2024-04-20T16:45:13.082 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-20T16:45:13.082 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench} --name yuriw-2024-04-20_15:32:38-rados-wip-yuriw-testing-20240419.185239-main-distro-default-smithi --verbose --archive /home/teuthworker/archive/yuriw-2024-04-20_15:32:38-rados-wip-yuriw-testing-20240419.185239-main-distro-default-smithi/7664810 --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-04-20_15:32:38-rados-wip-yuriw-testing-20240419.185239-main-distro-default-smithi/7664810/orig.config.yaml 2024-04-20T16:45:13.127 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-20T16:45:13.209 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-20_15:32:38-rados-wip-yuriw-testing-20240419.185239-main-distro-default-smithi/7664810 branch: wip-yuriw-testing-20240419.185239-main description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench} email: yweinste@redhat.com first_in_suite: false job_id: '7664810' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-04-20_15:32:38-rados-wip-yuriw-testing-20240419.185239-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuriw-testing-20240419.185239-main ceph: conf: client.2: debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd_pool_default_min_size: 2 osd_pool_default_size: 2 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd heartbeat use min delay socket: true osd map cache size: 1 osd max backfills: 6 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd scrub during recovery: false osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - osd_map_cache_size - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum mon_bind_msgr2: false sha1: 36c371567dddacf6207ea36f2535396ab31415fc ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 36c371567dddacf6207ea36f2535396ab31415fc selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuriw-testing-20240419.185239-main sha1: 36c371567dddacf6207ea36f2535396ab31415fc owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 317 sha1: 36c371567dddacf6207ea36f2535396ab31415fc sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuriw-testing-20240419.185239-main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_36c371567dddacf6207ea36f2535396ab31415fc/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 36c371567dddacf6207ea36f2535396ab31415fc targets: smithi005.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJfMFwSPgQGEdAO4p0iQci9eIYTRPcXnPfeUnyUQDTzKQGpqpb6bqvXpPOdoZpD4EQK3+wMhMU292jcNygMRYLA= smithi106.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHbMICTpir/0oEL5S4+LyKYMdSx5m/gasVlEaOPGrfpQCtBJyThqhcPoCqqv5hdQBNOEqldwehx0hojv7O3QO4c= smithi171.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMDFTDumLh8k/iLaIxuZqDoERBk9s1/MtgV73jCR0fPJzKr9Cjh/Q6E4+9kzar6GnzAjr7VdItI/eS8o3x3xzCE= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: nautilus exclude_packages: - cephadm - ceph-mgr-cephadm - ceph-immutable-object-cache - python3-rados - python3-rgw - python3-rbd - python3-cephfs - ceph-volume extra_packages: - python-rados - python-rgw - python-rbd - python-cephfs - cephadm: conf: mon: auth allow insecure global id reclaim: true - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph balancer mode crush-compat - ceph balancer on - thrashosds: chance_pgnum_grow: 0.25 chance_pgpnum_fix: 0.25 chance_test_map_discontinuity: 2 timeout: 1800 - full_sequential: - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-20_15:32:38 tube: smithi user: yuriw verbose: true verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-20T16:45:13.209 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_36c371567dddacf6207ea36f2535396ab31415fc/qa; will attempt to use it 2024-04-20T16:45:13.210 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_36c371567dddacf6207ea36f2535396ab31415fc/qa/tasks 2024-04-20T16:45:13.210 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-20T16:45:13.211 INFO:teuthology.task.internal:Saving configuration 2024-04-20T16:45:13.230 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-20T16:45:13.236 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-20T16:45:13.259 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi005.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-20_15:32:38-rados-wip-yuriw-testing-20240419.185239-main-distro-default-smithi/7664810', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-20 16:39:55.568854', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJfMFwSPgQGEdAO4p0iQci9eIYTRPcXnPfeUnyUQDTzKQGpqpb6bqvXpPOdoZpD4EQK3+wMhMU292jcNygMRYLA='} 2024-04-20T16:45:13.285 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi106.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-20_15:32:38-rados-wip-yuriw-testing-20240419.185239-main-distro-default-smithi/7664810', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-20 16:39:55.570207', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHbMICTpir/0oEL5S4+LyKYMdSx5m/gasVlEaOPGrfpQCtBJyThqhcPoCqqv5hdQBNOEqldwehx0hojv7O3QO4c='} 2024-04-20T16:45:13.306 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi171.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-20_15:32:38-rados-wip-yuriw-testing-20240419.185239-main-distro-default-smithi/7664810', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-20 16:39:55.571172', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMDFTDumLh8k/iLaIxuZqDoERBk9s1/MtgV73jCR0fPJzKr9Cjh/Q6E4+9kzar6GnzAjr7VdItI/eS8o3x3xzCE='} 2024-04-20T16:45:13.306 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-20T16:45:13.313 INFO:teuthology.task.internal:roles: ubuntu@smithi005.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-20T16:45:13.313 INFO:teuthology.task.internal:roles: ubuntu@smithi106.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-04-20T16:45:13.313 INFO:teuthology.task.internal:roles: ubuntu@smithi171.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-04-20T16:45:13.313 INFO:teuthology.run_tasks:Running task console_log... 2024-04-20T16:45:13.420 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9b85323f70>, signals=[15]) 2024-04-20T16:45:13.421 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-20T16:45:13.428 INFO:teuthology.task.internal:Opening connections... 2024-04-20T16:45:13.428 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi005.front.sepia.ceph.com 2024-04-20T16:45:13.431 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T16:45:13.509 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi106.front.sepia.ceph.com 2024-04-20T16:45:13.510 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T16:45:13.582 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi171.front.sepia.ceph.com 2024-04-20T16:45:13.583 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T16:45:13.656 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-20T16:45:13.662 DEBUG:teuthology.orchestra.run.smithi005:> uname -m 2024-04-20T16:45:13.686 INFO:teuthology.orchestra.run.smithi005.stdout:x86_64 2024-04-20T16:45:13.686 DEBUG:teuthology.orchestra.run.smithi005:> cat /etc/os-release 2024-04-20T16:45:13.749 INFO:teuthology.orchestra.run.smithi005.stdout:NAME="CentOS Stream" 2024-04-20T16:45:13.749 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION="8" 2024-04-20T16:45:13.749 INFO:teuthology.orchestra.run.smithi005.stdout:ID="centos" 2024-04-20T16:45:13.749 INFO:teuthology.orchestra.run.smithi005.stdout:ID_LIKE="rhel fedora" 2024-04-20T16:45:13.749 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION_ID="8" 2024-04-20T16:45:13.749 INFO:teuthology.orchestra.run.smithi005.stdout:PLATFORM_ID="platform:el8" 2024-04-20T16:45:13.749 INFO:teuthology.orchestra.run.smithi005.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-20T16:45:13.749 INFO:teuthology.orchestra.run.smithi005.stdout:ANSI_COLOR="0;31" 2024-04-20T16:45:13.749 INFO:teuthology.orchestra.run.smithi005.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-20T16:45:13.750 INFO:teuthology.orchestra.run.smithi005.stdout:HOME_URL="https://centos.org/" 2024-04-20T16:45:13.750 INFO:teuthology.orchestra.run.smithi005.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-20T16:45:13.750 INFO:teuthology.orchestra.run.smithi005.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-20T16:45:13.750 INFO:teuthology.orchestra.run.smithi005.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-20T16:45:13.750 INFO:teuthology.lock.ops:Updating smithi005.front.sepia.ceph.com on lock server 2024-04-20T16:45:13.787 DEBUG:teuthology.orchestra.run.smithi106:> uname -m 2024-04-20T16:45:13.810 INFO:teuthology.orchestra.run.smithi106.stdout:x86_64 2024-04-20T16:45:13.810 DEBUG:teuthology.orchestra.run.smithi106:> cat /etc/os-release 2024-04-20T16:45:13.872 INFO:teuthology.orchestra.run.smithi106.stdout:NAME="CentOS Stream" 2024-04-20T16:45:13.872 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION="8" 2024-04-20T16:45:13.872 INFO:teuthology.orchestra.run.smithi106.stdout:ID="centos" 2024-04-20T16:45:13.872 INFO:teuthology.orchestra.run.smithi106.stdout:ID_LIKE="rhel fedora" 2024-04-20T16:45:13.872 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION_ID="8" 2024-04-20T16:45:13.872 INFO:teuthology.orchestra.run.smithi106.stdout:PLATFORM_ID="platform:el8" 2024-04-20T16:45:13.872 INFO:teuthology.orchestra.run.smithi106.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-20T16:45:13.872 INFO:teuthology.orchestra.run.smithi106.stdout:ANSI_COLOR="0;31" 2024-04-20T16:45:13.872 INFO:teuthology.orchestra.run.smithi106.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-20T16:45:13.872 INFO:teuthology.orchestra.run.smithi106.stdout:HOME_URL="https://centos.org/" 2024-04-20T16:45:13.872 INFO:teuthology.orchestra.run.smithi106.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-20T16:45:13.872 INFO:teuthology.orchestra.run.smithi106.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-20T16:45:13.873 INFO:teuthology.orchestra.run.smithi106.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-20T16:45:13.873 INFO:teuthology.lock.ops:Updating smithi106.front.sepia.ceph.com on lock server 2024-04-20T16:45:13.896 DEBUG:teuthology.orchestra.run.smithi171:> uname -m 2024-04-20T16:45:13.918 INFO:teuthology.orchestra.run.smithi171.stdout:x86_64 2024-04-20T16:45:13.919 DEBUG:teuthology.orchestra.run.smithi171:> cat /etc/os-release 2024-04-20T16:45:13.981 INFO:teuthology.orchestra.run.smithi171.stdout:NAME="CentOS Stream" 2024-04-20T16:45:13.982 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION="8" 2024-04-20T16:45:13.982 INFO:teuthology.orchestra.run.smithi171.stdout:ID="centos" 2024-04-20T16:45:13.982 INFO:teuthology.orchestra.run.smithi171.stdout:ID_LIKE="rhel fedora" 2024-04-20T16:45:13.982 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION_ID="8" 2024-04-20T16:45:13.983 INFO:teuthology.orchestra.run.smithi171.stdout:PLATFORM_ID="platform:el8" 2024-04-20T16:45:13.983 INFO:teuthology.orchestra.run.smithi171.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-20T16:45:13.983 INFO:teuthology.orchestra.run.smithi171.stdout:ANSI_COLOR="0;31" 2024-04-20T16:45:13.983 INFO:teuthology.orchestra.run.smithi171.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-20T16:45:13.983 INFO:teuthology.orchestra.run.smithi171.stdout:HOME_URL="https://centos.org/" 2024-04-20T16:45:13.983 INFO:teuthology.orchestra.run.smithi171.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-20T16:45:13.984 INFO:teuthology.orchestra.run.smithi171.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-20T16:45:13.984 INFO:teuthology.orchestra.run.smithi171.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-20T16:45:13.984 INFO:teuthology.lock.ops:Updating smithi171.front.sepia.ceph.com on lock server 2024-04-20T16:45:14.013 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-20T16:45:14.021 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-20T16:45:14.026 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-20T16:45:14.026 DEBUG:teuthology.orchestra.run.smithi005:> test '!' -e /home/ubuntu/cephtest 2024-04-20T16:45:14.029 DEBUG:teuthology.orchestra.run.smithi106:> test '!' -e /home/ubuntu/cephtest 2024-04-20T16:45:14.031 DEBUG:teuthology.orchestra.run.smithi171:> test '!' -e /home/ubuntu/cephtest 2024-04-20T16:45:14.050 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-20T16:45:14.056 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-20T16:45:14.056 DEBUG:teuthology.orchestra.run.smithi005:> test -z $(ls -A /var/lib/ceph) 2024-04-20T16:45:14.091 DEBUG:teuthology.orchestra.run.smithi106:> test -z $(ls -A /var/lib/ceph) 2024-04-20T16:45:14.094 DEBUG:teuthology.orchestra.run.smithi171:> test -z $(ls -A /var/lib/ceph) 2024-04-20T16:45:14.123 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-20T16:45:14.179 INFO:teuthology.run_tasks:Running task kernel... 2024-04-20T16:45:14.197 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-20T16:45:14.197 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-20T16:45:14.198 DEBUG:teuthology.orchestra.run.smithi005:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-20T16:45:14.198 DEBUG:teuthology.orchestra.run.smithi106:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-20T16:45:14.199 DEBUG:teuthology.orchestra.run.smithi171:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-20T16:45:14.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T16:45:14.220 DEBUG:teuthology.orchestra.run.smithi171:> uname -r 2024-04-20T16:45:14.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T16:45:14.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T16:45:14.223 DEBUG:teuthology.orchestra.run.smithi005:> uname -r 2024-04-20T16:45:14.223 DEBUG:teuthology.orchestra.run.smithi106:> uname -r 2024-04-20T16:45:14.280 INFO:teuthology.orchestra.run.smithi171.stdout:4.18.0-489.el8.x86_64 2024-04-20T16:45:14.281 INFO:teuthology.task.kernel:Running kernel on smithi171: 4.18.0-489.el8.x86_64 2024-04-20T16:45:14.281 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum install -y kernel 2024-04-20T16:45:14.286 INFO:teuthology.orchestra.run.smithi106.stdout:4.18.0-489.el8.x86_64 2024-04-20T16:45:14.287 INFO:teuthology.task.kernel:Running kernel on smithi106: 4.18.0-489.el8.x86_64 2024-04-20T16:45:14.287 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum install -y kernel 2024-04-20T16:45:14.288 INFO:teuthology.orchestra.run.smithi005.stdout:4.18.0-489.el8.x86_64 2024-04-20T16:45:14.288 INFO:teuthology.task.kernel:Running kernel on smithi005: 4.18.0-489.el8.x86_64 2024-04-20T16:45:14.288 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum install -y kernel 2024-04-20T16:45:18.802 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 8 - AppStream 14 MB/s | 28 MB 00:01 2024-04-20T16:45:19.058 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 8 - AppStream 13 MB/s | 28 MB 00:02 2024-04-20T16:45:19.070 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 8 - AppStream 13 MB/s | 28 MB 00:02 2024-04-20T16:45:24.512 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 8 - BaseOS 22 MB/s | 10 MB 00:00 2024-04-20T16:45:24.686 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 8 - BaseOS 20 MB/s | 10 MB 00:00 2024-04-20T16:45:24.717 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 8 - BaseOS 22 MB/s | 10 MB 00:00 2024-04-20T16:45:26.770 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 8 - Extras 48 kB/s | 18 kB 00:00 2024-04-20T16:45:27.505 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 8 - Extras 13 kB/s | 18 kB 00:01 2024-04-20T16:45:27.792 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 8 - Extras 13 kB/s | 18 kB 00:01 2024-04-20T16:45:27.871 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 8 - Extras common packages 30 kB/s | 7.7 kB 00:00 2024-04-20T16:45:28.195 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 8 - Extras common packages 5.9 kB/s | 7.7 kB 00:01 2024-04-20T16:45:28.206 INFO:teuthology.orchestra.run.smithi106.stdout:Copr repo for python3-asyncssh owned by ceph 14 kB/s | 3.5 kB 00:00 2024-04-20T16:45:28.409 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 8 - Extras common packages 15 kB/s | 7.7 kB 00:00 2024-04-20T16:45:28.455 INFO:teuthology.orchestra.run.smithi005.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2024-04-20T16:45:28.677 INFO:teuthology.orchestra.run.smithi171.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2024-04-20T16:45:30.022 INFO:teuthology.orchestra.run.smithi171.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2024-04-20T16:45:30.085 INFO:teuthology.orchestra.run.smithi106.stdout:Extra Packages for Enterprise Linux 9.1 MB/s | 16 MB 00:01 2024-04-20T16:45:31.356 INFO:teuthology.orchestra.run.smithi005.stdout:Extra Packages for Enterprise Linux 5.8 MB/s | 16 MB 00:02 2024-04-20T16:45:33.848 INFO:teuthology.orchestra.run.smithi106.stdout:lab-extras 453 kB/s | 24 kB 00:00 2024-04-20T16:45:33.853 INFO:teuthology.orchestra.run.smithi171.stdout:lab-extras 517 kB/s | 24 kB 00:00 2024-04-20T16:45:35.314 INFO:teuthology.orchestra.run.smithi005.stdout:lab-extras 436 kB/s | 24 kB 00:00 2024-04-20T16:45:36.808 INFO:teuthology.orchestra.run.smithi171.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-20T16:45:36.808 INFO:teuthology.orchestra.run.smithi171.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-20T16:45:36.920 INFO:teuthology.orchestra.run.smithi106.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-20T16:45:36.921 INFO:teuthology.orchestra.run.smithi106.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-20T16:45:36.942 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-04-20T16:45:36.943 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:45:36.943 INFO:teuthology.orchestra.run.smithi171.stdout: Package Architecture Version Repository Size 2024-04-20T16:45:36.943 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:45:36.943 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-04-20T16:45:36.943 INFO:teuthology.orchestra.run.smithi171.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-20T16:45:36.943 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-04-20T16:45:36.943 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-20T16:45:36.944 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-20T16:45:36.944 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:45:36.944 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-04-20T16:45:36.944 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:45:36.944 INFO:teuthology.orchestra.run.smithi171.stdout:Install 3 Packages 2024-04-20T16:45:36.944 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:45:36.944 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 90 M 2024-04-20T16:45:36.944 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 96 M 2024-04-20T16:45:36.945 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-04-20T16:45:37.038 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-20T16:45:37.039 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:45:37.039 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-04-20T16:45:37.039 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:45:37.039 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-20T16:45:37.040 INFO:teuthology.orchestra.run.smithi106.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-20T16:45:37.040 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-04-20T16:45:37.040 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-20T16:45:37.040 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-20T16:45:37.040 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:45:37.040 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-20T16:45:37.040 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:45:37.040 INFO:teuthology.orchestra.run.smithi106.stdout:Install 3 Packages 2024-04-20T16:45:37.040 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:45:37.041 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 90 M 2024-04-20T16:45:37.041 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 96 M 2024-04-20T16:45:37.041 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-20T16:45:37.346 INFO:teuthology.orchestra.run.smithi171.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 31 MB/s | 10 MB 00:00 2024-04-20T16:45:37.884 INFO:teuthology.orchestra.run.smithi106.stdout:(1/3): kernel-core-4.18.0-552.el8.x86_64.rpm 56 MB/s | 44 MB 00:00 2024-04-20T16:45:37.930 INFO:teuthology.orchestra.run.smithi171.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 40 MB/s | 36 MB 00:00 2024-04-20T16:45:37.992 INFO:teuthology.orchestra.run.smithi106.stdout:(2/3): kernel-4.18.0-552.el8.x86_64.rpm 12 MB/s | 10 MB 00:00 2024-04-20T16:45:38.380 INFO:teuthology.orchestra.run.smithi171.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 32 MB/s | 44 MB 00:01 2024-04-20T16:45:38.380 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:45:38.380 INFO:teuthology.orchestra.run.smithi171.stdout:Total 63 MB/s | 90 MB 00:01 2024-04-20T16:45:38.426 INFO:teuthology.orchestra.run.smithi106.stdout:(3/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 28 MB/s | 36 MB 00:01 2024-04-20T16:45:38.426 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:45:38.426 INFO:teuthology.orchestra.run.smithi106.stdout:Total 65 MB/s | 90 MB 00:01 2024-04-20T16:45:38.578 INFO:teuthology.orchestra.run.smithi005.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-20T16:45:38.578 INFO:teuthology.orchestra.run.smithi005.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-20T16:45:38.706 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-04-20T16:45:38.708 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:45:38.708 INFO:teuthology.orchestra.run.smithi005.stdout: Package Architecture Version Repository Size 2024-04-20T16:45:38.708 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:45:38.708 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-04-20T16:45:38.708 INFO:teuthology.orchestra.run.smithi005.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-20T16:45:38.708 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-04-20T16:45:38.708 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-20T16:45:38.708 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-20T16:45:38.708 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:45:38.708 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-04-20T16:45:38.708 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:45:38.709 INFO:teuthology.orchestra.run.smithi005.stdout:Install 3 Packages 2024-04-20T16:45:38.709 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:45:38.709 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 90 M 2024-04-20T16:45:38.709 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 96 M 2024-04-20T16:45:38.709 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-04-20T16:45:39.062 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-04-20T16:45:39.106 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-20T16:45:39.199 INFO:teuthology.orchestra.run.smithi005.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 44 MB/s | 10 MB 00:00 2024-04-20T16:45:39.499 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-04-20T16:45:39.499 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-04-20T16:45:39.524 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-20T16:45:39.525 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-20T16:45:39.841 INFO:teuthology.orchestra.run.smithi005.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 41 MB/s | 36 MB 00:00 2024-04-20T16:45:40.283 INFO:teuthology.orchestra.run.smithi005.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 33 MB/s | 44 MB 00:01 2024-04-20T16:45:40.283 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:45:40.283 INFO:teuthology.orchestra.run.smithi005.stdout:Total 57 MB/s | 90 MB 00:01 2024-04-20T16:45:40.747 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-04-20T16:45:40.759 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-20T16:45:40.776 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-04-20T16:45:40.783 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-20T16:45:41.000 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-04-20T16:45:41.460 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-04-20T16:45:41.461 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-04-20T16:45:42.750 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-04-20T16:45:42.771 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-04-20T16:45:43.545 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-20T16:45:43.741 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-04-20T16:45:44.114 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-20T16:45:44.267 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-20T16:45:45.719 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-04-20T16:45:46.217 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-20T16:45:47.695 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-20T16:45:47.967 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-20T16:45:48.105 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-20T16:45:48.458 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-20T16:45:49.964 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-20T16:45:50.462 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-20T16:45:51.535 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-20T16:45:51.827 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-20T16:45:51.899 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-20T16:45:52.190 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-20T16:45:54.143 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-20T16:45:54.429 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-20T16:46:13.823 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-20T16:46:13.846 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-20T16:46:14.334 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-20T16:46:14.335 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-20T16:46:14.335 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-20T16:46:14.352 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-20T16:46:14.377 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-20T16:46:14.726 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-20T16:46:14.726 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-20T16:46:14.727 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-20T16:46:14.812 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-20T16:46:14.812 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:46:14.812 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-20T16:46:14.812 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-20T16:46:14.812 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-20T16:46:14.812 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:46:14.812 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-20T16:46:14.980 DEBUG:teuthology.orchestra.run.smithi106:> rpm -q kernel | sort -rV | head -n 1 2024-04-20T16:46:15.193 INFO:teuthology.orchestra.run.smithi106.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-20T16:46:15.193 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-20T16:46:15.193 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-20T16:46:15.194 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-20T16:46:15.194 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-20T16:46:15.194 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-20T16:46:15.194 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi106.front.sepia.ceph.com, path=None, version=distro) 2024-04-20T16:46:15.194 DEBUG:teuthology.orchestra.run.smithi106:> rpm -q kernel | sort -rV | head -n 1 2024-04-20T16:46:15.403 INFO:teuthology.orchestra.run.smithi106.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-20T16:46:15.403 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-20T16:46:15.404 DEBUG:teuthology.orchestra.run.smithi106:> sudo rpm -qi grub2-tools 2024-04-20T16:46:15.437 INFO:teuthology.orchestra.run.smithi106.stdout:Name : grub2-tools 2024-04-20T16:46:15.437 INFO:teuthology.orchestra.run.smithi106.stdout:Epoch : 1 2024-04-20T16:46:15.437 INFO:teuthology.orchestra.run.smithi106.stdout:Version : 2.02 2024-04-20T16:46:15.437 INFO:teuthology.orchestra.run.smithi106.stdout:Release : 129.el8 2024-04-20T16:46:15.437 INFO:teuthology.orchestra.run.smithi106.stdout:Architecture: x86_64 2024-04-20T16:46:15.437 INFO:teuthology.orchestra.run.smithi106.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-20T16:46:15.437 INFO:teuthology.orchestra.run.smithi106.stdout:Group : System Environment/Base 2024-04-20T16:46:15.437 INFO:teuthology.orchestra.run.smithi106.stdout:Size : 9265586 2024-04-20T16:46:15.437 INFO:teuthology.orchestra.run.smithi106.stdout:License : GPLv3+ 2024-04-20T16:46:15.437 INFO:teuthology.orchestra.run.smithi106.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-20T16:46:15.437 INFO:teuthology.orchestra.run.smithi106.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-20T16:46:15.437 INFO:teuthology.orchestra.run.smithi106.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-20T16:46:15.438 INFO:teuthology.orchestra.run.smithi106.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-20T16:46:15.438 INFO:teuthology.orchestra.run.smithi106.stdout:Relocations : (not relocatable) 2024-04-20T16:46:15.438 INFO:teuthology.orchestra.run.smithi106.stdout:Packager : CentOS BuildSystem 2024-04-20T16:46:15.438 INFO:teuthology.orchestra.run.smithi106.stdout:Vendor : CentOS 2024-04-20T16:46:15.438 INFO:teuthology.orchestra.run.smithi106.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-20T16:46:15.438 INFO:teuthology.orchestra.run.smithi106.stdout:Summary : Support tools for GRUB. 2024-04-20T16:46:15.438 INFO:teuthology.orchestra.run.smithi106.stdout:Description : 2024-04-20T16:46:15.438 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:46:15.438 INFO:teuthology.orchestra.run.smithi106.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-20T16:46:15.438 INFO:teuthology.orchestra.run.smithi106.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-20T16:46:15.438 INFO:teuthology.orchestra.run.smithi106.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-20T16:46:15.438 INFO:teuthology.orchestra.run.smithi106.stdout:hardware devices. 2024-04-20T16:46:15.438 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:46:15.438 INFO:teuthology.orchestra.run.smithi106.stdout:This subpackage provides tools for support of all platforms. 2024-04-20T16:46:15.439 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-20T16:46:15.439 INFO:teuthology.task.kernel:Updating grub on smithi106 to boot 4.18.0-552.el8.x86_64 2024-04-20T16:46:15.440 DEBUG:teuthology.orchestra.run.smithi106:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-20T16:46:15.683 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-20T16:46:15.683 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:46:15.683 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-04-20T16:46:15.683 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-20T16:46:15.683 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-20T16:46:15.683 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:46:15.683 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-04-20T16:46:15.852 DEBUG:teuthology.orchestra.run.smithi171:> rpm -q kernel | sort -rV | head -n 1 2024-04-20T16:46:16.066 INFO:teuthology.orchestra.run.smithi171.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-20T16:46:16.066 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-20T16:46:16.066 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-20T16:46:16.066 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-20T16:46:16.066 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-20T16:46:16.066 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-04-20T16:46:16.066 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi171.front.sepia.ceph.com, path=None, version=distro) 2024-04-20T16:46:16.067 DEBUG:teuthology.orchestra.run.smithi171:> rpm -q kernel | sort -rV | head -n 1 2024-04-20T16:46:16.277 INFO:teuthology.orchestra.run.smithi171.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-20T16:46:16.278 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-20T16:46:16.278 DEBUG:teuthology.orchestra.run.smithi171:> sudo rpm -qi grub2-tools 2024-04-20T16:46:16.311 INFO:teuthology.orchestra.run.smithi171.stdout:Name : grub2-tools 2024-04-20T16:46:16.311 INFO:teuthology.orchestra.run.smithi171.stdout:Epoch : 1 2024-04-20T16:46:16.311 INFO:teuthology.orchestra.run.smithi171.stdout:Version : 2.02 2024-04-20T16:46:16.312 INFO:teuthology.orchestra.run.smithi171.stdout:Release : 129.el8 2024-04-20T16:46:16.312 INFO:teuthology.orchestra.run.smithi171.stdout:Architecture: x86_64 2024-04-20T16:46:16.312 INFO:teuthology.orchestra.run.smithi171.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-20T16:46:16.312 INFO:teuthology.orchestra.run.smithi171.stdout:Group : System Environment/Base 2024-04-20T16:46:16.312 INFO:teuthology.orchestra.run.smithi171.stdout:Size : 9265586 2024-04-20T16:46:16.312 INFO:teuthology.orchestra.run.smithi171.stdout:License : GPLv3+ 2024-04-20T16:46:16.312 INFO:teuthology.orchestra.run.smithi171.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-20T16:46:16.312 INFO:teuthology.orchestra.run.smithi171.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-20T16:46:16.312 INFO:teuthology.orchestra.run.smithi171.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-20T16:46:16.312 INFO:teuthology.orchestra.run.smithi171.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-20T16:46:16.312 INFO:teuthology.orchestra.run.smithi171.stdout:Relocations : (not relocatable) 2024-04-20T16:46:16.312 INFO:teuthology.orchestra.run.smithi171.stdout:Packager : CentOS BuildSystem 2024-04-20T16:46:16.312 INFO:teuthology.orchestra.run.smithi171.stdout:Vendor : CentOS 2024-04-20T16:46:16.312 INFO:teuthology.orchestra.run.smithi171.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-20T16:46:16.312 INFO:teuthology.orchestra.run.smithi171.stdout:Summary : Support tools for GRUB. 2024-04-20T16:46:16.313 INFO:teuthology.orchestra.run.smithi171.stdout:Description : 2024-04-20T16:46:16.313 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:46:16.313 INFO:teuthology.orchestra.run.smithi171.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-20T16:46:16.313 INFO:teuthology.orchestra.run.smithi171.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-20T16:46:16.313 INFO:teuthology.orchestra.run.smithi171.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-20T16:46:16.313 INFO:teuthology.orchestra.run.smithi171.stdout:hardware devices. 2024-04-20T16:46:16.313 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:46:16.313 INFO:teuthology.orchestra.run.smithi171.stdout:This subpackage provides tools for support of all platforms. 2024-04-20T16:46:16.314 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-20T16:46:16.314 INFO:teuthology.task.kernel:Updating grub on smithi171 to boot 4.18.0-552.el8.x86_64 2024-04-20T16:46:16.314 DEBUG:teuthology.orchestra.run.smithi171:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-20T16:46:16.864 INFO:teuthology.orchestra.run.smithi106.stderr:Generating grub configuration file ... 2024-04-20T16:46:17.753 INFO:teuthology.orchestra.run.smithi171.stderr:Generating grub configuration file ... 2024-04-20T16:46:18.196 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-20T16:46:18.221 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-20T16:46:18.584 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-20T16:46:18.584 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-20T16:46:18.584 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-20T16:46:18.738 INFO:teuthology.orchestra.run.smithi106.stderr:done 2024-04-20T16:46:18.739 DEBUG:teuthology.orchestra.run.smithi106:> mktemp 2024-04-20T16:46:18.763 INFO:teuthology.orchestra.run.smithi106.stdout:/tmp/tmp.QpuoF6wccP 2024-04-20T16:46:18.763 DEBUG:teuthology.orchestra.run.smithi106:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.QpuoF6wccP 2024-04-20T16:46:18.837 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0666 /tmp/tmp.QpuoF6wccP 2024-04-20T16:46:18.976 DEBUG:teuthology.orchestra.remote:smithi106:/tmp/tmp.QpuoF6wccP is 6KB 2024-04-20T16:46:19.024 DEBUG:teuthology.orchestra.run.smithi106:> rm -fr /tmp/tmp.QpuoF6wccP 2024-04-20T16:46:19.041 DEBUG:teuthology.orchestra.run.smithi106:> sudo /bin/ls /boot/loader/entries || true 2024-04-20T16:46:19.106 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-20T16:46:19.106 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:46:19.106 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-04-20T16:46:19.107 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-20T16:46:19.107 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-20T16:46:19.107 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:46:19.107 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-04-20T16:46:19.111 INFO:teuthology.orchestra.run.smithi106.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-20T16:46:19.111 INFO:teuthology.orchestra.run.smithi106.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-20T16:46:19.111 INFO:teuthology.orchestra.run.smithi106.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-20T16:46:19.111 INFO:teuthology.orchestra.run.smithi106.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-20T16:46:19.112 DEBUG:teuthology.orchestra.run.smithi106:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-20T16:46:19.251 DEBUG:teuthology.orchestra.run.smithi005:> rpm -q kernel | sort -rV | head -n 1 2024-04-20T16:46:19.280 DEBUG:teuthology.orchestra.run.smithi106:> sudo shutdown -r now 2024-04-20T16:46:19.479 INFO:teuthology.orchestra.run.smithi005.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-20T16:46:19.479 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-20T16:46:19.479 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-20T16:46:19.479 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-20T16:46:19.480 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-20T16:46:19.480 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-20T16:46:19.480 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi005.front.sepia.ceph.com, path=None, version=distro) 2024-04-20T16:46:19.480 DEBUG:teuthology.orchestra.run.smithi005:> rpm -q kernel | sort -rV | head -n 1 2024-04-20T16:46:19.700 INFO:teuthology.orchestra.run.smithi005.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-20T16:46:19.700 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-20T16:46:19.700 DEBUG:teuthology.orchestra.run.smithi005:> sudo rpm -qi grub2-tools 2024-04-20T16:46:19.734 INFO:teuthology.orchestra.run.smithi171.stderr:done 2024-04-20T16:46:19.735 DEBUG:teuthology.orchestra.run.smithi171:> mktemp 2024-04-20T16:46:19.736 INFO:teuthology.orchestra.run.smithi005.stdout:Name : grub2-tools 2024-04-20T16:46:19.736 INFO:teuthology.orchestra.run.smithi005.stdout:Epoch : 1 2024-04-20T16:46:19.736 INFO:teuthology.orchestra.run.smithi005.stdout:Version : 2.02 2024-04-20T16:46:19.736 INFO:teuthology.orchestra.run.smithi005.stdout:Release : 129.el8 2024-04-20T16:46:19.736 INFO:teuthology.orchestra.run.smithi005.stdout:Architecture: x86_64 2024-04-20T16:46:19.736 INFO:teuthology.orchestra.run.smithi005.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-20T16:46:19.736 INFO:teuthology.orchestra.run.smithi005.stdout:Group : System Environment/Base 2024-04-20T16:46:19.736 INFO:teuthology.orchestra.run.smithi005.stdout:Size : 9265586 2024-04-20T16:46:19.736 INFO:teuthology.orchestra.run.smithi005.stdout:License : GPLv3+ 2024-04-20T16:46:19.737 INFO:teuthology.orchestra.run.smithi005.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-20T16:46:19.737 INFO:teuthology.orchestra.run.smithi005.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-20T16:46:19.737 INFO:teuthology.orchestra.run.smithi005.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-20T16:46:19.737 INFO:teuthology.orchestra.run.smithi005.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-20T16:46:19.737 INFO:teuthology.orchestra.run.smithi005.stdout:Relocations : (not relocatable) 2024-04-20T16:46:19.737 INFO:teuthology.orchestra.run.smithi005.stdout:Packager : CentOS BuildSystem 2024-04-20T16:46:19.737 INFO:teuthology.orchestra.run.smithi005.stdout:Vendor : CentOS 2024-04-20T16:46:19.737 INFO:teuthology.orchestra.run.smithi005.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-20T16:46:19.737 INFO:teuthology.orchestra.run.smithi005.stdout:Summary : Support tools for GRUB. 2024-04-20T16:46:19.737 INFO:teuthology.orchestra.run.smithi005.stdout:Description : 2024-04-20T16:46:19.737 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:46:19.737 INFO:teuthology.orchestra.run.smithi005.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-20T16:46:19.738 INFO:teuthology.orchestra.run.smithi005.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-20T16:46:19.738 INFO:teuthology.orchestra.run.smithi005.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-20T16:46:19.738 INFO:teuthology.orchestra.run.smithi005.stdout:hardware devices. 2024-04-20T16:46:19.738 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:46:19.738 INFO:teuthology.orchestra.run.smithi005.stdout:This subpackage provides tools for support of all platforms. 2024-04-20T16:46:19.739 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-20T16:46:19.739 INFO:teuthology.task.kernel:Updating grub on smithi005 to boot 4.18.0-552.el8.x86_64 2024-04-20T16:46:19.739 DEBUG:teuthology.orchestra.run.smithi005:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-20T16:46:19.767 INFO:teuthology.orchestra.run.smithi171.stdout:/tmp/tmp.64dYzTbIEm 2024-04-20T16:46:19.767 DEBUG:teuthology.orchestra.run.smithi171:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.64dYzTbIEm 2024-04-20T16:46:19.841 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 0666 /tmp/tmp.64dYzTbIEm 2024-04-20T16:46:19.983 DEBUG:teuthology.orchestra.remote:smithi171:/tmp/tmp.64dYzTbIEm is 6KB 2024-04-20T16:46:20.032 DEBUG:teuthology.orchestra.run.smithi171:> rm -fr /tmp/tmp.64dYzTbIEm 2024-04-20T16:46:20.049 DEBUG:teuthology.orchestra.run.smithi171:> sudo /bin/ls /boot/loader/entries || true 2024-04-20T16:46:20.114 INFO:teuthology.orchestra.run.smithi171.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-20T16:46:20.114 INFO:teuthology.orchestra.run.smithi171.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-20T16:46:20.114 INFO:teuthology.orchestra.run.smithi171.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-20T16:46:20.114 INFO:teuthology.orchestra.run.smithi171.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-20T16:46:20.115 DEBUG:teuthology.orchestra.run.smithi171:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-20T16:46:20.299 DEBUG:teuthology.orchestra.run.smithi171:> sudo shutdown -r now 2024-04-20T16:46:21.123 INFO:teuthology.orchestra.run.smithi005.stderr:Generating grub configuration file ... 2024-04-20T16:46:23.321 INFO:teuthology.orchestra.run.smithi005.stderr:done 2024-04-20T16:46:23.323 DEBUG:teuthology.orchestra.run.smithi005:> mktemp 2024-04-20T16:46:23.363 INFO:teuthology.orchestra.run.smithi005.stdout:/tmp/tmp.w0A6L4bKMi 2024-04-20T16:46:23.364 DEBUG:teuthology.orchestra.run.smithi005:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.w0A6L4bKMi 2024-04-20T16:46:23.436 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0666 /tmp/tmp.w0A6L4bKMi 2024-04-20T16:46:23.583 DEBUG:teuthology.orchestra.remote:smithi005:/tmp/tmp.w0A6L4bKMi is 6KB 2024-04-20T16:46:23.630 DEBUG:teuthology.orchestra.run.smithi005:> rm -fr /tmp/tmp.w0A6L4bKMi 2024-04-20T16:46:23.648 DEBUG:teuthology.orchestra.run.smithi005:> sudo /bin/ls /boot/loader/entries || true 2024-04-20T16:46:23.713 INFO:teuthology.orchestra.run.smithi005.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-20T16:46:23.713 INFO:teuthology.orchestra.run.smithi005.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-20T16:46:23.714 INFO:teuthology.orchestra.run.smithi005.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-20T16:46:23.714 INFO:teuthology.orchestra.run.smithi005.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-20T16:46:23.715 DEBUG:teuthology.orchestra.run.smithi005:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-20T16:46:23.876 DEBUG:teuthology.orchestra.run.smithi005:> sudo shutdown -r now 2024-04-20T16:46:49.306 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-20T16:46:49.307 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-04-20T16:46:49.308 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T16:46:50.304 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-04-20T16:46:50.304 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-04-20T16:46:50.304 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T16:46:53.882 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-20T16:46:53.882 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-04-20T16:46:53.883 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T16:47:07.806 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.106 2024-04-20T16:47:08.830 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.171 2024-04-20T16:47:12.418 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.5 2024-04-20T16:47:16.810 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-04-20T16:47:16.811 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T16:47:17.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-04-20T16:47:17.832 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T16:47:19.870 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.106 2024-04-20T16:47:20.894 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.171 2024-04-20T16:47:21.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-04-20T16:47:21.420 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T16:47:31.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-04-20T16:47:31.879 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T16:47:32.896 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-04-20T16:47:32.897 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T16:47:39.806 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.5 2024-04-20T16:47:50.302 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.106 2024-04-20T16:47:51.326 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.171 2024-04-20T16:47:51.807 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-04-20T16:47:51.808 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T16:47:53.028 DEBUG:teuthology.orchestra.run.smithi005:> true 2024-04-20T16:47:53.376 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-04-20T16:47:53.376 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-552.el8.x86_64"... 2024-04-20T16:47:53.376 DEBUG:teuthology.orchestra.run.smithi005:> uname -r 2024-04-20T16:47:53.396 INFO:teuthology.orchestra.run.smithi005.stdout:4.18.0-552.el8.x86_64 2024-04-20T16:47:53.396 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-20T16:47:53.396 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-20T16:47:53.396 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-20T16:47:54.397 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-20T16:47:54.397 DEBUG:teuthology.orchestra.run.smithi005:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-20T16:47:54.503 INFO:teuthology.orchestra.run.smithi005.stdout:ttyS1 2024-04-20T16:47:54.518 DEBUG:teuthology.parallel:result is None 2024-04-20T16:48:05.303 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-04-20T16:48:05.303 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T16:48:06.027 DEBUG:teuthology.orchestra.run.smithi106:> true 2024-04-20T16:48:06.328 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-04-20T16:48:06.328 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T16:48:06.614 DEBUG:teuthology.orchestra.run.smithi171:> true 2024-04-20T16:48:06.923 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-04-20T16:48:06.924 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "4.18.0-552.el8.x86_64"... 2024-04-20T16:48:06.924 DEBUG:teuthology.orchestra.run.smithi171:> uname -r 2024-04-20T16:48:06.941 INFO:teuthology.orchestra.run.smithi171.stdout:4.18.0-552.el8.x86_64 2024-04-20T16:48:06.941 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-20T16:48:06.941 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-20T16:48:06.941 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-20T16:48:07.106 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-04-20T16:48:07.106 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-552.el8.x86_64"... 2024-04-20T16:48:07.106 DEBUG:teuthology.orchestra.run.smithi106:> uname -r 2024-04-20T16:48:07.122 INFO:teuthology.orchestra.run.smithi106.stdout:4.18.0-552.el8.x86_64 2024-04-20T16:48:07.122 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-20T16:48:07.122 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-20T16:48:07.122 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-20T16:48:07.942 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-04-20T16:48:07.943 DEBUG:teuthology.orchestra.run.smithi171:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-20T16:48:08.032 INFO:teuthology.orchestra.run.smithi171.stdout:ttyS1 2024-04-20T16:48:08.048 DEBUG:teuthology.parallel:result is None 2024-04-20T16:48:08.123 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-20T16:48:08.123 DEBUG:teuthology.orchestra.run.smithi106:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-20T16:48:08.183 INFO:teuthology.orchestra.run.smithi106.stdout:ttyS1 2024-04-20T16:48:08.195 DEBUG:teuthology.parallel:result is None 2024-04-20T16:48:08.195 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-20T16:48:08.206 INFO:teuthology.task.internal:Creating test directory... 2024-04-20T16:48:08.207 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-20T16:48:08.211 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-20T16:48:08.213 DEBUG:teuthology.orchestra.run.smithi171:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-20T16:48:08.233 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-20T16:48:08.276 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-20T16:48:08.292 INFO:teuthology.task.internal:Creating archive directory... 2024-04-20T16:48:08.293 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-20T16:48:08.295 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-20T16:48:08.297 DEBUG:teuthology.orchestra.run.smithi171:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-20T16:48:08.331 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-20T16:48:08.342 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-20T16:48:08.343 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-20T16:48:08.369 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-20T16:48:08.371 DEBUG:teuthology.orchestra.run.smithi171:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-20T16:48:08.403 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-20T16:48:08.410 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-20T16:48:08.413 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-20T16:48:08.415 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-20T16:48:08.421 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-20T16:48:08.426 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-20T16:48:08.427 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-20T16:48:08.439 INFO:teuthology.task.internal:Configuring sudo... 2024-04-20T16:48:08.439 DEBUG:teuthology.orchestra.run.smithi005:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-20T16:48:08.441 DEBUG:teuthology.orchestra.run.smithi106:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-20T16:48:08.458 DEBUG:teuthology.orchestra.run.smithi171:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-20T16:48:08.484 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-20T16:48:08.497 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-20T16:48:08.498 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-20T16:48:08.512 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-20T16:48:08.526 DEBUG:teuthology.orchestra.run.smithi171:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-20T16:48:08.544 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-20T16:48:08.611 DEBUG:teuthology.orchestra.run.smithi005:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-20T16:48:08.670 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-20T16:48:08.749 DEBUG:teuthology.orchestra.run.smithi005:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-20T16:48:08.802 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-04-20T16:48:08.803 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-20T16:48:08.876 DEBUG:teuthology.orchestra.run.smithi106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-20T16:48:08.916 DEBUG:teuthology.orchestra.run.smithi106:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-20T16:48:08.984 DEBUG:teuthology.orchestra.run.smithi106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-20T16:48:09.074 DEBUG:teuthology.orchestra.run.smithi106:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-20T16:48:09.124 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-20T16:48:09.124 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-20T16:48:09.206 DEBUG:teuthology.orchestra.run.smithi171:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-20T16:48:09.253 DEBUG:teuthology.orchestra.run.smithi171:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-20T16:48:09.315 DEBUG:teuthology.orchestra.run.smithi171:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-20T16:48:09.408 DEBUG:teuthology.orchestra.run.smithi171:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-20T16:48:09.439 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-04-20T16:48:09.439 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-20T16:48:09.503 DEBUG:teuthology.orchestra.run.smithi005:> sudo service rsyslog restart 2024-04-20T16:48:09.510 DEBUG:teuthology.orchestra.run.smithi106:> sudo service rsyslog restart 2024-04-20T16:48:09.519 DEBUG:teuthology.orchestra.run.smithi171:> sudo service rsyslog restart 2024-04-20T16:48:09.564 INFO:teuthology.orchestra.run.smithi106.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-20T16:48:09.595 INFO:teuthology.orchestra.run.smithi005.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-20T16:48:09.655 INFO:teuthology.orchestra.run.smithi171.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-20T16:48:10.053 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-20T16:48:10.064 INFO:teuthology.task.internal:Starting timer... 2024-04-20T16:48:10.064 INFO:teuthology.run_tasks:Running task pcp... 2024-04-20T16:48:10.079 INFO:teuthology.run_tasks:Running task selinux... 2024-04-20T16:48:10.116 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-20T16:48:10.116 DEBUG:teuthology.orchestra.run.smithi005:> sudo service auditd rotate 2024-04-20T16:48:10.208 INFO:teuthology.orchestra.run.smithi005.stdout:Rotating logs: 2024-04-20T16:48:10.211 DEBUG:teuthology.orchestra.run.smithi106:> sudo service auditd rotate 2024-04-20T16:48:10.289 INFO:teuthology.orchestra.run.smithi106.stdout:Rotating logs: 2024-04-20T16:48:10.290 DEBUG:teuthology.orchestra.run.smithi171:> sudo service auditd rotate 2024-04-20T16:48:10.988 INFO:teuthology.orchestra.run.smithi171.stdout:Rotating logs: 2024-04-20T16:48:11.012 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-20T16:48:11.013 DEBUG:teuthology.orchestra.run.smithi005:> /usr/sbin/getenforce 2024-04-20T16:48:11.040 INFO:teuthology.orchestra.run.smithi005.stdout:Permissive 2024-04-20T16:48:11.041 DEBUG:teuthology.orchestra.run.smithi106:> /usr/sbin/getenforce 2024-04-20T16:48:11.069 INFO:teuthology.orchestra.run.smithi106.stdout:Permissive 2024-04-20T16:48:11.069 DEBUG:teuthology.orchestra.run.smithi171:> /usr/sbin/getenforce 2024-04-20T16:48:11.550 INFO:teuthology.orchestra.run.smithi171.stdout:Permissive 2024-04-20T16:48:11.551 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi005.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi106.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi171.front.sepia.ceph.com': 'permissive'} 2024-04-20T16:48:11.551 DEBUG:teuthology.orchestra.run.smithi005:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-20T16:48:11.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T16:48:11.580 DEBUG:teuthology.orchestra.run.smithi106:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-20T16:48:11.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T16:48:11.608 DEBUG:teuthology.orchestra.run.smithi171:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-20T16:48:11.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T16:48:11.808 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-20T16:48:11.808 DEBUG:teuthology.orchestra.run.smithi005:> sudo /usr/sbin/setenforce permissive 2024-04-20T16:48:11.859 DEBUG:teuthology.orchestra.run.smithi106:> sudo /usr/sbin/setenforce permissive 2024-04-20T16:48:11.903 DEBUG:teuthology.orchestra.run.smithi171:> sudo /usr/sbin/setenforce permissive 2024-04-20T16:48:11.931 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-20T16:48:11.947 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-20T16:48:11.969 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-20T16:48:11.970 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi005.front.sepia.ceph.com,smithi106.front.sepia.ceph.com,smithi171.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-20T16:54:15.598 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi005.front.sepia.ceph.com'), Remote(name='ubuntu@smithi106.front.sepia.ceph.com'), Remote(name='ubuntu@smithi171.front.sepia.ceph.com')] 2024-04-20T16:54:15.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-04-20T16:54:15.600 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T16:54:15.676 DEBUG:teuthology.orchestra.run.smithi005:> true 2024-04-20T16:54:15.745 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-04-20T16:54:15.745 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-04-20T16:54:15.746 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T16:54:15.821 DEBUG:teuthology.orchestra.run.smithi106:> true 2024-04-20T16:54:15.888 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-04-20T16:54:15.888 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-04-20T16:54:15.889 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T16:54:15.962 DEBUG:teuthology.orchestra.run.smithi171:> true 2024-04-20T16:54:16.034 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-04-20T16:54:16.035 INFO:teuthology.run_tasks:Running task clock... 2024-04-20T16:54:16.045 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-20T16:54:16.045 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-20T16:54:16.046 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-20T16:54:16.048 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-20T16:54:16.049 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-20T16:54:16.051 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-20T16:54:16.051 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-20T16:54:16.080 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-20T16:54:16.080 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-20T16:54:16.094 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-20T16:54:16.095 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-20T16:54:16.108 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-20T16:54:16.123 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-20T16:54:16.169 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: ntpd: command not found 2024-04-20T16:54:16.178 INFO:teuthology.orchestra.run.smithi005.stderr:sudo: ntpd: command not found 2024-04-20T16:54:16.180 INFO:teuthology.orchestra.run.smithi106.stdout:506 Cannot talk to daemon 2024-04-20T16:54:16.183 INFO:teuthology.orchestra.run.smithi171.stderr:sudo: ntpd: command not found 2024-04-20T16:54:16.190 INFO:teuthology.orchestra.run.smithi005.stdout:506 Cannot talk to daemon 2024-04-20T16:54:16.194 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-20T16:54:16.194 INFO:teuthology.orchestra.run.smithi171.stdout:506 Cannot talk to daemon 2024-04-20T16:54:16.204 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-20T16:54:16.207 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-20T16:54:16.208 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-20T16:54:16.218 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-20T16:54:16.221 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-20T16:54:16.268 INFO:teuthology.orchestra.run.smithi106.stderr:bash: ntpq: command not found 2024-04-20T16:54:16.272 INFO:teuthology.orchestra.run.smithi106.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-20T16:54:16.272 INFO:teuthology.orchestra.run.smithi106.stdout:=============================================================================== 2024-04-20T16:54:16.272 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T16:54:16.272 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T16:54:16.272 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T16:54:16.272 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T16:54:16.283 INFO:teuthology.orchestra.run.smithi005.stderr:bash: ntpq: command not found 2024-04-20T16:54:16.286 INFO:teuthology.orchestra.run.smithi005.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-20T16:54:16.286 INFO:teuthology.orchestra.run.smithi005.stdout:=============================================================================== 2024-04-20T16:54:16.287 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T16:54:16.287 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T16:54:16.287 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T16:54:16.287 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T16:54:16.292 INFO:teuthology.orchestra.run.smithi171.stderr:bash: ntpq: command not found 2024-04-20T16:54:16.296 INFO:teuthology.orchestra.run.smithi171.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-20T16:54:16.296 INFO:teuthology.orchestra.run.smithi171.stdout:=============================================================================== 2024-04-20T16:54:16.296 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T16:54:16.296 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T16:54:16.296 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T16:54:16.296 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T16:54:16.297 INFO:teuthology.run_tasks:Running task pexec... 2024-04-20T16:54:16.316 INFO:teuthology.task.pexec:Executing custom commands... 2024-04-20T16:54:16.316 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi005.front.sepia.ceph.com 2024-04-20T16:54:16.317 DEBUG:teuthology.orchestra.run.smithi005:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-04-20T16:54:16.317 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi106.front.sepia.ceph.com 2024-04-20T16:54:16.317 DEBUG:teuthology.orchestra.run.smithi106:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-04-20T16:54:16.317 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi171.front.sepia.ceph.com 2024-04-20T16:54:16.318 DEBUG:teuthology.orchestra.run.smithi171:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-04-20T16:54:17.069 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:35 ago on Sat 20 Apr 2024 04:52:41 PM UTC. 2024-04-20T16:54:17.094 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:35 ago on Sat 20 Apr 2024 04:52:41 PM UTC. 2024-04-20T16:54:17.113 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:26 ago on Sat 20 Apr 2024 04:52:50 PM UTC. 2024-04-20T16:54:19.225 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-20T16:54:19.226 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:54:19.226 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-04-20T16:54:19.226 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:54:19.226 INFO:teuthology.orchestra.run.smithi106.stdout:Resetting modules: 2024-04-20T16:54:19.226 INFO:teuthology.orchestra.run.smithi106.stdout: container-tools 2024-04-20T16:54:19.227 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:54:19.227 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-20T16:54:19.227 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:54:19.227 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:54:19.227 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-04-20T16:54:19.228 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:54:19.228 INFO:teuthology.orchestra.run.smithi171.stdout: Package Architecture Version Repository Size 2024-04-20T16:54:19.229 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:54:19.229 INFO:teuthology.orchestra.run.smithi171.stdout:Resetting modules: 2024-04-20T16:54:19.229 INFO:teuthology.orchestra.run.smithi171.stdout: container-tools 2024-04-20T16:54:19.229 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:54:19.229 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-04-20T16:54:19.229 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:54:19.229 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:54:19.414 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-04-20T16:54:19.415 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:54:19.416 INFO:teuthology.orchestra.run.smithi005.stdout: Package Architecture Version Repository Size 2024-04-20T16:54:19.416 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:54:19.416 INFO:teuthology.orchestra.run.smithi005.stdout:Resetting modules: 2024-04-20T16:54:19.416 INFO:teuthology.orchestra.run.smithi005.stdout: container-tools 2024-04-20T16:54:19.416 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:54:19.416 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-04-20T16:54:19.417 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:54:19.417 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:54:19.473 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-20T16:54:19.478 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-04-20T16:54:19.713 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-04-20T16:54:20.247 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:39 ago on Sat 20 Apr 2024 04:52:41 PM UTC. 2024-04-20T16:54:20.248 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:39 ago on Sat 20 Apr 2024 04:52:41 PM UTC. 2024-04-20T16:54:20.529 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:30 ago on Sat 20 Apr 2024 04:52:50 PM UTC. 2024-04-20T16:54:22.358 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-20T16:54:22.395 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-04-20T16:54:22.396 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:54:22.396 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-04-20T16:54:22.396 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:54:22.396 INFO:teuthology.orchestra.run.smithi106.stdout:Installing group/module packages: 2024-04-20T16:54:22.396 INFO:teuthology.orchestra.run.smithi106.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-04-20T16:54:22.396 INFO:teuthology.orchestra.run.smithi106.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-04-20T16:54:22.396 INFO:teuthology.orchestra.run.smithi106.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-04-20T16:54:22.396 INFO:teuthology.orchestra.run.smithi106.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-04-20T16:54:22.396 INFO:teuthology.orchestra.run.smithi106.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-04-20T16:54:22.396 INFO:teuthology.orchestra.run.smithi106.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-04-20T16:54:22.396 INFO:teuthology.orchestra.run.smithi106.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-04-20T16:54:22.396 INFO:teuthology.orchestra.run.smithi106.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-04-20T16:54:22.396 INFO:teuthology.orchestra.run.smithi106.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-04-20T16:54:22.396 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-04-20T16:54:22.397 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-04-20T16:54:22.397 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-04-20T16:54:22.397 INFO:teuthology.orchestra.run.smithi106.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-04-20T16:54:22.397 INFO:teuthology.orchestra.run.smithi106.stdout:Installing weak dependencies: 2024-04-20T16:54:22.397 INFO:teuthology.orchestra.run.smithi106.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-04-20T16:54:22.397 INFO:teuthology.orchestra.run.smithi106.stdout:Installing module profiles: 2024-04-20T16:54:22.397 INFO:teuthology.orchestra.run.smithi106.stdout: container-tools/common 2024-04-20T16:54:22.397 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:54:22.397 INFO:teuthology.orchestra.run.smithi106.stdout:Enabling module streams: 2024-04-20T16:54:22.397 INFO:teuthology.orchestra.run.smithi106.stdout: container-tools rhel8 2024-04-20T16:54:22.397 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:54:22.397 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-20T16:54:22.397 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:54:22.397 INFO:teuthology.orchestra.run.smithi106.stdout:Install 13 Packages 2024-04-20T16:54:22.397 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:54:22.398 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 27 M 2024-04-20T16:54:22.398 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 91 M 2024-04-20T16:54:22.398 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-20T16:54:22.433 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:54:22.433 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-04-20T16:54:22.433 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:54:22.433 INFO:teuthology.orchestra.run.smithi171.stdout:Installing group/module packages: 2024-04-20T16:54:22.433 INFO:teuthology.orchestra.run.smithi171.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-04-20T16:54:22.433 INFO:teuthology.orchestra.run.smithi171.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-04-20T16:54:22.434 INFO:teuthology.orchestra.run.smithi171.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-04-20T16:54:22.434 INFO:teuthology.orchestra.run.smithi171.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-04-20T16:54:22.434 INFO:teuthology.orchestra.run.smithi171.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-04-20T16:54:22.434 INFO:teuthology.orchestra.run.smithi171.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-04-20T16:54:22.434 INFO:teuthology.orchestra.run.smithi171.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-04-20T16:54:22.434 INFO:teuthology.orchestra.run.smithi171.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-04-20T16:54:22.434 INFO:teuthology.orchestra.run.smithi171.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-04-20T16:54:22.434 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-04-20T16:54:22.434 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-04-20T16:54:22.434 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-04-20T16:54:22.434 INFO:teuthology.orchestra.run.smithi171.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-04-20T16:54:22.434 INFO:teuthology.orchestra.run.smithi171.stdout:Installing weak dependencies: 2024-04-20T16:54:22.434 INFO:teuthology.orchestra.run.smithi171.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-04-20T16:54:22.434 INFO:teuthology.orchestra.run.smithi171.stdout:Installing module profiles: 2024-04-20T16:54:22.434 INFO:teuthology.orchestra.run.smithi171.stdout: container-tools/common 2024-04-20T16:54:22.435 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:54:22.435 INFO:teuthology.orchestra.run.smithi171.stdout:Enabling module streams: 2024-04-20T16:54:22.435 INFO:teuthology.orchestra.run.smithi171.stdout: container-tools rhel8 2024-04-20T16:54:22.435 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:54:22.435 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-04-20T16:54:22.435 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:54:22.435 INFO:teuthology.orchestra.run.smithi171.stdout:Install 13 Packages 2024-04-20T16:54:22.435 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:54:22.435 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 27 M 2024-04-20T16:54:22.436 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 91 M 2024-04-20T16:54:22.436 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-04-20T16:54:22.820 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-04-20T16:54:22.862 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:54:22.862 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-04-20T16:54:22.863 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:54:22.863 INFO:teuthology.orchestra.run.smithi005.stdout:Installing group/module packages: 2024-04-20T16:54:22.863 INFO:teuthology.orchestra.run.smithi005.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-04-20T16:54:22.863 INFO:teuthology.orchestra.run.smithi005.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-04-20T16:54:22.863 INFO:teuthology.orchestra.run.smithi005.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-04-20T16:54:22.863 INFO:teuthology.orchestra.run.smithi005.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-04-20T16:54:22.863 INFO:teuthology.orchestra.run.smithi005.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-04-20T16:54:22.863 INFO:teuthology.orchestra.run.smithi005.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-04-20T16:54:22.863 INFO:teuthology.orchestra.run.smithi005.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-04-20T16:54:22.863 INFO:teuthology.orchestra.run.smithi005.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-04-20T16:54:22.863 INFO:teuthology.orchestra.run.smithi005.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-04-20T16:54:22.863 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-04-20T16:54:22.863 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-04-20T16:54:22.863 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-04-20T16:54:22.863 INFO:teuthology.orchestra.run.smithi005.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-04-20T16:54:22.864 INFO:teuthology.orchestra.run.smithi005.stdout:Installing weak dependencies: 2024-04-20T16:54:22.864 INFO:teuthology.orchestra.run.smithi005.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-04-20T16:54:22.864 INFO:teuthology.orchestra.run.smithi005.stdout:Installing module profiles: 2024-04-20T16:54:22.864 INFO:teuthology.orchestra.run.smithi005.stdout: container-tools/common 2024-04-20T16:54:22.864 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:54:22.864 INFO:teuthology.orchestra.run.smithi005.stdout:Enabling module streams: 2024-04-20T16:54:22.864 INFO:teuthology.orchestra.run.smithi005.stdout: container-tools rhel8 2024-04-20T16:54:22.864 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:54:22.864 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-04-20T16:54:22.864 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:54:22.864 INFO:teuthology.orchestra.run.smithi005.stdout:Install 13 Packages 2024-04-20T16:54:22.864 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:54:22.865 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 27 M 2024-04-20T16:54:22.865 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 91 M 2024-04-20T16:54:22.865 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-04-20T16:54:23.077 INFO:teuthology.orchestra.run.smithi171.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 1.2 MB/s | 689 kB 00:00 2024-04-20T16:54:23.084 INFO:teuthology.orchestra.run.smithi106.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 1.1 MB/s | 689 kB 00:00 2024-04-20T16:54:23.118 INFO:teuthology.orchestra.run.smithi106.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 1.5 MB/s | 1.0 MB 00:00 2024-04-20T16:54:23.119 INFO:teuthology.orchestra.run.smithi171.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 1.6 MB/s | 1.0 MB 00:00 2024-04-20T16:54:23.185 INFO:teuthology.orchestra.run.smithi106.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 379 kB/s | 38 kB 00:00 2024-04-20T16:54:23.186 INFO:teuthology.orchestra.run.smithi171.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 348 kB/s | 38 kB 00:00 2024-04-20T16:54:23.277 INFO:teuthology.orchestra.run.smithi106.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 1.6 MB/s | 258 kB 00:00 2024-04-20T16:54:23.535 INFO:teuthology.orchestra.run.smithi106.stdout:(5/13): buildah-1.33.6-2.module_el8+976+1136073 9.0 MB/s | 9.6 MB 00:01 2024-04-20T16:54:23.561 INFO:teuthology.orchestra.run.smithi106.stdout:(6/13): python3-podman-4.9.0-1.module_el8+847+7 635 kB/s | 180 kB 00:00 2024-04-20T16:54:23.577 INFO:teuthology.orchestra.run.smithi171.stdout:(4/13): buildah-1.33.6-2.module_el8+976+1136073 8.9 MB/s | 9.6 MB 00:01 2024-04-20T16:54:23.612 INFO:teuthology.orchestra.run.smithi171.stdout:(5/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 524 kB/s | 258 kB 00:00 2024-04-20T16:54:23.653 INFO:teuthology.orchestra.run.smithi106.stdout:(7/13): python3-pytoml-0.1.14-5.git7dea353.el8. 211 kB/s | 25 kB 00:00 2024-04-20T16:54:23.679 INFO:teuthology.orchestra.run.smithi171.stdout:(6/13): python3-podman-4.9.0-1.module_el8+847+7 1.7 MB/s | 180 kB 00:00 2024-04-20T16:54:23.705 INFO:teuthology.orchestra.run.smithi171.stdout:(7/13): python3-pytoml-0.1.14-5.git7dea353.el8. 268 kB/s | 25 kB 00:00 2024-04-20T16:54:23.745 INFO:teuthology.orchestra.run.smithi106.stdout:(8/13): netavark-1.10.3-1.module_el8+951+32019c 7.3 MB/s | 4.1 MB 00:00 2024-04-20T16:54:23.770 INFO:teuthology.orchestra.run.smithi106.stdout:(9/13): python3-pyxdg-0.25-16.el8.noarch.rpm 449 kB/s | 94 kB 00:00 2024-04-20T16:54:23.771 INFO:teuthology.orchestra.run.smithi171.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.0 MB/s | 94 kB 00:00 2024-04-20T16:54:23.822 INFO:teuthology.orchestra.run.smithi005.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 769 kB/s | 689 kB 00:00 2024-04-20T16:54:23.938 INFO:teuthology.orchestra.run.smithi171.stdout:(9/13): netavark-1.10.3-1.module_el8+951+32019c 5.5 MB/s | 4.1 MB 00:00 2024-04-20T16:54:23.997 INFO:teuthology.orchestra.run.smithi005.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 216 kB/s | 38 kB 00:00 2024-04-20T16:54:24.012 INFO:teuthology.orchestra.run.smithi106.stdout:(10/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 9.5 MB/s | 2.5 MB 00:00 2024-04-20T16:54:24.014 INFO:teuthology.orchestra.run.smithi171.stdout:(10/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 10 MB/s | 2.5 MB 00:00 2024-04-20T16:54:24.037 INFO:teuthology.orchestra.run.smithi106.stdout:(11/13): udica-0.2.6-20.module_el8+804+f131391c 197 kB/s | 53 kB 00:00 2024-04-20T16:54:24.056 INFO:teuthology.orchestra.run.smithi005.stdout:(3/13): aardvark-dns-1.10.0-1.module_el8+871+55 912 kB/s | 1.0 MB 00:01 2024-04-20T16:54:24.081 INFO:teuthology.orchestra.run.smithi171.stdout:(11/13): udica-0.2.6-20.module_el8+804+f131391c 370 kB/s | 53 kB 00:00 2024-04-20T16:54:24.123 INFO:teuthology.orchestra.run.smithi171.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 377 kB/s | 41 kB 00:00 2024-04-20T16:54:24.171 INFO:teuthology.orchestra.run.smithi106.stdout:(12/13): skopeo-1.14.3-0.1.module_el8+885+7da14 17 MB/s | 8.8 MB 00:00 2024-04-20T16:54:24.196 INFO:teuthology.orchestra.run.smithi106.stdout:(13/13): yajl-2.1.0-12.el8.x86_64.rpm 223 kB/s | 41 kB 00:00 2024-04-20T16:54:24.197 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:54:24.197 INFO:teuthology.orchestra.run.smithi106.stdout:Total 15 MB/s | 27 MB 00:01 2024-04-20T16:54:24.240 INFO:teuthology.orchestra.run.smithi005.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 1.0 MB/s | 258 kB 00:00 2024-04-20T16:54:24.415 INFO:teuthology.orchestra.run.smithi005.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 1.0 MB/s | 180 kB 00:00 2024-04-20T16:54:24.464 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-20T16:54:24.485 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-20T16:54:24.486 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-20T16:54:24.515 INFO:teuthology.orchestra.run.smithi005.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 248 kB/s | 25 kB 00:00 2024-04-20T16:54:24.624 INFO:teuthology.orchestra.run.smithi005.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 868 kB/s | 94 kB 00:00 2024-04-20T16:54:24.743 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-20T16:54:24.746 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-20T16:54:24.840 INFO:teuthology.orchestra.run.smithi171.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 7.8 MB/s | 8.8 MB 00:01 2024-04-20T16:54:24.840 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:54:24.841 INFO:teuthology.orchestra.run.smithi171.stdout:Total 11 MB/s | 27 MB 00:02 2024-04-20T16:54:25.115 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-04-20T16:54:25.137 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-04-20T16:54:25.137 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-04-20T16:54:25.355 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-20T16:54:25.396 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-04-20T16:54:25.399 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-04-20T16:54:26.055 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-04-20T16:54:26.293 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-04-20T16:54:26.514 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-04-20T16:54:26.676 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-04-20T16:54:26.877 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-04-20T16:54:27.050 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-04-20T16:54:27.111 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-04-20T16:54:27.259 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-04-20T16:54:27.437 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-04-20T16:54:27.668 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-04-20T16:54:27.906 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-04-20T16:54:27.915 INFO:teuthology.orchestra.run.smithi005.stdout:(8/13): buildah-1.33.6-2.module_el8+976+1136073 1.9 MB/s | 9.6 MB 00:04 2024-04-20T16:54:27.986 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-04-20T16:54:28.433 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-04-20T16:54:28.891 INFO:teuthology.orchestra.run.smithi005.stdout:(9/13): toolbox-0.0.99.5-2.module_el8+971+3d3df 2.6 MB/s | 2.5 MB 00:00 2024-04-20T16:54:28.933 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-04-20T16:54:29.177 INFO:teuthology.orchestra.run.smithi005.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 184 kB/s | 53 kB 00:00 2024-04-20T16:54:29.218 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-04-20T16:54:29.316 INFO:teuthology.orchestra.run.smithi005.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 294 kB/s | 41 kB 00:00 2024-04-20T16:54:29.430 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-04-20T16:54:29.441 INFO:teuthology.orchestra.run.smithi005.stdout:(12/13): netavark-1.10.3-1.module_el8+951+32019 780 kB/s | 4.1 MB 00:05 2024-04-20T16:54:29.603 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-04-20T16:54:29.858 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-04-20T16:54:29.882 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-04-20T16:54:30.060 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-20T16:54:30.103 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-04-20T16:54:30.263 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-20T16:54:30.483 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-20T16:54:30.569 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-04-20T16:54:30.773 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-20T16:54:30.902 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-20T16:54:30.902 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-04-20T16:54:30.902 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-04-20T16:54:30.902 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-04-20T16:54:30.903 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-04-20T16:54:30.903 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-04-20T16:54:30.903 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-04-20T16:54:30.903 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-04-20T16:54:30.903 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-04-20T16:54:30.903 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-04-20T16:54:30.903 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-04-20T16:54:30.903 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-20T16:54:30.903 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-20T16:54:30.994 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-20T16:54:31.017 INFO:teuthology.orchestra.run.smithi005.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 1.4 MB/s | 8.8 MB 00:06 2024-04-20T16:54:31.017 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:54:31.017 INFO:teuthology.orchestra.run.smithi005.stdout:Total 3.4 MB/s | 27 MB 00:08 2024-04-20T16:54:31.244 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-20T16:54:31.299 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-04-20T16:54:31.322 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-04-20T16:54:31.322 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-04-20T16:54:31.419 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-04-20T16:54:31.419 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:54:31.419 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-20T16:54:31.419 INFO:teuthology.orchestra.run.smithi106.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-04-20T16:54:31.419 INFO:teuthology.orchestra.run.smithi106.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-04-20T16:54:31.419 INFO:teuthology.orchestra.run.smithi106.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-04-20T16:54:31.419 INFO:teuthology.orchestra.run.smithi106.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-04-20T16:54:31.420 INFO:teuthology.orchestra.run.smithi106.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-04-20T16:54:31.420 INFO:teuthology.orchestra.run.smithi106.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-04-20T16:54:31.420 INFO:teuthology.orchestra.run.smithi106.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-04-20T16:54:31.420 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-04-20T16:54:31.420 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-04-20T16:54:31.420 INFO:teuthology.orchestra.run.smithi106.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-04-20T16:54:31.420 INFO:teuthology.orchestra.run.smithi106.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-04-20T16:54:31.420 INFO:teuthology.orchestra.run.smithi106.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-04-20T16:54:31.420 INFO:teuthology.orchestra.run.smithi106.stdout: yajl-2.1.0-12.el8.x86_64 2024-04-20T16:54:31.420 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:54:31.420 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-20T16:54:31.607 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-04-20T16:54:31.610 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-04-20T16:54:31.669 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-20T16:54:31.669 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-04-20T16:54:31.669 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-04-20T16:54:31.669 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-04-20T16:54:31.669 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-04-20T16:54:31.669 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-04-20T16:54:31.669 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-04-20T16:54:31.669 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-04-20T16:54:31.669 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-04-20T16:54:31.669 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-04-20T16:54:31.670 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-04-20T16:54:31.670 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-20T16:54:31.670 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-20T16:54:31.735 DEBUG:teuthology.parallel:result is None 2024-04-20T16:54:32.222 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-04-20T16:54:32.222 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:54:32.222 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-04-20T16:54:32.222 INFO:teuthology.orchestra.run.smithi171.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-04-20T16:54:32.222 INFO:teuthology.orchestra.run.smithi171.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-04-20T16:54:32.222 INFO:teuthology.orchestra.run.smithi171.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-04-20T16:54:32.223 INFO:teuthology.orchestra.run.smithi171.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-04-20T16:54:32.223 INFO:teuthology.orchestra.run.smithi171.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-04-20T16:54:32.223 INFO:teuthology.orchestra.run.smithi171.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-04-20T16:54:32.223 INFO:teuthology.orchestra.run.smithi171.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-04-20T16:54:32.223 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-04-20T16:54:32.223 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-04-20T16:54:32.223 INFO:teuthology.orchestra.run.smithi171.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-04-20T16:54:32.223 INFO:teuthology.orchestra.run.smithi171.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-04-20T16:54:32.223 INFO:teuthology.orchestra.run.smithi171.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-04-20T16:54:32.223 INFO:teuthology.orchestra.run.smithi171.stdout: yajl-2.1.0-12.el8.x86_64 2024-04-20T16:54:32.223 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:54:32.223 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-04-20T16:54:32.317 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-04-20T16:54:32.491 DEBUG:teuthology.parallel:result is None 2024-04-20T16:54:33.335 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-04-20T16:54:33.538 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-04-20T16:54:33.709 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-04-20T16:54:33.922 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-04-20T16:54:34.165 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-04-20T16:54:34.828 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-04-20T16:54:35.902 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-04-20T16:54:36.181 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-04-20T16:54:36.374 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-04-20T16:54:36.838 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-04-20T16:54:37.056 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-20T16:54:37.256 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-20T16:54:37.473 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-20T16:54:37.952 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-20T16:54:37.952 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-04-20T16:54:37.953 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-04-20T16:54:37.953 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-04-20T16:54:37.953 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-04-20T16:54:37.953 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-04-20T16:54:37.953 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-04-20T16:54:37.953 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-04-20T16:54:37.953 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-04-20T16:54:37.953 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-04-20T16:54:37.953 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-04-20T16:54:37.953 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-20T16:54:37.953 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-20T16:54:38.581 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-04-20T16:54:38.581 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:54:38.581 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-04-20T16:54:38.581 INFO:teuthology.orchestra.run.smithi005.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-04-20T16:54:38.581 INFO:teuthology.orchestra.run.smithi005.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-04-20T16:54:38.581 INFO:teuthology.orchestra.run.smithi005.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-04-20T16:54:38.582 INFO:teuthology.orchestra.run.smithi005.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-04-20T16:54:38.582 INFO:teuthology.orchestra.run.smithi005.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-04-20T16:54:38.582 INFO:teuthology.orchestra.run.smithi005.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-04-20T16:54:38.582 INFO:teuthology.orchestra.run.smithi005.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-04-20T16:54:38.582 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-04-20T16:54:38.582 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-04-20T16:54:38.582 INFO:teuthology.orchestra.run.smithi005.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-04-20T16:54:38.582 INFO:teuthology.orchestra.run.smithi005.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-04-20T16:54:38.583 INFO:teuthology.orchestra.run.smithi005.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-04-20T16:54:38.583 INFO:teuthology.orchestra.run.smithi005.stdout: yajl-2.1.0-12.el8.x86_64 2024-04-20T16:54:38.583 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:54:38.583 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-04-20T16:54:38.880 DEBUG:teuthology.parallel:result is None 2024-04-20T16:54:38.880 INFO:teuthology.run_tasks:Running task install... 2024-04-20T16:54:38.888 DEBUG:teuthology.task.install:project ceph 2024-04-20T16:54:38.888 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '36c371567dddacf6207ea36f2535396ab31415fc'}} 2024-04-20T16:54:38.889 DEBUG:teuthology.task.install:config {'branch': 'nautilus', 'exclude_packages': ['cephadm', 'ceph-mgr-cephadm', 'ceph-immutable-object-cache', 'python3-rados', 'python3-rgw', 'python3-rbd', 'python3-cephfs', 'ceph-volume'], 'extra_packages': ['python-rados', 'python-rgw', 'python-rbd', 'python-cephfs'], 'flavor': 'default', 'sha1': '36c371567dddacf6207ea36f2535396ab31415fc'} 2024-04-20T16:54:38.889 INFO:teuthology.task.install:Using flavor: default 2024-04-20T16:54:38.894 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-20T16:54:38.894 INFO:teuthology.task.install:extra packages: ['python-rados', 'python-rgw', 'python-rbd', 'python-cephfs'] 2024-04-20T16:54:38.895 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'nautilus', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['cephadm', 'ceph-mgr-cephadm', 'ceph-immutable-object-cache', 'python3-rados', 'python3-rgw', 'python3-rbd', 'python3-cephfs', 'ceph-volume'], 'extra_packages': ['python-rados', 'python-rgw', 'python-rbd', 'python-cephfs'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '36c371567dddacf6207ea36f2535396ab31415fc', 'tag': None, 'wait_for_package': False} 2024-04-20T16:54:38.895 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-20T16:54:38.895 INFO:teuthology.packaging:ref: None 2024-04-20T16:54:38.895 INFO:teuthology.packaging:tag: None 2024-04-20T16:54:38.895 INFO:teuthology.packaging:branch: nautilus 2024-04-20T16:54:38.895 INFO:teuthology.packaging:sha1: 36c371567dddacf6207ea36f2535396ab31415fc 2024-04-20T16:54:38.895 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=nautilus 2024-04-20T16:54:38.897 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'nautilus', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['cephadm', 'ceph-mgr-cephadm', 'ceph-immutable-object-cache', 'python3-rados', 'python3-rgw', 'python3-rbd', 'python3-cephfs', 'ceph-volume'], 'extra_packages': ['python-rados', 'python-rgw', 'python-rbd', 'python-cephfs'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '36c371567dddacf6207ea36f2535396ab31415fc', 'tag': None, 'wait_for_package': False} 2024-04-20T16:54:38.897 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-20T16:54:38.897 INFO:teuthology.packaging:ref: None 2024-04-20T16:54:38.897 INFO:teuthology.packaging:tag: None 2024-04-20T16:54:38.898 INFO:teuthology.packaging:branch: nautilus 2024-04-20T16:54:38.898 INFO:teuthology.packaging:sha1: 36c371567dddacf6207ea36f2535396ab31415fc 2024-04-20T16:54:38.898 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=nautilus 2024-04-20T16:54:38.899 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'nautilus', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['cephadm', 'ceph-mgr-cephadm', 'ceph-immutable-object-cache', 'python3-rados', 'python3-rgw', 'python3-rbd', 'python3-cephfs', 'ceph-volume'], 'extra_packages': ['python-rados', 'python-rgw', 'python-rbd', 'python-cephfs'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '36c371567dddacf6207ea36f2535396ab31415fc', 'tag': None, 'wait_for_package': False} 2024-04-20T16:54:38.899 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-20T16:54:38.899 INFO:teuthology.packaging:ref: None 2024-04-20T16:54:38.899 INFO:teuthology.packaging:tag: None 2024-04-20T16:54:38.899 INFO:teuthology.packaging:branch: nautilus 2024-04-20T16:54:38.899 INFO:teuthology.packaging:sha1: 36c371567dddacf6207ea36f2535396ab31415fc 2024-04-20T16:54:38.899 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=nautilus 2024-04-20T16:54:39.067 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/ 2024-04-20T16:54:39.067 INFO:teuthology.task.install.rpm:Package version is 14.2.22-0 2024-04-20T16:54:39.097 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-20T16:54:39.097 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-04-20T16:54:39.097 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-20T16:54:39.132 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, rbd-fuse, rbd-mirror, rbd-nbd, python-rados, python-rgw, python-rbd, python-cephfs on remote rpm x86_64 2024-04-20T16:54:39.133 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-20T16:54:39.133 INFO:teuthology.packaging:ref: None 2024-04-20T16:54:39.133 INFO:teuthology.packaging:tag: None 2024-04-20T16:54:39.133 INFO:teuthology.packaging:branch: nautilus 2024-04-20T16:54:39.133 INFO:teuthology.packaging:sha1: 36c371567dddacf6207ea36f2535396ab31415fc 2024-04-20T16:54:39.133 DEBUG:teuthology.orchestra.run.smithi171:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/nautilus/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-20T16:54:39.157 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/ 2024-04-20T16:54:39.158 INFO:teuthology.task.install.rpm:Package version is 14.2.22-0 2024-04-20T16:54:39.174 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/ 2024-04-20T16:54:39.174 INFO:teuthology.task.install.rpm:Package version is 14.2.22-0 2024-04-20T16:54:39.188 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-20T16:54:39.188 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-20T16:54:39.188 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-20T16:54:39.201 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/nautilus/ca74598065096e6fcbd8433c8779a2be0c889351/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-20T16:54:39.201 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-04-20T16:54:39.201 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-20T16:54:39.208 DEBUG:teuthology.orchestra.run.smithi171:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-20T16:54:39.224 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, rbd-fuse, rbd-mirror, rbd-nbd, python-rados, python-rgw, python-rbd, python-cephfs on remote rpm x86_64 2024-04-20T16:54:39.224 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-20T16:54:39.224 INFO:teuthology.packaging:ref: None 2024-04-20T16:54:39.224 INFO:teuthology.packaging:tag: None 2024-04-20T16:54:39.224 INFO:teuthology.packaging:branch: nautilus 2024-04-20T16:54:39.224 INFO:teuthology.packaging:sha1: 36c371567dddacf6207ea36f2535396ab31415fc 2024-04-20T16:54:39.224 DEBUG:teuthology.orchestra.run.smithi106:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/nautilus/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-20T16:54:39.229 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, rbd-fuse, rbd-mirror, rbd-nbd, python-rados, python-rgw, python-rbd, python-cephfs on remote rpm x86_64 2024-04-20T16:54:39.229 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-20T16:54:39.229 INFO:teuthology.packaging:ref: None 2024-04-20T16:54:39.229 INFO:teuthology.packaging:tag: None 2024-04-20T16:54:39.230 INFO:teuthology.packaging:branch: nautilus 2024-04-20T16:54:39.230 INFO:teuthology.packaging:sha1: 36c371567dddacf6207ea36f2535396ab31415fc 2024-04-20T16:54:39.230 DEBUG:teuthology.orchestra.run.smithi005:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/nautilus/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-20T16:54:39.288 DEBUG:teuthology.orchestra.run.smithi171:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-20T16:54:39.297 DEBUG:teuthology.orchestra.run.smithi005:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-20T16:54:39.298 DEBUG:teuthology.orchestra.run.smithi106:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-20T16:54:39.356 INFO:teuthology.orchestra.run.smithi171.stdout:check_obsoletes = 1 2024-04-20T16:54:39.357 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum clean all 2024-04-20T16:54:39.374 DEBUG:teuthology.orchestra.run.smithi005:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-20T16:54:39.383 DEBUG:teuthology.orchestra.run.smithi106:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-20T16:54:39.441 INFO:teuthology.orchestra.run.smithi005.stdout:check_obsoletes = 1 2024-04-20T16:54:39.443 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum clean all 2024-04-20T16:54:39.455 INFO:teuthology.orchestra.run.smithi106.stdout:check_obsoletes = 1 2024-04-20T16:54:39.457 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum clean all 2024-04-20T16:54:39.636 INFO:teuthology.orchestra.run.smithi171.stdout:78 files removed 2024-04-20T16:54:39.667 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-radosgw 2024-04-20T16:54:39.722 INFO:teuthology.orchestra.run.smithi005.stdout:78 files removed 2024-04-20T16:54:39.735 INFO:teuthology.orchestra.run.smithi106.stdout:78 files removed 2024-04-20T16:54:39.751 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-radosgw 2024-04-20T16:54:39.763 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-radosgw 2024-04-20T16:54:40.221 INFO:teuthology.orchestra.run.smithi171.stdout:ceph packages for x86_64 443 kB/s | 65 kB 00:00 2024-04-20T16:54:40.239 INFO:teuthology.orchestra.run.smithi106.stdout:ceph packages for x86_64 845 kB/s | 65 kB 00:00 2024-04-20T16:54:40.259 INFO:teuthology.orchestra.run.smithi005.stdout:ceph packages for x86_64 865 kB/s | 65 kB 00:00 2024-04-20T16:54:40.360 INFO:teuthology.orchestra.run.smithi005.stdout:ceph noarch packages 188 kB/s | 11 kB 00:00 2024-04-20T16:54:40.373 INFO:teuthology.orchestra.run.smithi106.stdout:ceph noarch packages 118 kB/s | 11 kB 00:00 2024-04-20T16:54:40.402 INFO:teuthology.orchestra.run.smithi171.stdout:ceph noarch packages 80 kB/s | 11 kB 00:00 2024-04-20T16:54:40.502 INFO:teuthology.orchestra.run.smithi005.stdout:ceph source packages 15 kB/s | 1.7 kB 00:00 2024-04-20T16:54:40.507 INFO:teuthology.orchestra.run.smithi171.stdout:ceph source packages 22 kB/s | 1.7 kB 00:00 2024-04-20T16:54:40.515 INFO:teuthology.orchestra.run.smithi106.stdout:ceph source packages 15 kB/s | 1.7 kB 00:00 2024-04-20T16:54:42.341 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 8 - AppStream 15 MB/s | 28 MB 00:01 2024-04-20T16:54:42.453 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 8 - AppStream 15 MB/s | 28 MB 00:01 2024-04-20T16:54:43.504 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 8 - AppStream 9.5 MB/s | 28 MB 00:02 2024-04-20T16:54:47.700 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 8 - BaseOS 29 MB/s | 10 MB 00:00 2024-04-20T16:54:48.253 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 8 - BaseOS 21 MB/s | 10 MB 00:00 2024-04-20T16:54:48.821 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 8 - BaseOS 29 MB/s | 10 MB 00:00 2024-04-20T16:54:49.567 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 8 - Extras 63 kB/s | 18 kB 00:00 2024-04-20T16:54:49.993 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 8 - Extras common packages 24 kB/s | 7.7 kB 00:00 2024-04-20T16:54:50.204 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 8 - Extras 65 kB/s | 18 kB 00:00 2024-04-20T16:54:50.497 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 8 - Extras common packages 43 kB/s | 7.7 kB 00:00 2024-04-20T16:54:50.800 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 8 - Extras 45 kB/s | 18 kB 00:00 2024-04-20T16:54:50.870 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 8 - PowerTools 6.3 MB/s | 5.0 MB 00:00 2024-04-20T16:54:51.217 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 8 - Extras common packages 25 kB/s | 7.7 kB 00:00 2024-04-20T16:54:51.231 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 8 - PowerTools 7.7 MB/s | 5.0 MB 00:00 2024-04-20T16:54:51.972 INFO:teuthology.orchestra.run.smithi171.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2024-04-20T16:54:51.994 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 8 - PowerTools 7.2 MB/s | 5.0 MB 00:00 2024-04-20T16:54:52.349 INFO:teuthology.orchestra.run.smithi005.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-04-20T16:54:53.046 INFO:teuthology.orchestra.run.smithi106.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-04-20T16:54:53.967 INFO:teuthology.orchestra.run.smithi005.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-04-20T16:54:57.992 INFO:teuthology.orchestra.run.smithi005.stdout:lab-extras 357 kB/s | 24 kB 00:00 2024-04-20T16:54:58.250 INFO:teuthology.orchestra.run.smithi171.stdout:Extra Packages for Enterprise Linux 2.6 MB/s | 16 MB 00:06 2024-04-20T16:55:01.828 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-04-20T16:55:01.831 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:55:01.831 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-04-20T16:55:01.831 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:55:01.831 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-04-20T16:55:01.831 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-radosgw x86_64 2:14.2.22-0.el8 ceph 6.0 M 2024-04-20T16:55:01.832 INFO:teuthology.orchestra.run.smithi005.stdout:Upgrading: 2024-04-20T16:55:01.832 INFO:teuthology.orchestra.run.smithi005.stdout: librados2 x86_64 2:14.2.22-0.el8 ceph 3.5 M 2024-04-20T16:55:01.832 INFO:teuthology.orchestra.run.smithi005.stdout: librbd1 x86_64 2:14.2.22-0.el8 ceph 1.7 M 2024-04-20T16:55:01.832 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-04-20T16:55:01.832 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-base x86_64 2:14.2.22-0.el8 ceph 5.4 M 2024-04-20T16:55:01.832 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-common x86_64 2:14.2.22-0.el8 ceph 19 M 2024-04-20T16:55:01.832 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-selinux x86_64 2:14.2.22-0.el8 ceph 25 k 2024-04-20T16:55:01.832 INFO:teuthology.orchestra.run.smithi005.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-20T16:55:01.832 INFO:teuthology.orchestra.run.smithi005.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-04-20T16:55:01.832 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs2 x86_64 2:14.2.22-0.el8 ceph 520 k 2024-04-20T16:55:01.832 INFO:teuthology.orchestra.run.smithi005.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-20T16:55:01.832 INFO:teuthology.orchestra.run.smithi005.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-20T16:55:01.832 INFO:teuthology.orchestra.run.smithi005.stdout: libradosstriper1 x86_64 2:14.2.22-0.el8 ceph 376 k 2024-04-20T16:55:01.833 INFO:teuthology.orchestra.run.smithi005.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-20T16:55:01.833 INFO:teuthology.orchestra.run.smithi005.stdout: librgw2 x86_64 2:14.2.22-0.el8 ceph 5.3 M 2024-04-20T16:55:01.833 INFO:teuthology.orchestra.run.smithi005.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-20T16:55:01.833 INFO:teuthology.orchestra.run.smithi005.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-20T16:55:01.833 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-argparse x86_64 2:14.2.22-0.el8 ceph 39 k 2024-04-20T16:55:01.833 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cephfs x86_64 2:14.2.22-0.el8 ceph 159 k 2024-04-20T16:55:01.833 INFO:teuthology.orchestra.run.smithi005.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-20T16:55:01.833 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rados x86_64 2:14.2.22-0.el8 ceph 323 k 2024-04-20T16:55:01.833 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rbd x86_64 2:14.2.22-0.el8 ceph 291 k 2024-04-20T16:55:01.833 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rgw x86_64 2:14.2.22-0.el8 ceph 113 k 2024-04-20T16:55:01.833 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:55:01.833 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-04-20T16:55:01.833 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:55:01.833 INFO:teuthology.orchestra.run.smithi005.stdout:Install 20 Packages 2024-04-20T16:55:01.834 INFO:teuthology.orchestra.run.smithi005.stdout:Upgrade 2 Packages 2024-04-20T16:55:01.834 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:55:01.834 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 45 M 2024-04-20T16:55:01.834 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-04-20T16:55:02.115 INFO:teuthology.orchestra.run.smithi171.stdout:lab-extras 399 kB/s | 24 kB 00:00 2024-04-20T16:55:04.576 INFO:teuthology.orchestra.run.smithi005.stdout:(1/22): ceph-base-14.2.22-0.el8.x86_64.rpm 37 MB/s | 5.4 MB 00:00 2024-04-20T16:55:04.601 INFO:teuthology.orchestra.run.smithi005.stdout:(2/22): ceph-selinux-14.2.22-0.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2024-04-20T16:55:04.645 INFO:teuthology.orchestra.run.smithi005.stdout:(3/22): libcephfs2-14.2.22-0.el8.x86_64.rpm 12 MB/s | 520 kB 00:00 2024-04-20T16:55:04.677 INFO:teuthology.orchestra.run.smithi005.stdout:(4/22): libradosstriper1-14.2.22-0.el8.x86_64.r 11 MB/s | 376 kB 00:00 2024-04-20T16:55:04.794 INFO:teuthology.orchestra.run.smithi005.stdout:(5/22): ceph-radosgw-14.2.22-0.el8.x86_64.rpm 17 MB/s | 6.0 MB 00:00 2024-04-20T16:55:04.819 INFO:teuthology.orchestra.run.smithi005.stdout:(6/22): python3-ceph-argparse-14.2.22-0.el8.x86 1.5 MB/s | 39 kB 00:00 2024-04-20T16:55:04.845 INFO:teuthology.orchestra.run.smithi005.stdout:(7/22): python3-cephfs-14.2.22-0.el8.x86_64.rpm 6.2 MB/s | 159 kB 00:00 2024-04-20T16:55:04.878 INFO:teuthology.orchestra.run.smithi005.stdout:(8/22): python3-rados-14.2.22-0.el8.x86_64.rpm 9.6 MB/s | 323 kB 00:00 2024-04-20T16:55:04.904 INFO:teuthology.orchestra.run.smithi005.stdout:(9/22): python3-rbd-14.2.22-0.el8.x86_64.rpm 11 MB/s | 291 kB 00:00 2024-04-20T16:55:04.929 INFO:teuthology.orchestra.run.smithi005.stdout:(10/22): python3-rgw-14.2.22-0.el8.x86_64.rpm 4.5 MB/s | 113 kB 00:00 2024-04-20T16:55:05.022 INFO:teuthology.orchestra.run.smithi005.stdout:(11/22): librgw2-14.2.22-0.el8.x86_64.rpm 15 MB/s | 5.3 MB 00:00 2024-04-20T16:55:05.363 INFO:teuthology.orchestra.run.smithi005.stdout:(12/22): ceph-common-14.2.22-0.el8.x86_64.rpm 21 MB/s | 19 MB 00:00 2024-04-20T16:55:05.636 INFO:teuthology.orchestra.run.smithi005.stdout:(13/22): librdkafka-1.6.1-1.el8.x86_64.rpm 957 kB/s | 677 kB 00:00 2024-04-20T16:55:05.689 INFO:teuthology.orchestra.run.smithi005.stdout:(14/22): python3-prettytable-0.7.2-14.el8.noarc 135 kB/s | 44 kB 00:00 2024-04-20T16:55:05.705 INFO:teuthology.orchestra.run.smithi005.stdout:(15/22): librabbitmq-0.9.0-5.el8.x86_64.rpm 687 kB/s | 47 kB 00:00 2024-04-20T16:55:05.756 INFO:teuthology.orchestra.run.smithi005.stdout:(16/22): lttng-ust-2.8.1-11.el8.x86_64.rpm 354 kB/s | 259 kB 00:00 2024-04-20T16:55:05.768 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-04-20T16:55:05.770 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:55:05.770 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-04-20T16:55:05.770 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:55:05.771 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-04-20T16:55:05.771 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-radosgw x86_64 2:14.2.22-0.el8 ceph 6.0 M 2024-04-20T16:55:05.771 INFO:teuthology.orchestra.run.smithi171.stdout:Upgrading: 2024-04-20T16:55:05.771 INFO:teuthology.orchestra.run.smithi171.stdout: librados2 x86_64 2:14.2.22-0.el8 ceph 3.5 M 2024-04-20T16:55:05.771 INFO:teuthology.orchestra.run.smithi171.stdout: librbd1 x86_64 2:14.2.22-0.el8 ceph 1.7 M 2024-04-20T16:55:05.771 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-04-20T16:55:05.771 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-base x86_64 2:14.2.22-0.el8 ceph 5.4 M 2024-04-20T16:55:05.771 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-common x86_64 2:14.2.22-0.el8 ceph 19 M 2024-04-20T16:55:05.771 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-selinux x86_64 2:14.2.22-0.el8 ceph 25 k 2024-04-20T16:55:05.771 INFO:teuthology.orchestra.run.smithi171.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-20T16:55:05.772 INFO:teuthology.orchestra.run.smithi171.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-04-20T16:55:05.772 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs2 x86_64 2:14.2.22-0.el8 ceph 520 k 2024-04-20T16:55:05.772 INFO:teuthology.orchestra.run.smithi171.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-20T16:55:05.772 INFO:teuthology.orchestra.run.smithi171.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-20T16:55:05.772 INFO:teuthology.orchestra.run.smithi171.stdout: libradosstriper1 x86_64 2:14.2.22-0.el8 ceph 376 k 2024-04-20T16:55:05.772 INFO:teuthology.orchestra.run.smithi171.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-20T16:55:05.772 INFO:teuthology.orchestra.run.smithi171.stdout: librgw2 x86_64 2:14.2.22-0.el8 ceph 5.3 M 2024-04-20T16:55:05.772 INFO:teuthology.orchestra.run.smithi171.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-20T16:55:05.772 INFO:teuthology.orchestra.run.smithi171.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-20T16:55:05.772 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-argparse x86_64 2:14.2.22-0.el8 ceph 39 k 2024-04-20T16:55:05.772 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cephfs x86_64 2:14.2.22-0.el8 ceph 159 k 2024-04-20T16:55:05.772 INFO:teuthology.orchestra.run.smithi171.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-20T16:55:05.772 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rados x86_64 2:14.2.22-0.el8 ceph 323 k 2024-04-20T16:55:05.772 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rbd x86_64 2:14.2.22-0.el8 ceph 291 k 2024-04-20T16:55:05.773 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rgw x86_64 2:14.2.22-0.el8 ceph 113 k 2024-04-20T16:55:05.773 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:55:05.773 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-04-20T16:55:05.773 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:55:05.773 INFO:teuthology.orchestra.run.smithi171.stdout:Install 20 Packages 2024-04-20T16:55:05.773 INFO:teuthology.orchestra.run.smithi171.stdout:Upgrade 2 Packages 2024-04-20T16:55:05.773 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:55:05.773 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 45 M 2024-04-20T16:55:05.774 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-04-20T16:55:05.861 INFO:teuthology.orchestra.run.smithi005.stdout:(17/22): leveldb-1.22-1.el8.x86_64.rpm 1.1 MB/s | 181 kB 00:00 2024-04-20T16:55:06.003 INFO:teuthology.orchestra.run.smithi005.stdout:(18/22): liboath-2.6.2-3.el8.x86_64.rpm 238 kB/s | 59 kB 00:00 2024-04-20T16:55:06.229 INFO:teuthology.orchestra.run.smithi005.stdout:(19/22): libunwind-1.3.1-3.el8.x86_64.rpm 204 kB/s | 75 kB 00:00 2024-04-20T16:55:06.541 INFO:teuthology.orchestra.run.smithi005.stdout:(20/22): gperftools-libs-2.7-9.el8.x86_64.rpm 359 kB/s | 306 kB 00:00 2024-04-20T16:55:06.750 INFO:teuthology.orchestra.run.smithi005.stdout:(21/22): librbd1-14.2.22-0.el8.x86_64.rpm 3.3 MB/s | 1.7 MB 00:00 2024-04-20T16:55:06.783 INFO:teuthology.orchestra.run.smithi171.stdout:(1/22): ceph-base-14.2.22-0.el8.x86_64.rpm 46 MB/s | 5.4 MB 00:00 2024-04-20T16:55:06.809 INFO:teuthology.orchestra.run.smithi171.stdout:(2/22): ceph-selinux-14.2.22-0.el8.x86_64.rpm 998 kB/s | 25 kB 00:00 2024-04-20T16:55:06.876 INFO:teuthology.orchestra.run.smithi171.stdout:(3/22): libcephfs2-14.2.22-0.el8.x86_64.rpm 7.6 MB/s | 520 kB 00:00 2024-04-20T16:55:06.910 INFO:teuthology.orchestra.run.smithi171.stdout:(4/22): libradosstriper1-14.2.22-0.el8.x86_64.r 11 MB/s | 376 kB 00:00 2024-04-20T16:55:06.958 INFO:teuthology.orchestra.run.smithi005.stdout:(22/22): librados2-14.2.22-0.el8.x86_64.rpm 3.7 MB/s | 3.5 MB 00:00 2024-04-20T16:55:06.960 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:55:06.961 INFO:teuthology.orchestra.run.smithi005.stdout:Total 8.7 MB/s | 45 MB 00:05 2024-04-20T16:55:07.001 INFO:teuthology.orchestra.run.smithi171.stdout:(5/22): ceph-radosgw-14.2.22-0.el8.x86_64.rpm 18 MB/s | 6.0 MB 00:00 2024-04-20T16:55:07.006 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-04-20T16:55:07.026 INFO:teuthology.orchestra.run.smithi171.stdout:(6/22): python3-ceph-argparse-14.2.22-0.el8.x86 1.6 MB/s | 39 kB 00:00 2024-04-20T16:55:07.052 INFO:teuthology.orchestra.run.smithi171.stdout:(7/22): python3-cephfs-14.2.22-0.el8.x86_64.rpm 6.2 MB/s | 159 kB 00:00 2024-04-20T16:55:07.058 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-04-20T16:55:07.058 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-04-20T16:55:07.077 INFO:teuthology.orchestra.run.smithi171.stdout:(8/22): python3-rados-14.2.22-0.el8.x86_64.rpm 13 MB/s | 323 kB 00:00 2024-04-20T16:55:07.111 INFO:teuthology.orchestra.run.smithi171.stdout:(9/22): python3-rbd-14.2.22-0.el8.x86_64.rpm 8.6 MB/s | 291 kB 00:00 2024-04-20T16:55:07.136 INFO:teuthology.orchestra.run.smithi171.stdout:(10/22): python3-rgw-14.2.22-0.el8.x86_64.rpm 4.4 MB/s | 113 kB 00:00 2024-04-20T16:55:07.229 INFO:teuthology.orchestra.run.smithi171.stdout:(11/22): librgw2-14.2.22-0.el8.x86_64.rpm 17 MB/s | 5.3 MB 00:00 2024-04-20T16:55:07.359 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-04-20T16:55:07.362 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-04-20T16:55:07.454 INFO:teuthology.orchestra.run.smithi171.stdout:(12/22): ceph-common-14.2.22-0.el8.x86_64.rpm 25 MB/s | 19 MB 00:00 2024-04-20T16:55:07.705 INFO:teuthology.orchestra.run.smithi171.stdout:(13/22): librdkafka-1.6.1-1.el8.x86_64.rpm 1.2 MB/s | 677 kB 00:00 2024-04-20T16:55:07.763 INFO:teuthology.orchestra.run.smithi171.stdout:(14/22): librabbitmq-0.9.0-5.el8.x86_64.rpm 809 kB/s | 47 kB 00:00 2024-04-20T16:55:07.789 INFO:teuthology.orchestra.run.smithi171.stdout:(15/22): python3-prettytable-0.7.2-14.el8.noarc 131 kB/s | 44 kB 00:00 2024-04-20T16:55:07.847 INFO:teuthology.orchestra.run.smithi171.stdout:(16/22): lttng-ust-2.8.1-11.el8.x86_64.rpm 420 kB/s | 259 kB 00:00 2024-04-20T16:55:08.156 INFO:teuthology.orchestra.run.smithi171.stdout:(17/22): liboath-2.6.2-3.el8.x86_64.rpm 197 kB/s | 59 kB 00:00 2024-04-20T16:55:08.190 INFO:teuthology.orchestra.run.smithi171.stdout:(18/22): leveldb-1.22-1.el8.x86_64.rpm 451 kB/s | 181 kB 00:00 2024-04-20T16:55:08.215 INFO:teuthology.orchestra.run.smithi171.stdout:(19/22): gperftools-libs-2.7-9.el8.x86_64.rpm 678 kB/s | 306 kB 00:00 2024-04-20T16:55:08.265 INFO:teuthology.orchestra.run.smithi171.stdout:(20/22): librbd1-14.2.22-0.el8.x86_64.rpm 34 MB/s | 1.7 MB 00:00 2024-04-20T16:55:08.291 INFO:teuthology.orchestra.run.smithi171.stdout:(21/22): libunwind-1.3.1-3.el8.x86_64.rpm 559 kB/s | 75 kB 00:00 2024-04-20T16:55:08.358 INFO:teuthology.orchestra.run.smithi171.stdout:(22/22): librados2-14.2.22-0.el8.x86_64.rpm 21 MB/s | 3.5 MB 00:00 2024-04-20T16:55:08.363 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:55:08.363 INFO:teuthology.orchestra.run.smithi171.stdout:Total 17 MB/s | 45 MB 00:02 2024-04-20T16:55:08.369 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-04-20T16:55:08.407 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-04-20T16:55:08.456 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-04-20T16:55:08.456 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-04-20T16:55:08.574 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/24 2024-04-20T16:55:08.698 INFO:teuthology.orchestra.run.smithi106.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 16 MB 00:15 2024-04-20T16:55:08.737 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-04-20T16:55:08.740 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-04-20T16:55:08.785 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/24 2024-04-20T16:55:09.161 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/24 2024-04-20T16:55:09.402 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : librados2-2:14.2.22-0.el8.x86_64 3/24 2024-04-20T16:55:09.513 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librados2-2:14.2.22-0.el8.x86_64 3/24 2024-04-20T16:55:09.716 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-04-20T16:55:09.721 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rados-2:14.2.22-0.el8.x86_64 4/24 2024-04-20T16:55:09.893 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-04-20T16:55:09.967 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/24 2024-04-20T16:55:10.114 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-04-20T16:55:10.214 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/24 2024-04-20T16:55:10.335 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : librbd1-2:14.2.22-0.el8.x86_64 6/24 2024-04-20T16:55:10.414 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librbd1-2:14.2.22-0.el8.x86_64 6/24 2024-04-20T16:55:10.586 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/24 2024-04-20T16:55:10.830 INFO:teuthology.orchestra.run.smithi171.stdout: Upgrading : librados2-2:14.2.22-0.el8.x86_64 3/24 2024-04-20T16:55:10.911 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librados2-2:14.2.22-0.el8.x86_64 3/24 2024-04-20T16:55:11.062 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/24 2024-04-20T16:55:11.138 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rados-2:14.2.22-0.el8.x86_64 4/24 2024-04-20T16:55:11.243 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librgw2-2:14.2.22-0.el8.x86_64 8/24 2024-04-20T16:55:11.320 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librgw2-2:14.2.22-0.el8.x86_64 8/24 2024-04-20T16:55:11.363 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-04-20T16:55:11.524 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : leveldb-1.22-1.el8.x86_64 9/24 2024-04-20T16:55:11.556 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-04-20T16:55:11.670 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-ceph-argparse-2:14.2.22-0.el8.x86_64 10/24 2024-04-20T16:55:11.802 INFO:teuthology.orchestra.run.smithi171.stdout: Upgrading : librbd1-2:14.2.22-0.el8.x86_64 6/24 2024-04-20T16:55:11.863 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cephfs-2:14.2.22-0.el8.x86_64 11/24 2024-04-20T16:55:11.882 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librbd1-2:14.2.22-0.el8.x86_64 6/24 2024-04-20T16:55:12.157 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rgw-2:14.2.22-0.el8.x86_64 12/24 2024-04-20T16:55:12.386 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rbd-2:14.2.22-0.el8.x86_64 13/24 2024-04-20T16:55:12.487 INFO:teuthology.orchestra.run.smithi106.stdout:lab-extras 399 kB/s | 24 kB 00:00 2024-04-20T16:55:12.506 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/24 2024-04-20T16:55:12.526 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libradosstriper1-2:14.2.22-0.el8.x86_64 14/24 2024-04-20T16:55:12.611 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libradosstriper1-2:14.2.22-0.el8.x86_64 14/24 2024-04-20T16:55:12.655 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librgw2-2:14.2.22-0.el8.x86_64 8/24 2024-04-20T16:55:12.713 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librgw2-2:14.2.22-0.el8.x86_64 8/24 2024-04-20T16:55:12.848 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/24 2024-04-20T16:55:12.926 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : leveldb-1.22-1.el8.x86_64 9/24 2024-04-20T16:55:13.069 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/24 2024-04-20T16:55:13.072 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-ceph-argparse-2:14.2.22-0.el8.x86_64 10/24 2024-04-20T16:55:13.283 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cephfs-2:14.2.22-0.el8.x86_64 11/24 2024-04-20T16:55:13.307 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 17/24 2024-04-20T16:55:13.460 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 18/24 2024-04-20T16:55:13.474 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rgw-2:14.2.22-0.el8.x86_64 12/24 2024-04-20T16:55:13.703 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rbd-2:14.2.22-0.el8.x86_64 13/24 2024-04-20T16:55:13.896 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libradosstriper1-2:14.2.22-0.el8.x86_64 14/24 2024-04-20T16:55:13.969 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libradosstriper1-2:14.2.22-0.el8.x86_64 14/24 2024-04-20T16:55:14.222 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/24 2024-04-20T16:55:14.413 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/24 2024-04-20T16:55:14.648 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 17/24 2024-04-20T16:55:14.847 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 18/24 2024-04-20T16:55:15.790 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-04-20T16:55:16.052 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-04-20T16:55:16.114 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-20T16:55:16.117 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:55:16.117 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-04-20T16:55:16.117 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:55:16.117 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-20T16:55:16.117 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-radosgw x86_64 2:14.2.22-0.el8 ceph 6.0 M 2024-04-20T16:55:16.117 INFO:teuthology.orchestra.run.smithi106.stdout:Upgrading: 2024-04-20T16:55:16.117 INFO:teuthology.orchestra.run.smithi106.stdout: librados2 x86_64 2:14.2.22-0.el8 ceph 3.5 M 2024-04-20T16:55:16.117 INFO:teuthology.orchestra.run.smithi106.stdout: librbd1 x86_64 2:14.2.22-0.el8 ceph 1.7 M 2024-04-20T16:55:16.117 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-04-20T16:55:16.117 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-base x86_64 2:14.2.22-0.el8 ceph 5.4 M 2024-04-20T16:55:16.118 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-common x86_64 2:14.2.22-0.el8 ceph 19 M 2024-04-20T16:55:16.118 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-selinux x86_64 2:14.2.22-0.el8 ceph 25 k 2024-04-20T16:55:16.118 INFO:teuthology.orchestra.run.smithi106.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-20T16:55:16.118 INFO:teuthology.orchestra.run.smithi106.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-04-20T16:55:16.118 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs2 x86_64 2:14.2.22-0.el8 ceph 520 k 2024-04-20T16:55:16.118 INFO:teuthology.orchestra.run.smithi106.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-20T16:55:16.118 INFO:teuthology.orchestra.run.smithi106.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-20T16:55:16.118 INFO:teuthology.orchestra.run.smithi106.stdout: libradosstriper1 x86_64 2:14.2.22-0.el8 ceph 376 k 2024-04-20T16:55:16.118 INFO:teuthology.orchestra.run.smithi106.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-20T16:55:16.118 INFO:teuthology.orchestra.run.smithi106.stdout: librgw2 x86_64 2:14.2.22-0.el8 ceph 5.3 M 2024-04-20T16:55:16.118 INFO:teuthology.orchestra.run.smithi106.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-20T16:55:16.118 INFO:teuthology.orchestra.run.smithi106.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-20T16:55:16.118 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-argparse x86_64 2:14.2.22-0.el8 ceph 39 k 2024-04-20T16:55:16.119 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cephfs x86_64 2:14.2.22-0.el8 ceph 159 k 2024-04-20T16:55:16.119 INFO:teuthology.orchestra.run.smithi106.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-20T16:55:16.119 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rados x86_64 2:14.2.22-0.el8 ceph 323 k 2024-04-20T16:55:16.119 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rbd x86_64 2:14.2.22-0.el8 ceph 291 k 2024-04-20T16:55:16.119 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rgw x86_64 2:14.2.22-0.el8 ceph 113 k 2024-04-20T16:55:16.119 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:55:16.119 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-20T16:55:16.119 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:55:16.119 INFO:teuthology.orchestra.run.smithi106.stdout:Install 20 Packages 2024-04-20T16:55:16.119 INFO:teuthology.orchestra.run.smithi106.stdout:Upgrade 2 Packages 2024-04-20T16:55:16.119 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:55:16.120 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 45 M 2024-04-20T16:55:16.120 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-20T16:55:16.706 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-04-20T16:55:16.830 INFO:teuthology.orchestra.run.smithi106.stdout:(1/22): ceph-base-14.2.22-0.el8.x86_64.rpm 42 MB/s | 5.4 MB 00:00 2024-04-20T16:55:16.855 INFO:teuthology.orchestra.run.smithi106.stdout:(2/22): ceph-selinux-14.2.22-0.el8.x86_64.rpm 998 kB/s | 25 kB 00:00 2024-04-20T16:55:16.889 INFO:teuthology.orchestra.run.smithi106.stdout:(3/22): libcephfs2-14.2.22-0.el8.x86_64.rpm 15 MB/s | 520 kB 00:00 2024-04-20T16:55:16.914 INFO:teuthology.orchestra.run.smithi106.stdout:(4/22): libradosstriper1-14.2.22-0.el8.x86_64.r 15 MB/s | 376 kB 00:00 2024-04-20T16:55:16.922 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-04-20T16:55:16.971 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-base-2:14.2.22-0.el8.x86_64 20/24 2024-04-20T16:55:16.998 INFO:teuthology.orchestra.run.smithi106.stdout:(5/22): ceph-radosgw-14.2.22-0.el8.x86_64.rpm 20 MB/s | 6.0 MB 00:00 2024-04-20T16:55:17.041 INFO:teuthology.orchestra.run.smithi106.stdout:(6/22): python3-ceph-argparse-14.2.22-0.el8.x86 929 kB/s | 39 kB 00:00 2024-04-20T16:55:17.082 INFO:teuthology.orchestra.run.smithi106.stdout:(7/22): python3-cephfs-14.2.22-0.el8.x86_64.rpm 3.8 MB/s | 159 kB 00:00 2024-04-20T16:55:17.108 INFO:teuthology.orchestra.run.smithi106.stdout:(8/22): python3-rados-14.2.22-0.el8.x86_64.rpm 13 MB/s | 323 kB 00:00 2024-04-20T16:55:17.133 INFO:teuthology.orchestra.run.smithi106.stdout:(9/22): python3-rbd-14.2.22-0.el8.x86_64.rpm 11 MB/s | 291 kB 00:00 2024-04-20T16:55:17.158 INFO:teuthology.orchestra.run.smithi106.stdout:(10/22): python3-rgw-14.2.22-0.el8.x86_64.rpm 4.4 MB/s | 113 kB 00:00 2024-04-20T16:55:17.172 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-04-20T16:55:17.200 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-base-2:14.2.22-0.el8.x86_64 20/24 2024-04-20T16:55:17.233 INFO:teuthology.orchestra.run.smithi106.stdout:(11/22): librgw2-14.2.22-0.el8.x86_64.rpm 17 MB/s | 5.3 MB 00:00 2024-04-20T16:55:17.426 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-selinux-2:14.2.22-0.el8.x86_64 21/24 2024-04-20T16:55:17.459 INFO:teuthology.orchestra.run.smithi106.stdout:(12/22): ceph-common-14.2.22-0.el8.x86_64.rpm 26 MB/s | 19 MB 00:00 2024-04-20T16:55:17.709 INFO:teuthology.orchestra.run.smithi106.stdout:(13/22): librdkafka-1.6.1-1.el8.x86_64.rpm 1.2 MB/s | 677 kB 00:00 2024-04-20T16:55:17.768 INFO:teuthology.orchestra.run.smithi106.stdout:(14/22): librabbitmq-0.9.0-5.el8.x86_64.rpm 809 kB/s | 47 kB 00:00 2024-04-20T16:55:17.793 INFO:teuthology.orchestra.run.smithi106.stdout:(15/22): python3-prettytable-0.7.2-14.el8.noarc 132 kB/s | 44 kB 00:00 2024-04-20T16:55:17.815 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-04-20T16:55:17.835 INFO:teuthology.orchestra.run.smithi106.stdout:(16/22): lttng-ust-2.8.1-11.el8.x86_64.rpm 432 kB/s | 259 kB 00:00 2024-04-20T16:55:17.902 INFO:teuthology.orchestra.run.smithi106.stdout:(17/22): gperftools-libs-2.7-9.el8.x86_64.rpm 2.2 MB/s | 306 kB 00:00 2024-04-20T16:55:17.977 INFO:teuthology.orchestra.run.smithi106.stdout:(18/22): liboath-2.6.2-3.el8.x86_64.rpm 414 kB/s | 59 kB 00:00 2024-04-20T16:55:18.052 INFO:teuthology.orchestra.run.smithi106.stdout:(19/22): librados2-14.2.22-0.el8.x86_64.rpm 47 MB/s | 3.5 MB 00:00 2024-04-20T16:55:18.070 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-base-2:14.2.22-0.el8.x86_64 20/24 2024-04-20T16:55:18.077 INFO:teuthology.orchestra.run.smithi106.stdout:(20/22): leveldb-1.22-1.el8.x86_64.rpm 636 kB/s | 181 kB 00:00 2024-04-20T16:55:18.103 INFO:teuthology.orchestra.run.smithi106.stdout:(21/22): libunwind-1.3.1-3.el8.x86_64.rpm 374 kB/s | 75 kB 00:00 2024-04-20T16:55:18.145 INFO:teuthology.orchestra.run.smithi106.stdout:(22/22): librbd1-14.2.22-0.el8.x86_64.rpm 19 MB/s | 1.7 MB 00:00 2024-04-20T16:55:18.147 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:55:18.147 INFO:teuthology.orchestra.run.smithi106.stdout:Total 22 MB/s | 45 MB 00:02 2024-04-20T16:55:18.189 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-20T16:55:18.204 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-base-2:14.2.22-0.el8.x86_64 20/24 2024-04-20T16:55:18.237 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-20T16:55:18.237 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-20T16:55:18.329 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-selinux-2:14.2.22-0.el8.x86_64 21/24 2024-04-20T16:55:18.525 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-20T16:55:18.528 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-20T16:55:19.476 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-20T16:55:19.692 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/24 2024-04-20T16:55:19.877 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/24 2024-04-20T16:55:20.290 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/24 2024-04-20T16:55:20.719 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : librados2-2:14.2.22-0.el8.x86_64 3/24 2024-04-20T16:55:20.814 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librados2-2:14.2.22-0.el8.x86_64 3/24 2024-04-20T16:55:21.061 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rados-2:14.2.22-0.el8.x86_64 4/24 2024-04-20T16:55:21.219 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-04-20T16:55:21.425 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-04-20T16:55:21.644 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : librbd1-2:14.2.22-0.el8.x86_64 6/24 2024-04-20T16:55:21.708 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librbd1-2:14.2.22-0.el8.x86_64 6/24 2024-04-20T16:55:22.291 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/24 2024-04-20T16:55:22.435 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librgw2-2:14.2.22-0.el8.x86_64 8/24 2024-04-20T16:55:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librgw2-2:14.2.22-0.el8.x86_64 8/24 2024-04-20T16:55:22.666 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : leveldb-1.22-1.el8.x86_64 9/24 2024-04-20T16:55:22.853 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-ceph-argparse-2:14.2.22-0.el8.x86_64 10/24 2024-04-20T16:55:23.038 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cephfs-2:14.2.22-0.el8.x86_64 11/24 2024-04-20T16:55:23.255 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rgw-2:14.2.22-0.el8.x86_64 12/24 2024-04-20T16:55:23.478 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rbd-2:14.2.22-0.el8.x86_64 13/24 2024-04-20T16:55:23.602 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libradosstriper1-2:14.2.22-0.el8.x86_64 14/24 2024-04-20T16:55:23.685 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libradosstriper1-2:14.2.22-0.el8.x86_64 14/24 2024-04-20T16:55:23.904 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/24 2024-04-20T16:55:24.118 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/24 2024-04-20T16:55:24.320 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 17/24 2024-04-20T16:55:24.461 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 18/24 2024-04-20T16:55:26.564 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-04-20T16:55:26.802 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-04-20T16:55:27.427 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-common-2:14.2.22-0.el8.x86_64 19/24 2024-04-20T16:55:27.673 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-base-2:14.2.22-0.el8.x86_64 20/24 2024-04-20T16:55:27.758 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-base-2:14.2.22-0.el8.x86_64 20/24 2024-04-20T16:55:27.885 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-selinux-2:14.2.22-0.el8.x86_64 21/24 2024-04-20T16:55:37.058 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-selinux-2:14.2.22-0.el8.x86_64 21/24 2024-04-20T16:55:37.058 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /sys 2024-04-20T16:55:37.058 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /proc 2024-04-20T16:55:37.058 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /dev 2024-04-20T16:55:37.058 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /run 2024-04-20T16:55:37.058 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /mnt 2024-04-20T16:55:37.058 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /var/tmp 2024-04-20T16:55:37.058 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /home 2024-04-20T16:55:37.058 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /tmp 2024-04-20T16:55:37.058 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /dev 2024-04-20T16:55:37.059 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:55:37.801 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-radosgw-2:14.2.22-0.el8.x86_64 22/24 2024-04-20T16:55:37.822 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-radosgw-2:14.2.22-0.el8.x86_64 22/24 2024-04-20T16:55:37.822 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 23/24 2024-04-20T16:55:37.891 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-selinux-2:14.2.22-0.el8.x86_64 21/24 2024-04-20T16:55:37.891 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /sys 2024-04-20T16:55:37.891 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /proc 2024-04-20T16:55:37.891 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /dev 2024-04-20T16:55:37.891 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /run 2024-04-20T16:55:37.891 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /mnt 2024-04-20T16:55:37.892 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /var/tmp 2024-04-20T16:55:37.892 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /home 2024-04-20T16:55:37.892 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /tmp 2024-04-20T16:55:37.892 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /dev 2024-04-20T16:55:37.892 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:55:38.044 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 23/24 2024-04-20T16:55:38.044 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 24/24 2024-04-20T16:55:38.609 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-radosgw-2:14.2.22-0.el8.x86_64 22/24 2024-04-20T16:55:38.628 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-radosgw-2:14.2.22-0.el8.x86_64 22/24 2024-04-20T16:55:38.628 INFO:teuthology.orchestra.run.smithi171.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 23/24 2024-04-20T16:55:38.767 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 24/24 2024-04-20T16:55:38.767 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-base-2:14.2.22-0.el8.x86_64 1/24 2024-04-20T16:55:38.767 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-common-2:14.2.22-0.el8.x86_64 2/24 2024-04-20T16:55:38.767 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-radosgw-2:14.2.22-0.el8.x86_64 3/24 2024-04-20T16:55:38.767 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-selinux-2:14.2.22-0.el8.x86_64 4/24 2024-04-20T16:55:38.767 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-04-20T16:55:38.767 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libradosstriper1-2:14.2.22-0.el8.x86_64 6/24 2024-04-20T16:55:38.767 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librgw2-2:14.2.22-0.el8.x86_64 7/24 2024-04-20T16:55:38.767 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-ceph-argparse-2:14.2.22-0.el8.x86_64 8/24 2024-04-20T16:55:38.767 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cephfs-2:14.2.22-0.el8.x86_64 9/24 2024-04-20T16:55:38.767 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rados-2:14.2.22-0.el8.x86_64 10/24 2024-04-20T16:55:38.768 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rbd-2:14.2.22-0.el8.x86_64 11/24 2024-04-20T16:55:38.768 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rgw-2:14.2.22-0.el8.x86_64 12/24 2024-04-20T16:55:38.768 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 13/24 2024-04-20T16:55:38.768 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 14/24 2024-04-20T16:55:38.768 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 15/24 2024-04-20T16:55:38.768 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 16/24 2024-04-20T16:55:38.768 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 17/24 2024-04-20T16:55:38.768 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : leveldb-1.22-1.el8.x86_64 18/24 2024-04-20T16:55:38.768 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/24 2024-04-20T16:55:38.768 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/24 2024-04-20T16:55:38.768 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librados2-2:14.2.22-0.el8.x86_64 21/24 2024-04-20T16:55:38.768 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 22/24 2024-04-20T16:55:38.769 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librbd1-2:14.2.22-0.el8.x86_64 23/24 2024-04-20T16:55:38.860 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 23/24 2024-04-20T16:55:38.860 INFO:teuthology.orchestra.run.smithi171.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 24/24 2024-04-20T16:55:39.645 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 24/24 2024-04-20T16:55:39.646 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:55:39.646 INFO:teuthology.orchestra.run.smithi005.stdout:Upgraded: 2024-04-20T16:55:39.646 INFO:teuthology.orchestra.run.smithi005.stdout: librados2-2:14.2.22-0.el8.x86_64 librbd1-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:39.646 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-04-20T16:55:39.646 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-base-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:39.646 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-common-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:39.646 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-radosgw-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:39.646 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-selinux-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:39.646 INFO:teuthology.orchestra.run.smithi005.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-20T16:55:39.646 INFO:teuthology.orchestra.run.smithi005.stdout: leveldb-1.22-1.el8.x86_64 2024-04-20T16:55:39.646 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs2-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:39.646 INFO:teuthology.orchestra.run.smithi005.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-20T16:55:39.646 INFO:teuthology.orchestra.run.smithi005.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-20T16:55:39.647 INFO:teuthology.orchestra.run.smithi005.stdout: libradosstriper1-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:39.647 INFO:teuthology.orchestra.run.smithi005.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-20T16:55:39.647 INFO:teuthology.orchestra.run.smithi005.stdout: librgw2-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:39.647 INFO:teuthology.orchestra.run.smithi005.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-20T16:55:39.647 INFO:teuthology.orchestra.run.smithi005.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-20T16:55:39.647 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-argparse-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:39.647 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cephfs-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:39.647 INFO:teuthology.orchestra.run.smithi005.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-20T16:55:39.647 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rados-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:39.647 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rbd-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:39.647 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rgw-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:39.647 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:55:39.647 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-04-20T16:55:39.670 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 24/24 2024-04-20T16:55:39.670 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-base-2:14.2.22-0.el8.x86_64 1/24 2024-04-20T16:55:39.670 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-common-2:14.2.22-0.el8.x86_64 2/24 2024-04-20T16:55:39.670 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-radosgw-2:14.2.22-0.el8.x86_64 3/24 2024-04-20T16:55:39.670 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-selinux-2:14.2.22-0.el8.x86_64 4/24 2024-04-20T16:55:39.670 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-04-20T16:55:39.670 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libradosstriper1-2:14.2.22-0.el8.x86_64 6/24 2024-04-20T16:55:39.670 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librgw2-2:14.2.22-0.el8.x86_64 7/24 2024-04-20T16:55:39.670 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-ceph-argparse-2:14.2.22-0.el8.x86_64 8/24 2024-04-20T16:55:39.670 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cephfs-2:14.2.22-0.el8.x86_64 9/24 2024-04-20T16:55:39.671 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rados-2:14.2.22-0.el8.x86_64 10/24 2024-04-20T16:55:39.671 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rbd-2:14.2.22-0.el8.x86_64 11/24 2024-04-20T16:55:39.671 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rgw-2:14.2.22-0.el8.x86_64 12/24 2024-04-20T16:55:39.671 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 13/24 2024-04-20T16:55:39.671 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 14/24 2024-04-20T16:55:39.671 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 15/24 2024-04-20T16:55:39.671 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 16/24 2024-04-20T16:55:39.671 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 17/24 2024-04-20T16:55:39.671 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : leveldb-1.22-1.el8.x86_64 18/24 2024-04-20T16:55:39.671 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/24 2024-04-20T16:55:39.671 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/24 2024-04-20T16:55:39.671 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librados2-2:14.2.22-0.el8.x86_64 21/24 2024-04-20T16:55:39.671 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 22/24 2024-04-20T16:55:39.671 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librbd1-2:14.2.22-0.el8.x86_64 23/24 2024-04-20T16:55:39.977 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-test 2024-04-20T16:55:40.522 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 24/24 2024-04-20T16:55:40.523 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:55:40.523 INFO:teuthology.orchestra.run.smithi171.stdout:Upgraded: 2024-04-20T16:55:40.523 INFO:teuthology.orchestra.run.smithi171.stdout: librados2-2:14.2.22-0.el8.x86_64 librbd1-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:40.523 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-04-20T16:55:40.523 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-base-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:40.523 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-common-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:40.523 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-radosgw-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:40.523 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-selinux-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:40.523 INFO:teuthology.orchestra.run.smithi171.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-20T16:55:40.523 INFO:teuthology.orchestra.run.smithi171.stdout: leveldb-1.22-1.el8.x86_64 2024-04-20T16:55:40.523 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs2-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:40.523 INFO:teuthology.orchestra.run.smithi171.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-20T16:55:40.524 INFO:teuthology.orchestra.run.smithi171.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-20T16:55:40.524 INFO:teuthology.orchestra.run.smithi171.stdout: libradosstriper1-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:40.524 INFO:teuthology.orchestra.run.smithi171.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-20T16:55:40.524 INFO:teuthology.orchestra.run.smithi171.stdout: librgw2-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:40.524 INFO:teuthology.orchestra.run.smithi171.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-20T16:55:40.524 INFO:teuthology.orchestra.run.smithi171.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-20T16:55:40.524 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-argparse-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:40.524 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cephfs-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:40.524 INFO:teuthology.orchestra.run.smithi171.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-20T16:55:40.524 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rados-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:40.524 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rbd-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:40.524 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rgw-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:40.524 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:55:40.525 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-04-20T16:55:40.734 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:00:43 ago on Sat 20 Apr 2024 04:54:57 PM UTC. 2024-04-20T16:55:40.869 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-test 2024-04-20T16:55:41.596 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:00:39 ago on Sat 20 Apr 2024 04:55:02 PM UTC. 2024-04-20T16:55:42.489 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-04-20T16:55:42.490 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:55:42.490 INFO:teuthology.orchestra.run.smithi005.stdout: Package Architecture Version Repository Size 2024-04-20T16:55:42.490 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:55:42.490 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-04-20T16:55:42.490 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-test x86_64 2:14.2.22-0.el8 ceph 35 M 2024-04-20T16:55:42.490 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-04-20T16:55:42.490 INFO:teuthology.orchestra.run.smithi005.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-20T16:55:42.491 INFO:teuthology.orchestra.run.smithi005.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-20T16:55:42.491 INFO:teuthology.orchestra.run.smithi005.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-20T16:55:42.491 INFO:teuthology.orchestra.run.smithi005.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-20T16:55:42.491 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:55:42.491 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-04-20T16:55:42.491 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:55:42.491 INFO:teuthology.orchestra.run.smithi005.stdout:Install 5 Packages 2024-04-20T16:55:42.491 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:55:42.492 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 36 M 2024-04-20T16:55:42.492 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 127 M 2024-04-20T16:55:42.492 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-04-20T16:55:43.065 INFO:teuthology.orchestra.run.smithi005.stdout:(1/5): ceph-test-14.2.22-0.el8.x86_64.rpm 68 MB/s | 35 MB 00:00 2024-04-20T16:55:43.090 INFO:teuthology.orchestra.run.smithi005.stdout:(2/5): oniguruma-6.8.2-3.el8.x86_64.rpm 349 kB/s | 188 kB 00:00 2024-04-20T16:55:43.115 INFO:teuthology.orchestra.run.smithi005.stdout:(3/5): jq-1.6-8.el8.x86_64.rpm 360 kB/s | 203 kB 00:00 2024-04-20T16:55:43.190 INFO:teuthology.orchestra.run.smithi005.stdout:(4/5): xmlstarlet-1.6.1-20.el8.x86_64.rpm 706 kB/s | 71 kB 00:00 2024-04-20T16:55:43.232 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-04-20T16:55:43.233 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:55:43.233 INFO:teuthology.orchestra.run.smithi171.stdout: Package Architecture Version Repository Size 2024-04-20T16:55:43.234 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:55:43.234 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-04-20T16:55:43.234 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-test x86_64 2:14.2.22-0.el8 ceph 35 M 2024-04-20T16:55:43.234 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-04-20T16:55:43.234 INFO:teuthology.orchestra.run.smithi171.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-20T16:55:43.234 INFO:teuthology.orchestra.run.smithi171.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-20T16:55:43.234 INFO:teuthology.orchestra.run.smithi171.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-20T16:55:43.234 INFO:teuthology.orchestra.run.smithi171.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-20T16:55:43.234 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:55:43.234 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-04-20T16:55:43.234 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:55:43.234 INFO:teuthology.orchestra.run.smithi171.stdout:Install 5 Packages 2024-04-20T16:55:43.234 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:55:43.235 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 36 M 2024-04-20T16:55:43.235 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 127 M 2024-04-20T16:55:43.235 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-04-20T16:55:43.598 INFO:teuthology.orchestra.run.smithi005.stdout:(5/5): socat-1.7.4.1-1.el8.x86_64.rpm 607 kB/s | 323 kB 00:00 2024-04-20T16:55:43.599 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:55:43.599 INFO:teuthology.orchestra.run.smithi005.stdout:Total 32 MB/s | 36 MB 00:01 2024-04-20T16:55:43.644 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-04-20T16:55:43.666 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-04-20T16:55:43.666 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-04-20T16:55:43.786 INFO:teuthology.orchestra.run.smithi171.stdout:(1/5): ceph-test-14.2.22-0.el8.x86_64.rpm 71 MB/s | 35 MB 00:00 2024-04-20T16:55:43.853 INFO:teuthology.orchestra.run.smithi171.stdout:(2/5): oniguruma-6.8.2-3.el8.x86_64.rpm 336 kB/s | 188 kB 00:00 2024-04-20T16:55:43.878 INFO:teuthology.orchestra.run.smithi171.stdout:(3/5): jq-1.6-8.el8.x86_64.rpm 347 kB/s | 203 kB 00:00 2024-04-20T16:55:43.900 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-04-20T16:55:43.903 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-04-20T16:55:43.945 INFO:teuthology.orchestra.run.smithi171.stdout:(4/5): xmlstarlet-1.6.1-20.el8.x86_64.rpm 768 kB/s | 71 kB 00:00 2024-04-20T16:55:44.271 INFO:teuthology.orchestra.run.smithi171.stdout:(5/5): socat-1.7.4.1-1.el8.x86_64.rpm 668 kB/s | 323 kB 00:00 2024-04-20T16:55:44.271 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:55:44.272 INFO:teuthology.orchestra.run.smithi171.stdout:Total 34 MB/s | 36 MB 00:01 2024-04-20T16:55:44.299 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-04-20T16:55:44.314 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-04-20T16:55:44.335 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-04-20T16:55:44.335 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-04-20T16:55:44.543 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/5 2024-04-20T16:55:44.558 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-04-20T16:55:44.561 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-04-20T16:55:44.722 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/5 2024-04-20T16:55:44.863 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-20T16:55:44.947 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-20T16:55:44.959 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-04-20T16:55:45.240 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/5 2024-04-20T16:55:45.444 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/5 2024-04-20T16:55:45.625 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-20T16:55:45.701 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-20T16:55:47.128 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-selinux-2:14.2.22-0.el8.x86_64 21/24 2024-04-20T16:55:47.128 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /sys 2024-04-20T16:55:47.128 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /proc 2024-04-20T16:55:47.128 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /dev 2024-04-20T16:55:47.129 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /run 2024-04-20T16:55:47.129 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /mnt 2024-04-20T16:55:47.129 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /var/tmp 2024-04-20T16:55:47.129 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /home 2024-04-20T16:55:47.129 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /tmp 2024-04-20T16:55:47.129 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /dev 2024-04-20T16:55:47.129 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:55:47.814 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-radosgw-2:14.2.22-0.el8.x86_64 22/24 2024-04-20T16:55:47.832 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-radosgw-2:14.2.22-0.el8.x86_64 22/24 2024-04-20T16:55:47.832 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 23/24 2024-04-20T16:55:48.042 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 23/24 2024-04-20T16:55:48.042 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 24/24 2024-04-20T16:55:48.377 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : jq-1.6-8.el8.x86_64 4/5 2024-04-20T16:55:48.612 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-test-2:14.2.22-0.el8.x86_64 5/5 2024-04-20T16:55:48.776 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 24/24 2024-04-20T16:55:48.776 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-base-2:14.2.22-0.el8.x86_64 1/24 2024-04-20T16:55:48.776 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-common-2:14.2.22-0.el8.x86_64 2/24 2024-04-20T16:55:48.776 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-radosgw-2:14.2.22-0.el8.x86_64 3/24 2024-04-20T16:55:48.777 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-selinux-2:14.2.22-0.el8.x86_64 4/24 2024-04-20T16:55:48.777 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libcephfs2-2:14.2.22-0.el8.x86_64 5/24 2024-04-20T16:55:48.777 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libradosstriper1-2:14.2.22-0.el8.x86_64 6/24 2024-04-20T16:55:48.777 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librgw2-2:14.2.22-0.el8.x86_64 7/24 2024-04-20T16:55:48.777 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-ceph-argparse-2:14.2.22-0.el8.x86_64 8/24 2024-04-20T16:55:48.777 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cephfs-2:14.2.22-0.el8.x86_64 9/24 2024-04-20T16:55:48.777 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rados-2:14.2.22-0.el8.x86_64 10/24 2024-04-20T16:55:48.777 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rbd-2:14.2.22-0.el8.x86_64 11/24 2024-04-20T16:55:48.777 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rgw-2:14.2.22-0.el8.x86_64 12/24 2024-04-20T16:55:48.777 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 13/24 2024-04-20T16:55:48.777 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 14/24 2024-04-20T16:55:48.777 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 15/24 2024-04-20T16:55:48.777 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 16/24 2024-04-20T16:55:48.777 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 17/24 2024-04-20T16:55:48.777 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : leveldb-1.22-1.el8.x86_64 18/24 2024-04-20T16:55:48.778 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/24 2024-04-20T16:55:48.778 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/24 2024-04-20T16:55:48.778 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librados2-2:14.2.22-0.el8.x86_64 21/24 2024-04-20T16:55:48.778 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 22/24 2024-04-20T16:55:48.778 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librbd1-2:14.2.22-0.el8.x86_64 23/24 2024-04-20T16:55:48.910 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-test-2:14.2.22-0.el8.x86_64 5/5 2024-04-20T16:55:48.910 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-test-2:14.2.22-0.el8.x86_64 1/5 2024-04-20T16:55:48.911 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : jq-1.6-8.el8.x86_64 2/5 2024-04-20T16:55:48.911 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-20T16:55:48.911 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/5 2024-04-20T16:55:48.986 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : jq-1.6-8.el8.x86_64 4/5 2024-04-20T16:55:49.248 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-test-2:14.2.22-0.el8.x86_64 5/5 2024-04-20T16:55:49.389 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 5/5 2024-04-20T16:55:49.389 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:55:49.390 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-04-20T16:55:49.390 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-test-2:14.2.22-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-04-20T16:55:49.390 INFO:teuthology.orchestra.run.smithi005.stdout: oniguruma-6.8.2-3.el8.x86_64 socat-1.7.4.1-1.el8.x86_64 2024-04-20T16:55:49.390 INFO:teuthology.orchestra.run.smithi005.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-20T16:55:49.390 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:55:49.390 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-04-20T16:55:49.536 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-test-2:14.2.22-0.el8.x86_64 5/5 2024-04-20T16:55:49.536 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-test-2:14.2.22-0.el8.x86_64 1/5 2024-04-20T16:55:49.537 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : jq-1.6-8.el8.x86_64 2/5 2024-04-20T16:55:49.537 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-20T16:55:49.537 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/5 2024-04-20T16:55:49.626 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph 2024-04-20T16:55:49.692 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 24/24 2024-04-20T16:55:49.692 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:55:49.692 INFO:teuthology.orchestra.run.smithi106.stdout:Upgraded: 2024-04-20T16:55:49.692 INFO:teuthology.orchestra.run.smithi106.stdout: librados2-2:14.2.22-0.el8.x86_64 librbd1-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:49.692 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-20T16:55:49.692 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-base-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:49.692 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-common-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:49.692 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-radosgw-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:49.692 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-selinux-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:49.692 INFO:teuthology.orchestra.run.smithi106.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-20T16:55:49.693 INFO:teuthology.orchestra.run.smithi106.stdout: leveldb-1.22-1.el8.x86_64 2024-04-20T16:55:49.693 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs2-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:49.693 INFO:teuthology.orchestra.run.smithi106.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-20T16:55:49.693 INFO:teuthology.orchestra.run.smithi106.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-20T16:55:49.693 INFO:teuthology.orchestra.run.smithi106.stdout: libradosstriper1-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:49.693 INFO:teuthology.orchestra.run.smithi106.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-20T16:55:49.693 INFO:teuthology.orchestra.run.smithi106.stdout: librgw2-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:49.693 INFO:teuthology.orchestra.run.smithi106.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-20T16:55:49.693 INFO:teuthology.orchestra.run.smithi106.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-20T16:55:49.693 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-argparse-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:49.693 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cephfs-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:49.693 INFO:teuthology.orchestra.run.smithi106.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-20T16:55:49.693 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rados-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:49.693 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rbd-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:49.693 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rgw-2:14.2.22-0.el8.x86_64 2024-04-20T16:55:49.694 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:55:49.694 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-20T16:55:49.934 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 5/5 2024-04-20T16:55:49.934 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:55:49.934 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-04-20T16:55:49.934 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-test-2:14.2.22-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-04-20T16:55:49.934 INFO:teuthology.orchestra.run.smithi171.stdout: oniguruma-6.8.2-3.el8.x86_64 socat-1.7.4.1-1.el8.x86_64 2024-04-20T16:55:49.934 INFO:teuthology.orchestra.run.smithi171.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-20T16:55:49.934 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:55:49.934 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-04-20T16:55:50.006 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-test 2024-04-20T16:55:50.161 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph 2024-04-20T16:55:50.378 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:00:53 ago on Sat 20 Apr 2024 04:54:57 PM UTC. 2024-04-20T16:55:50.730 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:00:38 ago on Sat 20 Apr 2024 04:55:12 PM UTC. 2024-04-20T16:55:50.895 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:00:48 ago on Sat 20 Apr 2024 04:55:02 PM UTC. 2024-04-20T16:55:52.116 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-04-20T16:55:52.120 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:55:52.120 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-04-20T16:55:52.120 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:55:52.120 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-04-20T16:55:52.120 INFO:teuthology.orchestra.run.smithi005.stdout: ceph x86_64 2:14.2.22-0.el8 ceph 6.3 k 2024-04-20T16:55:52.120 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-04-20T16:55:52.121 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mds x86_64 2:14.2.22-0.el8 ceph 2.0 M 2024-04-20T16:55:52.121 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr x86_64 2:14.2.22-0.el8 ceph 1.6 M 2024-04-20T16:55:52.121 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mon x86_64 2:14.2.22-0.el8 ceph 4.5 M 2024-04-20T16:55:52.121 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-osd x86_64 2:14.2.22-0.el8 ceph 17 M 2024-04-20T16:55:52.121 INFO:teuthology.orchestra.run.smithi005.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-20T16:55:52.121 INFO:teuthology.orchestra.run.smithi005.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-20T16:55:52.121 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-20T16:55:52.121 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-20T16:55:52.121 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-20T16:55:52.121 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-20T16:55:52.121 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-20T16:55:52.122 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-20T16:55:52.122 INFO:teuthology.orchestra.run.smithi005.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-20T16:55:52.122 INFO:teuthology.orchestra.run.smithi005.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-20T16:55:52.122 INFO:teuthology.orchestra.run.smithi005.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-20T16:55:52.122 INFO:teuthology.orchestra.run.smithi005.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-20T16:55:52.122 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-20T16:55:52.122 INFO:teuthology.orchestra.run.smithi005.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-20T16:55:52.122 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-20T16:55:52.122 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-20T16:55:52.122 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-20T16:55:52.122 INFO:teuthology.orchestra.run.smithi005.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-20T16:55:52.123 INFO:teuthology.orchestra.run.smithi005.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-20T16:55:52.123 INFO:teuthology.orchestra.run.smithi005.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-20T16:55:52.123 INFO:teuthology.orchestra.run.smithi005.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-20T16:55:52.123 INFO:teuthology.orchestra.run.smithi005.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-20T16:55:52.123 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-20T16:55:52.123 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-20T16:55:52.123 INFO:teuthology.orchestra.run.smithi005.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-20T16:55:52.123 INFO:teuthology.orchestra.run.smithi005.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-20T16:55:52.123 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:55:52.123 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-04-20T16:55:52.123 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:55:52.123 INFO:teuthology.orchestra.run.smithi005.stdout:Install 31 Packages 2024-04-20T16:55:52.123 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:55:52.124 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 28 M 2024-04-20T16:55:52.124 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 93 M 2024-04-20T16:55:52.124 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-04-20T16:55:52.365 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-20T16:55:52.366 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:55:52.366 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-04-20T16:55:52.366 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:55:52.366 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-20T16:55:52.367 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-test x86_64 2:14.2.22-0.el8 ceph 35 M 2024-04-20T16:55:52.367 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-04-20T16:55:52.367 INFO:teuthology.orchestra.run.smithi106.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-20T16:55:52.367 INFO:teuthology.orchestra.run.smithi106.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-20T16:55:52.367 INFO:teuthology.orchestra.run.smithi106.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-20T16:55:52.367 INFO:teuthology.orchestra.run.smithi106.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-20T16:55:52.367 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:55:52.367 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-20T16:55:52.367 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:55:52.367 INFO:teuthology.orchestra.run.smithi106.stdout:Install 5 Packages 2024-04-20T16:55:52.367 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:55:52.368 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 36 M 2024-04-20T16:55:52.368 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 127 M 2024-04-20T16:55:52.368 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-20T16:55:52.381 INFO:teuthology.orchestra.run.smithi005.stdout:(1/31): ceph-14.2.22-0.el8.x86_64.rpm 94 kB/s | 6.3 kB 00:00 2024-04-20T16:55:52.434 INFO:teuthology.orchestra.run.smithi005.stdout:(2/31): ceph-mgr-14.2.22-0.el8.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-04-20T16:55:52.485 INFO:teuthology.orchestra.run.smithi005.stdout:(3/31): ceph-mds-14.2.22-0.el8.x86_64.rpm 12 MB/s | 2.0 MB 00:00 2024-04-20T16:55:52.541 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-04-20T16:55:52.544 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:55:52.544 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-04-20T16:55:52.544 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:55:52.544 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-04-20T16:55:52.544 INFO:teuthology.orchestra.run.smithi171.stdout: ceph x86_64 2:14.2.22-0.el8 ceph 6.3 k 2024-04-20T16:55:52.544 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-04-20T16:55:52.545 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mds x86_64 2:14.2.22-0.el8 ceph 2.0 M 2024-04-20T16:55:52.545 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr x86_64 2:14.2.22-0.el8 ceph 1.6 M 2024-04-20T16:55:52.545 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mon x86_64 2:14.2.22-0.el8 ceph 4.5 M 2024-04-20T16:55:52.545 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-osd x86_64 2:14.2.22-0.el8 ceph 17 M 2024-04-20T16:55:52.545 INFO:teuthology.orchestra.run.smithi171.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-20T16:55:52.545 INFO:teuthology.orchestra.run.smithi171.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-20T16:55:52.545 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-20T16:55:52.545 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-20T16:55:52.545 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-20T16:55:52.545 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-20T16:55:52.545 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-20T16:55:52.545 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-20T16:55:52.545 INFO:teuthology.orchestra.run.smithi171.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-20T16:55:52.546 INFO:teuthology.orchestra.run.smithi171.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-20T16:55:52.546 INFO:teuthology.orchestra.run.smithi171.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-20T16:55:52.546 INFO:teuthology.orchestra.run.smithi171.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-20T16:55:52.546 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-20T16:55:52.546 INFO:teuthology.orchestra.run.smithi171.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-20T16:55:52.546 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-20T16:55:52.546 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-20T16:55:52.546 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-20T16:55:52.546 INFO:teuthology.orchestra.run.smithi171.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-20T16:55:52.546 INFO:teuthology.orchestra.run.smithi171.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-20T16:55:52.546 INFO:teuthology.orchestra.run.smithi171.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-20T16:55:52.547 INFO:teuthology.orchestra.run.smithi171.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-20T16:55:52.547 INFO:teuthology.orchestra.run.smithi171.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-20T16:55:52.547 INFO:teuthology.orchestra.run.smithi171.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-20T16:55:52.547 INFO:teuthology.orchestra.run.smithi171.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-20T16:55:52.547 INFO:teuthology.orchestra.run.smithi171.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-20T16:55:52.547 INFO:teuthology.orchestra.run.smithi171.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-20T16:55:52.547 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:55:52.547 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-04-20T16:55:52.547 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:55:52.547 INFO:teuthology.orchestra.run.smithi171.stdout:Install 31 Packages 2024-04-20T16:55:52.547 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:55:52.548 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 28 M 2024-04-20T16:55:52.548 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 93 M 2024-04-20T16:55:52.548 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-04-20T16:55:52.568 INFO:teuthology.orchestra.run.smithi005.stdout:(4/31): ceph-mon-14.2.22-0.el8.x86_64.rpm 24 MB/s | 4.5 MB 00:00 2024-04-20T16:55:52.793 INFO:teuthology.orchestra.run.smithi171.stdout:(1/31): ceph-14.2.22-0.el8.x86_64.rpm 104 kB/s | 6.3 kB 00:00 2024-04-20T16:55:52.802 INFO:teuthology.orchestra.run.smithi005.stdout:(5/31): ceph-osd-14.2.22-0.el8.x86_64.rpm 46 MB/s | 17 MB 00:00 2024-04-20T16:55:52.844 INFO:teuthology.orchestra.run.smithi171.stdout:(2/31): ceph-mgr-14.2.22-0.el8.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-04-20T16:55:52.886 INFO:teuthology.orchestra.run.smithi171.stdout:(3/31): ceph-mds-14.2.22-0.el8.x86_64.rpm 13 MB/s | 2.0 MB 00:00 2024-04-20T16:55:52.945 INFO:teuthology.orchestra.run.smithi005.stdout:(6/31): python3-mako-1.0.6-14.el8.noarch.rpm 342 kB/s | 157 kB 00:00 2024-04-20T16:55:52.970 INFO:teuthology.orchestra.run.smithi171.stdout:(4/31): ceph-mon-14.2.22-0.el8.x86_64.rpm 25 MB/s | 4.5 MB 00:00 2024-04-20T16:55:53.005 INFO:teuthology.orchestra.run.smithi106.stdout:(1/5): ceph-test-14.2.22-0.el8.x86_64.rpm 77 MB/s | 35 MB 00:00 2024-04-20T16:55:53.011 INFO:teuthology.orchestra.run.smithi005.stdout:(7/31): python3-markupsafe-0.23-19.el8.x86_64.r 88 kB/s | 39 kB 00:00 2024-04-20T16:55:53.031 INFO:teuthology.orchestra.run.smithi106.stdout:(2/5): oniguruma-6.8.2-3.el8.x86_64.rpm 395 kB/s | 188 kB 00:00 2024-04-20T16:55:53.036 INFO:teuthology.orchestra.run.smithi005.stdout:(8/31): python3-pytz-2017.2-11.el8.noarch.rpm 621 kB/s | 57 kB 00:00 2024-04-20T16:55:53.056 INFO:teuthology.orchestra.run.smithi106.stdout:(3/5): jq-1.6-8.el8.x86_64.rpm 405 kB/s | 203 kB 00:00 2024-04-20T16:55:53.120 INFO:teuthology.orchestra.run.smithi005.stdout:(9/31): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.9 MB/s | 249 kB 00:00 2024-04-20T16:55:53.131 INFO:teuthology.orchestra.run.smithi106.stdout:(4/5): xmlstarlet-1.6.1-20.el8.x86_64.rpm 705 kB/s | 71 kB 00:00 2024-04-20T16:55:53.195 INFO:teuthology.orchestra.run.smithi005.stdout:(10/31): python3-cryptography-3.2.1-7.el8.x86_6 8.4 MB/s | 641 kB 00:00 2024-04-20T16:55:53.228 INFO:teuthology.orchestra.run.smithi171.stdout:(5/31): ceph-osd-14.2.22-0.el8.x86_64.rpm 44 MB/s | 17 MB 00:00 2024-04-20T16:55:53.266 INFO:teuthology.orchestra.run.smithi005.stdout:(11/31): python3-pycparser-2.14-14.el8.noarch.r 1.5 MB/s | 109 kB 00:00 2024-04-20T16:55:53.296 INFO:teuthology.orchestra.run.smithi005.stdout:(12/31): python3-pyOpenSSL-19.0.0-1.el8.noarch. 210 kB/s | 103 kB 00:00 2024-04-20T16:55:53.304 INFO:teuthology.orchestra.run.smithi171.stdout:(6/31): python3-mako-1.0.6-14.el8.noarch.rpm 376 kB/s | 157 kB 00:00 2024-04-20T16:55:53.396 INFO:teuthology.orchestra.run.smithi005.stdout:(13/31): python3-werkzeug-0.12.2-4.el8.noarch.r 1.2 MB/s | 457 kB 00:00 2024-04-20T16:55:53.427 INFO:teuthology.orchestra.run.smithi171.stdout:(7/31): python3-markupsafe-0.23-19.el8.x86_64.r 85 kB/s | 39 kB 00:00 2024-04-20T16:55:53.454 INFO:teuthology.orchestra.run.smithi171.stdout:(8/31): python3-pytz-2017.2-11.el8.noarch.rpm 379 kB/s | 57 kB 00:00 2024-04-20T16:55:53.456 INFO:teuthology.orchestra.run.smithi106.stdout:(5/5): socat-1.7.4.1-1.el8.x86_64.rpm 717 kB/s | 323 kB 00:00 2024-04-20T16:55:53.458 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:55:53.458 INFO:teuthology.orchestra.run.smithi106.stdout:Total 33 MB/s | 36 MB 00:01 2024-04-20T16:55:53.500 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-20T16:55:53.520 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-20T16:55:53.520 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-20T16:55:53.530 INFO:teuthology.orchestra.run.smithi171.stdout:(9/31): python3-cffi-1.11.5-6.el8.x86_64.rpm 3.2 MB/s | 249 kB 00:00 2024-04-20T16:55:53.588 INFO:teuthology.orchestra.run.smithi171.stdout:(10/31): python3-cryptography-3.2.1-7.el8.x86_6 11 MB/s | 641 kB 00:00 2024-04-20T16:55:53.605 INFO:teuthology.orchestra.run.smithi005.stdout:(14/31): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 129 kB/s | 44 kB 00:00 2024-04-20T16:55:53.614 INFO:teuthology.orchestra.run.smithi171.stdout:(11/31): python3-pyOpenSSL-19.0.0-1.el8.noarch. 269 kB/s | 103 kB 00:00 2024-04-20T16:55:53.639 INFO:teuthology.orchestra.run.smithi171.stdout:(12/31): python3-pycparser-2.14-14.el8.noarch.r 2.1 MB/s | 109 kB 00:00 2024-04-20T16:55:53.663 INFO:teuthology.orchestra.run.smithi005.stdout:(15/31): python3-beautifulsoup4-4.6.3-2.el8.1.n 504 kB/s | 185 kB 00:00 2024-04-20T16:55:53.740 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-20T16:55:53.743 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-20T16:55:53.764 INFO:teuthology.orchestra.run.smithi005.stdout:(16/31): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2024-04-20T16:55:53.781 INFO:teuthology.orchestra.run.smithi171.stdout:(13/31): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 261 kB/s | 44 kB 00:00 2024-04-20T16:55:53.789 INFO:teuthology.orchestra.run.smithi005.stdout:(17/31): python3-cheroot-8.5.2-1.el8.noarch.rpm 440 kB/s | 173 kB 00:00 2024-04-20T16:55:53.815 INFO:teuthology.orchestra.run.smithi171.stdout:(14/31): python3-werkzeug-0.12.2-4.el8.noarch.r 1.2 MB/s | 457 kB 00:00 2024-04-20T16:55:53.840 INFO:teuthology.orchestra.run.smithi171.stdout:(15/31): python3-beautifulsoup4-4.6.3-2.el8.1.n 923 kB/s | 185 kB 00:00 2024-04-20T16:55:53.856 INFO:teuthology.orchestra.run.smithi005.stdout:(18/31): python3-jaraco-functools-2.0-4.el8.noa 191 kB/s | 18 kB 00:00 2024-04-20T16:55:53.881 INFO:teuthology.orchestra.run.smithi005.stdout:(19/31): python3-logutils-0.3.5-11.el8.noarch.r 530 kB/s | 49 kB 00:00 2024-04-20T16:55:53.915 INFO:teuthology.orchestra.run.smithi005.stdout:(20/31): python3-cherrypy-18.4.0-1.el8.noarch.r 1.2 MB/s | 384 kB 00:00 2024-04-20T16:55:53.949 INFO:teuthology.orchestra.run.smithi005.stdout:(21/31): python3-more-itertools-7.2.0-3.el8.noa 636 kB/s | 59 kB 00:00 2024-04-20T16:55:53.974 INFO:teuthology.orchestra.run.smithi171.stdout:(16/31): python3-cherrypy-18.4.0-1.el8.noarch.r 2.4 MB/s | 384 kB 00:00 2024-04-20T16:55:53.999 INFO:teuthology.orchestra.run.smithi171.stdout:(17/31): python3-jaraco-6.2-6.el8.noarch.rpm 70 kB/s | 11 kB 00:00 2024-04-20T16:55:54.024 INFO:teuthology.orchestra.run.smithi005.stdout:(22/31): python3-portend-2.6-1.el8.noarch.rpm 143 kB/s | 16 kB 00:00 2024-04-20T16:55:54.033 INFO:teuthology.orchestra.run.smithi171.stdout:(18/31): python3-cheroot-8.5.2-1.el8.noarch.rpm 686 kB/s | 173 kB 00:00 2024-04-20T16:55:54.050 INFO:teuthology.orchestra.run.smithi005.stdout:(23/31): python3-simplegeneric-0.8.1-17.el8.noa 186 kB/s | 19 kB 00:00 2024-04-20T16:55:54.100 INFO:teuthology.orchestra.run.smithi171.stdout:(19/31): python3-jaraco-functools-2.0-4.el8.noa 139 kB/s | 18 kB 00:00 2024-04-20T16:55:54.105 INFO:teuthology.orchestra.run.smithi005.stdout:(24/31): python3-pecan-1.3.2-9.el8.noarch.rpm 1.2 MB/s | 283 kB 00:00 2024-04-20T16:55:54.126 INFO:teuthology.orchestra.run.smithi171.stdout:(20/31): python3-logutils-0.3.5-11.el8.noarch.r 386 kB/s | 49 kB 00:00 2024-04-20T16:55:54.133 INFO:teuthology.orchestra.run.smithi005.stdout:(25/31): python3-singledispatch-3.4.0.3-18.el8. 219 kB/s | 24 kB 00:00 2024-04-20T16:55:54.143 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-20T16:55:54.151 INFO:teuthology.orchestra.run.smithi171.stdout:(21/31): python3-more-itertools-7.2.0-3.el8.noa 500 kB/s | 59 kB 00:00 2024-04-20T16:55:54.159 INFO:teuthology.orchestra.run.smithi005.stdout:(26/31): python3-tempora-1.14.1-5.el8.noarch.rp 268 kB/s | 29 kB 00:00 2024-04-20T16:55:54.201 INFO:teuthology.orchestra.run.smithi005.stdout:(27/31): python3-trustme-0.6.0-4.el8.noarch.rpm 290 kB/s | 27 kB 00:00 2024-04-20T16:55:54.234 INFO:teuthology.orchestra.run.smithi005.stdout:(28/31): python3-waitress-1.4.3-1.el8.noarch.rp 2.5 MB/s | 255 kB 00:00 2024-04-20T16:55:54.243 INFO:teuthology.orchestra.run.smithi171.stdout:(22/31): python3-pecan-1.3.2-9.el8.noarch.rpm 1.9 MB/s | 283 kB 00:00 2024-04-20T16:55:54.268 INFO:teuthology.orchestra.run.smithi171.stdout:(23/31): python3-portend-2.6-1.el8.noarch.rpm 109 kB/s | 16 kB 00:00 2024-04-20T16:55:54.293 INFO:teuthology.orchestra.run.smithi005.stdout:(29/31): python3-webtest-2.0.33-1.el8.noarch.rp 928 kB/s | 85 kB 00:00 2024-04-20T16:55:54.294 INFO:teuthology.orchestra.run.smithi171.stdout:(24/31): python3-simplegeneric-0.8.1-17.el8.noa 131 kB/s | 19 kB 00:00 2024-04-20T16:55:54.318 INFO:teuthology.orchestra.run.smithi005.stdout:(30/31): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.5 MB/s | 251 kB 00:00 2024-04-20T16:55:54.327 INFO:teuthology.orchestra.run.smithi171.stdout:(25/31): python3-singledispatch-3.4.0.3-18.el8. 285 kB/s | 24 kB 00:00 2024-04-20T16:55:54.334 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/5 2024-04-20T16:55:54.343 INFO:teuthology.orchestra.run.smithi005.stdout:(31/31): python3-zc-lockfile-2.0-2.el8.noarch.r 211 kB/s | 23 kB 00:00 2024-04-20T16:55:54.345 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:55:54.345 INFO:teuthology.orchestra.run.smithi005.stdout:Total 13 MB/s | 28 MB 00:02 2024-04-20T16:55:54.369 INFO:teuthology.orchestra.run.smithi171.stdout:(26/31): python3-tempora-1.14.1-5.el8.noarch.rp 290 kB/s | 29 kB 00:00 2024-04-20T16:55:54.420 INFO:teuthology.orchestra.run.smithi171.stdout:(27/31): python3-trustme-0.6.0-4.el8.noarch.rpm 219 kB/s | 27 kB 00:00 2024-04-20T16:55:54.431 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-04-20T16:55:54.453 INFO:teuthology.orchestra.run.smithi171.stdout:(28/31): python3-waitress-1.4.3-1.el8.noarch.rp 2.0 MB/s | 255 kB 00:00 2024-04-20T16:55:54.473 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-04-20T16:55:54.473 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-04-20T16:55:54.495 INFO:teuthology.orchestra.run.smithi171.stdout:(29/31): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.0 MB/s | 251 kB 00:00 2024-04-20T16:55:54.521 INFO:teuthology.orchestra.run.smithi171.stdout:(30/31): python3-webtest-2.0.33-1.el8.noarch.rp 850 kB/s | 85 kB 00:00 2024-04-20T16:55:54.562 INFO:teuthology.orchestra.run.smithi171.stdout:(31/31): python3-zc-lockfile-2.0-2.el8.noarch.r 211 kB/s | 23 kB 00:00 2024-04-20T16:55:54.565 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:55:54.565 INFO:teuthology.orchestra.run.smithi171.stdout:Total 14 MB/s | 28 MB 00:02 2024-04-20T16:55:54.567 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/5 2024-04-20T16:55:54.647 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-04-20T16:55:54.687 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-04-20T16:55:54.687 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-04-20T16:55:54.706 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-20T16:55:54.777 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-20T16:55:54.799 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-04-20T16:55:54.802 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-04-20T16:55:54.998 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-04-20T16:55:55.001 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-04-20T16:55:55.944 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-04-20T16:55:56.128 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-04-20T16:55:56.196 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/31 2024-04-20T16:55:56.354 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/31 2024-04-20T16:55:56.445 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/31 2024-04-20T16:55:56.631 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/31 2024-04-20T16:55:56.751 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/31 2024-04-20T16:55:56.922 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/31 2024-04-20T16:55:57.013 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/31 2024-04-20T16:55:57.182 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/31 2024-04-20T16:55:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/31 2024-04-20T16:55:57.374 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/31 2024-04-20T16:55:57.410 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/31 2024-04-20T16:55:57.574 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/31 2024-04-20T16:55:57.581 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/31 2024-04-20T16:55:57.718 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/31 2024-04-20T16:55:57.801 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/31 2024-04-20T16:55:57.947 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/31 2024-04-20T16:55:57.970 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/31 2024-04-20T16:55:58.000 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : jq-1.6-8.el8.x86_64 4/5 2024-04-20T16:55:58.159 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/31 2024-04-20T16:55:58.221 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/31 2024-04-20T16:55:58.221 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-test-2:14.2.22-0.el8.x86_64 5/5 2024-04-20T16:55:58.407 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/31 2024-04-20T16:55:58.491 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/31 2024-04-20T16:55:58.503 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-test-2:14.2.22-0.el8.x86_64 5/5 2024-04-20T16:55:58.503 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-test-2:14.2.22-0.el8.x86_64 1/5 2024-04-20T16:55:58.503 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : jq-1.6-8.el8.x86_64 2/5 2024-04-20T16:55:58.503 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-20T16:55:58.503 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/5 2024-04-20T16:55:58.662 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/31 2024-04-20T16:55:58.762 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/31 2024-04-20T16:55:58.970 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 5/5 2024-04-20T16:55:58.970 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:55:58.970 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-20T16:55:58.970 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-test-2:14.2.22-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-04-20T16:55:58.970 INFO:teuthology.orchestra.run.smithi106.stdout: oniguruma-6.8.2-3.el8.x86_64 socat-1.7.4.1-1.el8.x86_64 2024-04-20T16:55:58.971 INFO:teuthology.orchestra.run.smithi106.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-20T16:55:58.971 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:55:58.971 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-20T16:55:59.011 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/31 2024-04-20T16:55:59.067 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/31 2024-04-20T16:55:59.196 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph 2024-04-20T16:55:59.332 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/31 2024-04-20T16:55:59.445 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/31 2024-04-20T16:55:59.532 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/31 2024-04-20T16:55:59.741 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/31 2024-04-20T16:55:59.756 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/31 2024-04-20T16:55:59.931 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:00:47 ago on Sat 20 Apr 2024 04:55:12 PM UTC. 2024-04-20T16:55:59.945 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/31 2024-04-20T16:55:59.963 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/31 2024-04-20T16:56:00.238 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/31 2024-04-20T16:56:00.263 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/31 2024-04-20T16:56:00.507 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/31 2024-04-20T16:56:00.510 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/31 2024-04-20T16:56:00.707 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/31 2024-04-20T16:56:00.841 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/31 2024-04-20T16:56:00.851 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/31 2024-04-20T16:56:01.078 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/31 2024-04-20T16:56:01.146 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/31 2024-04-20T16:56:01.249 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/31 2024-04-20T16:56:01.375 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/31 2024-04-20T16:56:01.444 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/31 2024-04-20T16:56:01.570 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-20T16:56:01.573 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:56:01.573 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-04-20T16:56:01.573 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:56:01.573 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-20T16:56:01.573 INFO:teuthology.orchestra.run.smithi106.stdout: ceph x86_64 2:14.2.22-0.el8 ceph 6.3 k 2024-04-20T16:56:01.573 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-04-20T16:56:01.573 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mds x86_64 2:14.2.22-0.el8 ceph 2.0 M 2024-04-20T16:56:01.573 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr x86_64 2:14.2.22-0.el8 ceph 1.6 M 2024-04-20T16:56:01.574 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mon x86_64 2:14.2.22-0.el8 ceph 4.5 M 2024-04-20T16:56:01.574 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-osd x86_64 2:14.2.22-0.el8 ceph 17 M 2024-04-20T16:56:01.574 INFO:teuthology.orchestra.run.smithi106.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-20T16:56:01.574 INFO:teuthology.orchestra.run.smithi106.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-20T16:56:01.574 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-20T16:56:01.574 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-20T16:56:01.574 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-20T16:56:01.574 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-20T16:56:01.574 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-20T16:56:01.574 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-20T16:56:01.574 INFO:teuthology.orchestra.run.smithi106.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-20T16:56:01.574 INFO:teuthology.orchestra.run.smithi106.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-20T16:56:01.574 INFO:teuthology.orchestra.run.smithi106.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-20T16:56:01.574 INFO:teuthology.orchestra.run.smithi106.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-20T16:56:01.575 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-20T16:56:01.575 INFO:teuthology.orchestra.run.smithi106.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-20T16:56:01.575 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-20T16:56:01.575 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-20T16:56:01.575 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-20T16:56:01.575 INFO:teuthology.orchestra.run.smithi106.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-20T16:56:01.575 INFO:teuthology.orchestra.run.smithi106.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-20T16:56:01.575 INFO:teuthology.orchestra.run.smithi106.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-20T16:56:01.575 INFO:teuthology.orchestra.run.smithi106.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-20T16:56:01.575 INFO:teuthology.orchestra.run.smithi106.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-20T16:56:01.575 INFO:teuthology.orchestra.run.smithi106.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-20T16:56:01.575 INFO:teuthology.orchestra.run.smithi106.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-20T16:56:01.575 INFO:teuthology.orchestra.run.smithi106.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-20T16:56:01.575 INFO:teuthology.orchestra.run.smithi106.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-20T16:56:01.575 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:56:01.576 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-20T16:56:01.576 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:56:01.576 INFO:teuthology.orchestra.run.smithi106.stdout:Install 31 Packages 2024-04-20T16:56:01.576 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:56:01.576 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 28 M 2024-04-20T16:56:01.576 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 93 M 2024-04-20T16:56:01.576 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-20T16:56:01.596 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/31 2024-04-20T16:56:01.684 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/31 2024-04-20T16:56:01.917 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/31 2024-04-20T16:56:01.945 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/31 2024-04-20T16:56:02.090 INFO:teuthology.orchestra.run.smithi106.stdout:(1/31): ceph-14.2.22-0.el8.x86_64.rpm 101 kB/s | 6.3 kB 00:00 2024-04-20T16:56:02.140 INFO:teuthology.orchestra.run.smithi106.stdout:(2/31): ceph-mgr-14.2.22-0.el8.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-04-20T16:56:02.157 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/31 2024-04-20T16:56:02.199 INFO:teuthology.orchestra.run.smithi106.stdout:(3/31): ceph-mds-14.2.22-0.el8.x86_64.rpm 11 MB/s | 2.0 MB 00:00 2024-04-20T16:56:02.274 INFO:teuthology.orchestra.run.smithi106.stdout:(4/31): ceph-mon-14.2.22-0.el8.x86_64.rpm 24 MB/s | 4.5 MB 00:00 2024-04-20T16:56:02.392 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/31 2024-04-20T16:56:02.491 INFO:teuthology.orchestra.run.smithi106.stdout:(5/31): ceph-osd-14.2.22-0.el8.x86_64.rpm 48 MB/s | 17 MB 00:00 2024-04-20T16:56:02.520 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/31 2024-04-20T16:56:02.667 INFO:teuthology.orchestra.run.smithi106.stdout:(6/31): python3-mako-1.0.6-14.el8.noarch.rpm 336 kB/s | 157 kB 00:00 2024-04-20T16:56:02.680 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-2:14.2.22-0.el8.x86_64 27/31 2024-04-20T16:56:02.691 INFO:teuthology.orchestra.run.smithi106.stdout:(7/31): python3-markupsafe-0.23-19.el8.x86_64.r 93 kB/s | 39 kB 00:00 2024-04-20T16:56:02.775 INFO:teuthology.orchestra.run.smithi106.stdout:(8/31): python3-pytz-2017.2-11.el8.noarch.rpm 525 kB/s | 57 kB 00:00 2024-04-20T16:56:02.859 INFO:teuthology.orchestra.run.smithi106.stdout:(9/31): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.9 MB/s | 249 kB 00:00 2024-04-20T16:56:02.909 INFO:teuthology.orchestra.run.smithi106.stdout:(10/31): python3-cryptography-3.2.1-7.el8.x86_6 13 MB/s | 641 kB 00:00 2024-04-20T16:56:02.935 INFO:teuthology.orchestra.run.smithi106.stdout:(11/31): python3-pyOpenSSL-19.0.0-1.el8.noarch. 234 kB/s | 103 kB 00:00 2024-04-20T16:56:02.960 INFO:teuthology.orchestra.run.smithi106.stdout:(12/31): python3-pycparser-2.14-14.el8.noarch.r 2.1 MB/s | 109 kB 00:00 2024-04-20T16:56:02.990 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/31 2024-04-20T16:56:03.043 INFO:teuthology.orchestra.run.smithi106.stdout:(13/31): python3-werkzeug-0.12.2-4.el8.noarch.r 1.3 MB/s | 457 kB 00:00 2024-04-20T16:56:03.094 INFO:teuthology.orchestra.run.smithi106.stdout:(14/31): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 274 kB/s | 44 kB 00:00 2024-04-20T16:56:03.144 INFO:teuthology.orchestra.run.smithi106.stdout:(15/31): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.0 MB/s | 185 kB 00:00 2024-04-20T16:56:03.203 INFO:teuthology.orchestra.run.smithi106.stdout:(16/31): python3-jaraco-6.2-6.el8.noarch.rpm 191 kB/s | 11 kB 00:00 2024-04-20T16:56:03.228 INFO:teuthology.orchestra.run.smithi106.stdout:(17/31): python3-cherrypy-18.4.0-1.el8.noarch.r 2.8 MB/s | 384 kB 00:00 2024-04-20T16:56:03.253 INFO:teuthology.orchestra.run.smithi106.stdout:(18/31): python3-cheroot-8.5.2-1.el8.noarch.rpm 824 kB/s | 173 kB 00:00 2024-04-20T16:56:03.278 INFO:teuthology.orchestra.run.smithi106.stdout:(19/31): python3-jaraco-functools-2.0-4.el8.noa 232 kB/s | 18 kB 00:00 2024-04-20T16:56:03.279 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-2:14.2.22-0.el8.x86_64 27/31 2024-04-20T16:56:03.304 INFO:teuthology.orchestra.run.smithi106.stdout:(20/31): python3-logutils-0.3.5-11.el8.noarch.r 644 kB/s | 49 kB 00:00 2024-04-20T16:56:03.338 INFO:teuthology.orchestra.run.smithi106.stdout:(21/31): python3-more-itertools-7.2.0-3.el8.noa 698 kB/s | 59 kB 00:00 2024-04-20T16:56:03.363 INFO:teuthology.orchestra.run.smithi106.stdout:(22/31): python3-portend-2.6-1.el8.noarch.rpm 265 kB/s | 16 kB 00:00 2024-04-20T16:56:03.388 INFO:teuthology.orchestra.run.smithi106.stdout:(23/31): python3-pecan-1.3.2-9.el8.noarch.rpm 2.5 MB/s | 283 kB 00:00 2024-04-20T16:56:03.413 INFO:teuthology.orchestra.run.smithi106.stdout:(24/31): python3-simplegeneric-0.8.1-17.el8.noa 249 kB/s | 19 kB 00:00 2024-04-20T16:56:03.464 INFO:teuthology.orchestra.run.smithi106.stdout:(25/31): python3-singledispatch-3.4.0.3-18.el8. 238 kB/s | 24 kB 00:00 2024-04-20T16:56:03.489 INFO:teuthology.orchestra.run.smithi106.stdout:(26/31): python3-tempora-1.14.1-5.el8.noarch.rp 291 kB/s | 29 kB 00:00 2024-04-20T16:56:03.514 INFO:teuthology.orchestra.run.smithi106.stdout:(27/31): python3-trustme-0.6.0-4.el8.noarch.rpm 550 kB/s | 27 kB 00:00 2024-04-20T16:56:03.557 INFO:teuthology.orchestra.run.smithi106.stdout:(28/31): python3-waitress-1.4.3-1.el8.noarch.rp 2.7 MB/s | 255 kB 00:00 2024-04-20T16:56:03.581 INFO:teuthology.orchestra.run.smithi106.stdout:(29/31): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.7 MB/s | 251 kB 00:00 2024-04-20T16:56:03.607 INFO:teuthology.orchestra.run.smithi106.stdout:(30/31): python3-webtest-2.0.33-1.el8.noarch.rp 925 kB/s | 85 kB 00:00 2024-04-20T16:56:03.640 INFO:teuthology.orchestra.run.smithi106.stdout:(31/31): python3-zc-lockfile-2.0-2.el8.noarch.r 394 kB/s | 23 kB 00:00 2024-04-20T16:56:03.641 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:56:03.641 INFO:teuthology.orchestra.run.smithi106.stdout:Total 14 MB/s | 28 MB 00:02 2024-04-20T16:56:03.724 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-20T16:56:03.764 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-20T16:56:03.764 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-20T16:56:04.060 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-20T16:56:04.063 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-20T16:56:04.137 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-2:14.2.22-0.el8.x86_64 27/31 2024-04-20T16:56:04.306 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-osd-2:14.2.22-0.el8.x86_64 28/31 2024-04-20T16:56:04.646 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-2:14.2.22-0.el8.x86_64 27/31 2024-04-20T16:56:04.724 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-osd-2:14.2.22-0.el8.x86_64 28/31 2024-04-20T16:56:04.862 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-osd-2:14.2.22-0.el8.x86_64 28/31 2024-04-20T16:56:04.886 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mon-2:14.2.22-0.el8.x86_64 29/31 2024-04-20T16:56:05.087 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mon-2:14.2.22-0.el8.x86_64 29/31 2024-04-20T16:56:05.137 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-20T16:56:05.256 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-osd-2:14.2.22-0.el8.x86_64 28/31 2024-04-20T16:56:05.302 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mds-2:14.2.22-0.el8.x86_64 30/31 2024-04-20T16:56:05.322 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mds-2:14.2.22-0.el8.x86_64 30/31 2024-04-20T16:56:05.360 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/31 2024-04-20T16:56:05.444 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-2:14.2.22-0.el8.x86_64 31/31 2024-04-20T16:56:05.463 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mon-2:14.2.22-0.el8.x86_64 29/31 2024-04-20T16:56:05.591 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/31 2024-04-20T16:56:05.671 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mon-2:14.2.22-0.el8.x86_64 29/31 2024-04-20T16:56:05.821 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/31 2024-04-20T16:56:05.873 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-2:14.2.22-0.el8.x86_64 31/31 2024-04-20T16:56:05.874 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-2:14.2.22-0.el8.x86_64 1/31 2024-04-20T16:56:05.874 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mds-2:14.2.22-0.el8.x86_64 2/31 2024-04-20T16:56:05.874 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-2:14.2.22-0.el8.x86_64 3/31 2024-04-20T16:56:05.874 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mon-2:14.2.22-0.el8.x86_64 4/31 2024-04-20T16:56:05.874 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-osd-2:14.2.22-0.el8.x86_64 5/31 2024-04-20T16:56:05.874 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 6/31 2024-04-20T16:56:05.874 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 7/31 2024-04-20T16:56:05.874 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 8/31 2024-04-20T16:56:05.874 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 9/31 2024-04-20T16:56:05.874 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 10/31 2024-04-20T16:56:05.874 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 11/31 2024-04-20T16:56:05.874 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 12/31 2024-04-20T16:56:05.875 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 13/31 2024-04-20T16:56:05.875 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 14/31 2024-04-20T16:56:05.875 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 15/31 2024-04-20T16:56:05.875 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 16/31 2024-04-20T16:56:05.875 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 17/31 2024-04-20T16:56:05.875 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 18/31 2024-04-20T16:56:05.875 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 19/31 2024-04-20T16:56:05.875 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 20/31 2024-04-20T16:56:05.875 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 21/31 2024-04-20T16:56:05.875 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 22/31 2024-04-20T16:56:05.875 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-portend-2.6-1.el8.noarch 23/31 2024-04-20T16:56:05.875 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 24/31 2024-04-20T16:56:05.875 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 25/31 2024-04-20T16:56:05.876 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mds-2:14.2.22-0.el8.x86_64 30/31 2024-04-20T16:56:05.876 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 26/31 2024-04-20T16:56:05.877 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 27/31 2024-04-20T16:56:05.877 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 28/31 2024-04-20T16:56:05.877 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 29/31 2024-04-20T16:56:05.877 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 30/31 2024-04-20T16:56:05.889 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mds-2:14.2.22-0.el8.x86_64 30/31 2024-04-20T16:56:06.046 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-2:14.2.22-0.el8.x86_64 31/31 2024-04-20T16:56:06.075 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/31 2024-04-20T16:56:06.284 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/31 2024-04-20T16:56:06.461 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-2:14.2.22-0.el8.x86_64 31/31 2024-04-20T16:56:06.461 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-2:14.2.22-0.el8.x86_64 1/31 2024-04-20T16:56:06.461 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mds-2:14.2.22-0.el8.x86_64 2/31 2024-04-20T16:56:06.461 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-2:14.2.22-0.el8.x86_64 3/31 2024-04-20T16:56:06.461 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mon-2:14.2.22-0.el8.x86_64 4/31 2024-04-20T16:56:06.461 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-osd-2:14.2.22-0.el8.x86_64 5/31 2024-04-20T16:56:06.461 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 6/31 2024-04-20T16:56:06.461 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 7/31 2024-04-20T16:56:06.461 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 8/31 2024-04-20T16:56:06.461 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 9/31 2024-04-20T16:56:06.461 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 10/31 2024-04-20T16:56:06.462 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 11/31 2024-04-20T16:56:06.462 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 12/31 2024-04-20T16:56:06.462 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 13/31 2024-04-20T16:56:06.462 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 14/31 2024-04-20T16:56:06.462 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 15/31 2024-04-20T16:56:06.462 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 16/31 2024-04-20T16:56:06.462 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 17/31 2024-04-20T16:56:06.462 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 18/31 2024-04-20T16:56:06.462 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 19/31 2024-04-20T16:56:06.462 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 20/31 2024-04-20T16:56:06.462 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 21/31 2024-04-20T16:56:06.462 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 22/31 2024-04-20T16:56:06.462 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-portend-2.6-1.el8.noarch 23/31 2024-04-20T16:56:06.462 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 24/31 2024-04-20T16:56:06.463 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 25/31 2024-04-20T16:56:06.463 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 26/31 2024-04-20T16:56:06.463 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 27/31 2024-04-20T16:56:06.463 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 28/31 2024-04-20T16:56:06.463 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 29/31 2024-04-20T16:56:06.463 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 30/31 2024-04-20T16:56:06.464 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/31 2024-04-20T16:56:06.656 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 31/31 2024-04-20T16:56:06.657 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:56:06.657 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-04-20T16:56:06.657 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-2:14.2.22-0.el8.x86_64 2024-04-20T16:56:06.657 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mds-2:14.2.22-0.el8.x86_64 2024-04-20T16:56:06.657 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-2:14.2.22-0.el8.x86_64 2024-04-20T16:56:06.657 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mon-2:14.2.22-0.el8.x86_64 2024-04-20T16:56:06.657 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-osd-2:14.2.22-0.el8.x86_64 2024-04-20T16:56:06.657 INFO:teuthology.orchestra.run.smithi005.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-20T16:56:06.657 INFO:teuthology.orchestra.run.smithi005.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-20T16:56:06.657 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-20T16:56:06.657 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-20T16:56:06.657 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-20T16:56:06.657 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-20T16:56:06.658 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-20T16:56:06.658 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-20T16:56:06.658 INFO:teuthology.orchestra.run.smithi005.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-20T16:56:06.658 INFO:teuthology.orchestra.run.smithi005.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-20T16:56:06.658 INFO:teuthology.orchestra.run.smithi005.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-20T16:56:06.658 INFO:teuthology.orchestra.run.smithi005.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-20T16:56:06.658 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-20T16:56:06.658 INFO:teuthology.orchestra.run.smithi005.stdout: python3-portend-2.6-1.el8.noarch 2024-04-20T16:56:06.658 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-20T16:56:06.658 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-20T16:56:06.658 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-20T16:56:06.658 INFO:teuthology.orchestra.run.smithi005.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-20T16:56:06.658 INFO:teuthology.orchestra.run.smithi005.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-20T16:56:06.659 INFO:teuthology.orchestra.run.smithi005.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-20T16:56:06.659 INFO:teuthology.orchestra.run.smithi005.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-20T16:56:06.659 INFO:teuthology.orchestra.run.smithi005.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-20T16:56:06.659 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-20T16:56:06.659 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-20T16:56:06.659 INFO:teuthology.orchestra.run.smithi005.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-20T16:56:06.659 INFO:teuthology.orchestra.run.smithi005.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-20T16:56:06.659 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:56:06.659 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-04-20T16:56:06.691 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/31 2024-04-20T16:56:06.843 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/31 2024-04-20T16:56:06.971 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-base 2024-04-20T16:56:07.060 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/31 2024-04-20T16:56:07.281 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/31 2024-04-20T16:56:07.310 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 31/31 2024-04-20T16:56:07.311 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:56:07.311 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-04-20T16:56:07.311 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-2:14.2.22-0.el8.x86_64 2024-04-20T16:56:07.311 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mds-2:14.2.22-0.el8.x86_64 2024-04-20T16:56:07.311 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-2:14.2.22-0.el8.x86_64 2024-04-20T16:56:07.311 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mon-2:14.2.22-0.el8.x86_64 2024-04-20T16:56:07.311 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-osd-2:14.2.22-0.el8.x86_64 2024-04-20T16:56:07.311 INFO:teuthology.orchestra.run.smithi171.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-20T16:56:07.311 INFO:teuthology.orchestra.run.smithi171.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-20T16:56:07.311 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-20T16:56:07.311 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-20T16:56:07.311 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-20T16:56:07.312 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-20T16:56:07.312 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-20T16:56:07.312 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-20T16:56:07.312 INFO:teuthology.orchestra.run.smithi171.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-20T16:56:07.312 INFO:teuthology.orchestra.run.smithi171.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-20T16:56:07.312 INFO:teuthology.orchestra.run.smithi171.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-20T16:56:07.312 INFO:teuthology.orchestra.run.smithi171.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-20T16:56:07.312 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-20T16:56:07.312 INFO:teuthology.orchestra.run.smithi171.stdout: python3-portend-2.6-1.el8.noarch 2024-04-20T16:56:07.312 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-20T16:56:07.312 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-20T16:56:07.312 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-20T16:56:07.312 INFO:teuthology.orchestra.run.smithi171.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-20T16:56:07.312 INFO:teuthology.orchestra.run.smithi171.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-20T16:56:07.313 INFO:teuthology.orchestra.run.smithi171.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-20T16:56:07.313 INFO:teuthology.orchestra.run.smithi171.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-20T16:56:07.313 INFO:teuthology.orchestra.run.smithi171.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-20T16:56:07.313 INFO:teuthology.orchestra.run.smithi171.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-20T16:56:07.313 INFO:teuthology.orchestra.run.smithi171.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-20T16:56:07.313 INFO:teuthology.orchestra.run.smithi171.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-20T16:56:07.313 INFO:teuthology.orchestra.run.smithi171.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-20T16:56:07.313 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:56:07.313 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-04-20T16:56:07.527 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/31 2024-04-20T16:56:07.596 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-base 2024-04-20T16:56:07.728 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:10 ago on Sat 20 Apr 2024 04:54:57 PM UTC. 2024-04-20T16:56:07.784 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/31 2024-04-20T16:56:08.205 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/31 2024-04-20T16:56:08.328 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:06 ago on Sat 20 Apr 2024 04:55:02 PM UTC. 2024-04-20T16:56:08.440 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/31 2024-04-20T16:56:08.661 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/31 2024-04-20T16:56:08.938 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/31 2024-04-20T16:56:09.164 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/31 2024-04-20T16:56:09.418 INFO:teuthology.orchestra.run.smithi005.stdout:Package ceph-base-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-20T16:56:09.470 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-04-20T16:56:09.472 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-04-20T16:56:09.472 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-04-20T16:56:09.508 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/31 2024-04-20T16:56:09.598 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-mgr 2024-04-20T16:56:09.727 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/31 2024-04-20T16:56:09.897 INFO:teuthology.orchestra.run.smithi171.stdout:Package ceph-base-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-20T16:56:09.932 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/31 2024-04-20T16:56:09.945 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-04-20T16:56:09.946 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-04-20T16:56:09.946 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-04-20T16:56:10.090 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr 2024-04-20T16:56:10.109 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/31 2024-04-20T16:56:10.347 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/31 2024-04-20T16:56:10.354 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:13 ago on Sat 20 Apr 2024 04:54:57 PM UTC. 2024-04-20T16:56:10.591 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/31 2024-04-20T16:56:10.782 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/31 2024-04-20T16:56:10.812 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:08 ago on Sat 20 Apr 2024 04:55:02 PM UTC. 2024-04-20T16:56:11.093 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/31 2024-04-20T16:56:11.532 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/31 2024-04-20T16:56:11.784 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-2:14.2.22-0.el8.x86_64 27/31 2024-04-20T16:56:12.059 INFO:teuthology.orchestra.run.smithi005.stdout:Package ceph-mgr-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-20T16:56:12.112 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-04-20T16:56:12.113 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-04-20T16:56:12.113 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-04-20T16:56:12.239 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-mgr-dashboard 2024-04-20T16:56:12.445 INFO:teuthology.orchestra.run.smithi171.stdout:Package ceph-mgr-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-20T16:56:12.495 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-04-20T16:56:12.496 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-04-20T16:56:12.496 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-04-20T16:56:12.635 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr-dashboard 2024-04-20T16:56:12.994 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:15 ago on Sat 20 Apr 2024 04:54:57 PM UTC. 2024-04-20T16:56:13.153 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-2:14.2.22-0.el8.x86_64 27/31 2024-04-20T16:56:13.326 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-osd-2:14.2.22-0.el8.x86_64 28/31 2024-04-20T16:56:13.354 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:11 ago on Sat 20 Apr 2024 04:55:02 PM UTC. 2024-04-20T16:56:13.720 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-osd-2:14.2.22-0.el8.x86_64 28/31 2024-04-20T16:56:13.879 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mon-2:14.2.22-0.el8.x86_64 29/31 2024-04-20T16:56:14.065 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mon-2:14.2.22-0.el8.x86_64 29/31 2024-04-20T16:56:14.246 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mds-2:14.2.22-0.el8.x86_64 30/31 2024-04-20T16:56:14.263 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mds-2:14.2.22-0.el8.x86_64 30/31 2024-04-20T16:56:14.403 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-2:14.2.22-0.el8.x86_64 31/31 2024-04-20T16:56:14.759 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-04-20T16:56:14.760 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:56:14.760 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-04-20T16:56:14.760 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:56:14.760 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-04-20T16:56:14.760 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-dashboard noarch 2:14.2.22-0.el8 ceph-noarch 3.7 M 2024-04-20T16:56:14.761 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-04-20T16:56:14.761 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-grafana-dashboards noarch 2:14.2.22-0.el8 ceph-noarch 24 k 2024-04-20T16:56:14.761 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-04-20T16:56:14.761 INFO:teuthology.orchestra.run.smithi005.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-20T16:56:14.761 INFO:teuthology.orchestra.run.smithi005.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-20T16:56:14.761 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:56:14.761 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-04-20T16:56:14.761 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:56:14.761 INFO:teuthology.orchestra.run.smithi005.stdout:Install 5 Packages 2024-04-20T16:56:14.761 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:56:14.762 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 4.0 M 2024-04-20T16:56:14.762 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 17 M 2024-04-20T16:56:14.762 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-04-20T16:56:14.800 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-2:14.2.22-0.el8.x86_64 31/31 2024-04-20T16:56:14.800 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-2:14.2.22-0.el8.x86_64 1/31 2024-04-20T16:56:14.800 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mds-2:14.2.22-0.el8.x86_64 2/31 2024-04-20T16:56:14.800 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-2:14.2.22-0.el8.x86_64 3/31 2024-04-20T16:56:14.800 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mon-2:14.2.22-0.el8.x86_64 4/31 2024-04-20T16:56:14.800 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-osd-2:14.2.22-0.el8.x86_64 5/31 2024-04-20T16:56:14.800 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 6/31 2024-04-20T16:56:14.800 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 7/31 2024-04-20T16:56:14.800 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 8/31 2024-04-20T16:56:14.801 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 9/31 2024-04-20T16:56:14.801 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 10/31 2024-04-20T16:56:14.801 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 11/31 2024-04-20T16:56:14.801 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 12/31 2024-04-20T16:56:14.801 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 13/31 2024-04-20T16:56:14.801 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 14/31 2024-04-20T16:56:14.801 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 15/31 2024-04-20T16:56:14.801 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 16/31 2024-04-20T16:56:14.801 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 17/31 2024-04-20T16:56:14.801 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 18/31 2024-04-20T16:56:14.801 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 19/31 2024-04-20T16:56:14.801 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 20/31 2024-04-20T16:56:14.801 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 21/31 2024-04-20T16:56:14.801 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 22/31 2024-04-20T16:56:14.802 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-portend-2.6-1.el8.noarch 23/31 2024-04-20T16:56:14.802 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 24/31 2024-04-20T16:56:14.802 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 25/31 2024-04-20T16:56:14.802 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 26/31 2024-04-20T16:56:14.802 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 27/31 2024-04-20T16:56:14.802 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 28/31 2024-04-20T16:56:14.802 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 29/31 2024-04-20T16:56:14.802 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 30/31 2024-04-20T16:56:14.984 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-04-20T16:56:14.985 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:56:14.985 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-04-20T16:56:14.985 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:56:14.986 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-04-20T16:56:14.986 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-dashboard noarch 2:14.2.22-0.el8 ceph-noarch 3.7 M 2024-04-20T16:56:14.986 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-04-20T16:56:14.986 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-grafana-dashboards noarch 2:14.2.22-0.el8 ceph-noarch 24 k 2024-04-20T16:56:14.986 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-04-20T16:56:14.986 INFO:teuthology.orchestra.run.smithi171.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-20T16:56:14.986 INFO:teuthology.orchestra.run.smithi171.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-20T16:56:14.986 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:56:14.986 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-04-20T16:56:14.986 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:56:14.987 INFO:teuthology.orchestra.run.smithi171.stdout:Install 5 Packages 2024-04-20T16:56:14.987 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:56:14.987 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 4.0 M 2024-04-20T16:56:14.987 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 17 M 2024-04-20T16:56:14.987 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-04-20T16:56:15.018 INFO:teuthology.orchestra.run.smithi005.stdout:(1/5): ceph-grafana-dashboards-14.2.22-0.el8.no 449 kB/s | 24 kB 00:00 2024-04-20T16:56:15.052 INFO:teuthology.orchestra.run.smithi005.stdout:(2/5): python3-jwt-1.6.1-2.el8.noarch.rpm 490 kB/s | 43 kB 00:00 2024-04-20T16:56:15.127 INFO:teuthology.orchestra.run.smithi005.stdout:(3/5): ceph-mgr-dashboard-14.2.22-0.el8.noarch. 23 MB/s | 3.7 MB 00:00 2024-04-20T16:56:15.164 INFO:teuthology.orchestra.run.smithi171.stdout:(1/5): ceph-grafana-dashboards-14.2.22-0.el8.no 523 kB/s | 24 kB 00:00 2024-04-20T16:56:15.190 INFO:teuthology.orchestra.run.smithi171.stdout:(2/5): python3-jwt-1.6.1-2.el8.noarch.rpm 599 kB/s | 43 kB 00:00 2024-04-20T16:56:15.290 INFO:teuthology.orchestra.run.smithi171.stdout:(3/5): ceph-mgr-dashboard-14.2.22-0.el8.noarch. 22 MB/s | 3.7 MB 00:00 2024-04-20T16:56:15.416 INFO:teuthology.orchestra.run.smithi171.stdout:(4/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 134 kB/s | 34 kB 00:00 2024-04-20T16:56:15.474 INFO:teuthology.orchestra.run.smithi171.stdout:(5/5): python3-routes-2.4.1-12.el8.noarch.rpm 692 kB/s | 196 kB 00:00 2024-04-20T16:56:15.476 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:56:15.477 INFO:teuthology.orchestra.run.smithi171.stdout:Total 8.2 MB/s | 4.0 MB 00:00 2024-04-20T16:56:15.487 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-04-20T16:56:15.497 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-04-20T16:56:15.497 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-04-20T16:56:15.526 INFO:teuthology.orchestra.run.smithi005.stdout:(4/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 66 kB/s | 34 kB 00:00 2024-04-20T16:56:15.655 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 31/31 2024-04-20T16:56:15.655 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:56:15.655 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-20T16:56:15.656 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-2:14.2.22-0.el8.x86_64 2024-04-20T16:56:15.656 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mds-2:14.2.22-0.el8.x86_64 2024-04-20T16:56:15.656 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-2:14.2.22-0.el8.x86_64 2024-04-20T16:56:15.656 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mon-2:14.2.22-0.el8.x86_64 2024-04-20T16:56:15.656 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-osd-2:14.2.22-0.el8.x86_64 2024-04-20T16:56:15.656 INFO:teuthology.orchestra.run.smithi106.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-20T16:56:15.656 INFO:teuthology.orchestra.run.smithi106.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-20T16:56:15.656 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-20T16:56:15.656 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-20T16:56:15.656 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-20T16:56:15.656 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-20T16:56:15.656 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-20T16:56:15.656 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-20T16:56:15.656 INFO:teuthology.orchestra.run.smithi106.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-20T16:56:15.657 INFO:teuthology.orchestra.run.smithi106.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-20T16:56:15.657 INFO:teuthology.orchestra.run.smithi106.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-20T16:56:15.657 INFO:teuthology.orchestra.run.smithi106.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-20T16:56:15.657 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-20T16:56:15.657 INFO:teuthology.orchestra.run.smithi106.stdout: python3-portend-2.6-1.el8.noarch 2024-04-20T16:56:15.657 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-20T16:56:15.657 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-20T16:56:15.657 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-20T16:56:15.657 INFO:teuthology.orchestra.run.smithi106.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-20T16:56:15.657 INFO:teuthology.orchestra.run.smithi106.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-20T16:56:15.657 INFO:teuthology.orchestra.run.smithi106.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-20T16:56:15.657 INFO:teuthology.orchestra.run.smithi106.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-20T16:56:15.657 INFO:teuthology.orchestra.run.smithi106.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-20T16:56:15.657 INFO:teuthology.orchestra.run.smithi106.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-20T16:56:15.657 INFO:teuthology.orchestra.run.smithi106.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-20T16:56:15.658 INFO:teuthology.orchestra.run.smithi106.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-20T16:56:15.658 INFO:teuthology.orchestra.run.smithi106.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-20T16:56:15.658 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:56:15.658 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-20T16:56:15.658 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-04-20T16:56:15.662 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-04-20T16:56:15.677 INFO:teuthology.orchestra.run.smithi005.stdout:(5/5): python3-routes-2.4.1-12.el8.noarch.rpm 314 kB/s | 196 kB 00:00 2024-04-20T16:56:15.680 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:56:15.680 INFO:teuthology.orchestra.run.smithi005.stdout:Total 4.4 MB/s | 4.0 MB 00:00 2024-04-20T16:56:15.693 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-04-20T16:56:15.704 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-04-20T16:56:15.704 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-04-20T16:56:15.883 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-04-20T16:56:15.887 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-04-20T16:56:15.919 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-04-20T16:56:15.929 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-base 2024-04-20T16:56:16.117 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-04-20T16:56:16.168 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-04-20T16:56:16.368 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-04-20T16:56:16.395 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-04-20T16:56:16.635 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/5 2024-04-20T16:56:16.658 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:04 ago on Sat 20 Apr 2024 04:55:12 PM UTC. 2024-04-20T16:56:17.551 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 4/5 2024-04-20T16:56:17.648 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-04-20T16:56:17.867 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/5 2024-04-20T16:56:18.244 INFO:teuthology.orchestra.run.smithi106.stdout:Package ceph-base-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-20T16:56:18.291 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-20T16:56:18.293 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-04-20T16:56:18.293 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-20T16:56:18.407 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-mgr 2024-04-20T16:56:18.831 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 4/5 2024-04-20T16:56:19.095 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 5/5 2024-04-20T16:56:19.126 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:07 ago on Sat 20 Apr 2024 04:55:12 PM UTC. 2024-04-20T16:56:19.240 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 5/5 2024-04-20T16:56:19.405 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 5/5 2024-04-20T16:56:19.405 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 1/5 2024-04-20T16:56:19.405 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 2/5 2024-04-20T16:56:19.405 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 3/5 2024-04-20T16:56:19.405 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-04-20T16:56:19.544 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 5/5 2024-04-20T16:56:19.544 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 1/5 2024-04-20T16:56:19.544 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 2/5 2024-04-20T16:56:19.544 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 3/5 2024-04-20T16:56:19.544 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-04-20T16:56:19.841 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-04-20T16:56:19.841 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:56:19.841 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-04-20T16:56:19.841 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 2024-04-20T16:56:19.841 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 2024-04-20T16:56:19.841 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-20T16:56:19.841 INFO:teuthology.orchestra.run.smithi005.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-20T16:56:19.841 INFO:teuthology.orchestra.run.smithi005.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-20T16:56:19.841 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:56:19.841 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-04-20T16:56:19.929 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-04-20T16:56:19.929 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:56:19.929 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-04-20T16:56:19.929 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 2024-04-20T16:56:19.930 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 2024-04-20T16:56:19.930 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-20T16:56:19.930 INFO:teuthology.orchestra.run.smithi171.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-20T16:56:19.930 INFO:teuthology.orchestra.run.smithi171.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-20T16:56:19.930 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:56:19.930 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-04-20T16:56:19.976 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-20T16:56:20.067 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-20T16:56:20.701 INFO:teuthology.orchestra.run.smithi106.stdout:Package ceph-mgr-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-20T16:56:20.733 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:23 ago on Sat 20 Apr 2024 04:54:57 PM UTC. 2024-04-20T16:56:20.751 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-20T16:56:20.752 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-04-20T16:56:20.752 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-20T16:56:20.795 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:18 ago on Sat 20 Apr 2024 04:55:02 PM UTC. 2024-04-20T16:56:20.867 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-mgr-dashboard 2024-04-20T16:56:21.585 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:09 ago on Sat 20 Apr 2024 04:55:12 PM UTC. 2024-04-20T16:56:22.434 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-04-20T16:56:22.436 INFO:teuthology.orchestra.run.smithi171.stdout:============================================================================================ 2024-04-20T16:56:22.436 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-04-20T16:56:22.436 INFO:teuthology.orchestra.run.smithi171.stdout:============================================================================================ 2024-04-20T16:56:22.436 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-04-20T16:56:22.436 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-diskprediction-local noarch 2:14.2.22-0.el8 ceph-noarch 1.1 M 2024-04-20T16:56:22.436 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-04-20T16:56:22.436 INFO:teuthology.orchestra.run.smithi171.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-04-20T16:56:22.436 INFO:teuthology.orchestra.run.smithi171.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-04-20T16:56:22.436 INFO:teuthology.orchestra.run.smithi171.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-04-20T16:56:22.436 INFO:teuthology.orchestra.run.smithi171.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-04-20T16:56:22.436 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-04-20T16:56:22.437 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-04-20T16:56:22.437 INFO:teuthology.orchestra.run.smithi171.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-04-20T16:56:22.437 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:56:22.437 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-04-20T16:56:22.437 INFO:teuthology.orchestra.run.smithi171.stdout:============================================================================================ 2024-04-20T16:56:22.437 INFO:teuthology.orchestra.run.smithi171.stdout:Install 8 Packages 2024-04-20T16:56:22.437 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:56:22.437 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 30 M 2024-04-20T16:56:22.438 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 172 M 2024-04-20T16:56:22.438 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-04-20T16:56:22.490 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-04-20T16:56:22.491 INFO:teuthology.orchestra.run.smithi005.stdout:============================================================================================ 2024-04-20T16:56:22.491 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-04-20T16:56:22.491 INFO:teuthology.orchestra.run.smithi005.stdout:============================================================================================ 2024-04-20T16:56:22.491 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-04-20T16:56:22.491 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-diskprediction-local noarch 2:14.2.22-0.el8 ceph-noarch 1.1 M 2024-04-20T16:56:22.491 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-04-20T16:56:22.492 INFO:teuthology.orchestra.run.smithi005.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-04-20T16:56:22.492 INFO:teuthology.orchestra.run.smithi005.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-04-20T16:56:22.492 INFO:teuthology.orchestra.run.smithi005.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-04-20T16:56:22.492 INFO:teuthology.orchestra.run.smithi005.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-04-20T16:56:22.492 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-04-20T16:56:22.492 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-04-20T16:56:22.492 INFO:teuthology.orchestra.run.smithi005.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-04-20T16:56:22.492 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:56:22.492 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-04-20T16:56:22.492 INFO:teuthology.orchestra.run.smithi005.stdout:============================================================================================ 2024-04-20T16:56:22.492 INFO:teuthology.orchestra.run.smithi005.stdout:Install 8 Packages 2024-04-20T16:56:22.492 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:56:22.493 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 30 M 2024-04-20T16:56:22.493 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 172 M 2024-04-20T16:56:22.493 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-04-20T16:56:22.907 INFO:teuthology.orchestra.run.smithi171.stdout:(1/8): ceph-mgr-diskprediction-local-14.2.22-0. 18 MB/s | 1.1 MB 00:00 2024-04-20T16:56:22.923 INFO:teuthology.orchestra.run.smithi005.stdout:(1/8): ceph-mgr-diskprediction-local-14.2.22-0. 7.2 MB/s | 1.1 MB 00:00 2024-04-20T16:56:23.215 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-20T16:56:23.216 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:56:23.216 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-04-20T16:56:23.216 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:56:23.216 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-20T16:56:23.216 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-dashboard noarch 2:14.2.22-0.el8 ceph-noarch 3.7 M 2024-04-20T16:56:23.216 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-04-20T16:56:23.216 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-grafana-dashboards noarch 2:14.2.22-0.el8 ceph-noarch 24 k 2024-04-20T16:56:23.216 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-04-20T16:56:23.216 INFO:teuthology.orchestra.run.smithi106.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-20T16:56:23.216 INFO:teuthology.orchestra.run.smithi106.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-20T16:56:23.217 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:56:23.217 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-20T16:56:23.217 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:56:23.217 INFO:teuthology.orchestra.run.smithi106.stdout:Install 5 Packages 2024-04-20T16:56:23.217 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:56:23.217 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 4.0 M 2024-04-20T16:56:23.217 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 17 M 2024-04-20T16:56:23.218 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-20T16:56:23.489 INFO:teuthology.orchestra.run.smithi106.stdout:(1/5): ceph-grafana-dashboards-14.2.22-0.el8.no 374 kB/s | 24 kB 00:00 2024-04-20T16:56:23.514 INFO:teuthology.orchestra.run.smithi106.stdout:(2/5): python3-jwt-1.6.1-2.el8.noarch.rpm 478 kB/s | 43 kB 00:00 2024-04-20T16:56:23.589 INFO:teuthology.orchestra.run.smithi106.stdout:(3/5): ceph-mgr-dashboard-14.2.22-0.el8.noarch. 23 MB/s | 3.7 MB 00:00 2024-04-20T16:56:23.722 INFO:teuthology.orchestra.run.smithi106.stdout:(4/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 144 kB/s | 34 kB 00:00 2024-04-20T16:56:23.823 INFO:teuthology.orchestra.run.smithi106.stdout:(5/5): python3-routes-2.4.1-12.el8.noarch.rpm 636 kB/s | 196 kB 00:00 2024-04-20T16:56:23.824 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:56:23.824 INFO:teuthology.orchestra.run.smithi106.stdout:Total 6.6 MB/s | 4.0 MB 00:00 2024-04-20T16:56:23.834 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-20T16:56:23.843 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-20T16:56:23.843 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-20T16:56:23.848 INFO:teuthology.orchestra.run.smithi005.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 4.4 MB/s | 4.7 MB 00:01 2024-04-20T16:56:23.941 INFO:teuthology.orchestra.run.smithi005.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 3.7 MB/s | 3.7 MB 00:01 2024-04-20T16:56:23.950 INFO:teuthology.orchestra.run.smithi171.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 3.6 MB/s | 3.7 MB 00:01 2024-04-20T16:56:23.990 INFO:teuthology.orchestra.run.smithi005.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.6 MB/s | 225 kB 00:00 2024-04-20T16:56:23.992 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-20T16:56:23.995 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-20T16:56:24.017 INFO:teuthology.orchestra.run.smithi171.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 4.2 MB/s | 4.9 MB 00:01 2024-04-20T16:56:24.057 INFO:teuthology.orchestra.run.smithi005.stdout:(5/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 3.8 MB/s | 4.9 MB 00:01 2024-04-20T16:56:24.058 INFO:teuthology.orchestra.run.smithi171.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.0 MB/s | 225 kB 00:00 2024-04-20T16:56:24.150 INFO:teuthology.orchestra.run.smithi005.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.8 MB/s | 172 kB 00:00 2024-04-20T16:56:24.159 INFO:teuthology.orchestra.run.smithi171.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 6.3 MB/s | 645 kB 00:00 2024-04-20T16:56:24.193 INFO:teuthology.orchestra.run.smithi171.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 5.1 MB/s | 172 kB 00:00 2024-04-20T16:56:24.207 INFO:teuthology.orchestra.run.smithi005.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.9 MB/s | 645 kB 00:00 2024-04-20T16:56:24.263 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-20T16:56:24.343 INFO:teuthology.orchestra.run.smithi171.stdout:(7/8): openblas-0.3.15-4.el8.x86_64.rpm 3.2 MB/s | 4.7 MB 00:01 2024-04-20T16:56:24.475 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-04-20T16:56:24.770 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-04-20T16:56:24.925 INFO:teuthology.orchestra.run.smithi005.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 14 MB/s | 14 MB 00:00 2024-04-20T16:56:24.926 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:56:24.926 INFO:teuthology.orchestra.run.smithi005.stdout:Total 12 MB/s | 30 MB 00:02 2024-04-20T16:56:24.976 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/5 2024-04-20T16:56:25.135 INFO:teuthology.orchestra.run.smithi171.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 13 MB/s | 14 MB 00:01 2024-04-20T16:56:25.136 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:56:25.136 INFO:teuthology.orchestra.run.smithi171.stdout:Total 11 MB/s | 30 MB 00:02 2024-04-20T16:56:25.168 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-04-20T16:56:25.188 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-04-20T16:56:25.188 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-04-20T16:56:25.371 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-04-20T16:56:25.389 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-04-20T16:56:25.389 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-04-20T16:56:25.529 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-04-20T16:56:25.532 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-04-20T16:56:25.696 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-04-20T16:56:25.699 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-04-20T16:56:25.878 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 4/5 2024-04-20T16:56:26.100 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-04-20T16:56:26.148 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 5/5 2024-04-20T16:56:26.258 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-04-20T16:56:26.266 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-20T16:56:26.402 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-20T16:56:26.425 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 5/5 2024-04-20T16:56:26.425 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 1/5 2024-04-20T16:56:26.425 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 2/5 2024-04-20T16:56:26.425 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 3/5 2024-04-20T16:56:26.425 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-04-20T16:56:26.451 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-20T16:56:26.590 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-20T16:56:26.591 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-20T16:56:26.776 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-20T16:56:26.829 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-04-20T16:56:26.829 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:56:26.829 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-20T16:56:26.829 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-grafana-dashboards-2:14.2.22-0.el8.noarch 2024-04-20T16:56:26.829 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-dashboard-2:14.2.22-0.el8.noarch 2024-04-20T16:56:26.829 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-20T16:56:26.829 INFO:teuthology.orchestra.run.smithi106.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-20T16:56:26.829 INFO:teuthology.orchestra.run.smithi106.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-20T16:56:26.829 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:56:26.829 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-20T16:56:26.964 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-20T16:56:27.234 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-20T16:56:27.382 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-20T16:56:27.407 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-04-20T16:56:27.593 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-04-20T16:56:27.690 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:15 ago on Sat 20 Apr 2024 04:55:12 PM UTC. 2024-04-20T16:56:28.071 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-04-20T16:56:28.219 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-04-20T16:56:28.224 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-20T16:56:28.368 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-20T16:56:29.050 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-20T16:56:29.088 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-20T16:56:29.323 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-20T16:56:29.325 INFO:teuthology.orchestra.run.smithi106.stdout:============================================================================================ 2024-04-20T16:56:29.325 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-04-20T16:56:29.325 INFO:teuthology.orchestra.run.smithi106.stdout:============================================================================================ 2024-04-20T16:56:29.325 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-20T16:56:29.325 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-diskprediction-local noarch 2:14.2.22-0.el8 ceph-noarch 1.1 M 2024-04-20T16:56:29.325 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-04-20T16:56:29.325 INFO:teuthology.orchestra.run.smithi106.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-04-20T16:56:29.325 INFO:teuthology.orchestra.run.smithi106.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-04-20T16:56:29.325 INFO:teuthology.orchestra.run.smithi106.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-04-20T16:56:29.325 INFO:teuthology.orchestra.run.smithi106.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-04-20T16:56:29.325 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-04-20T16:56:29.326 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-04-20T16:56:29.326 INFO:teuthology.orchestra.run.smithi106.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-04-20T16:56:29.326 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:56:29.326 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-20T16:56:29.326 INFO:teuthology.orchestra.run.smithi106.stdout:============================================================================================ 2024-04-20T16:56:29.326 INFO:teuthology.orchestra.run.smithi106.stdout:Install 8 Packages 2024-04-20T16:56:29.326 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:56:29.327 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 30 M 2024-04-20T16:56:29.327 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 172 M 2024-04-20T16:56:29.327 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-20T16:56:29.413 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-20T16:56:29.416 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-20T16:56:29.634 INFO:teuthology.orchestra.run.smithi106.stdout:(1/8): ceph-mgr-diskprediction-local-14.2.22-0. 22 MB/s | 1.1 MB 00:00 2024-04-20T16:56:29.698 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-20T16:56:29.843 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-20T16:56:30.397 INFO:teuthology.orchestra.run.smithi106.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 5.8 MB/s | 4.7 MB 00:00 2024-04-20T16:56:30.459 INFO:teuthology.orchestra.run.smithi106.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.5 MB/s | 3.7 MB 00:00 2024-04-20T16:56:30.526 INFO:teuthology.orchestra.run.smithi106.stdout:(4/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 5.1 MB/s | 4.9 MB 00:00 2024-04-20T16:56:30.551 INFO:teuthology.orchestra.run.smithi106.stdout:(5/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.4 MB/s | 225 kB 00:00 2024-04-20T16:56:30.635 INFO:teuthology.orchestra.run.smithi106.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 2.0 MB/s | 172 kB 00:00 2024-04-20T16:56:30.660 INFO:teuthology.orchestra.run.smithi106.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 4.7 MB/s | 645 kB 00:00 2024-04-20T16:56:31.103 INFO:teuthology.orchestra.run.smithi106.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 22 MB/s | 14 MB 00:00 2024-04-20T16:56:31.103 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:56:31.103 INFO:teuthology.orchestra.run.smithi106.stdout:Total 17 MB/s | 30 MB 00:01 2024-04-20T16:56:31.331 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-20T16:56:31.349 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-20T16:56:31.350 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-20T16:56:31.581 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-20T16:56:31.599 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-20T16:56:31.666 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-20T16:56:31.669 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-20T16:56:32.156 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-04-20T16:56:32.197 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-04-20T16:56:32.213 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-20T16:56:32.331 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 8/8 2024-04-20T16:56:32.351 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-20T16:56:32.418 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 8/8 2024-04-20T16:56:32.509 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-20T16:56:32.689 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 8/8 2024-04-20T16:56:32.689 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 1/8 2024-04-20T16:56:32.689 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-04-20T16:56:32.689 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-04-20T16:56:32.689 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-20T16:56:32.689 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-20T16:56:32.689 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-04-20T16:56:32.689 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-04-20T16:56:32.733 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 8/8 2024-04-20T16:56:32.733 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 1/8 2024-04-20T16:56:32.733 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-04-20T16:56:32.733 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-04-20T16:56:32.734 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-20T16:56:32.734 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-20T16:56:32.734 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-04-20T16:56:32.734 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-04-20T16:56:32.818 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-20T16:56:33.165 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-04-20T16:56:33.165 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:56:33.165 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-04-20T16:56:33.165 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 2024-04-20T16:56:33.165 INFO:teuthology.orchestra.run.smithi171.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-04-20T16:56:33.166 INFO:teuthology.orchestra.run.smithi171.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-04-20T16:56:33.166 INFO:teuthology.orchestra.run.smithi171.stdout: openblas-0.3.15-4.el8.x86_64 2024-04-20T16:56:33.166 INFO:teuthology.orchestra.run.smithi171.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-04-20T16:56:33.166 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-20T16:56:33.166 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-20T16:56:33.166 INFO:teuthology.orchestra.run.smithi171.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-04-20T16:56:33.166 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:56:33.166 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-04-20T16:56:33.169 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-04-20T16:56:33.169 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:56:33.169 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-04-20T16:56:33.169 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 2024-04-20T16:56:33.169 INFO:teuthology.orchestra.run.smithi005.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-04-20T16:56:33.169 INFO:teuthology.orchestra.run.smithi005.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-04-20T16:56:33.169 INFO:teuthology.orchestra.run.smithi005.stdout: openblas-0.3.15-4.el8.x86_64 2024-04-20T16:56:33.169 INFO:teuthology.orchestra.run.smithi005.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-04-20T16:56:33.169 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-20T16:56:33.169 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-20T16:56:33.169 INFO:teuthology.orchestra.run.smithi005.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-04-20T16:56:33.169 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:56:33.169 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-04-20T16:56:33.308 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr-rook 2024-04-20T16:56:33.321 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-mgr-rook 2024-04-20T16:56:33.418 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-20T16:56:33.559 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-04-20T16:56:34.042 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:32 ago on Sat 20 Apr 2024 04:55:02 PM UTC. 2024-04-20T16:56:34.082 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:37 ago on Sat 20 Apr 2024 04:54:57 PM UTC. 2024-04-20T16:56:34.183 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-04-20T16:56:34.351 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-20T16:56:35.035 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-20T16:56:35.378 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-20T16:56:35.651 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-20T16:56:35.658 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-04-20T16:56:35.659 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:56:35.660 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-04-20T16:56:35.660 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:56:35.660 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-04-20T16:56:35.660 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-rook noarch 2:14.2.22-0.el8 ceph-noarch 15 k 2024-04-20T16:56:35.660 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-04-20T16:56:35.660 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-20T16:56:35.660 INFO:teuthology.orchestra.run.smithi171.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-20T16:56:35.660 INFO:teuthology.orchestra.run.smithi171.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-20T16:56:35.660 INFO:teuthology.orchestra.run.smithi171.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-20T16:56:35.660 INFO:teuthology.orchestra.run.smithi171.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-04-20T16:56:35.660 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-04-20T16:56:35.660 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-04-20T16:56:35.660 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-04-20T16:56:35.660 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-20T16:56:35.661 INFO:teuthology.orchestra.run.smithi171.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-20T16:56:35.661 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:56:35.661 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-04-20T16:56:35.661 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:56:35.661 INFO:teuthology.orchestra.run.smithi171.stdout:Install 11 Packages 2024-04-20T16:56:35.661 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:56:35.661 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 1.9 M 2024-04-20T16:56:35.662 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 24 M 2024-04-20T16:56:35.662 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-04-20T16:56:35.855 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-04-20T16:56:35.857 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:56:35.857 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-04-20T16:56:35.857 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:56:35.857 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-04-20T16:56:35.857 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-rook noarch 2:14.2.22-0.el8 ceph-noarch 15 k 2024-04-20T16:56:35.857 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-04-20T16:56:35.857 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-20T16:56:35.857 INFO:teuthology.orchestra.run.smithi005.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-20T16:56:35.857 INFO:teuthology.orchestra.run.smithi005.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-20T16:56:35.857 INFO:teuthology.orchestra.run.smithi005.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-20T16:56:35.857 INFO:teuthology.orchestra.run.smithi005.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-04-20T16:56:35.857 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-04-20T16:56:35.858 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-04-20T16:56:35.858 INFO:teuthology.orchestra.run.smithi005.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-04-20T16:56:35.858 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-20T16:56:35.858 INFO:teuthology.orchestra.run.smithi005.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-20T16:56:35.858 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:56:35.858 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-04-20T16:56:35.858 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:56:35.858 INFO:teuthology.orchestra.run.smithi005.stdout:Install 11 Packages 2024-04-20T16:56:35.858 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:56:35.859 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 1.9 M 2024-04-20T16:56:35.859 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 24 M 2024-04-20T16:56:35.859 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-04-20T16:56:36.068 INFO:teuthology.orchestra.run.smithi171.stdout:(1/11): ceph-mgr-rook-14.2.22-0.el8.noarch.rpm 215 kB/s | 15 kB 00:00 2024-04-20T16:56:36.160 INFO:teuthology.orchestra.run.smithi171.stdout:(2/11): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.7 MB/s | 155 kB 00:00 2024-04-20T16:56:36.210 INFO:teuthology.orchestra.run.smithi171.stdout:(3/11): python3-requests-oauthlib-1.0.0-1.el8.n 852 kB/s | 43 kB 00:00 2024-04-20T16:56:36.444 INFO:teuthology.orchestra.run.smithi171.stdout:(4/11): python3-cachetools-3.1.1-4.el8.noarch.r 141 kB/s | 33 kB 00:00 2024-04-20T16:56:36.492 INFO:teuthology.orchestra.run.smithi005.stdout:(1/11): ceph-mgr-rook-14.2.22-0.el8.noarch.rpm 224 kB/s | 15 kB 00:00 2024-04-20T16:56:36.511 INFO:teuthology.orchestra.run.smithi171.stdout:(5/11): python3-pyasn1-modules-0.3.7-6.el8.noar 214 kB/s | 110 kB 00:00 2024-04-20T16:56:36.528 INFO:teuthology.orchestra.run.smithi171.stdout:(6/11): python3-pyasn1-0.3.7-6.el8.noarch.rpm 238 kB/s | 126 kB 00:00 2024-04-20T16:56:36.554 INFO:teuthology.orchestra.run.smithi171.stdout:(7/11): python3-certifi-2018.10.15-7.el8.noarch 149 kB/s | 16 kB 00:00 2024-04-20T16:56:36.568 INFO:teuthology.orchestra.run.smithi005.stdout:(2/11): python3-oauthlib-2.1.0-1.el8.noarch.rpm 2.0 MB/s | 155 kB 00:00 2024-04-20T16:56:36.601 INFO:teuthology.orchestra.run.smithi005.stdout:(3/11): python3-requests-oauthlib-1.0.0-1.el8.n 1.3 MB/s | 43 kB 00:00 2024-04-20T16:56:36.721 INFO:teuthology.orchestra.run.smithi171.stdout:(8/11): python3-rsa-4.9-2.el8.noarch.rpm 385 kB/s | 64 kB 00:00 2024-04-20T16:56:36.810 INFO:teuthology.orchestra.run.smithi005.stdout:(4/11): python3-pyasn1-modules-0.3.7-6.el8.noar 285 kB/s | 110 kB 00:00 2024-04-20T16:56:36.821 INFO:teuthology.orchestra.run.smithi171.stdout:(9/11): python3-websocket-client-0.56.0-5.el8.n 613 kB/s | 61 kB 00:00 2024-04-20T16:56:36.835 INFO:teuthology.orchestra.run.smithi005.stdout:(5/11): python3-pyasn1-0.3.7-6.el8.noarch.rpm 307 kB/s | 126 kB 00:00 2024-04-20T16:56:36.846 INFO:teuthology.orchestra.run.smithi171.stdout:(10/11): python3-google-auth-1.1.1-10.el8.noarc 243 kB/s | 81 kB 00:00 2024-04-20T16:56:36.994 INFO:teuthology.orchestra.run.smithi005.stdout:(6/11): python3-cachetools-3.1.1-4.el8.noarch.r 84 kB/s | 33 kB 00:00 2024-04-20T16:56:37.130 INFO:teuthology.orchestra.run.smithi171.stdout:(11/11): python3-kubernetes-11.0.0-6.el8.noarch 1.9 MB/s | 1.2 MB 00:00 2024-04-20T16:56:37.131 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:56:37.131 INFO:teuthology.orchestra.run.smithi171.stdout:Total 1.3 MB/s | 1.9 MB 00:01 2024-04-20T16:56:37.173 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-04-20T16:56:37.186 INFO:teuthology.orchestra.run.smithi005.stdout:(7/11): python3-certifi-2018.10.15-7.el8.noarch 43 kB/s | 16 kB 00:00 2024-04-20T16:56:37.188 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-04-20T16:56:37.188 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-04-20T16:56:37.264 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-04-20T16:56:37.268 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-04-20T16:56:37.336 INFO:teuthology.orchestra.run.smithi005.stdout:(8/11): python3-google-auth-1.1.1-10.el8.noarch 162 kB/s | 81 kB 00:00 2024-04-20T16:56:37.404 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-20T16:56:37.420 INFO:teuthology.orchestra.run.smithi005.stdout:(9/11): python3-rsa-4.9-2.el8.noarch.rpm 275 kB/s | 64 kB 00:00 2024-04-20T16:56:37.495 INFO:teuthology.orchestra.run.smithi005.stdout:(10/11): python3-websocket-client-0.56.0-5.el8. 387 kB/s | 61 kB 00:00 2024-04-20T16:56:37.545 INFO:teuthology.orchestra.run.smithi005.stdout:(11/11): python3-kubernetes-11.0.0-6.el8.noarch 2.1 MB/s | 1.2 MB 00:00 2024-04-20T16:56:37.548 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:56:37.548 INFO:teuthology.orchestra.run.smithi005.stdout:Total 1.1 MB/s | 1.9 MB 00:01 2024-04-20T16:56:37.591 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-04-20T16:56:37.608 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-04-20T16:56:37.608 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-04-20T16:56:37.690 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-04-20T16:56:37.693 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-04-20T16:56:37.843 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-04-20T16:56:37.977 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-04-20T16:56:38.120 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/11 2024-04-20T16:56:38.238 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-04-20T16:56:38.260 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 8/8 2024-04-20T16:56:38.316 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/11 2024-04-20T16:56:38.469 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/11 2024-04-20T16:56:38.585 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/11 2024-04-20T16:56:38.646 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 8/8 2024-04-20T16:56:38.646 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 1/8 2024-04-20T16:56:38.646 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-04-20T16:56:38.646 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-04-20T16:56:38.647 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-20T16:56:38.647 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-20T16:56:38.647 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-04-20T16:56:38.647 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-04-20T16:56:38.657 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/11 2024-04-20T16:56:38.822 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/11 2024-04-20T16:56:38.901 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/11 2024-04-20T16:56:39.049 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-04-20T16:56:39.050 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:56:39.050 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-20T16:56:39.050 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-diskprediction-local-2:14.2.22-0.el8.noarch 2024-04-20T16:56:39.050 INFO:teuthology.orchestra.run.smithi106.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-04-20T16:56:39.050 INFO:teuthology.orchestra.run.smithi106.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-04-20T16:56:39.050 INFO:teuthology.orchestra.run.smithi106.stdout: openblas-0.3.15-4.el8.x86_64 2024-04-20T16:56:39.050 INFO:teuthology.orchestra.run.smithi106.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-04-20T16:56:39.050 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-20T16:56:39.050 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-20T16:56:39.050 INFO:teuthology.orchestra.run.smithi106.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-04-20T16:56:39.050 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:56:39.050 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-20T16:56:39.079 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/11 2024-04-20T16:56:39.189 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-mgr-rook 2024-04-20T16:56:39.286 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/11 2024-04-20T16:56:39.594 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/11 2024-04-20T16:56:39.749 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/11 2024-04-20T16:56:39.887 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/11 2024-04-20T16:56:39.917 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:27 ago on Sat 20 Apr 2024 04:55:12 PM UTC. 2024-04-20T16:56:39.989 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/11 2024-04-20T16:56:40.215 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/11 2024-04-20T16:56:40.518 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/11 2024-04-20T16:56:40.755 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/11 2024-04-20T16:56:40.982 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/11 2024-04-20T16:56:41.410 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/11 2024-04-20T16:56:41.554 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-20T16:56:41.556 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:56:41.556 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-04-20T16:56:41.556 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:56:41.556 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-20T16:56:41.556 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-rook noarch 2:14.2.22-0.el8 ceph-noarch 15 k 2024-04-20T16:56:41.556 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-04-20T16:56:41.556 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-20T16:56:41.556 INFO:teuthology.orchestra.run.smithi106.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-20T16:56:41.556 INFO:teuthology.orchestra.run.smithi106.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-20T16:56:41.556 INFO:teuthology.orchestra.run.smithi106.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-20T16:56:41.556 INFO:teuthology.orchestra.run.smithi106.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-04-20T16:56:41.556 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-04-20T16:56:41.556 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-04-20T16:56:41.557 INFO:teuthology.orchestra.run.smithi106.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-04-20T16:56:41.557 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-20T16:56:41.557 INFO:teuthology.orchestra.run.smithi106.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-20T16:56:41.557 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:56:41.557 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-20T16:56:41.557 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:56:41.557 INFO:teuthology.orchestra.run.smithi106.stdout:Install 11 Packages 2024-04-20T16:56:41.557 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:56:41.558 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 1.9 M 2024-04-20T16:56:41.558 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 24 M 2024-04-20T16:56:41.558 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-20T16:56:41.565 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-rook-2:14.2.22-0.el8.noarch 11/11 2024-04-20T16:56:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-rook-2:14.2.22-0.el8.noarch 11/11 2024-04-20T16:56:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-rook-2:14.2.22-0.el8.noarch 1/11 2024-04-20T16:56:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 2/11 2024-04-20T16:56:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/11 2024-04-20T16:56:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 4/11 2024-04-20T16:56:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/11 2024-04-20T16:56:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 6/11 2024-04-20T16:56:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 7/11 2024-04-20T16:56:41.858 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 8/11 2024-04-20T16:56:41.858 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 9/11 2024-04-20T16:56:41.858 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 10/11 2024-04-20T16:56:41.887 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/11 2024-04-20T16:56:42.133 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/11 2024-04-20T16:56:42.192 INFO:teuthology.orchestra.run.smithi106.stdout:(1/11): ceph-mgr-rook-14.2.22-0.el8.noarch.rpm 341 kB/s | 15 kB 00:00 2024-04-20T16:56:42.259 INFO:teuthology.orchestra.run.smithi106.stdout:(2/11): python3-oauthlib-2.1.0-1.el8.noarch.rpm 2.3 MB/s | 155 kB 00:00 2024-04-20T16:56:42.293 INFO:teuthology.orchestra.run.smithi106.stdout:(3/11): python3-requests-oauthlib-1.0.0-1.el8.n 1.3 MB/s | 43 kB 00:00 2024-04-20T16:56:42.300 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 11/11 2024-04-20T16:56:42.300 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:56:42.300 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-04-20T16:56:42.300 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-rook-2:14.2.22-0.el8.noarch 2024-04-20T16:56:42.300 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-20T16:56:42.300 INFO:teuthology.orchestra.run.smithi171.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-20T16:56:42.301 INFO:teuthology.orchestra.run.smithi171.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-20T16:56:42.301 INFO:teuthology.orchestra.run.smithi171.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-20T16:56:42.301 INFO:teuthology.orchestra.run.smithi171.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-20T16:56:42.301 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-20T16:56:42.301 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-20T16:56:42.301 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-20T16:56:42.301 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-20T16:56:42.301 INFO:teuthology.orchestra.run.smithi171.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-20T16:56:42.301 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:56:42.301 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-04-20T16:56:42.313 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-rook-2:14.2.22-0.el8.noarch 11/11 2024-04-20T16:56:42.453 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-fuse 2024-04-20T16:56:42.527 INFO:teuthology.orchestra.run.smithi106.stdout:(4/11): python3-cachetools-3.1.1-4.el8.noarch.r 141 kB/s | 33 kB 00:00 2024-04-20T16:56:42.552 INFO:teuthology.orchestra.run.smithi106.stdout:(5/11): python3-pyasn1-modules-0.3.7-6.el8.noar 272 kB/s | 110 kB 00:00 2024-04-20T16:56:42.577 INFO:teuthology.orchestra.run.smithi106.stdout:(6/11): python3-pyasn1-0.3.7-6.el8.noarch.rpm 294 kB/s | 126 kB 00:00 2024-04-20T16:56:42.609 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-rook-2:14.2.22-0.el8.noarch 11/11 2024-04-20T16:56:42.609 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-rook-2:14.2.22-0.el8.noarch 1/11 2024-04-20T16:56:42.609 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 2/11 2024-04-20T16:56:42.609 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/11 2024-04-20T16:56:42.609 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 4/11 2024-04-20T16:56:42.609 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/11 2024-04-20T16:56:42.609 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 6/11 2024-04-20T16:56:42.609 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 7/11 2024-04-20T16:56:42.609 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 8/11 2024-04-20T16:56:42.609 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 9/11 2024-04-20T16:56:42.610 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 10/11 2024-04-20T16:56:42.623 INFO:teuthology.orchestra.run.smithi106.stdout:(7/11): python3-certifi-2018.10.15-7.el8.noarch 170 kB/s | 16 kB 00:00 2024-04-20T16:56:42.720 INFO:teuthology.orchestra.run.smithi106.stdout:(8/11): python3-rsa-4.9-2.el8.noarch.rpm 735 kB/s | 64 kB 00:00 2024-04-20T16:56:42.778 INFO:teuthology.orchestra.run.smithi106.stdout:(9/11): python3-websocket-client-0.56.0-5.el8.n 1.0 MB/s | 61 kB 00:00 2024-04-20T16:56:42.812 INFO:teuthology.orchestra.run.smithi106.stdout:(10/11): python3-google-auth-1.1.1-10.el8.noarc 313 kB/s | 81 kB 00:00 2024-04-20T16:56:42.937 INFO:teuthology.orchestra.run.smithi106.stdout:(11/11): python3-kubernetes-11.0.0-6.el8.noarch 3.2 MB/s | 1.2 MB 00:00 2024-04-20T16:56:42.940 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:56:42.940 INFO:teuthology.orchestra.run.smithi106.stdout:Total 1.3 MB/s | 1.9 MB 00:01 2024-04-20T16:56:42.982 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-20T16:56:42.999 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-20T16:56:42.999 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-20T16:56:43.069 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 11/11 2024-04-20T16:56:43.069 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:56:43.069 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-04-20T16:56:43.069 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-rook-2:14.2.22-0.el8.noarch 2024-04-20T16:56:43.069 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-20T16:56:43.069 INFO:teuthology.orchestra.run.smithi005.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-20T16:56:43.069 INFO:teuthology.orchestra.run.smithi005.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-20T16:56:43.069 INFO:teuthology.orchestra.run.smithi005.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-20T16:56:43.070 INFO:teuthology.orchestra.run.smithi005.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-20T16:56:43.070 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-20T16:56:43.070 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-20T16:56:43.070 INFO:teuthology.orchestra.run.smithi005.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-20T16:56:43.070 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-20T16:56:43.070 INFO:teuthology.orchestra.run.smithi005.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-20T16:56:43.070 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:56:43.070 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-04-20T16:56:43.080 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-20T16:56:43.083 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-20T16:56:43.179 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:41 ago on Sat 20 Apr 2024 04:55:02 PM UTC. 2024-04-20T16:56:43.217 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-fuse 2024-04-20T16:56:43.982 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:46 ago on Sat 20 Apr 2024 04:54:57 PM UTC. 2024-04-20T16:56:44.817 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-04-20T16:56:44.818 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:56:44.819 INFO:teuthology.orchestra.run.smithi171.stdout: Package Architecture Version Repository Size 2024-04-20T16:56:44.819 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:56:44.819 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-04-20T16:56:44.819 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-fuse x86_64 2:14.2.22-0.el8 ceph 590 k 2024-04-20T16:56:44.819 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-04-20T16:56:44.819 INFO:teuthology.orchestra.run.smithi171.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-20T16:56:44.819 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:56:44.819 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-04-20T16:56:44.819 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:56:44.819 INFO:teuthology.orchestra.run.smithi171.stdout:Install 2 Packages 2024-04-20T16:56:44.819 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:56:44.820 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 673 k 2024-04-20T16:56:44.820 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 1.9 M 2024-04-20T16:56:44.820 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-04-20T16:56:44.856 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-20T16:56:44.931 INFO:teuthology.orchestra.run.smithi171.stdout:(1/2): ceph-fuse-14.2.22-0.el8.x86_64.rpm 11 MB/s | 590 kB 00:00 2024-04-20T16:56:44.998 INFO:teuthology.orchestra.run.smithi171.stdout:(2/2): fuse-2.9.7-19.el8.x86_64.rpm 701 kB/s | 83 kB 00:00 2024-04-20T16:56:44.998 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:56:44.998 INFO:teuthology.orchestra.run.smithi171.stdout:Total 3.7 MB/s | 673 kB 00:00 2024-04-20T16:56:45.009 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-04-20T16:56:45.031 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-04-20T16:56:45.031 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-04-20T16:56:45.077 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/11 2024-04-20T16:56:45.117 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-04-20T16:56:45.119 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-04-20T16:56:45.291 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/11 2024-04-20T16:56:45.320 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-04-20T16:56:45.526 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/11 2024-04-20T16:56:45.563 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-04-20T16:56:45.723 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/11 2024-04-20T16:56:45.723 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-04-20T16:56:45.724 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:56:45.724 INFO:teuthology.orchestra.run.smithi005.stdout: Package Architecture Version Repository Size 2024-04-20T16:56:45.725 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:56:45.725 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-04-20T16:56:45.725 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-fuse x86_64 2:14.2.22-0.el8 ceph 590 k 2024-04-20T16:56:45.725 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-04-20T16:56:45.725 INFO:teuthology.orchestra.run.smithi005.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-20T16:56:45.725 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:56:45.725 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-04-20T16:56:45.725 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:56:45.725 INFO:teuthology.orchestra.run.smithi005.stdout:Install 2 Packages 2024-04-20T16:56:45.725 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:56:45.726 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 673 k 2024-04-20T16:56:45.726 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 1.9 M 2024-04-20T16:56:45.726 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-04-20T16:56:45.781 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-fuse-2:14.2.22-0.el8.x86_64 2/2 2024-04-20T16:56:45.964 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/11 2024-04-20T16:56:45.969 INFO:teuthology.orchestra.run.smithi005.stdout:(1/2): ceph-fuse-14.2.22-0.el8.x86_64.rpm 11 MB/s | 590 kB 00:00 2024-04-20T16:56:45.994 INFO:teuthology.orchestra.run.smithi005.stdout:(2/2): fuse-2.9.7-19.el8.x86_64.rpm 1.1 MB/s | 83 kB 00:00 2024-04-20T16:56:45.995 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:56:45.995 INFO:teuthology.orchestra.run.smithi005.stdout:Total 2.4 MB/s | 673 kB 00:00 2024-04-20T16:56:46.005 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-04-20T16:56:46.029 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-04-20T16:56:46.029 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-04-20T16:56:46.120 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-04-20T16:56:46.123 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-04-20T16:56:46.157 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-fuse-2:14.2.22-0.el8.x86_64 2/2 2024-04-20T16:56:46.157 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-fuse-2:14.2.22-0.el8.x86_64 1/2 2024-04-20T16:56:46.177 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/11 2024-04-20T16:56:46.330 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-04-20T16:56:46.443 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/11 2024-04-20T16:56:46.580 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-04-20T16:56:46.620 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-04-20T16:56:46.620 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:56:46.620 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-04-20T16:56:46.620 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-fuse-2:14.2.22-0.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-04-20T16:56:46.620 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:56:46.620 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-04-20T16:56:46.686 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/11 2024-04-20T16:56:46.805 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-fuse-2:14.2.22-0.el8.x86_64 2/2 2024-04-20T16:56:46.832 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install librados-devel 2024-04-20T16:56:47.204 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-fuse-2:14.2.22-0.el8.x86_64 2/2 2024-04-20T16:56:47.205 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-fuse-2:14.2.22-0.el8.x86_64 1/2 2024-04-20T16:56:47.572 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:45 ago on Sat 20 Apr 2024 04:55:02 PM UTC. 2024-04-20T16:56:47.601 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-04-20T16:56:47.601 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:56:47.601 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-04-20T16:56:47.601 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-fuse-2:14.2.22-0.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-04-20T16:56:47.601 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:56:47.601 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-04-20T16:56:47.735 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/11 2024-04-20T16:56:47.825 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install librados-devel 2024-04-20T16:56:47.985 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/11 2024-04-20T16:56:48.132 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-rook-2:14.2.22-0.el8.noarch 11/11 2024-04-20T16:56:48.423 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-rook-2:14.2.22-0.el8.noarch 11/11 2024-04-20T16:56:48.423 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-rook-2:14.2.22-0.el8.noarch 1/11 2024-04-20T16:56:48.423 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 2/11 2024-04-20T16:56:48.423 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/11 2024-04-20T16:56:48.423 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 4/11 2024-04-20T16:56:48.424 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/11 2024-04-20T16:56:48.424 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 6/11 2024-04-20T16:56:48.424 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 7/11 2024-04-20T16:56:48.424 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 8/11 2024-04-20T16:56:48.424 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 9/11 2024-04-20T16:56:48.424 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 10/11 2024-04-20T16:56:48.586 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:51 ago on Sat 20 Apr 2024 04:54:57 PM UTC. 2024-04-20T16:56:48.874 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 11/11 2024-04-20T16:56:48.874 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:56:48.875 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-20T16:56:48.875 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-rook-2:14.2.22-0.el8.noarch 2024-04-20T16:56:48.875 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-20T16:56:48.875 INFO:teuthology.orchestra.run.smithi106.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-20T16:56:48.875 INFO:teuthology.orchestra.run.smithi106.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-20T16:56:48.875 INFO:teuthology.orchestra.run.smithi106.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-20T16:56:48.875 INFO:teuthology.orchestra.run.smithi106.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-20T16:56:48.875 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-20T16:56:48.875 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-20T16:56:48.875 INFO:teuthology.orchestra.run.smithi106.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-20T16:56:48.875 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-20T16:56:48.876 INFO:teuthology.orchestra.run.smithi106.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-20T16:56:48.876 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:56:48.876 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-20T16:56:48.992 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-fuse 2024-04-20T16:56:49.205 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-04-20T16:56:49.205 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:56:49.205 INFO:teuthology.orchestra.run.smithi171.stdout: Package Architecture Version Repository Size 2024-04-20T16:56:49.205 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:56:49.205 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-04-20T16:56:49.205 INFO:teuthology.orchestra.run.smithi171.stdout: librados-devel x86_64 2:14.2.22-0.el8 ceph 90 k 2024-04-20T16:56:49.205 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:56:49.205 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-04-20T16:56:49.206 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:56:49.206 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-04-20T16:56:49.206 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:56:49.206 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 90 k 2024-04-20T16:56:49.206 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 310 k 2024-04-20T16:56:49.206 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-04-20T16:56:49.281 INFO:teuthology.orchestra.run.smithi171.stdout:librados-devel-14.2.22-0.el8.x86_64.rpm 1.2 MB/s | 90 kB 00:00 2024-04-20T16:56:49.281 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:56:49.282 INFO:teuthology.orchestra.run.smithi171.stdout:Total 1.2 MB/s | 90 kB 00:00 2024-04-20T16:56:49.282 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-04-20T16:56:49.293 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-04-20T16:56:49.293 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-04-20T16:56:49.342 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-04-20T16:56:49.345 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-04-20T16:56:49.460 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-04-20T16:56:49.688 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:56:49.724 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:37 ago on Sat 20 Apr 2024 04:55:12 PM UTC. 2024-04-20T16:56:49.979 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:56:50.341 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-04-20T16:56:50.341 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:56:50.341 INFO:teuthology.orchestra.run.smithi005.stdout: Package Architecture Version Repository Size 2024-04-20T16:56:50.342 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:56:50.342 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-04-20T16:56:50.342 INFO:teuthology.orchestra.run.smithi005.stdout: librados-devel x86_64 2:14.2.22-0.el8 ceph 90 k 2024-04-20T16:56:50.342 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:56:50.342 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-04-20T16:56:50.342 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:56:50.342 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-04-20T16:56:50.342 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:56:50.342 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 90 k 2024-04-20T16:56:50.342 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 310 k 2024-04-20T16:56:50.342 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-04-20T16:56:50.394 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:56:50.394 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:56:50.394 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-04-20T16:56:50.394 INFO:teuthology.orchestra.run.smithi171.stdout: librados-devel-2:14.2.22-0.el8.x86_64 2024-04-20T16:56:50.394 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:56:50.394 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-04-20T16:56:50.514 INFO:teuthology.orchestra.run.smithi005.stdout:librados-devel-14.2.22-0.el8.x86_64.rpm 524 kB/s | 90 kB 00:00 2024-04-20T16:56:50.515 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:56:50.515 INFO:teuthology.orchestra.run.smithi005.stdout:Total 521 kB/s | 90 kB 00:00 2024-04-20T16:56:50.515 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-04-20T16:56:50.526 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-04-20T16:56:50.527 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-04-20T16:56:50.584 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-04-20T16:56:50.584 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-04-20T16:56:50.698 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install libcephfs2 2024-04-20T16:56:50.700 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-04-20T16:56:50.886 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:56:51.179 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:56:51.367 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-20T16:56:51.368 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:56:51.368 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-04-20T16:56:51.368 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:56:51.368 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-20T16:56:51.368 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-fuse x86_64 2:14.2.22-0.el8 ceph 590 k 2024-04-20T16:56:51.368 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-04-20T16:56:51.368 INFO:teuthology.orchestra.run.smithi106.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-20T16:56:51.368 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:56:51.368 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-20T16:56:51.368 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:56:51.368 INFO:teuthology.orchestra.run.smithi106.stdout:Install 2 Packages 2024-04-20T16:56:51.369 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:56:51.369 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 673 k 2024-04-20T16:56:51.369 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 1.9 M 2024-04-20T16:56:51.369 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-20T16:56:51.442 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:49 ago on Sat 20 Apr 2024 04:55:02 PM UTC. 2024-04-20T16:56:51.478 INFO:teuthology.orchestra.run.smithi106.stdout:(1/2): ceph-fuse-14.2.22-0.el8.x86_64.rpm 12 MB/s | 590 kB 00:00 2024-04-20T16:56:51.503 INFO:teuthology.orchestra.run.smithi106.stdout:(2/2): fuse-2.9.7-19.el8.x86_64.rpm 1.1 MB/s | 83 kB 00:00 2024-04-20T16:56:51.503 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:56:51.504 INFO:teuthology.orchestra.run.smithi106.stdout:Total 4.9 MB/s | 673 kB 00:00 2024-04-20T16:56:51.513 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-20T16:56:51.534 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-20T16:56:51.535 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-20T16:56:51.591 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:56:51.591 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:56:51.591 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-04-20T16:56:51.591 INFO:teuthology.orchestra.run.smithi005.stdout: librados-devel-2:14.2.22-0.el8.x86_64 2024-04-20T16:56:51.591 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:56:51.591 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-04-20T16:56:51.619 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-20T16:56:51.622 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-20T16:56:51.806 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-20T16:56:51.827 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install libcephfs2 2024-04-20T16:56:52.030 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-04-20T16:56:52.257 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-fuse-2:14.2.22-0.el8.x86_64 2/2 2024-04-20T16:56:52.591 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:55 ago on Sat 20 Apr 2024 04:54:57 PM UTC. 2024-04-20T16:56:52.638 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-fuse-2:14.2.22-0.el8.x86_64 2/2 2024-04-20T16:56:52.639 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-fuse-2:14.2.22-0.el8.x86_64 1/2 2024-04-20T16:56:52.992 INFO:teuthology.orchestra.run.smithi171.stdout:Package libcephfs2-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-20T16:56:53.036 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-04-20T16:56:53.036 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:56:53.036 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-20T16:56:53.036 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-fuse-2:14.2.22-0.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-04-20T16:56:53.036 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:56:53.036 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-20T16:56:53.040 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-04-20T16:56:53.042 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-04-20T16:56:53.042 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-04-20T16:56:53.164 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install libcephfs-devel 2024-04-20T16:56:53.244 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install librados-devel 2024-04-20T16:56:53.892 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:51 ago on Sat 20 Apr 2024 04:55:02 PM UTC. 2024-04-20T16:56:53.974 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:41 ago on Sat 20 Apr 2024 04:55:12 PM UTC. 2024-04-20T16:56:54.287 INFO:teuthology.orchestra.run.smithi005.stdout:Package libcephfs2-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-20T16:56:54.338 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-04-20T16:56:54.340 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-04-20T16:56:54.340 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-04-20T16:56:54.446 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install libcephfs-devel 2024-04-20T16:56:55.203 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:58 ago on Sat 20 Apr 2024 04:54:57 PM UTC. 2024-04-20T16:56:55.531 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-04-20T16:56:55.531 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:56:55.531 INFO:teuthology.orchestra.run.smithi171.stdout: Package Architecture Version Repository Size 2024-04-20T16:56:55.531 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:56:55.532 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-04-20T16:56:55.532 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs-devel x86_64 2:14.2.22-0.el8 ceph 22 k 2024-04-20T16:56:55.532 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:56:55.532 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-04-20T16:56:55.532 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:56:55.532 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-04-20T16:56:55.532 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:56:55.532 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 22 k 2024-04-20T16:56:55.533 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 77 k 2024-04-20T16:56:55.533 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-04-20T16:56:55.589 INFO:teuthology.orchestra.run.smithi171.stdout:libcephfs-devel-14.2.22-0.el8.x86_64.rpm 382 kB/s | 22 kB 00:00 2024-04-20T16:56:55.589 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:56:55.590 INFO:teuthology.orchestra.run.smithi171.stdout:Total 373 kB/s | 22 kB 00:00 2024-04-20T16:56:55.590 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-04-20T16:56:55.594 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-04-20T16:56:55.594 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-04-20T16:56:55.596 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-04-20T16:56:55.599 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-04-20T16:56:55.599 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-20T16:56:55.600 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:56:55.600 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-04-20T16:56:55.600 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:56:55.600 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-20T16:56:55.600 INFO:teuthology.orchestra.run.smithi106.stdout: librados-devel x86_64 2:14.2.22-0.el8 ceph 90 k 2024-04-20T16:56:55.600 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:56:55.600 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-20T16:56:55.600 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:56:55.600 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-20T16:56:55.600 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:56:55.601 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 90 k 2024-04-20T16:56:55.601 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 310 k 2024-04-20T16:56:55.601 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-20T16:56:55.640 INFO:teuthology.orchestra.run.smithi106.stdout:librados-devel-14.2.22-0.el8.x86_64.rpm 2.3 MB/s | 90 kB 00:00 2024-04-20T16:56:55.640 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:56:55.640 INFO:teuthology.orchestra.run.smithi106.stdout:Total 2.2 MB/s | 90 kB 00:00 2024-04-20T16:56:55.641 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-20T16:56:55.646 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-04-20T16:56:55.651 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-20T16:56:55.651 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-20T16:56:55.701 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-20T16:56:55.704 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-20T16:56:55.811 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-20T16:56:55.845 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:56:55.972 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:56:56.111 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:56:56.250 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:56:56.490 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:56:56.490 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:56:56.490 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-04-20T16:56:56.490 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs-devel-2:14.2.22-0.el8.x86_64 2024-04-20T16:56:56.491 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:56:56.491 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-04-20T16:56:56.591 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librados-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:56:56.591 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:56:56.591 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-20T16:56:56.591 INFO:teuthology.orchestra.run.smithi106.stdout: librados-devel-2:14.2.22-0.el8.x86_64 2024-04-20T16:56:56.592 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:56:56.592 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-20T16:56:56.632 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install librados2 2024-04-20T16:56:56.852 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install libcephfs2 2024-04-20T16:56:56.949 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-04-20T16:56:56.949 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:56:56.949 INFO:teuthology.orchestra.run.smithi005.stdout: Package Architecture Version Repository Size 2024-04-20T16:56:56.949 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:56:56.950 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-04-20T16:56:56.950 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs-devel x86_64 2:14.2.22-0.el8 ceph 22 k 2024-04-20T16:56:56.950 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:56:56.950 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-04-20T16:56:56.950 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:56:56.950 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-04-20T16:56:56.950 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:56:56.951 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 22 k 2024-04-20T16:56:56.951 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 77 k 2024-04-20T16:56:56.951 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-04-20T16:56:57.003 INFO:teuthology.orchestra.run.smithi005.stdout:libcephfs-devel-14.2.22-0.el8.x86_64.rpm 411 kB/s | 22 kB 00:00 2024-04-20T16:56:57.004 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:56:57.004 INFO:teuthology.orchestra.run.smithi005.stdout:Total 400 kB/s | 22 kB 00:00 2024-04-20T16:56:57.004 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-04-20T16:56:57.008 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-04-20T16:56:57.008 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-04-20T16:56:57.010 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-04-20T16:56:57.013 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-04-20T16:56:57.081 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-04-20T16:56:57.227 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:56:57.361 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:55 ago on Sat 20 Apr 2024 04:55:02 PM UTC. 2024-04-20T16:56:57.503 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:56:57.585 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:45 ago on Sat 20 Apr 2024 04:55:12 PM UTC. 2024-04-20T16:56:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:56:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:56:57.871 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-04-20T16:56:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs-devel-2:14.2.22-0.el8.x86_64 2024-04-20T16:56:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:56:57.871 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-04-20T16:56:58.008 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install librados2 2024-04-20T16:56:58.762 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:01 ago on Sat 20 Apr 2024 04:54:57 PM UTC. 2024-04-20T16:56:58.944 INFO:teuthology.orchestra.run.smithi171.stdout:Package librados2-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-20T16:56:58.992 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-04-20T16:56:58.993 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-04-20T16:56:58.993 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-04-20T16:56:59.112 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install librbd1 2024-04-20T16:56:59.171 INFO:teuthology.orchestra.run.smithi106.stdout:Package libcephfs2-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-20T16:56:59.218 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-20T16:56:59.219 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-04-20T16:56:59.219 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-20T16:56:59.350 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install libcephfs-devel 2024-04-20T16:56:59.838 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:57 ago on Sat 20 Apr 2024 04:55:02 PM UTC. 2024-04-20T16:57:00.075 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:48 ago on Sat 20 Apr 2024 04:55:12 PM UTC. 2024-04-20T16:57:00.478 INFO:teuthology.orchestra.run.smithi005.stdout:Package librados2-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-20T16:57:00.530 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-04-20T16:57:00.532 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-04-20T16:57:00.532 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-04-20T16:57:00.756 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install librbd1 2024-04-20T16:57:01.415 INFO:teuthology.orchestra.run.smithi171.stdout:Package librbd1-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-20T16:57:01.464 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-04-20T16:57:01.465 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-04-20T16:57:01.465 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-04-20T16:57:01.511 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:04 ago on Sat 20 Apr 2024 04:54:57 PM UTC. 2024-04-20T16:57:01.580 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install rbd-fuse 2024-04-20T16:57:01.708 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-20T16:57:01.708 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:57:01.708 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-04-20T16:57:01.708 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:57:01.709 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-20T16:57:01.709 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs-devel x86_64 2:14.2.22-0.el8 ceph 22 k 2024-04-20T16:57:01.709 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:57:01.709 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-20T16:57:01.709 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:57:01.709 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-20T16:57:01.709 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:57:01.710 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 22 k 2024-04-20T16:57:01.710 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 77 k 2024-04-20T16:57:01.710 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-20T16:57:01.762 INFO:teuthology.orchestra.run.smithi106.stdout:libcephfs-devel-14.2.22-0.el8.x86_64.rpm 410 kB/s | 22 kB 00:00 2024-04-20T16:57:01.763 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:57:01.763 INFO:teuthology.orchestra.run.smithi106.stdout:Total 401 kB/s | 22 kB 00:00 2024-04-20T16:57:01.763 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-20T16:57:01.767 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-20T16:57:01.767 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-20T16:57:01.769 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-20T16:57:01.772 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-20T16:57:01.847 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-20T16:57:02.005 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:02.264 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:02.323 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:00 ago on Sat 20 Apr 2024 04:55:02 PM UTC. 2024-04-20T16:57:02.609 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libcephfs-devel-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:02.609 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:57:02.610 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-20T16:57:02.610 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs-devel-2:14.2.22-0.el8.x86_64 2024-04-20T16:57:02.610 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:57:02.610 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-20T16:57:02.723 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install librados2 2024-04-20T16:57:03.217 INFO:teuthology.orchestra.run.smithi005.stdout:Package librbd1-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-20T16:57:03.269 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-04-20T16:57:03.270 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-04-20T16:57:03.270 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-04-20T16:57:03.380 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install rbd-fuse 2024-04-20T16:57:03.444 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:51 ago on Sat 20 Apr 2024 04:55:12 PM UTC. 2024-04-20T16:57:03.965 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-04-20T16:57:03.966 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:57:03.966 INFO:teuthology.orchestra.run.smithi171.stdout: Package Architecture Version Repository Size 2024-04-20T16:57:03.966 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:57:03.966 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-04-20T16:57:03.966 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-fuse x86_64 2:14.2.22-0.el8 ceph 73 k 2024-04-20T16:57:03.966 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:57:03.966 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-04-20T16:57:03.966 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:57:03.966 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-04-20T16:57:03.966 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:57:03.967 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 73 k 2024-04-20T16:57:03.967 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 188 k 2024-04-20T16:57:03.967 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-04-20T16:57:04.021 INFO:teuthology.orchestra.run.smithi171.stdout:rbd-fuse-14.2.22-0.el8.x86_64.rpm 1.3 MB/s | 73 kB 00:00 2024-04-20T16:57:04.022 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:57:04.022 INFO:teuthology.orchestra.run.smithi171.stdout:Total 1.3 MB/s | 73 kB 00:00 2024-04-20T16:57:04.022 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-04-20T16:57:04.033 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-04-20T16:57:04.033 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-04-20T16:57:04.083 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-04-20T16:57:04.086 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-04-20T16:57:04.135 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:07 ago on Sat 20 Apr 2024 04:54:57 PM UTC. 2024-04-20T16:57:04.222 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-04-20T16:57:04.435 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:04.750 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:05.020 INFO:teuthology.orchestra.run.smithi106.stdout:Package librados2-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-20T16:57:05.067 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-20T16:57:05.069 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-04-20T16:57:05.069 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-20T16:57:05.161 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:05.162 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:57:05.162 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-04-20T16:57:05.162 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-fuse-2:14.2.22-0.el8.x86_64 2024-04-20T16:57:05.162 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:57:05.162 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-04-20T16:57:05.169 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install librbd1 2024-04-20T16:57:05.373 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install rbd-mirror 2024-04-20T16:57:05.887 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-04-20T16:57:05.888 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:57:05.888 INFO:teuthology.orchestra.run.smithi005.stdout: Package Architecture Version Repository Size 2024-04-20T16:57:05.888 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:57:05.888 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-04-20T16:57:05.889 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-fuse x86_64 2:14.2.22-0.el8 ceph 73 k 2024-04-20T16:57:05.889 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:57:05.889 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-04-20T16:57:05.889 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:57:05.889 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-04-20T16:57:05.889 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:57:05.889 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 73 k 2024-04-20T16:57:05.890 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 188 k 2024-04-20T16:57:05.890 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-04-20T16:57:05.890 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:53 ago on Sat 20 Apr 2024 04:55:12 PM UTC. 2024-04-20T16:57:05.942 INFO:teuthology.orchestra.run.smithi005.stdout:rbd-fuse-14.2.22-0.el8.x86_64.rpm 1.3 MB/s | 73 kB 00:00 2024-04-20T16:57:05.942 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:57:05.943 INFO:teuthology.orchestra.run.smithi005.stdout:Total 1.3 MB/s | 73 kB 00:00 2024-04-20T16:57:05.943 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-04-20T16:57:05.954 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-04-20T16:57:05.955 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-04-20T16:57:06.007 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-04-20T16:57:06.010 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-04-20T16:57:06.112 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:04 ago on Sat 20 Apr 2024 04:55:02 PM UTC. 2024-04-20T16:57:06.117 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-04-20T16:57:06.367 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:06.664 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:07.453 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:07.454 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:57:07.454 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-04-20T16:57:07.454 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-fuse-2:14.2.22-0.el8.x86_64 2024-04-20T16:57:07.454 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:57:07.454 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-04-20T16:57:07.464 INFO:teuthology.orchestra.run.smithi106.stdout:Package librbd1-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-20T16:57:07.512 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-20T16:57:07.513 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-04-20T16:57:07.513 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-20T16:57:07.646 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install rbd-fuse 2024-04-20T16:57:07.695 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install rbd-mirror 2024-04-20T16:57:07.762 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-04-20T16:57:07.763 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:57:07.763 INFO:teuthology.orchestra.run.smithi171.stdout: Package Architecture Version Repository Size 2024-04-20T16:57:07.763 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:57:07.763 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-04-20T16:57:07.763 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-mirror x86_64 2:14.2.22-0.el8 ceph 2.3 M 2024-04-20T16:57:07.764 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:57:07.764 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-04-20T16:57:07.764 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:57:07.764 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-04-20T16:57:07.764 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:57:07.764 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 2.3 M 2024-04-20T16:57:07.764 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 8.4 M 2024-04-20T16:57:07.764 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-04-20T16:57:07.880 INFO:teuthology.orchestra.run.smithi171.stdout:rbd-mirror-14.2.22-0.el8.x86_64.rpm 20 MB/s | 2.3 MB 00:00 2024-04-20T16:57:07.880 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:57:07.881 INFO:teuthology.orchestra.run.smithi171.stdout:Total 20 MB/s | 2.3 MB 00:00 2024-04-20T16:57:07.881 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-04-20T16:57:07.893 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-04-20T16:57:07.893 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-04-20T16:57:07.951 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-04-20T16:57:07.954 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-04-20T16:57:08.257 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-04-20T16:57:08.368 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:56 ago on Sat 20 Apr 2024 04:55:12 PM UTC. 2024-04-20T16:57:08.454 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:11 ago on Sat 20 Apr 2024 04:54:57 PM UTC. 2024-04-20T16:57:08.511 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:08.935 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:09.308 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:09.308 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:57:09.308 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-04-20T16:57:09.308 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-mirror-2:14.2.22-0.el8.x86_64 2024-04-20T16:57:09.308 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:57:09.309 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-04-20T16:57:09.557 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install rbd-nbd 2024-04-20T16:57:10.007 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-20T16:57:10.008 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:57:10.008 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-04-20T16:57:10.008 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:57:10.008 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-20T16:57:10.008 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-fuse x86_64 2:14.2.22-0.el8 ceph 73 k 2024-04-20T16:57:10.008 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:57:10.008 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-20T16:57:10.008 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:57:10.008 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-20T16:57:10.008 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:57:10.009 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 73 k 2024-04-20T16:57:10.009 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 188 k 2024-04-20T16:57:10.009 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-20T16:57:10.049 INFO:teuthology.orchestra.run.smithi106.stdout:rbd-fuse-14.2.22-0.el8.x86_64.rpm 1.8 MB/s | 73 kB 00:00 2024-04-20T16:57:10.050 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:57:10.050 INFO:teuthology.orchestra.run.smithi106.stdout:Total 1.7 MB/s | 73 kB 00:00 2024-04-20T16:57:10.050 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-20T16:57:10.061 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-20T16:57:10.061 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-20T16:57:10.110 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-20T16:57:10.113 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-20T16:57:10.202 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-04-20T16:57:10.202 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:57:10.202 INFO:teuthology.orchestra.run.smithi005.stdout: Package Architecture Version Repository Size 2024-04-20T16:57:10.202 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:57:10.202 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-04-20T16:57:10.202 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-mirror x86_64 2:14.2.22-0.el8 ceph 2.3 M 2024-04-20T16:57:10.202 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:57:10.203 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-04-20T16:57:10.203 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:57:10.203 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-04-20T16:57:10.203 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:57:10.203 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 2.3 M 2024-04-20T16:57:10.203 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 8.4 M 2024-04-20T16:57:10.203 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-04-20T16:57:10.213 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-20T16:57:10.295 INFO:teuthology.orchestra.run.smithi005.stdout:rbd-mirror-14.2.22-0.el8.x86_64.rpm 25 MB/s | 2.3 MB 00:00 2024-04-20T16:57:10.296 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:57:10.296 INFO:teuthology.orchestra.run.smithi005.stdout:Total 25 MB/s | 2.3 MB 00:00 2024-04-20T16:57:10.296 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-04-20T16:57:10.297 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:08 ago on Sat 20 Apr 2024 04:55:02 PM UTC. 2024-04-20T16:57:10.310 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-04-20T16:57:10.310 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-04-20T16:57:10.371 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-04-20T16:57:10.374 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-04-20T16:57:10.437 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:10.699 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-04-20T16:57:10.737 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:10.901 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:11.343 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:11.539 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : rbd-fuse-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:11.540 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:57:11.540 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-20T16:57:11.540 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-fuse-2:14.2.22-0.el8.x86_64 2024-04-20T16:57:11.540 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:57:11.540 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-20T16:57:11.815 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:11.815 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:57:11.815 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-04-20T16:57:11.815 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-mirror-2:14.2.22-0.el8.x86_64 2024-04-20T16:57:11.816 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:57:11.816 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-04-20T16:57:11.929 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-04-20T16:57:11.929 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:57:11.930 INFO:teuthology.orchestra.run.smithi171.stdout: Package Architecture Version Repository Size 2024-04-20T16:57:11.930 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:57:11.930 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-04-20T16:57:11.930 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-nbd x86_64 2:14.2.22-0.el8 ceph 142 k 2024-04-20T16:57:11.930 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:57:11.930 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-04-20T16:57:11.930 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-04-20T16:57:11.930 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-04-20T16:57:11.930 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:57:11.931 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 142 k 2024-04-20T16:57:11.931 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 422 k 2024-04-20T16:57:11.931 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-04-20T16:57:11.976 INFO:teuthology.orchestra.run.smithi171.stdout:rbd-nbd-14.2.22-0.el8.x86_64.rpm 3.1 MB/s | 142 kB 00:00 2024-04-20T16:57:11.977 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:57:11.977 INFO:teuthology.orchestra.run.smithi171.stdout:Total 3.0 MB/s | 142 kB 00:00 2024-04-20T16:57:11.977 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-04-20T16:57:11.988 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-04-20T16:57:11.988 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-04-20T16:57:12.026 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install rbd-mirror 2024-04-20T16:57:12.047 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-04-20T16:57:12.050 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-04-20T16:57:12.058 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install rbd-nbd 2024-04-20T16:57:12.184 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-04-20T16:57:12.359 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:12.679 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:12.763 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:02:00 ago on Sat 20 Apr 2024 04:55:12 PM UTC. 2024-04-20T16:57:12.821 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:15 ago on Sat 20 Apr 2024 04:54:57 PM UTC. 2024-04-20T16:57:13.049 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:13.049 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:57:13.049 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-04-20T16:57:13.049 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-nbd-2:14.2.22-0.el8.x86_64 2024-04-20T16:57:13.049 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T16:57:13.049 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-04-20T16:57:13.996 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python-rados 2024-04-20T16:57:14.404 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-20T16:57:14.405 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:57:14.405 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-04-20T16:57:14.405 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:57:14.405 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-20T16:57:14.405 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-mirror x86_64 2:14.2.22-0.el8 ceph 2.3 M 2024-04-20T16:57:14.405 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:57:14.405 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-20T16:57:14.405 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:57:14.405 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-20T16:57:14.405 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:57:14.406 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 2.3 M 2024-04-20T16:57:14.406 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 8.4 M 2024-04-20T16:57:14.406 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-20T16:57:14.471 INFO:teuthology.orchestra.run.smithi106.stdout:rbd-mirror-14.2.22-0.el8.x86_64.rpm 36 MB/s | 2.3 MB 00:00 2024-04-20T16:57:14.471 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:57:14.472 INFO:teuthology.orchestra.run.smithi106.stdout:Total 35 MB/s | 2.3 MB 00:00 2024-04-20T16:57:14.472 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-20T16:57:14.484 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-20T16:57:14.484 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-20T16:57:14.541 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-20T16:57:14.544 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-20T16:57:14.581 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-04-20T16:57:14.581 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:57:14.581 INFO:teuthology.orchestra.run.smithi005.stdout: Package Architecture Version Repository Size 2024-04-20T16:57:14.581 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:57:14.581 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-04-20T16:57:14.581 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-nbd x86_64 2:14.2.22-0.el8 ceph 142 k 2024-04-20T16:57:14.582 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:57:14.582 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-04-20T16:57:14.582 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-04-20T16:57:14.582 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-04-20T16:57:14.582 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:57:14.582 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 142 k 2024-04-20T16:57:14.582 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 422 k 2024-04-20T16:57:14.582 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-04-20T16:57:14.658 INFO:teuthology.orchestra.run.smithi005.stdout:rbd-nbd-14.2.22-0.el8.x86_64.rpm 1.8 MB/s | 142 kB 00:00 2024-04-20T16:57:14.659 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:57:14.659 INFO:teuthology.orchestra.run.smithi005.stdout:Total 1.8 MB/s | 142 kB 00:00 2024-04-20T16:57:14.659 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-04-20T16:57:14.671 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-04-20T16:57:14.671 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-04-20T16:57:14.735 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-04-20T16:57:14.737 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:12 ago on Sat 20 Apr 2024 04:55:02 PM UTC. 2024-04-20T16:57:14.738 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-04-20T16:57:14.851 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-20T16:57:14.865 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-04-20T16:57:15.054 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:15.057 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:15.356 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:15.468 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:15.737 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:15.737 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:57:15.737 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-04-20T16:57:15.738 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-nbd-2:14.2.22-0.el8.x86_64 2024-04-20T16:57:15.738 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:57:15.738 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-04-20T16:57:15.833 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : rbd-mirror-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:15.833 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:57:15.833 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-20T16:57:15.833 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-mirror-2:14.2.22-0.el8.x86_64 2024-04-20T16:57:15.833 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:57:15.833 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-20T16:57:15.998 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install python-rados 2024-04-20T16:57:16.086 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install rbd-nbd 2024-04-20T16:57:16.330 INFO:teuthology.orchestra.run.smithi171.stdout:Package python3-rados-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-20T16:57:16.378 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-04-20T16:57:16.379 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-04-20T16:57:16.380 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-04-20T16:57:16.508 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python-rgw 2024-04-20T16:57:16.756 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:19 ago on Sat 20 Apr 2024 04:54:57 PM UTC. 2024-04-20T16:57:16.820 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:02:04 ago on Sat 20 Apr 2024 04:55:12 PM UTC. 2024-04-20T16:57:17.243 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:15 ago on Sat 20 Apr 2024 04:55:02 PM UTC. 2024-04-20T16:57:18.459 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-20T16:57:18.459 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:57:18.459 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-04-20T16:57:18.459 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:57:18.460 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-04-20T16:57:18.460 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-nbd x86_64 2:14.2.22-0.el8 ceph 142 k 2024-04-20T16:57:18.460 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:57:18.460 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-04-20T16:57:18.460 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-04-20T16:57:18.460 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-04-20T16:57:18.460 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:57:18.460 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 142 k 2024-04-20T16:57:18.460 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 422 k 2024-04-20T16:57:18.460 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-04-20T16:57:18.460 INFO:teuthology.orchestra.run.smithi005.stdout:Package python3-rados-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-20T16:57:18.514 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-04-20T16:57:18.515 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-04-20T16:57:18.515 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-04-20T16:57:18.525 INFO:teuthology.orchestra.run.smithi106.stdout:rbd-nbd-14.2.22-0.el8.x86_64.rpm 2.1 MB/s | 142 kB 00:00 2024-04-20T16:57:18.525 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:57:18.525 INFO:teuthology.orchestra.run.smithi106.stdout:Total 2.1 MB/s | 142 kB 00:00 2024-04-20T16:57:18.526 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-04-20T16:57:18.537 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-04-20T16:57:18.537 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-04-20T16:57:18.595 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-04-20T16:57:18.598 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-04-20T16:57:18.632 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install python-rgw 2024-04-20T16:57:18.710 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-04-20T16:57:18.822 INFO:teuthology.orchestra.run.smithi171.stdout:Package python3-rgw-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-20T16:57:18.871 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-04-20T16:57:18.872 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-04-20T16:57:18.873 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-04-20T16:57:18.894 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:18.997 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python-rbd 2024-04-20T16:57:19.187 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:19.391 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:22 ago on Sat 20 Apr 2024 04:54:57 PM UTC. 2024-04-20T16:57:19.564 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : rbd-nbd-2:14.2.22-0.el8.x86_64 1/1 2024-04-20T16:57:19.564 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:57:19.564 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-04-20T16:57:19.564 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-nbd-2:14.2.22-0.el8.x86_64 2024-04-20T16:57:19.564 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-04-20T16:57:19.564 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-20T16:57:19.726 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:17 ago on Sat 20 Apr 2024 04:55:02 PM UTC. 2024-04-20T16:57:19.791 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install python-rados 2024-04-20T16:57:20.525 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:02:08 ago on Sat 20 Apr 2024 04:55:12 PM UTC. 2024-04-20T16:57:21.101 INFO:teuthology.orchestra.run.smithi005.stdout:Package python3-rgw-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-20T16:57:21.154 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-04-20T16:57:21.156 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-04-20T16:57:21.156 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-04-20T16:57:21.267 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install python-rbd 2024-04-20T16:57:21.305 INFO:teuthology.orchestra.run.smithi171.stdout:Package python3-rbd-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-20T16:57:21.356 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-04-20T16:57:21.357 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-04-20T16:57:21.357 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-04-20T16:57:21.478 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python-cephfs 2024-04-20T16:57:22.024 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:25 ago on Sat 20 Apr 2024 04:54:57 PM UTC. 2024-04-20T16:57:22.100 INFO:teuthology.orchestra.run.smithi106.stdout:Package python3-rados-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-20T16:57:22.150 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-20T16:57:22.151 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-04-20T16:57:22.151 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-20T16:57:22.211 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:20 ago on Sat 20 Apr 2024 04:55:02 PM UTC. 2024-04-20T16:57:22.253 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install python-rgw 2024-04-20T16:57:22.976 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:02:10 ago on Sat 20 Apr 2024 04:55:12 PM UTC. 2024-04-20T16:57:23.725 INFO:teuthology.orchestra.run.smithi005.stdout:Package python3-rbd-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-20T16:57:23.780 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-04-20T16:57:23.782 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-04-20T16:57:23.782 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-04-20T16:57:23.802 INFO:teuthology.orchestra.run.smithi171.stdout:Package python3-cephfs-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-20T16:57:23.851 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-04-20T16:57:23.853 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-04-20T16:57:23.853 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-04-20T16:57:23.900 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install python-cephfs 2024-04-20T16:57:23.977 DEBUG:teuthology.parallel:result is None 2024-04-20T16:57:24.554 INFO:teuthology.orchestra.run.smithi106.stdout:Package python3-rgw-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-20T16:57:24.605 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-20T16:57:24.606 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-04-20T16:57:24.606 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-20T16:57:24.662 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:27 ago on Sat 20 Apr 2024 04:54:57 PM UTC. 2024-04-20T16:57:24.700 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install python-rbd 2024-04-20T16:57:25.422 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:02:13 ago on Sat 20 Apr 2024 04:55:12 PM UTC. 2024-04-20T16:57:26.357 INFO:teuthology.orchestra.run.smithi005.stdout:Package python3-cephfs-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-20T16:57:26.412 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-04-20T16:57:26.413 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-04-20T16:57:26.413 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-04-20T16:57:26.531 DEBUG:teuthology.parallel:result is None 2024-04-20T16:57:27.007 INFO:teuthology.orchestra.run.smithi106.stdout:Package python3-rbd-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-20T16:57:27.057 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-20T16:57:27.058 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-04-20T16:57:27.058 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-20T16:57:27.162 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install python-cephfs 2024-04-20T16:57:27.885 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:02:15 ago on Sat 20 Apr 2024 04:55:12 PM UTC. 2024-04-20T16:57:29.463 INFO:teuthology.orchestra.run.smithi106.stdout:Package python3-cephfs-2:14.2.22-0.el8.x86_64 is already installed. 2024-04-20T16:57:29.511 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-04-20T16:57:29.512 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-04-20T16:57:29.512 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-04-20T16:57:29.616 DEBUG:teuthology.parallel:result is None 2024-04-20T16:57:29.616 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-20T16:57:29.616 INFO:teuthology.packaging:ref: None 2024-04-20T16:57:29.616 INFO:teuthology.packaging:tag: None 2024-04-20T16:57:29.616 INFO:teuthology.packaging:branch: nautilus 2024-04-20T16:57:29.616 INFO:teuthology.packaging:sha1: 36c371567dddacf6207ea36f2535396ab31415fc 2024-04-20T16:57:29.617 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=nautilus 2024-04-20T16:57:29.884 DEBUG:teuthology.orchestra.run.smithi005:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-20T16:57:29.911 INFO:teuthology.orchestra.run.smithi005.stdout:14.2.22-0.el8 2024-04-20T16:57:29.912 INFO:teuthology.packaging:The installed version of ceph is 14.2.22-0.el8 2024-04-20T16:57:29.912 INFO:teuthology.task.install:The correct ceph version 14.2.22-0 is installed. 2024-04-20T16:57:29.913 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-20T16:57:29.913 INFO:teuthology.packaging:ref: None 2024-04-20T16:57:29.914 INFO:teuthology.packaging:tag: None 2024-04-20T16:57:29.914 INFO:teuthology.packaging:branch: nautilus 2024-04-20T16:57:29.914 INFO:teuthology.packaging:sha1: 36c371567dddacf6207ea36f2535396ab31415fc 2024-04-20T16:57:29.914 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=nautilus 2024-04-20T16:57:30.095 DEBUG:teuthology.orchestra.run.smithi106:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-20T16:57:30.119 INFO:teuthology.orchestra.run.smithi106.stdout:14.2.22-0.el8 2024-04-20T16:57:30.119 INFO:teuthology.packaging:The installed version of ceph is 14.2.22-0.el8 2024-04-20T16:57:30.119 INFO:teuthology.task.install:The correct ceph version 14.2.22-0 is installed. 2024-04-20T16:57:30.121 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-20T16:57:30.121 INFO:teuthology.packaging:ref: None 2024-04-20T16:57:30.121 INFO:teuthology.packaging:tag: None 2024-04-20T16:57:30.121 INFO:teuthology.packaging:branch: nautilus 2024-04-20T16:57:30.121 INFO:teuthology.packaging:sha1: 36c371567dddacf6207ea36f2535396ab31415fc 2024-04-20T16:57:30.122 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=nautilus 2024-04-20T16:57:30.377 DEBUG:teuthology.orchestra.run.smithi171:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-20T16:57:30.404 INFO:teuthology.orchestra.run.smithi171.stdout:14.2.22-0.el8 2024-04-20T16:57:30.404 INFO:teuthology.packaging:The installed version of ceph is 14.2.22-0.el8 2024-04-20T16:57:30.404 INFO:teuthology.task.install:The correct ceph version 14.2.22-0 is installed. 2024-04-20T16:57:30.406 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-20T16:57:30.406 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-04-20T16:57:30.406 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-20T16:57:30.440 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-20T16:57:30.440 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-20T16:57:30.470 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-04-20T16:57:30.470 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-20T16:57:30.503 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-20T16:57:30.503 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-04-20T16:57:30.503 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/daemon-helper 2024-04-20T16:57:30.535 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-20T16:57:30.606 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-20T16:57:30.606 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/daemon-helper 2024-04-20T16:57:30.636 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-20T16:57:30.706 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-04-20T16:57:30.706 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/daemon-helper 2024-04-20T16:57:30.738 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-20T16:57:30.808 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-20T16:57:30.808 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-04-20T16:57:30.808 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-20T16:57:30.841 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-20T16:57:30.913 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-20T16:57:30.913 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-20T16:57:30.942 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-20T16:57:31.011 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-04-20T16:57:31.011 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-20T16:57:31.041 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-20T16:57:31.111 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-20T16:57:31.111 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-04-20T16:57:31.111 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/stdin-killer 2024-04-20T16:57:31.145 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-20T16:57:31.218 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-20T16:57:31.218 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/stdin-killer 2024-04-20T16:57:31.247 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-20T16:57:31.315 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-04-20T16:57:31.316 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/stdin-killer 2024-04-20T16:57:31.345 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-20T16:57:31.414 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-20T16:57:31.513 INFO:tasks.cephadm:Config: {'conf': {'mon': {'auth allow insecure global id reclaim': True, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd heartbeat use min delay socket': True, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'mon_bind_msgr2': False, 'sha1': '36c371567dddacf6207ea36f2535396ab31415fc'} 2024-04-20T16:57:31.514 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc 2024-04-20T16:57:31.515 INFO:tasks.cephadm:Cluster fsid is 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T16:57:31.515 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-20T16:57:31.515 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.5:6789]', 'mon.b': '[v1:172.21.15.106:6789]', 'mon.c': '[v1:172.21.15.171:6789]'} 2024-04-20T16:57:31.515 INFO:tasks.cephadm:First mon is mon.a on smithi005 2024-04-20T16:57:31.515 INFO:tasks.cephadm:First mgr is y 2024-04-20T16:57:31.515 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-20T16:57:31.515 DEBUG:teuthology.orchestra.run.smithi005:> sudo hostname $(hostname -s) 2024-04-20T16:57:31.550 DEBUG:teuthology.orchestra.run.smithi106:> sudo hostname $(hostname -s) 2024-04-20T16:57:31.579 DEBUG:teuthology.orchestra.run.smithi171:> sudo hostname $(hostname -s) 2024-04-20T16:57:31.609 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-20T16:57:31.610 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=36c371567dddacf6207ea36f2535396ab31415fc 2024-04-20T16:57:31.751 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '36c371567dddacf6207ea36f2535396ab31415fc', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/78756/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3160-g36c37156', 'node_name': '172.21.5.32+adami02', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3160.g36c37156'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-yuriw-testing-20240419.185239-main/36c371567dddacf6207ea36f2535396ab31415fc/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2024-04-20 02:04:20.922528', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-yuriw-testing-20240419.185239-main', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-yuriw-testing-20240419.185239-main/36c371567dddacf6207ea36f2535396ab31415fc/centos/8/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-04-20T16:57:31.972 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-yuriw-testing-20240419.185239-main, sha1 36c371567dddacf6207ea36f2535396ab31415fc from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=36c371567dddacf6207ea36f2535396ab31415fc 2024-04-20T16:57:31.973 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-yuriw-testing-20240419.185239-main/36c371567dddacf6207ea36f2535396ab31415fc/centos/8/x86_64/flavors/default/cephadm 2024-04-20T16:57:31.973 DEBUG:teuthology.orchestra.run.smithi005:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-yuriw-testing-20240419.185239-main/36c371567dddacf6207ea36f2535396ab31415fc/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-20T16:57:32.712 INFO:teuthology.orchestra.run.smithi005.stdout:-rw-rw-r--. 1 ubuntu ubuntu 2294149 Apr 20 16:57 /home/ubuntu/cephtest/cephadm 2024-04-20T16:57:32.712 DEBUG:teuthology.orchestra.run.smithi106:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-yuriw-testing-20240419.185239-main/36c371567dddacf6207ea36f2535396ab31415fc/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-20T16:57:33.471 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-rw-r--. 1 ubuntu ubuntu 2294149 Apr 20 16:57 /home/ubuntu/cephtest/cephadm 2024-04-20T16:57:33.471 DEBUG:teuthology.orchestra.run.smithi171:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-yuriw-testing-20240419.185239-main/36c371567dddacf6207ea36f2535396ab31415fc/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-20T16:57:33.999 INFO:teuthology.orchestra.run.smithi171.stdout:-rw-rw-r--. 1 ubuntu ubuntu 2294149 Apr 20 16:57 /home/ubuntu/cephtest/cephadm 2024-04-20T16:57:34.000 DEBUG:teuthology.orchestra.run.smithi005:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-20T16:57:34.023 DEBUG:teuthology.orchestra.run.smithi106:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-20T16:57:34.046 DEBUG:teuthology.orchestra.run.smithi171:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-20T16:57:34.082 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc on all hosts... 2024-04-20T16:57:34.082 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc pull 2024-04-20T16:57:34.084 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc pull 2024-04-20T16:57:34.088 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc pull 2024-04-20T16:57:35.278 INFO:teuthology.orchestra.run.smithi106.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc... 2024-04-20T16:57:35.368 INFO:teuthology.orchestra.run.smithi171.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc... 2024-04-20T16:57:35.372 INFO:teuthology.orchestra.run.smithi005.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc... 2024-04-20T16:58:04.125 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T16:58:04.126 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_version": "ceph version 19.0.0-3160-g36c37156 (36c371567dddacf6207ea36f2535396ab31415fc) squid (dev)", 2024-04-20T16:58:04.126 INFO:teuthology.orchestra.run.smithi171.stdout: "image_id": "3963ae0cc69b2c056e395257d0fd7aa926df0f6f09e4d860cc3407d54ee2f33b", 2024-04-20T16:58:04.126 INFO:teuthology.orchestra.run.smithi171.stdout: "repo_digests": [ 2024-04-20T16:58:04.126 INFO:teuthology.orchestra.run.smithi171.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dc102f95bf5a3cb0e41ae2360714141bf44b1867f263b70ab9e489d297e81b79" 2024-04-20T16:58:04.126 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T16:58:04.126 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T16:58:11.255 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T16:58:11.255 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_version": "ceph version 19.0.0-3160-g36c37156 (36c371567dddacf6207ea36f2535396ab31415fc) squid (dev)", 2024-04-20T16:58:11.255 INFO:teuthology.orchestra.run.smithi106.stdout: "image_id": "3963ae0cc69b2c056e395257d0fd7aa926df0f6f09e4d860cc3407d54ee2f33b", 2024-04-20T16:58:11.255 INFO:teuthology.orchestra.run.smithi106.stdout: "repo_digests": [ 2024-04-20T16:58:11.255 INFO:teuthology.orchestra.run.smithi106.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dc102f95bf5a3cb0e41ae2360714141bf44b1867f263b70ab9e489d297e81b79" 2024-04-20T16:58:11.255 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T16:58:11.255 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T16:58:14.429 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T16:58:14.429 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph_version": "ceph version 19.0.0-3160-g36c37156 (36c371567dddacf6207ea36f2535396ab31415fc) squid (dev)", 2024-04-20T16:58:14.430 INFO:teuthology.orchestra.run.smithi005.stdout: "image_id": "3963ae0cc69b2c056e395257d0fd7aa926df0f6f09e4d860cc3407d54ee2f33b", 2024-04-20T16:58:14.430 INFO:teuthology.orchestra.run.smithi005.stdout: "repo_digests": [ 2024-04-20T16:58:14.430 INFO:teuthology.orchestra.run.smithi005.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dc102f95bf5a3cb0e41ae2360714141bf44b1867f263b70ab9e489d297e81b79" 2024-04-20T16:58:14.430 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T16:58:14.430 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T16:58:14.462 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph 2024-04-20T16:58:14.499 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /etc/ceph 2024-04-20T16:58:14.537 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /etc/ceph 2024-04-20T16:58:14.575 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 777 /etc/ceph 2024-04-20T16:58:14.607 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 777 /etc/ceph 2024-04-20T16:58:14.640 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 777 /etc/ceph 2024-04-20T16:58:14.674 INFO:tasks.cephadm:Writing seed config... 2024-04-20T16:58:14.675 INFO:tasks.cephadm: override: [mon] auth allow insecure global id reclaim = True 2024-04-20T16:58:14.675 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-20T16:58:14.675 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-20T16:58:14.675 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-20T16:58:14.676 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-04-20T16:58:14.676 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-04-20T16:58:14.676 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-04-20T16:58:14.676 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-04-20T16:58:14.676 INFO:tasks.cephadm: override: [mon] mon scrub interval = 300 2024-04-20T16:58:14.676 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-04-20T16:58:14.676 INFO:tasks.cephadm: override: [client.2] debug ms = 1 2024-04-20T16:58:14.676 INFO:tasks.cephadm: override: [client.2] debug objecter = 20 2024-04-20T16:58:14.676 INFO:tasks.cephadm: override: [client.2] debug rados = 20 2024-04-20T16:58:14.676 INFO:tasks.cephadm: override: [global] mon client directed command retry = 5 2024-04-20T16:58:14.676 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-04-20T16:58:14.676 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-04-20T16:58:14.676 INFO:tasks.cephadm: override: [global] ms inject socket failures = 5000 2024-04-20T16:58:14.676 INFO:tasks.cephadm: override: [global] ms type = async 2024-04-20T16:58:14.676 INFO:tasks.cephadm: override: [global] osd_pool_default_min_size = 2 2024-04-20T16:58:14.677 INFO:tasks.cephadm: override: [global] osd_pool_default_size = 2 2024-04-20T16:58:14.677 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-20T16:58:14.677 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-20T16:58:14.677 INFO:tasks.cephadm: override: [osd] bluestore zero block detection = True 2024-04-20T16:58:14.677 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-20T16:58:14.677 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-20T16:58:14.677 INFO:tasks.cephadm: override: [osd] osd backoff on peering = True 2024-04-20T16:58:14.677 INFO:tasks.cephadm: override: [osd] osd blocked scrub grace period = 3600 2024-04-20T16:58:14.677 INFO:tasks.cephadm: override: [osd] osd debug verify cached snaps = True 2024-04-20T16:58:14.677 INFO:tasks.cephadm: override: [osd] osd debug verify missing on start = True 2024-04-20T16:58:14.677 INFO:tasks.cephadm: override: [osd] osd heartbeat use min delay socket = True 2024-04-20T16:58:14.677 INFO:tasks.cephadm: override: [osd] osd map cache size = 1 2024-04-20T16:58:14.677 INFO:tasks.cephadm: override: [osd] osd max backfills = 6 2024-04-20T16:58:14.677 INFO:tasks.cephadm: override: [osd] osd max markdown count = 1000 2024-04-20T16:58:14.677 INFO:tasks.cephadm: override: [osd] osd mclock override recovery settings = True 2024-04-20T16:58:14.677 INFO:tasks.cephadm: override: [osd] osd mclock profile = high_recovery_ops 2024-04-20T16:58:14.678 INFO:tasks.cephadm: override: [osd] osd op queue = debug_random 2024-04-20T16:58:14.678 INFO:tasks.cephadm: override: [osd] osd op queue cut off = debug_random 2024-04-20T16:58:14.678 INFO:tasks.cephadm: override: [osd] osd scrub during recovery = False 2024-04-20T16:58:14.678 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-04-20T16:58:14.678 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-04-20T16:58:14.678 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-20T16:58:14.679 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-04-20T16:58:14.679 DEBUG:teuthology.orchestra.run.smithi005:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-20T16:58:14.708 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 13c6e76c-ff37-11ee-bc93-c7b262605968 mon client directed command retry = 5 mon election default strategy = 1 ms bind msgr2 = False ms inject socket failures = 5000 ms type = async osd_pool_default_min_size = 2 osd_pool_default_size = 2 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = True osd debug verify missing on start = True osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore zero block detection = True debug ms = 1 debug osd = 20 osd backoff on peering = True osd blocked scrub grace period = 3600 osd heartbeat use min delay socket = True osd map cache size = 1 osd max backfills = 6 osd max markdown count = 1000 osd mclock override recovery settings = True osd scrub during recovery = False osd scrub min interval = 60 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false auth allow insecure global id reclaim = True debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 mon scrub interval = 300 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.2] debug ms = 1 debug objecter = 20 debug rados = 20 2024-04-20T16:58:14.708 DEBUG:teuthology.orchestra.run.smithi005:mon.a> sudo journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@mon.a.service 2024-04-20T16:58:14.761 DEBUG:teuthology.orchestra.run.smithi005:mgr.y> sudo journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@mgr.y.service 2024-04-20T16:58:14.796 INFO:journalctl@ceph.mon.a.smithi005.stdout:-- Logs begin at Sat 2024-04-20 16:47:38 UTC. -- 2024-04-20T16:58:14.797 INFO:tasks.cephadm:Bootstrapping... 2024-04-20T16:58:14.797 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc -v bootstrap --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.5:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-20T16:58:14.828 INFO:journalctl@ceph.mgr.y.smithi005.stdout:-- Logs begin at Sat 2024-04-20 16:47:38 UTC. -- 2024-04-20T16:58:15.019 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-04-20T16:58:15.020 INFO:teuthology.orchestra.run.smithi005.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc', '-v', 'bootstrap', '--fsid', '13c6e76c-ff37-11ee-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.5:6789]', '--skip-admin-label'] 2024-04-20T16:58:15.020 INFO:teuthology.orchestra.run.smithi005.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-20T16:58:15.020 INFO:teuthology.orchestra.run.smithi005.stdout:Verifying podman|docker is present... 2024-04-20T16:58:15.064 INFO:teuthology.orchestra.run.smithi005.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-20T16:58:15.065 INFO:teuthology.orchestra.run.smithi005.stdout:Verifying lvm2 is present... 2024-04-20T16:58:15.065 INFO:teuthology.orchestra.run.smithi005.stdout:Verifying time synchronization is in place... 2024-04-20T16:58:15.071 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-20T16:58:15.071 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-20T16:58:15.077 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-20T16:58:15.077 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout inactive 2024-04-20T16:58:15.083 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout enabled 2024-04-20T16:58:15.089 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout active 2024-04-20T16:58:15.089 INFO:teuthology.orchestra.run.smithi005.stdout:Unit chronyd.service is enabled and running 2024-04-20T16:58:15.089 INFO:teuthology.orchestra.run.smithi005.stdout:Repeating the final host check... 2024-04-20T16:58:15.116 INFO:teuthology.orchestra.run.smithi005.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-20T16:58:15.116 INFO:teuthology.orchestra.run.smithi005.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-20T16:58:15.116 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl is present 2024-04-20T16:58:15.116 INFO:teuthology.orchestra.run.smithi005.stdout:lvcreate is present 2024-04-20T16:58:15.123 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-20T16:58:15.123 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-20T16:58:15.128 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-20T16:58:15.128 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout inactive 2024-04-20T16:58:15.134 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout enabled 2024-04-20T16:58:15.140 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout active 2024-04-20T16:58:15.140 INFO:teuthology.orchestra.run.smithi005.stdout:Unit chronyd.service is enabled and running 2024-04-20T16:58:15.141 INFO:teuthology.orchestra.run.smithi005.stdout:Host looks OK 2024-04-20T16:58:15.141 INFO:teuthology.orchestra.run.smithi005.stdout:Cluster fsid: 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T16:58:15.141 INFO:teuthology.orchestra.run.smithi005.stdout:Acquiring lock 140543296458368 on /run/cephadm/13c6e76c-ff37-11ee-bc93-c7b262605968.lock 2024-04-20T16:58:15.141 INFO:teuthology.orchestra.run.smithi005.stdout:Lock 140543296458368 acquired on /run/cephadm/13c6e76c-ff37-11ee-bc93-c7b262605968.lock 2024-04-20T16:58:15.141 INFO:teuthology.orchestra.run.smithi005.stdout:Verifying IP 172.21.15.5 port 6789 ... 2024-04-20T16:58:15.142 INFO:teuthology.orchestra.run.smithi005.stdout:Base mon IP(s) is [172.21.15.5:6789], mon addrv is [v1:172.21.15.5:6789] 2024-04-20T16:58:15.145 INFO:teuthology.orchestra.run.smithi005.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.5 metric 100 2024-04-20T16:58:15.146 INFO:teuthology.orchestra.run.smithi005.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.5 metric 100 2024-04-20T16:58:15.149 INFO:teuthology.orchestra.run.smithi005.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-20T16:58:15.149 INFO:teuthology.orchestra.run.smithi005.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-04-20T16:58:15.149 INFO:teuthology.orchestra.run.smithi005.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1799sec hoplimit 64 pref medium 2024-04-20T16:58:15.151 INFO:teuthology.orchestra.run.smithi005.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-20T16:58:15.151 INFO:teuthology.orchestra.run.smithi005.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-20T16:58:15.151 INFO:teuthology.orchestra.run.smithi005.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-20T16:58:15.151 INFO:teuthology.orchestra.run.smithi005.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-20T16:58:15.151 INFO:teuthology.orchestra.run.smithi005.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1442/64 scope link 2024-04-20T16:58:15.152 INFO:teuthology.orchestra.run.smithi005.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-20T16:58:15.152 INFO:teuthology.orchestra.run.smithi005.stdout:Mon IP `172.21.15.5` is in CIDR network `172.21.0.0/20` 2024-04-20T16:58:15.152 INFO:teuthology.orchestra.run.smithi005.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20'] 2024-04-20T16:58:15.153 INFO:teuthology.orchestra.run.smithi005.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-20T16:58:15.154 INFO:teuthology.orchestra.run.smithi005.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc... 2024-04-20T16:58:15.624 INFO:teuthology.orchestra.run.smithi005.stdout:/bin/podman: stdout 3963ae0cc69b2c056e395257d0fd7aa926df0f6f09e4d860cc3407d54ee2f33b 2024-04-20T16:58:15.624 INFO:teuthology.orchestra.run.smithi005.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc... 2024-04-20T16:58:15.624 INFO:teuthology.orchestra.run.smithi005.stdout:/bin/podman: stderr Getting image source signatures 2024-04-20T16:58:15.624 INFO:teuthology.orchestra.run.smithi005.stdout:/bin/podman: stderr Copying blob sha256:426fede9fffa2c7dfe40a4f9322a34794b72405170134d54d2c30d9cc6c6a5be 2024-04-20T16:58:15.624 INFO:teuthology.orchestra.run.smithi005.stdout:/bin/podman: stderr Copying blob sha256:51ab76a66a102d4e065c68d5e2b9705c94cbdae6bd4fc328834a9b887b816856 2024-04-20T16:58:15.624 INFO:teuthology.orchestra.run.smithi005.stdout:/bin/podman: stderr Copying config sha256:3963ae0cc69b2c056e395257d0fd7aa926df0f6f09e4d860cc3407d54ee2f33b 2024-04-20T16:58:15.624 INFO:teuthology.orchestra.run.smithi005.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-20T16:58:16.833 INFO:teuthology.orchestra.run.smithi005.stdout:ceph: stdout ceph version 19.0.0-3160-g36c37156 (36c371567dddacf6207ea36f2535396ab31415fc) squid (dev) 2024-04-20T16:58:16.834 INFO:teuthology.orchestra.run.smithi005.stdout:Ceph version: ceph version 19.0.0-3160-g36c37156 (36c371567dddacf6207ea36f2535396ab31415fc) squid (dev) 2024-04-20T16:58:16.834 INFO:teuthology.orchestra.run.smithi005.stdout:Extracting ceph user uid/gid from container image... 2024-04-20T16:58:17.972 INFO:teuthology.orchestra.run.smithi005.stdout:stat: stdout 167 167 2024-04-20T16:58:17.973 INFO:teuthology.orchestra.run.smithi005.stdout:Creating initial keys... 2024-04-20T16:58:19.240 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph-authtool: stdout AQAq9CNm4j8KGRAAOGiSTShW54j0birqUwgcgg== 2024-04-20T16:58:20.484 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph-authtool: stdout AQAr9CNm5gHCLhAA0drn5rhkmLEUcZdyi76fVw== 2024-04-20T16:58:21.667 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph-authtool: stdout AQAs9CNmz1yTOxAAQ7SOOE6eYRO3iDBTWzNaxw== 2024-04-20T16:58:21.667 INFO:teuthology.orchestra.run.smithi005.stdout:Creating initial monmap... 2024-04-20T16:58:22.836 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-20T16:58:22.836 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-20T16:58:22.836 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T16:58:22.836 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-20T16:58:22.836 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool for a [v1:172.21.15.5:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-20T16:58:22.836 INFO:teuthology.orchestra.run.smithi005.stdout:setting min_mon_release = quincy 2024-04-20T16:58:22.836 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/monmaptool: set fsid to 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T16:58:22.836 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-20T16:58:22.836 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:58:22.837 INFO:teuthology.orchestra.run.smithi005.stdout:Creating mon... 2024-04-20T16:58:24.018 INFO:teuthology.orchestra.run.smithi005.stdout:create mon.a on 2024-04-20T16:58:24.159 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-04-20T16:58:24.254 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-20T16:58:24.398 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-13c6e76c-ff37-11ee-bc93-c7b262605968.target → /etc/systemd/system/ceph-13c6e76c-ff37-11ee-bc93-c7b262605968.target. 2024-04-20T16:58:24.398 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-13c6e76c-ff37-11ee-bc93-c7b262605968.target → /etc/systemd/system/ceph-13c6e76c-ff37-11ee-bc93-c7b262605968.target. 2024-04-20T16:58:24.582 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@mon.a 2024-04-20T16:58:24.583 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stderr Failed to reset failed state of unit ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@mon.a.service: Unit ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@mon.a.service not loaded. 2024-04-20T16:58:24.681 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-13c6e76c-ff37-11ee-bc93-c7b262605968.target.wants/ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@.service. 2024-04-20T16:58:25.323 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-20T16:58:25.324 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout disabled 2024-04-20T16:58:25.336 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-20T16:58:25.336 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout inactive 2024-04-20T16:58:25.336 INFO:teuthology.orchestra.run.smithi005.stdout:firewalld.service is not enabled 2024-04-20T16:58:25.336 INFO:teuthology.orchestra.run.smithi005.stdout:Not possible to enable service . firewalld.service is not available 2024-04-20T16:58:25.336 INFO:teuthology.orchestra.run.smithi005.stdout:Waiting for mon to start... 2024-04-20T16:58:25.337 INFO:teuthology.orchestra.run.smithi005.stdout:Waiting for mon... 2024-04-20T16:58:25.597 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:25 smithi005 bash[39040]: 365d7008ec5a0827ed1cb88205d438b053f14fa619f9672b5c04da81df0a4068 2024-04-20T16:58:25.597 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:25 smithi005 systemd[1]: Started Ceph mon.a for 13c6e76c-ff37-11ee-bc93-c7b262605968. 2024-04-20T16:58:26.620 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout cluster: 2024-04-20T16:58:26.620 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout id: 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T16:58:26.620 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-20T16:58:26.621 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 2024-04-20T16:58:26.621 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout services: 2024-04-20T16:58:26.621 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.722161s) 2024-04-20T16:58:26.621 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-20T16:58:26.621 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-20T16:58:26.621 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 2024-04-20T16:58:26.621 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout data: 2024-04-20T16:58:26.621 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-20T16:58:26.621 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-20T16:58:26.621 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-20T16:58:26.621 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout pgs: 2024-04-20T16:58:26.621 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 2024-04-20T16:58:26.621 INFO:teuthology.orchestra.run.smithi005.stdout:mon is available 2024-04-20T16:58:26.621 INFO:teuthology.orchestra.run.smithi005.stdout:Assimilating anything we can from ceph.conf... 2024-04-20T16:58:26.622 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:26 smithi005 ceph-mon[39078]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-20T16:58:26.622 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:26 smithi005 ceph-mon[39078]: monmap epoch 1 2024-04-20T16:58:26.622 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:26 smithi005 ceph-mon[39078]: fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T16:58:26.622 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:26 smithi005 ceph-mon[39078]: last_changed 2024-04-20T16:58:22.127736+0000 2024-04-20T16:58:26.622 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:26 smithi005 ceph-mon[39078]: created 2024-04-20T16:58:22.127736+0000 2024-04-20T16:58:26.622 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:26 smithi005 ceph-mon[39078]: min_mon_release 19 (squid) 2024-04-20T16:58:26.622 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:26 smithi005 ceph-mon[39078]: election_strategy: 1 2024-04-20T16:58:26.622 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:26 smithi005 ceph-mon[39078]: 0: v1:172.21.15.5:6789/0 mon.a 2024-04-20T16:58:26.622 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:26 smithi005 ceph-mon[39078]: fsmap 2024-04-20T16:58:26.622 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:26 smithi005 ceph-mon[39078]: osdmap e1: 0 total, 0 up, 0 in 2024-04-20T16:58:26.622 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:26 smithi005 ceph-mon[39078]: mgrmap e1: no daemons active 2024-04-20T16:58:26.623 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:26 smithi005 ceph-mon[39078]: from='client.? v1:172.21.15.5:0/845916349' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-20T16:58:27.523 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:27 smithi005 ceph-mon[39078]: from='client.? v1:172.21.15.5:0/3453660584' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-20T16:58:27.523 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:27 smithi005 ceph-mon[39078]: from='client.? v1:172.21.15.5:0/3453660584' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-20T16:58:27.819 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 2024-04-20T16:58:27.819 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout [global] 2024-04-20T16:58:27.819 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout fsid = 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T16:58:27.819 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-20T16:58:27.819 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.5:6789] 2024-04-20T16:58:27.820 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-20T16:58:27.820 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-20T16:58:27.820 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-20T16:58:27.820 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-20T16:58:27.820 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 2024-04-20T16:58:27.820 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-20T16:58:27.820 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-20T16:58:27.820 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 2024-04-20T16:58:27.820 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout [osd] 2024-04-20T16:58:27.820 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-20T16:58:27.820 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-20T16:58:27.820 INFO:teuthology.orchestra.run.smithi005.stdout:Generating new minimal ceph.conf... 2024-04-20T16:58:28.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:28 smithi005 ceph-mon[39078]: from='client.? v1:172.21.15.5:0/2991743266' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T16:58:29.050 INFO:teuthology.orchestra.run.smithi005.stdout:Restarting the monitor... 2024-04-20T16:58:29.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:29 smithi005 systemd[1]: Stopping Ceph mon.a for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T16:58:29.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:29 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mon-a[39055]: 2024-04-20T16:58:29.288+0000 7f9352852700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T16:58:29.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:29 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mon-a[39055]: 2024-04-20T16:58:29.288+0000 7f9352852700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-20T16:58:29.926 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:29 smithi005 bash[39404]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mon-a 2024-04-20T16:58:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 systemd[1]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@mon.a.service: Succeeded. 2024-04-20T16:58:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 systemd[1]: Stopped Ceph mon.a for 13c6e76c-ff37-11ee-bc93-c7b262605968. 2024-04-20T16:58:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 systemd[1]: Starting Ceph mon.a for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T16:58:30.699 INFO:teuthology.orchestra.run.smithi005.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-20T16:58:30.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: set uid:gid to 167:167 (ceph:ceph) 2024-04-20T16:58:30.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: ceph version 19.0.0-3160-g36c37156 (36c371567dddacf6207ea36f2535396ab31415fc) squid (dev), process ceph-mon, pid 7 2024-04-20T16:58:30.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: pidfile_write: ignore empty --pid-file 2024-04-20T16:58:30.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: load: jerasure load: lrc 2024-04-20T16:58:30.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: RocksDB version: 7.9.2 2024-04-20T16:58:30.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Git sha 0 2024-04-20T16:58:30.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Compile date 2024-04-20 01:00:17 2024-04-20T16:58:30.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: DB SUMMARY 2024-04-20T16:58:30.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: DB Session ID: 5ZF0UINAFVGR4J4HH19M 2024-04-20T16:58:30.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: CURRENT file: CURRENT 2024-04-20T16:58:30.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: IDENTITY file: IDENTITY 2024-04-20T16:58:30.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-20T16:58:30.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-20T16:58:30.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 98256 ; 2024-04-20T16:58:30.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.error_if_exists: 0 2024-04-20T16:58:30.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.create_if_missing: 0 2024-04-20T16:58:30.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.paranoid_checks: 1 2024-04-20T16:58:30.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-20T16:58:30.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-20T16:58:30.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-20T16:58:30.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.env: 0x5643579b0cc0 2024-04-20T16:58:30.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.fs: PosixFileSystem 2024-04-20T16:58:30.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.info_log: 0x564358f27c40 2024-04-20T16:58:30.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.max_file_opening_threads: 16 2024-04-20T16:58:30.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.statistics: (nil) 2024-04-20T16:58:30.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.use_fsync: 0 2024-04-20T16:58:30.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.max_log_file_size: 0 2024-04-20T16:58:30.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-20T16:58:30.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-20T16:58:30.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.keep_log_file_num: 1000 2024-04-20T16:58:30.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.recycle_log_file_num: 0 2024-04-20T16:58:30.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.allow_fallocate: 1 2024-04-20T16:58:30.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.allow_mmap_reads: 0 2024-04-20T16:58:30.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.allow_mmap_writes: 0 2024-04-20T16:58:30.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.use_direct_reads: 0 2024-04-20T16:58:30.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-20T16:58:30.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.create_missing_column_families: 0 2024-04-20T16:58:30.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.db_log_dir: 2024-04-20T16:58:30.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.wal_dir: 2024-04-20T16:58:30.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-20T16:58:30.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-20T16:58:30.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-20T16:58:30.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-20T16:58:30.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-20T16:58:30.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-20T16:58:30.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.advise_random_on_open: 1 2024-04-20T16:58:30.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.db_write_buffer_size: 0 2024-04-20T16:58:30.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.write_buffer_manager: 0x564358ebb0e0 2024-04-20T16:58:30.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-20T16:58:30.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-20T16:58:30.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-20T16:58:30.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.rate_limiter: (nil) 2024-04-20T16:58:30.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-20T16:58:30.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.wal_recovery_mode: 2 2024-04-20T16:58:30.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.enable_thread_tracking: 0 2024-04-20T16:58:30.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.enable_pipelined_write: 0 2024-04-20T16:58:30.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.unordered_write: 0 2024-04-20T16:58:30.967 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-20T16:58:30.967 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-20T16:58:30.967 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-20T16:58:30.967 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-20T16:58:30.967 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.row_cache: None 2024-04-20T16:58:30.967 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.wal_filter: None 2024-04-20T16:58:30.967 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-20T16:58:30.967 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.allow_ingest_behind: 0 2024-04-20T16:58:30.967 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.two_write_queues: 0 2024-04-20T16:58:30.967 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.manual_wal_flush: 0 2024-04-20T16:58:30.967 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.wal_compression: 0 2024-04-20T16:58:30.967 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.atomic_flush: 0 2024-04-20T16:58:30.967 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-20T16:58:30.968 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-20T16:58:30.968 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-20T16:58:30.968 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.log_readahead_size: 0 2024-04-20T16:58:30.968 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-20T16:58:30.968 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.best_efforts_recovery: 0 2024-04-20T16:58:30.968 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-20T16:58:30.968 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-20T16:58:30.968 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.allow_data_in_errors: 0 2024-04-20T16:58:30.968 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.db_host_id: __hostname__ 2024-04-20T16:58:30.968 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-20T16:58:30.968 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.max_background_jobs: 2 2024-04-20T16:58:30.968 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.max_background_compactions: -1 2024-04-20T16:58:30.968 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.max_subcompactions: 1 2024-04-20T16:58:30.969 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-20T16:58:30.969 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-20T16:58:30.969 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-20T16:58:30.969 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.max_total_wal_size: 0 2024-04-20T16:58:30.969 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-20T16:58:30.969 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-20T16:58:30.969 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-20T16:58:30.969 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-20T16:58:30.969 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.max_open_files: -1 2024-04-20T16:58:30.969 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.bytes_per_sync: 0 2024-04-20T16:58:30.969 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-20T16:58:30.969 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-20T16:58:30.969 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.compaction_readahead_size: 0 2024-04-20T16:58:30.969 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.max_background_flushes: -1 2024-04-20T16:58:30.969 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Compression algorithms supported: 2024-04-20T16:58:30.970 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-20T16:58:30.970 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: kZSTD supported: 0 2024-04-20T16:58:30.970 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: kXpressCompression supported: 0 2024-04-20T16:58:30.970 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: kLZ4HCCompression supported: 1 2024-04-20T16:58:30.970 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: kZlibCompression supported: 1 2024-04-20T16:58:30.970 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: kSnappyCompression supported: 1 2024-04-20T16:58:30.970 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: kLZ4Compression supported: 1 2024-04-20T16:58:30.970 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: kBZip2Compression supported: 0 2024-04-20T16:58:30.970 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-20T16:58:30.970 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-20T16:58:30.970 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-20T16:58:30.970 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-20T16:58:30.970 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-20T16:58:30.970 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.merge_operator: 2024-04-20T16:58:30.971 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.compaction_filter: None 2024-04-20T16:58:30.971 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.compaction_filter_factory: None 2024-04-20T16:58:30.971 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.sst_partitioner_factory: None 2024-04-20T16:58:30.971 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-20T16:58:30.971 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-20T16:58:30.971 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564358f27e00) 2024-04-20T16:58:30.971 INFO:journalctl@ceph.mon.a.smithi005.stdout: cache_index_and_filter_blocks: 1 2024-04-20T16:58:30.971 INFO:journalctl@ceph.mon.a.smithi005.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-20T16:58:30.971 INFO:journalctl@ceph.mon.a.smithi005.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-20T16:58:30.971 INFO:journalctl@ceph.mon.a.smithi005.stdout: pin_top_level_index_and_filter: 1 2024-04-20T16:58:30.971 INFO:journalctl@ceph.mon.a.smithi005.stdout: index_type: 0 2024-04-20T16:58:30.971 INFO:journalctl@ceph.mon.a.smithi005.stdout: data_block_index_type: 0 2024-04-20T16:58:30.971 INFO:journalctl@ceph.mon.a.smithi005.stdout: index_shortening: 1 2024-04-20T16:58:30.972 INFO:journalctl@ceph.mon.a.smithi005.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-20T16:58:30.972 INFO:journalctl@ceph.mon.a.smithi005.stdout: checksum: 4 2024-04-20T16:58:30.972 INFO:journalctl@ceph.mon.a.smithi005.stdout: no_block_cache: 0 2024-04-20T16:58:30.972 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_cache: 0x564358eed1f0 2024-04-20T16:58:30.972 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_cache_name: BinnedLRUCache 2024-04-20T16:58:30.972 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_cache_options: 2024-04-20T16:58:30.972 INFO:journalctl@ceph.mon.a.smithi005.stdout: capacity : 536870912 2024-04-20T16:58:30.972 INFO:journalctl@ceph.mon.a.smithi005.stdout: num_shard_bits : 4 2024-04-20T16:58:30.972 INFO:journalctl@ceph.mon.a.smithi005.stdout: strict_capacity_limit : 0 2024-04-20T16:58:30.972 INFO:journalctl@ceph.mon.a.smithi005.stdout: high_pri_pool_ratio: 0.000 2024-04-20T16:58:30.972 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_cache_compressed: (nil) 2024-04-20T16:58:30.972 INFO:journalctl@ceph.mon.a.smithi005.stdout: persistent_cache: (nil) 2024-04-20T16:58:30.972 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_size: 4096 2024-04-20T16:58:30.973 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_size_deviation: 10 2024-04-20T16:58:30.973 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_restart_interval: 16 2024-04-20T16:58:30.973 INFO:journalctl@ceph.mon.a.smithi005.stdout: index_block_restart_interval: 1 2024-04-20T16:58:30.973 INFO:journalctl@ceph.mon.a.smithi005.stdout: metadata_block_size: 4096 2024-04-20T16:58:30.973 INFO:journalctl@ceph.mon.a.smithi005.stdout: partition_filters: 0 2024-04-20T16:58:30.973 INFO:journalctl@ceph.mon.a.smithi005.stdout: use_delta_encoding: 1 2024-04-20T16:58:30.973 INFO:journalctl@ceph.mon.a.smithi005.stdout: filter_policy: bloomfilter 2024-04-20T16:58:30.973 INFO:journalctl@ceph.mon.a.smithi005.stdout: whole_key_filtering: 1 2024-04-20T16:58:30.973 INFO:journalctl@ceph.mon.a.smithi005.stdout: verify_compression: 0 2024-04-20T16:58:30.973 INFO:journalctl@ceph.mon.a.smithi005.stdout: read_amp_bytes_per_bit: 0 2024-04-20T16:58:30.973 INFO:journalctl@ceph.mon.a.smithi005.stdout: format_version: 5 2024-04-20T16:58:30.973 INFO:journalctl@ceph.mon.a.smithi005.stdout: enable_index_compression: 1 2024-04-20T16:58:30.973 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_align: 0 2024-04-20T16:58:30.973 INFO:journalctl@ceph.mon.a.smithi005.stdout: max_auto_readahead_size: 262144 2024-04-20T16:58:30.973 INFO:journalctl@ceph.mon.a.smithi005.stdout: prepopulate_block_cache: 0 2024-04-20T16:58:30.973 INFO:journalctl@ceph.mon.a.smithi005.stdout: initial_auto_readahead_size: 8192 2024-04-20T16:58:30.973 INFO:journalctl@ceph.mon.a.smithi005.stdout: num_file_reads_for_auto_readahead: 2 2024-04-20T16:58:30.974 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.write_buffer_size: 33554432 2024-04-20T16:58:30.974 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.max_write_buffer_number: 2 2024-04-20T16:58:30.974 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.compression: NoCompression 2024-04-20T16:58:30.974 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.bottommost_compression: Disabled 2024-04-20T16:58:30.974 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.prefix_extractor: nullptr 2024-04-20T16:58:30.974 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-20T16:58:30.974 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.num_levels: 7 2024-04-20T16:58:30.974 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-20T16:58:30.974 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-20T16:58:30.974 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-20T16:58:30.974 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-20T16:58:30.974 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-20T16:58:30.974 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-20T16:58:30.974 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-20T16:58:30.975 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-20T16:58:30.975 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-20T16:58:30.975 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-20T16:58:30.975 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-20T16:58:30.975 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-20T16:58:30.975 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-20T16:58:30.975 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.compression_opts.level: 32767 2024-04-20T16:58:30.975 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.compression_opts.strategy: 0 2024-04-20T16:58:30.975 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-20T16:58:30.975 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-20T16:58:30.975 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-20T16:58:30.975 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-20T16:58:30.975 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.compression_opts.enabled: false 2024-04-20T16:58:30.975 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-20T16:58:30.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-20T16:58:30.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-20T16:58:30.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-20T16:58:30.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.target_file_size_base: 67108864 2024-04-20T16:58:30.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-20T16:58:30.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-20T16:58:30.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-20T16:58:30.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-20T16:58:30.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-20T16:58:30.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-20T16:58:30.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-20T16:58:30.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-20T16:58:30.977 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-20T16:58:30.977 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-20T16:58:30.977 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-20T16:58:30.977 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-20T16:58:30.977 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-20T16:58:30.977 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-20T16:58:30.977 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.arena_block_size: 1048576 2024-04-20T16:58:30.977 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-20T16:58:30.977 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-20T16:58:30.977 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.disable_auto_compactions: 0 2024-04-20T16:58:30.977 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-20T16:58:30.977 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-20T16:58:30.977 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-20T16:58:30.978 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-20T16:58:30.978 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-20T16:58:30.978 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-20T16:58:30.979 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-20T16:58:30.979 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-20T16:58:30.979 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-20T16:58:30.979 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-20T16:58:30.979 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-20T16:58:30.979 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.inplace_update_support: 0 2024-04-20T16:58:30.979 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-20T16:58:30.979 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-20T16:58:30.979 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-20T16:58:30.979 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-20T16:58:30.979 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.bloom_locality: 0 2024-04-20T16:58:30.979 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.max_successive_merges: 0 2024-04-20T16:58:30.979 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-20T16:58:30.980 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.paranoid_file_checks: 0 2024-04-20T16:58:30.980 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.force_consistency_checks: 1 2024-04-20T16:58:30.980 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.report_bg_io_stats: 0 2024-04-20T16:58:30.980 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.ttl: 2592000 2024-04-20T16:58:30.980 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-20T16:58:30.980 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-20T16:58:30.980 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-20T16:58:30.980 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.enable_blob_files: false 2024-04-20T16:58:30.980 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.min_blob_size: 0 2024-04-20T16:58:30.980 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.blob_file_size: 268435456 2024-04-20T16:58:30.980 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-20T16:58:30.980 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-20T16:58:30.980 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-20T16:58:30.980 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-20T16:58:30.981 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-20T16:58:30.981 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.blob_file_starting_level: 0 2024-04-20T16:58:30.981 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-20T16:58:30.981 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-20T16:58:30.981 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-20T16:58:30.981 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8ec926f3-4790-484f-95a8-ef587206cc7b 2024-04-20T16:58:30.981 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713632310662573, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-20T16:58:30.981 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-20T16:58:30.981 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713632310664173, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93642, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 306, "table_properties": {"data_size": 91643, "index_size": 255, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13221, "raw_average_key_size": 49, "raw_value_size": 84454, "raw_average_value_size": 318, "num_data_blocks": 11, "num_entries": 265, "num_filter_entries": 265, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713632310, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8ec926f3-4790-484f-95a8-ef587206cc7b", "db_session_id": "5ZF0UINAFVGR4J4HH19M", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-20T16:58:30.981 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713632310664295, "job": 1, "event": "recovery_finished"} 2024-04-20T16:58:30.981 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-20T16:58:30.981 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-20T16:58:30.981 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564359002000 2024-04-20T16:58:30.982 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: DB pointer 0x564358fec000 2024-04-20T16:58:30.982 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-20T16:58:30.982 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-20T16:58:30.982 INFO:journalctl@ceph.mon.a.smithi005.stdout: ** DB Stats ** 2024-04-20T16:58:30.982 INFO:journalctl@ceph.mon.a.smithi005.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-20T16:58:30.982 INFO:journalctl@ceph.mon.a.smithi005.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-20T16:58:30.982 INFO:journalctl@ceph.mon.a.smithi005.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-20T16:58:30.982 INFO:journalctl@ceph.mon.a.smithi005.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-20T16:58:30.982 INFO:journalctl@ceph.mon.a.smithi005.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-20T16:58:30.982 INFO:journalctl@ceph.mon.a.smithi005.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-20T16:58:30.982 INFO:journalctl@ceph.mon.a.smithi005.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-20T16:58:30.982 INFO:journalctl@ceph.mon.a.smithi005.stdout: 2024-04-20T16:58:30.983 INFO:journalctl@ceph.mon.a.smithi005.stdout: ** Compaction Stats [default] ** 2024-04-20T16:58:30.983 INFO:journalctl@ceph.mon.a.smithi005.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-20T16:58:30.983 INFO:journalctl@ceph.mon.a.smithi005.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-20T16:58:30.983 INFO:journalctl@ceph.mon.a.smithi005.stdout: L0 2/0 93.27 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 94.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-20T16:58:30.983 INFO:journalctl@ceph.mon.a.smithi005.stdout: Sum 2/0 93.27 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 94.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-20T16:58:30.983 INFO:journalctl@ceph.mon.a.smithi005.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 94.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-20T16:58:30.983 INFO:journalctl@ceph.mon.a.smithi005.stdout: 2024-04-20T16:58:30.983 INFO:journalctl@ceph.mon.a.smithi005.stdout: ** Compaction Stats [default] ** 2024-04-20T16:58:30.983 INFO:journalctl@ceph.mon.a.smithi005.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-20T16:58:30.983 INFO:journalctl@ceph.mon.a.smithi005.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-20T16:58:30.983 INFO:journalctl@ceph.mon.a.smithi005.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 94.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-20T16:58:30.983 INFO:journalctl@ceph.mon.a.smithi005.stdout: 2024-04-20T16:58:30.984 INFO:journalctl@ceph.mon.a.smithi005.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-20T16:58:30.984 INFO:journalctl@ceph.mon.a.smithi005.stdout: 2024-04-20T16:58:30.984 INFO:journalctl@ceph.mon.a.smithi005.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-20T16:58:30.984 INFO:journalctl@ceph.mon.a.smithi005.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-20T16:58:30.984 INFO:journalctl@ceph.mon.a.smithi005.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-20T16:58:30.984 INFO:journalctl@ceph.mon.a.smithi005.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-20T16:58:30.984 INFO:journalctl@ceph.mon.a.smithi005.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-20T16:58:30.984 INFO:journalctl@ceph.mon.a.smithi005.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-20T16:58:30.984 INFO:journalctl@ceph.mon.a.smithi005.stdout: Cumulative compaction: 0.00 GB write, 21.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-20T16:58:30.984 INFO:journalctl@ceph.mon.a.smithi005.stdout: Interval compaction: 0.00 GB write, 21.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-20T16:58:30.984 INFO:journalctl@ceph.mon.a.smithi005.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-20T16:58:30.984 INFO:journalctl@ceph.mon.a.smithi005.stdout: Block cache BinnedLRUCache@0x564358eed1f0#7 capacity: 512.00 MB usage: 1.33 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-20T16:58:30.985 INFO:journalctl@ceph.mon.a.smithi005.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.44 KB,8.34465e-05%) Misc(1,0.00 KB,0%) 2024-04-20T16:58:30.985 INFO:journalctl@ceph.mon.a.smithi005.stdout: 2024-04-20T16:58:30.985 INFO:journalctl@ceph.mon.a.smithi005.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-20T16:58:30.985 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: starting mon.a rank 0 at public addrs v1:172.21.15.5:6789/0 at bind addrs v1:172.21.15.5:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T16:58:30.985 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: mon.a@-1(???) e1 preinit fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T16:58:30.985 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: mon.a@-1(???).mds e1 new map 2024-04-20T16:58:30.985 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: mon.a@-1(???).mds e1 print_map 2024-04-20T16:58:30.985 INFO:journalctl@ceph.mon.a.smithi005.stdout: e1 2024-04-20T16:58:30.985 INFO:journalctl@ceph.mon.a.smithi005.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T16:58:30.985 INFO:journalctl@ceph.mon.a.smithi005.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-20T16:58:30.985 INFO:journalctl@ceph.mon.a.smithi005.stdout: legacy client fscid: -1 2024-04-20T16:58:30.985 INFO:journalctl@ceph.mon.a.smithi005.stdout: 2024-04-20T16:58:30.986 INFO:journalctl@ceph.mon.a.smithi005.stdout: No filesystems configured 2024-04-20T16:58:30.986 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-20T16:58:30.986 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-20T16:58:30.986 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-20T16:58:30.986 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-20T16:58:30.986 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-20T16:58:30.986 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 bash[39524]: 5b1da09d4b2a469c556e41dea7f39df813a6a6b5b639243059de42168bd88c1f 2024-04-20T16:58:30.986 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 systemd[1]: Started Ceph mon.a for 13c6e76c-ff37-11ee-bc93-c7b262605968. 2024-04-20T16:58:30.986 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-20T16:58:30.986 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: monmap epoch 1 2024-04-20T16:58:30.986 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T16:58:30.986 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: last_changed 2024-04-20T16:58:22.127736+0000 2024-04-20T16:58:30.986 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: created 2024-04-20T16:58:22.127736+0000 2024-04-20T16:58:30.986 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: min_mon_release 19 (squid) 2024-04-20T16:58:30.987 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: election_strategy: 1 2024-04-20T16:58:30.987 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: 0: v1:172.21.15.5:6789/0 mon.a 2024-04-20T16:58:30.987 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: fsmap 2024-04-20T16:58:30.987 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: osdmap e1: 0 total, 0 up, 0 in 2024-04-20T16:58:30.987 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:30 smithi005 ceph-mon[39562]: mgrmap e1: no daemons active 2024-04-20T16:58:31.945 INFO:teuthology.orchestra.run.smithi005.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-20T16:58:31.968 INFO:teuthology.orchestra.run.smithi005.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-20T16:58:31.968 INFO:teuthology.orchestra.run.smithi005.stdout:Creating mgr... 2024-04-20T16:58:31.969 INFO:teuthology.orchestra.run.smithi005.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-20T16:58:31.969 INFO:teuthology.orchestra.run.smithi005.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-20T16:58:32.185 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@mgr.y 2024-04-20T16:58:32.185 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stderr Failed to reset failed state of unit ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@mgr.y.service: Unit ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@mgr.y.service not loaded. 2024-04-20T16:58:32.282 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-13c6e76c-ff37-11ee-bc93-c7b262605968.target.wants/ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@.service. 2024-04-20T16:58:32.392 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:32 smithi005 systemd[1]: Starting Ceph mgr.y for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T16:58:32.654 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:32 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2937962111' entity='client.admin' 2024-04-20T16:58:32.945 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-20T16:58:32.945 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout disabled 2024-04-20T16:58:32.954 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-20T16:58:32.954 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout inactive 2024-04-20T16:58:32.954 INFO:teuthology.orchestra.run.smithi005.stdout:firewalld.service is not enabled 2024-04-20T16:58:32.954 INFO:teuthology.orchestra.run.smithi005.stdout:Not possible to enable service . firewalld.service is not available 2024-04-20T16:58:32.962 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-20T16:58:32.962 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout disabled 2024-04-20T16:58:32.972 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-20T16:58:32.972 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout inactive 2024-04-20T16:58:32.972 INFO:teuthology.orchestra.run.smithi005.stdout:firewalld.service is not enabled 2024-04-20T16:58:32.972 INFO:teuthology.orchestra.run.smithi005.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-20T16:58:32.972 INFO:teuthology.orchestra.run.smithi005.stdout:Waiting for mgr to start... 2024-04-20T16:58:32.972 INFO:teuthology.orchestra.run.smithi005.stdout:Waiting for mgr... 2024-04-20T16:58:33.211 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:32 smithi005 bash[39796]: c07185bec3d43852395a84aec43c36f1801947848a3c62aebd8490d3b20a7b1c 2024-04-20T16:58:33.212 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:32 smithi005 systemd[1]: Started Ceph mgr.y for 13c6e76c-ff37-11ee-bc93-c7b262605968. 2024-04-20T16:58:33.212 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:32 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:32.969+0000 7fe80a367200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-20T16:58:33.212 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:33 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:33.159+0000 7fe80a367200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-20T16:58:33.869 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:33 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:33.499+0000 7fe80a367200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-20T16:58:34.211 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:34 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:34.030+0000 7fe80a367200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-20T16:58:34.711 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:34 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:34.287+0000 7fe80a367200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-20T16:58:35.211 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:34 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:34.738+0000 7fe80a367200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-20T16:58:35.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:35 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1801434663' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-20T16:58:35.962 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:35 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:35.500+0000 7fe80a367200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-20T16:58:35.962 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:35 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:35.592+0000 7fe80a367200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-20T16:58:36.640 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:36 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:36.222+0000 7fe80a367200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-20T16:58:36.640 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:36 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:36.311+0000 7fe80a367200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-20T16:58:36.640 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:36 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:36.404+0000 7fe80a367200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-20T16:58:36.961 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:36 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:36.639+0000 7fe80a367200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-20T16:58:36.961 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:36 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:36.734+0000 7fe80a367200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-20T16:58:36.961 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:36 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:36.831+0000 7fe80a367200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-20T16:58:37.369 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:37 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:37.073+0000 7fe80a367200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-20T16:58:37.370 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:37 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:37.174+0000 7fe80a367200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-20T16:58:37.632 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:37 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:37.369+0000 7fe80a367200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-20T16:58:37.961 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:37 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:37.631+0000 7fe80a367200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-20T16:58:38.577 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:38 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:38.243+0000 7fe80a367200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-20T16:58:38.578 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:38 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:38.465+0000 7fe80a367200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-20T16:58:38.961 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:38 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:38.670+0000 7fe80a367200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-20T16:58:39.711 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:39 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:39.364+0000 7fe80a367200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-20T16:58:39.711 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:39 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:39.571+0000 7fe80a367200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-20T16:58:40.211 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:39 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:39.761+0000 7fe80a367200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-20T16:58:40.211 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:39 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:39.882+0000 7fe80a367200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-20T16:58:40.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:39 smithi005 ceph-mon[39562]: Activating manager daemon y 2024-04-20T16:58:40.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:39 smithi005 ceph-mon[39562]: mgrmap e2: y(active, starting, since 0.00318807s) 2024-04-20T16:58:40.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:39 smithi005 ceph-mon[39562]: from='mgr.14100 v1:172.21.15.5:0/1910016847' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-20T16:58:40.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:39 smithi005 ceph-mon[39562]: from='mgr.14100 v1:172.21.15.5:0/1910016847' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-20T16:58:40.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:39 smithi005 ceph-mon[39562]: from='mgr.14100 v1:172.21.15.5:0/1910016847' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-20T16:58:40.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:39 smithi005 ceph-mon[39562]: from='mgr.14100 v1:172.21.15.5:0/1910016847' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-20T16:58:40.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:39 smithi005 ceph-mon[39562]: from='mgr.14100 v1:172.21.15.5:0/1910016847' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-20T16:58:40.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:39 smithi005 ceph-mon[39562]: from='mgr.14100 v1:172.21.15.5:0/1910016847' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-20T16:58:40.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:39 smithi005 ceph-mon[39562]: from='mgr.14100 v1:172.21.15.5:0/1910016847' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T16:58:40.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:39 smithi005 ceph-mon[39562]: from='mgr.14100 v1:172.21.15.5:0/1910016847' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-20T16:58:40.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:39 smithi005 ceph-mon[39562]: Manager daemon y is now available 2024-04-20T16:58:40.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:39 smithi005 ceph-mon[39562]: from='mgr.14100 v1:172.21.15.5:0/1910016847' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-20T16:58:40.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:39 smithi005 ceph-mon[39562]: from='mgr.14100 v1:172.21.15.5:0/1910016847' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-20T16:58:40.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:39 smithi005 ceph-mon[39562]: from='mgr.14100 v1:172.21.15.5:0/1910016847' entity='mgr.y' 2024-04-20T16:58:40.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:39 smithi005 ceph-mon[39562]: from='mgr.14100 v1:172.21.15.5:0/1910016847' entity='mgr.y' 2024-04-20T16:58:40.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:39 smithi005 ceph-mon[39562]: from='mgr.14100 v1:172.21.15.5:0/1910016847' entity='mgr.y' 2024-04-20T16:58:40.813 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 2024-04-20T16:58:40.819 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout { 2024-04-20T16:58:40.819 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "fsid": "13c6e76c-ff37-11ee-bc93-c7b262605968", 2024-04-20T16:58:40.819 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "health": { 2024-04-20T16:58:40.819 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-20T16:58:40.819 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-20T16:58:40.819 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-20T16:58:40.819 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-04-20T16:58:40.819 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-20T16:58:40.820 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-20T16:58:40.820 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 0 2024-04-20T16:58:40.820 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout ], 2024-04-20T16:58:40.820 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-20T16:58:40.820 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "a" 2024-04-20T16:58:40.820 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout ], 2024-04-20T16:58:40.820 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "quorum_age": 4, 2024-04-20T16:58:40.820 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-20T16:58:40.820 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-20T16:58:40.820 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-20T16:58:40.820 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-20T16:58:40.820 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-04-20T16:58:40.820 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-20T16:58:40.820 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-20T16:58:40.820 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-20T16:58:40.821 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-20T16:58:40.821 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-20T16:58:40.821 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-20T16:58:40.821 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-20T16:58:40.821 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-20T16:58:40.821 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-04-20T16:58:40.821 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-20T16:58:40.821 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-20T16:58:40.821 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-20T16:58:40.821 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-20T16:58:40.821 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-20T16:58:40.821 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-20T16:58:40.821 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-20T16:58:40.821 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-20T16:58:40.822 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-20T16:58:40.822 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-04-20T16:58:40.822 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-20T16:58:40.822 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-20T16:58:40.822 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-20T16:58:40.822 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-20T16:58:40.822 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-04-20T16:58:40.822 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-20T16:58:40.822 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-20T16:58:40.822 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-20T16:58:40.822 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-20T16:58:40.822 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-20T16:58:40.822 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-20T16:58:40.822 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "restful" 2024-04-20T16:58:40.822 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout ], 2024-04-20T16:58:40.822 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-20T16:58:40.823 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-04-20T16:58:40.823 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-20T16:58:40.823 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-20T16:58:40.823 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "modified": "2024-04-20T16:58:25.273052+0000", 2024-04-20T16:58:40.823 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-20T16:58:40.823 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-04-20T16:58:40.823 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-20T16:58:40.823 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout } 2024-04-20T16:58:40.823 INFO:teuthology.orchestra.run.smithi005.stdout:mgr not available, waiting (1/15)... 2024-04-20T16:58:42.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:41 smithi005 ceph-mon[39562]: mgrmap e3: y(active, since 1.00724s) 2024-04-20T16:58:44.191 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:43 smithi005 ceph-mon[39562]: mgrmap e4: y(active, since 3s) 2024-04-20T16:58:45.198 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:44 smithi005 ceph-mon[39562]: from='client.14110 v1:172.21.15.5:0/61160406' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-20T16:58:45.470 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 2024-04-20T16:58:45.471 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout { 2024-04-20T16:58:45.471 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "fsid": "13c6e76c-ff37-11ee-bc93-c7b262605968", 2024-04-20T16:58:45.471 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "health": { 2024-04-20T16:58:45.471 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-20T16:58:45.471 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-20T16:58:45.471 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-20T16:58:45.471 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-04-20T16:58:45.471 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-20T16:58:45.471 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-20T16:58:45.471 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 0 2024-04-20T16:58:45.471 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout ], 2024-04-20T16:58:45.472 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-20T16:58:45.472 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "a" 2024-04-20T16:58:45.472 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout ], 2024-04-20T16:58:45.472 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-04-20T16:58:45.472 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-20T16:58:45.472 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-20T16:58:45.472 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-20T16:58:45.472 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-20T16:58:45.472 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-04-20T16:58:45.472 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-20T16:58:45.472 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-20T16:58:45.472 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-20T16:58:45.473 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-20T16:58:45.473 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-20T16:58:45.473 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-20T16:58:45.473 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-20T16:58:45.473 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-20T16:58:45.476 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-04-20T16:58:45.476 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-20T16:58:45.476 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-20T16:58:45.476 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-20T16:58:45.476 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-20T16:58:45.476 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-20T16:58:45.477 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-20T16:58:45.477 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-20T16:58:45.477 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-20T16:58:45.477 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-20T16:58:45.477 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-04-20T16:58:45.477 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-20T16:58:45.477 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-20T16:58:45.477 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-20T16:58:45.477 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-20T16:58:45.477 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-04-20T16:58:45.477 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-20T16:58:45.477 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-20T16:58:45.477 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-20T16:58:45.477 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-20T16:58:45.478 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-20T16:58:45.478 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-20T16:58:45.478 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "restful" 2024-04-20T16:58:45.478 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout ], 2024-04-20T16:58:45.478 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-20T16:58:45.478 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-04-20T16:58:45.478 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-20T16:58:45.478 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-20T16:58:45.478 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "modified": "2024-04-20T16:58:25.273052+0000", 2024-04-20T16:58:45.478 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-20T16:58:45.478 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-04-20T16:58:45.478 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-20T16:58:45.478 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout } 2024-04-20T16:58:45.478 INFO:teuthology.orchestra.run.smithi005.stdout:mgr is available 2024-04-20T16:58:46.632 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:46 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1604411256' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-20T16:58:46.979 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 2024-04-20T16:58:46.979 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout [global] 2024-04-20T16:58:46.979 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout fsid = 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T16:58:46.979 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-20T16:58:46.979 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-20T16:58:46.979 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-20T16:58:46.979 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-20T16:58:46.980 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-20T16:58:46.980 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 2024-04-20T16:58:46.980 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-20T16:58:46.980 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-20T16:58:46.980 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 2024-04-20T16:58:46.980 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout [osd] 2024-04-20T16:58:46.980 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-20T16:58:46.980 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-20T16:58:46.980 INFO:teuthology.orchestra.run.smithi005.stdout:Enabling cephadm module... 2024-04-20T16:58:47.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:47 smithi005 ceph-mon[39562]: from='mgr.14100 v1:172.21.15.5:0/1910016847' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T16:58:48.610 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:48 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3642985709' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-20T16:58:48.611 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:48 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: ignoring --setuser ceph since I am not root 2024-04-20T16:58:48.611 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:48 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: ignoring --setgroup ceph since I am not root 2024-04-20T16:58:48.611 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:48 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:48.429+0000 7f49057a6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-20T16:58:48.947 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:48 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:48.611+0000 7f49057a6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-20T16:58:49.211 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:48 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:48.948+0000 7f49057a6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-20T16:58:49.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:49 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3642985709' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-20T16:58:49.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:49 smithi005 ceph-mon[39562]: mgrmap e5: y(active, since 8s) 2024-04-20T16:58:49.711 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:49 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:49.492+0000 7f49057a6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-20T16:58:50.211 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:49 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:49.752+0000 7f49057a6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-20T16:58:50.541 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout { 2024-04-20T16:58:50.542 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-20T16:58:50.542 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-20T16:58:50.542 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-04-20T16:58:50.542 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-20T16:58:50.542 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout } 2024-04-20T16:58:50.542 INFO:teuthology.orchestra.run.smithi005.stdout:Waiting for the mgr to restart... 2024-04-20T16:58:50.542 INFO:teuthology.orchestra.run.smithi005.stdout:Waiting for mgr epoch 5... 2024-04-20T16:58:50.543 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:50 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:50.213+0000 7f49057a6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-20T16:58:50.543 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:50 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3548223365' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-20T16:58:51.211 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:50 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:50.979+0000 7f49057a6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-20T16:58:51.211 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:51 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:51.070+0000 7f49057a6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-20T16:58:51.961 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:51 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:51.699+0000 7f49057a6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-20T16:58:51.961 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:51 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:51.787+0000 7f49057a6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-20T16:58:51.961 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:51 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:51.878+0000 7f49057a6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-20T16:58:52.461 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:52 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:52.113+0000 7f49057a6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-20T16:58:52.461 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:52 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:52.209+0000 7f49057a6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-20T16:58:52.461 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:52 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:52.306+0000 7f49057a6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-20T16:58:52.840 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:52 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:52.547+0000 7f49057a6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-20T16:58:52.840 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:52 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:52.647+0000 7f49057a6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-20T16:58:52.840 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:52 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:52.840+0000 7f49057a6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-20T16:58:53.461 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:53 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:53.101+0000 7f49057a6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-20T16:58:54.134 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:53 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:53.712+0000 7f49057a6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-20T16:58:54.134 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:53 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:53.931+0000 7f49057a6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-20T16:58:55.211 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:54 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:54.134+0000 7f49057a6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-20T16:58:55.211 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:54 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:54.860+0000 7f49057a6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-20T16:58:55.211 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:55 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:55.067+0000 7f49057a6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-20T16:58:55.711 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:55 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:55.257+0000 7f49057a6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-20T16:58:55.711 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:58:55 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:58:55.379+0000 7f49057a6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-20T16:58:55.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:55 smithi005 ceph-mon[39562]: Active manager daemon y restarted 2024-04-20T16:58:55.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:55 smithi005 ceph-mon[39562]: Activating manager daemon y 2024-04-20T16:58:55.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:55 smithi005 ceph-mon[39562]: osdmap e2: 0 total, 0 up, 0 in 2024-04-20T16:58:55.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:55 smithi005 ceph-mon[39562]: mgrmap e6: y(active, starting, since 0.00320095s) 2024-04-20T16:58:55.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:55 smithi005 ceph-mon[39562]: from='mgr.14116 v1:172.21.15.5:0/1073194425' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T16:58:55.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:55 smithi005 ceph-mon[39562]: from='mgr.14116 v1:172.21.15.5:0/1073194425' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-20T16:58:55.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:55 smithi005 ceph-mon[39562]: from='mgr.14116 v1:172.21.15.5:0/1073194425' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-20T16:58:55.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:55 smithi005 ceph-mon[39562]: from='mgr.14116 v1:172.21.15.5:0/1073194425' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-20T16:58:55.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:55 smithi005 ceph-mon[39562]: from='mgr.14116 v1:172.21.15.5:0/1073194425' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-20T16:58:55.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:55 smithi005 ceph-mon[39562]: Manager daemon y is now available 2024-04-20T16:58:55.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:55 smithi005 ceph-mon[39562]: from='mgr.14116 v1:172.21.15.5:0/1073194425' entity='mgr.y' 2024-04-20T16:58:55.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:55 smithi005 ceph-mon[39562]: from='mgr.14116 v1:172.21.15.5:0/1073194425' entity='mgr.y' 2024-04-20T16:58:55.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:55 smithi005 ceph-mon[39562]: from='mgr.14116 v1:172.21.15.5:0/1073194425' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T16:58:56.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:56 smithi005 ceph-mon[39562]: Found migration_current of "None". Setting to last migration. 2024-04-20T16:58:56.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:56 smithi005 ceph-mon[39562]: from='mgr.14116 v1:172.21.15.5:0/1073194425' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T16:58:56.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:56 smithi005 ceph-mon[39562]: from='mgr.14116 v1:172.21.15.5:0/1073194425' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-20T16:58:56.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:56 smithi005 ceph-mon[39562]: from='mgr.14116 v1:172.21.15.5:0/1073194425' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-20T16:58:56.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:56 smithi005 ceph-mon[39562]: mgrmap e7: y(active, since 1.00527s) 2024-04-20T16:58:57.038 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout { 2024-04-20T16:58:57.039 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-20T16:58:57.039 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-20T16:58:57.039 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout } 2024-04-20T16:58:57.039 INFO:teuthology.orchestra.run.smithi005.stdout:mgr epoch 5 is available 2024-04-20T16:58:57.039 INFO:teuthology.orchestra.run.smithi005.stdout:Setting orchestrator backend to cephadm... 2024-04-20T16:58:58.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:57 smithi005 ceph-mon[39562]: from='client.14120 v1:172.21.15.5:0/202376039' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-20T16:58:58.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:57 smithi005 ceph-mon[39562]: from='client.14120 v1:172.21.15.5:0/202376039' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-20T16:58:58.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:57 smithi005 ceph-mon[39562]: from='mgr.14116 v1:172.21.15.5:0/1073194425' entity='mgr.y' 2024-04-20T16:58:58.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:57 smithi005 ceph-mon[39562]: from='mgr.14116 v1:172.21.15.5:0/1073194425' entity='mgr.y' 2024-04-20T16:58:58.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:57 smithi005 ceph-mon[39562]: from='mgr.14116 v1:172.21.15.5:0/1073194425' entity='mgr.y' 2024-04-20T16:58:58.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:57 smithi005 ceph-mon[39562]: from='mgr.14116 v1:172.21.15.5:0/1073194425' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T16:58:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:58 smithi005 ceph-mon[39562]: from='client.14128 v1:172.21.15.5:0/897870529' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T16:58:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:58 smithi005 ceph-mon[39562]: mgrmap e8: y(active, since 2s) 2024-04-20T16:58:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:58 smithi005 ceph-mon[39562]: from='mgr.14116 v1:172.21.15.5:0/1073194425' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T16:58:59.977 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-20T16:58:59.978 INFO:teuthology.orchestra.run.smithi005.stdout:Generating ssh key... 2024-04-20T16:59:00.252 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:59 smithi005 ceph-mon[39562]: [20/Apr/2024:16:58:58] ENGINE Bus STARTING 2024-04-20T16:59:00.253 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:59 smithi005 ceph-mon[39562]: [20/Apr/2024:16:58:58] ENGINE Serving on http://172.21.15.5:8765 2024-04-20T16:59:00.253 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:59 smithi005 ceph-mon[39562]: [20/Apr/2024:16:58:58] ENGINE Serving on https://172.21.15.5:7150 2024-04-20T16:59:00.253 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:58:59 smithi005 ceph-mon[39562]: [20/Apr/2024:16:58:58] ENGINE Bus STARTED 2024-04-20T16:59:01.272 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:00 smithi005 ceph-mon[39562]: from='client.14130 v1:172.21.15.5:0/539555812' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T16:59:01.272 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:01 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: Generating public/private rsa key pair. 2024-04-20T16:59:01.273 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:01 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: Your identification has been saved in /tmp/tmpdw2049sr/key. 2024-04-20T16:59:01.273 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:01 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: Your public key has been saved in /tmp/tmpdw2049sr/key.pub. 2024-04-20T16:59:01.273 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:01 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: The key fingerprint is: 2024-04-20T16:59:01.273 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:01 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: SHA256:FsAzuFv0QGsDVJC8VqOEGcrDSfeNg1VjZu1bHFg81S4 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T16:59:01.273 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:01 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: The key's randomart image is: 2024-04-20T16:59:01.273 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:01 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: +---[RSA 3072]----+ 2024-04-20T16:59:01.273 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:01 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: | ..B+O=*. +.... | 2024-04-20T16:59:01.273 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:01 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: |+.= O.#o.o + . | 2024-04-20T16:59:01.273 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:01 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: |.= o X+Bo . o . | 2024-04-20T16:59:01.273 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:01 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: | . =.o..o o E . | 2024-04-20T16:59:01.273 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:01 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: | . o S o . | 2024-04-20T16:59:01.273 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:01 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: | . . . | 2024-04-20T16:59:01.273 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:01 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: | | 2024-04-20T16:59:01.273 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:01 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: | | 2024-04-20T16:59:01.273 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:01 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: | | 2024-04-20T16:59:01.274 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:01 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: +----[SHA256]-----+ 2024-04-20T16:59:02.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:02 smithi005 ceph-mon[39562]: from='client.14132 v1:172.21.15.5:0/1086593182' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T16:59:02.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:02 smithi005 ceph-mon[39562]: Generating ssh key... 2024-04-20T16:59:02.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:02 smithi005 ceph-mon[39562]: from='mgr.14116 v1:172.21.15.5:0/1073194425' entity='mgr.y' 2024-04-20T16:59:02.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:02 smithi005 ceph-mon[39562]: from='mgr.14116 v1:172.21.15.5:0/1073194425' entity='mgr.y' 2024-04-20T16:59:03.415 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1zMuHh7yD0LXZZu8yNtDsGgrotaXiTcqh+cgdcq1RnWo3YJnwd76WTemfP/Y/zzg4bZQp7XL5CBr/acDs+BcGe0Ym2WJWsM26C8lF+rWCmeYe9xUYdhG21GVvTxYcJXRQthk9y9rCBXV2jrMg81ICpUnkP2Z814D+wCUBfTPSFUSnAIlcFdCDz4fHqIoAlnEQK8CXNfSp1lXLYQ/oJPPOt9MB4y5Yb4NqG+9pSyDaZ2lPUnWTtdhXFSvpHnZ+NJhXF5lxP5XgBa1u87hHXqR13w0UQwxs6DHEvEJdgwsUk9AQ+1Egoh8ruYK5UlCNlzwC35bShIrUoxWnpOZ3z9WwhoO7ZVgnpLkXT0cyrfiE3SVI+oP4NThBuM3b/czoCY1lfBqH9EovOjmvtEvmeB5tBrTAN/ut/+ZeWyNzAanmE7TgiZyiRpLjVyrmp9/dRDxnLBIyVhF2viouWVzz5o+W3GlrIbcNTAXVJ7F9glgKWFE1wr4/hJtxQGvZyElxJXc= ceph-13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T16:59:03.416 INFO:teuthology.orchestra.run.smithi005.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-20T16:59:03.416 INFO:teuthology.orchestra.run.smithi005.stdout:Adding key to root@localhost authorized_keys... 2024-04-20T16:59:03.417 INFO:teuthology.orchestra.run.smithi005.stdout:Adding host smithi005... 2024-04-20T16:59:03.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:03 smithi005 ceph-mon[39562]: from='client.14134 v1:172.21.15.5:0/2980180833' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T16:59:04.493 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:04 smithi005 ceph-mon[39562]: from='client.14136 v1:172.21.15.5:0/971536560' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi005", "addr": "172.21.15.5", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T16:59:05.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:05 smithi005 ceph-mon[39562]: Deploying cephadm binary to smithi005 2024-04-20T16:59:08.852 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout Added host 'smithi005' with addr '172.21.15.5' 2024-04-20T16:59:08.853 INFO:teuthology.orchestra.run.smithi005.stdout:Deploying unmanaged mon service... 2024-04-20T16:59:09.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:09 smithi005 ceph-mon[39562]: from='mgr.14116 v1:172.21.15.5:0/1073194425' entity='mgr.y' 2024-04-20T16:59:09.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:09 smithi005 ceph-mon[39562]: Added host smithi005 2024-04-20T16:59:09.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:09 smithi005 ceph-mon[39562]: from='mgr.14116 v1:172.21.15.5:0/1073194425' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T16:59:10.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:10 smithi005 ceph-mon[39562]: from='client.14138 v1:172.21.15.5:0/1033083352' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T16:59:10.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:10 smithi005 ceph-mon[39562]: Saving service mon spec with placement count:5 2024-04-20T16:59:10.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:10 smithi005 ceph-mon[39562]: from='mgr.14116 v1:172.21.15.5:0/1073194425' entity='mgr.y' 2024-04-20T16:59:10.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:10 smithi005 ceph-mon[39562]: from='mgr.14116 v1:172.21.15.5:0/1073194425' entity='mgr.y' 2024-04-20T16:59:11.157 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-20T16:59:11.157 INFO:teuthology.orchestra.run.smithi005.stdout:Deploying unmanaged mgr service... 2024-04-20T16:59:12.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:12 smithi005 ceph-mon[39562]: from='mgr.14116 v1:172.21.15.5:0/1073194425' entity='mgr.y' 2024-04-20T16:59:12.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:12 smithi005 ceph-mon[39562]: from='mgr.14116 v1:172.21.15.5:0/1073194425' entity='mgr.y' 2024-04-20T16:59:13.359 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-20T16:59:13.635 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:13 smithi005 ceph-mon[39562]: from='client.14140 v1:172.21.15.5:0/3717109019' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T16:59:13.635 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:13 smithi005 ceph-mon[39562]: Saving service mgr spec with placement count:2 2024-04-20T16:59:13.635 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:13 smithi005 ceph-mon[39562]: from='mgr.14116 v1:172.21.15.5:0/1073194425' entity='mgr.y' 2024-04-20T16:59:14.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:14 smithi005 ceph-mon[39562]: from='mgr.14116 v1:172.21.15.5:0/1073194425' entity='mgr.y' 2024-04-20T16:59:14.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:14 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2471832529' entity='client.admin' 2024-04-20T16:59:16.665 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:16 smithi005 ceph-mon[39562]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T16:59:17.221 INFO:teuthology.orchestra.run.smithi005.stdout:Enabling the dashboard module... 2024-04-20T16:59:17.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:17 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1077795857' entity='client.admin' 2024-04-20T16:59:18.961 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:18 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: ignoring --setuser ceph since I am not root 2024-04-20T16:59:18.961 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:18 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: ignoring --setgroup ceph since I am not root 2024-04-20T16:59:18.961 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:18 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:59:18.825+0000 7f0d5acea200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-20T16:59:18.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:18 smithi005 ceph-mon[39562]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T16:59:18.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:18 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1388829693' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-20T16:59:19.347 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:19 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:59:19.010+0000 7f0d5acea200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-20T16:59:19.640 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:19 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:59:19.347+0000 7f0d5acea200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-20T16:59:19.961 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:19 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:59:19.874+0000 7f0d5acea200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-20T16:59:19.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:19 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1388829693' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-20T16:59:19.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:19 smithi005 ceph-mon[39562]: mgrmap e9: y(active, since 23s) 2024-04-20T16:59:20.461 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:20 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:59:20.153+0000 7f0d5acea200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-20T16:59:20.951 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout { 2024-04-20T16:59:20.951 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-20T16:59:20.951 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-20T16:59:20.951 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-04-20T16:59:20.951 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-20T16:59:20.952 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout } 2024-04-20T16:59:20.952 INFO:teuthology.orchestra.run.smithi005.stdout:Waiting for the mgr to restart... 2024-04-20T16:59:20.952 INFO:teuthology.orchestra.run.smithi005.stdout:Waiting for mgr epoch 9... 2024-04-20T16:59:20.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:20 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:59:20.619+0000 7f0d5acea200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-20T16:59:20.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:20 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1938681870' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-20T16:59:21.711 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:21 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:59:21.386+0000 7f0d5acea200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-20T16:59:21.711 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:21 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:59:21.480+0000 7f0d5acea200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-20T16:59:22.461 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:22 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:59:22.112+0000 7f0d5acea200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-20T16:59:22.461 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:22 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:59:22.201+0000 7f0d5acea200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-20T16:59:22.461 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:22 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:59:22.295+0000 7f0d5acea200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-20T16:59:22.961 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:22 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:59:22.543+0000 7f0d5acea200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-20T16:59:22.961 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:22 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:59:22.639+0000 7f0d5acea200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-20T16:59:22.961 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:22 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:59:22.738+0000 7f0d5acea200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-20T16:59:23.277 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:22 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:59:22.981+0000 7f0d5acea200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-20T16:59:23.277 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:23 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:59:23.082+0000 7f0d5acea200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-20T16:59:23.539 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:23 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:59:23.277+0000 7f0d5acea200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-20T16:59:23.961 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:23 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:59:23.539+0000 7f0d5acea200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-20T16:59:24.461 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:24 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:59:24.152+0000 7f0d5acea200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-20T16:59:24.461 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:24 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:59:24.373+0000 7f0d5acea200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-20T16:59:24.961 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:24 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:59:24.578+0000 7f0d5acea200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-20T16:59:25.673 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:25 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:59:25.270+0000 7f0d5acea200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-20T16:59:25.673 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:25 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:59:25.480+0000 7f0d5acea200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-20T16:59:25.939 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:25 smithi005 ceph-mon[39562]: Active manager daemon y restarted 2024-04-20T16:59:25.939 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:25 smithi005 ceph-mon[39562]: Activating manager daemon y 2024-04-20T16:59:25.939 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:25 smithi005 ceph-mon[39562]: osdmap e3: 0 total, 0 up, 0 in 2024-04-20T16:59:25.939 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:25 smithi005 ceph-mon[39562]: mgrmap e10: y(active, starting, since 0.00314117s) 2024-04-20T16:59:25.939 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:25 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T16:59:25.939 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:25 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-20T16:59:25.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:25 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-20T16:59:25.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:25 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-20T16:59:25.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:25 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-20T16:59:25.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:25 smithi005 ceph-mon[39562]: Manager daemon y is now available 2024-04-20T16:59:25.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:25 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T16:59:25.940 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:25 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:59:25.672+0000 7f0d5acea200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-20T16:59:25.940 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 16:59:25 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T16:59:25.793+0000 7f0d5acea200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-20T16:59:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:26 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-20T16:59:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:26 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-20T16:59:27.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:26 smithi005 ceph-mon[39562]: mgrmap e11: y(active, since 1.00618s) 2024-04-20T16:59:27.728 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout { 2024-04-20T16:59:27.728 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-20T16:59:27.728 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-20T16:59:27.728 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout } 2024-04-20T16:59:27.728 INFO:teuthology.orchestra.run.smithi005.stdout:mgr epoch 9 is available 2024-04-20T16:59:27.728 INFO:teuthology.orchestra.run.smithi005.stdout:Generating a dashboard self-signed certificate... 2024-04-20T16:59:28.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:28 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T16:59:28.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:28 smithi005 ceph-mon[39562]: [20/Apr/2024:16:59:27] ENGINE Bus STARTING 2024-04-20T16:59:28.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:28 smithi005 ceph-mon[39562]: [20/Apr/2024:16:59:27] ENGINE Serving on https://172.21.15.5:7150 2024-04-20T16:59:28.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:28 smithi005 ceph-mon[39562]: [20/Apr/2024:16:59:27] ENGINE Serving on http://172.21.15.5:8765 2024-04-20T16:59:28.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:28 smithi005 ceph-mon[39562]: [20/Apr/2024:16:59:27] ENGINE Bus STARTED 2024-04-20T16:59:28.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:28 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T16:59:29.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:29 smithi005 ceph-mon[39562]: mgrmap e12: y(active, since 2s) 2024-04-20T16:59:29.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:29 smithi005 ceph-mon[39562]: from='client.14160 v1:172.21.15.5:0/1037384858' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T16:59:29.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:29 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T16:59:29.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:29 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T16:59:30.292 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-20T16:59:30.292 INFO:teuthology.orchestra.run.smithi005.stdout:Creating initial admin user... 2024-04-20T16:59:32.174 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$AwvXm89vbB1WnMDLLxpxLOcvL63MRcjWlffglDnrDZ46zkoTHuarS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713632371, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-20T16:59:32.175 INFO:teuthology.orchestra.run.smithi005.stdout:Fetching dashboard port number... 2024-04-20T16:59:32.449 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:32 smithi005 ceph-mon[39562]: from='client.14162 v1:172.21.15.5:0/3590020746' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T16:59:32.449 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T16:59:33.689 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 8443 2024-04-20T16:59:33.692 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:33 smithi005 ceph-mon[39562]: mgrmap e13: y(active, since 6s) 2024-04-20T16:59:33.692 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:33 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3598759849' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-20T16:59:33.701 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-20T16:59:33.701 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout disabled 2024-04-20T16:59:33.711 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-20T16:59:33.711 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout inactive 2024-04-20T16:59:33.711 INFO:teuthology.orchestra.run.smithi005.stdout:firewalld.service is not enabled 2024-04-20T16:59:33.711 INFO:teuthology.orchestra.run.smithi005.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-20T16:59:33.713 INFO:teuthology.orchestra.run.smithi005.stdout:Ceph Dashboard is now available at: 2024-04-20T16:59:33.713 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:59:33.713 INFO:teuthology.orchestra.run.smithi005.stdout: URL: https://smithi005.front.sepia.ceph.com:8443/ 2024-04-20T16:59:33.713 INFO:teuthology.orchestra.run.smithi005.stdout: User: admin 2024-04-20T16:59:33.713 INFO:teuthology.orchestra.run.smithi005.stdout: Password: q0f18bclg4 2024-04-20T16:59:33.713 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:59:33.713 INFO:teuthology.orchestra.run.smithi005.stdout:Saving cluster configuration to /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/config directory 2024-04-20T16:59:33.714 INFO:teuthology.orchestra.run.smithi005.stdout:Enabling autotune for osd_memory_target 2024-04-20T16:59:36.602 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-20T16:59:36.602 INFO:teuthology.orchestra.run.smithi005.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-20T16:59:36.602 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:59:36.602 INFO:teuthology.orchestra.run.smithi005.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-20T16:59:36.602 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:59:36.602 INFO:teuthology.orchestra.run.smithi005.stdout:Or, if you are only running a single cluster on this host: 2024-04-20T16:59:36.602 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:59:36.602 INFO:teuthology.orchestra.run.smithi005.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-20T16:59:36.602 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:59:36.603 INFO:teuthology.orchestra.run.smithi005.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-20T16:59:36.603 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:59:36.603 INFO:teuthology.orchestra.run.smithi005.stdout: ceph telemetry on 2024-04-20T16:59:36.603 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:59:36.603 INFO:teuthology.orchestra.run.smithi005.stdout:For more information see: 2024-04-20T16:59:36.603 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:59:36.603 INFO:teuthology.orchestra.run.smithi005.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-20T16:59:36.603 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:59:36.603 INFO:teuthology.orchestra.run.smithi005.stdout:Bootstrap complete. 2024-04-20T16:59:36.652 INFO:tasks.cephadm:Fetching config... 2024-04-20T16:59:36.652 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-04-20T16:59:36.652 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-20T16:59:36.671 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-20T16:59:36.671 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-04-20T16:59:36.672 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-20T16:59:36.730 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-20T16:59:36.730 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-04-20T16:59:36.730 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-20T16:59:36.798 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-20T16:59:36.798 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-04-20T16:59:36.798 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-20T16:59:36.857 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-20T16:59:36.857 DEBUG:teuthology.orchestra.run.smithi005:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1zMuHh7yD0LXZZu8yNtDsGgrotaXiTcqh+cgdcq1RnWo3YJnwd76WTemfP/Y/zzg4bZQp7XL5CBr/acDs+BcGe0Ym2WJWsM26C8lF+rWCmeYe9xUYdhG21GVvTxYcJXRQthk9y9rCBXV2jrMg81ICpUnkP2Z814D+wCUBfTPSFUSnAIlcFdCDz4fHqIoAlnEQK8CXNfSp1lXLYQ/oJPPOt9MB4y5Yb4NqG+9pSyDaZ2lPUnWTtdhXFSvpHnZ+NJhXF5lxP5XgBa1u87hHXqR13w0UQwxs6DHEvEJdgwsUk9AQ+1Egoh8ruYK5UlCNlzwC35bShIrUoxWnpOZ3z9WwhoO7ZVgnpLkXT0cyrfiE3SVI+oP4NThBuM3b/czoCY1lfBqH9EovOjmvtEvmeB5tBrTAN/ut/+ZeWyNzAanmE7TgiZyiRpLjVyrmp9/dRDxnLBIyVhF2viouWVzz5o+W3GlrIbcNTAXVJ7F9glgKWFE1wr4/hJtxQGvZyElxJXc= ceph-13c6e76c-ff37-11ee-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-20T16:59:36.939 INFO:teuthology.orchestra.run.smithi005.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1zMuHh7yD0LXZZu8yNtDsGgrotaXiTcqh+cgdcq1RnWo3YJnwd76WTemfP/Y/zzg4bZQp7XL5CBr/acDs+BcGe0Ym2WJWsM26C8lF+rWCmeYe9xUYdhG21GVvTxYcJXRQthk9y9rCBXV2jrMg81ICpUnkP2Z814D+wCUBfTPSFUSnAIlcFdCDz4fHqIoAlnEQK8CXNfSp1lXLYQ/oJPPOt9MB4y5Yb4NqG+9pSyDaZ2lPUnWTtdhXFSvpHnZ+NJhXF5lxP5XgBa1u87hHXqR13w0UQwxs6DHEvEJdgwsUk9AQ+1Egoh8ruYK5UlCNlzwC35bShIrUoxWnpOZ3z9WwhoO7ZVgnpLkXT0cyrfiE3SVI+oP4NThBuM3b/czoCY1lfBqH9EovOjmvtEvmeB5tBrTAN/ut/+ZeWyNzAanmE7TgiZyiRpLjVyrmp9/dRDxnLBIyVhF2viouWVzz5o+W3GlrIbcNTAXVJ7F9glgKWFE1wr4/hJtxQGvZyElxJXc= ceph-13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T16:59:36.952 DEBUG:teuthology.orchestra.run.smithi106:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1zMuHh7yD0LXZZu8yNtDsGgrotaXiTcqh+cgdcq1RnWo3YJnwd76WTemfP/Y/zzg4bZQp7XL5CBr/acDs+BcGe0Ym2WJWsM26C8lF+rWCmeYe9xUYdhG21GVvTxYcJXRQthk9y9rCBXV2jrMg81ICpUnkP2Z814D+wCUBfTPSFUSnAIlcFdCDz4fHqIoAlnEQK8CXNfSp1lXLYQ/oJPPOt9MB4y5Yb4NqG+9pSyDaZ2lPUnWTtdhXFSvpHnZ+NJhXF5lxP5XgBa1u87hHXqR13w0UQwxs6DHEvEJdgwsUk9AQ+1Egoh8ruYK5UlCNlzwC35bShIrUoxWnpOZ3z9WwhoO7ZVgnpLkXT0cyrfiE3SVI+oP4NThBuM3b/czoCY1lfBqH9EovOjmvtEvmeB5tBrTAN/ut/+ZeWyNzAanmE7TgiZyiRpLjVyrmp9/dRDxnLBIyVhF2viouWVzz5o+W3GlrIbcNTAXVJ7F9glgKWFE1wr4/hJtxQGvZyElxJXc= ceph-13c6e76c-ff37-11ee-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-20T16:59:36.998 INFO:teuthology.orchestra.run.smithi106.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1zMuHh7yD0LXZZu8yNtDsGgrotaXiTcqh+cgdcq1RnWo3YJnwd76WTemfP/Y/zzg4bZQp7XL5CBr/acDs+BcGe0Ym2WJWsM26C8lF+rWCmeYe9xUYdhG21GVvTxYcJXRQthk9y9rCBXV2jrMg81ICpUnkP2Z814D+wCUBfTPSFUSnAIlcFdCDz4fHqIoAlnEQK8CXNfSp1lXLYQ/oJPPOt9MB4y5Yb4NqG+9pSyDaZ2lPUnWTtdhXFSvpHnZ+NJhXF5lxP5XgBa1u87hHXqR13w0UQwxs6DHEvEJdgwsUk9AQ+1Egoh8ruYK5UlCNlzwC35bShIrUoxWnpOZ3z9WwhoO7ZVgnpLkXT0cyrfiE3SVI+oP4NThBuM3b/czoCY1lfBqH9EovOjmvtEvmeB5tBrTAN/ut/+ZeWyNzAanmE7TgiZyiRpLjVyrmp9/dRDxnLBIyVhF2viouWVzz5o+W3GlrIbcNTAXVJ7F9glgKWFE1wr4/hJtxQGvZyElxJXc= ceph-13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T16:59:37.009 DEBUG:teuthology.orchestra.run.smithi171:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1zMuHh7yD0LXZZu8yNtDsGgrotaXiTcqh+cgdcq1RnWo3YJnwd76WTemfP/Y/zzg4bZQp7XL5CBr/acDs+BcGe0Ym2WJWsM26C8lF+rWCmeYe9xUYdhG21GVvTxYcJXRQthk9y9rCBXV2jrMg81ICpUnkP2Z814D+wCUBfTPSFUSnAIlcFdCDz4fHqIoAlnEQK8CXNfSp1lXLYQ/oJPPOt9MB4y5Yb4NqG+9pSyDaZ2lPUnWTtdhXFSvpHnZ+NJhXF5lxP5XgBa1u87hHXqR13w0UQwxs6DHEvEJdgwsUk9AQ+1Egoh8ruYK5UlCNlzwC35bShIrUoxWnpOZ3z9WwhoO7ZVgnpLkXT0cyrfiE3SVI+oP4NThBuM3b/czoCY1lfBqH9EovOjmvtEvmeB5tBrTAN/ut/+ZeWyNzAanmE7TgiZyiRpLjVyrmp9/dRDxnLBIyVhF2viouWVzz5o+W3GlrIbcNTAXVJ7F9glgKWFE1wr4/hJtxQGvZyElxJXc= ceph-13c6e76c-ff37-11ee-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-20T16:59:37.053 INFO:teuthology.orchestra.run.smithi171.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1zMuHh7yD0LXZZu8yNtDsGgrotaXiTcqh+cgdcq1RnWo3YJnwd76WTemfP/Y/zzg4bZQp7XL5CBr/acDs+BcGe0Ym2WJWsM26C8lF+rWCmeYe9xUYdhG21GVvTxYcJXRQthk9y9rCBXV2jrMg81ICpUnkP2Z814D+wCUBfTPSFUSnAIlcFdCDz4fHqIoAlnEQK8CXNfSp1lXLYQ/oJPPOt9MB4y5Yb4NqG+9pSyDaZ2lPUnWTtdhXFSvpHnZ+NJhXF5lxP5XgBa1u87hHXqR13w0UQwxs6DHEvEJdgwsUk9AQ+1Egoh8ruYK5UlCNlzwC35bShIrUoxWnpOZ3z9WwhoO7ZVgnpLkXT0cyrfiE3SVI+oP4NThBuM3b/czoCY1lfBqH9EovOjmvtEvmeB5tBrTAN/ut/+ZeWyNzAanmE7TgiZyiRpLjVyrmp9/dRDxnLBIyVhF2viouWVzz5o+W3GlrIbcNTAXVJ7F9glgKWFE1wr4/hJtxQGvZyElxJXc= ceph-13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T16:59:37.066 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-20T16:59:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:36 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3742280813' entity='client.admin' 2024-04-20T16:59:37.339 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T16:59:40.798 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-20T16:59:40.798 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-20T16:59:41.093 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T16:59:41.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:40 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1357885622' entity='client.admin' 2024-04-20T16:59:43.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T16:59:43.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T16:59:43.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi005", "name": "osd_memory_target"}]: dispatch 2024-04-20T16:59:43.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T16:59:43.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T16:59:43.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T16:59:43.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:43 smithi005 ceph-mon[39562]: from='client.14172 v1:172.21.15.5:0/637627921' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T16:59:43.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T16:59:43.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T16:59:43.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T16:59:43.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T16:59:43.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:43 smithi005 ceph-mon[39562]: Updating smithi005:/etc/ceph/ceph.conf 2024-04-20T16:59:44.555 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi106 2024-04-20T16:59:44.555 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-20T16:59:44.555 DEBUG:teuthology.orchestra.run.smithi106:> dd of=/etc/ceph/ceph.conf 2024-04-20T16:59:44.577 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-20T16:59:44.577 DEBUG:teuthology.orchestra.run.smithi106:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-20T16:59:44.637 INFO:tasks.cephadm:Adding host smithi106 to orchestrator... 2024-04-20T16:59:44.637 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph orch host add smithi106 2024-04-20T16:59:44.907 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T16:59:45.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:45 smithi005 ceph-mon[39562]: Updating smithi005:/var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T16:59:45.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:45 smithi005 ceph-mon[39562]: Updating smithi005:/etc/ceph/ceph.client.admin.keyring 2024-04-20T16:59:45.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:45 smithi005 ceph-mon[39562]: Updating smithi005:/var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-20T16:59:45.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T16:59:45.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T16:59:45.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T16:59:47.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:47 smithi005 ceph-mon[39562]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T16:59:48.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:48 smithi005 ceph-mon[39562]: from='client.14174 v1:172.21.15.5:0/1232692276' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi106", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T16:59:48.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:48 smithi005 ceph-mon[39562]: Deploying cephadm binary to smithi106 2024-04-20T16:59:49.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:49 smithi005 ceph-mon[39562]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T16:59:50.726 INFO:teuthology.orchestra.run.smithi005.stdout:Added host 'smithi106' with addr '172.21.15.106' 2024-04-20T16:59:51.488 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-20T16:59:51.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:51 smithi005 ceph-mon[39562]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T16:59:51.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:51 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T16:59:51.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:51 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T16:59:51.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:51 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T16:59:51.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:51 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T16:59:51.762 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T16:59:52.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:52 smithi005 ceph-mon[39562]: Added host smithi106 2024-04-20T16:59:52.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:52 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T16:59:53.738 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T16:59:53.739 INFO:teuthology.orchestra.run.smithi005.stdout:[{"addr": "172.21.15.5", "hostname": "smithi005", "labels": [], "status": ""}, {"addr": "172.21.15.106", "hostname": "smithi106", "labels": [], "status": ""}] 2024-04-20T16:59:53.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:53 smithi005 ceph-mon[39562]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T16:59:54.381 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi171 2024-04-20T16:59:54.381 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-04-20T16:59:54.382 DEBUG:teuthology.orchestra.run.smithi171:> dd of=/etc/ceph/ceph.conf 2024-04-20T16:59:54.403 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-04-20T16:59:54.403 DEBUG:teuthology.orchestra.run.smithi171:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-20T16:59:54.464 INFO:tasks.cephadm:Adding host smithi171 to orchestrator... 2024-04-20T16:59:54.464 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph orch host add smithi171 2024-04-20T16:59:54.738 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T16:59:55.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:55 smithi005 ceph-mon[39562]: from='client.14176 v1:172.21.15.5:0/1627713615' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T16:59:55.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:55 smithi005 ceph-mon[39562]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T16:59:55.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:55 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T16:59:55.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:55 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T16:59:55.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:55 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T16:59:55.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:55 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T16:59:55.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:55 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi106", "name": "osd_memory_target"}]: dispatch 2024-04-20T16:59:55.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:55 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T16:59:55.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:55 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T16:59:56.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:56 smithi005 ceph-mon[39562]: Updating smithi106:/etc/ceph/ceph.conf 2024-04-20T16:59:56.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:56 smithi005 ceph-mon[39562]: Updating smithi106:/var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T16:59:56.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:56 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T16:59:56.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:56 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T16:59:56.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:56 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T16:59:57.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:57 smithi005 ceph-mon[39562]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T16:59:57.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:57 smithi005 ceph-mon[39562]: Updating smithi106:/etc/ceph/ceph.client.admin.keyring 2024-04-20T16:59:57.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:57 smithi005 ceph-mon[39562]: Updating smithi106:/var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-20T16:59:57.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:57 smithi005 ceph-mon[39562]: from='client.14178 v1:172.21.15.5:0/3328672003' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi171", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T16:59:57.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:57 smithi005 ceph-mon[39562]: Deploying cephadm binary to smithi171 2024-04-20T16:59:58.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 16:59:58 smithi005 ceph-mon[39562]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:00.739 INFO:teuthology.orchestra.run.smithi005.stdout:Added host 'smithi171' with addr '172.21.15.171' 2024-04-20T17:00:01.200 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:00 smithi005 ceph-mon[39562]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:01.200 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:00 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:01.200 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:00 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:00:01.523 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-20T17:00:01.800 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:00:02.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:02 smithi005 ceph-mon[39562]: Added host smithi171 2024-04-20T17:00:02.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:02 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:02.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:02 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:03.297 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:03 smithi005 ceph-mon[39562]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:03.297 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:03 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:03.812 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:00:03.812 INFO:teuthology.orchestra.run.smithi005.stdout:[{"addr": "172.21.15.5", "hostname": "smithi005", "labels": [], "status": ""}, {"addr": "172.21.15.106", "hostname": "smithi106", "labels": [], "status": ""}, {"addr": "172.21.15.171", "hostname": "smithi171", "labels": [], "status": ""}] 2024-04-20T17:00:04.536 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-20T17:00:04.536 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-20T17:00:04.809 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:00:05.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:05 smithi005 ceph-mon[39562]: from='client.14180 v1:172.21.15.5:0/3061992227' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:00:05.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:05 smithi005 ceph-mon[39562]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:06.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:06 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:06.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:06 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:06.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:06 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:06.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:06 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:06.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:06 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi171", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:00:06.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:06 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:06.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:06 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:00:06.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:06 smithi005 ceph-mon[39562]: Updating smithi171:/etc/ceph/ceph.conf 2024-04-20T17:00:06.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:06 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:06.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:06 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:06.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:06 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:07.558 INFO:teuthology.orchestra.run.smithi005.stderr:adjusted tunables profile to default 2024-04-20T17:00:07.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:07 smithi005 ceph-mon[39562]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:07.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:07 smithi005 ceph-mon[39562]: Updating smithi171:/var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T17:00:07.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:07 smithi005 ceph-mon[39562]: Updating smithi171:/etc/ceph/ceph.client.admin.keyring 2024-04-20T17:00:07.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:07 smithi005 ceph-mon[39562]: Updating smithi171:/var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-20T17:00:07.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:07 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/340361140' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-20T17:00:08.370 INFO:tasks.cephadm:Adding mon.a on smithi005 2024-04-20T17:00:08.370 INFO:tasks.cephadm:Adding mon.b on smithi106 2024-04-20T17:00:08.370 INFO:tasks.cephadm:Adding mon.c on smithi171 2024-04-20T17:00:08.370 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph orch apply mon '3;smithi005:[v1:172.21.15.5:6789]=a;smithi106:[v1:172.21.15.106:6789]=b;smithi171:[v1:172.21.15.171:6789]=c' 2024-04-20T17:00:08.671 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T17:00:08.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:08 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/340361140' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-20T17:00:08.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:08 smithi005 ceph-mon[39562]: osdmap e4: 0 total, 0 up, 0 in 2024-04-20T17:00:09.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:09 smithi005 ceph-mon[39562]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:10.797 INFO:teuthology.orchestra.run.smithi171.stdout:Scheduled mon update... 2024-04-20T17:00:11.507 DEBUG:teuthology.orchestra.run.smithi106:mon.b> sudo journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@mon.b.service 2024-04-20T17:00:11.509 DEBUG:teuthology.orchestra.run.smithi171:mon.c> sudo journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@mon.c.service 2024-04-20T17:00:11.511 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-20T17:00:11.512 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-20T17:00:11.544 INFO:journalctl@ceph.mon.c.smithi171.stdout:-- Logs begin at Sat 2024-04-20 16:47:48 UTC. -- 2024-04-20T17:00:11.546 INFO:journalctl@ceph.mon.b.smithi106.stdout:-- Logs begin at Sat 2024-04-20 16:47:44 UTC. -- 2024-04-20T17:00:11.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:11 smithi005 ceph-mon[39562]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:11.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:11 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:11.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:11 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:00:11.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:11 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:11.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:11 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:00:11.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:11 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:11.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:11 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-20T17:00:11.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:11 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:12.372 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T17:00:12.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:12 smithi005 ceph-mon[39562]: from='client.14184 v1:172.21.15.171:0/684140253' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi005:[v1:172.21.15.5:6789]=a;smithi106:[v1:172.21.15.106:6789]=b;smithi171:[v1:172.21.15.171:6789]=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:00:12.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:12 smithi005 ceph-mon[39562]: Saving service mon spec with placement smithi005:[v1:172.21.15.5:6789]=a;smithi106:[v1:172.21.15.106:6789]=b;smithi171:[v1:172.21.15.171:6789]=c;count:3 2024-04-20T17:00:12.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:12 smithi005 ceph-mon[39562]: Deploying daemon mon.c on smithi171 2024-04-20T17:00:13.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:13 smithi005 ceph-mon[39562]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:15.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:15 smithi005 ceph-mon[39562]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:16.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:15 smithi171 bash[44113]: c1373e25db313ea568a9bafce4ce0305dff5e10dc6c218ec065a5282eafed339 2024-04-20T17:00:16.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:15 smithi171 systemd[1]: Started Ceph mon.c for 13c6e76c-ff37-11ee-bc93-c7b262605968. 2024-04-20T17:00:20.943 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T17:00:20.943 INFO:teuthology.orchestra.run.smithi171.stdout:{"epoch":2,"fsid":"13c6e76c-ff37-11ee-bc93-c7b262605968","modified":"2024-04-20T17:00:15.907056Z","created":"2024-04-20T16:58:22.127736Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-20T17:00:20.946 INFO:teuthology.orchestra.run.smithi171.stderr:dumped monmap epoch 2 2024-04-20T17:00:21.308 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:21.308 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:00:21.308 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: mon.a calling monitor election 2024-04-20T17:00:21.308 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:00:21.308 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.171:0/3214808924' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T17:00:21.308 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:00:21.309 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:21.309 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:00:21.309 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: mon.c calling monitor election 2024-04-20T17:00:21.309 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:00:21.309 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:21.309 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:00:21.309 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:00:21.309 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-20T17:00:21.309 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: monmap epoch 2 2024-04-20T17:00:21.309 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T17:00:21.309 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: last_changed 2024-04-20T17:00:15.907056+0000 2024-04-20T17:00:21.309 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: created 2024-04-20T16:58:22.127736+0000 2024-04-20T17:00:21.309 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: min_mon_release 19 (squid) 2024-04-20T17:00:21.309 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: election_strategy: 1 2024-04-20T17:00:21.310 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: 0: v1:172.21.15.5:6789/0 mon.a 2024-04-20T17:00:21.310 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: 1: v1:172.21.15.171:6789/0 mon.c 2024-04-20T17:00:21.310 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: fsmap 2024-04-20T17:00:21.310 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: osdmap e4: 0 total, 0 up, 0 in 2024-04-20T17:00:21.310 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: mgrmap e13: y(active, since 55s) 2024-04-20T17:00:21.310 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: overall HEALTH_OK 2024-04-20T17:00:21.310 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:21.310 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:20 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:21.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:21.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:00:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: mon.a calling monitor election 2024-04-20T17:00:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:00:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.171:0/3214808924' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T17:00:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:00:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:00:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: mon.c calling monitor election 2024-04-20T17:00:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:00:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:00:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:00:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-20T17:00:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: monmap epoch 2 2024-04-20T17:00:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T17:00:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: last_changed 2024-04-20T17:00:15.907056+0000 2024-04-20T17:00:21.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: created 2024-04-20T16:58:22.127736+0000 2024-04-20T17:00:21.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: min_mon_release 19 (squid) 2024-04-20T17:00:21.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: election_strategy: 1 2024-04-20T17:00:21.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: 0: v1:172.21.15.5:6789/0 mon.a 2024-04-20T17:00:21.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: 1: v1:172.21.15.171:6789/0 mon.c 2024-04-20T17:00:21.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: fsmap 2024-04-20T17:00:21.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: osdmap e4: 0 total, 0 up, 0 in 2024-04-20T17:00:21.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: mgrmap e13: y(active, since 55s) 2024-04-20T17:00:21.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: overall HEALTH_OK 2024-04-20T17:00:21.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:21.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:20 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:21 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:21 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-20T17:00:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:21 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:21 smithi171 ceph-mon[44189]: Deploying daemon mon.b on smithi106 2024-04-20T17:00:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:21 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:00:22.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:21 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:22.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:21 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-20T17:00:22.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:21 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:22.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:21 smithi005 ceph-mon[39562]: Deploying daemon mon.b on smithi106 2024-04-20T17:00:22.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:21 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:00:22.612 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-20T17:00:22.612 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-20T17:00:22.886 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:00:23.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:23 smithi171 ceph-mon[44189]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:23.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:23 smithi005 ceph-mon[39562]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 systemd[1]: Starting Ceph mon.b for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:00:25.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: set uid:gid to 167:167 (ceph:ceph) 2024-04-20T17:00:25.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: ceph version 19.0.0-3160-g36c37156 (36c371567dddacf6207ea36f2535396ab31415fc) squid (dev), process ceph-mon, pid 7 2024-04-20T17:00:25.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: pidfile_write: ignore empty --pid-file 2024-04-20T17:00:25.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: load: jerasure load: lrc 2024-04-20T17:00:25.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: RocksDB version: 7.9.2 2024-04-20T17:00:25.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Git sha 0 2024-04-20T17:00:25.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Compile date 2024-04-20 01:00:17 2024-04-20T17:00:25.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: DB SUMMARY 2024-04-20T17:00:25.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: DB Session ID: LNUQV23XDNX6LPACL868 2024-04-20T17:00:25.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: CURRENT file: CURRENT 2024-04-20T17:00:25.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: IDENTITY file: IDENTITY 2024-04-20T17:00:25.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-20T17:00:25.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-20T17:00:25.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 566 ; 2024-04-20T17:00:25.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.error_if_exists: 0 2024-04-20T17:00:25.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.create_if_missing: 0 2024-04-20T17:00:25.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.paranoid_checks: 1 2024-04-20T17:00:25.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-20T17:00:25.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-20T17:00:25.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-20T17:00:25.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.env: 0x56095a2aacc0 2024-04-20T17:00:25.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.fs: PosixFileSystem 2024-04-20T17:00:25.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.info_log: 0x56095b2c1c20 2024-04-20T17:00:25.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.max_file_opening_threads: 16 2024-04-20T17:00:25.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.statistics: (nil) 2024-04-20T17:00:25.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.use_fsync: 0 2024-04-20T17:00:25.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.max_log_file_size: 0 2024-04-20T17:00:25.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-20T17:00:25.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-20T17:00:25.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.keep_log_file_num: 1000 2024-04-20T17:00:25.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.recycle_log_file_num: 0 2024-04-20T17:00:25.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.allow_fallocate: 1 2024-04-20T17:00:25.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.allow_mmap_reads: 0 2024-04-20T17:00:25.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.allow_mmap_writes: 0 2024-04-20T17:00:25.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.use_direct_reads: 0 2024-04-20T17:00:25.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-20T17:00:25.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.create_missing_column_families: 0 2024-04-20T17:00:25.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.db_log_dir: 2024-04-20T17:00:25.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.wal_dir: 2024-04-20T17:00:25.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-20T17:00:25.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-20T17:00:25.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-20T17:00:25.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-20T17:00:25.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-20T17:00:25.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-20T17:00:25.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.advise_random_on_open: 1 2024-04-20T17:00:25.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.db_write_buffer_size: 0 2024-04-20T17:00:25.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.write_buffer_manager: 0x56095b2550e0 2024-04-20T17:00:25.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-20T17:00:25.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-20T17:00:25.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-20T17:00:25.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.rate_limiter: (nil) 2024-04-20T17:00:25.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-20T17:00:25.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.wal_recovery_mode: 2 2024-04-20T17:00:25.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.enable_thread_tracking: 0 2024-04-20T17:00:25.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.enable_pipelined_write: 0 2024-04-20T17:00:25.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.unordered_write: 0 2024-04-20T17:00:25.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-20T17:00:25.017 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-20T17:00:25.017 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-20T17:00:25.017 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-20T17:00:25.017 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.row_cache: None 2024-04-20T17:00:25.017 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.wal_filter: None 2024-04-20T17:00:25.017 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-20T17:00:25.017 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.allow_ingest_behind: 0 2024-04-20T17:00:25.017 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.two_write_queues: 0 2024-04-20T17:00:25.017 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.manual_wal_flush: 0 2024-04-20T17:00:25.017 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.wal_compression: 0 2024-04-20T17:00:25.017 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.atomic_flush: 0 2024-04-20T17:00:25.017 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-20T17:00:25.017 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-20T17:00:25.017 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-20T17:00:25.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.log_readahead_size: 0 2024-04-20T17:00:25.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-20T17:00:25.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.best_efforts_recovery: 0 2024-04-20T17:00:25.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-20T17:00:25.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-20T17:00:25.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.allow_data_in_errors: 0 2024-04-20T17:00:25.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.db_host_id: __hostname__ 2024-04-20T17:00:25.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-20T17:00:25.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.max_background_jobs: 2 2024-04-20T17:00:25.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.max_background_compactions: -1 2024-04-20T17:00:25.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.max_subcompactions: 1 2024-04-20T17:00:25.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-20T17:00:25.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-20T17:00:25.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-20T17:00:25.019 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.max_total_wal_size: 0 2024-04-20T17:00:25.019 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-20T17:00:25.019 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-20T17:00:25.019 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-20T17:00:25.019 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-20T17:00:25.019 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.max_open_files: -1 2024-04-20T17:00:25.019 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.bytes_per_sync: 0 2024-04-20T17:00:25.019 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-20T17:00:25.019 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-20T17:00:25.019 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.compaction_readahead_size: 0 2024-04-20T17:00:25.019 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.max_background_flushes: -1 2024-04-20T17:00:25.019 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Compression algorithms supported: 2024-04-20T17:00:25.019 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-20T17:00:25.019 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: kZSTD supported: 0 2024-04-20T17:00:25.019 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: kXpressCompression supported: 0 2024-04-20T17:00:25.020 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: kLZ4HCCompression supported: 1 2024-04-20T17:00:25.020 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: kZlibCompression supported: 1 2024-04-20T17:00:25.020 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: kSnappyCompression supported: 1 2024-04-20T17:00:25.020 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: kLZ4Compression supported: 1 2024-04-20T17:00:25.020 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: kBZip2Compression supported: 0 2024-04-20T17:00:25.020 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-20T17:00:25.020 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-20T17:00:25.020 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-20T17:00:25.020 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-20T17:00:25.020 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-20T17:00:25.020 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.merge_operator: 2024-04-20T17:00:25.020 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.compaction_filter: None 2024-04-20T17:00:25.020 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.compaction_filter_factory: None 2024-04-20T17:00:25.020 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.sst_partitioner_factory: None 2024-04-20T17:00:25.020 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-20T17:00:25.021 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-20T17:00:25.021 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56095b2c1de0) 2024-04-20T17:00:25.021 INFO:journalctl@ceph.mon.b.smithi106.stdout: cache_index_and_filter_blocks: 1 2024-04-20T17:00:25.021 INFO:journalctl@ceph.mon.b.smithi106.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-20T17:00:25.021 INFO:journalctl@ceph.mon.b.smithi106.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-20T17:00:25.021 INFO:journalctl@ceph.mon.b.smithi106.stdout: pin_top_level_index_and_filter: 1 2024-04-20T17:00:25.021 INFO:journalctl@ceph.mon.b.smithi106.stdout: index_type: 0 2024-04-20T17:00:25.021 INFO:journalctl@ceph.mon.b.smithi106.stdout: data_block_index_type: 0 2024-04-20T17:00:25.021 INFO:journalctl@ceph.mon.b.smithi106.stdout: index_shortening: 1 2024-04-20T17:00:25.021 INFO:journalctl@ceph.mon.b.smithi106.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-20T17:00:25.021 INFO:journalctl@ceph.mon.b.smithi106.stdout: checksum: 4 2024-04-20T17:00:25.021 INFO:journalctl@ceph.mon.b.smithi106.stdout: no_block_cache: 0 2024-04-20T17:00:25.021 INFO:journalctl@ceph.mon.b.smithi106.stdout: block_cache: 0x56095b2871f0 2024-04-20T17:00:25.021 INFO:journalctl@ceph.mon.b.smithi106.stdout: block_cache_name: BinnedLRUCache 2024-04-20T17:00:25.021 INFO:journalctl@ceph.mon.b.smithi106.stdout: block_cache_options: 2024-04-20T17:00:25.022 INFO:journalctl@ceph.mon.b.smithi106.stdout: capacity : 536870912 2024-04-20T17:00:25.022 INFO:journalctl@ceph.mon.b.smithi106.stdout: num_shard_bits : 4 2024-04-20T17:00:25.022 INFO:journalctl@ceph.mon.b.smithi106.stdout: strict_capacity_limit : 0 2024-04-20T17:00:25.022 INFO:journalctl@ceph.mon.b.smithi106.stdout: high_pri_pool_ratio: 0.000 2024-04-20T17:00:25.022 INFO:journalctl@ceph.mon.b.smithi106.stdout: block_cache_compressed: (nil) 2024-04-20T17:00:25.022 INFO:journalctl@ceph.mon.b.smithi106.stdout: persistent_cache: (nil) 2024-04-20T17:00:25.022 INFO:journalctl@ceph.mon.b.smithi106.stdout: block_size: 4096 2024-04-20T17:00:25.022 INFO:journalctl@ceph.mon.b.smithi106.stdout: block_size_deviation: 10 2024-04-20T17:00:25.022 INFO:journalctl@ceph.mon.b.smithi106.stdout: block_restart_interval: 16 2024-04-20T17:00:25.022 INFO:journalctl@ceph.mon.b.smithi106.stdout: index_block_restart_interval: 1 2024-04-20T17:00:25.022 INFO:journalctl@ceph.mon.b.smithi106.stdout: metadata_block_size: 4096 2024-04-20T17:00:25.022 INFO:journalctl@ceph.mon.b.smithi106.stdout: partition_filters: 0 2024-04-20T17:00:25.022 INFO:journalctl@ceph.mon.b.smithi106.stdout: use_delta_encoding: 1 2024-04-20T17:00:25.022 INFO:journalctl@ceph.mon.b.smithi106.stdout: filter_policy: bloomfilter 2024-04-20T17:00:25.023 INFO:journalctl@ceph.mon.b.smithi106.stdout: whole_key_filtering: 1 2024-04-20T17:00:25.023 INFO:journalctl@ceph.mon.b.smithi106.stdout: verify_compression: 0 2024-04-20T17:00:25.023 INFO:journalctl@ceph.mon.b.smithi106.stdout: read_amp_bytes_per_bit: 0 2024-04-20T17:00:25.023 INFO:journalctl@ceph.mon.b.smithi106.stdout: format_version: 5 2024-04-20T17:00:25.023 INFO:journalctl@ceph.mon.b.smithi106.stdout: enable_index_compression: 1 2024-04-20T17:00:25.023 INFO:journalctl@ceph.mon.b.smithi106.stdout: block_align: 0 2024-04-20T17:00:25.023 INFO:journalctl@ceph.mon.b.smithi106.stdout: max_auto_readahead_size: 262144 2024-04-20T17:00:25.023 INFO:journalctl@ceph.mon.b.smithi106.stdout: prepopulate_block_cache: 0 2024-04-20T17:00:25.023 INFO:journalctl@ceph.mon.b.smithi106.stdout: initial_auto_readahead_size: 8192 2024-04-20T17:00:25.023 INFO:journalctl@ceph.mon.b.smithi106.stdout: num_file_reads_for_auto_readahead: 2 2024-04-20T17:00:25.023 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.write_buffer_size: 33554432 2024-04-20T17:00:25.023 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.max_write_buffer_number: 2 2024-04-20T17:00:25.023 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.compression: NoCompression 2024-04-20T17:00:25.023 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.bottommost_compression: Disabled 2024-04-20T17:00:25.023 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.prefix_extractor: nullptr 2024-04-20T17:00:25.023 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-20T17:00:25.024 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.num_levels: 7 2024-04-20T17:00:25.024 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-20T17:00:25.024 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-20T17:00:25.024 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-20T17:00:25.024 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-20T17:00:25.024 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-20T17:00:25.024 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-20T17:00:25.024 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-20T17:00:25.024 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-20T17:00:25.024 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-20T17:00:25.024 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-20T17:00:25.024 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-20T17:00:25.024 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-20T17:00:25.024 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-20T17:00:25.024 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.compression_opts.level: 32767 2024-04-20T17:00:25.025 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.compression_opts.strategy: 0 2024-04-20T17:00:25.025 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-20T17:00:25.025 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-20T17:00:25.025 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-20T17:00:25.025 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-20T17:00:25.025 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.compression_opts.enabled: false 2024-04-20T17:00:25.025 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-20T17:00:25.025 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-20T17:00:25.025 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-20T17:00:25.025 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-20T17:00:25.025 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.target_file_size_base: 67108864 2024-04-20T17:00:25.025 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-20T17:00:25.025 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-20T17:00:25.025 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-20T17:00:25.025 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-20T17:00:25.026 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-20T17:00:25.026 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-20T17:00:25.026 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-20T17:00:25.026 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-20T17:00:25.026 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-20T17:00:25.026 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-20T17:00:25.026 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-20T17:00:25.026 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-20T17:00:25.026 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-20T17:00:25.026 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-20T17:00:25.026 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.arena_block_size: 1048576 2024-04-20T17:00:25.026 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-20T17:00:25.026 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-20T17:00:25.026 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.disable_auto_compactions: 0 2024-04-20T17:00:25.026 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-20T17:00:25.027 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-20T17:00:25.027 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-20T17:00:25.027 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-20T17:00:25.027 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-20T17:00:25.027 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-20T17:00:25.027 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-20T17:00:25.027 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-20T17:00:25.027 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-20T17:00:25.027 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-20T17:00:25.027 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-20T17:00:25.027 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.inplace_update_support: 0 2024-04-20T17:00:25.027 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-20T17:00:25.027 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-20T17:00:25.027 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-20T17:00:25.027 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-20T17:00:25.028 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.bloom_locality: 0 2024-04-20T17:00:25.028 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.max_successive_merges: 0 2024-04-20T17:00:25.028 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-20T17:00:25.028 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.paranoid_file_checks: 0 2024-04-20T17:00:25.028 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.force_consistency_checks: 1 2024-04-20T17:00:25.028 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.report_bg_io_stats: 0 2024-04-20T17:00:25.028 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.ttl: 2592000 2024-04-20T17:00:25.028 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-20T17:00:25.028 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-20T17:00:25.028 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-20T17:00:25.028 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.enable_blob_files: false 2024-04-20T17:00:25.028 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.min_blob_size: 0 2024-04-20T17:00:25.028 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.blob_file_size: 268435456 2024-04-20T17:00:25.028 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-20T17:00:25.028 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-20T17:00:25.029 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-20T17:00:25.029 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-20T17:00:25.029 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-20T17:00:25.029 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.blob_file_starting_level: 0 2024-04-20T17:00:25.029 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-20T17:00:25.029 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-20T17:00:25.029 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-20T17:00:25.029 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 6665de69-9db3-44f1-b3f5-604972490afc 2024-04-20T17:00:25.029 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713632424624950, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-20T17:00:25.029 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-20T17:00:25.029 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713632424625482, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1703, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 578, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 456, "raw_average_value_size": 91, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713632424, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "6665de69-9db3-44f1-b3f5-604972490afc", "db_session_id": "LNUQV23XDNX6LPACL868", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-20T17:00:25.029 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713632424625583, "job": 1, "event": "recovery_finished"} 2024-04-20T17:00:25.029 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-20T17:00:25.029 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-20T17:00:25.029 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56095b39c000 2024-04-20T17:00:25.030 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: DB pointer 0x56095b386000 2024-04-20T17:00:25.030 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-20T17:00:25.030 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-20T17:00:25.030 INFO:journalctl@ceph.mon.b.smithi106.stdout: ** DB Stats ** 2024-04-20T17:00:25.030 INFO:journalctl@ceph.mon.b.smithi106.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-20T17:00:25.030 INFO:journalctl@ceph.mon.b.smithi106.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-20T17:00:25.030 INFO:journalctl@ceph.mon.b.smithi106.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-20T17:00:25.030 INFO:journalctl@ceph.mon.b.smithi106.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-20T17:00:25.030 INFO:journalctl@ceph.mon.b.smithi106.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-20T17:00:25.030 INFO:journalctl@ceph.mon.b.smithi106.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-20T17:00:25.030 INFO:journalctl@ceph.mon.b.smithi106.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-20T17:00:25.030 INFO:journalctl@ceph.mon.b.smithi106.stdout: 2024-04-20T17:00:25.030 INFO:journalctl@ceph.mon.b.smithi106.stdout: ** Compaction Stats [default] ** 2024-04-20T17:00:25.030 INFO:journalctl@ceph.mon.b.smithi106.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-20T17:00:25.030 INFO:journalctl@ceph.mon.b.smithi106.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-20T17:00:25.031 INFO:journalctl@ceph.mon.b.smithi106.stdout: L0 1/0 1.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-20T17:00:25.031 INFO:journalctl@ceph.mon.b.smithi106.stdout: Sum 1/0 1.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-20T17:00:25.031 INFO:journalctl@ceph.mon.b.smithi106.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-20T17:00:25.031 INFO:journalctl@ceph.mon.b.smithi106.stdout: 2024-04-20T17:00:25.031 INFO:journalctl@ceph.mon.b.smithi106.stdout: ** Compaction Stats [default] ** 2024-04-20T17:00:25.031 INFO:journalctl@ceph.mon.b.smithi106.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-20T17:00:25.031 INFO:journalctl@ceph.mon.b.smithi106.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-20T17:00:25.031 INFO:journalctl@ceph.mon.b.smithi106.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-20T17:00:25.031 INFO:journalctl@ceph.mon.b.smithi106.stdout: 2024-04-20T17:00:25.031 INFO:journalctl@ceph.mon.b.smithi106.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-20T17:00:25.031 INFO:journalctl@ceph.mon.b.smithi106.stdout: 2024-04-20T17:00:25.031 INFO:journalctl@ceph.mon.b.smithi106.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-20T17:00:25.031 INFO:journalctl@ceph.mon.b.smithi106.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-20T17:00:25.031 INFO:journalctl@ceph.mon.b.smithi106.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-20T17:00:25.031 INFO:journalctl@ceph.mon.b.smithi106.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-20T17:00:25.032 INFO:journalctl@ceph.mon.b.smithi106.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-20T17:00:25.032 INFO:journalctl@ceph.mon.b.smithi106.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-20T17:00:25.032 INFO:journalctl@ceph.mon.b.smithi106.stdout: Cumulative compaction: 0.00 GB write, 0.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-20T17:00:25.032 INFO:journalctl@ceph.mon.b.smithi106.stdout: Interval compaction: 0.00 GB write, 0.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-20T17:00:25.032 INFO:journalctl@ceph.mon.b.smithi106.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-20T17:00:25.032 INFO:journalctl@ceph.mon.b.smithi106.stdout: Block cache BinnedLRUCache@0x56095b2871f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-04-20T17:00:25.032 INFO:journalctl@ceph.mon.b.smithi106.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-20T17:00:25.032 INFO:journalctl@ceph.mon.b.smithi106.stdout: 2024-04-20T17:00:25.032 INFO:journalctl@ceph.mon.b.smithi106.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-20T17:00:25.032 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-20T17:00:25.032 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: using public_addrv v1:172.21.15.106:6789/0 2024-04-20T17:00:25.032 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: starting mon.b rank -1 at public addrs v1:172.21.15.106:6789/0 at bind addrs v1:172.21.15.106:6789/0 mon_data /var/lib/ceph/mon/ceph-b fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T17:00:25.032 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: mon.b@-1(???) e0 preinit fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T17:00:25.032 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #13. Immutable memtables: 0. 2024-04-20T17:00:25.032 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: (Original Log Time 2024/04/20-17:00:24.643448) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-04-20T17:00:25.033 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: [db/flush_job.cc:856] [default] [JOB 3] Flushing memtable with next log file: 13 2024-04-20T17:00:25.033 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713632424643576, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1190, "num_deletes": 20, "total_data_size": 2315374, "memory_usage": 2344080, "flush_reason": "Manual Compaction"} 2024-04-20T17:00:25.033 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: [db/flush_job.cc:885] [default] [JOB 3] Level-0 flush table #14: started 2024-04-20T17:00:25.033 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: mon.b@-1(synchronizing).mds e1 new map 2024-04-20T17:00:25.033 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-20T17:00:25.033 INFO:journalctl@ceph.mon.b.smithi106.stdout: e1 2024-04-20T17:00:25.033 INFO:journalctl@ceph.mon.b.smithi106.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-20T17:00:25.033 INFO:journalctl@ceph.mon.b.smithi106.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-20T17:00:25.033 INFO:journalctl@ceph.mon.b.smithi106.stdout: legacy client fscid: -1 2024-04-20T17:00:25.033 INFO:journalctl@ceph.mon.b.smithi106.stdout: 2024-04-20T17:00:25.033 INFO:journalctl@ceph.mon.b.smithi106.stdout: No filesystems configured 2024-04-20T17:00:25.033 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-20T17:00:25.033 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-20T17:00:25.033 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713632424646631, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 14, "file_size": 2035440, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 6, "largest_seqno": 1195, "table_properties": {"data_size": 2029480, "index_size": 2652, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2245, "raw_key_size": 27654, "raw_average_key_size": 30, "raw_value_size": 2010832, "raw_average_value_size": 2251, "num_data_blocks": 123, "num_entries": 893, "num_filter_entries": 893, "num_deletions": 20, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713632424, "oldest_key_time": 1713632424, "file_creation_time": 1713632424, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "6665de69-9db3-44f1-b3f5-604972490afc", "db_session_id": "LNUQV23XDNX6LPACL868", "orig_file_number": 14, "seqno_to_time_mapping": "N/A"}} 2024-04-20T17:00:25.033 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: [db/flush_job.cc:1019] [default] [JOB 3] Flush lasted 3211 microseconds, and 2273 cpu microseconds. 2024-04-20T17:00:25.034 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-20T17:00:25.034 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-20T17:00:25.034 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-20T17:00:25.034 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-20T17:00:25.034 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-20T17:00:25.034 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-20T17:00:25.034 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-20T17:00:25.034 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-20T17:00:25.034 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: (Original Log Time 2024/04/20-17:00:24.646660) [db/flush_job.cc:967] [default] [JOB 3] Level-0 flush table #14: 2035440 bytes OK 2024-04-20T17:00:25.034 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: (Original Log Time 2024/04/20-17:00:24.646677) [db/memtable_list.cc:519] [default] Level-0 commit table #14 started 2024-04-20T17:00:25.034 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: (Original Log Time 2024/04/20-17:00:24.647066) [db/memtable_list.cc:722] [default] Level-0 commit table #14: memtable #1 done 2024-04-20T17:00:25.034 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: (Original Log Time 2024/04/20-17:00:24.647077) EVENT_LOG_v1 {"time_micros": 1713632424647074, "job": 3, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [2, 0, 0, 0, 0, 0, 0], "immutable_memtables": 0} 2024-04-20T17:00:25.034 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: (Original Log Time 2024/04/20-17:00:24.647086) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: files[2 0 0 0 0 0 0] max score 0.50 2024-04-20T17:00:25.034 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 3] Try to delete WAL files size 2307072, prev total WAL file size 2307295, number of live WAL files 2. 2024-04-20T17:00:25.034 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: Added host smithi106 2024-04-20T17:00:25.035 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:25.035 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:25.035 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='client.14176 v1:172.21.15.5:0/1627713615' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:00:25.035 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:25.035 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:25.035 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:25.035 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:25.035 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:25.035 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi106", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:00:25.035 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:25.035 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:00:25.035 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: Updating smithi106:/etc/ceph/ceph.conf 2024-04-20T17:00:25.035 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: Updating smithi106:/var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T17:00:25.035 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:25.036 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:25.036 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:25.036 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:25.036 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: Updating smithi106:/etc/ceph/ceph.client.admin.keyring 2024-04-20T17:00:25.036 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: Updating smithi106:/var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-20T17:00:25.036 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='client.14178 v1:172.21.15.5:0/3328672003' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi171", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:00:25.036 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: Deploying cephadm binary to smithi171 2024-04-20T17:00:25.036 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:25.036 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:25.036 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:25.036 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:00:25.036 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: Added host smithi171 2024-04-20T17:00:25.036 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:25.036 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:25.037 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:25.037 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:25.037 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-20T17:00:25.037 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='client.14180 v1:172.21.15.5:0/3061992227' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:00:25.037 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:25.037 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:25.037 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:25.037 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:25.037 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:25.037 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi171", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:00:25.037 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:25.037 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:00:25.037 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: Updating smithi171:/etc/ceph/ceph.conf 2024-04-20T17:00:25.037 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:25.037 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:25.037 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:25.038 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:25.038 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: Updating smithi171:/var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T17:00:25.038 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: Updating smithi171:/etc/ceph/ceph.client.admin.keyring 2024-04-20T17:00:25.038 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: Updating smithi171:/var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-20T17:00:25.038 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/340361140' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-20T17:00:25.038 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/340361140' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-20T17:00:25.038 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: osdmap e4: 0 total, 0 up, 0 in 2024-04-20T17:00:25.039 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:25.039 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: (Original Log Time 2024/04/20-17:00:24.647529) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '6D6772737461740030' seq:72057594037927935, type:22 .. '6D677273746174003131' seq:0, type:0; will stop at (end) 2024-04-20T17:00:25.039 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 4] Compacting 2@0 files to L6, score -1.00 2024-04-20T17:00:25.039 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:25.039 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 3 Base level 0, inputs: [14(1987KB) 8(1703B)] 2024-04-20T17:00:25.039 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:25.039 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713632424647577, "job": 4, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [14, 8], "score": -1, "input_data_size": 2037143, "oldest_snapshot_seqno": -1} 2024-04-20T17:00:25.039 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:00:25.039 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:25.039 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:00:25.039 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:25.039 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-20T17:00:25.039 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:25.039 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='client.14184 v1:172.21.15.171:0/684140253' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi005:[v1:172.21.15.5:6789]=a;smithi106:[v1:172.21.15.106:6789]=b;smithi171:[v1:172.21.15.171:6789]=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:00:25.040 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: Saving service mon spec with placement smithi005:[v1:172.21.15.5:6789]=a;smithi106:[v1:172.21.15.106:6789]=b;smithi171:[v1:172.21.15.171:6789]=c;count:3 2024-04-20T17:00:25.040 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: Deploying daemon mon.c on smithi171 2024-04-20T17:00:25.040 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:25.040 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:25.040 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:25.040 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:00:25.040 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: mon.a calling monitor election 2024-04-20T17:00:25.040 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:00:25.040 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.171:0/3214808924' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T17:00:25.040 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:00:25.040 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:25.040 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:00:25.040 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: mon.c calling monitor election 2024-04-20T17:00:25.040 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:00:25.040 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:25.041 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:00:25.041 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:00:25.041 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-20T17:00:25.041 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: monmap epoch 2 2024-04-20T17:00:25.041 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T17:00:25.041 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: last_changed 2024-04-20T17:00:15.907056+0000 2024-04-20T17:00:25.041 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: created 2024-04-20T16:58:22.127736+0000 2024-04-20T17:00:25.041 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: min_mon_release 19 (squid) 2024-04-20T17:00:25.041 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: election_strategy: 1 2024-04-20T17:00:25.041 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: 0: v1:172.21.15.5:6789/0 mon.a 2024-04-20T17:00:25.041 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: 1: v1:172.21.15.171:6789/0 mon.c 2024-04-20T17:00:25.041 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: fsmap 2024-04-20T17:00:25.041 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: osdmap e4: 0 total, 0 up, 0 in 2024-04-20T17:00:25.041 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: mgrmap e13: y(active, since 55s) 2024-04-20T17:00:25.041 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: overall HEALTH_OK 2024-04-20T17:00:25.042 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:25.042 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:25.042 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:25.042 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-20T17:00:25.042 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:25.042 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: Deploying daemon mon.b on smithi106 2024-04-20T17:00:25.042 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:00:25.042 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:25.042 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 ceph-mon[44423]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-20T17:00:25.042 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 bash[44385]: a46b95169e18f08ed9aea1047d0a7f9b8eab4f4ca797043b21e74158184e65fa 2024-04-20T17:00:25.042 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:24 smithi106 systemd[1]: Started Ceph mon.b for 13c6e76c-ff37-11ee-bc93-c7b262605968. 2024-04-20T17:00:30.008 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-20T17:00:30.008 INFO:teuthology.orchestra.run.smithi171.stdout:{"epoch":3,"fsid":"13c6e76c-ff37-11ee-bc93-c7b262605968","modified":"2024-04-20T17:00:24.652511Z","created":"2024-04-20T16:58:22.127736Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6789","nonce":0}]},"addr":"172.21.15.106:6789/0","public_addr":"172.21.15.106:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-20T17:00:30.010 INFO:teuthology.orchestra.run.smithi171.stderr:dumped monmap epoch 3 2024-04-20T17:00:30.038 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:30.038 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:00:30.038 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: mon.a calling monitor election 2024-04-20T17:00:30.039 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: mon.c calling monitor election 2024-04-20T17:00:30.039 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:00:30.039 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:00:30.039 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:00:30.039 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:30.039 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:00:30.039 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:00:30.039 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:30.039 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:00:30.039 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:00:30.039 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-20T17:00:30.039 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: monmap epoch 3 2024-04-20T17:00:30.039 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T17:00:30.039 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: last_changed 2024-04-20T17:00:24.652511+0000 2024-04-20T17:00:30.039 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: created 2024-04-20T16:58:22.127736+0000 2024-04-20T17:00:30.040 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: min_mon_release 19 (squid) 2024-04-20T17:00:30.040 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: election_strategy: 1 2024-04-20T17:00:30.040 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: 0: v1:172.21.15.5:6789/0 mon.a 2024-04-20T17:00:30.040 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: 1: v1:172.21.15.171:6789/0 mon.c 2024-04-20T17:00:30.040 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: 2: v1:172.21.15.106:6789/0 mon.b 2024-04-20T17:00:30.040 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: fsmap 2024-04-20T17:00:30.040 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: osdmap e4: 0 total, 0 up, 0 in 2024-04-20T17:00:30.040 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: mgrmap e13: y(active, since 63s) 2024-04-20T17:00:30.040 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-20T17:00:30.040 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-20T17:00:30.040 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-20T17:00:30.040 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: mon.b (rank 2) addr v1:172.21.15.106:6789/0 is down (out of quorum) 2024-04-20T17:00:30.040 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:30.040 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:29 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:00:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: mon.a calling monitor election 2024-04-20T17:00:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: mon.c calling monitor election 2024-04-20T17:00:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:00:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:00:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:00:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:00:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:00:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:00:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:00:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-20T17:00:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: monmap epoch 3 2024-04-20T17:00:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T17:00:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: last_changed 2024-04-20T17:00:24.652511+0000 2024-04-20T17:00:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: created 2024-04-20T16:58:22.127736+0000 2024-04-20T17:00:30.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: min_mon_release 19 (squid) 2024-04-20T17:00:30.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: election_strategy: 1 2024-04-20T17:00:30.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: 0: v1:172.21.15.5:6789/0 mon.a 2024-04-20T17:00:30.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: 1: v1:172.21.15.171:6789/0 mon.c 2024-04-20T17:00:30.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: 2: v1:172.21.15.106:6789/0 mon.b 2024-04-20T17:00:30.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: fsmap 2024-04-20T17:00:30.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: osdmap e4: 0 total, 0 up, 0 in 2024-04-20T17:00:30.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: mgrmap e13: y(active, since 63s) 2024-04-20T17:00:30.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-20T17:00:30.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-20T17:00:30.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-20T17:00:30.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: mon.b (rank 2) addr v1:172.21.15.106:6789/0 is down (out of quorum) 2024-04-20T17:00:30.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:30.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:29 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:30.591 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-20T17:00:30.592 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-20T17:00:30.862 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:00:30.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:30 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:30.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:30 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:30.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:30 smithi005 ceph-mon[39562]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:30.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:30 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:00:30.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:30 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.171:0/1767571827' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T17:00:30.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:30 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:00:30.974 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:30 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:30.974 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:30 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:30.975 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:30 smithi171 ceph-mon[44189]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:30.975 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:30 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:00:30.975 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:30 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.171:0/1767571827' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T17:00:30.975 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:30 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:00:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:00:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: mon.a calling monitor election 2024-04-20T17:00:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: mon.c calling monitor election 2024-04-20T17:00:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:00:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:00:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:00:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:32.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:00:32.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:00:32.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:32.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:00:32.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:00:32.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-20T17:00:32.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: monmap epoch 3 2024-04-20T17:00:32.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T17:00:32.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: last_changed 2024-04-20T17:00:24.652511+0000 2024-04-20T17:00:32.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: created 2024-04-20T16:58:22.127736+0000 2024-04-20T17:00:32.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: min_mon_release 19 (squid) 2024-04-20T17:00:32.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: election_strategy: 1 2024-04-20T17:00:32.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: 0: v1:172.21.15.5:6789/0 mon.a 2024-04-20T17:00:32.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: 1: v1:172.21.15.171:6789/0 mon.c 2024-04-20T17:00:32.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: 2: v1:172.21.15.106:6789/0 mon.b 2024-04-20T17:00:32.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: fsmap 2024-04-20T17:00:32.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: osdmap e4: 0 total, 0 up, 0 in 2024-04-20T17:00:32.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: mgrmap e13: y(active, since 63s) 2024-04-20T17:00:32.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-20T17:00:32.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-20T17:00:32.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-20T17:00:32.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: mon.b (rank 2) addr v1:172.21.15.106:6789/0 is down (out of quorum) 2024-04-20T17:00:32.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:32.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:00:32.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.171:0/1767571827' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T17:00:32.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:31 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:00:32.548 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: mon.b calling monitor election 2024-04-20T17:00:32.549 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: Updating smithi005:/etc/ceph/ceph.conf 2024-04-20T17:00:32.549 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: Updating smithi106:/etc/ceph/ceph.conf 2024-04-20T17:00:32.549 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: Updating smithi171:/etc/ceph/ceph.conf 2024-04-20T17:00:32.549 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: Updating smithi171:/var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T17:00:32.549 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: Updating smithi106:/var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T17:00:32.549 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: Updating smithi005:/var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T17:00:32.549 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: mon.b calling monitor election 2024-04-20T17:00:32.549 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: mon.c calling monitor election 2024-04-20T17:00:32.549 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: mon.a calling monitor election 2024-04-20T17:00:32.549 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-20T17:00:32.549 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: monmap epoch 3 2024-04-20T17:00:32.550 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T17:00:32.550 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: last_changed 2024-04-20T17:00:24.652511+0000 2024-04-20T17:00:32.550 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: created 2024-04-20T16:58:22.127736+0000 2024-04-20T17:00:32.550 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: min_mon_release 19 (squid) 2024-04-20T17:00:32.550 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: election_strategy: 1 2024-04-20T17:00:32.550 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: 0: v1:172.21.15.5:6789/0 mon.a 2024-04-20T17:00:32.550 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: 1: v1:172.21.15.171:6789/0 mon.c 2024-04-20T17:00:32.550 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: 2: v1:172.21.15.106:6789/0 mon.b 2024-04-20T17:00:32.550 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: fsmap 2024-04-20T17:00:32.550 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: osdmap e4: 0 total, 0 up, 0 in 2024-04-20T17:00:32.550 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: mgrmap e13: y(active, since 65s) 2024-04-20T17:00:32.550 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-20T17:00:32.550 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:00:32.551 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: overall HEALTH_OK 2024-04-20T17:00:32.551 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.551 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.551 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.551 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.551 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.551 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.551 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.551 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.551 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.551 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.551 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.551 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-20T17:00:32.551 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-20T17:00:32.551 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: mon.b calling monitor election 2024-04-20T17:00:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: Updating smithi005:/etc/ceph/ceph.conf 2024-04-20T17:00:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: Updating smithi106:/etc/ceph/ceph.conf 2024-04-20T17:00:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: Updating smithi171:/etc/ceph/ceph.conf 2024-04-20T17:00:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: Updating smithi171:/var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T17:00:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: Updating smithi106:/var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T17:00:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: Updating smithi005:/var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T17:00:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: mon.b calling monitor election 2024-04-20T17:00:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: mon.c calling monitor election 2024-04-20T17:00:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: mon.a calling monitor election 2024-04-20T17:00:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-20T17:00:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: monmap epoch 3 2024-04-20T17:00:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T17:00:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: last_changed 2024-04-20T17:00:24.652511+0000 2024-04-20T17:00:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: created 2024-04-20T16:58:22.127736+0000 2024-04-20T17:00:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: min_mon_release 19 (squid) 2024-04-20T17:00:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: election_strategy: 1 2024-04-20T17:00:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: 0: v1:172.21.15.5:6789/0 mon.a 2024-04-20T17:00:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: 1: v1:172.21.15.171:6789/0 mon.c 2024-04-20T17:00:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: 2: v1:172.21.15.106:6789/0 mon.b 2024-04-20T17:00:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: fsmap 2024-04-20T17:00:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: osdmap e4: 0 total, 0 up, 0 in 2024-04-20T17:00:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: mgrmap e13: y(active, since 65s) 2024-04-20T17:00:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-20T17:00:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:00:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: overall HEALTH_OK 2024-04-20T17:00:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-20T17:00:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-20T17:00:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:32.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: mon.b calling monitor election 2024-04-20T17:00:32.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: Updating smithi005:/etc/ceph/ceph.conf 2024-04-20T17:00:32.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: Updating smithi106:/etc/ceph/ceph.conf 2024-04-20T17:00:32.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: Updating smithi171:/etc/ceph/ceph.conf 2024-04-20T17:00:32.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: Updating smithi171:/var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T17:00:32.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: Updating smithi106:/var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T17:00:32.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: Updating smithi005:/var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-20T17:00:32.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: mon.b calling monitor election 2024-04-20T17:00:32.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: mon.c calling monitor election 2024-04-20T17:00:32.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: mon.a calling monitor election 2024-04-20T17:00:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-20T17:00:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: monmap epoch 3 2024-04-20T17:00:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T17:00:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: last_changed 2024-04-20T17:00:24.652511+0000 2024-04-20T17:00:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: created 2024-04-20T16:58:22.127736+0000 2024-04-20T17:00:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: min_mon_release 19 (squid) 2024-04-20T17:00:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: election_strategy: 1 2024-04-20T17:00:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: 0: v1:172.21.15.5:6789/0 mon.a 2024-04-20T17:00:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: 1: v1:172.21.15.171:6789/0 mon.c 2024-04-20T17:00:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: 2: v1:172.21.15.106:6789/0 mon.b 2024-04-20T17:00:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: fsmap 2024-04-20T17:00:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: osdmap e4: 0 total, 0 up, 0 in 2024-04-20T17:00:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: mgrmap e13: y(active, since 65s) 2024-04-20T17:00:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-20T17:00:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:00:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: overall HEALTH_OK 2024-04-20T17:00:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-20T17:00:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-20T17:00:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:33.019 INFO:teuthology.orchestra.run.smithi005.stdout:# minimal ceph.conf for 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T17:00:33.019 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2024-04-20T17:00:33.019 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T17:00:33.020 INFO:teuthology.orchestra.run.smithi005.stdout: mon_host = 172.21.15.5:6789/0 172.21.15.106:6789/0 172.21.15.171:6789/0 2024-04-20T17:00:33.862 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:33 smithi005 ceph-mon[39562]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:33.862 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:33 smithi005 ceph-mon[39562]: Reconfiguring mon.a (unknown last config time)... 2024-04-20T17:00:33.863 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:33 smithi005 ceph-mon[39562]: Reconfiguring daemon mon.a on smithi005 2024-04-20T17:00:33.863 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:33 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:00:33.863 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:33 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/245273249' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:34.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:33 smithi106 ceph-mon[44423]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:33 smithi106 ceph-mon[44423]: Reconfiguring mon.a (unknown last config time)... 2024-04-20T17:00:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:33 smithi106 ceph-mon[44423]: Reconfiguring daemon mon.a on smithi005 2024-04-20T17:00:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:33 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:00:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:33 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/245273249' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:33 smithi171 ceph-mon[44189]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:33 smithi171 ceph-mon[44189]: Reconfiguring mon.a (unknown last config time)... 2024-04-20T17:00:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:33 smithi171 ceph-mon[44189]: Reconfiguring daemon mon.a on smithi005 2024-04-20T17:00:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:33 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:00:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:33 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/245273249' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:34.703 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-20T17:00:34.703 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-04-20T17:00:34.704 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/ceph/ceph.conf 2024-04-20T17:00:34.738 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-04-20T17:00:34.738 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-20T17:00:34.808 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-20T17:00:34.808 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/ceph.conf 2024-04-20T17:00:34.838 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-20T17:00:34.838 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-20T17:00:34.907 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-04-20T17:00:34.907 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/ceph/ceph.conf 2024-04-20T17:00:34.944 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-04-20T17:00:34.944 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-20T17:00:35.015 INFO:tasks.cephadm:Adding mgr.y on smithi005 2024-04-20T17:00:35.015 INFO:tasks.cephadm:Adding mgr.x on smithi106 2024-04-20T17:00:35.015 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph orch apply mgr '2;smithi005=y;smithi106=x' 2024-04-20T17:00:35.315 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:00:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:35 smithi005 ceph-mon[39562]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:35 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:35 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:35 smithi005 ceph-mon[39562]: Reconfiguring mon.b (monmap changed)... 2024-04-20T17:00:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:35 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-20T17:00:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:35 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-20T17:00:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:35 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:35 smithi005 ceph-mon[39562]: Reconfiguring daemon mon.b on smithi106 2024-04-20T17:00:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:35 smithi106 ceph-mon[44423]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:35 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:35 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:35 smithi106 ceph-mon[44423]: Reconfiguring mon.b (monmap changed)... 2024-04-20T17:00:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:35 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-20T17:00:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:35 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-20T17:00:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:35 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:35 smithi106 ceph-mon[44423]: Reconfiguring daemon mon.b on smithi106 2024-04-20T17:00:35.525 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:35 smithi171 ceph-mon[44189]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:35.526 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:35 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:35.526 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:35 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:35.526 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:35 smithi171 ceph-mon[44189]: Reconfiguring mon.b (monmap changed)... 2024-04-20T17:00:35.526 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:35 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-20T17:00:35.526 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:35 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-20T17:00:35.526 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:35 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:35.526 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:35 smithi171 ceph-mon[44189]: Reconfiguring daemon mon.b on smithi106 2024-04-20T17:00:36.782 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:36 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:36.782 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:36 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:36.782 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:36 smithi171 ceph-mon[44189]: Reconfiguring mon.c (monmap changed)... 2024-04-20T17:00:36.782 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:36 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-20T17:00:36.782 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:36 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-20T17:00:36.782 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:36 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:36.782 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:36 smithi171 ceph-mon[44189]: Reconfiguring daemon mon.c on smithi171 2024-04-20T17:00:36.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:36 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:36.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:36 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:36.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:36 smithi005 ceph-mon[39562]: Reconfiguring mon.c (monmap changed)... 2024-04-20T17:00:36.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:36 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-20T17:00:36.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:36 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-20T17:00:36.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:36 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:36.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:36 smithi005 ceph-mon[39562]: Reconfiguring daemon mon.c on smithi171 2024-04-20T17:00:37.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:36 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:37.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:36 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:36 smithi106 ceph-mon[44423]: Reconfiguring mon.c (monmap changed)... 2024-04-20T17:00:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:36 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-20T17:00:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:36 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-20T17:00:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:36 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:36 smithi106 ceph-mon[44423]: Reconfiguring daemon mon.c on smithi171 2024-04-20T17:00:37.317 INFO:teuthology.orchestra.run.smithi171.stdout:Scheduled mgr update... 2024-04-20T17:00:37.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:37 smithi005 ceph-mon[39562]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:37.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:37 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:38.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:37 smithi106 ceph-mon[44423]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:37 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:38.060 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:37 smithi171 ceph-mon[44189]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:38.060 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:37 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:38.782 DEBUG:teuthology.orchestra.run.smithi106:mgr.x> sudo journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@mgr.x.service 2024-04-20T17:00:38.784 INFO:tasks.cephadm:Deploying OSDs... 2024-04-20T17:00:38.784 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-04-20T17:00:38.784 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/scratch_devs of=/dev/stdout 2024-04-20T17:00:38.807 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-20T17:00:38.807 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_1 2024-04-20T17:00:38.812 INFO:journalctl@ceph.mgr.x.smithi106.stdout:-- Logs begin at Sat 2024-04-20 16:47:44 UTC. -- 2024-04-20T17:00:38.869 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-20T17:00:38.869 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T17:00:38.870 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 6h/6d Inode: 167878 Links: 1 2024-04-20T17:00:38.870 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T17:00:38.870 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T17:00:38.870 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-04-20 16:59:40.774137599 +0000 2024-04-20T17:00:38.870 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-04-20 16:53:42.802845475 +0000 2024-04-20T17:00:38.870 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-04-20 16:53:42.802845475 +0000 2024-04-20T17:00:38.870 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2024-04-20T17:00:38.870 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-20T17:00:38.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:38 smithi005 ceph-mon[39562]: from='client.24103 v1:172.21.15.171:0/1371852307' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi005=y;smithi106=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:00:38.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:38 smithi005 ceph-mon[39562]: Saving service mgr spec with placement smithi005=y;smithi106=x;count:2 2024-04-20T17:00:38.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:38.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:38.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:00:38.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:38.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:00:38.942 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:38.942 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-20T17:00:38.942 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-20T17:00:38.942 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-20T17:00:38.942 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:38.943 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-04-20T17:00:38.943 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-04-20T17:00:38.943 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000111825 s, 4.6 MB/s 2024-04-20T17:00:38.943 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-20T17:00:39.005 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_2 2024-04-20T17:00:39.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:38 smithi106 ceph-mon[44423]: from='client.24103 v1:172.21.15.171:0/1371852307' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi005=y;smithi106=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:00:39.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:38 smithi106 ceph-mon[44423]: Saving service mgr spec with placement smithi005=y;smithi106=x;count:2 2024-04-20T17:00:39.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:00:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:00:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-20T17:00:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-20T17:00:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-20T17:00:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:39.066 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-20T17:00:39.066 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T17:00:39.067 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 6h/6d Inode: 166497 Links: 1 2024-04-20T17:00:39.067 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T17:00:39.067 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T17:00:39.067 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-04-20 16:59:40.819136644 +0000 2024-04-20T17:00:39.067 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-04-20 16:53:43.133838424 +0000 2024-04-20T17:00:39.067 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-04-20 16:53:43.133838424 +0000 2024-04-20T17:00:39.067 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2024-04-20T17:00:39.067 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-20T17:00:39.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:38 smithi171 ceph-mon[44189]: from='client.24103 v1:172.21.15.171:0/1371852307' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi005=y;smithi106=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:00:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:38 smithi171 ceph-mon[44189]: Saving service mgr spec with placement smithi005=y;smithi106=x;count:2 2024-04-20T17:00:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:00:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:00:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-20T17:00:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-20T17:00:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-20T17:00:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:39.136 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-04-20T17:00:39.136 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-04-20T17:00:39.136 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 9.9311e-05 s, 5.2 MB/s 2024-04-20T17:00:39.137 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-20T17:00:39.198 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_3 2024-04-20T17:00:39.258 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-20T17:00:39.258 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T17:00:39.258 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 6h/6d Inode: 165813 Links: 1 2024-04-20T17:00:39.258 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T17:00:39.258 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T17:00:39.258 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-04-20 16:59:40.861135753 +0000 2024-04-20T17:00:39.258 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-04-20 16:53:43.483830968 +0000 2024-04-20T17:00:39.258 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-04-20 16:53:43.483830968 +0000 2024-04-20T17:00:39.258 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2024-04-20T17:00:39.259 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-20T17:00:39.326 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-04-20T17:00:39.326 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-04-20T17:00:39.326 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 9.3473e-05 s, 5.5 MB/s 2024-04-20T17:00:39.327 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-20T17:00:39.387 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_4 2024-04-20T17:00:39.446 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-20T17:00:39.446 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T17:00:39.446 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 6h/6d Inode: 170239 Links: 1 2024-04-20T17:00:39.447 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T17:00:39.447 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T17:00:39.447 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-04-20 16:59:40.907134777 +0000 2024-04-20T17:00:39.447 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-04-20 16:53:43.871822702 +0000 2024-04-20T17:00:39.447 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-04-20 16:53:43.871822702 +0000 2024-04-20T17:00:39.447 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2024-04-20T17:00:39.447 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-20T17:00:39.514 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-04-20T17:00:39.514 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-04-20T17:00:39.514 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000102626 s, 5.0 MB/s 2024-04-20T17:00:39.515 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-20T17:00:39.575 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-20T17:00:39.575 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/scratch_devs of=/dev/stdout 2024-04-20T17:00:39.593 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-20T17:00:39.594 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_1 2024-04-20T17:00:39.651 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-20T17:00:39.651 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T17:00:39.652 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 6h/6d Inode: 169640 Links: 1 2024-04-20T17:00:39.652 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T17:00:39.652 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T17:00:39.652 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-04-20 17:00:39.639455011 +0000 2024-04-20T17:00:39.652 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-04-20 16:53:31.235821859 +0000 2024-04-20T17:00:39.652 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-04-20 16:53:31.235821859 +0000 2024-04-20T17:00:39.652 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: - 2024-04-20T17:00:39.652 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-20T17:00:39.720 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-04-20T17:00:39.720 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-04-20T17:00:39.720 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000262426 s, 2.0 MB/s 2024-04-20T17:00:39.721 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-20T17:00:39.780 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_2 2024-04-20T17:00:39.841 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-20T17:00:39.841 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T17:00:39.841 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 6h/6d Inode: 171043 Links: 1 2024-04-20T17:00:39.841 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T17:00:39.841 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T17:00:39.842 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-04-20 17:00:39.774451853 +0000 2024-04-20T17:00:39.842 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-04-20 16:53:31.560814243 +0000 2024-04-20T17:00:39.842 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-04-20 16:53:31.560814243 +0000 2024-04-20T17:00:39.842 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: - 2024-04-20T17:00:39.842 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-20T17:00:39.909 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:39 smithi106 ceph-mon[44423]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:39.909 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:39 smithi106 ceph-mon[44423]: Deploying daemon mgr.x on smithi106 2024-04-20T17:00:39.909 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:39 smithi106 systemd[1]: Starting Ceph mgr.x for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:00:39.910 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-04-20T17:00:39.911 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-04-20T17:00:39.911 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000227978 s, 2.2 MB/s 2024-04-20T17:00:39.911 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-20T17:00:39.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:39 smithi005 ceph-mon[39562]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:39.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:39 smithi005 ceph-mon[39562]: Deploying daemon mgr.x on smithi106 2024-04-20T17:00:39.970 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_3 2024-04-20T17:00:40.028 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-20T17:00:40.028 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T17:00:40.028 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 6h/6d Inode: 163691 Links: 1 2024-04-20T17:00:40.029 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T17:00:40.029 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T17:00:40.029 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-04-20 17:00:39.774451853 +0000 2024-04-20T17:00:40.029 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-04-20 16:53:31.895806393 +0000 2024-04-20T17:00:40.029 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-04-20 16:53:31.895806393 +0000 2024-04-20T17:00:40.029 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: - 2024-04-20T17:00:40.029 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-20T17:00:40.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:39 smithi171 ceph-mon[44189]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:40.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:39 smithi171 ceph-mon[44189]: Deploying daemon mgr.x on smithi106 2024-04-20T17:00:40.095 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-04-20T17:00:40.095 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-04-20T17:00:40.095 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000176501 s, 2.9 MB/s 2024-04-20T17:00:40.096 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-20T17:00:40.155 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_4 2024-04-20T17:00:40.215 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-20T17:00:40.215 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T17:00:40.215 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 6h/6d Inode: 170094 Links: 1 2024-04-20T17:00:40.215 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T17:00:40.215 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T17:00:40.215 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-04-20 17:00:39.774451853 +0000 2024-04-20T17:00:40.215 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-04-20 16:53:32.239798333 +0000 2024-04-20T17:00:40.215 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-04-20 16:53:32.239798333 +0000 2024-04-20T17:00:40.215 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: - 2024-04-20T17:00:40.216 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-20T17:00:40.280 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-04-20T17:00:40.281 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-04-20T17:00:40.281 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000222108 s, 2.3 MB/s 2024-04-20T17:00:40.282 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-20T17:00:40.340 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-04-20T17:00:40.340 DEBUG:teuthology.orchestra.run.smithi171:> dd if=/scratch_devs of=/dev/stdout 2024-04-20T17:00:40.363 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-20T17:00:40.363 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_1 2024-04-20T17:00:40.424 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-20T17:00:40.424 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T17:00:40.424 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 6h/6d Inode: 172159 Links: 1 2024-04-20T17:00:40.424 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T17:00:40.424 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T17:00:40.424 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-04-20 17:00:14.791226295 +0000 2024-04-20T17:00:40.424 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-04-20 16:53:32.250681226 +0000 2024-04-20T17:00:40.424 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-04-20 16:53:32.250681226 +0000 2024-04-20T17:00:40.425 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: - 2024-04-20T17:00:40.425 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-20T17:00:40.496 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-04-20T17:00:40.496 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-04-20T17:00:40.496 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000297866 s, 1.7 MB/s 2024-04-20T17:00:40.497 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-20T17:00:40.558 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_2 2024-04-20T17:00:40.574 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:40 smithi106 ceph-mgr[46069]: -- 172.21.15.106:0/558599149 <== mon.2 v1:172.21.15.106:6789/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (unknown 3576950894 0 0) 0x562e00e1e5a0 con 0x562e00df2800 2024-04-20T17:00:40.574 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:40 smithi106 bash[45865]: 73dedcd754d937b8c6cdd885cc48acb4337a711435d0ed6fa5630b99e1bb95b2 2024-04-20T17:00:40.575 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:40 smithi106 systemd[1]: Started Ceph mgr.x for 13c6e76c-ff37-11ee-bc93-c7b262605968. 2024-04-20T17:00:40.618 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-20T17:00:40.618 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T17:00:40.618 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 6h/6d Inode: 172242 Links: 1 2024-04-20T17:00:40.618 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T17:00:40.618 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T17:00:40.618 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-04-20 17:00:14.791226295 +0000 2024-04-20T17:00:40.618 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-04-20 16:53:32.590680500 +0000 2024-04-20T17:00:40.618 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-04-20 16:53:32.590680500 +0000 2024-04-20T17:00:40.618 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: - 2024-04-20T17:00:40.619 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-20T17:00:40.686 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-04-20T17:00:40.687 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-04-20T17:00:40.687 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000215758 s, 2.4 MB/s 2024-04-20T17:00:40.688 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-20T17:00:40.748 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_3 2024-04-20T17:00:40.807 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-20T17:00:40.808 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T17:00:40.808 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 6h/6d Inode: 172320 Links: 1 2024-04-20T17:00:40.808 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T17:00:40.808 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T17:00:40.808 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-04-20 17:00:14.791226295 +0000 2024-04-20T17:00:40.808 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-04-20 16:53:32.938679756 +0000 2024-04-20T17:00:40.808 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-04-20 16:53:32.938679756 +0000 2024-04-20T17:00:40.808 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: - 2024-04-20T17:00:40.808 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-20T17:00:40.873 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-04-20T17:00:40.873 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-04-20T17:00:40.873 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000190305 s, 2.7 MB/s 2024-04-20T17:00:40.874 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-20T17:00:40.932 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_4 2024-04-20T17:00:40.989 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-20T17:00:40.989 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T17:00:40.989 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 6h/6d Inode: 170363 Links: 1 2024-04-20T17:00:40.989 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T17:00:40.989 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T17:00:40.990 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-04-20 17:00:14.791226295 +0000 2024-04-20T17:00:40.990 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-04-20 16:53:33.266679057 +0000 2024-04-20T17:00:40.990 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-04-20 16:53:33.266679057 +0000 2024-04-20T17:00:40.990 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: - 2024-04-20T17:00:40.990 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-20T17:00:41.010 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:40 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-x[45975]: 2024-04-20T17:00:40.574+0000 7f3e690ac200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-20T17:00:41.010 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:40 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-x[45975]: 2024-04-20T17:00:40.744+0000 7f3e690ac200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-20T17:00:41.054 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-04-20T17:00:41.054 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-04-20T17:00:41.054 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000225146 s, 2.3 MB/s 2024-04-20T17:00:41.055 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-20T17:00:41.113 INFO:tasks.cephadm:Deploying osd.0 on smithi005 with /dev/vg_nvme/lv_4... 2024-04-20T17:00:41.114 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-20T17:00:41.396 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:00:41.505 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:41 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-x[45975]: 2024-04-20T17:00:41.084+0000 7f3e690ac200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-20T17:00:41.761 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:41 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-x[45975]: 2024-04-20T17:00:41.621+0000 7f3e690ac200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-20T17:00:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:41 smithi106 ceph-mon[44423]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:41 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:41 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:41 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:41 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:41 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:00:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:41 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:41.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:41 smithi005 ceph-mon[39562]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:41.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:41 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:41.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:41 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:41.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:41 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:41.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:41 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:41.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:41 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:00:41.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:41 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:42.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:41 smithi171 ceph-mon[44189]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:42.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:41 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:42.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:41 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:42.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:41 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:42.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:41 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:42.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:41 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:00:42.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:41 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:42.260 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:41 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-x[45975]: 2024-04-20T17:00:41.867+0000 7f3e690ac200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-20T17:00:42.760 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:42 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-x[45975]: 2024-04-20T17:00:42.315+0000 7f3e690ac200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-20T17:00:43.053 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:42 smithi005 ceph-mon[39562]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:43.053 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:43.053 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:43.053 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:43.053 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:00:43.053 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:43.053 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:42 smithi005 ceph-mon[39562]: Reconfiguring mgr.y (unknown last config time)... 2024-04-20T17:00:43.054 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-20T17:00:43.054 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-20T17:00:43.054 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:43.054 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:42 smithi005 ceph-mon[39562]: Reconfiguring daemon mgr.y on smithi005 2024-04-20T17:00:43.260 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:43 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-x[45975]: 2024-04-20T17:00:43.023+0000 7f3e690ac200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-20T17:00:43.261 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:43 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-x[45975]: 2024-04-20T17:00:43.109+0000 7f3e690ac200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-20T17:00:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:42 smithi106 ceph-mon[44423]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:00:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:42 smithi106 ceph-mon[44423]: Reconfiguring mgr.y (unknown last config time)... 2024-04-20T17:00:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-20T17:00:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-20T17:00:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:42 smithi106 ceph-mon[44423]: Reconfiguring daemon mgr.y on smithi005 2024-04-20T17:00:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:42 smithi171 ceph-mon[44189]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:00:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:42 smithi171 ceph-mon[44189]: Reconfiguring mgr.y (unknown last config time)... 2024-04-20T17:00:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-20T17:00:43.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-20T17:00:43.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:43.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:42 smithi171 ceph-mon[44189]: Reconfiguring daemon mgr.y on smithi005 2024-04-20T17:00:44.010 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:43 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-x[45975]: 2024-04-20T17:00:43.694+0000 7f3e690ac200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-20T17:00:44.011 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:43 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-x[45975]: 2024-04-20T17:00:43.778+0000 7f3e690ac200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-20T17:00:44.011 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:43 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-x[45975]: 2024-04-20T17:00:43.865+0000 7f3e690ac200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-20T17:00:44.067 INFO:teuthology.orchestra.run.smithi005.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-20T17:00:44.067 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-20T17:00:44.067 INFO:teuthology.orchestra.run.smithi005.stderr: stderr: 10+0 records in 2024-04-20T17:00:44.067 INFO:teuthology.orchestra.run.smithi005.stderr:10+0 records out 2024-04-20T17:00:44.067 INFO:teuthology.orchestra.run.smithi005.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00831401 s, 1.3 GB/s 2024-04-20T17:00:44.067 INFO:teuthology.orchestra.run.smithi005.stderr:--> Zapping successful for: 2024-04-20T17:00:44.508 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:44 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-x[45975]: 2024-04-20T17:00:44.090+0000 7f3e690ac200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-20T17:00:44.508 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:44 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-x[45975]: 2024-04-20T17:00:44.181+0000 7f3e690ac200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-20T17:00:44.508 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:44 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-x[45975]: 2024-04-20T17:00:44.275+0000 7f3e690ac200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-20T17:00:44.508 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:44 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-x[45975]: 2024-04-20T17:00:44.507+0000 7f3e690ac200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-20T17:00:44.761 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:44 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-x[45975]: 2024-04-20T17:00:44.602+0000 7f3e690ac200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-20T17:00:44.837 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi005:vg_nvme/lv_4 2024-04-20T17:00:45.114 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:00:45.177 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:44 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-x[45975]: 2024-04-20T17:00:44.786+0000 7f3e690ac200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-20T17:00:45.177 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:45 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-x[45975]: 2024-04-20T17:00:45.033+0000 7f3e690ac200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-20T17:00:45.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:45 smithi005 ceph-mon[39562]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:45.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:45.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:45.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:00:45.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:45.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:00:45.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:45 smithi106 ceph-mon[44423]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:00:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:00:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:45.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:45 smithi171 ceph-mon[44189]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:45.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:45.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:45.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:00:45.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:45.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:00:45.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:00:46.010 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:45 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-x[45975]: 2024-04-20T17:00:45.612+0000 7f3e690ac200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-20T17:00:46.011 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:45 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-x[45975]: 2024-04-20T17:00:45.821+0000 7f3e690ac200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-20T17:00:46.510 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:46 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-x[45975]: 2024-04-20T17:00:46.015+0000 7f3e690ac200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-20T17:00:47.010 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:46 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-x[45975]: 2024-04-20T17:00:46.667+0000 7f3e690ac200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-20T17:00:47.011 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:46 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-x[45975]: 2024-04-20T17:00:46.865+0000 7f3e690ac200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-20T17:00:47.510 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:47 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-x[45975]: 2024-04-20T17:00:47.043+0000 7f3e690ac200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-20T17:00:47.511 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:00:47 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-x[45975]: 2024-04-20T17:00:47.157+0000 7f3e690ac200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-20T17:00:47.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:47 smithi005 ceph-mon[39562]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:47.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:47 smithi005 ceph-mon[39562]: Standby manager daemon x started 2024-04-20T17:00:47.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:47 smithi005 ceph-mon[39562]: from='mgr.? v1:172.21.15.106:0/815059948' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-20T17:00:47.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:47 smithi005 ceph-mon[39562]: from='mgr.? v1:172.21.15.106:0/815059948' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-20T17:00:47.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:47 smithi005 ceph-mon[39562]: from='mgr.? v1:172.21.15.106:0/815059948' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-20T17:00:47.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:47 smithi005 ceph-mon[39562]: from='mgr.? v1:172.21.15.106:0/815059948' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-20T17:00:47.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:47 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:00:47.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:47 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:00:47.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:47 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:47 smithi106 ceph-mon[44423]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:47 smithi106 ceph-mon[44423]: Standby manager daemon x started 2024-04-20T17:00:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:47 smithi106 ceph-mon[44423]: from='mgr.? v1:172.21.15.106:0/815059948' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-20T17:00:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:47 smithi106 ceph-mon[44423]: from='mgr.? v1:172.21.15.106:0/815059948' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-20T17:00:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:47 smithi106 ceph-mon[44423]: from='mgr.? v1:172.21.15.106:0/815059948' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-20T17:00:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:47 smithi106 ceph-mon[44423]: from='mgr.? v1:172.21.15.106:0/815059948' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-20T17:00:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:47 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:00:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:47 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:00:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:47 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:48.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:47 smithi171 ceph-mon[44189]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:48.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:47 smithi171 ceph-mon[44189]: Standby manager daemon x started 2024-04-20T17:00:48.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:47 smithi171 ceph-mon[44189]: from='mgr.? v1:172.21.15.106:0/815059948' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-20T17:00:48.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:47 smithi171 ceph-mon[44189]: from='mgr.? v1:172.21.15.106:0/815059948' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-20T17:00:48.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:47 smithi171 ceph-mon[44189]: from='mgr.? v1:172.21.15.106:0/815059948' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-20T17:00:48.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:47 smithi171 ceph-mon[44189]: from='mgr.? v1:172.21.15.106:0/815059948' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-20T17:00:48.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:47 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:00:48.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:47 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:00:48.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:47 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:48.857 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:48 smithi005 ceph-mon[39562]: from='client.24116 v1:172.21.15.5:0/1959942716' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:00:48.857 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:48 smithi005 ceph-mon[39562]: mgrmap e14: y(active, since 81s), standbys: x 2024-04-20T17:00:48.857 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:48 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-20T17:00:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:48 smithi106 ceph-mon[44423]: from='client.24116 v1:172.21.15.5:0/1959942716' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:00:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:48 smithi106 ceph-mon[44423]: mgrmap e14: y(active, since 81s), standbys: x 2024-04-20T17:00:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:48 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-20T17:00:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:48 smithi171 ceph-mon[44189]: from='client.24116 v1:172.21.15.5:0/1959942716' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:00:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:48 smithi171 ceph-mon[44189]: mgrmap e14: y(active, since 81s), standbys: x 2024-04-20T17:00:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:48 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-20T17:00:49.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:49 smithi005 ceph-mon[39562]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:50.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:49 smithi106 ceph-mon[44423]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:49 smithi171 ceph-mon[44189]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:50.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:50 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1497128924' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b"}]: dispatch 2024-04-20T17:00:50.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:50 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1497128924' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b"}]': finished 2024-04-20T17:00:50.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:50 smithi005 ceph-mon[39562]: osdmap e5: 1 total, 0 up, 1 in 2024-04-20T17:00:50.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:50 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T17:00:51.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:50 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1497128924' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b"}]: dispatch 2024-04-20T17:00:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:50 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1497128924' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b"}]': finished 2024-04-20T17:00:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:50 smithi106 ceph-mon[44423]: osdmap e5: 1 total, 0 up, 1 in 2024-04-20T17:00:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:50 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T17:00:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:50 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1497128924' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b"}]: dispatch 2024-04-20T17:00:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:50 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1497128924' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b"}]': finished 2024-04-20T17:00:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:50 smithi171 ceph-mon[44189]: osdmap e5: 1 total, 0 up, 1 in 2024-04-20T17:00:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:50 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T17:00:51.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:51 smithi005 ceph-mon[39562]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:51.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:51 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2577918477' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:00:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:51 smithi106 ceph-mon[44423]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:51 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2577918477' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:00:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:51 smithi171 ceph-mon[44189]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:51 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2577918477' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:00:53.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:53 smithi005 ceph-mon[39562]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:54.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:53 smithi106 ceph-mon[44423]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:54.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:53 smithi171 ceph-mon[44189]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:55.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:55 smithi005 ceph-mon[39562]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:56.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:55 smithi106 ceph-mon[44423]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:56.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:55 smithi171 ceph-mon[44189]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:56.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:56 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-20T17:00:56.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:56 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:57.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:56 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-20T17:00:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:56 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:57.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:56 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-20T17:00:57.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:56 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:00:58.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:57 smithi106 ceph-mon[44423]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:58.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:57 smithi106 ceph-mon[44423]: Deploying daemon osd.0 on smithi005 2024-04-20T17:00:58.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:57 smithi171 ceph-mon[44189]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:58.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:57 smithi171 ceph-mon[44189]: Deploying daemon osd.0 on smithi005 2024-04-20T17:00:58.177 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:57 smithi005 ceph-mon[39562]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:58.177 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:57 smithi005 ceph-mon[39562]: Deploying daemon osd.0 on smithi005 2024-04-20T17:00:58.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:00:58 smithi005 ceph-mon[39562]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:59.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:00:58 smithi106 ceph-mon[44423]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:00:59.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:00:58 smithi171 ceph-mon[44189]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:01:01.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:01 smithi005 ceph-mon[39562]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:01:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:01 smithi106 ceph-mon[44423]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:01:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:01 smithi171 ceph-mon[44189]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:01:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:03 smithi171 ceph-mon[44189]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:01:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:03 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:03 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:01:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:03 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:03.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:03 smithi005 ceph-mon[39562]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:01:03.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:03 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:03.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:03 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:01:03.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:03 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:03 smithi106 ceph-mon[44423]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:01:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:03 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:03 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:01:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:03 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:05 smithi171 ceph-mon[44189]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:01:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:05 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:05 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:05 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:05 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:01:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:05 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:05.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:05 smithi005 ceph-mon[39562]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:01:05.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:05 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:05.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:05 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:05 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:05 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:01:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:05 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:05 smithi106 ceph-mon[44423]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:01:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:05 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:05 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:05 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:05 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:01:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:05 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:06.480 INFO:teuthology.orchestra.run.smithi005.stdout:Created osd(s) 0 on host 'smithi005' 2024-04-20T17:01:06.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:06 smithi005 ceph-mon[39562]: from='osd.0 v1:172.21.15.5:6801/3901484705' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-20T17:01:06.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:06 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:06.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:06 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:06.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:06 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:01:06.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:06 smithi171 ceph-mon[44189]: from='osd.0 v1:172.21.15.5:6801/3901484705' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-20T17:01:06.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:06 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:06.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:06 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:06.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:06 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:01:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:06 smithi106 ceph-mon[44423]: from='osd.0 v1:172.21.15.5:6801/3901484705' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-20T17:01:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:06 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:06 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:06 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:01:07.163 DEBUG:teuthology.orchestra.run.smithi005:osd.0> sudo journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.0.service 2024-04-20T17:01:07.164 INFO:tasks.cephadm:Deploying osd.1 on smithi005 with /dev/vg_nvme/lv_3... 2024-04-20T17:01:07.165 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-20T17:01:07.198 INFO:journalctl@ceph.osd.0.smithi005.stdout:-- Logs begin at Sat 2024-04-20 16:47:38 UTC. -- 2024-04-20T17:01:07.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:07 smithi005 ceph-mon[39562]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:01:07.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:07 smithi005 ceph-mon[39562]: from='osd.0 v1:172.21.15.5:6801/3901484705' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-20T17:01:07.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:07 smithi005 ceph-mon[39562]: osdmap e6: 1 total, 0 up, 1 in 2024-04-20T17:01:07.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:07 smithi005 ceph-mon[39562]: from='osd.0 v1:172.21.15.5:6801/3901484705' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:01:07.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:07 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T17:01:07.961 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:01:07 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-0[57624]: 2024-04-20T17:01:07.861+0000 7fa616531700 -1 osd.0 0 waiting for initial osdmap 2024-04-20T17:01:07.961 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:01:07 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-0[57624]: 2024-04-20T17:01:07.863+0000 7fa60e31c700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T17:01:08.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:07 smithi106 ceph-mon[44423]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:01:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:07 smithi106 ceph-mon[44423]: from='osd.0 v1:172.21.15.5:6801/3901484705' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-20T17:01:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:07 smithi106 ceph-mon[44423]: osdmap e6: 1 total, 0 up, 1 in 2024-04-20T17:01:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:07 smithi106 ceph-mon[44423]: from='osd.0 v1:172.21.15.5:6801/3901484705' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:01:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:07 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T17:01:08.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:07 smithi171 ceph-mon[44189]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:01:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:07 smithi171 ceph-mon[44189]: from='osd.0 v1:172.21.15.5:6801/3901484705' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-20T17:01:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:07 smithi171 ceph-mon[44189]: osdmap e6: 1 total, 0 up, 1 in 2024-04-20T17:01:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:07 smithi171 ceph-mon[44189]: from='osd.0 v1:172.21.15.5:6801/3901484705' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:01:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:07 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T17:01:08.556 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:01:08.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:08 smithi005 ceph-mon[39562]: from='osd.0 v1:172.21.15.5:6801/3901484705' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2024-04-20T17:01:08.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:08 smithi005 ceph-mon[39562]: osdmap e7: 1 total, 0 up, 1 in 2024-04-20T17:01:08.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:08 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T17:01:08.837 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:08 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T17:01:08.837 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:08 smithi005 ceph-mon[39562]: from='osd.0 v1:172.21.15.5:6801/3901484705' entity='osd.0' 2024-04-20T17:01:08.837 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:08 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T17:01:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:08 smithi106 ceph-mon[44423]: from='osd.0 v1:172.21.15.5:6801/3901484705' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2024-04-20T17:01:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:08 smithi106 ceph-mon[44423]: osdmap e7: 1 total, 0 up, 1 in 2024-04-20T17:01:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:08 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T17:01:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:08 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T17:01:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:08 smithi106 ceph-mon[44423]: from='osd.0 v1:172.21.15.5:6801/3901484705' entity='osd.0' 2024-04-20T17:01:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:08 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T17:01:09.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:08 smithi171 ceph-mon[44189]: from='osd.0 v1:172.21.15.5:6801/3901484705' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2024-04-20T17:01:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:08 smithi171 ceph-mon[44189]: osdmap e7: 1 total, 0 up, 1 in 2024-04-20T17:01:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:08 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T17:01:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:08 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T17:01:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:08 smithi171 ceph-mon[44189]: from='osd.0 v1:172.21.15.5:6801/3901484705' entity='osd.0' 2024-04-20T17:01:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:08 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T17:01:09.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:09 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:01:09.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:09 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:01:09.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:01:09.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:09 smithi005 ceph-mon[39562]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:01:09.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:01:09.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:09 smithi005 ceph-mon[39562]: osd.0 v1:172.21.15.5:6801/3901484705 boot 2024-04-20T17:01:09.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:09 smithi005 ceph-mon[39562]: osdmap e8: 1 total, 1 up, 1 in 2024-04-20T17:01:09.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:09 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T17:01:10.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:09 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:01:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:09 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:01:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:01:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:09 smithi106 ceph-mon[44423]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:01:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:01:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:09 smithi106 ceph-mon[44423]: osd.0 v1:172.21.15.5:6801/3901484705 boot 2024-04-20T17:01:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:09 smithi106 ceph-mon[44423]: osdmap e8: 1 total, 1 up, 1 in 2024-04-20T17:01:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:09 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T17:01:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:09 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:01:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:09 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:01:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:01:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:09 smithi171 ceph-mon[44189]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-20T17:01:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:01:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:09 smithi171 ceph-mon[44189]: osd.0 v1:172.21.15.5:6801/3901484705 boot 2024-04-20T17:01:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:09 smithi171 ceph-mon[44189]: osdmap e8: 1 total, 1 up, 1 in 2024-04-20T17:01:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:09 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T17:01:11.226 INFO:teuthology.orchestra.run.smithi005.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-20T17:01:11.227 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-20T17:01:11.227 INFO:teuthology.orchestra.run.smithi005.stderr: stderr: 10+0 records in 2024-04-20T17:01:11.227 INFO:teuthology.orchestra.run.smithi005.stderr:10+0 records out 2024-04-20T17:01:11.227 INFO:teuthology.orchestra.run.smithi005.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00824124 s, 1.3 GB/s 2024-04-20T17:01:11.227 INFO:teuthology.orchestra.run.smithi005.stderr:--> Zapping successful for: 2024-04-20T17:01:11.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:11 smithi005 ceph-mon[39562]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:11.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:11 smithi005 ceph-mon[39562]: osdmap e9: 1 total, 1 up, 1 in 2024-04-20T17:01:12.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:11 smithi106 ceph-mon[44423]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:11 smithi106 ceph-mon[44423]: osdmap e9: 1 total, 1 up, 1 in 2024-04-20T17:01:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:11 smithi171 ceph-mon[44189]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:11 smithi171 ceph-mon[44189]: osdmap e9: 1 total, 1 up, 1 in 2024-04-20T17:01:12.287 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi005:vg_nvme/lv_3 2024-04-20T17:01:12.560 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:01:12.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:12.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:12.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:01:12.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:12 smithi005 ceph-mon[39562]: Adjusting osd_memory_target on smithi005 to 17126M 2024-04-20T17:01:12.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:12.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:12.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:01:12.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:01:12.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:01:12.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:01:12.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:01:12.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:12.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:01:12.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:12.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:01:12.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:01:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:12 smithi106 ceph-mon[44423]: Adjusting osd_memory_target on smithi005 to 17126M 2024-04-20T17:01:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:01:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:01:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:01:13.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:01:13.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:01:13.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:13.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:01:13.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:13.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:01:13.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:01:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:12 smithi171 ceph-mon[44189]: Adjusting osd_memory_target on smithi005 to 17126M 2024-04-20T17:01:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:01:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:01:13.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:01:13.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:01:13.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:01:13.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:13.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:01:13.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:13.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:01:13.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:13.883 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:13 smithi005 ceph-mon[39562]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:14.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:13 smithi106 ceph-mon[44423]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:14.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:13 smithi171 ceph-mon[44189]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:14 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:01:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:14 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:01:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:14 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:15.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:14 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:01:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:14 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:01:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:14 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:15.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:14 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:01:15.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:14 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:01:15.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:14 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:15 smithi106 ceph-mon[44423]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:15 smithi106 ceph-mon[44423]: from='client.14241 v1:172.21.15.5:0/2576983158' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:01:16.074 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:15 smithi005 ceph-mon[39562]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:16.074 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:15 smithi005 ceph-mon[39562]: from='client.14241 v1:172.21.15.5:0/2576983158' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:01:16.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:15 smithi171 ceph-mon[44189]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:16.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:15 smithi171 ceph-mon[44189]: from='client.14241 v1:172.21.15.5:0/2576983158' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:01:17.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:16 smithi106 ceph-mon[44423]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:17.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:16 smithi171 ceph-mon[44189]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:17.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:16 smithi005 ceph-mon[39562]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:18.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:17 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3451031234' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aed0c8eb-0e81-4e85-ab8f-a60420060656"}]: dispatch 2024-04-20T17:01:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:17 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3451031234' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aed0c8eb-0e81-4e85-ab8f-a60420060656"}]': finished 2024-04-20T17:01:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:17 smithi106 ceph-mon[44423]: osdmap e10: 2 total, 1 up, 2 in 2024-04-20T17:01:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:17 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T17:01:18.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:17 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3451031234' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aed0c8eb-0e81-4e85-ab8f-a60420060656"}]: dispatch 2024-04-20T17:01:18.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:17 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3451031234' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aed0c8eb-0e81-4e85-ab8f-a60420060656"}]': finished 2024-04-20T17:01:18.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:17 smithi171 ceph-mon[44189]: osdmap e10: 2 total, 1 up, 2 in 2024-04-20T17:01:18.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:17 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T17:01:18.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:17 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3451031234' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aed0c8eb-0e81-4e85-ab8f-a60420060656"}]: dispatch 2024-04-20T17:01:18.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:17 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3451031234' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aed0c8eb-0e81-4e85-ab8f-a60420060656"}]': finished 2024-04-20T17:01:18.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:17 smithi005 ceph-mon[39562]: osdmap e10: 2 total, 1 up, 2 in 2024-04-20T17:01:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:17 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T17:01:19.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:18 smithi106 ceph-mon[44423]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:19.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:18 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3439745962' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:01:19.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:18 smithi171 ceph-mon[44189]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:18 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3439745962' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:01:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:18 smithi005 ceph-mon[39562]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:18 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3439745962' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:01:21.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:21 smithi005 ceph-mon[39562]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:22.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:21 smithi106 ceph-mon[44423]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:22.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:21 smithi171 ceph-mon[44189]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:23.903 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:23 smithi005 ceph-mon[39562]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:24.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:23 smithi106 ceph-mon[44423]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:24.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:23 smithi171 ceph-mon[44189]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:24.784 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:24 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-20T17:01:24.784 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:24 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:25.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-20T17:01:25.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:24 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-20T17:01:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:24 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:25.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:25 smithi005 ceph-mon[39562]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:25.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:25 smithi005 ceph-mon[39562]: Deploying daemon osd.1 on smithi005 2024-04-20T17:01:26.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:25 smithi106 ceph-mon[44423]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:26.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:25 smithi106 ceph-mon[44423]: Deploying daemon osd.1 on smithi005 2024-04-20T17:01:26.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:25 smithi171 ceph-mon[44189]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:26.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:25 smithi171 ceph-mon[44189]: Deploying daemon osd.1 on smithi005 2024-04-20T17:01:27.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:27 smithi005 ceph-mon[39562]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:28.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:27 smithi106 ceph-mon[44423]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:27 smithi171 ceph-mon[44189]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:29.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:29 smithi005 ceph-mon[39562]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:30.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:29 smithi106 ceph-mon[44423]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:30.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:29 smithi171 ceph-mon[44189]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:31.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:30 smithi005 ceph-mon[39562]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:31.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:30 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:31.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:30 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:30 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:01:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:30 smithi106 ceph-mon[44423]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:30 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:30 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:31.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:30 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:01:31.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:30 smithi171 ceph-mon[44189]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:30 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:30 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:30 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:01:33.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:33 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:33.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:33 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:33.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:33 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:01:33.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:33 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:01:33.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:33 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:01:33.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:33 smithi005 ceph-mon[39562]: from='osd.1 v1:172.21.15.5:6805/3391893914' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-20T17:01:33.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:33 smithi005 ceph-mon[39562]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-20T17:01:33.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:33 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:33.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:33 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:01:33.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:33 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:33.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:33 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:01:33.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:33 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:33.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:33 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:01:33.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:33 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:33 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:33 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:33 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:01:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:33 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:01:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:33 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:01:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:33 smithi106 ceph-mon[44423]: from='osd.1 v1:172.21.15.5:6805/3391893914' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-20T17:01:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:33 smithi106 ceph-mon[44423]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-20T17:01:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:33 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:33 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:01:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:33 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:33 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:01:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:33 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:34.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:33 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:01:34.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:33 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:34.060 INFO:teuthology.orchestra.run.smithi005.stdout:Created osd(s) 1 on host 'smithi005' 2024-04-20T17:01:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:33 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:33 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:33 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:01:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:33 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:01:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:33 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:01:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:33 smithi171 ceph-mon[44189]: from='osd.1 v1:172.21.15.5:6805/3391893914' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-20T17:01:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:33 smithi171 ceph-mon[44189]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-20T17:01:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:33 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:33 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:01:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:33 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:33 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:01:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:33 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:33 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:01:34.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:33 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:34.692 DEBUG:teuthology.orchestra.run.smithi005:osd.1> sudo journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.1.service 2024-04-20T17:01:34.694 INFO:tasks.cephadm:Deploying osd.2 on smithi005 with /dev/vg_nvme/lv_2... 2024-04-20T17:01:34.695 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-20T17:01:34.728 INFO:journalctl@ceph.osd.1.smithi005.stdout:-- Logs begin at Sat 2024-04-20 16:47:38 UTC. -- 2024-04-20T17:01:34.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:34 smithi005 ceph-mon[39562]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-20T17:01:34.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:34 smithi005 ceph-mon[39562]: osdmap e11: 2 total, 1 up, 2 in 2024-04-20T17:01:34.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:34 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T17:01:34.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:34 smithi005 ceph-mon[39562]: from='osd.1 v1:172.21.15.5:6805/3391893914' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:01:34.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:34 smithi005 ceph-mon[39562]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:01:34.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:34 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:34.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:34 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:34.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:34 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:01:35.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:34 smithi106 ceph-mon[44423]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-20T17:01:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:34 smithi106 ceph-mon[44423]: osdmap e11: 2 total, 1 up, 2 in 2024-04-20T17:01:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:34 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T17:01:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:34 smithi106 ceph-mon[44423]: from='osd.1 v1:172.21.15.5:6805/3391893914' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:01:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:34 smithi106 ceph-mon[44423]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:01:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:34 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:34 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:34 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:01:35.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:34 smithi171 ceph-mon[44189]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-20T17:01:35.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:34 smithi171 ceph-mon[44189]: osdmap e11: 2 total, 1 up, 2 in 2024-04-20T17:01:35.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:34 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T17:01:35.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:34 smithi171 ceph-mon[44189]: from='osd.1 v1:172.21.15.5:6805/3391893914' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:01:35.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:34 smithi171 ceph-mon[44189]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:01:35.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:34 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:35.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:34 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:35.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:34 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:01:35.458 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:01:35.907 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:35 smithi005 ceph-mon[39562]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:35.907 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:35 smithi005 ceph-mon[39562]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2024-04-20T17:01:35.907 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:35 smithi005 ceph-mon[39562]: osdmap e12: 2 total, 1 up, 2 in 2024-04-20T17:01:35.907 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:35 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T17:01:35.907 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:35 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T17:01:36.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:35 smithi106 ceph-mon[44423]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:35 smithi106 ceph-mon[44423]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2024-04-20T17:01:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:35 smithi106 ceph-mon[44423]: osdmap e12: 2 total, 1 up, 2 in 2024-04-20T17:01:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:35 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T17:01:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:35 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T17:01:36.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:35 smithi171 ceph-mon[44189]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:36.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:35 smithi171 ceph-mon[44189]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2024-04-20T17:01:36.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:35 smithi171 ceph-mon[44189]: osdmap e12: 2 total, 1 up, 2 in 2024-04-20T17:01:36.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:35 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T17:01:36.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:35 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T17:01:36.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:36 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:01:36.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:36 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:01:36.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:01:36.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:36 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T17:01:36.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:36 smithi005 ceph-mon[39562]: osd.1 v1:172.21.15.5:6805/3391893914 boot 2024-04-20T17:01:36.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:36 smithi005 ceph-mon[39562]: osdmap e13: 2 total, 2 up, 2 in 2024-04-20T17:01:36.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:36 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T17:01:36.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:36 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:01:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:36 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:01:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:36 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:01:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:01:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:36 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T17:01:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:36 smithi106 ceph-mon[44423]: osd.1 v1:172.21.15.5:6805/3391893914 boot 2024-04-20T17:01:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:36 smithi106 ceph-mon[44423]: osdmap e13: 2 total, 2 up, 2 in 2024-04-20T17:01:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:36 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T17:01:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:36 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:01:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:36 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:01:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:36 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:01:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:01:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:36 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T17:01:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:36 smithi171 ceph-mon[44189]: osd.1 v1:172.21.15.5:6805/3391893914 boot 2024-04-20T17:01:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:36 smithi171 ceph-mon[44189]: osdmap e13: 2 total, 2 up, 2 in 2024-04-20T17:01:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:36 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T17:01:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:36 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:01:37.915 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:37 smithi171 ceph-mon[44189]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:37.915 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:37 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-20T17:01:37.916 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:37 smithi171 ceph-mon[44189]: osdmap e14: 2 total, 2 up, 2 in 2024-04-20T17:01:37.916 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:37 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:01:37.916 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:37 smithi171 sudo[46364]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-20T17:01:37.916 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:37 smithi171 sudo[46364]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-20T17:01:37.916 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:37 smithi171 sudo[46364]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-20T17:01:37.916 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:37 smithi171 sudo[46364]: pam_unix(sudo:session): session closed for user root 2024-04-20T17:01:37.961 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:01:37 smithi005 sudo[66666]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-20T17:01:37.962 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:01:37 smithi005 sudo[66666]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-20T17:01:37.962 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:01:37 smithi005 sudo[66666]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-20T17:01:37.962 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:01:37 smithi005 sudo[66666]: pam_unix(sudo:session): session closed for user root 2024-04-20T17:01:37.962 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:01:37 smithi005 sudo[66842]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-20T17:01:37.962 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:01:37 smithi005 sudo[66842]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-20T17:01:37.962 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:01:37 smithi005 sudo[66842]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-20T17:01:37.962 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:01:37 smithi005 sudo[66842]: pam_unix(sudo:session): session closed for user root 2024-04-20T17:01:37.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:37 smithi005 ceph-mon[39562]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:37.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:37 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-20T17:01:37.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:37 smithi005 ceph-mon[39562]: osdmap e14: 2 total, 2 up, 2 in 2024-04-20T17:01:37.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:37 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:01:37.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:37 smithi005 sudo[67361]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-20T17:01:37.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:37 smithi005 sudo[67361]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-20T17:01:37.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:37 smithi005 sudo[67361]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-20T17:01:37.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:37 smithi005 sudo[67361]: pam_unix(sudo:session): session closed for user root 2024-04-20T17:01:37.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:37 smithi005 sudo[67564]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-20T17:01:37.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:37 smithi005 sudo[67564]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-20T17:01:37.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:37 smithi005 sudo[67564]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-20T17:01:37.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:37 smithi005 sudo[67564]: pam_unix(sudo:session): session closed for user root 2024-04-20T17:01:37.963 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:01:37 smithi005 sudo[67054]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-20T17:01:37.963 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:01:37 smithi005 sudo[67054]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-20T17:01:37.963 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:01:37 smithi005 sudo[67054]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-20T17:01:37.964 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:01:37 smithi005 sudo[67054]: pam_unix(sudo:session): session closed for user root 2024-04-20T17:01:37.964 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:01:37 smithi005 sudo[67247]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-20T17:01:37.964 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:01:37 smithi005 sudo[67247]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-20T17:01:37.964 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:01:37 smithi005 sudo[67247]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-20T17:01:37.964 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:01:37 smithi005 sudo[67247]: pam_unix(sudo:session): session closed for user root 2024-04-20T17:01:37.964 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:37 smithi106 ceph-mon[44423]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-20T17:01:37.964 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:37 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-20T17:01:37.964 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:37 smithi106 ceph-mon[44423]: osdmap e14: 2 total, 2 up, 2 in 2024-04-20T17:01:37.964 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:37 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:01:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:37 smithi106 sudo[46805]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-20T17:01:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:37 smithi106 sudo[46805]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-20T17:01:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:37 smithi106 sudo[46805]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-20T17:01:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:37 smithi106 sudo[46805]: pam_unix(sudo:session): session closed for user root 2024-04-20T17:01:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:37 smithi106 sudo[46808]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-20T17:01:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:37 smithi106 sudo[46808]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-20T17:01:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:37 smithi106 sudo[46808]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-20T17:01:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:37 smithi106 sudo[46808]: pam_unix(sudo:session): session closed for user root 2024-04-20T17:01:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:37 smithi171 sudo[46367]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-20T17:01:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:37 smithi171 sudo[46367]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-20T17:01:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:37 smithi171 sudo[46367]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-20T17:01:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:37 smithi171 sudo[46367]: pam_unix(sudo:session): session closed for user root 2024-04-20T17:01:38.767 INFO:teuthology.orchestra.run.smithi005.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-20T17:01:38.767 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-20T17:01:38.767 INFO:teuthology.orchestra.run.smithi005.stderr: stderr: 10+0 records in 2024-04-20T17:01:38.767 INFO:teuthology.orchestra.run.smithi005.stderr:10+0 records out 2024-04-20T17:01:38.767 INFO:teuthology.orchestra.run.smithi005.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00827362 s, 1.3 GB/s 2024-04-20T17:01:38.767 INFO:teuthology.orchestra.run.smithi005.stderr:--> Zapping successful for: 2024-04-20T17:01:38.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-20T17:01:38.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:38 smithi005 ceph-mon[39562]: osdmap e15: 2 total, 2 up, 2 in 2024-04-20T17:01:38.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:38.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:01:38.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:01:38.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:01:38.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:38 smithi005 ceph-mon[39562]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-20T17:01:38.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:38 smithi005 ceph-mon[39562]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-20T17:01:38.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:01:38.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:01:38.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:38 smithi005 ceph-mon[39562]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-20T17:01:38.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:01:38.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:38 smithi005 ceph-mon[39562]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-20T17:01:38.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:38.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:01:38.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:38 smithi005 ceph-mon[39562]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-20T17:01:38.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:01:38.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:01:38.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:38 smithi005 ceph-mon[39562]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-20T17:01:38.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-20T17:01:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:38 smithi106 ceph-mon[44423]: osdmap e15: 2 total, 2 up, 2 in 2024-04-20T17:01:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:01:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:01:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:01:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:38 smithi106 ceph-mon[44423]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-20T17:01:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:38 smithi106 ceph-mon[44423]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-20T17:01:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:01:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:01:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:38 smithi106 ceph-mon[44423]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-20T17:01:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:01:39.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:38 smithi106 ceph-mon[44423]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-20T17:01:39.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:39.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:01:39.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:38 smithi106 ceph-mon[44423]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-20T17:01:39.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:01:39.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:01:39.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:38 smithi106 ceph-mon[44423]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-20T17:01:39.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-20T17:01:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:38 smithi171 ceph-mon[44189]: osdmap e15: 2 total, 2 up, 2 in 2024-04-20T17:01:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:01:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:01:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:01:39.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:38 smithi171 ceph-mon[44189]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-20T17:01:39.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:38 smithi171 ceph-mon[44189]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-20T17:01:39.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:01:39.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:01:39.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:38 smithi171 ceph-mon[44189]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-20T17:01:39.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:01:39.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:38 smithi171 ceph-mon[44189]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-20T17:01:39.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:39.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:01:39.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:38 smithi171 ceph-mon[44189]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-20T17:01:39.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:01:39.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:01:39.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:38 smithi171 ceph-mon[44189]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-20T17:01:39.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:39.721 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi005:vg_nvme/lv_2 2024-04-20T17:01:39.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:39 smithi005 ceph-mon[39562]: pgmap v71: 1 pgs: 1 unknown; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:39.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:39 smithi005 ceph-mon[39562]: osdmap e16: 2 total, 2 up, 2 in 2024-04-20T17:01:39.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:39 smithi005 ceph-mon[39562]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-20T17:01:39.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:39 smithi005 ceph-mon[39562]: Detected new or changed devices on smithi005 2024-04-20T17:01:39.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:39 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:39.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:39 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:39.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:39 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:01:39.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:39 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:01:39.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:39 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:01:39.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:39 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:01:39.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:39 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:01:39.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:39 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:01:39.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:39 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:39.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:39 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:39.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:39 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:01:39.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:39 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:39.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:39 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:01:39.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:39 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:39.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:39 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:01:39.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:39 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:39.996 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:01:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:39 smithi106 ceph-mon[44423]: pgmap v71: 1 pgs: 1 unknown; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:39 smithi106 ceph-mon[44423]: osdmap e16: 2 total, 2 up, 2 in 2024-04-20T17:01:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:39 smithi106 ceph-mon[44423]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-20T17:01:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:39 smithi106 ceph-mon[44423]: Detected new or changed devices on smithi005 2024-04-20T17:01:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:39 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:39 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:39 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:01:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:39 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:01:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:39 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:01:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:39 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:01:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:39 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:01:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:39 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:01:40.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:39 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:40.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:39 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:40.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:39 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:01:40.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:39 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:40.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:39 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:01:40.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:39 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:40.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:39 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:01:40.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:39 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:40.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:39 smithi171 ceph-mon[44189]: pgmap v71: 1 pgs: 1 unknown; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:40.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:39 smithi171 ceph-mon[44189]: osdmap e16: 2 total, 2 up, 2 in 2024-04-20T17:01:40.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:39 smithi171 ceph-mon[44189]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-20T17:01:40.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:39 smithi171 ceph-mon[44189]: Detected new or changed devices on smithi005 2024-04-20T17:01:40.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:39 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:40.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:39 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:40.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:39 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:01:40.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:39 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:01:40.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:39 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:01:40.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:39 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:01:40.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:39 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:01:40.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:39 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:01:40.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:39 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:40.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:39 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:40.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:39 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:01:40.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:39 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:40.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:39 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:01:40.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:39 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:40.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:39 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:01:40.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:39 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:40.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:40 smithi005 ceph-mon[39562]: Adjusting osd_memory_target on smithi005 to 8563M 2024-04-20T17:01:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:40 smithi106 ceph-mon[44423]: Adjusting osd_memory_target on smithi005 to 8563M 2024-04-20T17:01:41.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:40 smithi171 ceph-mon[44189]: Adjusting osd_memory_target on smithi005 to 8563M 2024-04-20T17:01:41.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:41 smithi005 ceph-mon[39562]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:42.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:41 smithi106 ceph-mon[44423]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:42.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:41 smithi171 ceph-mon[44189]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:42.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:01:42.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:01:42.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:43.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:01:43.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:01:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:43.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:01:43.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:01:43.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:43.868 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:43 smithi005 ceph-mon[39562]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:43.868 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:43 smithi005 ceph-mon[39562]: from='client.14277 v1:172.21.15.5:0/602571472' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:01:44.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:43 smithi106 ceph-mon[44423]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:44.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:43 smithi106 ceph-mon[44423]: from='client.14277 v1:172.21.15.5:0/602571472' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:01:44.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:43 smithi171 ceph-mon[44189]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:44.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:43 smithi171 ceph-mon[44189]: from='client.14277 v1:172.21.15.5:0/602571472' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:01:46.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:45 smithi106 ceph-mon[44423]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:45 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/827654043' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f733a6a2-19c2-4c7a-af9c-04e282062bb7"}]: dispatch 2024-04-20T17:01:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:45 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/827654043' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f733a6a2-19c2-4c7a-af9c-04e282062bb7"}]': finished 2024-04-20T17:01:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:45 smithi106 ceph-mon[44423]: osdmap e17: 3 total, 2 up, 3 in 2024-04-20T17:01:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T17:01:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:45 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1922634575' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:01:46.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:45 smithi171 ceph-mon[44189]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:46.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:45 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/827654043' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f733a6a2-19c2-4c7a-af9c-04e282062bb7"}]: dispatch 2024-04-20T17:01:46.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:45 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/827654043' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f733a6a2-19c2-4c7a-af9c-04e282062bb7"}]': finished 2024-04-20T17:01:46.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:45 smithi171 ceph-mon[44189]: osdmap e17: 3 total, 2 up, 3 in 2024-04-20T17:01:46.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T17:01:46.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:45 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1922634575' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:01:46.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:45 smithi005 ceph-mon[39562]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:46.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:45 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/827654043' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f733a6a2-19c2-4c7a-af9c-04e282062bb7"}]: dispatch 2024-04-20T17:01:46.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:45 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/827654043' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f733a6a2-19c2-4c7a-af9c-04e282062bb7"}]': finished 2024-04-20T17:01:46.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:45 smithi005 ceph-mon[39562]: osdmap e17: 3 total, 2 up, 3 in 2024-04-20T17:01:46.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T17:01:46.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:45 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1922634575' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:01:48.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:47 smithi106 ceph-mon[44423]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:48.030 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:47 smithi005 ceph-mon[39562]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:48.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:47 smithi171 ceph-mon[44189]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:50.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:49 smithi106 ceph-mon[44423]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:50.066 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:49 smithi005 ceph-mon[39562]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:50.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:49 smithi171 ceph-mon[44189]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:52.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:51 smithi106 ceph-mon[44423]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:51 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-20T17:01:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:51 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:51 smithi171 ceph-mon[44189]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:51 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-20T17:01:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:51 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:52.089 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:51 smithi005 ceph-mon[39562]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:52.089 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:51 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-20T17:01:52.089 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:51 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:01:53.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:52 smithi106 ceph-mon[44423]: Deploying daemon osd.2 on smithi005 2024-04-20T17:01:53.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:52 smithi171 ceph-mon[44189]: Deploying daemon osd.2 on smithi005 2024-04-20T17:01:53.096 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:52 smithi005 ceph-mon[39562]: Deploying daemon osd.2 on smithi005 2024-04-20T17:01:53.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:53 smithi005 ceph-mon[39562]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:54.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:53 smithi106 ceph-mon[44423]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:54.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:53 smithi171 ceph-mon[44189]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:54.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:54 smithi005 ceph-mon[39562]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:55.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:54 smithi106 ceph-mon[44423]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:55.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:54 smithi171 ceph-mon[44189]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:57.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:56 smithi005 ceph-mon[39562]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:57.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:56 smithi106 ceph-mon[44423]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:57.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:56 smithi171 ceph-mon[44189]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:58.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:57 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:58.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:57 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:58.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:57 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:01:58.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:57 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:58.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:57 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:58.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:57 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:01:58.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:57 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:58.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:57 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:01:58.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:57 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:01:59.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:01:58 smithi005 ceph-mon[39562]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:59.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:01:58 smithi106 ceph-mon[44423]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:01:59.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:01:58 smithi171 ceph-mon[44189]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:02:00.593 INFO:teuthology.orchestra.run.smithi005.stdout:Created osd(s) 2 on host 'smithi005' 2024-04-20T17:02:00.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:00 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:00.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:00 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:00.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:00 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:00.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:00 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:02:00.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:00 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:00.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:00 smithi171 ceph-mon[44189]: from='osd.2 v1:172.21.15.5:6809/2105468358' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-20T17:02:00.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:00 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:00.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:00 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:00.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:00 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:00.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:00 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:02:00.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:00 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:00.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:00 smithi005 ceph-mon[39562]: from='osd.2 v1:172.21.15.5:6809/2105468358' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-20T17:02:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:00 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:00 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:00 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:00 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:02:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:00 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:00 smithi106 ceph-mon[44423]: from='osd.2 v1:172.21.15.5:6809/2105468358' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-20T17:02:01.311 DEBUG:teuthology.orchestra.run.smithi005:osd.2> sudo journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.2.service 2024-04-20T17:02:01.313 INFO:tasks.cephadm:Deploying osd.3 on smithi005 with /dev/vg_nvme/lv_1... 2024-04-20T17:02:01.313 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-20T17:02:01.344 INFO:journalctl@ceph.osd.2.smithi005.stdout:-- Logs begin at Sat 2024-04-20 16:47:38 UTC. -- 2024-04-20T17:02:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:01 smithi171 ceph-mon[44189]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:02:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:01 smithi171 ceph-mon[44189]: from='osd.2 v1:172.21.15.5:6809/2105468358' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-20T17:02:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:01 smithi171 ceph-mon[44189]: osdmap e18: 3 total, 2 up, 3 in 2024-04-20T17:02:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:01 smithi171 ceph-mon[44189]: from='osd.2 v1:172.21.15.5:6809/2105468358' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:02:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:01 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T17:02:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:01 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:01 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:01 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:02:01.939 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:01 smithi005 ceph-mon[39562]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:02:01.939 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:01 smithi005 ceph-mon[39562]: from='osd.2 v1:172.21.15.5:6809/2105468358' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-20T17:02:01.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:01 smithi005 ceph-mon[39562]: osdmap e18: 3 total, 2 up, 3 in 2024-04-20T17:02:01.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:01 smithi005 ceph-mon[39562]: from='osd.2 v1:172.21.15.5:6809/2105468358' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:02:01.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:01 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T17:02:01.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:01 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:01.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:01 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:01.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:01 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:02:01.940 INFO:journalctl@ceph.osd.2.smithi005.stdout:Apr 20 17:02:01 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-2[72612]: 2024-04-20T17:02:01.559+0000 7f190951e700 -1 osd.2 0 waiting for initial osdmap 2024-04-20T17:02:01.940 INFO:journalctl@ceph.osd.2.smithi005.stdout:Apr 20 17:02:01 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-2[72612]: 2024-04-20T17:02:01.564+0000 7f1905311700 -1 osd.2 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T17:02:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:01 smithi106 ceph-mon[44423]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:02:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:01 smithi106 ceph-mon[44423]: from='osd.2 v1:172.21.15.5:6809/2105468358' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-20T17:02:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:01 smithi106 ceph-mon[44423]: osdmap e18: 3 total, 2 up, 3 in 2024-04-20T17:02:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:01 smithi106 ceph-mon[44423]: from='osd.2 v1:172.21.15.5:6809/2105468358' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:02:02.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:01 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T17:02:02.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:01 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:02.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:01 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:02.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:01 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:02:02.287 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:02:02.817 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:02 smithi005 ceph-mon[39562]: from='osd.2 v1:172.21.15.5:6809/2105468358' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2024-04-20T17:02:02.817 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:02 smithi005 ceph-mon[39562]: osdmap e19: 3 total, 2 up, 3 in 2024-04-20T17:02:02.817 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:02 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T17:02:02.817 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:02 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T17:02:02.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:02 smithi171 ceph-mon[44189]: from='osd.2 v1:172.21.15.5:6809/2105468358' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2024-04-20T17:02:02.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:02 smithi171 ceph-mon[44189]: osdmap e19: 3 total, 2 up, 3 in 2024-04-20T17:02:02.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:02 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T17:02:02.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:02 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T17:02:03.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:02 smithi106 ceph-mon[44423]: from='osd.2 v1:172.21.15.5:6809/2105468358' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2024-04-20T17:02:03.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:02 smithi106 ceph-mon[44423]: osdmap e19: 3 total, 2 up, 3 in 2024-04-20T17:02:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:02 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T17:02:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:02 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T17:02:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:03 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:02:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:03 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:02:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:02:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:03 smithi171 ceph-mon[44189]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:02:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:03 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T17:02:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:03 smithi171 ceph-mon[44189]: osd.2 v1:172.21.15.5:6809/2105468358 boot 2024-04-20T17:02:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:03 smithi171 ceph-mon[44189]: osdmap e20: 3 total, 3 up, 3 in 2024-04-20T17:02:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:03 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T17:02:03.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:03 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:02:03.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:03 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:02:03.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:02:03.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:03 smithi005 ceph-mon[39562]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:02:03.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:03 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T17:02:03.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:03 smithi005 ceph-mon[39562]: osd.2 v1:172.21.15.5:6809/2105468358 boot 2024-04-20T17:02:03.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:03 smithi005 ceph-mon[39562]: osdmap e20: 3 total, 3 up, 3 in 2024-04-20T17:02:03.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:03 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T17:02:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:03 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:02:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:03 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:02:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:02:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:03 smithi106 ceph-mon[44423]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 179 GiB / 179 GiB avail 2024-04-20T17:02:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:03 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T17:02:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:03 smithi106 ceph-mon[44423]: osd.2 v1:172.21.15.5:6809/2105468358 boot 2024-04-20T17:02:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:03 smithi106 ceph-mon[44423]: osdmap e20: 3 total, 3 up, 3 in 2024-04-20T17:02:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:03 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T17:02:04.693 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:04 smithi005 ceph-mon[39562]: osdmap e21: 3 total, 3 up, 3 in 2024-04-20T17:02:04.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:04 smithi171 ceph-mon[44189]: osdmap e21: 3 total, 3 up, 3 in 2024-04-20T17:02:05.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:04 smithi106 ceph-mon[44423]: osdmap e21: 3 total, 3 up, 3 in 2024-04-20T17:02:05.319 INFO:teuthology.orchestra.run.smithi005.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-20T17:02:05.319 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-20T17:02:05.319 INFO:teuthology.orchestra.run.smithi005.stderr: stderr: 10+0 records in 2024-04-20T17:02:05.319 INFO:teuthology.orchestra.run.smithi005.stderr:10+0 records out 2024-04-20T17:02:05.319 INFO:teuthology.orchestra.run.smithi005.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00806229 s, 1.3 GB/s 2024-04-20T17:02:05.319 INFO:teuthology.orchestra.run.smithi005.stderr:--> Zapping successful for: 2024-04-20T17:02:05.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:05 smithi005 ceph-mon[39562]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:05.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:05 smithi171 ceph-mon[44189]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:06.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:05 smithi106 ceph-mon[44423]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:06.358 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi005:vg_nvme/lv_1 2024-04-20T17:02:06.636 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:02:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:06 smithi171 ceph-mon[44189]: Detected new or changed devices on smithi005 2024-04-20T17:02:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:06 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:06 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:06 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:02:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:06 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:02:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:06 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:02:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:06 smithi171 ceph-mon[44189]: Adjusting osd_memory_target on smithi005 to 5708M 2024-04-20T17:02:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:06 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:06 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:07.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:06 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:02:07.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:06 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:07.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:06 smithi171 ceph-mon[44189]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:07.161 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:06 smithi005 ceph-mon[39562]: Detected new or changed devices on smithi005 2024-04-20T17:02:07.161 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:06 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:07.161 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:06 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:07.161 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:06 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:02:07.161 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:06 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:02:07.161 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:06 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:02:07.161 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:06 smithi005 ceph-mon[39562]: Adjusting osd_memory_target on smithi005 to 5708M 2024-04-20T17:02:07.161 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:06 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:07.161 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:06 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:07.162 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:06 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:02:07.162 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:06 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:07.162 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:06 smithi005 ceph-mon[39562]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:06 smithi106 ceph-mon[44423]: Detected new or changed devices on smithi005 2024-04-20T17:02:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:06 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:06 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:06 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:02:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:06 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:02:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:06 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:02:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:06 smithi106 ceph-mon[44423]: Adjusting osd_memory_target on smithi005 to 5708M 2024-04-20T17:02:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:06 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:06 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:06 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:02:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:06 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:06 smithi106 ceph-mon[44423]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:09.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:08 smithi005 ceph-mon[39562]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:09.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:08 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:02:09.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:08 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:02:09.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:08 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:09.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:08 smithi106 ceph-mon[44423]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:08 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:02:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:08 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:02:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:08 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:09.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:08 smithi171 ceph-mon[44189]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:08 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:02:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:08 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:02:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:08 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:10.204 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:09 smithi005 ceph-mon[39562]: from='client.14304 v1:172.21.15.5:0/2018386549' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:02:10.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:09 smithi106 ceph-mon[44423]: from='client.14304 v1:172.21.15.5:0/2018386549' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:02:10.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:09 smithi171 ceph-mon[44189]: from='client.14304 v1:172.21.15.5:0/2018386549' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:02:11.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:11 smithi005 ceph-mon[39562]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:11.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:11 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2444793734' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c92f1df-e298-4606-88ae-b8a8e79628a1"}]: dispatch 2024-04-20T17:02:11.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:11 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2444793734' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8c92f1df-e298-4606-88ae-b8a8e79628a1"}]': finished 2024-04-20T17:02:11.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:11 smithi005 ceph-mon[39562]: osdmap e22: 4 total, 3 up, 4 in 2024-04-20T17:02:11.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:11 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T17:02:12.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:11 smithi106 ceph-mon[44423]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:11 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2444793734' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c92f1df-e298-4606-88ae-b8a8e79628a1"}]: dispatch 2024-04-20T17:02:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:11 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2444793734' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8c92f1df-e298-4606-88ae-b8a8e79628a1"}]': finished 2024-04-20T17:02:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:11 smithi106 ceph-mon[44423]: osdmap e22: 4 total, 3 up, 4 in 2024-04-20T17:02:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:11 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T17:02:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:11 smithi171 ceph-mon[44189]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:11 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2444793734' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c92f1df-e298-4606-88ae-b8a8e79628a1"}]: dispatch 2024-04-20T17:02:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:11 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2444793734' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8c92f1df-e298-4606-88ae-b8a8e79628a1"}]': finished 2024-04-20T17:02:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:11 smithi171 ceph-mon[44189]: osdmap e22: 4 total, 3 up, 4 in 2024-04-20T17:02:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:11 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T17:02:13.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:12 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2978066374' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:02:13.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:12 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2978066374' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:02:13.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:12 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2978066374' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:02:14.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:13 smithi106 ceph-mon[44423]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:14.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:13 smithi171 ceph-mon[44189]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:14.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:13 smithi005 ceph-mon[39562]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:16.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:15 smithi106 ceph-mon[44423]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:16.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:15 smithi171 ceph-mon[44189]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:16.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:15 smithi005 ceph-mon[39562]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:17 smithi106 ceph-mon[44423]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:18.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:17 smithi171 ceph-mon[44189]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:18.144 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:17 smithi005 ceph-mon[39562]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:19.006 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:18 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-20T17:02:19.006 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:18 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:19.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:18 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-20T17:02:19.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:18 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:19.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:18 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-20T17:02:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:18 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:19.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:19 smithi005 ceph-mon[39562]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:19.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:19 smithi005 ceph-mon[39562]: Deploying daemon osd.3 on smithi005 2024-04-20T17:02:20.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:19 smithi106 ceph-mon[44423]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:19 smithi106 ceph-mon[44423]: Deploying daemon osd.3 on smithi005 2024-04-20T17:02:20.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:19 smithi171 ceph-mon[44189]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:19 smithi171 ceph-mon[44189]: Deploying daemon osd.3 on smithi005 2024-04-20T17:02:21.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:21 smithi005 ceph-mon[39562]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:22.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:21 smithi106 ceph-mon[44423]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:21 smithi171 ceph-mon[44189]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:23.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:22 smithi171 ceph-mon[44189]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:23.149 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:22 smithi005 ceph-mon[39562]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:23.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:22 smithi106 ceph-mon[44423]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:25.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:25 smithi005 ceph-mon[39562]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:25.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:25 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:25.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:25 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:25.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:25 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:02:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:25 smithi106 ceph-mon[44423]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:25 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:25 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:25 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:02:25.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:25 smithi171 ceph-mon[44189]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:25 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:25 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:25 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:02:27.850 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:27 smithi005 ceph-mon[39562]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:27.850 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:27 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:27.850 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:27 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:27.850 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:27 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:27.850 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:27 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:02:27.850 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:27 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:27.850 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:27 smithi005 ceph-mon[39562]: from='osd.3 v1:172.21.15.5:6813/377652039' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-20T17:02:27.850 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:27 smithi005 ceph-mon[39562]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-20T17:02:27.910 INFO:teuthology.orchestra.run.smithi005.stdout:Created osd(s) 3 on host 'smithi005' 2024-04-20T17:02:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:27 smithi106 ceph-mon[44423]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:27 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:27 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:27 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:27 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:02:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:27 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:27 smithi106 ceph-mon[44423]: from='osd.3 v1:172.21.15.5:6813/377652039' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-20T17:02:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:27 smithi106 ceph-mon[44423]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-20T17:02:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:27 smithi171 ceph-mon[44189]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:27 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:27 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:27 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:27 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:02:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:27 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:27 smithi171 ceph-mon[44189]: from='osd.3 v1:172.21.15.5:6813/377652039' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-20T17:02:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:27 smithi171 ceph-mon[44189]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-20T17:02:28.612 DEBUG:teuthology.orchestra.run.smithi005:osd.3> sudo journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.3.service 2024-04-20T17:02:28.614 INFO:tasks.cephadm:Deploying osd.4 on smithi106 with /dev/vg_nvme/lv_4... 2024-04-20T17:02:28.614 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-20T17:02:28.652 INFO:journalctl@ceph.osd.3.smithi005.stdout:-- Logs begin at Sat 2024-04-20 16:47:38 UTC. -- 2024-04-20T17:02:28.882 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:02:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:28 smithi106 ceph-mon[44423]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:28 smithi106 ceph-mon[44423]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-20T17:02:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:28 smithi106 ceph-mon[44423]: osdmap e23: 4 total, 3 up, 4 in 2024-04-20T17:02:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:28 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T17:02:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:28 smithi106 ceph-mon[44423]: from='osd.3 v1:172.21.15.5:6813/377652039' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:02:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:28 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:28 smithi106 ceph-mon[44423]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:02:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:28 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:28 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:02:29.179 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:28 smithi005 ceph-mon[39562]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:29.179 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:28 smithi005 ceph-mon[39562]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-20T17:02:29.179 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:28 smithi005 ceph-mon[39562]: osdmap e23: 4 total, 3 up, 4 in 2024-04-20T17:02:29.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:28 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T17:02:29.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:28 smithi005 ceph-mon[39562]: from='osd.3 v1:172.21.15.5:6813/377652039' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:02:29.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:28 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:29.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:28 smithi005 ceph-mon[39562]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:02:29.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:28 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:29.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:28 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:02:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:28 smithi171 ceph-mon[44189]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:28 smithi171 ceph-mon[44189]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-20T17:02:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:28 smithi171 ceph-mon[44189]: osdmap e23: 4 total, 3 up, 4 in 2024-04-20T17:02:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:28 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T17:02:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:28 smithi171 ceph-mon[44189]: from='osd.3 v1:172.21.15.5:6813/377652039' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:02:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:28 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:28 smithi171 ceph-mon[44189]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:02:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:28 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:28 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:02:29.461 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:02:29 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-3[80507]: 2024-04-20T17:02:29.178+0000 7f237538e700 -1 osd.3 0 waiting for initial osdmap 2024-04-20T17:02:29.461 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:02:29 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-3[80507]: 2024-04-20T17:02:29.182+0000 7f236d179700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T17:02:29.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:29 smithi005 ceph-mon[39562]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2024-04-20T17:02:29.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:29 smithi005 ceph-mon[39562]: osdmap e24: 4 total, 3 up, 4 in 2024-04-20T17:02:29.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:29 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T17:02:29.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:29 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T17:02:29.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:29 smithi005 ceph-mon[39562]: from='osd.3 ' entity='osd.3' 2024-04-20T17:02:30.106 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:29 smithi106 ceph-mon[44423]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2024-04-20T17:02:30.106 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:29 smithi106 ceph-mon[44423]: osdmap e24: 4 total, 3 up, 4 in 2024-04-20T17:02:30.106 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:29 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T17:02:30.106 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:29 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T17:02:30.106 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:29 smithi106 ceph-mon[44423]: from='osd.3 ' entity='osd.3' 2024-04-20T17:02:30.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:29 smithi171 ceph-mon[44189]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2024-04-20T17:02:30.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:29 smithi171 ceph-mon[44189]: osdmap e24: 4 total, 3 up, 4 in 2024-04-20T17:02:30.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:29 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T17:02:30.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:29 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T17:02:30.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:29 smithi171 ceph-mon[44189]: from='osd.3 ' entity='osd.3' 2024-04-20T17:02:30.931 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-20T17:02:30.931 INFO:teuthology.orchestra.run.smithi106.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-20T17:02:30.931 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10+0 records in 2024-04-20T17:02:30.931 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records out 2024-04-20T17:02:30.931 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110891 s, 946 MB/s 2024-04-20T17:02:30.931 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping successful for: 2024-04-20T17:02:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:30 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:02:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:30 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:02:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:02:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:02:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:30 smithi005 ceph-mon[39562]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:30 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T17:02:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:30 smithi005 ceph-mon[39562]: osd.3 v1:172.21.15.5:6813/377652039 boot 2024-04-20T17:02:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:30 smithi005 ceph-mon[39562]: osdmap e25: 4 total, 4 up, 4 in 2024-04-20T17:02:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:30 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T17:02:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:30 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:02:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:30 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:02:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:02:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:02:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:30 smithi106 ceph-mon[44423]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:30 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T17:02:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:30 smithi106 ceph-mon[44423]: osd.3 v1:172.21.15.5:6813/377652039 boot 2024-04-20T17:02:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:30 smithi106 ceph-mon[44423]: osdmap e25: 4 total, 4 up, 4 in 2024-04-20T17:02:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:30 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T17:02:31.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:30 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:02:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:30 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:02:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:02:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:02:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:30 smithi171 ceph-mon[44189]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-20T17:02:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:30 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T17:02:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:30 smithi171 ceph-mon[44189]: osd.3 v1:172.21.15.5:6813/377652039 boot 2024-04-20T17:02:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:30 smithi171 ceph-mon[44189]: osdmap e25: 4 total, 4 up, 4 in 2024-04-20T17:02:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:30 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T17:02:31.697 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi106:vg_nvme/lv_4 2024-04-20T17:02:31.956 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:02:32.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:32 smithi106 ceph-mon[44423]: osdmap e26: 4 total, 4 up, 4 in 2024-04-20T17:02:32.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:32.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:32.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:02:32.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:02:32.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:02:32.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:02:32.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:32.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:32.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:02:32.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:32.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:32 smithi005 ceph-mon[39562]: osdmap e26: 4 total, 4 up, 4 in 2024-04-20T17:02:32.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:32.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:32.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:02:32.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:02:32.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:02:32.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:02:32.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:32.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:32.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:02:32.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:32 smithi171 ceph-mon[44189]: osdmap e26: 4 total, 4 up, 4 in 2024-04-20T17:02:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:02:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:02:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:02:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:02:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:02:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:33.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:33 smithi005 ceph-mon[39562]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:33.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:33 smithi005 ceph-mon[39562]: Detected new or changed devices on smithi005 2024-04-20T17:02:33.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:33 smithi005 ceph-mon[39562]: Adjusting osd_memory_target on smithi005 to 4281M 2024-04-20T17:02:33.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:33 smithi106 ceph-mon[44423]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:33.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:33 smithi106 ceph-mon[44423]: Detected new or changed devices on smithi005 2024-04-20T17:02:33.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:33 smithi106 ceph-mon[44423]: Adjusting osd_memory_target on smithi005 to 4281M 2024-04-20T17:02:33.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:33 smithi171 ceph-mon[44189]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:33.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:33 smithi171 ceph-mon[44189]: Detected new or changed devices on smithi005 2024-04-20T17:02:33.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:33 smithi171 ceph-mon[44189]: Adjusting osd_memory_target on smithi005 to 4281M 2024-04-20T17:02:34.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:34 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:02:34.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:34 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:02:34.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:34 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:34.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:34 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:02:34.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:34 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:02:34.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:34 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:34.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:34 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:02:34.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:34 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:02:34.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:34 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:35 smithi005 ceph-mon[39562]: from='client.24233 v1:172.21.15.106:0/1025808507' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:02:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:35 smithi005 ceph-mon[39562]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:35.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:35 smithi106 ceph-mon[44423]: from='client.24233 v1:172.21.15.106:0/1025808507' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:02:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:35 smithi106 ceph-mon[44423]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:35 smithi171 ceph-mon[44189]: from='client.24233 v1:172.21.15.106:0/1025808507' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:02:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:35 smithi171 ceph-mon[44189]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:36 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:02:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:36 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:02:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:36 smithi106 ceph-mon[44423]: osdmap e27: 4 total, 4 up, 4 in 2024-04-20T17:02:36.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:36 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:02:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:36 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:02:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:36 smithi171 ceph-mon[44189]: osdmap e27: 4 total, 4 up, 4 in 2024-04-20T17:02:36.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:36 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:02:36.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:36 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:02:36.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:36 smithi005 ceph-mon[39562]: osdmap e27: 4 total, 4 up, 4 in 2024-04-20T17:02:37.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:37 smithi106 ceph-mon[44423]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:37 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.106:0/1456150227' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46df16f1-9607-4d7b-9be8-9d4236e160d1"}]: dispatch 2024-04-20T17:02:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:37 smithi106 ceph-mon[44423]: from='client.24239 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46df16f1-9607-4d7b-9be8-9d4236e160d1"}]: dispatch 2024-04-20T17:02:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:37 smithi106 ceph-mon[44423]: from='client.24239 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46df16f1-9607-4d7b-9be8-9d4236e160d1"}]': finished 2024-04-20T17:02:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:37 smithi106 ceph-mon[44423]: osdmap e28: 5 total, 4 up, 5 in 2024-04-20T17:02:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:37 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T17:02:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:37 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.106:0/2283959577' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:02:37.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:37 smithi171 ceph-mon[44189]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:37.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:37 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.106:0/1456150227' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46df16f1-9607-4d7b-9be8-9d4236e160d1"}]: dispatch 2024-04-20T17:02:37.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:37 smithi171 ceph-mon[44189]: from='client.24239 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46df16f1-9607-4d7b-9be8-9d4236e160d1"}]: dispatch 2024-04-20T17:02:37.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:37 smithi171 ceph-mon[44189]: from='client.24239 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46df16f1-9607-4d7b-9be8-9d4236e160d1"}]': finished 2024-04-20T17:02:37.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:37 smithi171 ceph-mon[44189]: osdmap e28: 5 total, 4 up, 5 in 2024-04-20T17:02:37.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:37 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T17:02:37.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:37 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.106:0/2283959577' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:02:37.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:37 smithi005 ceph-mon[39562]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:37.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:37 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.106:0/1456150227' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46df16f1-9607-4d7b-9be8-9d4236e160d1"}]: dispatch 2024-04-20T17:02:37.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:37 smithi005 ceph-mon[39562]: from='client.24239 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46df16f1-9607-4d7b-9be8-9d4236e160d1"}]: dispatch 2024-04-20T17:02:37.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:37 smithi005 ceph-mon[39562]: from='client.24239 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46df16f1-9607-4d7b-9be8-9d4236e160d1"}]': finished 2024-04-20T17:02:37.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:37 smithi005 ceph-mon[39562]: osdmap e28: 5 total, 4 up, 5 in 2024-04-20T17:02:37.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:37 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T17:02:37.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:37 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.106:0/2283959577' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:02:39.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:39 smithi106 ceph-mon[44423]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:39 smithi171 ceph-mon[44189]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:39.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:39 smithi005 ceph-mon[39562]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:41.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:41 smithi106 ceph-mon[44423]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:41.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:41 smithi171 ceph-mon[44189]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:41.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:41 smithi005 ceph-mon[39562]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:43.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:43 smithi106 ceph-mon[44423]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-20T17:02:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:43.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:43 smithi171 ceph-mon[44189]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:43.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-20T17:02:43.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:43.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:43 smithi005 ceph-mon[39562]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:43.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-20T17:02:43.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:44.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:44 smithi106 ceph-mon[44423]: Deploying daemon osd.4 on smithi106 2024-04-20T17:02:44.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:44 smithi171 ceph-mon[44189]: Deploying daemon osd.4 on smithi106 2024-04-20T17:02:44.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:44 smithi005 ceph-mon[39562]: Deploying daemon osd.4 on smithi106 2024-04-20T17:02:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:45 smithi106 ceph-mon[44423]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:45.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:45 smithi171 ceph-mon[44189]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:45.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:45 smithi005 ceph-mon[39562]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:47 smithi106 ceph-mon[44423]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:47 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:02:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:47 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:02:48.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:47 smithi171 ceph-mon[44189]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:48.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:47 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:02:48.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:47 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:02:48.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:47 smithi005 ceph-mon[39562]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:48.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:47 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:02:48.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:47 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:02:49.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:49 smithi005 ceph-mon[39562]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:49.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:49 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:49.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:49 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:49.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:49 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:02:49.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:49 smithi106 ceph-mon[44423]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:49.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:49 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:49.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:49 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:49.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:49 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:02:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:49 smithi171 ceph-mon[44189]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:49 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:49 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:49 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:02:51.635 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:51 smithi106 ceph-mon[44423]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:51.635 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:51 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:51.635 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:51 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:51.635 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:51 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:51.635 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:51 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:02:51.635 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:51 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:51.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:51 smithi005 ceph-mon[39562]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:51.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:51 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:51.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:51 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:51.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:51 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:51.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:51 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:02:51.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:51 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:51.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:51 smithi171 ceph-mon[44189]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:51.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:51 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:51.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:51 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:51.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:51 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:51.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:51 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:02:51.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:51 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:51.970 INFO:teuthology.orchestra.run.smithi106.stdout:Created osd(s) 4 on host 'smithi106' 2024-04-20T17:02:52.699 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:52 smithi106 ceph-mon[44423]: from='osd.4 v1:172.21.15.106:6800/1839524469' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-20T17:02:52.699 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:52 smithi106 ceph-mon[44423]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-20T17:02:52.699 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:52 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:52.699 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:52 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:52.700 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:52 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:02:52.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:52 smithi005 ceph-mon[39562]: from='osd.4 v1:172.21.15.106:6800/1839524469' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-20T17:02:52.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:52 smithi005 ceph-mon[39562]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-20T17:02:52.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:52 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:52.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:52 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:52.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:52 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:02:52.736 DEBUG:teuthology.orchestra.run.smithi106:osd.4> sudo journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.4.service 2024-04-20T17:02:52.738 INFO:tasks.cephadm:Deploying osd.5 on smithi106 with /dev/vg_nvme/lv_3... 2024-04-20T17:02:52.738 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-20T17:02:52.766 INFO:journalctl@ceph.osd.4.smithi106.stdout:-- Logs begin at Sat 2024-04-20 16:47:44 UTC. -- 2024-04-20T17:02:52.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:52 smithi171 ceph-mon[44189]: from='osd.4 v1:172.21.15.106:6800/1839524469' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-20T17:02:52.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:52 smithi171 ceph-mon[44189]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-20T17:02:52.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:52 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:52.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:52 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:52.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:52 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:02:53.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:53 smithi005 ceph-mon[39562]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:53.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:53 smithi005 ceph-mon[39562]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-20T17:02:53.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:53 smithi005 ceph-mon[39562]: osdmap e29: 5 total, 4 up, 5 in 2024-04-20T17:02:53.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:53 smithi005 ceph-mon[39562]: from='osd.4 v1:172.21.15.106:6800/1839524469' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:02:53.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:53 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T17:02:53.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:53 smithi005 ceph-mon[39562]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:02:53.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:53 smithi106 ceph-mon[44423]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:53.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:53 smithi106 ceph-mon[44423]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-20T17:02:53.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:53 smithi106 ceph-mon[44423]: osdmap e29: 5 total, 4 up, 5 in 2024-04-20T17:02:53.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:53 smithi106 ceph-mon[44423]: from='osd.4 v1:172.21.15.106:6800/1839524469' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:02:53.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:53 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T17:02:53.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:53 smithi106 ceph-mon[44423]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:02:53.761 INFO:journalctl@ceph.osd.4.smithi106.stdout:Apr 20 17:02:53 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-4[49872]: 2024-04-20T17:02:53.444+0000 7fb319be7700 -1 osd.4 0 waiting for initial osdmap 2024-04-20T17:02:53.761 INFO:journalctl@ceph.osd.4.smithi106.stdout:Apr 20 17:02:53 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-4[49872]: 2024-04-20T17:02:53.448+0000 7fb3159da700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T17:02:53.810 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:02:53.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:53 smithi171 ceph-mon[44189]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:53.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:53 smithi171 ceph-mon[44189]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-20T17:02:53.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:53 smithi171 ceph-mon[44189]: osdmap e29: 5 total, 4 up, 5 in 2024-04-20T17:02:53.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:53 smithi171 ceph-mon[44189]: from='osd.4 v1:172.21.15.106:6800/1839524469' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:02:53.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:53 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T17:02:53.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:53 smithi171 ceph-mon[44189]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:02:54.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:54 smithi005 ceph-mon[39562]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-04-20T17:02:54.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:54 smithi005 ceph-mon[39562]: osdmap e30: 5 total, 4 up, 5 in 2024-04-20T17:02:54.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:54 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T17:02:54.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:54 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T17:02:54.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:54 smithi106 ceph-mon[44423]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-04-20T17:02:54.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:54 smithi106 ceph-mon[44423]: osdmap e30: 5 total, 4 up, 5 in 2024-04-20T17:02:54.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:54 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T17:02:54.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:54 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T17:02:54.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:54 smithi171 ceph-mon[44189]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-04-20T17:02:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:54 smithi171 ceph-mon[44189]: osdmap e30: 5 total, 4 up, 5 in 2024-04-20T17:02:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:54 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T17:02:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:54 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T17:02:55.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:55 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:02:55.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:55 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:02:55.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:02:55.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:55 smithi005 ceph-mon[39562]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:55.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:55 smithi005 ceph-mon[39562]: osd.4 v1:172.21.15.106:6800/1839524469 boot 2024-04-20T17:02:55.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:55 smithi005 ceph-mon[39562]: osdmap e31: 5 total, 5 up, 5 in 2024-04-20T17:02:55.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:55 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T17:02:55.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:55 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:02:55.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:55 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:02:55.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:02:55.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:55 smithi106 ceph-mon[44423]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:55.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:55 smithi106 ceph-mon[44423]: osd.4 v1:172.21.15.106:6800/1839524469 boot 2024-04-20T17:02:55.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:55 smithi106 ceph-mon[44423]: osdmap e31: 5 total, 5 up, 5 in 2024-04-20T17:02:55.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:55 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T17:02:55.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:55 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:02:55.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:55 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:02:55.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:02:55.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:55 smithi171 ceph-mon[44189]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-20T17:02:55.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:55 smithi171 ceph-mon[44189]: osd.4 v1:172.21.15.106:6800/1839524469 boot 2024-04-20T17:02:55.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:55 smithi171 ceph-mon[44189]: osdmap e31: 5 total, 5 up, 5 in 2024-04-20T17:02:55.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:55 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-20T17:02:56.197 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-20T17:02:56.198 INFO:teuthology.orchestra.run.smithi106.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-20T17:02:56.198 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10+0 records in 2024-04-20T17:02:56.198 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records out 2024-04-20T17:02:56.198 INFO:teuthology.orchestra.run.smithi106.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123943 s, 846 MB/s 2024-04-20T17:02:56.198 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping successful for: 2024-04-20T17:02:56.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:56 smithi106 ceph-mon[44423]: osdmap e32: 5 total, 5 up, 5 in 2024-04-20T17:02:56.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:56 smithi005 ceph-mon[39562]: osdmap e32: 5 total, 5 up, 5 in 2024-04-20T17:02:56.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:56 smithi171 ceph-mon[44189]: osdmap e32: 5 total, 5 up, 5 in 2024-04-20T17:02:56.926 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi106:vg_nvme/lv_3 2024-04-20T17:02:57.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:57 smithi106 ceph-mon[44423]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:02:57.620 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:02:57.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:57 smithi005 ceph-mon[39562]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:02:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:57 smithi171 ceph-mon[44189]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:02:58.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:58 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:02:58.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:58 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:02:58.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:58 smithi106 ceph-mon[44423]: osdmap e33: 5 total, 5 up, 5 in 2024-04-20T17:02:58.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:58 smithi106 ceph-mon[44423]: Detected new or changed devices on smithi106 2024-04-20T17:02:58.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:58 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:58.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:58 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:58.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:58 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:02:58.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:58 smithi106 ceph-mon[44423]: Adjusting osd_memory_target on smithi106 to 17122M 2024-04-20T17:02:58.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:58 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:58.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:58 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:58.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:58 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:02:58.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:58 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:58.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:58 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:02:58.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:58 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:02:58.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:58 smithi171 ceph-mon[44189]: osdmap e33: 5 total, 5 up, 5 in 2024-04-20T17:02:58.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:58 smithi171 ceph-mon[44189]: Detected new or changed devices on smithi106 2024-04-20T17:02:58.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:58 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:58.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:58 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:58.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:58 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:02:58.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:58 smithi171 ceph-mon[44189]: Adjusting osd_memory_target on smithi106 to 17122M 2024-04-20T17:02:58.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:58 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:58.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:58 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:58.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:58 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:02:58.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:58 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:58.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:58 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:02:58.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:58 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:02:58.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:58 smithi005 ceph-mon[39562]: osdmap e33: 5 total, 5 up, 5 in 2024-04-20T17:02:58.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:58 smithi005 ceph-mon[39562]: Detected new or changed devices on smithi106 2024-04-20T17:02:58.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:58 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:58.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:58 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:58.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:58 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:02:58.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:58 smithi005 ceph-mon[39562]: Adjusting osd_memory_target on smithi106 to 17122M 2024-04-20T17:02:58.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:58 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:58.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:58 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:02:58.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:58 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:02:58.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:58 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:02:59.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:02:59 smithi005 ceph-mon[39562]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:00.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:02:59 smithi106 ceph-mon[44423]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:00.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:02:59 smithi171 ceph-mon[44189]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:00.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:00 smithi005 ceph-mon[39562]: from='client.24260 v1:172.21.15.106:0/4061905517' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:03:00.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:00 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:03:00.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:00 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:03:00.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:00 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:03:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:00 smithi106 ceph-mon[44423]: from='client.24260 v1:172.21.15.106:0/4061905517' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:03:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:00 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:03:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:00 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:03:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:00 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:03:01.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:00 smithi171 ceph-mon[44189]: from='client.24260 v1:172.21.15.106:0/4061905517' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:03:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:00 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:03:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:00 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:03:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:00 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:03:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:01 smithi106 ceph-mon[44423]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:02.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:01 smithi171 ceph-mon[44189]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:02.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:01 smithi005 ceph-mon[39562]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:02 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.106:0/745162300' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6603231d-1e5c-4575-bfaa-2eeca2c12215"}]: dispatch 2024-04-20T17:03:03.071 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:02 smithi106 ceph-mon[44423]: from='client.24266 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6603231d-1e5c-4575-bfaa-2eeca2c12215"}]: dispatch 2024-04-20T17:03:03.072 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:02 smithi106 ceph-mon[44423]: from='client.24266 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6603231d-1e5c-4575-bfaa-2eeca2c12215"}]': finished 2024-04-20T17:03:03.072 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:02 smithi106 ceph-mon[44423]: osdmap e34: 6 total, 5 up, 6 in 2024-04-20T17:03:03.072 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:02 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T17:03:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:02 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.106:0/745162300' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6603231d-1e5c-4575-bfaa-2eeca2c12215"}]: dispatch 2024-04-20T17:03:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:02 smithi171 ceph-mon[44189]: from='client.24266 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6603231d-1e5c-4575-bfaa-2eeca2c12215"}]: dispatch 2024-04-20T17:03:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:02 smithi171 ceph-mon[44189]: from='client.24266 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6603231d-1e5c-4575-bfaa-2eeca2c12215"}]': finished 2024-04-20T17:03:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:02 smithi171 ceph-mon[44189]: osdmap e34: 6 total, 5 up, 6 in 2024-04-20T17:03:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:02 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T17:03:03.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:02 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.106:0/745162300' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6603231d-1e5c-4575-bfaa-2eeca2c12215"}]: dispatch 2024-04-20T17:03:03.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:02 smithi005 ceph-mon[39562]: from='client.24266 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6603231d-1e5c-4575-bfaa-2eeca2c12215"}]: dispatch 2024-04-20T17:03:03.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:02 smithi005 ceph-mon[39562]: from='client.24266 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6603231d-1e5c-4575-bfaa-2eeca2c12215"}]': finished 2024-04-20T17:03:03.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:02 smithi005 ceph-mon[39562]: osdmap e34: 6 total, 5 up, 6 in 2024-04-20T17:03:03.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:02 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T17:03:04.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:03 smithi106 ceph-mon[44423]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:03 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.106:0/3241570368' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:03:04.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:03 smithi171 ceph-mon[44189]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:03 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.106:0/3241570368' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:03:04.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:03 smithi005 ceph-mon[39562]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:04.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:03 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.106:0/3241570368' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:03:06.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:05 smithi106 ceph-mon[44423]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:06.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:05 smithi171 ceph-mon[44189]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:06.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:05 smithi005 ceph-mon[39562]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:08.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:07 smithi106 ceph-mon[44423]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:07 smithi171 ceph-mon[44189]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:08.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:07 smithi005 ceph-mon[39562]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:08.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:08 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-20T17:03:08.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:08 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:03:09.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:08 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-20T17:03:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:08 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:03:09.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:08 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-20T17:03:09.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:08 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:03:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:09 smithi106 ceph-mon[44423]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:09 smithi106 ceph-mon[44423]: Deploying daemon osd.5 on smithi106 2024-04-20T17:03:10.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:09 smithi171 ceph-mon[44189]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:09 smithi171 ceph-mon[44189]: Deploying daemon osd.5 on smithi106 2024-04-20T17:03:10.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:09 smithi005 ceph-mon[39562]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:10.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:09 smithi005 ceph-mon[39562]: Deploying daemon osd.5 on smithi106 2024-04-20T17:03:11.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:10 smithi106 ceph-mon[44423]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:11.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:10 smithi171 ceph-mon[44189]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:11.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:10 smithi005 ceph-mon[39562]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:13.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:12 smithi005 ceph-mon[39562]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:13.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:12 smithi106 ceph-mon[44423]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:13.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:12 smithi171 ceph-mon[44189]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:15.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:15 smithi005 ceph-mon[39562]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:15.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:15 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:15.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:15 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:15.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:15 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:03:15.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:15 smithi106 ceph-mon[44423]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:15.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:15 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:15.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:15 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:15.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:15 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:03:15.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:15 smithi171 ceph-mon[44189]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:15.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:15 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:15.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:15 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:15.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:15 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:03:17.534 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:17 smithi106 ceph-mon[44423]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:17.535 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:17 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:17.535 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:17 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:17.535 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:17 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:03:17.535 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:17 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:03:17.535 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:17 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:17 smithi171 ceph-mon[44189]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:17 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:17 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:17 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:03:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:17 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:03:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:17 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:17.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:17 smithi005 ceph-mon[39562]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:17.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:17 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:17.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:17 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:17.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:17 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:03:17.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:17 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:03:17.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:17 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:17.965 INFO:teuthology.orchestra.run.smithi106.stdout:Created osd(s) 5 on host 'smithi106' 2024-04-20T17:03:18.675 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:18 smithi106 ceph-mon[44423]: from='osd.5 v1:172.21.15.106:6804/2450315908' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-20T17:03:18.675 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:18 smithi106 ceph-mon[44423]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-20T17:03:18.675 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:18 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:18.675 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:18 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:18.676 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:18 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:03:18.710 DEBUG:teuthology.orchestra.run.smithi106:osd.5> sudo journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.5.service 2024-04-20T17:03:18.712 INFO:tasks.cephadm:Deploying osd.6 on smithi106 with /dev/vg_nvme/lv_2... 2024-04-20T17:03:18.712 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-20T17:03:18.743 INFO:journalctl@ceph.osd.5.smithi106.stdout:-- Logs begin at Sat 2024-04-20 16:47:44 UTC. -- 2024-04-20T17:03:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:18 smithi171 ceph-mon[44189]: from='osd.5 v1:172.21.15.106:6804/2450315908' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-20T17:03:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:18 smithi171 ceph-mon[44189]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-20T17:03:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:18 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:18 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:18 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:03:18.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:18 smithi005 ceph-mon[39562]: from='osd.5 v1:172.21.15.106:6804/2450315908' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-20T17:03:18.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:18 smithi005 ceph-mon[39562]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-20T17:03:18.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:18 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:18.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:18 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:18.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:18 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:03:19.545 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:03:19.788 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:19 smithi106 ceph-mon[44423]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:19.788 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:19 smithi106 ceph-mon[44423]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-20T17:03:19.788 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:19 smithi106 ceph-mon[44423]: osdmap e35: 6 total, 5 up, 6 in 2024-04-20T17:03:19.789 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:19 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T17:03:19.789 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:19 smithi106 ceph-mon[44423]: from='osd.5 v1:172.21.15.106:6804/2450315908' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:03:19.789 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:19 smithi106 ceph-mon[44423]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:03:19.789 INFO:journalctl@ceph.osd.5.smithi106.stdout:Apr 20 17:03:19 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-5[57661]: 2024-04-20T17:03:19.541+0000 7fbd066d8700 -1 osd.5 0 waiting for initial osdmap 2024-04-20T17:03:19.789 INFO:journalctl@ceph.osd.5.smithi106.stdout:Apr 20 17:03:19 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-5[57661]: 2024-04-20T17:03:19.546+0000 7fbd024cb700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T17:03:19.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:19 smithi171 ceph-mon[44189]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:19.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:19 smithi171 ceph-mon[44189]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-20T17:03:19.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:19 smithi171 ceph-mon[44189]: osdmap e35: 6 total, 5 up, 6 in 2024-04-20T17:03:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:19 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T17:03:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:19 smithi171 ceph-mon[44189]: from='osd.5 v1:172.21.15.106:6804/2450315908' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:03:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:19 smithi171 ceph-mon[44189]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:03:19.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:19 smithi005 ceph-mon[39562]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:19.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:19 smithi005 ceph-mon[39562]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-20T17:03:19.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:19 smithi005 ceph-mon[39562]: osdmap e35: 6 total, 5 up, 6 in 2024-04-20T17:03:19.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:19 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T17:03:19.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:19 smithi005 ceph-mon[39562]: from='osd.5 v1:172.21.15.106:6804/2450315908' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:03:19.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:19 smithi005 ceph-mon[39562]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:03:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:20 smithi106 ceph-mon[44423]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-04-20T17:03:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:20 smithi106 ceph-mon[44423]: osdmap e36: 6 total, 5 up, 6 in 2024-04-20T17:03:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:20 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T17:03:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:20 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T17:03:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:20 smithi106 ceph-mon[44423]: osd.5 v1:172.21.15.106:6804/2450315908 boot 2024-04-20T17:03:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:20 smithi106 ceph-mon[44423]: osdmap e37: 6 total, 6 up, 6 in 2024-04-20T17:03:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:20 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T17:03:20.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:20 smithi171 ceph-mon[44189]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-04-20T17:03:20.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:20 smithi171 ceph-mon[44189]: osdmap e36: 6 total, 5 up, 6 in 2024-04-20T17:03:20.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:20 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T17:03:20.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:20 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T17:03:20.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:20 smithi171 ceph-mon[44189]: osd.5 v1:172.21.15.106:6804/2450315908 boot 2024-04-20T17:03:20.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:20 smithi171 ceph-mon[44189]: osdmap e37: 6 total, 6 up, 6 in 2024-04-20T17:03:20.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:20 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T17:03:20.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:20 smithi005 ceph-mon[39562]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-04-20T17:03:20.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:20 smithi005 ceph-mon[39562]: osdmap e36: 6 total, 5 up, 6 in 2024-04-20T17:03:20.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:20 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T17:03:20.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:20 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T17:03:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:20 smithi005 ceph-mon[39562]: osd.5 v1:172.21.15.106:6804/2450315908 boot 2024-04-20T17:03:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:20 smithi005 ceph-mon[39562]: osdmap e37: 6 total, 6 up, 6 in 2024-04-20T17:03:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:20 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-20T17:03:21.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:21 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:03:21.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:21 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:03:21.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:03:21.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:21 smithi171 ceph-mon[44189]: pgmap v143: 1 pgs: 1 unknown; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:21.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:21 smithi171 ceph-mon[44189]: osdmap e38: 6 total, 6 up, 6 in 2024-04-20T17:03:21.910 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:21 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:03:21.910 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:21 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:03:21.910 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:03:21.911 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:21 smithi106 ceph-mon[44423]: pgmap v143: 1 pgs: 1 unknown; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:21.911 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:21 smithi106 ceph-mon[44423]: osdmap e38: 6 total, 6 up, 6 in 2024-04-20T17:03:21.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:21 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:03:21.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:21 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:03:21.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:03:21.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:21 smithi005 ceph-mon[39562]: pgmap v143: 1 pgs: 1 unknown; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-20T17:03:21.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:21 smithi005 ceph-mon[39562]: osdmap e38: 6 total, 6 up, 6 in 2024-04-20T17:03:22.663 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-20T17:03:22.663 INFO:teuthology.orchestra.run.smithi106.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-20T17:03:22.664 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10+0 records in 2024-04-20T17:03:22.664 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records out 2024-04-20T17:03:22.664 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115873 s, 905 MB/s 2024-04-20T17:03:22.664 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping successful for: 2024-04-20T17:03:22.991 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:22 smithi106 ceph-mon[44423]: osdmap e39: 6 total, 6 up, 6 in 2024-04-20T17:03:23.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:22 smithi171 ceph-mon[44189]: osdmap e39: 6 total, 6 up, 6 in 2024-04-20T17:03:23.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:22 smithi005 ceph-mon[39562]: osdmap e39: 6 total, 6 up, 6 in 2024-04-20T17:03:23.365 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi106:vg_nvme/lv_2 2024-04-20T17:03:23.626 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:03:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:23 smithi106 ceph-mon[44423]: pgmap v147: 1 pgs: 1 unknown; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:23 smithi106 ceph-mon[44423]: osdmap e40: 6 total, 6 up, 6 in 2024-04-20T17:03:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:23 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:23 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:23 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:03:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:23 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:03:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:23 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:24.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:23 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:03:24.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:23 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:03:24.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:23 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:24.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:23 smithi171 ceph-mon[44189]: pgmap v147: 1 pgs: 1 unknown; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:24.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:23 smithi171 ceph-mon[44189]: osdmap e40: 6 total, 6 up, 6 in 2024-04-20T17:03:24.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:23 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:24.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:23 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:24.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:23 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:03:24.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:23 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:03:24.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:23 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:24.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:23 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:03:24.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:23 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:03:24.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:23 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:23 smithi005 ceph-mon[39562]: pgmap v147: 1 pgs: 1 unknown; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:23 smithi005 ceph-mon[39562]: osdmap e40: 6 total, 6 up, 6 in 2024-04-20T17:03:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:23 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:23 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:23 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:03:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:23 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:03:24.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:23 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:24.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:23 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:03:24.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:23 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:03:24.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:23 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:25.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:24 smithi106 ceph-mon[44423]: Detected new or changed devices on smithi106 2024-04-20T17:03:25.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:24 smithi106 ceph-mon[44423]: Adjusting osd_memory_target on smithi106 to 8561M 2024-04-20T17:03:25.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:24 smithi106 ceph-mon[44423]: osdmap e41: 6 total, 6 up, 6 in 2024-04-20T17:03:25.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:24 smithi171 ceph-mon[44189]: Detected new or changed devices on smithi106 2024-04-20T17:03:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:24 smithi171 ceph-mon[44189]: Adjusting osd_memory_target on smithi106 to 8561M 2024-04-20T17:03:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:24 smithi171 ceph-mon[44189]: osdmap e41: 6 total, 6 up, 6 in 2024-04-20T17:03:25.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:24 smithi005 ceph-mon[39562]: Detected new or changed devices on smithi106 2024-04-20T17:03:25.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:24 smithi005 ceph-mon[39562]: Adjusting osd_memory_target on smithi106 to 8561M 2024-04-20T17:03:25.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:24 smithi005 ceph-mon[39562]: osdmap e41: 6 total, 6 up, 6 in 2024-04-20T17:03:26.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:25 smithi106 ceph-mon[44423]: pgmap v150: 1 pgs: 1 unknown; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:26.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:25 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:03:26.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:25 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:03:26.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:25 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:03:26.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:25 smithi171 ceph-mon[44189]: pgmap v150: 1 pgs: 1 unknown; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:26.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:25 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:03:26.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:25 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:03:26.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:25 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:03:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:25 smithi005 ceph-mon[39562]: pgmap v150: 1 pgs: 1 unknown; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:25 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:03:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:25 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:03:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:25 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:03:27.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:26 smithi106 ceph-mon[44423]: from='client.24287 v1:172.21.15.106:0/3102584450' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:03:27.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:26 smithi106 ceph-mon[44423]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:26 smithi171 ceph-mon[44189]: from='client.24287 v1:172.21.15.106:0/3102584450' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:03:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:26 smithi171 ceph-mon[44189]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:26 smithi005 ceph-mon[39562]: from='client.24287 v1:172.21.15.106:0/3102584450' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:03:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:26 smithi005 ceph-mon[39562]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:29.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:28 smithi005 ceph-mon[39562]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:29.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:28 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.106:0/2351880932' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "783a55a6-f176-42be-8c8b-af8481d08dc9"}]: dispatch 2024-04-20T17:03:29.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:28 smithi005 ceph-mon[39562]: from='client.24293 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "783a55a6-f176-42be-8c8b-af8481d08dc9"}]: dispatch 2024-04-20T17:03:29.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:28 smithi005 ceph-mon[39562]: from='client.24293 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "783a55a6-f176-42be-8c8b-af8481d08dc9"}]': finished 2024-04-20T17:03:29.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:28 smithi005 ceph-mon[39562]: osdmap e42: 7 total, 6 up, 7 in 2024-04-20T17:03:29.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:28 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:03:29.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:28 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.106:0/1557223750' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:03:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:28 smithi106 ceph-mon[44423]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:28 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.106:0/2351880932' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "783a55a6-f176-42be-8c8b-af8481d08dc9"}]: dispatch 2024-04-20T17:03:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:28 smithi106 ceph-mon[44423]: from='client.24293 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "783a55a6-f176-42be-8c8b-af8481d08dc9"}]: dispatch 2024-04-20T17:03:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:28 smithi106 ceph-mon[44423]: from='client.24293 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "783a55a6-f176-42be-8c8b-af8481d08dc9"}]': finished 2024-04-20T17:03:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:28 smithi106 ceph-mon[44423]: osdmap e42: 7 total, 6 up, 7 in 2024-04-20T17:03:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:28 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:03:29.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:28 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.106:0/1557223750' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:03:29.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:28 smithi171 ceph-mon[44189]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:28 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.106:0/2351880932' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "783a55a6-f176-42be-8c8b-af8481d08dc9"}]: dispatch 2024-04-20T17:03:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:28 smithi171 ceph-mon[44189]: from='client.24293 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "783a55a6-f176-42be-8c8b-af8481d08dc9"}]: dispatch 2024-04-20T17:03:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:28 smithi171 ceph-mon[44189]: from='client.24293 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "783a55a6-f176-42be-8c8b-af8481d08dc9"}]': finished 2024-04-20T17:03:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:28 smithi171 ceph-mon[44189]: osdmap e42: 7 total, 6 up, 7 in 2024-04-20T17:03:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:28 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:03:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:28 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.106:0/1557223750' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:03:31.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:30 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:03:31.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:30 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:03:31.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:30 smithi005 ceph-mon[39562]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:31.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:30 smithi005 ceph-mon[39562]: osdmap e43: 7 total, 6 up, 7 in 2024-04-20T17:03:31.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:30 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:03:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:30 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:03:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:30 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:03:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:30 smithi106 ceph-mon[44423]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:30 smithi106 ceph-mon[44423]: osdmap e43: 7 total, 6 up, 7 in 2024-04-20T17:03:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:30 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:03:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:30 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:03:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:30 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:03:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:30 smithi171 ceph-mon[44189]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:30 smithi171 ceph-mon[44189]: osdmap e43: 7 total, 6 up, 7 in 2024-04-20T17:03:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:30 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:03:33.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:32 smithi005 ceph-mon[39562]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:33.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:32 smithi106 ceph-mon[44423]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:33.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:32 smithi171 ceph-mon[44189]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:35.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:34 smithi005 ceph-mon[39562]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:35.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:34 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-20T17:03:35.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:34 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:03:35.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:34 smithi106 ceph-mon[44423]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:34 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-20T17:03:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:34 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:03:35.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:34 smithi171 ceph-mon[44189]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:35.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:34 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-20T17:03:35.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:34 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:03:36.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:35 smithi005 ceph-mon[39562]: Deploying daemon osd.6 on smithi106 2024-04-20T17:03:36.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:35 smithi106 ceph-mon[44423]: Deploying daemon osd.6 on smithi106 2024-04-20T17:03:36.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:35 smithi171 ceph-mon[44189]: Deploying daemon osd.6 on smithi106 2024-04-20T17:03:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:36 smithi005 ceph-mon[39562]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:37.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:36 smithi106 ceph-mon[44423]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:37.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:36 smithi171 ceph-mon[44189]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:39.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:38 smithi005 ceph-mon[39562]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:39.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:38 smithi106 ceph-mon[44423]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:39.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:38 smithi171 ceph-mon[44189]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:41.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:41 smithi005 ceph-mon[39562]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:41.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:41 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:41.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:41 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:41.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:41 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:03:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:41 smithi106 ceph-mon[44423]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:41 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:41 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:41 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:03:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:41 smithi171 ceph-mon[44189]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:41 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:41 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:41 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:03:43.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:43 smithi005 ceph-mon[39562]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:43.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:43 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:03:43.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:43 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:03:43.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:43 smithi005 ceph-mon[39562]: osdmap e44: 7 total, 6 up, 7 in 2024-04-20T17:03:43.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:03:43.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:43.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:43 smithi106 ceph-mon[44423]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:43 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:03:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:43 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:03:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:43 smithi106 ceph-mon[44423]: osdmap e44: 7 total, 6 up, 7 in 2024-04-20T17:03:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:03:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:43 smithi171 ceph-mon[44189]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:43 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:03:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:43 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:03:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:43 smithi171 ceph-mon[44189]: osdmap e44: 7 total, 6 up, 7 in 2024-04-20T17:03:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:03:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:44.348 INFO:teuthology.orchestra.run.smithi106.stdout:Created osd(s) 6 on host 'smithi106' 2024-04-20T17:03:44.683 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:44 smithi106 ceph-mon[44423]: from='osd.6 v1:172.21.15.106:6808/2840237913' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-20T17:03:44.683 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:44 smithi106 ceph-mon[44423]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-20T17:03:44.683 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:03:44.683 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:03:44.683 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:44.683 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:44.683 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:44.683 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:03:44.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:44 smithi005 ceph-mon[39562]: from='osd.6 v1:172.21.15.106:6808/2840237913' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-20T17:03:44.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:44 smithi005 ceph-mon[39562]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-20T17:03:44.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:03:44.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:03:44.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:44.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:44.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:44.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:03:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:44 smithi171 ceph-mon[44189]: from='osd.6 v1:172.21.15.106:6808/2840237913' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-20T17:03:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:44 smithi171 ceph-mon[44189]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-20T17:03:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:03:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:03:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:03:45.011 DEBUG:teuthology.orchestra.run.smithi106:osd.6> sudo journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.6.service 2024-04-20T17:03:45.013 INFO:tasks.cephadm:Deploying osd.7 on smithi106 with /dev/vg_nvme/lv_1... 2024-04-20T17:03:45.013 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-20T17:03:45.042 INFO:journalctl@ceph.osd.6.smithi106.stdout:-- Logs begin at Sat 2024-04-20 16:47:44 UTC. -- 2024-04-20T17:03:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:45 smithi106 ceph-mon[44423]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:45 smithi106 ceph-mon[44423]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-20T17:03:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:45 smithi106 ceph-mon[44423]: osdmap e45: 7 total, 6 up, 7 in 2024-04-20T17:03:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:03:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:45 smithi106 ceph-mon[44423]: from='osd.6 v1:172.21.15.106:6808/2840237913' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:03:45.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:45 smithi106 ceph-mon[44423]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:03:45.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:45 smithi005 ceph-mon[39562]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:45.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:45 smithi005 ceph-mon[39562]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-20T17:03:45.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:45 smithi005 ceph-mon[39562]: osdmap e45: 7 total, 6 up, 7 in 2024-04-20T17:03:45.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:03:45.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:45 smithi005 ceph-mon[39562]: from='osd.6 v1:172.21.15.106:6808/2840237913' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:03:45.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:45 smithi005 ceph-mon[39562]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:03:45.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:45 smithi171 ceph-mon[44189]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:45 smithi171 ceph-mon[44189]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-20T17:03:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:45 smithi171 ceph-mon[44189]: osdmap e45: 7 total, 6 up, 7 in 2024-04-20T17:03:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:03:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:45 smithi171 ceph-mon[44189]: from='osd.6 v1:172.21.15.106:6808/2840237913' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:03:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:45 smithi171 ceph-mon[44189]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:03:46.196 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:03:45 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[65549]: 2024-04-20T17:03:45.830+0000 7f8826de8700 -1 osd.6 0 waiting for initial osdmap 2024-04-20T17:03:46.196 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:03:45 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[65549]: 2024-04-20T17:03:45.838+0000 7f881e3d2700 -1 osd.6 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T17:03:46.196 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:03:46.464 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:46 smithi106 ceph-mon[44423]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-04-20T17:03:46.465 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:46 smithi106 ceph-mon[44423]: osdmap e46: 7 total, 6 up, 7 in 2024-04-20T17:03:46.465 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:46 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:03:46.465 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:46 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:03:46.465 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:46 smithi106 ceph-mon[44423]: osdmap e47: 7 total, 6 up, 7 in 2024-04-20T17:03:46.465 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:46 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:03:46.465 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:46 smithi106 ceph-mon[44423]: from='osd.6 ' entity='osd.6' 2024-04-20T17:03:46.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:46 smithi005 ceph-mon[39562]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-04-20T17:03:46.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:46 smithi005 ceph-mon[39562]: osdmap e46: 7 total, 6 up, 7 in 2024-04-20T17:03:46.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:46 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:03:46.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:46 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:03:46.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:46 smithi005 ceph-mon[39562]: osdmap e47: 7 total, 6 up, 7 in 2024-04-20T17:03:46.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:46 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:03:46.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:46 smithi005 ceph-mon[39562]: from='osd.6 ' entity='osd.6' 2024-04-20T17:03:46.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:46 smithi171 ceph-mon[44189]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-04-20T17:03:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:46 smithi171 ceph-mon[44189]: osdmap e46: 7 total, 6 up, 7 in 2024-04-20T17:03:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:46 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:03:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:46 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:03:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:46 smithi171 ceph-mon[44189]: osdmap e47: 7 total, 6 up, 7 in 2024-04-20T17:03:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:46 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:03:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:46 smithi171 ceph-mon[44189]: from='osd.6 ' entity='osd.6' 2024-04-20T17:03:47.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:47 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:03:47.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:47 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:03:47.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:03:47.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:03:47.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:47 smithi005 ceph-mon[39562]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:47.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:47 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:03:47.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:47 smithi005 ceph-mon[39562]: osd.6 v1:172.21.15.106:6808/2840237913 boot 2024-04-20T17:03:47.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:47 smithi005 ceph-mon[39562]: osdmap e48: 7 total, 7 up, 7 in 2024-04-20T17:03:47.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:47 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:03:47.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:47 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:03:47.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:47 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:03:47.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:03:47.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:03:47.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:47 smithi106 ceph-mon[44423]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:47.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:47 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:03:47.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:47 smithi106 ceph-mon[44423]: osd.6 v1:172.21.15.106:6808/2840237913 boot 2024-04-20T17:03:47.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:47 smithi106 ceph-mon[44423]: osdmap e48: 7 total, 7 up, 7 in 2024-04-20T17:03:47.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:47 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:03:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:47 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:03:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:47 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:03:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:03:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:03:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:47 smithi171 ceph-mon[44189]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-20T17:03:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:47 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:03:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:47 smithi171 ceph-mon[44189]: osd.6 v1:172.21.15.106:6808/2840237913 boot 2024-04-20T17:03:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:47 smithi171 ceph-mon[44189]: osdmap e48: 7 total, 7 up, 7 in 2024-04-20T17:03:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:47 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:03:48.638 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-20T17:03:48.638 INFO:teuthology.orchestra.run.smithi106.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-20T17:03:48.638 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10+0 records in 2024-04-20T17:03:48.639 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records out 2024-04-20T17:03:48.639 INFO:teuthology.orchestra.run.smithi106.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0149415 s, 702 MB/s 2024-04-20T17:03:48.639 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping successful for: 2024-04-20T17:03:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:48 smithi106 ceph-mon[44423]: osdmap e49: 7 total, 7 up, 7 in 2024-04-20T17:03:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:48 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:03:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:48 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:03:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:48 smithi171 ceph-mon[44189]: osdmap e49: 7 total, 7 up, 7 in 2024-04-20T17:03:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:48 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:03:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:48 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:03:49.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:48 smithi005 ceph-mon[39562]: osdmap e49: 7 total, 7 up, 7 in 2024-04-20T17:03:49.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:48 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:03:49.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:48 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:03:49.533 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi106:vg_nvme/lv_1 2024-04-20T17:03:49.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:49 smithi106 ceph-mon[44423]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:03:49.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:49 smithi106 ceph-mon[44423]: osdmap e50: 7 total, 7 up, 7 in 2024-04-20T17:03:50.034 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:03:50.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:49 smithi171 ceph-mon[44189]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:03:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:49 smithi171 ceph-mon[44189]: osdmap e50: 7 total, 7 up, 7 in 2024-04-20T17:03:50.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:49 smithi005 ceph-mon[39562]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:03:50.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:49 smithi005 ceph-mon[39562]: osdmap e50: 7 total, 7 up, 7 in 2024-04-20T17:03:51.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:50 smithi106 ceph-mon[44423]: osdmap e51: 7 total, 7 up, 7 in 2024-04-20T17:03:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:50 smithi106 ceph-mon[44423]: pgmap v173: 1 pgs: 1 active+clean+remapped; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail; 258 KiB/s, 0 objects/s recovering 2024-04-20T17:03:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:50 smithi106 ceph-mon[44423]: Detected new or changed devices on smithi106 2024-04-20T17:03:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:50 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:50 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:50 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:03:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:50 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:03:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:50 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:03:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:50 smithi106 ceph-mon[44423]: Adjusting osd_memory_target on smithi106 to 5707M 2024-04-20T17:03:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:50 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:50 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:03:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:50 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:03:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:50 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:50 smithi171 ceph-mon[44189]: osdmap e51: 7 total, 7 up, 7 in 2024-04-20T17:03:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:50 smithi171 ceph-mon[44189]: pgmap v173: 1 pgs: 1 active+clean+remapped; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail; 258 KiB/s, 0 objects/s recovering 2024-04-20T17:03:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:50 smithi171 ceph-mon[44189]: Detected new or changed devices on smithi106 2024-04-20T17:03:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:50 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:50 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:50 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:03:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:50 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:03:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:50 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:03:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:50 smithi171 ceph-mon[44189]: Adjusting osd_memory_target on smithi106 to 5707M 2024-04-20T17:03:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:50 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:51.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:50 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:03:51.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:50 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:03:51.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:50 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:51.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:50 smithi005 ceph-mon[39562]: osdmap e51: 7 total, 7 up, 7 in 2024-04-20T17:03:51.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:50 smithi005 ceph-mon[39562]: pgmap v173: 1 pgs: 1 active+clean+remapped; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail; 258 KiB/s, 0 objects/s recovering 2024-04-20T17:03:51.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:50 smithi005 ceph-mon[39562]: Detected new or changed devices on smithi106 2024-04-20T17:03:51.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:50 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:51.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:50 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:51.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:50 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:03:51.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:50 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:03:51.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:50 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:03:51.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:50 smithi005 ceph-mon[39562]: Adjusting osd_memory_target on smithi106 to 5707M 2024-04-20T17:03:51.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:50 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:51.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:50 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:03:51.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:50 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:03:51.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:50 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:03:52.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:51 smithi106 ceph-mon[44423]: osdmap e52: 7 total, 7 up, 7 in 2024-04-20T17:03:52.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:51 smithi171 ceph-mon[44189]: osdmap e52: 7 total, 7 up, 7 in 2024-04-20T17:03:52.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:51 smithi005 ceph-mon[39562]: osdmap e52: 7 total, 7 up, 7 in 2024-04-20T17:03:52.758 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:52 smithi106 ceph-mon[44423]: pgmap v175: 1 pgs: 1 active+clean+remapped; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail; 251 KiB/s, 0 objects/s recovering 2024-04-20T17:03:53.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:52 smithi106 ceph-mon[44423]: from='client.24314 v1:172.21.15.106:0/2232175702' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:03:53.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:52 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:03:53.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:52 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:03:53.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:52 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:03:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:52 smithi171 ceph-mon[44189]: pgmap v175: 1 pgs: 1 active+clean+remapped; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail; 251 KiB/s, 0 objects/s recovering 2024-04-20T17:03:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:52 smithi171 ceph-mon[44189]: from='client.24314 v1:172.21.15.106:0/2232175702' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:03:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:52 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:03:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:52 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:03:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:52 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:03:53.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:52 smithi005 ceph-mon[39562]: pgmap v175: 1 pgs: 1 active+clean+remapped; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail; 251 KiB/s, 0 objects/s recovering 2024-04-20T17:03:53.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:52 smithi005 ceph-mon[39562]: from='client.24314 v1:172.21.15.106:0/2232175702' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:03:53.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:52 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:03:53.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:52 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:03:53.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:52 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:03:55.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:54 smithi005 ceph-mon[39562]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-20T17:03:55.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:54 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.106:0/1230556811' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4"}]: dispatch 2024-04-20T17:03:55.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:54 smithi005 ceph-mon[39562]: from='client.24320 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4"}]: dispatch 2024-04-20T17:03:55.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:54 smithi005 ceph-mon[39562]: from='client.24320 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4"}]': finished 2024-04-20T17:03:55.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:54 smithi005 ceph-mon[39562]: osdmap e53: 8 total, 7 up, 8 in 2024-04-20T17:03:55.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:54 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-20T17:03:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:54 smithi106 ceph-mon[44423]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-20T17:03:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:54 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.106:0/1230556811' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4"}]: dispatch 2024-04-20T17:03:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:54 smithi106 ceph-mon[44423]: from='client.24320 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4"}]: dispatch 2024-04-20T17:03:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:54 smithi106 ceph-mon[44423]: from='client.24320 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4"}]': finished 2024-04-20T17:03:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:54 smithi106 ceph-mon[44423]: osdmap e53: 8 total, 7 up, 8 in 2024-04-20T17:03:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:54 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-20T17:03:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:54 smithi171 ceph-mon[44189]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-20T17:03:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:54 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.106:0/1230556811' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4"}]: dispatch 2024-04-20T17:03:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:54 smithi171 ceph-mon[44189]: from='client.24320 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4"}]: dispatch 2024-04-20T17:03:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:54 smithi171 ceph-mon[44189]: from='client.24320 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4"}]': finished 2024-04-20T17:03:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:54 smithi171 ceph-mon[44189]: osdmap e53: 8 total, 7 up, 8 in 2024-04-20T17:03:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:54 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-20T17:03:56.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:55 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.106:0/1662450121' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:03:56.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:55 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.106:0/1662450121' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:03:56.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:55 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.106:0/1662450121' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:03:57.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:56 smithi005 ceph-mon[39562]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:03:57.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:56 smithi106 ceph-mon[44423]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:03:57.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:56 smithi171 ceph-mon[44189]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:03:59.141 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:58 smithi106 ceph-mon[44423]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:03:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:58 smithi005 ceph-mon[39562]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:03:59.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:58 smithi171 ceph-mon[44189]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:04:00.123 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:59 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:04:00.123 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:03:59 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:04:00.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:59 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:04:00.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:03:59 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:04:00.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:59 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:04:00.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:03:59 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:04:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:00 smithi005 ceph-mon[39562]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:04:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:00 smithi106 ceph-mon[44423]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:04:01.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:00 smithi171 ceph-mon[44189]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:04:02.133 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:01 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-20T17:04:02.133 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:01 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:04:02.133 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:01 smithi106 ceph-mon[44423]: Deploying daemon osd.7 on smithi106 2024-04-20T17:04:02.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:01 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-20T17:04:02.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:01 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:04:02.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:01 smithi005 ceph-mon[39562]: Deploying daemon osd.7 on smithi106 2024-04-20T17:04:02.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:01 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-20T17:04:02.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:01 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:04:02.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:01 smithi171 ceph-mon[44189]: Deploying daemon osd.7 on smithi106 2024-04-20T17:04:03.137 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:02 smithi106 ceph-mon[44423]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:04:03.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:02 smithi005 ceph-mon[39562]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:04:03.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:02 smithi171 ceph-mon[44189]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:04:05.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:04 smithi005 ceph-mon[39562]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:04:05.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:04 smithi106 ceph-mon[44423]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:04:05.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:04 smithi171 ceph-mon[44189]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:04:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:07 smithi106 ceph-mon[44423]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:04:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:07 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:07 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:07 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:04:08.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:07 smithi171 ceph-mon[44189]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:04:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:07 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:07 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:07 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:04:08.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:07 smithi005 ceph-mon[39562]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:04:08.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:07 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:08.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:07 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:08.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:07 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:04:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:09 smithi106 ceph-mon[44423]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:04:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:09 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:09 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:09 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:04:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:09 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:04:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:09 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:09 smithi171 ceph-mon[44189]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:04:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:09 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:09 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:09 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:04:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:09 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:04:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:09 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:10.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:09 smithi005 ceph-mon[39562]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:04:10.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:09 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:10.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:09 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:10.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:09 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:04:10.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:09 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:04:10.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:09 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:10.770 INFO:teuthology.orchestra.run.smithi106.stdout:Created osd(s) 7 on host 'smithi106' 2024-04-20T17:04:11.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:10 smithi106 ceph-mon[44423]: from='osd.7 v1:172.21.15.106:6812/1105371883' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-20T17:04:11.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:10 smithi106 ceph-mon[44423]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-20T17:04:11.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:10 smithi171 ceph-mon[44189]: from='osd.7 v1:172.21.15.106:6812/1105371883' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-20T17:04:11.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:10 smithi171 ceph-mon[44189]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-20T17:04:11.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:10 smithi005 ceph-mon[39562]: from='osd.7 v1:172.21.15.106:6812/1105371883' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-20T17:04:11.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:10 smithi005 ceph-mon[39562]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-20T17:04:11.433 DEBUG:teuthology.orchestra.run.smithi106:osd.7> sudo journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.7.service 2024-04-20T17:04:11.435 INFO:tasks.cephadm:Deploying osd.8 on smithi171 with /dev/vg_nvme/lv_4... 2024-04-20T17:04:11.436 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-20T17:04:11.466 INFO:journalctl@ceph.osd.7.smithi106.stdout:-- Logs begin at Sat 2024-04-20 16:47:44 UTC. -- 2024-04-20T17:04:11.706 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:04:12.009 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:11 smithi171 ceph-mon[44189]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:04:12.009 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:11 smithi171 ceph-mon[44189]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-20T17:04:12.009 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:11 smithi171 ceph-mon[44189]: osdmap e54: 8 total, 7 up, 8 in 2024-04-20T17:04:12.009 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:11 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-20T17:04:12.009 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:11 smithi171 ceph-mon[44189]: from='osd.7 v1:172.21.15.106:6812/1105371883' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:04:12.009 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:11 smithi171 ceph-mon[44189]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:04:12.009 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:11 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:12.009 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:11 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:12.009 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:11 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:04:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:11 smithi106 ceph-mon[44423]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:04:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:11 smithi106 ceph-mon[44423]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-20T17:04:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:11 smithi106 ceph-mon[44423]: osdmap e54: 8 total, 7 up, 8 in 2024-04-20T17:04:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:11 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-20T17:04:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:11 smithi106 ceph-mon[44423]: from='osd.7 v1:172.21.15.106:6812/1105371883' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:04:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:11 smithi106 ceph-mon[44423]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:04:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:11 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:11 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:11 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:04:12.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:11 smithi005 ceph-mon[39562]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:04:12.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:11 smithi005 ceph-mon[39562]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-20T17:04:12.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:11 smithi005 ceph-mon[39562]: osdmap e54: 8 total, 7 up, 8 in 2024-04-20T17:04:12.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:11 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-20T17:04:12.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:11 smithi005 ceph-mon[39562]: from='osd.7 v1:172.21.15.106:6812/1105371883' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:04:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:11 smithi005 ceph-mon[39562]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:04:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:11 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:11 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:11 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:04:12.511 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:04:12 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[73608]: 2024-04-20T17:04:12.184+0000 7fc50179a700 -1 osd.7 0 waiting for initial osdmap 2024-04-20T17:04:12.511 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:04:12 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[73608]: 2024-04-20T17:04:12.193+0000 7fc4f9585700 -1 osd.7 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T17:04:12.845 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:12 smithi106 ceph-mon[44423]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-04-20T17:04:12.845 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:12 smithi106 ceph-mon[44423]: osdmap e55: 8 total, 7 up, 8 in 2024-04-20T17:04:12.845 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-20T17:04:12.845 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-20T17:04:12.845 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:12 smithi106 ceph-mon[44423]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:04:12.846 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:12 smithi106 ceph-mon[44423]: from='osd.7 ' entity='osd.7' 2024-04-20T17:04:12.846 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:12 smithi106 ceph-mon[44423]: osd.7 v1:172.21.15.106:6812/1105371883 boot 2024-04-20T17:04:12.846 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:12 smithi106 ceph-mon[44423]: osdmap e56: 8 total, 8 up, 8 in 2024-04-20T17:04:12.846 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-20T17:04:13.034 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:12 smithi171 ceph-mon[44189]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-04-20T17:04:13.035 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:12 smithi171 ceph-mon[44189]: osdmap e55: 8 total, 7 up, 8 in 2024-04-20T17:04:13.035 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-20T17:04:13.035 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-20T17:04:13.035 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:12 smithi171 ceph-mon[44189]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:04:13.035 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:12 smithi171 ceph-mon[44189]: from='osd.7 ' entity='osd.7' 2024-04-20T17:04:13.035 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:12 smithi171 ceph-mon[44189]: osd.7 v1:172.21.15.106:6812/1105371883 boot 2024-04-20T17:04:13.035 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:12 smithi171 ceph-mon[44189]: osdmap e56: 8 total, 8 up, 8 in 2024-04-20T17:04:13.035 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-20T17:04:13.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:12 smithi005 ceph-mon[39562]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-04-20T17:04:13.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:12 smithi005 ceph-mon[39562]: osdmap e55: 8 total, 7 up, 8 in 2024-04-20T17:04:13.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-20T17:04:13.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-20T17:04:13.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:12 smithi005 ceph-mon[39562]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-20T17:04:13.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:12 smithi005 ceph-mon[39562]: from='osd.7 ' entity='osd.7' 2024-04-20T17:04:13.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:12 smithi005 ceph-mon[39562]: osd.7 v1:172.21.15.106:6812/1105371883 boot 2024-04-20T17:04:13.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:12 smithi005 ceph-mon[39562]: osdmap e56: 8 total, 8 up, 8 in 2024-04-20T17:04:13.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-20T17:04:13.955 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-20T17:04:13.955 INFO:teuthology.orchestra.run.smithi171.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-20T17:04:13.955 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10+0 records in 2024-04-20T17:04:13.955 INFO:teuthology.orchestra.run.smithi171.stderr:10+0 records out 2024-04-20T17:04:13.955 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111516 s, 940 MB/s 2024-04-20T17:04:13.956 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping successful for: 2024-04-20T17:04:14.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:13 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:04:14.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:13 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:04:14.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:13 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:04:14.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:13 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:04:14.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:13 smithi106 ceph-mon[44423]: osdmap e57: 8 total, 8 up, 8 in 2024-04-20T17:04:14.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:13 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:04:14.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:13 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:04:14.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:13 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:04:14.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:13 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:04:14.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:13 smithi171 ceph-mon[44189]: osdmap e57: 8 total, 8 up, 8 in 2024-04-20T17:04:14.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:13 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:04:14.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:13 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:04:14.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:13 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:04:14.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:13 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:04:14.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:13 smithi005 ceph-mon[39562]: osdmap e57: 8 total, 8 up, 8 in 2024-04-20T17:04:14.600 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi171:vg_nvme/lv_4 2024-04-20T17:04:14.862 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:04:15.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:15 smithi005 ceph-mon[39562]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:15.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:15 smithi005 ceph-mon[39562]: Detected new or changed devices on smithi106 2024-04-20T17:04:15.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:15 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:15.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:15 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:15.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:15 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:04:15.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:15 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:04:15.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:15 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:04:15.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:15 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:04:15.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:15 smithi005 ceph-mon[39562]: Adjusting osd_memory_target on smithi106 to 4280M 2024-04-20T17:04:15.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:15 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:15.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:15 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:04:15.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:15 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:04:15.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:15 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:15.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:15 smithi005 ceph-mon[39562]: osdmap e58: 8 total, 8 up, 8 in 2024-04-20T17:04:15.996 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:15 smithi171 ceph-mon[44189]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:15.997 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:15 smithi171 ceph-mon[44189]: Detected new or changed devices on smithi106 2024-04-20T17:04:15.997 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:15 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:15.997 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:15 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:15.997 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:15 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:04:15.997 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:15 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:04:15.997 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:15 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:04:15.997 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:15 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:04:15.997 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:15 smithi171 ceph-mon[44189]: Adjusting osd_memory_target on smithi106 to 4280M 2024-04-20T17:04:15.997 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:15 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:15.997 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:15 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:04:15.997 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:15 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:04:15.997 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:15 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:15.997 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:15 smithi171 ceph-mon[44189]: osdmap e58: 8 total, 8 up, 8 in 2024-04-20T17:04:16.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:15 smithi106 ceph-mon[44423]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:15 smithi106 ceph-mon[44423]: Detected new or changed devices on smithi106 2024-04-20T17:04:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:15 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:15 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:15 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:04:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:15 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:04:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:15 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:04:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:15 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:04:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:15 smithi106 ceph-mon[44423]: Adjusting osd_memory_target on smithi106 to 4280M 2024-04-20T17:04:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:15 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:15 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:04:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:15 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:04:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:15 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:15 smithi106 ceph-mon[44423]: osdmap e58: 8 total, 8 up, 8 in 2024-04-20T17:04:17.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:16 smithi106 ceph-mon[44423]: osdmap e59: 8 total, 8 up, 8 in 2024-04-20T17:04:17.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:16 smithi171 ceph-mon[44189]: osdmap e59: 8 total, 8 up, 8 in 2024-04-20T17:04:17.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:16 smithi005 ceph-mon[39562]: osdmap e59: 8 total, 8 up, 8 in 2024-04-20T17:04:17.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:17 smithi171 ceph-mon[44189]: pgmap v194: 1 pgs: 1 remapped+peering; 577 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-20T17:04:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:17 smithi171 ceph-mon[44189]: osdmap e60: 8 total, 8 up, 8 in 2024-04-20T17:04:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:17 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:04:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:17 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:04:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:17 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:04:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:17 smithi106 ceph-mon[44423]: pgmap v194: 1 pgs: 1 remapped+peering; 577 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-20T17:04:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:17 smithi106 ceph-mon[44423]: osdmap e60: 8 total, 8 up, 8 in 2024-04-20T17:04:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:17 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:04:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:17 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:04:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:17 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:04:18.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:17 smithi005 ceph-mon[39562]: pgmap v194: 1 pgs: 1 remapped+peering; 577 KiB data, 623 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-20T17:04:18.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:17 smithi005 ceph-mon[39562]: osdmap e60: 8 total, 8 up, 8 in 2024-04-20T17:04:18.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:17 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:04:18.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:17 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:04:18.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:17 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:04:19.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:18 smithi106 ceph-mon[44423]: from='client.24337 v1:172.21.15.171:0/3584303138' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:04:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:18 smithi171 ceph-mon[44189]: from='client.24337 v1:172.21.15.171:0/3584303138' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:04:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:18 smithi005 ceph-mon[39562]: from='client.24337 v1:172.21.15.171:0/3584303138' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:04:20.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:19 smithi106 ceph-mon[44423]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-04-20T17:04:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:19 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:04:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:19 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:04:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:19 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.171:0/3162159460' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1cb232ad-1f59-4a0a-a215-e67419ad6476"}]: dispatch 2024-04-20T17:04:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:19 smithi106 ceph-mon[44423]: from='client.24343 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1cb232ad-1f59-4a0a-a215-e67419ad6476"}]: dispatch 2024-04-20T17:04:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:19 smithi106 ceph-mon[44423]: from='client.24343 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1cb232ad-1f59-4a0a-a215-e67419ad6476"}]': finished 2024-04-20T17:04:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:19 smithi106 ceph-mon[44423]: osdmap e61: 9 total, 8 up, 9 in 2024-04-20T17:04:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:19 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-20T17:04:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:19 smithi171 ceph-mon[44189]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-04-20T17:04:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:19 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:04:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:19 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:04:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:19 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.171:0/3162159460' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1cb232ad-1f59-4a0a-a215-e67419ad6476"}]: dispatch 2024-04-20T17:04:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:19 smithi171 ceph-mon[44189]: from='client.24343 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1cb232ad-1f59-4a0a-a215-e67419ad6476"}]: dispatch 2024-04-20T17:04:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:19 smithi171 ceph-mon[44189]: from='client.24343 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1cb232ad-1f59-4a0a-a215-e67419ad6476"}]': finished 2024-04-20T17:04:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:19 smithi171 ceph-mon[44189]: osdmap e61: 9 total, 8 up, 9 in 2024-04-20T17:04:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:19 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-20T17:04:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:19 smithi005 ceph-mon[39562]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-04-20T17:04:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:19 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:04:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:19 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:04:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:19 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.171:0/3162159460' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1cb232ad-1f59-4a0a-a215-e67419ad6476"}]: dispatch 2024-04-20T17:04:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:19 smithi005 ceph-mon[39562]: from='client.24343 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1cb232ad-1f59-4a0a-a215-e67419ad6476"}]: dispatch 2024-04-20T17:04:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:19 smithi005 ceph-mon[39562]: from='client.24343 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1cb232ad-1f59-4a0a-a215-e67419ad6476"}]': finished 2024-04-20T17:04:20.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:19 smithi005 ceph-mon[39562]: osdmap e61: 9 total, 8 up, 9 in 2024-04-20T17:04:20.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:19 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-20T17:04:21.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:20 smithi106 ceph-mon[44423]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:20 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.171:0/3765189698' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:04:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:20 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:04:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:20 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:04:21.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:20 smithi171 ceph-mon[44189]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:21.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:20 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.171:0/3765189698' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:04:21.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:20 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:04:21.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:20 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:04:21.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:20 smithi005 ceph-mon[39562]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:21.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:20 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.171:0/3765189698' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:04:21.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:20 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:04:21.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:20 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:04:22.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:21 smithi106 ceph-mon[44423]: osdmap e62: 9 total, 8 up, 9 in 2024-04-20T17:04:22.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:21 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-20T17:04:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:21 smithi171 ceph-mon[44189]: osdmap e62: 9 total, 8 up, 9 in 2024-04-20T17:04:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:21 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-20T17:04:22.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:21 smithi005 ceph-mon[39562]: osdmap e62: 9 total, 8 up, 9 in 2024-04-20T17:04:22.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:21 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-20T17:04:23.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:22 smithi106 ceph-mon[44423]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:23.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:22 smithi171 ceph-mon[44189]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:23.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:22 smithi005 ceph-mon[39562]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:24 smithi171 ceph-mon[44189]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:25.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:24 smithi005 ceph-mon[39562]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:25.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:24 smithi106 ceph-mon[44423]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:25 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:04:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:25 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:04:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:25 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-20T17:04:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:25 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:04:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:25 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:04:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:25 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:04:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:25 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-20T17:04:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:25 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:04:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:25 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:04:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:25 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:04:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:25 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-20T17:04:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:25 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:04:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:26 smithi171 ceph-mon[44189]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:26 smithi171 ceph-mon[44189]: Deploying daemon osd.8 on smithi171 2024-04-20T17:04:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:26 smithi171 ceph-mon[44189]: osdmap e63: 9 total, 8 up, 9 in 2024-04-20T17:04:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:26 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-20T17:04:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:26 smithi005 ceph-mon[39562]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:26 smithi005 ceph-mon[39562]: Deploying daemon osd.8 on smithi171 2024-04-20T17:04:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:26 smithi005 ceph-mon[39562]: osdmap e63: 9 total, 8 up, 9 in 2024-04-20T17:04:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:26 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-20T17:04:27.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:26 smithi106 ceph-mon[44423]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:26 smithi106 ceph-mon[44423]: Deploying daemon osd.8 on smithi171 2024-04-20T17:04:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:26 smithi106 ceph-mon[44423]: osdmap e63: 9 total, 8 up, 9 in 2024-04-20T17:04:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:26 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-20T17:04:29.182 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:28 smithi171 ceph-mon[44189]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:29.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:28 smithi005 ceph-mon[39562]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:29.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:28 smithi106 ceph-mon[44423]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:31.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:30 smithi005 ceph-mon[39562]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:31.240 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:30 smithi171 ceph-mon[44189]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:31.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:30 smithi106 ceph-mon[44423]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:32.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:32.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:32.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:04:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:04:33.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:33.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:33.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:04:33.821 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:33 smithi171 ceph-mon[44189]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:33.821 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:33 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:04:33.821 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:33 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:04:33.821 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:33 smithi171 ceph-mon[44189]: osdmap e64: 9 total, 8 up, 9 in 2024-04-20T17:04:33.821 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:33 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-20T17:04:33.821 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:33 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:33.821 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:33 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:33.821 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:33 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:04:33.821 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:33 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:04:33.821 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:33 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:33.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:33 smithi005 ceph-mon[39562]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:33.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:33 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:04:33.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:33 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:04:33.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:33 smithi005 ceph-mon[39562]: osdmap e64: 9 total, 8 up, 9 in 2024-04-20T17:04:33.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:33 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-20T17:04:33.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:33 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:33.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:33 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:33.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:33 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:04:33.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:33 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:04:33.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:33 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:34.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:33 smithi106 ceph-mon[44423]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:33 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:04:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:33 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:04:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:33 smithi106 ceph-mon[44423]: osdmap e64: 9 total, 8 up, 9 in 2024-04-20T17:04:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:33 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-20T17:04:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:33 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:33 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:33 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:04:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:33 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:04:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:33 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:35.482 INFO:teuthology.orchestra.run.smithi171.stdout:Created osd(s) 8 on host 'smithi171' 2024-04-20T17:04:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:35 smithi171 ceph-mon[44189]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:35 smithi171 ceph-mon[44189]: from='osd.8 v1:172.21.15.171:6800/4123058942' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-20T17:04:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:35 smithi171 ceph-mon[44189]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-20T17:04:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:35 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:35 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:35 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:04:35.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:35 smithi005 ceph-mon[39562]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:35.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:35 smithi005 ceph-mon[39562]: from='osd.8 v1:172.21.15.171:6800/4123058942' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-20T17:04:35.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:35 smithi005 ceph-mon[39562]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-20T17:04:35.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:35 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:35.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:35 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:35.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:35 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:04:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:35 smithi106 ceph-mon[44423]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:35 smithi106 ceph-mon[44423]: from='osd.8 v1:172.21.15.171:6800/4123058942' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-20T17:04:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:35 smithi106 ceph-mon[44423]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-20T17:04:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:35 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:35 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:36.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:35 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:04:36.161 DEBUG:teuthology.orchestra.run.smithi171:osd.8> sudo journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.8.service 2024-04-20T17:04:36.163 INFO:tasks.cephadm:Deploying osd.9 on smithi171 with /dev/vg_nvme/lv_3... 2024-04-20T17:04:36.163 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-20T17:04:36.195 INFO:journalctl@ceph.osd.8.smithi171.stdout:-- Logs begin at Sat 2024-04-20 16:47:48 UTC. -- 2024-04-20T17:04:36.732 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:36 smithi171 ceph-mon[44189]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-20T17:04:36.733 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:36 smithi171 ceph-mon[44189]: osdmap e65: 9 total, 8 up, 9 in 2024-04-20T17:04:36.733 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:36 smithi171 ceph-mon[44189]: from='osd.8 v1:172.21.15.171:6800/4123058942' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-20T17:04:36.733 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:36 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-20T17:04:36.733 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:36 smithi171 ceph-mon[44189]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-20T17:04:36.733 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:36 smithi171 ceph-mon[44189]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-04-20T17:04:36.733 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:04:36 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:04:36.570+0000 7f72bcdae700 -1 osd.8 0 waiting for initial osdmap 2024-04-20T17:04:36.733 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:04:36 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:04:36.572+0000 7f72b452c700 -1 osd.8 0 failed to load OSD map for epoch 10, got 0 bytes 2024-04-20T17:04:36.733 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:04:36 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:04:36.581+0000 7f72b8ba1700 -1 osd.8 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T17:04:36.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:36 smithi005 ceph-mon[39562]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-20T17:04:36.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:36 smithi005 ceph-mon[39562]: osdmap e65: 9 total, 8 up, 9 in 2024-04-20T17:04:36.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:36 smithi005 ceph-mon[39562]: from='osd.8 v1:172.21.15.171:6800/4123058942' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-20T17:04:36.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:36 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-20T17:04:36.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:36 smithi005 ceph-mon[39562]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-20T17:04:36.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:36 smithi005 ceph-mon[39562]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-04-20T17:04:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:36 smithi106 ceph-mon[44423]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-20T17:04:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:36 smithi106 ceph-mon[44423]: osdmap e65: 9 total, 8 up, 9 in 2024-04-20T17:04:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:36 smithi106 ceph-mon[44423]: from='osd.8 v1:172.21.15.171:6800/4123058942' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-20T17:04:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:36 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-20T17:04:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:36 smithi106 ceph-mon[44423]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-20T17:04:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:36 smithi106 ceph-mon[44423]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-04-20T17:04:37.437 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:04:37.735 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:37 smithi171 ceph-mon[44189]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:37.735 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:37 smithi171 ceph-mon[44189]: osdmap e66: 9 total, 8 up, 9 in 2024-04-20T17:04:37.735 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:37 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-20T17:04:37.735 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:37 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-20T17:04:37.735 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:37 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-20T17:04:37.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:37 smithi005 ceph-mon[39562]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:37.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:37 smithi005 ceph-mon[39562]: osdmap e66: 9 total, 8 up, 9 in 2024-04-20T17:04:37.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:37 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-20T17:04:37.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:37 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-20T17:04:37.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:37 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-20T17:04:38.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:37 smithi106 ceph-mon[44423]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:37 smithi106 ceph-mon[44423]: osdmap e66: 9 total, 8 up, 9 in 2024-04-20T17:04:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:37 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-20T17:04:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:37 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-20T17:04:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:37 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-20T17:04:38.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:38 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:04:38.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:38 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:04:38.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:04:38.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:38 smithi005 ceph-mon[39562]: osd.8 v1:172.21.15.171:6800/4123058942 boot 2024-04-20T17:04:38.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:38 smithi005 ceph-mon[39562]: osdmap e67: 9 total, 9 up, 9 in 2024-04-20T17:04:38.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-20T17:04:39.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:38 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:04:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:38 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:04:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:04:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:38 smithi106 ceph-mon[44423]: osd.8 v1:172.21.15.171:6800/4123058942 boot 2024-04-20T17:04:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:38 smithi106 ceph-mon[44423]: osdmap e67: 9 total, 9 up, 9 in 2024-04-20T17:04:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-20T17:04:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:38 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:04:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:38 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:04:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:04:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:38 smithi171 ceph-mon[44189]: osd.8 v1:172.21.15.171:6800/4123058942 boot 2024-04-20T17:04:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:38 smithi171 ceph-mon[44189]: osdmap e67: 9 total, 9 up, 9 in 2024-04-20T17:04:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-20T17:04:39.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:39 smithi171 ceph-mon[44189]: pgmap v213: 1 pgs: 1 peering; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:39.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:39 smithi171 ceph-mon[44189]: osdmap e68: 9 total, 9 up, 9 in 2024-04-20T17:04:39.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:39 smithi005 ceph-mon[39562]: pgmap v213: 1 pgs: 1 peering; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:39.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:39 smithi005 ceph-mon[39562]: osdmap e68: 9 total, 9 up, 9 in 2024-04-20T17:04:40.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:39 smithi106 ceph-mon[44423]: pgmap v213: 1 pgs: 1 peering; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-20T17:04:40.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:39 smithi106 ceph-mon[44423]: osdmap e68: 9 total, 9 up, 9 in 2024-04-20T17:04:40.101 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-20T17:04:40.101 INFO:teuthology.orchestra.run.smithi171.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-20T17:04:40.101 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10+0 records in 2024-04-20T17:04:40.101 INFO:teuthology.orchestra.run.smithi171.stderr:10+0 records out 2024-04-20T17:04:40.101 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111502 s, 940 MB/s 2024-04-20T17:04:40.102 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping successful for: 2024-04-20T17:04:41.051 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi171:vg_nvme/lv_3 2024-04-20T17:04:41.310 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:04:41.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:41 smithi005 ceph-mon[39562]: pgmap v215: 1 pgs: 1 peering; 577 KiB data, 250 MiB used, 804 GiB / 805 GiB avail 2024-04-20T17:04:41.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:41 smithi005 ceph-mon[39562]: Detected new or changed devices on smithi171 2024-04-20T17:04:41.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:41 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:41.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:41 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:41.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:41 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:04:41.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:41 smithi005 ceph-mon[39562]: Adjusting osd_memory_target on smithi171 to 21218M 2024-04-20T17:04:41.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:41 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:41.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:41 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:04:41.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:41 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:04:41.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:41 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:41 smithi106 ceph-mon[44423]: pgmap v215: 1 pgs: 1 peering; 577 KiB data, 250 MiB used, 804 GiB / 805 GiB avail 2024-04-20T17:04:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:41 smithi106 ceph-mon[44423]: Detected new or changed devices on smithi171 2024-04-20T17:04:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:41 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:41 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:41 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:04:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:41 smithi106 ceph-mon[44423]: Adjusting osd_memory_target on smithi171 to 21218M 2024-04-20T17:04:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:41 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:41 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:04:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:41 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:04:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:41 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:41.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:41 smithi171 ceph-mon[44189]: pgmap v215: 1 pgs: 1 peering; 577 KiB data, 250 MiB used, 804 GiB / 805 GiB avail 2024-04-20T17:04:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:41 smithi171 ceph-mon[44189]: Detected new or changed devices on smithi171 2024-04-20T17:04:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:41 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:41 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:41 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:04:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:41 smithi171 ceph-mon[44189]: Adjusting osd_memory_target on smithi171 to 21218M 2024-04-20T17:04:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:41 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:41 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:04:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:41 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:04:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:41 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:04:43.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:43 smithi171 ceph-mon[44189]: pgmap v216: 1 pgs: 1 peering; 577 KiB data, 250 MiB used, 804 GiB / 805 GiB avail 2024-04-20T17:04:43.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:43 smithi005 ceph-mon[39562]: pgmap v216: 1 pgs: 1 peering; 577 KiB data, 250 MiB used, 804 GiB / 805 GiB avail 2024-04-20T17:04:43.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:43 smithi106 ceph-mon[44423]: pgmap v216: 1 pgs: 1 peering; 577 KiB data, 250 MiB used, 804 GiB / 805 GiB avail 2024-04-20T17:04:44.490 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:44 smithi171 ceph-mon[44189]: from='client.24364 v1:172.21.15.171:0/470258774' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:04:44.490 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:04:44.490 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:04:44.490 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:04:44.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:44 smithi005 ceph-mon[39562]: from='client.24364 v1:172.21.15.171:0/470258774' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:04:44.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:04:44.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:04:44.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:04:44.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:44 smithi106 ceph-mon[44423]: from='client.24364 v1:172.21.15.171:0/470258774' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:04:44.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:04:44.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:04:44.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:04:45.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:45 smithi171 ceph-mon[44189]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-20T17:04:45.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:45 smithi005 ceph-mon[39562]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-20T17:04:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:45 smithi106 ceph-mon[44423]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-20T17:04:47.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:46 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.171:0/84578109' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6c17147-b3dc-475b-bb20-f9f712fb99e8"}]: dispatch 2024-04-20T17:04:47.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:46 smithi106 ceph-mon[44423]: from='client.24370 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6c17147-b3dc-475b-bb20-f9f712fb99e8"}]: dispatch 2024-04-20T17:04:47.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:46 smithi106 ceph-mon[44423]: from='client.24370 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b6c17147-b3dc-475b-bb20-f9f712fb99e8"}]': finished 2024-04-20T17:04:47.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:46 smithi106 ceph-mon[44423]: osdmap e69: 10 total, 9 up, 10 in 2024-04-20T17:04:47.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:46 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-20T17:04:47.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:46 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.171:0/84578109' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6c17147-b3dc-475b-bb20-f9f712fb99e8"}]: dispatch 2024-04-20T17:04:47.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:46 smithi171 ceph-mon[44189]: from='client.24370 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6c17147-b3dc-475b-bb20-f9f712fb99e8"}]: dispatch 2024-04-20T17:04:47.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:46 smithi171 ceph-mon[44189]: from='client.24370 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b6c17147-b3dc-475b-bb20-f9f712fb99e8"}]': finished 2024-04-20T17:04:47.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:46 smithi171 ceph-mon[44189]: osdmap e69: 10 total, 9 up, 10 in 2024-04-20T17:04:47.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:46 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-20T17:04:47.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:46 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.171:0/84578109' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6c17147-b3dc-475b-bb20-f9f712fb99e8"}]: dispatch 2024-04-20T17:04:47.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:46 smithi005 ceph-mon[39562]: from='client.24370 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6c17147-b3dc-475b-bb20-f9f712fb99e8"}]: dispatch 2024-04-20T17:04:47.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:46 smithi005 ceph-mon[39562]: from='client.24370 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b6c17147-b3dc-475b-bb20-f9f712fb99e8"}]': finished 2024-04-20T17:04:47.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:46 smithi005 ceph-mon[39562]: osdmap e69: 10 total, 9 up, 10 in 2024-04-20T17:04:47.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:46 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-20T17:04:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:47 smithi106 ceph-mon[44423]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-20T17:04:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:47 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.171:0/404059735' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:04:48.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:47 smithi171 ceph-mon[44189]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-20T17:04:48.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:47 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.171:0/404059735' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:04:48.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:47 smithi005 ceph-mon[39562]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-20T17:04:48.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:47 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.171:0/404059735' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:04:49.993 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:49 smithi171 ceph-mon[44189]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-20T17:04:50.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:49 smithi106 ceph-mon[44423]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-20T17:04:50.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:49 smithi005 ceph-mon[39562]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-20T17:04:51.956 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:51 smithi171 ceph-mon[44189]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-20T17:04:52.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:51 smithi106 ceph-mon[44423]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-20T17:04:52.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:51 smithi005 ceph-mon[39562]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-20T17:04:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:52 smithi171 ceph-mon[44189]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-20T17:04:53.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:52 smithi005 ceph-mon[39562]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-20T17:04:53.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:52 smithi106 ceph-mon[44423]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-20T17:04:54.069 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:53 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-20T17:04:54.070 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:53 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:04:54.070 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:53 smithi171 ceph-mon[44189]: Deploying daemon osd.9 on smithi171 2024-04-20T17:04:54.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:53 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-20T17:04:54.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:53 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:04:54.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:53 smithi005 ceph-mon[39562]: Deploying daemon osd.9 on smithi171 2024-04-20T17:04:54.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:53 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-20T17:04:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:53 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:04:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:53 smithi106 ceph-mon[44423]: Deploying daemon osd.9 on smithi171 2024-04-20T17:04:55.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:54 smithi171 ceph-mon[44189]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-20T17:04:55.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:54 smithi005 ceph-mon[39562]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-20T17:04:55.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:54 smithi106 ceph-mon[44423]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-20T17:04:57.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:56 smithi005 ceph-mon[39562]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-20T17:04:57.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:56 smithi106 ceph-mon[44423]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-20T17:04:57.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:56 smithi171 ceph-mon[44189]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-20T17:04:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:58 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:04:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:58 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:04:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:58 smithi005 ceph-mon[39562]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-20T17:04:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:58 smithi005 ceph-mon[39562]: osdmap e70: 10 total, 9 up, 10 in 2024-04-20T17:04:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:58 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-20T17:04:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:58 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:04:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:58 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:04:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:58 smithi106 ceph-mon[44423]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-20T17:04:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:58 smithi106 ceph-mon[44423]: osdmap e70: 10 total, 9 up, 10 in 2024-04-20T17:04:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:58 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-20T17:04:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:58 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:04:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:58 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:04:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:58 smithi171 ceph-mon[44189]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-20T17:04:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:58 smithi171 ceph-mon[44189]: osdmap e70: 10 total, 9 up, 10 in 2024-04-20T17:04:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:58 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-20T17:05:00.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:59 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:00.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:59 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:00.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:04:59 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:05:00.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:59 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:00.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:59 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:00.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:04:59 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:05:00.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:59 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:00.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:59 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:00.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:04:59 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:05:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:01 smithi171 ceph-mon[44189]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-20T17:05:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:01 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:01 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:01 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:05:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:01 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:05:01.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:01 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:01.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:01 smithi005 ceph-mon[39562]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-20T17:05:01.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:01 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:01.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:01 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:01.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:01 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:05:01.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:01 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:05:01.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:01 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:01 smithi106 ceph-mon[44423]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-20T17:05:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:01 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:01 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:01 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:05:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:01 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:05:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:01 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:02.819 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:02 smithi171 ceph-mon[44189]: from='osd.9 v1:172.21.15.171:6804/2877589636' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-20T17:05:02.819 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:02 smithi171 ceph-mon[44189]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-20T17:05:02.905 INFO:teuthology.orchestra.run.smithi171.stdout:Created osd(s) 9 on host 'smithi171' 2024-04-20T17:05:02.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:02 smithi005 ceph-mon[39562]: from='osd.9 v1:172.21.15.171:6804/2877589636' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-20T17:05:02.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:02 smithi005 ceph-mon[39562]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-20T17:05:03.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:02 smithi106 ceph-mon[44423]: from='osd.9 v1:172.21.15.171:6804/2877589636' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-20T17:05:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:02 smithi106 ceph-mon[44423]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-20T17:05:03.753 DEBUG:teuthology.orchestra.run.smithi171:osd.9> sudo journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.9.service 2024-04-20T17:05:03.755 INFO:tasks.cephadm:Deploying osd.10 on smithi171 with /dev/vg_nvme/lv_2... 2024-04-20T17:05:03.755 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-20T17:05:03.785 INFO:journalctl@ceph.osd.9.smithi171.stdout:-- Logs begin at Sat 2024-04-20 16:47:48 UTC. -- 2024-04-20T17:05:03.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:03 smithi171 ceph-mon[44189]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-20T17:05:03.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:03 smithi171 ceph-mon[44189]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-20T17:05:03.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:03 smithi171 ceph-mon[44189]: osdmap e71: 10 total, 9 up, 10 in 2024-04-20T17:05:03.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:03 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-20T17:05:03.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:03 smithi171 ceph-mon[44189]: from='osd.9 v1:172.21.15.171:6804/2877589636' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-20T17:05:03.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:03 smithi171 ceph-mon[44189]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-20T17:05:03.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:03 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:03.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:03 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:03.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:03 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:05:03.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:03 smithi005 ceph-mon[39562]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-20T17:05:03.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:03 smithi005 ceph-mon[39562]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-20T17:05:03.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:03 smithi005 ceph-mon[39562]: osdmap e71: 10 total, 9 up, 10 in 2024-04-20T17:05:03.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:03 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-20T17:05:03.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:03 smithi005 ceph-mon[39562]: from='osd.9 v1:172.21.15.171:6804/2877589636' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-20T17:05:03.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:03 smithi005 ceph-mon[39562]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-20T17:05:03.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:03 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:03.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:03 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:03.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:03 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:05:04.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:03 smithi106 ceph-mon[44423]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-20T17:05:04.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:03 smithi106 ceph-mon[44423]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-20T17:05:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:03 smithi106 ceph-mon[44423]: osdmap e71: 10 total, 9 up, 10 in 2024-04-20T17:05:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:03 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-20T17:05:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:03 smithi106 ceph-mon[44423]: from='osd.9 v1:172.21.15.171:6804/2877589636' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-20T17:05:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:03 smithi106 ceph-mon[44423]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-20T17:05:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:03 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:03 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:03 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:05:04.084 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:05:04 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:05:04.004+0000 7fbb03c76700 -1 osd.9 0 waiting for initial osdmap 2024-04-20T17:05:04.084 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:05:04 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:05:04.004+0000 7fbaf73ec700 -1 osd.9 0 failed to load OSD map for epoch 10, got 0 bytes 2024-04-20T17:05:04.084 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:05:04 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:05:04.014+0000 7fbafba61700 -1 osd.9 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T17:05:04.529 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:05:04.819 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:04 smithi171 ceph-mon[44189]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-04-20T17:05:04.820 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:04 smithi171 ceph-mon[44189]: osdmap e72: 10 total, 9 up, 10 in 2024-04-20T17:05:04.820 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:04 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-20T17:05:04.820 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:04 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-20T17:05:04.820 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:04 smithi171 ceph-mon[44189]: from='osd.9 ' entity='osd.9' 2024-04-20T17:05:04.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:04 smithi005 ceph-mon[39562]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-04-20T17:05:04.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:04 smithi005 ceph-mon[39562]: osdmap e72: 10 total, 9 up, 10 in 2024-04-20T17:05:04.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:04 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-20T17:05:04.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:04 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-20T17:05:04.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:04 smithi005 ceph-mon[39562]: from='osd.9 ' entity='osd.9' 2024-04-20T17:05:05.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:04 smithi106 ceph-mon[44423]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-04-20T17:05:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:04 smithi106 ceph-mon[44423]: osdmap e72: 10 total, 9 up, 10 in 2024-04-20T17:05:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:04 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-20T17:05:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:04 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-20T17:05:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:04 smithi106 ceph-mon[44423]: from='osd.9 ' entity='osd.9' 2024-04-20T17:05:05.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:05 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:05:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:05 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:05:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:05:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:05 smithi171 ceph-mon[44189]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-20T17:05:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:05:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:05 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-20T17:05:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:05 smithi171 ceph-mon[44189]: osd.9 v1:172.21.15.171:6804/2877589636 boot 2024-04-20T17:05:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:05 smithi171 ceph-mon[44189]: osdmap e73: 10 total, 10 up, 10 in 2024-04-20T17:05:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:05 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-20T17:05:05.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:05 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:05:05.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:05 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:05:05.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:05:05.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:05 smithi005 ceph-mon[39562]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-20T17:05:05.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:05:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:05 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-20T17:05:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:05 smithi005 ceph-mon[39562]: osd.9 v1:172.21.15.171:6804/2877589636 boot 2024-04-20T17:05:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:05 smithi005 ceph-mon[39562]: osdmap e73: 10 total, 10 up, 10 in 2024-04-20T17:05:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:05 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-20T17:05:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:05 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:05:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:05 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:05:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:05:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:05 smithi106 ceph-mon[44423]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 251 MiB used, 804 GiB / 805 GiB avail 2024-04-20T17:05:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:05:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:05 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-20T17:05:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:05 smithi106 ceph-mon[44423]: osd.9 v1:172.21.15.171:6804/2877589636 boot 2024-04-20T17:05:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:05 smithi106 ceph-mon[44423]: osdmap e73: 10 total, 10 up, 10 in 2024-04-20T17:05:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:05 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-20T17:05:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:07 smithi171 ceph-mon[44189]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:07 smithi171 ceph-mon[44189]: osdmap e74: 10 total, 10 up, 10 in 2024-04-20T17:05:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:07 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:05:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:07 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:05:07.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:07 smithi005 ceph-mon[39562]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:07.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:07 smithi005 ceph-mon[39562]: osdmap e74: 10 total, 10 up, 10 in 2024-04-20T17:05:07.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:07 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:05:07.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:07 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:05:07.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:07 smithi106 ceph-mon[44423]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:07.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:07 smithi106 ceph-mon[44423]: osdmap e74: 10 total, 10 up, 10 in 2024-04-20T17:05:07.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:07 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:05:07.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:07 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:05:07.726 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-20T17:05:07.726 INFO:teuthology.orchestra.run.smithi171.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-20T17:05:07.726 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10+0 records in 2024-04-20T17:05:07.726 INFO:teuthology.orchestra.run.smithi171.stderr:10+0 records out 2024-04-20T17:05:07.726 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0138221 s, 759 MB/s 2024-04-20T17:05:07.726 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping successful for: 2024-04-20T17:05:08.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:08 smithi171 ceph-mon[44189]: osdmap e75: 10 total, 10 up, 10 in 2024-04-20T17:05:08.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:08 smithi005 ceph-mon[39562]: osdmap e75: 10 total, 10 up, 10 in 2024-04-20T17:05:08.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:08 smithi106 ceph-mon[44423]: osdmap e75: 10 total, 10 up, 10 in 2024-04-20T17:05:08.728 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi171:vg_nvme/lv_2 2024-04-20T17:05:09.167 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:05:09.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:09 smithi171 ceph-mon[44189]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:09.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:09 smithi005 ceph-mon[39562]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:09.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:09 smithi106 ceph-mon[44423]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:10 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:05:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:10 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:05:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:10 smithi171 ceph-mon[44189]: osdmap e76: 10 total, 10 up, 10 in 2024-04-20T17:05:10.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:10 smithi171 ceph-mon[44189]: Detected new or changed devices on smithi171 2024-04-20T17:05:10.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:10 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:10.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:10 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:10.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:10 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:05:10.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:10 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:05:10.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:10 smithi171 ceph-mon[44189]: Adjusting osd_memory_target on smithi171 to 10609M 2024-04-20T17:05:10.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:10 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:10.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:10 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:05:10.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:10 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:05:10.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:10 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:10.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:10 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:05:10.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:10 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:05:10.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:10 smithi005 ceph-mon[39562]: osdmap e76: 10 total, 10 up, 10 in 2024-04-20T17:05:10.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:10 smithi005 ceph-mon[39562]: Detected new or changed devices on smithi171 2024-04-20T17:05:10.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:10 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:10.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:10 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:10.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:10 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:05:10.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:10 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:05:10.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:10 smithi005 ceph-mon[39562]: Adjusting osd_memory_target on smithi171 to 10609M 2024-04-20T17:05:10.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:10 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:10.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:10 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:05:10.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:10 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:05:10.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:10 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:10.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:10 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:05:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:10 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:05:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:10 smithi106 ceph-mon[44423]: osdmap e76: 10 total, 10 up, 10 in 2024-04-20T17:05:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:10 smithi106 ceph-mon[44423]: Detected new or changed devices on smithi171 2024-04-20T17:05:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:10 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:10 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:10 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:05:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:10 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:05:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:10 smithi106 ceph-mon[44423]: Adjusting osd_memory_target on smithi171 to 10609M 2024-04-20T17:05:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:10 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:10 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:05:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:10 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:05:10.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:10 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:11 smithi106 ceph-mon[44423]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:11 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:05:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:11 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:05:12.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:11 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:05:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:11 smithi171 ceph-mon[44189]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:11 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:05:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:11 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:05:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:11 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:05:12.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:11 smithi005 ceph-mon[39562]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:12.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:11 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:05:12.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:11 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:05:12.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:11 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:05:13.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:12 smithi106 ceph-mon[44423]: from='client.24391 v1:172.21.15.171:0/1239144287' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:05:13.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:12 smithi171 ceph-mon[44189]: from='client.24391 v1:172.21.15.171:0/1239144287' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:05:13.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:12 smithi005 ceph-mon[39562]: from='client.24391 v1:172.21.15.171:0/1239144287' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:05:14.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:13 smithi106 ceph-mon[44423]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:14.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:13 smithi171 ceph-mon[44189]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:14.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:13 smithi005 ceph-mon[39562]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:14 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.171:0/77147849' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1adcdf55-f002-4a39-91e7-378e5fae849c"}]: dispatch 2024-04-20T17:05:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:14 smithi106 ceph-mon[44423]: from='client.24397 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1adcdf55-f002-4a39-91e7-378e5fae849c"}]: dispatch 2024-04-20T17:05:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:14 smithi106 ceph-mon[44423]: from='client.24397 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1adcdf55-f002-4a39-91e7-378e5fae849c"}]': finished 2024-04-20T17:05:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:14 smithi106 ceph-mon[44423]: osdmap e77: 11 total, 10 up, 11 in 2024-04-20T17:05:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:14 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:14 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.171:0/77147849' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1adcdf55-f002-4a39-91e7-378e5fae849c"}]: dispatch 2024-04-20T17:05:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:14 smithi171 ceph-mon[44189]: from='client.24397 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1adcdf55-f002-4a39-91e7-378e5fae849c"}]: dispatch 2024-04-20T17:05:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:14 smithi171 ceph-mon[44189]: from='client.24397 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1adcdf55-f002-4a39-91e7-378e5fae849c"}]': finished 2024-04-20T17:05:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:14 smithi171 ceph-mon[44189]: osdmap e77: 11 total, 10 up, 11 in 2024-04-20T17:05:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:14 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:15.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:14 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.171:0/77147849' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1adcdf55-f002-4a39-91e7-378e5fae849c"}]: dispatch 2024-04-20T17:05:15.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:14 smithi005 ceph-mon[39562]: from='client.24397 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1adcdf55-f002-4a39-91e7-378e5fae849c"}]: dispatch 2024-04-20T17:05:15.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:14 smithi005 ceph-mon[39562]: from='client.24397 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1adcdf55-f002-4a39-91e7-378e5fae849c"}]': finished 2024-04-20T17:05:15.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:14 smithi005 ceph-mon[39562]: osdmap e77: 11 total, 10 up, 11 in 2024-04-20T17:05:15.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:14 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:16.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:15 smithi106 ceph-mon[44423]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:15 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.171:0/4060750451' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:05:16.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:15 smithi171 ceph-mon[44189]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:16.093 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:15 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.171:0/4060750451' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:05:16.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:15 smithi005 ceph-mon[39562]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:16.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:15 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.171:0/4060750451' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:05:17.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:16 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:05:17.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:16 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:05:17.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:16 smithi171 ceph-mon[44189]: osdmap e78: 11 total, 10 up, 11 in 2024-04-20T17:05:17.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:16 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:17.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:16 smithi171 ceph-mon[44189]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:17.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:16 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:05:17.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:16 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:05:17.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:16 smithi005 ceph-mon[39562]: osdmap e78: 11 total, 10 up, 11 in 2024-04-20T17:05:17.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:16 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:17.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:16 smithi005 ceph-mon[39562]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:17.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:16 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:05:17.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:16 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:05:17.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:16 smithi106 ceph-mon[44423]: osdmap e78: 11 total, 10 up, 11 in 2024-04-20T17:05:17.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:16 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:17.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:16 smithi106 ceph-mon[44423]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:18 smithi005 ceph-mon[39562]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:19.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:18 smithi106 ceph-mon[44423]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:19.262 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:18 smithi171 ceph-mon[44189]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:21.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:20 smithi005 ceph-mon[39562]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:21.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:20 smithi106 ceph-mon[44423]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:21.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:20 smithi171 ceph-mon[44189]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:21.998 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:21 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-20T17:05:21.998 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:21 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:05:21.999 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:21 smithi171 ceph-mon[44189]: Deploying daemon osd.10 on smithi171 2024-04-20T17:05:22.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:21 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-20T17:05:22.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:21 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:05:22.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:21 smithi005 ceph-mon[39562]: Deploying daemon osd.10 on smithi171 2024-04-20T17:05:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:21 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-20T17:05:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:21 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:05:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:21 smithi106 ceph-mon[44423]: Deploying daemon osd.10 on smithi171 2024-04-20T17:05:23.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:22 smithi171 ceph-mon[44189]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:23.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:22 smithi005 ceph-mon[39562]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:23.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:22 smithi106 ceph-mon[44423]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:23 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:05:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:23 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:05:24.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:23 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:05:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:23 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:05:24.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:23 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:05:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:23 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:05:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:24 smithi171 ceph-mon[44189]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:24 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:05:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:24 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:05:25.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:24 smithi005 ceph-mon[39562]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:25.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:24 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:05:25.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:24 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:05:25.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:24 smithi106 ceph-mon[44423]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:25.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:24 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:05:25.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:24 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:05:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:25 smithi005 ceph-mon[39562]: osdmap e79: 11 total, 10 up, 11 in 2024-04-20T17:05:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:25 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:26.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:25 smithi106 ceph-mon[44423]: osdmap e79: 11 total, 10 up, 11 in 2024-04-20T17:05:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:25 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:26.268 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:25 smithi171 ceph-mon[44189]: osdmap e79: 11 total, 10 up, 11 in 2024-04-20T17:05:26.268 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:25 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:27.926 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:27 smithi171 ceph-mon[44189]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:27.927 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:27 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:27.927 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:27 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:27.927 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:27 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:05:28.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:27 smithi106 ceph-mon[44423]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:27 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:27 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:27 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:05:28.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:27 smithi005 ceph-mon[39562]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:28.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:27 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:27 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:27 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:05:30.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:29 smithi106 ceph-mon[44423]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:30.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:29 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:05:30.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:29 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:05:30.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:29 smithi106 ceph-mon[44423]: osdmap e80: 11 total, 10 up, 11 in 2024-04-20T17:05:30.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:29 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:30.019 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:29 smithi171 ceph-mon[44189]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:30.019 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:29 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:05:30.020 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:29 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:05:30.020 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:29 smithi171 ceph-mon[44189]: osdmap e80: 11 total, 10 up, 11 in 2024-04-20T17:05:30.020 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:29 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:29 smithi005 ceph-mon[39562]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:29 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:05:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:29 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:05:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:29 smithi005 ceph-mon[39562]: osdmap e80: 11 total, 10 up, 11 in 2024-04-20T17:05:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:29 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:31.008 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:30 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:05:31.008 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:30 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:05:31.008 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:30 smithi171 ceph-mon[44189]: osdmap e81: 11 total, 10 up, 11 in 2024-04-20T17:05:31.009 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:30 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:31.009 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:30 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:31.009 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:30 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:31.009 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:30 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:05:31.009 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:30 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:05:31.009 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:30 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:31.009 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:30 smithi171 ceph-mon[44189]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:31.009 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:30 smithi171 ceph-mon[44189]: from='osd.10 v1:172.21.15.171:6808/181490621' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-20T17:05:31.009 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:30 smithi171 ceph-mon[44189]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-20T17:05:31.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:30 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:05:31.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:30 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:05:31.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:30 smithi005 ceph-mon[39562]: osdmap e81: 11 total, 10 up, 11 in 2024-04-20T17:05:31.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:30 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:30 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:30 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:30 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:05:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:30 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:05:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:30 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:30 smithi005 ceph-mon[39562]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:30 smithi005 ceph-mon[39562]: from='osd.10 v1:172.21.15.171:6808/181490621' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-20T17:05:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:30 smithi005 ceph-mon[39562]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-20T17:05:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:30 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:05:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:30 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:05:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:30 smithi106 ceph-mon[44423]: osdmap e81: 11 total, 10 up, 11 in 2024-04-20T17:05:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:30 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:30 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:30 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:30 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:05:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:30 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:05:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:30 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:30 smithi106 ceph-mon[44423]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:30 smithi106 ceph-mon[44423]: from='osd.10 v1:172.21.15.171:6808/181490621' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-20T17:05:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:30 smithi106 ceph-mon[44423]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-20T17:05:31.473 INFO:teuthology.orchestra.run.smithi171.stdout:Created osd(s) 10 on host 'smithi171' 2024-04-20T17:05:32.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-20T17:05:32.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: osdmap e82: 11 total, 10 up, 11 in 2024-04-20T17:05:32.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:32.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: from='osd.10 v1:172.21.15.171:6808/181490621' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-20T17:05:32.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-20T17:05:32.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:32.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:32.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:05:32.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {auth=22,config=2,health=12,kv=60,logm=4} crc {auth=3353605203,config=1548763834,health=3897446751,kv=829483090,logm=1635173414}) 2024-04-20T17:05:32.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2044810697}) 2024-04-20T17:05:32.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=936390506}) 2024-04-20T17:05:32.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3886421572}) 2024-04-20T17:05:32.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1600610170}) 2024-04-20T17:05:32.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2827672901}) 2024-04-20T17:05:32.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3672714428}) 2024-04-20T17:05:32.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2451702695}) 2024-04-20T17:05:32.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1871511657}) 2024-04-20T17:05:32.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1566442998}) 2024-04-20T17:05:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1889280795}) 2024-04-20T17:05:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1159749145}) 2024-04-20T17:05:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2727802300}) 2024-04-20T17:05:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3784486818}) 2024-04-20T17:05:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1205477104}) 2024-04-20T17:05:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=550784044}) 2024-04-20T17:05:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=33,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=32} crc {logm=397782869,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=2300735773,mon_config_key=3654313052}) 2024-04-20T17:05:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=860848980}) 2024-04-20T17:05:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1192077000}) 2024-04-20T17:05:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=40,monmap=5,osd_metadata=10,osd_pg_creating=1,osdmap=44} crc {mon_config_key=1781521675,monmap=717113318,osd_metadata=3749010413,osd_pg_creating=803862196,osdmap=693211358}) 2024-04-20T17:05:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=79} crc {osdmap=3797573902}) 2024-04-20T17:05:32.123 DEBUG:teuthology.orchestra.run.smithi171:osd.10> sudo journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.10.service 2024-04-20T17:05:32.125 INFO:tasks.cephadm:Deploying osd.11 on smithi171 with /dev/vg_nvme/lv_1... 2024-04-20T17:05:32.125 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-20T17:05:32.156 INFO:journalctl@ceph.osd.10.smithi171.stdout:-- Logs begin at Sat 2024-04-20 16:47:48 UTC. -- 2024-04-20T17:05:32.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-20T17:05:32.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: osdmap e82: 11 total, 10 up, 11 in 2024-04-20T17:05:32.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: from='osd.10 v1:172.21.15.171:6808/181490621' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-20T17:05:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-20T17:05:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:05:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {auth=22,config=2,health=12,kv=60,logm=4} crc {auth=3353605203,config=1548763834,health=3897446751,kv=829483090,logm=1635173414}) 2024-04-20T17:05:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2044810697}) 2024-04-20T17:05:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=936390506}) 2024-04-20T17:05:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3886421572}) 2024-04-20T17:05:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1600610170}) 2024-04-20T17:05:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2827672901}) 2024-04-20T17:05:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3672714428}) 2024-04-20T17:05:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2451702695}) 2024-04-20T17:05:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1871511657}) 2024-04-20T17:05:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1566442998}) 2024-04-20T17:05:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1889280795}) 2024-04-20T17:05:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1159749145}) 2024-04-20T17:05:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2727802300}) 2024-04-20T17:05:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3784486818}) 2024-04-20T17:05:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1205477104}) 2024-04-20T17:05:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=550784044}) 2024-04-20T17:05:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=33,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=32} crc {logm=397782869,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=2300735773,mon_config_key=3654313052}) 2024-04-20T17:05:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=860848980}) 2024-04-20T17:05:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1192077000}) 2024-04-20T17:05:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=40,monmap=5,osd_metadata=10,osd_pg_creating=1,osdmap=44} crc {mon_config_key=1781521675,monmap=717113318,osd_metadata=3749010413,osd_pg_creating=803862196,osdmap=693211358}) 2024-04-20T17:05:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=79} crc {osdmap=3797573902}) 2024-04-20T17:05:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-20T17:05:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: osdmap e82: 11 total, 10 up, 11 in 2024-04-20T17:05:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: from='osd.10 v1:172.21.15.171:6808/181490621' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-20T17:05:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-20T17:05:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:05:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {auth=22,config=2,health=12,kv=60,logm=4} crc {auth=3353605203,config=1548763834,health=3897446751,kv=829483090,logm=1635173414}) 2024-04-20T17:05:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2044810697}) 2024-04-20T17:05:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=936390506}) 2024-04-20T17:05:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3886421572}) 2024-04-20T17:05:32.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1600610170}) 2024-04-20T17:05:32.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2827672901}) 2024-04-20T17:05:32.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3672714428}) 2024-04-20T17:05:32.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2451702695}) 2024-04-20T17:05:32.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1871511657}) 2024-04-20T17:05:32.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1566442998}) 2024-04-20T17:05:32.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1889280795}) 2024-04-20T17:05:32.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1159749145}) 2024-04-20T17:05:32.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2727802300}) 2024-04-20T17:05:32.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3784486818}) 2024-04-20T17:05:32.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1205477104}) 2024-04-20T17:05:32.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=550784044}) 2024-04-20T17:05:32.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=33,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=32} crc {logm=397782869,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=2300735773,mon_config_key=3654313052}) 2024-04-20T17:05:32.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=860848980}) 2024-04-20T17:05:32.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1192077000}) 2024-04-20T17:05:32.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=40,monmap=5,osd_metadata=10,osd_pg_creating=1,osdmap=44} crc {mon_config_key=1781521675,monmap=717113318,osd_metadata=3749010413,osd_pg_creating=803862196,osdmap=693211358}) 2024-04-20T17:05:32.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=79} crc {osdmap=3797573902}) 2024-04-20T17:05:32.334 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:05:32 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:05:32.254+0000 7f5cd2089700 -1 osd.10 0 waiting for initial osdmap 2024-04-20T17:05:32.334 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:05:32 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:05:32.254+0000 7f5cc57ff700 -1 osd.10 0 failed to load OSD map for epoch 22, got 0 bytes 2024-04-20T17:05:32.334 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:05:32 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:05:32.265+0000 7f5cc9e74700 -1 osd.10 83 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T17:05:33.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:32 smithi171 ceph-mon[44189]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-04-20T17:05:33.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:32 smithi171 ceph-mon[44189]: osdmap e83: 11 total, 10 up, 11 in 2024-04-20T17:05:33.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:33.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:33.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:32 smithi171 ceph-mon[44189]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:33.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:32 smithi171 ceph-mon[44189]: from='osd.10 ' entity='osd.10' 2024-04-20T17:05:33.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:32 smithi005 ceph-mon[39562]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-04-20T17:05:33.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:32 smithi005 ceph-mon[39562]: osdmap e83: 11 total, 10 up, 11 in 2024-04-20T17:05:33.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:33.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:32 smithi005 ceph-mon[39562]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:32 smithi005 ceph-mon[39562]: from='osd.10 ' entity='osd.10' 2024-04-20T17:05:33.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:32 smithi106 ceph-mon[44423]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-04-20T17:05:33.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:32 smithi106 ceph-mon[44423]: osdmap e83: 11 total, 10 up, 11 in 2024-04-20T17:05:33.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:33.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:33.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:32 smithi106 ceph-mon[44423]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 894 GiB / 894 GiB avail 2024-04-20T17:05:33.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:32 smithi106 ceph-mon[44423]: from='osd.10 ' entity='osd.10' 2024-04-20T17:05:33.333 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:05:33.875 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:33 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:05:33.875 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:33 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:05:33.875 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:05:33.875 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:05:33.875 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:33 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:33.875 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:33 smithi171 ceph-mon[44189]: osd.10 v1:172.21.15.171:6808/181490621 boot 2024-04-20T17:05:33.876 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:33 smithi171 ceph-mon[44189]: osdmap e84: 11 total, 11 up, 11 in 2024-04-20T17:05:33.876 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:33 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:34.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:33 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:05:34.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:33 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:05:34.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:05:34.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:05:34.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:33 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:34.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:33 smithi005 ceph-mon[39562]: osd.10 v1:172.21.15.171:6808/181490621 boot 2024-04-20T17:05:34.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:33 smithi005 ceph-mon[39562]: osdmap e84: 11 total, 11 up, 11 in 2024-04-20T17:05:34.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:33 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:34.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:33 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:05:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:33 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:05:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:05:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:05:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:33 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:33 smithi106 ceph-mon[44423]: osd.10 v1:172.21.15.171:6808/181490621 boot 2024-04-20T17:05:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:33 smithi106 ceph-mon[44423]: osdmap e84: 11 total, 11 up, 11 in 2024-04-20T17:05:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:33 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-20T17:05:35.540 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:35 smithi171 ceph-mon[44189]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:35.540 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:35 smithi171 ceph-mon[44189]: osdmap e85: 11 total, 11 up, 11 in 2024-04-20T17:05:35.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:35 smithi005 ceph-mon[39562]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:35.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:35 smithi005 ceph-mon[39562]: osdmap e85: 11 total, 11 up, 11 in 2024-04-20T17:05:35.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:35 smithi106 ceph-mon[44423]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:35 smithi106 ceph-mon[44423]: osdmap e85: 11 total, 11 up, 11 in 2024-04-20T17:05:36.722 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-20T17:05:36.722 INFO:teuthology.orchestra.run.smithi171.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-20T17:05:36.722 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10+0 records in 2024-04-20T17:05:36.722 INFO:teuthology.orchestra.run.smithi171.stderr:10+0 records out 2024-04-20T17:05:36.722 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110971 s, 945 MB/s 2024-04-20T17:05:36.722 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping successful for: 2024-04-20T17:05:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:37 smithi171 ceph-mon[44189]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:37 smithi171 ceph-mon[44189]: Detected new or changed devices on smithi171 2024-04-20T17:05:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:37 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:37 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:37 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:05:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:37 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:05:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:37 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:05:37.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:37 smithi171 ceph-mon[44189]: Adjusting osd_memory_target on smithi171 to 7072M 2024-04-20T17:05:37.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:37 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:37.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:37 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:05:37.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:37 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:05:37.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:37 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:37.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:37 smithi005 ceph-mon[39562]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:37.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:37 smithi005 ceph-mon[39562]: Detected new or changed devices on smithi171 2024-04-20T17:05:37.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:37 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:37.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:37 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:37.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:37 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:05:37.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:37 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:05:37.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:37 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:05:37.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:37 smithi005 ceph-mon[39562]: Adjusting osd_memory_target on smithi171 to 7072M 2024-04-20T17:05:37.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:37 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:37.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:37 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:05:37.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:37 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:05:37.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:37 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:37.495 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi171:vg_nvme/lv_1 2024-04-20T17:05:37.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:37 smithi106 ceph-mon[44423]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:37 smithi106 ceph-mon[44423]: Detected new or changed devices on smithi171 2024-04-20T17:05:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:37 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:37 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:37 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:05:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:37 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:05:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:37 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:05:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:37 smithi106 ceph-mon[44423]: Adjusting osd_memory_target on smithi171 to 7072M 2024-04-20T17:05:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:37 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:37 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:05:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:37 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:05:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:37 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:37.763 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:05:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:39 smithi171 ceph-mon[44189]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:39.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:39 smithi005 ceph-mon[39562]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:39 smithi106 ceph-mon[44423]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:41 smithi171 ceph-mon[44189]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:41 smithi171 ceph-mon[44189]: from='client.24418 v1:172.21.15.171:0/2770792891' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:05:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:41 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:05:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:41 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:05:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:41 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:05:41.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:41 smithi005 ceph-mon[39562]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:41.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:41 smithi005 ceph-mon[39562]: from='client.24418 v1:172.21.15.171:0/2770792891' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:05:41.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:41 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:05:41.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:41 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:05:41.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:41 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:05:41.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:41 smithi106 ceph-mon[44423]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:41.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:41 smithi106 ceph-mon[44423]: from='client.24418 v1:172.21.15.171:0/2770792891' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-20T17:05:41.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:41 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-20T17:05:41.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:41 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-20T17:05:41.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:41 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:05:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:43 smithi171 ceph-mon[44189]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:43 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.171:0/3812221323' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "737c5aec-694a-4968-819d-fccf22661f4f"}]: dispatch 2024-04-20T17:05:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:43 smithi171 ceph-mon[44189]: from='client.24424 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "737c5aec-694a-4968-819d-fccf22661f4f"}]: dispatch 2024-04-20T17:05:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:43 smithi171 ceph-mon[44189]: from='client.24424 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "737c5aec-694a-4968-819d-fccf22661f4f"}]': finished 2024-04-20T17:05:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:43 smithi171 ceph-mon[44189]: osdmap e86: 12 total, 11 up, 12 in 2024-04-20T17:05:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-20T17:05:43.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:43 smithi005 ceph-mon[39562]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:43.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:43 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.171:0/3812221323' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "737c5aec-694a-4968-819d-fccf22661f4f"}]: dispatch 2024-04-20T17:05:43.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:43 smithi005 ceph-mon[39562]: from='client.24424 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "737c5aec-694a-4968-819d-fccf22661f4f"}]: dispatch 2024-04-20T17:05:43.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:43 smithi005 ceph-mon[39562]: from='client.24424 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "737c5aec-694a-4968-819d-fccf22661f4f"}]': finished 2024-04-20T17:05:43.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:43 smithi005 ceph-mon[39562]: osdmap e86: 12 total, 11 up, 12 in 2024-04-20T17:05:43.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-20T17:05:43.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:43 smithi106 ceph-mon[44423]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:43 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.171:0/3812221323' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "737c5aec-694a-4968-819d-fccf22661f4f"}]: dispatch 2024-04-20T17:05:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:43 smithi106 ceph-mon[44423]: from='client.24424 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "737c5aec-694a-4968-819d-fccf22661f4f"}]: dispatch 2024-04-20T17:05:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:43 smithi106 ceph-mon[44423]: from='client.24424 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "737c5aec-694a-4968-819d-fccf22661f4f"}]': finished 2024-04-20T17:05:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:43 smithi106 ceph-mon[44423]: osdmap e86: 12 total, 11 up, 12 in 2024-04-20T17:05:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-20T17:05:44.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:44 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.171:0/1982171420' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:05:44.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:44 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.171:0/1982171420' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:05:44.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:44 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.171:0/1982171420' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-20T17:05:45.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:45 smithi171 ceph-mon[44189]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:45.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:45 smithi005 ceph-mon[39562]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:45 smithi106 ceph-mon[44423]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:46.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:46 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:05:46.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:46 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:05:46.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:46 smithi171 ceph-mon[44189]: osdmap e87: 12 total, 11 up, 12 in 2024-04-20T17:05:46.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:46 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-20T17:05:46.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:46 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:05:46.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:46 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:05:46.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:46 smithi005 ceph-mon[39562]: osdmap e87: 12 total, 11 up, 12 in 2024-04-20T17:05:46.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:46 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-20T17:05:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:46 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:05:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:46 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:05:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:46 smithi106 ceph-mon[44423]: osdmap e87: 12 total, 11 up, 12 in 2024-04-20T17:05:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:46 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-20T17:05:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:47 smithi106 ceph-mon[44423]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:48.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:47 smithi171 ceph-mon[44189]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:48.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:47 smithi005 ceph-mon[39562]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:50.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:49 smithi171 ceph-mon[44189]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:49 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:05:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:49 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:05:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:49 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-20T17:05:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:49 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:05:50.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:49 smithi005 ceph-mon[39562]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:50.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:49 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:05:50.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:49 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:05:50.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:49 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-20T17:05:50.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:49 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:05:50.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:49 smithi106 ceph-mon[44423]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:49 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:05:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:49 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:05:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:49 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-20T17:05:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:49 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:05:50.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:50 smithi171 ceph-mon[44189]: Deploying daemon osd.11 on smithi171 2024-04-20T17:05:50.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:50 smithi171 ceph-mon[44189]: osdmap e88: 12 total, 11 up, 12 in 2024-04-20T17:05:50.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:50 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-20T17:05:50.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:50 smithi171 ceph-mon[44189]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:51.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:50 smithi005 ceph-mon[39562]: Deploying daemon osd.11 on smithi171 2024-04-20T17:05:51.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:50 smithi005 ceph-mon[39562]: osdmap e88: 12 total, 11 up, 12 in 2024-04-20T17:05:51.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:50 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-20T17:05:51.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:50 smithi005 ceph-mon[39562]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:51.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:50 smithi106 ceph-mon[44423]: Deploying daemon osd.11 on smithi171 2024-04-20T17:05:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:50 smithi106 ceph-mon[44423]: osdmap e88: 12 total, 11 up, 12 in 2024-04-20T17:05:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:50 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-20T17:05:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:50 smithi106 ceph-mon[44423]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:52 smithi171 ceph-mon[44189]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:53.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:52 smithi005 ceph-mon[39562]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:53.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:52 smithi106 ceph-mon[44423]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:55.195 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:54 smithi171 ceph-mon[44189]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:55.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:54 smithi005 ceph-mon[39562]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:55.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:54 smithi106 ceph-mon[44423]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:56.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:56 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:56.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:56 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:56.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:56 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:05:56.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:56 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:56.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:56 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:56.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:56 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:05:56.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:56 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:56.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:56 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:56.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:56 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:05:57.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:57 smithi005 ceph-mon[39562]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:57.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:57 smithi106 ceph-mon[44423]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:57 smithi171 ceph-mon[44189]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:59.323 INFO:teuthology.orchestra.run.smithi171.stdout:Created osd(s) 11 on host 'smithi171' 2024-04-20T17:05:59.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:59 smithi171 ceph-mon[44189]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:59.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:59 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:59.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:59 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:59.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:59 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:05:59.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:59 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:05:59.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:59 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:59.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:59 smithi171 ceph-mon[44189]: from='osd.11 v1:172.21.15.171:6812/3373585701' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-20T17:05:59.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:59 smithi171 ceph-mon[44189]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-20T17:05:59.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:59 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:59.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:59 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:59.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:05:59 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:05:59.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:59 smithi005 ceph-mon[39562]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:59.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:59 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:59.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:59 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:59.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:59 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:05:59.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:59 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:05:59.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:59 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:59.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:59 smithi005 ceph-mon[39562]: from='osd.11 v1:172.21.15.171:6812/3373585701' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-20T17:05:59.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:59 smithi005 ceph-mon[39562]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-20T17:05:59.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:59 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:59.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:59 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:59.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:05:59 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:05:59.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:59 smithi106 ceph-mon[44423]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:05:59.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:59 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:59.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:59 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:59.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:59 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:05:59.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:59 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:05:59.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:59 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:59.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:59 smithi106 ceph-mon[44423]: from='osd.11 v1:172.21.15.171:6812/3373585701' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-20T17:05:59.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:59 smithi106 ceph-mon[44423]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-20T17:05:59.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:59 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:59.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:59 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:05:59.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:05:59 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:06:00.103 DEBUG:teuthology.orchestra.run.smithi171:osd.11> sudo journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.11.service 2024-04-20T17:06:00.105 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-20T17:06:00.105 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-20T17:06:00.136 INFO:journalctl@ceph.osd.11.smithi171.stdout:-- Logs begin at Sat 2024-04-20 16:47:48 UTC. -- 2024-04-20T17:06:00.393 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:06:00.658 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:00 smithi171 ceph-mon[44189]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-20T17:06:00.658 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:00 smithi171 ceph-mon[44189]: osdmap e89: 12 total, 11 up, 12 in 2024-04-20T17:06:00.658 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:00 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-20T17:06:00.658 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:00 smithi171 ceph-mon[44189]: from='osd.11 v1:172.21.15.171:6812/3373585701' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-20T17:06:00.658 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:00 smithi171 ceph-mon[44189]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-20T17:06:00.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:00 smithi106 ceph-mon[44423]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-20T17:06:00.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:00 smithi106 ceph-mon[44423]: osdmap e89: 12 total, 11 up, 12 in 2024-04-20T17:06:00.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:00 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-20T17:06:00.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:00 smithi106 ceph-mon[44423]: from='osd.11 v1:172.21.15.171:6812/3373585701' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-20T17:06:00.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:00 smithi106 ceph-mon[44423]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-20T17:06:00.768 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:00 smithi005 ceph-mon[39562]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-20T17:06:00.768 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:00 smithi005 ceph-mon[39562]: osdmap e89: 12 total, 11 up, 12 in 2024-04-20T17:06:00.768 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:00 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-20T17:06:00.768 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:00 smithi005 ceph-mon[39562]: from='osd.11 v1:172.21.15.171:6812/3373585701' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-20T17:06:00.768 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:00 smithi005 ceph-mon[39562]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-20T17:06:01.334 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:06:00 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:06:00.930+0000 7f113d196700 -1 osd.11 0 waiting for initial osdmap 2024-04-20T17:06:01.334 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:06:00 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:06:00.930+0000 7f113090c700 -1 osd.11 0 failed to load OSD map for epoch 34, got 0 bytes 2024-04-20T17:06:01.334 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:06:00 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:06:00.939+0000 7f1134f81700 -1 osd.11 91 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T17:06:01.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:01 smithi005 ceph-mon[39562]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:06:01.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:01 smithi005 ceph-mon[39562]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-04-20T17:06:01.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:01 smithi005 ceph-mon[39562]: osdmap e90: 12 total, 11 up, 12 in 2024-04-20T17:06:01.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:01 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-20T17:06:01.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:01 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-20T17:06:01.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:01 smithi005 ceph-mon[39562]: osdmap e91: 12 total, 11 up, 12 in 2024-04-20T17:06:01.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:01 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-20T17:06:01.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:01 smithi005 ceph-mon[39562]: from='osd.11 ' entity='osd.11' 2024-04-20T17:06:01.736 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:01 smithi171 ceph-mon[44189]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:06:01.736 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:01 smithi171 ceph-mon[44189]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-04-20T17:06:01.736 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:01 smithi171 ceph-mon[44189]: osdmap e90: 12 total, 11 up, 12 in 2024-04-20T17:06:01.736 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:01 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-20T17:06:01.737 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:01 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-20T17:06:01.737 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:01 smithi171 ceph-mon[44189]: osdmap e91: 12 total, 11 up, 12 in 2024-04-20T17:06:01.737 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:01 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-20T17:06:01.737 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:01 smithi171 ceph-mon[44189]: from='osd.11 ' entity='osd.11' 2024-04-20T17:06:01.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:01 smithi106 ceph-mon[44423]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:06:01.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:01 smithi106 ceph-mon[44423]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-04-20T17:06:01.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:01 smithi106 ceph-mon[44423]: osdmap e90: 12 total, 11 up, 12 in 2024-04-20T17:06:01.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:01 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-20T17:06:01.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:01 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-20T17:06:01.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:01 smithi106 ceph-mon[44423]: osdmap e91: 12 total, 11 up, 12 in 2024-04-20T17:06:01.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:01 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-20T17:06:01.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:01 smithi106 ceph-mon[44423]: from='osd.11 ' entity='osd.11' 2024-04-20T17:06:02.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:02 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:06:02.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:02 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:06:02.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:06:02.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:06:02.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:02 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-20T17:06:02.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:02 smithi005 ceph-mon[39562]: osd.11 v1:172.21.15.171:6812/3373585701 boot 2024-04-20T17:06:02.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:02 smithi005 ceph-mon[39562]: osdmap e92: 12 total, 12 up, 12 in 2024-04-20T17:06:02.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:02 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-20T17:06:02.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:02 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:06:02.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:02 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:06:02.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:06:02.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:06:02.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:02 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-20T17:06:02.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:02 smithi106 ceph-mon[44423]: osd.11 v1:172.21.15.171:6812/3373585701 boot 2024-04-20T17:06:02.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:02 smithi106 ceph-mon[44423]: osdmap e92: 12 total, 12 up, 12 in 2024-04-20T17:06:02.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:02 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-20T17:06:02.782 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:06:02.818 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:02 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:06:02.818 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:02 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:06:02.818 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:06:02.818 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:06:02.818 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:02 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-20T17:06:02.818 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:02 smithi171 ceph-mon[44189]: osd.11 v1:172.21.15.171:6812/3373585701 boot 2024-04-20T17:06:02.818 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:02 smithi171 ceph-mon[44189]: osdmap e92: 12 total, 12 up, 12 in 2024-04-20T17:06:02.818 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:02 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-20T17:06:03.550 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:03 smithi171 ceph-mon[44189]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:06:03.550 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:03 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2522875822' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-20T17:06:03.550 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:03 smithi171 ceph-mon[44189]: osdmap e93: 12 total, 12 up, 12 in 2024-04-20T17:06:03.666 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":92,"num_osds":12,"num_up_osds":12,"osd_up_since":1713632761,"num_in_osds":12,"osd_in_since":1713632742,"num_remapped_pgs":1} 2024-04-20T17:06:03.666 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-20T17:06:03.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:03 smithi106 ceph-mon[44423]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:06:03.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:03 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2522875822' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-20T17:06:03.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:03 smithi106 ceph-mon[44423]: osdmap e93: 12 total, 12 up, 12 in 2024-04-20T17:06:03.946 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:06:03.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:03 smithi005 ceph-mon[39562]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-20T17:06:03.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:03 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2522875822' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-20T17:06:03.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:03 smithi005 ceph-mon[39562]: osdmap e93: 12 total, 12 up, 12 in 2024-04-20T17:06:04.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:04 smithi005 ceph-mon[39562]: Detected new or changed devices on smithi171 2024-04-20T17:06:04.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:04 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:06:04.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:04 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:06:04.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:04 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:06:04.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:04 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:06:04.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:04 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:06:04.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:04 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:06:04.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:04 smithi005 ceph-mon[39562]: Adjusting osd_memory_target on smithi171 to 5304M 2024-04-20T17:06:04.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:04 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:06:04.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:04 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:06:04.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:04 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:06:04.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:04 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:06:04.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:04 smithi005 ceph-mon[39562]: osdmap e94: 12 total, 12 up, 12 in 2024-04-20T17:06:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:04 smithi106 ceph-mon[44423]: Detected new or changed devices on smithi171 2024-04-20T17:06:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:04 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:06:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:04 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:06:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:04 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:06:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:04 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:06:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:04 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:06:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:04 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:06:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:04 smithi106 ceph-mon[44423]: Adjusting osd_memory_target on smithi171 to 5304M 2024-04-20T17:06:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:04 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:06:05.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:04 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:06:05.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:04 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:06:05.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:04 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:06:05.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:04 smithi106 ceph-mon[44423]: osdmap e94: 12 total, 12 up, 12 in 2024-04-20T17:06:05.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:04 smithi171 ceph-mon[44189]: Detected new or changed devices on smithi171 2024-04-20T17:06:05.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:04 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:06:05.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:04 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:06:05.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:04 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:06:05.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:04 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:06:05.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:04 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:06:05.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:04 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-20T17:06:05.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:04 smithi171 ceph-mon[44189]: Adjusting osd_memory_target on smithi171 to 5304M 2024-04-20T17:06:05.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:04 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:06:05.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:04 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:06:05.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:04 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:06:05.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:04 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:06:05.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:04 smithi171 ceph-mon[44189]: osdmap e94: 12 total, 12 up, 12 in 2024-04-20T17:06:05.810 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:05 smithi005 ceph-mon[39562]: pgmap v283: 1 pgs: 1 remapped+peering; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:06.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:05 smithi106 ceph-mon[44423]: pgmap v283: 1 pgs: 1 remapped+peering; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:06.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:05 smithi171 ceph-mon[44189]: pgmap v283: 1 pgs: 1 remapped+peering; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:06.302 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:06:06.302 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":94,"fsid":"13c6e76c-ff37-11ee-bc93-c7b262605968","created":"2024-04-20T16:58:25.272557+0000","modified":"2024-04-20T17:06:03.795512+0000","last_up_change":"2024-04-20T17:06:01.788926+0000","last_in_change":"2024-04-20T17:05:42.849446+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T17:01:35.909493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6801","nonce":3901484705}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6802","nonce":3901484705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6804","nonce":3901484705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6803","nonce":3901484705}]},"public_addr":"172.21.15.5:6801/3901484705","cluster_addr":"172.21.15.5:6802/3901484705","heartbeat_back_addr":"172.21.15.5:6804/3901484705","heartbeat_front_addr":"172.21.15.5:6803/3901484705","state":["exists","up"]},{"osd":1,"uuid":"aed0c8eb-0e81-4e85-ab8f-a60420060656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6805","nonce":3391893914}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6806","nonce":3391893914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6808","nonce":3391893914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6807","nonce":3391893914}]},"public_addr":"172.21.15.5:6805/3391893914","cluster_addr":"172.21.15.5:6806/3391893914","heartbeat_back_addr":"172.21.15.5:6808/3391893914","heartbeat_front_addr":"172.21.15.5:6807/3391893914","state":["exists","up"]},{"osd":2,"uuid":"f733a6a2-19c2-4c7a-af9c-04e282062bb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6809","nonce":2105468358}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6810","nonce":2105468358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6812","nonce":2105468358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6811","nonce":2105468358}]},"public_addr":"172.21.15.5:6809/2105468358","cluster_addr":"172.21.15.5:6810/2105468358","heartbeat_back_addr":"172.21.15.5:6812/2105468358","heartbeat_front_addr":"172.21.15.5:6811/2105468358","state":["exists","up"]},{"osd":3,"uuid":"8c92f1df-e298-4606-88ae-b8a8e79628a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6813","nonce":377652039}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6814","nonce":377652039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6816","nonce":377652039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6815","nonce":377652039}]},"public_addr":"172.21.15.5:6813/377652039","cluster_addr":"172.21.15.5:6814/377652039","heartbeat_back_addr":"172.21.15.5:6816/377652039","heartbeat_front_addr":"172.21.15.5:6815/377652039","state":["exists","up"]},{"osd":4,"uuid":"46df16f1-9607-4d7b-9be8-9d4236e160d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6800","nonce":1839524469}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6801","nonce":1839524469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6803","nonce":1839524469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6802","nonce":1839524469}]},"public_addr":"172.21.15.106:6800/1839524469","cluster_addr":"172.21.15.106:6801/1839524469","heartbeat_back_addr":"172.21.15.106:6803/1839524469","heartbeat_front_addr":"172.21.15.106:6802/1839524469","state":["exists","up"]},{"osd":5,"uuid":"6603231d-1e5c-4575-bfaa-2eeca2c12215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6804","nonce":2450315908}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6805","nonce":2450315908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6807","nonce":2450315908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6806","nonce":2450315908}]},"public_addr":"172.21.15.106:6804/2450315908","cluster_addr":"172.21.15.106:6805/2450315908","heartbeat_back_addr":"172.21.15.106:6807/2450315908","heartbeat_front_addr":"172.21.15.106:6806/2450315908","state":["exists","up"]},{"osd":6,"uuid":"783a55a6-f176-42be-8c8b-af8481d08dc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6808","nonce":2840237913}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6809","nonce":2840237913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6811","nonce":2840237913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6810","nonce":2840237913}]},"public_addr":"172.21.15.106:6808/2840237913","cluster_addr":"172.21.15.106:6809/2840237913","heartbeat_back_addr":"172.21.15.106:6811/2840237913","heartbeat_front_addr":"172.21.15.106:6810/2840237913","state":["exists","up"]},{"osd":7,"uuid":"b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6812","nonce":1105371883}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6813","nonce":1105371883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6815","nonce":1105371883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6814","nonce":1105371883}]},"public_addr":"172.21.15.106:6812/1105371883","cluster_addr":"172.21.15.106:6813/1105371883","heartbeat_back_addr":"172.21.15.106:6815/1105371883","heartbeat_front_addr":"172.21.15.106:6814/1105371883","state":["exists","up"]},{"osd":8,"uuid":"1cb232ad-1f59-4a0a-a215-e67419ad6476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6800","nonce":4123058942}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6801","nonce":4123058942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6803","nonce":4123058942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6802","nonce":4123058942}]},"public_addr":"172.21.15.171:6800/4123058942","cluster_addr":"172.21.15.171:6801/4123058942","heartbeat_back_addr":"172.21.15.171:6803/4123058942","heartbeat_front_addr":"172.21.15.171:6802/4123058942","state":["exists","up"]},{"osd":9,"uuid":"b6c17147-b3dc-475b-bb20-f9f712fb99e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6804","nonce":2877589636}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6805","nonce":2877589636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6807","nonce":2877589636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6806","nonce":2877589636}]},"public_addr":"172.21.15.171:6804/2877589636","cluster_addr":"172.21.15.171:6805/2877589636","heartbeat_back_addr":"172.21.15.171:6807/2877589636","heartbeat_front_addr":"172.21.15.171:6806/2877589636","state":["exists","up"]},{"osd":10,"uuid":"1adcdf55-f002-4a39-91e7-378e5fae849c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6808","nonce":181490621}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6809","nonce":181490621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6811","nonce":181490621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6810","nonce":181490621}]},"public_addr":"172.21.15.171:6808/181490621","cluster_addr":"172.21.15.171:6809/181490621","heartbeat_back_addr":"172.21.15.171:6811/181490621","heartbeat_front_addr":"172.21.15.171:6810/181490621","state":["exists","up"]},{"osd":11,"uuid":"737c5aec-694a-4968-819d-fccf22661f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":92,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6812","nonce":3373585701}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6813","nonce":3373585701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6815","nonce":3373585701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6814","nonce":3373585701}]},"public_addr":"172.21.15.171:6812/3373585701","cluster_addr":"172.21.15.171:6813/3373585701","heartbeat_back_addr":"172.21.15.171:6815/3373585701","heartbeat_front_addr":"172.21.15.171:6814/3373585701","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:05:09.000834+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:05:28.631146+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:05:44.274626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:05:06.065627+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:05:29.574150+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:05:23.947765+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:04:57.594057+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:05:15.604773+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:05:49.589055+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:05:03.121230+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:05:31.405460+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:05:59.566716+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.5:0/3881726425":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/338649532":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/1227843664":"2024-04-21T16:58:55.392162+0000","172.21.15.5:6800/1876064631":"2024-04-21T16:58:55.392162+0000","172.21.15.5:0/160689449":"2024-04-21T16:58:55.392162+0000","172.21.15.5:6800/3521715286":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/3841663462":"2024-04-21T16:58:55.392162+0000","172.21.15.5:0/687690919":"2024-04-21T16:59:25.808337+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T17:06:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:06 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3614809074' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:06:07.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:06 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3614809074' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:06:07.164 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-20T17:01:35.909493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.760000228881836, 'score_stable': 11.760000228881836, 'optimal_score': 0.17000000178813934, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-20T17:06:07.165 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-20T17:06:07.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:06 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3614809074' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:06:07.445 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:06:07.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:07 smithi005 ceph-mon[39562]: pgmap v284: 1 pgs: 1 remapped+peering; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:07.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:07 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:06:07.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:07 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:06:08.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:07 smithi171 ceph-mon[44189]: pgmap v284: 1 pgs: 1 remapped+peering; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:07 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:06:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:07 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:06:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:07 smithi106 ceph-mon[44423]: pgmap v284: 1 pgs: 1 remapped+peering; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:07 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:06:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:07 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:06:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:08 smithi171 ceph-mon[44189]: osdmap e95: 12 total, 12 up, 12 in 2024-04-20T17:06:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:08 smithi171 ceph-mon[44189]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-20T17:06:09.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:08 smithi005 ceph-mon[39562]: osdmap e95: 12 total, 12 up, 12 in 2024-04-20T17:06:09.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:08 smithi005 ceph-mon[39562]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-20T17:06:09.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:08 smithi106 ceph-mon[44423]: osdmap e95: 12 total, 12 up, 12 in 2024-04-20T17:06:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:08 smithi106 ceph-mon[44423]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-20T17:06:09.726 INFO:teuthology.orchestra.run.smithi005.stdout:pg_num: 1 2024-04-20T17:06:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:09 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:06:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:09 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:06:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:09 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1940577727' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-20T17:06:10.173 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:09 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:06:10.174 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:09 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:06:10.174 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:09 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1940577727' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-20T17:06:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:09 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:06:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:09 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:06:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:09 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1940577727' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-20T17:06:10.671 INFO:tasks.cephadm:Setting up client nodes... 2024-04-20T17:06:10.672 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-20T17:06:10.953 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:06:10.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:10 smithi005 ceph-mon[39562]: osdmap e96: 12 total, 12 up, 12 in 2024-04-20T17:06:10.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:10 smithi005 ceph-mon[39562]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-20T17:06:11.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:10 smithi171 ceph-mon[44189]: osdmap e96: 12 total, 12 up, 12 in 2024-04-20T17:06:11.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:10 smithi171 ceph-mon[44189]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-20T17:06:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:10 smithi106 ceph-mon[44423]: osdmap e96: 12 total, 12 up, 12 in 2024-04-20T17:06:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:10 smithi106 ceph-mon[44423]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-20T17:06:13.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:12 smithi005 ceph-mon[39562]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-20T17:06:13.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:12 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:06:13.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:12 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:06:13.246 INFO:teuthology.orchestra.run.smithi005.stdout:[client.0] 2024-04-20T17:06:13.246 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQAF9iNmS0yfDhAARZUQJN66JTHSSEkNitDxAg== 2024-04-20T17:06:13.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:12 smithi106 ceph-mon[44423]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-20T17:06:13.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:12 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:06:13.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:12 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:06:13.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:12 smithi171 ceph-mon[44189]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-20T17:06:13.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:12 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:06:13.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:12 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:06:13.990 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-04-20T17:06:13.990 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-20T17:06:13.990 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-20T17:06:14.033 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-20T17:06:14.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:13 smithi005 ceph-mon[39562]: osdmap e97: 12 total, 12 up, 12 in 2024-04-20T17:06:14.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:13 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2294878513' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-20T17:06:14.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:13 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2294878513' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-20T17:06:14.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:13 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:06:14.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:13 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:06:14.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:13 smithi106 ceph-mon[44423]: osdmap e97: 12 total, 12 up, 12 in 2024-04-20T17:06:14.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:13 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2294878513' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-20T17:06:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:13 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2294878513' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-20T17:06:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:13 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:06:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:13 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:06:14.312 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:06:14.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:13 smithi171 ceph-mon[44189]: osdmap e97: 12 total, 12 up, 12 in 2024-04-20T17:06:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:13 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2294878513' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-20T17:06:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:13 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2294878513' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-20T17:06:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:13 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:06:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:13 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:06:15.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:15 smithi106 ceph-mon[44423]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:15.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:15 smithi106 ceph-mon[44423]: osdmap e98: 12 total, 12 up, 12 in 2024-04-20T17:06:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:15 smithi171 ceph-mon[44189]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:15 smithi171 ceph-mon[44189]: osdmap e98: 12 total, 12 up, 12 in 2024-04-20T17:06:15.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:15 smithi005 ceph-mon[39562]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:15.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:15 smithi005 ceph-mon[39562]: osdmap e98: 12 total, 12 up, 12 in 2024-04-20T17:06:16.445 INFO:teuthology.orchestra.run.smithi106.stdout:[client.1] 2024-04-20T17:06:16.445 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQAI9iNmK5IsGhAABi/L990VXB1IBfXfAmf5Og== 2024-04-20T17:06:17.133 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-04-20T17:06:17.133 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-20T17:06:17.133 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-20T17:06:17.177 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-20T17:06:17.456 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:06:17.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:17 smithi106 ceph-mon[44423]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:17.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:17 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.106:0/187220122' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-20T17:06:17.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:17 smithi106 ceph-mon[44423]: from='client.24461 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-20T17:06:17.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:17 smithi106 ceph-mon[44423]: from='client.24461 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-20T17:06:17.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:17 smithi171 ceph-mon[44189]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:17.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:17 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.106:0/187220122' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-20T17:06:17.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:17 smithi171 ceph-mon[44189]: from='client.24461 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-20T17:06:17.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:17 smithi171 ceph-mon[44189]: from='client.24461 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-20T17:06:17.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:17 smithi005 ceph-mon[39562]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:17.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:17 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.106:0/187220122' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-20T17:06:17.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:17 smithi005 ceph-mon[39562]: from='client.24461 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-20T17:06:17.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:17 smithi005 ceph-mon[39562]: from='client.24461 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-20T17:06:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:19 smithi171 ceph-mon[44189]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:19.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:19 smithi106 ceph-mon[44423]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:19.602 INFO:teuthology.orchestra.run.smithi171.stdout:[client.2] 2024-04-20T17:06:19.602 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQAL9iNm1K60IxAAFJHny2/z1auV1yHQcCWLNg== 2024-04-20T17:06:19.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:19 smithi005 ceph-mon[39562]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:20.573 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:20 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.171:0/699427825' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-20T17:06:20.573 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:20 smithi171 ceph-mon[44189]: from='client.24475 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-20T17:06:20.573 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:20 smithi171 ceph-mon[44189]: from='client.24475 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-20T17:06:20.601 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-04-20T17:06:20.601 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-04-20T17:06:20.602 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-04-20T17:06:20.642 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-20T17:06:20.643 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-20T17:06:20.643 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-20T17:06:20.674 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:20 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.171:0/699427825' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-20T17:06:20.674 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:20 smithi005 ceph-mon[39562]: from='client.24475 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-20T17:06:20.674 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:20 smithi005 ceph-mon[39562]: from='client.24475 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-20T17:06:20.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:20 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.171:0/699427825' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-20T17:06:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:20 smithi106 ceph-mon[44423]: from='client.24475 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-20T17:06:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:20 smithi106 ceph-mon[44423]: from='client.24475 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-20T17:06:20.931 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:06:21.879 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:21 smithi005 ceph-mon[39562]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:22.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:21 smithi106 ceph-mon[44423]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:21 smithi171 ceph-mon[44189]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:23.039 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:06:23.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:23 smithi005 ceph-mon[39562]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:23.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:23 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3526309225' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-20T17:06:24.000 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":15,"flags":0,"active_gid":14148,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6800","nonce":1706825907}]},"active_addr":"172.21.15.5:6800/1706825907","active_change":"2024-04-20T16:59:25.808446+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24110,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.5:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":1502248079}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":3459237423}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":2515502853}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":1739089445}]}]} 2024-04-20T17:06:24.004 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-20T17:06:24.004 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-20T17:06:24.005 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-20T17:06:24.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:23 smithi106 ceph-mon[44423]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:24.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:23 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3526309225' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-20T17:06:24.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:23 smithi171 ceph-mon[44189]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:24.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:23 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3526309225' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-20T17:06:24.321 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:06:25.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:25 smithi005 ceph-mon[39562]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:26.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:25 smithi106 ceph-mon[44423]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:26.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:25 smithi171 ceph-mon[44189]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:26.570 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:06:26.570 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":98,"fsid":"13c6e76c-ff37-11ee-bc93-c7b262605968","created":"2024-04-20T16:58:25.272557+0000","modified":"2024-04-20T17:06:14.247040+0000","last_up_change":"2024-04-20T17:06:01.788926+0000","last_in_change":"2024-04-20T17:05:42.849446+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T17:01:35.909493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6801","nonce":3901484705}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6802","nonce":3901484705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6804","nonce":3901484705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6803","nonce":3901484705}]},"public_addr":"172.21.15.5:6801/3901484705","cluster_addr":"172.21.15.5:6802/3901484705","heartbeat_back_addr":"172.21.15.5:6804/3901484705","heartbeat_front_addr":"172.21.15.5:6803/3901484705","state":["exists","up"]},{"osd":1,"uuid":"aed0c8eb-0e81-4e85-ab8f-a60420060656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6805","nonce":3391893914}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6806","nonce":3391893914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6808","nonce":3391893914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6807","nonce":3391893914}]},"public_addr":"172.21.15.5:6805/3391893914","cluster_addr":"172.21.15.5:6806/3391893914","heartbeat_back_addr":"172.21.15.5:6808/3391893914","heartbeat_front_addr":"172.21.15.5:6807/3391893914","state":["exists","up"]},{"osd":2,"uuid":"f733a6a2-19c2-4c7a-af9c-04e282062bb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6809","nonce":2105468358}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6810","nonce":2105468358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6812","nonce":2105468358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6811","nonce":2105468358}]},"public_addr":"172.21.15.5:6809/2105468358","cluster_addr":"172.21.15.5:6810/2105468358","heartbeat_back_addr":"172.21.15.5:6812/2105468358","heartbeat_front_addr":"172.21.15.5:6811/2105468358","state":["exists","up"]},{"osd":3,"uuid":"8c92f1df-e298-4606-88ae-b8a8e79628a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6813","nonce":377652039}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6814","nonce":377652039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6816","nonce":377652039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6815","nonce":377652039}]},"public_addr":"172.21.15.5:6813/377652039","cluster_addr":"172.21.15.5:6814/377652039","heartbeat_back_addr":"172.21.15.5:6816/377652039","heartbeat_front_addr":"172.21.15.5:6815/377652039","state":["exists","up"]},{"osd":4,"uuid":"46df16f1-9607-4d7b-9be8-9d4236e160d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6800","nonce":1839524469}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6801","nonce":1839524469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6803","nonce":1839524469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6802","nonce":1839524469}]},"public_addr":"172.21.15.106:6800/1839524469","cluster_addr":"172.21.15.106:6801/1839524469","heartbeat_back_addr":"172.21.15.106:6803/1839524469","heartbeat_front_addr":"172.21.15.106:6802/1839524469","state":["exists","up"]},{"osd":5,"uuid":"6603231d-1e5c-4575-bfaa-2eeca2c12215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6804","nonce":2450315908}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6805","nonce":2450315908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6807","nonce":2450315908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6806","nonce":2450315908}]},"public_addr":"172.21.15.106:6804/2450315908","cluster_addr":"172.21.15.106:6805/2450315908","heartbeat_back_addr":"172.21.15.106:6807/2450315908","heartbeat_front_addr":"172.21.15.106:6806/2450315908","state":["exists","up"]},{"osd":6,"uuid":"783a55a6-f176-42be-8c8b-af8481d08dc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6808","nonce":2840237913}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6809","nonce":2840237913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6811","nonce":2840237913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6810","nonce":2840237913}]},"public_addr":"172.21.15.106:6808/2840237913","cluster_addr":"172.21.15.106:6809/2840237913","heartbeat_back_addr":"172.21.15.106:6811/2840237913","heartbeat_front_addr":"172.21.15.106:6810/2840237913","state":["exists","up"]},{"osd":7,"uuid":"b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6812","nonce":1105371883}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6813","nonce":1105371883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6815","nonce":1105371883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6814","nonce":1105371883}]},"public_addr":"172.21.15.106:6812/1105371883","cluster_addr":"172.21.15.106:6813/1105371883","heartbeat_back_addr":"172.21.15.106:6815/1105371883","heartbeat_front_addr":"172.21.15.106:6814/1105371883","state":["exists","up"]},{"osd":8,"uuid":"1cb232ad-1f59-4a0a-a215-e67419ad6476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6800","nonce":4123058942}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6801","nonce":4123058942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6803","nonce":4123058942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6802","nonce":4123058942}]},"public_addr":"172.21.15.171:6800/4123058942","cluster_addr":"172.21.15.171:6801/4123058942","heartbeat_back_addr":"172.21.15.171:6803/4123058942","heartbeat_front_addr":"172.21.15.171:6802/4123058942","state":["exists","up"]},{"osd":9,"uuid":"b6c17147-b3dc-475b-bb20-f9f712fb99e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6804","nonce":2877589636}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6805","nonce":2877589636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6807","nonce":2877589636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6806","nonce":2877589636}]},"public_addr":"172.21.15.171:6804/2877589636","cluster_addr":"172.21.15.171:6805/2877589636","heartbeat_back_addr":"172.21.15.171:6807/2877589636","heartbeat_front_addr":"172.21.15.171:6806/2877589636","state":["exists","up"]},{"osd":10,"uuid":"1adcdf55-f002-4a39-91e7-378e5fae849c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6808","nonce":181490621}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6809","nonce":181490621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6811","nonce":181490621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6810","nonce":181490621}]},"public_addr":"172.21.15.171:6808/181490621","cluster_addr":"172.21.15.171:6809/181490621","heartbeat_back_addr":"172.21.15.171:6811/181490621","heartbeat_front_addr":"172.21.15.171:6810/181490621","state":["exists","up"]},{"osd":11,"uuid":"737c5aec-694a-4968-819d-fccf22661f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":92,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6812","nonce":3373585701}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6813","nonce":3373585701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6815","nonce":3373585701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6814","nonce":3373585701}]},"public_addr":"172.21.15.171:6812/3373585701","cluster_addr":"172.21.15.171:6813/3373585701","heartbeat_back_addr":"172.21.15.171:6815/3373585701","heartbeat_front_addr":"172.21.15.171:6814/3373585701","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:06:12.118225+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:05:28.631146+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:05:44.274626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:06:09.029449+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:05:29.574150+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:05:23.947765+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:06:13.267582+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:05:15.604773+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:05:49.589055+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:06:06.910191+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:05:31.405460+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:05:59.566716+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.5:0/3881726425":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/338649532":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/1227843664":"2024-04-21T16:58:55.392162+0000","172.21.15.5:6800/1876064631":"2024-04-21T16:58:55.392162+0000","172.21.15.5:0/160689449":"2024-04-21T16:58:55.392162+0000","172.21.15.5:6800/3521715286":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/3841663462":"2024-04-21T16:58:55.392162+0000","172.21.15.5:0/687690919":"2024-04-21T16:59:25.808337+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T17:06:26.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:26 smithi005 ceph-mon[39562]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:26.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:26 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3456162787' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:06:27.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:26 smithi171 ceph-mon[44189]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:26 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3456162787' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:06:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:26 smithi106 ceph-mon[44423]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:26 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3456162787' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:06:27.455 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-20T17:06:27.456 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-20T17:06:27.738 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:06:29.098 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:28 smithi005 ceph-mon[39562]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:29.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:28 smithi106 ceph-mon[44423]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:29.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:28 smithi171 ceph-mon[44189]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:29.883 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:06:29.883 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":98,"fsid":"13c6e76c-ff37-11ee-bc93-c7b262605968","created":"2024-04-20T16:58:25.272557+0000","modified":"2024-04-20T17:06:14.247040+0000","last_up_change":"2024-04-20T17:06:01.788926+0000","last_in_change":"2024-04-20T17:05:42.849446+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T17:01:35.909493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6801","nonce":3901484705}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6802","nonce":3901484705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6804","nonce":3901484705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6803","nonce":3901484705}]},"public_addr":"172.21.15.5:6801/3901484705","cluster_addr":"172.21.15.5:6802/3901484705","heartbeat_back_addr":"172.21.15.5:6804/3901484705","heartbeat_front_addr":"172.21.15.5:6803/3901484705","state":["exists","up"]},{"osd":1,"uuid":"aed0c8eb-0e81-4e85-ab8f-a60420060656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6805","nonce":3391893914}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6806","nonce":3391893914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6808","nonce":3391893914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6807","nonce":3391893914}]},"public_addr":"172.21.15.5:6805/3391893914","cluster_addr":"172.21.15.5:6806/3391893914","heartbeat_back_addr":"172.21.15.5:6808/3391893914","heartbeat_front_addr":"172.21.15.5:6807/3391893914","state":["exists","up"]},{"osd":2,"uuid":"f733a6a2-19c2-4c7a-af9c-04e282062bb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6809","nonce":2105468358}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6810","nonce":2105468358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6812","nonce":2105468358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6811","nonce":2105468358}]},"public_addr":"172.21.15.5:6809/2105468358","cluster_addr":"172.21.15.5:6810/2105468358","heartbeat_back_addr":"172.21.15.5:6812/2105468358","heartbeat_front_addr":"172.21.15.5:6811/2105468358","state":["exists","up"]},{"osd":3,"uuid":"8c92f1df-e298-4606-88ae-b8a8e79628a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6813","nonce":377652039}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6814","nonce":377652039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6816","nonce":377652039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6815","nonce":377652039}]},"public_addr":"172.21.15.5:6813/377652039","cluster_addr":"172.21.15.5:6814/377652039","heartbeat_back_addr":"172.21.15.5:6816/377652039","heartbeat_front_addr":"172.21.15.5:6815/377652039","state":["exists","up"]},{"osd":4,"uuid":"46df16f1-9607-4d7b-9be8-9d4236e160d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6800","nonce":1839524469}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6801","nonce":1839524469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6803","nonce":1839524469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6802","nonce":1839524469}]},"public_addr":"172.21.15.106:6800/1839524469","cluster_addr":"172.21.15.106:6801/1839524469","heartbeat_back_addr":"172.21.15.106:6803/1839524469","heartbeat_front_addr":"172.21.15.106:6802/1839524469","state":["exists","up"]},{"osd":5,"uuid":"6603231d-1e5c-4575-bfaa-2eeca2c12215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6804","nonce":2450315908}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6805","nonce":2450315908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6807","nonce":2450315908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6806","nonce":2450315908}]},"public_addr":"172.21.15.106:6804/2450315908","cluster_addr":"172.21.15.106:6805/2450315908","heartbeat_back_addr":"172.21.15.106:6807/2450315908","heartbeat_front_addr":"172.21.15.106:6806/2450315908","state":["exists","up"]},{"osd":6,"uuid":"783a55a6-f176-42be-8c8b-af8481d08dc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6808","nonce":2840237913}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6809","nonce":2840237913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6811","nonce":2840237913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6810","nonce":2840237913}]},"public_addr":"172.21.15.106:6808/2840237913","cluster_addr":"172.21.15.106:6809/2840237913","heartbeat_back_addr":"172.21.15.106:6811/2840237913","heartbeat_front_addr":"172.21.15.106:6810/2840237913","state":["exists","up"]},{"osd":7,"uuid":"b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6812","nonce":1105371883}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6813","nonce":1105371883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6815","nonce":1105371883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6814","nonce":1105371883}]},"public_addr":"172.21.15.106:6812/1105371883","cluster_addr":"172.21.15.106:6813/1105371883","heartbeat_back_addr":"172.21.15.106:6815/1105371883","heartbeat_front_addr":"172.21.15.106:6814/1105371883","state":["exists","up"]},{"osd":8,"uuid":"1cb232ad-1f59-4a0a-a215-e67419ad6476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6800","nonce":4123058942}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6801","nonce":4123058942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6803","nonce":4123058942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6802","nonce":4123058942}]},"public_addr":"172.21.15.171:6800/4123058942","cluster_addr":"172.21.15.171:6801/4123058942","heartbeat_back_addr":"172.21.15.171:6803/4123058942","heartbeat_front_addr":"172.21.15.171:6802/4123058942","state":["exists","up"]},{"osd":9,"uuid":"b6c17147-b3dc-475b-bb20-f9f712fb99e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6804","nonce":2877589636}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6805","nonce":2877589636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6807","nonce":2877589636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6806","nonce":2877589636}]},"public_addr":"172.21.15.171:6804/2877589636","cluster_addr":"172.21.15.171:6805/2877589636","heartbeat_back_addr":"172.21.15.171:6807/2877589636","heartbeat_front_addr":"172.21.15.171:6806/2877589636","state":["exists","up"]},{"osd":10,"uuid":"1adcdf55-f002-4a39-91e7-378e5fae849c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6808","nonce":181490621}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6809","nonce":181490621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6811","nonce":181490621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6810","nonce":181490621}]},"public_addr":"172.21.15.171:6808/181490621","cluster_addr":"172.21.15.171:6809/181490621","heartbeat_back_addr":"172.21.15.171:6811/181490621","heartbeat_front_addr":"172.21.15.171:6810/181490621","state":["exists","up"]},{"osd":11,"uuid":"737c5aec-694a-4968-819d-fccf22661f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":92,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6812","nonce":3373585701}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6813","nonce":3373585701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6815","nonce":3373585701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6814","nonce":3373585701}]},"public_addr":"172.21.15.171:6812/3373585701","cluster_addr":"172.21.15.171:6813/3373585701","heartbeat_back_addr":"172.21.15.171:6815/3373585701","heartbeat_front_addr":"172.21.15.171:6814/3373585701","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:06:12.118225+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:05:28.631146+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:05:44.274626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:06:09.029449+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:05:29.574150+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:05:23.947765+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:06:13.267582+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:05:15.604773+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:05:49.589055+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:06:06.910191+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:05:31.405460+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:05:59.566716+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.5:0/3881726425":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/338649532":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/1227843664":"2024-04-21T16:58:55.392162+0000","172.21.15.5:6800/1876064631":"2024-04-21T16:58:55.392162+0000","172.21.15.5:0/160689449":"2024-04-21T16:58:55.392162+0000","172.21.15.5:6800/3521715286":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/3841663462":"2024-04-21T16:58:55.392162+0000","172.21.15.5:0/687690919":"2024-04-21T16:59:25.808337+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T17:06:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:29 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2892262036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:06:30.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:29 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2892262036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:06:30.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:29 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2892262036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:06:30.632 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-20T17:06:30.633 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-20T17:06:30.633 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-20T17:06:30.634 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-20T17:06:30.634 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-20T17:06:30.634 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-20T17:06:30.635 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-20T17:06:30.635 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-20T17:06:30.636 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-20T17:06:30.636 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-20T17:06:30.637 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-20T17:06:30.637 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-20T17:06:31.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:30 smithi106 ceph-mon[44423]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:31.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:30 smithi171 ceph-mon[44189]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:31.382 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:06:31.384 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:06:31.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:30 smithi005 ceph-mon[39562]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:31.762 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:06:31.783 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:06:31.798 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:06:31.798 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:06:31.804 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:06:31.807 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:06:31.807 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:06:31.811 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:06:33.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:32 smithi106 ceph-mon[44423]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:33.287 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:32 smithi005 ceph-mon[39562]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:33.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:32 smithi171 ceph-mon[44189]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:34.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:33 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:06:34.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:33 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:06:34.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:33 smithi005 ceph-mon[39562]: osdmap e99: 12 total, 12 up, 12 in 2024-04-20T17:06:34.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:33 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:06:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:33 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:06:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:33 smithi106 ceph-mon[44423]: osdmap e99: 12 total, 12 up, 12 in 2024-04-20T17:06:34.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:33 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:06:34.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:33 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:06:34.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:33 smithi171 ceph-mon[44189]: osdmap e99: 12 total, 12 up, 12 in 2024-04-20T17:06:34.709 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:06:34.710 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:06:35.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:34 smithi106 ceph-mon[44423]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:35.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:34 smithi171 ceph-mon[44189]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:34 smithi005 ceph-mon[39562]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:36.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:36 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:06:36.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:36 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:06:36.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:36 smithi005 ceph-mon[39562]: osdmap e100: 12 total, 12 up, 12 in 2024-04-20T17:06:36.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:36 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:06:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:36 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:06:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:36 smithi106 ceph-mon[44423]: osdmap e100: 12 total, 12 up, 12 in 2024-04-20T17:06:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:36 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:06:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:36 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:06:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:36 smithi171 ceph-mon[44189]: osdmap e100: 12 total, 12 up, 12 in 2024-04-20T17:06:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:37 smithi005 ceph-mon[39562]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:37 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:06:37.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:37 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:06:37.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:37 smithi005 ceph-mon[39562]: osdmap e101: 12 total, 12 up, 12 in 2024-04-20T17:06:37.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:37 smithi106 ceph-mon[44423]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:37.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:37 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:06:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:37 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:06:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:37 smithi106 ceph-mon[44423]: osdmap e101: 12 total, 12 up, 12 in 2024-04-20T17:06:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:37 smithi171 ceph-mon[44189]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:37 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:06:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:37 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:06:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:37 smithi171 ceph-mon[44189]: osdmap e101: 12 total, 12 up, 12 in 2024-04-20T17:06:39.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:39 smithi005 ceph-mon[39562]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:39.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:39 smithi106 ceph-mon[44423]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:39.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:39 smithi171 ceph-mon[44189]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:40.665 INFO:teuthology.orchestra.run.smithi005.stdout:34359738437 2024-04-20T17:06:40.666 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-20T17:06:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:41 smithi106 ceph-mon[44423]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:41.273 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:41 smithi005 ceph-mon[39562]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:41 smithi171 ceph-mon[44189]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:42.304 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:42 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:06:42.304 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:42 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:06:42.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:42 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:06:42.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:42 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:06:42.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:42 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:06:42.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:42 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:06:43.111 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:43 smithi005 ceph-mon[39562]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:43.111 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:43 smithi005 ceph-mon[39562]: osdmap e102: 12 total, 12 up, 12 in 2024-04-20T17:06:43.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:43 smithi171 ceph-mon[44189]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:43 smithi171 ceph-mon[44189]: osdmap e102: 12 total, 12 up, 12 in 2024-04-20T17:06:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:43 smithi106 ceph-mon[44423]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:43 smithi106 ceph-mon[44423]: osdmap e102: 12 total, 12 up, 12 in 2024-04-20T17:06:45.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:45 smithi171 ceph-mon[44189]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:45.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:45 smithi005 ceph-mon[39562]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:45.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:45 smithi106 ceph-mon[44423]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:47.079 INFO:teuthology.orchestra.run.smithi005.stdout:133143986225 2024-04-20T17:06:47.079 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-20T17:06:47.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:47 smithi005 ceph-mon[39562]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:48.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:47 smithi171 ceph-mon[44189]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:48.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:47 smithi106 ceph-mon[44423]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:49.934 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:49 smithi005 ceph-mon[39562]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:49 smithi171 ceph-mon[44189]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:49 smithi106 ceph-mon[44423]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:51.849 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:51 smithi005 ceph-mon[39562]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:52.007 INFO:teuthology.orchestra.run.smithi005.stdout:287762808860 2024-04-20T17:06:52.007 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-20T17:06:52.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:51 smithi171 ceph-mon[44189]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:52.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:51 smithi106 ceph-mon[44423]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:53.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:52 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:06:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:52 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:06:53.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:52 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:06:53.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:52 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:06:53.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:52 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:06:53.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:52 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:06:53.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:53 smithi005 ceph-mon[39562]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:54.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:53 smithi171 ceph-mon[44189]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:54.169 INFO:teuthology.orchestra.run.smithi005.stdout:107374182453 2024-04-20T17:06:54.169 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-20T17:06:54.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:53 smithi106 ceph-mon[44423]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:54.502 INFO:teuthology.orchestra.run.smithi005.stdout:206158430248 2024-04-20T17:06:54.503 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-20T17:06:55.054 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:54 smithi005 ceph-mon[39562]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:55.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:54 smithi171 ceph-mon[44189]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:55.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:54 smithi106 ceph-mon[44423]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:55.821 INFO:teuthology.orchestra.run.smithi005.stdout:313532612633 2024-04-20T17:06:55.822 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-20T17:06:56.833 INFO:teuthology.orchestra.run.smithi005.stdout:395136991244 2024-04-20T17:06:56.833 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-20T17:06:56.949 INFO:teuthology.orchestra.run.smithi005.stdout:360777252883 2024-04-20T17:06:56.949 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-20T17:06:57.066 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:56 smithi005 ceph-mon[39562]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:57.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:56 smithi106 ceph-mon[44423]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:57.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:56 smithi171 ceph-mon[44189]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:58.207 INFO:teuthology.orchestra.run.smithi005.stdout:55834574914 2024-04-20T17:06:58.208 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-20T17:06:59.111 INFO:teuthology.orchestra.run.smithi005.stdout:240518168612 2024-04-20T17:06:59.111 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-20T17:06:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:58 smithi005 ceph-mon[39562]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:58 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:06:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:58 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:06:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:06:58 smithi005 ceph-mon[39562]: osdmap e103: 12 total, 12 up, 12 in 2024-04-20T17:06:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:58 smithi106 ceph-mon[44423]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:58 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:06:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:58 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:06:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:06:58 smithi106 ceph-mon[44423]: osdmap e103: 12 total, 12 up, 12 in 2024-04-20T17:06:59.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:58 smithi171 ceph-mon[44189]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:06:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:58 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:06:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:58 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:06:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:06:58 smithi171 ceph-mon[44189]: osdmap e103: 12 total, 12 up, 12 in 2024-04-20T17:06:59.614 INFO:teuthology.orchestra.run.smithi005.stdout:85899345979 2024-04-20T17:06:59.614 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-20T17:07:00.023 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:07:00.076 INFO:teuthology.orchestra.run.smithi005.stdout:158913789997 2024-04-20T17:07:00.076 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-20T17:07:00.096 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:07:00.349 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:07:00.353 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:07:00.354 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:07:00.356 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:07:00.357 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:07:00.362 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:07:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:00 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:07:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:00 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:07:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:00 smithi005 ceph-mon[39562]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:00 smithi005 ceph-mon[39562]: osdmap e104: 12 total, 12 up, 12 in 2024-04-20T17:07:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:00 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:07:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:00 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:07:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:00 smithi106 ceph-mon[44423]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:00 smithi106 ceph-mon[44423]: osdmap e104: 12 total, 12 up, 12 in 2024-04-20T17:07:01.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:00 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:07:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:00 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:07:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:00 smithi171 ceph-mon[44189]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:00 smithi171 ceph-mon[44189]: osdmap e104: 12 total, 12 up, 12 in 2024-04-20T17:07:03.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:02 smithi005 ceph-mon[39562]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:03.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:02 smithi106 ceph-mon[44423]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:03.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:02 smithi171 ceph-mon[44189]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:04.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:03 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:07:04.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:03 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:07:04.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:03 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:07:05.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:04 smithi106 ceph-mon[44423]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:04 smithi171 ceph-mon[44189]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:05.412 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:04 smithi005 ceph-mon[39562]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:07.952 INFO:teuthology.orchestra.run.smithi005.stdout:313532612636 2024-04-20T17:07:08.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:07 smithi171 ceph-mon[44189]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:07 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:07:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:07 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:07:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:07 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3592303798' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-20T17:07:08.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:07 smithi005 ceph-mon[39562]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:08.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:07 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:07:08.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:07 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:07:08.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:07 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3592303798' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-20T17:07:08.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:07 smithi106 ceph-mon[44423]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:07 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:07:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:07 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:07:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:07 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3592303798' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-20T17:07:08.747 INFO:teuthology.orchestra.run.smithi005.stdout:360777252885 2024-04-20T17:07:08.954 INFO:teuthology.orchestra.run.smithi005.stdout:287762808865 2024-04-20T17:07:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:08 smithi171 ceph-mon[44189]: osdmap e105: 12 total, 12 up, 12 in 2024-04-20T17:07:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:08 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1804049436' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-20T17:07:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:08 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2493362392' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-20T17:07:09.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:08 smithi005 ceph-mon[39562]: osdmap e105: 12 total, 12 up, 12 in 2024-04-20T17:07:09.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:08 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1804049436' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-20T17:07:09.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:08 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2493362392' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-20T17:07:09.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:08 smithi106 ceph-mon[44423]: osdmap e105: 12 total, 12 up, 12 in 2024-04-20T17:07:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:08 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1804049436' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-20T17:07:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:08 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2493362392' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-20T17:07:09.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:09 smithi005 ceph-mon[39562]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:10.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:09 smithi171 ceph-mon[44189]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:10.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:09 smithi106 ceph-mon[44423]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:10.386 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808860 got 287762808865 for osd.8 2024-04-20T17:07:10.386 DEBUG:teuthology.parallel:result is None 2024-04-20T17:07:10.698 INFO:teuthology.orchestra.run.smithi005.stdout:395136991249 2024-04-20T17:07:11.052 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:10 smithi005 ceph-mon[39562]: pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:11.052 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:10 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/214547904' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-20T17:07:11.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:10 smithi171 ceph-mon[44189]: pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:11.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:10 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/214547904' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-20T17:07:11.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:10 smithi106 ceph-mon[44423]: pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:10 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/214547904' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-20T17:07:12.126 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:11 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:07:12.127 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:11 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:07:12.127 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:11 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:07:12.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:11 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:07:12.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:11 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:07:12.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:11 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:07:12.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:11 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:07:12.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:11 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:07:12.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:11 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:07:12.450 INFO:tasks.cephadm.ceph_manager.ceph:need seq 360777252883 got 360777252885 for osd.10 2024-04-20T17:07:12.450 DEBUG:teuthology.parallel:result is None 2024-04-20T17:07:12.973 INFO:teuthology.orchestra.run.smithi005.stdout:206158430251 2024-04-20T17:07:13.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:12 smithi005 ceph-mon[39562]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:13.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:12 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/160795527' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-20T17:07:13.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:12 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1483061284' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-20T17:07:13.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:12 smithi106 ceph-mon[44423]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:13.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:12 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/160795527' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-20T17:07:13.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:12 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1483061284' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-20T17:07:13.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:12 smithi171 ceph-mon[44189]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:13.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:12 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/160795527' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-20T17:07:13.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:12 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1483061284' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-20T17:07:13.602 INFO:tasks.cephadm.ceph_manager.ceph:need seq 395136991244 got 395136991249 for osd.11 2024-04-20T17:07:13.602 DEBUG:teuthology.parallel:result is None 2024-04-20T17:07:13.926 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612633 got 313532612636 for osd.9 2024-04-20T17:07:13.926 DEBUG:teuthology.parallel:result is None 2024-04-20T17:07:14.592 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:07:14.593 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:07:14.593 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:07:15.041 INFO:teuthology.orchestra.run.smithi005.stdout:107374182459 2024-04-20T17:07:15.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:14 smithi106 ceph-mon[44423]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:15.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:14 smithi171 ceph-mon[44189]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:15.352 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:14 smithi005 ceph-mon[39562]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:15.378 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430248 got 206158430251 for osd.6 2024-04-20T17:07:15.378 DEBUG:teuthology.parallel:result is None 2024-04-20T17:07:16.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:15 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/610661031' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-20T17:07:16.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:15 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/610661031' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-20T17:07:16.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:15 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/610661031' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-20T17:07:17.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:17 smithi106 ceph-mon[44423]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:17.305 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:17 smithi005 ceph-mon[39562]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:17.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:17 smithi171 ceph-mon[44189]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:17.596 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182453 got 107374182459 for osd.3 2024-04-20T17:07:17.596 DEBUG:teuthology.parallel:result is None 2024-04-20T17:07:18.047 INFO:teuthology.orchestra.run.smithi005.stdout:34359738445 2024-04-20T17:07:18.530 INFO:teuthology.orchestra.run.smithi005.stdout:133143986230 2024-04-20T17:07:18.780 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:07:19.254 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:19 smithi005 ceph-mon[39562]: pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:19.254 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:19 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3055061058' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-20T17:07:19.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:19 smithi106 ceph-mon[44423]: pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:19.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:19 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3055061058' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-20T17:07:19.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:19 smithi171 ceph-mon[44189]: pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:19 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3055061058' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-20T17:07:20.154 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738437 got 34359738445 for osd.0 2024-04-20T17:07:20.155 DEBUG:teuthology.parallel:result is None 2024-04-20T17:07:20.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:20 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:07:20.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:20 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:07:20.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:20 smithi106 ceph-mon[44423]: osdmap e106: 12 total, 12 up, 12 in 2024-04-20T17:07:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:20 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:07:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:20 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:07:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:20 smithi171 ceph-mon[44189]: osdmap e106: 12 total, 12 up, 12 in 2024-04-20T17:07:20.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:20 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:07:20.388 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:20 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:07:20.388 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:20 smithi005 ceph-mon[39562]: osdmap e106: 12 total, 12 up, 12 in 2024-04-20T17:07:20.912 INFO:teuthology.orchestra.run.smithi005.stdout:240518168617 2024-04-20T17:07:21.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:21 smithi005 ceph-mon[39562]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:21.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:21 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:07:21.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:21 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:07:21.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:21 smithi005 ceph-mon[39562]: osdmap e107: 12 total, 12 up, 12 in 2024-04-20T17:07:21.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:21 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3780138518' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-20T17:07:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:21 smithi171 ceph-mon[44189]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:21 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:07:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:21 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:07:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:21 smithi171 ceph-mon[44189]: osdmap e107: 12 total, 12 up, 12 in 2024-04-20T17:07:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:21 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3780138518' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-20T17:07:21.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:21 smithi106 ceph-mon[44423]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:21.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:21 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:07:21.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:21 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:07:21.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:21 smithi106 ceph-mon[44423]: osdmap e107: 12 total, 12 up, 12 in 2024-04-20T17:07:21.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:21 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3780138518' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-20T17:07:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:22 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:07:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:22 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:07:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:22 smithi171 ceph-mon[44189]: osdmap e108: 12 total, 12 up, 12 in 2024-04-20T17:07:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:22 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4155560207' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-20T17:07:22.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:22 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:07:22.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:22 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:07:22.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:22 smithi005 ceph-mon[39562]: osdmap e108: 12 total, 12 up, 12 in 2024-04-20T17:07:22.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:22 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4155560207' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-20T17:07:22.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:22 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:07:22.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:22 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:07:22.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:22 smithi106 ceph-mon[44423]: osdmap e108: 12 total, 12 up, 12 in 2024-04-20T17:07:22.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:22 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4155560207' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-20T17:07:22.531 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986225 got 133143986230 for osd.4 2024-04-20T17:07:22.531 DEBUG:teuthology.parallel:result is None 2024-04-20T17:07:23.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:23 smithi005 ceph-mon[39562]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:23.271 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168612 got 240518168617 for osd.7 2024-04-20T17:07:23.271 DEBUG:teuthology.parallel:result is None 2024-04-20T17:07:23.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:23 smithi171 ceph-mon[44189]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:23.360 INFO:teuthology.orchestra.run.smithi005.stdout:158913790001 2024-04-20T17:07:23.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:23 smithi106 ceph-mon[44423]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:24.082 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789997 got 158913790001 for osd.5 2024-04-20T17:07:24.097 DEBUG:teuthology.parallel:result is None 2024-04-20T17:07:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:24 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:07:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:24 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:07:24.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:24 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:07:24.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:24 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:07:24.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:24 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:07:24.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:24 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:07:25.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:25 smithi171 ceph-mon[44189]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:25.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:25 smithi171 ceph-mon[44189]: osdmap e109: 12 total, 12 up, 12 in 2024-04-20T17:07:25.344 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:25 smithi005 ceph-mon[39562]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:25.344 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:25 smithi005 ceph-mon[39562]: osdmap e109: 12 total, 12 up, 12 in 2024-04-20T17:07:25.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:25 smithi106 ceph-mon[44423]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:25 smithi106 ceph-mon[44423]: osdmap e109: 12 total, 12 up, 12 in 2024-04-20T17:07:26.212 INFO:teuthology.orchestra.run.smithi005.stdout:55834574920 2024-04-20T17:07:27.000 INFO:teuthology.orchestra.run.smithi005.stdout:85899345986 2024-04-20T17:07:27.032 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:26 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2678062497' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-20T17:07:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:26 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2678062497' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-20T17:07:27.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:26 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2678062497' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-20T17:07:27.430 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574914 got 55834574920 for osd.1 2024-04-20T17:07:27.430 DEBUG:teuthology.parallel:result is None 2024-04-20T17:07:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:27 smithi171 ceph-mon[44189]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:27 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1935933457' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-20T17:07:28.160 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:27 smithi005 ceph-mon[39562]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:28.160 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:27 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1935933457' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-20T17:07:28.191 INFO:tasks.cephadm.ceph_manager.ceph:need seq 85899345979 got 85899345986 for osd.2 2024-04-20T17:07:28.191 DEBUG:teuthology.parallel:result is None 2024-04-20T17:07:28.192 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-20T17:07:28.192 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:07:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:27 smithi106 ceph-mon[44423]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:27 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1935933457' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-20T17:07:28.472 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:07:28.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:28 smithi005 ceph-mon[39562]: pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:29.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:28 smithi171 ceph-mon[44189]: pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:28 smithi106 ceph-mon[44423]: pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:30.904 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:07:30.906 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:07:31.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:30 smithi005 ceph-mon[39562]: pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:31.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:30 smithi106 ceph-mon[44423]: pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:30 smithi171 ceph-mon[44189]: pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:31.673 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":341,"stamp":"2024-04-20T17:07:29.902390+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":345480,"kb_used_data":8576,"kb_used_omap":18,"kb_used_meta":336557,"kb_avail":1124596344,"statfs":{"total":1151940427776,"available":1151586656256,"internally_reserved":0,"allocated":8781824,"data_stored":4461352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":344634681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.877156"},"pg_stats":[{"pgid":"1.0","version":"15'137","reported_seq":361,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-20T17:07:24.977497+0000","last_change":"2024-04-20T17:06:51.212018+0000","last_active":"2024-04-20T17:07:24.977497+0000","last_peered":"2024-04-20T17:07:24.977497+0000","last_clean":"2024-04-20T17:07:24.977497+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:07:24.977497+0000","last_undegraded":"2024-04-20T17:07:24.977497+0000","last_fullsized":"2024-04-20T17:07:24.977497+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:06:51.211902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:06:51.211902+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:08:20.549123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991253,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1175552,"data_stored":812536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777252890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31808,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713344,"statfs":{"total":95995035648,"available":95962464256,"internally_reserved":0,"allocated":643072,"data_stored":283608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612640,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717504,"statfs":{"total":95995035648,"available":95966724096,"internally_reserved":0,"allocated":643072,"data_stored":283628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762808870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717504,"statfs":{"total":95995035648,"available":95966724096,"internally_reserved":0,"allocated":643072,"data_stored":283628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168619,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1175552,"data_stored":812556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430255,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717504,"statfs":{"total":95995035648,"available":95966724096,"internally_reserved":0,"allocated":643072,"data_stored":283628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574921,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717504,"statfs":{"total":95995035648,"available":95966724096,"internally_reserved":0,"allocated":643072,"data_stored":283628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31808,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713344,"statfs":{"total":95995035648,"available":95962464256,"internally_reserved":0,"allocated":643072,"data_stored":283628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":20,"seq":85899345987,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717504,"statfs":{"total":95995035648,"available":95966724096,"internally_reserved":0,"allocated":643072,"data_stored":283628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182462,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31900,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713252,"statfs":{"total":95995035648,"available":95962370048,"internally_reserved":0,"allocated":643072,"data_stored":283628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986233,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717504,"statfs":{"total":95995035648,"available":95966724096,"internally_reserved":0,"allocated":643072,"data_stored":283628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790003,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27740,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717412,"statfs":{"total":95995035648,"available":95966629888,"internally_reserved":0,"allocated":643072,"data_stored":283628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:07:31.674 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:07:31.954 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:07:32.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:31 smithi005 ceph-mon[39562]: from='client.14721 v1:172.21.15.5:0/1965563708' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:07:32.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:31 smithi106 ceph-mon[44423]: from='client.14721 v1:172.21.15.5:0/1965563708' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:07:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:31 smithi171 ceph-mon[44189]: from='client.14721 v1:172.21.15.5:0/1965563708' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:07:33.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:32 smithi005 ceph-mon[39562]: pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:33.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:32 smithi106 ceph-mon[44423]: pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:32 smithi171 ceph-mon[44189]: pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:34.136 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:07:34.138 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:07:34.904 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-04-20T17:07:33.966548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":345480,"kb_used_data":8576,"kb_used_omap":18,"kb_used_meta":336557,"kb_avail":1124596344,"statfs":{"total":1151940427776,"available":1151586656256,"internally_reserved":0,"allocated":8781824,"data_stored":4461352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":344634681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.930419"},"pg_stats":[{"pgid":"1.0","version":"15'137","reported_seq":361,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-20T17:07:24.977497+0000","last_change":"2024-04-20T17:06:51.212018+0000","last_active":"2024-04-20T17:07:24.977497+0000","last_peered":"2024-04-20T17:07:24.977497+0000","last_clean":"2024-04-20T17:07:24.977497+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:07:24.977497+0000","last_undegraded":"2024-04-20T17:07:24.977497+0000","last_fullsized":"2024-04-20T17:07:24.977497+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:06:51.211902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:06:51.211902+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:08:20.549123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991254,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1175552,"data_stored":812536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777252891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31808,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713344,"statfs":{"total":95995035648,"available":95962464256,"internally_reserved":0,"allocated":643072,"data_stored":283608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612641,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717504,"statfs":{"total":95995035648,"available":95966724096,"internally_reserved":0,"allocated":643072,"data_stored":283628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762808871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717504,"statfs":{"total":95995035648,"available":95966724096,"internally_reserved":0,"allocated":643072,"data_stored":283628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168620,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1175552,"data_stored":812556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430256,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717504,"statfs":{"total":95995035648,"available":95966724096,"internally_reserved":0,"allocated":643072,"data_stored":283628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717504,"statfs":{"total":95995035648,"available":95966724096,"internally_reserved":0,"allocated":643072,"data_stored":283628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31808,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713344,"statfs":{"total":95995035648,"available":95962464256,"internally_reserved":0,"allocated":643072,"data_stored":283628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":20,"seq":85899345988,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717504,"statfs":{"total":95995035648,"available":95966724096,"internally_reserved":0,"allocated":643072,"data_stored":283628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182463,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31900,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713252,"statfs":{"total":95995035648,"available":95962370048,"internally_reserved":0,"allocated":643072,"data_stored":283628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986234,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717504,"statfs":{"total":95995035648,"available":95966724096,"internally_reserved":0,"allocated":643072,"data_stored":283628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27740,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717412,"statfs":{"total":95995035648,"available":95966629888,"internally_reserved":0,"allocated":643072,"data_stored":283628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:07:34.905 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-20T17:07:34.905 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-20T17:07:34.905 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-20T17:07:34.905 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph health --format=json 2024-04-20T17:07:35.184 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:07:35.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:34 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:07:35.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:34 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:07:35.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:34 smithi005 ceph-mon[39562]: pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:35.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:34 smithi005 ceph-mon[39562]: osdmap e110: 12 total, 12 up, 12 in 2024-04-20T17:07:35.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:34 smithi005 ceph-mon[39562]: from='client.14727 v1:172.21.15.5:0/1575718392' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:07:35.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:34 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:07:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:34 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:07:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:34 smithi106 ceph-mon[44423]: pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:34 smithi106 ceph-mon[44423]: osdmap e110: 12 total, 12 up, 12 in 2024-04-20T17:07:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:34 smithi106 ceph-mon[44423]: from='client.14727 v1:172.21.15.5:0/1575718392' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:07:35.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:34 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:07:35.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:34 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:07:35.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:34 smithi171 ceph-mon[44189]: pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:35.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:34 smithi171 ceph-mon[44189]: osdmap e110: 12 total, 12 up, 12 in 2024-04-20T17:07:35.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:34 smithi171 ceph-mon[44189]: from='client.14727 v1:172.21.15.5:0/1575718392' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:07:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:36 smithi005 ceph-mon[39562]: pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:36 smithi106 ceph-mon[44423]: pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:37.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:36 smithi171 ceph-mon[44189]: pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:37.443 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:07:37.443 INFO:teuthology.orchestra.run.smithi005.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-20T17:07:38.093 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:37 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3218409772' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-20T17:07:38.120 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-20T17:07:38.120 INFO:tasks.cephadm:Setup complete, yielding 2024-04-20T17:07:38.121 INFO:teuthology.run_tasks:Running task exec... 2024-04-20T17:07:38.131 INFO:teuthology.task.exec:Executing custom commands... 2024-04-20T17:07:38.131 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi005.front.sepia.ceph.com 2024-04-20T17:07:38.131 DEBUG:teuthology.orchestra.run.smithi005:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'while ! ceph balancer status ; do sleep 1 ; done' 2024-04-20T17:07:38.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:37 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3218409772' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-20T17:07:38.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:37 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3218409772' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-20T17:07:38.707 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:07:38.707 INFO:teuthology.orchestra.run.smithi005.stdout: "active": true, 2024-04-20T17:07:38.707 INFO:teuthology.orchestra.run.smithi005.stdout: "last_optimize_duration": "0:00:00.000370", 2024-04-20T17:07:38.707 INFO:teuthology.orchestra.run.smithi005.stdout: "last_optimize_started": "Sat Apr 20 17:07:25 2024", 2024-04-20T17:07:38.707 INFO:teuthology.orchestra.run.smithi005.stdout: "mode": "upmap", 2024-04-20T17:07:38.707 INFO:teuthology.orchestra.run.smithi005.stdout: "no_optimization_needed": true, 2024-04-20T17:07:38.707 INFO:teuthology.orchestra.run.smithi005.stdout: "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", 2024-04-20T17:07:38.707 INFO:teuthology.orchestra.run.smithi005.stdout: "plans": [] 2024-04-20T17:07:38.707 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:07:38.721 DEBUG:teuthology.orchestra.run.smithi005:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer mode crush-compat' 2024-04-20T17:07:39.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:38 smithi106 ceph-mon[44423]: pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:39.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:38 smithi106 ceph-mon[44423]: from='client.14739 v1:172.21.15.5:0/683320508' entity='client.admin' cmd=[{"prefix": "balancer status", "target": ["mgr", ""]}]: dispatch 2024-04-20T17:07:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:38 smithi171 ceph-mon[44189]: pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:38 smithi171 ceph-mon[44189]: from='client.14739 v1:172.21.15.5:0/683320508' entity='client.admin' cmd=[{"prefix": "balancer status", "target": ["mgr", ""]}]: dispatch 2024-04-20T17:07:39.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:38 smithi005 ceph-mon[39562]: pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:39.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:38 smithi005 ceph-mon[39562]: from='client.14739 v1:172.21.15.5:0/683320508' entity='client.admin' cmd=[{"prefix": "balancer status", "target": ["mgr", ""]}]: dispatch 2024-04-20T17:07:40.014 DEBUG:teuthology.orchestra.run.smithi005:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer on' 2024-04-20T17:07:40.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:39 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:07:40.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:39 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:07:40.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:39 smithi106 ceph-mon[44423]: osdmap e111: 12 total, 12 up, 12 in 2024-04-20T17:07:40.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:39 smithi106 ceph-mon[44423]: from='client.14745 v1:172.21.15.5:0/2216353420' entity='client.admin' cmd=[{"prefix": "balancer mode", "mode": "crush-compat", "target": ["mgr", ""]}]: dispatch 2024-04-20T17:07:40.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:39 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set create-compat", "format": "json"}]: dispatch 2024-04-20T17:07:40.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:39 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:07:40.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:39 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:07:40.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:39 smithi171 ceph-mon[44189]: osdmap e111: 12 total, 12 up, 12 in 2024-04-20T17:07:40.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:39 smithi171 ceph-mon[44189]: from='client.14745 v1:172.21.15.5:0/2216353420' entity='client.admin' cmd=[{"prefix": "balancer mode", "mode": "crush-compat", "target": ["mgr", ""]}]: dispatch 2024-04-20T17:07:40.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:39 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set create-compat", "format": "json"}]: dispatch 2024-04-20T17:07:40.406 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:39 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:07:40.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:39 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:07:40.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:39 smithi005 ceph-mon[39562]: osdmap e111: 12 total, 12 up, 12 in 2024-04-20T17:07:40.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:39 smithi005 ceph-mon[39562]: from='client.14745 v1:172.21.15.5:0/2216353420' entity='client.admin' cmd=[{"prefix": "balancer mode", "mode": "crush-compat", "target": ["mgr", ""]}]: dispatch 2024-04-20T17:07:40.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:39 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set create-compat", "format": "json"}]: dispatch 2024-04-20T17:07:40.408 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-20T17:07:40.420 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 0.25, 'chance_pgpnum_fix': 0.25, 'chance_test_map_discontinuity': 2, 'timeout': 1800, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-20T17:07:40.420 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-yuriw-testing-20240419.185239-main'}, 'ceph': {'conf': {'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd heartbeat use min delay socket': True, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'mon_bind_msgr2': False, 'sha1': '36c371567dddacf6207ea36f2535396ab31415fc'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '36c371567dddacf6207ea36f2535396ab31415fc'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-yuriw-testing-20240419.185239-main', 'sha1': '36c371567dddacf6207ea36f2535396ab31415fc'}} 2024-04-20T17:07:40.420 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 0.25, 'chance_pgpnum_fix': 0.25, 'chance_test_map_discontinuity': 2, 'timeout': 1800, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-20T17:07:40.420 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-04-20T17:07:40.691 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:07:41.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:40 smithi005 ceph-mon[39562]: pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:41.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:40 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set create-compat", "format": "json"}]': finished 2024-04-20T17:07:41.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:40 smithi005 ceph-mon[39562]: osdmap e112: 12 total, 12 up, 12 in 2024-04-20T17:07:41.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:40 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush dump", "format": "json"}]: dispatch 2024-04-20T17:07:41.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:40 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:07:41.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:40 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:07:41.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:40 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:07:41.156 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:40 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:07:41.156 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:40 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:07:41.156 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:40 smithi005 ceph-mon[39562]: from='client.14751 v1:172.21.15.5:0/2373781174' entity='client.admin' cmd=[{"prefix": "balancer on", "target": ["mgr", ""]}]: dispatch 2024-04-20T17:07:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:40 smithi106 ceph-mon[44423]: pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:40 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set create-compat", "format": "json"}]': finished 2024-04-20T17:07:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:40 smithi106 ceph-mon[44423]: osdmap e112: 12 total, 12 up, 12 in 2024-04-20T17:07:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:40 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush dump", "format": "json"}]: dispatch 2024-04-20T17:07:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:40 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:07:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:40 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:07:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:40 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:07:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:40 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:07:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:40 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:07:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:40 smithi106 ceph-mon[44423]: from='client.14751 v1:172.21.15.5:0/2373781174' entity='client.admin' cmd=[{"prefix": "balancer on", "target": ["mgr", ""]}]: dispatch 2024-04-20T17:07:41.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:40 smithi171 ceph-mon[44189]: pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:40 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set create-compat", "format": "json"}]': finished 2024-04-20T17:07:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:40 smithi171 ceph-mon[44189]: osdmap e112: 12 total, 12 up, 12 in 2024-04-20T17:07:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:40 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush dump", "format": "json"}]: dispatch 2024-04-20T17:07:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:40 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:07:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:40 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:07:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:40 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:07:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:40 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:07:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:40 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:07:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:40 smithi171 ceph-mon[44189]: from='client.14751 v1:172.21.15.5:0/2373781174' entity='client.admin' cmd=[{"prefix": "balancer on", "target": ["mgr", ""]}]: dispatch 2024-04-20T17:07:43.292 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-20T17:07:43.292 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-20T17:07:43.293 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:07:43.570 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:07:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:43 smithi171 ceph-mon[44189]: pgmap v350: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:43 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/931778216' entity='client.admin' 2024-04-20T17:07:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:07:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:07:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:07:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:07:43.878 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:43 smithi005 ceph-mon[39562]: pgmap v350: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:43.879 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:43 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/931778216' entity='client.admin' 2024-04-20T17:07:43.879 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:07:43.879 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:07:43.879 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:07:43.879 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:07:44.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:43 smithi106 ceph-mon[44423]: pgmap v350: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:44.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:43 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/931778216' entity='client.admin' 2024-04-20T17:07:44.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:07:44.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:07:44.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:07:44.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:07:45.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:45 smithi005 ceph-mon[39562]: pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:45.819 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:07:45.821 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:07:45.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:45 smithi171 ceph-mon[44189]: pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:46.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:45 smithi106 ceph-mon[44423]: pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:46.491 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-04-20T17:07:43.905676+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":345688,"kb_used_data":8948,"kb_used_omap":18,"kb_used_meta":336557,"kb_avail":1124596136,"statfs":{"total":1151940427776,"available":1151586443264,"internally_reserved":0,"allocated":9162752,"data_stored":4648701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":344634681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001720"},"pg_stats":[{"pgid":"1.0","version":"15'137","reported_seq":367,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-20T17:07:40.159268+0000","last_change":"2024-04-20T17:06:51.212018+0000","last_active":"2024-04-20T17:07:40.159268+0000","last_peered":"2024-04-20T17:07:40.159268+0000","last_clean":"2024-04-20T17:07:40.159268+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:07:40.159268+0000","last_undegraded":"2024-04-20T17:07:40.159268+0000","last_fullsized":"2024-04-20T17:07:40.159268+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:06:51.211902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:06:51.211902+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:08:20.549123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991256,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28204,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716948,"statfs":{"total":95995035648,"available":95966154752,"internally_reserved":0,"allocated":1212416,"data_stored":831045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777252893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31848,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713304,"statfs":{"total":95995035648,"available":95962423296,"internally_reserved":0,"allocated":679936,"data_stored":302117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612643,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27672,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717480,"statfs":{"total":95995035648,"available":95966699520,"internally_reserved":0,"allocated":667648,"data_stored":294850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":67,"seq":287762808873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":679936,"data_stored":302137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168622,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28208,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716944,"statfs":{"total":95995035648,"available":95966150656,"internally_reserved":0,"allocated":1212416,"data_stored":831065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430258,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":679936,"data_stored":302137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717468,"statfs":{"total":95995035648,"available":95966687232,"internally_reserved":0,"allocated":679936,"data_stored":302137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31820,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713332,"statfs":{"total":95995035648,"available":95962451968,"internally_reserved":0,"allocated":655360,"data_stored":289239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":20,"seq":85899345990,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717468,"statfs":{"total":95995035648,"available":95966687232,"internally_reserved":0,"allocated":679936,"data_stored":302137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182465,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31832,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713320,"statfs":{"total":95995035648,"available":95962439680,"internally_reserved":0,"allocated":667648,"data_stored":294850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986236,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27672,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717480,"statfs":{"total":95995035648,"available":95966699520,"internally_reserved":0,"allocated":667648,"data_stored":294850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790006,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":679936,"data_stored":302137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:07:46.492 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:07:46.768 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:07:47.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:47 smithi005 ceph-mon[39562]: from='client.14763 v1:172.21.15.5:0/4020657298' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:07:47.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:47 smithi005 ceph-mon[39562]: pgmap v352: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:48.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:47 smithi171 ceph-mon[44189]: from='client.14763 v1:172.21.15.5:0/4020657298' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:07:48.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:47 smithi171 ceph-mon[44189]: pgmap v352: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:48.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:47 smithi106 ceph-mon[44423]: from='client.14763 v1:172.21.15.5:0/4020657298' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:07:48.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:47 smithi106 ceph-mon[44423]: pgmap v352: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:48.693 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:07:48.695 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:07:49.423 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":353,"stamp":"2024-04-20T17:07:47.906415+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":345760,"kb_used_data":9008,"kb_used_omap":18,"kb_used_meta":336557,"kb_avail":1124596064,"statfs":{"total":1151940427776,"available":1151586369536,"internally_reserved":0,"allocated":9224192,"data_stored":4683460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":344634681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.925991"},"pg_stats":[{"pgid":"1.0","version":"15'137","reported_seq":367,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-04-20T17:07:40.159268+0000","last_change":"2024-04-20T17:06:51.212018+0000","last_active":"2024-04-20T17:07:40.159268+0000","last_peered":"2024-04-20T17:07:40.159268+0000","last_clean":"2024-04-20T17:07:40.159268+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:07:40.159268+0000","last_undegraded":"2024-04-20T17:07:40.159268+0000","last_fullsized":"2024-04-20T17:07:40.159268+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:06:51.211902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:06:51.211902+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:08:20.549123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991257,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28204,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716948,"statfs":{"total":95995035648,"available":95966154752,"internally_reserved":0,"allocated":1212416,"data_stored":831045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777252893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31848,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713304,"statfs":{"total":95995035648,"available":95962423296,"internally_reserved":0,"allocated":679936,"data_stored":302117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612644,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":679936,"data_stored":302137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762808873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":679936,"data_stored":302137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168623,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28208,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716944,"statfs":{"total":95995035648,"available":95966150656,"internally_reserved":0,"allocated":1212416,"data_stored":831065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":679936,"data_stored":302137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717468,"statfs":{"total":95995035648,"available":95966687232,"internally_reserved":0,"allocated":679936,"data_stored":302137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31848,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713304,"statfs":{"total":95995035648,"available":95962423296,"internally_reserved":0,"allocated":679936,"data_stored":302137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":20,"seq":85899345990,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717468,"statfs":{"total":95995035648,"available":95966687232,"internally_reserved":0,"allocated":679936,"data_stored":302137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182466,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31844,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713308,"statfs":{"total":95995035648,"available":95962427392,"internally_reserved":0,"allocated":679936,"data_stored":302137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986237,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":679936,"data_stored":302137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790007,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":679936,"data_stored":302137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:07:49.423 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-20T17:07:49.424 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd dump 2024-04-20T17:07:49.703 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:07:50.041 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:49 smithi005 ceph-mon[39562]: pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:50.041 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:49 smithi005 ceph-mon[39562]: from='client.14769 v1:172.21.15.5:0/3824059181' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:07:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:49 smithi171 ceph-mon[44189]: pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:49 smithi171 ceph-mon[44189]: from='client.14769 v1:172.21.15.5:0/3824059181' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:07:50.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:49 smithi106 ceph-mon[44423]: pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:49 smithi106 ceph-mon[44423]: from='client.14769 v1:172.21.15.5:0/3824059181' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:07:51.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:50 smithi171 ceph-mon[44189]: pgmap v354: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:51.175 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:50 smithi005 ceph-mon[39562]: pgmap v354: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:50 smithi106 ceph-mon[44423]: pgmap v354: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:52.002 INFO:teuthology.orchestra.run.smithi005.stdout:epoch 112 2024-04-20T17:07:52.003 INFO:teuthology.orchestra.run.smithi005.stdout:fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T17:07:52.003 INFO:teuthology.orchestra.run.smithi005.stdout:created 2024-04-20T16:58:25.272557+0000 2024-04-20T17:07:52.003 INFO:teuthology.orchestra.run.smithi005.stdout:modified 2024-04-20T17:07:39.980806+0000 2024-04-20T17:07:52.003 INFO:teuthology.orchestra.run.smithi005.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-20T17:07:52.003 INFO:teuthology.orchestra.run.smithi005.stdout:crush_version 27 2024-04-20T17:07:52.003 INFO:teuthology.orchestra.run.smithi005.stdout:full_ratio 0.95 2024-04-20T17:07:52.003 INFO:teuthology.orchestra.run.smithi005.stdout:backfillfull_ratio 0.9 2024-04-20T17:07:52.003 INFO:teuthology.orchestra.run.smithi005.stdout:nearfull_ratio 0.85 2024-04-20T17:07:52.003 INFO:teuthology.orchestra.run.smithi005.stdout:require_min_compat_client luminous 2024-04-20T17:07:52.003 INFO:teuthology.orchestra.run.smithi005.stdout:min_compat_client jewel 2024-04-20T17:07:52.003 INFO:teuthology.orchestra.run.smithi005.stdout:require_osd_release squid 2024-04-20T17:07:52.003 INFO:teuthology.orchestra.run.smithi005.stdout:stretch_mode_enabled false 2024-04-20T17:07:52.004 INFO:teuthology.orchestra.run.smithi005.stdout:pool 1 '.mgr' replicated size 2 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 16 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 11.76 2024-04-20T17:07:52.004 INFO:teuthology.orchestra.run.smithi005.stdout:max_osd 12 2024-04-20T17:07:52.004 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 up in weight 1 up_from 8 up_thru 58 down_at 0 last_clean_interval [0,0) v1:172.21.15.5:6801/3901484705 v1:172.21.15.5:6802/3901484705 exists,up ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b 2024-04-20T17:07:52.004 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) v1:172.21.15.5:6805/3391893914 v1:172.21.15.5:6806/3391893914 exists,up aed0c8eb-0e81-4e85-ab8f-a60420060656 2024-04-20T17:07:52.004 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.5:6809/2105468358 v1:172.21.15.5:6810/2105468358 exists,up f733a6a2-19c2-4c7a-af9c-04e282062bb7 2024-04-20T17:07:52.004 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.5:6813/377652039 v1:172.21.15.5:6814/377652039 exists,up 8c92f1df-e298-4606-88ae-b8a8e79628a1 2024-04-20T17:07:52.004 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.106:6800/1839524469 v1:172.21.15.106:6801/1839524469 exists,up 46df16f1-9607-4d7b-9be8-9d4236e160d1 2024-04-20T17:07:52.004 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 up in weight 1 up_from 37 up_thru 40 down_at 0 last_clean_interval [0,0) v1:172.21.15.106:6804/2450315908 v1:172.21.15.106:6805/2450315908 exists,up 6603231d-1e5c-4575-bfaa-2eeca2c12215 2024-04-20T17:07:52.004 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 up in weight 1 up_from 48 up_thru 51 down_at 0 last_clean_interval [0,0) v1:172.21.15.106:6808/2840237913 v1:172.21.15.106:6809/2840237913 exists,up 783a55a6-f176-42be-8c8b-af8481d08dc9 2024-04-20T17:07:52.004 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 up in weight 1 up_from 56 up_thru 93 down_at 0 last_clean_interval [0,0) v1:172.21.15.106:6812/1105371883 v1:172.21.15.106:6813/1105371883 exists,up b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4 2024-04-20T17:07:52.004 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8 up in weight 1 up_from 67 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.171:6800/4123058942 v1:172.21.15.171:6801/4123058942 exists,up 1cb232ad-1f59-4a0a-a215-e67419ad6476 2024-04-20T17:07:52.004 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9 up in weight 1 up_from 73 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.171:6804/2877589636 v1:172.21.15.171:6805/2877589636 exists,up b6c17147-b3dc-475b-bb20-f9f712fb99e8 2024-04-20T17:07:52.004 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10 up in weight 1 up_from 84 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.171:6808/181490621 v1:172.21.15.171:6809/181490621 exists,up 1adcdf55-f002-4a39-91e7-378e5fae849c 2024-04-20T17:07:52.005 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11 up in weight 1 up_from 92 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.171:6812/3373585701 v1:172.21.15.171:6813/3373585701 exists,up 737c5aec-694a-4968-819d-fccf22661f4f 2024-04-20T17:07:52.005 INFO:teuthology.orchestra.run.smithi005.stdout:blocklist 172.21.15.5:0/3881726425 expires 2024-04-21T16:59:25.808337+0000 2024-04-20T17:07:52.005 INFO:teuthology.orchestra.run.smithi005.stdout:blocklist 172.21.15.5:0/338649532 expires 2024-04-21T16:59:25.808337+0000 2024-04-20T17:07:52.005 INFO:teuthology.orchestra.run.smithi005.stdout:blocklist 172.21.15.5:0/1227843664 expires 2024-04-21T16:58:55.392162+0000 2024-04-20T17:07:52.005 INFO:teuthology.orchestra.run.smithi005.stdout:blocklist 172.21.15.5:6800/1876064631 expires 2024-04-21T16:58:55.392162+0000 2024-04-20T17:07:52.005 INFO:teuthology.orchestra.run.smithi005.stdout:blocklist 172.21.15.5:0/160689449 expires 2024-04-21T16:58:55.392162+0000 2024-04-20T17:07:52.005 INFO:teuthology.orchestra.run.smithi005.stdout:blocklist 172.21.15.5:6800/3521715286 expires 2024-04-21T16:59:25.808337+0000 2024-04-20T17:07:52.005 INFO:teuthology.orchestra.run.smithi005.stdout:blocklist 172.21.15.5:0/3841663462 expires 2024-04-21T16:58:55.392162+0000 2024-04-20T17:07:52.005 INFO:teuthology.orchestra.run.smithi005.stdout:blocklist 172.21.15.5:0/687690919 expires 2024-04-21T16:59:25.808337+0000 2024-04-20T17:07:52.833 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 58 down_at 0 last_clean_interval [0,0) v1:172.21.15.5:6801/3901484705 v1:172.21.15.5:6802/3901484705 exists,up ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b', 'osd.1 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) v1:172.21.15.5:6805/3391893914 v1:172.21.15.5:6806/3391893914 exists,up aed0c8eb-0e81-4e85-ab8f-a60420060656', 'osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.5:6809/2105468358 v1:172.21.15.5:6810/2105468358 exists,up f733a6a2-19c2-4c7a-af9c-04e282062bb7', 'osd.3 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.5:6813/377652039 v1:172.21.15.5:6814/377652039 exists,up 8c92f1df-e298-4606-88ae-b8a8e79628a1', 'osd.4 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.106:6800/1839524469 v1:172.21.15.106:6801/1839524469 exists,up 46df16f1-9607-4d7b-9be8-9d4236e160d1', 'osd.5 up in weight 1 up_from 37 up_thru 40 down_at 0 last_clean_interval [0,0) v1:172.21.15.106:6804/2450315908 v1:172.21.15.106:6805/2450315908 exists,up 6603231d-1e5c-4575-bfaa-2eeca2c12215', 'osd.6 up in weight 1 up_from 48 up_thru 51 down_at 0 last_clean_interval [0,0) v1:172.21.15.106:6808/2840237913 v1:172.21.15.106:6809/2840237913 exists,up 783a55a6-f176-42be-8c8b-af8481d08dc9', 'osd.7 up in weight 1 up_from 56 up_thru 93 down_at 0 last_clean_interval [0,0) v1:172.21.15.106:6812/1105371883 v1:172.21.15.106:6813/1105371883 exists,up b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4', 'osd.8 up in weight 1 up_from 67 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.171:6800/4123058942 v1:172.21.15.171:6801/4123058942 exists,up 1cb232ad-1f59-4a0a-a215-e67419ad6476', 'osd.9 up in weight 1 up_from 73 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.171:6804/2877589636 v1:172.21.15.171:6805/2877589636 exists,up b6c17147-b3dc-475b-bb20-f9f712fb99e8', 'osd.10 up in weight 1 up_from 84 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.171:6808/181490621 v1:172.21.15.171:6809/181490621 exists,up 1adcdf55-f002-4a39-91e7-378e5fae849c', 'osd.11 up in weight 1 up_from 92 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.171:6812/3373585701 v1:172.21.15.171:6813/3373585701 exists,up 737c5aec-694a-4968-819d-fccf22661f4f'] 2024-04-20T17:07:52.833 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon mon.a config show 2024-04-20T17:07:53.112 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:07:53.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:52 smithi005 ceph-mon[39562]: pgmap v355: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:53.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:52 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3911589417' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-20T17:07:53.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:52 smithi106 ceph-mon[44423]: pgmap v355: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:53.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:52 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3911589417' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-20T17:07:53.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:52 smithi171 ceph-mon[44189]: pgmap v355: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:53.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:52 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3911589417' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-20T17:07:55.064 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:07:55.064 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "mon.a", 2024-04-20T17:07:55.064 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster": "ceph", 2024-04-20T17:07:55.064 INFO:teuthology.orchestra.run.smithi005.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-20T17:07:55.065 INFO:teuthology.orchestra.run.smithi005.stdout: "admin_socket_mode": "", 2024-04-20T17:07:55.065 INFO:teuthology.orchestra.run.smithi005.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-04-20T17:07:55.065 INFO:teuthology.orchestra.run.smithi005.stdout: "auth_client_required": "cephx, none", 2024-04-20T17:07:55.065 INFO:teuthology.orchestra.run.smithi005.stdout: "auth_cluster_required": "cephx", 2024-04-20T17:07:55.065 INFO:teuthology.orchestra.run.smithi005.stdout: "auth_debug": "true", 2024-04-20T17:07:55.065 INFO:teuthology.orchestra.run.smithi005.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-20T17:07:55.065 INFO:teuthology.orchestra.run.smithi005.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-20T17:07:55.065 INFO:teuthology.orchestra.run.smithi005.stdout: "auth_service_required": "cephx", 2024-04-20T17:07:55.065 INFO:teuthology.orchestra.run.smithi005.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-20T17:07:55.065 INFO:teuthology.orchestra.run.smithi005.stdout: "auth_supported": "", 2024-04-20T17:07:55.065 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_aio": "true", 2024-04-20T17:07:55.065 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-20T17:07:55.065 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_aio_poll_ms": "250", 2024-04-20T17:07:55.065 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_aio_reap_max": "16", 2024-04-20T17:07:55.066 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_async_discard_threads": "0", 2024-04-20T17:07:55.066 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_block_size": "4096", 2024-04-20T17:07:55.066 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_debug_aio": "false", 2024-04-20T17:07:55.066 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-20T17:07:55.066 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-20T17:07:55.066 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_debug_inflight_ios": "false", 2024-04-20T17:07:55.066 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_enable_discard": "false", 2024-04-20T17:07:55.066 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_flock_retry": "3", 2024-04-20T17:07:55.066 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-20T17:07:55.066 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_inject_crash": "0", 2024-04-20T17:07:55.066 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-20T17:07:55.066 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_ioring": "false", 2024-04-20T17:07:55.066 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_ioring_hipri": "false", 2024-04-20T17:07:55.066 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-20T17:07:55.067 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-20T17:07:55.067 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-20T17:07:55.067 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-20T17:07:55.067 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_type": "", 2024-04-20T17:07:55.067 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_alloc_size": "1048576", 2024-04-20T17:07:55.067 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_allocator": "hybrid", 2024-04-20T17:07:55.067 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_buffered_io": "true", 2024-04-20T17:07:55.067 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_check_for_zeros": "false", 2024-04-20T17:07:55.067 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-20T17:07:55.067 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-20T17:07:55.067 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_compact_log_sync": "false", 2024-04-20T17:07:55.067 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-20T17:07:55.067 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-20T17:07:55.068 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-20T17:07:55.068 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-20T17:07:55.068 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_max_log_runway": "4194304", 2024-04-20T17:07:55.068 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_max_prefetch": "1048576", 2024-04-20T17:07:55.068 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_min_flush_size": "524288", 2024-04-20T17:07:55.068 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_min_log_runway": "1048576", 2024-04-20T17:07:55.068 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_replay_recovery": "false", 2024-04-20T17:07:55.068 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-20T17:07:55.068 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-20T17:07:55.068 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_sync_write": "false", 2024-04-20T17:07:55.068 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-20T17:07:55.068 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-20T17:07:55.068 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-20T17:07:55.068 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_allocation_from_file": "true", 2024-04-20T17:07:55.069 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_allocator": "hybrid", 2024-04-20T17:07:55.069 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-20T17:07:55.069 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-20T17:07:55.069 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-20T17:07:55.069 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-20T17:07:55.069 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-20T17:07:55.069 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-20T17:07:55.069 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-20T17:07:55.069 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_create": "true", 2024-04-20T17:07:55.069 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_db_create": "false", 2024-04-20T17:07:55.069 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_db_path": "", 2024-04-20T17:07:55.069 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_db_size": "0", 2024-04-20T17:07:55.069 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_path": "", 2024-04-20T17:07:55.070 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_preallocate_file": "false", 2024-04-20T17:07:55.070 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_size": "107374182400", 2024-04-20T17:07:55.070 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_wal_create": "false", 2024-04-20T17:07:55.070 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_wal_path": "", 2024-04-20T17:07:55.070 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_wal_size": "100663296", 2024-04-20T17:07:55.070 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_bluefs": "true", 2024-04-20T17:07:55.070 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-20T17:07:55.070 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-20T17:07:55.070 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-20T17:07:55.070 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-20T17:07:55.070 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-20T17:07:55.070 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-20T17:07:55.070 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-20T17:07:55.071 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-20T17:07:55.071 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_autotune": "true", 2024-04-20T17:07:55.071 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-20T17:07:55.071 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-20T17:07:55.071 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-20T17:07:55.071 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-20T17:07:55.071 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_size": "0", 2024-04-20T17:07:55.071 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-20T17:07:55.071 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-20T17:07:55.071 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-20T17:07:55.071 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-20T17:07:55.071 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_type": "2q", 2024-04-20T17:07:55.071 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-20T17:07:55.071 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_clone_cow": "true", 2024-04-20T17:07:55.072 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-20T17:07:55.072 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-20T17:07:55.072 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-20T17:07:55.072 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-20T17:07:55.072 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-20T17:07:55.072 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-20T17:07:55.072 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-20T17:07:55.073 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_compression_mode": "none", 2024-04-20T17:07:55.073 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-20T17:07:55.073 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_csum_type": "crc32c", 2024-04-20T17:07:55.073 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-04-20T17:07:55.073 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-20T17:07:55.073 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_freelist": "false", 2024-04-20T17:07:55.073 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-20T17:07:55.073 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-20T17:07:55.074 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-20T17:07:55.074 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-20T17:07:55.074 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-20T17:07:55.074 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-20T17:07:55.074 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-20T17:07:55.074 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-20T17:07:55.074 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-20T17:07:55.074 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-20T17:07:55.074 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-20T17:07:55.074 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-20T17:07:55.074 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-20T17:07:55.074 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_small_allocations": "0", 2024-04-20T17:07:55.074 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-20T17:07:55.074 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_default_buffered_read": "true", 2024-04-20T17:07:55.075 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_default_buffered_write": "false", 2024-04-20T17:07:55.075 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-20T17:07:55.075 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-20T17:07:55.075 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-20T17:07:55.075 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_elastic_shared_blobs": "true", 2024-04-20T17:07:55.075 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-20T17:07:55.075 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-20T17:07:55.075 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-20T17:07:55.075 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-20T17:07:55.075 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-20T17:07:55.075 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fail_eio": "false", 2024-04-20T17:07:55.075 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-20T17:07:55.075 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-20T17:07:55.076 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-20T17:07:55.076 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-20T17:07:55.076 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-20T17:07:55.076 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-20T17:07:55.076 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_on_mount": "false", 2024-04-20T17:07:55.076 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-20T17:07:55.076 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_on_umount": "false", 2024-04-20T17:07:55.076 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-20T17:07:55.076 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-20T17:07:55.076 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-20T17:07:55.076 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-20T17:07:55.076 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-20T17:07:55.076 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-20T17:07:55.076 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-20T17:07:55.077 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-20T17:07:55.077 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_ignore_data_csum": "false", 2024-04-20T17:07:55.077 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-20T17:07:55.077 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-20T17:07:55.077 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-20T17:07:55.077 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-20T17:07:55.077 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_log_op_age": "5.000000", 2024-04-20T17:07:55.077 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_max_alloc_size": "0", 2024-04-20T17:07:55.077 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_max_blob_size": "0", 2024-04-20T17:07:55.077 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-20T17:07:55.077 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-20T17:07:55.077 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-20T17:07:55.077 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_max_deferred_txc": "32", 2024-04-20T17:07:55.077 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_min_alloc_size": "0", 2024-04-20T17:07:55.078 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-20T17:07:55.078 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-20T17:07:55.078 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_nid_prealloc": "1024", 2024-04-20T17:07:55.078 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-20T17:07:55.078 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-20T17:07:55.078 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-20T17:07:55.078 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-20T17:07:55.078 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_retry_disk_reads": "3", 2024-04-20T17:07:55.078 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_rocksdb_cf": "true", 2024-04-20T17:07:55.078 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-04-20T17:07:55.078 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-04-20T17:07:55.078 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-20T17:07:55.079 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-20T17:07:55.079 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-20T17:07:55.079 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-20T17:07:55.079 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_spdk_mem": "512", 2024-04-20T17:07:55.079 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-20T17:07:55.079 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-20T17:07:55.079 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-20T17:07:55.079 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-20T17:07:55.079 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-20T17:07:55.079 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-20T17:07:55.079 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-20T17:07:55.079 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_tracing": "false", 2024-04-20T17:07:55.079 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-20T17:07:55.079 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-20T17:07:55.080 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-20T17:07:55.080 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-20T17:07:55.080 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-20T17:07:55.080 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-20T17:07:55.080 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-20T17:07:55.080 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-20T17:07:55.080 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-20T17:07:55.080 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_zero_block_detection": "false", 2024-04-20T17:07:55.080 INFO:teuthology.orchestra.run.smithi005.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-20T17:07:55.080 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-20T17:07:55.080 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-20T17:07:55.080 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-20T17:07:55.080 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-20T17:07:55.080 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-20T17:07:55.081 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-20T17:07:55.081 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-04-20T17:07:55.081 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-20T17:07:55.081 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-20T17:07:55.081 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-20T17:07:55.081 INFO:teuthology.orchestra.run.smithi005.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-20T17:07:55.081 INFO:teuthology.orchestra.run.smithi005.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-20T17:07:55.081 INFO:teuthology.orchestra.run.smithi005.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-20T17:07:55.081 INFO:teuthology.orchestra.run.smithi005.stdout: "cephx_cluster_require_signatures": "false", 2024-04-20T17:07:55.081 INFO:teuthology.orchestra.run.smithi005.stdout: "cephx_cluster_require_version": "2", 2024-04-20T17:07:55.081 INFO:teuthology.orchestra.run.smithi005.stdout: "cephx_require_signatures": "false", 2024-04-20T17:07:55.081 INFO:teuthology.orchestra.run.smithi005.stdout: "cephx_require_version": "2", 2024-04-20T17:07:55.081 INFO:teuthology.orchestra.run.smithi005.stdout: "cephx_service_require_signatures": "false", 2024-04-20T17:07:55.081 INFO:teuthology.orchestra.run.smithi005.stdout: "cephx_service_require_version": "2", 2024-04-20T17:07:55.082 INFO:teuthology.orchestra.run.smithi005.stdout: "cephx_sign_messages": "true", 2024-04-20T17:07:55.082 INFO:teuthology.orchestra.run.smithi005.stdout: "chdir": "", 2024-04-20T17:07:55.082 INFO:teuthology.orchestra.run.smithi005.stdout: "client_acl_type": "", 2024-04-20T17:07:55.082 INFO:teuthology.orchestra.run.smithi005.stdout: "client_asio_thread_count": "2", 2024-04-20T17:07:55.082 INFO:teuthology.orchestra.run.smithi005.stdout: "client_cache_mid": "0.750000", 2024-04-20T17:07:55.082 INFO:teuthology.orchestra.run.smithi005.stdout: "client_cache_size": "16384", 2024-04-20T17:07:55.082 INFO:teuthology.orchestra.run.smithi005.stdout: "client_caps_release_delay": "5", 2024-04-20T17:07:55.082 INFO:teuthology.orchestra.run.smithi005.stdout: "client_check_pool_perm": "true", 2024-04-20T17:07:55.082 INFO:teuthology.orchestra.run.smithi005.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-20T17:07:55.082 INFO:teuthology.orchestra.run.smithi005.stdout: "client_debug_force_sync_read": "false", 2024-04-20T17:07:55.082 INFO:teuthology.orchestra.run.smithi005.stdout: "client_debug_getattr_caps": "false", 2024-04-20T17:07:55.082 INFO:teuthology.orchestra.run.smithi005.stdout: "client_debug_inject_tick_delay": "0", 2024-04-20T17:07:55.082 INFO:teuthology.orchestra.run.smithi005.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-20T17:07:55.083 INFO:teuthology.orchestra.run.smithi005.stdout: "client_die_on_failed_remount": "false", 2024-04-20T17:07:55.083 INFO:teuthology.orchestra.run.smithi005.stdout: "client_dirsize_rbytes": "true", 2024-04-20T17:07:55.083 INFO:teuthology.orchestra.run.smithi005.stdout: "client_force_lazyio": "false", 2024-04-20T17:07:55.083 INFO:teuthology.orchestra.run.smithi005.stdout: "client_fs": "", 2024-04-20T17:07:55.083 INFO:teuthology.orchestra.run.smithi005.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-20T17:07:55.083 INFO:teuthology.orchestra.run.smithi005.stdout: "client_inject_release_failure": "false", 2024-04-20T17:07:55.083 INFO:teuthology.orchestra.run.smithi005.stdout: "client_max_inline_size": "4096", 2024-04-20T17:07:55.083 INFO:teuthology.orchestra.run.smithi005.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-20T17:07:55.083 INFO:teuthology.orchestra.run.smithi005.stdout: "client_mds_namespace": "", 2024-04-20T17:07:55.083 INFO:teuthology.orchestra.run.smithi005.stdout: "client_metadata": "", 2024-04-20T17:07:55.083 INFO:teuthology.orchestra.run.smithi005.stdout: "client_mount_gid": "-1", 2024-04-20T17:07:55.083 INFO:teuthology.orchestra.run.smithi005.stdout: "client_mount_timeout": "300", 2024-04-20T17:07:55.083 INFO:teuthology.orchestra.run.smithi005.stdout: "client_mount_uid": "-1", 2024-04-20T17:07:55.084 INFO:teuthology.orchestra.run.smithi005.stdout: "client_mountpoint": "/", 2024-04-20T17:07:55.084 INFO:teuthology.orchestra.run.smithi005.stdout: "client_notify_timeout": "10", 2024-04-20T17:07:55.084 INFO:teuthology.orchestra.run.smithi005.stdout: "client_oc": "true", 2024-04-20T17:07:55.084 INFO:teuthology.orchestra.run.smithi005.stdout: "client_oc_max_dirty": "104857600", 2024-04-20T17:07:55.084 INFO:teuthology.orchestra.run.smithi005.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-20T17:07:55.084 INFO:teuthology.orchestra.run.smithi005.stdout: "client_oc_max_objects": "1000", 2024-04-20T17:07:55.084 INFO:teuthology.orchestra.run.smithi005.stdout: "client_oc_size": "209715200", 2024-04-20T17:07:55.084 INFO:teuthology.orchestra.run.smithi005.stdout: "client_oc_target_dirty": "8388608", 2024-04-20T17:07:55.084 INFO:teuthology.orchestra.run.smithi005.stdout: "client_permissions": "true", 2024-04-20T17:07:55.084 INFO:teuthology.orchestra.run.smithi005.stdout: "client_quota": "true", 2024-04-20T17:07:55.084 INFO:teuthology.orchestra.run.smithi005.stdout: "client_quota_df": "true", 2024-04-20T17:07:55.084 INFO:teuthology.orchestra.run.smithi005.stdout: "client_readahead_max_bytes": "0", 2024-04-20T17:07:55.084 INFO:teuthology.orchestra.run.smithi005.stdout: "client_readahead_max_periods": "4", 2024-04-20T17:07:55.084 INFO:teuthology.orchestra.run.smithi005.stdout: "client_readahead_min": "131072", 2024-04-20T17:07:55.085 INFO:teuthology.orchestra.run.smithi005.stdout: "client_reconnect_stale": "false", 2024-04-20T17:07:55.085 INFO:teuthology.orchestra.run.smithi005.stdout: "client_shutdown_timeout": "30", 2024-04-20T17:07:55.085 INFO:teuthology.orchestra.run.smithi005.stdout: "client_snapdir": ".snap", 2024-04-20T17:07:55.085 INFO:teuthology.orchestra.run.smithi005.stdout: "client_tick_interval": "1", 2024-04-20T17:07:55.085 INFO:teuthology.orchestra.run.smithi005.stdout: "client_trace": "", 2024-04-20T17:07:55.085 INFO:teuthology.orchestra.run.smithi005.stdout: "client_try_dentry_invalidate": "false", 2024-04-20T17:07:55.085 INFO:teuthology.orchestra.run.smithi005.stdout: "client_use_faked_inos": "false", 2024-04-20T17:07:55.085 INFO:teuthology.orchestra.run.smithi005.stdout: "client_use_random_mds": "false", 2024-04-20T17:07:55.085 INFO:teuthology.orchestra.run.smithi005.stdout: "clog_to_graylog": "false", 2024-04-20T17:07:55.085 INFO:teuthology.orchestra.run.smithi005.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-20T17:07:55.085 INFO:teuthology.orchestra.run.smithi005.stdout: "clog_to_graylog_port": "12201", 2024-04-20T17:07:55.085 INFO:teuthology.orchestra.run.smithi005.stdout: "clog_to_monitors": "default=true", 2024-04-20T17:07:55.085 INFO:teuthology.orchestra.run.smithi005.stdout: "clog_to_syslog": "false", 2024-04-20T17:07:55.086 INFO:teuthology.orchestra.run.smithi005.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-20T17:07:55.086 INFO:teuthology.orchestra.run.smithi005.stdout: "clog_to_syslog_level": "info", 2024-04-20T17:07:55.086 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "-", 2024-04-20T17:07:55.086 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_network": "", 2024-04-20T17:07:55.086 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_network_interface": "", 2024-04-20T17:07:55.086 INFO:teuthology.orchestra.run.smithi005.stdout: "compressor_zlib_isal": "false", 2024-04-20T17:07:55.086 INFO:teuthology.orchestra.run.smithi005.stdout: "compressor_zlib_level": "5", 2024-04-20T17:07:55.086 INFO:teuthology.orchestra.run.smithi005.stdout: "compressor_zlib_winsize": "-15", 2024-04-20T17:07:55.086 INFO:teuthology.orchestra.run.smithi005.stdout: "compressor_zstd_level": "1", 2024-04-20T17:07:55.086 INFO:teuthology.orchestra.run.smithi005.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dc102f95bf5a3cb0e41ae2360714141bf44b1867f263b70ab9e489d297e81b79", 2024-04-20T17:07:55.086 INFO:teuthology.orchestra.run.smithi005.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-20T17:07:55.086 INFO:teuthology.orchestra.run.smithi005.stdout: "crimson_alien_op_num_threads": "6", 2024-04-20T17:07:55.086 INFO:teuthology.orchestra.run.smithi005.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-20T17:07:55.087 INFO:teuthology.orchestra.run.smithi005.stdout: "crimson_osd_obc_lru_size": "512", 2024-04-20T17:07:55.087 INFO:teuthology.orchestra.run.smithi005.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-20T17:07:55.087 INFO:teuthology.orchestra.run.smithi005.stdout: "crimson_osd_stat_interval": "0", 2024-04-20T17:07:55.087 INFO:teuthology.orchestra.run.smithi005.stdout: "crimson_seastar_cpu_cores": "", 2024-04-20T17:07:55.087 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "", 2024-04-20T17:07:55.087 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location_hook": "", 2024-04-20T17:07:55.087 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location_hook_timeout": "10", 2024-04-20T17:07:55.087 INFO:teuthology.orchestra.run.smithi005.stdout: "daemonize": "false", 2024-04-20T17:07:55.087 INFO:teuthology.orchestra.run.smithi005.stdout: "daos_pool": "tank", 2024-04-20T17:07:55.087 INFO:teuthology.orchestra.run.smithi005.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-04-20T17:07:55.087 INFO:teuthology.orchestra.run.smithi005.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-04-20T17:07:55.087 INFO:teuthology.orchestra.run.smithi005.stdout: "dbstore_db_name_prefix": "dbstore", 2024-04-20T17:07:55.087 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_alienstore": "0/5", 2024-04-20T17:07:55.087 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_allow_any_pool_priority": "false", 2024-04-20T17:07:55.088 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_asok": "1/5", 2024-04-20T17:07:55.088 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_asok_assert_abort": "false", 2024-04-20T17:07:55.088 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_asserts_on_shutdown": "true", 2024-04-20T17:07:55.088 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_auth": "1/5", 2024-04-20T17:07:55.088 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_bdev": "1/3", 2024-04-20T17:07:55.088 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_bluefs": "1/5", 2024-04-20T17:07:55.088 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_bluestore": "1/5", 2024-04-20T17:07:55.088 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_buffer": "0/1", 2024-04-20T17:07:55.088 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_ceph_exporter": "1/5", 2024-04-20T17:07:55.088 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_cephfs_mirror": "0/5", 2024-04-20T17:07:55.088 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_cephsqlite": "0/5", 2024-04-20T17:07:55.088 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_client": "0/5", 2024-04-20T17:07:55.088 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_compressor": "1/5", 2024-04-20T17:07:55.088 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_context": "0/1", 2024-04-20T17:07:55.088 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_crimson_interrupt": "0/5", 2024-04-20T17:07:55.089 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_crush": "1/1", 2024-04-20T17:07:55.089 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_crypto": "1/5", 2024-04-20T17:07:55.089 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_cyanstore": "0/5", 2024-04-20T17:07:55.089 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_deliberately_leak_memory": "false", 2024-04-20T17:07:55.089 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_disable_randomized_ping": "false", 2024-04-20T17:07:55.089 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_dpdk": "1/5", 2024-04-20T17:07:55.089 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_eventtrace": "1/5", 2024-04-20T17:07:55.089 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_filer": "0/1", 2024-04-20T17:07:55.089 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_filestore": "1/3", 2024-04-20T17:07:55.089 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_finisher": "1/1", 2024-04-20T17:07:55.089 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_fuse": "1/5", 2024-04-20T17:07:55.089 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_heartbeat_testing_span": "0", 2024-04-20T17:07:55.089 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_heartbeatmap": "1/5", 2024-04-20T17:07:55.089 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-20T17:07:55.090 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_javaclient": "1/5", 2024-04-20T17:07:55.090 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_journal": "1/3", 2024-04-20T17:07:55.090 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_journaler": "0/5", 2024-04-20T17:07:55.090 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_kstore": "1/5", 2024-04-20T17:07:55.090 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_lockdep": "0/1", 2024-04-20T17:07:55.090 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mclock": "1/5", 2024-04-20T17:07:55.090 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mds": "1/5", 2024-04-20T17:07:55.090 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mds_balancer": "1/5", 2024-04-20T17:07:55.090 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mds_locker": "1/5", 2024-04-20T17:07:55.090 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mds_log": "1/5", 2024-04-20T17:07:55.090 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mds_log_expire": "1/5", 2024-04-20T17:07:55.090 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mds_migrator": "1/5", 2024-04-20T17:07:55.090 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mds_quiesce": "3/5", 2024-04-20T17:07:55.090 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_memstore": "1/5", 2024-04-20T17:07:55.091 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mgr": "2/5", 2024-04-20T17:07:55.091 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mgrc": "1/5", 2024-04-20T17:07:55.091 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mon": "20/20", 2024-04-20T17:07:55.091 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_monc": "0/10", 2024-04-20T17:07:55.091 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_ms": "1/1", 2024-04-20T17:07:55.091 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_none": "0/5", 2024-04-20T17:07:55.091 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_objclass": "0/5", 2024-04-20T17:07:55.091 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_objectcacher": "0/5", 2024-04-20T17:07:55.091 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_objecter": "0/1", 2024-04-20T17:07:55.091 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_optracker": "0/5", 2024-04-20T17:07:55.091 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_osd": "1/5", 2024-04-20T17:07:55.091 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_paxos": "20/20", 2024-04-20T17:07:55.091 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_perfcounter": "1/5", 2024-04-20T17:07:55.091 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_prioritycache": "1/5", 2024-04-20T17:07:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rados": "0/5", 2024-04-20T17:07:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rbd": "0/5", 2024-04-20T17:07:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rbd_mirror": "0/5", 2024-04-20T17:07:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rbd_pwl": "0/5", 2024-04-20T17:07:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rbd_replay": "0/5", 2024-04-20T17:07:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_refs": "0/0", 2024-04-20T17:07:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_reserver": "1/1", 2024-04-20T17:07:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rgw": "1/5", 2024-04-20T17:07:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rgw_access": "1/5", 2024-04-20T17:07:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rgw_datacache": "1/5", 2024-04-20T17:07:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rgw_dbstore": "1/5", 2024-04-20T17:07:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rgw_flight": "1/5", 2024-04-20T17:07:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rgw_lifecycle": "1/5", 2024-04-20T17:07:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rgw_notification": "1/5", 2024-04-20T17:07:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rgw_sync": "1/5", 2024-04-20T17:07:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rocksdb": "4/5", 2024-04-20T17:07:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore": "0/5", 2024-04-20T17:07:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_backref": "0/5", 2024-04-20T17:07:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_cache": "0/5", 2024-04-20T17:07:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_cleaner": "0/5", 2024-04-20T17:07:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_device": "0/5", 2024-04-20T17:07:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_epm": "0/5", 2024-04-20T17:07:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-04-20T17:07:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_journal": "0/5", 2024-04-20T17:07:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_lba": "0/5", 2024-04-20T17:07:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_odata": "0/5", 2024-04-20T17:07:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_omap": "0/5", 2024-04-20T17:07:55.094 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_onode": "0/5", 2024-04-20T17:07:55.094 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_t": "0/5", 2024-04-20T17:07:55.094 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_tm": "0/5", 2024-04-20T17:07:55.094 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_striper": "0/1", 2024-04-20T17:07:55.094 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_test": "0/5", 2024-04-20T17:07:55.094 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_throttle": "1/1", 2024-04-20T17:07:55.094 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_timer": "0/1", 2024-04-20T17:07:55.094 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_tp": "0/5", 2024-04-20T17:07:55.094 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_trace": "1/5", 2024-04-20T17:07:55.094 INFO:teuthology.orchestra.run.smithi005.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-20T17:07:55.094 INFO:teuthology.orchestra.run.smithi005.stdout: "device_failure_prediction_mode": "none", 2024-04-20T17:07:55.094 INFO:teuthology.orchestra.run.smithi005.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-04-20T17:07:55.094 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-04-20T17:07:55.094 INFO:teuthology.orchestra.run.smithi005.stdout: "err_to_graylog": "false", 2024-04-20T17:07:55.095 INFO:teuthology.orchestra.run.smithi005.stdout: "err_to_journald": "false", 2024-04-20T17:07:55.095 INFO:teuthology.orchestra.run.smithi005.stdout: "err_to_stderr": "true", 2024-04-20T17:07:55.095 INFO:teuthology.orchestra.run.smithi005.stdout: "err_to_syslog": "false", 2024-04-20T17:07:55.095 INFO:teuthology.orchestra.run.smithi005.stdout: "event_tracing": "false", 2024-04-20T17:07:55.095 INFO:teuthology.orchestra.run.smithi005.stdout: "exporter_addr": "0.0.0.0", 2024-04-20T17:07:55.095 INFO:teuthology.orchestra.run.smithi005.stdout: "exporter_http_port": "9926", 2024-04-20T17:07:55.095 INFO:teuthology.orchestra.run.smithi005.stdout: "exporter_prio_limit": "5", 2024-04-20T17:07:55.095 INFO:teuthology.orchestra.run.smithi005.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-20T17:07:55.095 INFO:teuthology.orchestra.run.smithi005.stdout: "exporter_sort_metrics": "true", 2024-04-20T17:07:55.095 INFO:teuthology.orchestra.run.smithi005.stdout: "exporter_stats_period": "5", 2024-04-20T17:07:55.095 INFO:teuthology.orchestra.run.smithi005.stdout: "fake_statfs_for_testing": "0", 2024-04-20T17:07:55.095 INFO:teuthology.orchestra.run.smithi005.stdout: "fatal_signal_handlers": "true", 2024-04-20T17:07:55.095 INFO:teuthology.orchestra.run.smithi005.stdout: "filer_max_purge_ops": "10", 2024-04-20T17:07:55.096 INFO:teuthology.orchestra.run.smithi005.stdout: "filer_max_truncate_ops": "128", 2024-04-20T17:07:55.096 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_apply_finisher_threads": "1", 2024-04-20T17:07:55.096 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_blackhole": "false", 2024-04-20T17:07:55.096 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_btrfs_clone_range": "true", 2024-04-20T17:07:55.096 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_btrfs_snap": "true", 2024-04-20T17:07:55.096 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_caller_concurrency": "10", 2024-04-20T17:07:55.096 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_collect_device_partition_information": "true", 2024-04-20T17:07:55.096 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_commit_timeout": "600.000000", 2024-04-20T17:07:55.096 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_debug_inject_read_err": "false", 2024-04-20T17:07:55.096 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_debug_omap_check": "false", 2024-04-20T17:07:55.096 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_debug_verify_split": "false", 2024-04-20T17:07:55.096 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_dump_file": "", 2024-04-20T17:07:55.096 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-20T17:07:55.097 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-20T17:07:55.097 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_fadvise": "true", 2024-04-20T17:07:55.097 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_fail_eio": "true", 2024-04-20T17:07:55.097 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_fd_cache_shards": "16", 2024-04-20T17:07:55.097 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_fd_cache_size": "128", 2024-04-20T17:07:55.097 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_fiemap": "false", 2024-04-20T17:07:55.097 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_fiemap_threshold": "4096", 2024-04-20T17:07:55.097 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-20T17:07:55.097 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-20T17:07:55.097 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_inject_stall": "0", 2024-04-20T17:07:55.097 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_journal_parallel": "false", 2024-04-20T17:07:55.097 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_journal_trailing": "false", 2024-04-20T17:07:55.097 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_journal_writeahead": "false", 2024-04-20T17:07:55.097 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_kill_at": "0", 2024-04-20T17:07:55.098 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-20T17:07:55.098 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-20T17:07:55.098 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-20T17:07:55.098 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-20T17:07:55.098 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-20T17:07:55.098 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_inline_xattrs": "0", 2024-04-20T17:07:55.098 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-20T17:07:55.098 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-20T17:07:55.098 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-20T17:07:55.098 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-20T17:07:55.098 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_xattr_value_size": "0", 2024-04-20T17:07:55.098 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-20T17:07:55.099 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-20T17:07:55.099 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-20T17:07:55.099 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_merge_threshold": "-10", 2024-04-20T17:07:55.099 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-20T17:07:55.099 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_odsync_write": "false", 2024-04-20T17:07:55.099 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_omap_backend": "rocksdb", 2024-04-20T17:07:55.099 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_omap_backend_path": "", 2024-04-20T17:07:55.099 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-20T17:07:55.099 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-20T17:07:55.099 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-20T17:07:55.099 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_op_thread_timeout": "60", 2024-04-20T17:07:55.099 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_op_threads": "2", 2024-04-20T17:07:55.099 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_punch_hole": "false", 2024-04-20T17:07:55.100 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-20T17:07:55.100 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-20T17:07:55.100 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-20T17:07:55.100 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-20T17:07:55.100 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-20T17:07:55.100 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-20T17:07:55.100 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-20T17:07:55.100 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-20T17:07:55.100 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-20T17:07:55.100 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_max_ops": "50", 2024-04-20T17:07:55.100 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-20T17:07:55.100 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_seek_data_hole": "false", 2024-04-20T17:07:55.100 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_sloppy_crc": "false", 2024-04-20T17:07:55.101 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-20T17:07:55.101 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_splice": "false", 2024-04-20T17:07:55.101 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_split_multiple": "2", 2024-04-20T17:07:55.101 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_split_rand_factor": "20", 2024-04-20T17:07:55.101 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_update_to": "1000", 2024-04-20T17:07:55.101 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-20T17:07:55.101 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-20T17:07:55.101 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-20T17:07:55.101 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-20T17:07:55.101 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-20T17:07:55.101 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-20T17:07:55.101 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_enable": "true", 2024-04-20T17:07:55.101 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-20T17:07:55.101 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-20T17:07:55.102 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-20T17:07:55.102 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-20T17:07:55.102 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-20T17:07:55.102 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-20T17:07:55.102 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_xfs_extsize": "false", 2024-04-20T17:07:55.102 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_zfs_snap": "false", 2024-04-20T17:07:55.102 INFO:teuthology.orchestra.run.smithi005.stdout: "fio_dir": "/tmp/fio", 2024-04-20T17:07:55.102 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "13c6e76c-ff37-11ee-bc93-c7b262605968", 2024-04-20T17:07:55.102 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_allow_other": "true", 2024-04-20T17:07:55.102 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_atomic_o_trunc": "true", 2024-04-20T17:07:55.102 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_big_writes": "true", 2024-04-20T17:07:55.102 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_debug": "false", 2024-04-20T17:07:55.103 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_default_permissions": "false", 2024-04-20T17:07:55.103 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_disable_pagecache": "false", 2024-04-20T17:07:55.103 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_max_write": "0", 2024-04-20T17:07:55.103 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_multithreaded": "true", 2024-04-20T17:07:55.103 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_require_active_mds": "true", 2024-04-20T17:07:55.103 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_set_user_groups": "true", 2024-04-20T17:07:55.103 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_splice_move": "true", 2024-04-20T17:07:55.103 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_splice_read": "true", 2024-04-20T17:07:55.103 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_splice_write": "true", 2024-04-20T17:07:55.103 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-20T17:07:55.103 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_use_invalidate_cb": "true", 2024-04-20T17:07:55.103 INFO:teuthology.orchestra.run.smithi005.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-20T17:07:55.103 INFO:teuthology.orchestra.run.smithi005.stdout: "gss_target_name": "ceph", 2024-04-20T17:07:55.103 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_file": "", 2024-04-20T17:07:55.104 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_inject_failure": "0", 2024-04-20T17:07:55.104 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_interval": "5", 2024-04-20T17:07:55.104 INFO:teuthology.orchestra.run.smithi005.stdout: "host": "smithi005", 2024-04-20T17:07:55.104 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-20T17:07:55.104 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-20T17:07:55.104 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-20T17:07:55.104 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-20T17:07:55.104 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-20T17:07:55.104 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-20T17:07:55.104 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-20T17:07:55.104 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-20T17:07:55.104 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-20T17:07:55.104 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-20T17:07:55.104 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-20T17:07:55.105 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-20T17:07:55.105 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-20T17:07:55.105 INFO:teuthology.orchestra.run.smithi005.stdout: "inject_early_sigterm": "false", 2024-04-20T17:07:55.105 INFO:teuthology.orchestra.run.smithi005.stdout: "jaeger_agent_port": "6799", 2024-04-20T17:07:55.105 INFO:teuthology.orchestra.run.smithi005.stdout: "jaeger_tracing_enable": "false", 2024-04-20T17:07:55.105 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_aio": "true", 2024-04-20T17:07:55.105 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_align_min_size": "65536", 2024-04-20T17:07:55.105 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_block_align": "true", 2024-04-20T17:07:55.105 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_block_size": "4096", 2024-04-20T17:07:55.105 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_dio": "true", 2024-04-20T17:07:55.105 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_discard": "false", 2024-04-20T17:07:55.105 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_force_aio": "false", 2024-04-20T17:07:55.105 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_ignore_corruption": "false", 2024-04-20T17:07:55.106 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_max_write_bytes": "10485760", 2024-04-20T17:07:55.106 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_max_write_entries": "100", 2024-04-20T17:07:55.106 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_replay_from": "0", 2024-04-20T17:07:55.106 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-20T17:07:55.106 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-20T17:07:55.106 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-20T17:07:55.106 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-20T17:07:55.106 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_write_header_frequency": "0", 2024-04-20T17:07:55.106 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_zero_on_create": "false", 2024-04-20T17:07:55.106 INFO:teuthology.orchestra.run.smithi005.stdout: "journaler_prefetch_periods": "10", 2024-04-20T17:07:55.106 INFO:teuthology.orchestra.run.smithi005.stdout: "journaler_prezero_periods": "5", 2024-04-20T17:07:55.106 INFO:teuthology.orchestra.run.smithi005.stdout: "journaler_write_head_interval": "15", 2024-04-20T17:07:55.106 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "", 2024-04-20T17:07:55.106 INFO:teuthology.orchestra.run.smithi005.stdout: "keyfile": "", 2024-04-20T17:07:55.107 INFO:teuthology.orchestra.run.smithi005.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-20T17:07:55.107 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_backend": "rocksdb", 2024-04-20T17:07:55.107 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_default_stripe_size": "65536", 2024-04-20T17:07:55.107 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_fsck_on_mount": "false", 2024-04-20T17:07:55.107 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-20T17:07:55.107 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_max_bytes": "67108864", 2024-04-20T17:07:55.107 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_max_ops": "512", 2024-04-20T17:07:55.107 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_nid_prealloc": "1024", 2024-04-20T17:07:55.107 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_onode_map_size": "1024", 2024-04-20T17:07:55.107 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-20T17:07:55.107 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_sync_submit_transaction": "false", 2024-04-20T17:07:55.107 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_sync_transaction": "false", 2024-04-20T17:07:55.107 INFO:teuthology.orchestra.run.smithi005.stdout: "librados_thread_count": "2", 2024-04-20T17:07:55.108 INFO:teuthology.orchestra.run.smithi005.stdout: "lockdep": "false", 2024-04-20T17:07:55.108 INFO:teuthology.orchestra.run.smithi005.stdout: "lockdep_force_backtrace": "false", 2024-04-20T17:07:55.108 INFO:teuthology.orchestra.run.smithi005.stdout: "log_coarse_timestamps": "true", 2024-04-20T17:07:55.108 INFO:teuthology.orchestra.run.smithi005.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-20T17:07:55.108 INFO:teuthology.orchestra.run.smithi005.stdout: "log_flush_on_exit": "false", 2024-04-20T17:07:55.108 INFO:teuthology.orchestra.run.smithi005.stdout: "log_graylog_host": "127.0.0.1", 2024-04-20T17:07:55.108 INFO:teuthology.orchestra.run.smithi005.stdout: "log_graylog_port": "12201", 2024-04-20T17:07:55.108 INFO:teuthology.orchestra.run.smithi005.stdout: "log_max_new": "1000", 2024-04-20T17:07:55.108 INFO:teuthology.orchestra.run.smithi005.stdout: "log_max_recent": "10000", 2024-04-20T17:07:55.108 INFO:teuthology.orchestra.run.smithi005.stdout: "log_stderr_prefix": "", 2024-04-20T17:07:55.108 INFO:teuthology.orchestra.run.smithi005.stdout: "log_stop_at_utilization": "0.970000", 2024-04-20T17:07:55.108 INFO:teuthology.orchestra.run.smithi005.stdout: "log_to_file": "true", 2024-04-20T17:07:55.108 INFO:teuthology.orchestra.run.smithi005.stdout: "log_to_graylog": "false", 2024-04-20T17:07:55.108 INFO:teuthology.orchestra.run.smithi005.stdout: "log_to_journald": "false", 2024-04-20T17:07:55.109 INFO:teuthology.orchestra.run.smithi005.stdout: "log_to_stderr": "false", 2024-04-20T17:07:55.109 INFO:teuthology.orchestra.run.smithi005.stdout: "log_to_syslog": "false", 2024-04-20T17:07:55.109 INFO:teuthology.orchestra.run.smithi005.stdout: "max_rotating_auth_attempts": "10", 2024-04-20T17:07:55.109 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-20T17:07:55.109 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_action_on_write_error": "1", 2024-04-20T17:07:55.109 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_alternate_name_max": "8192", 2024-04-20T17:07:55.109 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_asio_thread_count": "2", 2024-04-20T17:07:55.109 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_export_pin": "true", 2024-04-20T17:07:55.109 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_fragment_dirs": "true", 2024-04-20T17:07:55.109 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-20T17:07:55.109 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_fragment_interval": "5", 2024-04-20T17:07:55.109 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-20T17:07:55.109 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-20T17:07:55.110 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_interval": "10", 2024-04-20T17:07:55.110 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_max": "-1", 2024-04-20T17:07:55.110 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_max_until": "-1", 2024-04-20T17:07:55.110 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_merge_size": "50", 2024-04-20T17:07:55.110 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_midchunk": "0.300000", 2024-04-20T17:07:55.110 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-20T17:07:55.110 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_min_start": "0.200000", 2024-04-20T17:07:55.110 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_minchunk": "0.001000", 2024-04-20T17:07:55.110 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_mode": "0", 2024-04-20T17:07:55.110 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_need_max": "1.200000", 2024-04-20T17:07:55.110 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_need_min": "0.800000", 2024-04-20T17:07:55.110 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_overload_epochs": "2", 2024-04-20T17:07:55.110 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-20T17:07:55.110 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-20T17:07:55.111 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_split_bits": "3", 2024-04-20T17:07:55.111 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-20T17:07:55.111 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_split_size": "10000", 2024-04-20T17:07:55.111 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-20T17:07:55.111 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_target_decay": "10.000000", 2024-04-20T17:07:55.111 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-20T17:07:55.111 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_beacon_grace": "15.000000", 2024-04-20T17:07:55.111 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_beacon_interval": "4.000000", 2024-04-20T17:07:55.111 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-20T17:07:55.111 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-20T17:07:55.111 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_mid": "0.700000", 2024-04-20T17:07:55.111 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-04-20T17:07:55.111 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_quiesce_delay": "0", 2024-04-20T17:07:55.111 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_quiesce_sleep": "200", 2024-04-20T17:07:55.112 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_quiesce_splitauth": "true", 2024-04-20T17:07:55.112 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_quiesce_threshold": "524288", 2024-04-20T17:07:55.112 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_release_free_interval": "10", 2024-04-20T17:07:55.112 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_reservation": "0.050000", 2024-04-20T17:07:55.112 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-20T17:07:55.112 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_trim_interval": "1", 2024-04-20T17:07:55.112 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_trim_threshold": "262144", 2024-04-20T17:07:55.112 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-20T17:07:55.112 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-20T17:07:55.112 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-20T17:07:55.112 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_client_prealloc_inos": "1000", 2024-04-20T17:07:55.112 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-20T17:07:55.112 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_connect_bootstrapping": "false", 2024-04-20T17:07:55.112 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_damage_table_max_entries": "10000", 2024-04-20T17:07:55.112 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-20T17:07:55.113 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_debug_auth_pins": "false", 2024-04-20T17:07:55.113 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_debug_frag": "false", 2024-04-20T17:07:55.113 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_debug_scatterstat": "false", 2024-04-20T17:07:55.113 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_debug_subtrees": "false", 2024-04-20T17:07:55.113 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_decay_halflife": "5.000000", 2024-04-20T17:07:55.113 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_default_dir_hash": "2", 2024-04-20T17:07:55.113 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_defer_session_stale": "true", 2024-04-20T17:07:55.113 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_deny_all_reconnect": "false", 2024-04-20T17:07:55.113 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_dir_keys_per_op": "16384", 2024-04-20T17:07:55.113 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_dir_max_commit_size": "10", 2024-04-20T17:07:55.113 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_dir_max_entries": "0", 2024-04-20T17:07:55.113 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_dir_prefetch": "true", 2024-04-20T17:07:55.113 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-20T17:07:55.113 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-20T17:07:55.114 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_dump_cache_on_map": "false", 2024-04-20T17:07:55.114 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-20T17:07:55.114 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-20T17:07:55.114 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_early_reply": "true", 2024-04-20T17:07:55.114 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_enable_op_tracker": "true", 2024-04-20T17:07:55.114 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_enforce_unique_name": "true", 2024-04-20T17:07:55.114 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-20T17:07:55.114 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-20T17:07:55.114 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_export_ephemeral_random": "true", 2024-04-20T17:07:55.114 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-20T17:07:55.114 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-04-20T17:07:55.114 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-20T17:07:55.114 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-20T17:07:55.115 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-04-20T17:07:55.115 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-20T17:07:55.115 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-20T17:07:55.115 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-20T17:07:55.115 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_health_summarize_threshold": "10", 2024-04-20T17:07:55.115 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-20T17:07:55.115 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-20T17:07:55.115 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_inject_health_dummy": "false", 2024-04-20T17:07:55.115 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-20T17:07:55.115 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_inject_migrator_session_race": "false", 2024-04-20T17:07:55.115 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-20T17:07:55.115 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-20T17:07:55.115 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-20T17:07:55.115 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_join_fs": "", 2024-04-20T17:07:55.118 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_journal_format": "1", 2024-04-20T17:07:55.118 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_create_at": "0", 2024-04-20T17:07:55.118 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_export_at": "0", 2024-04-20T17:07:55.118 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_import_at": "0", 2024-04-20T17:07:55.118 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_journal_at": "0", 2024-04-20T17:07:55.118 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_journal_expire_at": "0", 2024-04-20T17:07:55.118 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_journal_replay_at": "0", 2024-04-20T17:07:55.118 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_link_at": "0", 2024-04-20T17:07:55.118 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_mdstable_at": "0", 2024-04-20T17:07:55.118 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_openc_at": "0", 2024-04-20T17:07:55.118 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_rename_at": "0", 2024-04-20T17:07:55.119 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_shutdown_at": "0", 2024-04-20T17:07:55.119 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-20T17:07:55.119 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_event_large_threshold": "524288", 2024-04-20T17:07:55.119 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_events_per_segment": "1024", 2024-04-20T17:07:55.119 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_major_segment_event_ratio": "12", 2024-04-20T17:07:55.119 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_max_events": "-1", 2024-04-20T17:07:55.119 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_max_segments": "128", 2024-04-20T17:07:55.119 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_pause": "false", 2024-04-20T17:07:55.119 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_segment_size": "0", 2024-04-20T17:07:55.119 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-20T17:07:55.119 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_skip_unbounded_events": "false", 2024-04-20T17:07:55.119 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-04-20T17:07:55.119 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_trim_threshold": "128", 2024-04-20T17:07:55.120 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-04-20T17:07:55.120 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_warn_factor": "2.000000", 2024-04-20T17:07:55.120 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_caps_per_client": "1048576", 2024-04-20T17:07:55.120 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_completed_flushes": "100000", 2024-04-20T17:07:55.120 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_completed_requests": "100000", 2024-04-20T17:07:55.120 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_export_size": "20971520", 2024-04-20T17:07:55.120 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_file_recover": "32", 2024-04-20T17:07:55.120 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_purge_files": "64", 2024-04-20T17:07:55.120 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_purge_ops": "8192", 2024-04-20T17:07:55.120 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-20T17:07:55.120 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-20T17:07:55.120 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_snaps_per_dir": "100", 2024-04-20T17:07:55.120 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_metrics_update_interval": "2", 2024-04-20T17:07:55.120 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_min_caps_per_client": "100", 2024-04-20T17:07:55.120 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_min_caps_working_set": "10000", 2024-04-20T17:07:55.120 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-20T17:07:55.121 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_numa_node": "-1", 2024-04-20T17:07:55.121 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-20T17:07:55.121 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_op_complaint_time": "30.000000", 2024-04-20T17:07:55.121 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_op_history_duration": "600", 2024-04-20T17:07:55.121 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_op_history_size": "20", 2024-04-20T17:07:55.121 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_op_history_slow_op_size": "20", 2024-04-20T17:07:55.121 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-20T17:07:55.121 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_op_log_threshold": "5", 2024-04-20T17:07:55.121 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_ping_grace": "15", 2024-04-20T17:07:55.121 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_ping_interval": "5", 2024-04-20T17:07:55.121 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-20T17:07:55.121 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-20T17:07:55.121 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_recall_max_caps": "30000", 2024-04-20T17:07:55.121 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-20T17:07:55.121 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-20T17:07:55.122 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-20T17:07:55.122 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_recall_warning_threshold": "262144", 2024-04-20T17:07:55.122 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-20T17:07:55.122 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_replay_interval": "1.000000", 2024-04-20T17:07:55.122 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-20T17:07:55.122 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-20T17:07:55.122 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_root_ino_gid": "0", 2024-04-20T17:07:55.122 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_root_ino_uid": "0", 2024-04-20T17:07:55.122 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-20T17:07:55.122 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-20T17:07:55.122 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-20T17:07:55.122 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-20T17:07:55.122 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-20T17:07:55.122 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-20T17:07:55.122 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-20T17:07:55.123 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-20T17:07:55.123 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-20T17:07:55.123 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-20T17:07:55.123 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_shutdown_check": "0", 2024-04-20T17:07:55.123 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_skip_ino": "0", 2024-04-20T17:07:55.123 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-20T17:07:55.123 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_snap_max_uid": "4294967294", 2024-04-20T17:07:55.123 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_snap_min_uid": "0", 2024-04-20T17:07:55.123 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_snap_rstat": "false", 2024-04-20T17:07:55.123 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_standby_replay_damaged": "false", 2024-04-20T17:07:55.123 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_symlink_recovery": "true", 2024-04-20T17:07:55.123 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-20T17:07:55.123 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_thrash_exports": "0", 2024-04-20T17:07:55.123 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_thrash_fragments": "0", 2024-04-20T17:07:55.123 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_tick_interval": "5.000000", 2024-04-20T17:07:55.123 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_valgrind_exit": "false", 2024-04-20T17:07:55.123 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_verify_backtrace": "1", 2024-04-20T17:07:55.124 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_verify_scatter": "false", 2024-04-20T17:07:55.124 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-20T17:07:55.124 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_wipe_sessions": "false", 2024-04-20T17:07:55.124 INFO:teuthology.orchestra.run.smithi005.stdout: "mempool_debug": "false", 2024-04-20T17:07:55.124 INFO:teuthology.orchestra.run.smithi005.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-20T17:07:55.124 INFO:teuthology.orchestra.run.smithi005.stdout: "memstore_device_bytes": "1073741824", 2024-04-20T17:07:55.124 INFO:teuthology.orchestra.run.smithi005.stdout: "memstore_page_set": "false", 2024-04-20T17:07:55.124 INFO:teuthology.orchestra.run.smithi005.stdout: "memstore_page_size": "65536", 2024-04-20T17:07:55.124 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_client_bytes": "134217728", 2024-04-20T17:07:55.124 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_client_messages": "512", 2024-04-20T17:07:55.124 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-20T17:07:55.124 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-20T17:07:55.124 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-20T17:07:55.124 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-20T17:07:55.125 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_disabled_modules": "", 2024-04-20T17:07:55.125 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-20T17:07:55.125 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_max_pg_creating": "1024", 2024-04-20T17:07:55.125 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_max_pg_num_change": "128", 2024-04-20T17:07:55.125 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_mds_bytes": "134217728", 2024-04-20T17:07:55.125 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_mds_messages": "128", 2024-04-20T17:07:55.125 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-20T17:07:55.125 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_mon_bytes": "134217728", 2024-04-20T17:07:55.125 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_mon_messages": "128", 2024-04-20T17:07:55.125 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_osd_bytes": "536870912", 2024-04-20T17:07:55.125 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_osd_messages": "8192", 2024-04-20T17:07:55.125 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_pool": "true", 2024-04-20T17:07:55.125 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-20T17:07:55.126 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_standby_modules": "true", 2024-04-20T17:07:55.126 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_stats_period": "5", 2024-04-20T17:07:55.126 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_stats_threshold": "5", 2024-04-20T17:07:55.126 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_tick_period": "2", 2024-04-20T17:07:55.126 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-20T17:07:55.126 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-20T17:07:55.126 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_allow_pool_delete": "true", 2024-04-20T17:07:55.126 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_allow_pool_size_one": "false", 2024-04-20T17:07:55.126 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_auth_validate_all_caps": "true", 2024-04-20T17:07:55.126 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-20T17:07:55.126 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-20T17:07:55.126 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_bytes": "104857600", 2024-04-20T17:07:55.126 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_directed_command_retry": "5", 2024-04-20T17:07:55.127 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-20T17:07:55.127 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-20T17:07:55.127 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-20T17:07:55.127 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-20T17:07:55.127 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_hunt_parallel": "3", 2024-04-20T17:07:55.127 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_log_interval": "1.000000", 2024-04-20T17:07:55.127 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-20T17:07:55.127 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_ping_interval": "10.000000", 2024-04-20T17:07:55.127 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-20T17:07:55.127 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_target_rank": "-1", 2024-04-20T17:07:55.127 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-20T17:07:55.127 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-20T17:07:55.127 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-20T17:07:55.128 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_level": "debug", 2024-04-20T17:07:55.128 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_to_file": "true", 2024-04-20T17:07:55.128 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-20T17:07:55.128 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-20T17:07:55.128 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-20T17:07:55.128 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_to_journald": "true", 2024-04-20T17:07:55.128 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_to_stderr": "false", 2024-04-20T17:07:55.128 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-20T17:07:55.128 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-20T17:07:55.128 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_compact_on_bootstrap": "false", 2024-04-20T17:07:55.128 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_compact_on_start": "false", 2024-04-20T17:07:55.128 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_compact_on_trim": "true", 2024-04-20T17:07:55.128 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-20T17:07:55.128 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-20T17:07:55.129 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-20T17:07:55.129 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cpu_threads": "4", 2024-04-20T17:07:55.129 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_crush_min_required_version": "hammer", 2024-04-20T17:07:55.129 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_daemon_bytes": "419430400", 2024-04-20T17:07:55.129 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-20T17:07:55.129 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_data_avail_crit": "5", 2024-04-20T17:07:55.129 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_data_avail_warn": "5", 2024-04-20T17:07:55.129 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_data_size_warn": "16106127360", 2024-04-20T17:07:55.129 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-20T17:07:55.129 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-20T17:07:55.129 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_dump_json": "false", 2024-04-20T17:07:55.129 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-20T17:07:55.129 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_dump_transactions": "false", 2024-04-20T17:07:55.129 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_extra_checks": "false", 2024-04-20T17:07:55.129 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-20T17:07:55.130 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-20T17:07:55.130 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_no_require_reef": "false", 2024-04-20T17:07:55.130 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_no_require_squid": "false", 2024-04-20T17:07:55.130 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-20T17:07:55.130 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-20T17:07:55.130 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-20T17:07:55.130 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_down_mkfs_grace": "60", 2024-04-20T17:07:55.130 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_down_uptime_grace": "60", 2024-04-20T17:07:55.130 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_election_default_strategy": "1", 2024-04-20T17:07:55.130 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_election_timeout": "5.000000", 2024-04-20T17:07:55.130 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-20T17:07:55.130 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_elector_ping_divisor": "2", 2024-04-20T17:07:55.130 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-20T17:07:55.130 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_enable_op_tracker": "true", 2024-04-20T17:07:55.131 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_fake_pool_delete": "false", 2024-04-20T17:07:55.131 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_force_quorum_join": "false", 2024-04-20T17:07:55.131 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_globalid_prealloc": "10000", 2024-04-20T17:07:55.131 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_health_detail_to_clog": "true", 2024-04-20T17:07:55.131 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_health_log_update_period": "5", 2024-04-20T17:07:55.131 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_health_max_detail": "50", 2024-04-20T17:07:55.131 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_health_to_clog": "true", 2024-04-20T17:07:55.131 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_health_to_clog_interval": "600", 2024-04-20T17:07:55.131 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-20T17:07:55.131 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_host": "172.21.15.5:6789/0", 2024-04-20T17:07:55.131 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_host_override": "", 2024-04-20T17:07:55.131 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_initial_members": "", 2024-04-20T17:07:55.131 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-20T17:07:55.131 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-20T17:07:55.131 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-20T17:07:55.131 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-20T17:07:55.132 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-20T17:07:55.132 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_lease": "5.000000", 2024-04-20T17:07:55.132 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-20T17:07:55.132 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-20T17:07:55.132 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_log_full_interval": "50", 2024-04-20T17:07:55.132 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_log_max": "10000", 2024-04-20T17:07:55.132 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_log_max_summary": "50", 2024-04-20T17:07:55.132 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-20T17:07:55.132 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_max_log_epochs": "500", 2024-04-20T17:07:55.132 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-20T17:07:55.132 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-20T17:07:55.132 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_max_osd": "10000", 2024-04-20T17:07:55.132 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_max_pg_per_osd": "10000", 2024-04-20T17:07:55.132 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_max_pool_pg_num": "65536", 2024-04-20T17:07:55.132 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-20T17:07:55.132 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-20T17:07:55.133 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_mds_force_trim_to": "0", 2024-04-20T17:07:55.133 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_mds_skip_sanity": "false", 2024-04-20T17:07:55.133 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_memory_autotune": "true", 2024-04-20T17:07:55.133 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_memory_target": "2147483648", 2024-04-20T17:07:55.133 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_mgr_beacon_grace": "30", 2024-04-20T17:07:55.133 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-20T17:07:55.133 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_mgr_digest_period": "5", 2024-04-20T17:07:55.133 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_mgr_inactive_grace": "60", 2024-04-20T17:07:55.133 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-20T17:07:55.133 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-20T17:07:55.133 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_min_osdmap_epochs": "50", 2024-04-20T17:07:55.133 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_op_complaint_time": "30", 2024-04-20T17:07:55.133 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_op_history_duration": "600", 2024-04-20T17:07:55.133 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_op_history_size": "20", 2024-04-20T17:07:55.133 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_op_history_slow_op_size": "20", 2024-04-20T17:07:55.134 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-20T17:07:55.134 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_op_log_threshold": "5", 2024-04-20T17:07:55.134 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-20T17:07:55.134 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-20T17:07:55.134 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-20T17:07:55.134 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_auto_mark_in": "false", 2024-04-20T17:07:55.134 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-20T17:07:55.134 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-20T17:07:55.134 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-20T17:07:55.134 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_cache_size": "500", 2024-04-20T17:07:55.134 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-20T17:07:55.134 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-20T17:07:55.134 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-20T17:07:55.134 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_down_out_interval": "600", 2024-04-20T17:07:55.135 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-20T17:07:55.135 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-20T17:07:55.135 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_force_trim_to": "0", 2024-04-20T17:07:55.135 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-20T17:07:55.135 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-20T17:07:55.135 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-20T17:07:55.135 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-20T17:07:55.135 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-20T17:07:55.135 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-20T17:07:55.135 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-20T17:07:55.135 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_min_down_reporters": "2", 2024-04-20T17:07:55.135 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-20T17:07:55.135 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-20T17:07:55.135 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-20T17:07:55.136 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-20T17:07:55.136 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-20T17:07:55.136 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-20T17:07:55.136 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_report_timeout": "900", 2024-04-20T17:07:55.136 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-20T17:07:55.136 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-20T17:07:55.136 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-20T17:07:55.136 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-20T17:07:55.136 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-20T17:07:55.136 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osdmap_full_prune_interval": "2", 2024-04-20T17:07:55.136 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osdmap_full_prune_min": "15", 2024-04-20T17:07:55.136 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-04-20T17:07:55.136 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-20T17:07:55.137 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_pg_stuck_threshold": "60", 2024-04-20T17:07:55.137 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-20T17:07:55.137 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-20T17:07:55.137 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-20T17:07:55.137 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-20T17:07:55.137 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-20T17:07:55.137 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-20T17:07:55.137 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_probe_timeout": "2.000000", 2024-04-20T17:07:55.137 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_reweight_max_change": "0.050000", 2024-04-20T17:07:55.137 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_reweight_max_osds": "4", 2024-04-20T17:07:55.137 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-20T17:07:55.137 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-20T17:07:55.137 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-20T17:07:55.137 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-20T17:07:55.137 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-20T17:07:55.137 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_scrub_interval": "300", 2024-04-20T17:07:55.138 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_scrub_max_keys": "100", 2024-04-20T17:07:55.138 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_scrub_timeout": "300", 2024-04-20T17:07:55.138 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_session_timeout": "300", 2024-04-20T17:07:55.138 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_smart_report_timeout": "5", 2024-04-20T17:07:55.138 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_stat_smooth_intervals": "6", 2024-04-20T17:07:55.138 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-20T17:07:55.138 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_stretch_pool_min_size": "2", 2024-04-20T17:07:55.138 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_stretch_pool_size": "4", 2024-04-20T17:07:55.138 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-20T17:07:55.138 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-20T17:07:55.138 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_sync_debug": "false", 2024-04-20T17:07:55.138 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-20T17:07:55.138 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-20T17:07:55.138 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_sync_provider_kill_at": "0", 2024-04-20T17:07:55.139 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_sync_requester_kill_at": "0", 2024-04-20T17:07:55.139 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_sync_timeout": "60.000000", 2024-04-20T17:07:55.139 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_target_pg_per_osd": "100", 2024-04-20T17:07:55.139 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_tick_interval": "5", 2024-04-20T17:07:55.139 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_timecheck_interval": "300.000000", 2024-04-20T17:07:55.139 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-20T17:07:55.139 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_use_min_delay_socket": "false", 2024-04-20T17:07:55.139 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_older_version_delay": "604800", 2024-04-20T17:07:55.139 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-20T17:07:55.139 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-20T17:07:55.139 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-20T17:07:55.139 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_filestore_osds": "true", 2024-04-20T17:07:55.139 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-20T17:07:55.139 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-20T17:07:55.140 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-20T17:07:55.140 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_misplaced": "false", 2024-04-20T17:07:55.140 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-20T17:07:55.140 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_older_version": "true", 2024-04-20T17:07:55.140 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-20T17:07:55.140 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_pool_no_app": "true", 2024-04-20T17:07:55.140 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-04-20T17:07:55.140 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-20T17:07:55.140 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-20T17:07:55.140 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-20T17:07:55.140 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-20T17:07:55.140 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-20T17:07:55.140 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-20T17:07:55.140 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": "", 2024-04-20T17:07:55.141 INFO:teuthology.orchestra.run.smithi005.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-20T17:07:55.141 INFO:teuthology.orchestra.run.smithi005.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-04-20T17:07:55.141 INFO:teuthology.orchestra.run.smithi005.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-04-20T17:07:55.141 INFO:teuthology.orchestra.run.smithi005.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-20T17:07:55.141 INFO:teuthology.orchestra.run.smithi005.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-04-20T17:07:55.141 INFO:teuthology.orchestra.run.smithi005.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-04-20T17:07:55.141 INFO:teuthology.orchestra.run.smithi005.stdout: "motr_tracing_enabled": "false", 2024-04-20T17:07:55.141 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_op_threads": "3", 2024-04-20T17:07:55.141 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-20T17:07:55.141 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_cm": "false", 2024-04-20T17:07:55.141 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_device_name": "", 2024-04-20T17:07:55.141 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_dscp": "96", 2024-04-20T17:07:55.141 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-20T17:07:55.141 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-20T17:07:55.141 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_local_gid": "", 2024-04-20T17:07:55.142 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-20T17:07:55.142 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_port_num": "1", 2024-04-20T17:07:55.142 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-20T17:07:55.142 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-20T17:07:55.142 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-20T17:07:55.142 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-20T17:07:55.142 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_sl": "3", 2024-04-20T17:07:55.142 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_support_srq": "true", 2024-04-20T17:07:55.142 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_type": "ib", 2024-04-20T17:07:55.142 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_reap_threshold": "5", 2024-04-20T17:07:55.142 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_before_connect": "false", 2024-04-20T17:07:55.142 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_ipv4": "true", 2024-04-20T17:07:55.142 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_ipv6": "false", 2024-04-20T17:07:55.143 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_msgr1": "true", 2024-04-20T17:07:55.143 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_msgr2": "false", 2024-04-20T17:07:55.143 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_port_max": "7568", 2024-04-20T17:07:55.143 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_port_min": "6800", 2024-04-20T17:07:55.143 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-20T17:07:55.143 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_retry_count": "3", 2024-04-20T17:07:55.143 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_retry_delay": "5", 2024-04-20T17:07:55.143 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_blackhole_client": "false", 2024-04-20T17:07:55.143 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_blackhole_mds": "false", 2024-04-20T17:07:55.143 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_blackhole_mgr": "false", 2024-04-20T17:07:55.143 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_blackhole_mon": "false", 2024-04-20T17:07:55.143 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_blackhole_osd": "false", 2024-04-20T17:07:55.143 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_client_mode": "crc secure", 2024-04-20T17:07:55.143 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_cluster_mode": "crc secure", 2024-04-20T17:07:55.144 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_cluster_type": "", 2024-04-20T17:07:55.144 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_compress_secure": "false", 2024-04-20T17:07:55.144 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_connection_idle_timeout": "900", 2024-04-20T17:07:55.144 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_connection_ready_timeout": "10", 2024-04-20T17:07:55.144 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_crc_data": "true", 2024-04-20T17:07:55.144 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_crc_header": "true", 2024-04-20T17:07:55.144 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_die_on_bad_msg": "false", 2024-04-20T17:07:55.144 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_die_on_bug": "true", 2024-04-20T17:07:55.144 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_die_on_old_message": "true", 2024-04-20T17:07:55.144 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_die_on_skipped_message": "false", 2024-04-20T17:07:55.144 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-20T17:07:55.144 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-20T17:07:55.144 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_coremask": "0xF", 2024-04-20T17:07:55.144 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-20T17:07:55.145 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-20T17:07:55.145 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_enable_tso": "true", 2024-04-20T17:07:55.145 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-20T17:07:55.145 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-20T17:07:55.145 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_hugepages": "", 2024-04-20T17:07:55.145 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-20T17:07:55.145 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-20T17:07:55.145 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_lro": "true", 2024-04-20T17:07:55.145 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_memory_channel": "4", 2024-04-20T17:07:55.145 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-20T17:07:55.145 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_pmd": "", 2024-04-20T17:07:55.145 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_port_id": "0", 2024-04-20T17:07:55.145 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-20T17:07:55.145 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-20T17:07:55.145 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dump_on_send": "false", 2024-04-20T17:07:55.145 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_initial_backoff": "0.200000", 2024-04-20T17:07:55.146 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_inject_delay_max": "1.000000", 2024-04-20T17:07:55.146 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_inject_delay_probability": "0.000000", 2024-04-20T17:07:55.146 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_inject_delay_type": "", 2024-04-20T17:07:55.146 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_inject_internal_delays": "0.000000", 2024-04-20T17:07:55.146 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_inject_network_congestion": "0", 2024-04-20T17:07:55.146 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_inject_socket_failures": "5000", 2024-04-20T17:07:55.146 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_learn_addr_from_peer": "true", 2024-04-20T17:07:55.146 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_max_accept_failures": "4", 2024-04-20T17:07:55.146 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_max_backoff": "15.000000", 2024-04-20T17:07:55.146 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_mon_client_mode": "secure crc", 2024-04-20T17:07:55.146 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_mon_cluster_mode": "secure crc", 2024-04-20T17:07:55.146 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_mon_service_mode": "secure crc", 2024-04-20T17:07:55.146 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_osd_compress_min_size": "1024", 2024-04-20T17:07:55.146 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_osd_compress_mode": "none", 2024-04-20T17:07:55.147 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_osd_compression_algorithm": "snappy", 2024-04-20T17:07:55.147 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-20T17:07:55.147 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_pq_min_cost": "65536", 2024-04-20T17:07:55.147 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_public_type": "", 2024-04-20T17:07:55.147 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_service_mode": "crc secure", 2024-04-20T17:07:55.147 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_tcp_listen_backlog": "512", 2024-04-20T17:07:55.147 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_tcp_nodelay": "true", 2024-04-20T17:07:55.147 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-20T17:07:55.147 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_tcp_rcvbuf": "0", 2024-04-20T17:07:55.147 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_type": "async+posix", 2024-04-20T17:07:55.147 INFO:teuthology.orchestra.run.smithi005.stdout: "no_config_file": "false", 2024-04-20T17:07:55.147 INFO:teuthology.orchestra.run.smithi005.stdout: "objecter_completion_locks_per_session": "32", 2024-04-20T17:07:55.147 INFO:teuthology.orchestra.run.smithi005.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-20T17:07:55.147 INFO:teuthology.orchestra.run.smithi005.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-20T17:07:55.147 INFO:teuthology.orchestra.run.smithi005.stdout: "objecter_inflight_ops": "1024", 2024-04-20T17:07:55.148 INFO:teuthology.orchestra.run.smithi005.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-20T17:07:55.148 INFO:teuthology.orchestra.run.smithi005.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-20T17:07:55.148 INFO:teuthology.orchestra.run.smithi005.stdout: "objecter_tick_interval": "5.000000", 2024-04-20T17:07:55.148 INFO:teuthology.orchestra.run.smithi005.stdout: "objecter_timeout": "10.000000", 2024-04-20T17:07:55.148 INFO:teuthology.orchestra.run.smithi005.stdout: "objectstore_blackhole": "false", 2024-04-20T17:07:55.148 INFO:teuthology.orchestra.run.smithi005.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-04-20T17:07:55.148 INFO:teuthology.orchestra.run.smithi005.stdout: "openssl_engine_opts": "", 2024-04-20T17:07:55.148 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_agent_delay_time": "5.000000", 2024-04-20T17:07:55.148 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_agent_hist_halflife": "1000", 2024-04-20T17:07:55.148 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_agent_max_low_ops": "2", 2024-04-20T17:07:55.148 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_agent_max_ops": "4", 2024-04-20T17:07:55.148 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-20T17:07:55.148 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-20T17:07:55.148 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_agent_slop": "0.020000", 2024-04-20T17:07:55.148 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-20T17:07:55.149 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-20T17:07:55.149 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_asio_thread_count": "2", 2024-04-20T17:07:55.149 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_async_recovery_min_cost": "100", 2024-04-20T17:07:55.149 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-20T17:07:55.149 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-20T17:07:55.149 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_backfill_scan_max": "512", 2024-04-20T17:07:55.149 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_backfill_scan_min": "64", 2024-04-20T17:07:55.149 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_backoff_on_degraded": "false", 2024-04-20T17:07:55.149 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_backoff_on_peering": "false", 2024-04-20T17:07:55.149 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_backoff_on_unfound": "true", 2024-04-20T17:07:55.149 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_beacon_report_interval": "300", 2024-04-20T17:07:55.149 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_bench_duration": "30", 2024-04-20T17:07:55.149 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-20T17:07:55.149 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_bench_max_block_size": "67108864", 2024-04-20T17:07:55.149 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-20T17:07:55.149 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_blkin_trace_all": "false", 2024-04-20T17:07:55.150 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_blocked_scrub_grace_period": "120", 2024-04-20T17:07:55.150 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-20T17:07:55.150 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-04-20T17:07:55.150 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-20T17:07:55.150 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_check_for_log_corruption": "false", 2024-04-20T17:07:55.150 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-20T17:07:55.150 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-20T17:07:55.150 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-04-20T17:07:55.150 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-20T17:07:55.150 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_class_update_on_start": "true", 2024-04-20T17:07:55.150 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_client_message_cap": "256", 2024-04-20T17:07:55.150 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_client_message_size_cap": "524288000", 2024-04-20T17:07:55.150 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_client_op_priority": "63", 2024-04-20T17:07:55.150 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_client_watch_timeout": "30", 2024-04-20T17:07:55.151 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_command_max_records": "256", 2024-04-20T17:07:55.151 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_compact_on_start": "false", 2024-04-20T17:07:55.151 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-20T17:07:55.151 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_crush_chooseleaf_type": "1", 2024-04-20T17:07:55.151 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-20T17:07:55.151 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_crush_update_on_start": "true", 2024-04-20T17:07:55.151 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_crush_update_weight_set": "true", 2024-04-20T17:07:55.151 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-20T17:07:55.151 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-20T17:07:55.151 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-20T17:07:55.151 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-20T17:07:55.151 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-20T17:07:55.151 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_feed_pullee": "-1", 2024-04-20T17:07:55.151 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-20T17:07:55.151 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-20T17:07:55.151 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-04-20T17:07:55.152 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_misdirected_ops": "false", 2024-04-20T17:07:55.152 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_no_acting_change": "false", 2024-04-20T17:07:55.152 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_no_purge_strays": "false", 2024-04-20T17:07:55.152 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_op_order": "false", 2024-04-20T17:07:55.152 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-20T17:07:55.152 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-20T17:07:55.152 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-20T17:07:55.152 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-20T17:07:55.152 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_shutdown": "false", 2024-04-20T17:07:55.152 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-20T17:07:55.152 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-20T17:07:55.152 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_trim_objects": "false", 2024-04-20T17:07:55.152 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-20T17:07:55.152 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-20T17:07:55.153 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_verify_snaps": "false", 2024-04-20T17:07:55.153 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-20T17:07:55.153 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-20T17:07:55.153 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_deep_scrub_keys": "1024", 2024-04-20T17:07:55.153 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-20T17:07:55.153 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-20T17:07:55.153 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-20T17:07:55.153 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_deep_scrub_stride": "524288", 2024-04-20T17:07:55.153 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-20T17:07:55.153 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_default_data_pool_replay_window": "45", 2024-04-20T17:07:55.153 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_default_notify_timeout": "30", 2024-04-20T17:07:55.153 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_delete_sleep": "0.000000", 2024-04-20T17:07:55.153 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-20T17:07:55.153 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-20T17:07:55.153 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-20T17:07:55.154 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_discard_disconnected_ops": "true", 2024-04-20T17:07:55.154 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_enable_op_tracker": "true", 2024-04-20T17:07:55.154 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-20T17:07:55.154 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_extblkdev_plugins": "vdo", 2024-04-20T17:07:55.154 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-20T17:07:55.154 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-20T17:07:55.154 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_fast_info": "true", 2024-04-20T17:07:55.154 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_fast_shutdown": "true", 2024-04-20T17:07:55.154 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-20T17:07:55.155 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-20T17:07:55.155 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-20T17:07:55.155 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-20T17:07:55.155 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-20T17:07:55.155 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_function_tracing": "false", 2024-04-20T17:07:55.155 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_heartbeat_grace": "20", 2024-04-20T17:07:55.155 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_heartbeat_interval": "6", 2024-04-20T17:07:55.155 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-20T17:07:55.155 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_heartbeat_min_peers": "10", 2024-04-20T17:07:55.155 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_heartbeat_min_size": "2000", 2024-04-20T17:07:55.155 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_heartbeat_stale": "600", 2024-04-20T17:07:55.155 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-20T17:07:55.155 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_hit_set_max_size": "100000", 2024-04-20T17:07:55.155 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_hit_set_min_size": "1000", 2024-04-20T17:07:55.155 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-20T17:07:55.156 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-20T17:07:55.156 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-20T17:07:55.156 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-20T17:07:55.156 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-20T17:07:55.156 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-20T17:07:55.156 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_journal_size": "5120", 2024-04-20T17:07:55.156 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_kill_backfill_at": "0", 2024-04-20T17:07:55.156 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-20T17:07:55.156 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_map_cache_size": "50", 2024-04-20T17:07:55.156 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_map_dedup": "true", 2024-04-20T17:07:55.156 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_map_message_max": "40", 2024-04-20T17:07:55.156 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-20T17:07:55.156 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_map_share_max_epochs": "40", 2024-04-20T17:07:55.156 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_attr_name_len": "100", 2024-04-20T17:07:55.157 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_attr_size": "0", 2024-04-20T17:07:55.157 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_backfills": "1", 2024-04-20T17:07:55.157 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_markdown_count": "5", 2024-04-20T17:07:55.157 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_markdown_period": "600", 2024-04-20T17:07:55.157 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_object_name_len": "2048", 2024-04-20T17:07:55.157 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_object_namespace_len": "256", 2024-04-20T17:07:55.157 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_object_size": "134217728", 2024-04-20T17:07:55.157 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-20T17:07:55.157 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-20T17:07:55.157 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_pg_blocked_by": "16", 2024-04-20T17:07:55.157 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_pg_log_entries": "10000", 2024-04-20T17:07:55.157 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-20T17:07:55.157 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_pgls": "1024", 2024-04-20T17:07:55.157 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_push_cost": "8388608", 2024-04-20T17:07:55.157 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_push_objects": "10", 2024-04-20T17:07:55.158 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_scrubs": "3", 2024-04-20T17:07:55.158 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-20T17:07:55.158 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_trimming_pgs": "2", 2024-04-20T17:07:55.158 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_write_op_reply_len": "64", 2024-04-20T17:07:55.158 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_write_size": "90", 2024-04-20T17:07:55.158 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-20T17:07:55.158 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-20T17:07:55.158 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-20T17:07:55.158 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-20T17:07:55.158 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-20T17:07:55.158 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-20T17:07:55.158 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-20T17:07:55.158 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-20T17:07:55.159 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_profile": "balanced", 2024-04-20T17:07:55.159 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-20T17:07:55.159 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-20T17:07:55.159 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-20T17:07:55.159 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-20T17:07:55.159 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-20T17:07:55.159 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-20T17:07:55.159 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-20T17:07:55.159 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-20T17:07:55.159 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-20T17:07:55.159 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-20T17:07:55.159 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-20T17:07:55.159 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_memory_base": "805306368", 2024-04-20T17:07:55.160 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_memory_cache_min": "134217728", 2024-04-20T17:07:55.160 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-20T17:07:55.160 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-20T17:07:55.160 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_memory_target": "4294967296", 2024-04-20T17:07:55.160 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_memory_target_autotune": "false", 2024-04-20T17:07:55.160 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-20T17:07:55.160 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_min_pg_log_entries": "250", 2024-04-20T17:07:55.160 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_min_recovery_priority": "0", 2024-04-20T17:07:55.160 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-20T17:07:55.160 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-20T17:07:55.160 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mon_report_interval": "5", 2024-04-20T17:07:55.160 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-20T17:07:55.160 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-20T17:07:55.160 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_num_cache_shards": "32", 2024-04-20T17:07:55.160 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_num_op_tracker_shard": "32", 2024-04-20T17:07:55.160 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_numa_auto_affinity": "true", 2024-04-20T17:07:55.161 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_numa_node": "-1", 2024-04-20T17:07:55.161 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_numa_prefer_iface": "true", 2024-04-20T17:07:55.161 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-04-20T17:07:55.161 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_objecter_finishers": "1", 2024-04-20T17:07:55.161 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_objectstore": "bluestore", 2024-04-20T17:07:55.161 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_objectstore_fuse": "false", 2024-04-20T17:07:55.161 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_objectstore_tracing": "false", 2024-04-20T17:07:55.161 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_complaint_time": "30.000000", 2024-04-20T17:07:55.161 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_history_duration": "600", 2024-04-20T17:07:55.161 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_history_size": "20", 2024-04-20T17:07:55.161 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_history_slow_op_size": "20", 2024-04-20T17:07:55.161 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-20T17:07:55.161 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_log_threshold": "5", 2024-04-20T17:07:55.161 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_num_shards": "0", 2024-04-20T17:07:55.162 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_num_shards_hdd": "5", 2024-04-20T17:07:55.162 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_num_shards_ssd": "8", 2024-04-20T17:07:55.162 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-20T17:07:55.162 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-20T17:07:55.162 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-20T17:07:55.162 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-20T17:07:55.162 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_pq_min_cost": "65536", 2024-04-20T17:07:55.162 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-20T17:07:55.162 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_queue_cut_off": "high", 2024-04-20T17:07:55.162 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-20T17:07:55.162 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_thread_timeout": "15", 2024-04-20T17:07:55.162 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_open_classes_on_start": "true", 2024-04-20T17:07:55.162 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_os_flags": "0", 2024-04-20T17:07:55.162 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_peering_op_priority": "255", 2024-04-20T17:07:55.162 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_delete_cost": "1048576", 2024-04-20T17:07:55.163 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_delete_priority": "5", 2024-04-20T17:07:55.163 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-20T17:07:55.163 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-20T17:07:55.163 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-20T17:07:55.163 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_log_trim_max": "10000", 2024-04-20T17:07:55.163 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_log_trim_min": "100", 2024-04-20T17:07:55.163 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-20T17:07:55.163 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-20T17:07:55.163 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-04-20T17:07:55.163 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-04-20T17:07:55.163 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-20T17:07:55.163 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-20T17:07:55.163 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-20T17:07:55.163 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-20T17:07:55.163 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-20T17:07:55.164 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-20T17:07:55.164 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_crimson": "false", 2024-04-20T17:07:55.164 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-20T17:07:55.164 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-20T17:07:55.164 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-20T17:07:55.164 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-20T17:07:55.164 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-20T17:07:55.164 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-20T17:07:55.164 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-20T17:07:55.164 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-20T17:07:55.164 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_flags": "0", 2024-04-20T17:07:55.164 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-20T17:07:55.164 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_min_size": "2", 2024-04-20T17:07:55.164 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-20T17:07:55.164 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_pg_num": "32", 2024-04-20T17:07:55.164 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_pgp_num": "0", 2024-04-20T17:07:55.165 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-20T17:07:55.165 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_read_ratio": "70", 2024-04-20T17:07:55.165 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_size": "2", 2024-04-20T17:07:55.165 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_type": "replicated", 2024-04-20T17:07:55.165 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-20T17:07:55.165 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-20T17:07:55.165 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_push_per_object_cost": "1000", 2024-04-20T17:07:55.165 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-20T17:07:55.165 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recover_clone_overlap": "true", 2024-04-20T17:07:55.165 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-20T17:07:55.165 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_cost": "20971520", 2024-04-20T17:07:55.165 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-20T17:07:55.165 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_max_active": "0", 2024-04-20T17:07:55.165 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-20T17:07:55.166 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-20T17:07:55.166 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-20T17:07:55.166 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-20T17:07:55.166 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_max_single_start": "1", 2024-04-20T17:07:55.166 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_op_priority": "3", 2024-04-20T17:07:55.166 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-20T17:07:55.166 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_priority": "5", 2024-04-20T17:07:55.166 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-20T17:07:55.166 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_sleep": "0.000000", 2024-04-20T17:07:55.166 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-20T17:07:55.166 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-20T17:07:55.166 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-20T17:07:55.167 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_repair_during_recovery": "false", 2024-04-20T17:07:55.167 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_requested_scrub_priority": "120", 2024-04-20T17:07:55.167 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-20T17:07:55.167 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-20T17:07:55.167 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_auto_repair": "false", 2024-04-20T17:07:55.167 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-20T17:07:55.167 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-20T17:07:55.167 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_begin_hour": "0", 2024-04-20T17:07:55.167 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_begin_week_day": "0", 2024-04-20T17:07:55.167 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_chunk_max": "25", 2024-04-20T17:07:55.167 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_chunk_min": "5", 2024-04-20T17:07:55.167 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_cost": "52428800", 2024-04-20T17:07:55.167 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-04-20T17:07:55.168 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_during_recovery": "false", 2024-04-20T17:07:55.168 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_end_hour": "0", 2024-04-20T17:07:55.168 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_end_week_day": "0", 2024-04-20T17:07:55.168 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_event_cost": "4096", 2024-04-20T17:07:55.168 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-20T17:07:55.168 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-20T17:07:55.168 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_invalid_stats": "true", 2024-04-20T17:07:55.168 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-20T17:07:55.168 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-20T17:07:55.168 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_max_preemptions": "5", 2024-04-20T17:07:55.168 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-20T17:07:55.168 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_priority": "5", 2024-04-20T17:07:55.168 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_reservation_timeout": "300000", 2024-04-20T17:07:55.169 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_sleep": "0.000000", 2024-04-20T17:07:55.169 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-04-20T17:07:55.169 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-04-20T17:07:55.169 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-20T17:07:55.169 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-04-20T17:07:55.169 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_skip_data_digest": "false", 2024-04-20T17:07:55.169 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_smart_report_timeout": "5", 2024-04-20T17:07:55.169 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_snap_trim_cost": "1048576", 2024-04-20T17:07:55.169 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_snap_trim_priority": "5", 2024-04-20T17:07:55.169 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-20T17:07:55.169 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-20T17:07:55.169 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-20T17:07:55.170 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-20T17:07:55.170 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-04-20T17:07:55.170 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_stats_update_period_scrubbing": "15", 2024-04-20T17:07:55.170 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-20T17:07:55.170 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_target_transaction_size": "30", 2024-04-20T17:07:55.170 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-20T17:07:55.170 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-20T17:07:55.170 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-20T17:07:55.170 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-20T17:07:55.170 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-20T17:07:55.170 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-20T17:07:55.170 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-20T17:07:55.170 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-20T17:07:55.171 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-20T17:07:55.171 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-20T17:07:55.171 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tracing": "false", 2024-04-20T17:07:55.171 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_use_stale_snap": "false", 2024-04-20T17:07:55.171 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-20T17:07:55.171 INFO:teuthology.orchestra.run.smithi005.stdout: "osdc_blkin_trace_all": "false", 2024-04-20T17:07:55.171 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_kill_at": "0", 2024-04-20T17:07:55.171 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_max_join_drift": "10", 2024-04-20T17:07:55.171 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_min": "500", 2024-04-20T17:07:55.171 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_min_wait": "0.050000", 2024-04-20T17:07:55.171 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_propose_interval": "1.000000", 2024-04-20T17:07:55.171 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_service_trim_max": "500", 2024-04-20T17:07:55.171 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-20T17:07:55.172 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_service_trim_min": "10", 2024-04-20T17:07:55.172 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_stash_full_interval": "25", 2024-04-20T17:07:55.172 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_trim_max": "500", 2024-04-20T17:07:55.172 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_trim_min": "250", 2024-04-20T17:07:55.172 INFO:teuthology.orchestra.run.smithi005.stdout: "perf": "true", 2024-04-20T17:07:55.172 INFO:teuthology.orchestra.run.smithi005.stdout: "pid_file": "", 2024-04-20T17:07:55.172 INFO:teuthology.orchestra.run.smithi005.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-20T17:07:55.172 INFO:teuthology.orchestra.run.smithi005.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-04-20T17:07:55.172 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "-", 2024-04-20T17:07:55.172 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrv": "", 2024-04-20T17:07:55.172 INFO:teuthology.orchestra.run.smithi005.stdout: "public_bind_addr": "-", 2024-04-20T17:07:55.172 INFO:teuthology.orchestra.run.smithi005.stdout: "public_network": "", 2024-04-20T17:07:55.172 INFO:teuthology.orchestra.run.smithi005.stdout: "public_network_interface": "", 2024-04-20T17:07:55.172 INFO:teuthology.orchestra.run.smithi005.stdout: "qat_compressor_enabled": "false", 2024-04-20T17:07:55.172 INFO:teuthology.orchestra.run.smithi005.stdout: "qat_compressor_session_max_number": "256", 2024-04-20T17:07:55.173 INFO:teuthology.orchestra.run.smithi005.stdout: "rados_mon_op_timeout": "0", 2024-04-20T17:07:55.173 INFO:teuthology.orchestra.run.smithi005.stdout: "rados_osd_op_timeout": "0", 2024-04-20T17:07:55.173 INFO:teuthology.orchestra.run.smithi005.stdout: "rados_tracing": "false", 2024-04-20T17:07:55.173 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_atime_update_interval": "60", 2024-04-20T17:07:55.173 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-20T17:07:55.173 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_balance_parent_reads": "false", 2024-04-20T17:07:55.173 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_balance_snap_reads": "false", 2024-04-20T17:07:55.173 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_blkin_trace_all": "false", 2024-04-20T17:07:55.173 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-20T17:07:55.173 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-20T17:07:55.173 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_cache": "true", 2024-04-20T17:07:55.173 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-20T17:07:55.173 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-20T17:07:55.173 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-20T17:07:55.174 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-20T17:07:55.174 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_cache_policy": "writearound", 2024-04-20T17:07:55.174 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_cache_size": "33554432", 2024-04-20T17:07:55.174 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-20T17:07:55.174 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-20T17:07:55.174 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_clone_copy_on_read": "false", 2024-04-20T17:07:55.174 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_compression_hint": "none", 2024-04-20T17:07:55.174 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_concurrent_management_ops": "10", 2024-04-20T17:07:55.174 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-20T17:07:55.174 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_clone_format": "auto", 2024-04-20T17:07:55.174 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_data_pool": "", 2024-04-20T17:07:55.174 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_features": "61", 2024-04-20T17:07:55.174 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_format": "2", 2024-04-20T17:07:55.174 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_map_options": "", 2024-04-20T17:07:55.175 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_order": "22", 2024-04-20T17:07:55.175 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_pool": "rbd", 2024-04-20T17:07:55.175 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-20T17:07:55.175 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_stripe_count": "0", 2024-04-20T17:07:55.175 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_stripe_unit": "0", 2024-04-20T17:07:55.175 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-20T17:07:55.175 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-20T17:07:55.175 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-20T17:07:55.175 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_enable_alloc_hint": "true", 2024-04-20T17:07:55.175 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-20T17:07:55.175 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_io_scheduler": "simple", 2024-04-20T17:07:55.175 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-20T17:07:55.175 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-20T17:07:55.175 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-20T17:07:55.175 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-20T17:07:55.176 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-20T17:07:55.176 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-20T17:07:55.176 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-20T17:07:55.176 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-20T17:07:55.176 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-20T17:07:55.176 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_order": "24", 2024-04-20T17:07:55.176 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_pool": "", 2024-04-20T17:07:55.176 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_splay_width": "4", 2024-04-20T17:07:55.176 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_localize_parent_reads": "false", 2024-04-20T17:07:55.176 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_localize_snap_reads": "false", 2024-04-20T17:07:55.176 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-20T17:07:55.176 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-20T17:07:55.176 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-20T17:07:55.176 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-20T17:07:55.176 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-20T17:07:55.177 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-20T17:07:55.177 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-20T17:07:55.177 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-20T17:07:55.177 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-20T17:07:55.177 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-20T17:07:55.177 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-20T17:07:55.177 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-20T17:07:55.177 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-20T17:07:55.177 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-20T17:07:55.177 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-20T17:07:55.177 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-20T17:07:55.177 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-20T17:07:55.177 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-20T17:07:55.177 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-20T17:07:55.177 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-20T17:07:55.178 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-20T17:07:55.178 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-20T17:07:55.178 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-20T17:07:55.178 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-20T17:07:55.178 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-20T17:07:55.178 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-20T17:07:55.178 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-20T17:07:55.178 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-20T17:07:55.178 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-20T17:07:55.178 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-20T17:07:55.178 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-20T17:07:55.178 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mtime_update_interval": "60", 2024-04-20T17:07:55.178 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_non_blocking_aio": "true", 2024-04-20T17:07:55.178 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_op_thread_timeout": "60", 2024-04-20T17:07:55.179 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_op_threads": "1", 2024-04-20T17:07:55.179 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_parent_cache_enabled": "false", 2024-04-20T17:07:55.179 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-20T17:07:55.179 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-20T17:07:55.179 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-20T17:07:55.179 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_plugins": "", 2024-04-20T17:07:55.179 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_bps_burst": "0", 2024-04-20T17:07:55.179 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-20T17:07:55.179 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_bps_limit": "0", 2024-04-20T17:07:55.179 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_exclude_ops": "0", 2024-04-20T17:07:55.179 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_iops_burst": "0", 2024-04-20T17:07:55.179 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-20T17:07:55.179 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_iops_limit": "0", 2024-04-20T17:07:55.179 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-20T17:07:55.180 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-20T17:07:55.180 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-20T17:07:55.180 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-20T17:07:55.180 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-20T17:07:55.180 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-20T17:07:55.180 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-20T17:07:55.180 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-20T17:07:55.180 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-20T17:07:55.180 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-20T17:07:55.180 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-20T17:07:55.180 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-20T17:07:55.180 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-20T17:07:55.180 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-20T17:07:55.180 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_read_from_replica_policy": "default", 2024-04-20T17:07:55.181 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-20T17:07:55.181 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-20T17:07:55.181 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-20T17:07:55.181 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-20T17:07:55.181 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_skip_partial_discard": "true", 2024-04-20T17:07:55.181 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-20T17:07:55.181 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_tracing": "false", 2024-04-20T17:07:55.181 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_validate_names": "true", 2024-04-20T17:07:55.181 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_validate_pool": "true", 2024-04-20T17:07:55.181 INFO:teuthology.orchestra.run.smithi005.stdout: "restapi_base_url": "", 2024-04-20T17:07:55.181 INFO:teuthology.orchestra.run.smithi005.stdout: "restapi_log_level": "", 2024-04-20T17:07:55.181 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_acl_grants_max_num": "100", 2024-04-20T17:07:55.181 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_admin_entry": "admin", 2024-04-20T17:07:55.181 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-04-20T17:07:55.181 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_backend_store": "rados", 2024-04-20T17:07:55.182 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_barbican_url": "", 2024-04-20T17:07:55.182 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_beast_enable_async": "true", 2024-04-20T17:07:55.182 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_bucket_counters_cache": "false", 2024-04-20T17:07:55.182 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-04-20T17:07:55.182 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-20T17:07:55.182 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-20T17:07:55.182 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-20T17:07:55.182 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-04-20T17:07:55.182 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-20T17:07:55.182 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-20T17:07:55.182 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-04-20T17:07:55.182 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_cache_enabled": "true", 2024-04-20T17:07:55.182 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_cache_expiry_interval": "900", 2024-04-20T17:07:55.182 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_cache_lru_size": "10000", 2024-04-20T17:07:55.183 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_config_store": "rados", 2024-04-20T17:07:55.183 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_content_length_compat": "false", 2024-04-20T17:07:55.183 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_copy_obj_progress": "true", 2024-04-20T17:07:55.183 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-20T17:07:55.183 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_cors_rules_max_num": "100", 2024-04-20T17:07:55.183 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_cross_domain_policy": "", 2024-04-20T17:07:55.183 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-20T17:07:55.183 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_kmip_addr": "", 2024-04-20T17:07:55.183 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-20T17:07:55.183 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-20T17:07:55.183 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-20T17:07:55.183 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-20T17:07:55.183 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_kmip_password": "", 2024-04-20T17:07:55.183 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-20T17:07:55.184 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_kmip_username": "", 2024-04-20T17:07:55.184 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_require_ssl": "true", 2024-04-20T17:07:55.184 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-20T17:07:55.184 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-20T17:07:55.184 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-20T17:07:55.184 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-20T17:07:55.184 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-20T17:07:55.184 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-20T17:07:55.184 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-20T17:07:55.184 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-20T17:07:55.184 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-20T17:07:55.184 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-20T17:07:55.184 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-20T17:07:55.185 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-20T17:07:55.185 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-20T17:07:55.185 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-20T17:07:55.185 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-20T17:07:55.185 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_addr": "", 2024-04-20T17:07:55.185 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_auth": "token", 2024-04-20T17:07:55.185 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_namespace": "", 2024-04-20T17:07:55.185 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_prefix": "", 2024-04-20T17:07:55.185 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-20T17:07:55.185 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-20T17:07:55.185 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-20T17:07:55.185 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-20T17:07:55.185 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_token_file": "", 2024-04-20T17:07:55.185 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-20T17:07:55.185 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_curl_buffersize": "524288", 2024-04-20T17:07:55.185 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-20T17:07:55.186 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_curl_low_speed_time": "300", 2024-04-20T17:07:55.186 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_curl_tcp_keepalive": "0", 2024-04-20T17:07:55.186 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-20T17:07:55.186 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-20T17:07:55.186 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-20T17:07:55.186 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-20T17:07:55.186 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-20T17:07:55.186 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-20T17:07:55.186 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-20T17:07:55.186 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-20T17:07:55.186 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-20T17:07:55.186 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d4n_host": "127.0.0.1", 2024-04-20T17:07:55.186 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-04-20T17:07:55.187 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-04-20T17:07:55.187 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-04-20T17:07:55.187 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d4n_l1_fadvise": "4", 2024-04-20T17:07:55.187 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-04-20T17:07:55.187 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-04-20T17:07:55.187 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d4n_port": "6379", 2024-04-20T17:07:55.187 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-20T17:07:55.187 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_data_log_changes_size": "1000", 2024-04-20T17:07:55.187 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_data_log_num_shards": "128", 2024-04-20T17:07:55.187 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-20T17:07:55.187 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_data_log_window": "30", 2024-04-20T17:07:55.187 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_data_notify_interval_msec": "0", 2024-04-20T17:07:55.187 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_data_sync_poll_interval": "20", 2024-04-20T17:07:55.187 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_data_sync_spawn_window": "20", 2024-04-20T17:07:55.188 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-04-20T17:07:55.188 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-20T17:07:55.188 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-20T17:07:55.188 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-20T17:07:55.188 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-20T17:07:55.188 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-20T17:07:55.188 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-20T17:07:55.188 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-20T17:07:55.188 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-20T17:07:55.188 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-20T17:07:55.188 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-20T17:07:55.188 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-20T17:07:55.188 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-20T17:07:55.188 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-20T17:07:55.189 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-20T17:07:55.189 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-20T17:07:55.189 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-20T17:07:55.189 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-20T17:07:55.189 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-20T17:07:55.189 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-20T17:07:55.189 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-20T17:07:55.189 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-20T17:07:55.189 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dns_name": "", 2024-04-20T17:07:55.189 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dns_s3website_name": "", 2024-04-20T17:07:55.189 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dynamic_resharding": "true", 2024-04-20T17:07:55.189 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-20T17:07:55.189 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_enable_gc_threads": "true", 2024-04-20T17:07:55.189 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_enable_lc_threads": "true", 2024-04-20T17:07:55.189 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_enable_ops_log": "false", 2024-04-20T17:07:55.190 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_enable_quota_threads": "true", 2024-04-20T17:07:55.190 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_enable_static_website": "false", 2024-04-20T17:07:55.190 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_enable_usage_log": "false", 2024-04-20T17:07:55.190 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_enforce_swift_acls": "true", 2024-04-20T17:07:55.190 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_exit_timeout_secs": "120", 2024-04-20T17:07:55.190 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_expose_bucket": "false", 2024-04-20T17:07:55.190 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_extended_http_attrs": "", 2024-04-20T17:07:55.190 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_filter": "none", 2024-04-20T17:07:55.190 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-20T17:07:55.190 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_frontends": "beast port=7480", 2024-04-20T17:07:55.190 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-20T17:07:55.190 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_gc_max_deferred": "50", 2024-04-20T17:07:55.190 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-20T17:07:55.190 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_gc_max_objs": "32", 2024-04-20T17:07:55.191 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-20T17:07:55.191 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-20T17:07:55.191 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-20T17:07:55.191 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-20T17:07:55.191 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_gc_processor_period": "3600", 2024-04-20T17:07:55.191 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-20T17:07:55.191 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-20T17:07:55.191 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-20T17:07:55.191 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-20T17:07:55.191 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_init_timeout": "300", 2024-04-20T17:07:55.191 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-20T17:07:55.191 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-04-20T17:07:55.191 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_kafka_connection_idle": "300", 2024-04-20T17:07:55.191 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_kafka_message_timeout": "5000", 2024-04-20T17:07:55.191 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_kafka_sleep_timeout": "10", 2024-04-20T17:07:55.192 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-20T17:07:55.192 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-04-20T17:07:55.192 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-20T17:07:55.192 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_admin_domain": "", 2024-04-20T17:07:55.192 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_admin_password": "", 2024-04-20T17:07:55.192 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_admin_password_path": "", 2024-04-20T17:07:55.192 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_admin_project": "", 2024-04-20T17:07:55.192 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_admin_tenant": "", 2024-04-20T17:07:55.192 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_admin_token": "", 2024-04-20T17:07:55.192 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_admin_token_path": "", 2024-04-20T17:07:55.192 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_admin_user": "", 2024-04-20T17:07:55.192 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_api_version": "2", 2024-04-20T17:07:55.193 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_barbican_domain": "", 2024-04-20T17:07:55.193 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_barbican_password": "", 2024-04-20T17:07:55.193 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_barbican_project": "", 2024-04-20T17:07:55.193 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-20T17:07:55.193 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_barbican_user": "", 2024-04-20T17:07:55.193 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-20T17:07:55.193 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-20T17:07:55.193 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-20T17:07:55.193 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-20T17:07:55.193 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-20T17:07:55.193 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_url": "", 2024-04-20T17:07:55.193 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-20T17:07:55.193 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lc_debug_interval": "-1", 2024-04-20T17:07:55.194 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lc_lock_max_time": "90", 2024-04-20T17:07:55.194 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lc_max_objs": "32", 2024-04-20T17:07:55.194 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lc_max_rules": "1000", 2024-04-20T17:07:55.194 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lc_max_worker": "3", 2024-04-20T17:07:55.194 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-20T17:07:55.194 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lc_thread_delay": "0", 2024-04-20T17:07:55.194 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-20T17:07:55.194 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ldap_dnattr": "uid", 2024-04-20T17:07:55.194 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-20T17:07:55.194 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ldap_searchfilter": "", 2024-04-20T17:07:55.194 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-20T17:07:55.194 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ldap_uri": "", 2024-04-20T17:07:55.194 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lfuda_sync_frequency": "60", 2024-04-20T17:07:55.194 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-20T17:07:55.195 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-20T17:07:55.195 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-20T17:07:55.195 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_local_cache_address": "127.0.0.1:6379", 2024-04-20T17:07:55.195 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_log_http_headers": "", 2024-04-20T17:07:55.195 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-20T17:07:55.195 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-20T17:07:55.195 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_log_object_name_utc": "false", 2024-04-20T17:07:55.195 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-04-20T17:07:55.195 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-04-20T17:07:55.195 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_attr_name_len": "0", 2024-04-20T17:07:55.195 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_attr_size": "0", 2024-04-20T17:07:55.196 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-20T17:07:55.196 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_chunk_size": "4194304", 2024-04-20T17:07:55.196 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-20T17:07:55.196 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-04-20T17:07:55.196 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-20T17:07:55.196 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_listing_results": "1000", 2024-04-20T17:07:55.196 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_notify_retries": "10", 2024-04-20T17:07:55.196 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-20T17:07:55.196 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_put_param_size": "1048576", 2024-04-20T17:07:55.196 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_put_size": "5368709120", 2024-04-20T17:07:55.196 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_slo_entries": "1000", 2024-04-20T17:07:55.196 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_md_log_max_shards": "64", 2024-04-20T17:07:55.196 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-20T17:07:55.196 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_meta_sync_poll_interval": "20", 2024-04-20T17:07:55.197 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_meta_sync_spawn_window": "20", 2024-04-20T17:07:55.197 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-20T17:07:55.197 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_mp_lock_max_time": "600", 2024-04-20T17:07:55.197 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-20T17:07:55.197 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-20T17:07:55.197 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-20T17:07:55.197 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-20T17:07:55.197 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-20T17:07:55.197 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-04-20T17:07:55.197 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-20T17:07:55.197 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-20T17:07:55.197 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_max_gc": "300", 2024-04-20T17:07:55.197 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-20T17:07:55.198 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-20T17:07:55.198 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-20T17:07:55.198 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-20T17:07:55.198 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-20T17:07:55.198 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-20T17:07:55.198 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-20T17:07:55.198 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_num_async_rados_threads": "32", 2024-04-20T17:07:55.198 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_num_control_oids": "8", 2024-04-20T17:07:55.198 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_numa_node": "-1", 2024-04-20T17:07:55.198 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-20T17:07:55.198 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-20T17:07:55.198 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_objexp_chunk_size": "100", 2024-04-20T17:07:55.198 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_objexp_gc_interval": "600", 2024-04-20T17:07:55.199 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-20T17:07:55.199 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-20T17:07:55.199 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-20T17:07:55.199 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_op_thread_timeout": "600", 2024-04-20T17:07:55.199 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_op_tracing": "false", 2024-04-20T17:07:55.199 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_opa_token": "", 2024-04-20T17:07:55.199 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_opa_url": "", 2024-04-20T17:07:55.199 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_opa_verify_ssl": "true", 2024-04-20T17:07:55.199 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-20T17:07:55.199 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-04-20T17:07:55.199 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ops_log_rados": "false", 2024-04-20T17:07:55.199 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ops_log_socket_path": "", 2024-04-20T17:07:55.199 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-20T17:07:55.200 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-04-20T17:07:55.200 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-20T17:07:55.200 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_period_push_interval": "2.000000", 2024-04-20T17:07:55.200 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-20T17:07:55.200 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-20T17:07:55.200 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-04-20T17:07:55.200 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-04-20T17:07:55.200 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_posix_cache_lanes": "3", 2024-04-20T17:07:55.200 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-04-20T17:07:55.200 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_posix_cache_max_buckets": "100", 2024-04-20T17:07:55.200 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_posix_cache_partitions": "3", 2024-04-20T17:07:55.200 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-04-20T17:07:55.200 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_print_continue": "true", 2024-04-20T17:07:55.200 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-20T17:07:55.201 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-20T17:07:55.201 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-20T17:07:55.201 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-20T17:07:55.201 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-20T17:07:55.201 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_rados_tracing": "false", 2024-04-20T17:07:55.201 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_realm": "", 2024-04-20T17:07:55.201 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_realm_id": "", 2024-04-20T17:07:55.201 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-20T17:07:55.201 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_region": "", 2024-04-20T17:07:55.201 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-20T17:07:55.201 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-20T17:07:55.201 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-20T17:07:55.201 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-20T17:07:55.202 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_request_uri": "", 2024-04-20T17:07:55.202 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_reshard_batch_size": "64", 2024-04-20T17:07:55.202 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-20T17:07:55.202 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_reshard_max_aio": "128", 2024-04-20T17:07:55.202 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_reshard_num_logs": "16", 2024-04-20T17:07:55.202 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_reshard_thread_interval": "600", 2024-04-20T17:07:55.202 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_resolve_cname": "false", 2024-04-20T17:07:55.202 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-20T17:07:55.202 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_run_sync_thread": "true", 2024-04-20T17:07:55.202 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-04-20T17:07:55.202 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-20T17:07:55.202 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-20T17:07:55.202 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-20T17:07:55.202 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-20T17:07:55.203 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-20T17:07:55.203 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-20T17:07:55.203 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-20T17:07:55.203 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-20T17:07:55.203 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_scheduler_type": "throttler", 2024-04-20T17:07:55.203 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_script_uri": "", 2024-04-20T17:07:55.203 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_service_provider_name": "", 2024-04-20T17:07:55.203 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-20T17:07:55.203 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sts_client_id": "", 2024-04-20T17:07:55.203 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sts_client_secret": "", 2024-04-20T17:07:55.203 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sts_entry": "sts", 2024-04-20T17:07:55.203 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sts_key": "sts", 2024-04-20T17:07:55.203 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-20T17:07:55.203 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sts_min_session_duration": "900", 2024-04-20T17:07:55.203 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sts_token_introspection_url": "", 2024-04-20T17:07:55.204 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_account_in_url": "false", 2024-04-20T17:07:55.204 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_auth_entry": "auth", 2024-04-20T17:07:55.204 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_auth_url": "", 2024-04-20T17:07:55.204 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_custom_header": "", 2024-04-20T17:07:55.204 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-20T17:07:55.204 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_need_stats": "true", 2024-04-20T17:07:55.204 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_tenant_name": "", 2024-04-20T17:07:55.204 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_token_expiration": "86400", 2024-04-20T17:07:55.204 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_url": "", 2024-04-20T17:07:55.204 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_url_prefix": "swift", 2024-04-20T17:07:55.204 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-20T17:07:55.204 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-04-20T17:07:55.204 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-20T17:07:55.204 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_lease_period": "120", 2024-04-20T17:07:55.205 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-20T17:07:55.205 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-20T17:07:55.205 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-20T17:07:55.205 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-20T17:07:55.205 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-20T17:07:55.205 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-20T17:07:55.205 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-20T17:07:55.205 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-20T17:07:55.205 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-20T17:07:55.205 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_thread_pool_size": "512", 2024-04-20T17:07:55.205 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_topic_persistency_max_retries": "0", 2024-04-20T17:07:55.205 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-04-20T17:07:55.205 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-04-20T17:07:55.205 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_topic_require_publish_policy": "false", 2024-04-20T17:07:55.205 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_torrent_comment": "", 2024-04-20T17:07:55.206 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_torrent_createby": "", 2024-04-20T17:07:55.206 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_torrent_encoding": "", 2024-04-20T17:07:55.206 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_torrent_flag": "false", 2024-04-20T17:07:55.206 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_torrent_max_size": "5368709120", 2024-04-20T17:07:55.206 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_torrent_origin": "", 2024-04-20T17:07:55.206 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-20T17:07:55.206 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_torrent_tracker": "", 2024-04-20T17:07:55.206 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_trust_forwarded_https": "false", 2024-04-20T17:07:55.206 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-20T17:07:55.206 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-20T17:07:55.206 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_usage_max_shards": "32", 2024-04-20T17:07:55.206 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_usage_max_user_shards": "1", 2024-04-20T17:07:55.206 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_use_opa_authz": "false", 2024-04-20T17:07:55.207 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_counters_cache": "false", 2024-04-20T17:07:55.207 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_counters_cache_size": "10000", 2024-04-20T17:07:55.207 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-20T17:07:55.207 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-20T17:07:55.207 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_max_buckets": "1000", 2024-04-20T17:07:55.207 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_policies_max_num": "100", 2024-04-20T17:07:55.207 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-20T17:07:55.207 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-20T17:07:55.207 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-20T17:07:55.207 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-20T17:07:55.207 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_unique_email": "true", 2024-04-20T17:07:55.207 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_verify_ssl": "true", 2024-04-20T17:07:55.207 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-20T17:07:55.207 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_zone": "", 2024-04-20T17:07:55.207 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_zone_id": "", 2024-04-20T17:07:55.207 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-20T17:07:55.207 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_zonegroup": "", 2024-04-20T17:07:55.208 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_zonegroup_id": "", 2024-04-20T17:07:55.208 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-20T17:07:55.208 INFO:teuthology.orchestra.run.smithi005.stdout: "rgwlc_auto_session_clear": "true", 2024-04-20T17:07:55.208 INFO:teuthology.orchestra.run.smithi005.stdout: "rgwlc_skip_bucket_step": "false", 2024-04-20T17:07:55.208 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_block_size": "4096", 2024-04-20T17:07:55.208 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-20T17:07:55.208 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-20T17:07:55.208 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-20T17:07:55.208 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-20T17:07:55.208 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-20T17:07:55.208 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_cache_size": "536870912", 2024-04-20T17:07:55.208 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-20T17:07:55.208 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-20T17:07:55.208 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-20T17:07:55.208 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-20T17:07:55.209 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-20T17:07:55.209 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-20T17:07:55.209 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-20T17:07:55.209 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-20T17:07:55.209 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_index_type": "binary_search", 2024-04-20T17:07:55.209 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-20T17:07:55.209 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-20T17:07:55.209 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_partition_filters": "false", 2024-04-20T17:07:55.209 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_perf": "false", 2024-04-20T17:07:55.209 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-20T17:07:55.209 INFO:teuthology.orchestra.run.smithi005.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-20T17:07:55.209 INFO:teuthology.orchestra.run.smithi005.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-20T17:07:55.209 INFO:teuthology.orchestra.run.smithi005.stdout: "run_dir": "/var/run/ceph", 2024-04-20T17:07:55.209 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_block_create": "true", 2024-04-20T17:07:55.209 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_cache_lru_size": "67108864", 2024-04-20T17:07:55.210 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_cbjournal_size": "5368709120", 2024-04-20T17:07:55.210 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_data_delta_based_overwrite": "0", 2024-04-20T17:07:55.210 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_default_max_object_size": "16777216", 2024-04-20T17:07:55.210 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-20T17:07:55.210 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_device_size": "53687091200", 2024-04-20T17:07:55.210 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_journal_batch_capacity": "16", 2024-04-20T17:07:55.210 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-20T17:07:55.210 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-20T17:07:55.210 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-20T17:07:55.210 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_main_device_type": "SSD", 2024-04-20T17:07:55.210 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_max_concurrent_transactions": "8", 2024-04-20T17:07:55.210 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-04-20T17:07:55.210 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-04-20T17:07:55.210 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-04-20T17:07:55.211 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-04-20T17:07:55.211 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_segment_size": "67108864", 2024-04-20T17:07:55.211 INFO:teuthology.orchestra.run.smithi005.stdout: "set_keepcaps": "false", 2024-04-20T17:07:55.211 INFO:teuthology.orchestra.run.smithi005.stdout: "setgroup": "ceph", 2024-04-20T17:07:55.211 INFO:teuthology.orchestra.run.smithi005.stdout: "setuser": "ceph", 2024-04-20T17:07:55.211 INFO:teuthology.orchestra.run.smithi005.stdout: "setuser_match_path": "", 2024-04-20T17:07:55.211 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-20T17:07:55.211 INFO:teuthology.orchestra.run.smithi005.stdout: "thp": "false", 2024-04-20T17:07:55.211 INFO:teuthology.orchestra.run.smithi005.stdout: "threadpool_default_timeout": "60", 2024-04-20T17:07:55.211 INFO:teuthology.orchestra.run.smithi005.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-20T17:07:55.211 INFO:teuthology.orchestra.run.smithi005.stdout: "throttler_perf_counter": "true" 2024-04-20T17:07:55.211 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:07:55.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:54 smithi106 ceph-mon[44423]: pgmap v356: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:55.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:54 smithi171 ceph-mon[44189]: pgmap v356: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:55.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:54 smithi005 ceph-mon[39562]: pgmap v356: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:55.833 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-20T17:07:56.120 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:07:57.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:56 smithi005 ceph-mon[39562]: pgmap v357: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:57.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:56 smithi106 ceph-mon[44423]: pgmap v357: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:57.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:56 smithi171 ceph-mon[44189]: pgmap v357: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:57.831 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a: {} 2024-04-20T17:07:57.842 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c: {} 2024-04-20T17:07:57.854 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b: {} 2024-04-20T17:07:57.855 INFO:teuthology.orchestra.run.smithi005.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-20T17:07:57.855 INFO:teuthology.orchestra.run.smithi005.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-20T17:07:57.855 INFO:teuthology.orchestra.run.smithi005.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-20T17:07:58.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:57 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/301068437' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T17:07:58.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:57 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/301068437' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T17:07:58.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:57 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/301068437' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T17:07:58.553 INFO:teuthology.run_tasks:Running task full_sequential... 2024-04-20T17:07:58.570 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-20T17:07:58.572 INFO:tasks.radosbench:Beginning radosbench... 2024-04-20T17:07:58.572 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-04-20T17:07:58.572 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-04-20T17:07:58.573 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-20T17:07:58.573 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-20T17:07:58.573 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:07:58.573 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-04-20T17:07:58.573 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-20T17:07:58.573 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-20T17:07:58.574 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-04-20T17:07:58.574 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-04-20T17:07:58.574 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:07:58.575 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-20T17:07:58.575 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:07:58.575 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-04-20T17:07:58.576 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-20T17:07:58.576 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-04-20T17:07:58.576 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:07:58.620 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:07:58.642 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:07:58.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:07:58.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:07:58.750 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:07:58.791 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:07:58.823 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:07:58.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:07:58.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:07:58.928 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:07:58.937 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:07:58.955 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:07:58.975 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:07:59.003 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:07:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:07:58 smithi005 ceph-mon[39562]: pgmap v358: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:59.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:07:58 smithi106 ceph-mon[44423]: pgmap v358: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:07:59.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:07:58 smithi171 ceph-mon[44189]: pgmap v358: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:00.184 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:08:00.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:00.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:08:00.288 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:08:00.323 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:00.345 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:08:00.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:00.348 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:08:00.449 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:08:00.478 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:00.928 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:08:00.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:00.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:08:00.933 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:08:00.933 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:08:01.032 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:08:01.060 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:00 smithi005 ceph-mon[39562]: pgmap v359: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:00 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:08:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:00 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:08:01.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:00 smithi106 ceph-mon[44423]: pgmap v359: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:01.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:00 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:08:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:00 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:08:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:00 smithi171 ceph-mon[44189]: pgmap v359: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:00 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:08:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:00 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:08:02.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:01 smithi106 ceph-mon[44423]: osdmap e113: 12 total, 12 up, 12 in 2024-04-20T17:08:02.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:01 smithi171 ceph-mon[44189]: osdmap e113: 12 total, 12 up, 12 in 2024-04-20T17:08:02.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:01 smithi005 ceph-mon[39562]: osdmap e113: 12 total, 12 up, 12 in 2024-04-20T17:08:03.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:02 smithi106 ceph-mon[44423]: pgmap v361: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:03.261 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:02 smithi005 ceph-mon[39562]: pgmap v361: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:03.311 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:08:03.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:03.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:08:03.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:02 smithi171 ceph-mon[44189]: pgmap v361: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:03.416 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:08:03.449 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:03.470 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:08:03.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:03.473 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:08:03.575 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:08:03.603 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:03.623 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:08:03.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:03.626 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:08:03.727 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:08:03.755 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:03.774 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:08:03.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:03.777 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:08:03.878 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:08:03.904 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:03.923 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:08:03.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:03.926 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:08:04.027 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:08:04.053 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:04.072 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:08:04.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:04.075 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:08:04.176 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:08:04.208 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:05.000 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'unique_pool_0' created 2024-04-20T17:08:05.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:04 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:08:05.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:04 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:08:05.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:04 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:08:05.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:04 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:08:05.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:04 smithi106 ceph-mon[44423]: pgmap v362: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:05.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:04 smithi106 ceph-mon[44423]: osdmap e114: 12 total, 12 up, 12 in 2024-04-20T17:08:05.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:04 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3213814244' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-04-20T17:08:05.300 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:04 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:08:05.300 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:04 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:08:05.300 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:04 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:08:05.300 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:04 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:08:05.300 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:04 smithi005 ceph-mon[39562]: pgmap v362: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:05.300 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:04 smithi005 ceph-mon[39562]: osdmap e114: 12 total, 12 up, 12 in 2024-04-20T17:08:05.300 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:04 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3213814244' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-04-20T17:08:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:04 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:08:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:04 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:08:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:04 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:08:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:04 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:08:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:04 smithi171 ceph-mon[44189]: pgmap v362: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:04 smithi171 ceph-mon[44189]: osdmap e114: 12 total, 12 up, 12 in 2024-04-20T17:08:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:04 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3213814244' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-04-20T17:08:05.558 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:08:05.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:05.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:08:05.662 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:08:05.691 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:05.891 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:08:05.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:05.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:08:05.995 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:08:06.002 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:08:06.031 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:06.188 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:06 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3213814244' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-04-20T17:08:06.189 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:06 smithi005 ceph-mon[39562]: osdmap e115: 12 total, 12 up, 12 in 2024-04-20T17:08:06.189 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:06 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/986529021' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:08:06.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:06 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3213814244' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-04-20T17:08:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:06 smithi106 ceph-mon[44423]: osdmap e115: 12 total, 12 up, 12 in 2024-04-20T17:08:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:06 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/986529021' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:08:06.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:06 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3213814244' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-04-20T17:08:06.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:06 smithi171 ceph-mon[44189]: osdmap e115: 12 total, 12 up, 12 in 2024-04-20T17:08:06.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:06 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/986529021' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:08:06.361 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:08:06.361 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-20T17:08:06.361 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:08:06.758 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:08:06.777 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:08:06.797 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:08:06.816 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:08:06.836 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:08:06.857 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:08:06.877 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:08:06.897 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:08:06.918 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:08:06.938 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:08:06.959 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:08:06.980 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:08:06.981 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:08:06.981 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:08:06.981 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:08:06.981 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:08:06.981 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:08:06.981 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:08:06.981 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:08:06.981 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:08:06.982 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:08:06.982 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:08:06.982 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:08:06.982 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:08:07.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:07 smithi005 ceph-mon[39562]: pgmap v365: 17 pgs: 3 creating+peering, 1 active+clean, 13 unknown; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:07.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:07 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:08:07.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:07 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/986529021' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:08:07.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:07 smithi005 ceph-mon[39562]: osdmap e116: 12 total, 12 up, 12 in 2024-04-20T17:08:07.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:07 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1779977510' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:08:07.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:07 smithi005 ceph-mon[39562]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-20T17:08:07.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:07 smithi106 ceph-mon[44423]: pgmap v365: 17 pgs: 3 creating+peering, 1 active+clean, 13 unknown; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:07 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:08:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:07 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/986529021' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:08:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:07 smithi106 ceph-mon[44423]: osdmap e116: 12 total, 12 up, 12 in 2024-04-20T17:08:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:07 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1779977510' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:08:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:07 smithi106 ceph-mon[44423]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-20T17:08:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:07 smithi171 ceph-mon[44189]: pgmap v365: 17 pgs: 3 creating+peering, 1 active+clean, 13 unknown; 577 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:07 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:08:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:07 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/986529021' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:08:07.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:07 smithi171 ceph-mon[44189]: osdmap e116: 12 total, 12 up, 12 in 2024-04-20T17:08:07.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:07 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1779977510' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:08:07.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:07 smithi171 ceph-mon[44189]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-20T17:08:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:08:07.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:07.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:08:07.486 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:08:07.517 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:07.538 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:08:07.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:07.541 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:08:07.642 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:08:07.674 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:07.924 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:08:07.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:07.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:08:07.930 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-04-20T17:08:08.028 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:08:08.056 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:08.094 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:08:08.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:08.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:08:08.198 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:08:08.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:08 smithi005 ceph-mon[39562]: osdmap e117: 12 total, 12 up, 12 in 2024-04-20T17:08:08.230 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:08.251 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:08:08.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:08.254 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:08:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:08 smithi171 ceph-mon[44189]: osdmap e117: 12 total, 12 up, 12 in 2024-04-20T17:08:08.355 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:08:08.382 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:08.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:08 smithi106 ceph-mon[44423]: osdmap e117: 12 total, 12 up, 12 in 2024-04-20T17:08:09.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:09 smithi005 ceph-mon[39562]: pgmap v368: 17 pgs: 3 creating+peering, 9 active+clean, 5 unknown; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:09.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:09 smithi171 ceph-mon[44189]: pgmap v368: 17 pgs: 3 creating+peering, 9 active+clean, 5 unknown; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:09.460 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:08:09.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:09.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:08:09.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:09 smithi106 ceph-mon[44423]: pgmap v368: 17 pgs: 3 creating+peering, 9 active+clean, 5 unknown; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:09.565 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:08:09.597 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:09.618 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:08:09.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:09.622 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:08:09.723 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:08:09.755 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:09.777 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:08:09.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:09.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:08:09.881 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:08:09.912 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:09.985 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:08:09.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:09.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:08:10.089 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:08:10.105 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:08:10.124 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:10.134 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:08:10.134 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:08:10.145 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:08:10.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:10.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:08:10.249 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:08:10.282 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:10.303 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:08:10.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:10.306 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:08:10.407 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:08:10.435 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:10.667 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:08:10.685 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:08:10.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:10.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:08:10.789 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:08:10.817 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:11.018 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:08:11.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:11.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:08:11.122 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:08:11.154 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:11.175 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:08:11.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:11.179 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:08:11.280 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:08:11.307 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:11 smithi171 ceph-mon[44189]: pgmap v369: 17 pgs: 3 creating+peering, 14 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:11.309 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:11.329 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:08:11.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:11.332 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:08:11.433 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:08:11.449 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:08:11.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:11 smithi005 ceph-mon[39562]: pgmap v369: 17 pgs: 3 creating+peering, 14 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:11.463 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:11 smithi106 ceph-mon[44423]: pgmap v369: 17 pgs: 3 creating+peering, 14 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:11.466 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:11.487 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:08:11.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:11.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:08:11.530 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-20T17:08:11.591 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:08:11.591 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:08:11.621 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:11.641 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:08:11.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:11.644 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:08:11.745 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:08:11.777 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:11.798 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:08:11.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:11.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:08:11.902 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:08:11.932 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:11.952 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:08:11.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:11.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:08:12.056 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:08:12.088 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:12.109 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:08:12.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:12.111 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:08:12.212 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:08:12.242 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:12.262 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:08:12.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:12.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:08:12.366 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:08:12.375 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-20T17:08:12.375 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:08:12.375 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-20T17:08:12.375 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-20T17:08:12.398 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:12.419 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:08:12.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:12.422 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:08:12.523 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:08:12.554 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:12.574 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:08:12.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:12.577 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:08:12.678 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:08:12.710 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:12.731 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:08:12.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:12.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:08:12.835 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:08:12.864 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:12.996 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:08:12.996 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:08:13.113 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:08:13.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:13.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:08:13.223 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:08:13.255 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:13.276 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:08:13.277 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:13 smithi171 ceph-mon[44189]: pgmap v370: 17 pgs: 17 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:13.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:13.279 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:08:13.352 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:13 smithi005 ceph-mon[39562]: pgmap v370: 17 pgs: 17 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:13.379 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:08:13.408 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:13.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:13 smithi106 ceph-mon[44423]: pgmap v370: 17 pgs: 17 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:13.557 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:08:13.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:13.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:08:13.660 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:08:13.692 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:13.713 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:08:13.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:13.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:08:13.817 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:08:13.846 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:13.866 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:08:13.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:13.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:08:13.970 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:08:13.998 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:14.017 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:08:14.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:14.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:08:14.034 INFO:teuthology.orchestra.run.smithi005.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-04-20T17:08:14.121 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:08:14.151 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:14.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:14 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3375645241' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:08:14.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:14 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3375645241' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:08:14.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:14 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3375645241' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:08:15.201 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:08:15.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:15.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:08:15.217 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:08:15.305 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:08:15.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:15 smithi171 ceph-mon[44189]: pgmap v371: 17 pgs: 17 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:15.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:15 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3375645241' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-20T17:08:15.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:15 smithi171 ceph-mon[44189]: osdmap e118: 12 total, 12 up, 12 in 2024-04-20T17:08:15.334 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:15 smithi106 ceph-mon[44423]: pgmap v371: 17 pgs: 17 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:15.334 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:15 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3375645241' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-20T17:08:15.334 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:15 smithi106 ceph-mon[44423]: osdmap e118: 12 total, 12 up, 12 in 2024-04-20T17:08:15.337 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:15.358 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:08:15.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:15.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:08:15.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:15 smithi005 ceph-mon[39562]: pgmap v371: 17 pgs: 17 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:15.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:15 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3375645241' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-20T17:08:15.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:15 smithi005 ceph-mon[39562]: osdmap e118: 12 total, 12 up, 12 in 2024-04-20T17:08:15.461 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:08:15.490 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:15.510 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:08:15.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:15.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:08:15.614 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:08:15.646 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:16.176 DEBUG:teuthology.orchestra.run.smithi171:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-04-20T17:08:16.178 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-20T17:08:16.178 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-20T17:08:16.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:16 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:08:16.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:16 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:08:16.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:16 smithi005 ceph-mon[39562]: osdmap e119: 12 total, 12 up, 12 in 2024-04-20T17:08:16.218 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:08:16.218 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:08:16.237 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:08:16.257 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:08:16.277 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:08:16.297 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:08:16.317 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:08:16.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:16 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:08:16.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:16 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:08:16.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:16 smithi171 ceph-mon[44189]: osdmap e119: 12 total, 12 up, 12 in 2024-04-20T17:08:16.339 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:08:16.359 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:08:16.379 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:08:16.399 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:08:16.419 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:08:16.420 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:08:16.420 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:08:16.420 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:08:16.420 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:08:16.420 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:08:16.420 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:08:16.420 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:08:16.421 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:08:16.421 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:08:16.421 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:08:16.421 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:08:16.421 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:08:16.437 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:hints = 1 2024-04-20T17:08:16.437 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-20T17:08:16.437 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Object prefix: benchmark_data_smithi171_78918 2024-04-20T17:08:16.438 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:08:16.438 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 0 0 0 0 0 0 - 0 2024-04-20T17:08:16.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:16 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:08:16.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:16 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:08:16.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:16 smithi106 ceph-mon[44423]: osdmap e119: 12 total, 12 up, 12 in 2024-04-20T17:08:16.598 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:08:16.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:16.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:08:16.707 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:08:16.749 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:16.780 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:08:16.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:16.783 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:08:16.884 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:08:16.926 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:16.954 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:08:16.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:16.957 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:08:17.058 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:08:17.099 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:17.129 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:08:17.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:17.134 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:08:17.235 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:08:17.279 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:17.303 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:08:17.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:17 smithi171 ceph-mon[44189]: pgmap v374: 17 pgs: 17 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:17.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:17 smithi171 ceph-mon[44189]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-20T17:08:17.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:17 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2688921544' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:08:17.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:17.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:17.307 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:17.307 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:17.307 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:17.307 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:17.307 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:17.307 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:17.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:17.309 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:08:17.411 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:08:17.437 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 1 16 5734 5718 357.346 357.375 0.00260678 0.00279279 2024-04-20T17:08:17.441 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:17 smithi106 ceph-mon[44423]: pgmap v374: 17 pgs: 17 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:17.441 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:17 smithi106 ceph-mon[44423]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-20T17:08:17.441 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:17 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2688921544' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:08:17.442 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:17.442 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:17.442 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:17.442 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:17.442 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:17.442 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:17.442 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:17.442 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:17.443 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:17.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:17 smithi005 ceph-mon[39562]: pgmap v374: 17 pgs: 17 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:08:17.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:17 smithi005 ceph-mon[39562]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-20T17:08:17.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:17 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2688921544' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:08:17.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:17.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:17.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:17.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:17.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:17.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:17.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:17.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:17.468 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:08:17.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:17.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:08:17.573 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:08:17.616 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:18.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:18.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:18 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:08:18.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:18 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:08:18.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:18 smithi005 ceph-mon[39562]: osdmap e120: 12 total, 12 up, 12 in 2024-04-20T17:08:18.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:18 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:08:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:18 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:08:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:18 smithi171 ceph-mon[44189]: osdmap e120: 12 total, 12 up, 12 in 2024-04-20T17:08:18.377 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:08:18.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:18.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:08:18.438 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 2 16 10952 10936 341.723 326.125 0.00226114 0.00292168 2024-04-20T17:08:18.486 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:08:18.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:18.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:18.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:18.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:18.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:18 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:08:18.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:18 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:08:18.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:18 smithi106 ceph-mon[44423]: osdmap e120: 12 total, 12 up, 12 in 2024-04-20T17:08:18.529 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:19.090 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:08:19.091 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:08:19.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:19.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:08:19.126 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:19 smithi005 ceph-mon[39562]: pgmap v376: 17 pgs: 17 active+clean; 144 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s wr, 382 op/s 2024-04-20T17:08:19.126 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:19 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3677849191' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:08:19.126 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:19 smithi005 ceph-mon[39562]: from='client.24739 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:08:19.179 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:08:19.180 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:08:19.180 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:08:19.180 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:08:19.180 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:08:19.196 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:08:19.241 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:19.275 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:08:19.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:19.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:08:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:19 smithi171 ceph-mon[44189]: pgmap v376: 17 pgs: 17 active+clean; 144 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s wr, 382 op/s 2024-04-20T17:08:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:19 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3677849191' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:08:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:19 smithi171 ceph-mon[44189]: from='client.24739 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:08:19.381 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:08:19.426 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:19.437 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 3 16 15788 15772 328.557 302.25 0.00336501 0.00304129 2024-04-20T17:08:19.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:19 smithi106 ceph-mon[44423]: pgmap v376: 17 pgs: 17 active+clean; 144 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s wr, 382 op/s 2024-04-20T17:08:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:19 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3677849191' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:08:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:19 smithi106 ceph-mon[44423]: from='client.24739 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:08:19.634 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:08:19.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:19.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:08:19.741 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:08:19.781 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:19.815 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:08:19.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:19.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:08:19.927 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:08:19.958 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:19.983 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:08:19.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:19.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:08:20.087 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:08:20.129 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:20.156 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:08:20.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:20.159 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:08:20.260 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:08:20.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:20 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:08:20.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:20 smithi106 ceph-mon[44423]: from='client.24739 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:08:20.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:20 smithi106 ceph-mon[44423]: osdmap e121: 12 total, 12 up, 12 in 2024-04-20T17:08:20.300 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:20.328 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:08:20.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:20.332 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:08:20.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:20 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:08:20.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:20 smithi171 ceph-mon[44189]: from='client.24739 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:08:20.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:20 smithi171 ceph-mon[44189]: osdmap e121: 12 total, 12 up, 12 in 2024-04-20T17:08:20.363 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:20 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:08:20.363 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:20 smithi005 ceph-mon[39562]: from='client.24739 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:08:20.363 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:20 smithi005 ceph-mon[39562]: osdmap e121: 12 total, 12 up, 12 in 2024-04-20T17:08:20.406 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-20T17:08:20.433 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:08:20.438 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 4 16 20238 20222 315.943 278.125 0.00632954 0.00316306 2024-04-20T17:08:20.464 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:20.489 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:08:20.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:20.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:08:20.593 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:08:20.630 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:20.660 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:08:20.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:20.665 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:08:20.766 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:08:20.798 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:20.820 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:08:20.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:20.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:08:20.899 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:08:20.924 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:08:20.956 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:20.980 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:08:20.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:20.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:08:21.083 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:21 smithi106 ceph-mon[44423]: pgmap v378: 17 pgs: 17 active+clean; 371 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-04-20T17:08:21.084 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:08:21.120 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:21.142 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:08:21.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:21.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:08:21.207 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:21 smithi005 ceph-mon[39562]: pgmap v378: 17 pgs: 17 active+clean; 371 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-04-20T17:08:21.247 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:08:21.287 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:21.315 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:08:21.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:21.320 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:08:21.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:21 smithi171 ceph-mon[44189]: pgmap v378: 17 pgs: 17 active+clean; 371 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-04-20T17:08:21.421 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:08:21.438 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 5 16 24325 24309 303.838 255.438 0.00573653 0.00328874 2024-04-20T17:08:21.462 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:21.574 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:08:21.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:21.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:08:21.687 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:08:21.722 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:21.747 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:08:21.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:21.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:08:21.851 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:08:21.883 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:21.908 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:08:21.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:21.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:08:22.012 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:08:22.045 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:22.070 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:08:22.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:22.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:08:22.153 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:08:22.174 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:08:22.206 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:22.231 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:08:22.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:22.236 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:08:22.337 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:08:22.382 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:22.438 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 6 16 28133 28117 292.862 238 0.00371817 0.00341263 2024-04-20T17:08:22.464 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:08:22.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:22.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:08:22.571 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:08:22.617 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:22.829 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:08:22.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:22.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:08:22.934 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:08:22.975 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:23.004 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:08:23.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:23.010 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:08:23.095 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:23 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:08:23.096 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:23 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:08:23.096 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:23 smithi005 ceph-mon[39562]: pgmap v379: 17 pgs: 17 active+clean; 896 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 130 MiB/s wr, 2.08k op/s 2024-04-20T17:08:23.096 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:23 smithi005 ceph-mon[39562]: osdmap e122: 12 total, 12 up, 12 in 2024-04-20T17:08:23.111 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:08:23.167 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:23.200 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:08:23.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:23.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:08:23.306 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:08:23.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:23 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:08:23.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:23 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:08:23.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:23 smithi171 ceph-mon[44189]: pgmap v379: 17 pgs: 17 active+clean; 896 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 130 MiB/s wr, 2.08k op/s 2024-04-20T17:08:23.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:23 smithi171 ceph-mon[44189]: osdmap e122: 12 total, 12 up, 12 in 2024-04-20T17:08:23.349 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:23.438 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 7 16 31303 31287 279.325 198.125 0.00202443 0.0035205 2024-04-20T17:08:23.491 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:08:23.491 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:08:23.491 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 122, 2024-04-20T17:08:23.491 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "13c6e76c-ff37-11ee-bc93-c7b262605968", 2024-04-20T17:08:23.491 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-04-20T16:58:25.272557+0000", 2024-04-20T17:08:23.491 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-04-20T17:08:22.082835+0000", 2024-04-20T17:08:23.492 INFO:teuthology.orchestra.run.smithi005.stdout: "last_up_change": "2024-04-20T17:06:01.788926+0000", 2024-04-20T17:08:23.492 INFO:teuthology.orchestra.run.smithi005.stdout: "last_in_change": "2024-04-20T17:05:42.849446+0000", 2024-04-20T17:08:23.492 INFO:teuthology.orchestra.run.smithi005.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-20T17:08:23.492 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_num": 5806080, 2024-04-20T17:08:23.492 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_set": [ 2024-04-20T17:08:23.492 INFO:teuthology.orchestra.run.smithi005.stdout: "nodeep-scrub", 2024-04-20T17:08:23.492 INFO:teuthology.orchestra.run.smithi005.stdout: "noscrub", 2024-04-20T17:08:23.492 INFO:teuthology.orchestra.run.smithi005.stdout: "pglog_hardlimit", 2024-04-20T17:08:23.492 INFO:teuthology.orchestra.run.smithi005.stdout: "purged_snapdirs", 2024-04-20T17:08:23.492 INFO:teuthology.orchestra.run.smithi005.stdout: "recovery_deletes", 2024-04-20T17:08:23.492 INFO:teuthology.orchestra.run.smithi005.stdout: "sortbitwise" 2024-04-20T17:08:23.492 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:08:23.492 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_version": 27, 2024-04-20T17:08:23.492 INFO:teuthology.orchestra.run.smithi005.stdout: "full_ratio": 0.94999998807907104, 2024-04-20T17:08:23.492 INFO:teuthology.orchestra.run.smithi005.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-20T17:08:23.493 INFO:teuthology.orchestra.run.smithi005.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-20T17:08:23.493 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_snapshot": "", 2024-04-20T17:08:23.493 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_max": 2, 2024-04-20T17:08:23.493 INFO:teuthology.orchestra.run.smithi005.stdout: "max_osd": 12, 2024-04-20T17:08:23.493 INFO:teuthology.orchestra.run.smithi005.stdout: "require_min_compat_client": "luminous", 2024-04-20T17:08:23.493 INFO:teuthology.orchestra.run.smithi005.stdout: "min_compat_client": "jewel", 2024-04-20T17:08:23.493 INFO:teuthology.orchestra.run.smithi005.stdout: "require_osd_release": "squid", 2024-04-20T17:08:23.493 INFO:teuthology.orchestra.run.smithi005.stdout: "allow_crimson": false, 2024-04-20T17:08:23.493 INFO:teuthology.orchestra.run.smithi005.stdout: "pools": [ 2024-04-20T17:08:23.493 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.493 INFO:teuthology.orchestra.run.smithi005.stdout: "pool": 1, 2024-04-20T17:08:23.493 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_name": ".mgr", 2024-04-20T17:08:23.493 INFO:teuthology.orchestra.run.smithi005.stdout: "create_time": "2024-04-20T17:01:35.909493+0000", 2024-04-20T17:08:23.493 INFO:teuthology.orchestra.run.smithi005.stdout: "flags": 1, 2024-04-20T17:08:23.494 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_names": "hashpspool", 2024-04-20T17:08:23.494 INFO:teuthology.orchestra.run.smithi005.stdout: "type": 1, 2024-04-20T17:08:23.494 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 2, 2024-04-20T17:08:23.494 INFO:teuthology.orchestra.run.smithi005.stdout: "min_size": 2, 2024-04-20T17:08:23.494 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_rule": 0, 2024-04-20T17:08:23.494 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_count": 0, 2024-04-20T17:08:23.494 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_target": 0, 2024-04-20T17:08:23.494 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T17:08:23.494 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T17:08:23.494 INFO:teuthology.orchestra.run.smithi005.stdout: "object_hash": 2, 2024-04-20T17:08:23.494 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_autoscale_mode": "off", 2024-04-20T17:08:23.494 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num": 1, 2024-04-20T17:08:23.494 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num": 1, 2024-04-20T17:08:23.494 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num_target": 1, 2024-04-20T17:08:23.495 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_target": 1, 2024-04-20T17:08:23.495 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_pending": 1, 2024-04-20T17:08:23.495 INFO:teuthology.orchestra.run.smithi005.stdout: "last_pg_merge_meta": { 2024-04-20T17:08:23.495 INFO:teuthology.orchestra.run.smithi005.stdout: "source_pgid": "0.0", 2024-04-20T17:08:23.495 INFO:teuthology.orchestra.run.smithi005.stdout: "ready_epoch": 0, 2024-04-20T17:08:23.495 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_started": 0, 2024-04-20T17:08:23.495 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_clean": 0, 2024-04-20T17:08:23.495 INFO:teuthology.orchestra.run.smithi005.stdout: "source_version": "0'0", 2024-04-20T17:08:23.495 INFO:teuthology.orchestra.run.smithi005.stdout: "target_version": "0'0" 2024-04-20T17:08:23.495 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.495 INFO:teuthology.orchestra.run.smithi005.stdout: "last_change": "16", 2024-04-20T17:08:23.495 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend": "0", 2024-04-20T17:08:23.495 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-20T17:08:23.495 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_preluminous": "0", 2024-04-20T17:08:23.496 INFO:teuthology.orchestra.run.smithi005.stdout: "auid": 0, 2024-04-20T17:08:23.496 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_mode": "selfmanaged", 2024-04-20T17:08:23.496 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_seq": 0, 2024-04-20T17:08:23.496 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_epoch": 0, 2024-04-20T17:08:23.496 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_snaps": [], 2024-04-20T17:08:23.496 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_snaps": "[]", 2024-04-20T17:08:23.496 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_bytes": 0, 2024-04-20T17:08:23.496 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_objects": 0, 2024-04-20T17:08:23.496 INFO:teuthology.orchestra.run.smithi005.stdout: "tiers": [], 2024-04-20T17:08:23.496 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_of": -1, 2024-04-20T17:08:23.496 INFO:teuthology.orchestra.run.smithi005.stdout: "read_tier": -1, 2024-04-20T17:08:23.497 INFO:teuthology.orchestra.run.smithi005.stdout: "write_tier": -1, 2024-04-20T17:08:23.497 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_mode": "none", 2024-04-20T17:08:23.497 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_bytes": 0, 2024-04-20T17:08:23.497 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_objects": 0, 2024-04-20T17:08:23.497 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T17:08:23.497 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T17:08:23.497 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T17:08:23.497 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_flush_age": 0, 2024-04-20T17:08:23.497 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_evict_age": 0, 2024-04-20T17:08:23.497 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_profile": "", 2024-04-20T17:08:23.497 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_params": { 2024-04-20T17:08:23.497 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "none" 2024-04-20T17:08:23.497 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.498 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_period": 0, 2024-04-20T17:08:23.498 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_count": 0, 2024-04-20T17:08:23.498 INFO:teuthology.orchestra.run.smithi005.stdout: "use_gmt_hitset": true, 2024-04-20T17:08:23.498 INFO:teuthology.orchestra.run.smithi005.stdout: "min_read_recency_for_promote": 0, 2024-04-20T17:08:23.498 INFO:teuthology.orchestra.run.smithi005.stdout: "min_write_recency_for_promote": 0, 2024-04-20T17:08:23.498 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T17:08:23.498 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_search_last_n": 0, 2024-04-20T17:08:23.498 INFO:teuthology.orchestra.run.smithi005.stdout: "grade_table": [], 2024-04-20T17:08:23.498 INFO:teuthology.orchestra.run.smithi005.stdout: "stripe_width": 0, 2024-04-20T17:08:23.498 INFO:teuthology.orchestra.run.smithi005.stdout: "expected_num_objects": 0, 2024-04-20T17:08:23.498 INFO:teuthology.orchestra.run.smithi005.stdout: "fast_read": false, 2024-04-20T17:08:23.498 INFO:teuthology.orchestra.run.smithi005.stdout: "options": { 2024-04-20T17:08:23.498 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_max": 32, 2024-04-20T17:08:23.498 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_min": 1 2024-04-20T17:08:23.499 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.499 INFO:teuthology.orchestra.run.smithi005.stdout: "application_metadata": { 2024-04-20T17:08:23.499 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": {} 2024-04-20T17:08:23.499 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.499 INFO:teuthology.orchestra.run.smithi005.stdout: "read_balance": { 2024-04-20T17:08:23.499 INFO:teuthology.orchestra.run.smithi005.stdout: "score_type": "Fair distribution", 2024-04-20T17:08:23.499 INFO:teuthology.orchestra.run.smithi005.stdout: "score_acting": 11.760000228881836, 2024-04-20T17:08:23.499 INFO:teuthology.orchestra.run.smithi005.stdout: "score_stable": 11.760000228881836, 2024-04-20T17:08:23.499 INFO:teuthology.orchestra.run.smithi005.stdout: "optimal_score": 0.17000000178813934, 2024-04-20T17:08:23.499 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_acting": 2, 2024-04-20T17:08:23.499 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_stable": 2, 2024-04-20T17:08:23.499 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-04-20T17:08:23.500 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity": 1, 2024-04-20T17:08:23.500 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity_weighted": 1 2024-04-20T17:08:23.500 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.500 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.500 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.500 INFO:teuthology.orchestra.run.smithi005.stdout: "pool": 2, 2024-04-20T17:08:23.500 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_name": "unique_pool_0", 2024-04-20T17:08:23.500 INFO:teuthology.orchestra.run.smithi005.stdout: "create_time": "2024-04-20T17:08:04.904546+0000", 2024-04-20T17:08:23.500 INFO:teuthology.orchestra.run.smithi005.stdout: "flags": 1, 2024-04-20T17:08:23.500 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_names": "hashpspool", 2024-04-20T17:08:23.500 INFO:teuthology.orchestra.run.smithi005.stdout: "type": 1, 2024-04-20T17:08:23.501 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 2, 2024-04-20T17:08:23.501 INFO:teuthology.orchestra.run.smithi005.stdout: "min_size": 2, 2024-04-20T17:08:23.501 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_rule": 0, 2024-04-20T17:08:23.501 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_count": 0, 2024-04-20T17:08:23.501 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_target": 0, 2024-04-20T17:08:23.501 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T17:08:23.501 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T17:08:23.501 INFO:teuthology.orchestra.run.smithi005.stdout: "object_hash": 2, 2024-04-20T17:08:23.501 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_autoscale_mode": "off", 2024-04-20T17:08:23.501 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num": 16, 2024-04-20T17:08:23.501 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num": 16, 2024-04-20T17:08:23.502 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num_target": 16, 2024-04-20T17:08:23.502 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_target": 16, 2024-04-20T17:08:23.502 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_pending": 16, 2024-04-20T17:08:23.502 INFO:teuthology.orchestra.run.smithi005.stdout: "last_pg_merge_meta": { 2024-04-20T17:08:23.502 INFO:teuthology.orchestra.run.smithi005.stdout: "source_pgid": "0.0", 2024-04-20T17:08:23.502 INFO:teuthology.orchestra.run.smithi005.stdout: "ready_epoch": 0, 2024-04-20T17:08:23.502 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_started": 0, 2024-04-20T17:08:23.502 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_clean": 0, 2024-04-20T17:08:23.502 INFO:teuthology.orchestra.run.smithi005.stdout: "source_version": "0'0", 2024-04-20T17:08:23.502 INFO:teuthology.orchestra.run.smithi005.stdout: "target_version": "0'0" 2024-04-20T17:08:23.502 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.502 INFO:teuthology.orchestra.run.smithi005.stdout: "last_change": "118", 2024-04-20T17:08:23.502 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend": "0", 2024-04-20T17:08:23.502 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-20T17:08:23.503 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_preluminous": "0", 2024-04-20T17:08:23.503 INFO:teuthology.orchestra.run.smithi005.stdout: "auid": 0, 2024-04-20T17:08:23.503 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_mode": "selfmanaged", 2024-04-20T17:08:23.503 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_seq": 0, 2024-04-20T17:08:23.503 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_epoch": 0, 2024-04-20T17:08:23.503 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_snaps": [], 2024-04-20T17:08:23.503 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_snaps": "[]", 2024-04-20T17:08:23.503 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_bytes": 0, 2024-04-20T17:08:23.503 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_objects": 0, 2024-04-20T17:08:23.503 INFO:teuthology.orchestra.run.smithi005.stdout: "tiers": [], 2024-04-20T17:08:23.503 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_of": -1, 2024-04-20T17:08:23.503 INFO:teuthology.orchestra.run.smithi005.stdout: "read_tier": -1, 2024-04-20T17:08:23.503 INFO:teuthology.orchestra.run.smithi005.stdout: "write_tier": -1, 2024-04-20T17:08:23.503 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_mode": "none", 2024-04-20T17:08:23.503 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_bytes": 0, 2024-04-20T17:08:23.504 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_objects": 0, 2024-04-20T17:08:23.504 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T17:08:23.504 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T17:08:23.504 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T17:08:23.504 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_flush_age": 0, 2024-04-20T17:08:23.504 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_evict_age": 0, 2024-04-20T17:08:23.504 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_profile": "", 2024-04-20T17:08:23.504 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_params": { 2024-04-20T17:08:23.504 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "none" 2024-04-20T17:08:23.504 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.504 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_period": 0, 2024-04-20T17:08:23.504 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_count": 0, 2024-04-20T17:08:23.504 INFO:teuthology.orchestra.run.smithi005.stdout: "use_gmt_hitset": true, 2024-04-20T17:08:23.504 INFO:teuthology.orchestra.run.smithi005.stdout: "min_read_recency_for_promote": 0, 2024-04-20T17:08:23.504 INFO:teuthology.orchestra.run.smithi005.stdout: "min_write_recency_for_promote": 0, 2024-04-20T17:08:23.505 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T17:08:23.505 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_search_last_n": 0, 2024-04-20T17:08:23.505 INFO:teuthology.orchestra.run.smithi005.stdout: "grade_table": [], 2024-04-20T17:08:23.505 INFO:teuthology.orchestra.run.smithi005.stdout: "stripe_width": 0, 2024-04-20T17:08:23.505 INFO:teuthology.orchestra.run.smithi005.stdout: "expected_num_objects": 0, 2024-04-20T17:08:23.505 INFO:teuthology.orchestra.run.smithi005.stdout: "fast_read": false, 2024-04-20T17:08:23.505 INFO:teuthology.orchestra.run.smithi005.stdout: "options": {}, 2024-04-20T17:08:23.505 INFO:teuthology.orchestra.run.smithi005.stdout: "application_metadata": { 2024-04-20T17:08:23.505 INFO:teuthology.orchestra.run.smithi005.stdout: "rados": {} 2024-04-20T17:08:23.505 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.505 INFO:teuthology.orchestra.run.smithi005.stdout: "read_balance": { 2024-04-20T17:08:23.505 INFO:teuthology.orchestra.run.smithi005.stdout: "score_type": "Fair distribution", 2024-04-20T17:08:23.505 INFO:teuthology.orchestra.run.smithi005.stdout: "score_acting": 2.25, 2024-04-20T17:08:23.505 INFO:teuthology.orchestra.run.smithi005.stdout: "score_stable": 2.25, 2024-04-20T17:08:23.505 INFO:teuthology.orchestra.run.smithi005.stdout: "optimal_score": 1, 2024-04-20T17:08:23.506 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_acting": 2.25, 2024-04-20T17:08:23.506 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_stable": 2.25, 2024-04-20T17:08:23.506 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-04-20T17:08:23.506 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity": 1, 2024-04-20T17:08:23.506 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity_weighted": 1 2024-04-20T17:08:23.506 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.506 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.506 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:08:23.506 INFO:teuthology.orchestra.run.smithi005.stdout: "osds": [ 2024-04-20T17:08:23.506 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.506 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 0, 2024-04-20T17:08:23.506 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b", 2024-04-20T17:08:23.506 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:08:23.506 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:08:23.506 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:08:23.507 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:08:23.507 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:08:23.507 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:08:23.507 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 8, 2024-04-20T17:08:23.507 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 58, 2024-04-20T17:08:23.507 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:08:23.507 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:08:23.507 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:08:23.507 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.507 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.507 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.507 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6801", 2024-04-20T17:08:23.507 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:08:23.507 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.507 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.508 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.508 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:08:23.508 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.508 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.508 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.508 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6802", 2024-04-20T17:08:23.508 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:08:23.508 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.508 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.508 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.508 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:08:23.508 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.508 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.508 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.508 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6804", 2024-04-20T17:08:23.509 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:08:23.509 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.509 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.509 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.509 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:08:23.509 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.509 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.509 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.509 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6803", 2024-04-20T17:08:23.509 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:08:23.509 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.509 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.509 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.509 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6801/3901484705", 2024-04-20T17:08:23.510 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6802/3901484705", 2024-04-20T17:08:23.510 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6804/3901484705", 2024-04-20T17:08:23.510 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6803/3901484705", 2024-04-20T17:08:23.510 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:08:23.510 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:08:23.510 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:08:23.510 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.510 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.510 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.510 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 1, 2024-04-20T17:08:23.510 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "aed0c8eb-0e81-4e85-ab8f-a60420060656", 2024-04-20T17:08:23.510 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:08:23.510 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:08:23.510 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:08:23.511 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:08:23.511 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:08:23.511 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:08:23.511 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 13, 2024-04-20T17:08:23.511 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 115, 2024-04-20T17:08:23.511 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:08:23.511 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:08:23.511 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:08:23.511 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.511 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.511 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.511 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6805", 2024-04-20T17:08:23.511 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3391893914 2024-04-20T17:08:23.511 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.511 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.512 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.512 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:08:23.512 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.512 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.512 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.512 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6806", 2024-04-20T17:08:23.512 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3391893914 2024-04-20T17:08:23.512 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.512 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.512 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.512 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:08:23.512 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.512 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.512 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.512 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6808", 2024-04-20T17:08:23.513 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3391893914 2024-04-20T17:08:23.513 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.513 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.513 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.513 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:08:23.513 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.513 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.513 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.513 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6807", 2024-04-20T17:08:23.513 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3391893914 2024-04-20T17:08:23.513 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.513 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.513 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.513 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6805/3391893914", 2024-04-20T17:08:23.513 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6806/3391893914", 2024-04-20T17:08:23.514 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6808/3391893914", 2024-04-20T17:08:23.514 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6807/3391893914", 2024-04-20T17:08:23.514 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:08:23.514 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:08:23.514 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:08:23.514 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.514 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.514 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.514 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 2, 2024-04-20T17:08:23.514 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "f733a6a2-19c2-4c7a-af9c-04e282062bb7", 2024-04-20T17:08:23.514 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:08:23.514 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:08:23.514 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:08:23.514 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:08:23.515 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:08:23.515 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:08:23.515 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 20, 2024-04-20T17:08:23.515 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 115, 2024-04-20T17:08:23.515 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:08:23.515 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:08:23.515 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:08:23.515 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.515 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.515 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.515 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6809", 2024-04-20T17:08:23.515 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2105468358 2024-04-20T17:08:23.515 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.515 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.515 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.516 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:08:23.516 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.516 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.516 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.516 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6810", 2024-04-20T17:08:23.516 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2105468358 2024-04-20T17:08:23.516 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.516 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.516 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.516 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:08:23.516 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.516 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.516 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.516 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6812", 2024-04-20T17:08:23.516 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2105468358 2024-04-20T17:08:23.517 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.517 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.517 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.517 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:08:23.517 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.517 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.517 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.517 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6811", 2024-04-20T17:08:23.517 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2105468358 2024-04-20T17:08:23.517 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.517 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.517 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.517 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6809/2105468358", 2024-04-20T17:08:23.517 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6810/2105468358", 2024-04-20T17:08:23.518 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6812/2105468358", 2024-04-20T17:08:23.518 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6811/2105468358", 2024-04-20T17:08:23.518 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:08:23.518 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:08:23.518 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:08:23.518 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.518 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.518 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.518 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 3, 2024-04-20T17:08:23.518 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "8c92f1df-e298-4606-88ae-b8a8e79628a1", 2024-04-20T17:08:23.518 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:08:23.518 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:08:23.518 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:08:23.518 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:08:23.519 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:08:23.519 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:08:23.519 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 25, 2024-04-20T17:08:23.519 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 0, 2024-04-20T17:08:23.519 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:08:23.519 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:08:23.519 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:08:23.519 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.519 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.519 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.519 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6813", 2024-04-20T17:08:23.519 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 377652039 2024-04-20T17:08:23.519 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.519 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.519 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.520 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:08:23.520 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.520 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.520 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.520 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6814", 2024-04-20T17:08:23.520 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 377652039 2024-04-20T17:08:23.520 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.520 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.520 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.520 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:08:23.520 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.520 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.520 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.520 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6816", 2024-04-20T17:08:23.520 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 377652039 2024-04-20T17:08:23.521 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.521 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.521 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.521 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:08:23.521 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.521 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.521 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.521 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6815", 2024-04-20T17:08:23.521 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 377652039 2024-04-20T17:08:23.521 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.521 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.521 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.522 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6813/377652039", 2024-04-20T17:08:23.522 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6814/377652039", 2024-04-20T17:08:23.522 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6816/377652039", 2024-04-20T17:08:23.522 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6815/377652039", 2024-04-20T17:08:23.522 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:08:23.522 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:08:23.522 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:08:23.522 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.522 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.522 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.522 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 4, 2024-04-20T17:08:23.522 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "46df16f1-9607-4d7b-9be8-9d4236e160d1", 2024-04-20T17:08:23.522 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:08:23.522 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:08:23.523 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:08:23.523 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:08:23.523 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:08:23.523 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:08:23.523 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 31, 2024-04-20T17:08:23.523 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 0, 2024-04-20T17:08:23.523 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:08:23.523 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:08:23.523 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:08:23.523 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.523 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.523 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.523 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6800", 2024-04-20T17:08:23.523 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:08:23.523 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.524 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.524 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.524 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:08:23.524 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.524 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.524 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.524 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6801", 2024-04-20T17:08:23.524 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:08:23.524 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.524 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.524 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.524 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:08:23.524 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.524 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.524 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.525 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6803", 2024-04-20T17:08:23.525 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:08:23.525 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.525 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.525 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.525 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:08:23.525 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.525 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.525 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.525 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6802", 2024-04-20T17:08:23.525 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:08:23.525 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.525 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.525 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.525 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6800/1839524469", 2024-04-20T17:08:23.526 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6801/1839524469", 2024-04-20T17:08:23.526 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6803/1839524469", 2024-04-20T17:08:23.526 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6802/1839524469", 2024-04-20T17:08:23.526 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:08:23.526 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:08:23.526 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:08:23.526 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.526 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.526 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.526 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 5, 2024-04-20T17:08:23.526 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "6603231d-1e5c-4575-bfaa-2eeca2c12215", 2024-04-20T17:08:23.526 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:08:23.526 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:08:23.526 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:08:23.527 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:08:23.527 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:08:23.527 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:08:23.527 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 37, 2024-04-20T17:08:23.527 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 115, 2024-04-20T17:08:23.527 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:08:23.527 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:08:23.527 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:08:23.527 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.527 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.527 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.527 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6804", 2024-04-20T17:08:23.527 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:08:23.527 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.527 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.528 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.528 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:08:23.528 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.528 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.528 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.528 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6805", 2024-04-20T17:08:23.528 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:08:23.528 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.528 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.528 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.528 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:08:23.528 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.528 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.528 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.529 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6807", 2024-04-20T17:08:23.529 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:08:23.529 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.529 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.529 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.529 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:08:23.529 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.529 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.529 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.529 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6806", 2024-04-20T17:08:23.529 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:08:23.529 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.529 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.529 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.529 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6804/2450315908", 2024-04-20T17:08:23.530 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6805/2450315908", 2024-04-20T17:08:23.530 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6807/2450315908", 2024-04-20T17:08:23.530 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6806/2450315908", 2024-04-20T17:08:23.530 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:08:23.530 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:08:23.530 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:08:23.530 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.530 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.530 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.530 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 6, 2024-04-20T17:08:23.530 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "783a55a6-f176-42be-8c8b-af8481d08dc9", 2024-04-20T17:08:23.530 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:08:23.530 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:08:23.530 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:08:23.531 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:08:23.531 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:08:23.531 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:08:23.531 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 48, 2024-04-20T17:08:23.531 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 115, 2024-04-20T17:08:23.531 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:08:23.531 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:08:23.531 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:08:23.531 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.531 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.531 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.531 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6808", 2024-04-20T17:08:23.531 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2840237913 2024-04-20T17:08:23.531 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.531 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.532 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.532 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:08:23.532 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.532 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.532 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.532 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6809", 2024-04-20T17:08:23.532 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2840237913 2024-04-20T17:08:23.532 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.532 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.532 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.532 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:08:23.532 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.532 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.532 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.532 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6811", 2024-04-20T17:08:23.533 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2840237913 2024-04-20T17:08:23.533 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.533 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.533 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.533 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:08:23.533 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.533 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.533 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.533 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6810", 2024-04-20T17:08:23.533 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2840237913 2024-04-20T17:08:23.533 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.533 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.533 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.533 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6808/2840237913", 2024-04-20T17:08:23.534 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6809/2840237913", 2024-04-20T17:08:23.534 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6811/2840237913", 2024-04-20T17:08:23.534 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6810/2840237913", 2024-04-20T17:08:23.534 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:08:23.534 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:08:23.534 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:08:23.534 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.534 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.534 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.534 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 7, 2024-04-20T17:08:23.534 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4", 2024-04-20T17:08:23.534 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:08:23.534 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:08:23.534 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:08:23.535 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:08:23.535 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:08:23.535 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:08:23.535 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 56, 2024-04-20T17:08:23.535 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 115, 2024-04-20T17:08:23.535 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:08:23.535 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:08:23.535 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:08:23.535 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.535 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.535 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.535 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6812", 2024-04-20T17:08:23.536 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1105371883 2024-04-20T17:08:23.536 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.536 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.536 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.536 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:08:23.536 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.536 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.536 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.536 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6813", 2024-04-20T17:08:23.536 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1105371883 2024-04-20T17:08:23.536 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.536 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.536 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.536 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:08:23.536 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.537 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.537 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.537 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6815", 2024-04-20T17:08:23.537 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1105371883 2024-04-20T17:08:23.537 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.537 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.537 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.537 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:08:23.537 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.537 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.537 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.537 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6814", 2024-04-20T17:08:23.537 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1105371883 2024-04-20T17:08:23.537 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.537 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.538 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.538 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6812/1105371883", 2024-04-20T17:08:23.538 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6813/1105371883", 2024-04-20T17:08:23.538 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6815/1105371883", 2024-04-20T17:08:23.538 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6814/1105371883", 2024-04-20T17:08:23.538 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:08:23.538 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:08:23.538 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:08:23.538 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.538 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.538 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.538 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 8, 2024-04-20T17:08:23.538 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "1cb232ad-1f59-4a0a-a215-e67419ad6476", 2024-04-20T17:08:23.538 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:08:23.539 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:08:23.539 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:08:23.539 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:08:23.539 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:08:23.539 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:08:23.539 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 67, 2024-04-20T17:08:23.539 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 115, 2024-04-20T17:08:23.539 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:08:23.539 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:08:23.539 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:08:23.539 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.539 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.539 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.539 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6800", 2024-04-20T17:08:23.540 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:08:23.540 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.540 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.540 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.540 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:08:23.540 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.540 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.540 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.540 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6801", 2024-04-20T17:08:23.540 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:08:23.540 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.540 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.540 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.540 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:08:23.540 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.541 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.541 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.541 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6803", 2024-04-20T17:08:23.541 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:08:23.541 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.541 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.541 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.541 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:08:23.541 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.541 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.541 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.541 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6802", 2024-04-20T17:08:23.541 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:08:23.541 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.541 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.542 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.542 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6800/4123058942", 2024-04-20T17:08:23.542 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6801/4123058942", 2024-04-20T17:08:23.542 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6803/4123058942", 2024-04-20T17:08:23.542 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6802/4123058942", 2024-04-20T17:08:23.542 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:08:23.542 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:08:23.542 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:08:23.542 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.542 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.542 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.542 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 9, 2024-04-20T17:08:23.542 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "b6c17147-b3dc-475b-bb20-f9f712fb99e8", 2024-04-20T17:08:23.542 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:08:23.542 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:08:23.543 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:08:23.543 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:08:23.543 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:08:23.543 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:08:23.543 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 73, 2024-04-20T17:08:23.543 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 115, 2024-04-20T17:08:23.543 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:08:23.546 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:08:23.546 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:08:23.546 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.546 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.546 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.546 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6804", 2024-04-20T17:08:23.546 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:08:23.546 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.546 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.546 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:08:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6805", 2024-04-20T17:08:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:08:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:08:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6807", 2024-04-20T17:08:23.547 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:08:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:08:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6806", 2024-04-20T17:08:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:08:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6804/2877589636", 2024-04-20T17:08:23.548 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6805/2877589636", 2024-04-20T17:08:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6807/2877589636", 2024-04-20T17:08:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6806/2877589636", 2024-04-20T17:08:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:08:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:08:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:08:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 10, 2024-04-20T17:08:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "1adcdf55-f002-4a39-91e7-378e5fae849c", 2024-04-20T17:08:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:08:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:08:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:08:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:08:23.549 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:08:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:08:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 84, 2024-04-20T17:08:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 0, 2024-04-20T17:08:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:08:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:08:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:08:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6808", 2024-04-20T17:08:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:08:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.550 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.551 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:08:23.551 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.551 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.551 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.551 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6809", 2024-04-20T17:08:23.551 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:08:23.551 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.551 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.551 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.551 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:08:23.551 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.551 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.551 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.551 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6811", 2024-04-20T17:08:23.551 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:08:23.552 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.552 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.552 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.552 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:08:23.552 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.552 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.552 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.552 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6810", 2024-04-20T17:08:23.552 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:08:23.552 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.552 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.552 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.552 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6808/181490621", 2024-04-20T17:08:23.552 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6809/181490621", 2024-04-20T17:08:23.552 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6811/181490621", 2024-04-20T17:08:23.553 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6810/181490621", 2024-04-20T17:08:23.553 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:08:23.553 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:08:23.553 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:08:23.553 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.553 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.553 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.553 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 11, 2024-04-20T17:08:23.553 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "737c5aec-694a-4968-819d-fccf22661f4f", 2024-04-20T17:08:23.553 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:08:23.553 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:08:23.553 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:08:23.553 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:08:23.553 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:08:23.554 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:08:23.554 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 92, 2024-04-20T17:08:23.554 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 115, 2024-04-20T17:08:23.554 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:08:23.554 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:08:23.554 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:08:23.554 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.554 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.554 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.554 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6812", 2024-04-20T17:08:23.554 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:08:23.554 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.554 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.554 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.554 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:08:23.555 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.555 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.555 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.555 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6813", 2024-04-20T17:08:23.555 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:08:23.555 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.555 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.555 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.555 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:08:23.555 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.555 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.555 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.555 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6815", 2024-04-20T17:08:23.555 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:08:23.556 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.556 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.556 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.556 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:08:23.556 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:08:23.556 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.556 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:08:23.556 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6814", 2024-04-20T17:08:23.556 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:08:23.556 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.556 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.556 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.556 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6812/3373585701", 2024-04-20T17:08:23.556 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6813/3373585701", 2024-04-20T17:08:23.557 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6815/3373585701", 2024-04-20T17:08:23.557 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6814/3373585701", 2024-04-20T17:08:23.557 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:08:23.557 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:08:23.557 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:08:23.557 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:23.557 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.557 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:08:23.557 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_xinfo": [ 2024-04-20T17:08:23.557 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.557 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 0, 2024-04-20T17:08:23.557 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:08:23.557 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:08:23.557 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:08:23.558 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:08:23.558 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:08:23.558 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:07:23.532853+0000", 2024-04-20T17:08:23.558 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:08:23.558 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.558 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.558 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 1, 2024-04-20T17:08:23.558 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:08:23.558 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:08:23.558 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:08:23.558 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:08:23.558 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:08:23.558 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:08:03.255790+0000", 2024-04-20T17:08:23.558 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:08:23.559 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.559 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.559 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 2, 2024-04-20T17:08:23.559 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:08:23.559 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:08:23.559 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:08:23.559 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:08:23.559 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:08:23.559 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:08:14.773262+0000", 2024-04-20T17:08:23.559 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:08:23.559 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.559 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.559 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 3, 2024-04-20T17:08:23.559 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:08:23.559 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:08:23.560 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:08:23.560 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:08:23.560 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:08:23.560 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:07:20.640569+0000", 2024-04-20T17:08:23.560 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:08:23.560 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.560 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.560 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 4, 2024-04-20T17:08:23.560 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:08:23.560 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:08:23.560 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:08:23.560 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:08:23.560 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:08:23.560 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:08:00.396848+0000", 2024-04-20T17:08:23.560 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:08:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 5, 2024-04-20T17:08:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:08:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:08:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:08:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:08:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:08:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:08:03.693589+0000", 2024-04-20T17:08:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:08:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 6, 2024-04-20T17:08:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:08:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:08:23.561 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:08:23.562 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:08:23.562 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:08:23.562 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:08:21.893763+0000", 2024-04-20T17:08:23.562 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:08:23.562 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.562 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.562 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 7, 2024-04-20T17:08:23.562 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:08:23.562 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:08:23.562 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:08:23.562 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:08:23.562 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:08:23.562 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:07:38.130130+0000", 2024-04-20T17:08:23.562 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:08:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 8, 2024-04-20T17:08:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:08:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:08:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:08:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:08:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:08:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:07:07.526939+0000", 2024-04-20T17:08:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:08:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 9, 2024-04-20T17:08:23.563 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:08:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:08:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:08:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:08:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:08:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:07:33.613661+0000", 2024-04-20T17:08:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:08:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 10, 2024-04-20T17:08:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:08:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:08:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:08:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:08:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:08:23.564 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:08:16.452578+0000", 2024-04-20T17:08:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:08:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 11, 2024-04-20T17:08:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:08:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:08:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:08:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:08:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:08:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:07:19.942313+0000", 2024-04-20T17:08:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:08:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:08:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_upmap": [], 2024-04-20T17:08:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_upmap_items": [], 2024-04-20T17:08:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_upmap_primaries": [], 2024-04-20T17:08:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_temp": [], 2024-04-20T17:08:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_temp": [], 2024-04-20T17:08:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "blocklist": { 2024-04-20T17:08:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/3881726425": "2024-04-21T16:59:25.808337+0000", 2024-04-20T17:08:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/338649532": "2024-04-21T16:59:25.808337+0000", 2024-04-20T17:08:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/1227843664": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:08:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:6800/1876064631": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:08:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/160689449": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:08:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:6800/3521715286": "2024-04-21T16:59:25.808337+0000", 2024-04-20T17:08:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/3841663462": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:08:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/687690919": "2024-04-21T16:59:25.808337+0000" 2024-04-20T17:08:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "range_blocklist": {}, 2024-04-20T17:08:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_profiles": { 2024-04-20T17:08:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: "default": { 2024-04-20T17:08:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: "crush-failure-domain": "osd", 2024-04-20T17:08:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: "k": "2", 2024-04-20T17:08:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: "m": "1", 2024-04-20T17:08:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: "plugin": "jerasure", 2024-04-20T17:08:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: "technique": "reed_sol_van" 2024-04-20T17:08:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_snaps_queue": [], 2024-04-20T17:08:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: "new_removed_snaps": [], 2024-04-20T17:08:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: "new_purged_snaps": [], 2024-04-20T17:08:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_node_flags": {}, 2024-04-20T17:08:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: "device_class_flags": {}, 2024-04-20T17:08:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": { 2024-04-20T17:08:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode_enabled": false, 2024-04-20T17:08:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_bucket_count": 0, 2024-04-20T17:08:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: "degraded_stretch_mode": 0, 2024-04-20T17:08:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: "recovering_stretch_mode": 0, 2024-04-20T17:08:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode_bucket": 0 2024-04-20T17:08:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:23.568 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:08:23.569 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:23 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:08:23.569 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:23 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:08:23.569 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:23 smithi106 ceph-mon[44423]: pgmap v379: 17 pgs: 17 active+clean; 896 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 130 MiB/s wr, 2.08k op/s 2024-04-20T17:08:23.569 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:23 smithi106 ceph-mon[44423]: osdmap e122: 12 total, 12 up, 12 in 2024-04-20T17:08:23.569 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:08:23.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:23.570 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:08:23.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:08:23.624 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:08:23.671 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:08:23.703 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:23.726 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:08:23.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:23.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:08:23.830 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:08:23.874 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:23.902 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:08:23.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:23.906 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:08:24.007 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:08:24.037 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:24.060 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:08:24.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:24.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:08:24.164 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:08:24.212 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:24.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:24 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/314741225' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-20T17:08:24.428 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:08:24.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:24.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:08:24.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:24 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/314741225' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-20T17:08:24.438 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 8 16 34541 34525 269.704 202.375 0.00480717 0.00370587 2024-04-20T17:08:24.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:24 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/314741225' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-20T17:08:24.533 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:08:24.565 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:24.588 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:08:24.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:24.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:08:24.693 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:08:24.723 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:24.748 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:08:24.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:24.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:08:24.853 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:08:24.886 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:24.909 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:08:24.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:24.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:08:25.014 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:08:25.064 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:25.150 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:08:25.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:25.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:08:25.256 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:08:25.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:25 smithi106 ceph-mon[44423]: pgmap v381: 17 pgs: 17 active+clean; 1.7 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 248 MiB/s wr, 3.97k op/s 2024-04-20T17:08:25.315 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:25 smithi005 ceph-mon[39562]: pgmap v381: 17 pgs: 17 active+clean; 1.7 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 248 MiB/s wr, 3.97k op/s 2024-04-20T17:08:25.315 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:25.322 INFO:tasks.thrashosds.thrasher:j is {'epoch': 122, 'fsid': '13c6e76c-ff37-11ee-bc93-c7b262605968', 'created': '2024-04-20T16:58:25.272557+0000', 'modified': '2024-04-20T17:08:22.082835+0000', 'last_up_change': '2024-04-20T17:06:01.788926+0000', 'last_in_change': '2024-04-20T17:05:42.849446+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-20T17:01:35.909493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.760000228881836, 'score_stable': 11.760000228881836, 'optimal_score': 0.17000000178813934, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-04-20T17:08:04.904546+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 58, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6801', 'nonce': 3901484705}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6802', 'nonce': 3901484705}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6804', 'nonce': 3901484705}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6803', 'nonce': 3901484705}]}, 'public_addr': '172.21.15.5:6801/3901484705', 'cluster_addr': '172.21.15.5:6802/3901484705', 'heartbeat_back_addr': '172.21.15.5:6804/3901484705', 'heartbeat_front_addr': '172.21.15.5:6803/3901484705', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'aed0c8eb-0e81-4e85-ab8f-a60420060656', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6805', 'nonce': 3391893914}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6806', 'nonce': 3391893914}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6808', 'nonce': 3391893914}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6807', 'nonce': 3391893914}]}, 'public_addr': '172.21.15.5:6805/3391893914', 'cluster_addr': '172.21.15.5:6806/3391893914', 'heartbeat_back_addr': '172.21.15.5:6808/3391893914', 'heartbeat_front_addr': '172.21.15.5:6807/3391893914', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f733a6a2-19c2-4c7a-af9c-04e282062bb7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 20, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6809', 'nonce': 2105468358}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6810', 'nonce': 2105468358}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6812', 'nonce': 2105468358}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6811', 'nonce': 2105468358}]}, 'public_addr': '172.21.15.5:6809/2105468358', 'cluster_addr': '172.21.15.5:6810/2105468358', 'heartbeat_back_addr': '172.21.15.5:6812/2105468358', 'heartbeat_front_addr': '172.21.15.5:6811/2105468358', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '8c92f1df-e298-4606-88ae-b8a8e79628a1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6813', 'nonce': 377652039}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6814', 'nonce': 377652039}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6816', 'nonce': 377652039}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6815', 'nonce': 377652039}]}, 'public_addr': '172.21.15.5:6813/377652039', 'cluster_addr': '172.21.15.5:6814/377652039', 'heartbeat_back_addr': '172.21.15.5:6816/377652039', 'heartbeat_front_addr': '172.21.15.5:6815/377652039', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '46df16f1-9607-4d7b-9be8-9d4236e160d1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6800', 'nonce': 1839524469}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6801', 'nonce': 1839524469}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6803', 'nonce': 1839524469}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6802', 'nonce': 1839524469}]}, 'public_addr': '172.21.15.106:6800/1839524469', 'cluster_addr': '172.21.15.106:6801/1839524469', 'heartbeat_back_addr': '172.21.15.106:6803/1839524469', 'heartbeat_front_addr': '172.21.15.106:6802/1839524469', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6603231d-1e5c-4575-bfaa-2eeca2c12215', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6804', 'nonce': 2450315908}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6805', 'nonce': 2450315908}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6807', 'nonce': 2450315908}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6806', 'nonce': 2450315908}]}, 'public_addr': '172.21.15.106:6804/2450315908', 'cluster_addr': '172.21.15.106:6805/2450315908', 'heartbeat_back_addr': '172.21.15.106:6807/2450315908', 'heartbeat_front_addr': '172.21.15.106:6806/2450315908', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '783a55a6-f176-42be-8c8b-af8481d08dc9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6808', 'nonce': 2840237913}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6809', 'nonce': 2840237913}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6811', 'nonce': 2840237913}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6810', 'nonce': 2840237913}]}, 'public_addr': '172.21.15.106:6808/2840237913', 'cluster_addr': '172.21.15.106:6809/2840237913', 'heartbeat_back_addr': '172.21.15.106:6811/2840237913', 'heartbeat_front_addr': '172.21.15.106:6810/2840237913', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6812', 'nonce': 1105371883}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6813', 'nonce': 1105371883}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6815', 'nonce': 1105371883}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6814', 'nonce': 1105371883}]}, 'public_addr': '172.21.15.106:6812/1105371883', 'cluster_addr': '172.21.15.106:6813/1105371883', 'heartbeat_back_addr': '172.21.15.106:6815/1105371883', 'heartbeat_front_addr': '172.21.15.106:6814/1105371883', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '1cb232ad-1f59-4a0a-a215-e67419ad6476', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 67, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6800', 'nonce': 4123058942}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6801', 'nonce': 4123058942}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6803', 'nonce': 4123058942}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6802', 'nonce': 4123058942}]}, 'public_addr': '172.21.15.171:6800/4123058942', 'cluster_addr': '172.21.15.171:6801/4123058942', 'heartbeat_back_addr': '172.21.15.171:6803/4123058942', 'heartbeat_front_addr': '172.21.15.171:6802/4123058942', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'b6c17147-b3dc-475b-bb20-f9f712fb99e8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 73, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6804', 'nonce': 2877589636}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6805', 'nonce': 2877589636}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6807', 'nonce': 2877589636}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6806', 'nonce': 2877589636}]}, 'public_addr': '172.21.15.171:6804/2877589636', 'cluster_addr': '172.21.15.171:6805/2877589636', 'heartbeat_back_addr': '172.21.15.171:6807/2877589636', 'heartbeat_front_addr': '172.21.15.171:6806/2877589636', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '1adcdf55-f002-4a39-91e7-378e5fae849c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 84, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6808', 'nonce': 181490621}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6809', 'nonce': 181490621}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6811', 'nonce': 181490621}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6810', 'nonce': 181490621}]}, 'public_addr': '172.21.15.171:6808/181490621', 'cluster_addr': '172.21.15.171:6809/181490621', 'heartbeat_back_addr': '172.21.15.171:6811/181490621', 'heartbeat_front_addr': '172.21.15.171:6810/181490621', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '737c5aec-694a-4968-819d-fccf22661f4f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 92, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6812', 'nonce': 3373585701}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6813', 'nonce': 3373585701}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6815', 'nonce': 3373585701}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6814', 'nonce': 3373585701}]}, 'public_addr': '172.21.15.171:6812/3373585701', 'cluster_addr': '172.21.15.171:6813/3373585701', 'heartbeat_back_addr': '172.21.15.171:6815/3373585701', 'heartbeat_front_addr': '172.21.15.171:6814/3373585701', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:07:23.532853+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:08:03.255790+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:08:14.773262+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:07:20.640569+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:08:00.396848+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:08:03.693589+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:08:21.893763+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:07:38.130130+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:07:07.526939+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:07:33.613661+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:08:16.452578+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:07:19.942313+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.5:0/3881726425': '2024-04-21T16:59:25.808337+0000', '172.21.15.5:0/338649532': '2024-04-21T16:59:25.808337+0000', '172.21.15.5:0/1227843664': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:6800/1876064631': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:0/160689449': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:6800/3521715286': '2024-04-21T16:59:25.808337+0000', '172.21.15.5:0/3841663462': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:0/687690919': '2024-04-21T16:59:25.808337+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-20T17:08:25.323 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:08:25.438 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 9 16 37905 37889 263.096 210.25 0.00618182 0.00379921 2024-04-20T17:08:25.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:25 smithi171 ceph-mon[44189]: pgmap v381: 17 pgs: 17 active+clean; 1.7 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 248 MiB/s wr, 3.97k op/s 2024-04-20T17:08:25.688 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:08:25.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:25.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:08:25.792 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:08:25.824 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:25.847 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:08:25.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:25.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:08:25.951 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:08:25.992 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:26.020 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:08:26.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:26.024 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:08:26.125 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:08:26.159 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:26.182 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:08:26.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:26.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:08:26.286 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:08:26.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:26 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:08:26.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:26 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:08:26.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:26 smithi171 ceph-mon[44189]: osdmap e123: 12 total, 12 up, 12 in 2024-04-20T17:08:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:26 smithi171 ceph-mon[44189]: osdmap e124: 12 total, 12 up, 12 in 2024-04-20T17:08:26.337 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:26 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:08:26.337 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:26 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:08:26.337 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:26 smithi005 ceph-mon[39562]: osdmap e123: 12 total, 12 up, 12 in 2024-04-20T17:08:26.337 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:26 smithi005 ceph-mon[39562]: osdmap e124: 12 total, 12 up, 12 in 2024-04-20T17:08:26.338 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:26.438 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 10 15 41026 41011 256.298 195.125 0.00409603 0.00390004 2024-04-20T17:08:26.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:26 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:08:26.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:26 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:08:26.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:26 smithi106 ceph-mon[44423]: osdmap e123: 12 total, 12 up, 12 in 2024-04-20T17:08:26.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:26 smithi106 ceph-mon[44423]: osdmap e124: 12 total, 12 up, 12 in 2024-04-20T17:08:27.438 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 11 15 44278 44263 251.473 203.25 0.00694269 0.00397503 2024-04-20T17:08:27.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:27 smithi005 ceph-mon[39562]: pgmap v384: 17 pgs: 17 active+clean; 1.9 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 259 MiB/s wr, 4.14k op/s 2024-04-20T17:08:27.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:27 smithi106 ceph-mon[44423]: pgmap v384: 17 pgs: 17 active+clean; 1.9 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 259 MiB/s wr, 4.14k op/s 2024-04-20T17:08:27.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:27 smithi171 ceph-mon[44189]: pgmap v384: 17 pgs: 17 active+clean; 1.9 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 259 MiB/s wr, 4.14k op/s 2024-04-20T17:08:28.439 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 12 16 47278 47262 246.136 187.438 0.00760596 0.00406105 2024-04-20T17:08:28.470 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:08:28.470 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:08:28.470 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:08:28.471 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:08:28.471 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.471 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:10722 2.4 e120/115 2:27621a40:::benchmark_data_smithi171_78918_object10721:head v 120'721, mlcod=120'719)", 2024-04-20T17:08:28.471 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:18.387201+0000", 2024-04-20T17:08:28.471 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 8.940661145, 2024-04-20T17:08:28.471 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010314834, 2024-04-20T17:08:28.471 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:28.471 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:28.471 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:28.471 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:28.471 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.471 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:28.471 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:18.387201+0000", 2024-04-20T17:08:28.471 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.472 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.472 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.472 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:28.472 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:18.387201+0000", 2024-04-20T17:08:28.472 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.472 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.472 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.472 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:28.472 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:18.387203+0000", 2024-04-20T17:08:28.472 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.415e-06 2024-04-20T17:08:28.472 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.472 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.472 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:28.472 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:18.387229+0000", 2024-04-20T17:08:28.472 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5836000000000001e-05 2024-04-20T17:08:28.473 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.473 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.473 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:28.473 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:18.387241+0000", 2024-04-20T17:08:28.473 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2319e-05 2024-04-20T17:08:28.473 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.473 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.473 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:28.473 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:18.387248+0000", 2024-04-20T17:08:28.473 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3799999999999996e-06 2024-04-20T17:08:28.473 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.473 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.473 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:28.473 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:18.387343+0000", 2024-04-20T17:08:28.473 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4882000000000004e-05 2024-04-20T17:08:28.474 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.474 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.474 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:28.474 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:18.387375+0000", 2024-04-20T17:08:28.474 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2264000000000001e-05 2024-04-20T17:08:28.474 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.474 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.474 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:28.474 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:18.397461+0000", 2024-04-20T17:08:28.474 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010085781 2024-04-20T17:08:28.474 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.474 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.474 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:28.474 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:18.397516+0000", 2024-04-20T17:08:28.475 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4957000000000003e-05 2024-04-20T17:08:28.475 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.475 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:28.475 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.475 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.475 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.475 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:27191 2.c e122/115 2:39f351cb:::benchmark_data_smithi171_78918_object27190:head v 122'1689, mlcod=122'1686)", 2024-04-20T17:08:28.475 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:22.180955+0000", 2024-04-20T17:08:28.475 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 5.1469071749999999, 2024-04-20T17:08:28.475 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015147328999999999, 2024-04-20T17:08:28.475 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:28.475 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:28.475 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:28.475 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:28.475 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.476 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:28.476 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.180955+0000", 2024-04-20T17:08:28.476 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.476 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.476 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.476 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:28.476 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.180955+0000", 2024-04-20T17:08:28.476 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.476 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.476 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.476 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:28.476 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.180957+0000", 2024-04-20T17:08:28.476 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0269999999999998e-06 2024-04-20T17:08:28.476 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.476 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.477 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:28.477 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.181013+0000", 2024-04-20T17:08:28.477 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5149999999999999e-05 2024-04-20T17:08:28.477 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.477 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.477 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:28.477 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.181047+0000", 2024-04-20T17:08:28.477 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4563000000000001e-05 2024-04-20T17:08:28.477 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.477 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.477 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:28.477 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.181065+0000", 2024-04-20T17:08:28.477 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7450000000000001e-05 2024-04-20T17:08:28.477 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.477 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.478 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:28.478 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.181463+0000", 2024-04-20T17:08:28.478 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00039860699999999999 2024-04-20T17:08:28.478 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.478 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.478 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:28.478 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.181484+0000", 2024-04-20T17:08:28.478 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0415e-05 2024-04-20T17:08:28.478 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.478 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.478 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:28.478 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.196047+0000", 2024-04-20T17:08:28.478 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014563296 2024-04-20T17:08:28.478 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.479 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.479 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:28.479 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.196103+0000", 2024-04-20T17:08:28.479 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5821e-05 2024-04-20T17:08:28.479 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.479 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:28.479 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.479 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.479 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.479 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:27192 2.5 e122/115 2:abdcd067:::benchmark_data_smithi171_78918_object27191:head v 122'1667, mlcod=122'1664)", 2024-04-20T17:08:28.479 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:22.181125+0000", 2024-04-20T17:08:28.479 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 5.1467375950000003, 2024-04-20T17:08:28.479 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014972488000000001, 2024-04-20T17:08:28.479 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:28.480 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:28.480 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:28.480 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:28.480 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.480 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:28.480 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.181125+0000", 2024-04-20T17:08:28.480 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.480 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.480 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.480 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:28.480 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.181125+0000", 2024-04-20T17:08:28.480 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.480 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.480 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.480 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:28.481 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.181126+0000", 2024-04-20T17:08:28.481 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.119e-06 2024-04-20T17:08:28.481 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.481 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.481 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:28.481 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.181155+0000", 2024-04-20T17:08:28.481 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9354000000000001e-05 2024-04-20T17:08:28.481 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.481 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.481 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:28.481 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.181163+0000", 2024-04-20T17:08:28.481 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5340000000000002e-06 2024-04-20T17:08:28.482 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.482 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.482 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:28.482 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.181169+0000", 2024-04-20T17:08:28.482 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6080000000000003e-06 2024-04-20T17:08:28.482 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.482 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.482 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:28.482 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.181241+0000", 2024-04-20T17:08:28.482 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2428000000000005e-05 2024-04-20T17:08:28.482 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.482 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.482 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:28.482 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.181263+0000", 2024-04-20T17:08:28.483 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1912e-05 2024-04-20T17:08:28.483 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.483 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.483 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:28.483 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.195999+0000", 2024-04-20T17:08:28.483 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014735867 2024-04-20T17:08:28.483 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.483 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.483 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:28.483 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.196097+0000", 2024-04-20T17:08:28.483 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8665999999999994e-05 2024-04-20T17:08:28.483 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.483 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:28.483 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.483 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.483 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.484 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:27204 2.5 e122/115 2:aa1b9275:::benchmark_data_smithi171_78918_object27203:head v 122'1668, mlcod=122'1665)", 2024-04-20T17:08:28.484 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:22.185012+0000", 2024-04-20T17:08:28.484 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 5.1428510020000004, 2024-04-20T17:08:28.484 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011120649999999999, 2024-04-20T17:08:28.484 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:28.484 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:28.484 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:28.484 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:28.484 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.484 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:28.484 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.185012+0000", 2024-04-20T17:08:28.484 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.484 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.484 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.484 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:28.485 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.185012+0000", 2024-04-20T17:08:28.485 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.485 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.485 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.485 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:28.485 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.185018+0000", 2024-04-20T17:08:28.485 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5019999999999998e-06 2024-04-20T17:08:28.485 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.485 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.485 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:28.485 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.185250+0000", 2024-04-20T17:08:28.485 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00023224199999999999 2024-04-20T17:08:28.485 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.485 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.485 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:28.486 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.185258+0000", 2024-04-20T17:08:28.486 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3100000000000003e-06 2024-04-20T17:08:28.486 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.486 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.486 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:28.486 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.185263+0000", 2024-04-20T17:08:28.486 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7939999999999999e-06 2024-04-20T17:08:28.486 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.486 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.487 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:28.487 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.185383+0000", 2024-04-20T17:08:28.487 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011971199999999999 2024-04-20T17:08:28.487 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.487 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.487 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:28.487 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.185410+0000", 2024-04-20T17:08:28.487 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7161e-05 2024-04-20T17:08:28.487 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.487 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.487 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:28.487 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.196104+0000", 2024-04-20T17:08:28.487 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010693914000000001 2024-04-20T17:08:28.487 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.488 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.488 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:28.488 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.196132+0000", 2024-04-20T17:08:28.488 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8014999999999999e-05 2024-04-20T17:08:28.488 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.488 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:28.488 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.488 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.488 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.488 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:27205 2.c e122/115 2:3d217872:::benchmark_data_smithi171_78918_object27204:head v 122'1690, mlcod=122'1687)", 2024-04-20T17:08:28.488 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:22.185051+0000", 2024-04-20T17:08:28.489 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 5.1428111830000001, 2024-04-20T17:08:28.489 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.01109533, 2024-04-20T17:08:28.489 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:28.489 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:28.489 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:28.489 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:28.489 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.489 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:28.489 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.185051+0000", 2024-04-20T17:08:28.489 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.490 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.490 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.490 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:28.490 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.185051+0000", 2024-04-20T17:08:28.490 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.490 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.490 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.490 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:28.490 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.185052+0000", 2024-04-20T17:08:28.490 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9999999999999996e-07 2024-04-20T17:08:28.490 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.490 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.490 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:28.490 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.185172+0000", 2024-04-20T17:08:28.491 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000119648 2024-04-20T17:08:28.491 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.491 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.491 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:28.491 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.185191+0000", 2024-04-20T17:08:28.491 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9335999999999999e-05 2024-04-20T17:08:28.491 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.491 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.491 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:28.491 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.185201+0000", 2024-04-20T17:08:28.491 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0051000000000001e-05 2024-04-20T17:08:28.491 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.491 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.492 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:28.492 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.188205+0000", 2024-04-20T17:08:28.492 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0030034829999999999 2024-04-20T17:08:28.492 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.492 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.492 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:28.492 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.188227+0000", 2024-04-20T17:08:28.492 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2453e-05 2024-04-20T17:08:28.492 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.492 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.492 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:28.492 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.196107+0000", 2024-04-20T17:08:28.492 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0078801140000000006 2024-04-20T17:08:28.492 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.492 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.492 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:28.493 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.196147+0000", 2024-04-20T17:08:28.493 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9444999999999999e-05 2024-04-20T17:08:28.493 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.493 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:28.493 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.493 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.493 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.493 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:27207 2.5 e122/115 2:a11b0014:::benchmark_data_smithi171_78918_object27206:head v 122'1669, mlcod=122'1665)", 2024-04-20T17:08:28.493 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:22.186385+0000", 2024-04-20T17:08:28.493 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 5.1414773480000004, 2024-04-20T17:08:28.493 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0097729600000000007, 2024-04-20T17:08:28.493 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:28.493 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:28.493 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:28.493 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:28.494 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.494 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:28.494 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.186385+0000", 2024-04-20T17:08:28.494 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.494 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.494 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.494 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:28.494 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.186385+0000", 2024-04-20T17:08:28.494 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.494 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.494 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.494 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:28.494 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.186387+0000", 2024-04-20T17:08:28.494 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6559999999999999e-06 2024-04-20T17:08:28.494 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.495 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.495 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:28.495 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.186423+0000", 2024-04-20T17:08:28.495 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5991000000000002e-05 2024-04-20T17:08:28.495 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.495 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.495 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:28.495 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.186433+0000", 2024-04-20T17:08:28.495 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0231000000000001e-05 2024-04-20T17:08:28.495 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.495 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.495 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:28.495 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.186440+0000", 2024-04-20T17:08:28.495 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4370000000000001e-06 2024-04-20T17:08:28.496 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.496 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.496 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:28.496 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.186517+0000", 2024-04-20T17:08:28.496 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6804999999999994e-05 2024-04-20T17:08:28.496 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.496 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.496 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:28.496 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.186538+0000", 2024-04-20T17:08:28.496 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0775999999999999e-05 2024-04-20T17:08:28.496 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.496 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.496 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:28.496 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.196135+0000", 2024-04-20T17:08:28.496 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0095964959999999995 2024-04-20T17:08:28.497 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.497 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.497 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:28.497 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.196158+0000", 2024-04-20T17:08:28.497 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3567999999999999e-05 2024-04-20T17:08:28.497 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.497 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:28.497 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.497 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.497 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.497 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:27218 2.4 e122/115 2:28b03d47:::benchmark_data_smithi171_78918_object27217:head v 122'1743, mlcod=122'1740)", 2024-04-20T17:08:28.497 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:22.197744+0000", 2024-04-20T17:08:28.497 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 5.1301188780000002, 2024-04-20T17:08:28.497 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010330255, 2024-04-20T17:08:28.497 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:28.498 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:28.498 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:28.498 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:28.498 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.498 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:28.498 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.197744+0000", 2024-04-20T17:08:28.498 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.498 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.498 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.498 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:28.498 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.197744+0000", 2024-04-20T17:08:28.498 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.498 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.499 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.499 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:28.499 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.197747+0000", 2024-04-20T17:08:28.499 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.693e-06 2024-04-20T17:08:28.499 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.499 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.499 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:28.499 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.197773+0000", 2024-04-20T17:08:28.499 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5556999999999999e-05 2024-04-20T17:08:28.499 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.499 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.499 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:28.499 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.197789+0000", 2024-04-20T17:08:28.499 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6504e-05 2024-04-20T17:08:28.499 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.500 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.500 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:28.500 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.197798+0000", 2024-04-20T17:08:28.500 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6580000000000008e-06 2024-04-20T17:08:28.500 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.500 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.500 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:28.500 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.198119+0000", 2024-04-20T17:08:28.500 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00032074099999999998 2024-04-20T17:08:28.500 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.500 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.500 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:28.500 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.198141+0000", 2024-04-20T17:08:28.500 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2133e-05 2024-04-20T17:08:28.500 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.500 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.501 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:28.501 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.208028+0000", 2024-04-20T17:08:28.501 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0098866969999999998 2024-04-20T17:08:28.501 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.501 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.501 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:28.501 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.208074+0000", 2024-04-20T17:08:28.501 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6272e-05 2024-04-20T17:08:28.501 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.501 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:28.501 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.501 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.501 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.502 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:27222 2.4 e122/115 2:24681f3f:::benchmark_data_smithi171_78918_object27221:head v 122'1744, mlcod=122'1740)", 2024-04-20T17:08:28.502 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:22.197820+0000", 2024-04-20T17:08:28.502 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 5.1300421959999998, 2024-04-20T17:08:28.502 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010290675000000001, 2024-04-20T17:08:28.502 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:28.502 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:28.502 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:28.502 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:28.502 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.502 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:28.502 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.197820+0000", 2024-04-20T17:08:28.502 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.502 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.503 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.503 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:28.503 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.197820+0000", 2024-04-20T17:08:28.503 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.503 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.503 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.503 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:28.503 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.197821+0000", 2024-04-20T17:08:28.503 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0800000000000004e-07 2024-04-20T17:08:28.503 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.503 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.503 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:28.503 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.197843+0000", 2024-04-20T17:08:28.503 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1963e-05 2024-04-20T17:08:28.503 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.503 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.503 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:28.504 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.197849+0000", 2024-04-20T17:08:28.504 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5579999999999997e-06 2024-04-20T17:08:28.504 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.504 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.504 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:28.504 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.197855+0000", 2024-04-20T17:08:28.504 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.079e-06 2024-04-20T17:08:28.504 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.504 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.504 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:28.504 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.199070+0000", 2024-04-20T17:08:28.504 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0012154500000000001 2024-04-20T17:08:28.504 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.504 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.504 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:28.505 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.199081+0000", 2024-04-20T17:08:28.505 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1477999999999999e-05 2024-04-20T17:08:28.505 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.505 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.505 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:28.505 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.208077+0000", 2024-04-20T17:08:28.505 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0089956900000000006 2024-04-20T17:08:28.505 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.505 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.505 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:28.505 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.208111+0000", 2024-04-20T17:08:28.505 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3849000000000001e-05 2024-04-20T17:08:28.505 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.505 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:28.505 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.506 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.506 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.506 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:27224 2.5 e122/115 2:a5c3509f:::benchmark_data_smithi171_78918_object27223:head v 122'1672, mlcod=122'1665)", 2024-04-20T17:08:28.506 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:22.197896+0000", 2024-04-20T17:08:28.506 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 5.1299668470000004, 2024-04-20T17:08:28.506 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0097368000000000003, 2024-04-20T17:08:28.506 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:28.506 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:28.506 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:28.506 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:28.506 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.506 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:28.506 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.197896+0000", 2024-04-20T17:08:28.507 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.507 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.507 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.507 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:28.507 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.197896+0000", 2024-04-20T17:08:28.507 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.507 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.507 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.507 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:28.507 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.197897+0000", 2024-04-20T17:08:28.507 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.3200000000000003e-07 2024-04-20T17:08:28.507 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.507 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.507 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:28.507 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.197934+0000", 2024-04-20T17:08:28.508 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6915999999999997e-05 2024-04-20T17:08:28.508 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.508 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.508 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:28.508 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.197941+0000", 2024-04-20T17:08:28.508 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6860000000000006e-06 2024-04-20T17:08:28.508 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.508 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.508 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:28.508 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.197946+0000", 2024-04-20T17:08:28.508 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1739999999999999e-06 2024-04-20T17:08:28.508 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.508 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.508 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:28.508 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.197998+0000", 2024-04-20T17:08:28.509 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1836000000000003e-05 2024-04-20T17:08:28.509 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.509 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.509 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:28.509 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.198023+0000", 2024-04-20T17:08:28.509 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4816000000000001e-05 2024-04-20T17:08:28.509 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.509 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.509 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:28.509 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.207589+0000", 2024-04-20T17:08:28.509 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0095660050000000007 2024-04-20T17:08:28.509 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.509 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.509 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:28.510 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.207632+0000", 2024-04-20T17:08:28.510 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3435e-05 2024-04-20T17:08:28.510 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.510 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:28.510 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.510 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.510 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.510 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:29600 2.c e122/115 2:39223520:::benchmark_data_smithi171_78918_object29599:head v 122'1825, mlcod=122'1823)", 2024-04-20T17:08:28.510 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:22.842070+0000", 2024-04-20T17:08:28.510 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 4.4857927289999999, 2024-04-20T17:08:28.510 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010221996000000001, 2024-04-20T17:08:28.510 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:28.510 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:28.510 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:28.511 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:28.511 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.511 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:28.511 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.842070+0000", 2024-04-20T17:08:28.511 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.511 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.511 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.511 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:28.511 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.842070+0000", 2024-04-20T17:08:28.511 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.511 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.511 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.511 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:28.511 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.842080+0000", 2024-04-20T17:08:28.511 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0499000000000001e-05 2024-04-20T17:08:28.512 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.512 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.512 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:28.512 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.842126+0000", 2024-04-20T17:08:28.512 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6029999999999998e-05 2024-04-20T17:08:28.512 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.512 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.512 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:28.512 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.842141+0000", 2024-04-20T17:08:28.512 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4985e-05 2024-04-20T17:08:28.512 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.513 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.513 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:28.513 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.842150+0000", 2024-04-20T17:08:28.513 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.276e-06 2024-04-20T17:08:28.513 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.513 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.513 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:28.513 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.842242+0000", 2024-04-20T17:08:28.513 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2561999999999999e-05 2024-04-20T17:08:28.513 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.513 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.513 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:28.513 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.842262+0000", 2024-04-20T17:08:28.513 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0125000000000001e-05 2024-04-20T17:08:28.513 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.513 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.514 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:28.514 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.852248+0000", 2024-04-20T17:08:28.514 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.009985608 2024-04-20T17:08:28.514 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.514 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.514 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:28.514 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:22.852292+0000", 2024-04-20T17:08:28.514 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3911e-05 2024-04-20T17:08:28.514 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.514 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:28.514 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.514 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.514 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.514 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:34846 2.c e122/115 2:3006e5c8:::benchmark_data_smithi171_78918_object34845:head v 122'2140, mlcod=122'2138)", 2024-04-20T17:08:28.515 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:24.524593+0000", 2024-04-20T17:08:28.515 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 2.8032697789999999, 2024-04-20T17:08:28.515 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010562831999999999, 2024-04-20T17:08:28.515 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:28.515 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:28.515 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:28.515 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:28.515 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.515 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:28.515 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.524593+0000", 2024-04-20T17:08:28.515 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.515 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.515 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.515 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:28.515 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.524593+0000", 2024-04-20T17:08:28.515 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.516 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.516 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.516 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:28.516 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.524595+0000", 2024-04-20T17:08:28.516 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.857e-06 2024-04-20T17:08:28.516 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.516 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.516 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:28.516 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.524622+0000", 2024-04-20T17:08:28.516 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.758e-05 2024-04-20T17:08:28.516 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.516 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.516 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:28.516 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.524633+0000", 2024-04-20T17:08:28.516 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0791000000000001e-05 2024-04-20T17:08:28.516 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.517 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.517 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:28.517 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.524640+0000", 2024-04-20T17:08:28.517 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4769999999999997e-06 2024-04-20T17:08:28.517 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.517 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.517 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:28.517 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.524700+0000", 2024-04-20T17:08:28.517 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9993e-05 2024-04-20T17:08:28.517 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.517 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.517 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:28.517 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.524731+0000", 2024-04-20T17:08:28.517 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0224000000000001e-05 2024-04-20T17:08:28.518 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.518 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.518 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:28.518 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.535114+0000", 2024-04-20T17:08:28.518 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010383514999999999 2024-04-20T17:08:28.518 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.518 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.518 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:28.518 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.535156+0000", 2024-04-20T17:08:28.518 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1394999999999999e-05 2024-04-20T17:08:28.518 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.518 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:28.518 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.518 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.519 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.519 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:34848 2.5 e122/115 2:a970f203:::benchmark_data_smithi171_78918_object34847:head v 122'2162, mlcod=122'2159)", 2024-04-20T17:08:28.519 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:24.525316+0000", 2024-04-20T17:08:28.519 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 2.8025464499999999, 2024-04-20T17:08:28.519 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0099972140000000008, 2024-04-20T17:08:28.519 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:28.519 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:28.519 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:28.519 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:28.519 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.519 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:28.519 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.525316+0000", 2024-04-20T17:08:28.519 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.519 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.519 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.520 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:28.520 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.525316+0000", 2024-04-20T17:08:28.520 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.520 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.520 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.520 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:28.520 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.525317+0000", 2024-04-20T17:08:28.520 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3319999999999999e-06 2024-04-20T17:08:28.520 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.520 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.520 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:28.520 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.525349+0000", 2024-04-20T17:08:28.520 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1695000000000001e-05 2024-04-20T17:08:28.520 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.521 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.521 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:28.521 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.525359+0000", 2024-04-20T17:08:28.521 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7629999999999995e-06 2024-04-20T17:08:28.521 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.521 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.521 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:28.521 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.525366+0000", 2024-04-20T17:08:28.521 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6730000000000003e-06 2024-04-20T17:08:28.521 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.521 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.521 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:28.521 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.525439+0000", 2024-04-20T17:08:28.522 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3787000000000004e-05 2024-04-20T17:08:28.522 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.522 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.522 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:28.522 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.525457+0000", 2024-04-20T17:08:28.522 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7907999999999999e-05 2024-04-20T17:08:28.522 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.522 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.522 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:28.522 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.535254+0000", 2024-04-20T17:08:28.522 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0097968880000000001 2024-04-20T17:08:28.523 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.523 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.523 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:28.523 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.535313+0000", 2024-04-20T17:08:28.523 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9168e-05 2024-04-20T17:08:28.523 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.523 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:28.523 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.523 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.523 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.523 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:34849 2.4 e122/115 2:219c3b14:::benchmark_data_smithi171_78918_object34848:head v 122'2242, mlcod=122'2240)", 2024-04-20T17:08:28.523 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:24.525864+0000", 2024-04-20T17:08:28.523 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 2.8019989560000003, 2024-04-20T17:08:28.523 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0093261179999999996, 2024-04-20T17:08:28.523 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:28.523 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:28.524 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:28.524 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:28.524 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.524 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:28.524 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.525864+0000", 2024-04-20T17:08:28.524 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.524 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.524 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.524 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:28.524 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.525864+0000", 2024-04-20T17:08:28.525 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.525 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.525 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.525 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:28.525 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.525865+0000", 2024-04-20T17:08:28.525 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.578e-06 2024-04-20T17:08:28.525 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.525 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.525 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:28.525 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.525889+0000", 2024-04-20T17:08:28.525 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4182999999999999e-05 2024-04-20T17:08:28.526 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.526 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.526 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:28.526 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.525898+0000", 2024-04-20T17:08:28.526 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8410000000000006e-06 2024-04-20T17:08:28.526 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.526 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.526 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:28.526 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.525905+0000", 2024-04-20T17:08:28.526 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4749999999999998e-06 2024-04-20T17:08:28.526 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.526 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.527 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:28.527 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.526017+0000", 2024-04-20T17:08:28.527 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011188099999999999 2024-04-20T17:08:28.527 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.527 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.527 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:28.527 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.526035+0000", 2024-04-20T17:08:28.527 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8179999999999999e-05 2024-04-20T17:08:28.527 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.527 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.527 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:28.528 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.535158+0000", 2024-04-20T17:08:28.528 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0091236249999999998 2024-04-20T17:08:28.528 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.528 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.528 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:28.528 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.535190+0000", 2024-04-20T17:08:28.528 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1355e-05 2024-04-20T17:08:28.528 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.528 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:28.528 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.528 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.528 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.528 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:34850 2.4 e122/115 2:2441d0ff:::benchmark_data_smithi171_78918_object34849:head v 122'2243, mlcod=122'2240)", 2024-04-20T17:08:28.528 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:24.525922+0000", 2024-04-20T17:08:28.528 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 2.8019402950000001, 2024-04-20T17:08:28.529 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.009284446, 2024-04-20T17:08:28.529 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:28.529 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:28.529 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:28.529 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:28.529 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.529 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:28.529 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.525922+0000", 2024-04-20T17:08:28.529 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.529 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.529 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.529 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:28.529 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.525922+0000", 2024-04-20T17:08:28.530 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.530 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.530 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.530 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:28.530 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.525923+0000", 2024-04-20T17:08:28.530 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2500000000000005e-07 2024-04-20T17:08:28.530 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.530 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.530 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:28.530 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.525944+0000", 2024-04-20T17:08:28.530 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0959999999999999e-05 2024-04-20T17:08:28.530 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.531 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.531 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:28.531 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.525950+0000", 2024-04-20T17:08:28.531 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9220000000000002e-06 2024-04-20T17:08:28.531 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.531 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.531 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:28.531 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.525954+0000", 2024-04-20T17:08:28.531 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7010000000000004e-06 2024-04-20T17:08:28.531 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.532 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.532 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:28.532 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.527066+0000", 2024-04-20T17:08:28.532 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0011111179999999999 2024-04-20T17:08:28.532 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.532 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.532 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:28.532 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.527083+0000", 2024-04-20T17:08:28.532 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7047000000000001e-05 2024-04-20T17:08:28.532 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.532 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.532 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:28.533 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.535192+0000", 2024-04-20T17:08:28.533 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0081090669999999993 2024-04-20T17:08:28.533 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.533 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.533 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:28.533 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.535207+0000", 2024-04-20T17:08:28.533 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5006e-05 2024-04-20T17:08:28.533 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.533 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:28.533 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.533 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.533 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.533 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:34856 2.5 e122/115 2:afe3d8fa:::benchmark_data_smithi171_78918_object34855:head v 122'2163, mlcod=122'2159)", 2024-04-20T17:08:28.533 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:24.526453+0000", 2024-04-20T17:08:28.533 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 2.8014097680000001, 2024-04-20T17:08:28.534 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0088926019999999995, 2024-04-20T17:08:28.534 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:28.534 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:28.534 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:28.534 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:28.534 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.534 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:28.534 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.526453+0000", 2024-04-20T17:08:28.534 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.534 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.534 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.534 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:28.534 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.526453+0000", 2024-04-20T17:08:28.535 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.535 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.535 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.535 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:28.535 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.526454+0000", 2024-04-20T17:08:28.535 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6299999999999993e-07 2024-04-20T17:08:28.535 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.535 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.535 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:28.535 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.526485+0000", 2024-04-20T17:08:28.535 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1075999999999999e-05 2024-04-20T17:08:28.535 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.535 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.535 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:28.535 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.526493+0000", 2024-04-20T17:08:28.536 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2719999999999997e-06 2024-04-20T17:08:28.536 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.536 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.536 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:28.536 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.526500+0000", 2024-04-20T17:08:28.536 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4400000000000002e-06 2024-04-20T17:08:28.536 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.536 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.536 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:28.536 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.526544+0000", 2024-04-20T17:08:28.536 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4990000000000001e-05 2024-04-20T17:08:28.536 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.536 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.536 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:28.537 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.526562+0000", 2024-04-20T17:08:28.537 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7788e-05 2024-04-20T17:08:28.537 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.537 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.537 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:28.537 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.535316+0000", 2024-04-20T17:08:28.537 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0087540680000000003 2024-04-20T17:08:28.537 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.537 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.537 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:28.537 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:24.535345+0000", 2024-04-20T17:08:28.537 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9005000000000001e-05 2024-04-20T17:08:28.537 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.537 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:28.538 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.538 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.538 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.538 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:37526 2.c e123/115 2:394f2ff2:::benchmark_data_smithi171_78918_object37525:head v 123'2319, mlcod=123'2316)", 2024-04-20T17:08:28.538 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:25.321444+0000", 2024-04-20T17:08:28.538 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 2.006418992, 2024-04-20T17:08:28.538 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014108766, 2024-04-20T17:08:28.538 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:28.538 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:28.538 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:28.538 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:28.538 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.538 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.321444+0000", 2024-04-20T17:08:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.321444+0000", 2024-04-20T17:08:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.321444+0000", 2024-04-20T17:08:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2099999999999996e-07 2024-04-20T17:08:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:28.540 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.321501+0000", 2024-04-20T17:08:28.540 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7241999999999998e-05 2024-04-20T17:08:28.540 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.540 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.540 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:28.540 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.321509+0000", 2024-04-20T17:08:28.540 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7989999999999994e-06 2024-04-20T17:08:28.540 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.540 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.540 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:28.540 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.321516+0000", 2024-04-20T17:08:28.540 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2979999999999997e-06 2024-04-20T17:08:28.540 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.540 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.325339+0000", 2024-04-20T17:08:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003823178 2024-04-20T17:08:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.325358+0000", 2024-04-20T17:08:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8838000000000002e-05 2024-04-20T17:08:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.335502+0000", 2024-04-20T17:08:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010144468 2024-04-20T17:08:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.542 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:28.542 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.335552+0000", 2024-04-20T17:08:28.542 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0222000000000001e-05 2024-04-20T17:08:28.542 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.542 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:28.542 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.542 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.542 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.542 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:37533 2.c e123/115 2:39facf37:::benchmark_data_smithi171_78918_object37532:head v 123'2320, mlcod=123'2316)", 2024-04-20T17:08:28.542 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:25.324829+0000", 2024-04-20T17:08:28.542 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 2.0030340029999998, 2024-04-20T17:08:28.542 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010751884, 2024-04-20T17:08:28.542 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:28.542 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:28.543 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:28.543 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:28.543 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.543 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:28.543 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.324829+0000", 2024-04-20T17:08:28.543 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.543 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.543 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.543 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:28.543 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.324829+0000", 2024-04-20T17:08:28.543 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.543 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.544 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.544 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:28.544 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.324830+0000", 2024-04-20T17:08:28.544 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2419999999999999e-06 2024-04-20T17:08:28.544 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.544 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.544 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:28.544 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.324866+0000", 2024-04-20T17:08:28.544 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6390000000000002e-05 2024-04-20T17:08:28.544 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.544 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.544 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:28.544 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.324876+0000", 2024-04-20T17:08:28.544 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0015e-05 2024-04-20T17:08:28.544 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.545 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.545 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:28.545 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.324882+0000", 2024-04-20T17:08:28.545 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9510000000000003e-06 2024-04-20T17:08:28.545 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.545 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.545 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:28.545 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.326514+0000", 2024-04-20T17:08:28.545 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0016321770000000001 2024-04-20T17:08:28.545 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.545 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.545 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:28.546 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.326532+0000", 2024-04-20T17:08:28.546 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7320999999999999e-05 2024-04-20T17:08:28.546 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.546 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.546 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:28.546 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.335556+0000", 2024-04-20T17:08:28.546 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0090245280000000004 2024-04-20T17:08:28.546 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.546 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.546 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:28.546 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.335580+0000", 2024-04-20T17:08:28.546 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4260000000000002e-05 2024-04-20T17:08:28.546 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.547 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:28.547 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.547 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.547 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.547 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:37537 2.c e123/115 2:3ce510f3:::benchmark_data_smithi171_78918_object37536:head v 123'2321, mlcod=123'2316)", 2024-04-20T17:08:28.547 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:25.324894+0000", 2024-04-20T17:08:28.547 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 2.0029686849999999, 2024-04-20T17:08:28.547 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010712764, 2024-04-20T17:08:28.547 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:28.547 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:28.547 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:28.547 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:28.547 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.548 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:28.548 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.324894+0000", 2024-04-20T17:08:28.548 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.548 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.548 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.548 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:28.548 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.324894+0000", 2024-04-20T17:08:28.548 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.548 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.548 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.548 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:28.548 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.324894+0000", 2024-04-20T17:08:28.548 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4199999999999995e-07 2024-04-20T17:08:28.549 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.549 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.549 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:28.549 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.324932+0000", 2024-04-20T17:08:28.549 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7716999999999998e-05 2024-04-20T17:08:28.549 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.549 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.549 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:28.549 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.324939+0000", 2024-04-20T17:08:28.549 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2989999999999999e-06 2024-04-20T17:08:28.549 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.549 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.549 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:28.550 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.324945+0000", 2024-04-20T17:08:28.550 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5099999999999998e-06 2024-04-20T17:08:28.550 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.550 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.550 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:28.550 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.327671+0000", 2024-04-20T17:08:28.550 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0027262150000000001 2024-04-20T17:08:28.550 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.550 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.550 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:28.550 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.327702+0000", 2024-04-20T17:08:28.550 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0583999999999997e-05 2024-04-20T17:08:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.335583+0000", 2024-04-20T17:08:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0078814880000000007 2024-04-20T17:08:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.335607+0000", 2024-04-20T17:08:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3309e-05 2024-04-20T17:08:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.552 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.552 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:37538 2.c e123/115 2:34d1edd6:::benchmark_data_smithi171_78918_object37537:head v 123'2322, mlcod=123'2316)", 2024-04-20T17:08:28.552 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:25.326122+0000", 2024-04-20T17:08:28.552 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 2.0017401050000001, 2024-04-20T17:08:28.552 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0095054639999999999, 2024-04-20T17:08:28.552 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:28.552 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:28.552 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:28.552 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:28.552 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.552 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:28.552 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.326122+0000", 2024-04-20T17:08:28.552 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.553 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.553 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.553 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:28.553 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.326122+0000", 2024-04-20T17:08:28.553 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.553 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.553 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.553 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:28.553 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.326123+0000", 2024-04-20T17:08:28.553 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4799999999999997e-07 2024-04-20T17:08:28.554 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.554 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.554 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:28.554 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.326197+0000", 2024-04-20T17:08:28.554 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4066999999999994e-05 2024-04-20T17:08:28.554 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.554 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.554 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:28.554 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.326206+0000", 2024-04-20T17:08:28.554 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5320000000000006e-06 2024-04-20T17:08:28.554 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.554 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.554 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:28.555 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.326212+0000", 2024-04-20T17:08:28.555 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2049999999999999e-06 2024-04-20T17:08:28.555 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.555 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.555 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:28.555 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.328855+0000", 2024-04-20T17:08:28.555 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0026427239999999999 2024-04-20T17:08:28.555 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.555 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.555 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:28.555 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.328876+0000", 2024-04-20T17:08:28.555 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1379e-05 2024-04-20T17:08:28.555 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.335608+0000", 2024-04-20T17:08:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0067321209999999998 2024-04-20T17:08:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:25.335628+0000", 2024-04-20T17:08:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9488e-05 2024-04-20T17:08:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.557 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:40771 2.4 e124/115 2:242dd1e8:::benchmark_data_smithi171_78918_object40770:head v 124'2601, mlcod=124'2599)", 2024-04-20T17:08:28.557 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:26.362185+0000", 2024-04-20T17:08:28.557 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.96567781200000002, 2024-04-20T17:08:28.557 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011376412000000001, 2024-04-20T17:08:28.557 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:28.557 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:28.557 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:28.557 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:28.557 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.557 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:28.557 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:26.362185+0000", 2024-04-20T17:08:28.557 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.557 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.557 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.558 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:28.558 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:26.362185+0000", 2024-04-20T17:08:28.558 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:28.558 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.558 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.558 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:28.558 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:26.362187+0000", 2024-04-20T17:08:28.558 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1500000000000002e-06 2024-04-20T17:08:28.558 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.558 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.558 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:28.558 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:26.362213+0000", 2024-04-20T17:08:28.558 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5899999999999999e-05 2024-04-20T17:08:28.559 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.559 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.559 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:28.559 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:26.362231+0000", 2024-04-20T17:08:28.559 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8099999999999999e-05 2024-04-20T17:08:28.559 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.559 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.559 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:26.362240+0000", 2024-04-20T17:08:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.55e-06 2024-04-20T17:08:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:26.362439+0000", 2024-04-20T17:08:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000198591 2024-04-20T17:08:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:26.362463+0000", 2024-04-20T17:08:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3949999999999999e-05 2024-04-20T17:08:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:26.373510+0000", 2024-04-20T17:08:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011046795999999999 2024-04-20T17:08:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:26.373561+0000", 2024-04-20T17:08:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1375000000000002e-05 2024-04-20T17:08:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:28.562 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.562 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:28.562 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:28.562 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:08:29.438 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 13 16 50089 50073 240.716 175.688 0.00461381 0.00415258 2024-04-20T17:08:29.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:29 smithi005 ceph-mon[39562]: pgmap v385: 17 pgs: 17 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 256 MiB/s wr, 4.09k op/s 2024-04-20T17:08:29.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:29 smithi106 ceph-mon[44423]: pgmap v385: 17 pgs: 17 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 256 MiB/s wr, 4.09k op/s 2024-04-20T17:08:29.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:29 smithi171 ceph-mon[44189]: pgmap v385: 17 pgs: 17 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 256 MiB/s wr, 4.09k op/s 2024-04-20T17:08:30.439 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 14 16 52527 52511 234.405 152.375 0.00285253 0.00426427 2024-04-20T17:08:30.915 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:08:30.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:30.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:08:31.020 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:08:31.062 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:31.091 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:08:31.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:31.096 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:08:31.196 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:08:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:31 smithi005 ceph-mon[39562]: pgmap v386: 17 pgs: 17 active+clean; 2.7 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 236 MiB/s wr, 3.78k op/s 2024-04-20T17:08:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:31 smithi005 ceph-mon[39562]: osdmap e125: 12 total, 12 up, 12 in 2024-04-20T17:08:31.226 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:31.251 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:08:31.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:31.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:08:31.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:31 smithi171 ceph-mon[44189]: pgmap v386: 17 pgs: 17 active+clean; 2.7 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 236 MiB/s wr, 3.78k op/s 2024-04-20T17:08:31.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:31 smithi171 ceph-mon[44189]: osdmap e125: 12 total, 12 up, 12 in 2024-04-20T17:08:31.355 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:08:31.384 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:31 smithi106 ceph-mon[44423]: pgmap v386: 17 pgs: 17 active+clean; 2.7 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 236 MiB/s wr, 3.78k op/s 2024-04-20T17:08:31.384 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:31 smithi106 ceph-mon[44423]: osdmap e125: 12 total, 12 up, 12 in 2024-04-20T17:08:31.387 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:31.407 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:08:31.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:31.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:08:31.439 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 15 16 55241 55225 230.085 169.625 0.00665202 0.00434447 2024-04-20T17:08:31.512 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:08:31.553 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:31.582 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:08:31.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:31.585 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:08:31.686 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:08:31.726 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:32.439 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 16 16 57957 57941 226.313 169.75 0.00238364 0.00441667 2024-04-20T17:08:32.491 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:08:32.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:32.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:08:32.596 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:08:32.626 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:32.647 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:08:32.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:32.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:08:32.751 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:08:32.790 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:32.817 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:08:32.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:32.822 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:08:32.923 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:08:32.969 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:33.020 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:08:33.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:33.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:08:33.070 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-20T17:08:33.133 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:08:33.162 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:33.185 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:08:33.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:33.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:08:33.289 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:08:33.298 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:33 smithi005 ceph-mon[39562]: pgmap v388: 17 pgs: 17 active+clean; 3.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 200 MiB/s wr, 3.19k op/s 2024-04-20T17:08:33.333 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:08:33.344 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:08:33.381 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:33.439 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 17 16 60325 60309 221.706 148 0.00553654 0.00450828 2024-04-20T17:08:33.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:33 smithi106 ceph-mon[44423]: pgmap v388: 17 pgs: 17 active+clean; 3.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 200 MiB/s wr, 3.19k op/s 2024-04-20T17:08:33.517 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:08:33.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:33.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:08:33.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:33 smithi171 ceph-mon[44189]: pgmap v388: 17 pgs: 17 active+clean; 3.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 200 MiB/s wr, 3.19k op/s 2024-04-20T17:08:33.628 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:08:33.694 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:33.735 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:08:33.771 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:08:33.804 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:08:33.835 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:08:33.866 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:08:33.904 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:08:33.917 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:08:33.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:33.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:08:33.951 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:08:33.981 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:08:34.012 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:08:34.036 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:08:34.051 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:08:34.088 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:34.095 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:08:34.134 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:08:34.142 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:08:34.143 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:08:34.143 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:08:34.143 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:08:34.143 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:08:34.143 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:08:34.143 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:08:34.143 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:08:34.143 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:08:34.143 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:08:34.143 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:08:34.143 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:08:34.397 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:34 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4012354127' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:08:34.398 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:34.398 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:34.398 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:34.398 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:34.398 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:34.439 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 18 16 62598 62582 217.281 142.062 0.00878269 0.00460067 2024-04-20T17:08:34.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:34 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4012354127' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:08:34.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:34.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:34.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:34.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:34.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:34.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:34 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4012354127' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:08:34.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:34.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:34.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:34.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:34.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:35.439 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 19 16 65086 65070 214.029 155.5 0.00613675 0.00467064 2024-04-20T17:08:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:35 smithi005 ceph-mon[39562]: pgmap v389: 17 pgs: 17 active+clean; 3.5 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 206 MiB/s wr, 3.29k op/s 2024-04-20T17:08:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:35 smithi106 ceph-mon[44423]: pgmap v389: 17 pgs: 17 active+clean; 3.5 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 206 MiB/s wr, 3.29k op/s 2024-04-20T17:08:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:35.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:35 smithi171 ceph-mon[44189]: pgmap v389: 17 pgs: 17 active+clean; 3.5 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 206 MiB/s wr, 3.29k op/s 2024-04-20T17:08:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:36.439 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:08:36.440534 min lat: 0.00106268 max lat: 0.207047 avg lat: 0.00473603 2024-04-20T17:08:36.439 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:08:36.439 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 20 16 67561 67545 211.061 154.688 0.00348442 0.00473603 2024-04-20T17:08:36.463 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:08:36.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:36.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:08:36.569 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:08:36.620 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:36.972 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:08:36.983 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:08:36.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:36.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:08:37.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:36 smithi171 ceph-mon[44189]: osdmap e126: 12 total, 12 up, 12 in 2024-04-20T17:08:37.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:36 smithi171 ceph-mon[44189]: pgmap v391: 17 pgs: 17 active+clean; 3.7 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 164 MiB/s wr, 2.62k op/s 2024-04-20T17:08:37.090 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:08:37.116 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:36 smithi106 ceph-mon[44423]: osdmap e126: 12 total, 12 up, 12 in 2024-04-20T17:08:37.117 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:36 smithi106 ceph-mon[44423]: pgmap v391: 17 pgs: 17 active+clean; 3.7 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 164 MiB/s wr, 2.62k op/s 2024-04-20T17:08:37.119 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:37.140 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:08:37.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:37.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:08:37.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:36 smithi005 ceph-mon[39562]: osdmap e126: 12 total, 12 up, 12 in 2024-04-20T17:08:37.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:36 smithi005 ceph-mon[39562]: pgmap v391: 17 pgs: 17 active+clean; 3.7 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 164 MiB/s wr, 2.62k op/s 2024-04-20T17:08:37.243 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:08:37.270 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:37.292 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:08:37.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:37.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:08:37.396 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:08:37.439 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 21 16 69641 69625 207.2 130 0.00372811 0.00482389 2024-04-20T17:08:37.447 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:37.942 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:08:38.217 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:08:38.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:38.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:08:38.328 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:08:38.369 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:38.394 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:08:38.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:38.397 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:08:38.439 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 22 16 71823 71807 203.981 136.375 0.00339031 0.00490051 2024-04-20T17:08:38.498 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:08:38.529 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:38.551 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:08:38.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:38.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:08:38.656 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:08:38.711 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:38.875 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:08:38.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:38.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:08:38.978 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:08:39.036 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:39.114 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:08:39.128 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:08:39.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:38 smithi106 ceph-mon[44423]: pgmap v392: 17 pgs: 17 active+clean; 4.0 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 171 MiB/s wr, 2.74k op/s 2024-04-20T17:08:39.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:38 smithi171 ceph-mon[44189]: pgmap v392: 17 pgs: 17 active+clean; 4.0 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 171 MiB/s wr, 2.74k op/s 2024-04-20T17:08:39.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:38 smithi005 ceph-mon[39562]: pgmap v392: 17 pgs: 17 active+clean; 4.0 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 171 MiB/s wr, 2.74k op/s 2024-04-20T17:08:39.427 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:08:39.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:39.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:08:39.439 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 23 16 73766 73750 200.391 121.438 0.00887953 0.00498845 2024-04-20T17:08:39.534 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:08:39.575 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:39.607 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:08:39.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:39.612 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:08:39.713 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:08:39.755 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:39.782 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:08:39.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:39.787 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:08:39.888 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:08:39.923 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:39.945 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:08:39.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:39.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:08:40.003 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:08:40.049 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:08:40.080 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:40.103 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:08:40.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:40.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:08:40.208 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:08:40.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:39 smithi005 ceph-mon[39562]: from='client.24761 v1:172.21.15.5:0/4157891196' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:08:40.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:39 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/619666765' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:08:40.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:39 smithi005 ceph-mon[39562]: from='client.24763 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:08:40.234 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:39 smithi106 ceph-mon[44423]: from='client.24761 v1:172.21.15.5:0/4157891196' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:08:40.235 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:39 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/619666765' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:08:40.235 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:39 smithi106 ceph-mon[44423]: from='client.24763 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:08:40.237 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:40.260 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:08:40.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:40.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:08:40.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:39 smithi171 ceph-mon[44189]: from='client.24761 v1:172.21.15.5:0/4157891196' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:08:40.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:39 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/619666765' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:08:40.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:39 smithi171 ceph-mon[44189]: from='client.24763 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:08:40.365 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:08:40.417 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:40.439 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 24 16 75934 75918 197.687 135.5 0.00327261 0.00505687 2024-04-20T17:08:40.492 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:08:40.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:40.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:08:40.539 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2024-04-20T17:08:37.921120+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4315349536,"num_objects":65840,"num_object_clones":0,"num_object_copies":131680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65840,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":66019,"num_write_kb":4216570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65975,"ondisk_log_size":65975,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":8643288,"kb_used_data":8223408,"kb_used_omap":18,"kb_used_meta":419757,"kb_avail":1116298536,"statfs":{"total":1151940427776,"available":1143089700864,"internally_reserved":0,"allocated":8420769792,"data_stored":8415260948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":429831481},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,3,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1439105024,"num_objects":21959,"num_object_clones":0,"num_object_copies":43918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21959,"num_write_kb":1405376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003868"},"pg_stats":[{"pgid":"2.f","version":"125'3744","reported_seq":3773,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-20T17:08:33.583488+0000","last_change":"2024-04-20T17:08:06.006163+0000","last_active":"2024-04-20T17:08:33.583488+0000","last_peered":"2024-04-20T17:08:33.583488+0000","last_clean":"2024-04-20T17:08:33.583488+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:08:33.583488+0000","last_undegraded":"2024-04-20T17:08:33.583488+0000","last_fullsized":"2024-04-20T17:08:33.583488+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":3744,"log_dups_size":0,"ondisk_log_size":3744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:09:09.053339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245366784,"num_objects":3744,"num_object_clones":0,"num_object_copies":7488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3744,"num_write_kb":239616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"126'4230","reported_seq":4261,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-20T17:08:36.679861+0000","last_change":"2024-04-20T17:08:06.008342+0000","last_active":"2024-04-20T17:08:36.679861+0000","last_peered":"2024-04-20T17:08:36.679861+0000","last_clean":"2024-04-20T17:08:36.679861+0000","last_became_active":"2024-04-20T17:08:06.008052+0000","last_became_peered":"2024-04-20T17:08:06.008052+0000","last_unstale":"2024-04-20T17:08:36.679861+0000","last_undegraded":"2024-04-20T17:08:36.679861+0000","last_fullsized":"2024-04-20T17:08:36.679861+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":4230,"log_dups_size":0,"ondisk_log_size":4230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:09:09.771029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277217280,"num_objects":4230,"num_object_clones":0,"num_object_copies":8460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4230,"num_write_kb":270720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"125'4141","reported_seq":4170,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-20T17:08:35.566535+0000","last_change":"2024-04-20T17:08:06.013628+0000","last_active":"2024-04-20T17:08:35.566535+0000","last_peered":"2024-04-20T17:08:35.566535+0000","last_clean":"2024-04-20T17:08:35.566535+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:08:35.566535+0000","last_undegraded":"2024-04-20T17:08:35.566535+0000","last_fullsized":"2024-04-20T17:08:35.566535+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":4141,"log_dups_size":0,"ondisk_log_size":4141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:09:05.751594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271384576,"num_objects":4141,"num_object_clones":0,"num_object_copies":8282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4141,"num_write_kb":265024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"125'3784","reported_seq":3812,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-20T17:08:33.588068+0000","last_change":"2024-04-20T17:08:06.028087+0000","last_active":"2024-04-20T17:08:33.588068+0000","last_peered":"2024-04-20T17:08:33.588068+0000","last_clean":"2024-04-20T17:08:33.588068+0000","last_became_active":"2024-04-20T17:08:06.027804+0000","last_became_peered":"2024-04-20T17:08:06.027804+0000","last_unstale":"2024-04-20T17:08:33.588068+0000","last_undegraded":"2024-04-20T17:08:33.588068+0000","last_fullsized":"2024-04-20T17:08:33.588068+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":3784,"log_dups_size":0,"ondisk_log_size":3784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:09:06.449260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247988224,"num_objects":3784,"num_object_clones":0,"num_object_copies":7568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3784,"num_write_kb":242176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"125'4056","reported_seq":4084,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-20T17:08:35.557237+0000","last_change":"2024-04-20T17:08:06.010896+0000","last_active":"2024-04-20T17:08:35.557237+0000","last_peered":"2024-04-20T17:08:35.557237+0000","last_clean":"2024-04-20T17:08:35.557237+0000","last_became_active":"2024-04-20T17:08:06.010691+0000","last_became_peered":"2024-04-20T17:08:06.010691+0000","last_unstale":"2024-04-20T17:08:35.557237+0000","last_undegraded":"2024-04-20T17:08:35.557237+0000","last_fullsized":"2024-04-20T17:08:35.557237+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":4056,"log_dups_size":0,"ondisk_log_size":4056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:09:22.642113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265814016,"num_objects":4056,"num_object_clones":0,"num_object_copies":8112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4056,"num_write_kb":259584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":396,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-20T17:08:35.804341+0000","last_change":"2024-04-20T17:06:51.212018+0000","last_active":"2024-04-20T17:08:35.804341+0000","last_peered":"2024-04-20T17:08:35.804341+0000","last_clean":"2024-04-20T17:08:35.804341+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:08:35.804341+0000","last_undegraded":"2024-04-20T17:08:35.804341+0000","last_fullsized":"2024-04-20T17:08:35.804341+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:06:51.211902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:06:51.211902+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"126'4426","reported_seq":4457,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-20T17:08:37.773469+0000","last_change":"2024-04-20T17:08:06.045012+0000","last_active":"2024-04-20T17:08:37.773469+0000","last_peered":"2024-04-20T17:08:37.773469+0000","last_clean":"2024-04-20T17:08:37.773469+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:08:37.773469+0000","last_undegraded":"2024-04-20T17:08:37.773469+0000","last_fullsized":"2024-04-20T17:08:37.773469+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":4426,"log_dups_size":0,"ondisk_log_size":4426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:09:31.502776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290062336,"num_objects":4426,"num_object_clones":0,"num_object_copies":8852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4426,"num_write_kb":283264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"125'3808","reported_seq":3837,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-20T17:08:33.581936+0000","last_change":"2024-04-20T17:08:06.003337+0000","last_active":"2024-04-20T17:08:33.581936+0000","last_peered":"2024-04-20T17:08:33.581936+0000","last_clean":"2024-04-20T17:08:33.581936+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:08:33.581936+0000","last_undegraded":"2024-04-20T17:08:33.581936+0000","last_fullsized":"2024-04-20T17:08:33.581936+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":3808,"log_dups_size":0,"ondisk_log_size":3808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:09:21.431488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249561088,"num_objects":3808,"num_object_clones":0,"num_object_copies":7616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3808,"num_write_kb":243712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"126'4328","reported_seq":4359,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-20T17:08:36.672325+0000","last_change":"2024-04-20T17:08:06.009697+0000","last_active":"2024-04-20T17:08:36.672325+0000","last_peered":"2024-04-20T17:08:36.672325+0000","last_clean":"2024-04-20T17:08:36.672325+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:08:36.672325+0000","last_undegraded":"2024-04-20T17:08:36.672325+0000","last_fullsized":"2024-04-20T17:08:36.672325+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":4328,"log_dups_size":0,"ondisk_log_size":4328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:09:21.041435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283639808,"num_objects":4328,"num_object_clones":0,"num_object_copies":8656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4328,"num_write_kb":276992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"126'4495","reported_seq":4526,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-20T17:08:37.401614+0000","last_change":"2024-04-20T17:08:06.045085+0000","last_active":"2024-04-20T17:08:37.401614+0000","last_peered":"2024-04-20T17:08:37.401614+0000","last_clean":"2024-04-20T17:08:37.401614+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:08:37.401614+0000","last_undegraded":"2024-04-20T17:08:37.401614+0000","last_fullsized":"2024-04-20T17:08:37.401614+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":4495,"log_dups_size":0,"ondisk_log_size":4495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:09:15.615140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294584320,"num_objects":4495,"num_object_clones":0,"num_object_copies":8990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4495,"num_write_kb":287680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"126'4338","reported_seq":4369,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-20T17:08:37.774367+0000","last_change":"2024-04-20T17:08:06.006061+0000","last_active":"2024-04-20T17:08:37.774367+0000","last_peered":"2024-04-20T17:08:37.774367+0000","last_clean":"2024-04-20T17:08:37.774367+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:08:37.774367+0000","last_undegraded":"2024-04-20T17:08:37.774367+0000","last_fullsized":"2024-04-20T17:08:37.774367+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":4338,"log_dups_size":0,"ondisk_log_size":4338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:09:28.073363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284295168,"num_objects":4338,"num_object_clones":0,"num_object_copies":8676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4338,"num_write_kb":277632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"126'4225","reported_seq":4256,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-20T17:08:36.657340+0000","last_change":"2024-04-20T17:08:06.011184+0000","last_active":"2024-04-20T17:08:36.657340+0000","last_peered":"2024-04-20T17:08:36.657340+0000","last_clean":"2024-04-20T17:08:36.657340+0000","last_became_active":"2024-04-20T17:08:06.010879+0000","last_became_peered":"2024-04-20T17:08:06.010879+0000","last_unstale":"2024-04-20T17:08:36.657340+0000","last_undegraded":"2024-04-20T17:08:36.657340+0000","last_fullsized":"2024-04-20T17:08:36.657340+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":4225,"log_dups_size":0,"ondisk_log_size":4225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:09:26.988436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276889600,"num_objects":4225,"num_object_clones":0,"num_object_copies":8450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4225,"num_write_kb":270400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"126'4457","reported_seq":4488,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-20T17:08:37.459883+0000","last_change":"2024-04-20T17:08:06.028253+0000","last_active":"2024-04-20T17:08:37.459883+0000","last_peered":"2024-04-20T17:08:37.459883+0000","last_clean":"2024-04-20T17:08:37.459883+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:08:37.459883+0000","last_undegraded":"2024-04-20T17:08:37.459883+0000","last_fullsized":"2024-04-20T17:08:37.459883+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":4457,"log_dups_size":0,"ondisk_log_size":4457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:09:11.200730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292093952,"num_objects":4457,"num_object_clones":0,"num_object_copies":8914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4457,"num_write_kb":285248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"125'3700","reported_seq":3729,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-20T17:08:33.587715+0000","last_change":"2024-04-20T17:08:06.005814+0000","last_active":"2024-04-20T17:08:33.587715+0000","last_peered":"2024-04-20T17:08:33.587715+0000","last_clean":"2024-04-20T17:08:33.587715+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:08:33.587715+0000","last_undegraded":"2024-04-20T17:08:33.587715+0000","last_fullsized":"2024-04-20T17:08:33.587715+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":3700,"log_dups_size":0,"ondisk_log_size":3700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:09:32.391282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242483200,"num_objects":3700,"num_object_clones":0,"num_object_copies":7400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3700,"num_write_kb":236800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"126'4408","reported_seq":4438,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-20T17:08:37.486186+0000","last_change":"2024-04-20T17:08:06.006951+0000","last_active":"2024-04-20T17:08:37.486186+0000","last_peered":"2024-04-20T17:08:37.486186+0000","last_clean":"2024-04-20T17:08:37.486186+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:08:37.486186+0000","last_undegraded":"2024-04-20T17:08:37.486186+0000","last_fullsized":"2024-04-20T17:08:37.486186+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":4408,"log_dups_size":0,"ondisk_log_size":4408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:09:25.563784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288882688,"num_objects":4408,"num_object_clones":0,"num_object_copies":8816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":282112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"125'4004","reported_seq":4033,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-20T17:08:35.575921+0000","last_change":"2024-04-20T17:08:06.044934+0000","last_active":"2024-04-20T17:08:35.575921+0000","last_peered":"2024-04-20T17:08:35.575921+0000","last_clean":"2024-04-20T17:08:35.575921+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:08:35.575921+0000","last_undegraded":"2024-04-20T17:08:35.575921+0000","last_fullsized":"2024-04-20T17:08:35.575921+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":4004,"log_dups_size":0,"ondisk_log_size":4004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:09:15.034500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262406144,"num_objects":4004,"num_object_clones":0,"num_object_copies":8008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4004,"num_write_kb":256256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"125'3694","reported_seq":3723,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-04-20T17:08:33.594523+0000","last_change":"2024-04-20T17:08:06.028545+0000","last_active":"2024-04-20T17:08:33.594523+0000","last_peered":"2024-04-20T17:08:33.594523+0000","last_clean":"2024-04-20T17:08:33.594523+0000","last_became_active":"2024-04-20T17:08:06.028272+0000","last_became_peered":"2024-04-20T17:08:06.028272+0000","last_unstale":"2024-04-20T17:08:33.594523+0000","last_undegraded":"2024-04-20T17:08:33.594523+0000","last_fullsized":"2024-04-20T17:08:33.594523+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":3694,"log_dups_size":0,"ondisk_log_size":3694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:09:31.827174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242089984,"num_objects":3694,"num_object_clones":0,"num_object_copies":7388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3694,"num_write_kb":236416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":4314759168,"num_objects":65838,"num_object_clones":0,"num_object_copies":131676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65838,"num_write_kb":4213632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8554020864,"data_stored":8554020864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65838,"ondisk_log_size":65838,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991267,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":865296,"kb_used_data":838280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":92879856,"statfs":{"total":95995035648,"available":95108972544,"internally_reserved":0,"allocated":858398720,"data_stored":857932605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777252903,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":529404,"kb_used_data":485748,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":93215748,"statfs":{"total":95995035648,"available":95452925952,"internally_reserved":0,"allocated":497405952,"data_stored":496949491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":44693952},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612654,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":788732,"kb_used_data":749236,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":92956420,"statfs":{"total":95995035648,"available":95187374080,"internally_reserved":0,"allocated":767217664,"data_stored":766761223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762808883,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":975740,"kb_used_data":948724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":92769412,"statfs":{"total":95995035648,"available":94995877888,"internally_reserved":0,"allocated":971493376,"data_stored":971036935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168633,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":872848,"kb_used_data":841672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":92872304,"statfs":{"total":95995035648,"available":95101239296,"internally_reserved":0,"allocated":861872128,"data_stored":861406033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430269,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312460,"kb_used_data":281280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93432692,"statfs":{"total":95995035648,"available":95675076608,"internally_reserved":0,"allocated":288030720,"data_stored":287568177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1117772,"kb_used_data":1082432,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":92627380,"statfs":{"total":95995035648,"available":94850437120,"internally_reserved":0,"allocated":1108410368,"data_stored":1107947825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738461,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":790780,"kb_used_data":747124,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":92954372,"statfs":{"total":95995035648,"available":95185276928,"internally_reserved":0,"allocated":765054976,"data_stored":764598535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":20,"seq":85899346000,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":727804,"kb_used_data":696628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93017348,"statfs":{"total":95995035648,"available":95249764352,"internally_reserved":0,"allocated":713347072,"data_stored":712890631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182476,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":834812,"kb_used_data":791156,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":92910340,"statfs":{"total":95995035648,"available":95140188160,"internally_reserved":0,"allocated":810143744,"data_stored":809687303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":44693959},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986247,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":281596,"kb_used_data":254580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93463556,"statfs":{"total":95995035648,"available":95706681344,"internally_reserved":0,"allocated":260689920,"data_stored":260233479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790017,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":546044,"kb_used_data":506548,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93199108,"statfs":{"total":95995035648,"available":95435886592,"internally_reserved":0,"allocated":518705152,"data_stored":518248711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":776732672,"data_stored":776732672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1120206848,"data_stored":1120206848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":734855168,"data_stored":734855168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":815136768,"data_stored":815136768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":261029888,"data_stored":261029888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":537264128,"data_stored":537264128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":294649856,"data_stored":294649856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":861601792,"data_stored":861601792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":990773248,"data_stored":990773248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":791281664,"data_stored":791281664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":511574016,"data_stored":511574016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":858914816,"data_stored":858914816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:08:40.540 INFO:tasks.thrashosds.thrasher:Setting 2.4 to [6, 7, 1, 5] 2024-04-20T17:08:40.540 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.4', '6', '7', '1', '5'] 2024-04-20T17:08:40.540 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pg-upmap-items 2.4 6 7 1 5 2024-04-20T17:08:40.602 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:08:40.679 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:40.820 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:08:40.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:40.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:08:40.925 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:08:40.967 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:41.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:41 smithi106 ceph-mon[44423]: pgmap v393: 17 pgs: 17 active+clean; 4.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 156 MiB/s wr, 2.50k op/s 2024-04-20T17:08:41.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:41 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:08:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:41 smithi106 ceph-mon[44423]: from='client.24763 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:08:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:41 smithi106 ceph-mon[44423]: osdmap e127: 12 total, 12 up, 12 in 2024-04-20T17:08:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:41 smithi106 ceph-mon[44423]: osdmap e128: 12 total, 12 up, 12 in 2024-04-20T17:08:41.283 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:41 smithi005 ceph-mon[39562]: pgmap v393: 17 pgs: 17 active+clean; 4.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 156 MiB/s wr, 2.50k op/s 2024-04-20T17:08:41.283 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:41 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:08:41.283 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:41 smithi005 ceph-mon[39562]: from='client.24763 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:08:41.283 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:41 smithi005 ceph-mon[39562]: osdmap e127: 12 total, 12 up, 12 in 2024-04-20T17:08:41.283 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:41 smithi005 ceph-mon[39562]: osdmap e128: 12 total, 12 up, 12 in 2024-04-20T17:08:41.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:41 smithi171 ceph-mon[44189]: pgmap v393: 17 pgs: 17 active+clean; 4.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 156 MiB/s wr, 2.50k op/s 2024-04-20T17:08:41.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:41 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:08:41.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:41 smithi171 ceph-mon[44189]: from='client.24763 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:08:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:41 smithi171 ceph-mon[44189]: osdmap e127: 12 total, 12 up, 12 in 2024-04-20T17:08:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:41 smithi171 ceph-mon[44189]: osdmap e128: 12 total, 12 up, 12 in 2024-04-20T17:08:41.337 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:08:41.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:41.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:08:41.439 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 25 16 78027 78011 195.012 130.812 0.00354784 0.00512652 2024-04-20T17:08:41.441 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:08:41.486 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:41.611 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:08:41.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:41.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:08:41.617 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:08:41.664 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:08:41.664 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:08:41.717 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:08:41.760 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:41.849 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:08:41.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:41.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:08:41.956 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:08:41.993 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:42.057 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:08:42.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:42.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:08:42.162 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:08:42.199 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:42.225 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:08:42.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:42.230 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:08:42.331 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:08:42.377 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:42.409 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:08:42.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:42.414 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:08:42.440 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 26 16 80050 80034 192.374 126.438 0.0116428 0.00519649 2024-04-20T17:08:42.514 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:08:42.545 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:42.568 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:08:42.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:42.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:08:42.578 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-20T17:08:42.675 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:08:42.744 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:42.779 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:08:42.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:42.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:08:42.885 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:08:42.920 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:42.945 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:08:42.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:42.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:08:43.050 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:08:43.125 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:43.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:43 smithi106 ceph-mon[44423]: pgmap v396: 17 pgs: 17 active+clean; 4.5 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 138 MiB/s wr, 2.21k op/s 2024-04-20T17:08:43.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:08:43.324 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:08:43.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:43 smithi171 ceph-mon[44189]: pgmap v396: 17 pgs: 17 active+clean; 4.5 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 138 MiB/s wr, 2.21k op/s 2024-04-20T17:08:43.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:08:43.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:43 smithi005 ceph-mon[39562]: pgmap v396: 17 pgs: 17 active+clean; 4.5 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 138 MiB/s wr, 2.21k op/s 2024-04-20T17:08:43.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:08:43.440 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 27 16 81865 81849 189.45 113.438 0.00525518 0.005277 2024-04-20T17:08:43.448 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:08:43.497 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:08:43.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:43.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:08:43.532 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:08:43.603 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:08:43.672 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:43.817 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:08:43.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:43.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:08:43.930 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:08:43.966 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:43.995 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:08:43.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:43.999 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:08:44.100 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:08:44.130 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:44.153 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:08:44.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:44.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:08:44.256 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:08:44.294 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:44 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:08:44.295 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:44 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:08:44.295 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:44 smithi005 ceph-mon[39562]: osdmap e129: 12 total, 12 up, 12 in 2024-04-20T17:08:44.295 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:08:44.295 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:08:44.295 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:08:44.301 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:44.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:44 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:08:44.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:44 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:08:44.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:44 smithi171 ceph-mon[44189]: osdmap e129: 12 total, 12 up, 12 in 2024-04-20T17:08:44.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:08:44.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:08:44.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:08:44.440 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 28 16 83673 83657 186.719 113 0.00375194 0.00535426 2024-04-20T17:08:44.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:44 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:08:44.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:44 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:08:44.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:44 smithi106 ceph-mon[44423]: osdmap e129: 12 total, 12 up, 12 in 2024-04-20T17:08:44.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:08:44.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:08:44.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:08:44.668 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:08:44.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:44.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:08:44.774 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:08:44.807 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:44.828 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:08:44.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:44.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:08:44.932 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:08:44.974 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:45.002 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:08:45.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:45.007 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:08:45.107 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:08:45.147 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:45.173 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:08:45.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:45.178 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:08:45.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:45 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:08:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:45 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:08:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:45 smithi106 ceph-mon[44423]: pgmap v398: 17 pgs: 17 active+clean; 4.9 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 145 MiB/s wr, 2.31k op/s 2024-04-20T17:08:45.278 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:08:45.310 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:45.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:45 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:08:45.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:45 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:08:45.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:45 smithi171 ceph-mon[44189]: pgmap v398: 17 pgs: 17 active+clean; 4.9 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 145 MiB/s wr, 2.31k op/s 2024-04-20T17:08:45.334 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:08:45.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:45.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:08:45.400 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:45 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:08:45.400 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:45 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:08:45.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:45 smithi005 ceph-mon[39562]: pgmap v398: 17 pgs: 17 active+clean; 4.9 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 145 MiB/s wr, 2.31k op/s 2024-04-20T17:08:45.437 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:08:45.440 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 29 16 85700 85684 184.649 126.688 0.0086352 0.00541447 2024-04-20T17:08:45.481 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:45.738 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:08:45.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:45.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:08:45.842 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:08:45.882 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:45.906 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:08:45.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:45.910 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:08:46.011 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:08:46.057 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:46.087 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:08:46.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:46.091 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:08:46.192 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:08:46.236 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:46.275 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:08:46.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:46.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:08:46.330 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:08:46.385 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:08:46.415 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:46.437 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:08:46.440 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 30 16 87607 87591 182.466 119.188 0.0167302 0.00547871 2024-04-20T17:08:46.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:46.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:08:46.541 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:08:46.572 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:46.594 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:08:46.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:46.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:08:46.699 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:08:46.757 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:46.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:46 smithi005 ceph-mon[39562]: osdmap e130: 12 total, 12 up, 12 in 2024-04-20T17:08:46.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:46 smithi005 ceph-mon[39562]: pgmap v400: 17 pgs: 17 active+clean; 5.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 128 MiB/s wr, 2.04k op/s 2024-04-20T17:08:46.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:46 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:08:46.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:46 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:08:46.963 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:08:46.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:46.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:08:47.070 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:08:47.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:46 smithi171 ceph-mon[44189]: osdmap e130: 12 total, 12 up, 12 in 2024-04-20T17:08:47.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:46 smithi171 ceph-mon[44189]: pgmap v400: 17 pgs: 17 active+clean; 5.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 128 MiB/s wr, 2.04k op/s 2024-04-20T17:08:47.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:46 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:08:47.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:46 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:08:47.097 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:46 smithi106 ceph-mon[44423]: osdmap e130: 12 total, 12 up, 12 in 2024-04-20T17:08:47.097 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:46 smithi106 ceph-mon[44423]: pgmap v400: 17 pgs: 17 active+clean; 5.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 128 MiB/s wr, 2.04k op/s 2024-04-20T17:08:47.097 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:46 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:08:47.097 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:46 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:08:47.099 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:47.120 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:08:47.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:47.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:08:47.225 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:08:47.277 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:47.368 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:08:47.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:47.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:08:47.440 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 31 16 89526 89510 180.449 119.938 0.00391491 0.0055403 2024-04-20T17:08:47.475 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:08:47.555 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:47.859 INFO:teuthology.orchestra.run.smithi005.stderr:set 2.4 pg_upmap_items mapping to [6->7,1->5] 2024-04-20T17:08:48.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:47 smithi005 ceph-mon[39562]: osdmap e131: 12 total, 12 up, 12 in 2024-04-20T17:08:48.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:47 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:08:48.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:47 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:08:48.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:47 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3754182310' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.4", "id": [6, 7, 1, 5]}]: dispatch 2024-04-20T17:08:48.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:47 smithi106 ceph-mon[44423]: osdmap e131: 12 total, 12 up, 12 in 2024-04-20T17:08:48.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:47 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:08:48.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:47 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:08:48.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:47 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3754182310' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.4", "id": [6, 7, 1, 5]}]: dispatch 2024-04-20T17:08:48.326 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:08:48.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:48.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:08:48.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:47 smithi171 ceph-mon[44189]: osdmap e131: 12 total, 12 up, 12 in 2024-04-20T17:08:48.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:47 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:08:48.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:47 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:08:48.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:47 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3754182310' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.4", "id": [6, 7, 1, 5]}]: dispatch 2024-04-20T17:08:48.429 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:08:48.440 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 32 16 90611 90595 176.929 67.8125 0.00285903 0.00556398 2024-04-20T17:08:48.459 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:49.210 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:08:49.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:48 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3754182310' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.4", "id": [6, 7, 1, 5]}]': finished 2024-04-20T17:08:49.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:48 smithi005 ceph-mon[39562]: osdmap e132: 12 total, 12 up, 12 in 2024-04-20T17:08:49.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:48 smithi005 ceph-mon[39562]: pgmap v403: 17 pgs: 17 active+clean; 5.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 125 MiB/s wr, 1.99k op/s 2024-04-20T17:08:49.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:49.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:08:49.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:48 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3754182310' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.4", "id": [6, 7, 1, 5]}]': finished 2024-04-20T17:08:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:48 smithi106 ceph-mon[44423]: osdmap e132: 12 total, 12 up, 12 in 2024-04-20T17:08:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:48 smithi106 ceph-mon[44423]: pgmap v403: 17 pgs: 17 active+clean; 5.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 125 MiB/s wr, 1.99k op/s 2024-04-20T17:08:49.316 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:08:49.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:48 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3754182310' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.4", "id": [6, 7, 1, 5]}]': finished 2024-04-20T17:08:49.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:48 smithi171 ceph-mon[44189]: osdmap e132: 12 total, 12 up, 12 in 2024-04-20T17:08:49.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:48 smithi171 ceph-mon[44189]: pgmap v403: 17 pgs: 17 active+clean; 5.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 125 MiB/s wr, 1.99k op/s 2024-04-20T17:08:49.348 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:49.370 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:08:49.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:49.373 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:08:49.440 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 33 16 90611 90595 171.567 0 - 0.00556398 2024-04-20T17:08:49.474 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:08:49.507 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:49.528 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:08:49.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:49.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:08:49.634 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:08:49.665 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:49.686 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:08:49.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:49.689 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:08:49.790 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:08:49.819 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:49.839 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:08:49.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:49.842 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:08:49.943 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:08:49.973 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:50.048 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:08:50.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:50.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:08:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:49 smithi171 ceph-mon[44189]: osdmap e133: 12 total, 12 up, 12 in 2024-04-20T17:08:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:49 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/719936286' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:08:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:50.130 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:08:50.130 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:08:50.130 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:08:50.130 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:08:50.130 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:08:50.130 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:08:50.130 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:08:50.130 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:08:50.131 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:08:50.131 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:08:50.131 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:08:50.131 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:08:50.131 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:08:50.131 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:08:50.131 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:08:50.131 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:08:50.131 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:08:50.131 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:08:50.131 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:08:50.131 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:08:50.131 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:08:50.132 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:08:50.132 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:08:50.132 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:08:50.153 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:08:50.184 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:49 smithi106 ceph-mon[44423]: osdmap e133: 12 total, 12 up, 12 in 2024-04-20T17:08:50.184 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:49 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/719936286' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:08:50.184 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:50.186 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:50.208 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:08:50.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:49 smithi005 ceph-mon[39562]: osdmap e133: 12 total, 12 up, 12 in 2024-04-20T17:08:50.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:49 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/719936286' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:08:50.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:50.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:50.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:08:50.313 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:08:50.343 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:50.364 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:08:50.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:50.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:08:50.440 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 34 16 90611 90595 166.521 0 - 0.00556398 2024-04-20T17:08:50.468 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:08:50.496 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:50.520 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:08:50.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:50.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:08:50.624 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:08:50.654 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:50.684 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:08:50.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:50.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:08:50.766 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:08:50.766 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:08:50.766 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:08:50.766 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:08:50.766 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:08:50.789 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:08:50.818 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:50.837 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:08:50.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:50.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:08:50.876 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:08:50.907 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:50 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:08:50.907 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:50 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:08:50.907 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:50.907 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:50.907 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:50.907 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:50.908 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:50.908 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:50 smithi005 ceph-mon[39562]: osdmap e134: 12 total, 12 up, 12 in 2024-04-20T17:08:50.908 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:50.908 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:50.908 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:50.908 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:50 smithi005 ceph-mon[39562]: pgmap v406: 17 pgs: 1 remapped+peering, 16 active+clean; 5.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 118 MiB/s wr, 1.88k op/s 2024-04-20T17:08:50.908 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:50.908 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:50.908 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:50.908 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:50 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2462434037' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:08:50.908 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:50 smithi005 ceph-mon[39562]: from='client.24778 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:08:50.941 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:08:50.971 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:51.000 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:08:51.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:51.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:08:51.105 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:08:51.133 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:50 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:08:51.133 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:50 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:08:51.133 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:51.133 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:51.134 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:51.134 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:51.134 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:51.134 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:50 smithi106 ceph-mon[44423]: osdmap e134: 12 total, 12 up, 12 in 2024-04-20T17:08:51.134 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:51.135 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:51.135 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:51.135 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:50 smithi106 ceph-mon[44423]: pgmap v406: 17 pgs: 1 remapped+peering, 16 active+clean; 5.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 118 MiB/s wr, 1.88k op/s 2024-04-20T17:08:51.135 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:51.135 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:51.135 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:51.136 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:50 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2462434037' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:08:51.136 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:50 smithi106 ceph-mon[44423]: from='client.24778 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:08:51.136 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:51.158 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:08:51.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:51.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:08:51.263 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:08:51.326 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:50 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:08:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:50 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:08:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:50 smithi171 ceph-mon[44189]: osdmap e134: 12 total, 12 up, 12 in 2024-04-20T17:08:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:50 smithi171 ceph-mon[44189]: pgmap v406: 17 pgs: 1 remapped+peering, 16 active+clean; 5.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 118 MiB/s wr, 1.88k op/s 2024-04-20T17:08:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:50 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2462434037' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:08:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:50 smithi171 ceph-mon[44189]: from='client.24778 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:08:51.435 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:08:51.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:51.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:08:51.441 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 35 16 92016 92000 164.27 29.2708 0.00419416 0.00608614 2024-04-20T17:08:51.541 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:08:51.583 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:52.140 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:08:52.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:51 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:08:52.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:51 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:08:52.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:51 smithi005 ceph-mon[39562]: from='client.24778 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:08:52.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:51 smithi005 ceph-mon[39562]: osdmap e135: 12 total, 12 up, 12 in 2024-04-20T17:08:52.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:52.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:08:52.245 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:08:52.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:51 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:08:52.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:51 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:08:52.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:51 smithi106 ceph-mon[44423]: from='client.24778 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:08:52.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:51 smithi106 ceph-mon[44423]: osdmap e135: 12 total, 12 up, 12 in 2024-04-20T17:08:52.274 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:52.300 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:08:52.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:52.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:08:52.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:51 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:08:52.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:51 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:08:52.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:51 smithi171 ceph-mon[44189]: from='client.24778 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:08:52.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:51 smithi171 ceph-mon[44189]: osdmap e135: 12 total, 12 up, 12 in 2024-04-20T17:08:52.405 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:08:52.436 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:52.441 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 36 16 93793 93777 162.792 111.062 0.00786532 0.00614104 2024-04-20T17:08:52.458 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:08:52.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:52.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:08:52.563 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:08:52.602 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:52.644 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:08:52.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:52.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:08:52.699 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-20T17:08:52.760 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:08:52.799 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:52.826 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:08:52.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:52.830 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:08:52.878 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:52 smithi005 ceph-mon[39562]: pgmap v408: 17 pgs: 1 remapped+peering, 16 active+clean; 5.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-04-20T17:08:52.931 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:08:52.999 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:53.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:52 smithi171 ceph-mon[44189]: pgmap v408: 17 pgs: 1 remapped+peering, 16 active+clean; 5.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-04-20T17:08:53.224 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:08:53.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:52 smithi106 ceph-mon[44423]: pgmap v408: 17 pgs: 1 remapped+peering, 16 active+clean; 5.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-04-20T17:08:53.441 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 37 16 95460 95444 161.208 104.188 0.00315241 0.00620161 2024-04-20T17:08:53.565 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:08:53.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:53.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:08:53.623 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:08:53.664 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:08:53.679 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:08:53.714 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:53.742 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:08:53.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:53.745 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:08:53.846 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:08:53.884 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:53.955 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:08:53.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:53.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:08:54.062 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:08:54.093 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:54.120 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:08:54.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:54.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:08:54.226 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:08:54.257 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:54.283 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:08:54.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:54.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:08:54.389 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:08:54.419 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:54.441 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 38 16 97069 97053 159.612 100.562 0.017053 0.00626352 2024-04-20T17:08:54.442 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:08:54.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:54.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:08:54.546 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:08:54.577 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:54.601 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:08:54.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:54.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:08:54.707 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:08:54.741 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:54.762 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:08:54.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:54.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:08:54.866 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:08:54.909 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:55.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:54 smithi106 ceph-mon[44423]: pgmap v409: 17 pgs: 1 remapped+peering, 16 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-04-20T17:08:55.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:54 smithi005 ceph-mon[39562]: pgmap v409: 17 pgs: 1 remapped+peering, 16 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-04-20T17:08:55.284 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:08:55.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:55.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:08:55.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:54 smithi171 ceph-mon[44189]: pgmap v409: 17 pgs: 1 remapped+peering, 16 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-04-20T17:08:55.391 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:08:55.422 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:55.441 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 39 16 98777 98761 158.256 106.75 0.003326 0.00631733 2024-04-20T17:08:55.443 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:08:55.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:55.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:08:55.517 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:08:55.548 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:08:55.635 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:55.849 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:08:55.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:55.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:08:55.955 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:08:56.013 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:56.441 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:08:56.442606 min lat: 0.00106268 max lat: 2.76125 avg lat: 0.00637699 2024-04-20T17:08:56.441 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:08:56.441 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 40 16 100366 100350 156.782 99.3125 0.00839868 0.00637699 2024-04-20T17:08:56.574 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:08:56.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:56.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:08:56.679 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:08:56.731 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:56.909 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-20T17:08:56.909 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:08:56.909 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-20T17:08:56.910 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-20T17:08:56.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:56 smithi005 ceph-mon[39562]: osdmap e136: 12 total, 12 up, 12 in 2024-04-20T17:08:56.998 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:08:57.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:57.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:08:57.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:56 smithi171 ceph-mon[44189]: osdmap e136: 12 total, 12 up, 12 in 2024-04-20T17:08:57.106 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:08:57.106 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:08:57.106 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:08:57.106 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:08:57.106 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.106 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:48162 2.9 2:94e71f2d:::benchmark_data_smithi171_78918_object48161:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-20T17:08:57.106 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:28.755715+0000", 2024-04-20T17:08:57.106 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 28.327870761, 2024-04-20T17:08:57.106 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.064500852999999997, 2024-04-20T17:08:57.106 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:57.106 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:57.106 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:57.107 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:08:57.107 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:08:57.107 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:08:57.107 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 48162 2024-04-20T17:08:57.107 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.107 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:57.107 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.107 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:57.107 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.755715+0000", 2024-04-20T17:08:57.107 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.107 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.107 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.107 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:57.107 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.755715+0000", 2024-04-20T17:08:57.107 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.108 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.108 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.108 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:57.108 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.755721+0000", 2024-04-20T17:08:57.108 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4690000000000002e-06 2024-04-20T17:08:57.108 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.108 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.108 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:57.108 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.755796+0000", 2024-04-20T17:08:57.108 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4956000000000005e-05 2024-04-20T17:08:57.108 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.108 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.108 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:57.109 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.755807+0000", 2024-04-20T17:08:57.109 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0563e-05 2024-04-20T17:08:57.109 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.109 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.109 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:57.109 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.755814+0000", 2024-04-20T17:08:57.109 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3649999999999998e-06 2024-04-20T17:08:57.109 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.109 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.109 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:57.109 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.755859+0000", 2024-04-20T17:08:57.109 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4994999999999999e-05 2024-04-20T17:08:57.109 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.110 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.110 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:57.110 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.755926+0000", 2024-04-20T17:08:57.110 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6567000000000001e-05 2024-04-20T17:08:57.110 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.110 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.110 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 7", 2024-04-20T17:08:57.110 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.756003+0000", 2024-04-20T17:08:57.110 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7088000000000005e-05 2024-04-20T17:08:57.110 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.110 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.110 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:08:57.110 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.758522+0000", 2024-04-20T17:08:57.110 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025193339999999998 2024-04-20T17:08:57.111 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.111 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.111 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:08:57.111 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820108+0000", 2024-04-20T17:08:57.111 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.061586381000000003 2024-04-20T17:08:57.111 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.111 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.111 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:57.111 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820162+0000", 2024-04-20T17:08:57.111 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3091999999999999e-05 2024-04-20T17:08:57.111 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.111 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.111 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:57.111 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820216+0000", 2024-04-20T17:08:57.112 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4042999999999997e-05 2024-04-20T17:08:57.112 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.112 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:57.112 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.112 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.112 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.112 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:48163 2.9 2:9793a441:::benchmark_data_smithi171_78918_object48162:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-20T17:08:57.112 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:28.755829+0000", 2024-04-20T17:08:57.112 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 28.327757016, 2024-04-20T17:08:57.112 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.064495022999999999, 2024-04-20T17:08:57.112 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:57.112 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:57.112 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:57.112 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:08:57.113 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:08:57.113 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:08:57.113 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 48163 2024-04-20T17:08:57.113 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.113 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:57.113 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.113 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:57.113 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.755829+0000", 2024-04-20T17:08:57.113 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.113 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.113 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.113 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:57.113 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.755829+0000", 2024-04-20T17:08:57.113 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.114 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.114 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.114 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:57.114 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.755829+0000", 2024-04-20T17:08:57.114 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4799999999999997e-07 2024-04-20T17:08:57.114 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.114 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.114 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:57.114 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.756061+0000", 2024-04-20T17:08:57.114 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00023139499999999999 2024-04-20T17:08:57.114 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.114 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.114 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:57.115 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.756068+0000", 2024-04-20T17:08:57.115 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.571e-06 2024-04-20T17:08:57.115 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.115 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.115 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:57.115 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.756074+0000", 2024-04-20T17:08:57.115 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9649999999999996e-06 2024-04-20T17:08:57.115 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.115 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.115 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:57.115 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.757238+0000", 2024-04-20T17:08:57.116 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0011636050000000001 2024-04-20T17:08:57.116 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.116 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.116 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:57.116 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.757357+0000", 2024-04-20T17:08:57.116 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011934000000000001 2024-04-20T17:08:57.116 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.116 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.116 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 7", 2024-04-20T17:08:57.116 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.757478+0000", 2024-04-20T17:08:57.116 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000121012 2024-04-20T17:08:57.116 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.116 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.116 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:08:57.116 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820090+0000", 2024-04-20T17:08:57.117 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.062611539999999993 2024-04-20T17:08:57.117 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.117 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.117 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:08:57.117 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820224+0000", 2024-04-20T17:08:57.117 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013453 2024-04-20T17:08:57.117 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.117 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.117 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:57.117 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820269+0000", 2024-04-20T17:08:57.117 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4808999999999997e-05 2024-04-20T17:08:57.117 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.117 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.117 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:57.118 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820324+0000", 2024-04-20T17:08:57.118 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4508000000000001e-05 2024-04-20T17:08:57.118 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.118 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:57.118 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.118 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.118 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.118 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:48166 2.9 2:97d21563:::benchmark_data_smithi171_78918_object48165:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-20T17:08:57.118 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:28.756183+0000", 2024-04-20T17:08:57.118 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 28.327402755000001, 2024-04-20T17:08:57.118 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.064276050000000001, 2024-04-20T17:08:57.118 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:57.118 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:57.118 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:57.118 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:08:57.119 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:08:57.119 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:08:57.119 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 48166 2024-04-20T17:08:57.119 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.119 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:57.119 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.119 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:57.119 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.756183+0000", 2024-04-20T17:08:57.119 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.119 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.119 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.119 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:57.119 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.756183+0000", 2024-04-20T17:08:57.120 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.120 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.120 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.120 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:57.120 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.756184+0000", 2024-04-20T17:08:57.120 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.209e-06 2024-04-20T17:08:57.120 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.120 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.120 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:57.120 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.756316+0000", 2024-04-20T17:08:57.120 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013179399999999999 2024-04-20T17:08:57.120 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.120 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.120 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:57.121 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.756323+0000", 2024-04-20T17:08:57.121 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5059999999999998e-06 2024-04-20T17:08:57.121 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.121 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.121 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:57.121 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.756328+0000", 2024-04-20T17:08:57.121 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0810000000000001e-06 2024-04-20T17:08:57.121 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.121 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.121 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:57.121 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.758564+0000", 2024-04-20T17:08:57.121 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0022352660000000001 2024-04-20T17:08:57.121 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.122 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.122 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:57.122 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.758635+0000", 2024-04-20T17:08:57.122 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1095999999999998e-05 2024-04-20T17:08:57.122 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.122 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.122 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 7", 2024-04-20T17:08:57.122 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.758731+0000", 2024-04-20T17:08:57.122 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6529999999999999e-05 2024-04-20T17:08:57.122 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.122 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.122 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:08:57.122 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820369+0000", 2024-04-20T17:08:57.123 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.061637816999999998 2024-04-20T17:08:57.123 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.123 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.123 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:08:57.123 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820381+0000", 2024-04-20T17:08:57.123 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.168e-05 2024-04-20T17:08:57.123 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.123 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.123 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:57.123 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820423+0000", 2024-04-20T17:08:57.123 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2199e-05 2024-04-20T17:08:57.123 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.123 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.124 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:57.124 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820459+0000", 2024-04-20T17:08:57.124 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5871999999999998e-05 2024-04-20T17:08:57.124 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.124 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:57.124 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.124 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.124 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.124 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:48168 2.9 2:95f718a3:::benchmark_data_smithi171_78918_object48167:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-20T17:08:57.124 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:28.756922+0000", 2024-04-20T17:08:57.124 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 28.326663045, 2024-04-20T17:08:57.124 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.063662654999999999, 2024-04-20T17:08:57.124 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:57.124 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:57.125 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:57.125 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:08:57.125 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:08:57.125 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:08:57.125 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 48168 2024-04-20T17:08:57.125 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.125 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:57.125 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.125 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:57.125 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.756922+0000", 2024-04-20T17:08:57.125 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.125 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.125 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.125 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:57.126 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.756922+0000", 2024-04-20T17:08:57.126 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.126 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.126 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.126 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:57.126 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.756924+0000", 2024-04-20T17:08:57.126 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.511e-06 2024-04-20T17:08:57.126 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.126 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.126 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:57.126 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.757018+0000", 2024-04-20T17:08:57.126 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.3935000000000001e-05 2024-04-20T17:08:57.126 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.127 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.127 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:57.127 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.757025+0000", 2024-04-20T17:08:57.127 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.498e-06 2024-04-20T17:08:57.127 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.127 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.127 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:57.127 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.757030+0000", 2024-04-20T17:08:57.127 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.955e-06 2024-04-20T17:08:57.127 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.127 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.127 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:57.127 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.759678+0000", 2024-04-20T17:08:57.127 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0026472980000000002 2024-04-20T17:08:57.127 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.127 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.127 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:57.128 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.759787+0000", 2024-04-20T17:08:57.128 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000109635 2024-04-20T17:08:57.128 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.128 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.128 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 7", 2024-04-20T17:08:57.128 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.759911+0000", 2024-04-20T17:08:57.128 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012409699999999999 2024-04-20T17:08:57.128 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.128 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.128 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:08:57.128 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820466+0000", 2024-04-20T17:08:57.128 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.060554837 2024-04-20T17:08:57.128 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.128 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.128 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:08:57.128 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820507+0000", 2024-04-20T17:08:57.129 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0339999999999997e-05 2024-04-20T17:08:57.129 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.129 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.129 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:57.129 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820552+0000", 2024-04-20T17:08:57.129 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5184e-05 2024-04-20T17:08:57.129 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.129 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.129 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:57.129 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820585+0000", 2024-04-20T17:08:57.129 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3365000000000003e-05 2024-04-20T17:08:57.129 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.129 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:57.129 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.129 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.129 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.129 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(client.24736.0:48163 2.9 e124/115 ondisk, result = 0)", 2024-04-20T17:08:57.130 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:28.758608+0000", 2024-04-20T17:08:57.130 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 28.324977609000001, 2024-04-20T17:08:57.130 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.061718978000000001, 2024-04-20T17:08:57.130 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:57.130 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:57.130 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:08:57.130 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:57.130 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.130 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:57.130 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.758608+0000", 2024-04-20T17:08:57.130 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.130 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.130 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.130 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:57.130 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.758608+0000", 2024-04-20T17:08:57.130 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.131 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.131 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.131 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:57.131 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.758609+0000", 2024-04-20T17:08:57.131 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3510000000000001e-06 2024-04-20T17:08:57.131 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.131 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.131 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:57.131 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.758614+0000", 2024-04-20T17:08:57.131 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4120000000000003e-06 2024-04-20T17:08:57.131 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.131 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.131 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:57.131 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.758622+0000", 2024-04-20T17:08:57.131 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7390000000000005e-06 2024-04-20T17:08:57.131 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.131 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.132 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:57.132 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.758629+0000", 2024-04-20T17:08:57.132 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.776e-06 2024-04-20T17:08:57.132 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.132 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.132 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:57.132 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820073+0000", 2024-04-20T17:08:57.132 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.061444004000000003 2024-04-20T17:08:57.132 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.132 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.132 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:57.132 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820083+0000", 2024-04-20T17:08:57.132 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.5710000000000004e-06 2024-04-20T17:08:57.132 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.132 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.132 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:57.132 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820327+0000", 2024-04-20T17:08:57.133 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00024412500000000001 2024-04-20T17:08:57.133 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.133 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:57.133 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.133 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.133 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.133 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(client.24736.0:48166 2.9 e124/115 ondisk, result = 0)", 2024-04-20T17:08:57.133 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:28.759854+0000", 2024-04-20T17:08:57.133 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 28.323731816999999, 2024-04-20T17:08:57.133 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.060613538000000002, 2024-04-20T17:08:57.133 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:57.133 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:57.133 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:08:57.133 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:57.133 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.133 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:57.134 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.759854+0000", 2024-04-20T17:08:57.134 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.134 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.134 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.134 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:57.134 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.759854+0000", 2024-04-20T17:08:57.134 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.134 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.134 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.134 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:57.134 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.759855+0000", 2024-04-20T17:08:57.134 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.187e-06 2024-04-20T17:08:57.134 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.135 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.135 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:57.135 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.759859+0000", 2024-04-20T17:08:57.135 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5050000000000001e-06 2024-04-20T17:08:57.135 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.135 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.135 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:57.135 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.759869+0000", 2024-04-20T17:08:57.135 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2119999999999992e-06 2024-04-20T17:08:57.135 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.135 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.135 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:57.135 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.759875+0000", 2024-04-20T17:08:57.136 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6579999999999996e-06 2024-04-20T17:08:57.136 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.136 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.136 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:57.136 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820356+0000", 2024-04-20T17:08:57.136 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.060480994000000003 2024-04-20T17:08:57.136 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.136 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.136 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:57.136 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820363+0000", 2024-04-20T17:08:57.136 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0029999999999997e-06 2024-04-20T17:08:57.136 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.136 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.136 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:57.136 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820467+0000", 2024-04-20T17:08:57.137 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000103979 2024-04-20T17:08:57.137 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.137 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:57.137 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.137 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.137 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.137 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:48180 2.6 2:63803d39:::benchmark_data_smithi171_78918_object48179:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-20T17:08:57.137 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:28.760596+0000", 2024-04-20T17:08:57.137 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 28.322989519, 2024-04-20T17:08:57.137 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.060120881000000001, 2024-04-20T17:08:57.137 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:57.137 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:57.137 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:57.137 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:08:57.137 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:08:57.138 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:08:57.138 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 48180 2024-04-20T17:08:57.138 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.138 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:57.138 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.138 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:57.138 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.760596+0000", 2024-04-20T17:08:57.138 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.138 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.138 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.138 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:57.138 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.760596+0000", 2024-04-20T17:08:57.138 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.138 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.138 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.139 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:57.139 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.760598+0000", 2024-04-20T17:08:57.139 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.615e-06 2024-04-20T17:08:57.139 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.139 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.139 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:57.139 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.760704+0000", 2024-04-20T17:08:57.139 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000106251 2024-04-20T17:08:57.139 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.139 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.139 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:57.139 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.760714+0000", 2024-04-20T17:08:57.139 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0131e-05 2024-04-20T17:08:57.139 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.140 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.140 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:57.140 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.760722+0000", 2024-04-20T17:08:57.140 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.055e-06 2024-04-20T17:08:57.140 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.140 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.140 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:57.140 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.760777+0000", 2024-04-20T17:08:57.140 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4446e-05 2024-04-20T17:08:57.140 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.140 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.140 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:57.140 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.760864+0000", 2024-04-20T17:08:57.140 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7744999999999994e-05 2024-04-20T17:08:57.140 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.140 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.141 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:08:57.141 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.760977+0000", 2024-04-20T17:08:57.141 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000112873 2024-04-20T17:08:57.141 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.141 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.141 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:08:57.141 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820632+0000", 2024-04-20T17:08:57.141 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.059654730000000003 2024-04-20T17:08:57.141 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.141 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.141 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:08:57.141 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820643+0000", 2024-04-20T17:08:57.141 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1260000000000001e-05 2024-04-20T17:08:57.141 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.141 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.141 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:57.142 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820683+0000", 2024-04-20T17:08:57.142 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0123999999999997e-05 2024-04-20T17:08:57.142 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.142 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.142 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:57.142 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820717+0000", 2024-04-20T17:08:57.142 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3651e-05 2024-04-20T17:08:57.142 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.142 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:57.142 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.142 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.142 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.142 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(client.24736.0:48168 2.9 e124/115 ondisk, result = 0)", 2024-04-20T17:08:57.142 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:28.760989+0000", 2024-04-20T17:08:57.143 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 28.322596563000001, 2024-04-20T17:08:57.143 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.059602068000000001, 2024-04-20T17:08:57.143 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:57.143 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:57.143 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:08:57.143 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:57.143 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.143 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:57.143 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.760989+0000", 2024-04-20T17:08:57.143 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.143 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.143 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.143 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:57.143 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.760989+0000", 2024-04-20T17:08:57.143 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.144 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.144 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.144 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:57.144 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.760990+0000", 2024-04-20T17:08:57.144 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.097e-06 2024-04-20T17:08:57.144 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.144 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.144 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:57.144 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.760994+0000", 2024-04-20T17:08:57.144 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2740000000000001e-06 2024-04-20T17:08:57.144 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.144 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.144 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:57.144 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.761002+0000", 2024-04-20T17:08:57.144 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5299999999999999e-06 2024-04-20T17:08:57.144 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.145 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.145 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:57.145 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.761008+0000", 2024-04-20T17:08:57.145 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1090000000000003e-06 2024-04-20T17:08:57.145 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.145 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.145 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:57.145 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820495+0000", 2024-04-20T17:08:57.145 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.059486867999999998 2024-04-20T17:08:57.145 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.145 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.145 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:57.145 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820501+0000", 2024-04-20T17:08:57.145 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.567e-06 2024-04-20T17:08:57.146 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.146 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.146 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:57.146 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820591+0000", 2024-04-20T17:08:57.146 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9623000000000006e-05 2024-04-20T17:08:57.146 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.146 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:57.146 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.146 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.146 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.146 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:48186 2.9 2:9cebb0f3:::benchmark_data_smithi171_78918_object48185:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-20T17:08:57.146 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:28.762429+0000", 2024-04-20T17:08:57.146 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 28.321157030999998, 2024-04-20T17:08:57.146 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.062396351000000003, 2024-04-20T17:08:57.146 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:57.147 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:57.147 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:57.147 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:08:57.147 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:08:57.147 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:08:57.147 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 48186 2024-04-20T17:08:57.147 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.147 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:57.147 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.147 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:57.147 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.762429+0000", 2024-04-20T17:08:57.147 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.147 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.147 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.147 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:57.147 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.762429+0000", 2024-04-20T17:08:57.147 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.148 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.148 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.148 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:57.148 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.762430+0000", 2024-04-20T17:08:57.148 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.539e-06 2024-04-20T17:08:57.148 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.148 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.148 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:57.148 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.762483+0000", 2024-04-20T17:08:57.148 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2630999999999998e-05 2024-04-20T17:08:57.148 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.148 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.148 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:57.148 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820889+0000", 2024-04-20T17:08:57.148 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.058406288000000001 2024-04-20T17:08:57.148 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.149 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.149 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:57.149 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820901+0000", 2024-04-20T17:08:57.149 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2278e-05 2024-04-20T17:08:57.149 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.149 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.149 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:57.149 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.821336+0000", 2024-04-20T17:08:57.149 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00043430299999999998 2024-04-20T17:08:57.149 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.149 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.149 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:57.149 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.821454+0000", 2024-04-20T17:08:57.149 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000118435 2024-04-20T17:08:57.149 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.149 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.149 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 7", 2024-04-20T17:08:57.150 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.821592+0000", 2024-04-20T17:08:57.150 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013768999999999999 2024-04-20T17:08:57.150 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.150 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.150 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:08:57.150 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.824703+0000", 2024-04-20T17:08:57.150 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0031112179999999998 2024-04-20T17:08:57.150 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.150 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.150 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:08:57.150 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.824725+0000", 2024-04-20T17:08:57.150 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2330000000000001e-05 2024-04-20T17:08:57.150 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.150 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.151 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:57.151 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.824777+0000", 2024-04-20T17:08:57.151 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1888999999999997e-05 2024-04-20T17:08:57.151 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.151 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.151 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:57.151 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.824825+0000", 2024-04-20T17:08:57.151 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7750000000000002e-05 2024-04-20T17:08:57.151 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.151 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:57.151 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.151 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.151 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.152 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:48188 2.2 e124/115 2:48644282:::benchmark_data_smithi171_78918_object48187:head v 124'3088, mlcod=124'3086)", 2024-04-20T17:08:57.152 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:28.763277+0000", 2024-04-20T17:08:57.152 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 28.320308666999999, 2024-04-20T17:08:57.152 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.060238704999999997, 2024-04-20T17:08:57.152 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:57.152 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:57.152 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:57.152 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:57.152 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.152 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:57.152 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.763277+0000", 2024-04-20T17:08:57.152 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.153 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.153 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.153 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:57.153 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.763277+0000", 2024-04-20T17:08:57.153 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.153 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.153 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.153 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:57.153 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.763288+0000", 2024-04-20T17:08:57.153 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.153e-05 2024-04-20T17:08:57.153 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.153 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.153 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:57.153 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.763365+0000", 2024-04-20T17:08:57.154 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6246000000000001e-05 2024-04-20T17:08:57.154 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.154 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.154 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:57.154 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820062+0000", 2024-04-20T17:08:57.154 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.056697522 2024-04-20T17:08:57.154 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.154 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.154 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:57.154 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820074+0000", 2024-04-20T17:08:57.154 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1841999999999999e-05 2024-04-20T17:08:57.154 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.154 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820268+0000", 2024-04-20T17:08:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00019409700000000001 2024-04-20T17:08:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.820298+0000", 2024-04-20T17:08:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0137e-05 2024-04-20T17:08:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.823481+0000", 2024-04-20T17:08:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0031827719999999999 2024-04-20T17:08:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:28.823516+0000", 2024-04-20T17:08:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4558999999999999e-05 2024-04-20T17:08:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:50979 2.4 2:24d96ae0:::benchmark_data_smithi171_78918_object50978:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-20T17:08:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:29.776090+0000", 2024-04-20T17:08:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 27.307495230000001, 2024-04-20T17:08:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.071982978000000003, 2024-04-20T17:08:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:08:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:08:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:08:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 50979 2024-04-20T17:08:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776090+0000", 2024-04-20T17:08:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776090+0000", 2024-04-20T17:08:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776092+0000", 2024-04-20T17:08:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5290000000000001e-06 2024-04-20T17:08:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776151+0000", 2024-04-20T17:08:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9178000000000002e-05 2024-04-20T17:08:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776159+0000", 2024-04-20T17:08:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4549999999999995e-06 2024-04-20T17:08:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776165+0000", 2024-04-20T17:08:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2279999999999998e-06 2024-04-20T17:08:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776200+0000", 2024-04-20T17:08:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5509000000000002e-05 2024-04-20T17:08:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776274+0000", 2024-04-20T17:08:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4029000000000006e-05 2024-04-20T17:08:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 0", 2024-04-20T17:08:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776356+0000", 2024-04-20T17:08:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2262000000000006e-05 2024-04-20T17:08:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:08:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.847215+0000", 2024-04-20T17:08:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.070858043999999995 2024-04-20T17:08:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:08:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.847952+0000", 2024-04-20T17:08:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00073713000000000001 2024-04-20T17:08:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.848025+0000", 2024-04-20T17:08:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3139999999999994e-05 2024-04-20T17:08:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.848073+0000", 2024-04-20T17:08:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8473999999999997e-05 2024-04-20T17:08:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:50980 2.4 2:240bdb56:::benchmark_data_smithi171_78918_object50979:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-20T17:08:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:29.776175+0000", 2024-04-20T17:08:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 27.307410469000001, 2024-04-20T17:08:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.072009840000000006, 2024-04-20T17:08:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:08:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:08:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:08:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 50980 2024-04-20T17:08:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776175+0000", 2024-04-20T17:08:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776175+0000", 2024-04-20T17:08:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776176+0000", 2024-04-20T17:08:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9699999999999995e-07 2024-04-20T17:08:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776250+0000", 2024-04-20T17:08:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4370999999999997e-05 2024-04-20T17:08:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776257+0000", 2024-04-20T17:08:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5699999999999998e-06 2024-04-20T17:08:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776261+0000", 2024-04-20T17:08:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5279999999999997e-06 2024-04-20T17:08:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.777860+0000", 2024-04-20T17:08:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00159895 2024-04-20T17:08:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.777959+0000", 2024-04-20T17:08:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9122000000000004e-05 2024-04-20T17:08:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 0", 2024-04-20T17:08:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.778096+0000", 2024-04-20T17:08:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013646599999999999 2024-04-20T17:08:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:08:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.847474+0000", 2024-04-20T17:08:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.069378596000000001 2024-04-20T17:08:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:08:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.848119+0000", 2024-04-20T17:08:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00064424700000000003 2024-04-20T17:08:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.848149+0000", 2024-04-20T17:08:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0001e-05 2024-04-20T17:08:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.848185+0000", 2024-04-20T17:08:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6192000000000002e-05 2024-04-20T17:08:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:50987 2.9 2:9ca6e68f:::benchmark_data_smithi171_78918_object50986:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-20T17:08:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:29.776397+0000", 2024-04-20T17:08:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 27.307188256, 2024-04-20T17:08:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.072799267000000001, 2024-04-20T17:08:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:08:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:08:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:08:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 50987 2024-04-20T17:08:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776397+0000", 2024-04-20T17:08:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776397+0000", 2024-04-20T17:08:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776398+0000", 2024-04-20T17:08:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.108e-06 2024-04-20T17:08:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776523+0000", 2024-04-20T17:08:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000124771 2024-04-20T17:08:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776533+0000", 2024-04-20T17:08:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0101e-05 2024-04-20T17:08:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776541+0000", 2024-04-20T17:08:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3359999999999997e-06 2024-04-20T17:08:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776627+0000", 2024-04-20T17:08:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6254999999999993e-05 2024-04-20T17:08:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776714+0000", 2024-04-20T17:08:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7336000000000005e-05 2024-04-20T17:08:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 7", 2024-04-20T17:08:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776791+0000", 2024-04-20T17:08:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7235000000000002e-05 2024-04-20T17:08:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:08:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.779393+0000", 2024-04-20T17:08:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0026010790000000001 2024-04-20T17:08:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:08:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.849100+0000", 2024-04-20T17:08:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.069707993999999995 2024-04-20T17:08:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.849152+0000", 2024-04-20T17:08:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1604000000000003e-05 2024-04-20T17:08:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.849197+0000", 2024-04-20T17:08:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4447999999999999e-05 2024-04-20T17:08:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:50989 2.9 2:918d01f4:::benchmark_data_smithi171_78918_object50988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-20T17:08:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:29.776557+0000", 2024-04-20T17:08:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 27.307028181, 2024-04-20T17:08:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.072786545999999994, 2024-04-20T17:08:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:08:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:08:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:08:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 50989 2024-04-20T17:08:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776557+0000", 2024-04-20T17:08:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.174 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.174 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.174 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:57.174 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776557+0000", 2024-04-20T17:08:57.174 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.174 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.174 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.174 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:57.174 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776558+0000", 2024-04-20T17:08:57.174 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.031e-06 2024-04-20T17:08:57.174 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.174 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.174 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776627+0000", 2024-04-20T17:08:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8998000000000006e-05 2024-04-20T17:08:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776635+0000", 2024-04-20T17:08:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5259999999999996e-06 2024-04-20T17:08:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.776641+0000", 2024-04-20T17:08:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.037e-06 2024-04-20T17:08:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.778059+0000", 2024-04-20T17:08:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001418189 2024-04-20T17:08:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.778151+0000", 2024-04-20T17:08:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1899999999999998e-05 2024-04-20T17:08:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 7", 2024-04-20T17:08:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.778286+0000", 2024-04-20T17:08:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000135158 2024-04-20T17:08:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:08:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.847248+0000", 2024-04-20T17:08:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.068961795000000006 2024-04-20T17:08:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:08:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.849246+0000", 2024-04-20T17:08:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0019982149999999998 2024-04-20T17:08:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.849307+0000", 2024-04-20T17:08:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.122e-05 2024-04-20T17:08:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.849344+0000", 2024-04-20T17:08:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6477000000000003e-05 2024-04-20T17:08:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(client.24736.0:50987 2.9 e124/115 ondisk, result = 0)", 2024-04-20T17:08:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:29.779658+0000", 2024-04-20T17:08:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 27.303927267999999, 2024-04-20T17:08:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.069548448999999998, 2024-04-20T17:08:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:08:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.779658+0000", 2024-04-20T17:08:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.779658+0000", 2024-04-20T17:08:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.779660+0000", 2024-04-20T17:08:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6729999999999999e-06 2024-04-20T17:08:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.779668+0000", 2024-04-20T17:08:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7400000000000004e-06 2024-04-20T17:08:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.847187+0000", 2024-04-20T17:08:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.067519079999999995 2024-04-20T17:08:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.847197+0000", 2024-04-20T17:08:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0715e-05 2024-04-20T17:08:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.849088+0000", 2024-04-20T17:08:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001890028 2024-04-20T17:08:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.849095+0000", 2024-04-20T17:08:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2960000000000001e-06 2024-04-20T17:08:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.849207+0000", 2024-04-20T17:08:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011191700000000001 2024-04-20T17:08:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(client.24736.0:50979 2.4 e124/115 ondisk, result = 0)", 2024-04-20T17:08:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:29.779841+0000", 2024-04-20T17:08:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 27.303744232, 2024-04-20T17:08:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.068240882000000003, 2024-04-20T17:08:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:08:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.779841+0000", 2024-04-20T17:08:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.779841+0000", 2024-04-20T17:08:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.779842+0000", 2024-04-20T17:08:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.113e-06 2024-04-20T17:08:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.779846+0000", 2024-04-20T17:08:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0289999999999999e-06 2024-04-20T17:08:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.847251+0000", 2024-04-20T17:08:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.067404927000000003 2024-04-20T17:08:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:57.184 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.847262+0000", 2024-04-20T17:08:57.184 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0419e-05 2024-04-20T17:08:57.184 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.184 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.184 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:57.184 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.847931+0000", 2024-04-20T17:08:57.184 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00066889100000000002 2024-04-20T17:08:57.184 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.184 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.184 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:57.184 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.847946+0000", 2024-04-20T17:08:57.184 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4856e-05 2024-04-20T17:08:57.184 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.184 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.185 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:57.185 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.848082+0000", 2024-04-20T17:08:57.185 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013664699999999999 2024-04-20T17:08:57.185 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.185 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:57.185 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.185 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.185 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.185 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:50999 2.9 2:986026df:::benchmark_data_smithi171_78918_object50998:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-20T17:08:57.185 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:29.781634+0000", 2024-04-20T17:08:57.185 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 27.301951335999998, 2024-04-20T17:08:57.185 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.071084708999999996, 2024-04-20T17:08:57.185 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:57.185 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:57.185 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:57.185 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:08:57.185 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:08:57.186 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:08:57.186 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 50999 2024-04-20T17:08:57.186 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.186 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:57.186 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.186 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:57.186 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.781634+0000", 2024-04-20T17:08:57.186 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.186 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.186 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.186 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:57.186 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.781634+0000", 2024-04-20T17:08:57.186 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.186 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.186 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.187 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:57.187 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.781636+0000", 2024-04-20T17:08:57.187 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8589999999999999e-06 2024-04-20T17:08:57.187 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.187 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.187 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:57.187 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.781716+0000", 2024-04-20T17:08:57.187 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0239000000000002e-05 2024-04-20T17:08:57.187 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.187 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.187 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:57.187 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.847177+0000", 2024-04-20T17:08:57.187 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.065460687000000004 2024-04-20T17:08:57.187 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.187 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.187 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:57.187 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.847194+0000", 2024-04-20T17:08:57.188 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7136e-05 2024-04-20T17:08:57.188 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.188 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.188 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:57.188 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.847291+0000", 2024-04-20T17:08:57.188 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7163999999999993e-05 2024-04-20T17:08:57.188 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.188 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.188 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:57.188 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.847438+0000", 2024-04-20T17:08:57.188 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000146713 2024-04-20T17:08:57.188 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.188 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.188 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 7", 2024-04-20T17:08:57.188 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.847596+0000", 2024-04-20T17:08:57.188 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015758600000000001 2024-04-20T17:08:57.189 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.189 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.189 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:08:57.189 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.852638+0000", 2024-04-20T17:08:57.189 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0050424939999999998 2024-04-20T17:08:57.189 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.189 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.189 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:08:57.189 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.852649+0000", 2024-04-20T17:08:57.189 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.11e-05 2024-04-20T17:08:57.189 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.189 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.189 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:57.190 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.852683+0000", 2024-04-20T17:08:57.190 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4180999999999997e-05 2024-04-20T17:08:57.190 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.190 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.190 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:57.190 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.852719+0000", 2024-04-20T17:08:57.190 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5549999999999997e-05 2024-04-20T17:08:57.190 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.190 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:57.190 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.190 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.190 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.190 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:51423 2.6 2:60993269:::benchmark_data_smithi171_78918_object51422:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-20T17:08:57.190 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:29.985973+0000", 2024-04-20T17:08:57.190 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 27.097613017, 2024-04-20T17:08:57.191 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.059632194999999999, 2024-04-20T17:08:57.191 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:57.191 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:57.191 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:57.191 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:08:57.191 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:08:57.191 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:08:57.191 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 51423 2024-04-20T17:08:57.191 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.191 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:57.191 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.191 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:57.191 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.985973+0000", 2024-04-20T17:08:57.192 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.192 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.192 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.192 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:57.192 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.985973+0000", 2024-04-20T17:08:57.192 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.192 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.192 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.192 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:57.192 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.985974+0000", 2024-04-20T17:08:57.192 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3090000000000001e-06 2024-04-20T17:08:57.192 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.192 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.192 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:57.192 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.986035+0000", 2024-04-20T17:08:57.193 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0922999999999999e-05 2024-04-20T17:08:57.193 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.193 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.193 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:57.193 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.986044+0000", 2024-04-20T17:08:57.193 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1370000000000008e-06 2024-04-20T17:08:57.193 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.193 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.193 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:57.194 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.986049+0000", 2024-04-20T17:08:57.194 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1100000000000002e-06 2024-04-20T17:08:57.194 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.194 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.194 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:57.194 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.986086+0000", 2024-04-20T17:08:57.194 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6504000000000002e-05 2024-04-20T17:08:57.194 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.195 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.195 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:57.195 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.986154+0000", 2024-04-20T17:08:57.195 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8616999999999997e-05 2024-04-20T17:08:57.195 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.195 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.195 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:08:57.195 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.986285+0000", 2024-04-20T17:08:57.195 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013076400000000001 2024-04-20T17:08:57.195 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.195 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.195 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:08:57.195 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:30.040552+0000", 2024-04-20T17:08:57.195 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.054267259999999998 2024-04-20T17:08:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:08:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:30.045528+0000", 2024-04-20T17:08:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0049759490000000003 2024-04-20T17:08:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:30.045558+0000", 2024-04-20T17:08:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9745000000000001e-05 2024-04-20T17:08:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:30.045605+0000", 2024-04-20T17:08:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6876999999999999e-05 2024-04-20T17:08:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:51424 2.9 2:9a965e29:::benchmark_data_smithi171_78918_object51423:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-20T17:08:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:29.986180+0000", 2024-04-20T17:08:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 27.097405147, 2024-04-20T17:08:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.058400366000000002, 2024-04-20T17:08:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:08:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:08:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:08:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 51424 2024-04-20T17:08:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.986180+0000", 2024-04-20T17:08:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.986180+0000", 2024-04-20T17:08:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.986193+0000", 2024-04-20T17:08:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2566e-05 2024-04-20T17:08:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.986246+0000", 2024-04-20T17:08:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2675e-05 2024-04-20T17:08:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.986254+0000", 2024-04-20T17:08:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4060000000000005e-06 2024-04-20T17:08:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.986261+0000", 2024-04-20T17:08:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2969999999999998e-06 2024-04-20T17:08:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.986311+0000", 2024-04-20T17:08:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9526e-05 2024-04-20T17:08:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:30.038989+0000", 2024-04-20T17:08:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.052678142999999997 2024-04-20T17:08:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 7", 2024-04-20T17:08:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:30.039146+0000", 2024-04-20T17:08:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015684 2024-04-20T17:08:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:08:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:30.044436+0000", 2024-04-20T17:08:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0052902890000000001 2024-04-20T17:08:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:08:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:30.044477+0000", 2024-04-20T17:08:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1208000000000003e-05 2024-04-20T17:08:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:30.044531+0000", 2024-04-20T17:08:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3616999999999998e-05 2024-04-20T17:08:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:30.044581+0000", 2024-04-20T17:08:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9799000000000002e-05 2024-04-20T17:08:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:51428 2.4 2:2f883a9e:::benchmark_data_smithi171_78918_object51427:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-20T17:08:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:08:29.986463+0000", 2024-04-20T17:08:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 27.097122242000001, 2024-04-20T17:08:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.060414550999999997, 2024-04-20T17:08:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:08:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:08:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:08:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:08:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:08:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:08:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 51428 2024-04-20T17:08:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:08:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:08:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.986463+0000", 2024-04-20T17:08:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:08:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.986463+0000", 2024-04-20T17:08:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:08:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:08:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.986464+0000", 2024-04-20T17:08:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6899999999999997e-07 2024-04-20T17:08:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:08:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:29.986551+0000", 2024-04-20T17:08:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7026000000000003e-05 2024-04-20T17:08:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:08:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:30.038751+0000", 2024-04-20T17:08:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.052200152 2024-04-20T17:08:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:08:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:30.038761+0000", 2024-04-20T17:08:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0329999999999999e-05 2024-04-20T17:08:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:08:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:30.038832+0000", 2024-04-20T17:08:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0436000000000001e-05 2024-04-20T17:08:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:08:57.206 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:30.038985+0000", 2024-04-20T17:08:57.206 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015311 2024-04-20T17:08:57.206 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.206 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.206 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 0", 2024-04-20T17:08:57.206 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:30.039140+0000", 2024-04-20T17:08:57.206 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015466500000000001 2024-04-20T17:08:57.206 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.206 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.206 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:08:57.206 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:30.043921+0000", 2024-04-20T17:08:57.206 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0047808299999999998 2024-04-20T17:08:57.206 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:08:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:30.046786+0000", 2024-04-20T17:08:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0028651750000000002 2024-04-20T17:08:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:08:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:30.046836+0000", 2024-04-20T17:08:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0615999999999998e-05 2024-04-20T17:08:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:08:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:08:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:08:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:08:30.046878+0000", 2024-04-20T17:08:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1542000000000003e-05 2024-04-20T17:08:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.208 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:57.208 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.208 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:08:57.208 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:08:57.208 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:08:57.208 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:08:57.239 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:57.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:56 smithi106 ceph-mon[44423]: osdmap e136: 12 total, 12 up, 12 in 2024-04-20T17:08:57.265 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:08:57.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:57.268 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:08:57.369 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:08:57.419 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:57.441 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 41 16 101849 101833 155.219 92.6875 0.00864515 0.00644077 2024-04-20T17:08:57.473 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:08:57.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:57.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:08:57.479 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:08:57.578 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:08:57.616 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:57.616 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:08:57.616 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:08:57.616 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:08:57.617 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:08:57.617 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:08:57.622 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:08:57.647 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:08:57.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:57.652 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:08:57.658 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:08:57.698 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:08:57.735 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:08:57.753 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:08:57.761 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:08:57.783 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:57.787 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:08:57.805 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:08:57.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:57.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:08:57.821 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:08:57.832 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:08:57.832 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:08:57.832 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:08:57.832 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:08:57.833 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:08:57.833 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:08:57.833 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:08:57.833 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:08:57.833 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:08:57.833 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:08:57.833 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:08:57.833 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:08:57.909 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:08:57.936 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:57.959 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:08:57.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:57.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:08:58.063 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:08:58.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:57 smithi171 ceph-mon[44189]: pgmap v411: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.05k op/s; 5476/191016 objects misplaced (2.867%); 1.5 MiB/s, 23 objects/s recovering 2024-04-20T17:08:58.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:57 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1426949479' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:08:58.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:58.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:58.089 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:57 smithi106 ceph-mon[44423]: pgmap v411: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.05k op/s; 5476/191016 objects misplaced (2.867%); 1.5 MiB/s, 23 objects/s recovering 2024-04-20T17:08:58.089 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:57 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1426949479' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:08:58.090 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:58.090 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:58.092 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:58.114 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:08:58.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:08:58.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:08:58.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:57 smithi005 ceph-mon[39562]: pgmap v411: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 5.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.05k op/s; 5476/191016 objects misplaced (2.867%); 1.5 MiB/s, 23 objects/s recovering 2024-04-20T17:08:58.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:57 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1426949479' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:08:58.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:58.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:58.220 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:08:58.261 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:08:58.441 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 42 16 103344 103328 153.748 93.4375 0.0143282 0.00650252 2024-04-20T17:08:59.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:08:58 smithi171 ceph-mon[44189]: pgmap v412: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s; 5476/198114 objects misplaced (2.764%); 1.1 MiB/s, 17 objects/s recovering 2024-04-20T17:08:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:08:58 smithi005 ceph-mon[39562]: pgmap v412: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s; 5476/198114 objects misplaced (2.764%); 1.1 MiB/s, 17 objects/s recovering 2024-04-20T17:08:59.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:08:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:08:58 smithi106 ceph-mon[44423]: pgmap v412: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s; 5476/198114 objects misplaced (2.764%); 1.1 MiB/s, 17 objects/s recovering 2024-04-20T17:08:59.441 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 43 16 104995 104979 152.572 103.188 0.00502665 0.00655269 2024-04-20T17:09:00.441 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 44 16 106638 106622 151.438 102.688 0.00955321 0.0066019 2024-04-20T17:09:01.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:00 smithi106 ceph-mon[44423]: pgmap v413: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s; 5289/204070 objects misplaced (2.592%); 2.3 MiB/s, 36 objects/s recovering 2024-04-20T17:09:01.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:00 smithi171 ceph-mon[44189]: pgmap v413: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s; 5289/204070 objects misplaced (2.592%); 2.3 MiB/s, 36 objects/s recovering 2024-04-20T17:09:01.442 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 45 16 108241 108225 150.299 100.188 0.00384372 0.00665162 2024-04-20T17:09:01.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:00 smithi005 ceph-mon[39562]: pgmap v413: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s; 5289/204070 objects misplaced (2.592%); 2.3 MiB/s, 36 objects/s recovering 2024-04-20T17:09:02.442 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 46 15 109805 109790 149.158 97.8125 0.00366569 0.00670223 2024-04-20T17:09:03.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:02 smithi106 ceph-mon[44423]: pgmap v414: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s; 5289/208994 objects misplaced (2.531%); 2.1 MiB/s, 32 objects/s recovering 2024-04-20T17:09:03.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:02 smithi171 ceph-mon[44189]: pgmap v414: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s; 5289/208994 objects misplaced (2.531%); 2.1 MiB/s, 32 objects/s recovering 2024-04-20T17:09:03.442 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 47 16 111430 111414 148.143 101.5 0.0259391 0.00674866 2024-04-20T17:09:03.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:02 smithi005 ceph-mon[39562]: pgmap v414: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s; 5289/208994 objects misplaced (2.531%); 2.1 MiB/s, 32 objects/s recovering 2024-04-20T17:09:04.442 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 48 16 112976 112960 147.07 96.625 0.00758272 0.00679797 2024-04-20T17:09:05.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:04 smithi005 ceph-mon[39562]: pgmap v415: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.60k op/s; 5289/219030 objects misplaced (2.415%); 2.1 MiB/s, 32 objects/s recovering 2024-04-20T17:09:05.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:04 smithi106 ceph-mon[44423]: pgmap v415: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.60k op/s; 5289/219030 objects misplaced (2.415%); 2.1 MiB/s, 32 objects/s recovering 2024-04-20T17:09:05.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:04 smithi171 ceph-mon[44189]: pgmap v415: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.60k op/s; 5289/219030 objects misplaced (2.415%); 2.1 MiB/s, 32 objects/s recovering 2024-04-20T17:09:05.442 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 49 16 114480 114464 145.987 94 0.0143285 0.00684834 2024-04-20T17:09:06.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:06 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:09:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:06 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:09:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:06 smithi106 ceph-mon[44423]: osdmap e137: 12 total, 12 up, 12 in 2024-04-20T17:09:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:06 smithi106 ceph-mon[44423]: osdmap e138: 12 total, 12 up, 12 in 2024-04-20T17:09:06.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:06 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:09:06.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:06 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:09:06.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:06 smithi171 ceph-mon[44189]: osdmap e137: 12 total, 12 up, 12 in 2024-04-20T17:09:06.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:06 smithi171 ceph-mon[44189]: osdmap e138: 12 total, 12 up, 12 in 2024-04-20T17:09:06.442 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 50 16 116001 115985 144.968 95.0625 0.00358235 0.0068961 2024-04-20T17:09:06.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:06 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:09:06.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:06 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:09:06.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:06 smithi005 ceph-mon[39562]: osdmap e137: 12 total, 12 up, 12 in 2024-04-20T17:09:06.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:06 smithi005 ceph-mon[39562]: osdmap e138: 12 total, 12 up, 12 in 2024-04-20T17:09:07.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:07 smithi106 ceph-mon[44423]: pgmap v418: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s; 5113/223014 objects misplaced (2.293%); 2.8 MiB/s, 45 objects/s recovering 2024-04-20T17:09:07.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:07 smithi171 ceph-mon[44189]: pgmap v418: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s; 5113/223014 objects misplaced (2.293%); 2.8 MiB/s, 45 objects/s recovering 2024-04-20T17:09:07.442 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 51 16 117503 117487 143.966 93.875 0.00494602 0.00694514 2024-04-20T17:09:07.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:07 smithi005 ceph-mon[39562]: pgmap v418: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s; 5113/223014 objects misplaced (2.293%); 2.8 MiB/s, 45 objects/s recovering 2024-04-20T17:09:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:08 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:09:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:08 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:09:08.442 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 52 15 118933 118918 142.917 89.4375 0.0091437 0.00699542 2024-04-20T17:09:08.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:08 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:09:08.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:08 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:09:08.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:08 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:09:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:08 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:09:09.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:09 smithi171 ceph-mon[44189]: 2.1 scrub starts 2024-04-20T17:09:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:09 smithi171 ceph-mon[44189]: pgmap v419: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.59k op/s; 5113/229576 objects misplaced (2.227%); 1.4 MiB/s, 21 objects/s recovering 2024-04-20T17:09:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:09 smithi171 ceph-mon[44189]: osdmap e139: 12 total, 12 up, 12 in 2024-04-20T17:09:09.442 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 53 16 120322 120306 141.857 86.75 0.028678 0.0070476 2024-04-20T17:09:09.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:09 smithi005 ceph-mon[39562]: 2.1 scrub starts 2024-04-20T17:09:09.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:09 smithi005 ceph-mon[39562]: pgmap v419: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.59k op/s; 5113/229576 objects misplaced (2.227%); 1.4 MiB/s, 21 objects/s recovering 2024-04-20T17:09:09.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:09 smithi005 ceph-mon[39562]: osdmap e139: 12 total, 12 up, 12 in 2024-04-20T17:09:09.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:09 smithi106 ceph-mon[44423]: 2.1 scrub starts 2024-04-20T17:09:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:09 smithi106 ceph-mon[44423]: pgmap v419: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.59k op/s; 5113/229576 objects misplaced (2.227%); 1.4 MiB/s, 21 objects/s recovering 2024-04-20T17:09:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:09 smithi106 ceph-mon[44423]: osdmap e139: 12 total, 12 up, 12 in 2024-04-20T17:09:09.629 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-20T17:09:09.786 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:09:09.799 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:09:09.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:09.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:09:09.916 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:09:09.946 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:09.967 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:09:09.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:09.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:09:10.071 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:09:10.109 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:10.135 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:09:10.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:10.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:09:10.243 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:09:10.278 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:10.306 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:09:10.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:10.309 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:09:10.410 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:09:10.438 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:10.442 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 54 16 121718 121702 140.846 87.25 0.0249144 0.00709789 2024-04-20T17:09:10.462 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:09:10.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:10.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:09:10.569 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:09:10.601 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:10.625 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:09:10.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:10.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:09:10.731 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:09:10.778 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:10.838 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:09:10.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:10.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:09:10.947 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:09:10.987 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:11.011 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:09:11.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:11.018 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:09:11.119 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:09:11.170 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:11.214 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:09:11.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:11.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:09:11.324 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:09:11.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:11 smithi171 ceph-mon[44189]: 2.1 scrub ok 2024-04-20T17:09:11.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:11 smithi171 ceph-mon[44189]: pgmap v421: 17 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 15 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s; 4919/235194 objects misplaced (2.091%); 3.8 MiB/s, 61 objects/s recovering 2024-04-20T17:09:11.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:11 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:09:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:11 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:09:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:11 smithi171 ceph-mon[44189]: osdmap e140: 12 total, 12 up, 12 in 2024-04-20T17:09:11.361 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:11.393 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:09:11.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:11.396 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:09:11.442 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 55 16 123034 123018 139.781 82.25 0.0104903 0.00715253 2024-04-20T17:09:11.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:11 smithi005 ceph-mon[39562]: 2.1 scrub ok 2024-04-20T17:09:11.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:11 smithi005 ceph-mon[39562]: pgmap v421: 17 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 15 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s; 4919/235194 objects misplaced (2.091%); 3.8 MiB/s, 61 objects/s recovering 2024-04-20T17:09:11.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:11 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:09:11.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:11 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:09:11.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:11 smithi005 ceph-mon[39562]: osdmap e140: 12 total, 12 up, 12 in 2024-04-20T17:09:11.496 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:09:11.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:11 smithi106 ceph-mon[44423]: 2.1 scrub ok 2024-04-20T17:09:11.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:11 smithi106 ceph-mon[44423]: pgmap v421: 17 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 15 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s; 4919/235194 objects misplaced (2.091%); 3.8 MiB/s, 61 objects/s recovering 2024-04-20T17:09:11.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:11 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:09:11.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:11 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:09:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:11 smithi106 ceph-mon[44423]: osdmap e140: 12 total, 12 up, 12 in 2024-04-20T17:09:11.553 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:11.835 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:09:12.060 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:09:12.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:12 smithi005 ceph-mon[39562]: 2.f scrub starts 2024-04-20T17:09:12.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:12 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/722857922' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:09:12.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:12 smithi005 ceph-mon[39562]: from='client.24794 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:09:12.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:12 smithi171 ceph-mon[44189]: 2.f scrub starts 2024-04-20T17:09:12.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:12 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/722857922' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:09:12.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:12 smithi171 ceph-mon[44189]: from='client.24794 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:09:12.442 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 56 16 124363 124347 138.768 83.0625 0.0081062 0.00720511 2024-04-20T17:09:12.448 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:09:12.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:12.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:09:12.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:12 smithi106 ceph-mon[44423]: 2.f scrub starts 2024-04-20T17:09:12.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:12 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/722857922' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:09:12.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:12 smithi106 ceph-mon[44423]: from='client.24794 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:09:12.555 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:09:12.590 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:12.620 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:09:12.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:12.627 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:09:12.728 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:09:12.732 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:09:12.758 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:12.781 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:09:12.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:12.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:09:12.885 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:09:12.942 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:13.030 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:13 smithi005 ceph-mon[39562]: pgmap v423: 17 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 15 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.35k op/s; 4919/239648 objects misplaced (2.053%); 2.0 MiB/s, 31 objects/s recovering 2024-04-20T17:09:13.030 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:13 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:09:13.030 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:13 smithi005 ceph-mon[39562]: from='client.24794 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:09:13.030 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:13 smithi005 ceph-mon[39562]: osdmap e141: 12 total, 12 up, 12 in 2024-04-20T17:09:13.031 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:13 smithi171 ceph-mon[44189]: pgmap v423: 17 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 15 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.35k op/s; 4919/239648 objects misplaced (2.053%); 2.0 MiB/s, 31 objects/s recovering 2024-04-20T17:09:13.031 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:13 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:09:13.031 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:13 smithi171 ceph-mon[44189]: from='client.24794 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:09:13.031 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:13 smithi171 ceph-mon[44189]: osdmap e141: 12 total, 12 up, 12 in 2024-04-20T17:09:13.078 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:09:13.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:13.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:09:13.185 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:09:13.223 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:13.251 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:09:13.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:13.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:09:13.358 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:09:13.395 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:13.419 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:09:13.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:13.422 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:09:13.442 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 57 16 125729 125713 137.831 85.375 0.0105729 0.00725367 2024-04-20T17:09:13.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:13 smithi106 ceph-mon[44423]: pgmap v423: 17 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 15 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.35k op/s; 4919/239648 objects misplaced (2.053%); 2.0 MiB/s, 31 objects/s recovering 2024-04-20T17:09:13.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:13 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:09:13.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:13 smithi106 ceph-mon[44423]: from='client.24794 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:09:13.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:13 smithi106 ceph-mon[44423]: osdmap e141: 12 total, 12 up, 12 in 2024-04-20T17:09:13.523 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:09:13.563 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:14.401 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:09:14.401 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:09:14.401 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 142, 2024-04-20T17:09:14.401 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "13c6e76c-ff37-11ee-bc93-c7b262605968", 2024-04-20T17:09:14.401 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-04-20T16:58:25.272557+0000", 2024-04-20T17:09:14.401 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-04-20T17:09:14.029416+0000", 2024-04-20T17:09:14.402 INFO:teuthology.orchestra.run.smithi005.stdout: "last_up_change": "2024-04-20T17:06:01.788926+0000", 2024-04-20T17:09:14.402 INFO:teuthology.orchestra.run.smithi005.stdout: "last_in_change": "2024-04-20T17:05:42.849446+0000", 2024-04-20T17:09:14.402 INFO:teuthology.orchestra.run.smithi005.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-20T17:09:14.402 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_num": 5801984, 2024-04-20T17:09:14.402 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_set": [ 2024-04-20T17:09:14.402 INFO:teuthology.orchestra.run.smithi005.stdout: "noscrub", 2024-04-20T17:09:14.402 INFO:teuthology.orchestra.run.smithi005.stdout: "pglog_hardlimit", 2024-04-20T17:09:14.402 INFO:teuthology.orchestra.run.smithi005.stdout: "purged_snapdirs", 2024-04-20T17:09:14.402 INFO:teuthology.orchestra.run.smithi005.stdout: "recovery_deletes", 2024-04-20T17:09:14.402 INFO:teuthology.orchestra.run.smithi005.stdout: "sortbitwise" 2024-04-20T17:09:14.402 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:09:14.402 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_version": 27, 2024-04-20T17:09:14.402 INFO:teuthology.orchestra.run.smithi005.stdout: "full_ratio": 0.94999998807907104, 2024-04-20T17:09:14.402 INFO:teuthology.orchestra.run.smithi005.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-20T17:09:14.403 INFO:teuthology.orchestra.run.smithi005.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-20T17:09:14.403 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_snapshot": "", 2024-04-20T17:09:14.403 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_max": 2, 2024-04-20T17:09:14.403 INFO:teuthology.orchestra.run.smithi005.stdout: "max_osd": 12, 2024-04-20T17:09:14.403 INFO:teuthology.orchestra.run.smithi005.stdout: "require_min_compat_client": "luminous", 2024-04-20T17:09:14.403 INFO:teuthology.orchestra.run.smithi005.stdout: "min_compat_client": "luminous", 2024-04-20T17:09:14.403 INFO:teuthology.orchestra.run.smithi005.stdout: "require_osd_release": "squid", 2024-04-20T17:09:14.403 INFO:teuthology.orchestra.run.smithi005.stdout: "allow_crimson": false, 2024-04-20T17:09:14.403 INFO:teuthology.orchestra.run.smithi005.stdout: "pools": [ 2024-04-20T17:09:14.403 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.403 INFO:teuthology.orchestra.run.smithi005.stdout: "pool": 1, 2024-04-20T17:09:14.403 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_name": ".mgr", 2024-04-20T17:09:14.403 INFO:teuthology.orchestra.run.smithi005.stdout: "create_time": "2024-04-20T17:01:35.909493+0000", 2024-04-20T17:09:14.403 INFO:teuthology.orchestra.run.smithi005.stdout: "flags": 1, 2024-04-20T17:09:14.404 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_names": "hashpspool", 2024-04-20T17:09:14.404 INFO:teuthology.orchestra.run.smithi005.stdout: "type": 1, 2024-04-20T17:09:14.404 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 2, 2024-04-20T17:09:14.404 INFO:teuthology.orchestra.run.smithi005.stdout: "min_size": 2, 2024-04-20T17:09:14.404 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_rule": 0, 2024-04-20T17:09:14.404 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_count": 0, 2024-04-20T17:09:14.404 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_target": 0, 2024-04-20T17:09:14.404 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T17:09:14.404 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T17:09:14.404 INFO:teuthology.orchestra.run.smithi005.stdout: "object_hash": 2, 2024-04-20T17:09:14.404 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_autoscale_mode": "off", 2024-04-20T17:09:14.404 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num": 1, 2024-04-20T17:09:14.404 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num": 1, 2024-04-20T17:09:14.404 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num_target": 1, 2024-04-20T17:09:14.405 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_target": 1, 2024-04-20T17:09:14.405 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_pending": 1, 2024-04-20T17:09:14.405 INFO:teuthology.orchestra.run.smithi005.stdout: "last_pg_merge_meta": { 2024-04-20T17:09:14.405 INFO:teuthology.orchestra.run.smithi005.stdout: "source_pgid": "0.0", 2024-04-20T17:09:14.405 INFO:teuthology.orchestra.run.smithi005.stdout: "ready_epoch": 0, 2024-04-20T17:09:14.405 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_started": 0, 2024-04-20T17:09:14.405 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_clean": 0, 2024-04-20T17:09:14.405 INFO:teuthology.orchestra.run.smithi005.stdout: "source_version": "0'0", 2024-04-20T17:09:14.405 INFO:teuthology.orchestra.run.smithi005.stdout: "target_version": "0'0" 2024-04-20T17:09:14.405 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.405 INFO:teuthology.orchestra.run.smithi005.stdout: "last_change": "16", 2024-04-20T17:09:14.405 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend": "0", 2024-04-20T17:09:14.405 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-20T17:09:14.405 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_preluminous": "0", 2024-04-20T17:09:14.406 INFO:teuthology.orchestra.run.smithi005.stdout: "auid": 0, 2024-04-20T17:09:14.406 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_mode": "selfmanaged", 2024-04-20T17:09:14.406 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_seq": 0, 2024-04-20T17:09:14.406 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_epoch": 0, 2024-04-20T17:09:14.406 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_snaps": [], 2024-04-20T17:09:14.406 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_snaps": "[]", 2024-04-20T17:09:14.406 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_bytes": 0, 2024-04-20T17:09:14.406 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_objects": 0, 2024-04-20T17:09:14.406 INFO:teuthology.orchestra.run.smithi005.stdout: "tiers": [], 2024-04-20T17:09:14.406 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_of": -1, 2024-04-20T17:09:14.406 INFO:teuthology.orchestra.run.smithi005.stdout: "read_tier": -1, 2024-04-20T17:09:14.406 INFO:teuthology.orchestra.run.smithi005.stdout: "write_tier": -1, 2024-04-20T17:09:14.406 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_mode": "none", 2024-04-20T17:09:14.407 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_bytes": 0, 2024-04-20T17:09:14.407 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_objects": 0, 2024-04-20T17:09:14.407 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T17:09:14.407 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T17:09:14.407 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T17:09:14.407 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_flush_age": 0, 2024-04-20T17:09:14.407 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_evict_age": 0, 2024-04-20T17:09:14.407 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_profile": "", 2024-04-20T17:09:14.407 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_params": { 2024-04-20T17:09:14.407 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "none" 2024-04-20T17:09:14.408 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.408 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_period": 0, 2024-04-20T17:09:14.408 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_count": 0, 2024-04-20T17:09:14.408 INFO:teuthology.orchestra.run.smithi005.stdout: "use_gmt_hitset": true, 2024-04-20T17:09:14.408 INFO:teuthology.orchestra.run.smithi005.stdout: "min_read_recency_for_promote": 0, 2024-04-20T17:09:14.408 INFO:teuthology.orchestra.run.smithi005.stdout: "min_write_recency_for_promote": 0, 2024-04-20T17:09:14.408 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T17:09:14.408 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_search_last_n": 0, 2024-04-20T17:09:14.408 INFO:teuthology.orchestra.run.smithi005.stdout: "grade_table": [], 2024-04-20T17:09:14.408 INFO:teuthology.orchestra.run.smithi005.stdout: "stripe_width": 0, 2024-04-20T17:09:14.408 INFO:teuthology.orchestra.run.smithi005.stdout: "expected_num_objects": 0, 2024-04-20T17:09:14.408 INFO:teuthology.orchestra.run.smithi005.stdout: "fast_read": false, 2024-04-20T17:09:14.408 INFO:teuthology.orchestra.run.smithi005.stdout: "options": { 2024-04-20T17:09:14.408 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_max": 32, 2024-04-20T17:09:14.409 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_min": 1 2024-04-20T17:09:14.409 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.409 INFO:teuthology.orchestra.run.smithi005.stdout: "application_metadata": { 2024-04-20T17:09:14.409 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": {} 2024-04-20T17:09:14.409 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.409 INFO:teuthology.orchestra.run.smithi005.stdout: "read_balance": { 2024-04-20T17:09:14.409 INFO:teuthology.orchestra.run.smithi005.stdout: "score_type": "Fair distribution", 2024-04-20T17:09:14.409 INFO:teuthology.orchestra.run.smithi005.stdout: "score_acting": 11.760000228881836, 2024-04-20T17:09:14.409 INFO:teuthology.orchestra.run.smithi005.stdout: "score_stable": 11.760000228881836, 2024-04-20T17:09:14.409 INFO:teuthology.orchestra.run.smithi005.stdout: "optimal_score": 0.17000000178813934, 2024-04-20T17:09:14.409 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_acting": 2, 2024-04-20T17:09:14.409 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_stable": 2, 2024-04-20T17:09:14.409 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-04-20T17:09:14.409 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity": 1, 2024-04-20T17:09:14.410 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity_weighted": 1 2024-04-20T17:09:14.410 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.410 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.410 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.410 INFO:teuthology.orchestra.run.smithi005.stdout: "pool": 2, 2024-04-20T17:09:14.410 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_name": "unique_pool_0", 2024-04-20T17:09:14.410 INFO:teuthology.orchestra.run.smithi005.stdout: "create_time": "2024-04-20T17:08:04.904546+0000", 2024-04-20T17:09:14.410 INFO:teuthology.orchestra.run.smithi005.stdout: "flags": 1, 2024-04-20T17:09:14.410 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_names": "hashpspool", 2024-04-20T17:09:14.410 INFO:teuthology.orchestra.run.smithi005.stdout: "type": 1, 2024-04-20T17:09:14.411 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 2, 2024-04-20T17:09:14.411 INFO:teuthology.orchestra.run.smithi005.stdout: "min_size": 2, 2024-04-20T17:09:14.411 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_rule": 0, 2024-04-20T17:09:14.411 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_count": 0, 2024-04-20T17:09:14.411 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_target": 0, 2024-04-20T17:09:14.411 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T17:09:14.411 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T17:09:14.411 INFO:teuthology.orchestra.run.smithi005.stdout: "object_hash": 2, 2024-04-20T17:09:14.411 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_autoscale_mode": "off", 2024-04-20T17:09:14.411 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num": 16, 2024-04-20T17:09:14.411 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num": 16, 2024-04-20T17:09:14.411 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num_target": 16, 2024-04-20T17:09:14.411 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_target": 16, 2024-04-20T17:09:14.411 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_pending": 16, 2024-04-20T17:09:14.412 INFO:teuthology.orchestra.run.smithi005.stdout: "last_pg_merge_meta": { 2024-04-20T17:09:14.412 INFO:teuthology.orchestra.run.smithi005.stdout: "source_pgid": "0.0", 2024-04-20T17:09:14.412 INFO:teuthology.orchestra.run.smithi005.stdout: "ready_epoch": 0, 2024-04-20T17:09:14.412 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_started": 0, 2024-04-20T17:09:14.412 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_clean": 0, 2024-04-20T17:09:14.412 INFO:teuthology.orchestra.run.smithi005.stdout: "source_version": "0'0", 2024-04-20T17:09:14.412 INFO:teuthology.orchestra.run.smithi005.stdout: "target_version": "0'0" 2024-04-20T17:09:14.412 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.412 INFO:teuthology.orchestra.run.smithi005.stdout: "last_change": "118", 2024-04-20T17:09:14.412 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend": "0", 2024-04-20T17:09:14.412 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-20T17:09:14.412 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_preluminous": "0", 2024-04-20T17:09:14.412 INFO:teuthology.orchestra.run.smithi005.stdout: "auid": 0, 2024-04-20T17:09:14.412 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_mode": "selfmanaged", 2024-04-20T17:09:14.413 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_seq": 0, 2024-04-20T17:09:14.413 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_epoch": 0, 2024-04-20T17:09:14.413 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_snaps": [], 2024-04-20T17:09:14.413 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_snaps": "[]", 2024-04-20T17:09:14.413 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_bytes": 0, 2024-04-20T17:09:14.413 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_objects": 0, 2024-04-20T17:09:14.413 INFO:teuthology.orchestra.run.smithi005.stdout: "tiers": [], 2024-04-20T17:09:14.413 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_of": -1, 2024-04-20T17:09:14.413 INFO:teuthology.orchestra.run.smithi005.stdout: "read_tier": -1, 2024-04-20T17:09:14.413 INFO:teuthology.orchestra.run.smithi005.stdout: "write_tier": -1, 2024-04-20T17:09:14.413 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_mode": "none", 2024-04-20T17:09:14.413 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_bytes": 0, 2024-04-20T17:09:14.413 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_objects": 0, 2024-04-20T17:09:14.413 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T17:09:14.413 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T17:09:14.414 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T17:09:14.414 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_flush_age": 0, 2024-04-20T17:09:14.414 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_evict_age": 0, 2024-04-20T17:09:14.414 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_profile": "", 2024-04-20T17:09:14.414 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_params": { 2024-04-20T17:09:14.414 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "none" 2024-04-20T17:09:14.414 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.414 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_period": 0, 2024-04-20T17:09:14.414 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_count": 0, 2024-04-20T17:09:14.414 INFO:teuthology.orchestra.run.smithi005.stdout: "use_gmt_hitset": true, 2024-04-20T17:09:14.414 INFO:teuthology.orchestra.run.smithi005.stdout: "min_read_recency_for_promote": 0, 2024-04-20T17:09:14.414 INFO:teuthology.orchestra.run.smithi005.stdout: "min_write_recency_for_promote": 0, 2024-04-20T17:09:14.414 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T17:09:14.414 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_search_last_n": 0, 2024-04-20T17:09:14.414 INFO:teuthology.orchestra.run.smithi005.stdout: "grade_table": [], 2024-04-20T17:09:14.415 INFO:teuthology.orchestra.run.smithi005.stdout: "stripe_width": 0, 2024-04-20T17:09:14.415 INFO:teuthology.orchestra.run.smithi005.stdout: "expected_num_objects": 0, 2024-04-20T17:09:14.415 INFO:teuthology.orchestra.run.smithi005.stdout: "fast_read": false, 2024-04-20T17:09:14.415 INFO:teuthology.orchestra.run.smithi005.stdout: "options": {}, 2024-04-20T17:09:14.415 INFO:teuthology.orchestra.run.smithi005.stdout: "application_metadata": { 2024-04-20T17:09:14.415 INFO:teuthology.orchestra.run.smithi005.stdout: "rados": {} 2024-04-20T17:09:14.415 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.415 INFO:teuthology.orchestra.run.smithi005.stdout: "read_balance": { 2024-04-20T17:09:14.415 INFO:teuthology.orchestra.run.smithi005.stdout: "score_type": "Fair distribution", 2024-04-20T17:09:14.415 INFO:teuthology.orchestra.run.smithi005.stdout: "score_acting": 2.25, 2024-04-20T17:09:14.415 INFO:teuthology.orchestra.run.smithi005.stdout: "score_stable": 2.25, 2024-04-20T17:09:14.415 INFO:teuthology.orchestra.run.smithi005.stdout: "optimal_score": 1, 2024-04-20T17:09:14.415 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_acting": 2.25, 2024-04-20T17:09:14.415 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_stable": 2.25, 2024-04-20T17:09:14.416 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-04-20T17:09:14.416 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity": 1, 2024-04-20T17:09:14.416 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity_weighted": 1 2024-04-20T17:09:14.416 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.416 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.416 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:09:14.416 INFO:teuthology.orchestra.run.smithi005.stdout: "osds": [ 2024-04-20T17:09:14.416 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.416 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 0, 2024-04-20T17:09:14.416 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b", 2024-04-20T17:09:14.416 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:09:14.416 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:09:14.416 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:09:14.416 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:09:14.416 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:09:14.417 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:09:14.417 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 8, 2024-04-20T17:09:14.417 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 133, 2024-04-20T17:09:14.417 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:09:14.417 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:09:14.417 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:09:14.417 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.417 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.417 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.417 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6801", 2024-04-20T17:09:14.417 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:09:14.417 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.417 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.417 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.417 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:09:14.418 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.418 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.418 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.418 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6802", 2024-04-20T17:09:14.418 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:09:14.418 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.418 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.418 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.418 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:09:14.418 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.418 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.418 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.419 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6804", 2024-04-20T17:09:14.419 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:09:14.419 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.419 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.419 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.419 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:09:14.419 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.419 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.419 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.419 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6803", 2024-04-20T17:09:14.419 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:09:14.419 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.420 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.420 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.420 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6801/3901484705", 2024-04-20T17:09:14.420 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6802/3901484705", 2024-04-20T17:09:14.420 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6804/3901484705", 2024-04-20T17:09:14.420 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6803/3901484705", 2024-04-20T17:09:14.420 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:09:14.420 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:09:14.420 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:09:14.420 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.421 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.421 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.421 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 1, 2024-04-20T17:09:14.421 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "aed0c8eb-0e81-4e85-ab8f-a60420060656", 2024-04-20T17:09:14.421 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:09:14.421 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:09:14.421 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:09:14.421 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:09:14.421 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:09:14.421 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:09:14.421 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 13, 2024-04-20T17:09:14.421 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 115, 2024-04-20T17:09:14.421 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:09:14.421 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:09:14.421 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:09:14.422 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.422 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.422 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.422 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6805", 2024-04-20T17:09:14.422 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3391893914 2024-04-20T17:09:14.422 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.422 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.422 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.422 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:09:14.422 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.422 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.422 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.422 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6806", 2024-04-20T17:09:14.422 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3391893914 2024-04-20T17:09:14.423 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.423 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.423 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.423 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:09:14.423 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.423 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.423 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.423 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6808", 2024-04-20T17:09:14.423 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3391893914 2024-04-20T17:09:14.423 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.423 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.423 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.423 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:09:14.423 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.424 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.424 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.424 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6807", 2024-04-20T17:09:14.424 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3391893914 2024-04-20T17:09:14.424 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.424 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.424 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.424 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6805/3391893914", 2024-04-20T17:09:14.424 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6806/3391893914", 2024-04-20T17:09:14.424 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6808/3391893914", 2024-04-20T17:09:14.424 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6807/3391893914", 2024-04-20T17:09:14.424 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:09:14.424 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:09:14.424 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:09:14.425 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.425 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.425 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.425 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 2, 2024-04-20T17:09:14.425 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "f733a6a2-19c2-4c7a-af9c-04e282062bb7", 2024-04-20T17:09:14.425 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:09:14.425 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:09:14.425 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:09:14.425 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:09:14.425 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:09:14.425 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:09:14.425 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 20, 2024-04-20T17:09:14.425 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 115, 2024-04-20T17:09:14.425 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:09:14.425 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:09:14.426 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:09:14.426 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.426 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.426 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.426 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6809", 2024-04-20T17:09:14.426 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2105468358 2024-04-20T17:09:14.426 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.426 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.426 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.426 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:09:14.426 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.426 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.426 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.426 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6810", 2024-04-20T17:09:14.427 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2105468358 2024-04-20T17:09:14.427 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.427 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.427 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.427 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:09:14.427 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.427 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.427 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.427 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6812", 2024-04-20T17:09:14.427 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2105468358 2024-04-20T17:09:14.427 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.427 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.427 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.427 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:09:14.427 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.428 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.428 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.428 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6811", 2024-04-20T17:09:14.428 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2105468358 2024-04-20T17:09:14.428 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.428 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.428 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.428 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6809/2105468358", 2024-04-20T17:09:14.428 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6810/2105468358", 2024-04-20T17:09:14.428 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6812/2105468358", 2024-04-20T17:09:14.428 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6811/2105468358", 2024-04-20T17:09:14.428 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:09:14.428 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:09:14.428 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:09:14.428 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.429 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.429 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.429 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 3, 2024-04-20T17:09:14.429 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "8c92f1df-e298-4606-88ae-b8a8e79628a1", 2024-04-20T17:09:14.429 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:09:14.429 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:09:14.429 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:09:14.429 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:09:14.429 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:09:14.429 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:09:14.429 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 25, 2024-04-20T17:09:14.429 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 0, 2024-04-20T17:09:14.429 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:09:14.429 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:09:14.429 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:09:14.430 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.430 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.430 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.430 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6813", 2024-04-20T17:09:14.430 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 377652039 2024-04-20T17:09:14.430 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.430 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.430 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.430 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:09:14.430 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.430 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.430 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.430 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6814", 2024-04-20T17:09:14.430 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 377652039 2024-04-20T17:09:14.431 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.431 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.431 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.431 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:09:14.431 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.431 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.431 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.431 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6816", 2024-04-20T17:09:14.431 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 377652039 2024-04-20T17:09:14.431 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.431 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.431 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.431 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:09:14.431 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.431 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.431 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.432 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6815", 2024-04-20T17:09:14.432 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 377652039 2024-04-20T17:09:14.432 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.432 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.432 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.432 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6813/377652039", 2024-04-20T17:09:14.432 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6814/377652039", 2024-04-20T17:09:14.432 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6816/377652039", 2024-04-20T17:09:14.432 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6815/377652039", 2024-04-20T17:09:14.432 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:09:14.432 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:09:14.432 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:09:14.432 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.432 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.433 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.433 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 4, 2024-04-20T17:09:14.433 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "46df16f1-9607-4d7b-9be8-9d4236e160d1", 2024-04-20T17:09:14.433 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:09:14.433 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:09:14.433 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:09:14.433 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:09:14.433 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:09:14.433 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:09:14.433 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 31, 2024-04-20T17:09:14.433 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 0, 2024-04-20T17:09:14.433 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:09:14.433 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:09:14.433 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:09:14.433 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.434 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.434 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.434 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6800", 2024-04-20T17:09:14.434 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:09:14.434 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.434 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.434 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.434 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:09:14.434 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.434 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.434 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.434 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6801", 2024-04-20T17:09:14.434 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:09:14.434 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.434 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.435 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.435 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:09:14.435 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.435 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.435 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.435 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6803", 2024-04-20T17:09:14.435 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:09:14.435 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.435 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.435 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.435 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:09:14.435 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.435 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.435 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.435 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6802", 2024-04-20T17:09:14.436 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:09:14.436 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.436 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.436 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.436 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6800/1839524469", 2024-04-20T17:09:14.436 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6801/1839524469", 2024-04-20T17:09:14.436 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6803/1839524469", 2024-04-20T17:09:14.436 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6802/1839524469", 2024-04-20T17:09:14.436 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:09:14.436 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:09:14.436 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:09:14.436 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.436 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.436 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.436 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 5, 2024-04-20T17:09:14.437 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "6603231d-1e5c-4575-bfaa-2eeca2c12215", 2024-04-20T17:09:14.437 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:09:14.437 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:09:14.437 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:09:14.437 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:09:14.437 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:09:14.437 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:09:14.437 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 37, 2024-04-20T17:09:14.437 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 132, 2024-04-20T17:09:14.437 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:09:14.437 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:09:14.437 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:09:14.437 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.437 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.437 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.438 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6804", 2024-04-20T17:09:14.438 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:09:14.438 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.438 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.438 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.438 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:09:14.438 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.438 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.438 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.438 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6805", 2024-04-20T17:09:14.438 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:09:14.438 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.438 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.438 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.439 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:09:14.439 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.439 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.439 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.439 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6807", 2024-04-20T17:09:14.439 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:09:14.439 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.439 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.439 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.439 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:09:14.439 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.439 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.439 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.439 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6806", 2024-04-20T17:09:14.440 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:09:14.440 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.440 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.440 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.440 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6804/2450315908", 2024-04-20T17:09:14.440 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6805/2450315908", 2024-04-20T17:09:14.440 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6807/2450315908", 2024-04-20T17:09:14.440 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6806/2450315908", 2024-04-20T17:09:14.440 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:09:14.440 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:09:14.440 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:09:14.440 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.440 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.440 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.441 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 6, 2024-04-20T17:09:14.441 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "783a55a6-f176-42be-8c8b-af8481d08dc9", 2024-04-20T17:09:14.441 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:09:14.441 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:09:14.441 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:09:14.441 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:09:14.441 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:09:14.441 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:09:14.441 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 48, 2024-04-20T17:09:14.441 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 115, 2024-04-20T17:09:14.441 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:09:14.441 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:09:14.441 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:09:14.441 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.442 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.442 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.442 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6808", 2024-04-20T17:09:14.442 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2840237913 2024-04-20T17:09:14.442 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.442 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.442 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.442 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:09:14.442 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.442 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.442 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.442 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6809", 2024-04-20T17:09:14.442 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2840237913 2024-04-20T17:09:14.442 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.442 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.443 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.443 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:09:14.443 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.443 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.443 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.443 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6811", 2024-04-20T17:09:14.443 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2840237913 2024-04-20T17:09:14.443 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.443 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.443 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.443 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:09:14.443 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.443 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.443 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.444 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6810", 2024-04-20T17:09:14.444 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2840237913 2024-04-20T17:09:14.444 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.444 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.444 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.444 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6808/2840237913", 2024-04-20T17:09:14.444 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6809/2840237913", 2024-04-20T17:09:14.444 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6811/2840237913", 2024-04-20T17:09:14.444 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6810/2840237913", 2024-04-20T17:09:14.444 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:09:14.444 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:09:14.444 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:09:14.444 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.444 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.444 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.445 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 7, 2024-04-20T17:09:14.445 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4", 2024-04-20T17:09:14.445 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:09:14.445 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:09:14.445 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:09:14.445 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:09:14.445 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:09:14.445 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:09:14.445 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 56, 2024-04-20T17:09:14.445 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 115, 2024-04-20T17:09:14.445 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:09:14.445 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:09:14.445 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:09:14.445 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.445 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.446 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.446 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6812", 2024-04-20T17:09:14.446 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1105371883 2024-04-20T17:09:14.446 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.446 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.446 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.446 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:09:14.446 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.446 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.446 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.446 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6813", 2024-04-20T17:09:14.446 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1105371883 2024-04-20T17:09:14.446 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.446 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.447 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.447 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:09:14.447 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.447 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.447 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.447 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6815", 2024-04-20T17:09:14.447 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1105371883 2024-04-20T17:09:14.447 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.447 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.447 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.447 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:09:14.447 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.447 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.447 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.447 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6814", 2024-04-20T17:09:14.448 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1105371883 2024-04-20T17:09:14.448 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.448 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.448 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.448 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6812/1105371883", 2024-04-20T17:09:14.448 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6813/1105371883", 2024-04-20T17:09:14.448 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6815/1105371883", 2024-04-20T17:09:14.448 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6814/1105371883", 2024-04-20T17:09:14.448 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:09:14.448 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:09:14.448 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:09:14.448 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.448 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.448 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.449 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 8, 2024-04-20T17:09:14.449 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "1cb232ad-1f59-4a0a-a215-e67419ad6476", 2024-04-20T17:09:14.449 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:09:14.449 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:09:14.449 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:09:14.449 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:09:14.449 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:09:14.449 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:09:14.449 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 67, 2024-04-20T17:09:14.449 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 115, 2024-04-20T17:09:14.449 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:09:14.449 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:09:14.449 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:09:14.449 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.449 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.450 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.450 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6800", 2024-04-20T17:09:14.450 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:09:14.450 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.450 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.450 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.450 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:09:14.450 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.450 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.450 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.450 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6801", 2024-04-20T17:09:14.450 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:09:14.450 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.450 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.450 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.451 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:09:14.451 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.451 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.451 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.451 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6803", 2024-04-20T17:09:14.451 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:09:14.451 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.451 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.451 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.452 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:09:14.452 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.452 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.452 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.452 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6802", 2024-04-20T17:09:14.452 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:09:14.452 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.452 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.452 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.452 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6800/4123058942", 2024-04-20T17:09:14.452 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6801/4123058942", 2024-04-20T17:09:14.452 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6803/4123058942", 2024-04-20T17:09:14.452 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6802/4123058942", 2024-04-20T17:09:14.452 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:09:14.453 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:09:14.453 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:09:14.453 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.453 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.453 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.453 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 9, 2024-04-20T17:09:14.453 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "b6c17147-b3dc-475b-bb20-f9f712fb99e8", 2024-04-20T17:09:14.453 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:09:14.453 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:09:14.453 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:09:14.453 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:09:14.453 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:09:14.453 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:09:14.453 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 73, 2024-04-20T17:09:14.453 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 115, 2024-04-20T17:09:14.454 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:09:14.454 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:09:14.454 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:09:14.454 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.454 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.454 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.454 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6804", 2024-04-20T17:09:14.454 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:09:14.454 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.454 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.454 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.454 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:09:14.454 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.454 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.454 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.455 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6805", 2024-04-20T17:09:14.455 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:09:14.455 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.455 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.455 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.455 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:09:14.455 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.455 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.455 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.455 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6807", 2024-04-20T17:09:14.455 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:09:14.455 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.455 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.455 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.455 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:09:14.456 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.456 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.456 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.456 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6806", 2024-04-20T17:09:14.456 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:09:14.456 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.456 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.456 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.456 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6804/2877589636", 2024-04-20T17:09:14.456 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6805/2877589636", 2024-04-20T17:09:14.456 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6807/2877589636", 2024-04-20T17:09:14.456 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6806/2877589636", 2024-04-20T17:09:14.456 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:09:14.456 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:09:14.456 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:09:14.457 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.457 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.457 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.457 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 10, 2024-04-20T17:09:14.457 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "1adcdf55-f002-4a39-91e7-378e5fae849c", 2024-04-20T17:09:14.457 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:09:14.457 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:09:14.457 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:09:14.457 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:09:14.457 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:09:14.457 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:09:14.457 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 84, 2024-04-20T17:09:14.457 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 0, 2024-04-20T17:09:14.458 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:09:14.458 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:09:14.458 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:09:14.458 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.458 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.458 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.458 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6808", 2024-04-20T17:09:14.458 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:09:14.458 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.458 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.458 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.458 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:09:14.458 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.458 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.459 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.459 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6809", 2024-04-20T17:09:14.459 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:09:14.459 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.459 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.459 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.459 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:09:14.459 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.459 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.459 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.459 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6811", 2024-04-20T17:09:14.459 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:09:14.459 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.459 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.460 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.460 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:09:14.460 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.460 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.460 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.460 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6810", 2024-04-20T17:09:14.460 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:09:14.460 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.460 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.460 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.460 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6808/181490621", 2024-04-20T17:09:14.460 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6809/181490621", 2024-04-20T17:09:14.460 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6811/181490621", 2024-04-20T17:09:14.460 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6810/181490621", 2024-04-20T17:09:14.461 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:09:14.461 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:09:14.461 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:09:14.461 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.461 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.461 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.461 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 11, 2024-04-20T17:09:14.461 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "737c5aec-694a-4968-819d-fccf22661f4f", 2024-04-20T17:09:14.461 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:09:14.461 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:09:14.461 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:09:14.461 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:09:14.461 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:09:14.461 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:09:14.461 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 92, 2024-04-20T17:09:14.462 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 115, 2024-04-20T17:09:14.462 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:09:14.462 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:09:14.462 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:09:14.462 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.462 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.462 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.462 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6812", 2024-04-20T17:09:14.462 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:09:14.462 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.462 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.462 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.462 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:09:14.463 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.463 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.463 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.463 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6813", 2024-04-20T17:09:14.463 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:09:14.463 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.463 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.463 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.463 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:09:14.463 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.463 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.463 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.463 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6815", 2024-04-20T17:09:14.463 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:09:14.464 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.464 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.464 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.464 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:09:14.464 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:09:14.464 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.464 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:09:14.464 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6814", 2024-04-20T17:09:14.464 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:09:14.464 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.464 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.464 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.464 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6812/3373585701", 2024-04-20T17:09:14.464 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6813/3373585701", 2024-04-20T17:09:14.464 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6815/3373585701", 2024-04-20T17:09:14.465 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6814/3373585701", 2024-04-20T17:09:14.465 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:09:14.465 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:09:14.465 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:09:14.465 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.465 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.465 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:09:14.465 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_xinfo": [ 2024-04-20T17:09:14.465 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.465 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 0, 2024-04-20T17:09:14.465 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:09:14.465 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:09:14.465 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:09:14.465 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:09:14.466 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:09:14.466 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:08:46.110149+0000", 2024-04-20T17:09:14.466 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:09:14.466 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.466 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.466 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 1, 2024-04-20T17:09:14.466 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:09:14.466 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:09:14.466 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:09:14.466 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:09:14.466 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:09:14.466 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:09:13.276482+0000", 2024-04-20T17:09:14.466 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:09:14.467 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.467 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.467 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 2, 2024-04-20T17:09:14.467 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:09:14.467 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:09:14.467 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:09:14.467 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:09:14.467 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:09:14.467 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:08:14.773262+0000", 2024-04-20T17:09:14.467 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:09:14.467 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.467 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.467 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 3, 2024-04-20T17:09:14.467 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:09:14.467 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:09:14.467 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:09:14.468 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:09:14.468 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:09:14.468 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:08:42.721685+0000", 2024-04-20T17:09:14.468 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:09:14.468 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.468 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.468 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 4, 2024-04-20T17:09:14.468 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:09:14.468 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:09:14.468 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:09:14.468 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:09:14.468 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:09:14.468 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:09:10.189022+0000", 2024-04-20T17:09:14.468 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:09:14.468 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.469 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.469 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 5, 2024-04-20T17:09:14.469 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:09:14.469 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:09:14.469 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:09:14.469 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:09:14.469 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:09:14.469 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:09:07.140169+0000", 2024-04-20T17:09:14.469 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:09:14.469 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.469 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.469 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 6, 2024-04-20T17:09:14.469 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:09:14.469 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:09:14.470 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:09:14.470 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:09:14.470 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:09:14.470 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:08:21.893763+0000", 2024-04-20T17:09:14.470 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:09:14.470 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.470 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.470 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 7, 2024-04-20T17:09:14.470 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:09:14.470 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:09:14.470 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:09:14.470 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:09:14.470 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:09:14.470 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:09:04.202179+0000", 2024-04-20T17:09:14.471 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:09:14.471 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.471 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.471 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 8, 2024-04-20T17:09:14.471 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:09:14.471 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:09:14.471 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:09:14.471 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:09:14.471 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:09:14.471 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:08:24.889880+0000", 2024-04-20T17:09:14.471 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:09:14.471 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.471 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.471 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 9, 2024-04-20T17:09:14.472 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:09:14.472 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:09:14.472 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:09:14.472 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:09:14.472 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:09:14.472 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:08:49.673293+0000", 2024-04-20T17:09:14.472 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:09:14.472 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.472 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.472 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 10, 2024-04-20T17:09:14.472 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:09:14.472 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:09:14.472 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:09:14.472 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:09:14.472 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:09:14.473 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:08:16.452578+0000", 2024-04-20T17:09:14.473 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:09:14.473 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.473 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.473 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 11, 2024-04-20T17:09:14.473 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:09:14.473 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:09:14.473 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:09:14.473 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:09:14.473 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:09:14.473 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:08:47.110274+0000", 2024-04-20T17:09:14.473 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:09:14.473 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.473 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:09:14.473 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_upmap": [], 2024-04-20T17:09:14.474 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_upmap_items": [ 2024-04-20T17:09:14.474 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.474 INFO:teuthology.orchestra.run.smithi005.stdout: "pgid": "2.4", 2024-04-20T17:09:14.474 INFO:teuthology.orchestra.run.smithi005.stdout: "mappings": [ 2024-04-20T17:09:14.474 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.474 INFO:teuthology.orchestra.run.smithi005.stdout: "from": 1, 2024-04-20T17:09:14.474 INFO:teuthology.orchestra.run.smithi005.stdout: "to": 5 2024-04-20T17:09:14.474 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.474 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.474 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.474 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:09:14.474 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_upmap_primaries": [], 2024-04-20T17:09:14.474 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_temp": [ 2024-04-20T17:09:14.474 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:09:14.475 INFO:teuthology.orchestra.run.smithi005.stdout: "pgid": "2.4", 2024-04-20T17:09:14.475 INFO:teuthology.orchestra.run.smithi005.stdout: "osds": [ 2024-04-20T17:09:14.475 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-04-20T17:09:14.475 INFO:teuthology.orchestra.run.smithi005.stdout: 5 2024-04-20T17:09:14.475 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:09:14.475 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.475 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:09:14.475 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_temp": [], 2024-04-20T17:09:14.475 INFO:teuthology.orchestra.run.smithi005.stdout: "blocklist": { 2024-04-20T17:09:14.475 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/3881726425": "2024-04-21T16:59:25.808337+0000", 2024-04-20T17:09:14.475 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/338649532": "2024-04-21T16:59:25.808337+0000", 2024-04-20T17:09:14.475 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/1227843664": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:09:14.475 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:6800/1876064631": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:09:14.475 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/160689449": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:09:14.475 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:6800/3521715286": "2024-04-21T16:59:25.808337+0000", 2024-04-20T17:09:14.476 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/3841663462": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:09:14.476 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/687690919": "2024-04-21T16:59:25.808337+0000" 2024-04-20T17:09:14.476 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.476 INFO:teuthology.orchestra.run.smithi005.stdout: "range_blocklist": {}, 2024-04-20T17:09:14.476 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_profiles": { 2024-04-20T17:09:14.476 INFO:teuthology.orchestra.run.smithi005.stdout: "default": { 2024-04-20T17:09:14.476 INFO:teuthology.orchestra.run.smithi005.stdout: "crush-failure-domain": "osd", 2024-04-20T17:09:14.476 INFO:teuthology.orchestra.run.smithi005.stdout: "k": "2", 2024-04-20T17:09:14.476 INFO:teuthology.orchestra.run.smithi005.stdout: "m": "1", 2024-04-20T17:09:14.476 INFO:teuthology.orchestra.run.smithi005.stdout: "plugin": "jerasure", 2024-04-20T17:09:14.476 INFO:teuthology.orchestra.run.smithi005.stdout: "technique": "reed_sol_van" 2024-04-20T17:09:14.476 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.476 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:09:14.476 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_snaps_queue": [], 2024-04-20T17:09:14.476 INFO:teuthology.orchestra.run.smithi005.stdout: "new_removed_snaps": [], 2024-04-20T17:09:14.477 INFO:teuthology.orchestra.run.smithi005.stdout: "new_purged_snaps": [], 2024-04-20T17:09:14.477 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_node_flags": {}, 2024-04-20T17:09:14.477 INFO:teuthology.orchestra.run.smithi005.stdout: "device_class_flags": {}, 2024-04-20T17:09:14.477 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": { 2024-04-20T17:09:14.477 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode_enabled": false, 2024-04-20T17:09:14.477 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_bucket_count": 0, 2024-04-20T17:09:14.477 INFO:teuthology.orchestra.run.smithi005.stdout: "degraded_stretch_mode": 0, 2024-04-20T17:09:14.477 INFO:teuthology.orchestra.run.smithi005.stdout: "recovering_stretch_mode": 0, 2024-04-20T17:09:14.477 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode_bucket": 0 2024-04-20T17:09:14.477 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:09:14.477 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:09:14.478 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 58 16 127135 127119 136.969 87.875 0.00772823 0.00729919 2024-04-20T17:09:15.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:15 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:09:15.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:15 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:09:15.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:15 smithi005 ceph-mon[39562]: pgmap v425: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.56k op/s; 4919/248276 objects misplaced (1.981%); 2.0 MiB/s, 32 objects/s recovering 2024-04-20T17:09:15.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:15 smithi005 ceph-mon[39562]: osdmap e142: 12 total, 12 up, 12 in 2024-04-20T17:09:15.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:15 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/917420824' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-20T17:09:15.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:15 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:09:15.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:15 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:09:15.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:15 smithi171 ceph-mon[44189]: pgmap v425: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.56k op/s; 4919/248276 objects misplaced (1.981%); 2.0 MiB/s, 32 objects/s recovering 2024-04-20T17:09:15.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:15 smithi171 ceph-mon[44189]: osdmap e142: 12 total, 12 up, 12 in 2024-04-20T17:09:15.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:15 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/917420824' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-20T17:09:15.393 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:09:15.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:15.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:09:15.416 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:09:15.443 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 59 16 128526 128510 136.121 86.9375 0.0100087 0.0073452 2024-04-20T17:09:15.499 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:09:15.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:15 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:09:15.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:15 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:09:15.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:15 smithi106 ceph-mon[44423]: pgmap v425: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.56k op/s; 4919/248276 objects misplaced (1.981%); 2.0 MiB/s, 32 objects/s recovering 2024-04-20T17:09:15.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:15 smithi106 ceph-mon[44423]: osdmap e142: 12 total, 12 up, 12 in 2024-04-20T17:09:15.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:15 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/917420824' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-20T17:09:15.531 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:15.556 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:09:15.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:15.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:09:15.661 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:09:15.702 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:15.730 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:09:15.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:15.735 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:09:15.809 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:09:15.836 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:09:15.873 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:09:15.873 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:09:15.921 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:16.198 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:09:16.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:16.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:09:16.308 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:09:16.339 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:16.360 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:09:16.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:16.363 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:09:16.443 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:09:16.444217 min lat: 0.00106268 max lat: 2.76125 avg lat: 0.00739449 2024-04-20T17:09:16.443 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:09:16.443 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 60 16 129822 129806 135.203 81 0.00866198 0.00739449 2024-04-20T17:09:16.463 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:09:16.508 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:16.726 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:09:16.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:16.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:09:16.774 INFO:tasks.thrashosds.thrasher:j is {'epoch': 142, 'fsid': '13c6e76c-ff37-11ee-bc93-c7b262605968', 'created': '2024-04-20T16:58:25.272557+0000', 'modified': '2024-04-20T17:09:14.029416+0000', 'last_up_change': '2024-04-20T17:06:01.788926+0000', 'last_in_change': '2024-04-20T17:05:42.849446+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-20T17:01:35.909493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.760000228881836, 'score_stable': 11.760000228881836, 'optimal_score': 0.17000000178813934, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-04-20T17:08:04.904546+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 133, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6801', 'nonce': 3901484705}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6802', 'nonce': 3901484705}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6804', 'nonce': 3901484705}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6803', 'nonce': 3901484705}]}, 'public_addr': '172.21.15.5:6801/3901484705', 'cluster_addr': '172.21.15.5:6802/3901484705', 'heartbeat_back_addr': '172.21.15.5:6804/3901484705', 'heartbeat_front_addr': '172.21.15.5:6803/3901484705', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'aed0c8eb-0e81-4e85-ab8f-a60420060656', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6805', 'nonce': 3391893914}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6806', 'nonce': 3391893914}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6808', 'nonce': 3391893914}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6807', 'nonce': 3391893914}]}, 'public_addr': '172.21.15.5:6805/3391893914', 'cluster_addr': '172.21.15.5:6806/3391893914', 'heartbeat_back_addr': '172.21.15.5:6808/3391893914', 'heartbeat_front_addr': '172.21.15.5:6807/3391893914', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f733a6a2-19c2-4c7a-af9c-04e282062bb7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 20, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6809', 'nonce': 2105468358}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6810', 'nonce': 2105468358}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6812', 'nonce': 2105468358}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6811', 'nonce': 2105468358}]}, 'public_addr': '172.21.15.5:6809/2105468358', 'cluster_addr': '172.21.15.5:6810/2105468358', 'heartbeat_back_addr': '172.21.15.5:6812/2105468358', 'heartbeat_front_addr': '172.21.15.5:6811/2105468358', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '8c92f1df-e298-4606-88ae-b8a8e79628a1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6813', 'nonce': 377652039}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6814', 'nonce': 377652039}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6816', 'nonce': 377652039}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6815', 'nonce': 377652039}]}, 'public_addr': '172.21.15.5:6813/377652039', 'cluster_addr': '172.21.15.5:6814/377652039', 'heartbeat_back_addr': '172.21.15.5:6816/377652039', 'heartbeat_front_addr': '172.21.15.5:6815/377652039', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '46df16f1-9607-4d7b-9be8-9d4236e160d1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6800', 'nonce': 1839524469}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6801', 'nonce': 1839524469}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6803', 'nonce': 1839524469}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6802', 'nonce': 1839524469}]}, 'public_addr': '172.21.15.106:6800/1839524469', 'cluster_addr': '172.21.15.106:6801/1839524469', 'heartbeat_back_addr': '172.21.15.106:6803/1839524469', 'heartbeat_front_addr': '172.21.15.106:6802/1839524469', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6603231d-1e5c-4575-bfaa-2eeca2c12215', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 132, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6804', 'nonce': 2450315908}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6805', 'nonce': 2450315908}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6807', 'nonce': 2450315908}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6806', 'nonce': 2450315908}]}, 'public_addr': '172.21.15.106:6804/2450315908', 'cluster_addr': '172.21.15.106:6805/2450315908', 'heartbeat_back_addr': '172.21.15.106:6807/2450315908', 'heartbeat_front_addr': '172.21.15.106:6806/2450315908', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '783a55a6-f176-42be-8c8b-af8481d08dc9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6808', 'nonce': 2840237913}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6809', 'nonce': 2840237913}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6811', 'nonce': 2840237913}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6810', 'nonce': 2840237913}]}, 'public_addr': '172.21.15.106:6808/2840237913', 'cluster_addr': '172.21.15.106:6809/2840237913', 'heartbeat_back_addr': '172.21.15.106:6811/2840237913', 'heartbeat_front_addr': '172.21.15.106:6810/2840237913', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6812', 'nonce': 1105371883}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6813', 'nonce': 1105371883}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6815', 'nonce': 1105371883}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6814', 'nonce': 1105371883}]}, 'public_addr': '172.21.15.106:6812/1105371883', 'cluster_addr': '172.21.15.106:6813/1105371883', 'heartbeat_back_addr': '172.21.15.106:6815/1105371883', 'heartbeat_front_addr': '172.21.15.106:6814/1105371883', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '1cb232ad-1f59-4a0a-a215-e67419ad6476', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 67, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6800', 'nonce': 4123058942}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6801', 'nonce': 4123058942}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6803', 'nonce': 4123058942}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6802', 'nonce': 4123058942}]}, 'public_addr': '172.21.15.171:6800/4123058942', 'cluster_addr': '172.21.15.171:6801/4123058942', 'heartbeat_back_addr': '172.21.15.171:6803/4123058942', 'heartbeat_front_addr': '172.21.15.171:6802/4123058942', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'b6c17147-b3dc-475b-bb20-f9f712fb99e8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 73, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6804', 'nonce': 2877589636}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6805', 'nonce': 2877589636}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6807', 'nonce': 2877589636}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6806', 'nonce': 2877589636}]}, 'public_addr': '172.21.15.171:6804/2877589636', 'cluster_addr': '172.21.15.171:6805/2877589636', 'heartbeat_back_addr': '172.21.15.171:6807/2877589636', 'heartbeat_front_addr': '172.21.15.171:6806/2877589636', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '1adcdf55-f002-4a39-91e7-378e5fae849c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 84, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6808', 'nonce': 181490621}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6809', 'nonce': 181490621}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6811', 'nonce': 181490621}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6810', 'nonce': 181490621}]}, 'public_addr': '172.21.15.171:6808/181490621', 'cluster_addr': '172.21.15.171:6809/181490621', 'heartbeat_back_addr': '172.21.15.171:6811/181490621', 'heartbeat_front_addr': '172.21.15.171:6810/181490621', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '737c5aec-694a-4968-819d-fccf22661f4f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 92, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6812', 'nonce': 3373585701}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6813', 'nonce': 3373585701}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6815', 'nonce': 3373585701}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6814', 'nonce': 3373585701}]}, 'public_addr': '172.21.15.171:6812/3373585701', 'cluster_addr': '172.21.15.171:6813/3373585701', 'heartbeat_back_addr': '172.21.15.171:6815/3373585701', 'heartbeat_front_addr': '172.21.15.171:6814/3373585701', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:08:46.110149+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:09:13.276482+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:08:14.773262+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:08:42.721685+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:09:10.189022+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:09:07.140169+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:08:21.893763+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:09:04.202179+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:08:24.889880+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:08:49.673293+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:08:16.452578+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:08:47.110274+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.4', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '2.4', 'osds': [0, 5]}], 'primary_temp': [], 'blocklist': {'172.21.15.5:0/3881726425': '2024-04-21T16:59:25.808337+0000', '172.21.15.5:0/338649532': '2024-04-21T16:59:25.808337+0000', '172.21.15.5:0/1227843664': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:6800/1876064631': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:0/160689449': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:6800/3521715286': '2024-04-21T16:59:25.808337+0000', '172.21.15.5:0/3841663462': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:0/687690919': '2024-04-21T16:59:25.808337+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-20T17:09:16.775 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:09:16.841 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:09:16.903 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:17.253 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-20T17:09:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:17 smithi171 ceph-mon[44189]: pgmap v427: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s; 4741/251660 objects misplaced (1.884%); 1.9 MiB/s, 29 objects/s recovering 2024-04-20T17:09:17.400 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:17 smithi005 ceph-mon[39562]: pgmap v427: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s; 4741/251660 objects misplaced (1.884%); 1.9 MiB/s, 29 objects/s recovering 2024-04-20T17:09:17.443 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 61 16 131135 131119 134.331 82.0625 0.0251828 0.00744239 2024-04-20T17:09:17.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:17 smithi106 ceph-mon[44423]: pgmap v427: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s; 4741/251660 objects misplaced (1.884%); 1.9 MiB/s, 29 objects/s recovering 2024-04-20T17:09:17.718 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:09:17.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:17.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:09:17.826 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:09:17.858 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:17.889 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:09:17.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:17.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:09:17.993 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:09:18.050 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:18.258 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:09:18.262 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:09:18.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:18.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:09:18.340 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:09:18.340 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:09:18.374 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:09:18.411 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:18.437 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:09:18.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:18.441 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:09:18.443 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 62 16 132436 132420 133.476 81.3125 0.0101428 0.00749032 2024-04-20T17:09:18.541 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:09:18.601 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:18.865 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:09:18.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:18.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:09:18.974 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:09:19.032 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:19.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:19 smithi005 ceph-mon[39562]: pgmap v428: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s; 4741/257544 objects misplaced (1.841%); 1.6 MiB/s, 24 objects/s recovering 2024-04-20T17:09:19.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:19 smithi171 ceph-mon[44189]: pgmap v428: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s; 4741/257544 objects misplaced (1.841%); 1.6 MiB/s, 24 objects/s recovering 2024-04-20T17:09:19.443 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 63 16 133767 133751 132.678 83.1875 0.0113493 0.00753541 2024-04-20T17:09:19.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:19 smithi106 ceph-mon[44423]: pgmap v428: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s; 4741/257544 objects misplaced (1.841%); 1.6 MiB/s, 24 objects/s recovering 2024-04-20T17:09:20.332 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:20 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3782235166' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:09:20.332 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:20.332 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:20.332 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:20.443 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 64 15 135106 135091 131.913 83.75 0.00620115 0.00757868 2024-04-20T17:09:20.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:20 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3782235166' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:09:20.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:20 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3782235166' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:09:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:20.715 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:09:20.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:20.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:09:20.822 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:09:20.852 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:20.877 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:09:20.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:20.881 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:09:20.983 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:09:21.018 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:21.039 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:09:21.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:21.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:09:21.146 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:09:21.186 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:21.186 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:21.186 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:21.186 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:21.186 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:21.186 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:21.187 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:21.187 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:21 smithi171 ceph-mon[44189]: pgmap v429: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s; 4546/262572 objects misplaced (1.731%); 2.9 MiB/s, 46 objects/s recovering 2024-04-20T17:09:21.187 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:21.187 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:21.189 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:21.210 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:09:21.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:21.213 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:09:21.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:21.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:21 smithi106 ceph-mon[44423]: pgmap v429: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s; 4546/262572 objects misplaced (1.731%); 2.9 MiB/s, 46 objects/s recovering 2024-04-20T17:09:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:21.277 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:21.277 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:21.277 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:21.277 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:21.277 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:21.277 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:21.277 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:21.277 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:21 smithi005 ceph-mon[39562]: pgmap v429: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s; 4546/262572 objects misplaced (1.731%); 2.9 MiB/s, 46 objects/s recovering 2024-04-20T17:09:21.277 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:21.277 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:21.314 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:09:21.354 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:21.394 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:09:21.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:21.403 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:09:21.443 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 65 16 136468 136452 131.192 85.0625 0.00484633 0.00762096 2024-04-20T17:09:21.475 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:09:21.475 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:09:21.475 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:09:21.475 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:09:21.475 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:09:21.476 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:09:21.476 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:09:21.476 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:09:21.476 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:09:21.476 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:09:21.476 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:09:21.476 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:09:21.476 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:09:21.476 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:09:21.476 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:09:21.476 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:09:21.476 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:09:21.476 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:09:21.477 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:09:21.477 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:09:21.477 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:09:21.477 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:09:21.477 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:09:21.477 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:09:21.503 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:09:21.532 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:21.554 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:09:21.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:21.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:09:21.657 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:09:21.689 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:21.718 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:09:21.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:21.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:09:21.827 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:09:21.863 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:21.891 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:09:21.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:21.895 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:09:21.996 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:09:22.030 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:22.062 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:09:22.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:22.069 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:09:22.170 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:09:22.221 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:22.350 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:09:22.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:22.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:09:22.443 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 66 16 137802 137786 130.467 83.375 0.0154054 0.00766322 2024-04-20T17:09:22.460 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:09:22.489 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:22.522 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:09:22.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:22.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:09:22.627 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:09:22.661 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:22.681 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:09:22.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:22.685 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:09:22.786 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:09:22.834 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:22.865 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:09:22.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:22.871 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:09:22.972 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:09:23.004 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:23.028 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:09:23.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:23.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:09:23.136 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:09:23.151 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:09:23.163 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:23.197 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:09:23.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:23.200 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:09:23.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:23 smithi106 ceph-mon[44423]: pgmap v430: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s; 4546/266700 objects misplaced (1.705%); 2.4 MiB/s, 37 objects/s recovering 2024-04-20T17:09:23.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:23 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2163463307' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:09:23.301 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:09:23.335 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:23.366 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:09:23.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:23.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:09:23.443 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 67 16 139075 139059 129.708 79.5625 0.00702802 0.00770811 2024-04-20T17:09:23.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:23 smithi005 ceph-mon[39562]: pgmap v430: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s; 4546/266700 objects misplaced (1.705%); 2.4 MiB/s, 37 objects/s recovering 2024-04-20T17:09:23.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:23 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2163463307' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:09:23.470 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:09:23.500 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:23.526 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:09:23.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:23.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:09:23.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:23 smithi171 ceph-mon[44189]: pgmap v430: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s; 4546/266700 objects misplaced (1.705%); 2.4 MiB/s, 37 objects/s recovering 2024-04-20T17:09:23.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:23 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2163463307' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:09:23.633 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:09:23.674 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:24.277 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:09:24.443 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 68 16 140409 140393 129.026 83.375 0.0186429 0.00774837 2024-04-20T17:09:24.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:24 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:09:24.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:24 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2163463307' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:09:24.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:24 smithi005 ceph-mon[39562]: osdmap e143: 12 total, 12 up, 12 in 2024-04-20T17:09:24.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:24 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:09:24.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:24 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2163463307' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:09:24.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:24 smithi106 ceph-mon[44423]: osdmap e143: 12 total, 12 up, 12 in 2024-04-20T17:09:24.542 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:09:24.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:24.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:09:24.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:24 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:09:24.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:24 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2163463307' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:09:24.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:24 smithi171 ceph-mon[44189]: osdmap e143: 12 total, 12 up, 12 in 2024-04-20T17:09:24.654 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:09:24.687 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:24.709 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:09:24.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:24.713 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:09:24.814 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:09:24.865 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:24.961 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:09:24.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:24.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:09:25.067 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:09:25.110 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:25.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:25 smithi005 ceph-mon[39562]: pgmap v432: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s; 4546/275046 objects misplaced (1.653%); 2.4 MiB/s, 37 objects/s recovering 2024-04-20T17:09:25.240 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:09:25.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:25.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:09:25.347 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:09:25.373 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:25 smithi106 ceph-mon[44423]: pgmap v432: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s; 4546/275046 objects misplaced (1.653%); 2.4 MiB/s, 37 objects/s recovering 2024-04-20T17:09:25.375 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:25.411 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:09:25.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:25.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:09:25.443 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 69 16 141677 141661 128.305 79.25 0.00468113 0.00779226 2024-04-20T17:09:25.515 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:09:25.583 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:25.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:25 smithi171 ceph-mon[44189]: pgmap v432: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s; 4546/275046 objects misplaced (1.653%); 2.4 MiB/s, 37 objects/s recovering 2024-04-20T17:09:25.771 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:09:25.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:25.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:09:25.875 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:09:25.908 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:25.938 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:09:25.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:25.944 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:09:25.965 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:09:25.965 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:09:25.965 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:09:25.965 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:09:25.965 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:09:26.045 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:09:26.093 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:26.119 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:09:26.138 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:09:26.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:26.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:09:26.149 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:09:26.245 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:09:26.282 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:26.309 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:09:26.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:26.311 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:09:26.412 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:09:26.443 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 70 15 142910 142895 127.573 77.125 0.00586477 0.00783686 2024-04-20T17:09:26.468 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:26.580 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:09:26.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:26.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:09:26.688 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:09:26.731 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:09:26.745 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:27.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:26 smithi171 ceph-mon[44189]: osdmap e144: 12 total, 12 up, 12 in 2024-04-20T17:09:27.099 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:26 smithi005 ceph-mon[39562]: osdmap e144: 12 total, 12 up, 12 in 2024-04-20T17:09:27.105 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:09:27.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:27.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:09:27.147 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":434,"stamp":"2024-04-20T17:09:25.952460+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":9118745120,"num_objects":139134,"num_object_clones":0,"num_object_copies":278268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4356,"num_objects_unfound":0,"num_objects_dirty":139134,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":139313,"num_write_kb":8907386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1271,"num_bytes_recovered":84887680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139269,"ondisk_log_size":139269,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":18428860,"kb_used_data":17829836,"kb_used_omap":18,"kb_used_meta":598893,"kb_avail":1106512964,"statfs":{"total":1151940427776,"available":1133069275136,"internally_reserved":0,"allocated":18257752064,"data_stored":18251005596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":613266745},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,6,8,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":679084032,"num_objects":10362,"num_object_clones":0,"num_object_copies":20724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-385,"num_objects_unfound":0,"num_objects_dirty":10362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10362,"num_write_kb":663168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":385,"num_bytes_recovered":25231360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001592"},"pg_stats":[{"pgid":"2.f","version":"143'8738","reported_seq":8873,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-20T17:09:23.625034+0000","last_change":"2024-04-20T17:09:12.070221+0000","last_active":"2024-04-20T17:09:23.625034+0000","last_peered":"2024-04-20T17:09:23.625034+0000","last_clean":"2024-04-20T17:09:23.625034+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:09:23.625034+0000","last_undegraded":"2024-04-20T17:09:23.625034+0000","last_fullsized":"2024-04-20T17:09:23.625034+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":6799,"log_size":8738,"log_dups_size":0,"ondisk_log_size":8738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":572653568,"num_objects":8738,"num_object_clones":0,"num_object_copies":17476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8738,"num_write_kb":559232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"143'8778","reported_seq":11371,"reported_epoch":143,"state":"active+recovering+remapped","last_fresh":"2024-04-20T17:09:24.729394+0000","last_change":"2024-04-20T17:08:50.616759+0000","last_active":"2024-04-20T17:09:24.729394+0000","last_peered":"2024-04-20T17:09:24.729394+0000","last_clean":"2024-04-20T17:08:47.829425+0000","last_became_active":"2024-04-20T17:08:50.590271+0000","last_became_peered":"2024-04-20T17:08:50.590271+0000","last_unstale":"2024-04-20T17:09:24.729394+0000","last_undegraded":"2024-04-20T17:09:24.729394+0000","last_fullsized":"2024-04-20T17:09:24.729394+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":8778,"log_dups_size":0,"ondisk_log_size":8778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575275008,"num_objects":8778,"num_object_clones":0,"num_object_copies":17556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4356,"num_objects_unfound":0,"num_objects_dirty":8778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8778,"num_write_kb":561792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1263,"num_bytes_recovered":82771968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[0,5],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":4356},{"shards":"0,5","objects":4422}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"143'8953","reported_seq":9018,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-20T17:09:25.574889+0000","last_change":"2024-04-20T17:08:06.013628+0000","last_active":"2024-04-20T17:09:25.574889+0000","last_peered":"2024-04-20T17:09:25.574889+0000","last_clean":"2024-04-20T17:09:25.574889+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:09:25.574889+0000","last_undegraded":"2024-04-20T17:09:25.574889+0000","last_fullsized":"2024-04-20T17:09:25.574889+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":8953,"log_dups_size":0,"ondisk_log_size":8953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586743808,"num_objects":8953,"num_object_clones":0,"num_object_copies":17906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8953,"num_write_kb":572992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"143'8713","reported_seq":8862,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-20T17:09:23.615243+0000","last_change":"2024-04-20T17:09:09.239641+0000","last_active":"2024-04-20T17:09:23.615243+0000","last_peered":"2024-04-20T17:09:23.615243+0000","last_clean":"2024-04-20T17:09:23.615243+0000","last_became_active":"2024-04-20T17:08:06.027804+0000","last_became_peered":"2024-04-20T17:08:06.027804+0000","last_unstale":"2024-04-20T17:09:23.615243+0000","last_undegraded":"2024-04-20T17:09:23.615243+0000","last_fullsized":"2024-04-20T17:09:23.615243+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'7483","last_scrub_stamp":"2024-04-20T17:09:09.239533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:09:09.239533+0000","objects_scrubbed":7410,"log_size":8713,"log_dups_size":0,"ondisk_log_size":8713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:10:21.574586+0000","scrub_duration":1594,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":571015168,"num_objects":8713,"num_object_clones":0,"num_object_copies":17426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8713,"num_write_kb":557632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"143'8705","reported_seq":8770,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-20T17:09:25.554848+0000","last_change":"2024-04-20T17:08:06.010896+0000","last_active":"2024-04-20T17:09:25.554848+0000","last_peered":"2024-04-20T17:09:25.554848+0000","last_clean":"2024-04-20T17:09:25.554848+0000","last_became_active":"2024-04-20T17:08:06.010691+0000","last_became_peered":"2024-04-20T17:08:06.010691+0000","last_unstale":"2024-04-20T17:09:25.554848+0000","last_undegraded":"2024-04-20T17:09:25.554848+0000","last_fullsized":"2024-04-20T17:09:25.554848+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":8705,"log_dups_size":0,"ondisk_log_size":8705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570490880,"num_objects":8705,"num_object_clones":0,"num_object_copies":17410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8705,"num_write_kb":557120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":436,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-20T17:09:14.049207+0000","last_change":"2024-04-20T17:08:43.762640+0000","last_active":"2024-04-20T17:09:14.049207+0000","last_peered":"2024-04-20T17:09:14.049207+0000","last_clean":"2024-04-20T17:09:14.049207+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:09:14.049207+0000","last_undegraded":"2024-04-20T17:09:14.049207+0000","last_fullsized":"2024-04-20T17:09:14.049207+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:08:43.762505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:08:43.762505+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:09:49.835061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"142'8706","reported_seq":8769,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-20T17:09:22.772717+0000","last_change":"2024-04-20T17:08:06.045012+0000","last_active":"2024-04-20T17:09:22.772717+0000","last_peered":"2024-04-20T17:09:22.772717+0000","last_clean":"2024-04-20T17:09:22.772717+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:09:22.772717+0000","last_undegraded":"2024-04-20T17:09:22.772717+0000","last_fullsized":"2024-04-20T17:09:22.772717+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":8706,"log_dups_size":0,"ondisk_log_size":8706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:09:31.502776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570556416,"num_objects":8706,"num_object_clones":0,"num_object_copies":17412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8706,"num_write_kb":557184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"143'8811","reported_seq":8876,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-20T17:09:23.602987+0000","last_change":"2024-04-20T17:08:06.003337+0000","last_active":"2024-04-20T17:09:23.602987+0000","last_peered":"2024-04-20T17:09:23.602987+0000","last_clean":"2024-04-20T17:09:23.602987+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:09:23.602987+0000","last_undegraded":"2024-04-20T17:09:23.602987+0000","last_fullsized":"2024-04-20T17:09:23.602987+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":8811,"log_dups_size":0,"ondisk_log_size":8811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:09:27.121093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":577437696,"num_objects":8811,"num_object_clones":0,"num_object_copies":17622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8811,"num_write_kb":563904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"142'8536","reported_seq":8599,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-20T17:09:21.648205+0000","last_change":"2024-04-20T17:08:06.009697+0000","last_active":"2024-04-20T17:09:21.648205+0000","last_peered":"2024-04-20T17:09:21.648205+0000","last_clean":"2024-04-20T17:09:21.648205+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:09:21.648205+0000","last_undegraded":"2024-04-20T17:09:21.648205+0000","last_fullsized":"2024-04-20T17:09:21.648205+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":8536,"log_dups_size":0,"ondisk_log_size":8536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":559415296,"num_objects":8536,"num_object_clones":0,"num_object_copies":17072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8536,"num_write_kb":546304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"142'8654","reported_seq":8717,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-20T17:09:22.401773+0000","last_change":"2024-04-20T17:08:06.045085+0000","last_active":"2024-04-20T17:09:22.401773+0000","last_peered":"2024-04-20T17:09:22.401773+0000","last_clean":"2024-04-20T17:09:22.401773+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:09:22.401773+0000","last_undegraded":"2024-04-20T17:09:22.401773+0000","last_fullsized":"2024-04-20T17:09:22.401773+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":8654,"log_dups_size":0,"ondisk_log_size":8654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:09:26.473268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":567148544,"num_objects":8654,"num_object_clones":0,"num_object_copies":17308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8654,"num_write_kb":553856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"142'8604","reported_seq":8667,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-20T17:09:22.800988+0000","last_change":"2024-04-20T17:08:06.006061+0000","last_active":"2024-04-20T17:09:22.800988+0000","last_peered":"2024-04-20T17:09:22.800988+0000","last_clean":"2024-04-20T17:09:22.800988+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:09:22.800988+0000","last_undegraded":"2024-04-20T17:09:22.800988+0000","last_fullsized":"2024-04-20T17:09:22.800988+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":8604,"log_dups_size":0,"ondisk_log_size":8604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:09:28.073363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":563871744,"num_objects":8604,"num_object_clones":0,"num_object_copies":17208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8604,"num_write_kb":550656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"142'8633","reported_seq":8695,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-20T17:09:21.684171+0000","last_change":"2024-04-20T17:08:06.011184+0000","last_active":"2024-04-20T17:09:21.684171+0000","last_peered":"2024-04-20T17:09:21.684171+0000","last_clean":"2024-04-20T17:09:21.684171+0000","last_became_active":"2024-04-20T17:08:06.010879+0000","last_became_peered":"2024-04-20T17:08:06.010879+0000","last_unstale":"2024-04-20T17:09:21.684171+0000","last_undegraded":"2024-04-20T17:09:21.684171+0000","last_fullsized":"2024-04-20T17:09:21.684171+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":8633,"log_dups_size":0,"ondisk_log_size":8633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:09:26.988436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":565772288,"num_objects":8633,"num_object_clones":0,"num_object_copies":17266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8633,"num_write_kb":552512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"142'8739","reported_seq":8802,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-20T17:09:22.493033+0000","last_change":"2024-04-20T17:08:06.028253+0000","last_active":"2024-04-20T17:09:22.493033+0000","last_peered":"2024-04-20T17:09:22.493033+0000","last_clean":"2024-04-20T17:09:22.493033+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:09:22.493033+0000","last_undegraded":"2024-04-20T17:09:22.493033+0000","last_fullsized":"2024-04-20T17:09:22.493033+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":8739,"log_dups_size":0,"ondisk_log_size":8739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:09:27.384592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":572719104,"num_objects":8739,"num_object_clones":0,"num_object_copies":17478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8739,"num_write_kb":559296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"143'8555","reported_seq":8620,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-20T17:09:23.616347+0000","last_change":"2024-04-20T17:08:06.005814+0000","last_active":"2024-04-20T17:09:23.616347+0000","last_peered":"2024-04-20T17:09:23.616347+0000","last_clean":"2024-04-20T17:09:23.616347+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:09:23.616347+0000","last_undegraded":"2024-04-20T17:09:23.616347+0000","last_fullsized":"2024-04-20T17:09:23.616347+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":8555,"log_dups_size":0,"ondisk_log_size":8555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:09:32.391282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":560660480,"num_objects":8555,"num_object_clones":0,"num_object_copies":17110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8555,"num_write_kb":547520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"142'8570","reported_seq":8633,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-04-20T17:09:22.478363+0000","last_change":"2024-04-20T17:08:06.006951+0000","last_active":"2024-04-20T17:09:22.478363+0000","last_peered":"2024-04-20T17:09:22.478363+0000","last_clean":"2024-04-20T17:09:22.478363+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:09:22.478363+0000","last_undegraded":"2024-04-20T17:09:22.478363+0000","last_fullsized":"2024-04-20T17:09:22.478363+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":8570,"log_dups_size":0,"ondisk_log_size":8570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:09:25.563784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561643520,"num_objects":8570,"num_object_clones":0,"num_object_copies":17140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8570,"num_write_kb":548480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"143'8785","reported_seq":8850,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-20T17:09:25.567449+0000","last_change":"2024-04-20T17:08:06.044934+0000","last_active":"2024-04-20T17:09:25.567449+0000","last_peered":"2024-04-20T17:09:25.567449+0000","last_clean":"2024-04-20T17:09:25.567449+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:09:25.567449+0000","last_undegraded":"2024-04-20T17:09:25.567449+0000","last_fullsized":"2024-04-20T17:09:25.567449+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":8785,"log_dups_size":0,"ondisk_log_size":8785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:09:29.479643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575733760,"num_objects":8785,"num_object_clones":0,"num_object_copies":17570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8785,"num_write_kb":562240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"143'8652","reported_seq":8717,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-04-20T17:09:23.618678+0000","last_change":"2024-04-20T17:08:06.028545+0000","last_active":"2024-04-20T17:09:23.618678+0000","last_peered":"2024-04-20T17:09:23.618678+0000","last_clean":"2024-04-20T17:09:23.618678+0000","last_became_active":"2024-04-20T17:08:06.028272+0000","last_became_peered":"2024-04-20T17:08:06.028272+0000","last_unstale":"2024-04-20T17:09:23.618678+0000","last_undegraded":"2024-04-20T17:09:23.618678+0000","last_fullsized":"2024-04-20T17:09:23.618678+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":8652,"log_dups_size":0,"ondisk_log_size":8652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:09:31.827174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":567017472,"num_objects":8652,"num_object_clones":0,"num_object_copies":17304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8652,"num_write_kb":553728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":9118154752,"num_objects":139132,"num_object_clones":0,"num_object_copies":278264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4356,"num_objects_unfound":0,"num_objects_dirty":139132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":139132,"num_write_kb":8904448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1263,"num_bytes_recovered":82771968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18331664384,"data_stored":18331664384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139132,"ondisk_log_size":139132,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991276,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1693968,"kb_used_data":1648904,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":92051184,"statfs":{"total":95995035648,"available":94260412416,"internally_reserved":0,"allocated":1688477696,"data_stored":1687915032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":46135748},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777252913,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1166216,"kb_used_data":1116992,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92578936,"statfs":{"total":95995035648,"available":94800830464,"internally_reserved":0,"allocated":1143799808,"data_stored":1143240696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612664,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1747288,"kb_used_data":1698060,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":91997864,"statfs":{"total":95995035648,"available":94205812736,"internally_reserved":0,"allocated":1738813440,"data_stored":1738248304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762808893,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2288392,"kb_used_data":2216960,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91456760,"statfs":{"total":95995035648,"available":93651722240,"internally_reserved":0,"allocated":2270167040,"data_stored":2269607948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168642,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1702800,"kb_used_data":1653576,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92042352,"statfs":{"total":95995035648,"available":94251368448,"internally_reserved":0,"allocated":1693261824,"data_stored":1692699180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430278,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":580360,"kb_used_data":549184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93164792,"statfs":{"total":95995035648,"available":95400747008,"internally_reserved":0,"allocated":562364416,"data_stored":561805324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574944,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2057224,"kb_used_data":2003840,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":91687928,"statfs":{"total":95995035648,"available":93888438272,"internally_reserved":0,"allocated":2051932160,"data_stored":2051373068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738471,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1729688,"kb_used_data":1684620,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":92015464,"statfs":{"total":95995035648,"available":94223835136,"internally_reserved":0,"allocated":1725050880,"data_stored":1724485744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":20,"seq":85899346010,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1712216,"kb_used_data":1658828,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92032936,"statfs":{"total":95995035648,"available":94241726464,"internally_reserved":0,"allocated":1698639872,"data_stored":1698074736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182485,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1693192,"kb_used_data":1635648,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92051960,"statfs":{"total":95995035648,"available":94261207040,"internally_reserved":0,"allocated":1674903552,"data_stored":1674344460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986257,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":585240,"kb_used_data":558220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93159912,"statfs":{"total":95995035648,"available":95395749888,"internally_reserved":0,"allocated":571617280,"data_stored":571052144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790027,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1472276,"kb_used_data":1405004,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92272876,"statfs":{"total":95995035648,"available":94487425024,"internally_reserved":0,"allocated":1438724096,"data_stored":1438158960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1732509696,"data_stored":1732509696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2061107200,"data_stored":2061107200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1699020800,"data_stored":1699020800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1674248192,"data_stored":1674248192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":575340544,"data_stored":575340544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1454178304,"data_stored":1454178304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":567148544,"data_stored":567148544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1707671552,"data_stored":1707671552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":2280783872,"data_stored":2280783872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1738670080,"data_stored":1738670080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1152188416,"data_stored":1152188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1688797184,"data_stored":1688797184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:09:27.148 INFO:tasks.thrashosds.thrasher:Setting 2.3 to [2, 10, 5, 0] 2024-04-20T17:09:27.148 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.3', '2', '10', '5', '0'] 2024-04-20T17:09:27.149 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pg-upmap-items 2.3 2 10 5 0 2024-04-20T17:09:27.213 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:09:27.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:26 smithi106 ceph-mon[44423]: osdmap e144: 12 total, 12 up, 12 in 2024-04-20T17:09:27.280 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:27.444 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 71 16 144163 144147 126.879 78.25 0.00565815 0.00788023 2024-04-20T17:09:27.526 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:09:27.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:27.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:09:27.561 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-20T17:09:27.581 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:09:27.594 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:09:27.636 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:09:27.665 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:09:27.748 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:27.837 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:27 smithi005 ceph-mon[39562]: pgmap v434: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s; 4356/278268 objects misplaced (1.565%); 3.0 MiB/s, 48 objects/s recovering 2024-04-20T17:09:27.837 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:27 smithi005 ceph-mon[39562]: from='client.24815 v1:172.21.15.5:0/3278841992' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:09:27.985 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:09:27.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:27.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:09:28.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:27 smithi171 ceph-mon[44189]: pgmap v434: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s; 4356/278268 objects misplaced (1.565%); 3.0 MiB/s, 48 objects/s recovering 2024-04-20T17:09:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:27 smithi171 ceph-mon[44189]: from='client.24815 v1:172.21.15.5:0/3278841992' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:09:28.094 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:09:28.147 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:28.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:27 smithi106 ceph-mon[44423]: pgmap v434: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s; 4356/278268 objects misplaced (1.565%); 3.0 MiB/s, 48 objects/s recovering 2024-04-20T17:09:28.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:27 smithi106 ceph-mon[44423]: from='client.24815 v1:172.21.15.5:0/3278841992' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:09:28.299 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:09:28.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:28.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:09:28.410 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:09:28.443 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:28.444 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 72 16 145337 145321 126.136 73.375 0.0132283 0.00792676 2024-04-20T17:09:28.472 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:09:28.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:28.475 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:09:28.576 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:09:28.607 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:28.636 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:09:28.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:28.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:09:28.742 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:09:28.779 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:28.807 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:09:28.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:28.819 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:09:28.841 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:28 smithi005 ceph-mon[39562]: pgmap v435: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s; 4356/283876 objects misplaced (1.534%); 1.5 MiB/s, 23 objects/s recovering 2024-04-20T17:09:28.841 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:28 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:09:28.841 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:28 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:09:28.919 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:09:28.968 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:28.994 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:09:28.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:28.997 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:09:29.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:28 smithi106 ceph-mon[44423]: pgmap v435: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s; 4356/283876 objects misplaced (1.534%); 1.5 MiB/s, 23 objects/s recovering 2024-04-20T17:09:29.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:28 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:09:29.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:28 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:09:29.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:28 smithi171 ceph-mon[44189]: pgmap v435: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s; 4356/283876 objects misplaced (1.534%); 1.5 MiB/s, 23 objects/s recovering 2024-04-20T17:09:29.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:28 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:09:29.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:28 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:09:29.098 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:09:29.159 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:29.432 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:09:29.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:29.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:09:29.444 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 73 16 146631 146615 125.515 80.875 0.0456577 0.00796493 2024-04-20T17:09:29.537 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:09:29.572 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:29.595 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:09:29.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:29.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:09:29.701 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:09:29.734 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:29.761 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:09:29.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:29.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:09:29.844 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:29 smithi106 ceph-mon[44423]: osdmap e145: 12 total, 12 up, 12 in 2024-04-20T17:09:29.867 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:09:29.921 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:29 smithi005 ceph-mon[39562]: osdmap e145: 12 total, 12 up, 12 in 2024-04-20T17:09:30.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:29 smithi171 ceph-mon[44189]: osdmap e145: 12 total, 12 up, 12 in 2024-04-20T17:09:30.444 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 74 16 147915 147899 124.904 80.25 0.00892522 0.00800436 2024-04-20T17:09:30.485 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:09:30.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:30.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:09:30.512 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:09:30.600 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:09:30.640 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:30.668 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:09:30.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:30.672 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:09:30.772 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:09:30.814 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:30.844 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:09:30.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:30.848 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:09:30.860 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:30 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:09:30.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:30 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:09:30.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:30 smithi005 ceph-mon[39562]: osdmap e146: 12 total, 12 up, 12 in 2024-04-20T17:09:30.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:30 smithi005 ceph-mon[39562]: pgmap v438: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s; 4174/288546 objects misplaced (1.447%); 3.9 MiB/s, 61 objects/s recovering 2024-04-20T17:09:30.949 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:09:31.012 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:31.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:30 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:09:31.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:30 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:09:31.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:30 smithi171 ceph-mon[44189]: osdmap e146: 12 total, 12 up, 12 in 2024-04-20T17:09:31.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:30 smithi171 ceph-mon[44189]: pgmap v438: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s; 4174/288546 objects misplaced (1.447%); 3.9 MiB/s, 61 objects/s recovering 2024-04-20T17:09:31.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:30 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:09:31.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:30 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:09:31.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:30 smithi106 ceph-mon[44423]: osdmap e146: 12 total, 12 up, 12 in 2024-04-20T17:09:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:30 smithi106 ceph-mon[44423]: pgmap v438: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s; 4174/288546 objects misplaced (1.447%); 3.9 MiB/s, 61 objects/s recovering 2024-04-20T17:09:31.316 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:09:31.366 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:09:31.407 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:09:31.433 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:09:31.444 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 75 16 149078 149062 124.207 72.6875 0.0052521 0.008049 2024-04-20T17:09:31.482 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:09:31.508 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:09:31.533 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:09:31.563 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:09:31.595 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:09:31.626 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:09:31.665 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:09:31.693 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:09:31.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:31.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:09:31.714 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:09:31.733 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:09:31.733 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:09:31.733 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:09:31.733 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:09:31.733 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:09:31.733 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:09:31.733 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:09:31.733 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:09:31.733 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:09:31.733 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:09:31.733 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:09:31.733 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:09:31.798 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:09:31.829 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:31.848 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:09:31.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:31.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:09:31.952 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:09:31.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:31 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/449132355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:09:31.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:31.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:31.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:31.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:31.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:31.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:32.007 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:32.044 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:09:32.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:32.050 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:09:32.151 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:09:32.176 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:31 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/449132355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:09:32.176 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:32.176 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:32.177 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:32.177 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:32.177 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:32.177 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:32.178 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:32.199 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:09:32.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:32.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:09:32.304 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:09:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:31 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/449132355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:09:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:32.364 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:32.425 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:09:32.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:32.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:09:32.444 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 76 16 150326 150310 123.599 78 0.00597343 0.00808923 2024-04-20T17:09:32.533 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:09:32.563 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:32.588 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:09:32.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:32.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:09:32.692 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:09:32.723 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:32.753 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:09:32.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:32.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:09:32.856 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:09:32.935 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:33.016 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:09:33.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:33.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:09:33.124 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:09:33.157 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:33.158 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:32 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:33.158 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:32 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:33.158 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:32 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:33.158 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:32 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:33.158 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:32 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:33.159 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:32 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:33.159 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:32 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:09:33.159 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:32 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:09:33.159 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:32 smithi106 ceph-mon[44423]: pgmap v439: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s; 4174/292474 objects misplaced (1.427%); 1.9 MiB/s, 29 objects/s recovering 2024-04-20T17:09:33.177 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:09:33.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:33.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:09:33.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:32 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:33.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:32 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:33.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:32 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:32 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:32 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:32 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:32 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:09:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:32 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:09:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:32 smithi005 ceph-mon[39562]: pgmap v439: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s; 4174/292474 objects misplaced (1.427%); 1.9 MiB/s, 29 objects/s recovering 2024-04-20T17:09:33.280 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:09:33.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:32 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:32 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:32 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:32 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:32 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:32 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:09:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:32 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:09:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:32 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:09:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:32 smithi171 ceph-mon[44189]: pgmap v439: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 8.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s; 4174/292474 objects misplaced (1.427%); 1.9 MiB/s, 29 objects/s recovering 2024-04-20T17:09:33.337 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:33.427 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:09:33.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:33.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:09:33.444 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 77 16 151505 151489 122.951 73.6875 0.0254823 0.00813181 2024-04-20T17:09:33.534 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:09:33.568 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:33.596 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:09:33.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:33.601 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:09:33.702 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:09:33.747 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:33.872 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:09:33.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:33.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:09:33.923 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:09:33.986 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:09:34.024 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:34.050 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:09:34.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:34.053 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:09:34.154 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:09:34.181 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:33 smithi106 ceph-mon[44423]: osdmap e147: 12 total, 12 up, 12 in 2024-04-20T17:09:34.181 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:33 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2746487278' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:09:34.181 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:33 smithi106 ceph-mon[44423]: from='client.24817 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:09:34.183 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:34.204 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:09:34.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:34.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:09:34.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:33 smithi005 ceph-mon[39562]: osdmap e147: 12 total, 12 up, 12 in 2024-04-20T17:09:34.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:33 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2746487278' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:09:34.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:33 smithi005 ceph-mon[39562]: from='client.24817 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:09:34.308 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:09:34.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:33 smithi171 ceph-mon[44189]: osdmap e147: 12 total, 12 up, 12 in 2024-04-20T17:09:34.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:33 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2746487278' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:09:34.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:33 smithi171 ceph-mon[44189]: from='client.24817 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:09:34.355 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:34.444 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 78 16 152720 152704 122.348 75.9375 0.0144365 0.00817227 2024-04-20T17:09:34.575 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:09:34.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:34.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:09:34.682 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:09:34.724 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:35.127 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:09:35.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:35.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:09:35.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:34 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:09:35.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:34 smithi005 ceph-mon[39562]: from='client.24817 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:09:35.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:34 smithi005 ceph-mon[39562]: osdmap e148: 12 total, 12 up, 12 in 2024-04-20T17:09:35.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:34 smithi005 ceph-mon[39562]: pgmap v442: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 9.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s; 4174/300228 objects misplaced (1.390%); 2.2 MiB/s, 35 objects/s recovering 2024-04-20T17:09:35.234 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:09:35.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:34 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:09:35.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:34 smithi106 ceph-mon[44423]: from='client.24817 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:09:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:34 smithi106 ceph-mon[44423]: osdmap e148: 12 total, 12 up, 12 in 2024-04-20T17:09:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:34 smithi106 ceph-mon[44423]: pgmap v442: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 9.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s; 4174/300228 objects misplaced (1.390%); 2.2 MiB/s, 35 objects/s recovering 2024-04-20T17:09:35.296 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:35.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:34 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:09:35.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:34 smithi171 ceph-mon[44189]: from='client.24817 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:09:35.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:34 smithi171 ceph-mon[44189]: osdmap e148: 12 total, 12 up, 12 in 2024-04-20T17:09:35.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:34 smithi171 ceph-mon[44189]: pgmap v442: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 9.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s; 4174/300228 objects misplaced (1.390%); 2.2 MiB/s, 35 objects/s recovering 2024-04-20T17:09:35.341 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:09:35.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:35.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:09:35.444 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 79 16 153841 153825 121.686 70.0625 0.0200384 0.0082164 2024-04-20T17:09:35.461 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:09:35.507 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:35.685 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:09:35.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:35.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:09:35.796 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:09:35.839 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:35.905 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:09:35.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:35 smithi005 ceph-mon[39562]: 2.9 scrub starts 2024-04-20T17:09:35.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:35 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1312464970' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.3", "id": [2, 10, 5, 0]}]: dispatch 2024-04-20T17:09:35.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:35 smithi005 ceph-mon[39562]: from='client.24830 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.3", "id": [2, 10, 5, 0]}]: dispatch 2024-04-20T17:09:35.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:35 smithi005 ceph-mon[39562]: 2.f scrub starts 2024-04-20T17:09:36.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:35 smithi106 ceph-mon[44423]: 2.9 scrub starts 2024-04-20T17:09:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:35 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1312464970' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.3", "id": [2, 10, 5, 0]}]: dispatch 2024-04-20T17:09:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:35 smithi106 ceph-mon[44423]: from='client.24830 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.3", "id": [2, 10, 5, 0]}]: dispatch 2024-04-20T17:09:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:35 smithi106 ceph-mon[44423]: 2.f scrub starts 2024-04-20T17:09:36.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:35 smithi171 ceph-mon[44189]: 2.9 scrub starts 2024-04-20T17:09:36.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:35 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1312464970' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.3", "id": [2, 10, 5, 0]}]: dispatch 2024-04-20T17:09:36.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:35 smithi171 ceph-mon[44189]: from='client.24830 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.3", "id": [2, 10, 5, 0]}]: dispatch 2024-04-20T17:09:36.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:35 smithi171 ceph-mon[44189]: 2.f scrub starts 2024-04-20T17:09:36.445 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:09:36.445965 min lat: 0.00106268 max lat: 2.76125 avg lat: 0.0082438 2024-04-20T17:09:36.445 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:09:36.445 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 80 16 154598 154582 120.757 47.3125 0.00483465 0.0082438 2024-04-20T17:09:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:36 smithi005 ceph-mon[39562]: from='client.24830 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.3", "id": [2, 10, 5, 0]}]': finished 2024-04-20T17:09:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:36 smithi005 ceph-mon[39562]: osdmap e149: 12 total, 12 up, 12 in 2024-04-20T17:09:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:36 smithi005 ceph-mon[39562]: pgmap v444: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 9.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s; 3971/303158 objects misplaced (1.310%); 2.1 MiB/s, 33 objects/s recovering 2024-04-20T17:09:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:36 smithi005 ceph-mon[39562]: 2.f scrub ok 2024-04-20T17:09:37.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:36 smithi106 ceph-mon[44423]: from='client.24830 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.3", "id": [2, 10, 5, 0]}]': finished 2024-04-20T17:09:37.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:36 smithi106 ceph-mon[44423]: osdmap e149: 12 total, 12 up, 12 in 2024-04-20T17:09:37.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:36 smithi106 ceph-mon[44423]: pgmap v444: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 9.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s; 3971/303158 objects misplaced (1.310%); 2.1 MiB/s, 33 objects/s recovering 2024-04-20T17:09:37.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:36 smithi106 ceph-mon[44423]: 2.f scrub ok 2024-04-20T17:09:37.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:36 smithi171 ceph-mon[44189]: from='client.24830 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.3", "id": [2, 10, 5, 0]}]': finished 2024-04-20T17:09:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:36 smithi171 ceph-mon[44189]: osdmap e149: 12 total, 12 up, 12 in 2024-04-20T17:09:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:36 smithi171 ceph-mon[44189]: pgmap v444: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 9.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s; 3971/303158 objects misplaced (1.310%); 2.1 MiB/s, 33 objects/s recovering 2024-04-20T17:09:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:36 smithi171 ceph-mon[44189]: 2.f scrub ok 2024-04-20T17:09:37.444 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 81 16 154598 154582 119.266 0 - 0.0082438 2024-04-20T17:09:37.721 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:09:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:37 smithi005 ceph-mon[39562]: 2.9 scrub ok 2024-04-20T17:09:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:37 smithi005 ceph-mon[39562]: 2.e scrub starts 2024-04-20T17:09:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:37 smithi005 ceph-mon[39562]: osdmap e150: 12 total, 12 up, 12 in 2024-04-20T17:09:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:37 smithi005 ceph-mon[39562]: 2.5 scrub starts 2024-04-20T17:09:38.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:37 smithi106 ceph-mon[44423]: 2.9 scrub ok 2024-04-20T17:09:38.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:37 smithi106 ceph-mon[44423]: 2.e scrub starts 2024-04-20T17:09:38.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:37 smithi106 ceph-mon[44423]: osdmap e150: 12 total, 12 up, 12 in 2024-04-20T17:09:38.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:37 smithi106 ceph-mon[44423]: 2.5 scrub starts 2024-04-20T17:09:38.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:37 smithi171 ceph-mon[44189]: 2.9 scrub ok 2024-04-20T17:09:38.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:37 smithi171 ceph-mon[44189]: 2.e scrub starts 2024-04-20T17:09:38.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:37 smithi171 ceph-mon[44189]: osdmap e150: 12 total, 12 up, 12 in 2024-04-20T17:09:38.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:37 smithi171 ceph-mon[44189]: 2.5 scrub starts 2024-04-20T17:09:38.445 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 82 16 154598 154582 117.811 0 - 0.0082438 2024-04-20T17:09:39.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:38 smithi005 ceph-mon[39562]: 2.e scrub ok 2024-04-20T17:09:39.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:39 smithi005 ceph-mon[39562]: 2.7 scrub starts 2024-04-20T17:09:39.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:39 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:09:39.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:39 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:09:39.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:39 smithi005 ceph-mon[39562]: osdmap e151: 12 total, 12 up, 12 in 2024-04-20T17:09:39.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:39 smithi005 ceph-mon[39562]: pgmap v447: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 9.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 827 op/s; 3971/306940 objects misplaced (1.294%); 3.1 MiB/s, 50 objects/s recovering 2024-04-20T17:09:39.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:39 smithi005 ceph-mon[39562]: 2.5 scrub ok 2024-04-20T17:09:39.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:39 smithi005 ceph-mon[39562]: 2.b scrub starts 2024-04-20T17:09:39.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:39 smithi005 ceph-mon[39562]: 2.7 scrub ok 2024-04-20T17:09:39.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:39 smithi005 ceph-mon[39562]: 2.a scrub starts 2024-04-20T17:09:39.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:38 smithi106 ceph-mon[44423]: 2.e scrub ok 2024-04-20T17:09:39.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:38 smithi106 ceph-mon[44423]: 2.7 scrub starts 2024-04-20T17:09:39.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:38 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:09:39.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:38 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:09:39.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:38 smithi106 ceph-mon[44423]: osdmap e151: 12 total, 12 up, 12 in 2024-04-20T17:09:39.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:38 smithi106 ceph-mon[44423]: pgmap v447: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 9.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 827 op/s; 3971/306940 objects misplaced (1.294%); 3.1 MiB/s, 50 objects/s recovering 2024-04-20T17:09:39.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:38 smithi106 ceph-mon[44423]: 2.5 scrub ok 2024-04-20T17:09:39.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:38 smithi106 ceph-mon[44423]: 2.b scrub starts 2024-04-20T17:09:39.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:38 smithi106 ceph-mon[44423]: 2.7 scrub ok 2024-04-20T17:09:39.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:38 smithi106 ceph-mon[44423]: 2.a scrub starts 2024-04-20T17:09:39.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:38 smithi171 ceph-mon[44189]: 2.e scrub ok 2024-04-20T17:09:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:38 smithi171 ceph-mon[44189]: 2.7 scrub starts 2024-04-20T17:09:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:38 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:09:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:38 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:09:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:38 smithi171 ceph-mon[44189]: osdmap e151: 12 total, 12 up, 12 in 2024-04-20T17:09:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:38 smithi171 ceph-mon[44189]: pgmap v447: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 9.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 827 op/s; 3971/306940 objects misplaced (1.294%); 3.1 MiB/s, 50 objects/s recovering 2024-04-20T17:09:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:38 smithi171 ceph-mon[44189]: 2.5 scrub ok 2024-04-20T17:09:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:38 smithi171 ceph-mon[44189]: 2.b scrub starts 2024-04-20T17:09:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:38 smithi171 ceph-mon[44189]: 2.7 scrub ok 2024-04-20T17:09:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:38 smithi171 ceph-mon[44189]: 2.a scrub starts 2024-04-20T17:09:39.448 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 83 16 155151 155135 116.804 11.5208 0.0064772 0.00855886 2024-04-20T17:09:40.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:39 smithi005 ceph-mon[39562]: 2.8 scrub starts 2024-04-20T17:09:40.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:39 smithi005 ceph-mon[39562]: 2.b scrub ok 2024-04-20T17:09:40.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:39 smithi005 ceph-mon[39562]: 2.c scrub starts 2024-04-20T17:09:40.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:39 smithi106 ceph-mon[44423]: 2.8 scrub starts 2024-04-20T17:09:40.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:39 smithi106 ceph-mon[44423]: 2.b scrub ok 2024-04-20T17:09:40.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:39 smithi106 ceph-mon[44423]: 2.c scrub starts 2024-04-20T17:09:40.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:39 smithi171 ceph-mon[44189]: 2.8 scrub starts 2024-04-20T17:09:40.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:39 smithi171 ceph-mon[44189]: 2.b scrub ok 2024-04-20T17:09:40.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:39 smithi171 ceph-mon[44189]: 2.c scrub starts 2024-04-20T17:09:40.448 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 84 16 155907 155891 115.976 47.25 0.00769695 0.00861816 2024-04-20T17:09:40.843 INFO:teuthology.orchestra.run.smithi005.stderr:set 2.3 pg_upmap_items mapping to [2->10,5->0] 2024-04-20T17:09:41.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:40 smithi005 ceph-mon[39562]: 2.6 scrub starts 2024-04-20T17:09:41.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:40 smithi005 ceph-mon[39562]: 2.0 scrub starts 2024-04-20T17:09:41.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:40 smithi005 ceph-mon[39562]: 2.a scrub ok 2024-04-20T17:09:41.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:40 smithi005 ceph-mon[39562]: pgmap v448: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 1 activating+degraded+remapped, 14 active+clean; 9.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 738 op/s; 9436/309094 objects degraded (3.053%); 13025/309094 objects misplaced (4.214%); 6.1 MiB/s, 97 objects/s recovering 2024-04-20T17:09:41.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:40 smithi005 ceph-mon[39562]: osdmap e152: 12 total, 12 up, 12 in 2024-04-20T17:09:41.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:40 smithi106 ceph-mon[44423]: 2.6 scrub starts 2024-04-20T17:09:41.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:40 smithi106 ceph-mon[44423]: 2.0 scrub starts 2024-04-20T17:09:41.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:40 smithi106 ceph-mon[44423]: 2.a scrub ok 2024-04-20T17:09:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:40 smithi106 ceph-mon[44423]: pgmap v448: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 1 activating+degraded+remapped, 14 active+clean; 9.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 738 op/s; 9436/309094 objects degraded (3.053%); 13025/309094 objects misplaced (4.214%); 6.1 MiB/s, 97 objects/s recovering 2024-04-20T17:09:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:40 smithi106 ceph-mon[44423]: osdmap e152: 12 total, 12 up, 12 in 2024-04-20T17:09:41.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:40 smithi171 ceph-mon[44189]: 2.6 scrub starts 2024-04-20T17:09:41.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:40 smithi171 ceph-mon[44189]: 2.0 scrub starts 2024-04-20T17:09:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:40 smithi171 ceph-mon[44189]: 2.a scrub ok 2024-04-20T17:09:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:40 smithi171 ceph-mon[44189]: pgmap v448: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 1 activating+degraded+remapped, 14 active+clean; 9.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 738 op/s; 9436/309094 objects degraded (3.053%); 13025/309094 objects misplaced (4.214%); 6.1 MiB/s, 97 objects/s recovering 2024-04-20T17:09:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:40 smithi171 ceph-mon[44189]: osdmap e152: 12 total, 12 up, 12 in 2024-04-20T17:09:41.448 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 85 16 156664 156648 115.168 47.3125 0.014464 0.0086809 2024-04-20T17:09:42.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:41 smithi005 ceph-mon[39562]: 2.8 scrub ok 2024-04-20T17:09:42.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:41 smithi005 ceph-mon[39562]: Health check failed: Degraded data redundancy: 9436/309094 objects degraded (3.053%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:09:42.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:41 smithi106 ceph-mon[44423]: 2.8 scrub ok 2024-04-20T17:09:42.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:41 smithi106 ceph-mon[44423]: Health check failed: Degraded data redundancy: 9436/309094 objects degraded (3.053%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:09:42.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:41 smithi171 ceph-mon[44189]: 2.8 scrub ok 2024-04-20T17:09:42.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:41 smithi171 ceph-mon[44189]: Health check failed: Degraded data redundancy: 9436/309094 objects degraded (3.053%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:09:42.448 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 86 16 157546 157530 114.47 55.125 0.00767209 0.00873227 2024-04-20T17:09:43.134 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:09:43.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:43.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:09:43.244 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:09:43.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:42 smithi106 ceph-mon[44423]: 2.0 scrub ok 2024-04-20T17:09:43.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:42 smithi106 ceph-mon[44423]: pgmap v450: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering+remapped, 1 activating+degraded+remapped, 13 active+clean; 9.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 590 op/s; 9436/310244 objects degraded (3.041%); 13025/310244 objects misplaced (4.198%); 4.0 MiB/s, 63 objects/s recovering 2024-04-20T17:09:43.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:42 smithi106 ceph-mon[44423]: 2.c scrub ok 2024-04-20T17:09:43.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:42 smithi106 ceph-mon[44423]: 2.6 scrub ok 2024-04-20T17:09:43.268 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:42 smithi171 ceph-mon[44189]: 2.0 scrub ok 2024-04-20T17:09:43.268 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:42 smithi171 ceph-mon[44189]: pgmap v450: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering+remapped, 1 activating+degraded+remapped, 13 active+clean; 9.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 590 op/s; 9436/310244 objects degraded (3.041%); 13025/310244 objects misplaced (4.198%); 4.0 MiB/s, 63 objects/s recovering 2024-04-20T17:09:43.268 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:42 smithi171 ceph-mon[44189]: 2.c scrub ok 2024-04-20T17:09:43.268 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:42 smithi171 ceph-mon[44189]: 2.6 scrub ok 2024-04-20T17:09:43.270 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:43.305 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:09:43.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:43.310 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:09:43.411 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:09:43.448 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 87 16 158414 158398 113.778 54.25 0.00940343 0.00878708 2024-04-20T17:09:43.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:42 smithi005 ceph-mon[39562]: 2.0 scrub ok 2024-04-20T17:09:43.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:42 smithi005 ceph-mon[39562]: pgmap v450: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering+remapped, 1 activating+degraded+remapped, 13 active+clean; 9.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 590 op/s; 9436/310244 objects degraded (3.041%); 13025/310244 objects misplaced (4.198%); 4.0 MiB/s, 63 objects/s recovering 2024-04-20T17:09:43.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:42 smithi005 ceph-mon[39562]: 2.c scrub ok 2024-04-20T17:09:43.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:42 smithi005 ceph-mon[39562]: 2.6 scrub ok 2024-04-20T17:09:43.460 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:44.332 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:09:44.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:44 smithi171 ceph-mon[44189]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9436/310244 objects degraded (3.041%), 1 pg degraded) 2024-04-20T17:09:44.334 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:44 smithi005 ceph-mon[39562]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9436/310244 objects degraded (3.041%), 1 pg degraded) 2024-04-20T17:09:44.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:44.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:09:44.445 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:09:44.448 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 88 16 159213 159197 113.052 49.9375 0.0178927 0.00884382 2024-04-20T17:09:44.473 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:44 smithi106 ceph-mon[44423]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9436/310244 objects degraded (3.041%), 1 pg degraded) 2024-04-20T17:09:44.475 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:44.499 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:09:44.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:44.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:09:44.606 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:09:44.667 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:45.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:45 smithi171 ceph-mon[44189]: 2.d scrub starts 2024-04-20T17:09:45.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:45 smithi171 ceph-mon[44189]: pgmap v451: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+remapped, 14 active+clean; 9.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 765 op/s; 22347/315164 objects misplaced (7.091%); 3.9 MiB/s, 62 objects/s recovering 2024-04-20T17:09:45.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:09:45.449 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 89 16 160123 160107 112.421 56.875 0.00596388 0.00889216 2024-04-20T17:09:45.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:45 smithi005 ceph-mon[39562]: 2.d scrub starts 2024-04-20T17:09:45.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:45 smithi005 ceph-mon[39562]: pgmap v451: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+remapped, 14 active+clean; 9.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 765 op/s; 22347/315164 objects misplaced (7.091%); 3.9 MiB/s, 62 objects/s recovering 2024-04-20T17:09:45.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:09:45.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:45 smithi106 ceph-mon[44423]: 2.d scrub starts 2024-04-20T17:09:45.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:45 smithi106 ceph-mon[44423]: pgmap v451: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+remapped, 14 active+clean; 9.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 765 op/s; 22347/315164 objects misplaced (7.091%); 3.9 MiB/s, 62 objects/s recovering 2024-04-20T17:09:45.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:09:46.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:46 smithi171 ceph-mon[44189]: 2.d scrub ok 2024-04-20T17:09:46.448 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 90 15 160991 160976 111.775 54.3125 0.0166089 0.00894242 2024-04-20T17:09:46.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:46 smithi005 ceph-mon[39562]: 2.d scrub ok 2024-04-20T17:09:46.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:46 smithi106 ceph-mon[44423]: 2.d scrub ok 2024-04-20T17:09:47.449 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Total time run: 90.0262 2024-04-20T17:09:47.450 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Total writes made: 160991 2024-04-20T17:09:47.450 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Write size: 65536 2024-04-20T17:09:47.450 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Object size: 65536 2024-04-20T17:09:47.450 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Bandwidth (MB/sec): 111.767 2024-04-20T17:09:47.450 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Stddev Bandwidth: 67.1861 2024-04-20T17:09:47.450 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Max bandwidth (MB/sec): 357.375 2024-04-20T17:09:47.450 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Min bandwidth (MB/sec): 0 2024-04-20T17:09:47.450 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Average IOPS: 1788 2024-04-20T17:09:47.450 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Stddev IOPS: 1074.99 2024-04-20T17:09:47.450 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Max IOPS: 5718 2024-04-20T17:09:47.450 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Min IOPS: 0 2024-04-20T17:09:47.450 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Average Latency(s): 0.00894642 2024-04-20T17:09:47.450 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Stddev Latency(s): 0.0391358 2024-04-20T17:09:47.450 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Max latency(s): 2.85259 2024-04-20T17:09:47.451 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Min latency(s): 0.00106268 2024-04-20T17:09:47.454 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Cleaning up (deleting benchmark objects) 2024-04-20T17:09:48.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:47 smithi171 ceph-mon[44189]: pgmap v452: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+remapped, 14 active+clean; 9.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 606 op/s; 22099/316688 objects misplaced (6.978%); 5.3 MiB/s, 85 objects/s recovering 2024-04-20T17:09:48.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:47 smithi005 ceph-mon[39562]: pgmap v452: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+remapped, 14 active+clean; 9.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 606 op/s; 22099/316688 objects misplaced (6.978%); 5.3 MiB/s, 85 objects/s recovering 2024-04-20T17:09:48.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:47 smithi106 ceph-mon[44423]: pgmap v452: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+remapped, 14 active+clean; 9.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 606 op/s; 22099/316688 objects misplaced (6.978%); 5.3 MiB/s, 85 objects/s recovering 2024-04-20T17:09:49.377 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:09:49.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:49.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:09:49.401 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:09:49.484 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:09:49.533 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:49.556 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:09:49.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:49.559 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:09:49.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:49 smithi171 ceph-mon[44189]: pgmap v453: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 648 op/s; 22099/319688 objects misplaced (6.913%); 4.3 MiB/s, 68 objects/s recovering 2024-04-20T17:09:49.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:49 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:09:49.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:49 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:09:49.659 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:09:49.710 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:49.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:49 smithi005 ceph-mon[39562]: pgmap v453: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 648 op/s; 22099/319688 objects misplaced (6.913%); 4.3 MiB/s, 68 objects/s recovering 2024-04-20T17:09:49.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:49 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:09:49.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:49 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:09:49.737 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:09:49.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:49.741 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:09:49.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:49 smithi106 ceph-mon[44423]: pgmap v453: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 648 op/s; 22099/319688 objects misplaced (6.913%); 4.3 MiB/s, 68 objects/s recovering 2024-04-20T17:09:49.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:49 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:09:49.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:49 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:09:49.841 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:09:49.871 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:49.900 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:09:49.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:49.907 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:09:50.008 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:09:50.041 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:50.062 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:09:50.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:50.065 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:09:50.166 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:09:50.192 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:50.212 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:09:50.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:50.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:09:50.316 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:09:50.343 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:50.362 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:09:50.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:50.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:09:50.466 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:09:50.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:50 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:09:50.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:50 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:09:50.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:50 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:09:50.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:50 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:09:50.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:50 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:09:50.518 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:50 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:09:50.518 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:50 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:09:50.518 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:50 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:09:50.518 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:50 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:09:50.518 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:50 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:09:50.528 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:50.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:50 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:09:50.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:50 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:09:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:50 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:09:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:50 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:09:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:50 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:09:50.896 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:09:50.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:50.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:09:51.008 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:09:51.035 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:51.060 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:09:51.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:51.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:09:51.164 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:09:51.215 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:51.378 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:09:51.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:51.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:09:51.487 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:09:51.535 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:51.596 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:09:51.596 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:09:51.939 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:51 smithi005 ceph-mon[39562]: pgmap v454: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 703 op/s; 21601/320514 objects misplaced (6.739%); 4.5 MiB/s, 71 objects/s recovering 2024-04-20T17:09:52.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:51 smithi106 ceph-mon[44423]: pgmap v454: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 703 op/s; 21601/320514 objects misplaced (6.739%); 4.5 MiB/s, 71 objects/s recovering 2024-04-20T17:09:52.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:51 smithi171 ceph-mon[44189]: pgmap v454: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 703 op/s; 21601/320514 objects misplaced (6.739%); 4.5 MiB/s, 71 objects/s recovering 2024-04-20T17:09:52.307 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:09:52.310 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:09:52.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:52.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:09:52.356 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-20T17:09:52.417 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:09:52.445 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:52.467 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:09:52.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:52.472 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:09:52.572 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:09:52.651 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:53.635 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:09:53.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:53.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:09:53.648 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:09:53.738 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:09:53.746 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:09:53.776 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:53.795 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:09:53.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:53.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:09:53.899 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:09:53.931 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:53.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:53 smithi005 ceph-mon[39562]: pgmap v455: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 91 B/s rd, 35 MiB/s wr, 681 op/s; 21601/319158 objects misplaced (6.768%); 4.0 MiB/s, 64 objects/s recovering 2024-04-20T17:09:53.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:53 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3562080735' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:09:53.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:53 smithi005 ceph-mon[39562]: from='client.24833 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:09:53.963 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:09:53.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:53.966 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:09:54.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:53 smithi106 ceph-mon[44423]: pgmap v455: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 91 B/s rd, 35 MiB/s wr, 681 op/s; 21601/319158 objects misplaced (6.768%); 4.0 MiB/s, 64 objects/s recovering 2024-04-20T17:09:54.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:53 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3562080735' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:09:54.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:53 smithi106 ceph-mon[44423]: from='client.24833 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:09:54.067 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:09:54.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:53 smithi171 ceph-mon[44189]: pgmap v455: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 91 B/s rd, 35 MiB/s wr, 681 op/s; 21601/319158 objects misplaced (6.768%); 4.0 MiB/s, 64 objects/s recovering 2024-04-20T17:09:54.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:53 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3562080735' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:09:54.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:53 smithi171 ceph-mon[44189]: from='client.24833 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:09:54.095 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:54.115 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:09:54.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:54.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:09:54.222 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:09:54.295 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:09:54.353 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-20T17:09:54.353 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:09:54.353 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-20T17:09:54.354 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-20T17:09:55.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:54 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:09:55.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:54 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:09:55.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:54 smithi106 ceph-mon[44423]: from='client.24833 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:09:55.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:54 smithi106 ceph-mon[44423]: osdmap e153: 12 total, 12 up, 12 in 2024-04-20T17:09:55.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:54 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:09:55.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:54 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:09:55.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:54 smithi171 ceph-mon[44189]: from='client.24833 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:09:55.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:54 smithi171 ceph-mon[44189]: osdmap e153: 12 total, 12 up, 12 in 2024-04-20T17:09:55.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:54 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:09:55.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:54 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:09:55.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:54 smithi005 ceph-mon[39562]: from='client.24833 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:09:55.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:54 smithi005 ceph-mon[39562]: osdmap e153: 12 total, 12 up, 12 in 2024-04-20T17:09:56.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:55 smithi106 ceph-mon[44423]: pgmap v457: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 21 MiB/s wr, 756 op/s; 21197/313672 objects misplaced (6.758%); 5.4 MiB/s, 86 objects/s recovering 2024-04-20T17:09:56.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:55 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:09:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:55 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:09:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:55 smithi106 ceph-mon[44423]: osdmap e154: 12 total, 12 up, 12 in 2024-04-20T17:09:56.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:55 smithi171 ceph-mon[44189]: pgmap v457: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 21 MiB/s wr, 756 op/s; 21197/313672 objects misplaced (6.758%); 5.4 MiB/s, 86 objects/s recovering 2024-04-20T17:09:56.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:55 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:09:56.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:55 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:09:56.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:55 smithi171 ceph-mon[44189]: osdmap e154: 12 total, 12 up, 12 in 2024-04-20T17:09:56.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:55 smithi005 ceph-mon[39562]: pgmap v457: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 21 MiB/s wr, 756 op/s; 21197/313672 objects misplaced (6.758%); 5.4 MiB/s, 86 objects/s recovering 2024-04-20T17:09:56.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:55 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:09:56.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:55 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:09:56.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:55 smithi005 ceph-mon[39562]: osdmap e154: 12 total, 12 up, 12 in 2024-04-20T17:09:58.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:57 smithi171 ceph-mon[44189]: pgmap v459: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 8.5 MiB/s wr, 737 op/s; 20889/312270 objects misplaced (6.689%); 7.2 MiB/s, 115 objects/s recovering 2024-04-20T17:09:58.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:57 smithi005 ceph-mon[39562]: pgmap v459: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 8.5 MiB/s wr, 737 op/s; 20889/312270 objects misplaced (6.689%); 7.2 MiB/s, 115 objects/s recovering 2024-04-20T17:09:58.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:57 smithi106 ceph-mon[44423]: pgmap v459: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 8.5 MiB/s wr, 737 op/s; 20889/312270 objects misplaced (6.689%); 7.2 MiB/s, 115 objects/s recovering 2024-04-20T17:09:59.841 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:09:59 smithi005 ceph-mon[39562]: pgmap v460: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 592 KiB/s wr, 743 op/s; 20889/308916 objects misplaced (6.762%); 4.0 MiB/s, 63 objects/s recovering 2024-04-20T17:09:59.926 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:09:59.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:09:59.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:10:00.031 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:10:00.079 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:00.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:09:59 smithi106 ceph-mon[44423]: pgmap v460: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 592 KiB/s wr, 743 op/s; 20889/308916 objects misplaced (6.762%); 4.0 MiB/s, 63 objects/s recovering 2024-04-20T17:10:00.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:09:59 smithi171 ceph-mon[44189]: pgmap v460: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 592 KiB/s wr, 743 op/s; 20889/308916 objects misplaced (6.762%); 4.0 MiB/s, 63 objects/s recovering 2024-04-20T17:10:00.445 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:10:00.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:00.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:10:00.555 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:10:00.586 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:00.608 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:10:00.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:00.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:10:00.713 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:10:00.742 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:00.770 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:10:00.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:00.774 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:10:00.875 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:10:00.912 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:00.948 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:10:00.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:00.954 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:10:01.001 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:00 smithi005 ceph-mon[39562]: overall HEALTH_OK 2024-04-20T17:10:01.054 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:10:01.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:00 smithi171 ceph-mon[44189]: overall HEALTH_OK 2024-04-20T17:10:01.106 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:01.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:00 smithi106 ceph-mon[44423]: overall HEALTH_OK 2024-04-20T17:10:01.289 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:10:01.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:01.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:10:01.401 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:10:01.473 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:10:01.474 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:10:01.515 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:01.525 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:10:01.623 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:10:01.623 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:10:01.623 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:10:01.623 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:10:01.623 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:10:01.623 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:10:01.623 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:10:01.642 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:10:01.702 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:10:01.736 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:10:01.769 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:10:01.805 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:10:01.817 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:10:01.817 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:10:01.817 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:10:01.817 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:10:01.817 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:10:01.817 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:10:01.817 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:10:01.817 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:10:01.817 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:10:01.817 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:10:01.817 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:10:01.818 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:10:01.850 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:01 smithi005 ceph-mon[39562]: pgmap v461: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 864 op/s; 20046/305326 objects misplaced (6.565%); 8.6 MiB/s, 137 objects/s recovering 2024-04-20T17:10:01.850 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:01 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/498720215' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:10:01.850 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:01.850 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:01.850 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:01.850 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:02.169 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:10:02.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:02.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:10:02.189 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:10:02.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:01 smithi106 ceph-mon[44423]: pgmap v461: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 864 op/s; 20046/305326 objects misplaced (6.565%); 8.6 MiB/s, 137 objects/s recovering 2024-04-20T17:10:02.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:01 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/498720215' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:10:02.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:02.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:02.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:02.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:02.274 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:10:02.304 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:01 smithi171 ceph-mon[44189]: pgmap v461: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 864 op/s; 20046/305326 objects misplaced (6.565%); 8.6 MiB/s, 137 objects/s recovering 2024-04-20T17:10:02.304 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:01 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/498720215' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:10:02.304 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:02.304 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:02.305 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:02.305 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:02.307 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:02.332 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:10:02.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:02.337 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:10:02.437 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:10:02.495 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:02.715 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:10:02.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:02.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:10:02.819 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:10:02.851 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:02.853 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:02.853 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:02.853 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:02.853 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:02.853 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:02.853 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:02.853 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:02.853 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:02.885 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:10:02.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:02.888 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:10:02.989 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:10:03.017 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:03.040 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:10:03.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:03.045 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:10:03.146 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:10:03.201 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:03.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:03.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:03.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:03.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:03.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:03.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:03.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:03.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:03.283 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:10:03.283 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:10:03.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:03.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:10:03.329 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-20T17:10:03.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:03.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:03.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:03.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:03.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:03.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:03.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:03.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:03.392 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:10:03.478 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:03.969 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:10:03.973 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:03 smithi005 ceph-mon[39562]: pgmap v462: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 20046/303278 objects misplaced (6.610%); 8.4 MiB/s, 133 objects/s recovering 2024-04-20T17:10:03.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:03.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:10:04.076 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:10:04.104 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:03 smithi106 ceph-mon[44423]: pgmap v462: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 20046/303278 objects misplaced (6.610%); 8.4 MiB/s, 133 objects/s recovering 2024-04-20T17:10:04.105 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:04.125 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:10:04.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:04.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:10:04.229 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:10:04.281 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:04.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:03 smithi171 ceph-mon[44189]: pgmap v462: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 20046/303278 objects misplaced (6.610%); 8.4 MiB/s, 133 objects/s recovering 2024-04-20T17:10:04.340 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:10:04.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:04.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:10:04.447 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:10:04.490 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:04.735 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:10:05.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:04 smithi005 ceph-mon[39562]: pgmap v463: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 781 op/s; 19538/298048 objects misplaced (6.555%); 7.2 MiB/s, 115 objects/s recovering 2024-04-20T17:10:05.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:04 smithi106 ceph-mon[44423]: pgmap v463: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 781 op/s; 19538/298048 objects misplaced (6.555%); 7.2 MiB/s, 115 objects/s recovering 2024-04-20T17:10:05.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:04 smithi171 ceph-mon[44189]: pgmap v463: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 781 op/s; 19538/298048 objects misplaced (6.555%); 7.2 MiB/s, 115 objects/s recovering 2024-04-20T17:10:06.076 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:10:06.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:06.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:10:06.186 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:10:06.228 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:06.264 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:10:06.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:06.268 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:10:06.369 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:10:06.421 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:06.582 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:10:06.584 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:10:06.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:06.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:10:06.690 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:10:06.718 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:06.739 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:10:06.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:06.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:10:06.843 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:10:06.870 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:06.890 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:10:06.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:06.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:10:06.994 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:10:07.022 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:07.044 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:10:07.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:07.047 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:10:07.148 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:10:07.195 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:07.281 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:10:07.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:07.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:10:07.348 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:10:07.348 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:10:07.348 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 154, 2024-04-20T17:10:07.348 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "13c6e76c-ff37-11ee-bc93-c7b262605968", 2024-04-20T17:10:07.348 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-04-20T16:58:25.272557+0000", 2024-04-20T17:10:07.348 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-04-20T17:09:54.738652+0000", 2024-04-20T17:10:07.348 INFO:teuthology.orchestra.run.smithi005.stdout: "last_up_change": "2024-04-20T17:06:01.788926+0000", 2024-04-20T17:10:07.348 INFO:teuthology.orchestra.run.smithi005.stdout: "last_in_change": "2024-04-20T17:05:42.849446+0000", 2024-04-20T17:10:07.348 INFO:teuthology.orchestra.run.smithi005.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-20T17:10:07.348 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_num": 5799936, 2024-04-20T17:10:07.348 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_set": [ 2024-04-20T17:10:07.349 INFO:teuthology.orchestra.run.smithi005.stdout: "pglog_hardlimit", 2024-04-20T17:10:07.349 INFO:teuthology.orchestra.run.smithi005.stdout: "purged_snapdirs", 2024-04-20T17:10:07.349 INFO:teuthology.orchestra.run.smithi005.stdout: "recovery_deletes", 2024-04-20T17:10:07.349 INFO:teuthology.orchestra.run.smithi005.stdout: "sortbitwise" 2024-04-20T17:10:07.349 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:10:07.349 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_version": 27, 2024-04-20T17:10:07.349 INFO:teuthology.orchestra.run.smithi005.stdout: "full_ratio": 0.94999998807907104, 2024-04-20T17:10:07.349 INFO:teuthology.orchestra.run.smithi005.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-20T17:10:07.349 INFO:teuthology.orchestra.run.smithi005.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-20T17:10:07.349 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_snapshot": "", 2024-04-20T17:10:07.349 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_max": 2, 2024-04-20T17:10:07.349 INFO:teuthology.orchestra.run.smithi005.stdout: "max_osd": 12, 2024-04-20T17:10:07.349 INFO:teuthology.orchestra.run.smithi005.stdout: "require_min_compat_client": "luminous", 2024-04-20T17:10:07.349 INFO:teuthology.orchestra.run.smithi005.stdout: "min_compat_client": "luminous", 2024-04-20T17:10:07.349 INFO:teuthology.orchestra.run.smithi005.stdout: "require_osd_release": "squid", 2024-04-20T17:10:07.350 INFO:teuthology.orchestra.run.smithi005.stdout: "allow_crimson": false, 2024-04-20T17:10:07.350 INFO:teuthology.orchestra.run.smithi005.stdout: "pools": [ 2024-04-20T17:10:07.350 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.350 INFO:teuthology.orchestra.run.smithi005.stdout: "pool": 1, 2024-04-20T17:10:07.350 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_name": ".mgr", 2024-04-20T17:10:07.350 INFO:teuthology.orchestra.run.smithi005.stdout: "create_time": "2024-04-20T17:01:35.909493+0000", 2024-04-20T17:10:07.350 INFO:teuthology.orchestra.run.smithi005.stdout: "flags": 1, 2024-04-20T17:10:07.350 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_names": "hashpspool", 2024-04-20T17:10:07.350 INFO:teuthology.orchestra.run.smithi005.stdout: "type": 1, 2024-04-20T17:10:07.350 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 2, 2024-04-20T17:10:07.350 INFO:teuthology.orchestra.run.smithi005.stdout: "min_size": 2, 2024-04-20T17:10:07.350 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_rule": 0, 2024-04-20T17:10:07.350 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_count": 0, 2024-04-20T17:10:07.351 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_target": 0, 2024-04-20T17:10:07.351 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T17:10:07.351 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T17:10:07.351 INFO:teuthology.orchestra.run.smithi005.stdout: "object_hash": 2, 2024-04-20T17:10:07.351 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_autoscale_mode": "off", 2024-04-20T17:10:07.351 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num": 1, 2024-04-20T17:10:07.351 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num": 1, 2024-04-20T17:10:07.351 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num_target": 1, 2024-04-20T17:10:07.351 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_target": 1, 2024-04-20T17:10:07.351 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_pending": 1, 2024-04-20T17:10:07.351 INFO:teuthology.orchestra.run.smithi005.stdout: "last_pg_merge_meta": { 2024-04-20T17:10:07.351 INFO:teuthology.orchestra.run.smithi005.stdout: "source_pgid": "0.0", 2024-04-20T17:10:07.351 INFO:teuthology.orchestra.run.smithi005.stdout: "ready_epoch": 0, 2024-04-20T17:10:07.351 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_started": 0, 2024-04-20T17:10:07.351 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_clean": 0, 2024-04-20T17:10:07.352 INFO:teuthology.orchestra.run.smithi005.stdout: "source_version": "0'0", 2024-04-20T17:10:07.352 INFO:teuthology.orchestra.run.smithi005.stdout: "target_version": "0'0" 2024-04-20T17:10:07.352 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.352 INFO:teuthology.orchestra.run.smithi005.stdout: "last_change": "16", 2024-04-20T17:10:07.352 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend": "0", 2024-04-20T17:10:07.352 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-20T17:10:07.352 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_preluminous": "0", 2024-04-20T17:10:07.352 INFO:teuthology.orchestra.run.smithi005.stdout: "auid": 0, 2024-04-20T17:10:07.352 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_mode": "selfmanaged", 2024-04-20T17:10:07.352 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_seq": 0, 2024-04-20T17:10:07.352 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_epoch": 0, 2024-04-20T17:10:07.352 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_snaps": [], 2024-04-20T17:10:07.352 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_snaps": "[]", 2024-04-20T17:10:07.352 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_bytes": 0, 2024-04-20T17:10:07.352 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_objects": 0, 2024-04-20T17:10:07.352 INFO:teuthology.orchestra.run.smithi005.stdout: "tiers": [], 2024-04-20T17:10:07.353 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_of": -1, 2024-04-20T17:10:07.353 INFO:teuthology.orchestra.run.smithi005.stdout: "read_tier": -1, 2024-04-20T17:10:07.353 INFO:teuthology.orchestra.run.smithi005.stdout: "write_tier": -1, 2024-04-20T17:10:07.353 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_mode": "none", 2024-04-20T17:10:07.353 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_bytes": 0, 2024-04-20T17:10:07.353 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_objects": 0, 2024-04-20T17:10:07.353 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T17:10:07.353 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T17:10:07.353 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T17:10:07.353 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_flush_age": 0, 2024-04-20T17:10:07.353 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_evict_age": 0, 2024-04-20T17:10:07.353 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_profile": "", 2024-04-20T17:10:07.353 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_params": { 2024-04-20T17:10:07.354 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "none" 2024-04-20T17:10:07.354 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.354 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_period": 0, 2024-04-20T17:10:07.354 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_count": 0, 2024-04-20T17:10:07.354 INFO:teuthology.orchestra.run.smithi005.stdout: "use_gmt_hitset": true, 2024-04-20T17:10:07.354 INFO:teuthology.orchestra.run.smithi005.stdout: "min_read_recency_for_promote": 0, 2024-04-20T17:10:07.354 INFO:teuthology.orchestra.run.smithi005.stdout: "min_write_recency_for_promote": 0, 2024-04-20T17:10:07.354 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T17:10:07.354 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_search_last_n": 0, 2024-04-20T17:10:07.354 INFO:teuthology.orchestra.run.smithi005.stdout: "grade_table": [], 2024-04-20T17:10:07.354 INFO:teuthology.orchestra.run.smithi005.stdout: "stripe_width": 0, 2024-04-20T17:10:07.354 INFO:teuthology.orchestra.run.smithi005.stdout: "expected_num_objects": 0, 2024-04-20T17:10:07.354 INFO:teuthology.orchestra.run.smithi005.stdout: "fast_read": false, 2024-04-20T17:10:07.354 INFO:teuthology.orchestra.run.smithi005.stdout: "options": { 2024-04-20T17:10:07.355 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_max": 32, 2024-04-20T17:10:07.355 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_min": 1 2024-04-20T17:10:07.355 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.355 INFO:teuthology.orchestra.run.smithi005.stdout: "application_metadata": { 2024-04-20T17:10:07.355 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": {} 2024-04-20T17:10:07.355 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.355 INFO:teuthology.orchestra.run.smithi005.stdout: "read_balance": { 2024-04-20T17:10:07.355 INFO:teuthology.orchestra.run.smithi005.stdout: "score_type": "Fair distribution", 2024-04-20T17:10:07.355 INFO:teuthology.orchestra.run.smithi005.stdout: "score_acting": 11.760000228881836, 2024-04-20T17:10:07.355 INFO:teuthology.orchestra.run.smithi005.stdout: "score_stable": 11.760000228881836, 2024-04-20T17:10:07.355 INFO:teuthology.orchestra.run.smithi005.stdout: "optimal_score": 0.17000000178813934, 2024-04-20T17:10:07.355 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_acting": 2, 2024-04-20T17:10:07.355 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_stable": 2, 2024-04-20T17:10:07.355 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-04-20T17:10:07.356 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity": 1, 2024-04-20T17:10:07.356 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity_weighted": 1 2024-04-20T17:10:07.356 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.356 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.356 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.356 INFO:teuthology.orchestra.run.smithi005.stdout: "pool": 2, 2024-04-20T17:10:07.356 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_name": "unique_pool_0", 2024-04-20T17:10:07.356 INFO:teuthology.orchestra.run.smithi005.stdout: "create_time": "2024-04-20T17:08:04.904546+0000", 2024-04-20T17:10:07.356 INFO:teuthology.orchestra.run.smithi005.stdout: "flags": 1, 2024-04-20T17:10:07.356 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_names": "hashpspool", 2024-04-20T17:10:07.356 INFO:teuthology.orchestra.run.smithi005.stdout: "type": 1, 2024-04-20T17:10:07.356 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 2, 2024-04-20T17:10:07.356 INFO:teuthology.orchestra.run.smithi005.stdout: "min_size": 2, 2024-04-20T17:10:07.356 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_rule": 0, 2024-04-20T17:10:07.356 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_count": 0, 2024-04-20T17:10:07.357 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_target": 0, 2024-04-20T17:10:07.357 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T17:10:07.357 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T17:10:07.357 INFO:teuthology.orchestra.run.smithi005.stdout: "object_hash": 2, 2024-04-20T17:10:07.357 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_autoscale_mode": "off", 2024-04-20T17:10:07.357 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num": 16, 2024-04-20T17:10:07.357 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num": 16, 2024-04-20T17:10:07.357 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num_target": 16, 2024-04-20T17:10:07.357 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_target": 16, 2024-04-20T17:10:07.357 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_pending": 16, 2024-04-20T17:10:07.357 INFO:teuthology.orchestra.run.smithi005.stdout: "last_pg_merge_meta": { 2024-04-20T17:10:07.357 INFO:teuthology.orchestra.run.smithi005.stdout: "source_pgid": "0.0", 2024-04-20T17:10:07.357 INFO:teuthology.orchestra.run.smithi005.stdout: "ready_epoch": 0, 2024-04-20T17:10:07.357 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_started": 0, 2024-04-20T17:10:07.357 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_clean": 0, 2024-04-20T17:10:07.358 INFO:teuthology.orchestra.run.smithi005.stdout: "source_version": "0'0", 2024-04-20T17:10:07.358 INFO:teuthology.orchestra.run.smithi005.stdout: "target_version": "0'0" 2024-04-20T17:10:07.358 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.358 INFO:teuthology.orchestra.run.smithi005.stdout: "last_change": "118", 2024-04-20T17:10:07.358 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend": "0", 2024-04-20T17:10:07.358 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-20T17:10:07.358 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_preluminous": "0", 2024-04-20T17:10:07.358 INFO:teuthology.orchestra.run.smithi005.stdout: "auid": 0, 2024-04-20T17:10:07.358 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_mode": "selfmanaged", 2024-04-20T17:10:07.358 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_seq": 0, 2024-04-20T17:10:07.358 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_epoch": 0, 2024-04-20T17:10:07.358 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_snaps": [], 2024-04-20T17:10:07.358 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_snaps": "[]", 2024-04-20T17:10:07.359 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_bytes": 0, 2024-04-20T17:10:07.359 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_objects": 0, 2024-04-20T17:10:07.359 INFO:teuthology.orchestra.run.smithi005.stdout: "tiers": [], 2024-04-20T17:10:07.359 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_of": -1, 2024-04-20T17:10:07.359 INFO:teuthology.orchestra.run.smithi005.stdout: "read_tier": -1, 2024-04-20T17:10:07.359 INFO:teuthology.orchestra.run.smithi005.stdout: "write_tier": -1, 2024-04-20T17:10:07.359 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_mode": "none", 2024-04-20T17:10:07.359 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_bytes": 0, 2024-04-20T17:10:07.359 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_objects": 0, 2024-04-20T17:10:07.359 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T17:10:07.359 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T17:10:07.359 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T17:10:07.359 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_flush_age": 0, 2024-04-20T17:10:07.359 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_evict_age": 0, 2024-04-20T17:10:07.360 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_profile": "", 2024-04-20T17:10:07.360 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_params": { 2024-04-20T17:10:07.360 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "none" 2024-04-20T17:10:07.360 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.360 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_period": 0, 2024-04-20T17:10:07.360 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_count": 0, 2024-04-20T17:10:07.360 INFO:teuthology.orchestra.run.smithi005.stdout: "use_gmt_hitset": true, 2024-04-20T17:10:07.360 INFO:teuthology.orchestra.run.smithi005.stdout: "min_read_recency_for_promote": 0, 2024-04-20T17:10:07.360 INFO:teuthology.orchestra.run.smithi005.stdout: "min_write_recency_for_promote": 0, 2024-04-20T17:10:07.360 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T17:10:07.360 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_search_last_n": 0, 2024-04-20T17:10:07.360 INFO:teuthology.orchestra.run.smithi005.stdout: "grade_table": [], 2024-04-20T17:10:07.360 INFO:teuthology.orchestra.run.smithi005.stdout: "stripe_width": 0, 2024-04-20T17:10:07.360 INFO:teuthology.orchestra.run.smithi005.stdout: "expected_num_objects": 0, 2024-04-20T17:10:07.360 INFO:teuthology.orchestra.run.smithi005.stdout: "fast_read": false, 2024-04-20T17:10:07.361 INFO:teuthology.orchestra.run.smithi005.stdout: "options": {}, 2024-04-20T17:10:07.361 INFO:teuthology.orchestra.run.smithi005.stdout: "application_metadata": { 2024-04-20T17:10:07.361 INFO:teuthology.orchestra.run.smithi005.stdout: "rados": {} 2024-04-20T17:10:07.361 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.361 INFO:teuthology.orchestra.run.smithi005.stdout: "read_balance": { 2024-04-20T17:10:07.361 INFO:teuthology.orchestra.run.smithi005.stdout: "score_type": "Fair distribution", 2024-04-20T17:10:07.361 INFO:teuthology.orchestra.run.smithi005.stdout: "score_acting": 2.25, 2024-04-20T17:10:07.361 INFO:teuthology.orchestra.run.smithi005.stdout: "score_stable": 2.25, 2024-04-20T17:10:07.361 INFO:teuthology.orchestra.run.smithi005.stdout: "optimal_score": 1, 2024-04-20T17:10:07.361 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_acting": 2.25, 2024-04-20T17:10:07.361 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_stable": 2.25, 2024-04-20T17:10:07.361 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-04-20T17:10:07.361 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity": 1, 2024-04-20T17:10:07.361 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity_weighted": 1 2024-04-20T17:10:07.361 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.362 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.362 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:10:07.362 INFO:teuthology.orchestra.run.smithi005.stdout: "osds": [ 2024-04-20T17:10:07.362 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.362 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 0, 2024-04-20T17:10:07.362 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b", 2024-04-20T17:10:07.362 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:10:07.362 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:10:07.362 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:10:07.362 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:10:07.362 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:10:07.362 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:10:07.362 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 8, 2024-04-20T17:10:07.362 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 149, 2024-04-20T17:10:07.362 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:10:07.363 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:10:07.363 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:10:07.363 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.363 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.363 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.363 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6801", 2024-04-20T17:10:07.363 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:10:07.363 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.363 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.363 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.363 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:10:07.363 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.363 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.363 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.363 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6802", 2024-04-20T17:10:07.363 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:10:07.364 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.364 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.364 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.364 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:10:07.364 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.364 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.364 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.364 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6804", 2024-04-20T17:10:07.364 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:10:07.364 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.364 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.364 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.364 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:10:07.364 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.364 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.364 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.365 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6803", 2024-04-20T17:10:07.365 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:10:07.365 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.365 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.365 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.365 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6801/3901484705", 2024-04-20T17:10:07.365 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6802/3901484705", 2024-04-20T17:10:07.365 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6804/3901484705", 2024-04-20T17:10:07.365 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6803/3901484705", 2024-04-20T17:10:07.365 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:10:07.365 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:10:07.365 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:10:07.365 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.365 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.365 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.366 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 1, 2024-04-20T17:10:07.366 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "aed0c8eb-0e81-4e85-ab8f-a60420060656", 2024-04-20T17:10:07.366 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:10:07.366 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:10:07.366 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:10:07.366 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:10:07.366 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:10:07.366 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:10:07.366 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 13, 2024-04-20T17:10:07.366 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 115, 2024-04-20T17:10:07.366 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:10:07.366 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:10:07.366 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:10:07.366 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.367 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.367 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.367 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6805", 2024-04-20T17:10:07.367 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3391893914 2024-04-20T17:10:07.367 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.367 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.367 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.367 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:10:07.367 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.367 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.367 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.367 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6806", 2024-04-20T17:10:07.367 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3391893914 2024-04-20T17:10:07.367 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.368 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.368 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.368 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:10:07.368 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.368 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.368 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.368 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6808", 2024-04-20T17:10:07.368 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3391893914 2024-04-20T17:10:07.368 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.368 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.368 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.368 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:10:07.368 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.368 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.368 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.369 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6807", 2024-04-20T17:10:07.369 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3391893914 2024-04-20T17:10:07.369 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.369 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.369 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.369 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6805/3391893914", 2024-04-20T17:10:07.369 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6806/3391893914", 2024-04-20T17:10:07.369 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6808/3391893914", 2024-04-20T17:10:07.369 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6807/3391893914", 2024-04-20T17:10:07.369 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:10:07.369 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:10:07.369 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:10:07.369 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.369 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.370 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.370 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 2, 2024-04-20T17:10:07.370 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "f733a6a2-19c2-4c7a-af9c-04e282062bb7", 2024-04-20T17:10:07.370 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:10:07.370 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:10:07.370 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:10:07.370 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:10:07.370 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:10:07.370 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:10:07.370 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 20, 2024-04-20T17:10:07.370 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 150, 2024-04-20T17:10:07.370 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:10:07.370 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:10:07.370 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:10:07.370 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.371 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.371 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.371 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6809", 2024-04-20T17:10:07.371 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2105468358 2024-04-20T17:10:07.371 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.371 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.371 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.371 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:10:07.371 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.371 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.371 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.371 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6810", 2024-04-20T17:10:07.371 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2105468358 2024-04-20T17:10:07.371 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.371 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.372 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.372 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:10:07.372 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.372 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.372 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.372 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6812", 2024-04-20T17:10:07.372 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2105468358 2024-04-20T17:10:07.372 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.372 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.372 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.372 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:10:07.372 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.372 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.372 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.372 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6811", 2024-04-20T17:10:07.373 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2105468358 2024-04-20T17:10:07.373 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.373 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.373 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.373 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6809/2105468358", 2024-04-20T17:10:07.373 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6810/2105468358", 2024-04-20T17:10:07.373 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6812/2105468358", 2024-04-20T17:10:07.373 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6811/2105468358", 2024-04-20T17:10:07.373 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:10:07.373 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:10:07.373 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:10:07.373 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.373 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.373 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.373 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 3, 2024-04-20T17:10:07.374 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "8c92f1df-e298-4606-88ae-b8a8e79628a1", 2024-04-20T17:10:07.374 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:10:07.374 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:10:07.374 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:10:07.374 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:10:07.374 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:10:07.374 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:10:07.374 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 25, 2024-04-20T17:10:07.374 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 0, 2024-04-20T17:10:07.374 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:10:07.374 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:10:07.374 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:10:07.374 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.374 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.375 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.375 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6813", 2024-04-20T17:10:07.375 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 377652039 2024-04-20T17:10:07.375 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.375 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.375 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.375 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:10:07.375 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.375 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.375 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.375 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6814", 2024-04-20T17:10:07.375 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 377652039 2024-04-20T17:10:07.375 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.376 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.376 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.376 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:10:07.376 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.376 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.376 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.376 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6816", 2024-04-20T17:10:07.376 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 377652039 2024-04-20T17:10:07.376 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.376 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.376 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.376 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:10:07.376 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.376 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.376 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.377 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6815", 2024-04-20T17:10:07.377 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 377652039 2024-04-20T17:10:07.377 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.377 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.377 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.377 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6813/377652039", 2024-04-20T17:10:07.377 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6814/377652039", 2024-04-20T17:10:07.377 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6816/377652039", 2024-04-20T17:10:07.377 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6815/377652039", 2024-04-20T17:10:07.377 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:10:07.377 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:10:07.377 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:10:07.377 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.377 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.378 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.378 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 4, 2024-04-20T17:10:07.378 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "46df16f1-9607-4d7b-9be8-9d4236e160d1", 2024-04-20T17:10:07.378 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:10:07.378 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:10:07.378 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:10:07.378 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:10:07.378 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:10:07.378 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:10:07.378 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 31, 2024-04-20T17:10:07.378 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 0, 2024-04-20T17:10:07.378 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:10:07.378 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:10:07.378 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:10:07.379 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.379 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.379 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.379 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6800", 2024-04-20T17:10:07.379 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:10:07.379 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.379 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.379 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.379 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:10:07.379 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.379 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.379 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.379 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6801", 2024-04-20T17:10:07.379 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:10:07.379 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.379 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.380 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.380 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:10:07.380 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.380 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.380 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.380 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6803", 2024-04-20T17:10:07.380 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:10:07.380 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.380 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.380 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.380 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:10:07.380 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.380 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.380 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.381 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6802", 2024-04-20T17:10:07.381 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:10:07.381 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.381 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.381 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.381 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6800/1839524469", 2024-04-20T17:10:07.381 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6801/1839524469", 2024-04-20T17:10:07.381 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6803/1839524469", 2024-04-20T17:10:07.381 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6802/1839524469", 2024-04-20T17:10:07.381 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:10:07.381 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:10:07.381 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:10:07.381 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.381 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.381 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.382 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 5, 2024-04-20T17:10:07.382 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "6603231d-1e5c-4575-bfaa-2eeca2c12215", 2024-04-20T17:10:07.382 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:10:07.382 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:10:07.382 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:10:07.382 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:10:07.382 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:10:07.382 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:10:07.382 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 37, 2024-04-20T17:10:07.382 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 132, 2024-04-20T17:10:07.382 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:10:07.382 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:10:07.382 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:10:07.382 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.383 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.383 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.383 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6804", 2024-04-20T17:10:07.383 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:10:07.383 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.383 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.383 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.383 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:10:07.383 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.383 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.383 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.383 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6805", 2024-04-20T17:10:07.383 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:10:07.383 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.383 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.384 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.384 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:10:07.384 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.384 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.384 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.384 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6807", 2024-04-20T17:10:07.384 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:10:07.384 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.384 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.384 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.384 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:10:07.384 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.384 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.385 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.385 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6806", 2024-04-20T17:10:07.385 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:10:07.385 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.385 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.385 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.385 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6804/2450315908", 2024-04-20T17:10:07.385 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6805/2450315908", 2024-04-20T17:10:07.385 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6807/2450315908", 2024-04-20T17:10:07.385 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6806/2450315908", 2024-04-20T17:10:07.385 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:10:07.385 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:10:07.385 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:10:07.385 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.386 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.386 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.386 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 6, 2024-04-20T17:10:07.386 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "783a55a6-f176-42be-8c8b-af8481d08dc9", 2024-04-20T17:10:07.386 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:10:07.386 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:10:07.386 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:10:07.386 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:10:07.386 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:10:07.386 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:10:07.386 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 48, 2024-04-20T17:10:07.386 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 115, 2024-04-20T17:10:07.386 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:10:07.386 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:10:07.387 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:10:07.387 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.387 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.387 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.387 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6808", 2024-04-20T17:10:07.387 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2840237913 2024-04-20T17:10:07.387 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.387 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.387 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.387 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:10:07.387 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.387 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.387 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.387 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6809", 2024-04-20T17:10:07.387 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2840237913 2024-04-20T17:10:07.388 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.388 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.388 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.388 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:10:07.388 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.388 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.388 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.388 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6811", 2024-04-20T17:10:07.388 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2840237913 2024-04-20T17:10:07.388 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.388 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.388 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.388 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:10:07.388 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.389 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.389 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.389 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6810", 2024-04-20T17:10:07.389 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2840237913 2024-04-20T17:10:07.389 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.389 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.389 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.389 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6808/2840237913", 2024-04-20T17:10:07.389 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6809/2840237913", 2024-04-20T17:10:07.389 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6811/2840237913", 2024-04-20T17:10:07.389 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6810/2840237913", 2024-04-20T17:10:07.389 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:10:07.389 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:10:07.389 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:10:07.389 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.390 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.390 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.390 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 7, 2024-04-20T17:10:07.390 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4", 2024-04-20T17:10:07.390 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:10:07.390 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:10:07.390 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:10:07.390 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:10:07.390 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:10:07.390 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:10:07.390 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 56, 2024-04-20T17:10:07.390 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 115, 2024-04-20T17:10:07.390 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:10:07.390 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:10:07.391 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:10:07.391 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.391 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.391 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.391 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6812", 2024-04-20T17:10:07.391 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1105371883 2024-04-20T17:10:07.391 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.391 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.391 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.391 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:10:07.391 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.391 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.391 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.391 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6813", 2024-04-20T17:10:07.391 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1105371883 2024-04-20T17:10:07.392 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.392 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.392 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.392 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:10:07.392 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.392 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.392 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.392 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6815", 2024-04-20T17:10:07.392 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1105371883 2024-04-20T17:10:07.392 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.392 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.392 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.392 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:10:07.392 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.392 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.393 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.393 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6814", 2024-04-20T17:10:07.393 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1105371883 2024-04-20T17:10:07.393 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.393 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.393 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.393 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6812/1105371883", 2024-04-20T17:10:07.393 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6813/1105371883", 2024-04-20T17:10:07.393 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6815/1105371883", 2024-04-20T17:10:07.393 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6814/1105371883", 2024-04-20T17:10:07.393 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:10:07.393 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:10:07.393 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:10:07.393 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.393 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.394 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.394 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 8, 2024-04-20T17:10:07.394 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "1cb232ad-1f59-4a0a-a215-e67419ad6476", 2024-04-20T17:10:07.394 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:10:07.394 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:10:07.394 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:10:07.394 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:10:07.394 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:10:07.394 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:10:07.394 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 67, 2024-04-20T17:10:07.394 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 115, 2024-04-20T17:10:07.394 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:10:07.394 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:10:07.394 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:10:07.395 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.395 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.395 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.395 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6800", 2024-04-20T17:10:07.395 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:10:07.395 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.395 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.395 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.395 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:10:07.395 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.395 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.395 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.395 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6801", 2024-04-20T17:10:07.395 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:10:07.396 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.396 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.396 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.396 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:10:07.396 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.396 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.396 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.396 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6803", 2024-04-20T17:10:07.396 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:10:07.396 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.396 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.396 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.396 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:10:07.396 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.397 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.397 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.397 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6802", 2024-04-20T17:10:07.397 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:10:07.397 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.397 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.397 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.397 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6800/4123058942", 2024-04-20T17:10:07.397 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6801/4123058942", 2024-04-20T17:10:07.397 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6803/4123058942", 2024-04-20T17:10:07.397 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6802/4123058942", 2024-04-20T17:10:07.397 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:10:07.397 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:10:07.397 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:10:07.397 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.398 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.398 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.398 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 9, 2024-04-20T17:10:07.398 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "b6c17147-b3dc-475b-bb20-f9f712fb99e8", 2024-04-20T17:10:07.398 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:10:07.398 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:10:07.398 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:10:07.398 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:10:07.398 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:10:07.398 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:10:07.398 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 73, 2024-04-20T17:10:07.398 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 115, 2024-04-20T17:10:07.398 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:10:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:10:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:10:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6804", 2024-04-20T17:10:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:10:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:10:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6805", 2024-04-20T17:10:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:10:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:10:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6807", 2024-04-20T17:10:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:10:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:10:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6806", 2024-04-20T17:10:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:10:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6804/2877589636", 2024-04-20T17:10:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6805/2877589636", 2024-04-20T17:10:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6807/2877589636", 2024-04-20T17:10:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6806/2877589636", 2024-04-20T17:10:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:10:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:10:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:10:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 10, 2024-04-20T17:10:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "1adcdf55-f002-4a39-91e7-378e5fae849c", 2024-04-20T17:10:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:10:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:10:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:10:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:10:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:10:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:10:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 84, 2024-04-20T17:10:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 0, 2024-04-20T17:10:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:10:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:10:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:10:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6808", 2024-04-20T17:10:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:10:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:10:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6809", 2024-04-20T17:10:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:10:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:10:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6811", 2024-04-20T17:10:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:10:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:10:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6810", 2024-04-20T17:10:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:10:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6808/181490621", 2024-04-20T17:10:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6809/181490621", 2024-04-20T17:10:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6811/181490621", 2024-04-20T17:10:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6810/181490621", 2024-04-20T17:10:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:10:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:10:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:10:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 11, 2024-04-20T17:10:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "737c5aec-694a-4968-819d-fccf22661f4f", 2024-04-20T17:10:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:10:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:10:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:10:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:10:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:10:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:10:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 92, 2024-04-20T17:10:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 115, 2024-04-20T17:10:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:10:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:10:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:10:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6812", 2024-04-20T17:10:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:10:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:10:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6813", 2024-04-20T17:10:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:10:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:10:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6815", 2024-04-20T17:10:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:10:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:10:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:10:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:10:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6814", 2024-04-20T17:10:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:10:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6812/3373585701", 2024-04-20T17:10:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6813/3373585701", 2024-04-20T17:10:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6815/3373585701", 2024-04-20T17:10:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6814/3373585701", 2024-04-20T17:10:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:10:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:10:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:10:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:10:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_xinfo": [ 2024-04-20T17:10:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 0, 2024-04-20T17:10:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:10:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:10:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:10:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:10:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:10:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:09:54.370068+0000", 2024-04-20T17:10:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:10:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 1, 2024-04-20T17:10:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:10:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:10:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:10:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:10:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:10:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:09:13.276482+0000", 2024-04-20T17:10:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:10:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 2, 2024-04-20T17:10:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:10:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:10:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:10:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:10:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:10:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:09:29.816427+0000", 2024-04-20T17:10:07.412 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:10:07.412 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.412 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.412 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 3, 2024-04-20T17:10:07.412 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:10:07.412 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:10:07.412 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:10:07.412 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:10:07.412 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:10:07.412 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:08:42.721685+0000", 2024-04-20T17:10:07.412 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:10:07.412 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.412 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.412 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 4, 2024-04-20T17:10:07.412 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:10:07.412 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:10:07.412 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:10:07.413 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:10:07.413 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:10:07.413 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:09:10.189022+0000", 2024-04-20T17:10:07.413 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:10:07.413 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.413 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.413 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 5, 2024-04-20T17:10:07.413 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:10:07.413 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:10:07.413 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:10:07.413 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:10:07.413 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:10:07.413 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:09:07.140169+0000", 2024-04-20T17:10:07.413 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:10:07.413 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.414 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.414 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 6, 2024-04-20T17:10:07.414 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:10:07.414 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:10:07.414 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:10:07.414 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:10:07.414 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:10:07.414 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:09:37.917111+0000", 2024-04-20T17:10:07.414 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:10:07.414 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.414 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.414 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 7, 2024-04-20T17:10:07.414 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:10:07.414 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:10:07.415 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:10:07.415 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:10:07.415 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:10:07.415 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:09:04.202179+0000", 2024-04-20T17:10:07.415 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:10:07.415 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.415 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.415 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 8, 2024-04-20T17:10:07.415 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:10:07.415 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:10:07.415 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:10:07.415 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:10:07.415 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:10:07.415 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:09:28.160922+0000", 2024-04-20T17:10:07.415 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:10:07.415 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.416 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.416 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 9, 2024-04-20T17:10:07.416 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:10:07.416 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:10:07.416 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:10:07.416 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:10:07.416 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:10:07.416 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:08:49.673293+0000", 2024-04-20T17:10:07.416 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:10:07.416 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.416 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.416 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 10, 2024-04-20T17:10:07.416 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:10:07.416 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:10:07.416 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:10:07.417 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:10:07.417 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:10:07.417 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:09:31.872180+0000", 2024-04-20T17:10:07.417 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:10:07.417 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.417 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.417 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 11, 2024-04-20T17:10:07.417 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:10:07.417 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:10:07.417 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:10:07.417 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:10:07.417 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:10:07.417 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:08:47.110274+0000", 2024-04-20T17:10:07.417 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:10:07.417 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.418 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:10:07.418 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_upmap": [], 2024-04-20T17:10:07.418 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_upmap_items": [ 2024-04-20T17:10:07.418 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.418 INFO:teuthology.orchestra.run.smithi005.stdout: "pgid": "2.3", 2024-04-20T17:10:07.418 INFO:teuthology.orchestra.run.smithi005.stdout: "mappings": [ 2024-04-20T17:10:07.418 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.418 INFO:teuthology.orchestra.run.smithi005.stdout: "from": 2, 2024-04-20T17:10:07.418 INFO:teuthology.orchestra.run.smithi005.stdout: "to": 10 2024-04-20T17:10:07.418 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.418 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.418 INFO:teuthology.orchestra.run.smithi005.stdout: "from": 5, 2024-04-20T17:10:07.418 INFO:teuthology.orchestra.run.smithi005.stdout: "to": 0 2024-04-20T17:10:07.418 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.419 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.419 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.419 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.419 INFO:teuthology.orchestra.run.smithi005.stdout: "pgid": "2.4", 2024-04-20T17:10:07.419 INFO:teuthology.orchestra.run.smithi005.stdout: "mappings": [ 2024-04-20T17:10:07.419 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.419 INFO:teuthology.orchestra.run.smithi005.stdout: "from": 1, 2024-04-20T17:10:07.419 INFO:teuthology.orchestra.run.smithi005.stdout: "to": 5 2024-04-20T17:10:07.419 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.419 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.419 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.419 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:10:07.419 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_upmap_primaries": [], 2024-04-20T17:10:07.419 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_temp": [ 2024-04-20T17:10:07.420 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.420 INFO:teuthology.orchestra.run.smithi005.stdout: "pgid": "2.3", 2024-04-20T17:10:07.420 INFO:teuthology.orchestra.run.smithi005.stdout: "osds": [ 2024-04-20T17:10:07.420 INFO:teuthology.orchestra.run.smithi005.stdout: 2, 2024-04-20T17:10:07.420 INFO:teuthology.orchestra.run.smithi005.stdout: 0 2024-04-20T17:10:07.420 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.420 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.420 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:10:07.420 INFO:teuthology.orchestra.run.smithi005.stdout: "pgid": "2.4", 2024-04-20T17:10:07.420 INFO:teuthology.orchestra.run.smithi005.stdout: "osds": [ 2024-04-20T17:10:07.420 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-04-20T17:10:07.420 INFO:teuthology.orchestra.run.smithi005.stdout: 5 2024-04-20T17:10:07.420 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:10:07.420 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.420 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:10:07.420 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_temp": [], 2024-04-20T17:10:07.421 INFO:teuthology.orchestra.run.smithi005.stdout: "blocklist": { 2024-04-20T17:10:07.421 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/3881726425": "2024-04-21T16:59:25.808337+0000", 2024-04-20T17:10:07.421 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/338649532": "2024-04-21T16:59:25.808337+0000", 2024-04-20T17:10:07.421 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/1227843664": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:10:07.421 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:6800/1876064631": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:10:07.421 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/160689449": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:10:07.421 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:6800/3521715286": "2024-04-21T16:59:25.808337+0000", 2024-04-20T17:10:07.421 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/3841663462": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:10:07.421 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/687690919": "2024-04-21T16:59:25.808337+0000" 2024-04-20T17:10:07.421 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.421 INFO:teuthology.orchestra.run.smithi005.stdout: "range_blocklist": {}, 2024-04-20T17:10:07.421 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_profiles": { 2024-04-20T17:10:07.421 INFO:teuthology.orchestra.run.smithi005.stdout: "default": { 2024-04-20T17:10:07.421 INFO:teuthology.orchestra.run.smithi005.stdout: "crush-failure-domain": "osd", 2024-04-20T17:10:07.421 INFO:teuthology.orchestra.run.smithi005.stdout: "k": "2", 2024-04-20T17:10:07.421 INFO:teuthology.orchestra.run.smithi005.stdout: "m": "1", 2024-04-20T17:10:07.422 INFO:teuthology.orchestra.run.smithi005.stdout: "plugin": "jerasure", 2024-04-20T17:10:07.422 INFO:teuthology.orchestra.run.smithi005.stdout: "technique": "reed_sol_van" 2024-04-20T17:10:07.422 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.422 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:10:07.422 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_snaps_queue": [], 2024-04-20T17:10:07.422 INFO:teuthology.orchestra.run.smithi005.stdout: "new_removed_snaps": [], 2024-04-20T17:10:07.422 INFO:teuthology.orchestra.run.smithi005.stdout: "new_purged_snaps": [], 2024-04-20T17:10:07.422 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_node_flags": {}, 2024-04-20T17:10:07.422 INFO:teuthology.orchestra.run.smithi005.stdout: "device_class_flags": {}, 2024-04-20T17:10:07.422 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": { 2024-04-20T17:10:07.422 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode_enabled": false, 2024-04-20T17:10:07.422 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_bucket_count": 0, 2024-04-20T17:10:07.422 INFO:teuthology.orchestra.run.smithi005.stdout: "degraded_stretch_mode": 0, 2024-04-20T17:10:07.422 INFO:teuthology.orchestra.run.smithi005.stdout: "recovering_stretch_mode": 0, 2024-04-20T17:10:07.422 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode_bucket": 0 2024-04-20T17:10:07.423 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:10:07.423 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:10:07.423 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:10:07.450 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:07.469 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:10:07.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:07.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:10:07.573 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:10:07.608 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:07.637 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:10:07.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:07.648 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:10:07.749 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:10:07.776 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:07.796 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:10:07.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:07.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:10:07.857 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:10:07.861 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:07 smithi171 ceph-mon[44189]: pgmap v464: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 762 op/s; 19181/296564 objects misplaced (6.468%); 8.4 MiB/s, 134 objects/s recovering 2024-04-20T17:10:07.861 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:07 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/719769491' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:10:07.861 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:07 smithi171 ceph-mon[44189]: from='client.24845 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:10:07.861 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:07 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2660104158' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-20T17:10:07.899 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:10:07.966 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:08.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:07 smithi005 ceph-mon[39562]: pgmap v464: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 762 op/s; 19181/296564 objects misplaced (6.468%); 8.4 MiB/s, 134 objects/s recovering 2024-04-20T17:10:08.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:07 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/719769491' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:10:08.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:07 smithi005 ceph-mon[39562]: from='client.24845 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:10:08.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:07 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2660104158' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-20T17:10:08.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:07 smithi106 ceph-mon[44423]: pgmap v464: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 762 op/s; 19181/296564 objects misplaced (6.468%); 8.4 MiB/s, 134 objects/s recovering 2024-04-20T17:10:08.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:07 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/719769491' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:10:08.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:07 smithi106 ceph-mon[44423]: from='client.24845 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:10:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:07 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2660104158' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-20T17:10:08.865 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:08 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:10:08.865 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:08 smithi005 ceph-mon[39562]: from='client.24845 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:10:08.865 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:08 smithi005 ceph-mon[39562]: osdmap e155: 12 total, 12 up, 12 in 2024-04-20T17:10:08.866 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:08 smithi005 ceph-mon[39562]: pgmap v466: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 795 op/s; 19181/293012 objects misplaced (6.546%); 7.5 MiB/s, 120 objects/s recovering 2024-04-20T17:10:09.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:08 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:10:09.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:08 smithi106 ceph-mon[44423]: from='client.24845 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:10:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:08 smithi106 ceph-mon[44423]: osdmap e155: 12 total, 12 up, 12 in 2024-04-20T17:10:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:08 smithi106 ceph-mon[44423]: pgmap v466: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 795 op/s; 19181/293012 objects misplaced (6.546%); 7.5 MiB/s, 120 objects/s recovering 2024-04-20T17:10:09.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:08 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:10:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:08 smithi171 ceph-mon[44189]: from='client.24845 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:10:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:08 smithi171 ceph-mon[44189]: osdmap e155: 12 total, 12 up, 12 in 2024-04-20T17:10:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:08 smithi171 ceph-mon[44189]: pgmap v466: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 795 op/s; 19181/293012 objects misplaced (6.546%); 7.5 MiB/s, 120 objects/s recovering 2024-04-20T17:10:09.401 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:10:09.459 INFO:tasks.thrashosds.thrasher:j is {'epoch': 154, 'fsid': '13c6e76c-ff37-11ee-bc93-c7b262605968', 'created': '2024-04-20T16:58:25.272557+0000', 'modified': '2024-04-20T17:09:54.738652+0000', 'last_up_change': '2024-04-20T17:06:01.788926+0000', 'last_in_change': '2024-04-20T17:05:42.849446+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-20T17:01:35.909493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.760000228881836, 'score_stable': 11.760000228881836, 'optimal_score': 0.17000000178813934, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-04-20T17:08:04.904546+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 149, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6801', 'nonce': 3901484705}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6802', 'nonce': 3901484705}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6804', 'nonce': 3901484705}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6803', 'nonce': 3901484705}]}, 'public_addr': '172.21.15.5:6801/3901484705', 'cluster_addr': '172.21.15.5:6802/3901484705', 'heartbeat_back_addr': '172.21.15.5:6804/3901484705', 'heartbeat_front_addr': '172.21.15.5:6803/3901484705', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'aed0c8eb-0e81-4e85-ab8f-a60420060656', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6805', 'nonce': 3391893914}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6806', 'nonce': 3391893914}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6808', 'nonce': 3391893914}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6807', 'nonce': 3391893914}]}, 'public_addr': '172.21.15.5:6805/3391893914', 'cluster_addr': '172.21.15.5:6806/3391893914', 'heartbeat_back_addr': '172.21.15.5:6808/3391893914', 'heartbeat_front_addr': '172.21.15.5:6807/3391893914', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f733a6a2-19c2-4c7a-af9c-04e282062bb7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 20, 'up_thru': 150, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6809', 'nonce': 2105468358}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6810', 'nonce': 2105468358}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6812', 'nonce': 2105468358}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6811', 'nonce': 2105468358}]}, 'public_addr': '172.21.15.5:6809/2105468358', 'cluster_addr': '172.21.15.5:6810/2105468358', 'heartbeat_back_addr': '172.21.15.5:6812/2105468358', 'heartbeat_front_addr': '172.21.15.5:6811/2105468358', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '8c92f1df-e298-4606-88ae-b8a8e79628a1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6813', 'nonce': 377652039}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6814', 'nonce': 377652039}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6816', 'nonce': 377652039}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6815', 'nonce': 377652039}]}, 'public_addr': '172.21.15.5:6813/377652039', 'cluster_addr': '172.21.15.5:6814/377652039', 'heartbeat_back_addr': '172.21.15.5:6816/377652039', 'heartbeat_front_addr': '172.21.15.5:6815/377652039', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '46df16f1-9607-4d7b-9be8-9d4236e160d1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6800', 'nonce': 1839524469}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6801', 'nonce': 1839524469}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6803', 'nonce': 1839524469}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6802', 'nonce': 1839524469}]}, 'public_addr': '172.21.15.106:6800/1839524469', 'cluster_addr': '172.21.15.106:6801/1839524469', 'heartbeat_back_addr': '172.21.15.106:6803/1839524469', 'heartbeat_front_addr': '172.21.15.106:6802/1839524469', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6603231d-1e5c-4575-bfaa-2eeca2c12215', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 132, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6804', 'nonce': 2450315908}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6805', 'nonce': 2450315908}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6807', 'nonce': 2450315908}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6806', 'nonce': 2450315908}]}, 'public_addr': '172.21.15.106:6804/2450315908', 'cluster_addr': '172.21.15.106:6805/2450315908', 'heartbeat_back_addr': '172.21.15.106:6807/2450315908', 'heartbeat_front_addr': '172.21.15.106:6806/2450315908', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '783a55a6-f176-42be-8c8b-af8481d08dc9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6808', 'nonce': 2840237913}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6809', 'nonce': 2840237913}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6811', 'nonce': 2840237913}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6810', 'nonce': 2840237913}]}, 'public_addr': '172.21.15.106:6808/2840237913', 'cluster_addr': '172.21.15.106:6809/2840237913', 'heartbeat_back_addr': '172.21.15.106:6811/2840237913', 'heartbeat_front_addr': '172.21.15.106:6810/2840237913', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6812', 'nonce': 1105371883}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6813', 'nonce': 1105371883}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6815', 'nonce': 1105371883}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6814', 'nonce': 1105371883}]}, 'public_addr': '172.21.15.106:6812/1105371883', 'cluster_addr': '172.21.15.106:6813/1105371883', 'heartbeat_back_addr': '172.21.15.106:6815/1105371883', 'heartbeat_front_addr': '172.21.15.106:6814/1105371883', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '1cb232ad-1f59-4a0a-a215-e67419ad6476', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 67, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6800', 'nonce': 4123058942}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6801', 'nonce': 4123058942}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6803', 'nonce': 4123058942}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6802', 'nonce': 4123058942}]}, 'public_addr': '172.21.15.171:6800/4123058942', 'cluster_addr': '172.21.15.171:6801/4123058942', 'heartbeat_back_addr': '172.21.15.171:6803/4123058942', 'heartbeat_front_addr': '172.21.15.171:6802/4123058942', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'b6c17147-b3dc-475b-bb20-f9f712fb99e8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 73, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6804', 'nonce': 2877589636}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6805', 'nonce': 2877589636}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6807', 'nonce': 2877589636}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6806', 'nonce': 2877589636}]}, 'public_addr': '172.21.15.171:6804/2877589636', 'cluster_addr': '172.21.15.171:6805/2877589636', 'heartbeat_back_addr': '172.21.15.171:6807/2877589636', 'heartbeat_front_addr': '172.21.15.171:6806/2877589636', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '1adcdf55-f002-4a39-91e7-378e5fae849c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 84, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6808', 'nonce': 181490621}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6809', 'nonce': 181490621}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6811', 'nonce': 181490621}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6810', 'nonce': 181490621}]}, 'public_addr': '172.21.15.171:6808/181490621', 'cluster_addr': '172.21.15.171:6809/181490621', 'heartbeat_back_addr': '172.21.15.171:6811/181490621', 'heartbeat_front_addr': '172.21.15.171:6810/181490621', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '737c5aec-694a-4968-819d-fccf22661f4f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 92, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6812', 'nonce': 3373585701}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6813', 'nonce': 3373585701}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6815', 'nonce': 3373585701}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6814', 'nonce': 3373585701}]}, 'public_addr': '172.21.15.171:6812/3373585701', 'cluster_addr': '172.21.15.171:6813/3373585701', 'heartbeat_back_addr': '172.21.15.171:6815/3373585701', 'heartbeat_front_addr': '172.21.15.171:6814/3373585701', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:09:54.370068+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:09:13.276482+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:09:29.816427+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:08:42.721685+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:09:10.189022+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:09:07.140169+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:09:37.917111+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:09:04.202179+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:09:28.160922+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:08:49.673293+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:09:31.872180+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:08:47.110274+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.3', 'mappings': [{'from': 2, 'to': 10}, {'from': 5, 'to': 0}]}, {'pgid': '2.4', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '2.3', 'osds': [2, 0]}, {'pgid': '2.4', 'osds': [0, 5]}], 'primary_temp': [], 'blocklist': {'172.21.15.5:0/3881726425': '2024-04-21T16:59:25.808337+0000', '172.21.15.5:0/338649532': '2024-04-21T16:59:25.808337+0000', '172.21.15.5:0/1227843664': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:6800/1876064631': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:0/160689449': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:6800/3521715286': '2024-04-21T16:59:25.808337+0000', '172.21.15.5:0/3841663462': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:0/687690919': '2024-04-21T16:59:25.808337+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-20T17:10:09.459 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:10:10.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:09 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:10:10.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:09 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:10:10.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:09 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:10:10.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:09 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:10:10.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:09 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:10:10.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:09 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:10:10.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:09 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:10:10.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:09 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:10:10.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:09 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:10:10.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:09 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:10:10.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:09 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:10:10.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:09 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:10:10.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:09 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:10:10.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:09 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:10:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:09 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:10:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:09 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:10:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:09 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:10:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:09 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:10:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:09 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:10:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:09 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:10:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:09 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:10:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:09 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:10:10.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:09 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:10:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:09 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:10:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:09 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:10:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:09 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:10:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:09 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:10:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:09 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:10:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:09 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:10:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:09 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:10:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:09 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:10:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:09 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:10:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:09 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:10:10.356 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:10:10.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:10.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:10:10.464 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:10:10.496 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:10.525 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:10:10.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:10.528 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:10:10.629 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:10:10.658 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:10.684 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:10:10.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:10.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:10:10.789 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:10:10.822 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:10.846 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:10:10.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:10.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:10:10.948 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:10:10.951 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:10:10.988 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:11.007 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:10:11.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:11.011 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:10:11.112 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:10:11.191 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:11.427 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:10:11.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:11.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:10:11.457 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:10:11.532 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:10:11.593 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:11.734 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:10:11.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:11.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:10:11.840 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:10:11.867 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:11.888 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:10:11.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:11.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:10:11.991 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:10:12.034 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:12.062 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:10:12.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:12.068 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:10:12.169 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:10:12.194 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:11 smithi106 ceph-mon[44423]: pgmap v467: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 786 op/s; 18334/289592 objects misplaced (6.331%); 7.6 MiB/s, 121 objects/s recovering 2024-04-20T17:10:12.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:11 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:10:12.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:11 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:10:12.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:11 smithi106 ceph-mon[44423]: osdmap e156: 12 total, 12 up, 12 in 2024-04-20T17:10:12.196 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:12.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:11 smithi005 ceph-mon[39562]: pgmap v467: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 786 op/s; 18334/289592 objects misplaced (6.331%); 7.6 MiB/s, 121 objects/s recovering 2024-04-20T17:10:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:11 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:10:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:11 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:10:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:11 smithi005 ceph-mon[39562]: osdmap e156: 12 total, 12 up, 12 in 2024-04-20T17:10:12.217 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:10:12.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:12.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:10:12.323 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:10:12.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:11 smithi171 ceph-mon[44189]: pgmap v467: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 786 op/s; 18334/289592 objects misplaced (6.331%); 7.6 MiB/s, 121 objects/s recovering 2024-04-20T17:10:12.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:11 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:10:12.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:11 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:10:12.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:11 smithi171 ceph-mon[44189]: osdmap e156: 12 total, 12 up, 12 in 2024-04-20T17:10:12.353 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:12.377 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:10:12.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:12.384 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:10:12.473 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:10:12.473 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:10:12.473 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:10:12.473 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:10:12.473 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:10:12.484 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:10:12.514 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:12.535 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:10:12.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:12.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:10:12.639 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:10:12.675 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:12.702 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:10:12.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:12.704 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:10:12.723 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:10:12.805 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:10:12.838 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:12.865 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:10:12.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:12.870 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:10:12.971 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:10:13.000 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:13.021 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:10:13.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:13.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:10:13.125 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:10:13.151 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:13.173 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:10:13.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:13.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:10:13.280 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:10:13.307 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:13.327 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:10:13.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:13.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:10:13.431 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:10:13.509 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:13.557 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:10:13.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:13.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:10:13.664 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:10:13.745 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:14.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:13 smithi005 ceph-mon[39562]: pgmap v469: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 656 op/s; 18334/287534 objects misplaced (6.376%); 7.5 MiB/s, 120 objects/s recovering 2024-04-20T17:10:14.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:13 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:10:14.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:13 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:10:14.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:13 smithi005 ceph-mon[39562]: osdmap e157: 12 total, 12 up, 12 in 2024-04-20T17:10:14.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:13 smithi106 ceph-mon[44423]: pgmap v469: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 656 op/s; 18334/287534 objects misplaced (6.376%); 7.5 MiB/s, 120 objects/s recovering 2024-04-20T17:10:14.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:13 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:10:14.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:13 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:10:14.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:13 smithi106 ceph-mon[44423]: osdmap e157: 12 total, 12 up, 12 in 2024-04-20T17:10:14.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:13 smithi171 ceph-mon[44189]: pgmap v469: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 656 op/s; 18334/287534 objects misplaced (6.376%); 7.5 MiB/s, 120 objects/s recovering 2024-04-20T17:10:14.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:13 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:10:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:13 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:10:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:13 smithi171 ceph-mon[44189]: osdmap e157: 12 total, 12 up, 12 in 2024-04-20T17:10:14.405 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-20T17:10:14.718 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:10:14.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:14.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:10:14.830 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:10:14.858 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:14.878 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:10:14.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:14.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:10:14.981 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:10:15.013 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:15.036 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:10:15.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:15.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:10:15.063 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:10:15.063 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:10:15.103 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:10:15.140 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:10:15.168 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:15.188 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:10:15.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:15.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:10:15.202 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:10:15.202 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:10:15.202 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:10:15.232 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:10:15.274 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:10:15.292 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:10:15.304 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:10:15.335 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:10:15.345 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:15.367 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:10:15.407 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:10:15.410 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:10:15.410 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:10:15.410 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:10:15.410 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:10:15.411 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:10:15.411 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:10:15.411 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:10:15.411 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:10:15.411 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:10:15.411 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:10:15.411 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:10:15.411 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:10:16.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:15 smithi005 ceph-mon[39562]: pgmap v471: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 916 op/s; 17855/281780 objects misplaced (6.337%); 8.6 MiB/s, 137 objects/s recovering 2024-04-20T17:10:16.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:15 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:10:16.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:15 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:10:16.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:15 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1014966302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:10:16.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:15 smithi106 ceph-mon[44423]: pgmap v471: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 916 op/s; 17855/281780 objects misplaced (6.337%); 8.6 MiB/s, 137 objects/s recovering 2024-04-20T17:10:16.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:15 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:10:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:15 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:10:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:15 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1014966302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:10:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:15 smithi171 ceph-mon[44189]: pgmap v471: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 916 op/s; 17855/281780 objects misplaced (6.337%); 8.6 MiB/s, 137 objects/s recovering 2024-04-20T17:10:16.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:15 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:10:16.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:15 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:10:16.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:15 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1014966302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:10:16.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:16.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:17.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:17.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:17.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:16 smithi005 ceph-mon[39562]: osdmap e158: 12 total, 12 up, 12 in 2024-04-20T17:10:17.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:17.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:17.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:16 smithi106 ceph-mon[44423]: osdmap e158: 12 total, 12 up, 12 in 2024-04-20T17:10:17.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:16 smithi171 ceph-mon[44189]: osdmap e158: 12 total, 12 up, 12 in 2024-04-20T17:10:17.619 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:10:17.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:17.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:10:17.726 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:10:17.788 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:17.881 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:10:17.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:17.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:10:17.995 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:10:18.060 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:18.103 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:10:18.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:18.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:10:18.207 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:10:18.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:17 smithi005 ceph-mon[39562]: pgmap v473: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 780 op/s; 17452/280216 objects misplaced (6.228%); 6.7 MiB/s, 106 objects/s recovering 2024-04-20T17:10:18.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:17 smithi106 ceph-mon[44423]: pgmap v473: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 780 op/s; 17452/280216 objects misplaced (6.228%); 6.7 MiB/s, 106 objects/s recovering 2024-04-20T17:10:18.264 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:10:18.266 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:18.302 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:10:18.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:18.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:10:18.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:17 smithi171 ceph-mon[44189]: pgmap v473: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 780 op/s; 17452/280216 objects misplaced (6.228%); 6.7 MiB/s, 106 objects/s recovering 2024-04-20T17:10:18.410 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:10:18.482 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:18.591 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:10:18.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:18.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:10:18.661 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:10:18.673 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:10:18.699 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:10:18.726 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:18.745 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:10:18.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:18.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:10:18.849 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:10:18.884 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:18 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:10:18.884 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:18 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:10:18.884 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:18 smithi005 ceph-mon[39562]: osdmap e159: 12 total, 12 up, 12 in 2024-04-20T17:10:18.885 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:18 smithi005 ceph-mon[39562]: pgmap v475: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 925 op/s; 17452/276420 objects misplaced (6.314%); 6.7 MiB/s, 106 objects/s recovering 2024-04-20T17:10:18.885 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:18 smithi005 ceph-mon[39562]: from='client.24863 v1:172.21.15.5:0/3944833483' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:10:18.912 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:18.942 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:10:18.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:18.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:10:19.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:18 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:10:19.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:18 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:10:19.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:18 smithi106 ceph-mon[44423]: osdmap e159: 12 total, 12 up, 12 in 2024-04-20T17:10:19.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:18 smithi106 ceph-mon[44423]: pgmap v475: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 925 op/s; 17452/276420 objects misplaced (6.314%); 6.7 MiB/s, 106 objects/s recovering 2024-04-20T17:10:19.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:18 smithi106 ceph-mon[44423]: from='client.24863 v1:172.21.15.5:0/3944833483' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:10:19.046 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:10:19.075 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:19.096 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:10:19.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:19.099 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:10:19.200 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:10:19.226 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:19.247 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:10:19.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:19.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:10:19.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:18 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:10:19.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:18 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:10:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:18 smithi171 ceph-mon[44189]: osdmap e159: 12 total, 12 up, 12 in 2024-04-20T17:10:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:18 smithi171 ceph-mon[44189]: pgmap v475: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 925 op/s; 17452/276420 objects misplaced (6.314%); 6.7 MiB/s, 106 objects/s recovering 2024-04-20T17:10:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:18 smithi171 ceph-mon[44189]: from='client.24863 v1:172.21.15.5:0/3944833483' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:10:19.351 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:10:19.404 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:19.657 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:10:19.933 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:10:20.084 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:19 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2066324211' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:10:20.084 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:19 smithi005 ceph-mon[39562]: from='client.24869 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:10:20.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:19 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2066324211' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:10:20.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:19 smithi106 ceph-mon[44423]: from='client.24869 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:10:20.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:19 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2066324211' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:10:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:19 smithi171 ceph-mon[44189]: from='client.24869 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:10:20.735 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:10:20.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:20.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:10:20.844 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:10:20.873 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:20.898 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:10:20.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:20.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:10:21.003 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:10:21.058 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:20 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:10:21.058 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:20 smithi005 ceph-mon[39562]: from='client.24869 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:10:21.058 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:20 smithi005 ceph-mon[39562]: osdmap e160: 12 total, 12 up, 12 in 2024-04-20T17:10:21.058 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:20 smithi005 ceph-mon[39562]: pgmap v477: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s; 16469/272962 objects misplaced (6.033%); 12 MiB/s, 188 objects/s recovering 2024-04-20T17:10:21.060 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:21.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:20 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:10:21.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:20 smithi106 ceph-mon[44423]: from='client.24869 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:10:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:20 smithi106 ceph-mon[44423]: osdmap e160: 12 total, 12 up, 12 in 2024-04-20T17:10:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:20 smithi106 ceph-mon[44423]: pgmap v477: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s; 16469/272962 objects misplaced (6.033%); 12 MiB/s, 188 objects/s recovering 2024-04-20T17:10:21.268 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":475,"stamp":"2024-04-20T17:10:17.973749+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":9058124344,"num_objects":138210,"num_object_clones":0,"num_object_copies":276420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":17452,"num_objects_unfound":0,"num_objects_dirty":138210,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":183957,"num_write_kb":10306363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5716,"num_bytes_recovered":376195200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161313,"ondisk_log_size":161313,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":37,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":18919388,"kb_used_data":18166792,"kb_used_omap":18,"kb_used_meta":751597,"kb_avail":1106022436,"statfs":{"total":1151940427776,"available":1132566974464,"internally_reserved":0,"allocated":18602795008,"data_stored":18598606124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":769635641},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,2,10,5,2,3,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-364183552,"num_objects":-5557,"num_object_clones":0,"num_object_copies":-11114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-882,"num_objects_unfound":0,"num_objects_dirty":-5557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5557,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":642,"num_bytes_recovered":42074112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.004478"},"pg_stats":[{"pgid":"2.f","version":"157'11375","reported_seq":11643,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-20T17:10:13.651640+0000","last_change":"2024-04-20T17:09:36.467412+0000","last_active":"2024-04-20T17:10:13.651640+0000","last_peered":"2024-04-20T17:10:13.651640+0000","last_clean":"2024-04-20T17:10:13.651640+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:10:13.651640+0000","last_undegraded":"2024-04-20T17:10:13.651640+0000","last_fullsized":"2024-04-20T17:10:13.651640+0000","mapping_epoch":115,"log_start":"121'1300","ondisk_log_start":"121'1300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'9696","last_scrub_stamp":"2024-04-20T17:09:36.467296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:09:36.467296+0000","objects_scrubbed":9674,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:10:53.491139+0000","scrub_duration":1325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":577175552,"num_objects":8807,"num_object_clones":0,"num_object_copies":17614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11375,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"157'11449","reported_seq":20300,"reported_epoch":157,"state":"active+recovering+remapped","last_fresh":"2024-04-20T17:10:14.860368+0000","last_change":"2024-04-20T17:08:50.616759+0000","last_active":"2024-04-20T17:10:14.860368+0000","last_peered":"2024-04-20T17:10:14.860368+0000","last_clean":"2024-04-20T17:08:47.829425+0000","last_became_active":"2024-04-20T17:08:50.590271+0000","last_became_peered":"2024-04-20T17:08:50.590271+0000","last_unstale":"2024-04-20T17:10:14.860368+0000","last_undegraded":"2024-04-20T17:10:14.860368+0000","last_fullsized":"2024-04-20T17:10:14.860368+0000","mapping_epoch":133,"log_start":"121'1400","ondisk_log_start":"121'1400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":564068352,"num_objects":8607,"num_object_clones":0,"num_object_copies":17214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1241,"num_objects_unfound":0,"num_objects_dirty":8607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11449,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4378,"num_bytes_recovered":286916608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[0,5],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1241},{"shards":"0,5","objects":7366}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"157'11674","reported_seq":11768,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-20T17:10:15.522039+0000","last_change":"2024-04-20T17:08:06.013628+0000","last_active":"2024-04-20T17:10:15.522039+0000","last_peered":"2024-04-20T17:10:15.522039+0000","last_clean":"2024-04-20T17:10:15.522039+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:10:15.522039+0000","last_undegraded":"2024-04-20T17:10:15.522039+0000","last_fullsized":"2024-04-20T17:10:15.522039+0000","mapping_epoch":115,"log_start":"121'1600","ondisk_log_start":"121'1600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":577241088,"num_objects":8808,"num_object_clones":0,"num_object_copies":17616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11674,"num_write_kb":655424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"157'11496","reported_seq":11673,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-20T17:10:13.859529+0000","last_change":"2024-04-20T17:09:09.239641+0000","last_active":"2024-04-20T17:10:13.859529+0000","last_peered":"2024-04-20T17:10:13.859529+0000","last_clean":"2024-04-20T17:10:13.859529+0000","last_became_active":"2024-04-20T17:08:06.027804+0000","last_became_peered":"2024-04-20T17:08:06.027804+0000","last_unstale":"2024-04-20T17:10:13.859529+0000","last_undegraded":"2024-04-20T17:10:13.859529+0000","last_fullsized":"2024-04-20T17:10:13.859529+0000","mapping_epoch":115,"log_start":"121'1400","ondisk_log_start":"121'1400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'7483","last_scrub_stamp":"2024-04-20T17:09:09.239533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:09:09.239533+0000","objects_scrubbed":7410,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:10:21.574586+0000","scrub_duration":1594,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":572653568,"num_objects":8738,"num_object_clones":0,"num_object_copies":17476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11496,"num_write_kb":647488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"157'11194","reported_seq":15280,"reported_epoch":157,"state":"active+recovering+remapped","last_fresh":"2024-04-20T17:10:13.955744+0000","last_change":"2024-04-20T17:09:38.883695+0000","last_active":"2024-04-20T17:10:13.955744+0000","last_peered":"2024-04-20T17:10:13.955744+0000","last_clean":"2024-04-20T17:09:35.882603+0000","last_became_active":"2024-04-20T17:09:38.726106+0000","last_became_peered":"2024-04-20T17:09:38.726106+0000","last_unstale":"2024-04-20T17:10:13.955744+0000","last_undegraded":"2024-04-20T17:10:13.955744+0000","last_fullsized":"2024-04-20T17:10:13.955744+0000","mapping_epoch":150,"log_start":"121'1100","ondisk_log_start":"121'1100","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":559415296,"num_objects":8536,"num_object_clones":0,"num_object_copies":17072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":16211,"num_objects_unfound":0,"num_objects_dirty":8536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11194,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1330,"num_bytes_recovered":87162880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[2,0],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"0,2","objects":430},{"shards":"2,5","objects":8105},{"shards":"2,5,10","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":477,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-20T17:10:15.863924+0000","last_change":"2024-04-20T17:09:49.876424+0000","last_active":"2024-04-20T17:10:15.863924+0000","last_peered":"2024-04-20T17:10:15.863924+0000","last_clean":"2024-04-20T17:10:15.863924+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:10:15.863924+0000","last_undegraded":"2024-04-20T17:10:15.863924+0000","last_fullsized":"2024-04-20T17:10:15.863924+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:09:49.876296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:09:49.876296+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:11:09.614452+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"158'11691","reported_seq":11891,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-20T17:10:17.815502+0000","last_change":"2024-04-20T17:09:41.085134+0000","last_active":"2024-04-20T17:10:17.815502+0000","last_peered":"2024-04-20T17:10:17.815502+0000","last_clean":"2024-04-20T17:10:17.815502+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:10:17.815502+0000","last_undegraded":"2024-04-20T17:10:17.815502+0000","last_fullsized":"2024-04-20T17:10:17.815502+0000","mapping_epoch":115,"log_start":"121'1600","ondisk_log_start":"121'1600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'9846","last_scrub_stamp":"2024-04-20T17:09:41.084919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:09:41.084919+0000","objects_scrubbed":9825,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:10:56.964219+0000","scrub_duration":1238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561053696,"num_objects":8561,"num_object_clones":0,"num_object_copies":17122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11691,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"157'11448","reported_seq":11645,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-20T17:10:13.661636+0000","last_change":"2024-04-20T17:09:38.038373+0000","last_active":"2024-04-20T17:10:13.661636+0000","last_peered":"2024-04-20T17:10:13.661636+0000","last_clean":"2024-04-20T17:10:13.661636+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:10:13.661636+0000","last_undegraded":"2024-04-20T17:10:13.661636+0000","last_fullsized":"2024-04-20T17:10:13.661636+0000","mapping_epoch":115,"log_start":"121'1400","ondisk_log_start":"121'1400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'9758","last_scrub_stamp":"2024-04-20T17:09:38.038264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:09:38.038264+0000","objects_scrubbed":9758,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:10:52.010775+0000","scrub_duration":859,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":581697536,"num_objects":8876,"num_object_clones":0,"num_object_copies":17752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11448,"num_write_kb":650368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"158'11559","reported_seq":11760,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-20T17:10:16.714923+0000","last_change":"2024-04-20T17:09:42.012666+0000","last_active":"2024-04-20T17:10:16.714923+0000","last_peered":"2024-04-20T17:10:16.714923+0000","last_clean":"2024-04-20T17:10:16.714923+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:10:16.714923+0000","last_undegraded":"2024-04-20T17:10:16.714923+0000","last_fullsized":"2024-04-20T17:10:16.714923+0000","mapping_epoch":115,"log_start":"121'1500","ondisk_log_start":"121'1500","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'9786","last_scrub_stamp":"2024-04-20T17:09:42.012491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:09:42.012491+0000","objects_scrubbed":9731,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:11:03.726366+0000","scrub_duration":2202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":555941888,"num_objects":8483,"num_object_clones":0,"num_object_copies":16966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11559,"num_write_kb":641344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"158'11689","reported_seq":11888,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-20T17:10:17.398605+0000","last_change":"2024-04-20T17:09:38.307566+0000","last_active":"2024-04-20T17:10:17.398605+0000","last_peered":"2024-04-20T17:10:17.398605+0000","last_clean":"2024-04-20T17:10:17.398605+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:10:17.398605+0000","last_undegraded":"2024-04-20T17:10:17.398605+0000","last_fullsized":"2024-04-20T17:10:17.398605+0000","mapping_epoch":115,"log_start":"121'1600","ondisk_log_start":"121'1600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'9753","last_scrub_stamp":"2024-04-20T17:09:38.307455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:09:38.307455+0000","objects_scrubbed":9753,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:10:50.482990+0000","scrub_duration":756,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":559087616,"num_objects":8531,"num_object_clones":0,"num_object_copies":17062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11689,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"158'11556","reported_seq":11755,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-20T17:10:17.798599+0000","last_change":"2024-04-20T17:09:40.270114+0000","last_active":"2024-04-20T17:10:17.798599+0000","last_peered":"2024-04-20T17:10:17.798599+0000","last_clean":"2024-04-20T17:10:17.798599+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:10:17.798599+0000","last_undegraded":"2024-04-20T17:10:17.798599+0000","last_fullsized":"2024-04-20T17:10:17.798599+0000","mapping_epoch":115,"log_start":"121'1500","ondisk_log_start":"121'1500","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'9691","last_scrub_stamp":"2024-04-20T17:09:40.269994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:09:40.269994+0000","objects_scrubbed":9667,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:11:08.194290+0000","scrub_duration":1400,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":556793856,"num_objects":8496,"num_object_clones":0,"num_object_copies":16992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11556,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"158'11600","reported_seq":11801,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-20T17:10:16.765603+0000","last_change":"2024-04-20T17:09:36.433743+0000","last_active":"2024-04-20T17:10:16.765603+0000","last_peered":"2024-04-20T17:10:16.765603+0000","last_clean":"2024-04-20T17:10:16.765603+0000","last_became_active":"2024-04-20T17:08:06.010879+0000","last_became_peered":"2024-04-20T17:08:06.010879+0000","last_unstale":"2024-04-20T17:10:16.765603+0000","last_undegraded":"2024-04-20T17:10:16.765603+0000","last_fullsized":"2024-04-20T17:10:16.765603+0000","mapping_epoch":115,"log_start":"121'1500","ondisk_log_start":"121'1500","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'9713","last_scrub_stamp":"2024-04-20T17:09:36.433628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:09:36.433628+0000","objects_scrubbed":9684,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:10:37.181937+0000","scrub_duration":1511,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":567672832,"num_objects":8662,"num_object_clones":0,"num_object_copies":17324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":648384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"158'11691","reported_seq":11892,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-20T17:10:17.489106+0000","last_change":"2024-04-20T17:09:39.939901+0000","last_active":"2024-04-20T17:10:17.489106+0000","last_peered":"2024-04-20T17:10:17.489106+0000","last_clean":"2024-04-20T17:10:17.489106+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:10:17.489106+0000","last_undegraded":"2024-04-20T17:10:17.489106+0000","last_fullsized":"2024-04-20T17:10:17.489106+0000","mapping_epoch":115,"log_start":"121'1600","ondisk_log_start":"121'1600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'9825","last_scrub_stamp":"2024-04-20T17:09:39.939725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:09:39.939725+0000","objects_scrubbed":9787,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:10:49.080406+0000","scrub_duration":1483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":565248000,"num_objects":8625,"num_object_clones":0,"num_object_copies":17250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11691,"num_write_kb":650112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"157'11244","reported_seq":11441,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-20T17:10:13.637476+0000","last_change":"2024-04-20T17:09:39.236644+0000","last_active":"2024-04-20T17:10:13.637476+0000","last_peered":"2024-04-20T17:10:13.637476+0000","last_clean":"2024-04-20T17:10:13.637476+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:10:13.637476+0000","last_undegraded":"2024-04-20T17:10:13.637476+0000","last_fullsized":"2024-04-20T17:10:13.637476+0000","mapping_epoch":115,"log_start":"121'1200","ondisk_log_start":"121'1200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'9545","last_scrub_stamp":"2024-04-20T17:09:39.236546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:09:39.236546+0000","objects_scrubbed":9527,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:10:46.162514+0000","scrub_duration":1088,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":564920320,"num_objects":8620,"num_object_clones":0,"num_object_copies":17240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11244,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"158'11564","reported_seq":11764,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-04-20T17:10:17.433989+0000","last_change":"2024-04-20T17:09:41.990059+0000","last_active":"2024-04-20T17:10:17.433989+0000","last_peered":"2024-04-20T17:10:17.433989+0000","last_clean":"2024-04-20T17:10:17.433989+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:10:17.433989+0000","last_undegraded":"2024-04-20T17:10:17.433989+0000","last_fullsized":"2024-04-20T17:10:17.433989+0000","mapping_epoch":115,"log_start":"121'1500","ondisk_log_start":"121'1500","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'9783","last_scrub_stamp":"2024-04-20T17:09:41.989853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:09:41.989853+0000","objects_scrubbed":9721,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:10:53.012725+0000","scrub_duration":2567,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":559546368,"num_objects":8538,"num_object_clones":0,"num_object_copies":17076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11564,"num_write_kb":643264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"157'11277","reported_seq":11477,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-20T17:10:15.506852+0000","last_change":"2024-04-20T17:09:44.962738+0000","last_active":"2024-04-20T17:10:15.506852+0000","last_peered":"2024-04-20T17:10:15.506852+0000","last_clean":"2024-04-20T17:10:15.506852+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:10:15.506852+0000","last_undegraded":"2024-04-20T17:10:15.506852+0000","last_fullsized":"2024-04-20T17:10:15.506852+0000","mapping_epoch":115,"log_start":"121'1200","ondisk_log_start":"121'1200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'9863","last_scrub_stamp":"2024-04-20T17:09:44.962573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:09:44.962573+0000","objects_scrubbed":9826,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:11:11.494538+0000","scrub_duration":1521,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":563085336,"num_objects":8593,"num_object_clones":0,"num_object_copies":17186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8593,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11277,"num_write_kb":635777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"157'11269","reported_seq":11465,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-04-20T17:10:13.935912+0000","last_change":"2024-04-20T17:09:37.406864+0000","last_active":"2024-04-20T17:10:13.935912+0000","last_peered":"2024-04-20T17:10:13.935912+0000","last_clean":"2024-04-20T17:10:13.935912+0000","last_became_active":"2024-04-20T17:08:06.028272+0000","last_became_peered":"2024-04-20T17:08:06.028272+0000","last_unstale":"2024-04-20T17:10:13.935912+0000","last_undegraded":"2024-04-20T17:10:13.935912+0000","last_fullsized":"2024-04-20T17:10:13.935912+0000","mapping_epoch":115,"log_start":"121'1200","ondisk_log_start":"121'1200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'9589","last_scrub_stamp":"2024-04-20T17:09:37.406724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:09:37.406724+0000","objects_scrubbed":9589,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:10:38.400223+0000","scrub_duration":886,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":571932672,"num_objects":8727,"num_object_clones":0,"num_object_copies":17454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11269,"num_write_kb":639872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":9057533976,"num_objects":138208,"num_object_clones":0,"num_object_copies":276416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":17452,"num_objects_unfound":0,"num_objects_dirty":138208,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183776,"num_write_kb":10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5708,"num_bytes_recovered":374079488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18538438656,"data_stored":18538430512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161176,"ondisk_log_size":161176,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991287,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1707372,"kb_used_data":1644260,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92037780,"statfs":{"total":95995035648,"available":94246686720,"internally_reserved":0,"allocated":1683722240,"data_stored":1683375819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":64616900},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777252923,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1221608,"kb_used_data":1154088,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92523544,"statfs":{"total":95995035648,"available":94744109056,"internally_reserved":0,"allocated":1181786112,"data_stored":1181435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":68876736},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612674,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1723508,"kb_used_data":1660396,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92021644,"statfs":{"total":95995035648,"available":94230163456,"internally_reserved":0,"allocated":1700245504,"data_stored":1699891157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":64616896},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762808903,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2309480,"kb_used_data":2246184,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91435672,"statfs":{"total":95995035648,"available":93630128128,"internally_reserved":0,"allocated":2300092416,"data_stored":2299742141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168653,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1724424,"kb_used_data":1652964,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92020728,"statfs":{"total":95995035648,"available":94229225472,"internally_reserved":0,"allocated":1692635136,"data_stored":1692288735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430289,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":573888,"kb_used_data":546844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93171264,"statfs":{"total":95995035648,"available":95407374336,"internally_reserved":0,"allocated":559968256,"data_stored":559625407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574955,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2097024,"kb_used_data":2025564,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91648128,"statfs":{"total":95995035648,"available":93847683072,"internally_reserved":0,"allocated":2074177536,"data_stored":2073834687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738481,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1776360,"kb_used_data":1704744,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91968792,"statfs":{"total":95995035648,"available":94176043008,"internally_reserved":0,"allocated":1745657856,"data_stored":1745307581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":20,"seq":85899346020,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1733416,"kb_used_data":1665960,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92011736,"statfs":{"total":95995035648,"available":94220017664,"internally_reserved":0,"allocated":1705943040,"data_stored":1705592765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,7,5,2,3,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182496,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1748528,"kb_used_data":1672936,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91996624,"statfs":{"total":95995035648,"available":94204542976,"internally_reserved":0,"allocated":1713086464,"data_stored":1712736189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986267,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":590196,"kb_used_data":550700,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93154956,"statfs":{"total":95995035648,"available":95390674944,"internally_reserved":0,"allocated":563916800,"data_stored":563562453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790037,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1713584,"kb_used_data":1642152,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92031568,"statfs":{"total":95995035648,"available":94240325632,"internally_reserved":0,"allocated":1681563648,"data_stored":1681213373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1736704000,"data_stored":1736704000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2063990784,"data_stored":2063990784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1703936000,"data_stored":1703936000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1704591360,"data_stored":1704591360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":563220480,"data_stored":563216408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1681326080,"data_stored":1681326080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":559087616,"data_stored":559087616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1681653760,"data_stored":1681653760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":2289827840,"data_stored":2289827840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1699221504,"data_stored":1699217432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1175846912,"data_stored":1175846912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1679032320,"data_stored":1679032320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:10:21.269 INFO:tasks.thrashosds.thrasher:Setting 2.9 to [7, 10] 2024-04-20T17:10:21.269 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.9', '7', '10'] 2024-04-20T17:10:21.269 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pg-upmap 2.9 7 10 2024-04-20T17:10:21.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:20 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:10:21.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:20 smithi171 ceph-mon[44189]: from='client.24869 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:10:21.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:20 smithi171 ceph-mon[44189]: osdmap e160: 12 total, 12 up, 12 in 2024-04-20T17:10:21.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:20 smithi171 ceph-mon[44189]: pgmap v477: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s; 16469/272962 objects misplaced (6.033%); 12 MiB/s, 188 objects/s recovering 2024-04-20T17:10:21.641 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:10:21.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:21.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:10:21.738 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:10:21.756 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:10:21.788 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:21.797 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:10:21.814 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:10:21.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:21.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:10:21.918 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:10:21.966 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:22.003 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:10:22.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:22.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:10:22.112 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:10:22.142 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:22.162 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:10:22.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:22.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:10:22.265 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:10:22.313 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:22.433 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:10:22.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:22.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:10:22.540 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:10:22.614 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:22.784 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:10:22.802 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:10:22.803 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:10:23.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:23 smithi171 ceph-mon[44189]: pgmap v478: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 757 op/s; 16469/270956 objects misplaced (6.078%); 7.4 MiB/s, 119 objects/s recovering 2024-04-20T17:10:23.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:23.401 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:10:23.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:10:23.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:23 smithi005 ceph-mon[39562]: pgmap v478: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 757 op/s; 16469/270956 objects misplaced (6.078%); 7.4 MiB/s, 119 objects/s recovering 2024-04-20T17:10:23.502 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:10:23.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:23 smithi106 ceph-mon[44423]: pgmap v478: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 757 op/s; 16469/270956 objects misplaced (6.078%); 7.4 MiB/s, 119 objects/s recovering 2024-04-20T17:10:23.532 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:23.553 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:10:23.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:23.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:10:23.657 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:10:23.685 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:23.704 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:10:23.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:23.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:10:23.808 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:10:23.835 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:23.855 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:10:23.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:23.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:10:23.959 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:10:23.995 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:24.019 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:10:24.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:24.025 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:10:24.126 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:10:24.172 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:24.220 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:10:24.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:24.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:10:24.325 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:10:24.351 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:24.373 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:10:24.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:24.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:10:24.476 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:10:24.521 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:24.738 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:10:24.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:24.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:10:24.849 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:10:24.882 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:24.910 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:10:24.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:24.913 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:10:25.014 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:10:25.063 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:25.148 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:10:25.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:25.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:10:25.201 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-20T17:10:25.262 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:10:25.307 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:25 smithi005 ceph-mon[39562]: pgmap v479: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 835 op/s; 16469/266844 objects misplaced (6.172%); 5.7 MiB/s, 90 objects/s recovering 2024-04-20T17:10:25.307 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:25 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:10:25.307 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:25 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:10:25.316 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:25.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:25 smithi171 ceph-mon[44189]: pgmap v479: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 835 op/s; 16469/266844 objects misplaced (6.172%); 5.7 MiB/s, 90 objects/s recovering 2024-04-20T17:10:25.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:25 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:10:25.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:25 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:10:25.436 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:10:25.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:25.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:10:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:25 smithi106 ceph-mon[44423]: pgmap v479: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 835 op/s; 16469/266844 objects misplaced (6.172%); 5.7 MiB/s, 90 objects/s recovering 2024-04-20T17:10:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:25 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:10:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:25 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:10:25.516 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:10:25.541 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:10:25.568 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:25.721 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:10:25.768 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:10:25.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:25.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:10:25.817 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:10:25.852 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:10:25.876 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:10:25.893 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:10:25.911 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:25.921 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:10:25.959 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:10:25.990 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:10:26.029 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:10:26.060 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:10:26.106 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:10:26.143 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:10:26.173 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:10:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:26 smithi005 ceph-mon[39562]: osdmap e161: 12 total, 12 up, 12 in 2024-04-20T17:10:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:26 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/953787575' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:10:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:26 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:26.216 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:10:26.230 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:10:26.230 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:10:26.230 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:10:26.231 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:10:26.231 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:10:26.231 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:10:26.231 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:10:26.231 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:10:26.231 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:10:26.231 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:10:26.231 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:10:26.231 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:10:26.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:26 smithi106 ceph-mon[44423]: osdmap e161: 12 total, 12 up, 12 in 2024-04-20T17:10:26.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:26 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/953787575' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:10:26.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:26 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:26.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:26 smithi171 ceph-mon[44189]: osdmap e161: 12 total, 12 up, 12 in 2024-04-20T17:10:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:26 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/953787575' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:10:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:26 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:26.362 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:10:26.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:26.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:10:26.466 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:10:26.516 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:27.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:27 smithi106 ceph-mon[44423]: pgmap v481: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s; 15451/263482 objects misplaced (5.864%); 12 MiB/s, 186 objects/s recovering 2024-04-20T17:10:27.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.328 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.329 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.329 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.329 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.329 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:27 smithi005 ceph-mon[39562]: pgmap v481: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s; 15451/263482 objects misplaced (5.864%); 12 MiB/s, 186 objects/s recovering 2024-04-20T17:10:27.329 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.329 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.329 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.329 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.329 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.329 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.329 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:27 smithi171 ceph-mon[44189]: pgmap v481: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 8.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s; 15451/263482 objects misplaced (5.864%); 12 MiB/s, 186 objects/s recovering 2024-04-20T17:10:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:27.564 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:10:27.564 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:10:27.564 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:27.564 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:193698 2.d e161/115 2:bfe7a396:::benchmark_data_smithi171_78918_object32704:head v 161'11867, mlcod=161'11865)", 2024-04-20T17:10:27.564 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:27.563606+0000", 2024-04-20T17:10:27.564 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.001473715, 2024-04-20T17:10:27.565 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0014963190000000001, 2024-04-20T17:10:27.565 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:27.565 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:27.565 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-04-20T17:10:27.565 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:27.565 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:27.565 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:27.565 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:27.563606+0000", 2024-04-20T17:10:27.565 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:27.565 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:27.565 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:27.565 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:27.565 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:27.563606+0000", 2024-04-20T17:10:27.565 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:27.566 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:27.566 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:27.566 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:27.566 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:27.563608+0000", 2024-04-20T17:10:27.566 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.441e-06 2024-04-20T17:10:27.566 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:27.566 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:27.566 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:27.566 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:27.563619+0000", 2024-04-20T17:10:27.566 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0379e-05 2024-04-20T17:10:27.566 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:27.566 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:27.566 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:27.566 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:27.563634+0000", 2024-04-20T17:10:27.567 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5500999999999999e-05 2024-04-20T17:10:27.567 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:27.567 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:27.567 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:27.567 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:27.563645+0000", 2024-04-20T17:10:27.567 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0283e-05 2024-04-20T17:10:27.567 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:27.567 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:27.567 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:27.567 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:27.563724+0000", 2024-04-20T17:10:27.567 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9604000000000006e-05 2024-04-20T17:10:27.567 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:27.567 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:27.567 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:27.567 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:27.563740+0000", 2024-04-20T17:10:27.568 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6130999999999999e-05 2024-04-20T17:10:27.568 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:27.568 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:27.568 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:27.568 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:27.568 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-04-20T17:10:27.568 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 1 2024-04-20T17:10:27.568 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:10:28.074 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:10:28.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:28.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:10:28.183 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:10:28.211 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:28.231 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:10:28.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:28.234 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:10:28.334 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:10:28.350 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-20T17:10:28.360 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:28.378 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:10:28.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:28.381 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:10:28.482 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:10:28.512 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:28.611 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:10:28.761 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:10:28.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:28.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:10:28.864 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:10:28.896 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:28.929 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:10:28.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:28.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:10:29.034 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:10:29.066 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:28 smithi106 ceph-mon[44423]: osdmap e162: 12 total, 12 up, 12 in 2024-04-20T17:10:29.067 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:28 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:10:29.067 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:28 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:10:29.074 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:29.087 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:28 smithi005 ceph-mon[39562]: osdmap e162: 12 total, 12 up, 12 in 2024-04-20T17:10:29.088 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:28 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:10:29.088 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:28 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:10:29.099 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:10:29.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:29.104 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:10:29.205 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:10:29.234 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:29.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:28 smithi171 ceph-mon[44189]: osdmap e162: 12 total, 12 up, 12 in 2024-04-20T17:10:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:28 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:10:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:28 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:10:29.434 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:10:29.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:29.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:10:29.538 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:10:29.566 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:29.587 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:10:29.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:29.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:10:29.694 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:10:29.747 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:29.933 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:10:29.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:29.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:10:29.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:29 smithi005 ceph-mon[39562]: pgmap v483: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 7.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 826 op/s; 15451/259724 objects misplaced (5.949%); 6.0 MiB/s, 95 objects/s recovering 2024-04-20T17:10:29.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:29 smithi005 ceph-mon[39562]: osdmap e163: 12 total, 12 up, 12 in 2024-04-20T17:10:30.041 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:10:30.071 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:30.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:29 smithi106 ceph-mon[44423]: pgmap v483: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 7.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 826 op/s; 15451/259724 objects misplaced (5.949%); 6.0 MiB/s, 95 objects/s recovering 2024-04-20T17:10:30.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:29 smithi106 ceph-mon[44423]: osdmap e163: 12 total, 12 up, 12 in 2024-04-20T17:10:30.272 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:10:30.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:30.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:10:30.293 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:10:30.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:29 smithi171 ceph-mon[44189]: pgmap v483: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 7.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 826 op/s; 15451/259724 objects misplaced (5.949%); 6.0 MiB/s, 95 objects/s recovering 2024-04-20T17:10:30.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:29 smithi171 ceph-mon[44189]: osdmap e163: 12 total, 12 up, 12 in 2024-04-20T17:10:30.378 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:10:30.468 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:30.500 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:10:30.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:30.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:10:30.551 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:10:30.551 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:10:30.551 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-04-20T17:10:30.551 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-04-20T17:10:30.551 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:10:30.610 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:10:30.641 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:30.752 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:10:30.790 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:10:30.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:30.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:10:30.882 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:10:30.888 INFO:teuthology.orchestra.run.smithi005.stderr:set 2.9 pg_upmap mapping to [7,10] 2024-04-20T17:10:30.894 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:10:30.957 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:31.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:30 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/709229053' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:10:31.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:30 smithi005 ceph-mon[39562]: from='client.24878 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:10:31.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:30 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3446682635' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.9", "id": [7, 10]}]: dispatch 2024-04-20T17:10:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:30 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/709229053' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:10:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:30 smithi106 ceph-mon[44423]: from='client.24878 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:10:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:30 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3446682635' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.9", "id": [7, 10]}]: dispatch 2024-04-20T17:10:31.311 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-20T17:10:31.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:30 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/709229053' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:10:31.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:30 smithi171 ceph-mon[44189]: from='client.24878 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:10:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:30 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3446682635' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.9", "id": [7, 10]}]: dispatch 2024-04-20T17:10:31.568 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:10:32.104 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: pgmap v485: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 7.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 821 op/s; 14921/256974 objects misplaced (5.806%); 11 MiB/s, 171 objects/s recovering 2024-04-20T17:10:32.104 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:10:32.104 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: from='client.24878 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:10:32.104 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3446682635' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.9", "id": [7, 10]}]': finished 2024-04-20T17:10:32.104 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: osdmap e164: 12 total, 12 up, 12 in 2024-04-20T17:10:32.104 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=53,logm=4} crc {auth=1473329160,config=1460075751,health=1997216375,kv=3483784192,logm=522194934}) 2024-04-20T17:10:32.104 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=200644115}) 2024-04-20T17:10:32.105 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1007500435}) 2024-04-20T17:10:32.105 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=436468810}) 2024-04-20T17:10:32.105 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4111209462}) 2024-04-20T17:10:32.105 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=886681059}) 2024-04-20T17:10:32.105 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474581252}) 2024-04-20T17:10:32.105 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1889235499}) 2024-04-20T17:10:32.105 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3341591124}) 2024-04-20T17:10:32.105 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2935635388}) 2024-04-20T17:10:32.105 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3049217228}) 2024-04-20T17:10:32.105 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2050735444}) 2024-04-20T17:10:32.105 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3816016640}) 2024-04-20T17:10:32.105 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=823390865}) 2024-04-20T17:10:32.105 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=475184182}) 2024-04-20T17:10:32.105 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1608293772}) 2024-04-20T17:10:32.106 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3935314925}) 2024-04-20T17:10:32.106 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3148006791}) 2024-04-20T17:10:32.106 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2518991235}) 2024-04-20T17:10:32.106 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1136166705}) 2024-04-20T17:10:32.106 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3470644104}) 2024-04-20T17:10:32.106 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1907464442}) 2024-04-20T17:10:32.106 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2702449849}) 2024-04-20T17:10:32.106 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3642763275}) 2024-04-20T17:10:32.106 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3296691705}) 2024-04-20T17:10:32.106 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=71,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=5} crc {logm=1580899885,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=3918386640}) 2024-04-20T17:10:32.106 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=6,mon_config_key=94} crc {mgrstat=3344077316,mon_config_key=3685716827}) 2024-04-20T17:10:32.106 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=657250526}) 2024-04-20T17:10:32.106 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=96,monmap=4} crc {mon_config_key=3565758000,monmap=3953046592}) 2024-04-20T17:10:32.106 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {monmap=1,osd_metadata=12,osd_pg_creating=1,osdmap=86} crc {monmap=3678701417,osd_metadata=2755308478,osd_pg_creating=801759104,osdmap=207868114}) 2024-04-20T17:10:32.107 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:31 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=39} crc {osdmap=3879954672}) 2024-04-20T17:10:32.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: pgmap v485: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 7.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 821 op/s; 14921/256974 objects misplaced (5.806%); 11 MiB/s, 171 objects/s recovering 2024-04-20T17:10:32.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:10:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: from='client.24878 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:10:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3446682635' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.9", "id": [7, 10]}]': finished 2024-04-20T17:10:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: osdmap e164: 12 total, 12 up, 12 in 2024-04-20T17:10:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=53,logm=4} crc {auth=1473329160,config=1460075751,health=1997216375,kv=3483784192,logm=522194934}) 2024-04-20T17:10:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=200644115}) 2024-04-20T17:10:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1007500435}) 2024-04-20T17:10:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=436468810}) 2024-04-20T17:10:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4111209462}) 2024-04-20T17:10:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=886681059}) 2024-04-20T17:10:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474581252}) 2024-04-20T17:10:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1889235499}) 2024-04-20T17:10:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3341591124}) 2024-04-20T17:10:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2935635388}) 2024-04-20T17:10:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3049217228}) 2024-04-20T17:10:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2050735444}) 2024-04-20T17:10:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3816016640}) 2024-04-20T17:10:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=823390865}) 2024-04-20T17:10:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=475184182}) 2024-04-20T17:10:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1608293772}) 2024-04-20T17:10:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3935314925}) 2024-04-20T17:10:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3148006791}) 2024-04-20T17:10:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2518991235}) 2024-04-20T17:10:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1136166705}) 2024-04-20T17:10:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3470644104}) 2024-04-20T17:10:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1907464442}) 2024-04-20T17:10:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2702449849}) 2024-04-20T17:10:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3642763275}) 2024-04-20T17:10:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3296691705}) 2024-04-20T17:10:32.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=71,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=5} crc {logm=1580899885,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=3918386640}) 2024-04-20T17:10:32.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=6,mon_config_key=94} crc {mgrstat=3344077316,mon_config_key=3685716827}) 2024-04-20T17:10:32.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=657250526}) 2024-04-20T17:10:32.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=96,monmap=4} crc {mon_config_key=3565758000,monmap=3953046592}) 2024-04-20T17:10:32.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {monmap=1,osd_metadata=12,osd_pg_creating=1,osdmap=86} crc {monmap=3678701417,osd_metadata=2755308478,osd_pg_creating=801759104,osdmap=207868114}) 2024-04-20T17:10:32.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:31 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=39} crc {osdmap=3879954672}) 2024-04-20T17:10:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: pgmap v485: 17 pgs: 2 active+recovering+remapped, 15 active+clean; 7.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 821 op/s; 14921/256974 objects misplaced (5.806%); 11 MiB/s, 171 objects/s recovering 2024-04-20T17:10:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:10:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: from='client.24878 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:10:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3446682635' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.9", "id": [7, 10]}]': finished 2024-04-20T17:10:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: osdmap e164: 12 total, 12 up, 12 in 2024-04-20T17:10:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=53,logm=4} crc {auth=1473329160,config=1460075751,health=1997216375,kv=3483784192,logm=522194934}) 2024-04-20T17:10:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=200644115}) 2024-04-20T17:10:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1007500435}) 2024-04-20T17:10:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=436468810}) 2024-04-20T17:10:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4111209462}) 2024-04-20T17:10:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=886681059}) 2024-04-20T17:10:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474581252}) 2024-04-20T17:10:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1889235499}) 2024-04-20T17:10:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3341591124}) 2024-04-20T17:10:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2935635388}) 2024-04-20T17:10:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3049217228}) 2024-04-20T17:10:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2050735444}) 2024-04-20T17:10:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3816016640}) 2024-04-20T17:10:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=823390865}) 2024-04-20T17:10:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=475184182}) 2024-04-20T17:10:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1608293772}) 2024-04-20T17:10:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3935314925}) 2024-04-20T17:10:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3148006791}) 2024-04-20T17:10:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2518991235}) 2024-04-20T17:10:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1136166705}) 2024-04-20T17:10:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3470644104}) 2024-04-20T17:10:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1907464442}) 2024-04-20T17:10:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2702449849}) 2024-04-20T17:10:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3642763275}) 2024-04-20T17:10:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3296691705}) 2024-04-20T17:10:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=71,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=5} crc {logm=1580899885,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=3918386640}) 2024-04-20T17:10:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=6,mon_config_key=94} crc {mgrstat=3344077316,mon_config_key=3685716827}) 2024-04-20T17:10:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=657250526}) 2024-04-20T17:10:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=96,monmap=4} crc {mon_config_key=3565758000,monmap=3953046592}) 2024-04-20T17:10:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {monmap=1,osd_metadata=12,osd_pg_creating=1,osdmap=86} crc {monmap=3678701417,osd_metadata=2755308478,osd_pg_creating=801759104,osdmap=207868114}) 2024-04-20T17:10:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:31 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=39} crc {osdmap=3879954672}) 2024-04-20T17:10:33.100 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:32 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:10:33.100 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:32 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:10:33.100 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:32 smithi106 ceph-mon[44423]: osdmap e165: 12 total, 12 up, 12 in 2024-04-20T17:10:33.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:32 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:10:33.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:32 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:10:33.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:32 smithi005 ceph-mon[39562]: osdmap e165: 12 total, 12 up, 12 in 2024-04-20T17:10:33.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:32 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:10:33.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:32 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:10:33.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:32 smithi171 ceph-mon[44189]: osdmap e165: 12 total, 12 up, 12 in 2024-04-20T17:10:33.417 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:10:33.417 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-04-20T17:10:33.417 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-04-20T17:10:33.418 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:10:33.418 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.418 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:135893 2.d e142/115 2:b4f3b9bd:::benchmark_data_smithi171_78918_object135892:head v 142'8408, mlcod=142'8406)", 2024-04-20T17:10:33.418 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:09:21.013815+0000", 2024-04-20T17:10:33.418 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 72.401441094000006, 2024-04-20T17:10:33.418 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0077621640000000002, 2024-04-20T17:10:33.418 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:33.418 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:33.418 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:33.418 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:33.418 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.418 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:33.418 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.013815+0000", 2024-04-20T17:10:33.419 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.419 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.419 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.419 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:33.419 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.013815+0000", 2024-04-20T17:10:33.419 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.419 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.419 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.419 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:33.419 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.013817+0000", 2024-04-20T17:10:33.419 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2460000000000002e-06 2024-04-20T17:10:33.419 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.419 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.419 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:33.420 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.013992+0000", 2024-04-20T17:10:33.420 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00017494900000000001 2024-04-20T17:10:33.420 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.420 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.420 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:33.420 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.014012+0000", 2024-04-20T17:10:33.420 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9618000000000001e-05 2024-04-20T17:10:33.420 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.420 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.420 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:33.420 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.014021+0000", 2024-04-20T17:10:33.420 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4860000000000003e-06 2024-04-20T17:10:33.420 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.420 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.421 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:33.421 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.014092+0000", 2024-04-20T17:10:33.421 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0197999999999994e-05 2024-04-20T17:10:33.421 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.421 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.421 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:33.421 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.014114+0000", 2024-04-20T17:10:33.421 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2132000000000001e-05 2024-04-20T17:10:33.421 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.421 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.421 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:33.421 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.021524+0000", 2024-04-20T17:10:33.421 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0074102919999999997 2024-04-20T17:10:33.421 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.422 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.422 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:33.422 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.021577+0000", 2024-04-20T17:10:33.422 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3242999999999998e-05 2024-04-20T17:10:33.422 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.422 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:33.422 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.422 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.422 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.422 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:135900 2.d e142/115 2:b066403e:::benchmark_data_smithi171_78918_object135899:head v 142'8409, mlcod=142'8406)", 2024-04-20T17:10:33.422 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:09:21.017880+0000", 2024-04-20T17:10:33.422 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 72.397376475000002, 2024-04-20T17:10:33.422 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012669634000000001, 2024-04-20T17:10:33.422 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:33.423 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:33.423 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:33.423 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:33.423 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.423 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:33.423 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.017880+0000", 2024-04-20T17:10:33.423 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.423 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.423 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.423 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:33.423 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.017880+0000", 2024-04-20T17:10:33.423 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.423 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.424 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.424 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:33.424 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.017883+0000", 2024-04-20T17:10:33.424 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.196e-06 2024-04-20T17:10:33.424 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.424 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.424 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:33.424 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.017976+0000", 2024-04-20T17:10:33.424 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2625000000000002e-05 2024-04-20T17:10:33.424 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.424 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.424 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:33.424 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.017993+0000", 2024-04-20T17:10:33.424 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7181e-05 2024-04-20T17:10:33.424 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.424 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.425 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:33.425 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.018004+0000", 2024-04-20T17:10:33.425 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1474999999999999e-05 2024-04-20T17:10:33.425 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.425 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.425 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:33.425 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.018061+0000", 2024-04-20T17:10:33.425 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7043000000000003e-05 2024-04-20T17:10:33.425 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.425 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.425 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:33.425 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.018086+0000", 2024-04-20T17:10:33.425 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4368000000000001e-05 2024-04-20T17:10:33.425 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.426 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.426 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:33.426 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.030488+0000", 2024-04-20T17:10:33.426 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012402616999999999 2024-04-20T17:10:33.426 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.426 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.426 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:33.426 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.030549+0000", 2024-04-20T17:10:33.426 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1129000000000004e-05 2024-04-20T17:10:33.426 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.426 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:33.426 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.426 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.427 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.427 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:135901 2.d e142/115 2:b72cd141:::benchmark_data_smithi171_78918_object135900:head v 142'8410, mlcod=142'8406)", 2024-04-20T17:10:33.427 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:09:21.021973+0000", 2024-04-20T17:10:33.427 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 72.393283564000001, 2024-04-20T17:10:33.427 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0088093230000000008, 2024-04-20T17:10:33.427 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:33.427 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:33.427 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:33.427 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:33.427 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.427 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:33.427 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.021973+0000", 2024-04-20T17:10:33.427 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.427 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.428 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.428 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:33.428 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.021973+0000", 2024-04-20T17:10:33.428 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.428 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.428 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.428 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:33.428 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.021975+0000", 2024-04-20T17:10:33.428 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9099999999999999e-06 2024-04-20T17:10:33.428 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.428 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.428 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:33.428 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.022010+0000", 2024-04-20T17:10:33.428 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5830000000000001e-05 2024-04-20T17:10:33.429 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.429 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.429 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:33.429 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.022022+0000", 2024-04-20T17:10:33.429 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1402e-05 2024-04-20T17:10:33.429 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.429 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.429 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:33.429 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.022030+0000", 2024-04-20T17:10:33.429 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4789999999999996e-06 2024-04-20T17:10:33.429 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.429 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.429 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:33.429 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.027015+0000", 2024-04-20T17:10:33.429 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0049849730000000002 2024-04-20T17:10:33.430 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.430 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.430 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:33.430 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.027039+0000", 2024-04-20T17:10:33.430 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3873999999999999e-05 2024-04-20T17:10:33.430 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.430 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.430 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:33.430 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.030751+0000", 2024-04-20T17:10:33.430 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037120059999999999 2024-04-20T17:10:33.430 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.430 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.430 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:33.430 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.030782+0000", 2024-04-20T17:10:33.431 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0849000000000002e-05 2024-04-20T17:10:33.431 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.431 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:33.431 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.431 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.431 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.431 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:136713 2.d e142/115 2:b651f5f4:::benchmark_data_smithi171_78918_object136712:head v 142'8450, mlcod=142'8448)", 2024-04-20T17:10:33.431 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:09:21.626332+0000", 2024-04-20T17:10:33.431 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 71.788924097999995, 2024-04-20T17:10:33.431 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0088084979999999997, 2024-04-20T17:10:33.431 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:33.431 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:33.431 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:33.431 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:33.432 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.432 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:33.432 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.626332+0000", 2024-04-20T17:10:33.432 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.432 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.432 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.432 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:33.432 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.626332+0000", 2024-04-20T17:10:33.432 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.432 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.432 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.432 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:33.432 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.626335+0000", 2024-04-20T17:10:33.432 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.407e-06 2024-04-20T17:10:33.433 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.433 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.433 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:33.433 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.626390+0000", 2024-04-20T17:10:33.433 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5377000000000002e-05 2024-04-20T17:10:33.433 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.433 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.433 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:33.433 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.626409+0000", 2024-04-20T17:10:33.433 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9065000000000001e-05 2024-04-20T17:10:33.433 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.433 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.433 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:33.433 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.626416+0000", 2024-04-20T17:10:33.434 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0060000000000003e-06 2024-04-20T17:10:33.434 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.434 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.434 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:33.434 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.626470+0000", 2024-04-20T17:10:33.434 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4233000000000001e-05 2024-04-20T17:10:33.434 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.434 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.434 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:33.434 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.626487+0000", 2024-04-20T17:10:33.434 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6656000000000001e-05 2024-04-20T17:10:33.434 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.434 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.434 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:33.435 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.635096+0000", 2024-04-20T17:10:33.435 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0086095640000000001 2024-04-20T17:10:33.435 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.435 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.435 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:33.435 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.635141+0000", 2024-04-20T17:10:33.435 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4190000000000002e-05 2024-04-20T17:10:33.435 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.435 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:33.435 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.435 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.435 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.435 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:136715 2.d e142/115 2:ba482ea7:::benchmark_data_smithi171_78918_object136714:head v 142'8451, mlcod=142'8448)", 2024-04-20T17:10:33.436 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:09:21.627185+0000", 2024-04-20T17:10:33.436 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 71.788071295999998, 2024-04-20T17:10:33.436 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0079756380000000002, 2024-04-20T17:10:33.436 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:33.436 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:33.436 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:33.436 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:33.436 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.436 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:33.436 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.627185+0000", 2024-04-20T17:10:33.436 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.436 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.436 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.436 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:33.437 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.627185+0000", 2024-04-20T17:10:33.437 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.437 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.437 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.437 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:33.437 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.627190+0000", 2024-04-20T17:10:33.437 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8199999999999996e-06 2024-04-20T17:10:33.437 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.437 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.437 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:33.437 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.627283+0000", 2024-04-20T17:10:33.437 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3307000000000006e-05 2024-04-20T17:10:33.437 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.437 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.437 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:33.438 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.627293+0000", 2024-04-20T17:10:33.438 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6250000000000002e-06 2024-04-20T17:10:33.438 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.438 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.438 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:33.438 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.627300+0000", 2024-04-20T17:10:33.438 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9800000000000001e-06 2024-04-20T17:10:33.438 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.438 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.438 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:33.438 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.629635+0000", 2024-04-20T17:10:33.438 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0023352830000000001 2024-04-20T17:10:33.438 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.438 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.438 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:33.439 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.629651+0000", 2024-04-20T17:10:33.439 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5747999999999999e-05 2024-04-20T17:10:33.439 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.439 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.439 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:33.439 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.635145+0000", 2024-04-20T17:10:33.439 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0054941019999999998 2024-04-20T17:10:33.439 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.439 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.439 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:33.439 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.635161+0000", 2024-04-20T17:10:33.440 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5773e-05 2024-04-20T17:10:33.440 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.440 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:33.440 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.440 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.440 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.440 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:137322 2.d e142/115 2:b9ec57db:::benchmark_data_smithi171_78918_object137321:head v 142'8488, mlcod=142'8485)", 2024-04-20T17:10:33.440 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:09:22.079067+0000", 2024-04-20T17:10:33.440 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 71.336189532999995, 2024-04-20T17:10:33.440 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.009095611, 2024-04-20T17:10:33.440 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:33.440 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:33.440 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:33.440 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:33.440 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.441 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:33.441 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:22.079067+0000", 2024-04-20T17:10:33.441 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.441 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.441 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.441 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:33.441 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:22.079067+0000", 2024-04-20T17:10:33.441 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.441 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.441 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.441 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:33.441 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:22.079068+0000", 2024-04-20T17:10:33.441 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5409999999999999e-06 2024-04-20T17:10:33.441 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.442 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.442 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:33.442 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:22.079129+0000", 2024-04-20T17:10:33.442 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0946000000000002e-05 2024-04-20T17:10:33.442 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.442 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.442 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:33.442 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:22.079137+0000", 2024-04-20T17:10:33.442 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9529999999999999e-06 2024-04-20T17:10:33.442 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.442 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.442 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:33.442 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:22.079145+0000", 2024-04-20T17:10:33.442 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5979999999999999e-06 2024-04-20T17:10:33.442 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.443 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.443 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:33.443 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:22.081774+0000", 2024-04-20T17:10:33.443 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0026295110000000002 2024-04-20T17:10:33.443 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.443 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.443 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:33.443 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:22.081789+0000", 2024-04-20T17:10:33.443 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4388e-05 2024-04-20T17:10:33.443 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.443 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.443 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:33.443 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:22.088109+0000", 2024-04-20T17:10:33.443 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0063202220000000003 2024-04-20T17:10:33.443 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.443 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.444 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:33.444 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:22.088162+0000", 2024-04-20T17:10:33.444 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3452000000000003e-05 2024-04-20T17:10:33.444 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.444 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:33.444 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.444 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.444 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.444 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:137420 2.d e142/115 2:ba7d949b:::benchmark_data_smithi171_78918_object137419:head v 142'8494, mlcod=142'8490)", 2024-04-20T17:10:33.444 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:09:22.157519+0000", 2024-04-20T17:10:33.444 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 71.257736769000005, 2024-04-20T17:10:33.444 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0079286410000000002, 2024-04-20T17:10:33.444 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:33.444 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:33.444 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:33.445 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:33.445 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.445 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:33.445 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:22.157519+0000", 2024-04-20T17:10:33.445 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.445 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.445 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.445 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:33.445 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:22.157519+0000", 2024-04-20T17:10:33.445 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.445 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.445 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.445 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:33.445 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:22.157520+0000", 2024-04-20T17:10:33.445 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0100000000000004e-07 2024-04-20T17:10:33.446 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.446 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.446 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:33.446 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:22.157554+0000", 2024-04-20T17:10:33.446 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3858e-05 2024-04-20T17:10:33.446 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.446 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.446 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:33.446 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:22.157560+0000", 2024-04-20T17:10:33.446 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8259999999999998e-06 2024-04-20T17:10:33.446 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.446 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.446 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:33.446 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:22.157565+0000", 2024-04-20T17:10:33.447 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7530000000000001e-06 2024-04-20T17:10:33.447 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.447 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.447 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:33.447 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:22.160003+0000", 2024-04-20T17:10:33.447 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002438001 2024-04-20T17:10:33.447 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.447 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.447 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:33.447 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:22.160024+0000", 2024-04-20T17:10:33.447 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1776e-05 2024-04-20T17:10:33.447 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.447 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.447 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:33.447 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:22.165403+0000", 2024-04-20T17:10:33.448 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0053790349999999999 2024-04-20T17:10:33.448 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.448 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.448 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:33.448 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:22.165448+0000", 2024-04-20T17:10:33.448 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4591e-05 2024-04-20T17:10:33.448 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.448 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:33.448 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.448 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.448 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.448 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:138877 2.d e143/115 2:b0e7a751:::benchmark_data_smithi171_78918_object138876:head v 143'8589, mlcod=143'8586)", 2024-04-20T17:10:33.448 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:09:23.304417+0000", 2024-04-20T17:10:33.448 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 70.110838959999995, 2024-04-20T17:10:33.448 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0079632790000000002, 2024-04-20T17:10:33.449 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:33.449 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:33.449 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:33.449 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:33.449 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.449 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:33.449 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.304417+0000", 2024-04-20T17:10:33.449 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.449 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.449 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.449 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:33.449 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.304417+0000", 2024-04-20T17:10:33.449 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.449 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.450 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.450 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:33.450 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.304419+0000", 2024-04-20T17:10:33.450 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.457e-06 2024-04-20T17:10:33.450 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.450 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.450 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:33.450 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.305087+0000", 2024-04-20T17:10:33.450 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00066843899999999995 2024-04-20T17:10:33.450 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.450 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.450 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:33.450 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.305099+0000", 2024-04-20T17:10:33.450 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1883e-05 2024-04-20T17:10:33.451 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.451 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.451 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:33.451 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.305108+0000", 2024-04-20T17:10:33.451 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2359999999999993e-06 2024-04-20T17:10:33.451 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.451 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.451 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:33.451 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.305179+0000", 2024-04-20T17:10:33.451 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1173000000000004e-05 2024-04-20T17:10:33.451 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.451 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.451 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:33.451 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.305201+0000", 2024-04-20T17:10:33.451 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1322e-05 2024-04-20T17:10:33.452 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.452 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.452 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:33.452 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.312320+0000", 2024-04-20T17:10:33.452 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0071195659999999999 2024-04-20T17:10:33.452 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.452 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.452 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:33.452 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.312381+0000", 2024-04-20T17:10:33.452 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0202999999999999e-05 2024-04-20T17:10:33.452 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.452 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:33.452 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.452 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.453 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.453 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:138919 2.d e143/115 2:b227306f:::benchmark_data_smithi171_78918_object138918:head v 143'8595, mlcod=143'8592)", 2024-04-20T17:10:33.453 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:09:23.338496+0000", 2024-04-20T17:10:33.453 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 70.076759918999997, 2024-04-20T17:10:33.453 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0098117770000000007, 2024-04-20T17:10:33.453 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:33.453 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:33.453 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:33.453 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:33.453 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.453 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:33.453 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338496+0000", 2024-04-20T17:10:33.453 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.453 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.453 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.454 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:33.454 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338496+0000", 2024-04-20T17:10:33.454 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.454 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.454 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.454 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:33.454 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338499+0000", 2024-04-20T17:10:33.454 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4700000000000001e-06 2024-04-20T17:10:33.454 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.454 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.454 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:33.454 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338647+0000", 2024-04-20T17:10:33.454 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014819599999999999 2024-04-20T17:10:33.454 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.455 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.455 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:33.455 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338663+0000", 2024-04-20T17:10:33.455 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6532e-05 2024-04-20T17:10:33.455 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.455 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.455 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:33.455 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338671+0000", 2024-04-20T17:10:33.455 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1779999999999997e-06 2024-04-20T17:10:33.455 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.455 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.455 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:33.455 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338719+0000", 2024-04-20T17:10:33.455 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8152000000000002e-05 2024-04-20T17:10:33.455 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.456 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.456 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:33.456 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338737+0000", 2024-04-20T17:10:33.456 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8117e-05 2024-04-20T17:10:33.456 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.456 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.456 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:33.456 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.348258+0000", 2024-04-20T17:10:33.456 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0095206400000000004 2024-04-20T17:10:33.456 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.456 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.456 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:33.456 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.348308+0000", 2024-04-20T17:10:33.456 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0491999999999997e-05 2024-04-20T17:10:33.456 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.457 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:33.457 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.457 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.457 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.457 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:138920 2.d e143/115 2:ba4e1b76:::benchmark_data_smithi171_78918_object138919:head v 143'8596, mlcod=143'8593)", 2024-04-20T17:10:33.457 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:09:23.338925+0000", 2024-04-20T17:10:33.457 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 70.076331155999995, 2024-04-20T17:10:33.457 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0094244870000000005, 2024-04-20T17:10:33.457 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:33.457 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:33.457 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:33.457 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:33.457 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.457 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:33.457 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338925+0000", 2024-04-20T17:10:33.457 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.457 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.458 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.458 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:33.458 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338925+0000", 2024-04-20T17:10:33.458 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.458 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.458 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.458 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:33.458 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338926+0000", 2024-04-20T17:10:33.458 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6700000000000002e-07 2024-04-20T17:10:33.458 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.458 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.458 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:33.458 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.339107+0000", 2024-04-20T17:10:33.458 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000181397 2024-04-20T17:10:33.459 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.459 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.459 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:33.459 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.339116+0000", 2024-04-20T17:10:33.459 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9600000000000006e-06 2024-04-20T17:10:33.459 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.459 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.459 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:33.459 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.339123+0000", 2024-04-20T17:10:33.459 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1250000000000004e-06 2024-04-20T17:10:33.459 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.459 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.459 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:33.459 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.341580+0000", 2024-04-20T17:10:33.459 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0024563979999999998 2024-04-20T17:10:33.460 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.460 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.460 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:33.460 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.341606+0000", 2024-04-20T17:10:33.460 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5919999999999999e-05 2024-04-20T17:10:33.460 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.460 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.460 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:33.460 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.348316+0000", 2024-04-20T17:10:33.460 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0067097709999999998 2024-04-20T17:10:33.460 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.460 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.460 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:33.460 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.348350+0000", 2024-04-20T17:10:33.460 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3949000000000003e-05 2024-04-20T17:10:33.461 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.461 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:33.461 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.461 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.461 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.461 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:138924 2.d e143/115 2:b7ae273a:::benchmark_data_smithi171_78918_object138923:head v 143'8597, mlcod=143'8593)", 2024-04-20T17:10:33.461 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:09:23.342603+0000", 2024-04-20T17:10:33.461 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 70.072653649000003, 2024-04-20T17:10:33.461 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0090249479999999997, 2024-04-20T17:10:33.461 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:33.461 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:33.461 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:33.461 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:33.461 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.461 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:33.462 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.342603+0000", 2024-04-20T17:10:33.462 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.462 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.462 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.462 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:33.462 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.342603+0000", 2024-04-20T17:10:33.462 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.462 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.462 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.462 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:33.462 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.342605+0000", 2024-04-20T17:10:33.462 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8459999999999999e-06 2024-04-20T17:10:33.462 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.462 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.462 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:33.463 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.342740+0000", 2024-04-20T17:10:33.463 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000134191 2024-04-20T17:10:33.463 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.463 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.463 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:33.463 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.342755+0000", 2024-04-20T17:10:33.463 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5760000000000002e-05 2024-04-20T17:10:33.463 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.463 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.463 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:33.463 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.342766+0000", 2024-04-20T17:10:33.463 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0574000000000001e-05 2024-04-20T17:10:33.463 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.463 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.464 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:33.464 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.344686+0000", 2024-04-20T17:10:33.464 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001919612 2024-04-20T17:10:33.464 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.464 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.464 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:33.464 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.344707+0000", 2024-04-20T17:10:33.464 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1489e-05 2024-04-20T17:10:33.464 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.464 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.464 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:33.464 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.351577+0000", 2024-04-20T17:10:33.464 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0068703879999999998 2024-04-20T17:10:33.464 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.465 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.465 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:33.465 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.351627+0000", 2024-04-20T17:10:33.465 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0087999999999999e-05 2024-04-20T17:10:33.465 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.465 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:33.465 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.465 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.465 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.465 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:139935 2.d e143/115 2:b6d73f0f:::benchmark_data_smithi171_78918_object139934:head v 143'8658, mlcod=143'8655)", 2024-04-20T17:10:33.465 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:09:24.080675+0000", 2024-04-20T17:10:33.465 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 69.334581686999996, 2024-04-20T17:10:33.465 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0088639749999999996, 2024-04-20T17:10:33.465 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:33.466 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:33.466 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:33.466 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:33.466 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.466 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:33.466 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:24.080675+0000", 2024-04-20T17:10:33.466 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.466 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.466 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.466 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:33.466 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:24.080675+0000", 2024-04-20T17:10:33.466 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.466 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.466 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.466 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:33.467 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:24.080676+0000", 2024-04-20T17:10:33.467 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.575e-06 2024-04-20T17:10:33.467 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.467 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.467 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:33.467 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:24.080774+0000", 2024-04-20T17:10:33.467 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7695000000000006e-05 2024-04-20T17:10:33.467 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.467 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.467 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:33.467 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:24.080785+0000", 2024-04-20T17:10:33.467 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0818e-05 2024-04-20T17:10:33.467 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.467 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.467 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:33.468 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:24.080791+0000", 2024-04-20T17:10:33.468 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0990000000000004e-06 2024-04-20T17:10:33.468 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.468 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.468 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:33.468 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:24.080835+0000", 2024-04-20T17:10:33.468 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4416000000000003e-05 2024-04-20T17:10:33.468 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.468 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.468 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:33.468 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:24.080850+0000", 2024-04-20T17:10:33.468 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4568e-05 2024-04-20T17:10:33.468 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.468 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.468 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:33.469 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:24.089499+0000", 2024-04-20T17:10:33.469 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0086491829999999995 2024-04-20T17:10:33.469 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.469 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.469 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:33.469 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:24.089538+0000", 2024-04-20T17:10:33.469 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9620999999999999e-05 2024-04-20T17:10:33.469 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.469 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:33.469 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.469 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.469 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.469 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:139943 2.d e143/115 2:b5777e5c:::benchmark_data_smithi171_78918_object139942:head v 143'8659, mlcod=143'8656)", 2024-04-20T17:10:33.469 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:09:24.081037+0000", 2024-04-20T17:10:33.469 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 69.334218985000007, 2024-04-20T17:10:33.470 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0085207809999999998, 2024-04-20T17:10:33.470 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:33.470 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:33.470 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:33.470 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:33.470 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.470 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:33.470 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:24.081037+0000", 2024-04-20T17:10:33.470 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.470 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.470 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.470 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:33.470 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:24.081037+0000", 2024-04-20T17:10:33.470 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.470 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.471 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.471 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:33.471 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:24.081038+0000", 2024-04-20T17:10:33.471 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0240000000000001e-06 2024-04-20T17:10:33.471 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.471 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.471 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:33.471 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:24.081105+0000", 2024-04-20T17:10:33.471 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6839000000000002e-05 2024-04-20T17:10:33.471 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.471 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.471 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:33.471 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:24.081113+0000", 2024-04-20T17:10:33.471 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3119999999999994e-06 2024-04-20T17:10:33.471 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.472 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.472 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:33.472 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:24.081119+0000", 2024-04-20T17:10:33.472 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.896e-06 2024-04-20T17:10:33.472 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.472 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.472 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:33.472 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:24.083435+0000", 2024-04-20T17:10:33.472 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0023155469999999998 2024-04-20T17:10:33.472 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.472 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.472 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:33.472 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:24.083458+0000", 2024-04-20T17:10:33.472 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3152000000000001e-05 2024-04-20T17:10:33.472 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.472 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.473 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:33.473 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:24.089542+0000", 2024-04-20T17:10:33.473 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0060837560000000001 2024-04-20T17:10:33.473 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.473 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.473 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:33.473 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:24.089558+0000", 2024-04-20T17:10:33.473 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6254999999999999e-05 2024-04-20T17:10:33.473 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.473 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:33.473 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.473 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.473 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.473 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:145115 2.d e144/115 2:b248f228:::benchmark_data_smithi171_78918_object145114:head v 144'9005, mlcod=144'9003)", 2024-04-20T17:10:33.473 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:09:28.271599+0000", 2024-04-20T17:10:33.474 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 65.143657372999996, 2024-04-20T17:10:33.474 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0078175069999999996, 2024-04-20T17:10:33.474 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:33.474 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:33.474 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:33.474 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:33.474 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.474 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:33.474 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.271599+0000", 2024-04-20T17:10:33.474 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.474 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.474 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.474 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:33.474 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.271599+0000", 2024-04-20T17:10:33.474 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.475 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.475 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.475 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:33.475 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.271601+0000", 2024-04-20T17:10:33.475 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1270000000000001e-06 2024-04-20T17:10:33.475 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.475 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.475 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:33.475 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.271754+0000", 2024-04-20T17:10:33.475 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000153389 2024-04-20T17:10:33.475 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.475 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.475 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:33.475 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.271770+0000", 2024-04-20T17:10:33.475 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6157000000000001e-05 2024-04-20T17:10:33.476 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.476 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.476 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:33.476 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.271780+0000", 2024-04-20T17:10:33.476 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1500000000000005e-06 2024-04-20T17:10:33.476 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.476 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.476 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:33.476 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.271835+0000", 2024-04-20T17:10:33.476 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5229000000000003e-05 2024-04-20T17:10:33.476 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.476 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.476 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:33.476 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.271857+0000", 2024-04-20T17:10:33.476 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1855e-05 2024-04-20T17:10:33.477 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.477 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.477 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:33.477 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.279376+0000", 2024-04-20T17:10:33.477 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0075196949999999999 2024-04-20T17:10:33.477 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.477 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.477 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:33.477 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.279416+0000", 2024-04-20T17:10:33.477 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9904999999999998e-05 2024-04-20T17:10:33.477 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.477 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:33.477 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.477 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.477 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.477 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:145540 2.d e144/115 2:bbadbea3:::benchmark_data_smithi171_78918_object145539:head v 144'9033, mlcod=144'9030)", 2024-04-20T17:10:33.478 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:09:28.610761+0000", 2024-04-20T17:10:33.478 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 64.804495368000005, 2024-04-20T17:10:33.478 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0095878439999999999, 2024-04-20T17:10:33.478 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:33.478 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:33.478 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:33.478 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:33.478 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.478 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:33.478 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.610761+0000", 2024-04-20T17:10:33.478 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.478 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.478 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.478 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:33.479 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.610761+0000", 2024-04-20T17:10:33.479 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.479 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.479 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.479 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:33.479 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.610763+0000", 2024-04-20T17:10:33.479 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8649999999999999e-06 2024-04-20T17:10:33.479 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.479 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.479 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:33.479 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.610874+0000", 2024-04-20T17:10:33.479 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000111185 2024-04-20T17:10:33.479 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.479 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.479 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:33.479 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.610886+0000", 2024-04-20T17:10:33.480 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2379e-05 2024-04-20T17:10:33.480 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.480 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.480 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:33.480 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.610895+0000", 2024-04-20T17:10:33.480 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3210000000000005e-06 2024-04-20T17:10:33.480 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.480 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.480 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:33.480 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.613942+0000", 2024-04-20T17:10:33.480 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003047892 2024-04-20T17:10:33.480 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.480 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.480 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:33.480 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.613966+0000", 2024-04-20T17:10:33.480 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4000999999999999e-05 2024-04-20T17:10:33.481 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.481 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.481 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:33.481 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.620302+0000", 2024-04-20T17:10:33.481 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0063351930000000002 2024-04-20T17:10:33.481 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.481 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.481 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:33.481 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.620349+0000", 2024-04-20T17:10:33.481 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7008000000000001e-05 2024-04-20T17:10:33.481 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.481 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:33.481 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.481 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.481 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.481 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:146284 2.d e145/115 2:ba16209d:::benchmark_data_smithi171_78918_object146283:head v 145'9080, mlcod=145'9077)", 2024-04-20T17:10:33.482 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:09:29.190321+0000", 2024-04-20T17:10:33.482 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 64.224935212999995, 2024-04-20T17:10:33.482 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.009650166, 2024-04-20T17:10:33.482 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:33.482 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:33.482 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:33.482 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:33.482 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.482 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:33.482 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.190321+0000", 2024-04-20T17:10:33.482 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.482 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.482 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.482 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:33.483 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.190321+0000", 2024-04-20T17:10:33.483 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.483 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.483 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.483 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:33.483 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.190322+0000", 2024-04-20T17:10:33.483 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2500000000000005e-07 2024-04-20T17:10:33.483 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.483 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.483 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:33.483 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.190371+0000", 2024-04-20T17:10:33.483 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9458e-05 2024-04-20T17:10:33.483 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.483 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.483 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:33.484 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.190376+0000", 2024-04-20T17:10:33.484 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5619999999999997e-06 2024-04-20T17:10:33.484 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.484 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.485 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:33.485 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.190379+0000", 2024-04-20T17:10:33.485 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.557e-06 2024-04-20T17:10:33.485 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.485 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.485 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:33.485 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.193010+0000", 2024-04-20T17:10:33.485 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0026303440000000002 2024-04-20T17:10:33.485 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.485 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.485 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:33.486 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.193033+0000", 2024-04-20T17:10:33.486 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3027000000000002e-05 2024-04-20T17:10:33.486 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.486 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.486 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:33.486 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.199918+0000", 2024-04-20T17:10:33.486 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0068856550000000001 2024-04-20T17:10:33.486 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.486 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.486 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:33.486 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.199971+0000", 2024-04-20T17:10:33.486 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2837999999999998e-05 2024-04-20T17:10:33.486 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.487 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:33.487 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.487 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.487 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.487 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:146791 2.d e145/115 2:b71aaf1f:::benchmark_data_smithi171_78918_object146790:head v 145'9113, mlcod=145'9111)", 2024-04-20T17:10:33.487 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:09:29.574392+0000", 2024-04-20T17:10:33.487 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 63.840863732000003, 2024-04-20T17:10:33.487 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0079932820000000009, 2024-04-20T17:10:33.487 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:33.487 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:33.487 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:33.487 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:33.487 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.487 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:33.487 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.574392+0000", 2024-04-20T17:10:33.488 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.488 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.488 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.488 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:33.488 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.574392+0000", 2024-04-20T17:10:33.488 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.488 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.488 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.488 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:33.488 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.574394+0000", 2024-04-20T17:10:33.488 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.584e-06 2024-04-20T17:10:33.488 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.488 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.488 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:33.488 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.574521+0000", 2024-04-20T17:10:33.489 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012711499999999999 2024-04-20T17:10:33.489 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.489 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.489 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:33.489 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.574531+0000", 2024-04-20T17:10:33.489 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8840000000000006e-06 2024-04-20T17:10:33.489 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.489 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.489 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:33.489 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.574537+0000", 2024-04-20T17:10:33.489 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0639999999999999e-06 2024-04-20T17:10:33.489 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.489 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.490 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:33.490 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.574587+0000", 2024-04-20T17:10:33.490 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0135e-05 2024-04-20T17:10:33.490 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.490 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.490 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:33.490 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.574610+0000", 2024-04-20T17:10:33.490 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2354e-05 2024-04-20T17:10:33.490 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.490 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.490 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:33.490 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.582334+0000", 2024-04-20T17:10:33.490 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0077245100000000004 2024-04-20T17:10:33.490 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.491 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.491 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:33.491 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.582386+0000", 2024-04-20T17:10:33.491 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1635999999999998e-05 2024-04-20T17:10:33.491 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.491 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:33.491 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.491 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.491 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.491 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:180296 2.d e156/115 2:b3863235:::benchmark_data_smithi171_78918_object19302:head v 156'11076, mlcod=156'11074)", 2024-04-20T17:10:33.491 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:11.606812+0000", 2024-04-20T17:10:33.491 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 21.808444430000002, 2024-04-20T17:10:33.491 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010515645000000001, 2024-04-20T17:10:33.491 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:33.492 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:33.492 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:33.492 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:33.492 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.492 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:33.492 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.606812+0000", 2024-04-20T17:10:33.492 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.492 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.492 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.492 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:33.492 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.606812+0000", 2024-04-20T17:10:33.492 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.492 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.492 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.493 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:33.493 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.606814+0000", 2024-04-20T17:10:33.493 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2819999999999999e-06 2024-04-20T17:10:33.493 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.493 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.493 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:33.493 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.606822+0000", 2024-04-20T17:10:33.493 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0700000000000007e-06 2024-04-20T17:10:33.493 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.493 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.493 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:33.493 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.606835+0000", 2024-04-20T17:10:33.493 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3154e-05 2024-04-20T17:10:33.493 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.493 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.494 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:33.494 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.606843+0000", 2024-04-20T17:10:33.494 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2290000000000003e-06 2024-04-20T17:10:33.494 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.494 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.494 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:33.494 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.606904+0000", 2024-04-20T17:10:33.494 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0714000000000002e-05 2024-04-20T17:10:33.494 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.494 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.494 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:33.494 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.606928+0000", 2024-04-20T17:10:33.494 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.336e-05 2024-04-20T17:10:33.494 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.495 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.495 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:33.495 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.617278+0000", 2024-04-20T17:10:33.495 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010350597 2024-04-20T17:10:33.495 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.495 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.495 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:33.495 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.617327+0000", 2024-04-20T17:10:33.495 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9239000000000001e-05 2024-04-20T17:10:33.495 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.495 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:33.495 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.495 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.495 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.495 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:196241 2.d e166/115 2:bf40b2e5:::benchmark_data_smithi171_78918_object35247:head v 166'12028, mlcod=164'12025)", 2024-04-20T17:10:33.495 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:33.095025+0000", 2024-04-20T17:10:33.496 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.32023107899999997, 2024-04-20T17:10:33.496 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0081707010000000007, 2024-04-20T17:10:33.496 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:33.496 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:33.496 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:33.496 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:33.496 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.496 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:33.496 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.095025+0000", 2024-04-20T17:10:33.496 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.496 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.496 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.496 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:33.496 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.095025+0000", 2024-04-20T17:10:33.496 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.497 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.497 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.497 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:33.497 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.095026+0000", 2024-04-20T17:10:33.497 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2759999999999999e-06 2024-04-20T17:10:33.497 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.497 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.497 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:33.497 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.095033+0000", 2024-04-20T17:10:33.497 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6139999999999997e-06 2024-04-20T17:10:33.497 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.497 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.497 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:33.497 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.095045+0000", 2024-04-20T17:10:33.497 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2179e-05 2024-04-20T17:10:33.497 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.498 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.498 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:33.498 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.095055+0000", 2024-04-20T17:10:33.498 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0186999999999999e-05 2024-04-20T17:10:33.498 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.498 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.498 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:33.498 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.095109+0000", 2024-04-20T17:10:33.498 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3859000000000001e-05 2024-04-20T17:10:33.498 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.498 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.498 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:33.498 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.095132+0000", 2024-04-20T17:10:33.498 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3136e-05 2024-04-20T17:10:33.498 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.499 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.499 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:33.499 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.103151+0000", 2024-04-20T17:10:33.499 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0080181839999999994 2024-04-20T17:10:33.499 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.499 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.499 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:33.499 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.103196+0000", 2024-04-20T17:10:33.499 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5266000000000003e-05 2024-04-20T17:10:33.499 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.499 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:33.499 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.499 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.499 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.500 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:196493 2.d e166/115 2:bd3d8133:::benchmark_data_smithi171_78918_object35499:head v 166'12053, mlcod=166'12050)", 2024-04-20T17:10:33.500 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:33.327689+0000", 2024-04-20T17:10:33.500 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.087566799000000001, 2024-04-20T17:10:33.500 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0076988009999999999, 2024-04-20T17:10:33.500 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:33.500 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:33.500 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:33.500 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:33.500 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.500 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:33.500 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.327689+0000", 2024-04-20T17:10:33.500 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.500 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.500 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.500 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:33.501 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.327689+0000", 2024-04-20T17:10:33.501 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:33.501 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.501 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.501 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:33.501 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.327691+0000", 2024-04-20T17:10:33.501 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.48e-06 2024-04-20T17:10:33.501 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.501 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.501 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:33.501 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.327698+0000", 2024-04-20T17:10:33.501 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.669e-06 2024-04-20T17:10:33.501 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.501 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.501 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:33.501 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.327706+0000", 2024-04-20T17:10:33.502 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6649999999999992e-06 2024-04-20T17:10:33.502 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.502 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.502 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:33.502 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.327715+0000", 2024-04-20T17:10:33.502 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4840000000000004e-06 2024-04-20T17:10:33.502 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.502 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.502 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:33.502 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.327759+0000", 2024-04-20T17:10:33.502 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4635000000000002e-05 2024-04-20T17:10:33.502 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.502 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.502 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:33.502 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.327775+0000", 2024-04-20T17:10:33.503 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5176e-05 2024-04-20T17:10:33.503 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.503 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.503 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:33.503 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.335340+0000", 2024-04-20T17:10:33.503 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0075650070000000003 2024-04-20T17:10:33.503 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:33.503 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:33.503 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:33.503 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.335388+0000", 2024-04-20T17:10:33.503 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8684999999999999e-05 2024-04-20T17:10:33.503 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.503 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:33.503 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.503 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:33.504 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:33.504 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:10:34.149 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:33 smithi106 ceph-mon[44423]: pgmap v488: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 563 op/s; 14666/255080 objects misplaced (5.750%); 7.9 MiB/s, 126 objects/s recovering 2024-04-20T17:10:34.149 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:33 smithi106 ceph-mon[44423]: osdmap e166: 12 total, 12 up, 12 in 2024-04-20T17:10:34.150 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:33 smithi106 ceph-mon[44423]: 2.9 starting backfill to osd.10 from (0'0,0'0] MAX to 163'12281 2024-04-20T17:10:34.188 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-20T17:10:34.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:33 smithi005 ceph-mon[39562]: pgmap v488: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 563 op/s; 14666/255080 objects misplaced (5.750%); 7.9 MiB/s, 126 objects/s recovering 2024-04-20T17:10:34.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:33 smithi005 ceph-mon[39562]: osdmap e166: 12 total, 12 up, 12 in 2024-04-20T17:10:34.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:33 smithi005 ceph-mon[39562]: 2.9 starting backfill to osd.10 from (0'0,0'0] MAX to 163'12281 2024-04-20T17:10:34.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:33 smithi171 ceph-mon[44189]: pgmap v488: 17 pgs: 1 active+recovering+remapped, 16 active+clean; 7.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 563 op/s; 14666/255080 objects misplaced (5.750%); 7.9 MiB/s, 126 objects/s recovering 2024-04-20T17:10:34.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:33 smithi171 ceph-mon[44189]: osdmap e166: 12 total, 12 up, 12 in 2024-04-20T17:10:34.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:33 smithi171 ceph-mon[44189]: 2.9 starting backfill to osd.10 from (0'0,0'0] MAX to 163'12281 2024-04-20T17:10:34.514 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:10:34.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:34 smithi005 ceph-mon[39562]: 2.2 scrub starts 2024-04-20T17:10:35.056 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:34 smithi106 ceph-mon[44423]: 2.2 scrub starts 2024-04-20T17:10:35.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:34 smithi171 ceph-mon[44189]: 2.2 scrub starts 2024-04-20T17:10:36.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:35 smithi005 ceph-mon[39562]: pgmap v490: 17 pgs: 1 remapped+peering, 1 active+recovering+remapped, 15 active+clean; 7.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s; 14666/252190 objects misplaced (5.815%); 6.3 MiB/s, 101 objects/s recovering 2024-04-20T17:10:36.241 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:35 smithi106 ceph-mon[44423]: pgmap v490: 17 pgs: 1 remapped+peering, 1 active+recovering+remapped, 15 active+clean; 7.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s; 14666/252190 objects misplaced (5.815%); 6.3 MiB/s, 101 objects/s recovering 2024-04-20T17:10:36.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:35 smithi171 ceph-mon[44189]: pgmap v490: 17 pgs: 1 remapped+peering, 1 active+recovering+remapped, 15 active+clean; 7.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s; 14666/252190 objects misplaced (5.815%); 6.3 MiB/s, 101 objects/s recovering 2024-04-20T17:10:36.522 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:10:36.523 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:10:36.523 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-04-20T17:10:36.523 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:10:36.887 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:36 smithi106 ceph-mon[44423]: 2.2 scrub ok 2024-04-20T17:10:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:36 smithi005 ceph-mon[39562]: 2.2 scrub ok 2024-04-20T17:10:37.252 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-20T17:10:37.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:36 smithi171 ceph-mon[44189]: 2.2 scrub ok 2024-04-20T17:10:37.568 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:10:38.085 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:37 smithi106 ceph-mon[44423]: pgmap v491: 17 pgs: 1 active+recovering+remapped, 1 remapped+peering, 15 active+clean; 7.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 573 op/s; 14308/250092 objects misplaced (5.721%); 4.5 MiB/s, 72 objects/s recovering 2024-04-20T17:10:38.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:37 smithi005 ceph-mon[39562]: pgmap v491: 17 pgs: 1 active+recovering+remapped, 1 remapped+peering, 15 active+clean; 7.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 573 op/s; 14308/250092 objects misplaced (5.721%); 4.5 MiB/s, 72 objects/s recovering 2024-04-20T17:10:38.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:37 smithi171 ceph-mon[44189]: pgmap v491: 17 pgs: 1 active+recovering+remapped, 1 remapped+peering, 15 active+clean; 7.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 573 op/s; 14308/250092 objects misplaced (5.721%); 4.5 MiB/s, 72 objects/s recovering 2024-04-20T17:10:39.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:38 smithi005 ceph-mon[39562]: pgmap v492: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 15 active+clean; 7.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 630 op/s; 20526/246552 objects misplaced (8.325%); 15 MiB/s, 241 objects/s recovering 2024-04-20T17:10:39.233 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:38 smithi106 ceph-mon[44423]: pgmap v492: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 15 active+clean; 7.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 630 op/s; 20526/246552 objects misplaced (8.325%); 15 MiB/s, 241 objects/s recovering 2024-04-20T17:10:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:38 smithi171 ceph-mon[44189]: pgmap v492: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 15 active+clean; 7.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 630 op/s; 20526/246552 objects misplaced (8.325%); 15 MiB/s, 241 objects/s recovering 2024-04-20T17:10:39.467 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:10:39.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:39.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:10:39.575 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:10:39.578 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:10:39.578 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:10:39.578 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-04-20T17:10:39.578 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-04-20T17:10:39.578 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:10:39.612 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:39.841 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:10:39.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:39.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:10:39.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:39 smithi106 ceph-mon[44423]: 2.4 scrub starts 2024-04-20T17:10:39.945 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:10:39.988 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:40.015 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:10:40.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:40.022 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:10:40.123 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:10:40.160 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:40.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:39 smithi005 ceph-mon[39562]: 2.4 scrub starts 2024-04-20T17:10:40.305 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:10:40.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:40.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:10:40.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:39 smithi171 ceph-mon[44189]: 2.4 scrub starts 2024-04-20T17:10:40.352 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-20T17:10:40.416 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:10:40.457 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:40.717 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:10:41.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:40 smithi005 ceph-mon[39562]: pgmap v493: 17 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 15 active+clean; 7.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 728 op/s; 20018/243262 objects misplaced (8.229%); 15 MiB/s, 243 objects/s recovering 2024-04-20T17:10:41.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:40 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:10:41.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:40 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:10:41.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:40 smithi005 ceph-mon[39562]: 2.4 scrub ok 2024-04-20T17:10:41.255 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:40 smithi106 ceph-mon[44423]: pgmap v493: 17 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 15 active+clean; 7.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 728 op/s; 20018/243262 objects misplaced (8.229%); 15 MiB/s, 243 objects/s recovering 2024-04-20T17:10:41.255 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:40 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:10:41.255 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:40 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:10:41.255 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:40 smithi106 ceph-mon[44423]: 2.4 scrub ok 2024-04-20T17:10:41.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:40 smithi171 ceph-mon[44189]: pgmap v493: 17 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 15 active+clean; 7.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 728 op/s; 20018/243262 objects misplaced (8.229%); 15 MiB/s, 243 objects/s recovering 2024-04-20T17:10:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:40 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:10:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:40 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:10:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:40 smithi171 ceph-mon[44189]: 2.4 scrub ok 2024-04-20T17:10:41.915 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:41 smithi005 ceph-mon[39562]: osdmap e167: 12 total, 12 up, 12 in 2024-04-20T17:10:42.119 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:41 smithi106 ceph-mon[44423]: osdmap e167: 12 total, 12 up, 12 in 2024-04-20T17:10:42.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:41 smithi171 ceph-mon[44189]: osdmap e167: 12 total, 12 up, 12 in 2024-04-20T17:10:42.791 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:10:42.791 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-04-20T17:10:42.791 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-04-20T17:10:42.791 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:10:42.791 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.791 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:48188 2.2 2:48644282:::benchmark_data_smithi171_78918_object48187:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-20T17:10:42.791 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:08:28.762794+0000", 2024-04-20T17:10:42.791 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 134.02509037999999, 2024-04-20T17:10:42.791 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.061042962999999999, 2024-04-20T17:10:42.791 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:42.791 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:42.792 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:42.792 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:10:42.792 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:10:42.792 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:10:42.792 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 48188 2024-04-20T17:10:42.792 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.792 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:42.792 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.792 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:42.792 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:28.762794+0000", 2024-04-20T17:10:42.792 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.792 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.792 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.792 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:42.792 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:28.762794+0000", 2024-04-20T17:10:42.793 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.793 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.793 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.793 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:42.793 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:28.762795+0000", 2024-04-20T17:10:42.793 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.082e-06 2024-04-20T17:10:42.793 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.793 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.793 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:42.793 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:28.762863+0000", 2024-04-20T17:10:42.793 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7705000000000003e-05 2024-04-20T17:10:42.793 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.793 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.793 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:42.793 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:28.762871+0000", 2024-04-20T17:10:42.794 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6629999999999993e-06 2024-04-20T17:10:42.794 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.794 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.794 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:42.794 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:28.762876+0000", 2024-04-20T17:10:42.794 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0490000000000003e-06 2024-04-20T17:10:42.794 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.794 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.794 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.794 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:28.762928+0000", 2024-04-20T17:10:42.794 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1989e-05 2024-04-20T17:10:42.794 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.794 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.795 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:42.795 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:28.763047+0000", 2024-04-20T17:10:42.795 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000119539 2024-04-20T17:10:42.795 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.795 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.795 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:10:42.795 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:28.763176+0000", 2024-04-20T17:10:42.795 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012855399999999999 2024-04-20T17:10:42.795 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.795 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.795 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:10:42.795 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:28.764606+0000", 2024-04-20T17:10:42.795 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001430349 2024-04-20T17:10:42.795 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.796 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.796 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:10:42.796 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:28.823743+0000", 2024-04-20T17:10:42.796 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.059136497000000003 2024-04-20T17:10:42.796 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.796 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.796 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:42.796 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:28.823798+0000", 2024-04-20T17:10:42.796 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5590999999999997e-05 2024-04-20T17:10:42.796 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.796 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.796 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:42.796 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:28.823837+0000", 2024-04-20T17:10:42.796 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8945000000000001e-05 2024-04-20T17:10:42.797 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.797 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:42.797 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.797 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.797 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.797 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:50998 2.3 2:c9eb4ddf:::benchmark_data_smithi171_78918_object50997:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-20T17:10:42.797 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:08:29.781567+0000", 2024-04-20T17:10:42.797 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 133.00631798500001, 2024-04-20T17:10:42.797 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.066159045, 2024-04-20T17:10:42.797 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:42.797 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:42.797 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:42.797 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:10:42.797 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:10:42.797 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:10:42.798 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 50998 2024-04-20T17:10:42.798 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.798 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:42.798 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.798 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:42.798 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:29.781567+0000", 2024-04-20T17:10:42.798 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.798 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.798 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.798 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:42.798 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:29.781567+0000", 2024-04-20T17:10:42.798 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.798 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.799 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.799 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:42.799 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:29.781568+0000", 2024-04-20T17:10:42.799 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4389999999999999e-06 2024-04-20T17:10:42.799 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.799 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.799 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:42.799 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:29.781636+0000", 2024-04-20T17:10:42.799 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7890000000000002e-05 2024-04-20T17:10:42.799 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.799 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.799 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:42.799 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:29.781647+0000", 2024-04-20T17:10:42.799 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1361e-05 2024-04-20T17:10:42.799 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.800 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.800 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:42.800 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:29.781655+0000", 2024-04-20T17:10:42.800 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1270000000000003e-06 2024-04-20T17:10:42.800 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.800 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.800 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.800 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:29.781696+0000", 2024-04-20T17:10:42.800 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1130000000000001e-05 2024-04-20T17:10:42.800 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.800 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.800 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:42.800 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:29.781778+0000", 2024-04-20T17:10:42.800 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1920000000000002e-05 2024-04-20T17:10:42.800 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.801 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.801 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2", 2024-04-20T17:10:42.801 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:29.781866+0000", 2024-04-20T17:10:42.801 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7664000000000006e-05 2024-04-20T17:10:42.801 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.801 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.801 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:10:42.801 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:29.783168+0000", 2024-04-20T17:10:42.801 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013017339999999999 2024-04-20T17:10:42.801 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.801 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.801 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:10:42.801 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:29.847658+0000", 2024-04-20T17:10:42.801 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.064489872000000004 2024-04-20T17:10:42.802 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.802 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.802 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:42.802 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:29.847691+0000", 2024-04-20T17:10:42.802 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3800999999999997e-05 2024-04-20T17:10:42.802 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.802 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.802 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:42.802 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:29.847726+0000", 2024-04-20T17:10:42.802 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4106999999999997e-05 2024-04-20T17:10:42.802 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.802 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:42.802 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.802 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.802 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.803 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:51018 2.2 2:49a37c75:::benchmark_data_smithi171_78918_object51017:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-20T17:10:42.803 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:08:29.793415+0000", 2024-04-20T17:10:42.803 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 132.99446977100001, 2024-04-20T17:10:42.803 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.057207517999999999, 2024-04-20T17:10:42.803 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:42.803 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:42.803 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:42.803 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:10:42.803 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:10:42.803 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:10:42.803 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 51018 2024-04-20T17:10:42.803 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.803 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:42.803 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.804 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:42.804 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:29.793415+0000", 2024-04-20T17:10:42.804 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.804 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.804 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:42.804 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:29.793415+0000", 2024-04-20T17:10:42.804 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.804 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.804 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:42.804 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:29.793417+0000", 2024-04-20T17:10:42.804 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0810000000000001e-06 2024-04-20T17:10:42.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.805 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.805 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:42.805 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:29.793460+0000", 2024-04-20T17:10:42.805 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3430000000000003e-05 2024-04-20T17:10:42.805 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.805 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.805 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:42.805 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:29.793473+0000", 2024-04-20T17:10:42.805 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2475e-05 2024-04-20T17:10:42.805 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.805 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.805 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:42.805 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:29.793482+0000", 2024-04-20T17:10:42.805 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9649999999999997e-06 2024-04-20T17:10:42.805 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.806 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.806 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.806 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:29.793530+0000", 2024-04-20T17:10:42.806 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7984e-05 2024-04-20T17:10:42.806 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.806 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.806 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:42.806 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:29.793611+0000", 2024-04-20T17:10:42.806 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1102999999999999e-05 2024-04-20T17:10:42.806 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.806 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.806 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:10:42.806 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:29.793695+0000", 2024-04-20T17:10:42.806 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4138e-05 2024-04-20T17:10:42.807 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.807 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.807 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:10:42.807 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:29.794941+0000", 2024-04-20T17:10:42.807 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0012457200000000001 2024-04-20T17:10:42.807 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.807 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.807 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:10:42.807 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:29.850562+0000", 2024-04-20T17:10:42.807 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.055621727000000003 2024-04-20T17:10:42.807 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.807 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.807 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:42.807 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:29.850595+0000", 2024-04-20T17:10:42.807 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3008e-05 2024-04-20T17:10:42.808 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.808 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.808 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:42.808 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:08:29.850622+0000", 2024-04-20T17:10:42.808 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6886999999999999e-05 2024-04-20T17:10:42.808 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.808 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:42.808 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.808 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.808 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.808 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:194018 2.4 2:27a9c215:::benchmark_data_smithi171_78918_object33024:head [delete] snapc 0=[] ondisk+write+known_if_redirected e163)", 2024-04-20T17:10:42.808 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:28.864443+0000", 2024-04-20T17:10:42.808 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 13.923441875, 2024-04-20T17:10:42.808 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.062669097000000007, 2024-04-20T17:10:42.809 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:42.809 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:42.809 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:42.809 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:10:42.809 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:10:42.809 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:10:42.809 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 194018 2024-04-20T17:10:42.809 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.809 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:42.809 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.809 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:42.809 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864443+0000", 2024-04-20T17:10:42.809 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.809 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.809 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.810 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:42.810 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864443+0000", 2024-04-20T17:10:42.810 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.810 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.810 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.810 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:42.810 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864443+0000", 2024-04-20T17:10:42.810 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6899999999999998e-07 2024-04-20T17:10:42.810 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.810 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.810 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:42.810 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864445+0000", 2024-04-20T17:10:42.810 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.632e-06 2024-04-20T17:10:42.810 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.810 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.811 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:42.811 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864448+0000", 2024-04-20T17:10:42.811 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1300000000000001e-06 2024-04-20T17:10:42.811 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.811 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.811 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:42.811 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864452+0000", 2024-04-20T17:10:42.811 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1649999999999999e-06 2024-04-20T17:10:42.811 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.811 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.811 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.811 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864569+0000", 2024-04-20T17:10:42.811 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000117321 2024-04-20T17:10:42.811 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.812 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.812 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:10:42.812 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864572+0000", 2024-04-20T17:10:42.812 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1209999999999998e-06 2024-04-20T17:10:42.812 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.812 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.812 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.812 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.893144+0000", 2024-04-20T17:10:42.812 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.028571745999999999 2024-04-20T17:10:42.812 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.812 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.812 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:42.812 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.893260+0000", 2024-04-20T17:10:42.812 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000115869 2024-04-20T17:10:42.812 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.813 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.813 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:10:42.813 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.893380+0000", 2024-04-20T17:10:42.813 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000119837 2024-04-20T17:10:42.813 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.813 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.813 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:10:42.813 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.904596+0000", 2024-04-20T17:10:42.813 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011216271 2024-04-20T17:10:42.813 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.813 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.813 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:10:42.813 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.926863+0000", 2024-04-20T17:10:42.813 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.022266828999999998 2024-04-20T17:10:42.813 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.813 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.814 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:42.814 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.927080+0000", 2024-04-20T17:10:42.814 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00021687 2024-04-20T17:10:42.814 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.814 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.814 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:42.814 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.927112+0000", 2024-04-20T17:10:42.814 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1937000000000003e-05 2024-04-20T17:10:42.814 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.814 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:42.814 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.814 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.814 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.814 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:194042 2.4 2:22fc9479:::benchmark_data_smithi171_78918_object33048:head [delete] snapc 0=[] ondisk+write+known_if_redirected e163)", 2024-04-20T17:10:42.815 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:28.864458+0000", 2024-04-20T17:10:42.815 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 13.923426896, 2024-04-20T17:10:42.815 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.062804993000000003, 2024-04-20T17:10:42.815 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:42.815 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:42.815 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:42.815 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:10:42.815 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:10:42.815 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:10:42.815 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 194042 2024-04-20T17:10:42.815 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.815 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:42.815 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.815 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:42.816 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864458+0000", 2024-04-20T17:10:42.816 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.816 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.816 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.816 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:42.816 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864458+0000", 2024-04-20T17:10:42.816 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.816 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.816 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.816 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:42.816 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864458+0000", 2024-04-20T17:10:42.816 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5200000000000002e-07 2024-04-20T17:10:42.816 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.816 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.816 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:42.817 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864460+0000", 2024-04-20T17:10:42.817 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0880000000000002e-06 2024-04-20T17:10:42.817 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.817 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.817 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:42.817 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864465+0000", 2024-04-20T17:10:42.817 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5480000000000003e-06 2024-04-20T17:10:42.817 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.817 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.817 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:42.817 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864469+0000", 2024-04-20T17:10:42.817 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6140000000000002e-06 2024-04-20T17:10:42.817 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.817 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.818 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.818 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864602+0000", 2024-04-20T17:10:42.818 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000132342 2024-04-20T17:10:42.818 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.818 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.818 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:10:42.818 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864608+0000", 2024-04-20T17:10:42.818 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3459999999999999e-06 2024-04-20T17:10:42.818 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.818 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.818 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.818 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.895141+0000", 2024-04-20T17:10:42.818 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.030533385 2024-04-20T17:10:42.818 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.818 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.819 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:42.819 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.895235+0000", 2024-04-20T17:10:42.819 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3633000000000003e-05 2024-04-20T17:10:42.819 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.819 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.819 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:10:42.819 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.895303+0000", 2024-04-20T17:10:42.819 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7793000000000006e-05 2024-04-20T17:10:42.819 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.819 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.819 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:10:42.819 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.904603+0000", 2024-04-20T17:10:42.819 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0093000429999999992 2024-04-20T17:10:42.819 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.820 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.820 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:10:42.820 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.927163+0000", 2024-04-20T17:10:42.820 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.022559795000000001 2024-04-20T17:10:42.820 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.820 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.820 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:42.820 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.927218+0000", 2024-04-20T17:10:42.820 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5537e-05 2024-04-20T17:10:42.820 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.820 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.820 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:42.820 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.927263+0000", 2024-04-20T17:10:42.821 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4416999999999998e-05 2024-04-20T17:10:42.821 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.821 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:42.821 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.821 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.821 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.821 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:194094 2.4 2:2b689924:::benchmark_data_smithi171_78918_object33100:head [delete] snapc 0=[] ondisk+write+known_if_redirected e163)", 2024-04-20T17:10:42.821 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:28.864480+0000", 2024-04-20T17:10:42.821 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 13.923404795, 2024-04-20T17:10:42.821 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.067369730000000003, 2024-04-20T17:10:42.821 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:42.821 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:42.821 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:42.821 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:10:42.821 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:10:42.821 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:10:42.822 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 194094 2024-04-20T17:10:42.822 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.822 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:42.822 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.822 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:42.822 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864480+0000", 2024-04-20T17:10:42.822 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.822 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.822 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.822 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:42.822 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864480+0000", 2024-04-20T17:10:42.822 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.822 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.822 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.823 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:42.823 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864480+0000", 2024-04-20T17:10:42.823 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6499999999999999e-07 2024-04-20T17:10:42.823 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.823 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.823 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:42.823 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864483+0000", 2024-04-20T17:10:42.823 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.306e-06 2024-04-20T17:10:42.823 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.823 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.823 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:42.823 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864487+0000", 2024-04-20T17:10:42.823 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7149999999999997e-06 2024-04-20T17:10:42.823 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.823 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864493+0000", 2024-04-20T17:10:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5090000000000001e-06 2024-04-20T17:10:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864631+0000", 2024-04-20T17:10:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013836400000000001 2024-04-20T17:10:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:10:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864634+0000", 2024-04-20T17:10:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1829999999999998e-06 2024-04-20T17:10:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.897021+0000", 2024-04-20T17:10:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.032386854999999999 2024-04-20T17:10:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.897088+0000", 2024-04-20T17:10:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6322000000000006e-05 2024-04-20T17:10:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:10:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.897152+0000", 2024-04-20T17:10:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4330000000000002e-05 2024-04-20T17:10:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:10:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.908260+0000", 2024-04-20T17:10:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011107911999999999 2024-04-20T17:10:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:10:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.931751+0000", 2024-04-20T17:10:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.023491452999999999 2024-04-20T17:10:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.931806+0000", 2024-04-20T17:10:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4557e-05 2024-04-20T17:10:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.931849+0000", 2024-04-20T17:10:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3559000000000001e-05 2024-04-20T17:10:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:194109 2.4 2:221bd37d:::benchmark_data_smithi171_78918_object33115:head [delete] snapc 0=[] ondisk+write+known_if_redirected e163)", 2024-04-20T17:10:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:28.864504+0000", 2024-04-20T17:10:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 13.923380386, 2024-04-20T17:10:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.067444737000000005, 2024-04-20T17:10:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:42.828 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:10:42.828 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:10:42.828 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:10:42.828 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 194109 2024-04-20T17:10:42.828 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.828 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:42.828 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.828 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:42.828 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864504+0000", 2024-04-20T17:10:42.828 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.828 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.828 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.828 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:42.828 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864504+0000", 2024-04-20T17:10:42.828 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864504+0000", 2024-04-20T17:10:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4700000000000002e-07 2024-04-20T17:10:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864506+0000", 2024-04-20T17:10:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6249999999999999e-06 2024-04-20T17:10:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864510+0000", 2024-04-20T17:10:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7440000000000001e-06 2024-04-20T17:10:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864513+0000", 2024-04-20T17:10:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2799999999999999e-06 2024-04-20T17:10:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864659+0000", 2024-04-20T17:10:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014595999999999999 2024-04-20T17:10:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:10:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864663+0000", 2024-04-20T17:10:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6109999999999998e-06 2024-04-20T17:10:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.899017+0000", 2024-04-20T17:10:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.034354572 2024-04-20T17:10:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.899105+0000", 2024-04-20T17:10:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7593e-05 2024-04-20T17:10:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:10:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.899199+0000", 2024-04-20T17:10:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3713000000000002e-05 2024-04-20T17:10:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:10:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.908284+0000", 2024-04-20T17:10:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0090854170000000001 2024-04-20T17:10:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:10:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.931883+0000", 2024-04-20T17:10:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.023599196999999999 2024-04-20T17:10:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.931912+0000", 2024-04-20T17:10:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8374e-05 2024-04-20T17:10:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.931949+0000", 2024-04-20T17:10:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7304000000000001e-05 2024-04-20T17:10:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:194115 2.4 2:2492cd19:::benchmark_data_smithi171_78918_object33121:head [delete] snapc 0=[] ondisk+write+known_if_redirected e163)", 2024-04-20T17:10:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:28.864518+0000", 2024-04-20T17:10:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 13.923366056999999, 2024-04-20T17:10:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.072661744, 2024-04-20T17:10:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:42.834 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:10:42.834 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:10:42.834 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:10:42.834 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 194115 2024-04-20T17:10:42.834 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.834 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:42.834 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.834 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:42.834 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864518+0000", 2024-04-20T17:10:42.834 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.834 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.834 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.834 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864518+0000", 2024-04-20T17:10:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864519+0000", 2024-04-20T17:10:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.51e-07 2024-04-20T17:10:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864521+0000", 2024-04-20T17:10:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9379999999999999e-06 2024-04-20T17:10:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864525+0000", 2024-04-20T17:10:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5909999999999998e-06 2024-04-20T17:10:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864530+0000", 2024-04-20T17:10:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8969999999999999e-06 2024-04-20T17:10:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864681+0000", 2024-04-20T17:10:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015036199999999999 2024-04-20T17:10:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:10:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864684+0000", 2024-04-20T17:10:42.837 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1669999999999999e-06 2024-04-20T17:10:42.837 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.837 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.837 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.837 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.900866+0000", 2024-04-20T17:10:42.837 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.036182178000000002 2024-04-20T17:10:42.837 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.837 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.837 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:42.837 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.900959+0000", 2024-04-20T17:10:42.837 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3410000000000002e-05 2024-04-20T17:10:42.837 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.837 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.837 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:10:42.837 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.901038+0000", 2024-04-20T17:10:42.838 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9041000000000004e-05 2024-04-20T17:10:42.838 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.838 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.838 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:10:42.838 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.908290+0000", 2024-04-20T17:10:42.838 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0072511820000000001 2024-04-20T17:10:42.838 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.838 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.838 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:10:42.838 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.937092+0000", 2024-04-20T17:10:42.838 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.028801941000000001 2024-04-20T17:10:42.838 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.838 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.838 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:42.839 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.937148+0000", 2024-04-20T17:10:42.839 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6276000000000001e-05 2024-04-20T17:10:42.839 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.839 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.839 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:42.839 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.937180+0000", 2024-04-20T17:10:42.839 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2310000000000001e-05 2024-04-20T17:10:42.839 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.839 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:42.839 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.839 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.839 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.839 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:194120 2.4 2:24f46cbd:::benchmark_data_smithi171_78918_object33126:head [delete] snapc 0=[] ondisk+write+known_if_redirected e163)", 2024-04-20T17:10:42.839 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:28.864535+0000", 2024-04-20T17:10:42.839 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 13.923349198, 2024-04-20T17:10:42.840 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.081574351000000003, 2024-04-20T17:10:42.840 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:42.840 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:42.840 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:42.840 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:10:42.840 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:10:42.840 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:10:42.840 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 194120 2024-04-20T17:10:42.840 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.840 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:42.840 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.840 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:42.840 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864535+0000", 2024-04-20T17:10:42.840 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.840 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.841 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.841 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:42.841 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864535+0000", 2024-04-20T17:10:42.841 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.841 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.841 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.841 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:42.841 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864536+0000", 2024-04-20T17:10:42.841 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3700000000000001e-07 2024-04-20T17:10:42.841 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.841 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.841 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:42.841 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864538+0000", 2024-04-20T17:10:42.841 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.294e-06 2024-04-20T17:10:42.841 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.841 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.842 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:42.842 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864542+0000", 2024-04-20T17:10:42.842 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4390000000000002e-06 2024-04-20T17:10:42.842 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.842 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.842 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:42.842 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864547+0000", 2024-04-20T17:10:42.842 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7180000000000004e-06 2024-04-20T17:10:42.842 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.842 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.842 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.842 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864703+0000", 2024-04-20T17:10:42.842 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015556899999999999 2024-04-20T17:10:42.842 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.842 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.842 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:10:42.843 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864706+0000", 2024-04-20T17:10:42.843 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1030000000000002e-06 2024-04-20T17:10:42.843 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.843 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.843 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.843 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.902671+0000", 2024-04-20T17:10:42.843 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.037965153000000001 2024-04-20T17:10:42.843 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.843 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.843 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:42.843 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.902734+0000", 2024-04-20T17:10:42.843 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2800999999999997e-05 2024-04-20T17:10:42.843 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.844 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.844 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:10:42.844 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.902800+0000", 2024-04-20T17:10:42.844 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5881000000000002e-05 2024-04-20T17:10:42.844 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.844 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.844 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:10:42.844 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.908296+0000", 2024-04-20T17:10:42.844 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0054963160000000002 2024-04-20T17:10:42.844 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.844 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.844 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:10:42.844 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.946010+0000", 2024-04-20T17:10:42.844 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.037714152000000001 2024-04-20T17:10:42.844 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.845 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.845 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:42.845 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.946064+0000", 2024-04-20T17:10:42.845 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3724999999999998e-05 2024-04-20T17:10:42.845 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.845 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.845 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:42.845 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.946110+0000", 2024-04-20T17:10:42.845 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5763000000000002e-05 2024-04-20T17:10:42.845 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.845 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:42.845 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.845 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.845 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.845 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:194128 2.4 2:25cb3f7f:::benchmark_data_smithi171_78918_object33134:head [delete] snapc 0=[] ondisk+write+known_if_redirected e163)", 2024-04-20T17:10:42.846 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:28.864566+0000", 2024-04-20T17:10:42.846 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 13.923318654999999, 2024-04-20T17:10:42.846 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.083445169999999999, 2024-04-20T17:10:42.846 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:42.846 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:42.846 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:42.846 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:10:42.846 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:10:42.846 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:10:42.846 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 194128 2024-04-20T17:10:42.846 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.846 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:42.846 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.846 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:42.847 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864566+0000", 2024-04-20T17:10:42.847 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.847 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.847 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.847 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:42.847 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864566+0000", 2024-04-20T17:10:42.847 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.847 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.847 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.847 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:42.847 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864566+0000", 2024-04-20T17:10:42.847 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5499999999999999e-07 2024-04-20T17:10:42.847 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.847 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.848 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:42.848 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864568+0000", 2024-04-20T17:10:42.848 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6080000000000001e-06 2024-04-20T17:10:42.848 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.848 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.848 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:42.848 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864571+0000", 2024-04-20T17:10:42.848 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0450000000000001e-06 2024-04-20T17:10:42.848 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.848 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.848 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:42.848 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864574+0000", 2024-04-20T17:10:42.848 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5470000000000001e-06 2024-04-20T17:10:42.848 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.849 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.849 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.849 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864727+0000", 2024-04-20T17:10:42.849 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015271600000000001 2024-04-20T17:10:42.849 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.849 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.849 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:10:42.849 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864730+0000", 2024-04-20T17:10:42.849 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0750000000000002e-06 2024-04-20T17:10:42.849 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.849 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.849 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.849 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.904621+0000", 2024-04-20T17:10:42.849 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.039891031 2024-04-20T17:10:42.849 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.849 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.850 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:42.850 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.904687+0000", 2024-04-20T17:10:42.850 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5309999999999996e-05 2024-04-20T17:10:42.850 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.850 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.850 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:10:42.850 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.904758+0000", 2024-04-20T17:10:42.850 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0996999999999998e-05 2024-04-20T17:10:42.850 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.850 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.850 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:10:42.850 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.912312+0000", 2024-04-20T17:10:42.850 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0075546420000000003 2024-04-20T17:10:42.850 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.850 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.851 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:10:42.851 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.947961+0000", 2024-04-20T17:10:42.851 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.035649040999999999 2024-04-20T17:10:42.851 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.851 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.851 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:42.851 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.947988+0000", 2024-04-20T17:10:42.851 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6741999999999999e-05 2024-04-20T17:10:42.851 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.851 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.851 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:42.851 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.948011+0000", 2024-04-20T17:10:42.851 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3061000000000002e-05 2024-04-20T17:10:42.851 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.852 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:42.852 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.852 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.852 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.852 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:194135 2.4 2:2e111232:::benchmark_data_smithi171_78918_object33141:head [delete] snapc 0=[] ondisk+write+known_if_redirected e163)", 2024-04-20T17:10:42.852 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:28.864580+0000", 2024-04-20T17:10:42.852 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 13.923304796, 2024-04-20T17:10:42.852 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.095628708000000007, 2024-04-20T17:10:42.852 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:42.852 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:42.852 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:42.852 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:10:42.852 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:10:42.852 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:10:42.853 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 194135 2024-04-20T17:10:42.853 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.853 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:42.853 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.853 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:42.853 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864580+0000", 2024-04-20T17:10:42.853 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.853 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.853 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.853 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:42.853 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864580+0000", 2024-04-20T17:10:42.853 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.853 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.853 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.853 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:42.854 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864580+0000", 2024-04-20T17:10:42.854 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4499999999999997e-07 2024-04-20T17:10:42.854 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.854 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.854 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:42.854 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864582+0000", 2024-04-20T17:10:42.854 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.933e-06 2024-04-20T17:10:42.854 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.854 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.854 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:42.854 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864591+0000", 2024-04-20T17:10:42.854 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9209999999999998e-06 2024-04-20T17:10:42.854 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.854 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.855 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:42.855 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864597+0000", 2024-04-20T17:10:42.855 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8810000000000001e-06 2024-04-20T17:10:42.855 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.855 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.855 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.855 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864749+0000", 2024-04-20T17:10:42.855 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000152072 2024-04-20T17:10:42.855 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.855 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.855 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:10:42.855 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864752+0000", 2024-04-20T17:10:42.855 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0429999999999999e-06 2024-04-20T17:10:42.855 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.855 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.855 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.855 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.906387+0000", 2024-04-20T17:10:42.856 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.041634717000000002 2024-04-20T17:10:42.856 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.856 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.856 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:42.856 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.906498+0000", 2024-04-20T17:10:42.856 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000111109 2024-04-20T17:10:42.856 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.856 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.856 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:10:42.856 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.906614+0000", 2024-04-20T17:10:42.856 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000116396 2024-04-20T17:10:42.856 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.856 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.856 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:10:42.857 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.912573+0000", 2024-04-20T17:10:42.857 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0059592669999999999 2024-04-20T17:10:42.857 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.857 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.857 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:10:42.857 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.960103+0000", 2024-04-20T17:10:42.857 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.047529505 2024-04-20T17:10:42.857 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.857 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.857 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:42.857 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.960159+0000", 2024-04-20T17:10:42.857 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6390000000000001e-05 2024-04-20T17:10:42.857 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.857 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.857 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:42.858 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.960208+0000", 2024-04-20T17:10:42.858 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9029000000000001e-05 2024-04-20T17:10:42.858 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.858 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:42.858 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.858 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.858 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.858 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:194155 2.4 2:2056e495:::benchmark_data_smithi171_78918_object33161:head [delete] snapc 0=[] ondisk+write+known_if_redirected e163)", 2024-04-20T17:10:42.858 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:28.864604+0000", 2024-04-20T17:10:42.858 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 13.923280126, 2024-04-20T17:10:42.858 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.099538138999999998, 2024-04-20T17:10:42.858 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:42.858 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:42.858 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:42.858 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:10:42.859 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:10:42.859 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:10:42.859 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 194155 2024-04-20T17:10:42.859 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.859 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:42.859 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.859 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:42.859 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864604+0000", 2024-04-20T17:10:42.859 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.859 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.859 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.859 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:42.859 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864604+0000", 2024-04-20T17:10:42.859 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.859 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.860 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.860 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:42.860 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864605+0000", 2024-04-20T17:10:42.860 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5499999999999998e-07 2024-04-20T17:10:42.860 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.860 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.860 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:42.860 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864607+0000", 2024-04-20T17:10:42.860 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2129999999999998e-06 2024-04-20T17:10:42.860 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.860 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.860 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:42.860 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864612+0000", 2024-04-20T17:10:42.860 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6820000000000002e-06 2024-04-20T17:10:42.861 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.861 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.861 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:42.861 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864617+0000", 2024-04-20T17:10:42.861 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7890000000000002e-06 2024-04-20T17:10:42.861 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.861 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.861 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.861 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864771+0000", 2024-04-20T17:10:42.861 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015395200000000001 2024-04-20T17:10:42.861 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.861 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.861 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:10:42.861 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864774+0000", 2024-04-20T17:10:42.861 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0350000000000002e-06 2024-04-20T17:10:42.862 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.862 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.862 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.862 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.908334+0000", 2024-04-20T17:10:42.862 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.04356061 2024-04-20T17:10:42.862 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.862 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.862 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:42.862 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.908408+0000", 2024-04-20T17:10:42.862 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4017999999999995e-05 2024-04-20T17:10:42.862 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.862 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.862 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:10:42.862 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.908483+0000", 2024-04-20T17:10:42.863 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5022000000000001e-05 2024-04-20T17:10:42.863 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.863 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.863 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:10:42.863 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.912581+0000", 2024-04-20T17:10:42.863 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0040974380000000001 2024-04-20T17:10:42.863 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.863 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.863 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:10:42.863 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.964041+0000", 2024-04-20T17:10:42.863 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.051460469000000002 2024-04-20T17:10:42.863 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.863 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.863 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:42.863 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.964098+0000", 2024-04-20T17:10:42.864 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6424000000000001e-05 2024-04-20T17:10:42.864 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.864 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.864 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:42.864 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.964143+0000", 2024-04-20T17:10:42.864 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4932000000000003e-05 2024-04-20T17:10:42.864 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.864 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:42.864 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.864 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.864 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.864 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:194181 2.4 2:216b496b:::benchmark_data_smithi171_78918_object33187:head [delete] snapc 0=[] ondisk+write+known_if_redirected e163)", 2024-04-20T17:10:42.864 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:28.864628+0000", 2024-04-20T17:10:42.864 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 13.923256317, 2024-04-20T17:10:42.864 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.099622924000000002, 2024-04-20T17:10:42.865 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:42.865 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:42.865 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:42.865 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:10:42.865 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:10:42.865 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:10:42.865 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 194181 2024-04-20T17:10:42.865 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.865 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:42.865 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.865 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:42.865 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864628+0000", 2024-04-20T17:10:42.866 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.866 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.866 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.866 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:42.866 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864628+0000", 2024-04-20T17:10:42.866 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.866 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.866 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.866 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:42.866 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864629+0000", 2024-04-20T17:10:42.866 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8599999999999999e-07 2024-04-20T17:10:42.866 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.867 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.867 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:42.867 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864630+0000", 2024-04-20T17:10:42.867 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.409e-06 2024-04-20T17:10:42.867 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.867 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.867 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:42.867 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864633+0000", 2024-04-20T17:10:42.867 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.016e-06 2024-04-20T17:10:42.867 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.867 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.867 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:42.867 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864636+0000", 2024-04-20T17:10:42.867 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.495e-06 2024-04-20T17:10:42.868 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.868 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.868 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.868 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864785+0000", 2024-04-20T17:10:42.868 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014829900000000001 2024-04-20T17:10:42.868 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.868 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.868 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:10:42.868 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864788+0000", 2024-04-20T17:10:42.868 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0120000000000001e-06 2024-04-20T17:10:42.868 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.868 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.868 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.868 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.910423+0000", 2024-04-20T17:10:42.868 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.045635697000000003 2024-04-20T17:10:42.869 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.869 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.869 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:42.869 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.910531+0000", 2024-04-20T17:10:42.869 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000107558 2024-04-20T17:10:42.869 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.869 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.869 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:10:42.869 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.910652+0000", 2024-04-20T17:10:42.869 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000120567 2024-04-20T17:10:42.869 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.869 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.869 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:10:42.869 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.914643+0000", 2024-04-20T17:10:42.869 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0039910190000000002 2024-04-20T17:10:42.870 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.870 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.870 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:10:42.870 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.964187+0000", 2024-04-20T17:10:42.870 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.049544461999999997 2024-04-20T17:10:42.870 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.870 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.870 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:42.870 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.964222+0000", 2024-04-20T17:10:42.870 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4786000000000002e-05 2024-04-20T17:10:42.870 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.870 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.870 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:42.870 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.964251+0000", 2024-04-20T17:10:42.871 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9218e-05 2024-04-20T17:10:42.871 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.871 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:42.871 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.871 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.871 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.871 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:194184 2.4 2:2b9fb9ba:::benchmark_data_smithi171_78918_object33190:head [delete] snapc 0=[] ondisk+write+known_if_redirected e163)", 2024-04-20T17:10:42.871 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:28.901968+0000", 2024-04-20T17:10:42.871 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 13.885916803000001, 2024-04-20T17:10:42.871 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.062420349999999999, 2024-04-20T17:10:42.871 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:42.871 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:42.871 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:42.871 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:10:42.872 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:10:42.872 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:10:42.872 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 194184 2024-04-20T17:10:42.872 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.872 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:42.872 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.872 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:42.872 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.901968+0000", 2024-04-20T17:10:42.872 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.872 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.872 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.872 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:42.872 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.901968+0000", 2024-04-20T17:10:42.872 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.872 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.873 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.873 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:42.873 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.901970+0000", 2024-04-20T17:10:42.873 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8309999999999999e-06 2024-04-20T17:10:42.873 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.873 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.873 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:42.873 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.901975+0000", 2024-04-20T17:10:42.873 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0329999999999999e-06 2024-04-20T17:10:42.873 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.873 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.873 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:42.873 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.901985+0000", 2024-04-20T17:10:42.874 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0841e-05 2024-04-20T17:10:42.874 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.874 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.874 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:42.874 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.901992+0000", 2024-04-20T17:10:42.874 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4019999999999999e-06 2024-04-20T17:10:42.874 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.874 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.874 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.874 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.912601+0000", 2024-04-20T17:10:42.874 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010609142 2024-04-20T17:10:42.874 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.874 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.874 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:42.875 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.912677+0000", 2024-04-20T17:10:42.875 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6341999999999995e-05 2024-04-20T17:10:42.875 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.875 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.875 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:10:42.875 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.912741+0000", 2024-04-20T17:10:42.875 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4208000000000006e-05 2024-04-20T17:10:42.875 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.875 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.875 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:10:42.875 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.916428+0000", 2024-04-20T17:10:42.875 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0036868030000000002 2024-04-20T17:10:42.875 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.875 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.876 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:10:42.876 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.964297+0000", 2024-04-20T17:10:42.876 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.047868850999999997 2024-04-20T17:10:42.876 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.876 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.876 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:42.876 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.964346+0000", 2024-04-20T17:10:42.876 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8999999999999998e-05 2024-04-20T17:10:42.876 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.876 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.876 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:42.876 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.964388+0000", 2024-04-20T17:10:42.876 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1897000000000002e-05 2024-04-20T17:10:42.877 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.877 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:42.877 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.877 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.877 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.877 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:197556 2.2 2:433198d9:::benchmark_data_smithi171_78918_object36562:head [delete] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-04-20T17:10:42.877 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:34.332716+0000", 2024-04-20T17:10:42.877 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 8.4551682409999991, 2024-04-20T17:10:42.877 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.068642177999999998, 2024-04-20T17:10:42.877 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:42.877 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:42.877 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:42.877 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:10:42.877 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:10:42.877 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:10:42.877 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 197556 2024-04-20T17:10:42.878 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.878 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:42.878 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.878 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:42.878 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.332716+0000", 2024-04-20T17:10:42.878 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.878 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.878 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.878 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:42.878 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.332716+0000", 2024-04-20T17:10:42.878 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.878 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.878 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.879 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:42.879 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.332718+0000", 2024-04-20T17:10:42.879 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0169999999999999e-06 2024-04-20T17:10:42.879 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.879 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.879 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:42.879 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.332724+0000", 2024-04-20T17:10:42.879 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8000000000000004e-06 2024-04-20T17:10:42.879 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.879 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.879 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:42.879 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.332733+0000", 2024-04-20T17:10:42.879 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8829999999999995e-06 2024-04-20T17:10:42.879 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.879 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.880 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:42.880 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.332739+0000", 2024-04-20T17:10:42.880 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1510000000000001e-06 2024-04-20T17:10:42.880 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.880 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.880 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.880 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.332784+0000", 2024-04-20T17:10:42.880 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4787999999999999e-05 2024-04-20T17:10:42.880 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.880 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.880 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:42.880 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.332862+0000", 2024-04-20T17:10:42.880 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7624000000000001e-05 2024-04-20T17:10:42.880 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.880 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.881 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:10:42.881 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.332927+0000", 2024-04-20T17:10:42.881 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5748999999999997e-05 2024-04-20T17:10:42.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.881 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:10:42.881 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.336407+0000", 2024-04-20T17:10:42.881 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0034794909999999999 2024-04-20T17:10:42.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.881 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:10:42.881 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.401233+0000", 2024-04-20T17:10:42.881 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.064826243000000006 2024-04-20T17:10:42.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.882 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:42.882 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.401296+0000", 2024-04-20T17:10:42.882 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2558000000000007e-05 2024-04-20T17:10:42.882 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.882 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.882 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:42.882 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.401358+0000", 2024-04-20T17:10:42.882 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2873999999999995e-05 2024-04-20T17:10:42.882 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.882 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:42.885 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.885 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.886 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:197569 2.2 2:470ccb18:::benchmark_data_smithi171_78918_object36575:head [delete] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-04-20T17:10:42.886 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:34.343803+0000", 2024-04-20T17:10:42.886 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 8.4440819519999994, 2024-04-20T17:10:42.886 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.057676193000000001, 2024-04-20T17:10:42.886 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:42.886 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:42.886 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:42.886 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:10:42.886 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:10:42.886 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:10:42.886 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 197569 2024-04-20T17:10:42.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.887 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:42.887 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.887 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:42.887 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.343803+0000", 2024-04-20T17:10:42.887 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.887 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.887 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:42.887 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.343803+0000", 2024-04-20T17:10:42.887 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.887 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.887 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:42.887 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.343804+0000", 2024-04-20T17:10:42.888 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8929999999999999e-06 2024-04-20T17:10:42.888 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.888 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.888 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:42.888 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.343809+0000", 2024-04-20T17:10:42.888 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7389999999999999e-06 2024-04-20T17:10:42.888 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.888 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.888 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:42.888 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.343820+0000", 2024-04-20T17:10:42.888 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0526999999999999e-05 2024-04-20T17:10:42.888 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.888 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.888 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:42.888 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.343828+0000", 2024-04-20T17:10:42.888 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3240000000000003e-06 2024-04-20T17:10:42.889 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.889 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.889 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.343887+0000", 2024-04-20T17:10:42.889 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.944e-05 2024-04-20T17:10:42.889 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.889 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:42.889 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344033+0000", 2024-04-20T17:10:42.889 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014574299999999999 2024-04-20T17:10:42.889 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.889 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:10:42.890 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344172+0000", 2024-04-20T17:10:42.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000139219 2024-04-20T17:10:42.890 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.890 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:10:42.890 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.354796+0000", 2024-04-20T17:10:42.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010623169 2024-04-20T17:10:42.890 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.890 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:10:42.890 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.401402+0000", 2024-04-20T17:10:42.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.046606389999999998 2024-04-20T17:10:42.890 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.891 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:42.891 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.401432+0000", 2024-04-20T17:10:42.891 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9845e-05 2024-04-20T17:10:42.891 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.891 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.891 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:42.891 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.401479+0000", 2024-04-20T17:10:42.891 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6903999999999997e-05 2024-04-20T17:10:42.891 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.891 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:42.891 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.891 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.891 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.891 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:197573 2.2 2:4b63adf6:::benchmark_data_smithi171_78918_object36579:head [delete] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-04-20T17:10:42.891 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:34.344613+0000", 2024-04-20T17:10:42.892 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 8.4432715789999992, 2024-04-20T17:10:42.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.057018594999999998, 2024-04-20T17:10:42.892 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:42.892 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:42.892 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:42.892 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:10:42.892 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:10:42.892 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:10:42.892 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 197573 2024-04-20T17:10:42.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.892 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:42.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.892 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:42.892 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344613+0000", 2024-04-20T17:10:42.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:42.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344613+0000", 2024-04-20T17:10:42.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:42.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344614+0000", 2024-04-20T17:10:42.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4670000000000001e-06 2024-04-20T17:10:42.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:42.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344619+0000", 2024-04-20T17:10:42.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.933e-06 2024-04-20T17:10:42.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.894 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:42.894 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344631+0000", 2024-04-20T17:10:42.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2108e-05 2024-04-20T17:10:42.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.894 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:42.894 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344639+0000", 2024-04-20T17:10:42.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3250000000000002e-06 2024-04-20T17:10:42.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.894 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.894 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.347516+0000", 2024-04-20T17:10:42.895 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0028768959999999999 2024-04-20T17:10:42.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.895 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:42.895 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.347642+0000", 2024-04-20T17:10:42.895 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012597199999999999 2024-04-20T17:10:42.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.895 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:10:42.895 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.347773+0000", 2024-04-20T17:10:42.895 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013104600000000001 2024-04-20T17:10:42.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.895 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:10:42.895 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.354799+0000", 2024-04-20T17:10:42.896 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0070264330000000003 2024-04-20T17:10:42.896 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.896 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:10:42.896 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.401527+0000", 2024-04-20T17:10:42.896 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.046728262999999999 2024-04-20T17:10:42.896 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.896 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:42.896 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.401578+0000", 2024-04-20T17:10:42.896 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0299000000000001e-05 2024-04-20T17:10:42.896 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.896 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:42.896 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.401632+0000", 2024-04-20T17:10:42.897 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3853000000000001e-05 2024-04-20T17:10:42.897 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.897 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:42.897 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.897 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.897 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:197574 2.2 2:453bbb18:::benchmark_data_smithi171_78918_object36580:head [delete] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-04-20T17:10:42.897 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:34.344795+0000", 2024-04-20T17:10:42.897 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 8.4430889909999998, 2024-04-20T17:10:42.897 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.056943622999999999, 2024-04-20T17:10:42.897 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:42.897 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:42.897 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:42.897 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:10:42.897 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:10:42.897 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:10:42.898 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 197574 2024-04-20T17:10:42.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.898 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:42.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:42.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344795+0000", 2024-04-20T17:10:42.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:42.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344795+0000", 2024-04-20T17:10:42.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:42.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344797+0000", 2024-04-20T17:10:42.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5e-06 2024-04-20T17:10:42.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:42.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344802+0000", 2024-04-20T17:10:42.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9350000000000002e-06 2024-04-20T17:10:42.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:42.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344814+0000", 2024-04-20T17:10:42.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2300000000000001e-05 2024-04-20T17:10:42.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:42.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344822+0000", 2024-04-20T17:10:42.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1519999999999992e-06 2024-04-20T17:10:42.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.351164+0000", 2024-04-20T17:10:42.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0063413799999999998 2024-04-20T17:10:42.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:42.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.351280+0000", 2024-04-20T17:10:42.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011615700000000001 2024-04-20T17:10:42.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:10:42.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.351359+0000", 2024-04-20T17:10:42.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8974000000000007e-05 2024-04-20T17:10:42.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:10:42.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.354931+0000", 2024-04-20T17:10:42.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0035719570000000002 2024-04-20T17:10:42.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:10:42.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.401670+0000", 2024-04-20T17:10:42.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.046739347000000001 2024-04-20T17:10:42.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:42.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.401699+0000", 2024-04-20T17:10:42.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9179999999999998e-05 2024-04-20T17:10:42.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:42.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.401739+0000", 2024-04-20T17:10:42.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9740999999999998e-05 2024-04-20T17:10:42.902 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.902 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:42.902 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.902 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:202243 2.4 2:2e1fd5a1:::benchmark_data_smithi171_78918_object41249:head [delete] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-04-20T17:10:42.903 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:39.784900+0000", 2024-04-20T17:10:42.903 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 3.002984616, 2024-04-20T17:10:42.903 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.062568198000000005, 2024-04-20T17:10:42.903 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:42.903 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:42.903 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:42.903 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:10:42.903 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:10:42.903 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:10:42.903 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 202243 2024-04-20T17:10:42.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.903 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:42.903 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.903 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:42.904 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.784900+0000", 2024-04-20T17:10:42.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.904 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:42.904 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.784900+0000", 2024-04-20T17:10:42.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.904 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:42.904 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.784903+0000", 2024-04-20T17:10:42.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.633e-06 2024-04-20T17:10:42.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:42.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.784908+0000", 2024-04-20T17:10:42.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5679999999999999e-06 2024-04-20T17:10:42.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:42.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.784919+0000", 2024-04-20T17:10:42.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0583e-05 2024-04-20T17:10:42.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:42.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.784926+0000", 2024-04-20T17:10:42.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.515e-06 2024-04-20T17:10:42.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.785293+0000", 2024-04-20T17:10:42.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000366898 2024-04-20T17:10:42.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:42.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.785405+0000", 2024-04-20T17:10:42.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000111912 2024-04-20T17:10:42.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:10:42.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.785512+0000", 2024-04-20T17:10:42.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010692900000000001 2024-04-20T17:10:42.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:10:42.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.789865+0000", 2024-04-20T17:10:42.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0043533360000000002 2024-04-20T17:10:42.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:10:42.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.847336+0000", 2024-04-20T17:10:42.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.057471181000000003 2024-04-20T17:10:42.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:42.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.847403+0000", 2024-04-20T17:10:42.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6317999999999998e-05 2024-04-20T17:10:42.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.908 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:42.908 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.847468+0000", 2024-04-20T17:10:42.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5325000000000002e-05 2024-04-20T17:10:42.908 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.908 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:42.908 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.908 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:202253 2.4 2:2c7ba7dc:::benchmark_data_smithi171_78918_object41259:head [delete] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-04-20T17:10:42.908 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:39.790264+0000", 2024-04-20T17:10:42.908 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 2.99762087, 2024-04-20T17:10:42.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.057390243000000001, 2024-04-20T17:10:42.908 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:10:42.908 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:10:42.909 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:10:42.909 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:10:42.909 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:10:42.909 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:10:42.909 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 202253 2024-04-20T17:10:42.909 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.909 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:10:42.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.909 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:10:42.909 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.790264+0000", 2024-04-20T17:10:42.909 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.909 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.909 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:10:42.910 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.790264+0000", 2024-04-20T17:10:42.910 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:10:42.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.910 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:10:42.910 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.790265+0000", 2024-04-20T17:10:42.910 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7379999999999999e-06 2024-04-20T17:10:42.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.910 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:10:42.910 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.790270+0000", 2024-04-20T17:10:42.910 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7570000000000004e-06 2024-04-20T17:10:42.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.910 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:10:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.790278+0000", 2024-04-20T17:10:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2649999999999996e-06 2024-04-20T17:10:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:10:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.790284+0000", 2024-04-20T17:10:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2299999999999999e-06 2024-04-20T17:10:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:10:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.790490+0000", 2024-04-20T17:10:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00020615999999999999 2024-04-20T17:10:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:10:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.790561+0000", 2024-04-20T17:10:42.912 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1202e-05 2024-04-20T17:10:42.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.912 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:10:42.912 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.790627+0000", 2024-04-20T17:10:42.912 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6366999999999997e-05 2024-04-20T17:10:42.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:10:42.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.793569+0000", 2024-04-20T17:10:42.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0029416799999999999 2024-04-20T17:10:42.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:10:42.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.847528+0000", 2024-04-20T17:10:42.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.053958675999999997 2024-04-20T17:10:42.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:10:42.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.847587+0000", 2024-04-20T17:10:42.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8881000000000001e-05 2024-04-20T17:10:42.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:10:42.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:10:42.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:10:42.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.847654+0000", 2024-04-20T17:10:42.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7286999999999995e-05 2024-04-20T17:10:42.914 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.914 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:42.914 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.914 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:10:42.914 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:10:42.914 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:10:43.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:42 smithi005 ceph-mon[39562]: pgmap v495: 17 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 15 active+clean; 7.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 676 op/s; 20018/240918 objects misplaced (8.309%); 14 MiB/s, 216 objects/s recovering 2024-04-20T17:10:43.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:42 smithi106 ceph-mon[44423]: pgmap v495: 17 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 15 active+clean; 7.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 676 op/s; 20018/240918 objects misplaced (8.309%); 14 MiB/s, 216 objects/s recovering 2024-04-20T17:10:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:42 smithi171 ceph-mon[44189]: pgmap v495: 17 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 15 active+clean; 7.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 676 op/s; 20018/240918 objects misplaced (8.309%); 14 MiB/s, 216 objects/s recovering 2024-04-20T17:10:45.069 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:10:45.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:45.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:10:45.173 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:10:45.203 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:45 smithi106 ceph-mon[44423]: pgmap v496: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 15 active+clean; 7.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 820 op/s; 18398/235782 objects misplaced (7.803%); 21 MiB/s, 336 objects/s recovering 2024-04-20T17:10:45.207 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:45.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:45 smithi171 ceph-mon[44189]: pgmap v496: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 15 active+clean; 7.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 820 op/s; 18398/235782 objects misplaced (7.803%); 21 MiB/s, 336 objects/s recovering 2024-04-20T17:10:45.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:45 smithi005 ceph-mon[39562]: pgmap v496: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 15 active+clean; 7.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 820 op/s; 18398/235782 objects misplaced (7.803%); 21 MiB/s, 336 objects/s recovering 2024-04-20T17:10:48.191 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:10:48.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:47 smithi005 ceph-mon[39562]: pgmap v497: 17 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 15 active+clean; 7.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 885 op/s; 17870/232374 objects misplaced (7.690%); 22 MiB/s, 345 objects/s recovering 2024-04-20T17:10:48.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:47 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:10:48.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:47 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:10:48.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:47 smithi106 ceph-mon[44423]: pgmap v497: 17 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 15 active+clean; 7.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 885 op/s; 17870/232374 objects misplaced (7.690%); 22 MiB/s, 345 objects/s recovering 2024-04-20T17:10:48.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:47 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:10:48.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:47 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:10:48.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:47 smithi171 ceph-mon[44189]: pgmap v497: 17 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 15 active+clean; 7.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 885 op/s; 17870/232374 objects misplaced (7.690%); 22 MiB/s, 345 objects/s recovering 2024-04-20T17:10:48.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:47 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:10:48.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:47 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:10:48.651 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:10:48.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:48.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:10:48.663 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-20T17:10:48.755 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:10:48.803 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:48.891 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:48 smithi005 ceph-mon[39562]: osdmap e168: 12 total, 12 up, 12 in 2024-04-20T17:10:48.891 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:48 smithi005 ceph-mon[39562]: 2.b scrub starts 2024-04-20T17:10:48.894 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:10:48.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:48.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:10:48.957 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:10:48.983 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-20T17:10:48.983 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:10:48.983 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-20T17:10:48.983 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-20T17:10:48.983 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.2 2024-04-20T17:10:48.990 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:10:49.005 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:10:49.014 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:10:49.044 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:10:49.046 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:49.081 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:10:49.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:49.086 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:10:49.106 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:10:49.140 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:10:49.179 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:10:49.187 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:10:49.213 INFO:journalctl@ceph.osd.2.smithi005.stdout:Apr 20 17:10:49 smithi005 systemd[1]: Stopping Ceph osd.2 for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:10:49.218 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:10:49.234 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:49.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:48 smithi106 ceph-mon[44423]: osdmap e168: 12 total, 12 up, 12 in 2024-04-20T17:10:49.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:48 smithi106 ceph-mon[44423]: 2.b scrub starts 2024-04-20T17:10:49.263 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:10:49.307 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:10:49.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:48 smithi171 ceph-mon[44189]: osdmap e168: 12 total, 12 up, 12 in 2024-04-20T17:10:49.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:48 smithi171 ceph-mon[44189]: 2.b scrub starts 2024-04-20T17:10:49.341 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:10:49.370 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:10:49.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:49.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:10:49.399 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:10:49.443 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:10:49.484 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:10:49.492 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:10:49.514 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:10:49.514 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:10:49.514 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:10:49.514 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:10:49.514 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:10:49.514 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:10:49.514 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:10:49.514 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:10:49.514 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:10:49.514 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:10:49.514 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:10:49.514 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:10:49.523 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:49.554 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:10:49.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:49.561 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:10:49.662 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:10:49.693 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:49.892 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:10:49.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:49.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:10:49.997 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:10:50.045 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:50.076 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:10:50.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:50.083 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:10:50.184 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:10:50.209 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:49 smithi005 ceph-mon[39562]: pgmap v499: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 15 active+clean; 7.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 929 op/s; 16337/228380 objects misplaced (7.153%); 24 MiB/s, 379 objects/s recovering 2024-04-20T17:10:50.209 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:49 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3679920918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:10:50.209 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:50.209 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:50.209 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:49 smithi005 ceph-mon[39562]: 2.b scrub ok 2024-04-20T17:10:50.210 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:50.210 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:50.210 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:50.210 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:50.210 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:50.210 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:50.210 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:50.210 INFO:journalctl@ceph.osd.2.smithi005.stdout:Apr 20 17:10:50 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-2[72612]: 2024-04-20T17:10:50.086+0000 7f190a520700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T17:10:50.211 INFO:journalctl@ceph.osd.2.smithi005.stdout:Apr 20 17:10:50 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-2[72612]: 2024-04-20T17:10:50.086+0000 7f190a520700 -1 osd.2 169 *** Got signal Terminated *** 2024-04-20T17:10:50.211 INFO:journalctl@ceph.osd.2.smithi005.stdout:Apr 20 17:10:50 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-2[72612]: 2024-04-20T17:10:50.086+0000 7f190a520700 -1 osd.2 169 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T17:10:50.228 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:49 smithi171 ceph-mon[44189]: pgmap v499: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 15 active+clean; 7.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 929 op/s; 16337/228380 objects misplaced (7.153%); 24 MiB/s, 379 objects/s recovering 2024-04-20T17:10:50.228 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:49 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3679920918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:10:50.228 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:50.228 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:50.228 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:49 smithi171 ceph-mon[44189]: 2.b scrub ok 2024-04-20T17:10:50.228 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:50.228 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:50.229 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:50.229 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:50.229 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:50.229 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:50.229 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:50.229 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:50.252 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:10:50.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:50.255 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:10:50.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:49 smithi106 ceph-mon[44423]: pgmap v499: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 15 active+clean; 7.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 929 op/s; 16337/228380 objects misplaced (7.153%); 24 MiB/s, 379 objects/s recovering 2024-04-20T17:10:50.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:49 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3679920918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:10:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:49 smithi106 ceph-mon[44423]: 2.b scrub ok 2024-04-20T17:10:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:50.356 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:10:50.385 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:50.411 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:10:50.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:50.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:10:50.516 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:10:50.570 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:50.604 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:10:50.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:50.610 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:10:50.711 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:10:50.758 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:50.791 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:10:50.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:50.799 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:10:50.900 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:10:50.940 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:50.961 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:10:50.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:50.964 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:10:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:50 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:10:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:50 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:10:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:50 smithi106 ceph-mon[44423]: osdmap e169: 12 total, 12 up, 12 in 2024-04-20T17:10:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:50 smithi106 ceph-mon[44423]: pgmap v501: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 1 active+remapped+backfilling, 14 active+clean; 6.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 996 op/s; 15817/224968 objects misplaced (7.031%); 26 MiB/s, 412 objects/s recovering 2024-04-20T17:10:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:50 smithi106 ceph-mon[44423]: osd.2 marked itself down and dead 2024-04-20T17:10:51.057 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:10:51.057 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:10:51.065 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:10:51.092 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:51.113 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:10:51.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:51.117 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:10:51.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:51.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:51.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:51.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:50 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:10:51.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:50 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:10:51.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:50 smithi005 ceph-mon[39562]: osdmap e169: 12 total, 12 up, 12 in 2024-04-20T17:10:51.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:50 smithi005 ceph-mon[39562]: pgmap v501: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 1 active+remapped+backfilling, 14 active+clean; 6.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 996 op/s; 15817/224968 objects misplaced (7.031%); 26 MiB/s, 412 objects/s recovering 2024-04-20T17:10:51.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:50 smithi005 ceph-mon[39562]: osd.2 marked itself down and dead 2024-04-20T17:10:51.218 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:10:51.244 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:51.244 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:51.244 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:10:51.244 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:50 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:10:51.244 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:50 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:10:51.244 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:50 smithi171 ceph-mon[44189]: osdmap e169: 12 total, 12 up, 12 in 2024-04-20T17:10:51.244 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:50 smithi171 ceph-mon[44189]: pgmap v501: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 1 active+remapped+backfilling, 14 active+clean; 6.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 996 op/s; 15817/224968 objects misplaced (7.031%); 26 MiB/s, 412 objects/s recovering 2024-04-20T17:10:51.245 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:50 smithi171 ceph-mon[44189]: osd.2 marked itself down and dead 2024-04-20T17:10:51.246 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:51.266 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:10:51.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:51.270 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:10:51.371 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:10:51.400 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:51.751 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:10:51.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:51.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:10:51.755 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-20T17:10:51.855 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:10:51.882 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:51.901 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:10:51.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:51.904 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:10:52.004 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:10:52.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:51 smithi106 ceph-mon[44423]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T17:10:52.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:51 smithi106 ceph-mon[44423]: osdmap e170: 12 total, 11 up, 12 in 2024-04-20T17:10:52.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:51 smithi106 ceph-mon[44423]: 2.a scrub starts 2024-04-20T17:10:52.031 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:52.051 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:10:52.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:52.053 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:10:52.080 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:10:52.154 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:10:52.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:51 smithi005 ceph-mon[39562]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T17:10:52.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:51 smithi005 ceph-mon[39562]: osdmap e170: 12 total, 11 up, 12 in 2024-04-20T17:10:52.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:51 smithi005 ceph-mon[39562]: 2.a scrub starts 2024-04-20T17:10:52.183 INFO:journalctl@ceph.osd.2.smithi005.stdout:Apr 20 17:10:52 smithi005 bash[109600]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-2 2024-04-20T17:10:52.186 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:52.317 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:10:52.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:52.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:10:52.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:51 smithi171 ceph-mon[44189]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T17:10:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:51 smithi171 ceph-mon[44189]: osdmap e170: 12 total, 11 up, 12 in 2024-04-20T17:10:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:51 smithi171 ceph-mon[44189]: 2.a scrub starts 2024-04-20T17:10:52.420 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:10:52.450 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:52.574 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:10:52.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:52.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:10:52.677 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:10:52.703 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:52.724 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:10:52.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:52.727 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:10:52.828 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:10:52.854 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:52.874 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:10:52.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:52.877 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:10:52.922 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:10:52.978 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:10:53.006 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:53.026 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:10:53.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:53.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:10:53.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:52 smithi171 ceph-mon[44189]: 2.a scrub ok 2024-04-20T17:10:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:52 smithi171 ceph-mon[44189]: osdmap e171: 12 total, 11 up, 12 in 2024-04-20T17:10:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:52 smithi171 ceph-mon[44189]: pgmap v504: 17 pgs: 2 peering, 1 stale+active+recovering+remapped, 1 active+clean+scrubbing, 1 active+remapped+backfilling, 12 active+clean; 6.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 722 op/s; 15817/222434 objects misplaced (7.111%); 3.9 MiB/s, 63 objects/s recovering 2024-04-20T17:10:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:52 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3018647021' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:10:53.129 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:10:53.156 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:53.175 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:10:53.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:53.178 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:10:53.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:52 smithi005 ceph-mon[39562]: 2.a scrub ok 2024-04-20T17:10:53.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:52 smithi005 ceph-mon[39562]: osdmap e171: 12 total, 11 up, 12 in 2024-04-20T17:10:53.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:52 smithi005 ceph-mon[39562]: pgmap v504: 17 pgs: 2 peering, 1 stale+active+recovering+remapped, 1 active+clean+scrubbing, 1 active+remapped+backfilling, 12 active+clean; 6.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 722 op/s; 15817/222434 objects misplaced (7.111%); 3.9 MiB/s, 63 objects/s recovering 2024-04-20T17:10:53.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:52 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3018647021' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:10:53.253 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:10:53.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:52 smithi106 ceph-mon[44423]: 2.a scrub ok 2024-04-20T17:10:53.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:52 smithi106 ceph-mon[44423]: osdmap e171: 12 total, 11 up, 12 in 2024-04-20T17:10:53.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:52 smithi106 ceph-mon[44423]: pgmap v504: 17 pgs: 2 peering, 1 stale+active+recovering+remapped, 1 active+clean+scrubbing, 1 active+remapped+backfilling, 12 active+clean; 6.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 722 op/s; 15817/222434 objects misplaced (7.111%); 3.9 MiB/s, 63 objects/s recovering 2024-04-20T17:10:53.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:52 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3018647021' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:10:53.279 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:10:53.305 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:53.324 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:10:53.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:53.327 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:10:53.428 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:10:53.458 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:54.001 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:10:54.001 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:10:54.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:53 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:10:54.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:53 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3018647021' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:10:54.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:53 smithi106 ceph-mon[44423]: osdmap e172: 12 total, 11 up, 12 in 2024-04-20T17:10:54.110 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:10:54.112 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:53 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:10:54.112 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:53 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3018647021' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:10:54.112 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:53 smithi005 ceph-mon[39562]: osdmap e172: 12 total, 11 up, 12 in 2024-04-20T17:10:54.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:54.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:10:54.179 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:10:54.214 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:10:54.246 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:53 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:10:54.246 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:53 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3018647021' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:10:54.246 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:53 smithi171 ceph-mon[44189]: osdmap e172: 12 total, 11 up, 12 in 2024-04-20T17:10:54.248 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:54.274 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:10:54.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:54.277 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:10:54.378 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:10:54.408 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:54.431 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:10:54.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:54.435 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:10:54.535 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:10:54.564 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:54.687 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:10:54.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:54.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:10:54.693 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-20T17:10:54.791 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:10:54.817 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:54.836 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:10:54.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:54.838 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:10:54.939 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:10:54.967 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:54.986 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:10:54.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:54.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:10:54.995 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:10:55.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:54 smithi171 ceph-mon[44189]: 2.7 scrub starts 2024-04-20T17:10:55.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:54 smithi171 ceph-mon[44189]: 2.5 scrub starts 2024-04-20T17:10:55.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:54 smithi171 ceph-mon[44189]: pgmap v506: 17 pgs: 1 active+remapped+backfilling, 2 peering, 1 stale+active+recovering+remapped, 13 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 682 op/s; 13735/219386 objects misplaced (6.261%); 31 MiB/s, 489 objects/s recovering 2024-04-20T17:10:55.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:54 smithi171 ceph-mon[44189]: 2.c scrub starts 2024-04-20T17:10:55.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:54 smithi171 ceph-mon[44189]: 2.7 scrub ok 2024-04-20T17:10:55.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:54 smithi171 ceph-mon[44189]: 2.5 scrub ok 2024-04-20T17:10:55.090 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:10:55.116 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:55.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:54 smithi005 ceph-mon[39562]: 2.7 scrub starts 2024-04-20T17:10:55.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:54 smithi005 ceph-mon[39562]: 2.5 scrub starts 2024-04-20T17:10:55.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:54 smithi005 ceph-mon[39562]: pgmap v506: 17 pgs: 1 active+remapped+backfilling, 2 peering, 1 stale+active+recovering+remapped, 13 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 682 op/s; 13735/219386 objects misplaced (6.261%); 31 MiB/s, 489 objects/s recovering 2024-04-20T17:10:55.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:54 smithi005 ceph-mon[39562]: 2.c scrub starts 2024-04-20T17:10:55.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:54 smithi005 ceph-mon[39562]: 2.7 scrub ok 2024-04-20T17:10:55.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:54 smithi005 ceph-mon[39562]: 2.5 scrub ok 2024-04-20T17:10:55.249 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:54 smithi106 ceph-mon[44423]: 2.7 scrub starts 2024-04-20T17:10:55.249 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:54 smithi106 ceph-mon[44423]: 2.5 scrub starts 2024-04-20T17:10:55.249 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:54 smithi106 ceph-mon[44423]: pgmap v506: 17 pgs: 1 active+remapped+backfilling, 2 peering, 1 stale+active+recovering+remapped, 13 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 682 op/s; 13735/219386 objects misplaced (6.261%); 31 MiB/s, 489 objects/s recovering 2024-04-20T17:10:55.249 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:54 smithi106 ceph-mon[44423]: 2.c scrub starts 2024-04-20T17:10:55.249 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:54 smithi106 ceph-mon[44423]: 2.7 scrub ok 2024-04-20T17:10:55.249 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:54 smithi106 ceph-mon[44423]: 2.5 scrub ok 2024-04-20T17:10:55.265 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:10:55.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:55.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:10:55.370 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:10:55.397 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:55.745 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:10:55.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:55.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:10:55.748 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:10:55.848 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:10:55.879 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:55.901 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:10:55.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:55.904 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:10:56.005 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:10:56.033 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:56.167 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:55 smithi106 ceph-mon[44423]: 2.c scrub ok 2024-04-20T17:10:56.167 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:55 smithi106 ceph-mon[44423]: osdmap e173: 12 total, 11 up, 12 in 2024-04-20T17:10:56.181 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:10:56.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:56.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:10:56.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:55 smithi005 ceph-mon[39562]: 2.c scrub ok 2024-04-20T17:10:56.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:55 smithi005 ceph-mon[39562]: osdmap e173: 12 total, 11 up, 12 in 2024-04-20T17:10:56.240 DEBUG:teuthology.orchestra.run.smithi005:> sudo pkill -f 'journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.2.service' 2024-04-20T17:10:56.286 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:10:56.292 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T17:10:56.292 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-20T17:10:56.292 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.2 2024-04-20T17:10:56.292 INFO:tasks.thrashosds.thrasher:remote for osd 2 is ubuntu@smithi005.front.sepia.ceph.com 2024-04-20T17:10:56.293 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.2 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op list-pgs 2024-04-20T17:10:56.312 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:56.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:55 smithi171 ceph-mon[44189]: 2.c scrub ok 2024-04-20T17:10:56.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:55 smithi171 ceph-mon[44189]: osdmap e173: 12 total, 11 up, 12 in 2024-04-20T17:10:56.510 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:10:56.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:56.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:10:56.601 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.2/config 2024-04-20T17:10:56.613 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:10:56.652 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:56.763 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:10:56.763 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:10:56.901 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:10:56.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:56.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:10:57.006 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:10:57.035 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:56 smithi106 ceph-mon[44423]: 2.f scrub starts 2024-04-20T17:10:57.035 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:56 smithi106 ceph-mon[44423]: osdmap e174: 12 total, 11 up, 12 in 2024-04-20T17:10:57.035 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:56 smithi106 ceph-mon[44423]: pgmap v509: 17 pgs: 1 active+remapped+backfilling, 2 peering, 1 stale+active+recovering+remapped, 13 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 388 op/s; 13735/219272 objects misplaced (6.264%); 31 MiB/s, 491 objects/s recovering 2024-04-20T17:10:57.035 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:56 smithi106 ceph-mon[44423]: 2.f scrub ok 2024-04-20T17:10:57.038 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:57.161 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:10:57.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:57.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:10:57.201 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:56 smithi005 ceph-mon[39562]: 2.f scrub starts 2024-04-20T17:10:57.201 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:56 smithi005 ceph-mon[39562]: osdmap e174: 12 total, 11 up, 12 in 2024-04-20T17:10:57.201 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:56 smithi005 ceph-mon[39562]: pgmap v509: 17 pgs: 1 active+remapped+backfilling, 2 peering, 1 stale+active+recovering+remapped, 13 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 388 op/s; 13735/219272 objects misplaced (6.264%); 31 MiB/s, 491 objects/s recovering 2024-04-20T17:10:57.201 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:56 smithi005 ceph-mon[39562]: 2.f scrub ok 2024-04-20T17:10:57.264 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:10:57.291 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:57.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:56 smithi171 ceph-mon[44189]: 2.f scrub starts 2024-04-20T17:10:57.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:56 smithi171 ceph-mon[44189]: osdmap e174: 12 total, 11 up, 12 in 2024-04-20T17:10:57.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:56 smithi171 ceph-mon[44189]: pgmap v509: 17 pgs: 1 active+remapped+backfilling, 2 peering, 1 stale+active+recovering+remapped, 13 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 388 op/s; 13735/219272 objects misplaced (6.264%); 31 MiB/s, 491 objects/s recovering 2024-04-20T17:10:57.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:56 smithi171 ceph-mon[44189]: 2.f scrub ok 2024-04-20T17:10:57.438 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:10:57.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:57.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:10:57.458 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-20T17:10:57.542 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:10:57.569 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:57.588 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:10:57.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:57.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:10:57.684 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:10:57.692 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:10:57.719 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:57.729 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:10:57.740 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:10:57.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:57.743 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:10:57.844 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:10:57.874 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:58.007 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:57 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1939316057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:10:58.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:57 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1939316057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:10:58.227 INFO:teuthology.orchestra.run.smithi005.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:10:58.228 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:10:58.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:58.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:10:58.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:57 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1939316057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:10:58.332 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:10:58.365 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:58.386 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:10:58.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:58.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:10:58.491 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:10:58.520 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:58.593 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:10:58.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:58.596 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:10:58.697 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:10:58.727 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:58.878 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:10:58.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:58.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:10:58.983 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:10:59.012 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:58 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:10:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:58 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:10:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:58 smithi005 ceph-mon[39562]: osdmap e175: 12 total, 11 up, 12 in 2024-04-20T17:10:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:58 smithi005 ceph-mon[39562]: pgmap v511: 17 pgs: 2 undersized+degraded+peered, 1 stale+active+recovering+remapped, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 87 op/s; 13743/219272 objects degraded (6.268%); 12752/219272 objects misplaced (5.816%); 12 MiB/s, 193 objects/s recovering 2024-04-20T17:10:59.222 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:10:59.232 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:58 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:10:59.232 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:58 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:10:59.232 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:58 smithi171 ceph-mon[44189]: osdmap e175: 12 total, 11 up, 12 in 2024-04-20T17:10:59.232 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:58 smithi171 ceph-mon[44189]: pgmap v511: 17 pgs: 2 undersized+degraded+peered, 1 stale+active+recovering+remapped, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 87 op/s; 13743/219272 objects degraded (6.268%); 12752/219272 objects misplaced (5.816%); 12 MiB/s, 193 objects/s recovering 2024-04-20T17:10:59.249 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:10:59.260 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:10:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:58 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:10:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:58 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:10:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:58 smithi106 ceph-mon[44423]: osdmap e175: 12 total, 11 up, 12 in 2024-04-20T17:10:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:58 smithi106 ceph-mon[44423]: pgmap v511: 17 pgs: 2 undersized+degraded+peered, 1 stale+active+recovering+remapped, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 87 op/s; 13743/219272 objects degraded (6.268%); 12752/219272 objects misplaced (5.816%); 12 MiB/s, 193 objects/s recovering 2024-04-20T17:10:59.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:59.265 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:10:59.269 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:10:59.290 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:10:59.311 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:10:59.335 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:10:59.366 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:10:59.367 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:10:59.392 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:59.404 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:10:59.438 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:10:59.469 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:10:59.492 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:10:59.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:59.495 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:10:59.496 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:10:59.496 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:10:59.496 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:10:59.497 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-20T17:10:59.497 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:10:59.497 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:10:59.497 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:10:59.497 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:10:59.497 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:10:59.497 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:10:59.497 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:10:59.497 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:10:59.497 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:10:59.595 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:10:59.623 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:59.650 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:10:59.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:59.653 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:10:59.754 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:10:59.784 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:59.793 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:10:59.793 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:10:59.793 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 0 2024-04-20T17:10:59.793 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:10:59.804 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:10:59.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:59.808 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:10:59.909 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:10:59.943 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:10:59.963 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:10:59.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:10:59.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:11:00.067 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:11:00.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:59 smithi171 ceph-mon[44189]: Health check failed: Degraded data redundancy: 13743/219272 objects degraded (6.268%), 2 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:00.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:00.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:00.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:00.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:00.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:00.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:00.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:10:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:00.093 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:00.117 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:11:00.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:00.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:11:00.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:59 smithi005 ceph-mon[39562]: Health check failed: Degraded data redundancy: 13743/219272 objects degraded (6.268%), 2 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:00.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:00.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:00.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:00.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:00.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:00.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:00.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:10:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:00.221 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:11:00.245 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:59 smithi106 ceph-mon[44423]: Health check failed: Degraded data redundancy: 13743/219272 objects degraded (6.268%), 2 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:00.245 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:00.245 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:00.245 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:00.245 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:00.245 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:00.245 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:00.245 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:10:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:00.247 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:00.266 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:11:00.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:00.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:11:00.370 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:11:00.399 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:00.747 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:11:00.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:00.750 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:11:00.768 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-20T17:11:00.847 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:11:00.848 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:11:00.851 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:11:00.877 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:00.896 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:11:00.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:00.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:11:01.002 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:11:01.034 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:01.056 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:11:01.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:01.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:11:01.071 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:11:01.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:01.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:01.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:00 smithi171 ceph-mon[44189]: 2.0 scrub starts 2024-04-20T17:11:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:00 smithi171 ceph-mon[44189]: pgmap v512: 17 pgs: 2 undersized+degraded+peered, 1 active+recovery_wait+degraded, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 24 op/s; 26311/219088 objects degraded (12.009%); 11 MiB/s, 178 objects/s recovering 2024-04-20T17:11:01.159 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:11:01.185 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:01.188 INFO:teuthology.orchestra.run.smithi005.stdout:2.1 2024-04-20T17:11:01.188 INFO:teuthology.orchestra.run.smithi005.stdout:2.e 2024-04-20T17:11:01.188 INFO:teuthology.orchestra.run.smithi005.stdout:2.3 2024-04-20T17:11:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:00 smithi005 ceph-mon[39562]: 2.0 scrub starts 2024-04-20T17:11:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:00 smithi005 ceph-mon[39562]: pgmap v512: 17 pgs: 2 undersized+degraded+peered, 1 active+recovery_wait+degraded, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 24 op/s; 26311/219088 objects degraded (12.009%); 11 MiB/s, 178 objects/s recovering 2024-04-20T17:11:01.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:00 smithi106 ceph-mon[44423]: 2.0 scrub starts 2024-04-20T17:11:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:00 smithi106 ceph-mon[44423]: pgmap v512: 17 pgs: 2 undersized+degraded+peered, 1 active+recovery_wait+degraded, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 24 op/s; 26311/219088 objects degraded (12.009%); 11 MiB/s, 178 objects/s recovering 2024-04-20T17:11:01.337 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:11:01.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:01.339 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:11:01.440 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:11:01.467 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:01.486 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:11:01.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:01.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:11:01.590 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:11:01.618 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:01.950 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:11:01.967 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:11:01.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:01.970 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:11:02.071 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:11:02.105 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:02.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:01 smithi005 ceph-mon[39562]: 2.0 scrub ok 2024-04-20T17:11:02.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:01 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3415435525' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:11:02.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:01 smithi106 ceph-mon[44423]: 2.0 scrub ok 2024-04-20T17:11:02.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:01 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3415435525' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:11:02.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:01 smithi171 ceph-mon[44189]: 2.0 scrub ok 2024-04-20T17:11:02.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:01 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3415435525' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:11:02.514 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.2 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op export --pgid 2.e --file /var/log/ceph/exp.2.e.2 2024-04-20T17:11:02.755 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:11:02.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:02.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:11:02.781 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.2/config 2024-04-20T17:11:02.848 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:11:02.860 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:11:02.893 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:02.914 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:11:02.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:02.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:11:03.019 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:11:03.048 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:03.068 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:11:03.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:03.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:11:03.092 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:02 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:11:03.092 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:02 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3415435525' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:11:03.092 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:02 smithi005 ceph-mon[39562]: osdmap e176: 12 total, 11 up, 12 in 2024-04-20T17:11:03.092 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:02 smithi005 ceph-mon[39562]: pgmap v514: 17 pgs: 2 undersized+degraded+peered, 1 active+recovery_wait+degraded, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 16 op/s; 26311/219072 objects degraded (12.010%); 11 MiB/s, 176 objects/s recovering 2024-04-20T17:11:03.161 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:11:03.161 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:11:03.161 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-04-20T17:11:03.161 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-04-20T17:11:03.161 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:11:03.171 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:11:03.180 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:02 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:11:03.180 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:02 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3415435525' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:11:03.181 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:02 smithi171 ceph-mon[44189]: osdmap e176: 12 total, 11 up, 12 in 2024-04-20T17:11:03.181 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:02 smithi171 ceph-mon[44189]: pgmap v514: 17 pgs: 2 undersized+degraded+peered, 1 active+recovery_wait+degraded, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 16 op/s; 26311/219072 objects degraded (12.010%); 11 MiB/s, 176 objects/s recovering 2024-04-20T17:11:03.201 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:03.221 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:11:03.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:03.224 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:11:03.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:02 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:11:03.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:02 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3415435525' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:11:03.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:02 smithi106 ceph-mon[44423]: osdmap e176: 12 total, 11 up, 12 in 2024-04-20T17:11:03.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:02 smithi106 ceph-mon[44423]: pgmap v514: 17 pgs: 2 undersized+degraded+peered, 1 active+recovery_wait+degraded, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 16 op/s; 26311/219072 objects degraded (12.010%); 11 MiB/s, 176 objects/s recovering 2024-04-20T17:11:03.325 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:11:03.355 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:03.375 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:11:03.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:03.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:11:03.479 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:11:03.509 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:03.635 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:11:03.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:03.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:11:03.740 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:11:03.768 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:03.788 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:11:03.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:03.790 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:11:03.808 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-20T17:11:03.891 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:11:03.924 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:03.946 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:11:03.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:03.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:11:04.050 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:11:04.077 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:03 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:11:04.077 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:03 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:11:04.079 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:04.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:03 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:11:04.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:03 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:11:04.107 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:11:04.118 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:11:04.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:04.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:11:04.148 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:11:04.222 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:11:04.250 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:04.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:03 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:11:04.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:03 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:11:04.449 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:11:04.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:04.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:11:04.553 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:11:04.581 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:04.929 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:11:04.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:04.932 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:11:05.033 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:11:05.058 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:04 smithi005 ceph-mon[39562]: osdmap e177: 12 total, 11 up, 12 in 2024-04-20T17:11:05.059 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:04 smithi005 ceph-mon[39562]: pgmap v516: 17 pgs: 2 undersized+degraded+peered, 1 active+recovery_wait+degraded, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 21 op/s; 26311/219006 objects degraded (12.014%); 931 KiB/s, 14 objects/s recovering 2024-04-20T17:11:05.062 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:05.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:04 smithi106 ceph-mon[44423]: osdmap e177: 12 total, 11 up, 12 in 2024-04-20T17:11:05.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:04 smithi106 ceph-mon[44423]: pgmap v516: 17 pgs: 2 undersized+degraded+peered, 1 active+recovery_wait+degraded, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 21 op/s; 26311/219006 objects degraded (12.014%); 931 KiB/s, 14 objects/s recovering 2024-04-20T17:11:05.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:04 smithi171 ceph-mon[44189]: osdmap e177: 12 total, 11 up, 12 in 2024-04-20T17:11:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:04 smithi171 ceph-mon[44189]: pgmap v516: 17 pgs: 2 undersized+degraded+peered, 1 active+recovery_wait+degraded, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 21 op/s; 26311/219006 objects degraded (12.014%); 931 KiB/s, 14 objects/s recovering 2024-04-20T17:11:05.645 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:11:06.121 INFO:teuthology.orchestra.run.smithi005.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:11:06.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:05 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 26311/219006 objects degraded (12.014%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:06.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:05 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 26311/219006 objects degraded (12.014%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:06.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:05 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 26311/219006 objects degraded (12.014%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:07.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:06 smithi106 ceph-mon[44423]: pgmap v517: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 18 op/s; 19978/218982 objects degraded (9.123%); 6237/218982 objects misplaced (2.848%); 1.1 MiB/s, 16 objects/s recovering 2024-04-20T17:11:07.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:06 smithi171 ceph-mon[44189]: pgmap v517: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 18 op/s; 19978/218982 objects degraded (9.123%); 6237/218982 objects misplaced (2.848%); 1.1 MiB/s, 16 objects/s recovering 2024-04-20T17:11:07.355 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:06 smithi005 ceph-mon[39562]: pgmap v517: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 18 op/s; 19978/218982 objects degraded (9.123%); 6237/218982 objects misplaced (2.848%); 1.1 MiB/s, 16 objects/s recovering 2024-04-20T17:11:07.714 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:11:07.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:07.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:11:07.805 INFO:teuthology.orchestra.run.smithi005.stderr:Exporting 2.e info 2.e( v 169'13122 (124'3100,169'13122] local-lis/les=115/116 n=6874 ec=115/115 lis/c=115/115 les/c/f=116/116/0 sis=115) 2024-04-20T17:11:07.819 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:11:07.852 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:07.853 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7000bc6a:::benchmark_data_smithi171_78918_object82634:head# 2024-04-20T17:11:07.853 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70015d79:::benchmark_data_smithi171_78918_object155850:head# 2024-04-20T17:11:07.853 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:700197b5:::benchmark_data_smithi171_78918_object127114:head# 2024-04-20T17:11:07.854 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7001b431:::benchmark_data_smithi171_78918_object107278:head# 2024-04-20T17:11:07.854 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7001ddbd:::benchmark_data_smithi171_78918_object129901:head# 2024-04-20T17:11:07.854 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7002cc2d:::benchmark_data_smithi171_78918_object117882:head# 2024-04-20T17:11:07.857 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70034c34:::benchmark_data_smithi171_78918_object87278:head# 2024-04-20T17:11:07.857 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70044159:::benchmark_data_smithi171_78918_object103640:head# 2024-04-20T17:11:07.857 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7004a58e:::benchmark_data_smithi171_78918_object103906:head# 2024-04-20T17:11:07.857 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:700586a8:::benchmark_data_smithi171_78918_object98955:head# 2024-04-20T17:11:07.857 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7007365f:::benchmark_data_smithi171_78918_object74153:head# 2024-04-20T17:11:07.857 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7007d04f:::benchmark_data_smithi171_78918_object143991:head# 2024-04-20T17:11:07.857 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:700907eb:::benchmark_data_smithi171_78918_object98383:head# 2024-04-20T17:11:07.858 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70096acc:::benchmark_data_smithi171_78918_object136914:head# 2024-04-20T17:11:07.858 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70097321:::benchmark_data_smithi171_78918_object150953:head# 2024-04-20T17:11:07.858 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70097406:::benchmark_data_smithi171_78918_object122216:head# 2024-04-20T17:11:07.858 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:700a359d:::benchmark_data_smithi171_78918_object133687:head# 2024-04-20T17:11:07.858 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:700a541f:::benchmark_data_smithi171_78918_object64253:head# 2024-04-20T17:11:07.858 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:700b7861:::benchmark_data_smithi171_78918_object61952:head# 2024-04-20T17:11:07.858 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:700be0c2:::benchmark_data_smithi171_78918_object63282:head# 2024-04-20T17:11:07.858 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:700c9327:::benchmark_data_smithi171_78918_object154704:head# 2024-04-20T17:11:07.858 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:700cb37d:::benchmark_data_smithi171_78918_object88807:head# 2024-04-20T17:11:07.858 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:700dc333:::benchmark_data_smithi171_78918_object81185:head# 2024-04-20T17:11:07.858 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:700e9018:::benchmark_data_smithi171_78918_object77225:head# 2024-04-20T17:11:07.858 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:700f2a35:::benchmark_data_smithi171_78918_object145504:head# 2024-04-20T17:11:07.858 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70103e23:::benchmark_data_smithi171_78918_object113823:head# 2024-04-20T17:11:07.859 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7010deb6:::benchmark_data_smithi171_78918_object118449:head# 2024-04-20T17:11:07.859 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:701168d0:::benchmark_data_smithi171_78918_object113466:head# 2024-04-20T17:11:07.859 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70127c53:::benchmark_data_smithi171_78918_object116578:head# 2024-04-20T17:11:07.859 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7012c51c:::benchmark_data_smithi171_78918_object152990:head# 2024-04-20T17:11:07.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70141c23:::benchmark_data_smithi171_78918_object95591:head# 2024-04-20T17:11:07.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70144fa2:::benchmark_data_smithi171_78918_object133363:head# 2024-04-20T17:11:07.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7014e6fa:::benchmark_data_smithi171_78918_object94911:head# 2024-04-20T17:11:07.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7015d971:::benchmark_data_smithi171_78918_object56344:head# 2024-04-20T17:11:07.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7015de0f:::benchmark_data_smithi171_78918_object115017:head# 2024-04-20T17:11:07.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7015e9d0:::benchmark_data_smithi171_78918_object85827:head# 2024-04-20T17:11:07.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7015ed40:::benchmark_data_smithi171_78918_object156013:head# 2024-04-20T17:11:07.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70164f4b:::benchmark_data_smithi171_78918_object69039:head# 2024-04-20T17:11:07.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:701678ed:::benchmark_data_smithi171_78918_object79234:head# 2024-04-20T17:11:07.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7016b541:::benchmark_data_smithi171_78918_object145831:head# 2024-04-20T17:11:07.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7016e3c6:::benchmark_data_smithi171_78918_object153224:head# 2024-04-20T17:11:07.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70176526:::benchmark_data_smithi171_78918_object83229:head# 2024-04-20T17:11:07.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70178ebc:::benchmark_data_smithi171_78918_object58235:head# 2024-04-20T17:11:07.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7017accd:::benchmark_data_smithi171_78918_object143934:head# 2024-04-20T17:11:07.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7017e8b2:::benchmark_data_smithi171_78918_object75407:head# 2024-04-20T17:11:07.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70184a55:::benchmark_data_smithi171_78918_object83752:head# 2024-04-20T17:11:07.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7018777e:::benchmark_data_smithi171_78918_object143906:head# 2024-04-20T17:11:07.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70188004:::benchmark_data_smithi171_78918_object111170:head# 2024-04-20T17:11:07.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7018c049:::benchmark_data_smithi171_78918_object131212:head# 2024-04-20T17:11:07.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7019dc0c:::benchmark_data_smithi171_78918_object128848:head# 2024-04-20T17:11:07.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:701ad703:::benchmark_data_smithi171_78918_object141311:head# 2024-04-20T17:11:07.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:701b6502:::benchmark_data_smithi171_78918_object100549:head# 2024-04-20T17:11:07.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:701bda2c:::benchmark_data_smithi171_78918_object151968:head# 2024-04-20T17:11:07.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:701c1edf:::benchmark_data_smithi171_78918_object116371:head# 2024-04-20T17:11:07.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:701cc349:::benchmark_data_smithi171_78918_object147087:head# 2024-04-20T17:11:07.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:701d0ecd:::benchmark_data_smithi171_78918_object160904:head# 2024-04-20T17:11:07.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:701dce6d:::benchmark_data_smithi171_78918_object107752:head# 2024-04-20T17:11:07.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:701e15ca:::benchmark_data_smithi171_78918_object54877:head# 2024-04-20T17:11:07.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:701e75a8:::benchmark_data_smithi171_78918_object108083:head# 2024-04-20T17:11:07.866 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:701ead85:::benchmark_data_smithi171_78918_object108260:head# 2024-04-20T17:11:07.866 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:701eb6c3:::benchmark_data_smithi171_78918_object101269:head# 2024-04-20T17:11:07.866 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:701f1472:::benchmark_data_smithi171_78918_object72778:head# 2024-04-20T17:11:07.866 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:701fbbd1:::benchmark_data_smithi171_78918_object95833:head# 2024-04-20T17:11:07.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70201de0:::benchmark_data_smithi171_78918_object126816:head# 2024-04-20T17:11:07.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70202874:::benchmark_data_smithi171_78918_object93496:head# 2024-04-20T17:11:07.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7021adf0:::benchmark_data_smithi171_78918_object146245:head# 2024-04-20T17:11:07.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7022d2ee:::benchmark_data_smithi171_78918_object151189:head# 2024-04-20T17:11:07.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70243ae5:::benchmark_data_smithi171_78918_object57938:head# 2024-04-20T17:11:07.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70243bcb:::benchmark_data_smithi171_78918_object108353:head# 2024-04-20T17:11:07.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7024ea64:::benchmark_data_smithi171_78918_object109789:head# 2024-04-20T17:11:07.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70251e13:::benchmark_data_smithi171_78918_object132866:head# 2024-04-20T17:11:07.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7025514b:::benchmark_data_smithi171_78918_object72346:head# 2024-04-20T17:11:07.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:702590a7:::benchmark_data_smithi171_78918_object149833:head# 2024-04-20T17:11:07.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70259bc2:::benchmark_data_smithi171_78918_object86226:head# 2024-04-20T17:11:07.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7025eabb:::benchmark_data_smithi171_78918_object51877:head# 2024-04-20T17:11:07.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:702622c1:::benchmark_data_smithi171_78918_object57456:head# 2024-04-20T17:11:07.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70271589:::benchmark_data_smithi171_78918_object83554:head# 2024-04-20T17:11:07.868 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7027b65f:::benchmark_data_smithi171_78918_object59313:head# 2024-04-20T17:11:07.868 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7029849a:::benchmark_data_smithi171_78918_object113889:head# 2024-04-20T17:11:07.868 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7029e801:::benchmark_data_smithi171_78918_object86767:head# 2024-04-20T17:11:07.868 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:702a27a6:::benchmark_data_smithi171_78918_object66426:head# 2024-04-20T17:11:07.868 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:702a7ec9:::benchmark_data_smithi171_78918_object120265:head# 2024-04-20T17:11:07.868 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:702b2705:::benchmark_data_smithi171_78918_object99547:head# 2024-04-20T17:11:07.868 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:702bcd58:::benchmark_data_smithi171_78918_object70152:head# 2024-04-20T17:11:07.868 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:702c3a0b:::benchmark_data_smithi171_78918_object55483:head# 2024-04-20T17:11:07.871 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:702c60b8:::benchmark_data_smithi171_78918_object79336:head# 2024-04-20T17:11:07.872 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:702cb0e9:::benchmark_data_smithi171_78918_object56201:head# 2024-04-20T17:11:07.872 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:702cbb41:::benchmark_data_smithi171_78918_object56889:head# 2024-04-20T17:11:07.872 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:702d1976:::benchmark_data_smithi171_78918_object104110:head# 2024-04-20T17:11:07.872 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:702d38ac:::benchmark_data_smithi171_78918_object66528:head# 2024-04-20T17:11:07.872 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:702ddb0a:::benchmark_data_smithi171_78918_object139351:head# 2024-04-20T17:11:07.872 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:702e51fa:::benchmark_data_smithi171_78918_object143550:head# 2024-04-20T17:11:07.872 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:702f5978:::benchmark_data_smithi171_78918_object63847:head# 2024-04-20T17:11:07.872 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:702fb0c1:::benchmark_data_smithi171_78918_object94314:head# 2024-04-20T17:11:07.872 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7030205d:::benchmark_data_smithi171_78918_object133681:head# 2024-04-20T17:11:07.872 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7031e6dd:::benchmark_data_smithi171_78918_object115758:head# 2024-04-20T17:11:07.872 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70339456:::benchmark_data_smithi171_78918_object84572:head# 2024-04-20T17:11:07.872 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7033e4bb:::benchmark_data_smithi171_78918_object64579:head# 2024-04-20T17:11:07.872 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:703437b4:::benchmark_data_smithi171_78918_object78631:head# 2024-04-20T17:11:07.872 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:703448ad:::benchmark_data_smithi171_78918_object65007:head# 2024-04-20T17:11:07.873 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7034a433:::benchmark_data_smithi171_78918_object99108:head# 2024-04-20T17:11:07.873 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7034c015:::benchmark_data_smithi171_78918_object135596:head# 2024-04-20T17:11:07.873 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7034fcae:::benchmark_data_smithi171_78918_object63846:head# 2024-04-20T17:11:07.873 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:703538a5:::benchmark_data_smithi171_78918_object101381:head# 2024-04-20T17:11:07.873 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70355ec8:::benchmark_data_smithi171_78918_object153673:head# 2024-04-20T17:11:07.873 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70365ddd:::benchmark_data_smithi171_78918_object78146:head# 2024-04-20T17:11:07.873 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7037091b:::benchmark_data_smithi171_78918_object107008:head# 2024-04-20T17:11:07.873 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70377a1d:::benchmark_data_smithi171_78918_object156036:head# 2024-04-20T17:11:07.873 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:703897fb:::benchmark_data_smithi171_78918_object128829:head# 2024-04-20T17:11:07.873 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70393cb8:::benchmark_data_smithi171_78918_object99788:head# 2024-04-20T17:11:07.874 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:703960a1:::benchmark_data_smithi171_78918_object111841:head# 2024-04-20T17:11:07.874 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:703b7fc7:::benchmark_data_smithi171_78918_object113696:head# 2024-04-20T17:11:07.874 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:11:07.878 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:703ece31:::benchmark_data_smithi171_78918_object63307:head# 2024-04-20T17:11:07.878 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7040b3e0:::benchmark_data_smithi171_78918_object73292:head# 2024-04-20T17:11:07.878 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7040c563:::benchmark_data_smithi171_78918_object112545:head# 2024-04-20T17:11:07.878 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7041c63c:::benchmark_data_smithi171_78918_object125434:head# 2024-04-20T17:11:07.878 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7042c7c9:::benchmark_data_smithi171_78918_object107473:head# 2024-04-20T17:11:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70433bcc:::benchmark_data_smithi171_78918_object76023:head# 2024-04-20T17:11:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70435b7b:::benchmark_data_smithi171_78918_object116135:head# 2024-04-20T17:11:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7043b50a:::benchmark_data_smithi171_78918_object120853:head# 2024-04-20T17:11:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7043d748:::benchmark_data_smithi171_78918_object75455:head# 2024-04-20T17:11:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70440aa9:::benchmark_data_smithi171_78918_object155556:head# 2024-04-20T17:11:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7046b9d3:::benchmark_data_smithi171_78918_object96248:head# 2024-04-20T17:11:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7046c15b:::benchmark_data_smithi171_78918_object99986:head# 2024-04-20T17:11:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:704756d5:::benchmark_data_smithi171_78918_object68986:head# 2024-04-20T17:11:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7047e5fd:::benchmark_data_smithi171_78918_object84447:head# 2024-04-20T17:11:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:704cbe9d:::benchmark_data_smithi171_78918_object105597:head# 2024-04-20T17:11:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:704f9344:::benchmark_data_smithi171_78918_object124264:head# 2024-04-20T17:11:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7050acee:::benchmark_data_smithi171_78918_object124986:head# 2024-04-20T17:11:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7050cdc2:::benchmark_data_smithi171_78918_object147847:head# 2024-04-20T17:11:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7050dd39:::benchmark_data_smithi171_78918_object132730:head# 2024-04-20T17:11:07.879 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7051283f:::benchmark_data_smithi171_78918_object101858:head# 2024-04-20T17:11:07.880 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7051ffc6:::benchmark_data_smithi171_78918_object94343:head# 2024-04-20T17:11:07.880 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70526b62:::benchmark_data_smithi171_78918_object108263:head# 2024-04-20T17:11:07.880 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70531a24:::benchmark_data_smithi171_78918_object154268:head# 2024-04-20T17:11:07.880 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7053937f:::benchmark_data_smithi171_78918_object147473:head# 2024-04-20T17:11:07.880 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:705509cf:::benchmark_data_smithi171_78918_object111543:head# 2024-04-20T17:11:07.880 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70558802:::benchmark_data_smithi171_78918_object108159:head# 2024-04-20T17:11:07.880 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70559ee1:::benchmark_data_smithi171_78918_object102846:head# 2024-04-20T17:11:07.880 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7055aa97:::benchmark_data_smithi171_78918_object129405:head# 2024-04-20T17:11:07.880 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70569c37:::benchmark_data_smithi171_78918_object151344:head# 2024-04-20T17:11:07.880 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7056ef54:::benchmark_data_smithi171_78918_object148985:head# 2024-04-20T17:11:07.880 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7057492c:::benchmark_data_smithi171_78918_object123110:head# 2024-04-20T17:11:07.880 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7057811d:::benchmark_data_smithi171_78918_object155161:head# 2024-04-20T17:11:07.880 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:705814ad:::benchmark_data_smithi171_78918_object64080:head# 2024-04-20T17:11:07.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:07.882 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:11:07.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:11:07.890 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70585c56:::benchmark_data_smithi171_78918_object68881:head# 2024-04-20T17:11:07.890 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70596494:::benchmark_data_smithi171_78918_object119192:head# 2024-04-20T17:11:07.890 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7059bf82:::benchmark_data_smithi171_78918_object128922:head# 2024-04-20T17:11:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:705a464a:::benchmark_data_smithi171_78918_object98066:head# 2024-04-20T17:11:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:705a72e7:::benchmark_data_smithi171_78918_object82792:head# 2024-04-20T17:11:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:705a9d0e:::benchmark_data_smithi171_78918_object118935:head# 2024-04-20T17:11:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:705a9f04:::benchmark_data_smithi171_78918_object54605:head# 2024-04-20T17:11:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:705b657c:::benchmark_data_smithi171_78918_object138261:head# 2024-04-20T17:11:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:705cdc1c:::benchmark_data_smithi171_78918_object132927:head# 2024-04-20T17:11:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:705e09d1:::benchmark_data_smithi171_78918_object130376:head# 2024-04-20T17:11:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:705fd2d8:::benchmark_data_smithi171_78918_object117723:head# 2024-04-20T17:11:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:705fe038:::benchmark_data_smithi171_78918_object121076:head# 2024-04-20T17:11:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7061615f:::benchmark_data_smithi171_78918_object137762:head# 2024-04-20T17:11:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7061660b:::benchmark_data_smithi171_78918_object157687:head# 2024-04-20T17:11:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70616b1a:::benchmark_data_smithi171_78918_object153364:head# 2024-04-20T17:11:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:706220a8:::benchmark_data_smithi171_78918_object53127:head# 2024-04-20T17:11:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70623a10:::benchmark_data_smithi171_78918_object61305:head# 2024-04-20T17:11:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70623f96:::benchmark_data_smithi171_78918_object126319:head# 2024-04-20T17:11:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70626ff7:::benchmark_data_smithi171_78918_object156458:head# 2024-04-20T17:11:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7063c342:::benchmark_data_smithi171_78918_object96455:head# 2024-04-20T17:11:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7063d358:::benchmark_data_smithi171_78918_object121530:head# 2024-04-20T17:11:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70648fa7:::benchmark_data_smithi171_78918_object159856:head# 2024-04-20T17:11:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7066712e:::benchmark_data_smithi171_78918_object71120:head# 2024-04-20T17:11:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7066cc72:::benchmark_data_smithi171_78918_object81973:head# 2024-04-20T17:11:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7067c860:::benchmark_data_smithi171_78918_object152687:head# 2024-04-20T17:11:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70683a4a:::benchmark_data_smithi171_78918_object78103:head# 2024-04-20T17:11:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:706937d6:::benchmark_data_smithi171_78918_object81341:head# 2024-04-20T17:11:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:706aa7dc:::benchmark_data_smithi171_78918_object99897:head# 2024-04-20T17:11:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:706baa39:::benchmark_data_smithi171_78918_object137567:head# 2024-04-20T17:11:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:706c1992:::benchmark_data_smithi171_78918_object63963:head# 2024-04-20T17:11:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:706ca2d2:::benchmark_data_smithi171_78918_object144139:head# 2024-04-20T17:11:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:706cb15e:::benchmark_data_smithi171_78918_object133012:head# 2024-04-20T17:11:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:706d32cd:::benchmark_data_smithi171_78918_object72573:head# 2024-04-20T17:11:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:706d6142:::benchmark_data_smithi171_78918_object67446:head# 2024-04-20T17:11:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:706ea5bd:::benchmark_data_smithi171_78918_object74644:head# 2024-04-20T17:11:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:706f621a:::benchmark_data_smithi171_78918_object67924:head# 2024-04-20T17:11:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7073c2ea:::benchmark_data_smithi171_78918_object139307:head# 2024-04-20T17:11:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7076501e:::benchmark_data_smithi171_78918_object69258:head# 2024-04-20T17:11:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7076570f:::benchmark_data_smithi171_78918_object150473:head# 2024-04-20T17:11:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:707696d5:::benchmark_data_smithi171_78918_object156398:head# 2024-04-20T17:11:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7076a585:::benchmark_data_smithi171_78918_object98246:head# 2024-04-20T17:11:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7076bead:::benchmark_data_smithi171_78918_object105986:head# 2024-04-20T17:11:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7076d976:::benchmark_data_smithi171_78918_object125313:head# 2024-04-20T17:11:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7077212a:::benchmark_data_smithi171_78918_object100021:head# 2024-04-20T17:11:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70775258:::benchmark_data_smithi171_78918_object55095:head# 2024-04-20T17:11:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7077d355:::benchmark_data_smithi171_78918_object144963:head# 2024-04-20T17:11:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70782680:::benchmark_data_smithi171_78918_object69736:head# 2024-04-20T17:11:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7078c52c:::benchmark_data_smithi171_78918_object141921:head# 2024-04-20T17:11:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7078cf61:::benchmark_data_smithi171_78918_object134862:head# 2024-04-20T17:11:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70791903:::benchmark_data_smithi171_78918_object133936:head# 2024-04-20T17:11:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:707939d1:::benchmark_data_smithi171_78918_object146777:head# 2024-04-20T17:11:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7079dd49:::benchmark_data_smithi171_78918_object118111:head# 2024-04-20T17:11:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7079e0b1:::benchmark_data_smithi171_78918_object108246:head# 2024-04-20T17:11:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:707b8823:::benchmark_data_smithi171_78918_object131510:head# 2024-04-20T17:11:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:707bed49:::benchmark_data_smithi171_78918_object120978:head# 2024-04-20T17:11:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:707bfae2:::benchmark_data_smithi171_78918_object158094:head# 2024-04-20T17:11:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:707c0a3a:::benchmark_data_smithi171_78918_object108021:head# 2024-04-20T17:11:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:707ca639:::benchmark_data_smithi171_78918_object55377:head# 2024-04-20T17:11:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:707cd2c7:::benchmark_data_smithi171_78918_object158689:head# 2024-04-20T17:11:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:707d8e9b:::benchmark_data_smithi171_78918_object160455:head# 2024-04-20T17:11:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:707db2ea:::benchmark_data_smithi171_78918_object141370:head# 2024-04-20T17:11:07.895 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:707f100e:::benchmark_data_smithi171_78918_object129664:head# 2024-04-20T17:11:07.895 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:707f1496:::benchmark_data_smithi171_78918_object95821:head# 2024-04-20T17:11:07.895 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:707faf55:::benchmark_data_smithi171_78918_object102092:head# 2024-04-20T17:11:07.895 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:708046a2:::benchmark_data_smithi171_78918_object112667:head# 2024-04-20T17:11:07.895 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7080cfc8:::benchmark_data_smithi171_78918_object123677:head# 2024-04-20T17:11:07.895 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70813187:::benchmark_data_smithi171_78918_object71686:head# 2024-04-20T17:11:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70819812:::benchmark_data_smithi171_78918_object111607:head# 2024-04-20T17:11:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7082f78d:::benchmark_data_smithi171_78918_object158265:head# 2024-04-20T17:11:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:708305a6:::benchmark_data_smithi171_78918_object139737:head# 2024-04-20T17:11:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70834dc5:::benchmark_data_smithi171_78918_object86994:head# 2024-04-20T17:11:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7083f38e:::benchmark_data_smithi171_78918_object116103:head# 2024-04-20T17:11:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:708468c5:::benchmark_data_smithi171_78918_object125352:head# 2024-04-20T17:11:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70861021:::benchmark_data_smithi171_78918_object156037:head# 2024-04-20T17:11:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7086ce95:::benchmark_data_smithi171_78918_object154375:head# 2024-04-20T17:11:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7086ee52:::benchmark_data_smithi171_78918_object131588:head# 2024-04-20T17:11:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7087fdf6:::benchmark_data_smithi171_78918_object63856:head# 2024-04-20T17:11:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7088d093:::benchmark_data_smithi171_78918_object149072:head# 2024-04-20T17:11:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7089a6e5:::benchmark_data_smithi171_78918_object63071:head# 2024-04-20T17:11:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7089fbb1:::benchmark_data_smithi171_78918_object78935:head# 2024-04-20T17:11:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:708b00ef:::benchmark_data_smithi171_78918_object75378:head# 2024-04-20T17:11:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:708d6c6a:::benchmark_data_smithi171_78918_object117001:head# 2024-04-20T17:11:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:708e093c:::benchmark_data_smithi171_78918_object105406:head# 2024-04-20T17:11:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:708e999b:::benchmark_data_smithi171_78918_object145600:head# 2024-04-20T17:11:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:708eefc4:::benchmark_data_smithi171_78918_object65873:head# 2024-04-20T17:11:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:708f26f8:::benchmark_data_smithi171_78918_object100347:head# 2024-04-20T17:11:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:708f2d76:::benchmark_data_smithi171_78918_object148963:head# 2024-04-20T17:11:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70902f22:::benchmark_data_smithi171_78918_object153239:head# 2024-04-20T17:11:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7092abd8:::benchmark_data_smithi171_78918_object52334:head# 2024-04-20T17:11:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70934c7e:::benchmark_data_smithi171_78918_object153259:head# 2024-04-20T17:11:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70934ec7:::benchmark_data_smithi171_78918_object124915:head# 2024-04-20T17:11:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7093dfba:::benchmark_data_smithi171_78918_object152497:head# 2024-04-20T17:11:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:709576e0:::benchmark_data_smithi171_78918_object104503:head# 2024-04-20T17:11:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7096c4c5:::benchmark_data_smithi171_78918_object151652:head# 2024-04-20T17:11:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7096ed96:::benchmark_data_smithi171_78918_object119004:head# 2024-04-20T17:11:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:709707c6:::benchmark_data_smithi171_78918_object158075:head# 2024-04-20T17:11:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70984489:::benchmark_data_smithi171_78918_object145198:head# 2024-04-20T17:11:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70990f80:::benchmark_data_smithi171_78918_object130967:head# 2024-04-20T17:11:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70997065:::benchmark_data_smithi171_78918_object148213:head# 2024-04-20T17:11:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7099c197:::benchmark_data_smithi171_78918_object153587:head# 2024-04-20T17:11:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:709a4c39:::benchmark_data_smithi171_78918_object58693:head# 2024-04-20T17:11:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:709a7d7b:::benchmark_data_smithi171_78918_object138989:head# 2024-04-20T17:11:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:709ade1d:::benchmark_data_smithi171_78918_object71203:head# 2024-04-20T17:11:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:709b80af:::benchmark_data_smithi171_78918_object118770:head# 2024-04-20T17:11:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:709ba2e1:::benchmark_data_smithi171_78918_object118904:head# 2024-04-20T17:11:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:709bffdf:::benchmark_data_smithi171_78918_object126046:head# 2024-04-20T17:11:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:709cb0be:::benchmark_data_smithi171_78918_object159062:head# 2024-04-20T17:11:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:709cd6e3:::benchmark_data_smithi171_78918_object157616:head# 2024-04-20T17:11:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:709d37c0:::benchmark_data_smithi171_78918_object69263:head# 2024-04-20T17:11:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:709d380e:::benchmark_data_smithi171_78918_object68300:head# 2024-04-20T17:11:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:709d462d:::benchmark_data_smithi171_78918_object128562:head# 2024-04-20T17:11:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:709dd84d:::benchmark_data_smithi171_78918_object64746:head# 2024-04-20T17:11:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:709fb3d1:::benchmark_data_smithi171_78918_object153648:head# 2024-04-20T17:11:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:709fdc83:::benchmark_data_smithi171_78918_object148759:head# 2024-04-20T17:11:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70a04232:::benchmark_data_smithi171_78918_object156069:head# 2024-04-20T17:11:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70a0d721:::benchmark_data_smithi171_78918_object121985:head# 2024-04-20T17:11:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70a13ae7:::benchmark_data_smithi171_78918_object156266:head# 2024-04-20T17:11:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70a26038:::benchmark_data_smithi171_78918_object150105:head# 2024-04-20T17:11:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70a31f40:::benchmark_data_smithi171_78918_object145825:head# 2024-04-20T17:11:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70a357cd:::benchmark_data_smithi171_78918_object135656:head# 2024-04-20T17:11:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70a3bf31:::benchmark_data_smithi171_78918_object105714:head# 2024-04-20T17:11:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70a46107:::benchmark_data_smithi171_78918_object92098:head# 2024-04-20T17:11:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70a47023:::benchmark_data_smithi171_78918_object125232:head# 2024-04-20T17:11:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70a520c4:::benchmark_data_smithi171_78918_object157991:head# 2024-04-20T17:11:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70a54a2a:::benchmark_data_smithi171_78918_object111053:head# 2024-04-20T17:11:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70a5ee0d:::benchmark_data_smithi171_78918_object132978:head# 2024-04-20T17:11:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70a6b4d4:::benchmark_data_smithi171_78918_object63689:head# 2024-04-20T17:11:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70a734ce:::benchmark_data_smithi171_78918_object136964:head# 2024-04-20T17:11:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70a77200:::benchmark_data_smithi171_78918_object60089:head# 2024-04-20T17:11:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70a92ff6:::benchmark_data_smithi171_78918_object94304:head# 2024-04-20T17:11:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70a93333:::benchmark_data_smithi171_78918_object95001:head# 2024-04-20T17:11:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70a9a8c9:::benchmark_data_smithi171_78918_object144609:head# 2024-04-20T17:11:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70aae84d:::benchmark_data_smithi171_78918_object63069:head# 2024-04-20T17:11:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70aba567:::benchmark_data_smithi171_78918_object87452:head# 2024-04-20T17:11:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70abb8d4:::benchmark_data_smithi171_78918_object78649:head# 2024-04-20T17:11:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70acd477:::benchmark_data_smithi171_78918_object60816:head# 2024-04-20T17:11:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70adb2ef:::benchmark_data_smithi171_78918_object134064:head# 2024-04-20T17:11:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70adf96e:::benchmark_data_smithi171_78918_object60780:head# 2024-04-20T17:11:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70ae6587:::benchmark_data_smithi171_78918_object92310:head# 2024-04-20T17:11:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70afc687:::benchmark_data_smithi171_78918_object58129:head# 2024-04-20T17:11:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70b012be:::benchmark_data_smithi171_78918_object54532:head# 2024-04-20T17:11:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70b02fa0:::benchmark_data_smithi171_78918_object139104:head# 2024-04-20T17:11:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70b11be4:::benchmark_data_smithi171_78918_object104332:head# 2024-04-20T17:11:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70b1cc26:::benchmark_data_smithi171_78918_object71427:head# 2024-04-20T17:11:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70b2eb7c:::benchmark_data_smithi171_78918_object77163:head# 2024-04-20T17:11:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70b423ab:::benchmark_data_smithi171_78918_object102872:head# 2024-04-20T17:11:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70b43539:::benchmark_data_smithi171_78918_object85424:head# 2024-04-20T17:11:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70b58a08:::benchmark_data_smithi171_78918_object63959:head# 2024-04-20T17:11:07.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70b63d0c:::benchmark_data_smithi171_78918_object142254:head# 2024-04-20T17:11:07.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70b669c1:::benchmark_data_smithi171_78918_object60798:head# 2024-04-20T17:11:07.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70b68142:::benchmark_data_smithi171_78918_object149088:head# 2024-04-20T17:11:07.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70b6cab8:::benchmark_data_smithi171_78918_object129829:head# 2024-04-20T17:11:07.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70b6f371:::benchmark_data_smithi171_78918_object79721:head# 2024-04-20T17:11:07.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70b7136b:::benchmark_data_smithi171_78918_object132178:head# 2024-04-20T17:11:07.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70b78def:::benchmark_data_smithi171_78918_object159704:head# 2024-04-20T17:11:07.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70b85f3e:::benchmark_data_smithi171_78918_object51954:head# 2024-04-20T17:11:07.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70b88757:::benchmark_data_smithi171_78918_object76844:head# 2024-04-20T17:11:07.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70b8c4ce:::benchmark_data_smithi171_78918_object53496:head# 2024-04-20T17:11:07.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70b8d21c:::benchmark_data_smithi171_78918_object106148:head# 2024-04-20T17:11:07.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70b8ff0b:::benchmark_data_smithi171_78918_object96071:head# 2024-04-20T17:11:07.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70b9a6ac:::benchmark_data_smithi171_78918_object113975:head# 2024-04-20T17:11:07.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70bb3363:::benchmark_data_smithi171_78918_object137237:head# 2024-04-20T17:11:07.915 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70bc5d8e:::benchmark_data_smithi171_78918_object142905:head# 2024-04-20T17:11:07.915 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70bc661a:::benchmark_data_smithi171_78918_object87995:head# 2024-04-20T17:11:07.923 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70bdabbd:::benchmark_data_smithi171_78918_object155961:head# 2024-04-20T17:11:07.923 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70bdb2f1:::benchmark_data_smithi171_78918_object86121:head# 2024-04-20T17:11:07.924 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70be23c9:::benchmark_data_smithi171_78918_object140513:head# 2024-04-20T17:11:07.924 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70be549f:::benchmark_data_smithi171_78918_object93974:head# 2024-04-20T17:11:07.924 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70beb3ef:::benchmark_data_smithi171_78918_object116457:head# 2024-04-20T17:11:07.924 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70bf69c9:::benchmark_data_smithi171_78918_object75870:head# 2024-04-20T17:11:07.924 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70c0f77a:::benchmark_data_smithi171_78918_object138179:head# 2024-04-20T17:11:07.924 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70c11f8f:::benchmark_data_smithi171_78918_object77818:head# 2024-04-20T17:11:07.924 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70c1be98:::benchmark_data_smithi171_78918_object104278:head# 2024-04-20T17:11:07.924 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70c21f8b:::benchmark_data_smithi171_78918_object82982:head# 2024-04-20T17:11:07.924 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70c26e83:::benchmark_data_smithi171_78918_object142493:head# 2024-04-20T17:11:07.924 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70c29f13:::benchmark_data_smithi171_78918_object125574:head# 2024-04-20T17:11:07.924 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70c42c0e:::benchmark_data_smithi171_78918_object114469:head# 2024-04-20T17:11:07.924 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70c47f9c:::benchmark_data_smithi171_78918_object146604:head# 2024-04-20T17:11:07.924 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70c48c50:::benchmark_data_smithi171_78918_object155753:head# 2024-04-20T17:11:07.924 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70c645a8:::benchmark_data_smithi171_78918_object142713:head# 2024-04-20T17:11:07.925 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70c6750f:::benchmark_data_smithi171_78918_object83520:head# 2024-04-20T17:11:07.925 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70c71abe:::benchmark_data_smithi171_78918_object117895:head# 2024-04-20T17:11:07.925 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70c7ab08:::benchmark_data_smithi171_78918_object69245:head# 2024-04-20T17:11:07.925 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70c884ee:::benchmark_data_smithi171_78918_object146490:head# 2024-04-20T17:11:07.925 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70c98e42:::benchmark_data_smithi171_78918_object100037:head# 2024-04-20T17:11:07.925 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70c99df1:::benchmark_data_smithi171_78918_object96272:head# 2024-04-20T17:11:07.925 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70c9ce48:::benchmark_data_smithi171_78918_object104172:head# 2024-04-20T17:11:07.925 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70ca4933:::benchmark_data_smithi171_78918_object85898:head# 2024-04-20T17:11:07.925 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70cb4067:::benchmark_data_smithi171_78918_object64269:head# 2024-04-20T17:11:07.925 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70cee318:::benchmark_data_smithi171_78918_object92624:head# 2024-04-20T17:11:07.925 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70cef44a:::benchmark_data_smithi171_78918_object148216:head# 2024-04-20T17:11:07.925 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70cfa1cc:::benchmark_data_smithi171_78918_object72721:head# 2024-04-20T17:11:07.925 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70d02ec6:::benchmark_data_smithi171_78918_object69006:head# 2024-04-20T17:11:07.925 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70d2765b:::benchmark_data_smithi171_78918_object145100:head# 2024-04-20T17:11:07.925 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70d39dab:::benchmark_data_smithi171_78918_object110149:head# 2024-04-20T17:11:07.926 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70d3a9be:::benchmark_data_smithi171_78918_object92222:head# 2024-04-20T17:11:07.926 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70d65e07:::benchmark_data_smithi171_78918_object85082:head# 2024-04-20T17:11:07.926 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70d68334:::benchmark_data_smithi171_78918_object131521:head# 2024-04-20T17:11:07.926 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70d68a60:::benchmark_data_smithi171_78918_object148833:head# 2024-04-20T17:11:07.926 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70d73763:::benchmark_data_smithi171_78918_object152339:head# 2024-04-20T17:11:07.926 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70d7a0e6:::benchmark_data_smithi171_78918_object108686:head# 2024-04-20T17:11:07.926 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70d7c7f0:::benchmark_data_smithi171_78918_object145740:head# 2024-04-20T17:11:07.926 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70d831e0:::benchmark_data_smithi171_78918_object54664:head# 2024-04-20T17:11:07.926 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70d9c42d:::benchmark_data_smithi171_78918_object123331:head# 2024-04-20T17:11:07.926 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70dc0e8d:::benchmark_data_smithi171_78918_object134593:head# 2024-04-20T17:11:07.926 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70dc9117:::benchmark_data_smithi171_78918_object72992:head# 2024-04-20T17:11:07.926 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70de6371:::benchmark_data_smithi171_78918_object126374:head# 2024-04-20T17:11:07.926 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70df1017:::benchmark_data_smithi171_78918_object75247:head# 2024-04-20T17:11:07.926 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70e34ff0:::benchmark_data_smithi171_78918_object69371:head# 2024-04-20T17:11:07.927 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70e367d5:::benchmark_data_smithi171_78918_object138457:head# 2024-04-20T17:11:07.927 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70e552cd:::benchmark_data_smithi171_78918_object63644:head# 2024-04-20T17:11:07.927 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70e639b1:::benchmark_data_smithi171_78918_object74520:head# 2024-04-20T17:11:07.927 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70e76dbe:::benchmark_data_smithi171_78918_object147120:head# 2024-04-20T17:11:07.927 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70e868ab:::benchmark_data_smithi171_78918_object114400:head# 2024-04-20T17:11:07.927 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70e8d486:::benchmark_data_smithi171_78918_object55798:head# 2024-04-20T17:11:07.927 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70e90bd8:::benchmark_data_smithi171_78918_object155310:head# 2024-04-20T17:11:07.927 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70e9692b:::benchmark_data_smithi171_78918_object77645:head# 2024-04-20T17:11:07.927 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70ec4b00:::benchmark_data_smithi171_78918_object113326:head# 2024-04-20T17:11:07.927 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70ec5abb:::benchmark_data_smithi171_78918_object138469:head# 2024-04-20T17:11:07.927 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70ed24b6:::benchmark_data_smithi171_78918_object111802:head# 2024-04-20T17:11:07.927 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70ed9519:::benchmark_data_smithi171_78918_object75204:head# 2024-04-20T17:11:07.927 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70ede9fd:::benchmark_data_smithi171_78918_object156435:head# 2024-04-20T17:11:07.927 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70ee0922:::benchmark_data_smithi171_78918_object158740:head# 2024-04-20T17:11:07.927 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70ee1d0d:::benchmark_data_smithi171_78918_object64640:head# 2024-04-20T17:11:07.928 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70ee5308:::benchmark_data_smithi171_78918_object98114:head# 2024-04-20T17:11:07.928 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70eeb801:::benchmark_data_smithi171_78918_object125604:head# 2024-04-20T17:11:07.928 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70eec461:::benchmark_data_smithi171_78918_object72019:head# 2024-04-20T17:11:07.928 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70eee9dd:::benchmark_data_smithi171_78918_object156659:head# 2024-04-20T17:11:07.928 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70ef13e5:::benchmark_data_smithi171_78918_object75825:head# 2024-04-20T17:11:07.928 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70ef3418:::benchmark_data_smithi171_78918_object113011:head# 2024-04-20T17:11:07.928 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70ef8a7c:::benchmark_data_smithi171_78918_object67130:head# 2024-04-20T17:11:07.928 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70efc44a:::benchmark_data_smithi171_78918_object85162:head# 2024-04-20T17:11:07.928 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70efc988:::benchmark_data_smithi171_78918_object151330:head# 2024-04-20T17:11:07.928 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70f00c7d:::benchmark_data_smithi171_78918_object79834:head# 2024-04-20T17:11:07.928 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70f0dbcc:::benchmark_data_smithi171_78918_object101133:head# 2024-04-20T17:11:07.928 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70f19dd3:::benchmark_data_smithi171_78918_object87272:head# 2024-04-20T17:11:07.928 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70f3416b:::benchmark_data_smithi171_78918_object133561:head# 2024-04-20T17:11:07.928 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70f36001:::benchmark_data_smithi171_78918_object148964:head# 2024-04-20T17:11:07.929 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70f3fd17:::benchmark_data_smithi171_78918_object91331:head# 2024-04-20T17:11:07.929 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70f43a86:::benchmark_data_smithi171_78918_object141262:head# 2024-04-20T17:11:07.929 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70f43e36:::benchmark_data_smithi171_78918_object132081:head# 2024-04-20T17:11:07.929 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70f46aa2:::benchmark_data_smithi171_78918_object144360:head# 2024-04-20T17:11:07.929 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70f538ab:::benchmark_data_smithi171_78918_object77978:head# 2024-04-20T17:11:07.929 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70f57190:::benchmark_data_smithi171_78918_object83369:head# 2024-04-20T17:11:07.929 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70f63d77:::benchmark_data_smithi171_78918_object135989:head# 2024-04-20T17:11:07.929 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70f6479b:::benchmark_data_smithi171_78918_object70449:head# 2024-04-20T17:11:07.929 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70f68952:::benchmark_data_smithi171_78918_object105671:head# 2024-04-20T17:11:07.929 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70f700ba:::benchmark_data_smithi171_78918_object74125:head# 2024-04-20T17:11:07.929 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70f9ed4b:::benchmark_data_smithi171_78918_object125726:head# 2024-04-20T17:11:07.929 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70f9ee83:::benchmark_data_smithi171_78918_object93295:head# 2024-04-20T17:11:07.937 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70fac79f:::benchmark_data_smithi171_78918_object110255:head# 2024-04-20T17:11:07.937 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70fb42a9:::benchmark_data_smithi171_78918_object123716:head# 2024-04-20T17:11:07.938 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70fc9d70:::benchmark_data_smithi171_78918_object108420:head# 2024-04-20T17:11:07.938 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70fdbee9:::benchmark_data_smithi171_78918_object123230:head# 2024-04-20T17:11:07.938 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70fe5812:::benchmark_data_smithi171_78918_object69141:head# 2024-04-20T17:11:07.938 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:70fff3f9:::benchmark_data_smithi171_78918_object81557:head# 2024-04-20T17:11:07.938 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71002455:::benchmark_data_smithi171_78918_object79442:head# 2024-04-20T17:11:07.938 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:710082ef:::benchmark_data_smithi171_78918_object137694:head# 2024-04-20T17:11:07.938 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7100e7c1:::benchmark_data_smithi171_78918_object65955:head# 2024-04-20T17:11:07.938 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71012fed:::benchmark_data_smithi171_78918_object147882:head# 2024-04-20T17:11:07.938 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71020993:::benchmark_data_smithi171_78918_object145248:head# 2024-04-20T17:11:07.938 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71036f13:::benchmark_data_smithi171_78918_object54240:head# 2024-04-20T17:11:07.938 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71039c7c:::benchmark_data_smithi171_78918_object160080:head# 2024-04-20T17:11:07.938 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7103daf6:::benchmark_data_smithi171_78918_object54651:head# 2024-04-20T17:11:07.938 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7103e5dc:::benchmark_data_smithi171_78918_object139537:head# 2024-04-20T17:11:07.938 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7103fb4d:::benchmark_data_smithi171_78918_object60663:head# 2024-04-20T17:11:07.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71059bb7:::benchmark_data_smithi171_78918_object93791:head# 2024-04-20T17:11:07.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7105c147:::benchmark_data_smithi171_78918_object78944:head# 2024-04-20T17:11:07.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:710713c3:::benchmark_data_smithi171_78918_object76020:head# 2024-04-20T17:11:07.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71078d31:::benchmark_data_smithi171_78918_object123113:head# 2024-04-20T17:11:07.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7107eee8:::benchmark_data_smithi171_78918_object142563:head# 2024-04-20T17:11:07.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71093187:::benchmark_data_smithi171_78918_object137986:head# 2024-04-20T17:11:07.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7109e6ba:::benchmark_data_smithi171_78918_object63155:head# 2024-04-20T17:11:07.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:710b8e3f:::benchmark_data_smithi171_78918_object157756:head# 2024-04-20T17:11:07.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:710c716b:::benchmark_data_smithi171_78918_object76866:head# 2024-04-20T17:11:07.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:710d1318:::benchmark_data_smithi171_78918_object88081:head# 2024-04-20T17:11:07.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:710daa6c:::benchmark_data_smithi171_78918_object160739:head# 2024-04-20T17:11:07.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:710e2010:::benchmark_data_smithi171_78918_object97039:head# 2024-04-20T17:11:07.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:710f92bb:::benchmark_data_smithi171_78918_object84056:head# 2024-04-20T17:11:07.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:711156fc:::benchmark_data_smithi171_78918_object52169:head# 2024-04-20T17:11:07.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71118a44:::benchmark_data_smithi171_78918_object129173:head# 2024-04-20T17:11:07.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71123ff6:::benchmark_data_smithi171_78918_object78578:head# 2024-04-20T17:11:07.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71125a4e:::benchmark_data_smithi171_78918_object101188:head# 2024-04-20T17:11:07.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71129d9c:::benchmark_data_smithi171_78918_object85934:head# 2024-04-20T17:11:07.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71129faf:::benchmark_data_smithi171_78918_object123384:head# 2024-04-20T17:11:07.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7112eff6:::benchmark_data_smithi171_78918_object101908:head# 2024-04-20T17:11:07.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71155bed:::benchmark_data_smithi171_78918_object64964:head# 2024-04-20T17:11:07.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71160949:::benchmark_data_smithi171_78918_object98856:head# 2024-04-20T17:11:07.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71163705:::benchmark_data_smithi171_78918_object102766:head# 2024-04-20T17:11:07.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7117515e:::benchmark_data_smithi171_78918_object96374:head# 2024-04-20T17:11:07.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71177421:::benchmark_data_smithi171_78918_object124974:head# 2024-04-20T17:11:07.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7117f3a7:::benchmark_data_smithi171_78918_object70871:head# 2024-04-20T17:11:07.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71191682:::benchmark_data_smithi171_78918_object147305:head# 2024-04-20T17:11:07.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:711afa96:::benchmark_data_smithi171_78918_object76503:head# 2024-04-20T17:11:07.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:711c5ac5:::benchmark_data_smithi171_78918_object135495:head# 2024-04-20T17:11:07.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:711c70ce:::benchmark_data_smithi171_78918_object95207:head# 2024-04-20T17:11:07.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:711cb416:::benchmark_data_smithi171_78918_object139062:head# 2024-04-20T17:11:07.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:711d0b1a:::benchmark_data_smithi171_78918_object128477:head# 2024-04-20T17:11:07.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:711d32c7:::benchmark_data_smithi171_78918_object52450:head# 2024-04-20T17:11:07.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:711dbf62:::benchmark_data_smithi171_78918_object126000:head# 2024-04-20T17:11:07.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:711dfb3d:::benchmark_data_smithi171_78918_object81069:head# 2024-04-20T17:11:07.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:711e09fa:::benchmark_data_smithi171_78918_object138192:head# 2024-04-20T17:11:07.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:711e96d8:::benchmark_data_smithi171_78918_object54286:head# 2024-04-20T17:11:07.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:711e9f69:::benchmark_data_smithi171_78918_object155503:head# 2024-04-20T17:11:07.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:711edd6e:::benchmark_data_smithi171_78918_object69550:head# 2024-04-20T17:11:07.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:711f6efe:::benchmark_data_smithi171_78918_object89069:head# 2024-04-20T17:11:07.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:711f7c70:::benchmark_data_smithi171_78918_object79241:head# 2024-04-20T17:11:07.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:711fd41d:::benchmark_data_smithi171_78918_object140984:head# 2024-04-20T17:11:07.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7121b998:::benchmark_data_smithi171_78918_object92343:head# 2024-04-20T17:11:07.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7121f7b2:::benchmark_data_smithi171_78918_object55693:head# 2024-04-20T17:11:07.942 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7123bfcc:::benchmark_data_smithi171_78918_object83344:head# 2024-04-20T17:11:07.942 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:712401ed:::benchmark_data_smithi171_78918_object71827:head# 2024-04-20T17:11:07.942 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71240d4a:::benchmark_data_smithi171_78918_object128440:head# 2024-04-20T17:11:07.942 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71249119:::benchmark_data_smithi171_78918_object132810:head# 2024-04-20T17:11:07.942 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7124ce0c:::benchmark_data_smithi171_78918_object96149:head# 2024-04-20T17:11:07.942 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7124dc54:::benchmark_data_smithi171_78918_object66558:head# 2024-04-20T17:11:07.942 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71259ed6:::benchmark_data_smithi171_78918_object63423:head# 2024-04-20T17:11:07.942 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7126b1ba:::benchmark_data_smithi171_78918_object138592:head# 2024-04-20T17:11:07.942 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:712706ba:::benchmark_data_smithi171_78918_object71000:head# 2024-04-20T17:11:07.942 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7127737c:::benchmark_data_smithi171_78918_object81601:head# 2024-04-20T17:11:07.942 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:712820b3:::benchmark_data_smithi171_78918_object90827:head# 2024-04-20T17:11:07.942 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71287615:::benchmark_data_smithi171_78918_object57653:head# 2024-04-20T17:11:07.942 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7128e79a:::benchmark_data_smithi171_78918_object58783:head# 2024-04-20T17:11:07.942 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7129a9fc:::benchmark_data_smithi171_78918_object141033:head# 2024-04-20T17:11:07.948 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:712a153a:::benchmark_data_smithi171_78918_object55780:head# 2024-04-20T17:11:07.949 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:712aa335:::benchmark_data_smithi171_78918_object86322:head# 2024-04-20T17:11:07.949 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:712b9ec3:::benchmark_data_smithi171_78918_object125806:head# 2024-04-20T17:11:07.949 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:712d2f84:::benchmark_data_smithi171_78918_object141076:head# 2024-04-20T17:11:07.949 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:712ed77f:::benchmark_data_smithi171_78918_object100963:head# 2024-04-20T17:11:07.949 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:712f1f2c:::benchmark_data_smithi171_78918_object105552:head# 2024-04-20T17:11:07.949 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:712f8e79:::benchmark_data_smithi171_78918_object133417:head# 2024-04-20T17:11:07.949 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:713068d9:::benchmark_data_smithi171_78918_object73012:head# 2024-04-20T17:11:07.949 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:713095ba:::benchmark_data_smithi171_78918_object122100:head# 2024-04-20T17:11:07.949 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7131ff36:::benchmark_data_smithi171_78918_object151713:head# 2024-04-20T17:11:07.949 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7132abfc:::benchmark_data_smithi171_78918_object59592:head# 2024-04-20T17:11:07.949 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71339a0f:::benchmark_data_smithi171_78918_object89008:head# 2024-04-20T17:11:07.949 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71353291:::benchmark_data_smithi171_78918_object125667:head# 2024-04-20T17:11:07.949 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:713555e7:::benchmark_data_smithi171_78918_object98429:head# 2024-04-20T17:11:07.949 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71358e7c:::benchmark_data_smithi171_78918_object65159:head# 2024-04-20T17:11:07.950 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7135ddf1:::benchmark_data_smithi171_78918_object154573:head# 2024-04-20T17:11:07.950 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71360bba:::benchmark_data_smithi171_78918_object75724:head# 2024-04-20T17:11:07.950 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7136197b:::benchmark_data_smithi171_78918_object90404:head# 2024-04-20T17:11:07.950 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7138b967:::benchmark_data_smithi171_78918_object127969:head# 2024-04-20T17:11:07.950 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71398106:::benchmark_data_smithi171_78918_object102965:head# 2024-04-20T17:11:07.950 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:713a109c:::benchmark_data_smithi171_78918_object63311:head# 2024-04-20T17:11:07.950 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:713b0aab:::benchmark_data_smithi171_78918_object76846:head# 2024-04-20T17:11:07.950 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:713b3569:::benchmark_data_smithi171_78918_object107766:head# 2024-04-20T17:11:07.950 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:713b4d91:::benchmark_data_smithi171_78918_object117245:head# 2024-04-20T17:11:07.950 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:713c1858:::benchmark_data_smithi171_78918_object92975:head# 2024-04-20T17:11:07.950 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:713c56e3:::benchmark_data_smithi171_78918_object63510:head# 2024-04-20T17:11:07.950 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:713ce652:::benchmark_data_smithi171_78918_object60864:head# 2024-04-20T17:11:07.950 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:713eab21:::benchmark_data_smithi171_78918_object101721:head# 2024-04-20T17:11:07.950 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:713f9037:::benchmark_data_smithi171_78918_object140693:head# 2024-04-20T17:11:07.951 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:713fa5f8:::benchmark_data_smithi171_78918_object143197:head# 2024-04-20T17:11:07.951 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:714172a0:::benchmark_data_smithi171_78918_object85005:head# 2024-04-20T17:11:07.951 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:714193fe:::benchmark_data_smithi171_78918_object119559:head# 2024-04-20T17:11:07.951 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7141e5cd:::benchmark_data_smithi171_78918_object73700:head# 2024-04-20T17:11:07.951 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71428de9:::benchmark_data_smithi171_78918_object99086:head# 2024-04-20T17:11:07.951 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7142b2fa:::benchmark_data_smithi171_78918_object90851:head# 2024-04-20T17:11:07.951 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71439674:::benchmark_data_smithi171_78918_object122875:head# 2024-04-20T17:11:07.951 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71451821:::benchmark_data_smithi171_78918_object103241:head# 2024-04-20T17:11:07.951 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71455265:::benchmark_data_smithi171_78918_object155452:head# 2024-04-20T17:11:07.951 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71489085:::benchmark_data_smithi171_78918_object102289:head# 2024-04-20T17:11:07.951 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7148a402:::benchmark_data_smithi171_78918_object91116:head# 2024-04-20T17:11:07.951 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7148d7ed:::benchmark_data_smithi171_78918_object132794:head# 2024-04-20T17:11:07.951 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:714ae465:::benchmark_data_smithi171_78918_object60716:head# 2024-04-20T17:11:07.951 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:714c883c:::benchmark_data_smithi171_78918_object54247:head# 2024-04-20T17:11:07.952 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:714d2710:::benchmark_data_smithi171_78918_object142085:head# 2024-04-20T17:11:07.952 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:714ded77:::benchmark_data_smithi171_78918_object138650:head# 2024-04-20T17:11:07.952 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:714e22d4:::benchmark_data_smithi171_78918_object111731:head# 2024-04-20T17:11:07.952 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:714e4ac1:::benchmark_data_smithi171_78918_object144220:head# 2024-04-20T17:11:07.952 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:714e8a9d:::benchmark_data_smithi171_78918_object140408:head# 2024-04-20T17:11:07.952 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:714ef2eb:::benchmark_data_smithi171_78918_object135496:head# 2024-04-20T17:11:07.952 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:714f4504:::benchmark_data_smithi171_78918_object118075:head# 2024-04-20T17:11:07.952 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:714f94ee:::benchmark_data_smithi171_78918_object140514:head# 2024-04-20T17:11:07.952 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:714fc3a5:::benchmark_data_smithi171_78918_object105909:head# 2024-04-20T17:11:07.952 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:714fe1e7:::benchmark_data_smithi171_78918_object105565:head# 2024-04-20T17:11:07.952 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71500596:::benchmark_data_smithi171_78918_object81683:head# 2024-04-20T17:11:07.952 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:715033c0:::benchmark_data_smithi171_78918_object141568:head# 2024-04-20T17:11:07.952 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:715224f9:::benchmark_data_smithi171_78918_object118331:head# 2024-04-20T17:11:07.952 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7152d99f:::benchmark_data_smithi171_78918_object141393:head# 2024-04-20T17:11:07.953 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:715357bd:::benchmark_data_smithi171_78918_object52451:head# 2024-04-20T17:11:07.953 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71536853:::benchmark_data_smithi171_78918_object78906:head# 2024-04-20T17:11:07.953 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71542352:::benchmark_data_smithi171_78918_object77844:head# 2024-04-20T17:11:07.953 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7154356b:::benchmark_data_smithi171_78918_object80380:head# 2024-04-20T17:11:07.953 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7155a749:::benchmark_data_smithi171_78918_object107960:head# 2024-04-20T17:11:07.953 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7155cd40:::benchmark_data_smithi171_78918_object145240:head# 2024-04-20T17:11:07.962 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7155f901:::benchmark_data_smithi171_78918_object82425:head# 2024-04-20T17:11:07.962 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7156858f:::benchmark_data_smithi171_78918_object103048:head# 2024-04-20T17:11:07.962 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:715702ac:::benchmark_data_smithi171_78918_object130607:head# 2024-04-20T17:11:07.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71571783:::benchmark_data_smithi171_78918_object103690:head# 2024-04-20T17:11:07.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7157c326:::benchmark_data_smithi171_78918_object147489:head# 2024-04-20T17:11:07.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:715895ce:::benchmark_data_smithi171_78918_object84320:head# 2024-04-20T17:11:07.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7158acd3:::benchmark_data_smithi171_78918_object72796:head# 2024-04-20T17:11:07.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7158ff61:::benchmark_data_smithi171_78918_object154861:head# 2024-04-20T17:11:07.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7159587c:::benchmark_data_smithi171_78918_object70548:head# 2024-04-20T17:11:07.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:715a18e3:::benchmark_data_smithi171_78918_object156269:head# 2024-04-20T17:11:07.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:715a3a42:::benchmark_data_smithi171_78918_object152775:head# 2024-04-20T17:11:07.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:715b2e44:::benchmark_data_smithi171_78918_object108684:head# 2024-04-20T17:11:07.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:715bd50c:::benchmark_data_smithi171_78918_object115544:head# 2024-04-20T17:11:07.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:715c5be7:::benchmark_data_smithi171_78918_object98962:head# 2024-04-20T17:11:07.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:715c87ef:::benchmark_data_smithi171_78918_object90765:head# 2024-04-20T17:11:07.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:715cc600:::benchmark_data_smithi171_78918_object122903:head# 2024-04-20T17:11:07.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:715d23d8:::benchmark_data_smithi171_78918_object156358:head# 2024-04-20T17:11:07.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:715d836a:::benchmark_data_smithi171_78918_object59909:head# 2024-04-20T17:11:07.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:715df2d0:::benchmark_data_smithi171_78918_object58277:head# 2024-04-20T17:11:07.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:715e2091:::benchmark_data_smithi171_78918_object61293:head# 2024-04-20T17:11:07.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:715e3cf1:::benchmark_data_smithi171_78918_object122009:head# 2024-04-20T17:11:07.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:715e7b7b:::benchmark_data_smithi171_78918_object62277:head# 2024-04-20T17:11:07.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:715e841e:::benchmark_data_smithi171_78918_object144551:head# 2024-04-20T17:11:07.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:715ec029:::benchmark_data_smithi171_78918_object131499:head# 2024-04-20T17:11:07.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:715f662e:::benchmark_data_smithi171_78918_object154087:head# 2024-04-20T17:11:07.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:715f9311:::benchmark_data_smithi171_78918_object117244:head# 2024-04-20T17:11:07.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:716160cd:::benchmark_data_smithi171_78918_object115362:head# 2024-04-20T17:11:07.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71616e16:::benchmark_data_smithi171_78918_object76482:head# 2024-04-20T17:11:07.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71618f6a:::benchmark_data_smithi171_78918_object52718:head# 2024-04-20T17:11:07.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:716324ee:::benchmark_data_smithi171_78918_object68826:head# 2024-04-20T17:11:07.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7163550b:::benchmark_data_smithi171_78918_object118982:head# 2024-04-20T17:11:07.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7163d6ba:::benchmark_data_smithi171_78918_object102546:head# 2024-04-20T17:11:07.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:716405f9:::benchmark_data_smithi171_78918_object115765:head# 2024-04-20T17:11:07.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7164cb11:::benchmark_data_smithi171_78918_object99873:head# 2024-04-20T17:11:07.965 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71656f89:::benchmark_data_smithi171_78918_object154719:head# 2024-04-20T17:11:07.965 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71661ed6:::benchmark_data_smithi171_78918_object101386:head# 2024-04-20T17:11:07.965 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:716788e7:::benchmark_data_smithi171_78918_object160322:head# 2024-04-20T17:11:07.965 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7169d397:::benchmark_data_smithi171_78918_object78758:head# 2024-04-20T17:11:07.965 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:716a5e05:::benchmark_data_smithi171_78918_object130635:head# 2024-04-20T17:11:07.965 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:716aa09e:::benchmark_data_smithi171_78918_object121038:head# 2024-04-20T17:11:07.965 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:716b9428:::benchmark_data_smithi171_78918_object76046:head# 2024-04-20T17:11:07.965 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:716bec3d:::benchmark_data_smithi171_78918_object100036:head# 2024-04-20T17:11:07.965 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:716c00d4:::benchmark_data_smithi171_78918_object100275:head# 2024-04-20T17:11:07.965 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:716c54d5:::benchmark_data_smithi171_78918_object153659:head# 2024-04-20T17:11:07.965 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:716c60a2:::benchmark_data_smithi171_78918_object133834:head# 2024-04-20T17:11:07.965 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:716e4475:::benchmark_data_smithi171_78918_object117360:head# 2024-04-20T17:11:07.965 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:716eb862:::benchmark_data_smithi171_78918_object66753:head# 2024-04-20T17:11:07.965 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:716ebe7e:::benchmark_data_smithi171_78918_object98162:head# 2024-04-20T17:11:07.966 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71716738:::benchmark_data_smithi171_78918_object111774:head# 2024-04-20T17:11:07.966 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7172b0fb:::benchmark_data_smithi171_78918_object69852:head# 2024-04-20T17:11:07.966 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7172ea3f:::benchmark_data_smithi171_78918_object113910:head# 2024-04-20T17:11:07.966 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7173aea7:::benchmark_data_smithi171_78918_object63404:head# 2024-04-20T17:11:07.966 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7173c779:::benchmark_data_smithi171_78918_object72168:head# 2024-04-20T17:11:07.966 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71745dbe:::benchmark_data_smithi171_78918_object66508:head# 2024-04-20T17:11:07.966 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7175a236:::benchmark_data_smithi171_78918_object131370:head# 2024-04-20T17:11:07.966 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:717687b7:::benchmark_data_smithi171_78918_object96737:head# 2024-04-20T17:11:07.966 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7176f2f4:::benchmark_data_smithi171_78918_object107701:head# 2024-04-20T17:11:07.966 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7178807c:::benchmark_data_smithi171_78918_object111464:head# 2024-04-20T17:11:07.966 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:717a010b:::benchmark_data_smithi171_78918_object102990:head# 2024-04-20T17:11:07.966 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:717a9565:::benchmark_data_smithi171_78918_object93723:head# 2024-04-20T17:11:07.966 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:717aa73e:::benchmark_data_smithi171_78918_object84896:head# 2024-04-20T17:11:07.966 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:717bcf76:::benchmark_data_smithi171_78918_object61930:head# 2024-04-20T17:11:07.966 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:717c733c:::benchmark_data_smithi171_78918_object61008:head# 2024-04-20T17:11:07.966 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:717c7c20:::benchmark_data_smithi171_78918_object145929:head# 2024-04-20T17:11:07.967 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:717d2d76:::benchmark_data_smithi171_78918_object131532:head# 2024-04-20T17:11:07.967 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:717db82c:::benchmark_data_smithi171_78918_object153085:head# 2024-04-20T17:11:07.967 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:717f0ff6:::benchmark_data_smithi171_78918_object60240:head# 2024-04-20T17:11:07.967 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71801557:::benchmark_data_smithi171_78918_object147728:head# 2024-04-20T17:11:07.967 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7180e963:::benchmark_data_smithi171_78918_object91442:head# 2024-04-20T17:11:07.967 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7180f8af:::benchmark_data_smithi171_78918_object154698:head# 2024-04-20T17:11:07.967 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71822b1e:::benchmark_data_smithi171_78918_object67282:head# 2024-04-20T17:11:07.967 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71822d00:::benchmark_data_smithi171_78918_object134524:head# 2024-04-20T17:11:07.967 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:718241f6:::benchmark_data_smithi171_78918_object126942:head# 2024-04-20T17:11:07.967 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71832571:::benchmark_data_smithi171_78918_object118136:head# 2024-04-20T17:11:07.967 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7184cd5c:::benchmark_data_smithi171_78918_object107579:head# 2024-04-20T17:11:07.967 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71868c70:::benchmark_data_smithi171_78918_object106136:head# 2024-04-20T17:11:07.973 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7186997a:::benchmark_data_smithi171_78918_object114499:head# 2024-04-20T17:11:07.973 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7186e13d:::benchmark_data_smithi171_78918_object87970:head# 2024-04-20T17:11:07.973 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7187f419:::benchmark_data_smithi171_78918_object118581:head# 2024-04-20T17:11:07.973 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71882186:::benchmark_data_smithi171_78918_object136727:head# 2024-04-20T17:11:07.973 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71885861:::benchmark_data_smithi171_78918_object133078:head# 2024-04-20T17:11:07.973 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71891daa:::benchmark_data_smithi171_78918_object77141:head# 2024-04-20T17:11:07.973 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:718930b4:::benchmark_data_smithi171_78918_object98569:head# 2024-04-20T17:11:07.973 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71893ee9:::benchmark_data_smithi171_78918_object124084:head# 2024-04-20T17:11:07.973 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71894949:::benchmark_data_smithi171_78918_object81398:head# 2024-04-20T17:11:07.973 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:718a7ed9:::benchmark_data_smithi171_78918_object114890:head# 2024-04-20T17:11:07.973 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:718b2972:::benchmark_data_smithi171_78918_object144238:head# 2024-04-20T17:11:07.973 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:718b4640:::benchmark_data_smithi171_78918_object98404:head# 2024-04-20T17:11:07.973 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:718c88dc:::benchmark_data_smithi171_78918_object108999:head# 2024-04-20T17:11:07.974 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:718ce256:::benchmark_data_smithi171_78918_object120598:head# 2024-04-20T17:11:07.974 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:718d399b:::benchmark_data_smithi171_78918_object126587:head# 2024-04-20T17:11:07.974 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:718db564:::benchmark_data_smithi171_78918_object131948:head# 2024-04-20T17:11:07.974 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:718db7af:::benchmark_data_smithi171_78918_object150080:head# 2024-04-20T17:11:07.974 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:718e0fcf:::benchmark_data_smithi171_78918_object68784:head# 2024-04-20T17:11:07.974 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:718e7031:::benchmark_data_smithi171_78918_object125327:head# 2024-04-20T17:11:07.974 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:718e9335:::benchmark_data_smithi171_78918_object134203:head# 2024-04-20T17:11:07.974 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:718ea60e:::benchmark_data_smithi171_78918_object100602:head# 2024-04-20T17:11:07.974 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:718ed8ca:::benchmark_data_smithi171_78918_object64333:head# 2024-04-20T17:11:07.974 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:719135c8:::benchmark_data_smithi171_78918_object69757:head# 2024-04-20T17:11:07.974 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7191546d:::benchmark_data_smithi171_78918_object141779:head# 2024-04-20T17:11:07.974 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71915f5b:::benchmark_data_smithi171_78918_object140176:head# 2024-04-20T17:11:07.974 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:719176ec:::benchmark_data_smithi171_78918_object152890:head# 2024-04-20T17:11:07.974 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7191c4b1:::benchmark_data_smithi171_78918_object102600:head# 2024-04-20T17:11:07.974 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71925136:::benchmark_data_smithi171_78918_object127919:head# 2024-04-20T17:11:07.975 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7193bbce:::benchmark_data_smithi171_78918_object129725:head# 2024-04-20T17:11:07.975 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:719413da:::benchmark_data_smithi171_78918_object82809:head# 2024-04-20T17:11:07.975 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7194c86c:::benchmark_data_smithi171_78918_object160211:head# 2024-04-20T17:11:07.975 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71952815:::benchmark_data_smithi171_78918_object110489:head# 2024-04-20T17:11:07.975 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71954d50:::benchmark_data_smithi171_78918_object152032:head# 2024-04-20T17:11:07.975 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:719630e8:::benchmark_data_smithi171_78918_object121353:head# 2024-04-20T17:11:07.975 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71964383:::benchmark_data_smithi171_78918_object117526:head# 2024-04-20T17:11:07.975 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71966bfd:::benchmark_data_smithi171_78918_object125099:head# 2024-04-20T17:11:07.975 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7196aced:::benchmark_data_smithi171_78918_object134658:head# 2024-04-20T17:11:07.975 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7196b4b9:::benchmark_data_smithi171_78918_object105557:head# 2024-04-20T17:11:07.975 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7196d530:::benchmark_data_smithi171_78918_object149411:head# 2024-04-20T17:11:07.975 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71971618:::benchmark_data_smithi171_78918_object95654:head# 2024-04-20T17:11:07.975 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7197216d:::benchmark_data_smithi171_78918_object159360:head# 2024-04-20T17:11:07.975 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:719790c6:::benchmark_data_smithi171_78918_object128212:head# 2024-04-20T17:11:07.975 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71989823:::benchmark_data_smithi171_78918_object113551:head# 2024-04-20T17:11:07.975 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:719aa2f1:::benchmark_data_smithi171_78918_object98290:head# 2024-04-20T17:11:07.976 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:719c897e:::benchmark_data_smithi171_78918_object134169:head# 2024-04-20T17:11:07.976 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:719cf7fc:::benchmark_data_smithi171_78918_object63686:head# 2024-04-20T17:11:07.976 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:719d8333:::benchmark_data_smithi171_78918_object124213:head# 2024-04-20T17:11:07.976 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:719ec08c:::benchmark_data_smithi171_78918_object152695:head# 2024-04-20T17:11:07.976 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:719ed440:::benchmark_data_smithi171_78918_object93962:head# 2024-04-20T17:11:07.976 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:719f2354:::benchmark_data_smithi171_78918_object140595:head# 2024-04-20T17:11:07.976 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:719ff6d3:::benchmark_data_smithi171_78918_object115824:head# 2024-04-20T17:11:07.976 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71a111ad:::benchmark_data_smithi171_78918_object63529:head# 2024-04-20T17:11:07.976 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71a18d23:::benchmark_data_smithi171_78918_object122119:head# 2024-04-20T17:11:07.976 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71a19c52:::benchmark_data_smithi171_78918_object127880:head# 2024-04-20T17:11:07.979 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71a329da:::benchmark_data_smithi171_78918_object64208:head# 2024-04-20T17:11:07.979 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71a34872:::benchmark_data_smithi171_78918_object88279:head# 2024-04-20T17:11:07.979 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71a3583e:::benchmark_data_smithi171_78918_object130481:head# 2024-04-20T17:11:07.979 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71a41217:::benchmark_data_smithi171_78918_object83736:head# 2024-04-20T17:11:07.979 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71a47747:::benchmark_data_smithi171_78918_object105089:head# 2024-04-20T17:11:07.979 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71a54d54:::benchmark_data_smithi171_78918_object54033:head# 2024-04-20T17:11:07.979 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71a57232:::benchmark_data_smithi171_78918_object138224:head# 2024-04-20T17:11:07.979 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71a60901:::benchmark_data_smithi171_78918_object82304:head# 2024-04-20T17:11:07.979 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71a6e724:::benchmark_data_smithi171_78918_object153720:head# 2024-04-20T17:11:07.979 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71a71d9d:::benchmark_data_smithi171_78918_object160173:head# 2024-04-20T17:11:07.979 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71a7355f:::benchmark_data_smithi171_78918_object85477:head# 2024-04-20T17:11:07.979 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71a7ad55:::benchmark_data_smithi171_78918_object92393:head# 2024-04-20T17:11:07.979 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71a8ba6d:::benchmark_data_smithi171_78918_object118086:head# 2024-04-20T17:11:07.980 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71a8d6e1:::benchmark_data_smithi171_78918_object99936:head# 2024-04-20T17:11:07.980 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71a9345b:::benchmark_data_smithi171_78918_object156338:head# 2024-04-20T17:11:07.980 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71a94873:::benchmark_data_smithi171_78918_object149368:head# 2024-04-20T17:11:07.980 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71a9c30b:::benchmark_data_smithi171_78918_object96630:head# 2024-04-20T17:11:07.980 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71aa2257:::benchmark_data_smithi171_78918_object76363:head# 2024-04-20T17:11:07.980 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71aa8ea2:::benchmark_data_smithi171_78918_object131919:head# 2024-04-20T17:11:07.980 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71aabbb5:::benchmark_data_smithi171_78918_object154399:head# 2024-04-20T17:11:07.980 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71ab3738:::benchmark_data_smithi171_78918_object54603:head# 2024-04-20T17:11:07.980 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71ab54ab:::benchmark_data_smithi171_78918_object94614:head# 2024-04-20T17:11:07.980 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71abae29:::benchmark_data_smithi171_78918_object99806:head# 2024-04-20T17:11:07.980 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71ac2adf:::benchmark_data_smithi171_78918_object117158:head# 2024-04-20T17:11:07.980 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71ac8b85:::benchmark_data_smithi171_78918_object156029:head# 2024-04-20T17:11:07.980 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71ad6220:::benchmark_data_smithi171_78918_object96430:head# 2024-04-20T17:11:07.980 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71ad8886:::benchmark_data_smithi171_78918_object144110:head# 2024-04-20T17:11:07.980 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71ae5dd5:::benchmark_data_smithi171_78918_object104750:head# 2024-04-20T17:11:07.983 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71af4fbc:::benchmark_data_smithi171_78918_object55635:head# 2024-04-20T17:11:07.983 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71af89f8:::benchmark_data_smithi171_78918_object81576:head# 2024-04-20T17:11:07.983 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71aff8e2:::benchmark_data_smithi171_78918_object98791:head# 2024-04-20T17:11:07.983 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71b046fa:::benchmark_data_smithi171_78918_object78420:head# 2024-04-20T17:11:07.983 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71b059e8:::benchmark_data_smithi171_78918_object150893:head# 2024-04-20T17:11:07.983 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71b0d259:::benchmark_data_smithi171_78918_object119237:head# 2024-04-20T17:11:07.983 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71b12106:::benchmark_data_smithi171_78918_object100273:head# 2024-04-20T17:11:07.983 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71b12fa6:::benchmark_data_smithi171_78918_object68196:head# 2024-04-20T17:11:07.983 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71b146c8:::benchmark_data_smithi171_78918_object130330:head# 2024-04-20T17:11:07.983 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71b18ba4:::benchmark_data_smithi171_78918_object100664:head# 2024-04-20T17:11:07.983 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71b27d13:::benchmark_data_smithi171_78918_object74545:head# 2024-04-20T17:11:07.983 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71b30f72:::benchmark_data_smithi171_78918_object105350:head# 2024-04-20T17:11:07.984 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71b38c2b:::benchmark_data_smithi171_78918_object81771:head# 2024-04-20T17:11:07.984 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71b3dcb8:::benchmark_data_smithi171_78918_object153756:head# 2024-04-20T17:11:07.984 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71b4d926:::benchmark_data_smithi171_78918_object118571:head# 2024-04-20T17:11:07.984 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71b51a4e:::benchmark_data_smithi171_78918_object84323:head# 2024-04-20T17:11:07.984 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71b52b71:::benchmark_data_smithi171_78918_object55124:head# 2024-04-20T17:11:07.984 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71b58bf5:::benchmark_data_smithi171_78918_object129201:head# 2024-04-20T17:11:07.984 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71b58fb0:::benchmark_data_smithi171_78918_object118131:head# 2024-04-20T17:11:07.984 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71b5ff01:::benchmark_data_smithi171_78918_object111629:head# 2024-04-20T17:11:07.984 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:11:07.987 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71b6228f:::benchmark_data_smithi171_78918_object119728:head# 2024-04-20T17:11:07.987 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71b6eb8b:::benchmark_data_smithi171_78918_object144750:head# 2024-04-20T17:11:07.987 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71b78e12:::benchmark_data_smithi171_78918_object133818:head# 2024-04-20T17:11:07.987 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71b7dfce:::benchmark_data_smithi171_78918_object117141:head# 2024-04-20T17:11:07.987 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71b8246e:::benchmark_data_smithi171_78918_object156033:head# 2024-04-20T17:11:07.987 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71b878c4:::benchmark_data_smithi171_78918_object132253:head# 2024-04-20T17:11:07.987 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71b955ac:::benchmark_data_smithi171_78918_object142628:head# 2024-04-20T17:11:07.987 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71ba0d12:::benchmark_data_smithi171_78918_object139909:head# 2024-04-20T17:11:07.987 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71ba6525:::benchmark_data_smithi171_78918_object82006:head# 2024-04-20T17:11:07.987 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71baa820:::benchmark_data_smithi171_78918_object143811:head# 2024-04-20T17:11:07.987 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71bb0d03:::benchmark_data_smithi171_78918_object155223:head# 2024-04-20T17:11:07.987 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71bb26c3:::benchmark_data_smithi171_78918_object68807:head# 2024-04-20T17:11:07.988 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71bb7e68:::benchmark_data_smithi171_78918_object126249:head# 2024-04-20T17:11:07.988 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71bc9f51:::benchmark_data_smithi171_78918_object60112:head# 2024-04-20T17:11:07.988 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71bdc3d2:::benchmark_data_smithi171_78918_object77273:head# 2024-04-20T17:11:07.988 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71be1d74:::benchmark_data_smithi171_78918_object89366:head# 2024-04-20T17:11:07.989 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71beca88:::benchmark_data_smithi171_78918_object57398:head# 2024-04-20T17:11:07.989 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71c0436a:::benchmark_data_smithi171_78918_object101686:head# 2024-04-20T17:11:07.989 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71c04778:::benchmark_data_smithi171_78918_object88439:head# 2024-04-20T17:11:07.990 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71c0d168:::benchmark_data_smithi171_78918_object67612:head# 2024-04-20T17:11:07.990 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71c1e545:::benchmark_data_smithi171_78918_object130379:head# 2024-04-20T17:11:07.990 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71c3285a:::benchmark_data_smithi171_78918_object109846:head# 2024-04-20T17:11:07.990 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71c39b87:::benchmark_data_smithi171_78918_object81261:head# 2024-04-20T17:11:07.991 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71c49944:::benchmark_data_smithi171_78918_object143284:head# 2024-04-20T17:11:07.991 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71c543e7:::benchmark_data_smithi171_78918_object100161:head# 2024-04-20T17:11:07.991 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71c575a3:::benchmark_data_smithi171_78918_object109115:head# 2024-04-20T17:11:07.991 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71c5eda9:::benchmark_data_smithi171_78918_object101875:head# 2024-04-20T17:11:07.991 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71c62baa:::benchmark_data_smithi171_78918_object72683:head# 2024-04-20T17:11:07.991 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71c647be:::benchmark_data_smithi171_78918_object116346:head# 2024-04-20T17:11:07.991 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71c75e42:::benchmark_data_smithi171_78918_object113200:head# 2024-04-20T17:11:07.991 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71c7d8e8:::benchmark_data_smithi171_78918_object65498:head# 2024-04-20T17:11:07.992 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71c7f54c:::benchmark_data_smithi171_78918_object79414:head# 2024-04-20T17:11:07.993 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71c832a2:::benchmark_data_smithi171_78918_object68923:head# 2024-04-20T17:11:07.993 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71c859ad:::benchmark_data_smithi171_78918_object121351:head# 2024-04-20T17:11:07.993 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71c9550b:::benchmark_data_smithi171_78918_object129616:head# 2024-04-20T17:11:07.993 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71c95bb4:::benchmark_data_smithi171_78918_object125494:head# 2024-04-20T17:11:07.993 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71c9f2f0:::benchmark_data_smithi171_78918_object113410:head# 2024-04-20T17:11:07.993 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71cad3bd:::benchmark_data_smithi171_78918_object55754:head# 2024-04-20T17:11:07.993 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71cc0338:::benchmark_data_smithi171_78918_object108667:head# 2024-04-20T17:11:07.993 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71cc3454:::benchmark_data_smithi171_78918_object86034:head# 2024-04-20T17:11:07.993 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71ccc415:::benchmark_data_smithi171_78918_object75108:head# 2024-04-20T17:11:07.993 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71ccf23c:::benchmark_data_smithi171_78918_object52550:head# 2024-04-20T17:11:07.993 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71ce41cb:::benchmark_data_smithi171_78918_object103531:head# 2024-04-20T17:11:07.994 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71cebd85:::benchmark_data_smithi171_78918_object114109:head# 2024-04-20T17:11:07.994 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71cfcc77:::benchmark_data_smithi171_78918_object80421:head# 2024-04-20T17:11:07.994 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71d027b8:::benchmark_data_smithi171_78918_object140731:head# 2024-04-20T17:11:07.994 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71d04bdc:::benchmark_data_smithi171_78918_object64324:head# 2024-04-20T17:11:07.995 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71d0a867:::benchmark_data_smithi171_78918_object67504:head# 2024-04-20T17:11:07.996 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71d162f3:::benchmark_data_smithi171_78918_object59559:head# 2024-04-20T17:11:07.996 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71d39ccf:::benchmark_data_smithi171_78918_object125651:head# 2024-04-20T17:11:07.996 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71d3c437:::benchmark_data_smithi171_78918_object125122:head# 2024-04-20T17:11:07.996 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71d409cb:::benchmark_data_smithi171_78918_object101593:head# 2024-04-20T17:11:07.996 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71d52d6d:::benchmark_data_smithi171_78918_object123531:head# 2024-04-20T17:11:07.996 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71d54fc2:::benchmark_data_smithi171_78918_object149009:head# 2024-04-20T17:11:07.996 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71d55987:::benchmark_data_smithi171_78918_object112538:head# 2024-04-20T17:11:07.996 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71d58706:::benchmark_data_smithi171_78918_object156420:head# 2024-04-20T17:11:07.996 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71d69f11:::benchmark_data_smithi171_78918_object73610:head# 2024-04-20T17:11:07.998 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71d7863a:::benchmark_data_smithi171_78918_object85078:head# 2024-04-20T17:11:07.998 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71d8989e:::benchmark_data_smithi171_78918_object115726:head# 2024-04-20T17:11:07.998 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71d8ea1a:::benchmark_data_smithi171_78918_object67985:head# 2024-04-20T17:11:07.998 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71d95d14:::benchmark_data_smithi171_78918_object132584:head# 2024-04-20T17:11:07.998 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71d97120:::benchmark_data_smithi171_78918_object96764:head# 2024-04-20T17:11:07.998 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71da73a5:::benchmark_data_smithi171_78918_object58972:head# 2024-04-20T17:11:07.998 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71db7e8f:::benchmark_data_smithi171_78918_object87916:head# 2024-04-20T17:11:07.998 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71dbd663:::benchmark_data_smithi171_78918_object143657:head# 2024-04-20T17:11:07.998 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71dc1c09:::benchmark_data_smithi171_78918_object85323:head# 2024-04-20T17:11:07.998 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71dc1e09:::benchmark_data_smithi171_78918_object117105:head# 2024-04-20T17:11:07.999 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71df2552:::benchmark_data_smithi171_78918_object115214:head# 2024-04-20T17:11:07.999 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71df9b55:::benchmark_data_smithi171_78918_object102227:head# 2024-04-20T17:11:07.999 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71dfbc14:::benchmark_data_smithi171_78918_object58636:head# 2024-04-20T17:11:07.999 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71e00d01:::benchmark_data_smithi171_78918_object69353:head# 2024-04-20T17:11:07.999 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71e13d95:::benchmark_data_smithi171_78918_object53544:head# 2024-04-20T17:11:08.002 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71e1a7c1:::benchmark_data_smithi171_78918_object132463:head# 2024-04-20T17:11:08.002 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71e52916:::benchmark_data_smithi171_78918_object59812:head# 2024-04-20T17:11:08.002 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71e54295:::benchmark_data_smithi171_78918_object52210:head# 2024-04-20T17:11:08.002 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71e57046:::benchmark_data_smithi171_78918_object154022:head# 2024-04-20T17:11:08.003 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71e5e508:::benchmark_data_smithi171_78918_object130161:head# 2024-04-20T17:11:08.003 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71e73498:::benchmark_data_smithi171_78918_object86660:head# 2024-04-20T17:11:08.003 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71e73b87:::benchmark_data_smithi171_78918_object86306:head# 2024-04-20T17:11:08.003 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71e7b7e9:::benchmark_data_smithi171_78918_object72953:head# 2024-04-20T17:11:08.003 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71e7fd38:::benchmark_data_smithi171_78918_object120043:head# 2024-04-20T17:11:08.003 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71e874b0:::benchmark_data_smithi171_78918_object55248:head# 2024-04-20T17:11:08.003 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71e8c231:::benchmark_data_smithi171_78918_object99293:head# 2024-04-20T17:11:08.003 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71e95252:::benchmark_data_smithi171_78918_object99148:head# 2024-04-20T17:11:08.003 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71eb4730:::benchmark_data_smithi171_78918_object107523:head# 2024-04-20T17:11:08.003 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71eb9894:::benchmark_data_smithi171_78918_object90572:head# 2024-04-20T17:11:08.003 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71ebe7a7:::benchmark_data_smithi171_78918_object56471:head# 2024-04-20T17:11:08.003 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71ec3c1b:::benchmark_data_smithi171_78918_object102906:head# 2024-04-20T17:11:08.004 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71ec4ca8:::benchmark_data_smithi171_78918_object109732:head# 2024-04-20T17:11:08.004 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71ec6399:::benchmark_data_smithi171_78918_object64291:head# 2024-04-20T17:11:08.004 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71ec9493:::benchmark_data_smithi171_78918_object114829:head# 2024-04-20T17:11:08.005 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71eeddbb:::benchmark_data_smithi171_78918_object70993:head# 2024-04-20T17:11:08.005 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71eeeca4:::benchmark_data_smithi171_78918_object59252:head# 2024-04-20T17:11:08.006 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71ef265a:::benchmark_data_smithi171_78918_object137904:head# 2024-04-20T17:11:08.006 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71efa570:::benchmark_data_smithi171_78918_object59320:head# 2024-04-20T17:11:08.006 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71efb132:::benchmark_data_smithi171_78918_object113997:head# 2024-04-20T17:11:08.006 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71f377b9:::benchmark_data_smithi171_78918_object97143:head# 2024-04-20T17:11:08.006 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71f38573:::benchmark_data_smithi171_78918_object102895:head# 2024-04-20T17:11:08.006 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71f3bae1:::benchmark_data_smithi171_78918_object79576:head# 2024-04-20T17:11:08.006 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71f3d612:::benchmark_data_smithi171_78918_object95972:head# 2024-04-20T17:11:08.006 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71f3eded:::benchmark_data_smithi171_78918_object121701:head# 2024-04-20T17:11:08.006 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71f598ad:::benchmark_data_smithi171_78918_object107375:head# 2024-04-20T17:11:08.006 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71f5b2bd:::benchmark_data_smithi171_78918_object103796:head# 2024-04-20T17:11:08.006 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71f765f5:::benchmark_data_smithi171_78918_object52920:head# 2024-04-20T17:11:08.006 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71f7fa42:::benchmark_data_smithi171_78918_object108916:head# 2024-04-20T17:11:08.007 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71f88abf:::benchmark_data_smithi171_78918_object145267:head# 2024-04-20T17:11:08.007 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71f9b510:::benchmark_data_smithi171_78918_object122601:head# 2024-04-20T17:11:08.007 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71fa1043:::benchmark_data_smithi171_78918_object72285:head# 2024-04-20T17:11:08.008 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71fa886d:::benchmark_data_smithi171_78918_object146956:head# 2024-04-20T17:11:08.008 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71faf492:::benchmark_data_smithi171_78918_object83447:head# 2024-04-20T17:11:08.008 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71fb1a5a:::benchmark_data_smithi171_78918_object116846:head# 2024-04-20T17:11:08.008 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71fb7678:::benchmark_data_smithi171_78918_object72207:head# 2024-04-20T17:11:08.008 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71fc04ea:::benchmark_data_smithi171_78918_object86757:head# 2024-04-20T17:11:08.008 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71fc5948:::benchmark_data_smithi171_78918_object141916:head# 2024-04-20T17:11:08.009 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71fd1357:::benchmark_data_smithi171_78918_object74693:head# 2024-04-20T17:11:08.009 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71fde422:::benchmark_data_smithi171_78918_object81728:head# 2024-04-20T17:11:08.009 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71fe62d5:::benchmark_data_smithi171_78918_object137287:head# 2024-04-20T17:11:08.009 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:71fffd07:::benchmark_data_smithi171_78918_object73498:head# 2024-04-20T17:11:08.009 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:720124ac:::benchmark_data_smithi171_78918_object91699:head# 2024-04-20T17:11:08.009 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:720177cc:::benchmark_data_smithi171_78918_object132995:head# 2024-04-20T17:11:08.009 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72019708:::benchmark_data_smithi171_78918_object160525:head# 2024-04-20T17:11:08.009 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7201d7d9:::benchmark_data_smithi171_78918_object80036:head# 2024-04-20T17:11:08.009 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7202586b:::benchmark_data_smithi171_78918_object120283:head# 2024-04-20T17:11:08.010 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:720289a4:::benchmark_data_smithi171_78918_object131831:head# 2024-04-20T17:11:08.011 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7202b200:::benchmark_data_smithi171_78918_object140480:head# 2024-04-20T17:11:08.011 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7202c2db:::benchmark_data_smithi171_78918_object124863:head# 2024-04-20T17:11:08.011 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7203651b:::benchmark_data_smithi171_78918_object119210:head# 2024-04-20T17:11:08.011 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7203e11a:::benchmark_data_smithi171_78918_object159621:head# 2024-04-20T17:11:08.011 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7203eadf:::benchmark_data_smithi171_78918_object112066:head# 2024-04-20T17:11:08.011 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72051a34:::benchmark_data_smithi171_78918_object125166:head# 2024-04-20T17:11:08.011 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:720540d7:::benchmark_data_smithi171_78918_object98609:head# 2024-04-20T17:11:08.011 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7205aec7:::benchmark_data_smithi171_78918_object79075:head# 2024-04-20T17:11:08.011 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:720634cd:::benchmark_data_smithi171_78918_object108426:head# 2024-04-20T17:11:08.012 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7206974f:::benchmark_data_smithi171_78918_object121729:head# 2024-04-20T17:11:08.012 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7206b9da:::benchmark_data_smithi171_78918_object154874:head# 2024-04-20T17:11:08.013 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72077a40:::benchmark_data_smithi171_78918_object131385:head# 2024-04-20T17:11:08.013 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7207b255:::benchmark_data_smithi171_78918_object68670:head# 2024-04-20T17:11:08.013 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7207dba9:::benchmark_data_smithi171_78918_object81707:head# 2024-04-20T17:11:08.013 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7207dc3e:::benchmark_data_smithi171_78918_object111383:head# 2024-04-20T17:11:08.013 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72081ec8:::benchmark_data_smithi171_78918_object115742:head# 2024-04-20T17:11:08.013 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7208b9c3:::benchmark_data_smithi171_78918_object77619:head# 2024-04-20T17:11:08.013 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:720928c7:::benchmark_data_smithi171_78918_object56145:head# 2024-04-20T17:11:08.013 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72099331:::benchmark_data_smithi171_78918_object59515:head# 2024-04-20T17:11:08.013 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7209e72a:::benchmark_data_smithi171_78918_object129540:head# 2024-04-20T17:11:08.014 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:720a7add:::benchmark_data_smithi171_78918_object69549:head# 2024-04-20T17:11:08.014 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:720af474:::benchmark_data_smithi171_78918_object54111:head# 2024-04-20T17:11:08.015 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:720b4a79:::benchmark_data_smithi171_78918_object70463:head# 2024-04-20T17:11:08.015 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:720b50bf:::benchmark_data_smithi171_78918_object160436:head# 2024-04-20T17:11:08.015 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:720b52df:::benchmark_data_smithi171_78918_object53818:head# 2024-04-20T17:11:08.015 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:720bdd22:::benchmark_data_smithi171_78918_object155190:head# 2024-04-20T17:11:08.015 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:720beafc:::benchmark_data_smithi171_78918_object115354:head# 2024-04-20T17:11:08.015 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:720cb6cf:::benchmark_data_smithi171_78918_object109091:head# 2024-04-20T17:11:08.015 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:720cbeda:::benchmark_data_smithi171_78918_object120749:head# 2024-04-20T17:11:08.015 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:720cfce1:::benchmark_data_smithi171_78918_object156893:head# 2024-04-20T17:11:08.016 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:720d2dee:::benchmark_data_smithi171_78918_object157008:head# 2024-04-20T17:11:08.016 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:720deae2:::benchmark_data_smithi171_78918_object91311:head# 2024-04-20T17:11:08.016 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:720f85ea:::benchmark_data_smithi171_78918_object83631:head# 2024-04-20T17:11:08.017 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72105a2e:::benchmark_data_smithi171_78918_object89736:head# 2024-04-20T17:11:08.017 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7211adf7:::benchmark_data_smithi171_78918_object85012:head# 2024-04-20T17:11:08.017 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72123cfd:::benchmark_data_smithi171_78918_object55977:head# 2024-04-20T17:11:08.017 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7212da1e:::benchmark_data_smithi171_78918_object72271:head# 2024-04-20T17:11:08.017 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72133c9c:::benchmark_data_smithi171_78918_object104066:head# 2024-04-20T17:11:08.017 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72138040:::benchmark_data_smithi171_78918_object151103:head# 2024-04-20T17:11:08.017 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72160790:::benchmark_data_smithi171_78918_object158248:head# 2024-04-20T17:11:08.018 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7216642a:::benchmark_data_smithi171_78918_object99353:head# 2024-04-20T17:11:08.018 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72167de5:::benchmark_data_smithi171_78918_object126732:head# 2024-04-20T17:11:08.018 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72168ccf:::benchmark_data_smithi171_78918_object72921:head# 2024-04-20T17:11:08.018 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:721719e3:::benchmark_data_smithi171_78918_object106142:head# 2024-04-20T17:11:08.018 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72172a31:::benchmark_data_smithi171_78918_object160629:head# 2024-04-20T17:11:08.019 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72181d1d:::benchmark_data_smithi171_78918_object84979:head# 2024-04-20T17:11:08.019 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72190c5f:::benchmark_data_smithi171_78918_object56280:head# 2024-04-20T17:11:08.019 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:721a46de:::benchmark_data_smithi171_78918_object93847:head# 2024-04-20T17:11:08.019 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:721b3a41:::benchmark_data_smithi171_78918_object87996:head# 2024-04-20T17:11:08.019 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:721bb503:::benchmark_data_smithi171_78918_object103605:head# 2024-04-20T17:11:08.019 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:721c30d6:::benchmark_data_smithi171_78918_object71885:head# 2024-04-20T17:11:08.019 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:721cb266:::benchmark_data_smithi171_78918_object130393:head# 2024-04-20T17:11:08.019 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:721ccd9f:::benchmark_data_smithi171_78918_object152513:head# 2024-04-20T17:11:08.019 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:721d01c1:::benchmark_data_smithi171_78918_object103506:head# 2024-04-20T17:11:08.020 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:721d2029:::benchmark_data_smithi171_78918_object99598:head# 2024-04-20T17:11:08.020 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:721d57b0:::benchmark_data_smithi171_78918_object152152:head# 2024-04-20T17:11:08.020 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:721db6a5:::benchmark_data_smithi171_78918_object96489:head# 2024-04-20T17:11:08.020 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:721e6426:::benchmark_data_smithi171_78918_object98572:head# 2024-04-20T17:11:08.021 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:721f1355:::benchmark_data_smithi171_78918_object110262:head# 2024-04-20T17:11:08.022 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:721f74ce:::benchmark_data_smithi171_78918_object129780:head# 2024-04-20T17:11:08.022 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:721fddb4:::benchmark_data_smithi171_78918_object113149:head# 2024-04-20T17:11:08.022 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7220c8c2:::benchmark_data_smithi171_78918_object98086:head# 2024-04-20T17:11:08.022 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72225150:::benchmark_data_smithi171_78918_object52953:head# 2024-04-20T17:11:08.022 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7222882b:::benchmark_data_smithi171_78918_object128624:head# 2024-04-20T17:11:08.022 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:722388bb:::benchmark_data_smithi171_78918_object117616:head# 2024-04-20T17:11:08.023 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72241313:::benchmark_data_smithi171_78918_object144153:head# 2024-04-20T17:11:08.023 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7224d533:::benchmark_data_smithi171_78918_object153376:head# 2024-04-20T17:11:08.023 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72254b51:::benchmark_data_smithi171_78918_object64790:head# 2024-04-20T17:11:08.023 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72260e8b:::benchmark_data_smithi171_78918_object159173:head# 2024-04-20T17:11:08.023 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:08.024 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7226b469:::benchmark_data_smithi171_78918_object142181:head# 2024-04-20T17:11:08.024 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7226edfb:::benchmark_data_smithi171_78918_object120520:head# 2024-04-20T17:11:08.024 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:722824db:::benchmark_data_smithi171_78918_object105356:head# 2024-04-20T17:11:08.024 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7228a128:::benchmark_data_smithi171_78918_object88972:head# 2024-04-20T17:11:08.024 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72292146:::benchmark_data_smithi171_78918_object94515:head# 2024-04-20T17:11:08.024 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72293b72:::benchmark_data_smithi171_78918_object83103:head# 2024-04-20T17:11:08.024 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7229cf99:::benchmark_data_smithi171_78918_object133584:head# 2024-04-20T17:11:08.024 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7229daf7:::benchmark_data_smithi171_78918_object115198:head# 2024-04-20T17:11:08.025 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:722a8bc2:::benchmark_data_smithi171_78918_object52510:head# 2024-04-20T17:11:08.025 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:722b2268:::benchmark_data_smithi171_78918_object93572:head# 2024-04-20T17:11:08.025 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:722b4014:::benchmark_data_smithi171_78918_object57633:head# 2024-04-20T17:11:08.025 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:722b97cb:::benchmark_data_smithi171_78918_object105829:head# 2024-04-20T17:11:08.025 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:722d8df4:::benchmark_data_smithi171_78918_object104064:head# 2024-04-20T17:11:08.025 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:722de1fb:::benchmark_data_smithi171_78918_object114633:head# 2024-04-20T17:11:08.026 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:722e434f:::benchmark_data_smithi171_78918_object143231:head# 2024-04-20T17:11:08.027 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:722fa89d:::benchmark_data_smithi171_78918_object148258:head# 2024-04-20T17:11:08.027 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7230bf58:::benchmark_data_smithi171_78918_object59895:head# 2024-04-20T17:11:08.027 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72310bc2:::benchmark_data_smithi171_78918_object101752:head# 2024-04-20T17:11:08.027 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72312ca5:::benchmark_data_smithi171_78918_object135341:head# 2024-04-20T17:11:08.027 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7232227d:::benchmark_data_smithi171_78918_object69993:head# 2024-04-20T17:11:08.027 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:723226fe:::benchmark_data_smithi171_78918_object156228:head# 2024-04-20T17:11:08.027 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72324d13:::benchmark_data_smithi171_78918_object122747:head# 2024-04-20T17:11:08.027 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72333e37:::benchmark_data_smithi171_78918_object149870:head# 2024-04-20T17:11:08.030 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7233d833:::benchmark_data_smithi171_78918_object157983:head# 2024-04-20T17:11:08.030 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7233ef65:::benchmark_data_smithi171_78918_object111090:head# 2024-04-20T17:11:08.030 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7233f0f8:::benchmark_data_smithi171_78918_object73166:head# 2024-04-20T17:11:08.030 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:723456aa:::benchmark_data_smithi171_78918_object130671:head# 2024-04-20T17:11:08.030 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7234f61c:::benchmark_data_smithi171_78918_object77609:head# 2024-04-20T17:11:08.030 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7234f706:::benchmark_data_smithi171_78918_object104849:head# 2024-04-20T17:11:08.030 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7235a1f3:::benchmark_data_smithi171_78918_object159872:head# 2024-04-20T17:11:08.030 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72367b29:::benchmark_data_smithi171_78918_object124791:head# 2024-04-20T17:11:08.030 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7237c25e:::benchmark_data_smithi171_78918_object79260:head# 2024-04-20T17:11:08.030 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72381da1:::benchmark_data_smithi171_78918_object59497:head# 2024-04-20T17:11:08.030 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72384175:::benchmark_data_smithi171_78918_object93986:head# 2024-04-20T17:11:08.030 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72385766:::benchmark_data_smithi171_78918_object143660:head# 2024-04-20T17:11:08.030 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72393f45:::benchmark_data_smithi171_78918_object70655:head# 2024-04-20T17:11:08.030 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:723b62ff:::benchmark_data_smithi171_78918_object70966:head# 2024-04-20T17:11:08.031 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:723b6b78:::benchmark_data_smithi171_78918_object64645:head# 2024-04-20T17:11:08.033 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:723b75f4:::benchmark_data_smithi171_78918_object150460:head# 2024-04-20T17:11:08.034 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:723ba4aa:::benchmark_data_smithi171_78918_object137723:head# 2024-04-20T17:11:08.034 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:723c0f50:::benchmark_data_smithi171_78918_object144454:head# 2024-04-20T17:11:08.034 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:723c872d:::benchmark_data_smithi171_78918_object67936:head# 2024-04-20T17:11:08.034 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:723ca525:::benchmark_data_smithi171_78918_object159115:head# 2024-04-20T17:11:08.034 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:723ce83d:::benchmark_data_smithi171_78918_object145357:head# 2024-04-20T17:11:08.034 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:723ceff0:::benchmark_data_smithi171_78918_object63732:head# 2024-04-20T17:11:08.034 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:723d00d2:::benchmark_data_smithi171_78918_object62151:head# 2024-04-20T17:11:08.034 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:723d9e48:::benchmark_data_smithi171_78918_object153975:head# 2024-04-20T17:11:08.034 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:723e0040:::benchmark_data_smithi171_78918_object129045:head# 2024-04-20T17:11:08.034 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:723e1cd4:::benchmark_data_smithi171_78918_object120288:head# 2024-04-20T17:11:08.034 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:723e3fec:::benchmark_data_smithi171_78918_object54015:head# 2024-04-20T17:11:08.034 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:723f29bd:::benchmark_data_smithi171_78918_object143519:head# 2024-04-20T17:11:08.035 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:723f63b4:::benchmark_data_smithi171_78918_object99668:head# 2024-04-20T17:11:08.035 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72402b72:::benchmark_data_smithi171_78918_object95226:head# 2024-04-20T17:11:08.035 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72407dd9:::benchmark_data_smithi171_78918_object59933:head# 2024-04-20T17:11:08.035 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7240a0a8:::benchmark_data_smithi171_78918_object55631:head# 2024-04-20T17:11:08.035 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72415603:::benchmark_data_smithi171_78918_object75815:head# 2024-04-20T17:11:08.035 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72435475:::benchmark_data_smithi171_78918_object70926:head# 2024-04-20T17:11:08.035 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7244ebf3:::benchmark_data_smithi171_78918_object80954:head# 2024-04-20T17:11:08.035 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72457fed:::benchmark_data_smithi171_78918_object56468:head# 2024-04-20T17:11:08.035 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72458aaf:::benchmark_data_smithi171_78918_object74998:head# 2024-04-20T17:11:08.035 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7245b86c:::benchmark_data_smithi171_78918_object83789:head# 2024-04-20T17:11:08.036 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72469e9a:::benchmark_data_smithi171_78918_object66479:head# 2024-04-20T17:11:08.036 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7247ab9a:::benchmark_data_smithi171_78918_object160231:head# 2024-04-20T17:11:08.036 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7247b299:::benchmark_data_smithi171_78918_object113273:head# 2024-04-20T17:11:08.036 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72491e0d:::benchmark_data_smithi171_78918_object149897:head# 2024-04-20T17:11:08.036 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7249c838:::benchmark_data_smithi171_78918_object81334:head# 2024-04-20T17:11:08.037 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7249ccf4:::benchmark_data_smithi171_78918_object142184:head# 2024-04-20T17:11:08.037 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:724a1568:::benchmark_data_smithi171_78918_object160935:head# 2024-04-20T17:11:08.037 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:724a607c:::benchmark_data_smithi171_78918_object56526:head# 2024-04-20T17:11:08.037 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:724ac229:::benchmark_data_smithi171_78918_object68365:head# 2024-04-20T17:11:08.039 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:724b00e5:::benchmark_data_smithi171_78918_object139646:head# 2024-04-20T17:11:08.039 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:724b091e:::benchmark_data_smithi171_78918_object55040:head# 2024-04-20T17:11:08.040 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:724ceffd:::benchmark_data_smithi171_78918_object80204:head# 2024-04-20T17:11:08.040 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:724d1661:::benchmark_data_smithi171_78918_object150235:head# 2024-04-20T17:11:08.040 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:724d4ead:::benchmark_data_smithi171_78918_object122054:head# 2024-04-20T17:11:08.040 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:724d92b9:::benchmark_data_smithi171_78918_object62582:head# 2024-04-20T17:11:08.040 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:724d94ce:::benchmark_data_smithi171_78918_object82914:head# 2024-04-20T17:11:08.040 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:724df54c:::benchmark_data_smithi171_78918_object84918:head# 2024-04-20T17:11:08.040 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:724e8d16:::benchmark_data_smithi171_78918_object122089:head# 2024-04-20T17:11:08.040 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:724f0844:::benchmark_data_smithi171_78918_object136836:head# 2024-04-20T17:11:08.040 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:724fc0bc:::benchmark_data_smithi171_78918_object80596:head# 2024-04-20T17:11:08.040 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72502700:::benchmark_data_smithi171_78918_object90242:head# 2024-04-20T17:11:08.040 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7250db1a:::benchmark_data_smithi171_78918_object140520:head# 2024-04-20T17:11:08.040 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72517694:::benchmark_data_smithi171_78918_object129520:head# 2024-04-20T17:11:08.040 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7251d92f:::benchmark_data_smithi171_78918_object130750:head# 2024-04-20T17:11:08.040 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72525417:::benchmark_data_smithi171_78918_object116331:head# 2024-04-20T17:11:08.041 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7252878a:::benchmark_data_smithi171_78918_object94259:head# 2024-04-20T17:11:08.041 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7253ec42:::benchmark_data_smithi171_78918_object104175:head# 2024-04-20T17:11:08.041 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:725468e5:::benchmark_data_smithi171_78918_object146200:head# 2024-04-20T17:11:08.041 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72553881:::benchmark_data_smithi171_78918_object125761:head# 2024-04-20T17:11:08.042 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72553d2d:::benchmark_data_smithi171_78918_object128579:head# 2024-04-20T17:11:08.042 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7255d513:::benchmark_data_smithi171_78918_object133959:head# 2024-04-20T17:11:08.042 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7255ddef:::benchmark_data_smithi171_78918_object120774:head# 2024-04-20T17:11:08.042 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7255ee52:::benchmark_data_smithi171_78918_object60836:head# 2024-04-20T17:11:08.042 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:725616e1:::benchmark_data_smithi171_78918_object118876:head# 2024-04-20T17:11:08.043 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7256afa3:::benchmark_data_smithi171_78918_object88588:head# 2024-04-20T17:11:08.043 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7258eb9f:::benchmark_data_smithi171_78918_object154265:head# 2024-04-20T17:11:08.043 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72596f9f:::benchmark_data_smithi171_78918_object65524:head# 2024-04-20T17:11:08.043 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:725a7eac:::benchmark_data_smithi171_78918_object72983:head# 2024-04-20T17:11:08.043 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:725b7891:::benchmark_data_smithi171_78918_object71854:head# 2024-04-20T17:11:08.043 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:725bfaea:::benchmark_data_smithi171_78918_object149821:head# 2024-04-20T17:11:08.044 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:725d06a8:::benchmark_data_smithi171_78918_object73178:head# 2024-04-20T17:11:08.044 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:725e64ef:::benchmark_data_smithi171_78918_object76277:head# 2024-04-20T17:11:08.044 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:725f1177:::benchmark_data_smithi171_78918_object110633:head# 2024-04-20T17:11:08.044 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:725f9eb3:::benchmark_data_smithi171_78918_object79127:head# 2024-04-20T17:11:08.044 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:725fd83b:::benchmark_data_smithi171_78918_object119006:head# 2024-04-20T17:11:08.056 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:725ffe2d:::benchmark_data_smithi171_78918_object132767:head# 2024-04-20T17:11:08.056 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7260409d:::benchmark_data_smithi171_78918_object96875:head# 2024-04-20T17:11:08.056 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72614020:::benchmark_data_smithi171_78918_object68553:head# 2024-04-20T17:11:08.056 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7261a39b:::benchmark_data_smithi171_78918_object76278:head# 2024-04-20T17:11:08.056 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72629f83:::benchmark_data_smithi171_78918_object102718:head# 2024-04-20T17:11:08.056 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:726339cb:::benchmark_data_smithi171_78918_object77637:head# 2024-04-20T17:11:08.057 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7263be89:::benchmark_data_smithi171_78918_object83483:head# 2024-04-20T17:11:08.057 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7265206d:::benchmark_data_smithi171_78918_object156240:head# 2024-04-20T17:11:08.057 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72663ae9:::benchmark_data_smithi171_78918_object96626:head# 2024-04-20T17:11:08.057 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:726860c6:::benchmark_data_smithi171_78918_object97945:head# 2024-04-20T17:11:08.057 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:726a09b1:::benchmark_data_smithi171_78918_object136576:head# 2024-04-20T17:11:08.057 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:726b161e:::benchmark_data_smithi171_78918_object141698:head# 2024-04-20T17:11:08.057 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:726c4533:::benchmark_data_smithi171_78918_object64115:head# 2024-04-20T17:11:08.057 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:726cd40d:::benchmark_data_smithi171_78918_object117828:head# 2024-04-20T17:11:08.057 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:726d2a15:::benchmark_data_smithi171_78918_object138629:head# 2024-04-20T17:11:08.057 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:726d2c9a:::benchmark_data_smithi171_78918_object108913:head# 2024-04-20T17:11:08.057 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:726e6fc1:::benchmark_data_smithi171_78918_object159146:head# 2024-04-20T17:11:08.057 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:726eb95a:::benchmark_data_smithi171_78918_object53611:head# 2024-04-20T17:11:08.057 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:726ec7ad:::benchmark_data_smithi171_78918_object76040:head# 2024-04-20T17:11:08.057 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:726f513b:::benchmark_data_smithi171_78918_object125558:head# 2024-04-20T17:11:08.057 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:726f79b9:::benchmark_data_smithi171_78918_object108494:head# 2024-04-20T17:11:08.058 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:726fe853:::benchmark_data_smithi171_78918_object93633:head# 2024-04-20T17:11:08.058 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7270177d:::benchmark_data_smithi171_78918_object124994:head# 2024-04-20T17:11:08.058 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7270eece:::benchmark_data_smithi171_78918_object66350:head# 2024-04-20T17:11:08.058 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7271b1e4:::benchmark_data_smithi171_78918_object119945:head# 2024-04-20T17:11:08.058 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:727340f2:::benchmark_data_smithi171_78918_object127221:head# 2024-04-20T17:11:08.058 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72738d4d:::benchmark_data_smithi171_78918_object110763:head# 2024-04-20T17:11:08.058 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7273c7e2:::benchmark_data_smithi171_78918_object102508:head# 2024-04-20T17:11:08.058 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7274a6a3:::benchmark_data_smithi171_78918_object120104:head# 2024-04-20T17:11:08.058 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7274e51c:::benchmark_data_smithi171_78918_object59053:head# 2024-04-20T17:11:08.058 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72758a51:::benchmark_data_smithi171_78918_object125957:head# 2024-04-20T17:11:08.058 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:727667af:::benchmark_data_smithi171_78918_object130266:head# 2024-04-20T17:11:08.058 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7276b188:::benchmark_data_smithi171_78918_object137179:head# 2024-04-20T17:11:08.058 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:727796a6:::benchmark_data_smithi171_78918_object118095:head# 2024-04-20T17:11:08.058 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72787e03:::benchmark_data_smithi171_78918_object115510:head# 2024-04-20T17:11:08.059 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7279a177:::benchmark_data_smithi171_78918_object63985:head# 2024-04-20T17:11:08.059 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7279e431:::benchmark_data_smithi171_78918_object96358:head# 2024-04-20T17:11:08.059 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:727a0de1:::benchmark_data_smithi171_78918_object157675:head# 2024-04-20T17:11:08.059 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:727b8af0:::benchmark_data_smithi171_78918_object160377:head# 2024-04-20T17:11:08.059 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:727b8dc2:::benchmark_data_smithi171_78918_object115524:head# 2024-04-20T17:11:08.059 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:727bc1d0:::benchmark_data_smithi171_78918_object103648:head# 2024-04-20T17:11:08.059 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:727bfde5:::benchmark_data_smithi171_78918_object121482:head# 2024-04-20T17:11:08.059 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:727c7786:::benchmark_data_smithi171_78918_object156807:head# 2024-04-20T17:11:08.059 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:727cd513:::benchmark_data_smithi171_78918_object108305:head# 2024-04-20T17:11:08.059 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:727d4f05:::benchmark_data_smithi171_78918_object128561:head# 2024-04-20T17:11:08.059 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:727e04b6:::benchmark_data_smithi171_78918_object129241:head# 2024-04-20T17:11:08.059 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:727e19d8:::benchmark_data_smithi171_78918_object120275:head# 2024-04-20T17:11:08.059 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:727e39be:::benchmark_data_smithi171_78918_object160182:head# 2024-04-20T17:11:08.059 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:727e6009:::benchmark_data_smithi171_78918_object159379:head# 2024-04-20T17:11:08.059 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:727ea785:::benchmark_data_smithi171_78918_object129017:head# 2024-04-20T17:11:08.060 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:727f5636:::benchmark_data_smithi171_78918_object76580:head# 2024-04-20T17:11:08.060 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:728097dc:::benchmark_data_smithi171_78918_object127551:head# 2024-04-20T17:11:08.060 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:728114ed:::benchmark_data_smithi171_78918_object155877:head# 2024-04-20T17:11:08.060 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:728195a6:::benchmark_data_smithi171_78918_object83459:head# 2024-04-20T17:11:08.060 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:728223d9:::benchmark_data_smithi171_78918_object160175:head# 2024-04-20T17:11:08.060 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72823666:::benchmark_data_smithi171_78918_object139449:head# 2024-04-20T17:11:08.060 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:728272b8:::benchmark_data_smithi171_78918_object55598:head# 2024-04-20T17:11:08.060 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72836dd4:::benchmark_data_smithi171_78918_object73107:head# 2024-04-20T17:11:08.060 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7283ccf0:::benchmark_data_smithi171_78918_object67259:head# 2024-04-20T17:11:08.060 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7283ee19:::benchmark_data_smithi171_78918_object141072:head# 2024-04-20T17:11:08.060 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7283efdf:::benchmark_data_smithi171_78918_object104077:head# 2024-04-20T17:11:08.060 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72840220:::benchmark_data_smithi171_78918_object114933:head# 2024-04-20T17:11:08.060 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72847ad3:::benchmark_data_smithi171_78918_object59353:head# 2024-04-20T17:11:08.060 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72852496:::benchmark_data_smithi171_78918_object71089:head# 2024-04-20T17:11:08.061 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7287d50f:::benchmark_data_smithi171_78918_object144527:head# 2024-04-20T17:11:08.061 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72880a5e:::benchmark_data_smithi171_78918_object57235:head# 2024-04-20T17:11:08.061 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72880e58:::benchmark_data_smithi171_78918_object122565:head# 2024-04-20T17:11:08.061 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:728810db:::benchmark_data_smithi171_78918_object68079:head# 2024-04-20T17:11:08.061 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72886bf4:::benchmark_data_smithi171_78918_object141631:head# 2024-04-20T17:11:08.065 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7288bc61:::benchmark_data_smithi171_78918_object86199:head# 2024-04-20T17:11:08.065 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72892bdb:::benchmark_data_smithi171_78918_object111324:head# 2024-04-20T17:11:08.065 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7289c783:::benchmark_data_smithi171_78918_object119351:head# 2024-04-20T17:11:08.066 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:728b0af0:::benchmark_data_smithi171_78918_object130269:head# 2024-04-20T17:11:08.066 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:728b0b26:::benchmark_data_smithi171_78918_object95630:head# 2024-04-20T17:11:08.066 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:728b94f9:::benchmark_data_smithi171_78918_object68343:head# 2024-04-20T17:11:08.066 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:728bb3f4:::benchmark_data_smithi171_78918_object110857:head# 2024-04-20T17:11:08.066 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:728c44c5:::benchmark_data_smithi171_78918_object126961:head# 2024-04-20T17:11:08.066 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:728cf480:::benchmark_data_smithi171_78918_object158776:head# 2024-04-20T17:11:08.066 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:728d5d54:::benchmark_data_smithi171_78918_object127607:head# 2024-04-20T17:11:08.066 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:728db7a2:::benchmark_data_smithi171_78918_object87744:head# 2024-04-20T17:11:08.066 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:728e2e95:::benchmark_data_smithi171_78918_object89626:head# 2024-04-20T17:11:08.066 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:728f42df:::benchmark_data_smithi171_78918_object62491:head# 2024-04-20T17:11:08.066 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:728f5b77:::benchmark_data_smithi171_78918_object122837:head# 2024-04-20T17:11:08.066 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:728fdff1:::benchmark_data_smithi171_78918_object104970:head# 2024-04-20T17:11:08.066 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7290304f:::benchmark_data_smithi171_78918_object73879:head# 2024-04-20T17:11:08.066 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7291add7:::benchmark_data_smithi171_78918_object105164:head# 2024-04-20T17:11:08.067 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:729215ad:::benchmark_data_smithi171_78918_object93536:head# 2024-04-20T17:11:08.067 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72932ef7:::benchmark_data_smithi171_78918_object61108:head# 2024-04-20T17:11:08.067 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72933161:::benchmark_data_smithi171_78918_object158683:head# 2024-04-20T17:11:08.067 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72944392:::benchmark_data_smithi171_78918_object70057:head# 2024-04-20T17:11:08.067 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7294b554:::benchmark_data_smithi171_78918_object54947:head# 2024-04-20T17:11:08.067 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7294cebe:::benchmark_data_smithi171_78918_object101391:head# 2024-04-20T17:11:08.067 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72950266:::benchmark_data_smithi171_78918_object160566:head# 2024-04-20T17:11:08.067 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72955856:::benchmark_data_smithi171_78918_object117172:head# 2024-04-20T17:11:08.067 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72970965:::benchmark_data_smithi171_78918_object101666:head# 2024-04-20T17:11:08.067 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72976361:::benchmark_data_smithi171_78918_object55914:head# 2024-04-20T17:11:08.067 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72988258:::benchmark_data_smithi171_78918_object145220:head# 2024-04-20T17:11:08.067 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72989d3d:::benchmark_data_smithi171_78918_object81225:head# 2024-04-20T17:11:08.067 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72990b71:::benchmark_data_smithi171_78918_object151624:head# 2024-04-20T17:11:08.068 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7299381d:::benchmark_data_smithi171_78918_object69896:head# 2024-04-20T17:11:08.068 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:729acf53:::benchmark_data_smithi171_78918_object71269:head# 2024-04-20T17:11:08.068 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:729bb5f2:::benchmark_data_smithi171_78918_object74961:head# 2024-04-20T17:11:08.068 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:729c0581:::benchmark_data_smithi171_78918_object54491:head# 2024-04-20T17:11:08.068 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:729ca71f:::benchmark_data_smithi171_78918_object56343:head# 2024-04-20T17:11:08.068 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:729d2c72:::benchmark_data_smithi171_78918_object111085:head# 2024-04-20T17:11:08.068 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:729d7e36:::benchmark_data_smithi171_78918_object135996:head# 2024-04-20T17:11:08.068 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:729ea3aa:::benchmark_data_smithi171_78918_object61322:head# 2024-04-20T17:11:08.068 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72a0a6de:::benchmark_data_smithi171_78918_object123196:head# 2024-04-20T17:11:08.068 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72a0b5eb:::benchmark_data_smithi171_78918_object136497:head# 2024-04-20T17:11:08.068 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72a0bb09:::benchmark_data_smithi171_78918_object130250:head# 2024-04-20T17:11:08.068 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72a1093b:::benchmark_data_smithi171_78918_object118122:head# 2024-04-20T17:11:08.068 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72a1f273:::benchmark_data_smithi171_78918_object118302:head# 2024-04-20T17:11:08.068 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72a33681:::benchmark_data_smithi171_78918_object143155:head# 2024-04-20T17:11:08.074 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72a35796:::benchmark_data_smithi171_78918_object147075:head# 2024-04-20T17:11:08.074 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72a359c9:::benchmark_data_smithi171_78918_object104127:head# 2024-04-20T17:11:08.074 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72a4b8a5:::benchmark_data_smithi171_78918_object57088:head# 2024-04-20T17:11:08.075 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72a4fe82:::benchmark_data_smithi171_78918_object98538:head# 2024-04-20T17:11:08.075 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72a50362:::benchmark_data_smithi171_78918_object114123:head# 2024-04-20T17:11:08.075 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72a5d7d5:::benchmark_data_smithi171_78918_object101176:head# 2024-04-20T17:11:08.075 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72a640fe:::benchmark_data_smithi171_78918_object88085:head# 2024-04-20T17:11:08.075 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72a6dfb3:::benchmark_data_smithi171_78918_object122948:head# 2024-04-20T17:11:08.075 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72a7372c:::benchmark_data_smithi171_78918_object122517:head# 2024-04-20T17:11:08.075 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72a79fb6:::benchmark_data_smithi171_78918_object58370:head# 2024-04-20T17:11:08.075 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72a85389:::benchmark_data_smithi171_78918_object68002:head# 2024-04-20T17:11:08.075 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72a89c5f:::benchmark_data_smithi171_78918_object141107:head# 2024-04-20T17:11:08.075 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72a8bb3a:::benchmark_data_smithi171_78918_object156949:head# 2024-04-20T17:11:08.075 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72a8f445:::benchmark_data_smithi171_78918_object96912:head# 2024-04-20T17:11:08.075 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72a9ca71:::benchmark_data_smithi171_78918_object135430:head# 2024-04-20T17:11:08.075 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72ab056c:::benchmark_data_smithi171_78918_object118537:head# 2024-04-20T17:11:08.075 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72abe8a8:::benchmark_data_smithi171_78918_object104351:head# 2024-04-20T17:11:08.076 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72abece7:::benchmark_data_smithi171_78918_object114653:head# 2024-04-20T17:11:08.076 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72acb1bd:::benchmark_data_smithi171_78918_object79173:head# 2024-04-20T17:11:08.076 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72afb955:::benchmark_data_smithi171_78918_object108609:head# 2024-04-20T17:11:08.076 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72aff883:::benchmark_data_smithi171_78918_object132440:head# 2024-04-20T17:11:08.076 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72b0fb87:::benchmark_data_smithi171_78918_object139556:head# 2024-04-20T17:11:08.076 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72b1b618:::benchmark_data_smithi171_78918_object123821:head# 2024-04-20T17:11:08.076 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72b211a2:::benchmark_data_smithi171_78918_object55423:head# 2024-04-20T17:11:08.076 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72b2cf83:::benchmark_data_smithi171_78918_object155005:head# 2024-04-20T17:11:08.076 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72b3c365:::benchmark_data_smithi171_78918_object97227:head# 2024-04-20T17:11:08.076 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72b3ca42:::benchmark_data_smithi171_78918_object127634:head# 2024-04-20T17:11:08.076 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72b3f38e:::benchmark_data_smithi171_78918_object142858:head# 2024-04-20T17:11:08.076 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72b4d287:::benchmark_data_smithi171_78918_object67143:head# 2024-04-20T17:11:08.076 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72b53f2e:::benchmark_data_smithi171_78918_object54815:head# 2024-04-20T17:11:08.076 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72b55974:::benchmark_data_smithi171_78918_object86965:head# 2024-04-20T17:11:08.076 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72b63372:::benchmark_data_smithi171_78918_object110610:head# 2024-04-20T17:11:08.077 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72b6f02e:::benchmark_data_smithi171_78918_object122070:head# 2024-04-20T17:11:08.077 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72b83a72:::benchmark_data_smithi171_78918_object138546:head# 2024-04-20T17:11:08.077 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72b849c8:::benchmark_data_smithi171_78918_object159837:head# 2024-04-20T17:11:08.077 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72b909dd:::benchmark_data_smithi171_78918_object86912:head# 2024-04-20T17:11:08.077 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72b98bed:::benchmark_data_smithi171_78918_object123414:head# 2024-04-20T17:11:08.077 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72ba003a:::benchmark_data_smithi171_78918_object122308:head# 2024-04-20T17:11:08.077 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72ba0acf:::benchmark_data_smithi171_78918_object132025:head# 2024-04-20T17:11:08.077 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72ba4434:::benchmark_data_smithi171_78918_object53859:head# 2024-04-20T17:11:08.077 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72bb040f:::benchmark_data_smithi171_78918_object76380:head# 2024-04-20T17:11:08.077 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72bb1b62:::benchmark_data_smithi171_78918_object111887:head# 2024-04-20T17:11:08.077 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72bbdd4a:::benchmark_data_smithi171_78918_object79185:head# 2024-04-20T17:11:08.077 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72bbf0a8:::benchmark_data_smithi171_78918_object67688:head# 2024-04-20T17:11:08.077 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72bc3fbd:::benchmark_data_smithi171_78918_object90502:head# 2024-04-20T17:11:08.077 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72bcd598:::benchmark_data_smithi171_78918_object97991:head# 2024-04-20T17:11:08.084 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72bcf3af:::benchmark_data_smithi171_78918_object133688:head# 2024-04-20T17:11:08.084 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72bd3845:::benchmark_data_smithi171_78918_object75394:head# 2024-04-20T17:11:08.084 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72bd9ed8:::benchmark_data_smithi171_78918_object82068:head# 2024-04-20T17:11:08.084 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72becce9:::benchmark_data_smithi171_78918_object111030:head# 2024-04-20T17:11:08.084 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72bf70dc:::benchmark_data_smithi171_78918_object139271:head# 2024-04-20T17:11:08.085 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72bfb760:::benchmark_data_smithi171_78918_object112217:head# 2024-04-20T17:11:08.085 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72bfed41:::benchmark_data_smithi171_78918_object54523:head# 2024-04-20T17:11:08.085 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72c0adfb:::benchmark_data_smithi171_78918_object60693:head# 2024-04-20T17:11:08.085 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72c1126b:::benchmark_data_smithi171_78918_object73976:head# 2024-04-20T17:11:08.085 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72c13c90:::benchmark_data_smithi171_78918_object117241:head# 2024-04-20T17:11:08.085 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72c30fe5:::benchmark_data_smithi171_78918_object83009:head# 2024-04-20T17:11:08.085 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72c3af9f:::benchmark_data_smithi171_78918_object136707:head# 2024-04-20T17:11:08.085 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72c5d6a4:::benchmark_data_smithi171_78918_object104688:head# 2024-04-20T17:11:08.085 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72c5fce5:::benchmark_data_smithi171_78918_object117692:head# 2024-04-20T17:11:08.085 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72c712df:::benchmark_data_smithi171_78918_object80046:head# 2024-04-20T17:11:08.085 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72c72918:::benchmark_data_smithi171_78918_object80774:head# 2024-04-20T17:11:08.085 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72c73e47:::benchmark_data_smithi171_78918_object155212:head# 2024-04-20T17:11:08.085 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72c80a93:::benchmark_data_smithi171_78918_object118433:head# 2024-04-20T17:11:08.085 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72c93063:::benchmark_data_smithi171_78918_object72724:head# 2024-04-20T17:11:08.085 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72c98bf0:::benchmark_data_smithi171_78918_object87390:head# 2024-04-20T17:11:08.085 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72c99eea:::benchmark_data_smithi171_78918_object128550:head# 2024-04-20T17:11:08.086 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72c9a08d:::benchmark_data_smithi171_78918_object121428:head# 2024-04-20T17:11:08.086 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72ca43c5:::benchmark_data_smithi171_78918_object75927:head# 2024-04-20T17:11:08.086 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72ca6ac1:::benchmark_data_smithi171_78918_object107860:head# 2024-04-20T17:11:08.086 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72cb5db3:::benchmark_data_smithi171_78918_object129075:head# 2024-04-20T17:11:08.086 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72ccd005:::benchmark_data_smithi171_78918_object51456:head# 2024-04-20T17:11:08.086 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72cd2df5:::benchmark_data_smithi171_78918_object129186:head# 2024-04-20T17:11:08.086 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72ced765:::benchmark_data_smithi171_78918_object134086:head# 2024-04-20T17:11:08.086 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72cf6268:::benchmark_data_smithi171_78918_object101185:head# 2024-04-20T17:11:08.086 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72cfcde2:::benchmark_data_smithi171_78918_object147000:head# 2024-04-20T17:11:08.086 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72d00b8c:::benchmark_data_smithi171_78918_object51452:head# 2024-04-20T17:11:08.086 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72d09203:::benchmark_data_smithi171_78918_object149742:head# 2024-04-20T17:11:08.086 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72d09543:::benchmark_data_smithi171_78918_object97397:head# 2024-04-20T17:11:08.086 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72d0fb16:::benchmark_data_smithi171_78918_object107787:head# 2024-04-20T17:11:08.086 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72d12cfc:::benchmark_data_smithi171_78918_object117944:head# 2024-04-20T17:11:08.087 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72d272f3:::benchmark_data_smithi171_78918_object156082:head# 2024-04-20T17:11:08.087 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72d3cb09:::benchmark_data_smithi171_78918_object148333:head# 2024-04-20T17:11:08.087 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72d45d5d:::benchmark_data_smithi171_78918_object123176:head# 2024-04-20T17:11:08.087 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72d5109d:::benchmark_data_smithi171_78918_object159054:head# 2024-04-20T17:11:08.087 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72d5124b:::benchmark_data_smithi171_78918_object113876:head# 2024-04-20T17:11:08.087 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72d5b074:::benchmark_data_smithi171_78918_object109748:head# 2024-04-20T17:11:08.087 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72d6713b:::benchmark_data_smithi171_78918_object131663:head# 2024-04-20T17:11:08.087 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72d70c3e:::benchmark_data_smithi171_78918_object58960:head# 2024-04-20T17:11:08.087 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72d73571:::benchmark_data_smithi171_78918_object106769:head# 2024-04-20T17:11:08.087 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72d75f82:::benchmark_data_smithi171_78918_object64389:head# 2024-04-20T17:11:08.087 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72d784b7:::benchmark_data_smithi171_78918_object70069:head# 2024-04-20T17:11:08.087 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72d839da:::benchmark_data_smithi171_78918_object69557:head# 2024-04-20T17:11:08.087 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72d8c84f:::benchmark_data_smithi171_78918_object150826:head# 2024-04-20T17:11:08.087 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72d93cb3:::benchmark_data_smithi171_78918_object119379:head# 2024-04-20T17:11:08.087 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72d97ee1:::benchmark_data_smithi171_78918_object73629:head# 2024-04-20T17:11:08.088 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72d9b55d:::benchmark_data_smithi171_78918_object76928:head# 2024-04-20T17:11:08.088 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72dc0c3f:::benchmark_data_smithi171_78918_object76022:head# 2024-04-20T17:11:08.088 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72dc5f38:::benchmark_data_smithi171_78918_object104913:head# 2024-04-20T17:11:08.088 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72ddd330:::benchmark_data_smithi171_78918_object99935:head# 2024-04-20T17:11:08.088 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72de2110:::benchmark_data_smithi171_78918_object126845:head# 2024-04-20T17:11:08.092 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72de4812:::benchmark_data_smithi171_78918_object82592:head# 2024-04-20T17:11:08.093 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72de7ca8:::benchmark_data_smithi171_78918_object56085:head# 2024-04-20T17:11:08.093 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72df0b53:::benchmark_data_smithi171_78918_object134005:head# 2024-04-20T17:11:08.093 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72dfa472:::benchmark_data_smithi171_78918_object147166:head# 2024-04-20T17:11:08.093 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72e03efc:::benchmark_data_smithi171_78918_object141695:head# 2024-04-20T17:11:08.093 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72e1512e:::benchmark_data_smithi171_78918_object84985:head# 2024-04-20T17:11:08.093 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72e240a8:::benchmark_data_smithi171_78918_object154514:head# 2024-04-20T17:11:08.093 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72e30fd9:::benchmark_data_smithi171_78918_object118524:head# 2024-04-20T17:11:08.093 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72e39b95:::benchmark_data_smithi171_78918_object123458:head# 2024-04-20T17:11:08.093 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72e48e31:::benchmark_data_smithi171_78918_object99464:head# 2024-04-20T17:11:08.093 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72e550e6:::benchmark_data_smithi171_78918_object132064:head# 2024-04-20T17:11:08.093 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72e57c93:::benchmark_data_smithi171_78918_object89481:head# 2024-04-20T17:11:08.093 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72e580f6:::benchmark_data_smithi171_78918_object148687:head# 2024-04-20T17:11:08.093 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72e59554:::benchmark_data_smithi171_78918_object69728:head# 2024-04-20T17:11:08.094 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72e59a15:::benchmark_data_smithi171_78918_object138120:head# 2024-04-20T17:11:08.094 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72e705d9:::benchmark_data_smithi171_78918_object72839:head# 2024-04-20T17:11:08.094 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72e745d3:::benchmark_data_smithi171_78918_object127355:head# 2024-04-20T17:11:08.094 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72e7ad9c:::benchmark_data_smithi171_78918_object109445:head# 2024-04-20T17:11:08.094 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72e7c8b0:::benchmark_data_smithi171_78918_object122480:head# 2024-04-20T17:11:08.094 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72e7de25:::benchmark_data_smithi171_78918_object64191:head# 2024-04-20T17:11:08.094 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72e822b7:::benchmark_data_smithi171_78918_object158711:head# 2024-04-20T17:11:08.094 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72e88075:::benchmark_data_smithi171_78918_object137155:head# 2024-04-20T17:11:08.094 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72e9848b:::benchmark_data_smithi171_78918_object103486:head# 2024-04-20T17:11:08.094 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72ea09f3:::benchmark_data_smithi171_78918_object84019:head# 2024-04-20T17:11:08.094 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72ea461b:::benchmark_data_smithi171_78918_object130086:head# 2024-04-20T17:11:08.094 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72ea66af:::benchmark_data_smithi171_78918_object143179:head# 2024-04-20T17:11:08.094 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72eb458b:::benchmark_data_smithi171_78918_object109798:head# 2024-04-20T17:11:08.094 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72eb551d:::benchmark_data_smithi171_78918_object136697:head# 2024-04-20T17:11:08.095 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72eb6667:::benchmark_data_smithi171_78918_object160952:head# 2024-04-20T17:11:08.095 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72eb6ab0:::benchmark_data_smithi171_78918_object128184:head# 2024-04-20T17:11:08.095 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72ec0b70:::benchmark_data_smithi171_78918_object71635:head# 2024-04-20T17:11:08.095 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72ec9c10:::benchmark_data_smithi171_78918_object57275:head# 2024-04-20T17:11:08.095 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72ed2a31:::benchmark_data_smithi171_78918_object129881:head# 2024-04-20T17:11:08.095 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72edc490:::benchmark_data_smithi171_78918_object120778:head# 2024-04-20T17:11:08.095 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72ee1b26:::benchmark_data_smithi171_78918_object131573:head# 2024-04-20T17:11:08.095 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72ee8d95:::benchmark_data_smithi171_78918_object139794:head# 2024-04-20T17:11:08.095 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72eeab4e:::benchmark_data_smithi171_78918_object75445:head# 2024-04-20T17:11:08.095 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72eed0b3:::benchmark_data_smithi171_78918_object147122:head# 2024-04-20T17:11:08.095 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72eed494:::benchmark_data_smithi171_78918_object115495:head# 2024-04-20T17:11:08.095 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72ef09b9:::benchmark_data_smithi171_78918_object83724:head# 2024-04-20T17:11:08.095 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72ef6e01:::benchmark_data_smithi171_78918_object116691:head# 2024-04-20T17:11:08.095 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72ef9dd8:::benchmark_data_smithi171_78918_object86285:head# 2024-04-20T17:11:08.095 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72f0111f:::benchmark_data_smithi171_78918_object143094:head# 2024-04-20T17:11:08.096 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72f037cd:::benchmark_data_smithi171_78918_object64314:head# 2024-04-20T17:11:08.096 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72f05231:::benchmark_data_smithi171_78918_object62507:head# 2024-04-20T17:11:08.096 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72f05717:::benchmark_data_smithi171_78918_object147317:head# 2024-04-20T17:11:08.096 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72f16e03:::benchmark_data_smithi171_78918_object114189:head# 2024-04-20T17:11:08.099 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72f1df68:::benchmark_data_smithi171_78918_object68227:head# 2024-04-20T17:11:08.100 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72f246a4:::benchmark_data_smithi171_78918_object118860:head# 2024-04-20T17:11:08.100 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72f291c8:::benchmark_data_smithi171_78918_object144201:head# 2024-04-20T17:11:08.100 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72f2d3d6:::benchmark_data_smithi171_78918_object104617:head# 2024-04-20T17:11:08.100 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72f2ea09:::benchmark_data_smithi171_78918_object146174:head# 2024-04-20T17:11:08.100 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72f3e163:::benchmark_data_smithi171_78918_object135396:head# 2024-04-20T17:11:08.100 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72f46221:::benchmark_data_smithi171_78918_object116513:head# 2024-04-20T17:11:08.100 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72f4c798:::benchmark_data_smithi171_78918_object112751:head# 2024-04-20T17:11:08.100 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72f5243e:::benchmark_data_smithi171_78918_object124570:head# 2024-04-20T17:11:08.100 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72f55d44:::benchmark_data_smithi171_78918_object149698:head# 2024-04-20T17:11:08.100 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72f56c07:::benchmark_data_smithi171_78918_object86375:head# 2024-04-20T17:11:08.100 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72f5bbb0:::benchmark_data_smithi171_78918_object67248:head# 2024-04-20T17:11:08.100 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72f67b69:::benchmark_data_smithi171_78918_object147380:head# 2024-04-20T17:11:08.100 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72f6aa21:::benchmark_data_smithi171_78918_object100459:head# 2024-04-20T17:11:08.100 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72f72d89:::benchmark_data_smithi171_78918_object60379:head# 2024-04-20T17:11:08.101 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72f8262e:::benchmark_data_smithi171_78918_object113372:head# 2024-04-20T17:11:08.101 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72f99926:::benchmark_data_smithi171_78918_object103885:head# 2024-04-20T17:11:08.101 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72f99cf2:::benchmark_data_smithi171_78918_object149134:head# 2024-04-20T17:11:08.101 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72f9b9be:::benchmark_data_smithi171_78918_object59808:head# 2024-04-20T17:11:08.101 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72f9c107:::benchmark_data_smithi171_78918_object157961:head# 2024-04-20T17:11:08.101 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72f9e505:::benchmark_data_smithi171_78918_object118934:head# 2024-04-20T17:11:08.101 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72fa83d4:::benchmark_data_smithi171_78918_object146791:head# 2024-04-20T17:11:08.101 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72fb7264:::benchmark_data_smithi171_78918_object123976:head# 2024-04-20T17:11:08.101 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72fc87be:::benchmark_data_smithi171_78918_object70487:head# 2024-04-20T17:11:08.101 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72fcaeab:::benchmark_data_smithi171_78918_object147264:head# 2024-04-20T17:11:08.101 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72fcf39b:::benchmark_data_smithi171_78918_object79166:head# 2024-04-20T17:11:08.101 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:72ffb630:::benchmark_data_smithi171_78918_object110042:head# 2024-04-20T17:11:08.101 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73005442:::benchmark_data_smithi171_78918_object131424:head# 2024-04-20T17:11:08.101 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7300949d:::benchmark_data_smithi171_78918_object147602:head# 2024-04-20T17:11:08.101 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7300bcc1:::benchmark_data_smithi171_78918_object134679:head# 2024-04-20T17:11:08.102 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7300c87c:::benchmark_data_smithi171_78918_object123437:head# 2024-04-20T17:11:08.102 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7300d1f4:::benchmark_data_smithi171_78918_object69890:head# 2024-04-20T17:11:08.102 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7300ed16:::benchmark_data_smithi171_78918_object143812:head# 2024-04-20T17:11:08.102 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7301399d:::benchmark_data_smithi171_78918_object68721:head# 2024-04-20T17:11:08.102 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7301aaba:::benchmark_data_smithi171_78918_object98887:head# 2024-04-20T17:11:08.102 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73020be9:::benchmark_data_smithi171_78918_object118918:head# 2024-04-20T17:11:08.102 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73025101:::benchmark_data_smithi171_78918_object83692:head# 2024-04-20T17:11:08.102 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:730284a1:::benchmark_data_smithi171_78918_object75129:head# 2024-04-20T17:11:08.102 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7302c336:::benchmark_data_smithi171_78918_object54644:head# 2024-04-20T17:11:08.110 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73043eea:::benchmark_data_smithi171_78918_object134074:head# 2024-04-20T17:11:08.110 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7304d64d:::benchmark_data_smithi171_78918_object129718:head# 2024-04-20T17:11:08.110 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73051b06:::benchmark_data_smithi171_78918_object139925:head# 2024-04-20T17:11:08.110 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7305e53c:::benchmark_data_smithi171_78918_object156307:head# 2024-04-20T17:11:08.110 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:730617c0:::benchmark_data_smithi171_78918_object160571:head# 2024-04-20T17:11:08.110 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:730637f3:::benchmark_data_smithi171_78918_object74833:head# 2024-04-20T17:11:08.110 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73064f7d:::benchmark_data_smithi171_78918_object73937:head# 2024-04-20T17:11:08.110 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7306a98e:::benchmark_data_smithi171_78918_object123342:head# 2024-04-20T17:11:08.110 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7306c35c:::benchmark_data_smithi171_78918_object79986:head# 2024-04-20T17:11:08.110 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73070152:::benchmark_data_smithi171_78918_object154371:head# 2024-04-20T17:11:08.111 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:730710de:::benchmark_data_smithi171_78918_object63642:head# 2024-04-20T17:11:08.111 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:730724a6:::benchmark_data_smithi171_78918_object76154:head# 2024-04-20T17:11:08.111 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73077eda:::benchmark_data_smithi171_78918_object94878:head# 2024-04-20T17:11:08.111 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7307847f:::benchmark_data_smithi171_78918_object145326:head# 2024-04-20T17:11:08.111 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73078ad9:::benchmark_data_smithi171_78918_object160289:head# 2024-04-20T17:11:08.111 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73079185:::benchmark_data_smithi171_78918_object56424:head# 2024-04-20T17:11:08.111 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7307da5a:::benchmark_data_smithi171_78918_object152168:head# 2024-04-20T17:11:08.111 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:730899fe:::benchmark_data_smithi171_78918_object63920:head# 2024-04-20T17:11:08.111 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:730965e6:::benchmark_data_smithi171_78918_object98274:head# 2024-04-20T17:11:08.111 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73099698:::benchmark_data_smithi171_78918_object53273:head# 2024-04-20T17:11:08.111 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7309a94f:::benchmark_data_smithi171_78918_object62332:head# 2024-04-20T17:11:08.111 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:730ab928:::benchmark_data_smithi171_78918_object57106:head# 2024-04-20T17:11:08.111 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:730c69ed:::benchmark_data_smithi171_78918_object154464:head# 2024-04-20T17:11:08.111 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:730cecd7:::benchmark_data_smithi171_78918_object137565:head# 2024-04-20T17:11:08.111 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:730cfeb5:::benchmark_data_smithi171_78918_object78946:head# 2024-04-20T17:11:08.112 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:730e2391:::benchmark_data_smithi171_78918_object124407:head# 2024-04-20T17:11:08.112 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:730e3cf1:::benchmark_data_smithi171_78918_object80707:head# 2024-04-20T17:11:08.112 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:730f75d6:::benchmark_data_smithi171_78918_object77025:head# 2024-04-20T17:11:08.112 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:730fcb6f:::benchmark_data_smithi171_78918_object153377:head# 2024-04-20T17:11:08.112 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73109b31:::benchmark_data_smithi171_78918_object60660:head# 2024-04-20T17:11:08.112 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7310c4f5:::benchmark_data_smithi171_78918_object130766:head# 2024-04-20T17:11:08.112 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73112da4:::benchmark_data_smithi171_78918_object147246:head# 2024-04-20T17:11:08.112 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7311c2b5:::benchmark_data_smithi171_78918_object70036:head# 2024-04-20T17:11:08.112 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73125c2e:::benchmark_data_smithi171_78918_object127050:head# 2024-04-20T17:11:08.112 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7312c906:::benchmark_data_smithi171_78918_object103023:head# 2024-04-20T17:11:08.112 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73136903:::benchmark_data_smithi171_78918_object87294:head# 2024-04-20T17:11:08.112 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7314c03c:::benchmark_data_smithi171_78918_object112182:head# 2024-04-20T17:11:08.112 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7314ef58:::benchmark_data_smithi171_78918_object56966:head# 2024-04-20T17:11:08.112 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73160d03:::benchmark_data_smithi171_78918_object84279:head# 2024-04-20T17:11:08.112 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:731627f1:::benchmark_data_smithi171_78918_object130975:head# 2024-04-20T17:11:08.113 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73174b47:::benchmark_data_smithi171_78918_object88173:head# 2024-04-20T17:11:08.113 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7317b6e1:::benchmark_data_smithi171_78918_object151441:head# 2024-04-20T17:11:08.113 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7318b28d:::benchmark_data_smithi171_78918_object142578:head# 2024-04-20T17:11:08.113 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73191a35:::benchmark_data_smithi171_78918_object109053:head# 2024-04-20T17:11:08.113 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:731ab6b8:::benchmark_data_smithi171_78918_object142782:head# 2024-04-20T17:11:08.113 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:731b3e0b:::benchmark_data_smithi171_78918_object129176:head# 2024-04-20T17:11:08.113 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:731c59a0:::benchmark_data_smithi171_78918_object107167:head# 2024-04-20T17:11:08.113 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:731cef08:::benchmark_data_smithi171_78918_object100659:head# 2024-04-20T17:11:08.113 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:731cfc66:::benchmark_data_smithi171_78918_object158585:head# 2024-04-20T17:11:08.113 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:731dd245:::benchmark_data_smithi171_78918_object84607:head# 2024-04-20T17:11:08.113 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:731e00b4:::benchmark_data_smithi171_78918_object101162:head# 2024-04-20T17:11:08.113 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:731e2eab:::benchmark_data_smithi171_78918_object128183:head# 2024-04-20T17:11:08.113 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:731e7844:::benchmark_data_smithi171_78918_object119073:head# 2024-04-20T17:11:08.113 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:731e7f44:::benchmark_data_smithi171_78918_object148588:head# 2024-04-20T17:11:08.114 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73205fdc:::benchmark_data_smithi171_78918_object121743:head# 2024-04-20T17:11:08.114 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7320b01e:::benchmark_data_smithi171_78918_object81523:head# 2024-04-20T17:11:08.114 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7320fec9:::benchmark_data_smithi171_78918_object158820:head# 2024-04-20T17:11:08.114 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73220dd2:::benchmark_data_smithi171_78918_object54350:head# 2024-04-20T17:11:08.114 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7322ac4a:::benchmark_data_smithi171_78918_object68425:head# 2024-04-20T17:11:08.114 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7323c4ad:::benchmark_data_smithi171_78918_object148397:head# 2024-04-20T17:11:08.114 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7323d0ad:::benchmark_data_smithi171_78918_object151350:head# 2024-04-20T17:11:08.120 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73246b1d:::benchmark_data_smithi171_78918_object58790:head# 2024-04-20T17:11:08.120 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7325df28:::benchmark_data_smithi171_78918_object125097:head# 2024-04-20T17:11:08.120 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73270a41:::benchmark_data_smithi171_78918_object84064:head# 2024-04-20T17:11:08.120 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73278b08:::benchmark_data_smithi171_78918_object59858:head# 2024-04-20T17:11:08.120 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73289fef:::benchmark_data_smithi171_78918_object86777:head# 2024-04-20T17:11:08.120 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7328e1d0:::benchmark_data_smithi171_78918_object55065:head# 2024-04-20T17:11:08.120 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7329b5a0:::benchmark_data_smithi171_78918_object126678:head# 2024-04-20T17:11:08.120 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:732b5387:::benchmark_data_smithi171_78918_object129591:head# 2024-04-20T17:11:08.120 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:732c3db0:::benchmark_data_smithi171_78918_object71487:head# 2024-04-20T17:11:08.120 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:732c477d:::benchmark_data_smithi171_78918_object124219:head# 2024-04-20T17:11:08.120 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:732c8659:::benchmark_data_smithi171_78918_object103341:head# 2024-04-20T17:11:08.120 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:732d22c3:::benchmark_data_smithi171_78918_object122506:head# 2024-04-20T17:11:08.120 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:732dce03:::benchmark_data_smithi171_78918_object156352:head# 2024-04-20T17:11:08.120 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:732f52d3:::benchmark_data_smithi171_78918_object106776:head# 2024-04-20T17:11:08.121 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:732f5456:::benchmark_data_smithi171_78918_object138500:head# 2024-04-20T17:11:08.121 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73307f98:::benchmark_data_smithi171_78918_object97204:head# 2024-04-20T17:11:08.121 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7330f967:::benchmark_data_smithi171_78918_object66399:head# 2024-04-20T17:11:08.121 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73310747:::benchmark_data_smithi171_78918_object62011:head# 2024-04-20T17:11:08.121 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:733109cd:::benchmark_data_smithi171_78918_object118500:head# 2024-04-20T17:11:08.121 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7331efba:::benchmark_data_smithi171_78918_object118376:head# 2024-04-20T17:11:08.121 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:733234e7:::benchmark_data_smithi171_78918_object117183:head# 2024-04-20T17:11:08.121 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73339021:::benchmark_data_smithi171_78918_object68460:head# 2024-04-20T17:11:08.121 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7333a0e7:::benchmark_data_smithi171_78918_object121436:head# 2024-04-20T17:11:08.121 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7334d79b:::benchmark_data_smithi171_78918_object134774:head# 2024-04-20T17:11:08.121 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73352710:::benchmark_data_smithi171_78918_object103193:head# 2024-04-20T17:11:08.121 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7335dbaf:::benchmark_data_smithi171_78918_object94889:head# 2024-04-20T17:11:08.121 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73369cd4:::benchmark_data_smithi171_78918_object71991:head# 2024-04-20T17:11:08.121 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7336ce98:::benchmark_data_smithi171_78918_object101285:head# 2024-04-20T17:11:08.121 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73384911:::benchmark_data_smithi171_78918_object113148:head# 2024-04-20T17:11:08.122 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7338d367:::benchmark_data_smithi171_78918_object108167:head# 2024-04-20T17:11:08.122 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:733a9edb:::benchmark_data_smithi171_78918_object54178:head# 2024-04-20T17:11:08.122 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:733aecb2:::benchmark_data_smithi171_78918_object75359:head# 2024-04-20T17:11:08.122 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:733b18d6:::benchmark_data_smithi171_78918_object72529:head# 2024-04-20T17:11:08.122 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:733bfa14:::benchmark_data_smithi171_78918_object99798:head# 2024-04-20T17:11:08.122 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:733d22be:::benchmark_data_smithi171_78918_object82016:head# 2024-04-20T17:11:08.122 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:733f023c:::benchmark_data_smithi171_78918_object149612:head# 2024-04-20T17:11:08.122 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:733f88d7:::benchmark_data_smithi171_78918_object143988:head# 2024-04-20T17:11:08.122 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:734038f6:::benchmark_data_smithi171_78918_object131579:head# 2024-04-20T17:11:08.122 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7340394c:::benchmark_data_smithi171_78918_object93965:head# 2024-04-20T17:11:08.122 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7340a1ce:::benchmark_data_smithi171_78918_object95549:head# 2024-04-20T17:11:08.122 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:734136ac:::benchmark_data_smithi171_78918_object93653:head# 2024-04-20T17:11:08.122 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73418b4e:::benchmark_data_smithi171_78918_object60803:head# 2024-04-20T17:11:08.122 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7344aefb:::benchmark_data_smithi171_78918_object160045:head# 2024-04-20T17:11:08.123 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7344f0ba:::benchmark_data_smithi171_78918_object87305:head# 2024-04-20T17:11:08.123 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7345384b:::benchmark_data_smithi171_78918_object82823:head# 2024-04-20T17:11:08.123 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:734629f7:::benchmark_data_smithi171_78918_object131738:head# 2024-04-20T17:11:08.123 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:734716c5:::benchmark_data_smithi171_78918_object95752:head# 2024-04-20T17:11:08.123 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73476f04:::benchmark_data_smithi171_78918_object63786:head# 2024-04-20T17:11:08.123 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73487d77:::benchmark_data_smithi171_78918_object109248:head# 2024-04-20T17:11:08.123 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7348e1b9:::benchmark_data_smithi171_78918_object80454:head# 2024-04-20T17:11:08.123 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7349a6e1:::benchmark_data_smithi171_78918_object56714:head# 2024-04-20T17:11:08.123 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7349b422:::benchmark_data_smithi171_78918_object108537:head# 2024-04-20T17:11:08.123 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7349bd71:::benchmark_data_smithi171_78918_object62081:head# 2024-04-20T17:11:08.123 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7349f87d:::benchmark_data_smithi171_78918_object106523:head# 2024-04-20T17:11:08.123 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:734a21ce:::benchmark_data_smithi171_78918_object68306:head# 2024-04-20T17:11:08.123 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:734a9ed9:::benchmark_data_smithi171_78918_object129545:head# 2024-04-20T17:11:08.123 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:734ace02:::benchmark_data_smithi171_78918_object128529:head# 2024-04-20T17:11:08.124 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:734b6d60:::benchmark_data_smithi171_78918_object76086:head# 2024-04-20T17:11:08.124 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:734b834d:::benchmark_data_smithi171_78918_object60502:head# 2024-04-20T17:11:08.124 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:734bdaf3:::benchmark_data_smithi171_78918_object100500:head# 2024-04-20T17:11:08.131 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:734c17e6:::benchmark_data_smithi171_78918_object80739:head# 2024-04-20T17:11:08.131 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:734c5f0d:::benchmark_data_smithi171_78918_object150772:head# 2024-04-20T17:11:08.131 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:734d6320:::benchmark_data_smithi171_78918_object114442:head# 2024-04-20T17:11:08.131 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:734d90ff:::benchmark_data_smithi171_78918_object93338:head# 2024-04-20T17:11:08.131 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:734da1a7:::benchmark_data_smithi171_78918_object113925:head# 2024-04-20T17:11:08.131 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:734df947:::benchmark_data_smithi171_78918_object155946:head# 2024-04-20T17:11:08.131 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:734e9497:::benchmark_data_smithi171_78918_object120149:head# 2024-04-20T17:11:08.131 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:734f527b:::benchmark_data_smithi171_78918_object51671:head# 2024-04-20T17:11:08.131 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73500e01:::benchmark_data_smithi171_78918_object157659:head# 2024-04-20T17:11:08.131 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7350e328:::benchmark_data_smithi171_78918_object73117:head# 2024-04-20T17:11:08.131 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:735225d0:::benchmark_data_smithi171_78918_object100254:head# 2024-04-20T17:11:08.132 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7352b82f:::benchmark_data_smithi171_78918_object74813:head# 2024-04-20T17:11:08.132 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:735305af:::benchmark_data_smithi171_78918_object153081:head# 2024-04-20T17:11:08.132 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73547d7f:::benchmark_data_smithi171_78918_object104793:head# 2024-04-20T17:11:08.132 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73559900:::benchmark_data_smithi171_78918_object134101:head# 2024-04-20T17:11:08.132 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73564887:::benchmark_data_smithi171_78918_object134276:head# 2024-04-20T17:11:08.132 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:735702d4:::benchmark_data_smithi171_78918_object90478:head# 2024-04-20T17:11:08.132 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:735708cb:::benchmark_data_smithi171_78918_object83744:head# 2024-04-20T17:11:08.132 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:735735f6:::benchmark_data_smithi171_78918_object54878:head# 2024-04-20T17:11:08.132 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73573989:::benchmark_data_smithi171_78918_object83823:head# 2024-04-20T17:11:08.132 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7357a715:::benchmark_data_smithi171_78918_object62732:head# 2024-04-20T17:11:08.132 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:735b3019:::benchmark_data_smithi171_78918_object59656:head# 2024-04-20T17:11:08.132 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:735b3816:::benchmark_data_smithi171_78918_object117032:head# 2024-04-20T17:11:08.132 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:735b876c:::benchmark_data_smithi171_78918_object68020:head# 2024-04-20T17:11:08.132 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:735baef5:::benchmark_data_smithi171_78918_object86531:head# 2024-04-20T17:11:08.133 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:735c9353:::benchmark_data_smithi171_78918_object86966:head# 2024-04-20T17:11:08.133 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:735cce66:::benchmark_data_smithi171_78918_object72844:head# 2024-04-20T17:11:08.133 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:735cfd29:::benchmark_data_smithi171_78918_object124479:head# 2024-04-20T17:11:08.133 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:735f2854:::benchmark_data_smithi171_78918_object141786:head# 2024-04-20T17:11:08.133 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73603bcf:::benchmark_data_smithi171_78918_object133258:head# 2024-04-20T17:11:08.133 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73616717:::benchmark_data_smithi171_78918_object58296:head# 2024-04-20T17:11:08.133 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7361fe70:::benchmark_data_smithi171_78918_object67894:head# 2024-04-20T17:11:08.133 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7362f9d4:::benchmark_data_smithi171_78918_object54332:head# 2024-04-20T17:11:08.133 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73635320:::benchmark_data_smithi171_78918_object129414:head# 2024-04-20T17:11:08.133 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:736446b0:::benchmark_data_smithi171_78918_object60607:head# 2024-04-20T17:11:08.133 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7365583b:::benchmark_data_smithi171_78918_object98516:head# 2024-04-20T17:11:08.133 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73664dcc:::benchmark_data_smithi171_78918_object89303:head# 2024-04-20T17:11:08.133 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7366c0dd:::benchmark_data_smithi171_78918_object125830:head# 2024-04-20T17:11:08.133 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:736796bf:::benchmark_data_smithi171_78918_object120842:head# 2024-04-20T17:11:08.133 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7368964e:::benchmark_data_smithi171_78918_object124638:head# 2024-04-20T17:11:08.134 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73694728:::benchmark_data_smithi171_78918_object138765:head# 2024-04-20T17:11:08.134 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73695d8c:::benchmark_data_smithi171_78918_object127821:head# 2024-04-20T17:11:08.134 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7369da22:::benchmark_data_smithi171_78918_object85621:head# 2024-04-20T17:11:08.134 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:736a2395:::benchmark_data_smithi171_78918_object87542:head# 2024-04-20T17:11:08.134 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:736a2ebb:::benchmark_data_smithi171_78918_object145310:head# 2024-04-20T17:11:08.134 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:736a440c:::benchmark_data_smithi171_78918_object130477:head# 2024-04-20T17:11:08.134 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:736a64eb:::benchmark_data_smithi171_78918_object60424:head# 2024-04-20T17:11:08.134 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:736acfaf:::benchmark_data_smithi171_78918_object73725:head# 2024-04-20T17:11:08.134 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:736d00e3:::benchmark_data_smithi171_78918_object75586:head# 2024-04-20T17:11:08.134 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:736db14b:::benchmark_data_smithi171_78918_object112720:head# 2024-04-20T17:11:08.134 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:736fcb0f:::benchmark_data_smithi171_78918_object69650:head# 2024-04-20T17:11:08.134 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:736fe145:::benchmark_data_smithi171_78918_object100640:head# 2024-04-20T17:11:08.134 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:736fe98a:::benchmark_data_smithi171_78918_object120245:head# 2024-04-20T17:11:08.134 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73701813:::benchmark_data_smithi171_78918_object82687:head# 2024-04-20T17:11:08.135 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7370772b:::benchmark_data_smithi171_78918_object96180:head# 2024-04-20T17:11:08.135 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73711ce7:::benchmark_data_smithi171_78918_object135167:head# 2024-04-20T17:11:08.135 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7371719b:::benchmark_data_smithi171_78918_object120281:head# 2024-04-20T17:11:08.135 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7371e766:::benchmark_data_smithi171_78918_object159611:head# 2024-04-20T17:11:08.135 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73727a28:::benchmark_data_smithi171_78918_object69126:head# 2024-04-20T17:11:08.135 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73730481:::benchmark_data_smithi171_78918_object156493:head# 2024-04-20T17:11:08.135 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73733cc8:::benchmark_data_smithi171_78918_object112738:head# 2024-04-20T17:11:08.135 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73744807:::benchmark_data_smithi171_78918_object63301:head# 2024-04-20T17:11:08.135 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7374ea32:::benchmark_data_smithi171_78918_object65041:head# 2024-04-20T17:11:08.135 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7374ff84:::benchmark_data_smithi171_78918_object59469:head# 2024-04-20T17:11:08.135 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73750abd:::benchmark_data_smithi171_78918_object149989:head# 2024-04-20T17:11:08.135 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73750deb:::benchmark_data_smithi171_78918_object68380:head# 2024-04-20T17:11:08.135 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73759a27:::benchmark_data_smithi171_78918_object103096:head# 2024-04-20T17:11:08.135 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7375dd2e:::benchmark_data_smithi171_78918_object106554:head# 2024-04-20T17:11:08.143 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73781ca3:::benchmark_data_smithi171_78918_object154348:head# 2024-04-20T17:11:08.143 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:737856ba:::benchmark_data_smithi171_78918_object90433:head# 2024-04-20T17:11:08.143 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:737894b5:::benchmark_data_smithi171_78918_object131966:head# 2024-04-20T17:11:08.144 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7379026a:::benchmark_data_smithi171_78918_object93012:head# 2024-04-20T17:11:08.144 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73794d9a:::benchmark_data_smithi171_78918_object71298:head# 2024-04-20T17:11:08.144 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:737962af:::benchmark_data_smithi171_78918_object100376:head# 2024-04-20T17:11:08.144 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:737a1c49:::benchmark_data_smithi171_78918_object95138:head# 2024-04-20T17:11:08.144 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:737b3a78:::benchmark_data_smithi171_78918_object85372:head# 2024-04-20T17:11:08.144 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:737c18d8:::benchmark_data_smithi171_78918_object133647:head# 2024-04-20T17:11:08.144 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:737c2031:::benchmark_data_smithi171_78918_object152606:head# 2024-04-20T17:11:08.144 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:737c5589:::benchmark_data_smithi171_78918_object145838:head# 2024-04-20T17:11:08.144 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:737c73a7:::benchmark_data_smithi171_78918_object106175:head# 2024-04-20T17:11:08.144 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:737d0527:::benchmark_data_smithi171_78918_object150310:head# 2024-04-20T17:11:08.144 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:737d4b74:::benchmark_data_smithi171_78918_object85442:head# 2024-04-20T17:11:08.144 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:737df269:::benchmark_data_smithi171_78918_object141993:head# 2024-04-20T17:11:08.144 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:737e906b:::benchmark_data_smithi171_78918_object86480:head# 2024-04-20T17:11:08.144 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:737f8d48:::benchmark_data_smithi171_78918_object60930:head# 2024-04-20T17:11:08.144 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73805bbd:::benchmark_data_smithi171_78918_object81512:head# 2024-04-20T17:11:08.145 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7381c4d0:::benchmark_data_smithi171_78918_object102487:head# 2024-04-20T17:11:08.145 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73820c2a:::benchmark_data_smithi171_78918_object132395:head# 2024-04-20T17:11:08.145 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73825588:::benchmark_data_smithi171_78918_object135552:head# 2024-04-20T17:11:08.145 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73833545:::benchmark_data_smithi171_78918_object123540:head# 2024-04-20T17:11:08.145 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:738366a8:::benchmark_data_smithi171_78918_object158278:head# 2024-04-20T17:11:08.145 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:738385cd:::benchmark_data_smithi171_78918_object99859:head# 2024-04-20T17:11:08.145 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7384f90c:::benchmark_data_smithi171_78918_object127896:head# 2024-04-20T17:11:08.145 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:738613a1:::benchmark_data_smithi171_78918_object70324:head# 2024-04-20T17:11:08.145 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73868a66:::benchmark_data_smithi171_78918_object144874:head# 2024-04-20T17:11:08.145 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73878f2b:::benchmark_data_smithi171_78918_object79587:head# 2024-04-20T17:11:08.145 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7387cbc1:::benchmark_data_smithi171_78918_object138594:head# 2024-04-20T17:11:08.145 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73886388:::benchmark_data_smithi171_78918_object100999:head# 2024-04-20T17:11:08.145 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:738a0c21:::benchmark_data_smithi171_78918_object95239:head# 2024-04-20T17:11:08.145 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:738a3aa0:::benchmark_data_smithi171_78918_object90161:head# 2024-04-20T17:11:08.145 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:738ae46d:::benchmark_data_smithi171_78918_object111695:head# 2024-04-20T17:11:08.146 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:738bd814:::benchmark_data_smithi171_78918_object82818:head# 2024-04-20T17:11:08.146 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:738c5651:::benchmark_data_smithi171_78918_object106847:head# 2024-04-20T17:11:08.146 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:738ce44e:::benchmark_data_smithi171_78918_object147771:head# 2024-04-20T17:11:08.146 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:738d9641:::benchmark_data_smithi171_78918_object85558:head# 2024-04-20T17:11:08.146 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:738e430d:::benchmark_data_smithi171_78918_object111290:head# 2024-04-20T17:11:08.146 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:738efe76:::benchmark_data_smithi171_78918_object104510:head# 2024-04-20T17:11:08.146 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:738f0ec2:::benchmark_data_smithi171_78918_object138237:head# 2024-04-20T17:11:08.146 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:738f43b6:::benchmark_data_smithi171_78918_object100367:head# 2024-04-20T17:11:08.146 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:738f5083:::benchmark_data_smithi171_78918_object158191:head# 2024-04-20T17:11:08.146 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:738fb3ed:::benchmark_data_smithi171_78918_object152718:head# 2024-04-20T17:11:08.146 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7390693c:::benchmark_data_smithi171_78918_object130003:head# 2024-04-20T17:11:08.146 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73910acb:::benchmark_data_smithi171_78918_object78264:head# 2024-04-20T17:11:08.146 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73913e77:::benchmark_data_smithi171_78918_object124387:head# 2024-04-20T17:11:08.146 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73917d76:::benchmark_data_smithi171_78918_object56496:head# 2024-04-20T17:11:08.147 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:739265ed:::benchmark_data_smithi171_78918_object132413:head# 2024-04-20T17:11:08.147 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7392a318:::benchmark_data_smithi171_78918_object128788:head# 2024-04-20T17:11:08.147 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7392c255:::benchmark_data_smithi171_78918_object86954:head# 2024-04-20T17:11:08.147 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:739334d3:::benchmark_data_smithi171_78918_object57728:head# 2024-04-20T17:11:08.147 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73934db4:::benchmark_data_smithi171_78918_object125534:head# 2024-04-20T17:11:08.147 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73937499:::benchmark_data_smithi171_78918_object112982:head# 2024-04-20T17:11:08.147 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7394e831:::benchmark_data_smithi171_78918_object124329:head# 2024-04-20T17:11:08.147 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73950e06:::benchmark_data_smithi171_78918_object91252:head# 2024-04-20T17:11:08.147 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7396494e:::benchmark_data_smithi171_78918_object63159:head# 2024-04-20T17:11:08.147 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73967d99:::benchmark_data_smithi171_78918_object93117:head# 2024-04-20T17:11:08.147 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73970ddc:::benchmark_data_smithi171_78918_object140726:head# 2024-04-20T17:11:08.147 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7397db0a:::benchmark_data_smithi171_78918_object146409:head# 2024-04-20T17:11:08.147 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73982d5c:::benchmark_data_smithi171_78918_object114662:head# 2024-04-20T17:11:08.147 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:739837b1:::benchmark_data_smithi171_78918_object112285:head# 2024-04-20T17:11:08.148 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:739856fe:::benchmark_data_smithi171_78918_object94085:head# 2024-04-20T17:11:08.148 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7399e7ad:::benchmark_data_smithi171_78918_object96022:head# 2024-04-20T17:11:08.148 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:739a130c:::benchmark_data_smithi171_78918_object64568:head# 2024-04-20T17:11:08.148 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:739a45a8:::benchmark_data_smithi171_78918_object140593:head# 2024-04-20T17:11:08.148 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:739a710f:::benchmark_data_smithi171_78918_object84823:head# 2024-04-20T17:11:08.148 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:739abdfa:::benchmark_data_smithi171_78918_object115076:head# 2024-04-20T17:11:08.148 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:739c126a:::benchmark_data_smithi171_78918_object135399:head# 2024-04-20T17:11:08.148 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:739c27c8:::benchmark_data_smithi171_78918_object115543:head# 2024-04-20T17:11:08.158 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:739d4609:::benchmark_data_smithi171_78918_object121926:head# 2024-04-20T17:11:08.158 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:739d8a3e:::benchmark_data_smithi171_78918_object95517:head# 2024-04-20T17:11:08.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:739ecb02:::benchmark_data_smithi171_78918_object62098:head# 2024-04-20T17:11:08.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73a02272:::benchmark_data_smithi171_78918_object77817:head# 2024-04-20T17:11:08.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73a09637:::benchmark_data_smithi171_78918_object148651:head# 2024-04-20T17:11:08.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73a0b22a:::benchmark_data_smithi171_78918_object89332:head# 2024-04-20T17:11:08.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73a12c89:::benchmark_data_smithi171_78918_object93854:head# 2024-04-20T17:11:08.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73a1b01b:::benchmark_data_smithi171_78918_object111825:head# 2024-04-20T17:11:08.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73a1b8dd:::benchmark_data_smithi171_78918_object58361:head# 2024-04-20T17:11:08.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73a25690:::benchmark_data_smithi171_78918_object86003:head# 2024-04-20T17:11:08.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73a2d121:::benchmark_data_smithi171_78918_object152944:head# 2024-04-20T17:11:08.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73a3000e:::benchmark_data_smithi171_78918_object90243:head# 2024-04-20T17:11:08.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73a3a4b8:::benchmark_data_smithi171_78918_object132087:head# 2024-04-20T17:11:08.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73a42317:::benchmark_data_smithi171_78918_object89949:head# 2024-04-20T17:11:08.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73a44673:::benchmark_data_smithi171_78918_object133264:head# 2024-04-20T17:11:08.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73a4530b:::benchmark_data_smithi171_78918_object91760:head# 2024-04-20T17:11:08.160 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73a52de6:::benchmark_data_smithi171_78918_object54213:head# 2024-04-20T17:11:08.160 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73a5859b:::benchmark_data_smithi171_78918_object53574:head# 2024-04-20T17:11:08.160 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73a6b8c6:::benchmark_data_smithi171_78918_object85281:head# 2024-04-20T17:11:08.160 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73a6d086:::benchmark_data_smithi171_78918_object111188:head# 2024-04-20T17:11:08.160 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73a859da:::benchmark_data_smithi171_78918_object62499:head# 2024-04-20T17:11:08.160 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73a89798:::benchmark_data_smithi171_78918_object56507:head# 2024-04-20T17:11:08.160 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73a8f40c:::benchmark_data_smithi171_78918_object152689:head# 2024-04-20T17:11:08.160 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73a90159:::benchmark_data_smithi171_78918_object108629:head# 2024-04-20T17:11:08.160 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73aa34a5:::benchmark_data_smithi171_78918_object127812:head# 2024-04-20T17:11:08.160 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73aa6525:::benchmark_data_smithi171_78918_object105726:head# 2024-04-20T17:11:08.160 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73ab4eaa:::benchmark_data_smithi171_78918_object144738:head# 2024-04-20T17:11:08.160 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73abd773:::benchmark_data_smithi171_78918_object66263:head# 2024-04-20T17:11:08.160 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73abe10c:::benchmark_data_smithi171_78918_object102250:head# 2024-04-20T17:11:08.160 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73abe92e:::benchmark_data_smithi171_78918_object69847:head# 2024-04-20T17:11:08.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73aca2a3:::benchmark_data_smithi171_78918_object154412:head# 2024-04-20T17:11:08.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73acc083:::benchmark_data_smithi171_78918_object87307:head# 2024-04-20T17:11:08.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73acd694:::benchmark_data_smithi171_78918_object53413:head# 2024-04-20T17:11:08.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73ad6e3d:::benchmark_data_smithi171_78918_object121032:head# 2024-04-20T17:11:08.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73ae2ffa:::benchmark_data_smithi171_78918_object102561:head# 2024-04-20T17:11:08.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73aea62c:::benchmark_data_smithi171_78918_object90547:head# 2024-04-20T17:11:08.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73aea9cb:::benchmark_data_smithi171_78918_object103154:head# 2024-04-20T17:11:08.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73aece78:::benchmark_data_smithi171_78918_object89057:head# 2024-04-20T17:11:08.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73af7af3:::benchmark_data_smithi171_78918_object130361:head# 2024-04-20T17:11:08.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73af85e0:::benchmark_data_smithi171_78918_object54562:head# 2024-04-20T17:11:08.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73af8cd7:::benchmark_data_smithi171_78918_object157352:head# 2024-04-20T17:11:08.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73afe5a2:::benchmark_data_smithi171_78918_object133725:head# 2024-04-20T17:11:08.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73affd67:::benchmark_data_smithi171_78918_object75502:head# 2024-04-20T17:11:08.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73b03c71:::benchmark_data_smithi171_78918_object152080:head# 2024-04-20T17:11:08.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73b0a918:::benchmark_data_smithi171_78918_object148017:head# 2024-04-20T17:11:08.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73b1d5da:::benchmark_data_smithi171_78918_object101080:head# 2024-04-20T17:11:08.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73b278cc:::benchmark_data_smithi171_78918_object67537:head# 2024-04-20T17:11:08.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73b2a9bf:::benchmark_data_smithi171_78918_object61810:head# 2024-04-20T17:11:08.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73b3adf8:::benchmark_data_smithi171_78918_object154633:head# 2024-04-20T17:11:08.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73b3e157:::benchmark_data_smithi171_78918_object94481:head# 2024-04-20T17:11:08.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73b3e693:::benchmark_data_smithi171_78918_object147713:head# 2024-04-20T17:11:08.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73b3f920:::benchmark_data_smithi171_78918_object145402:head# 2024-04-20T17:11:08.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73b42231:::benchmark_data_smithi171_78918_object100572:head# 2024-04-20T17:11:08.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73b4d6be:::benchmark_data_smithi171_78918_object90363:head# 2024-04-20T17:11:08.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73b50096:::benchmark_data_smithi171_78918_object113980:head# 2024-04-20T17:11:08.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73b5d5b1:::benchmark_data_smithi171_78918_object56172:head# 2024-04-20T17:11:08.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73b6b59c:::benchmark_data_smithi171_78918_object84595:head# 2024-04-20T17:11:08.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73b8649d:::benchmark_data_smithi171_78918_object99006:head# 2024-04-20T17:11:08.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73b888f0:::benchmark_data_smithi171_78918_object160277:head# 2024-04-20T17:11:08.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73b94c1a:::benchmark_data_smithi171_78918_object136987:head# 2024-04-20T17:11:08.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73b97d0f:::benchmark_data_smithi171_78918_object63650:head# 2024-04-20T17:11:08.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73baa460:::benchmark_data_smithi171_78918_object129583:head# 2024-04-20T17:11:08.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73babced:::benchmark_data_smithi171_78918_object149372:head# 2024-04-20T17:11:08.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73bb1f43:::benchmark_data_smithi171_78918_object110817:head# 2024-04-20T17:11:08.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73bc011f:::benchmark_data_smithi171_78918_object154765:head# 2024-04-20T17:11:08.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73bc30cb:::benchmark_data_smithi171_78918_object68854:head# 2024-04-20T17:11:08.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73bc59fc:::benchmark_data_smithi171_78918_object122885:head# 2024-04-20T17:11:08.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73bca4e6:::benchmark_data_smithi171_78918_object140808:head# 2024-04-20T17:11:08.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73bce7ef:::benchmark_data_smithi171_78918_object64488:head# 2024-04-20T17:11:08.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73bd70d5:::benchmark_data_smithi171_78918_object137561:head# 2024-04-20T17:11:08.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73bf1239:::benchmark_data_smithi171_78918_object87853:head# 2024-04-20T17:11:08.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73bf8efd:::benchmark_data_smithi171_78918_object56285:head# 2024-04-20T17:11:08.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73c0360d:::benchmark_data_smithi171_78918_object84832:head# 2024-04-20T17:11:08.164 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73c2cbdd:::benchmark_data_smithi171_78918_object148217:head# 2024-04-20T17:11:08.164 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73c3abcb:::benchmark_data_smithi171_78918_object111286:head# 2024-04-20T17:11:08.164 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73c40044:::benchmark_data_smithi171_78918_object65697:head# 2024-04-20T17:11:08.168 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73c41de4:::benchmark_data_smithi171_78918_object58535:head# 2024-04-20T17:11:08.168 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73c43acc:::benchmark_data_smithi171_78918_object140380:head# 2024-04-20T17:11:08.168 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73c45c1e:::benchmark_data_smithi171_78918_object62218:head# 2024-04-20T17:11:08.168 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73c51702:::benchmark_data_smithi171_78918_object135216:head# 2024-04-20T17:11:08.168 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73c5f191:::benchmark_data_smithi171_78918_object98381:head# 2024-04-20T17:11:08.168 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73c6dc5f:::benchmark_data_smithi171_78918_object101683:head# 2024-04-20T17:11:08.168 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73c74bd8:::benchmark_data_smithi171_78918_object107826:head# 2024-04-20T17:11:08.168 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73c78f74:::benchmark_data_smithi171_78918_object83994:head# 2024-04-20T17:11:08.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73c7df6c:::benchmark_data_smithi171_78918_object79194:head# 2024-04-20T17:11:08.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73c7e4f7:::benchmark_data_smithi171_78918_object120559:head# 2024-04-20T17:11:08.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73c7fe0a:::benchmark_data_smithi171_78918_object159851:head# 2024-04-20T17:11:08.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73c9f4d7:::benchmark_data_smithi171_78918_object133326:head# 2024-04-20T17:11:08.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73ca5bec:::benchmark_data_smithi171_78918_object66846:head# 2024-04-20T17:11:08.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73cadb34:::benchmark_data_smithi171_78918_object75660:head# 2024-04-20T17:11:08.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73cb52db:::benchmark_data_smithi171_78918_object148072:head# 2024-04-20T17:11:08.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73cc5f15:::benchmark_data_smithi171_78918_object136600:head# 2024-04-20T17:11:08.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73cd272a:::benchmark_data_smithi171_78918_object92787:head# 2024-04-20T17:11:08.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73cd9fe2:::benchmark_data_smithi171_78918_object125394:head# 2024-04-20T17:11:08.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73ce0eac:::benchmark_data_smithi171_78918_object93621:head# 2024-04-20T17:11:08.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73ce1e48:::benchmark_data_smithi171_78918_object118295:head# 2024-04-20T17:11:08.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73cf0b21:::benchmark_data_smithi171_78918_object135701:head# 2024-04-20T17:11:08.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73d06ccc:::benchmark_data_smithi171_78918_object143453:head# 2024-04-20T17:11:08.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73d12cca:::benchmark_data_smithi171_78918_object144364:head# 2024-04-20T17:11:08.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73d205fc:::benchmark_data_smithi171_78918_object150387:head# 2024-04-20T17:11:08.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73d320cb:::benchmark_data_smithi171_78918_object148871:head# 2024-04-20T17:11:08.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73d37b1c:::benchmark_data_smithi171_78918_object58514:head# 2024-04-20T17:11:08.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73d492f4:::benchmark_data_smithi171_78918_object117570:head# 2024-04-20T17:11:08.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73d4b4ae:::benchmark_data_smithi171_78918_object149632:head# 2024-04-20T17:11:08.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73d55c3b:::benchmark_data_smithi171_78918_object109151:head# 2024-04-20T17:11:08.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73d650bc:::benchmark_data_smithi171_78918_object99009:head# 2024-04-20T17:11:08.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73d75289:::benchmark_data_smithi171_78918_object116159:head# 2024-04-20T17:11:08.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73d7602b:::benchmark_data_smithi171_78918_object149697:head# 2024-04-20T17:11:08.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73d784ff:::benchmark_data_smithi171_78918_object93837:head# 2024-04-20T17:11:08.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73d7b463:::benchmark_data_smithi171_78918_object146432:head# 2024-04-20T17:11:08.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73d82718:::benchmark_data_smithi171_78918_object88681:head# 2024-04-20T17:11:08.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73d8c1f1:::benchmark_data_smithi171_78918_object83464:head# 2024-04-20T17:11:08.171 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73da0e4e:::benchmark_data_smithi171_78918_object122829:head# 2024-04-20T17:11:08.171 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73da590b:::benchmark_data_smithi171_78918_object71544:head# 2024-04-20T17:11:08.171 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73db1b0b:::benchmark_data_smithi171_78918_object118947:head# 2024-04-20T17:11:08.171 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73db8371:::benchmark_data_smithi171_78918_object64853:head# 2024-04-20T17:11:08.171 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73dcc0a6:::benchmark_data_smithi171_78918_object75823:head# 2024-04-20T17:11:08.171 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73dd3b71:::benchmark_data_smithi171_78918_object140740:head# 2024-04-20T17:11:08.177 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73ddd2a8:::benchmark_data_smithi171_78918_object66506:head# 2024-04-20T17:11:08.177 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73dee512:::benchmark_data_smithi171_78918_object136751:head# 2024-04-20T17:11:08.177 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73df9c7e:::benchmark_data_smithi171_78918_object123593:head# 2024-04-20T17:11:08.177 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73dfe46b:::benchmark_data_smithi171_78918_object56999:head# 2024-04-20T17:11:08.177 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73e01080:::benchmark_data_smithi171_78918_object121834:head# 2024-04-20T17:11:08.177 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73e01c91:::benchmark_data_smithi171_78918_object125992:head# 2024-04-20T17:11:08.177 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73e01dad:::benchmark_data_smithi171_78918_object66975:head# 2024-04-20T17:11:08.177 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73e0b831:::benchmark_data_smithi171_78918_object160615:head# 2024-04-20T17:11:08.177 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73e0c351:::benchmark_data_smithi171_78918_object51774:head# 2024-04-20T17:11:08.177 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73e0f793:::benchmark_data_smithi171_78918_object94124:head# 2024-04-20T17:11:08.177 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73e203ff:::benchmark_data_smithi171_78918_object55803:head# 2024-04-20T17:11:08.177 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73e27e1e:::benchmark_data_smithi171_78918_object51533:head# 2024-04-20T17:11:08.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73e3b2a3:::benchmark_data_smithi171_78918_object131529:head# 2024-04-20T17:11:08.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73e41f41:::benchmark_data_smithi171_78918_object129304:head# 2024-04-20T17:11:08.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73e4561f:::benchmark_data_smithi171_78918_object80845:head# 2024-04-20T17:11:08.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73e48826:::benchmark_data_smithi171_78918_object138739:head# 2024-04-20T17:11:08.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73e58cdc:::benchmark_data_smithi171_78918_object125234:head# 2024-04-20T17:11:08.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73e5b915:::benchmark_data_smithi171_78918_object52789:head# 2024-04-20T17:11:08.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73e5dbcc:::benchmark_data_smithi171_78918_object149497:head# 2024-04-20T17:11:08.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73e6cbbf:::benchmark_data_smithi171_78918_object125134:head# 2024-04-20T17:11:08.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73e741fa:::benchmark_data_smithi171_78918_object101501:head# 2024-04-20T17:11:08.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73e922e2:::benchmark_data_smithi171_78918_object105581:head# 2024-04-20T17:11:08.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73e96c41:::benchmark_data_smithi171_78918_object145877:head# 2024-04-20T17:11:08.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73e9bc80:::benchmark_data_smithi171_78918_object94112:head# 2024-04-20T17:11:08.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73ea35fa:::benchmark_data_smithi171_78918_object57791:head# 2024-04-20T17:11:08.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73eb572f:::benchmark_data_smithi171_78918_object112278:head# 2024-04-20T17:11:08.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73ec33c8:::benchmark_data_smithi171_78918_object56610:head# 2024-04-20T17:11:08.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73ec3df1:::benchmark_data_smithi171_78918_object62858:head# 2024-04-20T17:11:08.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73ec932c:::benchmark_data_smithi171_78918_object53299:head# 2024-04-20T17:11:08.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73ec9e5d:::benchmark_data_smithi171_78918_object97365:head# 2024-04-20T17:11:08.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73edae88:::benchmark_data_smithi171_78918_object56203:head# 2024-04-20T17:11:08.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73edf052:::benchmark_data_smithi171_78918_object56324:head# 2024-04-20T17:11:08.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73ee8746:::benchmark_data_smithi171_78918_object119710:head# 2024-04-20T17:11:08.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73ef4d80:::benchmark_data_smithi171_78918_object98483:head# 2024-04-20T17:11:08.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73efe43d:::benchmark_data_smithi171_78918_object55379:head# 2024-04-20T17:11:08.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73f02128:::benchmark_data_smithi171_78918_object158325:head# 2024-04-20T17:11:08.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73f23b84:::benchmark_data_smithi171_78918_object156565:head# 2024-04-20T17:11:08.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73f27569:::benchmark_data_smithi171_78918_object59773:head# 2024-04-20T17:11:08.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73f3864d:::benchmark_data_smithi171_78918_object78425:head# 2024-04-20T17:11:08.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73f45050:::benchmark_data_smithi171_78918_object57431:head# 2024-04-20T17:11:08.180 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73f4686a:::benchmark_data_smithi171_78918_object71051:head# 2024-04-20T17:11:08.180 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73f527da:::benchmark_data_smithi171_78918_object82323:head# 2024-04-20T17:11:08.180 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73f584b3:::benchmark_data_smithi171_78918_object131190:head# 2024-04-20T17:11:08.180 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73f63191:::benchmark_data_smithi171_78918_object141322:head# 2024-04-20T17:11:08.180 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73f6c025:::benchmark_data_smithi171_78918_object59087:head# 2024-04-20T17:11:08.180 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73f6f3d3:::benchmark_data_smithi171_78918_object51579:head# 2024-04-20T17:11:08.180 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73f785c1:::benchmark_data_smithi171_78918_object80996:head# 2024-04-20T17:11:08.180 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73f84908:::benchmark_data_smithi171_78918_object118088:head# 2024-04-20T17:11:08.180 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73f87028:::benchmark_data_smithi171_78918_object133277:head# 2024-04-20T17:11:08.180 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73f916cd:::benchmark_data_smithi171_78918_object160121:head# 2024-04-20T17:11:08.180 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73f9df22:::benchmark_data_smithi171_78918_object116506:head# 2024-04-20T17:11:08.185 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73fa2614:::benchmark_data_smithi171_78918_object157491:head# 2024-04-20T17:11:08.186 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73fa749e:::benchmark_data_smithi171_78918_object65303:head# 2024-04-20T17:11:08.186 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73fa9c9f:::benchmark_data_smithi171_78918_object137763:head# 2024-04-20T17:11:08.186 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73fb6daa:::benchmark_data_smithi171_78918_object152007:head# 2024-04-20T17:11:08.186 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73fc592c:::benchmark_data_smithi171_78918_object111670:head# 2024-04-20T17:11:08.186 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73fcb9f2:::benchmark_data_smithi171_78918_object149973:head# 2024-04-20T17:11:08.186 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73fd93bc:::benchmark_data_smithi171_78918_object91318:head# 2024-04-20T17:11:08.186 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73fdbdd2:::benchmark_data_smithi171_78918_object140795:head# 2024-04-20T17:11:08.186 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73fdd745:::benchmark_data_smithi171_78918_object84519:head# 2024-04-20T17:11:08.186 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73fdd9aa:::benchmark_data_smithi171_78918_object121558:head# 2024-04-20T17:11:08.186 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:73ff4132:::benchmark_data_smithi171_78918_object148394:head# 2024-04-20T17:11:08.186 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74013248:::benchmark_data_smithi171_78918_object113743:head# 2024-04-20T17:11:08.186 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7401397f:::benchmark_data_smithi171_78918_object110567:head# 2024-04-20T17:11:08.186 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7401bb58:::benchmark_data_smithi171_78918_object87279:head# 2024-04-20T17:11:08.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:740366a8:::benchmark_data_smithi171_78918_object66637:head# 2024-04-20T17:11:08.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74037d5b:::benchmark_data_smithi171_78918_object99130:head# 2024-04-20T17:11:08.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7403b18e:::benchmark_data_smithi171_78918_object134562:head# 2024-04-20T17:11:08.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7403c8fe:::benchmark_data_smithi171_78918_object60492:head# 2024-04-20T17:11:08.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7403d9db:::benchmark_data_smithi171_78918_object156986:head# 2024-04-20T17:11:08.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74040df3:::benchmark_data_smithi171_78918_object153448:head# 2024-04-20T17:11:08.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74047381:::benchmark_data_smithi171_78918_object103426:head# 2024-04-20T17:11:08.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74051395:::benchmark_data_smithi171_78918_object103749:head# 2024-04-20T17:11:08.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7405151d:::benchmark_data_smithi171_78918_object51816:head# 2024-04-20T17:11:08.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74054e0a:::benchmark_data_smithi171_78918_object117063:head# 2024-04-20T17:11:08.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7405850a:::benchmark_data_smithi171_78918_object56404:head# 2024-04-20T17:11:08.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7407ddb5:::benchmark_data_smithi171_78918_object91488:head# 2024-04-20T17:11:08.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7409ab7c:::benchmark_data_smithi171_78918_object154149:head# 2024-04-20T17:11:08.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7409bbf9:::benchmark_data_smithi171_78918_object82759:head# 2024-04-20T17:11:08.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:740abb3a:::benchmark_data_smithi171_78918_object80159:head# 2024-04-20T17:11:08.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:740bd7c6:::benchmark_data_smithi171_78918_object137601:head# 2024-04-20T17:11:08.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:740bfa5d:::benchmark_data_smithi171_78918_object146167:head# 2024-04-20T17:11:08.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:740c8547:::benchmark_data_smithi171_78918_object129007:head# 2024-04-20T17:11:08.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:740d3435:::benchmark_data_smithi171_78918_object72050:head# 2024-04-20T17:11:08.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:740e3e45:::benchmark_data_smithi171_78918_object152352:head# 2024-04-20T17:11:08.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:740ef6b7:::benchmark_data_smithi171_78918_object86987:head# 2024-04-20T17:11:08.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:740f77e3:::benchmark_data_smithi171_78918_object97277:head# 2024-04-20T17:11:08.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:740fb200:::benchmark_data_smithi171_78918_object82242:head# 2024-04-20T17:11:08.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:740ffc42:::benchmark_data_smithi171_78918_object141990:head# 2024-04-20T17:11:08.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:741019ae:::benchmark_data_smithi171_78918_object79876:head# 2024-04-20T17:11:08.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74101e22:::benchmark_data_smithi171_78918_object137499:head# 2024-04-20T17:11:08.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74104489:::benchmark_data_smithi171_78918_object80122:head# 2024-04-20T17:11:08.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74111ece:::benchmark_data_smithi171_78918_object65646:head# 2024-04-20T17:11:08.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74115881:::benchmark_data_smithi171_78918_object138381:head# 2024-04-20T17:11:08.189 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74121f97:::benchmark_data_smithi171_78918_object114984:head# 2024-04-20T17:11:08.189 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74122053:::benchmark_data_smithi171_78918_object147691:head# 2024-04-20T17:11:08.189 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74128053:::benchmark_data_smithi171_78918_object67386:head# 2024-04-20T17:11:08.189 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7413e3c9:::benchmark_data_smithi171_78918_object103163:head# 2024-04-20T17:11:08.189 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74147234:::benchmark_data_smithi171_78918_object57249:head# 2024-04-20T17:11:08.189 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7414ac16:::benchmark_data_smithi171_78918_object94812:head# 2024-04-20T17:11:08.193 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74155dd3:::benchmark_data_smithi171_78918_object52287:head# 2024-04-20T17:11:08.193 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:741568dd:::benchmark_data_smithi171_78918_object53129:head# 2024-04-20T17:11:08.193 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74157f19:::benchmark_data_smithi171_78918_object57894:head# 2024-04-20T17:11:08.193 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74169e54:::benchmark_data_smithi171_78918_object70473:head# 2024-04-20T17:11:08.193 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7416f5d8:::benchmark_data_smithi171_78918_object140495:head# 2024-04-20T17:11:08.193 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74176ea6:::benchmark_data_smithi171_78918_object80734:head# 2024-04-20T17:11:08.193 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:741854bb:::benchmark_data_smithi171_78918_object80899:head# 2024-04-20T17:11:08.193 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:741896a1:::benchmark_data_smithi171_78918_object146358:head# 2024-04-20T17:11:08.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7419c2e9:::benchmark_data_smithi171_78918_object151481:head# 2024-04-20T17:11:08.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:741aeaa5:::benchmark_data_smithi171_78918_object152150:head# 2024-04-20T17:11:08.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:741b3b0b:::benchmark_data_smithi171_78918_object115664:head# 2024-04-20T17:11:08.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:741b8141:::benchmark_data_smithi171_78918_object66602:head# 2024-04-20T17:11:08.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:741c34c4:::benchmark_data_smithi171_78918_object153806:head# 2024-04-20T17:11:08.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:741c6aa8:::benchmark_data_smithi171_78918_object87202:head# 2024-04-20T17:11:08.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:741c8d6c:::benchmark_data_smithi171_78918_object88557:head# 2024-04-20T17:11:08.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:741d6394:::benchmark_data_smithi171_78918_object61602:head# 2024-04-20T17:11:08.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:741d6924:::benchmark_data_smithi171_78918_object117915:head# 2024-04-20T17:11:08.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:741f3267:::benchmark_data_smithi171_78918_object122162:head# 2024-04-20T17:11:08.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7422f3f7:::benchmark_data_smithi171_78918_object131516:head# 2024-04-20T17:11:08.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7423be70:::benchmark_data_smithi171_78918_object88067:head# 2024-04-20T17:11:08.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74253a92:::benchmark_data_smithi171_78918_object83989:head# 2024-04-20T17:11:08.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7425800f:::benchmark_data_smithi171_78918_object99539:head# 2024-04-20T17:11:08.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7425d823:::benchmark_data_smithi171_78918_object106131:head# 2024-04-20T17:11:08.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7426049f:::benchmark_data_smithi171_78918_object80973:head# 2024-04-20T17:11:08.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74262fd7:::benchmark_data_smithi171_78918_object160814:head# 2024-04-20T17:11:08.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74268afb:::benchmark_data_smithi171_78918_object132327:head# 2024-04-20T17:11:08.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7426d13a:::benchmark_data_smithi171_78918_object146827:head# 2024-04-20T17:11:08.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:742740f3:::benchmark_data_smithi171_78918_object159984:head# 2024-04-20T17:11:08.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74287d76:::benchmark_data_smithi171_78918_object117820:head# 2024-04-20T17:11:08.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7428a459:::benchmark_data_smithi171_78918_object55276:head# 2024-04-20T17:11:08.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74291cb5:::benchmark_data_smithi171_78918_object57146:head# 2024-04-20T17:11:08.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74293c93:::benchmark_data_smithi171_78918_object160801:head# 2024-04-20T17:11:08.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74294d95:::benchmark_data_smithi171_78918_object142283:head# 2024-04-20T17:11:08.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:742abfe6:::benchmark_data_smithi171_78918_object99647:head# 2024-04-20T17:11:08.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:742bb116:::benchmark_data_smithi171_78918_object52460:head# 2024-04-20T17:11:08.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:742cc3e3:::benchmark_data_smithi171_78918_object114515:head# 2024-04-20T17:11:08.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:742ec82a:::benchmark_data_smithi171_78918_object116138:head# 2024-04-20T17:11:08.196 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:742f9697:::benchmark_data_smithi171_78918_object135117:head# 2024-04-20T17:11:08.196 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7430cfea:::benchmark_data_smithi171_78918_object77636:head# 2024-04-20T17:11:08.208 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7431695d:::benchmark_data_smithi171_78918_object116058:head# 2024-04-20T17:11:08.208 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7431a6e5:::benchmark_data_smithi171_78918_object83484:head# 2024-04-20T17:11:08.208 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7432e34f:::benchmark_data_smithi171_78918_object56781:head# 2024-04-20T17:11:08.208 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74341726:::benchmark_data_smithi171_78918_object152761:head# 2024-04-20T17:11:08.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74342d5f:::benchmark_data_smithi171_78918_object80258:head# 2024-04-20T17:11:08.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7434d5d1:::benchmark_data_smithi171_78918_object53585:head# 2024-04-20T17:11:08.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7434dd58:::benchmark_data_smithi171_78918_object102023:head# 2024-04-20T17:11:08.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74353b1b:::benchmark_data_smithi171_78918_object125948:head# 2024-04-20T17:11:08.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7435dd33:::benchmark_data_smithi171_78918_object97291:head# 2024-04-20T17:11:08.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7437e67f:::benchmark_data_smithi171_78918_object138009:head# 2024-04-20T17:11:08.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7438c24c:::benchmark_data_smithi171_78918_object71921:head# 2024-04-20T17:11:08.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7438f7f2:::benchmark_data_smithi171_78918_object53580:head# 2024-04-20T17:11:08.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:743aa16d:::benchmark_data_smithi171_78918_object119182:head# 2024-04-20T17:11:08.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:743ac9ef:::benchmark_data_smithi171_78918_object113879:head# 2024-04-20T17:11:08.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:743b773b:::benchmark_data_smithi171_78918_object81965:head# 2024-04-20T17:11:08.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:743cd6df:::benchmark_data_smithi171_78918_object91121:head# 2024-04-20T17:11:08.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:743d5878:::benchmark_data_smithi171_78918_object149263:head# 2024-04-20T17:11:08.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:743da0b2:::benchmark_data_smithi171_78918_object100986:head# 2024-04-20T17:11:08.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:743f2058:::benchmark_data_smithi171_78918_object52039:head# 2024-04-20T17:11:08.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:743f5382:::benchmark_data_smithi171_78918_object112292:head# 2024-04-20T17:11:08.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7440a945:::benchmark_data_smithi171_78918_object89684:head# 2024-04-20T17:11:08.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:744135d4:::benchmark_data_smithi171_78918_object64789:head# 2024-04-20T17:11:08.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:744139b3:::benchmark_data_smithi171_78918_object79068:head# 2024-04-20T17:11:08.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74418db9:::benchmark_data_smithi171_78918_object77262:head# 2024-04-20T17:11:08.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7442aa6f:::benchmark_data_smithi171_78918_object131291:head# 2024-04-20T17:11:08.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7442f31d:::benchmark_data_smithi171_78918_object145465:head# 2024-04-20T17:11:08.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:744300f2:::benchmark_data_smithi171_78918_object51599:head# 2024-04-20T17:11:08.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7443d5ef:::benchmark_data_smithi171_78918_object147783:head# 2024-04-20T17:11:08.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74446779:::benchmark_data_smithi171_78918_object83787:head# 2024-04-20T17:11:08.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7444d817:::benchmark_data_smithi171_78918_object63269:head# 2024-04-20T17:11:08.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7444f01d:::benchmark_data_smithi171_78918_object78099:head# 2024-04-20T17:11:08.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:744567c2:::benchmark_data_smithi171_78918_object154787:head# 2024-04-20T17:11:08.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74462b05:::benchmark_data_smithi171_78918_object73250:head# 2024-04-20T17:11:08.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7446fee5:::benchmark_data_smithi171_78918_object56310:head# 2024-04-20T17:11:08.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74488eac:::benchmark_data_smithi171_78918_object89972:head# 2024-04-20T17:11:08.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7449b950:::benchmark_data_smithi171_78918_object126858:head# 2024-04-20T17:11:08.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:744a383c:::benchmark_data_smithi171_78918_object70809:head# 2024-04-20T17:11:08.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:744aa670:::benchmark_data_smithi171_78918_object83264:head# 2024-04-20T17:11:08.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:744ab235:::benchmark_data_smithi171_78918_object133657:head# 2024-04-20T17:11:08.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:744ac922:::benchmark_data_smithi171_78918_object73115:head# 2024-04-20T17:11:08.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:744c47ee:::benchmark_data_smithi171_78918_object120233:head# 2024-04-20T17:11:08.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:744daadc:::benchmark_data_smithi171_78918_object149758:head# 2024-04-20T17:11:08.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:744e916b:::benchmark_data_smithi171_78918_object156042:head# 2024-04-20T17:11:08.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:744f3258:::benchmark_data_smithi171_78918_object141834:head# 2024-04-20T17:11:08.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7450054c:::benchmark_data_smithi171_78918_object93233:head# 2024-04-20T17:11:08.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7450f8f0:::benchmark_data_smithi171_78918_object61176:head# 2024-04-20T17:11:08.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74512801:::benchmark_data_smithi171_78918_object131507:head# 2024-04-20T17:11:08.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74517053:::benchmark_data_smithi171_78918_object148156:head# 2024-04-20T17:11:08.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74523564:::benchmark_data_smithi171_78918_object105160:head# 2024-04-20T17:11:08.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7452e035:::benchmark_data_smithi171_78918_object151341:head# 2024-04-20T17:11:08.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7454f989:::benchmark_data_smithi171_78918_object76852:head# 2024-04-20T17:11:08.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74552236:::benchmark_data_smithi171_78918_object153312:head# 2024-04-20T17:11:08.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7455a63c:::benchmark_data_smithi171_78918_object157400:head# 2024-04-20T17:11:08.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7455fa01:::benchmark_data_smithi171_78918_object138965:head# 2024-04-20T17:11:08.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74564c95:::benchmark_data_smithi171_78918_object110121:head# 2024-04-20T17:11:08.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:745820f3:::benchmark_data_smithi171_78918_object55890:head# 2024-04-20T17:11:08.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74586996:::benchmark_data_smithi171_78918_object160147:head# 2024-04-20T17:11:08.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74588b2b:::benchmark_data_smithi171_78918_object124856:head# 2024-04-20T17:11:08.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7459a9c4:::benchmark_data_smithi171_78918_object142732:head# 2024-04-20T17:11:08.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7459e84d:::benchmark_data_smithi171_78918_object147996:head# 2024-04-20T17:11:08.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:745a9e69:::benchmark_data_smithi171_78918_object93468:head# 2024-04-20T17:11:08.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:745b2c6c:::benchmark_data_smithi171_78918_object71025:head# 2024-04-20T17:11:08.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:745b5099:::benchmark_data_smithi171_78918_object114981:head# 2024-04-20T17:11:08.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:745bfe1f:::benchmark_data_smithi171_78918_object75698:head# 2024-04-20T17:11:08.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:745d7489:::benchmark_data_smithi171_78918_object129614:head# 2024-04-20T17:11:08.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:745d8f60:::benchmark_data_smithi171_78918_object105440:head# 2024-04-20T17:11:08.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:745dbdff:::benchmark_data_smithi171_78918_object138644:head# 2024-04-20T17:11:08.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:745dec67:::benchmark_data_smithi171_78918_object121513:head# 2024-04-20T17:11:08.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74608ce6:::benchmark_data_smithi171_78918_object66270:head# 2024-04-20T17:11:08.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7461fae8:::benchmark_data_smithi171_78918_object107248:head# 2024-04-20T17:11:08.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74629941:::benchmark_data_smithi171_78918_object88932:head# 2024-04-20T17:11:08.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7462b6a6:::benchmark_data_smithi171_78918_object109554:head# 2024-04-20T17:11:08.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7462c068:::benchmark_data_smithi171_78918_object124105:head# 2024-04-20T17:11:08.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:746328a2:::benchmark_data_smithi171_78918_object90761:head# 2024-04-20T17:11:08.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74643f1a:::benchmark_data_smithi171_78918_object140747:head# 2024-04-20T17:11:08.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:746481f1:::benchmark_data_smithi171_78918_object72639:head# 2024-04-20T17:11:08.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7464a9d7:::benchmark_data_smithi171_78918_object86829:head# 2024-04-20T17:11:08.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74666899:::benchmark_data_smithi171_78918_object74184:head# 2024-04-20T17:11:08.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74667694:::benchmark_data_smithi171_78918_object66348:head# 2024-04-20T17:11:08.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74667e7f:::benchmark_data_smithi171_78918_object136773:head# 2024-04-20T17:11:08.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:746726f5:::benchmark_data_smithi171_78918_object152940:head# 2024-04-20T17:11:08.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74678a16:::benchmark_data_smithi171_78918_object88897:head# 2024-04-20T17:11:08.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7468e49c:::benchmark_data_smithi171_78918_object77103:head# 2024-04-20T17:11:08.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7468e52f:::benchmark_data_smithi171_78918_object157577:head# 2024-04-20T17:11:08.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7469db15:::benchmark_data_smithi171_78918_object145425:head# 2024-04-20T17:11:08.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:746b0a4a:::benchmark_data_smithi171_78918_object145790:head# 2024-04-20T17:11:08.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:746b403b:::benchmark_data_smithi171_78918_object105743:head# 2024-04-20T17:11:08.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:746bf146:::benchmark_data_smithi171_78918_object92698:head# 2024-04-20T17:11:08.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:746c3f8c:::benchmark_data_smithi171_78918_object95895:head# 2024-04-20T17:11:08.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:746cf2ea:::benchmark_data_smithi171_78918_object63974:head# 2024-04-20T17:11:08.243 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7470d3e9:::benchmark_data_smithi171_78918_object79884:head# 2024-04-20T17:11:08.243 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:747269db:::benchmark_data_smithi171_78918_object138355:head# 2024-04-20T17:11:08.243 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7472857e:::benchmark_data_smithi171_78918_object119676:head# 2024-04-20T17:11:08.243 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7472eafc:::benchmark_data_smithi171_78918_object95594:head# 2024-04-20T17:11:08.243 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74734166:::benchmark_data_smithi171_78918_object113899:head# 2024-04-20T17:11:08.243 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7474af29:::benchmark_data_smithi171_78918_object121464:head# 2024-04-20T17:11:08.243 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:747562e3:::benchmark_data_smithi171_78918_object109974:head# 2024-04-20T17:11:08.243 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7476bc54:::benchmark_data_smithi171_78918_object104081:head# 2024-04-20T17:11:08.243 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7477f5e4:::benchmark_data_smithi171_78918_object155542:head# 2024-04-20T17:11:08.243 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7478436e:::benchmark_data_smithi171_78918_object115473:head# 2024-04-20T17:11:08.243 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7478ff45:::benchmark_data_smithi171_78918_object151402:head# 2024-04-20T17:11:08.244 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74796db3:::benchmark_data_smithi171_78918_object54383:head# 2024-04-20T17:11:08.244 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:747990e3:::benchmark_data_smithi171_78918_object126141:head# 2024-04-20T17:11:08.244 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7479d76e:::benchmark_data_smithi171_78918_object66924:head# 2024-04-20T17:11:08.244 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:747a94d5:::benchmark_data_smithi171_78918_object119275:head# 2024-04-20T17:11:08.244 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:747c0909:::benchmark_data_smithi171_78918_object62084:head# 2024-04-20T17:11:08.244 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:747c41cc:::benchmark_data_smithi171_78918_object82748:head# 2024-04-20T17:11:08.244 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:747c6535:::benchmark_data_smithi171_78918_object119074:head# 2024-04-20T17:11:08.244 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:747c88d6:::benchmark_data_smithi171_78918_object135965:head# 2024-04-20T17:11:08.244 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:747caf3a:::benchmark_data_smithi171_78918_object146928:head# 2024-04-20T17:11:08.244 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:747cc2b7:::benchmark_data_smithi171_78918_object131179:head# 2024-04-20T17:11:08.244 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:747d75e6:::benchmark_data_smithi171_78918_object129559:head# 2024-04-20T17:11:08.244 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:747db7f5:::benchmark_data_smithi171_78918_object139625:head# 2024-04-20T17:11:08.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:747e864a:::benchmark_data_smithi171_78918_object77960:head# 2024-04-20T17:11:08.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74805053:::benchmark_data_smithi171_78918_object107467:head# 2024-04-20T17:11:08.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7480d232:::benchmark_data_smithi171_78918_object149357:head# 2024-04-20T17:11:08.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7482bb21:::benchmark_data_smithi171_78918_object56995:head# 2024-04-20T17:11:08.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74833c6a:::benchmark_data_smithi171_78918_object131949:head# 2024-04-20T17:11:08.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:748439eb:::benchmark_data_smithi171_78918_object155748:head# 2024-04-20T17:11:08.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7485cc8a:::benchmark_data_smithi171_78918_object152112:head# 2024-04-20T17:11:08.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7485e337:::benchmark_data_smithi171_78918_object105504:head# 2024-04-20T17:11:08.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74865980:::benchmark_data_smithi171_78918_object70396:head# 2024-04-20T17:11:08.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7486bb50:::benchmark_data_smithi171_78918_object153519:head# 2024-04-20T17:11:08.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74876e12:::benchmark_data_smithi171_78918_object79002:head# 2024-04-20T17:11:08.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7487d07a:::benchmark_data_smithi171_78918_object121225:head# 2024-04-20T17:11:08.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74881cda:::benchmark_data_smithi171_78918_object71343:head# 2024-04-20T17:11:08.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74888814:::benchmark_data_smithi171_78918_object75965:head# 2024-04-20T17:11:08.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7488c3b8:::benchmark_data_smithi171_78918_object112363:head# 2024-04-20T17:11:08.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7488ccb0:::benchmark_data_smithi171_78918_object73880:head# 2024-04-20T17:11:08.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7489e11e:::benchmark_data_smithi171_78918_object148994:head# 2024-04-20T17:11:08.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:748a8240:::benchmark_data_smithi171_78918_object115213:head# 2024-04-20T17:11:08.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:748abd3d:::benchmark_data_smithi171_78918_object144649:head# 2024-04-20T17:11:08.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:748b1167:::benchmark_data_smithi171_78918_object76143:head# 2024-04-20T17:11:08.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:748b16fa:::benchmark_data_smithi171_78918_object72419:head# 2024-04-20T17:11:08.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:748b2f75:::benchmark_data_smithi171_78918_object95192:head# 2024-04-20T17:11:08.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:748c8d39:::benchmark_data_smithi171_78918_object158084:head# 2024-04-20T17:11:08.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:748d12bc:::benchmark_data_smithi171_78918_object160358:head# 2024-04-20T17:11:08.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:748e558e:::benchmark_data_smithi171_78918_object104084:head# 2024-04-20T17:11:08.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:748f2d12:::benchmark_data_smithi171_78918_object119091:head# 2024-04-20T17:11:08.247 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:748fe9d7:::benchmark_data_smithi171_78918_object67875:head# 2024-04-20T17:11:08.247 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74901fba:::benchmark_data_smithi171_78918_object63041:head# 2024-04-20T17:11:08.247 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7490abf4:::benchmark_data_smithi171_78918_object146406:head# 2024-04-20T17:11:08.247 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7491cf2f:::benchmark_data_smithi171_78918_object153989:head# 2024-04-20T17:11:08.247 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:749363b7:::benchmark_data_smithi171_78918_object152522:head# 2024-04-20T17:11:08.247 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74944366:::benchmark_data_smithi171_78918_object86941:head# 2024-04-20T17:11:08.247 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:749520b7:::benchmark_data_smithi171_78918_object112386:head# 2024-04-20T17:11:08.247 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7495ddb4:::benchmark_data_smithi171_78918_object146462:head# 2024-04-20T17:11:08.247 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:749755e5:::benchmark_data_smithi171_78918_object96829:head# 2024-04-20T17:11:08.247 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74980855:::benchmark_data_smithi171_78918_object95438:head# 2024-04-20T17:11:08.247 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74988126:::benchmark_data_smithi171_78918_object128603:head# 2024-04-20T17:11:08.247 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7498f2ce:::benchmark_data_smithi171_78918_object82273:head# 2024-04-20T17:11:08.247 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74991fc8:::benchmark_data_smithi171_78918_object149291:head# 2024-04-20T17:11:08.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:749a79d4:::benchmark_data_smithi171_78918_object83192:head# 2024-04-20T17:11:08.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:749be937:::benchmark_data_smithi171_78918_object133526:head# 2024-04-20T17:11:08.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:749c3aec:::benchmark_data_smithi171_78918_object87255:head# 2024-04-20T17:11:08.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:749c675c:::benchmark_data_smithi171_78918_object74574:head# 2024-04-20T17:11:08.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:749c94a5:::benchmark_data_smithi171_78918_object159384:head# 2024-04-20T17:11:08.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:749cf490:::benchmark_data_smithi171_78918_object103006:head# 2024-04-20T17:11:08.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:749d0d3e:::benchmark_data_smithi171_78918_object81134:head# 2024-04-20T17:11:08.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:749dd95e:::benchmark_data_smithi171_78918_object155781:head# 2024-04-20T17:11:08.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:749f33bc:::benchmark_data_smithi171_78918_object150459:head# 2024-04-20T17:11:08.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:749f6647:::benchmark_data_smithi171_78918_object56602:head# 2024-04-20T17:11:08.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:749ff1ca:::benchmark_data_smithi171_78918_object154907:head# 2024-04-20T17:11:08.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74a012f0:::benchmark_data_smithi171_78918_object61308:head# 2024-04-20T17:11:08.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74a0f5cf:::benchmark_data_smithi171_78918_object115809:head# 2024-04-20T17:11:08.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74a25725:::benchmark_data_smithi171_78918_object153743:head# 2024-04-20T17:11:08.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74a2918d:::benchmark_data_smithi171_78918_object115721:head# 2024-04-20T17:11:08.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74a33215:::benchmark_data_smithi171_78918_object111194:head# 2024-04-20T17:11:08.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74a33aa9:::benchmark_data_smithi171_78918_object135398:head# 2024-04-20T17:11:08.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74a3796a:::benchmark_data_smithi171_78918_object86864:head# 2024-04-20T17:11:08.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74a46e9c:::benchmark_data_smithi171_78918_object60458:head# 2024-04-20T17:11:08.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74a59b89:::benchmark_data_smithi171_78918_object113946:head# 2024-04-20T17:11:08.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74a5eabb:::benchmark_data_smithi171_78918_object141545:head# 2024-04-20T17:11:08.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74a6209f:::benchmark_data_smithi171_78918_object104719:head# 2024-04-20T17:11:08.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74a6563a:::benchmark_data_smithi171_78918_object131787:head# 2024-04-20T17:11:08.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74a6e28a:::benchmark_data_smithi171_78918_object57935:head# 2024-04-20T17:11:08.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74a75380:::benchmark_data_smithi171_78918_object96611:head# 2024-04-20T17:11:08.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74a7771c:::benchmark_data_smithi171_78918_object94426:head# 2024-04-20T17:11:08.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74a804cd:::benchmark_data_smithi171_78918_object106665:head# 2024-04-20T17:11:08.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74a872ec:::benchmark_data_smithi171_78918_object54430:head# 2024-04-20T17:11:08.250 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74a87cb5:::benchmark_data_smithi171_78918_object95235:head# 2024-04-20T17:11:08.250 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74a8ff72:::benchmark_data_smithi171_78918_object147277:head# 2024-04-20T17:11:08.250 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74a97616:::benchmark_data_smithi171_78918_object72233:head# 2024-04-20T17:11:08.250 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74a97e73:::benchmark_data_smithi171_78918_object107632:head# 2024-04-20T17:11:08.250 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74a9a979:::benchmark_data_smithi171_78918_object91736:head# 2024-04-20T17:11:08.250 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74aae40f:::benchmark_data_smithi171_78918_object109042:head# 2024-04-20T17:11:08.250 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74ac1c02:::benchmark_data_smithi171_78918_object156804:head# 2024-04-20T17:11:08.250 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74ac2e7a:::benchmark_data_smithi171_78918_object123503:head# 2024-04-20T17:11:08.250 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74ac9c73:::benchmark_data_smithi171_78918_object76171:head# 2024-04-20T17:11:08.250 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74acbf08:::benchmark_data_smithi171_78918_object85489:head# 2024-04-20T17:11:08.250 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74ad16d4:::benchmark_data_smithi171_78918_object56125:head# 2024-04-20T17:11:08.250 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74ae1c1e:::benchmark_data_smithi171_78918_object122197:head# 2024-04-20T17:11:08.250 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74ae5a00:::benchmark_data_smithi171_78918_object130057:head# 2024-04-20T17:11:08.250 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74ae690a:::benchmark_data_smithi171_78918_object115814:head# 2024-04-20T17:11:08.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74ae95f5:::benchmark_data_smithi171_78918_object132892:head# 2024-04-20T17:11:08.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74afb660:::benchmark_data_smithi171_78918_object122222:head# 2024-04-20T17:11:08.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74afe660:::benchmark_data_smithi171_78918_object79953:head# 2024-04-20T17:11:08.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74b01278:::benchmark_data_smithi171_78918_object155158:head# 2024-04-20T17:11:08.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74b0d9ae:::benchmark_data_smithi171_78918_object79595:head# 2024-04-20T17:11:08.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74b155d2:::benchmark_data_smithi171_78918_object79346:head# 2024-04-20T17:11:08.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74b202ed:::benchmark_data_smithi171_78918_object94004:head# 2024-04-20T17:11:08.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74b39ab7:::benchmark_data_smithi171_78918_object57740:head# 2024-04-20T17:11:08.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74b492cd:::benchmark_data_smithi171_78918_object155287:head# 2024-04-20T17:11:08.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74b4d785:::benchmark_data_smithi171_78918_object156341:head# 2024-04-20T17:11:08.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74b55b25:::benchmark_data_smithi171_78918_object63639:head# 2024-04-20T17:11:08.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74b56c89:::benchmark_data_smithi171_78918_object145598:head# 2024-04-20T17:11:08.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74b58aca:::benchmark_data_smithi171_78918_object98799:head# 2024-04-20T17:11:08.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74b5c73d:::benchmark_data_smithi171_78918_object125116:head# 2024-04-20T17:11:08.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74b62b5a:::benchmark_data_smithi171_78918_object128650:head# 2024-04-20T17:11:08.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74b6f633:::benchmark_data_smithi171_78918_object97057:head# 2024-04-20T17:11:08.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74b7dccc:::benchmark_data_smithi171_78918_object60052:head# 2024-04-20T17:11:08.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74b80067:::benchmark_data_smithi171_78918_object148904:head# 2024-04-20T17:11:08.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74b8f0f0:::benchmark_data_smithi171_78918_object60970:head# 2024-04-20T17:11:08.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74b935d6:::benchmark_data_smithi171_78918_object81287:head# 2024-04-20T17:11:08.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74ba78e4:::benchmark_data_smithi171_78918_object148288:head# 2024-04-20T17:11:08.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74bab2ed:::benchmark_data_smithi171_78918_object104357:head# 2024-04-20T17:11:08.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74bb1879:::benchmark_data_smithi171_78918_object91481:head# 2024-04-20T17:11:08.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74bb889a:::benchmark_data_smithi171_78918_object65609:head# 2024-04-20T17:11:08.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74bbc173:::benchmark_data_smithi171_78918_object144490:head# 2024-04-20T17:11:08.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74bc4430:::benchmark_data_smithi171_78918_object106264:head# 2024-04-20T17:11:08.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74be3971:::benchmark_data_smithi171_78918_object119563:head# 2024-04-20T17:11:08.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74bf2671:::benchmark_data_smithi171_78918_object93411:head# 2024-04-20T17:11:08.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74c08515:::benchmark_data_smithi171_78918_object112385:head# 2024-04-20T17:11:08.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74c10e36:::benchmark_data_smithi171_78918_object134551:head# 2024-04-20T17:11:08.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74c1502d:::benchmark_data_smithi171_78918_object63465:head# 2024-04-20T17:11:08.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74c19511:::benchmark_data_smithi171_78918_object113621:head# 2024-04-20T17:11:08.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74c224e0:::benchmark_data_smithi171_78918_object110594:head# 2024-04-20T17:11:08.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74c24d87:::benchmark_data_smithi171_78918_object159641:head# 2024-04-20T17:11:08.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74c25ec5:::benchmark_data_smithi171_78918_object78805:head# 2024-04-20T17:11:08.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74c3d220:::benchmark_data_smithi171_78918_object69677:head# 2024-04-20T17:11:08.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74c46019:::benchmark_data_smithi171_78918_object94839:head# 2024-04-20T17:11:08.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74c4ba36:::benchmark_data_smithi171_78918_object156903:head# 2024-04-20T17:11:08.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74c5537f:::benchmark_data_smithi171_78918_object97139:head# 2024-04-20T17:11:08.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74c62277:::benchmark_data_smithi171_78918_object134764:head# 2024-04-20T17:11:08.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74c77e06:::benchmark_data_smithi171_78918_object85190:head# 2024-04-20T17:11:08.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74c8cf6f:::benchmark_data_smithi171_78918_object149793:head# 2024-04-20T17:11:08.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74c93af6:::benchmark_data_smithi171_78918_object73861:head# 2024-04-20T17:11:08.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74c99bff:::benchmark_data_smithi171_78918_object76825:head# 2024-04-20T17:11:08.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74ca2c40:::benchmark_data_smithi171_78918_object96769:head# 2024-04-20T17:11:08.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74cab782:::benchmark_data_smithi171_78918_object109325:head# 2024-04-20T17:11:08.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74caeb9f:::benchmark_data_smithi171_78918_object57703:head# 2024-04-20T17:11:08.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74cb62a0:::benchmark_data_smithi171_78918_object94227:head# 2024-04-20T17:11:08.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74cc1c3f:::benchmark_data_smithi171_78918_object76738:head# 2024-04-20T17:11:08.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74cd428a:::benchmark_data_smithi171_78918_object88711:head# 2024-04-20T17:11:08.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74ce99a3:::benchmark_data_smithi171_78918_object56360:head# 2024-04-20T17:11:08.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74cebaf1:::benchmark_data_smithi171_78918_object149046:head# 2024-04-20T17:11:08.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74cef167:::benchmark_data_smithi171_78918_object71415:head# 2024-04-20T17:11:08.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74d07826:::benchmark_data_smithi171_78918_object85159:head# 2024-04-20T17:11:08.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74d08054:::benchmark_data_smithi171_78918_object59162:head# 2024-04-20T17:11:08.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74d0b40e:::benchmark_data_smithi171_78918_object142975:head# 2024-04-20T17:11:08.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74d110f2:::benchmark_data_smithi171_78918_object58433:head# 2024-04-20T17:11:08.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74d1da19:::benchmark_data_smithi171_78918_object53001:head# 2024-04-20T17:11:08.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74d1ff92:::benchmark_data_smithi171_78918_object155081:head# 2024-04-20T17:11:08.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74d2d676:::benchmark_data_smithi171_78918_object130201:head# 2024-04-20T17:11:08.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74d2e401:::benchmark_data_smithi171_78918_object111693:head# 2024-04-20T17:11:08.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74d3d5aa:::benchmark_data_smithi171_78918_object71061:head# 2024-04-20T17:11:08.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74d3eb7d:::benchmark_data_smithi171_78918_object84626:head# 2024-04-20T17:11:08.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74d4efdf:::benchmark_data_smithi171_78918_object138357:head# 2024-04-20T17:11:08.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74d572f2:::benchmark_data_smithi171_78918_object122332:head# 2024-04-20T17:11:08.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74d590ae:::benchmark_data_smithi171_78918_object89041:head# 2024-04-20T17:11:08.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74d59cb7:::benchmark_data_smithi171_78918_object68431:head# 2024-04-20T17:11:08.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74d5ff05:::benchmark_data_smithi171_78918_object113865:head# 2024-04-20T17:11:08.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74d6cb55:::benchmark_data_smithi171_78918_object89330:head# 2024-04-20T17:11:08.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74d6d1d3:::benchmark_data_smithi171_78918_object128463:head# 2024-04-20T17:11:08.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74d76f33:::benchmark_data_smithi171_78918_object98736:head# 2024-04-20T17:11:08.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74d7dacd:::benchmark_data_smithi171_78918_object113891:head# 2024-04-20T17:11:08.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74d8f9a0:::benchmark_data_smithi171_78918_object103789:head# 2024-04-20T17:11:08.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74d96fd2:::benchmark_data_smithi171_78918_object88079:head# 2024-04-20T17:11:08.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74d99bd4:::benchmark_data_smithi171_78918_object66238:head# 2024-04-20T17:11:08.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74d9dba6:::benchmark_data_smithi171_78918_object142577:head# 2024-04-20T17:11:08.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74da2f3e:::benchmark_data_smithi171_78918_object139846:head# 2024-04-20T17:11:08.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74da2fd7:::benchmark_data_smithi171_78918_object108529:head# 2024-04-20T17:11:08.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74da6bf2:::benchmark_data_smithi171_78918_object147014:head# 2024-04-20T17:11:08.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74dad6a7:::benchmark_data_smithi171_78918_object117236:head# 2024-04-20T17:11:08.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74db5432:::benchmark_data_smithi171_78918_object124795:head# 2024-04-20T17:11:08.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74dc60d0:::benchmark_data_smithi171_78918_object106109:head# 2024-04-20T17:11:08.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74dcbaa1:::benchmark_data_smithi171_78918_object89853:head# 2024-04-20T17:11:08.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74dd4756:::benchmark_data_smithi171_78918_object129696:head# 2024-04-20T17:11:08.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74ddc84b:::benchmark_data_smithi171_78918_object102143:head# 2024-04-20T17:11:08.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74de7876:::benchmark_data_smithi171_78918_object100885:head# 2024-04-20T17:11:08.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74deadcf:::benchmark_data_smithi171_78918_object153579:head# 2024-04-20T17:11:08.257 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74df0541:::benchmark_data_smithi171_78918_object52968:head# 2024-04-20T17:11:08.257 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74df2cf5:::benchmark_data_smithi171_78918_object67328:head# 2024-04-20T17:11:08.257 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74df71b2:::benchmark_data_smithi171_78918_object70091:head# 2024-04-20T17:11:08.257 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74e03769:::benchmark_data_smithi171_78918_object63976:head# 2024-04-20T17:11:08.257 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74e06de0:::benchmark_data_smithi171_78918_object76698:head# 2024-04-20T17:11:08.274 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74e129ab:::benchmark_data_smithi171_78918_object75286:head# 2024-04-20T17:11:08.274 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74e1b168:::benchmark_data_smithi171_78918_object55620:head# 2024-04-20T17:11:08.274 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74e1fead:::benchmark_data_smithi171_78918_object129011:head# 2024-04-20T17:11:08.274 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74e43d12:::benchmark_data_smithi171_78918_object74497:head# 2024-04-20T17:11:08.274 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74e45c64:::benchmark_data_smithi171_78918_object72112:head# 2024-04-20T17:11:08.274 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74e5f60a:::benchmark_data_smithi171_78918_object132414:head# 2024-04-20T17:11:08.274 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74e6109f:::benchmark_data_smithi171_78918_object84612:head# 2024-04-20T17:11:08.274 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74e61870:::benchmark_data_smithi171_78918_object136068:head# 2024-04-20T17:11:08.274 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74e63d5d:::benchmark_data_smithi171_78918_object56384:head# 2024-04-20T17:11:08.274 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74e64259:::benchmark_data_smithi171_78918_object117516:head# 2024-04-20T17:11:08.274 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74e7d41d:::benchmark_data_smithi171_78918_object149825:head# 2024-04-20T17:11:08.275 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74e828b9:::benchmark_data_smithi171_78918_object140835:head# 2024-04-20T17:11:08.275 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74e95ce9:::benchmark_data_smithi171_78918_object57822:head# 2024-04-20T17:11:08.275 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74e9b562:::benchmark_data_smithi171_78918_object120634:head# 2024-04-20T17:11:08.275 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74eadfad:::benchmark_data_smithi171_78918_object51798:head# 2024-04-20T17:11:08.275 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74ebbb4f:::benchmark_data_smithi171_78918_object126058:head# 2024-04-20T17:11:08.275 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74ec17e3:::benchmark_data_smithi171_78918_object144190:head# 2024-04-20T17:11:08.275 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74ec2a7e:::benchmark_data_smithi171_78918_object154044:head# 2024-04-20T17:11:08.275 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74ecb9e5:::benchmark_data_smithi171_78918_object59721:head# 2024-04-20T17:11:08.275 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74ecc54f:::benchmark_data_smithi171_78918_object96065:head# 2024-04-20T17:11:08.275 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74ed66ca:::benchmark_data_smithi171_78918_object84304:head# 2024-04-20T17:11:08.275 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74edeaa6:::benchmark_data_smithi171_78918_object126314:head# 2024-04-20T17:11:08.275 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74eeb926:::benchmark_data_smithi171_78918_object99526:head# 2024-04-20T17:11:08.275 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74f01619:::benchmark_data_smithi171_78918_object134414:head# 2024-04-20T17:11:08.275 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74f16c78:::benchmark_data_smithi171_78918_object58326:head# 2024-04-20T17:11:08.276 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74f23d6e:::benchmark_data_smithi171_78918_object93151:head# 2024-04-20T17:11:08.276 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74f25ae9:::benchmark_data_smithi171_78918_object107336:head# 2024-04-20T17:11:08.276 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74f2d24c:::benchmark_data_smithi171_78918_object90775:head# 2024-04-20T17:11:08.276 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74f2d4e4:::benchmark_data_smithi171_78918_object99410:head# 2024-04-20T17:11:08.276 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74f420bf:::benchmark_data_smithi171_78918_object142724:head# 2024-04-20T17:11:08.276 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74f69b28:::benchmark_data_smithi171_78918_object147163:head# 2024-04-20T17:11:08.276 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74f6efa7:::benchmark_data_smithi171_78918_object60936:head# 2024-04-20T17:11:08.276 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74f756d3:::benchmark_data_smithi171_78918_object121831:head# 2024-04-20T17:11:08.276 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74f8004b:::benchmark_data_smithi171_78918_object87000:head# 2024-04-20T17:11:08.276 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74f834a0:::benchmark_data_smithi171_78918_object60512:head# 2024-04-20T17:11:08.276 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74f9e695:::benchmark_data_smithi171_78918_object120596:head# 2024-04-20T17:11:08.276 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74fae7c2:::benchmark_data_smithi171_78918_object58679:head# 2024-04-20T17:11:08.276 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74fafd0f:::benchmark_data_smithi171_78918_object123346:head# 2024-04-20T17:11:08.276 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74fbfef0:::benchmark_data_smithi171_78918_object73089:head# 2024-04-20T17:11:08.276 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74fceeaf:::benchmark_data_smithi171_78918_object56426:head# 2024-04-20T17:11:08.277 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74fcf88a:::benchmark_data_smithi171_78918_object143101:head# 2024-04-20T17:11:08.277 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74fdf3ba:::benchmark_data_smithi171_78918_object114078:head# 2024-04-20T17:11:08.277 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74ff8c6b:::benchmark_data_smithi171_78918_object58771:head# 2024-04-20T17:11:08.277 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:74ffbd12:::benchmark_data_smithi171_78918_object70976:head# 2024-04-20T17:11:08.277 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:750003dc:::benchmark_data_smithi171_78918_object76280:head# 2024-04-20T17:11:08.277 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75016edd:::benchmark_data_smithi171_78918_object54655:head# 2024-04-20T17:11:08.277 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7503095f:::benchmark_data_smithi171_78918_object117703:head# 2024-04-20T17:11:08.277 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7504e6c3:::benchmark_data_smithi171_78918_object144909:head# 2024-04-20T17:11:08.277 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:750624b1:::benchmark_data_smithi171_78918_object117058:head# 2024-04-20T17:11:08.277 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75063539:::benchmark_data_smithi171_78918_object102192:head# 2024-04-20T17:11:08.277 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75066cd0:::benchmark_data_smithi171_78918_object154841:head# 2024-04-20T17:11:08.277 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75066d4b:::benchmark_data_smithi171_78918_object80788:head# 2024-04-20T17:11:08.277 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:750876f0:::benchmark_data_smithi171_78918_object122995:head# 2024-04-20T17:11:08.277 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75090bc0:::benchmark_data_smithi171_78918_object97451:head# 2024-04-20T17:11:08.277 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75092a3e:::benchmark_data_smithi171_78918_object68743:head# 2024-04-20T17:11:08.278 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:750a6f76:::benchmark_data_smithi171_78918_object136740:head# 2024-04-20T17:11:08.278 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:750a7d1e:::benchmark_data_smithi171_78918_object93155:head# 2024-04-20T17:11:08.278 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:750bef35:::benchmark_data_smithi171_78918_object109698:head# 2024-04-20T17:11:08.278 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:750cea33:::benchmark_data_smithi171_78918_object160634:head# 2024-04-20T17:11:08.278 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:750df830:::benchmark_data_smithi171_78918_object155102:head# 2024-04-20T17:11:08.278 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:750ed6eb:::benchmark_data_smithi171_78918_object101727:head# 2024-04-20T17:11:08.278 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:750f604e:::benchmark_data_smithi171_78918_object113184:head# 2024-04-20T17:11:08.278 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:750f8135:::benchmark_data_smithi171_78918_object127669:head# 2024-04-20T17:11:08.278 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:750fc5df:::benchmark_data_smithi171_78918_object116124:head# 2024-04-20T17:11:08.278 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:750ffe2f:::benchmark_data_smithi171_78918_object89231:head# 2024-04-20T17:11:08.278 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:751070f1:::benchmark_data_smithi171_78918_object141372:head# 2024-04-20T17:11:08.278 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7510859a:::benchmark_data_smithi171_78918_object97633:head# 2024-04-20T17:11:08.278 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7512e0b5:::benchmark_data_smithi171_78918_object82666:head# 2024-04-20T17:11:08.278 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:751379c4:::benchmark_data_smithi171_78918_object95383:head# 2024-04-20T17:11:08.278 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7513a82b:::benchmark_data_smithi171_78918_object150107:head# 2024-04-20T17:11:08.279 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7514ebae:::benchmark_data_smithi171_78918_object142200:head# 2024-04-20T17:11:08.279 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7514f192:::benchmark_data_smithi171_78918_object76756:head# 2024-04-20T17:11:08.279 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75151a00:::benchmark_data_smithi171_78918_object57194:head# 2024-04-20T17:11:08.279 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75158d51:::benchmark_data_smithi171_78918_object135653:head# 2024-04-20T17:11:08.279 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7515c183:::benchmark_data_smithi171_78918_object63899:head# 2024-04-20T17:11:08.279 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7515dd9c:::benchmark_data_smithi171_78918_object131764:head# 2024-04-20T17:11:08.279 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7517bee5:::benchmark_data_smithi171_78918_object150550:head# 2024-04-20T17:11:08.279 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75186b4a:::benchmark_data_smithi171_78918_object76645:head# 2024-04-20T17:11:08.279 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7518de7f:::benchmark_data_smithi171_78918_object87755:head# 2024-04-20T17:11:08.279 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:751ada3b:::benchmark_data_smithi171_78918_object56407:head# 2024-04-20T17:11:08.279 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:751b147b:::benchmark_data_smithi171_78918_object68606:head# 2024-04-20T17:11:08.279 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:751b16c1:::benchmark_data_smithi171_78918_object122793:head# 2024-04-20T17:11:08.279 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:751d2158:::benchmark_data_smithi171_78918_object54458:head# 2024-04-20T17:11:08.279 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:751dfe90:::benchmark_data_smithi171_78918_object63871:head# 2024-04-20T17:11:08.279 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:751f30f2:::benchmark_data_smithi171_78918_object153675:head# 2024-04-20T17:11:08.280 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:751f3c6c:::benchmark_data_smithi171_78918_object69310:head# 2024-04-20T17:11:08.280 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:751f7266:::benchmark_data_smithi171_78918_object136393:head# 2024-04-20T17:11:08.280 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:751f86e0:::benchmark_data_smithi171_78918_object151040:head# 2024-04-20T17:11:08.280 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75200f1c:::benchmark_data_smithi171_78918_object91732:head# 2024-04-20T17:11:08.280 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75203ec3:::benchmark_data_smithi171_78918_object114432:head# 2024-04-20T17:11:08.280 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75215229:::benchmark_data_smithi171_78918_object146031:head# 2024-04-20T17:11:08.280 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7521b3d3:::benchmark_data_smithi171_78918_object139631:head# 2024-04-20T17:11:08.280 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7521e6ee:::benchmark_data_smithi171_78918_object120119:head# 2024-04-20T17:11:08.280 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75227344:::benchmark_data_smithi171_78918_object155997:head# 2024-04-20T17:11:08.280 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7522b527:::benchmark_data_smithi171_78918_object156362:head# 2024-04-20T17:11:08.280 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7522c516:::benchmark_data_smithi171_78918_object92695:head# 2024-04-20T17:11:08.280 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7522d0fd:::benchmark_data_smithi171_78918_object144255:head# 2024-04-20T17:11:08.280 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7522d4ff:::benchmark_data_smithi171_78918_object117629:head# 2024-04-20T17:11:08.280 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7522fce9:::benchmark_data_smithi171_78918_object101481:head# 2024-04-20T17:11:08.280 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7523db4e:::benchmark_data_smithi171_78918_object140788:head# 2024-04-20T17:11:08.280 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7527806a:::benchmark_data_smithi171_78918_object100284:head# 2024-04-20T17:11:08.280 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7527e55c:::benchmark_data_smithi171_78918_object148647:head# 2024-04-20T17:11:08.281 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7528bab1:::benchmark_data_smithi171_78918_object80644:head# 2024-04-20T17:11:08.281 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:752907ee:::benchmark_data_smithi171_78918_object154084:head# 2024-04-20T17:11:08.281 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7529bf2e:::benchmark_data_smithi171_78918_object86880:head# 2024-04-20T17:11:08.281 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:752bc499:::benchmark_data_smithi171_78918_object138595:head# 2024-04-20T17:11:08.281 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:752d134e:::benchmark_data_smithi171_78918_object53398:head# 2024-04-20T17:11:08.281 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:752d1437:::benchmark_data_smithi171_78918_object56157:head# 2024-04-20T17:11:08.281 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:752d4e38:::benchmark_data_smithi171_78918_object66250:head# 2024-04-20T17:11:08.281 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:752d60d0:::benchmark_data_smithi171_78918_object87118:head# 2024-04-20T17:11:08.281 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:752d7141:::benchmark_data_smithi171_78918_object57749:head# 2024-04-20T17:11:08.281 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:752dd864:::benchmark_data_smithi171_78918_object152478:head# 2024-04-20T17:11:08.281 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:752dfd7c:::benchmark_data_smithi171_78918_object76550:head# 2024-04-20T17:11:08.281 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:752ef716:::benchmark_data_smithi171_78918_object106851:head# 2024-04-20T17:11:08.281 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:752f9594:::benchmark_data_smithi171_78918_object121452:head# 2024-04-20T17:11:08.281 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7530b14a:::benchmark_data_smithi171_78918_object136278:head# 2024-04-20T17:11:08.281 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75310406:::benchmark_data_smithi171_78918_object118933:head# 2024-04-20T17:11:08.281 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7532a0da:::benchmark_data_smithi171_78918_object107024:head# 2024-04-20T17:11:08.282 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75336909:::benchmark_data_smithi171_78918_object115665:head# 2024-04-20T17:11:08.282 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7534b206:::benchmark_data_smithi171_78918_object141350:head# 2024-04-20T17:11:08.282 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7535101d:::benchmark_data_smithi171_78918_object81376:head# 2024-04-20T17:11:08.282 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75351191:::benchmark_data_smithi171_78918_object64211:head# 2024-04-20T17:11:08.282 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75353192:::benchmark_data_smithi171_78918_object65292:head# 2024-04-20T17:11:08.282 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75358610:::benchmark_data_smithi171_78918_object59432:head# 2024-04-20T17:11:08.282 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7535a46a:::benchmark_data_smithi171_78918_object115662:head# 2024-04-20T17:11:08.282 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75360450:::benchmark_data_smithi171_78918_object60867:head# 2024-04-20T17:11:08.282 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75378b8d:::benchmark_data_smithi171_78918_object158293:head# 2024-04-20T17:11:08.282 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7537f0a3:::benchmark_data_smithi171_78918_object105711:head# 2024-04-20T17:11:08.282 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:753909d3:::benchmark_data_smithi171_78918_object157528:head# 2024-04-20T17:11:08.282 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:753a428e:::benchmark_data_smithi171_78918_object159591:head# 2024-04-20T17:11:08.282 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:753b2848:::benchmark_data_smithi171_78918_object154866:head# 2024-04-20T17:11:08.282 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:753bc389:::benchmark_data_smithi171_78918_object159981:head# 2024-04-20T17:11:08.283 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:753c010b:::benchmark_data_smithi171_78918_object148870:head# 2024-04-20T17:11:08.283 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:753ec688:::benchmark_data_smithi171_78918_object61220:head# 2024-04-20T17:11:08.283 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:753eee16:::benchmark_data_smithi171_78918_object120929:head# 2024-04-20T17:11:08.283 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:753f0751:::benchmark_data_smithi171_78918_object106614:head# 2024-04-20T17:11:08.283 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:753f756b:::benchmark_data_smithi171_78918_object88455:head# 2024-04-20T17:11:08.283 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:753fa799:::benchmark_data_smithi171_78918_object140972:head# 2024-04-20T17:11:08.283 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:753fbbb9:::benchmark_data_smithi171_78918_object63608:head# 2024-04-20T17:11:08.283 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75418e15:::benchmark_data_smithi171_78918_object145654:head# 2024-04-20T17:11:08.283 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7542167b:::benchmark_data_smithi171_78918_object108438:head# 2024-04-20T17:11:08.283 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7542608e:::benchmark_data_smithi171_78918_object151591:head# 2024-04-20T17:11:08.283 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7542cb29:::benchmark_data_smithi171_78918_object97200:head# 2024-04-20T17:11:08.283 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75477a23:::benchmark_data_smithi171_78918_object124882:head# 2024-04-20T17:11:08.283 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:754982ef:::benchmark_data_smithi171_78918_object78531:head# 2024-04-20T17:11:08.283 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7549b59a:::benchmark_data_smithi171_78918_object160741:head# 2024-04-20T17:11:08.284 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7549cf36:::benchmark_data_smithi171_78918_object120603:head# 2024-04-20T17:11:08.284 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:754af90e:::benchmark_data_smithi171_78918_object89054:head# 2024-04-20T17:11:08.284 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:754beb04:::benchmark_data_smithi171_78918_object121839:head# 2024-04-20T17:11:08.284 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:754c38d1:::benchmark_data_smithi171_78918_object57646:head# 2024-04-20T17:11:08.284 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:754c73f0:::benchmark_data_smithi171_78918_object155464:head# 2024-04-20T17:11:08.284 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:754ded62:::benchmark_data_smithi171_78918_object106338:head# 2024-04-20T17:11:08.284 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:754e2282:::benchmark_data_smithi171_78918_object81591:head# 2024-04-20T17:11:08.284 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:754ed2d3:::benchmark_data_smithi171_78918_object70197:head# 2024-04-20T17:11:08.284 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75509144:::benchmark_data_smithi171_78918_object91497:head# 2024-04-20T17:11:08.296 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7551037b:::benchmark_data_smithi171_78918_object131917:head# 2024-04-20T17:11:08.296 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75510836:::benchmark_data_smithi171_78918_object153671:head# 2024-04-20T17:11:08.296 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75546620:::benchmark_data_smithi171_78918_object118963:head# 2024-04-20T17:11:08.296 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7555230b:::benchmark_data_smithi171_78918_object110795:head# 2024-04-20T17:11:08.296 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75557aa9:::benchmark_data_smithi171_78918_object106789:head# 2024-04-20T17:11:08.296 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7555cccc:::benchmark_data_smithi171_78918_object123585:head# 2024-04-20T17:11:08.296 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7555d75a:::benchmark_data_smithi171_78918_object51866:head# 2024-04-20T17:11:08.296 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75560546:::benchmark_data_smithi171_78918_object151337:head# 2024-04-20T17:11:08.296 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75566d41:::benchmark_data_smithi171_78918_object119650:head# 2024-04-20T17:11:08.296 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:755673ba:::benchmark_data_smithi171_78918_object110569:head# 2024-04-20T17:11:08.297 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75586ba5:::benchmark_data_smithi171_78918_object141962:head# 2024-04-20T17:11:08.297 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75591249:::benchmark_data_smithi171_78918_object64514:head# 2024-04-20T17:11:08.297 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7559a55a:::benchmark_data_smithi171_78918_object97006:head# 2024-04-20T17:11:08.297 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7559cd0d:::benchmark_data_smithi171_78918_object141353:head# 2024-04-20T17:11:08.297 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:755a5dd0:::benchmark_data_smithi171_78918_object123725:head# 2024-04-20T17:11:08.297 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:755ba3eb:::benchmark_data_smithi171_78918_object158659:head# 2024-04-20T17:11:08.297 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:755bb667:::benchmark_data_smithi171_78918_object79420:head# 2024-04-20T17:11:08.297 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:755c7793:::benchmark_data_smithi171_78918_object107902:head# 2024-04-20T17:11:08.297 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:755d5a53:::benchmark_data_smithi171_78918_object68356:head# 2024-04-20T17:11:08.297 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:755d954c:::benchmark_data_smithi171_78918_object128436:head# 2024-04-20T17:11:08.297 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:755dc865:::benchmark_data_smithi171_78918_object156985:head# 2024-04-20T17:11:08.297 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:755e1aad:::benchmark_data_smithi171_78918_object87570:head# 2024-04-20T17:11:08.297 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:755f9ce5:::benchmark_data_smithi171_78918_object112225:head# 2024-04-20T17:11:08.297 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:755fe6b7:::benchmark_data_smithi171_78918_object159319:head# 2024-04-20T17:11:08.297 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7560d30f:::benchmark_data_smithi171_78918_object83337:head# 2024-04-20T17:11:08.298 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75633626:::benchmark_data_smithi171_78918_object115629:head# 2024-04-20T17:11:08.298 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:756445bc:::benchmark_data_smithi171_78918_object75961:head# 2024-04-20T17:11:08.298 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75644f78:::benchmark_data_smithi171_78918_object64630:head# 2024-04-20T17:11:08.298 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:756461fd:::benchmark_data_smithi171_78918_object92127:head# 2024-04-20T17:11:08.298 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7564d7ec:::benchmark_data_smithi171_78918_object159230:head# 2024-04-20T17:11:08.298 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:756503e9:::benchmark_data_smithi171_78918_object54062:head# 2024-04-20T17:11:08.298 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7566da37:::benchmark_data_smithi171_78918_object116271:head# 2024-04-20T17:11:08.298 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7567f892:::benchmark_data_smithi171_78918_object158616:head# 2024-04-20T17:11:08.298 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7567fd99:::benchmark_data_smithi171_78918_object79257:head# 2024-04-20T17:11:08.298 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75680cd8:::benchmark_data_smithi171_78918_object154184:head# 2024-04-20T17:11:08.298 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:756868e7:::benchmark_data_smithi171_78918_object75400:head# 2024-04-20T17:11:08.298 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7569a8c2:::benchmark_data_smithi171_78918_object131739:head# 2024-04-20T17:11:08.298 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:756a8387:::benchmark_data_smithi171_78918_object128839:head# 2024-04-20T17:11:08.298 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:756b2a4a:::benchmark_data_smithi171_78918_object110172:head# 2024-04-20T17:11:08.299 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:756bdb95:::benchmark_data_smithi171_78918_object139115:head# 2024-04-20T17:11:08.299 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:756c5bb6:::benchmark_data_smithi171_78918_object72634:head# 2024-04-20T17:11:08.299 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:756c94d4:::benchmark_data_smithi171_78918_object151900:head# 2024-04-20T17:11:08.299 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:756ca218:::benchmark_data_smithi171_78918_object60893:head# 2024-04-20T17:11:08.299 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:756d2ba7:::benchmark_data_smithi171_78918_object117182:head# 2024-04-20T17:11:08.299 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:756d5b6a:::benchmark_data_smithi171_78918_object62856:head# 2024-04-20T17:11:08.299 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:756e23d2:::benchmark_data_smithi171_78918_object94459:head# 2024-04-20T17:11:08.299 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:756e26b5:::benchmark_data_smithi171_78918_object119413:head# 2024-04-20T17:11:08.299 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:756e4b35:::benchmark_data_smithi171_78918_object98844:head# 2024-04-20T17:11:08.299 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:756e9d97:::benchmark_data_smithi171_78918_object115468:head# 2024-04-20T17:11:08.299 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:756eae9c:::benchmark_data_smithi171_78918_object111712:head# 2024-04-20T17:11:08.299 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:756ee5e1:::benchmark_data_smithi171_78918_object121213:head# 2024-04-20T17:11:08.299 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:756f859f:::benchmark_data_smithi171_78918_object96153:head# 2024-04-20T17:11:08.299 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:756fd72f:::benchmark_data_smithi171_78918_object136561:head# 2024-04-20T17:11:08.300 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:756ff8ca:::benchmark_data_smithi171_78918_object111158:head# 2024-04-20T17:11:08.300 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7573012f:::benchmark_data_smithi171_78918_object79192:head# 2024-04-20T17:11:08.300 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75742e8a:::benchmark_data_smithi171_78918_object65902:head# 2024-04-20T17:11:08.300 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75746714:::benchmark_data_smithi171_78918_object150809:head# 2024-04-20T17:11:08.300 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75749bd6:::benchmark_data_smithi171_78918_object69663:head# 2024-04-20T17:11:08.300 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7574d0a7:::benchmark_data_smithi171_78918_object78943:head# 2024-04-20T17:11:08.300 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75756c40:::benchmark_data_smithi171_78918_object58051:head# 2024-04-20T17:11:08.300 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75756d80:::benchmark_data_smithi171_78918_object143967:head# 2024-04-20T17:11:08.300 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7575a4e3:::benchmark_data_smithi171_78918_object90849:head# 2024-04-20T17:11:08.300 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7575ede8:::benchmark_data_smithi171_78918_object74940:head# 2024-04-20T17:11:08.300 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:757672b1:::benchmark_data_smithi171_78918_object149244:head# 2024-04-20T17:11:08.300 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75775fb1:::benchmark_data_smithi171_78918_object67843:head# 2024-04-20T17:11:08.300 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7578bae2:::benchmark_data_smithi171_78918_object138066:head# 2024-04-20T17:11:08.300 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:757995ca:::benchmark_data_smithi171_78918_object153612:head# 2024-04-20T17:11:08.300 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:757a55a2:::benchmark_data_smithi171_78918_object95578:head# 2024-04-20T17:11:08.300 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:757bc168:::benchmark_data_smithi171_78918_object155138:head# 2024-04-20T17:11:08.300 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:757c1b30:::benchmark_data_smithi171_78918_object57529:head# 2024-04-20T17:11:08.301 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:757ce38b:::benchmark_data_smithi171_78918_object148205:head# 2024-04-20T17:11:08.301 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:757d10f8:::benchmark_data_smithi171_78918_object128300:head# 2024-04-20T17:11:08.301 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:757d4e3c:::benchmark_data_smithi171_78918_object102716:head# 2024-04-20T17:11:08.301 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:757db0ab:::benchmark_data_smithi171_78918_object120804:head# 2024-04-20T17:11:08.301 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:757f1142:::benchmark_data_smithi171_78918_object123639:head# 2024-04-20T17:11:08.301 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:757fc3f4:::benchmark_data_smithi171_78918_object98903:head# 2024-04-20T17:11:08.301 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75809a50:::benchmark_data_smithi171_78918_object77396:head# 2024-04-20T17:11:08.301 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7580f431:::benchmark_data_smithi171_78918_object56435:head# 2024-04-20T17:11:08.301 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7581737b:::benchmark_data_smithi171_78918_object60845:head# 2024-04-20T17:11:08.301 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7582ce93:::benchmark_data_smithi171_78918_object82969:head# 2024-04-20T17:11:08.301 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7584d1e8:::benchmark_data_smithi171_78918_object149482:head# 2024-04-20T17:11:08.301 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:758583c8:::benchmark_data_smithi171_78918_object144801:head# 2024-04-20T17:11:08.301 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7585d243:::benchmark_data_smithi171_78918_object103402:head# 2024-04-20T17:11:08.301 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7585e1db:::benchmark_data_smithi171_78918_object154747:head# 2024-04-20T17:11:08.301 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:758731cb:::benchmark_data_smithi171_78918_object119461:head# 2024-04-20T17:11:08.301 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75875eb3:::benchmark_data_smithi171_78918_object139763:head# 2024-04-20T17:11:08.302 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75879604:::benchmark_data_smithi171_78918_object159777:head# 2024-04-20T17:11:08.302 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7587f8ad:::benchmark_data_smithi171_78918_object91973:head# 2024-04-20T17:11:08.302 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7588383c:::benchmark_data_smithi171_78918_object60974:head# 2024-04-20T17:11:08.302 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7588bb66:::benchmark_data_smithi171_78918_object160730:head# 2024-04-20T17:11:08.302 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7588f855:::benchmark_data_smithi171_78918_object59273:head# 2024-04-20T17:11:08.302 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75890ead:::benchmark_data_smithi171_78918_object102521:head# 2024-04-20T17:11:08.302 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7589acb8:::benchmark_data_smithi171_78918_object95675:head# 2024-04-20T17:11:08.302 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:758a8b19:::benchmark_data_smithi171_78918_object97037:head# 2024-04-20T17:11:08.302 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:758b263d:::benchmark_data_smithi171_78918_object94799:head# 2024-04-20T17:11:08.302 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:758b2718:::benchmark_data_smithi171_78918_object73596:head# 2024-04-20T17:11:08.302 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:758c1114:::benchmark_data_smithi171_78918_object156433:head# 2024-04-20T17:11:08.302 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:758c634f:::benchmark_data_smithi171_78918_object74331:head# 2024-04-20T17:11:08.302 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:758d03b4:::benchmark_data_smithi171_78918_object74795:head# 2024-04-20T17:11:08.302 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:758dc1a2:::benchmark_data_smithi171_78918_object100007:head# 2024-04-20T17:11:08.302 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:758ea6d0:::benchmark_data_smithi171_78918_object69069:head# 2024-04-20T17:11:08.303 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:758f744b:::benchmark_data_smithi171_78918_object104371:head# 2024-04-20T17:11:08.303 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:758f7e9a:::benchmark_data_smithi171_78918_object145666:head# 2024-04-20T17:11:08.303 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75908e62:::benchmark_data_smithi171_78918_object82285:head# 2024-04-20T17:11:08.303 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7590abda:::benchmark_data_smithi171_78918_object160583:head# 2024-04-20T17:11:08.303 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75912ead:::benchmark_data_smithi171_78918_object89120:head# 2024-04-20T17:11:08.303 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:759152fd:::benchmark_data_smithi171_78918_object141643:head# 2024-04-20T17:11:08.303 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75915aaa:::benchmark_data_smithi171_78918_object137173:head# 2024-04-20T17:11:08.303 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7593289a:::benchmark_data_smithi171_78918_object77357:head# 2024-04-20T17:11:08.303 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75933794:::benchmark_data_smithi171_78918_object147249:head# 2024-04-20T17:11:08.303 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75936f7d:::benchmark_data_smithi171_78918_object122360:head# 2024-04-20T17:11:08.303 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7594c3c0:::benchmark_data_smithi171_78918_object81770:head# 2024-04-20T17:11:08.303 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75951d24:::benchmark_data_smithi171_78918_object150213:head# 2024-04-20T17:11:08.303 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75955500:::benchmark_data_smithi171_78918_object58806:head# 2024-04-20T17:11:08.303 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75956b8a:::benchmark_data_smithi171_78918_object94933:head# 2024-04-20T17:11:08.304 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7595e535:::benchmark_data_smithi171_78918_object78269:head# 2024-04-20T17:11:08.304 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75963fde:::benchmark_data_smithi171_78918_object73559:head# 2024-04-20T17:11:08.304 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75969d05:::benchmark_data_smithi171_78918_object83131:head# 2024-04-20T17:11:08.304 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75977df2:::benchmark_data_smithi171_78918_object97048:head# 2024-04-20T17:11:08.312 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75993f16:::benchmark_data_smithi171_78918_object120859:head# 2024-04-20T17:11:08.312 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7599ae29:::benchmark_data_smithi171_78918_object137675:head# 2024-04-20T17:11:08.312 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:759a3bc8:::benchmark_data_smithi171_78918_object93851:head# 2024-04-20T17:11:08.312 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:759ab0b8:::benchmark_data_smithi171_78918_object142437:head# 2024-04-20T17:11:08.312 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:759af54c:::benchmark_data_smithi171_78918_object84592:head# 2024-04-20T17:11:08.312 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:759ba13a:::benchmark_data_smithi171_78918_object151210:head# 2024-04-20T17:11:08.312 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:759cb00c:::benchmark_data_smithi171_78918_object160877:head# 2024-04-20T17:11:08.312 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:759d1d63:::benchmark_data_smithi171_78918_object100525:head# 2024-04-20T17:11:08.313 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:759d8a76:::benchmark_data_smithi171_78918_object122096:head# 2024-04-20T17:11:08.313 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:759df595:::benchmark_data_smithi171_78918_object133806:head# 2024-04-20T17:11:08.313 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:759eb829:::benchmark_data_smithi171_78918_object121717:head# 2024-04-20T17:11:08.313 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:759ff877:::benchmark_data_smithi171_78918_object101505:head# 2024-04-20T17:11:08.313 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75a0a0c5:::benchmark_data_smithi171_78918_object94301:head# 2024-04-20T17:11:08.313 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75a0b2c0:::benchmark_data_smithi171_78918_object97958:head# 2024-04-20T17:11:08.313 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75a0bae7:::benchmark_data_smithi171_78918_object105456:head# 2024-04-20T17:11:08.313 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75a0d857:::benchmark_data_smithi171_78918_object68703:head# 2024-04-20T17:11:08.313 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75a0e405:::benchmark_data_smithi171_78918_object138174:head# 2024-04-20T17:11:08.313 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75a16be1:::benchmark_data_smithi171_78918_object108457:head# 2024-04-20T17:11:08.313 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75a222e8:::benchmark_data_smithi171_78918_object96255:head# 2024-04-20T17:11:08.313 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75a225c7:::benchmark_data_smithi171_78918_object114733:head# 2024-04-20T17:11:08.313 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75a2a617:::benchmark_data_smithi171_78918_object94666:head# 2024-04-20T17:11:08.313 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75a3c709:::benchmark_data_smithi171_78918_object158493:head# 2024-04-20T17:11:08.314 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75a41bc8:::benchmark_data_smithi171_78918_object142047:head# 2024-04-20T17:11:08.314 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75a47929:::benchmark_data_smithi171_78918_object120992:head# 2024-04-20T17:11:08.314 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75a52623:::benchmark_data_smithi171_78918_object60200:head# 2024-04-20T17:11:08.314 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75a5ed01:::benchmark_data_smithi171_78918_object56862:head# 2024-04-20T17:11:08.314 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75a6baa7:::benchmark_data_smithi171_78918_object155573:head# 2024-04-20T17:11:08.314 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75a6f648:::benchmark_data_smithi171_78918_object98375:head# 2024-04-20T17:11:08.314 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75a91baa:::benchmark_data_smithi171_78918_object114234:head# 2024-04-20T17:11:08.314 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75aa89d2:::benchmark_data_smithi171_78918_object104978:head# 2024-04-20T17:11:08.314 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75aadaeb:::benchmark_data_smithi171_78918_object60791:head# 2024-04-20T17:11:08.314 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75ab94f9:::benchmark_data_smithi171_78918_object62380:head# 2024-04-20T17:11:08.314 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75aba466:::benchmark_data_smithi171_78918_object132564:head# 2024-04-20T17:11:08.314 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75abd79d:::benchmark_data_smithi171_78918_object113476:head# 2024-04-20T17:11:08.314 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75ac16da:::benchmark_data_smithi171_78918_object80187:head# 2024-04-20T17:11:08.314 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75adb5be:::benchmark_data_smithi171_78918_object94492:head# 2024-04-20T17:11:08.315 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75adcefb:::benchmark_data_smithi171_78918_object129100:head# 2024-04-20T17:11:08.315 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75ae1460:::benchmark_data_smithi171_78918_object116459:head# 2024-04-20T17:11:08.315 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75ae5f67:::benchmark_data_smithi171_78918_object110480:head# 2024-04-20T17:11:08.315 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75ae7040:::benchmark_data_smithi171_78918_object93434:head# 2024-04-20T17:11:08.315 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75aef45d:::benchmark_data_smithi171_78918_object117826:head# 2024-04-20T17:11:08.315 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75af201d:::benchmark_data_smithi171_78918_object82461:head# 2024-04-20T17:11:08.315 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75af22e1:::benchmark_data_smithi171_78918_object140200:head# 2024-04-20T17:11:08.315 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75aff7d6:::benchmark_data_smithi171_78918_object104548:head# 2024-04-20T17:11:08.315 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75b1dea2:::benchmark_data_smithi171_78918_object115248:head# 2024-04-20T17:11:08.315 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75b2fb77:::benchmark_data_smithi171_78918_object136433:head# 2024-04-20T17:11:08.315 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75b39fc3:::benchmark_data_smithi171_78918_object58788:head# 2024-04-20T17:11:08.315 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75b3ab3d:::benchmark_data_smithi171_78918_object76119:head# 2024-04-20T17:11:08.315 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75b3e3d0:::benchmark_data_smithi171_78918_object77836:head# 2024-04-20T17:11:08.315 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75b4945e:::benchmark_data_smithi171_78918_object86202:head# 2024-04-20T17:11:08.315 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75b63a2e:::benchmark_data_smithi171_78918_object143734:head# 2024-04-20T17:11:08.316 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75b6fe5f:::benchmark_data_smithi171_78918_object111042:head# 2024-04-20T17:11:08.316 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75b95907:::benchmark_data_smithi171_78918_object66095:head# 2024-04-20T17:11:08.316 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75b9601f:::benchmark_data_smithi171_78918_object150117:head# 2024-04-20T17:11:08.316 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75baf317:::benchmark_data_smithi171_78918_object88942:head# 2024-04-20T17:11:08.316 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75bc3d97:::benchmark_data_smithi171_78918_object54204:head# 2024-04-20T17:11:08.316 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75bd3811:::benchmark_data_smithi171_78918_object144282:head# 2024-04-20T17:11:08.316 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75be0d0c:::benchmark_data_smithi171_78918_object101985:head# 2024-04-20T17:11:08.316 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75beaf89:::benchmark_data_smithi171_78918_object122831:head# 2024-04-20T17:11:08.316 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75bf67a7:::benchmark_data_smithi171_78918_object139806:head# 2024-04-20T17:11:08.316 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75bff26c:::benchmark_data_smithi171_78918_object70673:head# 2024-04-20T17:11:08.316 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75c0367e:::benchmark_data_smithi171_78918_object139663:head# 2024-04-20T17:11:08.316 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75c04bbf:::benchmark_data_smithi171_78918_object95331:head# 2024-04-20T17:11:08.316 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75c05fa9:::benchmark_data_smithi171_78918_object123240:head# 2024-04-20T17:11:08.316 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75c11545:::benchmark_data_smithi171_78918_object90949:head# 2024-04-20T17:11:08.317 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75c13108:::benchmark_data_smithi171_78918_object84863:head# 2024-04-20T17:11:08.317 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75c1445f:::benchmark_data_smithi171_78918_object153767:head# 2024-04-20T17:11:08.317 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75c1ba98:::benchmark_data_smithi171_78918_object69346:head# 2024-04-20T17:11:08.317 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75c1e43c:::benchmark_data_smithi171_78918_object115149:head# 2024-04-20T17:11:08.317 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75c31ad5:::benchmark_data_smithi171_78918_object147655:head# 2024-04-20T17:11:08.317 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75c53ff4:::benchmark_data_smithi171_78918_object87218:head# 2024-04-20T17:11:08.317 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75c606d3:::benchmark_data_smithi171_78918_object126424:head# 2024-04-20T17:11:08.317 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75c657aa:::benchmark_data_smithi171_78918_object143985:head# 2024-04-20T17:11:08.317 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75c85aa6:::benchmark_data_smithi171_78918_object81096:head# 2024-04-20T17:11:08.317 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75c8c6c8:::benchmark_data_smithi171_78918_object88537:head# 2024-04-20T17:11:08.317 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75c8f223:::benchmark_data_smithi171_78918_object67543:head# 2024-04-20T17:11:08.317 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75c90f92:::benchmark_data_smithi171_78918_object113290:head# 2024-04-20T17:11:08.317 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75c972be:::benchmark_data_smithi171_78918_object89190:head# 2024-04-20T17:11:08.317 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75caaf15:::benchmark_data_smithi171_78918_object111199:head# 2024-04-20T17:11:08.318 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75cb378d:::benchmark_data_smithi171_78918_object121566:head# 2024-04-20T17:11:08.318 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75cbfb44:::benchmark_data_smithi171_78918_object82726:head# 2024-04-20T17:11:08.318 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75cd1a12:::benchmark_data_smithi171_78918_object93081:head# 2024-04-20T17:11:08.318 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75cd6999:::benchmark_data_smithi171_78918_object63823:head# 2024-04-20T17:11:08.324 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75cd9fbf:::benchmark_data_smithi171_78918_object82773:head# 2024-04-20T17:11:08.324 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75ce78e6:::benchmark_data_smithi171_78918_object117832:head# 2024-04-20T17:11:08.324 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75ce7f27:::benchmark_data_smithi171_78918_object132170:head# 2024-04-20T17:11:08.325 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75d29691:::benchmark_data_smithi171_78918_object153788:head# 2024-04-20T17:11:08.325 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75d2bf05:::benchmark_data_smithi171_78918_object101362:head# 2024-04-20T17:11:08.325 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75d414d4:::benchmark_data_smithi171_78918_object138971:head# 2024-04-20T17:11:08.325 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75d43ee6:::benchmark_data_smithi171_78918_object149366:head# 2024-04-20T17:11:08.325 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75d47d13:::benchmark_data_smithi171_78918_object153935:head# 2024-04-20T17:11:08.325 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75d48ec8:::benchmark_data_smithi171_78918_object126912:head# 2024-04-20T17:11:08.325 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75d5cd12:::benchmark_data_smithi171_78918_object115186:head# 2024-04-20T17:11:08.325 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75d69685:::benchmark_data_smithi171_78918_object153707:head# 2024-04-20T17:11:08.325 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75d89244:::benchmark_data_smithi171_78918_object114904:head# 2024-04-20T17:11:08.325 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75d98542:::benchmark_data_smithi171_78918_object89027:head# 2024-04-20T17:11:08.325 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75d9876a:::benchmark_data_smithi171_78918_object156230:head# 2024-04-20T17:11:08.325 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75dbd144:::benchmark_data_smithi171_78918_object120707:head# 2024-04-20T17:11:08.325 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75dbfb4b:::benchmark_data_smithi171_78918_object135537:head# 2024-04-20T17:11:08.325 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75dc0aaa:::benchmark_data_smithi171_78918_object103339:head# 2024-04-20T17:11:08.326 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75dd54b4:::benchmark_data_smithi171_78918_object131228:head# 2024-04-20T17:11:08.326 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75ded61e:::benchmark_data_smithi171_78918_object132057:head# 2024-04-20T17:11:08.326 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75df12aa:::benchmark_data_smithi171_78918_object150511:head# 2024-04-20T17:11:08.326 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75e02af4:::benchmark_data_smithi171_78918_object57715:head# 2024-04-20T17:11:08.326 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75e23fee:::benchmark_data_smithi171_78918_object108233:head# 2024-04-20T17:11:08.326 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75e24459:::benchmark_data_smithi171_78918_object149700:head# 2024-04-20T17:11:08.326 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75e24d35:::benchmark_data_smithi171_78918_object56417:head# 2024-04-20T17:11:08.326 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75e385fc:::benchmark_data_smithi171_78918_object81676:head# 2024-04-20T17:11:08.326 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75e40439:::benchmark_data_smithi171_78918_object79092:head# 2024-04-20T17:11:08.326 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75e5c16c:::benchmark_data_smithi171_78918_object90889:head# 2024-04-20T17:11:08.326 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75e65f1a:::benchmark_data_smithi171_78918_object157001:head# 2024-04-20T17:11:08.326 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75eab0f4:::benchmark_data_smithi171_78918_object104748:head# 2024-04-20T17:11:08.326 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75ec7d8a:::benchmark_data_smithi171_78918_object71261:head# 2024-04-20T17:11:08.326 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75eca705:::benchmark_data_smithi171_78918_object121440:head# 2024-04-20T17:11:08.327 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75ecf51d:::benchmark_data_smithi171_78918_object66968:head# 2024-04-20T17:11:08.327 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75edbe51:::benchmark_data_smithi171_78918_object64875:head# 2024-04-20T17:11:08.327 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75edf3b0:::benchmark_data_smithi171_78918_object108078:head# 2024-04-20T17:11:08.327 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75ee591d:::benchmark_data_smithi171_78918_object97040:head# 2024-04-20T17:11:08.327 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75ee94e5:::benchmark_data_smithi171_78918_object66893:head# 2024-04-20T17:11:08.327 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75f0285b:::benchmark_data_smithi171_78918_object85989:head# 2024-04-20T17:11:08.327 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75f1f1c9:::benchmark_data_smithi171_78918_object112293:head# 2024-04-20T17:11:08.327 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75f1fdd7:::benchmark_data_smithi171_78918_object79366:head# 2024-04-20T17:11:08.327 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75f3791b:::benchmark_data_smithi171_78918_object137743:head# 2024-04-20T17:11:08.327 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75f3daea:::benchmark_data_smithi171_78918_object72891:head# 2024-04-20T17:11:08.327 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75f4b161:::benchmark_data_smithi171_78918_object82371:head# 2024-04-20T17:11:08.327 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75f4eb77:::benchmark_data_smithi171_78918_object77910:head# 2024-04-20T17:11:08.327 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75f4eefb:::benchmark_data_smithi171_78918_object150466:head# 2024-04-20T17:11:08.327 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75f55905:::benchmark_data_smithi171_78918_object106909:head# 2024-04-20T17:11:08.327 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75f6ad8b:::benchmark_data_smithi171_78918_object67423:head# 2024-04-20T17:11:08.328 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75f797fa:::benchmark_data_smithi171_78918_object96332:head# 2024-04-20T17:11:08.328 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75f84e2a:::benchmark_data_smithi171_78918_object92607:head# 2024-04-20T17:11:08.328 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75f8f92c:::benchmark_data_smithi171_78918_object51932:head# 2024-04-20T17:11:08.328 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75f9a035:::benchmark_data_smithi171_78918_object52832:head# 2024-04-20T17:11:08.328 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75f9bc7a:::benchmark_data_smithi171_78918_object91576:head# 2024-04-20T17:11:08.328 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75f9fcac:::benchmark_data_smithi171_78918_object143187:head# 2024-04-20T17:11:08.328 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75fa0ea3:::benchmark_data_smithi171_78918_object80398:head# 2024-04-20T17:11:08.328 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75fb9709:::benchmark_data_smithi171_78918_object107137:head# 2024-04-20T17:11:08.328 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75fc16e1:::benchmark_data_smithi171_78918_object125930:head# 2024-04-20T17:11:08.328 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75fd8d35:::benchmark_data_smithi171_78918_object133454:head# 2024-04-20T17:11:08.328 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75fe19db:::benchmark_data_smithi171_78918_object107925:head# 2024-04-20T17:11:08.328 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75fe4e77:::benchmark_data_smithi171_78918_object68972:head# 2024-04-20T17:11:08.328 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:75fedfcc:::benchmark_data_smithi171_78918_object70529:head# 2024-04-20T17:11:08.328 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76001691:::benchmark_data_smithi171_78918_object58179:head# 2024-04-20T17:11:08.328 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7600fdb7:::benchmark_data_smithi171_78918_object157566:head# 2024-04-20T17:11:08.329 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76014c28:::benchmark_data_smithi171_78918_object154245:head# 2024-04-20T17:11:08.329 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7601ec69:::benchmark_data_smithi171_78918_object86104:head# 2024-04-20T17:11:08.329 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76033447:::benchmark_data_smithi171_78918_object59212:head# 2024-04-20T17:11:08.329 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76035d91:::benchmark_data_smithi171_78918_object137372:head# 2024-04-20T17:11:08.329 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76037e0c:::benchmark_data_smithi171_78918_object63641:head# 2024-04-20T17:11:08.329 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7603a42b:::benchmark_data_smithi171_78918_object51460:head# 2024-04-20T17:11:08.329 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7604ef3b:::benchmark_data_smithi171_78918_object145622:head# 2024-04-20T17:11:08.329 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7605a613:::benchmark_data_smithi171_78918_object149438:head# 2024-04-20T17:11:08.329 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7606a356:::benchmark_data_smithi171_78918_object153771:head# 2024-04-20T17:11:08.335 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7606bb9c:::benchmark_data_smithi171_78918_object84206:head# 2024-04-20T17:11:08.335 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7606e7a9:::benchmark_data_smithi171_78918_object54841:head# 2024-04-20T17:11:08.335 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76073bce:::benchmark_data_smithi171_78918_object159272:head# 2024-04-20T17:11:08.335 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76085828:::benchmark_data_smithi171_78918_object79570:head# 2024-04-20T17:11:08.335 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:760b2128:::benchmark_data_smithi171_78918_object64346:head# 2024-04-20T17:11:08.335 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:760b8f6c:::benchmark_data_smithi171_78918_object86379:head# 2024-04-20T17:11:08.335 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:760bac7d:::benchmark_data_smithi171_78918_object99882:head# 2024-04-20T17:11:08.335 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:760c22fa:::benchmark_data_smithi171_78918_object143970:head# 2024-04-20T17:11:08.335 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:760cc699:::benchmark_data_smithi171_78918_object159495:head# 2024-04-20T17:11:08.335 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:760ce839:::benchmark_data_smithi171_78918_object57288:head# 2024-04-20T17:11:08.335 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:760d10cf:::benchmark_data_smithi171_78918_object92464:head# 2024-04-20T17:11:08.335 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:760d6487:::benchmark_data_smithi171_78918_object72164:head# 2024-04-20T17:11:08.335 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:760d79ba:::benchmark_data_smithi171_78918_object110132:head# 2024-04-20T17:11:08.335 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:760d7dae:::benchmark_data_smithi171_78918_object141185:head# 2024-04-20T17:11:08.336 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:760e6dbf:::benchmark_data_smithi171_78918_object117392:head# 2024-04-20T17:11:08.336 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:760e7f45:::benchmark_data_smithi171_78918_object72537:head# 2024-04-20T17:11:08.336 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:760f02b5:::benchmark_data_smithi171_78918_object81631:head# 2024-04-20T17:11:08.336 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:760f2db9:::benchmark_data_smithi171_78918_object53718:head# 2024-04-20T17:11:08.336 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:760fa722:::benchmark_data_smithi171_78918_object69203:head# 2024-04-20T17:11:08.336 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:761035e7:::benchmark_data_smithi171_78918_object130271:head# 2024-04-20T17:11:08.336 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76103c18:::benchmark_data_smithi171_78918_object81509:head# 2024-04-20T17:11:08.336 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7610c5d1:::benchmark_data_smithi171_78918_object98474:head# 2024-04-20T17:11:08.336 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7612cb9f:::benchmark_data_smithi171_78918_object149938:head# 2024-04-20T17:11:08.336 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:761397d3:::benchmark_data_smithi171_78918_object63312:head# 2024-04-20T17:11:08.336 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7613e345:::benchmark_data_smithi171_78918_object141138:head# 2024-04-20T17:11:08.336 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76141960:::benchmark_data_smithi171_78918_object52675:head# 2024-04-20T17:11:08.336 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7614b08c:::benchmark_data_smithi171_78918_object80924:head# 2024-04-20T17:11:08.336 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7615e823:::benchmark_data_smithi171_78918_object136970:head# 2024-04-20T17:11:08.337 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76164391:::benchmark_data_smithi171_78918_object70941:head# 2024-04-20T17:11:08.337 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7616761a:::benchmark_data_smithi171_78918_object153925:head# 2024-04-20T17:11:08.337 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7616e1c0:::benchmark_data_smithi171_78918_object96274:head# 2024-04-20T17:11:08.337 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7616f480:::benchmark_data_smithi171_78918_object122528:head# 2024-04-20T17:11:08.337 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7617d112:::benchmark_data_smithi171_78918_object130001:head# 2024-04-20T17:11:08.337 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76186003:::benchmark_data_smithi171_78918_object61084:head# 2024-04-20T17:11:08.337 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:761936bb:::benchmark_data_smithi171_78918_object153394:head# 2024-04-20T17:11:08.337 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:761963ea:::benchmark_data_smithi171_78918_object96431:head# 2024-04-20T17:11:08.337 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76199aa9:::benchmark_data_smithi171_78918_object93163:head# 2024-04-20T17:11:08.337 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:761a21c8:::benchmark_data_smithi171_78918_object101069:head# 2024-04-20T17:11:08.337 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:761af3cb:::benchmark_data_smithi171_78918_object54210:head# 2024-04-20T17:11:08.337 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:761c2d8f:::benchmark_data_smithi171_78918_object57508:head# 2024-04-20T17:11:08.337 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:761d0857:::benchmark_data_smithi171_78918_object118251:head# 2024-04-20T17:11:08.337 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:761d09d2:::benchmark_data_smithi171_78918_object133830:head# 2024-04-20T17:11:08.338 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:761d13d0:::benchmark_data_smithi171_78918_object93271:head# 2024-04-20T17:11:08.338 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:761d4044:::benchmark_data_smithi171_78918_object63420:head# 2024-04-20T17:11:08.338 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:761dfd09:::benchmark_data_smithi171_78918_object104596:head# 2024-04-20T17:11:08.338 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:761fbdc8:::benchmark_data_smithi171_78918_object132426:head# 2024-04-20T17:11:08.338 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:762017d5:::benchmark_data_smithi171_78918_object55820:head# 2024-04-20T17:11:08.338 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7620600e:::benchmark_data_smithi171_78918_object89816:head# 2024-04-20T17:11:08.338 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76207534:::benchmark_data_smithi171_78918_object115825:head# 2024-04-20T17:11:08.338 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76210853:::benchmark_data_smithi171_78918_object70270:head# 2024-04-20T17:11:08.338 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:762191ae:::benchmark_data_smithi171_78918_object114299:head# 2024-04-20T17:11:08.338 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7624730c:::benchmark_data_smithi171_78918_object86919:head# 2024-04-20T17:11:08.338 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:762536b2:::benchmark_data_smithi171_78918_object111588:head# 2024-04-20T17:11:08.338 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76265edb:::benchmark_data_smithi171_78918_object154835:head# 2024-04-20T17:11:08.338 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7626aa3f:::benchmark_data_smithi171_78918_object68080:head# 2024-04-20T17:11:08.338 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76273193:::benchmark_data_smithi171_78918_object149733:head# 2024-04-20T17:11:08.339 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76293d1d:::benchmark_data_smithi171_78918_object76030:head# 2024-04-20T17:11:08.339 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:762b696d:::benchmark_data_smithi171_78918_object91773:head# 2024-04-20T17:11:08.339 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:762bbc94:::benchmark_data_smithi171_78918_object92876:head# 2024-04-20T17:11:08.339 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:762c7d09:::benchmark_data_smithi171_78918_object125041:head# 2024-04-20T17:11:08.352 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:762d479f:::benchmark_data_smithi171_78918_object66900:head# 2024-04-20T17:11:08.352 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:762ddac7:::benchmark_data_smithi171_78918_object145691:head# 2024-04-20T17:11:08.352 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:762e0d76:::benchmark_data_smithi171_78918_object99287:head# 2024-04-20T17:11:08.352 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:762e2bca:::benchmark_data_smithi171_78918_object71538:head# 2024-04-20T17:11:08.352 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:762e86a8:::benchmark_data_smithi171_78918_object58128:head# 2024-04-20T17:11:08.352 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:762f124a:::benchmark_data_smithi171_78918_object74854:head# 2024-04-20T17:11:08.352 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:762fe4de:::benchmark_data_smithi171_78918_object152065:head# 2024-04-20T17:11:08.352 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76304c0f:::benchmark_data_smithi171_78918_object67796:head# 2024-04-20T17:11:08.352 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7630f96b:::benchmark_data_smithi171_78918_object87109:head# 2024-04-20T17:11:08.352 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76319501:::benchmark_data_smithi171_78918_object144455:head# 2024-04-20T17:11:08.352 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7631c1aa:::benchmark_data_smithi171_78918_object60364:head# 2024-04-20T17:11:08.352 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76337fc9:::benchmark_data_smithi171_78918_object59528:head# 2024-04-20T17:11:08.353 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7633c400:::benchmark_data_smithi171_78918_object73441:head# 2024-04-20T17:11:08.353 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:763480d6:::benchmark_data_smithi171_78918_object64071:head# 2024-04-20T17:11:08.353 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76353579:::benchmark_data_smithi171_78918_object111917:head# 2024-04-20T17:11:08.353 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76377ec2:::benchmark_data_smithi171_78918_object112295:head# 2024-04-20T17:11:08.353 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7638113d:::benchmark_data_smithi171_78918_object120661:head# 2024-04-20T17:11:08.353 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7638c6ce:::benchmark_data_smithi171_78918_object155150:head# 2024-04-20T17:11:08.353 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7638d22a:::benchmark_data_smithi171_78918_object65934:head# 2024-04-20T17:11:08.353 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:763aae91:::benchmark_data_smithi171_78918_object54388:head# 2024-04-20T17:11:08.353 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:763b0c69:::benchmark_data_smithi171_78918_object92069:head# 2024-04-20T17:11:08.353 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:763b4733:::benchmark_data_smithi171_78918_object62358:head# 2024-04-20T17:11:08.353 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:763b7cbc:::benchmark_data_smithi171_78918_object122268:head# 2024-04-20T17:11:08.353 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:763beaa5:::benchmark_data_smithi171_78918_object156540:head# 2024-04-20T17:11:08.353 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:763c2a0f:::benchmark_data_smithi171_78918_object57899:head# 2024-04-20T17:11:08.353 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:763d648b:::benchmark_data_smithi171_78918_object62943:head# 2024-04-20T17:11:08.354 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76408d44:::benchmark_data_smithi171_78918_object96631:head# 2024-04-20T17:11:08.354 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76413fc9:::benchmark_data_smithi171_78918_object84074:head# 2024-04-20T17:11:08.354 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76431abd:::benchmark_data_smithi171_78918_object156774:head# 2024-04-20T17:11:08.354 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76437c45:::benchmark_data_smithi171_78918_object106787:head# 2024-04-20T17:11:08.354 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:764394b9:::benchmark_data_smithi171_78918_object71811:head# 2024-04-20T17:11:08.354 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7643e05e:::benchmark_data_smithi171_78918_object132235:head# 2024-04-20T17:11:08.354 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:764692c0:::benchmark_data_smithi171_78918_object114632:head# 2024-04-20T17:11:08.354 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76478b39:::benchmark_data_smithi171_78918_object101151:head# 2024-04-20T17:11:08.354 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76495760:::benchmark_data_smithi171_78918_object57521:head# 2024-04-20T17:11:08.354 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7649eea4:::benchmark_data_smithi171_78918_object143634:head# 2024-04-20T17:11:08.354 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:764b10ce:::benchmark_data_smithi171_78918_object68520:head# 2024-04-20T17:11:08.354 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:764bc5d0:::benchmark_data_smithi171_78918_object83889:head# 2024-04-20T17:11:08.354 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:764bf601:::benchmark_data_smithi171_78918_object75754:head# 2024-04-20T17:11:08.354 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:764c9356:::benchmark_data_smithi171_78918_object65458:head# 2024-04-20T17:11:08.354 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:764ce36e:::benchmark_data_smithi171_78918_object72184:head# 2024-04-20T17:11:08.355 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:764d9bed:::benchmark_data_smithi171_78918_object126364:head# 2024-04-20T17:11:08.355 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:764db2de:::benchmark_data_smithi171_78918_object71046:head# 2024-04-20T17:11:08.355 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:764dc0ba:::benchmark_data_smithi171_78918_object96494:head# 2024-04-20T17:11:08.355 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:764e5a89:::benchmark_data_smithi171_78918_object138265:head# 2024-04-20T17:11:08.355 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:764f37db:::benchmark_data_smithi171_78918_object130978:head# 2024-04-20T17:11:08.355 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:764f5d79:::benchmark_data_smithi171_78918_object61645:head# 2024-04-20T17:11:08.355 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:764f636f:::benchmark_data_smithi171_78918_object141317:head# 2024-04-20T17:11:08.355 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:765006a9:::benchmark_data_smithi171_78918_object67662:head# 2024-04-20T17:11:08.355 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:765012c2:::benchmark_data_smithi171_78918_object128358:head# 2024-04-20T17:11:08.355 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7650ab22:::benchmark_data_smithi171_78918_object119470:head# 2024-04-20T17:11:08.355 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76515165:::benchmark_data_smithi171_78918_object73603:head# 2024-04-20T17:11:08.355 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7651890b:::benchmark_data_smithi171_78918_object60113:head# 2024-04-20T17:11:08.355 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7652069c:::benchmark_data_smithi171_78918_object136336:head# 2024-04-20T17:11:08.355 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76520bfd:::benchmark_data_smithi171_78918_object57245:head# 2024-04-20T17:11:08.356 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:765282f6:::benchmark_data_smithi171_78918_object126588:head# 2024-04-20T17:11:08.356 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76534489:::benchmark_data_smithi171_78918_object105779:head# 2024-04-20T17:11:08.356 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7654aac9:::benchmark_data_smithi171_78918_object116590:head# 2024-04-20T17:11:08.356 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7654bb82:::benchmark_data_smithi171_78918_object69818:head# 2024-04-20T17:11:08.356 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:765601b4:::benchmark_data_smithi171_78918_object149853:head# 2024-04-20T17:11:08.356 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76569381:::benchmark_data_smithi171_78918_object96013:head# 2024-04-20T17:11:08.356 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7656ce28:::benchmark_data_smithi171_78918_object94100:head# 2024-04-20T17:11:08.356 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7657f2cd:::benchmark_data_smithi171_78918_object154778:head# 2024-04-20T17:11:08.356 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76581cc2:::benchmark_data_smithi171_78918_object65756:head# 2024-04-20T17:11:08.356 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7658b391:::benchmark_data_smithi171_78918_object58699:head# 2024-04-20T17:11:08.356 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7658bca9:::benchmark_data_smithi171_78918_object62353:head# 2024-04-20T17:11:08.356 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76591710:::benchmark_data_smithi171_78918_object118165:head# 2024-04-20T17:11:08.356 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7659e390:::benchmark_data_smithi171_78918_object152738:head# 2024-04-20T17:11:08.356 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:765a0794:::benchmark_data_smithi171_78918_object106163:head# 2024-04-20T17:11:08.356 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:765a6c63:::benchmark_data_smithi171_78918_object112118:head# 2024-04-20T17:11:08.356 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:765ac0d0:::benchmark_data_smithi171_78918_object148077:head# 2024-04-20T17:11:08.357 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:765b65cd:::benchmark_data_smithi171_78918_object89219:head# 2024-04-20T17:11:08.357 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:765b7b37:::benchmark_data_smithi171_78918_object102634:head# 2024-04-20T17:11:08.357 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:765bb833:::benchmark_data_smithi171_78918_object151490:head# 2024-04-20T17:11:08.357 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:765d3ba4:::benchmark_data_smithi171_78918_object56777:head# 2024-04-20T17:11:08.357 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:765de407:::benchmark_data_smithi171_78918_object104819:head# 2024-04-20T17:11:08.357 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:765e0fcd:::benchmark_data_smithi171_78918_object150636:head# 2024-04-20T17:11:08.357 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:765e3103:::benchmark_data_smithi171_78918_object147990:head# 2024-04-20T17:11:08.357 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:765f8119:::benchmark_data_smithi171_78918_object97618:head# 2024-04-20T17:11:08.357 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:765fa3fe:::benchmark_data_smithi171_78918_object151914:head# 2024-04-20T17:11:08.357 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:766125d1:::benchmark_data_smithi171_78918_object154124:head# 2024-04-20T17:11:08.357 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7661597f:::benchmark_data_smithi171_78918_object59464:head# 2024-04-20T17:11:08.357 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7661d85a:::benchmark_data_smithi171_78918_object110085:head# 2024-04-20T17:11:08.357 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7662c415:::benchmark_data_smithi171_78918_object96165:head# 2024-04-20T17:11:08.357 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76634707:::benchmark_data_smithi171_78918_object92162:head# 2024-04-20T17:11:08.358 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76641312:::benchmark_data_smithi171_78918_object129938:head# 2024-04-20T17:11:08.358 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:766512e6:::benchmark_data_smithi171_78918_object84495:head# 2024-04-20T17:11:08.358 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:766515f9:::benchmark_data_smithi171_78918_object72009:head# 2024-04-20T17:11:08.358 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7665fbf6:::benchmark_data_smithi171_78918_object123666:head# 2024-04-20T17:11:08.358 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:766691a5:::benchmark_data_smithi171_78918_object98454:head# 2024-04-20T17:11:08.358 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7666a161:::benchmark_data_smithi171_78918_object87029:head# 2024-04-20T17:11:08.358 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76674731:::benchmark_data_smithi171_78918_object90956:head# 2024-04-20T17:11:08.358 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76675974:::benchmark_data_smithi171_78918_object117619:head# 2024-04-20T17:11:08.358 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7667870e:::benchmark_data_smithi171_78918_object117211:head# 2024-04-20T17:11:08.358 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:766799d6:::benchmark_data_smithi171_78918_object52161:head# 2024-04-20T17:11:08.358 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7668f102:::benchmark_data_smithi171_78918_object151162:head# 2024-04-20T17:11:08.372 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:766a9c78:::benchmark_data_smithi171_78918_object109477:head# 2024-04-20T17:11:08.372 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:766af39d:::benchmark_data_smithi171_78918_object145667:head# 2024-04-20T17:11:08.372 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:766c82cb:::benchmark_data_smithi171_78918_object78681:head# 2024-04-20T17:11:08.372 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:766c9d47:::benchmark_data_smithi171_78918_object81660:head# 2024-04-20T17:11:08.372 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:766ce5e7:::benchmark_data_smithi171_78918_object96808:head# 2024-04-20T17:11:08.372 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:766d3b7d:::benchmark_data_smithi171_78918_object147874:head# 2024-04-20T17:11:08.372 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:766d7e39:::benchmark_data_smithi171_78918_object137167:head# 2024-04-20T17:11:08.372 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:766d849a:::benchmark_data_smithi171_78918_object83976:head# 2024-04-20T17:11:08.372 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:766dfe64:::benchmark_data_smithi171_78918_object84576:head# 2024-04-20T17:11:08.372 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:766e3b95:::benchmark_data_smithi171_78918_object120715:head# 2024-04-20T17:11:08.372 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:766e6aed:::benchmark_data_smithi171_78918_object108767:head# 2024-04-20T17:11:08.372 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:766eb628:::benchmark_data_smithi171_78918_object134289:head# 2024-04-20T17:11:08.372 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:766f577d:::benchmark_data_smithi171_78918_object151393:head# 2024-04-20T17:11:08.373 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:766f8e06:::benchmark_data_smithi171_78918_object52886:head# 2024-04-20T17:11:08.373 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7670dc2d:::benchmark_data_smithi171_78918_object123500:head# 2024-04-20T17:11:08.373 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7670f95c:::benchmark_data_smithi171_78918_object142735:head# 2024-04-20T17:11:08.373 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7672948d:::benchmark_data_smithi171_78918_object67790:head# 2024-04-20T17:11:08.373 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7672abc4:::benchmark_data_smithi171_78918_object114899:head# 2024-04-20T17:11:08.373 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7672b154:::benchmark_data_smithi171_78918_object51464:head# 2024-04-20T17:11:08.373 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7672d815:::benchmark_data_smithi171_78918_object121251:head# 2024-04-20T17:11:08.373 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7673c08b:::benchmark_data_smithi171_78918_object147955:head# 2024-04-20T17:11:08.373 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7674ea5a:::benchmark_data_smithi171_78918_object68233:head# 2024-04-20T17:11:08.373 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76761b69:::benchmark_data_smithi171_78918_object79326:head# 2024-04-20T17:11:08.373 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7676755b:::benchmark_data_smithi171_78918_object53991:head# 2024-04-20T17:11:08.373 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76768391:::benchmark_data_smithi171_78918_object145722:head# 2024-04-20T17:11:08.373 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7676cc3e:::benchmark_data_smithi171_78918_object59372:head# 2024-04-20T17:11:08.373 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7677d4b4:::benchmark_data_smithi171_78918_object52803:head# 2024-04-20T17:11:08.373 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76780b34:::benchmark_data_smithi171_78918_object85673:head# 2024-04-20T17:11:08.374 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76784a02:::benchmark_data_smithi171_78918_object70146:head# 2024-04-20T17:11:08.374 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7679e4c1:::benchmark_data_smithi171_78918_object124542:head# 2024-04-20T17:11:08.374 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:767b09fa:::benchmark_data_smithi171_78918_object153365:head# 2024-04-20T17:11:08.374 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:767b4f19:::benchmark_data_smithi171_78918_object127486:head# 2024-04-20T17:11:08.374 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:767bfde7:::benchmark_data_smithi171_78918_object147234:head# 2024-04-20T17:11:08.374 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:767ceb2e:::benchmark_data_smithi171_78918_object118656:head# 2024-04-20T17:11:08.374 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:767d5763:::benchmark_data_smithi171_78918_object78721:head# 2024-04-20T17:11:08.374 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:767d84a6:::benchmark_data_smithi171_78918_object131434:head# 2024-04-20T17:11:08.374 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:767d853b:::benchmark_data_smithi171_78918_object56636:head# 2024-04-20T17:11:08.374 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:767df3b7:::benchmark_data_smithi171_78918_object141931:head# 2024-04-20T17:11:08.374 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:767df4d2:::benchmark_data_smithi171_78918_object141523:head# 2024-04-20T17:11:08.374 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:767dfa51:::benchmark_data_smithi171_78918_object57285:head# 2024-04-20T17:11:08.374 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:767e0e7c:::benchmark_data_smithi171_78918_object108126:head# 2024-04-20T17:11:08.374 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:767f3f1e:::benchmark_data_smithi171_78918_object115983:head# 2024-04-20T17:11:08.374 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:768090fa:::benchmark_data_smithi171_78918_object75397:head# 2024-04-20T17:11:08.375 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:768308d4:::benchmark_data_smithi171_78918_object93823:head# 2024-04-20T17:11:08.375 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76830c7b:::benchmark_data_smithi171_78918_object158214:head# 2024-04-20T17:11:08.375 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76833c7b:::benchmark_data_smithi171_78918_object152929:head# 2024-04-20T17:11:08.375 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76837fa9:::benchmark_data_smithi171_78918_object82895:head# 2024-04-20T17:11:08.375 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:768383a7:::benchmark_data_smithi171_78918_object152404:head# 2024-04-20T17:11:08.375 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76848aa9:::benchmark_data_smithi171_78918_object115222:head# 2024-04-20T17:11:08.375 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76850c16:::benchmark_data_smithi171_78918_object120855:head# 2024-04-20T17:11:08.375 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76853962:::benchmark_data_smithi171_78918_object98532:head# 2024-04-20T17:11:08.375 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76866e96:::benchmark_data_smithi171_78918_object97771:head# 2024-04-20T17:11:08.375 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76869299:::benchmark_data_smithi171_78918_object117881:head# 2024-04-20T17:11:08.375 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7686b588:::benchmark_data_smithi171_78918_object105266:head# 2024-04-20T17:11:08.375 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7686d6b6:::benchmark_data_smithi171_78918_object155049:head# 2024-04-20T17:11:08.375 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7687a398:::benchmark_data_smithi171_78918_object71628:head# 2024-04-20T17:11:08.375 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7687eec4:::benchmark_data_smithi171_78918_object71426:head# 2024-04-20T17:11:08.375 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7688d2c2:::benchmark_data_smithi171_78918_object128022:head# 2024-04-20T17:11:08.375 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7688f086:::benchmark_data_smithi171_78918_object138666:head# 2024-04-20T17:11:08.376 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7689c218:::benchmark_data_smithi171_78918_object133094:head# 2024-04-20T17:11:08.376 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:768a2836:::benchmark_data_smithi171_78918_object72390:head# 2024-04-20T17:11:08.376 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:768af297:::benchmark_data_smithi171_78918_object103938:head# 2024-04-20T17:11:08.376 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:768ba3a3:::benchmark_data_smithi171_78918_object89462:head# 2024-04-20T17:11:08.376 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:768bb3d3:::benchmark_data_smithi171_78918_object82980:head# 2024-04-20T17:11:08.376 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:768c2542:::benchmark_data_smithi171_78918_object53139:head# 2024-04-20T17:11:08.376 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:768d9e1c:::benchmark_data_smithi171_78918_object137142:head# 2024-04-20T17:11:08.376 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:768de494:::benchmark_data_smithi171_78918_object108175:head# 2024-04-20T17:11:08.376 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:768e9225:::benchmark_data_smithi171_78918_object56776:head# 2024-04-20T17:11:08.376 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:768eebfd:::benchmark_data_smithi171_78918_object67484:head# 2024-04-20T17:11:08.376 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:768f1ba0:::benchmark_data_smithi171_78918_object65919:head# 2024-04-20T17:11:08.376 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:768f70aa:::benchmark_data_smithi171_78918_object128403:head# 2024-04-20T17:11:08.376 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:768fcc3f:::benchmark_data_smithi171_78918_object160429:head# 2024-04-20T17:11:08.376 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:768fe6af:::benchmark_data_smithi171_78918_object128125:head# 2024-04-20T17:11:08.376 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7690825c:::benchmark_data_smithi171_78918_object70632:head# 2024-04-20T17:11:08.377 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7690b711:::benchmark_data_smithi171_78918_object134365:head# 2024-04-20T17:11:08.377 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7690c3fd:::benchmark_data_smithi171_78918_object79543:head# 2024-04-20T17:11:08.377 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7690ca9e:::benchmark_data_smithi171_78918_object98306:head# 2024-04-20T17:11:08.377 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7690f53f:::benchmark_data_smithi171_78918_object55214:head# 2024-04-20T17:11:08.377 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7691b03f:::benchmark_data_smithi171_78918_object137116:head# 2024-04-20T17:11:08.377 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7691ec07:::benchmark_data_smithi171_78918_object75531:head# 2024-04-20T17:11:08.377 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7692c647:::benchmark_data_smithi171_78918_object111318:head# 2024-04-20T17:11:08.377 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76930bf6:::benchmark_data_smithi171_78918_object61993:head# 2024-04-20T17:11:08.377 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76934312:::benchmark_data_smithi171_78918_object141812:head# 2024-04-20T17:11:08.377 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7693746d:::benchmark_data_smithi171_78918_object147616:head# 2024-04-20T17:11:08.377 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76937c46:::benchmark_data_smithi171_78918_object55775:head# 2024-04-20T17:11:08.377 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76938d11:::benchmark_data_smithi171_78918_object141869:head# 2024-04-20T17:11:08.377 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:769416bb:::benchmark_data_smithi171_78918_object139190:head# 2024-04-20T17:11:08.377 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7694f107:::benchmark_data_smithi171_78918_object138197:head# 2024-04-20T17:11:08.377 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7694fc46:::benchmark_data_smithi171_78918_object64761:head# 2024-04-20T17:11:08.377 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7695b375:::benchmark_data_smithi171_78918_object156581:head# 2024-04-20T17:11:08.378 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76971b16:::benchmark_data_smithi171_78918_object82565:head# 2024-04-20T17:11:08.378 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76974081:::benchmark_data_smithi171_78918_object99373:head# 2024-04-20T17:11:08.378 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7697d878:::benchmark_data_smithi171_78918_object147181:head# 2024-04-20T17:11:08.378 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76985828:::benchmark_data_smithi171_78918_object106780:head# 2024-04-20T17:11:08.378 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:769947a3:::benchmark_data_smithi171_78918_object57981:head# 2024-04-20T17:11:08.378 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:769a9e1e:::benchmark_data_smithi171_78918_object80441:head# 2024-04-20T17:11:08.378 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:769b2767:::benchmark_data_smithi171_78918_object105902:head# 2024-04-20T17:11:08.378 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:769c519e:::benchmark_data_smithi171_78918_object67317:head# 2024-04-20T17:11:08.384 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:769c9cbe:::benchmark_data_smithi171_78918_object56769:head# 2024-04-20T17:11:08.384 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:769d6327:::benchmark_data_smithi171_78918_object79428:head# 2024-04-20T17:11:08.384 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:769d99ee:::benchmark_data_smithi171_78918_object86201:head# 2024-04-20T17:11:08.384 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:769dd68d:::benchmark_data_smithi171_78918_object114373:head# 2024-04-20T17:11:08.385 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:769f39f6:::benchmark_data_smithi171_78918_object145017:head# 2024-04-20T17:11:08.385 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76a02770:::benchmark_data_smithi171_78918_object136505:head# 2024-04-20T17:11:08.385 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76a07ad8:::benchmark_data_smithi171_78918_object85121:head# 2024-04-20T17:11:08.385 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76a1104d:::benchmark_data_smithi171_78918_object131673:head# 2024-04-20T17:11:08.385 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76a18780:::benchmark_data_smithi171_78918_object130061:head# 2024-04-20T17:11:08.385 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76a25c24:::benchmark_data_smithi171_78918_object101989:head# 2024-04-20T17:11:08.385 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76a2da82:::benchmark_data_smithi171_78918_object156170:head# 2024-04-20T17:11:08.385 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76a34331:::benchmark_data_smithi171_78918_object152585:head# 2024-04-20T17:11:08.385 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76a3bc2a:::benchmark_data_smithi171_78918_object78289:head# 2024-04-20T17:11:08.385 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76a3ce83:::benchmark_data_smithi171_78918_object53520:head# 2024-04-20T17:11:08.385 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76a3ecb1:::benchmark_data_smithi171_78918_object67721:head# 2024-04-20T17:11:08.385 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76a57e8b:::benchmark_data_smithi171_78918_object117412:head# 2024-04-20T17:11:08.385 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76a5a037:::benchmark_data_smithi171_78918_object143227:head# 2024-04-20T17:11:08.385 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76a5d893:::benchmark_data_smithi171_78918_object55846:head# 2024-04-20T17:11:08.385 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76a8982f:::benchmark_data_smithi171_78918_object148612:head# 2024-04-20T17:11:08.385 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76a90303:::benchmark_data_smithi171_78918_object57672:head# 2024-04-20T17:11:08.386 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76a97483:::benchmark_data_smithi171_78918_object125250:head# 2024-04-20T17:11:08.386 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76aa336f:::benchmark_data_smithi171_78918_object145894:head# 2024-04-20T17:11:08.386 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76aa4431:::benchmark_data_smithi171_78918_object126903:head# 2024-04-20T17:11:08.386 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76aba524:::benchmark_data_smithi171_78918_object115237:head# 2024-04-20T17:11:08.386 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76abe011:::benchmark_data_smithi171_78918_object104601:head# 2024-04-20T17:11:08.386 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76abebd0:::benchmark_data_smithi171_78918_object61354:head# 2024-04-20T17:11:08.386 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76ac700d:::benchmark_data_smithi171_78918_object115474:head# 2024-04-20T17:11:08.386 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76ad360c:::benchmark_data_smithi171_78918_object144108:head# 2024-04-20T17:11:08.386 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76ad506f:::benchmark_data_smithi171_78918_object71446:head# 2024-04-20T17:11:08.386 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76ad7281:::benchmark_data_smithi171_78918_object105223:head# 2024-04-20T17:11:08.386 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76adddcd:::benchmark_data_smithi171_78918_object133655:head# 2024-04-20T17:11:08.386 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76adf0d9:::benchmark_data_smithi171_78918_object143418:head# 2024-04-20T17:11:08.386 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76b1ac4a:::benchmark_data_smithi171_78918_object146426:head# 2024-04-20T17:11:08.386 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76b25cde:::benchmark_data_smithi171_78918_object128640:head# 2024-04-20T17:11:08.387 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76b316b5:::benchmark_data_smithi171_78918_object61490:head# 2024-04-20T17:11:08.387 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76b34253:::benchmark_data_smithi171_78918_object62748:head# 2024-04-20T17:11:08.387 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76b3a1a3:::benchmark_data_smithi171_78918_object93752:head# 2024-04-20T17:11:08.387 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76b3bedf:::benchmark_data_smithi171_78918_object153629:head# 2024-04-20T17:11:08.387 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76b40982:::benchmark_data_smithi171_78918_object109794:head# 2024-04-20T17:11:08.387 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76b42455:::benchmark_data_smithi171_78918_object104420:head# 2024-04-20T17:11:08.387 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76b479a6:::benchmark_data_smithi171_78918_object136681:head# 2024-04-20T17:11:08.387 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76b5e234:::benchmark_data_smithi171_78918_object60553:head# 2024-04-20T17:11:08.387 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76b6a2ba:::benchmark_data_smithi171_78918_object86817:head# 2024-04-20T17:11:08.387 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76b6b376:::benchmark_data_smithi171_78918_object87061:head# 2024-04-20T17:11:08.387 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76b75f1b:::benchmark_data_smithi171_78918_object71755:head# 2024-04-20T17:11:08.387 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76b7ac59:::benchmark_data_smithi171_78918_object104520:head# 2024-04-20T17:11:08.387 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76b82f95:::benchmark_data_smithi171_78918_object63045:head# 2024-04-20T17:11:08.387 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76b898c4:::benchmark_data_smithi171_78918_object72436:head# 2024-04-20T17:11:08.387 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76b8aecf:::benchmark_data_smithi171_78918_object92116:head# 2024-04-20T17:11:08.388 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76ba6831:::benchmark_data_smithi171_78918_object66214:head# 2024-04-20T17:11:08.388 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76ba9165:::benchmark_data_smithi171_78918_object116356:head# 2024-04-20T17:11:08.388 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76bb9763:::benchmark_data_smithi171_78918_object134809:head# 2024-04-20T17:11:08.388 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76bc41f2:::benchmark_data_smithi171_78918_object96688:head# 2024-04-20T17:11:08.388 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76c002d5:::benchmark_data_smithi171_78918_object74287:head# 2024-04-20T17:11:08.388 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76c01612:::benchmark_data_smithi171_78918_object147586:head# 2024-04-20T17:11:08.388 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76c09396:::benchmark_data_smithi171_78918_object120604:head# 2024-04-20T17:11:08.388 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76c0f2ff:::benchmark_data_smithi171_78918_object151555:head# 2024-04-20T17:11:08.388 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76c1c664:::benchmark_data_smithi171_78918_object58057:head# 2024-04-20T17:11:08.388 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76c21f3e:::benchmark_data_smithi171_78918_object92667:head# 2024-04-20T17:11:08.388 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76c2b9af:::benchmark_data_smithi171_78918_object123946:head# 2024-04-20T17:11:08.388 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76c48d94:::benchmark_data_smithi171_78918_object124997:head# 2024-04-20T17:11:08.388 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76c6362b:::benchmark_data_smithi171_78918_object58860:head# 2024-04-20T17:11:08.388 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76c69fff:::benchmark_data_smithi171_78918_object76297:head# 2024-04-20T17:11:08.388 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76c6b046:::benchmark_data_smithi171_78918_object155693:head# 2024-04-20T17:11:08.389 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76c8d30d:::benchmark_data_smithi171_78918_object138892:head# 2024-04-20T17:11:08.389 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76c8e412:::benchmark_data_smithi171_78918_object149474:head# 2024-04-20T17:11:08.394 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76c927d8:::benchmark_data_smithi171_78918_object89798:head# 2024-04-20T17:11:08.394 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76c9fb1b:::benchmark_data_smithi171_78918_object79560:head# 2024-04-20T17:11:08.394 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76cabb26:::benchmark_data_smithi171_78918_object132288:head# 2024-04-20T17:11:08.394 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76cc77ad:::benchmark_data_smithi171_78918_object55960:head# 2024-04-20T17:11:08.394 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76ce47d5:::benchmark_data_smithi171_78918_object101853:head# 2024-04-20T17:11:08.394 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76ce7449:::benchmark_data_smithi171_78918_object109567:head# 2024-04-20T17:11:08.394 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76ce92c6:::benchmark_data_smithi171_78918_object109676:head# 2024-04-20T17:11:08.395 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76ced28d:::benchmark_data_smithi171_78918_object150144:head# 2024-04-20T17:11:08.395 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76cf40ea:::benchmark_data_smithi171_78918_object62716:head# 2024-04-20T17:11:08.395 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76d00da3:::benchmark_data_smithi171_78918_object144104:head# 2024-04-20T17:11:08.395 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76d120c9:::benchmark_data_smithi171_78918_object101035:head# 2024-04-20T17:11:08.395 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76d19510:::benchmark_data_smithi171_78918_object111330:head# 2024-04-20T17:11:08.395 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76d24cc5:::benchmark_data_smithi171_78918_object90712:head# 2024-04-20T17:11:08.395 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76d26b81:::benchmark_data_smithi171_78918_object114671:head# 2024-04-20T17:11:08.395 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76d29afd:::benchmark_data_smithi171_78918_object113473:head# 2024-04-20T17:11:08.395 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76d33096:::benchmark_data_smithi171_78918_object61459:head# 2024-04-20T17:11:08.395 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76d36a25:::benchmark_data_smithi171_78918_object134012:head# 2024-04-20T17:11:08.395 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76d38e40:::benchmark_data_smithi171_78918_object65645:head# 2024-04-20T17:11:08.395 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76d39073:::benchmark_data_smithi171_78918_object110193:head# 2024-04-20T17:11:08.395 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76d41cf2:::benchmark_data_smithi171_78918_object136459:head# 2024-04-20T17:11:08.396 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76d64c89:::benchmark_data_smithi171_78918_object70121:head# 2024-04-20T17:11:08.396 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76d76276:::benchmark_data_smithi171_78918_object108244:head# 2024-04-20T17:11:08.396 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76d7e50b:::benchmark_data_smithi171_78918_object60345:head# 2024-04-20T17:11:08.396 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76d7f30e:::benchmark_data_smithi171_78918_object65083:head# 2024-04-20T17:11:08.396 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76d84b18:::benchmark_data_smithi171_78918_object119049:head# 2024-04-20T17:11:08.396 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76d8c805:::benchmark_data_smithi171_78918_object137838:head# 2024-04-20T17:11:08.396 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76d8d3f3:::benchmark_data_smithi171_78918_object127229:head# 2024-04-20T17:11:08.396 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76d8d41b:::benchmark_data_smithi171_78918_object123215:head# 2024-04-20T17:11:08.396 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76d9b451:::benchmark_data_smithi171_78918_object104860:head# 2024-04-20T17:11:08.396 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76dbc447:::benchmark_data_smithi171_78918_object160233:head# 2024-04-20T17:11:08.396 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76dc0719:::benchmark_data_smithi171_78918_object150381:head# 2024-04-20T17:11:08.396 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76ddb5b3:::benchmark_data_smithi171_78918_object137313:head# 2024-04-20T17:11:08.396 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76ddd978:::benchmark_data_smithi171_78918_object54412:head# 2024-04-20T17:11:08.396 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76ddedaf:::benchmark_data_smithi171_78918_object74415:head# 2024-04-20T17:11:08.397 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76de1def:::benchmark_data_smithi171_78918_object134749:head# 2024-04-20T17:11:08.397 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76de6811:::benchmark_data_smithi171_78918_object109433:head# 2024-04-20T17:11:08.397 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76df7e26:::benchmark_data_smithi171_78918_object134025:head# 2024-04-20T17:11:08.397 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76e0063a:::benchmark_data_smithi171_78918_object96741:head# 2024-04-20T17:11:08.397 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76e095a8:::benchmark_data_smithi171_78918_object123438:head# 2024-04-20T17:11:08.397 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76e0bd19:::benchmark_data_smithi171_78918_object130274:head# 2024-04-20T17:11:08.397 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76e14eea:::benchmark_data_smithi171_78918_object129415:head# 2024-04-20T17:11:08.397 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76e22924:::benchmark_data_smithi171_78918_object104846:head# 2024-04-20T17:11:08.397 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76e41161:::benchmark_data_smithi171_78918_object76341:head# 2024-04-20T17:11:08.397 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76e584b5:::benchmark_data_smithi171_78918_object57435:head# 2024-04-20T17:11:08.397 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76e704b3:::benchmark_data_smithi171_78918_object53221:head# 2024-04-20T17:11:08.397 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76e797c4:::benchmark_data_smithi171_78918_object57638:head# 2024-04-20T17:11:08.397 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76e7f387:::benchmark_data_smithi171_78918_object139364:head# 2024-04-20T17:11:08.397 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76e8ca04:::benchmark_data_smithi171_78918_object81938:head# 2024-04-20T17:11:08.397 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76e9816e:::benchmark_data_smithi171_78918_object102028:head# 2024-04-20T17:11:08.398 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76e9f8e7:::benchmark_data_smithi171_78918_object142694:head# 2024-04-20T17:11:08.398 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76e9ff36:::benchmark_data_smithi171_78918_object133092:head# 2024-04-20T17:11:08.398 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76ea0602:::benchmark_data_smithi171_78918_object151473:head# 2024-04-20T17:11:08.398 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76ea90f9:::benchmark_data_smithi171_78918_object86581:head# 2024-04-20T17:11:08.398 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76eaacd0:::benchmark_data_smithi171_78918_object75302:head# 2024-04-20T17:11:08.398 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76eb0dbe:::benchmark_data_smithi171_78918_object119007:head# 2024-04-20T17:11:08.398 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76ec7b48:::benchmark_data_smithi171_78918_object90899:head# 2024-04-20T17:11:08.402 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76ecc1db:::benchmark_data_smithi171_78918_object68317:head# 2024-04-20T17:11:08.402 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76ed8ebc:::benchmark_data_smithi171_78918_object157288:head# 2024-04-20T17:11:08.402 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76eee17d:::benchmark_data_smithi171_78918_object74500:head# 2024-04-20T17:11:08.402 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76ef030c:::benchmark_data_smithi171_78918_object69339:head# 2024-04-20T17:11:08.402 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76ef0ea6:::benchmark_data_smithi171_78918_object59165:head# 2024-04-20T17:11:08.402 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76ef268b:::benchmark_data_smithi171_78918_object118202:head# 2024-04-20T17:11:08.402 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76f0535f:::benchmark_data_smithi171_78918_object63163:head# 2024-04-20T17:11:08.402 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76f11063:::benchmark_data_smithi171_78918_object86064:head# 2024-04-20T17:11:08.403 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76f19101:::benchmark_data_smithi171_78918_object137288:head# 2024-04-20T17:11:08.403 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76f230bc:::benchmark_data_smithi171_78918_object91420:head# 2024-04-20T17:11:08.403 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76f27194:::benchmark_data_smithi171_78918_object87727:head# 2024-04-20T17:11:08.403 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76f2779b:::benchmark_data_smithi171_78918_object129055:head# 2024-04-20T17:11:08.403 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76f37afe:::benchmark_data_smithi171_78918_object152151:head# 2024-04-20T17:11:08.403 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76f3a73e:::benchmark_data_smithi171_78918_object55593:head# 2024-04-20T17:11:08.403 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76f5a4bd:::benchmark_data_smithi171_78918_object104090:head# 2024-04-20T17:11:08.403 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76f5f948:::benchmark_data_smithi171_78918_object143545:head# 2024-04-20T17:11:08.403 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76f723be:::benchmark_data_smithi171_78918_object91452:head# 2024-04-20T17:11:08.403 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76f74aa8:::benchmark_data_smithi171_78918_object137213:head# 2024-04-20T17:11:08.403 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76f765f6:::benchmark_data_smithi171_78918_object56624:head# 2024-04-20T17:11:08.403 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76f7fd08:::benchmark_data_smithi171_78918_object156759:head# 2024-04-20T17:11:08.403 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76f83544:::benchmark_data_smithi171_78918_object139569:head# 2024-04-20T17:11:08.404 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76f89d3b:::benchmark_data_smithi171_78918_object121507:head# 2024-04-20T17:11:08.404 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76f8e88e:::benchmark_data_smithi171_78918_object151867:head# 2024-04-20T17:11:08.404 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76f8ecab:::benchmark_data_smithi171_78918_object121034:head# 2024-04-20T17:11:08.404 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76f97cde:::benchmark_data_smithi171_78918_object97079:head# 2024-04-20T17:11:08.404 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76f9bd7a:::benchmark_data_smithi171_78918_object150039:head# 2024-04-20T17:11:08.404 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76fa11f0:::benchmark_data_smithi171_78918_object112960:head# 2024-04-20T17:11:08.404 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76fa94e5:::benchmark_data_smithi171_78918_object115549:head# 2024-04-20T17:11:08.404 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76facc8a:::benchmark_data_smithi171_78918_object134800:head# 2024-04-20T17:11:08.404 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76faf8f5:::benchmark_data_smithi171_78918_object144411:head# 2024-04-20T17:11:08.404 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76fb163a:::benchmark_data_smithi171_78918_object155632:head# 2024-04-20T17:11:08.404 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76fb7d33:::benchmark_data_smithi171_78918_object143672:head# 2024-04-20T17:11:08.404 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76fbc398:::benchmark_data_smithi171_78918_object69917:head# 2024-04-20T17:11:08.404 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76fc3108:::benchmark_data_smithi171_78918_object136953:head# 2024-04-20T17:11:08.404 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76fd5cdf:::benchmark_data_smithi171_78918_object72780:head# 2024-04-20T17:11:08.404 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76fd6b1f:::benchmark_data_smithi171_78918_object157744:head# 2024-04-20T17:11:08.405 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76fe17c3:::benchmark_data_smithi171_78918_object72497:head# 2024-04-20T17:11:08.405 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76fe342a:::benchmark_data_smithi171_78918_object71492:head# 2024-04-20T17:11:08.405 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76fe7d16:::benchmark_data_smithi171_78918_object128176:head# 2024-04-20T17:11:08.405 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:76ffd846:::benchmark_data_smithi171_78918_object76577:head# 2024-04-20T17:11:08.405 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7700dc7c:::benchmark_data_smithi171_78918_object105439:head# 2024-04-20T17:11:08.408 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7700f02e:::benchmark_data_smithi171_78918_object122149:head# 2024-04-20T17:11:08.409 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77012bf3:::benchmark_data_smithi171_78918_object158077:head# 2024-04-20T17:11:08.409 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77019136:::benchmark_data_smithi171_78918_object117030:head# 2024-04-20T17:11:08.409 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77027bca:::benchmark_data_smithi171_78918_object73961:head# 2024-04-20T17:11:08.409 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77040a5e:::benchmark_data_smithi171_78918_object103324:head# 2024-04-20T17:11:08.409 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77050d22:::benchmark_data_smithi171_78918_object81375:head# 2024-04-20T17:11:08.409 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77051296:::benchmark_data_smithi171_78918_object56957:head# 2024-04-20T17:11:08.409 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77054e12:::benchmark_data_smithi171_78918_object104522:head# 2024-04-20T17:11:08.409 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7706ea88:::benchmark_data_smithi171_78918_object60043:head# 2024-04-20T17:11:08.409 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7707614a:::benchmark_data_smithi171_78918_object86770:head# 2024-04-20T17:11:08.409 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7707bace:::benchmark_data_smithi171_78918_object94443:head# 2024-04-20T17:11:08.409 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:770820d4:::benchmark_data_smithi171_78918_object118872:head# 2024-04-20T17:11:08.409 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7708af2f:::benchmark_data_smithi171_78918_object54941:head# 2024-04-20T17:11:08.409 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7708ea0a:::benchmark_data_smithi171_78918_object141654:head# 2024-04-20T17:11:08.409 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:770969c8:::benchmark_data_smithi171_78918_object107191:head# 2024-04-20T17:11:08.409 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:770b6b18:::benchmark_data_smithi171_78918_object121047:head# 2024-04-20T17:11:08.410 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:770b8ed5:::benchmark_data_smithi171_78918_object150787:head# 2024-04-20T17:11:08.410 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:770be7c7:::benchmark_data_smithi171_78918_object111623:head# 2024-04-20T17:11:08.410 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:770c6579:::benchmark_data_smithi171_78918_object149967:head# 2024-04-20T17:11:08.410 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:770cad40:::benchmark_data_smithi171_78918_object58425:head# 2024-04-20T17:11:08.410 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:770db127:::benchmark_data_smithi171_78918_object64206:head# 2024-04-20T17:11:08.410 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:770dce0f:::benchmark_data_smithi171_78918_object95466:head# 2024-04-20T17:11:08.410 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:770ea725:::benchmark_data_smithi171_78918_object149281:head# 2024-04-20T17:11:08.410 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:770f849d:::benchmark_data_smithi171_78918_object81661:head# 2024-04-20T17:11:08.410 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77102f9e:::benchmark_data_smithi171_78918_object58478:head# 2024-04-20T17:11:08.410 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:771175f7:::benchmark_data_smithi171_78918_object148455:head# 2024-04-20T17:11:08.410 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7711c7d5:::benchmark_data_smithi171_78918_object153688:head# 2024-04-20T17:11:08.410 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7711cc10:::benchmark_data_smithi171_78918_object136755:head# 2024-04-20T17:11:08.410 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7712ab32:::benchmark_data_smithi171_78918_object131055:head# 2024-04-20T17:11:08.410 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:771384a0:::benchmark_data_smithi171_78918_object146631:head# 2024-04-20T17:11:08.410 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77145c62:::benchmark_data_smithi171_78918_object158839:head# 2024-04-20T17:11:08.411 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7715c3b6:::benchmark_data_smithi171_78918_object145304:head# 2024-04-20T17:11:08.411 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:771660a4:::benchmark_data_smithi171_78918_object79801:head# 2024-04-20T17:11:08.411 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7716ede2:::benchmark_data_smithi171_78918_object121163:head# 2024-04-20T17:11:08.411 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7717bf43:::benchmark_data_smithi171_78918_object95172:head# 2024-04-20T17:11:08.412 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7718d5c6:::benchmark_data_smithi171_78918_object158175:head# 2024-04-20T17:11:08.413 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7718d88a:::benchmark_data_smithi171_78918_object150040:head# 2024-04-20T17:11:08.413 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77192f47:::benchmark_data_smithi171_78918_object73036:head# 2024-04-20T17:11:08.413 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:771a18ff:::benchmark_data_smithi171_78918_object62045:head# 2024-04-20T17:11:08.413 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:771b250c:::benchmark_data_smithi171_78918_object89950:head# 2024-04-20T17:11:08.413 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:771bc468:::benchmark_data_smithi171_78918_object105060:head# 2024-04-20T17:11:08.413 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:771c4a1f:::benchmark_data_smithi171_78918_object90036:head# 2024-04-20T17:11:08.413 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:771c61e0:::benchmark_data_smithi171_78918_object128091:head# 2024-04-20T17:11:08.413 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:771ccb88:::benchmark_data_smithi171_78918_object149765:head# 2024-04-20T17:11:08.413 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:771d0884:::benchmark_data_smithi171_78918_object137238:head# 2024-04-20T17:11:08.413 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:771ef6f7:::benchmark_data_smithi171_78918_object59064:head# 2024-04-20T17:11:08.413 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:771ff16a:::benchmark_data_smithi171_78918_object124423:head# 2024-04-20T17:11:08.413 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77208b60:::benchmark_data_smithi171_78918_object141051:head# 2024-04-20T17:11:08.413 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7720da55:::benchmark_data_smithi171_78918_object99602:head# 2024-04-20T17:11:08.414 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7720fb79:::benchmark_data_smithi171_78918_object157160:head# 2024-04-20T17:11:08.414 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7721257b:::benchmark_data_smithi171_78918_object57543:head# 2024-04-20T17:11:08.414 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7722f2f1:::benchmark_data_smithi171_78918_object77296:head# 2024-04-20T17:11:08.417 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77233c29:::benchmark_data_smithi171_78918_object115278:head# 2024-04-20T17:11:08.418 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7724957c:::benchmark_data_smithi171_78918_object81321:head# 2024-04-20T17:11:08.418 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7724977c:::benchmark_data_smithi171_78918_object160253:head# 2024-04-20T17:11:08.418 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7724bff1:::benchmark_data_smithi171_78918_object55184:head# 2024-04-20T17:11:08.418 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77259bcd:::benchmark_data_smithi171_78918_object122074:head# 2024-04-20T17:11:08.418 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:772673ac:::benchmark_data_smithi171_78918_object141821:head# 2024-04-20T17:11:08.418 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7726d354:::benchmark_data_smithi171_78918_object76324:head# 2024-04-20T17:11:08.418 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77288cf1:::benchmark_data_smithi171_78918_object159377:head# 2024-04-20T17:11:08.418 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77296e7c:::benchmark_data_smithi171_78918_object53005:head# 2024-04-20T17:11:08.418 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77299da1:::benchmark_data_smithi171_78918_object87024:head# 2024-04-20T17:11:08.418 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:772b0a43:::benchmark_data_smithi171_78918_object65162:head# 2024-04-20T17:11:08.418 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:772b37d1:::benchmark_data_smithi171_78918_object53900:head# 2024-04-20T17:11:08.418 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:772bc92b:::benchmark_data_smithi171_78918_object76712:head# 2024-04-20T17:11:08.418 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:772c8dbb:::benchmark_data_smithi171_78918_object78166:head# 2024-04-20T17:11:08.419 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:772c9105:::benchmark_data_smithi171_78918_object95755:head# 2024-04-20T17:11:08.419 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:772ce678:::benchmark_data_smithi171_78918_object134358:head# 2024-04-20T17:11:08.419 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:772d4236:::benchmark_data_smithi171_78918_object133237:head# 2024-04-20T17:11:08.419 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:772dd52b:::benchmark_data_smithi171_78918_object78182:head# 2024-04-20T17:11:08.419 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:772e132e:::benchmark_data_smithi171_78918_object158559:head# 2024-04-20T17:11:08.419 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:772ecb72:::benchmark_data_smithi171_78918_object100311:head# 2024-04-20T17:11:08.419 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7730b7b7:::benchmark_data_smithi171_78918_object87687:head# 2024-04-20T17:11:08.419 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7730bd7b:::benchmark_data_smithi171_78918_object122804:head# 2024-04-20T17:11:08.419 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7731424d:::benchmark_data_smithi171_78918_object117450:head# 2024-04-20T17:11:08.419 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:773143a6:::benchmark_data_smithi171_78918_object83836:head# 2024-04-20T17:11:08.419 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77314686:::benchmark_data_smithi171_78918_object90947:head# 2024-04-20T17:11:08.419 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7731702a:::benchmark_data_smithi171_78918_object126808:head# 2024-04-20T17:11:08.419 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:773176cb:::benchmark_data_smithi171_78918_object58768:head# 2024-04-20T17:11:08.421 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77320bbf:::benchmark_data_smithi171_78918_object81169:head# 2024-04-20T17:11:08.421 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77325705:::benchmark_data_smithi171_78918_object126725:head# 2024-04-20T17:11:08.422 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7733f09b:::benchmark_data_smithi171_78918_object63040:head# 2024-04-20T17:11:08.422 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77340b22:::benchmark_data_smithi171_78918_object92500:head# 2024-04-20T17:11:08.422 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7734daec:::benchmark_data_smithi171_78918_object75985:head# 2024-04-20T17:11:08.422 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7735a850:::benchmark_data_smithi171_78918_object82691:head# 2024-04-20T17:11:08.422 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7735b579:::benchmark_data_smithi171_78918_object122105:head# 2024-04-20T17:11:08.422 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:773634b9:::benchmark_data_smithi171_78918_object120658:head# 2024-04-20T17:11:08.422 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77369bdd:::benchmark_data_smithi171_78918_object139054:head# 2024-04-20T17:11:08.422 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7736f1ed:::benchmark_data_smithi171_78918_object122671:head# 2024-04-20T17:11:08.422 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7736f39d:::benchmark_data_smithi171_78918_object63421:head# 2024-04-20T17:11:08.422 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:773790f8:::benchmark_data_smithi171_78918_object72936:head# 2024-04-20T17:11:08.422 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7737b555:::benchmark_data_smithi171_78918_object155151:head# 2024-04-20T17:11:08.422 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:773803df:::benchmark_data_smithi171_78918_object160688:head# 2024-04-20T17:11:08.422 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77384bba:::benchmark_data_smithi171_78918_object110259:head# 2024-04-20T17:11:08.422 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77385538:::benchmark_data_smithi171_78918_object95941:head# 2024-04-20T17:11:08.423 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7739eff9:::benchmark_data_smithi171_78918_object88025:head# 2024-04-20T17:11:08.424 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7739fd6d:::benchmark_data_smithi171_78918_object125730:head# 2024-04-20T17:11:08.424 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:773aaa76:::benchmark_data_smithi171_78918_object84550:head# 2024-04-20T17:11:08.424 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:773b136b:::benchmark_data_smithi171_78918_object157396:head# 2024-04-20T17:11:08.424 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:773e591e:::benchmark_data_smithi171_78918_object153480:head# 2024-04-20T17:11:08.424 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:773eb55a:::benchmark_data_smithi171_78918_object72749:head# 2024-04-20T17:11:08.425 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:773f5027:::benchmark_data_smithi171_78918_object133587:head# 2024-04-20T17:11:08.425 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7740c137:::benchmark_data_smithi171_78918_object93046:head# 2024-04-20T17:11:08.425 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:774175b7:::benchmark_data_smithi171_78918_object115057:head# 2024-04-20T17:11:08.425 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7741e919:::benchmark_data_smithi171_78918_object68687:head# 2024-04-20T17:11:08.425 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77423b83:::benchmark_data_smithi171_78918_object92705:head# 2024-04-20T17:11:08.425 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77446e24:::benchmark_data_smithi171_78918_object130991:head# 2024-04-20T17:11:08.426 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7744d8cd:::benchmark_data_smithi171_78918_object87553:head# 2024-04-20T17:11:08.426 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77455849:::benchmark_data_smithi171_78918_object120883:head# 2024-04-20T17:11:08.426 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77456b53:::benchmark_data_smithi171_78918_object155121:head# 2024-04-20T17:11:08.426 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:774882c2:::benchmark_data_smithi171_78918_object81670:head# 2024-04-20T17:11:08.426 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77493b00:::benchmark_data_smithi171_78918_object105484:head# 2024-04-20T17:11:08.426 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7749a976:::benchmark_data_smithi171_78918_object76396:head# 2024-04-20T17:11:08.426 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:774a650e:::benchmark_data_smithi171_78918_object81007:head# 2024-04-20T17:11:08.426 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:774be869:::benchmark_data_smithi171_78918_object103401:head# 2024-04-20T17:11:08.430 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:774c5dc1:::benchmark_data_smithi171_78918_object153380:head# 2024-04-20T17:11:08.430 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:774cef92:::benchmark_data_smithi171_78918_object137491:head# 2024-04-20T17:11:08.430 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:774de089:::benchmark_data_smithi171_78918_object124115:head# 2024-04-20T17:11:08.430 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:774e3d97:::benchmark_data_smithi171_78918_object152895:head# 2024-04-20T17:11:08.430 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:774e58df:::benchmark_data_smithi171_78918_object160953:head# 2024-04-20T17:11:08.430 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:774e5d43:::benchmark_data_smithi171_78918_object62105:head# 2024-04-20T17:11:08.430 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:774e7353:::benchmark_data_smithi171_78918_object76070:head# 2024-04-20T17:11:08.430 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:774ec25c:::benchmark_data_smithi171_78918_object131647:head# 2024-04-20T17:11:08.430 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:774eee72:::benchmark_data_smithi171_78918_object67501:head# 2024-04-20T17:11:08.430 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:774faacb:::benchmark_data_smithi171_78918_object141891:head# 2024-04-20T17:11:08.430 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:774fb006:::benchmark_data_smithi171_78918_object101926:head# 2024-04-20T17:11:08.431 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:774fbe12:::benchmark_data_smithi171_78918_object58027:head# 2024-04-20T17:11:08.431 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:774ffaea:::benchmark_data_smithi171_78918_object70368:head# 2024-04-20T17:11:08.431 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7750611d:::benchmark_data_smithi171_78918_object110969:head# 2024-04-20T17:11:08.431 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:775127aa:::benchmark_data_smithi171_78918_object72217:head# 2024-04-20T17:11:08.431 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77516c1c:::benchmark_data_smithi171_78918_object148070:head# 2024-04-20T17:11:08.431 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7751e5b7:::benchmark_data_smithi171_78918_object143764:head# 2024-04-20T17:11:08.431 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77522d1b:::benchmark_data_smithi171_78918_object128165:head# 2024-04-20T17:11:08.431 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7752327b:::benchmark_data_smithi171_78918_object131655:head# 2024-04-20T17:11:08.442 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77527306:::benchmark_data_smithi171_78918_object112799:head# 2024-04-20T17:11:08.442 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77527f03:::benchmark_data_smithi171_78918_object73792:head# 2024-04-20T17:11:08.442 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7752e338:::benchmark_data_smithi171_78918_object86992:head# 2024-04-20T17:11:08.442 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7752ff3e:::benchmark_data_smithi171_78918_object154478:head# 2024-04-20T17:11:08.443 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77545c5f:::benchmark_data_smithi171_78918_object131207:head# 2024-04-20T17:11:08.443 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:775480a3:::benchmark_data_smithi171_78918_object71052:head# 2024-04-20T17:11:08.443 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77557df3:::benchmark_data_smithi171_78918_object67320:head# 2024-04-20T17:11:08.443 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77558290:::benchmark_data_smithi171_78918_object60824:head# 2024-04-20T17:11:08.443 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7755d51b:::benchmark_data_smithi171_78918_object115104:head# 2024-04-20T17:11:08.443 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7755dcbc:::benchmark_data_smithi171_78918_object74847:head# 2024-04-20T17:11:08.443 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:775690e8:::benchmark_data_smithi171_78918_object88568:head# 2024-04-20T17:11:08.443 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77574816:::benchmark_data_smithi171_78918_object92953:head# 2024-04-20T17:11:08.443 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7758d513:::benchmark_data_smithi171_78918_object75672:head# 2024-04-20T17:11:08.443 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:775910ce:::benchmark_data_smithi171_78918_object120437:head# 2024-04-20T17:11:08.443 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:775a2798:::benchmark_data_smithi171_78918_object112822:head# 2024-04-20T17:11:08.443 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:775c613c:::benchmark_data_smithi171_78918_object126809:head# 2024-04-20T17:11:08.443 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:775d0331:::benchmark_data_smithi171_78918_object70490:head# 2024-04-20T17:11:08.443 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:775d8234:::benchmark_data_smithi171_78918_object64024:head# 2024-04-20T17:11:08.443 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:775dddbd:::benchmark_data_smithi171_78918_object160267:head# 2024-04-20T17:11:08.444 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:775e1316:::benchmark_data_smithi171_78918_object143542:head# 2024-04-20T17:11:08.444 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:775e1e36:::benchmark_data_smithi171_78918_object64280:head# 2024-04-20T17:11:08.444 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:775e4cdd:::benchmark_data_smithi171_78918_object157153:head# 2024-04-20T17:11:08.444 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:775fac77:::benchmark_data_smithi171_78918_object68280:head# 2024-04-20T17:11:08.444 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77604786:::benchmark_data_smithi171_78918_object92487:head# 2024-04-20T17:11:08.444 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7760c053:::benchmark_data_smithi171_78918_object70308:head# 2024-04-20T17:11:08.444 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7760eb50:::benchmark_data_smithi171_78918_object88635:head# 2024-04-20T17:11:08.444 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:776100a0:::benchmark_data_smithi171_78918_object58767:head# 2024-04-20T17:11:08.444 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77611120:::benchmark_data_smithi171_78918_object85340:head# 2024-04-20T17:11:08.444 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77614197:::benchmark_data_smithi171_78918_object59325:head# 2024-04-20T17:11:08.444 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77625ff8:::benchmark_data_smithi171_78918_object67656:head# 2024-04-20T17:11:08.444 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7762900f:::benchmark_data_smithi171_78918_object96154:head# 2024-04-20T17:11:08.444 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7762ad8f:::benchmark_data_smithi171_78918_object86272:head# 2024-04-20T17:11:08.444 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7762b080:::benchmark_data_smithi171_78918_object115754:head# 2024-04-20T17:11:08.445 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7763f760:::benchmark_data_smithi171_78918_object126035:head# 2024-04-20T17:11:08.445 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7764fe41:::benchmark_data_smithi171_78918_object143088:head# 2024-04-20T17:11:08.445 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:776544f6:::benchmark_data_smithi171_78918_object96738:head# 2024-04-20T17:11:08.445 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77660b97:::benchmark_data_smithi171_78918_object117743:head# 2024-04-20T17:11:08.445 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77669b87:::benchmark_data_smithi171_78918_object114815:head# 2024-04-20T17:11:08.445 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7766bb9e:::benchmark_data_smithi171_78918_object96479:head# 2024-04-20T17:11:08.445 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77673392:::benchmark_data_smithi171_78918_object86792:head# 2024-04-20T17:11:08.445 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7767450c:::benchmark_data_smithi171_78918_object89117:head# 2024-04-20T17:11:08.445 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:776762e0:::benchmark_data_smithi171_78918_object67545:head# 2024-04-20T17:11:08.445 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:776797d0:::benchmark_data_smithi171_78918_object63568:head# 2024-04-20T17:11:08.445 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7767a538:::benchmark_data_smithi171_78918_object63603:head# 2024-04-20T17:11:08.445 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7767a86e:::benchmark_data_smithi171_78918_object62535:head# 2024-04-20T17:11:08.445 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7767aa8e:::benchmark_data_smithi171_78918_object152129:head# 2024-04-20T17:11:08.445 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7767d82c:::benchmark_data_smithi171_78918_object110183:head# 2024-04-20T17:11:08.445 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77697318:::benchmark_data_smithi171_78918_object144706:head# 2024-04-20T17:11:08.446 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:776b3789:::benchmark_data_smithi171_78918_object158696:head# 2024-04-20T17:11:08.446 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:776bbca6:::benchmark_data_smithi171_78918_object133612:head# 2024-04-20T17:11:08.446 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:776be6cf:::benchmark_data_smithi171_78918_object55548:head# 2024-04-20T17:11:08.446 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:776c17f1:::benchmark_data_smithi171_78918_object132746:head# 2024-04-20T17:11:08.446 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:776cf86f:::benchmark_data_smithi171_78918_object69102:head# 2024-04-20T17:11:08.446 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:776e67d7:::benchmark_data_smithi171_78918_object64258:head# 2024-04-20T17:11:08.446 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:776e6ac9:::benchmark_data_smithi171_78918_object138654:head# 2024-04-20T17:11:08.446 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:776f41d8:::benchmark_data_smithi171_78918_object157657:head# 2024-04-20T17:11:08.446 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77704173:::benchmark_data_smithi171_78918_object141086:head# 2024-04-20T17:11:08.446 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:777088ee:::benchmark_data_smithi171_78918_object130195:head# 2024-04-20T17:11:08.446 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7770cd27:::benchmark_data_smithi171_78918_object72256:head# 2024-04-20T17:11:08.446 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77714498:::benchmark_data_smithi171_78918_object141861:head# 2024-04-20T17:11:08.446 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77723962:::benchmark_data_smithi171_78918_object67873:head# 2024-04-20T17:11:08.446 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77727dcc:::benchmark_data_smithi171_78918_object93755:head# 2024-04-20T17:11:08.447 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7772840c:::benchmark_data_smithi171_78918_object139653:head# 2024-04-20T17:11:08.447 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77729b0d:::benchmark_data_smithi171_78918_object75798:head# 2024-04-20T17:11:08.447 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77733511:::benchmark_data_smithi171_78918_object66894:head# 2024-04-20T17:11:08.447 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7773aab7:::benchmark_data_smithi171_78918_object74503:head# 2024-04-20T17:11:08.460 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7773f59e:::benchmark_data_smithi171_78918_object100454:head# 2024-04-20T17:11:08.460 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77741ad6:::benchmark_data_smithi171_78918_object90206:head# 2024-04-20T17:11:08.460 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:777514c4:::benchmark_data_smithi171_78918_object74941:head# 2024-04-20T17:11:08.460 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:777536c1:::benchmark_data_smithi171_78918_object150943:head# 2024-04-20T17:11:08.460 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77756101:::benchmark_data_smithi171_78918_object119768:head# 2024-04-20T17:11:08.460 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77760d33:::benchmark_data_smithi171_78918_object135577:head# 2024-04-20T17:11:08.460 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77763282:::benchmark_data_smithi171_78918_object88272:head# 2024-04-20T17:11:08.461 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77767ae1:::benchmark_data_smithi171_78918_object118976:head# 2024-04-20T17:11:08.461 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7776eab1:::benchmark_data_smithi171_78918_object120085:head# 2024-04-20T17:11:08.461 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77790df1:::benchmark_data_smithi171_78918_object69254:head# 2024-04-20T17:11:08.461 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7779a018:::benchmark_data_smithi171_78918_object102437:head# 2024-04-20T17:11:08.461 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:777a2c5f:::benchmark_data_smithi171_78918_object84353:head# 2024-04-20T17:11:08.461 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:777aa8d0:::benchmark_data_smithi171_78918_object158944:head# 2024-04-20T17:11:08.461 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:777b50f6:::benchmark_data_smithi171_78918_object138119:head# 2024-04-20T17:11:08.461 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:777bec65:::benchmark_data_smithi171_78918_object143814:head# 2024-04-20T17:11:08.461 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:777c10d7:::benchmark_data_smithi171_78918_object96447:head# 2024-04-20T17:11:08.461 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:777d5881:::benchmark_data_smithi171_78918_object54468:head# 2024-04-20T17:11:08.461 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:777e6d1a:::benchmark_data_smithi171_78918_object66687:head# 2024-04-20T17:11:08.461 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:777e7e52:::benchmark_data_smithi171_78918_object140155:head# 2024-04-20T17:11:08.461 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:777eb26b:::benchmark_data_smithi171_78918_object81649:head# 2024-04-20T17:11:08.461 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:777f0d9b:::benchmark_data_smithi171_78918_object146033:head# 2024-04-20T17:11:08.461 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:777f5ea5:::benchmark_data_smithi171_78918_object128294:head# 2024-04-20T17:11:08.462 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:777fa098:::benchmark_data_smithi171_78918_object76361:head# 2024-04-20T17:11:08.462 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:777fac05:::benchmark_data_smithi171_78918_object151168:head# 2024-04-20T17:11:08.462 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:777fbae3:::benchmark_data_smithi171_78918_object61767:head# 2024-04-20T17:11:08.462 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77800347:::benchmark_data_smithi171_78918_object112498:head# 2024-04-20T17:11:08.462 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7780b17a:::benchmark_data_smithi171_78918_object160287:head# 2024-04-20T17:11:08.462 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7780cad0:::benchmark_data_smithi171_78918_object106174:head# 2024-04-20T17:11:08.462 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77815210:::benchmark_data_smithi171_78918_object87180:head# 2024-04-20T17:11:08.462 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77827b31:::benchmark_data_smithi171_78918_object69724:head# 2024-04-20T17:11:08.462 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7782e5cf:::benchmark_data_smithi171_78918_object136863:head# 2024-04-20T17:11:08.462 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7783010a:::benchmark_data_smithi171_78918_object108899:head# 2024-04-20T17:11:08.462 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77834a3d:::benchmark_data_smithi171_78918_object110782:head# 2024-04-20T17:11:08.462 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7784161d:::benchmark_data_smithi171_78918_object93379:head# 2024-04-20T17:11:08.462 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77842269:::benchmark_data_smithi171_78918_object147304:head# 2024-04-20T17:11:08.462 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77852c7d:::benchmark_data_smithi171_78918_object136297:head# 2024-04-20T17:11:08.463 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77867398:::benchmark_data_smithi171_78918_object123910:head# 2024-04-20T17:11:08.463 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77870921:::benchmark_data_smithi171_78918_object77671:head# 2024-04-20T17:11:08.463 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77886708:::benchmark_data_smithi171_78918_object148914:head# 2024-04-20T17:11:08.463 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7789177a:::benchmark_data_smithi171_78918_object133147:head# 2024-04-20T17:11:08.463 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77895615:::benchmark_data_smithi171_78918_object74005:head# 2024-04-20T17:11:08.463 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7789c4a6:::benchmark_data_smithi171_78918_object120109:head# 2024-04-20T17:11:08.463 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7789fd18:::benchmark_data_smithi171_78918_object153316:head# 2024-04-20T17:11:08.463 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:778afbe1:::benchmark_data_smithi171_78918_object56383:head# 2024-04-20T17:11:08.463 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:778b9b38:::benchmark_data_smithi171_78918_object117379:head# 2024-04-20T17:11:08.463 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:778bf57d:::benchmark_data_smithi171_78918_object71069:head# 2024-04-20T17:11:08.463 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:778bfbbf:::benchmark_data_smithi171_78918_object88123:head# 2024-04-20T17:11:08.463 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:778c2905:::benchmark_data_smithi171_78918_object147813:head# 2024-04-20T17:11:08.463 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:778d2fcc:::benchmark_data_smithi171_78918_object57773:head# 2024-04-20T17:11:08.463 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:778db06b:::benchmark_data_smithi171_78918_object144021:head# 2024-04-20T17:11:08.463 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:778ddb38:::benchmark_data_smithi171_78918_object66512:head# 2024-04-20T17:11:08.463 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:778de13f:::benchmark_data_smithi171_78918_object102925:head# 2024-04-20T17:11:08.464 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:778df32e:::benchmark_data_smithi171_78918_object101258:head# 2024-04-20T17:11:08.464 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:778df766:::benchmark_data_smithi171_78918_object106527:head# 2024-04-20T17:11:08.464 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:778e15d6:::benchmark_data_smithi171_78918_object88044:head# 2024-04-20T17:11:08.464 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7790162f:::benchmark_data_smithi171_78918_object90224:head# 2024-04-20T17:11:08.464 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77906897:::benchmark_data_smithi171_78918_object140860:head# 2024-04-20T17:11:08.464 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77908fa6:::benchmark_data_smithi171_78918_object128687:head# 2024-04-20T17:11:08.464 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7790e7ed:::benchmark_data_smithi171_78918_object131211:head# 2024-04-20T17:11:08.464 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7790fa28:::benchmark_data_smithi171_78918_object65369:head# 2024-04-20T17:11:08.464 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77930921:::benchmark_data_smithi171_78918_object128012:head# 2024-04-20T17:11:08.464 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77935254:::benchmark_data_smithi171_78918_object121494:head# 2024-04-20T17:11:08.464 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77935521:::benchmark_data_smithi171_78918_object109801:head# 2024-04-20T17:11:08.464 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77938671:::benchmark_data_smithi171_78918_object156262:head# 2024-04-20T17:11:08.464 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7793974f:::benchmark_data_smithi171_78918_object85997:head# 2024-04-20T17:11:08.464 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7793e261:::benchmark_data_smithi171_78918_object115225:head# 2024-04-20T17:11:08.464 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77944ae2:::benchmark_data_smithi171_78918_object96955:head# 2024-04-20T17:11:08.464 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77948c6c:::benchmark_data_smithi171_78918_object70785:head# 2024-04-20T17:11:08.465 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7794ed35:::benchmark_data_smithi171_78918_object153284:head# 2024-04-20T17:11:08.465 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7795351b:::benchmark_data_smithi171_78918_object83711:head# 2024-04-20T17:11:08.465 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77975ca9:::benchmark_data_smithi171_78918_object61227:head# 2024-04-20T17:11:08.465 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77992364:::benchmark_data_smithi171_78918_object65993:head# 2024-04-20T17:11:08.465 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77998cb5:::benchmark_data_smithi171_78918_object128401:head# 2024-04-20T17:11:08.465 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:779a05a1:::benchmark_data_smithi171_78918_object137667:head# 2024-04-20T17:11:08.465 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:779a2887:::benchmark_data_smithi171_78918_object105886:head# 2024-04-20T17:11:08.465 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:779abf70:::benchmark_data_smithi171_78918_object116761:head# 2024-04-20T17:11:08.465 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:779b6cc0:::benchmark_data_smithi171_78918_object109822:head# 2024-04-20T17:11:08.465 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:779b89b6:::benchmark_data_smithi171_78918_object116850:head# 2024-04-20T17:11:08.465 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:779c8653:::benchmark_data_smithi171_78918_object63854:head# 2024-04-20T17:11:08.465 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:779ddf50:::benchmark_data_smithi171_78918_object73683:head# 2024-04-20T17:11:08.465 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:779e3a12:::benchmark_data_smithi171_78918_object104410:head# 2024-04-20T17:11:08.465 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:779e6ef9:::benchmark_data_smithi171_78918_object155296:head# 2024-04-20T17:11:08.465 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:779e8009:::benchmark_data_smithi171_78918_object133217:head# 2024-04-20T17:11:08.465 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:779ec94a:::benchmark_data_smithi171_78918_object84305:head# 2024-04-20T17:11:08.472 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:779f8ad4:::benchmark_data_smithi171_78918_object134695:head# 2024-04-20T17:11:08.472 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77a34cf5:::benchmark_data_smithi171_78918_object99212:head# 2024-04-20T17:11:08.472 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77a34d78:::benchmark_data_smithi171_78918_object83683:head# 2024-04-20T17:11:08.472 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77a3b94e:::benchmark_data_smithi171_78918_object158895:head# 2024-04-20T17:11:08.472 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77a3cdca:::benchmark_data_smithi171_78918_object70088:head# 2024-04-20T17:11:08.473 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77a420f6:::benchmark_data_smithi171_78918_object148573:head# 2024-04-20T17:11:08.473 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77a448e6:::benchmark_data_smithi171_78918_object76783:head# 2024-04-20T17:11:08.473 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77a452a0:::benchmark_data_smithi171_78918_object149026:head# 2024-04-20T17:11:08.473 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77a4b85d:::benchmark_data_smithi171_78918_object64588:head# 2024-04-20T17:11:08.473 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77a6f095:::benchmark_data_smithi171_78918_object66424:head# 2024-04-20T17:11:08.473 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77a73127:::benchmark_data_smithi171_78918_object64075:head# 2024-04-20T17:11:08.473 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77a7f0b1:::benchmark_data_smithi171_78918_object86758:head# 2024-04-20T17:11:08.473 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77a843bb:::benchmark_data_smithi171_78918_object73500:head# 2024-04-20T17:11:08.473 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77aa0dc1:::benchmark_data_smithi171_78918_object123669:head# 2024-04-20T17:11:08.473 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77aa4e65:::benchmark_data_smithi171_78918_object158340:head# 2024-04-20T17:11:08.473 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77aa76bc:::benchmark_data_smithi171_78918_object64276:head# 2024-04-20T17:11:08.473 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77aab0d8:::benchmark_data_smithi171_78918_object160632:head# 2024-04-20T17:11:08.473 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77ab0806:::benchmark_data_smithi171_78918_object95795:head# 2024-04-20T17:11:08.473 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77ab4f3c:::benchmark_data_smithi171_78918_object97894:head# 2024-04-20T17:11:08.474 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77acc159:::benchmark_data_smithi171_78918_object128124:head# 2024-04-20T17:11:08.474 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77ae3a04:::benchmark_data_smithi171_78918_object155694:head# 2024-04-20T17:11:08.474 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77ae3d8e:::benchmark_data_smithi171_78918_object137857:head# 2024-04-20T17:11:08.474 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77ae8054:::benchmark_data_smithi171_78918_object146499:head# 2024-04-20T17:11:08.474 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77ae8605:::benchmark_data_smithi171_78918_object66372:head# 2024-04-20T17:11:08.474 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77af9662:::benchmark_data_smithi171_78918_object81787:head# 2024-04-20T17:11:08.474 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77b12e63:::benchmark_data_smithi171_78918_object57237:head# 2024-04-20T17:11:08.474 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77b27aa6:::benchmark_data_smithi171_78918_object153131:head# 2024-04-20T17:11:08.474 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77b3a75b:::benchmark_data_smithi171_78918_object99135:head# 2024-04-20T17:11:08.474 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77b46412:::benchmark_data_smithi171_78918_object73405:head# 2024-04-20T17:11:08.474 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77b4e97e:::benchmark_data_smithi171_78918_object80430:head# 2024-04-20T17:11:08.474 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77b4f255:::benchmark_data_smithi171_78918_object59979:head# 2024-04-20T17:11:08.474 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77b5466d:::benchmark_data_smithi171_78918_object76168:head# 2024-04-20T17:11:08.475 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77b613b6:::benchmark_data_smithi171_78918_object130476:head# 2024-04-20T17:11:08.475 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77b6a228:::benchmark_data_smithi171_78918_object105505:head# 2024-04-20T17:11:08.475 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77b7c433:::benchmark_data_smithi171_78918_object54913:head# 2024-04-20T17:11:08.475 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77b7d816:::benchmark_data_smithi171_78918_object99279:head# 2024-04-20T17:11:08.475 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77b8264b:::benchmark_data_smithi171_78918_object113581:head# 2024-04-20T17:11:08.475 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77b831a0:::benchmark_data_smithi171_78918_object73566:head# 2024-04-20T17:11:08.475 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77b8bf57:::benchmark_data_smithi171_78918_object64621:head# 2024-04-20T17:11:08.475 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77b8e48a:::benchmark_data_smithi171_78918_object107242:head# 2024-04-20T17:11:08.475 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77b96293:::benchmark_data_smithi171_78918_object93675:head# 2024-04-20T17:11:08.475 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77ba00fb:::benchmark_data_smithi171_78918_object130747:head# 2024-04-20T17:11:08.475 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77ba06c5:::benchmark_data_smithi171_78918_object95555:head# 2024-04-20T17:11:08.475 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77ba5e4b:::benchmark_data_smithi171_78918_object135308:head# 2024-04-20T17:11:08.475 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77ba797c:::benchmark_data_smithi171_78918_object107553:head# 2024-04-20T17:11:08.475 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77bb6706:::benchmark_data_smithi171_78918_object137837:head# 2024-04-20T17:11:08.476 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77bb9dcc:::benchmark_data_smithi171_78918_object102291:head# 2024-04-20T17:11:08.476 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77bc6839:::benchmark_data_smithi171_78918_object64874:head# 2024-04-20T17:11:08.476 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77bcca1f:::benchmark_data_smithi171_78918_object97335:head# 2024-04-20T17:11:08.476 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77bd6137:::benchmark_data_smithi171_78918_object64021:head# 2024-04-20T17:11:08.476 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77bd6569:::benchmark_data_smithi171_78918_object146085:head# 2024-04-20T17:11:08.476 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77becea1:::benchmark_data_smithi171_78918_object91404:head# 2024-04-20T17:11:08.476 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77bf7888:::benchmark_data_smithi171_78918_object135361:head# 2024-04-20T17:11:08.476 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77bff712:::benchmark_data_smithi171_78918_object99378:head# 2024-04-20T17:11:08.476 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77c000ce:::benchmark_data_smithi171_78918_object105957:head# 2024-04-20T17:11:08.476 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77c160e2:::benchmark_data_smithi171_78918_object143361:head# 2024-04-20T17:11:08.476 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77c1a6f7:::benchmark_data_smithi171_78918_object118878:head# 2024-04-20T17:11:08.476 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77c1e9fe:::benchmark_data_smithi171_78918_object93390:head# 2024-04-20T17:11:08.476 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77c2e733:::benchmark_data_smithi171_78918_object77460:head# 2024-04-20T17:11:08.476 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77c31b1e:::benchmark_data_smithi171_78918_object132679:head# 2024-04-20T17:11:08.476 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77c3984b:::benchmark_data_smithi171_78918_object84518:head# 2024-04-20T17:11:08.477 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77c4473f:::benchmark_data_smithi171_78918_object82105:head# 2024-04-20T17:11:08.477 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77c45aa7:::benchmark_data_smithi171_78918_object124103:head# 2024-04-20T17:11:08.477 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77c4a4ea:::benchmark_data_smithi171_78918_object55469:head# 2024-04-20T17:11:08.477 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77c67c22:::benchmark_data_smithi171_78918_object71575:head# 2024-04-20T17:11:08.477 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77c79f7b:::benchmark_data_smithi171_78918_object55104:head# 2024-04-20T17:11:08.477 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77c906ff:::benchmark_data_smithi171_78918_object121555:head# 2024-04-20T17:11:08.477 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77c90732:::benchmark_data_smithi171_78918_object126595:head# 2024-04-20T17:11:08.477 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77c973eb:::benchmark_data_smithi171_78918_object160451:head# 2024-04-20T17:11:08.477 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77c9762b:::benchmark_data_smithi171_78918_object160773:head# 2024-04-20T17:11:08.482 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77c977ed:::benchmark_data_smithi171_78918_object151107:head# 2024-04-20T17:11:08.482 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77ca9ddb:::benchmark_data_smithi171_78918_object129090:head# 2024-04-20T17:11:08.483 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77cadf5f:::benchmark_data_smithi171_78918_object109363:head# 2024-04-20T17:11:08.483 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77ccc89d:::benchmark_data_smithi171_78918_object143396:head# 2024-04-20T17:11:08.483 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77cce6d3:::benchmark_data_smithi171_78918_object138403:head# 2024-04-20T17:11:08.483 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77cd66e8:::benchmark_data_smithi171_78918_object99352:head# 2024-04-20T17:11:08.483 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77ce3365:::benchmark_data_smithi171_78918_object116528:head# 2024-04-20T17:11:08.483 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77ce88a4:::benchmark_data_smithi171_78918_object106463:head# 2024-04-20T17:11:08.483 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77d03889:::benchmark_data_smithi171_78918_object91927:head# 2024-04-20T17:11:08.483 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77d09b37:::benchmark_data_smithi171_78918_object73084:head# 2024-04-20T17:11:08.483 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77d148e4:::benchmark_data_smithi171_78918_object102331:head# 2024-04-20T17:11:08.483 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77d21efb:::benchmark_data_smithi171_78918_object75774:head# 2024-04-20T17:11:08.483 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77d294cd:::benchmark_data_smithi171_78918_object152278:head# 2024-04-20T17:11:08.483 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77d3d5bc:::benchmark_data_smithi171_78918_object157387:head# 2024-04-20T17:11:08.483 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77d42bf5:::benchmark_data_smithi171_78918_object121235:head# 2024-04-20T17:11:08.483 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77d4c515:::benchmark_data_smithi171_78918_object156844:head# 2024-04-20T17:11:08.483 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77d5c8aa:::benchmark_data_smithi171_78918_object103608:head# 2024-04-20T17:11:08.483 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77d5dc8d:::benchmark_data_smithi171_78918_object93912:head# 2024-04-20T17:11:08.484 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77d5dcee:::benchmark_data_smithi171_78918_object87386:head# 2024-04-20T17:11:08.484 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77d6263e:::benchmark_data_smithi171_78918_object68332:head# 2024-04-20T17:11:08.484 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77d64f95:::benchmark_data_smithi171_78918_object127094:head# 2024-04-20T17:11:08.484 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77d653ba:::benchmark_data_smithi171_78918_object141848:head# 2024-04-20T17:11:08.484 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77d6fb38:::benchmark_data_smithi171_78918_object157004:head# 2024-04-20T17:11:08.484 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77d749e6:::benchmark_data_smithi171_78918_object85719:head# 2024-04-20T17:11:08.484 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77d91aa5:::benchmark_data_smithi171_78918_object160329:head# 2024-04-20T17:11:08.484 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77d9435f:::benchmark_data_smithi171_78918_object148245:head# 2024-04-20T17:11:08.484 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77d96e00:::benchmark_data_smithi171_78918_object139095:head# 2024-04-20T17:11:08.484 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77dbe43a:::benchmark_data_smithi171_78918_object59670:head# 2024-04-20T17:11:08.484 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77dcca11:::benchmark_data_smithi171_78918_object93314:head# 2024-04-20T17:11:08.484 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77dd302e:::benchmark_data_smithi171_78918_object54590:head# 2024-04-20T17:11:08.484 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77dda201:::benchmark_data_smithi171_78918_object86882:head# 2024-04-20T17:11:08.484 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77df7fe5:::benchmark_data_smithi171_78918_object103270:head# 2024-04-20T17:11:08.485 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77df9bf1:::benchmark_data_smithi171_78918_object71351:head# 2024-04-20T17:11:08.485 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77e0e61b:::benchmark_data_smithi171_78918_object109314:head# 2024-04-20T17:11:08.485 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77e0eba1:::benchmark_data_smithi171_78918_object96893:head# 2024-04-20T17:11:08.485 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77e2b7bf:::benchmark_data_smithi171_78918_object104239:head# 2024-04-20T17:11:08.485 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77e312f1:::benchmark_data_smithi171_78918_object109726:head# 2024-04-20T17:11:08.485 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77e4d255:::benchmark_data_smithi171_78918_object132467:head# 2024-04-20T17:11:08.485 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77e4d8da:::benchmark_data_smithi171_78918_object85089:head# 2024-04-20T17:11:08.485 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77e5acc2:::benchmark_data_smithi171_78918_object153062:head# 2024-04-20T17:11:08.485 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77e5ed24:::benchmark_data_smithi171_78918_object120160:head# 2024-04-20T17:11:08.485 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77e5f2be:::benchmark_data_smithi171_78918_object84780:head# 2024-04-20T17:11:08.485 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77e6641e:::benchmark_data_smithi171_78918_object68403:head# 2024-04-20T17:11:08.485 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77e6b5f0:::benchmark_data_smithi171_78918_object103997:head# 2024-04-20T17:11:08.485 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77e6e310:::benchmark_data_smithi171_78918_object100656:head# 2024-04-20T17:11:08.485 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77e71f4d:::benchmark_data_smithi171_78918_object141302:head# 2024-04-20T17:11:08.485 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77e75503:::benchmark_data_smithi171_78918_object80221:head# 2024-04-20T17:11:08.485 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77e77071:::benchmark_data_smithi171_78918_object55650:head# 2024-04-20T17:11:08.486 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77e862f3:::benchmark_data_smithi171_78918_object142172:head# 2024-04-20T17:11:08.486 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77e873ca:::benchmark_data_smithi171_78918_object140203:head# 2024-04-20T17:11:08.486 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77e88e23:::benchmark_data_smithi171_78918_object113334:head# 2024-04-20T17:11:08.486 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77ed2975:::benchmark_data_smithi171_78918_object75959:head# 2024-04-20T17:11:08.486 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77ed3649:::benchmark_data_smithi171_78918_object151610:head# 2024-04-20T17:11:08.486 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77ed389b:::benchmark_data_smithi171_78918_object88297:head# 2024-04-20T17:11:08.486 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77ed81e7:::benchmark_data_smithi171_78918_object82984:head# 2024-04-20T17:11:08.486 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77eda153:::benchmark_data_smithi171_78918_object132728:head# 2024-04-20T17:11:08.492 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77ee4aa0:::benchmark_data_smithi171_78918_object114694:head# 2024-04-20T17:11:08.492 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77ee6c98:::benchmark_data_smithi171_78918_object143190:head# 2024-04-20T17:11:08.492 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77ee86f9:::benchmark_data_smithi171_78918_object108652:head# 2024-04-20T17:11:08.493 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77eea9c5:::benchmark_data_smithi171_78918_object53621:head# 2024-04-20T17:11:08.493 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77eebb9a:::benchmark_data_smithi171_78918_object52579:head# 2024-04-20T17:11:08.493 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77f20ff5:::benchmark_data_smithi171_78918_object95862:head# 2024-04-20T17:11:08.493 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77f2287b:::benchmark_data_smithi171_78918_object133577:head# 2024-04-20T17:11:08.493 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77f32adf:::benchmark_data_smithi171_78918_object65838:head# 2024-04-20T17:11:08.493 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77f3429c:::benchmark_data_smithi171_78918_object91920:head# 2024-04-20T17:11:08.493 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77f35f9e:::benchmark_data_smithi171_78918_object139048:head# 2024-04-20T17:11:08.493 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77f41e55:::benchmark_data_smithi171_78918_object74828:head# 2024-04-20T17:11:08.493 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77f69da1:::benchmark_data_smithi171_78918_object83306:head# 2024-04-20T17:11:08.493 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77f6bf25:::benchmark_data_smithi171_78918_object114872:head# 2024-04-20T17:11:08.493 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77f7061f:::benchmark_data_smithi171_78918_object132759:head# 2024-04-20T17:11:08.493 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77f77a3b:::benchmark_data_smithi171_78918_object121781:head# 2024-04-20T17:11:08.493 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77f80d1e:::benchmark_data_smithi171_78918_object140856:head# 2024-04-20T17:11:08.493 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77f856fe:::benchmark_data_smithi171_78918_object73317:head# 2024-04-20T17:11:08.493 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77f8a7dd:::benchmark_data_smithi171_78918_object157157:head# 2024-04-20T17:11:08.493 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77f97c55:::benchmark_data_smithi171_78918_object116419:head# 2024-04-20T17:11:08.494 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77f988ee:::benchmark_data_smithi171_78918_object108633:head# 2024-04-20T17:11:08.494 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77fa8cf1:::benchmark_data_smithi171_78918_object81124:head# 2024-04-20T17:11:08.494 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77fb2a1e:::benchmark_data_smithi171_78918_object113715:head# 2024-04-20T17:11:08.494 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77fbbec8:::benchmark_data_smithi171_78918_object79555:head# 2024-04-20T17:11:08.494 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77fc643b:::benchmark_data_smithi171_78918_object78936:head# 2024-04-20T17:11:08.494 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77fd5d86:::benchmark_data_smithi171_78918_object97434:head# 2024-04-20T17:11:08.494 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77fdc10b:::benchmark_data_smithi171_78918_object90093:head# 2024-04-20T17:11:08.494 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77fdc671:::benchmark_data_smithi171_78918_object134142:head# 2024-04-20T17:11:08.494 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77fef3e6:::benchmark_data_smithi171_78918_object101018:head# 2024-04-20T17:11:08.494 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77ff1ae0:::benchmark_data_smithi171_78918_object155197:head# 2024-04-20T17:11:08.494 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77ff5a44:::benchmark_data_smithi171_78918_object159294:head# 2024-04-20T17:11:08.494 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:77ff964f:::benchmark_data_smithi171_78918_object62839:head# 2024-04-20T17:11:08.494 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78005178:::benchmark_data_smithi171_78918_object100438:head# 2024-04-20T17:11:08.495 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7801a9e4:::benchmark_data_smithi171_78918_object145889:head# 2024-04-20T17:11:08.495 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78022274:::benchmark_data_smithi171_78918_object132191:head# 2024-04-20T17:11:08.495 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78026648:::benchmark_data_smithi171_78918_object79546:head# 2024-04-20T17:11:08.495 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7802fd34:::benchmark_data_smithi171_78918_object117879:head# 2024-04-20T17:11:08.495 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7804677d:::benchmark_data_smithi171_78918_object127309:head# 2024-04-20T17:11:08.495 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7804730a:::benchmark_data_smithi171_78918_object109668:head# 2024-04-20T17:11:08.495 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78047412:::benchmark_data_smithi171_78918_object54068:head# 2024-04-20T17:11:08.495 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78058d17:::benchmark_data_smithi171_78918_object144576:head# 2024-04-20T17:11:08.495 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7805d055:::benchmark_data_smithi171_78918_object52626:head# 2024-04-20T17:11:08.495 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7806b281:::benchmark_data_smithi171_78918_object61823:head# 2024-04-20T17:11:08.495 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7806ee99:::benchmark_data_smithi171_78918_object77667:head# 2024-04-20T17:11:08.495 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78072cdf:::benchmark_data_smithi171_78918_object141717:head# 2024-04-20T17:11:08.495 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7808056a:::benchmark_data_smithi171_78918_object86761:head# 2024-04-20T17:11:08.496 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78084f5c:::benchmark_data_smithi171_78918_object58729:head# 2024-04-20T17:11:08.496 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78091f4a:::benchmark_data_smithi171_78918_object149457:head# 2024-04-20T17:11:08.496 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:780a9d0b:::benchmark_data_smithi171_78918_object63583:head# 2024-04-20T17:11:08.496 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:780af866:::benchmark_data_smithi171_78918_object152873:head# 2024-04-20T17:11:08.496 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:780b1e3a:::benchmark_data_smithi171_78918_object156075:head# 2024-04-20T17:11:08.496 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:780cbab9:::benchmark_data_smithi171_78918_object143474:head# 2024-04-20T17:11:08.496 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:780da192:::benchmark_data_smithi171_78918_object55336:head# 2024-04-20T17:11:08.496 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:780f499c:::benchmark_data_smithi171_78918_object88582:head# 2024-04-20T17:11:08.496 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78103c83:::benchmark_data_smithi171_78918_object120546:head# 2024-04-20T17:11:08.496 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78104e7d:::benchmark_data_smithi171_78918_object132061:head# 2024-04-20T17:11:08.496 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7810b494:::benchmark_data_smithi171_78918_object77216:head# 2024-04-20T17:11:08.496 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78111ac9:::benchmark_data_smithi171_78918_object92852:head# 2024-04-20T17:11:08.496 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7811585c:::benchmark_data_smithi171_78918_object141288:head# 2024-04-20T17:11:08.496 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:781480e2:::benchmark_data_smithi171_78918_object67660:head# 2024-04-20T17:11:08.496 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7814cd43:::benchmark_data_smithi171_78918_object70960:head# 2024-04-20T17:11:08.497 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:781592c0:::benchmark_data_smithi171_78918_object110988:head# 2024-04-20T17:11:08.497 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:781641c6:::benchmark_data_smithi171_78918_object109086:head# 2024-04-20T17:11:08.497 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78174b40:::benchmark_data_smithi171_78918_object104876:head# 2024-04-20T17:11:08.497 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78182e88:::benchmark_data_smithi171_78918_object121799:head# 2024-04-20T17:11:08.497 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78185f3c:::benchmark_data_smithi171_78918_object154368:head# 2024-04-20T17:11:08.506 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:11:08.506 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:11:08.506 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:11:08.506 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:11:08.506 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.506 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:138161 2.5 2:a4f53bd8:::benchmark_data_smithi171_78918_object138160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-20T17:11:08.506 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:09:22.723185+0000", 2024-04-20T17:11:08.506 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 105.773585254, 2024-04-20T17:11:08.506 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.037369641000000002, 2024-04-20T17:11:08.507 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:08.507 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:08.507 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:08.507 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:08.507 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:08.507 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:08.507 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 138161 2024-04-20T17:11:08.507 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.507 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:08.507 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.507 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:08.507 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.723185+0000", 2024-04-20T17:11:08.507 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.507 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.508 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.508 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:08.508 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.723185+0000", 2024-04-20T17:11:08.508 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.508 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.508 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.508 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:08.508 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.723186+0000", 2024-04-20T17:11:08.508 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4000000000000001e-07 2024-04-20T17:11:08.508 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.508 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.508 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:08.508 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.723231+0000", 2024-04-20T17:11:08.509 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5262999999999997e-05 2024-04-20T17:11:08.509 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.509 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.509 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:08.509 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.723240+0000", 2024-04-20T17:11:08.509 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9649999999999997e-06 2024-04-20T17:11:08.509 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.509 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.509 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:08.509 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.723246+0000", 2024-04-20T17:11:08.509 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.207e-06 2024-04-20T17:11:08.509 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.509 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.509 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:08.510 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.727449+0000", 2024-04-20T17:11:08.510 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0042026570000000003 2024-04-20T17:11:08.510 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.510 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.510 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:08.510 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.727582+0000", 2024-04-20T17:11:08.510 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013308600000000001 2024-04-20T17:11:08.510 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.510 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.510 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:08.510 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.727720+0000", 2024-04-20T17:11:08.510 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000138348 2024-04-20T17:11:08.510 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.510 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.511 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:08.511 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.731368+0000", 2024-04-20T17:11:08.511 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0036483380000000001 2024-04-20T17:11:08.511 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.511 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.511 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:08.511 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.760485+0000", 2024-04-20T17:11:08.511 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.029116454 2024-04-20T17:11:08.511 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.511 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.511 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:08.511 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.760520+0000", 2024-04-20T17:11:08.511 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5349999999999999e-05 2024-04-20T17:11:08.511 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.511 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.512 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:08.512 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.760554+0000", 2024-04-20T17:11:08.512 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4233000000000002e-05 2024-04-20T17:11:08.512 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.512 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:08.512 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.512 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.512 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.512 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:142704 2.5 2:a19ddc2a:::benchmark_data_smithi171_78918_object142703:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-04-20T17:11:08.512 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:09:26.285445+0000", 2024-04-20T17:11:08.512 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 102.21132471599999, 2024-04-20T17:11:08.512 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.031034718999999999, 2024-04-20T17:11:08.512 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:08.512 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:08.512 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:08.513 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:08.513 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:08.513 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:08.513 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 142704 2024-04-20T17:11:08.513 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.513 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:08.513 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.513 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:08.513 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:26.285445+0000", 2024-04-20T17:11:08.513 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.513 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.513 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.513 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:08.513 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:26.285445+0000", 2024-04-20T17:11:08.514 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.514 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.514 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.514 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:08.514 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:26.285447+0000", 2024-04-20T17:11:08.514 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8670000000000001e-06 2024-04-20T17:11:08.514 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.514 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.514 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:08.514 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:26.285467+0000", 2024-04-20T17:11:08.514 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0108000000000001e-05 2024-04-20T17:11:08.514 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.514 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.514 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:08.515 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:26.285477+0000", 2024-04-20T17:11:08.515 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0169e-05 2024-04-20T17:11:08.515 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.515 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.515 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:08.515 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:26.285483+0000", 2024-04-20T17:11:08.515 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9200000000000001e-06 2024-04-20T17:11:08.515 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.515 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.515 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:08.515 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:26.288586+0000", 2024-04-20T17:11:08.515 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0031026059999999999 2024-04-20T17:11:08.515 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.516 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.516 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:08.516 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:26.288735+0000", 2024-04-20T17:11:08.516 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014930400000000001 2024-04-20T17:11:08.516 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.516 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.516 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:08.516 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:26.288899+0000", 2024-04-20T17:11:08.516 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00016379 2024-04-20T17:11:08.516 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.516 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.516 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:08.516 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:26.292320+0000", 2024-04-20T17:11:08.516 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0034206470000000002 2024-04-20T17:11:08.517 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.517 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.517 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:08.517 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:26.316415+0000", 2024-04-20T17:11:08.517 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024095343000000002 2024-04-20T17:11:08.517 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.517 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.517 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:08.517 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:26.316449+0000", 2024-04-20T17:11:08.517 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4078999999999997e-05 2024-04-20T17:11:08.517 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.517 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.517 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:08.517 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:26.316480+0000", 2024-04-20T17:11:08.517 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0886000000000002e-05 2024-04-20T17:11:08.518 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.518 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:08.518 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.518 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.518 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.518 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:145783 2.b 2:d30281bf:::benchmark_data_smithi171_78918_object145782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-04-20T17:11:08.518 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:09:28.793289+0000", 2024-04-20T17:11:08.518 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 99.703481468999996, 2024-04-20T17:11:08.518 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.031262359000000003, 2024-04-20T17:11:08.518 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:08.518 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:08.518 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:08.518 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:08.518 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:08.519 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:08.519 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 145783 2024-04-20T17:11:08.519 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.519 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:08.519 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.519 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:08.519 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:28.793289+0000", 2024-04-20T17:11:08.519 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.519 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.519 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.519 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:08.519 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:28.793289+0000", 2024-04-20T17:11:08.519 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.519 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.520 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.520 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:08.520 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:28.793291+0000", 2024-04-20T17:11:08.520 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7329999999999999e-06 2024-04-20T17:11:08.520 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.520 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.520 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:08.520 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:28.793319+0000", 2024-04-20T17:11:08.520 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8201e-05 2024-04-20T17:11:08.520 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.520 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.520 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:08.520 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:28.793332+0000", 2024-04-20T17:11:08.520 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2734999999999999e-05 2024-04-20T17:11:08.520 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.521 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.521 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:08.521 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:28.793339+0000", 2024-04-20T17:11:08.521 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8120000000000002e-06 2024-04-20T17:11:08.521 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.521 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.521 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:08.521 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:28.798844+0000", 2024-04-20T17:11:08.521 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.005504854 2024-04-20T17:11:08.521 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.521 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.521 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:08.521 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:28.799003+0000", 2024-04-20T17:11:08.521 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015915699999999999 2024-04-20T17:11:08.522 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.522 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.522 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:11:08.522 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:28.799126+0000", 2024-04-20T17:11:08.522 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012265999999999999 2024-04-20T17:11:08.522 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.522 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.522 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:08.522 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:28.814268+0000", 2024-04-20T17:11:08.522 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015142043000000001 2024-04-20T17:11:08.522 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.522 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.522 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:08.522 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:28.824459+0000", 2024-04-20T17:11:08.523 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010191155 2024-04-20T17:11:08.523 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.523 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.523 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:08.523 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:28.824510+0000", 2024-04-20T17:11:08.523 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1579000000000002e-05 2024-04-20T17:11:08.523 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.523 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.523 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:08.523 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:28.824551+0000", 2024-04-20T17:11:08.523 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0429999999999997e-05 2024-04-20T17:11:08.523 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.523 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:08.523 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.523 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.524 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.524 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:146863 2.5 2:ae859954:::benchmark_data_smithi171_78918_object146862:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e145)", 2024-04-20T17:11:08.524 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:09:29.631385+0000", 2024-04-20T17:11:08.524 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 98.865385168000003, 2024-04-20T17:11:08.524 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030717557, 2024-04-20T17:11:08.524 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:08.524 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:08.524 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:08.524 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:08.524 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:08.524 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:08.524 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 146863 2024-04-20T17:11:08.524 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.524 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:08.525 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.525 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:08.525 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:29.631385+0000", 2024-04-20T17:11:08.525 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.525 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.525 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.525 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:08.525 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:29.631385+0000", 2024-04-20T17:11:08.525 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.525 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.525 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.525 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:08.525 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:29.631387+0000", 2024-04-20T17:11:08.525 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2089999999999999e-06 2024-04-20T17:11:08.525 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.526 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.526 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:08.526 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:29.631414+0000", 2024-04-20T17:11:08.526 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6455e-05 2024-04-20T17:11:08.526 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.526 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.526 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:08.526 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:29.631428+0000", 2024-04-20T17:11:08.526 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4571e-05 2024-04-20T17:11:08.526 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.526 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.526 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:08.526 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:29.631437+0000", 2024-04-20T17:11:08.526 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8170000000000005e-06 2024-04-20T17:11:08.527 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.527 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.527 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:08.527 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:29.631490+0000", 2024-04-20T17:11:08.527 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3560000000000002e-05 2024-04-20T17:11:08.527 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.527 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.527 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:08.527 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:29.631621+0000", 2024-04-20T17:11:08.527 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013083600000000001 2024-04-20T17:11:08.527 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.527 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.527 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:08.527 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:29.631754+0000", 2024-04-20T17:11:08.528 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013223 2024-04-20T17:11:08.528 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.528 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.528 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:08.528 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:29.638049+0000", 2024-04-20T17:11:08.528 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0062951279999999997 2024-04-20T17:11:08.528 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.528 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.528 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:08.528 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:29.661999+0000", 2024-04-20T17:11:08.528 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.023950374999999999 2024-04-20T17:11:08.528 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.528 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.528 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:08.528 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:29.662056+0000", 2024-04-20T17:11:08.528 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7249e-05 2024-04-20T17:11:08.529 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.529 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.529 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:08.529 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:29.662102+0000", 2024-04-20T17:11:08.529 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6127000000000001e-05 2024-04-20T17:11:08.529 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.529 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:08.529 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.529 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.529 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.529 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:149084 2.5 2:a8d12acd:::benchmark_data_smithi171_78918_object149083:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-04-20T17:11:08.529 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:09:31.451318+0000", 2024-04-20T17:11:08.529 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 97.045452350999994, 2024-04-20T17:11:08.529 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032349261999999997, 2024-04-20T17:11:08.529 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:08.529 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:08.530 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:08.530 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:08.530 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:08.530 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:08.530 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 149084 2024-04-20T17:11:08.530 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.530 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:08.530 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.530 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:08.530 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.451318+0000", 2024-04-20T17:11:08.530 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.530 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.530 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.530 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:08.531 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.451318+0000", 2024-04-20T17:11:08.531 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.531 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.531 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.531 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:08.531 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.451319+0000", 2024-04-20T17:11:08.531 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1850000000000001e-06 2024-04-20T17:11:08.531 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.531 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.531 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:08.531 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.451339+0000", 2024-04-20T17:11:08.531 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0503999999999999e-05 2024-04-20T17:11:08.531 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.531 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.532 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:08.532 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.451350+0000", 2024-04-20T17:11:08.532 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0139999999999999e-05 2024-04-20T17:11:08.532 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.532 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.532 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:08.532 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.451356+0000", 2024-04-20T17:11:08.532 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7750000000000004e-06 2024-04-20T17:11:08.532 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.532 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.532 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:08.532 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.454427+0000", 2024-04-20T17:11:08.532 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0030711079999999999 2024-04-20T17:11:08.532 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.532 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.533 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:08.533 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.454560+0000", 2024-04-20T17:11:08.533 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013242199999999999 2024-04-20T17:11:08.533 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.533 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.533 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:08.533 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.454713+0000", 2024-04-20T17:11:08.533 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015271800000000001 2024-04-20T17:11:08.533 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.533 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.533 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:08.533 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.461714+0000", 2024-04-20T17:11:08.533 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0070017220000000002 2024-04-20T17:11:08.534 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.534 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.534 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:08.534 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.483577+0000", 2024-04-20T17:11:08.534 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021862765999999999 2024-04-20T17:11:08.534 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.534 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.534 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:08.534 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.483631+0000", 2024-04-20T17:11:08.534 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4240000000000002e-05 2024-04-20T17:11:08.534 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.534 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.534 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:08.535 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.483667+0000", 2024-04-20T17:11:08.535 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5682000000000002e-05 2024-04-20T17:11:08.535 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.535 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:08.535 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.535 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.535 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.535 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:173492 2.5 2:a8fc7228:::benchmark_data_smithi171_78918_object12498:head [delete] snapc 0=[] ondisk+write+known_if_redirected e154)", 2024-04-20T17:11:08.535 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:03.266628+0000", 2024-04-20T17:11:08.535 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 65.230142021999995, 2024-04-20T17:11:08.535 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.033769500000000001, 2024-04-20T17:11:08.535 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:08.535 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:08.535 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:08.535 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:08.536 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:08.536 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:08.536 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 173492 2024-04-20T17:11:08.536 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.536 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:08.536 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.536 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:08.536 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:03.266628+0000", 2024-04-20T17:11:08.536 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.536 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.536 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.536 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:08.536 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:03.266628+0000", 2024-04-20T17:11:08.536 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.537 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.537 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.537 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:08.537 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:03.266629+0000", 2024-04-20T17:11:08.537 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1570000000000001e-06 2024-04-20T17:11:08.537 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.537 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.537 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:08.537 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:03.266633+0000", 2024-04-20T17:11:08.537 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5020000000000001e-06 2024-04-20T17:11:08.537 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.537 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.537 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:08.537 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:03.266641+0000", 2024-04-20T17:11:08.538 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0679999999999997e-06 2024-04-20T17:11:08.538 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.538 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.538 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:08.538 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:03.266646+0000", 2024-04-20T17:11:08.538 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3009999999999997e-06 2024-04-20T17:11:08.538 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.538 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.538 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:08.538 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:03.266688+0000", 2024-04-20T17:11:08.538 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2271000000000002e-05 2024-04-20T17:11:08.538 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.538 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.538 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:08.538 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:03.266777+0000", 2024-04-20T17:11:08.539 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9111999999999997e-05 2024-04-20T17:11:08.539 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.539 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.539 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:08.539 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:03.266862+0000", 2024-04-20T17:11:08.539 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4185999999999997e-05 2024-04-20T17:11:08.539 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.539 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.539 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:08.539 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:03.271573+0000", 2024-04-20T17:11:08.539 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0047111890000000002 2024-04-20T17:11:08.539 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.539 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.539 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:08.539 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:03.300303+0000", 2024-04-20T17:11:08.540 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028730545999999999 2024-04-20T17:11:08.540 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.540 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.540 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:08.540 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:03.300356+0000", 2024-04-20T17:11:08.540 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2386000000000003e-05 2024-04-20T17:11:08.540 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.540 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.540 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:08.540 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:03.300398+0000", 2024-04-20T17:11:08.540 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1782000000000001e-05 2024-04-20T17:11:08.540 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.540 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:08.540 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.541 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.541 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.541 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:180641 2.5 2:a1c1b464:::benchmark_data_smithi171_78918_object19647:head [delete] snapc 0=[] ondisk+write+known_if_redirected e156)", 2024-04-20T17:11:08.541 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:11.969181+0000", 2024-04-20T17:11:08.541 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 56.527588973, 2024-04-20T17:11:08.541 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.033084681999999997, 2024-04-20T17:11:08.541 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:08.541 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:08.541 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:08.541 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:08.541 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:08.541 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:08.541 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 180641 2024-04-20T17:11:08.541 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.541 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:08.542 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.542 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:08.542 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:11.969181+0000", 2024-04-20T17:11:08.542 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.542 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.542 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.542 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:08.542 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:11.969181+0000", 2024-04-20T17:11:08.542 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.542 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.542 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.542 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:08.542 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:11.969184+0000", 2024-04-20T17:11:08.542 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9399999999999998e-06 2024-04-20T17:11:08.543 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.543 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.543 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:08.543 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:11.969194+0000", 2024-04-20T17:11:08.543 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6970000000000005e-06 2024-04-20T17:11:08.543 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.543 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.543 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:08.543 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:11.969229+0000", 2024-04-20T17:11:08.543 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5478000000000002e-05 2024-04-20T17:11:08.543 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.543 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.543 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:08.543 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:11.969240+0000", 2024-04-20T17:11:08.543 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0862e-05 2024-04-20T17:11:08.544 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.544 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.544 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:08.544 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:11.969305+0000", 2024-04-20T17:11:08.544 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5411000000000001e-05 2024-04-20T17:11:08.544 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.544 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.544 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:08.544 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:11.969433+0000", 2024-04-20T17:11:08.544 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000127704 2024-04-20T17:11:08.544 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.544 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.544 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:08.544 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:11.969551+0000", 2024-04-20T17:11:08.544 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011782600000000001 2024-04-20T17:11:08.545 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.545 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.545 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:08.545 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:11.974394+0000", 2024-04-20T17:11:08.545 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.004843216 2024-04-20T17:11:08.545 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.545 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.545 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:08.545 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.002156+0000", 2024-04-20T17:11:08.545 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.027762044 2024-04-20T17:11:08.545 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.545 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.545 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:08.545 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.002219+0000", 2024-04-20T17:11:08.546 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2848999999999994e-05 2024-04-20T17:11:08.546 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.546 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.546 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:08.546 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.002266+0000", 2024-04-20T17:11:08.546 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6655e-05 2024-04-20T17:11:08.546 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.546 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:08.546 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.546 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.546 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.546 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:180926 2.5 2:a386948d:::benchmark_data_smithi171_78918_object19932:head [delete] snapc 0=[] ondisk+write+known_if_redirected e156)", 2024-04-20T17:11:08.546 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:12.318586+0000", 2024-04-20T17:11:08.546 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 56.178183953000001, 2024-04-20T17:11:08.547 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.033074789, 2024-04-20T17:11:08.547 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:08.547 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:08.547 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:08.547 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:08.547 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:08.547 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:08.547 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 180926 2024-04-20T17:11:08.547 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.547 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:08.547 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.547 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:08.547 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.318586+0000", 2024-04-20T17:11:08.547 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.547 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.548 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.548 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:08.548 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.318586+0000", 2024-04-20T17:11:08.548 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.548 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.548 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.548 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:08.548 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.318589+0000", 2024-04-20T17:11:08.548 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8669999999999998e-06 2024-04-20T17:11:08.548 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.548 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.548 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:08.548 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.318598+0000", 2024-04-20T17:11:08.548 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0289999999999994e-06 2024-04-20T17:11:08.549 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.549 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.549 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:08.549 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.318614+0000", 2024-04-20T17:11:08.549 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6282000000000001e-05 2024-04-20T17:11:08.549 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.549 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.549 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:08.549 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.318624+0000", 2024-04-20T17:11:08.549 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.251e-06 2024-04-20T17:11:08.549 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.549 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.549 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:08.549 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.318683+0000", 2024-04-20T17:11:08.550 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9358000000000003e-05 2024-04-20T17:11:08.550 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.550 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.550 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:08.550 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.318779+0000", 2024-04-20T17:11:08.550 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5789000000000001e-05 2024-04-20T17:11:08.550 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.550 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.550 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:08.550 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.318857+0000", 2024-04-20T17:11:08.550 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8078999999999996e-05 2024-04-20T17:11:08.550 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.550 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.550 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:08.551 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.326912+0000", 2024-04-20T17:11:08.551 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0080557010000000002 2024-04-20T17:11:08.551 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.551 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.551 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:08.551 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.351571+0000", 2024-04-20T17:11:08.551 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024658594999999998 2024-04-20T17:11:08.551 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.551 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.551 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:08.551 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.351624+0000", 2024-04-20T17:11:08.551 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2729999999999998e-05 2024-04-20T17:11:08.551 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.551 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.552 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:08.552 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.351661+0000", 2024-04-20T17:11:08.552 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7107999999999998e-05 2024-04-20T17:11:08.552 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.552 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:08.552 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.552 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.552 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.552 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:180927 2.5 2:aaedb533:::benchmark_data_smithi171_78918_object19933:head [delete] snapc 0=[] ondisk+write+known_if_redirected e156)", 2024-04-20T17:11:08.552 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:12.318653+0000", 2024-04-20T17:11:08.552 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 56.178116844999998, 2024-04-20T17:11:08.552 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.033123749000000001, 2024-04-20T17:11:08.552 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:08.552 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:08.553 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:08.553 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:08.553 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:08.553 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:08.553 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 180927 2024-04-20T17:11:08.553 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.553 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:08.553 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.553 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:08.553 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.318653+0000", 2024-04-20T17:11:08.553 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.553 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.553 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.553 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:08.554 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.318653+0000", 2024-04-20T17:11:08.554 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.554 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.554 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.554 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:08.554 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.318654+0000", 2024-04-20T17:11:08.554 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1099999999999995e-07 2024-04-20T17:11:08.554 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.554 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.554 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:08.554 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.318659+0000", 2024-04-20T17:11:08.554 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0699999999999997e-06 2024-04-20T17:11:08.554 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.554 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.555 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:08.555 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.318665+0000", 2024-04-20T17:11:08.555 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3759999999999996e-06 2024-04-20T17:11:08.555 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.555 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.555 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:08.555 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.318671+0000", 2024-04-20T17:11:08.555 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6250000000000004e-06 2024-04-20T17:11:08.555 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.555 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.555 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:08.555 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.323237+0000", 2024-04-20T17:11:08.555 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0045659450000000001 2024-04-20T17:11:08.555 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.556 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.556 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:08.556 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.323361+0000", 2024-04-20T17:11:08.556 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012377300000000001 2024-04-20T17:11:08.556 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.556 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.556 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:08.556 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.323485+0000", 2024-04-20T17:11:08.556 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012416900000000001 2024-04-20T17:11:08.556 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.556 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.556 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:08.556 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.331721+0000", 2024-04-20T17:11:08.556 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0082364789999999997 2024-04-20T17:11:08.556 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.557 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.557 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:08.557 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.351702+0000", 2024-04-20T17:11:08.557 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019980940999999999 2024-04-20T17:11:08.557 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.557 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.557 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:08.557 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.351746+0000", 2024-04-20T17:11:08.557 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3974999999999998e-05 2024-04-20T17:11:08.557 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.557 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.557 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:08.557 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.351777+0000", 2024-04-20T17:11:08.557 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0685000000000002e-05 2024-04-20T17:11:08.558 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.558 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:08.558 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.558 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.558 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.558 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:180928 2.5 2:a520eb16:::benchmark_data_smithi171_78918_object19934:head [delete] snapc 0=[] ondisk+write+known_if_redirected e156)", 2024-04-20T17:11:08.558 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:12.318681+0000", 2024-04-20T17:11:08.558 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 56.178089264999997, 2024-04-20T17:11:08.558 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.033250865999999997, 2024-04-20T17:11:08.558 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:08.558 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:08.558 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:08.558 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:08.558 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:08.558 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:08.559 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 180928 2024-04-20T17:11:08.559 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.559 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:08.559 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.559 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:08.559 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.318681+0000", 2024-04-20T17:11:08.559 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.559 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.559 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.559 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:08.559 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.318681+0000", 2024-04-20T17:11:08.559 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.559 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.559 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.559 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:08.559 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.318681+0000", 2024-04-20T17:11:08.560 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4499999999999997e-07 2024-04-20T17:11:08.560 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.560 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.560 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:08.560 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.318683+0000", 2024-04-20T17:11:08.560 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.165e-06 2024-04-20T17:11:08.560 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.560 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.560 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:08.560 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.318687+0000", 2024-04-20T17:11:08.560 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.569e-06 2024-04-20T17:11:08.560 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.560 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.560 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:08.560 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.318690+0000", 2024-04-20T17:11:08.561 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1410000000000001e-06 2024-04-20T17:11:08.561 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.561 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.561 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:08.561 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.326948+0000", 2024-04-20T17:11:08.561 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0082579999999999997 2024-04-20T17:11:08.561 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.561 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.561 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:08.561 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.327068+0000", 2024-04-20T17:11:08.561 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000120188 2024-04-20T17:11:08.561 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.561 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.562 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:08.562 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.327183+0000", 2024-04-20T17:11:08.562 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000115166 2024-04-20T17:11:08.562 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.562 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.562 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:08.562 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.335777+0000", 2024-04-20T17:11:08.562 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0085932119999999994 2024-04-20T17:11:08.562 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.562 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.562 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:08.562 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.351837+0000", 2024-04-20T17:11:08.562 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.016060259 2024-04-20T17:11:08.562 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.562 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.563 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:08.563 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.351888+0000", 2024-04-20T17:11:08.563 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0775000000000001e-05 2024-04-20T17:11:08.563 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.563 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.563 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:08.563 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.351932+0000", 2024-04-20T17:11:08.563 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3946000000000002e-05 2024-04-20T17:11:08.563 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.563 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:08.563 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.563 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.563 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.563 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:180940 2.5 2:a5bcee2b:::benchmark_data_smithi171_78918_object19946:head [delete] snapc 0=[] ondisk+write+known_if_redirected e156)", 2024-04-20T17:11:08.564 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:12.334478+0000", 2024-04-20T17:11:08.564 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 56.162292524999998, 2024-04-20T17:11:08.564 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036018191999999997, 2024-04-20T17:11:08.564 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:08.564 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:08.564 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:08.564 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:08.564 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:08.564 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:08.564 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 180940 2024-04-20T17:11:08.564 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.564 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:08.564 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.564 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:08.564 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.334478+0000", 2024-04-20T17:11:08.564 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.565 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.565 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.565 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:08.565 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.334478+0000", 2024-04-20T17:11:08.565 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.565 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.565 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.565 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:08.565 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.334479+0000", 2024-04-20T17:11:08.565 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9700000000000002e-06 2024-04-20T17:11:08.565 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.565 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.565 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:08.566 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.334485+0000", 2024-04-20T17:11:08.566 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8579999999999996e-06 2024-04-20T17:11:08.566 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.566 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.566 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:08.566 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.334498+0000", 2024-04-20T17:11:08.566 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2641e-05 2024-04-20T17:11:08.566 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.566 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.566 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:08.566 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.334506+0000", 2024-04-20T17:11:08.566 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0050000000000004e-06 2024-04-20T17:11:08.566 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.566 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.567 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:08.567 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.335793+0000", 2024-04-20T17:11:08.567 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0012874589999999999 2024-04-20T17:11:08.567 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.567 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.567 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:08.567 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.335883+0000", 2024-04-20T17:11:08.567 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.954e-05 2024-04-20T17:11:08.567 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.567 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.567 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:08.567 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.335971+0000", 2024-04-20T17:11:08.567 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.7756000000000005e-05 2024-04-20T17:11:08.568 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.568 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.568 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:08.568 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.342095+0000", 2024-04-20T17:11:08.568 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0061244769999999997 2024-04-20T17:11:08.568 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.568 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.568 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:08.568 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.370381+0000", 2024-04-20T17:11:08.568 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028286252000000001 2024-04-20T17:11:08.568 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.568 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.568 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:08.568 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.370441+0000", 2024-04-20T17:11:08.568 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9216999999999999e-05 2024-04-20T17:11:08.569 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.569 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.569 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:08.569 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:12.370496+0000", 2024-04-20T17:11:08.569 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5016999999999999e-05 2024-04-20T17:11:08.569 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.569 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:08.569 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.569 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.569 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.569 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:183345 2.5 2:a587617f:::benchmark_data_smithi171_78918_object22351:head [delete] snapc 0=[] ondisk+write+known_if_redirected e157)", 2024-04-20T17:11:08.569 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:15.179751+0000", 2024-04-20T17:11:08.569 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 53.317018590000004, 2024-04-20T17:11:08.569 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.033445415999999999, 2024-04-20T17:11:08.569 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:08.570 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:08.570 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:08.570 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:08.570 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:08.570 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:08.570 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 183345 2024-04-20T17:11:08.570 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.570 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:08.570 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.570 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:08.570 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:15.179751+0000", 2024-04-20T17:11:08.570 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.570 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.571 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.571 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:08.571 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:15.179751+0000", 2024-04-20T17:11:08.571 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.571 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.571 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.571 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:08.571 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:15.179753+0000", 2024-04-20T17:11:08.571 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.581e-06 2024-04-20T17:11:08.571 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.571 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.571 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:08.571 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:15.179759+0000", 2024-04-20T17:11:08.571 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4840000000000003e-06 2024-04-20T17:11:08.571 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.572 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.572 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:08.572 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:15.179771+0000", 2024-04-20T17:11:08.572 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2326e-05 2024-04-20T17:11:08.572 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.572 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.572 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:08.572 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:15.179778+0000", 2024-04-20T17:11:08.572 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0790000000000003e-06 2024-04-20T17:11:08.572 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.572 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.572 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:08.572 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:15.179834+0000", 2024-04-20T17:11:08.572 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5608e-05 2024-04-20T17:11:08.573 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.573 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.573 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:08.573 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:15.179939+0000", 2024-04-20T17:11:08.573 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000105043 2024-04-20T17:11:08.573 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.573 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.573 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:08.573 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:15.180038+0000", 2024-04-20T17:11:08.573 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9784000000000006e-05 2024-04-20T17:11:08.573 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.573 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.573 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:08.574 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:15.188887+0000", 2024-04-20T17:11:08.574 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0088483750000000003 2024-04-20T17:11:08.574 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.574 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.574 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:08.574 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:15.213121+0000", 2024-04-20T17:11:08.574 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024234737999999999 2024-04-20T17:11:08.574 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.574 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.574 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:08.574 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:15.213162+0000", 2024-04-20T17:11:08.574 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0207999999999999e-05 2024-04-20T17:11:08.574 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.574 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.575 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:08.575 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:15.213197+0000", 2024-04-20T17:11:08.575 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.519e-05 2024-04-20T17:11:08.575 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.575 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:08.575 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.575 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.575 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.575 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:195841 2.5 2:a970f203:::benchmark_data_smithi171_78918_object34847:head [delete] snapc 0=[] ondisk+write+known_if_redirected e163)", 2024-04-20T17:11:08.575 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:30.697234+0000", 2024-04-20T17:11:08.575 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 37.799535788, 2024-04-20T17:11:08.575 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.037855572999999997, 2024-04-20T17:11:08.575 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:08.575 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:08.576 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:08.576 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:08.576 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:08.576 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:08.576 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 195841 2024-04-20T17:11:08.576 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.576 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:08.576 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.576 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:08.576 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.697234+0000", 2024-04-20T17:11:08.576 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.576 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.576 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.576 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:08.576 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.697234+0000", 2024-04-20T17:11:08.576 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.577 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.577 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.577 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:08.577 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.697235+0000", 2024-04-20T17:11:08.577 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1939999999999999e-06 2024-04-20T17:11:08.577 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.577 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.577 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:08.577 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.697239+0000", 2024-04-20T17:11:08.577 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0589999999999996e-06 2024-04-20T17:11:08.577 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.577 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.577 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:08.577 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.697247+0000", 2024-04-20T17:11:08.577 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5909999999999998e-06 2024-04-20T17:11:08.578 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.578 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.578 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:08.578 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.697254+0000", 2024-04-20T17:11:08.578 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5760000000000002e-06 2024-04-20T17:11:08.578 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.578 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.578 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:08.578 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.701087+0000", 2024-04-20T17:11:08.578 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.003833192 2024-04-20T17:11:08.578 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.578 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.578 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:08.578 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.701179+0000", 2024-04-20T17:11:08.579 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2037e-05 2024-04-20T17:11:08.579 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.579 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.579 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:08.579 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.701266+0000", 2024-04-20T17:11:08.579 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6891999999999994e-05 2024-04-20T17:11:08.579 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.579 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.579 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:08.579 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.704805+0000", 2024-04-20T17:11:08.579 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0035387330000000001 2024-04-20T17:11:08.579 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.579 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.579 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:08.579 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.735014+0000", 2024-04-20T17:11:08.579 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030209192999999999 2024-04-20T17:11:08.580 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.580 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.580 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:08.580 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.735056+0000", 2024-04-20T17:11:08.580 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2469000000000003e-05 2024-04-20T17:11:08.580 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.580 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.580 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:08.580 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.735090+0000", 2024-04-20T17:11:08.580 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3637000000000003e-05 2024-04-20T17:11:08.580 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.580 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:08.580 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.580 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.580 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.580 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:202267 2.5 2:abe7d7a2:::benchmark_data_smithi171_78918_object41273:head [delete] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-04-20T17:11:08.581 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:39.807427+0000", 2024-04-20T17:11:08.581 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 28.689343203, 2024-04-20T17:11:08.581 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.038790863000000002, 2024-04-20T17:11:08.581 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:08.581 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:08.581 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:08.581 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:08.581 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:08.581 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:08.581 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 202267 2024-04-20T17:11:08.581 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.581 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:08.581 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.581 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:08.582 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.807427+0000", 2024-04-20T17:11:08.582 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.582 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.582 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.582 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:08.582 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.807427+0000", 2024-04-20T17:11:08.582 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.582 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.582 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.582 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:08.582 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.807429+0000", 2024-04-20T17:11:08.582 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0959999999999999e-06 2024-04-20T17:11:08.582 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.582 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.582 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:08.583 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.807435+0000", 2024-04-20T17:11:08.583 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9499999999999998e-06 2024-04-20T17:11:08.583 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.583 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.583 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:08.583 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.807447+0000", 2024-04-20T17:11:08.583 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1654000000000001e-05 2024-04-20T17:11:08.583 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.583 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.583 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:08.583 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.807454+0000", 2024-04-20T17:11:08.583 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2790000000000001e-06 2024-04-20T17:11:08.583 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.583 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.583 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:08.583 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.807508+0000", 2024-04-20T17:11:08.583 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4370999999999998e-05 2024-04-20T17:11:08.584 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.584 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.584 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:08.584 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.807648+0000", 2024-04-20T17:11:08.584 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000139734 2024-04-20T17:11:08.584 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.584 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.584 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:08.584 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.807766+0000", 2024-04-20T17:11:08.584 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000118279 2024-04-20T17:11:08.584 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.584 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.584 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:08.584 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.811542+0000", 2024-04-20T17:11:08.585 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0037755639999999999 2024-04-20T17:11:08.585 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.585 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.585 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:08.585 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.846127+0000", 2024-04-20T17:11:08.585 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.034585171999999997 2024-04-20T17:11:08.585 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.585 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.585 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:08.585 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.846164+0000", 2024-04-20T17:11:08.585 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6671999999999997e-05 2024-04-20T17:11:08.585 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.585 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.585 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:08.586 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.846218+0000", 2024-04-20T17:11:08.586 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4092000000000003e-05 2024-04-20T17:11:08.586 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.586 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:08.586 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.586 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.586 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.586 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:205499 2.b 2:d1d4321c:::benchmark_data_smithi171_78918_object44505:head [delete] snapc 0=[] ondisk+write+known_if_redirected e167)", 2024-04-20T17:11:08.586 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:43.228332+0000", 2024-04-20T17:11:08.586 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 25.268438306, 2024-04-20T17:11:08.586 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.031662832000000002, 2024-04-20T17:11:08.586 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:08.586 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:08.586 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:08.587 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:08.587 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:08.587 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:08.587 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 205499 2024-04-20T17:11:08.587 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.587 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:08.587 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.587 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:08.587 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.228332+0000", 2024-04-20T17:11:08.587 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.587 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.587 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.587 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:08.587 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.228332+0000", 2024-04-20T17:11:08.587 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.588 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.588 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.588 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:08.588 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.228332+0000", 2024-04-20T17:11:08.588 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.37e-07 2024-04-20T17:11:08.588 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.588 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.588 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:08.588 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.228335+0000", 2024-04-20T17:11:08.588 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5909999999999998e-06 2024-04-20T17:11:08.588 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.588 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.588 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:08.588 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.228340+0000", 2024-04-20T17:11:08.589 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8810000000000004e-06 2024-04-20T17:11:08.589 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.589 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.589 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:08.589 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.228345+0000", 2024-04-20T17:11:08.589 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2789999999999998e-06 2024-04-20T17:11:08.589 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.589 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.589 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:08.589 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.241464+0000", 2024-04-20T17:11:08.589 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013118635999999999 2024-04-20T17:11:08.589 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.589 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.589 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:08.590 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.241552+0000", 2024-04-20T17:11:08.590 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8233999999999996e-05 2024-04-20T17:11:08.590 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.590 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.590 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:11:08.590 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.241640+0000", 2024-04-20T17:11:08.590 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8233999999999996e-05 2024-04-20T17:11:08.590 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.590 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.590 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:08.590 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.259850+0000", 2024-04-20T17:11:08.590 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018210088999999999 2024-04-20T17:11:08.590 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.590 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.591 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:08.591 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.259903+0000", 2024-04-20T17:11:08.591 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2132000000000002e-05 2024-04-20T17:11:08.591 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.591 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.591 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:08.591 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.259953+0000", 2024-04-20T17:11:08.591 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0788999999999998e-05 2024-04-20T17:11:08.591 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.591 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.591 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:08.591 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.259995+0000", 2024-04-20T17:11:08.591 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1230000000000003e-05 2024-04-20T17:11:08.591 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.592 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:08.592 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.592 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.592 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.592 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:205502 2.b 2:da81e759:::benchmark_data_smithi171_78918_object44508:head [delete] snapc 0=[] ondisk+write+known_if_redirected e167)", 2024-04-20T17:11:08.592 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:43.228493+0000", 2024-04-20T17:11:08.592 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 25.268276759999999, 2024-04-20T17:11:08.592 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036099956000000002, 2024-04-20T17:11:08.592 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:08.592 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:08.592 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:08.592 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:08.592 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:08.592 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:08.593 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 205502 2024-04-20T17:11:08.593 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.593 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:08.593 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.593 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:08.593 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.228493+0000", 2024-04-20T17:11:08.593 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.593 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.593 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.593 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:08.593 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.228493+0000", 2024-04-20T17:11:08.593 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.593 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.594 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.594 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:08.594 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.228495+0000", 2024-04-20T17:11:08.594 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5370000000000001e-06 2024-04-20T17:11:08.594 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.594 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.594 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:08.594 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.228501+0000", 2024-04-20T17:11:08.594 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8939999999999998e-06 2024-04-20T17:11:08.594 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.594 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.594 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:08.594 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.228510+0000", 2024-04-20T17:11:08.594 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6509999999999996e-06 2024-04-20T17:11:08.595 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.595 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.595 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:08.595 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.228519+0000", 2024-04-20T17:11:08.595 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6600000000000001e-06 2024-04-20T17:11:08.595 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.595 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.595 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:08.595 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.245447+0000", 2024-04-20T17:11:08.595 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.016928144999999999 2024-04-20T17:11:08.595 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.595 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.595 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:08.595 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.245548+0000", 2024-04-20T17:11:08.596 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010089 2024-04-20T17:11:08.596 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.596 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.596 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:11:08.596 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.245644+0000", 2024-04-20T17:11:08.596 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6254000000000004e-05 2024-04-20T17:11:08.596 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.596 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.596 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:08.596 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.260040+0000", 2024-04-20T17:11:08.596 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014396153 2024-04-20T17:11:08.596 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.596 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.596 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:08.596 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.264494+0000", 2024-04-20T17:11:08.597 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0044534120000000003 2024-04-20T17:11:08.597 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.597 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.597 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:08.597 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.264544+0000", 2024-04-20T17:11:08.597 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0451000000000002e-05 2024-04-20T17:11:08.597 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.597 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.597 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:08.597 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.264593+0000", 2024-04-20T17:11:08.597 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8909000000000002e-05 2024-04-20T17:11:08.597 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.597 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:08.597 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.597 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.598 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.598 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:209428 2.5 2:aeb1019e:::benchmark_data_smithi171_78918_object48434:head [delete] snapc 0=[] ondisk+write+known_if_redirected e167)", 2024-04-20T17:11:08.598 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:47.597346+0000", 2024-04-20T17:11:08.598 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 20.899424057000001, 2024-04-20T17:11:08.598 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.031794458999999997, 2024-04-20T17:11:08.598 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:08.598 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:08.598 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:08.598 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:08.598 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:08.598 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:08.598 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 209428 2024-04-20T17:11:08.598 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.598 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:08.598 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.599 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:08.599 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.597346+0000", 2024-04-20T17:11:08.599 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.599 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.599 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.599 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:08.599 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.597346+0000", 2024-04-20T17:11:08.599 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.599 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.599 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.599 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:08.599 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.597347+0000", 2024-04-20T17:11:08.599 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7999999999999993e-07 2024-04-20T17:11:08.599 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.600 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.600 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:08.600 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.597351+0000", 2024-04-20T17:11:08.600 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7239999999999998e-06 2024-04-20T17:11:08.600 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.600 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.600 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:08.600 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.597357+0000", 2024-04-20T17:11:08.600 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1029999999999998e-06 2024-04-20T17:11:08.600 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.600 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.600 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:08.600 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.597362+0000", 2024-04-20T17:11:08.600 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5960000000000003e-06 2024-04-20T17:11:08.600 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.601 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.601 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:08.601 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.602369+0000", 2024-04-20T17:11:08.601 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0050068650000000001 2024-04-20T17:11:08.601 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.601 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.601 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:08.601 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.602477+0000", 2024-04-20T17:11:08.601 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010818 2024-04-20T17:11:08.601 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.601 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.601 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:08.601 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.602594+0000", 2024-04-20T17:11:08.601 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011668 2024-04-20T17:11:08.601 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.602 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.602 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:08.602 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.613321+0000", 2024-04-20T17:11:08.602 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.01072686 2024-04-20T17:11:08.602 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.602 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.602 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:08.602 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.629075+0000", 2024-04-20T17:11:08.602 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015753612 2024-04-20T17:11:08.602 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.602 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.602 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:08.602 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.629104+0000", 2024-04-20T17:11:08.602 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9535000000000001e-05 2024-04-20T17:11:08.603 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.603 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.603 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:08.603 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.629140+0000", 2024-04-20T17:11:08.603 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6324e-05 2024-04-20T17:11:08.603 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.603 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:08.603 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.603 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.603 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.603 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:209959 2.f 2:f774094c:::benchmark_data_smithi171_78918_object48965:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-04-20T17:11:08.603 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:48.192736+0000", 2024-04-20T17:11:08.603 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 20.304034368, 2024-04-20T17:11:08.603 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.035735277000000003, 2024-04-20T17:11:08.604 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:08.604 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:08.604 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:08.604 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:08.604 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:08.604 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:08.604 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 209959 2024-04-20T17:11:08.604 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.604 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:08.604 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.604 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:08.604 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.192736+0000", 2024-04-20T17:11:08.604 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.604 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.605 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.605 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:08.605 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.192736+0000", 2024-04-20T17:11:08.605 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.605 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.605 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.605 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:08.605 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.192737+0000", 2024-04-20T17:11:08.605 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.085e-06 2024-04-20T17:11:08.605 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.605 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.605 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:08.605 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.192741+0000", 2024-04-20T17:11:08.605 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7610000000000001e-06 2024-04-20T17:11:08.605 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.606 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.606 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:08.606 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.192746+0000", 2024-04-20T17:11:08.606 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6559999999999997e-06 2024-04-20T17:11:08.606 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.606 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.606 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:08.606 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.192752+0000", 2024-04-20T17:11:08.606 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4389999999999999e-06 2024-04-20T17:11:08.606 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.606 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.606 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:08.606 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.197509+0000", 2024-04-20T17:11:08.606 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0047570579999999998 2024-04-20T17:11:08.607 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.607 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.607 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:08.607 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.197611+0000", 2024-04-20T17:11:08.607 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000102668 2024-04-20T17:11:08.607 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.607 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.607 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:11:08.607 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.197716+0000", 2024-04-20T17:11:08.607 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000104348 2024-04-20T17:11:08.607 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.607 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.607 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:08.607 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.213918+0000", 2024-04-20T17:11:08.608 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.016202145000000001 2024-04-20T17:11:08.608 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.608 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.608 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:08.608 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.228388+0000", 2024-04-20T17:11:08.608 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014470255 2024-04-20T17:11:08.608 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.608 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.608 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:08.608 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.228441+0000", 2024-04-20T17:11:08.608 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3340000000000001e-05 2024-04-20T17:11:08.608 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.608 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.608 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:08.608 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.228471+0000", 2024-04-20T17:11:08.609 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9522e-05 2024-04-20T17:11:08.609 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.609 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:08.609 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.609 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.609 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.609 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:209961 2.f 2:f3563409:::benchmark_data_smithi171_78918_object48967:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-04-20T17:11:08.609 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:48.193106+0000", 2024-04-20T17:11:08.609 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 20.303664292000001, 2024-04-20T17:11:08.609 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.035446084000000003, 2024-04-20T17:11:08.609 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:08.609 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:08.609 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:08.609 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:08.610 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:08.610 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:08.610 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 209961 2024-04-20T17:11:08.610 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.610 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:08.610 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.610 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:08.610 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.193106+0000", 2024-04-20T17:11:08.610 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.610 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.610 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.610 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:08.610 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.193106+0000", 2024-04-20T17:11:08.610 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.611 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.611 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.611 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:08.611 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.193107+0000", 2024-04-20T17:11:08.611 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4839999999999999e-06 2024-04-20T17:11:08.611 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.611 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.611 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:08.611 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.193112+0000", 2024-04-20T17:11:08.611 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0649999999999998e-06 2024-04-20T17:11:08.611 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.611 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.611 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:08.611 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.193125+0000", 2024-04-20T17:11:08.611 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3182e-05 2024-04-20T17:11:08.611 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.612 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.612 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:08.612 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.193133+0000", 2024-04-20T17:11:08.612 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8310000000000001e-06 2024-04-20T17:11:08.612 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.612 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.612 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:08.612 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.202343+0000", 2024-04-20T17:11:08.612 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0092099420000000005 2024-04-20T17:11:08.612 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.612 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.612 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:08.612 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.202439+0000", 2024-04-20T17:11:08.612 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6224999999999994e-05 2024-04-20T17:11:08.612 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.613 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.613 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:11:08.613 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.202529+0000", 2024-04-20T17:11:08.613 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9407999999999995e-05 2024-04-20T17:11:08.613 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.613 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.613 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:08.613 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.213923+0000", 2024-04-20T17:11:08.613 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.01139401 2024-04-20T17:11:08.613 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.613 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.613 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:08.613 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.228494+0000", 2024-04-20T17:11:08.613 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014571459 2024-04-20T17:11:08.613 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.614 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.614 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:08.614 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.228519+0000", 2024-04-20T17:11:08.614 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4861000000000001e-05 2024-04-20T17:11:08.614 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.614 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.614 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:08.614 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.228552+0000", 2024-04-20T17:11:08.614 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2617000000000003e-05 2024-04-20T17:11:08.614 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.614 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:08.614 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.614 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.615 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.615 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:209965 2.f 2:f12687c9:::benchmark_data_smithi171_78918_object48971:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-04-20T17:11:08.615 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:48.196514+0000", 2024-04-20T17:11:08.615 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 20.300255602, 2024-04-20T17:11:08.615 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032143523, 2024-04-20T17:11:08.615 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:08.615 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:08.615 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:08.615 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:08.615 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:08.615 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:08.615 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 209965 2024-04-20T17:11:08.615 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.615 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:08.616 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.616 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:08.616 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.196514+0000", 2024-04-20T17:11:08.616 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.616 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.616 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.616 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:08.616 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.196514+0000", 2024-04-20T17:11:08.616 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:08.616 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.616 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.616 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:08.616 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.196517+0000", 2024-04-20T17:11:08.616 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3089999999999998e-06 2024-04-20T17:11:08.617 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.617 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.617 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:08.617 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.196524+0000", 2024-04-20T17:11:08.617 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.396e-06 2024-04-20T17:11:08.617 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.617 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.617 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:08.617 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.196535+0000", 2024-04-20T17:11:08.617 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0402e-05 2024-04-20T17:11:08.617 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.617 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.617 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:08.617 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.196543+0000", 2024-04-20T17:11:08.617 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2700000000000004e-06 2024-04-20T17:11:08.618 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.618 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.618 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:08.618 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.206109+0000", 2024-04-20T17:11:08.618 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0095658240000000005 2024-04-20T17:11:08.618 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.618 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.618 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:08.618 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.206193+0000", 2024-04-20T17:11:08.618 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4606999999999998e-05 2024-04-20T17:11:08.618 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.618 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.618 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:11:08.618 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.206334+0000", 2024-04-20T17:11:08.619 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000141099 2024-04-20T17:11:08.619 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.619 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.619 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:08.619 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.213930+0000", 2024-04-20T17:11:08.619 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0075955579999999997 2024-04-20T17:11:08.619 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.619 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.619 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:08.619 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.228586+0000", 2024-04-20T17:11:08.619 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014656199999999999 2024-04-20T17:11:08.619 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.619 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.619 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:08.619 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.228632+0000", 2024-04-20T17:11:08.620 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6007000000000002e-05 2024-04-20T17:11:08.620 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:08.620 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:08.620 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:08.620 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.228658+0000", 2024-04-20T17:11:08.620 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5851e-05 2024-04-20T17:11:08.620 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.620 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:08.620 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.620 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:08.620 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:08.620 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:11:08.652 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7818adb2:::benchmark_data_smithi171_78918_object61587:head# 2024-04-20T17:11:08.652 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7819af95:::benchmark_data_smithi171_78918_object124871:head# 2024-04-20T17:11:08.652 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:781a73c6:::benchmark_data_smithi171_78918_object89946:head# 2024-04-20T17:11:08.652 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:781ad95b:::benchmark_data_smithi171_78918_object63807:head# 2024-04-20T17:11:08.652 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:781b12cf:::benchmark_data_smithi171_78918_object105447:head# 2024-04-20T17:11:08.652 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:781b9c4f:::benchmark_data_smithi171_78918_object67418:head# 2024-04-20T17:11:08.652 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:781bab05:::benchmark_data_smithi171_78918_object120063:head# 2024-04-20T17:11:08.652 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:781bb0e1:::benchmark_data_smithi171_78918_object131013:head# 2024-04-20T17:11:08.652 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:781c34b0:::benchmark_data_smithi171_78918_object120665:head# 2024-04-20T17:11:08.652 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:781cc934:::benchmark_data_smithi171_78918_object154418:head# 2024-04-20T17:11:08.652 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:781cdc1b:::benchmark_data_smithi171_78918_object159949:head# 2024-04-20T17:11:08.652 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:781cfc1d:::benchmark_data_smithi171_78918_object120981:head# 2024-04-20T17:11:08.653 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:781d9ab2:::benchmark_data_smithi171_78918_object116475:head# 2024-04-20T17:11:08.653 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:781dae61:::benchmark_data_smithi171_78918_object128194:head# 2024-04-20T17:11:08.653 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:781e60e7:::benchmark_data_smithi171_78918_object109806:head# 2024-04-20T17:11:08.653 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:781ecca6:::benchmark_data_smithi171_78918_object149885:head# 2024-04-20T17:11:08.653 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:781ffe87:::benchmark_data_smithi171_78918_object147857:head# 2024-04-20T17:11:08.653 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78206256:::benchmark_data_smithi171_78918_object126627:head# 2024-04-20T17:11:08.653 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7820abcc:::benchmark_data_smithi171_78918_object160854:head# 2024-04-20T17:11:08.653 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7821cf64:::benchmark_data_smithi171_78918_object150874:head# 2024-04-20T17:11:08.653 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7821d91f:::benchmark_data_smithi171_78918_object153101:head# 2024-04-20T17:11:08.653 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7823f7b7:::benchmark_data_smithi171_78918_object60430:head# 2024-04-20T17:11:08.653 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78244162:::benchmark_data_smithi171_78918_object153099:head# 2024-04-20T17:11:08.653 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78250fe8:::benchmark_data_smithi171_78918_object101605:head# 2024-04-20T17:11:08.653 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78262c16:::benchmark_data_smithi171_78918_object74480:head# 2024-04-20T17:11:08.653 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:782789b1:::benchmark_data_smithi171_78918_object82454:head# 2024-04-20T17:11:08.654 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78278a46:::benchmark_data_smithi171_78918_object123768:head# 2024-04-20T17:11:08.654 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7827df7c:::benchmark_data_smithi171_78918_object84479:head# 2024-04-20T17:11:08.654 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:782812f6:::benchmark_data_smithi171_78918_object79186:head# 2024-04-20T17:11:08.654 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7829830d:::benchmark_data_smithi171_78918_object126497:head# 2024-04-20T17:11:08.654 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78299511:::benchmark_data_smithi171_78918_object76826:head# 2024-04-20T17:11:08.654 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7829eded:::benchmark_data_smithi171_78918_object102506:head# 2024-04-20T17:11:08.654 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:782a1a05:::benchmark_data_smithi171_78918_object113421:head# 2024-04-20T17:11:08.654 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:782aa200:::benchmark_data_smithi171_78918_object70082:head# 2024-04-20T17:11:08.654 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:782c1fd9:::benchmark_data_smithi171_78918_object92142:head# 2024-04-20T17:11:08.654 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:782d7fff:::benchmark_data_smithi171_78918_object107061:head# 2024-04-20T17:11:08.654 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:782e28d2:::benchmark_data_smithi171_78918_object51501:head# 2024-04-20T17:11:08.654 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:782e586f:::benchmark_data_smithi171_78918_object104591:head# 2024-04-20T17:11:08.654 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:782e9bd3:::benchmark_data_smithi171_78918_object65714:head# 2024-04-20T17:11:08.654 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:782ee108:::benchmark_data_smithi171_78918_object89974:head# 2024-04-20T17:11:08.655 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:782f249b:::benchmark_data_smithi171_78918_object81402:head# 2024-04-20T17:11:08.655 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:782ff18d:::benchmark_data_smithi171_78918_object140869:head# 2024-04-20T17:11:08.655 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78306fdf:::benchmark_data_smithi171_78918_object112175:head# 2024-04-20T17:11:08.655 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78309ae2:::benchmark_data_smithi171_78918_object141024:head# 2024-04-20T17:11:08.655 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7830d7c3:::benchmark_data_smithi171_78918_object101477:head# 2024-04-20T17:11:08.655 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:783164b5:::benchmark_data_smithi171_78918_object156663:head# 2024-04-20T17:11:08.655 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:783196ed:::benchmark_data_smithi171_78918_object136640:head# 2024-04-20T17:11:08.655 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78320652:::benchmark_data_smithi171_78918_object58223:head# 2024-04-20T17:11:08.655 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78320d81:::benchmark_data_smithi171_78918_object112722:head# 2024-04-20T17:11:08.655 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78326be7:::benchmark_data_smithi171_78918_object144518:head# 2024-04-20T17:11:08.655 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7832a25a:::benchmark_data_smithi171_78918_object141609:head# 2024-04-20T17:11:08.655 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7832d0e6:::benchmark_data_smithi171_78918_object106353:head# 2024-04-20T17:11:08.655 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7832e1e0:::benchmark_data_smithi171_78918_object113833:head# 2024-04-20T17:11:08.655 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78338b88:::benchmark_data_smithi171_78918_object57479:head# 2024-04-20T17:11:08.656 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78344ca8:::benchmark_data_smithi171_78918_object122937:head# 2024-04-20T17:11:08.656 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7835816b:::benchmark_data_smithi171_78918_object108634:head# 2024-04-20T17:11:08.656 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:783635b7:::benchmark_data_smithi171_78918_object148184:head# 2024-04-20T17:11:08.656 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7837ba08:::benchmark_data_smithi171_78918_object143157:head# 2024-04-20T17:11:08.656 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7838d0a7:::benchmark_data_smithi171_78918_object57604:head# 2024-04-20T17:11:08.656 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78393da7:::benchmark_data_smithi171_78918_object61319:head# 2024-04-20T17:11:08.656 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78395dbd:::benchmark_data_smithi171_78918_object111866:head# 2024-04-20T17:11:08.656 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7839b8a6:::benchmark_data_smithi171_78918_object134671:head# 2024-04-20T17:11:08.656 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:783a566b:::benchmark_data_smithi171_78918_object56168:head# 2024-04-20T17:11:08.656 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:783a8926:::benchmark_data_smithi171_78918_object138204:head# 2024-04-20T17:11:08.656 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:783b052b:::benchmark_data_smithi171_78918_object148578:head# 2024-04-20T17:11:08.656 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:783c3214:::benchmark_data_smithi171_78918_object92995:head# 2024-04-20T17:11:08.656 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:783c38a1:::benchmark_data_smithi171_78918_object137570:head# 2024-04-20T17:11:08.656 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:783cd5d9:::benchmark_data_smithi171_78918_object65557:head# 2024-04-20T17:11:08.656 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:783d6b79:::benchmark_data_smithi171_78918_object123517:head# 2024-04-20T17:11:08.657 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:783eef18:::benchmark_data_smithi171_78918_object73703:head# 2024-04-20T17:11:08.657 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7840dfa6:::benchmark_data_smithi171_78918_object116395:head# 2024-04-20T17:11:08.657 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7840fc3e:::benchmark_data_smithi171_78918_object61164:head# 2024-04-20T17:11:08.657 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78444d9f:::benchmark_data_smithi171_78918_object148539:head# 2024-04-20T17:11:08.657 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78478e32:::benchmark_data_smithi171_78918_object113868:head# 2024-04-20T17:11:08.657 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7847b6db:::benchmark_data_smithi171_78918_object129720:head# 2024-04-20T17:11:08.657 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:784889c0:::benchmark_data_smithi171_78918_object137631:head# 2024-04-20T17:11:08.657 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7848cf78:::benchmark_data_smithi171_78918_object124446:head# 2024-04-20T17:11:08.657 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7848f025:::benchmark_data_smithi171_78918_object90781:head# 2024-04-20T17:11:08.657 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:784949bc:::benchmark_data_smithi171_78918_object100457:head# 2024-04-20T17:11:08.657 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78498bf3:::benchmark_data_smithi171_78918_object94039:head# 2024-04-20T17:11:08.657 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7849d12f:::benchmark_data_smithi171_78918_object81424:head# 2024-04-20T17:11:08.657 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7849fbe5:::benchmark_data_smithi171_78918_object58169:head# 2024-04-20T17:11:08.657 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:784a2526:::benchmark_data_smithi171_78918_object127141:head# 2024-04-20T17:11:08.657 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:784b222f:::benchmark_data_smithi171_78918_object71493:head# 2024-04-20T17:11:08.658 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:784bc2d7:::benchmark_data_smithi171_78918_object141167:head# 2024-04-20T17:11:08.658 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:784c1464:::benchmark_data_smithi171_78918_object58041:head# 2024-04-20T17:11:08.658 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:784c8da2:::benchmark_data_smithi171_78918_object132033:head# 2024-04-20T17:11:08.658 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:784ca073:::benchmark_data_smithi171_78918_object77148:head# 2024-04-20T17:11:08.658 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:784f27bd:::benchmark_data_smithi171_78918_object118452:head# 2024-04-20T17:11:08.658 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:784f38e2:::benchmark_data_smithi171_78918_object148650:head# 2024-04-20T17:11:08.658 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:784ffdd4:::benchmark_data_smithi171_78918_object119438:head# 2024-04-20T17:11:08.658 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78503e99:::benchmark_data_smithi171_78918_object130164:head# 2024-04-20T17:11:08.658 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7851cbd7:::benchmark_data_smithi171_78918_object133927:head# 2024-04-20T17:11:08.658 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78522319:::benchmark_data_smithi171_78918_object81892:head# 2024-04-20T17:11:08.658 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78529951:::benchmark_data_smithi171_78918_object115601:head# 2024-04-20T17:11:08.658 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7852ba5d:::benchmark_data_smithi171_78918_object115033:head# 2024-04-20T17:11:08.658 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78533049:::benchmark_data_smithi171_78918_object106380:head# 2024-04-20T17:11:08.658 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78541932:::benchmark_data_smithi171_78918_object87812:head# 2024-04-20T17:11:08.659 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78545916:::benchmark_data_smithi171_78918_object105845:head# 2024-04-20T17:11:08.659 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78551312:::benchmark_data_smithi171_78918_object160306:head# 2024-04-20T17:11:08.659 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78553c3b:::benchmark_data_smithi171_78918_object151396:head# 2024-04-20T17:11:08.659 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7856724c:::benchmark_data_smithi171_78918_object96871:head# 2024-04-20T17:11:08.659 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:785832b0:::benchmark_data_smithi171_78918_object59448:head# 2024-04-20T17:11:08.659 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7858cc0a:::benchmark_data_smithi171_78918_object79079:head# 2024-04-20T17:11:08.659 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78590c79:::benchmark_data_smithi171_78918_object68288:head# 2024-04-20T17:11:08.659 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78593461:::benchmark_data_smithi171_78918_object85273:head# 2024-04-20T17:11:08.659 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78598c64:::benchmark_data_smithi171_78918_object121220:head# 2024-04-20T17:11:08.659 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78599e3e:::benchmark_data_smithi171_78918_object134587:head# 2024-04-20T17:11:08.659 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7859ecb2:::benchmark_data_smithi171_78918_object144247:head# 2024-04-20T17:11:08.659 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:785a17f0:::benchmark_data_smithi171_78918_object104317:head# 2024-04-20T17:11:08.659 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:785a63ee:::benchmark_data_smithi171_78918_object82377:head# 2024-04-20T17:11:08.659 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:785aedb3:::benchmark_data_smithi171_78918_object85568:head# 2024-04-20T17:11:08.659 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:785c40a7:::benchmark_data_smithi171_78918_object132125:head# 2024-04-20T17:11:08.660 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:785d9fd0:::benchmark_data_smithi171_78918_object156257:head# 2024-04-20T17:11:08.660 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:785e7d2a:::benchmark_data_smithi171_78918_object72427:head# 2024-04-20T17:11:08.660 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:785ea99b:::benchmark_data_smithi171_78918_object70320:head# 2024-04-20T17:11:08.660 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:785f420b:::benchmark_data_smithi171_78918_object108188:head# 2024-04-20T17:11:08.660 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:785fd5b1:::benchmark_data_smithi171_78918_object160190:head# 2024-04-20T17:11:08.660 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7860fbe9:::benchmark_data_smithi171_78918_object119895:head# 2024-04-20T17:11:08.660 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7861082d:::benchmark_data_smithi171_78918_object65038:head# 2024-04-20T17:11:08.660 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7861a007:::benchmark_data_smithi171_78918_object158747:head# 2024-04-20T17:11:08.660 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7861c564:::benchmark_data_smithi171_78918_object89578:head# 2024-04-20T17:11:08.660 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78636c7e:::benchmark_data_smithi171_78918_object136152:head# 2024-04-20T17:11:08.660 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7863863e:::benchmark_data_smithi171_78918_object102447:head# 2024-04-20T17:11:08.660 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7863bffc:::benchmark_data_smithi171_78918_object111544:head# 2024-04-20T17:11:08.660 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:786401b3:::benchmark_data_smithi171_78918_object130202:head# 2024-04-20T17:11:08.660 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78660cc3:::benchmark_data_smithi171_78918_object70132:head# 2024-04-20T17:11:08.661 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78664a83:::benchmark_data_smithi171_78918_object120668:head# 2024-04-20T17:11:08.661 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7867155e:::benchmark_data_smithi171_78918_object157882:head# 2024-04-20T17:11:08.661 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78674719:::benchmark_data_smithi171_78918_object72255:head# 2024-04-20T17:11:08.661 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7868222d:::benchmark_data_smithi171_78918_object102000:head# 2024-04-20T17:11:08.661 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78688d32:::benchmark_data_smithi171_78918_object93557:head# 2024-04-20T17:11:08.661 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7869199f:::benchmark_data_smithi171_78918_object145916:head# 2024-04-20T17:11:08.661 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78697a19:::benchmark_data_smithi171_78918_object117712:head# 2024-04-20T17:11:08.661 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78698795:::benchmark_data_smithi171_78918_object129686:head# 2024-04-20T17:11:08.661 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7869ace8:::benchmark_data_smithi171_78918_object72438:head# 2024-04-20T17:11:08.661 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:786ab3d3:::benchmark_data_smithi171_78918_object121695:head# 2024-04-20T17:11:08.661 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:786adb10:::benchmark_data_smithi171_78918_object74725:head# 2024-04-20T17:11:08.661 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:786af079:::benchmark_data_smithi171_78918_object110269:head# 2024-04-20T17:11:08.661 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:786b9ec7:::benchmark_data_smithi171_78918_object63777:head# 2024-04-20T17:11:08.661 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:786bec71:::benchmark_data_smithi171_78918_object128248:head# 2024-04-20T17:11:08.662 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:786ca068:::benchmark_data_smithi171_78918_object60359:head# 2024-04-20T17:11:08.662 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:786d1892:::benchmark_data_smithi171_78918_object84931:head# 2024-04-20T17:11:08.662 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:786e6a46:::benchmark_data_smithi171_78918_object88863:head# 2024-04-20T17:11:08.662 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:786e99eb:::benchmark_data_smithi171_78918_object94479:head# 2024-04-20T17:11:08.662 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:786f1da7:::benchmark_data_smithi171_78918_object106486:head# 2024-04-20T17:11:08.662 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:786f8b2b:::benchmark_data_smithi171_78918_object143172:head# 2024-04-20T17:11:08.662 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:786fb6f4:::benchmark_data_smithi171_78918_object116064:head# 2024-04-20T17:11:08.662 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:786fe480:::benchmark_data_smithi171_78918_object141089:head# 2024-04-20T17:11:08.662 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78700c16:::benchmark_data_smithi171_78918_object67647:head# 2024-04-20T17:11:08.662 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:787018a5:::benchmark_data_smithi171_78918_object156140:head# 2024-04-20T17:11:08.662 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78705d9e:::benchmark_data_smithi171_78918_object69035:head# 2024-04-20T17:11:08.662 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7870f2fb:::benchmark_data_smithi171_78918_object111235:head# 2024-04-20T17:11:08.662 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78716a1d:::benchmark_data_smithi171_78918_object132221:head# 2024-04-20T17:11:08.662 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7871ca3b:::benchmark_data_smithi171_78918_object112999:head# 2024-04-20T17:11:08.663 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7872c7ef:::benchmark_data_smithi171_78918_object138380:head# 2024-04-20T17:11:08.663 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:787300af:::benchmark_data_smithi171_78918_object145988:head# 2024-04-20T17:11:08.663 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:787310da:::benchmark_data_smithi171_78918_object101889:head# 2024-04-20T17:11:08.663 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7873160c:::benchmark_data_smithi171_78918_object72955:head# 2024-04-20T17:11:08.663 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78744faf:::benchmark_data_smithi171_78918_object96331:head# 2024-04-20T17:11:08.663 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:787481db:::benchmark_data_smithi171_78918_object110353:head# 2024-04-20T17:11:08.663 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7874fdf8:::benchmark_data_smithi171_78918_object61922:head# 2024-04-20T17:11:08.663 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78751b36:::benchmark_data_smithi171_78918_object62656:head# 2024-04-20T17:11:08.663 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78763493:::benchmark_data_smithi171_78918_object138443:head# 2024-04-20T17:11:08.663 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7877c642:::benchmark_data_smithi171_78918_object60121:head# 2024-04-20T17:11:08.663 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:787842bd:::benchmark_data_smithi171_78918_object120647:head# 2024-04-20T17:11:08.663 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7879ed82:::benchmark_data_smithi171_78918_object90288:head# 2024-04-20T17:11:08.663 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:787a190e:::benchmark_data_smithi171_78918_object158281:head# 2024-04-20T17:11:08.663 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:787a26e3:::benchmark_data_smithi171_78918_object91447:head# 2024-04-20T17:11:08.664 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:787a3474:::benchmark_data_smithi171_78918_object131438:head# 2024-04-20T17:11:08.664 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:787a5b76:::benchmark_data_smithi171_78918_object112724:head# 2024-04-20T17:11:08.664 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:787b1594:::benchmark_data_smithi171_78918_object127738:head# 2024-04-20T17:11:08.664 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:787bcd6b:::benchmark_data_smithi171_78918_object121274:head# 2024-04-20T17:11:08.664 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:787cda41:::benchmark_data_smithi171_78918_object138077:head# 2024-04-20T17:11:08.664 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:787d3104:::benchmark_data_smithi171_78918_object85998:head# 2024-04-20T17:11:08.664 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:787dc92e:::benchmark_data_smithi171_78918_object81141:head# 2024-04-20T17:11:08.664 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:787dffa6:::benchmark_data_smithi171_78918_object129148:head# 2024-04-20T17:11:08.664 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:787e2b79:::benchmark_data_smithi171_78918_object145665:head# 2024-04-20T17:11:08.664 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:788197b0:::benchmark_data_smithi171_78918_object113018:head# 2024-04-20T17:11:08.664 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78828387:::benchmark_data_smithi171_78918_object114622:head# 2024-04-20T17:11:08.664 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:788429dc:::benchmark_data_smithi171_78918_object86109:head# 2024-04-20T17:11:08.664 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78846324:::benchmark_data_smithi171_78918_object88420:head# 2024-04-20T17:11:08.664 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:788527f9:::benchmark_data_smithi171_78918_object128774:head# 2024-04-20T17:11:08.664 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7886b9f0:::benchmark_data_smithi171_78918_object103215:head# 2024-04-20T17:11:08.665 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7888de65:::benchmark_data_smithi171_78918_object97068:head# 2024-04-20T17:11:08.665 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:788a55c7:::benchmark_data_smithi171_78918_object112235:head# 2024-04-20T17:11:08.665 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:788aba18:::benchmark_data_smithi171_78918_object61087:head# 2024-04-20T17:11:08.665 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:788b3cf1:::benchmark_data_smithi171_78918_object144194:head# 2024-04-20T17:11:08.665 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:788bded8:::benchmark_data_smithi171_78918_object60087:head# 2024-04-20T17:11:08.665 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:788bf50d:::benchmark_data_smithi171_78918_object142449:head# 2024-04-20T17:11:08.665 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:788c8db1:::benchmark_data_smithi171_78918_object65512:head# 2024-04-20T17:11:08.665 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:788ca52d:::benchmark_data_smithi171_78918_object127981:head# 2024-04-20T17:11:08.665 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:788d7fc9:::benchmark_data_smithi171_78918_object73517:head# 2024-04-20T17:11:08.665 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:788d9919:::benchmark_data_smithi171_78918_object53482:head# 2024-04-20T17:11:08.665 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:788f63d9:::benchmark_data_smithi171_78918_object76671:head# 2024-04-20T17:11:08.665 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78912e14:::benchmark_data_smithi171_78918_object86103:head# 2024-04-20T17:11:08.665 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:789221fd:::benchmark_data_smithi171_78918_object54609:head# 2024-04-20T17:11:08.665 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78928e9f:::benchmark_data_smithi171_78918_object94357:head# 2024-04-20T17:11:08.665 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78935072:::benchmark_data_smithi171_78918_object125299:head# 2024-04-20T17:11:08.666 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:789589d7:::benchmark_data_smithi171_78918_object129726:head# 2024-04-20T17:11:08.666 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7895995e:::benchmark_data_smithi171_78918_object99493:head# 2024-04-20T17:11:08.666 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7895bcba:::benchmark_data_smithi171_78918_object74671:head# 2024-04-20T17:11:08.666 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7896109e:::benchmark_data_smithi171_78918_object83616:head# 2024-04-20T17:11:08.666 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78968459:::benchmark_data_smithi171_78918_object72157:head# 2024-04-20T17:11:08.666 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7896b2f0:::benchmark_data_smithi171_78918_object92873:head# 2024-04-20T17:11:08.666 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7896c1fb:::benchmark_data_smithi171_78918_object144363:head# 2024-04-20T17:11:08.666 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7897ec01:::benchmark_data_smithi171_78918_object91172:head# 2024-04-20T17:11:08.666 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78982477:::benchmark_data_smithi171_78918_object125162:head# 2024-04-20T17:11:08.666 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7898e41c:::benchmark_data_smithi171_78918_object137660:head# 2024-04-20T17:11:08.666 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:789aefe0:::benchmark_data_smithi171_78918_object124520:head# 2024-04-20T17:11:08.666 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:789b9fce:::benchmark_data_smithi171_78918_object160419:head# 2024-04-20T17:11:08.666 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:789d76ba:::benchmark_data_smithi171_78918_object141754:head# 2024-04-20T17:11:08.666 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:789db0c6:::benchmark_data_smithi171_78918_object106712:head# 2024-04-20T17:11:08.667 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:789de345:::benchmark_data_smithi171_78918_object75328:head# 2024-04-20T17:11:08.667 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:789de88d:::benchmark_data_smithi171_78918_object55203:head# 2024-04-20T17:11:08.667 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:789dfba2:::benchmark_data_smithi171_78918_object125222:head# 2024-04-20T17:11:08.667 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:789e96e8:::benchmark_data_smithi171_78918_object55962:head# 2024-04-20T17:11:08.667 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:789f2df5:::benchmark_data_smithi171_78918_object97898:head# 2024-04-20T17:11:08.667 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78a1ed76:::benchmark_data_smithi171_78918_object120971:head# 2024-04-20T17:11:08.667 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78a38ffb:::benchmark_data_smithi171_78918_object119746:head# 2024-04-20T17:11:08.667 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78a3efe5:::benchmark_data_smithi171_78918_object120890:head# 2024-04-20T17:11:08.667 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78a5482a:::benchmark_data_smithi171_78918_object55025:head# 2024-04-20T17:11:08.667 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78a662eb:::benchmark_data_smithi171_78918_object75731:head# 2024-04-20T17:11:08.667 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78a7362b:::benchmark_data_smithi171_78918_object129311:head# 2024-04-20T17:11:08.667 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78a73c44:::benchmark_data_smithi171_78918_object87241:head# 2024-04-20T17:11:08.667 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78a8fbca:::benchmark_data_smithi171_78918_object98065:head# 2024-04-20T17:11:08.667 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78a937eb:::benchmark_data_smithi171_78918_object97106:head# 2024-04-20T17:11:08.667 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78a972eb:::benchmark_data_smithi171_78918_object54785:head# 2024-04-20T17:11:08.668 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78a9ed53:::benchmark_data_smithi171_78918_object126224:head# 2024-04-20T17:11:08.668 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78aa1b31:::benchmark_data_smithi171_78918_object103526:head# 2024-04-20T17:11:08.668 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78acac3d:::benchmark_data_smithi171_78918_object129648:head# 2024-04-20T17:11:08.668 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78acd321:::benchmark_data_smithi171_78918_object103798:head# 2024-04-20T17:11:08.668 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78acf578:::benchmark_data_smithi171_78918_object119369:head# 2024-04-20T17:11:08.668 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78ad0a0b:::benchmark_data_smithi171_78918_object118704:head# 2024-04-20T17:11:08.668 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78ad1c1e:::benchmark_data_smithi171_78918_object70744:head# 2024-04-20T17:11:08.668 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78ae71bb:::benchmark_data_smithi171_78918_object89755:head# 2024-04-20T17:11:08.668 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78aeae65:::benchmark_data_smithi171_78918_object134516:head# 2024-04-20T17:11:08.668 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78af09f8:::benchmark_data_smithi171_78918_object130444:head# 2024-04-20T17:11:08.668 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78af953a:::benchmark_data_smithi171_78918_object71317:head# 2024-04-20T17:11:08.668 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78b07079:::benchmark_data_smithi171_78918_object78339:head# 2024-04-20T17:11:08.668 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78b10b3d:::benchmark_data_smithi171_78918_object122735:head# 2024-04-20T17:11:08.669 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78b155aa:::benchmark_data_smithi171_78918_object54970:head# 2024-04-20T17:11:08.669 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78b2127d:::benchmark_data_smithi171_78918_object82116:head# 2024-04-20T17:11:08.669 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78b2307e:::benchmark_data_smithi171_78918_object142428:head# 2024-04-20T17:11:08.669 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78b3964d:::benchmark_data_smithi171_78918_object61461:head# 2024-04-20T17:11:08.669 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78b4d98a:::benchmark_data_smithi171_78918_object149746:head# 2024-04-20T17:11:08.669 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78b580b8:::benchmark_data_smithi171_78918_object71458:head# 2024-04-20T17:11:08.669 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78b7605d:::benchmark_data_smithi171_78918_object119440:head# 2024-04-20T17:11:08.669 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78b7e789:::benchmark_data_smithi171_78918_object110967:head# 2024-04-20T17:11:08.669 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78b7f8fe:::benchmark_data_smithi171_78918_object133006:head# 2024-04-20T17:11:08.669 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78b83deb:::benchmark_data_smithi171_78918_object107759:head# 2024-04-20T17:11:08.669 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78b87530:::benchmark_data_smithi171_78918_object85609:head# 2024-04-20T17:11:08.669 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78b8c293:::benchmark_data_smithi171_78918_object155403:head# 2024-04-20T17:11:08.669 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78b8e3b9:::benchmark_data_smithi171_78918_object82564:head# 2024-04-20T17:11:08.669 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78b8ebd7:::benchmark_data_smithi171_78918_object95690:head# 2024-04-20T17:11:08.670 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78b965ba:::benchmark_data_smithi171_78918_object90567:head# 2024-04-20T17:11:08.670 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78b9c2de:::benchmark_data_smithi171_78918_object98840:head# 2024-04-20T17:11:08.670 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78b9d434:::benchmark_data_smithi171_78918_object57042:head# 2024-04-20T17:11:08.670 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78ba8037:::benchmark_data_smithi171_78918_object131330:head# 2024-04-20T17:11:08.670 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78bb05a2:::benchmark_data_smithi171_78918_object118635:head# 2024-04-20T17:11:08.670 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78bb49f0:::benchmark_data_smithi171_78918_object150602:head# 2024-04-20T17:11:08.670 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78bc5af3:::benchmark_data_smithi171_78918_object65767:head# 2024-04-20T17:11:08.670 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78bcfab0:::benchmark_data_smithi171_78918_object136439:head# 2024-04-20T17:11:08.670 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78bd61f7:::benchmark_data_smithi171_78918_object73896:head# 2024-04-20T17:11:08.670 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78bdb709:::benchmark_data_smithi171_78918_object135196:head# 2024-04-20T17:11:08.670 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78bdecff:::benchmark_data_smithi171_78918_object83405:head# 2024-04-20T17:11:08.670 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78be5deb:::benchmark_data_smithi171_78918_object131592:head# 2024-04-20T17:11:08.670 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78bed614:::benchmark_data_smithi171_78918_object100020:head# 2024-04-20T17:11:08.670 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78bf096a:::benchmark_data_smithi171_78918_object132702:head# 2024-04-20T17:11:08.671 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78bf5db7:::benchmark_data_smithi171_78918_object155293:head# 2024-04-20T17:11:08.671 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78c0439c:::benchmark_data_smithi171_78918_object59308:head# 2024-04-20T17:11:08.671 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78c117a5:::benchmark_data_smithi171_78918_object89940:head# 2024-04-20T17:11:08.671 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78c14794:::benchmark_data_smithi171_78918_object130485:head# 2024-04-20T17:11:08.671 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78c1b90b:::benchmark_data_smithi171_78918_object65258:head# 2024-04-20T17:11:08.671 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78c1bf92:::benchmark_data_smithi171_78918_object126952:head# 2024-04-20T17:11:08.671 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78c1ca18:::benchmark_data_smithi171_78918_object138743:head# 2024-04-20T17:11:08.671 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78c29ffa:::benchmark_data_smithi171_78918_object88164:head# 2024-04-20T17:11:08.671 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78c2b0c9:::benchmark_data_smithi171_78918_object144334:head# 2024-04-20T17:11:08.671 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78c338f9:::benchmark_data_smithi171_78918_object78767:head# 2024-04-20T17:11:08.671 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78c476f9:::benchmark_data_smithi171_78918_object107949:head# 2024-04-20T17:11:08.671 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78c51a36:::benchmark_data_smithi171_78918_object83829:head# 2024-04-20T17:11:08.671 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78c6157b:::benchmark_data_smithi171_78918_object150962:head# 2024-04-20T17:11:08.671 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78c65705:::benchmark_data_smithi171_78918_object130425:head# 2024-04-20T17:11:08.671 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78c88cfc:::benchmark_data_smithi171_78918_object131053:head# 2024-04-20T17:11:08.672 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78c95f27:::benchmark_data_smithi171_78918_object56860:head# 2024-04-20T17:11:08.672 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78c96f7e:::benchmark_data_smithi171_78918_object74649:head# 2024-04-20T17:11:08.672 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78c9a0cc:::benchmark_data_smithi171_78918_object56418:head# 2024-04-20T17:11:08.672 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78ca01ad:::benchmark_data_smithi171_78918_object129411:head# 2024-04-20T17:11:08.672 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78ca1f43:::benchmark_data_smithi171_78918_object77391:head# 2024-04-20T17:11:08.672 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78ca3e5e:::benchmark_data_smithi171_78918_object103011:head# 2024-04-20T17:11:08.672 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78ca64d7:::benchmark_data_smithi171_78918_object79278:head# 2024-04-20T17:11:08.672 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78cb5cbc:::benchmark_data_smithi171_78918_object127632:head# 2024-04-20T17:11:08.672 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78cc7ba4:::benchmark_data_smithi171_78918_object115830:head# 2024-04-20T17:11:08.672 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78cdf7bb:::benchmark_data_smithi171_78918_object156232:head# 2024-04-20T17:11:08.672 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78ce5e44:::benchmark_data_smithi171_78918_object102750:head# 2024-04-20T17:11:08.672 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78ce9f05:::benchmark_data_smithi171_78918_object156382:head# 2024-04-20T17:11:08.672 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78cf91c1:::benchmark_data_smithi171_78918_object116714:head# 2024-04-20T17:11:08.672 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78cf9410:::benchmark_data_smithi171_78918_object69248:head# 2024-04-20T17:11:08.672 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78cfff83:::benchmark_data_smithi171_78918_object111635:head# 2024-04-20T17:11:08.673 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78d04d25:::benchmark_data_smithi171_78918_object158152:head# 2024-04-20T17:11:08.673 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78d08b9b:::benchmark_data_smithi171_78918_object81238:head# 2024-04-20T17:11:08.673 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78d1c1ec:::benchmark_data_smithi171_78918_object80810:head# 2024-04-20T17:11:08.673 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78d1ca1b:::benchmark_data_smithi171_78918_object109825:head# 2024-04-20T17:11:08.673 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78d33cef:::benchmark_data_smithi171_78918_object152368:head# 2024-04-20T17:11:08.673 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78d4507c:::benchmark_data_smithi171_78918_object152292:head# 2024-04-20T17:11:08.673 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78d52406:::benchmark_data_smithi171_78918_object118999:head# 2024-04-20T17:11:08.673 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78d64ada:::benchmark_data_smithi171_78918_object94486:head# 2024-04-20T17:11:08.673 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78d6c9be:::benchmark_data_smithi171_78918_object80801:head# 2024-04-20T17:11:08.673 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78d6d23b:::benchmark_data_smithi171_78918_object154601:head# 2024-04-20T17:11:08.673 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78d831d6:::benchmark_data_smithi171_78918_object86679:head# 2024-04-20T17:11:08.673 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78d832d2:::benchmark_data_smithi171_78918_object67865:head# 2024-04-20T17:11:08.673 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78d879c4:::benchmark_data_smithi171_78918_object114213:head# 2024-04-20T17:11:08.673 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78d89ced:::benchmark_data_smithi171_78918_object117689:head# 2024-04-20T17:11:08.674 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78d95905:::benchmark_data_smithi171_78918_object97733:head# 2024-04-20T17:11:08.674 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78d9b454:::benchmark_data_smithi171_78918_object122915:head# 2024-04-20T17:11:08.674 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78db5864:::benchmark_data_smithi171_78918_object65553:head# 2024-04-20T17:11:08.674 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78db9ad8:::benchmark_data_smithi171_78918_object85920:head# 2024-04-20T17:11:08.674 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78dbe1b3:::benchmark_data_smithi171_78918_object56307:head# 2024-04-20T17:11:08.674 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78dc36f3:::benchmark_data_smithi171_78918_object93349:head# 2024-04-20T17:11:08.674 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78dc77a7:::benchmark_data_smithi171_78918_object123228:head# 2024-04-20T17:11:08.674 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78dd8843:::benchmark_data_smithi171_78918_object126262:head# 2024-04-20T17:11:08.674 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78ddc6ea:::benchmark_data_smithi171_78918_object82892:head# 2024-04-20T17:11:08.674 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78ddec80:::benchmark_data_smithi171_78918_object94122:head# 2024-04-20T17:11:08.674 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78de0189:::benchmark_data_smithi171_78918_object157522:head# 2024-04-20T17:11:08.674 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78de35e2:::benchmark_data_smithi171_78918_object51753:head# 2024-04-20T17:11:08.674 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78de65dd:::benchmark_data_smithi171_78918_object129046:head# 2024-04-20T17:11:08.674 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78e0297a:::benchmark_data_smithi171_78918_object100041:head# 2024-04-20T17:11:08.675 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78e0665f:::benchmark_data_smithi171_78918_object77263:head# 2024-04-20T17:11:08.675 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78e085dd:::benchmark_data_smithi171_78918_object134927:head# 2024-04-20T17:11:08.675 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78e0b1a6:::benchmark_data_smithi171_78918_object62643:head# 2024-04-20T17:11:08.675 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78e16e8a:::benchmark_data_smithi171_78918_object97216:head# 2024-04-20T17:11:08.675 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78e3a6b2:::benchmark_data_smithi171_78918_object108807:head# 2024-04-20T17:11:08.675 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78e4b17c:::benchmark_data_smithi171_78918_object101579:head# 2024-04-20T17:11:08.675 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78e5019c:::benchmark_data_smithi171_78918_object72603:head# 2024-04-20T17:11:08.675 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78e5558b:::benchmark_data_smithi171_78918_object77493:head# 2024-04-20T17:11:08.675 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78e5e450:::benchmark_data_smithi171_78918_object73882:head# 2024-04-20T17:11:08.675 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78e5e9d4:::benchmark_data_smithi171_78918_object96304:head# 2024-04-20T17:11:08.675 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78e67bf6:::benchmark_data_smithi171_78918_object128227:head# 2024-04-20T17:11:08.675 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78e67c6e:::benchmark_data_smithi171_78918_object74032:head# 2024-04-20T17:11:08.675 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78e67e0d:::benchmark_data_smithi171_78918_object151901:head# 2024-04-20T17:11:08.675 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78e68bc9:::benchmark_data_smithi171_78918_object84524:head# 2024-04-20T17:11:08.676 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78e78203:::benchmark_data_smithi171_78918_object52573:head# 2024-04-20T17:11:08.676 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78e7af3e:::benchmark_data_smithi171_78918_object54977:head# 2024-04-20T17:11:08.676 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78e81195:::benchmark_data_smithi171_78918_object86823:head# 2024-04-20T17:11:08.676 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78e8261d:::benchmark_data_smithi171_78918_object59357:head# 2024-04-20T17:11:08.676 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78e84b41:::benchmark_data_smithi171_78918_object89464:head# 2024-04-20T17:11:08.676 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78e852d8:::benchmark_data_smithi171_78918_object133426:head# 2024-04-20T17:11:08.676 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78e8b28b:::benchmark_data_smithi171_78918_object100741:head# 2024-04-20T17:11:08.676 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78e8e8a4:::benchmark_data_smithi171_78918_object91138:head# 2024-04-20T17:11:08.676 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78e8fd8e:::benchmark_data_smithi171_78918_object97392:head# 2024-04-20T17:11:08.676 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78e91915:::benchmark_data_smithi171_78918_object132849:head# 2024-04-20T17:11:08.676 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78e9c86d:::benchmark_data_smithi171_78918_object140116:head# 2024-04-20T17:11:08.676 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78ebc26c:::benchmark_data_smithi171_78918_object94046:head# 2024-04-20T17:11:08.676 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78ebd482:::benchmark_data_smithi171_78918_object129833:head# 2024-04-20T17:11:08.676 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78ee1b6e:::benchmark_data_smithi171_78918_object121595:head# 2024-04-20T17:11:08.676 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78eec418:::benchmark_data_smithi171_78918_object159634:head# 2024-04-20T17:11:08.677 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78efb1e3:::benchmark_data_smithi171_78918_object51670:head# 2024-04-20T17:11:08.677 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78efda83:::benchmark_data_smithi171_78918_object69501:head# 2024-04-20T17:11:08.677 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78f04015:::benchmark_data_smithi171_78918_object105887:head# 2024-04-20T17:11:08.677 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:11:08.691 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78f05d75:::benchmark_data_smithi171_78918_object96150:head# 2024-04-20T17:11:08.691 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78f0a099:::benchmark_data_smithi171_78918_object70070:head# 2024-04-20T17:11:08.691 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78f0f67a:::benchmark_data_smithi171_78918_object101899:head# 2024-04-20T17:11:08.691 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78f1a002:::benchmark_data_smithi171_78918_object159261:head# 2024-04-20T17:11:08.691 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78f20e57:::benchmark_data_smithi171_78918_object159689:head# 2024-04-20T17:11:08.692 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78f210b0:::benchmark_data_smithi171_78918_object110642:head# 2024-04-20T17:11:08.692 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78f2f50d:::benchmark_data_smithi171_78918_object139723:head# 2024-04-20T17:11:08.692 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78f3c171:::benchmark_data_smithi171_78918_object87697:head# 2024-04-20T17:11:08.692 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78f43fa2:::benchmark_data_smithi171_78918_object155692:head# 2024-04-20T17:11:08.692 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78f47cd0:::benchmark_data_smithi171_78918_object119124:head# 2024-04-20T17:11:08.692 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78f4a6ae:::benchmark_data_smithi171_78918_object82392:head# 2024-04-20T17:11:08.692 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78f4e5f7:::benchmark_data_smithi171_78918_object121421:head# 2024-04-20T17:11:08.692 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78f56d93:::benchmark_data_smithi171_78918_object80795:head# 2024-04-20T17:11:08.692 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78f570b7:::benchmark_data_smithi171_78918_object86998:head# 2024-04-20T17:11:08.692 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78f5878b:::benchmark_data_smithi171_78918_object147173:head# 2024-04-20T17:11:08.692 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78f7ad2b:::benchmark_data_smithi171_78918_object137551:head# 2024-04-20T17:11:08.692 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78f8c939:::benchmark_data_smithi171_78918_object52189:head# 2024-04-20T17:11:08.692 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78fa98ef:::benchmark_data_smithi171_78918_object146813:head# 2024-04-20T17:11:08.692 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78fb1c5f:::benchmark_data_smithi171_78918_object74988:head# 2024-04-20T17:11:08.693 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78fb2a87:::benchmark_data_smithi171_78918_object117626:head# 2024-04-20T17:11:08.693 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78fc347c:::benchmark_data_smithi171_78918_object81890:head# 2024-04-20T17:11:08.693 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78fcea13:::benchmark_data_smithi171_78918_object137813:head# 2024-04-20T17:11:08.693 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78fceb38:::benchmark_data_smithi171_78918_object101464:head# 2024-04-20T17:11:08.693 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78fde233:::benchmark_data_smithi171_78918_object119589:head# 2024-04-20T17:11:08.693 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78fec9c7:::benchmark_data_smithi171_78918_object54225:head# 2024-04-20T17:11:08.693 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78ff0dc2:::benchmark_data_smithi171_78918_object139308:head# 2024-04-20T17:11:08.693 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78ff244e:::benchmark_data_smithi171_78918_object81821:head# 2024-04-20T17:11:08.693 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:78ff509c:::benchmark_data_smithi171_78918_object60647:head# 2024-04-20T17:11:08.693 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:790058f3:::benchmark_data_smithi171_78918_object156491:head# 2024-04-20T17:11:08.693 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79009e57:::benchmark_data_smithi171_78918_object102911:head# 2024-04-20T17:11:08.693 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7900ab4e:::benchmark_data_smithi171_78918_object65764:head# 2024-04-20T17:11:08.693 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:790169ef:::benchmark_data_smithi171_78918_object95397:head# 2024-04-20T17:11:08.693 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79018034:::benchmark_data_smithi171_78918_object98758:head# 2024-04-20T17:11:08.694 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79026ce0:::benchmark_data_smithi171_78918_object92999:head# 2024-04-20T17:11:08.694 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7902add7:::benchmark_data_smithi171_78918_object65346:head# 2024-04-20T17:11:08.694 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79030dfd:::benchmark_data_smithi171_78918_object99107:head# 2024-04-20T17:11:08.694 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:790563d0:::benchmark_data_smithi171_78918_object138527:head# 2024-04-20T17:11:08.694 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7905af6e:::benchmark_data_smithi171_78918_object148856:head# 2024-04-20T17:11:08.694 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7905b259:::benchmark_data_smithi171_78918_object158217:head# 2024-04-20T17:11:08.694 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7905ba76:::benchmark_data_smithi171_78918_object69902:head# 2024-04-20T17:11:08.694 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79060ee0:::benchmark_data_smithi171_78918_object124964:head# 2024-04-20T17:11:08.694 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7906384f:::benchmark_data_smithi171_78918_object87496:head# 2024-04-20T17:11:08.694 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79064dd8:::benchmark_data_smithi171_78918_object70030:head# 2024-04-20T17:11:08.694 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7906cbab:::benchmark_data_smithi171_78918_object97103:head# 2024-04-20T17:11:08.694 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79079987:::benchmark_data_smithi171_78918_object63096:head# 2024-04-20T17:11:08.694 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7908d39f:::benchmark_data_smithi171_78918_object93059:head# 2024-04-20T17:11:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:790915e5:::benchmark_data_smithi171_78918_object87093:head# 2024-04-20T17:11:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:790a4775:::benchmark_data_smithi171_78918_object127726:head# 2024-04-20T17:11:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:790a59d6:::benchmark_data_smithi171_78918_object96782:head# 2024-04-20T17:11:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:790b5aec:::benchmark_data_smithi171_78918_object103764:head# 2024-04-20T17:11:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:790bb1b2:::benchmark_data_smithi171_78918_object122824:head# 2024-04-20T17:11:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:790bd32a:::benchmark_data_smithi171_78918_object57611:head# 2024-04-20T17:11:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:790c8153:::benchmark_data_smithi171_78918_object137110:head# 2024-04-20T17:11:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:790ce320:::benchmark_data_smithi171_78918_object127434:head# 2024-04-20T17:11:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:790d9219:::benchmark_data_smithi171_78918_object68017:head# 2024-04-20T17:11:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:790dd235:::benchmark_data_smithi171_78918_object99026:head# 2024-04-20T17:11:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:790e2b7f:::benchmark_data_smithi171_78918_object61018:head# 2024-04-20T17:11:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:790e5476:::benchmark_data_smithi171_78918_object107805:head# 2024-04-20T17:11:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:790e8067:::benchmark_data_smithi171_78918_object68783:head# 2024-04-20T17:11:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:790edff0:::benchmark_data_smithi171_78918_object115253:head# 2024-04-20T17:11:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:790fd5c8:::benchmark_data_smithi171_78918_object119324:head# 2024-04-20T17:11:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79109dc8:::benchmark_data_smithi171_78918_object153079:head# 2024-04-20T17:11:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79117b40:::benchmark_data_smithi171_78918_object117109:head# 2024-04-20T17:11:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79130eed:::benchmark_data_smithi171_78918_object64131:head# 2024-04-20T17:11:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7913230c:::benchmark_data_smithi171_78918_object116363:head# 2024-04-20T17:11:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7913aff1:::benchmark_data_smithi171_78918_object73583:head# 2024-04-20T17:11:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:791404f1:::benchmark_data_smithi171_78918_object80379:head# 2024-04-20T17:11:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79142af0:::benchmark_data_smithi171_78918_object137993:head# 2024-04-20T17:11:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7914d58c:::benchmark_data_smithi171_78918_object78684:head# 2024-04-20T17:11:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7915f1bc:::benchmark_data_smithi171_78918_object109164:head# 2024-04-20T17:11:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7915faf9:::benchmark_data_smithi171_78918_object146712:head# 2024-04-20T17:11:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79177595:::benchmark_data_smithi171_78918_object93684:head# 2024-04-20T17:11:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7917af5e:::benchmark_data_smithi171_78918_object107758:head# 2024-04-20T17:11:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7917ee3d:::benchmark_data_smithi171_78918_object135192:head# 2024-04-20T17:11:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7917fccf:::benchmark_data_smithi171_78918_object102367:head# 2024-04-20T17:11:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79186c02:::benchmark_data_smithi171_78918_object115970:head# 2024-04-20T17:11:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7918b4ad:::benchmark_data_smithi171_78918_object110007:head# 2024-04-20T17:11:08.697 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7918fc0f:::benchmark_data_smithi171_78918_object56807:head# 2024-04-20T17:11:08.697 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79195039:::benchmark_data_smithi171_78918_object130614:head# 2024-04-20T17:11:08.697 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7919a303:::benchmark_data_smithi171_78918_object118507:head# 2024-04-20T17:11:08.697 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:791b3674:::benchmark_data_smithi171_78918_object81265:head# 2024-04-20T17:11:08.697 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:791bcf16:::benchmark_data_smithi171_78918_object113206:head# 2024-04-20T17:11:08.697 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:791be73f:::benchmark_data_smithi171_78918_object138809:head# 2024-04-20T17:11:08.697 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:791c849b:::benchmark_data_smithi171_78918_object61216:head# 2024-04-20T17:11:08.697 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:791dee3e:::benchmark_data_smithi171_78918_object112255:head# 2024-04-20T17:11:08.697 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:791e4a56:::benchmark_data_smithi171_78918_object97314:head# 2024-04-20T17:11:08.697 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79208c37:::benchmark_data_smithi171_78918_object145367:head# 2024-04-20T17:11:08.697 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7920f948:::benchmark_data_smithi171_78918_object64785:head# 2024-04-20T17:11:08.697 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79211cb3:::benchmark_data_smithi171_78918_object142322:head# 2024-04-20T17:11:08.697 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:792168a6:::benchmark_data_smithi171_78918_object63671:head# 2024-04-20T17:11:08.697 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79217b5d:::benchmark_data_smithi171_78918_object97098:head# 2024-04-20T17:11:08.697 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7921b5d3:::benchmark_data_smithi171_78918_object103207:head# 2024-04-20T17:11:08.698 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7921c9f4:::benchmark_data_smithi171_78918_object151811:head# 2024-04-20T17:11:08.698 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7921dfc1:::benchmark_data_smithi171_78918_object128606:head# 2024-04-20T17:11:08.698 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79228808:::benchmark_data_smithi171_78918_object75349:head# 2024-04-20T17:11:08.698 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79229c1e:::benchmark_data_smithi171_78918_object159926:head# 2024-04-20T17:11:08.698 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7922b7fb:::benchmark_data_smithi171_78918_object119885:head# 2024-04-20T17:11:08.698 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:792376bb:::benchmark_data_smithi171_78918_object52693:head# 2024-04-20T17:11:08.698 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79239c1b:::benchmark_data_smithi171_78918_object115061:head# 2024-04-20T17:11:08.698 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7924060e:::benchmark_data_smithi171_78918_object91631:head# 2024-04-20T17:11:08.698 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79243e78:::benchmark_data_smithi171_78918_object134947:head# 2024-04-20T17:11:08.698 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79246921:::benchmark_data_smithi171_78918_object65022:head# 2024-04-20T17:11:08.698 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:792490a0:::benchmark_data_smithi171_78918_object155927:head# 2024-04-20T17:11:08.698 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79257fd8:::benchmark_data_smithi171_78918_object131898:head# 2024-04-20T17:11:08.698 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:792633ac:::benchmark_data_smithi171_78918_object100535:head# 2024-04-20T17:11:08.699 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7926a093:::benchmark_data_smithi171_78918_object117704:head# 2024-04-20T17:11:08.699 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:792b0f18:::benchmark_data_smithi171_78918_object86215:head# 2024-04-20T17:11:08.699 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:792b2dc5:::benchmark_data_smithi171_78918_object86186:head# 2024-04-20T17:11:08.699 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:792b80a2:::benchmark_data_smithi171_78918_object148603:head# 2024-04-20T17:11:08.699 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:792bee67:::benchmark_data_smithi171_78918_object56953:head# 2024-04-20T17:11:08.699 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:792c69f6:::benchmark_data_smithi171_78918_object78250:head# 2024-04-20T17:11:08.699 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:792c7b07:::benchmark_data_smithi171_78918_object77884:head# 2024-04-20T17:11:08.699 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:792ca4d5:::benchmark_data_smithi171_78918_object62889:head# 2024-04-20T17:11:08.699 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:792df878:::benchmark_data_smithi171_78918_object79134:head# 2024-04-20T17:11:08.699 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:792ec530:::benchmark_data_smithi171_78918_object154033:head# 2024-04-20T17:11:08.699 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:792f1cbb:::benchmark_data_smithi171_78918_object82460:head# 2024-04-20T17:11:08.699 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:792fc817:::benchmark_data_smithi171_78918_object54139:head# 2024-04-20T17:11:08.699 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79306783:::benchmark_data_smithi171_78918_object150541:head# 2024-04-20T17:11:08.699 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7930a0d5:::benchmark_data_smithi171_78918_object121209:head# 2024-04-20T17:11:08.699 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:793190f1:::benchmark_data_smithi171_78918_object56035:head# 2024-04-20T17:11:08.699 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7931bbde:::benchmark_data_smithi171_78918_object134875:head# 2024-04-20T17:11:08.700 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79329ff3:::benchmark_data_smithi171_78918_object126072:head# 2024-04-20T17:11:08.700 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79336cf4:::benchmark_data_smithi171_78918_object55387:head# 2024-04-20T17:11:08.700 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79337f70:::benchmark_data_smithi171_78918_object63110:head# 2024-04-20T17:11:08.700 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7937a16c:::benchmark_data_smithi171_78918_object65264:head# 2024-04-20T17:11:08.700 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7938559f:::benchmark_data_smithi171_78918_object53148:head# 2024-04-20T17:11:08.700 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79395706:::benchmark_data_smithi171_78918_object117827:head# 2024-04-20T17:11:08.700 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:793a5ace:::benchmark_data_smithi171_78918_object150966:head# 2024-04-20T17:11:08.700 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:793ac66c:::benchmark_data_smithi171_78918_object67960:head# 2024-04-20T17:11:08.700 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:793bae04:::benchmark_data_smithi171_78918_object107403:head# 2024-04-20T17:11:08.700 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:793bbaf8:::benchmark_data_smithi171_78918_object158433:head# 2024-04-20T17:11:08.700 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:793d8c54:::benchmark_data_smithi171_78918_object159241:head# 2024-04-20T17:11:08.700 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:793d93a3:::benchmark_data_smithi171_78918_object86091:head# 2024-04-20T17:11:08.700 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:793d9e04:::benchmark_data_smithi171_78918_object121645:head# 2024-04-20T17:11:08.701 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:793dd06f:::benchmark_data_smithi171_78918_object89835:head# 2024-04-20T17:11:08.701 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:793edb95:::benchmark_data_smithi171_78918_object116215:head# 2024-04-20T17:11:08.701 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:793f0f1d:::benchmark_data_smithi171_78918_object141544:head# 2024-04-20T17:11:08.701 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:793f6aa2:::benchmark_data_smithi171_78918_object111937:head# 2024-04-20T17:11:08.701 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:793f9092:::benchmark_data_smithi171_78918_object125322:head# 2024-04-20T17:11:08.701 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:794139c1:::benchmark_data_smithi171_78918_object63812:head# 2024-04-20T17:11:08.701 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7941d33b:::benchmark_data_smithi171_78918_object118010:head# 2024-04-20T17:11:08.701 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79428db6:::benchmark_data_smithi171_78918_object103185:head# 2024-04-20T17:11:08.701 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79428ffe:::benchmark_data_smithi171_78918_object147637:head# 2024-04-20T17:11:08.701 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79429430:::benchmark_data_smithi171_78918_object131357:head# 2024-04-20T17:11:08.701 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7942e338:::benchmark_data_smithi171_78918_object55003:head# 2024-04-20T17:11:08.701 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79436490:::benchmark_data_smithi171_78918_object69230:head# 2024-04-20T17:11:08.701 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79436bfe:::benchmark_data_smithi171_78918_object146305:head# 2024-04-20T17:11:08.701 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7944a05e:::benchmark_data_smithi171_78918_object138673:head# 2024-04-20T17:11:08.702 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:11:08.723 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79451a46:::benchmark_data_smithi171_78918_object104799:head# 2024-04-20T17:11:08.724 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7945cb47:::benchmark_data_smithi171_78918_object144321:head# 2024-04-20T17:11:08.724 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7945d66a:::benchmark_data_smithi171_78918_object133096:head# 2024-04-20T17:11:08.724 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7945fbfe:::benchmark_data_smithi171_78918_object83495:head# 2024-04-20T17:11:08.724 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:794629fc:::benchmark_data_smithi171_78918_object89184:head# 2024-04-20T17:11:08.724 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:794947f0:::benchmark_data_smithi171_78918_object145592:head# 2024-04-20T17:11:08.724 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79495fbd:::benchmark_data_smithi171_78918_object93032:head# 2024-04-20T17:11:08.724 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7949a836:::benchmark_data_smithi171_78918_object62242:head# 2024-04-20T17:11:08.724 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:794ac0a1:::benchmark_data_smithi171_78918_object84030:head# 2024-04-20T17:11:08.724 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:794ad50f:::benchmark_data_smithi171_78918_object125713:head# 2024-04-20T17:11:08.724 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:794b95e1:::benchmark_data_smithi171_78918_object72347:head# 2024-04-20T17:11:08.724 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:794c22c5:::benchmark_data_smithi171_78918_object73956:head# 2024-04-20T17:11:08.724 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:794c3740:::benchmark_data_smithi171_78918_object91431:head# 2024-04-20T17:11:08.724 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:794c800f:::benchmark_data_smithi171_78918_object55600:head# 2024-04-20T17:11:08.724 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:794d3caa:::benchmark_data_smithi171_78918_object90402:head# 2024-04-20T17:11:08.725 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:794e920f:::benchmark_data_smithi171_78918_object96793:head# 2024-04-20T17:11:08.725 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:794ef673:::benchmark_data_smithi171_78918_object141492:head# 2024-04-20T17:11:08.725 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79501dd6:::benchmark_data_smithi171_78918_object65752:head# 2024-04-20T17:11:08.725 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:795031ee:::benchmark_data_smithi171_78918_object148929:head# 2024-04-20T17:11:08.725 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7950a031:::benchmark_data_smithi171_78918_object152649:head# 2024-04-20T17:11:08.725 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7950d587:::benchmark_data_smithi171_78918_object110358:head# 2024-04-20T17:11:08.725 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7950e4ab:::benchmark_data_smithi171_78918_object66977:head# 2024-04-20T17:11:08.725 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7951801a:::benchmark_data_smithi171_78918_object102154:head# 2024-04-20T17:11:08.725 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79522a53:::benchmark_data_smithi171_78918_object85426:head# 2024-04-20T17:11:08.725 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:795249f8:::benchmark_data_smithi171_78918_object68635:head# 2024-04-20T17:11:08.725 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7952ae9f:::benchmark_data_smithi171_78918_object100080:head# 2024-04-20T17:11:08.725 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7952b387:::benchmark_data_smithi171_78918_object132741:head# 2024-04-20T17:11:08.725 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7952e035:::benchmark_data_smithi171_78918_object74704:head# 2024-04-20T17:11:08.725 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79531121:::benchmark_data_smithi171_78918_object98826:head# 2024-04-20T17:11:08.726 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:795379ce:::benchmark_data_smithi171_78918_object105068:head# 2024-04-20T17:11:08.726 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7953e126:::benchmark_data_smithi171_78918_object160821:head# 2024-04-20T17:11:08.726 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79545a0f:::benchmark_data_smithi171_78918_object153127:head# 2024-04-20T17:11:08.726 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79545cdf:::benchmark_data_smithi171_78918_object132368:head# 2024-04-20T17:11:08.726 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7954e890:::benchmark_data_smithi171_78918_object99028:head# 2024-04-20T17:11:08.726 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79554997:::benchmark_data_smithi171_78918_object156098:head# 2024-04-20T17:11:08.726 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7955b17b:::benchmark_data_smithi171_78918_object79805:head# 2024-04-20T17:11:08.726 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79561c65:::benchmark_data_smithi171_78918_object108009:head# 2024-04-20T17:11:08.726 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79563338:::benchmark_data_smithi171_78918_object63721:head# 2024-04-20T17:11:08.726 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7956a4ac:::benchmark_data_smithi171_78918_object75459:head# 2024-04-20T17:11:08.726 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79570cdb:::benchmark_data_smithi171_78918_object153443:head# 2024-04-20T17:11:08.726 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:795755e6:::benchmark_data_smithi171_78918_object76651:head# 2024-04-20T17:11:08.726 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7957db6e:::benchmark_data_smithi171_78918_object109312:head# 2024-04-20T17:11:08.726 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79587101:::benchmark_data_smithi171_78918_object107428:head# 2024-04-20T17:11:08.727 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79593b3e:::benchmark_data_smithi171_78918_object139243:head# 2024-04-20T17:11:08.727 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79599922:::benchmark_data_smithi171_78918_object125289:head# 2024-04-20T17:11:08.727 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7959ac5f:::benchmark_data_smithi171_78918_object71794:head# 2024-04-20T17:11:08.727 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:795a7066:::benchmark_data_smithi171_78918_object80554:head# 2024-04-20T17:11:08.727 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:795ad9b4:::benchmark_data_smithi171_78918_object68890:head# 2024-04-20T17:11:08.727 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:795b2c0c:::benchmark_data_smithi171_78918_object65473:head# 2024-04-20T17:11:08.727 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:795b67ef:::benchmark_data_smithi171_78918_object142553:head# 2024-04-20T17:11:08.727 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:795c31dd:::benchmark_data_smithi171_78918_object94198:head# 2024-04-20T17:11:08.727 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:795c8791:::benchmark_data_smithi171_78918_object69692:head# 2024-04-20T17:11:08.727 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:795d3662:::benchmark_data_smithi171_78918_object117693:head# 2024-04-20T17:11:08.727 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:795d648d:::benchmark_data_smithi171_78918_object66485:head# 2024-04-20T17:11:08.727 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:795dbdbe:::benchmark_data_smithi171_78918_object104875:head# 2024-04-20T17:11:08.727 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:795df80b:::benchmark_data_smithi171_78918_object55616:head# 2024-04-20T17:11:08.727 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:795e1191:::benchmark_data_smithi171_78918_object154740:head# 2024-04-20T17:11:08.728 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:795fba15:::benchmark_data_smithi171_78918_object127877:head# 2024-04-20T17:11:08.728 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7960448e:::benchmark_data_smithi171_78918_object110468:head# 2024-04-20T17:11:08.728 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79614fc1:::benchmark_data_smithi171_78918_object52477:head# 2024-04-20T17:11:08.728 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7961a32d:::benchmark_data_smithi171_78918_object94467:head# 2024-04-20T17:11:08.728 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7961ea3b:::benchmark_data_smithi171_78918_object120289:head# 2024-04-20T17:11:08.728 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7962e59d:::benchmark_data_smithi171_78918_object152854:head# 2024-04-20T17:11:08.728 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7963073e:::benchmark_data_smithi171_78918_object81723:head# 2024-04-20T17:11:08.728 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:796378a5:::benchmark_data_smithi171_78918_object125126:head# 2024-04-20T17:11:08.728 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7964da80:::benchmark_data_smithi171_78918_object113796:head# 2024-04-20T17:11:08.728 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79663673:::benchmark_data_smithi171_78918_object129970:head# 2024-04-20T17:11:08.728 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79664b06:::benchmark_data_smithi171_78918_object133427:head# 2024-04-20T17:11:08.728 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79671ce8:::benchmark_data_smithi171_78918_object137180:head# 2024-04-20T17:11:08.728 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79675072:::benchmark_data_smithi171_78918_object74075:head# 2024-04-20T17:11:08.728 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7967b604:::benchmark_data_smithi171_78918_object85873:head# 2024-04-20T17:11:08.729 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7968a9d4:::benchmark_data_smithi171_78918_object121399:head# 2024-04-20T17:11:08.729 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7968bd9d:::benchmark_data_smithi171_78918_object102571:head# 2024-04-20T17:11:08.729 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7968c896:::benchmark_data_smithi171_78918_object75262:head# 2024-04-20T17:11:08.729 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:796ab515:::benchmark_data_smithi171_78918_object77567:head# 2024-04-20T17:11:08.729 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:796afc7c:::benchmark_data_smithi171_78918_object148737:head# 2024-04-20T17:11:08.729 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:796b28ef:::benchmark_data_smithi171_78918_object126655:head# 2024-04-20T17:11:08.729 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:796b2aac:::benchmark_data_smithi171_78918_object91898:head# 2024-04-20T17:11:08.729 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:796c1e73:::benchmark_data_smithi171_78918_object147436:head# 2024-04-20T17:11:08.729 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:796c6da3:::benchmark_data_smithi171_78918_object121004:head# 2024-04-20T17:11:08.729 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:796cc161:::benchmark_data_smithi171_78918_object153437:head# 2024-04-20T17:11:08.729 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:796cd779:::benchmark_data_smithi171_78918_object83169:head# 2024-04-20T17:11:08.729 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:796f9fc3:::benchmark_data_smithi171_78918_object68251:head# 2024-04-20T17:11:08.729 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:796ff920:::benchmark_data_smithi171_78918_object147769:head# 2024-04-20T17:11:08.729 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79702930:::benchmark_data_smithi171_78918_object140302:head# 2024-04-20T17:11:08.729 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7970537b:::benchmark_data_smithi171_78918_object97116:head# 2024-04-20T17:11:08.729 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79708e00:::benchmark_data_smithi171_78918_object72481:head# 2024-04-20T17:11:08.730 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7970affc:::benchmark_data_smithi171_78918_object68212:head# 2024-04-20T17:11:08.730 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7970b0ed:::benchmark_data_smithi171_78918_object55038:head# 2024-04-20T17:11:08.730 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7970cd43:::benchmark_data_smithi171_78918_object57267:head# 2024-04-20T17:11:08.730 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7970d8eb:::benchmark_data_smithi171_78918_object59211:head# 2024-04-20T17:11:08.730 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:797121ba:::benchmark_data_smithi171_78918_object132391:head# 2024-04-20T17:11:08.730 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:797216f0:::benchmark_data_smithi171_78918_object125052:head# 2024-04-20T17:11:08.730 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7972de0c:::benchmark_data_smithi171_78918_object122153:head# 2024-04-20T17:11:08.730 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79737203:::benchmark_data_smithi171_78918_object152105:head# 2024-04-20T17:11:08.730 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:797456f3:::benchmark_data_smithi171_78918_object59000:head# 2024-04-20T17:11:08.730 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7976bd47:::benchmark_data_smithi171_78918_object63745:head# 2024-04-20T17:11:08.730 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7977075d:::benchmark_data_smithi171_78918_object63076:head# 2024-04-20T17:11:08.730 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79775d2c:::benchmark_data_smithi171_78918_object78552:head# 2024-04-20T17:11:08.730 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:797762f2:::benchmark_data_smithi171_78918_object56568:head# 2024-04-20T17:11:08.730 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79777a11:::benchmark_data_smithi171_78918_object84696:head# 2024-04-20T17:11:08.731 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7977ba49:::benchmark_data_smithi171_78918_object158018:head# 2024-04-20T17:11:08.731 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7977fae8:::benchmark_data_smithi171_78918_object87299:head# 2024-04-20T17:11:08.731 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79784d0d:::benchmark_data_smithi171_78918_object58625:head# 2024-04-20T17:11:08.731 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7979d19f:::benchmark_data_smithi171_78918_object123738:head# 2024-04-20T17:11:08.731 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7979f6ea:::benchmark_data_smithi171_78918_object92665:head# 2024-04-20T17:11:08.731 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:797a3e34:::benchmark_data_smithi171_78918_object71972:head# 2024-04-20T17:11:08.731 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:797b5ec2:::benchmark_data_smithi171_78918_object115224:head# 2024-04-20T17:11:08.731 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:797bba50:::benchmark_data_smithi171_78918_object81657:head# 2024-04-20T17:11:08.731 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:797bd2cb:::benchmark_data_smithi171_78918_object90275:head# 2024-04-20T17:11:08.731 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:797c2d1d:::benchmark_data_smithi171_78918_object99858:head# 2024-04-20T17:11:08.731 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:797c7366:::benchmark_data_smithi171_78918_object136904:head# 2024-04-20T17:11:08.731 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:797d23a6:::benchmark_data_smithi171_78918_object151182:head# 2024-04-20T17:11:08.731 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:797d42a8:::benchmark_data_smithi171_78918_object86500:head# 2024-04-20T17:11:08.731 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:797d5bb5:::benchmark_data_smithi171_78918_object66036:head# 2024-04-20T17:11:08.732 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:797df35d:::benchmark_data_smithi171_78918_object126536:head# 2024-04-20T17:11:08.732 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:797e3e43:::benchmark_data_smithi171_78918_object87155:head# 2024-04-20T17:11:08.732 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:797e5764:::benchmark_data_smithi171_78918_object84833:head# 2024-04-20T17:11:08.732 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:797f4b9e:::benchmark_data_smithi171_78918_object160890:head# 2024-04-20T17:11:08.732 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7980500e:::benchmark_data_smithi171_78918_object121751:head# 2024-04-20T17:11:08.732 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7980f935:::benchmark_data_smithi171_78918_object60350:head# 2024-04-20T17:11:08.732 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79829d84:::benchmark_data_smithi171_78918_object63297:head# 2024-04-20T17:11:08.732 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79833cee:::benchmark_data_smithi171_78918_object133374:head# 2024-04-20T17:11:08.732 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79835c1a:::benchmark_data_smithi171_78918_object70435:head# 2024-04-20T17:11:08.732 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79845a25:::benchmark_data_smithi171_78918_object62082:head# 2024-04-20T17:11:08.732 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7984c6ff:::benchmark_data_smithi171_78918_object75377:head# 2024-04-20T17:11:08.732 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7984ccd2:::benchmark_data_smithi171_78918_object123059:head# 2024-04-20T17:11:08.732 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:798508e8:::benchmark_data_smithi171_78918_object130722:head# 2024-04-20T17:11:08.732 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7986be57:::benchmark_data_smithi171_78918_object151883:head# 2024-04-20T17:11:08.732 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79870bd7:::benchmark_data_smithi171_78918_object111333:head# 2024-04-20T17:11:08.733 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79873f58:::benchmark_data_smithi171_78918_object66909:head# 2024-04-20T17:11:08.733 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7987cea2:::benchmark_data_smithi171_78918_object95576:head# 2024-04-20T17:11:08.733 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7988e54f:::benchmark_data_smithi171_78918_object157547:head# 2024-04-20T17:11:08.733 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7988fd9c:::benchmark_data_smithi171_78918_object151212:head# 2024-04-20T17:11:08.733 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:798914fd:::benchmark_data_smithi171_78918_object56479:head# 2024-04-20T17:11:08.733 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79899530:::benchmark_data_smithi171_78918_object116471:head# 2024-04-20T17:11:08.733 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7989e2b8:::benchmark_data_smithi171_78918_object122168:head# 2024-04-20T17:11:08.733 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:798a48ea:::benchmark_data_smithi171_78918_object81686:head# 2024-04-20T17:11:08.733 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:798c2721:::benchmark_data_smithi171_78918_object138318:head# 2024-04-20T17:11:08.733 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:798c5ae7:::benchmark_data_smithi171_78918_object105056:head# 2024-04-20T17:11:08.733 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:798d06de:::benchmark_data_smithi171_78918_object105492:head# 2024-04-20T17:11:08.733 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:798d30e5:::benchmark_data_smithi171_78918_object141738:head# 2024-04-20T17:11:08.733 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:798ef7ac:::benchmark_data_smithi171_78918_object148792:head# 2024-04-20T17:11:08.733 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79908528:::benchmark_data_smithi171_78918_object145096:head# 2024-04-20T17:11:08.733 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79918554:::benchmark_data_smithi171_78918_object131930:head# 2024-04-20T17:11:08.734 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79919737:::benchmark_data_smithi171_78918_object120731:head# 2024-04-20T17:11:08.734 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7991a042:::benchmark_data_smithi171_78918_object81796:head# 2024-04-20T17:11:08.734 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79927237:::benchmark_data_smithi171_78918_object125673:head# 2024-04-20T17:11:08.734 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7992e201:::benchmark_data_smithi171_78918_object59358:head# 2024-04-20T17:11:08.734 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7992f7da:::benchmark_data_smithi171_78918_object145634:head# 2024-04-20T17:11:08.734 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79939ea7:::benchmark_data_smithi171_78918_object57246:head# 2024-04-20T17:11:08.734 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79940a71:::benchmark_data_smithi171_78918_object116829:head# 2024-04-20T17:11:08.734 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79947113:::benchmark_data_smithi171_78918_object117731:head# 2024-04-20T17:11:08.734 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79950d42:::benchmark_data_smithi171_78918_object114268:head# 2024-04-20T17:11:08.734 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:799522ca:::benchmark_data_smithi171_78918_object160811:head# 2024-04-20T17:11:08.734 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79975b2c:::benchmark_data_smithi171_78918_object54088:head# 2024-04-20T17:11:08.734 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79977d6a:::benchmark_data_smithi171_78918_object80325:head# 2024-04-20T17:11:08.734 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7997dcaf:::benchmark_data_smithi171_78918_object58281:head# 2024-04-20T17:11:08.734 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79980cb7:::benchmark_data_smithi171_78918_object135127:head# 2024-04-20T17:11:08.734 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79981fd2:::benchmark_data_smithi171_78918_object86127:head# 2024-04-20T17:11:08.735 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:11:08.754 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79989df3:::benchmark_data_smithi171_78918_object66054:head# 2024-04-20T17:11:08.754 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7998ccfc:::benchmark_data_smithi171_78918_object141133:head# 2024-04-20T17:11:08.754 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:799a185a:::benchmark_data_smithi171_78918_object75450:head# 2024-04-20T17:11:08.754 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:799c8c55:::benchmark_data_smithi171_78918_object75713:head# 2024-04-20T17:11:08.754 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:799d2971:::benchmark_data_smithi171_78918_object130180:head# 2024-04-20T17:11:08.755 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:799d578f:::benchmark_data_smithi171_78918_object135882:head# 2024-04-20T17:11:08.755 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:799da661:::benchmark_data_smithi171_78918_object141137:head# 2024-04-20T17:11:08.755 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:799efdc7:::benchmark_data_smithi171_78918_object89710:head# 2024-04-20T17:11:08.755 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:799f9b6b:::benchmark_data_smithi171_78918_object78104:head# 2024-04-20T17:11:08.755 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79a0054e:::benchmark_data_smithi171_78918_object120069:head# 2024-04-20T17:11:08.755 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79a01253:::benchmark_data_smithi171_78918_object156843:head# 2024-04-20T17:11:08.755 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79a3fe80:::benchmark_data_smithi171_78918_object143367:head# 2024-04-20T17:11:08.755 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79a51e57:::benchmark_data_smithi171_78918_object116430:head# 2024-04-20T17:11:08.755 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79a6224e:::benchmark_data_smithi171_78918_object141419:head# 2024-04-20T17:11:08.755 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79a66437:::benchmark_data_smithi171_78918_object77552:head# 2024-04-20T17:11:08.755 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79a6b52d:::benchmark_data_smithi171_78918_object82205:head# 2024-04-20T17:11:08.755 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79a6da2a:::benchmark_data_smithi171_78918_object143901:head# 2024-04-20T17:11:08.755 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79a6f330:::benchmark_data_smithi171_78918_object64175:head# 2024-04-20T17:11:08.755 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79a6f7eb:::benchmark_data_smithi171_78918_object94544:head# 2024-04-20T17:11:08.756 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79a82037:::benchmark_data_smithi171_78918_object79073:head# 2024-04-20T17:11:08.756 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79a8306f:::benchmark_data_smithi171_78918_object157074:head# 2024-04-20T17:11:08.756 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79a8b7a9:::benchmark_data_smithi171_78918_object103204:head# 2024-04-20T17:11:08.756 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79a919d3:::benchmark_data_smithi171_78918_object133499:head# 2024-04-20T17:11:08.756 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79a94515:::benchmark_data_smithi171_78918_object136273:head# 2024-04-20T17:11:08.756 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79a98c80:::benchmark_data_smithi171_78918_object90713:head# 2024-04-20T17:11:08.756 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79aa2e7b:::benchmark_data_smithi171_78918_object160654:head# 2024-04-20T17:11:08.756 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79ada5c0:::benchmark_data_smithi171_78918_object149216:head# 2024-04-20T17:11:08.756 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79ae9149:::benchmark_data_smithi171_78918_object110787:head# 2024-04-20T17:11:08.756 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79aebd2f:::benchmark_data_smithi171_78918_object115360:head# 2024-04-20T17:11:08.756 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79aecd90:::benchmark_data_smithi171_78918_object102793:head# 2024-04-20T17:11:08.756 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79af1148:::benchmark_data_smithi171_78918_object158573:head# 2024-04-20T17:11:08.756 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79af6220:::benchmark_data_smithi171_78918_object88578:head# 2024-04-20T17:11:08.756 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79afaadd:::benchmark_data_smithi171_78918_object80184:head# 2024-04-20T17:11:08.757 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79affa3b:::benchmark_data_smithi171_78918_object82384:head# 2024-04-20T17:11:08.757 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79b0950f:::benchmark_data_smithi171_78918_object54671:head# 2024-04-20T17:11:08.757 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79b10321:::benchmark_data_smithi171_78918_object67112:head# 2024-04-20T17:11:08.757 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79b22870:::benchmark_data_smithi171_78918_object111740:head# 2024-04-20T17:11:08.757 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79b2a8af:::benchmark_data_smithi171_78918_object144307:head# 2024-04-20T17:11:08.757 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79b3591c:::benchmark_data_smithi171_78918_object146559:head# 2024-04-20T17:11:08.757 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79b59123:::benchmark_data_smithi171_78918_object111711:head# 2024-04-20T17:11:08.757 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79b5c0ad:::benchmark_data_smithi171_78918_object59505:head# 2024-04-20T17:11:08.757 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79b5ce88:::benchmark_data_smithi171_78918_object131281:head# 2024-04-20T17:11:08.757 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79b5f4e1:::benchmark_data_smithi171_78918_object108996:head# 2024-04-20T17:11:08.757 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79b62000:::benchmark_data_smithi171_78918_object122884:head# 2024-04-20T17:11:08.757 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79b65f25:::benchmark_data_smithi171_78918_object102135:head# 2024-04-20T17:11:08.757 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79b71d62:::benchmark_data_smithi171_78918_object72318:head# 2024-04-20T17:11:08.757 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79b79997:::benchmark_data_smithi171_78918_object90020:head# 2024-04-20T17:11:08.758 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79b99450:::benchmark_data_smithi171_78918_object152076:head# 2024-04-20T17:11:08.758 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79ba82e0:::benchmark_data_smithi171_78918_object70135:head# 2024-04-20T17:11:08.758 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79bade6d:::benchmark_data_smithi171_78918_object150615:head# 2024-04-20T17:11:08.758 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79bb2ec0:::benchmark_data_smithi171_78918_object123465:head# 2024-04-20T17:11:08.758 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79bbd948:::benchmark_data_smithi171_78918_object77805:head# 2024-04-20T17:11:08.758 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79bbffae:::benchmark_data_smithi171_78918_object148125:head# 2024-04-20T17:11:08.758 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79bc27b3:::benchmark_data_smithi171_78918_object157739:head# 2024-04-20T17:11:08.758 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79bd1c7c:::benchmark_data_smithi171_78918_object129703:head# 2024-04-20T17:11:08.758 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79bd5b87:::benchmark_data_smithi171_78918_object147835:head# 2024-04-20T17:11:08.758 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79bd5d68:::benchmark_data_smithi171_78918_object56939:head# 2024-04-20T17:11:08.758 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79be87f5:::benchmark_data_smithi171_78918_object81041:head# 2024-04-20T17:11:08.758 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79bed745:::benchmark_data_smithi171_78918_object133517:head# 2024-04-20T17:11:08.758 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79bf62c8:::benchmark_data_smithi171_78918_object56719:head# 2024-04-20T17:11:08.758 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79bf7715:::benchmark_data_smithi171_78918_object122447:head# 2024-04-20T17:11:08.759 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79bf7833:::benchmark_data_smithi171_78918_object144543:head# 2024-04-20T17:11:08.759 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79c1039d:::benchmark_data_smithi171_78918_object81498:head# 2024-04-20T17:11:08.759 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79c10ee9:::benchmark_data_smithi171_78918_object58984:head# 2024-04-20T17:11:08.759 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79c2350f:::benchmark_data_smithi171_78918_object70733:head# 2024-04-20T17:11:08.759 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79c26c57:::benchmark_data_smithi171_78918_object83612:head# 2024-04-20T17:11:08.759 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79c31240:::benchmark_data_smithi171_78918_object121297:head# 2024-04-20T17:11:08.759 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79c3ce22:::benchmark_data_smithi171_78918_object128174:head# 2024-04-20T17:11:08.759 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79c3d152:::benchmark_data_smithi171_78918_object98952:head# 2024-04-20T17:11:08.759 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79c40004:::benchmark_data_smithi171_78918_object98604:head# 2024-04-20T17:11:08.759 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79c5552f:::benchmark_data_smithi171_78918_object81832:head# 2024-04-20T17:11:08.759 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79c6294b:::benchmark_data_smithi171_78918_object150918:head# 2024-04-20T17:11:08.759 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79c65a35:::benchmark_data_smithi171_78918_object152400:head# 2024-04-20T17:11:08.759 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79c687e5:::benchmark_data_smithi171_78918_object103979:head# 2024-04-20T17:11:08.760 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79c88338:::benchmark_data_smithi171_78918_object124875:head# 2024-04-20T17:11:08.760 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79ca2d24:::benchmark_data_smithi171_78918_object72717:head# 2024-04-20T17:11:08.760 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79ca3968:::benchmark_data_smithi171_78918_object93812:head# 2024-04-20T17:11:08.760 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79cb32b6:::benchmark_data_smithi171_78918_object124979:head# 2024-04-20T17:11:08.760 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79cbed8c:::benchmark_data_smithi171_78918_object124019:head# 2024-04-20T17:11:08.760 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79cc36fe:::benchmark_data_smithi171_78918_object154054:head# 2024-04-20T17:11:08.760 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79cc7bc3:::benchmark_data_smithi171_78918_object143226:head# 2024-04-20T17:11:08.760 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79ce02aa:::benchmark_data_smithi171_78918_object86697:head# 2024-04-20T17:11:08.760 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79ce0966:::benchmark_data_smithi171_78918_object107898:head# 2024-04-20T17:11:08.760 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79ce6e81:::benchmark_data_smithi171_78918_object108153:head# 2024-04-20T17:11:08.760 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79cef47a:::benchmark_data_smithi171_78918_object160056:head# 2024-04-20T17:11:08.760 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79cf84fe:::benchmark_data_smithi171_78918_object142967:head# 2024-04-20T17:11:08.760 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79d018e4:::benchmark_data_smithi171_78918_object154725:head# 2024-04-20T17:11:08.760 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79d04c4a:::benchmark_data_smithi171_78918_object157388:head# 2024-04-20T17:11:08.760 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79d0593d:::benchmark_data_smithi171_78918_object121057:head# 2024-04-20T17:11:08.760 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79d0626d:::benchmark_data_smithi171_78918_object86630:head# 2024-04-20T17:11:08.761 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79d08467:::benchmark_data_smithi171_78918_object53370:head# 2024-04-20T17:11:08.761 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79d102d0:::benchmark_data_smithi171_78918_object150832:head# 2024-04-20T17:11:08.761 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79d12421:::benchmark_data_smithi171_78918_object64772:head# 2024-04-20T17:11:08.761 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79d161b5:::benchmark_data_smithi171_78918_object103103:head# 2024-04-20T17:11:08.761 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79d24f2b:::benchmark_data_smithi171_78918_object92457:head# 2024-04-20T17:11:08.761 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79d3af5d:::benchmark_data_smithi171_78918_object74776:head# 2024-04-20T17:11:08.761 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79d41b7f:::benchmark_data_smithi171_78918_object93782:head# 2024-04-20T17:11:08.761 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79d4452c:::benchmark_data_smithi171_78918_object93254:head# 2024-04-20T17:11:08.761 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79d45c1a:::benchmark_data_smithi171_78918_object57366:head# 2024-04-20T17:11:08.761 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79d49906:::benchmark_data_smithi171_78918_object95339:head# 2024-04-20T17:11:08.761 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79d692ee:::benchmark_data_smithi171_78918_object67310:head# 2024-04-20T17:11:08.761 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79d6db27:::benchmark_data_smithi171_78918_object58290:head# 2024-04-20T17:11:08.761 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79d76e12:::benchmark_data_smithi171_78918_object94164:head# 2024-04-20T17:11:08.761 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79d89689:::benchmark_data_smithi171_78918_object62384:head# 2024-04-20T17:11:08.761 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79d8db8b:::benchmark_data_smithi171_78918_object132045:head# 2024-04-20T17:11:08.761 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79d94f67:::benchmark_data_smithi171_78918_object129683:head# 2024-04-20T17:11:08.762 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79d95c6c:::benchmark_data_smithi171_78918_object108001:head# 2024-04-20T17:11:08.762 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79d98da5:::benchmark_data_smithi171_78918_object105376:head# 2024-04-20T17:11:08.762 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79d9d3f2:::benchmark_data_smithi171_78918_object143998:head# 2024-04-20T17:11:08.762 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79da019d:::benchmark_data_smithi171_78918_object58030:head# 2024-04-20T17:11:08.762 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79da8fe3:::benchmark_data_smithi171_78918_object158312:head# 2024-04-20T17:11:08.762 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79dd45b2:::benchmark_data_smithi171_78918_object70149:head# 2024-04-20T17:11:08.762 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79dd843f:::benchmark_data_smithi171_78918_object88444:head# 2024-04-20T17:11:08.762 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79dec167:::benchmark_data_smithi171_78918_object87081:head# 2024-04-20T17:11:08.762 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79df19f6:::benchmark_data_smithi171_78918_object120254:head# 2024-04-20T17:11:08.762 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79df3dff:::benchmark_data_smithi171_78918_object109598:head# 2024-04-20T17:11:08.762 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79dfc33e:::benchmark_data_smithi171_78918_object158357:head# 2024-04-20T17:11:08.762 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79e00df7:::benchmark_data_smithi171_78918_object155323:head# 2024-04-20T17:11:08.762 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79e015b7:::benchmark_data_smithi171_78918_object103580:head# 2024-04-20T17:11:08.762 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79e2d17e:::benchmark_data_smithi171_78918_object159896:head# 2024-04-20T17:11:08.763 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79e3ae10:::benchmark_data_smithi171_78918_object52761:head# 2024-04-20T17:11:08.763 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79e47ea1:::benchmark_data_smithi171_78918_object145156:head# 2024-04-20T17:11:08.763 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79e4e8da:::benchmark_data_smithi171_78918_object56008:head# 2024-04-20T17:11:08.763 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79e51876:::benchmark_data_smithi171_78918_object160062:head# 2024-04-20T17:11:08.763 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79e74298:::benchmark_data_smithi171_78918_object119749:head# 2024-04-20T17:11:08.763 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79e75d8f:::benchmark_data_smithi171_78918_object95261:head# 2024-04-20T17:11:08.763 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79e849c4:::benchmark_data_smithi171_78918_object156874:head# 2024-04-20T17:11:08.763 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79e870dd:::benchmark_data_smithi171_78918_object124598:head# 2024-04-20T17:11:08.763 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79e8d034:::benchmark_data_smithi171_78918_object70266:head# 2024-04-20T17:11:08.763 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79e97be8:::benchmark_data_smithi171_78918_object94899:head# 2024-04-20T17:11:08.763 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79e989f8:::benchmark_data_smithi171_78918_object114944:head# 2024-04-20T17:11:08.763 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79ea4388:::benchmark_data_smithi171_78918_object118545:head# 2024-04-20T17:11:08.763 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79eaf098:::benchmark_data_smithi171_78918_object139484:head# 2024-04-20T17:11:08.764 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79ebaa20:::benchmark_data_smithi171_78918_object131481:head# 2024-04-20T17:11:08.764 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79eccca1:::benchmark_data_smithi171_78918_object82103:head# 2024-04-20T17:11:08.764 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79ed2623:::benchmark_data_smithi171_78918_object100624:head# 2024-04-20T17:11:08.764 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79ed4965:::benchmark_data_smithi171_78918_object52674:head# 2024-04-20T17:11:08.764 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79ed8303:::benchmark_data_smithi171_78918_object76859:head# 2024-04-20T17:11:08.764 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79ed9e0f:::benchmark_data_smithi171_78918_object111122:head# 2024-04-20T17:11:08.764 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79edacf3:::benchmark_data_smithi171_78918_object149586:head# 2024-04-20T17:11:08.764 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79ee18d6:::benchmark_data_smithi171_78918_object115060:head# 2024-04-20T17:11:08.764 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79ef0fed:::benchmark_data_smithi171_78918_object88914:head# 2024-04-20T17:11:08.764 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79ef1cb6:::benchmark_data_smithi171_78918_object104260:head# 2024-04-20T17:11:08.764 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79ef3ca6:::benchmark_data_smithi171_78918_object152973:head# 2024-04-20T17:11:08.764 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79efd519:::benchmark_data_smithi171_78918_object73551:head# 2024-04-20T17:11:08.764 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79f04790:::benchmark_data_smithi171_78918_object121030:head# 2024-04-20T17:11:08.764 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79f0a7ac:::benchmark_data_smithi171_78918_object116428:head# 2024-04-20T17:11:08.764 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79f0fb61:::benchmark_data_smithi171_78918_object140064:head# 2024-04-20T17:11:08.765 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79f13a4b:::benchmark_data_smithi171_78918_object120616:head# 2024-04-20T17:11:08.765 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79f2664d:::benchmark_data_smithi171_78918_object158765:head# 2024-04-20T17:11:08.765 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79f28409:::benchmark_data_smithi171_78918_object132205:head# 2024-04-20T17:11:08.765 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79f447a5:::benchmark_data_smithi171_78918_object159484:head# 2024-04-20T17:11:08.765 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79f45b25:::benchmark_data_smithi171_78918_object156885:head# 2024-04-20T17:11:08.765 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79f465fd:::benchmark_data_smithi171_78918_object125894:head# 2024-04-20T17:11:08.765 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79f46b11:::benchmark_data_smithi171_78918_object87436:head# 2024-04-20T17:11:08.765 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79f4cff7:::benchmark_data_smithi171_78918_object156913:head# 2024-04-20T17:11:08.765 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79f5d782:::benchmark_data_smithi171_78918_object147024:head# 2024-04-20T17:11:08.765 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79f6352f:::benchmark_data_smithi171_78918_object91863:head# 2024-04-20T17:11:08.765 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79f7f347:::benchmark_data_smithi171_78918_object122267:head# 2024-04-20T17:11:08.765 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79f8f4d3:::benchmark_data_smithi171_78918_object113455:head# 2024-04-20T17:11:08.765 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79f97775:::benchmark_data_smithi171_78918_object160561:head# 2024-04-20T17:11:08.766 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:11:08.799 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79f97a2f:::benchmark_data_smithi171_78918_object129426:head# 2024-04-20T17:11:08.800 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79fadb2b:::benchmark_data_smithi171_78918_object76485:head# 2024-04-20T17:11:08.800 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79fbee36:::benchmark_data_smithi171_78918_object157651:head# 2024-04-20T17:11:08.800 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79fc71f3:::benchmark_data_smithi171_78918_object56713:head# 2024-04-20T17:11:08.800 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79fceef6:::benchmark_data_smithi171_78918_object137738:head# 2024-04-20T17:11:08.800 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79fd7bbc:::benchmark_data_smithi171_78918_object146562:head# 2024-04-20T17:11:08.800 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79fe63bb:::benchmark_data_smithi171_78918_object112804:head# 2024-04-20T17:11:08.800 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79fef87c:::benchmark_data_smithi171_78918_object83211:head# 2024-04-20T17:11:08.800 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79ff772e:::benchmark_data_smithi171_78918_object107851:head# 2024-04-20T17:11:08.800 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:79fff31e:::benchmark_data_smithi171_78918_object124813:head# 2024-04-20T17:11:08.800 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a00aa85:::benchmark_data_smithi171_78918_object117993:head# 2024-04-20T17:11:08.800 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a01026f:::benchmark_data_smithi171_78918_object141238:head# 2024-04-20T17:11:08.800 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a017d76:::benchmark_data_smithi171_78918_object103060:head# 2024-04-20T17:11:08.800 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a024da8:::benchmark_data_smithi171_78918_object118585:head# 2024-04-20T17:11:08.801 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a02f633:::benchmark_data_smithi171_78918_object114876:head# 2024-04-20T17:11:08.801 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a03f476:::benchmark_data_smithi171_78918_object140668:head# 2024-04-20T17:11:08.801 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a042b1b:::benchmark_data_smithi171_78918_object87800:head# 2024-04-20T17:11:08.801 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a04de71:::benchmark_data_smithi171_78918_object56182:head# 2024-04-20T17:11:08.801 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a067298:::benchmark_data_smithi171_78918_object75152:head# 2024-04-20T17:11:08.801 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a0685c4:::benchmark_data_smithi171_78918_object73809:head# 2024-04-20T17:11:08.801 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a09ce2a:::benchmark_data_smithi171_78918_object121323:head# 2024-04-20T17:11:08.801 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a0a866d:::benchmark_data_smithi171_78918_object54774:head# 2024-04-20T17:11:08.801 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a0b8ac2:::benchmark_data_smithi171_78918_object68878:head# 2024-04-20T17:11:08.801 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a0d956b:::benchmark_data_smithi171_78918_object84024:head# 2024-04-20T17:11:08.801 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a0db227:::benchmark_data_smithi171_78918_object126170:head# 2024-04-20T17:11:08.801 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a0f7c74:::benchmark_data_smithi171_78918_object133784:head# 2024-04-20T17:11:08.801 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a0f83a7:::benchmark_data_smithi171_78918_object76508:head# 2024-04-20T17:11:08.801 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a1274c7:::benchmark_data_smithi171_78918_object135682:head# 2024-04-20T17:11:08.801 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a12d091:::benchmark_data_smithi171_78918_object69980:head# 2024-04-20T17:11:08.802 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a1316dc:::benchmark_data_smithi171_78918_object106948:head# 2024-04-20T17:11:08.802 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a1343db:::benchmark_data_smithi171_78918_object52175:head# 2024-04-20T17:11:08.802 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a13490a:::benchmark_data_smithi171_78918_object82145:head# 2024-04-20T17:11:08.802 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a138ede:::benchmark_data_smithi171_78918_object59243:head# 2024-04-20T17:11:08.802 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a13a75a:::benchmark_data_smithi171_78918_object152208:head# 2024-04-20T17:11:08.802 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a145329:::benchmark_data_smithi171_78918_object74135:head# 2024-04-20T17:11:08.802 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a14c82f:::benchmark_data_smithi171_78918_object135347:head# 2024-04-20T17:11:08.802 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a1560f1:::benchmark_data_smithi171_78918_object53236:head# 2024-04-20T17:11:08.802 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a1578a4:::benchmark_data_smithi171_78918_object76262:head# 2024-04-20T17:11:08.802 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a16b4fc:::benchmark_data_smithi171_78918_object59616:head# 2024-04-20T17:11:08.802 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a18ba5e:::benchmark_data_smithi171_78918_object141264:head# 2024-04-20T17:11:08.802 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a18f67c:::benchmark_data_smithi171_78918_object65847:head# 2024-04-20T17:11:08.802 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a18fb80:::benchmark_data_smithi171_78918_object104095:head# 2024-04-20T17:11:08.803 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a19a054:::benchmark_data_smithi171_78918_object97236:head# 2024-04-20T17:11:08.803 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a19e172:::benchmark_data_smithi171_78918_object157831:head# 2024-04-20T17:11:08.803 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a1a2803:::benchmark_data_smithi171_78918_object90034:head# 2024-04-20T17:11:08.803 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a1a6b2b:::benchmark_data_smithi171_78918_object128837:head# 2024-04-20T17:11:08.803 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a1b0871:::benchmark_data_smithi171_78918_object88550:head# 2024-04-20T17:11:08.803 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a1b3739:::benchmark_data_smithi171_78918_object134512:head# 2024-04-20T17:11:08.803 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a1c2655:::benchmark_data_smithi171_78918_object61814:head# 2024-04-20T17:11:08.803 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a1ccdfe:::benchmark_data_smithi171_78918_object120806:head# 2024-04-20T17:11:08.803 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a1cf5a4:::benchmark_data_smithi171_78918_object98256:head# 2024-04-20T17:11:08.803 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a1d80fc:::benchmark_data_smithi171_78918_object97319:head# 2024-04-20T17:11:08.803 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a1dbd3e:::benchmark_data_smithi171_78918_object74071:head# 2024-04-20T17:11:08.803 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a1e4f2a:::benchmark_data_smithi171_78918_object125568:head# 2024-04-20T17:11:08.803 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a1f91c6:::benchmark_data_smithi171_78918_object124369:head# 2024-04-20T17:11:08.803 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a1fffce:::benchmark_data_smithi171_78918_object143644:head# 2024-04-20T17:11:08.803 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a2007fb:::benchmark_data_smithi171_78918_object157354:head# 2024-04-20T17:11:08.804 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a201b8e:::benchmark_data_smithi171_78918_object90268:head# 2024-04-20T17:11:08.804 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a2082f3:::benchmark_data_smithi171_78918_object55348:head# 2024-04-20T17:11:08.804 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a213807:::benchmark_data_smithi171_78918_object150935:head# 2024-04-20T17:11:08.804 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a22b3e8:::benchmark_data_smithi171_78918_object155430:head# 2024-04-20T17:11:08.804 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a23a335:::benchmark_data_smithi171_78918_object114719:head# 2024-04-20T17:11:08.804 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a23dcbe:::benchmark_data_smithi171_78918_object118344:head# 2024-04-20T17:11:08.804 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a248def:::benchmark_data_smithi171_78918_object158876:head# 2024-04-20T17:11:08.804 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a263335:::benchmark_data_smithi171_78918_object66310:head# 2024-04-20T17:11:08.804 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a268ee2:::benchmark_data_smithi171_78918_object148150:head# 2024-04-20T17:11:08.804 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a278592:::benchmark_data_smithi171_78918_object118602:head# 2024-04-20T17:11:08.804 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a27fce5:::benchmark_data_smithi171_78918_object113068:head# 2024-04-20T17:11:08.804 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a2820fe:::benchmark_data_smithi171_78918_object62575:head# 2024-04-20T17:11:08.804 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a290220:::benchmark_data_smithi171_78918_object75451:head# 2024-04-20T17:11:08.804 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a2a26bc:::benchmark_data_smithi171_78918_object140697:head# 2024-04-20T17:11:08.805 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a2a37e9:::benchmark_data_smithi171_78918_object118036:head# 2024-04-20T17:11:08.805 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a2abd31:::benchmark_data_smithi171_78918_object148752:head# 2024-04-20T17:11:08.805 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a2bae2c:::benchmark_data_smithi171_78918_object63742:head# 2024-04-20T17:11:08.805 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a2cf599:::benchmark_data_smithi171_78918_object99230:head# 2024-04-20T17:11:08.805 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a2e6ae6:::benchmark_data_smithi171_78918_object117352:head# 2024-04-20T17:11:08.805 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a2e7e29:::benchmark_data_smithi171_78918_object83606:head# 2024-04-20T17:11:08.805 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a2eeab6:::benchmark_data_smithi171_78918_object85235:head# 2024-04-20T17:11:08.805 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a2ef06b:::benchmark_data_smithi171_78918_object81645:head# 2024-04-20T17:11:08.805 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a2fe154:::benchmark_data_smithi171_78918_object105205:head# 2024-04-20T17:11:08.805 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a306219:::benchmark_data_smithi171_78918_object73057:head# 2024-04-20T17:11:08.805 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a32a9f7:::benchmark_data_smithi171_78918_object73326:head# 2024-04-20T17:11:08.805 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a332315:::benchmark_data_smithi171_78918_object151241:head# 2024-04-20T17:11:08.805 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a34e4ce:::benchmark_data_smithi171_78918_object96197:head# 2024-04-20T17:11:08.805 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a35a6d5:::benchmark_data_smithi171_78918_object65308:head# 2024-04-20T17:11:08.805 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a372337:::benchmark_data_smithi171_78918_object153251:head# 2024-04-20T17:11:08.806 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a3738ca:::benchmark_data_smithi171_78918_object71041:head# 2024-04-20T17:11:08.806 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a373a7d:::benchmark_data_smithi171_78918_object139042:head# 2024-04-20T17:11:08.806 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a379fd0:::benchmark_data_smithi171_78918_object143771:head# 2024-04-20T17:11:08.806 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a37af1e:::benchmark_data_smithi171_78918_object155812:head# 2024-04-20T17:11:08.806 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a37b408:::benchmark_data_smithi171_78918_object119617:head# 2024-04-20T17:11:08.806 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a380704:::benchmark_data_smithi171_78918_object81429:head# 2024-04-20T17:11:08.806 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a38dd1f:::benchmark_data_smithi171_78918_object92321:head# 2024-04-20T17:11:08.806 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a394757:::benchmark_data_smithi171_78918_object55877:head# 2024-04-20T17:11:08.806 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a39a3f7:::benchmark_data_smithi171_78918_object85374:head# 2024-04-20T17:11:08.806 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a39f134:::benchmark_data_smithi171_78918_object154908:head# 2024-04-20T17:11:08.806 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a3d55bd:::benchmark_data_smithi171_78918_object137206:head# 2024-04-20T17:11:08.806 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a3ec32a:::benchmark_data_smithi171_78918_object63744:head# 2024-04-20T17:11:08.806 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a3f1a77:::benchmark_data_smithi171_78918_object138990:head# 2024-04-20T17:11:08.806 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a3f2591:::benchmark_data_smithi171_78918_object82372:head# 2024-04-20T17:11:08.807 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a400b19:::benchmark_data_smithi171_78918_object83147:head# 2024-04-20T17:11:08.807 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a404279:::benchmark_data_smithi171_78918_object104935:head# 2024-04-20T17:11:08.807 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a413b55:::benchmark_data_smithi171_78918_object154964:head# 2024-04-20T17:11:08.807 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a41c5ac:::benchmark_data_smithi171_78918_object67864:head# 2024-04-20T17:11:08.807 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a41cfd8:::benchmark_data_smithi171_78918_object105441:head# 2024-04-20T17:11:08.807 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a41ed2b:::benchmark_data_smithi171_78918_object58587:head# 2024-04-20T17:11:08.807 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a431147:::benchmark_data_smithi171_78918_object117436:head# 2024-04-20T17:11:08.807 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a433370:::benchmark_data_smithi171_78918_object124949:head# 2024-04-20T17:11:08.807 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a43c878:::benchmark_data_smithi171_78918_object129843:head# 2024-04-20T17:11:08.807 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a43db4d:::benchmark_data_smithi171_78918_object68014:head# 2024-04-20T17:11:08.807 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a447806:::benchmark_data_smithi171_78918_object91504:head# 2024-04-20T17:11:08.807 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a44fdb6:::benchmark_data_smithi171_78918_object147383:head# 2024-04-20T17:11:08.807 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a4622a2:::benchmark_data_smithi171_78918_object85099:head# 2024-04-20T17:11:08.807 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a4715a0:::benchmark_data_smithi171_78918_object67795:head# 2024-04-20T17:11:08.808 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a47c8f2:::benchmark_data_smithi171_78918_object68370:head# 2024-04-20T17:11:08.808 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a47e564:::benchmark_data_smithi171_78918_object94241:head# 2024-04-20T17:11:08.808 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a480f9f:::benchmark_data_smithi171_78918_object59017:head# 2024-04-20T17:11:08.808 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a49a60a:::benchmark_data_smithi171_78918_object117756:head# 2024-04-20T17:11:08.808 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a4adc2b:::benchmark_data_smithi171_78918_object69048:head# 2024-04-20T17:11:08.808 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a4b8a32:::benchmark_data_smithi171_78918_object62543:head# 2024-04-20T17:11:08.808 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a4bff46:::benchmark_data_smithi171_78918_object144903:head# 2024-04-20T17:11:08.808 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a4c7dc4:::benchmark_data_smithi171_78918_object53418:head# 2024-04-20T17:11:08.808 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a4e3d65:::benchmark_data_smithi171_78918_object62736:head# 2024-04-20T17:11:08.808 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a4edbba:::benchmark_data_smithi171_78918_object133247:head# 2024-04-20T17:11:08.808 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a4f1906:::benchmark_data_smithi171_78918_object158478:head# 2024-04-20T17:11:08.808 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a4f3cf2:::benchmark_data_smithi171_78918_object97581:head# 2024-04-20T17:11:08.808 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a4fe739:::benchmark_data_smithi171_78918_object139017:head# 2024-04-20T17:11:08.808 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a507f94:::benchmark_data_smithi171_78918_object62737:head# 2024-04-20T17:11:08.809 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a5099d2:::benchmark_data_smithi171_78918_object151312:head# 2024-04-20T17:11:08.809 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a50e71f:::benchmark_data_smithi171_78918_object99918:head# 2024-04-20T17:11:08.809 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a51097f:::benchmark_data_smithi171_78918_object124078:head# 2024-04-20T17:11:08.809 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a51126b:::benchmark_data_smithi171_78918_object77361:head# 2024-04-20T17:11:08.809 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a514473:::benchmark_data_smithi171_78918_object138411:head# 2024-04-20T17:11:08.809 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a52687f:::benchmark_data_smithi171_78918_object110049:head# 2024-04-20T17:11:08.809 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a5287c7:::benchmark_data_smithi171_78918_object75308:head# 2024-04-20T17:11:08.809 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a52fe90:::benchmark_data_smithi171_78918_object154701:head# 2024-04-20T17:11:08.809 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a53380b:::benchmark_data_smithi171_78918_object51547:head# 2024-04-20T17:11:08.809 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a53d651:::benchmark_data_smithi171_78918_object57875:head# 2024-04-20T17:11:08.809 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a542bf3:::benchmark_data_smithi171_78918_object85004:head# 2024-04-20T17:11:08.809 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a54abbf:::benchmark_data_smithi171_78918_object137436:head# 2024-04-20T17:11:08.809 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a54d71a:::benchmark_data_smithi171_78918_object97582:head# 2024-04-20T17:11:08.809 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a54e9ee:::benchmark_data_smithi171_78918_object52256:head# 2024-04-20T17:11:08.810 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a551dcd:::benchmark_data_smithi171_78918_object141841:head# 2024-04-20T17:11:08.810 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a58599b:::benchmark_data_smithi171_78918_object115785:head# 2024-04-20T17:11:08.810 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a58c730:::benchmark_data_smithi171_78918_object63354:head# 2024-04-20T17:11:08.810 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a58d855:::benchmark_data_smithi171_78918_object109170:head# 2024-04-20T17:11:08.810 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a597d97:::benchmark_data_smithi171_78918_object58596:head# 2024-04-20T17:11:08.810 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a5a00c7:::benchmark_data_smithi171_78918_object134779:head# 2024-04-20T17:11:08.810 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a5a8b57:::benchmark_data_smithi171_78918_object102509:head# 2024-04-20T17:11:08.810 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a5b63e0:::benchmark_data_smithi171_78918_object148464:head# 2024-04-20T17:11:08.810 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a5b8b0e:::benchmark_data_smithi171_78918_object144534:head# 2024-04-20T17:11:08.810 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a5c3a3a:::benchmark_data_smithi171_78918_object118220:head# 2024-04-20T17:11:08.810 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a5e225f:::benchmark_data_smithi171_78918_object61349:head# 2024-04-20T17:11:08.810 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a5e3d4d:::benchmark_data_smithi171_78918_object84264:head# 2024-04-20T17:11:08.810 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a5e7199:::benchmark_data_smithi171_78918_object80205:head# 2024-04-20T17:11:08.810 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a5e7f07:::benchmark_data_smithi171_78918_object54633:head# 2024-04-20T17:11:08.811 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a5e9268:::benchmark_data_smithi171_78918_object145963:head# 2024-04-20T17:11:08.811 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a5f2d82:::benchmark_data_smithi171_78918_object72999:head# 2024-04-20T17:11:08.811 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a5fbab9:::benchmark_data_smithi171_78918_object111927:head# 2024-04-20T17:11:08.811 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a623299:::benchmark_data_smithi171_78918_object114885:head# 2024-04-20T17:11:08.811 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a637f81:::benchmark_data_smithi171_78918_object53367:head# 2024-04-20T17:11:08.811 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a64d791:::benchmark_data_smithi171_78918_object120482:head# 2024-04-20T17:11:08.811 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a64e737:::benchmark_data_smithi171_78918_object136224:head# 2024-04-20T17:11:08.811 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a64f7c3:::benchmark_data_smithi171_78918_object79826:head# 2024-04-20T17:11:08.811 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a657784:::benchmark_data_smithi171_78918_object123971:head# 2024-04-20T17:11:08.811 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a658ba5:::benchmark_data_smithi171_78918_object157875:head# 2024-04-20T17:11:08.811 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a66b75e:::benchmark_data_smithi171_78918_object146030:head# 2024-04-20T17:11:08.811 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a678977:::benchmark_data_smithi171_78918_object100060:head# 2024-04-20T17:11:08.811 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a68f03f:::benchmark_data_smithi171_78918_object134463:head# 2024-04-20T17:11:08.811 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a6b3fc1:::benchmark_data_smithi171_78918_object150607:head# 2024-04-20T17:11:08.812 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a6bfb2f:::benchmark_data_smithi171_78918_object116177:head# 2024-04-20T17:11:08.812 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a6c6962:::benchmark_data_smithi171_78918_object103882:head# 2024-04-20T17:11:08.812 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a6d72e2:::benchmark_data_smithi171_78918_object145560:head# 2024-04-20T17:11:08.812 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a6ead15:::benchmark_data_smithi171_78918_object134969:head# 2024-04-20T17:11:08.812 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a6ec082:::benchmark_data_smithi171_78918_object120759:head# 2024-04-20T17:11:08.812 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a6fcc4c:::benchmark_data_smithi171_78918_object102650:head# 2024-04-20T17:11:08.812 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a710eec:::benchmark_data_smithi171_78918_object51724:head# 2024-04-20T17:11:08.812 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a71151a:::benchmark_data_smithi171_78918_object114041:head# 2024-04-20T17:11:08.812 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a718310:::benchmark_data_smithi171_78918_object102802:head# 2024-04-20T17:11:08.812 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a7279ca:::benchmark_data_smithi171_78918_object71486:head# 2024-04-20T17:11:08.812 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a729a26:::benchmark_data_smithi171_78918_object76748:head# 2024-04-20T17:11:08.812 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a7417e5:::benchmark_data_smithi171_78918_object60302:head# 2024-04-20T17:11:08.812 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a7418b8:::benchmark_data_smithi171_78918_object101633:head# 2024-04-20T17:11:08.812 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a748e69:::benchmark_data_smithi171_78918_object142521:head# 2024-04-20T17:11:08.812 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a74c99e:::benchmark_data_smithi171_78918_object116906:head# 2024-04-20T17:11:08.812 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a785edb:::benchmark_data_smithi171_78918_object77749:head# 2024-04-20T17:11:08.813 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a7a5ab7:::benchmark_data_smithi171_78918_object89767:head# 2024-04-20T17:11:08.813 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a7a9675:::benchmark_data_smithi171_78918_object113701:head# 2024-04-20T17:11:08.813 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a7ab3ff:::benchmark_data_smithi171_78918_object97456:head# 2024-04-20T17:11:08.813 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a7b02d1:::benchmark_data_smithi171_78918_object155383:head# 2024-04-20T17:11:08.813 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a7be279:::benchmark_data_smithi171_78918_object98809:head# 2024-04-20T17:11:08.813 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a7cb4f4:::benchmark_data_smithi171_78918_object111861:head# 2024-04-20T17:11:08.813 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a7d2954:::benchmark_data_smithi171_78918_object72126:head# 2024-04-20T17:11:08.813 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a7d69bb:::benchmark_data_smithi171_78918_object94442:head# 2024-04-20T17:11:08.813 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a7d7708:::benchmark_data_smithi171_78918_object130618:head# 2024-04-20T17:11:08.813 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a7e2ab1:::benchmark_data_smithi171_78918_object79674:head# 2024-04-20T17:11:08.813 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a7e4124:::benchmark_data_smithi171_78918_object93341:head# 2024-04-20T17:11:08.813 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a801f58:::benchmark_data_smithi171_78918_object65509:head# 2024-04-20T17:11:08.813 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a810958:::benchmark_data_smithi171_78918_object69056:head# 2024-04-20T17:11:08.813 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a81f4a6:::benchmark_data_smithi171_78918_object109106:head# 2024-04-20T17:11:08.813 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a82bb25:::benchmark_data_smithi171_78918_object127046:head# 2024-04-20T17:11:08.814 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a82f213:::benchmark_data_smithi171_78918_object106394:head# 2024-04-20T17:11:08.814 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a83c1ce:::benchmark_data_smithi171_78918_object77116:head# 2024-04-20T17:11:08.814 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a8402d7:::benchmark_data_smithi171_78918_object96287:head# 2024-04-20T17:11:08.814 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a84173c:::benchmark_data_smithi171_78918_object92512:head# 2024-04-20T17:11:08.814 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a84527c:::benchmark_data_smithi171_78918_object149016:head# 2024-04-20T17:11:08.814 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a85c2c7:::benchmark_data_smithi171_78918_object100589:head# 2024-04-20T17:11:08.814 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a85c338:::benchmark_data_smithi171_78918_object74034:head# 2024-04-20T17:11:08.814 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:11:08.834 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a860aa0:::benchmark_data_smithi171_78918_object109278:head# 2024-04-20T17:11:08.834 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a866a9e:::benchmark_data_smithi171_78918_object61756:head# 2024-04-20T17:11:08.834 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a87f270:::benchmark_data_smithi171_78918_object76539:head# 2024-04-20T17:11:08.834 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a8823e9:::benchmark_data_smithi171_78918_object112084:head# 2024-04-20T17:11:08.834 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a88447f:::benchmark_data_smithi171_78918_object94098:head# 2024-04-20T17:11:08.834 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a89836f:::benchmark_data_smithi171_78918_object130014:head# 2024-04-20T17:11:08.835 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a8a0c4e:::benchmark_data_smithi171_78918_object98405:head# 2024-04-20T17:11:08.835 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a8a1b80:::benchmark_data_smithi171_78918_object69542:head# 2024-04-20T17:11:08.835 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a8a3c4b:::benchmark_data_smithi171_78918_object65598:head# 2024-04-20T17:11:08.835 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a8b2539:::benchmark_data_smithi171_78918_object144641:head# 2024-04-20T17:11:08.835 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a8b333d:::benchmark_data_smithi171_78918_object94246:head# 2024-04-20T17:11:08.835 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a8c0b57:::benchmark_data_smithi171_78918_object144166:head# 2024-04-20T17:11:08.835 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a8c8ee0:::benchmark_data_smithi171_78918_object108897:head# 2024-04-20T17:11:08.835 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a8cafe2:::benchmark_data_smithi171_78918_object149610:head# 2024-04-20T17:11:08.835 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a8d437c:::benchmark_data_smithi171_78918_object60386:head# 2024-04-20T17:11:08.835 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a8d4b65:::benchmark_data_smithi171_78918_object101734:head# 2024-04-20T17:11:08.836 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a8dd24e:::benchmark_data_smithi171_78918_object148429:head# 2024-04-20T17:11:08.836 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a8f4af5:::benchmark_data_smithi171_78918_object69704:head# 2024-04-20T17:11:08.836 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a8faef1:::benchmark_data_smithi171_78918_object148408:head# 2024-04-20T17:11:08.836 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a8fc73e:::benchmark_data_smithi171_78918_object77961:head# 2024-04-20T17:11:08.836 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a900dc7:::benchmark_data_smithi171_78918_object104442:head# 2024-04-20T17:11:08.836 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a901b5c:::benchmark_data_smithi171_78918_object98820:head# 2024-04-20T17:11:08.836 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a9181d2:::benchmark_data_smithi171_78918_object136694:head# 2024-04-20T17:11:08.836 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a920921:::benchmark_data_smithi171_78918_object69654:head# 2024-04-20T17:11:08.836 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a924d16:::benchmark_data_smithi171_78918_object144892:head# 2024-04-20T17:11:08.836 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a934be6:::benchmark_data_smithi171_78918_object125638:head# 2024-04-20T17:11:08.836 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a939f0a:::benchmark_data_smithi171_78918_object144970:head# 2024-04-20T17:11:08.836 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a948730:::benchmark_data_smithi171_78918_object88101:head# 2024-04-20T17:11:08.836 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a949fae:::benchmark_data_smithi171_78918_object139987:head# 2024-04-20T17:11:08.836 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a956a5d:::benchmark_data_smithi171_78918_object83303:head# 2024-04-20T17:11:08.836 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a965d5c:::benchmark_data_smithi171_78918_object106661:head# 2024-04-20T17:11:08.837 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a968113:::benchmark_data_smithi171_78918_object122600:head# 2024-04-20T17:11:08.837 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a96e129:::benchmark_data_smithi171_78918_object121819:head# 2024-04-20T17:11:08.837 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a96ea78:::benchmark_data_smithi171_78918_object94924:head# 2024-04-20T17:11:08.837 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a97054f:::benchmark_data_smithi171_78918_object63429:head# 2024-04-20T17:11:08.837 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a972daf:::benchmark_data_smithi171_78918_object157879:head# 2024-04-20T17:11:08.837 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a9880d1:::benchmark_data_smithi171_78918_object117331:head# 2024-04-20T17:11:08.837 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a98b261:::benchmark_data_smithi171_78918_object91371:head# 2024-04-20T17:11:08.837 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a997594:::benchmark_data_smithi171_78918_object109678:head# 2024-04-20T17:11:08.837 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a99c9f1:::benchmark_data_smithi171_78918_object117928:head# 2024-04-20T17:11:08.837 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a9a3d5c:::benchmark_data_smithi171_78918_object136921:head# 2024-04-20T17:11:08.837 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a9ac781:::benchmark_data_smithi171_78918_object130420:head# 2024-04-20T17:11:08.837 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a9c1aac:::benchmark_data_smithi171_78918_object137393:head# 2024-04-20T17:11:08.837 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a9daa17:::benchmark_data_smithi171_78918_object107094:head# 2024-04-20T17:11:08.837 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a9dd3d3:::benchmark_data_smithi171_78918_object117794:head# 2024-04-20T17:11:08.838 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a9dfa41:::benchmark_data_smithi171_78918_object94221:head# 2024-04-20T17:11:08.838 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a9f4440:::benchmark_data_smithi171_78918_object93932:head# 2024-04-20T17:11:08.838 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a9f707e:::benchmark_data_smithi171_78918_object139543:head# 2024-04-20T17:11:08.838 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7a9fa216:::benchmark_data_smithi171_78918_object125618:head# 2024-04-20T17:11:08.838 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aa06351:::benchmark_data_smithi171_78918_object93124:head# 2024-04-20T17:11:08.838 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aa16fc5:::benchmark_data_smithi171_78918_object80787:head# 2024-04-20T17:11:08.838 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aa1b2f5:::benchmark_data_smithi171_78918_object91716:head# 2024-04-20T17:11:08.838 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aa1ee7e:::benchmark_data_smithi171_78918_object89331:head# 2024-04-20T17:11:08.838 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aa1ef5a:::benchmark_data_smithi171_78918_object120973:head# 2024-04-20T17:11:08.838 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aa49a58:::benchmark_data_smithi171_78918_object93565:head# 2024-04-20T17:11:08.838 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aa56b60:::benchmark_data_smithi171_78918_object96546:head# 2024-04-20T17:11:08.838 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aa577cf:::benchmark_data_smithi171_78918_object129392:head# 2024-04-20T17:11:08.838 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aa6849f:::benchmark_data_smithi171_78918_object121019:head# 2024-04-20T17:11:08.838 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aa6d159:::benchmark_data_smithi171_78918_object115691:head# 2024-04-20T17:11:08.839 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aa6ddbc:::benchmark_data_smithi171_78918_object116991:head# 2024-04-20T17:11:08.839 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aa876ff:::benchmark_data_smithi171_78918_object82438:head# 2024-04-20T17:11:08.839 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aa89b23:::benchmark_data_smithi171_78918_object149592:head# 2024-04-20T17:11:08.839 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aa8b110:::benchmark_data_smithi171_78918_object53427:head# 2024-04-20T17:11:08.839 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aa91585:::benchmark_data_smithi171_78918_object54916:head# 2024-04-20T17:11:08.839 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aa9f5e4:::benchmark_data_smithi171_78918_object93793:head# 2024-04-20T17:11:08.839 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aaa09a2:::benchmark_data_smithi171_78918_object65975:head# 2024-04-20T17:11:08.839 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aab6aa8:::benchmark_data_smithi171_78918_object128666:head# 2024-04-20T17:11:08.839 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aab8f67:::benchmark_data_smithi171_78918_object94482:head# 2024-04-20T17:11:08.839 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aac1a66:::benchmark_data_smithi171_78918_object107525:head# 2024-04-20T17:11:08.839 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aac44f6:::benchmark_data_smithi171_78918_object108293:head# 2024-04-20T17:11:08.839 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aad619c:::benchmark_data_smithi171_78918_object104480:head# 2024-04-20T17:11:08.839 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aadb1af:::benchmark_data_smithi171_78918_object71823:head# 2024-04-20T17:11:08.839 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aadd6b6:::benchmark_data_smithi171_78918_object155449:head# 2024-04-20T17:11:08.840 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aae1491:::benchmark_data_smithi171_78918_object80247:head# 2024-04-20T17:11:08.840 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aaee729:::benchmark_data_smithi171_78918_object118011:head# 2024-04-20T17:11:08.840 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aaf0952:::benchmark_data_smithi171_78918_object96349:head# 2024-04-20T17:11:08.840 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aaf32b0:::benchmark_data_smithi171_78918_object158661:head# 2024-04-20T17:11:08.840 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ab05290:::benchmark_data_smithi171_78918_object72194:head# 2024-04-20T17:11:08.840 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ab09f20:::benchmark_data_smithi171_78918_object152191:head# 2024-04-20T17:11:08.840 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ab0b8b0:::benchmark_data_smithi171_78918_object85524:head# 2024-04-20T17:11:08.840 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ab142ea:::benchmark_data_smithi171_78918_object69320:head# 2024-04-20T17:11:08.840 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ab1d9f0:::benchmark_data_smithi171_78918_object111297:head# 2024-04-20T17:11:08.840 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ab20bc8:::benchmark_data_smithi171_78918_object121930:head# 2024-04-20T17:11:08.840 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ab23357:::benchmark_data_smithi171_78918_object135084:head# 2024-04-20T17:11:08.840 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ab271bc:::benchmark_data_smithi171_78918_object95482:head# 2024-04-20T17:11:08.840 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ab2af3b:::benchmark_data_smithi171_78918_object115090:head# 2024-04-20T17:11:08.840 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ab2c3cd:::benchmark_data_smithi171_78918_object107612:head# 2024-04-20T17:11:08.840 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ab3a2d7:::benchmark_data_smithi171_78918_object56454:head# 2024-04-20T17:11:08.840 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ab3e395:::benchmark_data_smithi171_78918_object156245:head# 2024-04-20T17:11:08.840 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ab48427:::benchmark_data_smithi171_78918_object158371:head# 2024-04-20T17:11:08.841 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ab48464:::benchmark_data_smithi171_78918_object149879:head# 2024-04-20T17:11:08.841 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ab5498b:::benchmark_data_smithi171_78918_object66194:head# 2024-04-20T17:11:08.841 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ab71906:::benchmark_data_smithi171_78918_object86768:head# 2024-04-20T17:11:08.841 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ab7b9c7:::benchmark_data_smithi171_78918_object69581:head# 2024-04-20T17:11:08.841 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ab89adc:::benchmark_data_smithi171_78918_object122583:head# 2024-04-20T17:11:08.841 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ab90e31:::benchmark_data_smithi171_78918_object109298:head# 2024-04-20T17:11:08.841 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aba5ebd:::benchmark_data_smithi171_78918_object118006:head# 2024-04-20T17:11:08.841 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aba8f61:::benchmark_data_smithi171_78918_object66761:head# 2024-04-20T17:11:08.841 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7abbe63b:::benchmark_data_smithi171_78918_object108506:head# 2024-04-20T17:11:08.841 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7abc344d:::benchmark_data_smithi171_78918_object141253:head# 2024-04-20T17:11:08.841 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7abd235c:::benchmark_data_smithi171_78918_object130472:head# 2024-04-20T17:11:08.841 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7abd8f8d:::benchmark_data_smithi171_78918_object88514:head# 2024-04-20T17:11:08.841 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7abdff38:::benchmark_data_smithi171_78918_object142041:head# 2024-04-20T17:11:08.841 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7abfd3f6:::benchmark_data_smithi171_78918_object123811:head# 2024-04-20T17:11:08.841 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7abfe683:::benchmark_data_smithi171_78918_object66933:head# 2024-04-20T17:11:08.842 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7abffe21:::benchmark_data_smithi171_78918_object144957:head# 2024-04-20T17:11:08.842 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ac05ac1:::benchmark_data_smithi171_78918_object69953:head# 2024-04-20T17:11:08.842 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ac1cc1e:::benchmark_data_smithi171_78918_object52919:head# 2024-04-20T17:11:08.842 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ac222c6:::benchmark_data_smithi171_78918_object112641:head# 2024-04-20T17:11:08.842 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ac23f51:::benchmark_data_smithi171_78918_object107696:head# 2024-04-20T17:11:08.842 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ac2a117:::benchmark_data_smithi171_78918_object91983:head# 2024-04-20T17:11:08.842 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ac2eda4:::benchmark_data_smithi171_78918_object65155:head# 2024-04-20T17:11:08.842 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ac2f744:::benchmark_data_smithi171_78918_object119588:head# 2024-04-20T17:11:08.842 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ac3490b:::benchmark_data_smithi171_78918_object144231:head# 2024-04-20T17:11:08.842 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ac3bb80:::benchmark_data_smithi171_78918_object150355:head# 2024-04-20T17:11:08.842 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ac40720:::benchmark_data_smithi171_78918_object84432:head# 2024-04-20T17:11:08.842 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ac499a8:::benchmark_data_smithi171_78918_object52683:head# 2024-04-20T17:11:08.842 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ac4b4e7:::benchmark_data_smithi171_78918_object72997:head# 2024-04-20T17:11:08.842 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ac4bbbd:::benchmark_data_smithi171_78918_object95599:head# 2024-04-20T17:11:08.843 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ac5668b:::benchmark_data_smithi171_78918_object148703:head# 2024-04-20T17:11:08.843 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ac567e1:::benchmark_data_smithi171_78918_object117585:head# 2024-04-20T17:11:08.843 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ac5ca8f:::benchmark_data_smithi171_78918_object131328:head# 2024-04-20T17:11:08.843 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ac6d533:::benchmark_data_smithi171_78918_object89371:head# 2024-04-20T17:11:08.843 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ac8b5f6:::benchmark_data_smithi171_78918_object113402:head# 2024-04-20T17:11:08.843 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ac900a8:::benchmark_data_smithi171_78918_object159733:head# 2024-04-20T17:11:08.843 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aca14d6:::benchmark_data_smithi171_78918_object143393:head# 2024-04-20T17:11:08.843 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aca2b1e:::benchmark_data_smithi171_78918_object149007:head# 2024-04-20T17:11:08.843 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aca7579:::benchmark_data_smithi171_78918_object120270:head# 2024-04-20T17:11:08.843 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7acaca8a:::benchmark_data_smithi171_78918_object61534:head# 2024-04-20T17:11:08.843 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7acae55f:::benchmark_data_smithi171_78918_object70273:head# 2024-04-20T17:11:08.843 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7acb7d43:::benchmark_data_smithi171_78918_object51691:head# 2024-04-20T17:11:08.843 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7accbad6:::benchmark_data_smithi171_78918_object145405:head# 2024-04-20T17:11:08.843 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7acd02ff:::benchmark_data_smithi171_78918_object102006:head# 2024-04-20T17:11:08.844 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7acda07c:::benchmark_data_smithi171_78918_object85846:head# 2024-04-20T17:11:08.844 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ace42b6:::benchmark_data_smithi171_78918_object160962:head# 2024-04-20T17:11:08.844 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7acee622:::benchmark_data_smithi171_78918_object128574:head# 2024-04-20T17:11:08.844 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7acf219d:::benchmark_data_smithi171_78918_object63907:head# 2024-04-20T17:11:08.844 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ad011e8:::benchmark_data_smithi171_78918_object72558:head# 2024-04-20T17:11:08.844 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ad0fc77:::benchmark_data_smithi171_78918_object98492:head# 2024-04-20T17:11:08.844 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ad10857:::benchmark_data_smithi171_78918_object56246:head# 2024-04-20T17:11:08.844 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ad39eef:::benchmark_data_smithi171_78918_object115553:head# 2024-04-20T17:11:08.844 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ad48999:::benchmark_data_smithi171_78918_object118381:head# 2024-04-20T17:11:08.844 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ad52017:::benchmark_data_smithi171_78918_object116187:head# 2024-04-20T17:11:08.844 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ad522d3:::benchmark_data_smithi171_78918_object91618:head# 2024-04-20T17:11:08.844 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ad5d0b9:::benchmark_data_smithi171_78918_object89931:head# 2024-04-20T17:11:08.844 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ad60172:::benchmark_data_smithi171_78918_object150295:head# 2024-04-20T17:11:08.844 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ad68ef8:::benchmark_data_smithi171_78918_object141341:head# 2024-04-20T17:11:08.845 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ad6c2a9:::benchmark_data_smithi171_78918_object56950:head# 2024-04-20T17:11:08.845 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ad7c157:::benchmark_data_smithi171_78918_object65516:head# 2024-04-20T17:11:08.845 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ad8498b:::benchmark_data_smithi171_78918_object136507:head# 2024-04-20T17:11:08.845 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ad8e5e7:::benchmark_data_smithi171_78918_object124721:head# 2024-04-20T17:11:08.845 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ad8eea1:::benchmark_data_smithi171_78918_object157147:head# 2024-04-20T17:11:08.845 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ad92563:::benchmark_data_smithi171_78918_object71495:head# 2024-04-20T17:11:08.845 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ad9d1f9:::benchmark_data_smithi171_78918_object138113:head# 2024-04-20T17:11:08.845 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ad9f734:::benchmark_data_smithi171_78918_object148079:head# 2024-04-20T17:11:08.845 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7adac2aa:::benchmark_data_smithi171_78918_object121569:head# 2024-04-20T17:11:08.845 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7adafd99:::benchmark_data_smithi171_78918_object132462:head# 2024-04-20T17:11:08.845 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7adb76a5:::benchmark_data_smithi171_78918_object137545:head# 2024-04-20T17:11:08.845 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7adbf09b:::benchmark_data_smithi171_78918_object91321:head# 2024-04-20T17:11:08.845 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7adc783f:::benchmark_data_smithi171_78918_object65117:head# 2024-04-20T17:11:08.845 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7add1096:::benchmark_data_smithi171_78918_object117452:head# 2024-04-20T17:11:08.845 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7add6918:::benchmark_data_smithi171_78918_object118741:head# 2024-04-20T17:11:08.846 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7addbd83:::benchmark_data_smithi171_78918_object91414:head# 2024-04-20T17:11:08.846 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7addef3a:::benchmark_data_smithi171_78918_object85601:head# 2024-04-20T17:11:08.846 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7adf9a8c:::benchmark_data_smithi171_78918_object105747:head# 2024-04-20T17:11:08.846 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ae01536:::benchmark_data_smithi171_78918_object87968:head# 2024-04-20T17:11:08.846 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ae05bd7:::benchmark_data_smithi171_78918_object55223:head# 2024-04-20T17:11:08.846 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ae0b897:::benchmark_data_smithi171_78918_object85640:head# 2024-04-20T17:11:08.846 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ae342c6:::benchmark_data_smithi171_78918_object82381:head# 2024-04-20T17:11:08.846 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ae49fa7:::benchmark_data_smithi171_78918_object52347:head# 2024-04-20T17:11:08.846 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ae53319:::benchmark_data_smithi171_78918_object93910:head# 2024-04-20T17:11:08.846 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ae7b7a2:::benchmark_data_smithi171_78918_object124395:head# 2024-04-20T17:11:08.846 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ae8655e:::benchmark_data_smithi171_78918_object110753:head# 2024-04-20T17:11:08.846 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ae9e58f:::benchmark_data_smithi171_78918_object92305:head# 2024-04-20T17:11:08.846 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aea5bba:::benchmark_data_smithi171_78918_object59172:head# 2024-04-20T17:11:08.846 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aea7d83:::benchmark_data_smithi171_78918_object113631:head# 2024-04-20T17:11:08.847 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aeb88bc:::benchmark_data_smithi171_78918_object150222:head# 2024-04-20T17:11:08.847 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aeb8aae:::benchmark_data_smithi171_78918_object104113:head# 2024-04-20T17:11:08.847 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:11:08.861 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aeba70c:::benchmark_data_smithi171_78918_object84946:head# 2024-04-20T17:11:08.861 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aebbc12:::benchmark_data_smithi171_78918_object160946:head# 2024-04-20T17:11:08.861 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aec20f2:::benchmark_data_smithi171_78918_object54891:head# 2024-04-20T17:11:08.861 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aec2502:::benchmark_data_smithi171_78918_object123237:head# 2024-04-20T17:11:08.861 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aed4aa9:::benchmark_data_smithi171_78918_object146290:head# 2024-04-20T17:11:08.861 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aedc8a7:::benchmark_data_smithi171_78918_object97535:head# 2024-04-20T17:11:08.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aee27ef:::benchmark_data_smithi171_78918_object140909:head# 2024-04-20T17:11:08.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aee9cac:::benchmark_data_smithi171_78918_object155943:head# 2024-04-20T17:11:08.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aef60fe:::benchmark_data_smithi171_78918_object100168:head# 2024-04-20T17:11:08.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aef8a01:::benchmark_data_smithi171_78918_object97257:head# 2024-04-20T17:11:08.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7af1e9d9:::benchmark_data_smithi171_78918_object105806:head# 2024-04-20T17:11:08.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7af2d54f:::benchmark_data_smithi171_78918_object130836:head# 2024-04-20T17:11:08.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7af2d56a:::benchmark_data_smithi171_78918_object107060:head# 2024-04-20T17:11:08.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7af36879:::benchmark_data_smithi171_78918_object106158:head# 2024-04-20T17:11:08.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7af4b680:::benchmark_data_smithi171_78918_object124519:head# 2024-04-20T17:11:08.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7af4fb80:::benchmark_data_smithi171_78918_object103194:head# 2024-04-20T17:11:08.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7af5628b:::benchmark_data_smithi171_78918_object122577:head# 2024-04-20T17:11:08.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7af5b86a:::benchmark_data_smithi171_78918_object111088:head# 2024-04-20T17:11:08.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7af5cbc5:::benchmark_data_smithi171_78918_object113995:head# 2024-04-20T17:11:08.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7af68260:::benchmark_data_smithi171_78918_object87774:head# 2024-04-20T17:11:08.862 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7af6c8b8:::benchmark_data_smithi171_78918_object97870:head# 2024-04-20T17:11:08.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7af7e061:::benchmark_data_smithi171_78918_object72367:head# 2024-04-20T17:11:08.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7af90d6a:::benchmark_data_smithi171_78918_object142915:head# 2024-04-20T17:11:08.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7af93362:::benchmark_data_smithi171_78918_object148894:head# 2024-04-20T17:11:08.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7af9485a:::benchmark_data_smithi171_78918_object158742:head# 2024-04-20T17:11:08.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7af9fbe8:::benchmark_data_smithi171_78918_object60174:head# 2024-04-20T17:11:08.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7afa0b9a:::benchmark_data_smithi171_78918_object92981:head# 2024-04-20T17:11:08.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7afa2972:::benchmark_data_smithi171_78918_object92435:head# 2024-04-20T17:11:08.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7afb5c10:::benchmark_data_smithi171_78918_object131259:head# 2024-04-20T17:11:08.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7afb8024:::benchmark_data_smithi171_78918_object142912:head# 2024-04-20T17:11:08.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7afbc658:::benchmark_data_smithi171_78918_object52449:head# 2024-04-20T17:11:08.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7afbcab4:::benchmark_data_smithi171_78918_object56559:head# 2024-04-20T17:11:08.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7afc0d63:::benchmark_data_smithi171_78918_object72189:head# 2024-04-20T17:11:08.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7afc13c5:::benchmark_data_smithi171_78918_object131176:head# 2024-04-20T17:11:08.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7afc5966:::benchmark_data_smithi171_78918_object148071:head# 2024-04-20T17:11:08.863 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7afcb98b:::benchmark_data_smithi171_78918_object136107:head# 2024-04-20T17:11:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7afdac7b:::benchmark_data_smithi171_78918_object105547:head# 2024-04-20T17:11:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7afe01cc:::benchmark_data_smithi171_78918_object64830:head# 2024-04-20T17:11:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7afeb688:::benchmark_data_smithi171_78918_object155071:head# 2024-04-20T17:11:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aff0624:::benchmark_data_smithi171_78918_object128495:head# 2024-04-20T17:11:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aff47d3:::benchmark_data_smithi171_78918_object97460:head# 2024-04-20T17:11:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7aff7137:::benchmark_data_smithi171_78918_object89002:head# 2024-04-20T17:11:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b00c6e5:::benchmark_data_smithi171_78918_object160160:head# 2024-04-20T17:11:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b014390:::benchmark_data_smithi171_78918_object125447:head# 2024-04-20T17:11:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b021c17:::benchmark_data_smithi171_78918_object135839:head# 2024-04-20T17:11:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b0336fc:::benchmark_data_smithi171_78918_object122099:head# 2024-04-20T17:11:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b0371e2:::benchmark_data_smithi171_78918_object58688:head# 2024-04-20T17:11:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b048462:::benchmark_data_smithi171_78918_object110878:head# 2024-04-20T17:11:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b057579:::benchmark_data_smithi171_78918_object109506:head# 2024-04-20T17:11:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b060c66:::benchmark_data_smithi171_78918_object65278:head# 2024-04-20T17:11:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b06a271:::benchmark_data_smithi171_78918_object85504:head# 2024-04-20T17:11:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b07b1d7:::benchmark_data_smithi171_78918_object150911:head# 2024-04-20T17:11:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b07c291:::benchmark_data_smithi171_78918_object67723:head# 2024-04-20T17:11:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b084db3:::benchmark_data_smithi171_78918_object91770:head# 2024-04-20T17:11:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b090426:::benchmark_data_smithi171_78918_object56965:head# 2024-04-20T17:11:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b09ad6c:::benchmark_data_smithi171_78918_object123741:head# 2024-04-20T17:11:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b0aef6e:::benchmark_data_smithi171_78918_object68865:head# 2024-04-20T17:11:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b0b4e14:::benchmark_data_smithi171_78918_object141009:head# 2024-04-20T17:11:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b0b8a24:::benchmark_data_smithi171_78918_object73457:head# 2024-04-20T17:11:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b0c23d2:::benchmark_data_smithi171_78918_object55653:head# 2024-04-20T17:11:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b0c7240:::benchmark_data_smithi171_78918_object71648:head# 2024-04-20T17:11:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b0d0a7b:::benchmark_data_smithi171_78918_object115539:head# 2024-04-20T17:11:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b0d5115:::benchmark_data_smithi171_78918_object58765:head# 2024-04-20T17:11:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b0e0ccb:::benchmark_data_smithi171_78918_object72137:head# 2024-04-20T17:11:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b0e344b:::benchmark_data_smithi171_78918_object111680:head# 2024-04-20T17:11:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b0efc66:::benchmark_data_smithi171_78918_object63152:head# 2024-04-20T17:11:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b0f1246:::benchmark_data_smithi171_78918_object86713:head# 2024-04-20T17:11:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b0f44ee:::benchmark_data_smithi171_78918_object157199:head# 2024-04-20T17:11:08.866 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b0f7760:::benchmark_data_smithi171_78918_object55337:head# 2024-04-20T17:11:08.866 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b0fc0ec:::benchmark_data_smithi171_78918_object64599:head# 2024-04-20T17:11:08.866 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b0fc64f:::benchmark_data_smithi171_78918_object135580:head# 2024-04-20T17:11:08.866 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b1011cf:::benchmark_data_smithi171_78918_object52645:head# 2024-04-20T17:11:08.866 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b10ee97:::benchmark_data_smithi171_78918_object130833:head# 2024-04-20T17:11:08.866 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b124a03:::benchmark_data_smithi171_78918_object104347:head# 2024-04-20T17:11:08.866 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b131d79:::benchmark_data_smithi171_78918_object147540:head# 2024-04-20T17:11:08.866 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b152382:::benchmark_data_smithi171_78918_object96620:head# 2024-04-20T17:11:08.866 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b154a1d:::benchmark_data_smithi171_78918_object77828:head# 2024-04-20T17:11:08.866 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b15c3ab:::benchmark_data_smithi171_78918_object58748:head# 2024-04-20T17:11:08.866 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b16d68e:::benchmark_data_smithi171_78918_object153298:head# 2024-04-20T17:11:08.866 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b179871:::benchmark_data_smithi171_78918_object53293:head# 2024-04-20T17:11:08.866 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b17d82e:::benchmark_data_smithi171_78918_object132826:head# 2024-04-20T17:11:08.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b1893dc:::benchmark_data_smithi171_78918_object151662:head# 2024-04-20T17:11:08.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b18eeb1:::benchmark_data_smithi171_78918_object119170:head# 2024-04-20T17:11:08.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b1a1503:::benchmark_data_smithi171_78918_object62236:head# 2024-04-20T17:11:08.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b1a3354:::benchmark_data_smithi171_78918_object130459:head# 2024-04-20T17:11:08.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b1a9cc4:::benchmark_data_smithi171_78918_object146175:head# 2024-04-20T17:11:08.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b1adb3c:::benchmark_data_smithi171_78918_object132091:head# 2024-04-20T17:11:08.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b1ae31a:::benchmark_data_smithi171_78918_object125621:head# 2024-04-20T17:11:08.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b1b1c04:::benchmark_data_smithi171_78918_object117462:head# 2024-04-20T17:11:08.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b1c2e93:::benchmark_data_smithi171_78918_object145315:head# 2024-04-20T17:11:08.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b1c8b17:::benchmark_data_smithi171_78918_object142749:head# 2024-04-20T17:11:08.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b1cb753:::benchmark_data_smithi171_78918_object126638:head# 2024-04-20T17:11:08.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b1d572b:::benchmark_data_smithi171_78918_object116895:head# 2024-04-20T17:11:08.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b1e12b7:::benchmark_data_smithi171_78918_object83700:head# 2024-04-20T17:11:08.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b1e5211:::benchmark_data_smithi171_78918_object92620:head# 2024-04-20T17:11:08.867 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b1e58e7:::benchmark_data_smithi171_78918_object126646:head# 2024-04-20T17:11:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b1f33e9:::benchmark_data_smithi171_78918_object136506:head# 2024-04-20T17:11:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b1f81ca:::benchmark_data_smithi171_78918_object54555:head# 2024-04-20T17:11:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b202567:::benchmark_data_smithi171_78918_object73815:head# 2024-04-20T17:11:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b207270:::benchmark_data_smithi171_78918_object60941:head# 2024-04-20T17:11:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b20aa54:::benchmark_data_smithi171_78918_object62351:head# 2024-04-20T17:11:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b21d15c:::benchmark_data_smithi171_78918_object128005:head# 2024-04-20T17:11:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b21d42a:::benchmark_data_smithi171_78918_object123521:head# 2024-04-20T17:11:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b223e63:::benchmark_data_smithi171_78918_object120073:head# 2024-04-20T17:11:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b22b72f:::benchmark_data_smithi171_78918_object72550:head# 2024-04-20T17:11:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b2366f2:::benchmark_data_smithi171_78918_object107053:head# 2024-04-20T17:11:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b248d88:::benchmark_data_smithi171_78918_object150846:head# 2024-04-20T17:11:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b253ec8:::benchmark_data_smithi171_78918_object146014:head# 2024-04-20T17:11:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b25acfa:::benchmark_data_smithi171_78918_object85042:head# 2024-04-20T17:11:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b270f9d:::benchmark_data_smithi171_78918_object125648:head# 2024-04-20T17:11:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b27378b:::benchmark_data_smithi171_78918_object57934:head# 2024-04-20T17:11:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b27b465:::benchmark_data_smithi171_78918_object106570:head# 2024-04-20T17:11:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b27dba8:::benchmark_data_smithi171_78918_object82008:head# 2024-04-20T17:11:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b28c72a:::benchmark_data_smithi171_78918_object79509:head# 2024-04-20T17:11:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b290c8f:::benchmark_data_smithi171_78918_object121055:head# 2024-04-20T17:11:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b29d2ac:::benchmark_data_smithi171_78918_object98402:head# 2024-04-20T17:11:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b29fc3d:::benchmark_data_smithi171_78918_object73254:head# 2024-04-20T17:11:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b2c54f6:::benchmark_data_smithi171_78918_object113846:head# 2024-04-20T17:11:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b2c8cba:::benchmark_data_smithi171_78918_object155864:head# 2024-04-20T17:11:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b2d02af:::benchmark_data_smithi171_78918_object82247:head# 2024-04-20T17:11:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b2e5e5d:::benchmark_data_smithi171_78918_object85124:head# 2024-04-20T17:11:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b2e7ea8:::benchmark_data_smithi171_78918_object144159:head# 2024-04-20T17:11:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b2efe68:::benchmark_data_smithi171_78918_object69951:head# 2024-04-20T17:11:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b2f3ee4:::benchmark_data_smithi171_78918_object152571:head# 2024-04-20T17:11:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b2fd2b6:::benchmark_data_smithi171_78918_object59127:head# 2024-04-20T17:11:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b301a8f:::benchmark_data_smithi171_78918_object69233:head# 2024-04-20T17:11:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b3057d4:::benchmark_data_smithi171_78918_object80195:head# 2024-04-20T17:11:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b305b78:::benchmark_data_smithi171_78918_object141272:head# 2024-04-20T17:11:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b30982c:::benchmark_data_smithi171_78918_object82694:head# 2024-04-20T17:11:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b30e89f:::benchmark_data_smithi171_78918_object92677:head# 2024-04-20T17:11:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b3232ea:::benchmark_data_smithi171_78918_object100152:head# 2024-04-20T17:11:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b33c771:::benchmark_data_smithi171_78918_object59030:head# 2024-04-20T17:11:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b33cf93:::benchmark_data_smithi171_78918_object114036:head# 2024-04-20T17:11:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b341e8f:::benchmark_data_smithi171_78918_object104219:head# 2024-04-20T17:11:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b345ee4:::benchmark_data_smithi171_78918_object113992:head# 2024-04-20T17:11:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b346716:::benchmark_data_smithi171_78918_object66639:head# 2024-04-20T17:11:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b36038a:::benchmark_data_smithi171_78918_object109694:head# 2024-04-20T17:11:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b364202:::benchmark_data_smithi171_78918_object87461:head# 2024-04-20T17:11:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b367c3f:::benchmark_data_smithi171_78918_object56574:head# 2024-04-20T17:11:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b37cf86:::benchmark_data_smithi171_78918_object153681:head# 2024-04-20T17:11:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b380aa5:::benchmark_data_smithi171_78918_object101808:head# 2024-04-20T17:11:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b3869e3:::benchmark_data_smithi171_78918_object83522:head# 2024-04-20T17:11:08.871 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b396f8c:::benchmark_data_smithi171_78918_object72882:head# 2024-04-20T17:11:08.871 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b3a350f:::benchmark_data_smithi171_78918_object111246:head# 2024-04-20T17:11:08.871 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b3a4309:::benchmark_data_smithi171_78918_object87343:head# 2024-04-20T17:11:08.871 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b3ac102:::benchmark_data_smithi171_78918_object122883:head# 2024-04-20T17:11:08.871 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b3afcde:::benchmark_data_smithi171_78918_object72382:head# 2024-04-20T17:11:08.871 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b3bea25:::benchmark_data_smithi171_78918_object78544:head# 2024-04-20T17:11:08.871 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b3c01e6:::benchmark_data_smithi171_78918_object58839:head# 2024-04-20T17:11:08.871 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b3cd138:::benchmark_data_smithi171_78918_object70954:head# 2024-04-20T17:11:08.871 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b3cf622:::benchmark_data_smithi171_78918_object158924:head# 2024-04-20T17:11:08.871 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:11:08.884 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b3d2db6:::benchmark_data_smithi171_78918_object154938:head# 2024-04-20T17:11:08.884 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b3d8341:::benchmark_data_smithi171_78918_object119940:head# 2024-04-20T17:11:08.884 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b3f4880:::benchmark_data_smithi171_78918_object138220:head# 2024-04-20T17:11:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b40262e:::benchmark_data_smithi171_78918_object152305:head# 2024-04-20T17:11:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b4041ba:::benchmark_data_smithi171_78918_object89688:head# 2024-04-20T17:11:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b406719:::benchmark_data_smithi171_78918_object117270:head# 2024-04-20T17:11:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b40a0d5:::benchmark_data_smithi171_78918_object106090:head# 2024-04-20T17:11:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b414c59:::benchmark_data_smithi171_78918_object66478:head# 2024-04-20T17:11:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b42b794:::benchmark_data_smithi171_78918_object134334:head# 2024-04-20T17:11:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b43eed7:::benchmark_data_smithi171_78918_object77773:head# 2024-04-20T17:11:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b44cf40:::benchmark_data_smithi171_78918_object61637:head# 2024-04-20T17:11:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b457932:::benchmark_data_smithi171_78918_object53762:head# 2024-04-20T17:11:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b45e59c:::benchmark_data_smithi171_78918_object106218:head# 2024-04-20T17:11:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b45ff63:::benchmark_data_smithi171_78918_object94649:head# 2024-04-20T17:11:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b4768eb:::benchmark_data_smithi171_78918_object92297:head# 2024-04-20T17:11:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b49ffe7:::benchmark_data_smithi171_78918_object75003:head# 2024-04-20T17:11:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b4a268a:::benchmark_data_smithi171_78918_object66972:head# 2024-04-20T17:11:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b4a66e9:::benchmark_data_smithi171_78918_object156106:head# 2024-04-20T17:11:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b4ae93e:::benchmark_data_smithi171_78918_object69613:head# 2024-04-20T17:11:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b4b6744:::benchmark_data_smithi171_78918_object110059:head# 2024-04-20T17:11:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b4b7b4b:::benchmark_data_smithi171_78918_object146837:head# 2024-04-20T17:11:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b4b942e:::benchmark_data_smithi171_78918_object54675:head# 2024-04-20T17:11:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b4bb9c1:::benchmark_data_smithi171_78918_object113272:head# 2024-04-20T17:11:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b4c6033:::benchmark_data_smithi171_78918_object120699:head# 2024-04-20T17:11:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b4c9d9b:::benchmark_data_smithi171_78918_object128292:head# 2024-04-20T17:11:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b4ce76d:::benchmark_data_smithi171_78918_object157042:head# 2024-04-20T17:11:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b4d425b:::benchmark_data_smithi171_78918_object145011:head# 2024-04-20T17:11:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b4da1b8:::benchmark_data_smithi171_78918_object121637:head# 2024-04-20T17:11:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b4dbf4a:::benchmark_data_smithi171_78918_object151072:head# 2024-04-20T17:11:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b4f0cbb:::benchmark_data_smithi171_78918_object75030:head# 2024-04-20T17:11:08.887 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b4f29ee:::benchmark_data_smithi171_78918_object58668:head# 2024-04-20T17:11:08.887 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b4f2f6e:::benchmark_data_smithi171_78918_object63632:head# 2024-04-20T17:11:08.887 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b4fedea:::benchmark_data_smithi171_78918_object127940:head# 2024-04-20T17:11:08.887 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b4ff736:::benchmark_data_smithi171_78918_object136357:head# 2024-04-20T17:11:08.887 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b509de8:::benchmark_data_smithi171_78918_object154807:head# 2024-04-20T17:11:08.887 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b5136d1:::benchmark_data_smithi171_78918_object53090:head# 2024-04-20T17:11:08.887 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b51ef18:::benchmark_data_smithi171_78918_object53808:head# 2024-04-20T17:11:08.887 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b52567d:::benchmark_data_smithi171_78918_object118425:head# 2024-04-20T17:11:08.887 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b52b48d:::benchmark_data_smithi171_78918_object71185:head# 2024-04-20T17:11:08.887 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b52ff8a:::benchmark_data_smithi171_78918_object76142:head# 2024-04-20T17:11:08.887 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b536387:::benchmark_data_smithi171_78918_object153452:head# 2024-04-20T17:11:08.887 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b539385:::benchmark_data_smithi171_78918_object99345:head# 2024-04-20T17:11:08.887 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b541eeb:::benchmark_data_smithi171_78918_object112596:head# 2024-04-20T17:11:08.887 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b541f8c:::benchmark_data_smithi171_78918_object100213:head# 2024-04-20T17:11:08.887 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b557d55:::benchmark_data_smithi171_78918_object157103:head# 2024-04-20T17:11:08.888 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b559c1c:::benchmark_data_smithi171_78918_object81746:head# 2024-04-20T17:11:08.888 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b56b25b:::benchmark_data_smithi171_78918_object109604:head# 2024-04-20T17:11:08.888 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b56dab9:::benchmark_data_smithi171_78918_object70014:head# 2024-04-20T17:11:08.888 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b57426b:::benchmark_data_smithi171_78918_object125872:head# 2024-04-20T17:11:08.888 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b575c71:::benchmark_data_smithi171_78918_object92548:head# 2024-04-20T17:11:08.888 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b575ef9:::benchmark_data_smithi171_78918_object116790:head# 2024-04-20T17:11:08.888 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b586a3f:::benchmark_data_smithi171_78918_object91985:head# 2024-04-20T17:11:08.888 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b588b3f:::benchmark_data_smithi171_78918_object102881:head# 2024-04-20T17:11:08.888 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b5aa760:::benchmark_data_smithi171_78918_object72516:head# 2024-04-20T17:11:08.888 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b5b1c03:::benchmark_data_smithi171_78918_object92856:head# 2024-04-20T17:11:08.888 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b5b4005:::benchmark_data_smithi171_78918_object133519:head# 2024-04-20T17:11:08.888 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b5d6c2b:::benchmark_data_smithi171_78918_object127895:head# 2024-04-20T17:11:08.888 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b5e16a8:::benchmark_data_smithi171_78918_object113352:head# 2024-04-20T17:11:08.888 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b5f5ddf:::benchmark_data_smithi171_78918_object107410:head# 2024-04-20T17:11:08.888 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b605572:::benchmark_data_smithi171_78918_object96692:head# 2024-04-20T17:11:08.889 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b611d50:::benchmark_data_smithi171_78918_object142131:head# 2024-04-20T17:11:08.889 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b6134c3:::benchmark_data_smithi171_78918_object78813:head# 2024-04-20T17:11:08.889 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b6200bf:::benchmark_data_smithi171_78918_object61742:head# 2024-04-20T17:11:08.889 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b624990:::benchmark_data_smithi171_78918_object145374:head# 2024-04-20T17:11:08.889 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b6380d7:::benchmark_data_smithi171_78918_object73235:head# 2024-04-20T17:11:08.889 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b639c33:::benchmark_data_smithi171_78918_object61250:head# 2024-04-20T17:11:08.889 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b63a62e:::benchmark_data_smithi171_78918_object154487:head# 2024-04-20T17:11:08.889 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b6419d4:::benchmark_data_smithi171_78918_object152996:head# 2024-04-20T17:11:08.889 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b64ea58:::benchmark_data_smithi171_78918_object128817:head# 2024-04-20T17:11:08.889 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b64fbab:::benchmark_data_smithi171_78918_object86430:head# 2024-04-20T17:11:08.889 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b652ea2:::benchmark_data_smithi171_78918_object98716:head# 2024-04-20T17:11:08.889 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b656efb:::benchmark_data_smithi171_78918_object76625:head# 2024-04-20T17:11:08.889 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b65a7bc:::benchmark_data_smithi171_78918_object103654:head# 2024-04-20T17:11:08.889 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b65acf7:::benchmark_data_smithi171_78918_object65897:head# 2024-04-20T17:11:08.890 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b65d1f6:::benchmark_data_smithi171_78918_object59711:head# 2024-04-20T17:11:08.890 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b65faa7:::benchmark_data_smithi171_78918_object118998:head# 2024-04-20T17:11:08.890 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b66c7f6:::benchmark_data_smithi171_78918_object57969:head# 2024-04-20T17:11:08.890 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b683c8d:::benchmark_data_smithi171_78918_object131520:head# 2024-04-20T17:11:08.890 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b684c45:::benchmark_data_smithi171_78918_object90510:head# 2024-04-20T17:11:08.890 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b68a005:::benchmark_data_smithi171_78918_object72655:head# 2024-04-20T17:11:08.890 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b695315:::benchmark_data_smithi171_78918_object138798:head# 2024-04-20T17:11:08.890 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b695c71:::benchmark_data_smithi171_78918_object118492:head# 2024-04-20T17:11:08.890 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b6a5277:::benchmark_data_smithi171_78918_object140144:head# 2024-04-20T17:11:08.890 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b6a8594:::benchmark_data_smithi171_78918_object138485:head# 2024-04-20T17:11:08.890 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b6aa37c:::benchmark_data_smithi171_78918_object128372:head# 2024-04-20T17:11:08.890 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b6ae8be:::benchmark_data_smithi171_78918_object133091:head# 2024-04-20T17:11:08.890 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b6c2c97:::benchmark_data_smithi171_78918_object141556:head# 2024-04-20T17:11:08.890 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b6d4ef3:::benchmark_data_smithi171_78918_object96011:head# 2024-04-20T17:11:08.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b6d7b04:::benchmark_data_smithi171_78918_object95781:head# 2024-04-20T17:11:08.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b6db674:::benchmark_data_smithi171_78918_object95789:head# 2024-04-20T17:11:08.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b6e5c84:::benchmark_data_smithi171_78918_object105744:head# 2024-04-20T17:11:08.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b6e73a0:::benchmark_data_smithi171_78918_object82898:head# 2024-04-20T17:11:08.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b6eefa4:::benchmark_data_smithi171_78918_object124363:head# 2024-04-20T17:11:08.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b6f8f6b:::benchmark_data_smithi171_78918_object90906:head# 2024-04-20T17:11:08.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b70d886:::benchmark_data_smithi171_78918_object53415:head# 2024-04-20T17:11:08.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b71b3dd:::benchmark_data_smithi171_78918_object79571:head# 2024-04-20T17:11:08.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b71b43b:::benchmark_data_smithi171_78918_object53133:head# 2024-04-20T17:11:08.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b72447c:::benchmark_data_smithi171_78918_object121110:head# 2024-04-20T17:11:08.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b724906:::benchmark_data_smithi171_78918_object71054:head# 2024-04-20T17:11:08.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b72acce:::benchmark_data_smithi171_78918_object138533:head# 2024-04-20T17:11:08.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b72b600:::benchmark_data_smithi171_78918_object78448:head# 2024-04-20T17:11:08.891 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b72d599:::benchmark_data_smithi171_78918_object89805:head# 2024-04-20T17:11:08.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b73496f:::benchmark_data_smithi171_78918_object135761:head# 2024-04-20T17:11:08.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b74b8e6:::benchmark_data_smithi171_78918_object139799:head# 2024-04-20T17:11:08.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b74ed13:::benchmark_data_smithi171_78918_object84477:head# 2024-04-20T17:11:08.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b75869f:::benchmark_data_smithi171_78918_object113990:head# 2024-04-20T17:11:08.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b75aa7b:::benchmark_data_smithi171_78918_object107363:head# 2024-04-20T17:11:08.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b761c39:::benchmark_data_smithi171_78918_object107932:head# 2024-04-20T17:11:08.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b777040:::benchmark_data_smithi171_78918_object98361:head# 2024-04-20T17:11:08.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b77fedf:::benchmark_data_smithi171_78918_object96964:head# 2024-04-20T17:11:08.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b78d097:::benchmark_data_smithi171_78918_object147184:head# 2024-04-20T17:11:08.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b78de25:::benchmark_data_smithi171_78918_object108514:head# 2024-04-20T17:11:08.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b796400:::benchmark_data_smithi171_78918_object107837:head# 2024-04-20T17:11:08.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b799d38:::benchmark_data_smithi171_78918_object133111:head# 2024-04-20T17:11:08.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b7a95b6:::benchmark_data_smithi171_78918_object141068:head# 2024-04-20T17:11:08.892 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b7b8715:::benchmark_data_smithi171_78918_object110662:head# 2024-04-20T17:11:08.893 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b7c0e45:::benchmark_data_smithi171_78918_object153022:head# 2024-04-20T17:11:08.893 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b7c593b:::benchmark_data_smithi171_78918_object146785:head# 2024-04-20T17:11:08.893 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b7c5fd8:::benchmark_data_smithi171_78918_object107235:head# 2024-04-20T17:11:08.893 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b7c6bb1:::benchmark_data_smithi171_78918_object119176:head# 2024-04-20T17:11:08.893 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b7c96af:::benchmark_data_smithi171_78918_object123269:head# 2024-04-20T17:11:08.893 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b7d2f81:::benchmark_data_smithi171_78918_object145470:head# 2024-04-20T17:11:08.893 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b7ea781:::benchmark_data_smithi171_78918_object106529:head# 2024-04-20T17:11:08.893 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b800807:::benchmark_data_smithi171_78918_object123152:head# 2024-04-20T17:11:08.893 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b806852:::benchmark_data_smithi171_78918_object99126:head# 2024-04-20T17:11:08.893 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b80aec5:::benchmark_data_smithi171_78918_object75175:head# 2024-04-20T17:11:08.893 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b8112a1:::benchmark_data_smithi171_78918_object64834:head# 2024-04-20T17:11:08.893 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b8136be:::benchmark_data_smithi171_78918_object80332:head# 2024-04-20T17:11:08.893 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b81aa57:::benchmark_data_smithi171_78918_object119340:head# 2024-04-20T17:11:08.894 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b841ff7:::benchmark_data_smithi171_78918_object125609:head# 2024-04-20T17:11:08.894 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b843c2d:::benchmark_data_smithi171_78918_object131931:head# 2024-04-20T17:11:08.894 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b848cb7:::benchmark_data_smithi171_78918_object82835:head# 2024-04-20T17:11:08.894 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b84ca8d:::benchmark_data_smithi171_78918_object78402:head# 2024-04-20T17:11:08.894 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b855709:::benchmark_data_smithi171_78918_object154462:head# 2024-04-20T17:11:08.894 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b85744d:::benchmark_data_smithi171_78918_object126670:head# 2024-04-20T17:11:08.894 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b8599fb:::benchmark_data_smithi171_78918_object109553:head# 2024-04-20T17:11:08.894 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b866601:::benchmark_data_smithi171_78918_object62317:head# 2024-04-20T17:11:08.894 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:11:08.909 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b868688:::benchmark_data_smithi171_78918_object77162:head# 2024-04-20T17:11:08.909 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b870337:::benchmark_data_smithi171_78918_object72351:head# 2024-04-20T17:11:08.909 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b8740d3:::benchmark_data_smithi171_78918_object61639:head# 2024-04-20T17:11:08.909 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b874416:::benchmark_data_smithi171_78918_object91940:head# 2024-04-20T17:11:08.909 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b884f90:::benchmark_data_smithi171_78918_object153979:head# 2024-04-20T17:11:08.909 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b897884:::benchmark_data_smithi171_78918_object61766:head# 2024-04-20T17:11:08.909 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b8ac232:::benchmark_data_smithi171_78918_object78142:head# 2024-04-20T17:11:08.909 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b8ba401:::benchmark_data_smithi171_78918_object128168:head# 2024-04-20T17:11:08.909 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b8bd2b8:::benchmark_data_smithi171_78918_object82038:head# 2024-04-20T17:11:08.909 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b8ca35b:::benchmark_data_smithi171_78918_object136315:head# 2024-04-20T17:11:08.909 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b8ceec3:::benchmark_data_smithi171_78918_object117317:head# 2024-04-20T17:11:08.909 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b8d471d:::benchmark_data_smithi171_78918_object85334:head# 2024-04-20T17:11:08.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b8dabc8:::benchmark_data_smithi171_78918_object76067:head# 2024-04-20T17:11:08.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b8f690a:::benchmark_data_smithi171_78918_object66838:head# 2024-04-20T17:11:08.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b8f71c4:::benchmark_data_smithi171_78918_object117163:head# 2024-04-20T17:11:08.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b8fc56e:::benchmark_data_smithi171_78918_object108704:head# 2024-04-20T17:11:08.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b8fe53c:::benchmark_data_smithi171_78918_object64345:head# 2024-04-20T17:11:08.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b90b4d4:::benchmark_data_smithi171_78918_object97888:head# 2024-04-20T17:11:08.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b90cbe5:::benchmark_data_smithi171_78918_object86239:head# 2024-04-20T17:11:08.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b9183ed:::benchmark_data_smithi171_78918_object103375:head# 2024-04-20T17:11:08.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b91abfb:::benchmark_data_smithi171_78918_object98930:head# 2024-04-20T17:11:08.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b91ebc7:::benchmark_data_smithi171_78918_object144560:head# 2024-04-20T17:11:08.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b926be1:::benchmark_data_smithi171_78918_object108057:head# 2024-04-20T17:11:08.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b926d23:::benchmark_data_smithi171_78918_object70583:head# 2024-04-20T17:11:08.910 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b928f79:::benchmark_data_smithi171_78918_object135011:head# 2024-04-20T17:11:08.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b94f24b:::benchmark_data_smithi171_78918_object152336:head# 2024-04-20T17:11:08.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b956d53:::benchmark_data_smithi171_78918_object71530:head# 2024-04-20T17:11:08.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b963365:::benchmark_data_smithi171_78918_object70051:head# 2024-04-20T17:11:08.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b9645ae:::benchmark_data_smithi171_78918_object113672:head# 2024-04-20T17:11:08.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b964a66:::benchmark_data_smithi171_78918_object94103:head# 2024-04-20T17:11:08.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b96c800:::benchmark_data_smithi171_78918_object53533:head# 2024-04-20T17:11:08.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b9761d1:::benchmark_data_smithi171_78918_object140684:head# 2024-04-20T17:11:08.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b976e55:::benchmark_data_smithi171_78918_object75225:head# 2024-04-20T17:11:08.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b986905:::benchmark_data_smithi171_78918_object151180:head# 2024-04-20T17:11:08.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b98fc3e:::benchmark_data_smithi171_78918_object119918:head# 2024-04-20T17:11:08.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b994694:::benchmark_data_smithi171_78918_object139274:head# 2024-04-20T17:11:08.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b999d41:::benchmark_data_smithi171_78918_object94130:head# 2024-04-20T17:11:08.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b99aff5:::benchmark_data_smithi171_78918_object128324:head# 2024-04-20T17:11:08.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b99e81a:::benchmark_data_smithi171_78918_object146338:head# 2024-04-20T17:11:08.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b9a1969:::benchmark_data_smithi171_78918_object101866:head# 2024-04-20T17:11:08.911 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b9a338d:::benchmark_data_smithi171_78918_object144065:head# 2024-04-20T17:11:08.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b9b11bd:::benchmark_data_smithi171_78918_object98196:head# 2024-04-20T17:11:08.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b9b43a6:::benchmark_data_smithi171_78918_object80335:head# 2024-04-20T17:11:08.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b9b6a1c:::benchmark_data_smithi171_78918_object58584:head# 2024-04-20T17:11:08.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b9c0883:::benchmark_data_smithi171_78918_object135270:head# 2024-04-20T17:11:08.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b9c35e3:::benchmark_data_smithi171_78918_object158030:head# 2024-04-20T17:11:08.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b9d0972:::benchmark_data_smithi171_78918_object113756:head# 2024-04-20T17:11:08.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b9d82bb:::benchmark_data_smithi171_78918_object126822:head# 2024-04-20T17:11:08.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b9dbd6f:::benchmark_data_smithi171_78918_object125297:head# 2024-04-20T17:11:08.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b9de85d:::benchmark_data_smithi171_78918_object159737:head# 2024-04-20T17:11:08.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b9f03fd:::benchmark_data_smithi171_78918_object76034:head# 2024-04-20T17:11:08.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b9f11be:::benchmark_data_smithi171_78918_object76606:head# 2024-04-20T17:11:08.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b9f5957:::benchmark_data_smithi171_78918_object102908:head# 2024-04-20T17:11:08.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7b9fcadf:::benchmark_data_smithi171_78918_object125543:head# 2024-04-20T17:11:08.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ba03753:::benchmark_data_smithi171_78918_object95446:head# 2024-04-20T17:11:08.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ba0af9a:::benchmark_data_smithi171_78918_object94908:head# 2024-04-20T17:11:08.912 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ba0d4fd:::benchmark_data_smithi171_78918_object118791:head# 2024-04-20T17:11:08.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ba0da9c:::benchmark_data_smithi171_78918_object120903:head# 2024-04-20T17:11:08.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ba15c65:::benchmark_data_smithi171_78918_object154439:head# 2024-04-20T17:11:08.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ba17471:::benchmark_data_smithi171_78918_object155243:head# 2024-04-20T17:11:08.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ba18339:::benchmark_data_smithi171_78918_object64485:head# 2024-04-20T17:11:08.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ba1a815:::benchmark_data_smithi171_78918_object110089:head# 2024-04-20T17:11:08.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ba1b86b:::benchmark_data_smithi171_78918_object112705:head# 2024-04-20T17:11:08.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ba1c6f7:::benchmark_data_smithi171_78918_object112786:head# 2024-04-20T17:11:08.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ba246c9:::benchmark_data_smithi171_78918_object100114:head# 2024-04-20T17:11:08.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ba31673:::benchmark_data_smithi171_78918_object124314:head# 2024-04-20T17:11:08.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ba3e943:::benchmark_data_smithi171_78918_object70401:head# 2024-04-20T17:11:08.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ba4032b:::benchmark_data_smithi171_78918_object78549:head# 2024-04-20T17:11:08.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ba40bd4:::benchmark_data_smithi171_78918_object87692:head# 2024-04-20T17:11:08.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ba42159:::benchmark_data_smithi171_78918_object106472:head# 2024-04-20T17:11:08.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ba4af18:::benchmark_data_smithi171_78918_object108252:head# 2024-04-20T17:11:08.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ba52b55:::benchmark_data_smithi171_78918_object85624:head# 2024-04-20T17:11:08.913 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ba597c1:::benchmark_data_smithi171_78918_object67801:head# 2024-04-20T17:11:08.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ba5c826:::benchmark_data_smithi171_78918_object140963:head# 2024-04-20T17:11:08.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ba6a96b:::benchmark_data_smithi171_78918_object110187:head# 2024-04-20T17:11:08.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ba70383:::benchmark_data_smithi171_78918_object77178:head# 2024-04-20T17:11:08.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ba83bf4:::benchmark_data_smithi171_78918_object77515:head# 2024-04-20T17:11:08.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ba93815:::benchmark_data_smithi171_78918_object82080:head# 2024-04-20T17:11:08.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ba97604:::benchmark_data_smithi171_78918_object53376:head# 2024-04-20T17:11:08.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ba99ef6:::benchmark_data_smithi171_78918_object83120:head# 2024-04-20T17:11:08.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7baa23ce:::benchmark_data_smithi171_78918_object109573:head# 2024-04-20T17:11:08.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bab0ad5:::benchmark_data_smithi171_78918_object77865:head# 2024-04-20T17:11:08.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bab2c90:::benchmark_data_smithi171_78918_object108665:head# 2024-04-20T17:11:08.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bab8028:::benchmark_data_smithi171_78918_object103195:head# 2024-04-20T17:11:08.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bad247f:::benchmark_data_smithi171_78918_object58387:head# 2024-04-20T17:11:08.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bad7a66:::benchmark_data_smithi171_78918_object92564:head# 2024-04-20T17:11:08.914 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7baf56dd:::benchmark_data_smithi171_78918_object99685:head# 2024-04-20T17:11:08.915 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bafaa47:::benchmark_data_smithi171_78918_object117433:head# 2024-04-20T17:11:08.915 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bb02544:::benchmark_data_smithi171_78918_object134617:head# 2024-04-20T17:11:08.915 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bb027b8:::benchmark_data_smithi171_78918_object137304:head# 2024-04-20T17:11:08.915 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bb1b873:::benchmark_data_smithi171_78918_object62855:head# 2024-04-20T17:11:08.915 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bb23dea:::benchmark_data_smithi171_78918_object141399:head# 2024-04-20T17:11:08.915 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bb29d46:::benchmark_data_smithi171_78918_object62291:head# 2024-04-20T17:11:08.915 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bb3a483:::benchmark_data_smithi171_78918_object88976:head# 2024-04-20T17:11:08.915 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bb464db:::benchmark_data_smithi171_78918_object84027:head# 2024-04-20T17:11:08.915 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bb592aa:::benchmark_data_smithi171_78918_object96513:head# 2024-04-20T17:11:08.915 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bb5ac2e:::benchmark_data_smithi171_78918_object137434:head# 2024-04-20T17:11:08.915 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bb5c99a:::benchmark_data_smithi171_78918_object143989:head# 2024-04-20T17:11:08.915 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bb6edb7:::benchmark_data_smithi171_78918_object157523:head# 2024-04-20T17:11:08.915 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bb75453:::benchmark_data_smithi171_78918_object145629:head# 2024-04-20T17:11:08.915 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bb774e6:::benchmark_data_smithi171_78918_object54230:head# 2024-04-20T17:11:08.916 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bb7d468:::benchmark_data_smithi171_78918_object157652:head# 2024-04-20T17:11:08.916 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bb850ed:::benchmark_data_smithi171_78918_object115537:head# 2024-04-20T17:11:08.916 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bb8ce99:::benchmark_data_smithi171_78918_object115609:head# 2024-04-20T17:11:08.916 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bbae6ce:::benchmark_data_smithi171_78918_object130849:head# 2024-04-20T17:11:08.916 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bbb828b:::benchmark_data_smithi171_78918_object152655:head# 2024-04-20T17:11:08.916 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bbbc268:::benchmark_data_smithi171_78918_object116161:head# 2024-04-20T17:11:08.916 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bbcca12:::benchmark_data_smithi171_78918_object82893:head# 2024-04-20T17:11:08.916 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bbd6c5b:::benchmark_data_smithi171_78918_object107854:head# 2024-04-20T17:11:08.916 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bbda960:::benchmark_data_smithi171_78918_object71376:head# 2024-04-20T17:11:08.916 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bbe0161:::benchmark_data_smithi171_78918_object147358:head# 2024-04-20T17:11:08.916 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bbefad8:::benchmark_data_smithi171_78918_object157311:head# 2024-04-20T17:11:08.916 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bbf58d9:::benchmark_data_smithi171_78918_object156299:head# 2024-04-20T17:11:08.916 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bbf5925:::benchmark_data_smithi171_78918_object127866:head# 2024-04-20T17:11:08.916 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bbf7f67:::benchmark_data_smithi171_78918_object160533:head# 2024-04-20T17:11:08.916 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bbfd29c:::benchmark_data_smithi171_78918_object104943:head# 2024-04-20T17:11:08.917 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bc01192:::benchmark_data_smithi171_78918_object131829:head# 2024-04-20T17:11:08.917 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bc040bb:::benchmark_data_smithi171_78918_object82901:head# 2024-04-20T17:11:08.917 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bc04d95:::benchmark_data_smithi171_78918_object150888:head# 2024-04-20T17:11:08.917 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bc14271:::benchmark_data_smithi171_78918_object155601:head# 2024-04-20T17:11:08.917 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bc21d10:::benchmark_data_smithi171_78918_object126438:head# 2024-04-20T17:11:08.917 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bc21dbe:::benchmark_data_smithi171_78918_object63363:head# 2024-04-20T17:11:08.917 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bc220e3:::benchmark_data_smithi171_78918_object101312:head# 2024-04-20T17:11:08.917 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bc2d7df:::benchmark_data_smithi171_78918_object152291:head# 2024-04-20T17:11:08.917 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bc3f964:::benchmark_data_smithi171_78918_object95283:head# 2024-04-20T17:11:08.917 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bc454c9:::benchmark_data_smithi171_78918_object123478:head# 2024-04-20T17:11:08.917 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bc4e420:::benchmark_data_smithi171_78918_object78819:head# 2024-04-20T17:11:08.917 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bc56250:::benchmark_data_smithi171_78918_object59579:head# 2024-04-20T17:11:08.917 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bc5d9fc:::benchmark_data_smithi171_78918_object86361:head# 2024-04-20T17:11:08.917 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bc6cd3c:::benchmark_data_smithi171_78918_object133739:head# 2024-04-20T17:11:08.918 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bc77ed0:::benchmark_data_smithi171_78918_object128147:head# 2024-04-20T17:11:08.918 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bc913a8:::benchmark_data_smithi171_78918_object59632:head# 2024-04-20T17:11:08.918 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bc9bf7f:::benchmark_data_smithi171_78918_object134647:head# 2024-04-20T17:11:08.918 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bca61d0:::benchmark_data_smithi171_78918_object152349:head# 2024-04-20T17:11:08.918 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bcabf1a:::benchmark_data_smithi171_78918_object157401:head# 2024-04-20T17:11:08.918 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bcc3b62:::benchmark_data_smithi171_78918_object68893:head# 2024-04-20T17:11:08.918 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bcc4fb2:::benchmark_data_smithi171_78918_object114304:head# 2024-04-20T17:11:08.918 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bcc819e:::benchmark_data_smithi171_78918_object143111:head# 2024-04-20T17:11:08.918 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bcd046c:::benchmark_data_smithi171_78918_object90228:head# 2024-04-20T17:11:08.918 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bcd0bfd:::benchmark_data_smithi171_78918_object139009:head# 2024-04-20T17:11:08.918 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bce5ff9:::benchmark_data_smithi171_78918_object158123:head# 2024-04-20T17:11:08.918 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bced057:::benchmark_data_smithi171_78918_object56332:head# 2024-04-20T17:11:08.918 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bcf63d9:::benchmark_data_smithi171_78918_object79917:head# 2024-04-20T17:11:08.918 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bd10505:::benchmark_data_smithi171_78918_object88349:head# 2024-04-20T17:11:08.919 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bd16849:::benchmark_data_smithi171_78918_object83603:head# 2024-04-20T17:11:08.919 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bd17f8c:::benchmark_data_smithi171_78918_object142677:head# 2024-04-20T17:11:08.919 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:11:08.933 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bd1b9d6:::benchmark_data_smithi171_78918_object76077:head# 2024-04-20T17:11:08.933 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bd1fb98:::benchmark_data_smithi171_78918_object119851:head# 2024-04-20T17:11:08.933 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bd25408:::benchmark_data_smithi171_78918_object121547:head# 2024-04-20T17:11:08.933 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bd2b586:::benchmark_data_smithi171_78918_object119994:head# 2024-04-20T17:11:08.933 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bd38523:::benchmark_data_smithi171_78918_object79118:head# 2024-04-20T17:11:08.933 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bd38f15:::benchmark_data_smithi171_78918_object82487:head# 2024-04-20T17:11:08.933 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bd3ffb0:::benchmark_data_smithi171_78918_object74613:head# 2024-04-20T17:11:08.933 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bd4b774:::benchmark_data_smithi171_78918_object67264:head# 2024-04-20T17:11:08.933 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bd60762:::benchmark_data_smithi171_78918_object76983:head# 2024-04-20T17:11:08.933 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bd61f7b:::benchmark_data_smithi171_78918_object84396:head# 2024-04-20T17:11:08.933 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bd626d5:::benchmark_data_smithi171_78918_object58173:head# 2024-04-20T17:11:08.933 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bd68ee5:::benchmark_data_smithi171_78918_object140857:head# 2024-04-20T17:11:08.933 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bd6b825:::benchmark_data_smithi171_78918_object102473:head# 2024-04-20T17:11:08.934 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bd6e906:::benchmark_data_smithi171_78918_object61118:head# 2024-04-20T17:11:08.934 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bd7d402:::benchmark_data_smithi171_78918_object55597:head# 2024-04-20T17:11:08.934 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bd81d8a:::benchmark_data_smithi171_78918_object122459:head# 2024-04-20T17:11:08.934 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bd8d0f8:::benchmark_data_smithi171_78918_object124975:head# 2024-04-20T17:11:08.934 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bd8d761:::benchmark_data_smithi171_78918_object90938:head# 2024-04-20T17:11:08.934 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bd902ac:::benchmark_data_smithi171_78918_object67113:head# 2024-04-20T17:11:08.934 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bd9ad2b:::benchmark_data_smithi171_78918_object55299:head# 2024-04-20T17:11:08.934 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bda8b20:::benchmark_data_smithi171_78918_object134559:head# 2024-04-20T17:11:08.934 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bda9c9c:::benchmark_data_smithi171_78918_object91910:head# 2024-04-20T17:11:08.934 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bdc060d:::benchmark_data_smithi171_78918_object137407:head# 2024-04-20T17:11:08.934 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bdcc612:::benchmark_data_smithi171_78918_object110265:head# 2024-04-20T17:11:08.934 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bddd23a:::benchmark_data_smithi171_78918_object160784:head# 2024-04-20T17:11:08.934 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bde21bb:::benchmark_data_smithi171_78918_object144851:head# 2024-04-20T17:11:08.935 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bde9dd9:::benchmark_data_smithi171_78918_object146770:head# 2024-04-20T17:11:08.935 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bdf23d6:::benchmark_data_smithi171_78918_object88551:head# 2024-04-20T17:11:08.935 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bdf4151:::benchmark_data_smithi171_78918_object145236:head# 2024-04-20T17:11:08.935 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7be00319:::benchmark_data_smithi171_78918_object120137:head# 2024-04-20T17:11:08.935 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7be094ce:::benchmark_data_smithi171_78918_object85169:head# 2024-04-20T17:11:08.935 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7be10889:::benchmark_data_smithi171_78918_object116752:head# 2024-04-20T17:11:08.935 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7be47bc8:::benchmark_data_smithi171_78918_object154077:head# 2024-04-20T17:11:08.935 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7be4c786:::benchmark_data_smithi171_78918_object121490:head# 2024-04-20T17:11:08.935 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7be59df7:::benchmark_data_smithi171_78918_object144552:head# 2024-04-20T17:11:08.935 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7be5a179:::benchmark_data_smithi171_78918_object61399:head# 2024-04-20T17:11:08.935 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7be7ba05:::benchmark_data_smithi171_78918_object150763:head# 2024-04-20T17:11:08.935 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7be7f2e4:::benchmark_data_smithi171_78918_object120766:head# 2024-04-20T17:11:08.935 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7be818bd:::benchmark_data_smithi171_78918_object56817:head# 2024-04-20T17:11:08.935 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7be9ac79:::benchmark_data_smithi171_78918_object113443:head# 2024-04-20T17:11:08.936 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bea63a0:::benchmark_data_smithi171_78918_object103458:head# 2024-04-20T17:11:08.936 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7beaa37b:::benchmark_data_smithi171_78918_object96047:head# 2024-04-20T17:11:08.936 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7beb1b65:::benchmark_data_smithi171_78918_object133425:head# 2024-04-20T17:11:08.936 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bebb3f1:::benchmark_data_smithi171_78918_object150250:head# 2024-04-20T17:11:08.936 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bebbc7d:::benchmark_data_smithi171_78918_object89881:head# 2024-04-20T17:11:08.936 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bec7864:::benchmark_data_smithi171_78918_object134355:head# 2024-04-20T17:11:08.936 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bed0ae9:::benchmark_data_smithi171_78918_object146994:head# 2024-04-20T17:11:08.936 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bed8af3:::benchmark_data_smithi171_78918_object160754:head# 2024-04-20T17:11:08.936 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bedb958:::benchmark_data_smithi171_78918_object64999:head# 2024-04-20T17:11:08.936 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7beeaf67:::benchmark_data_smithi171_78918_object98779:head# 2024-04-20T17:11:08.936 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7beebd41:::benchmark_data_smithi171_78918_object84411:head# 2024-04-20T17:11:08.936 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7beee111:::benchmark_data_smithi171_78918_object55766:head# 2024-04-20T17:11:08.936 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7befb5d7:::benchmark_data_smithi171_78918_object153360:head# 2024-04-20T17:11:08.936 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bf0cdb2:::benchmark_data_smithi171_78918_object149504:head# 2024-04-20T17:11:08.937 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bf1d0ad:::benchmark_data_smithi171_78918_object62300:head# 2024-04-20T17:11:08.937 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bf367b9:::benchmark_data_smithi171_78918_object86411:head# 2024-04-20T17:11:08.937 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bf4e5c9:::benchmark_data_smithi171_78918_object91569:head# 2024-04-20T17:11:08.937 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bf59a81:::benchmark_data_smithi171_78918_object143549:head# 2024-04-20T17:11:08.937 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bf61c40:::benchmark_data_smithi171_78918_object67463:head# 2024-04-20T17:11:08.937 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bf6652b:::benchmark_data_smithi171_78918_object76834:head# 2024-04-20T17:11:08.937 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bf72074:::benchmark_data_smithi171_78918_object98224:head# 2024-04-20T17:11:08.937 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bf73565:::benchmark_data_smithi171_78918_object86404:head# 2024-04-20T17:11:08.937 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bf9628d:::benchmark_data_smithi171_78918_object71338:head# 2024-04-20T17:11:08.937 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bfe3c33:::benchmark_data_smithi171_78918_object82896:head# 2024-04-20T17:11:08.937 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bfe3e42:::benchmark_data_smithi171_78918_object124015:head# 2024-04-20T17:11:08.937 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bfeb0d0:::benchmark_data_smithi171_78918_object68226:head# 2024-04-20T17:11:08.937 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7bff2dfc:::benchmark_data_smithi171_78918_object56284:head# 2024-04-20T17:11:08.937 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c005670:::benchmark_data_smithi171_78918_object84733:head# 2024-04-20T17:11:08.937 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c015a0e:::benchmark_data_smithi171_78918_object62709:head# 2024-04-20T17:11:08.938 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c015fde:::benchmark_data_smithi171_78918_object85779:head# 2024-04-20T17:11:08.938 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c01625b:::benchmark_data_smithi171_78918_object83903:head# 2024-04-20T17:11:08.938 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c01daed:::benchmark_data_smithi171_78918_object98168:head# 2024-04-20T17:11:08.938 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c026f8e:::benchmark_data_smithi171_78918_object59502:head# 2024-04-20T17:11:08.938 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c038fe7:::benchmark_data_smithi171_78918_object120788:head# 2024-04-20T17:11:08.938 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c050efb:::benchmark_data_smithi171_78918_object132468:head# 2024-04-20T17:11:08.938 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c056754:::benchmark_data_smithi171_78918_object156041:head# 2024-04-20T17:11:08.938 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c0632f5:::benchmark_data_smithi171_78918_object85191:head# 2024-04-20T17:11:08.938 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c067421:::benchmark_data_smithi171_78918_object112992:head# 2024-04-20T17:11:08.938 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c068298:::benchmark_data_smithi171_78918_object146970:head# 2024-04-20T17:11:08.938 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c06f266:::benchmark_data_smithi171_78918_object69760:head# 2024-04-20T17:11:08.938 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c06f2dd:::benchmark_data_smithi171_78918_object126369:head# 2024-04-20T17:11:08.938 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c071bf8:::benchmark_data_smithi171_78918_object95539:head# 2024-04-20T17:11:08.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c0840a0:::benchmark_data_smithi171_78918_object63013:head# 2024-04-20T17:11:08.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c0a8361:::benchmark_data_smithi171_78918_object146800:head# 2024-04-20T17:11:08.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c0b1850:::benchmark_data_smithi171_78918_object74509:head# 2024-04-20T17:11:08.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c0ca1b8:::benchmark_data_smithi171_78918_object91816:head# 2024-04-20T17:11:08.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c0e094c:::benchmark_data_smithi171_78918_object74465:head# 2024-04-20T17:11:08.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c0efb1a:::benchmark_data_smithi171_78918_object54442:head# 2024-04-20T17:11:08.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c0f0b08:::benchmark_data_smithi171_78918_object109136:head# 2024-04-20T17:11:08.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c10ebe6:::benchmark_data_smithi171_78918_object93249:head# 2024-04-20T17:11:08.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c11ba4c:::benchmark_data_smithi171_78918_object103081:head# 2024-04-20T17:11:08.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c120034:::benchmark_data_smithi171_78918_object151934:head# 2024-04-20T17:11:08.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c129d5b:::benchmark_data_smithi171_78918_object59646:head# 2024-04-20T17:11:08.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c139db6:::benchmark_data_smithi171_78918_object106133:head# 2024-04-20T17:11:08.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c13d259:::benchmark_data_smithi171_78918_object150873:head# 2024-04-20T17:11:08.939 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c14342f:::benchmark_data_smithi171_78918_object132668:head# 2024-04-20T17:11:08.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c1497ef:::benchmark_data_smithi171_78918_object123883:head# 2024-04-20T17:11:08.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c150abc:::benchmark_data_smithi171_78918_object76921:head# 2024-04-20T17:11:08.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c151948:::benchmark_data_smithi171_78918_object139805:head# 2024-04-20T17:11:08.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c15a5ca:::benchmark_data_smithi171_78918_object160460:head# 2024-04-20T17:11:08.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c15fb1c:::benchmark_data_smithi171_78918_object132586:head# 2024-04-20T17:11:08.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c17215d:::benchmark_data_smithi171_78918_object108793:head# 2024-04-20T17:11:08.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c173530:::benchmark_data_smithi171_78918_object157784:head# 2024-04-20T17:11:08.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c17cf34:::benchmark_data_smithi171_78918_object118775:head# 2024-04-20T17:11:08.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c17eb2d:::benchmark_data_smithi171_78918_object160481:head# 2024-04-20T17:11:08.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c1827f1:::benchmark_data_smithi171_78918_object110054:head# 2024-04-20T17:11:08.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c185a1c:::benchmark_data_smithi171_78918_object145127:head# 2024-04-20T17:11:08.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c19d961:::benchmark_data_smithi171_78918_object67055:head# 2024-04-20T17:11:08.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c1b6920:::benchmark_data_smithi171_78918_object65430:head# 2024-04-20T17:11:08.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c1cca6e:::benchmark_data_smithi171_78918_object159074:head# 2024-04-20T17:11:08.940 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c1d0536:::benchmark_data_smithi171_78918_object82913:head# 2024-04-20T17:11:08.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c1da1e4:::benchmark_data_smithi171_78918_object74268:head# 2024-04-20T17:11:08.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c1dbd9c:::benchmark_data_smithi171_78918_object148066:head# 2024-04-20T17:11:08.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c1dc15e:::benchmark_data_smithi171_78918_object127029:head# 2024-04-20T17:11:08.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c1dfb4c:::benchmark_data_smithi171_78918_object139521:head# 2024-04-20T17:11:08.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c1dff47:::benchmark_data_smithi171_78918_object104083:head# 2024-04-20T17:11:08.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c1e65a3:::benchmark_data_smithi171_78918_object115838:head# 2024-04-20T17:11:08.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c1f34ac:::benchmark_data_smithi171_78918_object88896:head# 2024-04-20T17:11:08.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c1f3eed:::benchmark_data_smithi171_78918_object132227:head# 2024-04-20T17:11:08.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c1f7617:::benchmark_data_smithi171_78918_object107959:head# 2024-04-20T17:11:08.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c20327c:::benchmark_data_smithi171_78918_object66244:head# 2024-04-20T17:11:08.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c203b76:::benchmark_data_smithi171_78918_object52746:head# 2024-04-20T17:11:08.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c208297:::benchmark_data_smithi171_78918_object75417:head# 2024-04-20T17:11:08.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c211d66:::benchmark_data_smithi171_78918_object98614:head# 2024-04-20T17:11:08.941 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c2149ab:::benchmark_data_smithi171_78918_object99606:head# 2024-04-20T17:11:08.942 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c2184a9:::benchmark_data_smithi171_78918_object96090:head# 2024-04-20T17:11:08.942 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c21acfd:::benchmark_data_smithi171_78918_object134535:head# 2024-04-20T17:11:08.942 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c22a62b:::benchmark_data_smithi171_78918_object160755:head# 2024-04-20T17:11:08.942 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c22f3f2:::benchmark_data_smithi171_78918_object95281:head# 2024-04-20T17:11:08.942 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c24066c:::benchmark_data_smithi171_78918_object113871:head# 2024-04-20T17:11:08.942 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c241f7b:::benchmark_data_smithi171_78918_object90267:head# 2024-04-20T17:11:08.942 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c249d4b:::benchmark_data_smithi171_78918_object87156:head# 2024-04-20T17:11:08.942 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c253fc7:::benchmark_data_smithi171_78918_object137866:head# 2024-04-20T17:11:08.942 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c25594a:::benchmark_data_smithi171_78918_object105741:head# 2024-04-20T17:11:08.942 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c255b04:::benchmark_data_smithi171_78918_object84808:head# 2024-04-20T17:11:08.942 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c2648b8:::benchmark_data_smithi171_78918_object69062:head# 2024-04-20T17:11:08.943 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:11:08.955 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c27afcc:::benchmark_data_smithi171_78918_object136150:head# 2024-04-20T17:11:08.955 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c27e4d5:::benchmark_data_smithi171_78918_object108454:head# 2024-04-20T17:11:08.955 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c280a7f:::benchmark_data_smithi171_78918_object136235:head# 2024-04-20T17:11:08.956 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c286387:::benchmark_data_smithi171_78918_object112211:head# 2024-04-20T17:11:08.956 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c2a56a4:::benchmark_data_smithi171_78918_object58754:head# 2024-04-20T17:11:08.956 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c2ad414:::benchmark_data_smithi171_78918_object77340:head# 2024-04-20T17:11:08.956 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c2c9f27:::benchmark_data_smithi171_78918_object98739:head# 2024-04-20T17:11:08.956 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c2ca997:::benchmark_data_smithi171_78918_object136524:head# 2024-04-20T17:11:08.956 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c2db120:::benchmark_data_smithi171_78918_object99210:head# 2024-04-20T17:11:08.956 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c2eaeee:::benchmark_data_smithi171_78918_object144366:head# 2024-04-20T17:11:08.956 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c2f6c8c:::benchmark_data_smithi171_78918_object124843:head# 2024-04-20T17:11:08.956 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c2fcd41:::benchmark_data_smithi171_78918_object107729:head# 2024-04-20T17:11:08.956 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c304e4d:::benchmark_data_smithi171_78918_object146863:head# 2024-04-20T17:11:08.956 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c308c85:::benchmark_data_smithi171_78918_object81491:head# 2024-04-20T17:11:08.956 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c327d79:::benchmark_data_smithi171_78918_object157677:head# 2024-04-20T17:11:08.956 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c329b01:::benchmark_data_smithi171_78918_object116313:head# 2024-04-20T17:11:08.956 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c33c134:::benchmark_data_smithi171_78918_object148124:head# 2024-04-20T17:11:08.957 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c33e96e:::benchmark_data_smithi171_78918_object113601:head# 2024-04-20T17:11:08.957 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c3415e3:::benchmark_data_smithi171_78918_object111096:head# 2024-04-20T17:11:08.957 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c3467cc:::benchmark_data_smithi171_78918_object136992:head# 2024-04-20T17:11:08.957 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c34d052:::benchmark_data_smithi171_78918_object57234:head# 2024-04-20T17:11:08.957 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c35624c:::benchmark_data_smithi171_78918_object156994:head# 2024-04-20T17:11:08.957 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c3567a5:::benchmark_data_smithi171_78918_object84168:head# 2024-04-20T17:11:08.957 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c359910:::benchmark_data_smithi171_78918_object151051:head# 2024-04-20T17:11:08.957 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c367447:::benchmark_data_smithi171_78918_object138724:head# 2024-04-20T17:11:08.957 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c369fda:::benchmark_data_smithi171_78918_object117940:head# 2024-04-20T17:11:08.957 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c377a69:::benchmark_data_smithi171_78918_object149863:head# 2024-04-20T17:11:08.957 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c381111:::benchmark_data_smithi171_78918_object134002:head# 2024-04-20T17:11:08.957 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c383197:::benchmark_data_smithi171_78918_object141240:head# 2024-04-20T17:11:08.957 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c38b1f0:::benchmark_data_smithi171_78918_object135858:head# 2024-04-20T17:11:08.957 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c392f1b:::benchmark_data_smithi171_78918_object148381:head# 2024-04-20T17:11:08.957 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c39e408:::benchmark_data_smithi171_78918_object55815:head# 2024-04-20T17:11:08.958 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c3a6705:::benchmark_data_smithi171_78918_object99672:head# 2024-04-20T17:11:08.958 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c3b1019:::benchmark_data_smithi171_78918_object68269:head# 2024-04-20T17:11:08.958 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c3bb779:::benchmark_data_smithi171_78918_object76956:head# 2024-04-20T17:11:08.958 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c3bf25a:::benchmark_data_smithi171_78918_object135386:head# 2024-04-20T17:11:08.958 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c3d0f4d:::benchmark_data_smithi171_78918_object129252:head# 2024-04-20T17:11:08.958 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c3d1c83:::benchmark_data_smithi171_78918_object148911:head# 2024-04-20T17:11:08.958 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c3d74fb:::benchmark_data_smithi171_78918_object87608:head# 2024-04-20T17:11:08.958 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c3e78b9:::benchmark_data_smithi171_78918_object154611:head# 2024-04-20T17:11:08.958 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c3ee3c2:::benchmark_data_smithi171_78918_object124744:head# 2024-04-20T17:11:08.958 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c3f56d1:::benchmark_data_smithi171_78918_object147921:head# 2024-04-20T17:11:08.958 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c3faa57:::benchmark_data_smithi171_78918_object136752:head# 2024-04-20T17:11:08.958 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c4091aa:::benchmark_data_smithi171_78918_object134952:head# 2024-04-20T17:11:08.958 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c40b100:::benchmark_data_smithi171_78918_object123695:head# 2024-04-20T17:11:08.958 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c412541:::benchmark_data_smithi171_78918_object75977:head# 2024-04-20T17:11:08.959 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c41b255:::benchmark_data_smithi171_78918_object155248:head# 2024-04-20T17:11:08.959 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c42110b:::benchmark_data_smithi171_78918_object66307:head# 2024-04-20T17:11:08.959 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c42bb55:::benchmark_data_smithi171_78918_object149444:head# 2024-04-20T17:11:08.959 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c42cf07:::benchmark_data_smithi171_78918_object97206:head# 2024-04-20T17:11:08.959 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c4385d2:::benchmark_data_smithi171_78918_object153130:head# 2024-04-20T17:11:08.959 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c439c26:::benchmark_data_smithi171_78918_object125017:head# 2024-04-20T17:11:08.959 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c44c943:::benchmark_data_smithi171_78918_object155352:head# 2024-04-20T17:11:08.959 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c462dd0:::benchmark_data_smithi171_78918_object99563:head# 2024-04-20T17:11:08.959 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c4a1f54:::benchmark_data_smithi171_78918_object145535:head# 2024-04-20T17:11:08.959 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c4a4c28:::benchmark_data_smithi171_78918_object117384:head# 2024-04-20T17:11:08.959 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c4b333c:::benchmark_data_smithi171_78918_object100294:head# 2024-04-20T17:11:08.959 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c4b336d:::benchmark_data_smithi171_78918_object102657:head# 2024-04-20T17:11:08.959 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c4c2551:::benchmark_data_smithi171_78918_object55189:head# 2024-04-20T17:11:08.959 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c4c69e8:::benchmark_data_smithi171_78918_object149523:head# 2024-04-20T17:11:08.959 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c4cd8a7:::benchmark_data_smithi171_78918_object131355:head# 2024-04-20T17:11:08.960 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c4d1e8a:::benchmark_data_smithi171_78918_object62311:head# 2024-04-20T17:11:08.960 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c4d6eb1:::benchmark_data_smithi171_78918_object62685:head# 2024-04-20T17:11:08.960 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c4e4573:::benchmark_data_smithi171_78918_object148799:head# 2024-04-20T17:11:08.960 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c4e70d2:::benchmark_data_smithi171_78918_object148561:head# 2024-04-20T17:11:08.960 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c4ecddc:::benchmark_data_smithi171_78918_object141598:head# 2024-04-20T17:11:08.960 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c4f9911:::benchmark_data_smithi171_78918_object107799:head# 2024-04-20T17:11:08.960 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c504b08:::benchmark_data_smithi171_78918_object72616:head# 2024-04-20T17:11:08.960 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c50f95a:::benchmark_data_smithi171_78918_object134955:head# 2024-04-20T17:11:08.960 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c51f23d:::benchmark_data_smithi171_78918_object159938:head# 2024-04-20T17:11:08.960 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c52b68e:::benchmark_data_smithi171_78918_object71961:head# 2024-04-20T17:11:08.960 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c52c981:::benchmark_data_smithi171_78918_object105494:head# 2024-04-20T17:11:08.960 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c53d0a5:::benchmark_data_smithi171_78918_object80017:head# 2024-04-20T17:11:08.960 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c54eb4a:::benchmark_data_smithi171_78918_object80896:head# 2024-04-20T17:11:08.960 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c5526c3:::benchmark_data_smithi171_78918_object105304:head# 2024-04-20T17:11:08.960 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c55a34c:::benchmark_data_smithi171_78918_object123174:head# 2024-04-20T17:11:08.961 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c5613c1:::benchmark_data_smithi171_78918_object56823:head# 2024-04-20T17:11:08.961 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c567a3f:::benchmark_data_smithi171_78918_object111980:head# 2024-04-20T17:11:08.961 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c568ccf:::benchmark_data_smithi171_78918_object65859:head# 2024-04-20T17:11:08.961 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c56a8fb:::benchmark_data_smithi171_78918_object135056:head# 2024-04-20T17:11:08.961 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c57847c:::benchmark_data_smithi171_78918_object151871:head# 2024-04-20T17:11:08.961 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c57a076:::benchmark_data_smithi171_78918_object114871:head# 2024-04-20T17:11:08.961 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c59b79e:::benchmark_data_smithi171_78918_object132967:head# 2024-04-20T17:11:08.961 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c5a7732:::benchmark_data_smithi171_78918_object138253:head# 2024-04-20T17:11:08.961 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c5a863f:::benchmark_data_smithi171_78918_object114742:head# 2024-04-20T17:11:08.961 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c5aca39:::benchmark_data_smithi171_78918_object70965:head# 2024-04-20T17:11:08.961 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c5add18:::benchmark_data_smithi171_78918_object51518:head# 2024-04-20T17:11:08.961 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c5b1f21:::benchmark_data_smithi171_78918_object97472:head# 2024-04-20T17:11:08.961 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c5b7732:::benchmark_data_smithi171_78918_object56345:head# 2024-04-20T17:11:08.961 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c5c52a8:::benchmark_data_smithi171_78918_object107895:head# 2024-04-20T17:11:08.961 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c5c5405:::benchmark_data_smithi171_78918_object95774:head# 2024-04-20T17:11:08.962 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c5c65d0:::benchmark_data_smithi171_78918_object84125:head# 2024-04-20T17:11:08.962 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c5c85a5:::benchmark_data_smithi171_78918_object62958:head# 2024-04-20T17:11:08.962 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c5d0eb1:::benchmark_data_smithi171_78918_object149867:head# 2024-04-20T17:11:08.962 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c5e8b8b:::benchmark_data_smithi171_78918_object89034:head# 2024-04-20T17:11:08.962 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c5f91cd:::benchmark_data_smithi171_78918_object77476:head# 2024-04-20T17:11:08.962 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c60dc94:::benchmark_data_smithi171_78918_object156658:head# 2024-04-20T17:11:08.962 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c61ac1e:::benchmark_data_smithi171_78918_object87296:head# 2024-04-20T17:11:08.962 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c62416d:::benchmark_data_smithi171_78918_object136234:head# 2024-04-20T17:11:08.962 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c62737a:::benchmark_data_smithi171_78918_object75978:head# 2024-04-20T17:11:08.962 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c62eb34:::benchmark_data_smithi171_78918_object70118:head# 2024-04-20T17:11:08.962 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c633558:::benchmark_data_smithi171_78918_object82296:head# 2024-04-20T17:11:08.962 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c63514d:::benchmark_data_smithi171_78918_object136659:head# 2024-04-20T17:11:08.962 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c6494b7:::benchmark_data_smithi171_78918_object152334:head# 2024-04-20T17:11:08.962 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c652c90:::benchmark_data_smithi171_78918_object70592:head# 2024-04-20T17:11:08.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c65695b:::benchmark_data_smithi171_78918_object73323:head# 2024-04-20T17:11:08.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c65d530:::benchmark_data_smithi171_78918_object74923:head# 2024-04-20T17:11:08.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c65d65d:::benchmark_data_smithi171_78918_object98707:head# 2024-04-20T17:11:08.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c6664cf:::benchmark_data_smithi171_78918_object115072:head# 2024-04-20T17:11:08.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c670001:::benchmark_data_smithi171_78918_object78381:head# 2024-04-20T17:11:08.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c671212:::benchmark_data_smithi171_78918_object132585:head# 2024-04-20T17:11:08.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c678aa9:::benchmark_data_smithi171_78918_object79454:head# 2024-04-20T17:11:08.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c68c0f2:::benchmark_data_smithi171_78918_object100918:head# 2024-04-20T17:11:08.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c68f14c:::benchmark_data_smithi171_78918_object93192:head# 2024-04-20T17:11:08.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c69321e:::benchmark_data_smithi171_78918_object103439:head# 2024-04-20T17:11:08.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c6949a7:::benchmark_data_smithi171_78918_object126617:head# 2024-04-20T17:11:08.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c697ea2:::benchmark_data_smithi171_78918_object159957:head# 2024-04-20T17:11:08.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c6a99ea:::benchmark_data_smithi171_78918_object53088:head# 2024-04-20T17:11:08.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c6ba814:::benchmark_data_smithi171_78918_object158873:head# 2024-04-20T17:11:08.963 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c6c95b2:::benchmark_data_smithi171_78918_object149417:head# 2024-04-20T17:11:08.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c6e4727:::benchmark_data_smithi171_78918_object95231:head# 2024-04-20T17:11:08.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c6eb802:::benchmark_data_smithi171_78918_object73377:head# 2024-04-20T17:11:08.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c6f5676:::benchmark_data_smithi171_78918_object144667:head# 2024-04-20T17:11:08.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c716a9e:::benchmark_data_smithi171_78918_object121062:head# 2024-04-20T17:11:08.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c71d600:::benchmark_data_smithi171_78918_object132522:head# 2024-04-20T17:11:08.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c73df55:::benchmark_data_smithi171_78918_object78045:head# 2024-04-20T17:11:08.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c741ec9:::benchmark_data_smithi171_78918_object117809:head# 2024-04-20T17:11:08.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c7448ba:::benchmark_data_smithi171_78918_object130328:head# 2024-04-20T17:11:08.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c7453dd:::benchmark_data_smithi171_78918_object59176:head# 2024-04-20T17:11:08.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c751bf5:::benchmark_data_smithi171_78918_object92977:head# 2024-04-20T17:11:08.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c7620a0:::benchmark_data_smithi171_78918_object149070:head# 2024-04-20T17:11:08.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c768710:::benchmark_data_smithi171_78918_object122409:head# 2024-04-20T17:11:08.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c783ecf:::benchmark_data_smithi171_78918_object99417:head# 2024-04-20T17:11:08.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c7863bf:::benchmark_data_smithi171_78918_object114517:head# 2024-04-20T17:11:08.964 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c78d98b:::benchmark_data_smithi171_78918_object83218:head# 2024-04-20T17:11:08.965 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c78fdf8:::benchmark_data_smithi171_78918_object149175:head# 2024-04-20T17:11:08.965 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c79a799:::benchmark_data_smithi171_78918_object78248:head# 2024-04-20T17:11:08.965 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c79b6d9:::benchmark_data_smithi171_78918_object91194:head# 2024-04-20T17:11:08.965 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c7a8c5a:::benchmark_data_smithi171_78918_object66448:head# 2024-04-20T17:11:08.965 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:11:08.965 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c7a9955:::benchmark_data_smithi171_78918_object123497:head# 2024-04-20T17:11:08.966 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:11:08.966 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:11:08.966 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-20T17:11:08.966 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:11:08.966 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:11:08.966 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:11:08.966 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:11:08.966 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:11:08.966 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:11:08.966 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:11:08.966 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:11:08.966 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:11:09.155 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c7c53f4:::benchmark_data_smithi171_78918_object124510:head# 2024-04-20T17:11:09.155 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c7caf49:::benchmark_data_smithi171_78918_object81184:head# 2024-04-20T17:11:09.155 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c7ce5e8:::benchmark_data_smithi171_78918_object92394:head# 2024-04-20T17:11:09.155 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c7ce994:::benchmark_data_smithi171_78918_object77283:head# 2024-04-20T17:11:09.155 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c7ce9a9:::benchmark_data_smithi171_78918_object101235:head# 2024-04-20T17:11:09.155 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c7de34a:::benchmark_data_smithi171_78918_object105477:head# 2024-04-20T17:11:09.156 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c7e7ec1:::benchmark_data_smithi171_78918_object58328:head# 2024-04-20T17:11:09.156 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c7e8d29:::benchmark_data_smithi171_78918_object84486:head# 2024-04-20T17:11:09.156 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c8093a2:::benchmark_data_smithi171_78918_object92482:head# 2024-04-20T17:11:09.156 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c80d3e8:::benchmark_data_smithi171_78918_object102274:head# 2024-04-20T17:11:09.156 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c8137c2:::benchmark_data_smithi171_78918_object79252:head# 2024-04-20T17:11:09.156 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c830f18:::benchmark_data_smithi171_78918_object125053:head# 2024-04-20T17:11:09.156 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c8314b9:::benchmark_data_smithi171_78918_object122921:head# 2024-04-20T17:11:09.156 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c83b36c:::benchmark_data_smithi171_78918_object62077:head# 2024-04-20T17:11:09.156 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c84e5cd:::benchmark_data_smithi171_78918_object64693:head# 2024-04-20T17:11:09.156 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c857fe8:::benchmark_data_smithi171_78918_object123021:head# 2024-04-20T17:11:09.156 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c85860b:::benchmark_data_smithi171_78918_object152780:head# 2024-04-20T17:11:09.156 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c8812e0:::benchmark_data_smithi171_78918_object134544:head# 2024-04-20T17:11:09.156 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c89b1f3:::benchmark_data_smithi171_78918_object149400:head# 2024-04-20T17:11:09.156 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c89eafb:::benchmark_data_smithi171_78918_object67868:head# 2024-04-20T17:11:09.157 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c8ab34d:::benchmark_data_smithi171_78918_object106232:head# 2024-04-20T17:11:09.157 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c8b8c03:::benchmark_data_smithi171_78918_object87401:head# 2024-04-20T17:11:09.157 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c8c3b71:::benchmark_data_smithi171_78918_object73407:head# 2024-04-20T17:11:09.157 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c8df72b:::benchmark_data_smithi171_78918_object154766:head# 2024-04-20T17:11:09.157 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c8f20bf:::benchmark_data_smithi171_78918_object105300:head# 2024-04-20T17:11:09.157 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c8fa5ed:::benchmark_data_smithi171_78918_object91541:head# 2024-04-20T17:11:09.157 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c8fd0a1:::benchmark_data_smithi171_78918_object141537:head# 2024-04-20T17:11:09.157 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c9052e1:::benchmark_data_smithi171_78918_object128626:head# 2024-04-20T17:11:09.157 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c90a698:::benchmark_data_smithi171_78918_object128349:head# 2024-04-20T17:11:09.157 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c912a84:::benchmark_data_smithi171_78918_object128196:head# 2024-04-20T17:11:09.157 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c914074:::benchmark_data_smithi171_78918_object54188:head# 2024-04-20T17:11:09.157 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c91da57:::benchmark_data_smithi171_78918_object87929:head# 2024-04-20T17:11:09.157 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c9263bd:::benchmark_data_smithi171_78918_object52840:head# 2024-04-20T17:11:09.157 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c92d997:::benchmark_data_smithi171_78918_object62775:head# 2024-04-20T17:11:09.158 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c93228e:::benchmark_data_smithi171_78918_object141278:head# 2024-04-20T17:11:09.158 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c936b02:::benchmark_data_smithi171_78918_object159392:head# 2024-04-20T17:11:09.158 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c93e16b:::benchmark_data_smithi171_78918_object125205:head# 2024-04-20T17:11:09.158 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c9439ce:::benchmark_data_smithi171_78918_object84311:head# 2024-04-20T17:11:09.158 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c944336:::benchmark_data_smithi171_78918_object141539:head# 2024-04-20T17:11:09.158 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c953acd:::benchmark_data_smithi171_78918_object62970:head# 2024-04-20T17:11:09.158 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c958a48:::benchmark_data_smithi171_78918_object149806:head# 2024-04-20T17:11:09.158 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c95acbe:::benchmark_data_smithi171_78918_object158383:head# 2024-04-20T17:11:09.158 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c95eee0:::benchmark_data_smithi171_78918_object148953:head# 2024-04-20T17:11:09.158 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c962852:::benchmark_data_smithi171_78918_object70169:head# 2024-04-20T17:11:09.158 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c96539c:::benchmark_data_smithi171_78918_object110791:head# 2024-04-20T17:11:09.158 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c96c065:::benchmark_data_smithi171_78918_object53687:head# 2024-04-20T17:11:09.158 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c97280d:::benchmark_data_smithi171_78918_object62383:head# 2024-04-20T17:11:09.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c97e251:::benchmark_data_smithi171_78918_object77885:head# 2024-04-20T17:11:09.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c99b246:::benchmark_data_smithi171_78918_object110836:head# 2024-04-20T17:11:09.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c99ce2c:::benchmark_data_smithi171_78918_object130048:head# 2024-04-20T17:11:09.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c9a8aec:::benchmark_data_smithi171_78918_object70524:head# 2024-04-20T17:11:09.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c9ba171:::benchmark_data_smithi171_78918_object119979:head# 2024-04-20T17:11:09.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c9baadd:::benchmark_data_smithi171_78918_object65661:head# 2024-04-20T17:11:09.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c9c4986:::benchmark_data_smithi171_78918_object118574:head# 2024-04-20T17:11:09.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c9c4e4c:::benchmark_data_smithi171_78918_object70156:head# 2024-04-20T17:11:09.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c9d69dd:::benchmark_data_smithi171_78918_object149846:head# 2024-04-20T17:11:09.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c9dba5e:::benchmark_data_smithi171_78918_object54981:head# 2024-04-20T17:11:09.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c9dfb70:::benchmark_data_smithi171_78918_object122320:head# 2024-04-20T17:11:09.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c9e368e:::benchmark_data_smithi171_78918_object107373:head# 2024-04-20T17:11:09.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c9e5352:::benchmark_data_smithi171_78918_object108860:head# 2024-04-20T17:11:09.159 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7c9e543b:::benchmark_data_smithi171_78918_object55420:head# 2024-04-20T17:11:09.160 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ca00302:::benchmark_data_smithi171_78918_object88012:head# 2024-04-20T17:11:09.160 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ca00853:::benchmark_data_smithi171_78918_object86307:head# 2024-04-20T17:11:09.160 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ca02802:::benchmark_data_smithi171_78918_object57692:head# 2024-04-20T17:11:09.160 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ca150a1:::benchmark_data_smithi171_78918_object91453:head# 2024-04-20T17:11:09.160 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ca1a562:::benchmark_data_smithi171_78918_object77827:head# 2024-04-20T17:11:09.160 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ca212f6:::benchmark_data_smithi171_78918_object117972:head# 2024-04-20T17:11:09.160 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ca2fc91:::benchmark_data_smithi171_78918_object100682:head# 2024-04-20T17:11:09.160 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ca36f1d:::benchmark_data_smithi171_78918_object145767:head# 2024-04-20T17:11:09.160 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ca443a7:::benchmark_data_smithi171_78918_object57259:head# 2024-04-20T17:11:09.160 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ca4c9fa:::benchmark_data_smithi171_78918_object133979:head# 2024-04-20T17:11:09.160 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ca52454:::benchmark_data_smithi171_78918_object153338:head# 2024-04-20T17:11:09.160 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ca57b5b:::benchmark_data_smithi171_78918_object76074:head# 2024-04-20T17:11:09.160 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ca5adce:::benchmark_data_smithi171_78918_object61165:head# 2024-04-20T17:11:09.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ca634ed:::benchmark_data_smithi171_78918_object158530:head# 2024-04-20T17:11:09.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ca79f26:::benchmark_data_smithi171_78918_object147278:head# 2024-04-20T17:11:09.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ca7a9ac:::benchmark_data_smithi171_78918_object145630:head# 2024-04-20T17:11:09.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ca7acf0:::benchmark_data_smithi171_78918_object74767:head# 2024-04-20T17:11:09.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ca7f4dc:::benchmark_data_smithi171_78918_object83493:head# 2024-04-20T17:11:09.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ca8411d:::benchmark_data_smithi171_78918_object160787:head# 2024-04-20T17:11:09.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ca89053:::benchmark_data_smithi171_78918_object160918:head# 2024-04-20T17:11:09.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ca8a68b:::benchmark_data_smithi171_78918_object160955:head# 2024-04-20T17:11:09.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ca95bfd:::benchmark_data_smithi171_78918_object98865:head# 2024-04-20T17:11:09.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7caa3368:::benchmark_data_smithi171_78918_object134765:head# 2024-04-20T17:11:09.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7caa9238:::benchmark_data_smithi171_78918_object82140:head# 2024-04-20T17:11:09.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7caabfb9:::benchmark_data_smithi171_78918_object113413:head# 2024-04-20T17:11:09.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cabbc3e:::benchmark_data_smithi171_78918_object102627:head# 2024-04-20T17:11:09.161 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cac48ae:::benchmark_data_smithi171_78918_object132538:head# 2024-04-20T17:11:09.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cad524a:::benchmark_data_smithi171_78918_object80751:head# 2024-04-20T17:11:09.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cadfc15:::benchmark_data_smithi171_78918_object88126:head# 2024-04-20T17:11:09.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7caf64ec:::benchmark_data_smithi171_78918_object74552:head# 2024-04-20T17:11:09.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7caf7e05:::benchmark_data_smithi171_78918_object93405:head# 2024-04-20T17:11:09.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cafd7f8:::benchmark_data_smithi171_78918_object58053:head# 2024-04-20T17:11:09.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cb02768:::benchmark_data_smithi171_78918_object159245:head# 2024-04-20T17:11:09.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cb08973:::benchmark_data_smithi171_78918_object60775:head# 2024-04-20T17:11:09.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cb1a478:::benchmark_data_smithi171_78918_object138767:head# 2024-04-20T17:11:09.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cb1da81:::benchmark_data_smithi171_78918_object151136:head# 2024-04-20T17:11:09.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cb200f7:::benchmark_data_smithi171_78918_object71964:head# 2024-04-20T17:11:09.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cb231b5:::benchmark_data_smithi171_78918_object131208:head# 2024-04-20T17:11:09.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cb24b10:::benchmark_data_smithi171_78918_object130749:head# 2024-04-20T17:11:09.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cb24ca7:::benchmark_data_smithi171_78918_object156466:head# 2024-04-20T17:11:09.162 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cb36efb:::benchmark_data_smithi171_78918_object60891:head# 2024-04-20T17:11:09.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cb3d96e:::benchmark_data_smithi171_78918_object149709:head# 2024-04-20T17:11:09.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cb43401:::benchmark_data_smithi171_78918_object159766:head# 2024-04-20T17:11:09.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cb59678:::benchmark_data_smithi171_78918_object135861:head# 2024-04-20T17:11:09.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cb5edea:::benchmark_data_smithi171_78918_object69358:head# 2024-04-20T17:11:09.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cb5ffb8:::benchmark_data_smithi171_78918_object149227:head# 2024-04-20T17:11:09.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cb6b907:::benchmark_data_smithi171_78918_object133914:head# 2024-04-20T17:11:09.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cb77713:::benchmark_data_smithi171_78918_object153194:head# 2024-04-20T17:11:09.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cb7c3c8:::benchmark_data_smithi171_78918_object53462:head# 2024-04-20T17:11:09.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cb7d320:::benchmark_data_smithi171_78918_object98351:head# 2024-04-20T17:11:09.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cb7ea26:::benchmark_data_smithi171_78918_object141174:head# 2024-04-20T17:11:09.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cb814bf:::benchmark_data_smithi171_78918_object148910:head# 2024-04-20T17:11:09.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cb85b88:::benchmark_data_smithi171_78918_object125057:head# 2024-04-20T17:11:09.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cb87f18:::benchmark_data_smithi171_78918_object154138:head# 2024-04-20T17:11:09.163 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cb8e16f:::benchmark_data_smithi171_78918_object59385:head# 2024-04-20T17:11:09.164 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cba2435:::benchmark_data_smithi171_78918_object120640:head# 2024-04-20T17:11:09.164 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cbb9f0c:::benchmark_data_smithi171_78918_object91490:head# 2024-04-20T17:11:09.164 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cbba842:::benchmark_data_smithi171_78918_object53406:head# 2024-04-20T17:11:09.164 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cbc01cb:::benchmark_data_smithi171_78918_object78230:head# 2024-04-20T17:11:09.164 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cbd5409:::benchmark_data_smithi171_78918_object146526:head# 2024-04-20T17:11:09.164 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cbd9a0e:::benchmark_data_smithi171_78918_object108207:head# 2024-04-20T17:11:09.164 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cbe581d:::benchmark_data_smithi171_78918_object152315:head# 2024-04-20T17:11:09.164 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cbfd1ac:::benchmark_data_smithi171_78918_object143120:head# 2024-04-20T17:11:09.164 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cc23b60:::benchmark_data_smithi171_78918_object159825:head# 2024-04-20T17:11:09.164 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cc32604:::benchmark_data_smithi171_78918_object56104:head# 2024-04-20T17:11:09.164 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cc44dbd:::benchmark_data_smithi171_78918_object128119:head# 2024-04-20T17:11:09.164 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cc46e50:::benchmark_data_smithi171_78918_object75422:head# 2024-04-20T17:11:09.164 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cc76391:::benchmark_data_smithi171_78918_object158815:head# 2024-04-20T17:11:09.164 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cc91af6:::benchmark_data_smithi171_78918_object62385:head# 2024-04-20T17:11:09.165 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ccad61b:::benchmark_data_smithi171_78918_object79853:head# 2024-04-20T17:11:09.165 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ccca959:::benchmark_data_smithi171_78918_object144023:head# 2024-04-20T17:11:09.165 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ccd8c1d:::benchmark_data_smithi171_78918_object72249:head# 2024-04-20T17:11:09.165 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ccdb0f2:::benchmark_data_smithi171_78918_object93647:head# 2024-04-20T17:11:09.165 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ccdbcf4:::benchmark_data_smithi171_78918_object125866:head# 2024-04-20T17:11:09.165 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cce3477:::benchmark_data_smithi171_78918_object153551:head# 2024-04-20T17:11:09.165 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cceaa9e:::benchmark_data_smithi171_78918_object144589:head# 2024-04-20T17:11:09.165 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ccf441e:::benchmark_data_smithi171_78918_object133415:head# 2024-04-20T17:11:09.165 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cd0487f:::benchmark_data_smithi171_78918_object54709:head# 2024-04-20T17:11:09.165 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cd14069:::benchmark_data_smithi171_78918_object157622:head# 2024-04-20T17:11:09.165 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cd1b320:::benchmark_data_smithi171_78918_object129014:head# 2024-04-20T17:11:09.165 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cd1d0e5:::benchmark_data_smithi171_78918_object121725:head# 2024-04-20T17:11:09.165 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cd3b2b5:::benchmark_data_smithi171_78918_object160119:head# 2024-04-20T17:11:09.165 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cd49e8f:::benchmark_data_smithi171_78918_object128957:head# 2024-04-20T17:11:09.166 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cd4b995:::benchmark_data_smithi171_78918_object61706:head# 2024-04-20T17:11:09.166 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cd5abbe:::benchmark_data_smithi171_78918_object155385:head# 2024-04-20T17:11:09.166 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cd5c987:::benchmark_data_smithi171_78918_object97476:head# 2024-04-20T17:11:09.166 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cd61314:::benchmark_data_smithi171_78918_object110465:head# 2024-04-20T17:11:09.166 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cd61b58:::benchmark_data_smithi171_78918_object101050:head# 2024-04-20T17:11:09.166 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cd61b68:::benchmark_data_smithi171_78918_object140360:head# 2024-04-20T17:11:09.166 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cd6dd2b:::benchmark_data_smithi171_78918_object151788:head# 2024-04-20T17:11:09.166 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cd75d32:::benchmark_data_smithi171_78918_object105688:head# 2024-04-20T17:11:09.166 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cd8a6f1:::benchmark_data_smithi171_78918_object73635:head# 2024-04-20T17:11:09.166 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cd90640:::benchmark_data_smithi171_78918_object69225:head# 2024-04-20T17:11:09.166 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cd966cb:::benchmark_data_smithi171_78918_object108546:head# 2024-04-20T17:11:09.166 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cda0fd9:::benchmark_data_smithi171_78918_object128475:head# 2024-04-20T17:11:09.166 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cdb7ed3:::benchmark_data_smithi171_78918_object107677:head# 2024-04-20T17:11:09.166 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cdc14f2:::benchmark_data_smithi171_78918_object81716:head# 2024-04-20T17:11:09.167 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cdc7fec:::benchmark_data_smithi171_78918_object100675:head# 2024-04-20T17:11:09.167 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cdd3c1e:::benchmark_data_smithi171_78918_object79263:head# 2024-04-20T17:11:09.167 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cde0245:::benchmark_data_smithi171_78918_object158234:head# 2024-04-20T17:11:09.167 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cdf2203:::benchmark_data_smithi171_78918_object88576:head# 2024-04-20T17:11:09.167 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cdf612a:::benchmark_data_smithi171_78918_object118950:head# 2024-04-20T17:11:09.167 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cdf923d:::benchmark_data_smithi171_78918_object111458:head# 2024-04-20T17:11:09.167 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cdfe463:::benchmark_data_smithi171_78918_object135831:head# 2024-04-20T17:11:09.167 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ce03c66:::benchmark_data_smithi171_78918_object64061:head# 2024-04-20T17:11:09.167 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ce05365:::benchmark_data_smithi171_78918_object130742:head# 2024-04-20T17:11:09.167 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ce0caed:::benchmark_data_smithi171_78918_object128931:head# 2024-04-20T17:11:09.167 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ce19355:::benchmark_data_smithi171_78918_object139776:head# 2024-04-20T17:11:09.167 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ce1a8e8:::benchmark_data_smithi171_78918_object137995:head# 2024-04-20T17:11:09.167 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ce2addf:::benchmark_data_smithi171_78918_object103641:head# 2024-04-20T17:11:09.168 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ce2cfe7:::benchmark_data_smithi171_78918_object137231:head# 2024-04-20T17:11:09.168 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ce35f67:::benchmark_data_smithi171_78918_object105603:head# 2024-04-20T17:11:09.168 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ce414ca:::benchmark_data_smithi171_78918_object70579:head# 2024-04-20T17:11:09.168 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ce52720:::benchmark_data_smithi171_78918_object65924:head# 2024-04-20T17:11:09.168 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ce614a3:::benchmark_data_smithi171_78918_object91671:head# 2024-04-20T17:11:09.168 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ce722ee:::benchmark_data_smithi171_78918_object145013:head# 2024-04-20T17:11:09.168 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ce7f141:::benchmark_data_smithi171_78918_object71264:head# 2024-04-20T17:11:09.168 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ce881f1:::benchmark_data_smithi171_78918_object130335:head# 2024-04-20T17:11:09.168 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ce8b9e3:::benchmark_data_smithi171_78918_object118186:head# 2024-04-20T17:11:09.168 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ce906d1:::benchmark_data_smithi171_78918_object135330:head# 2024-04-20T17:11:09.168 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ce92bf2:::benchmark_data_smithi171_78918_object58183:head# 2024-04-20T17:11:09.168 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ce93433:::benchmark_data_smithi171_78918_object142548:head# 2024-04-20T17:11:09.168 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ce987b8:::benchmark_data_smithi171_78918_object88649:head# 2024-04-20T17:11:09.168 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ce9dde8:::benchmark_data_smithi171_78918_object73591:head# 2024-04-20T17:11:09.168 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cea4f61:::benchmark_data_smithi171_78918_object104994:head# 2024-04-20T17:11:09.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ceb4724:::benchmark_data_smithi171_78918_object91085:head# 2024-04-20T17:11:09.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cec097e:::benchmark_data_smithi171_78918_object122007:head# 2024-04-20T17:11:09.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cec709d:::benchmark_data_smithi171_78918_object105935:head# 2024-04-20T17:11:09.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cedb906:::benchmark_data_smithi171_78918_object122028:head# 2024-04-20T17:11:09.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cedcbb8:::benchmark_data_smithi171_78918_object90007:head# 2024-04-20T17:11:09.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cee355b:::benchmark_data_smithi171_78918_object98286:head# 2024-04-20T17:11:09.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cef32e2:::benchmark_data_smithi171_78918_object61103:head# 2024-04-20T17:11:09.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cf05c4a:::benchmark_data_smithi171_78918_object104667:head# 2024-04-20T17:11:09.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cf07c4b:::benchmark_data_smithi171_78918_object153189:head# 2024-04-20T17:11:09.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cf0939f:::benchmark_data_smithi171_78918_object63801:head# 2024-04-20T17:11:09.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cf0a9a2:::benchmark_data_smithi171_78918_object55560:head# 2024-04-20T17:11:09.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cf0b434:::benchmark_data_smithi171_78918_object106100:head# 2024-04-20T17:11:09.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cf0ed17:::benchmark_data_smithi171_78918_object115917:head# 2024-04-20T17:11:09.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cf12525:::benchmark_data_smithi171_78918_object75505:head# 2024-04-20T17:11:09.169 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cf14ef7:::benchmark_data_smithi171_78918_object111654:head# 2024-04-20T17:11:09.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cf1bbfa:::benchmark_data_smithi171_78918_object81741:head# 2024-04-20T17:11:09.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cf218df:::benchmark_data_smithi171_78918_object128061:head# 2024-04-20T17:11:09.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cf25070:::benchmark_data_smithi171_78918_object121622:head# 2024-04-20T17:11:09.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cf25aea:::benchmark_data_smithi171_78918_object96973:head# 2024-04-20T17:11:09.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cf26421:::benchmark_data_smithi171_78918_object143175:head# 2024-04-20T17:11:09.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cf38f34:::benchmark_data_smithi171_78918_object106395:head# 2024-04-20T17:11:09.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cf3f7e3:::benchmark_data_smithi171_78918_object116617:head# 2024-04-20T17:11:09.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cf4271e:::benchmark_data_smithi171_78918_object143296:head# 2024-04-20T17:11:09.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cf68b07:::benchmark_data_smithi171_78918_object83107:head# 2024-04-20T17:11:09.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cf6ce58:::benchmark_data_smithi171_78918_object95359:head# 2024-04-20T17:11:09.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cf787b3:::benchmark_data_smithi171_78918_object103027:head# 2024-04-20T17:11:09.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cf80480:::benchmark_data_smithi171_78918_object68255:head# 2024-04-20T17:11:09.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cf83bdc:::benchmark_data_smithi171_78918_object102410:head# 2024-04-20T17:11:09.170 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cf91957:::benchmark_data_smithi171_78918_object102765:head# 2024-04-20T17:11:09.171 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cf9d8dd:::benchmark_data_smithi171_78918_object119313:head# 2024-04-20T17:11:09.171 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cf9ea3c:::benchmark_data_smithi171_78918_object129652:head# 2024-04-20T17:11:09.171 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cfa2ae4:::benchmark_data_smithi171_78918_object89463:head# 2024-04-20T17:11:09.171 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cfb188d:::benchmark_data_smithi171_78918_object73048:head# 2024-04-20T17:11:09.171 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cfb9134:::benchmark_data_smithi171_78918_object101337:head# 2024-04-20T17:11:09.171 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cfbbdca:::benchmark_data_smithi171_78918_object99763:head# 2024-04-20T17:11:09.171 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cfebc94:::benchmark_data_smithi171_78918_object123514:head# 2024-04-20T17:11:09.171 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cfeee65:::benchmark_data_smithi171_78918_object93253:head# 2024-04-20T17:11:09.171 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cfef941:::benchmark_data_smithi171_78918_object104678:head# 2024-04-20T17:11:09.171 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7cffda2d:::benchmark_data_smithi171_78918_object66745:head# 2024-04-20T17:11:09.171 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d0156bf:::benchmark_data_smithi171_78918_object140632:head# 2024-04-20T17:11:09.171 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d02df28:::benchmark_data_smithi171_78918_object158491:head# 2024-04-20T17:11:09.171 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d035b1e:::benchmark_data_smithi171_78918_object97613:head# 2024-04-20T17:11:09.171 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d044c43:::benchmark_data_smithi171_78918_object57124:head# 2024-04-20T17:11:09.171 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d049775:::benchmark_data_smithi171_78918_object60562:head# 2024-04-20T17:11:09.172 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d04b8f2:::benchmark_data_smithi171_78918_object99282:head# 2024-04-20T17:11:09.172 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d05a332:::benchmark_data_smithi171_78918_object112465:head# 2024-04-20T17:11:09.172 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d06b8a5:::benchmark_data_smithi171_78918_object115530:head# 2024-04-20T17:11:09.172 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d07c8f6:::benchmark_data_smithi171_78918_object94762:head# 2024-04-20T17:11:09.172 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d07dc8d:::benchmark_data_smithi171_78918_object70494:head# 2024-04-20T17:11:09.172 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d08cf5f:::benchmark_data_smithi171_78918_object136932:head# 2024-04-20T17:11:09.172 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d09ccaa:::benchmark_data_smithi171_78918_object75950:head# 2024-04-20T17:11:09.172 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d0a1494:::benchmark_data_smithi171_78918_object65314:head# 2024-04-20T17:11:09.172 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d0a3ee3:::benchmark_data_smithi171_78918_object109487:head# 2024-04-20T17:11:09.172 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d0b132f:::benchmark_data_smithi171_78918_object81929:head# 2024-04-20T17:11:09.172 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d0b9075:::benchmark_data_smithi171_78918_object158695:head# 2024-04-20T17:11:09.172 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d0d0868:::benchmark_data_smithi171_78918_object101495:head# 2024-04-20T17:11:09.172 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d0d4761:::benchmark_data_smithi171_78918_object101497:head# 2024-04-20T17:11:09.172 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d0d4f67:::benchmark_data_smithi171_78918_object62238:head# 2024-04-20T17:11:09.172 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d0d9fc3:::benchmark_data_smithi171_78918_object111066:head# 2024-04-20T17:11:09.173 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d0effe0:::benchmark_data_smithi171_78918_object70264:head# 2024-04-20T17:11:09.173 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d105384:::benchmark_data_smithi171_78918_object97316:head# 2024-04-20T17:11:09.173 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d108c41:::benchmark_data_smithi171_78918_object133347:head# 2024-04-20T17:11:09.173 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d111656:::benchmark_data_smithi171_78918_object86198:head# 2024-04-20T17:11:09.173 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d113741:::benchmark_data_smithi171_78918_object127508:head# 2024-04-20T17:11:09.173 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d124902:::benchmark_data_smithi171_78918_object57008:head# 2024-04-20T17:11:09.173 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d13167b:::benchmark_data_smithi171_78918_object129126:head# 2024-04-20T17:11:09.173 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d139e0d:::benchmark_data_smithi171_78918_object63636:head# 2024-04-20T17:11:09.173 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d1408b1:::benchmark_data_smithi171_78918_object97377:head# 2024-04-20T17:11:09.173 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d161ced:::benchmark_data_smithi171_78918_object139408:head# 2024-04-20T17:11:09.173 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d162cf6:::benchmark_data_smithi171_78918_object65667:head# 2024-04-20T17:11:09.173 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d165491:::benchmark_data_smithi171_78918_object75908:head# 2024-04-20T17:11:09.173 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d16680c:::benchmark_data_smithi171_78918_object91653:head# 2024-04-20T17:11:09.173 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d1694a3:::benchmark_data_smithi171_78918_object57868:head# 2024-04-20T17:11:09.173 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d16f632:::benchmark_data_smithi171_78918_object108528:head# 2024-04-20T17:11:09.174 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d174e97:::benchmark_data_smithi171_78918_object122425:head# 2024-04-20T17:11:09.174 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d17a2c4:::benchmark_data_smithi171_78918_object80524:head# 2024-04-20T17:11:09.174 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d1830aa:::benchmark_data_smithi171_78918_object79364:head# 2024-04-20T17:11:09.174 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d185e52:::benchmark_data_smithi171_78918_object63059:head# 2024-04-20T17:11:09.174 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d18727b:::benchmark_data_smithi171_78918_object60648:head# 2024-04-20T17:11:09.174 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d1984bb:::benchmark_data_smithi171_78918_object90436:head# 2024-04-20T17:11:09.174 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d19a8c7:::benchmark_data_smithi171_78918_object108608:head# 2024-04-20T17:11:09.174 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d19f4f9:::benchmark_data_smithi171_78918_object147973:head# 2024-04-20T17:11:09.174 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d1a1b34:::benchmark_data_smithi171_78918_object151920:head# 2024-04-20T17:11:09.174 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d1af9b5:::benchmark_data_smithi171_78918_object122477:head# 2024-04-20T17:11:09.174 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d1bbd0f:::benchmark_data_smithi171_78918_object101577:head# 2024-04-20T17:11:09.174 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d1bd950:::benchmark_data_smithi171_78918_object88900:head# 2024-04-20T17:11:09.174 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d1c048f:::benchmark_data_smithi171_78918_object145149:head# 2024-04-20T17:11:09.174 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d1c5431:::benchmark_data_smithi171_78918_object97704:head# 2024-04-20T17:11:09.175 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d1e5c18:::benchmark_data_smithi171_78918_object158064:head# 2024-04-20T17:11:09.175 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d1f4221:::benchmark_data_smithi171_78918_object158350:head# 2024-04-20T17:11:09.175 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d2037a9:::benchmark_data_smithi171_78918_object83682:head# 2024-04-20T17:11:09.175 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d20a03c:::benchmark_data_smithi171_78918_object138987:head# 2024-04-20T17:11:09.175 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d21ccf7:::benchmark_data_smithi171_78918_object116915:head# 2024-04-20T17:11:09.175 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d21f53f:::benchmark_data_smithi171_78918_object51846:head# 2024-04-20T17:11:09.175 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d226f34:::benchmark_data_smithi171_78918_object150659:head# 2024-04-20T17:11:09.175 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d22a3e3:::benchmark_data_smithi171_78918_object109646:head# 2024-04-20T17:11:09.175 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d22e0b0:::benchmark_data_smithi171_78918_object91276:head# 2024-04-20T17:11:09.175 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d235139:::benchmark_data_smithi171_78918_object57972:head# 2024-04-20T17:11:09.175 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d23def6:::benchmark_data_smithi171_78918_object131823:head# 2024-04-20T17:11:09.175 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d24e018:::benchmark_data_smithi171_78918_object127168:head# 2024-04-20T17:11:09.175 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d24ee04:::benchmark_data_smithi171_78918_object110920:head# 2024-04-20T17:11:09.175 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d252c4a:::benchmark_data_smithi171_78918_object135583:head# 2024-04-20T17:11:09.176 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d252e78:::benchmark_data_smithi171_78918_object52203:head# 2024-04-20T17:11:09.176 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d25897e:::benchmark_data_smithi171_78918_object64819:head# 2024-04-20T17:11:09.176 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d259341:::benchmark_data_smithi171_78918_object73852:head# 2024-04-20T17:11:09.176 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d25b964:::benchmark_data_smithi171_78918_object115760:head# 2024-04-20T17:11:09.176 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d269800:::benchmark_data_smithi171_78918_object74167:head# 2024-04-20T17:11:09.176 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d26fe58:::benchmark_data_smithi171_78918_object130373:head# 2024-04-20T17:11:09.176 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d28856d:::benchmark_data_smithi171_78918_object106880:head# 2024-04-20T17:11:09.176 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d294e30:::benchmark_data_smithi171_78918_object98149:head# 2024-04-20T17:11:09.176 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d29c4a7:::benchmark_data_smithi171_78918_object62055:head# 2024-04-20T17:11:09.176 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d29de13:::benchmark_data_smithi171_78918_object130311:head# 2024-04-20T17:11:09.176 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d2a1d00:::benchmark_data_smithi171_78918_object150334:head# 2024-04-20T17:11:09.176 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d2ab24c:::benchmark_data_smithi171_78918_object61990:head# 2024-04-20T17:11:09.176 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d2af757:::benchmark_data_smithi171_78918_object132646:head# 2024-04-20T17:11:09.176 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d2b91ab:::benchmark_data_smithi171_78918_object58121:head# 2024-04-20T17:11:09.177 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d2c5fa0:::benchmark_data_smithi171_78918_object63944:head# 2024-04-20T17:11:09.177 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d2c7dbe:::benchmark_data_smithi171_78918_object108204:head# 2024-04-20T17:11:09.177 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d2c9d98:::benchmark_data_smithi171_78918_object125553:head# 2024-04-20T17:11:09.177 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d2d2c7a:::benchmark_data_smithi171_78918_object108978:head# 2024-04-20T17:11:09.177 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d2dd40b:::benchmark_data_smithi171_78918_object108750:head# 2024-04-20T17:11:09.177 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d2de0b6:::benchmark_data_smithi171_78918_object123670:head# 2024-04-20T17:11:09.177 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d2df290:::benchmark_data_smithi171_78918_object73442:head# 2024-04-20T17:11:09.177 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d2eac87:::benchmark_data_smithi171_78918_object116421:head# 2024-04-20T17:11:09.177 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d300538:::benchmark_data_smithi171_78918_object84951:head# 2024-04-20T17:11:09.177 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d3048bc:::benchmark_data_smithi171_78918_object130224:head# 2024-04-20T17:11:09.177 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d308df4:::benchmark_data_smithi171_78918_object105355:head# 2024-04-20T17:11:09.177 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d30b2b8:::benchmark_data_smithi171_78918_object158606:head# 2024-04-20T17:11:09.177 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d30cd8b:::benchmark_data_smithi171_78918_object65077:head# 2024-04-20T17:11:09.177 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d317762:::benchmark_data_smithi171_78918_object75611:head# 2024-04-20T17:11:09.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d31b2ac:::benchmark_data_smithi171_78918_object138091:head# 2024-04-20T17:11:09.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d31fce1:::benchmark_data_smithi171_78918_object127811:head# 2024-04-20T17:11:09.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d32903d:::benchmark_data_smithi171_78918_object119894:head# 2024-04-20T17:11:09.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d33bda7:::benchmark_data_smithi171_78918_object123444:head# 2024-04-20T17:11:09.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d349713:::benchmark_data_smithi171_78918_object126264:head# 2024-04-20T17:11:09.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d34c3af:::benchmark_data_smithi171_78918_object132455:head# 2024-04-20T17:11:09.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d355592:::benchmark_data_smithi171_78918_object80741:head# 2024-04-20T17:11:09.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d358d92:::benchmark_data_smithi171_78918_object70720:head# 2024-04-20T17:11:09.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d36a98a:::benchmark_data_smithi171_78918_object108782:head# 2024-04-20T17:11:09.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d36dd27:::benchmark_data_smithi171_78918_object104208:head# 2024-04-20T17:11:09.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d376563:::benchmark_data_smithi171_78918_object61245:head# 2024-04-20T17:11:09.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d380e6a:::benchmark_data_smithi171_78918_object69386:head# 2024-04-20T17:11:09.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d391c1b:::benchmark_data_smithi171_78918_object100676:head# 2024-04-20T17:11:09.178 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d39e72e:::benchmark_data_smithi171_78918_object108191:head# 2024-04-20T17:11:09.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d3aedc8:::benchmark_data_smithi171_78918_object79739:head# 2024-04-20T17:11:09.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d3c6064:::benchmark_data_smithi171_78918_object108205:head# 2024-04-20T17:11:09.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d3ccbbc:::benchmark_data_smithi171_78918_object133322:head# 2024-04-20T17:11:09.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d3ce6e1:::benchmark_data_smithi171_78918_object145231:head# 2024-04-20T17:11:09.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d3cec1a:::benchmark_data_smithi171_78918_object109571:head# 2024-04-20T17:11:09.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d3e309d:::benchmark_data_smithi171_78918_object143406:head# 2024-04-20T17:11:09.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d3ea5b4:::benchmark_data_smithi171_78918_object52247:head# 2024-04-20T17:11:09.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d3eff99:::benchmark_data_smithi171_78918_object113571:head# 2024-04-20T17:11:09.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d3f6c74:::benchmark_data_smithi171_78918_object71948:head# 2024-04-20T17:11:09.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d3f9d9e:::benchmark_data_smithi171_78918_object72879:head# 2024-04-20T17:11:09.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d4068f0:::benchmark_data_smithi171_78918_object114666:head# 2024-04-20T17:11:09.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d407772:::benchmark_data_smithi171_78918_object115065:head# 2024-04-20T17:11:09.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d40c941:::benchmark_data_smithi171_78918_object67852:head# 2024-04-20T17:11:09.179 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d40cd17:::benchmark_data_smithi171_78918_object137885:head# 2024-04-20T17:11:09.180 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d40f1fc:::benchmark_data_smithi171_78918_object76243:head# 2024-04-20T17:11:09.180 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d4148c7:::benchmark_data_smithi171_78918_object156366:head# 2024-04-20T17:11:09.180 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d419604:::benchmark_data_smithi171_78918_object103001:head# 2024-04-20T17:11:09.180 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d41fb95:::benchmark_data_smithi171_78918_object54693:head# 2024-04-20T17:11:09.180 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d43cf43:::benchmark_data_smithi171_78918_object76010:head# 2024-04-20T17:11:09.180 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d44af4e:::benchmark_data_smithi171_78918_object113584:head# 2024-04-20T17:11:09.180 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d4538ff:::benchmark_data_smithi171_78918_object108625:head# 2024-04-20T17:11:09.180 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d466012:::benchmark_data_smithi171_78918_object70671:head# 2024-04-20T17:11:09.180 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d494529:::benchmark_data_smithi171_78918_object57083:head# 2024-04-20T17:11:09.180 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d4a08c8:::benchmark_data_smithi171_78918_object84194:head# 2024-04-20T17:11:09.180 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d4aa2a7:::benchmark_data_smithi171_78918_object83302:head# 2024-04-20T17:11:09.180 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d4b080f:::benchmark_data_smithi171_78918_object105525:head# 2024-04-20T17:11:09.180 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d4c3963:::benchmark_data_smithi171_78918_object119719:head# 2024-04-20T17:11:09.180 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d4c80db:::benchmark_data_smithi171_78918_object152434:head# 2024-04-20T17:11:09.181 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d4cefac:::benchmark_data_smithi171_78918_object55164:head# 2024-04-20T17:11:09.181 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d4d704c:::benchmark_data_smithi171_78918_object116778:head# 2024-04-20T17:11:09.181 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d4dae9e:::benchmark_data_smithi171_78918_object69413:head# 2024-04-20T17:11:09.181 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d4dd2c9:::benchmark_data_smithi171_78918_object125769:head# 2024-04-20T17:11:09.181 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d4e59c6:::benchmark_data_smithi171_78918_object56267:head# 2024-04-20T17:11:09.181 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d4fbe9e:::benchmark_data_smithi171_78918_object122681:head# 2024-04-20T17:11:09.181 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d4feacc:::benchmark_data_smithi171_78918_object82358:head# 2024-04-20T17:11:09.181 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d506516:::benchmark_data_smithi171_78918_object106188:head# 2024-04-20T17:11:09.181 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d516613:::benchmark_data_smithi171_78918_object61327:head# 2024-04-20T17:11:09.181 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d524295:::benchmark_data_smithi171_78918_object127714:head# 2024-04-20T17:11:09.181 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d524b4d:::benchmark_data_smithi171_78918_object87086:head# 2024-04-20T17:11:09.181 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d52de91:::benchmark_data_smithi171_78918_object74209:head# 2024-04-20T17:11:09.181 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d53bdbc:::benchmark_data_smithi171_78918_object115359:head# 2024-04-20T17:11:09.181 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d53d863:::benchmark_data_smithi171_78918_object124077:head# 2024-04-20T17:11:09.182 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d556759:::benchmark_data_smithi171_78918_object58145:head# 2024-04-20T17:11:09.182 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d55b4b1:::benchmark_data_smithi171_78918_object99588:head# 2024-04-20T17:11:09.182 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d55d48a:::benchmark_data_smithi171_78918_object92040:head# 2024-04-20T17:11:09.182 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d56053e:::benchmark_data_smithi171_78918_object94826:head# 2024-04-20T17:11:09.182 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d57315b:::benchmark_data_smithi171_78918_object130617:head# 2024-04-20T17:11:09.182 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d577491:::benchmark_data_smithi171_78918_object108096:head# 2024-04-20T17:11:09.182 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d57846d:::benchmark_data_smithi171_78918_object86611:head# 2024-04-20T17:11:09.182 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d586d46:::benchmark_data_smithi171_78918_object147519:head# 2024-04-20T17:11:09.182 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d58d052:::benchmark_data_smithi171_78918_object64064:head# 2024-04-20T17:11:09.182 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d593344:::benchmark_data_smithi171_78918_object144081:head# 2024-04-20T17:11:09.182 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d5977fa:::benchmark_data_smithi171_78918_object120913:head# 2024-04-20T17:11:09.182 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d59a7c2:::benchmark_data_smithi171_78918_object131692:head# 2024-04-20T17:11:09.182 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d59febd:::benchmark_data_smithi171_78918_object73906:head# 2024-04-20T17:11:09.182 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d5b345c:::benchmark_data_smithi171_78918_object151569:head# 2024-04-20T17:11:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d5b866c:::benchmark_data_smithi171_78918_object51797:head# 2024-04-20T17:11:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d5c44ab:::benchmark_data_smithi171_78918_object148551:head# 2024-04-20T17:11:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d5d3fe7:::benchmark_data_smithi171_78918_object71940:head# 2024-04-20T17:11:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d5d98d3:::benchmark_data_smithi171_78918_object160734:head# 2024-04-20T17:11:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d5dce2e:::benchmark_data_smithi171_78918_object65340:head# 2024-04-20T17:11:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d5e629b:::benchmark_data_smithi171_78918_object94070:head# 2024-04-20T17:11:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d5f9fb8:::benchmark_data_smithi171_78918_object160646:head# 2024-04-20T17:11:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d6092f6:::benchmark_data_smithi171_78918_object130276:head# 2024-04-20T17:11:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d60f8a9:::benchmark_data_smithi171_78918_object156110:head# 2024-04-20T17:11:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d61d4b7:::benchmark_data_smithi171_78918_object153414:head# 2024-04-20T17:11:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d61d842:::benchmark_data_smithi171_78918_object152290:head# 2024-04-20T17:11:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d61dea3:::benchmark_data_smithi171_78918_object68128:head# 2024-04-20T17:11:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d620944:::benchmark_data_smithi171_78918_object120975:head# 2024-04-20T17:11:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d638429:::benchmark_data_smithi171_78918_object149473:head# 2024-04-20T17:11:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d6435b2:::benchmark_data_smithi171_78918_object118030:head# 2024-04-20T17:11:09.184 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d643c48:::benchmark_data_smithi171_78918_object60980:head# 2024-04-20T17:11:09.184 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d6502a8:::benchmark_data_smithi171_78918_object78313:head# 2024-04-20T17:11:09.184 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d65b927:::benchmark_data_smithi171_78918_object146225:head# 2024-04-20T17:11:09.184 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d65bae8:::benchmark_data_smithi171_78918_object129397:head# 2024-04-20T17:11:09.184 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d662e4d:::benchmark_data_smithi171_78918_object155536:head# 2024-04-20T17:11:09.184 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d66bbce:::benchmark_data_smithi171_78918_object111941:head# 2024-04-20T17:11:09.184 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d67184e:::benchmark_data_smithi171_78918_object77232:head# 2024-04-20T17:11:09.184 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d677e76:::benchmark_data_smithi171_78918_object57184:head# 2024-04-20T17:11:09.184 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d682021:::benchmark_data_smithi171_78918_object158226:head# 2024-04-20T17:11:09.184 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d6893c1:::benchmark_data_smithi171_78918_object94844:head# 2024-04-20T17:11:09.184 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d699c9f:::benchmark_data_smithi171_78918_object78489:head# 2024-04-20T17:11:09.184 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d69a137:::benchmark_data_smithi171_78918_object87377:head# 2024-04-20T17:11:09.184 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d6aee9d:::benchmark_data_smithi171_78918_object149374:head# 2024-04-20T17:11:09.184 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d6c5d33:::benchmark_data_smithi171_78918_object87486:head# 2024-04-20T17:11:09.185 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d6c8d99:::benchmark_data_smithi171_78918_object98675:head# 2024-04-20T17:11:09.185 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d6d801a:::benchmark_data_smithi171_78918_object129340:head# 2024-04-20T17:11:09.185 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d6dd465:::benchmark_data_smithi171_78918_object78899:head# 2024-04-20T17:11:09.185 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d6eb1ff:::benchmark_data_smithi171_78918_object52079:head# 2024-04-20T17:11:09.185 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d6ec749:::benchmark_data_smithi171_78918_object116979:head# 2024-04-20T17:11:09.185 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d6f526d:::benchmark_data_smithi171_78918_object145576:head# 2024-04-20T17:11:09.185 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d6f6f82:::benchmark_data_smithi171_78918_object154657:head# 2024-04-20T17:11:09.185 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d6fa128:::benchmark_data_smithi171_78918_object108697:head# 2024-04-20T17:11:09.185 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d704380:::benchmark_data_smithi171_78918_object66405:head# 2024-04-20T17:11:09.185 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d7115f1:::benchmark_data_smithi171_78918_object110157:head# 2024-04-20T17:11:09.185 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d725acd:::benchmark_data_smithi171_78918_object160874:head# 2024-04-20T17:11:09.185 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d73731a:::benchmark_data_smithi171_78918_object123913:head# 2024-04-20T17:11:09.185 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d74005a:::benchmark_data_smithi171_78918_object64197:head# 2024-04-20T17:11:09.185 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d743e69:::benchmark_data_smithi171_78918_object98117:head# 2024-04-20T17:11:09.186 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d758c93:::benchmark_data_smithi171_78918_object85982:head# 2024-04-20T17:11:09.186 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d75a47d:::benchmark_data_smithi171_78918_object110650:head# 2024-04-20T17:11:09.186 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d76f612:::benchmark_data_smithi171_78918_object131383:head# 2024-04-20T17:11:09.186 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d76fbbe:::benchmark_data_smithi171_78918_object70067:head# 2024-04-20T17:11:09.186 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d7752a1:::benchmark_data_smithi171_78918_object84844:head# 2024-04-20T17:11:09.186 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d78d5c1:::benchmark_data_smithi171_78918_object139730:head# 2024-04-20T17:11:09.186 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d799a42:::benchmark_data_smithi171_78918_object79614:head# 2024-04-20T17:11:09.186 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d7a45a6:::benchmark_data_smithi171_78918_object77484:head# 2024-04-20T17:11:09.186 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d7ba25d:::benchmark_data_smithi171_78918_object101476:head# 2024-04-20T17:11:09.186 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d7bb5cb:::benchmark_data_smithi171_78918_object118529:head# 2024-04-20T17:11:09.186 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d7bfdac:::benchmark_data_smithi171_78918_object116028:head# 2024-04-20T17:11:09.186 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d7ce577:::benchmark_data_smithi171_78918_object60025:head# 2024-04-20T17:11:09.186 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d7de219:::benchmark_data_smithi171_78918_object93053:head# 2024-04-20T17:11:09.186 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d7ebb2d:::benchmark_data_smithi171_78918_object143051:head# 2024-04-20T17:11:09.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d7ec13b:::benchmark_data_smithi171_78918_object130369:head# 2024-04-20T17:11:09.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d8140dc:::benchmark_data_smithi171_78918_object77985:head# 2024-04-20T17:11:09.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d82f644:::benchmark_data_smithi171_78918_object105308:head# 2024-04-20T17:11:09.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d8429cb:::benchmark_data_smithi171_78918_object90506:head# 2024-04-20T17:11:09.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d843b11:::benchmark_data_smithi171_78918_object129332:head# 2024-04-20T17:11:09.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d84a79f:::benchmark_data_smithi171_78918_object86295:head# 2024-04-20T17:11:09.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d84f251:::benchmark_data_smithi171_78918_object118129:head# 2024-04-20T17:11:09.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d8572e3:::benchmark_data_smithi171_78918_object133061:head# 2024-04-20T17:11:09.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d85b4d2:::benchmark_data_smithi171_78918_object154969:head# 2024-04-20T17:11:09.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d85e126:::benchmark_data_smithi171_78918_object53820:head# 2024-04-20T17:11:09.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d861cb5:::benchmark_data_smithi171_78918_object135234:head# 2024-04-20T17:11:09.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d863d39:::benchmark_data_smithi171_78918_object137625:head# 2024-04-20T17:11:09.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d86444c:::benchmark_data_smithi171_78918_object123526:head# 2024-04-20T17:11:09.187 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d86b4af:::benchmark_data_smithi171_78918_object95664:head# 2024-04-20T17:11:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d8714a0:::benchmark_data_smithi171_78918_object124684:head# 2024-04-20T17:11:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d87b64a:::benchmark_data_smithi171_78918_object74104:head# 2024-04-20T17:11:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d880e11:::benchmark_data_smithi171_78918_object116429:head# 2024-04-20T17:11:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d881f27:::benchmark_data_smithi171_78918_object74218:head# 2024-04-20T17:11:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d883aa4:::benchmark_data_smithi171_78918_object104151:head# 2024-04-20T17:11:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d88e73f:::benchmark_data_smithi171_78918_object73340:head# 2024-04-20T17:11:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d88e905:::benchmark_data_smithi171_78918_object56059:head# 2024-04-20T17:11:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d8967ab:::benchmark_data_smithi171_78918_object65974:head# 2024-04-20T17:11:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d8a3bc5:::benchmark_data_smithi171_78918_object75571:head# 2024-04-20T17:11:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d8b5372:::benchmark_data_smithi171_78918_object115889:head# 2024-04-20T17:11:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d8c1254:::benchmark_data_smithi171_78918_object115166:head# 2024-04-20T17:11:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d8c7b70:::benchmark_data_smithi171_78918_object99611:head# 2024-04-20T17:11:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d8cecf1:::benchmark_data_smithi171_78918_object87780:head# 2024-04-20T17:11:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d8d241a:::benchmark_data_smithi171_78918_object99745:head# 2024-04-20T17:11:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d8e1628:::benchmark_data_smithi171_78918_object63456:head# 2024-04-20T17:11:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d8e407a:::benchmark_data_smithi171_78918_object101180:head# 2024-04-20T17:11:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d8e5008:::benchmark_data_smithi171_78918_object88377:head# 2024-04-20T17:11:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d8ee0a7:::benchmark_data_smithi171_78918_object95811:head# 2024-04-20T17:11:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d8f4dad:::benchmark_data_smithi171_78918_object94963:head# 2024-04-20T17:11:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d905bad:::benchmark_data_smithi171_78918_object113329:head# 2024-04-20T17:11:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d91715a:::benchmark_data_smithi171_78918_object139020:head# 2024-04-20T17:11:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d920235:::benchmark_data_smithi171_78918_object67186:head# 2024-04-20T17:11:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d922859:::benchmark_data_smithi171_78918_object86180:head# 2024-04-20T17:11:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d9412df:::benchmark_data_smithi171_78918_object66942:head# 2024-04-20T17:11:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d951415:::benchmark_data_smithi171_78918_object127467:head# 2024-04-20T17:11:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d95f675:::benchmark_data_smithi171_78918_object86444:head# 2024-04-20T17:11:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d96948f:::benchmark_data_smithi171_78918_object119850:head# 2024-04-20T17:11:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d982014:::benchmark_data_smithi171_78918_object86167:head# 2024-04-20T17:11:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d9851c7:::benchmark_data_smithi171_78918_object74170:head# 2024-04-20T17:11:09.190 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d9876ef:::benchmark_data_smithi171_78918_object144458:head# 2024-04-20T17:11:09.190 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d9aec82:::benchmark_data_smithi171_78918_object80333:head# 2024-04-20T17:11:09.190 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d9b2f15:::benchmark_data_smithi171_78918_object118516:head# 2024-04-20T17:11:09.190 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d9c601b:::benchmark_data_smithi171_78918_object66550:head# 2024-04-20T17:11:09.190 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d9d0c97:::benchmark_data_smithi171_78918_object106753:head# 2024-04-20T17:11:09.190 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d9d4850:::benchmark_data_smithi171_78918_object54048:head# 2024-04-20T17:11:09.190 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d9dfbeb:::benchmark_data_smithi171_78918_object121361:head# 2024-04-20T17:11:09.190 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d9f222d:::benchmark_data_smithi171_78918_object119552:head# 2024-04-20T17:11:09.190 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7d9f6228:::benchmark_data_smithi171_78918_object80467:head# 2024-04-20T17:11:09.190 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7da0c1fe:::benchmark_data_smithi171_78918_object129681:head# 2024-04-20T17:11:09.190 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7da1b0bf:::benchmark_data_smithi171_78918_object150678:head# 2024-04-20T17:11:09.190 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7da1d2f3:::benchmark_data_smithi171_78918_object138498:head# 2024-04-20T17:11:09.190 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7da1dc0b:::benchmark_data_smithi171_78918_object81343:head# 2024-04-20T17:11:09.190 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7da208a7:::benchmark_data_smithi171_78918_object74744:head# 2024-04-20T17:11:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7da23e70:::benchmark_data_smithi171_78918_object143957:head# 2024-04-20T17:11:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7da24a35:::benchmark_data_smithi171_78918_object119874:head# 2024-04-20T17:11:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7da34de4:::benchmark_data_smithi171_78918_object152964:head# 2024-04-20T17:11:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7da3ef2d:::benchmark_data_smithi171_78918_object61970:head# 2024-04-20T17:11:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7da4b643:::benchmark_data_smithi171_78918_object150016:head# 2024-04-20T17:11:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7da4da83:::benchmark_data_smithi171_78918_object158745:head# 2024-04-20T17:11:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7da635b0:::benchmark_data_smithi171_78918_object75115:head# 2024-04-20T17:11:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7da8b751:::benchmark_data_smithi171_78918_object83758:head# 2024-04-20T17:11:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7da99c2a:::benchmark_data_smithi171_78918_object93257:head# 2024-04-20T17:11:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7daa8f03:::benchmark_data_smithi171_78918_object61707:head# 2024-04-20T17:11:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7daaaeb2:::benchmark_data_smithi171_78918_object132483:head# 2024-04-20T17:11:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dad59a8:::benchmark_data_smithi171_78918_object75112:head# 2024-04-20T17:11:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dad8442:::benchmark_data_smithi171_78918_object113807:head# 2024-04-20T17:11:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7db0c9f9:::benchmark_data_smithi171_78918_object131877:head# 2024-04-20T17:11:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7db12f0e:::benchmark_data_smithi171_78918_object144437:head# 2024-04-20T17:11:09.192 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7db20459:::benchmark_data_smithi171_78918_object156212:head# 2024-04-20T17:11:09.192 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7db30299:::benchmark_data_smithi171_78918_object104854:head# 2024-04-20T17:11:09.192 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7db39588:::benchmark_data_smithi171_78918_object116025:head# 2024-04-20T17:11:09.192 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7db3df0d:::benchmark_data_smithi171_78918_object124147:head# 2024-04-20T17:11:09.192 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7db4a419:::benchmark_data_smithi171_78918_object132612:head# 2024-04-20T17:11:09.192 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7db510d0:::benchmark_data_smithi171_78918_object60199:head# 2024-04-20T17:11:09.192 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7db5516b:::benchmark_data_smithi171_78918_object83368:head# 2024-04-20T17:11:09.192 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7db66b31:::benchmark_data_smithi171_78918_object133792:head# 2024-04-20T17:11:09.192 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7db6df7a:::benchmark_data_smithi171_78918_object100461:head# 2024-04-20T17:11:09.192 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7db972db:::benchmark_data_smithi171_78918_object97324:head# 2024-04-20T17:11:09.192 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dba0468:::benchmark_data_smithi171_78918_object156214:head# 2024-04-20T17:11:09.192 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dbc88c9:::benchmark_data_smithi171_78918_object144849:head# 2024-04-20T17:11:09.192 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dbcfab7:::benchmark_data_smithi171_78918_object101592:head# 2024-04-20T17:11:09.192 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dbe5bce:::benchmark_data_smithi171_78918_object52818:head# 2024-04-20T17:11:09.192 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dbebea0:::benchmark_data_smithi171_78918_object94681:head# 2024-04-20T17:11:09.193 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dbf6fa2:::benchmark_data_smithi171_78918_object92402:head# 2024-04-20T17:11:09.193 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dbfb53c:::benchmark_data_smithi171_78918_object145617:head# 2024-04-20T17:11:09.193 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dc01b80:::benchmark_data_smithi171_78918_object102416:head# 2024-04-20T17:11:09.193 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dc19f5d:::benchmark_data_smithi171_78918_object85755:head# 2024-04-20T17:11:09.193 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dc2356e:::benchmark_data_smithi171_78918_object100804:head# 2024-04-20T17:11:09.193 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dc27be2:::benchmark_data_smithi171_78918_object146321:head# 2024-04-20T17:11:09.193 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dc2cc11:::benchmark_data_smithi171_78918_object142800:head# 2024-04-20T17:11:09.193 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dc34a84:::benchmark_data_smithi171_78918_object81697:head# 2024-04-20T17:11:09.193 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dc3b529:::benchmark_data_smithi171_78918_object140013:head# 2024-04-20T17:11:09.193 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dc44643:::benchmark_data_smithi171_78918_object141671:head# 2024-04-20T17:11:09.193 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dc546ee:::benchmark_data_smithi171_78918_object143690:head# 2024-04-20T17:11:09.193 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dc57482:::benchmark_data_smithi171_78918_object67991:head# 2024-04-20T17:11:09.193 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dc5b32c:::benchmark_data_smithi171_78918_object105054:head# 2024-04-20T17:11:09.193 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dc5fca2:::benchmark_data_smithi171_78918_object100409:head# 2024-04-20T17:11:09.193 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dc61ed3:::benchmark_data_smithi171_78918_object99956:head# 2024-04-20T17:11:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dc9c99a:::benchmark_data_smithi171_78918_object151856:head# 2024-04-20T17:11:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dcaa244:::benchmark_data_smithi171_78918_object155903:head# 2024-04-20T17:11:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dcabd6c:::benchmark_data_smithi171_78918_object82431:head# 2024-04-20T17:11:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dcaeed1:::benchmark_data_smithi171_78918_object156610:head# 2024-04-20T17:11:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dcb064b:::benchmark_data_smithi171_78918_object129000:head# 2024-04-20T17:11:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dcb64ed:::benchmark_data_smithi171_78918_object98451:head# 2024-04-20T17:11:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dcbac86:::benchmark_data_smithi171_78918_object103886:head# 2024-04-20T17:11:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dcbc8b8:::benchmark_data_smithi171_78918_object128731:head# 2024-04-20T17:11:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dcbe101:::benchmark_data_smithi171_78918_object79423:head# 2024-04-20T17:11:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dcc59ed:::benchmark_data_smithi171_78918_object85246:head# 2024-04-20T17:11:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dcf4aaf:::benchmark_data_smithi171_78918_object100878:head# 2024-04-20T17:11:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dd023a5:::benchmark_data_smithi171_78918_object146093:head# 2024-04-20T17:11:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dd0a3dc:::benchmark_data_smithi171_78918_object145161:head# 2024-04-20T17:11:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dd19eb8:::benchmark_data_smithi171_78918_object115961:head# 2024-04-20T17:11:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dd2819d:::benchmark_data_smithi171_78918_object123100:head# 2024-04-20T17:11:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dd2a69a:::benchmark_data_smithi171_78918_object119227:head# 2024-04-20T17:11:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dd388a4:::benchmark_data_smithi171_78918_object57509:head# 2024-04-20T17:11:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dd4e6c2:::benchmark_data_smithi171_78918_object101648:head# 2024-04-20T17:11:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dd5510a:::benchmark_data_smithi171_78918_object62744:head# 2024-04-20T17:11:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dd67b84:::benchmark_data_smithi171_78918_object139524:head# 2024-04-20T17:11:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dd7046d:::benchmark_data_smithi171_78918_object129767:head# 2024-04-20T17:11:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dd76ed4:::benchmark_data_smithi171_78918_object92631:head# 2024-04-20T17:11:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dd80e5b:::benchmark_data_smithi171_78918_object75469:head# 2024-04-20T17:11:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dd847da:::benchmark_data_smithi171_78918_object80171:head# 2024-04-20T17:11:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dd872dd:::benchmark_data_smithi171_78918_object123679:head# 2024-04-20T17:11:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dd8e944:::benchmark_data_smithi171_78918_object109213:head# 2024-04-20T17:11:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dd9ec36:::benchmark_data_smithi171_78918_object70213:head# 2024-04-20T17:11:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dda8567:::benchmark_data_smithi171_78918_object122200:head# 2024-04-20T17:11:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ddacd04:::benchmark_data_smithi171_78918_object81590:head# 2024-04-20T17:11:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ddb1651:::benchmark_data_smithi171_78918_object120716:head# 2024-04-20T17:11:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ddbafe9:::benchmark_data_smithi171_78918_object87633:head# 2024-04-20T17:11:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ddbcf32:::benchmark_data_smithi171_78918_object135548:head# 2024-04-20T17:11:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ddbe32d:::benchmark_data_smithi171_78918_object114235:head# 2024-04-20T17:11:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ddc0e57:::benchmark_data_smithi171_78918_object109952:head# 2024-04-20T17:11:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ddc3bbc:::benchmark_data_smithi171_78918_object130303:head# 2024-04-20T17:11:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ddc52e7:::benchmark_data_smithi171_78918_object75897:head# 2024-04-20T17:11:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ddcf85e:::benchmark_data_smithi171_78918_object131469:head# 2024-04-20T17:11:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ddcfd87:::benchmark_data_smithi171_78918_object58390:head# 2024-04-20T17:11:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ddd8595:::benchmark_data_smithi171_78918_object110976:head# 2024-04-20T17:11:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ddde422:::benchmark_data_smithi171_78918_object129532:head# 2024-04-20T17:11:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dddec9e:::benchmark_data_smithi171_78918_object104884:head# 2024-04-20T17:11:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dde50ae:::benchmark_data_smithi171_78918_object126349:head# 2024-04-20T17:11:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dde5c71:::benchmark_data_smithi171_78918_object142835:head# 2024-04-20T17:11:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dde8a03:::benchmark_data_smithi171_78918_object103086:head# 2024-04-20T17:11:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ddede20:::benchmark_data_smithi171_78918_object147658:head# 2024-04-20T17:11:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ddf1b37:::benchmark_data_smithi171_78918_object112434:head# 2024-04-20T17:11:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ddf7df7:::benchmark_data_smithi171_78918_object65095:head# 2024-04-20T17:11:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7de044f8:::benchmark_data_smithi171_78918_object90258:head# 2024-04-20T17:11:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7de0ffac:::benchmark_data_smithi171_78918_object80652:head# 2024-04-20T17:11:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7de1843a:::benchmark_data_smithi171_78918_object86095:head# 2024-04-20T17:11:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7de29f0d:::benchmark_data_smithi171_78918_object63300:head# 2024-04-20T17:11:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7de2aa17:::benchmark_data_smithi171_78918_object133785:head# 2024-04-20T17:11:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7de36593:::benchmark_data_smithi171_78918_object66094:head# 2024-04-20T17:11:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7de3b974:::benchmark_data_smithi171_78918_object104080:head# 2024-04-20T17:11:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7de402c7:::benchmark_data_smithi171_78918_object131596:head# 2024-04-20T17:11:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7de48837:::benchmark_data_smithi171_78918_object71828:head# 2024-04-20T17:11:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7de4abea:::benchmark_data_smithi171_78918_object75338:head# 2024-04-20T17:11:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7de4b8cb:::benchmark_data_smithi171_78918_object71676:head# 2024-04-20T17:11:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7de5b257:::benchmark_data_smithi171_78918_object80686:head# 2024-04-20T17:11:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7de5e5a3:::benchmark_data_smithi171_78918_object52989:head# 2024-04-20T17:11:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7de69663:::benchmark_data_smithi171_78918_object105637:head# 2024-04-20T17:11:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7de74b84:::benchmark_data_smithi171_78918_object121471:head# 2024-04-20T17:11:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7de7cad3:::benchmark_data_smithi171_78918_object98805:head# 2024-04-20T17:11:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7de82fb0:::benchmark_data_smithi171_78918_object84758:head# 2024-04-20T17:11:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7de87863:::benchmark_data_smithi171_78918_object58310:head# 2024-04-20T17:11:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7de897fa:::benchmark_data_smithi171_78918_object155205:head# 2024-04-20T17:11:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7de8e54e:::benchmark_data_smithi171_78918_object64129:head# 2024-04-20T17:11:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7de8fb59:::benchmark_data_smithi171_78918_object98494:head# 2024-04-20T17:11:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7de95931:::benchmark_data_smithi171_78918_object121017:head# 2024-04-20T17:11:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7de96c73:::benchmark_data_smithi171_78918_object152625:head# 2024-04-20T17:11:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7de9ccfe:::benchmark_data_smithi171_78918_object160788:head# 2024-04-20T17:11:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7deb5556:::benchmark_data_smithi171_78918_object110898:head# 2024-04-20T17:11:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7deca210:::benchmark_data_smithi171_78918_object133741:head# 2024-04-20T17:11:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dedd360:::benchmark_data_smithi171_78918_object96345:head# 2024-04-20T17:11:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dee1255:::benchmark_data_smithi171_78918_object132067:head# 2024-04-20T17:11:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dee5d76:::benchmark_data_smithi171_78918_object145517:head# 2024-04-20T17:11:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7def864d:::benchmark_data_smithi171_78918_object57207:head# 2024-04-20T17:11:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7df0076f:::benchmark_data_smithi171_78918_object93033:head# 2024-04-20T17:11:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7df03b81:::benchmark_data_smithi171_78918_object118438:head# 2024-04-20T17:11:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7df0866e:::benchmark_data_smithi171_78918_object113105:head# 2024-04-20T17:11:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7df0a483:::benchmark_data_smithi171_78918_object59219:head# 2024-04-20T17:11:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7df0d3f1:::benchmark_data_smithi171_78918_object113919:head# 2024-04-20T17:11:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7df0ec8a:::benchmark_data_smithi171_78918_object133246:head# 2024-04-20T17:11:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7df2bad1:::benchmark_data_smithi171_78918_object51475:head# 2024-04-20T17:11:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7df3009f:::benchmark_data_smithi171_78918_object56898:head# 2024-04-20T17:11:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7df30e0d:::benchmark_data_smithi171_78918_object57258:head# 2024-04-20T17:11:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7df52758:::benchmark_data_smithi171_78918_object83453:head# 2024-04-20T17:11:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7df79020:::benchmark_data_smithi171_78918_object55790:head# 2024-04-20T17:11:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7df7bbc0:::benchmark_data_smithi171_78918_object104727:head# 2024-04-20T17:11:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7df80237:::benchmark_data_smithi171_78918_object92593:head# 2024-04-20T17:11:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7df8046e:::benchmark_data_smithi171_78918_object110616:head# 2024-04-20T17:11:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7df81c83:::benchmark_data_smithi171_78918_object55844:head# 2024-04-20T17:11:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7df81ec5:::benchmark_data_smithi171_78918_object154109:head# 2024-04-20T17:11:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7df81f6d:::benchmark_data_smithi171_78918_object107809:head# 2024-04-20T17:11:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7df97e69:::benchmark_data_smithi171_78918_object52438:head# 2024-04-20T17:11:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dfa61be:::benchmark_data_smithi171_78918_object97219:head# 2024-04-20T17:11:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dfbd367:::benchmark_data_smithi171_78918_object111071:head# 2024-04-20T17:11:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dfcb11c:::benchmark_data_smithi171_78918_object88231:head# 2024-04-20T17:11:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dfd0162:::benchmark_data_smithi171_78918_object129899:head# 2024-04-20T17:11:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dfe16ff:::benchmark_data_smithi171_78918_object152769:head# 2024-04-20T17:11:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dff81f0:::benchmark_data_smithi171_78918_object143138:head# 2024-04-20T17:11:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7dffff9c:::benchmark_data_smithi171_78918_object144074:head# 2024-04-20T17:11:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e000aa3:::benchmark_data_smithi171_78918_object99531:head# 2024-04-20T17:11:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e002270:::benchmark_data_smithi171_78918_object73721:head# 2024-04-20T17:11:09.201 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e01a4dd:::benchmark_data_smithi171_78918_object93075:head# 2024-04-20T17:11:09.201 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e01c34b:::benchmark_data_smithi171_78918_object65456:head# 2024-04-20T17:11:09.201 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e03ae5b:::benchmark_data_smithi171_78918_object106303:head# 2024-04-20T17:11:09.201 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e03ff35:::benchmark_data_smithi171_78918_object61342:head# 2024-04-20T17:11:09.201 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e042e28:::benchmark_data_smithi171_78918_object57904:head# 2024-04-20T17:11:09.201 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e05932e:::benchmark_data_smithi171_78918_object59686:head# 2024-04-20T17:11:09.201 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e05bb77:::benchmark_data_smithi171_78918_object97942:head# 2024-04-20T17:11:09.201 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e067ae5:::benchmark_data_smithi171_78918_object123216:head# 2024-04-20T17:11:09.201 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e06e892:::benchmark_data_smithi171_78918_object135193:head# 2024-04-20T17:11:09.201 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e06f620:::benchmark_data_smithi171_78918_object132286:head# 2024-04-20T17:11:09.201 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e07de3d:::benchmark_data_smithi171_78918_object147954:head# 2024-04-20T17:11:09.201 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e0a7928:::benchmark_data_smithi171_78918_object74679:head# 2024-04-20T17:11:09.201 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e0ac5df:::benchmark_data_smithi171_78918_object107767:head# 2024-04-20T17:11:09.201 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e0b8637:::benchmark_data_smithi171_78918_object148336:head# 2024-04-20T17:11:09.201 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e0be41c:::benchmark_data_smithi171_78918_object108282:head# 2024-04-20T17:11:09.202 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e0d257d:::benchmark_data_smithi171_78918_object75345:head# 2024-04-20T17:11:09.202 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e0d5e84:::benchmark_data_smithi171_78918_object78178:head# 2024-04-20T17:11:09.202 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e0e19a0:::benchmark_data_smithi171_78918_object60694:head# 2024-04-20T17:11:09.202 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e0e88c3:::benchmark_data_smithi171_78918_object110229:head# 2024-04-20T17:11:09.202 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e0f2db6:::benchmark_data_smithi171_78918_object127621:head# 2024-04-20T17:11:09.202 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e0fa08f:::benchmark_data_smithi171_78918_object152911:head# 2024-04-20T17:11:09.202 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e0fe87f:::benchmark_data_smithi171_78918_object67774:head# 2024-04-20T17:11:09.202 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e10fbf0:::benchmark_data_smithi171_78918_object142326:head# 2024-04-20T17:11:09.202 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e110c54:::benchmark_data_smithi171_78918_object124141:head# 2024-04-20T17:11:09.202 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e11598e:::benchmark_data_smithi171_78918_object142116:head# 2024-04-20T17:11:09.202 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e116d01:::benchmark_data_smithi171_78918_object147915:head# 2024-04-20T17:11:09.202 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e11929c:::benchmark_data_smithi171_78918_object65245:head# 2024-04-20T17:11:09.202 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e11b830:::benchmark_data_smithi171_78918_object140123:head# 2024-04-20T17:11:09.202 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e150aef:::benchmark_data_smithi171_78918_object65597:head# 2024-04-20T17:11:09.203 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e1526f4:::benchmark_data_smithi171_78918_object123908:head# 2024-04-20T17:11:09.203 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e156d12:::benchmark_data_smithi171_78918_object52649:head# 2024-04-20T17:11:09.203 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e15b36a:::benchmark_data_smithi171_78918_object135446:head# 2024-04-20T17:11:09.203 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e15b629:::benchmark_data_smithi171_78918_object91191:head# 2024-04-20T17:11:09.203 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e173765:::benchmark_data_smithi171_78918_object70061:head# 2024-04-20T17:11:09.203 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e184bfe:::benchmark_data_smithi171_78918_object51857:head# 2024-04-20T17:11:09.203 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e1852c8:::benchmark_data_smithi171_78918_object140755:head# 2024-04-20T17:11:09.203 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e1858e7:::benchmark_data_smithi171_78918_object74884:head# 2024-04-20T17:11:09.203 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e186158:::benchmark_data_smithi171_78918_object97063:head# 2024-04-20T17:11:09.203 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e18eb20:::benchmark_data_smithi171_78918_object100233:head# 2024-04-20T17:11:09.203 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e195ff1:::benchmark_data_smithi171_78918_object124640:head# 2024-04-20T17:11:09.203 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e1a0e32:::benchmark_data_smithi171_78918_object75268:head# 2024-04-20T17:11:09.203 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e1a411c:::benchmark_data_smithi171_78918_object92513:head# 2024-04-20T17:11:09.203 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e1a8bb7:::benchmark_data_smithi171_78918_object105965:head# 2024-04-20T17:11:09.203 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e1b1aee:::benchmark_data_smithi171_78918_object149835:head# 2024-04-20T17:11:09.204 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e1ba566:::benchmark_data_smithi171_78918_object116295:head# 2024-04-20T17:11:09.204 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e1be5b3:::benchmark_data_smithi171_78918_object150986:head# 2024-04-20T17:11:09.204 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e1c056d:::benchmark_data_smithi171_78918_object106995:head# 2024-04-20T17:11:09.204 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e1c40cc:::benchmark_data_smithi171_78918_object103966:head# 2024-04-20T17:11:09.204 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e1cbc03:::benchmark_data_smithi171_78918_object99384:head# 2024-04-20T17:11:09.204 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e1d0690:::benchmark_data_smithi171_78918_object60270:head# 2024-04-20T17:11:09.204 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e1e47de:::benchmark_data_smithi171_78918_object76965:head# 2024-04-20T17:11:09.204 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e209fbf:::benchmark_data_smithi171_78918_object59034:head# 2024-04-20T17:11:09.204 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e21343b:::benchmark_data_smithi171_78918_object89513:head# 2024-04-20T17:11:09.204 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e226c42:::benchmark_data_smithi171_78918_object107064:head# 2024-04-20T17:11:09.204 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e23ac8b:::benchmark_data_smithi171_78918_object110628:head# 2024-04-20T17:11:09.204 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e250a82:::benchmark_data_smithi171_78918_object58992:head# 2024-04-20T17:11:09.204 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e2579a9:::benchmark_data_smithi171_78918_object53515:head# 2024-04-20T17:11:09.204 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e25f17a:::benchmark_data_smithi171_78918_object120686:head# 2024-04-20T17:11:09.204 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e28195c:::benchmark_data_smithi171_78918_object115753:head# 2024-04-20T17:11:09.205 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e2899ce:::benchmark_data_smithi171_78918_object138333:head# 2024-04-20T17:11:09.205 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e290452:::benchmark_data_smithi171_78918_object60773:head# 2024-04-20T17:11:09.205 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e2913ea:::benchmark_data_smithi171_78918_object119405:head# 2024-04-20T17:11:09.205 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e2a3b46:::benchmark_data_smithi171_78918_object128509:head# 2024-04-20T17:11:09.205 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e2a7fd0:::benchmark_data_smithi171_78918_object149616:head# 2024-04-20T17:11:09.205 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e2a8379:::benchmark_data_smithi171_78918_object90877:head# 2024-04-20T17:11:09.205 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e2b8eeb:::benchmark_data_smithi171_78918_object151769:head# 2024-04-20T17:11:09.205 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e2c663f:::benchmark_data_smithi171_78918_object104834:head# 2024-04-20T17:11:09.205 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e2cec26:::benchmark_data_smithi171_78918_object132041:head# 2024-04-20T17:11:09.205 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e2d0086:::benchmark_data_smithi171_78918_object158671:head# 2024-04-20T17:11:09.205 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e2d3d30:::benchmark_data_smithi171_78918_object92183:head# 2024-04-20T17:11:09.205 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e2da5ea:::benchmark_data_smithi171_78918_object121408:head# 2024-04-20T17:11:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e2dd8ea:::benchmark_data_smithi171_78918_object154989:head# 2024-04-20T17:11:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e2edf83:::benchmark_data_smithi171_78918_object84764:head# 2024-04-20T17:11:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e2f0f74:::benchmark_data_smithi171_78918_object118116:head# 2024-04-20T17:11:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e2f59af:::benchmark_data_smithi171_78918_object85467:head# 2024-04-20T17:11:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e2faa73:::benchmark_data_smithi171_78918_object116404:head# 2024-04-20T17:11:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e305eef:::benchmark_data_smithi171_78918_object94743:head# 2024-04-20T17:11:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e308607:::benchmark_data_smithi171_78918_object78986:head# 2024-04-20T17:11:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e30920b:::benchmark_data_smithi171_78918_object54148:head# 2024-04-20T17:11:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e30f069:::benchmark_data_smithi171_78918_object97806:head# 2024-04-20T17:11:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e316777:::benchmark_data_smithi171_78918_object148536:head# 2024-04-20T17:11:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e319b9c:::benchmark_data_smithi171_78918_object94307:head# 2024-04-20T17:11:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e319db3:::benchmark_data_smithi171_78918_object68408:head# 2024-04-20T17:11:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e31aca6:::benchmark_data_smithi171_78918_object155872:head# 2024-04-20T17:11:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e31c751:::benchmark_data_smithi171_78918_object98049:head# 2024-04-20T17:11:09.207 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e32131b:::benchmark_data_smithi171_78918_object53884:head# 2024-04-20T17:11:09.207 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e336cf8:::benchmark_data_smithi171_78918_object145653:head# 2024-04-20T17:11:09.207 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e33ed51:::benchmark_data_smithi171_78918_object60969:head# 2024-04-20T17:11:09.207 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e341c57:::benchmark_data_smithi171_78918_object79504:head# 2024-04-20T17:11:09.207 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e350c53:::benchmark_data_smithi171_78918_object104621:head# 2024-04-20T17:11:09.207 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e35f266:::benchmark_data_smithi171_78918_object105493:head# 2024-04-20T17:11:09.207 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e372e76:::benchmark_data_smithi171_78918_object145204:head# 2024-04-20T17:11:09.207 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e378968:::benchmark_data_smithi171_78918_object160496:head# 2024-04-20T17:11:09.207 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e3809d2:::benchmark_data_smithi171_78918_object52783:head# 2024-04-20T17:11:09.207 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e385f1c:::benchmark_data_smithi171_78918_object80060:head# 2024-04-20T17:11:09.207 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e386a37:::benchmark_data_smithi171_78918_object142564:head# 2024-04-20T17:11:09.207 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e38cc21:::benchmark_data_smithi171_78918_object85237:head# 2024-04-20T17:11:09.207 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e39d5dc:::benchmark_data_smithi171_78918_object81329:head# 2024-04-20T17:11:09.207 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e3a1e39:::benchmark_data_smithi171_78918_object124694:head# 2024-04-20T17:11:09.208 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e3ac7c0:::benchmark_data_smithi171_78918_object134224:head# 2024-04-20T17:11:09.208 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e3b2099:::benchmark_data_smithi171_78918_object153494:head# 2024-04-20T17:11:09.208 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e3b222d:::benchmark_data_smithi171_78918_object159049:head# 2024-04-20T17:11:09.208 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e3df0e6:::benchmark_data_smithi171_78918_object145225:head# 2024-04-20T17:11:09.208 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e3e4ba7:::benchmark_data_smithi171_78918_object67871:head# 2024-04-20T17:11:09.208 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e3e51f7:::benchmark_data_smithi171_78918_object112302:head# 2024-04-20T17:11:09.208 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e3e69b1:::benchmark_data_smithi171_78918_object70110:head# 2024-04-20T17:11:09.208 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e3e91fb:::benchmark_data_smithi171_78918_object102848:head# 2024-04-20T17:11:09.208 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e3ef0b4:::benchmark_data_smithi171_78918_object119479:head# 2024-04-20T17:11:09.208 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e3f1ca6:::benchmark_data_smithi171_78918_object108483:head# 2024-04-20T17:11:09.208 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e3f1fad:::benchmark_data_smithi171_78918_object133109:head# 2024-04-20T17:11:09.208 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e3f23dd:::benchmark_data_smithi171_78918_object153306:head# 2024-04-20T17:11:09.208 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e3f2dbc:::benchmark_data_smithi171_78918_object145739:head# 2024-04-20T17:11:09.208 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e3f9829:::benchmark_data_smithi171_78918_object126006:head# 2024-04-20T17:11:09.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e404556:::benchmark_data_smithi171_78918_object79612:head# 2024-04-20T17:11:09.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e404ece:::benchmark_data_smithi171_78918_object72286:head# 2024-04-20T17:11:09.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e41280e:::benchmark_data_smithi171_78918_object148648:head# 2024-04-20T17:11:09.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e423e34:::benchmark_data_smithi171_78918_object94685:head# 2024-04-20T17:11:09.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e4287e8:::benchmark_data_smithi171_78918_object135638:head# 2024-04-20T17:11:09.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e430658:::benchmark_data_smithi171_78918_object88905:head# 2024-04-20T17:11:09.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e43346d:::benchmark_data_smithi171_78918_object140589:head# 2024-04-20T17:11:09.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e44253a:::benchmark_data_smithi171_78918_object147309:head# 2024-04-20T17:11:09.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e4438d8:::benchmark_data_smithi171_78918_object67094:head# 2024-04-20T17:11:09.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e446638:::benchmark_data_smithi171_78918_object143780:head# 2024-04-20T17:11:09.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e449551:::benchmark_data_smithi171_78918_object77019:head# 2024-04-20T17:11:09.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e456119:::benchmark_data_smithi171_78918_object120914:head# 2024-04-20T17:11:09.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e463e15:::benchmark_data_smithi171_78918_object114062:head# 2024-04-20T17:11:09.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e4647a9:::benchmark_data_smithi171_78918_object159471:head# 2024-04-20T17:11:09.209 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e4818b2:::benchmark_data_smithi171_78918_object100964:head# 2024-04-20T17:11:09.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e4a929d:::benchmark_data_smithi171_78918_object97364:head# 2024-04-20T17:11:09.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e4a96c1:::benchmark_data_smithi171_78918_object80083:head# 2024-04-20T17:11:09.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e4ac13c:::benchmark_data_smithi171_78918_object73252:head# 2024-04-20T17:11:09.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e4b755b:::benchmark_data_smithi171_78918_object121943:head# 2024-04-20T17:11:09.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e4c3a8b:::benchmark_data_smithi171_78918_object125586:head# 2024-04-20T17:11:09.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e4cd560:::benchmark_data_smithi171_78918_object129410:head# 2024-04-20T17:11:09.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e4d040a:::benchmark_data_smithi171_78918_object83227:head# 2024-04-20T17:11:09.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e4dd7e4:::benchmark_data_smithi171_78918_object116026:head# 2024-04-20T17:11:09.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e4e1b83:::benchmark_data_smithi171_78918_object120911:head# 2024-04-20T17:11:09.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e4ea9e2:::benchmark_data_smithi171_78918_object145874:head# 2024-04-20T17:11:09.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e4facfb:::benchmark_data_smithi171_78918_object134623:head# 2024-04-20T17:11:09.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e5004ea:::benchmark_data_smithi171_78918_object51644:head# 2024-04-20T17:11:09.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e501be1:::benchmark_data_smithi171_78918_object81219:head# 2024-04-20T17:11:09.210 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e50780b:::benchmark_data_smithi171_78918_object99874:head# 2024-04-20T17:11:09.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e519a3d:::benchmark_data_smithi171_78918_object91246:head# 2024-04-20T17:11:09.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e521d4d:::benchmark_data_smithi171_78918_object53965:head# 2024-04-20T17:11:09.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e525dca:::benchmark_data_smithi171_78918_object137386:head# 2024-04-20T17:11:09.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e5413f5:::benchmark_data_smithi171_78918_object138852:head# 2024-04-20T17:11:09.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e547528:::benchmark_data_smithi171_78918_object137735:head# 2024-04-20T17:11:09.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e550012:::benchmark_data_smithi171_78918_object64855:head# 2024-04-20T17:11:09.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e555383:::benchmark_data_smithi171_78918_object128792:head# 2024-04-20T17:11:09.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e55c65a:::benchmark_data_smithi171_78918_object67014:head# 2024-04-20T17:11:09.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e565b0a:::benchmark_data_smithi171_78918_object144177:head# 2024-04-20T17:11:09.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e5692de:::benchmark_data_smithi171_78918_object52720:head# 2024-04-20T17:11:09.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e56a752:::benchmark_data_smithi171_78918_object102733:head# 2024-04-20T17:11:09.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e571309:::benchmark_data_smithi171_78918_object114783:head# 2024-04-20T17:11:09.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e574308:::benchmark_data_smithi171_78918_object99698:head# 2024-04-20T17:11:09.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e583057:::benchmark_data_smithi171_78918_object152995:head# 2024-04-20T17:11:09.211 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e590cd3:::benchmark_data_smithi171_78918_object67631:head# 2024-04-20T17:11:09.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e5a5ad2:::benchmark_data_smithi171_78918_object126998:head# 2024-04-20T17:11:09.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e5ac092:::benchmark_data_smithi171_78918_object76335:head# 2024-04-20T17:11:09.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e5b95aa:::benchmark_data_smithi171_78918_object96054:head# 2024-04-20T17:11:09.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e5ba037:::benchmark_data_smithi171_78918_object134642:head# 2024-04-20T17:11:09.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e5be2b7:::benchmark_data_smithi171_78918_object128166:head# 2024-04-20T17:11:09.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e5c0cb6:::benchmark_data_smithi171_78918_object60882:head# 2024-04-20T17:11:09.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e5c1f9e:::benchmark_data_smithi171_78918_object158554:head# 2024-04-20T17:11:09.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e5cd211:::benchmark_data_smithi171_78918_object107717:head# 2024-04-20T17:11:09.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e5ce0e1:::benchmark_data_smithi171_78918_object88425:head# 2024-04-20T17:11:09.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e5d56f1:::benchmark_data_smithi171_78918_object144031:head# 2024-04-20T17:11:09.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e5e6483:::benchmark_data_smithi171_78918_object94214:head# 2024-04-20T17:11:09.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e5fb3db:::benchmark_data_smithi171_78918_object88145:head# 2024-04-20T17:11:09.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e60a531:::benchmark_data_smithi171_78918_object109808:head# 2024-04-20T17:11:09.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e6109cc:::benchmark_data_smithi171_78918_object95305:head# 2024-04-20T17:11:09.212 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e62077a:::benchmark_data_smithi171_78918_object95094:head# 2024-04-20T17:11:09.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e620799:::benchmark_data_smithi171_78918_object149483:head# 2024-04-20T17:11:09.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e6387b8:::benchmark_data_smithi171_78918_object111236:head# 2024-04-20T17:11:09.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e63e008:::benchmark_data_smithi171_78918_object150234:head# 2024-04-20T17:11:09.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e63ef49:::benchmark_data_smithi171_78918_object100361:head# 2024-04-20T17:11:09.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e644f19:::benchmark_data_smithi171_78918_object73527:head# 2024-04-20T17:11:09.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e65189f:::benchmark_data_smithi171_78918_object112391:head# 2024-04-20T17:11:09.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e65c434:::benchmark_data_smithi171_78918_object66878:head# 2024-04-20T17:11:09.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e65f059:::benchmark_data_smithi171_78918_object83540:head# 2024-04-20T17:11:09.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e66bbf8:::benchmark_data_smithi171_78918_object106545:head# 2024-04-20T17:11:09.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e676318:::benchmark_data_smithi171_78918_object158781:head# 2024-04-20T17:11:09.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e697634:::benchmark_data_smithi171_78918_object159785:head# 2024-04-20T17:11:09.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e69972e:::benchmark_data_smithi171_78918_object80131:head# 2024-04-20T17:11:09.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e69bb07:::benchmark_data_smithi171_78918_object92132:head# 2024-04-20T17:11:09.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e69ecfa:::benchmark_data_smithi171_78918_object72556:head# 2024-04-20T17:11:09.213 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e6a5bee:::benchmark_data_smithi171_78918_object148646:head# 2024-04-20T17:11:09.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e6aad0d:::benchmark_data_smithi171_78918_object139135:head# 2024-04-20T17:11:09.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e6b1248:::benchmark_data_smithi171_78918_object88084:head# 2024-04-20T17:11:09.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e6ce734:::benchmark_data_smithi171_78918_object110503:head# 2024-04-20T17:11:09.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e6d8e5c:::benchmark_data_smithi171_78918_object113142:head# 2024-04-20T17:11:09.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e6de035:::benchmark_data_smithi171_78918_object112207:head# 2024-04-20T17:11:09.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e6de12d:::benchmark_data_smithi171_78918_object68345:head# 2024-04-20T17:11:09.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e6ed711:::benchmark_data_smithi171_78918_object142433:head# 2024-04-20T17:11:09.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e6fda91:::benchmark_data_smithi171_78918_object52459:head# 2024-04-20T17:11:09.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e70383c:::benchmark_data_smithi171_78918_object124136:head# 2024-04-20T17:11:09.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e7092c0:::benchmark_data_smithi171_78918_object86915:head# 2024-04-20T17:11:09.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e71700e:::benchmark_data_smithi171_78918_object70090:head# 2024-04-20T17:11:09.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e719a47:::benchmark_data_smithi171_78918_object131150:head# 2024-04-20T17:11:09.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e71bb13:::benchmark_data_smithi171_78918_object69841:head# 2024-04-20T17:11:09.214 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e7258fd:::benchmark_data_smithi171_78918_object87301:head# 2024-04-20T17:11:09.215 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e72d8aa:::benchmark_data_smithi171_78918_object121711:head# 2024-04-20T17:11:09.215 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e73695d:::benchmark_data_smithi171_78918_object97331:head# 2024-04-20T17:11:09.215 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e73a865:::benchmark_data_smithi171_78918_object71091:head# 2024-04-20T17:11:09.215 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e73ce71:::benchmark_data_smithi171_78918_object110636:head# 2024-04-20T17:11:09.215 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e7406d6:::benchmark_data_smithi171_78918_object63192:head# 2024-04-20T17:11:09.215 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e75c297:::benchmark_data_smithi171_78918_object86055:head# 2024-04-20T17:11:09.215 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e764917:::benchmark_data_smithi171_78918_object111553:head# 2024-04-20T17:11:09.215 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e76a608:::benchmark_data_smithi171_78918_object105988:head# 2024-04-20T17:11:09.215 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e777277:::benchmark_data_smithi171_78918_object130475:head# 2024-04-20T17:11:09.215 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e7a3af1:::benchmark_data_smithi171_78918_object137725:head# 2024-04-20T17:11:09.215 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e7ad01b:::benchmark_data_smithi171_78918_object66138:head# 2024-04-20T17:11:09.215 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e7b99af:::benchmark_data_smithi171_78918_object138040:head# 2024-04-20T17:11:09.215 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e7c717d:::benchmark_data_smithi171_78918_object91354:head# 2024-04-20T17:11:09.215 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e7e86ac:::benchmark_data_smithi171_78918_object86631:head# 2024-04-20T17:11:09.216 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e82414c:::benchmark_data_smithi171_78918_object89035:head# 2024-04-20T17:11:09.216 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e8361bb:::benchmark_data_smithi171_78918_object119260:head# 2024-04-20T17:11:09.216 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e839561:::benchmark_data_smithi171_78918_object54116:head# 2024-04-20T17:11:09.216 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e83edd6:::benchmark_data_smithi171_78918_object110864:head# 2024-04-20T17:11:09.216 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e840c1d:::benchmark_data_smithi171_78918_object147302:head# 2024-04-20T17:11:09.216 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e8485c0:::benchmark_data_smithi171_78918_object97490:head# 2024-04-20T17:11:09.216 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e84a013:::benchmark_data_smithi171_78918_object127060:head# 2024-04-20T17:11:09.216 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e856c64:::benchmark_data_smithi171_78918_object157061:head# 2024-04-20T17:11:09.216 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e85defc:::benchmark_data_smithi171_78918_object65356:head# 2024-04-20T17:11:09.216 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e86bdf1:::benchmark_data_smithi171_78918_object107743:head# 2024-04-20T17:11:09.216 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e86ff8f:::benchmark_data_smithi171_78918_object51799:head# 2024-04-20T17:11:09.216 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e88748e:::benchmark_data_smithi171_78918_object121098:head# 2024-04-20T17:11:09.216 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e89d0c8:::benchmark_data_smithi171_78918_object62952:head# 2024-04-20T17:11:09.216 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e8b4a9f:::benchmark_data_smithi171_78918_object92779:head# 2024-04-20T17:11:09.216 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e8b98a1:::benchmark_data_smithi171_78918_object146058:head# 2024-04-20T17:11:09.217 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e8ce609:::benchmark_data_smithi171_78918_object125401:head# 2024-04-20T17:11:09.217 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e8d1ff7:::benchmark_data_smithi171_78918_object53825:head# 2024-04-20T17:11:09.217 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e8de3d9:::benchmark_data_smithi171_78918_object98976:head# 2024-04-20T17:11:09.217 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e8f78df:::benchmark_data_smithi171_78918_object113890:head# 2024-04-20T17:11:09.217 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e8f8fc9:::benchmark_data_smithi171_78918_object73825:head# 2024-04-20T17:11:09.217 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e8fc659:::benchmark_data_smithi171_78918_object144383:head# 2024-04-20T17:11:09.217 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e8fdacd:::benchmark_data_smithi171_78918_object152388:head# 2024-04-20T17:11:09.217 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e90266b:::benchmark_data_smithi171_78918_object98985:head# 2024-04-20T17:11:09.217 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e91c678:::benchmark_data_smithi171_78918_object66231:head# 2024-04-20T17:11:09.217 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e92a28b:::benchmark_data_smithi171_78918_object67180:head# 2024-04-20T17:11:09.217 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e92b5b2:::benchmark_data_smithi171_78918_object74735:head# 2024-04-20T17:11:09.217 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e92bbf7:::benchmark_data_smithi171_78918_object68539:head# 2024-04-20T17:11:09.217 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e930616:::benchmark_data_smithi171_78918_object128772:head# 2024-04-20T17:11:09.217 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e936ffb:::benchmark_data_smithi171_78918_object153314:head# 2024-04-20T17:11:09.217 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e963cfa:::benchmark_data_smithi171_78918_object82525:head# 2024-04-20T17:11:09.218 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e97226c:::benchmark_data_smithi171_78918_object116565:head# 2024-04-20T17:11:09.218 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e97af0d:::benchmark_data_smithi171_78918_object147663:head# 2024-04-20T17:11:09.218 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e97c83d:::benchmark_data_smithi171_78918_object120549:head# 2024-04-20T17:11:09.218 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e98d6ff:::benchmark_data_smithi171_78918_object133666:head# 2024-04-20T17:11:09.218 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e9a29c0:::benchmark_data_smithi171_78918_object139445:head# 2024-04-20T17:11:09.218 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e9ab6d0:::benchmark_data_smithi171_78918_object55463:head# 2024-04-20T17:11:09.218 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e9b37a1:::benchmark_data_smithi171_78918_object108761:head# 2024-04-20T17:11:09.218 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e9b5cf2:::benchmark_data_smithi171_78918_object136866:head# 2024-04-20T17:11:09.218 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e9bbb6b:::benchmark_data_smithi171_78918_object72947:head# 2024-04-20T17:11:09.218 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e9bdbad:::benchmark_data_smithi171_78918_object77991:head# 2024-04-20T17:11:09.218 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e9c3540:::benchmark_data_smithi171_78918_object91346:head# 2024-04-20T17:11:09.218 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e9cd4fa:::benchmark_data_smithi171_78918_object118957:head# 2024-04-20T17:11:09.218 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e9d033a:::benchmark_data_smithi171_78918_object81153:head# 2024-04-20T17:11:09.218 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e9d386e:::benchmark_data_smithi171_78918_object86408:head# 2024-04-20T17:11:09.219 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e9d402b:::benchmark_data_smithi171_78918_object155736:head# 2024-04-20T17:11:09.219 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e9d9e38:::benchmark_data_smithi171_78918_object72801:head# 2024-04-20T17:11:09.219 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e9e34de:::benchmark_data_smithi171_78918_object89191:head# 2024-04-20T17:11:09.219 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7e9f55c1:::benchmark_data_smithi171_78918_object154325:head# 2024-04-20T17:11:09.219 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ea0bd6e:::benchmark_data_smithi171_78918_object72282:head# 2024-04-20T17:11:09.219 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ea0fdd2:::benchmark_data_smithi171_78918_object102671:head# 2024-04-20T17:11:09.219 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ea23e93:::benchmark_data_smithi171_78918_object122021:head# 2024-04-20T17:11:09.219 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ea40abc:::benchmark_data_smithi171_78918_object84166:head# 2024-04-20T17:11:09.219 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ea4a293:::benchmark_data_smithi171_78918_object56774:head# 2024-04-20T17:11:09.219 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ea553f1:::benchmark_data_smithi171_78918_object102093:head# 2024-04-20T17:11:09.219 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ea661b4:::benchmark_data_smithi171_78918_object74235:head# 2024-04-20T17:11:09.219 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ea6a283:::benchmark_data_smithi171_78918_object56746:head# 2024-04-20T17:11:09.219 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ea6eb9a:::benchmark_data_smithi171_78918_object145732:head# 2024-04-20T17:11:09.219 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ea77eb1:::benchmark_data_smithi171_78918_object134139:head# 2024-04-20T17:11:09.219 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ea79e96:::benchmark_data_smithi171_78918_object99286:head# 2024-04-20T17:11:09.220 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ea7c200:::benchmark_data_smithi171_78918_object122895:head# 2024-04-20T17:11:09.220 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ea82242:::benchmark_data_smithi171_78918_object132898:head# 2024-04-20T17:11:09.220 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ea93b5e:::benchmark_data_smithi171_78918_object87879:head# 2024-04-20T17:11:09.220 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eaa3661:::benchmark_data_smithi171_78918_object119922:head# 2024-04-20T17:11:09.220 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eaa650e:::benchmark_data_smithi171_78918_object151304:head# 2024-04-20T17:11:09.220 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eab060f:::benchmark_data_smithi171_78918_object106199:head# 2024-04-20T17:11:09.220 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eac9ed3:::benchmark_data_smithi171_78918_object79707:head# 2024-04-20T17:11:09.220 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eacb35c:::benchmark_data_smithi171_78918_object105314:head# 2024-04-20T17:11:09.220 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eacfaf1:::benchmark_data_smithi171_78918_object63019:head# 2024-04-20T17:11:09.220 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eade42f:::benchmark_data_smithi171_78918_object128004:head# 2024-04-20T17:11:09.220 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eae991b:::benchmark_data_smithi171_78918_object115504:head# 2024-04-20T17:11:09.220 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eaeb31a:::benchmark_data_smithi171_78918_object126133:head# 2024-04-20T17:11:09.220 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eaf2e43:::benchmark_data_smithi171_78918_object90959:head# 2024-04-20T17:11:09.220 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eafd516:::benchmark_data_smithi171_78918_object152862:head# 2024-04-20T17:11:09.220 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eb02cee:::benchmark_data_smithi171_78918_object145972:head# 2024-04-20T17:11:09.221 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eb10ee4:::benchmark_data_smithi171_78918_object93040:head# 2024-04-20T17:11:09.221 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eb14a76:::benchmark_data_smithi171_78918_object122496:head# 2024-04-20T17:11:09.221 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eb21d50:::benchmark_data_smithi171_78918_object107259:head# 2024-04-20T17:11:09.221 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eb380b3:::benchmark_data_smithi171_78918_object72632:head# 2024-04-20T17:11:09.221 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eb42ea1:::benchmark_data_smithi171_78918_object142488:head# 2024-04-20T17:11:09.221 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eb4e77c:::benchmark_data_smithi171_78918_object156431:head# 2024-04-20T17:11:09.221 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eb5b0de:::benchmark_data_smithi171_78918_object148705:head# 2024-04-20T17:11:09.221 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eb5fa19:::benchmark_data_smithi171_78918_object93523:head# 2024-04-20T17:11:09.221 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eb68267:::benchmark_data_smithi171_78918_object149611:head# 2024-04-20T17:11:09.221 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eb7148f:::benchmark_data_smithi171_78918_object83458:head# 2024-04-20T17:11:09.221 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eb72c51:::benchmark_data_smithi171_78918_object62426:head# 2024-04-20T17:11:09.221 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eb7a3fc:::benchmark_data_smithi171_78918_object56139:head# 2024-04-20T17:11:09.221 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eb80876:::benchmark_data_smithi171_78918_object102818:head# 2024-04-20T17:11:09.221 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eb9e3fb:::benchmark_data_smithi171_78918_object51537:head# 2024-04-20T17:11:09.221 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ebb063c:::benchmark_data_smithi171_78918_object125034:head# 2024-04-20T17:11:09.222 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ebbdce0:::benchmark_data_smithi171_78918_object95908:head# 2024-04-20T17:11:09.222 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ebc00b0:::benchmark_data_smithi171_78918_object137067:head# 2024-04-20T17:11:09.222 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ebc6194:::benchmark_data_smithi171_78918_object57533:head# 2024-04-20T17:11:09.222 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ebcdcdf:::benchmark_data_smithi171_78918_object109484:head# 2024-04-20T17:11:09.222 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ebeb9ac:::benchmark_data_smithi171_78918_object115986:head# 2024-04-20T17:11:09.222 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ebedb5a:::benchmark_data_smithi171_78918_object135694:head# 2024-04-20T17:11:09.222 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ebee1c9:::benchmark_data_smithi171_78918_object137021:head# 2024-04-20T17:11:09.222 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ebf2c08:::benchmark_data_smithi171_78918_object66957:head# 2024-04-20T17:11:09.222 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ec08fb0:::benchmark_data_smithi171_78918_object138427:head# 2024-04-20T17:11:09.222 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ec19bf0:::benchmark_data_smithi171_78918_object79588:head# 2024-04-20T17:11:09.222 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ec23643:::benchmark_data_smithi171_78918_object142612:head# 2024-04-20T17:11:09.222 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ec25912:::benchmark_data_smithi171_78918_object81940:head# 2024-04-20T17:11:09.222 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ec2b986:::benchmark_data_smithi171_78918_object151386:head# 2024-04-20T17:11:09.223 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ec2ff6e:::benchmark_data_smithi171_78918_object94368:head# 2024-04-20T17:11:09.223 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ec3597e:::benchmark_data_smithi171_78918_object104321:head# 2024-04-20T17:11:09.223 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ec382fb:::benchmark_data_smithi171_78918_object79379:head# 2024-04-20T17:11:09.223 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ec3a6f4:::benchmark_data_smithi171_78918_object85007:head# 2024-04-20T17:11:09.223 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ec3cbd5:::benchmark_data_smithi171_78918_object81483:head# 2024-04-20T17:11:09.223 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ec57878:::benchmark_data_smithi171_78918_object61827:head# 2024-04-20T17:11:09.223 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ec5c2e0:::benchmark_data_smithi171_78918_object158673:head# 2024-04-20T17:11:09.223 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ec67ce2:::benchmark_data_smithi171_78918_object128899:head# 2024-04-20T17:11:09.223 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ec79ef6:::benchmark_data_smithi171_78918_object139587:head# 2024-04-20T17:11:09.223 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ec955f7:::benchmark_data_smithi171_78918_object130871:head# 2024-04-20T17:11:09.223 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ec95a4c:::benchmark_data_smithi171_78918_object76620:head# 2024-04-20T17:11:09.223 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ec98bd8:::benchmark_data_smithi171_78918_object94475:head# 2024-04-20T17:11:09.223 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ec99bf5:::benchmark_data_smithi171_78918_object95316:head# 2024-04-20T17:11:09.223 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ec9e5f5:::benchmark_data_smithi171_78918_object99405:head# 2024-04-20T17:11:09.223 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ec9e95d:::benchmark_data_smithi171_78918_object124357:head# 2024-04-20T17:11:09.224 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eca806a:::benchmark_data_smithi171_78918_object59014:head# 2024-04-20T17:11:09.224 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ecaff24:::benchmark_data_smithi171_78918_object52283:head# 2024-04-20T17:11:09.224 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ecb05fd:::benchmark_data_smithi171_78918_object64063:head# 2024-04-20T17:11:09.224 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ecb8673:::benchmark_data_smithi171_78918_object54023:head# 2024-04-20T17:11:09.224 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ecc4732:::benchmark_data_smithi171_78918_object101700:head# 2024-04-20T17:11:09.224 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ecc91d4:::benchmark_data_smithi171_78918_object154957:head# 2024-04-20T17:11:09.224 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eccaf8a:::benchmark_data_smithi171_78918_object121689:head# 2024-04-20T17:11:09.224 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eccd407:::benchmark_data_smithi171_78918_object126054:head# 2024-04-20T17:11:09.224 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eccfd9d:::benchmark_data_smithi171_78918_object61670:head# 2024-04-20T17:11:09.224 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ecd69be:::benchmark_data_smithi171_78918_object72240:head# 2024-04-20T17:11:09.224 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ecdcd5d:::benchmark_data_smithi171_78918_object149572:head# 2024-04-20T17:11:09.224 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ecde22d:::benchmark_data_smithi171_78918_object115013:head# 2024-04-20T17:11:09.224 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ecec938:::benchmark_data_smithi171_78918_object105435:head# 2024-04-20T17:11:09.224 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ecf04a3:::benchmark_data_smithi171_78918_object78858:head# 2024-04-20T17:11:09.224 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ecf3f5f:::benchmark_data_smithi171_78918_object135655:head# 2024-04-20T17:11:09.225 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ecf8c90:::benchmark_data_smithi171_78918_object79251:head# 2024-04-20T17:11:09.225 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ecff65d:::benchmark_data_smithi171_78918_object91673:head# 2024-04-20T17:11:09.225 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ed00f4e:::benchmark_data_smithi171_78918_object97623:head# 2024-04-20T17:11:09.225 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ed0f9c7:::benchmark_data_smithi171_78918_object148849:head# 2024-04-20T17:11:09.225 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ed148a8:::benchmark_data_smithi171_78918_object67397:head# 2024-04-20T17:11:09.225 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ed1e532:::benchmark_data_smithi171_78918_object100998:head# 2024-04-20T17:11:09.225 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ed20ca6:::benchmark_data_smithi171_78918_object72925:head# 2024-04-20T17:11:09.225 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ed224d6:::benchmark_data_smithi171_78918_object104826:head# 2024-04-20T17:11:09.225 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ed24b79:::benchmark_data_smithi171_78918_object104028:head# 2024-04-20T17:11:09.225 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ed2766e:::benchmark_data_smithi171_78918_object133717:head# 2024-04-20T17:11:09.225 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ed2cbd1:::benchmark_data_smithi171_78918_object108776:head# 2024-04-20T17:11:09.225 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ed397d1:::benchmark_data_smithi171_78918_object140960:head# 2024-04-20T17:11:09.225 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ed4e8f0:::benchmark_data_smithi171_78918_object63612:head# 2024-04-20T17:11:09.225 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ed515da:::benchmark_data_smithi171_78918_object110540:head# 2024-04-20T17:11:09.225 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ed5e9c8:::benchmark_data_smithi171_78918_object54706:head# 2024-04-20T17:11:09.226 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ed606fe:::benchmark_data_smithi171_78918_object72398:head# 2024-04-20T17:11:09.226 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ed69762:::benchmark_data_smithi171_78918_object95669:head# 2024-04-20T17:11:09.226 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ed6b050:::benchmark_data_smithi171_78918_object69042:head# 2024-04-20T17:11:09.226 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ed7c637:::benchmark_data_smithi171_78918_object151614:head# 2024-04-20T17:11:09.226 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ed7dad0:::benchmark_data_smithi171_78918_object79969:head# 2024-04-20T17:11:09.226 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ed99ed0:::benchmark_data_smithi171_78918_object119141:head# 2024-04-20T17:11:09.226 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ed9aa82:::benchmark_data_smithi171_78918_object118880:head# 2024-04-20T17:11:09.226 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7edae832:::benchmark_data_smithi171_78918_object58099:head# 2024-04-20T17:11:09.226 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7edb1d35:::benchmark_data_smithi171_78918_object84079:head# 2024-04-20T17:11:09.226 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7edb7f92:::benchmark_data_smithi171_78918_object155481:head# 2024-04-20T17:11:09.226 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7edc1403:::benchmark_data_smithi171_78918_object97135:head# 2024-04-20T17:11:09.226 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7edc4fc7:::benchmark_data_smithi171_78918_object88255:head# 2024-04-20T17:11:09.226 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7edcbadb:::benchmark_data_smithi171_78918_object97498:head# 2024-04-20T17:11:09.226 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7edddc74:::benchmark_data_smithi171_78918_object66050:head# 2024-04-20T17:11:09.227 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ede2248:::benchmark_data_smithi171_78918_object54912:head# 2024-04-20T17:11:09.227 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ede84a3:::benchmark_data_smithi171_78918_object102225:head# 2024-04-20T17:11:09.227 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ee060b0:::benchmark_data_smithi171_78918_object92878:head# 2024-04-20T17:11:09.227 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ee12f4a:::benchmark_data_smithi171_78918_object126676:head# 2024-04-20T17:11:09.227 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ee14e62:::benchmark_data_smithi171_78918_object108706:head# 2024-04-20T17:11:09.227 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ee19831:::benchmark_data_smithi171_78918_object93252:head# 2024-04-20T17:11:09.227 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ee2f6cd:::benchmark_data_smithi171_78918_object148327:head# 2024-04-20T17:11:09.227 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ee39b21:::benchmark_data_smithi171_78918_object94279:head# 2024-04-20T17:11:09.227 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ee3b7d2:::benchmark_data_smithi171_78918_object139021:head# 2024-04-20T17:11:09.227 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ee44288:::benchmark_data_smithi171_78918_object102475:head# 2024-04-20T17:11:09.227 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ee48479:::benchmark_data_smithi171_78918_object92906:head# 2024-04-20T17:11:09.227 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ee4d3ae:::benchmark_data_smithi171_78918_object121135:head# 2024-04-20T17:11:09.227 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ee59544:::benchmark_data_smithi171_78918_object147005:head# 2024-04-20T17:11:09.227 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ee60f84:::benchmark_data_smithi171_78918_object84296:head# 2024-04-20T17:11:09.227 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ee6188e:::benchmark_data_smithi171_78918_object124255:head# 2024-04-20T17:11:09.228 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ee7ed48:::benchmark_data_smithi171_78918_object90942:head# 2024-04-20T17:11:09.228 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ee896a0:::benchmark_data_smithi171_78918_object58581:head# 2024-04-20T17:11:09.228 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ee96eb0:::benchmark_data_smithi171_78918_object152086:head# 2024-04-20T17:11:09.228 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ee9d0e1:::benchmark_data_smithi171_78918_object110877:head# 2024-04-20T17:11:09.228 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eeab659:::benchmark_data_smithi171_78918_object128465:head# 2024-04-20T17:11:09.228 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eeac2ee:::benchmark_data_smithi171_78918_object52265:head# 2024-04-20T17:11:09.228 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eeaf6d6:::benchmark_data_smithi171_78918_object53460:head# 2024-04-20T17:11:09.228 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eeba42e:::benchmark_data_smithi171_78918_object155321:head# 2024-04-20T17:11:09.228 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eec16f7:::benchmark_data_smithi171_78918_object108615:head# 2024-04-20T17:11:09.228 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eed1deb:::benchmark_data_smithi171_78918_object117628:head# 2024-04-20T17:11:09.228 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eed50d0:::benchmark_data_smithi171_78918_object72258:head# 2024-04-20T17:11:09.228 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eee398f:::benchmark_data_smithi171_78918_object60453:head# 2024-04-20T17:11:09.228 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eeea0d6:::benchmark_data_smithi171_78918_object117072:head# 2024-04-20T17:11:09.228 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eef30e9:::benchmark_data_smithi171_78918_object136265:head# 2024-04-20T17:11:09.229 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eef3d9d:::benchmark_data_smithi171_78918_object158663:head# 2024-04-20T17:11:09.229 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ef00405:::benchmark_data_smithi171_78918_object119545:head# 2024-04-20T17:11:09.229 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ef08d25:::benchmark_data_smithi171_78918_object145860:head# 2024-04-20T17:11:09.229 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ef0b1cb:::benchmark_data_smithi171_78918_object90771:head# 2024-04-20T17:11:09.229 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ef0e40c:::benchmark_data_smithi171_78918_object96889:head# 2024-04-20T17:11:09.229 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ef1319c:::benchmark_data_smithi171_78918_object106050:head# 2024-04-20T17:11:09.229 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ef1c186:::benchmark_data_smithi171_78918_object82400:head# 2024-04-20T17:11:09.229 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ef27a01:::benchmark_data_smithi171_78918_object108843:head# 2024-04-20T17:11:09.229 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ef29e23:::benchmark_data_smithi171_78918_object156912:head# 2024-04-20T17:11:09.229 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ef2b1e5:::benchmark_data_smithi171_78918_object127547:head# 2024-04-20T17:11:09.229 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ef31534:::benchmark_data_smithi171_78918_object77904:head# 2024-04-20T17:11:09.229 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ef49980:::benchmark_data_smithi171_78918_object78779:head# 2024-04-20T17:11:09.229 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ef4ba47:::benchmark_data_smithi171_78918_object110404:head# 2024-04-20T17:11:09.229 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ef4bd3b:::benchmark_data_smithi171_78918_object103452:head# 2024-04-20T17:11:09.229 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ef5a2d7:::benchmark_data_smithi171_78918_object99044:head# 2024-04-20T17:11:09.230 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ef64ee3:::benchmark_data_smithi171_78918_object61215:head# 2024-04-20T17:11:09.230 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ef6f598:::benchmark_data_smithi171_78918_object123449:head# 2024-04-20T17:11:09.230 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ef70734:::benchmark_data_smithi171_78918_object116851:head# 2024-04-20T17:11:09.230 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ef78b72:::benchmark_data_smithi171_78918_object68445:head# 2024-04-20T17:11:09.230 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ef7ac70:::benchmark_data_smithi171_78918_object87107:head# 2024-04-20T17:11:09.230 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ef7eed3:::benchmark_data_smithi171_78918_object106798:head# 2024-04-20T17:11:09.230 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ef85b26:::benchmark_data_smithi171_78918_object96992:head# 2024-04-20T17:11:09.230 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ef91165:::benchmark_data_smithi171_78918_object145300:head# 2024-04-20T17:11:09.230 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ef99143:::benchmark_data_smithi171_78918_object156576:head# 2024-04-20T17:11:09.230 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7efa0d4d:::benchmark_data_smithi171_78918_object63023:head# 2024-04-20T17:11:09.230 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7efa4162:::benchmark_data_smithi171_78918_object139648:head# 2024-04-20T17:11:09.230 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7efc85c0:::benchmark_data_smithi171_78918_object115965:head# 2024-04-20T17:11:09.230 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7efdbdaa:::benchmark_data_smithi171_78918_object119705:head# 2024-04-20T17:11:09.231 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7efe1d7e:::benchmark_data_smithi171_78918_object77180:head# 2024-04-20T17:11:09.231 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7efe8ad0:::benchmark_data_smithi171_78918_object110699:head# 2024-04-20T17:11:09.231 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eff07e6:::benchmark_data_smithi171_78918_object140091:head# 2024-04-20T17:11:09.231 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7eff7bfd:::benchmark_data_smithi171_78918_object94817:head# 2024-04-20T17:11:09.231 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f017fc4:::benchmark_data_smithi171_78918_object74152:head# 2024-04-20T17:11:09.231 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f01a100:::benchmark_data_smithi171_78918_object128171:head# 2024-04-20T17:11:09.231 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f01ded7:::benchmark_data_smithi171_78918_object58406:head# 2024-04-20T17:11:09.231 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f0299d8:::benchmark_data_smithi171_78918_object130442:head# 2024-04-20T17:11:09.231 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f02ba54:::benchmark_data_smithi171_78918_object133291:head# 2024-04-20T17:11:09.231 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f038661:::benchmark_data_smithi171_78918_object138694:head# 2024-04-20T17:11:09.231 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f046cbc:::benchmark_data_smithi171_78918_object101624:head# 2024-04-20T17:11:09.231 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f05708e:::benchmark_data_smithi171_78918_object128759:head# 2024-04-20T17:11:09.231 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f05af06:::benchmark_data_smithi171_78918_object66253:head# 2024-04-20T17:11:09.231 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f0641d9:::benchmark_data_smithi171_78918_object122605:head# 2024-04-20T17:11:09.231 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f065ed4:::benchmark_data_smithi171_78918_object135958:head# 2024-04-20T17:11:09.232 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f0691c8:::benchmark_data_smithi171_78918_object139796:head# 2024-04-20T17:11:09.232 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f0717e4:::benchmark_data_smithi171_78918_object107439:head# 2024-04-20T17:11:09.232 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f0799eb:::benchmark_data_smithi171_78918_object152387:head# 2024-04-20T17:11:09.232 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f08272a:::benchmark_data_smithi171_78918_object136296:head# 2024-04-20T17:11:09.232 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f08342c:::benchmark_data_smithi171_78918_object138812:head# 2024-04-20T17:11:09.232 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f08826a:::benchmark_data_smithi171_78918_object68523:head# 2024-04-20T17:11:09.232 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f08e9ef:::benchmark_data_smithi171_78918_object128141:head# 2024-04-20T17:11:09.232 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f090cf7:::benchmark_data_smithi171_78918_object156739:head# 2024-04-20T17:11:09.232 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f096d92:::benchmark_data_smithi171_78918_object145164:head# 2024-04-20T17:11:09.232 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f0970be:::benchmark_data_smithi171_78918_object53215:head# 2024-04-20T17:11:09.232 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f0a8b6e:::benchmark_data_smithi171_78918_object103169:head# 2024-04-20T17:11:09.232 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f0aa46a:::benchmark_data_smithi171_78918_object76615:head# 2024-04-20T17:11:09.233 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f0d0c94:::benchmark_data_smithi171_78918_object135626:head# 2024-04-20T17:11:09.233 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f0d1d02:::benchmark_data_smithi171_78918_object140599:head# 2024-04-20T17:11:09.233 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f0defe3:::benchmark_data_smithi171_78918_object65604:head# 2024-04-20T17:11:09.233 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f0e18c4:::benchmark_data_smithi171_78918_object87151:head# 2024-04-20T17:11:09.233 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f0e7576:::benchmark_data_smithi171_78918_object76850:head# 2024-04-20T17:11:09.233 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f0e927f:::benchmark_data_smithi171_78918_object133121:head# 2024-04-20T17:11:09.233 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f0fbdd2:::benchmark_data_smithi171_78918_object156835:head# 2024-04-20T17:11:09.233 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f0fc282:::benchmark_data_smithi171_78918_object83585:head# 2024-04-20T17:11:09.233 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f10d115:::benchmark_data_smithi171_78918_object87404:head# 2024-04-20T17:11:09.233 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f11d4ce:::benchmark_data_smithi171_78918_object124734:head# 2024-04-20T17:11:09.233 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f12cb49:::benchmark_data_smithi171_78918_object137815:head# 2024-04-20T17:11:09.233 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f12db11:::benchmark_data_smithi171_78918_object88450:head# 2024-04-20T17:11:09.233 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f13e184:::benchmark_data_smithi171_78918_object75668:head# 2024-04-20T17:11:09.233 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f148e7b:::benchmark_data_smithi171_78918_object78633:head# 2024-04-20T17:11:09.233 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f1539f7:::benchmark_data_smithi171_78918_object72005:head# 2024-04-20T17:11:09.233 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f15e8e8:::benchmark_data_smithi171_78918_object92288:head# 2024-04-20T17:11:09.234 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f16640c:::benchmark_data_smithi171_78918_object81163:head# 2024-04-20T17:11:09.234 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f169e57:::benchmark_data_smithi171_78918_object133139:head# 2024-04-20T17:11:09.234 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f16a498:::benchmark_data_smithi171_78918_object93090:head# 2024-04-20T17:11:09.234 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f17709e:::benchmark_data_smithi171_78918_object89794:head# 2024-04-20T17:11:09.234 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f179a5c:::benchmark_data_smithi171_78918_object51468:head# 2024-04-20T17:11:09.234 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f18acc3:::benchmark_data_smithi171_78918_object133764:head# 2024-04-20T17:11:09.234 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f195824:::benchmark_data_smithi171_78918_object130897:head# 2024-04-20T17:11:09.234 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f1b461a:::benchmark_data_smithi171_78918_object82437:head# 2024-04-20T17:11:09.234 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f1c15ef:::benchmark_data_smithi171_78918_object124778:head# 2024-04-20T17:11:09.234 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f1cb917:::benchmark_data_smithi171_78918_object114950:head# 2024-04-20T17:11:09.234 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f1d3afc:::benchmark_data_smithi171_78918_object57136:head# 2024-04-20T17:11:09.234 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f1e1c2c:::benchmark_data_smithi171_78918_object67550:head# 2024-04-20T17:11:09.234 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f1e5ed1:::benchmark_data_smithi171_78918_object80555:head# 2024-04-20T17:11:09.234 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f1e8803:::benchmark_data_smithi171_78918_object118652:head# 2024-04-20T17:11:09.235 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f1fbb46:::benchmark_data_smithi171_78918_object135534:head# 2024-04-20T17:11:09.235 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f203e80:::benchmark_data_smithi171_78918_object146964:head# 2024-04-20T17:11:09.235 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f21a1e3:::benchmark_data_smithi171_78918_object117283:head# 2024-04-20T17:11:09.235 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f21a764:::benchmark_data_smithi171_78918_object128656:head# 2024-04-20T17:11:09.235 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f21f05c:::benchmark_data_smithi171_78918_object122224:head# 2024-04-20T17:11:09.235 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f21f760:::benchmark_data_smithi171_78918_object123792:head# 2024-04-20T17:11:09.235 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f2381e6:::benchmark_data_smithi171_78918_object93392:head# 2024-04-20T17:11:09.235 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f2432b1:::benchmark_data_smithi171_78918_object121306:head# 2024-04-20T17:11:09.235 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f245b74:::benchmark_data_smithi171_78918_object150396:head# 2024-04-20T17:11:09.235 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f24f481:::benchmark_data_smithi171_78918_object86624:head# 2024-04-20T17:11:09.235 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f252025:::benchmark_data_smithi171_78918_object63492:head# 2024-04-20T17:11:09.235 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f252f2b:::benchmark_data_smithi171_78918_object145833:head# 2024-04-20T17:11:09.235 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f25b082:::benchmark_data_smithi171_78918_object153193:head# 2024-04-20T17:11:09.235 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f2644fa:::benchmark_data_smithi171_78918_object90301:head# 2024-04-20T17:11:09.236 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f264d33:::benchmark_data_smithi171_78918_object58218:head# 2024-04-20T17:11:09.236 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f265f37:::benchmark_data_smithi171_78918_object84996:head# 2024-04-20T17:11:09.236 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f266611:::benchmark_data_smithi171_78918_object67862:head# 2024-04-20T17:11:09.236 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f26d509:::benchmark_data_smithi171_78918_object157916:head# 2024-04-20T17:11:09.236 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f26e5d8:::benchmark_data_smithi171_78918_object141212:head# 2024-04-20T17:11:09.236 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f285cfd:::benchmark_data_smithi171_78918_object58861:head# 2024-04-20T17:11:09.236 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f289234:::benchmark_data_smithi171_78918_object114843:head# 2024-04-20T17:11:09.236 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f291dc8:::benchmark_data_smithi171_78918_object135814:head# 2024-04-20T17:11:09.236 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f29cd26:::benchmark_data_smithi171_78918_object87300:head# 2024-04-20T17:11:09.236 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f2a03f9:::benchmark_data_smithi171_78918_object76028:head# 2024-04-20T17:11:09.236 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f2a252f:::benchmark_data_smithi171_78918_object151763:head# 2024-04-20T17:11:09.236 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f2a502c:::benchmark_data_smithi171_78918_object109265:head# 2024-04-20T17:11:09.236 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f2a5c73:::benchmark_data_smithi171_78918_object136144:head# 2024-04-20T17:11:09.236 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f2b3809:::benchmark_data_smithi171_78918_object148095:head# 2024-04-20T17:11:09.237 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f2c3e62:::benchmark_data_smithi171_78918_object66173:head# 2024-04-20T17:11:09.237 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f2c8947:::benchmark_data_smithi171_78918_object112684:head# 2024-04-20T17:11:09.237 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f2d78f2:::benchmark_data_smithi171_78918_object57241:head# 2024-04-20T17:11:09.237 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f2ec708:::benchmark_data_smithi171_78918_object86795:head# 2024-04-20T17:11:09.237 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f305481:::benchmark_data_smithi171_78918_object104537:head# 2024-04-20T17:11:09.237 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f3054b9:::benchmark_data_smithi171_78918_object155437:head# 2024-04-20T17:11:09.237 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f30d111:::benchmark_data_smithi171_78918_object78749:head# 2024-04-20T17:11:09.237 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f326d72:::benchmark_data_smithi171_78918_object52374:head# 2024-04-20T17:11:09.237 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f336635:::benchmark_data_smithi171_78918_object105489:head# 2024-04-20T17:11:09.237 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f33df7e:::benchmark_data_smithi171_78918_object101718:head# 2024-04-20T17:11:09.237 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f34d18d:::benchmark_data_smithi171_78918_object155725:head# 2024-04-20T17:11:09.237 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f357244:::benchmark_data_smithi171_78918_object73257:head# 2024-04-20T17:11:09.237 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f3632d0:::benchmark_data_smithi171_78918_object160503:head# 2024-04-20T17:11:09.237 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f36c0e5:::benchmark_data_smithi171_78918_object150518:head# 2024-04-20T17:11:09.237 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f36e5c6:::benchmark_data_smithi171_78918_object134398:head# 2024-04-20T17:11:09.238 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f36e5dc:::benchmark_data_smithi171_78918_object59494:head# 2024-04-20T17:11:09.238 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f377462:::benchmark_data_smithi171_78918_object102113:head# 2024-04-20T17:11:09.238 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f37fc0d:::benchmark_data_smithi171_78918_object115576:head# 2024-04-20T17:11:09.238 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f38cc5c:::benchmark_data_smithi171_78918_object58590:head# 2024-04-20T17:11:09.238 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f397656:::benchmark_data_smithi171_78918_object150517:head# 2024-04-20T17:11:09.238 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f39977c:::benchmark_data_smithi171_78918_object134137:head# 2024-04-20T17:11:09.238 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f39efb5:::benchmark_data_smithi171_78918_object112771:head# 2024-04-20T17:11:09.238 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f3a69d5:::benchmark_data_smithi171_78918_object148377:head# 2024-04-20T17:11:09.238 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f3a7230:::benchmark_data_smithi171_78918_object120581:head# 2024-04-20T17:11:09.238 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f3a8399:::benchmark_data_smithi171_78918_object145449:head# 2024-04-20T17:11:09.238 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f3ae10d:::benchmark_data_smithi171_78918_object81177:head# 2024-04-20T17:11:09.238 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f3bc155:::benchmark_data_smithi171_78918_object149425:head# 2024-04-20T17:11:09.238 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f3bd8a6:::benchmark_data_smithi171_78918_object123211:head# 2024-04-20T17:11:09.238 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f3c130d:::benchmark_data_smithi171_78918_object56937:head# 2024-04-20T17:11:09.239 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f3ca6ec:::benchmark_data_smithi171_78918_object73982:head# 2024-04-20T17:11:09.239 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f3d4c52:::benchmark_data_smithi171_78918_object58227:head# 2024-04-20T17:11:09.239 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f3d8552:::benchmark_data_smithi171_78918_object102819:head# 2024-04-20T17:11:09.239 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f3db23c:::benchmark_data_smithi171_78918_object89012:head# 2024-04-20T17:11:09.239 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f3de4e5:::benchmark_data_smithi171_78918_object64910:head# 2024-04-20T17:11:09.239 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f3e6cfb:::benchmark_data_smithi171_78918_object135457:head# 2024-04-20T17:11:09.239 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f405333:::benchmark_data_smithi171_78918_object136774:head# 2024-04-20T17:11:09.239 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f4066b4:::benchmark_data_smithi171_78918_object57282:head# 2024-04-20T17:11:09.239 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f41f435:::benchmark_data_smithi171_78918_object144019:head# 2024-04-20T17:11:09.239 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f4329e4:::benchmark_data_smithi171_78918_object101755:head# 2024-04-20T17:11:09.239 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f438849:::benchmark_data_smithi171_78918_object79088:head# 2024-04-20T17:11:09.239 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f44fe61:::benchmark_data_smithi171_78918_object133245:head# 2024-04-20T17:11:09.239 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f4592ce:::benchmark_data_smithi171_78918_object131368:head# 2024-04-20T17:11:09.239 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f4636cf:::benchmark_data_smithi171_78918_object149199:head# 2024-04-20T17:11:09.239 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f48c972:::benchmark_data_smithi171_78918_object134900:head# 2024-04-20T17:11:09.240 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f48f36c:::benchmark_data_smithi171_78918_object52168:head# 2024-04-20T17:11:09.240 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f491d92:::benchmark_data_smithi171_78918_object65704:head# 2024-04-20T17:11:09.240 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f4a2e91:::benchmark_data_smithi171_78918_object136671:head# 2024-04-20T17:11:09.240 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f4bb22b:::benchmark_data_smithi171_78918_object120243:head# 2024-04-20T17:11:09.240 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f4be5fb:::benchmark_data_smithi171_78918_object148691:head# 2024-04-20T17:11:09.240 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f4c2fb2:::benchmark_data_smithi171_78918_object70567:head# 2024-04-20T17:11:09.240 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f4d8e43:::benchmark_data_smithi171_78918_object93553:head# 2024-04-20T17:11:09.240 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f4e235d:::benchmark_data_smithi171_78918_object74353:head# 2024-04-20T17:11:09.240 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f5015e7:::benchmark_data_smithi171_78918_object137197:head# 2024-04-20T17:11:09.240 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f51f5ec:::benchmark_data_smithi171_78918_object129214:head# 2024-04-20T17:11:09.240 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f52a549:::benchmark_data_smithi171_78918_object52541:head# 2024-04-20T17:11:09.240 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f52f506:::benchmark_data_smithi171_78918_object75095:head# 2024-04-20T17:11:09.240 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f53ea1e:::benchmark_data_smithi171_78918_object80073:head# 2024-04-20T17:11:09.240 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f543de3:::benchmark_data_smithi171_78918_object85987:head# 2024-04-20T17:11:09.241 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f5493a6:::benchmark_data_smithi171_78918_object81251:head# 2024-04-20T17:11:09.241 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f54ed05:::benchmark_data_smithi171_78918_object159656:head# 2024-04-20T17:11:09.241 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f550192:::benchmark_data_smithi171_78918_object155242:head# 2024-04-20T17:11:09.241 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f55644e:::benchmark_data_smithi171_78918_object140756:head# 2024-04-20T17:11:09.241 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f559d66:::benchmark_data_smithi171_78918_object88873:head# 2024-04-20T17:11:09.241 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f55b995:::benchmark_data_smithi171_78918_object150474:head# 2024-04-20T17:11:09.241 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f56c7ef:::benchmark_data_smithi171_78918_object101374:head# 2024-04-20T17:11:09.241 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f56e1b8:::benchmark_data_smithi171_78918_object54180:head# 2024-04-20T17:11:09.241 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f5750e6:::benchmark_data_smithi171_78918_object55414:head# 2024-04-20T17:11:09.241 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f5762e7:::benchmark_data_smithi171_78918_object147381:head# 2024-04-20T17:11:09.241 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f583c85:::benchmark_data_smithi171_78918_object128846:head# 2024-04-20T17:11:09.241 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f58647a:::benchmark_data_smithi171_78918_object116821:head# 2024-04-20T17:11:09.241 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f590c28:::benchmark_data_smithi171_78918_object90789:head# 2024-04-20T17:11:09.241 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f5966f8:::benchmark_data_smithi171_78918_object127363:head# 2024-04-20T17:11:09.242 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f59f91e:::benchmark_data_smithi171_78918_object98858:head# 2024-04-20T17:11:09.242 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f5a69b6:::benchmark_data_smithi171_78918_object62724:head# 2024-04-20T17:11:09.242 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f5a7d32:::benchmark_data_smithi171_78918_object69707:head# 2024-04-20T17:11:09.242 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f5a814b:::benchmark_data_smithi171_78918_object104542:head# 2024-04-20T17:11:09.242 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f5aba44:::benchmark_data_smithi171_78918_object69708:head# 2024-04-20T17:11:09.242 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f5b00b0:::benchmark_data_smithi171_78918_object138946:head# 2024-04-20T17:11:09.242 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f5bc5d5:::benchmark_data_smithi171_78918_object101735:head# 2024-04-20T17:11:09.242 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f5c742f:::benchmark_data_smithi171_78918_object160700:head# 2024-04-20T17:11:09.242 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f5ddaff:::benchmark_data_smithi171_78918_object146864:head# 2024-04-20T17:11:09.242 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f5e0d36:::benchmark_data_smithi171_78918_object110824:head# 2024-04-20T17:11:09.242 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f5eaa0f:::benchmark_data_smithi171_78918_object153936:head# 2024-04-20T17:11:09.242 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f600bc9:::benchmark_data_smithi171_78918_object149522:head# 2024-04-20T17:11:09.242 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f60b61b:::benchmark_data_smithi171_78918_object127502:head# 2024-04-20T17:11:09.242 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f61b8ae:::benchmark_data_smithi171_78918_object153972:head# 2024-04-20T17:11:09.243 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f61b8e2:::benchmark_data_smithi171_78918_object104264:head# 2024-04-20T17:11:09.243 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f620f53:::benchmark_data_smithi171_78918_object93156:head# 2024-04-20T17:11:09.243 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f62750d:::benchmark_data_smithi171_78918_object140970:head# 2024-04-20T17:11:09.243 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f62e90e:::benchmark_data_smithi171_78918_object53309:head# 2024-04-20T17:11:09.243 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f6448cc:::benchmark_data_smithi171_78918_object152092:head# 2024-04-20T17:11:09.243 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f644b4f:::benchmark_data_smithi171_78918_object75739:head# 2024-04-20T17:11:09.243 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f645b85:::benchmark_data_smithi171_78918_object63060:head# 2024-04-20T17:11:09.243 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f6481f2:::benchmark_data_smithi171_78918_object75444:head# 2024-04-20T17:11:09.243 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f65016b:::benchmark_data_smithi171_78918_object126593:head# 2024-04-20T17:11:09.243 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f652845:::benchmark_data_smithi171_78918_object64886:head# 2024-04-20T17:11:09.243 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f660921:::benchmark_data_smithi171_78918_object124242:head# 2024-04-20T17:11:09.243 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f66adc9:::benchmark_data_smithi171_78918_object116178:head# 2024-04-20T17:11:09.243 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f67834a:::benchmark_data_smithi171_78918_object133797:head# 2024-04-20T17:11:09.243 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f678b1c:::benchmark_data_smithi171_78918_object84017:head# 2024-04-20T17:11:09.244 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f679b7d:::benchmark_data_smithi171_78918_object66130:head# 2024-04-20T17:11:09.244 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f688978:::benchmark_data_smithi171_78918_object116511:head# 2024-04-20T17:11:09.244 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f68fe78:::benchmark_data_smithi171_78918_object154818:head# 2024-04-20T17:11:09.244 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f694ce2:::benchmark_data_smithi171_78918_object60205:head# 2024-04-20T17:11:09.244 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f69bb46:::benchmark_data_smithi171_78918_object85278:head# 2024-04-20T17:11:09.244 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f6a0fff:::benchmark_data_smithi171_78918_object67944:head# 2024-04-20T17:11:09.244 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f6a1572:::benchmark_data_smithi171_78918_object95746:head# 2024-04-20T17:11:09.244 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f6a7aa7:::benchmark_data_smithi171_78918_object121515:head# 2024-04-20T17:11:09.244 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f6abf8a:::benchmark_data_smithi171_78918_object111698:head# 2024-04-20T17:11:09.244 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f6ae97f:::benchmark_data_smithi171_78918_object137705:head# 2024-04-20T17:11:09.244 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f6bdc0f:::benchmark_data_smithi171_78918_object127173:head# 2024-04-20T17:11:09.244 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f6ca743:::benchmark_data_smithi171_78918_object81288:head# 2024-04-20T17:11:09.244 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f6dc778:::benchmark_data_smithi171_78918_object116567:head# 2024-04-20T17:11:09.244 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f6e5220:::benchmark_data_smithi171_78918_object158080:head# 2024-04-20T17:11:09.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f6e6108:::benchmark_data_smithi171_78918_object102636:head# 2024-04-20T17:11:09.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f6e8309:::benchmark_data_smithi171_78918_object71142:head# 2024-04-20T17:11:09.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f6ec039:::benchmark_data_smithi171_78918_object148622:head# 2024-04-20T17:11:09.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f6f1088:::benchmark_data_smithi171_78918_object116887:head# 2024-04-20T17:11:09.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f707558:::benchmark_data_smithi171_78918_object111119:head# 2024-04-20T17:11:09.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f716b42:::benchmark_data_smithi171_78918_object62526:head# 2024-04-20T17:11:09.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f72421f:::benchmark_data_smithi171_78918_object108036:head# 2024-04-20T17:11:09.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f726366:::benchmark_data_smithi171_78918_object156549:head# 2024-04-20T17:11:09.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f7507dc:::benchmark_data_smithi171_78918_object72890:head# 2024-04-20T17:11:09.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f758ab9:::benchmark_data_smithi171_78918_object134182:head# 2024-04-20T17:11:09.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f75f505:::benchmark_data_smithi171_78918_object69546:head# 2024-04-20T17:11:09.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f760908:::benchmark_data_smithi171_78918_object145403:head# 2024-04-20T17:11:09.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f7625fa:::benchmark_data_smithi171_78918_object87914:head# 2024-04-20T17:11:09.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f77a6e2:::benchmark_data_smithi171_78918_object115029:head# 2024-04-20T17:11:09.245 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f783a4a:::benchmark_data_smithi171_78918_object79372:head# 2024-04-20T17:11:09.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f79aeef:::benchmark_data_smithi171_78918_object145086:head# 2024-04-20T17:11:09.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f7a4471:::benchmark_data_smithi171_78918_object103444:head# 2024-04-20T17:11:09.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f7b54af:::benchmark_data_smithi171_78918_object120617:head# 2024-04-20T17:11:09.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f7bf6c5:::benchmark_data_smithi171_78918_object76452:head# 2024-04-20T17:11:09.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f7d10bc:::benchmark_data_smithi171_78918_object127732:head# 2024-04-20T17:11:09.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f7d3fa2:::benchmark_data_smithi171_78918_object147950:head# 2024-04-20T17:11:09.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f7da0d9:::benchmark_data_smithi171_78918_object108651:head# 2024-04-20T17:11:09.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f7dec31:::benchmark_data_smithi171_78918_object57768:head# 2024-04-20T17:11:09.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f7e2510:::benchmark_data_smithi171_78918_object123322:head# 2024-04-20T17:11:09.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f7fb0c6:::benchmark_data_smithi171_78918_object126278:head# 2024-04-20T17:11:09.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f8048d6:::benchmark_data_smithi171_78918_object91574:head# 2024-04-20T17:11:09.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f81b7ba:::benchmark_data_smithi171_78918_object136737:head# 2024-04-20T17:11:09.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f83776e:::benchmark_data_smithi171_78918_object117039:head# 2024-04-20T17:11:09.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f839b2e:::benchmark_data_smithi171_78918_object62917:head# 2024-04-20T17:11:09.246 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f845f7b:::benchmark_data_smithi171_78918_object127362:head# 2024-04-20T17:11:09.247 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f846394:::benchmark_data_smithi171_78918_object120489:head# 2024-04-20T17:11:09.247 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f84a251:::benchmark_data_smithi171_78918_object91167:head# 2024-04-20T17:11:09.247 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f84e80c:::benchmark_data_smithi171_78918_object133682:head# 2024-04-20T17:11:09.247 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f866a29:::benchmark_data_smithi171_78918_object101508:head# 2024-04-20T17:11:09.247 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f879b76:::benchmark_data_smithi171_78918_object145154:head# 2024-04-20T17:11:09.247 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f87a66b:::benchmark_data_smithi171_78918_object126759:head# 2024-04-20T17:11:09.247 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f87f578:::benchmark_data_smithi171_78918_object119635:head# 2024-04-20T17:11:09.247 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f884653:::benchmark_data_smithi171_78918_object120772:head# 2024-04-20T17:11:09.247 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f88553b:::benchmark_data_smithi171_78918_object140385:head# 2024-04-20T17:11:09.247 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f89103a:::benchmark_data_smithi171_78918_object79370:head# 2024-04-20T17:11:09.247 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f8a4f6e:::benchmark_data_smithi171_78918_object122927:head# 2024-04-20T17:11:09.247 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f8aec4f:::benchmark_data_smithi171_78918_object123827:head# 2024-04-20T17:11:09.247 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f8b5215:::benchmark_data_smithi171_78918_object80726:head# 2024-04-20T17:11:09.247 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f8be229:::benchmark_data_smithi171_78918_object68993:head# 2024-04-20T17:11:09.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f8c72c2:::benchmark_data_smithi171_78918_object87671:head# 2024-04-20T17:11:09.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f8e665a:::benchmark_data_smithi171_78918_object113500:head# 2024-04-20T17:11:09.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f8e8ff9:::benchmark_data_smithi171_78918_object129714:head# 2024-04-20T17:11:09.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f8fa4a7:::benchmark_data_smithi171_78918_object158162:head# 2024-04-20T17:11:09.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f90aaa5:::benchmark_data_smithi171_78918_object114763:head# 2024-04-20T17:11:09.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f90e995:::benchmark_data_smithi171_78918_object113951:head# 2024-04-20T17:11:09.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f911849:::benchmark_data_smithi171_78918_object122423:head# 2024-04-20T17:11:09.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f91e068:::benchmark_data_smithi171_78918_object142183:head# 2024-04-20T17:11:09.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f925174:::benchmark_data_smithi171_78918_object157318:head# 2024-04-20T17:11:09.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f928be0:::benchmark_data_smithi171_78918_object77122:head# 2024-04-20T17:11:09.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f92eb13:::benchmark_data_smithi171_78918_object111668:head# 2024-04-20T17:11:09.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f9480e6:::benchmark_data_smithi171_78918_object124379:head# 2024-04-20T17:11:09.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f95286d:::benchmark_data_smithi171_78918_object109982:head# 2024-04-20T17:11:09.248 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f954e39:::benchmark_data_smithi171_78918_object107441:head# 2024-04-20T17:11:09.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f95ca56:::benchmark_data_smithi171_78918_object144069:head# 2024-04-20T17:11:09.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f9649e0:::benchmark_data_smithi171_78918_object99701:head# 2024-04-20T17:11:09.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f96890d:::benchmark_data_smithi171_78918_object128175:head# 2024-04-20T17:11:09.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f977efd:::benchmark_data_smithi171_78918_object70171:head# 2024-04-20T17:11:09.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f9811ea:::benchmark_data_smithi171_78918_object84227:head# 2024-04-20T17:11:09.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f98a4cb:::benchmark_data_smithi171_78918_object155620:head# 2024-04-20T17:11:09.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f993a2f:::benchmark_data_smithi171_78918_object79502:head# 2024-04-20T17:11:09.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f995894:::benchmark_data_smithi171_78918_object144117:head# 2024-04-20T17:11:09.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f99ba94:::benchmark_data_smithi171_78918_object141251:head# 2024-04-20T17:11:09.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f9a075e:::benchmark_data_smithi171_78918_object91015:head# 2024-04-20T17:11:09.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f9a163a:::benchmark_data_smithi171_78918_object51772:head# 2024-04-20T17:11:09.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f9ab684:::benchmark_data_smithi171_78918_object82039:head# 2024-04-20T17:11:09.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f9b051b:::benchmark_data_smithi171_78918_object150122:head# 2024-04-20T17:11:09.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f9b9530:::benchmark_data_smithi171_78918_object129290:head# 2024-04-20T17:11:09.249 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f9d09e0:::benchmark_data_smithi171_78918_object115435:head# 2024-04-20T17:11:09.250 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f9d7714:::benchmark_data_smithi171_78918_object53123:head# 2024-04-20T17:11:09.250 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f9db9df:::benchmark_data_smithi171_78918_object153928:head# 2024-04-20T17:11:09.250 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f9de6d5:::benchmark_data_smithi171_78918_object157948:head# 2024-04-20T17:11:09.250 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f9e0372:::benchmark_data_smithi171_78918_object93994:head# 2024-04-20T17:11:09.250 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f9e1376:::benchmark_data_smithi171_78918_object62152:head# 2024-04-20T17:11:09.250 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f9e5698:::benchmark_data_smithi171_78918_object144439:head# 2024-04-20T17:11:09.250 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7f9eec05:::benchmark_data_smithi171_78918_object127201:head# 2024-04-20T17:11:09.250 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fa0c715:::benchmark_data_smithi171_78918_object116456:head# 2024-04-20T17:11:09.250 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fa0f0fe:::benchmark_data_smithi171_78918_object83591:head# 2024-04-20T17:11:09.250 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fa34835:::benchmark_data_smithi171_78918_object128314:head# 2024-04-20T17:11:09.250 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fa386d8:::benchmark_data_smithi171_78918_object73965:head# 2024-04-20T17:11:09.250 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fa3fa93:::benchmark_data_smithi171_78918_object52905:head# 2024-04-20T17:11:09.250 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fa4a3d2:::benchmark_data_smithi171_78918_object103696:head# 2024-04-20T17:11:09.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fa5d80b:::benchmark_data_smithi171_78918_object143846:head# 2024-04-20T17:11:09.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fa5ffaa:::benchmark_data_smithi171_78918_object69504:head# 2024-04-20T17:11:09.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fa60edc:::benchmark_data_smithi171_78918_object57272:head# 2024-04-20T17:11:09.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fa6a015:::benchmark_data_smithi171_78918_object59226:head# 2024-04-20T17:11:09.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fa6bf11:::benchmark_data_smithi171_78918_object158760:head# 2024-04-20T17:11:09.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fa715c6:::benchmark_data_smithi171_78918_object78022:head# 2024-04-20T17:11:09.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fa7b71b:::benchmark_data_smithi171_78918_object136631:head# 2024-04-20T17:11:09.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fa7dd38:::benchmark_data_smithi171_78918_object63084:head# 2024-04-20T17:11:09.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fa8ee6c:::benchmark_data_smithi171_78918_object53835:head# 2024-04-20T17:11:09.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fa9a3cb:::benchmark_data_smithi171_78918_object110104:head# 2024-04-20T17:11:09.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fa9f186:::benchmark_data_smithi171_78918_object89923:head# 2024-04-20T17:11:09.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7faa2ba9:::benchmark_data_smithi171_78918_object99696:head# 2024-04-20T17:11:09.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7faab486:::benchmark_data_smithi171_78918_object86425:head# 2024-04-20T17:11:09.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fabb0e1:::benchmark_data_smithi171_78918_object153570:head# 2024-04-20T17:11:09.251 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fabb98f:::benchmark_data_smithi171_78918_object61948:head# 2024-04-20T17:11:09.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fabd30d:::benchmark_data_smithi171_78918_object153583:head# 2024-04-20T17:11:09.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fabda3a:::benchmark_data_smithi171_78918_object104473:head# 2024-04-20T17:11:09.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fac606e:::benchmark_data_smithi171_78918_object55758:head# 2024-04-20T17:11:09.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fac77a2:::benchmark_data_smithi171_78918_object96285:head# 2024-04-20T17:11:09.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7facb511:::benchmark_data_smithi171_78918_object83095:head# 2024-04-20T17:11:09.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7facc2fb:::benchmark_data_smithi171_78918_object106128:head# 2024-04-20T17:11:09.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fad20a0:::benchmark_data_smithi171_78918_object117403:head# 2024-04-20T17:11:09.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fad391a:::benchmark_data_smithi171_78918_object120824:head# 2024-04-20T17:11:09.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fae0e12:::benchmark_data_smithi171_78918_object101831:head# 2024-04-20T17:11:09.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7faf1f35:::benchmark_data_smithi171_78918_object104527:head# 2024-04-20T17:11:09.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7faf9370:::benchmark_data_smithi171_78918_object122049:head# 2024-04-20T17:11:09.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fafc47a:::benchmark_data_smithi171_78918_object146961:head# 2024-04-20T17:11:09.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fb025b4:::benchmark_data_smithi171_78918_object122183:head# 2024-04-20T17:11:09.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fb04c44:::benchmark_data_smithi171_78918_object97241:head# 2024-04-20T17:11:09.252 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fb0e9d3:::benchmark_data_smithi171_78918_object59248:head# 2024-04-20T17:11:09.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fb1c2bd:::benchmark_data_smithi171_78918_object156210:head# 2024-04-20T17:11:09.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fb2c034:::benchmark_data_smithi171_78918_object89591:head# 2024-04-20T17:11:09.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fb31ddd:::benchmark_data_smithi171_78918_object63797:head# 2024-04-20T17:11:09.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fb3719a:::benchmark_data_smithi171_78918_object156343:head# 2024-04-20T17:11:09.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fb3a23d:::benchmark_data_smithi171_78918_object132547:head# 2024-04-20T17:11:09.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fb42348:::benchmark_data_smithi171_78918_object147154:head# 2024-04-20T17:11:09.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fb4afe7:::benchmark_data_smithi171_78918_object60315:head# 2024-04-20T17:11:09.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fb55ab8:::benchmark_data_smithi171_78918_object86845:head# 2024-04-20T17:11:09.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fb66485:::benchmark_data_smithi171_78918_object77276:head# 2024-04-20T17:11:09.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fb6d72a:::benchmark_data_smithi171_78918_object91706:head# 2024-04-20T17:11:09.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fb6f1c2:::benchmark_data_smithi171_78918_object91923:head# 2024-04-20T17:11:09.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fb7ded0:::benchmark_data_smithi171_78918_object59865:head# 2024-04-20T17:11:09.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fb9ffa2:::benchmark_data_smithi171_78918_object146133:head# 2024-04-20T17:11:09.253 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fba428d:::benchmark_data_smithi171_78918_object122688:head# 2024-04-20T17:11:09.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fba758f:::benchmark_data_smithi171_78918_object118583:head# 2024-04-20T17:11:09.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fbc3348:::benchmark_data_smithi171_78918_object107435:head# 2024-04-20T17:11:09.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fbc9598:::benchmark_data_smithi171_78918_object53098:head# 2024-04-20T17:11:09.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fbca922:::benchmark_data_smithi171_78918_object75418:head# 2024-04-20T17:11:09.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fbe8619:::benchmark_data_smithi171_78918_object155291:head# 2024-04-20T17:11:09.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fbfd5fa:::benchmark_data_smithi171_78918_object79400:head# 2024-04-20T17:11:09.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fc03a14:::benchmark_data_smithi171_78918_object121639:head# 2024-04-20T17:11:09.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fc2215f:::benchmark_data_smithi171_78918_object141342:head# 2024-04-20T17:11:09.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fc2346a:::benchmark_data_smithi171_78918_object122379:head# 2024-04-20T17:11:09.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fc2b643:::benchmark_data_smithi171_78918_object91944:head# 2024-04-20T17:11:09.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fc2bc2d:::benchmark_data_smithi171_78918_object64113:head# 2024-04-20T17:11:09.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fc2e994:::benchmark_data_smithi171_78918_object90648:head# 2024-04-20T17:11:09.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fc36123:::benchmark_data_smithi171_78918_object94761:head# 2024-04-20T17:11:09.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fc3799e:::benchmark_data_smithi171_78918_object159366:head# 2024-04-20T17:11:09.254 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fc3eb6e:::benchmark_data_smithi171_78918_object136693:head# 2024-04-20T17:11:09.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fc438ed:::benchmark_data_smithi171_78918_object159829:head# 2024-04-20T17:11:09.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fc4433b:::benchmark_data_smithi171_78918_object86079:head# 2024-04-20T17:11:09.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fc48be9:::benchmark_data_smithi171_78918_object76892:head# 2024-04-20T17:11:09.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fc5245d:::benchmark_data_smithi171_78918_object98700:head# 2024-04-20T17:11:09.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fc540f9:::benchmark_data_smithi171_78918_object97725:head# 2024-04-20T17:11:09.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fc5e3ef:::benchmark_data_smithi171_78918_object119456:head# 2024-04-20T17:11:09.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fc7575a:::benchmark_data_smithi171_78918_object66978:head# 2024-04-20T17:11:09.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fc7756d:::benchmark_data_smithi171_78918_object61770:head# 2024-04-20T17:11:09.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fc83f88:::benchmark_data_smithi171_78918_object77630:head# 2024-04-20T17:11:09.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fc845a5:::benchmark_data_smithi171_78918_object65307:head# 2024-04-20T17:11:09.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fc9a265:::benchmark_data_smithi171_78918_object129906:head# 2024-04-20T17:11:09.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fcad362:::benchmark_data_smithi171_78918_object114101:head# 2024-04-20T17:11:09.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fcb0b72:::benchmark_data_smithi171_78918_object124849:head# 2024-04-20T17:11:09.255 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fcbebb4:::benchmark_data_smithi171_78918_object107268:head# 2024-04-20T17:11:09.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fcbf5aa:::benchmark_data_smithi171_78918_object51978:head# 2024-04-20T17:11:09.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fcc1616:::benchmark_data_smithi171_78918_object104192:head# 2024-04-20T17:11:09.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fccf11b:::benchmark_data_smithi171_78918_object122562:head# 2024-04-20T17:11:09.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fce427a:::benchmark_data_smithi171_78918_object63693:head# 2024-04-20T17:11:09.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fce67d7:::benchmark_data_smithi171_78918_object93298:head# 2024-04-20T17:11:09.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fcee39b:::benchmark_data_smithi171_78918_object129627:head# 2024-04-20T17:11:09.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fcf6791:::benchmark_data_smithi171_78918_object152720:head# 2024-04-20T17:11:09.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fcf89bf:::benchmark_data_smithi171_78918_object152236:head# 2024-04-20T17:11:09.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fcffd00:::benchmark_data_smithi171_78918_object158824:head# 2024-04-20T17:11:09.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fd05632:::benchmark_data_smithi171_78918_object99929:head# 2024-04-20T17:11:09.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fd0ab0c:::benchmark_data_smithi171_78918_object79344:head# 2024-04-20T17:11:09.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fd18d56:::benchmark_data_smithi171_78918_object94822:head# 2024-04-20T17:11:09.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fd2979e:::benchmark_data_smithi171_78918_object141802:head# 2024-04-20T17:11:09.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fd38cc4:::benchmark_data_smithi171_78918_object105745:head# 2024-04-20T17:11:09.256 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fd3acfa:::benchmark_data_smithi171_78918_object157587:head# 2024-04-20T17:11:09.257 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fd3dc79:::benchmark_data_smithi171_78918_object111858:head# 2024-04-20T17:11:09.257 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fd469df:::benchmark_data_smithi171_78918_object88683:head# 2024-04-20T17:11:09.257 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fd4c365:::benchmark_data_smithi171_78918_object80284:head# 2024-04-20T17:11:09.257 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fd52b65:::benchmark_data_smithi171_78918_object117720:head# 2024-04-20T17:11:09.257 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fd52fb9:::benchmark_data_smithi171_78918_object114051:head# 2024-04-20T17:11:09.257 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fd542e6:::benchmark_data_smithi171_78918_object85797:head# 2024-04-20T17:11:09.257 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fd6265b:::benchmark_data_smithi171_78918_object112367:head# 2024-04-20T17:11:09.257 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fd6c35f:::benchmark_data_smithi171_78918_object86851:head# 2024-04-20T17:11:09.257 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fd78ce1:::benchmark_data_smithi171_78918_object105914:head# 2024-04-20T17:11:09.257 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fd7ecfc:::benchmark_data_smithi171_78918_object142842:head# 2024-04-20T17:11:09.257 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fd8c6e1:::benchmark_data_smithi171_78918_object117740:head# 2024-04-20T17:11:09.257 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fd9ca9e:::benchmark_data_smithi171_78918_object76545:head# 2024-04-20T17:11:09.257 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fda2e6e:::benchmark_data_smithi171_78918_object97214:head# 2024-04-20T17:11:09.257 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fdab88f:::benchmark_data_smithi171_78918_object118483:head# 2024-04-20T17:11:09.257 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fdae1b6:::benchmark_data_smithi171_78918_object141623:head# 2024-04-20T17:11:09.258 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fdc11fe:::benchmark_data_smithi171_78918_object78290:head# 2024-04-20T17:11:09.258 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fdc3107:::benchmark_data_smithi171_78918_object155702:head# 2024-04-20T17:11:09.258 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fdc6dc2:::benchmark_data_smithi171_78918_object103284:head# 2024-04-20T17:11:09.258 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fdce067:::benchmark_data_smithi171_78918_object67095:head# 2024-04-20T17:11:09.258 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fdd84b3:::benchmark_data_smithi171_78918_object91335:head# 2024-04-20T17:11:09.258 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fde1dbe:::benchmark_data_smithi171_78918_object84129:head# 2024-04-20T17:11:09.258 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fde3e81:::benchmark_data_smithi171_78918_object105159:head# 2024-04-20T17:11:09.258 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fde4f03:::benchmark_data_smithi171_78918_object147443:head# 2024-04-20T17:11:09.258 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fe03632:::benchmark_data_smithi171_78918_object102912:head# 2024-04-20T17:11:09.258 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fe07f36:::benchmark_data_smithi171_78918_object96979:head# 2024-04-20T17:11:09.258 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fe0b601:::benchmark_data_smithi171_78918_object122929:head# 2024-04-20T17:11:09.258 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fe1da77:::benchmark_data_smithi171_78918_object71992:head# 2024-04-20T17:11:09.258 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fe22673:::benchmark_data_smithi171_78918_object106246:head# 2024-04-20T17:11:09.258 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fe41b6f:::benchmark_data_smithi171_78918_object115382:head# 2024-04-20T17:11:09.259 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fe4c2c8:::benchmark_data_smithi171_78918_object76700:head# 2024-04-20T17:11:09.259 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fe5aab9:::benchmark_data_smithi171_78918_object130996:head# 2024-04-20T17:11:09.259 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fe5d3f0:::benchmark_data_smithi171_78918_object60428:head# 2024-04-20T17:11:09.259 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fe68b09:::benchmark_data_smithi171_78918_object125668:head# 2024-04-20T17:11:09.259 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fe91cee:::benchmark_data_smithi171_78918_object62845:head# 2024-04-20T17:11:09.259 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fe952fa:::benchmark_data_smithi171_78918_object61835:head# 2024-04-20T17:11:09.259 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fe96b14:::benchmark_data_smithi171_78918_object128452:head# 2024-04-20T17:11:09.259 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fea5044:::benchmark_data_smithi171_78918_object56883:head# 2024-04-20T17:11:09.259 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7feaf7ac:::benchmark_data_smithi171_78918_object67832:head# 2024-04-20T17:11:09.259 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7febab3d:::benchmark_data_smithi171_78918_object118456:head# 2024-04-20T17:11:09.259 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7febb5ff:::benchmark_data_smithi171_78918_object52216:head# 2024-04-20T17:11:09.259 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7febccda:::benchmark_data_smithi171_78918_object148253:head# 2024-04-20T17:11:09.259 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fee2557:::benchmark_data_smithi171_78918_object86659:head# 2024-04-20T17:11:09.259 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fee5c32:::benchmark_data_smithi171_78918_object154329:head# 2024-04-20T17:11:09.259 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7feeec89:::benchmark_data_smithi171_78918_object78889:head# 2024-04-20T17:11:09.260 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ff0b874:::benchmark_data_smithi171_78918_object142060:head# 2024-04-20T17:11:09.260 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ff19787:::benchmark_data_smithi171_78918_object73497:head# 2024-04-20T17:11:09.260 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ff2e6a6:::benchmark_data_smithi171_78918_object138004:head# 2024-04-20T17:11:09.260 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ff2fbf0:::benchmark_data_smithi171_78918_object131199:head# 2024-04-20T17:11:09.260 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ff3054e:::benchmark_data_smithi171_78918_object68022:head# 2024-04-20T17:11:09.260 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ff30a64:::benchmark_data_smithi171_78918_object118094:head# 2024-04-20T17:11:09.260 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ff321a9:::benchmark_data_smithi171_78918_object112908:head# 2024-04-20T17:11:09.260 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ff3bbdc:::benchmark_data_smithi171_78918_object118765:head# 2024-04-20T17:11:09.260 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ff3f5f3:::benchmark_data_smithi171_78918_object153048:head# 2024-04-20T17:11:09.260 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ff41ee0:::benchmark_data_smithi171_78918_object148683:head# 2024-04-20T17:11:09.260 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ff4597a:::benchmark_data_smithi171_78918_object109950:head# 2024-04-20T17:11:09.260 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ff67a64:::benchmark_data_smithi171_78918_object87044:head# 2024-04-20T17:11:09.260 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ff741bd:::benchmark_data_smithi171_78918_object91014:head# 2024-04-20T17:11:09.260 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ff7c5de:::benchmark_data_smithi171_78918_object112519:head# 2024-04-20T17:11:09.260 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ff7e8bf:::benchmark_data_smithi171_78918_object77831:head# 2024-04-20T17:11:09.261 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ff8e0c3:::benchmark_data_smithi171_78918_object58026:head# 2024-04-20T17:11:09.261 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ffae170:::benchmark_data_smithi171_78918_object136355:head# 2024-04-20T17:11:09.261 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ffc80cf:::benchmark_data_smithi171_78918_object120428:head# 2024-04-20T17:11:09.261 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ffcd54c:::benchmark_data_smithi171_78918_object107688:head# 2024-04-20T17:11:09.261 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ffe1948:::benchmark_data_smithi171_78918_object131509:head# 2024-04-20T17:11:09.261 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7ffe5828:::benchmark_data_smithi171_78918_object101417:head# 2024-04-20T17:11:09.261 INFO:teuthology.orchestra.run.smithi005.stderr:Read #2:7fffea73:::benchmark_data_smithi171_78918_object158314:head# 2024-04-20T17:11:09.261 INFO:teuthology.orchestra.run.smithi005.stderr:Export successful 2024-04-20T17:11:09.262 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:09 smithi171 ceph-mon[44189]: pgmap v518: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 6 op/s; 19978/218982 objects degraded (9.123%); 6237/218982 objects misplaced (2.848%); 376 KiB/s, 5 objects/s recovering 2024-04-20T17:11:09.262 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:09 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/981466524' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:11:09.262 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:09.262 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:09.263 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-20T17:11:09.384 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:09 smithi005 ceph-mon[39562]: pgmap v518: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 6 op/s; 19978/218982 objects degraded (9.123%); 6237/218982 objects misplaced (2.848%); 376 KiB/s, 5 objects/s recovering 2024-04-20T17:11:09.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:09 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/981466524' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:11:09.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:09.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:09.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:09 smithi106 ceph-mon[44423]: pgmap v518: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 6 op/s; 19978/218982 objects degraded (9.123%); 6237/218982 objects misplaced (2.848%); 376 KiB/s, 5 objects/s recovering 2024-04-20T17:11:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:09 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/981466524' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:11:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:09.525 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:11:09.939 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:11:09.939 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:11:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:10 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:11:10.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:10.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:10.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:10.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:10.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:10.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:10.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:10.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:10.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:10.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:10 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:11:10.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:10 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:11:10.770 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.2 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --force --op remove --pgid 2.e 2024-04-20T17:11:11.100 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.2/config 2024-04-20T17:11:11.106 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:11:11.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:11.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:11:11.185 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:11 smithi171 ceph-mon[44189]: pgmap v519: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 5 op/s; 19923/218982 objects degraded (9.098%); 6181/218982 objects misplaced (2.823%); 820 KiB/s, 12 objects/s recovering 2024-04-20T17:11:11.185 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:11 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 19923/218982 objects degraded (9.098%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:11.185 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:11 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:11:11.185 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:11 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:11:11.185 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:11 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:11:11.211 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:11:11.240 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:11.313 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:11:11.317 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:11 smithi005 ceph-mon[39562]: pgmap v519: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 5 op/s; 19923/218982 objects degraded (9.098%); 6181/218982 objects misplaced (2.823%); 820 KiB/s, 12 objects/s recovering 2024-04-20T17:11:11.317 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:11 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 19923/218982 objects degraded (9.098%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:11.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:11 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:11:11.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:11 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:11:11.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:11 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:11:11.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:11.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:11:11.420 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:11:11.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:11 smithi106 ceph-mon[44423]: pgmap v519: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 5 op/s; 19923/218982 objects degraded (9.098%); 6181/218982 objects misplaced (2.823%); 820 KiB/s, 12 objects/s recovering 2024-04-20T17:11:11.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:11 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 19923/218982 objects degraded (9.098%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:11.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:11 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:11:11.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:11 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:11:11.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:11 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:11:11.448 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:11.468 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:11:11.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:11.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:11:11.487 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:11:11.487 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:11:11.572 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:11:11.601 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:11.748 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:11:11.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:11.751 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:11:11.852 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:11:11.879 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:11.940 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:11:12.185 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-20T17:11:12.228 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:11:12.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:12.230 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:11:12.330 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:11:12.363 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:12.385 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:11:12.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:12.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:11:12.485 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:11:12.488 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:11:12.520 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:12.578 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:11:12.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:12.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:11:12.682 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:11:12.715 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:12.736 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:11:12.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:12.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:11:12.840 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:11:12.867 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:12.887 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:11:12.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:12.890 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:11:12.991 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:11:13.017 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:13.021 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:11:13.054 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:11:13.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:13.058 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:11:13.061 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:11:13.074 INFO:teuthology.orchestra.run.smithi005.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:11:13.159 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:11:13.192 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:13.213 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:11:13.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:13.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:11:13.317 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:11:13.333 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:13 smithi005 ceph-mon[39562]: pgmap v520: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 4 op/s; 19923/218982 objects degraded (9.098%); 6181/218982 objects misplaced (2.823%); 659 KiB/s, 10 objects/s recovering 2024-04-20T17:11:13.333 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:13 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2044903574' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:11:13.333 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:13 smithi005 ceph-mon[39562]: from='client.24914 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:11:13.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:13 smithi171 ceph-mon[44189]: pgmap v520: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 4 op/s; 19923/218982 objects degraded (9.098%); 6181/218982 objects misplaced (2.823%); 659 KiB/s, 10 objects/s recovering 2024-04-20T17:11:13.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:13 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2044903574' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:11:13.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:13 smithi171 ceph-mon[44189]: from='client.24914 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:11:13.344 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:13 smithi106 ceph-mon[44423]: pgmap v520: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 4 op/s; 19923/218982 objects degraded (9.098%); 6181/218982 objects misplaced (2.823%); 659 KiB/s, 10 objects/s recovering 2024-04-20T17:11:13.344 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:13 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2044903574' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:11:13.344 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:13 smithi106 ceph-mon[44423]: from='client.24914 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:11:13.346 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:13.366 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:11:13.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:13.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:11:13.470 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:11:13.497 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:14.226 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:14 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:11:14.226 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:14 smithi171 ceph-mon[44189]: from='client.24914 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:11:14.226 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:14 smithi171 ceph-mon[44189]: osdmap e178: 12 total, 11 up, 12 in 2024-04-20T17:11:14.246 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:11:14.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:14.249 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:11:14.350 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:11:14.379 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:14.398 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:11:14.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:14.401 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:11:14.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:14 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:11:14.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:14 smithi005 ceph-mon[39562]: from='client.24914 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:11:14.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:14 smithi005 ceph-mon[39562]: osdmap e178: 12 total, 11 up, 12 in 2024-04-20T17:11:14.502 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:11:14.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:14 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:11:14.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:14 smithi106 ceph-mon[44423]: from='client.24914 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:11:14.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:14 smithi106 ceph-mon[44423]: osdmap e178: 12 total, 11 up, 12 in 2024-04-20T17:11:14.529 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:14.529 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:11:14.529 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:11:14.548 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:11:14.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:14.550 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:11:14.651 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:11:14.682 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:14.883 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:11:14.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:14.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:11:14.989 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:11:15.022 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:15.043 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:11:15.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:15.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:11:15.147 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:11:15.177 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:15.197 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:11:15.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:15.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:11:15.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:15 smithi005 ceph-mon[39562]: pgmap v522: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1 op/s; 19923/218982 objects degraded (9.098%); 6181/218982 objects misplaced (2.823%); 659 KiB/s, 10 objects/s recovering 2024-04-20T17:11:15.225 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:15 smithi171 ceph-mon[44189]: pgmap v522: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1 op/s; 19923/218982 objects degraded (9.098%); 6181/218982 objects misplaced (2.823%); 659 KiB/s, 10 objects/s recovering 2024-04-20T17:11:15.250 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-20T17:11:15.301 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:11:15.334 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:15.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:15 smithi106 ceph-mon[44423]: pgmap v522: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1 op/s; 19923/218982 objects degraded (9.098%); 6181/218982 objects misplaced (2.823%); 659 KiB/s, 10 objects/s recovering 2024-04-20T17:11:15.511 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:11:15.587 INFO:teuthology.orchestra.run.smithi005.stdout: marking collection for removal 2024-04-20T17:11:15.588 INFO:teuthology.orchestra.run.smithi005.stdout:setting '_remove' omap key 2024-04-20T17:11:15.589 INFO:teuthology.orchestra.run.smithi005.stdout:finish_remove_pgs 2.e_head removing 2.e 2024-04-20T17:11:15.684 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:11:15.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:15.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:11:15.704 INFO:teuthology.orchestra.run.smithi005.stdout:Remove successful 2024-04-20T17:11:15.791 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:11:15.822 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:16.071 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:11:16.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:16.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:11:16.177 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:11:16.209 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:16.230 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:11:16.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:16.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:11:16.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:16 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 19866/218982 objects degraded (9.072%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:16.334 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:11:16.361 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:16.414 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:11:16.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:16.417 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:11:16.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:16 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 19866/218982 objects degraded (9.072%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:16.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:16 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 19866/218982 objects degraded (9.072%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:16.518 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:11:16.544 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:11:16.544 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:11:16.544 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:11:16.544 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:11:16.544 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:11:16.544 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:11:16.544 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:11:16.544 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:11:16.544 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:11:16.545 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:11:16.548 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:16.566 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:11:16.567 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:11:16.567 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:11:16.567 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-20T17:11:16.567 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:11:16.567 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:11:16.567 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:11:16.567 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:11:16.567 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:11:16.567 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:11:16.567 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:11:16.567 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:11:16.567 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:11:16.568 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:11:16.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:16.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:11:16.660 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:11:16.672 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:11:16.734 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:17.135 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.2 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.e.2 2024-04-20T17:11:17.222 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:17 smithi171 ceph-mon[44189]: pgmap v523: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19866/218982 objects degraded (9.072%); 6123/218982 objects misplaced (2.796%); 729 KiB/s, 11 objects/s recovering 2024-04-20T17:11:17.222 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:17 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/922834618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:11:17.222 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:17.222 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:17.222 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:17.222 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:17.222 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:17.223 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:17.223 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:17.391 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:11:17.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:17 smithi005 ceph-mon[39562]: pgmap v523: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19866/218982 objects degraded (9.072%); 6123/218982 objects misplaced (2.796%); 729 KiB/s, 11 objects/s recovering 2024-04-20T17:11:17.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:17 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/922834618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:11:17.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:17.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:17.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:17.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:17.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:17.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:17.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:17.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:17.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:11:17.405 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.2/config 2024-04-20T17:11:17.427 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:11:17.497 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:11:17.498 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:11:17.499 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:11:17.499 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:11:17.499 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:11:17.499 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.499 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:137417 2.d 2:b75f1f6e:::benchmark_data_smithi171_78918_object137416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-20T17:11:17.499 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:09:22.149036+0000", 2024-04-20T17:11:17.499 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 115.34601001199999, 2024-04-20T17:11:17.499 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021732422000000001, 2024-04-20T17:11:17.499 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:17.499 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:17.499 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:17.499 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:17.499 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:17.499 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:17.500 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 137417 2024-04-20T17:11:17.500 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.500 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:17.500 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.500 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:17.500 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.149036+0000", 2024-04-20T17:11:17.500 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.500 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.500 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.500 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:17.500 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.149036+0000", 2024-04-20T17:11:17.500 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.500 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.500 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.500 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:17.500 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.149037+0000", 2024-04-20T17:11:17.500 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8400000000000003e-07 2024-04-20T17:11:17.501 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.501 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.501 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:17.501 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.149059+0000", 2024-04-20T17:11:17.501 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1903e-05 2024-04-20T17:11:17.501 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.501 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.501 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:17.501 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.149066+0000", 2024-04-20T17:11:17.501 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.328e-06 2024-04-20T17:11:17.501 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.501 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.501 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:17.501 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.149072+0000", 2024-04-20T17:11:17.502 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5339999999999996e-06 2024-04-20T17:11:17.502 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.502 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.502 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:17.502 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.152593+0000", 2024-04-20T17:11:17.502 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0035202219999999999 2024-04-20T17:11:17.502 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.502 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.502 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:17.503 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.152733+0000", 2024-04-20T17:11:17.503 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000139951 2024-04-20T17:11:17.503 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.503 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.503 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4", 2024-04-20T17:11:17.503 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.152868+0000", 2024-04-20T17:11:17.503 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013591800000000001 2024-04-20T17:11:17.503 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.503 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.503 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:17.503 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.162924+0000", 2024-04-20T17:11:17.503 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010055036 2024-04-20T17:11:17.503 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.504 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.504 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:17.504 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.170653+0000", 2024-04-20T17:11:17.504 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.007729166 2024-04-20T17:11:17.504 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.504 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.504 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:17.504 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.170716+0000", 2024-04-20T17:11:17.504 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3816e-05 2024-04-20T17:11:17.504 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.504 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.504 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:17.504 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.170768+0000", 2024-04-20T17:11:17.505 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1764000000000001e-05 2024-04-20T17:11:17.505 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.505 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:17.505 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.505 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.505 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.505 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:137420 2.d 2:ba7d949b:::benchmark_data_smithi171_78918_object137419:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-20T17:11:17.505 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:09:22.149081+0000", 2024-04-20T17:11:17.505 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 115.34596466399999, 2024-04-20T17:11:17.505 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021865292000000001, 2024-04-20T17:11:17.505 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:17.505 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:17.505 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:17.505 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:17.505 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:17.505 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:17.506 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 137420 2024-04-20T17:11:17.506 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.506 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:17.506 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.506 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:17.506 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.149081+0000", 2024-04-20T17:11:17.506 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.506 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.506 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.506 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:17.506 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.149081+0000", 2024-04-20T17:11:17.506 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.506 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.507 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.507 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:17.507 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.149082+0000", 2024-04-20T17:11:17.507 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8899999999999998e-07 2024-04-20T17:11:17.507 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.507 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.507 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:17.507 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.149102+0000", 2024-04-20T17:11:17.507 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9621000000000001e-05 2024-04-20T17:11:17.507 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.507 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.507 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:17.507 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.149107+0000", 2024-04-20T17:11:17.507 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.699e-06 2024-04-20T17:11:17.507 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.507 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.508 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:17.508 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.149112+0000", 2024-04-20T17:11:17.508 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1800000000000004e-06 2024-04-20T17:11:17.508 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.508 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.508 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:17.508 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.156571+0000", 2024-04-20T17:11:17.508 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0074587129999999996 2024-04-20T17:11:17.508 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.508 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.508 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:17.508 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.156683+0000", 2024-04-20T17:11:17.508 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000111694 2024-04-20T17:11:17.508 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.508 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.508 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4", 2024-04-20T17:11:17.509 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.156822+0000", 2024-04-20T17:11:17.509 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013901699999999999 2024-04-20T17:11:17.509 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.509 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.509 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:17.509 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.162929+0000", 2024-04-20T17:11:17.509 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.006107215 2024-04-20T17:11:17.509 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.509 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.509 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:17.509 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.170844+0000", 2024-04-20T17:11:17.509 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0079146919999999992 2024-04-20T17:11:17.509 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.510 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.510 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:17.510 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.170902+0000", 2024-04-20T17:11:17.510 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8662999999999997e-05 2024-04-20T17:11:17.510 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.510 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.510 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:17.510 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.170946+0000", 2024-04-20T17:11:17.510 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4008999999999998e-05 2024-04-20T17:11:17.510 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.510 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:17.510 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.510 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.511 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.511 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:145569 2.d 2:b4461c6e:::benchmark_data_smithi171_78918_object145568:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-04-20T17:11:17.511 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:09:28.629859+0000", 2024-04-20T17:11:17.511 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 108.865187212, 2024-04-20T17:11:17.511 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022452033, 2024-04-20T17:11:17.511 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:17.511 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:17.511 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:17.511 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:17.511 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:17.511 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:17.511 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 145569 2024-04-20T17:11:17.511 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.511 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:17.512 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.512 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:17.512 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:28.629859+0000", 2024-04-20T17:11:17.512 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.512 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.512 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.512 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:17.512 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:28.629859+0000", 2024-04-20T17:11:17.512 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.512 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.512 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.512 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:17.512 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:28.629860+0000", 2024-04-20T17:11:17.513 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.852e-06 2024-04-20T17:11:17.513 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.513 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.513 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:17.513 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:28.629882+0000", 2024-04-20T17:11:17.513 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1999999999999999e-05 2024-04-20T17:11:17.513 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.513 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.513 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:17.513 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:28.629893+0000", 2024-04-20T17:11:17.513 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0285000000000001e-05 2024-04-20T17:11:17.513 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.513 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.513 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:17.514 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:28.629900+0000", 2024-04-20T17:11:17.514 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3409999999999997e-06 2024-04-20T17:11:17.514 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.514 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.514 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:17.514 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:28.633152+0000", 2024-04-20T17:11:17.514 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0032520309999999998 2024-04-20T17:11:17.514 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.514 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.514 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:17.514 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:28.633304+0000", 2024-04-20T17:11:17.514 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015139900000000001 2024-04-20T17:11:17.515 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.515 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.515 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4", 2024-04-20T17:11:17.515 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:28.633397+0000", 2024-04-20T17:11:17.515 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2984000000000003e-05 2024-04-20T17:11:17.515 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.515 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.515 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:17.515 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:28.644593+0000", 2024-04-20T17:11:17.515 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011196245000000001 2024-04-20T17:11:17.515 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.515 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.515 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:17.516 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:28.652247+0000", 2024-04-20T17:11:17.516 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0076545939999999998 2024-04-20T17:11:17.516 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.516 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.516 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:17.516 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:28.652279+0000", 2024-04-20T17:11:17.516 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1285999999999998e-05 2024-04-20T17:11:17.516 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.516 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.516 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:17.516 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:28.652311+0000", 2024-04-20T17:11:17.516 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2016e-05 2024-04-20T17:11:17.516 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.517 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:17.517 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.517 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.517 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.517 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:146277 2.d 2:be62896a:::benchmark_data_smithi171_78918_object146276:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e145)", 2024-04-20T17:11:17.517 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:09:29.176417+0000", 2024-04-20T17:11:17.517 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 108.318628451, 2024-04-20T17:11:17.517 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026120431999999999, 2024-04-20T17:11:17.517 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:17.517 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:17.517 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:17.517 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:17.517 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:17.517 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:17.518 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 146277 2024-04-20T17:11:17.518 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.518 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:17.518 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.518 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:17.518 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:29.176417+0000", 2024-04-20T17:11:17.518 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.518 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.518 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.518 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:17.518 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:29.176417+0000", 2024-04-20T17:11:17.518 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.518 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.519 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.519 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:17.519 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:29.176418+0000", 2024-04-20T17:11:17.519 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.3099999999999996e-07 2024-04-20T17:11:17.519 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.519 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.519 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:17.519 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:29.176437+0000", 2024-04-20T17:11:17.519 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8912000000000001e-05 2024-04-20T17:11:17.519 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.519 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.519 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:17.519 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:29.176443+0000", 2024-04-20T17:11:17.520 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9800000000000003e-06 2024-04-20T17:11:17.520 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.520 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.520 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:17.520 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:29.176448+0000", 2024-04-20T17:11:17.520 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0780000000000003e-06 2024-04-20T17:11:17.520 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.520 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.520 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:17.520 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:29.185519+0000", 2024-04-20T17:11:17.520 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0090703750000000003 2024-04-20T17:11:17.520 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.520 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.520 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:17.521 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:29.185662+0000", 2024-04-20T17:11:17.521 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014380600000000001 2024-04-20T17:11:17.521 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.521 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.521 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4", 2024-04-20T17:11:17.521 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:29.185801+0000", 2024-04-20T17:11:17.521 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013815600000000001 2024-04-20T17:11:17.521 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.521 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.521 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:17.521 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:29.194240+0000", 2024-04-20T17:11:17.521 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0084392140000000004 2024-04-20T17:11:17.521 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.521 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.521 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:17.522 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:29.202434+0000", 2024-04-20T17:11:17.522 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0081946399999999996 2024-04-20T17:11:17.522 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.522 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.522 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:17.522 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:29.202488+0000", 2024-04-20T17:11:17.522 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3542000000000003e-05 2024-04-20T17:11:17.522 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.522 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.522 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:17.522 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:29.202538+0000", 2024-04-20T17:11:17.522 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9898000000000002e-05 2024-04-20T17:11:17.522 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.522 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:17.523 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.523 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.523 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.523 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:194457 2.8 e163/115 2:196a0b1f:::benchmark_data_smithi171_78918_object33463:head v 163'12077, mlcod=163'12074)", 2024-04-20T17:11:17.523 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:29.179243+0000", 2024-04-20T17:11:17.523 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 48.315802771999998, 2024-04-20T17:11:17.523 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030984006000000001, 2024-04-20T17:11:17.523 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:17.523 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:17.523 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:17.523 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:17.523 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.523 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:17.523 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.179243+0000", 2024-04-20T17:11:17.523 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.524 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.524 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.524 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:17.524 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.179243+0000", 2024-04-20T17:11:17.524 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.524 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.524 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.524 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:17.524 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.179246+0000", 2024-04-20T17:11:17.524 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8899999999999999e-06 2024-04-20T17:11:17.524 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.524 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.524 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:17.524 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.179259+0000", 2024-04-20T17:11:17.524 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3015e-05 2024-04-20T17:11:17.524 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.525 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.525 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:17.525 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.179271+0000", 2024-04-20T17:11:17.525 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2232000000000001e-05 2024-04-20T17:11:17.525 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.525 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.525 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:17.525 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.179281+0000", 2024-04-20T17:11:17.525 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7780000000000002e-06 2024-04-20T17:11:17.525 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.525 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.525 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:17.525 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.181674+0000", 2024-04-20T17:11:17.526 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0023931289999999999 2024-04-20T17:11:17.526 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.526 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.526 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:17.526 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.181699+0000", 2024-04-20T17:11:17.526 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5320000000000002e-05 2024-04-20T17:11:17.526 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.526 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.526 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:17.526 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.210179+0000", 2024-04-20T17:11:17.526 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028479263000000001 2024-04-20T17:11:17.526 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.526 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.526 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:17.527 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.210227+0000", 2024-04-20T17:11:17.527 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8378999999999999e-05 2024-04-20T17:11:17.527 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.527 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:17.527 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.527 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.527 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.527 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:194458 2.8 e163/115 2:121b74d8:::benchmark_data_smithi171_78918_object33464:head v 163'12078, mlcod=163'12074)", 2024-04-20T17:11:17.527 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:29.181647+0000", 2024-04-20T17:11:17.527 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 48.313398454999998, 2024-04-20T17:11:17.527 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028612980999999999, 2024-04-20T17:11:17.527 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:17.527 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:17.527 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:17.527 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:17.528 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.528 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:17.528 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.181647+0000", 2024-04-20T17:11:17.528 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.528 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.528 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.528 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:17.528 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.181647+0000", 2024-04-20T17:11:17.528 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.528 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.528 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.528 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:17.528 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.181649+0000", 2024-04-20T17:11:17.529 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.438e-06 2024-04-20T17:11:17.529 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.529 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.529 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:17.529 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.181658+0000", 2024-04-20T17:11:17.529 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.4530000000000008e-06 2024-04-20T17:11:17.529 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.529 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.529 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:17.529 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.181667+0000", 2024-04-20T17:11:17.529 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5040000000000002e-06 2024-04-20T17:11:17.529 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.529 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.529 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:17.529 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.181674+0000", 2024-04-20T17:11:17.530 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2389999999999996e-06 2024-04-20T17:11:17.530 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.530 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.530 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:17.530 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.186299+0000", 2024-04-20T17:11:17.530 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0046246969999999997 2024-04-20T17:11:17.530 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.530 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.530 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:17.530 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.186321+0000", 2024-04-20T17:11:17.530 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2099e-05 2024-04-20T17:11:17.530 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.530 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.531 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:17.531 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.210231+0000", 2024-04-20T17:11:17.531 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.023910424999999999 2024-04-20T17:11:17.531 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.531 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.531 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:17.531 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.210260+0000", 2024-04-20T17:11:17.531 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9125999999999999e-05 2024-04-20T17:11:17.531 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.531 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:17.531 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.531 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.531 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.531 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:194465 2.8 e163/115 2:11f5df4f:::benchmark_data_smithi171_78918_object33471:head v 163'12079, mlcod=163'12075)", 2024-04-20T17:11:17.532 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:29.185641+0000", 2024-04-20T17:11:17.532 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 48.309405042999998, 2024-04-20T17:11:17.532 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024652615999999999, 2024-04-20T17:11:17.532 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:17.532 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:17.532 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:17.532 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:17.532 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.532 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:17.532 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.185641+0000", 2024-04-20T17:11:17.532 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.532 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.532 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.532 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:17.532 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.185641+0000", 2024-04-20T17:11:17.533 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.533 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.533 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.533 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:17.533 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.185643+0000", 2024-04-20T17:11:17.533 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7120000000000001e-06 2024-04-20T17:11:17.533 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.533 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.533 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:17.533 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.185651+0000", 2024-04-20T17:11:17.533 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.3720000000000005e-06 2024-04-20T17:11:17.533 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.533 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.533 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:17.533 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.185659+0000", 2024-04-20T17:11:17.534 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7319999999999998e-06 2024-04-20T17:11:17.534 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.534 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.534 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:17.534 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.185664+0000", 2024-04-20T17:11:17.534 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1429999999999997e-06 2024-04-20T17:11:17.534 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.534 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.534 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:17.534 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.190070+0000", 2024-04-20T17:11:17.534 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0044059040000000004 2024-04-20T17:11:17.534 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.534 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.534 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:17.535 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.190095+0000", 2024-04-20T17:11:17.535 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5565e-05 2024-04-20T17:11:17.535 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.535 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.535 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:17.535 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.210263+0000", 2024-04-20T17:11:17.535 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020168174000000001 2024-04-20T17:11:17.535 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.535 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.535 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:17.535 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.210293+0000", 2024-04-20T17:11:17.535 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0014000000000001e-05 2024-04-20T17:11:17.535 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.535 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:17.535 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.535 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.536 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.536 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:196524 2.d 2:bde318a3:::benchmark_data_smithi171_78918_object35530:head [delete] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-04-20T17:11:17.536 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:33.348334+0000", 2024-04-20T17:11:17.536 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 44.146711967999998, 2024-04-20T17:11:17.536 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022420002000000001, 2024-04-20T17:11:17.536 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:17.536 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:17.536 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:17.536 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:17.536 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:17.536 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:17.536 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 196524 2024-04-20T17:11:17.536 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.536 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:17.536 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.536 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:17.537 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.348334+0000", 2024-04-20T17:11:17.537 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.537 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.537 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.537 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:17.537 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.348334+0000", 2024-04-20T17:11:17.537 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.537 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.537 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.537 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:17.537 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.348336+0000", 2024-04-20T17:11:17.537 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.385e-06 2024-04-20T17:11:17.537 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.537 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.537 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:17.538 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.348343+0000", 2024-04-20T17:11:17.538 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1069999999999999e-06 2024-04-20T17:11:17.538 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.538 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.538 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:17.538 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.348358+0000", 2024-04-20T17:11:17.538 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4278e-05 2024-04-20T17:11:17.538 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.538 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.538 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:17.538 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.348364+0000", 2024-04-20T17:11:17.538 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.596e-06 2024-04-20T17:11:17.538 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.539 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.539 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:17.539 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.348422+0000", 2024-04-20T17:11:17.539 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7886000000000001e-05 2024-04-20T17:11:17.539 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.539 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.539 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:17.539 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.348552+0000", 2024-04-20T17:11:17.539 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013014400000000001 2024-04-20T17:11:17.539 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.539 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.539 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4", 2024-04-20T17:11:17.539 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.348671+0000", 2024-04-20T17:11:17.539 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011856199999999999 2024-04-20T17:11:17.540 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.540 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.540 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:17.540 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.360091+0000", 2024-04-20T17:11:17.540 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011420559 2024-04-20T17:11:17.540 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.540 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.540 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:17.540 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.370659+0000", 2024-04-20T17:11:17.540 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010567437000000001 2024-04-20T17:11:17.540 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.540 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.540 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:17.540 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.370712+0000", 2024-04-20T17:11:17.540 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2858000000000001e-05 2024-04-20T17:11:17.541 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.541 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.541 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:17.541 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.370754+0000", 2024-04-20T17:11:17.541 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2190000000000001e-05 2024-04-20T17:11:17.541 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.541 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:17.541 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.541 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.541 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.541 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:201288 2.f e166/115 2:f7c6cf78:::benchmark_data_smithi171_78918_object40294:head v 166'12596, mlcod=166'12592)", 2024-04-20T17:11:17.541 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:38.669960+0000", 2024-04-20T17:11:17.541 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 38.825085614999999, 2024-04-20T17:11:17.541 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021307833000000002, 2024-04-20T17:11:17.541 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:17.542 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:17.542 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:17.542 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:17.542 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.542 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:17.542 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.669960+0000", 2024-04-20T17:11:17.542 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.542 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.542 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.542 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:17.542 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.669960+0000", 2024-04-20T17:11:17.542 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.542 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.542 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.543 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:17.543 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.669962+0000", 2024-04-20T17:11:17.543 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.618e-06 2024-04-20T17:11:17.543 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.543 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.543 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:17.543 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.669968+0000", 2024-04-20T17:11:17.543 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6100000000000002e-06 2024-04-20T17:11:17.543 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.543 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.543 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:17.543 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.669988+0000", 2024-04-20T17:11:17.543 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9979e-05 2024-04-20T17:11:17.543 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.543 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.544 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:17.544 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.670011+0000", 2024-04-20T17:11:17.544 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2229e-05 2024-04-20T17:11:17.544 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.544 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.544 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:17.544 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.672581+0000", 2024-04-20T17:11:17.544 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0025701750000000001 2024-04-20T17:11:17.544 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.544 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.544 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:17.544 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.672600+0000", 2024-04-20T17:11:17.544 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9318999999999999e-05 2024-04-20T17:11:17.545 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.545 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.545 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:17.545 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.691218+0000", 2024-04-20T17:11:17.545 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018617788 2024-04-20T17:11:17.545 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.545 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.545 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:17.545 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.691268+0000", 2024-04-20T17:11:17.545 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0114999999999997e-05 2024-04-20T17:11:17.545 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.545 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:17.545 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.545 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.545 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.545 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:202150 2.d 2:bdc569ca:::benchmark_data_smithi171_78918_object41156:head [delete] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-04-20T17:11:17.546 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:39.692595+0000", 2024-04-20T17:11:17.546 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 37.802451128999998, 2024-04-20T17:11:17.546 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021286863999999999, 2024-04-20T17:11:17.546 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:17.546 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:17.546 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:17.546 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:17.546 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:17.546 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:17.546 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 202150 2024-04-20T17:11:17.546 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.546 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:17.546 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.547 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:17.547 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.692595+0000", 2024-04-20T17:11:17.547 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.547 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.547 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.547 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:17.547 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.692595+0000", 2024-04-20T17:11:17.547 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.547 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.547 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.547 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:17.547 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.692596+0000", 2024-04-20T17:11:17.547 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0020000000000001e-06 2024-04-20T17:11:17.547 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.547 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.548 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:17.548 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.692599+0000", 2024-04-20T17:11:17.548 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7740000000000002e-06 2024-04-20T17:11:17.548 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.548 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.548 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:17.548 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.692606+0000", 2024-04-20T17:11:17.548 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.02e-06 2024-04-20T17:11:17.548 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.548 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.548 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:17.548 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.692613+0000", 2024-04-20T17:11:17.548 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4089999999999997e-06 2024-04-20T17:11:17.548 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.548 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.549 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:17.549 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.702019+0000", 2024-04-20T17:11:17.549 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.009406312 2024-04-20T17:11:17.549 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.549 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.549 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:17.549 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.702135+0000", 2024-04-20T17:11:17.549 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011589 2024-04-20T17:11:17.549 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.549 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.549 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4", 2024-04-20T17:11:17.549 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.702261+0000", 2024-04-20T17:11:17.549 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012546899999999999 2024-04-20T17:11:17.549 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.549 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.549 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:17.549 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.710105+0000", 2024-04-20T17:11:17.550 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0078449060000000004 2024-04-20T17:11:17.550 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.550 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.550 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:17.550 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.713827+0000", 2024-04-20T17:11:17.550 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0037214430000000001 2024-04-20T17:11:17.550 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.550 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.550 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:17.550 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.713850+0000", 2024-04-20T17:11:17.550 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3223000000000001e-05 2024-04-20T17:11:17.550 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.550 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.550 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:17.550 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.713882+0000", 2024-04-20T17:11:17.550 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1415999999999999e-05 2024-04-20T17:11:17.551 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.551 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:17.551 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.551 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.551 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.552 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:202407 2.d 2:be74e115:::benchmark_data_smithi171_78918_object41413:head [delete] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-04-20T17:11:17.552 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:39.978176+0000", 2024-04-20T17:11:17.552 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 37.516869534000001, 2024-04-20T17:11:17.552 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022642580999999998, 2024-04-20T17:11:17.552 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:17.552 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:17.552 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:17.552 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:17.552 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:17.552 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:17.552 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 202407 2024-04-20T17:11:17.552 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.552 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:17.553 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.553 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:17.553 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.978176+0000", 2024-04-20T17:11:17.553 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.553 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.553 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.553 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:17.553 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.978176+0000", 2024-04-20T17:11:17.553 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.553 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.553 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.553 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:17.553 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.978179+0000", 2024-04-20T17:11:17.554 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3750000000000001e-06 2024-04-20T17:11:17.554 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.554 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.554 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:17.554 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.978185+0000", 2024-04-20T17:11:17.554 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4509999999999997e-06 2024-04-20T17:11:17.554 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.554 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.554 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:17.554 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.978198+0000", 2024-04-20T17:11:17.554 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3283e-05 2024-04-20T17:11:17.554 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.554 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.555 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:17.555 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.978216+0000", 2024-04-20T17:11:17.555 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7243000000000001e-05 2024-04-20T17:11:17.555 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.555 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.555 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:17.555 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.985378+0000", 2024-04-20T17:11:17.555 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0071626099999999998 2024-04-20T17:11:17.555 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.555 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.555 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:17.555 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.985516+0000", 2024-04-20T17:11:17.555 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000137806 2024-04-20T17:11:17.555 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.555 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.556 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4", 2024-04-20T17:11:17.556 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.985641+0000", 2024-04-20T17:11:17.556 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012463400000000001 2024-04-20T17:11:17.556 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.556 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.556 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:17.556 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.989774+0000", 2024-04-20T17:11:17.556 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0041336469999999998 2024-04-20T17:11:17.556 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.556 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.556 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:17.556 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.000709+0000", 2024-04-20T17:11:17.556 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010935034 2024-04-20T17:11:17.556 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.556 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.557 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:17.557 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.000763+0000", 2024-04-20T17:11:17.557 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3214999999999998e-05 2024-04-20T17:11:17.557 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.557 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.557 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:17.557 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.000819+0000", 2024-04-20T17:11:17.557 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6283000000000003e-05 2024-04-20T17:11:17.557 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.557 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:17.557 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.557 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.557 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.557 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:203783 2.d 2:bd77818c:::benchmark_data_smithi171_78918_object42789:head [delete] snapc 0=[] ondisk+write+known_if_redirected e167)", 2024-04-20T17:11:17.557 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:41.427785+0000", 2024-04-20T17:11:17.557 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 36.067260675999997, 2024-04-20T17:11:17.558 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.023141172000000002, 2024-04-20T17:11:17.558 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:17.558 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:17.558 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:17.558 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:17.558 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:17.558 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:17.558 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 203783 2024-04-20T17:11:17.558 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.558 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:17.558 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.558 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:17.559 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:41.427785+0000", 2024-04-20T17:11:17.559 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.559 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.559 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.559 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:17.559 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:41.427785+0000", 2024-04-20T17:11:17.559 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.559 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.559 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.559 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:17.559 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:41.427787+0000", 2024-04-20T17:11:17.559 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.905e-06 2024-04-20T17:11:17.559 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.559 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.559 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:17.560 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:41.427794+0000", 2024-04-20T17:11:17.560 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6420000000000001e-06 2024-04-20T17:11:17.560 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.560 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.560 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:17.560 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:41.427804+0000", 2024-04-20T17:11:17.560 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0242e-05 2024-04-20T17:11:17.560 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.560 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.560 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:17.560 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:41.427811+0000", 2024-04-20T17:11:17.560 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2690000000000002e-06 2024-04-20T17:11:17.560 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.560 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.560 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:17.561 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:41.427892+0000", 2024-04-20T17:11:17.561 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0885999999999998e-05 2024-04-20T17:11:17.561 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.561 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.561 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:17.561 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:41.428026+0000", 2024-04-20T17:11:17.561 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000133437 2024-04-20T17:11:17.561 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.561 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.561 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4", 2024-04-20T17:11:17.561 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:41.428147+0000", 2024-04-20T17:11:17.561 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012188000000000001 2024-04-20T17:11:17.562 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.562 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.562 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:17.562 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:41.441159+0000", 2024-04-20T17:11:17.562 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013011317 2024-04-20T17:11:17.562 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.562 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.562 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:17.562 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:41.450831+0000", 2024-04-20T17:11:17.562 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.009672043 2024-04-20T17:11:17.562 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.562 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.562 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:17.562 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:41.450885+0000", 2024-04-20T17:11:17.563 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4460999999999999e-05 2024-04-20T17:11:17.563 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.563 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.563 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:17.563 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:41.450926+0000", 2024-04-20T17:11:17.563 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1090000000000001e-05 2024-04-20T17:11:17.563 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.563 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:17.563 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.563 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.563 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.563 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:203786 2.d 2:bdbb436c:::benchmark_data_smithi171_78918_object42792:head [delete] snapc 0=[] ondisk+write+known_if_redirected e167)", 2024-04-20T17:11:17.563 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:41.427957+0000", 2024-04-20T17:11:17.563 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 36.067089314999997, 2024-04-20T17:11:17.563 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.023114319000000001, 2024-04-20T17:11:17.564 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:17.564 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:17.564 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:17.564 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:17.564 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:17.564 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:17.564 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 203786 2024-04-20T17:11:17.564 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.564 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:17.564 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.564 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:17.564 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:41.427957+0000", 2024-04-20T17:11:17.564 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.564 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.565 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.565 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:17.565 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:41.427957+0000", 2024-04-20T17:11:17.565 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.565 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.565 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.565 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:17.565 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:41.427958+0000", 2024-04-20T17:11:17.565 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.491e-06 2024-04-20T17:11:17.565 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.565 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.565 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:17.565 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:41.427963+0000", 2024-04-20T17:11:17.565 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3850000000000001e-06 2024-04-20T17:11:17.565 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.566 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.566 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:17.566 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:41.427974+0000", 2024-04-20T17:11:17.566 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0128e-05 2024-04-20T17:11:17.566 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.566 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.566 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:17.566 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:41.427980+0000", 2024-04-20T17:11:17.566 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9149999999999999e-06 2024-04-20T17:11:17.566 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.566 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.566 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:17.566 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:41.431884+0000", 2024-04-20T17:11:17.566 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0039031830000000002 2024-04-20T17:11:17.567 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.567 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.567 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:17.567 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:41.432020+0000", 2024-04-20T17:11:17.567 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013676699999999999 2024-04-20T17:11:17.568 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.568 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.568 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4", 2024-04-20T17:11:17.568 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:41.432529+0000", 2024-04-20T17:11:17.568 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00050814700000000003 2024-04-20T17:11:17.568 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.568 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.568 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:17.568 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:41.450977+0000", 2024-04-20T17:11:17.568 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018448084999999999 2024-04-20T17:11:17.568 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.568 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.568 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:17.568 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:41.450988+0000", 2024-04-20T17:11:17.569 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1698e-05 2024-04-20T17:11:17.569 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.569 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.569 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:17.569 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:41.451033+0000", 2024-04-20T17:11:17.569 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4270999999999997e-05 2024-04-20T17:11:17.569 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.569 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.569 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:17.569 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:41.451071+0000", 2024-04-20T17:11:17.569 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8248999999999999e-05 2024-04-20T17:11:17.569 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.569 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:17.569 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.570 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.570 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.570 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:209399 2.d 2:bf37fbbc:::benchmark_data_smithi171_78918_object48405:head [delete] snapc 0=[] ondisk+write+known_if_redirected e167)", 2024-04-20T17:11:17.570 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:47.569377+0000", 2024-04-20T17:11:17.570 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 29.925668914999999, 2024-04-20T17:11:17.570 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022517832000000002, 2024-04-20T17:11:17.570 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:17.570 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:17.570 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:17.570 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:17.570 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:17.570 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:17.570 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 209399 2024-04-20T17:11:17.570 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.571 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:17.571 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.571 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:17.571 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.569377+0000", 2024-04-20T17:11:17.571 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.571 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.571 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.571 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:17.571 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.569377+0000", 2024-04-20T17:11:17.571 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.571 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.571 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.571 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:17.571 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.569379+0000", 2024-04-20T17:11:17.572 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3740000000000001e-06 2024-04-20T17:11:17.572 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.572 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.572 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:17.572 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.569386+0000", 2024-04-20T17:11:17.572 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0949999999999998e-06 2024-04-20T17:11:17.572 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.572 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.572 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:17.572 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.569401+0000", 2024-04-20T17:11:17.572 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4406999999999999e-05 2024-04-20T17:11:17.572 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.572 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.572 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:17.573 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.569410+0000", 2024-04-20T17:11:17.573 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0680000000000003e-06 2024-04-20T17:11:17.573 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.573 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.573 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:17.573 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.569472+0000", 2024-04-20T17:11:17.573 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1812999999999999e-05 2024-04-20T17:11:17.573 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.573 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.573 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:17.573 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.569603+0000", 2024-04-20T17:11:17.573 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013110500000000001 2024-04-20T17:11:17.573 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.573 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.573 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4", 2024-04-20T17:11:17.574 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.569689+0000", 2024-04-20T17:11:17.574 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6501000000000004e-05 2024-04-20T17:11:17.574 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.574 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.574 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:17.574 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.583179+0000", 2024-04-20T17:11:17.574 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013489707 2024-04-20T17:11:17.574 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.574 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.574 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:17.574 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.591788+0000", 2024-04-20T17:11:17.574 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0086088620000000001 2024-04-20T17:11:17.574 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.574 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.575 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:17.575 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.591847+0000", 2024-04-20T17:11:17.575 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9222000000000003e-05 2024-04-20T17:11:17.575 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.575 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.575 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:17.575 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.591895+0000", 2024-04-20T17:11:17.575 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7678e-05 2024-04-20T17:11:17.575 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.575 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:17.575 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.575 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.575 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.575 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:209400 2.d 2:bc25b4ca:::benchmark_data_smithi171_78918_object48406:head [delete] snapc 0=[] ondisk+write+known_if_redirected e167)", 2024-04-20T17:11:17.576 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:47.569446+0000", 2024-04-20T17:11:17.576 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 29.925599568999999, 2024-04-20T17:11:17.576 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022593223999999999, 2024-04-20T17:11:17.576 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:17.576 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:17.576 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:17.576 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:17.576 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:17.576 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:17.576 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 209400 2024-04-20T17:11:17.576 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.576 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:17.576 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.576 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:17.576 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.569446+0000", 2024-04-20T17:11:17.577 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.577 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.577 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.577 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:17.577 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.569446+0000", 2024-04-20T17:11:17.577 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.577 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.577 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.577 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:17.577 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.569447+0000", 2024-04-20T17:11:17.577 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.139e-06 2024-04-20T17:11:17.577 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.577 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.577 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:17.577 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.569451+0000", 2024-04-20T17:11:17.578 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.946e-06 2024-04-20T17:11:17.578 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.578 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.578 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:17.578 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.569458+0000", 2024-04-20T17:11:17.578 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7599999999999997e-06 2024-04-20T17:11:17.578 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.578 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.578 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:17.578 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.569464+0000", 2024-04-20T17:11:17.578 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.037e-06 2024-04-20T17:11:17.578 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.578 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.579 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:17.579 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.573545+0000", 2024-04-20T17:11:17.579 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0040810949999999999 2024-04-20T17:11:17.579 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.579 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.579 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:17.579 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.573677+0000", 2024-04-20T17:11:17.579 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013182100000000001 2024-04-20T17:11:17.579 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.579 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.579 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4", 2024-04-20T17:11:17.579 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.573798+0000", 2024-04-20T17:11:17.579 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000120827 2024-04-20T17:11:17.579 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.580 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.580 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:17.580 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.583222+0000", 2024-04-20T17:11:17.580 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0094237769999999995 2024-04-20T17:11:17.580 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.580 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.580 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:17.580 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.591959+0000", 2024-04-20T17:11:17.580 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0087374150000000001 2024-04-20T17:11:17.580 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.580 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.580 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:17.580 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.592001+0000", 2024-04-20T17:11:17.580 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2194000000000003e-05 2024-04-20T17:11:17.580 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.580 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.581 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:17.581 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.592039+0000", 2024-04-20T17:11:17.581 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8213000000000002e-05 2024-04-20T17:11:17.581 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.581 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:17.581 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.581 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.581 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.581 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:209410 2.d 2:b43a5a5a:::benchmark_data_smithi171_78918_object48416:head [delete] snapc 0=[] ondisk+write+known_if_redirected e167)", 2024-04-20T17:11:17.581 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:47.574804+0000", 2024-04-20T17:11:17.581 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 29.920242019, 2024-04-20T17:11:17.581 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022457639000000001, 2024-04-20T17:11:17.581 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:17.581 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:17.582 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:17.582 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:17.582 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:17.582 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:17.582 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 209410 2024-04-20T17:11:17.582 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.582 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:17.582 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.582 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:17.582 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.574804+0000", 2024-04-20T17:11:17.582 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.582 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.582 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.582 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:17.583 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.574804+0000", 2024-04-20T17:11:17.583 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.583 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.583 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.583 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:17.583 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.574805+0000", 2024-04-20T17:11:17.583 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.64e-07 2024-04-20T17:11:17.583 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.583 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.583 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:17.583 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.574808+0000", 2024-04-20T17:11:17.583 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5370000000000002e-06 2024-04-20T17:11:17.583 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.584 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.584 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:17.584 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.574816+0000", 2024-04-20T17:11:17.584 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4259999999999997e-06 2024-04-20T17:11:17.584 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.584 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.584 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:17.584 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.574823+0000", 2024-04-20T17:11:17.584 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8940000000000004e-06 2024-04-20T17:11:17.584 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.584 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.584 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:17.584 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.587528+0000", 2024-04-20T17:11:17.585 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.012705591 2024-04-20T17:11:17.585 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.585 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.585 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:17.585 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.587616+0000", 2024-04-20T17:11:17.585 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8123999999999999e-05 2024-04-20T17:11:17.585 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.585 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.585 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4", 2024-04-20T17:11:17.585 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.587691+0000", 2024-04-20T17:11:17.585 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4999999999999993e-05 2024-04-20T17:11:17.585 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.585 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.585 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:17.586 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.597138+0000", 2024-04-20T17:11:17.586 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0094465450000000006 2024-04-20T17:11:17.586 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.586 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.586 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:17.586 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.597158+0000", 2024-04-20T17:11:17.586 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0023000000000001e-05 2024-04-20T17:11:17.586 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.586 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.586 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:17.586 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.597216+0000", 2024-04-20T17:11:17.586 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8082999999999999e-05 2024-04-20T17:11:17.586 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.586 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.587 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:17.587 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:47.597261+0000", 2024-04-20T17:11:17.587 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5451999999999998e-05 2024-04-20T17:11:17.587 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.587 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:17.587 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.587 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.587 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.587 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:209959 2.f e168/115 2:f774094c:::benchmark_data_smithi171_78918_object48965:head v 168'13137, mlcod=168'13134)", 2024-04-20T17:11:17.587 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:48.205229+0000", 2024-04-20T17:11:17.587 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 29.289816665, 2024-04-20T17:11:17.587 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022188847000000001, 2024-04-20T17:11:17.587 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:17.588 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:17.588 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:17.588 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:17.588 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.588 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:17.588 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.205229+0000", 2024-04-20T17:11:17.588 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.588 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.588 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.588 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:17.588 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.205229+0000", 2024-04-20T17:11:17.588 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.588 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.588 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.589 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:17.589 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.205231+0000", 2024-04-20T17:11:17.589 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0099999999999998e-06 2024-04-20T17:11:17.589 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.589 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.589 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:17.589 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.205240+0000", 2024-04-20T17:11:17.589 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0510000000000002e-06 2024-04-20T17:11:17.589 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.589 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.589 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:17.590 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.205251+0000", 2024-04-20T17:11:17.590 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0468e-05 2024-04-20T17:11:17.590 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.590 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.590 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:17.590 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.205257+0000", 2024-04-20T17:11:17.590 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6270000000000002e-06 2024-04-20T17:11:17.590 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.590 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.590 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:17.590 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.205308+0000", 2024-04-20T17:11:17.590 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0288999999999999e-05 2024-04-20T17:11:17.590 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.591 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.591 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:17.591 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.205330+0000", 2024-04-20T17:11:17.591 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2087e-05 2024-04-20T17:11:17.591 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.591 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.591 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:17.591 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.227371+0000", 2024-04-20T17:11:17.591 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022040904 2024-04-20T17:11:17.591 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.591 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.591 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:17.591 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.227418+0000", 2024-04-20T17:11:17.591 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7410999999999997e-05 2024-04-20T17:11:17.592 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.592 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:17.592 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.592 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.592 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.592 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:209961 2.f e168/115 2:f3563409:::benchmark_data_smithi171_78918_object48967:head v 168'13138, mlcod=168'13134)", 2024-04-20T17:11:17.592 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:48.205277+0000", 2024-04-20T17:11:17.592 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 29.289769018000001, 2024-04-20T17:11:17.592 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022168667, 2024-04-20T17:11:17.592 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:17.592 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:17.592 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:17.592 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:17.592 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.592 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:17.593 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.205277+0000", 2024-04-20T17:11:17.593 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.593 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.593 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.593 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:17.593 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.205277+0000", 2024-04-20T17:11:17.593 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.593 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.593 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.593 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:17.593 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.205277+0000", 2024-04-20T17:11:17.593 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3799999999999997e-07 2024-04-20T17:11:17.593 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.593 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.593 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:17.594 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.205282+0000", 2024-04-20T17:11:17.594 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9439999999999996e-06 2024-04-20T17:11:17.594 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.594 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.594 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:17.594 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.205288+0000", 2024-04-20T17:11:17.594 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8640000000000001e-06 2024-04-20T17:11:17.594 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.594 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.594 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:17.594 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.205294+0000", 2024-04-20T17:11:17.594 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3369999999999999e-06 2024-04-20T17:11:17.594 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.595 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.595 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:17.595 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.209308+0000", 2024-04-20T17:11:17.595 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0040149449999999998 2024-04-20T17:11:17.595 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.595 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.595 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:17.595 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.209328+0000", 2024-04-20T17:11:17.595 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9809999999999998e-05 2024-04-20T17:11:17.595 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.595 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.595 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:17.595 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.227421+0000", 2024-04-20T17:11:17.595 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018093001000000001 2024-04-20T17:11:17.596 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.596 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.596 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:17.596 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.227445+0000", 2024-04-20T17:11:17.596 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4227999999999999e-05 2024-04-20T17:11:17.596 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.596 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:17.596 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.596 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.596 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.596 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:210034 2.f e168/115 2:f5070602:::benchmark_data_smithi171_78918_object49040:head v 168'13143, mlcod=168'13141)", 2024-04-20T17:11:17.596 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:48.267198+0000", 2024-04-20T17:11:17.596 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 29.227848293000001, 2024-04-20T17:11:17.596 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022806882000000001, 2024-04-20T17:11:17.596 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:17.597 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:17.597 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:17.597 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:17.597 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.597 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:17.597 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.267198+0000", 2024-04-20T17:11:17.597 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.597 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.597 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.597 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:17.597 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.267198+0000", 2024-04-20T17:11:17.597 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.597 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.597 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.597 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:17.598 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.267200+0000", 2024-04-20T17:11:17.598 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7819999999999998e-06 2024-04-20T17:11:17.598 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.598 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.598 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:17.598 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.267220+0000", 2024-04-20T17:11:17.598 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9632000000000001e-05 2024-04-20T17:11:17.598 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.598 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.598 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:17.598 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.267239+0000", 2024-04-20T17:11:17.598 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8977000000000001e-05 2024-04-20T17:11:17.598 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.598 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.598 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:17.599 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.267248+0000", 2024-04-20T17:11:17.599 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0969999999999995e-06 2024-04-20T17:11:17.599 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.599 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.599 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:17.599 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.267303+0000", 2024-04-20T17:11:17.599 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5135e-05 2024-04-20T17:11:17.599 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.599 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.599 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:17.599 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.267327+0000", 2024-04-20T17:11:17.599 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3533999999999999e-05 2024-04-20T17:11:17.599 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.600 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.600 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:17.600 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.289959+0000", 2024-04-20T17:11:17.600 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022632106999999999 2024-04-20T17:11:17.600 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.600 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.600 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:17.600 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.290004+0000", 2024-04-20T17:11:17.600 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5618000000000003e-05 2024-04-20T17:11:17.600 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.600 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:17.600 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.600 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.600 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.600 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:210465 2.d 2:bf240863:::benchmark_data_smithi171_78918_object49471:head [delete] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-04-20T17:11:17.601 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:48.744870+0000", 2024-04-20T17:11:17.601 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 28.750175471999999, 2024-04-20T17:11:17.601 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.023416211999999999, 2024-04-20T17:11:17.601 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:17.601 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:17.601 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:17.601 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:17.601 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:17.601 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:17.601 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 210465 2024-04-20T17:11:17.601 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.601 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:17.601 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.601 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:17.601 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.744870+0000", 2024-04-20T17:11:17.602 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.602 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.602 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.602 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:17.602 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.744870+0000", 2024-04-20T17:11:17.602 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:17.602 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.602 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.602 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:17.602 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.744873+0000", 2024-04-20T17:11:17.602 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3039999999999999e-06 2024-04-20T17:11:17.602 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.602 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.603 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:17.603 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.744879+0000", 2024-04-20T17:11:17.603 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0349999999999998e-06 2024-04-20T17:11:17.603 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.603 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.603 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:17.603 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.744891+0000", 2024-04-20T17:11:17.603 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2791e-05 2024-04-20T17:11:17.603 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.603 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.603 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:17.603 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.744900+0000", 2024-04-20T17:11:17.603 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5089999999999993e-06 2024-04-20T17:11:17.603 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.604 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.604 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:17.604 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.745097+0000", 2024-04-20T17:11:17.604 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00019676900000000001 2024-04-20T17:11:17.604 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.604 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.604 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:17.604 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.745191+0000", 2024-04-20T17:11:17.604 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.4539999999999999e-05 2024-04-20T17:11:17.604 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.604 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.604 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4", 2024-04-20T17:11:17.604 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.745308+0000", 2024-04-20T17:11:17.604 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000116729 2024-04-20T17:11:17.604 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.605 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.605 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:17.605 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.753191+0000", 2024-04-20T17:11:17.605 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0078833219999999999 2024-04-20T17:11:17.605 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.605 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.605 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:17.605 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.768180+0000", 2024-04-20T17:11:17.605 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.01498822 2024-04-20T17:11:17.605 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.605 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.605 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:17.605 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.768240+0000", 2024-04-20T17:11:17.605 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.033e-05 2024-04-20T17:11:17.605 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:17.605 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:17.606 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:17.606 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:48.768287+0000", 2024-04-20T17:11:17.606 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6662999999999997e-05 2024-04-20T17:11:17.606 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.606 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:17.606 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.606 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:17.606 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:17.606 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:11:17.607 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:17 smithi106 ceph-mon[44423]: pgmap v523: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19866/218982 objects degraded (9.072%); 6123/218982 objects misplaced (2.796%); 729 KiB/s, 11 objects/s recovering 2024-04-20T17:11:17.607 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:17 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/922834618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:11:17.607 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:17.607 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:17.607 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:17.607 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:17.607 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:17.607 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:17.608 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:17.635 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:17.758 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:11:17.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:17.760 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:11:17.861 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:11:17.890 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:18.266 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:11:18.267 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:11:18.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:18.375 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:18.376 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:18.376 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:18.376 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:18.407 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-20T17:11:18.438 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:11:18.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:18.440 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:11:18.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:18.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:18.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:18.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:18.541 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:11:18.573 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:18.593 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:11:18.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:18.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:11:18.696 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:11:18.697 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:11:18.727 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:18.751 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:11:18.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:18.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:11:18.855 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:11:18.881 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:18.978 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:11:18.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:18.981 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:11:19.082 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:11:19.115 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:19.137 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:11:19.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:19.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:11:19.241 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:11:19.245 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:19 smithi171 ceph-mon[44189]: pgmap v524: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19866/218982 objects degraded (9.072%); 6123/218982 objects misplaced (2.796%); 729 KiB/s, 11 objects/s recovering 2024-04-20T17:11:19.269 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:19.281 INFO:teuthology.orchestra.run.smithi005.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:11:19.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:19 smithi005 ceph-mon[39562]: pgmap v524: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19866/218982 objects degraded (9.072%); 6123/218982 objects misplaced (2.796%); 729 KiB/s, 11 objects/s recovering 2024-04-20T17:11:19.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:19 smithi106 ceph-mon[44423]: pgmap v524: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19866/218982 objects degraded (9.072%); 6123/218982 objects misplaced (2.796%); 729 KiB/s, 11 objects/s recovering 2024-04-20T17:11:20.119 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:11:20.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:20.123 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:11:20.224 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:11:20.251 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:20.268 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:11:20.341 INFO:teuthology.orchestra.run.smithi005.stdout:get_pg_num_history pg_num_history pg_num_history(e169 pg_nums {1={14=1},2={115=16}} deleted_pools ) 2024-04-20T17:11:20.342 INFO:teuthology.orchestra.run.smithi005.stdout:Importing pgid 2.e 2024-04-20T17:11:20.358 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7000bc6a:::benchmark_data_smithi171_78918_object82634:head# 2024-04-20T17:11:20.359 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.359 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70015d79:::benchmark_data_smithi171_78918_object155850:head# 2024-04-20T17:11:20.359 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.359 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:700197b5:::benchmark_data_smithi171_78918_object127114:head# 2024-04-20T17:11:20.360 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.360 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7001b431:::benchmark_data_smithi171_78918_object107278:head# 2024-04-20T17:11:20.360 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.360 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7001ddbd:::benchmark_data_smithi171_78918_object129901:head# 2024-04-20T17:11:20.360 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.360 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7002cc2d:::benchmark_data_smithi171_78918_object117882:head# 2024-04-20T17:11:20.360 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.362 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70034c34:::benchmark_data_smithi171_78918_object87278:head# 2024-04-20T17:11:20.362 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.362 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70044159:::benchmark_data_smithi171_78918_object103640:head# 2024-04-20T17:11:20.362 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.362 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7004a58e:::benchmark_data_smithi171_78918_object103906:head# 2024-04-20T17:11:20.362 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.362 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:700586a8:::benchmark_data_smithi171_78918_object98955:head# 2024-04-20T17:11:20.362 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.362 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7007365f:::benchmark_data_smithi171_78918_object74153:head# 2024-04-20T17:11:20.362 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.362 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7007d04f:::benchmark_data_smithi171_78918_object143991:head# 2024-04-20T17:11:20.363 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.364 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:700907eb:::benchmark_data_smithi171_78918_object98383:head# 2024-04-20T17:11:20.364 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.364 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70096acc:::benchmark_data_smithi171_78918_object136914:head# 2024-04-20T17:11:20.364 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.364 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70097321:::benchmark_data_smithi171_78918_object150953:head# 2024-04-20T17:11:20.364 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.365 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70097406:::benchmark_data_smithi171_78918_object122216:head# 2024-04-20T17:11:20.365 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.365 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:700a359d:::benchmark_data_smithi171_78918_object133687:head# 2024-04-20T17:11:20.365 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.365 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:700a541f:::benchmark_data_smithi171_78918_object64253:head# 2024-04-20T17:11:20.365 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.365 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:700b7861:::benchmark_data_smithi171_78918_object61952:head# 2024-04-20T17:11:20.365 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.365 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:700be0c2:::benchmark_data_smithi171_78918_object63282:head# 2024-04-20T17:11:20.365 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.365 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:700c9327:::benchmark_data_smithi171_78918_object154704:head# 2024-04-20T17:11:20.365 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.365 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:700cb37d:::benchmark_data_smithi171_78918_object88807:head# 2024-04-20T17:11:20.365 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.367 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:700dc333:::benchmark_data_smithi171_78918_object81185:head# 2024-04-20T17:11:20.367 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.367 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:700e9018:::benchmark_data_smithi171_78918_object77225:head# 2024-04-20T17:11:20.367 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.367 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:700f2a35:::benchmark_data_smithi171_78918_object145504:head# 2024-04-20T17:11:20.367 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.368 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70103e23:::benchmark_data_smithi171_78918_object113823:head# 2024-04-20T17:11:20.368 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.368 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7010deb6:::benchmark_data_smithi171_78918_object118449:head# 2024-04-20T17:11:20.368 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.368 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:701168d0:::benchmark_data_smithi171_78918_object113466:head# 2024-04-20T17:11:20.368 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.368 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70127c53:::benchmark_data_smithi171_78918_object116578:head# 2024-04-20T17:11:20.368 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.368 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7012c51c:::benchmark_data_smithi171_78918_object152990:head# 2024-04-20T17:11:20.368 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.368 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70141c23:::benchmark_data_smithi171_78918_object95591:head# 2024-04-20T17:11:20.368 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.368 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70144fa2:::benchmark_data_smithi171_78918_object133363:head# 2024-04-20T17:11:20.368 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.369 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7014e6fa:::benchmark_data_smithi171_78918_object94911:head# 2024-04-20T17:11:20.369 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.369 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7015d971:::benchmark_data_smithi171_78918_object56344:head# 2024-04-20T17:11:20.369 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.369 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7015de0f:::benchmark_data_smithi171_78918_object115017:head# 2024-04-20T17:11:20.369 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.370 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7015e9d0:::benchmark_data_smithi171_78918_object85827:head# 2024-04-20T17:11:20.371 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.371 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7015ed40:::benchmark_data_smithi171_78918_object156013:head# 2024-04-20T17:11:20.371 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.371 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70164f4b:::benchmark_data_smithi171_78918_object69039:head# 2024-04-20T17:11:20.371 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.371 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:701678ed:::benchmark_data_smithi171_78918_object79234:head# 2024-04-20T17:11:20.371 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.371 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7016b541:::benchmark_data_smithi171_78918_object145831:head# 2024-04-20T17:11:20.371 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.371 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7016e3c6:::benchmark_data_smithi171_78918_object153224:head# 2024-04-20T17:11:20.371 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.371 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70176526:::benchmark_data_smithi171_78918_object83229:head# 2024-04-20T17:11:20.371 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.372 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70178ebc:::benchmark_data_smithi171_78918_object58235:head# 2024-04-20T17:11:20.372 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.372 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7017accd:::benchmark_data_smithi171_78918_object143934:head# 2024-04-20T17:11:20.372 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.372 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7017e8b2:::benchmark_data_smithi171_78918_object75407:head# 2024-04-20T17:11:20.372 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.372 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70184a55:::benchmark_data_smithi171_78918_object83752:head# 2024-04-20T17:11:20.372 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.372 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7018777e:::benchmark_data_smithi171_78918_object143906:head# 2024-04-20T17:11:20.372 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.372 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70188004:::benchmark_data_smithi171_78918_object111170:head# 2024-04-20T17:11:20.372 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.375 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7018c049:::benchmark_data_smithi171_78918_object131212:head# 2024-04-20T17:11:20.377 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.377 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7019dc0c:::benchmark_data_smithi171_78918_object128848:head# 2024-04-20T17:11:20.377 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.377 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:701ad703:::benchmark_data_smithi171_78918_object141311:head# 2024-04-20T17:11:20.377 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.377 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:701b6502:::benchmark_data_smithi171_78918_object100549:head# 2024-04-20T17:11:20.377 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.377 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:701bda2c:::benchmark_data_smithi171_78918_object151968:head# 2024-04-20T17:11:20.377 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.377 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:701c1edf:::benchmark_data_smithi171_78918_object116371:head# 2024-04-20T17:11:20.378 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.378 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:701cc349:::benchmark_data_smithi171_78918_object147087:head# 2024-04-20T17:11:20.378 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.378 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:701d0ecd:::benchmark_data_smithi171_78918_object160904:head# 2024-04-20T17:11:20.378 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.378 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:701dce6d:::benchmark_data_smithi171_78918_object107752:head# 2024-04-20T17:11:20.378 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.378 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:701e15ca:::benchmark_data_smithi171_78918_object54877:head# 2024-04-20T17:11:20.378 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.378 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:701e75a8:::benchmark_data_smithi171_78918_object108083:head# 2024-04-20T17:11:20.378 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.378 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:701ead85:::benchmark_data_smithi171_78918_object108260:head# 2024-04-20T17:11:20.378 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.378 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:701eb6c3:::benchmark_data_smithi171_78918_object101269:head# 2024-04-20T17:11:20.379 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.379 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:701f1472:::benchmark_data_smithi171_78918_object72778:head# 2024-04-20T17:11:20.379 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.379 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:701fbbd1:::benchmark_data_smithi171_78918_object95833:head# 2024-04-20T17:11:20.379 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.379 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70201de0:::benchmark_data_smithi171_78918_object126816:head# 2024-04-20T17:11:20.379 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.379 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70202874:::benchmark_data_smithi171_78918_object93496:head# 2024-04-20T17:11:20.379 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.379 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7021adf0:::benchmark_data_smithi171_78918_object146245:head# 2024-04-20T17:11:20.379 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.379 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7022d2ee:::benchmark_data_smithi171_78918_object151189:head# 2024-04-20T17:11:20.379 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.379 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70243ae5:::benchmark_data_smithi171_78918_object57938:head# 2024-04-20T17:11:20.379 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.380 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70243bcb:::benchmark_data_smithi171_78918_object108353:head# 2024-04-20T17:11:20.380 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.387 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7024ea64:::benchmark_data_smithi171_78918_object109789:head# 2024-04-20T17:11:20.387 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.387 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70251e13:::benchmark_data_smithi171_78918_object132866:head# 2024-04-20T17:11:20.387 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.388 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7025514b:::benchmark_data_smithi171_78918_object72346:head# 2024-04-20T17:11:20.388 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.388 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:702590a7:::benchmark_data_smithi171_78918_object149833:head# 2024-04-20T17:11:20.388 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.388 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70259bc2:::benchmark_data_smithi171_78918_object86226:head# 2024-04-20T17:11:20.388 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.388 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7025eabb:::benchmark_data_smithi171_78918_object51877:head# 2024-04-20T17:11:20.388 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.388 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:702622c1:::benchmark_data_smithi171_78918_object57456:head# 2024-04-20T17:11:20.388 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.388 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70271589:::benchmark_data_smithi171_78918_object83554:head# 2024-04-20T17:11:20.388 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.388 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7027b65f:::benchmark_data_smithi171_78918_object59313:head# 2024-04-20T17:11:20.388 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.388 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7029849a:::benchmark_data_smithi171_78918_object113889:head# 2024-04-20T17:11:20.389 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.389 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7029e801:::benchmark_data_smithi171_78918_object86767:head# 2024-04-20T17:11:20.389 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.389 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:702a27a6:::benchmark_data_smithi171_78918_object66426:head# 2024-04-20T17:11:20.389 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.389 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:702a7ec9:::benchmark_data_smithi171_78918_object120265:head# 2024-04-20T17:11:20.389 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.389 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:702b2705:::benchmark_data_smithi171_78918_object99547:head# 2024-04-20T17:11:20.389 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.389 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:702bcd58:::benchmark_data_smithi171_78918_object70152:head# 2024-04-20T17:11:20.389 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.389 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:702c3a0b:::benchmark_data_smithi171_78918_object55483:head# 2024-04-20T17:11:20.389 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.390 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:702c60b8:::benchmark_data_smithi171_78918_object79336:head# 2024-04-20T17:11:20.390 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.390 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:702cb0e9:::benchmark_data_smithi171_78918_object56201:head# 2024-04-20T17:11:20.390 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.390 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:702cbb41:::benchmark_data_smithi171_78918_object56889:head# 2024-04-20T17:11:20.390 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.390 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:702d1976:::benchmark_data_smithi171_78918_object104110:head# 2024-04-20T17:11:20.390 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.390 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:702d38ac:::benchmark_data_smithi171_78918_object66528:head# 2024-04-20T17:11:20.390 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.390 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:702ddb0a:::benchmark_data_smithi171_78918_object139351:head# 2024-04-20T17:11:20.390 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.390 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:702e51fa:::benchmark_data_smithi171_78918_object143550:head# 2024-04-20T17:11:20.390 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.391 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:702f5978:::benchmark_data_smithi171_78918_object63847:head# 2024-04-20T17:11:20.391 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.391 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:702fb0c1:::benchmark_data_smithi171_78918_object94314:head# 2024-04-20T17:11:20.391 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.391 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7030205d:::benchmark_data_smithi171_78918_object133681:head# 2024-04-20T17:11:20.391 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.391 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7031e6dd:::benchmark_data_smithi171_78918_object115758:head# 2024-04-20T17:11:20.391 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.391 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70339456:::benchmark_data_smithi171_78918_object84572:head# 2024-04-20T17:11:20.391 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.391 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7033e4bb:::benchmark_data_smithi171_78918_object64579:head# 2024-04-20T17:11:20.391 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.391 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:703437b4:::benchmark_data_smithi171_78918_object78631:head# 2024-04-20T17:11:20.391 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.391 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:703448ad:::benchmark_data_smithi171_78918_object65007:head# 2024-04-20T17:11:20.392 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.392 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7034a433:::benchmark_data_smithi171_78918_object99108:head# 2024-04-20T17:11:20.392 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.392 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7034c015:::benchmark_data_smithi171_78918_object135596:head# 2024-04-20T17:11:20.392 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.392 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7034fcae:::benchmark_data_smithi171_78918_object63846:head# 2024-04-20T17:11:20.392 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.392 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:703538a5:::benchmark_data_smithi171_78918_object101381:head# 2024-04-20T17:11:20.392 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.392 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70355ec8:::benchmark_data_smithi171_78918_object153673:head# 2024-04-20T17:11:20.392 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.392 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70365ddd:::benchmark_data_smithi171_78918_object78146:head# 2024-04-20T17:11:20.392 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.393 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7037091b:::benchmark_data_smithi171_78918_object107008:head# 2024-04-20T17:11:20.393 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.393 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70377a1d:::benchmark_data_smithi171_78918_object156036:head# 2024-04-20T17:11:20.393 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.393 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:703897fb:::benchmark_data_smithi171_78918_object128829:head# 2024-04-20T17:11:20.393 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.393 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70393cb8:::benchmark_data_smithi171_78918_object99788:head# 2024-04-20T17:11:20.393 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.393 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:703960a1:::benchmark_data_smithi171_78918_object111841:head# 2024-04-20T17:11:20.393 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.393 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:703b7fc7:::benchmark_data_smithi171_78918_object113696:head# 2024-04-20T17:11:20.393 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.393 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:703ece31:::benchmark_data_smithi171_78918_object63307:head# 2024-04-20T17:11:20.393 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.393 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7040b3e0:::benchmark_data_smithi171_78918_object73292:head# 2024-04-20T17:11:20.394 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.394 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7040c563:::benchmark_data_smithi171_78918_object112545:head# 2024-04-20T17:11:20.394 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.394 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7041c63c:::benchmark_data_smithi171_78918_object125434:head# 2024-04-20T17:11:20.394 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.403 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7042c7c9:::benchmark_data_smithi171_78918_object107473:head# 2024-04-20T17:11:20.403 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.403 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70433bcc:::benchmark_data_smithi171_78918_object76023:head# 2024-04-20T17:11:20.403 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.403 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70435b7b:::benchmark_data_smithi171_78918_object116135:head# 2024-04-20T17:11:20.403 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.403 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7043b50a:::benchmark_data_smithi171_78918_object120853:head# 2024-04-20T17:11:20.403 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.403 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7043d748:::benchmark_data_smithi171_78918_object75455:head# 2024-04-20T17:11:20.403 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.403 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70440aa9:::benchmark_data_smithi171_78918_object155556:head# 2024-04-20T17:11:20.404 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.404 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7046b9d3:::benchmark_data_smithi171_78918_object96248:head# 2024-04-20T17:11:20.404 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.404 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7046c15b:::benchmark_data_smithi171_78918_object99986:head# 2024-04-20T17:11:20.404 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.404 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:704756d5:::benchmark_data_smithi171_78918_object68986:head# 2024-04-20T17:11:20.404 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.404 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7047e5fd:::benchmark_data_smithi171_78918_object84447:head# 2024-04-20T17:11:20.404 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.404 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:704cbe9d:::benchmark_data_smithi171_78918_object105597:head# 2024-04-20T17:11:20.404 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.404 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:704f9344:::benchmark_data_smithi171_78918_object124264:head# 2024-04-20T17:11:20.404 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.404 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7050acee:::benchmark_data_smithi171_78918_object124986:head# 2024-04-20T17:11:20.405 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.405 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7050cdc2:::benchmark_data_smithi171_78918_object147847:head# 2024-04-20T17:11:20.405 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.405 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7050dd39:::benchmark_data_smithi171_78918_object132730:head# 2024-04-20T17:11:20.405 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.405 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7051283f:::benchmark_data_smithi171_78918_object101858:head# 2024-04-20T17:11:20.405 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.405 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7051ffc6:::benchmark_data_smithi171_78918_object94343:head# 2024-04-20T17:11:20.405 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.405 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70526b62:::benchmark_data_smithi171_78918_object108263:head# 2024-04-20T17:11:20.405 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.405 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70531a24:::benchmark_data_smithi171_78918_object154268:head# 2024-04-20T17:11:20.405 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.405 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7053937f:::benchmark_data_smithi171_78918_object147473:head# 2024-04-20T17:11:20.406 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.406 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:705509cf:::benchmark_data_smithi171_78918_object111543:head# 2024-04-20T17:11:20.406 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.406 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70558802:::benchmark_data_smithi171_78918_object108159:head# 2024-04-20T17:11:20.406 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.406 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70559ee1:::benchmark_data_smithi171_78918_object102846:head# 2024-04-20T17:11:20.406 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.406 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7055aa97:::benchmark_data_smithi171_78918_object129405:head# 2024-04-20T17:11:20.406 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.406 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70569c37:::benchmark_data_smithi171_78918_object151344:head# 2024-04-20T17:11:20.406 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.406 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7056ef54:::benchmark_data_smithi171_78918_object148985:head# 2024-04-20T17:11:20.406 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.406 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7057492c:::benchmark_data_smithi171_78918_object123110:head# 2024-04-20T17:11:20.407 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.407 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7057811d:::benchmark_data_smithi171_78918_object155161:head# 2024-04-20T17:11:20.407 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.407 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:705814ad:::benchmark_data_smithi171_78918_object64080:head# 2024-04-20T17:11:20.407 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.407 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70585c56:::benchmark_data_smithi171_78918_object68881:head# 2024-04-20T17:11:20.407 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.407 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70596494:::benchmark_data_smithi171_78918_object119192:head# 2024-04-20T17:11:20.407 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.407 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7059bf82:::benchmark_data_smithi171_78918_object128922:head# 2024-04-20T17:11:20.407 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.407 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:705a464a:::benchmark_data_smithi171_78918_object98066:head# 2024-04-20T17:11:20.407 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.407 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:705a72e7:::benchmark_data_smithi171_78918_object82792:head# 2024-04-20T17:11:20.407 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.408 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:705a9d0e:::benchmark_data_smithi171_78918_object118935:head# 2024-04-20T17:11:20.408 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.408 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:705a9f04:::benchmark_data_smithi171_78918_object54605:head# 2024-04-20T17:11:20.408 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.408 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:705b657c:::benchmark_data_smithi171_78918_object138261:head# 2024-04-20T17:11:20.408 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.408 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:705cdc1c:::benchmark_data_smithi171_78918_object132927:head# 2024-04-20T17:11:20.408 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.408 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:705e09d1:::benchmark_data_smithi171_78918_object130376:head# 2024-04-20T17:11:20.408 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.408 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:705fd2d8:::benchmark_data_smithi171_78918_object117723:head# 2024-04-20T17:11:20.408 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.408 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:705fe038:::benchmark_data_smithi171_78918_object121076:head# 2024-04-20T17:11:20.408 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.408 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7061615f:::benchmark_data_smithi171_78918_object137762:head# 2024-04-20T17:11:20.409 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.409 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7061660b:::benchmark_data_smithi171_78918_object157687:head# 2024-04-20T17:11:20.409 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.409 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70616b1a:::benchmark_data_smithi171_78918_object153364:head# 2024-04-20T17:11:20.409 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.409 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:706220a8:::benchmark_data_smithi171_78918_object53127:head# 2024-04-20T17:11:20.409 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.409 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70623a10:::benchmark_data_smithi171_78918_object61305:head# 2024-04-20T17:11:20.409 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.409 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70623f96:::benchmark_data_smithi171_78918_object126319:head# 2024-04-20T17:11:20.409 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.409 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70626ff7:::benchmark_data_smithi171_78918_object156458:head# 2024-04-20T17:11:20.409 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.409 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7063c342:::benchmark_data_smithi171_78918_object96455:head# 2024-04-20T17:11:20.409 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.409 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7063d358:::benchmark_data_smithi171_78918_object121530:head# 2024-04-20T17:11:20.410 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.410 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70648fa7:::benchmark_data_smithi171_78918_object159856:head# 2024-04-20T17:11:20.410 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.410 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7066712e:::benchmark_data_smithi171_78918_object71120:head# 2024-04-20T17:11:20.410 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.410 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7066cc72:::benchmark_data_smithi171_78918_object81973:head# 2024-04-20T17:11:20.410 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.410 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7067c860:::benchmark_data_smithi171_78918_object152687:head# 2024-04-20T17:11:20.410 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.410 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70683a4a:::benchmark_data_smithi171_78918_object78103:head# 2024-04-20T17:11:20.410 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.410 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:706937d6:::benchmark_data_smithi171_78918_object81341:head# 2024-04-20T17:11:20.410 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.410 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:706aa7dc:::benchmark_data_smithi171_78918_object99897:head# 2024-04-20T17:11:20.410 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.419 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:706baa39:::benchmark_data_smithi171_78918_object137567:head# 2024-04-20T17:11:20.419 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.419 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:706c1992:::benchmark_data_smithi171_78918_object63963:head# 2024-04-20T17:11:20.420 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.420 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:706ca2d2:::benchmark_data_smithi171_78918_object144139:head# 2024-04-20T17:11:20.420 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.420 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:706cb15e:::benchmark_data_smithi171_78918_object133012:head# 2024-04-20T17:11:20.420 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.420 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:706d32cd:::benchmark_data_smithi171_78918_object72573:head# 2024-04-20T17:11:20.420 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.420 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:706d6142:::benchmark_data_smithi171_78918_object67446:head# 2024-04-20T17:11:20.420 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.420 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:706ea5bd:::benchmark_data_smithi171_78918_object74644:head# 2024-04-20T17:11:20.420 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.420 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:706f621a:::benchmark_data_smithi171_78918_object67924:head# 2024-04-20T17:11:20.420 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.421 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7073c2ea:::benchmark_data_smithi171_78918_object139307:head# 2024-04-20T17:11:20.421 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.421 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7076501e:::benchmark_data_smithi171_78918_object69258:head# 2024-04-20T17:11:20.421 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.421 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7076570f:::benchmark_data_smithi171_78918_object150473:head# 2024-04-20T17:11:20.421 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.421 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:707696d5:::benchmark_data_smithi171_78918_object156398:head# 2024-04-20T17:11:20.421 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.421 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7076a585:::benchmark_data_smithi171_78918_object98246:head# 2024-04-20T17:11:20.421 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.421 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7076bead:::benchmark_data_smithi171_78918_object105986:head# 2024-04-20T17:11:20.421 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.421 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7076d976:::benchmark_data_smithi171_78918_object125313:head# 2024-04-20T17:11:20.421 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.421 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7077212a:::benchmark_data_smithi171_78918_object100021:head# 2024-04-20T17:11:20.422 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.422 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70775258:::benchmark_data_smithi171_78918_object55095:head# 2024-04-20T17:11:20.422 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.422 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7077d355:::benchmark_data_smithi171_78918_object144963:head# 2024-04-20T17:11:20.422 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.422 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70782680:::benchmark_data_smithi171_78918_object69736:head# 2024-04-20T17:11:20.422 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.422 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7078c52c:::benchmark_data_smithi171_78918_object141921:head# 2024-04-20T17:11:20.422 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.422 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7078cf61:::benchmark_data_smithi171_78918_object134862:head# 2024-04-20T17:11:20.422 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.422 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70791903:::benchmark_data_smithi171_78918_object133936:head# 2024-04-20T17:11:20.422 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.423 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:707939d1:::benchmark_data_smithi171_78918_object146777:head# 2024-04-20T17:11:20.423 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.423 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7079dd49:::benchmark_data_smithi171_78918_object118111:head# 2024-04-20T17:11:20.423 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.423 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7079e0b1:::benchmark_data_smithi171_78918_object108246:head# 2024-04-20T17:11:20.423 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.423 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:707b8823:::benchmark_data_smithi171_78918_object131510:head# 2024-04-20T17:11:20.423 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.423 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:707bed49:::benchmark_data_smithi171_78918_object120978:head# 2024-04-20T17:11:20.423 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.423 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:707bfae2:::benchmark_data_smithi171_78918_object158094:head# 2024-04-20T17:11:20.423 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.423 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:707c0a3a:::benchmark_data_smithi171_78918_object108021:head# 2024-04-20T17:11:20.423 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.424 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:707ca639:::benchmark_data_smithi171_78918_object55377:head# 2024-04-20T17:11:20.424 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.424 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:707cd2c7:::benchmark_data_smithi171_78918_object158689:head# 2024-04-20T17:11:20.424 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.424 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:707d8e9b:::benchmark_data_smithi171_78918_object160455:head# 2024-04-20T17:11:20.424 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.424 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:707db2ea:::benchmark_data_smithi171_78918_object141370:head# 2024-04-20T17:11:20.424 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.424 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:707f100e:::benchmark_data_smithi171_78918_object129664:head# 2024-04-20T17:11:20.424 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.424 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:707f1496:::benchmark_data_smithi171_78918_object95821:head# 2024-04-20T17:11:20.424 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.424 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:707faf55:::benchmark_data_smithi171_78918_object102092:head# 2024-04-20T17:11:20.424 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.425 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:708046a2:::benchmark_data_smithi171_78918_object112667:head# 2024-04-20T17:11:20.425 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.425 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7080cfc8:::benchmark_data_smithi171_78918_object123677:head# 2024-04-20T17:11:20.425 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.425 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70813187:::benchmark_data_smithi171_78918_object71686:head# 2024-04-20T17:11:20.425 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.425 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70819812:::benchmark_data_smithi171_78918_object111607:head# 2024-04-20T17:11:20.425 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.425 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7082f78d:::benchmark_data_smithi171_78918_object158265:head# 2024-04-20T17:11:20.425 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.425 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:708305a6:::benchmark_data_smithi171_78918_object139737:head# 2024-04-20T17:11:20.425 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.425 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70834dc5:::benchmark_data_smithi171_78918_object86994:head# 2024-04-20T17:11:20.425 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.426 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7083f38e:::benchmark_data_smithi171_78918_object116103:head# 2024-04-20T17:11:20.426 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.426 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:708468c5:::benchmark_data_smithi171_78918_object125352:head# 2024-04-20T17:11:20.426 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.426 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70861021:::benchmark_data_smithi171_78918_object156037:head# 2024-04-20T17:11:20.426 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.426 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7086ce95:::benchmark_data_smithi171_78918_object154375:head# 2024-04-20T17:11:20.426 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.426 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7086ee52:::benchmark_data_smithi171_78918_object131588:head# 2024-04-20T17:11:20.426 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.426 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7087fdf6:::benchmark_data_smithi171_78918_object63856:head# 2024-04-20T17:11:20.426 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.426 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7088d093:::benchmark_data_smithi171_78918_object149072:head# 2024-04-20T17:11:20.426 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.426 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7089a6e5:::benchmark_data_smithi171_78918_object63071:head# 2024-04-20T17:11:20.427 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.427 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7089fbb1:::benchmark_data_smithi171_78918_object78935:head# 2024-04-20T17:11:20.427 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.427 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:708b00ef:::benchmark_data_smithi171_78918_object75378:head# 2024-04-20T17:11:20.427 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.427 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:708d6c6a:::benchmark_data_smithi171_78918_object117001:head# 2024-04-20T17:11:20.427 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.427 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:708e093c:::benchmark_data_smithi171_78918_object105406:head# 2024-04-20T17:11:20.427 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.427 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:708e999b:::benchmark_data_smithi171_78918_object145600:head# 2024-04-20T17:11:20.427 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.427 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:708eefc4:::benchmark_data_smithi171_78918_object65873:head# 2024-04-20T17:11:20.427 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.427 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:708f26f8:::benchmark_data_smithi171_78918_object100347:head# 2024-04-20T17:11:20.427 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.428 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:708f2d76:::benchmark_data_smithi171_78918_object148963:head# 2024-04-20T17:11:20.428 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.428 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70902f22:::benchmark_data_smithi171_78918_object153239:head# 2024-04-20T17:11:20.428 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.428 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7092abd8:::benchmark_data_smithi171_78918_object52334:head# 2024-04-20T17:11:20.428 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.428 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70934c7e:::benchmark_data_smithi171_78918_object153259:head# 2024-04-20T17:11:20.428 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.428 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70934ec7:::benchmark_data_smithi171_78918_object124915:head# 2024-04-20T17:11:20.428 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.428 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7093dfba:::benchmark_data_smithi171_78918_object152497:head# 2024-04-20T17:11:20.428 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.428 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:709576e0:::benchmark_data_smithi171_78918_object104503:head# 2024-04-20T17:11:20.428 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.434 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7096c4c5:::benchmark_data_smithi171_78918_object151652:head# 2024-04-20T17:11:20.434 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.434 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7096ed96:::benchmark_data_smithi171_78918_object119004:head# 2024-04-20T17:11:20.434 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.434 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:709707c6:::benchmark_data_smithi171_78918_object158075:head# 2024-04-20T17:11:20.434 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.434 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70984489:::benchmark_data_smithi171_78918_object145198:head# 2024-04-20T17:11:20.434 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.434 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70990f80:::benchmark_data_smithi171_78918_object130967:head# 2024-04-20T17:11:20.435 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.435 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70997065:::benchmark_data_smithi171_78918_object148213:head# 2024-04-20T17:11:20.435 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.435 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7099c197:::benchmark_data_smithi171_78918_object153587:head# 2024-04-20T17:11:20.435 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.435 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:709a4c39:::benchmark_data_smithi171_78918_object58693:head# 2024-04-20T17:11:20.435 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.435 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:709a7d7b:::benchmark_data_smithi171_78918_object138989:head# 2024-04-20T17:11:20.435 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.435 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:709ade1d:::benchmark_data_smithi171_78918_object71203:head# 2024-04-20T17:11:20.435 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.435 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:709b80af:::benchmark_data_smithi171_78918_object118770:head# 2024-04-20T17:11:20.435 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.436 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:709ba2e1:::benchmark_data_smithi171_78918_object118904:head# 2024-04-20T17:11:20.436 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.436 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:709bffdf:::benchmark_data_smithi171_78918_object126046:head# 2024-04-20T17:11:20.436 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.436 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:709cb0be:::benchmark_data_smithi171_78918_object159062:head# 2024-04-20T17:11:20.436 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.436 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:709cd6e3:::benchmark_data_smithi171_78918_object157616:head# 2024-04-20T17:11:20.436 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.436 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:709d37c0:::benchmark_data_smithi171_78918_object69263:head# 2024-04-20T17:11:20.436 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.436 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:709d380e:::benchmark_data_smithi171_78918_object68300:head# 2024-04-20T17:11:20.436 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.436 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:709d462d:::benchmark_data_smithi171_78918_object128562:head# 2024-04-20T17:11:20.437 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.437 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:709dd84d:::benchmark_data_smithi171_78918_object64746:head# 2024-04-20T17:11:20.437 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.437 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:709fb3d1:::benchmark_data_smithi171_78918_object153648:head# 2024-04-20T17:11:20.437 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.437 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:709fdc83:::benchmark_data_smithi171_78918_object148759:head# 2024-04-20T17:11:20.437 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.437 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70a04232:::benchmark_data_smithi171_78918_object156069:head# 2024-04-20T17:11:20.437 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.437 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70a0d721:::benchmark_data_smithi171_78918_object121985:head# 2024-04-20T17:11:20.437 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.437 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70a13ae7:::benchmark_data_smithi171_78918_object156266:head# 2024-04-20T17:11:20.437 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.437 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70a26038:::benchmark_data_smithi171_78918_object150105:head# 2024-04-20T17:11:20.438 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.438 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70a31f40:::benchmark_data_smithi171_78918_object145825:head# 2024-04-20T17:11:20.438 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.438 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70a357cd:::benchmark_data_smithi171_78918_object135656:head# 2024-04-20T17:11:20.438 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.438 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70a3bf31:::benchmark_data_smithi171_78918_object105714:head# 2024-04-20T17:11:20.438 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.438 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70a46107:::benchmark_data_smithi171_78918_object92098:head# 2024-04-20T17:11:20.438 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.438 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70a47023:::benchmark_data_smithi171_78918_object125232:head# 2024-04-20T17:11:20.438 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.438 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70a520c4:::benchmark_data_smithi171_78918_object157991:head# 2024-04-20T17:11:20.438 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.438 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70a54a2a:::benchmark_data_smithi171_78918_object111053:head# 2024-04-20T17:11:20.439 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.439 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70a5ee0d:::benchmark_data_smithi171_78918_object132978:head# 2024-04-20T17:11:20.439 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.439 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70a6b4d4:::benchmark_data_smithi171_78918_object63689:head# 2024-04-20T17:11:20.439 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.439 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70a734ce:::benchmark_data_smithi171_78918_object136964:head# 2024-04-20T17:11:20.439 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.439 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70a77200:::benchmark_data_smithi171_78918_object60089:head# 2024-04-20T17:11:20.439 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.439 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70a92ff6:::benchmark_data_smithi171_78918_object94304:head# 2024-04-20T17:11:20.439 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.439 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70a93333:::benchmark_data_smithi171_78918_object95001:head# 2024-04-20T17:11:20.439 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.439 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70a9a8c9:::benchmark_data_smithi171_78918_object144609:head# 2024-04-20T17:11:20.440 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.440 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70aae84d:::benchmark_data_smithi171_78918_object63069:head# 2024-04-20T17:11:20.440 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.440 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70aba567:::benchmark_data_smithi171_78918_object87452:head# 2024-04-20T17:11:20.440 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.440 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70abb8d4:::benchmark_data_smithi171_78918_object78649:head# 2024-04-20T17:11:20.440 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.440 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70acd477:::benchmark_data_smithi171_78918_object60816:head# 2024-04-20T17:11:20.440 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.440 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70adb2ef:::benchmark_data_smithi171_78918_object134064:head# 2024-04-20T17:11:20.440 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.440 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70adf96e:::benchmark_data_smithi171_78918_object60780:head# 2024-04-20T17:11:20.440 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.440 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70ae6587:::benchmark_data_smithi171_78918_object92310:head# 2024-04-20T17:11:20.441 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.441 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70afc687:::benchmark_data_smithi171_78918_object58129:head# 2024-04-20T17:11:20.441 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.441 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70b012be:::benchmark_data_smithi171_78918_object54532:head# 2024-04-20T17:11:20.441 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.441 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70b02fa0:::benchmark_data_smithi171_78918_object139104:head# 2024-04-20T17:11:20.441 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.441 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70b11be4:::benchmark_data_smithi171_78918_object104332:head# 2024-04-20T17:11:20.441 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.441 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70b1cc26:::benchmark_data_smithi171_78918_object71427:head# 2024-04-20T17:11:20.441 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.441 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70b2eb7c:::benchmark_data_smithi171_78918_object77163:head# 2024-04-20T17:11:20.441 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.441 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70b423ab:::benchmark_data_smithi171_78918_object102872:head# 2024-04-20T17:11:20.441 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.442 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70b43539:::benchmark_data_smithi171_78918_object85424:head# 2024-04-20T17:11:20.442 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.442 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70b58a08:::benchmark_data_smithi171_78918_object63959:head# 2024-04-20T17:11:20.442 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.442 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70b63d0c:::benchmark_data_smithi171_78918_object142254:head# 2024-04-20T17:11:20.442 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.442 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70b669c1:::benchmark_data_smithi171_78918_object60798:head# 2024-04-20T17:11:20.442 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.442 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70b68142:::benchmark_data_smithi171_78918_object149088:head# 2024-04-20T17:11:20.442 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.442 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70b6cab8:::benchmark_data_smithi171_78918_object129829:head# 2024-04-20T17:11:20.442 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.442 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70b6f371:::benchmark_data_smithi171_78918_object79721:head# 2024-04-20T17:11:20.442 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.449 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70b7136b:::benchmark_data_smithi171_78918_object132178:head# 2024-04-20T17:11:20.449 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.449 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70b78def:::benchmark_data_smithi171_78918_object159704:head# 2024-04-20T17:11:20.449 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.449 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70b85f3e:::benchmark_data_smithi171_78918_object51954:head# 2024-04-20T17:11:20.449 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.449 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70b88757:::benchmark_data_smithi171_78918_object76844:head# 2024-04-20T17:11:20.449 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.449 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70b8c4ce:::benchmark_data_smithi171_78918_object53496:head# 2024-04-20T17:11:20.449 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.449 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70b8d21c:::benchmark_data_smithi171_78918_object106148:head# 2024-04-20T17:11:20.450 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.450 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70b8ff0b:::benchmark_data_smithi171_78918_object96071:head# 2024-04-20T17:11:20.450 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.450 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70b9a6ac:::benchmark_data_smithi171_78918_object113975:head# 2024-04-20T17:11:20.450 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.450 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70bb3363:::benchmark_data_smithi171_78918_object137237:head# 2024-04-20T17:11:20.450 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.450 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70bc5d8e:::benchmark_data_smithi171_78918_object142905:head# 2024-04-20T17:11:20.450 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.450 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70bc661a:::benchmark_data_smithi171_78918_object87995:head# 2024-04-20T17:11:20.450 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.450 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70bdabbd:::benchmark_data_smithi171_78918_object155961:head# 2024-04-20T17:11:20.450 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.450 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70bdb2f1:::benchmark_data_smithi171_78918_object86121:head# 2024-04-20T17:11:20.451 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.451 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70be23c9:::benchmark_data_smithi171_78918_object140513:head# 2024-04-20T17:11:20.451 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.451 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70be549f:::benchmark_data_smithi171_78918_object93974:head# 2024-04-20T17:11:20.451 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.451 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70beb3ef:::benchmark_data_smithi171_78918_object116457:head# 2024-04-20T17:11:20.451 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.451 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70bf69c9:::benchmark_data_smithi171_78918_object75870:head# 2024-04-20T17:11:20.451 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.451 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70c0f77a:::benchmark_data_smithi171_78918_object138179:head# 2024-04-20T17:11:20.451 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.451 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70c11f8f:::benchmark_data_smithi171_78918_object77818:head# 2024-04-20T17:11:20.451 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.452 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70c1be98:::benchmark_data_smithi171_78918_object104278:head# 2024-04-20T17:11:20.452 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.452 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70c21f8b:::benchmark_data_smithi171_78918_object82982:head# 2024-04-20T17:11:20.452 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.452 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70c26e83:::benchmark_data_smithi171_78918_object142493:head# 2024-04-20T17:11:20.452 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.452 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70c29f13:::benchmark_data_smithi171_78918_object125574:head# 2024-04-20T17:11:20.452 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.452 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70c42c0e:::benchmark_data_smithi171_78918_object114469:head# 2024-04-20T17:11:20.452 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.453 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70c47f9c:::benchmark_data_smithi171_78918_object146604:head# 2024-04-20T17:11:20.453 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.453 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70c48c50:::benchmark_data_smithi171_78918_object155753:head# 2024-04-20T17:11:20.453 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.453 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70c645a8:::benchmark_data_smithi171_78918_object142713:head# 2024-04-20T17:11:20.453 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.453 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70c6750f:::benchmark_data_smithi171_78918_object83520:head# 2024-04-20T17:11:20.453 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.453 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70c71abe:::benchmark_data_smithi171_78918_object117895:head# 2024-04-20T17:11:20.453 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.454 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70c7ab08:::benchmark_data_smithi171_78918_object69245:head# 2024-04-20T17:11:20.454 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.454 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70c884ee:::benchmark_data_smithi171_78918_object146490:head# 2024-04-20T17:11:20.454 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.454 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70c98e42:::benchmark_data_smithi171_78918_object100037:head# 2024-04-20T17:11:20.454 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.454 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70c99df1:::benchmark_data_smithi171_78918_object96272:head# 2024-04-20T17:11:20.454 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.454 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70c9ce48:::benchmark_data_smithi171_78918_object104172:head# 2024-04-20T17:11:20.454 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.454 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70ca4933:::benchmark_data_smithi171_78918_object85898:head# 2024-04-20T17:11:20.454 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.454 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70cb4067:::benchmark_data_smithi171_78918_object64269:head# 2024-04-20T17:11:20.455 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.455 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70cee318:::benchmark_data_smithi171_78918_object92624:head# 2024-04-20T17:11:20.455 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.455 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70cef44a:::benchmark_data_smithi171_78918_object148216:head# 2024-04-20T17:11:20.455 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.455 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70cfa1cc:::benchmark_data_smithi171_78918_object72721:head# 2024-04-20T17:11:20.455 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.455 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70d02ec6:::benchmark_data_smithi171_78918_object69006:head# 2024-04-20T17:11:20.455 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.455 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70d2765b:::benchmark_data_smithi171_78918_object145100:head# 2024-04-20T17:11:20.455 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.455 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70d39dab:::benchmark_data_smithi171_78918_object110149:head# 2024-04-20T17:11:20.455 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.455 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70d3a9be:::benchmark_data_smithi171_78918_object92222:head# 2024-04-20T17:11:20.456 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.456 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70d65e07:::benchmark_data_smithi171_78918_object85082:head# 2024-04-20T17:11:20.456 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.456 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70d68334:::benchmark_data_smithi171_78918_object131521:head# 2024-04-20T17:11:20.456 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.456 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70d68a60:::benchmark_data_smithi171_78918_object148833:head# 2024-04-20T17:11:20.456 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.456 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70d73763:::benchmark_data_smithi171_78918_object152339:head# 2024-04-20T17:11:20.456 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.456 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70d7a0e6:::benchmark_data_smithi171_78918_object108686:head# 2024-04-20T17:11:20.456 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.456 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70d7c7f0:::benchmark_data_smithi171_78918_object145740:head# 2024-04-20T17:11:20.456 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.457 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70d831e0:::benchmark_data_smithi171_78918_object54664:head# 2024-04-20T17:11:20.457 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.457 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70d9c42d:::benchmark_data_smithi171_78918_object123331:head# 2024-04-20T17:11:20.457 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.457 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70dc0e8d:::benchmark_data_smithi171_78918_object134593:head# 2024-04-20T17:11:20.457 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.457 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70dc9117:::benchmark_data_smithi171_78918_object72992:head# 2024-04-20T17:11:20.457 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.457 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70de6371:::benchmark_data_smithi171_78918_object126374:head# 2024-04-20T17:11:20.457 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.457 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70df1017:::benchmark_data_smithi171_78918_object75247:head# 2024-04-20T17:11:20.457 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.457 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70e34ff0:::benchmark_data_smithi171_78918_object69371:head# 2024-04-20T17:11:20.457 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.458 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70e367d5:::benchmark_data_smithi171_78918_object138457:head# 2024-04-20T17:11:20.458 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.458 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70e552cd:::benchmark_data_smithi171_78918_object63644:head# 2024-04-20T17:11:20.458 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.458 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70e639b1:::benchmark_data_smithi171_78918_object74520:head# 2024-04-20T17:11:20.458 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.458 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70e76dbe:::benchmark_data_smithi171_78918_object147120:head# 2024-04-20T17:11:20.458 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.458 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70e868ab:::benchmark_data_smithi171_78918_object114400:head# 2024-04-20T17:11:20.458 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.458 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70e8d486:::benchmark_data_smithi171_78918_object55798:head# 2024-04-20T17:11:20.458 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.468 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70e90bd8:::benchmark_data_smithi171_78918_object155310:head# 2024-04-20T17:11:20.468 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.468 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70e9692b:::benchmark_data_smithi171_78918_object77645:head# 2024-04-20T17:11:20.468 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.468 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70ec4b00:::benchmark_data_smithi171_78918_object113326:head# 2024-04-20T17:11:20.468 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.468 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70ec5abb:::benchmark_data_smithi171_78918_object138469:head# 2024-04-20T17:11:20.468 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.468 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70ed24b6:::benchmark_data_smithi171_78918_object111802:head# 2024-04-20T17:11:20.468 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.468 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70ed9519:::benchmark_data_smithi171_78918_object75204:head# 2024-04-20T17:11:20.468 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.468 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70ede9fd:::benchmark_data_smithi171_78918_object156435:head# 2024-04-20T17:11:20.469 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.469 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70ee0922:::benchmark_data_smithi171_78918_object158740:head# 2024-04-20T17:11:20.469 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.469 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70ee1d0d:::benchmark_data_smithi171_78918_object64640:head# 2024-04-20T17:11:20.469 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.469 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70ee5308:::benchmark_data_smithi171_78918_object98114:head# 2024-04-20T17:11:20.469 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.469 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70eeb801:::benchmark_data_smithi171_78918_object125604:head# 2024-04-20T17:11:20.469 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.469 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70eec461:::benchmark_data_smithi171_78918_object72019:head# 2024-04-20T17:11:20.469 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.469 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70eee9dd:::benchmark_data_smithi171_78918_object156659:head# 2024-04-20T17:11:20.469 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.470 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70ef13e5:::benchmark_data_smithi171_78918_object75825:head# 2024-04-20T17:11:20.470 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.470 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70ef3418:::benchmark_data_smithi171_78918_object113011:head# 2024-04-20T17:11:20.470 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.470 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70ef8a7c:::benchmark_data_smithi171_78918_object67130:head# 2024-04-20T17:11:20.470 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.470 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70efc44a:::benchmark_data_smithi171_78918_object85162:head# 2024-04-20T17:11:20.470 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.470 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70efc988:::benchmark_data_smithi171_78918_object151330:head# 2024-04-20T17:11:20.470 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.470 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70f00c7d:::benchmark_data_smithi171_78918_object79834:head# 2024-04-20T17:11:20.470 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.470 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70f0dbcc:::benchmark_data_smithi171_78918_object101133:head# 2024-04-20T17:11:20.470 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.471 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70f19dd3:::benchmark_data_smithi171_78918_object87272:head# 2024-04-20T17:11:20.471 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.471 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70f3416b:::benchmark_data_smithi171_78918_object133561:head# 2024-04-20T17:11:20.471 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.471 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70f36001:::benchmark_data_smithi171_78918_object148964:head# 2024-04-20T17:11:20.471 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.471 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70f3fd17:::benchmark_data_smithi171_78918_object91331:head# 2024-04-20T17:11:20.471 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.471 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70f43a86:::benchmark_data_smithi171_78918_object141262:head# 2024-04-20T17:11:20.471 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.471 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70f43e36:::benchmark_data_smithi171_78918_object132081:head# 2024-04-20T17:11:20.471 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.471 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70f46aa2:::benchmark_data_smithi171_78918_object144360:head# 2024-04-20T17:11:20.472 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.472 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70f538ab:::benchmark_data_smithi171_78918_object77978:head# 2024-04-20T17:11:20.472 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.472 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70f57190:::benchmark_data_smithi171_78918_object83369:head# 2024-04-20T17:11:20.472 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.472 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70f63d77:::benchmark_data_smithi171_78918_object135989:head# 2024-04-20T17:11:20.472 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.472 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70f6479b:::benchmark_data_smithi171_78918_object70449:head# 2024-04-20T17:11:20.472 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.472 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70f68952:::benchmark_data_smithi171_78918_object105671:head# 2024-04-20T17:11:20.472 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.472 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70f700ba:::benchmark_data_smithi171_78918_object74125:head# 2024-04-20T17:11:20.472 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.472 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70f9ed4b:::benchmark_data_smithi171_78918_object125726:head# 2024-04-20T17:11:20.472 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.473 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70f9ee83:::benchmark_data_smithi171_78918_object93295:head# 2024-04-20T17:11:20.473 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.473 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70fac79f:::benchmark_data_smithi171_78918_object110255:head# 2024-04-20T17:11:20.473 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.473 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70fb42a9:::benchmark_data_smithi171_78918_object123716:head# 2024-04-20T17:11:20.473 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.473 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70fc9d70:::benchmark_data_smithi171_78918_object108420:head# 2024-04-20T17:11:20.473 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.473 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70fdbee9:::benchmark_data_smithi171_78918_object123230:head# 2024-04-20T17:11:20.473 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.473 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70fe5812:::benchmark_data_smithi171_78918_object69141:head# 2024-04-20T17:11:20.473 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.473 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:70fff3f9:::benchmark_data_smithi171_78918_object81557:head# 2024-04-20T17:11:20.473 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.474 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71002455:::benchmark_data_smithi171_78918_object79442:head# 2024-04-20T17:11:20.474 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.474 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:710082ef:::benchmark_data_smithi171_78918_object137694:head# 2024-04-20T17:11:20.474 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.474 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7100e7c1:::benchmark_data_smithi171_78918_object65955:head# 2024-04-20T17:11:20.474 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.474 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71012fed:::benchmark_data_smithi171_78918_object147882:head# 2024-04-20T17:11:20.474 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.474 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71020993:::benchmark_data_smithi171_78918_object145248:head# 2024-04-20T17:11:20.474 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.474 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71036f13:::benchmark_data_smithi171_78918_object54240:head# 2024-04-20T17:11:20.474 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.474 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71039c7c:::benchmark_data_smithi171_78918_object160080:head# 2024-04-20T17:11:20.474 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.475 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7103daf6:::benchmark_data_smithi171_78918_object54651:head# 2024-04-20T17:11:20.475 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.475 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7103e5dc:::benchmark_data_smithi171_78918_object139537:head# 2024-04-20T17:11:20.475 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.475 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7103fb4d:::benchmark_data_smithi171_78918_object60663:head# 2024-04-20T17:11:20.475 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.475 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71059bb7:::benchmark_data_smithi171_78918_object93791:head# 2024-04-20T17:11:20.475 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.475 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7105c147:::benchmark_data_smithi171_78918_object78944:head# 2024-04-20T17:11:20.475 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.475 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:710713c3:::benchmark_data_smithi171_78918_object76020:head# 2024-04-20T17:11:20.475 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.475 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71078d31:::benchmark_data_smithi171_78918_object123113:head# 2024-04-20T17:11:20.475 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.476 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7107eee8:::benchmark_data_smithi171_78918_object142563:head# 2024-04-20T17:11:20.476 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.476 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71093187:::benchmark_data_smithi171_78918_object137986:head# 2024-04-20T17:11:20.476 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.476 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7109e6ba:::benchmark_data_smithi171_78918_object63155:head# 2024-04-20T17:11:20.476 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.476 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:710b8e3f:::benchmark_data_smithi171_78918_object157756:head# 2024-04-20T17:11:20.476 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.476 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:710c716b:::benchmark_data_smithi171_78918_object76866:head# 2024-04-20T17:11:20.476 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.476 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:710d1318:::benchmark_data_smithi171_78918_object88081:head# 2024-04-20T17:11:20.476 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.476 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:710daa6c:::benchmark_data_smithi171_78918_object160739:head# 2024-04-20T17:11:20.476 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.477 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:710e2010:::benchmark_data_smithi171_78918_object97039:head# 2024-04-20T17:11:20.477 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.477 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:710f92bb:::benchmark_data_smithi171_78918_object84056:head# 2024-04-20T17:11:20.477 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.477 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:711156fc:::benchmark_data_smithi171_78918_object52169:head# 2024-04-20T17:11:20.477 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.477 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71118a44:::benchmark_data_smithi171_78918_object129173:head# 2024-04-20T17:11:20.477 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.477 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71123ff6:::benchmark_data_smithi171_78918_object78578:head# 2024-04-20T17:11:20.477 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.477 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71125a4e:::benchmark_data_smithi171_78918_object101188:head# 2024-04-20T17:11:20.477 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.477 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71129d9c:::benchmark_data_smithi171_78918_object85934:head# 2024-04-20T17:11:20.478 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.478 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71129faf:::benchmark_data_smithi171_78918_object123384:head# 2024-04-20T17:11:20.478 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.478 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7112eff6:::benchmark_data_smithi171_78918_object101908:head# 2024-04-20T17:11:20.478 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.478 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71155bed:::benchmark_data_smithi171_78918_object64964:head# 2024-04-20T17:11:20.478 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.478 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71160949:::benchmark_data_smithi171_78918_object98856:head# 2024-04-20T17:11:20.478 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.478 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71163705:::benchmark_data_smithi171_78918_object102766:head# 2024-04-20T17:11:20.478 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.478 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7117515e:::benchmark_data_smithi171_78918_object96374:head# 2024-04-20T17:11:20.478 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.478 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71177421:::benchmark_data_smithi171_78918_object124974:head# 2024-04-20T17:11:20.479 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.479 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7117f3a7:::benchmark_data_smithi171_78918_object70871:head# 2024-04-20T17:11:20.479 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.479 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:11:20.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:20.486 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:11:20.497 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71191682:::benchmark_data_smithi171_78918_object147305:head# 2024-04-20T17:11:20.497 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.497 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:711afa96:::benchmark_data_smithi171_78918_object76503:head# 2024-04-20T17:11:20.497 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.497 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:711c5ac5:::benchmark_data_smithi171_78918_object135495:head# 2024-04-20T17:11:20.498 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.498 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:711c70ce:::benchmark_data_smithi171_78918_object95207:head# 2024-04-20T17:11:20.498 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.498 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:711cb416:::benchmark_data_smithi171_78918_object139062:head# 2024-04-20T17:11:20.498 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.498 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:711d0b1a:::benchmark_data_smithi171_78918_object128477:head# 2024-04-20T17:11:20.498 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.498 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:711d32c7:::benchmark_data_smithi171_78918_object52450:head# 2024-04-20T17:11:20.498 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.498 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:711dbf62:::benchmark_data_smithi171_78918_object126000:head# 2024-04-20T17:11:20.498 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.499 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:711dfb3d:::benchmark_data_smithi171_78918_object81069:head# 2024-04-20T17:11:20.499 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.499 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:711e09fa:::benchmark_data_smithi171_78918_object138192:head# 2024-04-20T17:11:20.499 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.499 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:711e96d8:::benchmark_data_smithi171_78918_object54286:head# 2024-04-20T17:11:20.499 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.499 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:711e9f69:::benchmark_data_smithi171_78918_object155503:head# 2024-04-20T17:11:20.499 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.499 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:711edd6e:::benchmark_data_smithi171_78918_object69550:head# 2024-04-20T17:11:20.499 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.499 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:711f6efe:::benchmark_data_smithi171_78918_object89069:head# 2024-04-20T17:11:20.499 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.499 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:711f7c70:::benchmark_data_smithi171_78918_object79241:head# 2024-04-20T17:11:20.500 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.500 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:711fd41d:::benchmark_data_smithi171_78918_object140984:head# 2024-04-20T17:11:20.500 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.500 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7121b998:::benchmark_data_smithi171_78918_object92343:head# 2024-04-20T17:11:20.500 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.500 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7121f7b2:::benchmark_data_smithi171_78918_object55693:head# 2024-04-20T17:11:20.500 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.500 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7123bfcc:::benchmark_data_smithi171_78918_object83344:head# 2024-04-20T17:11:20.500 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.500 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:712401ed:::benchmark_data_smithi171_78918_object71827:head# 2024-04-20T17:11:20.500 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.500 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71240d4a:::benchmark_data_smithi171_78918_object128440:head# 2024-04-20T17:11:20.500 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.501 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71249119:::benchmark_data_smithi171_78918_object132810:head# 2024-04-20T17:11:20.501 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.501 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7124ce0c:::benchmark_data_smithi171_78918_object96149:head# 2024-04-20T17:11:20.501 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.501 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7124dc54:::benchmark_data_smithi171_78918_object66558:head# 2024-04-20T17:11:20.501 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.501 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71259ed6:::benchmark_data_smithi171_78918_object63423:head# 2024-04-20T17:11:20.501 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.501 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7126b1ba:::benchmark_data_smithi171_78918_object138592:head# 2024-04-20T17:11:20.501 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.501 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:712706ba:::benchmark_data_smithi171_78918_object71000:head# 2024-04-20T17:11:20.501 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.501 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7127737c:::benchmark_data_smithi171_78918_object81601:head# 2024-04-20T17:11:20.502 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.502 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:712820b3:::benchmark_data_smithi171_78918_object90827:head# 2024-04-20T17:11:20.502 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.502 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71287615:::benchmark_data_smithi171_78918_object57653:head# 2024-04-20T17:11:20.502 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.502 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7128e79a:::benchmark_data_smithi171_78918_object58783:head# 2024-04-20T17:11:20.502 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.502 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7129a9fc:::benchmark_data_smithi171_78918_object141033:head# 2024-04-20T17:11:20.502 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.502 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:712a153a:::benchmark_data_smithi171_78918_object55780:head# 2024-04-20T17:11:20.502 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.502 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:712aa335:::benchmark_data_smithi171_78918_object86322:head# 2024-04-20T17:11:20.502 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.503 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:712b9ec3:::benchmark_data_smithi171_78918_object125806:head# 2024-04-20T17:11:20.503 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.503 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:712d2f84:::benchmark_data_smithi171_78918_object141076:head# 2024-04-20T17:11:20.503 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.503 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:712ed77f:::benchmark_data_smithi171_78918_object100963:head# 2024-04-20T17:11:20.503 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.503 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:712f1f2c:::benchmark_data_smithi171_78918_object105552:head# 2024-04-20T17:11:20.503 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.503 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:712f8e79:::benchmark_data_smithi171_78918_object133417:head# 2024-04-20T17:11:20.503 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.503 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:713068d9:::benchmark_data_smithi171_78918_object73012:head# 2024-04-20T17:11:20.503 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.504 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:713095ba:::benchmark_data_smithi171_78918_object122100:head# 2024-04-20T17:11:20.504 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.504 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7131ff36:::benchmark_data_smithi171_78918_object151713:head# 2024-04-20T17:11:20.504 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.504 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7132abfc:::benchmark_data_smithi171_78918_object59592:head# 2024-04-20T17:11:20.504 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.504 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71339a0f:::benchmark_data_smithi171_78918_object89008:head# 2024-04-20T17:11:20.504 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.504 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71353291:::benchmark_data_smithi171_78918_object125667:head# 2024-04-20T17:11:20.504 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.504 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:713555e7:::benchmark_data_smithi171_78918_object98429:head# 2024-04-20T17:11:20.504 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.504 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71358e7c:::benchmark_data_smithi171_78918_object65159:head# 2024-04-20T17:11:20.505 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.505 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7135ddf1:::benchmark_data_smithi171_78918_object154573:head# 2024-04-20T17:11:20.505 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.505 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71360bba:::benchmark_data_smithi171_78918_object75724:head# 2024-04-20T17:11:20.505 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.505 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7136197b:::benchmark_data_smithi171_78918_object90404:head# 2024-04-20T17:11:20.505 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.505 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7138b967:::benchmark_data_smithi171_78918_object127969:head# 2024-04-20T17:11:20.505 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.505 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71398106:::benchmark_data_smithi171_78918_object102965:head# 2024-04-20T17:11:20.505 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.506 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:713a109c:::benchmark_data_smithi171_78918_object63311:head# 2024-04-20T17:11:20.506 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.506 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:713b0aab:::benchmark_data_smithi171_78918_object76846:head# 2024-04-20T17:11:20.506 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.506 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:713b3569:::benchmark_data_smithi171_78918_object107766:head# 2024-04-20T17:11:20.506 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.506 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:713b4d91:::benchmark_data_smithi171_78918_object117245:head# 2024-04-20T17:11:20.506 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.506 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:713c1858:::benchmark_data_smithi171_78918_object92975:head# 2024-04-20T17:11:20.506 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.506 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:713c56e3:::benchmark_data_smithi171_78918_object63510:head# 2024-04-20T17:11:20.506 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.506 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:713ce652:::benchmark_data_smithi171_78918_object60864:head# 2024-04-20T17:11:20.507 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.507 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:713eab21:::benchmark_data_smithi171_78918_object101721:head# 2024-04-20T17:11:20.507 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.507 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:713f9037:::benchmark_data_smithi171_78918_object140693:head# 2024-04-20T17:11:20.507 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.507 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:713fa5f8:::benchmark_data_smithi171_78918_object143197:head# 2024-04-20T17:11:20.507 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.507 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:714172a0:::benchmark_data_smithi171_78918_object85005:head# 2024-04-20T17:11:20.507 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.507 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:714193fe:::benchmark_data_smithi171_78918_object119559:head# 2024-04-20T17:11:20.507 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.507 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7141e5cd:::benchmark_data_smithi171_78918_object73700:head# 2024-04-20T17:11:20.507 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.507 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71428de9:::benchmark_data_smithi171_78918_object99086:head# 2024-04-20T17:11:20.507 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.508 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7142b2fa:::benchmark_data_smithi171_78918_object90851:head# 2024-04-20T17:11:20.508 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.508 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71439674:::benchmark_data_smithi171_78918_object122875:head# 2024-04-20T17:11:20.508 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.508 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71451821:::benchmark_data_smithi171_78918_object103241:head# 2024-04-20T17:11:20.508 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.508 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71455265:::benchmark_data_smithi171_78918_object155452:head# 2024-04-20T17:11:20.508 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.508 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71489085:::benchmark_data_smithi171_78918_object102289:head# 2024-04-20T17:11:20.508 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.508 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7148a402:::benchmark_data_smithi171_78918_object91116:head# 2024-04-20T17:11:20.508 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.508 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7148d7ed:::benchmark_data_smithi171_78918_object132794:head# 2024-04-20T17:11:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:714ae465:::benchmark_data_smithi171_78918_object60716:head# 2024-04-20T17:11:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:714c883c:::benchmark_data_smithi171_78918_object54247:head# 2024-04-20T17:11:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:714d2710:::benchmark_data_smithi171_78918_object142085:head# 2024-04-20T17:11:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:714ded77:::benchmark_data_smithi171_78918_object138650:head# 2024-04-20T17:11:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:714e22d4:::benchmark_data_smithi171_78918_object111731:head# 2024-04-20T17:11:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:714e4ac1:::benchmark_data_smithi171_78918_object144220:head# 2024-04-20T17:11:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:714e8a9d:::benchmark_data_smithi171_78918_object140408:head# 2024-04-20T17:11:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.510 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:714ef2eb:::benchmark_data_smithi171_78918_object135496:head# 2024-04-20T17:11:20.510 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.510 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:714f4504:::benchmark_data_smithi171_78918_object118075:head# 2024-04-20T17:11:20.510 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.510 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:714f94ee:::benchmark_data_smithi171_78918_object140514:head# 2024-04-20T17:11:20.510 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.510 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:714fc3a5:::benchmark_data_smithi171_78918_object105909:head# 2024-04-20T17:11:20.510 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.510 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:714fe1e7:::benchmark_data_smithi171_78918_object105565:head# 2024-04-20T17:11:20.510 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.510 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71500596:::benchmark_data_smithi171_78918_object81683:head# 2024-04-20T17:11:20.510 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.510 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:715033c0:::benchmark_data_smithi171_78918_object141568:head# 2024-04-20T17:11:20.511 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.511 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:715224f9:::benchmark_data_smithi171_78918_object118331:head# 2024-04-20T17:11:20.511 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.511 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7152d99f:::benchmark_data_smithi171_78918_object141393:head# 2024-04-20T17:11:20.511 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.511 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:715357bd:::benchmark_data_smithi171_78918_object52451:head# 2024-04-20T17:11:20.511 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.511 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71536853:::benchmark_data_smithi171_78918_object78906:head# 2024-04-20T17:11:20.511 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.511 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71542352:::benchmark_data_smithi171_78918_object77844:head# 2024-04-20T17:11:20.511 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.511 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7154356b:::benchmark_data_smithi171_78918_object80380:head# 2024-04-20T17:11:20.511 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.511 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7155a749:::benchmark_data_smithi171_78918_object107960:head# 2024-04-20T17:11:20.511 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.512 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7155cd40:::benchmark_data_smithi171_78918_object145240:head# 2024-04-20T17:11:20.512 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.512 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7155f901:::benchmark_data_smithi171_78918_object82425:head# 2024-04-20T17:11:20.512 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.512 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7156858f:::benchmark_data_smithi171_78918_object103048:head# 2024-04-20T17:11:20.512 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.512 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:715702ac:::benchmark_data_smithi171_78918_object130607:head# 2024-04-20T17:11:20.512 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.512 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71571783:::benchmark_data_smithi171_78918_object103690:head# 2024-04-20T17:11:20.512 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.512 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7157c326:::benchmark_data_smithi171_78918_object147489:head# 2024-04-20T17:11:20.512 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.513 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:715895ce:::benchmark_data_smithi171_78918_object84320:head# 2024-04-20T17:11:20.513 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.513 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7158acd3:::benchmark_data_smithi171_78918_object72796:head# 2024-04-20T17:11:20.513 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.513 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7158ff61:::benchmark_data_smithi171_78918_object154861:head# 2024-04-20T17:11:20.513 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.513 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7159587c:::benchmark_data_smithi171_78918_object70548:head# 2024-04-20T17:11:20.513 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.513 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:715a18e3:::benchmark_data_smithi171_78918_object156269:head# 2024-04-20T17:11:20.513 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.513 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:715a3a42:::benchmark_data_smithi171_78918_object152775:head# 2024-04-20T17:11:20.513 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.513 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:715b2e44:::benchmark_data_smithi171_78918_object108684:head# 2024-04-20T17:11:20.513 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.514 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:715bd50c:::benchmark_data_smithi171_78918_object115544:head# 2024-04-20T17:11:20.514 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.514 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:715c5be7:::benchmark_data_smithi171_78918_object98962:head# 2024-04-20T17:11:20.514 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.514 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:715c87ef:::benchmark_data_smithi171_78918_object90765:head# 2024-04-20T17:11:20.514 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.514 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:715cc600:::benchmark_data_smithi171_78918_object122903:head# 2024-04-20T17:11:20.514 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.514 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:715d23d8:::benchmark_data_smithi171_78918_object156358:head# 2024-04-20T17:11:20.514 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.514 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:715d836a:::benchmark_data_smithi171_78918_object59909:head# 2024-04-20T17:11:20.514 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.514 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:715df2d0:::benchmark_data_smithi171_78918_object58277:head# 2024-04-20T17:11:20.515 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.530 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:715e2091:::benchmark_data_smithi171_78918_object61293:head# 2024-04-20T17:11:20.530 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.530 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:715e3cf1:::benchmark_data_smithi171_78918_object122009:head# 2024-04-20T17:11:20.530 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.531 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:715e7b7b:::benchmark_data_smithi171_78918_object62277:head# 2024-04-20T17:11:20.531 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.531 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:715e841e:::benchmark_data_smithi171_78918_object144551:head# 2024-04-20T17:11:20.531 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.531 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:715ec029:::benchmark_data_smithi171_78918_object131499:head# 2024-04-20T17:11:20.531 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.531 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:715f662e:::benchmark_data_smithi171_78918_object154087:head# 2024-04-20T17:11:20.531 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.531 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:715f9311:::benchmark_data_smithi171_78918_object117244:head# 2024-04-20T17:11:20.531 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.531 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:716160cd:::benchmark_data_smithi171_78918_object115362:head# 2024-04-20T17:11:20.531 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.531 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71616e16:::benchmark_data_smithi171_78918_object76482:head# 2024-04-20T17:11:20.531 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.532 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71618f6a:::benchmark_data_smithi171_78918_object52718:head# 2024-04-20T17:11:20.532 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.532 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:716324ee:::benchmark_data_smithi171_78918_object68826:head# 2024-04-20T17:11:20.532 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.532 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7163550b:::benchmark_data_smithi171_78918_object118982:head# 2024-04-20T17:11:20.532 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.532 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7163d6ba:::benchmark_data_smithi171_78918_object102546:head# 2024-04-20T17:11:20.532 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.532 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:716405f9:::benchmark_data_smithi171_78918_object115765:head# 2024-04-20T17:11:20.532 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.532 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7164cb11:::benchmark_data_smithi171_78918_object99873:head# 2024-04-20T17:11:20.532 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.532 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71656f89:::benchmark_data_smithi171_78918_object154719:head# 2024-04-20T17:11:20.532 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.533 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71661ed6:::benchmark_data_smithi171_78918_object101386:head# 2024-04-20T17:11:20.533 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.533 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:716788e7:::benchmark_data_smithi171_78918_object160322:head# 2024-04-20T17:11:20.533 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.533 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7169d397:::benchmark_data_smithi171_78918_object78758:head# 2024-04-20T17:11:20.533 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.533 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:716a5e05:::benchmark_data_smithi171_78918_object130635:head# 2024-04-20T17:11:20.533 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.533 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:716aa09e:::benchmark_data_smithi171_78918_object121038:head# 2024-04-20T17:11:20.533 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.533 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:716b9428:::benchmark_data_smithi171_78918_object76046:head# 2024-04-20T17:11:20.533 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.533 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:716bec3d:::benchmark_data_smithi171_78918_object100036:head# 2024-04-20T17:11:20.533 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.534 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:716c00d4:::benchmark_data_smithi171_78918_object100275:head# 2024-04-20T17:11:20.534 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.534 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:716c54d5:::benchmark_data_smithi171_78918_object153659:head# 2024-04-20T17:11:20.534 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.534 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:716c60a2:::benchmark_data_smithi171_78918_object133834:head# 2024-04-20T17:11:20.534 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.534 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:716e4475:::benchmark_data_smithi171_78918_object117360:head# 2024-04-20T17:11:20.534 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.534 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:716eb862:::benchmark_data_smithi171_78918_object66753:head# 2024-04-20T17:11:20.534 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.534 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:716ebe7e:::benchmark_data_smithi171_78918_object98162:head# 2024-04-20T17:11:20.534 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.534 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71716738:::benchmark_data_smithi171_78918_object111774:head# 2024-04-20T17:11:20.535 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.535 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7172b0fb:::benchmark_data_smithi171_78918_object69852:head# 2024-04-20T17:11:20.535 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.535 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7172ea3f:::benchmark_data_smithi171_78918_object113910:head# 2024-04-20T17:11:20.535 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.535 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7173aea7:::benchmark_data_smithi171_78918_object63404:head# 2024-04-20T17:11:20.535 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.535 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7173c779:::benchmark_data_smithi171_78918_object72168:head# 2024-04-20T17:11:20.535 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.535 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71745dbe:::benchmark_data_smithi171_78918_object66508:head# 2024-04-20T17:11:20.535 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.535 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7175a236:::benchmark_data_smithi171_78918_object131370:head# 2024-04-20T17:11:20.535 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.535 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:717687b7:::benchmark_data_smithi171_78918_object96737:head# 2024-04-20T17:11:20.536 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.536 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7176f2f4:::benchmark_data_smithi171_78918_object107701:head# 2024-04-20T17:11:20.536 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.536 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7178807c:::benchmark_data_smithi171_78918_object111464:head# 2024-04-20T17:11:20.536 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.536 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:717a010b:::benchmark_data_smithi171_78918_object102990:head# 2024-04-20T17:11:20.536 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.536 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:717a9565:::benchmark_data_smithi171_78918_object93723:head# 2024-04-20T17:11:20.536 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.536 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:717aa73e:::benchmark_data_smithi171_78918_object84896:head# 2024-04-20T17:11:20.536 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.536 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:717bcf76:::benchmark_data_smithi171_78918_object61930:head# 2024-04-20T17:11:20.536 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.536 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:717c733c:::benchmark_data_smithi171_78918_object61008:head# 2024-04-20T17:11:20.537 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.537 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:717c7c20:::benchmark_data_smithi171_78918_object145929:head# 2024-04-20T17:11:20.537 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.537 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:717d2d76:::benchmark_data_smithi171_78918_object131532:head# 2024-04-20T17:11:20.537 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.537 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:717db82c:::benchmark_data_smithi171_78918_object153085:head# 2024-04-20T17:11:20.537 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.537 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:717f0ff6:::benchmark_data_smithi171_78918_object60240:head# 2024-04-20T17:11:20.537 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.537 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71801557:::benchmark_data_smithi171_78918_object147728:head# 2024-04-20T17:11:20.537 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.537 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7180e963:::benchmark_data_smithi171_78918_object91442:head# 2024-04-20T17:11:20.537 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.537 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7180f8af:::benchmark_data_smithi171_78918_object154698:head# 2024-04-20T17:11:20.538 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.538 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71822b1e:::benchmark_data_smithi171_78918_object67282:head# 2024-04-20T17:11:20.538 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.538 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71822d00:::benchmark_data_smithi171_78918_object134524:head# 2024-04-20T17:11:20.538 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.538 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:718241f6:::benchmark_data_smithi171_78918_object126942:head# 2024-04-20T17:11:20.538 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.538 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71832571:::benchmark_data_smithi171_78918_object118136:head# 2024-04-20T17:11:20.538 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.538 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7184cd5c:::benchmark_data_smithi171_78918_object107579:head# 2024-04-20T17:11:20.538 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.538 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71868c70:::benchmark_data_smithi171_78918_object106136:head# 2024-04-20T17:11:20.538 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.539 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7186997a:::benchmark_data_smithi171_78918_object114499:head# 2024-04-20T17:11:20.539 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.539 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7186e13d:::benchmark_data_smithi171_78918_object87970:head# 2024-04-20T17:11:20.539 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.539 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7187f419:::benchmark_data_smithi171_78918_object118581:head# 2024-04-20T17:11:20.539 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.539 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71882186:::benchmark_data_smithi171_78918_object136727:head# 2024-04-20T17:11:20.539 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.539 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71885861:::benchmark_data_smithi171_78918_object133078:head# 2024-04-20T17:11:20.539 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.539 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71891daa:::benchmark_data_smithi171_78918_object77141:head# 2024-04-20T17:11:20.539 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.539 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:718930b4:::benchmark_data_smithi171_78918_object98569:head# 2024-04-20T17:11:20.539 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.539 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71893ee9:::benchmark_data_smithi171_78918_object124084:head# 2024-04-20T17:11:20.540 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.540 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71894949:::benchmark_data_smithi171_78918_object81398:head# 2024-04-20T17:11:20.540 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.540 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:718a7ed9:::benchmark_data_smithi171_78918_object114890:head# 2024-04-20T17:11:20.540 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.540 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:718b2972:::benchmark_data_smithi171_78918_object144238:head# 2024-04-20T17:11:20.540 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.540 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:718b4640:::benchmark_data_smithi171_78918_object98404:head# 2024-04-20T17:11:20.540 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.540 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:718c88dc:::benchmark_data_smithi171_78918_object108999:head# 2024-04-20T17:11:20.540 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.540 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:718ce256:::benchmark_data_smithi171_78918_object120598:head# 2024-04-20T17:11:20.540 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.540 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:718d399b:::benchmark_data_smithi171_78918_object126587:head# 2024-04-20T17:11:20.541 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.541 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:718db564:::benchmark_data_smithi171_78918_object131948:head# 2024-04-20T17:11:20.541 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.541 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:718db7af:::benchmark_data_smithi171_78918_object150080:head# 2024-04-20T17:11:20.541 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.541 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:718e0fcf:::benchmark_data_smithi171_78918_object68784:head# 2024-04-20T17:11:20.541 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.541 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:718e7031:::benchmark_data_smithi171_78918_object125327:head# 2024-04-20T17:11:20.541 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.541 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:718e9335:::benchmark_data_smithi171_78918_object134203:head# 2024-04-20T17:11:20.541 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.541 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:718ea60e:::benchmark_data_smithi171_78918_object100602:head# 2024-04-20T17:11:20.541 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.542 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:718ed8ca:::benchmark_data_smithi171_78918_object64333:head# 2024-04-20T17:11:20.542 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.542 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:719135c8:::benchmark_data_smithi171_78918_object69757:head# 2024-04-20T17:11:20.542 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.542 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7191546d:::benchmark_data_smithi171_78918_object141779:head# 2024-04-20T17:11:20.542 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.542 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71915f5b:::benchmark_data_smithi171_78918_object140176:head# 2024-04-20T17:11:20.542 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.542 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:719176ec:::benchmark_data_smithi171_78918_object152890:head# 2024-04-20T17:11:20.542 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.542 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7191c4b1:::benchmark_data_smithi171_78918_object102600:head# 2024-04-20T17:11:20.542 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.543 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71925136:::benchmark_data_smithi171_78918_object127919:head# 2024-04-20T17:11:20.543 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.543 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7193bbce:::benchmark_data_smithi171_78918_object129725:head# 2024-04-20T17:11:20.543 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.543 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:719413da:::benchmark_data_smithi171_78918_object82809:head# 2024-04-20T17:11:20.543 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.543 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7194c86c:::benchmark_data_smithi171_78918_object160211:head# 2024-04-20T17:11:20.543 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.543 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71952815:::benchmark_data_smithi171_78918_object110489:head# 2024-04-20T17:11:20.543 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.543 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71954d50:::benchmark_data_smithi171_78918_object152032:head# 2024-04-20T17:11:20.543 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.543 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:719630e8:::benchmark_data_smithi171_78918_object121353:head# 2024-04-20T17:11:20.543 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.544 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71964383:::benchmark_data_smithi171_78918_object117526:head# 2024-04-20T17:11:20.544 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.544 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71966bfd:::benchmark_data_smithi171_78918_object125099:head# 2024-04-20T17:11:20.544 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.544 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7196aced:::benchmark_data_smithi171_78918_object134658:head# 2024-04-20T17:11:20.544 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.544 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7196b4b9:::benchmark_data_smithi171_78918_object105557:head# 2024-04-20T17:11:20.544 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.544 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7196d530:::benchmark_data_smithi171_78918_object149411:head# 2024-04-20T17:11:20.544 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.544 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71971618:::benchmark_data_smithi171_78918_object95654:head# 2024-04-20T17:11:20.544 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.545 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7197216d:::benchmark_data_smithi171_78918_object159360:head# 2024-04-20T17:11:20.545 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.545 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:719790c6:::benchmark_data_smithi171_78918_object128212:head# 2024-04-20T17:11:20.545 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.545 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71989823:::benchmark_data_smithi171_78918_object113551:head# 2024-04-20T17:11:20.545 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.545 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:719aa2f1:::benchmark_data_smithi171_78918_object98290:head# 2024-04-20T17:11:20.545 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.545 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:719c897e:::benchmark_data_smithi171_78918_object134169:head# 2024-04-20T17:11:20.545 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.545 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:719cf7fc:::benchmark_data_smithi171_78918_object63686:head# 2024-04-20T17:11:20.545 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.545 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:719d8333:::benchmark_data_smithi171_78918_object124213:head# 2024-04-20T17:11:20.546 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.546 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:719ec08c:::benchmark_data_smithi171_78918_object152695:head# 2024-04-20T17:11:20.546 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.546 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:719ed440:::benchmark_data_smithi171_78918_object93962:head# 2024-04-20T17:11:20.546 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.546 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:719f2354:::benchmark_data_smithi171_78918_object140595:head# 2024-04-20T17:11:20.546 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.546 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:719ff6d3:::benchmark_data_smithi171_78918_object115824:head# 2024-04-20T17:11:20.546 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.546 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71a111ad:::benchmark_data_smithi171_78918_object63529:head# 2024-04-20T17:11:20.546 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.546 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71a18d23:::benchmark_data_smithi171_78918_object122119:head# 2024-04-20T17:11:20.546 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.547 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71a19c52:::benchmark_data_smithi171_78918_object127880:head# 2024-04-20T17:11:20.547 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.547 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71a329da:::benchmark_data_smithi171_78918_object64208:head# 2024-04-20T17:11:20.547 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.547 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71a34872:::benchmark_data_smithi171_78918_object88279:head# 2024-04-20T17:11:20.547 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.547 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71a3583e:::benchmark_data_smithi171_78918_object130481:head# 2024-04-20T17:11:20.547 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.547 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71a41217:::benchmark_data_smithi171_78918_object83736:head# 2024-04-20T17:11:20.547 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.547 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71a47747:::benchmark_data_smithi171_78918_object105089:head# 2024-04-20T17:11:20.547 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.547 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71a54d54:::benchmark_data_smithi171_78918_object54033:head# 2024-04-20T17:11:20.548 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.548 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71a57232:::benchmark_data_smithi171_78918_object138224:head# 2024-04-20T17:11:20.548 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.548 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71a60901:::benchmark_data_smithi171_78918_object82304:head# 2024-04-20T17:11:20.548 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.548 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71a6e724:::benchmark_data_smithi171_78918_object153720:head# 2024-04-20T17:11:20.548 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.548 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71a71d9d:::benchmark_data_smithi171_78918_object160173:head# 2024-04-20T17:11:20.548 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.548 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71a7355f:::benchmark_data_smithi171_78918_object85477:head# 2024-04-20T17:11:20.548 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.548 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71a7ad55:::benchmark_data_smithi171_78918_object92393:head# 2024-04-20T17:11:20.548 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.548 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71a8ba6d:::benchmark_data_smithi171_78918_object118086:head# 2024-04-20T17:11:20.548 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.549 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71a8d6e1:::benchmark_data_smithi171_78918_object99936:head# 2024-04-20T17:11:20.549 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.549 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71a9345b:::benchmark_data_smithi171_78918_object156338:head# 2024-04-20T17:11:20.549 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.549 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71a94873:::benchmark_data_smithi171_78918_object149368:head# 2024-04-20T17:11:20.549 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.581 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71a9c30b:::benchmark_data_smithi171_78918_object96630:head# 2024-04-20T17:11:20.581 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.581 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71aa2257:::benchmark_data_smithi171_78918_object76363:head# 2024-04-20T17:11:20.581 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.581 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71aa8ea2:::benchmark_data_smithi171_78918_object131919:head# 2024-04-20T17:11:20.581 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.581 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71aabbb5:::benchmark_data_smithi171_78918_object154399:head# 2024-04-20T17:11:20.582 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.582 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71ab3738:::benchmark_data_smithi171_78918_object54603:head# 2024-04-20T17:11:20.582 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.582 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71ab54ab:::benchmark_data_smithi171_78918_object94614:head# 2024-04-20T17:11:20.582 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.582 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71abae29:::benchmark_data_smithi171_78918_object99806:head# 2024-04-20T17:11:20.582 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.582 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71ac2adf:::benchmark_data_smithi171_78918_object117158:head# 2024-04-20T17:11:20.582 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.582 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71ac8b85:::benchmark_data_smithi171_78918_object156029:head# 2024-04-20T17:11:20.582 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.582 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71ad6220:::benchmark_data_smithi171_78918_object96430:head# 2024-04-20T17:11:20.582 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.582 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71ad8886:::benchmark_data_smithi171_78918_object144110:head# 2024-04-20T17:11:20.583 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.583 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71ae5dd5:::benchmark_data_smithi171_78918_object104750:head# 2024-04-20T17:11:20.583 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.583 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71af4fbc:::benchmark_data_smithi171_78918_object55635:head# 2024-04-20T17:11:20.583 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.583 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71af89f8:::benchmark_data_smithi171_78918_object81576:head# 2024-04-20T17:11:20.583 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.583 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71aff8e2:::benchmark_data_smithi171_78918_object98791:head# 2024-04-20T17:11:20.585 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.585 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71b046fa:::benchmark_data_smithi171_78918_object78420:head# 2024-04-20T17:11:20.585 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.585 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71b059e8:::benchmark_data_smithi171_78918_object150893:head# 2024-04-20T17:11:20.586 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.586 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71b0d259:::benchmark_data_smithi171_78918_object119237:head# 2024-04-20T17:11:20.586 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.586 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71b12106:::benchmark_data_smithi171_78918_object100273:head# 2024-04-20T17:11:20.586 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.586 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71b12fa6:::benchmark_data_smithi171_78918_object68196:head# 2024-04-20T17:11:20.586 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.586 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71b146c8:::benchmark_data_smithi171_78918_object130330:head# 2024-04-20T17:11:20.586 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.586 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71b18ba4:::benchmark_data_smithi171_78918_object100664:head# 2024-04-20T17:11:20.586 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.586 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71b27d13:::benchmark_data_smithi171_78918_object74545:head# 2024-04-20T17:11:20.586 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.586 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71b30f72:::benchmark_data_smithi171_78918_object105350:head# 2024-04-20T17:11:20.587 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.587 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71b38c2b:::benchmark_data_smithi171_78918_object81771:head# 2024-04-20T17:11:20.587 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.587 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71b3dcb8:::benchmark_data_smithi171_78918_object153756:head# 2024-04-20T17:11:20.587 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.587 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71b4d926:::benchmark_data_smithi171_78918_object118571:head# 2024-04-20T17:11:20.587 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.587 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71b51a4e:::benchmark_data_smithi171_78918_object84323:head# 2024-04-20T17:11:20.587 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.587 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71b52b71:::benchmark_data_smithi171_78918_object55124:head# 2024-04-20T17:11:20.587 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.587 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71b58bf5:::benchmark_data_smithi171_78918_object129201:head# 2024-04-20T17:11:20.587 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.587 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71b58fb0:::benchmark_data_smithi171_78918_object118131:head# 2024-04-20T17:11:20.587 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.588 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71b5ff01:::benchmark_data_smithi171_78918_object111629:head# 2024-04-20T17:11:20.588 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.588 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71b6228f:::benchmark_data_smithi171_78918_object119728:head# 2024-04-20T17:11:20.588 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.588 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71b6eb8b:::benchmark_data_smithi171_78918_object144750:head# 2024-04-20T17:11:20.588 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.588 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71b78e12:::benchmark_data_smithi171_78918_object133818:head# 2024-04-20T17:11:20.588 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.588 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71b7dfce:::benchmark_data_smithi171_78918_object117141:head# 2024-04-20T17:11:20.588 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.588 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71b8246e:::benchmark_data_smithi171_78918_object156033:head# 2024-04-20T17:11:20.588 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.588 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71b878c4:::benchmark_data_smithi171_78918_object132253:head# 2024-04-20T17:11:20.588 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.589 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71b955ac:::benchmark_data_smithi171_78918_object142628:head# 2024-04-20T17:11:20.589 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.589 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71ba0d12:::benchmark_data_smithi171_78918_object139909:head# 2024-04-20T17:11:20.589 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.589 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71ba6525:::benchmark_data_smithi171_78918_object82006:head# 2024-04-20T17:11:20.589 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.589 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71baa820:::benchmark_data_smithi171_78918_object143811:head# 2024-04-20T17:11:20.589 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.589 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71bb0d03:::benchmark_data_smithi171_78918_object155223:head# 2024-04-20T17:11:20.589 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.589 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71bb26c3:::benchmark_data_smithi171_78918_object68807:head# 2024-04-20T17:11:20.589 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.589 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71bb7e68:::benchmark_data_smithi171_78918_object126249:head# 2024-04-20T17:11:20.589 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.590 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71bc9f51:::benchmark_data_smithi171_78918_object60112:head# 2024-04-20T17:11:20.590 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.590 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71bdc3d2:::benchmark_data_smithi171_78918_object77273:head# 2024-04-20T17:11:20.590 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.590 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71be1d74:::benchmark_data_smithi171_78918_object89366:head# 2024-04-20T17:11:20.590 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.590 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71beca88:::benchmark_data_smithi171_78918_object57398:head# 2024-04-20T17:11:20.590 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.590 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71c0436a:::benchmark_data_smithi171_78918_object101686:head# 2024-04-20T17:11:20.590 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.590 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71c04778:::benchmark_data_smithi171_78918_object88439:head# 2024-04-20T17:11:20.590 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.590 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71c0d168:::benchmark_data_smithi171_78918_object67612:head# 2024-04-20T17:11:20.590 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.591 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71c1e545:::benchmark_data_smithi171_78918_object130379:head# 2024-04-20T17:11:20.591 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.591 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71c3285a:::benchmark_data_smithi171_78918_object109846:head# 2024-04-20T17:11:20.591 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.591 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71c39b87:::benchmark_data_smithi171_78918_object81261:head# 2024-04-20T17:11:20.591 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.591 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71c49944:::benchmark_data_smithi171_78918_object143284:head# 2024-04-20T17:11:20.591 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.591 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71c543e7:::benchmark_data_smithi171_78918_object100161:head# 2024-04-20T17:11:20.591 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.591 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71c575a3:::benchmark_data_smithi171_78918_object109115:head# 2024-04-20T17:11:20.591 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.591 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71c5eda9:::benchmark_data_smithi171_78918_object101875:head# 2024-04-20T17:11:20.591 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.591 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71c62baa:::benchmark_data_smithi171_78918_object72683:head# 2024-04-20T17:11:20.592 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.592 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71c647be:::benchmark_data_smithi171_78918_object116346:head# 2024-04-20T17:11:20.592 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.592 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71c75e42:::benchmark_data_smithi171_78918_object113200:head# 2024-04-20T17:11:20.592 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.592 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71c7d8e8:::benchmark_data_smithi171_78918_object65498:head# 2024-04-20T17:11:20.592 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.592 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71c7f54c:::benchmark_data_smithi171_78918_object79414:head# 2024-04-20T17:11:20.592 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.592 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71c832a2:::benchmark_data_smithi171_78918_object68923:head# 2024-04-20T17:11:20.592 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.592 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71c859ad:::benchmark_data_smithi171_78918_object121351:head# 2024-04-20T17:11:20.592 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.592 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71c9550b:::benchmark_data_smithi171_78918_object129616:head# 2024-04-20T17:11:20.592 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.593 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71c95bb4:::benchmark_data_smithi171_78918_object125494:head# 2024-04-20T17:11:20.593 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.593 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71c9f2f0:::benchmark_data_smithi171_78918_object113410:head# 2024-04-20T17:11:20.593 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.593 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71cad3bd:::benchmark_data_smithi171_78918_object55754:head# 2024-04-20T17:11:20.593 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.593 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71cc0338:::benchmark_data_smithi171_78918_object108667:head# 2024-04-20T17:11:20.593 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.593 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71cc3454:::benchmark_data_smithi171_78918_object86034:head# 2024-04-20T17:11:20.593 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.593 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71ccc415:::benchmark_data_smithi171_78918_object75108:head# 2024-04-20T17:11:20.593 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.593 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71ccf23c:::benchmark_data_smithi171_78918_object52550:head# 2024-04-20T17:11:20.593 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.594 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71ce41cb:::benchmark_data_smithi171_78918_object103531:head# 2024-04-20T17:11:20.594 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.594 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71cebd85:::benchmark_data_smithi171_78918_object114109:head# 2024-04-20T17:11:20.594 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.594 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71cfcc77:::benchmark_data_smithi171_78918_object80421:head# 2024-04-20T17:11:20.594 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.594 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71d027b8:::benchmark_data_smithi171_78918_object140731:head# 2024-04-20T17:11:20.594 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.594 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71d04bdc:::benchmark_data_smithi171_78918_object64324:head# 2024-04-20T17:11:20.594 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.594 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71d0a867:::benchmark_data_smithi171_78918_object67504:head# 2024-04-20T17:11:20.594 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.594 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71d162f3:::benchmark_data_smithi171_78918_object59559:head# 2024-04-20T17:11:20.594 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.595 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71d39ccf:::benchmark_data_smithi171_78918_object125651:head# 2024-04-20T17:11:20.595 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.595 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71d3c437:::benchmark_data_smithi171_78918_object125122:head# 2024-04-20T17:11:20.595 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.595 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71d409cb:::benchmark_data_smithi171_78918_object101593:head# 2024-04-20T17:11:20.595 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.595 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71d52d6d:::benchmark_data_smithi171_78918_object123531:head# 2024-04-20T17:11:20.595 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.595 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71d54fc2:::benchmark_data_smithi171_78918_object149009:head# 2024-04-20T17:11:20.595 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.595 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71d55987:::benchmark_data_smithi171_78918_object112538:head# 2024-04-20T17:11:20.595 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.595 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71d58706:::benchmark_data_smithi171_78918_object156420:head# 2024-04-20T17:11:20.595 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.595 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71d69f11:::benchmark_data_smithi171_78918_object73610:head# 2024-04-20T17:11:20.596 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.596 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71d7863a:::benchmark_data_smithi171_78918_object85078:head# 2024-04-20T17:11:20.596 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.596 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71d8989e:::benchmark_data_smithi171_78918_object115726:head# 2024-04-20T17:11:20.596 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.596 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71d8ea1a:::benchmark_data_smithi171_78918_object67985:head# 2024-04-20T17:11:20.596 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.596 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71d95d14:::benchmark_data_smithi171_78918_object132584:head# 2024-04-20T17:11:20.596 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.596 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71d97120:::benchmark_data_smithi171_78918_object96764:head# 2024-04-20T17:11:20.596 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.596 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71da73a5:::benchmark_data_smithi171_78918_object58972:head# 2024-04-20T17:11:20.596 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.596 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71db7e8f:::benchmark_data_smithi171_78918_object87916:head# 2024-04-20T17:11:20.597 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.597 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71dbd663:::benchmark_data_smithi171_78918_object143657:head# 2024-04-20T17:11:20.597 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.597 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71dc1c09:::benchmark_data_smithi171_78918_object85323:head# 2024-04-20T17:11:20.597 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.597 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71dc1e09:::benchmark_data_smithi171_78918_object117105:head# 2024-04-20T17:11:20.597 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.597 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71df2552:::benchmark_data_smithi171_78918_object115214:head# 2024-04-20T17:11:20.597 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.597 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71df9b55:::benchmark_data_smithi171_78918_object102227:head# 2024-04-20T17:11:20.597 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.597 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71dfbc14:::benchmark_data_smithi171_78918_object58636:head# 2024-04-20T17:11:20.597 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.597 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71e00d01:::benchmark_data_smithi171_78918_object69353:head# 2024-04-20T17:11:20.597 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.598 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71e13d95:::benchmark_data_smithi171_78918_object53544:head# 2024-04-20T17:11:20.598 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.598 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71e1a7c1:::benchmark_data_smithi171_78918_object132463:head# 2024-04-20T17:11:20.598 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.598 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71e52916:::benchmark_data_smithi171_78918_object59812:head# 2024-04-20T17:11:20.598 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.598 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71e54295:::benchmark_data_smithi171_78918_object52210:head# 2024-04-20T17:11:20.598 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.598 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71e57046:::benchmark_data_smithi171_78918_object154022:head# 2024-04-20T17:11:20.598 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.598 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71e5e508:::benchmark_data_smithi171_78918_object130161:head# 2024-04-20T17:11:20.598 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.598 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71e73498:::benchmark_data_smithi171_78918_object86660:head# 2024-04-20T17:11:20.598 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.599 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71e73b87:::benchmark_data_smithi171_78918_object86306:head# 2024-04-20T17:11:20.599 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.599 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71e7b7e9:::benchmark_data_smithi171_78918_object72953:head# 2024-04-20T17:11:20.599 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.599 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71e7fd38:::benchmark_data_smithi171_78918_object120043:head# 2024-04-20T17:11:20.599 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.599 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71e874b0:::benchmark_data_smithi171_78918_object55248:head# 2024-04-20T17:11:20.599 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.599 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71e8c231:::benchmark_data_smithi171_78918_object99293:head# 2024-04-20T17:11:20.599 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.599 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71e95252:::benchmark_data_smithi171_78918_object99148:head# 2024-04-20T17:11:20.599 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.599 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71eb4730:::benchmark_data_smithi171_78918_object107523:head# 2024-04-20T17:11:20.599 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.599 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71eb9894:::benchmark_data_smithi171_78918_object90572:head# 2024-04-20T17:11:20.600 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.600 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71ebe7a7:::benchmark_data_smithi171_78918_object56471:head# 2024-04-20T17:11:20.600 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.600 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71ec3c1b:::benchmark_data_smithi171_78918_object102906:head# 2024-04-20T17:11:20.600 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.600 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71ec4ca8:::benchmark_data_smithi171_78918_object109732:head# 2024-04-20T17:11:20.600 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.600 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71ec6399:::benchmark_data_smithi171_78918_object64291:head# 2024-04-20T17:11:20.600 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.600 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71ec9493:::benchmark_data_smithi171_78918_object114829:head# 2024-04-20T17:11:20.600 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.600 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71eeddbb:::benchmark_data_smithi171_78918_object70993:head# 2024-04-20T17:11:20.600 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.600 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71eeeca4:::benchmark_data_smithi171_78918_object59252:head# 2024-04-20T17:11:20.600 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.601 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71ef265a:::benchmark_data_smithi171_78918_object137904:head# 2024-04-20T17:11:20.601 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.601 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71efa570:::benchmark_data_smithi171_78918_object59320:head# 2024-04-20T17:11:20.601 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.601 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71efb132:::benchmark_data_smithi171_78918_object113997:head# 2024-04-20T17:11:20.601 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.601 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71f377b9:::benchmark_data_smithi171_78918_object97143:head# 2024-04-20T17:11:20.601 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.601 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71f38573:::benchmark_data_smithi171_78918_object102895:head# 2024-04-20T17:11:20.601 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.601 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71f3bae1:::benchmark_data_smithi171_78918_object79576:head# 2024-04-20T17:11:20.601 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.601 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71f3d612:::benchmark_data_smithi171_78918_object95972:head# 2024-04-20T17:11:20.601 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.602 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71f3eded:::benchmark_data_smithi171_78918_object121701:head# 2024-04-20T17:11:20.602 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.602 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71f598ad:::benchmark_data_smithi171_78918_object107375:head# 2024-04-20T17:11:20.602 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.602 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71f5b2bd:::benchmark_data_smithi171_78918_object103796:head# 2024-04-20T17:11:20.602 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.602 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71f765f5:::benchmark_data_smithi171_78918_object52920:head# 2024-04-20T17:11:20.602 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.602 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71f7fa42:::benchmark_data_smithi171_78918_object108916:head# 2024-04-20T17:11:20.602 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.602 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71f88abf:::benchmark_data_smithi171_78918_object145267:head# 2024-04-20T17:11:20.602 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.602 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71f9b510:::benchmark_data_smithi171_78918_object122601:head# 2024-04-20T17:11:20.602 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.602 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71fa1043:::benchmark_data_smithi171_78918_object72285:head# 2024-04-20T17:11:20.603 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.603 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71fa886d:::benchmark_data_smithi171_78918_object146956:head# 2024-04-20T17:11:20.603 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.603 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71faf492:::benchmark_data_smithi171_78918_object83447:head# 2024-04-20T17:11:20.603 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.603 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71fb1a5a:::benchmark_data_smithi171_78918_object116846:head# 2024-04-20T17:11:20.603 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.603 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71fb7678:::benchmark_data_smithi171_78918_object72207:head# 2024-04-20T17:11:20.603 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.603 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71fc04ea:::benchmark_data_smithi171_78918_object86757:head# 2024-04-20T17:11:20.603 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.603 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71fc5948:::benchmark_data_smithi171_78918_object141916:head# 2024-04-20T17:11:20.603 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.603 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71fd1357:::benchmark_data_smithi171_78918_object74693:head# 2024-04-20T17:11:20.604 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.604 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71fde422:::benchmark_data_smithi171_78918_object81728:head# 2024-04-20T17:11:20.604 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.604 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71fe62d5:::benchmark_data_smithi171_78918_object137287:head# 2024-04-20T17:11:20.604 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.604 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:71fffd07:::benchmark_data_smithi171_78918_object73498:head# 2024-04-20T17:11:20.604 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.604 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:720124ac:::benchmark_data_smithi171_78918_object91699:head# 2024-04-20T17:11:20.604 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.604 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:720177cc:::benchmark_data_smithi171_78918_object132995:head# 2024-04-20T17:11:20.604 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.604 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72019708:::benchmark_data_smithi171_78918_object160525:head# 2024-04-20T17:11:20.604 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.604 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7201d7d9:::benchmark_data_smithi171_78918_object80036:head# 2024-04-20T17:11:20.605 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.605 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7202586b:::benchmark_data_smithi171_78918_object120283:head# 2024-04-20T17:11:20.605 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.605 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:720289a4:::benchmark_data_smithi171_78918_object131831:head# 2024-04-20T17:11:20.605 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.605 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7202b200:::benchmark_data_smithi171_78918_object140480:head# 2024-04-20T17:11:20.605 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.605 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7202c2db:::benchmark_data_smithi171_78918_object124863:head# 2024-04-20T17:11:20.605 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.605 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7203651b:::benchmark_data_smithi171_78918_object119210:head# 2024-04-20T17:11:20.605 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.605 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7203e11a:::benchmark_data_smithi171_78918_object159621:head# 2024-04-20T17:11:20.605 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.605 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7203eadf:::benchmark_data_smithi171_78918_object112066:head# 2024-04-20T17:11:20.606 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.606 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72051a34:::benchmark_data_smithi171_78918_object125166:head# 2024-04-20T17:11:20.606 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.606 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:720540d7:::benchmark_data_smithi171_78918_object98609:head# 2024-04-20T17:11:20.606 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.606 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7205aec7:::benchmark_data_smithi171_78918_object79075:head# 2024-04-20T17:11:20.606 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.606 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:720634cd:::benchmark_data_smithi171_78918_object108426:head# 2024-04-20T17:11:20.606 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.606 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7206974f:::benchmark_data_smithi171_78918_object121729:head# 2024-04-20T17:11:20.606 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.606 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7206b9da:::benchmark_data_smithi171_78918_object154874:head# 2024-04-20T17:11:20.606 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.606 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72077a40:::benchmark_data_smithi171_78918_object131385:head# 2024-04-20T17:11:20.607 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.607 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7207b255:::benchmark_data_smithi171_78918_object68670:head# 2024-04-20T17:11:20.607 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.607 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7207dba9:::benchmark_data_smithi171_78918_object81707:head# 2024-04-20T17:11:20.607 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.607 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7207dc3e:::benchmark_data_smithi171_78918_object111383:head# 2024-04-20T17:11:20.607 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.607 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72081ec8:::benchmark_data_smithi171_78918_object115742:head# 2024-04-20T17:11:20.607 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.607 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7208b9c3:::benchmark_data_smithi171_78918_object77619:head# 2024-04-20T17:11:20.607 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.607 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:720928c7:::benchmark_data_smithi171_78918_object56145:head# 2024-04-20T17:11:20.607 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.607 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72099331:::benchmark_data_smithi171_78918_object59515:head# 2024-04-20T17:11:20.607 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.608 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7209e72a:::benchmark_data_smithi171_78918_object129540:head# 2024-04-20T17:11:20.608 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.608 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:720a7add:::benchmark_data_smithi171_78918_object69549:head# 2024-04-20T17:11:20.608 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.608 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:720af474:::benchmark_data_smithi171_78918_object54111:head# 2024-04-20T17:11:20.608 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.608 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:720b4a79:::benchmark_data_smithi171_78918_object70463:head# 2024-04-20T17:11:20.608 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.608 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:720b50bf:::benchmark_data_smithi171_78918_object160436:head# 2024-04-20T17:11:20.608 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.608 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:720b52df:::benchmark_data_smithi171_78918_object53818:head# 2024-04-20T17:11:20.608 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.608 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:720bdd22:::benchmark_data_smithi171_78918_object155190:head# 2024-04-20T17:11:20.608 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.608 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:720beafc:::benchmark_data_smithi171_78918_object115354:head# 2024-04-20T17:11:20.609 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.609 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:720cb6cf:::benchmark_data_smithi171_78918_object109091:head# 2024-04-20T17:11:20.609 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.609 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:720cbeda:::benchmark_data_smithi171_78918_object120749:head# 2024-04-20T17:11:20.609 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.609 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:720cfce1:::benchmark_data_smithi171_78918_object156893:head# 2024-04-20T17:11:20.609 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.609 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:720d2dee:::benchmark_data_smithi171_78918_object157008:head# 2024-04-20T17:11:20.609 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.609 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:720deae2:::benchmark_data_smithi171_78918_object91311:head# 2024-04-20T17:11:20.609 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.609 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:720f85ea:::benchmark_data_smithi171_78918_object83631:head# 2024-04-20T17:11:20.609 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.609 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72105a2e:::benchmark_data_smithi171_78918_object89736:head# 2024-04-20T17:11:20.609 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.610 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7211adf7:::benchmark_data_smithi171_78918_object85012:head# 2024-04-20T17:11:20.610 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.610 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72123cfd:::benchmark_data_smithi171_78918_object55977:head# 2024-04-20T17:11:20.610 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.610 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7212da1e:::benchmark_data_smithi171_78918_object72271:head# 2024-04-20T17:11:20.610 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.610 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72133c9c:::benchmark_data_smithi171_78918_object104066:head# 2024-04-20T17:11:20.610 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.610 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72138040:::benchmark_data_smithi171_78918_object151103:head# 2024-04-20T17:11:20.610 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.610 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72160790:::benchmark_data_smithi171_78918_object158248:head# 2024-04-20T17:11:20.610 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.610 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7216642a:::benchmark_data_smithi171_78918_object99353:head# 2024-04-20T17:11:20.611 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.611 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72167de5:::benchmark_data_smithi171_78918_object126732:head# 2024-04-20T17:11:20.611 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.611 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72168ccf:::benchmark_data_smithi171_78918_object72921:head# 2024-04-20T17:11:20.611 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.611 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:721719e3:::benchmark_data_smithi171_78918_object106142:head# 2024-04-20T17:11:20.611 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.611 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72172a31:::benchmark_data_smithi171_78918_object160629:head# 2024-04-20T17:11:20.611 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.611 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72181d1d:::benchmark_data_smithi171_78918_object84979:head# 2024-04-20T17:11:20.611 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.611 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72190c5f:::benchmark_data_smithi171_78918_object56280:head# 2024-04-20T17:11:20.611 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.611 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:721a46de:::benchmark_data_smithi171_78918_object93847:head# 2024-04-20T17:11:20.612 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.612 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:721b3a41:::benchmark_data_smithi171_78918_object87996:head# 2024-04-20T17:11:20.612 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.612 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:721bb503:::benchmark_data_smithi171_78918_object103605:head# 2024-04-20T17:11:20.612 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.612 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:721c30d6:::benchmark_data_smithi171_78918_object71885:head# 2024-04-20T17:11:20.612 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.612 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:721cb266:::benchmark_data_smithi171_78918_object130393:head# 2024-04-20T17:11:20.612 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.612 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:721ccd9f:::benchmark_data_smithi171_78918_object152513:head# 2024-04-20T17:11:20.612 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.612 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:721d01c1:::benchmark_data_smithi171_78918_object103506:head# 2024-04-20T17:11:20.612 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.612 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:721d2029:::benchmark_data_smithi171_78918_object99598:head# 2024-04-20T17:11:20.612 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.613 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:721d57b0:::benchmark_data_smithi171_78918_object152152:head# 2024-04-20T17:11:20.613 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.613 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:721db6a5:::benchmark_data_smithi171_78918_object96489:head# 2024-04-20T17:11:20.613 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.613 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:721e6426:::benchmark_data_smithi171_78918_object98572:head# 2024-04-20T17:11:20.613 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.613 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:721f1355:::benchmark_data_smithi171_78918_object110262:head# 2024-04-20T17:11:20.613 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.613 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:721f74ce:::benchmark_data_smithi171_78918_object129780:head# 2024-04-20T17:11:20.613 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.613 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:721fddb4:::benchmark_data_smithi171_78918_object113149:head# 2024-04-20T17:11:20.613 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.613 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7220c8c2:::benchmark_data_smithi171_78918_object98086:head# 2024-04-20T17:11:20.613 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.613 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72225150:::benchmark_data_smithi171_78918_object52953:head# 2024-04-20T17:11:20.614 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.614 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7222882b:::benchmark_data_smithi171_78918_object128624:head# 2024-04-20T17:11:20.614 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.614 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:722388bb:::benchmark_data_smithi171_78918_object117616:head# 2024-04-20T17:11:20.614 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.614 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:11:20.649 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:20.659 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72241313:::benchmark_data_smithi171_78918_object144153:head# 2024-04-20T17:11:20.659 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.659 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7224d533:::benchmark_data_smithi171_78918_object153376:head# 2024-04-20T17:11:20.659 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.659 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72254b51:::benchmark_data_smithi171_78918_object64790:head# 2024-04-20T17:11:20.659 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.659 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72260e8b:::benchmark_data_smithi171_78918_object159173:head# 2024-04-20T17:11:20.659 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.660 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7226b469:::benchmark_data_smithi171_78918_object142181:head# 2024-04-20T17:11:20.660 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.660 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7226edfb:::benchmark_data_smithi171_78918_object120520:head# 2024-04-20T17:11:20.660 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.660 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:722824db:::benchmark_data_smithi171_78918_object105356:head# 2024-04-20T17:11:20.660 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.660 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7228a128:::benchmark_data_smithi171_78918_object88972:head# 2024-04-20T17:11:20.660 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.660 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72292146:::benchmark_data_smithi171_78918_object94515:head# 2024-04-20T17:11:20.660 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.660 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72293b72:::benchmark_data_smithi171_78918_object83103:head# 2024-04-20T17:11:20.660 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.660 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7229cf99:::benchmark_data_smithi171_78918_object133584:head# 2024-04-20T17:11:20.660 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.661 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7229daf7:::benchmark_data_smithi171_78918_object115198:head# 2024-04-20T17:11:20.661 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.661 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:722a8bc2:::benchmark_data_smithi171_78918_object52510:head# 2024-04-20T17:11:20.661 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.661 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:722b2268:::benchmark_data_smithi171_78918_object93572:head# 2024-04-20T17:11:20.661 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.661 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:722b4014:::benchmark_data_smithi171_78918_object57633:head# 2024-04-20T17:11:20.661 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.661 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:722b97cb:::benchmark_data_smithi171_78918_object105829:head# 2024-04-20T17:11:20.661 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.661 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:722d8df4:::benchmark_data_smithi171_78918_object104064:head# 2024-04-20T17:11:20.661 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.661 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:722de1fb:::benchmark_data_smithi171_78918_object114633:head# 2024-04-20T17:11:20.661 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.662 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:722e434f:::benchmark_data_smithi171_78918_object143231:head# 2024-04-20T17:11:20.662 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.662 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:722fa89d:::benchmark_data_smithi171_78918_object148258:head# 2024-04-20T17:11:20.662 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.662 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7230bf58:::benchmark_data_smithi171_78918_object59895:head# 2024-04-20T17:11:20.662 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.662 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72310bc2:::benchmark_data_smithi171_78918_object101752:head# 2024-04-20T17:11:20.662 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.662 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72312ca5:::benchmark_data_smithi171_78918_object135341:head# 2024-04-20T17:11:20.662 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.662 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7232227d:::benchmark_data_smithi171_78918_object69993:head# 2024-04-20T17:11:20.662 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.662 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:723226fe:::benchmark_data_smithi171_78918_object156228:head# 2024-04-20T17:11:20.662 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.663 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72324d13:::benchmark_data_smithi171_78918_object122747:head# 2024-04-20T17:11:20.663 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.663 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72333e37:::benchmark_data_smithi171_78918_object149870:head# 2024-04-20T17:11:20.663 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.663 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7233d833:::benchmark_data_smithi171_78918_object157983:head# 2024-04-20T17:11:20.663 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.663 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7233ef65:::benchmark_data_smithi171_78918_object111090:head# 2024-04-20T17:11:20.663 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.663 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7233f0f8:::benchmark_data_smithi171_78918_object73166:head# 2024-04-20T17:11:20.663 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.663 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:723456aa:::benchmark_data_smithi171_78918_object130671:head# 2024-04-20T17:11:20.663 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.663 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7234f61c:::benchmark_data_smithi171_78918_object77609:head# 2024-04-20T17:11:20.663 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.664 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7234f706:::benchmark_data_smithi171_78918_object104849:head# 2024-04-20T17:11:20.664 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.664 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7235a1f3:::benchmark_data_smithi171_78918_object159872:head# 2024-04-20T17:11:20.664 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.664 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72367b29:::benchmark_data_smithi171_78918_object124791:head# 2024-04-20T17:11:20.664 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.664 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7237c25e:::benchmark_data_smithi171_78918_object79260:head# 2024-04-20T17:11:20.664 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.664 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72381da1:::benchmark_data_smithi171_78918_object59497:head# 2024-04-20T17:11:20.664 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.664 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72384175:::benchmark_data_smithi171_78918_object93986:head# 2024-04-20T17:11:20.664 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.664 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72385766:::benchmark_data_smithi171_78918_object143660:head# 2024-04-20T17:11:20.664 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.664 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72393f45:::benchmark_data_smithi171_78918_object70655:head# 2024-04-20T17:11:20.665 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.665 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:723b62ff:::benchmark_data_smithi171_78918_object70966:head# 2024-04-20T17:11:20.665 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.665 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:723b6b78:::benchmark_data_smithi171_78918_object64645:head# 2024-04-20T17:11:20.665 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.665 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:723b75f4:::benchmark_data_smithi171_78918_object150460:head# 2024-04-20T17:11:20.665 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.665 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:723ba4aa:::benchmark_data_smithi171_78918_object137723:head# 2024-04-20T17:11:20.665 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.665 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:723c0f50:::benchmark_data_smithi171_78918_object144454:head# 2024-04-20T17:11:20.665 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.665 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:723c872d:::benchmark_data_smithi171_78918_object67936:head# 2024-04-20T17:11:20.665 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.665 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:723ca525:::benchmark_data_smithi171_78918_object159115:head# 2024-04-20T17:11:20.666 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.666 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:723ce83d:::benchmark_data_smithi171_78918_object145357:head# 2024-04-20T17:11:20.666 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.666 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:723ceff0:::benchmark_data_smithi171_78918_object63732:head# 2024-04-20T17:11:20.666 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.666 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:723d00d2:::benchmark_data_smithi171_78918_object62151:head# 2024-04-20T17:11:20.666 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.666 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:723d9e48:::benchmark_data_smithi171_78918_object153975:head# 2024-04-20T17:11:20.666 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.666 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:723e0040:::benchmark_data_smithi171_78918_object129045:head# 2024-04-20T17:11:20.666 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.666 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:723e1cd4:::benchmark_data_smithi171_78918_object120288:head# 2024-04-20T17:11:20.666 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.666 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:723e3fec:::benchmark_data_smithi171_78918_object54015:head# 2024-04-20T17:11:20.666 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.667 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:723f29bd:::benchmark_data_smithi171_78918_object143519:head# 2024-04-20T17:11:20.667 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.667 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:723f63b4:::benchmark_data_smithi171_78918_object99668:head# 2024-04-20T17:11:20.667 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.667 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72402b72:::benchmark_data_smithi171_78918_object95226:head# 2024-04-20T17:11:20.667 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.667 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72407dd9:::benchmark_data_smithi171_78918_object59933:head# 2024-04-20T17:11:20.667 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.667 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7240a0a8:::benchmark_data_smithi171_78918_object55631:head# 2024-04-20T17:11:20.667 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.667 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72415603:::benchmark_data_smithi171_78918_object75815:head# 2024-04-20T17:11:20.667 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.667 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72435475:::benchmark_data_smithi171_78918_object70926:head# 2024-04-20T17:11:20.667 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.667 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7244ebf3:::benchmark_data_smithi171_78918_object80954:head# 2024-04-20T17:11:20.668 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.668 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72457fed:::benchmark_data_smithi171_78918_object56468:head# 2024-04-20T17:11:20.668 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.668 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72458aaf:::benchmark_data_smithi171_78918_object74998:head# 2024-04-20T17:11:20.668 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.668 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7245b86c:::benchmark_data_smithi171_78918_object83789:head# 2024-04-20T17:11:20.668 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.668 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72469e9a:::benchmark_data_smithi171_78918_object66479:head# 2024-04-20T17:11:20.668 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.668 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7247ab9a:::benchmark_data_smithi171_78918_object160231:head# 2024-04-20T17:11:20.668 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.668 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7247b299:::benchmark_data_smithi171_78918_object113273:head# 2024-04-20T17:11:20.668 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.668 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72491e0d:::benchmark_data_smithi171_78918_object149897:head# 2024-04-20T17:11:20.669 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.669 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7249c838:::benchmark_data_smithi171_78918_object81334:head# 2024-04-20T17:11:20.669 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.669 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7249ccf4:::benchmark_data_smithi171_78918_object142184:head# 2024-04-20T17:11:20.669 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.669 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:724a1568:::benchmark_data_smithi171_78918_object160935:head# 2024-04-20T17:11:20.669 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.669 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:724a607c:::benchmark_data_smithi171_78918_object56526:head# 2024-04-20T17:11:20.669 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.669 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:724ac229:::benchmark_data_smithi171_78918_object68365:head# 2024-04-20T17:11:20.669 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.669 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:724b00e5:::benchmark_data_smithi171_78918_object139646:head# 2024-04-20T17:11:20.669 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.669 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:724b091e:::benchmark_data_smithi171_78918_object55040:head# 2024-04-20T17:11:20.669 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.670 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:724ceffd:::benchmark_data_smithi171_78918_object80204:head# 2024-04-20T17:11:20.670 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.670 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:724d1661:::benchmark_data_smithi171_78918_object150235:head# 2024-04-20T17:11:20.670 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.670 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:724d4ead:::benchmark_data_smithi171_78918_object122054:head# 2024-04-20T17:11:20.670 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.670 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:724d92b9:::benchmark_data_smithi171_78918_object62582:head# 2024-04-20T17:11:20.670 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.670 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:724d94ce:::benchmark_data_smithi171_78918_object82914:head# 2024-04-20T17:11:20.670 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.670 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:724df54c:::benchmark_data_smithi171_78918_object84918:head# 2024-04-20T17:11:20.670 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.670 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:724e8d16:::benchmark_data_smithi171_78918_object122089:head# 2024-04-20T17:11:20.671 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.671 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:724f0844:::benchmark_data_smithi171_78918_object136836:head# 2024-04-20T17:11:20.671 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.671 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:724fc0bc:::benchmark_data_smithi171_78918_object80596:head# 2024-04-20T17:11:20.671 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.671 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72502700:::benchmark_data_smithi171_78918_object90242:head# 2024-04-20T17:11:20.671 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.671 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7250db1a:::benchmark_data_smithi171_78918_object140520:head# 2024-04-20T17:11:20.671 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.671 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72517694:::benchmark_data_smithi171_78918_object129520:head# 2024-04-20T17:11:20.671 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.671 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7251d92f:::benchmark_data_smithi171_78918_object130750:head# 2024-04-20T17:11:20.671 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.671 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72525417:::benchmark_data_smithi171_78918_object116331:head# 2024-04-20T17:11:20.671 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.672 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7252878a:::benchmark_data_smithi171_78918_object94259:head# 2024-04-20T17:11:20.672 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.672 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7253ec42:::benchmark_data_smithi171_78918_object104175:head# 2024-04-20T17:11:20.672 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.672 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:725468e5:::benchmark_data_smithi171_78918_object146200:head# 2024-04-20T17:11:20.672 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.672 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72553881:::benchmark_data_smithi171_78918_object125761:head# 2024-04-20T17:11:20.672 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.672 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72553d2d:::benchmark_data_smithi171_78918_object128579:head# 2024-04-20T17:11:20.672 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.672 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7255d513:::benchmark_data_smithi171_78918_object133959:head# 2024-04-20T17:11:20.672 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.672 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7255ddef:::benchmark_data_smithi171_78918_object120774:head# 2024-04-20T17:11:20.672 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.673 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7255ee52:::benchmark_data_smithi171_78918_object60836:head# 2024-04-20T17:11:20.673 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.673 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:725616e1:::benchmark_data_smithi171_78918_object118876:head# 2024-04-20T17:11:20.673 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.673 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7256afa3:::benchmark_data_smithi171_78918_object88588:head# 2024-04-20T17:11:20.673 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.673 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7258eb9f:::benchmark_data_smithi171_78918_object154265:head# 2024-04-20T17:11:20.673 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.673 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72596f9f:::benchmark_data_smithi171_78918_object65524:head# 2024-04-20T17:11:20.673 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.673 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:725a7eac:::benchmark_data_smithi171_78918_object72983:head# 2024-04-20T17:11:20.673 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.673 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:725b7891:::benchmark_data_smithi171_78918_object71854:head# 2024-04-20T17:11:20.673 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.673 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:725bfaea:::benchmark_data_smithi171_78918_object149821:head# 2024-04-20T17:11:20.674 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.674 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:725d06a8:::benchmark_data_smithi171_78918_object73178:head# 2024-04-20T17:11:20.674 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.674 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:725e64ef:::benchmark_data_smithi171_78918_object76277:head# 2024-04-20T17:11:20.674 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.674 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:725f1177:::benchmark_data_smithi171_78918_object110633:head# 2024-04-20T17:11:20.674 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.674 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:725f9eb3:::benchmark_data_smithi171_78918_object79127:head# 2024-04-20T17:11:20.674 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.674 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:725fd83b:::benchmark_data_smithi171_78918_object119006:head# 2024-04-20T17:11:20.674 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.674 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:725ffe2d:::benchmark_data_smithi171_78918_object132767:head# 2024-04-20T17:11:20.674 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.674 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7260409d:::benchmark_data_smithi171_78918_object96875:head# 2024-04-20T17:11:20.675 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.675 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72614020:::benchmark_data_smithi171_78918_object68553:head# 2024-04-20T17:11:20.675 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.675 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7261a39b:::benchmark_data_smithi171_78918_object76278:head# 2024-04-20T17:11:20.675 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.675 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72629f83:::benchmark_data_smithi171_78918_object102718:head# 2024-04-20T17:11:20.675 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.675 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:726339cb:::benchmark_data_smithi171_78918_object77637:head# 2024-04-20T17:11:20.675 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.675 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7263be89:::benchmark_data_smithi171_78918_object83483:head# 2024-04-20T17:11:20.675 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.675 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7265206d:::benchmark_data_smithi171_78918_object156240:head# 2024-04-20T17:11:20.675 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.675 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72663ae9:::benchmark_data_smithi171_78918_object96626:head# 2024-04-20T17:11:20.676 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.676 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:726860c6:::benchmark_data_smithi171_78918_object97945:head# 2024-04-20T17:11:20.676 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.676 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:726a09b1:::benchmark_data_smithi171_78918_object136576:head# 2024-04-20T17:11:20.676 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.676 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:726b161e:::benchmark_data_smithi171_78918_object141698:head# 2024-04-20T17:11:20.676 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.676 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:726c4533:::benchmark_data_smithi171_78918_object64115:head# 2024-04-20T17:11:20.676 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.676 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:726cd40d:::benchmark_data_smithi171_78918_object117828:head# 2024-04-20T17:11:20.676 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.676 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:726d2a15:::benchmark_data_smithi171_78918_object138629:head# 2024-04-20T17:11:20.676 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.676 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:726d2c9a:::benchmark_data_smithi171_78918_object108913:head# 2024-04-20T17:11:20.677 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.677 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:726e6fc1:::benchmark_data_smithi171_78918_object159146:head# 2024-04-20T17:11:20.677 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.677 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:726eb95a:::benchmark_data_smithi171_78918_object53611:head# 2024-04-20T17:11:20.677 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.677 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:726ec7ad:::benchmark_data_smithi171_78918_object76040:head# 2024-04-20T17:11:20.677 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.677 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:726f513b:::benchmark_data_smithi171_78918_object125558:head# 2024-04-20T17:11:20.677 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.677 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:726f79b9:::benchmark_data_smithi171_78918_object108494:head# 2024-04-20T17:11:20.677 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.677 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:726fe853:::benchmark_data_smithi171_78918_object93633:head# 2024-04-20T17:11:20.677 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.677 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7270177d:::benchmark_data_smithi171_78918_object124994:head# 2024-04-20T17:11:20.678 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.678 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7270eece:::benchmark_data_smithi171_78918_object66350:head# 2024-04-20T17:11:20.678 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.678 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7271b1e4:::benchmark_data_smithi171_78918_object119945:head# 2024-04-20T17:11:20.678 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.678 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:727340f2:::benchmark_data_smithi171_78918_object127221:head# 2024-04-20T17:11:20.678 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.678 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72738d4d:::benchmark_data_smithi171_78918_object110763:head# 2024-04-20T17:11:20.678 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.678 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7273c7e2:::benchmark_data_smithi171_78918_object102508:head# 2024-04-20T17:11:20.678 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.678 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7274a6a3:::benchmark_data_smithi171_78918_object120104:head# 2024-04-20T17:11:20.679 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.679 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7274e51c:::benchmark_data_smithi171_78918_object59053:head# 2024-04-20T17:11:20.679 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.679 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72758a51:::benchmark_data_smithi171_78918_object125957:head# 2024-04-20T17:11:20.679 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.679 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:727667af:::benchmark_data_smithi171_78918_object130266:head# 2024-04-20T17:11:20.679 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.679 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7276b188:::benchmark_data_smithi171_78918_object137179:head# 2024-04-20T17:11:20.679 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.679 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:727796a6:::benchmark_data_smithi171_78918_object118095:head# 2024-04-20T17:11:20.679 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.679 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72787e03:::benchmark_data_smithi171_78918_object115510:head# 2024-04-20T17:11:20.679 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.679 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7279a177:::benchmark_data_smithi171_78918_object63985:head# 2024-04-20T17:11:20.680 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.680 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7279e431:::benchmark_data_smithi171_78918_object96358:head# 2024-04-20T17:11:20.680 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.680 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:727a0de1:::benchmark_data_smithi171_78918_object157675:head# 2024-04-20T17:11:20.680 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.680 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:727b8af0:::benchmark_data_smithi171_78918_object160377:head# 2024-04-20T17:11:20.680 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.680 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:727b8dc2:::benchmark_data_smithi171_78918_object115524:head# 2024-04-20T17:11:20.680 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.680 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:727bc1d0:::benchmark_data_smithi171_78918_object103648:head# 2024-04-20T17:11:20.680 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.680 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:727bfde5:::benchmark_data_smithi171_78918_object121482:head# 2024-04-20T17:11:20.680 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.680 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:727c7786:::benchmark_data_smithi171_78918_object156807:head# 2024-04-20T17:11:20.681 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.681 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:727cd513:::benchmark_data_smithi171_78918_object108305:head# 2024-04-20T17:11:20.681 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.681 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:727d4f05:::benchmark_data_smithi171_78918_object128561:head# 2024-04-20T17:11:20.681 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.681 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:727e04b6:::benchmark_data_smithi171_78918_object129241:head# 2024-04-20T17:11:20.681 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.681 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:727e19d8:::benchmark_data_smithi171_78918_object120275:head# 2024-04-20T17:11:20.681 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.681 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:727e39be:::benchmark_data_smithi171_78918_object160182:head# 2024-04-20T17:11:20.681 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.681 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:727e6009:::benchmark_data_smithi171_78918_object159379:head# 2024-04-20T17:11:20.681 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.681 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:727ea785:::benchmark_data_smithi171_78918_object129017:head# 2024-04-20T17:11:20.681 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.682 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:727f5636:::benchmark_data_smithi171_78918_object76580:head# 2024-04-20T17:11:20.682 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.682 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:728097dc:::benchmark_data_smithi171_78918_object127551:head# 2024-04-20T17:11:20.682 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.682 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:728114ed:::benchmark_data_smithi171_78918_object155877:head# 2024-04-20T17:11:20.682 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.682 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:728195a6:::benchmark_data_smithi171_78918_object83459:head# 2024-04-20T17:11:20.682 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.682 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:728223d9:::benchmark_data_smithi171_78918_object160175:head# 2024-04-20T17:11:20.682 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.682 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72823666:::benchmark_data_smithi171_78918_object139449:head# 2024-04-20T17:11:20.682 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.682 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:728272b8:::benchmark_data_smithi171_78918_object55598:head# 2024-04-20T17:11:20.683 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.683 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72836dd4:::benchmark_data_smithi171_78918_object73107:head# 2024-04-20T17:11:20.683 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.683 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7283ccf0:::benchmark_data_smithi171_78918_object67259:head# 2024-04-20T17:11:20.683 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.683 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7283ee19:::benchmark_data_smithi171_78918_object141072:head# 2024-04-20T17:11:20.683 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.683 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7283efdf:::benchmark_data_smithi171_78918_object104077:head# 2024-04-20T17:11:20.683 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.683 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72840220:::benchmark_data_smithi171_78918_object114933:head# 2024-04-20T17:11:20.683 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.683 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72847ad3:::benchmark_data_smithi171_78918_object59353:head# 2024-04-20T17:11:20.683 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.683 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72852496:::benchmark_data_smithi171_78918_object71089:head# 2024-04-20T17:11:20.683 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.684 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7287d50f:::benchmark_data_smithi171_78918_object144527:head# 2024-04-20T17:11:20.684 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.684 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72880a5e:::benchmark_data_smithi171_78918_object57235:head# 2024-04-20T17:11:20.684 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.684 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72880e58:::benchmark_data_smithi171_78918_object122565:head# 2024-04-20T17:11:20.684 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.684 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:728810db:::benchmark_data_smithi171_78918_object68079:head# 2024-04-20T17:11:20.684 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.684 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72886bf4:::benchmark_data_smithi171_78918_object141631:head# 2024-04-20T17:11:20.684 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.684 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7288bc61:::benchmark_data_smithi171_78918_object86199:head# 2024-04-20T17:11:20.684 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.684 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72892bdb:::benchmark_data_smithi171_78918_object111324:head# 2024-04-20T17:11:20.684 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.684 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7289c783:::benchmark_data_smithi171_78918_object119351:head# 2024-04-20T17:11:20.685 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.685 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:728b0af0:::benchmark_data_smithi171_78918_object130269:head# 2024-04-20T17:11:20.685 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.685 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:728b0b26:::benchmark_data_smithi171_78918_object95630:head# 2024-04-20T17:11:20.685 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.685 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:728b94f9:::benchmark_data_smithi171_78918_object68343:head# 2024-04-20T17:11:20.685 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.685 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:728bb3f4:::benchmark_data_smithi171_78918_object110857:head# 2024-04-20T17:11:20.685 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.685 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:728c44c5:::benchmark_data_smithi171_78918_object126961:head# 2024-04-20T17:11:20.685 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.685 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:728cf480:::benchmark_data_smithi171_78918_object158776:head# 2024-04-20T17:11:20.685 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.685 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:728d5d54:::benchmark_data_smithi171_78918_object127607:head# 2024-04-20T17:11:20.685 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.686 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:728db7a2:::benchmark_data_smithi171_78918_object87744:head# 2024-04-20T17:11:20.686 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.686 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:728e2e95:::benchmark_data_smithi171_78918_object89626:head# 2024-04-20T17:11:20.686 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.686 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:728f42df:::benchmark_data_smithi171_78918_object62491:head# 2024-04-20T17:11:20.686 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.686 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:728f5b77:::benchmark_data_smithi171_78918_object122837:head# 2024-04-20T17:11:20.686 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.686 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:728fdff1:::benchmark_data_smithi171_78918_object104970:head# 2024-04-20T17:11:20.686 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.686 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7290304f:::benchmark_data_smithi171_78918_object73879:head# 2024-04-20T17:11:20.686 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.686 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7291add7:::benchmark_data_smithi171_78918_object105164:head# 2024-04-20T17:11:20.686 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.687 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:729215ad:::benchmark_data_smithi171_78918_object93536:head# 2024-04-20T17:11:20.687 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.687 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72932ef7:::benchmark_data_smithi171_78918_object61108:head# 2024-04-20T17:11:20.687 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.687 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72933161:::benchmark_data_smithi171_78918_object158683:head# 2024-04-20T17:11:20.687 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.687 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72944392:::benchmark_data_smithi171_78918_object70057:head# 2024-04-20T17:11:20.687 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.687 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7294b554:::benchmark_data_smithi171_78918_object54947:head# 2024-04-20T17:11:20.687 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.687 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7294cebe:::benchmark_data_smithi171_78918_object101391:head# 2024-04-20T17:11:20.687 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.687 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72950266:::benchmark_data_smithi171_78918_object160566:head# 2024-04-20T17:11:20.687 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.687 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72955856:::benchmark_data_smithi171_78918_object117172:head# 2024-04-20T17:11:20.688 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.688 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72970965:::benchmark_data_smithi171_78918_object101666:head# 2024-04-20T17:11:20.688 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.688 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72976361:::benchmark_data_smithi171_78918_object55914:head# 2024-04-20T17:11:20.688 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.688 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72988258:::benchmark_data_smithi171_78918_object145220:head# 2024-04-20T17:11:20.688 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.688 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72989d3d:::benchmark_data_smithi171_78918_object81225:head# 2024-04-20T17:11:20.688 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.688 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72990b71:::benchmark_data_smithi171_78918_object151624:head# 2024-04-20T17:11:20.688 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.688 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7299381d:::benchmark_data_smithi171_78918_object69896:head# 2024-04-20T17:11:20.688 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.689 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:729acf53:::benchmark_data_smithi171_78918_object71269:head# 2024-04-20T17:11:20.689 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.689 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:729bb5f2:::benchmark_data_smithi171_78918_object74961:head# 2024-04-20T17:11:20.689 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.689 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:729c0581:::benchmark_data_smithi171_78918_object54491:head# 2024-04-20T17:11:20.689 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.689 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:729ca71f:::benchmark_data_smithi171_78918_object56343:head# 2024-04-20T17:11:20.689 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.689 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:729d2c72:::benchmark_data_smithi171_78918_object111085:head# 2024-04-20T17:11:20.689 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.689 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:729d7e36:::benchmark_data_smithi171_78918_object135996:head# 2024-04-20T17:11:20.689 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.689 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:729ea3aa:::benchmark_data_smithi171_78918_object61322:head# 2024-04-20T17:11:20.689 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.690 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72a0a6de:::benchmark_data_smithi171_78918_object123196:head# 2024-04-20T17:11:20.690 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.690 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72a0b5eb:::benchmark_data_smithi171_78918_object136497:head# 2024-04-20T17:11:20.690 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.690 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72a0bb09:::benchmark_data_smithi171_78918_object130250:head# 2024-04-20T17:11:20.690 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.690 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72a1093b:::benchmark_data_smithi171_78918_object118122:head# 2024-04-20T17:11:20.690 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.690 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72a1f273:::benchmark_data_smithi171_78918_object118302:head# 2024-04-20T17:11:20.690 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.690 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72a33681:::benchmark_data_smithi171_78918_object143155:head# 2024-04-20T17:11:20.690 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.690 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72a35796:::benchmark_data_smithi171_78918_object147075:head# 2024-04-20T17:11:20.690 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.691 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72a359c9:::benchmark_data_smithi171_78918_object104127:head# 2024-04-20T17:11:20.691 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.691 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72a4b8a5:::benchmark_data_smithi171_78918_object57088:head# 2024-04-20T17:11:20.691 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.691 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72a4fe82:::benchmark_data_smithi171_78918_object98538:head# 2024-04-20T17:11:20.691 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.691 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72a50362:::benchmark_data_smithi171_78918_object114123:head# 2024-04-20T17:11:20.691 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.691 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72a5d7d5:::benchmark_data_smithi171_78918_object101176:head# 2024-04-20T17:11:20.691 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.691 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72a640fe:::benchmark_data_smithi171_78918_object88085:head# 2024-04-20T17:11:20.691 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.691 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72a6dfb3:::benchmark_data_smithi171_78918_object122948:head# 2024-04-20T17:11:20.692 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.692 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72a7372c:::benchmark_data_smithi171_78918_object122517:head# 2024-04-20T17:11:20.692 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.692 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72a79fb6:::benchmark_data_smithi171_78918_object58370:head# 2024-04-20T17:11:20.692 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.692 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72a85389:::benchmark_data_smithi171_78918_object68002:head# 2024-04-20T17:11:20.692 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.692 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72a89c5f:::benchmark_data_smithi171_78918_object141107:head# 2024-04-20T17:11:20.692 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.692 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72a8bb3a:::benchmark_data_smithi171_78918_object156949:head# 2024-04-20T17:11:20.692 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.692 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72a8f445:::benchmark_data_smithi171_78918_object96912:head# 2024-04-20T17:11:20.692 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.692 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72a9ca71:::benchmark_data_smithi171_78918_object135430:head# 2024-04-20T17:11:20.692 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.693 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72ab056c:::benchmark_data_smithi171_78918_object118537:head# 2024-04-20T17:11:20.693 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.693 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72abe8a8:::benchmark_data_smithi171_78918_object104351:head# 2024-04-20T17:11:20.693 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.693 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72abece7:::benchmark_data_smithi171_78918_object114653:head# 2024-04-20T17:11:20.693 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.693 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72acb1bd:::benchmark_data_smithi171_78918_object79173:head# 2024-04-20T17:11:20.693 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.693 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72afb955:::benchmark_data_smithi171_78918_object108609:head# 2024-04-20T17:11:20.693 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.693 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72aff883:::benchmark_data_smithi171_78918_object132440:head# 2024-04-20T17:11:20.693 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.693 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72b0fb87:::benchmark_data_smithi171_78918_object139556:head# 2024-04-20T17:11:20.693 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.693 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72b1b618:::benchmark_data_smithi171_78918_object123821:head# 2024-04-20T17:11:20.694 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.694 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72b211a2:::benchmark_data_smithi171_78918_object55423:head# 2024-04-20T17:11:20.694 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.694 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72b2cf83:::benchmark_data_smithi171_78918_object155005:head# 2024-04-20T17:11:20.694 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.694 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72b3c365:::benchmark_data_smithi171_78918_object97227:head# 2024-04-20T17:11:20.694 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.694 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72b3ca42:::benchmark_data_smithi171_78918_object127634:head# 2024-04-20T17:11:20.694 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.694 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72b3f38e:::benchmark_data_smithi171_78918_object142858:head# 2024-04-20T17:11:20.694 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.694 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72b4d287:::benchmark_data_smithi171_78918_object67143:head# 2024-04-20T17:11:20.694 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.694 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72b53f2e:::benchmark_data_smithi171_78918_object54815:head# 2024-04-20T17:11:20.695 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.695 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72b55974:::benchmark_data_smithi171_78918_object86965:head# 2024-04-20T17:11:20.695 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.695 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72b63372:::benchmark_data_smithi171_78918_object110610:head# 2024-04-20T17:11:20.695 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.695 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72b6f02e:::benchmark_data_smithi171_78918_object122070:head# 2024-04-20T17:11:20.695 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.695 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72b83a72:::benchmark_data_smithi171_78918_object138546:head# 2024-04-20T17:11:20.695 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.695 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72b849c8:::benchmark_data_smithi171_78918_object159837:head# 2024-04-20T17:11:20.695 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.695 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72b909dd:::benchmark_data_smithi171_78918_object86912:head# 2024-04-20T17:11:20.695 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.696 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72b98bed:::benchmark_data_smithi171_78918_object123414:head# 2024-04-20T17:11:20.696 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.696 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72ba003a:::benchmark_data_smithi171_78918_object122308:head# 2024-04-20T17:11:20.696 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.696 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72ba0acf:::benchmark_data_smithi171_78918_object132025:head# 2024-04-20T17:11:20.696 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.696 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72ba4434:::benchmark_data_smithi171_78918_object53859:head# 2024-04-20T17:11:20.696 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.696 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72bb040f:::benchmark_data_smithi171_78918_object76380:head# 2024-04-20T17:11:20.696 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.696 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72bb1b62:::benchmark_data_smithi171_78918_object111887:head# 2024-04-20T17:11:20.696 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.696 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72bbdd4a:::benchmark_data_smithi171_78918_object79185:head# 2024-04-20T17:11:20.696 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.697 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72bbf0a8:::benchmark_data_smithi171_78918_object67688:head# 2024-04-20T17:11:20.697 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.697 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72bc3fbd:::benchmark_data_smithi171_78918_object90502:head# 2024-04-20T17:11:20.697 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.697 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72bcd598:::benchmark_data_smithi171_78918_object97991:head# 2024-04-20T17:11:20.697 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.697 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72bcf3af:::benchmark_data_smithi171_78918_object133688:head# 2024-04-20T17:11:20.697 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.697 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72bd3845:::benchmark_data_smithi171_78918_object75394:head# 2024-04-20T17:11:20.697 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.697 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72bd9ed8:::benchmark_data_smithi171_78918_object82068:head# 2024-04-20T17:11:20.697 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.697 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72becce9:::benchmark_data_smithi171_78918_object111030:head# 2024-04-20T17:11:20.697 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.697 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72bf70dc:::benchmark_data_smithi171_78918_object139271:head# 2024-04-20T17:11:20.698 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.698 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72bfb760:::benchmark_data_smithi171_78918_object112217:head# 2024-04-20T17:11:20.698 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.698 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72bfed41:::benchmark_data_smithi171_78918_object54523:head# 2024-04-20T17:11:20.698 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.698 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72c0adfb:::benchmark_data_smithi171_78918_object60693:head# 2024-04-20T17:11:20.698 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.698 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72c1126b:::benchmark_data_smithi171_78918_object73976:head# 2024-04-20T17:11:20.698 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.698 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72c13c90:::benchmark_data_smithi171_78918_object117241:head# 2024-04-20T17:11:20.698 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.698 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72c30fe5:::benchmark_data_smithi171_78918_object83009:head# 2024-04-20T17:11:20.698 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.698 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72c3af9f:::benchmark_data_smithi171_78918_object136707:head# 2024-04-20T17:11:20.699 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.699 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72c5d6a4:::benchmark_data_smithi171_78918_object104688:head# 2024-04-20T17:11:20.699 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.699 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72c5fce5:::benchmark_data_smithi171_78918_object117692:head# 2024-04-20T17:11:20.699 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.699 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72c712df:::benchmark_data_smithi171_78918_object80046:head# 2024-04-20T17:11:20.699 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.699 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72c72918:::benchmark_data_smithi171_78918_object80774:head# 2024-04-20T17:11:20.699 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.699 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72c73e47:::benchmark_data_smithi171_78918_object155212:head# 2024-04-20T17:11:20.699 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.699 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72c80a93:::benchmark_data_smithi171_78918_object118433:head# 2024-04-20T17:11:20.699 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.699 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72c93063:::benchmark_data_smithi171_78918_object72724:head# 2024-04-20T17:11:20.699 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.700 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72c98bf0:::benchmark_data_smithi171_78918_object87390:head# 2024-04-20T17:11:20.700 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.700 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72c99eea:::benchmark_data_smithi171_78918_object128550:head# 2024-04-20T17:11:20.700 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.700 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72c9a08d:::benchmark_data_smithi171_78918_object121428:head# 2024-04-20T17:11:20.700 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.700 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72ca43c5:::benchmark_data_smithi171_78918_object75927:head# 2024-04-20T17:11:20.700 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.700 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72ca6ac1:::benchmark_data_smithi171_78918_object107860:head# 2024-04-20T17:11:20.700 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.700 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72cb5db3:::benchmark_data_smithi171_78918_object129075:head# 2024-04-20T17:11:20.700 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.700 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72ccd005:::benchmark_data_smithi171_78918_object51456:head# 2024-04-20T17:11:20.700 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.700 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72cd2df5:::benchmark_data_smithi171_78918_object129186:head# 2024-04-20T17:11:20.701 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.701 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72ced765:::benchmark_data_smithi171_78918_object134086:head# 2024-04-20T17:11:20.701 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.701 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72cf6268:::benchmark_data_smithi171_78918_object101185:head# 2024-04-20T17:11:20.701 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.701 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72cfcde2:::benchmark_data_smithi171_78918_object147000:head# 2024-04-20T17:11:20.701 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.701 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72d00b8c:::benchmark_data_smithi171_78918_object51452:head# 2024-04-20T17:11:20.701 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.701 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72d09203:::benchmark_data_smithi171_78918_object149742:head# 2024-04-20T17:11:20.701 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.701 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72d09543:::benchmark_data_smithi171_78918_object97397:head# 2024-04-20T17:11:20.701 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.701 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72d0fb16:::benchmark_data_smithi171_78918_object107787:head# 2024-04-20T17:11:20.702 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.702 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72d12cfc:::benchmark_data_smithi171_78918_object117944:head# 2024-04-20T17:11:20.702 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.702 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72d272f3:::benchmark_data_smithi171_78918_object156082:head# 2024-04-20T17:11:20.702 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.702 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72d3cb09:::benchmark_data_smithi171_78918_object148333:head# 2024-04-20T17:11:20.702 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.702 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72d45d5d:::benchmark_data_smithi171_78918_object123176:head# 2024-04-20T17:11:20.702 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.702 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72d5109d:::benchmark_data_smithi171_78918_object159054:head# 2024-04-20T17:11:20.702 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.702 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72d5124b:::benchmark_data_smithi171_78918_object113876:head# 2024-04-20T17:11:20.702 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.703 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72d5b074:::benchmark_data_smithi171_78918_object109748:head# 2024-04-20T17:11:20.703 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.703 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72d6713b:::benchmark_data_smithi171_78918_object131663:head# 2024-04-20T17:11:20.703 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.703 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72d70c3e:::benchmark_data_smithi171_78918_object58960:head# 2024-04-20T17:11:20.703 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.703 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72d73571:::benchmark_data_smithi171_78918_object106769:head# 2024-04-20T17:11:20.703 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.703 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72d75f82:::benchmark_data_smithi171_78918_object64389:head# 2024-04-20T17:11:20.703 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.703 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72d784b7:::benchmark_data_smithi171_78918_object70069:head# 2024-04-20T17:11:20.703 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.703 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72d839da:::benchmark_data_smithi171_78918_object69557:head# 2024-04-20T17:11:20.703 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.703 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72d8c84f:::benchmark_data_smithi171_78918_object150826:head# 2024-04-20T17:11:20.704 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.763 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72d93cb3:::benchmark_data_smithi171_78918_object119379:head# 2024-04-20T17:11:20.763 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.764 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72d97ee1:::benchmark_data_smithi171_78918_object73629:head# 2024-04-20T17:11:20.764 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.764 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72d9b55d:::benchmark_data_smithi171_78918_object76928:head# 2024-04-20T17:11:20.764 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.764 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72dc0c3f:::benchmark_data_smithi171_78918_object76022:head# 2024-04-20T17:11:20.764 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.764 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72dc5f38:::benchmark_data_smithi171_78918_object104913:head# 2024-04-20T17:11:20.764 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.764 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72ddd330:::benchmark_data_smithi171_78918_object99935:head# 2024-04-20T17:11:20.764 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.764 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72de2110:::benchmark_data_smithi171_78918_object126845:head# 2024-04-20T17:11:20.764 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.764 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72de4812:::benchmark_data_smithi171_78918_object82592:head# 2024-04-20T17:11:20.764 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.765 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72de7ca8:::benchmark_data_smithi171_78918_object56085:head# 2024-04-20T17:11:20.765 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.765 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72df0b53:::benchmark_data_smithi171_78918_object134005:head# 2024-04-20T17:11:20.765 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.765 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72dfa472:::benchmark_data_smithi171_78918_object147166:head# 2024-04-20T17:11:20.765 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.765 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72e03efc:::benchmark_data_smithi171_78918_object141695:head# 2024-04-20T17:11:20.765 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.765 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72e1512e:::benchmark_data_smithi171_78918_object84985:head# 2024-04-20T17:11:20.765 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.765 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72e240a8:::benchmark_data_smithi171_78918_object154514:head# 2024-04-20T17:11:20.765 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.765 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72e30fd9:::benchmark_data_smithi171_78918_object118524:head# 2024-04-20T17:11:20.765 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.765 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72e39b95:::benchmark_data_smithi171_78918_object123458:head# 2024-04-20T17:11:20.766 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.766 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72e48e31:::benchmark_data_smithi171_78918_object99464:head# 2024-04-20T17:11:20.766 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.766 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72e550e6:::benchmark_data_smithi171_78918_object132064:head# 2024-04-20T17:11:20.766 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.766 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72e57c93:::benchmark_data_smithi171_78918_object89481:head# 2024-04-20T17:11:20.766 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.766 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72e580f6:::benchmark_data_smithi171_78918_object148687:head# 2024-04-20T17:11:20.766 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.766 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72e59554:::benchmark_data_smithi171_78918_object69728:head# 2024-04-20T17:11:20.766 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.766 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72e59a15:::benchmark_data_smithi171_78918_object138120:head# 2024-04-20T17:11:20.766 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.766 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72e705d9:::benchmark_data_smithi171_78918_object72839:head# 2024-04-20T17:11:20.767 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.767 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72e745d3:::benchmark_data_smithi171_78918_object127355:head# 2024-04-20T17:11:20.767 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.767 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72e7ad9c:::benchmark_data_smithi171_78918_object109445:head# 2024-04-20T17:11:20.767 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.767 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72e7c8b0:::benchmark_data_smithi171_78918_object122480:head# 2024-04-20T17:11:20.767 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.767 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72e7de25:::benchmark_data_smithi171_78918_object64191:head# 2024-04-20T17:11:20.767 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.767 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72e822b7:::benchmark_data_smithi171_78918_object158711:head# 2024-04-20T17:11:20.767 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.767 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72e88075:::benchmark_data_smithi171_78918_object137155:head# 2024-04-20T17:11:20.767 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.767 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72e9848b:::benchmark_data_smithi171_78918_object103486:head# 2024-04-20T17:11:20.767 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.768 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72ea09f3:::benchmark_data_smithi171_78918_object84019:head# 2024-04-20T17:11:20.768 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.768 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72ea461b:::benchmark_data_smithi171_78918_object130086:head# 2024-04-20T17:11:20.768 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.768 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72ea66af:::benchmark_data_smithi171_78918_object143179:head# 2024-04-20T17:11:20.768 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.768 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72eb458b:::benchmark_data_smithi171_78918_object109798:head# 2024-04-20T17:11:20.768 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.768 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72eb551d:::benchmark_data_smithi171_78918_object136697:head# 2024-04-20T17:11:20.768 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.768 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72eb6667:::benchmark_data_smithi171_78918_object160952:head# 2024-04-20T17:11:20.768 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.768 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72eb6ab0:::benchmark_data_smithi171_78918_object128184:head# 2024-04-20T17:11:20.768 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.769 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72ec0b70:::benchmark_data_smithi171_78918_object71635:head# 2024-04-20T17:11:20.769 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.769 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72ec9c10:::benchmark_data_smithi171_78918_object57275:head# 2024-04-20T17:11:20.769 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.769 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72ed2a31:::benchmark_data_smithi171_78918_object129881:head# 2024-04-20T17:11:20.769 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.769 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72edc490:::benchmark_data_smithi171_78918_object120778:head# 2024-04-20T17:11:20.769 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.769 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72ee1b26:::benchmark_data_smithi171_78918_object131573:head# 2024-04-20T17:11:20.769 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.769 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72ee8d95:::benchmark_data_smithi171_78918_object139794:head# 2024-04-20T17:11:20.769 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.769 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72eeab4e:::benchmark_data_smithi171_78918_object75445:head# 2024-04-20T17:11:20.769 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.769 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72eed0b3:::benchmark_data_smithi171_78918_object147122:head# 2024-04-20T17:11:20.770 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.770 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72eed494:::benchmark_data_smithi171_78918_object115495:head# 2024-04-20T17:11:20.770 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.770 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72ef09b9:::benchmark_data_smithi171_78918_object83724:head# 2024-04-20T17:11:20.770 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.770 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72ef6e01:::benchmark_data_smithi171_78918_object116691:head# 2024-04-20T17:11:20.770 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.770 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72ef9dd8:::benchmark_data_smithi171_78918_object86285:head# 2024-04-20T17:11:20.770 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.770 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72f0111f:::benchmark_data_smithi171_78918_object143094:head# 2024-04-20T17:11:20.770 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.770 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72f037cd:::benchmark_data_smithi171_78918_object64314:head# 2024-04-20T17:11:20.770 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.770 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72f05231:::benchmark_data_smithi171_78918_object62507:head# 2024-04-20T17:11:20.770 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.771 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72f05717:::benchmark_data_smithi171_78918_object147317:head# 2024-04-20T17:11:20.771 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.771 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72f16e03:::benchmark_data_smithi171_78918_object114189:head# 2024-04-20T17:11:20.771 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.771 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72f1df68:::benchmark_data_smithi171_78918_object68227:head# 2024-04-20T17:11:20.771 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.771 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72f246a4:::benchmark_data_smithi171_78918_object118860:head# 2024-04-20T17:11:20.771 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.771 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72f291c8:::benchmark_data_smithi171_78918_object144201:head# 2024-04-20T17:11:20.771 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.771 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72f2d3d6:::benchmark_data_smithi171_78918_object104617:head# 2024-04-20T17:11:20.771 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.771 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72f2ea09:::benchmark_data_smithi171_78918_object146174:head# 2024-04-20T17:11:20.771 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.772 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72f3e163:::benchmark_data_smithi171_78918_object135396:head# 2024-04-20T17:11:20.772 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.772 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72f46221:::benchmark_data_smithi171_78918_object116513:head# 2024-04-20T17:11:20.772 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.772 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72f4c798:::benchmark_data_smithi171_78918_object112751:head# 2024-04-20T17:11:20.772 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.772 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72f5243e:::benchmark_data_smithi171_78918_object124570:head# 2024-04-20T17:11:20.772 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.772 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72f55d44:::benchmark_data_smithi171_78918_object149698:head# 2024-04-20T17:11:20.772 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.772 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72f56c07:::benchmark_data_smithi171_78918_object86375:head# 2024-04-20T17:11:20.772 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.772 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72f5bbb0:::benchmark_data_smithi171_78918_object67248:head# 2024-04-20T17:11:20.772 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.772 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72f67b69:::benchmark_data_smithi171_78918_object147380:head# 2024-04-20T17:11:20.773 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.773 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72f6aa21:::benchmark_data_smithi171_78918_object100459:head# 2024-04-20T17:11:20.773 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.773 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72f72d89:::benchmark_data_smithi171_78918_object60379:head# 2024-04-20T17:11:20.773 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.773 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72f8262e:::benchmark_data_smithi171_78918_object113372:head# 2024-04-20T17:11:20.773 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.773 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72f99926:::benchmark_data_smithi171_78918_object103885:head# 2024-04-20T17:11:20.773 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.773 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72f99cf2:::benchmark_data_smithi171_78918_object149134:head# 2024-04-20T17:11:20.773 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.773 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72f9b9be:::benchmark_data_smithi171_78918_object59808:head# 2024-04-20T17:11:20.773 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.773 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72f9c107:::benchmark_data_smithi171_78918_object157961:head# 2024-04-20T17:11:20.773 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.774 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72f9e505:::benchmark_data_smithi171_78918_object118934:head# 2024-04-20T17:11:20.774 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.774 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72fa83d4:::benchmark_data_smithi171_78918_object146791:head# 2024-04-20T17:11:20.774 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.774 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72fb7264:::benchmark_data_smithi171_78918_object123976:head# 2024-04-20T17:11:20.774 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.774 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72fc87be:::benchmark_data_smithi171_78918_object70487:head# 2024-04-20T17:11:20.774 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.774 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72fcaeab:::benchmark_data_smithi171_78918_object147264:head# 2024-04-20T17:11:20.774 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.774 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72fcf39b:::benchmark_data_smithi171_78918_object79166:head# 2024-04-20T17:11:20.774 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.774 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:72ffb630:::benchmark_data_smithi171_78918_object110042:head# 2024-04-20T17:11:20.774 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.774 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73005442:::benchmark_data_smithi171_78918_object131424:head# 2024-04-20T17:11:20.775 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.775 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7300949d:::benchmark_data_smithi171_78918_object147602:head# 2024-04-20T17:11:20.775 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.775 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7300bcc1:::benchmark_data_smithi171_78918_object134679:head# 2024-04-20T17:11:20.775 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.775 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7300c87c:::benchmark_data_smithi171_78918_object123437:head# 2024-04-20T17:11:20.775 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.775 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7300d1f4:::benchmark_data_smithi171_78918_object69890:head# 2024-04-20T17:11:20.775 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.775 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7300ed16:::benchmark_data_smithi171_78918_object143812:head# 2024-04-20T17:11:20.775 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.775 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7301399d:::benchmark_data_smithi171_78918_object68721:head# 2024-04-20T17:11:20.775 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.775 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7301aaba:::benchmark_data_smithi171_78918_object98887:head# 2024-04-20T17:11:20.775 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.776 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73020be9:::benchmark_data_smithi171_78918_object118918:head# 2024-04-20T17:11:20.776 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.776 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73025101:::benchmark_data_smithi171_78918_object83692:head# 2024-04-20T17:11:20.776 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.776 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:730284a1:::benchmark_data_smithi171_78918_object75129:head# 2024-04-20T17:11:20.776 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.776 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7302c336:::benchmark_data_smithi171_78918_object54644:head# 2024-04-20T17:11:20.776 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.776 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73043eea:::benchmark_data_smithi171_78918_object134074:head# 2024-04-20T17:11:20.776 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.776 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7304d64d:::benchmark_data_smithi171_78918_object129718:head# 2024-04-20T17:11:20.776 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.776 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73051b06:::benchmark_data_smithi171_78918_object139925:head# 2024-04-20T17:11:20.776 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.776 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7305e53c:::benchmark_data_smithi171_78918_object156307:head# 2024-04-20T17:11:20.777 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.777 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:730617c0:::benchmark_data_smithi171_78918_object160571:head# 2024-04-20T17:11:20.777 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.777 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:730637f3:::benchmark_data_smithi171_78918_object74833:head# 2024-04-20T17:11:20.777 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.777 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73064f7d:::benchmark_data_smithi171_78918_object73937:head# 2024-04-20T17:11:20.777 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.777 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7306a98e:::benchmark_data_smithi171_78918_object123342:head# 2024-04-20T17:11:20.777 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.777 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7306c35c:::benchmark_data_smithi171_78918_object79986:head# 2024-04-20T17:11:20.777 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.777 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73070152:::benchmark_data_smithi171_78918_object154371:head# 2024-04-20T17:11:20.777 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.777 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:730710de:::benchmark_data_smithi171_78918_object63642:head# 2024-04-20T17:11:20.777 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.778 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:730724a6:::benchmark_data_smithi171_78918_object76154:head# 2024-04-20T17:11:20.778 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.778 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73077eda:::benchmark_data_smithi171_78918_object94878:head# 2024-04-20T17:11:20.778 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.778 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7307847f:::benchmark_data_smithi171_78918_object145326:head# 2024-04-20T17:11:20.778 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.778 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73078ad9:::benchmark_data_smithi171_78918_object160289:head# 2024-04-20T17:11:20.778 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.778 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73079185:::benchmark_data_smithi171_78918_object56424:head# 2024-04-20T17:11:20.778 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.778 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7307da5a:::benchmark_data_smithi171_78918_object152168:head# 2024-04-20T17:11:20.778 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.778 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:730899fe:::benchmark_data_smithi171_78918_object63920:head# 2024-04-20T17:11:20.778 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.779 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:730965e6:::benchmark_data_smithi171_78918_object98274:head# 2024-04-20T17:11:20.779 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.779 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73099698:::benchmark_data_smithi171_78918_object53273:head# 2024-04-20T17:11:20.779 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.779 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7309a94f:::benchmark_data_smithi171_78918_object62332:head# 2024-04-20T17:11:20.779 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.779 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:730ab928:::benchmark_data_smithi171_78918_object57106:head# 2024-04-20T17:11:20.779 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.779 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:730c69ed:::benchmark_data_smithi171_78918_object154464:head# 2024-04-20T17:11:20.779 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.779 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:730cecd7:::benchmark_data_smithi171_78918_object137565:head# 2024-04-20T17:11:20.779 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.779 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:730cfeb5:::benchmark_data_smithi171_78918_object78946:head# 2024-04-20T17:11:20.779 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.779 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:730e2391:::benchmark_data_smithi171_78918_object124407:head# 2024-04-20T17:11:20.780 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.780 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:730e3cf1:::benchmark_data_smithi171_78918_object80707:head# 2024-04-20T17:11:20.780 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.780 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:730f75d6:::benchmark_data_smithi171_78918_object77025:head# 2024-04-20T17:11:20.780 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.780 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:730fcb6f:::benchmark_data_smithi171_78918_object153377:head# 2024-04-20T17:11:20.780 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.780 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73109b31:::benchmark_data_smithi171_78918_object60660:head# 2024-04-20T17:11:20.780 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.780 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7310c4f5:::benchmark_data_smithi171_78918_object130766:head# 2024-04-20T17:11:20.780 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.780 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73112da4:::benchmark_data_smithi171_78918_object147246:head# 2024-04-20T17:11:20.780 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.780 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7311c2b5:::benchmark_data_smithi171_78918_object70036:head# 2024-04-20T17:11:20.780 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.781 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73125c2e:::benchmark_data_smithi171_78918_object127050:head# 2024-04-20T17:11:20.781 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.781 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7312c906:::benchmark_data_smithi171_78918_object103023:head# 2024-04-20T17:11:20.781 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.781 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73136903:::benchmark_data_smithi171_78918_object87294:head# 2024-04-20T17:11:20.781 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.781 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7314c03c:::benchmark_data_smithi171_78918_object112182:head# 2024-04-20T17:11:20.781 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.781 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7314ef58:::benchmark_data_smithi171_78918_object56966:head# 2024-04-20T17:11:20.781 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.781 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73160d03:::benchmark_data_smithi171_78918_object84279:head# 2024-04-20T17:11:20.781 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.781 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:731627f1:::benchmark_data_smithi171_78918_object130975:head# 2024-04-20T17:11:20.781 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.782 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73174b47:::benchmark_data_smithi171_78918_object88173:head# 2024-04-20T17:11:20.782 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.782 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7317b6e1:::benchmark_data_smithi171_78918_object151441:head# 2024-04-20T17:11:20.782 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.782 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7318b28d:::benchmark_data_smithi171_78918_object142578:head# 2024-04-20T17:11:20.782 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.782 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73191a35:::benchmark_data_smithi171_78918_object109053:head# 2024-04-20T17:11:20.782 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.782 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:731ab6b8:::benchmark_data_smithi171_78918_object142782:head# 2024-04-20T17:11:20.782 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.782 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:731b3e0b:::benchmark_data_smithi171_78918_object129176:head# 2024-04-20T17:11:20.782 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.782 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:731c59a0:::benchmark_data_smithi171_78918_object107167:head# 2024-04-20T17:11:20.782 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.782 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:731cef08:::benchmark_data_smithi171_78918_object100659:head# 2024-04-20T17:11:20.783 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.783 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:731cfc66:::benchmark_data_smithi171_78918_object158585:head# 2024-04-20T17:11:20.783 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.783 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:731dd245:::benchmark_data_smithi171_78918_object84607:head# 2024-04-20T17:11:20.783 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.783 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:731e00b4:::benchmark_data_smithi171_78918_object101162:head# 2024-04-20T17:11:20.783 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.783 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:731e2eab:::benchmark_data_smithi171_78918_object128183:head# 2024-04-20T17:11:20.783 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.783 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:731e7844:::benchmark_data_smithi171_78918_object119073:head# 2024-04-20T17:11:20.783 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.783 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:731e7f44:::benchmark_data_smithi171_78918_object148588:head# 2024-04-20T17:11:20.783 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.783 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73205fdc:::benchmark_data_smithi171_78918_object121743:head# 2024-04-20T17:11:20.784 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.784 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7320b01e:::benchmark_data_smithi171_78918_object81523:head# 2024-04-20T17:11:20.784 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.784 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7320fec9:::benchmark_data_smithi171_78918_object158820:head# 2024-04-20T17:11:20.784 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.784 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73220dd2:::benchmark_data_smithi171_78918_object54350:head# 2024-04-20T17:11:20.784 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.784 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7322ac4a:::benchmark_data_smithi171_78918_object68425:head# 2024-04-20T17:11:20.784 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.784 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7323c4ad:::benchmark_data_smithi171_78918_object148397:head# 2024-04-20T17:11:20.784 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.784 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7323d0ad:::benchmark_data_smithi171_78918_object151350:head# 2024-04-20T17:11:20.784 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.784 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73246b1d:::benchmark_data_smithi171_78918_object58790:head# 2024-04-20T17:11:20.784 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.785 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7325df28:::benchmark_data_smithi171_78918_object125097:head# 2024-04-20T17:11:20.785 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.785 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73270a41:::benchmark_data_smithi171_78918_object84064:head# 2024-04-20T17:11:20.785 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.785 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73278b08:::benchmark_data_smithi171_78918_object59858:head# 2024-04-20T17:11:20.785 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.785 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73289fef:::benchmark_data_smithi171_78918_object86777:head# 2024-04-20T17:11:20.785 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.785 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7328e1d0:::benchmark_data_smithi171_78918_object55065:head# 2024-04-20T17:11:20.785 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.785 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7329b5a0:::benchmark_data_smithi171_78918_object126678:head# 2024-04-20T17:11:20.785 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.785 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:732b5387:::benchmark_data_smithi171_78918_object129591:head# 2024-04-20T17:11:20.785 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.785 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:732c3db0:::benchmark_data_smithi171_78918_object71487:head# 2024-04-20T17:11:20.786 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.786 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:732c477d:::benchmark_data_smithi171_78918_object124219:head# 2024-04-20T17:11:20.786 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.786 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:732c8659:::benchmark_data_smithi171_78918_object103341:head# 2024-04-20T17:11:20.786 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.786 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:732d22c3:::benchmark_data_smithi171_78918_object122506:head# 2024-04-20T17:11:20.786 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.786 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:732dce03:::benchmark_data_smithi171_78918_object156352:head# 2024-04-20T17:11:20.786 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.786 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:732f52d3:::benchmark_data_smithi171_78918_object106776:head# 2024-04-20T17:11:20.786 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.786 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:732f5456:::benchmark_data_smithi171_78918_object138500:head# 2024-04-20T17:11:20.786 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.786 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73307f98:::benchmark_data_smithi171_78918_object97204:head# 2024-04-20T17:11:20.786 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.787 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7330f967:::benchmark_data_smithi171_78918_object66399:head# 2024-04-20T17:11:20.787 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.787 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73310747:::benchmark_data_smithi171_78918_object62011:head# 2024-04-20T17:11:20.787 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.787 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:733109cd:::benchmark_data_smithi171_78918_object118500:head# 2024-04-20T17:11:20.787 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.787 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7331efba:::benchmark_data_smithi171_78918_object118376:head# 2024-04-20T17:11:20.787 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.787 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:733234e7:::benchmark_data_smithi171_78918_object117183:head# 2024-04-20T17:11:20.787 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.787 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73339021:::benchmark_data_smithi171_78918_object68460:head# 2024-04-20T17:11:20.787 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.787 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7333a0e7:::benchmark_data_smithi171_78918_object121436:head# 2024-04-20T17:11:20.787 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.787 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7334d79b:::benchmark_data_smithi171_78918_object134774:head# 2024-04-20T17:11:20.788 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.788 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73352710:::benchmark_data_smithi171_78918_object103193:head# 2024-04-20T17:11:20.788 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.788 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7335dbaf:::benchmark_data_smithi171_78918_object94889:head# 2024-04-20T17:11:20.788 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.788 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73369cd4:::benchmark_data_smithi171_78918_object71991:head# 2024-04-20T17:11:20.788 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.788 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7336ce98:::benchmark_data_smithi171_78918_object101285:head# 2024-04-20T17:11:20.788 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.788 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73384911:::benchmark_data_smithi171_78918_object113148:head# 2024-04-20T17:11:20.788 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.788 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7338d367:::benchmark_data_smithi171_78918_object108167:head# 2024-04-20T17:11:20.788 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.788 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:733a9edb:::benchmark_data_smithi171_78918_object54178:head# 2024-04-20T17:11:20.788 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.789 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:733aecb2:::benchmark_data_smithi171_78918_object75359:head# 2024-04-20T17:11:20.789 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.789 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:733b18d6:::benchmark_data_smithi171_78918_object72529:head# 2024-04-20T17:11:20.789 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.789 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:733bfa14:::benchmark_data_smithi171_78918_object99798:head# 2024-04-20T17:11:20.789 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.789 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:733d22be:::benchmark_data_smithi171_78918_object82016:head# 2024-04-20T17:11:20.789 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.789 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:733f023c:::benchmark_data_smithi171_78918_object149612:head# 2024-04-20T17:11:20.789 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.789 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:733f88d7:::benchmark_data_smithi171_78918_object143988:head# 2024-04-20T17:11:20.789 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.789 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:734038f6:::benchmark_data_smithi171_78918_object131579:head# 2024-04-20T17:11:20.789 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.789 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7340394c:::benchmark_data_smithi171_78918_object93965:head# 2024-04-20T17:11:20.790 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.790 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7340a1ce:::benchmark_data_smithi171_78918_object95549:head# 2024-04-20T17:11:20.790 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.790 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:734136ac:::benchmark_data_smithi171_78918_object93653:head# 2024-04-20T17:11:20.790 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.790 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73418b4e:::benchmark_data_smithi171_78918_object60803:head# 2024-04-20T17:11:20.790 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.790 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7344aefb:::benchmark_data_smithi171_78918_object160045:head# 2024-04-20T17:11:20.790 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.790 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7344f0ba:::benchmark_data_smithi171_78918_object87305:head# 2024-04-20T17:11:20.790 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.790 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7345384b:::benchmark_data_smithi171_78918_object82823:head# 2024-04-20T17:11:20.790 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.790 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:734629f7:::benchmark_data_smithi171_78918_object131738:head# 2024-04-20T17:11:20.791 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.791 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:734716c5:::benchmark_data_smithi171_78918_object95752:head# 2024-04-20T17:11:20.791 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.791 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73476f04:::benchmark_data_smithi171_78918_object63786:head# 2024-04-20T17:11:20.791 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.791 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73487d77:::benchmark_data_smithi171_78918_object109248:head# 2024-04-20T17:11:20.791 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.791 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7348e1b9:::benchmark_data_smithi171_78918_object80454:head# 2024-04-20T17:11:20.791 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.791 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7349a6e1:::benchmark_data_smithi171_78918_object56714:head# 2024-04-20T17:11:20.791 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.791 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7349b422:::benchmark_data_smithi171_78918_object108537:head# 2024-04-20T17:11:20.791 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.791 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7349bd71:::benchmark_data_smithi171_78918_object62081:head# 2024-04-20T17:11:20.791 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.792 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7349f87d:::benchmark_data_smithi171_78918_object106523:head# 2024-04-20T17:11:20.792 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.792 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:734a21ce:::benchmark_data_smithi171_78918_object68306:head# 2024-04-20T17:11:20.792 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.792 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:734a9ed9:::benchmark_data_smithi171_78918_object129545:head# 2024-04-20T17:11:20.792 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.792 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:734ace02:::benchmark_data_smithi171_78918_object128529:head# 2024-04-20T17:11:20.792 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.792 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:734b6d60:::benchmark_data_smithi171_78918_object76086:head# 2024-04-20T17:11:20.792 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.792 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:734b834d:::benchmark_data_smithi171_78918_object60502:head# 2024-04-20T17:11:20.792 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.792 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:734bdaf3:::benchmark_data_smithi171_78918_object100500:head# 2024-04-20T17:11:20.792 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.793 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:734c17e6:::benchmark_data_smithi171_78918_object80739:head# 2024-04-20T17:11:20.793 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.793 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:734c5f0d:::benchmark_data_smithi171_78918_object150772:head# 2024-04-20T17:11:20.793 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.793 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:734d6320:::benchmark_data_smithi171_78918_object114442:head# 2024-04-20T17:11:20.793 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.793 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:734d90ff:::benchmark_data_smithi171_78918_object93338:head# 2024-04-20T17:11:20.793 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.793 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:734da1a7:::benchmark_data_smithi171_78918_object113925:head# 2024-04-20T17:11:20.793 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.793 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:734df947:::benchmark_data_smithi171_78918_object155946:head# 2024-04-20T17:11:20.793 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.793 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:734e9497:::benchmark_data_smithi171_78918_object120149:head# 2024-04-20T17:11:20.793 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.793 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:734f527b:::benchmark_data_smithi171_78918_object51671:head# 2024-04-20T17:11:20.794 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.794 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73500e01:::benchmark_data_smithi171_78918_object157659:head# 2024-04-20T17:11:20.794 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.794 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7350e328:::benchmark_data_smithi171_78918_object73117:head# 2024-04-20T17:11:20.794 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.794 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:735225d0:::benchmark_data_smithi171_78918_object100254:head# 2024-04-20T17:11:20.794 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.794 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7352b82f:::benchmark_data_smithi171_78918_object74813:head# 2024-04-20T17:11:20.794 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.794 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:735305af:::benchmark_data_smithi171_78918_object153081:head# 2024-04-20T17:11:20.794 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.794 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73547d7f:::benchmark_data_smithi171_78918_object104793:head# 2024-04-20T17:11:20.794 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.794 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73559900:::benchmark_data_smithi171_78918_object134101:head# 2024-04-20T17:11:20.795 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.795 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73564887:::benchmark_data_smithi171_78918_object134276:head# 2024-04-20T17:11:20.795 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.795 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:735702d4:::benchmark_data_smithi171_78918_object90478:head# 2024-04-20T17:11:20.795 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.795 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:735708cb:::benchmark_data_smithi171_78918_object83744:head# 2024-04-20T17:11:20.795 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.795 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:735735f6:::benchmark_data_smithi171_78918_object54878:head# 2024-04-20T17:11:20.795 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.795 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73573989:::benchmark_data_smithi171_78918_object83823:head# 2024-04-20T17:11:20.795 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.795 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7357a715:::benchmark_data_smithi171_78918_object62732:head# 2024-04-20T17:11:20.795 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.795 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:735b3019:::benchmark_data_smithi171_78918_object59656:head# 2024-04-20T17:11:20.795 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.796 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:735b3816:::benchmark_data_smithi171_78918_object117032:head# 2024-04-20T17:11:20.796 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.796 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:735b876c:::benchmark_data_smithi171_78918_object68020:head# 2024-04-20T17:11:20.796 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.796 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:735baef5:::benchmark_data_smithi171_78918_object86531:head# 2024-04-20T17:11:20.796 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.796 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:735c9353:::benchmark_data_smithi171_78918_object86966:head# 2024-04-20T17:11:20.796 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.796 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:735cce66:::benchmark_data_smithi171_78918_object72844:head# 2024-04-20T17:11:20.796 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.796 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:735cfd29:::benchmark_data_smithi171_78918_object124479:head# 2024-04-20T17:11:20.796 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.796 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:735f2854:::benchmark_data_smithi171_78918_object141786:head# 2024-04-20T17:11:20.796 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.796 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73603bcf:::benchmark_data_smithi171_78918_object133258:head# 2024-04-20T17:11:20.797 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.797 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73616717:::benchmark_data_smithi171_78918_object58296:head# 2024-04-20T17:11:20.797 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.797 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7361fe70:::benchmark_data_smithi171_78918_object67894:head# 2024-04-20T17:11:20.797 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.797 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7362f9d4:::benchmark_data_smithi171_78918_object54332:head# 2024-04-20T17:11:20.797 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.797 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73635320:::benchmark_data_smithi171_78918_object129414:head# 2024-04-20T17:11:20.797 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.797 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:736446b0:::benchmark_data_smithi171_78918_object60607:head# 2024-04-20T17:11:20.797 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.797 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7365583b:::benchmark_data_smithi171_78918_object98516:head# 2024-04-20T17:11:20.797 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.797 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73664dcc:::benchmark_data_smithi171_78918_object89303:head# 2024-04-20T17:11:20.798 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.798 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7366c0dd:::benchmark_data_smithi171_78918_object125830:head# 2024-04-20T17:11:20.798 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.798 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:736796bf:::benchmark_data_smithi171_78918_object120842:head# 2024-04-20T17:11:20.798 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.798 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7368964e:::benchmark_data_smithi171_78918_object124638:head# 2024-04-20T17:11:20.798 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.798 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73694728:::benchmark_data_smithi171_78918_object138765:head# 2024-04-20T17:11:20.798 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.798 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73695d8c:::benchmark_data_smithi171_78918_object127821:head# 2024-04-20T17:11:20.798 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.798 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7369da22:::benchmark_data_smithi171_78918_object85621:head# 2024-04-20T17:11:20.798 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.798 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:736a2395:::benchmark_data_smithi171_78918_object87542:head# 2024-04-20T17:11:20.799 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.799 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:736a2ebb:::benchmark_data_smithi171_78918_object145310:head# 2024-04-20T17:11:20.799 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.799 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:736a440c:::benchmark_data_smithi171_78918_object130477:head# 2024-04-20T17:11:20.799 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.799 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:736a64eb:::benchmark_data_smithi171_78918_object60424:head# 2024-04-20T17:11:20.799 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.799 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:736acfaf:::benchmark_data_smithi171_78918_object73725:head# 2024-04-20T17:11:20.799 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.799 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:736d00e3:::benchmark_data_smithi171_78918_object75586:head# 2024-04-20T17:11:20.799 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.799 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:736db14b:::benchmark_data_smithi171_78918_object112720:head# 2024-04-20T17:11:20.799 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.799 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:736fcb0f:::benchmark_data_smithi171_78918_object69650:head# 2024-04-20T17:11:20.799 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.800 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:736fe145:::benchmark_data_smithi171_78918_object100640:head# 2024-04-20T17:11:20.800 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.800 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:736fe98a:::benchmark_data_smithi171_78918_object120245:head# 2024-04-20T17:11:20.800 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.800 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73701813:::benchmark_data_smithi171_78918_object82687:head# 2024-04-20T17:11:20.800 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.800 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7370772b:::benchmark_data_smithi171_78918_object96180:head# 2024-04-20T17:11:20.800 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.800 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73711ce7:::benchmark_data_smithi171_78918_object135167:head# 2024-04-20T17:11:20.800 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.800 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7371719b:::benchmark_data_smithi171_78918_object120281:head# 2024-04-20T17:11:20.800 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.800 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7371e766:::benchmark_data_smithi171_78918_object159611:head# 2024-04-20T17:11:20.800 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.800 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73727a28:::benchmark_data_smithi171_78918_object69126:head# 2024-04-20T17:11:20.801 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.801 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73730481:::benchmark_data_smithi171_78918_object156493:head# 2024-04-20T17:11:20.801 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.801 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73733cc8:::benchmark_data_smithi171_78918_object112738:head# 2024-04-20T17:11:20.801 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.801 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73744807:::benchmark_data_smithi171_78918_object63301:head# 2024-04-20T17:11:20.801 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.801 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7374ea32:::benchmark_data_smithi171_78918_object65041:head# 2024-04-20T17:11:20.801 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.801 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7374ff84:::benchmark_data_smithi171_78918_object59469:head# 2024-04-20T17:11:20.801 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.801 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73750abd:::benchmark_data_smithi171_78918_object149989:head# 2024-04-20T17:11:20.801 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.801 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73750deb:::benchmark_data_smithi171_78918_object68380:head# 2024-04-20T17:11:20.801 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.802 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73759a27:::benchmark_data_smithi171_78918_object103096:head# 2024-04-20T17:11:20.802 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.802 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7375dd2e:::benchmark_data_smithi171_78918_object106554:head# 2024-04-20T17:11:20.802 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.802 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73781ca3:::benchmark_data_smithi171_78918_object154348:head# 2024-04-20T17:11:20.802 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.802 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:737856ba:::benchmark_data_smithi171_78918_object90433:head# 2024-04-20T17:11:20.802 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.802 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:737894b5:::benchmark_data_smithi171_78918_object131966:head# 2024-04-20T17:11:20.802 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.802 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7379026a:::benchmark_data_smithi171_78918_object93012:head# 2024-04-20T17:11:20.802 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.802 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73794d9a:::benchmark_data_smithi171_78918_object71298:head# 2024-04-20T17:11:20.802 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.803 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:737962af:::benchmark_data_smithi171_78918_object100376:head# 2024-04-20T17:11:20.803 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.803 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:737a1c49:::benchmark_data_smithi171_78918_object95138:head# 2024-04-20T17:11:20.803 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.803 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:737b3a78:::benchmark_data_smithi171_78918_object85372:head# 2024-04-20T17:11:20.803 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.803 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:737c18d8:::benchmark_data_smithi171_78918_object133647:head# 2024-04-20T17:11:20.803 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.803 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:737c2031:::benchmark_data_smithi171_78918_object152606:head# 2024-04-20T17:11:20.803 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.803 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:737c5589:::benchmark_data_smithi171_78918_object145838:head# 2024-04-20T17:11:20.803 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.803 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:737c73a7:::benchmark_data_smithi171_78918_object106175:head# 2024-04-20T17:11:20.803 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.803 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:737d0527:::benchmark_data_smithi171_78918_object150310:head# 2024-04-20T17:11:20.804 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.804 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:737d4b74:::benchmark_data_smithi171_78918_object85442:head# 2024-04-20T17:11:20.804 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.804 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:737df269:::benchmark_data_smithi171_78918_object141993:head# 2024-04-20T17:11:20.804 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.804 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:737e906b:::benchmark_data_smithi171_78918_object86480:head# 2024-04-20T17:11:20.804 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.804 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:737f8d48:::benchmark_data_smithi171_78918_object60930:head# 2024-04-20T17:11:20.804 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.804 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73805bbd:::benchmark_data_smithi171_78918_object81512:head# 2024-04-20T17:11:20.804 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.804 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7381c4d0:::benchmark_data_smithi171_78918_object102487:head# 2024-04-20T17:11:20.804 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.804 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73820c2a:::benchmark_data_smithi171_78918_object132395:head# 2024-04-20T17:11:20.804 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.805 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73825588:::benchmark_data_smithi171_78918_object135552:head# 2024-04-20T17:11:20.805 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.805 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73833545:::benchmark_data_smithi171_78918_object123540:head# 2024-04-20T17:11:20.805 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.805 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:738366a8:::benchmark_data_smithi171_78918_object158278:head# 2024-04-20T17:11:20.805 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.805 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:738385cd:::benchmark_data_smithi171_78918_object99859:head# 2024-04-20T17:11:20.805 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.805 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7384f90c:::benchmark_data_smithi171_78918_object127896:head# 2024-04-20T17:11:20.805 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.805 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:738613a1:::benchmark_data_smithi171_78918_object70324:head# 2024-04-20T17:11:20.805 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.805 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73868a66:::benchmark_data_smithi171_78918_object144874:head# 2024-04-20T17:11:20.805 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.806 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73878f2b:::benchmark_data_smithi171_78918_object79587:head# 2024-04-20T17:11:20.806 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.806 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7387cbc1:::benchmark_data_smithi171_78918_object138594:head# 2024-04-20T17:11:20.806 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.806 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73886388:::benchmark_data_smithi171_78918_object100999:head# 2024-04-20T17:11:20.806 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.806 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:738a0c21:::benchmark_data_smithi171_78918_object95239:head# 2024-04-20T17:11:20.806 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.806 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:738a3aa0:::benchmark_data_smithi171_78918_object90161:head# 2024-04-20T17:11:20.806 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.807 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:738ae46d:::benchmark_data_smithi171_78918_object111695:head# 2024-04-20T17:11:20.807 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.807 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:738bd814:::benchmark_data_smithi171_78918_object82818:head# 2024-04-20T17:11:20.807 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.807 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:738c5651:::benchmark_data_smithi171_78918_object106847:head# 2024-04-20T17:11:20.807 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.807 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:738ce44e:::benchmark_data_smithi171_78918_object147771:head# 2024-04-20T17:11:20.807 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.807 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:738d9641:::benchmark_data_smithi171_78918_object85558:head# 2024-04-20T17:11:20.807 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.807 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:738e430d:::benchmark_data_smithi171_78918_object111290:head# 2024-04-20T17:11:20.807 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.807 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:738efe76:::benchmark_data_smithi171_78918_object104510:head# 2024-04-20T17:11:20.807 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.808 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:738f0ec2:::benchmark_data_smithi171_78918_object138237:head# 2024-04-20T17:11:20.808 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.808 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:738f43b6:::benchmark_data_smithi171_78918_object100367:head# 2024-04-20T17:11:20.808 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.808 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:738f5083:::benchmark_data_smithi171_78918_object158191:head# 2024-04-20T17:11:20.808 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.808 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:738fb3ed:::benchmark_data_smithi171_78918_object152718:head# 2024-04-20T17:11:20.808 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.808 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7390693c:::benchmark_data_smithi171_78918_object130003:head# 2024-04-20T17:11:20.808 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.808 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73910acb:::benchmark_data_smithi171_78918_object78264:head# 2024-04-20T17:11:20.808 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.808 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73913e77:::benchmark_data_smithi171_78918_object124387:head# 2024-04-20T17:11:20.808 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.808 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73917d76:::benchmark_data_smithi171_78918_object56496:head# 2024-04-20T17:11:20.809 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.809 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:739265ed:::benchmark_data_smithi171_78918_object132413:head# 2024-04-20T17:11:20.809 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.809 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7392a318:::benchmark_data_smithi171_78918_object128788:head# 2024-04-20T17:11:20.809 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.809 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7392c255:::benchmark_data_smithi171_78918_object86954:head# 2024-04-20T17:11:20.809 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.809 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:739334d3:::benchmark_data_smithi171_78918_object57728:head# 2024-04-20T17:11:20.809 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.809 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73934db4:::benchmark_data_smithi171_78918_object125534:head# 2024-04-20T17:11:20.809 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.809 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73937499:::benchmark_data_smithi171_78918_object112982:head# 2024-04-20T17:11:20.809 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.809 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7394e831:::benchmark_data_smithi171_78918_object124329:head# 2024-04-20T17:11:20.810 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.810 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73950e06:::benchmark_data_smithi171_78918_object91252:head# 2024-04-20T17:11:20.810 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.810 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7396494e:::benchmark_data_smithi171_78918_object63159:head# 2024-04-20T17:11:20.810 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.810 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73967d99:::benchmark_data_smithi171_78918_object93117:head# 2024-04-20T17:11:20.810 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.810 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73970ddc:::benchmark_data_smithi171_78918_object140726:head# 2024-04-20T17:11:20.810 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.810 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7397db0a:::benchmark_data_smithi171_78918_object146409:head# 2024-04-20T17:11:20.810 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.810 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73982d5c:::benchmark_data_smithi171_78918_object114662:head# 2024-04-20T17:11:20.810 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.810 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:739837b1:::benchmark_data_smithi171_78918_object112285:head# 2024-04-20T17:11:20.811 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.811 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:739856fe:::benchmark_data_smithi171_78918_object94085:head# 2024-04-20T17:11:20.811 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.811 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7399e7ad:::benchmark_data_smithi171_78918_object96022:head# 2024-04-20T17:11:20.811 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.811 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:739a130c:::benchmark_data_smithi171_78918_object64568:head# 2024-04-20T17:11:20.811 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.811 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:739a45a8:::benchmark_data_smithi171_78918_object140593:head# 2024-04-20T17:11:20.811 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.811 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:739a710f:::benchmark_data_smithi171_78918_object84823:head# 2024-04-20T17:11:20.811 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.811 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:739abdfa:::benchmark_data_smithi171_78918_object115076:head# 2024-04-20T17:11:20.811 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.811 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:739c126a:::benchmark_data_smithi171_78918_object135399:head# 2024-04-20T17:11:20.811 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.812 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:739c27c8:::benchmark_data_smithi171_78918_object115543:head# 2024-04-20T17:11:20.812 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.812 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:739d4609:::benchmark_data_smithi171_78918_object121926:head# 2024-04-20T17:11:20.812 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.812 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:739d8a3e:::benchmark_data_smithi171_78918_object95517:head# 2024-04-20T17:11:20.812 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.812 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:739ecb02:::benchmark_data_smithi171_78918_object62098:head# 2024-04-20T17:11:20.812 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.812 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73a02272:::benchmark_data_smithi171_78918_object77817:head# 2024-04-20T17:11:20.812 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.812 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73a09637:::benchmark_data_smithi171_78918_object148651:head# 2024-04-20T17:11:20.812 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.812 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73a0b22a:::benchmark_data_smithi171_78918_object89332:head# 2024-04-20T17:11:20.812 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.812 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73a12c89:::benchmark_data_smithi171_78918_object93854:head# 2024-04-20T17:11:20.813 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.813 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73a1b01b:::benchmark_data_smithi171_78918_object111825:head# 2024-04-20T17:11:20.813 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.813 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73a1b8dd:::benchmark_data_smithi171_78918_object58361:head# 2024-04-20T17:11:20.813 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.813 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73a25690:::benchmark_data_smithi171_78918_object86003:head# 2024-04-20T17:11:20.813 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.813 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73a2d121:::benchmark_data_smithi171_78918_object152944:head# 2024-04-20T17:11:20.813 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.813 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73a3000e:::benchmark_data_smithi171_78918_object90243:head# 2024-04-20T17:11:20.813 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.813 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73a3a4b8:::benchmark_data_smithi171_78918_object132087:head# 2024-04-20T17:11:20.813 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.813 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73a42317:::benchmark_data_smithi171_78918_object89949:head# 2024-04-20T17:11:20.814 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.814 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73a44673:::benchmark_data_smithi171_78918_object133264:head# 2024-04-20T17:11:20.814 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.814 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73a4530b:::benchmark_data_smithi171_78918_object91760:head# 2024-04-20T17:11:20.814 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.814 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73a52de6:::benchmark_data_smithi171_78918_object54213:head# 2024-04-20T17:11:20.814 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.814 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73a5859b:::benchmark_data_smithi171_78918_object53574:head# 2024-04-20T17:11:20.814 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.814 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73a6b8c6:::benchmark_data_smithi171_78918_object85281:head# 2024-04-20T17:11:20.814 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.814 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73a6d086:::benchmark_data_smithi171_78918_object111188:head# 2024-04-20T17:11:20.814 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.814 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73a859da:::benchmark_data_smithi171_78918_object62499:head# 2024-04-20T17:11:20.814 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.815 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73a89798:::benchmark_data_smithi171_78918_object56507:head# 2024-04-20T17:11:20.815 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.815 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73a8f40c:::benchmark_data_smithi171_78918_object152689:head# 2024-04-20T17:11:20.815 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.815 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73a90159:::benchmark_data_smithi171_78918_object108629:head# 2024-04-20T17:11:20.815 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.815 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73aa34a5:::benchmark_data_smithi171_78918_object127812:head# 2024-04-20T17:11:20.815 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.815 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73aa6525:::benchmark_data_smithi171_78918_object105726:head# 2024-04-20T17:11:20.815 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.815 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73ab4eaa:::benchmark_data_smithi171_78918_object144738:head# 2024-04-20T17:11:20.815 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.815 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73abd773:::benchmark_data_smithi171_78918_object66263:head# 2024-04-20T17:11:20.815 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.815 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73abe10c:::benchmark_data_smithi171_78918_object102250:head# 2024-04-20T17:11:20.816 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.816 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73abe92e:::benchmark_data_smithi171_78918_object69847:head# 2024-04-20T17:11:20.816 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.816 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73aca2a3:::benchmark_data_smithi171_78918_object154412:head# 2024-04-20T17:11:20.816 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.816 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73acc083:::benchmark_data_smithi171_78918_object87307:head# 2024-04-20T17:11:20.816 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.816 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73acd694:::benchmark_data_smithi171_78918_object53413:head# 2024-04-20T17:11:20.816 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.816 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73ad6e3d:::benchmark_data_smithi171_78918_object121032:head# 2024-04-20T17:11:20.816 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.816 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73ae2ffa:::benchmark_data_smithi171_78918_object102561:head# 2024-04-20T17:11:20.816 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.816 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73aea62c:::benchmark_data_smithi171_78918_object90547:head# 2024-04-20T17:11:20.817 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.817 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73aea9cb:::benchmark_data_smithi171_78918_object103154:head# 2024-04-20T17:11:20.817 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.817 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73aece78:::benchmark_data_smithi171_78918_object89057:head# 2024-04-20T17:11:20.817 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.817 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73af7af3:::benchmark_data_smithi171_78918_object130361:head# 2024-04-20T17:11:20.817 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.817 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73af85e0:::benchmark_data_smithi171_78918_object54562:head# 2024-04-20T17:11:20.817 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.817 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73af8cd7:::benchmark_data_smithi171_78918_object157352:head# 2024-04-20T17:11:20.817 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.817 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73afe5a2:::benchmark_data_smithi171_78918_object133725:head# 2024-04-20T17:11:20.817 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.817 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73affd67:::benchmark_data_smithi171_78918_object75502:head# 2024-04-20T17:11:20.817 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.818 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73b03c71:::benchmark_data_smithi171_78918_object152080:head# 2024-04-20T17:11:20.818 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.818 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73b0a918:::benchmark_data_smithi171_78918_object148017:head# 2024-04-20T17:11:20.818 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.818 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73b1d5da:::benchmark_data_smithi171_78918_object101080:head# 2024-04-20T17:11:20.818 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.818 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73b278cc:::benchmark_data_smithi171_78918_object67537:head# 2024-04-20T17:11:20.818 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.818 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73b2a9bf:::benchmark_data_smithi171_78918_object61810:head# 2024-04-20T17:11:20.818 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.818 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73b3adf8:::benchmark_data_smithi171_78918_object154633:head# 2024-04-20T17:11:20.818 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.818 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73b3e157:::benchmark_data_smithi171_78918_object94481:head# 2024-04-20T17:11:20.818 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.819 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73b3e693:::benchmark_data_smithi171_78918_object147713:head# 2024-04-20T17:11:20.819 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.819 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73b3f920:::benchmark_data_smithi171_78918_object145402:head# 2024-04-20T17:11:20.819 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.819 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73b42231:::benchmark_data_smithi171_78918_object100572:head# 2024-04-20T17:11:20.819 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.819 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73b4d6be:::benchmark_data_smithi171_78918_object90363:head# 2024-04-20T17:11:20.819 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.819 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73b50096:::benchmark_data_smithi171_78918_object113980:head# 2024-04-20T17:11:20.819 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.819 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73b5d5b1:::benchmark_data_smithi171_78918_object56172:head# 2024-04-20T17:11:20.819 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.819 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73b6b59c:::benchmark_data_smithi171_78918_object84595:head# 2024-04-20T17:11:20.819 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.819 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73b8649d:::benchmark_data_smithi171_78918_object99006:head# 2024-04-20T17:11:20.820 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.820 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73b888f0:::benchmark_data_smithi171_78918_object160277:head# 2024-04-20T17:11:20.820 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.820 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73b94c1a:::benchmark_data_smithi171_78918_object136987:head# 2024-04-20T17:11:20.820 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.820 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73b97d0f:::benchmark_data_smithi171_78918_object63650:head# 2024-04-20T17:11:20.820 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.820 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73baa460:::benchmark_data_smithi171_78918_object129583:head# 2024-04-20T17:11:20.820 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.820 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73babced:::benchmark_data_smithi171_78918_object149372:head# 2024-04-20T17:11:20.820 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.820 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73bb1f43:::benchmark_data_smithi171_78918_object110817:head# 2024-04-20T17:11:20.820 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.820 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73bc011f:::benchmark_data_smithi171_78918_object154765:head# 2024-04-20T17:11:20.821 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.821 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73bc30cb:::benchmark_data_smithi171_78918_object68854:head# 2024-04-20T17:11:20.821 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.821 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73bc59fc:::benchmark_data_smithi171_78918_object122885:head# 2024-04-20T17:11:20.821 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.821 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73bca4e6:::benchmark_data_smithi171_78918_object140808:head# 2024-04-20T17:11:20.821 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.821 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73bce7ef:::benchmark_data_smithi171_78918_object64488:head# 2024-04-20T17:11:20.821 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.821 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73bd70d5:::benchmark_data_smithi171_78918_object137561:head# 2024-04-20T17:11:20.821 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.821 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73bf1239:::benchmark_data_smithi171_78918_object87853:head# 2024-04-20T17:11:20.821 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.821 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73bf8efd:::benchmark_data_smithi171_78918_object56285:head# 2024-04-20T17:11:20.821 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.822 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73c0360d:::benchmark_data_smithi171_78918_object84832:head# 2024-04-20T17:11:20.822 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.822 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73c2cbdd:::benchmark_data_smithi171_78918_object148217:head# 2024-04-20T17:11:20.822 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.822 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73c3abcb:::benchmark_data_smithi171_78918_object111286:head# 2024-04-20T17:11:20.822 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.822 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73c40044:::benchmark_data_smithi171_78918_object65697:head# 2024-04-20T17:11:20.822 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.822 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73c41de4:::benchmark_data_smithi171_78918_object58535:head# 2024-04-20T17:11:20.822 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.822 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73c43acc:::benchmark_data_smithi171_78918_object140380:head# 2024-04-20T17:11:20.822 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.822 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73c45c1e:::benchmark_data_smithi171_78918_object62218:head# 2024-04-20T17:11:20.822 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.823 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73c51702:::benchmark_data_smithi171_78918_object135216:head# 2024-04-20T17:11:20.823 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.823 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73c5f191:::benchmark_data_smithi171_78918_object98381:head# 2024-04-20T17:11:20.823 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.823 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73c6dc5f:::benchmark_data_smithi171_78918_object101683:head# 2024-04-20T17:11:20.823 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.823 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73c74bd8:::benchmark_data_smithi171_78918_object107826:head# 2024-04-20T17:11:20.823 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.823 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73c78f74:::benchmark_data_smithi171_78918_object83994:head# 2024-04-20T17:11:20.823 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.823 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73c7df6c:::benchmark_data_smithi171_78918_object79194:head# 2024-04-20T17:11:20.823 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.823 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73c7e4f7:::benchmark_data_smithi171_78918_object120559:head# 2024-04-20T17:11:20.823 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.823 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73c7fe0a:::benchmark_data_smithi171_78918_object159851:head# 2024-04-20T17:11:20.824 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.824 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73c9f4d7:::benchmark_data_smithi171_78918_object133326:head# 2024-04-20T17:11:20.824 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.824 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73ca5bec:::benchmark_data_smithi171_78918_object66846:head# 2024-04-20T17:11:20.824 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.824 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73cadb34:::benchmark_data_smithi171_78918_object75660:head# 2024-04-20T17:11:20.824 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.824 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73cb52db:::benchmark_data_smithi171_78918_object148072:head# 2024-04-20T17:11:20.824 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.824 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73cc5f15:::benchmark_data_smithi171_78918_object136600:head# 2024-04-20T17:11:20.824 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.824 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73cd272a:::benchmark_data_smithi171_78918_object92787:head# 2024-04-20T17:11:20.824 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.824 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73cd9fe2:::benchmark_data_smithi171_78918_object125394:head# 2024-04-20T17:11:20.824 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.825 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73ce0eac:::benchmark_data_smithi171_78918_object93621:head# 2024-04-20T17:11:20.825 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.825 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73ce1e48:::benchmark_data_smithi171_78918_object118295:head# 2024-04-20T17:11:20.825 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.825 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73cf0b21:::benchmark_data_smithi171_78918_object135701:head# 2024-04-20T17:11:20.825 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.825 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73d06ccc:::benchmark_data_smithi171_78918_object143453:head# 2024-04-20T17:11:20.825 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.825 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73d12cca:::benchmark_data_smithi171_78918_object144364:head# 2024-04-20T17:11:20.825 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.825 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73d205fc:::benchmark_data_smithi171_78918_object150387:head# 2024-04-20T17:11:20.825 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.825 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73d320cb:::benchmark_data_smithi171_78918_object148871:head# 2024-04-20T17:11:20.825 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.825 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73d37b1c:::benchmark_data_smithi171_78918_object58514:head# 2024-04-20T17:11:20.826 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.826 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73d492f4:::benchmark_data_smithi171_78918_object117570:head# 2024-04-20T17:11:20.826 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.826 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73d4b4ae:::benchmark_data_smithi171_78918_object149632:head# 2024-04-20T17:11:20.826 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.826 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73d55c3b:::benchmark_data_smithi171_78918_object109151:head# 2024-04-20T17:11:20.826 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:20.828 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:11:20.828 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:11:20.829 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:11:20.829 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:11:20.829 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 0 2024-04-20T17:11:20.829 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:11:20.913 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73d650bc:::benchmark_data_smithi171_78918_object99009:head# 2024-04-20T17:11:20.913 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.913 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73d75289:::benchmark_data_smithi171_78918_object116159:head# 2024-04-20T17:11:20.913 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.914 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73d7602b:::benchmark_data_smithi171_78918_object149697:head# 2024-04-20T17:11:20.914 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.914 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73d784ff:::benchmark_data_smithi171_78918_object93837:head# 2024-04-20T17:11:20.914 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.914 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73d7b463:::benchmark_data_smithi171_78918_object146432:head# 2024-04-20T17:11:20.914 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.914 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73d82718:::benchmark_data_smithi171_78918_object88681:head# 2024-04-20T17:11:20.914 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.914 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73d8c1f1:::benchmark_data_smithi171_78918_object83464:head# 2024-04-20T17:11:20.914 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.914 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73da0e4e:::benchmark_data_smithi171_78918_object122829:head# 2024-04-20T17:11:20.914 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.914 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73da590b:::benchmark_data_smithi171_78918_object71544:head# 2024-04-20T17:11:20.915 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.915 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73db1b0b:::benchmark_data_smithi171_78918_object118947:head# 2024-04-20T17:11:20.915 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.915 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73db8371:::benchmark_data_smithi171_78918_object64853:head# 2024-04-20T17:11:20.915 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.915 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73dcc0a6:::benchmark_data_smithi171_78918_object75823:head# 2024-04-20T17:11:20.915 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.915 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73dd3b71:::benchmark_data_smithi171_78918_object140740:head# 2024-04-20T17:11:20.915 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.915 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73ddd2a8:::benchmark_data_smithi171_78918_object66506:head# 2024-04-20T17:11:20.915 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.915 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73dee512:::benchmark_data_smithi171_78918_object136751:head# 2024-04-20T17:11:20.915 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.915 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73df9c7e:::benchmark_data_smithi171_78918_object123593:head# 2024-04-20T17:11:20.915 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.916 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73dfe46b:::benchmark_data_smithi171_78918_object56999:head# 2024-04-20T17:11:20.916 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.916 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73e01080:::benchmark_data_smithi171_78918_object121834:head# 2024-04-20T17:11:20.916 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.916 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73e01c91:::benchmark_data_smithi171_78918_object125992:head# 2024-04-20T17:11:20.916 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.916 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73e01dad:::benchmark_data_smithi171_78918_object66975:head# 2024-04-20T17:11:20.916 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.916 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73e0b831:::benchmark_data_smithi171_78918_object160615:head# 2024-04-20T17:11:20.916 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.916 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73e0c351:::benchmark_data_smithi171_78918_object51774:head# 2024-04-20T17:11:20.916 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.916 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73e0f793:::benchmark_data_smithi171_78918_object94124:head# 2024-04-20T17:11:20.917 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.917 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73e203ff:::benchmark_data_smithi171_78918_object55803:head# 2024-04-20T17:11:20.917 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.917 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73e27e1e:::benchmark_data_smithi171_78918_object51533:head# 2024-04-20T17:11:20.917 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.917 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73e3b2a3:::benchmark_data_smithi171_78918_object131529:head# 2024-04-20T17:11:20.917 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.917 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73e41f41:::benchmark_data_smithi171_78918_object129304:head# 2024-04-20T17:11:20.917 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.917 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73e4561f:::benchmark_data_smithi171_78918_object80845:head# 2024-04-20T17:11:20.917 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.917 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73e48826:::benchmark_data_smithi171_78918_object138739:head# 2024-04-20T17:11:20.917 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.917 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73e58cdc:::benchmark_data_smithi171_78918_object125234:head# 2024-04-20T17:11:20.918 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.918 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73e5b915:::benchmark_data_smithi171_78918_object52789:head# 2024-04-20T17:11:20.918 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.918 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73e5dbcc:::benchmark_data_smithi171_78918_object149497:head# 2024-04-20T17:11:20.918 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.918 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73e6cbbf:::benchmark_data_smithi171_78918_object125134:head# 2024-04-20T17:11:20.918 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.918 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73e741fa:::benchmark_data_smithi171_78918_object101501:head# 2024-04-20T17:11:20.918 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.918 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73e922e2:::benchmark_data_smithi171_78918_object105581:head# 2024-04-20T17:11:20.918 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.918 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73e96c41:::benchmark_data_smithi171_78918_object145877:head# 2024-04-20T17:11:20.918 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.919 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73e9bc80:::benchmark_data_smithi171_78918_object94112:head# 2024-04-20T17:11:20.919 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.919 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73ea35fa:::benchmark_data_smithi171_78918_object57791:head# 2024-04-20T17:11:20.919 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.919 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73eb572f:::benchmark_data_smithi171_78918_object112278:head# 2024-04-20T17:11:20.919 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.919 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73ec33c8:::benchmark_data_smithi171_78918_object56610:head# 2024-04-20T17:11:20.919 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.919 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73ec3df1:::benchmark_data_smithi171_78918_object62858:head# 2024-04-20T17:11:20.919 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.919 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73ec932c:::benchmark_data_smithi171_78918_object53299:head# 2024-04-20T17:11:20.919 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.919 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73ec9e5d:::benchmark_data_smithi171_78918_object97365:head# 2024-04-20T17:11:20.919 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.919 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73edae88:::benchmark_data_smithi171_78918_object56203:head# 2024-04-20T17:11:20.920 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.920 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73edf052:::benchmark_data_smithi171_78918_object56324:head# 2024-04-20T17:11:20.920 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.920 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73ee8746:::benchmark_data_smithi171_78918_object119710:head# 2024-04-20T17:11:20.920 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.920 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73ef4d80:::benchmark_data_smithi171_78918_object98483:head# 2024-04-20T17:11:20.920 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.920 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73efe43d:::benchmark_data_smithi171_78918_object55379:head# 2024-04-20T17:11:20.920 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.920 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73f02128:::benchmark_data_smithi171_78918_object158325:head# 2024-04-20T17:11:20.920 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.920 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73f23b84:::benchmark_data_smithi171_78918_object156565:head# 2024-04-20T17:11:20.920 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.920 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73f27569:::benchmark_data_smithi171_78918_object59773:head# 2024-04-20T17:11:20.921 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.921 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73f3864d:::benchmark_data_smithi171_78918_object78425:head# 2024-04-20T17:11:20.921 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.921 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73f45050:::benchmark_data_smithi171_78918_object57431:head# 2024-04-20T17:11:20.921 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.921 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73f4686a:::benchmark_data_smithi171_78918_object71051:head# 2024-04-20T17:11:20.921 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.921 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73f527da:::benchmark_data_smithi171_78918_object82323:head# 2024-04-20T17:11:20.921 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.921 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73f584b3:::benchmark_data_smithi171_78918_object131190:head# 2024-04-20T17:11:20.921 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.921 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73f63191:::benchmark_data_smithi171_78918_object141322:head# 2024-04-20T17:11:20.921 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.921 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73f6c025:::benchmark_data_smithi171_78918_object59087:head# 2024-04-20T17:11:20.922 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.922 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73f6f3d3:::benchmark_data_smithi171_78918_object51579:head# 2024-04-20T17:11:20.922 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.922 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73f785c1:::benchmark_data_smithi171_78918_object80996:head# 2024-04-20T17:11:20.922 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.922 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73f84908:::benchmark_data_smithi171_78918_object118088:head# 2024-04-20T17:11:20.922 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.922 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73f87028:::benchmark_data_smithi171_78918_object133277:head# 2024-04-20T17:11:20.922 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.922 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73f916cd:::benchmark_data_smithi171_78918_object160121:head# 2024-04-20T17:11:20.922 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.922 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73f9df22:::benchmark_data_smithi171_78918_object116506:head# 2024-04-20T17:11:20.922 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.922 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73fa2614:::benchmark_data_smithi171_78918_object157491:head# 2024-04-20T17:11:20.922 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.923 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73fa749e:::benchmark_data_smithi171_78918_object65303:head# 2024-04-20T17:11:20.923 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.923 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73fa9c9f:::benchmark_data_smithi171_78918_object137763:head# 2024-04-20T17:11:20.923 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.923 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73fb6daa:::benchmark_data_smithi171_78918_object152007:head# 2024-04-20T17:11:20.923 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.923 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73fc592c:::benchmark_data_smithi171_78918_object111670:head# 2024-04-20T17:11:20.923 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.923 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73fcb9f2:::benchmark_data_smithi171_78918_object149973:head# 2024-04-20T17:11:20.923 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.923 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73fd93bc:::benchmark_data_smithi171_78918_object91318:head# 2024-04-20T17:11:20.923 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.923 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73fdbdd2:::benchmark_data_smithi171_78918_object140795:head# 2024-04-20T17:11:20.924 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.924 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73fdd745:::benchmark_data_smithi171_78918_object84519:head# 2024-04-20T17:11:20.924 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.924 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73fdd9aa:::benchmark_data_smithi171_78918_object121558:head# 2024-04-20T17:11:20.924 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.924 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:73ff4132:::benchmark_data_smithi171_78918_object148394:head# 2024-04-20T17:11:20.924 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.924 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74013248:::benchmark_data_smithi171_78918_object113743:head# 2024-04-20T17:11:20.924 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.924 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7401397f:::benchmark_data_smithi171_78918_object110567:head# 2024-04-20T17:11:20.924 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.924 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7401bb58:::benchmark_data_smithi171_78918_object87279:head# 2024-04-20T17:11:20.925 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.925 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:740366a8:::benchmark_data_smithi171_78918_object66637:head# 2024-04-20T17:11:20.925 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.925 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74037d5b:::benchmark_data_smithi171_78918_object99130:head# 2024-04-20T17:11:20.925 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.925 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7403b18e:::benchmark_data_smithi171_78918_object134562:head# 2024-04-20T17:11:20.925 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.925 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7403c8fe:::benchmark_data_smithi171_78918_object60492:head# 2024-04-20T17:11:20.925 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.925 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7403d9db:::benchmark_data_smithi171_78918_object156986:head# 2024-04-20T17:11:20.925 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.925 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74040df3:::benchmark_data_smithi171_78918_object153448:head# 2024-04-20T17:11:20.925 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.925 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74047381:::benchmark_data_smithi171_78918_object103426:head# 2024-04-20T17:11:20.926 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.926 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74051395:::benchmark_data_smithi171_78918_object103749:head# 2024-04-20T17:11:20.926 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.926 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7405151d:::benchmark_data_smithi171_78918_object51816:head# 2024-04-20T17:11:20.926 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.926 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74054e0a:::benchmark_data_smithi171_78918_object117063:head# 2024-04-20T17:11:20.926 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.926 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7405850a:::benchmark_data_smithi171_78918_object56404:head# 2024-04-20T17:11:20.926 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.926 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7407ddb5:::benchmark_data_smithi171_78918_object91488:head# 2024-04-20T17:11:20.926 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.926 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7409ab7c:::benchmark_data_smithi171_78918_object154149:head# 2024-04-20T17:11:20.926 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.926 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7409bbf9:::benchmark_data_smithi171_78918_object82759:head# 2024-04-20T17:11:20.927 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.927 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:740abb3a:::benchmark_data_smithi171_78918_object80159:head# 2024-04-20T17:11:20.927 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.927 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:740bd7c6:::benchmark_data_smithi171_78918_object137601:head# 2024-04-20T17:11:20.927 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.927 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:740bfa5d:::benchmark_data_smithi171_78918_object146167:head# 2024-04-20T17:11:20.927 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.927 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:740c8547:::benchmark_data_smithi171_78918_object129007:head# 2024-04-20T17:11:20.927 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.927 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:740d3435:::benchmark_data_smithi171_78918_object72050:head# 2024-04-20T17:11:20.927 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.927 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:740e3e45:::benchmark_data_smithi171_78918_object152352:head# 2024-04-20T17:11:20.928 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.928 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:740ef6b7:::benchmark_data_smithi171_78918_object86987:head# 2024-04-20T17:11:20.928 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.928 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:740f77e3:::benchmark_data_smithi171_78918_object97277:head# 2024-04-20T17:11:20.928 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.928 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:740fb200:::benchmark_data_smithi171_78918_object82242:head# 2024-04-20T17:11:20.928 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.928 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:740ffc42:::benchmark_data_smithi171_78918_object141990:head# 2024-04-20T17:11:20.928 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.928 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:741019ae:::benchmark_data_smithi171_78918_object79876:head# 2024-04-20T17:11:20.928 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.929 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74101e22:::benchmark_data_smithi171_78918_object137499:head# 2024-04-20T17:11:20.929 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.929 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74104489:::benchmark_data_smithi171_78918_object80122:head# 2024-04-20T17:11:20.929 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.929 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74111ece:::benchmark_data_smithi171_78918_object65646:head# 2024-04-20T17:11:20.929 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.929 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74115881:::benchmark_data_smithi171_78918_object138381:head# 2024-04-20T17:11:20.929 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.929 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74121f97:::benchmark_data_smithi171_78918_object114984:head# 2024-04-20T17:11:20.930 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.930 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74122053:::benchmark_data_smithi171_78918_object147691:head# 2024-04-20T17:11:20.930 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.930 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74128053:::benchmark_data_smithi171_78918_object67386:head# 2024-04-20T17:11:20.930 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.930 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7413e3c9:::benchmark_data_smithi171_78918_object103163:head# 2024-04-20T17:11:20.930 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.930 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74147234:::benchmark_data_smithi171_78918_object57249:head# 2024-04-20T17:11:20.930 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.931 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7414ac16:::benchmark_data_smithi171_78918_object94812:head# 2024-04-20T17:11:20.931 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.931 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74155dd3:::benchmark_data_smithi171_78918_object52287:head# 2024-04-20T17:11:20.931 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.931 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:741568dd:::benchmark_data_smithi171_78918_object53129:head# 2024-04-20T17:11:20.931 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.931 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74157f19:::benchmark_data_smithi171_78918_object57894:head# 2024-04-20T17:11:20.932 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.932 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74169e54:::benchmark_data_smithi171_78918_object70473:head# 2024-04-20T17:11:20.932 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.932 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7416f5d8:::benchmark_data_smithi171_78918_object140495:head# 2024-04-20T17:11:20.932 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.932 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74176ea6:::benchmark_data_smithi171_78918_object80734:head# 2024-04-20T17:11:20.932 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.932 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:741854bb:::benchmark_data_smithi171_78918_object80899:head# 2024-04-20T17:11:20.932 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.933 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:741896a1:::benchmark_data_smithi171_78918_object146358:head# 2024-04-20T17:11:20.933 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.933 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7419c2e9:::benchmark_data_smithi171_78918_object151481:head# 2024-04-20T17:11:20.933 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.933 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:741aeaa5:::benchmark_data_smithi171_78918_object152150:head# 2024-04-20T17:11:20.933 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.933 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:741b3b0b:::benchmark_data_smithi171_78918_object115664:head# 2024-04-20T17:11:20.933 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.933 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:741b8141:::benchmark_data_smithi171_78918_object66602:head# 2024-04-20T17:11:20.933 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.934 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:741c34c4:::benchmark_data_smithi171_78918_object153806:head# 2024-04-20T17:11:20.934 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.934 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:741c6aa8:::benchmark_data_smithi171_78918_object87202:head# 2024-04-20T17:11:20.934 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.934 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:741c8d6c:::benchmark_data_smithi171_78918_object88557:head# 2024-04-20T17:11:20.934 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.934 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:741d6394:::benchmark_data_smithi171_78918_object61602:head# 2024-04-20T17:11:20.934 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.934 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:741d6924:::benchmark_data_smithi171_78918_object117915:head# 2024-04-20T17:11:20.934 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.934 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:741f3267:::benchmark_data_smithi171_78918_object122162:head# 2024-04-20T17:11:20.934 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.934 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7422f3f7:::benchmark_data_smithi171_78918_object131516:head# 2024-04-20T17:11:20.934 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.935 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7423be70:::benchmark_data_smithi171_78918_object88067:head# 2024-04-20T17:11:20.935 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.935 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74253a92:::benchmark_data_smithi171_78918_object83989:head# 2024-04-20T17:11:20.935 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.935 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7425800f:::benchmark_data_smithi171_78918_object99539:head# 2024-04-20T17:11:20.935 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.935 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7425d823:::benchmark_data_smithi171_78918_object106131:head# 2024-04-20T17:11:20.935 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.935 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7426049f:::benchmark_data_smithi171_78918_object80973:head# 2024-04-20T17:11:20.935 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.935 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74262fd7:::benchmark_data_smithi171_78918_object160814:head# 2024-04-20T17:11:20.935 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.935 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74268afb:::benchmark_data_smithi171_78918_object132327:head# 2024-04-20T17:11:20.936 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.936 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7426d13a:::benchmark_data_smithi171_78918_object146827:head# 2024-04-20T17:11:20.936 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.936 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:742740f3:::benchmark_data_smithi171_78918_object159984:head# 2024-04-20T17:11:20.936 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.936 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74287d76:::benchmark_data_smithi171_78918_object117820:head# 2024-04-20T17:11:20.936 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.936 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7428a459:::benchmark_data_smithi171_78918_object55276:head# 2024-04-20T17:11:20.936 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.936 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74291cb5:::benchmark_data_smithi171_78918_object57146:head# 2024-04-20T17:11:20.936 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.936 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74293c93:::benchmark_data_smithi171_78918_object160801:head# 2024-04-20T17:11:20.936 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.936 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74294d95:::benchmark_data_smithi171_78918_object142283:head# 2024-04-20T17:11:20.937 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.937 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:742abfe6:::benchmark_data_smithi171_78918_object99647:head# 2024-04-20T17:11:20.937 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.937 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:742bb116:::benchmark_data_smithi171_78918_object52460:head# 2024-04-20T17:11:20.937 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.937 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:742cc3e3:::benchmark_data_smithi171_78918_object114515:head# 2024-04-20T17:11:20.937 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.937 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:742ec82a:::benchmark_data_smithi171_78918_object116138:head# 2024-04-20T17:11:20.937 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.937 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:742f9697:::benchmark_data_smithi171_78918_object135117:head# 2024-04-20T17:11:20.937 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.937 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7430cfea:::benchmark_data_smithi171_78918_object77636:head# 2024-04-20T17:11:20.937 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.938 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7431695d:::benchmark_data_smithi171_78918_object116058:head# 2024-04-20T17:11:20.938 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.938 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7431a6e5:::benchmark_data_smithi171_78918_object83484:head# 2024-04-20T17:11:20.938 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.938 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7432e34f:::benchmark_data_smithi171_78918_object56781:head# 2024-04-20T17:11:20.938 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.938 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74341726:::benchmark_data_smithi171_78918_object152761:head# 2024-04-20T17:11:20.938 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.938 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74342d5f:::benchmark_data_smithi171_78918_object80258:head# 2024-04-20T17:11:20.938 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.938 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7434d5d1:::benchmark_data_smithi171_78918_object53585:head# 2024-04-20T17:11:20.938 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.938 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7434dd58:::benchmark_data_smithi171_78918_object102023:head# 2024-04-20T17:11:20.938 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.939 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74353b1b:::benchmark_data_smithi171_78918_object125948:head# 2024-04-20T17:11:20.939 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.939 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7435dd33:::benchmark_data_smithi171_78918_object97291:head# 2024-04-20T17:11:20.939 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.939 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7437e67f:::benchmark_data_smithi171_78918_object138009:head# 2024-04-20T17:11:20.939 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.939 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7438c24c:::benchmark_data_smithi171_78918_object71921:head# 2024-04-20T17:11:20.939 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.939 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7438f7f2:::benchmark_data_smithi171_78918_object53580:head# 2024-04-20T17:11:20.939 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.939 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:743aa16d:::benchmark_data_smithi171_78918_object119182:head# 2024-04-20T17:11:20.939 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.939 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:743ac9ef:::benchmark_data_smithi171_78918_object113879:head# 2024-04-20T17:11:20.939 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.940 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:743b773b:::benchmark_data_smithi171_78918_object81965:head# 2024-04-20T17:11:20.940 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.940 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:743cd6df:::benchmark_data_smithi171_78918_object91121:head# 2024-04-20T17:11:20.940 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.940 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:743d5878:::benchmark_data_smithi171_78918_object149263:head# 2024-04-20T17:11:20.940 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.940 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:743da0b2:::benchmark_data_smithi171_78918_object100986:head# 2024-04-20T17:11:20.940 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.940 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:743f2058:::benchmark_data_smithi171_78918_object52039:head# 2024-04-20T17:11:20.940 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.940 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:743f5382:::benchmark_data_smithi171_78918_object112292:head# 2024-04-20T17:11:20.940 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.941 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7440a945:::benchmark_data_smithi171_78918_object89684:head# 2024-04-20T17:11:20.941 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.941 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:744135d4:::benchmark_data_smithi171_78918_object64789:head# 2024-04-20T17:11:20.941 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.941 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:744139b3:::benchmark_data_smithi171_78918_object79068:head# 2024-04-20T17:11:20.941 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.941 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74418db9:::benchmark_data_smithi171_78918_object77262:head# 2024-04-20T17:11:20.941 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.941 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7442aa6f:::benchmark_data_smithi171_78918_object131291:head# 2024-04-20T17:11:20.941 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.941 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7442f31d:::benchmark_data_smithi171_78918_object145465:head# 2024-04-20T17:11:20.941 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.941 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:744300f2:::benchmark_data_smithi171_78918_object51599:head# 2024-04-20T17:11:20.942 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.942 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7443d5ef:::benchmark_data_smithi171_78918_object147783:head# 2024-04-20T17:11:20.942 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.942 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74446779:::benchmark_data_smithi171_78918_object83787:head# 2024-04-20T17:11:20.942 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.942 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7444d817:::benchmark_data_smithi171_78918_object63269:head# 2024-04-20T17:11:20.942 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.942 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7444f01d:::benchmark_data_smithi171_78918_object78099:head# 2024-04-20T17:11:20.942 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.942 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:744567c2:::benchmark_data_smithi171_78918_object154787:head# 2024-04-20T17:11:20.942 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.942 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74462b05:::benchmark_data_smithi171_78918_object73250:head# 2024-04-20T17:11:20.942 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.942 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7446fee5:::benchmark_data_smithi171_78918_object56310:head# 2024-04-20T17:11:20.943 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.943 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74488eac:::benchmark_data_smithi171_78918_object89972:head# 2024-04-20T17:11:20.943 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.943 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7449b950:::benchmark_data_smithi171_78918_object126858:head# 2024-04-20T17:11:20.943 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.943 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:744a383c:::benchmark_data_smithi171_78918_object70809:head# 2024-04-20T17:11:20.943 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.943 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:744aa670:::benchmark_data_smithi171_78918_object83264:head# 2024-04-20T17:11:20.943 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.943 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:744ab235:::benchmark_data_smithi171_78918_object133657:head# 2024-04-20T17:11:20.943 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.943 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:744ac922:::benchmark_data_smithi171_78918_object73115:head# 2024-04-20T17:11:20.943 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.944 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:744c47ee:::benchmark_data_smithi171_78918_object120233:head# 2024-04-20T17:11:20.944 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.944 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:744daadc:::benchmark_data_smithi171_78918_object149758:head# 2024-04-20T17:11:20.944 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.944 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:744e916b:::benchmark_data_smithi171_78918_object156042:head# 2024-04-20T17:11:20.944 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.944 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:744f3258:::benchmark_data_smithi171_78918_object141834:head# 2024-04-20T17:11:20.944 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.944 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7450054c:::benchmark_data_smithi171_78918_object93233:head# 2024-04-20T17:11:20.944 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.945 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7450f8f0:::benchmark_data_smithi171_78918_object61176:head# 2024-04-20T17:11:20.945 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.945 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74512801:::benchmark_data_smithi171_78918_object131507:head# 2024-04-20T17:11:20.945 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.945 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74517053:::benchmark_data_smithi171_78918_object148156:head# 2024-04-20T17:11:20.945 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.945 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74523564:::benchmark_data_smithi171_78918_object105160:head# 2024-04-20T17:11:20.945 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.945 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7452e035:::benchmark_data_smithi171_78918_object151341:head# 2024-04-20T17:11:20.945 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.946 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7454f989:::benchmark_data_smithi171_78918_object76852:head# 2024-04-20T17:11:20.946 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.946 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74552236:::benchmark_data_smithi171_78918_object153312:head# 2024-04-20T17:11:20.946 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.946 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7455a63c:::benchmark_data_smithi171_78918_object157400:head# 2024-04-20T17:11:20.946 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.946 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7455fa01:::benchmark_data_smithi171_78918_object138965:head# 2024-04-20T17:11:20.946 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.946 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74564c95:::benchmark_data_smithi171_78918_object110121:head# 2024-04-20T17:11:20.946 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.946 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:745820f3:::benchmark_data_smithi171_78918_object55890:head# 2024-04-20T17:11:20.946 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.946 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74586996:::benchmark_data_smithi171_78918_object160147:head# 2024-04-20T17:11:20.947 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.947 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74588b2b:::benchmark_data_smithi171_78918_object124856:head# 2024-04-20T17:11:20.947 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.947 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7459a9c4:::benchmark_data_smithi171_78918_object142732:head# 2024-04-20T17:11:20.947 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.947 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7459e84d:::benchmark_data_smithi171_78918_object147996:head# 2024-04-20T17:11:20.947 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.947 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:745a9e69:::benchmark_data_smithi171_78918_object93468:head# 2024-04-20T17:11:20.947 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.947 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:745b2c6c:::benchmark_data_smithi171_78918_object71025:head# 2024-04-20T17:11:20.947 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.947 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:745b5099:::benchmark_data_smithi171_78918_object114981:head# 2024-04-20T17:11:20.947 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.947 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:745bfe1f:::benchmark_data_smithi171_78918_object75698:head# 2024-04-20T17:11:20.948 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.948 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:745d7489:::benchmark_data_smithi171_78918_object129614:head# 2024-04-20T17:11:20.948 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.948 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:745d8f60:::benchmark_data_smithi171_78918_object105440:head# 2024-04-20T17:11:20.948 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.948 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:745dbdff:::benchmark_data_smithi171_78918_object138644:head# 2024-04-20T17:11:20.948 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.948 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:745dec67:::benchmark_data_smithi171_78918_object121513:head# 2024-04-20T17:11:20.948 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.948 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74608ce6:::benchmark_data_smithi171_78918_object66270:head# 2024-04-20T17:11:20.948 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.948 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7461fae8:::benchmark_data_smithi171_78918_object107248:head# 2024-04-20T17:11:20.948 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.948 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74629941:::benchmark_data_smithi171_78918_object88932:head# 2024-04-20T17:11:20.949 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.949 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7462b6a6:::benchmark_data_smithi171_78918_object109554:head# 2024-04-20T17:11:20.949 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.949 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7462c068:::benchmark_data_smithi171_78918_object124105:head# 2024-04-20T17:11:20.949 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.949 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:746328a2:::benchmark_data_smithi171_78918_object90761:head# 2024-04-20T17:11:20.949 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.949 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74643f1a:::benchmark_data_smithi171_78918_object140747:head# 2024-04-20T17:11:20.949 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.949 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:746481f1:::benchmark_data_smithi171_78918_object72639:head# 2024-04-20T17:11:20.949 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.949 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7464a9d7:::benchmark_data_smithi171_78918_object86829:head# 2024-04-20T17:11:20.949 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.950 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74666899:::benchmark_data_smithi171_78918_object74184:head# 2024-04-20T17:11:20.950 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.950 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74667694:::benchmark_data_smithi171_78918_object66348:head# 2024-04-20T17:11:20.950 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.950 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74667e7f:::benchmark_data_smithi171_78918_object136773:head# 2024-04-20T17:11:20.950 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.950 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:746726f5:::benchmark_data_smithi171_78918_object152940:head# 2024-04-20T17:11:20.950 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.950 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74678a16:::benchmark_data_smithi171_78918_object88897:head# 2024-04-20T17:11:20.950 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.950 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7468e49c:::benchmark_data_smithi171_78918_object77103:head# 2024-04-20T17:11:20.950 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.951 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7468e52f:::benchmark_data_smithi171_78918_object157577:head# 2024-04-20T17:11:20.951 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.951 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7469db15:::benchmark_data_smithi171_78918_object145425:head# 2024-04-20T17:11:20.951 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.951 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:746b0a4a:::benchmark_data_smithi171_78918_object145790:head# 2024-04-20T17:11:20.951 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.951 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:746b403b:::benchmark_data_smithi171_78918_object105743:head# 2024-04-20T17:11:20.951 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.951 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:746bf146:::benchmark_data_smithi171_78918_object92698:head# 2024-04-20T17:11:20.951 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.951 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:746c3f8c:::benchmark_data_smithi171_78918_object95895:head# 2024-04-20T17:11:20.951 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.951 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:746cf2ea:::benchmark_data_smithi171_78918_object63974:head# 2024-04-20T17:11:20.951 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.952 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7470d3e9:::benchmark_data_smithi171_78918_object79884:head# 2024-04-20T17:11:20.952 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.952 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:747269db:::benchmark_data_smithi171_78918_object138355:head# 2024-04-20T17:11:20.952 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.952 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7472857e:::benchmark_data_smithi171_78918_object119676:head# 2024-04-20T17:11:20.952 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.952 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7472eafc:::benchmark_data_smithi171_78918_object95594:head# 2024-04-20T17:11:20.952 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.952 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74734166:::benchmark_data_smithi171_78918_object113899:head# 2024-04-20T17:11:20.952 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.952 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7474af29:::benchmark_data_smithi171_78918_object121464:head# 2024-04-20T17:11:20.952 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.952 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:747562e3:::benchmark_data_smithi171_78918_object109974:head# 2024-04-20T17:11:20.952 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.953 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7476bc54:::benchmark_data_smithi171_78918_object104081:head# 2024-04-20T17:11:20.953 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.953 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7477f5e4:::benchmark_data_smithi171_78918_object155542:head# 2024-04-20T17:11:20.953 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.953 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7478436e:::benchmark_data_smithi171_78918_object115473:head# 2024-04-20T17:11:20.953 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.953 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7478ff45:::benchmark_data_smithi171_78918_object151402:head# 2024-04-20T17:11:20.953 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.953 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74796db3:::benchmark_data_smithi171_78918_object54383:head# 2024-04-20T17:11:20.953 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.953 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:747990e3:::benchmark_data_smithi171_78918_object126141:head# 2024-04-20T17:11:20.953 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.953 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7479d76e:::benchmark_data_smithi171_78918_object66924:head# 2024-04-20T17:11:20.953 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.954 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:747a94d5:::benchmark_data_smithi171_78918_object119275:head# 2024-04-20T17:11:20.954 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.954 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:747c0909:::benchmark_data_smithi171_78918_object62084:head# 2024-04-20T17:11:20.954 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.954 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:747c41cc:::benchmark_data_smithi171_78918_object82748:head# 2024-04-20T17:11:20.954 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.954 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:747c6535:::benchmark_data_smithi171_78918_object119074:head# 2024-04-20T17:11:20.954 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.954 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:747c88d6:::benchmark_data_smithi171_78918_object135965:head# 2024-04-20T17:11:20.954 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.954 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:747caf3a:::benchmark_data_smithi171_78918_object146928:head# 2024-04-20T17:11:20.954 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.954 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:747cc2b7:::benchmark_data_smithi171_78918_object131179:head# 2024-04-20T17:11:20.954 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.955 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:747d75e6:::benchmark_data_smithi171_78918_object129559:head# 2024-04-20T17:11:20.955 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.955 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:747db7f5:::benchmark_data_smithi171_78918_object139625:head# 2024-04-20T17:11:20.955 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.955 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:747e864a:::benchmark_data_smithi171_78918_object77960:head# 2024-04-20T17:11:20.955 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.955 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74805053:::benchmark_data_smithi171_78918_object107467:head# 2024-04-20T17:11:20.955 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.955 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7480d232:::benchmark_data_smithi171_78918_object149357:head# 2024-04-20T17:11:20.955 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.955 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7482bb21:::benchmark_data_smithi171_78918_object56995:head# 2024-04-20T17:11:20.955 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.955 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74833c6a:::benchmark_data_smithi171_78918_object131949:head# 2024-04-20T17:11:20.955 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.956 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:748439eb:::benchmark_data_smithi171_78918_object155748:head# 2024-04-20T17:11:20.956 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.956 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7485cc8a:::benchmark_data_smithi171_78918_object152112:head# 2024-04-20T17:11:20.956 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.956 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7485e337:::benchmark_data_smithi171_78918_object105504:head# 2024-04-20T17:11:20.956 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.956 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74865980:::benchmark_data_smithi171_78918_object70396:head# 2024-04-20T17:11:20.956 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.956 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7486bb50:::benchmark_data_smithi171_78918_object153519:head# 2024-04-20T17:11:20.956 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.956 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74876e12:::benchmark_data_smithi171_78918_object79002:head# 2024-04-20T17:11:20.956 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.956 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7487d07a:::benchmark_data_smithi171_78918_object121225:head# 2024-04-20T17:11:20.956 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.956 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74881cda:::benchmark_data_smithi171_78918_object71343:head# 2024-04-20T17:11:20.957 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.957 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74888814:::benchmark_data_smithi171_78918_object75965:head# 2024-04-20T17:11:20.957 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.957 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7488c3b8:::benchmark_data_smithi171_78918_object112363:head# 2024-04-20T17:11:20.957 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.957 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7488ccb0:::benchmark_data_smithi171_78918_object73880:head# 2024-04-20T17:11:20.957 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.957 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7489e11e:::benchmark_data_smithi171_78918_object148994:head# 2024-04-20T17:11:20.957 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.957 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:748a8240:::benchmark_data_smithi171_78918_object115213:head# 2024-04-20T17:11:20.957 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.957 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:748abd3d:::benchmark_data_smithi171_78918_object144649:head# 2024-04-20T17:11:20.957 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.957 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:748b1167:::benchmark_data_smithi171_78918_object76143:head# 2024-04-20T17:11:20.957 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.958 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:748b16fa:::benchmark_data_smithi171_78918_object72419:head# 2024-04-20T17:11:20.958 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.958 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:748b2f75:::benchmark_data_smithi171_78918_object95192:head# 2024-04-20T17:11:20.958 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.958 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:748c8d39:::benchmark_data_smithi171_78918_object158084:head# 2024-04-20T17:11:20.958 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.958 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:748d12bc:::benchmark_data_smithi171_78918_object160358:head# 2024-04-20T17:11:20.958 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.958 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:748e558e:::benchmark_data_smithi171_78918_object104084:head# 2024-04-20T17:11:20.958 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.958 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:748f2d12:::benchmark_data_smithi171_78918_object119091:head# 2024-04-20T17:11:20.958 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.958 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:748fe9d7:::benchmark_data_smithi171_78918_object67875:head# 2024-04-20T17:11:20.958 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.959 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74901fba:::benchmark_data_smithi171_78918_object63041:head# 2024-04-20T17:11:20.959 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.959 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7490abf4:::benchmark_data_smithi171_78918_object146406:head# 2024-04-20T17:11:20.959 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.959 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7491cf2f:::benchmark_data_smithi171_78918_object153989:head# 2024-04-20T17:11:20.959 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.959 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:749363b7:::benchmark_data_smithi171_78918_object152522:head# 2024-04-20T17:11:20.959 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.959 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74944366:::benchmark_data_smithi171_78918_object86941:head# 2024-04-20T17:11:20.959 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.959 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:749520b7:::benchmark_data_smithi171_78918_object112386:head# 2024-04-20T17:11:20.959 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.959 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7495ddb4:::benchmark_data_smithi171_78918_object146462:head# 2024-04-20T17:11:20.959 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.960 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:749755e5:::benchmark_data_smithi171_78918_object96829:head# 2024-04-20T17:11:20.960 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.960 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74980855:::benchmark_data_smithi171_78918_object95438:head# 2024-04-20T17:11:20.960 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.960 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74988126:::benchmark_data_smithi171_78918_object128603:head# 2024-04-20T17:11:20.960 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.960 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7498f2ce:::benchmark_data_smithi171_78918_object82273:head# 2024-04-20T17:11:20.960 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.960 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74991fc8:::benchmark_data_smithi171_78918_object149291:head# 2024-04-20T17:11:20.960 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.960 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:749a79d4:::benchmark_data_smithi171_78918_object83192:head# 2024-04-20T17:11:20.960 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.960 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:749be937:::benchmark_data_smithi171_78918_object133526:head# 2024-04-20T17:11:20.960 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.961 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:749c3aec:::benchmark_data_smithi171_78918_object87255:head# 2024-04-20T17:11:20.961 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.961 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:749c675c:::benchmark_data_smithi171_78918_object74574:head# 2024-04-20T17:11:20.961 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.961 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:749c94a5:::benchmark_data_smithi171_78918_object159384:head# 2024-04-20T17:11:20.961 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.961 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:749cf490:::benchmark_data_smithi171_78918_object103006:head# 2024-04-20T17:11:20.961 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.961 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:749d0d3e:::benchmark_data_smithi171_78918_object81134:head# 2024-04-20T17:11:20.961 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.961 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:749dd95e:::benchmark_data_smithi171_78918_object155781:head# 2024-04-20T17:11:20.961 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.961 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:749f33bc:::benchmark_data_smithi171_78918_object150459:head# 2024-04-20T17:11:20.961 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.961 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:749f6647:::benchmark_data_smithi171_78918_object56602:head# 2024-04-20T17:11:20.962 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.962 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:749ff1ca:::benchmark_data_smithi171_78918_object154907:head# 2024-04-20T17:11:20.962 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.962 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74a012f0:::benchmark_data_smithi171_78918_object61308:head# 2024-04-20T17:11:20.962 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.962 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74a0f5cf:::benchmark_data_smithi171_78918_object115809:head# 2024-04-20T17:11:20.962 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.962 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74a25725:::benchmark_data_smithi171_78918_object153743:head# 2024-04-20T17:11:20.962 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.962 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74a2918d:::benchmark_data_smithi171_78918_object115721:head# 2024-04-20T17:11:20.962 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.962 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74a33215:::benchmark_data_smithi171_78918_object111194:head# 2024-04-20T17:11:20.962 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.962 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74a33aa9:::benchmark_data_smithi171_78918_object135398:head# 2024-04-20T17:11:20.963 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.963 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74a3796a:::benchmark_data_smithi171_78918_object86864:head# 2024-04-20T17:11:20.963 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.963 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74a46e9c:::benchmark_data_smithi171_78918_object60458:head# 2024-04-20T17:11:20.963 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.963 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74a59b89:::benchmark_data_smithi171_78918_object113946:head# 2024-04-20T17:11:20.963 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.963 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74a5eabb:::benchmark_data_smithi171_78918_object141545:head# 2024-04-20T17:11:20.963 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.963 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74a6209f:::benchmark_data_smithi171_78918_object104719:head# 2024-04-20T17:11:20.963 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.963 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74a6563a:::benchmark_data_smithi171_78918_object131787:head# 2024-04-20T17:11:20.963 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.963 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74a6e28a:::benchmark_data_smithi171_78918_object57935:head# 2024-04-20T17:11:20.964 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.964 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74a75380:::benchmark_data_smithi171_78918_object96611:head# 2024-04-20T17:11:20.964 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.964 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74a7771c:::benchmark_data_smithi171_78918_object94426:head# 2024-04-20T17:11:20.964 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.964 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74a804cd:::benchmark_data_smithi171_78918_object106665:head# 2024-04-20T17:11:20.964 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.964 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74a872ec:::benchmark_data_smithi171_78918_object54430:head# 2024-04-20T17:11:20.964 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.964 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74a87cb5:::benchmark_data_smithi171_78918_object95235:head# 2024-04-20T17:11:20.964 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.964 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74a8ff72:::benchmark_data_smithi171_78918_object147277:head# 2024-04-20T17:11:20.964 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.965 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74a97616:::benchmark_data_smithi171_78918_object72233:head# 2024-04-20T17:11:20.965 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.965 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74a97e73:::benchmark_data_smithi171_78918_object107632:head# 2024-04-20T17:11:20.965 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.965 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74a9a979:::benchmark_data_smithi171_78918_object91736:head# 2024-04-20T17:11:20.965 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.965 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74aae40f:::benchmark_data_smithi171_78918_object109042:head# 2024-04-20T17:11:20.965 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.965 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74ac1c02:::benchmark_data_smithi171_78918_object156804:head# 2024-04-20T17:11:20.965 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.965 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74ac2e7a:::benchmark_data_smithi171_78918_object123503:head# 2024-04-20T17:11:20.965 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.965 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74ac9c73:::benchmark_data_smithi171_78918_object76171:head# 2024-04-20T17:11:20.965 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.965 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74acbf08:::benchmark_data_smithi171_78918_object85489:head# 2024-04-20T17:11:20.966 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.966 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74ad16d4:::benchmark_data_smithi171_78918_object56125:head# 2024-04-20T17:11:20.966 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.966 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74ae1c1e:::benchmark_data_smithi171_78918_object122197:head# 2024-04-20T17:11:20.966 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.966 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74ae5a00:::benchmark_data_smithi171_78918_object130057:head# 2024-04-20T17:11:20.966 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.966 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74ae690a:::benchmark_data_smithi171_78918_object115814:head# 2024-04-20T17:11:20.966 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.966 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74ae95f5:::benchmark_data_smithi171_78918_object132892:head# 2024-04-20T17:11:20.966 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.966 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74afb660:::benchmark_data_smithi171_78918_object122222:head# 2024-04-20T17:11:20.966 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.966 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74afe660:::benchmark_data_smithi171_78918_object79953:head# 2024-04-20T17:11:20.967 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.967 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74b01278:::benchmark_data_smithi171_78918_object155158:head# 2024-04-20T17:11:20.967 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.967 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74b0d9ae:::benchmark_data_smithi171_78918_object79595:head# 2024-04-20T17:11:20.967 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.967 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74b155d2:::benchmark_data_smithi171_78918_object79346:head# 2024-04-20T17:11:20.967 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.967 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74b202ed:::benchmark_data_smithi171_78918_object94004:head# 2024-04-20T17:11:20.967 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.967 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74b39ab7:::benchmark_data_smithi171_78918_object57740:head# 2024-04-20T17:11:20.967 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.967 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74b492cd:::benchmark_data_smithi171_78918_object155287:head# 2024-04-20T17:11:20.967 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.967 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74b4d785:::benchmark_data_smithi171_78918_object156341:head# 2024-04-20T17:11:20.968 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.968 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74b55b25:::benchmark_data_smithi171_78918_object63639:head# 2024-04-20T17:11:20.968 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.968 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74b56c89:::benchmark_data_smithi171_78918_object145598:head# 2024-04-20T17:11:20.968 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.968 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74b58aca:::benchmark_data_smithi171_78918_object98799:head# 2024-04-20T17:11:20.968 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.968 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74b5c73d:::benchmark_data_smithi171_78918_object125116:head# 2024-04-20T17:11:20.968 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.968 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74b62b5a:::benchmark_data_smithi171_78918_object128650:head# 2024-04-20T17:11:20.968 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.968 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74b6f633:::benchmark_data_smithi171_78918_object97057:head# 2024-04-20T17:11:20.968 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.968 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74b7dccc:::benchmark_data_smithi171_78918_object60052:head# 2024-04-20T17:11:20.968 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.969 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74b80067:::benchmark_data_smithi171_78918_object148904:head# 2024-04-20T17:11:20.969 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.969 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74b8f0f0:::benchmark_data_smithi171_78918_object60970:head# 2024-04-20T17:11:20.969 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.969 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74b935d6:::benchmark_data_smithi171_78918_object81287:head# 2024-04-20T17:11:20.969 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.969 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74ba78e4:::benchmark_data_smithi171_78918_object148288:head# 2024-04-20T17:11:20.969 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.969 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74bab2ed:::benchmark_data_smithi171_78918_object104357:head# 2024-04-20T17:11:20.969 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.969 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74bb1879:::benchmark_data_smithi171_78918_object91481:head# 2024-04-20T17:11:20.969 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.969 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74bb889a:::benchmark_data_smithi171_78918_object65609:head# 2024-04-20T17:11:20.969 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.970 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74bbc173:::benchmark_data_smithi171_78918_object144490:head# 2024-04-20T17:11:20.970 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.970 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74bc4430:::benchmark_data_smithi171_78918_object106264:head# 2024-04-20T17:11:20.970 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.970 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74be3971:::benchmark_data_smithi171_78918_object119563:head# 2024-04-20T17:11:20.970 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.970 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74bf2671:::benchmark_data_smithi171_78918_object93411:head# 2024-04-20T17:11:20.970 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.970 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74c08515:::benchmark_data_smithi171_78918_object112385:head# 2024-04-20T17:11:20.970 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.970 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74c10e36:::benchmark_data_smithi171_78918_object134551:head# 2024-04-20T17:11:20.970 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.970 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74c1502d:::benchmark_data_smithi171_78918_object63465:head# 2024-04-20T17:11:20.970 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.971 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74c19511:::benchmark_data_smithi171_78918_object113621:head# 2024-04-20T17:11:20.971 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.971 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74c224e0:::benchmark_data_smithi171_78918_object110594:head# 2024-04-20T17:11:20.971 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.971 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74c24d87:::benchmark_data_smithi171_78918_object159641:head# 2024-04-20T17:11:20.971 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.971 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74c25ec5:::benchmark_data_smithi171_78918_object78805:head# 2024-04-20T17:11:20.971 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.971 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74c3d220:::benchmark_data_smithi171_78918_object69677:head# 2024-04-20T17:11:20.971 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.971 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74c46019:::benchmark_data_smithi171_78918_object94839:head# 2024-04-20T17:11:20.971 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.971 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74c4ba36:::benchmark_data_smithi171_78918_object156903:head# 2024-04-20T17:11:20.971 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.971 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74c5537f:::benchmark_data_smithi171_78918_object97139:head# 2024-04-20T17:11:20.972 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.972 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74c62277:::benchmark_data_smithi171_78918_object134764:head# 2024-04-20T17:11:20.972 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.972 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74c77e06:::benchmark_data_smithi171_78918_object85190:head# 2024-04-20T17:11:20.972 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.972 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74c8cf6f:::benchmark_data_smithi171_78918_object149793:head# 2024-04-20T17:11:20.972 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.972 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74c93af6:::benchmark_data_smithi171_78918_object73861:head# 2024-04-20T17:11:20.972 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.972 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74c99bff:::benchmark_data_smithi171_78918_object76825:head# 2024-04-20T17:11:20.972 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.972 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74ca2c40:::benchmark_data_smithi171_78918_object96769:head# 2024-04-20T17:11:20.972 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.972 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74cab782:::benchmark_data_smithi171_78918_object109325:head# 2024-04-20T17:11:20.973 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.973 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74caeb9f:::benchmark_data_smithi171_78918_object57703:head# 2024-04-20T17:11:20.973 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.973 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74cb62a0:::benchmark_data_smithi171_78918_object94227:head# 2024-04-20T17:11:20.973 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.973 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74cc1c3f:::benchmark_data_smithi171_78918_object76738:head# 2024-04-20T17:11:20.973 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.973 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74cd428a:::benchmark_data_smithi171_78918_object88711:head# 2024-04-20T17:11:20.973 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.973 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74ce99a3:::benchmark_data_smithi171_78918_object56360:head# 2024-04-20T17:11:20.973 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.973 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74cebaf1:::benchmark_data_smithi171_78918_object149046:head# 2024-04-20T17:11:20.973 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.973 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74cef167:::benchmark_data_smithi171_78918_object71415:head# 2024-04-20T17:11:20.973 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.974 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74d07826:::benchmark_data_smithi171_78918_object85159:head# 2024-04-20T17:11:20.974 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.974 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74d08054:::benchmark_data_smithi171_78918_object59162:head# 2024-04-20T17:11:20.974 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.974 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74d0b40e:::benchmark_data_smithi171_78918_object142975:head# 2024-04-20T17:11:20.974 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.974 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74d110f2:::benchmark_data_smithi171_78918_object58433:head# 2024-04-20T17:11:20.974 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.974 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74d1da19:::benchmark_data_smithi171_78918_object53001:head# 2024-04-20T17:11:20.974 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.974 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74d1ff92:::benchmark_data_smithi171_78918_object155081:head# 2024-04-20T17:11:20.974 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.974 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74d2d676:::benchmark_data_smithi171_78918_object130201:head# 2024-04-20T17:11:20.975 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.975 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74d2e401:::benchmark_data_smithi171_78918_object111693:head# 2024-04-20T17:11:20.975 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.975 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74d3d5aa:::benchmark_data_smithi171_78918_object71061:head# 2024-04-20T17:11:20.975 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.975 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74d3eb7d:::benchmark_data_smithi171_78918_object84626:head# 2024-04-20T17:11:20.975 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.975 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74d4efdf:::benchmark_data_smithi171_78918_object138357:head# 2024-04-20T17:11:20.975 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.975 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74d572f2:::benchmark_data_smithi171_78918_object122332:head# 2024-04-20T17:11:20.975 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.975 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74d590ae:::benchmark_data_smithi171_78918_object89041:head# 2024-04-20T17:11:20.975 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.975 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74d59cb7:::benchmark_data_smithi171_78918_object68431:head# 2024-04-20T17:11:20.975 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.976 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74d5ff05:::benchmark_data_smithi171_78918_object113865:head# 2024-04-20T17:11:20.976 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.976 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74d6cb55:::benchmark_data_smithi171_78918_object89330:head# 2024-04-20T17:11:20.976 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.976 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74d6d1d3:::benchmark_data_smithi171_78918_object128463:head# 2024-04-20T17:11:20.976 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.976 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74d76f33:::benchmark_data_smithi171_78918_object98736:head# 2024-04-20T17:11:20.976 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.976 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74d7dacd:::benchmark_data_smithi171_78918_object113891:head# 2024-04-20T17:11:20.976 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.976 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74d8f9a0:::benchmark_data_smithi171_78918_object103789:head# 2024-04-20T17:11:20.976 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.976 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74d96fd2:::benchmark_data_smithi171_78918_object88079:head# 2024-04-20T17:11:20.977 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.977 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74d99bd4:::benchmark_data_smithi171_78918_object66238:head# 2024-04-20T17:11:20.977 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.977 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74d9dba6:::benchmark_data_smithi171_78918_object142577:head# 2024-04-20T17:11:20.977 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.977 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74da2f3e:::benchmark_data_smithi171_78918_object139846:head# 2024-04-20T17:11:20.977 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.977 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74da2fd7:::benchmark_data_smithi171_78918_object108529:head# 2024-04-20T17:11:20.977 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.977 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74da6bf2:::benchmark_data_smithi171_78918_object147014:head# 2024-04-20T17:11:20.977 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.977 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74dad6a7:::benchmark_data_smithi171_78918_object117236:head# 2024-04-20T17:11:20.977 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.977 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74db5432:::benchmark_data_smithi171_78918_object124795:head# 2024-04-20T17:11:20.978 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.978 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74dc60d0:::benchmark_data_smithi171_78918_object106109:head# 2024-04-20T17:11:20.978 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.978 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74dcbaa1:::benchmark_data_smithi171_78918_object89853:head# 2024-04-20T17:11:20.978 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.978 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74dd4756:::benchmark_data_smithi171_78918_object129696:head# 2024-04-20T17:11:20.978 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.978 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74ddc84b:::benchmark_data_smithi171_78918_object102143:head# 2024-04-20T17:11:20.978 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.978 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74de7876:::benchmark_data_smithi171_78918_object100885:head# 2024-04-20T17:11:20.978 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.978 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74deadcf:::benchmark_data_smithi171_78918_object153579:head# 2024-04-20T17:11:20.978 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.978 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74df0541:::benchmark_data_smithi171_78918_object52968:head# 2024-04-20T17:11:20.979 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.979 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74df2cf5:::benchmark_data_smithi171_78918_object67328:head# 2024-04-20T17:11:20.979 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.979 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74df71b2:::benchmark_data_smithi171_78918_object70091:head# 2024-04-20T17:11:20.979 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.979 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74e03769:::benchmark_data_smithi171_78918_object63976:head# 2024-04-20T17:11:20.979 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.979 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74e06de0:::benchmark_data_smithi171_78918_object76698:head# 2024-04-20T17:11:20.979 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.979 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74e129ab:::benchmark_data_smithi171_78918_object75286:head# 2024-04-20T17:11:20.980 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.980 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74e1b168:::benchmark_data_smithi171_78918_object55620:head# 2024-04-20T17:11:20.980 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.980 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74e1fead:::benchmark_data_smithi171_78918_object129011:head# 2024-04-20T17:11:20.980 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.980 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74e43d12:::benchmark_data_smithi171_78918_object74497:head# 2024-04-20T17:11:20.980 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.980 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74e45c64:::benchmark_data_smithi171_78918_object72112:head# 2024-04-20T17:11:20.980 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.980 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74e5f60a:::benchmark_data_smithi171_78918_object132414:head# 2024-04-20T17:11:20.981 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.981 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74e6109f:::benchmark_data_smithi171_78918_object84612:head# 2024-04-20T17:11:20.981 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.981 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74e61870:::benchmark_data_smithi171_78918_object136068:head# 2024-04-20T17:11:20.981 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.981 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74e63d5d:::benchmark_data_smithi171_78918_object56384:head# 2024-04-20T17:11:20.981 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.981 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74e64259:::benchmark_data_smithi171_78918_object117516:head# 2024-04-20T17:11:20.981 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.981 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74e7d41d:::benchmark_data_smithi171_78918_object149825:head# 2024-04-20T17:11:20.981 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.981 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74e828b9:::benchmark_data_smithi171_78918_object140835:head# 2024-04-20T17:11:20.982 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.982 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74e95ce9:::benchmark_data_smithi171_78918_object57822:head# 2024-04-20T17:11:20.982 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.982 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74e9b562:::benchmark_data_smithi171_78918_object120634:head# 2024-04-20T17:11:20.982 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.982 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74eadfad:::benchmark_data_smithi171_78918_object51798:head# 2024-04-20T17:11:20.982 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.982 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74ebbb4f:::benchmark_data_smithi171_78918_object126058:head# 2024-04-20T17:11:20.982 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.982 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74ec17e3:::benchmark_data_smithi171_78918_object144190:head# 2024-04-20T17:11:20.982 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.982 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74ec2a7e:::benchmark_data_smithi171_78918_object154044:head# 2024-04-20T17:11:20.982 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.982 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74ecb9e5:::benchmark_data_smithi171_78918_object59721:head# 2024-04-20T17:11:20.983 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.983 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74ecc54f:::benchmark_data_smithi171_78918_object96065:head# 2024-04-20T17:11:20.983 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.983 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74ed66ca:::benchmark_data_smithi171_78918_object84304:head# 2024-04-20T17:11:20.983 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.983 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74edeaa6:::benchmark_data_smithi171_78918_object126314:head# 2024-04-20T17:11:20.983 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.983 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74eeb926:::benchmark_data_smithi171_78918_object99526:head# 2024-04-20T17:11:20.983 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.983 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74f01619:::benchmark_data_smithi171_78918_object134414:head# 2024-04-20T17:11:20.983 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.983 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74f16c78:::benchmark_data_smithi171_78918_object58326:head# 2024-04-20T17:11:20.983 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.983 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74f23d6e:::benchmark_data_smithi171_78918_object93151:head# 2024-04-20T17:11:20.984 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.984 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74f25ae9:::benchmark_data_smithi171_78918_object107336:head# 2024-04-20T17:11:20.984 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.984 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74f2d24c:::benchmark_data_smithi171_78918_object90775:head# 2024-04-20T17:11:20.984 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.984 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74f2d4e4:::benchmark_data_smithi171_78918_object99410:head# 2024-04-20T17:11:20.984 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.984 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74f420bf:::benchmark_data_smithi171_78918_object142724:head# 2024-04-20T17:11:20.984 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.984 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74f69b28:::benchmark_data_smithi171_78918_object147163:head# 2024-04-20T17:11:20.984 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.984 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74f6efa7:::benchmark_data_smithi171_78918_object60936:head# 2024-04-20T17:11:20.984 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.984 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74f756d3:::benchmark_data_smithi171_78918_object121831:head# 2024-04-20T17:11:20.984 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.985 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74f8004b:::benchmark_data_smithi171_78918_object87000:head# 2024-04-20T17:11:20.985 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.985 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74f834a0:::benchmark_data_smithi171_78918_object60512:head# 2024-04-20T17:11:20.985 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.985 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74f9e695:::benchmark_data_smithi171_78918_object120596:head# 2024-04-20T17:11:20.985 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.985 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74fae7c2:::benchmark_data_smithi171_78918_object58679:head# 2024-04-20T17:11:20.985 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.985 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74fafd0f:::benchmark_data_smithi171_78918_object123346:head# 2024-04-20T17:11:20.985 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.985 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74fbfef0:::benchmark_data_smithi171_78918_object73089:head# 2024-04-20T17:11:20.985 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.985 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74fceeaf:::benchmark_data_smithi171_78918_object56426:head# 2024-04-20T17:11:20.985 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.985 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74fcf88a:::benchmark_data_smithi171_78918_object143101:head# 2024-04-20T17:11:20.986 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.986 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74fdf3ba:::benchmark_data_smithi171_78918_object114078:head# 2024-04-20T17:11:20.986 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.986 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74ff8c6b:::benchmark_data_smithi171_78918_object58771:head# 2024-04-20T17:11:20.986 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.986 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:74ffbd12:::benchmark_data_smithi171_78918_object70976:head# 2024-04-20T17:11:20.986 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.986 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:750003dc:::benchmark_data_smithi171_78918_object76280:head# 2024-04-20T17:11:20.986 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.986 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75016edd:::benchmark_data_smithi171_78918_object54655:head# 2024-04-20T17:11:20.986 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.987 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7503095f:::benchmark_data_smithi171_78918_object117703:head# 2024-04-20T17:11:20.987 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.987 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7504e6c3:::benchmark_data_smithi171_78918_object144909:head# 2024-04-20T17:11:20.987 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.987 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:750624b1:::benchmark_data_smithi171_78918_object117058:head# 2024-04-20T17:11:20.987 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.987 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75063539:::benchmark_data_smithi171_78918_object102192:head# 2024-04-20T17:11:20.987 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.987 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75066cd0:::benchmark_data_smithi171_78918_object154841:head# 2024-04-20T17:11:20.987 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.987 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75066d4b:::benchmark_data_smithi171_78918_object80788:head# 2024-04-20T17:11:20.987 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.988 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:750876f0:::benchmark_data_smithi171_78918_object122995:head# 2024-04-20T17:11:20.988 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.988 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75090bc0:::benchmark_data_smithi171_78918_object97451:head# 2024-04-20T17:11:20.988 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.988 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75092a3e:::benchmark_data_smithi171_78918_object68743:head# 2024-04-20T17:11:20.988 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.988 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:750a6f76:::benchmark_data_smithi171_78918_object136740:head# 2024-04-20T17:11:20.988 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.988 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:750a7d1e:::benchmark_data_smithi171_78918_object93155:head# 2024-04-20T17:11:20.988 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.988 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:750bef35:::benchmark_data_smithi171_78918_object109698:head# 2024-04-20T17:11:20.988 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.988 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:750cea33:::benchmark_data_smithi171_78918_object160634:head# 2024-04-20T17:11:20.988 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.989 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:750df830:::benchmark_data_smithi171_78918_object155102:head# 2024-04-20T17:11:20.989 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.989 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:750ed6eb:::benchmark_data_smithi171_78918_object101727:head# 2024-04-20T17:11:20.989 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.989 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:750f604e:::benchmark_data_smithi171_78918_object113184:head# 2024-04-20T17:11:20.989 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.989 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:750f8135:::benchmark_data_smithi171_78918_object127669:head# 2024-04-20T17:11:20.989 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.989 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:750fc5df:::benchmark_data_smithi171_78918_object116124:head# 2024-04-20T17:11:20.989 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.989 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:750ffe2f:::benchmark_data_smithi171_78918_object89231:head# 2024-04-20T17:11:20.989 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.989 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:751070f1:::benchmark_data_smithi171_78918_object141372:head# 2024-04-20T17:11:20.989 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.990 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7510859a:::benchmark_data_smithi171_78918_object97633:head# 2024-04-20T17:11:20.990 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.990 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7512e0b5:::benchmark_data_smithi171_78918_object82666:head# 2024-04-20T17:11:20.990 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.990 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:751379c4:::benchmark_data_smithi171_78918_object95383:head# 2024-04-20T17:11:20.990 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.990 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7513a82b:::benchmark_data_smithi171_78918_object150107:head# 2024-04-20T17:11:20.990 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.990 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7514ebae:::benchmark_data_smithi171_78918_object142200:head# 2024-04-20T17:11:20.990 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.990 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7514f192:::benchmark_data_smithi171_78918_object76756:head# 2024-04-20T17:11:20.990 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.993 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75151a00:::benchmark_data_smithi171_78918_object57194:head# 2024-04-20T17:11:20.993 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.993 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75158d51:::benchmark_data_smithi171_78918_object135653:head# 2024-04-20T17:11:20.993 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.994 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7515c183:::benchmark_data_smithi171_78918_object63899:head# 2024-04-20T17:11:20.994 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.994 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7515dd9c:::benchmark_data_smithi171_78918_object131764:head# 2024-04-20T17:11:20.994 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.994 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7517bee5:::benchmark_data_smithi171_78918_object150550:head# 2024-04-20T17:11:20.994 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.994 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75186b4a:::benchmark_data_smithi171_78918_object76645:head# 2024-04-20T17:11:20.994 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.994 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7518de7f:::benchmark_data_smithi171_78918_object87755:head# 2024-04-20T17:11:20.994 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.994 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:751ada3b:::benchmark_data_smithi171_78918_object56407:head# 2024-04-20T17:11:20.995 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.995 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:751b147b:::benchmark_data_smithi171_78918_object68606:head# 2024-04-20T17:11:20.995 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.995 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:751b16c1:::benchmark_data_smithi171_78918_object122793:head# 2024-04-20T17:11:20.995 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.995 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:751d2158:::benchmark_data_smithi171_78918_object54458:head# 2024-04-20T17:11:20.995 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.995 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:751dfe90:::benchmark_data_smithi171_78918_object63871:head# 2024-04-20T17:11:20.995 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.995 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:751f30f2:::benchmark_data_smithi171_78918_object153675:head# 2024-04-20T17:11:20.995 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.996 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:751f3c6c:::benchmark_data_smithi171_78918_object69310:head# 2024-04-20T17:11:20.996 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.996 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:751f7266:::benchmark_data_smithi171_78918_object136393:head# 2024-04-20T17:11:20.996 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.996 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:751f86e0:::benchmark_data_smithi171_78918_object151040:head# 2024-04-20T17:11:20.996 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.996 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75200f1c:::benchmark_data_smithi171_78918_object91732:head# 2024-04-20T17:11:20.996 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.996 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75203ec3:::benchmark_data_smithi171_78918_object114432:head# 2024-04-20T17:11:20.997 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.997 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75215229:::benchmark_data_smithi171_78918_object146031:head# 2024-04-20T17:11:20.997 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.997 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7521b3d3:::benchmark_data_smithi171_78918_object139631:head# 2024-04-20T17:11:20.997 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.997 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7521e6ee:::benchmark_data_smithi171_78918_object120119:head# 2024-04-20T17:11:20.997 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.997 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75227344:::benchmark_data_smithi171_78918_object155997:head# 2024-04-20T17:11:20.997 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.997 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7522b527:::benchmark_data_smithi171_78918_object156362:head# 2024-04-20T17:11:20.997 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.998 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7522c516:::benchmark_data_smithi171_78918_object92695:head# 2024-04-20T17:11:20.998 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.998 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7522d0fd:::benchmark_data_smithi171_78918_object144255:head# 2024-04-20T17:11:20.998 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.998 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7522d4ff:::benchmark_data_smithi171_78918_object117629:head# 2024-04-20T17:11:20.998 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.998 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7522fce9:::benchmark_data_smithi171_78918_object101481:head# 2024-04-20T17:11:20.998 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.998 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7523db4e:::benchmark_data_smithi171_78918_object140788:head# 2024-04-20T17:11:20.998 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.998 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7527806a:::benchmark_data_smithi171_78918_object100284:head# 2024-04-20T17:11:20.998 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.999 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7527e55c:::benchmark_data_smithi171_78918_object148647:head# 2024-04-20T17:11:20.999 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.999 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7528bab1:::benchmark_data_smithi171_78918_object80644:head# 2024-04-20T17:11:20.999 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.999 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:752907ee:::benchmark_data_smithi171_78918_object154084:head# 2024-04-20T17:11:20.999 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.999 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7529bf2e:::benchmark_data_smithi171_78918_object86880:head# 2024-04-20T17:11:20.999 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.999 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:752bc499:::benchmark_data_smithi171_78918_object138595:head# 2024-04-20T17:11:20.999 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.999 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:752d134e:::benchmark_data_smithi171_78918_object53398:head# 2024-04-20T17:11:20.999 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:20.999 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:752d1437:::benchmark_data_smithi171_78918_object56157:head# 2024-04-20T17:11:20.999 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.000 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:752d4e38:::benchmark_data_smithi171_78918_object66250:head# 2024-04-20T17:11:21.000 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.000 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:752d60d0:::benchmark_data_smithi171_78918_object87118:head# 2024-04-20T17:11:21.000 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.000 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:752d7141:::benchmark_data_smithi171_78918_object57749:head# 2024-04-20T17:11:21.000 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.000 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:752dd864:::benchmark_data_smithi171_78918_object152478:head# 2024-04-20T17:11:21.000 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.000 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:752dfd7c:::benchmark_data_smithi171_78918_object76550:head# 2024-04-20T17:11:21.000 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.000 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:752ef716:::benchmark_data_smithi171_78918_object106851:head# 2024-04-20T17:11:21.000 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.000 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:752f9594:::benchmark_data_smithi171_78918_object121452:head# 2024-04-20T17:11:21.001 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.001 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7530b14a:::benchmark_data_smithi171_78918_object136278:head# 2024-04-20T17:11:21.001 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.001 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75310406:::benchmark_data_smithi171_78918_object118933:head# 2024-04-20T17:11:21.001 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.001 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7532a0da:::benchmark_data_smithi171_78918_object107024:head# 2024-04-20T17:11:21.001 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.001 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75336909:::benchmark_data_smithi171_78918_object115665:head# 2024-04-20T17:11:21.001 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.001 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7534b206:::benchmark_data_smithi171_78918_object141350:head# 2024-04-20T17:11:21.001 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.002 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7535101d:::benchmark_data_smithi171_78918_object81376:head# 2024-04-20T17:11:21.002 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.002 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75351191:::benchmark_data_smithi171_78918_object64211:head# 2024-04-20T17:11:21.002 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.002 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75353192:::benchmark_data_smithi171_78918_object65292:head# 2024-04-20T17:11:21.002 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.002 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75358610:::benchmark_data_smithi171_78918_object59432:head# 2024-04-20T17:11:21.002 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.002 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7535a46a:::benchmark_data_smithi171_78918_object115662:head# 2024-04-20T17:11:21.002 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.003 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75360450:::benchmark_data_smithi171_78918_object60867:head# 2024-04-20T17:11:21.003 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.003 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75378b8d:::benchmark_data_smithi171_78918_object158293:head# 2024-04-20T17:11:21.003 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.003 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7537f0a3:::benchmark_data_smithi171_78918_object105711:head# 2024-04-20T17:11:21.003 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.003 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:753909d3:::benchmark_data_smithi171_78918_object157528:head# 2024-04-20T17:11:21.003 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.003 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:753a428e:::benchmark_data_smithi171_78918_object159591:head# 2024-04-20T17:11:21.003 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.003 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:753b2848:::benchmark_data_smithi171_78918_object154866:head# 2024-04-20T17:11:21.003 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.004 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:753bc389:::benchmark_data_smithi171_78918_object159981:head# 2024-04-20T17:11:21.004 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.004 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:753c010b:::benchmark_data_smithi171_78918_object148870:head# 2024-04-20T17:11:21.004 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.004 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:753ec688:::benchmark_data_smithi171_78918_object61220:head# 2024-04-20T17:11:21.004 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.004 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:753eee16:::benchmark_data_smithi171_78918_object120929:head# 2024-04-20T17:11:21.004 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.004 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:753f0751:::benchmark_data_smithi171_78918_object106614:head# 2024-04-20T17:11:21.004 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.004 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:753f756b:::benchmark_data_smithi171_78918_object88455:head# 2024-04-20T17:11:21.004 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.004 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:753fa799:::benchmark_data_smithi171_78918_object140972:head# 2024-04-20T17:11:21.005 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.005 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:753fbbb9:::benchmark_data_smithi171_78918_object63608:head# 2024-04-20T17:11:21.005 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.005 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75418e15:::benchmark_data_smithi171_78918_object145654:head# 2024-04-20T17:11:21.005 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.005 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7542167b:::benchmark_data_smithi171_78918_object108438:head# 2024-04-20T17:11:21.005 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.005 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7542608e:::benchmark_data_smithi171_78918_object151591:head# 2024-04-20T17:11:21.005 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.005 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7542cb29:::benchmark_data_smithi171_78918_object97200:head# 2024-04-20T17:11:21.005 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.006 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75477a23:::benchmark_data_smithi171_78918_object124882:head# 2024-04-20T17:11:21.006 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.006 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:754982ef:::benchmark_data_smithi171_78918_object78531:head# 2024-04-20T17:11:21.006 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.006 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7549b59a:::benchmark_data_smithi171_78918_object160741:head# 2024-04-20T17:11:21.006 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.006 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7549cf36:::benchmark_data_smithi171_78918_object120603:head# 2024-04-20T17:11:21.006 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.007 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:754af90e:::benchmark_data_smithi171_78918_object89054:head# 2024-04-20T17:11:21.007 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.007 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:754beb04:::benchmark_data_smithi171_78918_object121839:head# 2024-04-20T17:11:21.007 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.007 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:754c38d1:::benchmark_data_smithi171_78918_object57646:head# 2024-04-20T17:11:21.007 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.007 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:754c73f0:::benchmark_data_smithi171_78918_object155464:head# 2024-04-20T17:11:21.007 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.007 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:754ded62:::benchmark_data_smithi171_78918_object106338:head# 2024-04-20T17:11:21.007 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.008 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:754e2282:::benchmark_data_smithi171_78918_object81591:head# 2024-04-20T17:11:21.008 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.008 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:754ed2d3:::benchmark_data_smithi171_78918_object70197:head# 2024-04-20T17:11:21.008 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.008 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75509144:::benchmark_data_smithi171_78918_object91497:head# 2024-04-20T17:11:21.008 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.008 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7551037b:::benchmark_data_smithi171_78918_object131917:head# 2024-04-20T17:11:21.008 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.008 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75510836:::benchmark_data_smithi171_78918_object153671:head# 2024-04-20T17:11:21.008 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.008 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75546620:::benchmark_data_smithi171_78918_object118963:head# 2024-04-20T17:11:21.008 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.009 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7555230b:::benchmark_data_smithi171_78918_object110795:head# 2024-04-20T17:11:21.009 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.009 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75557aa9:::benchmark_data_smithi171_78918_object106789:head# 2024-04-20T17:11:21.009 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.009 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7555cccc:::benchmark_data_smithi171_78918_object123585:head# 2024-04-20T17:11:21.009 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.009 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7555d75a:::benchmark_data_smithi171_78918_object51866:head# 2024-04-20T17:11:21.009 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.009 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75560546:::benchmark_data_smithi171_78918_object151337:head# 2024-04-20T17:11:21.009 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.009 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75566d41:::benchmark_data_smithi171_78918_object119650:head# 2024-04-20T17:11:21.009 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.009 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:755673ba:::benchmark_data_smithi171_78918_object110569:head# 2024-04-20T17:11:21.010 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.010 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75586ba5:::benchmark_data_smithi171_78918_object141962:head# 2024-04-20T17:11:21.010 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.010 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75591249:::benchmark_data_smithi171_78918_object64514:head# 2024-04-20T17:11:21.010 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.010 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7559a55a:::benchmark_data_smithi171_78918_object97006:head# 2024-04-20T17:11:21.010 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.010 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7559cd0d:::benchmark_data_smithi171_78918_object141353:head# 2024-04-20T17:11:21.010 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.010 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:755a5dd0:::benchmark_data_smithi171_78918_object123725:head# 2024-04-20T17:11:21.010 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.010 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:755ba3eb:::benchmark_data_smithi171_78918_object158659:head# 2024-04-20T17:11:21.011 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.011 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:755bb667:::benchmark_data_smithi171_78918_object79420:head# 2024-04-20T17:11:21.011 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.011 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:755c7793:::benchmark_data_smithi171_78918_object107902:head# 2024-04-20T17:11:21.011 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.011 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:755d5a53:::benchmark_data_smithi171_78918_object68356:head# 2024-04-20T17:11:21.011 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.011 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:755d954c:::benchmark_data_smithi171_78918_object128436:head# 2024-04-20T17:11:21.011 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.011 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:755dc865:::benchmark_data_smithi171_78918_object156985:head# 2024-04-20T17:11:21.011 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.011 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:755e1aad:::benchmark_data_smithi171_78918_object87570:head# 2024-04-20T17:11:21.011 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.011 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:755f9ce5:::benchmark_data_smithi171_78918_object112225:head# 2024-04-20T17:11:21.011 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.012 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:755fe6b7:::benchmark_data_smithi171_78918_object159319:head# 2024-04-20T17:11:21.012 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.012 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7560d30f:::benchmark_data_smithi171_78918_object83337:head# 2024-04-20T17:11:21.012 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.012 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75633626:::benchmark_data_smithi171_78918_object115629:head# 2024-04-20T17:11:21.012 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.012 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:756445bc:::benchmark_data_smithi171_78918_object75961:head# 2024-04-20T17:11:21.012 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.012 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75644f78:::benchmark_data_smithi171_78918_object64630:head# 2024-04-20T17:11:21.012 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.012 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:756461fd:::benchmark_data_smithi171_78918_object92127:head# 2024-04-20T17:11:21.012 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.012 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7564d7ec:::benchmark_data_smithi171_78918_object159230:head# 2024-04-20T17:11:21.012 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.013 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:756503e9:::benchmark_data_smithi171_78918_object54062:head# 2024-04-20T17:11:21.013 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.013 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7566da37:::benchmark_data_smithi171_78918_object116271:head# 2024-04-20T17:11:21.013 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.013 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7567f892:::benchmark_data_smithi171_78918_object158616:head# 2024-04-20T17:11:21.013 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.013 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:11:21.113 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7567fd99:::benchmark_data_smithi171_78918_object79257:head# 2024-04-20T17:11:21.113 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.113 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75680cd8:::benchmark_data_smithi171_78918_object154184:head# 2024-04-20T17:11:21.113 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.113 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:756868e7:::benchmark_data_smithi171_78918_object75400:head# 2024-04-20T17:11:21.113 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.113 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7569a8c2:::benchmark_data_smithi171_78918_object131739:head# 2024-04-20T17:11:21.113 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.113 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:756a8387:::benchmark_data_smithi171_78918_object128839:head# 2024-04-20T17:11:21.113 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.114 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:756b2a4a:::benchmark_data_smithi171_78918_object110172:head# 2024-04-20T17:11:21.114 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.114 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:756bdb95:::benchmark_data_smithi171_78918_object139115:head# 2024-04-20T17:11:21.114 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.114 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:756c5bb6:::benchmark_data_smithi171_78918_object72634:head# 2024-04-20T17:11:21.114 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.114 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:756c94d4:::benchmark_data_smithi171_78918_object151900:head# 2024-04-20T17:11:21.114 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.114 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:756ca218:::benchmark_data_smithi171_78918_object60893:head# 2024-04-20T17:11:21.114 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.114 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:756d2ba7:::benchmark_data_smithi171_78918_object117182:head# 2024-04-20T17:11:21.114 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.114 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:756d5b6a:::benchmark_data_smithi171_78918_object62856:head# 2024-04-20T17:11:21.114 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.115 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:756e23d2:::benchmark_data_smithi171_78918_object94459:head# 2024-04-20T17:11:21.115 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.115 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:756e26b5:::benchmark_data_smithi171_78918_object119413:head# 2024-04-20T17:11:21.115 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.115 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:756e4b35:::benchmark_data_smithi171_78918_object98844:head# 2024-04-20T17:11:21.115 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.115 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:756e9d97:::benchmark_data_smithi171_78918_object115468:head# 2024-04-20T17:11:21.115 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.115 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:756eae9c:::benchmark_data_smithi171_78918_object111712:head# 2024-04-20T17:11:21.115 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.115 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:756ee5e1:::benchmark_data_smithi171_78918_object121213:head# 2024-04-20T17:11:21.115 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.115 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:756f859f:::benchmark_data_smithi171_78918_object96153:head# 2024-04-20T17:11:21.115 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.116 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:756fd72f:::benchmark_data_smithi171_78918_object136561:head# 2024-04-20T17:11:21.116 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.116 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:756ff8ca:::benchmark_data_smithi171_78918_object111158:head# 2024-04-20T17:11:21.116 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.116 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7573012f:::benchmark_data_smithi171_78918_object79192:head# 2024-04-20T17:11:21.116 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.116 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75742e8a:::benchmark_data_smithi171_78918_object65902:head# 2024-04-20T17:11:21.116 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.116 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75746714:::benchmark_data_smithi171_78918_object150809:head# 2024-04-20T17:11:21.116 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.116 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75749bd6:::benchmark_data_smithi171_78918_object69663:head# 2024-04-20T17:11:21.116 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.116 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7574d0a7:::benchmark_data_smithi171_78918_object78943:head# 2024-04-20T17:11:21.116 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.117 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75756c40:::benchmark_data_smithi171_78918_object58051:head# 2024-04-20T17:11:21.117 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.117 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75756d80:::benchmark_data_smithi171_78918_object143967:head# 2024-04-20T17:11:21.117 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.117 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7575a4e3:::benchmark_data_smithi171_78918_object90849:head# 2024-04-20T17:11:21.117 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.117 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7575ede8:::benchmark_data_smithi171_78918_object74940:head# 2024-04-20T17:11:21.117 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.117 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:757672b1:::benchmark_data_smithi171_78918_object149244:head# 2024-04-20T17:11:21.117 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.117 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75775fb1:::benchmark_data_smithi171_78918_object67843:head# 2024-04-20T17:11:21.117 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.117 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7578bae2:::benchmark_data_smithi171_78918_object138066:head# 2024-04-20T17:11:21.117 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.118 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:757995ca:::benchmark_data_smithi171_78918_object153612:head# 2024-04-20T17:11:21.118 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.118 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:757a55a2:::benchmark_data_smithi171_78918_object95578:head# 2024-04-20T17:11:21.118 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.118 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:757bc168:::benchmark_data_smithi171_78918_object155138:head# 2024-04-20T17:11:21.118 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.118 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:757c1b30:::benchmark_data_smithi171_78918_object57529:head# 2024-04-20T17:11:21.118 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.118 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:757ce38b:::benchmark_data_smithi171_78918_object148205:head# 2024-04-20T17:11:21.118 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.118 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:757d10f8:::benchmark_data_smithi171_78918_object128300:head# 2024-04-20T17:11:21.118 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.118 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:757d4e3c:::benchmark_data_smithi171_78918_object102716:head# 2024-04-20T17:11:21.119 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.119 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:757db0ab:::benchmark_data_smithi171_78918_object120804:head# 2024-04-20T17:11:21.119 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.119 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:757f1142:::benchmark_data_smithi171_78918_object123639:head# 2024-04-20T17:11:21.119 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.119 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:757fc3f4:::benchmark_data_smithi171_78918_object98903:head# 2024-04-20T17:11:21.119 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.119 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75809a50:::benchmark_data_smithi171_78918_object77396:head# 2024-04-20T17:11:21.119 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.119 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7580f431:::benchmark_data_smithi171_78918_object56435:head# 2024-04-20T17:11:21.119 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.119 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7581737b:::benchmark_data_smithi171_78918_object60845:head# 2024-04-20T17:11:21.120 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.120 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7582ce93:::benchmark_data_smithi171_78918_object82969:head# 2024-04-20T17:11:21.120 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.120 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7584d1e8:::benchmark_data_smithi171_78918_object149482:head# 2024-04-20T17:11:21.120 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.120 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:758583c8:::benchmark_data_smithi171_78918_object144801:head# 2024-04-20T17:11:21.120 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.120 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7585d243:::benchmark_data_smithi171_78918_object103402:head# 2024-04-20T17:11:21.120 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.121 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7585e1db:::benchmark_data_smithi171_78918_object154747:head# 2024-04-20T17:11:21.121 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.121 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:758731cb:::benchmark_data_smithi171_78918_object119461:head# 2024-04-20T17:11:21.121 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.121 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75875eb3:::benchmark_data_smithi171_78918_object139763:head# 2024-04-20T17:11:21.121 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.121 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75879604:::benchmark_data_smithi171_78918_object159777:head# 2024-04-20T17:11:21.121 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.121 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7587f8ad:::benchmark_data_smithi171_78918_object91973:head# 2024-04-20T17:11:21.122 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.122 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7588383c:::benchmark_data_smithi171_78918_object60974:head# 2024-04-20T17:11:21.122 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.122 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7588bb66:::benchmark_data_smithi171_78918_object160730:head# 2024-04-20T17:11:21.122 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.122 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7588f855:::benchmark_data_smithi171_78918_object59273:head# 2024-04-20T17:11:21.122 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.122 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75890ead:::benchmark_data_smithi171_78918_object102521:head# 2024-04-20T17:11:21.122 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.122 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7589acb8:::benchmark_data_smithi171_78918_object95675:head# 2024-04-20T17:11:21.122 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.123 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:758a8b19:::benchmark_data_smithi171_78918_object97037:head# 2024-04-20T17:11:21.123 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.123 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:758b263d:::benchmark_data_smithi171_78918_object94799:head# 2024-04-20T17:11:21.123 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.123 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:758b2718:::benchmark_data_smithi171_78918_object73596:head# 2024-04-20T17:11:21.123 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.123 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:758c1114:::benchmark_data_smithi171_78918_object156433:head# 2024-04-20T17:11:21.123 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.123 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:758c634f:::benchmark_data_smithi171_78918_object74331:head# 2024-04-20T17:11:21.123 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.123 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:758d03b4:::benchmark_data_smithi171_78918_object74795:head# 2024-04-20T17:11:21.123 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.123 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:758dc1a2:::benchmark_data_smithi171_78918_object100007:head# 2024-04-20T17:11:21.124 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.124 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:758ea6d0:::benchmark_data_smithi171_78918_object69069:head# 2024-04-20T17:11:21.124 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.124 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:758f744b:::benchmark_data_smithi171_78918_object104371:head# 2024-04-20T17:11:21.124 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.124 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:758f7e9a:::benchmark_data_smithi171_78918_object145666:head# 2024-04-20T17:11:21.124 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.124 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75908e62:::benchmark_data_smithi171_78918_object82285:head# 2024-04-20T17:11:21.124 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.124 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7590abda:::benchmark_data_smithi171_78918_object160583:head# 2024-04-20T17:11:21.124 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.124 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75912ead:::benchmark_data_smithi171_78918_object89120:head# 2024-04-20T17:11:21.124 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.124 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:759152fd:::benchmark_data_smithi171_78918_object141643:head# 2024-04-20T17:11:21.125 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.125 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75915aaa:::benchmark_data_smithi171_78918_object137173:head# 2024-04-20T17:11:21.125 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.125 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7593289a:::benchmark_data_smithi171_78918_object77357:head# 2024-04-20T17:11:21.125 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.125 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75933794:::benchmark_data_smithi171_78918_object147249:head# 2024-04-20T17:11:21.125 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.125 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75936f7d:::benchmark_data_smithi171_78918_object122360:head# 2024-04-20T17:11:21.125 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.125 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7594c3c0:::benchmark_data_smithi171_78918_object81770:head# 2024-04-20T17:11:21.125 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.125 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75951d24:::benchmark_data_smithi171_78918_object150213:head# 2024-04-20T17:11:21.125 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.125 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75955500:::benchmark_data_smithi171_78918_object58806:head# 2024-04-20T17:11:21.126 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.126 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75956b8a:::benchmark_data_smithi171_78918_object94933:head# 2024-04-20T17:11:21.126 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.126 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7595e535:::benchmark_data_smithi171_78918_object78269:head# 2024-04-20T17:11:21.126 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.126 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75963fde:::benchmark_data_smithi171_78918_object73559:head# 2024-04-20T17:11:21.126 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.126 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75969d05:::benchmark_data_smithi171_78918_object83131:head# 2024-04-20T17:11:21.126 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.126 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75977df2:::benchmark_data_smithi171_78918_object97048:head# 2024-04-20T17:11:21.126 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.126 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75993f16:::benchmark_data_smithi171_78918_object120859:head# 2024-04-20T17:11:21.126 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.126 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7599ae29:::benchmark_data_smithi171_78918_object137675:head# 2024-04-20T17:11:21.127 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.127 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:759a3bc8:::benchmark_data_smithi171_78918_object93851:head# 2024-04-20T17:11:21.127 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.127 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:759ab0b8:::benchmark_data_smithi171_78918_object142437:head# 2024-04-20T17:11:21.127 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.127 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:759af54c:::benchmark_data_smithi171_78918_object84592:head# 2024-04-20T17:11:21.127 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.127 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:759ba13a:::benchmark_data_smithi171_78918_object151210:head# 2024-04-20T17:11:21.127 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.127 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:759cb00c:::benchmark_data_smithi171_78918_object160877:head# 2024-04-20T17:11:21.127 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.127 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:759d1d63:::benchmark_data_smithi171_78918_object100525:head# 2024-04-20T17:11:21.127 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.127 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:759d8a76:::benchmark_data_smithi171_78918_object122096:head# 2024-04-20T17:11:21.127 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.128 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:759df595:::benchmark_data_smithi171_78918_object133806:head# 2024-04-20T17:11:21.128 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.128 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:759eb829:::benchmark_data_smithi171_78918_object121717:head# 2024-04-20T17:11:21.128 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.128 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:759ff877:::benchmark_data_smithi171_78918_object101505:head# 2024-04-20T17:11:21.128 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.128 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75a0a0c5:::benchmark_data_smithi171_78918_object94301:head# 2024-04-20T17:11:21.128 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.128 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75a0b2c0:::benchmark_data_smithi171_78918_object97958:head# 2024-04-20T17:11:21.128 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.128 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75a0bae7:::benchmark_data_smithi171_78918_object105456:head# 2024-04-20T17:11:21.129 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.129 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75a0d857:::benchmark_data_smithi171_78918_object68703:head# 2024-04-20T17:11:21.129 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.129 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75a0e405:::benchmark_data_smithi171_78918_object138174:head# 2024-04-20T17:11:21.129 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.129 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75a16be1:::benchmark_data_smithi171_78918_object108457:head# 2024-04-20T17:11:21.129 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.129 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75a222e8:::benchmark_data_smithi171_78918_object96255:head# 2024-04-20T17:11:21.129 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.129 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75a225c7:::benchmark_data_smithi171_78918_object114733:head# 2024-04-20T17:11:21.129 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.130 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75a2a617:::benchmark_data_smithi171_78918_object94666:head# 2024-04-20T17:11:21.130 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.130 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75a3c709:::benchmark_data_smithi171_78918_object158493:head# 2024-04-20T17:11:21.130 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.130 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75a41bc8:::benchmark_data_smithi171_78918_object142047:head# 2024-04-20T17:11:21.130 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.130 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75a47929:::benchmark_data_smithi171_78918_object120992:head# 2024-04-20T17:11:21.130 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.130 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75a52623:::benchmark_data_smithi171_78918_object60200:head# 2024-04-20T17:11:21.130 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.130 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75a5ed01:::benchmark_data_smithi171_78918_object56862:head# 2024-04-20T17:11:21.130 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.131 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75a6baa7:::benchmark_data_smithi171_78918_object155573:head# 2024-04-20T17:11:21.131 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.131 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75a6f648:::benchmark_data_smithi171_78918_object98375:head# 2024-04-20T17:11:21.131 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.131 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75a91baa:::benchmark_data_smithi171_78918_object114234:head# 2024-04-20T17:11:21.131 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.131 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75aa89d2:::benchmark_data_smithi171_78918_object104978:head# 2024-04-20T17:11:21.131 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.131 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75aadaeb:::benchmark_data_smithi171_78918_object60791:head# 2024-04-20T17:11:21.131 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.131 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75ab94f9:::benchmark_data_smithi171_78918_object62380:head# 2024-04-20T17:11:21.131 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.131 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75aba466:::benchmark_data_smithi171_78918_object132564:head# 2024-04-20T17:11:21.131 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.132 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75abd79d:::benchmark_data_smithi171_78918_object113476:head# 2024-04-20T17:11:21.132 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.132 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75ac16da:::benchmark_data_smithi171_78918_object80187:head# 2024-04-20T17:11:21.132 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.132 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75adb5be:::benchmark_data_smithi171_78918_object94492:head# 2024-04-20T17:11:21.132 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.132 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75adcefb:::benchmark_data_smithi171_78918_object129100:head# 2024-04-20T17:11:21.132 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.132 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75ae1460:::benchmark_data_smithi171_78918_object116459:head# 2024-04-20T17:11:21.132 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.132 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75ae5f67:::benchmark_data_smithi171_78918_object110480:head# 2024-04-20T17:11:21.132 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.132 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75ae7040:::benchmark_data_smithi171_78918_object93434:head# 2024-04-20T17:11:21.132 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.133 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75aef45d:::benchmark_data_smithi171_78918_object117826:head# 2024-04-20T17:11:21.133 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.133 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75af201d:::benchmark_data_smithi171_78918_object82461:head# 2024-04-20T17:11:21.133 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.133 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75af22e1:::benchmark_data_smithi171_78918_object140200:head# 2024-04-20T17:11:21.133 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.133 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75aff7d6:::benchmark_data_smithi171_78918_object104548:head# 2024-04-20T17:11:21.133 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.133 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75b1dea2:::benchmark_data_smithi171_78918_object115248:head# 2024-04-20T17:11:21.133 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.133 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75b2fb77:::benchmark_data_smithi171_78918_object136433:head# 2024-04-20T17:11:21.133 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.133 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75b39fc3:::benchmark_data_smithi171_78918_object58788:head# 2024-04-20T17:11:21.133 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.133 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75b3ab3d:::benchmark_data_smithi171_78918_object76119:head# 2024-04-20T17:11:21.134 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.134 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75b3e3d0:::benchmark_data_smithi171_78918_object77836:head# 2024-04-20T17:11:21.134 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.134 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75b4945e:::benchmark_data_smithi171_78918_object86202:head# 2024-04-20T17:11:21.134 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.134 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75b63a2e:::benchmark_data_smithi171_78918_object143734:head# 2024-04-20T17:11:21.134 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.134 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75b6fe5f:::benchmark_data_smithi171_78918_object111042:head# 2024-04-20T17:11:21.134 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.134 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75b95907:::benchmark_data_smithi171_78918_object66095:head# 2024-04-20T17:11:21.134 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.134 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75b9601f:::benchmark_data_smithi171_78918_object150117:head# 2024-04-20T17:11:21.134 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.135 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75baf317:::benchmark_data_smithi171_78918_object88942:head# 2024-04-20T17:11:21.135 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.135 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75bc3d97:::benchmark_data_smithi171_78918_object54204:head# 2024-04-20T17:11:21.135 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.135 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75bd3811:::benchmark_data_smithi171_78918_object144282:head# 2024-04-20T17:11:21.135 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.135 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75be0d0c:::benchmark_data_smithi171_78918_object101985:head# 2024-04-20T17:11:21.135 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.135 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75beaf89:::benchmark_data_smithi171_78918_object122831:head# 2024-04-20T17:11:21.135 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.135 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75bf67a7:::benchmark_data_smithi171_78918_object139806:head# 2024-04-20T17:11:21.135 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.135 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75bff26c:::benchmark_data_smithi171_78918_object70673:head# 2024-04-20T17:11:21.135 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.136 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75c0367e:::benchmark_data_smithi171_78918_object139663:head# 2024-04-20T17:11:21.136 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.136 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75c04bbf:::benchmark_data_smithi171_78918_object95331:head# 2024-04-20T17:11:21.136 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.136 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75c05fa9:::benchmark_data_smithi171_78918_object123240:head# 2024-04-20T17:11:21.136 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.136 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75c11545:::benchmark_data_smithi171_78918_object90949:head# 2024-04-20T17:11:21.136 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.136 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75c13108:::benchmark_data_smithi171_78918_object84863:head# 2024-04-20T17:11:21.136 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.136 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75c1445f:::benchmark_data_smithi171_78918_object153767:head# 2024-04-20T17:11:21.136 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.136 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75c1ba98:::benchmark_data_smithi171_78918_object69346:head# 2024-04-20T17:11:21.136 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.136 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75c1e43c:::benchmark_data_smithi171_78918_object115149:head# 2024-04-20T17:11:21.137 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.137 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75c31ad5:::benchmark_data_smithi171_78918_object147655:head# 2024-04-20T17:11:21.137 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.137 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75c53ff4:::benchmark_data_smithi171_78918_object87218:head# 2024-04-20T17:11:21.137 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.137 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75c606d3:::benchmark_data_smithi171_78918_object126424:head# 2024-04-20T17:11:21.137 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.137 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75c657aa:::benchmark_data_smithi171_78918_object143985:head# 2024-04-20T17:11:21.137 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.137 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75c85aa6:::benchmark_data_smithi171_78918_object81096:head# 2024-04-20T17:11:21.137 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.137 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75c8c6c8:::benchmark_data_smithi171_78918_object88537:head# 2024-04-20T17:11:21.137 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.137 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75c8f223:::benchmark_data_smithi171_78918_object67543:head# 2024-04-20T17:11:21.138 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.138 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75c90f92:::benchmark_data_smithi171_78918_object113290:head# 2024-04-20T17:11:21.138 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.138 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75c972be:::benchmark_data_smithi171_78918_object89190:head# 2024-04-20T17:11:21.138 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.138 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75caaf15:::benchmark_data_smithi171_78918_object111199:head# 2024-04-20T17:11:21.138 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.138 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75cb378d:::benchmark_data_smithi171_78918_object121566:head# 2024-04-20T17:11:21.138 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.138 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75cbfb44:::benchmark_data_smithi171_78918_object82726:head# 2024-04-20T17:11:21.138 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.138 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75cd1a12:::benchmark_data_smithi171_78918_object93081:head# 2024-04-20T17:11:21.138 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.138 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75cd6999:::benchmark_data_smithi171_78918_object63823:head# 2024-04-20T17:11:21.139 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.139 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75cd9fbf:::benchmark_data_smithi171_78918_object82773:head# 2024-04-20T17:11:21.139 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.139 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75ce78e6:::benchmark_data_smithi171_78918_object117832:head# 2024-04-20T17:11:21.139 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.139 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75ce7f27:::benchmark_data_smithi171_78918_object132170:head# 2024-04-20T17:11:21.139 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.139 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75d29691:::benchmark_data_smithi171_78918_object153788:head# 2024-04-20T17:11:21.139 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.139 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75d2bf05:::benchmark_data_smithi171_78918_object101362:head# 2024-04-20T17:11:21.139 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.139 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75d414d4:::benchmark_data_smithi171_78918_object138971:head# 2024-04-20T17:11:21.139 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.139 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75d43ee6:::benchmark_data_smithi171_78918_object149366:head# 2024-04-20T17:11:21.139 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.140 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75d47d13:::benchmark_data_smithi171_78918_object153935:head# 2024-04-20T17:11:21.140 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.140 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75d48ec8:::benchmark_data_smithi171_78918_object126912:head# 2024-04-20T17:11:21.140 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.140 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75d5cd12:::benchmark_data_smithi171_78918_object115186:head# 2024-04-20T17:11:21.140 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.140 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75d69685:::benchmark_data_smithi171_78918_object153707:head# 2024-04-20T17:11:21.140 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.140 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75d89244:::benchmark_data_smithi171_78918_object114904:head# 2024-04-20T17:11:21.140 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.140 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75d98542:::benchmark_data_smithi171_78918_object89027:head# 2024-04-20T17:11:21.140 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.140 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75d9876a:::benchmark_data_smithi171_78918_object156230:head# 2024-04-20T17:11:21.140 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.140 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75dbd144:::benchmark_data_smithi171_78918_object120707:head# 2024-04-20T17:11:21.141 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.141 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75dbfb4b:::benchmark_data_smithi171_78918_object135537:head# 2024-04-20T17:11:21.141 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.141 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75dc0aaa:::benchmark_data_smithi171_78918_object103339:head# 2024-04-20T17:11:21.141 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.141 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75dd54b4:::benchmark_data_smithi171_78918_object131228:head# 2024-04-20T17:11:21.141 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.141 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75ded61e:::benchmark_data_smithi171_78918_object132057:head# 2024-04-20T17:11:21.141 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.141 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75df12aa:::benchmark_data_smithi171_78918_object150511:head# 2024-04-20T17:11:21.141 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.141 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75e02af4:::benchmark_data_smithi171_78918_object57715:head# 2024-04-20T17:11:21.141 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.141 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75e23fee:::benchmark_data_smithi171_78918_object108233:head# 2024-04-20T17:11:21.142 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.142 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75e24459:::benchmark_data_smithi171_78918_object149700:head# 2024-04-20T17:11:21.142 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.142 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75e24d35:::benchmark_data_smithi171_78918_object56417:head# 2024-04-20T17:11:21.142 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.142 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75e385fc:::benchmark_data_smithi171_78918_object81676:head# 2024-04-20T17:11:21.142 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.142 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75e40439:::benchmark_data_smithi171_78918_object79092:head# 2024-04-20T17:11:21.142 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.142 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75e5c16c:::benchmark_data_smithi171_78918_object90889:head# 2024-04-20T17:11:21.142 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.142 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75e65f1a:::benchmark_data_smithi171_78918_object157001:head# 2024-04-20T17:11:21.143 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.143 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75eab0f4:::benchmark_data_smithi171_78918_object104748:head# 2024-04-20T17:11:21.143 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.143 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75ec7d8a:::benchmark_data_smithi171_78918_object71261:head# 2024-04-20T17:11:21.143 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.143 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75eca705:::benchmark_data_smithi171_78918_object121440:head# 2024-04-20T17:11:21.143 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.143 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75ecf51d:::benchmark_data_smithi171_78918_object66968:head# 2024-04-20T17:11:21.144 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.144 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75edbe51:::benchmark_data_smithi171_78918_object64875:head# 2024-04-20T17:11:21.144 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.144 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75edf3b0:::benchmark_data_smithi171_78918_object108078:head# 2024-04-20T17:11:21.144 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.144 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75ee591d:::benchmark_data_smithi171_78918_object97040:head# 2024-04-20T17:11:21.144 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.144 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75ee94e5:::benchmark_data_smithi171_78918_object66893:head# 2024-04-20T17:11:21.144 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.144 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75f0285b:::benchmark_data_smithi171_78918_object85989:head# 2024-04-20T17:11:21.144 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.144 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75f1f1c9:::benchmark_data_smithi171_78918_object112293:head# 2024-04-20T17:11:21.144 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.145 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75f1fdd7:::benchmark_data_smithi171_78918_object79366:head# 2024-04-20T17:11:21.145 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.145 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75f3791b:::benchmark_data_smithi171_78918_object137743:head# 2024-04-20T17:11:21.145 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.145 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75f3daea:::benchmark_data_smithi171_78918_object72891:head# 2024-04-20T17:11:21.145 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.145 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75f4b161:::benchmark_data_smithi171_78918_object82371:head# 2024-04-20T17:11:21.145 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.145 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75f4eb77:::benchmark_data_smithi171_78918_object77910:head# 2024-04-20T17:11:21.145 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.145 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75f4eefb:::benchmark_data_smithi171_78918_object150466:head# 2024-04-20T17:11:21.145 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.145 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75f55905:::benchmark_data_smithi171_78918_object106909:head# 2024-04-20T17:11:21.146 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.146 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75f6ad8b:::benchmark_data_smithi171_78918_object67423:head# 2024-04-20T17:11:21.146 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.146 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75f797fa:::benchmark_data_smithi171_78918_object96332:head# 2024-04-20T17:11:21.146 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.146 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75f84e2a:::benchmark_data_smithi171_78918_object92607:head# 2024-04-20T17:11:21.146 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.146 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75f8f92c:::benchmark_data_smithi171_78918_object51932:head# 2024-04-20T17:11:21.146 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.146 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75f9a035:::benchmark_data_smithi171_78918_object52832:head# 2024-04-20T17:11:21.146 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.146 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75f9bc7a:::benchmark_data_smithi171_78918_object91576:head# 2024-04-20T17:11:21.146 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.146 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75f9fcac:::benchmark_data_smithi171_78918_object143187:head# 2024-04-20T17:11:21.147 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.147 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75fa0ea3:::benchmark_data_smithi171_78918_object80398:head# 2024-04-20T17:11:21.147 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.147 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75fb9709:::benchmark_data_smithi171_78918_object107137:head# 2024-04-20T17:11:21.147 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.147 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75fc16e1:::benchmark_data_smithi171_78918_object125930:head# 2024-04-20T17:11:21.147 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.147 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75fd8d35:::benchmark_data_smithi171_78918_object133454:head# 2024-04-20T17:11:21.147 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.147 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75fe19db:::benchmark_data_smithi171_78918_object107925:head# 2024-04-20T17:11:21.147 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.147 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75fe4e77:::benchmark_data_smithi171_78918_object68972:head# 2024-04-20T17:11:21.148 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.148 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:75fedfcc:::benchmark_data_smithi171_78918_object70529:head# 2024-04-20T17:11:21.148 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.148 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76001691:::benchmark_data_smithi171_78918_object58179:head# 2024-04-20T17:11:21.148 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.148 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7600fdb7:::benchmark_data_smithi171_78918_object157566:head# 2024-04-20T17:11:21.148 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.148 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76014c28:::benchmark_data_smithi171_78918_object154245:head# 2024-04-20T17:11:21.148 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.148 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7601ec69:::benchmark_data_smithi171_78918_object86104:head# 2024-04-20T17:11:21.148 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.149 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76033447:::benchmark_data_smithi171_78918_object59212:head# 2024-04-20T17:11:21.149 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.149 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76035d91:::benchmark_data_smithi171_78918_object137372:head# 2024-04-20T17:11:21.149 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.149 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76037e0c:::benchmark_data_smithi171_78918_object63641:head# 2024-04-20T17:11:21.149 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.149 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7603a42b:::benchmark_data_smithi171_78918_object51460:head# 2024-04-20T17:11:21.149 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.149 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7604ef3b:::benchmark_data_smithi171_78918_object145622:head# 2024-04-20T17:11:21.149 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.150 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7605a613:::benchmark_data_smithi171_78918_object149438:head# 2024-04-20T17:11:21.150 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.150 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7606a356:::benchmark_data_smithi171_78918_object153771:head# 2024-04-20T17:11:21.150 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.150 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7606bb9c:::benchmark_data_smithi171_78918_object84206:head# 2024-04-20T17:11:21.150 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.150 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7606e7a9:::benchmark_data_smithi171_78918_object54841:head# 2024-04-20T17:11:21.150 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.150 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76073bce:::benchmark_data_smithi171_78918_object159272:head# 2024-04-20T17:11:21.150 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.150 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76085828:::benchmark_data_smithi171_78918_object79570:head# 2024-04-20T17:11:21.150 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.150 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:760b2128:::benchmark_data_smithi171_78918_object64346:head# 2024-04-20T17:11:21.150 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.151 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:760b8f6c:::benchmark_data_smithi171_78918_object86379:head# 2024-04-20T17:11:21.151 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.151 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:760bac7d:::benchmark_data_smithi171_78918_object99882:head# 2024-04-20T17:11:21.151 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.151 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:760c22fa:::benchmark_data_smithi171_78918_object143970:head# 2024-04-20T17:11:21.151 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.151 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:760cc699:::benchmark_data_smithi171_78918_object159495:head# 2024-04-20T17:11:21.151 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.151 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:760ce839:::benchmark_data_smithi171_78918_object57288:head# 2024-04-20T17:11:21.151 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.151 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:760d10cf:::benchmark_data_smithi171_78918_object92464:head# 2024-04-20T17:11:21.151 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.151 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:760d6487:::benchmark_data_smithi171_78918_object72164:head# 2024-04-20T17:11:21.151 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.152 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:760d79ba:::benchmark_data_smithi171_78918_object110132:head# 2024-04-20T17:11:21.152 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.152 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:760d7dae:::benchmark_data_smithi171_78918_object141185:head# 2024-04-20T17:11:21.152 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.152 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:760e6dbf:::benchmark_data_smithi171_78918_object117392:head# 2024-04-20T17:11:21.152 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.152 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:760e7f45:::benchmark_data_smithi171_78918_object72537:head# 2024-04-20T17:11:21.152 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.152 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:760f02b5:::benchmark_data_smithi171_78918_object81631:head# 2024-04-20T17:11:21.152 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.152 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:760f2db9:::benchmark_data_smithi171_78918_object53718:head# 2024-04-20T17:11:21.152 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.152 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:760fa722:::benchmark_data_smithi171_78918_object69203:head# 2024-04-20T17:11:21.153 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.153 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:761035e7:::benchmark_data_smithi171_78918_object130271:head# 2024-04-20T17:11:21.153 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.153 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76103c18:::benchmark_data_smithi171_78918_object81509:head# 2024-04-20T17:11:21.153 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.153 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7610c5d1:::benchmark_data_smithi171_78918_object98474:head# 2024-04-20T17:11:21.153 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.153 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7612cb9f:::benchmark_data_smithi171_78918_object149938:head# 2024-04-20T17:11:21.153 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.153 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:761397d3:::benchmark_data_smithi171_78918_object63312:head# 2024-04-20T17:11:21.153 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.153 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7613e345:::benchmark_data_smithi171_78918_object141138:head# 2024-04-20T17:11:21.153 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.154 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76141960:::benchmark_data_smithi171_78918_object52675:head# 2024-04-20T17:11:21.154 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.154 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7614b08c:::benchmark_data_smithi171_78918_object80924:head# 2024-04-20T17:11:21.154 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.154 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7615e823:::benchmark_data_smithi171_78918_object136970:head# 2024-04-20T17:11:21.154 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.154 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76164391:::benchmark_data_smithi171_78918_object70941:head# 2024-04-20T17:11:21.154 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.154 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7616761a:::benchmark_data_smithi171_78918_object153925:head# 2024-04-20T17:11:21.154 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.154 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7616e1c0:::benchmark_data_smithi171_78918_object96274:head# 2024-04-20T17:11:21.155 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.155 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7616f480:::benchmark_data_smithi171_78918_object122528:head# 2024-04-20T17:11:21.155 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.155 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7617d112:::benchmark_data_smithi171_78918_object130001:head# 2024-04-20T17:11:21.155 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.155 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76186003:::benchmark_data_smithi171_78918_object61084:head# 2024-04-20T17:11:21.155 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.155 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:761936bb:::benchmark_data_smithi171_78918_object153394:head# 2024-04-20T17:11:21.155 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.155 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:761963ea:::benchmark_data_smithi171_78918_object96431:head# 2024-04-20T17:11:21.155 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.155 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76199aa9:::benchmark_data_smithi171_78918_object93163:head# 2024-04-20T17:11:21.155 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.155 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:761a21c8:::benchmark_data_smithi171_78918_object101069:head# 2024-04-20T17:11:21.156 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.156 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:761af3cb:::benchmark_data_smithi171_78918_object54210:head# 2024-04-20T17:11:21.156 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.156 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:761c2d8f:::benchmark_data_smithi171_78918_object57508:head# 2024-04-20T17:11:21.156 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.156 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:761d0857:::benchmark_data_smithi171_78918_object118251:head# 2024-04-20T17:11:21.156 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.156 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:761d09d2:::benchmark_data_smithi171_78918_object133830:head# 2024-04-20T17:11:21.156 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.156 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:761d13d0:::benchmark_data_smithi171_78918_object93271:head# 2024-04-20T17:11:21.156 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.156 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:761d4044:::benchmark_data_smithi171_78918_object63420:head# 2024-04-20T17:11:21.156 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.156 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:761dfd09:::benchmark_data_smithi171_78918_object104596:head# 2024-04-20T17:11:21.156 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.157 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:761fbdc8:::benchmark_data_smithi171_78918_object132426:head# 2024-04-20T17:11:21.157 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.157 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:762017d5:::benchmark_data_smithi171_78918_object55820:head# 2024-04-20T17:11:21.157 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.157 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7620600e:::benchmark_data_smithi171_78918_object89816:head# 2024-04-20T17:11:21.157 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.157 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76207534:::benchmark_data_smithi171_78918_object115825:head# 2024-04-20T17:11:21.157 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.157 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76210853:::benchmark_data_smithi171_78918_object70270:head# 2024-04-20T17:11:21.157 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.157 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:762191ae:::benchmark_data_smithi171_78918_object114299:head# 2024-04-20T17:11:21.157 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.157 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7624730c:::benchmark_data_smithi171_78918_object86919:head# 2024-04-20T17:11:21.158 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.158 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:762536b2:::benchmark_data_smithi171_78918_object111588:head# 2024-04-20T17:11:21.158 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.158 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76265edb:::benchmark_data_smithi171_78918_object154835:head# 2024-04-20T17:11:21.158 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.158 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7626aa3f:::benchmark_data_smithi171_78918_object68080:head# 2024-04-20T17:11:21.158 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.158 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76273193:::benchmark_data_smithi171_78918_object149733:head# 2024-04-20T17:11:21.158 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.158 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76293d1d:::benchmark_data_smithi171_78918_object76030:head# 2024-04-20T17:11:21.158 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.158 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:762b696d:::benchmark_data_smithi171_78918_object91773:head# 2024-04-20T17:11:21.158 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.158 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:762bbc94:::benchmark_data_smithi171_78918_object92876:head# 2024-04-20T17:11:21.159 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.159 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:762c7d09:::benchmark_data_smithi171_78918_object125041:head# 2024-04-20T17:11:21.159 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.159 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:762d479f:::benchmark_data_smithi171_78918_object66900:head# 2024-04-20T17:11:21.159 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.159 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:762ddac7:::benchmark_data_smithi171_78918_object145691:head# 2024-04-20T17:11:21.159 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.159 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:762e0d76:::benchmark_data_smithi171_78918_object99287:head# 2024-04-20T17:11:21.159 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.159 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:762e2bca:::benchmark_data_smithi171_78918_object71538:head# 2024-04-20T17:11:21.159 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.159 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:762e86a8:::benchmark_data_smithi171_78918_object58128:head# 2024-04-20T17:11:21.159 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.160 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:762f124a:::benchmark_data_smithi171_78918_object74854:head# 2024-04-20T17:11:21.160 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.160 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:762fe4de:::benchmark_data_smithi171_78918_object152065:head# 2024-04-20T17:11:21.160 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.160 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76304c0f:::benchmark_data_smithi171_78918_object67796:head# 2024-04-20T17:11:21.160 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.160 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7630f96b:::benchmark_data_smithi171_78918_object87109:head# 2024-04-20T17:11:21.160 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.160 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76319501:::benchmark_data_smithi171_78918_object144455:head# 2024-04-20T17:11:21.160 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.160 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7631c1aa:::benchmark_data_smithi171_78918_object60364:head# 2024-04-20T17:11:21.160 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.160 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76337fc9:::benchmark_data_smithi171_78918_object59528:head# 2024-04-20T17:11:21.160 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.160 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7633c400:::benchmark_data_smithi171_78918_object73441:head# 2024-04-20T17:11:21.161 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.161 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:763480d6:::benchmark_data_smithi171_78918_object64071:head# 2024-04-20T17:11:21.161 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.161 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76353579:::benchmark_data_smithi171_78918_object111917:head# 2024-04-20T17:11:21.161 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.161 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76377ec2:::benchmark_data_smithi171_78918_object112295:head# 2024-04-20T17:11:21.161 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.161 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7638113d:::benchmark_data_smithi171_78918_object120661:head# 2024-04-20T17:11:21.161 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.161 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7638c6ce:::benchmark_data_smithi171_78918_object155150:head# 2024-04-20T17:11:21.161 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.161 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7638d22a:::benchmark_data_smithi171_78918_object65934:head# 2024-04-20T17:11:21.161 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.161 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:763aae91:::benchmark_data_smithi171_78918_object54388:head# 2024-04-20T17:11:21.162 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.162 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:763b0c69:::benchmark_data_smithi171_78918_object92069:head# 2024-04-20T17:11:21.162 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.162 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:763b4733:::benchmark_data_smithi171_78918_object62358:head# 2024-04-20T17:11:21.162 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.162 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:763b7cbc:::benchmark_data_smithi171_78918_object122268:head# 2024-04-20T17:11:21.162 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.162 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:763beaa5:::benchmark_data_smithi171_78918_object156540:head# 2024-04-20T17:11:21.162 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.162 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:763c2a0f:::benchmark_data_smithi171_78918_object57899:head# 2024-04-20T17:11:21.162 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.162 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:763d648b:::benchmark_data_smithi171_78918_object62943:head# 2024-04-20T17:11:21.162 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.162 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76408d44:::benchmark_data_smithi171_78918_object96631:head# 2024-04-20T17:11:21.163 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.163 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76413fc9:::benchmark_data_smithi171_78918_object84074:head# 2024-04-20T17:11:21.163 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.163 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76431abd:::benchmark_data_smithi171_78918_object156774:head# 2024-04-20T17:11:21.163 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.163 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76437c45:::benchmark_data_smithi171_78918_object106787:head# 2024-04-20T17:11:21.163 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.163 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:764394b9:::benchmark_data_smithi171_78918_object71811:head# 2024-04-20T17:11:21.163 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.163 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7643e05e:::benchmark_data_smithi171_78918_object132235:head# 2024-04-20T17:11:21.163 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.163 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:764692c0:::benchmark_data_smithi171_78918_object114632:head# 2024-04-20T17:11:21.163 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.164 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76478b39:::benchmark_data_smithi171_78918_object101151:head# 2024-04-20T17:11:21.164 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.164 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76495760:::benchmark_data_smithi171_78918_object57521:head# 2024-04-20T17:11:21.164 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.164 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7649eea4:::benchmark_data_smithi171_78918_object143634:head# 2024-04-20T17:11:21.164 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.164 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:764b10ce:::benchmark_data_smithi171_78918_object68520:head# 2024-04-20T17:11:21.164 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.164 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:764bc5d0:::benchmark_data_smithi171_78918_object83889:head# 2024-04-20T17:11:21.164 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.164 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:764bf601:::benchmark_data_smithi171_78918_object75754:head# 2024-04-20T17:11:21.164 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.164 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:764c9356:::benchmark_data_smithi171_78918_object65458:head# 2024-04-20T17:11:21.165 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.165 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:764ce36e:::benchmark_data_smithi171_78918_object72184:head# 2024-04-20T17:11:21.165 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.165 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:764d9bed:::benchmark_data_smithi171_78918_object126364:head# 2024-04-20T17:11:21.165 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.165 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:764db2de:::benchmark_data_smithi171_78918_object71046:head# 2024-04-20T17:11:21.165 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.165 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:764dc0ba:::benchmark_data_smithi171_78918_object96494:head# 2024-04-20T17:11:21.165 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.165 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:764e5a89:::benchmark_data_smithi171_78918_object138265:head# 2024-04-20T17:11:21.165 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.165 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:764f37db:::benchmark_data_smithi171_78918_object130978:head# 2024-04-20T17:11:21.165 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.165 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:764f5d79:::benchmark_data_smithi171_78918_object61645:head# 2024-04-20T17:11:21.165 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.166 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:764f636f:::benchmark_data_smithi171_78918_object141317:head# 2024-04-20T17:11:21.166 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.166 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:765006a9:::benchmark_data_smithi171_78918_object67662:head# 2024-04-20T17:11:21.166 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.166 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:765012c2:::benchmark_data_smithi171_78918_object128358:head# 2024-04-20T17:11:21.166 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.166 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7650ab22:::benchmark_data_smithi171_78918_object119470:head# 2024-04-20T17:11:21.166 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.166 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76515165:::benchmark_data_smithi171_78918_object73603:head# 2024-04-20T17:11:21.166 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.166 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7651890b:::benchmark_data_smithi171_78918_object60113:head# 2024-04-20T17:11:21.166 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.166 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7652069c:::benchmark_data_smithi171_78918_object136336:head# 2024-04-20T17:11:21.166 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.167 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76520bfd:::benchmark_data_smithi171_78918_object57245:head# 2024-04-20T17:11:21.167 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.167 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:765282f6:::benchmark_data_smithi171_78918_object126588:head# 2024-04-20T17:11:21.167 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.167 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76534489:::benchmark_data_smithi171_78918_object105779:head# 2024-04-20T17:11:21.167 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.167 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7654aac9:::benchmark_data_smithi171_78918_object116590:head# 2024-04-20T17:11:21.167 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.167 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7654bb82:::benchmark_data_smithi171_78918_object69818:head# 2024-04-20T17:11:21.167 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.167 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:765601b4:::benchmark_data_smithi171_78918_object149853:head# 2024-04-20T17:11:21.167 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.167 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76569381:::benchmark_data_smithi171_78918_object96013:head# 2024-04-20T17:11:21.167 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.167 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7656ce28:::benchmark_data_smithi171_78918_object94100:head# 2024-04-20T17:11:21.168 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.168 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7657f2cd:::benchmark_data_smithi171_78918_object154778:head# 2024-04-20T17:11:21.168 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.168 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76581cc2:::benchmark_data_smithi171_78918_object65756:head# 2024-04-20T17:11:21.168 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.168 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7658b391:::benchmark_data_smithi171_78918_object58699:head# 2024-04-20T17:11:21.168 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.168 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7658bca9:::benchmark_data_smithi171_78918_object62353:head# 2024-04-20T17:11:21.168 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.168 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76591710:::benchmark_data_smithi171_78918_object118165:head# 2024-04-20T17:11:21.168 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.168 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7659e390:::benchmark_data_smithi171_78918_object152738:head# 2024-04-20T17:11:21.168 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.168 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:765a0794:::benchmark_data_smithi171_78918_object106163:head# 2024-04-20T17:11:21.168 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.169 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:765a6c63:::benchmark_data_smithi171_78918_object112118:head# 2024-04-20T17:11:21.169 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.169 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:765ac0d0:::benchmark_data_smithi171_78918_object148077:head# 2024-04-20T17:11:21.169 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.169 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:765b65cd:::benchmark_data_smithi171_78918_object89219:head# 2024-04-20T17:11:21.169 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.169 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:765b7b37:::benchmark_data_smithi171_78918_object102634:head# 2024-04-20T17:11:21.169 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.169 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:765bb833:::benchmark_data_smithi171_78918_object151490:head# 2024-04-20T17:11:21.169 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.169 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:765d3ba4:::benchmark_data_smithi171_78918_object56777:head# 2024-04-20T17:11:21.169 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.169 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:765de407:::benchmark_data_smithi171_78918_object104819:head# 2024-04-20T17:11:21.169 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.169 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:765e0fcd:::benchmark_data_smithi171_78918_object150636:head# 2024-04-20T17:11:21.170 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.170 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:765e3103:::benchmark_data_smithi171_78918_object147990:head# 2024-04-20T17:11:21.170 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.170 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:765f8119:::benchmark_data_smithi171_78918_object97618:head# 2024-04-20T17:11:21.170 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.170 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:765fa3fe:::benchmark_data_smithi171_78918_object151914:head# 2024-04-20T17:11:21.170 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.170 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:766125d1:::benchmark_data_smithi171_78918_object154124:head# 2024-04-20T17:11:21.170 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.170 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7661597f:::benchmark_data_smithi171_78918_object59464:head# 2024-04-20T17:11:21.170 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.170 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7661d85a:::benchmark_data_smithi171_78918_object110085:head# 2024-04-20T17:11:21.170 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.170 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7662c415:::benchmark_data_smithi171_78918_object96165:head# 2024-04-20T17:11:21.171 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.171 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76634707:::benchmark_data_smithi171_78918_object92162:head# 2024-04-20T17:11:21.171 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.171 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76641312:::benchmark_data_smithi171_78918_object129938:head# 2024-04-20T17:11:21.171 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.171 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:766512e6:::benchmark_data_smithi171_78918_object84495:head# 2024-04-20T17:11:21.171 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.171 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:766515f9:::benchmark_data_smithi171_78918_object72009:head# 2024-04-20T17:11:21.171 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.171 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7665fbf6:::benchmark_data_smithi171_78918_object123666:head# 2024-04-20T17:11:21.171 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.171 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:766691a5:::benchmark_data_smithi171_78918_object98454:head# 2024-04-20T17:11:21.171 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.171 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7666a161:::benchmark_data_smithi171_78918_object87029:head# 2024-04-20T17:11:21.172 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.172 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76674731:::benchmark_data_smithi171_78918_object90956:head# 2024-04-20T17:11:21.172 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.172 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76675974:::benchmark_data_smithi171_78918_object117619:head# 2024-04-20T17:11:21.172 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.172 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7667870e:::benchmark_data_smithi171_78918_object117211:head# 2024-04-20T17:11:21.172 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.172 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:766799d6:::benchmark_data_smithi171_78918_object52161:head# 2024-04-20T17:11:21.172 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.172 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7668f102:::benchmark_data_smithi171_78918_object151162:head# 2024-04-20T17:11:21.172 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.172 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:766a9c78:::benchmark_data_smithi171_78918_object109477:head# 2024-04-20T17:11:21.172 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.172 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:766af39d:::benchmark_data_smithi171_78918_object145667:head# 2024-04-20T17:11:21.173 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.173 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:766c82cb:::benchmark_data_smithi171_78918_object78681:head# 2024-04-20T17:11:21.173 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.173 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:766c9d47:::benchmark_data_smithi171_78918_object81660:head# 2024-04-20T17:11:21.173 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.173 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:766ce5e7:::benchmark_data_smithi171_78918_object96808:head# 2024-04-20T17:11:21.173 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.173 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:766d3b7d:::benchmark_data_smithi171_78918_object147874:head# 2024-04-20T17:11:21.173 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.173 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:766d7e39:::benchmark_data_smithi171_78918_object137167:head# 2024-04-20T17:11:21.173 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.173 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:766d849a:::benchmark_data_smithi171_78918_object83976:head# 2024-04-20T17:11:21.173 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.173 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:766dfe64:::benchmark_data_smithi171_78918_object84576:head# 2024-04-20T17:11:21.173 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.174 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:766e3b95:::benchmark_data_smithi171_78918_object120715:head# 2024-04-20T17:11:21.174 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.174 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:766e6aed:::benchmark_data_smithi171_78918_object108767:head# 2024-04-20T17:11:21.174 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.174 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:766eb628:::benchmark_data_smithi171_78918_object134289:head# 2024-04-20T17:11:21.174 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.174 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:766f577d:::benchmark_data_smithi171_78918_object151393:head# 2024-04-20T17:11:21.174 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.174 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:766f8e06:::benchmark_data_smithi171_78918_object52886:head# 2024-04-20T17:11:21.174 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.174 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7670dc2d:::benchmark_data_smithi171_78918_object123500:head# 2024-04-20T17:11:21.174 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.174 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7670f95c:::benchmark_data_smithi171_78918_object142735:head# 2024-04-20T17:11:21.174 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.175 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7672948d:::benchmark_data_smithi171_78918_object67790:head# 2024-04-20T17:11:21.175 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.175 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7672abc4:::benchmark_data_smithi171_78918_object114899:head# 2024-04-20T17:11:21.175 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.175 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7672b154:::benchmark_data_smithi171_78918_object51464:head# 2024-04-20T17:11:21.175 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.175 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7672d815:::benchmark_data_smithi171_78918_object121251:head# 2024-04-20T17:11:21.175 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.175 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7673c08b:::benchmark_data_smithi171_78918_object147955:head# 2024-04-20T17:11:21.175 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.175 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7674ea5a:::benchmark_data_smithi171_78918_object68233:head# 2024-04-20T17:11:21.175 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.175 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76761b69:::benchmark_data_smithi171_78918_object79326:head# 2024-04-20T17:11:21.175 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.176 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7676755b:::benchmark_data_smithi171_78918_object53991:head# 2024-04-20T17:11:21.176 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.176 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76768391:::benchmark_data_smithi171_78918_object145722:head# 2024-04-20T17:11:21.176 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.176 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7676cc3e:::benchmark_data_smithi171_78918_object59372:head# 2024-04-20T17:11:21.176 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.176 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7677d4b4:::benchmark_data_smithi171_78918_object52803:head# 2024-04-20T17:11:21.176 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.176 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76780b34:::benchmark_data_smithi171_78918_object85673:head# 2024-04-20T17:11:21.176 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.176 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76784a02:::benchmark_data_smithi171_78918_object70146:head# 2024-04-20T17:11:21.176 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.176 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7679e4c1:::benchmark_data_smithi171_78918_object124542:head# 2024-04-20T17:11:21.176 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.176 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:767b09fa:::benchmark_data_smithi171_78918_object153365:head# 2024-04-20T17:11:21.177 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.177 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:767b4f19:::benchmark_data_smithi171_78918_object127486:head# 2024-04-20T17:11:21.177 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.177 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:767bfde7:::benchmark_data_smithi171_78918_object147234:head# 2024-04-20T17:11:21.177 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.177 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:767ceb2e:::benchmark_data_smithi171_78918_object118656:head# 2024-04-20T17:11:21.177 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.177 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:767d5763:::benchmark_data_smithi171_78918_object78721:head# 2024-04-20T17:11:21.177 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.177 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:767d84a6:::benchmark_data_smithi171_78918_object131434:head# 2024-04-20T17:11:21.177 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.177 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:767d853b:::benchmark_data_smithi171_78918_object56636:head# 2024-04-20T17:11:21.177 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.177 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:767df3b7:::benchmark_data_smithi171_78918_object141931:head# 2024-04-20T17:11:21.178 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.178 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:767df4d2:::benchmark_data_smithi171_78918_object141523:head# 2024-04-20T17:11:21.178 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.178 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:767dfa51:::benchmark_data_smithi171_78918_object57285:head# 2024-04-20T17:11:21.178 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.178 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:767e0e7c:::benchmark_data_smithi171_78918_object108126:head# 2024-04-20T17:11:21.178 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.178 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:767f3f1e:::benchmark_data_smithi171_78918_object115983:head# 2024-04-20T17:11:21.178 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.178 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:768090fa:::benchmark_data_smithi171_78918_object75397:head# 2024-04-20T17:11:21.178 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.178 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:768308d4:::benchmark_data_smithi171_78918_object93823:head# 2024-04-20T17:11:21.178 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.178 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76830c7b:::benchmark_data_smithi171_78918_object158214:head# 2024-04-20T17:11:21.178 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.179 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76833c7b:::benchmark_data_smithi171_78918_object152929:head# 2024-04-20T17:11:21.179 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.179 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76837fa9:::benchmark_data_smithi171_78918_object82895:head# 2024-04-20T17:11:21.179 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.179 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:768383a7:::benchmark_data_smithi171_78918_object152404:head# 2024-04-20T17:11:21.179 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.179 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76848aa9:::benchmark_data_smithi171_78918_object115222:head# 2024-04-20T17:11:21.179 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.179 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76850c16:::benchmark_data_smithi171_78918_object120855:head# 2024-04-20T17:11:21.179 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.179 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76853962:::benchmark_data_smithi171_78918_object98532:head# 2024-04-20T17:11:21.179 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.179 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76866e96:::benchmark_data_smithi171_78918_object97771:head# 2024-04-20T17:11:21.179 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.180 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76869299:::benchmark_data_smithi171_78918_object117881:head# 2024-04-20T17:11:21.180 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.180 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7686b588:::benchmark_data_smithi171_78918_object105266:head# 2024-04-20T17:11:21.180 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.180 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7686d6b6:::benchmark_data_smithi171_78918_object155049:head# 2024-04-20T17:11:21.180 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.180 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7687a398:::benchmark_data_smithi171_78918_object71628:head# 2024-04-20T17:11:21.180 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.180 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7687eec4:::benchmark_data_smithi171_78918_object71426:head# 2024-04-20T17:11:21.180 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.180 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7688d2c2:::benchmark_data_smithi171_78918_object128022:head# 2024-04-20T17:11:21.180 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.180 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7688f086:::benchmark_data_smithi171_78918_object138666:head# 2024-04-20T17:11:21.180 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.181 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7689c218:::benchmark_data_smithi171_78918_object133094:head# 2024-04-20T17:11:21.181 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.181 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:768a2836:::benchmark_data_smithi171_78918_object72390:head# 2024-04-20T17:11:21.181 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.181 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:768af297:::benchmark_data_smithi171_78918_object103938:head# 2024-04-20T17:11:21.181 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.181 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:768ba3a3:::benchmark_data_smithi171_78918_object89462:head# 2024-04-20T17:11:21.181 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.181 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:768bb3d3:::benchmark_data_smithi171_78918_object82980:head# 2024-04-20T17:11:21.181 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.181 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:768c2542:::benchmark_data_smithi171_78918_object53139:head# 2024-04-20T17:11:21.181 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.181 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:768d9e1c:::benchmark_data_smithi171_78918_object137142:head# 2024-04-20T17:11:21.181 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.181 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:768de494:::benchmark_data_smithi171_78918_object108175:head# 2024-04-20T17:11:21.182 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.182 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:768e9225:::benchmark_data_smithi171_78918_object56776:head# 2024-04-20T17:11:21.182 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.182 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:768eebfd:::benchmark_data_smithi171_78918_object67484:head# 2024-04-20T17:11:21.182 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.182 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:768f1ba0:::benchmark_data_smithi171_78918_object65919:head# 2024-04-20T17:11:21.182 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.182 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:768f70aa:::benchmark_data_smithi171_78918_object128403:head# 2024-04-20T17:11:21.182 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.182 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:768fcc3f:::benchmark_data_smithi171_78918_object160429:head# 2024-04-20T17:11:21.182 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.182 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:768fe6af:::benchmark_data_smithi171_78918_object128125:head# 2024-04-20T17:11:21.182 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.182 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7690825c:::benchmark_data_smithi171_78918_object70632:head# 2024-04-20T17:11:21.182 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.183 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7690b711:::benchmark_data_smithi171_78918_object134365:head# 2024-04-20T17:11:21.183 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.183 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7690c3fd:::benchmark_data_smithi171_78918_object79543:head# 2024-04-20T17:11:21.183 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.183 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7690ca9e:::benchmark_data_smithi171_78918_object98306:head# 2024-04-20T17:11:21.183 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.183 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7690f53f:::benchmark_data_smithi171_78918_object55214:head# 2024-04-20T17:11:21.183 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.183 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7691b03f:::benchmark_data_smithi171_78918_object137116:head# 2024-04-20T17:11:21.183 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.184 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7691ec07:::benchmark_data_smithi171_78918_object75531:head# 2024-04-20T17:11:21.184 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.184 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7692c647:::benchmark_data_smithi171_78918_object111318:head# 2024-04-20T17:11:21.184 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.184 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76930bf6:::benchmark_data_smithi171_78918_object61993:head# 2024-04-20T17:11:21.184 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.184 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76934312:::benchmark_data_smithi171_78918_object141812:head# 2024-04-20T17:11:21.184 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.184 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7693746d:::benchmark_data_smithi171_78918_object147616:head# 2024-04-20T17:11:21.184 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.184 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76937c46:::benchmark_data_smithi171_78918_object55775:head# 2024-04-20T17:11:21.184 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.184 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76938d11:::benchmark_data_smithi171_78918_object141869:head# 2024-04-20T17:11:21.185 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.185 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:769416bb:::benchmark_data_smithi171_78918_object139190:head# 2024-04-20T17:11:21.185 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.185 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7694f107:::benchmark_data_smithi171_78918_object138197:head# 2024-04-20T17:11:21.185 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.185 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7694fc46:::benchmark_data_smithi171_78918_object64761:head# 2024-04-20T17:11:21.185 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.185 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7695b375:::benchmark_data_smithi171_78918_object156581:head# 2024-04-20T17:11:21.185 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.185 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76971b16:::benchmark_data_smithi171_78918_object82565:head# 2024-04-20T17:11:21.185 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.185 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76974081:::benchmark_data_smithi171_78918_object99373:head# 2024-04-20T17:11:21.185 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.185 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7697d878:::benchmark_data_smithi171_78918_object147181:head# 2024-04-20T17:11:21.185 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.186 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76985828:::benchmark_data_smithi171_78918_object106780:head# 2024-04-20T17:11:21.186 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.186 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:769947a3:::benchmark_data_smithi171_78918_object57981:head# 2024-04-20T17:11:21.186 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.186 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:769a9e1e:::benchmark_data_smithi171_78918_object80441:head# 2024-04-20T17:11:21.186 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.186 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:769b2767:::benchmark_data_smithi171_78918_object105902:head# 2024-04-20T17:11:21.186 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.186 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:769c519e:::benchmark_data_smithi171_78918_object67317:head# 2024-04-20T17:11:21.186 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.186 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:769c9cbe:::benchmark_data_smithi171_78918_object56769:head# 2024-04-20T17:11:21.186 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.186 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:769d6327:::benchmark_data_smithi171_78918_object79428:head# 2024-04-20T17:11:21.187 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.187 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:769d99ee:::benchmark_data_smithi171_78918_object86201:head# 2024-04-20T17:11:21.187 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.187 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:769dd68d:::benchmark_data_smithi171_78918_object114373:head# 2024-04-20T17:11:21.187 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.187 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:769f39f6:::benchmark_data_smithi171_78918_object145017:head# 2024-04-20T17:11:21.187 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.187 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76a02770:::benchmark_data_smithi171_78918_object136505:head# 2024-04-20T17:11:21.187 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.187 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76a07ad8:::benchmark_data_smithi171_78918_object85121:head# 2024-04-20T17:11:21.188 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.188 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76a1104d:::benchmark_data_smithi171_78918_object131673:head# 2024-04-20T17:11:21.188 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.188 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76a18780:::benchmark_data_smithi171_78918_object130061:head# 2024-04-20T17:11:21.188 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.188 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76a25c24:::benchmark_data_smithi171_78918_object101989:head# 2024-04-20T17:11:21.188 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.188 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76a2da82:::benchmark_data_smithi171_78918_object156170:head# 2024-04-20T17:11:21.188 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.189 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76a34331:::benchmark_data_smithi171_78918_object152585:head# 2024-04-20T17:11:21.189 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.189 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76a3bc2a:::benchmark_data_smithi171_78918_object78289:head# 2024-04-20T17:11:21.189 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.189 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76a3ce83:::benchmark_data_smithi171_78918_object53520:head# 2024-04-20T17:11:21.189 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.189 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76a3ecb1:::benchmark_data_smithi171_78918_object67721:head# 2024-04-20T17:11:21.189 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.189 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76a57e8b:::benchmark_data_smithi171_78918_object117412:head# 2024-04-20T17:11:21.189 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.189 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76a5a037:::benchmark_data_smithi171_78918_object143227:head# 2024-04-20T17:11:21.189 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.189 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76a5d893:::benchmark_data_smithi171_78918_object55846:head# 2024-04-20T17:11:21.190 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.190 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76a8982f:::benchmark_data_smithi171_78918_object148612:head# 2024-04-20T17:11:21.190 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.190 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76a90303:::benchmark_data_smithi171_78918_object57672:head# 2024-04-20T17:11:21.190 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.190 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76a97483:::benchmark_data_smithi171_78918_object125250:head# 2024-04-20T17:11:21.190 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.190 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76aa336f:::benchmark_data_smithi171_78918_object145894:head# 2024-04-20T17:11:21.190 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.190 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76aa4431:::benchmark_data_smithi171_78918_object126903:head# 2024-04-20T17:11:21.190 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.190 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76aba524:::benchmark_data_smithi171_78918_object115237:head# 2024-04-20T17:11:21.190 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.190 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76abe011:::benchmark_data_smithi171_78918_object104601:head# 2024-04-20T17:11:21.191 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.191 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76abebd0:::benchmark_data_smithi171_78918_object61354:head# 2024-04-20T17:11:21.191 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.191 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76ac700d:::benchmark_data_smithi171_78918_object115474:head# 2024-04-20T17:11:21.191 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.191 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76ad360c:::benchmark_data_smithi171_78918_object144108:head# 2024-04-20T17:11:21.191 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.191 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76ad506f:::benchmark_data_smithi171_78918_object71446:head# 2024-04-20T17:11:21.191 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.191 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76ad7281:::benchmark_data_smithi171_78918_object105223:head# 2024-04-20T17:11:21.191 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.191 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76adddcd:::benchmark_data_smithi171_78918_object133655:head# 2024-04-20T17:11:21.191 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.191 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76adf0d9:::benchmark_data_smithi171_78918_object143418:head# 2024-04-20T17:11:21.192 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.192 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76b1ac4a:::benchmark_data_smithi171_78918_object146426:head# 2024-04-20T17:11:21.192 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.192 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76b25cde:::benchmark_data_smithi171_78918_object128640:head# 2024-04-20T17:11:21.192 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.192 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76b316b5:::benchmark_data_smithi171_78918_object61490:head# 2024-04-20T17:11:21.192 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.192 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76b34253:::benchmark_data_smithi171_78918_object62748:head# 2024-04-20T17:11:21.192 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.192 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76b3a1a3:::benchmark_data_smithi171_78918_object93752:head# 2024-04-20T17:11:21.192 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.192 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76b3bedf:::benchmark_data_smithi171_78918_object153629:head# 2024-04-20T17:11:21.192 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.192 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76b40982:::benchmark_data_smithi171_78918_object109794:head# 2024-04-20T17:11:21.193 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.193 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76b42455:::benchmark_data_smithi171_78918_object104420:head# 2024-04-20T17:11:21.193 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.193 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76b479a6:::benchmark_data_smithi171_78918_object136681:head# 2024-04-20T17:11:21.193 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.193 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76b5e234:::benchmark_data_smithi171_78918_object60553:head# 2024-04-20T17:11:21.193 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.193 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76b6a2ba:::benchmark_data_smithi171_78918_object86817:head# 2024-04-20T17:11:21.193 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.193 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76b6b376:::benchmark_data_smithi171_78918_object87061:head# 2024-04-20T17:11:21.193 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.193 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76b75f1b:::benchmark_data_smithi171_78918_object71755:head# 2024-04-20T17:11:21.193 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.194 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76b7ac59:::benchmark_data_smithi171_78918_object104520:head# 2024-04-20T17:11:21.194 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.194 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76b82f95:::benchmark_data_smithi171_78918_object63045:head# 2024-04-20T17:11:21.194 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.194 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76b898c4:::benchmark_data_smithi171_78918_object72436:head# 2024-04-20T17:11:21.194 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.194 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76b8aecf:::benchmark_data_smithi171_78918_object92116:head# 2024-04-20T17:11:21.194 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.194 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76ba6831:::benchmark_data_smithi171_78918_object66214:head# 2024-04-20T17:11:21.194 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.194 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76ba9165:::benchmark_data_smithi171_78918_object116356:head# 2024-04-20T17:11:21.197 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.197 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76bb9763:::benchmark_data_smithi171_78918_object134809:head# 2024-04-20T17:11:21.198 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.198 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76bc41f2:::benchmark_data_smithi171_78918_object96688:head# 2024-04-20T17:11:21.198 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.198 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76c002d5:::benchmark_data_smithi171_78918_object74287:head# 2024-04-20T17:11:21.198 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.198 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76c01612:::benchmark_data_smithi171_78918_object147586:head# 2024-04-20T17:11:21.198 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.198 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76c09396:::benchmark_data_smithi171_78918_object120604:head# 2024-04-20T17:11:21.198 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.198 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76c0f2ff:::benchmark_data_smithi171_78918_object151555:head# 2024-04-20T17:11:21.198 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.198 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76c1c664:::benchmark_data_smithi171_78918_object58057:head# 2024-04-20T17:11:21.198 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.199 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76c21f3e:::benchmark_data_smithi171_78918_object92667:head# 2024-04-20T17:11:21.199 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.199 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76c2b9af:::benchmark_data_smithi171_78918_object123946:head# 2024-04-20T17:11:21.199 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.199 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76c48d94:::benchmark_data_smithi171_78918_object124997:head# 2024-04-20T17:11:21.199 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.199 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76c6362b:::benchmark_data_smithi171_78918_object58860:head# 2024-04-20T17:11:21.199 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.199 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76c69fff:::benchmark_data_smithi171_78918_object76297:head# 2024-04-20T17:11:21.199 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.199 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76c6b046:::benchmark_data_smithi171_78918_object155693:head# 2024-04-20T17:11:21.199 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.199 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76c8d30d:::benchmark_data_smithi171_78918_object138892:head# 2024-04-20T17:11:21.199 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.200 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76c8e412:::benchmark_data_smithi171_78918_object149474:head# 2024-04-20T17:11:21.200 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.200 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76c927d8:::benchmark_data_smithi171_78918_object89798:head# 2024-04-20T17:11:21.200 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.200 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76c9fb1b:::benchmark_data_smithi171_78918_object79560:head# 2024-04-20T17:11:21.200 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.200 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76cabb26:::benchmark_data_smithi171_78918_object132288:head# 2024-04-20T17:11:21.200 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.200 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76cc77ad:::benchmark_data_smithi171_78918_object55960:head# 2024-04-20T17:11:21.200 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.200 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76ce47d5:::benchmark_data_smithi171_78918_object101853:head# 2024-04-20T17:11:21.200 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.200 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76ce7449:::benchmark_data_smithi171_78918_object109567:head# 2024-04-20T17:11:21.200 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.200 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76ce92c6:::benchmark_data_smithi171_78918_object109676:head# 2024-04-20T17:11:21.201 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.201 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76ced28d:::benchmark_data_smithi171_78918_object150144:head# 2024-04-20T17:11:21.201 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.201 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76cf40ea:::benchmark_data_smithi171_78918_object62716:head# 2024-04-20T17:11:21.201 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.201 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76d00da3:::benchmark_data_smithi171_78918_object144104:head# 2024-04-20T17:11:21.201 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.201 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76d120c9:::benchmark_data_smithi171_78918_object101035:head# 2024-04-20T17:11:21.201 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.201 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76d19510:::benchmark_data_smithi171_78918_object111330:head# 2024-04-20T17:11:21.201 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.201 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76d24cc5:::benchmark_data_smithi171_78918_object90712:head# 2024-04-20T17:11:21.201 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.202 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76d26b81:::benchmark_data_smithi171_78918_object114671:head# 2024-04-20T17:11:21.202 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.202 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76d29afd:::benchmark_data_smithi171_78918_object113473:head# 2024-04-20T17:11:21.202 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.202 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76d33096:::benchmark_data_smithi171_78918_object61459:head# 2024-04-20T17:11:21.202 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.202 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76d36a25:::benchmark_data_smithi171_78918_object134012:head# 2024-04-20T17:11:21.202 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.202 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76d38e40:::benchmark_data_smithi171_78918_object65645:head# 2024-04-20T17:11:21.202 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.202 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76d39073:::benchmark_data_smithi171_78918_object110193:head# 2024-04-20T17:11:21.202 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.202 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76d41cf2:::benchmark_data_smithi171_78918_object136459:head# 2024-04-20T17:11:21.203 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.203 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76d64c89:::benchmark_data_smithi171_78918_object70121:head# 2024-04-20T17:11:21.203 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.203 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76d76276:::benchmark_data_smithi171_78918_object108244:head# 2024-04-20T17:11:21.203 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.203 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76d7e50b:::benchmark_data_smithi171_78918_object60345:head# 2024-04-20T17:11:21.203 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.203 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76d7f30e:::benchmark_data_smithi171_78918_object65083:head# 2024-04-20T17:11:21.203 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.203 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76d84b18:::benchmark_data_smithi171_78918_object119049:head# 2024-04-20T17:11:21.203 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.203 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76d8c805:::benchmark_data_smithi171_78918_object137838:head# 2024-04-20T17:11:21.203 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.203 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76d8d3f3:::benchmark_data_smithi171_78918_object127229:head# 2024-04-20T17:11:21.203 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.204 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76d8d41b:::benchmark_data_smithi171_78918_object123215:head# 2024-04-20T17:11:21.204 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.204 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76d9b451:::benchmark_data_smithi171_78918_object104860:head# 2024-04-20T17:11:21.204 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.204 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76dbc447:::benchmark_data_smithi171_78918_object160233:head# 2024-04-20T17:11:21.204 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.204 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76dc0719:::benchmark_data_smithi171_78918_object150381:head# 2024-04-20T17:11:21.204 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.204 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76ddb5b3:::benchmark_data_smithi171_78918_object137313:head# 2024-04-20T17:11:21.204 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.204 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76ddd978:::benchmark_data_smithi171_78918_object54412:head# 2024-04-20T17:11:21.204 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.204 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76ddedaf:::benchmark_data_smithi171_78918_object74415:head# 2024-04-20T17:11:21.204 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.205 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76de1def:::benchmark_data_smithi171_78918_object134749:head# 2024-04-20T17:11:21.205 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.205 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76de6811:::benchmark_data_smithi171_78918_object109433:head# 2024-04-20T17:11:21.205 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.205 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76df7e26:::benchmark_data_smithi171_78918_object134025:head# 2024-04-20T17:11:21.205 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.205 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76e0063a:::benchmark_data_smithi171_78918_object96741:head# 2024-04-20T17:11:21.205 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.205 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76e095a8:::benchmark_data_smithi171_78918_object123438:head# 2024-04-20T17:11:21.205 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.205 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76e0bd19:::benchmark_data_smithi171_78918_object130274:head# 2024-04-20T17:11:21.205 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.205 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76e14eea:::benchmark_data_smithi171_78918_object129415:head# 2024-04-20T17:11:21.205 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.206 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76e22924:::benchmark_data_smithi171_78918_object104846:head# 2024-04-20T17:11:21.206 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.206 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76e41161:::benchmark_data_smithi171_78918_object76341:head# 2024-04-20T17:11:21.206 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.206 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76e584b5:::benchmark_data_smithi171_78918_object57435:head# 2024-04-20T17:11:21.206 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.206 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76e704b3:::benchmark_data_smithi171_78918_object53221:head# 2024-04-20T17:11:21.206 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.206 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76e797c4:::benchmark_data_smithi171_78918_object57638:head# 2024-04-20T17:11:21.206 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.206 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76e7f387:::benchmark_data_smithi171_78918_object139364:head# 2024-04-20T17:11:21.206 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.206 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76e8ca04:::benchmark_data_smithi171_78918_object81938:head# 2024-04-20T17:11:21.207 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.207 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76e9816e:::benchmark_data_smithi171_78918_object102028:head# 2024-04-20T17:11:21.207 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.207 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76e9f8e7:::benchmark_data_smithi171_78918_object142694:head# 2024-04-20T17:11:21.207 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.207 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76e9ff36:::benchmark_data_smithi171_78918_object133092:head# 2024-04-20T17:11:21.207 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.207 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76ea0602:::benchmark_data_smithi171_78918_object151473:head# 2024-04-20T17:11:21.207 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.207 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76ea90f9:::benchmark_data_smithi171_78918_object86581:head# 2024-04-20T17:11:21.207 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.207 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76eaacd0:::benchmark_data_smithi171_78918_object75302:head# 2024-04-20T17:11:21.207 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.207 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76eb0dbe:::benchmark_data_smithi171_78918_object119007:head# 2024-04-20T17:11:21.208 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.208 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76ec7b48:::benchmark_data_smithi171_78918_object90899:head# 2024-04-20T17:11:21.208 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.208 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76ecc1db:::benchmark_data_smithi171_78918_object68317:head# 2024-04-20T17:11:21.208 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.208 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76ed8ebc:::benchmark_data_smithi171_78918_object157288:head# 2024-04-20T17:11:21.208 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.208 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76eee17d:::benchmark_data_smithi171_78918_object74500:head# 2024-04-20T17:11:21.208 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.208 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76ef030c:::benchmark_data_smithi171_78918_object69339:head# 2024-04-20T17:11:21.208 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.208 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76ef0ea6:::benchmark_data_smithi171_78918_object59165:head# 2024-04-20T17:11:21.208 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.208 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76ef268b:::benchmark_data_smithi171_78918_object118202:head# 2024-04-20T17:11:21.209 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.209 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76f0535f:::benchmark_data_smithi171_78918_object63163:head# 2024-04-20T17:11:21.209 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.209 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76f11063:::benchmark_data_smithi171_78918_object86064:head# 2024-04-20T17:11:21.209 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.209 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76f19101:::benchmark_data_smithi171_78918_object137288:head# 2024-04-20T17:11:21.209 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.209 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76f230bc:::benchmark_data_smithi171_78918_object91420:head# 2024-04-20T17:11:21.209 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.209 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76f27194:::benchmark_data_smithi171_78918_object87727:head# 2024-04-20T17:11:21.209 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.209 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76f2779b:::benchmark_data_smithi171_78918_object129055:head# 2024-04-20T17:11:21.209 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.210 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76f37afe:::benchmark_data_smithi171_78918_object152151:head# 2024-04-20T17:11:21.210 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.210 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76f3a73e:::benchmark_data_smithi171_78918_object55593:head# 2024-04-20T17:11:21.210 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.210 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76f5a4bd:::benchmark_data_smithi171_78918_object104090:head# 2024-04-20T17:11:21.210 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.210 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76f5f948:::benchmark_data_smithi171_78918_object143545:head# 2024-04-20T17:11:21.210 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.210 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76f723be:::benchmark_data_smithi171_78918_object91452:head# 2024-04-20T17:11:21.210 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.210 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76f74aa8:::benchmark_data_smithi171_78918_object137213:head# 2024-04-20T17:11:21.210 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.210 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76f765f6:::benchmark_data_smithi171_78918_object56624:head# 2024-04-20T17:11:21.211 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.211 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76f7fd08:::benchmark_data_smithi171_78918_object156759:head# 2024-04-20T17:11:21.211 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.211 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76f83544:::benchmark_data_smithi171_78918_object139569:head# 2024-04-20T17:11:21.211 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.211 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76f89d3b:::benchmark_data_smithi171_78918_object121507:head# 2024-04-20T17:11:21.211 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.211 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76f8e88e:::benchmark_data_smithi171_78918_object151867:head# 2024-04-20T17:11:21.211 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.211 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76f8ecab:::benchmark_data_smithi171_78918_object121034:head# 2024-04-20T17:11:21.211 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.211 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76f97cde:::benchmark_data_smithi171_78918_object97079:head# 2024-04-20T17:11:21.211 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.211 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76f9bd7a:::benchmark_data_smithi171_78918_object150039:head# 2024-04-20T17:11:21.211 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.212 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76fa11f0:::benchmark_data_smithi171_78918_object112960:head# 2024-04-20T17:11:21.212 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.212 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76fa94e5:::benchmark_data_smithi171_78918_object115549:head# 2024-04-20T17:11:21.212 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.212 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76facc8a:::benchmark_data_smithi171_78918_object134800:head# 2024-04-20T17:11:21.212 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.212 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76faf8f5:::benchmark_data_smithi171_78918_object144411:head# 2024-04-20T17:11:21.212 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.212 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76fb163a:::benchmark_data_smithi171_78918_object155632:head# 2024-04-20T17:11:21.212 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.212 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76fb7d33:::benchmark_data_smithi171_78918_object143672:head# 2024-04-20T17:11:21.212 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.212 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76fbc398:::benchmark_data_smithi171_78918_object69917:head# 2024-04-20T17:11:21.212 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.213 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76fc3108:::benchmark_data_smithi171_78918_object136953:head# 2024-04-20T17:11:21.213 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.213 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76fd5cdf:::benchmark_data_smithi171_78918_object72780:head# 2024-04-20T17:11:21.213 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.213 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76fd6b1f:::benchmark_data_smithi171_78918_object157744:head# 2024-04-20T17:11:21.213 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.213 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76fe17c3:::benchmark_data_smithi171_78918_object72497:head# 2024-04-20T17:11:21.213 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.213 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76fe342a:::benchmark_data_smithi171_78918_object71492:head# 2024-04-20T17:11:21.213 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.213 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76fe7d16:::benchmark_data_smithi171_78918_object128176:head# 2024-04-20T17:11:21.213 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.213 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:76ffd846:::benchmark_data_smithi171_78918_object76577:head# 2024-04-20T17:11:21.213 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.214 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7700dc7c:::benchmark_data_smithi171_78918_object105439:head# 2024-04-20T17:11:21.214 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.214 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7700f02e:::benchmark_data_smithi171_78918_object122149:head# 2024-04-20T17:11:21.214 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.214 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77012bf3:::benchmark_data_smithi171_78918_object158077:head# 2024-04-20T17:11:21.214 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.214 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77019136:::benchmark_data_smithi171_78918_object117030:head# 2024-04-20T17:11:21.214 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.214 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77027bca:::benchmark_data_smithi171_78918_object73961:head# 2024-04-20T17:11:21.214 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.214 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77040a5e:::benchmark_data_smithi171_78918_object103324:head# 2024-04-20T17:11:21.214 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.214 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77050d22:::benchmark_data_smithi171_78918_object81375:head# 2024-04-20T17:11:21.214 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.214 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77051296:::benchmark_data_smithi171_78918_object56957:head# 2024-04-20T17:11:21.215 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.215 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77054e12:::benchmark_data_smithi171_78918_object104522:head# 2024-04-20T17:11:21.215 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.215 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7706ea88:::benchmark_data_smithi171_78918_object60043:head# 2024-04-20T17:11:21.215 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.215 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7707614a:::benchmark_data_smithi171_78918_object86770:head# 2024-04-20T17:11:21.215 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.215 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7707bace:::benchmark_data_smithi171_78918_object94443:head# 2024-04-20T17:11:21.215 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.215 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:770820d4:::benchmark_data_smithi171_78918_object118872:head# 2024-04-20T17:11:21.215 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.223 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7708af2f:::benchmark_data_smithi171_78918_object54941:head# 2024-04-20T17:11:21.223 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.224 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7708ea0a:::benchmark_data_smithi171_78918_object141654:head# 2024-04-20T17:11:21.224 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.224 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:770969c8:::benchmark_data_smithi171_78918_object107191:head# 2024-04-20T17:11:21.224 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.224 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:770b6b18:::benchmark_data_smithi171_78918_object121047:head# 2024-04-20T17:11:21.224 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.224 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:770b8ed5:::benchmark_data_smithi171_78918_object150787:head# 2024-04-20T17:11:21.224 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.224 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:770be7c7:::benchmark_data_smithi171_78918_object111623:head# 2024-04-20T17:11:21.224 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.224 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:770c6579:::benchmark_data_smithi171_78918_object149967:head# 2024-04-20T17:11:21.224 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.224 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:770cad40:::benchmark_data_smithi171_78918_object58425:head# 2024-04-20T17:11:21.224 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.225 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:770db127:::benchmark_data_smithi171_78918_object64206:head# 2024-04-20T17:11:21.225 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.225 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:770dce0f:::benchmark_data_smithi171_78918_object95466:head# 2024-04-20T17:11:21.225 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.225 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:770ea725:::benchmark_data_smithi171_78918_object149281:head# 2024-04-20T17:11:21.225 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.225 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:770f849d:::benchmark_data_smithi171_78918_object81661:head# 2024-04-20T17:11:21.225 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.225 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77102f9e:::benchmark_data_smithi171_78918_object58478:head# 2024-04-20T17:11:21.225 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.225 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:771175f7:::benchmark_data_smithi171_78918_object148455:head# 2024-04-20T17:11:21.225 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.225 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7711c7d5:::benchmark_data_smithi171_78918_object153688:head# 2024-04-20T17:11:21.225 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.226 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7711cc10:::benchmark_data_smithi171_78918_object136755:head# 2024-04-20T17:11:21.226 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.226 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7712ab32:::benchmark_data_smithi171_78918_object131055:head# 2024-04-20T17:11:21.226 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.226 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:771384a0:::benchmark_data_smithi171_78918_object146631:head# 2024-04-20T17:11:21.226 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.226 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77145c62:::benchmark_data_smithi171_78918_object158839:head# 2024-04-20T17:11:21.226 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.226 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7715c3b6:::benchmark_data_smithi171_78918_object145304:head# 2024-04-20T17:11:21.226 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.226 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:771660a4:::benchmark_data_smithi171_78918_object79801:head# 2024-04-20T17:11:21.226 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.226 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7716ede2:::benchmark_data_smithi171_78918_object121163:head# 2024-04-20T17:11:21.226 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.227 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7717bf43:::benchmark_data_smithi171_78918_object95172:head# 2024-04-20T17:11:21.227 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.227 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7718d5c6:::benchmark_data_smithi171_78918_object158175:head# 2024-04-20T17:11:21.227 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.227 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7718d88a:::benchmark_data_smithi171_78918_object150040:head# 2024-04-20T17:11:21.227 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.227 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77192f47:::benchmark_data_smithi171_78918_object73036:head# 2024-04-20T17:11:21.227 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.227 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:771a18ff:::benchmark_data_smithi171_78918_object62045:head# 2024-04-20T17:11:21.227 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.227 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:771b250c:::benchmark_data_smithi171_78918_object89950:head# 2024-04-20T17:11:21.227 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.227 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:771bc468:::benchmark_data_smithi171_78918_object105060:head# 2024-04-20T17:11:21.227 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.228 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:771c4a1f:::benchmark_data_smithi171_78918_object90036:head# 2024-04-20T17:11:21.228 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.228 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:771c61e0:::benchmark_data_smithi171_78918_object128091:head# 2024-04-20T17:11:21.228 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.228 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:771ccb88:::benchmark_data_smithi171_78918_object149765:head# 2024-04-20T17:11:21.228 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.228 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:771d0884:::benchmark_data_smithi171_78918_object137238:head# 2024-04-20T17:11:21.228 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.228 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:771ef6f7:::benchmark_data_smithi171_78918_object59064:head# 2024-04-20T17:11:21.228 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.228 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:771ff16a:::benchmark_data_smithi171_78918_object124423:head# 2024-04-20T17:11:21.228 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.228 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77208b60:::benchmark_data_smithi171_78918_object141051:head# 2024-04-20T17:11:21.228 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.229 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7720da55:::benchmark_data_smithi171_78918_object99602:head# 2024-04-20T17:11:21.229 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.229 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7720fb79:::benchmark_data_smithi171_78918_object157160:head# 2024-04-20T17:11:21.229 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.229 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7721257b:::benchmark_data_smithi171_78918_object57543:head# 2024-04-20T17:11:21.229 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.229 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7722f2f1:::benchmark_data_smithi171_78918_object77296:head# 2024-04-20T17:11:21.229 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.229 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77233c29:::benchmark_data_smithi171_78918_object115278:head# 2024-04-20T17:11:21.229 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.229 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7724957c:::benchmark_data_smithi171_78918_object81321:head# 2024-04-20T17:11:21.229 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.229 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7724977c:::benchmark_data_smithi171_78918_object160253:head# 2024-04-20T17:11:21.230 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.230 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7724bff1:::benchmark_data_smithi171_78918_object55184:head# 2024-04-20T17:11:21.230 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.230 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77259bcd:::benchmark_data_smithi171_78918_object122074:head# 2024-04-20T17:11:21.230 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.230 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:772673ac:::benchmark_data_smithi171_78918_object141821:head# 2024-04-20T17:11:21.230 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.230 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7726d354:::benchmark_data_smithi171_78918_object76324:head# 2024-04-20T17:11:21.230 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.230 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77288cf1:::benchmark_data_smithi171_78918_object159377:head# 2024-04-20T17:11:21.230 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.230 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77296e7c:::benchmark_data_smithi171_78918_object53005:head# 2024-04-20T17:11:21.230 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.230 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77299da1:::benchmark_data_smithi171_78918_object87024:head# 2024-04-20T17:11:21.231 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.231 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:772b0a43:::benchmark_data_smithi171_78918_object65162:head# 2024-04-20T17:11:21.231 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.231 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:772b37d1:::benchmark_data_smithi171_78918_object53900:head# 2024-04-20T17:11:21.231 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.231 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:772bc92b:::benchmark_data_smithi171_78918_object76712:head# 2024-04-20T17:11:21.231 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.231 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:772c8dbb:::benchmark_data_smithi171_78918_object78166:head# 2024-04-20T17:11:21.231 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.231 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:772c9105:::benchmark_data_smithi171_78918_object95755:head# 2024-04-20T17:11:21.231 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.231 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:772ce678:::benchmark_data_smithi171_78918_object134358:head# 2024-04-20T17:11:21.231 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.231 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:772d4236:::benchmark_data_smithi171_78918_object133237:head# 2024-04-20T17:11:21.231 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.232 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:772dd52b:::benchmark_data_smithi171_78918_object78182:head# 2024-04-20T17:11:21.232 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.232 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:772e132e:::benchmark_data_smithi171_78918_object158559:head# 2024-04-20T17:11:21.232 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.232 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:772ecb72:::benchmark_data_smithi171_78918_object100311:head# 2024-04-20T17:11:21.232 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.232 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7730b7b7:::benchmark_data_smithi171_78918_object87687:head# 2024-04-20T17:11:21.232 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.232 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7730bd7b:::benchmark_data_smithi171_78918_object122804:head# 2024-04-20T17:11:21.232 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.232 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7731424d:::benchmark_data_smithi171_78918_object117450:head# 2024-04-20T17:11:21.232 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.232 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:773143a6:::benchmark_data_smithi171_78918_object83836:head# 2024-04-20T17:11:21.233 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.233 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77314686:::benchmark_data_smithi171_78918_object90947:head# 2024-04-20T17:11:21.233 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.233 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7731702a:::benchmark_data_smithi171_78918_object126808:head# 2024-04-20T17:11:21.233 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.233 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:773176cb:::benchmark_data_smithi171_78918_object58768:head# 2024-04-20T17:11:21.233 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.233 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77320bbf:::benchmark_data_smithi171_78918_object81169:head# 2024-04-20T17:11:21.233 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.233 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77325705:::benchmark_data_smithi171_78918_object126725:head# 2024-04-20T17:11:21.233 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.233 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7733f09b:::benchmark_data_smithi171_78918_object63040:head# 2024-04-20T17:11:21.233 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.233 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77340b22:::benchmark_data_smithi171_78918_object92500:head# 2024-04-20T17:11:21.233 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.234 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7734daec:::benchmark_data_smithi171_78918_object75985:head# 2024-04-20T17:11:21.234 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.234 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7735a850:::benchmark_data_smithi171_78918_object82691:head# 2024-04-20T17:11:21.234 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.234 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7735b579:::benchmark_data_smithi171_78918_object122105:head# 2024-04-20T17:11:21.234 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.234 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:773634b9:::benchmark_data_smithi171_78918_object120658:head# 2024-04-20T17:11:21.234 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.234 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77369bdd:::benchmark_data_smithi171_78918_object139054:head# 2024-04-20T17:11:21.234 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.234 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7736f1ed:::benchmark_data_smithi171_78918_object122671:head# 2024-04-20T17:11:21.234 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.234 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7736f39d:::benchmark_data_smithi171_78918_object63421:head# 2024-04-20T17:11:21.234 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.234 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:773790f8:::benchmark_data_smithi171_78918_object72936:head# 2024-04-20T17:11:21.235 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.235 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7737b555:::benchmark_data_smithi171_78918_object155151:head# 2024-04-20T17:11:21.235 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.235 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:773803df:::benchmark_data_smithi171_78918_object160688:head# 2024-04-20T17:11:21.235 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.235 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77384bba:::benchmark_data_smithi171_78918_object110259:head# 2024-04-20T17:11:21.235 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.235 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77385538:::benchmark_data_smithi171_78918_object95941:head# 2024-04-20T17:11:21.235 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.235 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7739eff9:::benchmark_data_smithi171_78918_object88025:head# 2024-04-20T17:11:21.235 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.235 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7739fd6d:::benchmark_data_smithi171_78918_object125730:head# 2024-04-20T17:11:21.235 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.235 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:773aaa76:::benchmark_data_smithi171_78918_object84550:head# 2024-04-20T17:11:21.235 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.236 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:773b136b:::benchmark_data_smithi171_78918_object157396:head# 2024-04-20T17:11:21.236 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.236 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:773e591e:::benchmark_data_smithi171_78918_object153480:head# 2024-04-20T17:11:21.236 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.236 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:773eb55a:::benchmark_data_smithi171_78918_object72749:head# 2024-04-20T17:11:21.236 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.236 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:773f5027:::benchmark_data_smithi171_78918_object133587:head# 2024-04-20T17:11:21.236 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.236 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7740c137:::benchmark_data_smithi171_78918_object93046:head# 2024-04-20T17:11:21.236 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.236 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:774175b7:::benchmark_data_smithi171_78918_object115057:head# 2024-04-20T17:11:21.236 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.236 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7741e919:::benchmark_data_smithi171_78918_object68687:head# 2024-04-20T17:11:21.236 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.237 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77423b83:::benchmark_data_smithi171_78918_object92705:head# 2024-04-20T17:11:21.237 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.237 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77446e24:::benchmark_data_smithi171_78918_object130991:head# 2024-04-20T17:11:21.237 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.237 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7744d8cd:::benchmark_data_smithi171_78918_object87553:head# 2024-04-20T17:11:21.237 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.237 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77455849:::benchmark_data_smithi171_78918_object120883:head# 2024-04-20T17:11:21.237 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.237 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77456b53:::benchmark_data_smithi171_78918_object155121:head# 2024-04-20T17:11:21.237 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.237 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:774882c2:::benchmark_data_smithi171_78918_object81670:head# 2024-04-20T17:11:21.237 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.238 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:21.301 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77493b00:::benchmark_data_smithi171_78918_object105484:head# 2024-04-20T17:11:21.301 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.301 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7749a976:::benchmark_data_smithi171_78918_object76396:head# 2024-04-20T17:11:21.301 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.301 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:774a650e:::benchmark_data_smithi171_78918_object81007:head# 2024-04-20T17:11:21.301 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.301 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:774be869:::benchmark_data_smithi171_78918_object103401:head# 2024-04-20T17:11:21.301 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.301 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:774c5dc1:::benchmark_data_smithi171_78918_object153380:head# 2024-04-20T17:11:21.302 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.302 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:774cef92:::benchmark_data_smithi171_78918_object137491:head# 2024-04-20T17:11:21.302 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.302 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:774de089:::benchmark_data_smithi171_78918_object124115:head# 2024-04-20T17:11:21.302 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.302 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:774e3d97:::benchmark_data_smithi171_78918_object152895:head# 2024-04-20T17:11:21.302 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.302 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:774e58df:::benchmark_data_smithi171_78918_object160953:head# 2024-04-20T17:11:21.302 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.302 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:774e5d43:::benchmark_data_smithi171_78918_object62105:head# 2024-04-20T17:11:21.302 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.302 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:774e7353:::benchmark_data_smithi171_78918_object76070:head# 2024-04-20T17:11:21.302 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.303 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:774ec25c:::benchmark_data_smithi171_78918_object131647:head# 2024-04-20T17:11:21.303 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.303 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:774eee72:::benchmark_data_smithi171_78918_object67501:head# 2024-04-20T17:11:21.303 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.303 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:774faacb:::benchmark_data_smithi171_78918_object141891:head# 2024-04-20T17:11:21.303 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.303 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:774fb006:::benchmark_data_smithi171_78918_object101926:head# 2024-04-20T17:11:21.303 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.303 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:774fbe12:::benchmark_data_smithi171_78918_object58027:head# 2024-04-20T17:11:21.303 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.303 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:774ffaea:::benchmark_data_smithi171_78918_object70368:head# 2024-04-20T17:11:21.303 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.303 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7750611d:::benchmark_data_smithi171_78918_object110969:head# 2024-04-20T17:11:21.303 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.304 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:775127aa:::benchmark_data_smithi171_78918_object72217:head# 2024-04-20T17:11:21.304 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.304 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77516c1c:::benchmark_data_smithi171_78918_object148070:head# 2024-04-20T17:11:21.304 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.304 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7751e5b7:::benchmark_data_smithi171_78918_object143764:head# 2024-04-20T17:11:21.304 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.304 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77522d1b:::benchmark_data_smithi171_78918_object128165:head# 2024-04-20T17:11:21.304 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.304 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7752327b:::benchmark_data_smithi171_78918_object131655:head# 2024-04-20T17:11:21.304 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.304 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77527306:::benchmark_data_smithi171_78918_object112799:head# 2024-04-20T17:11:21.304 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.304 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77527f03:::benchmark_data_smithi171_78918_object73792:head# 2024-04-20T17:11:21.304 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.305 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7752e338:::benchmark_data_smithi171_78918_object86992:head# 2024-04-20T17:11:21.305 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.305 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7752ff3e:::benchmark_data_smithi171_78918_object154478:head# 2024-04-20T17:11:21.305 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.305 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77545c5f:::benchmark_data_smithi171_78918_object131207:head# 2024-04-20T17:11:21.305 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.305 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:775480a3:::benchmark_data_smithi171_78918_object71052:head# 2024-04-20T17:11:21.305 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.305 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77557df3:::benchmark_data_smithi171_78918_object67320:head# 2024-04-20T17:11:21.305 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.305 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77558290:::benchmark_data_smithi171_78918_object60824:head# 2024-04-20T17:11:21.305 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.305 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7755d51b:::benchmark_data_smithi171_78918_object115104:head# 2024-04-20T17:11:21.305 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.305 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7755dcbc:::benchmark_data_smithi171_78918_object74847:head# 2024-04-20T17:11:21.306 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.306 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:775690e8:::benchmark_data_smithi171_78918_object88568:head# 2024-04-20T17:11:21.306 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.306 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77574816:::benchmark_data_smithi171_78918_object92953:head# 2024-04-20T17:11:21.306 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.306 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7758d513:::benchmark_data_smithi171_78918_object75672:head# 2024-04-20T17:11:21.306 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.306 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:775910ce:::benchmark_data_smithi171_78918_object120437:head# 2024-04-20T17:11:21.306 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.306 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:775a2798:::benchmark_data_smithi171_78918_object112822:head# 2024-04-20T17:11:21.306 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.306 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:775c613c:::benchmark_data_smithi171_78918_object126809:head# 2024-04-20T17:11:21.306 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.307 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:775d0331:::benchmark_data_smithi171_78918_object70490:head# 2024-04-20T17:11:21.307 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.307 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:775d8234:::benchmark_data_smithi171_78918_object64024:head# 2024-04-20T17:11:21.307 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.307 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:775dddbd:::benchmark_data_smithi171_78918_object160267:head# 2024-04-20T17:11:21.307 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.307 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:775e1316:::benchmark_data_smithi171_78918_object143542:head# 2024-04-20T17:11:21.307 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.307 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:775e1e36:::benchmark_data_smithi171_78918_object64280:head# 2024-04-20T17:11:21.307 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.307 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:775e4cdd:::benchmark_data_smithi171_78918_object157153:head# 2024-04-20T17:11:21.307 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.307 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:775fac77:::benchmark_data_smithi171_78918_object68280:head# 2024-04-20T17:11:21.308 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.308 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77604786:::benchmark_data_smithi171_78918_object92487:head# 2024-04-20T17:11:21.308 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.308 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7760c053:::benchmark_data_smithi171_78918_object70308:head# 2024-04-20T17:11:21.308 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.308 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7760eb50:::benchmark_data_smithi171_78918_object88635:head# 2024-04-20T17:11:21.308 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.308 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:776100a0:::benchmark_data_smithi171_78918_object58767:head# 2024-04-20T17:11:21.308 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.308 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77611120:::benchmark_data_smithi171_78918_object85340:head# 2024-04-20T17:11:21.308 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.308 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77614197:::benchmark_data_smithi171_78918_object59325:head# 2024-04-20T17:11:21.308 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.308 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77625ff8:::benchmark_data_smithi171_78918_object67656:head# 2024-04-20T17:11:21.309 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.309 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7762900f:::benchmark_data_smithi171_78918_object96154:head# 2024-04-20T17:11:21.309 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.309 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7762ad8f:::benchmark_data_smithi171_78918_object86272:head# 2024-04-20T17:11:21.309 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.309 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7762b080:::benchmark_data_smithi171_78918_object115754:head# 2024-04-20T17:11:21.309 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.309 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7763f760:::benchmark_data_smithi171_78918_object126035:head# 2024-04-20T17:11:21.309 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.309 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7764fe41:::benchmark_data_smithi171_78918_object143088:head# 2024-04-20T17:11:21.309 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.309 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:776544f6:::benchmark_data_smithi171_78918_object96738:head# 2024-04-20T17:11:21.309 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.309 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77660b97:::benchmark_data_smithi171_78918_object117743:head# 2024-04-20T17:11:21.310 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.310 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77669b87:::benchmark_data_smithi171_78918_object114815:head# 2024-04-20T17:11:21.310 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.310 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7766bb9e:::benchmark_data_smithi171_78918_object96479:head# 2024-04-20T17:11:21.310 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.310 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77673392:::benchmark_data_smithi171_78918_object86792:head# 2024-04-20T17:11:21.310 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.310 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7767450c:::benchmark_data_smithi171_78918_object89117:head# 2024-04-20T17:11:21.310 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.310 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:776762e0:::benchmark_data_smithi171_78918_object67545:head# 2024-04-20T17:11:21.310 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.310 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:776797d0:::benchmark_data_smithi171_78918_object63568:head# 2024-04-20T17:11:21.310 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.310 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7767a538:::benchmark_data_smithi171_78918_object63603:head# 2024-04-20T17:11:21.311 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.311 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7767a86e:::benchmark_data_smithi171_78918_object62535:head# 2024-04-20T17:11:21.311 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.311 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7767aa8e:::benchmark_data_smithi171_78918_object152129:head# 2024-04-20T17:11:21.311 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.311 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7767d82c:::benchmark_data_smithi171_78918_object110183:head# 2024-04-20T17:11:21.311 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.311 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77697318:::benchmark_data_smithi171_78918_object144706:head# 2024-04-20T17:11:21.311 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.311 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:776b3789:::benchmark_data_smithi171_78918_object158696:head# 2024-04-20T17:11:21.311 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.311 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:776bbca6:::benchmark_data_smithi171_78918_object133612:head# 2024-04-20T17:11:21.311 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.311 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:776be6cf:::benchmark_data_smithi171_78918_object55548:head# 2024-04-20T17:11:21.311 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.312 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:776c17f1:::benchmark_data_smithi171_78918_object132746:head# 2024-04-20T17:11:21.312 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.312 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:776cf86f:::benchmark_data_smithi171_78918_object69102:head# 2024-04-20T17:11:21.312 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.312 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:776e67d7:::benchmark_data_smithi171_78918_object64258:head# 2024-04-20T17:11:21.312 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.312 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:776e6ac9:::benchmark_data_smithi171_78918_object138654:head# 2024-04-20T17:11:21.312 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.312 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:776f41d8:::benchmark_data_smithi171_78918_object157657:head# 2024-04-20T17:11:21.312 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.312 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77704173:::benchmark_data_smithi171_78918_object141086:head# 2024-04-20T17:11:21.312 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.312 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:777088ee:::benchmark_data_smithi171_78918_object130195:head# 2024-04-20T17:11:21.312 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.313 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7770cd27:::benchmark_data_smithi171_78918_object72256:head# 2024-04-20T17:11:21.313 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.313 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77714498:::benchmark_data_smithi171_78918_object141861:head# 2024-04-20T17:11:21.313 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.313 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77723962:::benchmark_data_smithi171_78918_object67873:head# 2024-04-20T17:11:21.313 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.313 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77727dcc:::benchmark_data_smithi171_78918_object93755:head# 2024-04-20T17:11:21.313 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.313 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7772840c:::benchmark_data_smithi171_78918_object139653:head# 2024-04-20T17:11:21.313 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.313 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77729b0d:::benchmark_data_smithi171_78918_object75798:head# 2024-04-20T17:11:21.313 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.313 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77733511:::benchmark_data_smithi171_78918_object66894:head# 2024-04-20T17:11:21.313 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.313 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7773aab7:::benchmark_data_smithi171_78918_object74503:head# 2024-04-20T17:11:21.314 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.314 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7773f59e:::benchmark_data_smithi171_78918_object100454:head# 2024-04-20T17:11:21.314 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.314 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77741ad6:::benchmark_data_smithi171_78918_object90206:head# 2024-04-20T17:11:21.314 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.314 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:777514c4:::benchmark_data_smithi171_78918_object74941:head# 2024-04-20T17:11:21.314 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.314 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:777536c1:::benchmark_data_smithi171_78918_object150943:head# 2024-04-20T17:11:21.314 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.314 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77756101:::benchmark_data_smithi171_78918_object119768:head# 2024-04-20T17:11:21.314 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.314 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77760d33:::benchmark_data_smithi171_78918_object135577:head# 2024-04-20T17:11:21.314 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.314 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77763282:::benchmark_data_smithi171_78918_object88272:head# 2024-04-20T17:11:21.315 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.315 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77767ae1:::benchmark_data_smithi171_78918_object118976:head# 2024-04-20T17:11:21.315 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.315 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7776eab1:::benchmark_data_smithi171_78918_object120085:head# 2024-04-20T17:11:21.315 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.315 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77790df1:::benchmark_data_smithi171_78918_object69254:head# 2024-04-20T17:11:21.315 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.315 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7779a018:::benchmark_data_smithi171_78918_object102437:head# 2024-04-20T17:11:21.315 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.315 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:777a2c5f:::benchmark_data_smithi171_78918_object84353:head# 2024-04-20T17:11:21.315 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.315 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:777aa8d0:::benchmark_data_smithi171_78918_object158944:head# 2024-04-20T17:11:21.315 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.316 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:777b50f6:::benchmark_data_smithi171_78918_object138119:head# 2024-04-20T17:11:21.316 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.316 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:777bec65:::benchmark_data_smithi171_78918_object143814:head# 2024-04-20T17:11:21.316 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.316 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:777c10d7:::benchmark_data_smithi171_78918_object96447:head# 2024-04-20T17:11:21.316 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.316 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:777d5881:::benchmark_data_smithi171_78918_object54468:head# 2024-04-20T17:11:21.316 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.316 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:777e6d1a:::benchmark_data_smithi171_78918_object66687:head# 2024-04-20T17:11:21.316 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.316 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:777e7e52:::benchmark_data_smithi171_78918_object140155:head# 2024-04-20T17:11:21.316 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.316 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:777eb26b:::benchmark_data_smithi171_78918_object81649:head# 2024-04-20T17:11:21.316 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.317 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:777f0d9b:::benchmark_data_smithi171_78918_object146033:head# 2024-04-20T17:11:21.317 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.317 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:777f5ea5:::benchmark_data_smithi171_78918_object128294:head# 2024-04-20T17:11:21.317 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.317 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:777fa098:::benchmark_data_smithi171_78918_object76361:head# 2024-04-20T17:11:21.317 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.317 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:777fac05:::benchmark_data_smithi171_78918_object151168:head# 2024-04-20T17:11:21.317 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.317 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:777fbae3:::benchmark_data_smithi171_78918_object61767:head# 2024-04-20T17:11:21.317 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.317 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77800347:::benchmark_data_smithi171_78918_object112498:head# 2024-04-20T17:11:21.317 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.317 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7780b17a:::benchmark_data_smithi171_78918_object160287:head# 2024-04-20T17:11:21.317 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.317 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7780cad0:::benchmark_data_smithi171_78918_object106174:head# 2024-04-20T17:11:21.318 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.318 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77815210:::benchmark_data_smithi171_78918_object87180:head# 2024-04-20T17:11:21.318 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.318 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77827b31:::benchmark_data_smithi171_78918_object69724:head# 2024-04-20T17:11:21.318 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.318 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7782e5cf:::benchmark_data_smithi171_78918_object136863:head# 2024-04-20T17:11:21.318 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.318 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7783010a:::benchmark_data_smithi171_78918_object108899:head# 2024-04-20T17:11:21.318 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.318 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77834a3d:::benchmark_data_smithi171_78918_object110782:head# 2024-04-20T17:11:21.318 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.318 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7784161d:::benchmark_data_smithi171_78918_object93379:head# 2024-04-20T17:11:21.318 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.318 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77842269:::benchmark_data_smithi171_78918_object147304:head# 2024-04-20T17:11:21.319 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.319 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77852c7d:::benchmark_data_smithi171_78918_object136297:head# 2024-04-20T17:11:21.319 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.319 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77867398:::benchmark_data_smithi171_78918_object123910:head# 2024-04-20T17:11:21.319 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.319 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77870921:::benchmark_data_smithi171_78918_object77671:head# 2024-04-20T17:11:21.319 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.319 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77886708:::benchmark_data_smithi171_78918_object148914:head# 2024-04-20T17:11:21.319 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.319 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7789177a:::benchmark_data_smithi171_78918_object133147:head# 2024-04-20T17:11:21.319 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.319 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77895615:::benchmark_data_smithi171_78918_object74005:head# 2024-04-20T17:11:21.319 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.319 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7789c4a6:::benchmark_data_smithi171_78918_object120109:head# 2024-04-20T17:11:21.319 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.320 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7789fd18:::benchmark_data_smithi171_78918_object153316:head# 2024-04-20T17:11:21.320 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.320 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:778afbe1:::benchmark_data_smithi171_78918_object56383:head# 2024-04-20T17:11:21.320 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.320 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:778b9b38:::benchmark_data_smithi171_78918_object117379:head# 2024-04-20T17:11:21.320 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.320 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:778bf57d:::benchmark_data_smithi171_78918_object71069:head# 2024-04-20T17:11:21.320 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.320 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:778bfbbf:::benchmark_data_smithi171_78918_object88123:head# 2024-04-20T17:11:21.320 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.320 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:778c2905:::benchmark_data_smithi171_78918_object147813:head# 2024-04-20T17:11:21.320 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.320 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:778d2fcc:::benchmark_data_smithi171_78918_object57773:head# 2024-04-20T17:11:21.320 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.320 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:778db06b:::benchmark_data_smithi171_78918_object144021:head# 2024-04-20T17:11:21.321 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.321 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:778ddb38:::benchmark_data_smithi171_78918_object66512:head# 2024-04-20T17:11:21.321 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.321 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:778de13f:::benchmark_data_smithi171_78918_object102925:head# 2024-04-20T17:11:21.321 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.321 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:778df32e:::benchmark_data_smithi171_78918_object101258:head# 2024-04-20T17:11:21.321 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.321 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:778df766:::benchmark_data_smithi171_78918_object106527:head# 2024-04-20T17:11:21.321 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.321 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:778e15d6:::benchmark_data_smithi171_78918_object88044:head# 2024-04-20T17:11:21.321 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.321 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7790162f:::benchmark_data_smithi171_78918_object90224:head# 2024-04-20T17:11:21.321 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.321 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77906897:::benchmark_data_smithi171_78918_object140860:head# 2024-04-20T17:11:21.322 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.322 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77908fa6:::benchmark_data_smithi171_78918_object128687:head# 2024-04-20T17:11:21.322 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.322 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7790e7ed:::benchmark_data_smithi171_78918_object131211:head# 2024-04-20T17:11:21.322 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.322 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7790fa28:::benchmark_data_smithi171_78918_object65369:head# 2024-04-20T17:11:21.322 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.322 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77930921:::benchmark_data_smithi171_78918_object128012:head# 2024-04-20T17:11:21.322 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.322 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77935254:::benchmark_data_smithi171_78918_object121494:head# 2024-04-20T17:11:21.322 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.322 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77935521:::benchmark_data_smithi171_78918_object109801:head# 2024-04-20T17:11:21.322 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.322 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77938671:::benchmark_data_smithi171_78918_object156262:head# 2024-04-20T17:11:21.323 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.323 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7793974f:::benchmark_data_smithi171_78918_object85997:head# 2024-04-20T17:11:21.323 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.323 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7793e261:::benchmark_data_smithi171_78918_object115225:head# 2024-04-20T17:11:21.323 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.323 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77944ae2:::benchmark_data_smithi171_78918_object96955:head# 2024-04-20T17:11:21.323 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.323 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77948c6c:::benchmark_data_smithi171_78918_object70785:head# 2024-04-20T17:11:21.323 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.323 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7794ed35:::benchmark_data_smithi171_78918_object153284:head# 2024-04-20T17:11:21.323 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.323 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7795351b:::benchmark_data_smithi171_78918_object83711:head# 2024-04-20T17:11:21.323 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.323 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77975ca9:::benchmark_data_smithi171_78918_object61227:head# 2024-04-20T17:11:21.323 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.323 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77992364:::benchmark_data_smithi171_78918_object65993:head# 2024-04-20T17:11:21.324 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.324 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77998cb5:::benchmark_data_smithi171_78918_object128401:head# 2024-04-20T17:11:21.324 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.324 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:779a05a1:::benchmark_data_smithi171_78918_object137667:head# 2024-04-20T17:11:21.324 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.324 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:779a2887:::benchmark_data_smithi171_78918_object105886:head# 2024-04-20T17:11:21.324 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.324 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:779abf70:::benchmark_data_smithi171_78918_object116761:head# 2024-04-20T17:11:21.324 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.324 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:779b6cc0:::benchmark_data_smithi171_78918_object109822:head# 2024-04-20T17:11:21.324 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.324 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:779b89b6:::benchmark_data_smithi171_78918_object116850:head# 2024-04-20T17:11:21.324 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.324 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:779c8653:::benchmark_data_smithi171_78918_object63854:head# 2024-04-20T17:11:21.324 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.324 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:779ddf50:::benchmark_data_smithi171_78918_object73683:head# 2024-04-20T17:11:21.325 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.325 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:779e3a12:::benchmark_data_smithi171_78918_object104410:head# 2024-04-20T17:11:21.325 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.325 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:779e6ef9:::benchmark_data_smithi171_78918_object155296:head# 2024-04-20T17:11:21.325 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.325 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:779e8009:::benchmark_data_smithi171_78918_object133217:head# 2024-04-20T17:11:21.325 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.325 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:779ec94a:::benchmark_data_smithi171_78918_object84305:head# 2024-04-20T17:11:21.325 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.325 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:779f8ad4:::benchmark_data_smithi171_78918_object134695:head# 2024-04-20T17:11:21.325 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.325 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77a34cf5:::benchmark_data_smithi171_78918_object99212:head# 2024-04-20T17:11:21.325 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.325 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77a34d78:::benchmark_data_smithi171_78918_object83683:head# 2024-04-20T17:11:21.326 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.326 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77a3b94e:::benchmark_data_smithi171_78918_object158895:head# 2024-04-20T17:11:21.326 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.326 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77a3cdca:::benchmark_data_smithi171_78918_object70088:head# 2024-04-20T17:11:21.326 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.326 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77a420f6:::benchmark_data_smithi171_78918_object148573:head# 2024-04-20T17:11:21.326 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.326 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77a448e6:::benchmark_data_smithi171_78918_object76783:head# 2024-04-20T17:11:21.326 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.326 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77a452a0:::benchmark_data_smithi171_78918_object149026:head# 2024-04-20T17:11:21.326 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.326 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77a4b85d:::benchmark_data_smithi171_78918_object64588:head# 2024-04-20T17:11:21.326 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.326 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77a6f095:::benchmark_data_smithi171_78918_object66424:head# 2024-04-20T17:11:21.327 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.327 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77a73127:::benchmark_data_smithi171_78918_object64075:head# 2024-04-20T17:11:21.327 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.327 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77a7f0b1:::benchmark_data_smithi171_78918_object86758:head# 2024-04-20T17:11:21.327 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.327 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77a843bb:::benchmark_data_smithi171_78918_object73500:head# 2024-04-20T17:11:21.327 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.327 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77aa0dc1:::benchmark_data_smithi171_78918_object123669:head# 2024-04-20T17:11:21.327 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.327 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77aa4e65:::benchmark_data_smithi171_78918_object158340:head# 2024-04-20T17:11:21.327 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.327 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77aa76bc:::benchmark_data_smithi171_78918_object64276:head# 2024-04-20T17:11:21.327 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.327 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77aab0d8:::benchmark_data_smithi171_78918_object160632:head# 2024-04-20T17:11:21.327 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.327 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77ab0806:::benchmark_data_smithi171_78918_object95795:head# 2024-04-20T17:11:21.327 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.328 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77ab4f3c:::benchmark_data_smithi171_78918_object97894:head# 2024-04-20T17:11:21.328 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.328 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77acc159:::benchmark_data_smithi171_78918_object128124:head# 2024-04-20T17:11:21.328 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.328 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77ae3a04:::benchmark_data_smithi171_78918_object155694:head# 2024-04-20T17:11:21.328 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.328 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77ae3d8e:::benchmark_data_smithi171_78918_object137857:head# 2024-04-20T17:11:21.328 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.328 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77ae8054:::benchmark_data_smithi171_78918_object146499:head# 2024-04-20T17:11:21.328 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.328 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77ae8605:::benchmark_data_smithi171_78918_object66372:head# 2024-04-20T17:11:21.328 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.328 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77af9662:::benchmark_data_smithi171_78918_object81787:head# 2024-04-20T17:11:21.328 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.328 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77b12e63:::benchmark_data_smithi171_78918_object57237:head# 2024-04-20T17:11:21.329 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.329 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77b27aa6:::benchmark_data_smithi171_78918_object153131:head# 2024-04-20T17:11:21.329 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.329 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77b3a75b:::benchmark_data_smithi171_78918_object99135:head# 2024-04-20T17:11:21.329 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.329 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77b46412:::benchmark_data_smithi171_78918_object73405:head# 2024-04-20T17:11:21.329 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.329 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77b4e97e:::benchmark_data_smithi171_78918_object80430:head# 2024-04-20T17:11:21.329 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.329 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77b4f255:::benchmark_data_smithi171_78918_object59979:head# 2024-04-20T17:11:21.329 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.329 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77b5466d:::benchmark_data_smithi171_78918_object76168:head# 2024-04-20T17:11:21.329 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.329 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77b613b6:::benchmark_data_smithi171_78918_object130476:head# 2024-04-20T17:11:21.329 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.330 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77b6a228:::benchmark_data_smithi171_78918_object105505:head# 2024-04-20T17:11:21.330 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.330 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77b7c433:::benchmark_data_smithi171_78918_object54913:head# 2024-04-20T17:11:21.330 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.330 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77b7d816:::benchmark_data_smithi171_78918_object99279:head# 2024-04-20T17:11:21.330 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.330 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77b8264b:::benchmark_data_smithi171_78918_object113581:head# 2024-04-20T17:11:21.330 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.330 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77b831a0:::benchmark_data_smithi171_78918_object73566:head# 2024-04-20T17:11:21.330 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.330 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77b8bf57:::benchmark_data_smithi171_78918_object64621:head# 2024-04-20T17:11:21.330 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.330 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77b8e48a:::benchmark_data_smithi171_78918_object107242:head# 2024-04-20T17:11:21.330 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.331 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77b96293:::benchmark_data_smithi171_78918_object93675:head# 2024-04-20T17:11:21.331 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.331 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77ba00fb:::benchmark_data_smithi171_78918_object130747:head# 2024-04-20T17:11:21.331 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.331 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77ba06c5:::benchmark_data_smithi171_78918_object95555:head# 2024-04-20T17:11:21.331 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.331 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77ba5e4b:::benchmark_data_smithi171_78918_object135308:head# 2024-04-20T17:11:21.331 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.331 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77ba797c:::benchmark_data_smithi171_78918_object107553:head# 2024-04-20T17:11:21.331 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.331 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77bb6706:::benchmark_data_smithi171_78918_object137837:head# 2024-04-20T17:11:21.331 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.331 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77bb9dcc:::benchmark_data_smithi171_78918_object102291:head# 2024-04-20T17:11:21.331 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.331 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77bc6839:::benchmark_data_smithi171_78918_object64874:head# 2024-04-20T17:11:21.332 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.332 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77bcca1f:::benchmark_data_smithi171_78918_object97335:head# 2024-04-20T17:11:21.332 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.332 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77bd6137:::benchmark_data_smithi171_78918_object64021:head# 2024-04-20T17:11:21.332 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.332 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77bd6569:::benchmark_data_smithi171_78918_object146085:head# 2024-04-20T17:11:21.332 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.332 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77becea1:::benchmark_data_smithi171_78918_object91404:head# 2024-04-20T17:11:21.332 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.332 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77bf7888:::benchmark_data_smithi171_78918_object135361:head# 2024-04-20T17:11:21.332 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.332 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77bff712:::benchmark_data_smithi171_78918_object99378:head# 2024-04-20T17:11:21.332 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.332 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77c000ce:::benchmark_data_smithi171_78918_object105957:head# 2024-04-20T17:11:21.332 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.332 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77c160e2:::benchmark_data_smithi171_78918_object143361:head# 2024-04-20T17:11:21.333 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.333 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77c1a6f7:::benchmark_data_smithi171_78918_object118878:head# 2024-04-20T17:11:21.333 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.333 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77c1e9fe:::benchmark_data_smithi171_78918_object93390:head# 2024-04-20T17:11:21.333 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.333 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77c2e733:::benchmark_data_smithi171_78918_object77460:head# 2024-04-20T17:11:21.333 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.333 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77c31b1e:::benchmark_data_smithi171_78918_object132679:head# 2024-04-20T17:11:21.333 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.333 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77c3984b:::benchmark_data_smithi171_78918_object84518:head# 2024-04-20T17:11:21.333 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.333 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77c4473f:::benchmark_data_smithi171_78918_object82105:head# 2024-04-20T17:11:21.333 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.333 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77c45aa7:::benchmark_data_smithi171_78918_object124103:head# 2024-04-20T17:11:21.334 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.334 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77c4a4ea:::benchmark_data_smithi171_78918_object55469:head# 2024-04-20T17:11:21.334 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.334 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77c67c22:::benchmark_data_smithi171_78918_object71575:head# 2024-04-20T17:11:21.334 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.334 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77c79f7b:::benchmark_data_smithi171_78918_object55104:head# 2024-04-20T17:11:21.334 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.334 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77c906ff:::benchmark_data_smithi171_78918_object121555:head# 2024-04-20T17:11:21.334 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.334 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77c90732:::benchmark_data_smithi171_78918_object126595:head# 2024-04-20T17:11:21.334 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.334 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77c973eb:::benchmark_data_smithi171_78918_object160451:head# 2024-04-20T17:11:21.334 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.334 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77c9762b:::benchmark_data_smithi171_78918_object160773:head# 2024-04-20T17:11:21.335 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.335 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77c977ed:::benchmark_data_smithi171_78918_object151107:head# 2024-04-20T17:11:21.335 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.335 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77ca9ddb:::benchmark_data_smithi171_78918_object129090:head# 2024-04-20T17:11:21.335 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.335 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77cadf5f:::benchmark_data_smithi171_78918_object109363:head# 2024-04-20T17:11:21.335 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.335 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77ccc89d:::benchmark_data_smithi171_78918_object143396:head# 2024-04-20T17:11:21.335 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.335 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77cce6d3:::benchmark_data_smithi171_78918_object138403:head# 2024-04-20T17:11:21.335 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.335 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77cd66e8:::benchmark_data_smithi171_78918_object99352:head# 2024-04-20T17:11:21.335 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.335 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77ce3365:::benchmark_data_smithi171_78918_object116528:head# 2024-04-20T17:11:21.335 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.335 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77ce88a4:::benchmark_data_smithi171_78918_object106463:head# 2024-04-20T17:11:21.336 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.336 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77d03889:::benchmark_data_smithi171_78918_object91927:head# 2024-04-20T17:11:21.336 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.336 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77d09b37:::benchmark_data_smithi171_78918_object73084:head# 2024-04-20T17:11:21.336 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.336 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77d148e4:::benchmark_data_smithi171_78918_object102331:head# 2024-04-20T17:11:21.336 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.336 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77d21efb:::benchmark_data_smithi171_78918_object75774:head# 2024-04-20T17:11:21.336 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.336 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77d294cd:::benchmark_data_smithi171_78918_object152278:head# 2024-04-20T17:11:21.336 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.336 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77d3d5bc:::benchmark_data_smithi171_78918_object157387:head# 2024-04-20T17:11:21.336 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.336 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77d42bf5:::benchmark_data_smithi171_78918_object121235:head# 2024-04-20T17:11:21.336 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.337 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77d4c515:::benchmark_data_smithi171_78918_object156844:head# 2024-04-20T17:11:21.337 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.337 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77d5c8aa:::benchmark_data_smithi171_78918_object103608:head# 2024-04-20T17:11:21.337 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.337 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77d5dc8d:::benchmark_data_smithi171_78918_object93912:head# 2024-04-20T17:11:21.337 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.337 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77d5dcee:::benchmark_data_smithi171_78918_object87386:head# 2024-04-20T17:11:21.337 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.337 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77d6263e:::benchmark_data_smithi171_78918_object68332:head# 2024-04-20T17:11:21.337 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.337 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77d64f95:::benchmark_data_smithi171_78918_object127094:head# 2024-04-20T17:11:21.337 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.337 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77d653ba:::benchmark_data_smithi171_78918_object141848:head# 2024-04-20T17:11:21.337 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.337 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77d6fb38:::benchmark_data_smithi171_78918_object157004:head# 2024-04-20T17:11:21.338 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.338 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77d749e6:::benchmark_data_smithi171_78918_object85719:head# 2024-04-20T17:11:21.338 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.338 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77d91aa5:::benchmark_data_smithi171_78918_object160329:head# 2024-04-20T17:11:21.338 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.338 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77d9435f:::benchmark_data_smithi171_78918_object148245:head# 2024-04-20T17:11:21.338 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.338 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77d96e00:::benchmark_data_smithi171_78918_object139095:head# 2024-04-20T17:11:21.338 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.338 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77dbe43a:::benchmark_data_smithi171_78918_object59670:head# 2024-04-20T17:11:21.338 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.338 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77dcca11:::benchmark_data_smithi171_78918_object93314:head# 2024-04-20T17:11:21.338 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.338 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77dd302e:::benchmark_data_smithi171_78918_object54590:head# 2024-04-20T17:11:21.338 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.339 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77dda201:::benchmark_data_smithi171_78918_object86882:head# 2024-04-20T17:11:21.339 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.339 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77df7fe5:::benchmark_data_smithi171_78918_object103270:head# 2024-04-20T17:11:21.339 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.339 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77df9bf1:::benchmark_data_smithi171_78918_object71351:head# 2024-04-20T17:11:21.339 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.339 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77e0e61b:::benchmark_data_smithi171_78918_object109314:head# 2024-04-20T17:11:21.339 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.339 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77e0eba1:::benchmark_data_smithi171_78918_object96893:head# 2024-04-20T17:11:21.339 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.339 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77e2b7bf:::benchmark_data_smithi171_78918_object104239:head# 2024-04-20T17:11:21.339 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.339 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77e312f1:::benchmark_data_smithi171_78918_object109726:head# 2024-04-20T17:11:21.339 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.339 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77e4d255:::benchmark_data_smithi171_78918_object132467:head# 2024-04-20T17:11:21.339 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.340 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77e4d8da:::benchmark_data_smithi171_78918_object85089:head# 2024-04-20T17:11:21.340 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.340 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77e5acc2:::benchmark_data_smithi171_78918_object153062:head# 2024-04-20T17:11:21.340 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.340 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77e5ed24:::benchmark_data_smithi171_78918_object120160:head# 2024-04-20T17:11:21.340 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.340 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77e5f2be:::benchmark_data_smithi171_78918_object84780:head# 2024-04-20T17:11:21.340 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.340 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77e6641e:::benchmark_data_smithi171_78918_object68403:head# 2024-04-20T17:11:21.340 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.340 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77e6b5f0:::benchmark_data_smithi171_78918_object103997:head# 2024-04-20T17:11:21.340 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.340 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77e6e310:::benchmark_data_smithi171_78918_object100656:head# 2024-04-20T17:11:21.340 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.341 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77e71f4d:::benchmark_data_smithi171_78918_object141302:head# 2024-04-20T17:11:21.341 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.341 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77e75503:::benchmark_data_smithi171_78918_object80221:head# 2024-04-20T17:11:21.341 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.341 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77e77071:::benchmark_data_smithi171_78918_object55650:head# 2024-04-20T17:11:21.341 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.341 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77e862f3:::benchmark_data_smithi171_78918_object142172:head# 2024-04-20T17:11:21.341 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.341 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77e873ca:::benchmark_data_smithi171_78918_object140203:head# 2024-04-20T17:11:21.341 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.341 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77e88e23:::benchmark_data_smithi171_78918_object113334:head# 2024-04-20T17:11:21.341 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.341 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77ed2975:::benchmark_data_smithi171_78918_object75959:head# 2024-04-20T17:11:21.341 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.342 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77ed3649:::benchmark_data_smithi171_78918_object151610:head# 2024-04-20T17:11:21.342 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.342 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77ed389b:::benchmark_data_smithi171_78918_object88297:head# 2024-04-20T17:11:21.342 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.342 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77ed81e7:::benchmark_data_smithi171_78918_object82984:head# 2024-04-20T17:11:21.342 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.342 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77eda153:::benchmark_data_smithi171_78918_object132728:head# 2024-04-20T17:11:21.342 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.342 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77ee4aa0:::benchmark_data_smithi171_78918_object114694:head# 2024-04-20T17:11:21.342 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.342 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77ee6c98:::benchmark_data_smithi171_78918_object143190:head# 2024-04-20T17:11:21.342 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.342 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77ee86f9:::benchmark_data_smithi171_78918_object108652:head# 2024-04-20T17:11:21.342 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.343 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77eea9c5:::benchmark_data_smithi171_78918_object53621:head# 2024-04-20T17:11:21.343 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.343 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77eebb9a:::benchmark_data_smithi171_78918_object52579:head# 2024-04-20T17:11:21.343 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.343 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77f20ff5:::benchmark_data_smithi171_78918_object95862:head# 2024-04-20T17:11:21.343 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.343 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77f2287b:::benchmark_data_smithi171_78918_object133577:head# 2024-04-20T17:11:21.343 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.343 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77f32adf:::benchmark_data_smithi171_78918_object65838:head# 2024-04-20T17:11:21.343 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.343 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77f3429c:::benchmark_data_smithi171_78918_object91920:head# 2024-04-20T17:11:21.343 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.343 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77f35f9e:::benchmark_data_smithi171_78918_object139048:head# 2024-04-20T17:11:21.343 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.343 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77f41e55:::benchmark_data_smithi171_78918_object74828:head# 2024-04-20T17:11:21.343 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.344 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77f69da1:::benchmark_data_smithi171_78918_object83306:head# 2024-04-20T17:11:21.344 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.344 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77f6bf25:::benchmark_data_smithi171_78918_object114872:head# 2024-04-20T17:11:21.344 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.344 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77f7061f:::benchmark_data_smithi171_78918_object132759:head# 2024-04-20T17:11:21.344 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.344 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77f77a3b:::benchmark_data_smithi171_78918_object121781:head# 2024-04-20T17:11:21.344 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.344 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77f80d1e:::benchmark_data_smithi171_78918_object140856:head# 2024-04-20T17:11:21.344 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.344 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77f856fe:::benchmark_data_smithi171_78918_object73317:head# 2024-04-20T17:11:21.344 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.344 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77f8a7dd:::benchmark_data_smithi171_78918_object157157:head# 2024-04-20T17:11:21.344 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.344 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77f97c55:::benchmark_data_smithi171_78918_object116419:head# 2024-04-20T17:11:21.345 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.345 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77f988ee:::benchmark_data_smithi171_78918_object108633:head# 2024-04-20T17:11:21.345 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.345 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77fa8cf1:::benchmark_data_smithi171_78918_object81124:head# 2024-04-20T17:11:21.345 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.345 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77fb2a1e:::benchmark_data_smithi171_78918_object113715:head# 2024-04-20T17:11:21.345 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.345 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77fbbec8:::benchmark_data_smithi171_78918_object79555:head# 2024-04-20T17:11:21.345 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.345 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77fc643b:::benchmark_data_smithi171_78918_object78936:head# 2024-04-20T17:11:21.345 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.345 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77fd5d86:::benchmark_data_smithi171_78918_object97434:head# 2024-04-20T17:11:21.345 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.345 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77fdc10b:::benchmark_data_smithi171_78918_object90093:head# 2024-04-20T17:11:21.345 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.345 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77fdc671:::benchmark_data_smithi171_78918_object134142:head# 2024-04-20T17:11:21.346 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.346 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77fef3e6:::benchmark_data_smithi171_78918_object101018:head# 2024-04-20T17:11:21.346 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.346 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77ff1ae0:::benchmark_data_smithi171_78918_object155197:head# 2024-04-20T17:11:21.346 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.346 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77ff5a44:::benchmark_data_smithi171_78918_object159294:head# 2024-04-20T17:11:21.346 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.346 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:77ff964f:::benchmark_data_smithi171_78918_object62839:head# 2024-04-20T17:11:21.346 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.346 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78005178:::benchmark_data_smithi171_78918_object100438:head# 2024-04-20T17:11:21.346 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.346 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7801a9e4:::benchmark_data_smithi171_78918_object145889:head# 2024-04-20T17:11:21.346 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.346 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78022274:::benchmark_data_smithi171_78918_object132191:head# 2024-04-20T17:11:21.347 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.347 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78026648:::benchmark_data_smithi171_78918_object79546:head# 2024-04-20T17:11:21.347 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.347 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7802fd34:::benchmark_data_smithi171_78918_object117879:head# 2024-04-20T17:11:21.347 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.347 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7804677d:::benchmark_data_smithi171_78918_object127309:head# 2024-04-20T17:11:21.347 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.347 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7804730a:::benchmark_data_smithi171_78918_object109668:head# 2024-04-20T17:11:21.347 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.347 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78047412:::benchmark_data_smithi171_78918_object54068:head# 2024-04-20T17:11:21.347 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.347 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78058d17:::benchmark_data_smithi171_78918_object144576:head# 2024-04-20T17:11:21.347 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.347 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7805d055:::benchmark_data_smithi171_78918_object52626:head# 2024-04-20T17:11:21.347 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.347 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7806b281:::benchmark_data_smithi171_78918_object61823:head# 2024-04-20T17:11:21.348 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.348 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7806ee99:::benchmark_data_smithi171_78918_object77667:head# 2024-04-20T17:11:21.348 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.348 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78072cdf:::benchmark_data_smithi171_78918_object141717:head# 2024-04-20T17:11:21.348 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.348 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7808056a:::benchmark_data_smithi171_78918_object86761:head# 2024-04-20T17:11:21.348 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.348 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78084f5c:::benchmark_data_smithi171_78918_object58729:head# 2024-04-20T17:11:21.348 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.348 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78091f4a:::benchmark_data_smithi171_78918_object149457:head# 2024-04-20T17:11:21.348 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.348 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:780a9d0b:::benchmark_data_smithi171_78918_object63583:head# 2024-04-20T17:11:21.348 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.348 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:780af866:::benchmark_data_smithi171_78918_object152873:head# 2024-04-20T17:11:21.348 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.349 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:780b1e3a:::benchmark_data_smithi171_78918_object156075:head# 2024-04-20T17:11:21.349 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.349 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:780cbab9:::benchmark_data_smithi171_78918_object143474:head# 2024-04-20T17:11:21.349 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.349 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:780da192:::benchmark_data_smithi171_78918_object55336:head# 2024-04-20T17:11:21.349 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.349 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:780f499c:::benchmark_data_smithi171_78918_object88582:head# 2024-04-20T17:11:21.349 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.349 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78103c83:::benchmark_data_smithi171_78918_object120546:head# 2024-04-20T17:11:21.349 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.349 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78104e7d:::benchmark_data_smithi171_78918_object132061:head# 2024-04-20T17:11:21.349 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.349 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7810b494:::benchmark_data_smithi171_78918_object77216:head# 2024-04-20T17:11:21.349 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.349 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78111ac9:::benchmark_data_smithi171_78918_object92852:head# 2024-04-20T17:11:21.349 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.350 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7811585c:::benchmark_data_smithi171_78918_object141288:head# 2024-04-20T17:11:21.350 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.350 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:781480e2:::benchmark_data_smithi171_78918_object67660:head# 2024-04-20T17:11:21.350 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.350 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7814cd43:::benchmark_data_smithi171_78918_object70960:head# 2024-04-20T17:11:21.350 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.350 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:781592c0:::benchmark_data_smithi171_78918_object110988:head# 2024-04-20T17:11:21.350 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.350 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:781641c6:::benchmark_data_smithi171_78918_object109086:head# 2024-04-20T17:11:21.350 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.350 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78174b40:::benchmark_data_smithi171_78918_object104876:head# 2024-04-20T17:11:21.350 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.350 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78182e88:::benchmark_data_smithi171_78918_object121799:head# 2024-04-20T17:11:21.350 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.351 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78185f3c:::benchmark_data_smithi171_78918_object154368:head# 2024-04-20T17:11:21.351 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.351 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7818adb2:::benchmark_data_smithi171_78918_object61587:head# 2024-04-20T17:11:21.351 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.351 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7819af95:::benchmark_data_smithi171_78918_object124871:head# 2024-04-20T17:11:21.351 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.351 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:781a73c6:::benchmark_data_smithi171_78918_object89946:head# 2024-04-20T17:11:21.351 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.351 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:781ad95b:::benchmark_data_smithi171_78918_object63807:head# 2024-04-20T17:11:21.351 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.351 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:781b12cf:::benchmark_data_smithi171_78918_object105447:head# 2024-04-20T17:11:21.351 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.351 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:781b9c4f:::benchmark_data_smithi171_78918_object67418:head# 2024-04-20T17:11:21.351 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.352 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:781bab05:::benchmark_data_smithi171_78918_object120063:head# 2024-04-20T17:11:21.352 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.352 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:781bb0e1:::benchmark_data_smithi171_78918_object131013:head# 2024-04-20T17:11:21.352 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.352 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:781c34b0:::benchmark_data_smithi171_78918_object120665:head# 2024-04-20T17:11:21.352 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.352 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:781cc934:::benchmark_data_smithi171_78918_object154418:head# 2024-04-20T17:11:21.352 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.352 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:781cdc1b:::benchmark_data_smithi171_78918_object159949:head# 2024-04-20T17:11:21.352 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.352 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:781cfc1d:::benchmark_data_smithi171_78918_object120981:head# 2024-04-20T17:11:21.352 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.352 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:781d9ab2:::benchmark_data_smithi171_78918_object116475:head# 2024-04-20T17:11:21.352 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.352 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:781dae61:::benchmark_data_smithi171_78918_object128194:head# 2024-04-20T17:11:21.353 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.353 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:781e60e7:::benchmark_data_smithi171_78918_object109806:head# 2024-04-20T17:11:21.353 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.353 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:781ecca6:::benchmark_data_smithi171_78918_object149885:head# 2024-04-20T17:11:21.353 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.353 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:781ffe87:::benchmark_data_smithi171_78918_object147857:head# 2024-04-20T17:11:21.353 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.353 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78206256:::benchmark_data_smithi171_78918_object126627:head# 2024-04-20T17:11:21.353 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.353 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7820abcc:::benchmark_data_smithi171_78918_object160854:head# 2024-04-20T17:11:21.353 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.353 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7821cf64:::benchmark_data_smithi171_78918_object150874:head# 2024-04-20T17:11:21.353 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.353 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7821d91f:::benchmark_data_smithi171_78918_object153101:head# 2024-04-20T17:11:21.354 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.354 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7823f7b7:::benchmark_data_smithi171_78918_object60430:head# 2024-04-20T17:11:21.354 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.354 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78244162:::benchmark_data_smithi171_78918_object153099:head# 2024-04-20T17:11:21.354 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.354 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78250fe8:::benchmark_data_smithi171_78918_object101605:head# 2024-04-20T17:11:21.354 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.354 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78262c16:::benchmark_data_smithi171_78918_object74480:head# 2024-04-20T17:11:21.354 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.354 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:782789b1:::benchmark_data_smithi171_78918_object82454:head# 2024-04-20T17:11:21.354 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.354 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78278a46:::benchmark_data_smithi171_78918_object123768:head# 2024-04-20T17:11:21.354 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.354 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7827df7c:::benchmark_data_smithi171_78918_object84479:head# 2024-04-20T17:11:21.355 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.355 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:782812f6:::benchmark_data_smithi171_78918_object79186:head# 2024-04-20T17:11:21.355 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.355 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7829830d:::benchmark_data_smithi171_78918_object126497:head# 2024-04-20T17:11:21.355 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.355 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78299511:::benchmark_data_smithi171_78918_object76826:head# 2024-04-20T17:11:21.355 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.355 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7829eded:::benchmark_data_smithi171_78918_object102506:head# 2024-04-20T17:11:21.355 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.355 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:782a1a05:::benchmark_data_smithi171_78918_object113421:head# 2024-04-20T17:11:21.355 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.355 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:782aa200:::benchmark_data_smithi171_78918_object70082:head# 2024-04-20T17:11:21.355 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.355 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:782c1fd9:::benchmark_data_smithi171_78918_object92142:head# 2024-04-20T17:11:21.356 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.356 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:782d7fff:::benchmark_data_smithi171_78918_object107061:head# 2024-04-20T17:11:21.356 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.356 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:782e28d2:::benchmark_data_smithi171_78918_object51501:head# 2024-04-20T17:11:21.356 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.356 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:782e586f:::benchmark_data_smithi171_78918_object104591:head# 2024-04-20T17:11:21.356 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.356 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:782e9bd3:::benchmark_data_smithi171_78918_object65714:head# 2024-04-20T17:11:21.356 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.356 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:782ee108:::benchmark_data_smithi171_78918_object89974:head# 2024-04-20T17:11:21.356 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.356 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:782f249b:::benchmark_data_smithi171_78918_object81402:head# 2024-04-20T17:11:21.356 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.356 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:782ff18d:::benchmark_data_smithi171_78918_object140869:head# 2024-04-20T17:11:21.357 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.357 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78306fdf:::benchmark_data_smithi171_78918_object112175:head# 2024-04-20T17:11:21.357 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.357 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78309ae2:::benchmark_data_smithi171_78918_object141024:head# 2024-04-20T17:11:21.357 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.357 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7830d7c3:::benchmark_data_smithi171_78918_object101477:head# 2024-04-20T17:11:21.357 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.357 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:783164b5:::benchmark_data_smithi171_78918_object156663:head# 2024-04-20T17:11:21.357 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.357 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:783196ed:::benchmark_data_smithi171_78918_object136640:head# 2024-04-20T17:11:21.357 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.357 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78320652:::benchmark_data_smithi171_78918_object58223:head# 2024-04-20T17:11:21.357 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.357 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78320d81:::benchmark_data_smithi171_78918_object112722:head# 2024-04-20T17:11:21.358 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.358 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78326be7:::benchmark_data_smithi171_78918_object144518:head# 2024-04-20T17:11:21.358 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.358 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7832a25a:::benchmark_data_smithi171_78918_object141609:head# 2024-04-20T17:11:21.358 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.358 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7832d0e6:::benchmark_data_smithi171_78918_object106353:head# 2024-04-20T17:11:21.358 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.358 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7832e1e0:::benchmark_data_smithi171_78918_object113833:head# 2024-04-20T17:11:21.358 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.358 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78338b88:::benchmark_data_smithi171_78918_object57479:head# 2024-04-20T17:11:21.358 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.358 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78344ca8:::benchmark_data_smithi171_78918_object122937:head# 2024-04-20T17:11:21.358 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.358 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7835816b:::benchmark_data_smithi171_78918_object108634:head# 2024-04-20T17:11:21.359 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.359 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:783635b7:::benchmark_data_smithi171_78918_object148184:head# 2024-04-20T17:11:21.359 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.359 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7837ba08:::benchmark_data_smithi171_78918_object143157:head# 2024-04-20T17:11:21.359 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.359 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7838d0a7:::benchmark_data_smithi171_78918_object57604:head# 2024-04-20T17:11:21.359 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.359 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78393da7:::benchmark_data_smithi171_78918_object61319:head# 2024-04-20T17:11:21.359 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.359 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78395dbd:::benchmark_data_smithi171_78918_object111866:head# 2024-04-20T17:11:21.359 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.359 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7839b8a6:::benchmark_data_smithi171_78918_object134671:head# 2024-04-20T17:11:21.359 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.359 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:783a566b:::benchmark_data_smithi171_78918_object56168:head# 2024-04-20T17:11:21.359 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.360 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:783a8926:::benchmark_data_smithi171_78918_object138204:head# 2024-04-20T17:11:21.360 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.360 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:783b052b:::benchmark_data_smithi171_78918_object148578:head# 2024-04-20T17:11:21.360 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.360 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:783c3214:::benchmark_data_smithi171_78918_object92995:head# 2024-04-20T17:11:21.360 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.360 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:783c38a1:::benchmark_data_smithi171_78918_object137570:head# 2024-04-20T17:11:21.360 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.360 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:783cd5d9:::benchmark_data_smithi171_78918_object65557:head# 2024-04-20T17:11:21.360 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.360 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:783d6b79:::benchmark_data_smithi171_78918_object123517:head# 2024-04-20T17:11:21.360 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.360 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:783eef18:::benchmark_data_smithi171_78918_object73703:head# 2024-04-20T17:11:21.360 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.361 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7840dfa6:::benchmark_data_smithi171_78918_object116395:head# 2024-04-20T17:11:21.361 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.361 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7840fc3e:::benchmark_data_smithi171_78918_object61164:head# 2024-04-20T17:11:21.361 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.361 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78444d9f:::benchmark_data_smithi171_78918_object148539:head# 2024-04-20T17:11:21.361 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.361 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78478e32:::benchmark_data_smithi171_78918_object113868:head# 2024-04-20T17:11:21.361 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.361 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7847b6db:::benchmark_data_smithi171_78918_object129720:head# 2024-04-20T17:11:21.361 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.361 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:784889c0:::benchmark_data_smithi171_78918_object137631:head# 2024-04-20T17:11:21.361 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.361 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7848cf78:::benchmark_data_smithi171_78918_object124446:head# 2024-04-20T17:11:21.361 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.361 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7848f025:::benchmark_data_smithi171_78918_object90781:head# 2024-04-20T17:11:21.362 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.362 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:784949bc:::benchmark_data_smithi171_78918_object100457:head# 2024-04-20T17:11:21.362 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.362 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78498bf3:::benchmark_data_smithi171_78918_object94039:head# 2024-04-20T17:11:21.362 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.362 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7849d12f:::benchmark_data_smithi171_78918_object81424:head# 2024-04-20T17:11:21.362 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.362 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7849fbe5:::benchmark_data_smithi171_78918_object58169:head# 2024-04-20T17:11:21.362 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.362 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:784a2526:::benchmark_data_smithi171_78918_object127141:head# 2024-04-20T17:11:21.362 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.362 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:784b222f:::benchmark_data_smithi171_78918_object71493:head# 2024-04-20T17:11:21.362 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.362 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:784bc2d7:::benchmark_data_smithi171_78918_object141167:head# 2024-04-20T17:11:21.363 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.363 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:784c1464:::benchmark_data_smithi171_78918_object58041:head# 2024-04-20T17:11:21.363 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.363 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:784c8da2:::benchmark_data_smithi171_78918_object132033:head# 2024-04-20T17:11:21.363 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.363 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:784ca073:::benchmark_data_smithi171_78918_object77148:head# 2024-04-20T17:11:21.363 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.363 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:784f27bd:::benchmark_data_smithi171_78918_object118452:head# 2024-04-20T17:11:21.363 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.363 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:784f38e2:::benchmark_data_smithi171_78918_object148650:head# 2024-04-20T17:11:21.363 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.363 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:784ffdd4:::benchmark_data_smithi171_78918_object119438:head# 2024-04-20T17:11:21.363 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.363 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78503e99:::benchmark_data_smithi171_78918_object130164:head# 2024-04-20T17:11:21.364 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.364 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7851cbd7:::benchmark_data_smithi171_78918_object133927:head# 2024-04-20T17:11:21.364 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.364 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78522319:::benchmark_data_smithi171_78918_object81892:head# 2024-04-20T17:11:21.364 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.364 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78529951:::benchmark_data_smithi171_78918_object115601:head# 2024-04-20T17:11:21.364 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.364 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7852ba5d:::benchmark_data_smithi171_78918_object115033:head# 2024-04-20T17:11:21.364 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.364 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78533049:::benchmark_data_smithi171_78918_object106380:head# 2024-04-20T17:11:21.364 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.364 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78541932:::benchmark_data_smithi171_78918_object87812:head# 2024-04-20T17:11:21.364 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.364 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78545916:::benchmark_data_smithi171_78918_object105845:head# 2024-04-20T17:11:21.364 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.365 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78551312:::benchmark_data_smithi171_78918_object160306:head# 2024-04-20T17:11:21.365 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.365 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78553c3b:::benchmark_data_smithi171_78918_object151396:head# 2024-04-20T17:11:21.365 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.365 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7856724c:::benchmark_data_smithi171_78918_object96871:head# 2024-04-20T17:11:21.365 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.365 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:785832b0:::benchmark_data_smithi171_78918_object59448:head# 2024-04-20T17:11:21.365 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.365 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7858cc0a:::benchmark_data_smithi171_78918_object79079:head# 2024-04-20T17:11:21.365 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.365 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78590c79:::benchmark_data_smithi171_78918_object68288:head# 2024-04-20T17:11:21.365 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.365 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78593461:::benchmark_data_smithi171_78918_object85273:head# 2024-04-20T17:11:21.365 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.365 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78598c64:::benchmark_data_smithi171_78918_object121220:head# 2024-04-20T17:11:21.366 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.366 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78599e3e:::benchmark_data_smithi171_78918_object134587:head# 2024-04-20T17:11:21.366 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.366 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7859ecb2:::benchmark_data_smithi171_78918_object144247:head# 2024-04-20T17:11:21.366 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.366 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:785a17f0:::benchmark_data_smithi171_78918_object104317:head# 2024-04-20T17:11:21.366 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.366 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:785a63ee:::benchmark_data_smithi171_78918_object82377:head# 2024-04-20T17:11:21.366 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.366 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:785aedb3:::benchmark_data_smithi171_78918_object85568:head# 2024-04-20T17:11:21.366 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.366 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:785c40a7:::benchmark_data_smithi171_78918_object132125:head# 2024-04-20T17:11:21.366 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.366 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:785d9fd0:::benchmark_data_smithi171_78918_object156257:head# 2024-04-20T17:11:21.367 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.367 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:785e7d2a:::benchmark_data_smithi171_78918_object72427:head# 2024-04-20T17:11:21.367 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.367 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:785ea99b:::benchmark_data_smithi171_78918_object70320:head# 2024-04-20T17:11:21.367 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.367 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:785f420b:::benchmark_data_smithi171_78918_object108188:head# 2024-04-20T17:11:21.367 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.367 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:785fd5b1:::benchmark_data_smithi171_78918_object160190:head# 2024-04-20T17:11:21.367 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.367 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7860fbe9:::benchmark_data_smithi171_78918_object119895:head# 2024-04-20T17:11:21.367 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.367 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7861082d:::benchmark_data_smithi171_78918_object65038:head# 2024-04-20T17:11:21.367 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.367 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7861a007:::benchmark_data_smithi171_78918_object158747:head# 2024-04-20T17:11:21.368 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.368 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7861c564:::benchmark_data_smithi171_78918_object89578:head# 2024-04-20T17:11:21.368 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.368 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78636c7e:::benchmark_data_smithi171_78918_object136152:head# 2024-04-20T17:11:21.368 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.368 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7863863e:::benchmark_data_smithi171_78918_object102447:head# 2024-04-20T17:11:21.368 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.368 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7863bffc:::benchmark_data_smithi171_78918_object111544:head# 2024-04-20T17:11:21.368 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.368 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:786401b3:::benchmark_data_smithi171_78918_object130202:head# 2024-04-20T17:11:21.368 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.368 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78660cc3:::benchmark_data_smithi171_78918_object70132:head# 2024-04-20T17:11:21.368 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.368 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78664a83:::benchmark_data_smithi171_78918_object120668:head# 2024-04-20T17:11:21.369 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.369 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7867155e:::benchmark_data_smithi171_78918_object157882:head# 2024-04-20T17:11:21.369 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.369 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78674719:::benchmark_data_smithi171_78918_object72255:head# 2024-04-20T17:11:21.369 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.369 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7868222d:::benchmark_data_smithi171_78918_object102000:head# 2024-04-20T17:11:21.369 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.369 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78688d32:::benchmark_data_smithi171_78918_object93557:head# 2024-04-20T17:11:21.369 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.369 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7869199f:::benchmark_data_smithi171_78918_object145916:head# 2024-04-20T17:11:21.369 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.369 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78697a19:::benchmark_data_smithi171_78918_object117712:head# 2024-04-20T17:11:21.369 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.369 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78698795:::benchmark_data_smithi171_78918_object129686:head# 2024-04-20T17:11:21.369 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.370 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7869ace8:::benchmark_data_smithi171_78918_object72438:head# 2024-04-20T17:11:21.370 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.370 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:786ab3d3:::benchmark_data_smithi171_78918_object121695:head# 2024-04-20T17:11:21.370 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.370 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:786adb10:::benchmark_data_smithi171_78918_object74725:head# 2024-04-20T17:11:21.370 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.370 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:786af079:::benchmark_data_smithi171_78918_object110269:head# 2024-04-20T17:11:21.370 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.370 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:786b9ec7:::benchmark_data_smithi171_78918_object63777:head# 2024-04-20T17:11:21.370 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.370 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:786bec71:::benchmark_data_smithi171_78918_object128248:head# 2024-04-20T17:11:21.370 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.370 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:786ca068:::benchmark_data_smithi171_78918_object60359:head# 2024-04-20T17:11:21.370 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.371 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:786d1892:::benchmark_data_smithi171_78918_object84931:head# 2024-04-20T17:11:21.371 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.371 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:786e6a46:::benchmark_data_smithi171_78918_object88863:head# 2024-04-20T17:11:21.371 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.371 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:786e99eb:::benchmark_data_smithi171_78918_object94479:head# 2024-04-20T17:11:21.371 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.371 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:786f1da7:::benchmark_data_smithi171_78918_object106486:head# 2024-04-20T17:11:21.371 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.371 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:786f8b2b:::benchmark_data_smithi171_78918_object143172:head# 2024-04-20T17:11:21.371 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.371 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:786fb6f4:::benchmark_data_smithi171_78918_object116064:head# 2024-04-20T17:11:21.371 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.371 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:786fe480:::benchmark_data_smithi171_78918_object141089:head# 2024-04-20T17:11:21.371 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.372 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78700c16:::benchmark_data_smithi171_78918_object67647:head# 2024-04-20T17:11:21.372 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.372 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:787018a5:::benchmark_data_smithi171_78918_object156140:head# 2024-04-20T17:11:21.372 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.372 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78705d9e:::benchmark_data_smithi171_78918_object69035:head# 2024-04-20T17:11:21.372 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.372 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7870f2fb:::benchmark_data_smithi171_78918_object111235:head# 2024-04-20T17:11:21.372 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.372 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78716a1d:::benchmark_data_smithi171_78918_object132221:head# 2024-04-20T17:11:21.372 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.372 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7871ca3b:::benchmark_data_smithi171_78918_object112999:head# 2024-04-20T17:11:21.372 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.372 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7872c7ef:::benchmark_data_smithi171_78918_object138380:head# 2024-04-20T17:11:21.372 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.373 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:787300af:::benchmark_data_smithi171_78918_object145988:head# 2024-04-20T17:11:21.373 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.373 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:787310da:::benchmark_data_smithi171_78918_object101889:head# 2024-04-20T17:11:21.373 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.373 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7873160c:::benchmark_data_smithi171_78918_object72955:head# 2024-04-20T17:11:21.373 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.373 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78744faf:::benchmark_data_smithi171_78918_object96331:head# 2024-04-20T17:11:21.373 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.373 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:787481db:::benchmark_data_smithi171_78918_object110353:head# 2024-04-20T17:11:21.373 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.373 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7874fdf8:::benchmark_data_smithi171_78918_object61922:head# 2024-04-20T17:11:21.373 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.373 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78751b36:::benchmark_data_smithi171_78918_object62656:head# 2024-04-20T17:11:21.373 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.373 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78763493:::benchmark_data_smithi171_78918_object138443:head# 2024-04-20T17:11:21.374 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.374 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7877c642:::benchmark_data_smithi171_78918_object60121:head# 2024-04-20T17:11:21.374 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.374 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:787842bd:::benchmark_data_smithi171_78918_object120647:head# 2024-04-20T17:11:21.374 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.374 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7879ed82:::benchmark_data_smithi171_78918_object90288:head# 2024-04-20T17:11:21.374 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.374 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:787a190e:::benchmark_data_smithi171_78918_object158281:head# 2024-04-20T17:11:21.374 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.374 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:787a26e3:::benchmark_data_smithi171_78918_object91447:head# 2024-04-20T17:11:21.374 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.374 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:787a3474:::benchmark_data_smithi171_78918_object131438:head# 2024-04-20T17:11:21.374 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.374 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:787a5b76:::benchmark_data_smithi171_78918_object112724:head# 2024-04-20T17:11:21.375 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.375 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:787b1594:::benchmark_data_smithi171_78918_object127738:head# 2024-04-20T17:11:21.375 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.375 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:787bcd6b:::benchmark_data_smithi171_78918_object121274:head# 2024-04-20T17:11:21.375 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.375 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:787cda41:::benchmark_data_smithi171_78918_object138077:head# 2024-04-20T17:11:21.375 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.375 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:787d3104:::benchmark_data_smithi171_78918_object85998:head# 2024-04-20T17:11:21.375 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.375 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:787dc92e:::benchmark_data_smithi171_78918_object81141:head# 2024-04-20T17:11:21.375 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.375 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:787dffa6:::benchmark_data_smithi171_78918_object129148:head# 2024-04-20T17:11:21.375 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.375 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:787e2b79:::benchmark_data_smithi171_78918_object145665:head# 2024-04-20T17:11:21.375 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.376 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:788197b0:::benchmark_data_smithi171_78918_object113018:head# 2024-04-20T17:11:21.376 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.376 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78828387:::benchmark_data_smithi171_78918_object114622:head# 2024-04-20T17:11:21.376 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.376 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:788429dc:::benchmark_data_smithi171_78918_object86109:head# 2024-04-20T17:11:21.376 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.376 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78846324:::benchmark_data_smithi171_78918_object88420:head# 2024-04-20T17:11:21.376 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.376 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:788527f9:::benchmark_data_smithi171_78918_object128774:head# 2024-04-20T17:11:21.376 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.376 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7886b9f0:::benchmark_data_smithi171_78918_object103215:head# 2024-04-20T17:11:21.376 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.376 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7888de65:::benchmark_data_smithi171_78918_object97068:head# 2024-04-20T17:11:21.376 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.377 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:788a55c7:::benchmark_data_smithi171_78918_object112235:head# 2024-04-20T17:11:21.377 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.377 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:788aba18:::benchmark_data_smithi171_78918_object61087:head# 2024-04-20T17:11:21.377 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.377 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:788b3cf1:::benchmark_data_smithi171_78918_object144194:head# 2024-04-20T17:11:21.377 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.377 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:788bded8:::benchmark_data_smithi171_78918_object60087:head# 2024-04-20T17:11:21.377 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.377 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:788bf50d:::benchmark_data_smithi171_78918_object142449:head# 2024-04-20T17:11:21.377 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.377 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:788c8db1:::benchmark_data_smithi171_78918_object65512:head# 2024-04-20T17:11:21.377 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.377 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:788ca52d:::benchmark_data_smithi171_78918_object127981:head# 2024-04-20T17:11:21.377 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.378 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:788d7fc9:::benchmark_data_smithi171_78918_object73517:head# 2024-04-20T17:11:21.378 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.378 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:788d9919:::benchmark_data_smithi171_78918_object53482:head# 2024-04-20T17:11:21.378 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.378 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:788f63d9:::benchmark_data_smithi171_78918_object76671:head# 2024-04-20T17:11:21.378 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.378 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78912e14:::benchmark_data_smithi171_78918_object86103:head# 2024-04-20T17:11:21.378 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.378 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:789221fd:::benchmark_data_smithi171_78918_object54609:head# 2024-04-20T17:11:21.378 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.378 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78928e9f:::benchmark_data_smithi171_78918_object94357:head# 2024-04-20T17:11:21.378 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.378 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78935072:::benchmark_data_smithi171_78918_object125299:head# 2024-04-20T17:11:21.379 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.379 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:789589d7:::benchmark_data_smithi171_78918_object129726:head# 2024-04-20T17:11:21.379 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.379 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7895995e:::benchmark_data_smithi171_78918_object99493:head# 2024-04-20T17:11:21.379 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.379 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7895bcba:::benchmark_data_smithi171_78918_object74671:head# 2024-04-20T17:11:21.379 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.379 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7896109e:::benchmark_data_smithi171_78918_object83616:head# 2024-04-20T17:11:21.379 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.379 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78968459:::benchmark_data_smithi171_78918_object72157:head# 2024-04-20T17:11:21.379 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.379 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7896b2f0:::benchmark_data_smithi171_78918_object92873:head# 2024-04-20T17:11:21.379 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.379 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7896c1fb:::benchmark_data_smithi171_78918_object144363:head# 2024-04-20T17:11:21.379 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.379 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7897ec01:::benchmark_data_smithi171_78918_object91172:head# 2024-04-20T17:11:21.380 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.380 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78982477:::benchmark_data_smithi171_78918_object125162:head# 2024-04-20T17:11:21.380 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.380 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7898e41c:::benchmark_data_smithi171_78918_object137660:head# 2024-04-20T17:11:21.380 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.380 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:789aefe0:::benchmark_data_smithi171_78918_object124520:head# 2024-04-20T17:11:21.380 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.380 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:789b9fce:::benchmark_data_smithi171_78918_object160419:head# 2024-04-20T17:11:21.380 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.380 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:789d76ba:::benchmark_data_smithi171_78918_object141754:head# 2024-04-20T17:11:21.380 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.380 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:789db0c6:::benchmark_data_smithi171_78918_object106712:head# 2024-04-20T17:11:21.380 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.380 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:789de345:::benchmark_data_smithi171_78918_object75328:head# 2024-04-20T17:11:21.380 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.381 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:789de88d:::benchmark_data_smithi171_78918_object55203:head# 2024-04-20T17:11:21.381 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.381 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:789dfba2:::benchmark_data_smithi171_78918_object125222:head# 2024-04-20T17:11:21.381 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.381 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:789e96e8:::benchmark_data_smithi171_78918_object55962:head# 2024-04-20T17:11:21.381 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.381 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:789f2df5:::benchmark_data_smithi171_78918_object97898:head# 2024-04-20T17:11:21.381 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.381 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78a1ed76:::benchmark_data_smithi171_78918_object120971:head# 2024-04-20T17:11:21.381 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.381 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78a38ffb:::benchmark_data_smithi171_78918_object119746:head# 2024-04-20T17:11:21.381 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.381 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78a3efe5:::benchmark_data_smithi171_78918_object120890:head# 2024-04-20T17:11:21.381 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.381 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78a5482a:::benchmark_data_smithi171_78918_object55025:head# 2024-04-20T17:11:21.382 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.382 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78a662eb:::benchmark_data_smithi171_78918_object75731:head# 2024-04-20T17:11:21.382 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.382 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78a7362b:::benchmark_data_smithi171_78918_object129311:head# 2024-04-20T17:11:21.382 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.382 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78a73c44:::benchmark_data_smithi171_78918_object87241:head# 2024-04-20T17:11:21.382 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.382 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78a8fbca:::benchmark_data_smithi171_78918_object98065:head# 2024-04-20T17:11:21.382 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.382 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78a937eb:::benchmark_data_smithi171_78918_object97106:head# 2024-04-20T17:11:21.382 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.382 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78a972eb:::benchmark_data_smithi171_78918_object54785:head# 2024-04-20T17:11:21.382 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.382 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78a9ed53:::benchmark_data_smithi171_78918_object126224:head# 2024-04-20T17:11:21.383 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.383 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78aa1b31:::benchmark_data_smithi171_78918_object103526:head# 2024-04-20T17:11:21.383 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.383 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78acac3d:::benchmark_data_smithi171_78918_object129648:head# 2024-04-20T17:11:21.383 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.383 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78acd321:::benchmark_data_smithi171_78918_object103798:head# 2024-04-20T17:11:21.383 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.383 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78acf578:::benchmark_data_smithi171_78918_object119369:head# 2024-04-20T17:11:21.383 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.383 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78ad0a0b:::benchmark_data_smithi171_78918_object118704:head# 2024-04-20T17:11:21.383 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.383 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78ad1c1e:::benchmark_data_smithi171_78918_object70744:head# 2024-04-20T17:11:21.383 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.383 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78ae71bb:::benchmark_data_smithi171_78918_object89755:head# 2024-04-20T17:11:21.383 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.384 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78aeae65:::benchmark_data_smithi171_78918_object134516:head# 2024-04-20T17:11:21.384 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.384 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78af09f8:::benchmark_data_smithi171_78918_object130444:head# 2024-04-20T17:11:21.384 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.384 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78af953a:::benchmark_data_smithi171_78918_object71317:head# 2024-04-20T17:11:21.384 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.384 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78b07079:::benchmark_data_smithi171_78918_object78339:head# 2024-04-20T17:11:21.384 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.384 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78b10b3d:::benchmark_data_smithi171_78918_object122735:head# 2024-04-20T17:11:21.384 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.384 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78b155aa:::benchmark_data_smithi171_78918_object54970:head# 2024-04-20T17:11:21.384 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.384 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78b2127d:::benchmark_data_smithi171_78918_object82116:head# 2024-04-20T17:11:21.385 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.385 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78b2307e:::benchmark_data_smithi171_78918_object142428:head# 2024-04-20T17:11:21.385 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.385 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78b3964d:::benchmark_data_smithi171_78918_object61461:head# 2024-04-20T17:11:21.385 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.385 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78b4d98a:::benchmark_data_smithi171_78918_object149746:head# 2024-04-20T17:11:21.385 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.385 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78b580b8:::benchmark_data_smithi171_78918_object71458:head# 2024-04-20T17:11:21.385 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.385 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78b7605d:::benchmark_data_smithi171_78918_object119440:head# 2024-04-20T17:11:21.385 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.385 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78b7e789:::benchmark_data_smithi171_78918_object110967:head# 2024-04-20T17:11:21.385 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.385 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78b7f8fe:::benchmark_data_smithi171_78918_object133006:head# 2024-04-20T17:11:21.385 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.386 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78b83deb:::benchmark_data_smithi171_78918_object107759:head# 2024-04-20T17:11:21.386 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.386 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78b87530:::benchmark_data_smithi171_78918_object85609:head# 2024-04-20T17:11:21.386 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.386 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78b8c293:::benchmark_data_smithi171_78918_object155403:head# 2024-04-20T17:11:21.386 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.386 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78b8e3b9:::benchmark_data_smithi171_78918_object82564:head# 2024-04-20T17:11:21.386 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.386 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78b8ebd7:::benchmark_data_smithi171_78918_object95690:head# 2024-04-20T17:11:21.386 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.386 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78b965ba:::benchmark_data_smithi171_78918_object90567:head# 2024-04-20T17:11:21.386 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.386 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78b9c2de:::benchmark_data_smithi171_78918_object98840:head# 2024-04-20T17:11:21.386 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.386 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78b9d434:::benchmark_data_smithi171_78918_object57042:head# 2024-04-20T17:11:21.387 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.387 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78ba8037:::benchmark_data_smithi171_78918_object131330:head# 2024-04-20T17:11:21.387 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.387 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78bb05a2:::benchmark_data_smithi171_78918_object118635:head# 2024-04-20T17:11:21.387 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.387 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78bb49f0:::benchmark_data_smithi171_78918_object150602:head# 2024-04-20T17:11:21.387 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.387 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78bc5af3:::benchmark_data_smithi171_78918_object65767:head# 2024-04-20T17:11:21.387 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.387 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78bcfab0:::benchmark_data_smithi171_78918_object136439:head# 2024-04-20T17:11:21.387 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.387 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78bd61f7:::benchmark_data_smithi171_78918_object73896:head# 2024-04-20T17:11:21.387 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.387 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78bdb709:::benchmark_data_smithi171_78918_object135196:head# 2024-04-20T17:11:21.387 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.388 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78bdecff:::benchmark_data_smithi171_78918_object83405:head# 2024-04-20T17:11:21.388 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.388 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78be5deb:::benchmark_data_smithi171_78918_object131592:head# 2024-04-20T17:11:21.388 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.388 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78bed614:::benchmark_data_smithi171_78918_object100020:head# 2024-04-20T17:11:21.388 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.388 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78bf096a:::benchmark_data_smithi171_78918_object132702:head# 2024-04-20T17:11:21.388 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.388 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78bf5db7:::benchmark_data_smithi171_78918_object155293:head# 2024-04-20T17:11:21.388 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.388 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78c0439c:::benchmark_data_smithi171_78918_object59308:head# 2024-04-20T17:11:21.388 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.388 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78c117a5:::benchmark_data_smithi171_78918_object89940:head# 2024-04-20T17:11:21.388 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.389 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78c14794:::benchmark_data_smithi171_78918_object130485:head# 2024-04-20T17:11:21.389 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.389 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78c1b90b:::benchmark_data_smithi171_78918_object65258:head# 2024-04-20T17:11:21.389 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.389 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78c1bf92:::benchmark_data_smithi171_78918_object126952:head# 2024-04-20T17:11:21.389 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.389 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78c1ca18:::benchmark_data_smithi171_78918_object138743:head# 2024-04-20T17:11:21.389 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.389 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78c29ffa:::benchmark_data_smithi171_78918_object88164:head# 2024-04-20T17:11:21.389 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.389 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78c2b0c9:::benchmark_data_smithi171_78918_object144334:head# 2024-04-20T17:11:21.389 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.389 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78c338f9:::benchmark_data_smithi171_78918_object78767:head# 2024-04-20T17:11:21.389 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.389 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78c476f9:::benchmark_data_smithi171_78918_object107949:head# 2024-04-20T17:11:21.390 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.390 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78c51a36:::benchmark_data_smithi171_78918_object83829:head# 2024-04-20T17:11:21.390 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.390 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78c6157b:::benchmark_data_smithi171_78918_object150962:head# 2024-04-20T17:11:21.390 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.390 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78c65705:::benchmark_data_smithi171_78918_object130425:head# 2024-04-20T17:11:21.390 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.390 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78c88cfc:::benchmark_data_smithi171_78918_object131053:head# 2024-04-20T17:11:21.390 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.390 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78c95f27:::benchmark_data_smithi171_78918_object56860:head# 2024-04-20T17:11:21.390 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.390 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78c96f7e:::benchmark_data_smithi171_78918_object74649:head# 2024-04-20T17:11:21.390 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.390 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78c9a0cc:::benchmark_data_smithi171_78918_object56418:head# 2024-04-20T17:11:21.390 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.391 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78ca01ad:::benchmark_data_smithi171_78918_object129411:head# 2024-04-20T17:11:21.391 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.391 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78ca1f43:::benchmark_data_smithi171_78918_object77391:head# 2024-04-20T17:11:21.391 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.391 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78ca3e5e:::benchmark_data_smithi171_78918_object103011:head# 2024-04-20T17:11:21.391 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.391 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78ca64d7:::benchmark_data_smithi171_78918_object79278:head# 2024-04-20T17:11:21.391 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.391 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78cb5cbc:::benchmark_data_smithi171_78918_object127632:head# 2024-04-20T17:11:21.391 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.391 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78cc7ba4:::benchmark_data_smithi171_78918_object115830:head# 2024-04-20T17:11:21.391 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.391 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78cdf7bb:::benchmark_data_smithi171_78918_object156232:head# 2024-04-20T17:11:21.391 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.391 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78ce5e44:::benchmark_data_smithi171_78918_object102750:head# 2024-04-20T17:11:21.392 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.392 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78ce9f05:::benchmark_data_smithi171_78918_object156382:head# 2024-04-20T17:11:21.392 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.392 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78cf91c1:::benchmark_data_smithi171_78918_object116714:head# 2024-04-20T17:11:21.392 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.392 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78cf9410:::benchmark_data_smithi171_78918_object69248:head# 2024-04-20T17:11:21.392 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.392 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78cfff83:::benchmark_data_smithi171_78918_object111635:head# 2024-04-20T17:11:21.392 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.392 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78d04d25:::benchmark_data_smithi171_78918_object158152:head# 2024-04-20T17:11:21.392 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.392 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78d08b9b:::benchmark_data_smithi171_78918_object81238:head# 2024-04-20T17:11:21.392 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.392 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78d1c1ec:::benchmark_data_smithi171_78918_object80810:head# 2024-04-20T17:11:21.392 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.393 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78d1ca1b:::benchmark_data_smithi171_78918_object109825:head# 2024-04-20T17:11:21.393 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.393 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78d33cef:::benchmark_data_smithi171_78918_object152368:head# 2024-04-20T17:11:21.393 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.393 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78d4507c:::benchmark_data_smithi171_78918_object152292:head# 2024-04-20T17:11:21.393 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.393 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78d52406:::benchmark_data_smithi171_78918_object118999:head# 2024-04-20T17:11:21.393 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.393 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78d64ada:::benchmark_data_smithi171_78918_object94486:head# 2024-04-20T17:11:21.393 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.393 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78d6c9be:::benchmark_data_smithi171_78918_object80801:head# 2024-04-20T17:11:21.393 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.393 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78d6d23b:::benchmark_data_smithi171_78918_object154601:head# 2024-04-20T17:11:21.393 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.394 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78d831d6:::benchmark_data_smithi171_78918_object86679:head# 2024-04-20T17:11:21.394 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.394 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78d832d2:::benchmark_data_smithi171_78918_object67865:head# 2024-04-20T17:11:21.394 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.394 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78d879c4:::benchmark_data_smithi171_78918_object114213:head# 2024-04-20T17:11:21.394 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.394 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78d89ced:::benchmark_data_smithi171_78918_object117689:head# 2024-04-20T17:11:21.394 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.394 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78d95905:::benchmark_data_smithi171_78918_object97733:head# 2024-04-20T17:11:21.394 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.394 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78d9b454:::benchmark_data_smithi171_78918_object122915:head# 2024-04-20T17:11:21.394 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.394 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78db5864:::benchmark_data_smithi171_78918_object65553:head# 2024-04-20T17:11:21.394 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.395 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78db9ad8:::benchmark_data_smithi171_78918_object85920:head# 2024-04-20T17:11:21.395 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.395 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78dbe1b3:::benchmark_data_smithi171_78918_object56307:head# 2024-04-20T17:11:21.395 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.395 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78dc36f3:::benchmark_data_smithi171_78918_object93349:head# 2024-04-20T17:11:21.395 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.395 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78dc77a7:::benchmark_data_smithi171_78918_object123228:head# 2024-04-20T17:11:21.395 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.395 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78dd8843:::benchmark_data_smithi171_78918_object126262:head# 2024-04-20T17:11:21.395 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.395 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78ddc6ea:::benchmark_data_smithi171_78918_object82892:head# 2024-04-20T17:11:21.395 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.395 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78ddec80:::benchmark_data_smithi171_78918_object94122:head# 2024-04-20T17:11:21.395 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.396 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78de0189:::benchmark_data_smithi171_78918_object157522:head# 2024-04-20T17:11:21.396 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.396 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78de35e2:::benchmark_data_smithi171_78918_object51753:head# 2024-04-20T17:11:21.396 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.396 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78de65dd:::benchmark_data_smithi171_78918_object129046:head# 2024-04-20T17:11:21.396 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.396 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78e0297a:::benchmark_data_smithi171_78918_object100041:head# 2024-04-20T17:11:21.396 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.396 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78e0665f:::benchmark_data_smithi171_78918_object77263:head# 2024-04-20T17:11:21.396 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.396 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78e085dd:::benchmark_data_smithi171_78918_object134927:head# 2024-04-20T17:11:21.396 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.396 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78e0b1a6:::benchmark_data_smithi171_78918_object62643:head# 2024-04-20T17:11:21.396 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.397 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78e16e8a:::benchmark_data_smithi171_78918_object97216:head# 2024-04-20T17:11:21.397 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.397 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78e3a6b2:::benchmark_data_smithi171_78918_object108807:head# 2024-04-20T17:11:21.397 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.397 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78e4b17c:::benchmark_data_smithi171_78918_object101579:head# 2024-04-20T17:11:21.397 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.397 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78e5019c:::benchmark_data_smithi171_78918_object72603:head# 2024-04-20T17:11:21.397 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.397 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78e5558b:::benchmark_data_smithi171_78918_object77493:head# 2024-04-20T17:11:21.397 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.397 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78e5e450:::benchmark_data_smithi171_78918_object73882:head# 2024-04-20T17:11:21.397 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.397 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78e5e9d4:::benchmark_data_smithi171_78918_object96304:head# 2024-04-20T17:11:21.397 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.398 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78e67bf6:::benchmark_data_smithi171_78918_object128227:head# 2024-04-20T17:11:21.398 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.398 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78e67c6e:::benchmark_data_smithi171_78918_object74032:head# 2024-04-20T17:11:21.398 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.398 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78e67e0d:::benchmark_data_smithi171_78918_object151901:head# 2024-04-20T17:11:21.398 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.398 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78e68bc9:::benchmark_data_smithi171_78918_object84524:head# 2024-04-20T17:11:21.398 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.398 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78e78203:::benchmark_data_smithi171_78918_object52573:head# 2024-04-20T17:11:21.398 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.398 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78e7af3e:::benchmark_data_smithi171_78918_object54977:head# 2024-04-20T17:11:21.398 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.398 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78e81195:::benchmark_data_smithi171_78918_object86823:head# 2024-04-20T17:11:21.398 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.398 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78e8261d:::benchmark_data_smithi171_78918_object59357:head# 2024-04-20T17:11:21.399 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.401 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78e84b41:::benchmark_data_smithi171_78918_object89464:head# 2024-04-20T17:11:21.401 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.401 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78e852d8:::benchmark_data_smithi171_78918_object133426:head# 2024-04-20T17:11:21.401 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.401 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78e8b28b:::benchmark_data_smithi171_78918_object100741:head# 2024-04-20T17:11:21.401 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.401 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78e8e8a4:::benchmark_data_smithi171_78918_object91138:head# 2024-04-20T17:11:21.402 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.402 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78e8fd8e:::benchmark_data_smithi171_78918_object97392:head# 2024-04-20T17:11:21.402 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.402 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78e91915:::benchmark_data_smithi171_78918_object132849:head# 2024-04-20T17:11:21.402 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.402 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78e9c86d:::benchmark_data_smithi171_78918_object140116:head# 2024-04-20T17:11:21.402 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.402 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78ebc26c:::benchmark_data_smithi171_78918_object94046:head# 2024-04-20T17:11:21.402 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.402 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78ebd482:::benchmark_data_smithi171_78918_object129833:head# 2024-04-20T17:11:21.402 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.402 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78ee1b6e:::benchmark_data_smithi171_78918_object121595:head# 2024-04-20T17:11:21.402 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.402 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78eec418:::benchmark_data_smithi171_78918_object159634:head# 2024-04-20T17:11:21.403 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.403 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78efb1e3:::benchmark_data_smithi171_78918_object51670:head# 2024-04-20T17:11:21.403 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.403 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78efda83:::benchmark_data_smithi171_78918_object69501:head# 2024-04-20T17:11:21.403 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.403 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78f04015:::benchmark_data_smithi171_78918_object105887:head# 2024-04-20T17:11:21.403 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.403 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78f05d75:::benchmark_data_smithi171_78918_object96150:head# 2024-04-20T17:11:21.403 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.403 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78f0a099:::benchmark_data_smithi171_78918_object70070:head# 2024-04-20T17:11:21.403 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.403 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78f0f67a:::benchmark_data_smithi171_78918_object101899:head# 2024-04-20T17:11:21.403 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.403 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78f1a002:::benchmark_data_smithi171_78918_object159261:head# 2024-04-20T17:11:21.403 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.404 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78f20e57:::benchmark_data_smithi171_78918_object159689:head# 2024-04-20T17:11:21.404 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.404 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78f210b0:::benchmark_data_smithi171_78918_object110642:head# 2024-04-20T17:11:21.404 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.404 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78f2f50d:::benchmark_data_smithi171_78918_object139723:head# 2024-04-20T17:11:21.404 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.404 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78f3c171:::benchmark_data_smithi171_78918_object87697:head# 2024-04-20T17:11:21.404 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.404 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78f43fa2:::benchmark_data_smithi171_78918_object155692:head# 2024-04-20T17:11:21.404 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.404 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78f47cd0:::benchmark_data_smithi171_78918_object119124:head# 2024-04-20T17:11:21.404 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.404 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78f4a6ae:::benchmark_data_smithi171_78918_object82392:head# 2024-04-20T17:11:21.404 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.405 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78f4e5f7:::benchmark_data_smithi171_78918_object121421:head# 2024-04-20T17:11:21.405 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.405 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78f56d93:::benchmark_data_smithi171_78918_object80795:head# 2024-04-20T17:11:21.405 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.405 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78f570b7:::benchmark_data_smithi171_78918_object86998:head# 2024-04-20T17:11:21.405 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.405 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78f5878b:::benchmark_data_smithi171_78918_object147173:head# 2024-04-20T17:11:21.405 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.405 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78f7ad2b:::benchmark_data_smithi171_78918_object137551:head# 2024-04-20T17:11:21.405 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.405 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78f8c939:::benchmark_data_smithi171_78918_object52189:head# 2024-04-20T17:11:21.405 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.405 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78fa98ef:::benchmark_data_smithi171_78918_object146813:head# 2024-04-20T17:11:21.405 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.405 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78fb1c5f:::benchmark_data_smithi171_78918_object74988:head# 2024-04-20T17:11:21.405 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.406 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78fb2a87:::benchmark_data_smithi171_78918_object117626:head# 2024-04-20T17:11:21.406 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.406 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78fc347c:::benchmark_data_smithi171_78918_object81890:head# 2024-04-20T17:11:21.406 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.406 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78fcea13:::benchmark_data_smithi171_78918_object137813:head# 2024-04-20T17:11:21.406 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.406 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78fceb38:::benchmark_data_smithi171_78918_object101464:head# 2024-04-20T17:11:21.406 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.406 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78fde233:::benchmark_data_smithi171_78918_object119589:head# 2024-04-20T17:11:21.406 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.406 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78fec9c7:::benchmark_data_smithi171_78918_object54225:head# 2024-04-20T17:11:21.406 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.406 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78ff0dc2:::benchmark_data_smithi171_78918_object139308:head# 2024-04-20T17:11:21.406 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.407 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78ff244e:::benchmark_data_smithi171_78918_object81821:head# 2024-04-20T17:11:21.407 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.407 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:78ff509c:::benchmark_data_smithi171_78918_object60647:head# 2024-04-20T17:11:21.407 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.407 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:790058f3:::benchmark_data_smithi171_78918_object156491:head# 2024-04-20T17:11:21.407 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.407 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79009e57:::benchmark_data_smithi171_78918_object102911:head# 2024-04-20T17:11:21.407 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.407 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7900ab4e:::benchmark_data_smithi171_78918_object65764:head# 2024-04-20T17:11:21.407 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.407 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:790169ef:::benchmark_data_smithi171_78918_object95397:head# 2024-04-20T17:11:21.407 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.407 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79018034:::benchmark_data_smithi171_78918_object98758:head# 2024-04-20T17:11:21.407 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.407 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79026ce0:::benchmark_data_smithi171_78918_object92999:head# 2024-04-20T17:11:21.408 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.408 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7902add7:::benchmark_data_smithi171_78918_object65346:head# 2024-04-20T17:11:21.408 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.408 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79030dfd:::benchmark_data_smithi171_78918_object99107:head# 2024-04-20T17:11:21.408 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.408 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:790563d0:::benchmark_data_smithi171_78918_object138527:head# 2024-04-20T17:11:21.408 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.408 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7905af6e:::benchmark_data_smithi171_78918_object148856:head# 2024-04-20T17:11:21.408 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.408 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7905b259:::benchmark_data_smithi171_78918_object158217:head# 2024-04-20T17:11:21.408 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.408 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7905ba76:::benchmark_data_smithi171_78918_object69902:head# 2024-04-20T17:11:21.408 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.408 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79060ee0:::benchmark_data_smithi171_78918_object124964:head# 2024-04-20T17:11:21.408 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.408 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7906384f:::benchmark_data_smithi171_78918_object87496:head# 2024-04-20T17:11:21.409 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.409 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79064dd8:::benchmark_data_smithi171_78918_object70030:head# 2024-04-20T17:11:21.409 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.409 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7906cbab:::benchmark_data_smithi171_78918_object97103:head# 2024-04-20T17:11:21.409 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.409 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79079987:::benchmark_data_smithi171_78918_object63096:head# 2024-04-20T17:11:21.409 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.409 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7908d39f:::benchmark_data_smithi171_78918_object93059:head# 2024-04-20T17:11:21.409 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.409 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:790915e5:::benchmark_data_smithi171_78918_object87093:head# 2024-04-20T17:11:21.409 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.409 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:790a4775:::benchmark_data_smithi171_78918_object127726:head# 2024-04-20T17:11:21.409 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.409 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:790a59d6:::benchmark_data_smithi171_78918_object96782:head# 2024-04-20T17:11:21.409 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.410 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:790b5aec:::benchmark_data_smithi171_78918_object103764:head# 2024-04-20T17:11:21.410 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.410 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:790bb1b2:::benchmark_data_smithi171_78918_object122824:head# 2024-04-20T17:11:21.410 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.410 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:790bd32a:::benchmark_data_smithi171_78918_object57611:head# 2024-04-20T17:11:21.410 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.410 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:790c8153:::benchmark_data_smithi171_78918_object137110:head# 2024-04-20T17:11:21.410 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.525 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:790ce320:::benchmark_data_smithi171_78918_object127434:head# 2024-04-20T17:11:21.525 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.525 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:790d9219:::benchmark_data_smithi171_78918_object68017:head# 2024-04-20T17:11:21.525 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.525 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:790dd235:::benchmark_data_smithi171_78918_object99026:head# 2024-04-20T17:11:21.525 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.525 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:790e2b7f:::benchmark_data_smithi171_78918_object61018:head# 2024-04-20T17:11:21.525 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.526 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:790e5476:::benchmark_data_smithi171_78918_object107805:head# 2024-04-20T17:11:21.526 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.526 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:790e8067:::benchmark_data_smithi171_78918_object68783:head# 2024-04-20T17:11:21.526 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.526 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:790edff0:::benchmark_data_smithi171_78918_object115253:head# 2024-04-20T17:11:21.526 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.526 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:790fd5c8:::benchmark_data_smithi171_78918_object119324:head# 2024-04-20T17:11:21.526 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.526 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79109dc8:::benchmark_data_smithi171_78918_object153079:head# 2024-04-20T17:11:21.526 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.526 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79117b40:::benchmark_data_smithi171_78918_object117109:head# 2024-04-20T17:11:21.526 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.526 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79130eed:::benchmark_data_smithi171_78918_object64131:head# 2024-04-20T17:11:21.527 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.527 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7913230c:::benchmark_data_smithi171_78918_object116363:head# 2024-04-20T17:11:21.527 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.527 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7913aff1:::benchmark_data_smithi171_78918_object73583:head# 2024-04-20T17:11:21.527 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.527 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:791404f1:::benchmark_data_smithi171_78918_object80379:head# 2024-04-20T17:11:21.527 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.527 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79142af0:::benchmark_data_smithi171_78918_object137993:head# 2024-04-20T17:11:21.527 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.527 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7914d58c:::benchmark_data_smithi171_78918_object78684:head# 2024-04-20T17:11:21.527 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.527 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7915f1bc:::benchmark_data_smithi171_78918_object109164:head# 2024-04-20T17:11:21.527 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.527 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7915faf9:::benchmark_data_smithi171_78918_object146712:head# 2024-04-20T17:11:21.528 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.528 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79177595:::benchmark_data_smithi171_78918_object93684:head# 2024-04-20T17:11:21.528 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.528 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7917af5e:::benchmark_data_smithi171_78918_object107758:head# 2024-04-20T17:11:21.528 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.528 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7917ee3d:::benchmark_data_smithi171_78918_object135192:head# 2024-04-20T17:11:21.528 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.528 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7917fccf:::benchmark_data_smithi171_78918_object102367:head# 2024-04-20T17:11:21.528 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.528 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79186c02:::benchmark_data_smithi171_78918_object115970:head# 2024-04-20T17:11:21.528 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.528 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7918b4ad:::benchmark_data_smithi171_78918_object110007:head# 2024-04-20T17:11:21.528 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.528 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7918fc0f:::benchmark_data_smithi171_78918_object56807:head# 2024-04-20T17:11:21.528 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.529 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79195039:::benchmark_data_smithi171_78918_object130614:head# 2024-04-20T17:11:21.529 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.529 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7919a303:::benchmark_data_smithi171_78918_object118507:head# 2024-04-20T17:11:21.529 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.529 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:791b3674:::benchmark_data_smithi171_78918_object81265:head# 2024-04-20T17:11:21.529 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.529 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:791bcf16:::benchmark_data_smithi171_78918_object113206:head# 2024-04-20T17:11:21.529 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.529 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:791be73f:::benchmark_data_smithi171_78918_object138809:head# 2024-04-20T17:11:21.529 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.529 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:791c849b:::benchmark_data_smithi171_78918_object61216:head# 2024-04-20T17:11:21.529 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.529 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:791dee3e:::benchmark_data_smithi171_78918_object112255:head# 2024-04-20T17:11:21.529 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.530 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:791e4a56:::benchmark_data_smithi171_78918_object97314:head# 2024-04-20T17:11:21.530 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.530 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79208c37:::benchmark_data_smithi171_78918_object145367:head# 2024-04-20T17:11:21.530 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.530 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7920f948:::benchmark_data_smithi171_78918_object64785:head# 2024-04-20T17:11:21.530 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.530 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79211cb3:::benchmark_data_smithi171_78918_object142322:head# 2024-04-20T17:11:21.530 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.530 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:792168a6:::benchmark_data_smithi171_78918_object63671:head# 2024-04-20T17:11:21.530 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.530 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79217b5d:::benchmark_data_smithi171_78918_object97098:head# 2024-04-20T17:11:21.530 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.530 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7921b5d3:::benchmark_data_smithi171_78918_object103207:head# 2024-04-20T17:11:21.530 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.531 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7921c9f4:::benchmark_data_smithi171_78918_object151811:head# 2024-04-20T17:11:21.531 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.531 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7921dfc1:::benchmark_data_smithi171_78918_object128606:head# 2024-04-20T17:11:21.531 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.531 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79228808:::benchmark_data_smithi171_78918_object75349:head# 2024-04-20T17:11:21.531 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.531 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79229c1e:::benchmark_data_smithi171_78918_object159926:head# 2024-04-20T17:11:21.531 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.531 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7922b7fb:::benchmark_data_smithi171_78918_object119885:head# 2024-04-20T17:11:21.531 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.531 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:792376bb:::benchmark_data_smithi171_78918_object52693:head# 2024-04-20T17:11:21.531 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.531 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79239c1b:::benchmark_data_smithi171_78918_object115061:head# 2024-04-20T17:11:21.531 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.532 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7924060e:::benchmark_data_smithi171_78918_object91631:head# 2024-04-20T17:11:21.532 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.532 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79243e78:::benchmark_data_smithi171_78918_object134947:head# 2024-04-20T17:11:21.532 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.532 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79246921:::benchmark_data_smithi171_78918_object65022:head# 2024-04-20T17:11:21.532 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.532 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:792490a0:::benchmark_data_smithi171_78918_object155927:head# 2024-04-20T17:11:21.532 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.532 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79257fd8:::benchmark_data_smithi171_78918_object131898:head# 2024-04-20T17:11:21.532 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.532 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:792633ac:::benchmark_data_smithi171_78918_object100535:head# 2024-04-20T17:11:21.532 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.532 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7926a093:::benchmark_data_smithi171_78918_object117704:head# 2024-04-20T17:11:21.532 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.532 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:792b0f18:::benchmark_data_smithi171_78918_object86215:head# 2024-04-20T17:11:21.533 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.533 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:792b2dc5:::benchmark_data_smithi171_78918_object86186:head# 2024-04-20T17:11:21.533 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.533 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:792b80a2:::benchmark_data_smithi171_78918_object148603:head# 2024-04-20T17:11:21.533 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.533 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:792bee67:::benchmark_data_smithi171_78918_object56953:head# 2024-04-20T17:11:21.533 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.533 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:792c69f6:::benchmark_data_smithi171_78918_object78250:head# 2024-04-20T17:11:21.533 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.533 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:792c7b07:::benchmark_data_smithi171_78918_object77884:head# 2024-04-20T17:11:21.533 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.533 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:792ca4d5:::benchmark_data_smithi171_78918_object62889:head# 2024-04-20T17:11:21.533 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.533 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:792df878:::benchmark_data_smithi171_78918_object79134:head# 2024-04-20T17:11:21.534 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.534 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:792ec530:::benchmark_data_smithi171_78918_object154033:head# 2024-04-20T17:11:21.534 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.534 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:792f1cbb:::benchmark_data_smithi171_78918_object82460:head# 2024-04-20T17:11:21.534 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.534 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:792fc817:::benchmark_data_smithi171_78918_object54139:head# 2024-04-20T17:11:21.534 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.534 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79306783:::benchmark_data_smithi171_78918_object150541:head# 2024-04-20T17:11:21.534 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.534 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7930a0d5:::benchmark_data_smithi171_78918_object121209:head# 2024-04-20T17:11:21.534 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.534 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:793190f1:::benchmark_data_smithi171_78918_object56035:head# 2024-04-20T17:11:21.534 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.534 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7931bbde:::benchmark_data_smithi171_78918_object134875:head# 2024-04-20T17:11:21.535 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.535 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79329ff3:::benchmark_data_smithi171_78918_object126072:head# 2024-04-20T17:11:21.535 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.535 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79336cf4:::benchmark_data_smithi171_78918_object55387:head# 2024-04-20T17:11:21.535 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.535 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79337f70:::benchmark_data_smithi171_78918_object63110:head# 2024-04-20T17:11:21.535 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.535 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7937a16c:::benchmark_data_smithi171_78918_object65264:head# 2024-04-20T17:11:21.535 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.535 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7938559f:::benchmark_data_smithi171_78918_object53148:head# 2024-04-20T17:11:21.535 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.535 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79395706:::benchmark_data_smithi171_78918_object117827:head# 2024-04-20T17:11:21.535 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.535 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:793a5ace:::benchmark_data_smithi171_78918_object150966:head# 2024-04-20T17:11:21.536 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.536 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:793ac66c:::benchmark_data_smithi171_78918_object67960:head# 2024-04-20T17:11:21.536 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.536 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:793bae04:::benchmark_data_smithi171_78918_object107403:head# 2024-04-20T17:11:21.536 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.536 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:793bbaf8:::benchmark_data_smithi171_78918_object158433:head# 2024-04-20T17:11:21.536 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.536 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:793d8c54:::benchmark_data_smithi171_78918_object159241:head# 2024-04-20T17:11:21.536 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.536 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:793d93a3:::benchmark_data_smithi171_78918_object86091:head# 2024-04-20T17:11:21.536 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.536 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:793d9e04:::benchmark_data_smithi171_78918_object121645:head# 2024-04-20T17:11:21.536 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.536 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:793dd06f:::benchmark_data_smithi171_78918_object89835:head# 2024-04-20T17:11:21.536 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.537 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:793edb95:::benchmark_data_smithi171_78918_object116215:head# 2024-04-20T17:11:21.537 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.537 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:793f0f1d:::benchmark_data_smithi171_78918_object141544:head# 2024-04-20T17:11:21.537 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.537 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:793f6aa2:::benchmark_data_smithi171_78918_object111937:head# 2024-04-20T17:11:21.537 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.537 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:793f9092:::benchmark_data_smithi171_78918_object125322:head# 2024-04-20T17:11:21.537 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.537 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:794139c1:::benchmark_data_smithi171_78918_object63812:head# 2024-04-20T17:11:21.537 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.537 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7941d33b:::benchmark_data_smithi171_78918_object118010:head# 2024-04-20T17:11:21.537 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.537 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79428db6:::benchmark_data_smithi171_78918_object103185:head# 2024-04-20T17:11:21.537 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.538 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79428ffe:::benchmark_data_smithi171_78918_object147637:head# 2024-04-20T17:11:21.538 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.538 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79429430:::benchmark_data_smithi171_78918_object131357:head# 2024-04-20T17:11:21.538 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.538 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7942e338:::benchmark_data_smithi171_78918_object55003:head# 2024-04-20T17:11:21.538 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.538 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79436490:::benchmark_data_smithi171_78918_object69230:head# 2024-04-20T17:11:21.538 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.538 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79436bfe:::benchmark_data_smithi171_78918_object146305:head# 2024-04-20T17:11:21.538 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.538 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7944a05e:::benchmark_data_smithi171_78918_object138673:head# 2024-04-20T17:11:21.538 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.538 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79451a46:::benchmark_data_smithi171_78918_object104799:head# 2024-04-20T17:11:21.538 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.538 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7945cb47:::benchmark_data_smithi171_78918_object144321:head# 2024-04-20T17:11:21.539 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.539 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7945d66a:::benchmark_data_smithi171_78918_object133096:head# 2024-04-20T17:11:21.539 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.539 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7945fbfe:::benchmark_data_smithi171_78918_object83495:head# 2024-04-20T17:11:21.539 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.539 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:794629fc:::benchmark_data_smithi171_78918_object89184:head# 2024-04-20T17:11:21.539 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.539 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:794947f0:::benchmark_data_smithi171_78918_object145592:head# 2024-04-20T17:11:21.539 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.539 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79495fbd:::benchmark_data_smithi171_78918_object93032:head# 2024-04-20T17:11:21.539 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.539 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7949a836:::benchmark_data_smithi171_78918_object62242:head# 2024-04-20T17:11:21.539 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.539 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:794ac0a1:::benchmark_data_smithi171_78918_object84030:head# 2024-04-20T17:11:21.539 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.540 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:794ad50f:::benchmark_data_smithi171_78918_object125713:head# 2024-04-20T17:11:21.540 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.540 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:794b95e1:::benchmark_data_smithi171_78918_object72347:head# 2024-04-20T17:11:21.540 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.540 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:794c22c5:::benchmark_data_smithi171_78918_object73956:head# 2024-04-20T17:11:21.540 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.540 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:794c3740:::benchmark_data_smithi171_78918_object91431:head# 2024-04-20T17:11:21.540 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.540 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:794c800f:::benchmark_data_smithi171_78918_object55600:head# 2024-04-20T17:11:21.540 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.540 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:794d3caa:::benchmark_data_smithi171_78918_object90402:head# 2024-04-20T17:11:21.540 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.540 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:794e920f:::benchmark_data_smithi171_78918_object96793:head# 2024-04-20T17:11:21.541 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.541 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:794ef673:::benchmark_data_smithi171_78918_object141492:head# 2024-04-20T17:11:21.541 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.541 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79501dd6:::benchmark_data_smithi171_78918_object65752:head# 2024-04-20T17:11:21.541 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.541 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:795031ee:::benchmark_data_smithi171_78918_object148929:head# 2024-04-20T17:11:21.541 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.541 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7950a031:::benchmark_data_smithi171_78918_object152649:head# 2024-04-20T17:11:21.541 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.541 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7950d587:::benchmark_data_smithi171_78918_object110358:head# 2024-04-20T17:11:21.541 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.541 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7950e4ab:::benchmark_data_smithi171_78918_object66977:head# 2024-04-20T17:11:21.541 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.541 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7951801a:::benchmark_data_smithi171_78918_object102154:head# 2024-04-20T17:11:21.541 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.542 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79522a53:::benchmark_data_smithi171_78918_object85426:head# 2024-04-20T17:11:21.542 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.542 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:795249f8:::benchmark_data_smithi171_78918_object68635:head# 2024-04-20T17:11:21.542 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.542 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7952ae9f:::benchmark_data_smithi171_78918_object100080:head# 2024-04-20T17:11:21.542 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.542 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7952b387:::benchmark_data_smithi171_78918_object132741:head# 2024-04-20T17:11:21.542 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.542 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7952e035:::benchmark_data_smithi171_78918_object74704:head# 2024-04-20T17:11:21.542 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.542 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79531121:::benchmark_data_smithi171_78918_object98826:head# 2024-04-20T17:11:21.542 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.542 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:795379ce:::benchmark_data_smithi171_78918_object105068:head# 2024-04-20T17:11:21.542 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.542 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7953e126:::benchmark_data_smithi171_78918_object160821:head# 2024-04-20T17:11:21.543 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.543 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79545a0f:::benchmark_data_smithi171_78918_object153127:head# 2024-04-20T17:11:21.543 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.543 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79545cdf:::benchmark_data_smithi171_78918_object132368:head# 2024-04-20T17:11:21.543 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.543 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7954e890:::benchmark_data_smithi171_78918_object99028:head# 2024-04-20T17:11:21.543 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.543 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79554997:::benchmark_data_smithi171_78918_object156098:head# 2024-04-20T17:11:21.543 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.543 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7955b17b:::benchmark_data_smithi171_78918_object79805:head# 2024-04-20T17:11:21.543 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.543 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79561c65:::benchmark_data_smithi171_78918_object108009:head# 2024-04-20T17:11:21.543 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.543 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79563338:::benchmark_data_smithi171_78918_object63721:head# 2024-04-20T17:11:21.543 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.544 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7956a4ac:::benchmark_data_smithi171_78918_object75459:head# 2024-04-20T17:11:21.544 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.544 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79570cdb:::benchmark_data_smithi171_78918_object153443:head# 2024-04-20T17:11:21.544 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.544 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:795755e6:::benchmark_data_smithi171_78918_object76651:head# 2024-04-20T17:11:21.544 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.544 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7957db6e:::benchmark_data_smithi171_78918_object109312:head# 2024-04-20T17:11:21.544 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.544 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79587101:::benchmark_data_smithi171_78918_object107428:head# 2024-04-20T17:11:21.544 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.544 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79593b3e:::benchmark_data_smithi171_78918_object139243:head# 2024-04-20T17:11:21.544 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.545 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79599922:::benchmark_data_smithi171_78918_object125289:head# 2024-04-20T17:11:21.545 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.545 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7959ac5f:::benchmark_data_smithi171_78918_object71794:head# 2024-04-20T17:11:21.545 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.545 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:795a7066:::benchmark_data_smithi171_78918_object80554:head# 2024-04-20T17:11:21.545 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.545 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:795ad9b4:::benchmark_data_smithi171_78918_object68890:head# 2024-04-20T17:11:21.545 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.545 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:795b2c0c:::benchmark_data_smithi171_78918_object65473:head# 2024-04-20T17:11:21.545 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.545 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:795b67ef:::benchmark_data_smithi171_78918_object142553:head# 2024-04-20T17:11:21.545 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.545 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:795c31dd:::benchmark_data_smithi171_78918_object94198:head# 2024-04-20T17:11:21.545 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.546 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:795c8791:::benchmark_data_smithi171_78918_object69692:head# 2024-04-20T17:11:21.546 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.546 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:795d3662:::benchmark_data_smithi171_78918_object117693:head# 2024-04-20T17:11:21.546 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.546 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:795d648d:::benchmark_data_smithi171_78918_object66485:head# 2024-04-20T17:11:21.546 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.546 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:795dbdbe:::benchmark_data_smithi171_78918_object104875:head# 2024-04-20T17:11:21.546 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.546 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:795df80b:::benchmark_data_smithi171_78918_object55616:head# 2024-04-20T17:11:21.546 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.546 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:795e1191:::benchmark_data_smithi171_78918_object154740:head# 2024-04-20T17:11:21.546 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.546 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:795fba15:::benchmark_data_smithi171_78918_object127877:head# 2024-04-20T17:11:21.546 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.547 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7960448e:::benchmark_data_smithi171_78918_object110468:head# 2024-04-20T17:11:21.547 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.547 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79614fc1:::benchmark_data_smithi171_78918_object52477:head# 2024-04-20T17:11:21.547 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.547 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7961a32d:::benchmark_data_smithi171_78918_object94467:head# 2024-04-20T17:11:21.547 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.547 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7961ea3b:::benchmark_data_smithi171_78918_object120289:head# 2024-04-20T17:11:21.547 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.547 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7962e59d:::benchmark_data_smithi171_78918_object152854:head# 2024-04-20T17:11:21.547 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.547 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7963073e:::benchmark_data_smithi171_78918_object81723:head# 2024-04-20T17:11:21.547 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.547 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:796378a5:::benchmark_data_smithi171_78918_object125126:head# 2024-04-20T17:11:21.547 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.547 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7964da80:::benchmark_data_smithi171_78918_object113796:head# 2024-04-20T17:11:21.548 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.548 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79663673:::benchmark_data_smithi171_78918_object129970:head# 2024-04-20T17:11:21.548 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.548 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79664b06:::benchmark_data_smithi171_78918_object133427:head# 2024-04-20T17:11:21.548 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.548 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79671ce8:::benchmark_data_smithi171_78918_object137180:head# 2024-04-20T17:11:21.548 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.548 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79675072:::benchmark_data_smithi171_78918_object74075:head# 2024-04-20T17:11:21.548 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.548 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7967b604:::benchmark_data_smithi171_78918_object85873:head# 2024-04-20T17:11:21.548 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.548 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7968a9d4:::benchmark_data_smithi171_78918_object121399:head# 2024-04-20T17:11:21.548 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.548 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7968bd9d:::benchmark_data_smithi171_78918_object102571:head# 2024-04-20T17:11:21.548 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.549 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7968c896:::benchmark_data_smithi171_78918_object75262:head# 2024-04-20T17:11:21.549 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.549 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:796ab515:::benchmark_data_smithi171_78918_object77567:head# 2024-04-20T17:11:21.549 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.549 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:796afc7c:::benchmark_data_smithi171_78918_object148737:head# 2024-04-20T17:11:21.549 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.549 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:796b28ef:::benchmark_data_smithi171_78918_object126655:head# 2024-04-20T17:11:21.549 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.549 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:796b2aac:::benchmark_data_smithi171_78918_object91898:head# 2024-04-20T17:11:21.549 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.549 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:796c1e73:::benchmark_data_smithi171_78918_object147436:head# 2024-04-20T17:11:21.549 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.549 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:796c6da3:::benchmark_data_smithi171_78918_object121004:head# 2024-04-20T17:11:21.549 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.550 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:796cc161:::benchmark_data_smithi171_78918_object153437:head# 2024-04-20T17:11:21.550 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.550 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:796cd779:::benchmark_data_smithi171_78918_object83169:head# 2024-04-20T17:11:21.550 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.550 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:796f9fc3:::benchmark_data_smithi171_78918_object68251:head# 2024-04-20T17:11:21.550 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.550 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:796ff920:::benchmark_data_smithi171_78918_object147769:head# 2024-04-20T17:11:21.550 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.550 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79702930:::benchmark_data_smithi171_78918_object140302:head# 2024-04-20T17:11:21.550 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.550 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7970537b:::benchmark_data_smithi171_78918_object97116:head# 2024-04-20T17:11:21.550 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.550 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79708e00:::benchmark_data_smithi171_78918_object72481:head# 2024-04-20T17:11:21.550 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.550 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7970affc:::benchmark_data_smithi171_78918_object68212:head# 2024-04-20T17:11:21.551 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.551 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7970b0ed:::benchmark_data_smithi171_78918_object55038:head# 2024-04-20T17:11:21.551 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.551 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7970cd43:::benchmark_data_smithi171_78918_object57267:head# 2024-04-20T17:11:21.551 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.551 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7970d8eb:::benchmark_data_smithi171_78918_object59211:head# 2024-04-20T17:11:21.551 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.551 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:797121ba:::benchmark_data_smithi171_78918_object132391:head# 2024-04-20T17:11:21.551 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.551 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:797216f0:::benchmark_data_smithi171_78918_object125052:head# 2024-04-20T17:11:21.551 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.551 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7972de0c:::benchmark_data_smithi171_78918_object122153:head# 2024-04-20T17:11:21.551 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.551 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79737203:::benchmark_data_smithi171_78918_object152105:head# 2024-04-20T17:11:21.551 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.552 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:797456f3:::benchmark_data_smithi171_78918_object59000:head# 2024-04-20T17:11:21.552 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.552 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7976bd47:::benchmark_data_smithi171_78918_object63745:head# 2024-04-20T17:11:21.552 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.552 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7977075d:::benchmark_data_smithi171_78918_object63076:head# 2024-04-20T17:11:21.552 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.552 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79775d2c:::benchmark_data_smithi171_78918_object78552:head# 2024-04-20T17:11:21.552 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.552 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:797762f2:::benchmark_data_smithi171_78918_object56568:head# 2024-04-20T17:11:21.552 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.552 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79777a11:::benchmark_data_smithi171_78918_object84696:head# 2024-04-20T17:11:21.552 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.552 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7977ba49:::benchmark_data_smithi171_78918_object158018:head# 2024-04-20T17:11:21.552 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.553 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7977fae8:::benchmark_data_smithi171_78918_object87299:head# 2024-04-20T17:11:21.553 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.553 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79784d0d:::benchmark_data_smithi171_78918_object58625:head# 2024-04-20T17:11:21.553 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.553 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7979d19f:::benchmark_data_smithi171_78918_object123738:head# 2024-04-20T17:11:21.553 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.553 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7979f6ea:::benchmark_data_smithi171_78918_object92665:head# 2024-04-20T17:11:21.553 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.553 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:797a3e34:::benchmark_data_smithi171_78918_object71972:head# 2024-04-20T17:11:21.553 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.553 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:797b5ec2:::benchmark_data_smithi171_78918_object115224:head# 2024-04-20T17:11:21.553 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.553 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:797bba50:::benchmark_data_smithi171_78918_object81657:head# 2024-04-20T17:11:21.553 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.553 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:797bd2cb:::benchmark_data_smithi171_78918_object90275:head# 2024-04-20T17:11:21.554 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.554 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:797c2d1d:::benchmark_data_smithi171_78918_object99858:head# 2024-04-20T17:11:21.554 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.554 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:797c7366:::benchmark_data_smithi171_78918_object136904:head# 2024-04-20T17:11:21.554 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.554 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:797d23a6:::benchmark_data_smithi171_78918_object151182:head# 2024-04-20T17:11:21.554 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.554 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:797d42a8:::benchmark_data_smithi171_78918_object86500:head# 2024-04-20T17:11:21.554 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.554 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:797d5bb5:::benchmark_data_smithi171_78918_object66036:head# 2024-04-20T17:11:21.554 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.554 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:797df35d:::benchmark_data_smithi171_78918_object126536:head# 2024-04-20T17:11:21.554 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.554 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:797e3e43:::benchmark_data_smithi171_78918_object87155:head# 2024-04-20T17:11:21.555 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.555 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:797e5764:::benchmark_data_smithi171_78918_object84833:head# 2024-04-20T17:11:21.555 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.555 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:797f4b9e:::benchmark_data_smithi171_78918_object160890:head# 2024-04-20T17:11:21.555 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.555 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7980500e:::benchmark_data_smithi171_78918_object121751:head# 2024-04-20T17:11:21.555 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.555 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7980f935:::benchmark_data_smithi171_78918_object60350:head# 2024-04-20T17:11:21.555 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.555 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79829d84:::benchmark_data_smithi171_78918_object63297:head# 2024-04-20T17:11:21.555 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.555 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79833cee:::benchmark_data_smithi171_78918_object133374:head# 2024-04-20T17:11:21.555 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.555 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79835c1a:::benchmark_data_smithi171_78918_object70435:head# 2024-04-20T17:11:21.555 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.556 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79845a25:::benchmark_data_smithi171_78918_object62082:head# 2024-04-20T17:11:21.556 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.556 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7984c6ff:::benchmark_data_smithi171_78918_object75377:head# 2024-04-20T17:11:21.556 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.556 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7984ccd2:::benchmark_data_smithi171_78918_object123059:head# 2024-04-20T17:11:21.556 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.556 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:798508e8:::benchmark_data_smithi171_78918_object130722:head# 2024-04-20T17:11:21.556 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.556 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7986be57:::benchmark_data_smithi171_78918_object151883:head# 2024-04-20T17:11:21.556 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.556 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79870bd7:::benchmark_data_smithi171_78918_object111333:head# 2024-04-20T17:11:21.556 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.556 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79873f58:::benchmark_data_smithi171_78918_object66909:head# 2024-04-20T17:11:21.556 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.556 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7987cea2:::benchmark_data_smithi171_78918_object95576:head# 2024-04-20T17:11:21.557 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.557 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7988e54f:::benchmark_data_smithi171_78918_object157547:head# 2024-04-20T17:11:21.557 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.557 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7988fd9c:::benchmark_data_smithi171_78918_object151212:head# 2024-04-20T17:11:21.557 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.557 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:798914fd:::benchmark_data_smithi171_78918_object56479:head# 2024-04-20T17:11:21.557 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.557 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79899530:::benchmark_data_smithi171_78918_object116471:head# 2024-04-20T17:11:21.557 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.557 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7989e2b8:::benchmark_data_smithi171_78918_object122168:head# 2024-04-20T17:11:21.557 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.557 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:798a48ea:::benchmark_data_smithi171_78918_object81686:head# 2024-04-20T17:11:21.557 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.557 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:798c2721:::benchmark_data_smithi171_78918_object138318:head# 2024-04-20T17:11:21.558 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.558 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:798c5ae7:::benchmark_data_smithi171_78918_object105056:head# 2024-04-20T17:11:21.558 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.558 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:798d06de:::benchmark_data_smithi171_78918_object105492:head# 2024-04-20T17:11:21.558 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.558 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:798d30e5:::benchmark_data_smithi171_78918_object141738:head# 2024-04-20T17:11:21.558 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.558 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:798ef7ac:::benchmark_data_smithi171_78918_object148792:head# 2024-04-20T17:11:21.558 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.558 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79908528:::benchmark_data_smithi171_78918_object145096:head# 2024-04-20T17:11:21.558 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.558 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79918554:::benchmark_data_smithi171_78918_object131930:head# 2024-04-20T17:11:21.558 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.558 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79919737:::benchmark_data_smithi171_78918_object120731:head# 2024-04-20T17:11:21.558 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.559 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7991a042:::benchmark_data_smithi171_78918_object81796:head# 2024-04-20T17:11:21.559 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.559 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79927237:::benchmark_data_smithi171_78918_object125673:head# 2024-04-20T17:11:21.559 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.559 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7992e201:::benchmark_data_smithi171_78918_object59358:head# 2024-04-20T17:11:21.559 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.559 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7992f7da:::benchmark_data_smithi171_78918_object145634:head# 2024-04-20T17:11:21.559 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.559 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79939ea7:::benchmark_data_smithi171_78918_object57246:head# 2024-04-20T17:11:21.559 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.559 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79940a71:::benchmark_data_smithi171_78918_object116829:head# 2024-04-20T17:11:21.559 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.559 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79947113:::benchmark_data_smithi171_78918_object117731:head# 2024-04-20T17:11:21.559 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.559 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79950d42:::benchmark_data_smithi171_78918_object114268:head# 2024-04-20T17:11:21.560 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.560 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:799522ca:::benchmark_data_smithi171_78918_object160811:head# 2024-04-20T17:11:21.560 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.560 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79975b2c:::benchmark_data_smithi171_78918_object54088:head# 2024-04-20T17:11:21.560 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.560 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79977d6a:::benchmark_data_smithi171_78918_object80325:head# 2024-04-20T17:11:21.560 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.560 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7997dcaf:::benchmark_data_smithi171_78918_object58281:head# 2024-04-20T17:11:21.560 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.560 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79980cb7:::benchmark_data_smithi171_78918_object135127:head# 2024-04-20T17:11:21.560 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.560 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79981fd2:::benchmark_data_smithi171_78918_object86127:head# 2024-04-20T17:11:21.560 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.560 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79989df3:::benchmark_data_smithi171_78918_object66054:head# 2024-04-20T17:11:21.561 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.561 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7998ccfc:::benchmark_data_smithi171_78918_object141133:head# 2024-04-20T17:11:21.561 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.561 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:799a185a:::benchmark_data_smithi171_78918_object75450:head# 2024-04-20T17:11:21.561 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.561 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:799c8c55:::benchmark_data_smithi171_78918_object75713:head# 2024-04-20T17:11:21.561 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.561 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:799d2971:::benchmark_data_smithi171_78918_object130180:head# 2024-04-20T17:11:21.561 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.561 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:799d578f:::benchmark_data_smithi171_78918_object135882:head# 2024-04-20T17:11:21.561 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.561 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:799da661:::benchmark_data_smithi171_78918_object141137:head# 2024-04-20T17:11:21.561 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.561 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:799efdc7:::benchmark_data_smithi171_78918_object89710:head# 2024-04-20T17:11:21.561 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.562 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:799f9b6b:::benchmark_data_smithi171_78918_object78104:head# 2024-04-20T17:11:21.562 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.562 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79a0054e:::benchmark_data_smithi171_78918_object120069:head# 2024-04-20T17:11:21.562 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.562 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79a01253:::benchmark_data_smithi171_78918_object156843:head# 2024-04-20T17:11:21.562 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.562 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79a3fe80:::benchmark_data_smithi171_78918_object143367:head# 2024-04-20T17:11:21.562 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.562 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79a51e57:::benchmark_data_smithi171_78918_object116430:head# 2024-04-20T17:11:21.562 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.562 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79a6224e:::benchmark_data_smithi171_78918_object141419:head# 2024-04-20T17:11:21.562 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.562 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79a66437:::benchmark_data_smithi171_78918_object77552:head# 2024-04-20T17:11:21.562 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.563 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79a6b52d:::benchmark_data_smithi171_78918_object82205:head# 2024-04-20T17:11:21.563 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.563 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79a6da2a:::benchmark_data_smithi171_78918_object143901:head# 2024-04-20T17:11:21.563 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.563 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79a6f330:::benchmark_data_smithi171_78918_object64175:head# 2024-04-20T17:11:21.563 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.563 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79a6f7eb:::benchmark_data_smithi171_78918_object94544:head# 2024-04-20T17:11:21.563 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.563 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79a82037:::benchmark_data_smithi171_78918_object79073:head# 2024-04-20T17:11:21.563 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.563 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79a8306f:::benchmark_data_smithi171_78918_object157074:head# 2024-04-20T17:11:21.563 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.563 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79a8b7a9:::benchmark_data_smithi171_78918_object103204:head# 2024-04-20T17:11:21.563 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.564 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79a919d3:::benchmark_data_smithi171_78918_object133499:head# 2024-04-20T17:11:21.564 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.564 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79a94515:::benchmark_data_smithi171_78918_object136273:head# 2024-04-20T17:11:21.564 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.564 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79a98c80:::benchmark_data_smithi171_78918_object90713:head# 2024-04-20T17:11:21.564 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.564 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79aa2e7b:::benchmark_data_smithi171_78918_object160654:head# 2024-04-20T17:11:21.564 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.564 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79ada5c0:::benchmark_data_smithi171_78918_object149216:head# 2024-04-20T17:11:21.564 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.564 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79ae9149:::benchmark_data_smithi171_78918_object110787:head# 2024-04-20T17:11:21.564 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.564 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79aebd2f:::benchmark_data_smithi171_78918_object115360:head# 2024-04-20T17:11:21.564 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.565 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79aecd90:::benchmark_data_smithi171_78918_object102793:head# 2024-04-20T17:11:21.565 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.565 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79af1148:::benchmark_data_smithi171_78918_object158573:head# 2024-04-20T17:11:21.565 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.565 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79af6220:::benchmark_data_smithi171_78918_object88578:head# 2024-04-20T17:11:21.565 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.565 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79afaadd:::benchmark_data_smithi171_78918_object80184:head# 2024-04-20T17:11:21.565 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.565 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79affa3b:::benchmark_data_smithi171_78918_object82384:head# 2024-04-20T17:11:21.565 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.565 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79b0950f:::benchmark_data_smithi171_78918_object54671:head# 2024-04-20T17:11:21.565 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.565 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79b10321:::benchmark_data_smithi171_78918_object67112:head# 2024-04-20T17:11:21.565 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.565 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79b22870:::benchmark_data_smithi171_78918_object111740:head# 2024-04-20T17:11:21.566 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.566 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79b2a8af:::benchmark_data_smithi171_78918_object144307:head# 2024-04-20T17:11:21.566 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.566 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79b3591c:::benchmark_data_smithi171_78918_object146559:head# 2024-04-20T17:11:21.566 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.566 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79b59123:::benchmark_data_smithi171_78918_object111711:head# 2024-04-20T17:11:21.566 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.566 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79b5c0ad:::benchmark_data_smithi171_78918_object59505:head# 2024-04-20T17:11:21.566 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.566 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79b5ce88:::benchmark_data_smithi171_78918_object131281:head# 2024-04-20T17:11:21.566 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.566 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79b5f4e1:::benchmark_data_smithi171_78918_object108996:head# 2024-04-20T17:11:21.566 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.567 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79b62000:::benchmark_data_smithi171_78918_object122884:head# 2024-04-20T17:11:21.567 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.567 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79b65f25:::benchmark_data_smithi171_78918_object102135:head# 2024-04-20T17:11:21.567 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.567 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79b71d62:::benchmark_data_smithi171_78918_object72318:head# 2024-04-20T17:11:21.567 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.567 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79b79997:::benchmark_data_smithi171_78918_object90020:head# 2024-04-20T17:11:21.567 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.567 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79b99450:::benchmark_data_smithi171_78918_object152076:head# 2024-04-20T17:11:21.567 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.567 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79ba82e0:::benchmark_data_smithi171_78918_object70135:head# 2024-04-20T17:11:21.567 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.567 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79bade6d:::benchmark_data_smithi171_78918_object150615:head# 2024-04-20T17:11:21.567 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.567 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79bb2ec0:::benchmark_data_smithi171_78918_object123465:head# 2024-04-20T17:11:21.568 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.568 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79bbd948:::benchmark_data_smithi171_78918_object77805:head# 2024-04-20T17:11:21.568 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.568 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79bbffae:::benchmark_data_smithi171_78918_object148125:head# 2024-04-20T17:11:21.568 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.568 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79bc27b3:::benchmark_data_smithi171_78918_object157739:head# 2024-04-20T17:11:21.568 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.568 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79bd1c7c:::benchmark_data_smithi171_78918_object129703:head# 2024-04-20T17:11:21.568 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.568 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79bd5b87:::benchmark_data_smithi171_78918_object147835:head# 2024-04-20T17:11:21.568 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.568 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79bd5d68:::benchmark_data_smithi171_78918_object56939:head# 2024-04-20T17:11:21.568 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.568 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79be87f5:::benchmark_data_smithi171_78918_object81041:head# 2024-04-20T17:11:21.569 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.569 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79bed745:::benchmark_data_smithi171_78918_object133517:head# 2024-04-20T17:11:21.569 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.569 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79bf62c8:::benchmark_data_smithi171_78918_object56719:head# 2024-04-20T17:11:21.569 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.569 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79bf7715:::benchmark_data_smithi171_78918_object122447:head# 2024-04-20T17:11:21.569 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.569 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79bf7833:::benchmark_data_smithi171_78918_object144543:head# 2024-04-20T17:11:21.569 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.569 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79c1039d:::benchmark_data_smithi171_78918_object81498:head# 2024-04-20T17:11:21.569 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.569 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79c10ee9:::benchmark_data_smithi171_78918_object58984:head# 2024-04-20T17:11:21.569 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.569 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79c2350f:::benchmark_data_smithi171_78918_object70733:head# 2024-04-20T17:11:21.569 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.570 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79c26c57:::benchmark_data_smithi171_78918_object83612:head# 2024-04-20T17:11:21.570 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.570 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79c31240:::benchmark_data_smithi171_78918_object121297:head# 2024-04-20T17:11:21.570 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.570 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79c3ce22:::benchmark_data_smithi171_78918_object128174:head# 2024-04-20T17:11:21.570 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.570 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79c3d152:::benchmark_data_smithi171_78918_object98952:head# 2024-04-20T17:11:21.570 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.570 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79c40004:::benchmark_data_smithi171_78918_object98604:head# 2024-04-20T17:11:21.570 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.570 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79c5552f:::benchmark_data_smithi171_78918_object81832:head# 2024-04-20T17:11:21.570 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.570 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79c6294b:::benchmark_data_smithi171_78918_object150918:head# 2024-04-20T17:11:21.571 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.571 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79c65a35:::benchmark_data_smithi171_78918_object152400:head# 2024-04-20T17:11:21.571 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.571 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79c687e5:::benchmark_data_smithi171_78918_object103979:head# 2024-04-20T17:11:21.571 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.571 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79c88338:::benchmark_data_smithi171_78918_object124875:head# 2024-04-20T17:11:21.571 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.571 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79ca2d24:::benchmark_data_smithi171_78918_object72717:head# 2024-04-20T17:11:21.571 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.571 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79ca3968:::benchmark_data_smithi171_78918_object93812:head# 2024-04-20T17:11:21.571 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.571 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79cb32b6:::benchmark_data_smithi171_78918_object124979:head# 2024-04-20T17:11:21.571 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.571 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79cbed8c:::benchmark_data_smithi171_78918_object124019:head# 2024-04-20T17:11:21.572 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.572 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79cc36fe:::benchmark_data_smithi171_78918_object154054:head# 2024-04-20T17:11:21.572 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.572 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79cc7bc3:::benchmark_data_smithi171_78918_object143226:head# 2024-04-20T17:11:21.572 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.572 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79ce02aa:::benchmark_data_smithi171_78918_object86697:head# 2024-04-20T17:11:21.572 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.572 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79ce0966:::benchmark_data_smithi171_78918_object107898:head# 2024-04-20T17:11:21.572 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.572 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79ce6e81:::benchmark_data_smithi171_78918_object108153:head# 2024-04-20T17:11:21.572 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.572 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79cef47a:::benchmark_data_smithi171_78918_object160056:head# 2024-04-20T17:11:21.572 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.572 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79cf84fe:::benchmark_data_smithi171_78918_object142967:head# 2024-04-20T17:11:21.573 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.573 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79d018e4:::benchmark_data_smithi171_78918_object154725:head# 2024-04-20T17:11:21.573 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.573 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79d04c4a:::benchmark_data_smithi171_78918_object157388:head# 2024-04-20T17:11:21.573 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.573 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79d0593d:::benchmark_data_smithi171_78918_object121057:head# 2024-04-20T17:11:21.573 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.573 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79d0626d:::benchmark_data_smithi171_78918_object86630:head# 2024-04-20T17:11:21.573 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.573 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79d08467:::benchmark_data_smithi171_78918_object53370:head# 2024-04-20T17:11:21.573 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.573 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79d102d0:::benchmark_data_smithi171_78918_object150832:head# 2024-04-20T17:11:21.573 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.573 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79d12421:::benchmark_data_smithi171_78918_object64772:head# 2024-04-20T17:11:21.573 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.574 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79d161b5:::benchmark_data_smithi171_78918_object103103:head# 2024-04-20T17:11:21.574 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.574 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79d24f2b:::benchmark_data_smithi171_78918_object92457:head# 2024-04-20T17:11:21.574 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.574 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79d3af5d:::benchmark_data_smithi171_78918_object74776:head# 2024-04-20T17:11:21.574 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.574 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79d41b7f:::benchmark_data_smithi171_78918_object93782:head# 2024-04-20T17:11:21.574 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.574 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79d4452c:::benchmark_data_smithi171_78918_object93254:head# 2024-04-20T17:11:21.574 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.574 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79d45c1a:::benchmark_data_smithi171_78918_object57366:head# 2024-04-20T17:11:21.574 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.574 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79d49906:::benchmark_data_smithi171_78918_object95339:head# 2024-04-20T17:11:21.574 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.575 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79d692ee:::benchmark_data_smithi171_78918_object67310:head# 2024-04-20T17:11:21.575 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.575 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79d6db27:::benchmark_data_smithi171_78918_object58290:head# 2024-04-20T17:11:21.575 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.575 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79d76e12:::benchmark_data_smithi171_78918_object94164:head# 2024-04-20T17:11:21.575 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.575 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79d89689:::benchmark_data_smithi171_78918_object62384:head# 2024-04-20T17:11:21.575 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.575 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79d8db8b:::benchmark_data_smithi171_78918_object132045:head# 2024-04-20T17:11:21.575 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.575 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79d94f67:::benchmark_data_smithi171_78918_object129683:head# 2024-04-20T17:11:21.575 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.575 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79d95c6c:::benchmark_data_smithi171_78918_object108001:head# 2024-04-20T17:11:21.575 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.575 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79d98da5:::benchmark_data_smithi171_78918_object105376:head# 2024-04-20T17:11:21.576 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.576 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79d9d3f2:::benchmark_data_smithi171_78918_object143998:head# 2024-04-20T17:11:21.576 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.576 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79da019d:::benchmark_data_smithi171_78918_object58030:head# 2024-04-20T17:11:21.576 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.576 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79da8fe3:::benchmark_data_smithi171_78918_object158312:head# 2024-04-20T17:11:21.576 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.576 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79dd45b2:::benchmark_data_smithi171_78918_object70149:head# 2024-04-20T17:11:21.576 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.576 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79dd843f:::benchmark_data_smithi171_78918_object88444:head# 2024-04-20T17:11:21.576 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.576 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79dec167:::benchmark_data_smithi171_78918_object87081:head# 2024-04-20T17:11:21.576 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.576 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79df19f6:::benchmark_data_smithi171_78918_object120254:head# 2024-04-20T17:11:21.577 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.577 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79df3dff:::benchmark_data_smithi171_78918_object109598:head# 2024-04-20T17:11:21.577 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.577 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79dfc33e:::benchmark_data_smithi171_78918_object158357:head# 2024-04-20T17:11:21.577 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.577 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79e00df7:::benchmark_data_smithi171_78918_object155323:head# 2024-04-20T17:11:21.577 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.577 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79e015b7:::benchmark_data_smithi171_78918_object103580:head# 2024-04-20T17:11:21.577 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.577 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79e2d17e:::benchmark_data_smithi171_78918_object159896:head# 2024-04-20T17:11:21.577 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.577 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79e3ae10:::benchmark_data_smithi171_78918_object52761:head# 2024-04-20T17:11:21.577 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.577 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79e47ea1:::benchmark_data_smithi171_78918_object145156:head# 2024-04-20T17:11:21.577 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.578 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79e4e8da:::benchmark_data_smithi171_78918_object56008:head# 2024-04-20T17:11:21.578 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.578 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79e51876:::benchmark_data_smithi171_78918_object160062:head# 2024-04-20T17:11:21.578 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.578 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79e74298:::benchmark_data_smithi171_78918_object119749:head# 2024-04-20T17:11:21.578 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.578 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79e75d8f:::benchmark_data_smithi171_78918_object95261:head# 2024-04-20T17:11:21.578 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.578 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79e849c4:::benchmark_data_smithi171_78918_object156874:head# 2024-04-20T17:11:21.578 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.578 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79e870dd:::benchmark_data_smithi171_78918_object124598:head# 2024-04-20T17:11:21.578 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.578 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79e8d034:::benchmark_data_smithi171_78918_object70266:head# 2024-04-20T17:11:21.578 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.579 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79e97be8:::benchmark_data_smithi171_78918_object94899:head# 2024-04-20T17:11:21.579 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.579 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79e989f8:::benchmark_data_smithi171_78918_object114944:head# 2024-04-20T17:11:21.579 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.579 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79ea4388:::benchmark_data_smithi171_78918_object118545:head# 2024-04-20T17:11:21.579 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.579 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79eaf098:::benchmark_data_smithi171_78918_object139484:head# 2024-04-20T17:11:21.579 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.579 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79ebaa20:::benchmark_data_smithi171_78918_object131481:head# 2024-04-20T17:11:21.579 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.579 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79eccca1:::benchmark_data_smithi171_78918_object82103:head# 2024-04-20T17:11:21.579 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.579 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79ed2623:::benchmark_data_smithi171_78918_object100624:head# 2024-04-20T17:11:21.579 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.579 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79ed4965:::benchmark_data_smithi171_78918_object52674:head# 2024-04-20T17:11:21.580 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.580 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79ed8303:::benchmark_data_smithi171_78918_object76859:head# 2024-04-20T17:11:21.580 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.580 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79ed9e0f:::benchmark_data_smithi171_78918_object111122:head# 2024-04-20T17:11:21.580 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.580 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79edacf3:::benchmark_data_smithi171_78918_object149586:head# 2024-04-20T17:11:21.580 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.580 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79ee18d6:::benchmark_data_smithi171_78918_object115060:head# 2024-04-20T17:11:21.580 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.580 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79ef0fed:::benchmark_data_smithi171_78918_object88914:head# 2024-04-20T17:11:21.580 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.580 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79ef1cb6:::benchmark_data_smithi171_78918_object104260:head# 2024-04-20T17:11:21.580 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.580 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79ef3ca6:::benchmark_data_smithi171_78918_object152973:head# 2024-04-20T17:11:21.581 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.581 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79efd519:::benchmark_data_smithi171_78918_object73551:head# 2024-04-20T17:11:21.581 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.581 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79f04790:::benchmark_data_smithi171_78918_object121030:head# 2024-04-20T17:11:21.581 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.581 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79f0a7ac:::benchmark_data_smithi171_78918_object116428:head# 2024-04-20T17:11:21.581 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.581 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79f0fb61:::benchmark_data_smithi171_78918_object140064:head# 2024-04-20T17:11:21.581 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.581 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79f13a4b:::benchmark_data_smithi171_78918_object120616:head# 2024-04-20T17:11:21.581 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.581 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79f2664d:::benchmark_data_smithi171_78918_object158765:head# 2024-04-20T17:11:21.581 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.581 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79f28409:::benchmark_data_smithi171_78918_object132205:head# 2024-04-20T17:11:21.582 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.582 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79f447a5:::benchmark_data_smithi171_78918_object159484:head# 2024-04-20T17:11:21.582 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.582 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79f45b25:::benchmark_data_smithi171_78918_object156885:head# 2024-04-20T17:11:21.582 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.582 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79f465fd:::benchmark_data_smithi171_78918_object125894:head# 2024-04-20T17:11:21.582 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.582 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79f46b11:::benchmark_data_smithi171_78918_object87436:head# 2024-04-20T17:11:21.582 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.582 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79f4cff7:::benchmark_data_smithi171_78918_object156913:head# 2024-04-20T17:11:21.582 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.582 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79f5d782:::benchmark_data_smithi171_78918_object147024:head# 2024-04-20T17:11:21.582 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.582 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79f6352f:::benchmark_data_smithi171_78918_object91863:head# 2024-04-20T17:11:21.583 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.583 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79f7f347:::benchmark_data_smithi171_78918_object122267:head# 2024-04-20T17:11:21.583 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.583 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79f8f4d3:::benchmark_data_smithi171_78918_object113455:head# 2024-04-20T17:11:21.583 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.583 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79f97775:::benchmark_data_smithi171_78918_object160561:head# 2024-04-20T17:11:21.583 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.583 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79f97a2f:::benchmark_data_smithi171_78918_object129426:head# 2024-04-20T17:11:21.583 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.583 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79fadb2b:::benchmark_data_smithi171_78918_object76485:head# 2024-04-20T17:11:21.583 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.583 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79fbee36:::benchmark_data_smithi171_78918_object157651:head# 2024-04-20T17:11:21.583 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.583 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79fc71f3:::benchmark_data_smithi171_78918_object56713:head# 2024-04-20T17:11:21.584 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.584 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79fceef6:::benchmark_data_smithi171_78918_object137738:head# 2024-04-20T17:11:21.584 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.584 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79fd7bbc:::benchmark_data_smithi171_78918_object146562:head# 2024-04-20T17:11:21.584 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.584 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79fe63bb:::benchmark_data_smithi171_78918_object112804:head# 2024-04-20T17:11:21.584 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.584 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79fef87c:::benchmark_data_smithi171_78918_object83211:head# 2024-04-20T17:11:21.584 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.584 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79ff772e:::benchmark_data_smithi171_78918_object107851:head# 2024-04-20T17:11:21.584 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.584 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:79fff31e:::benchmark_data_smithi171_78918_object124813:head# 2024-04-20T17:11:21.584 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.584 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a00aa85:::benchmark_data_smithi171_78918_object117993:head# 2024-04-20T17:11:21.585 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.585 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a01026f:::benchmark_data_smithi171_78918_object141238:head# 2024-04-20T17:11:21.585 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.585 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a017d76:::benchmark_data_smithi171_78918_object103060:head# 2024-04-20T17:11:21.585 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.585 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a024da8:::benchmark_data_smithi171_78918_object118585:head# 2024-04-20T17:11:21.585 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.585 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a02f633:::benchmark_data_smithi171_78918_object114876:head# 2024-04-20T17:11:21.585 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.585 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a03f476:::benchmark_data_smithi171_78918_object140668:head# 2024-04-20T17:11:21.585 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.585 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a042b1b:::benchmark_data_smithi171_78918_object87800:head# 2024-04-20T17:11:21.585 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.585 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a04de71:::benchmark_data_smithi171_78918_object56182:head# 2024-04-20T17:11:21.585 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.585 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a067298:::benchmark_data_smithi171_78918_object75152:head# 2024-04-20T17:11:21.586 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.586 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a0685c4:::benchmark_data_smithi171_78918_object73809:head# 2024-04-20T17:11:21.586 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.586 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a09ce2a:::benchmark_data_smithi171_78918_object121323:head# 2024-04-20T17:11:21.586 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.586 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a0a866d:::benchmark_data_smithi171_78918_object54774:head# 2024-04-20T17:11:21.586 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.586 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a0b8ac2:::benchmark_data_smithi171_78918_object68878:head# 2024-04-20T17:11:21.586 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.586 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a0d956b:::benchmark_data_smithi171_78918_object84024:head# 2024-04-20T17:11:21.586 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.586 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a0db227:::benchmark_data_smithi171_78918_object126170:head# 2024-04-20T17:11:21.586 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.586 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a0f7c74:::benchmark_data_smithi171_78918_object133784:head# 2024-04-20T17:11:21.587 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.587 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a0f83a7:::benchmark_data_smithi171_78918_object76508:head# 2024-04-20T17:11:21.587 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.587 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a1274c7:::benchmark_data_smithi171_78918_object135682:head# 2024-04-20T17:11:21.587 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.587 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a12d091:::benchmark_data_smithi171_78918_object69980:head# 2024-04-20T17:11:21.587 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.587 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a1316dc:::benchmark_data_smithi171_78918_object106948:head# 2024-04-20T17:11:21.587 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.587 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a1343db:::benchmark_data_smithi171_78918_object52175:head# 2024-04-20T17:11:21.587 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.587 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a13490a:::benchmark_data_smithi171_78918_object82145:head# 2024-04-20T17:11:21.587 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.587 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a138ede:::benchmark_data_smithi171_78918_object59243:head# 2024-04-20T17:11:21.588 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.588 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a13a75a:::benchmark_data_smithi171_78918_object152208:head# 2024-04-20T17:11:21.588 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.588 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a145329:::benchmark_data_smithi171_78918_object74135:head# 2024-04-20T17:11:21.588 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.588 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a14c82f:::benchmark_data_smithi171_78918_object135347:head# 2024-04-20T17:11:21.588 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.588 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a1560f1:::benchmark_data_smithi171_78918_object53236:head# 2024-04-20T17:11:21.588 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.588 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a1578a4:::benchmark_data_smithi171_78918_object76262:head# 2024-04-20T17:11:21.588 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.588 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a16b4fc:::benchmark_data_smithi171_78918_object59616:head# 2024-04-20T17:11:21.588 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.588 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a18ba5e:::benchmark_data_smithi171_78918_object141264:head# 2024-04-20T17:11:21.588 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.588 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a18f67c:::benchmark_data_smithi171_78918_object65847:head# 2024-04-20T17:11:21.589 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.589 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a18fb80:::benchmark_data_smithi171_78918_object104095:head# 2024-04-20T17:11:21.589 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.589 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a19a054:::benchmark_data_smithi171_78918_object97236:head# 2024-04-20T17:11:21.589 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.589 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a19e172:::benchmark_data_smithi171_78918_object157831:head# 2024-04-20T17:11:21.589 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.589 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a1a2803:::benchmark_data_smithi171_78918_object90034:head# 2024-04-20T17:11:21.589 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.589 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a1a6b2b:::benchmark_data_smithi171_78918_object128837:head# 2024-04-20T17:11:21.589 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.589 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a1b0871:::benchmark_data_smithi171_78918_object88550:head# 2024-04-20T17:11:21.589 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.589 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a1b3739:::benchmark_data_smithi171_78918_object134512:head# 2024-04-20T17:11:21.589 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.590 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a1c2655:::benchmark_data_smithi171_78918_object61814:head# 2024-04-20T17:11:21.590 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.590 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a1ccdfe:::benchmark_data_smithi171_78918_object120806:head# 2024-04-20T17:11:21.590 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.590 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a1cf5a4:::benchmark_data_smithi171_78918_object98256:head# 2024-04-20T17:11:21.590 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.590 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a1d80fc:::benchmark_data_smithi171_78918_object97319:head# 2024-04-20T17:11:21.590 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.590 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a1dbd3e:::benchmark_data_smithi171_78918_object74071:head# 2024-04-20T17:11:21.590 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.590 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a1e4f2a:::benchmark_data_smithi171_78918_object125568:head# 2024-04-20T17:11:21.590 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.590 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a1f91c6:::benchmark_data_smithi171_78918_object124369:head# 2024-04-20T17:11:21.590 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.590 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a1fffce:::benchmark_data_smithi171_78918_object143644:head# 2024-04-20T17:11:21.591 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.591 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a2007fb:::benchmark_data_smithi171_78918_object157354:head# 2024-04-20T17:11:21.591 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.591 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a201b8e:::benchmark_data_smithi171_78918_object90268:head# 2024-04-20T17:11:21.591 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.591 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a2082f3:::benchmark_data_smithi171_78918_object55348:head# 2024-04-20T17:11:21.591 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.591 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a213807:::benchmark_data_smithi171_78918_object150935:head# 2024-04-20T17:11:21.591 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.591 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a22b3e8:::benchmark_data_smithi171_78918_object155430:head# 2024-04-20T17:11:21.591 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.591 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a23a335:::benchmark_data_smithi171_78918_object114719:head# 2024-04-20T17:11:21.591 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.591 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a23dcbe:::benchmark_data_smithi171_78918_object118344:head# 2024-04-20T17:11:21.591 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.592 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a248def:::benchmark_data_smithi171_78918_object158876:head# 2024-04-20T17:11:21.592 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.592 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a263335:::benchmark_data_smithi171_78918_object66310:head# 2024-04-20T17:11:21.592 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.592 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a268ee2:::benchmark_data_smithi171_78918_object148150:head# 2024-04-20T17:11:21.592 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.592 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a278592:::benchmark_data_smithi171_78918_object118602:head# 2024-04-20T17:11:21.592 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.592 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a27fce5:::benchmark_data_smithi171_78918_object113068:head# 2024-04-20T17:11:21.592 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.592 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a2820fe:::benchmark_data_smithi171_78918_object62575:head# 2024-04-20T17:11:21.592 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.592 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a290220:::benchmark_data_smithi171_78918_object75451:head# 2024-04-20T17:11:21.592 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.592 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a2a26bc:::benchmark_data_smithi171_78918_object140697:head# 2024-04-20T17:11:21.593 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.593 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a2a37e9:::benchmark_data_smithi171_78918_object118036:head# 2024-04-20T17:11:21.593 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.593 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a2abd31:::benchmark_data_smithi171_78918_object148752:head# 2024-04-20T17:11:21.593 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.593 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a2bae2c:::benchmark_data_smithi171_78918_object63742:head# 2024-04-20T17:11:21.593 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.593 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a2cf599:::benchmark_data_smithi171_78918_object99230:head# 2024-04-20T17:11:21.593 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.593 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a2e6ae6:::benchmark_data_smithi171_78918_object117352:head# 2024-04-20T17:11:21.593 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.593 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a2e7e29:::benchmark_data_smithi171_78918_object83606:head# 2024-04-20T17:11:21.593 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.593 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a2eeab6:::benchmark_data_smithi171_78918_object85235:head# 2024-04-20T17:11:21.593 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.594 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a2ef06b:::benchmark_data_smithi171_78918_object81645:head# 2024-04-20T17:11:21.594 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.594 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a2fe154:::benchmark_data_smithi171_78918_object105205:head# 2024-04-20T17:11:21.594 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.594 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a306219:::benchmark_data_smithi171_78918_object73057:head# 2024-04-20T17:11:21.594 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.594 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a32a9f7:::benchmark_data_smithi171_78918_object73326:head# 2024-04-20T17:11:21.594 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.594 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a332315:::benchmark_data_smithi171_78918_object151241:head# 2024-04-20T17:11:21.594 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.594 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a34e4ce:::benchmark_data_smithi171_78918_object96197:head# 2024-04-20T17:11:21.594 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.594 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a35a6d5:::benchmark_data_smithi171_78918_object65308:head# 2024-04-20T17:11:21.594 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.595 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a372337:::benchmark_data_smithi171_78918_object153251:head# 2024-04-20T17:11:21.595 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.595 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a3738ca:::benchmark_data_smithi171_78918_object71041:head# 2024-04-20T17:11:21.595 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.595 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a373a7d:::benchmark_data_smithi171_78918_object139042:head# 2024-04-20T17:11:21.595 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.595 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a379fd0:::benchmark_data_smithi171_78918_object143771:head# 2024-04-20T17:11:21.595 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.595 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a37af1e:::benchmark_data_smithi171_78918_object155812:head# 2024-04-20T17:11:21.595 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.595 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a37b408:::benchmark_data_smithi171_78918_object119617:head# 2024-04-20T17:11:21.595 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.595 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a380704:::benchmark_data_smithi171_78918_object81429:head# 2024-04-20T17:11:21.595 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.595 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a38dd1f:::benchmark_data_smithi171_78918_object92321:head# 2024-04-20T17:11:21.596 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.596 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a394757:::benchmark_data_smithi171_78918_object55877:head# 2024-04-20T17:11:21.596 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.596 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a39a3f7:::benchmark_data_smithi171_78918_object85374:head# 2024-04-20T17:11:21.596 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.596 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a39f134:::benchmark_data_smithi171_78918_object154908:head# 2024-04-20T17:11:21.596 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.596 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a3d55bd:::benchmark_data_smithi171_78918_object137206:head# 2024-04-20T17:11:21.596 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.596 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a3ec32a:::benchmark_data_smithi171_78918_object63744:head# 2024-04-20T17:11:21.596 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.596 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a3f1a77:::benchmark_data_smithi171_78918_object138990:head# 2024-04-20T17:11:21.596 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.596 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a3f2591:::benchmark_data_smithi171_78918_object82372:head# 2024-04-20T17:11:21.596 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.597 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a400b19:::benchmark_data_smithi171_78918_object83147:head# 2024-04-20T17:11:21.597 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.597 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a404279:::benchmark_data_smithi171_78918_object104935:head# 2024-04-20T17:11:21.597 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.597 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a413b55:::benchmark_data_smithi171_78918_object154964:head# 2024-04-20T17:11:21.597 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.597 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a41c5ac:::benchmark_data_smithi171_78918_object67864:head# 2024-04-20T17:11:21.597 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.597 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a41cfd8:::benchmark_data_smithi171_78918_object105441:head# 2024-04-20T17:11:21.597 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.597 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a41ed2b:::benchmark_data_smithi171_78918_object58587:head# 2024-04-20T17:11:21.597 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.597 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a431147:::benchmark_data_smithi171_78918_object117436:head# 2024-04-20T17:11:21.597 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.597 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a433370:::benchmark_data_smithi171_78918_object124949:head# 2024-04-20T17:11:21.598 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.598 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a43c878:::benchmark_data_smithi171_78918_object129843:head# 2024-04-20T17:11:21.598 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.598 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a43db4d:::benchmark_data_smithi171_78918_object68014:head# 2024-04-20T17:11:21.598 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.598 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a447806:::benchmark_data_smithi171_78918_object91504:head# 2024-04-20T17:11:21.598 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.598 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a44fdb6:::benchmark_data_smithi171_78918_object147383:head# 2024-04-20T17:11:21.598 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.598 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a4622a2:::benchmark_data_smithi171_78918_object85099:head# 2024-04-20T17:11:21.598 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.598 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a4715a0:::benchmark_data_smithi171_78918_object67795:head# 2024-04-20T17:11:21.598 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.598 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a47c8f2:::benchmark_data_smithi171_78918_object68370:head# 2024-04-20T17:11:21.598 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.599 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a47e564:::benchmark_data_smithi171_78918_object94241:head# 2024-04-20T17:11:21.599 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.599 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a480f9f:::benchmark_data_smithi171_78918_object59017:head# 2024-04-20T17:11:21.599 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.599 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a49a60a:::benchmark_data_smithi171_78918_object117756:head# 2024-04-20T17:11:21.599 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.599 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a4adc2b:::benchmark_data_smithi171_78918_object69048:head# 2024-04-20T17:11:21.599 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.599 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a4b8a32:::benchmark_data_smithi171_78918_object62543:head# 2024-04-20T17:11:21.599 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.599 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a4bff46:::benchmark_data_smithi171_78918_object144903:head# 2024-04-20T17:11:21.599 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.599 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a4c7dc4:::benchmark_data_smithi171_78918_object53418:head# 2024-04-20T17:11:21.599 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.600 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a4e3d65:::benchmark_data_smithi171_78918_object62736:head# 2024-04-20T17:11:21.600 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.600 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a4edbba:::benchmark_data_smithi171_78918_object133247:head# 2024-04-20T17:11:21.600 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.600 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a4f1906:::benchmark_data_smithi171_78918_object158478:head# 2024-04-20T17:11:21.600 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.600 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a4f3cf2:::benchmark_data_smithi171_78918_object97581:head# 2024-04-20T17:11:21.600 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.600 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a4fe739:::benchmark_data_smithi171_78918_object139017:head# 2024-04-20T17:11:21.600 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.600 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a507f94:::benchmark_data_smithi171_78918_object62737:head# 2024-04-20T17:11:21.600 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.600 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a5099d2:::benchmark_data_smithi171_78918_object151312:head# 2024-04-20T17:11:21.600 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.600 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a50e71f:::benchmark_data_smithi171_78918_object99918:head# 2024-04-20T17:11:21.601 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.601 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a51097f:::benchmark_data_smithi171_78918_object124078:head# 2024-04-20T17:11:21.601 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.601 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a51126b:::benchmark_data_smithi171_78918_object77361:head# 2024-04-20T17:11:21.601 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.601 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a514473:::benchmark_data_smithi171_78918_object138411:head# 2024-04-20T17:11:21.601 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.601 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a52687f:::benchmark_data_smithi171_78918_object110049:head# 2024-04-20T17:11:21.601 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.601 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a5287c7:::benchmark_data_smithi171_78918_object75308:head# 2024-04-20T17:11:21.601 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.601 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a52fe90:::benchmark_data_smithi171_78918_object154701:head# 2024-04-20T17:11:21.601 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.601 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a53380b:::benchmark_data_smithi171_78918_object51547:head# 2024-04-20T17:11:21.601 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.602 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a53d651:::benchmark_data_smithi171_78918_object57875:head# 2024-04-20T17:11:21.602 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.602 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a542bf3:::benchmark_data_smithi171_78918_object85004:head# 2024-04-20T17:11:21.602 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.602 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a54abbf:::benchmark_data_smithi171_78918_object137436:head# 2024-04-20T17:11:21.602 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.602 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a54d71a:::benchmark_data_smithi171_78918_object97582:head# 2024-04-20T17:11:21.602 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.602 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a54e9ee:::benchmark_data_smithi171_78918_object52256:head# 2024-04-20T17:11:21.602 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.602 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a551dcd:::benchmark_data_smithi171_78918_object141841:head# 2024-04-20T17:11:21.602 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.602 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a58599b:::benchmark_data_smithi171_78918_object115785:head# 2024-04-20T17:11:21.602 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.602 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a58c730:::benchmark_data_smithi171_78918_object63354:head# 2024-04-20T17:11:21.603 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.603 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a58d855:::benchmark_data_smithi171_78918_object109170:head# 2024-04-20T17:11:21.603 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.603 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a597d97:::benchmark_data_smithi171_78918_object58596:head# 2024-04-20T17:11:21.603 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.603 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a5a00c7:::benchmark_data_smithi171_78918_object134779:head# 2024-04-20T17:11:21.603 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.603 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a5a8b57:::benchmark_data_smithi171_78918_object102509:head# 2024-04-20T17:11:21.603 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.603 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a5b63e0:::benchmark_data_smithi171_78918_object148464:head# 2024-04-20T17:11:21.603 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.603 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a5b8b0e:::benchmark_data_smithi171_78918_object144534:head# 2024-04-20T17:11:21.603 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.603 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a5c3a3a:::benchmark_data_smithi171_78918_object118220:head# 2024-04-20T17:11:21.603 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.604 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a5e225f:::benchmark_data_smithi171_78918_object61349:head# 2024-04-20T17:11:21.604 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.604 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a5e3d4d:::benchmark_data_smithi171_78918_object84264:head# 2024-04-20T17:11:21.604 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.604 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a5e7199:::benchmark_data_smithi171_78918_object80205:head# 2024-04-20T17:11:21.604 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.604 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a5e7f07:::benchmark_data_smithi171_78918_object54633:head# 2024-04-20T17:11:21.604 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.604 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a5e9268:::benchmark_data_smithi171_78918_object145963:head# 2024-04-20T17:11:21.604 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.604 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a5f2d82:::benchmark_data_smithi171_78918_object72999:head# 2024-04-20T17:11:21.604 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.604 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a5fbab9:::benchmark_data_smithi171_78918_object111927:head# 2024-04-20T17:11:21.604 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.605 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a623299:::benchmark_data_smithi171_78918_object114885:head# 2024-04-20T17:11:21.605 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.605 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a637f81:::benchmark_data_smithi171_78918_object53367:head# 2024-04-20T17:11:21.605 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.605 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a64d791:::benchmark_data_smithi171_78918_object120482:head# 2024-04-20T17:11:21.605 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.605 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a64e737:::benchmark_data_smithi171_78918_object136224:head# 2024-04-20T17:11:21.605 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.605 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a64f7c3:::benchmark_data_smithi171_78918_object79826:head# 2024-04-20T17:11:21.605 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.605 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a657784:::benchmark_data_smithi171_78918_object123971:head# 2024-04-20T17:11:21.605 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.605 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a658ba5:::benchmark_data_smithi171_78918_object157875:head# 2024-04-20T17:11:21.605 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.605 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a66b75e:::benchmark_data_smithi171_78918_object146030:head# 2024-04-20T17:11:21.606 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.606 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a678977:::benchmark_data_smithi171_78918_object100060:head# 2024-04-20T17:11:21.606 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.606 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a68f03f:::benchmark_data_smithi171_78918_object134463:head# 2024-04-20T17:11:21.606 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.606 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a6b3fc1:::benchmark_data_smithi171_78918_object150607:head# 2024-04-20T17:11:21.606 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.606 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a6bfb2f:::benchmark_data_smithi171_78918_object116177:head# 2024-04-20T17:11:21.606 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.606 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a6c6962:::benchmark_data_smithi171_78918_object103882:head# 2024-04-20T17:11:21.606 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.606 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a6d72e2:::benchmark_data_smithi171_78918_object145560:head# 2024-04-20T17:11:21.606 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.606 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a6ead15:::benchmark_data_smithi171_78918_object134969:head# 2024-04-20T17:11:21.607 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.607 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a6ec082:::benchmark_data_smithi171_78918_object120759:head# 2024-04-20T17:11:21.609 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.609 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a6fcc4c:::benchmark_data_smithi171_78918_object102650:head# 2024-04-20T17:11:21.609 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.609 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a710eec:::benchmark_data_smithi171_78918_object51724:head# 2024-04-20T17:11:21.609 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.609 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a71151a:::benchmark_data_smithi171_78918_object114041:head# 2024-04-20T17:11:21.609 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.610 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a718310:::benchmark_data_smithi171_78918_object102802:head# 2024-04-20T17:11:21.610 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.610 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a7279ca:::benchmark_data_smithi171_78918_object71486:head# 2024-04-20T17:11:21.610 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.610 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a729a26:::benchmark_data_smithi171_78918_object76748:head# 2024-04-20T17:11:21.610 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.610 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a7417e5:::benchmark_data_smithi171_78918_object60302:head# 2024-04-20T17:11:21.610 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.610 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a7418b8:::benchmark_data_smithi171_78918_object101633:head# 2024-04-20T17:11:21.610 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.610 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a748e69:::benchmark_data_smithi171_78918_object142521:head# 2024-04-20T17:11:21.610 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.610 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a74c99e:::benchmark_data_smithi171_78918_object116906:head# 2024-04-20T17:11:21.610 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.610 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a785edb:::benchmark_data_smithi171_78918_object77749:head# 2024-04-20T17:11:21.611 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.611 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a7a5ab7:::benchmark_data_smithi171_78918_object89767:head# 2024-04-20T17:11:21.611 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.611 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a7a9675:::benchmark_data_smithi171_78918_object113701:head# 2024-04-20T17:11:21.611 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.611 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a7ab3ff:::benchmark_data_smithi171_78918_object97456:head# 2024-04-20T17:11:21.611 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.611 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a7b02d1:::benchmark_data_smithi171_78918_object155383:head# 2024-04-20T17:11:21.611 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.611 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a7be279:::benchmark_data_smithi171_78918_object98809:head# 2024-04-20T17:11:21.611 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.611 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a7cb4f4:::benchmark_data_smithi171_78918_object111861:head# 2024-04-20T17:11:21.611 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.611 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a7d2954:::benchmark_data_smithi171_78918_object72126:head# 2024-04-20T17:11:21.612 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.612 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a7d69bb:::benchmark_data_smithi171_78918_object94442:head# 2024-04-20T17:11:21.612 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.612 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a7d7708:::benchmark_data_smithi171_78918_object130618:head# 2024-04-20T17:11:21.612 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.612 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a7e2ab1:::benchmark_data_smithi171_78918_object79674:head# 2024-04-20T17:11:21.612 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.612 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a7e4124:::benchmark_data_smithi171_78918_object93341:head# 2024-04-20T17:11:21.612 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.612 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a801f58:::benchmark_data_smithi171_78918_object65509:head# 2024-04-20T17:11:21.612 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.612 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a810958:::benchmark_data_smithi171_78918_object69056:head# 2024-04-20T17:11:21.612 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.612 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a81f4a6:::benchmark_data_smithi171_78918_object109106:head# 2024-04-20T17:11:21.612 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.613 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a82bb25:::benchmark_data_smithi171_78918_object127046:head# 2024-04-20T17:11:21.613 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.613 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a82f213:::benchmark_data_smithi171_78918_object106394:head# 2024-04-20T17:11:21.613 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.613 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a83c1ce:::benchmark_data_smithi171_78918_object77116:head# 2024-04-20T17:11:21.613 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.613 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a8402d7:::benchmark_data_smithi171_78918_object96287:head# 2024-04-20T17:11:21.613 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.613 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a84173c:::benchmark_data_smithi171_78918_object92512:head# 2024-04-20T17:11:21.613 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.613 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a84527c:::benchmark_data_smithi171_78918_object149016:head# 2024-04-20T17:11:21.613 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.613 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a85c2c7:::benchmark_data_smithi171_78918_object100589:head# 2024-04-20T17:11:21.613 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.614 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a85c338:::benchmark_data_smithi171_78918_object74034:head# 2024-04-20T17:11:21.614 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.614 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a860aa0:::benchmark_data_smithi171_78918_object109278:head# 2024-04-20T17:11:21.614 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.614 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a866a9e:::benchmark_data_smithi171_78918_object61756:head# 2024-04-20T17:11:21.614 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.614 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a87f270:::benchmark_data_smithi171_78918_object76539:head# 2024-04-20T17:11:21.614 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.614 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a8823e9:::benchmark_data_smithi171_78918_object112084:head# 2024-04-20T17:11:21.614 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.614 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a88447f:::benchmark_data_smithi171_78918_object94098:head# 2024-04-20T17:11:21.614 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.614 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a89836f:::benchmark_data_smithi171_78918_object130014:head# 2024-04-20T17:11:21.614 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.615 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a8a0c4e:::benchmark_data_smithi171_78918_object98405:head# 2024-04-20T17:11:21.615 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.615 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a8a1b80:::benchmark_data_smithi171_78918_object69542:head# 2024-04-20T17:11:21.615 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.615 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a8a3c4b:::benchmark_data_smithi171_78918_object65598:head# 2024-04-20T17:11:21.615 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.615 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a8b2539:::benchmark_data_smithi171_78918_object144641:head# 2024-04-20T17:11:21.615 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.615 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a8b333d:::benchmark_data_smithi171_78918_object94246:head# 2024-04-20T17:11:21.615 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.615 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a8c0b57:::benchmark_data_smithi171_78918_object144166:head# 2024-04-20T17:11:21.615 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.615 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a8c8ee0:::benchmark_data_smithi171_78918_object108897:head# 2024-04-20T17:11:21.615 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.616 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a8cafe2:::benchmark_data_smithi171_78918_object149610:head# 2024-04-20T17:11:21.616 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.616 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a8d437c:::benchmark_data_smithi171_78918_object60386:head# 2024-04-20T17:11:21.616 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.616 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a8d4b65:::benchmark_data_smithi171_78918_object101734:head# 2024-04-20T17:11:21.616 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.616 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a8dd24e:::benchmark_data_smithi171_78918_object148429:head# 2024-04-20T17:11:21.616 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.616 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a8f4af5:::benchmark_data_smithi171_78918_object69704:head# 2024-04-20T17:11:21.616 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.616 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a8faef1:::benchmark_data_smithi171_78918_object148408:head# 2024-04-20T17:11:21.616 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.616 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a8fc73e:::benchmark_data_smithi171_78918_object77961:head# 2024-04-20T17:11:21.616 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.617 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a900dc7:::benchmark_data_smithi171_78918_object104442:head# 2024-04-20T17:11:21.617 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.617 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a901b5c:::benchmark_data_smithi171_78918_object98820:head# 2024-04-20T17:11:21.617 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.617 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a9181d2:::benchmark_data_smithi171_78918_object136694:head# 2024-04-20T17:11:21.617 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.617 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a920921:::benchmark_data_smithi171_78918_object69654:head# 2024-04-20T17:11:21.617 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.617 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a924d16:::benchmark_data_smithi171_78918_object144892:head# 2024-04-20T17:11:21.617 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.617 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a934be6:::benchmark_data_smithi171_78918_object125638:head# 2024-04-20T17:11:21.617 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.617 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a939f0a:::benchmark_data_smithi171_78918_object144970:head# 2024-04-20T17:11:21.617 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.618 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a948730:::benchmark_data_smithi171_78918_object88101:head# 2024-04-20T17:11:21.618 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.618 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a949fae:::benchmark_data_smithi171_78918_object139987:head# 2024-04-20T17:11:21.618 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.618 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a956a5d:::benchmark_data_smithi171_78918_object83303:head# 2024-04-20T17:11:21.618 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.618 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a965d5c:::benchmark_data_smithi171_78918_object106661:head# 2024-04-20T17:11:21.618 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.618 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a968113:::benchmark_data_smithi171_78918_object122600:head# 2024-04-20T17:11:21.618 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.618 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a96e129:::benchmark_data_smithi171_78918_object121819:head# 2024-04-20T17:11:21.618 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.618 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a96ea78:::benchmark_data_smithi171_78918_object94924:head# 2024-04-20T17:11:21.618 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.618 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a97054f:::benchmark_data_smithi171_78918_object63429:head# 2024-04-20T17:11:21.619 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.619 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a972daf:::benchmark_data_smithi171_78918_object157879:head# 2024-04-20T17:11:21.619 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.619 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a9880d1:::benchmark_data_smithi171_78918_object117331:head# 2024-04-20T17:11:21.619 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.619 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a98b261:::benchmark_data_smithi171_78918_object91371:head# 2024-04-20T17:11:21.619 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.619 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a997594:::benchmark_data_smithi171_78918_object109678:head# 2024-04-20T17:11:21.619 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.619 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a99c9f1:::benchmark_data_smithi171_78918_object117928:head# 2024-04-20T17:11:21.619 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.619 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a9a3d5c:::benchmark_data_smithi171_78918_object136921:head# 2024-04-20T17:11:21.619 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.619 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a9ac781:::benchmark_data_smithi171_78918_object130420:head# 2024-04-20T17:11:21.619 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.620 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a9c1aac:::benchmark_data_smithi171_78918_object137393:head# 2024-04-20T17:11:21.620 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.620 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a9daa17:::benchmark_data_smithi171_78918_object107094:head# 2024-04-20T17:11:21.620 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.620 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a9dd3d3:::benchmark_data_smithi171_78918_object117794:head# 2024-04-20T17:11:21.620 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.620 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a9dfa41:::benchmark_data_smithi171_78918_object94221:head# 2024-04-20T17:11:21.620 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.620 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a9f4440:::benchmark_data_smithi171_78918_object93932:head# 2024-04-20T17:11:21.620 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.620 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a9f707e:::benchmark_data_smithi171_78918_object139543:head# 2024-04-20T17:11:21.620 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.620 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7a9fa216:::benchmark_data_smithi171_78918_object125618:head# 2024-04-20T17:11:21.620 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.620 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aa06351:::benchmark_data_smithi171_78918_object93124:head# 2024-04-20T17:11:21.621 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.621 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aa16fc5:::benchmark_data_smithi171_78918_object80787:head# 2024-04-20T17:11:21.621 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.621 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aa1b2f5:::benchmark_data_smithi171_78918_object91716:head# 2024-04-20T17:11:21.621 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.621 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aa1ee7e:::benchmark_data_smithi171_78918_object89331:head# 2024-04-20T17:11:21.621 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.621 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aa1ef5a:::benchmark_data_smithi171_78918_object120973:head# 2024-04-20T17:11:21.621 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.621 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aa49a58:::benchmark_data_smithi171_78918_object93565:head# 2024-04-20T17:11:21.621 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.621 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aa56b60:::benchmark_data_smithi171_78918_object96546:head# 2024-04-20T17:11:21.621 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.621 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aa577cf:::benchmark_data_smithi171_78918_object129392:head# 2024-04-20T17:11:21.622 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.622 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aa6849f:::benchmark_data_smithi171_78918_object121019:head# 2024-04-20T17:11:21.622 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.622 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aa6d159:::benchmark_data_smithi171_78918_object115691:head# 2024-04-20T17:11:21.622 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.622 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aa6ddbc:::benchmark_data_smithi171_78918_object116991:head# 2024-04-20T17:11:21.622 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.622 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aa876ff:::benchmark_data_smithi171_78918_object82438:head# 2024-04-20T17:11:21.622 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.622 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aa89b23:::benchmark_data_smithi171_78918_object149592:head# 2024-04-20T17:11:21.622 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.622 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aa8b110:::benchmark_data_smithi171_78918_object53427:head# 2024-04-20T17:11:21.622 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.622 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aa91585:::benchmark_data_smithi171_78918_object54916:head# 2024-04-20T17:11:21.623 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.623 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aa9f5e4:::benchmark_data_smithi171_78918_object93793:head# 2024-04-20T17:11:21.623 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.623 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aaa09a2:::benchmark_data_smithi171_78918_object65975:head# 2024-04-20T17:11:21.623 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.623 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aab6aa8:::benchmark_data_smithi171_78918_object128666:head# 2024-04-20T17:11:21.623 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.623 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aab8f67:::benchmark_data_smithi171_78918_object94482:head# 2024-04-20T17:11:21.623 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.623 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aac1a66:::benchmark_data_smithi171_78918_object107525:head# 2024-04-20T17:11:21.623 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.623 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aac44f6:::benchmark_data_smithi171_78918_object108293:head# 2024-04-20T17:11:21.623 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.623 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aad619c:::benchmark_data_smithi171_78918_object104480:head# 2024-04-20T17:11:21.624 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.624 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aadb1af:::benchmark_data_smithi171_78918_object71823:head# 2024-04-20T17:11:21.624 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.624 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aadd6b6:::benchmark_data_smithi171_78918_object155449:head# 2024-04-20T17:11:21.624 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.624 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aae1491:::benchmark_data_smithi171_78918_object80247:head# 2024-04-20T17:11:21.624 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.624 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aaee729:::benchmark_data_smithi171_78918_object118011:head# 2024-04-20T17:11:21.624 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.624 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aaf0952:::benchmark_data_smithi171_78918_object96349:head# 2024-04-20T17:11:21.624 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.624 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aaf32b0:::benchmark_data_smithi171_78918_object158661:head# 2024-04-20T17:11:21.624 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.624 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ab05290:::benchmark_data_smithi171_78918_object72194:head# 2024-04-20T17:11:21.624 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.625 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ab09f20:::benchmark_data_smithi171_78918_object152191:head# 2024-04-20T17:11:21.625 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.625 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ab0b8b0:::benchmark_data_smithi171_78918_object85524:head# 2024-04-20T17:11:21.625 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.625 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ab142ea:::benchmark_data_smithi171_78918_object69320:head# 2024-04-20T17:11:21.625 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.625 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ab1d9f0:::benchmark_data_smithi171_78918_object111297:head# 2024-04-20T17:11:21.625 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.625 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ab20bc8:::benchmark_data_smithi171_78918_object121930:head# 2024-04-20T17:11:21.625 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.625 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ab23357:::benchmark_data_smithi171_78918_object135084:head# 2024-04-20T17:11:21.625 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.625 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ab271bc:::benchmark_data_smithi171_78918_object95482:head# 2024-04-20T17:11:21.625 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.626 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ab2af3b:::benchmark_data_smithi171_78918_object115090:head# 2024-04-20T17:11:21.626 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.626 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ab2c3cd:::benchmark_data_smithi171_78918_object107612:head# 2024-04-20T17:11:21.626 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.626 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ab3a2d7:::benchmark_data_smithi171_78918_object56454:head# 2024-04-20T17:11:21.626 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.626 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ab3e395:::benchmark_data_smithi171_78918_object156245:head# 2024-04-20T17:11:21.626 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.626 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ab48427:::benchmark_data_smithi171_78918_object158371:head# 2024-04-20T17:11:21.626 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.626 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ab48464:::benchmark_data_smithi171_78918_object149879:head# 2024-04-20T17:11:21.626 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.626 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ab5498b:::benchmark_data_smithi171_78918_object66194:head# 2024-04-20T17:11:21.626 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.626 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ab71906:::benchmark_data_smithi171_78918_object86768:head# 2024-04-20T17:11:21.627 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.627 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ab7b9c7:::benchmark_data_smithi171_78918_object69581:head# 2024-04-20T17:11:21.627 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.627 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ab89adc:::benchmark_data_smithi171_78918_object122583:head# 2024-04-20T17:11:21.627 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.627 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ab90e31:::benchmark_data_smithi171_78918_object109298:head# 2024-04-20T17:11:21.627 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.627 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aba5ebd:::benchmark_data_smithi171_78918_object118006:head# 2024-04-20T17:11:21.627 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.627 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aba8f61:::benchmark_data_smithi171_78918_object66761:head# 2024-04-20T17:11:21.627 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.627 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7abbe63b:::benchmark_data_smithi171_78918_object108506:head# 2024-04-20T17:11:21.627 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.627 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7abc344d:::benchmark_data_smithi171_78918_object141253:head# 2024-04-20T17:11:21.628 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.628 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7abd235c:::benchmark_data_smithi171_78918_object130472:head# 2024-04-20T17:11:21.628 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.628 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7abd8f8d:::benchmark_data_smithi171_78918_object88514:head# 2024-04-20T17:11:21.628 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.628 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7abdff38:::benchmark_data_smithi171_78918_object142041:head# 2024-04-20T17:11:21.628 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.628 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7abfd3f6:::benchmark_data_smithi171_78918_object123811:head# 2024-04-20T17:11:21.628 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.628 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7abfe683:::benchmark_data_smithi171_78918_object66933:head# 2024-04-20T17:11:21.628 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.628 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7abffe21:::benchmark_data_smithi171_78918_object144957:head# 2024-04-20T17:11:21.628 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.628 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ac05ac1:::benchmark_data_smithi171_78918_object69953:head# 2024-04-20T17:11:21.629 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.629 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ac1cc1e:::benchmark_data_smithi171_78918_object52919:head# 2024-04-20T17:11:21.629 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.629 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ac222c6:::benchmark_data_smithi171_78918_object112641:head# 2024-04-20T17:11:21.629 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.629 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ac23f51:::benchmark_data_smithi171_78918_object107696:head# 2024-04-20T17:11:21.629 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.629 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ac2a117:::benchmark_data_smithi171_78918_object91983:head# 2024-04-20T17:11:21.629 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.629 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ac2eda4:::benchmark_data_smithi171_78918_object65155:head# 2024-04-20T17:11:21.629 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.629 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ac2f744:::benchmark_data_smithi171_78918_object119588:head# 2024-04-20T17:11:21.629 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.629 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ac3490b:::benchmark_data_smithi171_78918_object144231:head# 2024-04-20T17:11:21.629 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.630 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ac3bb80:::benchmark_data_smithi171_78918_object150355:head# 2024-04-20T17:11:21.630 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.630 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ac40720:::benchmark_data_smithi171_78918_object84432:head# 2024-04-20T17:11:21.630 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.630 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ac499a8:::benchmark_data_smithi171_78918_object52683:head# 2024-04-20T17:11:21.630 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.630 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ac4b4e7:::benchmark_data_smithi171_78918_object72997:head# 2024-04-20T17:11:21.630 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.630 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ac4bbbd:::benchmark_data_smithi171_78918_object95599:head# 2024-04-20T17:11:21.630 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.630 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ac5668b:::benchmark_data_smithi171_78918_object148703:head# 2024-04-20T17:11:21.630 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.630 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ac567e1:::benchmark_data_smithi171_78918_object117585:head# 2024-04-20T17:11:21.630 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.630 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ac5ca8f:::benchmark_data_smithi171_78918_object131328:head# 2024-04-20T17:11:21.631 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.631 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ac6d533:::benchmark_data_smithi171_78918_object89371:head# 2024-04-20T17:11:21.631 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.631 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ac8b5f6:::benchmark_data_smithi171_78918_object113402:head# 2024-04-20T17:11:21.631 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.631 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ac900a8:::benchmark_data_smithi171_78918_object159733:head# 2024-04-20T17:11:21.631 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.631 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aca14d6:::benchmark_data_smithi171_78918_object143393:head# 2024-04-20T17:11:21.631 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.631 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aca2b1e:::benchmark_data_smithi171_78918_object149007:head# 2024-04-20T17:11:21.631 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.631 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aca7579:::benchmark_data_smithi171_78918_object120270:head# 2024-04-20T17:11:21.631 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.631 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7acaca8a:::benchmark_data_smithi171_78918_object61534:head# 2024-04-20T17:11:21.631 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.632 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7acae55f:::benchmark_data_smithi171_78918_object70273:head# 2024-04-20T17:11:21.632 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.632 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7acb7d43:::benchmark_data_smithi171_78918_object51691:head# 2024-04-20T17:11:21.632 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.632 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7accbad6:::benchmark_data_smithi171_78918_object145405:head# 2024-04-20T17:11:21.632 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.632 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7acd02ff:::benchmark_data_smithi171_78918_object102006:head# 2024-04-20T17:11:21.632 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.632 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7acda07c:::benchmark_data_smithi171_78918_object85846:head# 2024-04-20T17:11:21.632 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.632 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ace42b6:::benchmark_data_smithi171_78918_object160962:head# 2024-04-20T17:11:21.632 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.632 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7acee622:::benchmark_data_smithi171_78918_object128574:head# 2024-04-20T17:11:21.632 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.632 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7acf219d:::benchmark_data_smithi171_78918_object63907:head# 2024-04-20T17:11:21.633 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.633 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ad011e8:::benchmark_data_smithi171_78918_object72558:head# 2024-04-20T17:11:21.633 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.633 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ad0fc77:::benchmark_data_smithi171_78918_object98492:head# 2024-04-20T17:11:21.633 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.633 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ad10857:::benchmark_data_smithi171_78918_object56246:head# 2024-04-20T17:11:21.633 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.633 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ad39eef:::benchmark_data_smithi171_78918_object115553:head# 2024-04-20T17:11:21.633 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.633 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ad48999:::benchmark_data_smithi171_78918_object118381:head# 2024-04-20T17:11:21.633 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.633 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ad52017:::benchmark_data_smithi171_78918_object116187:head# 2024-04-20T17:11:21.633 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.633 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ad522d3:::benchmark_data_smithi171_78918_object91618:head# 2024-04-20T17:11:21.634 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.634 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ad5d0b9:::benchmark_data_smithi171_78918_object89931:head# 2024-04-20T17:11:21.634 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.634 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ad60172:::benchmark_data_smithi171_78918_object150295:head# 2024-04-20T17:11:21.634 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.634 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ad68ef8:::benchmark_data_smithi171_78918_object141341:head# 2024-04-20T17:11:21.634 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.634 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ad6c2a9:::benchmark_data_smithi171_78918_object56950:head# 2024-04-20T17:11:21.634 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.634 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ad7c157:::benchmark_data_smithi171_78918_object65516:head# 2024-04-20T17:11:21.634 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.634 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ad8498b:::benchmark_data_smithi171_78918_object136507:head# 2024-04-20T17:11:21.634 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.634 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ad8e5e7:::benchmark_data_smithi171_78918_object124721:head# 2024-04-20T17:11:21.635 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.635 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ad8eea1:::benchmark_data_smithi171_78918_object157147:head# 2024-04-20T17:11:21.635 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.635 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ad92563:::benchmark_data_smithi171_78918_object71495:head# 2024-04-20T17:11:21.635 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.635 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ad9d1f9:::benchmark_data_smithi171_78918_object138113:head# 2024-04-20T17:11:21.635 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.635 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ad9f734:::benchmark_data_smithi171_78918_object148079:head# 2024-04-20T17:11:21.635 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.635 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7adac2aa:::benchmark_data_smithi171_78918_object121569:head# 2024-04-20T17:11:21.635 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.635 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7adafd99:::benchmark_data_smithi171_78918_object132462:head# 2024-04-20T17:11:21.635 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.635 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7adb76a5:::benchmark_data_smithi171_78918_object137545:head# 2024-04-20T17:11:21.636 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.636 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7adbf09b:::benchmark_data_smithi171_78918_object91321:head# 2024-04-20T17:11:21.636 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.636 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7adc783f:::benchmark_data_smithi171_78918_object65117:head# 2024-04-20T17:11:21.636 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.636 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7add1096:::benchmark_data_smithi171_78918_object117452:head# 2024-04-20T17:11:21.636 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.636 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7add6918:::benchmark_data_smithi171_78918_object118741:head# 2024-04-20T17:11:21.636 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.636 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7addbd83:::benchmark_data_smithi171_78918_object91414:head# 2024-04-20T17:11:21.636 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.636 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7addef3a:::benchmark_data_smithi171_78918_object85601:head# 2024-04-20T17:11:21.636 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.636 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7adf9a8c:::benchmark_data_smithi171_78918_object105747:head# 2024-04-20T17:11:21.637 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.637 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ae01536:::benchmark_data_smithi171_78918_object87968:head# 2024-04-20T17:11:21.637 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.637 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ae05bd7:::benchmark_data_smithi171_78918_object55223:head# 2024-04-20T17:11:21.637 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.637 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ae0b897:::benchmark_data_smithi171_78918_object85640:head# 2024-04-20T17:11:21.637 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.637 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ae342c6:::benchmark_data_smithi171_78918_object82381:head# 2024-04-20T17:11:21.637 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.637 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ae49fa7:::benchmark_data_smithi171_78918_object52347:head# 2024-04-20T17:11:21.637 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.637 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ae53319:::benchmark_data_smithi171_78918_object93910:head# 2024-04-20T17:11:21.637 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.637 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ae7b7a2:::benchmark_data_smithi171_78918_object124395:head# 2024-04-20T17:11:21.638 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.638 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ae8655e:::benchmark_data_smithi171_78918_object110753:head# 2024-04-20T17:11:21.638 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.638 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ae9e58f:::benchmark_data_smithi171_78918_object92305:head# 2024-04-20T17:11:21.638 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.638 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aea5bba:::benchmark_data_smithi171_78918_object59172:head# 2024-04-20T17:11:21.638 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.638 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aea7d83:::benchmark_data_smithi171_78918_object113631:head# 2024-04-20T17:11:21.638 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.638 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aeb88bc:::benchmark_data_smithi171_78918_object150222:head# 2024-04-20T17:11:21.638 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.638 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aeb8aae:::benchmark_data_smithi171_78918_object104113:head# 2024-04-20T17:11:21.638 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.638 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aeba70c:::benchmark_data_smithi171_78918_object84946:head# 2024-04-20T17:11:21.639 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.639 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aebbc12:::benchmark_data_smithi171_78918_object160946:head# 2024-04-20T17:11:21.639 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.639 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aec20f2:::benchmark_data_smithi171_78918_object54891:head# 2024-04-20T17:11:21.639 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.639 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aec2502:::benchmark_data_smithi171_78918_object123237:head# 2024-04-20T17:11:21.639 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.639 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aed4aa9:::benchmark_data_smithi171_78918_object146290:head# 2024-04-20T17:11:21.639 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.639 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aedc8a7:::benchmark_data_smithi171_78918_object97535:head# 2024-04-20T17:11:21.639 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.639 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aee27ef:::benchmark_data_smithi171_78918_object140909:head# 2024-04-20T17:11:21.639 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.640 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aee9cac:::benchmark_data_smithi171_78918_object155943:head# 2024-04-20T17:11:21.640 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.640 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aef60fe:::benchmark_data_smithi171_78918_object100168:head# 2024-04-20T17:11:21.640 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.640 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aef8a01:::benchmark_data_smithi171_78918_object97257:head# 2024-04-20T17:11:21.640 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.640 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7af1e9d9:::benchmark_data_smithi171_78918_object105806:head# 2024-04-20T17:11:21.640 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.640 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7af2d54f:::benchmark_data_smithi171_78918_object130836:head# 2024-04-20T17:11:21.640 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.640 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7af2d56a:::benchmark_data_smithi171_78918_object107060:head# 2024-04-20T17:11:21.640 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.640 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7af36879:::benchmark_data_smithi171_78918_object106158:head# 2024-04-20T17:11:21.640 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.640 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7af4b680:::benchmark_data_smithi171_78918_object124519:head# 2024-04-20T17:11:21.641 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.641 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7af4fb80:::benchmark_data_smithi171_78918_object103194:head# 2024-04-20T17:11:21.641 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.641 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7af5628b:::benchmark_data_smithi171_78918_object122577:head# 2024-04-20T17:11:21.641 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.641 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7af5b86a:::benchmark_data_smithi171_78918_object111088:head# 2024-04-20T17:11:21.641 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.641 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7af5cbc5:::benchmark_data_smithi171_78918_object113995:head# 2024-04-20T17:11:21.641 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.641 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7af68260:::benchmark_data_smithi171_78918_object87774:head# 2024-04-20T17:11:21.641 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.641 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7af6c8b8:::benchmark_data_smithi171_78918_object97870:head# 2024-04-20T17:11:21.641 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.641 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7af7e061:::benchmark_data_smithi171_78918_object72367:head# 2024-04-20T17:11:21.641 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.642 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7af90d6a:::benchmark_data_smithi171_78918_object142915:head# 2024-04-20T17:11:21.642 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.642 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7af93362:::benchmark_data_smithi171_78918_object148894:head# 2024-04-20T17:11:21.642 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.642 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7af9485a:::benchmark_data_smithi171_78918_object158742:head# 2024-04-20T17:11:21.642 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.642 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7af9fbe8:::benchmark_data_smithi171_78918_object60174:head# 2024-04-20T17:11:21.642 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.642 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7afa0b9a:::benchmark_data_smithi171_78918_object92981:head# 2024-04-20T17:11:21.642 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.642 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7afa2972:::benchmark_data_smithi171_78918_object92435:head# 2024-04-20T17:11:21.642 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.642 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7afb5c10:::benchmark_data_smithi171_78918_object131259:head# 2024-04-20T17:11:21.642 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.643 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7afb8024:::benchmark_data_smithi171_78918_object142912:head# 2024-04-20T17:11:21.643 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.643 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7afbc658:::benchmark_data_smithi171_78918_object52449:head# 2024-04-20T17:11:21.643 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.643 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7afbcab4:::benchmark_data_smithi171_78918_object56559:head# 2024-04-20T17:11:21.643 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.643 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7afc0d63:::benchmark_data_smithi171_78918_object72189:head# 2024-04-20T17:11:21.643 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.643 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7afc13c5:::benchmark_data_smithi171_78918_object131176:head# 2024-04-20T17:11:21.643 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.643 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7afc5966:::benchmark_data_smithi171_78918_object148071:head# 2024-04-20T17:11:21.643 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.643 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7afcb98b:::benchmark_data_smithi171_78918_object136107:head# 2024-04-20T17:11:21.643 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.644 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7afdac7b:::benchmark_data_smithi171_78918_object105547:head# 2024-04-20T17:11:21.644 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.644 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7afe01cc:::benchmark_data_smithi171_78918_object64830:head# 2024-04-20T17:11:21.644 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.644 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7afeb688:::benchmark_data_smithi171_78918_object155071:head# 2024-04-20T17:11:21.644 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.644 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aff0624:::benchmark_data_smithi171_78918_object128495:head# 2024-04-20T17:11:21.644 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.644 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aff47d3:::benchmark_data_smithi171_78918_object97460:head# 2024-04-20T17:11:21.644 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.644 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7aff7137:::benchmark_data_smithi171_78918_object89002:head# 2024-04-20T17:11:21.644 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.644 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b00c6e5:::benchmark_data_smithi171_78918_object160160:head# 2024-04-20T17:11:21.644 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.644 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b014390:::benchmark_data_smithi171_78918_object125447:head# 2024-04-20T17:11:21.645 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.645 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b021c17:::benchmark_data_smithi171_78918_object135839:head# 2024-04-20T17:11:21.645 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.645 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b0336fc:::benchmark_data_smithi171_78918_object122099:head# 2024-04-20T17:11:21.645 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.645 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b0371e2:::benchmark_data_smithi171_78918_object58688:head# 2024-04-20T17:11:21.645 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.645 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b048462:::benchmark_data_smithi171_78918_object110878:head# 2024-04-20T17:11:21.645 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.645 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b057579:::benchmark_data_smithi171_78918_object109506:head# 2024-04-20T17:11:21.645 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.645 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b060c66:::benchmark_data_smithi171_78918_object65278:head# 2024-04-20T17:11:21.645 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.645 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b06a271:::benchmark_data_smithi171_78918_object85504:head# 2024-04-20T17:11:21.646 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.646 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b07b1d7:::benchmark_data_smithi171_78918_object150911:head# 2024-04-20T17:11:21.646 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.646 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b07c291:::benchmark_data_smithi171_78918_object67723:head# 2024-04-20T17:11:21.646 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.646 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b084db3:::benchmark_data_smithi171_78918_object91770:head# 2024-04-20T17:11:21.646 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.646 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b090426:::benchmark_data_smithi171_78918_object56965:head# 2024-04-20T17:11:21.646 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.646 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b09ad6c:::benchmark_data_smithi171_78918_object123741:head# 2024-04-20T17:11:21.646 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.646 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b0aef6e:::benchmark_data_smithi171_78918_object68865:head# 2024-04-20T17:11:21.646 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.646 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b0b4e14:::benchmark_data_smithi171_78918_object141009:head# 2024-04-20T17:11:21.646 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.647 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b0b8a24:::benchmark_data_smithi171_78918_object73457:head# 2024-04-20T17:11:21.647 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.647 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b0c23d2:::benchmark_data_smithi171_78918_object55653:head# 2024-04-20T17:11:21.647 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.647 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b0c7240:::benchmark_data_smithi171_78918_object71648:head# 2024-04-20T17:11:21.647 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.647 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b0d0a7b:::benchmark_data_smithi171_78918_object115539:head# 2024-04-20T17:11:21.647 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.647 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b0d5115:::benchmark_data_smithi171_78918_object58765:head# 2024-04-20T17:11:21.647 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.647 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b0e0ccb:::benchmark_data_smithi171_78918_object72137:head# 2024-04-20T17:11:21.647 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.647 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b0e344b:::benchmark_data_smithi171_78918_object111680:head# 2024-04-20T17:11:21.647 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.648 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b0efc66:::benchmark_data_smithi171_78918_object63152:head# 2024-04-20T17:11:21.648 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.648 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b0f1246:::benchmark_data_smithi171_78918_object86713:head# 2024-04-20T17:11:21.648 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.648 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b0f44ee:::benchmark_data_smithi171_78918_object157199:head# 2024-04-20T17:11:21.648 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.648 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b0f7760:::benchmark_data_smithi171_78918_object55337:head# 2024-04-20T17:11:21.648 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.648 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b0fc0ec:::benchmark_data_smithi171_78918_object64599:head# 2024-04-20T17:11:21.648 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.648 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b0fc64f:::benchmark_data_smithi171_78918_object135580:head# 2024-04-20T17:11:21.648 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.648 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b1011cf:::benchmark_data_smithi171_78918_object52645:head# 2024-04-20T17:11:21.648 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.648 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b10ee97:::benchmark_data_smithi171_78918_object130833:head# 2024-04-20T17:11:21.648 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.649 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b124a03:::benchmark_data_smithi171_78918_object104347:head# 2024-04-20T17:11:21.649 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.649 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b131d79:::benchmark_data_smithi171_78918_object147540:head# 2024-04-20T17:11:21.649 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.649 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b152382:::benchmark_data_smithi171_78918_object96620:head# 2024-04-20T17:11:21.649 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.649 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b154a1d:::benchmark_data_smithi171_78918_object77828:head# 2024-04-20T17:11:21.649 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.649 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b15c3ab:::benchmark_data_smithi171_78918_object58748:head# 2024-04-20T17:11:21.649 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.649 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b16d68e:::benchmark_data_smithi171_78918_object153298:head# 2024-04-20T17:11:21.649 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.649 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b179871:::benchmark_data_smithi171_78918_object53293:head# 2024-04-20T17:11:21.649 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.650 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b17d82e:::benchmark_data_smithi171_78918_object132826:head# 2024-04-20T17:11:21.650 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.650 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b1893dc:::benchmark_data_smithi171_78918_object151662:head# 2024-04-20T17:11:21.650 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.650 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b18eeb1:::benchmark_data_smithi171_78918_object119170:head# 2024-04-20T17:11:21.650 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.650 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b1a1503:::benchmark_data_smithi171_78918_object62236:head# 2024-04-20T17:11:21.650 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.650 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b1a3354:::benchmark_data_smithi171_78918_object130459:head# 2024-04-20T17:11:21.650 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.650 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b1a9cc4:::benchmark_data_smithi171_78918_object146175:head# 2024-04-20T17:11:21.650 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.650 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b1adb3c:::benchmark_data_smithi171_78918_object132091:head# 2024-04-20T17:11:21.650 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.651 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b1ae31a:::benchmark_data_smithi171_78918_object125621:head# 2024-04-20T17:11:21.651 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.651 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b1b1c04:::benchmark_data_smithi171_78918_object117462:head# 2024-04-20T17:11:21.651 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.651 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b1c2e93:::benchmark_data_smithi171_78918_object145315:head# 2024-04-20T17:11:21.651 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.651 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b1c8b17:::benchmark_data_smithi171_78918_object142749:head# 2024-04-20T17:11:21.651 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.651 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b1cb753:::benchmark_data_smithi171_78918_object126638:head# 2024-04-20T17:11:21.651 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.651 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b1d572b:::benchmark_data_smithi171_78918_object116895:head# 2024-04-20T17:11:21.651 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.651 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b1e12b7:::benchmark_data_smithi171_78918_object83700:head# 2024-04-20T17:11:21.651 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.651 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b1e5211:::benchmark_data_smithi171_78918_object92620:head# 2024-04-20T17:11:21.652 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.652 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b1e58e7:::benchmark_data_smithi171_78918_object126646:head# 2024-04-20T17:11:21.652 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.652 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b1f33e9:::benchmark_data_smithi171_78918_object136506:head# 2024-04-20T17:11:21.652 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.652 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b1f81ca:::benchmark_data_smithi171_78918_object54555:head# 2024-04-20T17:11:21.652 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.652 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b202567:::benchmark_data_smithi171_78918_object73815:head# 2024-04-20T17:11:21.652 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.652 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b207270:::benchmark_data_smithi171_78918_object60941:head# 2024-04-20T17:11:21.652 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.652 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b20aa54:::benchmark_data_smithi171_78918_object62351:head# 2024-04-20T17:11:21.652 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.652 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b21d15c:::benchmark_data_smithi171_78918_object128005:head# 2024-04-20T17:11:21.652 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.653 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b21d42a:::benchmark_data_smithi171_78918_object123521:head# 2024-04-20T17:11:21.653 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.653 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b223e63:::benchmark_data_smithi171_78918_object120073:head# 2024-04-20T17:11:21.653 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.653 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b22b72f:::benchmark_data_smithi171_78918_object72550:head# 2024-04-20T17:11:21.653 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.653 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b2366f2:::benchmark_data_smithi171_78918_object107053:head# 2024-04-20T17:11:21.653 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.653 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b248d88:::benchmark_data_smithi171_78918_object150846:head# 2024-04-20T17:11:21.653 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.653 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b253ec8:::benchmark_data_smithi171_78918_object146014:head# 2024-04-20T17:11:21.653 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.653 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b25acfa:::benchmark_data_smithi171_78918_object85042:head# 2024-04-20T17:11:21.653 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.653 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b270f9d:::benchmark_data_smithi171_78918_object125648:head# 2024-04-20T17:11:21.654 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.654 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b27378b:::benchmark_data_smithi171_78918_object57934:head# 2024-04-20T17:11:21.654 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.654 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b27b465:::benchmark_data_smithi171_78918_object106570:head# 2024-04-20T17:11:21.654 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.654 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b27dba8:::benchmark_data_smithi171_78918_object82008:head# 2024-04-20T17:11:21.654 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.654 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b28c72a:::benchmark_data_smithi171_78918_object79509:head# 2024-04-20T17:11:21.654 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.654 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b290c8f:::benchmark_data_smithi171_78918_object121055:head# 2024-04-20T17:11:21.654 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.654 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b29d2ac:::benchmark_data_smithi171_78918_object98402:head# 2024-04-20T17:11:21.654 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.654 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b29fc3d:::benchmark_data_smithi171_78918_object73254:head# 2024-04-20T17:11:21.654 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.655 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b2c54f6:::benchmark_data_smithi171_78918_object113846:head# 2024-04-20T17:11:21.655 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.655 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b2c8cba:::benchmark_data_smithi171_78918_object155864:head# 2024-04-20T17:11:21.655 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.655 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b2d02af:::benchmark_data_smithi171_78918_object82247:head# 2024-04-20T17:11:21.655 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.655 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b2e5e5d:::benchmark_data_smithi171_78918_object85124:head# 2024-04-20T17:11:21.655 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.655 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b2e7ea8:::benchmark_data_smithi171_78918_object144159:head# 2024-04-20T17:11:21.655 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.655 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b2efe68:::benchmark_data_smithi171_78918_object69951:head# 2024-04-20T17:11:21.655 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.655 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b2f3ee4:::benchmark_data_smithi171_78918_object152571:head# 2024-04-20T17:11:21.655 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.655 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b2fd2b6:::benchmark_data_smithi171_78918_object59127:head# 2024-04-20T17:11:21.656 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.656 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b301a8f:::benchmark_data_smithi171_78918_object69233:head# 2024-04-20T17:11:21.656 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.656 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b3057d4:::benchmark_data_smithi171_78918_object80195:head# 2024-04-20T17:11:21.656 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.656 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b305b78:::benchmark_data_smithi171_78918_object141272:head# 2024-04-20T17:11:21.656 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.656 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b30982c:::benchmark_data_smithi171_78918_object82694:head# 2024-04-20T17:11:21.656 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.656 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b30e89f:::benchmark_data_smithi171_78918_object92677:head# 2024-04-20T17:11:21.656 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.656 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b3232ea:::benchmark_data_smithi171_78918_object100152:head# 2024-04-20T17:11:21.656 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.656 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b33c771:::benchmark_data_smithi171_78918_object59030:head# 2024-04-20T17:11:21.656 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.657 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b33cf93:::benchmark_data_smithi171_78918_object114036:head# 2024-04-20T17:11:21.657 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.657 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b341e8f:::benchmark_data_smithi171_78918_object104219:head# 2024-04-20T17:11:21.657 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.657 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b345ee4:::benchmark_data_smithi171_78918_object113992:head# 2024-04-20T17:11:21.657 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.657 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b346716:::benchmark_data_smithi171_78918_object66639:head# 2024-04-20T17:11:21.657 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.657 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b36038a:::benchmark_data_smithi171_78918_object109694:head# 2024-04-20T17:11:21.657 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.657 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b364202:::benchmark_data_smithi171_78918_object87461:head# 2024-04-20T17:11:21.657 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.657 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b367c3f:::benchmark_data_smithi171_78918_object56574:head# 2024-04-20T17:11:21.657 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.657 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b37cf86:::benchmark_data_smithi171_78918_object153681:head# 2024-04-20T17:11:21.658 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.658 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b380aa5:::benchmark_data_smithi171_78918_object101808:head# 2024-04-20T17:11:21.658 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.658 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b3869e3:::benchmark_data_smithi171_78918_object83522:head# 2024-04-20T17:11:21.658 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.660 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-20T17:11:21.776 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b396f8c:::benchmark_data_smithi171_78918_object72882:head# 2024-04-20T17:11:21.776 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.776 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b3a350f:::benchmark_data_smithi171_78918_object111246:head# 2024-04-20T17:11:21.776 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.776 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b3a4309:::benchmark_data_smithi171_78918_object87343:head# 2024-04-20T17:11:21.776 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.776 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b3ac102:::benchmark_data_smithi171_78918_object122883:head# 2024-04-20T17:11:21.776 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.776 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b3afcde:::benchmark_data_smithi171_78918_object72382:head# 2024-04-20T17:11:21.776 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.777 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b3bea25:::benchmark_data_smithi171_78918_object78544:head# 2024-04-20T17:11:21.777 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.777 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b3c01e6:::benchmark_data_smithi171_78918_object58839:head# 2024-04-20T17:11:21.777 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.777 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b3cd138:::benchmark_data_smithi171_78918_object70954:head# 2024-04-20T17:11:21.777 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.777 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b3cf622:::benchmark_data_smithi171_78918_object158924:head# 2024-04-20T17:11:21.777 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.777 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b3d2db6:::benchmark_data_smithi171_78918_object154938:head# 2024-04-20T17:11:21.777 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.777 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b3d8341:::benchmark_data_smithi171_78918_object119940:head# 2024-04-20T17:11:21.777 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.777 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b3f4880:::benchmark_data_smithi171_78918_object138220:head# 2024-04-20T17:11:21.777 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.778 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b40262e:::benchmark_data_smithi171_78918_object152305:head# 2024-04-20T17:11:21.778 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.778 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b4041ba:::benchmark_data_smithi171_78918_object89688:head# 2024-04-20T17:11:21.778 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.778 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b406719:::benchmark_data_smithi171_78918_object117270:head# 2024-04-20T17:11:21.778 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.778 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b40a0d5:::benchmark_data_smithi171_78918_object106090:head# 2024-04-20T17:11:21.778 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.778 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b414c59:::benchmark_data_smithi171_78918_object66478:head# 2024-04-20T17:11:21.778 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.778 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b42b794:::benchmark_data_smithi171_78918_object134334:head# 2024-04-20T17:11:21.778 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.778 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b43eed7:::benchmark_data_smithi171_78918_object77773:head# 2024-04-20T17:11:21.779 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.779 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b44cf40:::benchmark_data_smithi171_78918_object61637:head# 2024-04-20T17:11:21.779 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.779 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b457932:::benchmark_data_smithi171_78918_object53762:head# 2024-04-20T17:11:21.779 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.779 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b45e59c:::benchmark_data_smithi171_78918_object106218:head# 2024-04-20T17:11:21.779 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.779 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b45ff63:::benchmark_data_smithi171_78918_object94649:head# 2024-04-20T17:11:21.779 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.779 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b4768eb:::benchmark_data_smithi171_78918_object92297:head# 2024-04-20T17:11:21.779 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.779 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b49ffe7:::benchmark_data_smithi171_78918_object75003:head# 2024-04-20T17:11:21.779 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.779 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b4a268a:::benchmark_data_smithi171_78918_object66972:head# 2024-04-20T17:11:21.780 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.780 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b4a66e9:::benchmark_data_smithi171_78918_object156106:head# 2024-04-20T17:11:21.780 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.780 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b4ae93e:::benchmark_data_smithi171_78918_object69613:head# 2024-04-20T17:11:21.780 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.780 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b4b6744:::benchmark_data_smithi171_78918_object110059:head# 2024-04-20T17:11:21.780 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.780 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b4b7b4b:::benchmark_data_smithi171_78918_object146837:head# 2024-04-20T17:11:21.780 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.780 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b4b942e:::benchmark_data_smithi171_78918_object54675:head# 2024-04-20T17:11:21.780 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.780 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b4bb9c1:::benchmark_data_smithi171_78918_object113272:head# 2024-04-20T17:11:21.780 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.780 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b4c6033:::benchmark_data_smithi171_78918_object120699:head# 2024-04-20T17:11:21.781 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.781 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b4c9d9b:::benchmark_data_smithi171_78918_object128292:head# 2024-04-20T17:11:21.781 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.781 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b4ce76d:::benchmark_data_smithi171_78918_object157042:head# 2024-04-20T17:11:21.781 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.781 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b4d425b:::benchmark_data_smithi171_78918_object145011:head# 2024-04-20T17:11:21.781 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.781 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b4da1b8:::benchmark_data_smithi171_78918_object121637:head# 2024-04-20T17:11:21.781 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.781 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b4dbf4a:::benchmark_data_smithi171_78918_object151072:head# 2024-04-20T17:11:21.781 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.781 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b4f0cbb:::benchmark_data_smithi171_78918_object75030:head# 2024-04-20T17:11:21.781 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.782 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b4f29ee:::benchmark_data_smithi171_78918_object58668:head# 2024-04-20T17:11:21.782 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.782 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b4f2f6e:::benchmark_data_smithi171_78918_object63632:head# 2024-04-20T17:11:21.782 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.782 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b4fedea:::benchmark_data_smithi171_78918_object127940:head# 2024-04-20T17:11:21.782 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.782 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b4ff736:::benchmark_data_smithi171_78918_object136357:head# 2024-04-20T17:11:21.782 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.782 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b509de8:::benchmark_data_smithi171_78918_object154807:head# 2024-04-20T17:11:21.782 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.782 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b5136d1:::benchmark_data_smithi171_78918_object53090:head# 2024-04-20T17:11:21.782 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.782 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b51ef18:::benchmark_data_smithi171_78918_object53808:head# 2024-04-20T17:11:21.782 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.783 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b52567d:::benchmark_data_smithi171_78918_object118425:head# 2024-04-20T17:11:21.783 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.783 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b52b48d:::benchmark_data_smithi171_78918_object71185:head# 2024-04-20T17:11:21.783 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.783 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b52ff8a:::benchmark_data_smithi171_78918_object76142:head# 2024-04-20T17:11:21.783 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.783 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b536387:::benchmark_data_smithi171_78918_object153452:head# 2024-04-20T17:11:21.783 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.783 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b539385:::benchmark_data_smithi171_78918_object99345:head# 2024-04-20T17:11:21.783 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.783 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b541eeb:::benchmark_data_smithi171_78918_object112596:head# 2024-04-20T17:11:21.783 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.783 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b541f8c:::benchmark_data_smithi171_78918_object100213:head# 2024-04-20T17:11:21.783 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.784 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b557d55:::benchmark_data_smithi171_78918_object157103:head# 2024-04-20T17:11:21.784 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.784 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b559c1c:::benchmark_data_smithi171_78918_object81746:head# 2024-04-20T17:11:21.784 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.784 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b56b25b:::benchmark_data_smithi171_78918_object109604:head# 2024-04-20T17:11:21.784 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.784 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b56dab9:::benchmark_data_smithi171_78918_object70014:head# 2024-04-20T17:11:21.784 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.784 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b57426b:::benchmark_data_smithi171_78918_object125872:head# 2024-04-20T17:11:21.784 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.784 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b575c71:::benchmark_data_smithi171_78918_object92548:head# 2024-04-20T17:11:21.784 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.785 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b575ef9:::benchmark_data_smithi171_78918_object116790:head# 2024-04-20T17:11:21.785 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.785 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b586a3f:::benchmark_data_smithi171_78918_object91985:head# 2024-04-20T17:11:21.785 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.785 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b588b3f:::benchmark_data_smithi171_78918_object102881:head# 2024-04-20T17:11:21.785 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.785 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b5aa760:::benchmark_data_smithi171_78918_object72516:head# 2024-04-20T17:11:21.785 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.785 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b5b1c03:::benchmark_data_smithi171_78918_object92856:head# 2024-04-20T17:11:21.785 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.785 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b5b4005:::benchmark_data_smithi171_78918_object133519:head# 2024-04-20T17:11:21.785 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.785 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b5d6c2b:::benchmark_data_smithi171_78918_object127895:head# 2024-04-20T17:11:21.785 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.785 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b5e16a8:::benchmark_data_smithi171_78918_object113352:head# 2024-04-20T17:11:21.786 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.786 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b5f5ddf:::benchmark_data_smithi171_78918_object107410:head# 2024-04-20T17:11:21.786 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.786 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b605572:::benchmark_data_smithi171_78918_object96692:head# 2024-04-20T17:11:21.786 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.786 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b611d50:::benchmark_data_smithi171_78918_object142131:head# 2024-04-20T17:11:21.786 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.786 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b6134c3:::benchmark_data_smithi171_78918_object78813:head# 2024-04-20T17:11:21.786 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.786 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b6200bf:::benchmark_data_smithi171_78918_object61742:head# 2024-04-20T17:11:21.786 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.786 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b624990:::benchmark_data_smithi171_78918_object145374:head# 2024-04-20T17:11:21.786 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.786 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b6380d7:::benchmark_data_smithi171_78918_object73235:head# 2024-04-20T17:11:21.787 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.787 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b639c33:::benchmark_data_smithi171_78918_object61250:head# 2024-04-20T17:11:21.787 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.787 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b63a62e:::benchmark_data_smithi171_78918_object154487:head# 2024-04-20T17:11:21.787 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.787 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b6419d4:::benchmark_data_smithi171_78918_object152996:head# 2024-04-20T17:11:21.787 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.787 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b64ea58:::benchmark_data_smithi171_78918_object128817:head# 2024-04-20T17:11:21.787 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.787 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b64fbab:::benchmark_data_smithi171_78918_object86430:head# 2024-04-20T17:11:21.787 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.787 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b652ea2:::benchmark_data_smithi171_78918_object98716:head# 2024-04-20T17:11:21.787 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.788 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b656efb:::benchmark_data_smithi171_78918_object76625:head# 2024-04-20T17:11:21.788 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.788 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b65a7bc:::benchmark_data_smithi171_78918_object103654:head# 2024-04-20T17:11:21.788 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.788 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b65acf7:::benchmark_data_smithi171_78918_object65897:head# 2024-04-20T17:11:21.788 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.788 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b65d1f6:::benchmark_data_smithi171_78918_object59711:head# 2024-04-20T17:11:21.788 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.788 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b65faa7:::benchmark_data_smithi171_78918_object118998:head# 2024-04-20T17:11:21.788 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.788 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b66c7f6:::benchmark_data_smithi171_78918_object57969:head# 2024-04-20T17:11:21.788 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.788 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b683c8d:::benchmark_data_smithi171_78918_object131520:head# 2024-04-20T17:11:21.788 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.788 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b684c45:::benchmark_data_smithi171_78918_object90510:head# 2024-04-20T17:11:21.789 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.789 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b68a005:::benchmark_data_smithi171_78918_object72655:head# 2024-04-20T17:11:21.789 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.789 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b695315:::benchmark_data_smithi171_78918_object138798:head# 2024-04-20T17:11:21.789 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.789 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b695c71:::benchmark_data_smithi171_78918_object118492:head# 2024-04-20T17:11:21.789 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.789 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b6a5277:::benchmark_data_smithi171_78918_object140144:head# 2024-04-20T17:11:21.789 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.789 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b6a8594:::benchmark_data_smithi171_78918_object138485:head# 2024-04-20T17:11:21.789 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.789 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b6aa37c:::benchmark_data_smithi171_78918_object128372:head# 2024-04-20T17:11:21.789 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.789 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b6ae8be:::benchmark_data_smithi171_78918_object133091:head# 2024-04-20T17:11:21.789 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.790 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b6c2c97:::benchmark_data_smithi171_78918_object141556:head# 2024-04-20T17:11:21.790 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.790 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b6d4ef3:::benchmark_data_smithi171_78918_object96011:head# 2024-04-20T17:11:21.790 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.790 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b6d7b04:::benchmark_data_smithi171_78918_object95781:head# 2024-04-20T17:11:21.790 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.790 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b6db674:::benchmark_data_smithi171_78918_object95789:head# 2024-04-20T17:11:21.790 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.790 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b6e5c84:::benchmark_data_smithi171_78918_object105744:head# 2024-04-20T17:11:21.790 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.791 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b6e73a0:::benchmark_data_smithi171_78918_object82898:head# 2024-04-20T17:11:21.791 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.791 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b6eefa4:::benchmark_data_smithi171_78918_object124363:head# 2024-04-20T17:11:21.791 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.791 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b6f8f6b:::benchmark_data_smithi171_78918_object90906:head# 2024-04-20T17:11:21.791 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.791 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b70d886:::benchmark_data_smithi171_78918_object53415:head# 2024-04-20T17:11:21.791 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.791 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b71b3dd:::benchmark_data_smithi171_78918_object79571:head# 2024-04-20T17:11:21.791 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.791 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b71b43b:::benchmark_data_smithi171_78918_object53133:head# 2024-04-20T17:11:21.791 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.791 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b72447c:::benchmark_data_smithi171_78918_object121110:head# 2024-04-20T17:11:21.791 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.791 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b724906:::benchmark_data_smithi171_78918_object71054:head# 2024-04-20T17:11:21.792 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.792 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b72acce:::benchmark_data_smithi171_78918_object138533:head# 2024-04-20T17:11:21.792 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.792 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b72b600:::benchmark_data_smithi171_78918_object78448:head# 2024-04-20T17:11:21.792 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.792 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b72d599:::benchmark_data_smithi171_78918_object89805:head# 2024-04-20T17:11:21.792 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.792 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b73496f:::benchmark_data_smithi171_78918_object135761:head# 2024-04-20T17:11:21.792 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.792 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b74b8e6:::benchmark_data_smithi171_78918_object139799:head# 2024-04-20T17:11:21.792 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.792 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b74ed13:::benchmark_data_smithi171_78918_object84477:head# 2024-04-20T17:11:21.792 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.792 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b75869f:::benchmark_data_smithi171_78918_object113990:head# 2024-04-20T17:11:21.793 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.793 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b75aa7b:::benchmark_data_smithi171_78918_object107363:head# 2024-04-20T17:11:21.793 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.793 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b761c39:::benchmark_data_smithi171_78918_object107932:head# 2024-04-20T17:11:21.793 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.793 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b777040:::benchmark_data_smithi171_78918_object98361:head# 2024-04-20T17:11:21.793 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.793 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b77fedf:::benchmark_data_smithi171_78918_object96964:head# 2024-04-20T17:11:21.793 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.793 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b78d097:::benchmark_data_smithi171_78918_object147184:head# 2024-04-20T17:11:21.793 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.793 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b78de25:::benchmark_data_smithi171_78918_object108514:head# 2024-04-20T17:11:21.793 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.793 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b796400:::benchmark_data_smithi171_78918_object107837:head# 2024-04-20T17:11:21.794 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.794 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b799d38:::benchmark_data_smithi171_78918_object133111:head# 2024-04-20T17:11:21.794 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.794 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b7a95b6:::benchmark_data_smithi171_78918_object141068:head# 2024-04-20T17:11:21.794 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.794 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b7b8715:::benchmark_data_smithi171_78918_object110662:head# 2024-04-20T17:11:21.794 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.794 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b7c0e45:::benchmark_data_smithi171_78918_object153022:head# 2024-04-20T17:11:21.794 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.794 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b7c593b:::benchmark_data_smithi171_78918_object146785:head# 2024-04-20T17:11:21.794 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.794 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b7c5fd8:::benchmark_data_smithi171_78918_object107235:head# 2024-04-20T17:11:21.794 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.794 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b7c6bb1:::benchmark_data_smithi171_78918_object119176:head# 2024-04-20T17:11:21.795 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.795 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b7c96af:::benchmark_data_smithi171_78918_object123269:head# 2024-04-20T17:11:21.795 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.795 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b7d2f81:::benchmark_data_smithi171_78918_object145470:head# 2024-04-20T17:11:21.795 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.795 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b7ea781:::benchmark_data_smithi171_78918_object106529:head# 2024-04-20T17:11:21.795 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.795 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b800807:::benchmark_data_smithi171_78918_object123152:head# 2024-04-20T17:11:21.795 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.795 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b806852:::benchmark_data_smithi171_78918_object99126:head# 2024-04-20T17:11:21.795 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.795 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b80aec5:::benchmark_data_smithi171_78918_object75175:head# 2024-04-20T17:11:21.795 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.795 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b8112a1:::benchmark_data_smithi171_78918_object64834:head# 2024-04-20T17:11:21.795 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.796 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b8136be:::benchmark_data_smithi171_78918_object80332:head# 2024-04-20T17:11:21.796 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.796 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b81aa57:::benchmark_data_smithi171_78918_object119340:head# 2024-04-20T17:11:21.796 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.796 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b841ff7:::benchmark_data_smithi171_78918_object125609:head# 2024-04-20T17:11:21.796 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.796 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b843c2d:::benchmark_data_smithi171_78918_object131931:head# 2024-04-20T17:11:21.796 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.796 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b848cb7:::benchmark_data_smithi171_78918_object82835:head# 2024-04-20T17:11:21.796 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.796 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b84ca8d:::benchmark_data_smithi171_78918_object78402:head# 2024-04-20T17:11:21.796 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.796 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b855709:::benchmark_data_smithi171_78918_object154462:head# 2024-04-20T17:11:21.796 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.797 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b85744d:::benchmark_data_smithi171_78918_object126670:head# 2024-04-20T17:11:21.797 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.797 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b8599fb:::benchmark_data_smithi171_78918_object109553:head# 2024-04-20T17:11:21.797 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.797 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b866601:::benchmark_data_smithi171_78918_object62317:head# 2024-04-20T17:11:21.797 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.797 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b868688:::benchmark_data_smithi171_78918_object77162:head# 2024-04-20T17:11:21.797 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.797 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b870337:::benchmark_data_smithi171_78918_object72351:head# 2024-04-20T17:11:21.797 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.797 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b8740d3:::benchmark_data_smithi171_78918_object61639:head# 2024-04-20T17:11:21.797 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.797 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b874416:::benchmark_data_smithi171_78918_object91940:head# 2024-04-20T17:11:21.797 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.798 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b884f90:::benchmark_data_smithi171_78918_object153979:head# 2024-04-20T17:11:21.798 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.798 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b897884:::benchmark_data_smithi171_78918_object61766:head# 2024-04-20T17:11:21.798 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.798 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b8ac232:::benchmark_data_smithi171_78918_object78142:head# 2024-04-20T17:11:21.798 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.798 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b8ba401:::benchmark_data_smithi171_78918_object128168:head# 2024-04-20T17:11:21.798 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.798 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b8bd2b8:::benchmark_data_smithi171_78918_object82038:head# 2024-04-20T17:11:21.798 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.798 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b8ca35b:::benchmark_data_smithi171_78918_object136315:head# 2024-04-20T17:11:21.798 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.798 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b8ceec3:::benchmark_data_smithi171_78918_object117317:head# 2024-04-20T17:11:21.798 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.799 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b8d471d:::benchmark_data_smithi171_78918_object85334:head# 2024-04-20T17:11:21.799 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.799 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b8dabc8:::benchmark_data_smithi171_78918_object76067:head# 2024-04-20T17:11:21.799 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.799 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b8f690a:::benchmark_data_smithi171_78918_object66838:head# 2024-04-20T17:11:21.799 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.799 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b8f71c4:::benchmark_data_smithi171_78918_object117163:head# 2024-04-20T17:11:21.799 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.799 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b8fc56e:::benchmark_data_smithi171_78918_object108704:head# 2024-04-20T17:11:21.799 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.799 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b8fe53c:::benchmark_data_smithi171_78918_object64345:head# 2024-04-20T17:11:21.799 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.799 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b90b4d4:::benchmark_data_smithi171_78918_object97888:head# 2024-04-20T17:11:21.799 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.800 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b90cbe5:::benchmark_data_smithi171_78918_object86239:head# 2024-04-20T17:11:21.800 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.800 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b9183ed:::benchmark_data_smithi171_78918_object103375:head# 2024-04-20T17:11:21.800 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.800 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b91abfb:::benchmark_data_smithi171_78918_object98930:head# 2024-04-20T17:11:21.800 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.800 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b91ebc7:::benchmark_data_smithi171_78918_object144560:head# 2024-04-20T17:11:21.800 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.800 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b926be1:::benchmark_data_smithi171_78918_object108057:head# 2024-04-20T17:11:21.800 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.800 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b926d23:::benchmark_data_smithi171_78918_object70583:head# 2024-04-20T17:11:21.800 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.800 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b928f79:::benchmark_data_smithi171_78918_object135011:head# 2024-04-20T17:11:21.800 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.800 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b94f24b:::benchmark_data_smithi171_78918_object152336:head# 2024-04-20T17:11:21.800 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.801 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b956d53:::benchmark_data_smithi171_78918_object71530:head# 2024-04-20T17:11:21.801 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.801 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b963365:::benchmark_data_smithi171_78918_object70051:head# 2024-04-20T17:11:21.801 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.801 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b9645ae:::benchmark_data_smithi171_78918_object113672:head# 2024-04-20T17:11:21.801 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.801 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b964a66:::benchmark_data_smithi171_78918_object94103:head# 2024-04-20T17:11:21.801 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.801 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b96c800:::benchmark_data_smithi171_78918_object53533:head# 2024-04-20T17:11:21.801 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.801 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b9761d1:::benchmark_data_smithi171_78918_object140684:head# 2024-04-20T17:11:21.801 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.801 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b976e55:::benchmark_data_smithi171_78918_object75225:head# 2024-04-20T17:11:21.801 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.801 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b986905:::benchmark_data_smithi171_78918_object151180:head# 2024-04-20T17:11:21.802 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.802 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b98fc3e:::benchmark_data_smithi171_78918_object119918:head# 2024-04-20T17:11:21.802 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.802 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b994694:::benchmark_data_smithi171_78918_object139274:head# 2024-04-20T17:11:21.802 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.802 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b999d41:::benchmark_data_smithi171_78918_object94130:head# 2024-04-20T17:11:21.802 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.802 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b99aff5:::benchmark_data_smithi171_78918_object128324:head# 2024-04-20T17:11:21.802 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.802 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b99e81a:::benchmark_data_smithi171_78918_object146338:head# 2024-04-20T17:11:21.802 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.802 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b9a1969:::benchmark_data_smithi171_78918_object101866:head# 2024-04-20T17:11:21.802 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.802 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b9a338d:::benchmark_data_smithi171_78918_object144065:head# 2024-04-20T17:11:21.803 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.803 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b9b11bd:::benchmark_data_smithi171_78918_object98196:head# 2024-04-20T17:11:21.803 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.803 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b9b43a6:::benchmark_data_smithi171_78918_object80335:head# 2024-04-20T17:11:21.803 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.803 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b9b6a1c:::benchmark_data_smithi171_78918_object58584:head# 2024-04-20T17:11:21.803 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.803 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b9c0883:::benchmark_data_smithi171_78918_object135270:head# 2024-04-20T17:11:21.803 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.803 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b9c35e3:::benchmark_data_smithi171_78918_object158030:head# 2024-04-20T17:11:21.803 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.803 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b9d0972:::benchmark_data_smithi171_78918_object113756:head# 2024-04-20T17:11:21.803 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.803 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b9d82bb:::benchmark_data_smithi171_78918_object126822:head# 2024-04-20T17:11:21.803 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.804 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b9dbd6f:::benchmark_data_smithi171_78918_object125297:head# 2024-04-20T17:11:21.804 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.804 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b9de85d:::benchmark_data_smithi171_78918_object159737:head# 2024-04-20T17:11:21.804 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.804 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b9f03fd:::benchmark_data_smithi171_78918_object76034:head# 2024-04-20T17:11:21.804 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.804 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b9f11be:::benchmark_data_smithi171_78918_object76606:head# 2024-04-20T17:11:21.804 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.804 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b9f5957:::benchmark_data_smithi171_78918_object102908:head# 2024-04-20T17:11:21.804 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.804 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7b9fcadf:::benchmark_data_smithi171_78918_object125543:head# 2024-04-20T17:11:21.804 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.804 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ba03753:::benchmark_data_smithi171_78918_object95446:head# 2024-04-20T17:11:21.805 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.805 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ba0af9a:::benchmark_data_smithi171_78918_object94908:head# 2024-04-20T17:11:21.805 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.805 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ba0d4fd:::benchmark_data_smithi171_78918_object118791:head# 2024-04-20T17:11:21.805 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.805 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ba0da9c:::benchmark_data_smithi171_78918_object120903:head# 2024-04-20T17:11:21.805 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.805 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ba15c65:::benchmark_data_smithi171_78918_object154439:head# 2024-04-20T17:11:21.805 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.805 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ba17471:::benchmark_data_smithi171_78918_object155243:head# 2024-04-20T17:11:21.805 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.805 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ba18339:::benchmark_data_smithi171_78918_object64485:head# 2024-04-20T17:11:21.805 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.805 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ba1a815:::benchmark_data_smithi171_78918_object110089:head# 2024-04-20T17:11:21.806 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.806 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ba1b86b:::benchmark_data_smithi171_78918_object112705:head# 2024-04-20T17:11:21.806 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.806 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ba1c6f7:::benchmark_data_smithi171_78918_object112786:head# 2024-04-20T17:11:21.806 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.806 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ba246c9:::benchmark_data_smithi171_78918_object100114:head# 2024-04-20T17:11:21.806 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.806 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ba31673:::benchmark_data_smithi171_78918_object124314:head# 2024-04-20T17:11:21.806 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.806 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ba3e943:::benchmark_data_smithi171_78918_object70401:head# 2024-04-20T17:11:21.806 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.806 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ba4032b:::benchmark_data_smithi171_78918_object78549:head# 2024-04-20T17:11:21.806 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.806 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ba40bd4:::benchmark_data_smithi171_78918_object87692:head# 2024-04-20T17:11:21.807 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.807 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ba42159:::benchmark_data_smithi171_78918_object106472:head# 2024-04-20T17:11:21.807 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.807 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ba4af18:::benchmark_data_smithi171_78918_object108252:head# 2024-04-20T17:11:21.807 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.807 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ba52b55:::benchmark_data_smithi171_78918_object85624:head# 2024-04-20T17:11:21.807 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.807 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ba597c1:::benchmark_data_smithi171_78918_object67801:head# 2024-04-20T17:11:21.807 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.807 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ba5c826:::benchmark_data_smithi171_78918_object140963:head# 2024-04-20T17:11:21.807 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.807 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ba6a96b:::benchmark_data_smithi171_78918_object110187:head# 2024-04-20T17:11:21.807 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.807 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ba70383:::benchmark_data_smithi171_78918_object77178:head# 2024-04-20T17:11:21.808 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.808 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ba83bf4:::benchmark_data_smithi171_78918_object77515:head# 2024-04-20T17:11:21.808 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.808 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ba93815:::benchmark_data_smithi171_78918_object82080:head# 2024-04-20T17:11:21.808 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.808 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ba97604:::benchmark_data_smithi171_78918_object53376:head# 2024-04-20T17:11:21.808 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.808 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ba99ef6:::benchmark_data_smithi171_78918_object83120:head# 2024-04-20T17:11:21.808 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.808 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7baa23ce:::benchmark_data_smithi171_78918_object109573:head# 2024-04-20T17:11:21.808 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.808 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bab0ad5:::benchmark_data_smithi171_78918_object77865:head# 2024-04-20T17:11:21.808 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.808 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bab2c90:::benchmark_data_smithi171_78918_object108665:head# 2024-04-20T17:11:21.808 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.809 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bab8028:::benchmark_data_smithi171_78918_object103195:head# 2024-04-20T17:11:21.809 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.809 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bad247f:::benchmark_data_smithi171_78918_object58387:head# 2024-04-20T17:11:21.809 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.809 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bad7a66:::benchmark_data_smithi171_78918_object92564:head# 2024-04-20T17:11:21.809 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.809 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7baf56dd:::benchmark_data_smithi171_78918_object99685:head# 2024-04-20T17:11:21.809 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.809 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bafaa47:::benchmark_data_smithi171_78918_object117433:head# 2024-04-20T17:11:21.809 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.809 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bb02544:::benchmark_data_smithi171_78918_object134617:head# 2024-04-20T17:11:21.809 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.809 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bb027b8:::benchmark_data_smithi171_78918_object137304:head# 2024-04-20T17:11:21.809 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.810 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bb1b873:::benchmark_data_smithi171_78918_object62855:head# 2024-04-20T17:11:21.810 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.810 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bb23dea:::benchmark_data_smithi171_78918_object141399:head# 2024-04-20T17:11:21.810 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.810 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bb29d46:::benchmark_data_smithi171_78918_object62291:head# 2024-04-20T17:11:21.810 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.810 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bb3a483:::benchmark_data_smithi171_78918_object88976:head# 2024-04-20T17:11:21.810 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.810 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bb464db:::benchmark_data_smithi171_78918_object84027:head# 2024-04-20T17:11:21.810 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.810 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bb592aa:::benchmark_data_smithi171_78918_object96513:head# 2024-04-20T17:11:21.810 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.810 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bb5ac2e:::benchmark_data_smithi171_78918_object137434:head# 2024-04-20T17:11:21.810 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.810 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bb5c99a:::benchmark_data_smithi171_78918_object143989:head# 2024-04-20T17:11:21.811 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.811 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bb6edb7:::benchmark_data_smithi171_78918_object157523:head# 2024-04-20T17:11:21.811 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.811 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bb75453:::benchmark_data_smithi171_78918_object145629:head# 2024-04-20T17:11:21.811 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.811 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bb774e6:::benchmark_data_smithi171_78918_object54230:head# 2024-04-20T17:11:21.811 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.811 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bb7d468:::benchmark_data_smithi171_78918_object157652:head# 2024-04-20T17:11:21.811 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.811 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bb850ed:::benchmark_data_smithi171_78918_object115537:head# 2024-04-20T17:11:21.811 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.811 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bb8ce99:::benchmark_data_smithi171_78918_object115609:head# 2024-04-20T17:11:21.811 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.811 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bbae6ce:::benchmark_data_smithi171_78918_object130849:head# 2024-04-20T17:11:21.812 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.812 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bbb828b:::benchmark_data_smithi171_78918_object152655:head# 2024-04-20T17:11:21.812 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.812 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bbbc268:::benchmark_data_smithi171_78918_object116161:head# 2024-04-20T17:11:21.812 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.812 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bbcca12:::benchmark_data_smithi171_78918_object82893:head# 2024-04-20T17:11:21.812 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.812 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bbd6c5b:::benchmark_data_smithi171_78918_object107854:head# 2024-04-20T17:11:21.812 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.812 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bbda960:::benchmark_data_smithi171_78918_object71376:head# 2024-04-20T17:11:21.812 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.812 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bbe0161:::benchmark_data_smithi171_78918_object147358:head# 2024-04-20T17:11:21.812 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.812 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bbefad8:::benchmark_data_smithi171_78918_object157311:head# 2024-04-20T17:11:21.812 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.813 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bbf58d9:::benchmark_data_smithi171_78918_object156299:head# 2024-04-20T17:11:21.813 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.813 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bbf5925:::benchmark_data_smithi171_78918_object127866:head# 2024-04-20T17:11:21.813 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.813 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bbf7f67:::benchmark_data_smithi171_78918_object160533:head# 2024-04-20T17:11:21.813 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.813 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bbfd29c:::benchmark_data_smithi171_78918_object104943:head# 2024-04-20T17:11:21.813 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.813 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bc01192:::benchmark_data_smithi171_78918_object131829:head# 2024-04-20T17:11:21.813 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.813 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bc040bb:::benchmark_data_smithi171_78918_object82901:head# 2024-04-20T17:11:21.813 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.813 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bc04d95:::benchmark_data_smithi171_78918_object150888:head# 2024-04-20T17:11:21.813 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.814 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bc14271:::benchmark_data_smithi171_78918_object155601:head# 2024-04-20T17:11:21.814 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.814 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bc21d10:::benchmark_data_smithi171_78918_object126438:head# 2024-04-20T17:11:21.814 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.814 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bc21dbe:::benchmark_data_smithi171_78918_object63363:head# 2024-04-20T17:11:21.814 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.814 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bc220e3:::benchmark_data_smithi171_78918_object101312:head# 2024-04-20T17:11:21.814 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.814 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bc2d7df:::benchmark_data_smithi171_78918_object152291:head# 2024-04-20T17:11:21.814 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.814 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bc3f964:::benchmark_data_smithi171_78918_object95283:head# 2024-04-20T17:11:21.814 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.814 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bc454c9:::benchmark_data_smithi171_78918_object123478:head# 2024-04-20T17:11:21.814 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.815 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bc4e420:::benchmark_data_smithi171_78918_object78819:head# 2024-04-20T17:11:21.815 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.815 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bc56250:::benchmark_data_smithi171_78918_object59579:head# 2024-04-20T17:11:21.815 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.815 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bc5d9fc:::benchmark_data_smithi171_78918_object86361:head# 2024-04-20T17:11:21.815 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.815 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bc6cd3c:::benchmark_data_smithi171_78918_object133739:head# 2024-04-20T17:11:21.815 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.815 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bc77ed0:::benchmark_data_smithi171_78918_object128147:head# 2024-04-20T17:11:21.815 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.815 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bc913a8:::benchmark_data_smithi171_78918_object59632:head# 2024-04-20T17:11:21.815 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.815 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bc9bf7f:::benchmark_data_smithi171_78918_object134647:head# 2024-04-20T17:11:21.815 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.815 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bca61d0:::benchmark_data_smithi171_78918_object152349:head# 2024-04-20T17:11:21.816 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.816 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bcabf1a:::benchmark_data_smithi171_78918_object157401:head# 2024-04-20T17:11:21.816 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.816 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bcc3b62:::benchmark_data_smithi171_78918_object68893:head# 2024-04-20T17:11:21.816 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.816 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bcc4fb2:::benchmark_data_smithi171_78918_object114304:head# 2024-04-20T17:11:21.816 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.816 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bcc819e:::benchmark_data_smithi171_78918_object143111:head# 2024-04-20T17:11:21.816 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.816 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bcd046c:::benchmark_data_smithi171_78918_object90228:head# 2024-04-20T17:11:21.816 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.816 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bcd0bfd:::benchmark_data_smithi171_78918_object139009:head# 2024-04-20T17:11:21.816 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.816 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bce5ff9:::benchmark_data_smithi171_78918_object158123:head# 2024-04-20T17:11:21.816 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.817 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bced057:::benchmark_data_smithi171_78918_object56332:head# 2024-04-20T17:11:21.817 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.817 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bcf63d9:::benchmark_data_smithi171_78918_object79917:head# 2024-04-20T17:11:21.817 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.817 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bd10505:::benchmark_data_smithi171_78918_object88349:head# 2024-04-20T17:11:21.817 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.817 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bd16849:::benchmark_data_smithi171_78918_object83603:head# 2024-04-20T17:11:21.817 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.817 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bd17f8c:::benchmark_data_smithi171_78918_object142677:head# 2024-04-20T17:11:21.817 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.817 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bd1b9d6:::benchmark_data_smithi171_78918_object76077:head# 2024-04-20T17:11:21.817 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.817 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bd1fb98:::benchmark_data_smithi171_78918_object119851:head# 2024-04-20T17:11:21.817 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.817 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bd25408:::benchmark_data_smithi171_78918_object121547:head# 2024-04-20T17:11:21.818 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.818 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bd2b586:::benchmark_data_smithi171_78918_object119994:head# 2024-04-20T17:11:21.818 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.818 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bd38523:::benchmark_data_smithi171_78918_object79118:head# 2024-04-20T17:11:21.818 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.818 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bd38f15:::benchmark_data_smithi171_78918_object82487:head# 2024-04-20T17:11:21.818 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.818 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bd3ffb0:::benchmark_data_smithi171_78918_object74613:head# 2024-04-20T17:11:21.818 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.818 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bd4b774:::benchmark_data_smithi171_78918_object67264:head# 2024-04-20T17:11:21.818 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.818 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bd60762:::benchmark_data_smithi171_78918_object76983:head# 2024-04-20T17:11:21.818 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.818 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bd61f7b:::benchmark_data_smithi171_78918_object84396:head# 2024-04-20T17:11:21.819 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.819 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bd626d5:::benchmark_data_smithi171_78918_object58173:head# 2024-04-20T17:11:21.819 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.819 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bd68ee5:::benchmark_data_smithi171_78918_object140857:head# 2024-04-20T17:11:21.819 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.819 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bd6b825:::benchmark_data_smithi171_78918_object102473:head# 2024-04-20T17:11:21.819 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.819 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bd6e906:::benchmark_data_smithi171_78918_object61118:head# 2024-04-20T17:11:21.819 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.819 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bd7d402:::benchmark_data_smithi171_78918_object55597:head# 2024-04-20T17:11:21.819 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.819 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bd81d8a:::benchmark_data_smithi171_78918_object122459:head# 2024-04-20T17:11:21.819 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.819 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bd8d0f8:::benchmark_data_smithi171_78918_object124975:head# 2024-04-20T17:11:21.820 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.820 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bd8d761:::benchmark_data_smithi171_78918_object90938:head# 2024-04-20T17:11:21.820 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.820 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bd902ac:::benchmark_data_smithi171_78918_object67113:head# 2024-04-20T17:11:21.820 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.820 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bd9ad2b:::benchmark_data_smithi171_78918_object55299:head# 2024-04-20T17:11:21.820 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.820 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bda8b20:::benchmark_data_smithi171_78918_object134559:head# 2024-04-20T17:11:21.820 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.820 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bda9c9c:::benchmark_data_smithi171_78918_object91910:head# 2024-04-20T17:11:21.820 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.820 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bdc060d:::benchmark_data_smithi171_78918_object137407:head# 2024-04-20T17:11:21.820 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.820 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bdcc612:::benchmark_data_smithi171_78918_object110265:head# 2024-04-20T17:11:21.821 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.821 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bddd23a:::benchmark_data_smithi171_78918_object160784:head# 2024-04-20T17:11:21.821 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.821 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bde21bb:::benchmark_data_smithi171_78918_object144851:head# 2024-04-20T17:11:21.821 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.821 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bde9dd9:::benchmark_data_smithi171_78918_object146770:head# 2024-04-20T17:11:21.821 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.821 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bdf23d6:::benchmark_data_smithi171_78918_object88551:head# 2024-04-20T17:11:21.821 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.821 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bdf4151:::benchmark_data_smithi171_78918_object145236:head# 2024-04-20T17:11:21.821 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.821 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7be00319:::benchmark_data_smithi171_78918_object120137:head# 2024-04-20T17:11:21.821 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.821 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7be094ce:::benchmark_data_smithi171_78918_object85169:head# 2024-04-20T17:11:21.821 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.822 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7be10889:::benchmark_data_smithi171_78918_object116752:head# 2024-04-20T17:11:21.822 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.822 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7be47bc8:::benchmark_data_smithi171_78918_object154077:head# 2024-04-20T17:11:21.822 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.822 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7be4c786:::benchmark_data_smithi171_78918_object121490:head# 2024-04-20T17:11:21.822 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.822 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7be59df7:::benchmark_data_smithi171_78918_object144552:head# 2024-04-20T17:11:21.822 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.822 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7be5a179:::benchmark_data_smithi171_78918_object61399:head# 2024-04-20T17:11:21.822 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.822 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7be7ba05:::benchmark_data_smithi171_78918_object150763:head# 2024-04-20T17:11:21.822 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.822 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7be7f2e4:::benchmark_data_smithi171_78918_object120766:head# 2024-04-20T17:11:21.822 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.823 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7be818bd:::benchmark_data_smithi171_78918_object56817:head# 2024-04-20T17:11:21.823 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.823 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7be9ac79:::benchmark_data_smithi171_78918_object113443:head# 2024-04-20T17:11:21.823 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.823 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bea63a0:::benchmark_data_smithi171_78918_object103458:head# 2024-04-20T17:11:21.823 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.823 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7beaa37b:::benchmark_data_smithi171_78918_object96047:head# 2024-04-20T17:11:21.823 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.823 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7beb1b65:::benchmark_data_smithi171_78918_object133425:head# 2024-04-20T17:11:21.823 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.823 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bebb3f1:::benchmark_data_smithi171_78918_object150250:head# 2024-04-20T17:11:21.823 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.823 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bebbc7d:::benchmark_data_smithi171_78918_object89881:head# 2024-04-20T17:11:21.823 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.823 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bec7864:::benchmark_data_smithi171_78918_object134355:head# 2024-04-20T17:11:21.824 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.824 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bed0ae9:::benchmark_data_smithi171_78918_object146994:head# 2024-04-20T17:11:21.824 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.824 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bed8af3:::benchmark_data_smithi171_78918_object160754:head# 2024-04-20T17:11:21.824 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.824 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bedb958:::benchmark_data_smithi171_78918_object64999:head# 2024-04-20T17:11:21.824 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.824 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7beeaf67:::benchmark_data_smithi171_78918_object98779:head# 2024-04-20T17:11:21.824 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.824 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7beebd41:::benchmark_data_smithi171_78918_object84411:head# 2024-04-20T17:11:21.824 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.824 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7beee111:::benchmark_data_smithi171_78918_object55766:head# 2024-04-20T17:11:21.824 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.824 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7befb5d7:::benchmark_data_smithi171_78918_object153360:head# 2024-04-20T17:11:21.825 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.825 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bf0cdb2:::benchmark_data_smithi171_78918_object149504:head# 2024-04-20T17:11:21.825 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.825 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bf1d0ad:::benchmark_data_smithi171_78918_object62300:head# 2024-04-20T17:11:21.825 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.825 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bf367b9:::benchmark_data_smithi171_78918_object86411:head# 2024-04-20T17:11:21.825 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.825 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bf4e5c9:::benchmark_data_smithi171_78918_object91569:head# 2024-04-20T17:11:21.825 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.825 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bf59a81:::benchmark_data_smithi171_78918_object143549:head# 2024-04-20T17:11:21.825 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.825 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bf61c40:::benchmark_data_smithi171_78918_object67463:head# 2024-04-20T17:11:21.825 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.825 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bf6652b:::benchmark_data_smithi171_78918_object76834:head# 2024-04-20T17:11:21.826 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.826 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bf72074:::benchmark_data_smithi171_78918_object98224:head# 2024-04-20T17:11:21.826 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.826 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bf73565:::benchmark_data_smithi171_78918_object86404:head# 2024-04-20T17:11:21.826 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.826 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bf9628d:::benchmark_data_smithi171_78918_object71338:head# 2024-04-20T17:11:21.826 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.826 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bfe3c33:::benchmark_data_smithi171_78918_object82896:head# 2024-04-20T17:11:21.826 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.826 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bfe3e42:::benchmark_data_smithi171_78918_object124015:head# 2024-04-20T17:11:21.826 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.826 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bfeb0d0:::benchmark_data_smithi171_78918_object68226:head# 2024-04-20T17:11:21.826 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.826 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7bff2dfc:::benchmark_data_smithi171_78918_object56284:head# 2024-04-20T17:11:21.826 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.827 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c005670:::benchmark_data_smithi171_78918_object84733:head# 2024-04-20T17:11:21.827 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.827 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c015a0e:::benchmark_data_smithi171_78918_object62709:head# 2024-04-20T17:11:21.827 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.827 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c015fde:::benchmark_data_smithi171_78918_object85779:head# 2024-04-20T17:11:21.827 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.827 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c01625b:::benchmark_data_smithi171_78918_object83903:head# 2024-04-20T17:11:21.827 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.827 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c01daed:::benchmark_data_smithi171_78918_object98168:head# 2024-04-20T17:11:21.827 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.827 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c026f8e:::benchmark_data_smithi171_78918_object59502:head# 2024-04-20T17:11:21.827 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.827 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c038fe7:::benchmark_data_smithi171_78918_object120788:head# 2024-04-20T17:11:21.827 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.828 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c050efb:::benchmark_data_smithi171_78918_object132468:head# 2024-04-20T17:11:21.828 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.828 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c056754:::benchmark_data_smithi171_78918_object156041:head# 2024-04-20T17:11:21.828 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.828 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c0632f5:::benchmark_data_smithi171_78918_object85191:head# 2024-04-20T17:11:21.828 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.828 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c067421:::benchmark_data_smithi171_78918_object112992:head# 2024-04-20T17:11:21.828 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.828 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c068298:::benchmark_data_smithi171_78918_object146970:head# 2024-04-20T17:11:21.828 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.828 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c06f266:::benchmark_data_smithi171_78918_object69760:head# 2024-04-20T17:11:21.828 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.828 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c06f2dd:::benchmark_data_smithi171_78918_object126369:head# 2024-04-20T17:11:21.828 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.829 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c071bf8:::benchmark_data_smithi171_78918_object95539:head# 2024-04-20T17:11:21.829 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.829 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c0840a0:::benchmark_data_smithi171_78918_object63013:head# 2024-04-20T17:11:21.829 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.829 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c0a8361:::benchmark_data_smithi171_78918_object146800:head# 2024-04-20T17:11:21.829 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.829 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c0b1850:::benchmark_data_smithi171_78918_object74509:head# 2024-04-20T17:11:21.829 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.829 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c0ca1b8:::benchmark_data_smithi171_78918_object91816:head# 2024-04-20T17:11:21.829 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.829 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c0e094c:::benchmark_data_smithi171_78918_object74465:head# 2024-04-20T17:11:21.829 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.829 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c0efb1a:::benchmark_data_smithi171_78918_object54442:head# 2024-04-20T17:11:21.829 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.829 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c0f0b08:::benchmark_data_smithi171_78918_object109136:head# 2024-04-20T17:11:21.830 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.830 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c10ebe6:::benchmark_data_smithi171_78918_object93249:head# 2024-04-20T17:11:21.830 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.830 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c11ba4c:::benchmark_data_smithi171_78918_object103081:head# 2024-04-20T17:11:21.830 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.830 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c120034:::benchmark_data_smithi171_78918_object151934:head# 2024-04-20T17:11:21.830 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.830 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c129d5b:::benchmark_data_smithi171_78918_object59646:head# 2024-04-20T17:11:21.830 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.830 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c139db6:::benchmark_data_smithi171_78918_object106133:head# 2024-04-20T17:11:21.830 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.830 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c13d259:::benchmark_data_smithi171_78918_object150873:head# 2024-04-20T17:11:21.830 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.830 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c14342f:::benchmark_data_smithi171_78918_object132668:head# 2024-04-20T17:11:21.831 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.831 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c1497ef:::benchmark_data_smithi171_78918_object123883:head# 2024-04-20T17:11:21.831 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.831 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c150abc:::benchmark_data_smithi171_78918_object76921:head# 2024-04-20T17:11:21.831 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.831 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c151948:::benchmark_data_smithi171_78918_object139805:head# 2024-04-20T17:11:21.831 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.831 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c15a5ca:::benchmark_data_smithi171_78918_object160460:head# 2024-04-20T17:11:21.831 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.831 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c15fb1c:::benchmark_data_smithi171_78918_object132586:head# 2024-04-20T17:11:21.831 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.831 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c17215d:::benchmark_data_smithi171_78918_object108793:head# 2024-04-20T17:11:21.831 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.831 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c173530:::benchmark_data_smithi171_78918_object157784:head# 2024-04-20T17:11:21.832 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.832 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c17cf34:::benchmark_data_smithi171_78918_object118775:head# 2024-04-20T17:11:21.832 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.832 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c17eb2d:::benchmark_data_smithi171_78918_object160481:head# 2024-04-20T17:11:21.832 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.832 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c1827f1:::benchmark_data_smithi171_78918_object110054:head# 2024-04-20T17:11:21.832 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.832 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c185a1c:::benchmark_data_smithi171_78918_object145127:head# 2024-04-20T17:11:21.832 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.832 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c19d961:::benchmark_data_smithi171_78918_object67055:head# 2024-04-20T17:11:21.832 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.832 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c1b6920:::benchmark_data_smithi171_78918_object65430:head# 2024-04-20T17:11:21.832 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.832 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c1cca6e:::benchmark_data_smithi171_78918_object159074:head# 2024-04-20T17:11:21.833 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.833 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c1d0536:::benchmark_data_smithi171_78918_object82913:head# 2024-04-20T17:11:21.833 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.833 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c1da1e4:::benchmark_data_smithi171_78918_object74268:head# 2024-04-20T17:11:21.833 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.833 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c1dbd9c:::benchmark_data_smithi171_78918_object148066:head# 2024-04-20T17:11:21.833 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.833 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c1dc15e:::benchmark_data_smithi171_78918_object127029:head# 2024-04-20T17:11:21.833 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.833 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c1dfb4c:::benchmark_data_smithi171_78918_object139521:head# 2024-04-20T17:11:21.833 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.833 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c1dff47:::benchmark_data_smithi171_78918_object104083:head# 2024-04-20T17:11:21.833 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.833 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c1e65a3:::benchmark_data_smithi171_78918_object115838:head# 2024-04-20T17:11:21.833 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.834 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c1f34ac:::benchmark_data_smithi171_78918_object88896:head# 2024-04-20T17:11:21.834 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.834 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c1f3eed:::benchmark_data_smithi171_78918_object132227:head# 2024-04-20T17:11:21.834 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.834 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c1f7617:::benchmark_data_smithi171_78918_object107959:head# 2024-04-20T17:11:21.834 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.834 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c20327c:::benchmark_data_smithi171_78918_object66244:head# 2024-04-20T17:11:21.834 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.834 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c203b76:::benchmark_data_smithi171_78918_object52746:head# 2024-04-20T17:11:21.834 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.834 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c208297:::benchmark_data_smithi171_78918_object75417:head# 2024-04-20T17:11:21.834 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.834 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c211d66:::benchmark_data_smithi171_78918_object98614:head# 2024-04-20T17:11:21.834 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.835 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c2149ab:::benchmark_data_smithi171_78918_object99606:head# 2024-04-20T17:11:21.835 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.835 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c2184a9:::benchmark_data_smithi171_78918_object96090:head# 2024-04-20T17:11:21.835 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.835 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c21acfd:::benchmark_data_smithi171_78918_object134535:head# 2024-04-20T17:11:21.835 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.835 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c22a62b:::benchmark_data_smithi171_78918_object160755:head# 2024-04-20T17:11:21.835 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.835 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c22f3f2:::benchmark_data_smithi171_78918_object95281:head# 2024-04-20T17:11:21.835 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.835 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c24066c:::benchmark_data_smithi171_78918_object113871:head# 2024-04-20T17:11:21.835 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.835 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c241f7b:::benchmark_data_smithi171_78918_object90267:head# 2024-04-20T17:11:21.835 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.836 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c249d4b:::benchmark_data_smithi171_78918_object87156:head# 2024-04-20T17:11:21.836 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.836 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c253fc7:::benchmark_data_smithi171_78918_object137866:head# 2024-04-20T17:11:21.836 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.836 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c25594a:::benchmark_data_smithi171_78918_object105741:head# 2024-04-20T17:11:21.836 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.836 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c255b04:::benchmark_data_smithi171_78918_object84808:head# 2024-04-20T17:11:21.836 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.836 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c2648b8:::benchmark_data_smithi171_78918_object69062:head# 2024-04-20T17:11:21.836 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.836 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c27afcc:::benchmark_data_smithi171_78918_object136150:head# 2024-04-20T17:11:21.836 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.836 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c27e4d5:::benchmark_data_smithi171_78918_object108454:head# 2024-04-20T17:11:21.836 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.837 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c280a7f:::benchmark_data_smithi171_78918_object136235:head# 2024-04-20T17:11:21.837 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.837 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c286387:::benchmark_data_smithi171_78918_object112211:head# 2024-04-20T17:11:21.837 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.837 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c2a56a4:::benchmark_data_smithi171_78918_object58754:head# 2024-04-20T17:11:21.837 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.837 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c2ad414:::benchmark_data_smithi171_78918_object77340:head# 2024-04-20T17:11:21.837 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.837 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c2c9f27:::benchmark_data_smithi171_78918_object98739:head# 2024-04-20T17:11:21.837 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.837 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c2ca997:::benchmark_data_smithi171_78918_object136524:head# 2024-04-20T17:11:21.838 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.838 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c2db120:::benchmark_data_smithi171_78918_object99210:head# 2024-04-20T17:11:21.838 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.838 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c2eaeee:::benchmark_data_smithi171_78918_object144366:head# 2024-04-20T17:11:21.838 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.838 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c2f6c8c:::benchmark_data_smithi171_78918_object124843:head# 2024-04-20T17:11:21.838 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.838 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c2fcd41:::benchmark_data_smithi171_78918_object107729:head# 2024-04-20T17:11:21.838 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.838 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c304e4d:::benchmark_data_smithi171_78918_object146863:head# 2024-04-20T17:11:21.838 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.838 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c308c85:::benchmark_data_smithi171_78918_object81491:head# 2024-04-20T17:11:21.838 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.839 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c327d79:::benchmark_data_smithi171_78918_object157677:head# 2024-04-20T17:11:21.839 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.839 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c329b01:::benchmark_data_smithi171_78918_object116313:head# 2024-04-20T17:11:21.839 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.839 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c33c134:::benchmark_data_smithi171_78918_object148124:head# 2024-04-20T17:11:21.839 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.839 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c33e96e:::benchmark_data_smithi171_78918_object113601:head# 2024-04-20T17:11:21.839 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.839 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c3415e3:::benchmark_data_smithi171_78918_object111096:head# 2024-04-20T17:11:21.839 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.839 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c3467cc:::benchmark_data_smithi171_78918_object136992:head# 2024-04-20T17:11:21.839 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.839 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c34d052:::benchmark_data_smithi171_78918_object57234:head# 2024-04-20T17:11:21.839 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.839 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c35624c:::benchmark_data_smithi171_78918_object156994:head# 2024-04-20T17:11:21.840 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.840 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c3567a5:::benchmark_data_smithi171_78918_object84168:head# 2024-04-20T17:11:21.840 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.840 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c359910:::benchmark_data_smithi171_78918_object151051:head# 2024-04-20T17:11:21.840 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.840 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c367447:::benchmark_data_smithi171_78918_object138724:head# 2024-04-20T17:11:21.840 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.840 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c369fda:::benchmark_data_smithi171_78918_object117940:head# 2024-04-20T17:11:21.840 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.840 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c377a69:::benchmark_data_smithi171_78918_object149863:head# 2024-04-20T17:11:21.840 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.840 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c381111:::benchmark_data_smithi171_78918_object134002:head# 2024-04-20T17:11:21.840 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.840 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c383197:::benchmark_data_smithi171_78918_object141240:head# 2024-04-20T17:11:21.840 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.841 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c38b1f0:::benchmark_data_smithi171_78918_object135858:head# 2024-04-20T17:11:21.841 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.841 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c392f1b:::benchmark_data_smithi171_78918_object148381:head# 2024-04-20T17:11:21.841 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.841 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c39e408:::benchmark_data_smithi171_78918_object55815:head# 2024-04-20T17:11:21.841 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.841 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c3a6705:::benchmark_data_smithi171_78918_object99672:head# 2024-04-20T17:11:21.841 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.841 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c3b1019:::benchmark_data_smithi171_78918_object68269:head# 2024-04-20T17:11:21.841 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.841 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c3bb779:::benchmark_data_smithi171_78918_object76956:head# 2024-04-20T17:11:21.841 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.841 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c3bf25a:::benchmark_data_smithi171_78918_object135386:head# 2024-04-20T17:11:21.841 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.842 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c3d0f4d:::benchmark_data_smithi171_78918_object129252:head# 2024-04-20T17:11:21.842 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.842 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c3d1c83:::benchmark_data_smithi171_78918_object148911:head# 2024-04-20T17:11:21.842 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.842 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c3d74fb:::benchmark_data_smithi171_78918_object87608:head# 2024-04-20T17:11:21.842 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.842 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c3e78b9:::benchmark_data_smithi171_78918_object154611:head# 2024-04-20T17:11:21.842 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.842 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c3ee3c2:::benchmark_data_smithi171_78918_object124744:head# 2024-04-20T17:11:21.842 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.842 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c3f56d1:::benchmark_data_smithi171_78918_object147921:head# 2024-04-20T17:11:21.842 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.842 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c3faa57:::benchmark_data_smithi171_78918_object136752:head# 2024-04-20T17:11:21.842 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.843 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c4091aa:::benchmark_data_smithi171_78918_object134952:head# 2024-04-20T17:11:21.843 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.843 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c40b100:::benchmark_data_smithi171_78918_object123695:head# 2024-04-20T17:11:21.843 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.843 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c412541:::benchmark_data_smithi171_78918_object75977:head# 2024-04-20T17:11:21.843 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.843 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c41b255:::benchmark_data_smithi171_78918_object155248:head# 2024-04-20T17:11:21.843 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.843 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c42110b:::benchmark_data_smithi171_78918_object66307:head# 2024-04-20T17:11:21.843 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.843 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c42bb55:::benchmark_data_smithi171_78918_object149444:head# 2024-04-20T17:11:21.843 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.843 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c42cf07:::benchmark_data_smithi171_78918_object97206:head# 2024-04-20T17:11:21.843 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.843 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c4385d2:::benchmark_data_smithi171_78918_object153130:head# 2024-04-20T17:11:21.843 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.844 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c439c26:::benchmark_data_smithi171_78918_object125017:head# 2024-04-20T17:11:21.844 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.844 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c44c943:::benchmark_data_smithi171_78918_object155352:head# 2024-04-20T17:11:21.844 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.844 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c462dd0:::benchmark_data_smithi171_78918_object99563:head# 2024-04-20T17:11:21.844 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.844 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c4a1f54:::benchmark_data_smithi171_78918_object145535:head# 2024-04-20T17:11:21.844 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.844 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c4a4c28:::benchmark_data_smithi171_78918_object117384:head# 2024-04-20T17:11:21.844 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.844 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c4b333c:::benchmark_data_smithi171_78918_object100294:head# 2024-04-20T17:11:21.844 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.844 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c4b336d:::benchmark_data_smithi171_78918_object102657:head# 2024-04-20T17:11:21.844 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.844 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c4c2551:::benchmark_data_smithi171_78918_object55189:head# 2024-04-20T17:11:21.845 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.845 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c4c69e8:::benchmark_data_smithi171_78918_object149523:head# 2024-04-20T17:11:21.845 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.845 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c4cd8a7:::benchmark_data_smithi171_78918_object131355:head# 2024-04-20T17:11:21.845 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.845 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c4d1e8a:::benchmark_data_smithi171_78918_object62311:head# 2024-04-20T17:11:21.845 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.845 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c4d6eb1:::benchmark_data_smithi171_78918_object62685:head# 2024-04-20T17:11:21.845 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.845 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c4e4573:::benchmark_data_smithi171_78918_object148799:head# 2024-04-20T17:11:21.845 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.845 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c4e70d2:::benchmark_data_smithi171_78918_object148561:head# 2024-04-20T17:11:21.845 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.846 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c4ecddc:::benchmark_data_smithi171_78918_object141598:head# 2024-04-20T17:11:21.846 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.846 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c4f9911:::benchmark_data_smithi171_78918_object107799:head# 2024-04-20T17:11:21.846 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.846 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c504b08:::benchmark_data_smithi171_78918_object72616:head# 2024-04-20T17:11:21.846 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.846 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c50f95a:::benchmark_data_smithi171_78918_object134955:head# 2024-04-20T17:11:21.846 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.846 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c51f23d:::benchmark_data_smithi171_78918_object159938:head# 2024-04-20T17:11:21.846 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.846 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c52b68e:::benchmark_data_smithi171_78918_object71961:head# 2024-04-20T17:11:21.846 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.846 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c52c981:::benchmark_data_smithi171_78918_object105494:head# 2024-04-20T17:11:21.846 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.847 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c53d0a5:::benchmark_data_smithi171_78918_object80017:head# 2024-04-20T17:11:21.847 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.847 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c54eb4a:::benchmark_data_smithi171_78918_object80896:head# 2024-04-20T17:11:21.847 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.847 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c5526c3:::benchmark_data_smithi171_78918_object105304:head# 2024-04-20T17:11:21.847 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.847 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c55a34c:::benchmark_data_smithi171_78918_object123174:head# 2024-04-20T17:11:21.847 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.847 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c5613c1:::benchmark_data_smithi171_78918_object56823:head# 2024-04-20T17:11:21.847 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.847 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c567a3f:::benchmark_data_smithi171_78918_object111980:head# 2024-04-20T17:11:21.847 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.847 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c568ccf:::benchmark_data_smithi171_78918_object65859:head# 2024-04-20T17:11:21.847 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.848 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c56a8fb:::benchmark_data_smithi171_78918_object135056:head# 2024-04-20T17:11:21.848 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.848 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c57847c:::benchmark_data_smithi171_78918_object151871:head# 2024-04-20T17:11:21.848 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.848 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c57a076:::benchmark_data_smithi171_78918_object114871:head# 2024-04-20T17:11:21.848 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.848 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c59b79e:::benchmark_data_smithi171_78918_object132967:head# 2024-04-20T17:11:21.848 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.848 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c5a7732:::benchmark_data_smithi171_78918_object138253:head# 2024-04-20T17:11:21.848 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.848 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c5a863f:::benchmark_data_smithi171_78918_object114742:head# 2024-04-20T17:11:21.848 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.848 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c5aca39:::benchmark_data_smithi171_78918_object70965:head# 2024-04-20T17:11:21.849 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.849 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c5add18:::benchmark_data_smithi171_78918_object51518:head# 2024-04-20T17:11:21.849 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.849 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c5b1f21:::benchmark_data_smithi171_78918_object97472:head# 2024-04-20T17:11:21.849 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.849 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c5b7732:::benchmark_data_smithi171_78918_object56345:head# 2024-04-20T17:11:21.849 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.849 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c5c52a8:::benchmark_data_smithi171_78918_object107895:head# 2024-04-20T17:11:21.849 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.849 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c5c5405:::benchmark_data_smithi171_78918_object95774:head# 2024-04-20T17:11:21.849 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.849 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c5c65d0:::benchmark_data_smithi171_78918_object84125:head# 2024-04-20T17:11:21.849 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.849 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c5c85a5:::benchmark_data_smithi171_78918_object62958:head# 2024-04-20T17:11:21.849 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.850 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c5d0eb1:::benchmark_data_smithi171_78918_object149867:head# 2024-04-20T17:11:21.850 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.850 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c5e8b8b:::benchmark_data_smithi171_78918_object89034:head# 2024-04-20T17:11:21.850 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.850 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c5f91cd:::benchmark_data_smithi171_78918_object77476:head# 2024-04-20T17:11:21.850 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.850 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c60dc94:::benchmark_data_smithi171_78918_object156658:head# 2024-04-20T17:11:21.850 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.850 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c61ac1e:::benchmark_data_smithi171_78918_object87296:head# 2024-04-20T17:11:21.850 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.850 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c62416d:::benchmark_data_smithi171_78918_object136234:head# 2024-04-20T17:11:21.850 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.850 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c62737a:::benchmark_data_smithi171_78918_object75978:head# 2024-04-20T17:11:21.851 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.851 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c62eb34:::benchmark_data_smithi171_78918_object70118:head# 2024-04-20T17:11:21.851 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.851 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c633558:::benchmark_data_smithi171_78918_object82296:head# 2024-04-20T17:11:21.851 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.851 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c63514d:::benchmark_data_smithi171_78918_object136659:head# 2024-04-20T17:11:21.851 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.851 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c6494b7:::benchmark_data_smithi171_78918_object152334:head# 2024-04-20T17:11:21.851 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.851 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c652c90:::benchmark_data_smithi171_78918_object70592:head# 2024-04-20T17:11:21.851 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.851 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c65695b:::benchmark_data_smithi171_78918_object73323:head# 2024-04-20T17:11:21.851 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.851 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c65d530:::benchmark_data_smithi171_78918_object74923:head# 2024-04-20T17:11:21.852 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.852 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c65d65d:::benchmark_data_smithi171_78918_object98707:head# 2024-04-20T17:11:21.852 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.852 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c6664cf:::benchmark_data_smithi171_78918_object115072:head# 2024-04-20T17:11:21.852 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.852 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c670001:::benchmark_data_smithi171_78918_object78381:head# 2024-04-20T17:11:21.852 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.852 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c671212:::benchmark_data_smithi171_78918_object132585:head# 2024-04-20T17:11:21.852 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.852 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c678aa9:::benchmark_data_smithi171_78918_object79454:head# 2024-04-20T17:11:21.852 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.852 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c68c0f2:::benchmark_data_smithi171_78918_object100918:head# 2024-04-20T17:11:21.852 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.852 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c68f14c:::benchmark_data_smithi171_78918_object93192:head# 2024-04-20T17:11:21.853 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.853 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c69321e:::benchmark_data_smithi171_78918_object103439:head# 2024-04-20T17:11:21.853 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.853 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c6949a7:::benchmark_data_smithi171_78918_object126617:head# 2024-04-20T17:11:21.853 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.853 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c697ea2:::benchmark_data_smithi171_78918_object159957:head# 2024-04-20T17:11:21.853 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.853 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c6a99ea:::benchmark_data_smithi171_78918_object53088:head# 2024-04-20T17:11:21.853 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.853 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c6ba814:::benchmark_data_smithi171_78918_object158873:head# 2024-04-20T17:11:21.853 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.853 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c6c95b2:::benchmark_data_smithi171_78918_object149417:head# 2024-04-20T17:11:21.853 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.853 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c6e4727:::benchmark_data_smithi171_78918_object95231:head# 2024-04-20T17:11:21.853 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.854 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c6eb802:::benchmark_data_smithi171_78918_object73377:head# 2024-04-20T17:11:21.854 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.854 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c6f5676:::benchmark_data_smithi171_78918_object144667:head# 2024-04-20T17:11:21.854 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.854 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c716a9e:::benchmark_data_smithi171_78918_object121062:head# 2024-04-20T17:11:21.854 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.854 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c71d600:::benchmark_data_smithi171_78918_object132522:head# 2024-04-20T17:11:21.854 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.854 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c73df55:::benchmark_data_smithi171_78918_object78045:head# 2024-04-20T17:11:21.854 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.854 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c741ec9:::benchmark_data_smithi171_78918_object117809:head# 2024-04-20T17:11:21.854 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.854 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c7448ba:::benchmark_data_smithi171_78918_object130328:head# 2024-04-20T17:11:21.854 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.854 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c7453dd:::benchmark_data_smithi171_78918_object59176:head# 2024-04-20T17:11:21.855 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.855 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c751bf5:::benchmark_data_smithi171_78918_object92977:head# 2024-04-20T17:11:21.855 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.855 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c7620a0:::benchmark_data_smithi171_78918_object149070:head# 2024-04-20T17:11:21.855 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.855 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c768710:::benchmark_data_smithi171_78918_object122409:head# 2024-04-20T17:11:21.855 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.855 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c783ecf:::benchmark_data_smithi171_78918_object99417:head# 2024-04-20T17:11:21.855 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.855 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c7863bf:::benchmark_data_smithi171_78918_object114517:head# 2024-04-20T17:11:21.855 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.855 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c78d98b:::benchmark_data_smithi171_78918_object83218:head# 2024-04-20T17:11:21.855 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.855 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c78fdf8:::benchmark_data_smithi171_78918_object149175:head# 2024-04-20T17:11:21.856 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.856 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c79a799:::benchmark_data_smithi171_78918_object78248:head# 2024-04-20T17:11:21.856 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.856 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c79b6d9:::benchmark_data_smithi171_78918_object91194:head# 2024-04-20T17:11:21.856 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.856 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c7a8c5a:::benchmark_data_smithi171_78918_object66448:head# 2024-04-20T17:11:21.856 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.856 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c7a9955:::benchmark_data_smithi171_78918_object123497:head# 2024-04-20T17:11:21.856 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.856 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c7c53f4:::benchmark_data_smithi171_78918_object124510:head# 2024-04-20T17:11:21.856 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.856 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c7caf49:::benchmark_data_smithi171_78918_object81184:head# 2024-04-20T17:11:21.856 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.856 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c7ce5e8:::benchmark_data_smithi171_78918_object92394:head# 2024-04-20T17:11:21.856 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.857 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c7ce994:::benchmark_data_smithi171_78918_object77283:head# 2024-04-20T17:11:21.857 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.857 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c7ce9a9:::benchmark_data_smithi171_78918_object101235:head# 2024-04-20T17:11:21.857 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.857 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c7de34a:::benchmark_data_smithi171_78918_object105477:head# 2024-04-20T17:11:21.857 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.857 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c7e7ec1:::benchmark_data_smithi171_78918_object58328:head# 2024-04-20T17:11:21.857 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.857 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c7e8d29:::benchmark_data_smithi171_78918_object84486:head# 2024-04-20T17:11:21.857 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.857 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c8093a2:::benchmark_data_smithi171_78918_object92482:head# 2024-04-20T17:11:21.857 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.857 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c80d3e8:::benchmark_data_smithi171_78918_object102274:head# 2024-04-20T17:11:21.857 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.857 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c8137c2:::benchmark_data_smithi171_78918_object79252:head# 2024-04-20T17:11:21.857 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.858 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c830f18:::benchmark_data_smithi171_78918_object125053:head# 2024-04-20T17:11:21.858 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.858 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c8314b9:::benchmark_data_smithi171_78918_object122921:head# 2024-04-20T17:11:21.858 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.858 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c83b36c:::benchmark_data_smithi171_78918_object62077:head# 2024-04-20T17:11:21.858 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.858 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c84e5cd:::benchmark_data_smithi171_78918_object64693:head# 2024-04-20T17:11:21.858 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.858 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c857fe8:::benchmark_data_smithi171_78918_object123021:head# 2024-04-20T17:11:21.858 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.858 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c85860b:::benchmark_data_smithi171_78918_object152780:head# 2024-04-20T17:11:21.858 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.858 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c8812e0:::benchmark_data_smithi171_78918_object134544:head# 2024-04-20T17:11:21.859 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.859 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c89b1f3:::benchmark_data_smithi171_78918_object149400:head# 2024-04-20T17:11:21.859 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.859 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c89eafb:::benchmark_data_smithi171_78918_object67868:head# 2024-04-20T17:11:21.859 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.859 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c8ab34d:::benchmark_data_smithi171_78918_object106232:head# 2024-04-20T17:11:21.859 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.859 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c8b8c03:::benchmark_data_smithi171_78918_object87401:head# 2024-04-20T17:11:21.859 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.859 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c8c3b71:::benchmark_data_smithi171_78918_object73407:head# 2024-04-20T17:11:21.859 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.859 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c8df72b:::benchmark_data_smithi171_78918_object154766:head# 2024-04-20T17:11:21.859 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.859 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c8f20bf:::benchmark_data_smithi171_78918_object105300:head# 2024-04-20T17:11:21.859 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.859 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c8fa5ed:::benchmark_data_smithi171_78918_object91541:head# 2024-04-20T17:11:21.860 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.860 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c8fd0a1:::benchmark_data_smithi171_78918_object141537:head# 2024-04-20T17:11:21.860 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.860 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c9052e1:::benchmark_data_smithi171_78918_object128626:head# 2024-04-20T17:11:21.860 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.860 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c90a698:::benchmark_data_smithi171_78918_object128349:head# 2024-04-20T17:11:21.860 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.860 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c912a84:::benchmark_data_smithi171_78918_object128196:head# 2024-04-20T17:11:21.860 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.860 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c914074:::benchmark_data_smithi171_78918_object54188:head# 2024-04-20T17:11:21.860 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.860 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c91da57:::benchmark_data_smithi171_78918_object87929:head# 2024-04-20T17:11:21.860 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.861 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c9263bd:::benchmark_data_smithi171_78918_object52840:head# 2024-04-20T17:11:21.861 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.861 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c92d997:::benchmark_data_smithi171_78918_object62775:head# 2024-04-20T17:11:21.861 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.861 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c93228e:::benchmark_data_smithi171_78918_object141278:head# 2024-04-20T17:11:21.861 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.861 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c936b02:::benchmark_data_smithi171_78918_object159392:head# 2024-04-20T17:11:21.861 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.861 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c93e16b:::benchmark_data_smithi171_78918_object125205:head# 2024-04-20T17:11:21.861 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.861 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c9439ce:::benchmark_data_smithi171_78918_object84311:head# 2024-04-20T17:11:21.861 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.861 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c944336:::benchmark_data_smithi171_78918_object141539:head# 2024-04-20T17:11:21.861 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.862 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c953acd:::benchmark_data_smithi171_78918_object62970:head# 2024-04-20T17:11:21.862 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.862 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c958a48:::benchmark_data_smithi171_78918_object149806:head# 2024-04-20T17:11:21.862 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.862 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c95acbe:::benchmark_data_smithi171_78918_object158383:head# 2024-04-20T17:11:21.862 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.862 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c95eee0:::benchmark_data_smithi171_78918_object148953:head# 2024-04-20T17:11:21.862 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.862 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c962852:::benchmark_data_smithi171_78918_object70169:head# 2024-04-20T17:11:21.862 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.862 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c96539c:::benchmark_data_smithi171_78918_object110791:head# 2024-04-20T17:11:21.862 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.862 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c96c065:::benchmark_data_smithi171_78918_object53687:head# 2024-04-20T17:11:21.863 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.863 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c97280d:::benchmark_data_smithi171_78918_object62383:head# 2024-04-20T17:11:21.863 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.863 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c97e251:::benchmark_data_smithi171_78918_object77885:head# 2024-04-20T17:11:21.863 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.863 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c99b246:::benchmark_data_smithi171_78918_object110836:head# 2024-04-20T17:11:21.863 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.863 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c99ce2c:::benchmark_data_smithi171_78918_object130048:head# 2024-04-20T17:11:21.863 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.863 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c9a8aec:::benchmark_data_smithi171_78918_object70524:head# 2024-04-20T17:11:21.863 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.863 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c9ba171:::benchmark_data_smithi171_78918_object119979:head# 2024-04-20T17:11:21.863 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.863 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c9baadd:::benchmark_data_smithi171_78918_object65661:head# 2024-04-20T17:11:21.863 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.863 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c9c4986:::benchmark_data_smithi171_78918_object118574:head# 2024-04-20T17:11:21.864 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.864 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c9c4e4c:::benchmark_data_smithi171_78918_object70156:head# 2024-04-20T17:11:21.864 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.864 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c9d69dd:::benchmark_data_smithi171_78918_object149846:head# 2024-04-20T17:11:21.864 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.864 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c9dba5e:::benchmark_data_smithi171_78918_object54981:head# 2024-04-20T17:11:21.864 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.864 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c9dfb70:::benchmark_data_smithi171_78918_object122320:head# 2024-04-20T17:11:21.864 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.864 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c9e368e:::benchmark_data_smithi171_78918_object107373:head# 2024-04-20T17:11:21.864 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.864 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c9e5352:::benchmark_data_smithi171_78918_object108860:head# 2024-04-20T17:11:21.864 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.864 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7c9e543b:::benchmark_data_smithi171_78918_object55420:head# 2024-04-20T17:11:21.864 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.865 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ca00302:::benchmark_data_smithi171_78918_object88012:head# 2024-04-20T17:11:21.865 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.865 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ca00853:::benchmark_data_smithi171_78918_object86307:head# 2024-04-20T17:11:21.865 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.865 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ca02802:::benchmark_data_smithi171_78918_object57692:head# 2024-04-20T17:11:21.865 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.865 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ca150a1:::benchmark_data_smithi171_78918_object91453:head# 2024-04-20T17:11:21.865 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.865 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ca1a562:::benchmark_data_smithi171_78918_object77827:head# 2024-04-20T17:11:21.865 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.865 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ca212f6:::benchmark_data_smithi171_78918_object117972:head# 2024-04-20T17:11:21.865 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.865 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ca2fc91:::benchmark_data_smithi171_78918_object100682:head# 2024-04-20T17:11:21.865 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.866 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ca36f1d:::benchmark_data_smithi171_78918_object145767:head# 2024-04-20T17:11:21.866 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.866 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ca443a7:::benchmark_data_smithi171_78918_object57259:head# 2024-04-20T17:11:21.866 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.866 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ca4c9fa:::benchmark_data_smithi171_78918_object133979:head# 2024-04-20T17:11:21.866 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.866 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ca52454:::benchmark_data_smithi171_78918_object153338:head# 2024-04-20T17:11:21.866 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.866 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ca57b5b:::benchmark_data_smithi171_78918_object76074:head# 2024-04-20T17:11:21.866 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.866 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ca5adce:::benchmark_data_smithi171_78918_object61165:head# 2024-04-20T17:11:21.866 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.866 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ca634ed:::benchmark_data_smithi171_78918_object158530:head# 2024-04-20T17:11:21.866 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.867 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ca79f26:::benchmark_data_smithi171_78918_object147278:head# 2024-04-20T17:11:21.867 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.867 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ca7a9ac:::benchmark_data_smithi171_78918_object145630:head# 2024-04-20T17:11:21.867 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.867 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ca7acf0:::benchmark_data_smithi171_78918_object74767:head# 2024-04-20T17:11:21.867 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.867 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ca7f4dc:::benchmark_data_smithi171_78918_object83493:head# 2024-04-20T17:11:21.867 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.867 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ca8411d:::benchmark_data_smithi171_78918_object160787:head# 2024-04-20T17:11:21.867 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.867 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ca89053:::benchmark_data_smithi171_78918_object160918:head# 2024-04-20T17:11:21.867 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.867 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ca8a68b:::benchmark_data_smithi171_78918_object160955:head# 2024-04-20T17:11:21.867 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.868 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ca95bfd:::benchmark_data_smithi171_78918_object98865:head# 2024-04-20T17:11:21.868 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.868 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7caa3368:::benchmark_data_smithi171_78918_object134765:head# 2024-04-20T17:11:21.868 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.868 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7caa9238:::benchmark_data_smithi171_78918_object82140:head# 2024-04-20T17:11:21.868 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.868 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7caabfb9:::benchmark_data_smithi171_78918_object113413:head# 2024-04-20T17:11:21.868 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.868 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cabbc3e:::benchmark_data_smithi171_78918_object102627:head# 2024-04-20T17:11:21.868 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.868 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cac48ae:::benchmark_data_smithi171_78918_object132538:head# 2024-04-20T17:11:21.868 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.868 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cad524a:::benchmark_data_smithi171_78918_object80751:head# 2024-04-20T17:11:21.869 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.869 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cadfc15:::benchmark_data_smithi171_78918_object88126:head# 2024-04-20T17:11:21.869 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.869 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7caf64ec:::benchmark_data_smithi171_78918_object74552:head# 2024-04-20T17:11:21.869 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.869 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7caf7e05:::benchmark_data_smithi171_78918_object93405:head# 2024-04-20T17:11:21.869 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.869 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cafd7f8:::benchmark_data_smithi171_78918_object58053:head# 2024-04-20T17:11:21.869 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.869 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cb02768:::benchmark_data_smithi171_78918_object159245:head# 2024-04-20T17:11:21.869 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.869 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cb08973:::benchmark_data_smithi171_78918_object60775:head# 2024-04-20T17:11:21.869 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.869 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cb1a478:::benchmark_data_smithi171_78918_object138767:head# 2024-04-20T17:11:21.870 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.870 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cb1da81:::benchmark_data_smithi171_78918_object151136:head# 2024-04-20T17:11:21.870 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.870 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cb200f7:::benchmark_data_smithi171_78918_object71964:head# 2024-04-20T17:11:21.870 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.870 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cb231b5:::benchmark_data_smithi171_78918_object131208:head# 2024-04-20T17:11:21.870 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.870 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cb24b10:::benchmark_data_smithi171_78918_object130749:head# 2024-04-20T17:11:21.870 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.870 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cb24ca7:::benchmark_data_smithi171_78918_object156466:head# 2024-04-20T17:11:21.870 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.870 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cb36efb:::benchmark_data_smithi171_78918_object60891:head# 2024-04-20T17:11:21.870 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.870 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cb3d96e:::benchmark_data_smithi171_78918_object149709:head# 2024-04-20T17:11:21.871 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.871 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cb43401:::benchmark_data_smithi171_78918_object159766:head# 2024-04-20T17:11:21.871 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.871 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cb59678:::benchmark_data_smithi171_78918_object135861:head# 2024-04-20T17:11:21.871 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.871 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cb5edea:::benchmark_data_smithi171_78918_object69358:head# 2024-04-20T17:11:21.871 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.871 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cb5ffb8:::benchmark_data_smithi171_78918_object149227:head# 2024-04-20T17:11:21.871 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.871 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cb6b907:::benchmark_data_smithi171_78918_object133914:head# 2024-04-20T17:11:21.871 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.871 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cb77713:::benchmark_data_smithi171_78918_object153194:head# 2024-04-20T17:11:21.871 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.871 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cb7c3c8:::benchmark_data_smithi171_78918_object53462:head# 2024-04-20T17:11:21.871 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.872 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cb7d320:::benchmark_data_smithi171_78918_object98351:head# 2024-04-20T17:11:21.872 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.872 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cb7ea26:::benchmark_data_smithi171_78918_object141174:head# 2024-04-20T17:11:21.872 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.872 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cb814bf:::benchmark_data_smithi171_78918_object148910:head# 2024-04-20T17:11:21.872 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.872 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cb85b88:::benchmark_data_smithi171_78918_object125057:head# 2024-04-20T17:11:21.872 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.872 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cb87f18:::benchmark_data_smithi171_78918_object154138:head# 2024-04-20T17:11:21.872 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.872 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cb8e16f:::benchmark_data_smithi171_78918_object59385:head# 2024-04-20T17:11:21.872 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.872 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cba2435:::benchmark_data_smithi171_78918_object120640:head# 2024-04-20T17:11:21.873 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.873 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cbb9f0c:::benchmark_data_smithi171_78918_object91490:head# 2024-04-20T17:11:21.873 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.873 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cbba842:::benchmark_data_smithi171_78918_object53406:head# 2024-04-20T17:11:21.873 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.873 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cbc01cb:::benchmark_data_smithi171_78918_object78230:head# 2024-04-20T17:11:21.873 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.873 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cbd5409:::benchmark_data_smithi171_78918_object146526:head# 2024-04-20T17:11:21.873 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.873 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cbd9a0e:::benchmark_data_smithi171_78918_object108207:head# 2024-04-20T17:11:21.873 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.873 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cbe581d:::benchmark_data_smithi171_78918_object152315:head# 2024-04-20T17:11:21.873 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.873 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cbfd1ac:::benchmark_data_smithi171_78918_object143120:head# 2024-04-20T17:11:21.873 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.874 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cc23b60:::benchmark_data_smithi171_78918_object159825:head# 2024-04-20T17:11:21.874 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.874 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cc32604:::benchmark_data_smithi171_78918_object56104:head# 2024-04-20T17:11:21.874 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.874 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cc44dbd:::benchmark_data_smithi171_78918_object128119:head# 2024-04-20T17:11:21.874 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.874 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cc46e50:::benchmark_data_smithi171_78918_object75422:head# 2024-04-20T17:11:21.874 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.874 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cc76391:::benchmark_data_smithi171_78918_object158815:head# 2024-04-20T17:11:21.874 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.874 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cc91af6:::benchmark_data_smithi171_78918_object62385:head# 2024-04-20T17:11:21.874 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.874 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ccad61b:::benchmark_data_smithi171_78918_object79853:head# 2024-04-20T17:11:21.874 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.874 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ccca959:::benchmark_data_smithi171_78918_object144023:head# 2024-04-20T17:11:21.875 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.875 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ccd8c1d:::benchmark_data_smithi171_78918_object72249:head# 2024-04-20T17:11:21.875 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.875 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ccdb0f2:::benchmark_data_smithi171_78918_object93647:head# 2024-04-20T17:11:21.875 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.875 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ccdbcf4:::benchmark_data_smithi171_78918_object125866:head# 2024-04-20T17:11:21.875 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.875 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cce3477:::benchmark_data_smithi171_78918_object153551:head# 2024-04-20T17:11:21.875 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.875 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cceaa9e:::benchmark_data_smithi171_78918_object144589:head# 2024-04-20T17:11:21.875 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.875 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ccf441e:::benchmark_data_smithi171_78918_object133415:head# 2024-04-20T17:11:21.875 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.875 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cd0487f:::benchmark_data_smithi171_78918_object54709:head# 2024-04-20T17:11:21.876 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.876 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cd14069:::benchmark_data_smithi171_78918_object157622:head# 2024-04-20T17:11:21.876 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.876 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cd1b320:::benchmark_data_smithi171_78918_object129014:head# 2024-04-20T17:11:21.876 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.876 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cd1d0e5:::benchmark_data_smithi171_78918_object121725:head# 2024-04-20T17:11:21.876 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.876 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cd3b2b5:::benchmark_data_smithi171_78918_object160119:head# 2024-04-20T17:11:21.876 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.876 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cd49e8f:::benchmark_data_smithi171_78918_object128957:head# 2024-04-20T17:11:21.876 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.876 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cd4b995:::benchmark_data_smithi171_78918_object61706:head# 2024-04-20T17:11:21.876 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.876 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cd5abbe:::benchmark_data_smithi171_78918_object155385:head# 2024-04-20T17:11:21.876 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.877 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cd5c987:::benchmark_data_smithi171_78918_object97476:head# 2024-04-20T17:11:21.877 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.877 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cd61314:::benchmark_data_smithi171_78918_object110465:head# 2024-04-20T17:11:21.877 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.877 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cd61b58:::benchmark_data_smithi171_78918_object101050:head# 2024-04-20T17:11:21.877 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.877 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cd61b68:::benchmark_data_smithi171_78918_object140360:head# 2024-04-20T17:11:21.877 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.877 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cd6dd2b:::benchmark_data_smithi171_78918_object151788:head# 2024-04-20T17:11:21.877 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.877 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cd75d32:::benchmark_data_smithi171_78918_object105688:head# 2024-04-20T17:11:21.877 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.877 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cd8a6f1:::benchmark_data_smithi171_78918_object73635:head# 2024-04-20T17:11:21.877 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.877 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cd90640:::benchmark_data_smithi171_78918_object69225:head# 2024-04-20T17:11:21.877 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.878 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cd966cb:::benchmark_data_smithi171_78918_object108546:head# 2024-04-20T17:11:21.878 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.878 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cda0fd9:::benchmark_data_smithi171_78918_object128475:head# 2024-04-20T17:11:21.878 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.878 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cdb7ed3:::benchmark_data_smithi171_78918_object107677:head# 2024-04-20T17:11:21.878 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.878 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cdc14f2:::benchmark_data_smithi171_78918_object81716:head# 2024-04-20T17:11:21.878 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.878 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cdc7fec:::benchmark_data_smithi171_78918_object100675:head# 2024-04-20T17:11:21.878 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.878 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cdd3c1e:::benchmark_data_smithi171_78918_object79263:head# 2024-04-20T17:11:21.878 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.878 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cde0245:::benchmark_data_smithi171_78918_object158234:head# 2024-04-20T17:11:21.878 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.879 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cdf2203:::benchmark_data_smithi171_78918_object88576:head# 2024-04-20T17:11:21.879 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.879 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cdf612a:::benchmark_data_smithi171_78918_object118950:head# 2024-04-20T17:11:21.879 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.879 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cdf923d:::benchmark_data_smithi171_78918_object111458:head# 2024-04-20T17:11:21.879 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.879 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cdfe463:::benchmark_data_smithi171_78918_object135831:head# 2024-04-20T17:11:21.879 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.879 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ce03c66:::benchmark_data_smithi171_78918_object64061:head# 2024-04-20T17:11:21.879 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.879 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ce05365:::benchmark_data_smithi171_78918_object130742:head# 2024-04-20T17:11:21.879 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.879 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ce0caed:::benchmark_data_smithi171_78918_object128931:head# 2024-04-20T17:11:21.879 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.879 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ce19355:::benchmark_data_smithi171_78918_object139776:head# 2024-04-20T17:11:21.879 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.880 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ce1a8e8:::benchmark_data_smithi171_78918_object137995:head# 2024-04-20T17:11:21.880 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.880 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ce2addf:::benchmark_data_smithi171_78918_object103641:head# 2024-04-20T17:11:21.880 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.880 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ce2cfe7:::benchmark_data_smithi171_78918_object137231:head# 2024-04-20T17:11:21.880 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.880 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ce35f67:::benchmark_data_smithi171_78918_object105603:head# 2024-04-20T17:11:21.880 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.880 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ce414ca:::benchmark_data_smithi171_78918_object70579:head# 2024-04-20T17:11:21.880 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.880 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ce52720:::benchmark_data_smithi171_78918_object65924:head# 2024-04-20T17:11:21.880 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.880 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ce614a3:::benchmark_data_smithi171_78918_object91671:head# 2024-04-20T17:11:21.880 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.881 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ce722ee:::benchmark_data_smithi171_78918_object145013:head# 2024-04-20T17:11:21.881 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.881 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ce7f141:::benchmark_data_smithi171_78918_object71264:head# 2024-04-20T17:11:21.881 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.881 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ce881f1:::benchmark_data_smithi171_78918_object130335:head# 2024-04-20T17:11:21.881 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.881 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ce8b9e3:::benchmark_data_smithi171_78918_object118186:head# 2024-04-20T17:11:21.881 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.881 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ce906d1:::benchmark_data_smithi171_78918_object135330:head# 2024-04-20T17:11:21.881 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.881 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ce92bf2:::benchmark_data_smithi171_78918_object58183:head# 2024-04-20T17:11:21.881 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.881 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ce93433:::benchmark_data_smithi171_78918_object142548:head# 2024-04-20T17:11:21.881 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.881 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ce987b8:::benchmark_data_smithi171_78918_object88649:head# 2024-04-20T17:11:21.882 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.882 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ce9dde8:::benchmark_data_smithi171_78918_object73591:head# 2024-04-20T17:11:21.882 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.882 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cea4f61:::benchmark_data_smithi171_78918_object104994:head# 2024-04-20T17:11:21.882 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.882 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ceb4724:::benchmark_data_smithi171_78918_object91085:head# 2024-04-20T17:11:21.882 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.882 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cec097e:::benchmark_data_smithi171_78918_object122007:head# 2024-04-20T17:11:21.882 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.882 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cec709d:::benchmark_data_smithi171_78918_object105935:head# 2024-04-20T17:11:21.882 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.882 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cedb906:::benchmark_data_smithi171_78918_object122028:head# 2024-04-20T17:11:21.882 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.882 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cedcbb8:::benchmark_data_smithi171_78918_object90007:head# 2024-04-20T17:11:21.882 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.883 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cee355b:::benchmark_data_smithi171_78918_object98286:head# 2024-04-20T17:11:21.883 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.883 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cef32e2:::benchmark_data_smithi171_78918_object61103:head# 2024-04-20T17:11:21.883 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.883 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cf05c4a:::benchmark_data_smithi171_78918_object104667:head# 2024-04-20T17:11:21.883 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.883 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cf07c4b:::benchmark_data_smithi171_78918_object153189:head# 2024-04-20T17:11:21.883 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.883 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cf0939f:::benchmark_data_smithi171_78918_object63801:head# 2024-04-20T17:11:21.883 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.883 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cf0a9a2:::benchmark_data_smithi171_78918_object55560:head# 2024-04-20T17:11:21.883 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.883 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cf0b434:::benchmark_data_smithi171_78918_object106100:head# 2024-04-20T17:11:21.883 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.883 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cf0ed17:::benchmark_data_smithi171_78918_object115917:head# 2024-04-20T17:11:21.884 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.884 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cf12525:::benchmark_data_smithi171_78918_object75505:head# 2024-04-20T17:11:21.884 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.884 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cf14ef7:::benchmark_data_smithi171_78918_object111654:head# 2024-04-20T17:11:21.884 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.884 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cf1bbfa:::benchmark_data_smithi171_78918_object81741:head# 2024-04-20T17:11:21.884 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.884 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cf218df:::benchmark_data_smithi171_78918_object128061:head# 2024-04-20T17:11:21.884 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.884 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cf25070:::benchmark_data_smithi171_78918_object121622:head# 2024-04-20T17:11:21.884 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.884 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cf25aea:::benchmark_data_smithi171_78918_object96973:head# 2024-04-20T17:11:21.884 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.884 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cf26421:::benchmark_data_smithi171_78918_object143175:head# 2024-04-20T17:11:21.884 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.885 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cf38f34:::benchmark_data_smithi171_78918_object106395:head# 2024-04-20T17:11:21.885 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.885 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cf3f7e3:::benchmark_data_smithi171_78918_object116617:head# 2024-04-20T17:11:21.885 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.885 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cf4271e:::benchmark_data_smithi171_78918_object143296:head# 2024-04-20T17:11:21.885 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.885 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cf68b07:::benchmark_data_smithi171_78918_object83107:head# 2024-04-20T17:11:21.885 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.885 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cf6ce58:::benchmark_data_smithi171_78918_object95359:head# 2024-04-20T17:11:21.885 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.885 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cf787b3:::benchmark_data_smithi171_78918_object103027:head# 2024-04-20T17:11:21.885 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.885 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cf80480:::benchmark_data_smithi171_78918_object68255:head# 2024-04-20T17:11:21.885 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.886 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cf83bdc:::benchmark_data_smithi171_78918_object102410:head# 2024-04-20T17:11:21.886 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.886 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cf91957:::benchmark_data_smithi171_78918_object102765:head# 2024-04-20T17:11:21.886 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.886 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cf9d8dd:::benchmark_data_smithi171_78918_object119313:head# 2024-04-20T17:11:21.886 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.886 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cf9ea3c:::benchmark_data_smithi171_78918_object129652:head# 2024-04-20T17:11:21.886 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.886 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cfa2ae4:::benchmark_data_smithi171_78918_object89463:head# 2024-04-20T17:11:21.886 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.886 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cfb188d:::benchmark_data_smithi171_78918_object73048:head# 2024-04-20T17:11:21.886 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.886 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cfb9134:::benchmark_data_smithi171_78918_object101337:head# 2024-04-20T17:11:21.887 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.887 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cfbbdca:::benchmark_data_smithi171_78918_object99763:head# 2024-04-20T17:11:21.887 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.887 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cfebc94:::benchmark_data_smithi171_78918_object123514:head# 2024-04-20T17:11:21.887 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.887 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cfeee65:::benchmark_data_smithi171_78918_object93253:head# 2024-04-20T17:11:21.887 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.887 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cfef941:::benchmark_data_smithi171_78918_object104678:head# 2024-04-20T17:11:21.887 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.887 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7cffda2d:::benchmark_data_smithi171_78918_object66745:head# 2024-04-20T17:11:21.887 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.887 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d0156bf:::benchmark_data_smithi171_78918_object140632:head# 2024-04-20T17:11:21.887 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.887 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d02df28:::benchmark_data_smithi171_78918_object158491:head# 2024-04-20T17:11:21.888 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.888 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d035b1e:::benchmark_data_smithi171_78918_object97613:head# 2024-04-20T17:11:21.888 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.888 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d044c43:::benchmark_data_smithi171_78918_object57124:head# 2024-04-20T17:11:21.888 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.888 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d049775:::benchmark_data_smithi171_78918_object60562:head# 2024-04-20T17:11:21.888 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.888 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d04b8f2:::benchmark_data_smithi171_78918_object99282:head# 2024-04-20T17:11:21.888 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.888 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d05a332:::benchmark_data_smithi171_78918_object112465:head# 2024-04-20T17:11:21.888 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.888 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d06b8a5:::benchmark_data_smithi171_78918_object115530:head# 2024-04-20T17:11:21.888 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.888 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d07c8f6:::benchmark_data_smithi171_78918_object94762:head# 2024-04-20T17:11:21.889 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.889 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d07dc8d:::benchmark_data_smithi171_78918_object70494:head# 2024-04-20T17:11:21.889 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.889 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d08cf5f:::benchmark_data_smithi171_78918_object136932:head# 2024-04-20T17:11:21.889 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.889 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d09ccaa:::benchmark_data_smithi171_78918_object75950:head# 2024-04-20T17:11:21.889 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.889 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d0a1494:::benchmark_data_smithi171_78918_object65314:head# 2024-04-20T17:11:21.889 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.889 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d0a3ee3:::benchmark_data_smithi171_78918_object109487:head# 2024-04-20T17:11:21.889 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.889 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d0b132f:::benchmark_data_smithi171_78918_object81929:head# 2024-04-20T17:11:21.889 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.889 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d0b9075:::benchmark_data_smithi171_78918_object158695:head# 2024-04-20T17:11:21.889 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.890 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d0d0868:::benchmark_data_smithi171_78918_object101495:head# 2024-04-20T17:11:21.890 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.890 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d0d4761:::benchmark_data_smithi171_78918_object101497:head# 2024-04-20T17:11:21.890 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.890 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d0d4f67:::benchmark_data_smithi171_78918_object62238:head# 2024-04-20T17:11:21.890 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.890 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d0d9fc3:::benchmark_data_smithi171_78918_object111066:head# 2024-04-20T17:11:21.890 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.890 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d0effe0:::benchmark_data_smithi171_78918_object70264:head# 2024-04-20T17:11:21.890 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.890 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d105384:::benchmark_data_smithi171_78918_object97316:head# 2024-04-20T17:11:21.890 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.890 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d108c41:::benchmark_data_smithi171_78918_object133347:head# 2024-04-20T17:11:21.891 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.891 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d111656:::benchmark_data_smithi171_78918_object86198:head# 2024-04-20T17:11:21.891 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.891 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d113741:::benchmark_data_smithi171_78918_object127508:head# 2024-04-20T17:11:21.891 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.891 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d124902:::benchmark_data_smithi171_78918_object57008:head# 2024-04-20T17:11:21.891 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.891 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d13167b:::benchmark_data_smithi171_78918_object129126:head# 2024-04-20T17:11:21.891 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.891 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d139e0d:::benchmark_data_smithi171_78918_object63636:head# 2024-04-20T17:11:21.891 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.891 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d1408b1:::benchmark_data_smithi171_78918_object97377:head# 2024-04-20T17:11:21.891 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.891 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d161ced:::benchmark_data_smithi171_78918_object139408:head# 2024-04-20T17:11:21.891 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.892 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d162cf6:::benchmark_data_smithi171_78918_object65667:head# 2024-04-20T17:11:21.892 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.892 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d165491:::benchmark_data_smithi171_78918_object75908:head# 2024-04-20T17:11:21.892 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.892 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d16680c:::benchmark_data_smithi171_78918_object91653:head# 2024-04-20T17:11:21.892 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.892 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d1694a3:::benchmark_data_smithi171_78918_object57868:head# 2024-04-20T17:11:21.892 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.892 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d16f632:::benchmark_data_smithi171_78918_object108528:head# 2024-04-20T17:11:21.892 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.892 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d174e97:::benchmark_data_smithi171_78918_object122425:head# 2024-04-20T17:11:21.892 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.892 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d17a2c4:::benchmark_data_smithi171_78918_object80524:head# 2024-04-20T17:11:21.892 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.892 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d1830aa:::benchmark_data_smithi171_78918_object79364:head# 2024-04-20T17:11:21.893 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.893 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d185e52:::benchmark_data_smithi171_78918_object63059:head# 2024-04-20T17:11:21.893 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.893 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d18727b:::benchmark_data_smithi171_78918_object60648:head# 2024-04-20T17:11:21.893 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.893 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d1984bb:::benchmark_data_smithi171_78918_object90436:head# 2024-04-20T17:11:21.893 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.893 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d19a8c7:::benchmark_data_smithi171_78918_object108608:head# 2024-04-20T17:11:21.893 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.893 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d19f4f9:::benchmark_data_smithi171_78918_object147973:head# 2024-04-20T17:11:21.893 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.893 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d1a1b34:::benchmark_data_smithi171_78918_object151920:head# 2024-04-20T17:11:21.893 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.894 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d1af9b5:::benchmark_data_smithi171_78918_object122477:head# 2024-04-20T17:11:21.894 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.894 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d1bbd0f:::benchmark_data_smithi171_78918_object101577:head# 2024-04-20T17:11:21.894 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.894 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d1bd950:::benchmark_data_smithi171_78918_object88900:head# 2024-04-20T17:11:21.894 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.894 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d1c048f:::benchmark_data_smithi171_78918_object145149:head# 2024-04-20T17:11:21.894 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.894 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d1c5431:::benchmark_data_smithi171_78918_object97704:head# 2024-04-20T17:11:21.894 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.894 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d1e5c18:::benchmark_data_smithi171_78918_object158064:head# 2024-04-20T17:11:21.894 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.894 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d1f4221:::benchmark_data_smithi171_78918_object158350:head# 2024-04-20T17:11:21.894 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.894 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d2037a9:::benchmark_data_smithi171_78918_object83682:head# 2024-04-20T17:11:21.895 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.895 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d20a03c:::benchmark_data_smithi171_78918_object138987:head# 2024-04-20T17:11:21.895 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.895 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d21ccf7:::benchmark_data_smithi171_78918_object116915:head# 2024-04-20T17:11:21.895 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.895 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d21f53f:::benchmark_data_smithi171_78918_object51846:head# 2024-04-20T17:11:21.895 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.895 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d226f34:::benchmark_data_smithi171_78918_object150659:head# 2024-04-20T17:11:21.895 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.895 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d22a3e3:::benchmark_data_smithi171_78918_object109646:head# 2024-04-20T17:11:21.895 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.895 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d22e0b0:::benchmark_data_smithi171_78918_object91276:head# 2024-04-20T17:11:21.895 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.895 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d235139:::benchmark_data_smithi171_78918_object57972:head# 2024-04-20T17:11:21.896 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.896 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d23def6:::benchmark_data_smithi171_78918_object131823:head# 2024-04-20T17:11:21.896 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.896 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d24e018:::benchmark_data_smithi171_78918_object127168:head# 2024-04-20T17:11:21.896 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.896 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d24ee04:::benchmark_data_smithi171_78918_object110920:head# 2024-04-20T17:11:21.896 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.896 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d252c4a:::benchmark_data_smithi171_78918_object135583:head# 2024-04-20T17:11:21.896 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.896 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d252e78:::benchmark_data_smithi171_78918_object52203:head# 2024-04-20T17:11:21.896 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.896 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d25897e:::benchmark_data_smithi171_78918_object64819:head# 2024-04-20T17:11:21.896 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.896 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d259341:::benchmark_data_smithi171_78918_object73852:head# 2024-04-20T17:11:21.897 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.897 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d25b964:::benchmark_data_smithi171_78918_object115760:head# 2024-04-20T17:11:21.897 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.897 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d269800:::benchmark_data_smithi171_78918_object74167:head# 2024-04-20T17:11:21.897 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.897 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d26fe58:::benchmark_data_smithi171_78918_object130373:head# 2024-04-20T17:11:21.897 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.897 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d28856d:::benchmark_data_smithi171_78918_object106880:head# 2024-04-20T17:11:21.897 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.897 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d294e30:::benchmark_data_smithi171_78918_object98149:head# 2024-04-20T17:11:21.897 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.897 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d29c4a7:::benchmark_data_smithi171_78918_object62055:head# 2024-04-20T17:11:21.897 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.897 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d29de13:::benchmark_data_smithi171_78918_object130311:head# 2024-04-20T17:11:21.897 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.897 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d2a1d00:::benchmark_data_smithi171_78918_object150334:head# 2024-04-20T17:11:21.898 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.898 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d2ab24c:::benchmark_data_smithi171_78918_object61990:head# 2024-04-20T17:11:21.898 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.898 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d2af757:::benchmark_data_smithi171_78918_object132646:head# 2024-04-20T17:11:21.898 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.898 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d2b91ab:::benchmark_data_smithi171_78918_object58121:head# 2024-04-20T17:11:21.898 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.898 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d2c5fa0:::benchmark_data_smithi171_78918_object63944:head# 2024-04-20T17:11:21.898 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.898 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d2c7dbe:::benchmark_data_smithi171_78918_object108204:head# 2024-04-20T17:11:21.898 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.898 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d2c9d98:::benchmark_data_smithi171_78918_object125553:head# 2024-04-20T17:11:21.898 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.898 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d2d2c7a:::benchmark_data_smithi171_78918_object108978:head# 2024-04-20T17:11:21.899 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.899 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d2dd40b:::benchmark_data_smithi171_78918_object108750:head# 2024-04-20T17:11:21.899 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.899 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d2de0b6:::benchmark_data_smithi171_78918_object123670:head# 2024-04-20T17:11:21.899 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.899 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d2df290:::benchmark_data_smithi171_78918_object73442:head# 2024-04-20T17:11:21.899 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.899 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d2eac87:::benchmark_data_smithi171_78918_object116421:head# 2024-04-20T17:11:21.899 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.899 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d300538:::benchmark_data_smithi171_78918_object84951:head# 2024-04-20T17:11:21.899 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.899 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d3048bc:::benchmark_data_smithi171_78918_object130224:head# 2024-04-20T17:11:21.899 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.899 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d308df4:::benchmark_data_smithi171_78918_object105355:head# 2024-04-20T17:11:21.899 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.900 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d30b2b8:::benchmark_data_smithi171_78918_object158606:head# 2024-04-20T17:11:21.900 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.900 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d30cd8b:::benchmark_data_smithi171_78918_object65077:head# 2024-04-20T17:11:21.900 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.900 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d317762:::benchmark_data_smithi171_78918_object75611:head# 2024-04-20T17:11:21.900 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.900 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d31b2ac:::benchmark_data_smithi171_78918_object138091:head# 2024-04-20T17:11:21.900 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.900 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d31fce1:::benchmark_data_smithi171_78918_object127811:head# 2024-04-20T17:11:21.900 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.900 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d32903d:::benchmark_data_smithi171_78918_object119894:head# 2024-04-20T17:11:21.900 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.900 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d33bda7:::benchmark_data_smithi171_78918_object123444:head# 2024-04-20T17:11:21.900 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.901 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d349713:::benchmark_data_smithi171_78918_object126264:head# 2024-04-20T17:11:21.901 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.901 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d34c3af:::benchmark_data_smithi171_78918_object132455:head# 2024-04-20T17:11:21.901 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.901 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d355592:::benchmark_data_smithi171_78918_object80741:head# 2024-04-20T17:11:21.901 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.901 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d358d92:::benchmark_data_smithi171_78918_object70720:head# 2024-04-20T17:11:21.901 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.901 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d36a98a:::benchmark_data_smithi171_78918_object108782:head# 2024-04-20T17:11:21.901 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.901 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d36dd27:::benchmark_data_smithi171_78918_object104208:head# 2024-04-20T17:11:21.901 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.901 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d376563:::benchmark_data_smithi171_78918_object61245:head# 2024-04-20T17:11:21.901 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.901 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d380e6a:::benchmark_data_smithi171_78918_object69386:head# 2024-04-20T17:11:21.902 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.902 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d391c1b:::benchmark_data_smithi171_78918_object100676:head# 2024-04-20T17:11:21.902 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.902 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d39e72e:::benchmark_data_smithi171_78918_object108191:head# 2024-04-20T17:11:21.902 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.902 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d3aedc8:::benchmark_data_smithi171_78918_object79739:head# 2024-04-20T17:11:21.902 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.902 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d3c6064:::benchmark_data_smithi171_78918_object108205:head# 2024-04-20T17:11:21.902 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.902 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d3ccbbc:::benchmark_data_smithi171_78918_object133322:head# 2024-04-20T17:11:21.902 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.902 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d3ce6e1:::benchmark_data_smithi171_78918_object145231:head# 2024-04-20T17:11:21.902 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.903 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d3cec1a:::benchmark_data_smithi171_78918_object109571:head# 2024-04-20T17:11:21.903 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.903 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d3e309d:::benchmark_data_smithi171_78918_object143406:head# 2024-04-20T17:11:21.903 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.903 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d3ea5b4:::benchmark_data_smithi171_78918_object52247:head# 2024-04-20T17:11:21.903 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.903 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d3eff99:::benchmark_data_smithi171_78918_object113571:head# 2024-04-20T17:11:21.903 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.903 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d3f6c74:::benchmark_data_smithi171_78918_object71948:head# 2024-04-20T17:11:21.903 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.903 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d3f9d9e:::benchmark_data_smithi171_78918_object72879:head# 2024-04-20T17:11:21.903 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.903 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d4068f0:::benchmark_data_smithi171_78918_object114666:head# 2024-04-20T17:11:21.904 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.904 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d407772:::benchmark_data_smithi171_78918_object115065:head# 2024-04-20T17:11:21.904 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.904 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d40c941:::benchmark_data_smithi171_78918_object67852:head# 2024-04-20T17:11:21.904 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.904 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d40cd17:::benchmark_data_smithi171_78918_object137885:head# 2024-04-20T17:11:21.904 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.904 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d40f1fc:::benchmark_data_smithi171_78918_object76243:head# 2024-04-20T17:11:21.904 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.904 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d4148c7:::benchmark_data_smithi171_78918_object156366:head# 2024-04-20T17:11:21.904 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.904 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d419604:::benchmark_data_smithi171_78918_object103001:head# 2024-04-20T17:11:21.904 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.904 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d41fb95:::benchmark_data_smithi171_78918_object54693:head# 2024-04-20T17:11:21.904 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.905 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d43cf43:::benchmark_data_smithi171_78918_object76010:head# 2024-04-20T17:11:21.905 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.905 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d44af4e:::benchmark_data_smithi171_78918_object113584:head# 2024-04-20T17:11:21.905 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.905 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d4538ff:::benchmark_data_smithi171_78918_object108625:head# 2024-04-20T17:11:21.905 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.905 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d466012:::benchmark_data_smithi171_78918_object70671:head# 2024-04-20T17:11:21.905 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.905 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d494529:::benchmark_data_smithi171_78918_object57083:head# 2024-04-20T17:11:21.905 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.905 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d4a08c8:::benchmark_data_smithi171_78918_object84194:head# 2024-04-20T17:11:21.905 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.905 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d4aa2a7:::benchmark_data_smithi171_78918_object83302:head# 2024-04-20T17:11:21.905 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.906 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d4b080f:::benchmark_data_smithi171_78918_object105525:head# 2024-04-20T17:11:21.906 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.906 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d4c3963:::benchmark_data_smithi171_78918_object119719:head# 2024-04-20T17:11:21.906 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.906 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d4c80db:::benchmark_data_smithi171_78918_object152434:head# 2024-04-20T17:11:21.906 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.906 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d4cefac:::benchmark_data_smithi171_78918_object55164:head# 2024-04-20T17:11:21.906 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.906 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d4d704c:::benchmark_data_smithi171_78918_object116778:head# 2024-04-20T17:11:21.906 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.906 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d4dae9e:::benchmark_data_smithi171_78918_object69413:head# 2024-04-20T17:11:21.906 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.906 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d4dd2c9:::benchmark_data_smithi171_78918_object125769:head# 2024-04-20T17:11:21.906 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.907 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d4e59c6:::benchmark_data_smithi171_78918_object56267:head# 2024-04-20T17:11:21.907 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.907 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d4fbe9e:::benchmark_data_smithi171_78918_object122681:head# 2024-04-20T17:11:21.907 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.907 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d4feacc:::benchmark_data_smithi171_78918_object82358:head# 2024-04-20T17:11:21.907 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.907 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d506516:::benchmark_data_smithi171_78918_object106188:head# 2024-04-20T17:11:21.907 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.907 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d516613:::benchmark_data_smithi171_78918_object61327:head# 2024-04-20T17:11:21.907 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.907 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d524295:::benchmark_data_smithi171_78918_object127714:head# 2024-04-20T17:11:21.907 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.907 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d524b4d:::benchmark_data_smithi171_78918_object87086:head# 2024-04-20T17:11:21.907 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.907 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d52de91:::benchmark_data_smithi171_78918_object74209:head# 2024-04-20T17:11:21.908 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.908 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d53bdbc:::benchmark_data_smithi171_78918_object115359:head# 2024-04-20T17:11:21.908 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.908 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d53d863:::benchmark_data_smithi171_78918_object124077:head# 2024-04-20T17:11:21.908 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.908 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d556759:::benchmark_data_smithi171_78918_object58145:head# 2024-04-20T17:11:21.908 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.908 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d55b4b1:::benchmark_data_smithi171_78918_object99588:head# 2024-04-20T17:11:21.908 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.908 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d55d48a:::benchmark_data_smithi171_78918_object92040:head# 2024-04-20T17:11:21.908 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.908 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d56053e:::benchmark_data_smithi171_78918_object94826:head# 2024-04-20T17:11:21.908 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.908 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d57315b:::benchmark_data_smithi171_78918_object130617:head# 2024-04-20T17:11:21.909 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.909 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d577491:::benchmark_data_smithi171_78918_object108096:head# 2024-04-20T17:11:21.909 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.909 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d57846d:::benchmark_data_smithi171_78918_object86611:head# 2024-04-20T17:11:21.909 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.909 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d586d46:::benchmark_data_smithi171_78918_object147519:head# 2024-04-20T17:11:21.909 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.909 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d58d052:::benchmark_data_smithi171_78918_object64064:head# 2024-04-20T17:11:21.909 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.909 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d593344:::benchmark_data_smithi171_78918_object144081:head# 2024-04-20T17:11:21.909 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.909 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d5977fa:::benchmark_data_smithi171_78918_object120913:head# 2024-04-20T17:11:21.909 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.909 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d59a7c2:::benchmark_data_smithi171_78918_object131692:head# 2024-04-20T17:11:21.909 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.910 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d59febd:::benchmark_data_smithi171_78918_object73906:head# 2024-04-20T17:11:21.910 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.910 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d5b345c:::benchmark_data_smithi171_78918_object151569:head# 2024-04-20T17:11:21.910 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.910 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d5b866c:::benchmark_data_smithi171_78918_object51797:head# 2024-04-20T17:11:21.910 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.910 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d5c44ab:::benchmark_data_smithi171_78918_object148551:head# 2024-04-20T17:11:21.910 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.910 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d5d3fe7:::benchmark_data_smithi171_78918_object71940:head# 2024-04-20T17:11:21.910 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.910 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d5d98d3:::benchmark_data_smithi171_78918_object160734:head# 2024-04-20T17:11:21.910 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.910 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d5dce2e:::benchmark_data_smithi171_78918_object65340:head# 2024-04-20T17:11:21.910 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.910 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d5e629b:::benchmark_data_smithi171_78918_object94070:head# 2024-04-20T17:11:21.911 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.911 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d5f9fb8:::benchmark_data_smithi171_78918_object160646:head# 2024-04-20T17:11:21.911 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.911 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d6092f6:::benchmark_data_smithi171_78918_object130276:head# 2024-04-20T17:11:21.911 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.911 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d60f8a9:::benchmark_data_smithi171_78918_object156110:head# 2024-04-20T17:11:21.911 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.911 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d61d4b7:::benchmark_data_smithi171_78918_object153414:head# 2024-04-20T17:11:21.911 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.911 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d61d842:::benchmark_data_smithi171_78918_object152290:head# 2024-04-20T17:11:21.911 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.911 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d61dea3:::benchmark_data_smithi171_78918_object68128:head# 2024-04-20T17:11:21.911 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.911 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d620944:::benchmark_data_smithi171_78918_object120975:head# 2024-04-20T17:11:21.912 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.912 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d638429:::benchmark_data_smithi171_78918_object149473:head# 2024-04-20T17:11:21.912 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.912 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d6435b2:::benchmark_data_smithi171_78918_object118030:head# 2024-04-20T17:11:21.912 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.912 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d643c48:::benchmark_data_smithi171_78918_object60980:head# 2024-04-20T17:11:21.912 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.912 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d6502a8:::benchmark_data_smithi171_78918_object78313:head# 2024-04-20T17:11:21.912 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.912 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d65b927:::benchmark_data_smithi171_78918_object146225:head# 2024-04-20T17:11:21.912 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.912 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d65bae8:::benchmark_data_smithi171_78918_object129397:head# 2024-04-20T17:11:21.912 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.912 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d662e4d:::benchmark_data_smithi171_78918_object155536:head# 2024-04-20T17:11:21.913 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.913 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d66bbce:::benchmark_data_smithi171_78918_object111941:head# 2024-04-20T17:11:21.913 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.913 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d67184e:::benchmark_data_smithi171_78918_object77232:head# 2024-04-20T17:11:21.913 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.913 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d677e76:::benchmark_data_smithi171_78918_object57184:head# 2024-04-20T17:11:21.913 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.913 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d682021:::benchmark_data_smithi171_78918_object158226:head# 2024-04-20T17:11:21.913 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.913 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d6893c1:::benchmark_data_smithi171_78918_object94844:head# 2024-04-20T17:11:21.913 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.913 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d699c9f:::benchmark_data_smithi171_78918_object78489:head# 2024-04-20T17:11:21.913 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.913 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d69a137:::benchmark_data_smithi171_78918_object87377:head# 2024-04-20T17:11:21.913 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.914 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d6aee9d:::benchmark_data_smithi171_78918_object149374:head# 2024-04-20T17:11:21.914 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.914 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d6c5d33:::benchmark_data_smithi171_78918_object87486:head# 2024-04-20T17:11:21.914 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.914 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d6c8d99:::benchmark_data_smithi171_78918_object98675:head# 2024-04-20T17:11:21.914 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.914 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d6d801a:::benchmark_data_smithi171_78918_object129340:head# 2024-04-20T17:11:21.914 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.914 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d6dd465:::benchmark_data_smithi171_78918_object78899:head# 2024-04-20T17:11:21.914 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.914 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d6eb1ff:::benchmark_data_smithi171_78918_object52079:head# 2024-04-20T17:11:21.914 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.914 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d6ec749:::benchmark_data_smithi171_78918_object116979:head# 2024-04-20T17:11:21.914 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.915 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d6f526d:::benchmark_data_smithi171_78918_object145576:head# 2024-04-20T17:11:21.915 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.915 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d6f6f82:::benchmark_data_smithi171_78918_object154657:head# 2024-04-20T17:11:21.915 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.915 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d6fa128:::benchmark_data_smithi171_78918_object108697:head# 2024-04-20T17:11:21.915 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.915 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d704380:::benchmark_data_smithi171_78918_object66405:head# 2024-04-20T17:11:21.915 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.915 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d7115f1:::benchmark_data_smithi171_78918_object110157:head# 2024-04-20T17:11:21.915 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.915 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d725acd:::benchmark_data_smithi171_78918_object160874:head# 2024-04-20T17:11:21.915 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.915 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d73731a:::benchmark_data_smithi171_78918_object123913:head# 2024-04-20T17:11:21.916 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.916 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d74005a:::benchmark_data_smithi171_78918_object64197:head# 2024-04-20T17:11:21.916 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.916 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d743e69:::benchmark_data_smithi171_78918_object98117:head# 2024-04-20T17:11:21.916 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.916 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d758c93:::benchmark_data_smithi171_78918_object85982:head# 2024-04-20T17:11:21.916 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.916 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d75a47d:::benchmark_data_smithi171_78918_object110650:head# 2024-04-20T17:11:21.916 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.916 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d76f612:::benchmark_data_smithi171_78918_object131383:head# 2024-04-20T17:11:21.916 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.916 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d76fbbe:::benchmark_data_smithi171_78918_object70067:head# 2024-04-20T17:11:21.916 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.916 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d7752a1:::benchmark_data_smithi171_78918_object84844:head# 2024-04-20T17:11:21.916 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.917 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d78d5c1:::benchmark_data_smithi171_78918_object139730:head# 2024-04-20T17:11:21.917 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.917 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d799a42:::benchmark_data_smithi171_78918_object79614:head# 2024-04-20T17:11:21.917 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.917 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d7a45a6:::benchmark_data_smithi171_78918_object77484:head# 2024-04-20T17:11:21.917 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.917 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d7ba25d:::benchmark_data_smithi171_78918_object101476:head# 2024-04-20T17:11:21.917 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.917 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d7bb5cb:::benchmark_data_smithi171_78918_object118529:head# 2024-04-20T17:11:21.917 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.917 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d7bfdac:::benchmark_data_smithi171_78918_object116028:head# 2024-04-20T17:11:21.917 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.917 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d7ce577:::benchmark_data_smithi171_78918_object60025:head# 2024-04-20T17:11:21.917 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.917 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d7de219:::benchmark_data_smithi171_78918_object93053:head# 2024-04-20T17:11:21.918 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.918 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d7ebb2d:::benchmark_data_smithi171_78918_object143051:head# 2024-04-20T17:11:21.918 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.918 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d7ec13b:::benchmark_data_smithi171_78918_object130369:head# 2024-04-20T17:11:21.918 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.918 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d8140dc:::benchmark_data_smithi171_78918_object77985:head# 2024-04-20T17:11:21.918 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.918 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d82f644:::benchmark_data_smithi171_78918_object105308:head# 2024-04-20T17:11:21.918 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.918 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d8429cb:::benchmark_data_smithi171_78918_object90506:head# 2024-04-20T17:11:21.918 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.918 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d843b11:::benchmark_data_smithi171_78918_object129332:head# 2024-04-20T17:11:21.918 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.918 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d84a79f:::benchmark_data_smithi171_78918_object86295:head# 2024-04-20T17:11:21.919 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.919 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d84f251:::benchmark_data_smithi171_78918_object118129:head# 2024-04-20T17:11:21.919 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.919 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d8572e3:::benchmark_data_smithi171_78918_object133061:head# 2024-04-20T17:11:21.919 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.919 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d85b4d2:::benchmark_data_smithi171_78918_object154969:head# 2024-04-20T17:11:21.919 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.919 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d85e126:::benchmark_data_smithi171_78918_object53820:head# 2024-04-20T17:11:21.919 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.919 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d861cb5:::benchmark_data_smithi171_78918_object135234:head# 2024-04-20T17:11:21.919 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.919 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d863d39:::benchmark_data_smithi171_78918_object137625:head# 2024-04-20T17:11:21.919 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.919 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d86444c:::benchmark_data_smithi171_78918_object123526:head# 2024-04-20T17:11:21.919 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.920 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d86b4af:::benchmark_data_smithi171_78918_object95664:head# 2024-04-20T17:11:21.920 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.920 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d8714a0:::benchmark_data_smithi171_78918_object124684:head# 2024-04-20T17:11:21.920 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.920 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d87b64a:::benchmark_data_smithi171_78918_object74104:head# 2024-04-20T17:11:21.920 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.920 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d880e11:::benchmark_data_smithi171_78918_object116429:head# 2024-04-20T17:11:21.920 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.920 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d881f27:::benchmark_data_smithi171_78918_object74218:head# 2024-04-20T17:11:21.920 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.920 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d883aa4:::benchmark_data_smithi171_78918_object104151:head# 2024-04-20T17:11:21.920 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.920 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d88e73f:::benchmark_data_smithi171_78918_object73340:head# 2024-04-20T17:11:21.920 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.920 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d88e905:::benchmark_data_smithi171_78918_object56059:head# 2024-04-20T17:11:21.921 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.921 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d8967ab:::benchmark_data_smithi171_78918_object65974:head# 2024-04-20T17:11:21.921 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.921 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d8a3bc5:::benchmark_data_smithi171_78918_object75571:head# 2024-04-20T17:11:21.921 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.921 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d8b5372:::benchmark_data_smithi171_78918_object115889:head# 2024-04-20T17:11:21.921 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.921 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d8c1254:::benchmark_data_smithi171_78918_object115166:head# 2024-04-20T17:11:21.921 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.921 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d8c7b70:::benchmark_data_smithi171_78918_object99611:head# 2024-04-20T17:11:21.921 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.921 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d8cecf1:::benchmark_data_smithi171_78918_object87780:head# 2024-04-20T17:11:21.921 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.921 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d8d241a:::benchmark_data_smithi171_78918_object99745:head# 2024-04-20T17:11:21.922 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.922 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d8e1628:::benchmark_data_smithi171_78918_object63456:head# 2024-04-20T17:11:21.922 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.922 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d8e407a:::benchmark_data_smithi171_78918_object101180:head# 2024-04-20T17:11:21.922 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.922 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d8e5008:::benchmark_data_smithi171_78918_object88377:head# 2024-04-20T17:11:21.922 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.922 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d8ee0a7:::benchmark_data_smithi171_78918_object95811:head# 2024-04-20T17:11:21.922 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.922 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d8f4dad:::benchmark_data_smithi171_78918_object94963:head# 2024-04-20T17:11:21.922 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.922 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d905bad:::benchmark_data_smithi171_78918_object113329:head# 2024-04-20T17:11:21.922 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.922 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d91715a:::benchmark_data_smithi171_78918_object139020:head# 2024-04-20T17:11:21.922 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.923 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d920235:::benchmark_data_smithi171_78918_object67186:head# 2024-04-20T17:11:21.923 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.923 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d922859:::benchmark_data_smithi171_78918_object86180:head# 2024-04-20T17:11:21.923 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.923 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d9412df:::benchmark_data_smithi171_78918_object66942:head# 2024-04-20T17:11:21.923 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.923 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d951415:::benchmark_data_smithi171_78918_object127467:head# 2024-04-20T17:11:21.923 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.923 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d95f675:::benchmark_data_smithi171_78918_object86444:head# 2024-04-20T17:11:21.923 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.923 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d96948f:::benchmark_data_smithi171_78918_object119850:head# 2024-04-20T17:11:21.923 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.923 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d982014:::benchmark_data_smithi171_78918_object86167:head# 2024-04-20T17:11:21.923 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.924 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d9851c7:::benchmark_data_smithi171_78918_object74170:head# 2024-04-20T17:11:21.924 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.924 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d9876ef:::benchmark_data_smithi171_78918_object144458:head# 2024-04-20T17:11:21.924 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.924 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d9aec82:::benchmark_data_smithi171_78918_object80333:head# 2024-04-20T17:11:21.924 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.924 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d9b2f15:::benchmark_data_smithi171_78918_object118516:head# 2024-04-20T17:11:21.924 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.924 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d9c601b:::benchmark_data_smithi171_78918_object66550:head# 2024-04-20T17:11:21.924 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.924 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d9d0c97:::benchmark_data_smithi171_78918_object106753:head# 2024-04-20T17:11:21.924 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.924 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d9d4850:::benchmark_data_smithi171_78918_object54048:head# 2024-04-20T17:11:21.924 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.925 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d9dfbeb:::benchmark_data_smithi171_78918_object121361:head# 2024-04-20T17:11:21.925 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:21.926 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:11:22.023 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d9f222d:::benchmark_data_smithi171_78918_object119552:head# 2024-04-20T17:11:22.023 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.023 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7d9f6228:::benchmark_data_smithi171_78918_object80467:head# 2024-04-20T17:11:22.023 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.023 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7da0c1fe:::benchmark_data_smithi171_78918_object129681:head# 2024-04-20T17:11:22.023 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.024 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7da1b0bf:::benchmark_data_smithi171_78918_object150678:head# 2024-04-20T17:11:22.024 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.024 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7da1d2f3:::benchmark_data_smithi171_78918_object138498:head# 2024-04-20T17:11:22.024 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.024 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7da1dc0b:::benchmark_data_smithi171_78918_object81343:head# 2024-04-20T17:11:22.024 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.024 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7da208a7:::benchmark_data_smithi171_78918_object74744:head# 2024-04-20T17:11:22.024 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.024 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7da23e70:::benchmark_data_smithi171_78918_object143957:head# 2024-04-20T17:11:22.024 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.024 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7da24a35:::benchmark_data_smithi171_78918_object119874:head# 2024-04-20T17:11:22.024 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.024 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7da34de4:::benchmark_data_smithi171_78918_object152964:head# 2024-04-20T17:11:22.024 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.025 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7da3ef2d:::benchmark_data_smithi171_78918_object61970:head# 2024-04-20T17:11:22.025 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.025 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7da4b643:::benchmark_data_smithi171_78918_object150016:head# 2024-04-20T17:11:22.025 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.025 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7da4da83:::benchmark_data_smithi171_78918_object158745:head# 2024-04-20T17:11:22.025 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.025 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7da635b0:::benchmark_data_smithi171_78918_object75115:head# 2024-04-20T17:11:22.025 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.025 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7da8b751:::benchmark_data_smithi171_78918_object83758:head# 2024-04-20T17:11:22.025 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.025 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7da99c2a:::benchmark_data_smithi171_78918_object93257:head# 2024-04-20T17:11:22.025 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.025 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7daa8f03:::benchmark_data_smithi171_78918_object61707:head# 2024-04-20T17:11:22.025 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.025 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7daaaeb2:::benchmark_data_smithi171_78918_object132483:head# 2024-04-20T17:11:22.026 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.026 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dad59a8:::benchmark_data_smithi171_78918_object75112:head# 2024-04-20T17:11:22.026 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.026 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dad8442:::benchmark_data_smithi171_78918_object113807:head# 2024-04-20T17:11:22.026 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.026 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7db0c9f9:::benchmark_data_smithi171_78918_object131877:head# 2024-04-20T17:11:22.026 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.026 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7db12f0e:::benchmark_data_smithi171_78918_object144437:head# 2024-04-20T17:11:22.026 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.026 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7db20459:::benchmark_data_smithi171_78918_object156212:head# 2024-04-20T17:11:22.026 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.026 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7db30299:::benchmark_data_smithi171_78918_object104854:head# 2024-04-20T17:11:22.026 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.026 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7db39588:::benchmark_data_smithi171_78918_object116025:head# 2024-04-20T17:11:22.027 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.027 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7db3df0d:::benchmark_data_smithi171_78918_object124147:head# 2024-04-20T17:11:22.027 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.027 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7db4a419:::benchmark_data_smithi171_78918_object132612:head# 2024-04-20T17:11:22.027 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.027 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7db510d0:::benchmark_data_smithi171_78918_object60199:head# 2024-04-20T17:11:22.027 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.027 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7db5516b:::benchmark_data_smithi171_78918_object83368:head# 2024-04-20T17:11:22.027 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.027 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7db66b31:::benchmark_data_smithi171_78918_object133792:head# 2024-04-20T17:11:22.027 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.027 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7db6df7a:::benchmark_data_smithi171_78918_object100461:head# 2024-04-20T17:11:22.027 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.027 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7db972db:::benchmark_data_smithi171_78918_object97324:head# 2024-04-20T17:11:22.027 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.028 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dba0468:::benchmark_data_smithi171_78918_object156214:head# 2024-04-20T17:11:22.028 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.028 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dbc88c9:::benchmark_data_smithi171_78918_object144849:head# 2024-04-20T17:11:22.028 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.028 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dbcfab7:::benchmark_data_smithi171_78918_object101592:head# 2024-04-20T17:11:22.028 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.028 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dbe5bce:::benchmark_data_smithi171_78918_object52818:head# 2024-04-20T17:11:22.028 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.028 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dbebea0:::benchmark_data_smithi171_78918_object94681:head# 2024-04-20T17:11:22.028 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.028 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dbf6fa2:::benchmark_data_smithi171_78918_object92402:head# 2024-04-20T17:11:22.028 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.028 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dbfb53c:::benchmark_data_smithi171_78918_object145617:head# 2024-04-20T17:11:22.028 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.028 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dc01b80:::benchmark_data_smithi171_78918_object102416:head# 2024-04-20T17:11:22.029 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.029 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dc19f5d:::benchmark_data_smithi171_78918_object85755:head# 2024-04-20T17:11:22.029 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.029 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dc2356e:::benchmark_data_smithi171_78918_object100804:head# 2024-04-20T17:11:22.029 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.029 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dc27be2:::benchmark_data_smithi171_78918_object146321:head# 2024-04-20T17:11:22.029 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.029 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dc2cc11:::benchmark_data_smithi171_78918_object142800:head# 2024-04-20T17:11:22.029 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.029 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dc34a84:::benchmark_data_smithi171_78918_object81697:head# 2024-04-20T17:11:22.029 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.029 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dc3b529:::benchmark_data_smithi171_78918_object140013:head# 2024-04-20T17:11:22.029 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.029 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dc44643:::benchmark_data_smithi171_78918_object141671:head# 2024-04-20T17:11:22.030 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.030 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dc546ee:::benchmark_data_smithi171_78918_object143690:head# 2024-04-20T17:11:22.030 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.030 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dc57482:::benchmark_data_smithi171_78918_object67991:head# 2024-04-20T17:11:22.030 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.030 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dc5b32c:::benchmark_data_smithi171_78918_object105054:head# 2024-04-20T17:11:22.030 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.030 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dc5fca2:::benchmark_data_smithi171_78918_object100409:head# 2024-04-20T17:11:22.030 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.030 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dc61ed3:::benchmark_data_smithi171_78918_object99956:head# 2024-04-20T17:11:22.030 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.030 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dc9c99a:::benchmark_data_smithi171_78918_object151856:head# 2024-04-20T17:11:22.030 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.030 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dcaa244:::benchmark_data_smithi171_78918_object155903:head# 2024-04-20T17:11:22.030 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.031 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dcabd6c:::benchmark_data_smithi171_78918_object82431:head# 2024-04-20T17:11:22.031 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.031 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dcaeed1:::benchmark_data_smithi171_78918_object156610:head# 2024-04-20T17:11:22.031 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.031 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dcb064b:::benchmark_data_smithi171_78918_object129000:head# 2024-04-20T17:11:22.031 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.031 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dcb64ed:::benchmark_data_smithi171_78918_object98451:head# 2024-04-20T17:11:22.031 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.031 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dcbac86:::benchmark_data_smithi171_78918_object103886:head# 2024-04-20T17:11:22.031 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.031 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dcbc8b8:::benchmark_data_smithi171_78918_object128731:head# 2024-04-20T17:11:22.031 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.031 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dcbe101:::benchmark_data_smithi171_78918_object79423:head# 2024-04-20T17:11:22.032 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.032 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dcc59ed:::benchmark_data_smithi171_78918_object85246:head# 2024-04-20T17:11:22.032 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.032 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dcf4aaf:::benchmark_data_smithi171_78918_object100878:head# 2024-04-20T17:11:22.032 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.032 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dd023a5:::benchmark_data_smithi171_78918_object146093:head# 2024-04-20T17:11:22.032 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.032 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dd0a3dc:::benchmark_data_smithi171_78918_object145161:head# 2024-04-20T17:11:22.032 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.032 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dd19eb8:::benchmark_data_smithi171_78918_object115961:head# 2024-04-20T17:11:22.032 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.032 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dd2819d:::benchmark_data_smithi171_78918_object123100:head# 2024-04-20T17:11:22.032 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.032 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dd2a69a:::benchmark_data_smithi171_78918_object119227:head# 2024-04-20T17:11:22.032 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.033 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dd388a4:::benchmark_data_smithi171_78918_object57509:head# 2024-04-20T17:11:22.033 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.033 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dd4e6c2:::benchmark_data_smithi171_78918_object101648:head# 2024-04-20T17:11:22.033 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.033 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dd5510a:::benchmark_data_smithi171_78918_object62744:head# 2024-04-20T17:11:22.033 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.033 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dd67b84:::benchmark_data_smithi171_78918_object139524:head# 2024-04-20T17:11:22.033 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.033 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dd7046d:::benchmark_data_smithi171_78918_object129767:head# 2024-04-20T17:11:22.033 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.033 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dd76ed4:::benchmark_data_smithi171_78918_object92631:head# 2024-04-20T17:11:22.033 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.033 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dd80e5b:::benchmark_data_smithi171_78918_object75469:head# 2024-04-20T17:11:22.033 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.033 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dd847da:::benchmark_data_smithi171_78918_object80171:head# 2024-04-20T17:11:22.034 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.034 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dd872dd:::benchmark_data_smithi171_78918_object123679:head# 2024-04-20T17:11:22.034 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.034 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dd8e944:::benchmark_data_smithi171_78918_object109213:head# 2024-04-20T17:11:22.034 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.034 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dd9ec36:::benchmark_data_smithi171_78918_object70213:head# 2024-04-20T17:11:22.034 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.034 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dda8567:::benchmark_data_smithi171_78918_object122200:head# 2024-04-20T17:11:22.034 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.034 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ddacd04:::benchmark_data_smithi171_78918_object81590:head# 2024-04-20T17:11:22.034 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.034 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ddb1651:::benchmark_data_smithi171_78918_object120716:head# 2024-04-20T17:11:22.034 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.034 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ddbafe9:::benchmark_data_smithi171_78918_object87633:head# 2024-04-20T17:11:22.035 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.035 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ddbcf32:::benchmark_data_smithi171_78918_object135548:head# 2024-04-20T17:11:22.035 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.035 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ddbe32d:::benchmark_data_smithi171_78918_object114235:head# 2024-04-20T17:11:22.035 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.035 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ddc0e57:::benchmark_data_smithi171_78918_object109952:head# 2024-04-20T17:11:22.035 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.035 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ddc3bbc:::benchmark_data_smithi171_78918_object130303:head# 2024-04-20T17:11:22.035 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.035 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ddc52e7:::benchmark_data_smithi171_78918_object75897:head# 2024-04-20T17:11:22.035 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.035 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ddcf85e:::benchmark_data_smithi171_78918_object131469:head# 2024-04-20T17:11:22.035 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.035 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ddcfd87:::benchmark_data_smithi171_78918_object58390:head# 2024-04-20T17:11:22.035 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.036 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ddd8595:::benchmark_data_smithi171_78918_object110976:head# 2024-04-20T17:11:22.036 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.036 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ddde422:::benchmark_data_smithi171_78918_object129532:head# 2024-04-20T17:11:22.036 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.036 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dddec9e:::benchmark_data_smithi171_78918_object104884:head# 2024-04-20T17:11:22.036 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.036 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dde50ae:::benchmark_data_smithi171_78918_object126349:head# 2024-04-20T17:11:22.036 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.036 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dde5c71:::benchmark_data_smithi171_78918_object142835:head# 2024-04-20T17:11:22.036 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.036 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dde8a03:::benchmark_data_smithi171_78918_object103086:head# 2024-04-20T17:11:22.036 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.036 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ddede20:::benchmark_data_smithi171_78918_object147658:head# 2024-04-20T17:11:22.036 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.037 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ddf1b37:::benchmark_data_smithi171_78918_object112434:head# 2024-04-20T17:11:22.037 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.037 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ddf7df7:::benchmark_data_smithi171_78918_object65095:head# 2024-04-20T17:11:22.037 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.037 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7de044f8:::benchmark_data_smithi171_78918_object90258:head# 2024-04-20T17:11:22.037 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.037 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7de0ffac:::benchmark_data_smithi171_78918_object80652:head# 2024-04-20T17:11:22.037 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.037 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7de1843a:::benchmark_data_smithi171_78918_object86095:head# 2024-04-20T17:11:22.037 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.037 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7de29f0d:::benchmark_data_smithi171_78918_object63300:head# 2024-04-20T17:11:22.037 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.037 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7de2aa17:::benchmark_data_smithi171_78918_object133785:head# 2024-04-20T17:11:22.037 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.037 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7de36593:::benchmark_data_smithi171_78918_object66094:head# 2024-04-20T17:11:22.037 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.038 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7de3b974:::benchmark_data_smithi171_78918_object104080:head# 2024-04-20T17:11:22.038 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.038 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7de402c7:::benchmark_data_smithi171_78918_object131596:head# 2024-04-20T17:11:22.038 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.038 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7de48837:::benchmark_data_smithi171_78918_object71828:head# 2024-04-20T17:11:22.038 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.038 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7de4abea:::benchmark_data_smithi171_78918_object75338:head# 2024-04-20T17:11:22.038 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.038 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7de4b8cb:::benchmark_data_smithi171_78918_object71676:head# 2024-04-20T17:11:22.038 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.038 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7de5b257:::benchmark_data_smithi171_78918_object80686:head# 2024-04-20T17:11:22.038 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.038 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7de5e5a3:::benchmark_data_smithi171_78918_object52989:head# 2024-04-20T17:11:22.039 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.039 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7de69663:::benchmark_data_smithi171_78918_object105637:head# 2024-04-20T17:11:22.039 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.039 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7de74b84:::benchmark_data_smithi171_78918_object121471:head# 2024-04-20T17:11:22.039 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.039 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7de7cad3:::benchmark_data_smithi171_78918_object98805:head# 2024-04-20T17:11:22.039 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.039 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7de82fb0:::benchmark_data_smithi171_78918_object84758:head# 2024-04-20T17:11:22.039 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.039 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7de87863:::benchmark_data_smithi171_78918_object58310:head# 2024-04-20T17:11:22.039 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.039 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7de897fa:::benchmark_data_smithi171_78918_object155205:head# 2024-04-20T17:11:22.039 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.039 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7de8e54e:::benchmark_data_smithi171_78918_object64129:head# 2024-04-20T17:11:22.039 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.040 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7de8fb59:::benchmark_data_smithi171_78918_object98494:head# 2024-04-20T17:11:22.040 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.040 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7de95931:::benchmark_data_smithi171_78918_object121017:head# 2024-04-20T17:11:22.040 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.040 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7de96c73:::benchmark_data_smithi171_78918_object152625:head# 2024-04-20T17:11:22.040 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.040 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7de9ccfe:::benchmark_data_smithi171_78918_object160788:head# 2024-04-20T17:11:22.040 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.040 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7deb5556:::benchmark_data_smithi171_78918_object110898:head# 2024-04-20T17:11:22.040 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.040 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7deca210:::benchmark_data_smithi171_78918_object133741:head# 2024-04-20T17:11:22.040 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.040 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dedd360:::benchmark_data_smithi171_78918_object96345:head# 2024-04-20T17:11:22.041 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.041 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dee1255:::benchmark_data_smithi171_78918_object132067:head# 2024-04-20T17:11:22.041 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.041 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dee5d76:::benchmark_data_smithi171_78918_object145517:head# 2024-04-20T17:11:22.041 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.041 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7def864d:::benchmark_data_smithi171_78918_object57207:head# 2024-04-20T17:11:22.041 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.041 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7df0076f:::benchmark_data_smithi171_78918_object93033:head# 2024-04-20T17:11:22.041 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.041 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7df03b81:::benchmark_data_smithi171_78918_object118438:head# 2024-04-20T17:11:22.041 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.041 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7df0866e:::benchmark_data_smithi171_78918_object113105:head# 2024-04-20T17:11:22.041 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.041 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7df0a483:::benchmark_data_smithi171_78918_object59219:head# 2024-04-20T17:11:22.041 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.042 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7df0d3f1:::benchmark_data_smithi171_78918_object113919:head# 2024-04-20T17:11:22.042 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.042 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7df0ec8a:::benchmark_data_smithi171_78918_object133246:head# 2024-04-20T17:11:22.042 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.042 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7df2bad1:::benchmark_data_smithi171_78918_object51475:head# 2024-04-20T17:11:22.042 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.042 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7df3009f:::benchmark_data_smithi171_78918_object56898:head# 2024-04-20T17:11:22.042 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.042 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7df30e0d:::benchmark_data_smithi171_78918_object57258:head# 2024-04-20T17:11:22.042 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.042 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7df52758:::benchmark_data_smithi171_78918_object83453:head# 2024-04-20T17:11:22.042 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.042 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7df79020:::benchmark_data_smithi171_78918_object55790:head# 2024-04-20T17:11:22.042 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.042 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7df7bbc0:::benchmark_data_smithi171_78918_object104727:head# 2024-04-20T17:11:22.043 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.043 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7df80237:::benchmark_data_smithi171_78918_object92593:head# 2024-04-20T17:11:22.043 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.043 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7df8046e:::benchmark_data_smithi171_78918_object110616:head# 2024-04-20T17:11:22.043 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.043 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7df81c83:::benchmark_data_smithi171_78918_object55844:head# 2024-04-20T17:11:22.043 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.043 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7df81ec5:::benchmark_data_smithi171_78918_object154109:head# 2024-04-20T17:11:22.043 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.043 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7df81f6d:::benchmark_data_smithi171_78918_object107809:head# 2024-04-20T17:11:22.043 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.043 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7df97e69:::benchmark_data_smithi171_78918_object52438:head# 2024-04-20T17:11:22.043 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.043 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dfa61be:::benchmark_data_smithi171_78918_object97219:head# 2024-04-20T17:11:22.044 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.044 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dfbd367:::benchmark_data_smithi171_78918_object111071:head# 2024-04-20T17:11:22.044 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.044 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dfcb11c:::benchmark_data_smithi171_78918_object88231:head# 2024-04-20T17:11:22.044 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.044 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dfd0162:::benchmark_data_smithi171_78918_object129899:head# 2024-04-20T17:11:22.044 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.044 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dfe16ff:::benchmark_data_smithi171_78918_object152769:head# 2024-04-20T17:11:22.044 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.044 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dff81f0:::benchmark_data_smithi171_78918_object143138:head# 2024-04-20T17:11:22.044 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.044 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7dffff9c:::benchmark_data_smithi171_78918_object144074:head# 2024-04-20T17:11:22.044 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.044 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e000aa3:::benchmark_data_smithi171_78918_object99531:head# 2024-04-20T17:11:22.045 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.045 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e002270:::benchmark_data_smithi171_78918_object73721:head# 2024-04-20T17:11:22.045 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.045 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e01a4dd:::benchmark_data_smithi171_78918_object93075:head# 2024-04-20T17:11:22.045 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.045 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e01c34b:::benchmark_data_smithi171_78918_object65456:head# 2024-04-20T17:11:22.045 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.045 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e03ae5b:::benchmark_data_smithi171_78918_object106303:head# 2024-04-20T17:11:22.045 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.045 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e03ff35:::benchmark_data_smithi171_78918_object61342:head# 2024-04-20T17:11:22.045 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.045 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e042e28:::benchmark_data_smithi171_78918_object57904:head# 2024-04-20T17:11:22.045 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.045 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e05932e:::benchmark_data_smithi171_78918_object59686:head# 2024-04-20T17:11:22.046 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.046 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e05bb77:::benchmark_data_smithi171_78918_object97942:head# 2024-04-20T17:11:22.046 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.046 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e067ae5:::benchmark_data_smithi171_78918_object123216:head# 2024-04-20T17:11:22.046 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.046 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e06e892:::benchmark_data_smithi171_78918_object135193:head# 2024-04-20T17:11:22.046 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.046 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e06f620:::benchmark_data_smithi171_78918_object132286:head# 2024-04-20T17:11:22.046 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.046 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e07de3d:::benchmark_data_smithi171_78918_object147954:head# 2024-04-20T17:11:22.046 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.046 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e0a7928:::benchmark_data_smithi171_78918_object74679:head# 2024-04-20T17:11:22.046 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.046 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e0ac5df:::benchmark_data_smithi171_78918_object107767:head# 2024-04-20T17:11:22.046 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.047 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e0b8637:::benchmark_data_smithi171_78918_object148336:head# 2024-04-20T17:11:22.047 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.047 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e0be41c:::benchmark_data_smithi171_78918_object108282:head# 2024-04-20T17:11:22.047 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.047 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e0d257d:::benchmark_data_smithi171_78918_object75345:head# 2024-04-20T17:11:22.047 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.047 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e0d5e84:::benchmark_data_smithi171_78918_object78178:head# 2024-04-20T17:11:22.047 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.047 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e0e19a0:::benchmark_data_smithi171_78918_object60694:head# 2024-04-20T17:11:22.047 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.047 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e0e88c3:::benchmark_data_smithi171_78918_object110229:head# 2024-04-20T17:11:22.047 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.047 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e0f2db6:::benchmark_data_smithi171_78918_object127621:head# 2024-04-20T17:11:22.047 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.047 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e0fa08f:::benchmark_data_smithi171_78918_object152911:head# 2024-04-20T17:11:22.048 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.048 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e0fe87f:::benchmark_data_smithi171_78918_object67774:head# 2024-04-20T17:11:22.048 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.048 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e10fbf0:::benchmark_data_smithi171_78918_object142326:head# 2024-04-20T17:11:22.048 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.048 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e110c54:::benchmark_data_smithi171_78918_object124141:head# 2024-04-20T17:11:22.048 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.048 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e11598e:::benchmark_data_smithi171_78918_object142116:head# 2024-04-20T17:11:22.048 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.048 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e116d01:::benchmark_data_smithi171_78918_object147915:head# 2024-04-20T17:11:22.048 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.048 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e11929c:::benchmark_data_smithi171_78918_object65245:head# 2024-04-20T17:11:22.048 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.048 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e11b830:::benchmark_data_smithi171_78918_object140123:head# 2024-04-20T17:11:22.049 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.049 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e150aef:::benchmark_data_smithi171_78918_object65597:head# 2024-04-20T17:11:22.049 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.049 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e1526f4:::benchmark_data_smithi171_78918_object123908:head# 2024-04-20T17:11:22.049 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.049 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e156d12:::benchmark_data_smithi171_78918_object52649:head# 2024-04-20T17:11:22.049 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.049 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e15b36a:::benchmark_data_smithi171_78918_object135446:head# 2024-04-20T17:11:22.049 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.049 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e15b629:::benchmark_data_smithi171_78918_object91191:head# 2024-04-20T17:11:22.049 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.049 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e173765:::benchmark_data_smithi171_78918_object70061:head# 2024-04-20T17:11:22.049 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.049 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e184bfe:::benchmark_data_smithi171_78918_object51857:head# 2024-04-20T17:11:22.049 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.050 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e1852c8:::benchmark_data_smithi171_78918_object140755:head# 2024-04-20T17:11:22.050 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.050 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e1858e7:::benchmark_data_smithi171_78918_object74884:head# 2024-04-20T17:11:22.050 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.050 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e186158:::benchmark_data_smithi171_78918_object97063:head# 2024-04-20T17:11:22.050 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.050 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e18eb20:::benchmark_data_smithi171_78918_object100233:head# 2024-04-20T17:11:22.050 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.050 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e195ff1:::benchmark_data_smithi171_78918_object124640:head# 2024-04-20T17:11:22.050 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.050 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e1a0e32:::benchmark_data_smithi171_78918_object75268:head# 2024-04-20T17:11:22.050 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.050 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e1a411c:::benchmark_data_smithi171_78918_object92513:head# 2024-04-20T17:11:22.051 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.051 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e1a8bb7:::benchmark_data_smithi171_78918_object105965:head# 2024-04-20T17:11:22.051 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.051 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e1b1aee:::benchmark_data_smithi171_78918_object149835:head# 2024-04-20T17:11:22.051 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.051 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e1ba566:::benchmark_data_smithi171_78918_object116295:head# 2024-04-20T17:11:22.051 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.051 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e1be5b3:::benchmark_data_smithi171_78918_object150986:head# 2024-04-20T17:11:22.051 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.051 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e1c056d:::benchmark_data_smithi171_78918_object106995:head# 2024-04-20T17:11:22.051 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.051 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e1c40cc:::benchmark_data_smithi171_78918_object103966:head# 2024-04-20T17:11:22.051 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.051 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e1cbc03:::benchmark_data_smithi171_78918_object99384:head# 2024-04-20T17:11:22.052 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.052 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e1d0690:::benchmark_data_smithi171_78918_object60270:head# 2024-04-20T17:11:22.052 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.052 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e1e47de:::benchmark_data_smithi171_78918_object76965:head# 2024-04-20T17:11:22.052 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.052 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e209fbf:::benchmark_data_smithi171_78918_object59034:head# 2024-04-20T17:11:22.052 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.052 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e21343b:::benchmark_data_smithi171_78918_object89513:head# 2024-04-20T17:11:22.052 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.052 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e226c42:::benchmark_data_smithi171_78918_object107064:head# 2024-04-20T17:11:22.052 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.052 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e23ac8b:::benchmark_data_smithi171_78918_object110628:head# 2024-04-20T17:11:22.052 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.052 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e250a82:::benchmark_data_smithi171_78918_object58992:head# 2024-04-20T17:11:22.052 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.053 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e2579a9:::benchmark_data_smithi171_78918_object53515:head# 2024-04-20T17:11:22.053 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.053 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e25f17a:::benchmark_data_smithi171_78918_object120686:head# 2024-04-20T17:11:22.053 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.053 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e28195c:::benchmark_data_smithi171_78918_object115753:head# 2024-04-20T17:11:22.053 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.053 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e2899ce:::benchmark_data_smithi171_78918_object138333:head# 2024-04-20T17:11:22.053 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.053 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e290452:::benchmark_data_smithi171_78918_object60773:head# 2024-04-20T17:11:22.053 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.053 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e2913ea:::benchmark_data_smithi171_78918_object119405:head# 2024-04-20T17:11:22.053 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.053 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e2a3b46:::benchmark_data_smithi171_78918_object128509:head# 2024-04-20T17:11:22.053 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.054 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e2a7fd0:::benchmark_data_smithi171_78918_object149616:head# 2024-04-20T17:11:22.054 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.054 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e2a8379:::benchmark_data_smithi171_78918_object90877:head# 2024-04-20T17:11:22.054 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.054 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e2b8eeb:::benchmark_data_smithi171_78918_object151769:head# 2024-04-20T17:11:22.054 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.054 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e2c663f:::benchmark_data_smithi171_78918_object104834:head# 2024-04-20T17:11:22.054 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.054 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e2cec26:::benchmark_data_smithi171_78918_object132041:head# 2024-04-20T17:11:22.054 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.054 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e2d0086:::benchmark_data_smithi171_78918_object158671:head# 2024-04-20T17:11:22.054 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.054 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e2d3d30:::benchmark_data_smithi171_78918_object92183:head# 2024-04-20T17:11:22.054 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.055 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e2da5ea:::benchmark_data_smithi171_78918_object121408:head# 2024-04-20T17:11:22.055 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.055 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e2dd8ea:::benchmark_data_smithi171_78918_object154989:head# 2024-04-20T17:11:22.055 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.055 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e2edf83:::benchmark_data_smithi171_78918_object84764:head# 2024-04-20T17:11:22.055 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.055 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e2f0f74:::benchmark_data_smithi171_78918_object118116:head# 2024-04-20T17:11:22.055 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.055 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e2f59af:::benchmark_data_smithi171_78918_object85467:head# 2024-04-20T17:11:22.055 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.055 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e2faa73:::benchmark_data_smithi171_78918_object116404:head# 2024-04-20T17:11:22.055 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.055 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e305eef:::benchmark_data_smithi171_78918_object94743:head# 2024-04-20T17:11:22.055 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.055 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e308607:::benchmark_data_smithi171_78918_object78986:head# 2024-04-20T17:11:22.056 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.056 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e30920b:::benchmark_data_smithi171_78918_object54148:head# 2024-04-20T17:11:22.056 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.056 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e30f069:::benchmark_data_smithi171_78918_object97806:head# 2024-04-20T17:11:22.056 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.056 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e316777:::benchmark_data_smithi171_78918_object148536:head# 2024-04-20T17:11:22.056 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.056 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e319b9c:::benchmark_data_smithi171_78918_object94307:head# 2024-04-20T17:11:22.056 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.056 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e319db3:::benchmark_data_smithi171_78918_object68408:head# 2024-04-20T17:11:22.056 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.056 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e31aca6:::benchmark_data_smithi171_78918_object155872:head# 2024-04-20T17:11:22.056 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.056 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e31c751:::benchmark_data_smithi171_78918_object98049:head# 2024-04-20T17:11:22.056 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.057 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e32131b:::benchmark_data_smithi171_78918_object53884:head# 2024-04-20T17:11:22.057 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.057 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e336cf8:::benchmark_data_smithi171_78918_object145653:head# 2024-04-20T17:11:22.057 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.057 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e33ed51:::benchmark_data_smithi171_78918_object60969:head# 2024-04-20T17:11:22.057 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.057 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e341c57:::benchmark_data_smithi171_78918_object79504:head# 2024-04-20T17:11:22.057 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.057 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e350c53:::benchmark_data_smithi171_78918_object104621:head# 2024-04-20T17:11:22.057 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.057 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e35f266:::benchmark_data_smithi171_78918_object105493:head# 2024-04-20T17:11:22.057 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.057 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e372e76:::benchmark_data_smithi171_78918_object145204:head# 2024-04-20T17:11:22.057 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.058 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e378968:::benchmark_data_smithi171_78918_object160496:head# 2024-04-20T17:11:22.058 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.058 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e3809d2:::benchmark_data_smithi171_78918_object52783:head# 2024-04-20T17:11:22.058 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.058 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e385f1c:::benchmark_data_smithi171_78918_object80060:head# 2024-04-20T17:11:22.058 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.058 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e386a37:::benchmark_data_smithi171_78918_object142564:head# 2024-04-20T17:11:22.058 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.058 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e38cc21:::benchmark_data_smithi171_78918_object85237:head# 2024-04-20T17:11:22.058 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.058 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e39d5dc:::benchmark_data_smithi171_78918_object81329:head# 2024-04-20T17:11:22.058 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.058 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e3a1e39:::benchmark_data_smithi171_78918_object124694:head# 2024-04-20T17:11:22.058 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.058 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e3ac7c0:::benchmark_data_smithi171_78918_object134224:head# 2024-04-20T17:11:22.059 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.059 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e3b2099:::benchmark_data_smithi171_78918_object153494:head# 2024-04-20T17:11:22.059 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.059 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e3b222d:::benchmark_data_smithi171_78918_object159049:head# 2024-04-20T17:11:22.059 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.059 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e3df0e6:::benchmark_data_smithi171_78918_object145225:head# 2024-04-20T17:11:22.059 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.059 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e3e4ba7:::benchmark_data_smithi171_78918_object67871:head# 2024-04-20T17:11:22.059 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.059 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e3e51f7:::benchmark_data_smithi171_78918_object112302:head# 2024-04-20T17:11:22.059 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.059 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e3e69b1:::benchmark_data_smithi171_78918_object70110:head# 2024-04-20T17:11:22.059 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.059 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e3e91fb:::benchmark_data_smithi171_78918_object102848:head# 2024-04-20T17:11:22.059 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.060 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e3ef0b4:::benchmark_data_smithi171_78918_object119479:head# 2024-04-20T17:11:22.060 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.060 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e3f1ca6:::benchmark_data_smithi171_78918_object108483:head# 2024-04-20T17:11:22.060 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.060 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e3f1fad:::benchmark_data_smithi171_78918_object133109:head# 2024-04-20T17:11:22.060 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.060 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e3f23dd:::benchmark_data_smithi171_78918_object153306:head# 2024-04-20T17:11:22.060 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.060 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e3f2dbc:::benchmark_data_smithi171_78918_object145739:head# 2024-04-20T17:11:22.060 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.060 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e3f9829:::benchmark_data_smithi171_78918_object126006:head# 2024-04-20T17:11:22.060 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.060 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e404556:::benchmark_data_smithi171_78918_object79612:head# 2024-04-20T17:11:22.060 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.060 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e404ece:::benchmark_data_smithi171_78918_object72286:head# 2024-04-20T17:11:22.061 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.061 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e41280e:::benchmark_data_smithi171_78918_object148648:head# 2024-04-20T17:11:22.061 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.061 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e423e34:::benchmark_data_smithi171_78918_object94685:head# 2024-04-20T17:11:22.061 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.061 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e4287e8:::benchmark_data_smithi171_78918_object135638:head# 2024-04-20T17:11:22.061 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.061 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e430658:::benchmark_data_smithi171_78918_object88905:head# 2024-04-20T17:11:22.061 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.061 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e43346d:::benchmark_data_smithi171_78918_object140589:head# 2024-04-20T17:11:22.061 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.061 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e44253a:::benchmark_data_smithi171_78918_object147309:head# 2024-04-20T17:11:22.061 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.061 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e4438d8:::benchmark_data_smithi171_78918_object67094:head# 2024-04-20T17:11:22.062 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.062 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e446638:::benchmark_data_smithi171_78918_object143780:head# 2024-04-20T17:11:22.062 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.062 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e449551:::benchmark_data_smithi171_78918_object77019:head# 2024-04-20T17:11:22.062 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.062 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e456119:::benchmark_data_smithi171_78918_object120914:head# 2024-04-20T17:11:22.062 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.062 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e463e15:::benchmark_data_smithi171_78918_object114062:head# 2024-04-20T17:11:22.062 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.062 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e4647a9:::benchmark_data_smithi171_78918_object159471:head# 2024-04-20T17:11:22.062 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.062 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e4818b2:::benchmark_data_smithi171_78918_object100964:head# 2024-04-20T17:11:22.062 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.062 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e4a929d:::benchmark_data_smithi171_78918_object97364:head# 2024-04-20T17:11:22.062 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.063 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e4a96c1:::benchmark_data_smithi171_78918_object80083:head# 2024-04-20T17:11:22.063 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.063 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e4ac13c:::benchmark_data_smithi171_78918_object73252:head# 2024-04-20T17:11:22.063 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.063 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e4b755b:::benchmark_data_smithi171_78918_object121943:head# 2024-04-20T17:11:22.063 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.063 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e4c3a8b:::benchmark_data_smithi171_78918_object125586:head# 2024-04-20T17:11:22.063 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.063 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e4cd560:::benchmark_data_smithi171_78918_object129410:head# 2024-04-20T17:11:22.063 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.063 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e4d040a:::benchmark_data_smithi171_78918_object83227:head# 2024-04-20T17:11:22.063 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.063 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e4dd7e4:::benchmark_data_smithi171_78918_object116026:head# 2024-04-20T17:11:22.063 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.064 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e4e1b83:::benchmark_data_smithi171_78918_object120911:head# 2024-04-20T17:11:22.064 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.064 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e4ea9e2:::benchmark_data_smithi171_78918_object145874:head# 2024-04-20T17:11:22.064 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.064 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e4facfb:::benchmark_data_smithi171_78918_object134623:head# 2024-04-20T17:11:22.064 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.064 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e5004ea:::benchmark_data_smithi171_78918_object51644:head# 2024-04-20T17:11:22.064 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.064 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e501be1:::benchmark_data_smithi171_78918_object81219:head# 2024-04-20T17:11:22.064 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.064 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e50780b:::benchmark_data_smithi171_78918_object99874:head# 2024-04-20T17:11:22.064 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.064 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e519a3d:::benchmark_data_smithi171_78918_object91246:head# 2024-04-20T17:11:22.064 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.064 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e521d4d:::benchmark_data_smithi171_78918_object53965:head# 2024-04-20T17:11:22.065 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.065 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e525dca:::benchmark_data_smithi171_78918_object137386:head# 2024-04-20T17:11:22.065 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.065 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e5413f5:::benchmark_data_smithi171_78918_object138852:head# 2024-04-20T17:11:22.065 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.065 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e547528:::benchmark_data_smithi171_78918_object137735:head# 2024-04-20T17:11:22.065 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.065 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e550012:::benchmark_data_smithi171_78918_object64855:head# 2024-04-20T17:11:22.065 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.065 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e555383:::benchmark_data_smithi171_78918_object128792:head# 2024-04-20T17:11:22.065 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.065 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e55c65a:::benchmark_data_smithi171_78918_object67014:head# 2024-04-20T17:11:22.065 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.065 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e565b0a:::benchmark_data_smithi171_78918_object144177:head# 2024-04-20T17:11:22.066 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.066 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e5692de:::benchmark_data_smithi171_78918_object52720:head# 2024-04-20T17:11:22.066 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.066 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e56a752:::benchmark_data_smithi171_78918_object102733:head# 2024-04-20T17:11:22.066 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.066 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e571309:::benchmark_data_smithi171_78918_object114783:head# 2024-04-20T17:11:22.066 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.066 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e574308:::benchmark_data_smithi171_78918_object99698:head# 2024-04-20T17:11:22.066 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.066 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e583057:::benchmark_data_smithi171_78918_object152995:head# 2024-04-20T17:11:22.066 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.066 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e590cd3:::benchmark_data_smithi171_78918_object67631:head# 2024-04-20T17:11:22.066 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.067 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e5a5ad2:::benchmark_data_smithi171_78918_object126998:head# 2024-04-20T17:11:22.067 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.067 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e5ac092:::benchmark_data_smithi171_78918_object76335:head# 2024-04-20T17:11:22.067 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.067 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e5b95aa:::benchmark_data_smithi171_78918_object96054:head# 2024-04-20T17:11:22.067 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.067 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e5ba037:::benchmark_data_smithi171_78918_object134642:head# 2024-04-20T17:11:22.067 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.067 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e5be2b7:::benchmark_data_smithi171_78918_object128166:head# 2024-04-20T17:11:22.067 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.067 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e5c0cb6:::benchmark_data_smithi171_78918_object60882:head# 2024-04-20T17:11:22.067 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.067 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e5c1f9e:::benchmark_data_smithi171_78918_object158554:head# 2024-04-20T17:11:22.067 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.068 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e5cd211:::benchmark_data_smithi171_78918_object107717:head# 2024-04-20T17:11:22.068 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.068 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e5ce0e1:::benchmark_data_smithi171_78918_object88425:head# 2024-04-20T17:11:22.068 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.068 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e5d56f1:::benchmark_data_smithi171_78918_object144031:head# 2024-04-20T17:11:22.068 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.068 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e5e6483:::benchmark_data_smithi171_78918_object94214:head# 2024-04-20T17:11:22.068 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.068 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e5fb3db:::benchmark_data_smithi171_78918_object88145:head# 2024-04-20T17:11:22.068 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.068 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e60a531:::benchmark_data_smithi171_78918_object109808:head# 2024-04-20T17:11:22.068 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.068 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e6109cc:::benchmark_data_smithi171_78918_object95305:head# 2024-04-20T17:11:22.068 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.069 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e62077a:::benchmark_data_smithi171_78918_object95094:head# 2024-04-20T17:11:22.069 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.069 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e620799:::benchmark_data_smithi171_78918_object149483:head# 2024-04-20T17:11:22.069 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.069 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e6387b8:::benchmark_data_smithi171_78918_object111236:head# 2024-04-20T17:11:22.069 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.069 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e63e008:::benchmark_data_smithi171_78918_object150234:head# 2024-04-20T17:11:22.069 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.069 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e63ef49:::benchmark_data_smithi171_78918_object100361:head# 2024-04-20T17:11:22.069 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.069 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e644f19:::benchmark_data_smithi171_78918_object73527:head# 2024-04-20T17:11:22.069 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.069 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e65189f:::benchmark_data_smithi171_78918_object112391:head# 2024-04-20T17:11:22.069 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.070 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e65c434:::benchmark_data_smithi171_78918_object66878:head# 2024-04-20T17:11:22.070 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.070 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e65f059:::benchmark_data_smithi171_78918_object83540:head# 2024-04-20T17:11:22.070 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.070 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e66bbf8:::benchmark_data_smithi171_78918_object106545:head# 2024-04-20T17:11:22.070 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.070 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e676318:::benchmark_data_smithi171_78918_object158781:head# 2024-04-20T17:11:22.070 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.070 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e697634:::benchmark_data_smithi171_78918_object159785:head# 2024-04-20T17:11:22.070 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.070 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e69972e:::benchmark_data_smithi171_78918_object80131:head# 2024-04-20T17:11:22.070 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.070 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e69bb07:::benchmark_data_smithi171_78918_object92132:head# 2024-04-20T17:11:22.070 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.070 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e69ecfa:::benchmark_data_smithi171_78918_object72556:head# 2024-04-20T17:11:22.071 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.071 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e6a5bee:::benchmark_data_smithi171_78918_object148646:head# 2024-04-20T17:11:22.071 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.071 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e6aad0d:::benchmark_data_smithi171_78918_object139135:head# 2024-04-20T17:11:22.071 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.071 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e6b1248:::benchmark_data_smithi171_78918_object88084:head# 2024-04-20T17:11:22.071 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.071 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e6ce734:::benchmark_data_smithi171_78918_object110503:head# 2024-04-20T17:11:22.071 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.071 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e6d8e5c:::benchmark_data_smithi171_78918_object113142:head# 2024-04-20T17:11:22.071 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.071 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e6de035:::benchmark_data_smithi171_78918_object112207:head# 2024-04-20T17:11:22.071 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.071 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e6de12d:::benchmark_data_smithi171_78918_object68345:head# 2024-04-20T17:11:22.071 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.072 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e6ed711:::benchmark_data_smithi171_78918_object142433:head# 2024-04-20T17:11:22.072 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.072 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e6fda91:::benchmark_data_smithi171_78918_object52459:head# 2024-04-20T17:11:22.072 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.072 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e70383c:::benchmark_data_smithi171_78918_object124136:head# 2024-04-20T17:11:22.072 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.072 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e7092c0:::benchmark_data_smithi171_78918_object86915:head# 2024-04-20T17:11:22.072 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.072 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e71700e:::benchmark_data_smithi171_78918_object70090:head# 2024-04-20T17:11:22.072 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.072 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e719a47:::benchmark_data_smithi171_78918_object131150:head# 2024-04-20T17:11:22.072 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.072 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e71bb13:::benchmark_data_smithi171_78918_object69841:head# 2024-04-20T17:11:22.072 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.072 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e7258fd:::benchmark_data_smithi171_78918_object87301:head# 2024-04-20T17:11:22.073 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.073 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e72d8aa:::benchmark_data_smithi171_78918_object121711:head# 2024-04-20T17:11:22.073 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.073 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e73695d:::benchmark_data_smithi171_78918_object97331:head# 2024-04-20T17:11:22.073 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.073 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e73a865:::benchmark_data_smithi171_78918_object71091:head# 2024-04-20T17:11:22.073 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.073 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e73ce71:::benchmark_data_smithi171_78918_object110636:head# 2024-04-20T17:11:22.073 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.073 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e7406d6:::benchmark_data_smithi171_78918_object63192:head# 2024-04-20T17:11:22.073 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.073 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e75c297:::benchmark_data_smithi171_78918_object86055:head# 2024-04-20T17:11:22.073 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.073 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e764917:::benchmark_data_smithi171_78918_object111553:head# 2024-04-20T17:11:22.073 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.074 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e76a608:::benchmark_data_smithi171_78918_object105988:head# 2024-04-20T17:11:22.074 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.074 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e777277:::benchmark_data_smithi171_78918_object130475:head# 2024-04-20T17:11:22.074 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.074 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e7a3af1:::benchmark_data_smithi171_78918_object137725:head# 2024-04-20T17:11:22.074 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.074 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e7ad01b:::benchmark_data_smithi171_78918_object66138:head# 2024-04-20T17:11:22.074 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.074 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e7b99af:::benchmark_data_smithi171_78918_object138040:head# 2024-04-20T17:11:22.074 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.074 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e7c717d:::benchmark_data_smithi171_78918_object91354:head# 2024-04-20T17:11:22.074 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.074 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e7e86ac:::benchmark_data_smithi171_78918_object86631:head# 2024-04-20T17:11:22.074 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.075 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e82414c:::benchmark_data_smithi171_78918_object89035:head# 2024-04-20T17:11:22.075 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.075 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e8361bb:::benchmark_data_smithi171_78918_object119260:head# 2024-04-20T17:11:22.075 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.075 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e839561:::benchmark_data_smithi171_78918_object54116:head# 2024-04-20T17:11:22.075 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.075 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e83edd6:::benchmark_data_smithi171_78918_object110864:head# 2024-04-20T17:11:22.075 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.075 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e840c1d:::benchmark_data_smithi171_78918_object147302:head# 2024-04-20T17:11:22.075 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.075 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e8485c0:::benchmark_data_smithi171_78918_object97490:head# 2024-04-20T17:11:22.075 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.075 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e84a013:::benchmark_data_smithi171_78918_object127060:head# 2024-04-20T17:11:22.075 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.075 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e856c64:::benchmark_data_smithi171_78918_object157061:head# 2024-04-20T17:11:22.076 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.076 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e85defc:::benchmark_data_smithi171_78918_object65356:head# 2024-04-20T17:11:22.076 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.076 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e86bdf1:::benchmark_data_smithi171_78918_object107743:head# 2024-04-20T17:11:22.076 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.076 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e86ff8f:::benchmark_data_smithi171_78918_object51799:head# 2024-04-20T17:11:22.076 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.076 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e88748e:::benchmark_data_smithi171_78918_object121098:head# 2024-04-20T17:11:22.076 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.076 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e89d0c8:::benchmark_data_smithi171_78918_object62952:head# 2024-04-20T17:11:22.076 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.076 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e8b4a9f:::benchmark_data_smithi171_78918_object92779:head# 2024-04-20T17:11:22.076 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.076 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e8b98a1:::benchmark_data_smithi171_78918_object146058:head# 2024-04-20T17:11:22.077 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.077 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e8ce609:::benchmark_data_smithi171_78918_object125401:head# 2024-04-20T17:11:22.077 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.077 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e8d1ff7:::benchmark_data_smithi171_78918_object53825:head# 2024-04-20T17:11:22.077 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.077 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e8de3d9:::benchmark_data_smithi171_78918_object98976:head# 2024-04-20T17:11:22.077 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.077 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e8f78df:::benchmark_data_smithi171_78918_object113890:head# 2024-04-20T17:11:22.077 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.077 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e8f8fc9:::benchmark_data_smithi171_78918_object73825:head# 2024-04-20T17:11:22.077 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.077 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e8fc659:::benchmark_data_smithi171_78918_object144383:head# 2024-04-20T17:11:22.077 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.078 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e8fdacd:::benchmark_data_smithi171_78918_object152388:head# 2024-04-20T17:11:22.078 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.078 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e90266b:::benchmark_data_smithi171_78918_object98985:head# 2024-04-20T17:11:22.078 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.078 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e91c678:::benchmark_data_smithi171_78918_object66231:head# 2024-04-20T17:11:22.078 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.078 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e92a28b:::benchmark_data_smithi171_78918_object67180:head# 2024-04-20T17:11:22.078 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.078 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e92b5b2:::benchmark_data_smithi171_78918_object74735:head# 2024-04-20T17:11:22.078 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.078 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e92bbf7:::benchmark_data_smithi171_78918_object68539:head# 2024-04-20T17:11:22.079 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.079 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e930616:::benchmark_data_smithi171_78918_object128772:head# 2024-04-20T17:11:22.079 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.079 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e936ffb:::benchmark_data_smithi171_78918_object153314:head# 2024-04-20T17:11:22.079 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.079 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e963cfa:::benchmark_data_smithi171_78918_object82525:head# 2024-04-20T17:11:22.079 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.079 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e97226c:::benchmark_data_smithi171_78918_object116565:head# 2024-04-20T17:11:22.079 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.080 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e97af0d:::benchmark_data_smithi171_78918_object147663:head# 2024-04-20T17:11:22.080 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.080 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e97c83d:::benchmark_data_smithi171_78918_object120549:head# 2024-04-20T17:11:22.080 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.080 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e98d6ff:::benchmark_data_smithi171_78918_object133666:head# 2024-04-20T17:11:22.080 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.080 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e9a29c0:::benchmark_data_smithi171_78918_object139445:head# 2024-04-20T17:11:22.080 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.080 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e9ab6d0:::benchmark_data_smithi171_78918_object55463:head# 2024-04-20T17:11:22.081 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.081 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e9b37a1:::benchmark_data_smithi171_78918_object108761:head# 2024-04-20T17:11:22.081 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.081 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e9b5cf2:::benchmark_data_smithi171_78918_object136866:head# 2024-04-20T17:11:22.081 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.081 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e9bbb6b:::benchmark_data_smithi171_78918_object72947:head# 2024-04-20T17:11:22.081 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.081 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e9bdbad:::benchmark_data_smithi171_78918_object77991:head# 2024-04-20T17:11:22.081 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.081 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e9c3540:::benchmark_data_smithi171_78918_object91346:head# 2024-04-20T17:11:22.081 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.082 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e9cd4fa:::benchmark_data_smithi171_78918_object118957:head# 2024-04-20T17:11:22.082 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.082 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e9d033a:::benchmark_data_smithi171_78918_object81153:head# 2024-04-20T17:11:22.082 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.082 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e9d386e:::benchmark_data_smithi171_78918_object86408:head# 2024-04-20T17:11:22.082 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.082 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e9d402b:::benchmark_data_smithi171_78918_object155736:head# 2024-04-20T17:11:22.082 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.082 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e9d9e38:::benchmark_data_smithi171_78918_object72801:head# 2024-04-20T17:11:22.082 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.082 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e9e34de:::benchmark_data_smithi171_78918_object89191:head# 2024-04-20T17:11:22.082 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.082 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7e9f55c1:::benchmark_data_smithi171_78918_object154325:head# 2024-04-20T17:11:22.082 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.083 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ea0bd6e:::benchmark_data_smithi171_78918_object72282:head# 2024-04-20T17:11:22.083 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.083 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ea0fdd2:::benchmark_data_smithi171_78918_object102671:head# 2024-04-20T17:11:22.083 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.083 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ea23e93:::benchmark_data_smithi171_78918_object122021:head# 2024-04-20T17:11:22.083 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.083 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ea40abc:::benchmark_data_smithi171_78918_object84166:head# 2024-04-20T17:11:22.083 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.083 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ea4a293:::benchmark_data_smithi171_78918_object56774:head# 2024-04-20T17:11:22.083 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.083 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ea553f1:::benchmark_data_smithi171_78918_object102093:head# 2024-04-20T17:11:22.084 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.084 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ea661b4:::benchmark_data_smithi171_78918_object74235:head# 2024-04-20T17:11:22.084 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.084 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ea6a283:::benchmark_data_smithi171_78918_object56746:head# 2024-04-20T17:11:22.084 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.084 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ea6eb9a:::benchmark_data_smithi171_78918_object145732:head# 2024-04-20T17:11:22.084 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.084 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ea77eb1:::benchmark_data_smithi171_78918_object134139:head# 2024-04-20T17:11:22.084 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.084 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ea79e96:::benchmark_data_smithi171_78918_object99286:head# 2024-04-20T17:11:22.084 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.084 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ea7c200:::benchmark_data_smithi171_78918_object122895:head# 2024-04-20T17:11:22.085 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.085 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ea82242:::benchmark_data_smithi171_78918_object132898:head# 2024-04-20T17:11:22.085 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.085 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ea93b5e:::benchmark_data_smithi171_78918_object87879:head# 2024-04-20T17:11:22.085 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.085 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eaa3661:::benchmark_data_smithi171_78918_object119922:head# 2024-04-20T17:11:22.085 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.085 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eaa650e:::benchmark_data_smithi171_78918_object151304:head# 2024-04-20T17:11:22.085 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.085 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eab060f:::benchmark_data_smithi171_78918_object106199:head# 2024-04-20T17:11:22.085 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.085 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eac9ed3:::benchmark_data_smithi171_78918_object79707:head# 2024-04-20T17:11:22.086 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.086 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eacb35c:::benchmark_data_smithi171_78918_object105314:head# 2024-04-20T17:11:22.086 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.086 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eacfaf1:::benchmark_data_smithi171_78918_object63019:head# 2024-04-20T17:11:22.086 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.086 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eade42f:::benchmark_data_smithi171_78918_object128004:head# 2024-04-20T17:11:22.086 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.086 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eae991b:::benchmark_data_smithi171_78918_object115504:head# 2024-04-20T17:11:22.086 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.086 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eaeb31a:::benchmark_data_smithi171_78918_object126133:head# 2024-04-20T17:11:22.086 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.087 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eaf2e43:::benchmark_data_smithi171_78918_object90959:head# 2024-04-20T17:11:22.087 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.087 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eafd516:::benchmark_data_smithi171_78918_object152862:head# 2024-04-20T17:11:22.087 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.087 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eb02cee:::benchmark_data_smithi171_78918_object145972:head# 2024-04-20T17:11:22.087 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.087 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eb10ee4:::benchmark_data_smithi171_78918_object93040:head# 2024-04-20T17:11:22.087 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.087 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eb14a76:::benchmark_data_smithi171_78918_object122496:head# 2024-04-20T17:11:22.087 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.087 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eb21d50:::benchmark_data_smithi171_78918_object107259:head# 2024-04-20T17:11:22.087 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.087 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eb380b3:::benchmark_data_smithi171_78918_object72632:head# 2024-04-20T17:11:22.087 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.088 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eb42ea1:::benchmark_data_smithi171_78918_object142488:head# 2024-04-20T17:11:22.088 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.088 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eb4e77c:::benchmark_data_smithi171_78918_object156431:head# 2024-04-20T17:11:22.088 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.088 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eb5b0de:::benchmark_data_smithi171_78918_object148705:head# 2024-04-20T17:11:22.088 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.088 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eb5fa19:::benchmark_data_smithi171_78918_object93523:head# 2024-04-20T17:11:22.088 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.088 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eb68267:::benchmark_data_smithi171_78918_object149611:head# 2024-04-20T17:11:22.088 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.088 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eb7148f:::benchmark_data_smithi171_78918_object83458:head# 2024-04-20T17:11:22.088 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.088 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eb72c51:::benchmark_data_smithi171_78918_object62426:head# 2024-04-20T17:11:22.088 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.088 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eb7a3fc:::benchmark_data_smithi171_78918_object56139:head# 2024-04-20T17:11:22.089 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.089 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eb80876:::benchmark_data_smithi171_78918_object102818:head# 2024-04-20T17:11:22.089 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.089 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eb9e3fb:::benchmark_data_smithi171_78918_object51537:head# 2024-04-20T17:11:22.089 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.089 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ebb063c:::benchmark_data_smithi171_78918_object125034:head# 2024-04-20T17:11:22.089 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.089 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ebbdce0:::benchmark_data_smithi171_78918_object95908:head# 2024-04-20T17:11:22.089 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.089 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ebc00b0:::benchmark_data_smithi171_78918_object137067:head# 2024-04-20T17:11:22.089 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.089 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ebc6194:::benchmark_data_smithi171_78918_object57533:head# 2024-04-20T17:11:22.089 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.089 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ebcdcdf:::benchmark_data_smithi171_78918_object109484:head# 2024-04-20T17:11:22.090 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.090 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ebeb9ac:::benchmark_data_smithi171_78918_object115986:head# 2024-04-20T17:11:22.090 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.090 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ebedb5a:::benchmark_data_smithi171_78918_object135694:head# 2024-04-20T17:11:22.090 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.090 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ebee1c9:::benchmark_data_smithi171_78918_object137021:head# 2024-04-20T17:11:22.090 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.090 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ebf2c08:::benchmark_data_smithi171_78918_object66957:head# 2024-04-20T17:11:22.090 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.090 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ec08fb0:::benchmark_data_smithi171_78918_object138427:head# 2024-04-20T17:11:22.090 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.090 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ec19bf0:::benchmark_data_smithi171_78918_object79588:head# 2024-04-20T17:11:22.090 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.090 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ec23643:::benchmark_data_smithi171_78918_object142612:head# 2024-04-20T17:11:22.091 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.091 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ec25912:::benchmark_data_smithi171_78918_object81940:head# 2024-04-20T17:11:22.091 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.091 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ec2b986:::benchmark_data_smithi171_78918_object151386:head# 2024-04-20T17:11:22.091 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.091 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ec2ff6e:::benchmark_data_smithi171_78918_object94368:head# 2024-04-20T17:11:22.091 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.091 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ec3597e:::benchmark_data_smithi171_78918_object104321:head# 2024-04-20T17:11:22.091 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.091 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ec382fb:::benchmark_data_smithi171_78918_object79379:head# 2024-04-20T17:11:22.091 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.091 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ec3a6f4:::benchmark_data_smithi171_78918_object85007:head# 2024-04-20T17:11:22.091 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.091 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ec3cbd5:::benchmark_data_smithi171_78918_object81483:head# 2024-04-20T17:11:22.091 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.092 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ec57878:::benchmark_data_smithi171_78918_object61827:head# 2024-04-20T17:11:22.092 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.092 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ec5c2e0:::benchmark_data_smithi171_78918_object158673:head# 2024-04-20T17:11:22.092 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.092 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ec67ce2:::benchmark_data_smithi171_78918_object128899:head# 2024-04-20T17:11:22.092 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.092 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ec79ef6:::benchmark_data_smithi171_78918_object139587:head# 2024-04-20T17:11:22.092 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.092 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ec955f7:::benchmark_data_smithi171_78918_object130871:head# 2024-04-20T17:11:22.092 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.092 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ec95a4c:::benchmark_data_smithi171_78918_object76620:head# 2024-04-20T17:11:22.092 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.092 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ec98bd8:::benchmark_data_smithi171_78918_object94475:head# 2024-04-20T17:11:22.092 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.093 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ec99bf5:::benchmark_data_smithi171_78918_object95316:head# 2024-04-20T17:11:22.093 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.093 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ec9e5f5:::benchmark_data_smithi171_78918_object99405:head# 2024-04-20T17:11:22.093 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.093 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ec9e95d:::benchmark_data_smithi171_78918_object124357:head# 2024-04-20T17:11:22.093 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.093 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eca806a:::benchmark_data_smithi171_78918_object59014:head# 2024-04-20T17:11:22.093 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.093 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ecaff24:::benchmark_data_smithi171_78918_object52283:head# 2024-04-20T17:11:22.093 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.093 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ecb05fd:::benchmark_data_smithi171_78918_object64063:head# 2024-04-20T17:11:22.093 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.093 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ecb8673:::benchmark_data_smithi171_78918_object54023:head# 2024-04-20T17:11:22.093 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.094 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ecc4732:::benchmark_data_smithi171_78918_object101700:head# 2024-04-20T17:11:22.094 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.094 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ecc91d4:::benchmark_data_smithi171_78918_object154957:head# 2024-04-20T17:11:22.094 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.094 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eccaf8a:::benchmark_data_smithi171_78918_object121689:head# 2024-04-20T17:11:22.094 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.094 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eccd407:::benchmark_data_smithi171_78918_object126054:head# 2024-04-20T17:11:22.094 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.094 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eccfd9d:::benchmark_data_smithi171_78918_object61670:head# 2024-04-20T17:11:22.094 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.094 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ecd69be:::benchmark_data_smithi171_78918_object72240:head# 2024-04-20T17:11:22.094 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.094 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ecdcd5d:::benchmark_data_smithi171_78918_object149572:head# 2024-04-20T17:11:22.095 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.095 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ecde22d:::benchmark_data_smithi171_78918_object115013:head# 2024-04-20T17:11:22.095 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.095 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ecec938:::benchmark_data_smithi171_78918_object105435:head# 2024-04-20T17:11:22.095 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.095 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ecf04a3:::benchmark_data_smithi171_78918_object78858:head# 2024-04-20T17:11:22.095 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.095 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ecf3f5f:::benchmark_data_smithi171_78918_object135655:head# 2024-04-20T17:11:22.095 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.095 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ecf8c90:::benchmark_data_smithi171_78918_object79251:head# 2024-04-20T17:11:22.095 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.095 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ecff65d:::benchmark_data_smithi171_78918_object91673:head# 2024-04-20T17:11:22.095 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.095 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ed00f4e:::benchmark_data_smithi171_78918_object97623:head# 2024-04-20T17:11:22.095 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.096 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ed0f9c7:::benchmark_data_smithi171_78918_object148849:head# 2024-04-20T17:11:22.096 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.096 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ed148a8:::benchmark_data_smithi171_78918_object67397:head# 2024-04-20T17:11:22.096 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.096 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ed1e532:::benchmark_data_smithi171_78918_object100998:head# 2024-04-20T17:11:22.096 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.096 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ed20ca6:::benchmark_data_smithi171_78918_object72925:head# 2024-04-20T17:11:22.096 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.096 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ed224d6:::benchmark_data_smithi171_78918_object104826:head# 2024-04-20T17:11:22.096 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.096 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ed24b79:::benchmark_data_smithi171_78918_object104028:head# 2024-04-20T17:11:22.096 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.096 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ed2766e:::benchmark_data_smithi171_78918_object133717:head# 2024-04-20T17:11:22.096 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.097 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ed2cbd1:::benchmark_data_smithi171_78918_object108776:head# 2024-04-20T17:11:22.097 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.097 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ed397d1:::benchmark_data_smithi171_78918_object140960:head# 2024-04-20T17:11:22.097 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.097 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ed4e8f0:::benchmark_data_smithi171_78918_object63612:head# 2024-04-20T17:11:22.097 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.097 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ed515da:::benchmark_data_smithi171_78918_object110540:head# 2024-04-20T17:11:22.097 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.097 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ed5e9c8:::benchmark_data_smithi171_78918_object54706:head# 2024-04-20T17:11:22.097 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.097 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ed606fe:::benchmark_data_smithi171_78918_object72398:head# 2024-04-20T17:11:22.097 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.097 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ed69762:::benchmark_data_smithi171_78918_object95669:head# 2024-04-20T17:11:22.097 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.098 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ed6b050:::benchmark_data_smithi171_78918_object69042:head# 2024-04-20T17:11:22.098 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.098 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ed7c637:::benchmark_data_smithi171_78918_object151614:head# 2024-04-20T17:11:22.098 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.098 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ed7dad0:::benchmark_data_smithi171_78918_object79969:head# 2024-04-20T17:11:22.098 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.098 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ed99ed0:::benchmark_data_smithi171_78918_object119141:head# 2024-04-20T17:11:22.098 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.098 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ed9aa82:::benchmark_data_smithi171_78918_object118880:head# 2024-04-20T17:11:22.098 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.098 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7edae832:::benchmark_data_smithi171_78918_object58099:head# 2024-04-20T17:11:22.098 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.098 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7edb1d35:::benchmark_data_smithi171_78918_object84079:head# 2024-04-20T17:11:22.098 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.099 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7edb7f92:::benchmark_data_smithi171_78918_object155481:head# 2024-04-20T17:11:22.099 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.099 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7edc1403:::benchmark_data_smithi171_78918_object97135:head# 2024-04-20T17:11:22.099 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.099 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7edc4fc7:::benchmark_data_smithi171_78918_object88255:head# 2024-04-20T17:11:22.099 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.099 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7edcbadb:::benchmark_data_smithi171_78918_object97498:head# 2024-04-20T17:11:22.099 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.099 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7edddc74:::benchmark_data_smithi171_78918_object66050:head# 2024-04-20T17:11:22.099 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.099 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ede2248:::benchmark_data_smithi171_78918_object54912:head# 2024-04-20T17:11:22.099 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.099 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ede84a3:::benchmark_data_smithi171_78918_object102225:head# 2024-04-20T17:11:22.099 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.100 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ee060b0:::benchmark_data_smithi171_78918_object92878:head# 2024-04-20T17:11:22.100 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.100 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ee12f4a:::benchmark_data_smithi171_78918_object126676:head# 2024-04-20T17:11:22.100 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.100 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ee14e62:::benchmark_data_smithi171_78918_object108706:head# 2024-04-20T17:11:22.100 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.100 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ee19831:::benchmark_data_smithi171_78918_object93252:head# 2024-04-20T17:11:22.100 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.100 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ee2f6cd:::benchmark_data_smithi171_78918_object148327:head# 2024-04-20T17:11:22.100 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.100 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ee39b21:::benchmark_data_smithi171_78918_object94279:head# 2024-04-20T17:11:22.100 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.100 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ee3b7d2:::benchmark_data_smithi171_78918_object139021:head# 2024-04-20T17:11:22.100 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.100 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ee44288:::benchmark_data_smithi171_78918_object102475:head# 2024-04-20T17:11:22.101 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.101 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ee48479:::benchmark_data_smithi171_78918_object92906:head# 2024-04-20T17:11:22.101 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.101 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ee4d3ae:::benchmark_data_smithi171_78918_object121135:head# 2024-04-20T17:11:22.101 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.101 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ee59544:::benchmark_data_smithi171_78918_object147005:head# 2024-04-20T17:11:22.101 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.101 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ee60f84:::benchmark_data_smithi171_78918_object84296:head# 2024-04-20T17:11:22.101 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.101 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ee6188e:::benchmark_data_smithi171_78918_object124255:head# 2024-04-20T17:11:22.101 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.101 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ee7ed48:::benchmark_data_smithi171_78918_object90942:head# 2024-04-20T17:11:22.101 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.101 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ee896a0:::benchmark_data_smithi171_78918_object58581:head# 2024-04-20T17:11:22.102 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.102 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ee96eb0:::benchmark_data_smithi171_78918_object152086:head# 2024-04-20T17:11:22.102 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.102 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ee9d0e1:::benchmark_data_smithi171_78918_object110877:head# 2024-04-20T17:11:22.102 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.102 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eeab659:::benchmark_data_smithi171_78918_object128465:head# 2024-04-20T17:11:22.102 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.102 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eeac2ee:::benchmark_data_smithi171_78918_object52265:head# 2024-04-20T17:11:22.102 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.102 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eeaf6d6:::benchmark_data_smithi171_78918_object53460:head# 2024-04-20T17:11:22.102 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.102 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eeba42e:::benchmark_data_smithi171_78918_object155321:head# 2024-04-20T17:11:22.102 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.102 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eec16f7:::benchmark_data_smithi171_78918_object108615:head# 2024-04-20T17:11:22.103 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.103 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eed1deb:::benchmark_data_smithi171_78918_object117628:head# 2024-04-20T17:11:22.103 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.103 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eed50d0:::benchmark_data_smithi171_78918_object72258:head# 2024-04-20T17:11:22.103 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.103 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eee398f:::benchmark_data_smithi171_78918_object60453:head# 2024-04-20T17:11:22.103 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.103 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eeea0d6:::benchmark_data_smithi171_78918_object117072:head# 2024-04-20T17:11:22.103 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.103 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eef30e9:::benchmark_data_smithi171_78918_object136265:head# 2024-04-20T17:11:22.103 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.103 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eef3d9d:::benchmark_data_smithi171_78918_object158663:head# 2024-04-20T17:11:22.103 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.103 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ef00405:::benchmark_data_smithi171_78918_object119545:head# 2024-04-20T17:11:22.104 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.104 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ef08d25:::benchmark_data_smithi171_78918_object145860:head# 2024-04-20T17:11:22.104 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.104 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ef0b1cb:::benchmark_data_smithi171_78918_object90771:head# 2024-04-20T17:11:22.104 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.104 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ef0e40c:::benchmark_data_smithi171_78918_object96889:head# 2024-04-20T17:11:22.104 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.104 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ef1319c:::benchmark_data_smithi171_78918_object106050:head# 2024-04-20T17:11:22.104 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.104 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ef1c186:::benchmark_data_smithi171_78918_object82400:head# 2024-04-20T17:11:22.104 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.104 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ef27a01:::benchmark_data_smithi171_78918_object108843:head# 2024-04-20T17:11:22.104 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.104 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ef29e23:::benchmark_data_smithi171_78918_object156912:head# 2024-04-20T17:11:22.104 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.104 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ef2b1e5:::benchmark_data_smithi171_78918_object127547:head# 2024-04-20T17:11:22.105 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.105 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ef31534:::benchmark_data_smithi171_78918_object77904:head# 2024-04-20T17:11:22.105 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.105 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ef49980:::benchmark_data_smithi171_78918_object78779:head# 2024-04-20T17:11:22.105 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.105 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ef4ba47:::benchmark_data_smithi171_78918_object110404:head# 2024-04-20T17:11:22.105 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.105 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ef4bd3b:::benchmark_data_smithi171_78918_object103452:head# 2024-04-20T17:11:22.105 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.105 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ef5a2d7:::benchmark_data_smithi171_78918_object99044:head# 2024-04-20T17:11:22.105 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.105 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ef64ee3:::benchmark_data_smithi171_78918_object61215:head# 2024-04-20T17:11:22.105 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.105 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ef6f598:::benchmark_data_smithi171_78918_object123449:head# 2024-04-20T17:11:22.106 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.106 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ef70734:::benchmark_data_smithi171_78918_object116851:head# 2024-04-20T17:11:22.106 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.106 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ef78b72:::benchmark_data_smithi171_78918_object68445:head# 2024-04-20T17:11:22.106 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.106 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ef7ac70:::benchmark_data_smithi171_78918_object87107:head# 2024-04-20T17:11:22.106 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.106 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ef7eed3:::benchmark_data_smithi171_78918_object106798:head# 2024-04-20T17:11:22.106 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.106 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ef85b26:::benchmark_data_smithi171_78918_object96992:head# 2024-04-20T17:11:22.106 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.107 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ef91165:::benchmark_data_smithi171_78918_object145300:head# 2024-04-20T17:11:22.107 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.107 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ef99143:::benchmark_data_smithi171_78918_object156576:head# 2024-04-20T17:11:22.107 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.107 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7efa0d4d:::benchmark_data_smithi171_78918_object63023:head# 2024-04-20T17:11:22.107 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.107 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7efa4162:::benchmark_data_smithi171_78918_object139648:head# 2024-04-20T17:11:22.107 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.107 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7efc85c0:::benchmark_data_smithi171_78918_object115965:head# 2024-04-20T17:11:22.107 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.107 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7efdbdaa:::benchmark_data_smithi171_78918_object119705:head# 2024-04-20T17:11:22.107 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.107 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7efe1d7e:::benchmark_data_smithi171_78918_object77180:head# 2024-04-20T17:11:22.108 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.108 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7efe8ad0:::benchmark_data_smithi171_78918_object110699:head# 2024-04-20T17:11:22.108 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.108 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eff07e6:::benchmark_data_smithi171_78918_object140091:head# 2024-04-20T17:11:22.108 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.108 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7eff7bfd:::benchmark_data_smithi171_78918_object94817:head# 2024-04-20T17:11:22.108 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.108 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f017fc4:::benchmark_data_smithi171_78918_object74152:head# 2024-04-20T17:11:22.108 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.108 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f01a100:::benchmark_data_smithi171_78918_object128171:head# 2024-04-20T17:11:22.108 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.109 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f01ded7:::benchmark_data_smithi171_78918_object58406:head# 2024-04-20T17:11:22.109 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.109 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f0299d8:::benchmark_data_smithi171_78918_object130442:head# 2024-04-20T17:11:22.109 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.109 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f02ba54:::benchmark_data_smithi171_78918_object133291:head# 2024-04-20T17:11:22.109 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.109 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f038661:::benchmark_data_smithi171_78918_object138694:head# 2024-04-20T17:11:22.109 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.109 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f046cbc:::benchmark_data_smithi171_78918_object101624:head# 2024-04-20T17:11:22.110 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.110 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f05708e:::benchmark_data_smithi171_78918_object128759:head# 2024-04-20T17:11:22.110 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.110 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f05af06:::benchmark_data_smithi171_78918_object66253:head# 2024-04-20T17:11:22.110 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.110 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f0641d9:::benchmark_data_smithi171_78918_object122605:head# 2024-04-20T17:11:22.110 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.110 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f065ed4:::benchmark_data_smithi171_78918_object135958:head# 2024-04-20T17:11:22.110 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.110 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f0691c8:::benchmark_data_smithi171_78918_object139796:head# 2024-04-20T17:11:22.110 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.110 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f0717e4:::benchmark_data_smithi171_78918_object107439:head# 2024-04-20T17:11:22.111 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.111 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f0799eb:::benchmark_data_smithi171_78918_object152387:head# 2024-04-20T17:11:22.111 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.111 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f08272a:::benchmark_data_smithi171_78918_object136296:head# 2024-04-20T17:11:22.111 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.111 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f08342c:::benchmark_data_smithi171_78918_object138812:head# 2024-04-20T17:11:22.111 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.111 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f08826a:::benchmark_data_smithi171_78918_object68523:head# 2024-04-20T17:11:22.111 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.111 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f08e9ef:::benchmark_data_smithi171_78918_object128141:head# 2024-04-20T17:11:22.111 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.111 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f090cf7:::benchmark_data_smithi171_78918_object156739:head# 2024-04-20T17:11:22.111 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.112 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f096d92:::benchmark_data_smithi171_78918_object145164:head# 2024-04-20T17:11:22.112 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.112 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f0970be:::benchmark_data_smithi171_78918_object53215:head# 2024-04-20T17:11:22.112 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.112 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f0a8b6e:::benchmark_data_smithi171_78918_object103169:head# 2024-04-20T17:11:22.112 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.112 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f0aa46a:::benchmark_data_smithi171_78918_object76615:head# 2024-04-20T17:11:22.112 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.112 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f0d0c94:::benchmark_data_smithi171_78918_object135626:head# 2024-04-20T17:11:22.112 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.113 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f0d1d02:::benchmark_data_smithi171_78918_object140599:head# 2024-04-20T17:11:22.113 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.113 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f0defe3:::benchmark_data_smithi171_78918_object65604:head# 2024-04-20T17:11:22.113 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.113 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f0e18c4:::benchmark_data_smithi171_78918_object87151:head# 2024-04-20T17:11:22.113 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.113 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f0e7576:::benchmark_data_smithi171_78918_object76850:head# 2024-04-20T17:11:22.113 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.113 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f0e927f:::benchmark_data_smithi171_78918_object133121:head# 2024-04-20T17:11:22.114 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.114 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f0fbdd2:::benchmark_data_smithi171_78918_object156835:head# 2024-04-20T17:11:22.114 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.114 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f0fc282:::benchmark_data_smithi171_78918_object83585:head# 2024-04-20T17:11:22.114 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.114 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f10d115:::benchmark_data_smithi171_78918_object87404:head# 2024-04-20T17:11:22.114 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.114 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f11d4ce:::benchmark_data_smithi171_78918_object124734:head# 2024-04-20T17:11:22.114 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.114 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f12cb49:::benchmark_data_smithi171_78918_object137815:head# 2024-04-20T17:11:22.114 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.114 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f12db11:::benchmark_data_smithi171_78918_object88450:head# 2024-04-20T17:11:22.114 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.115 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f13e184:::benchmark_data_smithi171_78918_object75668:head# 2024-04-20T17:11:22.115 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.115 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f148e7b:::benchmark_data_smithi171_78918_object78633:head# 2024-04-20T17:11:22.115 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.115 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f1539f7:::benchmark_data_smithi171_78918_object72005:head# 2024-04-20T17:11:22.115 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.115 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f15e8e8:::benchmark_data_smithi171_78918_object92288:head# 2024-04-20T17:11:22.115 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.115 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f16640c:::benchmark_data_smithi171_78918_object81163:head# 2024-04-20T17:11:22.115 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.115 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f169e57:::benchmark_data_smithi171_78918_object133139:head# 2024-04-20T17:11:22.116 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.116 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f16a498:::benchmark_data_smithi171_78918_object93090:head# 2024-04-20T17:11:22.116 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.116 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f17709e:::benchmark_data_smithi171_78918_object89794:head# 2024-04-20T17:11:22.116 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.116 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f179a5c:::benchmark_data_smithi171_78918_object51468:head# 2024-04-20T17:11:22.116 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.116 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f18acc3:::benchmark_data_smithi171_78918_object133764:head# 2024-04-20T17:11:22.116 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.116 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f195824:::benchmark_data_smithi171_78918_object130897:head# 2024-04-20T17:11:22.117 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.117 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f1b461a:::benchmark_data_smithi171_78918_object82437:head# 2024-04-20T17:11:22.117 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.117 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f1c15ef:::benchmark_data_smithi171_78918_object124778:head# 2024-04-20T17:11:22.117 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.117 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f1cb917:::benchmark_data_smithi171_78918_object114950:head# 2024-04-20T17:11:22.117 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.117 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f1d3afc:::benchmark_data_smithi171_78918_object57136:head# 2024-04-20T17:11:22.117 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.117 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f1e1c2c:::benchmark_data_smithi171_78918_object67550:head# 2024-04-20T17:11:22.117 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.118 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f1e5ed1:::benchmark_data_smithi171_78918_object80555:head# 2024-04-20T17:11:22.118 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.118 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f1e8803:::benchmark_data_smithi171_78918_object118652:head# 2024-04-20T17:11:22.118 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.118 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f1fbb46:::benchmark_data_smithi171_78918_object135534:head# 2024-04-20T17:11:22.118 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.118 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f203e80:::benchmark_data_smithi171_78918_object146964:head# 2024-04-20T17:11:22.118 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.118 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f21a1e3:::benchmark_data_smithi171_78918_object117283:head# 2024-04-20T17:11:22.118 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.118 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f21a764:::benchmark_data_smithi171_78918_object128656:head# 2024-04-20T17:11:22.118 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.118 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f21f05c:::benchmark_data_smithi171_78918_object122224:head# 2024-04-20T17:11:22.119 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.119 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f21f760:::benchmark_data_smithi171_78918_object123792:head# 2024-04-20T17:11:22.119 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.119 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f2381e6:::benchmark_data_smithi171_78918_object93392:head# 2024-04-20T17:11:22.119 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.119 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f2432b1:::benchmark_data_smithi171_78918_object121306:head# 2024-04-20T17:11:22.119 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.119 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f245b74:::benchmark_data_smithi171_78918_object150396:head# 2024-04-20T17:11:22.119 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.119 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f24f481:::benchmark_data_smithi171_78918_object86624:head# 2024-04-20T17:11:22.119 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.119 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f252025:::benchmark_data_smithi171_78918_object63492:head# 2024-04-20T17:11:22.119 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.119 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f252f2b:::benchmark_data_smithi171_78918_object145833:head# 2024-04-20T17:11:22.120 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.120 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f25b082:::benchmark_data_smithi171_78918_object153193:head# 2024-04-20T17:11:22.120 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.120 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f2644fa:::benchmark_data_smithi171_78918_object90301:head# 2024-04-20T17:11:22.120 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.120 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f264d33:::benchmark_data_smithi171_78918_object58218:head# 2024-04-20T17:11:22.120 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.120 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f265f37:::benchmark_data_smithi171_78918_object84996:head# 2024-04-20T17:11:22.120 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.120 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f266611:::benchmark_data_smithi171_78918_object67862:head# 2024-04-20T17:11:22.120 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.120 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f26d509:::benchmark_data_smithi171_78918_object157916:head# 2024-04-20T17:11:22.120 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.120 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f26e5d8:::benchmark_data_smithi171_78918_object141212:head# 2024-04-20T17:11:22.120 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.121 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f285cfd:::benchmark_data_smithi171_78918_object58861:head# 2024-04-20T17:11:22.121 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.121 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f289234:::benchmark_data_smithi171_78918_object114843:head# 2024-04-20T17:11:22.121 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.121 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f291dc8:::benchmark_data_smithi171_78918_object135814:head# 2024-04-20T17:11:22.121 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.121 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f29cd26:::benchmark_data_smithi171_78918_object87300:head# 2024-04-20T17:11:22.121 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.121 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f2a03f9:::benchmark_data_smithi171_78918_object76028:head# 2024-04-20T17:11:22.121 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.121 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f2a252f:::benchmark_data_smithi171_78918_object151763:head# 2024-04-20T17:11:22.121 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.121 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f2a502c:::benchmark_data_smithi171_78918_object109265:head# 2024-04-20T17:11:22.121 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.122 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f2a5c73:::benchmark_data_smithi171_78918_object136144:head# 2024-04-20T17:11:22.122 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.122 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f2b3809:::benchmark_data_smithi171_78918_object148095:head# 2024-04-20T17:11:22.122 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.122 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f2c3e62:::benchmark_data_smithi171_78918_object66173:head# 2024-04-20T17:11:22.122 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.122 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f2c8947:::benchmark_data_smithi171_78918_object112684:head# 2024-04-20T17:11:22.122 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.122 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f2d78f2:::benchmark_data_smithi171_78918_object57241:head# 2024-04-20T17:11:22.122 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.122 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f2ec708:::benchmark_data_smithi171_78918_object86795:head# 2024-04-20T17:11:22.122 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.122 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f305481:::benchmark_data_smithi171_78918_object104537:head# 2024-04-20T17:11:22.122 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.123 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f3054b9:::benchmark_data_smithi171_78918_object155437:head# 2024-04-20T17:11:22.123 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.123 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f30d111:::benchmark_data_smithi171_78918_object78749:head# 2024-04-20T17:11:22.123 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.123 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f326d72:::benchmark_data_smithi171_78918_object52374:head# 2024-04-20T17:11:22.123 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.123 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f336635:::benchmark_data_smithi171_78918_object105489:head# 2024-04-20T17:11:22.123 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.123 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f33df7e:::benchmark_data_smithi171_78918_object101718:head# 2024-04-20T17:11:22.123 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.123 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f34d18d:::benchmark_data_smithi171_78918_object155725:head# 2024-04-20T17:11:22.123 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.123 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f357244:::benchmark_data_smithi171_78918_object73257:head# 2024-04-20T17:11:22.123 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.124 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f3632d0:::benchmark_data_smithi171_78918_object160503:head# 2024-04-20T17:11:22.124 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.124 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f36c0e5:::benchmark_data_smithi171_78918_object150518:head# 2024-04-20T17:11:22.124 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.124 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f36e5c6:::benchmark_data_smithi171_78918_object134398:head# 2024-04-20T17:11:22.124 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.124 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f36e5dc:::benchmark_data_smithi171_78918_object59494:head# 2024-04-20T17:11:22.124 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.124 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f377462:::benchmark_data_smithi171_78918_object102113:head# 2024-04-20T17:11:22.124 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.124 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f37fc0d:::benchmark_data_smithi171_78918_object115576:head# 2024-04-20T17:11:22.124 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.124 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f38cc5c:::benchmark_data_smithi171_78918_object58590:head# 2024-04-20T17:11:22.124 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.124 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f397656:::benchmark_data_smithi171_78918_object150517:head# 2024-04-20T17:11:22.125 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.125 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f39977c:::benchmark_data_smithi171_78918_object134137:head# 2024-04-20T17:11:22.125 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.125 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f39efb5:::benchmark_data_smithi171_78918_object112771:head# 2024-04-20T17:11:22.125 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.125 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f3a69d5:::benchmark_data_smithi171_78918_object148377:head# 2024-04-20T17:11:22.125 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.125 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f3a7230:::benchmark_data_smithi171_78918_object120581:head# 2024-04-20T17:11:22.125 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.125 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f3a8399:::benchmark_data_smithi171_78918_object145449:head# 2024-04-20T17:11:22.125 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.125 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f3ae10d:::benchmark_data_smithi171_78918_object81177:head# 2024-04-20T17:11:22.125 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.126 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f3bc155:::benchmark_data_smithi171_78918_object149425:head# 2024-04-20T17:11:22.126 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.126 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f3bd8a6:::benchmark_data_smithi171_78918_object123211:head# 2024-04-20T17:11:22.126 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.126 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f3c130d:::benchmark_data_smithi171_78918_object56937:head# 2024-04-20T17:11:22.126 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.126 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f3ca6ec:::benchmark_data_smithi171_78918_object73982:head# 2024-04-20T17:11:22.126 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.126 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f3d4c52:::benchmark_data_smithi171_78918_object58227:head# 2024-04-20T17:11:22.126 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.126 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f3d8552:::benchmark_data_smithi171_78918_object102819:head# 2024-04-20T17:11:22.126 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.126 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f3db23c:::benchmark_data_smithi171_78918_object89012:head# 2024-04-20T17:11:22.126 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.127 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f3de4e5:::benchmark_data_smithi171_78918_object64910:head# 2024-04-20T17:11:22.127 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.127 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f3e6cfb:::benchmark_data_smithi171_78918_object135457:head# 2024-04-20T17:11:22.127 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.127 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f405333:::benchmark_data_smithi171_78918_object136774:head# 2024-04-20T17:11:22.127 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.127 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f4066b4:::benchmark_data_smithi171_78918_object57282:head# 2024-04-20T17:11:22.127 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.127 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f41f435:::benchmark_data_smithi171_78918_object144019:head# 2024-04-20T17:11:22.127 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.127 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f4329e4:::benchmark_data_smithi171_78918_object101755:head# 2024-04-20T17:11:22.127 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.127 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f438849:::benchmark_data_smithi171_78918_object79088:head# 2024-04-20T17:11:22.127 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.128 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f44fe61:::benchmark_data_smithi171_78918_object133245:head# 2024-04-20T17:11:22.128 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.128 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f4592ce:::benchmark_data_smithi171_78918_object131368:head# 2024-04-20T17:11:22.128 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.128 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f4636cf:::benchmark_data_smithi171_78918_object149199:head# 2024-04-20T17:11:22.128 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.128 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f48c972:::benchmark_data_smithi171_78918_object134900:head# 2024-04-20T17:11:22.128 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.128 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f48f36c:::benchmark_data_smithi171_78918_object52168:head# 2024-04-20T17:11:22.128 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.128 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f491d92:::benchmark_data_smithi171_78918_object65704:head# 2024-04-20T17:11:22.128 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.128 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f4a2e91:::benchmark_data_smithi171_78918_object136671:head# 2024-04-20T17:11:22.128 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.128 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f4bb22b:::benchmark_data_smithi171_78918_object120243:head# 2024-04-20T17:11:22.129 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.129 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f4be5fb:::benchmark_data_smithi171_78918_object148691:head# 2024-04-20T17:11:22.129 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.129 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f4c2fb2:::benchmark_data_smithi171_78918_object70567:head# 2024-04-20T17:11:22.129 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.129 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f4d8e43:::benchmark_data_smithi171_78918_object93553:head# 2024-04-20T17:11:22.129 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.129 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f4e235d:::benchmark_data_smithi171_78918_object74353:head# 2024-04-20T17:11:22.129 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.129 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f5015e7:::benchmark_data_smithi171_78918_object137197:head# 2024-04-20T17:11:22.129 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.129 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f51f5ec:::benchmark_data_smithi171_78918_object129214:head# 2024-04-20T17:11:22.129 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.129 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f52a549:::benchmark_data_smithi171_78918_object52541:head# 2024-04-20T17:11:22.130 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.130 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f52f506:::benchmark_data_smithi171_78918_object75095:head# 2024-04-20T17:11:22.130 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.130 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f53ea1e:::benchmark_data_smithi171_78918_object80073:head# 2024-04-20T17:11:22.130 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.130 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f543de3:::benchmark_data_smithi171_78918_object85987:head# 2024-04-20T17:11:22.130 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.130 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f5493a6:::benchmark_data_smithi171_78918_object81251:head# 2024-04-20T17:11:22.130 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.130 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f54ed05:::benchmark_data_smithi171_78918_object159656:head# 2024-04-20T17:11:22.130 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.130 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f550192:::benchmark_data_smithi171_78918_object155242:head# 2024-04-20T17:11:22.130 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.130 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f55644e:::benchmark_data_smithi171_78918_object140756:head# 2024-04-20T17:11:22.131 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.131 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f559d66:::benchmark_data_smithi171_78918_object88873:head# 2024-04-20T17:11:22.131 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.131 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f55b995:::benchmark_data_smithi171_78918_object150474:head# 2024-04-20T17:11:22.131 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.131 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f56c7ef:::benchmark_data_smithi171_78918_object101374:head# 2024-04-20T17:11:22.131 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.131 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f56e1b8:::benchmark_data_smithi171_78918_object54180:head# 2024-04-20T17:11:22.131 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.131 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f5750e6:::benchmark_data_smithi171_78918_object55414:head# 2024-04-20T17:11:22.131 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.131 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f5762e7:::benchmark_data_smithi171_78918_object147381:head# 2024-04-20T17:11:22.131 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.131 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f583c85:::benchmark_data_smithi171_78918_object128846:head# 2024-04-20T17:11:22.132 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.132 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f58647a:::benchmark_data_smithi171_78918_object116821:head# 2024-04-20T17:11:22.132 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.132 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f590c28:::benchmark_data_smithi171_78918_object90789:head# 2024-04-20T17:11:22.132 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.132 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f5966f8:::benchmark_data_smithi171_78918_object127363:head# 2024-04-20T17:11:22.132 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.132 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f59f91e:::benchmark_data_smithi171_78918_object98858:head# 2024-04-20T17:11:22.132 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.132 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f5a69b6:::benchmark_data_smithi171_78918_object62724:head# 2024-04-20T17:11:22.132 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.132 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f5a7d32:::benchmark_data_smithi171_78918_object69707:head# 2024-04-20T17:11:22.132 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.133 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f5a814b:::benchmark_data_smithi171_78918_object104542:head# 2024-04-20T17:11:22.133 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.133 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f5aba44:::benchmark_data_smithi171_78918_object69708:head# 2024-04-20T17:11:22.133 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.133 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f5b00b0:::benchmark_data_smithi171_78918_object138946:head# 2024-04-20T17:11:22.133 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.133 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f5bc5d5:::benchmark_data_smithi171_78918_object101735:head# 2024-04-20T17:11:22.133 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.133 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f5c742f:::benchmark_data_smithi171_78918_object160700:head# 2024-04-20T17:11:22.133 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.133 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f5ddaff:::benchmark_data_smithi171_78918_object146864:head# 2024-04-20T17:11:22.133 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.133 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f5e0d36:::benchmark_data_smithi171_78918_object110824:head# 2024-04-20T17:11:22.133 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.133 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f5eaa0f:::benchmark_data_smithi171_78918_object153936:head# 2024-04-20T17:11:22.134 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.134 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f600bc9:::benchmark_data_smithi171_78918_object149522:head# 2024-04-20T17:11:22.134 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.134 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f60b61b:::benchmark_data_smithi171_78918_object127502:head# 2024-04-20T17:11:22.134 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.134 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f61b8ae:::benchmark_data_smithi171_78918_object153972:head# 2024-04-20T17:11:22.134 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.134 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f61b8e2:::benchmark_data_smithi171_78918_object104264:head# 2024-04-20T17:11:22.134 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.134 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f620f53:::benchmark_data_smithi171_78918_object93156:head# 2024-04-20T17:11:22.134 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.134 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f62750d:::benchmark_data_smithi171_78918_object140970:head# 2024-04-20T17:11:22.134 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.135 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f62e90e:::benchmark_data_smithi171_78918_object53309:head# 2024-04-20T17:11:22.135 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.135 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f6448cc:::benchmark_data_smithi171_78918_object152092:head# 2024-04-20T17:11:22.135 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.135 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f644b4f:::benchmark_data_smithi171_78918_object75739:head# 2024-04-20T17:11:22.135 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.135 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f645b85:::benchmark_data_smithi171_78918_object63060:head# 2024-04-20T17:11:22.135 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.136 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f6481f2:::benchmark_data_smithi171_78918_object75444:head# 2024-04-20T17:11:22.136 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.136 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f65016b:::benchmark_data_smithi171_78918_object126593:head# 2024-04-20T17:11:22.136 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.136 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f652845:::benchmark_data_smithi171_78918_object64886:head# 2024-04-20T17:11:22.136 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.136 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f660921:::benchmark_data_smithi171_78918_object124242:head# 2024-04-20T17:11:22.136 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.136 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f66adc9:::benchmark_data_smithi171_78918_object116178:head# 2024-04-20T17:11:22.136 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.136 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f67834a:::benchmark_data_smithi171_78918_object133797:head# 2024-04-20T17:11:22.136 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.136 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f678b1c:::benchmark_data_smithi171_78918_object84017:head# 2024-04-20T17:11:22.137 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.137 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f679b7d:::benchmark_data_smithi171_78918_object66130:head# 2024-04-20T17:11:22.137 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.137 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f688978:::benchmark_data_smithi171_78918_object116511:head# 2024-04-20T17:11:22.137 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.137 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f68fe78:::benchmark_data_smithi171_78918_object154818:head# 2024-04-20T17:11:22.137 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.137 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f694ce2:::benchmark_data_smithi171_78918_object60205:head# 2024-04-20T17:11:22.137 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.137 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f69bb46:::benchmark_data_smithi171_78918_object85278:head# 2024-04-20T17:11:22.137 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.137 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f6a0fff:::benchmark_data_smithi171_78918_object67944:head# 2024-04-20T17:11:22.138 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.138 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f6a1572:::benchmark_data_smithi171_78918_object95746:head# 2024-04-20T17:11:22.138 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.138 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f6a7aa7:::benchmark_data_smithi171_78918_object121515:head# 2024-04-20T17:11:22.138 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.138 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f6abf8a:::benchmark_data_smithi171_78918_object111698:head# 2024-04-20T17:11:22.138 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.138 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f6ae97f:::benchmark_data_smithi171_78918_object137705:head# 2024-04-20T17:11:22.138 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.138 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f6bdc0f:::benchmark_data_smithi171_78918_object127173:head# 2024-04-20T17:11:22.138 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.138 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f6ca743:::benchmark_data_smithi171_78918_object81288:head# 2024-04-20T17:11:22.139 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.139 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f6dc778:::benchmark_data_smithi171_78918_object116567:head# 2024-04-20T17:11:22.139 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.139 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f6e5220:::benchmark_data_smithi171_78918_object158080:head# 2024-04-20T17:11:22.139 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.139 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f6e6108:::benchmark_data_smithi171_78918_object102636:head# 2024-04-20T17:11:22.139 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.139 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f6e8309:::benchmark_data_smithi171_78918_object71142:head# 2024-04-20T17:11:22.139 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.139 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f6ec039:::benchmark_data_smithi171_78918_object148622:head# 2024-04-20T17:11:22.139 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.140 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f6f1088:::benchmark_data_smithi171_78918_object116887:head# 2024-04-20T17:11:22.140 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.140 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f707558:::benchmark_data_smithi171_78918_object111119:head# 2024-04-20T17:11:22.140 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.140 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f716b42:::benchmark_data_smithi171_78918_object62526:head# 2024-04-20T17:11:22.140 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.140 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f72421f:::benchmark_data_smithi171_78918_object108036:head# 2024-04-20T17:11:22.140 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.140 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f726366:::benchmark_data_smithi171_78918_object156549:head# 2024-04-20T17:11:22.140 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.141 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f7507dc:::benchmark_data_smithi171_78918_object72890:head# 2024-04-20T17:11:22.141 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.141 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f758ab9:::benchmark_data_smithi171_78918_object134182:head# 2024-04-20T17:11:22.141 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.141 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f75f505:::benchmark_data_smithi171_78918_object69546:head# 2024-04-20T17:11:22.141 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.141 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f760908:::benchmark_data_smithi171_78918_object145403:head# 2024-04-20T17:11:22.141 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.141 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f7625fa:::benchmark_data_smithi171_78918_object87914:head# 2024-04-20T17:11:22.141 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.142 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f77a6e2:::benchmark_data_smithi171_78918_object115029:head# 2024-04-20T17:11:22.142 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.142 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f783a4a:::benchmark_data_smithi171_78918_object79372:head# 2024-04-20T17:11:22.142 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.142 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f79aeef:::benchmark_data_smithi171_78918_object145086:head# 2024-04-20T17:11:22.142 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.142 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f7a4471:::benchmark_data_smithi171_78918_object103444:head# 2024-04-20T17:11:22.142 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.142 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f7b54af:::benchmark_data_smithi171_78918_object120617:head# 2024-04-20T17:11:22.142 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.142 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f7bf6c5:::benchmark_data_smithi171_78918_object76452:head# 2024-04-20T17:11:22.142 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.143 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f7d10bc:::benchmark_data_smithi171_78918_object127732:head# 2024-04-20T17:11:22.143 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.143 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f7d3fa2:::benchmark_data_smithi171_78918_object147950:head# 2024-04-20T17:11:22.143 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.143 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f7da0d9:::benchmark_data_smithi171_78918_object108651:head# 2024-04-20T17:11:22.143 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.143 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f7dec31:::benchmark_data_smithi171_78918_object57768:head# 2024-04-20T17:11:22.143 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.143 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f7e2510:::benchmark_data_smithi171_78918_object123322:head# 2024-04-20T17:11:22.143 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.143 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f7fb0c6:::benchmark_data_smithi171_78918_object126278:head# 2024-04-20T17:11:22.143 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.144 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f8048d6:::benchmark_data_smithi171_78918_object91574:head# 2024-04-20T17:11:22.144 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.144 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f81b7ba:::benchmark_data_smithi171_78918_object136737:head# 2024-04-20T17:11:22.144 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.144 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f83776e:::benchmark_data_smithi171_78918_object117039:head# 2024-04-20T17:11:22.144 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.144 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f839b2e:::benchmark_data_smithi171_78918_object62917:head# 2024-04-20T17:11:22.144 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.144 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f845f7b:::benchmark_data_smithi171_78918_object127362:head# 2024-04-20T17:11:22.144 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.144 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f846394:::benchmark_data_smithi171_78918_object120489:head# 2024-04-20T17:11:22.145 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.145 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f84a251:::benchmark_data_smithi171_78918_object91167:head# 2024-04-20T17:11:22.145 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.145 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f84e80c:::benchmark_data_smithi171_78918_object133682:head# 2024-04-20T17:11:22.145 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.145 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f866a29:::benchmark_data_smithi171_78918_object101508:head# 2024-04-20T17:11:22.145 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.145 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f879b76:::benchmark_data_smithi171_78918_object145154:head# 2024-04-20T17:11:22.145 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.145 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f87a66b:::benchmark_data_smithi171_78918_object126759:head# 2024-04-20T17:11:22.146 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.146 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f87f578:::benchmark_data_smithi171_78918_object119635:head# 2024-04-20T17:11:22.146 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.146 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f884653:::benchmark_data_smithi171_78918_object120772:head# 2024-04-20T17:11:22.146 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.146 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f88553b:::benchmark_data_smithi171_78918_object140385:head# 2024-04-20T17:11:22.146 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.146 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f89103a:::benchmark_data_smithi171_78918_object79370:head# 2024-04-20T17:11:22.147 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.147 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f8a4f6e:::benchmark_data_smithi171_78918_object122927:head# 2024-04-20T17:11:22.147 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.147 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f8aec4f:::benchmark_data_smithi171_78918_object123827:head# 2024-04-20T17:11:22.147 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.147 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f8b5215:::benchmark_data_smithi171_78918_object80726:head# 2024-04-20T17:11:22.147 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.147 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f8be229:::benchmark_data_smithi171_78918_object68993:head# 2024-04-20T17:11:22.147 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.147 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f8c72c2:::benchmark_data_smithi171_78918_object87671:head# 2024-04-20T17:11:22.148 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.148 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f8e665a:::benchmark_data_smithi171_78918_object113500:head# 2024-04-20T17:11:22.148 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.148 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f8e8ff9:::benchmark_data_smithi171_78918_object129714:head# 2024-04-20T17:11:22.148 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.148 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f8fa4a7:::benchmark_data_smithi171_78918_object158162:head# 2024-04-20T17:11:22.148 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.148 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f90aaa5:::benchmark_data_smithi171_78918_object114763:head# 2024-04-20T17:11:22.148 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.148 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f90e995:::benchmark_data_smithi171_78918_object113951:head# 2024-04-20T17:11:22.149 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.149 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f911849:::benchmark_data_smithi171_78918_object122423:head# 2024-04-20T17:11:22.149 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.149 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f91e068:::benchmark_data_smithi171_78918_object142183:head# 2024-04-20T17:11:22.149 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.149 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f925174:::benchmark_data_smithi171_78918_object157318:head# 2024-04-20T17:11:22.149 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.149 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f928be0:::benchmark_data_smithi171_78918_object77122:head# 2024-04-20T17:11:22.149 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.149 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f92eb13:::benchmark_data_smithi171_78918_object111668:head# 2024-04-20T17:11:22.149 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.149 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f9480e6:::benchmark_data_smithi171_78918_object124379:head# 2024-04-20T17:11:22.149 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.150 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f95286d:::benchmark_data_smithi171_78918_object109982:head# 2024-04-20T17:11:22.150 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.150 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f954e39:::benchmark_data_smithi171_78918_object107441:head# 2024-04-20T17:11:22.150 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.150 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f95ca56:::benchmark_data_smithi171_78918_object144069:head# 2024-04-20T17:11:22.150 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.150 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f9649e0:::benchmark_data_smithi171_78918_object99701:head# 2024-04-20T17:11:22.150 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.150 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f96890d:::benchmark_data_smithi171_78918_object128175:head# 2024-04-20T17:11:22.150 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.150 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f977efd:::benchmark_data_smithi171_78918_object70171:head# 2024-04-20T17:11:22.150 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.150 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f9811ea:::benchmark_data_smithi171_78918_object84227:head# 2024-04-20T17:11:22.151 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.151 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f98a4cb:::benchmark_data_smithi171_78918_object155620:head# 2024-04-20T17:11:22.151 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.151 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f993a2f:::benchmark_data_smithi171_78918_object79502:head# 2024-04-20T17:11:22.151 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.151 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f995894:::benchmark_data_smithi171_78918_object144117:head# 2024-04-20T17:11:22.151 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.151 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f99ba94:::benchmark_data_smithi171_78918_object141251:head# 2024-04-20T17:11:22.151 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.151 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f9a075e:::benchmark_data_smithi171_78918_object91015:head# 2024-04-20T17:11:22.151 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.151 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f9a163a:::benchmark_data_smithi171_78918_object51772:head# 2024-04-20T17:11:22.151 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.152 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f9ab684:::benchmark_data_smithi171_78918_object82039:head# 2024-04-20T17:11:22.152 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.152 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f9b051b:::benchmark_data_smithi171_78918_object150122:head# 2024-04-20T17:11:22.152 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.152 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f9b9530:::benchmark_data_smithi171_78918_object129290:head# 2024-04-20T17:11:22.152 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.152 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f9d09e0:::benchmark_data_smithi171_78918_object115435:head# 2024-04-20T17:11:22.152 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.152 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f9d7714:::benchmark_data_smithi171_78918_object53123:head# 2024-04-20T17:11:22.152 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.152 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f9db9df:::benchmark_data_smithi171_78918_object153928:head# 2024-04-20T17:11:22.153 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.153 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f9de6d5:::benchmark_data_smithi171_78918_object157948:head# 2024-04-20T17:11:22.153 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.153 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f9e0372:::benchmark_data_smithi171_78918_object93994:head# 2024-04-20T17:11:22.153 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.153 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f9e1376:::benchmark_data_smithi171_78918_object62152:head# 2024-04-20T17:11:22.153 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.153 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f9e5698:::benchmark_data_smithi171_78918_object144439:head# 2024-04-20T17:11:22.153 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.153 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7f9eec05:::benchmark_data_smithi171_78918_object127201:head# 2024-04-20T17:11:22.154 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.154 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fa0c715:::benchmark_data_smithi171_78918_object116456:head# 2024-04-20T17:11:22.154 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.154 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fa0f0fe:::benchmark_data_smithi171_78918_object83591:head# 2024-04-20T17:11:22.154 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.154 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fa34835:::benchmark_data_smithi171_78918_object128314:head# 2024-04-20T17:11:22.154 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.154 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fa386d8:::benchmark_data_smithi171_78918_object73965:head# 2024-04-20T17:11:22.154 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.154 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fa3fa93:::benchmark_data_smithi171_78918_object52905:head# 2024-04-20T17:11:22.154 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.154 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fa4a3d2:::benchmark_data_smithi171_78918_object103696:head# 2024-04-20T17:11:22.155 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.155 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fa5d80b:::benchmark_data_smithi171_78918_object143846:head# 2024-04-20T17:11:22.155 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.155 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fa5ffaa:::benchmark_data_smithi171_78918_object69504:head# 2024-04-20T17:11:22.155 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.155 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fa60edc:::benchmark_data_smithi171_78918_object57272:head# 2024-04-20T17:11:22.155 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.155 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fa6a015:::benchmark_data_smithi171_78918_object59226:head# 2024-04-20T17:11:22.155 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.155 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fa6bf11:::benchmark_data_smithi171_78918_object158760:head# 2024-04-20T17:11:22.155 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.155 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fa715c6:::benchmark_data_smithi171_78918_object78022:head# 2024-04-20T17:11:22.155 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.155 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fa7b71b:::benchmark_data_smithi171_78918_object136631:head# 2024-04-20T17:11:22.156 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.156 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fa7dd38:::benchmark_data_smithi171_78918_object63084:head# 2024-04-20T17:11:22.156 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.156 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fa8ee6c:::benchmark_data_smithi171_78918_object53835:head# 2024-04-20T17:11:22.156 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.156 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fa9a3cb:::benchmark_data_smithi171_78918_object110104:head# 2024-04-20T17:11:22.156 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.156 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fa9f186:::benchmark_data_smithi171_78918_object89923:head# 2024-04-20T17:11:22.156 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.156 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7faa2ba9:::benchmark_data_smithi171_78918_object99696:head# 2024-04-20T17:11:22.156 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.156 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7faab486:::benchmark_data_smithi171_78918_object86425:head# 2024-04-20T17:11:22.156 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.157 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fabb0e1:::benchmark_data_smithi171_78918_object153570:head# 2024-04-20T17:11:22.157 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.157 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fabb98f:::benchmark_data_smithi171_78918_object61948:head# 2024-04-20T17:11:22.157 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.157 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fabd30d:::benchmark_data_smithi171_78918_object153583:head# 2024-04-20T17:11:22.157 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.157 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fabda3a:::benchmark_data_smithi171_78918_object104473:head# 2024-04-20T17:11:22.157 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.157 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fac606e:::benchmark_data_smithi171_78918_object55758:head# 2024-04-20T17:11:22.157 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.157 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fac77a2:::benchmark_data_smithi171_78918_object96285:head# 2024-04-20T17:11:22.157 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.157 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7facb511:::benchmark_data_smithi171_78918_object83095:head# 2024-04-20T17:11:22.157 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.158 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7facc2fb:::benchmark_data_smithi171_78918_object106128:head# 2024-04-20T17:11:22.158 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.158 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fad20a0:::benchmark_data_smithi171_78918_object117403:head# 2024-04-20T17:11:22.158 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.158 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fad391a:::benchmark_data_smithi171_78918_object120824:head# 2024-04-20T17:11:22.158 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.158 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fae0e12:::benchmark_data_smithi171_78918_object101831:head# 2024-04-20T17:11:22.158 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.158 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7faf1f35:::benchmark_data_smithi171_78918_object104527:head# 2024-04-20T17:11:22.158 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.158 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7faf9370:::benchmark_data_smithi171_78918_object122049:head# 2024-04-20T17:11:22.158 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.158 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fafc47a:::benchmark_data_smithi171_78918_object146961:head# 2024-04-20T17:11:22.158 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.159 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fb025b4:::benchmark_data_smithi171_78918_object122183:head# 2024-04-20T17:11:22.159 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.159 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fb04c44:::benchmark_data_smithi171_78918_object97241:head# 2024-04-20T17:11:22.159 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.159 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fb0e9d3:::benchmark_data_smithi171_78918_object59248:head# 2024-04-20T17:11:22.159 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.159 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fb1c2bd:::benchmark_data_smithi171_78918_object156210:head# 2024-04-20T17:11:22.159 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.159 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fb2c034:::benchmark_data_smithi171_78918_object89591:head# 2024-04-20T17:11:22.159 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.159 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fb31ddd:::benchmark_data_smithi171_78918_object63797:head# 2024-04-20T17:11:22.159 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.159 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fb3719a:::benchmark_data_smithi171_78918_object156343:head# 2024-04-20T17:11:22.159 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.160 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fb3a23d:::benchmark_data_smithi171_78918_object132547:head# 2024-04-20T17:11:22.160 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.160 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fb42348:::benchmark_data_smithi171_78918_object147154:head# 2024-04-20T17:11:22.160 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.160 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fb4afe7:::benchmark_data_smithi171_78918_object60315:head# 2024-04-20T17:11:22.160 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.160 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fb55ab8:::benchmark_data_smithi171_78918_object86845:head# 2024-04-20T17:11:22.160 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.160 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fb66485:::benchmark_data_smithi171_78918_object77276:head# 2024-04-20T17:11:22.160 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.160 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fb6d72a:::benchmark_data_smithi171_78918_object91706:head# 2024-04-20T17:11:22.160 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.160 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fb6f1c2:::benchmark_data_smithi171_78918_object91923:head# 2024-04-20T17:11:22.160 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.161 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fb7ded0:::benchmark_data_smithi171_78918_object59865:head# 2024-04-20T17:11:22.161 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.161 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fb9ffa2:::benchmark_data_smithi171_78918_object146133:head# 2024-04-20T17:11:22.161 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.161 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fba428d:::benchmark_data_smithi171_78918_object122688:head# 2024-04-20T17:11:22.161 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.161 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fba758f:::benchmark_data_smithi171_78918_object118583:head# 2024-04-20T17:11:22.161 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.161 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fbc3348:::benchmark_data_smithi171_78918_object107435:head# 2024-04-20T17:11:22.161 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.161 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fbc9598:::benchmark_data_smithi171_78918_object53098:head# 2024-04-20T17:11:22.161 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.161 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fbca922:::benchmark_data_smithi171_78918_object75418:head# 2024-04-20T17:11:22.161 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.162 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fbe8619:::benchmark_data_smithi171_78918_object155291:head# 2024-04-20T17:11:22.162 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.162 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fbfd5fa:::benchmark_data_smithi171_78918_object79400:head# 2024-04-20T17:11:22.162 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.162 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fc03a14:::benchmark_data_smithi171_78918_object121639:head# 2024-04-20T17:11:22.162 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.162 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fc2215f:::benchmark_data_smithi171_78918_object141342:head# 2024-04-20T17:11:22.162 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.162 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fc2346a:::benchmark_data_smithi171_78918_object122379:head# 2024-04-20T17:11:22.162 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.162 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fc2b643:::benchmark_data_smithi171_78918_object91944:head# 2024-04-20T17:11:22.162 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.162 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fc2bc2d:::benchmark_data_smithi171_78918_object64113:head# 2024-04-20T17:11:22.162 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.163 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fc2e994:::benchmark_data_smithi171_78918_object90648:head# 2024-04-20T17:11:22.163 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.163 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fc36123:::benchmark_data_smithi171_78918_object94761:head# 2024-04-20T17:11:22.163 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.163 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fc3799e:::benchmark_data_smithi171_78918_object159366:head# 2024-04-20T17:11:22.163 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.163 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fc3eb6e:::benchmark_data_smithi171_78918_object136693:head# 2024-04-20T17:11:22.163 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.163 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fc438ed:::benchmark_data_smithi171_78918_object159829:head# 2024-04-20T17:11:22.163 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.163 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fc4433b:::benchmark_data_smithi171_78918_object86079:head# 2024-04-20T17:11:22.163 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.163 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fc48be9:::benchmark_data_smithi171_78918_object76892:head# 2024-04-20T17:11:22.163 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.163 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fc5245d:::benchmark_data_smithi171_78918_object98700:head# 2024-04-20T17:11:22.164 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.164 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fc540f9:::benchmark_data_smithi171_78918_object97725:head# 2024-04-20T17:11:22.164 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.164 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fc5e3ef:::benchmark_data_smithi171_78918_object119456:head# 2024-04-20T17:11:22.164 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.164 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fc7575a:::benchmark_data_smithi171_78918_object66978:head# 2024-04-20T17:11:22.164 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.164 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fc7756d:::benchmark_data_smithi171_78918_object61770:head# 2024-04-20T17:11:22.164 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.164 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fc83f88:::benchmark_data_smithi171_78918_object77630:head# 2024-04-20T17:11:22.164 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.164 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fc845a5:::benchmark_data_smithi171_78918_object65307:head# 2024-04-20T17:11:22.164 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.164 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fc9a265:::benchmark_data_smithi171_78918_object129906:head# 2024-04-20T17:11:22.164 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.165 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fcad362:::benchmark_data_smithi171_78918_object114101:head# 2024-04-20T17:11:22.165 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.165 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fcb0b72:::benchmark_data_smithi171_78918_object124849:head# 2024-04-20T17:11:22.165 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.165 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fcbebb4:::benchmark_data_smithi171_78918_object107268:head# 2024-04-20T17:11:22.165 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.165 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fcbf5aa:::benchmark_data_smithi171_78918_object51978:head# 2024-04-20T17:11:22.165 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.165 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fcc1616:::benchmark_data_smithi171_78918_object104192:head# 2024-04-20T17:11:22.165 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.165 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fccf11b:::benchmark_data_smithi171_78918_object122562:head# 2024-04-20T17:11:22.165 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.165 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fce427a:::benchmark_data_smithi171_78918_object63693:head# 2024-04-20T17:11:22.165 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.165 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fce67d7:::benchmark_data_smithi171_78918_object93298:head# 2024-04-20T17:11:22.166 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.166 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fcee39b:::benchmark_data_smithi171_78918_object129627:head# 2024-04-20T17:11:22.166 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.166 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fcf6791:::benchmark_data_smithi171_78918_object152720:head# 2024-04-20T17:11:22.166 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.166 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fcf89bf:::benchmark_data_smithi171_78918_object152236:head# 2024-04-20T17:11:22.166 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.166 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fcffd00:::benchmark_data_smithi171_78918_object158824:head# 2024-04-20T17:11:22.166 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.166 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fd05632:::benchmark_data_smithi171_78918_object99929:head# 2024-04-20T17:11:22.166 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.166 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fd0ab0c:::benchmark_data_smithi171_78918_object79344:head# 2024-04-20T17:11:22.166 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.166 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fd18d56:::benchmark_data_smithi171_78918_object94822:head# 2024-04-20T17:11:22.167 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.167 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fd2979e:::benchmark_data_smithi171_78918_object141802:head# 2024-04-20T17:11:22.167 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.167 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fd38cc4:::benchmark_data_smithi171_78918_object105745:head# 2024-04-20T17:11:22.167 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.167 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fd3acfa:::benchmark_data_smithi171_78918_object157587:head# 2024-04-20T17:11:22.167 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.167 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fd3dc79:::benchmark_data_smithi171_78918_object111858:head# 2024-04-20T17:11:22.167 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.167 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fd469df:::benchmark_data_smithi171_78918_object88683:head# 2024-04-20T17:11:22.167 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.167 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fd4c365:::benchmark_data_smithi171_78918_object80284:head# 2024-04-20T17:11:22.167 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.167 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fd52b65:::benchmark_data_smithi171_78918_object117720:head# 2024-04-20T17:11:22.167 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.168 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fd52fb9:::benchmark_data_smithi171_78918_object114051:head# 2024-04-20T17:11:22.168 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.168 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fd542e6:::benchmark_data_smithi171_78918_object85797:head# 2024-04-20T17:11:22.168 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.168 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fd6265b:::benchmark_data_smithi171_78918_object112367:head# 2024-04-20T17:11:22.168 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.168 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fd6c35f:::benchmark_data_smithi171_78918_object86851:head# 2024-04-20T17:11:22.168 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.168 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fd78ce1:::benchmark_data_smithi171_78918_object105914:head# 2024-04-20T17:11:22.168 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.168 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fd7ecfc:::benchmark_data_smithi171_78918_object142842:head# 2024-04-20T17:11:22.168 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.168 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fd8c6e1:::benchmark_data_smithi171_78918_object117740:head# 2024-04-20T17:11:22.168 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.169 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fd9ca9e:::benchmark_data_smithi171_78918_object76545:head# 2024-04-20T17:11:22.169 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.169 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fda2e6e:::benchmark_data_smithi171_78918_object97214:head# 2024-04-20T17:11:22.169 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.169 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fdab88f:::benchmark_data_smithi171_78918_object118483:head# 2024-04-20T17:11:22.169 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.169 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fdae1b6:::benchmark_data_smithi171_78918_object141623:head# 2024-04-20T17:11:22.169 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.169 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fdc11fe:::benchmark_data_smithi171_78918_object78290:head# 2024-04-20T17:11:22.169 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.169 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fdc3107:::benchmark_data_smithi171_78918_object155702:head# 2024-04-20T17:11:22.169 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.169 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fdc6dc2:::benchmark_data_smithi171_78918_object103284:head# 2024-04-20T17:11:22.169 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.169 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fdce067:::benchmark_data_smithi171_78918_object67095:head# 2024-04-20T17:11:22.170 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.170 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fdd84b3:::benchmark_data_smithi171_78918_object91335:head# 2024-04-20T17:11:22.170 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.170 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fde1dbe:::benchmark_data_smithi171_78918_object84129:head# 2024-04-20T17:11:22.170 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.170 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fde3e81:::benchmark_data_smithi171_78918_object105159:head# 2024-04-20T17:11:22.170 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.170 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fde4f03:::benchmark_data_smithi171_78918_object147443:head# 2024-04-20T17:11:22.170 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.170 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fe03632:::benchmark_data_smithi171_78918_object102912:head# 2024-04-20T17:11:22.170 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.170 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fe07f36:::benchmark_data_smithi171_78918_object96979:head# 2024-04-20T17:11:22.170 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.170 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fe0b601:::benchmark_data_smithi171_78918_object122929:head# 2024-04-20T17:11:22.171 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.171 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fe1da77:::benchmark_data_smithi171_78918_object71992:head# 2024-04-20T17:11:22.171 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.171 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fe22673:::benchmark_data_smithi171_78918_object106246:head# 2024-04-20T17:11:22.171 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.171 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fe41b6f:::benchmark_data_smithi171_78918_object115382:head# 2024-04-20T17:11:22.171 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.171 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fe4c2c8:::benchmark_data_smithi171_78918_object76700:head# 2024-04-20T17:11:22.171 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.171 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fe5aab9:::benchmark_data_smithi171_78918_object130996:head# 2024-04-20T17:11:22.171 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.171 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fe5d3f0:::benchmark_data_smithi171_78918_object60428:head# 2024-04-20T17:11:22.171 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.171 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fe68b09:::benchmark_data_smithi171_78918_object125668:head# 2024-04-20T17:11:22.172 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.172 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fe91cee:::benchmark_data_smithi171_78918_object62845:head# 2024-04-20T17:11:22.172 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.172 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fe952fa:::benchmark_data_smithi171_78918_object61835:head# 2024-04-20T17:11:22.172 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.172 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fe96b14:::benchmark_data_smithi171_78918_object128452:head# 2024-04-20T17:11:22.172 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.172 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fea5044:::benchmark_data_smithi171_78918_object56883:head# 2024-04-20T17:11:22.172 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.172 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7feaf7ac:::benchmark_data_smithi171_78918_object67832:head# 2024-04-20T17:11:22.172 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.172 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7febab3d:::benchmark_data_smithi171_78918_object118456:head# 2024-04-20T17:11:22.172 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.172 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7febb5ff:::benchmark_data_smithi171_78918_object52216:head# 2024-04-20T17:11:22.172 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.173 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7febccda:::benchmark_data_smithi171_78918_object148253:head# 2024-04-20T17:11:22.173 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.173 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fee2557:::benchmark_data_smithi171_78918_object86659:head# 2024-04-20T17:11:22.173 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.173 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fee5c32:::benchmark_data_smithi171_78918_object154329:head# 2024-04-20T17:11:22.173 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.173 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7feeec89:::benchmark_data_smithi171_78918_object78889:head# 2024-04-20T17:11:22.173 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.173 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ff0b874:::benchmark_data_smithi171_78918_object142060:head# 2024-04-20T17:11:22.173 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.173 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ff19787:::benchmark_data_smithi171_78918_object73497:head# 2024-04-20T17:11:22.173 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.173 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ff2e6a6:::benchmark_data_smithi171_78918_object138004:head# 2024-04-20T17:11:22.173 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.173 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ff2fbf0:::benchmark_data_smithi171_78918_object131199:head# 2024-04-20T17:11:22.174 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.174 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ff3054e:::benchmark_data_smithi171_78918_object68022:head# 2024-04-20T17:11:22.174 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.174 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ff30a64:::benchmark_data_smithi171_78918_object118094:head# 2024-04-20T17:11:22.174 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.174 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ff321a9:::benchmark_data_smithi171_78918_object112908:head# 2024-04-20T17:11:22.174 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.174 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ff3bbdc:::benchmark_data_smithi171_78918_object118765:head# 2024-04-20T17:11:22.174 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.174 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ff3f5f3:::benchmark_data_smithi171_78918_object153048:head# 2024-04-20T17:11:22.174 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.174 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ff41ee0:::benchmark_data_smithi171_78918_object148683:head# 2024-04-20T17:11:22.174 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.174 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ff4597a:::benchmark_data_smithi171_78918_object109950:head# 2024-04-20T17:11:22.175 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.175 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ff67a64:::benchmark_data_smithi171_78918_object87044:head# 2024-04-20T17:11:22.175 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.175 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ff741bd:::benchmark_data_smithi171_78918_object91014:head# 2024-04-20T17:11:22.175 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.175 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ff7c5de:::benchmark_data_smithi171_78918_object112519:head# 2024-04-20T17:11:22.175 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.175 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ff7e8bf:::benchmark_data_smithi171_78918_object77831:head# 2024-04-20T17:11:22.175 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.175 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ff8e0c3:::benchmark_data_smithi171_78918_object58026:head# 2024-04-20T17:11:22.175 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.175 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ffae170:::benchmark_data_smithi171_78918_object136355:head# 2024-04-20T17:11:22.175 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.175 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ffc80cf:::benchmark_data_smithi171_78918_object120428:head# 2024-04-20T17:11:22.175 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.176 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ffcd54c:::benchmark_data_smithi171_78918_object107688:head# 2024-04-20T17:11:22.176 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.176 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ffe1948:::benchmark_data_smithi171_78918_object131509:head# 2024-04-20T17:11:22.176 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.176 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7ffe5828:::benchmark_data_smithi171_78918_object101417:head# 2024-04-20T17:11:22.176 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.176 INFO:teuthology.orchestra.run.smithi005.stdout:Write #2:7fffea73:::benchmark_data_smithi171_78918_object158314:head# 2024-04-20T17:11:22.176 INFO:teuthology.orchestra.run.smithi005.stdout:snapset 0=[]:{} 2024-04-20T17:11:22.176 INFO:teuthology.orchestra.run.smithi005.stdout:write_pg epoch 169 info 2.e( v 169'13122 (124'3100,169'13122] local-lis/les=115/116 n=6874 ec=115/115 lis/c=115/115 les/c/f=116/116/0 sis=115) 2024-04-20T17:11:22.177 INFO:teuthology.orchestra.run.smithi005.stdout:Import successful 2024-04-20T17:11:22.177 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:21 smithi171 ceph-mon[44189]: pgmap v525: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19807/218982 objects degraded (9.045%); 6065/218982 objects misplaced (2.770%); 742 KiB/s, 11 objects/s recovering 2024-04-20T17:11:22.177 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:21 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:11:22.177 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:21 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:11:22.177 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:21 smithi171 ceph-mon[44189]: osdmap e179: 12 total, 11 up, 12 in 2024-04-20T17:11:22.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:21 smithi005 ceph-mon[39562]: pgmap v525: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19807/218982 objects degraded (9.045%); 6065/218982 objects misplaced (2.770%); 742 KiB/s, 11 objects/s recovering 2024-04-20T17:11:22.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:21 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:11:22.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:21 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:11:22.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:21 smithi005 ceph-mon[39562]: osdmap e179: 12 total, 11 up, 12 in 2024-04-20T17:11:22.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:21 smithi106 ceph-mon[44423]: pgmap v525: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19807/218982 objects degraded (9.045%); 6065/218982 objects misplaced (2.770%); 742 KiB/s, 11 objects/s recovering 2024-04-20T17:11:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:21 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:11:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:21 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:11:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:21 smithi106 ceph-mon[44423]: osdmap e179: 12 total, 11 up, 12 in 2024-04-20T17:11:22.867 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:11:22.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:22.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:11:22.972 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:11:23.005 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:23.026 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:11:23.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:23.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:11:23.130 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:11:23.160 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:23.180 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:11:23.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:23.184 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:11:23.285 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:11:23.318 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:23.338 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:11:23.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:23.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:11:23.442 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:11:23.471 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:23.490 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:11:23.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:23.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:11:23.594 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:11:23.628 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:23.794 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:11:23.795 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:11:23.795 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-04-20T17:11:23.795 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-04-20T17:11:23.795 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:11:23.866 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:11:24.079 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:11:24.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:23 smithi171 ceph-mon[44189]: pgmap v527: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19807/218982 objects degraded (9.045%); 6065/218982 objects misplaced (2.770%); 831 KiB/s, 12 objects/s recovering 2024-04-20T17:11:24.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:23 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1345590651' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:11:24.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:23 smithi171 ceph-mon[44189]: from='client.24929 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:11:24.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:24.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:11:24.103 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/exp.2.e.2 2024-04-20T17:11:24.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:23 smithi005 ceph-mon[39562]: pgmap v527: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19807/218982 objects degraded (9.045%); 6065/218982 objects misplaced (2.770%); 831 KiB/s, 12 objects/s recovering 2024-04-20T17:11:24.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:23 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1345590651' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:11:24.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:23 smithi005 ceph-mon[39562]: from='client.24929 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:11:24.185 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:11:24.214 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:24.236 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:11:24.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:24.255 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:11:24.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:23 smithi106 ceph-mon[44423]: pgmap v527: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19807/218982 objects degraded (9.045%); 6065/218982 objects misplaced (2.770%); 831 KiB/s, 12 objects/s recovering 2024-04-20T17:11:24.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:23 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1345590651' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:11:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:23 smithi106 ceph-mon[44423]: from='client.24929 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:11:24.345 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:11:24.356 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:11:24.389 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:24.411 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:11:24.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:24.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:11:24.515 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:11:24.542 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:24.594 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:11:24.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:24.597 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:11:24.622 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-20T17:11:24.699 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:11:24.731 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:24.758 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:11:24.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:24.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:11:24.862 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:11:24.891 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:24.912 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:11:24.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:24.915 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:11:24.934 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:11:25.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:24 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:11:25.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:24 smithi106 ceph-mon[44423]: from='client.24929 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:11:25.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:24 smithi106 ceph-mon[44423]: osdmap e180: 12 total, 11 up, 12 in 2024-04-20T17:11:25.015 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:11:25.045 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:25.193 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:24 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:11:25.194 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:24 smithi171 ceph-mon[44189]: from='client.24929 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:11:25.194 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:24 smithi171 ceph-mon[44189]: osdmap e180: 12 total, 11 up, 12 in 2024-04-20T17:11:25.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:24 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:11:25.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:24 smithi005 ceph-mon[39562]: from='client.24929 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:11:25.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:24 smithi005 ceph-mon[39562]: osdmap e180: 12 total, 11 up, 12 in 2024-04-20T17:11:25.398 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:11:25.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:25.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:11:25.502 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:11:25.531 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:25.729 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:11:25.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:25.732 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:11:25.833 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:11:25.868 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:25.892 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:11:25.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:25.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:11:25.996 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:11:26.027 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:26.048 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:11:26.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:26.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:11:26.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:25 smithi171 ceph-mon[44189]: pgmap v529: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19807/218982 objects degraded (9.045%); 6065/218982 objects misplaced (2.770%); 464 KiB/s, 7 objects/s recovering 2024-04-20T17:11:26.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:25 smithi171 ceph-mon[44189]: 2.9 scrub starts 2024-04-20T17:11:26.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:25 smithi171 ceph-mon[44189]: 2.6 scrub starts 2024-04-20T17:11:26.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:25 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 19807/218982 objects degraded (9.045%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:26.152 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:11:26.178 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:25 smithi106 ceph-mon[44423]: pgmap v529: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19807/218982 objects degraded (9.045%); 6065/218982 objects misplaced (2.770%); 464 KiB/s, 7 objects/s recovering 2024-04-20T17:11:26.178 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:25 smithi106 ceph-mon[44423]: 2.9 scrub starts 2024-04-20T17:11:26.178 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:25 smithi106 ceph-mon[44423]: 2.6 scrub starts 2024-04-20T17:11:26.178 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:25 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 19807/218982 objects degraded (9.045%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:26.180 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:26.199 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:11:26.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:26.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:11:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:25 smithi005 ceph-mon[39562]: pgmap v529: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19807/218982 objects degraded (9.045%); 6065/218982 objects misplaced (2.770%); 464 KiB/s, 7 objects/s recovering 2024-04-20T17:11:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:25 smithi005 ceph-mon[39562]: 2.9 scrub starts 2024-04-20T17:11:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:25 smithi005 ceph-mon[39562]: 2.6 scrub starts 2024-04-20T17:11:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:25 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 19807/218982 objects degraded (9.045%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:26.302 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:11:26.329 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:26.477 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:11:26.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:26.480 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:11:26.581 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:11:26.615 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:26.789 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:11:26.789 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:11:26.789 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:11:26.790 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:11:26.790 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.790 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:194660 2.3 e163/150 2:c260d96c:::benchmark_data_smithi171_78918_object33666:head v 163'11958, mlcod=122'2109)", 2024-04-20T17:11:26.790 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:29.397421+0000", 2024-04-20T17:11:26.790 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 57.390051845000002, 2024-04-20T17:11:26.790 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.039947886000000002, 2024-04-20T17:11:26.790 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:26.790 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:26.790 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:26.790 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:26.790 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.790 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:26.790 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.397421+0000", 2024-04-20T17:11:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.397421+0000", 2024-04-20T17:11:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.397423+0000", 2024-04-20T17:11:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7630000000000001e-06 2024-04-20T17:11:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.397432+0000", 2024-04-20T17:11:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.541e-06 2024-04-20T17:11:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.397443+0000", 2024-04-20T17:11:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1776999999999999e-05 2024-04-20T17:11:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.397450+0000", 2024-04-20T17:11:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7179999999999999e-06 2024-04-20T17:11:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.404629+0000", 2024-04-20T17:11:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.007178551 2024-04-20T17:11:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.404652+0000", 2024-04-20T17:11:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3575e-05 2024-04-20T17:11:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.437330+0000", 2024-04-20T17:11:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032677465000000003 2024-04-20T17:11:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.437369+0000", 2024-04-20T17:11:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9496000000000003e-05 2024-04-20T17:11:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:194796 2.3 e163/150 2:c824a312:::benchmark_data_smithi171_78918_object33802:head v 163'11966, mlcod=122'2112)", 2024-04-20T17:11:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:29.533354+0000", 2024-04-20T17:11:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 57.254119256999999, 2024-04-20T17:11:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0406288, 2024-04-20T17:11:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.533354+0000", 2024-04-20T17:11:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.533354+0000", 2024-04-20T17:11:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.533355+0000", 2024-04-20T17:11:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3999999999999999e-06 2024-04-20T17:11:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.533367+0000", 2024-04-20T17:11:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1691e-05 2024-04-20T17:11:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.533377+0000", 2024-04-20T17:11:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.8199999999999992e-06 2024-04-20T17:11:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.533393+0000", 2024-04-20T17:11:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6192999999999999e-05 2024-04-20T17:11:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.542134+0000", 2024-04-20T17:11:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0087409270000000008 2024-04-20T17:11:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.542162+0000", 2024-04-20T17:11:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7766000000000001e-05 2024-04-20T17:11:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.573938+0000", 2024-04-20T17:11:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.031776739999999998 2024-04-20T17:11:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.573983+0000", 2024-04-20T17:11:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4263e-05 2024-04-20T17:11:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:194810 2.3 e163/150 2:c314b47a:::benchmark_data_smithi171_78918_object33816:head v 163'11968, mlcod=122'2112)", 2024-04-20T17:11:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:29.541537+0000", 2024-04-20T17:11:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 57.245935760000002, 2024-04-20T17:11:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.046523396000000002, 2024-04-20T17:11:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.541537+0000", 2024-04-20T17:11:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.541537+0000", 2024-04-20T17:11:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.541539+0000", 2024-04-20T17:11:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4300000000000001e-06 2024-04-20T17:11:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.541547+0000", 2024-04-20T17:11:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1049999999999995e-06 2024-04-20T17:11:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.541558+0000", 2024-04-20T17:11:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0915e-05 2024-04-20T17:11:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.541565+0000", 2024-04-20T17:11:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8469999999999998e-06 2024-04-20T17:11:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.557126+0000", 2024-04-20T17:11:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015561478 2024-04-20T17:11:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.557145+0000", 2024-04-20T17:11:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8454e-05 2024-04-20T17:11:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.588028+0000", 2024-04-20T17:11:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030883058000000001 2024-04-20T17:11:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.588061+0000", 2024-04-20T17:11:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3108999999999998e-05 2024-04-20T17:11:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:194816 2.3 e163/150 2:c4e181df:::benchmark_data_smithi171_78918_object33822:head v 163'11969, mlcod=122'2112)", 2024-04-20T17:11:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:29.545777+0000", 2024-04-20T17:11:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 57.241696312000002, 2024-04-20T17:11:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.042317029999999999, 2024-04-20T17:11:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.545777+0000", 2024-04-20T17:11:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.545777+0000", 2024-04-20T17:11:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.545779+0000", 2024-04-20T17:11:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.717e-06 2024-04-20T17:11:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.545789+0000", 2024-04-20T17:11:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0498e-05 2024-04-20T17:11:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.545803+0000", 2024-04-20T17:11:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3423e-05 2024-04-20T17:11:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.545810+0000", 2024-04-20T17:11:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8869999999999992e-06 2024-04-20T17:11:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.565547+0000", 2024-04-20T17:11:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019736581999999999 2024-04-20T17:11:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.565579+0000", 2024-04-20T17:11:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2079999999999998e-05 2024-04-20T17:11:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.588064+0000", 2024-04-20T17:11:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022484671000000001 2024-04-20T17:11:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.588094+0000", 2024-04-20T17:11:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0171999999999998e-05 2024-04-20T17:11:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:196209 2.3 e166/150 2:ca2be9e5:::benchmark_data_smithi171_78918_object35215:head v 166'12051, mlcod=122'2264)", 2024-04-20T17:11:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:33.094691+0000", 2024-04-20T17:11:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 53.69278199, 2024-04-20T17:11:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043283582000000001, 2024-04-20T17:11:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.094691+0000", 2024-04-20T17:11:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.094691+0000", 2024-04-20T17:11:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.094692+0000", 2024-04-20T17:11:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9599999999999999e-07 2024-04-20T17:11:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.094696+0000", 2024-04-20T17:11:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3649999999999997e-06 2024-04-20T17:11:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.094701+0000", 2024-04-20T17:11:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6419999999999998e-06 2024-04-20T17:11:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.094705+0000", 2024-04-20T17:11:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4819999999999996e-06 2024-04-20T17:11:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.103126+0000", 2024-04-20T17:11:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0084207180000000006 2024-04-20T17:11:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.103155+0000", 2024-04-20T17:11:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9300000000000001e-05 2024-04-20T17:11:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.137934+0000", 2024-04-20T17:11:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.034779139000000001 2024-04-20T17:11:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.137975+0000", 2024-04-20T17:11:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0439999999999999e-05 2024-04-20T17:11:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:196690 2.3 e166/150 2:cb6b9f79:::benchmark_data_smithi171_78918_object35696:head v 166'12075, mlcod=122'2273)", 2024-04-20T17:11:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:33.527490+0000", 2024-04-20T17:11:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 53.259982745000002, 2024-04-20T17:11:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043966988999999998, 2024-04-20T17:11:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.527490+0000", 2024-04-20T17:11:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.527490+0000", 2024-04-20T17:11:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.527491+0000", 2024-04-20T17:11:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4900000000000005e-07 2024-04-20T17:11:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.527496+0000", 2024-04-20T17:11:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1309999999999997e-06 2024-04-20T17:11:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.527503+0000", 2024-04-20T17:11:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9569999999999996e-06 2024-04-20T17:11:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.527508+0000", 2024-04-20T17:11:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.118e-06 2024-04-20T17:11:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.527920+0000", 2024-04-20T17:11:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00041207700000000001 2024-04-20T17:11:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.527944+0000", 2024-04-20T17:11:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3935e-05 2024-04-20T17:11:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.571413+0000", 2024-04-20T17:11:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043468967999999997 2024-04-20T17:11:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.571457+0000", 2024-04-20T17:11:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4054000000000002e-05 2024-04-20T17:11:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:197148 2.3 e166/150 2:cc8bb281:::benchmark_data_smithi171_78918_object36154:head v 166'12100, mlcod=122'2280)", 2024-04-20T17:11:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:33.965330+0000", 2024-04-20T17:11:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 52.822143296999997, 2024-04-20T17:11:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044780069999999998, 2024-04-20T17:11:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.965330+0000", 2024-04-20T17:11:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.965330+0000", 2024-04-20T17:11:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.965331+0000", 2024-04-20T17:11:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4640000000000001e-06 2024-04-20T17:11:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.965343+0000", 2024-04-20T17:11:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1589e-05 2024-04-20T17:11:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.965357+0000", 2024-04-20T17:11:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4555999999999999e-05 2024-04-20T17:11:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.965363+0000", 2024-04-20T17:11:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.648e-06 2024-04-20T17:11:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.965422+0000", 2024-04-20T17:11:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8631000000000002e-05 2024-04-20T17:11:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.965445+0000", 2024-04-20T17:11:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3706999999999999e-05 2024-04-20T17:11:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.010066+0000", 2024-04-20T17:11:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044620303 2024-04-20T17:11:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.010110+0000", 2024-04-20T17:11:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4172000000000003e-05 2024-04-20T17:11:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:197153 2.3 e166/150 2:cb452060:::benchmark_data_smithi171_78918_object36159:head v 166'12101, mlcod=122'2280)", 2024-04-20T17:11:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:33.968794+0000", 2024-04-20T17:11:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 52.818679504000002, 2024-04-20T17:11:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.041343306000000003, 2024-04-20T17:11:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.968794+0000", 2024-04-20T17:11:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.968794+0000", 2024-04-20T17:11:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.968800+0000", 2024-04-20T17:11:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6409999999999996e-06 2024-04-20T17:11:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.968817+0000", 2024-04-20T17:11:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6310000000000001e-05 2024-04-20T17:11:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.968833+0000", 2024-04-20T17:11:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6028999999999998e-05 2024-04-20T17:11:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.968844+0000", 2024-04-20T17:11:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1484999999999999e-05 2024-04-20T17:11:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.974323+0000", 2024-04-20T17:11:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0054788629999999996 2024-04-20T17:11:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.974345+0000", 2024-04-20T17:11:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1639000000000001e-05 2024-04-20T17:11:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.010113+0000", 2024-04-20T17:11:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.035768056999999999 2024-04-20T17:11:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.010137+0000", 2024-04-20T17:11:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4281999999999999e-05 2024-04-20T17:11:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:197280 2.3 e166/150 2:c3bfc80e:::benchmark_data_smithi171_78918_object36286:head v 166'12109, mlcod=123'2282)", 2024-04-20T17:11:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:34.080902+0000", 2024-04-20T17:11:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 52.706570693000003, 2024-04-20T17:11:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043969018999999998, 2024-04-20T17:11:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.080902+0000", 2024-04-20T17:11:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.080902+0000", 2024-04-20T17:11:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.080904+0000", 2024-04-20T17:11:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.096e-06 2024-04-20T17:11:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.080910+0000", 2024-04-20T17:11:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5239999999999997e-06 2024-04-20T17:11:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.080917+0000", 2024-04-20T17:11:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5409999999999997e-06 2024-04-20T17:11:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.080922+0000", 2024-04-20T17:11:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.989e-06 2024-04-20T17:11:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.088738+0000", 2024-04-20T17:11:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0078159309999999999 2024-04-20T17:11:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.088756+0000", 2024-04-20T17:11:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8604e-05 2024-04-20T17:11:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.124825+0000", 2024-04-20T17:11:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036068950000000002 2024-04-20T17:11:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.124872+0000", 2024-04-20T17:11:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6384000000000002e-05 2024-04-20T17:11:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:197284 2.3 e166/150 2:c6c18a95:::benchmark_data_smithi171_78918_object36290:head v 166'12110, mlcod=123'2282)", 2024-04-20T17:11:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:34.085104+0000", 2024-04-20T17:11:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 52.702368958000001, 2024-04-20T17:11:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.039799240999999999, 2024-04-20T17:11:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.085104+0000", 2024-04-20T17:11:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.085104+0000", 2024-04-20T17:11:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.085106+0000", 2024-04-20T17:11:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.657e-06 2024-04-20T17:11:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.085114+0000", 2024-04-20T17:11:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7559999999999999e-06 2024-04-20T17:11:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.085125+0000", 2024-04-20T17:11:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.119e-05 2024-04-20T17:11:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.085132+0000", 2024-04-20T17:11:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1339999999999998e-06 2024-04-20T17:11:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.098328+0000", 2024-04-20T17:11:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013196505000000001 2024-04-20T17:11:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.098352+0000", 2024-04-20T17:11:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3306e-05 2024-04-20T17:11:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.124874+0000", 2024-04-20T17:11:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026522343 2024-04-20T17:11:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.124903+0000", 2024-04-20T17:11:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9349999999999999e-05 2024-04-20T17:11:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:197285 2.3 e166/150 2:ce3b9f7d:::benchmark_data_smithi171_78918_object36291:head v 166'12111, mlcod=123'2282)", 2024-04-20T17:11:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:34.089569+0000", 2024-04-20T17:11:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 52.697904436999998, 2024-04-20T17:11:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.042953340999999999, 2024-04-20T17:11:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.089569+0000", 2024-04-20T17:11:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.089569+0000", 2024-04-20T17:11:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.089570+0000", 2024-04-20T17:11:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3740000000000001e-06 2024-04-20T17:11:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.089577+0000", 2024-04-20T17:11:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8700000000000003e-06 2024-04-20T17:11:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.089589+0000", 2024-04-20T17:11:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1571e-05 2024-04-20T17:11:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.089595+0000", 2024-04-20T17:11:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2079999999999997e-06 2024-04-20T17:11:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.107695+0000", 2024-04-20T17:11:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018100574000000001 2024-04-20T17:11:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.107723+0000", 2024-04-20T17:11:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7291999999999999e-05 2024-04-20T17:11:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.132481+0000", 2024-04-20T17:11:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024757923000000001 2024-04-20T17:11:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.132522+0000", 2024-04-20T17:11:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1529000000000002e-05 2024-04-20T17:11:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:197311 2.3 e166/150 2:c92347ee:::benchmark_data_smithi171_78918_object36317:head v 166'12113, mlcod=123'2282)", 2024-04-20T17:11:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:34.100868+0000", 2024-04-20T17:11:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 52.686605409000002, 2024-04-20T17:11:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.047080359000000002, 2024-04-20T17:11:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.100868+0000", 2024-04-20T17:11:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.100868+0000", 2024-04-20T17:11:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.100870+0000", 2024-04-20T17:11:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5359999999999999e-06 2024-04-20T17:11:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.100882+0000", 2024-04-20T17:11:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1579999999999999e-05 2024-04-20T17:11:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.100893+0000", 2024-04-20T17:11:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.118e-05 2024-04-20T17:11:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.100900+0000", 2024-04-20T17:11:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3989999999999998e-06 2024-04-20T17:11:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.124930+0000", 2024-04-20T17:11:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024029600000000002 2024-04-20T17:11:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.124952+0000", 2024-04-20T17:11:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2024000000000001e-05 2024-04-20T17:11:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.147909+0000", 2024-04-20T17:11:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022956826999999999 2024-04-20T17:11:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.147948+0000", 2024-04-20T17:11:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9212999999999999e-05 2024-04-20T17:11:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:197313 2.3 e166/150 2:ce42bf9c:::benchmark_data_smithi171_78918_object36319:head v 166'12114, mlcod=123'2282)", 2024-04-20T17:11:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:34.104580+0000", 2024-04-20T17:11:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 52.682892903000003, 2024-04-20T17:11:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043407387999999998, 2024-04-20T17:11:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.104580+0000", 2024-04-20T17:11:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.104580+0000", 2024-04-20T17:11:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.104581+0000", 2024-04-20T17:11:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.203e-06 2024-04-20T17:11:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.104588+0000", 2024-04-20T17:11:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.951e-06 2024-04-20T17:11:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.104597+0000", 2024-04-20T17:11:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6090000000000001e-06 2024-04-20T17:11:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:26.844 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.104605+0000", 2024-04-20T17:11:26.844 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9710000000000004e-06 2024-04-20T17:11:26.844 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.844 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.844 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:26.844 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.132598+0000", 2024-04-20T17:11:26.844 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.027992929999999999 2024-04-20T17:11:26.844 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.844 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.844 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:26.844 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.132623+0000", 2024-04-20T17:11:26.844 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.529e-05 2024-04-20T17:11:26.844 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.844 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.844 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:26.845 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.147957+0000", 2024-04-20T17:11:26.845 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015334129 2024-04-20T17:11:26.845 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.845 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.845 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:26.845 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.147988+0000", 2024-04-20T17:11:26.845 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0304999999999999e-05 2024-04-20T17:11:26.845 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.845 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:26.845 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.845 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.845 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.845 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:197988 2.3 e166/150 2:cafb7b2e:::benchmark_data_smithi171_78918_object36994:head v 166'12151, mlcod=123'2294)", 2024-04-20T17:11:26.846 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:34.765325+0000", 2024-04-20T17:11:26.846 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 52.022148491999999, 2024-04-20T17:11:26.846 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044294795999999997, 2024-04-20T17:11:26.846 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:26.846 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:26.846 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:26.846 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:26.846 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.847 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:26.847 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.765325+0000", 2024-04-20T17:11:26.847 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.847 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.847 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.848 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:26.848 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.765325+0000", 2024-04-20T17:11:26.848 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.848 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.848 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.848 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:26.848 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.765325+0000", 2024-04-20T17:11:26.848 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7100000000000001e-07 2024-04-20T17:11:26.848 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.848 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.848 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:26.848 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.765333+0000", 2024-04-20T17:11:26.848 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6040000000000004e-06 2024-04-20T17:11:26.848 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.848 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.849 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:26.849 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.765342+0000", 2024-04-20T17:11:26.849 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3600000000000002e-06 2024-04-20T17:11:26.849 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.849 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.849 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:26.849 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.765349+0000", 2024-04-20T17:11:26.849 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8349999999999998e-06 2024-04-20T17:11:26.849 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.849 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.849 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:26.849 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.775787+0000", 2024-04-20T17:11:26.849 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.01043725 2024-04-20T17:11:26.849 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.849 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.850 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:26.850 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.775802+0000", 2024-04-20T17:11:26.850 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5911000000000001e-05 2024-04-20T17:11:26.850 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.850 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.850 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:26.850 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.809585+0000", 2024-04-20T17:11:26.850 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.033782464999999998 2024-04-20T17:11:26.850 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.850 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.850 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:26.850 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.809619+0000", 2024-04-20T17:11:26.850 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4600000000000001e-05 2024-04-20T17:11:26.850 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.851 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:26.851 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.851 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.851 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.851 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:197989 2.3 e166/150 2:c00840d5:::benchmark_data_smithi171_78918_object36995:head v 166'12152, mlcod=123'2294)", 2024-04-20T17:11:26.851 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:34.769894+0000", 2024-04-20T17:11:26.851 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 52.017579505, 2024-04-20T17:11:26.851 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.039770080999999999, 2024-04-20T17:11:26.851 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:26.851 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:26.851 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:26.851 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:26.851 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.851 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:26.851 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.769894+0000", 2024-04-20T17:11:26.852 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.852 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.852 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.852 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:26.852 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.769894+0000", 2024-04-20T17:11:26.852 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.852 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.852 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.852 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:26.852 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.769895+0000", 2024-04-20T17:11:26.852 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7399999999999991e-07 2024-04-20T17:11:26.852 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.852 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.852 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:26.852 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.769901+0000", 2024-04-20T17:11:26.853 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3559999999999998e-06 2024-04-20T17:11:26.853 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.853 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.853 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:26.853 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.769909+0000", 2024-04-20T17:11:26.853 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6310000000000003e-06 2024-04-20T17:11:26.853 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.853 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.853 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:26.853 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.769916+0000", 2024-04-20T17:11:26.853 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5700000000000004e-06 2024-04-20T17:11:26.853 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.853 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.854 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:26.854 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.784392+0000", 2024-04-20T17:11:26.854 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014475449 2024-04-20T17:11:26.854 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.854 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.854 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:26.854 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.784419+0000", 2024-04-20T17:11:26.854 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7633000000000001e-05 2024-04-20T17:11:26.854 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.854 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.854 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:26.854 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.809623+0000", 2024-04-20T17:11:26.854 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.025203204999999999 2024-04-20T17:11:26.854 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.855 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.855 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:26.855 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.809664+0000", 2024-04-20T17:11:26.855 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1263000000000001e-05 2024-04-20T17:11:26.855 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.855 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:26.855 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.855 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.855 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.855 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:198153 2.3 e166/150 2:c93b993b:::benchmark_data_smithi171_78918_object37159:head v 166'12166, mlcod=123'2301)", 2024-04-20T17:11:26.855 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:35.037480+0000", 2024-04-20T17:11:26.855 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 51.749992792, 2024-04-20T17:11:26.855 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.041818424999999999, 2024-04-20T17:11:26.855 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:26.856 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:26.856 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:26.856 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:26.856 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.856 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:26.856 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.037480+0000", 2024-04-20T17:11:26.856 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.856 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.856 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.856 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:26.856 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.037480+0000", 2024-04-20T17:11:26.856 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.856 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.856 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.857 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:26.857 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.037482+0000", 2024-04-20T17:11:26.857 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.319e-06 2024-04-20T17:11:26.857 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.857 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.857 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:26.857 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.037488+0000", 2024-04-20T17:11:26.857 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8189999999999997e-06 2024-04-20T17:11:26.857 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.857 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.857 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:26.857 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.037496+0000", 2024-04-20T17:11:26.857 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.7329999999999992e-06 2024-04-20T17:11:26.857 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.857 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.858 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:26.858 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.037502+0000", 2024-04-20T17:11:26.858 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.818e-06 2024-04-20T17:11:26.858 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.858 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.858 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:26.858 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.048850+0000", 2024-04-20T17:11:26.858 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011348008999999999 2024-04-20T17:11:26.858 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.858 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.858 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:26.858 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.048868+0000", 2024-04-20T17:11:26.858 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8148e-05 2024-04-20T17:11:26.859 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.859 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.859 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:26.859 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.079255+0000", 2024-04-20T17:11:26.859 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030387038000000002 2024-04-20T17:11:26.859 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.859 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.859 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:26.859 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.079299+0000", 2024-04-20T17:11:26.859 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3541000000000002e-05 2024-04-20T17:11:26.859 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.859 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:26.859 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.859 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.860 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.860 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:200778 2.3 e166/150 2:ca9ecfbd:::benchmark_data_smithi171_78918_object39784:head v 166'12349, mlcod=124'2485)", 2024-04-20T17:11:26.860 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:38.271498+0000", 2024-04-20T17:11:26.860 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 48.515974909000001, 2024-04-20T17:11:26.860 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.040696903, 2024-04-20T17:11:26.860 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:26.860 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:26.860 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:26.860 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:26.860 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.860 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:26.860 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.271498+0000", 2024-04-20T17:11:26.860 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.860 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.861 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.861 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:26.861 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.271498+0000", 2024-04-20T17:11:26.861 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.861 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.861 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.861 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:26.861 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.271499+0000", 2024-04-20T17:11:26.861 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7599999999999996e-07 2024-04-20T17:11:26.861 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.861 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.861 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:26.861 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.271505+0000", 2024-04-20T17:11:26.861 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1859999999999997e-06 2024-04-20T17:11:26.862 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.862 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.862 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:26.862 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.271513+0000", 2024-04-20T17:11:26.862 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0269999999999995e-06 2024-04-20T17:11:26.862 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.862 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.862 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:26.862 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.271519+0000", 2024-04-20T17:11:26.862 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6060000000000002e-06 2024-04-20T17:11:26.862 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.862 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.863 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:26.863 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.279364+0000", 2024-04-20T17:11:26.863 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0078446799999999997 2024-04-20T17:11:26.863 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.863 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.863 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:26.863 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.279381+0000", 2024-04-20T17:11:26.863 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7142999999999998e-05 2024-04-20T17:11:26.863 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.863 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.863 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:26.863 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.312170+0000", 2024-04-20T17:11:26.863 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032789670999999999 2024-04-20T17:11:26.863 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.864 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.864 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:26.864 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.312195+0000", 2024-04-20T17:11:26.864 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4814e-05 2024-04-20T17:11:26.864 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.864 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:26.864 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.864 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.864 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.864 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:201134 2.3 e166/150 2:c429e5fa:::benchmark_data_smithi171_78918_object40140:head v 166'12367, mlcod=124'2502)", 2024-04-20T17:11:26.864 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:38.559710+0000", 2024-04-20T17:11:26.864 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 48.227763494000001, 2024-04-20T17:11:26.864 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.048219026999999998, 2024-04-20T17:11:26.864 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:26.865 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:26.865 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:26.865 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:26.865 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.865 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:26.865 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.559710+0000", 2024-04-20T17:11:26.865 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.865 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.865 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.865 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:26.865 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.559710+0000", 2024-04-20T17:11:26.865 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.865 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.865 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.866 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:26.866 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.559712+0000", 2024-04-20T17:11:26.866 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0679999999999999e-06 2024-04-20T17:11:26.866 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.866 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.866 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:26.866 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.559721+0000", 2024-04-20T17:11:26.866 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1379999999999996e-06 2024-04-20T17:11:26.866 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.866 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.866 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:26.866 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.559733+0000", 2024-04-20T17:11:26.866 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2199e-05 2024-04-20T17:11:26.867 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.867 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.867 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:26.867 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.559742+0000", 2024-04-20T17:11:26.867 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9740000000000008e-06 2024-04-20T17:11:26.867 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.867 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.867 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:26.867 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.575041+0000", 2024-04-20T17:11:26.867 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015299046 2024-04-20T17:11:26.867 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.867 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.867 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:26.867 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.575057+0000", 2024-04-20T17:11:26.867 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5900999999999999e-05 2024-04-20T17:11:26.868 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.868 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.868 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:26.868 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.607891+0000", 2024-04-20T17:11:26.868 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032834224000000002 2024-04-20T17:11:26.868 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.868 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.868 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:26.868 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.607929+0000", 2024-04-20T17:11:26.868 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7477e-05 2024-04-20T17:11:26.868 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.868 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:26.868 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.868 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.869 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.869 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:201680 2.3 e166/150 2:cf5ca5b2:::benchmark_data_smithi171_78918_object40686:head v 166'12411, mlcod=124'2544)", 2024-04-20T17:11:26.869 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:39.306758+0000", 2024-04-20T17:11:26.869 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 47.480715590999999, 2024-04-20T17:11:26.869 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.040304325000000002, 2024-04-20T17:11:26.869 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:26.869 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:26.869 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:26.869 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:26.869 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.869 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:26.869 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.306758+0000", 2024-04-20T17:11:26.869 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.869 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.869 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.869 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:26.870 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.306758+0000", 2024-04-20T17:11:26.870 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.870 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.870 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.870 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:26.870 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.306759+0000", 2024-04-20T17:11:26.870 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.141e-06 2024-04-20T17:11:26.870 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.870 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.870 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:26.870 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.306765+0000", 2024-04-20T17:11:26.870 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8889999999999999e-06 2024-04-20T17:11:26.870 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.870 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.871 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:26.871 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.306772+0000", 2024-04-20T17:11:26.871 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2599999999999999e-06 2024-04-20T17:11:26.871 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.871 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.871 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:26.871 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.306778+0000", 2024-04-20T17:11:26.871 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1890000000000004e-06 2024-04-20T17:11:26.871 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.871 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.871 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:26.871 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.306829+0000", 2024-04-20T17:11:26.871 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0553999999999997e-05 2024-04-20T17:11:26.871 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.871 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.872 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:26.872 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.306853+0000", 2024-04-20T17:11:26.872 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4615999999999999e-05 2024-04-20T17:11:26.872 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.872 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.872 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:26.872 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.347027+0000", 2024-04-20T17:11:26.872 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.040173826000000003 2024-04-20T17:11:26.872 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.872 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.872 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:26.872 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.347062+0000", 2024-04-20T17:11:26.872 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.485e-05 2024-04-20T17:11:26.872 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.872 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:26.873 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.873 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.873 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.873 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:203113 2.3 e166/150 2:c71a3208:::benchmark_data_smithi171_78918_object42119:head v 166'12491, mlcod=124'2628)", 2024-04-20T17:11:26.873 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:40.855113+0000", 2024-04-20T17:11:26.873 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 45.932360502999998, 2024-04-20T17:11:26.873 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044317189999999999, 2024-04-20T17:11:26.873 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:26.873 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:26.873 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:26.873 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:26.873 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.873 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:26.873 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.855113+0000", 2024-04-20T17:11:26.873 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.874 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.874 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.874 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:26.874 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.855113+0000", 2024-04-20T17:11:26.874 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:26.874 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.874 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.874 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:26.874 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.855114+0000", 2024-04-20T17:11:26.874 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.649e-06 2024-04-20T17:11:26.874 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.874 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.874 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:26.875 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.855121+0000", 2024-04-20T17:11:26.875 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.985e-06 2024-04-20T17:11:26.875 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.875 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.875 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:26.875 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.855131+0000", 2024-04-20T17:11:26.875 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7890000000000006e-06 2024-04-20T17:11:26.875 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.875 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.875 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:26.875 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.855139+0000", 2024-04-20T17:11:26.875 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8949999999999998e-06 2024-04-20T17:11:26.875 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.875 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.876 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:26.876 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.865644+0000", 2024-04-20T17:11:26.876 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010505393999999999 2024-04-20T17:11:26.876 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.876 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.876 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:26.876 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.865672+0000", 2024-04-20T17:11:26.876 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8022e-05 2024-04-20T17:11:26.876 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.876 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.876 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:26.876 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.899392+0000", 2024-04-20T17:11:26.876 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.033719879000000001 2024-04-20T17:11:26.876 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:26.876 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:26.877 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:26.877 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.899430+0000", 2024-04-20T17:11:26.877 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7577000000000003e-05 2024-04-20T17:11:26.877 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.877 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:26.877 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.877 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:26.877 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:26.877 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:11:26.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:26.879 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:11:26.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:11:26.964 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:11:26.980 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:11:27.013 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:27.034 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:11:27.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:27.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:11:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:26 smithi171 ceph-mon[44189]: 2.9 scrub ok 2024-04-20T17:11:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:26 smithi171 ceph-mon[44189]: 2.6 scrub ok 2024-04-20T17:11:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:26 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:11:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:26 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:11:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:26 smithi171 ceph-mon[44189]: osdmap e181: 12 total, 11 up, 12 in 2024-04-20T17:11:27.139 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:11:27.162 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:11:27.166 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:26 smithi106 ceph-mon[44423]: 2.9 scrub ok 2024-04-20T17:11:27.166 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:26 smithi106 ceph-mon[44423]: 2.6 scrub ok 2024-04-20T17:11:27.166 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:26 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:11:27.166 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:26 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:11:27.166 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:26 smithi106 ceph-mon[44423]: osdmap e181: 12 total, 11 up, 12 in 2024-04-20T17:11:27.168 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:27.185 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:11:27.188 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:11:27.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:27.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:11:27.207 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:11:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:26 smithi005 ceph-mon[39562]: 2.9 scrub ok 2024-04-20T17:11:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:26 smithi005 ceph-mon[39562]: 2.6 scrub ok 2024-04-20T17:11:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:26 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:11:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:26 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:11:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:26 smithi005 ceph-mon[39562]: osdmap e181: 12 total, 11 up, 12 in 2024-04-20T17:11:27.229 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:11:27.263 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:11:27.267 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:11:27.288 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:11:27.292 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:11:27.310 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:11:27.320 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:27.337 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:11:27.340 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:11:27.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:27.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:11:27.357 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:11:27.384 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:11:27.405 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:11:27.405 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:11:27.405 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:11:27.405 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-20T17:11:27.405 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:11:27.405 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:11:27.405 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:11:27.406 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:11:27.406 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:11:27.406 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:11:27.406 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:11:27.406 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:11:27.406 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:11:27.443 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:11:27.472 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:27.545 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:11:27.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:27.550 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:11:27.574 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-20T17:11:27.651 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:11:27.681 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:27.808 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:11:27.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:27.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:11:27.883 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:11:27.912 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:11:27.939 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:27.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:27 smithi005 ceph-mon[39562]: pgmap v531: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19751/218982 objects degraded (9.019%); 6008/218982 objects misplaced (2.744%); 608 KiB/s, 9 objects/s recovering 2024-04-20T17:11:27.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:27 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2758587323' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:11:27.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:27.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:27.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:27.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:27.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:27.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:27.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:27.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:28.148 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:27 smithi171 ceph-mon[44189]: pgmap v531: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19751/218982 objects degraded (9.019%); 6008/218982 objects misplaced (2.744%); 608 KiB/s, 9 objects/s recovering 2024-04-20T17:11:28.148 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:27 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2758587323' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:11:28.148 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:28.148 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:28.148 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:28.148 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:28.148 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:28.148 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:28.148 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:28.149 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:28.187 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:11:28.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:28.190 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:11:28.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:27 smithi106 ceph-mon[44423]: pgmap v531: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19751/218982 objects degraded (9.019%); 6008/218982 objects misplaced (2.744%); 608 KiB/s, 9 objects/s recovering 2024-04-20T17:11:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:27 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2758587323' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:11:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:28.291 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:11:28.317 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:28.335 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:11:28.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:28.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:11:28.438 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:11:28.449 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:11:28.450 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:11:28.464 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:28.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:28.486 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:11:28.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:11:28.587 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:11:28.619 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:29.144 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:29.144 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:29.144 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:29.144 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:28 smithi005 ceph-mon[39562]: 2.8 scrub starts 2024-04-20T17:11:29.238 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-20T17:11:29.239 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:11:29.239 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-20T17:11:29.239 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-20T17:11:29.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:28 smithi106 ceph-mon[44423]: 2.8 scrub starts 2024-04-20T17:11:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:28 smithi171 ceph-mon[44189]: 2.8 scrub starts 2024-04-20T17:11:29.761 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:11:29.769 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:11:29.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:29.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:11:29.876 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:11:29.938 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:29.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:29 smithi005 ceph-mon[39562]: pgmap v532: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded, 2 undersized+degraded+peered, 13 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19751/218982 objects degraded (9.019%); 6008/218982 objects misplaced (2.744%); 531 KiB/s, 8 objects/s recovering 2024-04-20T17:11:29.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:29 smithi005 ceph-mon[39562]: 2.8 scrub ok 2024-04-20T17:11:29.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:29 smithi005 ceph-mon[39562]: 2.d scrub starts 2024-04-20T17:11:30.209 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:29 smithi171 ceph-mon[44189]: pgmap v532: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded, 2 undersized+degraded+peered, 13 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19751/218982 objects degraded (9.019%); 6008/218982 objects misplaced (2.744%); 531 KiB/s, 8 objects/s recovering 2024-04-20T17:11:30.209 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:29 smithi171 ceph-mon[44189]: 2.8 scrub ok 2024-04-20T17:11:30.210 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:29 smithi171 ceph-mon[44189]: 2.d scrub starts 2024-04-20T17:11:30.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:29 smithi106 ceph-mon[44423]: pgmap v532: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded, 2 undersized+degraded+peered, 13 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19751/218982 objects degraded (9.019%); 6008/218982 objects misplaced (2.744%); 531 KiB/s, 8 objects/s recovering 2024-04-20T17:11:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:29 smithi106 ceph-mon[44423]: 2.8 scrub ok 2024-04-20T17:11:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:29 smithi106 ceph-mon[44423]: 2.d scrub starts 2024-04-20T17:11:30.286 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:11:30.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:30.290 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:11:30.391 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:11:30.418 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:30.451 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:11:30.489 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:11:30.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:30.493 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:11:30.594 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:11:30.627 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:30.885 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:11:30.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:30.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:11:30.940 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:11:30.940 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:11:30.989 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:11:31.022 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:31.043 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:11:31.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:31.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:11:31.147 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:11:31.173 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:30 smithi005 ceph-mon[39562]: 2.d scrub ok 2024-04-20T17:11:31.173 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:30 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3269674038' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:11:31.173 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:30 smithi005 ceph-mon[39562]: from='client.24941 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:11:31.174 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:30 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 19751/218982 objects degraded (9.019%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:31.174 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:30 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:11:31.174 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:30 smithi005 ceph-mon[39562]: from='client.24941 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:11:31.174 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:30 smithi005 ceph-mon[39562]: osdmap e182: 12 total, 11 up, 12 in 2024-04-20T17:11:31.176 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:31.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:30 smithi106 ceph-mon[44423]: 2.d scrub ok 2024-04-20T17:11:31.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:30 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3269674038' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:11:31.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:30 smithi106 ceph-mon[44423]: from='client.24941 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:11:31.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:30 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 19751/218982 objects degraded (9.019%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:31.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:30 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:11:31.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:30 smithi106 ceph-mon[44423]: from='client.24941 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:11:31.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:30 smithi106 ceph-mon[44423]: osdmap e182: 12 total, 11 up, 12 in 2024-04-20T17:11:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:30 smithi171 ceph-mon[44189]: 2.d scrub ok 2024-04-20T17:11:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:30 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3269674038' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:11:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:30 smithi171 ceph-mon[44189]: from='client.24941 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:11:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:30 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 19751/218982 objects degraded (9.019%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:30 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:11:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:30 smithi171 ceph-mon[44189]: from='client.24941 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:11:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:30 smithi171 ceph-mon[44189]: osdmap e182: 12 total, 11 up, 12 in 2024-04-20T17:11:31.429 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:11:31.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:31.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:11:31.535 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:11:31.566 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:11:31.569 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:31.590 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:11:31.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:31.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:11:31.666 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-20T17:11:31.693 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:11:31.725 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:31.747 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:11:31.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:31.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:11:31.851 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:11:31.881 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:31.902 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:11:31.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:31.905 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:11:31.933 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:11:32.006 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:11:32.037 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:32.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:31 smithi005 ceph-mon[39562]: pgmap v533: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded, 2 undersized+degraded+peered, 13 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19691/218982 objects degraded (8.992%); 5948/218982 objects misplaced (2.716%); 936 KiB/s, 14 objects/s recovering 2024-04-20T17:11:32.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:31 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:11:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:31 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:11:32.214 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:31 smithi171 ceph-mon[44189]: pgmap v533: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded, 2 undersized+degraded+peered, 13 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19691/218982 objects degraded (8.992%); 5948/218982 objects misplaced (2.716%); 936 KiB/s, 14 objects/s recovering 2024-04-20T17:11:32.214 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:31 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:11:32.214 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:31 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:11:32.242 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:11:32.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:32.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:11:32.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:31 smithi106 ceph-mon[44423]: pgmap v533: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded, 2 undersized+degraded+peered, 13 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19691/218982 objects degraded (8.992%); 5948/218982 objects misplaced (2.716%); 936 KiB/s, 14 objects/s recovering 2024-04-20T17:11:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:31 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:11:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:31 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:11:32.332 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:11:32.352 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:11:32.384 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:32.406 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:11:32.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:32.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:11:32.510 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:11:32.543 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:32.564 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:11:32.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:32.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:11:32.668 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:11:32.697 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:32.717 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:11:32.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:32.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:11:32.821 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:11:32.850 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:32.869 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:11:32.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:32.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:11:32.972 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:11:33.002 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:33.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:32 smithi106 ceph-mon[44423]: pgmap v535: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded, 2 undersized+degraded+peered, 13 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19691/218982 objects degraded (8.992%); 5948/218982 objects misplaced (2.716%); 936 KiB/s, 14 objects/s recovering 2024-04-20T17:11:33.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:32 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:11:33.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:32 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:11:33.168 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:32 smithi005 ceph-mon[39562]: pgmap v535: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded, 2 undersized+degraded+peered, 13 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19691/218982 objects degraded (8.992%); 5948/218982 objects misplaced (2.716%); 936 KiB/s, 14 objects/s recovering 2024-04-20T17:11:33.168 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:32 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:11:33.168 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:32 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:11:33.279 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:11:33.279 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:11:33.279 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 183, 2024-04-20T17:11:33.279 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "13c6e76c-ff37-11ee-bc93-c7b262605968", 2024-04-20T17:11:33.279 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-04-20T16:58:25.272557+0000", 2024-04-20T17:11:33.279 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-04-20T17:11:32.896939+0000", 2024-04-20T17:11:33.279 INFO:teuthology.orchestra.run.smithi005.stdout: "last_up_change": "2024-04-20T17:10:50.899850+0000", 2024-04-20T17:11:33.279 INFO:teuthology.orchestra.run.smithi005.stdout: "last_in_change": "2024-04-20T17:05:42.849446+0000", 2024-04-20T17:11:33.279 INFO:teuthology.orchestra.run.smithi005.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-20T17:11:33.279 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_num": 5799936, 2024-04-20T17:11:33.280 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_set": [ 2024-04-20T17:11:33.280 INFO:teuthology.orchestra.run.smithi005.stdout: "pglog_hardlimit", 2024-04-20T17:11:33.280 INFO:teuthology.orchestra.run.smithi005.stdout: "purged_snapdirs", 2024-04-20T17:11:33.280 INFO:teuthology.orchestra.run.smithi005.stdout: "recovery_deletes", 2024-04-20T17:11:33.280 INFO:teuthology.orchestra.run.smithi005.stdout: "sortbitwise" 2024-04-20T17:11:33.280 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:11:33.280 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_version": 27, 2024-04-20T17:11:33.280 INFO:teuthology.orchestra.run.smithi005.stdout: "full_ratio": 0.94999998807907104, 2024-04-20T17:11:33.280 INFO:teuthology.orchestra.run.smithi005.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-20T17:11:33.280 INFO:teuthology.orchestra.run.smithi005.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-20T17:11:33.280 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_snapshot": "", 2024-04-20T17:11:33.280 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_max": 2, 2024-04-20T17:11:33.280 INFO:teuthology.orchestra.run.smithi005.stdout: "max_osd": 12, 2024-04-20T17:11:33.280 INFO:teuthology.orchestra.run.smithi005.stdout: "require_min_compat_client": "luminous", 2024-04-20T17:11:33.281 INFO:teuthology.orchestra.run.smithi005.stdout: "min_compat_client": "luminous", 2024-04-20T17:11:33.281 INFO:teuthology.orchestra.run.smithi005.stdout: "require_osd_release": "squid", 2024-04-20T17:11:33.281 INFO:teuthology.orchestra.run.smithi005.stdout: "allow_crimson": false, 2024-04-20T17:11:33.281 INFO:teuthology.orchestra.run.smithi005.stdout: "pools": [ 2024-04-20T17:11:33.281 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.281 INFO:teuthology.orchestra.run.smithi005.stdout: "pool": 1, 2024-04-20T17:11:33.281 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_name": ".mgr", 2024-04-20T17:11:33.281 INFO:teuthology.orchestra.run.smithi005.stdout: "create_time": "2024-04-20T17:01:35.909493+0000", 2024-04-20T17:11:33.281 INFO:teuthology.orchestra.run.smithi005.stdout: "flags": 1, 2024-04-20T17:11:33.281 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_names": "hashpspool", 2024-04-20T17:11:33.281 INFO:teuthology.orchestra.run.smithi005.stdout: "type": 1, 2024-04-20T17:11:33.281 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 2, 2024-04-20T17:11:33.281 INFO:teuthology.orchestra.run.smithi005.stdout: "min_size": 2, 2024-04-20T17:11:33.282 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_rule": 0, 2024-04-20T17:11:33.282 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_count": 0, 2024-04-20T17:11:33.282 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_target": 0, 2024-04-20T17:11:33.282 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T17:11:33.282 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T17:11:33.282 INFO:teuthology.orchestra.run.smithi005.stdout: "object_hash": 2, 2024-04-20T17:11:33.282 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_autoscale_mode": "off", 2024-04-20T17:11:33.282 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num": 1, 2024-04-20T17:11:33.282 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num": 1, 2024-04-20T17:11:33.282 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num_target": 1, 2024-04-20T17:11:33.282 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_target": 1, 2024-04-20T17:11:33.282 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_pending": 1, 2024-04-20T17:11:33.282 INFO:teuthology.orchestra.run.smithi005.stdout: "last_pg_merge_meta": { 2024-04-20T17:11:33.282 INFO:teuthology.orchestra.run.smithi005.stdout: "source_pgid": "0.0", 2024-04-20T17:11:33.283 INFO:teuthology.orchestra.run.smithi005.stdout: "ready_epoch": 0, 2024-04-20T17:11:33.283 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_started": 0, 2024-04-20T17:11:33.283 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_clean": 0, 2024-04-20T17:11:33.283 INFO:teuthology.orchestra.run.smithi005.stdout: "source_version": "0'0", 2024-04-20T17:11:33.283 INFO:teuthology.orchestra.run.smithi005.stdout: "target_version": "0'0" 2024-04-20T17:11:33.283 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.283 INFO:teuthology.orchestra.run.smithi005.stdout: "last_change": "16", 2024-04-20T17:11:33.283 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend": "0", 2024-04-20T17:11:33.283 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-20T17:11:33.283 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_preluminous": "0", 2024-04-20T17:11:33.283 INFO:teuthology.orchestra.run.smithi005.stdout: "auid": 0, 2024-04-20T17:11:33.283 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_mode": "selfmanaged", 2024-04-20T17:11:33.283 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_seq": 0, 2024-04-20T17:11:33.283 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_epoch": 0, 2024-04-20T17:11:33.283 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_snaps": [], 2024-04-20T17:11:33.284 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_snaps": "[]", 2024-04-20T17:11:33.284 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_bytes": 0, 2024-04-20T17:11:33.284 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_objects": 0, 2024-04-20T17:11:33.284 INFO:teuthology.orchestra.run.smithi005.stdout: "tiers": [], 2024-04-20T17:11:33.284 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_of": -1, 2024-04-20T17:11:33.284 INFO:teuthology.orchestra.run.smithi005.stdout: "read_tier": -1, 2024-04-20T17:11:33.284 INFO:teuthology.orchestra.run.smithi005.stdout: "write_tier": -1, 2024-04-20T17:11:33.284 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_mode": "none", 2024-04-20T17:11:33.284 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_bytes": 0, 2024-04-20T17:11:33.284 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_objects": 0, 2024-04-20T17:11:33.284 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T17:11:33.284 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T17:11:33.284 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T17:11:33.285 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_flush_age": 0, 2024-04-20T17:11:33.285 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_evict_age": 0, 2024-04-20T17:11:33.285 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_profile": "", 2024-04-20T17:11:33.285 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_params": { 2024-04-20T17:11:33.285 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "none" 2024-04-20T17:11:33.285 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.285 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_period": 0, 2024-04-20T17:11:33.285 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_count": 0, 2024-04-20T17:11:33.285 INFO:teuthology.orchestra.run.smithi005.stdout: "use_gmt_hitset": true, 2024-04-20T17:11:33.285 INFO:teuthology.orchestra.run.smithi005.stdout: "min_read_recency_for_promote": 0, 2024-04-20T17:11:33.285 INFO:teuthology.orchestra.run.smithi005.stdout: "min_write_recency_for_promote": 0, 2024-04-20T17:11:33.285 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T17:11:33.285 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_search_last_n": 0, 2024-04-20T17:11:33.285 INFO:teuthology.orchestra.run.smithi005.stdout: "grade_table": [], 2024-04-20T17:11:33.285 INFO:teuthology.orchestra.run.smithi005.stdout: "stripe_width": 0, 2024-04-20T17:11:33.286 INFO:teuthology.orchestra.run.smithi005.stdout: "expected_num_objects": 0, 2024-04-20T17:11:33.286 INFO:teuthology.orchestra.run.smithi005.stdout: "fast_read": false, 2024-04-20T17:11:33.286 INFO:teuthology.orchestra.run.smithi005.stdout: "options": { 2024-04-20T17:11:33.286 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_max": 32, 2024-04-20T17:11:33.286 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_min": 1 2024-04-20T17:11:33.286 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.286 INFO:teuthology.orchestra.run.smithi005.stdout: "application_metadata": { 2024-04-20T17:11:33.286 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": {} 2024-04-20T17:11:33.286 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.286 INFO:teuthology.orchestra.run.smithi005.stdout: "read_balance": { 2024-04-20T17:11:33.286 INFO:teuthology.orchestra.run.smithi005.stdout: "score_type": "Fair distribution", 2024-04-20T17:11:33.286 INFO:teuthology.orchestra.run.smithi005.stdout: "score_acting": 11.760000228881836, 2024-04-20T17:11:33.286 INFO:teuthology.orchestra.run.smithi005.stdout: "score_stable": 11.760000228881836, 2024-04-20T17:11:33.287 INFO:teuthology.orchestra.run.smithi005.stdout: "optimal_score": 0.17000000178813934, 2024-04-20T17:11:33.287 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_acting": 2, 2024-04-20T17:11:33.287 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_stable": 2, 2024-04-20T17:11:33.287 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-04-20T17:11:33.287 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity": 1, 2024-04-20T17:11:33.287 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity_weighted": 1 2024-04-20T17:11:33.287 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.287 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.287 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.287 INFO:teuthology.orchestra.run.smithi005.stdout: "pool": 2, 2024-04-20T17:11:33.287 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_name": "unique_pool_0", 2024-04-20T17:11:33.287 INFO:teuthology.orchestra.run.smithi005.stdout: "create_time": "2024-04-20T17:08:04.904546+0000", 2024-04-20T17:11:33.287 INFO:teuthology.orchestra.run.smithi005.stdout: "flags": 1, 2024-04-20T17:11:33.287 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_names": "hashpspool", 2024-04-20T17:11:33.288 INFO:teuthology.orchestra.run.smithi005.stdout: "type": 1, 2024-04-20T17:11:33.288 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 2, 2024-04-20T17:11:33.288 INFO:teuthology.orchestra.run.smithi005.stdout: "min_size": 2, 2024-04-20T17:11:33.288 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_rule": 0, 2024-04-20T17:11:33.288 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_count": 0, 2024-04-20T17:11:33.288 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_target": 0, 2024-04-20T17:11:33.288 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T17:11:33.288 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T17:11:33.288 INFO:teuthology.orchestra.run.smithi005.stdout: "object_hash": 2, 2024-04-20T17:11:33.288 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_autoscale_mode": "off", 2024-04-20T17:11:33.288 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num": 16, 2024-04-20T17:11:33.288 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num": 16, 2024-04-20T17:11:33.288 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num_target": 16, 2024-04-20T17:11:33.288 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_target": 16, 2024-04-20T17:11:33.288 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_pending": 16, 2024-04-20T17:11:33.289 INFO:teuthology.orchestra.run.smithi005.stdout: "last_pg_merge_meta": { 2024-04-20T17:11:33.289 INFO:teuthology.orchestra.run.smithi005.stdout: "source_pgid": "0.0", 2024-04-20T17:11:33.289 INFO:teuthology.orchestra.run.smithi005.stdout: "ready_epoch": 0, 2024-04-20T17:11:33.289 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_started": 0, 2024-04-20T17:11:33.289 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_clean": 0, 2024-04-20T17:11:33.289 INFO:teuthology.orchestra.run.smithi005.stdout: "source_version": "0'0", 2024-04-20T17:11:33.289 INFO:teuthology.orchestra.run.smithi005.stdout: "target_version": "0'0" 2024-04-20T17:11:33.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.289 INFO:teuthology.orchestra.run.smithi005.stdout: "last_change": "118", 2024-04-20T17:11:33.289 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend": "0", 2024-04-20T17:11:33.289 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-20T17:11:33.289 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_preluminous": "0", 2024-04-20T17:11:33.289 INFO:teuthology.orchestra.run.smithi005.stdout: "auid": 0, 2024-04-20T17:11:33.290 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_mode": "selfmanaged", 2024-04-20T17:11:33.290 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_seq": 0, 2024-04-20T17:11:33.290 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_epoch": 0, 2024-04-20T17:11:33.290 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_snaps": [], 2024-04-20T17:11:33.290 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_snaps": "[]", 2024-04-20T17:11:33.290 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_bytes": 0, 2024-04-20T17:11:33.290 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_objects": 0, 2024-04-20T17:11:33.290 INFO:teuthology.orchestra.run.smithi005.stdout: "tiers": [], 2024-04-20T17:11:33.290 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_of": -1, 2024-04-20T17:11:33.290 INFO:teuthology.orchestra.run.smithi005.stdout: "read_tier": -1, 2024-04-20T17:11:33.290 INFO:teuthology.orchestra.run.smithi005.stdout: "write_tier": -1, 2024-04-20T17:11:33.290 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_mode": "none", 2024-04-20T17:11:33.290 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_bytes": 0, 2024-04-20T17:11:33.290 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_objects": 0, 2024-04-20T17:11:33.290 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T17:11:33.291 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T17:11:33.291 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T17:11:33.291 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_flush_age": 0, 2024-04-20T17:11:33.291 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_evict_age": 0, 2024-04-20T17:11:33.291 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_profile": "", 2024-04-20T17:11:33.291 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_params": { 2024-04-20T17:11:33.291 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "none" 2024-04-20T17:11:33.291 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.291 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_period": 0, 2024-04-20T17:11:33.291 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_count": 0, 2024-04-20T17:11:33.291 INFO:teuthology.orchestra.run.smithi005.stdout: "use_gmt_hitset": true, 2024-04-20T17:11:33.291 INFO:teuthology.orchestra.run.smithi005.stdout: "min_read_recency_for_promote": 0, 2024-04-20T17:11:33.291 INFO:teuthology.orchestra.run.smithi005.stdout: "min_write_recency_for_promote": 0, 2024-04-20T17:11:33.291 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T17:11:33.292 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_search_last_n": 0, 2024-04-20T17:11:33.292 INFO:teuthology.orchestra.run.smithi005.stdout: "grade_table": [], 2024-04-20T17:11:33.292 INFO:teuthology.orchestra.run.smithi005.stdout: "stripe_width": 0, 2024-04-20T17:11:33.292 INFO:teuthology.orchestra.run.smithi005.stdout: "expected_num_objects": 0, 2024-04-20T17:11:33.292 INFO:teuthology.orchestra.run.smithi005.stdout: "fast_read": false, 2024-04-20T17:11:33.292 INFO:teuthology.orchestra.run.smithi005.stdout: "options": {}, 2024-04-20T17:11:33.292 INFO:teuthology.orchestra.run.smithi005.stdout: "application_metadata": { 2024-04-20T17:11:33.292 INFO:teuthology.orchestra.run.smithi005.stdout: "rados": {} 2024-04-20T17:11:33.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.292 INFO:teuthology.orchestra.run.smithi005.stdout: "read_balance": { 2024-04-20T17:11:33.292 INFO:teuthology.orchestra.run.smithi005.stdout: "score_type": "Fair distribution", 2024-04-20T17:11:33.292 INFO:teuthology.orchestra.run.smithi005.stdout: "score_acting": 2.2400000095367432, 2024-04-20T17:11:33.292 INFO:teuthology.orchestra.run.smithi005.stdout: "score_stable": 2.2400000095367432, 2024-04-20T17:11:33.292 INFO:teuthology.orchestra.run.smithi005.stdout: "optimal_score": 0.92000001668930054, 2024-04-20T17:11:33.293 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_acting": 2.059999942779541, 2024-04-20T17:11:33.293 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_stable": 2.059999942779541, 2024-04-20T17:11:33.293 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-04-20T17:11:33.293 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity": 1, 2024-04-20T17:11:33.293 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity_weighted": 1 2024-04-20T17:11:33.293 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.293 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.293 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:11:33.293 INFO:teuthology.orchestra.run.smithi005.stdout: "osds": [ 2024-04-20T17:11:33.293 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.293 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 0, 2024-04-20T17:11:33.293 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b", 2024-04-20T17:11:33.293 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:11:33.293 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:11:33.293 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:11:33.294 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:11:33.294 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:11:33.294 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:11:33.294 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 8, 2024-04-20T17:11:33.294 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 171, 2024-04-20T17:11:33.294 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:11:33.294 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:11:33.294 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:11:33.294 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.294 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.294 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.294 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6801", 2024-04-20T17:11:33.294 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:11:33.294 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.295 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.295 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.295 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:11:33.295 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.295 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.295 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6802", 2024-04-20T17:11:33.295 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:11:33.295 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.295 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.295 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.295 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:11:33.295 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.296 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.296 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6804", 2024-04-20T17:11:33.296 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:11:33.296 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.296 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.296 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.296 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:11:33.296 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.296 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.296 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6803", 2024-04-20T17:11:33.296 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:11:33.296 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.296 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.297 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6801/3901484705", 2024-04-20T17:11:33.297 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6802/3901484705", 2024-04-20T17:11:33.297 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6804/3901484705", 2024-04-20T17:11:33.297 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6803/3901484705", 2024-04-20T17:11:33.297 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:11:33.297 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:11:33.297 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:11:33.297 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.297 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.297 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 1, 2024-04-20T17:11:33.297 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "aed0c8eb-0e81-4e85-ab8f-a60420060656", 2024-04-20T17:11:33.297 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:11:33.298 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:11:33.298 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:11:33.298 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:11:33.298 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:11:33.298 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:11:33.298 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 13, 2024-04-20T17:11:33.298 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 115, 2024-04-20T17:11:33.298 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:11:33.298 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:11:33.298 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:11:33.298 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.298 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.298 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.298 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6805", 2024-04-20T17:11:33.299 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3391893914 2024-04-20T17:11:33.299 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.299 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.299 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.299 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:11:33.299 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.299 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.299 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.299 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6806", 2024-04-20T17:11:33.299 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3391893914 2024-04-20T17:11:33.299 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.299 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.299 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.299 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:11:33.300 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.300 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.300 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6808", 2024-04-20T17:11:33.300 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3391893914 2024-04-20T17:11:33.300 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.300 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.300 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.300 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:11:33.300 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.300 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.300 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6807", 2024-04-20T17:11:33.300 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3391893914 2024-04-20T17:11:33.301 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.301 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.301 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.301 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6805/3391893914", 2024-04-20T17:11:33.301 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6806/3391893914", 2024-04-20T17:11:33.301 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6808/3391893914", 2024-04-20T17:11:33.301 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6807/3391893914", 2024-04-20T17:11:33.301 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:11:33.301 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:11:33.301 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:11:33.301 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.301 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.301 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.301 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 2, 2024-04-20T17:11:33.301 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "f733a6a2-19c2-4c7a-af9c-04e282062bb7", 2024-04-20T17:11:33.302 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 0, 2024-04-20T17:11:33.302 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:11:33.302 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:11:33.302 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:11:33.302 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:11:33.302 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:11:33.302 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 20, 2024-04-20T17:11:33.302 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 150, 2024-04-20T17:11:33.302 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 170, 2024-04-20T17:11:33.302 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:11:33.302 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:11:33.302 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.302 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.303 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.303 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6809", 2024-04-20T17:11:33.303 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2105468358 2024-04-20T17:11:33.303 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.303 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.303 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.303 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:11:33.303 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.303 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.303 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.303 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6810", 2024-04-20T17:11:33.303 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2105468358 2024-04-20T17:11:33.303 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.303 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.304 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.304 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:11:33.304 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.304 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.304 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.304 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6812", 2024-04-20T17:11:33.304 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2105468358 2024-04-20T17:11:33.304 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.304 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.304 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.305 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:11:33.305 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.305 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.305 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.305 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6811", 2024-04-20T17:11:33.305 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2105468358 2024-04-20T17:11:33.305 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.305 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.305 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6809/2105468358", 2024-04-20T17:11:33.305 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6810/2105468358", 2024-04-20T17:11:33.305 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6812/2105468358", 2024-04-20T17:11:33.305 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6811/2105468358", 2024-04-20T17:11:33.305 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:11:33.306 INFO:teuthology.orchestra.run.smithi005.stdout: "exists" 2024-04-20T17:11:33.306 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.306 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.306 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 3, 2024-04-20T17:11:33.306 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "8c92f1df-e298-4606-88ae-b8a8e79628a1", 2024-04-20T17:11:33.306 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:11:33.306 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:11:33.306 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:11:33.306 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:11:33.306 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:11:33.306 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:11:33.307 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 25, 2024-04-20T17:11:33.307 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 170, 2024-04-20T17:11:33.307 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:11:33.307 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:11:33.307 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:11:33.307 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.307 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.307 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.307 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6813", 2024-04-20T17:11:33.307 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 377652039 2024-04-20T17:11:33.307 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.307 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.307 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.307 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:11:33.307 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.308 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.308 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.308 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6814", 2024-04-20T17:11:33.308 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 377652039 2024-04-20T17:11:33.308 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.308 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.308 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.308 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:11:33.308 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.308 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.308 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.308 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6816", 2024-04-20T17:11:33.308 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 377652039 2024-04-20T17:11:33.308 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.309 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.309 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.309 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:11:33.309 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.309 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.309 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.309 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6815", 2024-04-20T17:11:33.309 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 377652039 2024-04-20T17:11:33.309 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.309 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.309 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.309 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6813/377652039", 2024-04-20T17:11:33.309 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6814/377652039", 2024-04-20T17:11:33.310 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6816/377652039", 2024-04-20T17:11:33.310 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6815/377652039", 2024-04-20T17:11:33.310 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:11:33.310 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:11:33.310 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:11:33.310 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.310 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.310 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.310 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 4, 2024-04-20T17:11:33.310 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "46df16f1-9607-4d7b-9be8-9d4236e160d1", 2024-04-20T17:11:33.310 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:11:33.310 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:11:33.310 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:11:33.310 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:11:33.311 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:11:33.311 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:11:33.311 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 31, 2024-04-20T17:11:33.311 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 0, 2024-04-20T17:11:33.311 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:11:33.311 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:11:33.311 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:11:33.314 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.314 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.314 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.314 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6800", 2024-04-20T17:11:33.314 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:11:33.314 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.314 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.314 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.314 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:11:33.314 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.314 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.314 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.314 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6801", 2024-04-20T17:11:33.315 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:11:33.315 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.315 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.315 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.315 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:11:33.315 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.315 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.315 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.315 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6803", 2024-04-20T17:11:33.315 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:11:33.315 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.315 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.315 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.315 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:11:33.315 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.316 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.316 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.316 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6802", 2024-04-20T17:11:33.316 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:11:33.316 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.316 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.316 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.316 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6800/1839524469", 2024-04-20T17:11:33.316 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6801/1839524469", 2024-04-20T17:11:33.316 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6803/1839524469", 2024-04-20T17:11:33.316 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6802/1839524469", 2024-04-20T17:11:33.316 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:11:33.316 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:11:33.316 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:11:33.317 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.317 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.317 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.317 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 5, 2024-04-20T17:11:33.317 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "6603231d-1e5c-4575-bfaa-2eeca2c12215", 2024-04-20T17:11:33.317 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:11:33.317 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:11:33.317 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:11:33.318 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:11:33.318 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:11:33.318 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:11:33.318 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 37, 2024-04-20T17:11:33.318 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 162, 2024-04-20T17:11:33.318 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:11:33.318 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:11:33.318 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:11:33.318 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.318 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.318 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.319 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6804", 2024-04-20T17:11:33.319 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:11:33.319 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.319 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.319 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.319 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:11:33.319 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.319 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.319 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.319 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6805", 2024-04-20T17:11:33.319 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:11:33.319 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.319 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.319 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.319 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:11:33.320 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.320 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.320 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.320 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6807", 2024-04-20T17:11:33.320 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:11:33.320 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.320 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.320 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.320 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:11:33.320 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.320 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.320 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.320 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6806", 2024-04-20T17:11:33.321 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:11:33.321 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.321 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.321 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.321 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6804/2450315908", 2024-04-20T17:11:33.321 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6805/2450315908", 2024-04-20T17:11:33.321 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6807/2450315908", 2024-04-20T17:11:33.321 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6806/2450315908", 2024-04-20T17:11:33.321 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:11:33.321 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:11:33.321 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:11:33.321 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.321 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.321 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.322 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 6, 2024-04-20T17:11:33.322 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "783a55a6-f176-42be-8c8b-af8481d08dc9", 2024-04-20T17:11:33.322 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:11:33.322 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:11:33.322 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:11:33.322 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:11:33.322 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:11:33.322 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:11:33.322 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 48, 2024-04-20T17:11:33.322 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 115, 2024-04-20T17:11:33.322 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:11:33.322 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:11:33.322 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:11:33.322 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.323 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.323 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.323 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6808", 2024-04-20T17:11:33.323 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2840237913 2024-04-20T17:11:33.323 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.323 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.323 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.323 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:11:33.323 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.323 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.323 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.323 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6809", 2024-04-20T17:11:33.323 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2840237913 2024-04-20T17:11:33.323 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.324 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.324 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.324 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:11:33.324 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.324 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.324 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.324 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6811", 2024-04-20T17:11:33.324 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2840237913 2024-04-20T17:11:33.324 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.324 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.324 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.324 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:11:33.324 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.324 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.325 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.325 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6810", 2024-04-20T17:11:33.325 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2840237913 2024-04-20T17:11:33.325 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.325 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.325 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.325 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6808/2840237913", 2024-04-20T17:11:33.325 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6809/2840237913", 2024-04-20T17:11:33.325 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6811/2840237913", 2024-04-20T17:11:33.325 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6810/2840237913", 2024-04-20T17:11:33.325 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:11:33.325 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:11:33.325 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:11:33.325 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.326 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.326 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.326 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 7, 2024-04-20T17:11:33.326 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4", 2024-04-20T17:11:33.326 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:11:33.326 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:11:33.326 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:11:33.326 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:11:33.326 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:11:33.326 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:11:33.326 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 56, 2024-04-20T17:11:33.326 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 173, 2024-04-20T17:11:33.326 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:11:33.327 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:11:33.327 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:11:33.327 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.327 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.327 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.327 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6812", 2024-04-20T17:11:33.327 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1105371883 2024-04-20T17:11:33.327 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.327 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.327 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.327 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:11:33.327 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.327 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.328 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.328 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6813", 2024-04-20T17:11:33.328 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1105371883 2024-04-20T17:11:33.328 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.328 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.328 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.328 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:11:33.328 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.328 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.328 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.328 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6815", 2024-04-20T17:11:33.328 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1105371883 2024-04-20T17:11:33.328 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.328 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.329 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.329 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:11:33.329 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.329 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.329 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.329 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6814", 2024-04-20T17:11:33.329 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1105371883 2024-04-20T17:11:33.329 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.329 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.329 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.329 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6812/1105371883", 2024-04-20T17:11:33.329 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6813/1105371883", 2024-04-20T17:11:33.329 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6815/1105371883", 2024-04-20T17:11:33.329 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6814/1105371883", 2024-04-20T17:11:33.330 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:11:33.330 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:11:33.330 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:11:33.330 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.330 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.330 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.330 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 8, 2024-04-20T17:11:33.330 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "1cb232ad-1f59-4a0a-a215-e67419ad6476", 2024-04-20T17:11:33.330 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:11:33.330 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:11:33.330 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:11:33.330 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:11:33.330 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:11:33.330 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:11:33.331 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 67, 2024-04-20T17:11:33.331 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 115, 2024-04-20T17:11:33.331 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:11:33.331 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:11:33.331 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:11:33.331 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.331 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.331 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.331 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6800", 2024-04-20T17:11:33.331 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:11:33.331 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.331 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.331 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.331 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:11:33.332 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.332 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.332 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.332 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6801", 2024-04-20T17:11:33.332 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:11:33.332 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.332 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.332 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.332 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:11:33.332 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.332 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.332 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.332 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6803", 2024-04-20T17:11:33.332 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:11:33.332 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.333 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.333 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.333 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:11:33.333 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.333 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.333 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.333 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6802", 2024-04-20T17:11:33.333 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:11:33.333 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.333 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.333 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.333 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6800/4123058942", 2024-04-20T17:11:33.333 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6801/4123058942", 2024-04-20T17:11:33.333 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6803/4123058942", 2024-04-20T17:11:33.333 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6802/4123058942", 2024-04-20T17:11:33.334 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:11:33.334 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:11:33.334 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:11:33.334 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.334 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.334 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.334 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 9, 2024-04-20T17:11:33.334 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "b6c17147-b3dc-475b-bb20-f9f712fb99e8", 2024-04-20T17:11:33.334 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:11:33.334 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:11:33.334 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:11:33.334 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:11:33.334 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:11:33.334 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:11:33.334 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 73, 2024-04-20T17:11:33.335 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 115, 2024-04-20T17:11:33.335 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:11:33.335 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:11:33.335 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:11:33.335 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.335 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.335 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.335 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6804", 2024-04-20T17:11:33.335 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:11:33.335 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.335 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.335 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.335 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:11:33.335 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.335 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.336 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.336 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6805", 2024-04-20T17:11:33.336 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:11:33.336 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.336 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.336 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.336 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:11:33.336 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.336 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.336 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.336 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6807", 2024-04-20T17:11:33.336 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:11:33.336 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.336 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.337 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.337 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:11:33.337 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.337 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.337 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.337 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6806", 2024-04-20T17:11:33.337 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:11:33.337 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.337 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.337 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.337 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6804/2877589636", 2024-04-20T17:11:33.337 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6805/2877589636", 2024-04-20T17:11:33.337 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6807/2877589636", 2024-04-20T17:11:33.337 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6806/2877589636", 2024-04-20T17:11:33.338 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:11:33.338 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:11:33.338 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:11:33.338 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.338 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.338 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.338 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 10, 2024-04-20T17:11:33.338 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "1adcdf55-f002-4a39-91e7-378e5fae849c", 2024-04-20T17:11:33.338 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:11:33.338 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:11:33.338 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:11:33.338 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:11:33.338 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:11:33.338 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:11:33.338 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 84, 2024-04-20T17:11:33.339 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 0, 2024-04-20T17:11:33.339 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:11:33.339 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:11:33.339 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:11:33.339 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.339 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.339 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.339 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6808", 2024-04-20T17:11:33.339 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:11:33.339 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.339 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.339 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.339 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:11:33.339 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.340 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.340 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.340 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6809", 2024-04-20T17:11:33.340 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:11:33.340 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.340 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.340 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.340 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:11:33.340 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.340 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.340 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.340 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6811", 2024-04-20T17:11:33.340 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:11:33.340 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.341 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.341 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.341 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:11:33.341 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.341 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.341 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.341 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6810", 2024-04-20T17:11:33.341 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:11:33.341 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.341 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.341 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.341 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6808/181490621", 2024-04-20T17:11:33.341 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6809/181490621", 2024-04-20T17:11:33.341 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6811/181490621", 2024-04-20T17:11:33.341 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6810/181490621", 2024-04-20T17:11:33.342 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:11:33.342 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:11:33.342 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:11:33.342 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.342 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.342 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.342 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 11, 2024-04-20T17:11:33.342 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "737c5aec-694a-4968-819d-fccf22661f4f", 2024-04-20T17:11:33.342 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:11:33.342 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:11:33.342 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:11:33.342 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:11:33.342 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:11:33.343 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:11:33.343 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 92, 2024-04-20T17:11:33.343 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 115, 2024-04-20T17:11:33.343 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:11:33.343 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:11:33.343 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:11:33.343 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.343 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.343 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.343 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6812", 2024-04-20T17:11:33.343 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:11:33.343 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.343 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.343 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.343 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:11:33.344 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.344 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.344 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.344 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6813", 2024-04-20T17:11:33.344 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:11:33.344 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.344 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.344 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.344 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:11:33.344 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.344 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.344 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.344 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6815", 2024-04-20T17:11:33.345 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:11:33.345 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.345 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.345 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.345 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:11:33.345 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:11:33.345 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.345 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:11:33.345 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6814", 2024-04-20T17:11:33.345 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:11:33.345 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.345 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.345 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.345 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6812/3373585701", 2024-04-20T17:11:33.346 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6813/3373585701", 2024-04-20T17:11:33.346 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6815/3373585701", 2024-04-20T17:11:33.346 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6814/3373585701", 2024-04-20T17:11:33.346 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:11:33.346 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:11:33.346 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:11:33.346 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.346 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.346 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:11:33.346 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_xinfo": [ 2024-04-20T17:11:33.346 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.346 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 0, 2024-04-20T17:11:33.346 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:11:33.346 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:11:33.347 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:11:33.347 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:11:33.347 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:11:33.347 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:11:03.794045+0000", 2024-04-20T17:11:33.347 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:11:33.347 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.347 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.347 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 1, 2024-04-20T17:11:33.347 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:11:33.347 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:11:33.347 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:11:33.347 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:11:33.347 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:11:33.347 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:10:31.541261+0000", 2024-04-20T17:11:33.347 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:11:33.348 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.348 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.348 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 2, 2024-04-20T17:11:33.348 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:11:33.348 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:11:33.348 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:11:33.348 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:11:33.348 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:11:33.348 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:10:40.011156+0000", 2024-04-20T17:11:33.348 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 169 2024-04-20T17:11:33.348 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.348 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.348 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 3, 2024-04-20T17:11:33.348 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:11:33.349 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:11:33.349 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:11:33.349 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:11:33.349 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:11:33.349 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:11:20.567120+0000", 2024-04-20T17:11:33.349 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:11:33.349 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.349 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.349 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 4, 2024-04-20T17:11:33.349 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:11:33.349 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:11:33.349 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:11:33.349 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:11:33.349 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:11:33.349 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:10:28.350736+0000", 2024-04-20T17:11:33.350 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:11:33.350 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.350 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.350 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 5, 2024-04-20T17:11:33.350 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:11:33.350 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:11:33.350 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:11:33.350 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:11:33.350 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:11:33.350 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:10:24.075027+0000", 2024-04-20T17:11:33.350 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:11:33.350 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.350 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.351 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 6, 2024-04-20T17:11:33.351 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:11:33.351 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:11:33.351 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:11:33.351 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:11:33.351 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:11:33.351 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:10:46.893553+0000", 2024-04-20T17:11:33.351 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:11:33.351 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.351 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.351 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 7, 2024-04-20T17:11:33.351 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:11:33.351 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:11:33.351 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:11:33.351 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:11:33.351 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:11:33.352 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:11:25.621018+0000", 2024-04-20T17:11:33.352 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:11:33.352 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.352 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.352 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 8, 2024-04-20T17:11:33.352 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:11:33.352 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:11:33.352 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:11:33.352 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:11:33.352 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:11:33.352 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:10:57.759585+0000", 2024-04-20T17:11:33.352 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:11:33.352 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.352 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.352 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 9, 2024-04-20T17:11:33.353 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:11:33.353 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:11:33.353 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:11:33.353 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:11:33.353 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:11:33.353 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:10:17.692419+0000", 2024-04-20T17:11:33.353 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:11:33.353 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.353 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.353 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 10, 2024-04-20T17:11:33.353 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:11:33.353 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:11:33.353 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:11:33.354 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:11:33.354 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:11:33.354 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:10:49.739504+0000", 2024-04-20T17:11:33.354 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:11:33.354 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.354 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.354 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 11, 2024-04-20T17:11:33.354 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:11:33.354 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:11:33.354 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:11:33.354 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:11:33.354 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:11:33.354 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:11:32.296546+0000", 2024-04-20T17:11:33.355 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:11:33.355 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.355 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:11:33.355 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_upmap": [ 2024-04-20T17:11:33.355 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.355 INFO:teuthology.orchestra.run.smithi005.stdout: "pgid": "2.9", 2024-04-20T17:11:33.355 INFO:teuthology.orchestra.run.smithi005.stdout: "osds": [ 2024-04-20T17:11:33.355 INFO:teuthology.orchestra.run.smithi005.stdout: 7, 2024-04-20T17:11:33.355 INFO:teuthology.orchestra.run.smithi005.stdout: 10 2024-04-20T17:11:33.355 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.355 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.355 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:11:33.355 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_upmap_items": [ 2024-04-20T17:11:33.355 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.355 INFO:teuthology.orchestra.run.smithi005.stdout: "pgid": "2.3", 2024-04-20T17:11:33.356 INFO:teuthology.orchestra.run.smithi005.stdout: "mappings": [ 2024-04-20T17:11:33.356 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.356 INFO:teuthology.orchestra.run.smithi005.stdout: "from": 2, 2024-04-20T17:11:33.356 INFO:teuthology.orchestra.run.smithi005.stdout: "to": 10 2024-04-20T17:11:33.356 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.356 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.356 INFO:teuthology.orchestra.run.smithi005.stdout: "from": 5, 2024-04-20T17:11:33.356 INFO:teuthology.orchestra.run.smithi005.stdout: "to": 0 2024-04-20T17:11:33.356 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.356 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.356 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.356 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.356 INFO:teuthology.orchestra.run.smithi005.stdout: "pgid": "2.4", 2024-04-20T17:11:33.357 INFO:teuthology.orchestra.run.smithi005.stdout: "mappings": [ 2024-04-20T17:11:33.357 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:11:33.357 INFO:teuthology.orchestra.run.smithi005.stdout: "from": 1, 2024-04-20T17:11:33.357 INFO:teuthology.orchestra.run.smithi005.stdout: "to": 5 2024-04-20T17:11:33.357 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.357 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:11:33.357 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.357 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:11:33.357 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_upmap_primaries": [], 2024-04-20T17:11:33.357 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_temp": [], 2024-04-20T17:11:33.357 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_temp": [], 2024-04-20T17:11:33.357 INFO:teuthology.orchestra.run.smithi005.stdout: "blocklist": { 2024-04-20T17:11:33.357 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/3881726425": "2024-04-21T16:59:25.808337+0000", 2024-04-20T17:11:33.357 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/338649532": "2024-04-21T16:59:25.808337+0000", 2024-04-20T17:11:33.357 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/1227843664": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:11:33.358 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:6800/1876064631": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:11:33.358 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/160689449": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:11:33.358 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:6800/3521715286": "2024-04-21T16:59:25.808337+0000", 2024-04-20T17:11:33.358 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/3841663462": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:11:33.358 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/687690919": "2024-04-21T16:59:25.808337+0000" 2024-04-20T17:11:33.358 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.358 INFO:teuthology.orchestra.run.smithi005.stdout: "range_blocklist": {}, 2024-04-20T17:11:33.358 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_profiles": { 2024-04-20T17:11:33.358 INFO:teuthology.orchestra.run.smithi005.stdout: "default": { 2024-04-20T17:11:33.358 INFO:teuthology.orchestra.run.smithi005.stdout: "crush-failure-domain": "osd", 2024-04-20T17:11:33.358 INFO:teuthology.orchestra.run.smithi005.stdout: "k": "2", 2024-04-20T17:11:33.358 INFO:teuthology.orchestra.run.smithi005.stdout: "m": "1", 2024-04-20T17:11:33.358 INFO:teuthology.orchestra.run.smithi005.stdout: "plugin": "jerasure", 2024-04-20T17:11:33.359 INFO:teuthology.orchestra.run.smithi005.stdout: "technique": "reed_sol_van" 2024-04-20T17:11:33.359 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.359 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:11:33.359 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_snaps_queue": [], 2024-04-20T17:11:33.359 INFO:teuthology.orchestra.run.smithi005.stdout: "new_removed_snaps": [], 2024-04-20T17:11:33.359 INFO:teuthology.orchestra.run.smithi005.stdout: "new_purged_snaps": [], 2024-04-20T17:11:33.359 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_node_flags": {}, 2024-04-20T17:11:33.359 INFO:teuthology.orchestra.run.smithi005.stdout: "device_class_flags": {}, 2024-04-20T17:11:33.359 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": { 2024-04-20T17:11:33.359 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode_enabled": false, 2024-04-20T17:11:33.359 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_bucket_count": 0, 2024-04-20T17:11:33.359 INFO:teuthology.orchestra.run.smithi005.stdout: "degraded_stretch_mode": 0, 2024-04-20T17:11:33.359 INFO:teuthology.orchestra.run.smithi005.stdout: "recovering_stretch_mode": 0, 2024-04-20T17:11:33.359 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode_bucket": 0 2024-04-20T17:11:33.359 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:11:33.359 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:11:33.360 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:32 smithi171 ceph-mon[44189]: pgmap v535: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded, 2 undersized+degraded+peered, 13 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19691/218982 objects degraded (8.992%); 5948/218982 objects misplaced (2.716%); 936 KiB/s, 14 objects/s recovering 2024-04-20T17:11:33.360 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:32 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:11:33.360 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:32 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:11:33.679 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:11:33.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:33.683 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:11:33.784 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:11:33.812 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:33.917 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:11:33.917 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:33 smithi171 ceph-mon[44189]: osdmap e183: 12 total, 11 up, 12 in 2024-04-20T17:11:33.918 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:33 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2221758116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-20T17:11:33.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:33.919 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:11:33.941 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:11:33.941 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:11:34.021 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:11:34.054 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:34.077 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:11:34.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:34.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:11:34.182 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:11:34.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:33 smithi005 ceph-mon[39562]: osdmap e183: 12 total, 11 up, 12 in 2024-04-20T17:11:34.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:33 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2221758116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-20T17:11:34.214 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:33 smithi106 ceph-mon[44423]: osdmap e183: 12 total, 11 up, 12 in 2024-04-20T17:11:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:33 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2221758116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-20T17:11:34.268 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:11:34.277 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:11:34.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:34.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:11:34.380 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:11:34.409 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:34.675 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-20T17:11:34.765 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:11:34.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:34.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:11:34.776 INFO:tasks.thrashosds.thrasher:j is {'epoch': 183, 'fsid': '13c6e76c-ff37-11ee-bc93-c7b262605968', 'created': '2024-04-20T16:58:25.272557+0000', 'modified': '2024-04-20T17:11:32.896939+0000', 'last_up_change': '2024-04-20T17:10:50.899850+0000', 'last_in_change': '2024-04-20T17:05:42.849446+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-20T17:01:35.909493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.760000228881836, 'score_stable': 11.760000228881836, 'optimal_score': 0.17000000178813934, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-04-20T17:08:04.904546+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 171, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6801', 'nonce': 3901484705}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6802', 'nonce': 3901484705}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6804', 'nonce': 3901484705}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6803', 'nonce': 3901484705}]}, 'public_addr': '172.21.15.5:6801/3901484705', 'cluster_addr': '172.21.15.5:6802/3901484705', 'heartbeat_back_addr': '172.21.15.5:6804/3901484705', 'heartbeat_front_addr': '172.21.15.5:6803/3901484705', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'aed0c8eb-0e81-4e85-ab8f-a60420060656', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6805', 'nonce': 3391893914}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6806', 'nonce': 3391893914}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6808', 'nonce': 3391893914}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6807', 'nonce': 3391893914}]}, 'public_addr': '172.21.15.5:6805/3391893914', 'cluster_addr': '172.21.15.5:6806/3391893914', 'heartbeat_back_addr': '172.21.15.5:6808/3391893914', 'heartbeat_front_addr': '172.21.15.5:6807/3391893914', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f733a6a2-19c2-4c7a-af9c-04e282062bb7', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 20, 'up_thru': 150, 'down_at': 170, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6809', 'nonce': 2105468358}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6810', 'nonce': 2105468358}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6812', 'nonce': 2105468358}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6811', 'nonce': 2105468358}]}, 'public_addr': '172.21.15.5:6809/2105468358', 'cluster_addr': '172.21.15.5:6810/2105468358', 'heartbeat_back_addr': '172.21.15.5:6812/2105468358', 'heartbeat_front_addr': '172.21.15.5:6811/2105468358', 'state': ['exists']}, {'osd': 3, 'uuid': '8c92f1df-e298-4606-88ae-b8a8e79628a1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 170, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6813', 'nonce': 377652039}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6814', 'nonce': 377652039}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6816', 'nonce': 377652039}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6815', 'nonce': 377652039}]}, 'public_addr': '172.21.15.5:6813/377652039', 'cluster_addr': '172.21.15.5:6814/377652039', 'heartbeat_back_addr': '172.21.15.5:6816/377652039', 'heartbeat_front_addr': '172.21.15.5:6815/377652039', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '46df16f1-9607-4d7b-9be8-9d4236e160d1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6800', 'nonce': 1839524469}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6801', 'nonce': 1839524469}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6803', 'nonce': 1839524469}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6802', 'nonce': 1839524469}]}, 'public_addr': '172.21.15.106:6800/1839524469', 'cluster_addr': '172.21.15.106:6801/1839524469', 'heartbeat_back_addr': '172.21.15.106:6803/1839524469', 'heartbeat_front_addr': '172.21.15.106:6802/1839524469', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6603231d-1e5c-4575-bfaa-2eeca2c12215', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 162, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6804', 'nonce': 2450315908}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6805', 'nonce': 2450315908}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6807', 'nonce': 2450315908}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6806', 'nonce': 2450315908}]}, 'public_addr': '172.21.15.106:6804/2450315908', 'cluster_addr': '172.21.15.106:6805/2450315908', 'heartbeat_back_addr': '172.21.15.106:6807/2450315908', 'heartbeat_front_addr': '172.21.15.106:6806/2450315908', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '783a55a6-f176-42be-8c8b-af8481d08dc9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6808', 'nonce': 2840237913}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6809', 'nonce': 2840237913}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6811', 'nonce': 2840237913}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6810', 'nonce': 2840237913}]}, 'public_addr': '172.21.15.106:6808/2840237913', 'cluster_addr': '172.21.15.106:6809/2840237913', 'heartbeat_back_addr': '172.21.15.106:6811/2840237913', 'heartbeat_front_addr': '172.21.15.106:6810/2840237913', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 173, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6812', 'nonce': 1105371883}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6813', 'nonce': 1105371883}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6815', 'nonce': 1105371883}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6814', 'nonce': 1105371883}]}, 'public_addr': '172.21.15.106:6812/1105371883', 'cluster_addr': '172.21.15.106:6813/1105371883', 'heartbeat_back_addr': '172.21.15.106:6815/1105371883', 'heartbeat_front_addr': '172.21.15.106:6814/1105371883', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '1cb232ad-1f59-4a0a-a215-e67419ad6476', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 67, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6800', 'nonce': 4123058942}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6801', 'nonce': 4123058942}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6803', 'nonce': 4123058942}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6802', 'nonce': 4123058942}]}, 'public_addr': '172.21.15.171:6800/4123058942', 'cluster_addr': '172.21.15.171:6801/4123058942', 'heartbeat_back_addr': '172.21.15.171:6803/4123058942', 'heartbeat_front_addr': '172.21.15.171:6802/4123058942', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'b6c17147-b3dc-475b-bb20-f9f712fb99e8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 73, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6804', 'nonce': 2877589636}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6805', 'nonce': 2877589636}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6807', 'nonce': 2877589636}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6806', 'nonce': 2877589636}]}, 'public_addr': '172.21.15.171:6804/2877589636', 'cluster_addr': '172.21.15.171:6805/2877589636', 'heartbeat_back_addr': '172.21.15.171:6807/2877589636', 'heartbeat_front_addr': '172.21.15.171:6806/2877589636', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '1adcdf55-f002-4a39-91e7-378e5fae849c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 84, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6808', 'nonce': 181490621}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6809', 'nonce': 181490621}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6811', 'nonce': 181490621}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6810', 'nonce': 181490621}]}, 'public_addr': '172.21.15.171:6808/181490621', 'cluster_addr': '172.21.15.171:6809/181490621', 'heartbeat_back_addr': '172.21.15.171:6811/181490621', 'heartbeat_front_addr': '172.21.15.171:6810/181490621', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '737c5aec-694a-4968-819d-fccf22661f4f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 92, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6812', 'nonce': 3373585701}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6813', 'nonce': 3373585701}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6815', 'nonce': 3373585701}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6814', 'nonce': 3373585701}]}, 'public_addr': '172.21.15.171:6812/3373585701', 'cluster_addr': '172.21.15.171:6813/3373585701', 'heartbeat_back_addr': '172.21.15.171:6815/3373585701', 'heartbeat_front_addr': '172.21.15.171:6814/3373585701', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:11:03.794045+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:10:31.541261+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:10:40.011156+0000', 'dead_epoch': 169}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:11:20.567120+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:10:28.350736+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:10:24.075027+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:10:46.893553+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:11:25.621018+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:10:57.759585+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:10:17.692419+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:10:49.739504+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:11:32.296546+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.9', 'osds': [7, 10]}], 'pg_upmap_items': [{'pgid': '2.3', 'mappings': [{'from': 2, 'to': 10}, {'from': 5, 'to': 0}]}, {'pgid': '2.4', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.5:0/3881726425': '2024-04-21T16:59:25.808337+0000', '172.21.15.5:0/338649532': '2024-04-21T16:59:25.808337+0000', '172.21.15.5:0/1227843664': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:6800/1876064631': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:0/160689449': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:6800/3521715286': '2024-04-21T16:59:25.808337+0000', '172.21.15.5:0/3841663462': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:0/687690919': '2024-04-21T16:59:25.808337+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-20T17:11:34.776 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:11:34.869 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:11:34.895 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:34.917 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:11:34.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:34.921 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:11:34.982 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:11:35.021 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:11:35.048 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:35.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:34 smithi005 ceph-mon[39562]: pgmap v537: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19691/218982 objects degraded (8.992%); 5948/218982 objects misplaced (2.716%); 480 KiB/s, 7 objects/s recovering 2024-04-20T17:11:35.236 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:34 smithi171 ceph-mon[44189]: pgmap v537: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19691/218982 objects degraded (8.992%); 5948/218982 objects misplaced (2.716%); 480 KiB/s, 7 objects/s recovering 2024-04-20T17:11:35.245 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:11:35.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:35.248 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:11:35.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:34 smithi106 ceph-mon[44423]: pgmap v537: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19691/218982 objects degraded (8.992%); 5948/218982 objects misplaced (2.716%); 480 KiB/s, 7 objects/s recovering 2024-04-20T17:11:35.349 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:11:35.381 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:35.454 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:11:35.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:35.456 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:11:35.482 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:11:35.557 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:11:35.590 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:35.611 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:11:35.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:35.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:11:35.715 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:11:35.743 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:35.765 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:11:35.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:35.768 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:11:35.870 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:11:35.903 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:35.926 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:11:35.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:35.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:11:36.030 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:11:36.060 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:36.080 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:11:36.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:36.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:11:36.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:35 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 19691/218982 objects degraded (8.992%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:36.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:35 smithi171 ceph-mon[44189]: osdmap e184: 12 total, 11 up, 12 in 2024-04-20T17:11:36.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:35 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 19691/218982 objects degraded (8.992%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:36.151 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:35 smithi005 ceph-mon[39562]: osdmap e184: 12 total, 11 up, 12 in 2024-04-20T17:11:36.184 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:11:36.216 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:36.249 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:11:36.249 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:11:36.249 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:11:36.249 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:11:36.257 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:11:36.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:36.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:11:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:35 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 19691/218982 objects degraded (8.992%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:35 smithi106 ceph-mon[44423]: osdmap e184: 12 total, 11 up, 12 in 2024-04-20T17:11:36.272 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:11:36.293 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:11:36.313 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:11:36.334 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:11:36.354 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:11:36.361 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:11:36.373 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:11:36.392 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:36.398 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:11:36.399 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:11:36.399 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:11:36.399 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-20T17:11:36.399 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:11:36.399 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:11:36.399 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:11:36.399 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:11:36.399 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:11:36.399 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:11:36.399 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:11:36.399 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:11:36.399 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:11:36.413 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:11:36.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:36.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:11:36.517 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:11:36.546 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:36.643 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:11:36.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:36.647 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:11:36.748 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:11:36.776 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:36.796 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:11:36.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:36.798 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:11:36.899 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:11:36.928 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:11:36.929 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:11:36.929 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:11:36.929 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:11:36.929 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.929 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:47425 2.a 2:5eb129a8:::benchmark_data_smithi171_78918_object47424:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-20T17:11:36.929 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:08:28.489964+0000", 2024-04-20T17:11:36.930 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 188.43649466700001, 2024-04-20T17:11:36.930 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.052758462999999998, 2024-04-20T17:11:36.930 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:36.930 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:36.930 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:36.930 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:36.930 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:36.930 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:36.930 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 47425 2024-04-20T17:11:36.930 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.931 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:36.931 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.931 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:36.931 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.489964+0000", 2024-04-20T17:11:36.931 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:36.931 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.931 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.931 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:36.931 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.489964+0000", 2024-04-20T17:11:36.931 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:36.931 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.931 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.931 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:36.931 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.489965+0000", 2024-04-20T17:11:36.931 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3340000000000001e-06 2024-04-20T17:11:36.932 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.932 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.932 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:36.932 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.489984+0000", 2024-04-20T17:11:36.932 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8604e-05 2024-04-20T17:11:36.932 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.932 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.932 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:36.932 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.489994+0000", 2024-04-20T17:11:36.932 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0179e-05 2024-04-20T17:11:36.932 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.932 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.932 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:36.932 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.490002+0000", 2024-04-20T17:11:36.933 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5809999999999999e-06 2024-04-20T17:11:36.933 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.933 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.933 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:36.933 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.490068+0000", 2024-04-20T17:11:36.933 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5809000000000006e-05 2024-04-20T17:11:36.933 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.933 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.933 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:36.933 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.490207+0000", 2024-04-20T17:11:36.933 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013893400000000001 2024-04-20T17:11:36.933 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.933 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.933 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:11:36.934 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.490343+0000", 2024-04-20T17:11:36.934 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000136911 2024-04-20T17:11:36.934 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.934 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.934 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:36.934 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.494984+0000", 2024-04-20T17:11:36.934 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0046409820000000001 2024-04-20T17:11:36.934 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.934 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.934 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:36.934 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.542620+0000", 2024-04-20T17:11:36.934 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.047635499999999997 2024-04-20T17:11:36.934 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.934 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.935 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:36.935 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.542681+0000", 2024-04-20T17:11:36.935 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1426000000000004e-05 2024-04-20T17:11:36.935 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.935 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.935 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:36.935 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.542723+0000", 2024-04-20T17:11:36.935 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1202999999999998e-05 2024-04-20T17:11:36.935 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:36.935 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:36.935 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:36.935 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.935 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.936 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:47427 2.a 2:579a24bf:::benchmark_data_smithi171_78918_object47426:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-20T17:11:36.936 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:08:28.490506+0000", 2024-04-20T17:11:36.936 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 188.43595296699999, 2024-04-20T17:11:36.936 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.052400438000000001, 2024-04-20T17:11:36.936 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:36.936 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:36.936 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:36.936 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:36.936 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:36.936 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:36.936 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 47427 2024-04-20T17:11:36.936 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.936 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:36.936 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.937 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:36.937 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.490506+0000", 2024-04-20T17:11:36.937 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:36.937 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.937 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.937 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:36.937 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.490506+0000", 2024-04-20T17:11:36.937 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:36.937 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.937 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.937 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:36.937 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.490507+0000", 2024-04-20T17:11:36.937 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4750000000000001e-06 2024-04-20T17:11:36.937 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.937 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.938 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:36.938 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.490686+0000", 2024-04-20T17:11:36.938 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00017897099999999999 2024-04-20T17:11:36.938 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.938 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.938 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:36.938 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.490693+0000", 2024-04-20T17:11:36.938 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9650000000000002e-06 2024-04-20T17:11:36.938 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.938 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.938 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:36.938 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.490700+0000", 2024-04-20T17:11:36.938 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5540000000000003e-06 2024-04-20T17:11:36.938 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.939 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.939 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:36.939 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.491895+0000", 2024-04-20T17:11:36.939 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0011954139999999999 2024-04-20T17:11:36.939 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.939 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.939 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:36.939 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.492013+0000", 2024-04-20T17:11:36.939 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000117533 2024-04-20T17:11:36.939 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.939 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.939 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:11:36.939 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.492179+0000", 2024-04-20T17:11:36.939 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000166577 2024-04-20T17:11:36.939 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.940 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.940 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:36.940 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.495090+0000", 2024-04-20T17:11:36.940 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0029102239999999999 2024-04-20T17:11:36.940 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.940 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.940 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:36.940 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.542820+0000", 2024-04-20T17:11:36.940 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.047730677999999999 2024-04-20T17:11:36.940 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.940 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.940 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:36.940 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.542870+0000", 2024-04-20T17:11:36.940 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9805999999999997e-05 2024-04-20T17:11:36.940 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.941 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.941 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:36.941 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.542906+0000", 2024-04-20T17:11:36.941 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6241000000000001e-05 2024-04-20T17:11:36.941 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:36.941 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:36.941 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:36.941 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.941 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.941 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:47447 2.a 2:52d1136e:::benchmark_data_smithi171_78918_object47446:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-20T17:11:36.941 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:08:28.496000+0000", 2024-04-20T17:11:36.941 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 188.430459152, 2024-04-20T17:11:36.941 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.048016830000000003, 2024-04-20T17:11:36.941 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:36.941 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:36.942 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:36.942 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:36.942 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:36.942 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:36.942 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 47447 2024-04-20T17:11:36.942 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.942 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:36.942 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.942 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:36.942 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.496000+0000", 2024-04-20T17:11:36.942 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:36.943 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.943 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.943 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:36.943 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.496000+0000", 2024-04-20T17:11:36.943 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:36.943 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.943 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.943 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:36.943 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.496002+0000", 2024-04-20T17:11:36.943 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1280000000000002e-06 2024-04-20T17:11:36.943 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.943 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.943 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:36.943 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.496031+0000", 2024-04-20T17:11:36.944 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9723999999999999e-05 2024-04-20T17:11:36.944 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.944 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.944 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:36.944 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.496042+0000", 2024-04-20T17:11:36.944 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0524e-05 2024-04-20T17:11:36.944 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.944 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.944 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:36.944 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.496050+0000", 2024-04-20T17:11:36.944 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4570000000000005e-06 2024-04-20T17:11:36.944 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.944 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.945 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:36.945 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.496104+0000", 2024-04-20T17:11:36.945 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3792000000000003e-05 2024-04-20T17:11:36.945 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.945 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.945 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:36.945 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.496236+0000", 2024-04-20T17:11:36.945 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013201 2024-04-20T17:11:36.945 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.945 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.945 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:11:36.945 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.496377+0000", 2024-04-20T17:11:36.946 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014099199999999999 2024-04-20T17:11:36.946 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.946 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.946 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:36.946 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.497930+0000", 2024-04-20T17:11:36.946 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0015531900000000001 2024-04-20T17:11:36.946 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.946 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.946 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:36.946 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.543918+0000", 2024-04-20T17:11:36.946 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.045987871 2024-04-20T17:11:36.946 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.946 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.946 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:36.947 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.543976+0000", 2024-04-20T17:11:36.947 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7812000000000001e-05 2024-04-20T17:11:36.947 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.947 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.947 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:36.947 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:28.544016+0000", 2024-04-20T17:11:36.947 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0330000000000002e-05 2024-04-20T17:11:36.947 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:36.947 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:36.947 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:36.947 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.947 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.947 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:91249 2.c 2:30fac588:::benchmark_data_smithi171_78918_object91248:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e135)", 2024-04-20T17:11:36.947 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:08:51.003383+0000", 2024-04-20T17:11:36.947 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 165.92307573799999, 2024-04-20T17:11:36.948 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.051943793000000002, 2024-04-20T17:11:36.948 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:36.948 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:36.948 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:36.948 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:36.948 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:36.948 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:36.948 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 91249 2024-04-20T17:11:36.948 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.948 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:36.949 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.949 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:36.949 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:51.003383+0000", 2024-04-20T17:11:36.949 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:36.949 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.949 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.950 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:36.950 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:51.003383+0000", 2024-04-20T17:11:36.950 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:36.950 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.950 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.950 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:36.950 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:51.003385+0000", 2024-04-20T17:11:36.950 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3159999999999999e-06 2024-04-20T17:11:36.950 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.950 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.950 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:36.950 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:51.003410+0000", 2024-04-20T17:11:36.950 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.482e-05 2024-04-20T17:11:36.951 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.951 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.951 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:36.951 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:51.003425+0000", 2024-04-20T17:11:36.951 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4579000000000001e-05 2024-04-20T17:11:36.951 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.951 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.951 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:36.951 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:51.003434+0000", 2024-04-20T17:11:36.951 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2380000000000003e-06 2024-04-20T17:11:36.951 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.951 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.951 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:36.951 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:51.003489+0000", 2024-04-20T17:11:36.951 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5401e-05 2024-04-20T17:11:36.951 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.952 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.952 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:36.952 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:51.003629+0000", 2024-04-20T17:11:36.952 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013952800000000001 2024-04-20T17:11:36.952 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.952 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.952 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:36.952 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:51.003783+0000", 2024-04-20T17:11:36.952 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000154072 2024-04-20T17:11:36.952 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.952 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.952 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:36.952 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:51.006365+0000", 2024-04-20T17:11:36.952 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0025820349999999999 2024-04-20T17:11:36.953 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.953 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.953 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:36.953 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:51.055259+0000", 2024-04-20T17:11:36.953 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.048894182000000001 2024-04-20T17:11:36.953 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.954 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.954 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:36.954 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:51.055296+0000", 2024-04-20T17:11:36.954 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6853000000000001e-05 2024-04-20T17:11:36.954 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.954 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.954 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:36.954 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:51.055327+0000", 2024-04-20T17:11:36.954 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0769000000000003e-05 2024-04-20T17:11:36.955 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:36.955 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:36.955 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:36.955 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.955 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.955 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:101261 2.c 2:3d73fbb6:::benchmark_data_smithi171_78918_object101260:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e136)", 2024-04-20T17:11:36.955 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:08:57.026241+0000", 2024-04-20T17:11:36.955 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 159.900218271, 2024-04-20T17:11:36.955 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.055197837999999999, 2024-04-20T17:11:36.955 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:36.955 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:36.955 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:36.955 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:36.955 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:36.955 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:36.956 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 101261 2024-04-20T17:11:36.956 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.956 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:36.956 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.956 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:36.956 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:57.026241+0000", 2024-04-20T17:11:36.956 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:36.956 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.956 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.956 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:36.956 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:57.026241+0000", 2024-04-20T17:11:36.956 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:36.956 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.956 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.957 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:36.957 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:57.026243+0000", 2024-04-20T17:11:36.957 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7480000000000002e-06 2024-04-20T17:11:36.957 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.957 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.957 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:36.957 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:57.026270+0000", 2024-04-20T17:11:36.957 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6327999999999999e-05 2024-04-20T17:11:36.957 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.957 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.957 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:36.957 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:57.026283+0000", 2024-04-20T17:11:36.957 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3631999999999999e-05 2024-04-20T17:11:36.958 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.958 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.958 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:36.958 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:57.026291+0000", 2024-04-20T17:11:36.958 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8220000000000007e-06 2024-04-20T17:11:36.958 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.958 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.958 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:36.958 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:57.026350+0000", 2024-04-20T17:11:36.958 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9203000000000002e-05 2024-04-20T17:11:36.958 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.958 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.958 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:36.959 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:57.026525+0000", 2024-04-20T17:11:36.959 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00017447400000000001 2024-04-20T17:11:36.959 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.959 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.959 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:36.959 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:57.026671+0000", 2024-04-20T17:11:36.959 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000146592 2024-04-20T17:11:36.959 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.959 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.959 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:36.960 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:57.029274+0000", 2024-04-20T17:11:36.960 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0026023309999999998 2024-04-20T17:11:36.960 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.960 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.960 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:36.960 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:57.081347+0000", 2024-04-20T17:11:36.960 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.052072938999999999 2024-04-20T17:11:36.960 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.960 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.960 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:36.960 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:57.081398+0000", 2024-04-20T17:11:36.960 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1712999999999998e-05 2024-04-20T17:11:36.960 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.961 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.961 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:36.961 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:08:57.081438+0000", 2024-04-20T17:11:36.961 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0055999999999997e-05 2024-04-20T17:11:36.961 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:36.961 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:36.961 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:36.962 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.962 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.962 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:141062 2.c 2:3970e3d8:::benchmark_data_smithi171_78918_object141061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e143)", 2024-04-20T17:11:36.962 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:09:24.961311+0000", 2024-04-20T17:11:36.962 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 131.96514808200001, 2024-04-20T17:11:36.962 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.054121486000000003, 2024-04-20T17:11:36.962 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:36.962 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:36.962 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:36.962 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:36.963 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:36.963 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:36.963 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 141062 2024-04-20T17:11:36.963 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.963 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:36.963 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.963 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:36.963 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:24.961311+0000", 2024-04-20T17:11:36.963 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:36.963 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.963 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.963 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:36.963 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:24.961311+0000", 2024-04-20T17:11:36.964 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:36.964 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.964 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.964 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:36.964 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:24.961312+0000", 2024-04-20T17:11:36.964 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3939999999999999e-06 2024-04-20T17:11:36.964 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.964 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.964 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:36.964 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:24.961333+0000", 2024-04-20T17:11:36.964 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0922999999999999e-05 2024-04-20T17:11:36.964 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.964 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.964 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:36.965 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:24.961342+0000", 2024-04-20T17:11:36.965 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4889999999999995e-06 2024-04-20T17:11:36.965 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.965 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.965 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:36.965 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:24.961348+0000", 2024-04-20T17:11:36.965 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8279999999999997e-06 2024-04-20T17:11:36.965 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.965 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.965 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:36.965 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:24.964926+0000", 2024-04-20T17:11:36.965 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.003577551 2024-04-20T17:11:36.965 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.965 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.965 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:36.965 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:24.965067+0000", 2024-04-20T17:11:36.966 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014145499999999999 2024-04-20T17:11:36.966 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.966 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.966 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:36.966 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:24.965197+0000", 2024-04-20T17:11:36.966 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012948699999999999 2024-04-20T17:11:36.966 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.966 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.966 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:36.966 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:24.968663+0000", 2024-04-20T17:11:36.966 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0034657749999999999 2024-04-20T17:11:36.966 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.966 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.967 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:36.967 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:25.015344+0000", 2024-04-20T17:11:36.967 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.046681380000000001 2024-04-20T17:11:36.967 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.967 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.967 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:36.967 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:25.015384+0000", 2024-04-20T17:11:36.967 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0438000000000001e-05 2024-04-20T17:11:36.967 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.967 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.967 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:36.967 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:25.015432+0000", 2024-04-20T17:11:36.967 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7766000000000003e-05 2024-04-20T17:11:36.967 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:36.967 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:36.968 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:36.968 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.968 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.968 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:143722 2.c 2:39b55b09:::benchmark_data_smithi171_78918_object143721:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-04-20T17:11:36.968 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:09:27.080315+0000", 2024-04-20T17:11:36.968 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 129.84614405400001, 2024-04-20T17:11:36.968 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.057183052999999998, 2024-04-20T17:11:36.968 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:36.968 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:36.968 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:36.968 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:36.968 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:36.968 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:36.969 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 143722 2024-04-20T17:11:36.969 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.969 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:36.969 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.969 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:36.969 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.080315+0000", 2024-04-20T17:11:36.969 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:36.969 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.969 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.969 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:36.969 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.080315+0000", 2024-04-20T17:11:36.969 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:36.969 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.969 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.969 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:36.969 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.080316+0000", 2024-04-20T17:11:36.970 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.528e-06 2024-04-20T17:11:36.970 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.970 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.970 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:36.970 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.080338+0000", 2024-04-20T17:11:36.970 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1928000000000001e-05 2024-04-20T17:11:36.970 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.970 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.970 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:36.970 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.080349+0000", 2024-04-20T17:11:36.970 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0684e-05 2024-04-20T17:11:36.970 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.970 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.971 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:36.971 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.080357+0000", 2024-04-20T17:11:36.971 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0479999999999999e-06 2024-04-20T17:11:36.971 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.971 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.971 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:36.971 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.080410+0000", 2024-04-20T17:11:36.971 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3066000000000003e-05 2024-04-20T17:11:36.971 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.971 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.971 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:36.971 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.080524+0000", 2024-04-20T17:11:36.971 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011411199999999999 2024-04-20T17:11:36.971 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.971 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.971 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:36.972 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.080659+0000", 2024-04-20T17:11:36.972 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013531400000000001 2024-04-20T17:11:36.972 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.972 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.972 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:36.972 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.086934+0000", 2024-04-20T17:11:36.972 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0062750699999999998 2024-04-20T17:11:36.972 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.972 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.972 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:36.972 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.137406+0000", 2024-04-20T17:11:36.972 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.050471939 2024-04-20T17:11:36.972 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.972 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.972 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:36.972 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.137450+0000", 2024-04-20T17:11:36.973 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3118000000000003e-05 2024-04-20T17:11:36.973 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.973 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.973 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:36.973 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.137498+0000", 2024-04-20T17:11:36.973 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8245999999999999e-05 2024-04-20T17:11:36.973 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:36.973 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:36.973 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:36.973 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.973 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.973 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:144339 2.c 2:3d0a8c7c:::benchmark_data_smithi171_78918_object144338:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-04-20T17:11:36.973 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:09:27.623423+0000", 2024-04-20T17:11:36.974 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 129.30303556000001, 2024-04-20T17:11:36.974 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.058652787999999997, 2024-04-20T17:11:36.974 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:36.974 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:36.974 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:36.974 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:36.974 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:36.974 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:36.974 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 144339 2024-04-20T17:11:36.974 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.975 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:36.975 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.975 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:36.975 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.623423+0000", 2024-04-20T17:11:36.975 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:36.975 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.975 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.975 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:36.975 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.623423+0000", 2024-04-20T17:11:36.975 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:36.975 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.975 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.975 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:36.975 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.623425+0000", 2024-04-20T17:11:36.976 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4929999999999999e-06 2024-04-20T17:11:36.976 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.976 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.976 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:36.976 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.623449+0000", 2024-04-20T17:11:36.976 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3845999999999998e-05 2024-04-20T17:11:36.976 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.976 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.976 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:36.976 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.623461+0000", 2024-04-20T17:11:36.976 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2407e-05 2024-04-20T17:11:36.976 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.976 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.976 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:36.976 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.623469+0000", 2024-04-20T17:11:36.977 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6739999999999997e-06 2024-04-20T17:11:36.977 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.977 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.977 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:36.977 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.625818+0000", 2024-04-20T17:11:36.977 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0023498130000000001 2024-04-20T17:11:36.977 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.977 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.977 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:36.977 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.625955+0000", 2024-04-20T17:11:36.977 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000136453 2024-04-20T17:11:36.977 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.977 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.977 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:36.978 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.626111+0000", 2024-04-20T17:11:36.978 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015560699999999999 2024-04-20T17:11:36.978 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.978 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.978 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:36.978 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.634430+0000", 2024-04-20T17:11:36.979 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0083195839999999997 2024-04-20T17:11:36.979 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.979 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.979 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:36.979 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.681991+0000", 2024-04-20T17:11:36.979 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.047561084000000003 2024-04-20T17:11:36.979 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.979 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.979 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:36.979 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.682029+0000", 2024-04-20T17:11:36.979 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7960000000000002e-05 2024-04-20T17:11:36.979 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.979 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.980 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:36.980 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.682076+0000", 2024-04-20T17:11:36.980 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6866999999999997e-05 2024-04-20T17:11:36.980 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:36.980 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:36.980 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:36.980 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.980 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.980 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:144341 2.c 2:3f83532a:::benchmark_data_smithi171_78918_object144340:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-04-20T17:11:36.980 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:09:27.627092+0000", 2024-04-20T17:11:36.980 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 129.29936652999999, 2024-04-20T17:11:36.980 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.055133806, 2024-04-20T17:11:36.980 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:36.980 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:36.980 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:36.981 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:36.981 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:36.981 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:36.981 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 144341 2024-04-20T17:11:36.981 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.981 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:36.981 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.981 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:36.981 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.627092+0000", 2024-04-20T17:11:36.981 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:36.981 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.981 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.981 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:36.981 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.627092+0000", 2024-04-20T17:11:36.982 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:36.982 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.982 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.982 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:36.982 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.627094+0000", 2024-04-20T17:11:36.982 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1519999999999999e-06 2024-04-20T17:11:36.982 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.982 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.982 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:36.982 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.627117+0000", 2024-04-20T17:11:36.982 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.234e-05 2024-04-20T17:11:36.982 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.983 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.983 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:36.983 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.627127+0000", 2024-04-20T17:11:36.983 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0471e-05 2024-04-20T17:11:36.983 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.983 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.983 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:36.983 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.627134+0000", 2024-04-20T17:11:36.983 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0149999999999998e-06 2024-04-20T17:11:36.983 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.983 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.983 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:36.983 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.629949+0000", 2024-04-20T17:11:36.983 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.002814268 2024-04-20T17:11:36.983 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.984 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.984 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:36.984 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.630074+0000", 2024-04-20T17:11:36.984 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012503300000000001 2024-04-20T17:11:36.984 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.984 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.984 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:36.984 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.630221+0000", 2024-04-20T17:11:36.984 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014770699999999999 2024-04-20T17:11:36.984 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.984 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.984 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:36.984 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.634437+0000", 2024-04-20T17:11:36.984 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0042160799999999997 2024-04-20T17:11:36.984 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.984 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.985 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:36.985 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.682126+0000", 2024-04-20T17:11:36.985 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.047688213 2024-04-20T17:11:36.985 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.985 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.985 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:36.985 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.682182+0000", 2024-04-20T17:11:36.985 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6162999999999997e-05 2024-04-20T17:11:36.985 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.985 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.985 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:36.985 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.682226+0000", 2024-04-20T17:11:36.985 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4363999999999997e-05 2024-04-20T17:11:36.985 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:36.986 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:36.986 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:36.986 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.986 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.986 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:144350 2.c 2:3d6daa0b:::benchmark_data_smithi171_78918_object144349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-04-20T17:11:36.986 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:09:27.632986+0000", 2024-04-20T17:11:36.986 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 129.293472597, 2024-04-20T17:11:36.986 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.049390060999999999, 2024-04-20T17:11:36.986 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:36.986 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:36.986 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:36.987 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:36.987 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:36.987 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:36.987 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 144350 2024-04-20T17:11:36.987 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.987 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:36.987 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.987 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:36.987 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.632986+0000", 2024-04-20T17:11:36.987 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:36.987 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.987 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.987 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:36.987 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.632986+0000", 2024-04-20T17:11:36.988 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:36.988 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.988 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.988 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:36.988 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.632988+0000", 2024-04-20T17:11:36.988 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8789999999999999e-06 2024-04-20T17:11:36.988 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.988 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.988 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:36.988 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.633008+0000", 2024-04-20T17:11:36.988 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0217999999999999e-05 2024-04-20T17:11:36.988 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.988 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.988 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:36.988 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.633018+0000", 2024-04-20T17:11:36.989 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.009e-05 2024-04-20T17:11:36.989 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.989 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.989 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:36.989 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.633024+0000", 2024-04-20T17:11:36.989 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9750000000000004e-06 2024-04-20T17:11:36.989 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.989 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.989 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:36.989 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.638160+0000", 2024-04-20T17:11:36.989 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0051355810000000002 2024-04-20T17:11:36.989 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.989 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.989 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:36.989 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.638297+0000", 2024-04-20T17:11:36.989 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013696100000000001 2024-04-20T17:11:36.990 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.990 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.990 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:36.990 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.638417+0000", 2024-04-20T17:11:36.990 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000120396 2024-04-20T17:11:36.990 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.990 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.990 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:36.990 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.644910+0000", 2024-04-20T17:11:36.990 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.006492879 2024-04-20T17:11:36.990 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.990 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.990 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:36.991 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.682282+0000", 2024-04-20T17:11:36.991 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.037371595000000001 2024-04-20T17:11:36.991 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.991 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.991 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:36.991 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.682332+0000", 2024-04-20T17:11:36.991 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9835e-05 2024-04-20T17:11:36.991 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.991 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.991 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:36.991 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:27.682376+0000", 2024-04-20T17:11:36.991 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4651999999999999e-05 2024-04-20T17:11:36.991 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:36.992 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:36.992 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:36.992 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.992 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.992 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:147990 2.c 2:397286c7:::benchmark_data_smithi171_78918_object147989:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-04-20T17:11:36.992 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:09:30.507646+0000", 2024-04-20T17:11:36.992 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 126.41881298, 2024-04-20T17:11:36.992 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.053486158999999998, 2024-04-20T17:11:36.992 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:36.992 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:36.992 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:36.992 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:36.992 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:36.992 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:36.992 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 147990 2024-04-20T17:11:36.993 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.993 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:36.993 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.993 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:36.993 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:30.507646+0000", 2024-04-20T17:11:36.993 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:36.993 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.993 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.993 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:36.993 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:30.507646+0000", 2024-04-20T17:11:36.993 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:36.993 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.993 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.994 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:36.994 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:30.507648+0000", 2024-04-20T17:11:36.994 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.001e-06 2024-04-20T17:11:36.994 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.994 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.994 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:36.994 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:30.507667+0000", 2024-04-20T17:11:36.994 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9497e-05 2024-04-20T17:11:36.994 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.994 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.994 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:36.995 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:30.507678+0000", 2024-04-20T17:11:36.995 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0504e-05 2024-04-20T17:11:36.995 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.995 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.995 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:36.995 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:30.507686+0000", 2024-04-20T17:11:36.995 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0779999999999996e-06 2024-04-20T17:11:36.995 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.995 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.995 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:36.995 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:30.507735+0000", 2024-04-20T17:11:36.995 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8921999999999997e-05 2024-04-20T17:11:36.995 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.995 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.995 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:36.996 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:30.507842+0000", 2024-04-20T17:11:36.996 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000107075 2024-04-20T17:11:36.996 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.996 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.996 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:36.996 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:30.507936+0000", 2024-04-20T17:11:36.996 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.4550999999999996e-05 2024-04-20T17:11:36.996 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.996 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.996 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:36.996 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:30.511337+0000", 2024-04-20T17:11:36.997 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0034003359999999999 2024-04-20T17:11:36.997 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.997 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.997 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:36.997 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:30.561012+0000", 2024-04-20T17:11:36.997 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.049675142999999998 2024-04-20T17:11:36.997 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.997 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.997 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:36.997 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:30.561080+0000", 2024-04-20T17:11:36.997 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8071000000000006e-05 2024-04-20T17:11:36.998 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.998 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.998 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:36.998 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:30.561132+0000", 2024-04-20T17:11:36.998 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1981000000000003e-05 2024-04-20T17:11:36.998 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:36.998 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:36.998 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:36.998 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:36.998 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:36.998 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:148887 2.c 2:3eb10028:::benchmark_data_smithi171_78918_object148886:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-04-20T17:11:36.998 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:09:31.278165+0000", 2024-04-20T17:11:36.999 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 125.648294247, 2024-04-20T17:11:36.999 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.048085502000000002, 2024-04-20T17:11:36.999 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:36.999 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:36.999 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:36.999 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:36.999 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:36.999 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:37.000 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 148887 2024-04-20T17:11:37.000 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.000 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:37.000 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.000 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:37.001 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.278165+0000", 2024-04-20T17:11:37.001 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:37.001 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.001 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.001 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:37.001 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.278165+0000", 2024-04-20T17:11:37.001 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:37.001 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.002 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.002 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:37.002 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.278168+0000", 2024-04-20T17:11:37.002 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2289999999999999e-06 2024-04-20T17:11:37.002 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.002 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.002 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:37.002 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.278194+0000", 2024-04-20T17:11:37.002 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5741999999999998e-05 2024-04-20T17:11:37.002 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.003 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.003 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:37.003 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.278215+0000", 2024-04-20T17:11:37.003 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1234999999999999e-05 2024-04-20T17:11:37.003 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.003 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.003 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:37.003 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.278222+0000", 2024-04-20T17:11:37.003 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1110000000000002e-06 2024-04-20T17:11:37.003 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.003 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.003 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:37.003 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.278289+0000", 2024-04-20T17:11:37.003 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6828000000000005e-05 2024-04-20T17:11:37.003 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.003 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.004 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:37.004 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.278420+0000", 2024-04-20T17:11:37.004 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013122899999999999 2024-04-20T17:11:37.004 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.004 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.004 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:37.004 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.278538+0000", 2024-04-20T17:11:37.004 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000118257 2024-04-20T17:11:37.004 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.004 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.004 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:37.004 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.282984+0000", 2024-04-20T17:11:37.004 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0044460819999999996 2024-04-20T17:11:37.004 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.004 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.005 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:37.005 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.326139+0000", 2024-04-20T17:11:37.005 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043154813 2024-04-20T17:11:37.005 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.005 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.005 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:37.005 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.326198+0000", 2024-04-20T17:11:37.005 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9268999999999998e-05 2024-04-20T17:11:37.005 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.005 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.005 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:37.005 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.326250+0000", 2024-04-20T17:11:37.005 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1706999999999998e-05 2024-04-20T17:11:37.005 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:37.005 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:37.005 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:37.006 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.006 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.006 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:148906 2.c 2:3c57aaea:::benchmark_data_smithi171_78918_object148905:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e146)", 2024-04-20T17:11:37.006 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:09:31.289306+0000", 2024-04-20T17:11:37.006 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 125.637153243, 2024-04-20T17:11:37.006 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.050091911000000003, 2024-04-20T17:11:37.006 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:37.006 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:37.006 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:37.006 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:37.006 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:37.006 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:37.006 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 148906 2024-04-20T17:11:37.007 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.007 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:37.007 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.007 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:37.007 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.289306+0000", 2024-04-20T17:11:37.007 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:37.007 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.007 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.007 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:37.007 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.289306+0000", 2024-04-20T17:11:37.007 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:37.007 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.007 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.007 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:37.008 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.289307+0000", 2024-04-20T17:11:37.008 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.533e-06 2024-04-20T17:11:37.008 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.008 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.008 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:37.008 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.289327+0000", 2024-04-20T17:11:37.008 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9425000000000001e-05 2024-04-20T17:11:37.008 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.008 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.008 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:37.008 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.289337+0000", 2024-04-20T17:11:37.008 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.023e-05 2024-04-20T17:11:37.008 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.009 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.009 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:37.009 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.289342+0000", 2024-04-20T17:11:37.009 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.75e-06 2024-04-20T17:11:37.009 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.009 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.009 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:37.009 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.292362+0000", 2024-04-20T17:11:37.009 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0030190170000000001 2024-04-20T17:11:37.009 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.009 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.009 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:37.009 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.292482+0000", 2024-04-20T17:11:37.009 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000120683 2024-04-20T17:11:37.009 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.010 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.010 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:37.010 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.292584+0000", 2024-04-20T17:11:37.010 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000101524 2024-04-20T17:11:37.010 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.010 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.010 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:37.010 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.295677+0000", 2024-04-20T17:11:37.010 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0030930300000000001 2024-04-20T17:11:37.010 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.010 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.010 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:37.010 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.339284+0000", 2024-04-20T17:11:37.011 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043607435999999999 2024-04-20T17:11:37.011 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.011 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.011 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:37.011 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.339348+0000", 2024-04-20T17:11:37.011 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3473999999999996e-05 2024-04-20T17:11:37.011 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.011 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.011 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:37.011 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:31.339397+0000", 2024-04-20T17:11:37.011 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9808999999999997e-05 2024-04-20T17:11:37.011 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:37.011 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:37.012 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:37.012 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.012 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.012 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:176371 2.c 2:3d917d6f:::benchmark_data_smithi171_78918_object15377:head [delete] snapc 0=[] ondisk+write+known_if_redirected e154)", 2024-04-20T17:11:37.012 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:06.842220+0000", 2024-04-20T17:11:37.012 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 90.084239264999994, 2024-04-20T17:11:37.012 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.050929812999999997, 2024-04-20T17:11:37.012 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:37.012 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:37.012 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:37.012 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:37.012 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:37.012 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:37.012 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 176371 2024-04-20T17:11:37.012 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.013 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:37.013 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.013 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:37.013 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:06.842220+0000", 2024-04-20T17:11:37.013 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:37.013 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.013 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.013 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:37.013 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:06.842220+0000", 2024-04-20T17:11:37.013 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:37.013 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.013 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.013 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:37.013 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:06.842221+0000", 2024-04-20T17:11:37.013 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7120000000000001e-06 2024-04-20T17:11:37.013 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.013 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.014 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:37.014 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:06.842226+0000", 2024-04-20T17:11:37.014 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1250000000000001e-06 2024-04-20T17:11:37.014 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.014 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.014 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:37.014 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:06.842236+0000", 2024-04-20T17:11:37.014 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.4029999999999996e-06 2024-04-20T17:11:37.014 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.014 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.014 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:37.014 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:06.842241+0000", 2024-04-20T17:11:37.014 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7189999999999998e-06 2024-04-20T17:11:37.014 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.015 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.015 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:37.015 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:06.842304+0000", 2024-04-20T17:11:37.015 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2997999999999996e-05 2024-04-20T17:11:37.015 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.015 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.015 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:37.015 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:06.842435+0000", 2024-04-20T17:11:37.015 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013079899999999999 2024-04-20T17:11:37.015 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.015 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.015 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:37.015 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:06.842557+0000", 2024-04-20T17:11:37.015 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012123 2024-04-20T17:11:37.015 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.016 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.016 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:37.016 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:06.849634+0000", 2024-04-20T17:11:37.016 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0070778409999999996 2024-04-20T17:11:37.016 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.016 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.016 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:37.016 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:06.893043+0000", 2024-04-20T17:11:37.016 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043409097000000001 2024-04-20T17:11:37.016 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.016 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.016 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:37.016 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:06.893099+0000", 2024-04-20T17:11:37.016 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5658000000000001e-05 2024-04-20T17:11:37.017 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.017 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.017 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:37.017 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:06.893149+0000", 2024-04-20T17:11:37.017 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0231000000000001e-05 2024-04-20T17:11:37.017 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:37.017 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:37.017 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:37.017 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.017 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.017 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:176373 2.c 2:3e490d22:::benchmark_data_smithi171_78918_object15379:head [delete] snapc 0=[] ondisk+write+known_if_redirected e154)", 2024-04-20T17:11:37.017 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:06.846133+0000", 2024-04-20T17:11:37.017 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 90.080325724000005, 2024-04-20T17:11:37.017 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.047142680999999999, 2024-04-20T17:11:37.017 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:37.018 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:37.018 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:37.018 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:37.018 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:37.018 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:37.018 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 176373 2024-04-20T17:11:37.018 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.018 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:37.018 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.018 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:37.018 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:06.846133+0000", 2024-04-20T17:11:37.018 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:37.019 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.019 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.019 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:37.019 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:06.846133+0000", 2024-04-20T17:11:37.019 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:37.019 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.019 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.019 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:37.019 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:06.846136+0000", 2024-04-20T17:11:37.019 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.05e-06 2024-04-20T17:11:37.019 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.019 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.020 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:37.020 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:06.846144+0000", 2024-04-20T17:11:37.020 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9400000000000002e-06 2024-04-20T17:11:37.020 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.020 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.020 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:37.020 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:06.846159+0000", 2024-04-20T17:11:37.020 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4864e-05 2024-04-20T17:11:37.020 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.020 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.020 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:37.020 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:06.846169+0000", 2024-04-20T17:11:37.020 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6199999999999994e-06 2024-04-20T17:11:37.020 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.021 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.021 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:37.021 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:06.846229+0000", 2024-04-20T17:11:37.021 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0909000000000002e-05 2024-04-20T17:11:37.021 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.021 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.021 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:37.021 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:06.846323+0000", 2024-04-20T17:11:37.021 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3894999999999995e-05 2024-04-20T17:11:37.021 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.021 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.021 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:37.021 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:06.846400+0000", 2024-04-20T17:11:37.021 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6501000000000005e-05 2024-04-20T17:11:37.021 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.022 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.022 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:37.022 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:06.849806+0000", 2024-04-20T17:11:37.022 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0034058199999999999 2024-04-20T17:11:37.022 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.022 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.022 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:37.022 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:06.893213+0000", 2024-04-20T17:11:37.022 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043406861999999997 2024-04-20T17:11:37.022 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.022 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.022 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:37.022 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:06.893248+0000", 2024-04-20T17:11:37.023 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5831999999999998e-05 2024-04-20T17:11:37.023 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.023 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.023 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:37.023 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:06.893276+0000", 2024-04-20T17:11:37.023 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7387999999999999e-05 2024-04-20T17:11:37.023 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:37.023 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:37.023 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:37.023 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.023 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.023 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:181735 2.c 2:3e0c3f23:::benchmark_data_smithi171_78918_object20741:head [delete] snapc 0=[] ondisk+write+known_if_redirected e157)", 2024-04-20T17:11:37.023 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:13.287575+0000", 2024-04-20T17:11:37.024 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 83.638883401000001, 2024-04-20T17:11:37.024 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.056994733999999998, 2024-04-20T17:11:37.024 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:37.024 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:37.024 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:37.024 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:37.024 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:37.024 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:37.024 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 181735 2024-04-20T17:11:37.024 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.024 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:37.024 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.024 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:37.024 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.287575+0000", 2024-04-20T17:11:37.024 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:37.025 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.025 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.025 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:37.025 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.287575+0000", 2024-04-20T17:11:37.025 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:37.025 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.025 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.025 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:37.025 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.287577+0000", 2024-04-20T17:11:37.025 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.08e-06 2024-04-20T17:11:37.025 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.025 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.025 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:37.025 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.287584+0000", 2024-04-20T17:11:37.025 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1129999999999997e-06 2024-04-20T17:11:37.026 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.026 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.026 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:37.026 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.287596+0000", 2024-04-20T17:11:37.026 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2391999999999999e-05 2024-04-20T17:11:37.026 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.026 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.026 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:37.026 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.287605+0000", 2024-04-20T17:11:37.026 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5790000000000004e-06 2024-04-20T17:11:37.026 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.026 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.026 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:37.026 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.287679+0000", 2024-04-20T17:11:37.027 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4571000000000001e-05 2024-04-20T17:11:37.027 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.027 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.027 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:37.027 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.287811+0000", 2024-04-20T17:11:37.027 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013169200000000001 2024-04-20T17:11:37.027 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.027 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.027 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:37.027 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.287957+0000", 2024-04-20T17:11:37.027 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014658299999999999 2024-04-20T17:11:37.027 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.027 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.027 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:37.028 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.299326+0000", 2024-04-20T17:11:37.028 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011368723000000001 2024-04-20T17:11:37.028 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.028 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.028 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:37.028 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.344494+0000", 2024-04-20T17:11:37.028 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.045167455000000002 2024-04-20T17:11:37.028 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.028 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.028 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:37.028 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.344527+0000", 2024-04-20T17:11:37.028 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3775000000000001e-05 2024-04-20T17:11:37.029 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.029 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.029 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:37.029 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.344570+0000", 2024-04-20T17:11:37.029 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2771000000000001e-05 2024-04-20T17:11:37.029 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:37.029 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:37.029 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:37.029 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.029 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.029 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:181739 2.c 2:363e7d5a:::benchmark_data_smithi171_78918_object20745:head [delete] snapc 0=[] ondisk+write+known_if_redirected e157)", 2024-04-20T17:11:37.029 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:13.287946+0000", 2024-04-20T17:11:37.029 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 83.638512546000001, 2024-04-20T17:11:37.030 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.056758889999999999, 2024-04-20T17:11:37.030 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:37.030 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:37.030 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:37.030 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:37.030 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:37.030 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:37.030 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 181739 2024-04-20T17:11:37.030 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.030 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:37.030 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.030 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:37.030 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.287946+0000", 2024-04-20T17:11:37.031 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:37.031 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.031 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.031 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:37.031 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.287946+0000", 2024-04-20T17:11:37.031 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:37.031 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.031 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.031 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:37.031 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.287948+0000", 2024-04-20T17:11:37.031 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.494e-06 2024-04-20T17:11:37.031 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.031 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.031 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:37.031 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.287953+0000", 2024-04-20T17:11:37.032 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.276e-06 2024-04-20T17:11:37.032 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.032 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.032 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:37.032 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.287963+0000", 2024-04-20T17:11:37.032 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.8090000000000004e-06 2024-04-20T17:11:37.032 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.032 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.032 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:37.032 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.287970+0000", 2024-04-20T17:11:37.032 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5499999999999997e-06 2024-04-20T17:11:37.032 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.032 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.032 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:37.032 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.292612+0000", 2024-04-20T17:11:37.032 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0046411270000000001 2024-04-20T17:11:37.033 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.033 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.033 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:37.033 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.292719+0000", 2024-04-20T17:11:37.033 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000107377 2024-04-20T17:11:37.033 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.033 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.033 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:37.033 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.292801+0000", 2024-04-20T17:11:37.033 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2231999999999995e-05 2024-04-20T17:11:37.033 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.033 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.033 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:37.033 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.299329+0000", 2024-04-20T17:11:37.033 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0065280700000000004 2024-04-20T17:11:37.034 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.034 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.034 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:37.034 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.344619+0000", 2024-04-20T17:11:37.034 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.045289994 2024-04-20T17:11:37.034 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.034 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.034 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:37.034 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.344667+0000", 2024-04-20T17:11:37.034 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7540999999999998e-05 2024-04-20T17:11:37.034 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.034 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.034 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:37.034 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.344705+0000", 2024-04-20T17:11:37.035 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8420000000000001e-05 2024-04-20T17:11:37.035 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:37.035 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:37.035 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:37.035 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.035 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.035 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:181740 2.c 2:304bada1:::benchmark_data_smithi171_78918_object20746:head [delete] snapc 0=[] ondisk+write+known_if_redirected e157)", 2024-04-20T17:11:37.035 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:13.288097+0000", 2024-04-20T17:11:37.035 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 83.638361997000004, 2024-04-20T17:11:37.035 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.056743586999999998, 2024-04-20T17:11:37.035 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:37.035 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:37.035 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:37.035 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:37.035 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:37.035 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:37.036 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 181740 2024-04-20T17:11:37.036 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.036 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:37.036 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.036 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:37.036 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.288097+0000", 2024-04-20T17:11:37.036 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:37.036 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.036 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.036 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:37.036 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.288097+0000", 2024-04-20T17:11:37.036 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:37.036 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.036 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.036 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:37.037 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.288098+0000", 2024-04-20T17:11:37.037 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4330000000000001e-06 2024-04-20T17:11:37.037 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.037 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.037 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:37.037 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.288103+0000", 2024-04-20T17:11:37.037 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8099999999999997e-06 2024-04-20T17:11:37.037 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.037 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.037 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:37.037 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.288112+0000", 2024-04-20T17:11:37.037 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8929999999999994e-06 2024-04-20T17:11:37.037 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.038 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.038 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:37.038 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.288119+0000", 2024-04-20T17:11:37.038 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8940000000000004e-06 2024-04-20T17:11:37.038 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.038 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.038 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:37.038 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.295990+0000", 2024-04-20T17:11:37.038 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0078712049999999992 2024-04-20T17:11:37.038 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.038 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.038 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:37.038 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.296072+0000", 2024-04-20T17:11:37.038 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1891000000000006e-05 2024-04-20T17:11:37.039 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.039 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.039 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:37.039 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.296151+0000", 2024-04-20T17:11:37.039 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8903e-05 2024-04-20T17:11:37.039 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.039 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.039 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:37.039 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.299369+0000", 2024-04-20T17:11:37.039 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00321857 2024-04-20T17:11:37.039 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.039 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.039 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:37.039 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.344755+0000", 2024-04-20T17:11:37.040 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.045385389999999998 2024-04-20T17:11:37.040 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.040 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.040 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:37.040 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.344801+0000", 2024-04-20T17:11:37.040 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6566000000000001e-05 2024-04-20T17:11:37.040 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.040 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.040 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:37.040 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:13.344840+0000", 2024-04-20T17:11:37.040 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9032000000000002e-05 2024-04-20T17:11:37.040 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:37.040 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:37.040 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:37.040 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.041 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.041 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:182847 2.c 2:390ba3b0:::benchmark_data_smithi171_78918_object21853:head [delete] snapc 0=[] ondisk+write+known_if_redirected e157)", 2024-04-20T17:11:37.041 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:14.556286+0000", 2024-04-20T17:11:37.041 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 82.370173104000003, 2024-04-20T17:11:37.041 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.048300177999999999, 2024-04-20T17:11:37.041 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:37.041 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:37.041 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:37.041 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:37.041 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:37.041 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:37.041 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 182847 2024-04-20T17:11:37.041 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.041 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:37.041 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.042 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:37.042 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:14.556286+0000", 2024-04-20T17:11:37.042 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:37.042 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.042 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.042 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:37.042 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:14.556286+0000", 2024-04-20T17:11:37.042 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:37.042 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.042 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.042 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:37.042 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:14.556286+0000", 2024-04-20T17:11:37.042 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3799999999999996e-07 2024-04-20T17:11:37.042 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.043 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.043 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:37.043 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:14.556290+0000", 2024-04-20T17:11:37.043 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.292e-06 2024-04-20T17:11:37.043 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.043 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.043 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:37.043 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:14.556296+0000", 2024-04-20T17:11:37.043 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.686e-06 2024-04-20T17:11:37.043 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.043 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.043 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:37.043 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:14.556301+0000", 2024-04-20T17:11:37.043 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9450000000000001e-06 2024-04-20T17:11:37.044 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.044 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.044 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:37.044 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:14.560673+0000", 2024-04-20T17:11:37.044 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0043716060000000001 2024-04-20T17:11:37.044 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.044 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.044 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:37.044 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:14.560807+0000", 2024-04-20T17:11:37.044 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000134084 2024-04-20T17:11:37.044 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.044 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.044 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:37.044 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:14.560940+0000", 2024-04-20T17:11:37.044 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000132698 2024-04-20T17:11:37.044 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.045 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.045 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:37.045 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:14.564345+0000", 2024-04-20T17:11:37.045 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0034054580000000001 2024-04-20T17:11:37.045 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.045 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.045 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:37.045 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:14.604486+0000", 2024-04-20T17:11:37.045 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.040140677 2024-04-20T17:11:37.045 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.045 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.045 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:37.045 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:14.604537+0000", 2024-04-20T17:11:37.045 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1459999999999999e-05 2024-04-20T17:11:37.045 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.045 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.046 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:37.046 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:14.604586+0000", 2024-04-20T17:11:37.046 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8534e-05 2024-04-20T17:11:37.046 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:37.046 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:37.046 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:37.046 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.046 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.046 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:202245 2.c 2:39d65621:::benchmark_data_smithi171_78918_object41251:head [delete] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-04-20T17:11:37.046 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:39.785186+0000", 2024-04-20T17:11:37.047 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 57.141273167000001, 2024-04-20T17:11:37.047 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.057341021999999998, 2024-04-20T17:11:37.047 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:11:37.047 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:11:37.047 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:11:37.047 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:11:37.047 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:11:37.047 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:11:37.047 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 202245 2024-04-20T17:11:37.047 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.047 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:11:37.047 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.047 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:11:37.048 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.785186+0000", 2024-04-20T17:11:37.048 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:37.048 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.048 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.048 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:11:37.048 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.785186+0000", 2024-04-20T17:11:37.048 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:11:37.048 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.048 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.048 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:11:37.048 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.785189+0000", 2024-04-20T17:11:37.048 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.954e-06 2024-04-20T17:11:37.048 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.048 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.049 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:11:37.049 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.785196+0000", 2024-04-20T17:11:37.049 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0509999999999999e-06 2024-04-20T17:11:37.049 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.049 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.049 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:11:37.049 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.785214+0000", 2024-04-20T17:11:37.049 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.836e-05 2024-04-20T17:11:37.049 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.049 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.049 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:11:37.049 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.785223+0000", 2024-04-20T17:11:37.050 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.7099999999999996e-06 2024-04-20T17:11:37.050 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.050 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.050 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:11:37.050 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.785286+0000", 2024-04-20T17:11:37.050 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2964000000000002e-05 2024-04-20T17:11:37.050 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.050 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.050 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:11:37.050 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.785402+0000", 2024-04-20T17:11:37.051 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000116549 2024-04-20T17:11:37.051 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.051 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.051 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:11:37.051 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.785512+0000", 2024-04-20T17:11:37.051 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010943 2024-04-20T17:11:37.051 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.051 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.051 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:11:37.051 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.790112+0000", 2024-04-20T17:11:37.051 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0046002660000000004 2024-04-20T17:11:37.051 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.051 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.051 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:11:37.051 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.842463+0000", 2024-04-20T17:11:37.052 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.052350918000000003 2024-04-20T17:11:37.052 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.052 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.052 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:11:37.052 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.842498+0000", 2024-04-20T17:11:37.052 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5136999999999999e-05 2024-04-20T17:11:37.052 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:11:37.052 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:11:37.052 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:11:37.052 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.842527+0000", 2024-04-20T17:11:37.052 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8683e-05 2024-04-20T17:11:37.052 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:37.052 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:37.052 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:37.052 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:11:37.052 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:11:37.052 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:11:37.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:37.054 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:37.054 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:11:37.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:11:37.155 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:11:37.186 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:36 smithi106 ceph-mon[44423]: pgmap v539: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19631/218982 objects degraded (8.965%); 5888/218982 objects misplaced (2.689%); 640 KiB/s, 9 objects/s recovering 2024-04-20T17:11:37.186 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:36 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:11:37.186 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:36 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:11:37.186 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:36 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/273153112' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:11:37.186 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:37.186 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:37.187 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:37.187 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:37.187 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:37.187 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:37.187 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:37.187 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:37.188 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:37.209 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:11:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:36 smithi005 ceph-mon[39562]: pgmap v539: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19631/218982 objects degraded (8.965%); 5888/218982 objects misplaced (2.689%); 640 KiB/s, 9 objects/s recovering 2024-04-20T17:11:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:36 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:11:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:36 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:11:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:36 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/273153112' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:11:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:37.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:37.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:37.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:37.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:37.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:11:37.314 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:11:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:36 smithi171 ceph-mon[44189]: pgmap v539: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19631/218982 objects degraded (8.965%); 5888/218982 objects misplaced (2.689%); 640 KiB/s, 9 objects/s recovering 2024-04-20T17:11:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:36 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:11:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:36 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:11:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:36 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/273153112' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:11:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:37.344 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:37.740 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-20T17:11:37.793 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:11:37.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:37.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:11:37.828 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:11:37.829 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:11:37.897 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:11:37.927 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:37.947 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:11:37.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:37.950 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:11:38.051 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:11:38.076 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:38.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:38.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:38.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:37 smithi171 ceph-mon[44189]: osdmap e185: 12 total, 11 up, 12 in 2024-04-20T17:11:38.078 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:38.097 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:11:38.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:38.101 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:11:38.201 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:11:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:37 smithi005 ceph-mon[39562]: osdmap e185: 12 total, 11 up, 12 in 2024-04-20T17:11:38.232 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:38.232 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:38.232 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:38.232 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:37 smithi106 ceph-mon[44423]: osdmap e185: 12 total, 11 up, 12 in 2024-04-20T17:11:38.234 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:38.257 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:11:38.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:38.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:11:38.361 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:11:38.392 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:38.412 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:11:38.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:38.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:11:38.443 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:11:38.481 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:11:38.516 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:11:38.544 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:38.564 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:11:38.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:38.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:11:38.668 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:11:38.695 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:38.713 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:11:38.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:38.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:11:38.817 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:11:38.844 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:38.862 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:11:38.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:38.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:11:38.966 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:11:38.993 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:39.011 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:11:39.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:39.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:11:39.115 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:11:39.148 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:39.169 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:11:39.171 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:38 smithi171 ceph-mon[44189]: pgmap v541: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19631/218982 objects degraded (8.965%); 5888/218982 objects misplaced (2.689%); 640 KiB/s, 9 objects/s recovering 2024-04-20T17:11:39.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:39.173 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:11:39.187 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:38 smithi005 ceph-mon[39562]: pgmap v541: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19631/218982 objects degraded (8.965%); 5888/218982 objects misplaced (2.689%); 640 KiB/s, 9 objects/s recovering 2024-04-20T17:11:39.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:38 smithi106 ceph-mon[44423]: pgmap v541: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19631/218982 objects degraded (8.965%); 5888/218982 objects misplaced (2.689%); 640 KiB/s, 9 objects/s recovering 2024-04-20T17:11:39.274 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:11:39.304 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:39.830 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:11:40.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:39 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:11:40.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:39 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:11:40.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:39 smithi005 ceph-mon[39562]: osdmap e186: 12 total, 11 up, 12 in 2024-04-20T17:11:40.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:39 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:11:40.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:39 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:11:40.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:39 smithi106 ceph-mon[44423]: osdmap e186: 12 total, 11 up, 12 in 2024-04-20T17:11:40.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:39 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:11:40.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:39 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:11:40.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:39 smithi171 ceph-mon[44189]: osdmap e186: 12 total, 11 up, 12 in 2024-04-20T17:11:40.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:40 smithi005 ceph-mon[39562]: pgmap v543: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19570/218982 objects degraded (8.937%); 5828/218982 objects misplaced (2.661%); 1.2 MiB/s, 19 objects/s recovering 2024-04-20T17:11:40.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:40 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 19631/218982 objects degraded (8.965%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:40.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:40 smithi005 ceph-mon[39562]: osdmap e187: 12 total, 11 up, 12 in 2024-04-20T17:11:41.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:40 smithi106 ceph-mon[44423]: pgmap v543: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19570/218982 objects degraded (8.937%); 5828/218982 objects misplaced (2.661%); 1.2 MiB/s, 19 objects/s recovering 2024-04-20T17:11:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:40 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 19631/218982 objects degraded (8.965%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:40 smithi106 ceph-mon[44423]: osdmap e187: 12 total, 11 up, 12 in 2024-04-20T17:11:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:40 smithi171 ceph-mon[44189]: pgmap v543: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19570/218982 objects degraded (8.937%); 5828/218982 objects misplaced (2.661%); 1.2 MiB/s, 19 objects/s recovering 2024-04-20T17:11:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:40 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 19631/218982 objects degraded (8.965%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:40 smithi171 ceph-mon[44189]: osdmap e187: 12 total, 11 up, 12 in 2024-04-20T17:11:41.756 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:11:41.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:41.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:11:41.861 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:11:41.892 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:42.018 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:11:42.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:42.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:11:42.123 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:11:42.156 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:42.177 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:11:42.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:42.180 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:11:42.281 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:11:42.315 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:42.320 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:11:42.337 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:11:42.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:42.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:11:42.441 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:11:42.472 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:42.604 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:11:42.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:42.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:11:42.710 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:11:42.743 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:42.764 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:11:42.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:42.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:11:42.868 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:11:42.900 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:42.921 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:11:42.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:42.925 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:11:42.954 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:11:42.954 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:11:42.954 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-04-20T17:11:42.954 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:11:43.026 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:11:43.058 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:11:43.064 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:43 smithi171 ceph-mon[44189]: pgmap v545: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19570/218982 objects degraded (8.937%); 5828/218982 objects misplaced (2.661%); 640 KiB/s, 9 objects/s recovering 2024-04-20T17:11:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:43 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/666575899' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:11:43.417 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:11:43.418 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:43 smithi005 ceph-mon[39562]: pgmap v545: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19570/218982 objects degraded (8.937%); 5828/218982 objects misplaced (2.661%); 640 KiB/s, 9 objects/s recovering 2024-04-20T17:11:43.418 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:43 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/666575899' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:11:43.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:43.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:11:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:43 smithi106 ceph-mon[44423]: pgmap v545: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19570/218982 objects degraded (8.937%); 5828/218982 objects misplaced (2.661%); 640 KiB/s, 9 objects/s recovering 2024-04-20T17:11:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:43 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/666575899' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:11:43.520 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:11:43.553 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:43.575 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:11:43.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:43.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:11:43.679 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:11:43.708 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:44.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:44 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:11:44.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:44 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/666575899' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:11:44.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:44 smithi171 ceph-mon[44189]: osdmap e188: 12 total, 11 up, 12 in 2024-04-20T17:11:44.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:44 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:11:44.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:44 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/666575899' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:11:44.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:44 smithi005 ceph-mon[39562]: osdmap e188: 12 total, 11 up, 12 in 2024-04-20T17:11:44.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:44 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:11:44.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:44 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/666575899' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:11:44.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:44 smithi106 ceph-mon[44423]: osdmap e188: 12 total, 11 up, 12 in 2024-04-20T17:11:44.557 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:11:44.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:44.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:11:44.661 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:11:44.691 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:44.729 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:11:44.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:44.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:11:44.833 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:11:44.864 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:44.885 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:11:44.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:44.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:11:44.992 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:11:45.025 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:45.046 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:11:45.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:45.049 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:11:45.071 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:11:45.072 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:11:45.150 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:11:45.176 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:45 smithi005 ceph-mon[39562]: pgmap v547: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19570/218982 objects degraded (8.937%); 5828/218982 objects misplaced (2.661%); 640 KiB/s, 9 objects/s recovering 2024-04-20T17:11:45.179 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:45.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:45 smithi171 ceph-mon[44189]: pgmap v547: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19570/218982 objects degraded (8.937%); 5828/218982 objects misplaced (2.661%); 640 KiB/s, 9 objects/s recovering 2024-04-20T17:11:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:45 smithi106 ceph-mon[44423]: pgmap v547: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19570/218982 objects degraded (8.937%); 5828/218982 objects misplaced (2.661%); 640 KiB/s, 9 objects/s recovering 2024-04-20T17:11:46.029 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:11:46.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:46.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:11:46.058 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-20T17:11:46.133 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:11:46.166 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:46.187 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:11:46.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:46.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:11:46.291 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:11:46.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:46 smithi005 ceph-mon[39562]: from='client.24943 v1:172.21.15.5:0/813872175' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:11:46.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:46 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 19570/218982 objects degraded (8.937%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:46.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:46 smithi005 ceph-mon[39562]: osdmap e189: 12 total, 11 up, 12 in 2024-04-20T17:11:46.321 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:46.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:46 smithi171 ceph-mon[44189]: from='client.24943 v1:172.21.15.5:0/813872175' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:11:46.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:46 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 19570/218982 objects degraded (8.937%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:46.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:46 smithi171 ceph-mon[44189]: osdmap e189: 12 total, 11 up, 12 in 2024-04-20T17:11:46.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:46 smithi106 ceph-mon[44423]: from='client.24943 v1:172.21.15.5:0/813872175' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:11:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:46 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 19570/218982 objects degraded (8.937%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:46 smithi106 ceph-mon[44423]: osdmap e189: 12 total, 11 up, 12 in 2024-04-20T17:11:46.967 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":547,"stamp":"2024-04-20T17:11:43.995301+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7175995960,"num_objects":109491,"num_object_clones":0,"num_object_copies":218982,"num_objects_missing_on_primary":5827,"num_objects_missing":5827,"num_objects_degraded":19570,"num_objects_misplaced":5828,"num_objects_unfound":0,"num_objects_dirty":109491,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":212676,"num_write_kb":10306363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16527,"num_bytes_recovered":1084704896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160932,"ondisk_log_size":160932,"up":32,"acting":32,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":38,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":15828892,"kb_used_data":14917516,"kb_used_omap":18,"kb_used_meta":911213,"kb_avail":1109112932,"statfs":{"total":1151940427776,"available":1135731642368,"internally_reserved":0,"allocated":15275536384,"data_stored":15270407958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":933082425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":-61,"num_objects_missing":-61,"num_objects_degraded":-61,"num_objects_misplaced":-60,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":3932160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001139"},"pg_stats":[{"pgid":"2.f","version":"175'13293","reported_seq":13699,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-20T17:11:43.296313+0000","last_change":"2024-04-20T17:10:56.390079+0000","last_active":"2024-04-20T17:11:43.296313+0000","last_peered":"2024-04-20T17:11:43.296313+0000","last_clean":"2024-04-20T17:11:43.296313+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:11:43.296313+0000","last_undegraded":"2024-04-20T17:11:43.296313+0000","last_fullsized":"2024-04-20T17:11:43.296313+0000","mapping_epoch":115,"log_start":"124'3200","ondisk_log_start":"124'3200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'13292","last_scrub_stamp":"2024-04-20T17:10:56.389959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:10:56.389959+0000","objects_scrubbed":6890,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:12:23.073235+0000","scrub_duration":538,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451477504,"num_objects":6889,"num_object_clones":0,"num_object_copies":13778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13293,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"169'13276","reported_seq":24757,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-20T17:11:38.976774+0000","last_change":"2024-04-20T17:10:40.496871+0000","last_active":"2024-04-20T17:11:38.976774+0000","last_peered":"2024-04-20T17:11:38.976774+0000","last_clean":"2024-04-20T17:11:38.976774+0000","last_became_active":"2024-04-20T17:10:28.865670+0000","last_became_peered":"2024-04-20T17:10:28.865670+0000","last_unstale":"2024-04-20T17:11:38.976774+0000","last_undegraded":"2024-04-20T17:11:38.976774+0000","last_fullsized":"2024-04-20T17:11:38.976774+0000","mapping_epoch":162,"log_start":"124'3200","ondisk_log_start":"124'3200","created":115,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'12703","last_scrub_stamp":"2024-04-20T17:10:40.496707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:10:40.496707+0000","objects_scrubbed":7387,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:11:45.616990+0000","scrub_duration":1195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444334080,"num_objects":6780,"num_object_clones":0,"num_object_copies":13560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13276,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5619,"num_bytes_recovered":368246784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"175'13551","reported_seq":13790,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-20T17:11:38.976773+0000","last_change":"2024-04-20T17:10:35.632158+0000","last_active":"2024-04-20T17:11:38.976773+0000","last_peered":"2024-04-20T17:11:38.976773+0000","last_clean":"2024-04-20T17:11:38.976773+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:11:38.976773+0000","last_undegraded":"2024-04-20T17:11:38.976773+0000","last_fullsized":"2024-04-20T17:11:38.976773+0000","mapping_epoch":115,"log_start":"125'3500","ondisk_log_start":"125'3500","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'12643","last_scrub_stamp":"2024-04-20T17:10:35.632031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:10:35.632031+0000","objects_scrubbed":7872,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:11:55.425530+0000","scrub_duration":1332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454230016,"num_objects":6931,"num_object_clones":0,"num_object_copies":13862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13551,"num_write_kb":655424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"169'13365","reported_seq":13603,"reported_epoch":187,"state":"undersized+degraded+peered","last_fresh":"2024-04-20T17:11:40.864151+0000","last_change":"2024-04-20T17:10:51.918355+0000","last_active":"2024-04-20T17:10:50.919902+0000","last_peered":"2024-04-20T17:11:40.864151+0000","last_clean":"2024-04-20T17:10:50.905426+0000","last_became_active":"2024-04-20T17:08:06.027804+0000","last_became_peered":"2024-04-20T17:10:51.918355+0000","last_unstale":"2024-04-20T17:11:40.864151+0000","last_undegraded":"2024-04-20T17:10:51.917937+0000","last_fullsized":"2024-04-20T17:10:51.917860+0000","mapping_epoch":170,"log_start":"124'3300","ondisk_log_start":"124'3300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'7483","last_scrub_stamp":"2024-04-20T17:09:09.239533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:09:09.239533+0000","objects_scrubbed":7410,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:11:41.677031+0000","scrub_duration":1594,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450166784,"num_objects":6869,"num_object_clones":0,"num_object_copies":13738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6869,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13365,"num_write_kb":647488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":6869}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"175'13027","reported_seq":23997,"reported_epoch":186,"state":"active+recovering+degraded","last_fresh":"2024-04-20T17:11:39.397088+0000","last_change":"2024-04-20T17:10:59.345918+0000","last_active":"2024-04-20T17:11:39.397088+0000","last_peered":"2024-04-20T17:11:39.397088+0000","last_clean":"2024-04-20T17:09:35.882603+0000","last_became_active":"2024-04-20T17:10:54.470253+0000","last_became_peered":"2024-04-20T17:10:54.470253+0000","last_unstale":"2024-04-20T17:11:39.397088+0000","last_undegraded":"2024-04-20T17:10:54.461454+0000","last_fullsized":"2024-04-20T17:11:39.397088+0000","mapping_epoch":171,"log_start":"124'3000","ondisk_log_start":"124'3000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439287808,"num_objects":6703,"num_object_clones":0,"num_object_copies":13406,"num_objects_missing_on_primary":5827,"num_objects_missing":5827,"num_objects_degraded":5827,"num_objects_misplaced":5828,"num_objects_unfound":0,"num_objects_dirty":6703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13027,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3604,"num_bytes_recovered":236191744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":["5"],"object_location_counts":[{"shards":"0,5","objects":1},{"shards":"0,10","objects":875},{"shards":"5","objects":5827}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":542,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-20T17:11:40.882496+0000","last_change":"2024-04-20T17:11:30.691542+0000","last_active":"2024-04-20T17:11:40.882496+0000","last_peered":"2024-04-20T17:11:40.882496+0000","last_clean":"2024-04-20T17:11:40.882496+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:11:40.882496+0000","last_undegraded":"2024-04-20T17:11:40.882496+0000","last_fullsized":"2024-04-20T17:11:40.882496+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:11:30.691437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:11:30.691437+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:12:59.401862+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"175'13381","reported_seq":13716,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-20T17:11:40.882585+0000","last_change":"2024-04-20T17:11:00.352332+0000","last_active":"2024-04-20T17:11:40.882585+0000","last_peered":"2024-04-20T17:11:40.882585+0000","last_clean":"2024-04-20T17:11:40.882585+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:11:40.882585+0000","last_undegraded":"2024-04-20T17:11:40.882585+0000","last_fullsized":"2024-04-20T17:11:40.882585+0000","mapping_epoch":115,"log_start":"124'3300","ondisk_log_start":"124'3300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'13376","last_scrub_stamp":"2024-04-20T17:11:00.352212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:11:00.352212+0000","objects_scrubbed":6876,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:12:12.934458+0000","scrub_duration":602,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450297856,"num_objects":6871,"num_object_clones":0,"num_object_copies":13742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13381,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"175'13375","reported_seq":13712,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-20T17:11:43.296301+0000","last_change":"2024-04-20T17:10:54.602964+0000","last_active":"2024-04-20T17:11:43.296301+0000","last_peered":"2024-04-20T17:11:43.296301+0000","last_clean":"2024-04-20T17:11:43.296301+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:11:43.296301+0000","last_undegraded":"2024-04-20T17:11:43.296301+0000","last_fullsized":"2024-04-20T17:11:43.296301+0000","mapping_epoch":115,"log_start":"124'3300","ondisk_log_start":"124'3300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'13372","last_scrub_stamp":"2024-04-20T17:10:54.602783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:10:54.602783+0000","objects_scrubbed":6952,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:12:19.908628+0000","scrub_duration":699,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455409664,"num_objects":6949,"num_object_clones":0,"num_object_copies":13898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13375,"num_write_kb":650368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"175'13306","reported_seq":13639,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-20T17:11:40.881857+0000","last_change":"2024-04-20T17:11:25.505882+0000","last_active":"2024-04-20T17:11:40.881857+0000","last_peered":"2024-04-20T17:11:40.881857+0000","last_clean":"2024-04-20T17:11:40.881857+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:11:40.881857+0000","last_undegraded":"2024-04-20T17:11:40.881857+0000","last_fullsized":"2024-04-20T17:11:40.881857+0000","mapping_epoch":115,"log_start":"124'3300","ondisk_log_start":"124'3300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'13306","last_scrub_stamp":"2024-04-20T17:11:25.505756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:11:25.505756+0000","objects_scrubbed":6736,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:12:38.695007+0000","scrub_duration":572,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":441450496,"num_objects":6736,"num_object_clones":0,"num_object_copies":13472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13306,"num_write_kb":641344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"175'13415","reported_seq":13749,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-20T17:11:40.863658+0000","last_change":"2024-04-20T17:10:54.342672+0000","last_active":"2024-04-20T17:11:40.863658+0000","last_peered":"2024-04-20T17:11:40.863658+0000","last_clean":"2024-04-20T17:11:40.863658+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:11:40.863658+0000","last_undegraded":"2024-04-20T17:11:40.863658+0000","last_fullsized":"2024-04-20T17:11:40.863658+0000","mapping_epoch":115,"log_start":"124'3400","ondisk_log_start":"124'3400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'13413","last_scrub_stamp":"2024-04-20T17:10:54.342574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:10:54.342574+0000","objects_scrubbed":6807,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:12:20.034029+0000","scrub_duration":594,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445972480,"num_objects":6805,"num_object_clones":0,"num_object_copies":13610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13415,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"175'13219","reported_seq":13553,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-20T17:11:40.882493+0000","last_change":"2024-04-20T17:11:28.302573+0000","last_active":"2024-04-20T17:11:40.882493+0000","last_peered":"2024-04-20T17:11:40.882493+0000","last_clean":"2024-04-20T17:11:40.882493+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:11:40.882493+0000","last_undegraded":"2024-04-20T17:11:40.882493+0000","last_fullsized":"2024-04-20T17:11:40.882493+0000","mapping_epoch":115,"log_start":"124'3200","ondisk_log_start":"124'3200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'13219","last_scrub_stamp":"2024-04-20T17:11:28.302415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:11:28.302415+0000","objects_scrubbed":6833,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:12:45.284554+0000","scrub_duration":578,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447807488,"num_objects":6833,"num_object_clones":0,"num_object_copies":13666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13219,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"175'13313","reported_seq":28253,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-20T17:11:40.882370+0000","last_change":"2024-04-20T17:11:25.264437+0000","last_active":"2024-04-20T17:11:40.882370+0000","last_peered":"2024-04-20T17:11:40.882370+0000","last_clean":"2024-04-20T17:11:40.882370+0000","last_became_active":"2024-04-20T17:10:55.931751+0000","last_became_peered":"2024-04-20T17:10:55.931751+0000","last_unstale":"2024-04-20T17:11:40.882370+0000","last_undegraded":"2024-04-20T17:11:40.882370+0000","last_fullsized":"2024-04-20T17:11:40.882370+0000","mapping_epoch":173,"log_start":"124'3300","ondisk_log_start":"124'3300","created":115,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'13313","last_scrub_stamp":"2024-04-20T17:11:25.264325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:11:25.264325+0000","objects_scrubbed":6949,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:12:33.630987+0000","scrub_duration":575,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455409664,"num_objects":6949,"num_object_clones":0,"num_object_copies":13898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13313,"num_write_kb":648384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7296,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"175'13450","reported_seq":13783,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-20T17:11:40.882807+0000","last_change":"2024-04-20T17:10:51.686364+0000","last_active":"2024-04-20T17:11:40.882807+0000","last_peered":"2024-04-20T17:11:40.882807+0000","last_clean":"2024-04-20T17:11:40.882807+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:11:40.882807+0000","last_undegraded":"2024-04-20T17:11:40.882807+0000","last_fullsized":"2024-04-20T17:11:40.882807+0000","mapping_epoch":115,"log_start":"124'3400","ondisk_log_start":"124'3400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'13447","last_scrub_stamp":"2024-04-20T17:10:51.686248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:10:51.686248+0000","objects_scrubbed":6869,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:12:05.921427+0000","scrub_duration":581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449970176,"num_objects":6866,"num_object_clones":0,"num_object_copies":13732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13450,"num_write_kb":650112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"175'13097","reported_seq":13433,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-20T17:11:43.296281+0000","last_change":"2024-04-20T17:10:49.088022+0000","last_active":"2024-04-20T17:11:43.296281+0000","last_peered":"2024-04-20T17:11:43.296281+0000","last_clean":"2024-04-20T17:11:43.296281+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:11:43.296281+0000","last_undegraded":"2024-04-20T17:11:43.296281+0000","last_fullsized":"2024-04-20T17:11:43.296281+0000","mapping_epoch":115,"log_start":"124'3000","ondisk_log_start":"124'3000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'12982","last_scrub_stamp":"2024-04-20T17:10:49.087740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:10:49.087740+0000","objects_scrubbed":6907,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:12:11.511722+0000","scrub_duration":1078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443482112,"num_objects":6767,"num_object_clones":0,"num_object_copies":13534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13097,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"175'13241","reported_seq":13574,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-20T17:11:40.882621+0000","last_change":"2024-04-20T17:10:54.855660+0000","last_active":"2024-04-20T17:11:40.882621+0000","last_peered":"2024-04-20T17:11:40.882621+0000","last_clean":"2024-04-20T17:11:40.882621+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:11:40.882621+0000","last_undegraded":"2024-04-20T17:11:40.882621+0000","last_fullsized":"2024-04-20T17:11:40.882621+0000","mapping_epoch":115,"log_start":"124'3200","ondisk_log_start":"124'3200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'13236","last_scrub_stamp":"2024-04-20T17:10:54.855532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:10:54.855532+0000","objects_scrubbed":6866,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:12:12.033503+0000","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449642496,"num_objects":6861,"num_object_clones":0,"num_object_copies":13722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13241,"num_write_kb":643264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"175'13064","reported_seq":13399,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-04-20T17:11:38.922463+0000","last_change":"2024-04-20T17:11:29.185786+0000","last_active":"2024-04-20T17:11:38.922463+0000","last_peered":"2024-04-20T17:11:38.922463+0000","last_clean":"2024-04-20T17:11:38.922463+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:11:38.922463+0000","last_undegraded":"2024-04-20T17:11:38.922463+0000","last_fullsized":"2024-04-20T17:11:38.922463+0000","mapping_epoch":115,"log_start":"124'3000","ondisk_log_start":"124'3000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'13064","last_scrub_stamp":"2024-04-20T17:11:29.185665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:11:29.185665+0000","objects_scrubbed":6806,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:12:38.228491+0000","scrub_duration":558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445972504,"num_objects":6806,"num_object_clones":0,"num_object_copies":13612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6806,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13064,"num_write_kb":635777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"169'13122","reported_seq":13381,"reported_epoch":187,"state":"undersized+degraded+peered","last_fresh":"2024-04-20T17:11:40.864162+0000","last_change":"2024-04-20T17:10:51.918238+0000","last_active":"2024-04-20T17:10:50.920018+0000","last_peered":"2024-04-20T17:11:40.864162+0000","last_clean":"2024-04-20T17:10:50.881753+0000","last_became_active":"2024-04-20T17:08:06.028272+0000","last_became_peered":"2024-04-20T17:10:51.918238+0000","last_unstale":"2024-04-20T17:11:40.864162+0000","last_undegraded":"2024-04-20T17:10:51.917923+0000","last_fullsized":"2024-04-20T17:10:51.917837+0000","mapping_epoch":170,"log_start":"124'3100","ondisk_log_start":"124'3100","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'9589","last_scrub_stamp":"2024-04-20T17:09:37.406724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:09:37.406724+0000","objects_scrubbed":9589,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:11:41.677043+0000","scrub_duration":886,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450494464,"num_objects":6874,"num_object_clones":0,"num_object_copies":13748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6874,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13122,"num_write_kb":639872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":6874}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":7175405592,"num_objects":109489,"num_object_clones":0,"num_object_copies":218978,"num_objects_missing_on_primary":5827,"num_objects_missing":5827,"num_objects_degraded":19570,"num_objects_misplaced":5828,"num_objects_unfound":0,"num_objects_dirty":109489,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":212495,"num_write_kb":10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16519,"num_bytes_recovered":1082589184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15256133632,"data_stored":15256125488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160795,"ondisk_log_size":160795,"up":30,"acting":30,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991304,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1380284,"kb_used_data":1313008,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92364868,"statfs":{"total":95995035648,"available":94581624832,"internally_reserved":0,"allocated":1344520192,"data_stored":1344078076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":68876740},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777252941,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1527536,"kb_used_data":1379752,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92217616,"statfs":{"total":95995035648,"available":94430838784,"internally_reserved":0,"allocated":1412866048,"data_stored":1412427484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":151321024},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612691,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1381868,"kb_used_data":1314592,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92363284,"statfs":{"total":95995035648,"available":94580002816,"internally_reserved":0,"allocated":1346142208,"data_stored":1345705599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":68876736},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762808921,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1826296,"kb_used_data":1750696,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91918856,"statfs":{"total":95995035648,"available":94124908544,"internally_reserved":0,"allocated":1792712704,"data_stored":1792274160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168670,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1398780,"kb_used_data":1323184,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92346372,"statfs":{"total":95995035648,"available":94562684928,"internally_reserved":0,"allocated":1354940416,"data_stored":1354498320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430306,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":471736,"kb_used_data":436392,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93273416,"statfs":{"total":95995035648,"available":95511977984,"internally_reserved":0,"allocated":446865408,"data_stored":446426864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574972,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1529320,"kb_used_data":1453724,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92215832,"statfs":{"total":95995035648,"available":94429011968,"internally_reserved":0,"allocated":1488613376,"data_stored":1488180839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738497,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1486052,"kb_used_data":1374364,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92259100,"statfs":{"total":95995035648,"available":94473318400,"internally_reserved":0,"allocated":1407348736,"data_stored":1406916199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":20,"seq":85899346027,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1400588,"kb_used_data":1333316,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92344564,"statfs":{"total":95995035648,"available":94560833536,"internally_reserved":0,"allocated":1365315584,"data_stored":1364990388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182513,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1382952,"kb_used_data":1319836,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92362200,"statfs":{"total":95995035648,"available":94578892800,"internally_reserved":0,"allocated":1351512064,"data_stored":1351079527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986284,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":467564,"kb_used_data":436384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93277588,"statfs":{"total":95995035648,"available":95516250112,"internally_reserved":0,"allocated":446857216,"data_stored":446420607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790054,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1575916,"kb_used_data":1482268,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":92169236,"statfs":{"total":95995035648,"available":94381297664,"internally_reserved":0,"allocated":1517842432,"data_stored":1517409895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1406795776,"data_stored":1406795776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1483800576,"data_stored":1483800576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1359609856,"data_stored":1359609856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1350631424,"data_stored":1350631424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":445976576,"data_stored":445972504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1516961792,"data_stored":1516961792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":445972480,"data_stored":445972480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1353515008,"data_stored":1353515008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1791819776,"data_stored":1791819776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1345261568,"data_stored":1345257496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1412694016,"data_stored":1412694016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1343094784,"data_stored":1343094784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:11:46.969 INFO:tasks.thrashosds.thrasher:Setting 2.1 to [9, 6] 2024-04-20T17:11:46.969 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.1', '9', '6'] 2024-04-20T17:11:46.969 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pg-upmap 2.1 9 6 2024-04-20T17:11:46.971 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:11:46.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:46.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:11:47.075 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:11:47.107 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:47.129 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:11:47.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:47.132 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:11:47.233 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:11:47.261 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:47.513 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:11:47.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:47.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:11:47.525 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:11:47.548 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:11:47.620 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:11:47.652 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:47.673 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:11:47.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:47.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:11:47.777 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:11:47.805 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:47.824 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:11:47.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:47.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:11:47.860 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:11:47.927 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:11:47.959 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:47.980 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:11:47.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:47.983 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:11:48.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:47 smithi106 ceph-mon[44423]: pgmap v549: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19508/218982 objects degraded (8.908%); 5765/218982 objects misplaced (2.633%); 672 KiB/s, 10 objects/s recovering 2024-04-20T17:11:48.084 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:11:48.093 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:47 smithi005 ceph-mon[39562]: pgmap v549: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19508/218982 objects degraded (8.908%); 5765/218982 objects misplaced (2.633%); 672 KiB/s, 10 objects/s recovering 2024-04-20T17:11:48.114 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:48.134 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:11:48.136 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:47 smithi171 ceph-mon[44189]: pgmap v549: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19508/218982 objects degraded (8.908%); 5765/218982 objects misplaced (2.633%); 672 KiB/s, 10 objects/s recovering 2024-04-20T17:11:48.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:48.137 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:11:48.238 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:11:48.268 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:48.288 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:11:48.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:48.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:11:48.392 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:11:48.423 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:48.500 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:11:48.523 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:11:48.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:48.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:11:48.529 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:11:48.549 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:11:48.570 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:11:48.591 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:11:48.611 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:11:48.627 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:11:48.631 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:11:48.655 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:11:48.657 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:48.678 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:11:48.699 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:11:48.720 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:11:48.720 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:11:48.720 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:11:48.720 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-20T17:11:48.720 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:11:48.720 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:11:48.720 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:11:48.720 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:11:48.720 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:11:48.721 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:11:48.721 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:11:48.721 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:11:48.721 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:11:49.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:48 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1964468918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:11:49.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:49.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:49.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:49.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:49.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:48 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1964468918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:11:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:49.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:48 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1964468918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:11:49.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:49.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:49.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:49.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:49.508 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:11:49.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:49.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:11:49.611 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:11:49.644 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:49.665 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:11:49.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:49.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:11:49.769 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:11:49.800 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:49.821 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:11:49.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:49.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:11:49.877 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:49 smithi005 ceph-mon[39562]: pgmap v550: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19508/218982 objects degraded (8.908%); 5765/218982 objects misplaced (2.633%); 565 KiB/s, 8 objects/s recovering 2024-04-20T17:11:49.877 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:49.877 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:49.877 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:49.877 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:49.877 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:49.878 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:49.878 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:49.924 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:11:49.955 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:50.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:49 smithi106 ceph-mon[44423]: pgmap v550: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19508/218982 objects degraded (8.908%); 5765/218982 objects misplaced (2.633%); 565 KiB/s, 8 objects/s recovering 2024-04-20T17:11:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:50.241 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:11:50.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:50.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:11:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:49 smithi171 ceph-mon[44189]: pgmap v550: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19508/218982 objects degraded (8.908%); 5765/218982 objects misplaced (2.633%); 565 KiB/s, 8 objects/s recovering 2024-04-20T17:11:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:11:50.346 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:11:50.379 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:50.399 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:11:50.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:50.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:11:50.504 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:11:50.534 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:50.553 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:11:50.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:50.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:11:50.657 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:11:50.689 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:50.710 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:11:50.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:50.714 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:11:50.815 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:11:50.846 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:50 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 19508/218982 objects degraded (8.908%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:51.100 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:50 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 19508/218982 objects degraded (8.908%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:50 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 19508/218982 objects degraded (8.908%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:11:51.500 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:11:51.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:51.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:11:51.529 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:11:51.529 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:11:51.545 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:11:51.605 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:11:51.636 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:51.658 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:11:51.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:51.661 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:11:51.762 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:11:51.792 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:51.812 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:11:51.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:51.815 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:11:51.916 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:11:51.946 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:52.068 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:51 smithi005 ceph-mon[39562]: pgmap v551: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19445/218982 objects degraded (8.880%); 5703/218982 objects misplaced (2.604%); 1000 KiB/s, 15 objects/s recovering 2024-04-20T17:11:52.074 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:11:52.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:52.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:11:52.178 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:11:52.206 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:51 smithi171 ceph-mon[44189]: pgmap v551: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19445/218982 objects degraded (8.880%); 5703/218982 objects misplaced (2.604%); 1000 KiB/s, 15 objects/s recovering 2024-04-20T17:11:52.208 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:52.229 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:11:52.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:52.232 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:11:52.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:51 smithi106 ceph-mon[44423]: pgmap v551: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19445/218982 objects degraded (8.880%); 5703/218982 objects misplaced (2.604%); 1000 KiB/s, 15 objects/s recovering 2024-04-20T17:11:52.333 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:11:52.362 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:52.382 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:11:52.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:52.384 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:11:52.485 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:11:52.516 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:52.536 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:11:52.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:52.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:11:52.640 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:11:52.669 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:52.794 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:11:52.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:52.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:11:52.898 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:11:52.899 INFO:teuthology.orchestra.run.smithi005.stderr:set 2.1 pg_upmap mapping to [9,6] 2024-04-20T17:11:52.926 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:52 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3873092530' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.1", "id": [9, 6]}]: dispatch 2024-04-20T17:11:52.931 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:52.956 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:11:52.956 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:11:53.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:52 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3873092530' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.1", "id": [9, 6]}]: dispatch 2024-04-20T17:11:53.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:52 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3873092530' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.1", "id": [9, 6]}]: dispatch 2024-04-20T17:11:53.530 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:11:53.844 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-20T17:11:54.144 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:53 smithi005 ceph-mon[39562]: pgmap v552: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19445/218982 objects degraded (8.880%); 5703/218982 objects misplaced (2.604%); 895 KiB/s, 13 objects/s recovering 2024-04-20T17:11:54.145 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:53 smithi005 ceph-mon[39562]: Health check failed: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-04-20T17:11:54.145 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:53 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3873092530' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.1", "id": [9, 6]}]': finished 2024-04-20T17:11:54.145 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:53 smithi005 ceph-mon[39562]: osdmap e190: 12 total, 11 up, 12 in 2024-04-20T17:11:54.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:53 smithi106 ceph-mon[44423]: pgmap v552: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19445/218982 objects degraded (8.880%); 5703/218982 objects misplaced (2.604%); 895 KiB/s, 13 objects/s recovering 2024-04-20T17:11:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:53 smithi106 ceph-mon[44423]: Health check failed: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-04-20T17:11:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:53 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3873092530' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.1", "id": [9, 6]}]': finished 2024-04-20T17:11:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:53 smithi106 ceph-mon[44423]: osdmap e190: 12 total, 11 up, 12 in 2024-04-20T17:11:54.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:53 smithi171 ceph-mon[44189]: pgmap v552: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19445/218982 objects degraded (8.880%); 5703/218982 objects misplaced (2.604%); 895 KiB/s, 13 objects/s recovering 2024-04-20T17:11:54.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:53 smithi171 ceph-mon[44189]: Health check failed: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-04-20T17:11:54.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:53 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3873092530' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.1", "id": [9, 6]}]': finished 2024-04-20T17:11:54.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:53 smithi171 ceph-mon[44189]: osdmap e190: 12 total, 11 up, 12 in 2024-04-20T17:11:55.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:54 smithi005 ceph-mon[39562]: osdmap e191: 12 total, 11 up, 12 in 2024-04-20T17:11:55.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:54 smithi005 ceph-mon[39562]: pgmap v555: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19445/218982 objects degraded (8.880%); 5703/218982 objects misplaced (2.604%); 496 KiB/s, 7 objects/s recovering 2024-04-20T17:11:55.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:54 smithi106 ceph-mon[44423]: osdmap e191: 12 total, 11 up, 12 in 2024-04-20T17:11:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:54 smithi106 ceph-mon[44423]: pgmap v555: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19445/218982 objects degraded (8.880%); 5703/218982 objects misplaced (2.604%); 496 KiB/s, 7 objects/s recovering 2024-04-20T17:11:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:54 smithi171 ceph-mon[44189]: osdmap e191: 12 total, 11 up, 12 in 2024-04-20T17:11:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:54 smithi171 ceph-mon[44189]: pgmap v555: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19445/218982 objects degraded (8.880%); 5703/218982 objects misplaced (2.604%); 496 KiB/s, 7 objects/s recovering 2024-04-20T17:11:56.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:55 smithi005 ceph-mon[39562]: osdmap e192: 12 total, 11 up, 12 in 2024-04-20T17:11:56.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:55 smithi005 ceph-mon[39562]: 2.1 starting backfill to osd.6 from (0'0,0'0] MAX to 169'13365 2024-04-20T17:11:56.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:55 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:11:56.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:55 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:11:56.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:55 smithi005 ceph-mon[39562]: 2.1 starting backfill to osd.9 from (0'0,0'0] MAX to 169'13365 2024-04-20T17:11:56.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:55 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 19445/218982 objects degraded (8.880%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-20T17:11:56.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:55 smithi106 ceph-mon[44423]: osdmap e192: 12 total, 11 up, 12 in 2024-04-20T17:11:56.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:55 smithi106 ceph-mon[44423]: 2.1 starting backfill to osd.6 from (0'0,0'0] MAX to 169'13365 2024-04-20T17:11:56.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:55 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:11:56.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:55 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:11:56.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:55 smithi106 ceph-mon[44423]: 2.1 starting backfill to osd.9 from (0'0,0'0] MAX to 169'13365 2024-04-20T17:11:56.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:55 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 19445/218982 objects degraded (8.880%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-20T17:11:56.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:55 smithi171 ceph-mon[44189]: osdmap e192: 12 total, 11 up, 12 in 2024-04-20T17:11:56.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:55 smithi171 ceph-mon[44189]: 2.1 starting backfill to osd.6 from (0'0,0'0] MAX to 169'13365 2024-04-20T17:11:56.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:55 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:11:56.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:55 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:11:56.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:55 smithi171 ceph-mon[44189]: 2.1 starting backfill to osd.9 from (0'0,0'0] MAX to 169'13365 2024-04-20T17:11:56.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:55 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 19445/218982 objects degraded (8.880%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-20T17:11:57.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:56 smithi005 ceph-mon[39562]: osdmap e193: 12 total, 11 up, 12 in 2024-04-20T17:11:57.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:56 smithi005 ceph-mon[39562]: pgmap v558: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19382/218982 objects degraded (8.851%); 5639/218982 objects misplaced (2.575%); 1024 KiB/s, 15 objects/s recovering 2024-04-20T17:11:57.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:56 smithi106 ceph-mon[44423]: osdmap e193: 12 total, 11 up, 12 in 2024-04-20T17:11:57.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:56 smithi106 ceph-mon[44423]: pgmap v558: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19382/218982 objects degraded (8.851%); 5639/218982 objects misplaced (2.575%); 1024 KiB/s, 15 objects/s recovering 2024-04-20T17:11:57.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:56 smithi171 ceph-mon[44189]: osdmap e193: 12 total, 11 up, 12 in 2024-04-20T17:11:57.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:56 smithi171 ceph-mon[44189]: pgmap v558: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19382/218982 objects degraded (8.851%); 5639/218982 objects misplaced (2.575%); 1024 KiB/s, 15 objects/s recovering 2024-04-20T17:11:58.385 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:11:58.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:58.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:11:58.490 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:11:58.517 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:58.537 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:11:58.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:58.540 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:11:58.641 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:11:58.666 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:58.684 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:11:58.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:58.687 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:11:58.788 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:11:58.815 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:58.835 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:11:58.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:58.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:11:58.938 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:11:58.966 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:58.985 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:11:58.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:58.987 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:11:59.088 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:11:59.113 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:11:59 smithi106 ceph-mon[44423]: pgmap v559: 17 pgs: 1 undersized+degraded+peered, 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19093/218982 objects degraded (8.719%); 12219/218982 objects misplaced (5.580%); 4.3 MiB/s, 69 objects/s recovering 2024-04-20T17:11:59.115 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:59.134 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:11:59.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:59.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:11:59.238 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:11:59.262 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:11:59 smithi171 ceph-mon[44189]: pgmap v559: 17 pgs: 1 undersized+degraded+peered, 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19093/218982 objects degraded (8.719%); 12219/218982 objects misplaced (5.580%); 4.3 MiB/s, 69 objects/s recovering 2024-04-20T17:11:59.264 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:59.284 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:11:59.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:59.287 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:11:59.387 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:11:59.414 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:59.425 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:11:59 smithi005 ceph-mon[39562]: pgmap v559: 17 pgs: 1 undersized+degraded+peered, 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19093/218982 objects degraded (8.719%); 12219/218982 objects misplaced (5.580%); 4.3 MiB/s, 69 objects/s recovering 2024-04-20T17:11:59.433 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:11:59.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:59.436 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:11:59.537 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:11:59.564 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:59.584 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:11:59.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:59.588 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:11:59.689 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:11:59.714 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:59.733 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:11:59.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:59.735 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:11:59.836 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:11:59.863 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:11:59.882 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:11:59.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:11:59.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:11:59.986 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:12:00.012 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:00.031 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:12:00.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:00.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:12:00.135 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:12:00.159 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:00 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:12:00.159 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:00 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:12:00.159 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:00 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:12:00.159 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:00 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:12:00.159 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:00 smithi106 ceph-mon[44423]: osdmap e194: 12 total, 11 up, 12 in 2024-04-20T17:12:00.161 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:00.180 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:12:00.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:00.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:12:00.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:00 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:12:00.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:00 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:12:00.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:00 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:12:00.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:00 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:12:00.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:00 smithi005 ceph-mon[39562]: osdmap e194: 12 total, 11 up, 12 in 2024-04-20T17:12:00.263 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:12:00.284 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:12:00.312 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:12:00.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:00 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:12:00.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:00 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:12:00.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:00 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:12:00.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:00 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:12:00.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:00 smithi171 ceph-mon[44189]: osdmap e194: 12 total, 11 up, 12 in 2024-04-20T17:12:00.334 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:00.989 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:12:00.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:00.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:12:01.094 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:12:01.097 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:01 smithi005 ceph-mon[39562]: pgmap v561: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19031/218982 objects degraded (8.691%); 12158/218982 objects misplaced (5.552%); 4.3 MiB/s, 69 objects/s recovering 2024-04-20T17:12:01.097 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:01 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 19031/218982 objects degraded (8.691%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-04-20T17:12:01.121 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:01.141 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:12:01.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:01.143 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:12:01.246 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:12:01.273 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:01.292 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:12:01.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:01.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:12:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:01 smithi171 ceph-mon[44189]: pgmap v561: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19031/218982 objects degraded (8.691%); 12158/218982 objects misplaced (5.552%); 4.3 MiB/s, 69 objects/s recovering 2024-04-20T17:12:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:01 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 19031/218982 objects degraded (8.691%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-04-20T17:12:01.395 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:12:01.419 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:01 smithi106 ceph-mon[44423]: pgmap v561: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19031/218982 objects degraded (8.691%); 12158/218982 objects misplaced (5.552%); 4.3 MiB/s, 69 objects/s recovering 2024-04-20T17:12:01.419 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:01 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 19031/218982 objects degraded (8.691%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-04-20T17:12:01.421 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:01.440 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:12:01.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:01.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:12:01.544 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:12:01.578 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:01.782 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:12:01.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:01.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:12:01.887 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:12:01.915 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:01.935 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:12:01.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:01.938 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:12:02.039 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:12:02.073 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:02.725 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:12:02.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:02.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:12:02.829 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:12:02.855 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:02.874 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:12:02.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:02.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:12:02.978 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:12:03.011 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:03.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:03 smithi106 ceph-mon[44423]: pgmap v562: 17 pgs: 1 undersized+degraded+peered, 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15929/218982 objects degraded (7.274%); 9056/218982 objects misplaced (4.135%); 30 MiB/s, 486 objects/s recovering 2024-04-20T17:12:03.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:03 smithi171 ceph-mon[44189]: pgmap v562: 17 pgs: 1 undersized+degraded+peered, 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15929/218982 objects degraded (7.274%); 9056/218982 objects misplaced (4.135%); 30 MiB/s, 486 objects/s recovering 2024-04-20T17:12:03.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:03 smithi005 ceph-mon[39562]: pgmap v562: 17 pgs: 1 undersized+degraded+peered, 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 14 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15929/218982 objects degraded (7.274%); 9056/218982 objects misplaced (4.135%); 30 MiB/s, 486 objects/s recovering 2024-04-20T17:12:03.866 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:12:03.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:03.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:12:03.972 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:12:04.006 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:04.047 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:12:04.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:04.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:12:04.077 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:12:04.151 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:12:04.178 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:04.198 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:12:04.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:04.201 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:12:04.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:04 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2756001980' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:12:04.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:04 smithi005 ceph-mon[39562]: from='client.24983 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:12:04.212 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:12:04.213 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:12:04.301 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:12:04.326 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:04 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2756001980' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:12:04.326 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:04 smithi171 ceph-mon[44189]: from='client.24983 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:12:04.328 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:04.348 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:12:04.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:04.350 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:12:04.451 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:12:04.458 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-20T17:12:04.458 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:12:04.458 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-04-20T17:12:04.458 INFO:tasks.cephadm.osd.2:Restarting osd.2 (starting--it wasn't running)... 2024-04-20T17:12:04.458 DEBUG:teuthology.orchestra.run.smithi005:osd.2> sudo journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.2.service 2024-04-20T17:12:04.461 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl start ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.2 2024-04-20T17:12:04.478 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:04.495 INFO:journalctl@ceph.osd.2.smithi005.stdout:-- Logs begin at Sat 2024-04-20 16:47:38 UTC. -- 2024-04-20T17:12:04.497 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:12:04.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:04.500 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:12:04.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:04 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2756001980' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:12:04.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:04 smithi106 ceph-mon[44423]: from='client.24983 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:12:04.601 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:12:04.628 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:04.646 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:12:04.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:04.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:12:04.750 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:12:04.776 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:04.796 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:12:04.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:04.799 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:12:04.900 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:12:04.931 INFO:journalctl@ceph.osd.2.smithi005.stdout:Apr 20 17:12:04 smithi005 systemd[1]: Starting Ceph osd.2 for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:12:04.935 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:05.189 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:12:05.192 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:05 smithi005 ceph-mon[39562]: pgmap v563: 17 pgs: 1 undersized+degraded+peered, 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 14 active+clean; 6.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 15929/218982 objects degraded (7.274%); 9056/218982 objects misplaced (4.135%); 27 MiB/s, 427 objects/s recovering 2024-04-20T17:12:05.192 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:05 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:12:05.192 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:05 smithi005 ceph-mon[39562]: from='client.24983 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:12:05.192 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:05 smithi005 ceph-mon[39562]: osdmap e195: 12 total, 11 up, 12 in 2024-04-20T17:12:05.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:05.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:12:05.294 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:12:05.318 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:05 smithi171 ceph-mon[44189]: pgmap v563: 17 pgs: 1 undersized+degraded+peered, 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 14 active+clean; 6.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 15929/218982 objects degraded (7.274%); 9056/218982 objects misplaced (4.135%); 27 MiB/s, 427 objects/s recovering 2024-04-20T17:12:05.318 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:05 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:12:05.319 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:05 smithi171 ceph-mon[44189]: from='client.24983 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:12:05.319 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:05 smithi171 ceph-mon[44189]: osdmap e195: 12 total, 11 up, 12 in 2024-04-20T17:12:05.320 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:05.339 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:12:05.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:05.342 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:12:05.443 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:12:05.478 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:05.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:05 smithi106 ceph-mon[44423]: pgmap v563: 17 pgs: 1 undersized+degraded+peered, 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 14 active+clean; 6.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 15929/218982 objects degraded (7.274%); 9056/218982 objects misplaced (4.135%); 27 MiB/s, 427 objects/s recovering 2024-04-20T17:12:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:05 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:12:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:05 smithi106 ceph-mon[44423]: from='client.24983 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:12:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:05 smithi106 ceph-mon[44423]: osdmap e195: 12 total, 11 up, 12 in 2024-04-20T17:12:05.732 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:12:05.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:05.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:12:05.836 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:12:05.871 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:06.104 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:12:06.104 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:12:06.104 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:12:06.118 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:12:06.140 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:12:06.164 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:12:06.194 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:12:06.223 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:12:06.250 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:12:06.274 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:12:06.297 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:12:06.297 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:12:06.297 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:12:06.297 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-20T17:12:06.297 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:12:06.297 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:12:06.297 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:12:06.297 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:12:06.297 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:12:06.297 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:12:06.298 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:12:06.298 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:12:06.298 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:12:06.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:06 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 15929/218982 objects degraded (7.274%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-04-20T17:12:06.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:06 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2808040610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:12:06.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:06 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 15929/218982 objects degraded (7.274%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-04-20T17:12:06.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:06 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2808040610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:12:06.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:06 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 15929/218982 objects degraded (7.274%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-04-20T17:12:06.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:06 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2808040610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:12:06.615 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-20T17:12:07.036 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:12:07.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:07.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:12:07.146 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:12:07.173 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:07.192 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:12:07.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:07.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:12:07.196 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:12:07.295 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:12:07.322 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:07 smithi005 ceph-mon[39562]: pgmap v565: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 14 active+clean; 6.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 15875/218982 objects degraded (7.249%); 9001/218982 objects misplaced (4.110%); 25 MiB/s, 402 objects/s recovering 2024-04-20T17:12:07.322 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:07.322 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:07.322 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:07.322 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:07.322 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:07.322 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:07.322 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:07.322 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:07.323 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:07.324 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:07 smithi171 ceph-mon[44189]: pgmap v565: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 14 active+clean; 6.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 15875/218982 objects degraded (7.249%); 9001/218982 objects misplaced (4.110%); 25 MiB/s, 402 objects/s recovering 2024-04-20T17:12:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:07.378 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:12:07.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:07.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:12:07.410 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:12:07.410 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:12:07.421 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:12:07.485 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:12:07.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:07 smithi106 ceph-mon[44423]: pgmap v565: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 14 active+clean; 6.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 15875/218982 objects degraded (7.249%); 9001/218982 objects misplaced (4.110%); 25 MiB/s, 402 objects/s recovering 2024-04-20T17:12:07.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:07.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:07.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:07.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:07.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:07.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:07.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:07.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:07.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:07.514 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:07.715 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:12:07.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:07.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:12:07.820 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:12:07.849 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:07.869 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:12:07.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:07.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:12:07.973 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:12:08.001 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:08.020 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:12:08.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:08.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:12:08.124 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:12:08.149 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:08.149 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:08.149 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:08 smithi106 ceph-mon[44423]: osdmap e196: 12 total, 11 up, 12 in 2024-04-20T17:12:08.151 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:08.172 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:12:08.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:08.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:12:08.276 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:12:08.302 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:08.321 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:12:08.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:08.324 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:12:08.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:08 smithi171 ceph-mon[44189]: osdmap e196: 12 total, 11 up, 12 in 2024-04-20T17:12:08.424 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:12:08.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:08.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:08.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:08 smithi005 ceph-mon[39562]: osdmap e196: 12 total, 11 up, 12 in 2024-04-20T17:12:08.460 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:08.711 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:12:08.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:08.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:12:08.742 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:12:08.818 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:12:08.848 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:09.058 INFO:journalctl@ceph.osd.2.smithi005.stdout:Apr 20 17:12:08 smithi005 bash[118327]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T17:12:09.058 INFO:journalctl@ceph.osd.2.smithi005.stdout:Apr 20 17:12:08 smithi005 bash[118327]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-20T17:12:09.058 INFO:journalctl@ceph.osd.2.smithi005.stdout:Apr 20 17:12:08 smithi005 bash[118327]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-20T17:12:09.058 INFO:journalctl@ceph.osd.2.smithi005.stdout:Apr 20 17:12:08 smithi005 bash[118327]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-20T17:12:09.058 INFO:journalctl@ceph.osd.2.smithi005.stdout:Apr 20 17:12:08 smithi005 bash[118327]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-20T17:12:09.058 INFO:journalctl@ceph.osd.2.smithi005.stdout:Apr 20 17:12:08 smithi005 bash[118327]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-20T17:12:09.058 INFO:journalctl@ceph.osd.2.smithi005.stdout:Apr 20 17:12:08 smithi005 bash[118327]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-20T17:12:09.058 INFO:journalctl@ceph.osd.2.smithi005.stdout:Apr 20 17:12:08 smithi005 bash[118327]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-20T17:12:09.300 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:12:09.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:09.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:12:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:09 smithi171 ceph-mon[44189]: pgmap v567: 17 pgs: 1 undersized+degraded+peered, 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 14 active+clean; 6.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 12636/218982 objects degraded (5.770%); 5762/218982 objects misplaced (2.631%); 50 MiB/s, 799 objects/s recovering 2024-04-20T17:12:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:09 smithi171 ceph-mon[44189]: osdmap e197: 12 total, 11 up, 12 in 2024-04-20T17:12:09.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:09 smithi005 ceph-mon[39562]: pgmap v567: 17 pgs: 1 undersized+degraded+peered, 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 14 active+clean; 6.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 12636/218982 objects degraded (5.770%); 5762/218982 objects misplaced (2.631%); 50 MiB/s, 799 objects/s recovering 2024-04-20T17:12:09.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:09 smithi005 ceph-mon[39562]: osdmap e197: 12 total, 11 up, 12 in 2024-04-20T17:12:09.405 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:12:09.411 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:12:09.436 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:09.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:09 smithi106 ceph-mon[44423]: pgmap v567: 17 pgs: 1 undersized+degraded+peered, 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 14 active+clean; 6.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 12636/218982 objects degraded (5.770%); 5762/218982 objects misplaced (2.631%); 50 MiB/s, 799 objects/s recovering 2024-04-20T17:12:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:09 smithi106 ceph-mon[44423]: osdmap e197: 12 total, 11 up, 12 in 2024-04-20T17:12:10.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:10 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:12:10.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:10 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:12:10.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:10 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:12:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:10 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:12:10.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:10 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:12:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:10 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:12:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:11 smithi171 ceph-mon[44189]: pgmap v569: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 14 active+clean; 6.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 4 op/s; 12584/218922 objects degraded (5.748%); 5711/218922 objects misplaced (2.609%); 35 MiB/s, 557 objects/s recovering 2024-04-20T17:12:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:11 smithi171 ceph-mon[44189]: osdmap e198: 12 total, 11 up, 12 in 2024-04-20T17:12:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:11 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:12:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:11 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 12584/218922 objects degraded (5.748%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-04-20T17:12:11.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:11 smithi005 ceph-mon[39562]: pgmap v569: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 14 active+clean; 6.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 4 op/s; 12584/218922 objects degraded (5.748%); 5711/218922 objects misplaced (2.609%); 35 MiB/s, 557 objects/s recovering 2024-04-20T17:12:11.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:11 smithi005 ceph-mon[39562]: osdmap e198: 12 total, 11 up, 12 in 2024-04-20T17:12:11.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:11 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:12:11.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:11 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 12584/218922 objects degraded (5.748%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-04-20T17:12:11.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:11 smithi106 ceph-mon[44423]: pgmap v569: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 14 active+clean; 6.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 4 op/s; 12584/218922 objects degraded (5.748%); 5711/218922 objects misplaced (2.609%); 35 MiB/s, 557 objects/s recovering 2024-04-20T17:12:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:11 smithi106 ceph-mon[44423]: osdmap e198: 12 total, 11 up, 12 in 2024-04-20T17:12:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:11 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:12:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:11 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 12584/218922 objects degraded (5.748%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-04-20T17:12:12.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:12 smithi005 ceph-mon[39562]: Health check update: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-20T17:12:12.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:12 smithi106 ceph-mon[44423]: Health check update: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-20T17:12:12.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:12 smithi171 ceph-mon[44189]: Health check update: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-20T17:12:13.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:13 smithi005 ceph-mon[39562]: pgmap v571: 17 pgs: 1 undersized+degraded+peered, 1 active+recovering+degraded, 15 active+clean; 6.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 14 op/s; 12345/218806 objects degraded (5.642%); 5472/218806 objects misplaced (2.501%); 37 MiB/s, 587 objects/s recovering 2024-04-20T17:12:13.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:13 smithi106 ceph-mon[44423]: pgmap v571: 17 pgs: 1 undersized+degraded+peered, 1 active+recovering+degraded, 15 active+clean; 6.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 14 op/s; 12345/218806 objects degraded (5.642%); 5472/218806 objects misplaced (2.501%); 37 MiB/s, 587 objects/s recovering 2024-04-20T17:12:13.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:13 smithi171 ceph-mon[44189]: pgmap v571: 17 pgs: 1 undersized+degraded+peered, 1 active+recovering+degraded, 15 active+clean; 6.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 14 op/s; 12345/218806 objects degraded (5.642%); 5472/218806 objects misplaced (2.501%); 37 MiB/s, 587 objects/s recovering 2024-04-20T17:12:14.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:14 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:12:14.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:14 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:12:14.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:14 smithi005 ceph-mon[39562]: osdmap e199: 12 total, 11 up, 12 in 2024-04-20T17:12:14.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:14 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:12:14.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:14 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:12:14.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:14 smithi106 ceph-mon[44423]: osdmap e199: 12 total, 11 up, 12 in 2024-04-20T17:12:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:14 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:12:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:14 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:12:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:14 smithi171 ceph-mon[44189]: osdmap e199: 12 total, 11 up, 12 in 2024-04-20T17:12:15.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:15 smithi005 ceph-mon[39562]: pgmap v573: 17 pgs: 1 undersized+degraded+peered, 1 active+recovering+degraded, 15 active+clean; 6.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 25 op/s; 12345/218676 objects degraded (5.645%); 5472/218676 objects misplaced (2.502%); 3.0 MiB/s, 48 objects/s recovering 2024-04-20T17:12:15.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:15 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:12:15.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:15 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:12:15.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:15 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:12:15.474 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:12:15.474 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:12:15.474 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-04-20T17:12:15.474 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:12:15.491 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:12:15.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:15.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:12:15.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:15 smithi106 ceph-mon[44423]: pgmap v573: 17 pgs: 1 undersized+degraded+peered, 1 active+recovering+degraded, 15 active+clean; 6.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 25 op/s; 12345/218676 objects degraded (5.645%); 5472/218676 objects misplaced (2.502%); 3.0 MiB/s, 48 objects/s recovering 2024-04-20T17:12:15.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:15 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:12:15.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:15 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:12:15.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:15 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:12:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:15 smithi171 ceph-mon[44189]: pgmap v573: 17 pgs: 1 undersized+degraded+peered, 1 active+recovering+degraded, 15 active+clean; 6.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 25 op/s; 12345/218676 objects degraded (5.645%); 5472/218676 objects misplaced (2.502%); 3.0 MiB/s, 48 objects/s recovering 2024-04-20T17:12:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:15 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:12:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:15 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:12:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:15 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:12:15.599 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:12:15.630 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:15.653 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:12:15.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:15.657 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:12:15.757 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:12:15.790 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:15.812 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:12:15.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:15.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:12:15.915 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:12:15.950 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:15.989 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:12:15.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:15.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:12:16.093 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:12:16.125 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:16.147 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:12:16.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:16.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:12:16.198 INFO:journalctl@ceph.osd.2.smithi005.stdout:Apr 20 17:12:16 smithi005 ceph-osd[119165]: -- 172.21.15.5:0/590192089 <== mon.0 v1:172.21.15.5:6789/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (unknown 2629309483 0 0) 0x555f81e35680 con 0x555f81dff400 2024-04-20T17:12:16.198 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:16 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 12345/218676 objects degraded (5.645%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-04-20T17:12:16.203 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-20T17:12:16.250 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:12:16.280 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:16.451 INFO:journalctl@ceph.osd.2.smithi005.stdout:Apr 20 17:12:16 smithi005 bash[119068]: de6c5732c255f6e98b1f74fdb91665d0a00761114847b71e346d1bb7121cc9bf 2024-04-20T17:12:16.452 INFO:journalctl@ceph.osd.2.smithi005.stdout:Apr 20 17:12:16 smithi005 systemd[1]: Started Ceph osd.2 for 13c6e76c-ff37-11ee-bc93-c7b262605968. 2024-04-20T17:12:16.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:16 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 12345/218676 objects degraded (5.645%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-04-20T17:12:16.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:16 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 12345/218676 objects degraded (5.645%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-04-20T17:12:16.711 INFO:journalctl@ceph.osd.2.smithi005.stdout:Apr 20 17:12:16 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-2[119106]: 2024-04-20T17:12:16.667+0000 7faa55b3e7c0 -1 Falling back to public interface 2024-04-20T17:12:16.931 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:12:16.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:16.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:12:16.959 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-20T17:12:17.037 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:12:17.070 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:17.091 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:12:17.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:17.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:12:17.195 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:12:17.228 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:17.250 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:12:17.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:17.253 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:12:17.354 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:12:17.389 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:17.414 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:12:17.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:17.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:12:17.425 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:12:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:12:17.521 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:12:17.549 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:17.699 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:12:17.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:17.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:12:17.804 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:12:17.836 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:17.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:17 smithi005 ceph-mon[39562]: pgmap v574: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+peered, 15 active+clean; 6.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 19 op/s; 12275/218676 objects degraded (5.613%); 5402/218676 objects misplaced (2.470%); 2.8 MiB/s, 45 objects/s recovering 2024-04-20T17:12:18.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:17 smithi106 ceph-mon[44423]: pgmap v574: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+peered, 15 active+clean; 6.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 19 op/s; 12275/218676 objects degraded (5.613%); 5402/218676 objects misplaced (2.470%); 2.8 MiB/s, 45 objects/s recovering 2024-04-20T17:12:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:17 smithi171 ceph-mon[44189]: pgmap v574: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+peered, 15 active+clean; 6.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 19 op/s; 12275/218676 objects degraded (5.613%); 5402/218676 objects misplaced (2.470%); 2.8 MiB/s, 45 objects/s recovering 2024-04-20T17:12:18.884 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:12:18.909 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:18 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3717080650' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:12:19.057 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:12:19.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:18 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3717080650' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:12:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:18 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3717080650' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:12:19.387 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:12:19.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:19.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:12:19.493 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:12:19.525 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:19.546 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:12:19.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:19.548 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:12:19.649 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:12:19.682 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:19.704 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:12:19.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:19.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:12:19.808 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:12:19.838 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:19.858 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:12:19.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:19.860 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:12:19.961 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:12:19.991 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:19 smithi106 ceph-mon[44423]: pgmap v575: 17 pgs: 1 undersized+degraded+peered, 1 active+recovering+degraded, 15 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 15 op/s; 12275/218676 objects degraded (5.613%); 5402/218676 objects misplaced (2.470%); 2.4 MiB/s, 38 objects/s recovering 2024-04-20T17:12:19.992 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:19 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:12:19.992 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:19 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3717080650' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:12:19.992 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:19 smithi106 ceph-mon[44423]: osdmap e200: 12 total, 11 up, 12 in 2024-04-20T17:12:19.994 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:20.018 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:12:20.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:20.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:12:20.120 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:12:20.148 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:19 smithi171 ceph-mon[44189]: pgmap v575: 17 pgs: 1 undersized+degraded+peered, 1 active+recovering+degraded, 15 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 15 op/s; 12275/218676 objects degraded (5.613%); 5402/218676 objects misplaced (2.470%); 2.4 MiB/s, 38 objects/s recovering 2024-04-20T17:12:20.148 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:19 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:12:20.148 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:19 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3717080650' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:12:20.148 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:19 smithi171 ceph-mon[44189]: osdmap e200: 12 total, 11 up, 12 in 2024-04-20T17:12:20.150 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:20.170 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:12:20.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:20.173 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:12:20.197 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:19 smithi005 ceph-mon[39562]: pgmap v575: 17 pgs: 1 undersized+degraded+peered, 1 active+recovering+degraded, 15 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 15 op/s; 12275/218676 objects degraded (5.613%); 5402/218676 objects misplaced (2.470%); 2.4 MiB/s, 38 objects/s recovering 2024-04-20T17:12:20.197 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:19 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:12:20.197 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:19 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3717080650' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:12:20.197 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:19 smithi005 ceph-mon[39562]: osdmap e200: 12 total, 11 up, 12 in 2024-04-20T17:12:20.274 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:12:20.302 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:20.321 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:12:20.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:20.324 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:12:20.425 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:12:20.459 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:20.461 INFO:journalctl@ceph.osd.2.smithi005.stdout:Apr 20 17:12:20 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-2[119106]: 2024-04-20T17:12:20.195+0000 7faa55b3e7c0 -1 osd.2 169 log_to_monitors true 2024-04-20T17:12:20.479 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:12:20.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:20.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:12:20.583 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:12:20.614 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:20.634 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:12:20.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:20.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:12:20.737 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:12:20.773 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:21.195 INFO:journalctl@ceph.osd.2.smithi005.stdout:Apr 20 17:12:20 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-2[119106]: 2024-04-20T17:12:20.889+0000 7faa46f96700 -1 osd.2 169 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T17:12:21.195 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:20 smithi005 ceph-mon[39562]: from='osd.2 v1:172.21.15.5:6809/2120275501' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-20T17:12:21.195 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:20 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 12275/218676 objects degraded (5.613%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-04-20T17:12:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:20 smithi106 ceph-mon[44423]: from='osd.2 v1:172.21.15.5:6809/2120275501' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-20T17:12:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:20 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 12275/218676 objects degraded (5.613%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-04-20T17:12:21.323 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:12:21.324 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:12:21.324 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:12:21.324 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:12:21.324 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:12:21.324 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:12:21.324 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:12:21.325 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:12:21.325 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:12:21.325 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:12:21.325 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:12:21.326 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:12:21.326 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:12:21.326 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-20T17:12:21.326 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:12:21.326 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:12:21.326 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:12:21.327 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:12:21.327 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:12:21.327 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:12:21.327 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:12:21.327 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:12:21.327 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:12:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:20 smithi171 ceph-mon[44189]: from='osd.2 v1:172.21.15.5:6809/2120275501' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-20T17:12:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:20 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 12275/218676 objects degraded (5.613%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-04-20T17:12:21.523 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:12:21.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:21.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:12:21.626 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:12:21.653 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:21.672 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:12:21.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:21.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:12:21.776 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:12:21.802 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:21.820 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:12:21.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:21.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:12:21.924 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:12:21.951 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:21.969 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:12:21.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:21.972 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:12:22.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:21 smithi106 ceph-mon[44423]: pgmap v577: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+peered, 15 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 8 op/s; 12210/218676 objects degraded (5.584%); 5337/218676 objects misplaced (2.441%); 1.1 MiB/s, 16 objects/s recovering 2024-04-20T17:12:22.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:21 smithi106 ceph-mon[44423]: 2.7 scrub starts 2024-04-20T17:12:22.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:21 smithi106 ceph-mon[44423]: 2.0 scrub starts 2024-04-20T17:12:22.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:21 smithi106 ceph-mon[44423]: from='osd.2 v1:172.21.15.5:6809/2120275501' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-20T17:12:22.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:21 smithi106 ceph-mon[44423]: osdmap e201: 12 total, 11 up, 12 in 2024-04-20T17:12:22.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:21 smithi106 ceph-mon[44423]: from='osd.2 v1:172.21.15.5:6809/2120275501' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:12:22.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:21 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2386590106' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:12:22.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:21 smithi106 ceph-mon[44423]: 2.7 scrub ok 2024-04-20T17:12:22.073 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:12:22.100 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:22.113 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:12:22.114 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:12:22.119 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:12:22.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:22.122 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:12:22.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:21 smithi005 ceph-mon[39562]: pgmap v577: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+peered, 15 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 8 op/s; 12210/218676 objects degraded (5.584%); 5337/218676 objects misplaced (2.441%); 1.1 MiB/s, 16 objects/s recovering 2024-04-20T17:12:22.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:21 smithi005 ceph-mon[39562]: 2.7 scrub starts 2024-04-20T17:12:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:21 smithi005 ceph-mon[39562]: 2.0 scrub starts 2024-04-20T17:12:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:21 smithi005 ceph-mon[39562]: from='osd.2 v1:172.21.15.5:6809/2120275501' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-20T17:12:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:21 smithi005 ceph-mon[39562]: osdmap e201: 12 total, 11 up, 12 in 2024-04-20T17:12:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:21 smithi005 ceph-mon[39562]: from='osd.2 v1:172.21.15.5:6809/2120275501' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:12:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:21 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2386590106' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:12:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:21 smithi005 ceph-mon[39562]: 2.7 scrub ok 2024-04-20T17:12:22.223 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:12:22.247 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:21 smithi171 ceph-mon[44189]: pgmap v577: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+peered, 15 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 8 op/s; 12210/218676 objects degraded (5.584%); 5337/218676 objects misplaced (2.441%); 1.1 MiB/s, 16 objects/s recovering 2024-04-20T17:12:22.247 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:21 smithi171 ceph-mon[44189]: 2.7 scrub starts 2024-04-20T17:12:22.248 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:21 smithi171 ceph-mon[44189]: 2.0 scrub starts 2024-04-20T17:12:22.248 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:21 smithi171 ceph-mon[44189]: from='osd.2 v1:172.21.15.5:6809/2120275501' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-20T17:12:22.248 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:21 smithi171 ceph-mon[44189]: osdmap e201: 12 total, 11 up, 12 in 2024-04-20T17:12:22.248 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:21 smithi171 ceph-mon[44189]: from='osd.2 v1:172.21.15.5:6809/2120275501' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:12:22.248 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:21 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2386590106' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:12:22.248 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.248 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.248 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.248 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.248 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.248 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.248 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.248 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.248 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.248 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.249 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:22.249 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:21 smithi171 ceph-mon[44189]: 2.7 scrub ok 2024-04-20T17:12:22.249 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:22.267 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:12:22.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:22.269 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:12:22.370 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:12:22.397 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:22.421 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:12:22.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:22.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:12:22.524 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:12:22.556 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:22.657 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:12:22.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:22.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:12:22.762 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:12:22.789 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:22.808 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:12:22.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:22.811 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:12:22.913 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:12:22.940 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:22.959 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:12:22.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:22.961 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:12:23.062 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:12:23.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:22 smithi171 ceph-mon[44189]: 2.0 scrub ok 2024-04-20T17:12:23.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:22 smithi171 ceph-mon[44189]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T17:12:23.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:22 smithi171 ceph-mon[44189]: osd.2 v1:172.21.15.5:6809/2120275501 boot 2024-04-20T17:12:23.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:22 smithi171 ceph-mon[44189]: osdmap e202: 12 total, 12 up, 12 in 2024-04-20T17:12:23.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:22 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T17:12:23.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:22 smithi171 ceph-mon[44189]: 2.4 scrub starts 2024-04-20T17:12:23.088 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:23.107 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:12:23.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:23.110 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:12:23.211 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:12:23.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:22 smithi005 ceph-mon[39562]: 2.0 scrub ok 2024-04-20T17:12:23.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:22 smithi005 ceph-mon[39562]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T17:12:23.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:22 smithi005 ceph-mon[39562]: osd.2 v1:172.21.15.5:6809/2120275501 boot 2024-04-20T17:12:23.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:22 smithi005 ceph-mon[39562]: osdmap e202: 12 total, 12 up, 12 in 2024-04-20T17:12:23.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:22 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T17:12:23.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:22 smithi005 ceph-mon[39562]: 2.4 scrub starts 2024-04-20T17:12:23.236 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:22 smithi106 ceph-mon[44423]: 2.0 scrub ok 2024-04-20T17:12:23.236 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:22 smithi106 ceph-mon[44423]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T17:12:23.236 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:22 smithi106 ceph-mon[44423]: osd.2 v1:172.21.15.5:6809/2120275501 boot 2024-04-20T17:12:23.236 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:22 smithi106 ceph-mon[44423]: osdmap e202: 12 total, 12 up, 12 in 2024-04-20T17:12:23.236 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:22 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-20T17:12:23.236 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:22 smithi106 ceph-mon[44423]: 2.4 scrub starts 2024-04-20T17:12:23.238 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:23.256 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:12:23.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:23.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:12:23.360 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:12:23.386 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:23.405 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:12:23.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:23.408 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:12:23.509 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:12:23.542 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:23.647 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:12:23.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:23.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:12:23.753 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:12:23.784 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:24.115 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:12:24.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:24.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:23 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:12:24.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:23 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:12:24.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:23 smithi005 ceph-mon[39562]: pgmap v580: 17 pgs: 1 undersized+degraded+peered, 1 active+recovering+degraded, 15 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12210/218676 objects degraded (5.584%); 5337/218676 objects misplaced (2.441%); 693 KiB/s, 10 objects/s recovering 2024-04-20T17:12:24.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:23 smithi005 ceph-mon[39562]: 2.4 scrub ok 2024-04-20T17:12:24.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:23 smithi005 ceph-mon[39562]: osdmap e203: 12 total, 12 up, 12 in 2024-04-20T17:12:24.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:23 smithi005 ceph-mon[39562]: 2.2 scrub starts 2024-04-20T17:12:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:23 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:12:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:23 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:12:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:23 smithi106 ceph-mon[44423]: pgmap v580: 17 pgs: 1 undersized+degraded+peered, 1 active+recovering+degraded, 15 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12210/218676 objects degraded (5.584%); 5337/218676 objects misplaced (2.441%); 693 KiB/s, 10 objects/s recovering 2024-04-20T17:12:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:23 smithi106 ceph-mon[44423]: 2.4 scrub ok 2024-04-20T17:12:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:23 smithi106 ceph-mon[44423]: osdmap e203: 12 total, 12 up, 12 in 2024-04-20T17:12:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:23 smithi106 ceph-mon[44423]: 2.2 scrub starts 2024-04-20T17:12:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:23 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:12:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:23 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:12:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:23 smithi171 ceph-mon[44189]: pgmap v580: 17 pgs: 1 undersized+degraded+peered, 1 active+recovering+degraded, 15 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12210/218676 objects degraded (5.584%); 5337/218676 objects misplaced (2.441%); 693 KiB/s, 10 objects/s recovering 2024-04-20T17:12:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:23 smithi171 ceph-mon[44189]: 2.4 scrub ok 2024-04-20T17:12:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:23 smithi171 ceph-mon[44189]: osdmap e203: 12 total, 12 up, 12 in 2024-04-20T17:12:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:23 smithi171 ceph-mon[44189]: 2.2 scrub starts 2024-04-20T17:12:24.339 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:12:24.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:24.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:12:24.446 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:12:24.472 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:24.490 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:12:24.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:24.493 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:12:24.594 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:12:24.620 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:24.638 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:12:24.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:24.641 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:12:24.742 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:12:24.768 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:24.786 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:12:24.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:24.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:12:24.890 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:12:24.902 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:24 smithi005 ceph-mon[39562]: 2.5 scrub starts 2024-04-20T17:12:24.902 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:24 smithi005 ceph-mon[39562]: 2.2 scrub ok 2024-04-20T17:12:24.902 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:24 smithi005 ceph-mon[39562]: 2.5 scrub ok 2024-04-20T17:12:24.926 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:25.031 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:12:25.031 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:12:25.031 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-04-20T17:12:25.031 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:12:25.059 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:12:25.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:25.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:12:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:24 smithi171 ceph-mon[44189]: 2.5 scrub starts 2024-04-20T17:12:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:24 smithi171 ceph-mon[44189]: 2.2 scrub ok 2024-04-20T17:12:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:24 smithi171 ceph-mon[44189]: 2.5 scrub ok 2024-04-20T17:12:25.164 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:12:25.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:24 smithi106 ceph-mon[44423]: 2.5 scrub starts 2024-04-20T17:12:25.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:24 smithi106 ceph-mon[44423]: 2.2 scrub ok 2024-04-20T17:12:25.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:24 smithi106 ceph-mon[44423]: 2.5 scrub ok 2024-04-20T17:12:25.197 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:25.219 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:12:25.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:25.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:12:25.326 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:12:25.355 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:25.391 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:12:25.391 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:12:25.676 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:12:25.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:25 smithi005 ceph-mon[39562]: pgmap v582: 17 pgs: 1 undersized+degraded+peered, 1 active+recovering+degraded, 15 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 4 op/s; 12210/218628 objects degraded (5.585%); 5337/218628 objects misplaced (2.441%); 813 KiB/s, 12 objects/s recovering 2024-04-20T17:12:25.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:25 smithi005 ceph-mon[39562]: 2.e scrub starts 2024-04-20T17:12:25.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:25 smithi005 ceph-mon[39562]: 2.e scrub ok 2024-04-20T17:12:25.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:25 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 12210/218628 objects degraded (5.585%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-04-20T17:12:25.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:25 smithi005 ceph-mon[39562]: osdmap e204: 12 total, 12 up, 12 in 2024-04-20T17:12:26.242 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-20T17:12:26.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:25 smithi106 ceph-mon[44423]: pgmap v582: 17 pgs: 1 undersized+degraded+peered, 1 active+recovering+degraded, 15 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 4 op/s; 12210/218628 objects degraded (5.585%); 5337/218628 objects misplaced (2.441%); 813 KiB/s, 12 objects/s recovering 2024-04-20T17:12:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:25 smithi106 ceph-mon[44423]: 2.e scrub starts 2024-04-20T17:12:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:25 smithi106 ceph-mon[44423]: 2.e scrub ok 2024-04-20T17:12:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:25 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 12210/218628 objects degraded (5.585%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-04-20T17:12:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:25 smithi106 ceph-mon[44423]: osdmap e204: 12 total, 12 up, 12 in 2024-04-20T17:12:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:25 smithi171 ceph-mon[44189]: pgmap v582: 17 pgs: 1 undersized+degraded+peered, 1 active+recovering+degraded, 15 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 4 op/s; 12210/218628 objects degraded (5.585%); 5337/218628 objects misplaced (2.441%); 813 KiB/s, 12 objects/s recovering 2024-04-20T17:12:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:25 smithi171 ceph-mon[44189]: 2.e scrub starts 2024-04-20T17:12:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:25 smithi171 ceph-mon[44189]: 2.e scrub ok 2024-04-20T17:12:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:25 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 12210/218628 objects degraded (5.585%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-04-20T17:12:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:25 smithi171 ceph-mon[44189]: osdmap e204: 12 total, 12 up, 12 in 2024-04-20T17:12:26.905 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:12:26.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:26.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:12:26.946 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:12:26.946 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:12:27.009 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:12:27.039 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:27.112 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:12:27.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:27.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:12:27.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:26 smithi005 ceph-mon[39562]: 2.b scrub starts 2024-04-20T17:12:27.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:26 smithi005 ceph-mon[39562]: pgmap v584: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 17 op/s; 5336/218492 objects degraded (2.442%); 5337/218492 objects misplaced (2.443%) 2024-04-20T17:12:27.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:26 smithi005 ceph-mon[39562]: 2.a scrub starts 2024-04-20T17:12:27.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:26 smithi005 ceph-mon[39562]: 2.b scrub ok 2024-04-20T17:12:27.219 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:12:27.226 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:12:27.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:26 smithi106 ceph-mon[44423]: 2.b scrub starts 2024-04-20T17:12:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:26 smithi106 ceph-mon[44423]: pgmap v584: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 17 op/s; 5336/218492 objects degraded (2.442%); 5337/218492 objects misplaced (2.443%) 2024-04-20T17:12:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:26 smithi106 ceph-mon[44423]: 2.a scrub starts 2024-04-20T17:12:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:26 smithi106 ceph-mon[44423]: 2.b scrub ok 2024-04-20T17:12:27.263 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:26 smithi171 ceph-mon[44189]: 2.b scrub starts 2024-04-20T17:12:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:26 smithi171 ceph-mon[44189]: pgmap v584: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 17 op/s; 5336/218492 objects degraded (2.442%); 5337/218492 objects misplaced (2.443%) 2024-04-20T17:12:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:26 smithi171 ceph-mon[44189]: 2.a scrub starts 2024-04-20T17:12:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:26 smithi171 ceph-mon[44189]: 2.b scrub ok 2024-04-20T17:12:28.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:27 smithi005 ceph-mon[39562]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-20T17:12:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:27 smithi005 ceph-mon[39562]: 2.a scrub ok 2024-04-20T17:12:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:27 smithi005 ceph-mon[39562]: 2.c scrub starts 2024-04-20T17:12:28.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:27 smithi106 ceph-mon[44423]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-20T17:12:28.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:27 smithi106 ceph-mon[44423]: 2.a scrub ok 2024-04-20T17:12:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:27 smithi106 ceph-mon[44423]: 2.c scrub starts 2024-04-20T17:12:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:27 smithi171 ceph-mon[44189]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-20T17:12:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:27 smithi171 ceph-mon[44189]: 2.a scrub ok 2024-04-20T17:12:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:27 smithi171 ceph-mon[44189]: 2.c scrub starts 2024-04-20T17:12:29.027 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:28 smithi005 ceph-mon[39562]: 2.f scrub starts 2024-04-20T17:12:29.028 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:28 smithi005 ceph-mon[39562]: 2.c scrub ok 2024-04-20T17:12:29.028 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:28 smithi005 ceph-mon[39562]: pgmap v585: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 52 op/s; 10611/218032 objects misplaced (4.867%); 324 KiB/s, 5 objects/s recovering 2024-04-20T17:12:29.028 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:28 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:12:29.028 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:28 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:12:29.028 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:28 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:12:29.028 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:28 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:12:29.028 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:28 smithi005 ceph-mon[39562]: 2.f scrub ok 2024-04-20T17:12:29.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:28 smithi106 ceph-mon[44423]: 2.f scrub starts 2024-04-20T17:12:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:28 smithi106 ceph-mon[44423]: 2.c scrub ok 2024-04-20T17:12:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:28 smithi106 ceph-mon[44423]: pgmap v585: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 52 op/s; 10611/218032 objects misplaced (4.867%); 324 KiB/s, 5 objects/s recovering 2024-04-20T17:12:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:28 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:12:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:28 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:12:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:28 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:12:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:28 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:12:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:28 smithi106 ceph-mon[44423]: 2.f scrub ok 2024-04-20T17:12:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:28 smithi171 ceph-mon[44189]: 2.f scrub starts 2024-04-20T17:12:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:28 smithi171 ceph-mon[44189]: 2.c scrub ok 2024-04-20T17:12:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:28 smithi171 ceph-mon[44189]: pgmap v585: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 6.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 52 op/s; 10611/218032 objects misplaced (4.867%); 324 KiB/s, 5 objects/s recovering 2024-04-20T17:12:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:28 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:12:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:28 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:12:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:28 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:12:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:28 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:12:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:28 smithi171 ceph-mon[44189]: 2.f scrub ok 2024-04-20T17:12:29.415 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:12:29.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:29.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:12:29.522 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:12:29.549 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:29.568 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:12:29.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:29.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:12:29.673 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:12:29.700 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:29.721 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:12:29.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:29.724 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:12:29.825 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:12:29.851 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:29.871 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:12:29.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:29.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:12:29.924 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:12:29.975 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:12:30.009 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:30.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:29 smithi171 ceph-mon[44189]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5336/218492 objects degraded (2.442%), 1 pg degraded) 2024-04-20T17:12:30.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:29 smithi171 ceph-mon[44189]: osdmap e205: 12 total, 12 up, 12 in 2024-04-20T17:12:30.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:29 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4221418555' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:12:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:29 smithi005 ceph-mon[39562]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5336/218492 objects degraded (2.442%), 1 pg degraded) 2024-04-20T17:12:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:29 smithi005 ceph-mon[39562]: osdmap e205: 12 total, 12 up, 12 in 2024-04-20T17:12:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:29 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4221418555' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:12:30.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:29 smithi106 ceph-mon[44423]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5336/218492 objects degraded (2.442%), 1 pg degraded) 2024-04-20T17:12:30.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:29 smithi106 ceph-mon[44423]: osdmap e205: 12 total, 12 up, 12 in 2024-04-20T17:12:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:29 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4221418555' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:12:31.172 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:30 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:12:31.172 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:30 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:12:31.172 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:30 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4221418555' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:12:31.172 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:30 smithi005 ceph-mon[39562]: osdmap e206: 12 total, 12 up, 12 in 2024-04-20T17:12:31.172 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:30 smithi005 ceph-mon[39562]: pgmap v588: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 6.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 70 op/s; 10611/217778 objects misplaced (4.872%); 331 KiB/s, 5 objects/s recovering 2024-04-20T17:12:31.172 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:30 smithi005 ceph-mon[39562]: osdmap e207: 12 total, 12 up, 12 in 2024-04-20T17:12:31.188 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:12:31.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:31.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:12:31.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:30 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:12:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:30 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:12:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:30 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4221418555' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:12:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:30 smithi106 ceph-mon[44423]: osdmap e206: 12 total, 12 up, 12 in 2024-04-20T17:12:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:30 smithi106 ceph-mon[44423]: pgmap v588: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 6.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 70 op/s; 10611/217778 objects misplaced (4.872%); 331 KiB/s, 5 objects/s recovering 2024-04-20T17:12:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:30 smithi106 ceph-mon[44423]: osdmap e207: 12 total, 12 up, 12 in 2024-04-20T17:12:31.292 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:12:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:30 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:12:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:30 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:12:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:30 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4221418555' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:12:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:30 smithi171 ceph-mon[44189]: osdmap e206: 12 total, 12 up, 12 in 2024-04-20T17:12:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:30 smithi171 ceph-mon[44189]: pgmap v588: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 6.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 70 op/s; 10611/217778 objects misplaced (4.872%); 331 KiB/s, 5 objects/s recovering 2024-04-20T17:12:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:30 smithi171 ceph-mon[44189]: osdmap e207: 12 total, 12 up, 12 in 2024-04-20T17:12:31.344 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:31.364 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:12:31.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:31.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:12:31.468 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:12:31.494 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:31.515 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:12:31.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:31.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:12:31.619 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:12:31.654 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:31.734 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:12:31.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:31.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:12:31.839 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:12:31.865 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:31.867 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-04-20T17:12:31.867 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 0 1 2024-04-20T17:12:31.875 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:12:31.875 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:12:31.885 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:12:31.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:31.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:12:31.990 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:12:32.017 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:32.036 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:12:32.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:32.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:12:32.142 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:12:32.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:31 smithi005 ceph-mon[39562]: 2.1 deep-scrub starts 2024-04-20T17:12:32.172 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:32.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:31 smithi106 ceph-mon[44423]: 2.1 deep-scrub starts 2024-04-20T17:12:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:31 smithi171 ceph-mon[44189]: 2.1 deep-scrub starts 2024-04-20T17:12:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:32 smithi005 ceph-mon[39562]: pgmap v590: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 6.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 144 op/s; 10611/216752 objects misplaced (4.895%); 331 KiB/s, 5 objects/s recovering 2024-04-20T17:12:33.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:32 smithi106 ceph-mon[44423]: pgmap v590: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 6.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 144 op/s; 10611/216752 objects misplaced (4.895%); 331 KiB/s, 5 objects/s recovering 2024-04-20T17:12:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:32 smithi171 ceph-mon[44189]: pgmap v590: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 6.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 144 op/s; 10611/216752 objects misplaced (4.895%); 331 KiB/s, 5 objects/s recovering 2024-04-20T17:12:33.526 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:12:33.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:33.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:12:33.630 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:12:33.660 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:33.680 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:12:33.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:33.683 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:12:33.784 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:12:33.813 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:33.833 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:12:33.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:33.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:12:33.937 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:12:33.963 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:33.984 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:12:33.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:33.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:12:34.078 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:12:34.080 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-20T17:12:34.088 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:12:34.116 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:34.136 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:12:34.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:34.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:12:34.240 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:12:34.271 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:34.375 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:12:34.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:34.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:12:34.479 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:12:34.511 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:34.535 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:12:34.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:34.538 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:12:34.639 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:12:34.668 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:34.707 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:12:34.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:34.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:12:34.720 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:12:34.813 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:12:34.839 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:34.860 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:12:34.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:34.863 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:12:34.964 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:12:34.995 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:35.095 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:12:35.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:35.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:12:35.199 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:12:35.225 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:35 smithi171 ceph-mon[44189]: pgmap v591: 17 pgs: 1 active+recovering, 16 active+clean; 6.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 224 op/s; 10611/215334 objects misplaced (4.928%) 2024-04-20T17:12:35.227 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:35.250 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:12:35.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:35.253 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:12:35.267 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:35 smithi005 ceph-mon[39562]: pgmap v591: 17 pgs: 1 active+recovering, 16 active+clean; 6.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 224 op/s; 10611/215334 objects misplaced (4.928%) 2024-04-20T17:12:35.354 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:12:35.383 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:35.409 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:12:35.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:35.412 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:12:35.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:35 smithi106 ceph-mon[44423]: pgmap v591: 17 pgs: 1 active+recovering, 16 active+clean; 6.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 224 op/s; 10611/215334 objects misplaced (4.928%) 2024-04-20T17:12:35.513 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:12:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:35.897 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:12:35.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:35.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:12:36.003 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:12:36.031 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:36.055 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:12:36.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:36.061 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:12:36.162 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:12:36.194 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:36.294 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:12:36.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:36.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:12:36.398 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:12:36.408 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:12:36.426 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:36.446 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:12:36.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:36.449 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:12:36.450 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:12:36.456 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:12:36.484 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:12:36.504 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:12:36.525 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:12:36.550 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:12:36.559 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:12:36.580 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:36.582 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:12:36.606 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:12:36.630 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:12:36.653 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:12:36.674 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:12:36.695 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:12:36.709 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:12:36.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:36.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:12:36.726 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:12:36.727 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:12:36.727 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:12:36.728 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:12:36.728 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:12:36.728 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:12:36.728 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:12:36.728 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:12:36.728 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:12:36.728 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:12:36.728 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:12:36.728 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:12:36.728 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:12:36.813 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:12:36.842 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:36.867 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:12:36.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:36.870 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:12:36.883 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:36 smithi005 ceph-mon[39562]: osdmap e208: 12 total, 12 up, 12 in 2024-04-20T17:12:36.883 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:36 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2128623499' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:12:36.883 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:36.883 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:36.883 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:36.883 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:36.883 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:36.970 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:12:37.004 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:37.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:36 smithi106 ceph-mon[44423]: osdmap e208: 12 total, 12 up, 12 in 2024-04-20T17:12:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:36 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2128623499' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:12:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:36 smithi171 ceph-mon[44189]: osdmap e208: 12 total, 12 up, 12 in 2024-04-20T17:12:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:36 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2128623499' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:12:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:37.455 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:12:37.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:37.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:12:37.559 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:12:37.591 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:37.820 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:12:37.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:37.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:12:37.925 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:12:37.954 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:37.992 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:12:37.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:37.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:12:38.096 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:12:38.124 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:38.144 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:12:38.147 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:37 smithi171 ceph-mon[44189]: pgmap v593: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 261 op/s; 10611/214602 objects misplaced (4.945%) 2024-04-20T17:12:38.147 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:38.148 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:38.148 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:38.148 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:38.148 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:38.148 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:38.148 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:38.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:38.149 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:12:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:37 smithi005 ceph-mon[39562]: pgmap v593: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 261 op/s; 10611/214602 objects misplaced (4.945%) 2024-04-20T17:12:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:38.250 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:12:38.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:37 smithi106 ceph-mon[44423]: pgmap v593: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 261 op/s; 10611/214602 objects misplaced (4.945%) 2024-04-20T17:12:38.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:38.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:38.276 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:38.298 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:12:38.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:38.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:12:38.402 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:12:38.431 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:38.455 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:12:38.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:38.460 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:12:38.561 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:12:38.588 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:38.609 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:12:38.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:38.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:12:38.714 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:12:38.742 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:38.767 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:12:38.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:38.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:12:38.870 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:12:38.887 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:38 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:12:38.887 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:38 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:12:38.889 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:38 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:12:38.889 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:38 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:12:38.898 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:38.918 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:12:38.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:38.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:12:39.022 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:12:39.051 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:39.070 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:12:39.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:39.075 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:12:39.175 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:12:39.203 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:39.224 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:12:39.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:39.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:12:39.328 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:12:39.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:38 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:12:39.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:38 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:12:39.355 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:39.376 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:12:39.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:39.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:12:39.480 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:12:39.517 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:39.562 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:12:39.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:39.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:12:39.667 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:12:39.694 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:39.714 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:12:39.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:39.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:12:39.820 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:12:39.847 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:39.866 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:12:39.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:39.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:12:39.895 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.0 to 1 (10000) 2024-04-20T17:12:39.970 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:12:39.996 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:39 smithi106 ceph-mon[44423]: pgmap v594: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 279 op/s; 10610/213304 objects misplaced (4.974%) 2024-04-20T17:12:39.996 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:39 smithi106 ceph-mon[44423]: osdmap e209: 12 total, 12 up, 12 in 2024-04-20T17:12:39.996 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:39 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2142226917' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:12:39.996 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:39 smithi106 ceph-mon[44423]: from='client.25025 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:12:39.998 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:40.017 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:12:40.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:40.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:12:40.121 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:12:40.148 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:39 smithi171 ceph-mon[44189]: pgmap v594: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 279 op/s; 10610/213304 objects misplaced (4.974%) 2024-04-20T17:12:40.148 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:39 smithi171 ceph-mon[44189]: osdmap e209: 12 total, 12 up, 12 in 2024-04-20T17:12:40.148 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:39 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2142226917' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:12:40.148 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:39 smithi171 ceph-mon[44189]: from='client.25025 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:12:40.150 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:40.180 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:12:40.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:40.186 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:12:40.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:39 smithi005 ceph-mon[39562]: pgmap v594: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 279 op/s; 10610/213304 objects misplaced (4.974%) 2024-04-20T17:12:40.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:39 smithi005 ceph-mon[39562]: osdmap e209: 12 total, 12 up, 12 in 2024-04-20T17:12:40.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:39 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2142226917' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:12:40.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:39 smithi005 ceph-mon[39562]: from='client.25025 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:12:40.287 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:12:40.313 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:40.334 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:12:40.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:40.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:12:40.438 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:12:40.470 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:40.819 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:12:40.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:40.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:12:40.856 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 1 1 2024-04-20T17:12:40.925 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:12:40.953 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:12:40.976 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:41.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:40 smithi005 ceph-mon[39562]: from='client.25025 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-20T17:12:41.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:40 smithi005 ceph-mon[39562]: osdmap e210: 12 total, 12 up, 12 in 2024-04-20T17:12:41.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:40 smithi005 ceph-mon[39562]: 2.6 deep-scrub starts 2024-04-20T17:12:41.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:40 smithi106 ceph-mon[44423]: from='client.25025 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-20T17:12:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:40 smithi106 ceph-mon[44423]: osdmap e210: 12 total, 12 up, 12 in 2024-04-20T17:12:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:40 smithi106 ceph-mon[44423]: 2.6 deep-scrub starts 2024-04-20T17:12:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:40 smithi171 ceph-mon[44189]: from='client.25025 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-20T17:12:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:40 smithi171 ceph-mon[44189]: osdmap e210: 12 total, 12 up, 12 in 2024-04-20T17:12:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:40 smithi171 ceph-mon[44189]: 2.6 deep-scrub starts 2024-04-20T17:12:42.144 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:12:42.147 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:41 smithi005 ceph-mon[39562]: pgmap v597: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 210 op/s; 10610/212804 objects misplaced (4.986%) 2024-04-20T17:12:42.147 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:41 smithi005 ceph-mon[39562]: 2.d scrub starts 2024-04-20T17:12:42.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:42.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:12:42.250 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:12:42.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:41 smithi106 ceph-mon[44423]: pgmap v597: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 210 op/s; 10610/212804 objects misplaced (4.986%) 2024-04-20T17:12:42.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:41 smithi106 ceph-mon[44423]: 2.d scrub starts 2024-04-20T17:12:42.287 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:42.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:41 smithi171 ceph-mon[44189]: pgmap v597: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 210 op/s; 10610/212804 objects misplaced (4.986%) 2024-04-20T17:12:42.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:41 smithi171 ceph-mon[44189]: 2.d scrub starts 2024-04-20T17:12:42.369 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:12:42.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:42.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:12:42.476 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:12:42.504 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:42.524 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:12:42.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:42.530 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:12:42.630 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:12:42.633 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:12:42.633 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:12:42.633 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-04-20T17:12:42.633 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:12:42.662 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:42.867 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:12:42.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:42.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:12:42.889 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:12:42.893 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:12:42.933 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:12:42.974 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:12:43.003 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:43.027 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:12:43.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:43.032 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:12:43.133 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:12:43.161 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:42 smithi005 ceph-mon[39562]: 2.d scrub ok 2024-04-20T17:12:43.161 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:42 smithi005 ceph-mon[39562]: pgmap v598: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 6.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 262 op/s; 10610/211528 objects misplaced (5.016%) 2024-04-20T17:12:43.162 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:42 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/485723310' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:12:43.162 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:42 smithi005 ceph-mon[39562]: from='client.25031 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:12:43.164 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:42 smithi106 ceph-mon[44423]: 2.d scrub ok 2024-04-20T17:12:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:42 smithi106 ceph-mon[44423]: pgmap v598: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 6.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 262 op/s; 10610/211528 objects misplaced (5.016%) 2024-04-20T17:12:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:42 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/485723310' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:12:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:42 smithi106 ceph-mon[44423]: from='client.25031 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:12:43.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:42 smithi171 ceph-mon[44189]: 2.d scrub ok 2024-04-20T17:12:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:42 smithi171 ceph-mon[44189]: pgmap v598: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 6.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 262 op/s; 10610/211528 objects misplaced (5.016%) 2024-04-20T17:12:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:42 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/485723310' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:12:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:42 smithi171 ceph-mon[44189]: from='client.25031 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:12:43.920 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:12:43.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:43.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:12:44.024 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:12:44.051 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:44.072 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:12:44.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:44.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:12:44.176 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:12:44.200 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:43 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:12:44.201 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:43 smithi106 ceph-mon[44423]: from='client.25031 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:12:44.201 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:43 smithi106 ceph-mon[44423]: osdmap e211: 12 total, 12 up, 12 in 2024-04-20T17:12:44.201 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:43 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:12:44.201 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:43 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:12:44.203 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:44.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:43 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:12:44.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:43 smithi005 ceph-mon[39562]: from='client.25031 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:12:44.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:43 smithi005 ceph-mon[39562]: osdmap e211: 12 total, 12 up, 12 in 2024-04-20T17:12:44.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:43 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:12:44.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:43 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:12:44.223 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:12:44.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:44.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:12:44.326 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:12:44.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:43 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:12:44.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:43 smithi171 ceph-mon[44189]: from='client.25031 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:12:44.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:43 smithi171 ceph-mon[44189]: osdmap e211: 12 total, 12 up, 12 in 2024-04-20T17:12:44.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:43 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:12:44.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:43 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:12:44.354 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:44.373 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:12:44.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:44.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:12:44.476 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:12:44.506 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:44.532 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:12:44.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:44.536 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:12:44.637 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:12:44.672 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:44.701 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:12:44.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:44.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:12:44.811 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:12:44.841 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:45.092 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:12:45.094 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:44 smithi005 ceph-mon[39562]: osdmap e212: 12 total, 12 up, 12 in 2024-04-20T17:12:45.094 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:44 smithi005 ceph-mon[39562]: pgmap v601: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 6.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 293 op/s; 10610/209804 objects misplaced (5.057%) 2024-04-20T17:12:45.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:45.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:12:45.197 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:12:45.221 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:44 smithi106 ceph-mon[44423]: osdmap e212: 12 total, 12 up, 12 in 2024-04-20T17:12:45.221 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:44 smithi106 ceph-mon[44423]: pgmap v601: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 6.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 293 op/s; 10610/209804 objects misplaced (5.057%) 2024-04-20T17:12:45.224 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:45.244 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:12:45.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:45.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:12:45.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:44 smithi171 ceph-mon[44189]: osdmap e212: 12 total, 12 up, 12 in 2024-04-20T17:12:45.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:44 smithi171 ceph-mon[44189]: pgmap v601: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 6.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 293 op/s; 10610/209804 objects misplaced (5.057%) 2024-04-20T17:12:45.348 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:12:45.375 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:45.396 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:12:45.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:45.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:12:45.500 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:12:45.532 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:45.841 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-20T17:12:46.185 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:12:46.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:46.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:12:46.289 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:12:46.320 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:46.342 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:12:46.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:46.346 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:12:46.447 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:12:46.483 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:47.127 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:12:47.157 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:46 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/313245991' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:12:47.158 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:46 smithi005 ceph-mon[39562]: from='client.25037 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:12:47.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:46 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/313245991' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:12:47.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:46 smithi106 ceph-mon[44423]: from='client.25037 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:12:47.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:46 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/313245991' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:12:47.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:46 smithi171 ceph-mon[44189]: from='client.25037 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:12:47.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:47 smithi005 ceph-mon[39562]: pgmap v602: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 6.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 310 op/s; 10610/209002 objects misplaced (5.077%) 2024-04-20T17:12:47.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:47 smithi005 ceph-mon[39562]: from='client.25037 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-20T17:12:47.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:47 smithi005 ceph-mon[39562]: osdmap e213: 12 total, 12 up, 12 in 2024-04-20T17:12:48.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:47 smithi106 ceph-mon[44423]: pgmap v602: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 6.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 310 op/s; 10610/209002 objects misplaced (5.077%) 2024-04-20T17:12:48.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:47 smithi106 ceph-mon[44423]: from='client.25037 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-20T17:12:48.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:47 smithi106 ceph-mon[44423]: osdmap e213: 12 total, 12 up, 12 in 2024-04-20T17:12:48.275 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.1 to 1 (10000) 2024-04-20T17:12:48.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:47 smithi171 ceph-mon[44189]: pgmap v602: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 6.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 310 op/s; 10610/209002 objects misplaced (5.077%) 2024-04-20T17:12:48.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:47 smithi171 ceph-mon[44189]: from='client.25037 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-20T17:12:48.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:47 smithi171 ceph-mon[44189]: osdmap e213: 12 total, 12 up, 12 in 2024-04-20T17:12:48.637 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:12:48.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:48.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:12:48.743 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:12:48.769 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:48.788 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:12:48.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:48.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:12:48.891 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:12:48.918 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:48.936 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:12:48.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:48.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:12:49.040 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:12:49.067 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:49.086 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:12:49.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:49.090 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:12:49.116 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:12:49.157 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 2 1 2024-04-20T17:12:49.190 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:12:49.217 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:49.234 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:12:49.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:49.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:12:49.338 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:12:49.370 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:49.393 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:12:49.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:49.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:12:49.500 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:12:49.533 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:49.633 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:12:49.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:49.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:12:49.737 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:12:49.772 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:49.787 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:12:49.796 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:12:49.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:49.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:12:49.811 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:12:49.846 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:12:49.873 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:12:49.894 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:12:49.903 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:49 smithi005 ceph-mon[39562]: pgmap v604: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 6.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 306 op/s; 10610/207850 objects misplaced (5.105%) 2024-04-20T17:12:49.903 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:49 smithi005 ceph-mon[39562]: 2.1 deep-scrub ok 2024-04-20T17:12:49.903 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:49 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3452193662' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:12:49.903 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:49.904 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:12:49.918 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:12:49.934 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:49.942 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:12:49.970 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:12:49.996 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:12:50.017 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:12:50.045 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:12:50.070 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:12:50.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:50.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:12:50.080 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:12:50.089 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:12:50.089 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:12:50.089 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:12:50.090 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:12:50.090 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:12:50.090 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:12:50.090 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:12:50.090 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:12:50.090 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:12:50.090 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:12:50.090 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:12:50.090 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:12:50.123 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:12:50.176 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:12:50.218 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:50.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:49 smithi106 ceph-mon[44423]: pgmap v604: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 6.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 306 op/s; 10610/207850 objects misplaced (5.105%) 2024-04-20T17:12:50.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:49 smithi106 ceph-mon[44423]: 2.1 deep-scrub ok 2024-04-20T17:12:50.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:49 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3452193662' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:12:50.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:49 smithi171 ceph-mon[44189]: pgmap v604: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 6.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 306 op/s; 10610/207850 objects misplaced (5.105%) 2024-04-20T17:12:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:49 smithi171 ceph-mon[44189]: 2.1 deep-scrub ok 2024-04-20T17:12:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:49 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3452193662' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:12:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:50.566 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:12:50.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:50.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:12:50.670 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:12:50.697 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:50.716 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:12:50.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:50.719 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:12:50.820 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:12:50.852 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:50 smithi171 ceph-mon[44189]: osdmap e214: 12 total, 12 up, 12 in 2024-04-20T17:12:51.210 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:12:51.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:51.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:50 smithi005 ceph-mon[39562]: osdmap e214: 12 total, 12 up, 12 in 2024-04-20T17:12:51.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:12:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:12:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:50 smithi106 ceph-mon[44423]: osdmap e214: 12 total, 12 up, 12 in 2024-04-20T17:12:51.316 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:12:51.346 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:51.844 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:12:52.092 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:51 smithi005 ceph-mon[39562]: pgmap v605: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 6.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 204 op/s; 10610/207426 objects misplaced (5.115%) 2024-04-20T17:12:52.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:51 smithi106 ceph-mon[44423]: pgmap v605: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 6.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 204 op/s; 10610/207426 objects misplaced (5.115%) 2024-04-20T17:12:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:51 smithi171 ceph-mon[44189]: pgmap v605: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 6.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 204 op/s; 10610/207426 objects misplaced (5.115%) 2024-04-20T17:12:52.600 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:12:52.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:52.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:12:52.705 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:12:52.738 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:52.889 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:12:52.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:52.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:12:52.993 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:12:53.025 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:53.046 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:12:53.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:53.049 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:12:53.150 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:12:53.177 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:52 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:12:53.177 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:52 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:12:53.177 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:52 smithi005 ceph-mon[39562]: osdmap e215: 12 total, 12 up, 12 in 2024-04-20T17:12:53.177 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:52 smithi005 ceph-mon[39562]: pgmap v608: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 194 op/s; 10610/206672 objects misplaced (5.134%) 2024-04-20T17:12:53.179 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:53.237 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:12:53.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:53.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:12:53.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:52 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:12:53.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:52 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:12:53.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:52 smithi106 ceph-mon[44423]: osdmap e215: 12 total, 12 up, 12 in 2024-04-20T17:12:53.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:52 smithi106 ceph-mon[44423]: pgmap v608: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 194 op/s; 10610/206672 objects misplaced (5.134%) 2024-04-20T17:12:53.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:52 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:12:53.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:52 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:12:53.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:52 smithi171 ceph-mon[44189]: osdmap e215: 12 total, 12 up, 12 in 2024-04-20T17:12:53.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:52 smithi171 ceph-mon[44189]: pgmap v608: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 194 op/s; 10610/206672 objects misplaced (5.134%) 2024-04-20T17:12:53.340 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:12:53.366 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:53.385 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:12:53.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:53.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:12:53.488 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:12:53.515 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:53.533 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:12:53.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:53.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:12:53.636 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:12:53.662 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:53.681 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:12:53.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:53.684 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:12:53.785 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:12:53.812 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:53.831 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:12:53.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:53.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:12:53.937 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:12:53.973 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:12:53.974 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:12:53.977 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:54.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:53 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:12:54.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:53 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:12:54.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:53 smithi005 ceph-mon[39562]: osdmap e216: 12 total, 12 up, 12 in 2024-04-20T17:12:54.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:53 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:12:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:53 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:12:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:53 smithi106 ceph-mon[44423]: osdmap e216: 12 total, 12 up, 12 in 2024-04-20T17:12:54.330 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:12:54.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:53 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:12:54.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:53 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:12:54.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:53 smithi171 ceph-mon[44189]: osdmap e216: 12 total, 12 up, 12 in 2024-04-20T17:12:54.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:54.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:12:54.436 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:12:54.462 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:54.481 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:12:54.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:54.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:12:54.584 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:12:54.613 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:54.614 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:12:54.635 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:12:54.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:54.638 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:12:54.740 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:12:54.769 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:54.790 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:12:54.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:54.794 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:12:54.895 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:12:54.921 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:54.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:54 smithi005 ceph-mon[39562]: pgmap v610: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 217 op/s; 10610/205242 objects misplaced (5.170%) 2024-04-20T17:12:54.939 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:12:54.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:54.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:12:55.043 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:12:55.082 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:55.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:54 smithi106 ceph-mon[44423]: pgmap v610: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 217 op/s; 10610/205242 objects misplaced (5.170%) 2024-04-20T17:12:55.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:54 smithi171 ceph-mon[44189]: pgmap v610: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 217 op/s; 10610/205242 objects misplaced (5.170%) 2024-04-20T17:12:56.586 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-20T17:12:56.836 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:12:56.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:56.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:12:56.941 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:12:56.972 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:56.992 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:12:56.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:56.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:12:57.102 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:12:57.131 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:57.150 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:12:57.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:57.153 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:12:57.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:57 smithi005 ceph-mon[39562]: pgmap v611: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 241 op/s; 10610/204522 objects misplaced (5.188%) 2024-04-20T17:12:57.254 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:12:57.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:57 smithi106 ceph-mon[44423]: pgmap v611: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 241 op/s; 10610/204522 objects misplaced (5.188%) 2024-04-20T17:12:57.281 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:57.300 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:12:57.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:57.306 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:12:57.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:57 smithi171 ceph-mon[44189]: pgmap v611: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 241 op/s; 10610/204522 objects misplaced (5.188%) 2024-04-20T17:12:57.406 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:12:57.441 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:57.503 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:12:57.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:57.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:12:57.611 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:12:57.638 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:57.660 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:12:57.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:57.663 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:12:57.763 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:12:57.790 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:57.810 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:12:57.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:57.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:12:57.833 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:12:57.914 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:12:57.950 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:58.070 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.2 to 1 (10000) 2024-04-20T17:12:58.200 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:58 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3621928298' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-20T17:12:58.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:58 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3621928298' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-20T17:12:58.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:58 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3621928298' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-20T17:12:58.906 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:12:58.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:58.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:12:59.010 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:12:59.042 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:59.084 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:12:59.086 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:12:59.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:12:59.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:12:59.196 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:12:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:59 smithi005 ceph-mon[39562]: pgmap v612: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 259 op/s; 10610/203266 objects misplaced (5.220%) 2024-04-20T17:12:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:59 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3621928298' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-20T17:12:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:59 smithi005 ceph-mon[39562]: osdmap e217: 12 total, 12 up, 12 in 2024-04-20T17:12:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:59 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1945675070' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:12:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:12:59 smithi005 ceph-mon[39562]: from='client.25055 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:12:59.227 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:12:59.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:59 smithi171 ceph-mon[44189]: pgmap v612: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 259 op/s; 10610/203266 objects misplaced (5.220%) 2024-04-20T17:12:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:59 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3621928298' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-20T17:12:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:59 smithi171 ceph-mon[44189]: osdmap e217: 12 total, 12 up, 12 in 2024-04-20T17:12:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:59 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1945675070' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:12:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:12:59 smithi171 ceph-mon[44189]: from='client.25055 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:12:59.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:59 smithi106 ceph-mon[44423]: pgmap v612: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 259 op/s; 10610/203266 objects misplaced (5.220%) 2024-04-20T17:12:59.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:59 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3621928298' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-20T17:12:59.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:59 smithi106 ceph-mon[44423]: osdmap e217: 12 total, 12 up, 12 in 2024-04-20T17:12:59.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:59 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1945675070' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:12:59.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:12:59 smithi106 ceph-mon[44423]: from='client.25055 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:12:59.803 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:13:00.083 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:13:00.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:00.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:13:00.188 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:13:00.214 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:00.232 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:13:00.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:00.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:13:00.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:00 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:13:00.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:00 smithi171 ceph-mon[44189]: from='client.25055 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:13:00.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:00 smithi171 ceph-mon[44189]: osdmap e218: 12 total, 12 up, 12 in 2024-04-20T17:13:00.335 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:13:00.338 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:00 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:13:00.338 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:00 smithi005 ceph-mon[39562]: from='client.25055 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:13:00.338 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:00 smithi005 ceph-mon[39562]: osdmap e218: 12 total, 12 up, 12 in 2024-04-20T17:13:00.366 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:00.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:00 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:13:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:00 smithi106 ceph-mon[44423]: from='client.25055 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:13:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:00 smithi106 ceph-mon[44423]: osdmap e218: 12 total, 12 up, 12 in 2024-04-20T17:13:00.926 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 3 1 2024-04-20T17:13:01.816 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:13:01.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:01.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:13:01.920 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:13:01.947 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:01.953 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:13:01.953 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:13:01.965 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:13:01.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:01.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:13:02.068 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:13:02.099 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:02.118 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:13:02.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:02.122 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:13:02.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:01 smithi005 ceph-mon[39562]: pgmap v615: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 200 op/s; 10610/202838 objects misplaced (5.231%) 2024-04-20T17:13:02.223 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:13:02.259 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:02.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:01 smithi106 ceph-mon[44423]: pgmap v615: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 200 op/s; 10610/202838 objects misplaced (5.231%) 2024-04-20T17:13:02.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:01 smithi171 ceph-mon[44189]: pgmap v615: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 6.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 200 op/s; 10610/202838 objects misplaced (5.231%) 2024-04-20T17:13:02.789 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-20T17:13:02.806 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:13:02.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:02.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:13:02.910 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:13:02.940 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:03.071 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:13:03.726 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:13:03.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:03.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:13:03.741 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:13:03.800 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:13:03.831 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:13:03.863 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:03.883 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:13:03.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:03.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:13:03.987 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:13:04.014 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:04.087 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:13:04.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:04.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:13:04.190 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:13:04.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:03 smithi005 ceph-mon[39562]: pgmap v616: 17 pgs: 1 active+recovering, 16 active+clean; 6.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 219 op/s; 10610/201730 objects misplaced (5.260%) 2024-04-20T17:13:04.215 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:03 smithi106 ceph-mon[44423]: pgmap v616: 17 pgs: 1 active+recovering, 16 active+clean; 6.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 219 op/s; 10610/201730 objects misplaced (5.260%) 2024-04-20T17:13:04.217 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:04.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:03 smithi171 ceph-mon[44189]: pgmap v616: 17 pgs: 1 active+recovering, 16 active+clean; 6.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 219 op/s; 10610/201730 objects misplaced (5.260%) 2024-04-20T17:13:04.565 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:13:04.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:04.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:13:04.670 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:13:04.697 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:04.895 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:13:04.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:04.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:13:05.000 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:13:05.033 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:05.155 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:13:05.155 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:13:05.575 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:13:05.734 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:13:05.734 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:13:05.735 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:13:05.735 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:13:05.735 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:13:05.735 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:13:05.735 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:13:05.735 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:13:05.735 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:13:05.735 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:13:05.735 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:13:05.735 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:13:05.735 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:13:05.735 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:13:05.736 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:13:05.736 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:13:05.736 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:13:05.736 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:13:05.736 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:13:05.736 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:13:05.736 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:13:05.736 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:13:05.736 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:13:05.736 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:13:05.850 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-20T17:13:06.146 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:13:06.206 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:05 smithi005 ceph-mon[39562]: pgmap v617: 17 pgs: 1 active+recovering, 16 active+clean; 6.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 280 op/s; 10610/200030 objects misplaced (5.304%) 2024-04-20T17:13:06.206 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:05 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3511776181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:13:06.206 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:06.206 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:06.207 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:06.207 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:06.207 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:06.207 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:06.207 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:06.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:05 smithi106 ceph-mon[44423]: pgmap v617: 17 pgs: 1 active+recovering, 16 active+clean; 6.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 280 op/s; 10610/200030 objects misplaced (5.304%) 2024-04-20T17:13:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:05 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3511776181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:13:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:06.263 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:13:06.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:06.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:13:06.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:05 smithi171 ceph-mon[44189]: pgmap v617: 17 pgs: 1 active+recovering, 16 active+clean; 6.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 280 op/s; 10610/200030 objects misplaced (5.304%) 2024-04-20T17:13:06.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:05 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3511776181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:13:06.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:06.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:06.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:06.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:06.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:06.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:06.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:06.366 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:13:06.398 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:07.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:07.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:07.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:07.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:07.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:07.554 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:13:07.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:07.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:13:07.658 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:13:07.690 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:08.072 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:13:08.072 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:13:08.072 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-04-20T17:13:08.072 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-04-20T17:13:08.073 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:13:08.095 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:07 smithi106 ceph-mon[44423]: pgmap v618: 17 pgs: 1 active+recovering, 16 active+clean; 6.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 250 op/s; 10610/199264 objects misplaced (5.325%) 2024-04-20T17:13:08.139 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:13:08.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:08.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:13:08.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:07 smithi005 ceph-mon[39562]: pgmap v618: 17 pgs: 1 active+recovering, 16 active+clean; 6.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 250 op/s; 10610/199264 objects misplaced (5.325%) 2024-04-20T17:13:08.243 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:13:08.278 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:08.300 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:13:08.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:08.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:13:08.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:07 smithi171 ceph-mon[44189]: pgmap v618: 17 pgs: 1 active+recovering, 16 active+clean; 6.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 250 op/s; 10610/199264 objects misplaced (5.325%) 2024-04-20T17:13:08.405 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:13:08.434 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:08.639 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:13:08.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:08.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:13:08.743 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:13:08.772 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:08.834 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-20T17:13:08.899 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:13:08.909 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.3 to 1 (10000) 2024-04-20T17:13:09.108 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:13:09.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:08 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3114439446' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:13:09.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:08 smithi005 ceph-mon[39562]: from='client.25045 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:13:09.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:08 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3114439446' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:13:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:08 smithi106 ceph-mon[44423]: from='client.25045 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:13:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:08 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3114439446' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:13:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:08 smithi171 ceph-mon[44189]: from='client.25045 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:13:09.722 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:13:09.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:09.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:13:09.827 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:13:09.856 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:09.894 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:13:09.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:09.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:13:09.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:09 smithi005 ceph-mon[39562]: pgmap v619: 17 pgs: 1 active+recovering, 16 active+clean; 6.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 268 op/s; 10610/197924 objects misplaced (5.361%) 2024-04-20T17:13:09.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:09 smithi005 ceph-mon[39562]: from='client.25045 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-20T17:13:09.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:09 smithi005 ceph-mon[39562]: osdmap e219: 12 total, 12 up, 12 in 2024-04-20T17:13:10.005 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:13:10.035 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:10.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:09 smithi106 ceph-mon[44423]: pgmap v619: 17 pgs: 1 active+recovering, 16 active+clean; 6.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 268 op/s; 10610/197924 objects misplaced (5.361%) 2024-04-20T17:13:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:09 smithi106 ceph-mon[44423]: from='client.25045 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-20T17:13:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:09 smithi106 ceph-mon[44423]: osdmap e219: 12 total, 12 up, 12 in 2024-04-20T17:13:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:09 smithi171 ceph-mon[44189]: pgmap v619: 17 pgs: 1 active+recovering, 16 active+clean; 6.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 268 op/s; 10610/197924 objects misplaced (5.361%) 2024-04-20T17:13:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:09 smithi171 ceph-mon[44189]: from='client.25045 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-20T17:13:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:09 smithi171 ceph-mon[44189]: osdmap e219: 12 total, 12 up, 12 in 2024-04-20T17:13:10.385 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:13:10.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:10.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:13:10.397 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 4 1 2024-04-20T17:13:10.494 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:13:10.524 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:10.674 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:13:10.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:10.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:13:10.779 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:13:10.812 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:10.960 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:13:10.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:10.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:13:10.978 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:13:10.978 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-04-20T17:13:10.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-04-20T17:13:10.978 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:13:10.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.978 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:135900 2.d e142/115 2:b066403e:::benchmark_data_smithi171_78918_object135899:head v 142'8409, mlcod=142'8406)", 2024-04-20T17:13:10.978 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:09:21.017880+0000", 2024-04-20T17:13:10.978 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 229.95903259799999, 2024-04-20T17:13:10.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012669634000000001, 2024-04-20T17:13:10.979 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:10.979 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:10.979 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:10.979 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:10.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.979 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:10.979 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.017880+0000", 2024-04-20T17:13:10.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:10.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.979 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:10.979 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.017880+0000", 2024-04-20T17:13:10.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:10.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.017883+0000", 2024-04-20T17:13:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.196e-06 2024-04-20T17:13:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.017976+0000", 2024-04-20T17:13:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2625000000000002e-05 2024-04-20T17:13:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.017993+0000", 2024-04-20T17:13:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7181e-05 2024-04-20T17:13:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.018004+0000", 2024-04-20T17:13:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1474999999999999e-05 2024-04-20T17:13:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.018061+0000", 2024-04-20T17:13:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7043000000000003e-05 2024-04-20T17:13:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.018086+0000", 2024-04-20T17:13:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4368000000000001e-05 2024-04-20T17:13:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.982 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:10.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.030488+0000", 2024-04-20T17:13:10.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012402616999999999 2024-04-20T17:13:10.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.982 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:10.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.030549+0000", 2024-04-20T17:13:10.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1129000000000004e-05 2024-04-20T17:13:10.982 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:10.982 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:10.982 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:10.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.983 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:138919 2.d e143/115 2:b227306f:::benchmark_data_smithi171_78918_object138918:head v 143'8595, mlcod=143'8592)", 2024-04-20T17:13:10.983 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:09:23.338496+0000", 2024-04-20T17:13:10.983 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 227.63841604199999, 2024-04-20T17:13:10.983 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0098117770000000007, 2024-04-20T17:13:10.983 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:10.983 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:10.983 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:10.983 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:10.983 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.983 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:10.983 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338496+0000", 2024-04-20T17:13:10.983 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:10.983 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.983 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.984 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:10.984 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338496+0000", 2024-04-20T17:13:10.984 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:10.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.984 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:10.984 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338499+0000", 2024-04-20T17:13:10.984 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4700000000000001e-06 2024-04-20T17:13:10.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.984 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:10.984 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338647+0000", 2024-04-20T17:13:10.984 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014819599999999999 2024-04-20T17:13:10.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.984 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:10.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338663+0000", 2024-04-20T17:13:10.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6532e-05 2024-04-20T17:13:10.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:10.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338671+0000", 2024-04-20T17:13:10.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1779999999999997e-06 2024-04-20T17:13:10.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:10.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338719+0000", 2024-04-20T17:13:10.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8152000000000002e-05 2024-04-20T17:13:10.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338737+0000", 2024-04-20T17:13:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8117e-05 2024-04-20T17:13:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.348258+0000", 2024-04-20T17:13:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0095206400000000004 2024-04-20T17:13:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.348308+0000", 2024-04-20T17:13:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0491999999999997e-05 2024-04-20T17:13:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:138920 2.d e143/115 2:ba4e1b76:::benchmark_data_smithi171_78918_object138919:head v 143'8596, mlcod=143'8593)", 2024-04-20T17:13:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:09:23.338925+0000", 2024-04-20T17:13:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 227.63798727899999, 2024-04-20T17:13:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0094244870000000005, 2024-04-20T17:13:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338925+0000", 2024-04-20T17:13:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:10.988 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.988 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.988 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:10.988 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338925+0000", 2024-04-20T17:13:10.988 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:10.988 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.988 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.988 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:10.988 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338926+0000", 2024-04-20T17:13:10.988 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6700000000000002e-07 2024-04-20T17:13:10.988 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.988 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.339107+0000", 2024-04-20T17:13:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000181397 2024-04-20T17:13:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.339116+0000", 2024-04-20T17:13:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9600000000000006e-06 2024-04-20T17:13:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.339123+0000", 2024-04-20T17:13:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1250000000000004e-06 2024-04-20T17:13:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.990 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:10.990 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.341580+0000", 2024-04-20T17:13:10.990 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0024563979999999998 2024-04-20T17:13:10.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.990 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.990 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:10.990 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.341606+0000", 2024-04-20T17:13:10.990 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5919999999999999e-05 2024-04-20T17:13:10.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.990 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.990 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:10.990 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.348316+0000", 2024-04-20T17:13:10.990 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0067097709999999998 2024-04-20T17:13:10.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.990 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.991 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:10.991 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.348350+0000", 2024-04-20T17:13:10.991 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3949000000000003e-05 2024-04-20T17:13:10.991 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:10.991 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:10.991 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:10.991 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.991 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.991 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:145540 2.d e144/115 2:bbadbea3:::benchmark_data_smithi171_78918_object145539:head v 144'9033, mlcod=144'9030)", 2024-04-20T17:13:10.991 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:09:28.610761+0000", 2024-04-20T17:13:10.991 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 222.36615149100001, 2024-04-20T17:13:10.991 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0095878439999999999, 2024-04-20T17:13:10.991 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:10.991 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:10.992 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:10.992 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:10.992 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.992 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:10.992 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.610761+0000", 2024-04-20T17:13:10.992 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:10.992 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.992 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.992 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:10.992 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.610761+0000", 2024-04-20T17:13:10.992 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:10.992 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.992 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.992 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:10.992 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.610763+0000", 2024-04-20T17:13:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8649999999999999e-06 2024-04-20T17:13:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.610874+0000", 2024-04-20T17:13:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000111185 2024-04-20T17:13:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.610886+0000", 2024-04-20T17:13:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2379e-05 2024-04-20T17:13:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.610895+0000", 2024-04-20T17:13:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3210000000000005e-06 2024-04-20T17:13:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.613942+0000", 2024-04-20T17:13:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003047892 2024-04-20T17:13:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.613966+0000", 2024-04-20T17:13:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4000999999999999e-05 2024-04-20T17:13:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.995 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:10.995 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.620302+0000", 2024-04-20T17:13:10.995 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0063351930000000002 2024-04-20T17:13:10.995 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.995 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.995 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:10.995 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.620349+0000", 2024-04-20T17:13:10.995 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7008000000000001e-05 2024-04-20T17:13:10.995 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:10.995 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:10.995 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:10.996 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.996 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.996 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:146284 2.d e145/115 2:ba16209d:::benchmark_data_smithi171_78918_object146283:head v 145'9080, mlcod=145'9077)", 2024-04-20T17:13:10.996 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:09:29.190321+0000", 2024-04-20T17:13:10.996 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 221.78659133599999, 2024-04-20T17:13:10.996 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.009650166, 2024-04-20T17:13:10.996 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:10.996 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:10.996 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:10.996 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:10.996 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.996 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.190321+0000", 2024-04-20T17:13:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.190321+0000", 2024-04-20T17:13:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.190322+0000", 2024-04-20T17:13:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2500000000000005e-07 2024-04-20T17:13:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:10.998 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.190371+0000", 2024-04-20T17:13:10.998 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9458e-05 2024-04-20T17:13:10.998 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.998 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:10.998 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.190376+0000", 2024-04-20T17:13:10.998 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5619999999999997e-06 2024-04-20T17:13:10.998 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.998 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:10.998 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.190379+0000", 2024-04-20T17:13:10.998 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.557e-06 2024-04-20T17:13:10.998 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.999 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:10.999 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.193010+0000", 2024-04-20T17:13:10.999 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0026303440000000002 2024-04-20T17:13:10.999 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.999 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.999 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:10.999 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.193033+0000", 2024-04-20T17:13:10.999 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3027000000000002e-05 2024-04-20T17:13:10.999 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:10.999 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:10.999 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:11.000 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.199918+0000", 2024-04-20T17:13:11.000 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0068856550000000001 2024-04-20T17:13:11.000 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.000 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.000 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:11.000 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.199971+0000", 2024-04-20T17:13:11.000 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2837999999999998e-05 2024-04-20T17:13:11.000 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.000 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:11.000 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.000 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.000 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.000 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:180296 2.d e156/115 2:b3863235:::benchmark_data_smithi171_78918_object19302:head v 156'11076, mlcod=156'11074)", 2024-04-20T17:13:11.000 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:11.606812+0000", 2024-04-20T17:13:11.001 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 179.37010055299999, 2024-04-20T17:13:11.001 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010515645000000001, 2024-04-20T17:13:11.001 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:11.001 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:11.001 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:11.001 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:11.001 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.001 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:11.001 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.606812+0000", 2024-04-20T17:13:11.001 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.001 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.001 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.001 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:11.001 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.606812+0000", 2024-04-20T17:13:11.001 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.002 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:11.002 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.606814+0000", 2024-04-20T17:13:11.002 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2819999999999999e-06 2024-04-20T17:13:11.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.002 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:11.002 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.606822+0000", 2024-04-20T17:13:11.002 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0700000000000007e-06 2024-04-20T17:13:11.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.002 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:11.002 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.606835+0000", 2024-04-20T17:13:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3154e-05 2024-04-20T17:13:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.606843+0000", 2024-04-20T17:13:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2290000000000003e-06 2024-04-20T17:13:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.606904+0000", 2024-04-20T17:13:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0714000000000002e-05 2024-04-20T17:13:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.606928+0000", 2024-04-20T17:13:11.004 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.336e-05 2024-04-20T17:13:11.004 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.004 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.004 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:11.004 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.617278+0000", 2024-04-20T17:13:11.004 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010350597 2024-04-20T17:13:11.004 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.004 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.004 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:11.004 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.617327+0000", 2024-04-20T17:13:11.004 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9239000000000001e-05 2024-04-20T17:13:11.004 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.004 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:11.004 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.004 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.004 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.005 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:202407 2.d e166/115 2:be74e115:::benchmark_data_smithi171_78918_object41413:head v 166'12446, mlcod=166'12442)", 2024-04-20T17:13:11.005 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:39.985793+0000", 2024-04-20T17:13:11.005 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 150.99111920999999, 2024-04-20T17:13:11.005 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0095545500000000002, 2024-04-20T17:13:11.005 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:11.005 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:11.005 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:11.005 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:11.005 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.005 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:11.005 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.985793+0000", 2024-04-20T17:13:11.005 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.005 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.005 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.005 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:11.006 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.985793+0000", 2024-04-20T17:13:11.006 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.006 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.006 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.006 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:11.006 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.985794+0000", 2024-04-20T17:13:11.006 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3e-06 2024-04-20T17:13:11.006 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.006 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.006 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:11.006 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.985801+0000", 2024-04-20T17:13:11.006 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1890000000000004e-06 2024-04-20T17:13:11.006 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.006 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.007 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:11.007 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.985811+0000", 2024-04-20T17:13:11.007 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.06e-05 2024-04-20T17:13:11.007 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.007 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.007 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:11.007 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.985819+0000", 2024-04-20T17:13:11.007 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6580000000000002e-06 2024-04-20T17:13:11.007 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.007 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.007 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:11.007 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.990292+0000", 2024-04-20T17:13:11.007 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0044734869999999999 2024-04-20T17:13:11.007 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.008 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.008 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:11.008 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.990308+0000", 2024-04-20T17:13:11.008 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5389000000000001e-05 2024-04-20T17:13:11.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.008 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.008 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:11.008 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.995301+0000", 2024-04-20T17:13:11.008 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0049930010000000004 2024-04-20T17:13:11.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.008 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.008 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:11.008 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.995348+0000", 2024-04-20T17:13:11.008 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6925999999999998e-05 2024-04-20T17:13:11.008 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.008 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:11.009 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.009 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.009 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.009 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:202710 2.d e166/115 2:b3a6797d:::benchmark_data_smithi171_78918_object41716:head v 166'12468, mlcod=166'12466)", 2024-04-20T17:13:11.009 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:40.303984+0000", 2024-04-20T17:13:11.009 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 150.672928587, 2024-04-20T17:13:11.009 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.013242804, 2024-04-20T17:13:11.009 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:11.009 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:11.009 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:11.009 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:11.009 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.009 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:11.009 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.303984+0000", 2024-04-20T17:13:11.009 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.010 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.010 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.010 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:11.010 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.303984+0000", 2024-04-20T17:13:11.010 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.010 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.010 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.010 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:11.010 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.303986+0000", 2024-04-20T17:13:11.010 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0099999999999998e-06 2024-04-20T17:13:11.010 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.010 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.010 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:11.010 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.303997+0000", 2024-04-20T17:13:11.010 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0852999999999999e-05 2024-04-20T17:13:11.011 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.011 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.011 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:11.011 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.304011+0000", 2024-04-20T17:13:11.011 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4304000000000001e-05 2024-04-20T17:13:11.011 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.011 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.011 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:11.011 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.304020+0000", 2024-04-20T17:13:11.011 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6449999999999994e-06 2024-04-20T17:13:11.011 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.011 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.011 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:11.011 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.304086+0000", 2024-04-20T17:13:11.011 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6268999999999999e-05 2024-04-20T17:13:11.012 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.012 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.012 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:11.012 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.304111+0000", 2024-04-20T17:13:11.012 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5480999999999999e-05 2024-04-20T17:13:11.012 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.012 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.012 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:11.012 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.317176+0000", 2024-04-20T17:13:11.012 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.013064697 2024-04-20T17:13:11.012 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.012 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.012 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:11.012 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.317227+0000", 2024-04-20T17:13:11.012 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0544999999999998e-05 2024-04-20T17:13:11.012 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.013 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:11.013 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.013 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.013 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.013 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:202714 2.d e166/115 2:b3a8ef48:::benchmark_data_smithi171_78918_object41720:head v 166'12469, mlcod=166'12466)", 2024-04-20T17:13:11.013 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:40.308710+0000", 2024-04-20T17:13:11.013 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 150.66820214800001, 2024-04-20T17:13:11.013 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012740461, 2024-04-20T17:13:11.013 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:11.013 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:11.013 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:11.013 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:11.013 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.013 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:11.013 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.308710+0000", 2024-04-20T17:13:11.013 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.014 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.014 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.014 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:11.014 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.308710+0000", 2024-04-20T17:13:11.014 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.014 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.014 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.014 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:11.014 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.308712+0000", 2024-04-20T17:13:11.014 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0590000000000001e-06 2024-04-20T17:13:11.014 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.014 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.014 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:11.014 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.308722+0000", 2024-04-20T17:13:11.014 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0236e-05 2024-04-20T17:13:11.015 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.015 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:11.015 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.308735+0000", 2024-04-20T17:13:11.015 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2743e-05 2024-04-20T17:13:11.015 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.015 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:11.015 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.308744+0000", 2024-04-20T17:13:11.015 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8529999999999998e-06 2024-04-20T17:13:11.015 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.015 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:11.015 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.308809+0000", 2024-04-20T17:13:11.015 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4975999999999996e-05 2024-04-20T17:13:11.015 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.016 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.016 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:11.016 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.308834+0000", 2024-04-20T17:13:11.016 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5329000000000001e-05 2024-04-20T17:13:11.016 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.016 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.016 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:11.016 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.321396+0000", 2024-04-20T17:13:11.016 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012561948 2024-04-20T17:13:11.016 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.016 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.016 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:11.016 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.321451+0000", 2024-04-20T17:13:11.016 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4317000000000002e-05 2024-04-20T17:13:11.016 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.017 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:11.017 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.017 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.017 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.017 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:203759 2.d e167/115 2:b448a1cf:::benchmark_data_smithi171_78918_object42765:head v 167'12540, mlcod=167'12537)", 2024-04-20T17:13:11.017 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:41.412120+0000", 2024-04-20T17:13:11.017 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 149.56479255900001, 2024-04-20T17:13:11.017 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010611162, 2024-04-20T17:13:11.017 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:11.017 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:11.017 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:11.017 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:11.017 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.017 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:11.017 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.412120+0000", 2024-04-20T17:13:11.017 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.017 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.018 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.018 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:11.018 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.412120+0000", 2024-04-20T17:13:11.018 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.018 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.018 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.018 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:11.018 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.412121+0000", 2024-04-20T17:13:11.018 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7000000000000003e-07 2024-04-20T17:13:11.018 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.018 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.018 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:11.018 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.412126+0000", 2024-04-20T17:13:11.018 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8629999999999999e-06 2024-04-20T17:13:11.019 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.019 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.019 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:11.019 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.412132+0000", 2024-04-20T17:13:11.019 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3770000000000001e-06 2024-04-20T17:13:11.019 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.019 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.019 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:11.019 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.412136+0000", 2024-04-20T17:13:11.019 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2030000000000002e-06 2024-04-20T17:13:11.019 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.019 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.019 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:11.019 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.415923+0000", 2024-04-20T17:13:11.019 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037866380000000002 2024-04-20T17:13:11.019 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.020 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:11.020 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.415956+0000", 2024-04-20T17:13:11.020 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3308e-05 2024-04-20T17:13:11.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.020 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:11.020 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.422682+0000", 2024-04-20T17:13:11.020 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0067258149999999996 2024-04-20T17:13:11.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.020 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:11.020 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.422731+0000", 2024-04-20T17:13:11.020 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8987999999999999e-05 2024-04-20T17:13:11.020 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.020 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:203788 2.d e167/115 2:b22759bd:::benchmark_data_smithi171_78918_object42794:head v 167'12544, mlcod=167'12540)", 2024-04-20T17:13:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:41.437431+0000", 2024-04-20T17:13:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 149.53948098399999, 2024-04-20T17:13:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.009289472, 2024-04-20T17:13:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.437431+0000", 2024-04-20T17:13:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:11.022 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.437431+0000", 2024-04-20T17:13:11.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:11.022 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.437432+0000", 2024-04-20T17:13:11.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0809999999999999e-06 2024-04-20T17:13:11.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:11.022 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.437437+0000", 2024-04-20T17:13:11.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8459999999999999e-06 2024-04-20T17:13:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.437445+0000", 2024-04-20T17:13:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7249999999999997e-06 2024-04-20T17:13:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.437450+0000", 2024-04-20T17:13:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9250000000000003e-06 2024-04-20T17:13:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.441273+0000", 2024-04-20T17:13:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003822607 2024-04-20T17:13:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.441298+0000", 2024-04-20T17:13:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5850000000000002e-05 2024-04-20T17:13:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.446674+0000", 2024-04-20T17:13:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0053756129999999996 2024-04-20T17:13:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.446721+0000", 2024-04-20T17:13:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6825e-05 2024-04-20T17:13:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:207844 2.d e167/115 2:b4c70255:::benchmark_data_smithi171_78918_object46850:head v 167'12787, mlcod=167'12784)", 2024-04-20T17:13:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:45.866401+0000", 2024-04-20T17:13:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 145.110510968, 2024-04-20T17:13:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010044907, 2024-04-20T17:13:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866401+0000", 2024-04-20T17:13:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866401+0000", 2024-04-20T17:13:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866405+0000", 2024-04-20T17:13:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.67e-06 2024-04-20T17:13:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866421+0000", 2024-04-20T17:13:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6113e-05 2024-04-20T17:13:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866450+0000", 2024-04-20T17:13:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8915999999999999e-05 2024-04-20T17:13:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866470+0000", 2024-04-20T17:13:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.959e-05 2024-04-20T17:13:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866607+0000", 2024-04-20T17:13:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013733699999999999 2024-04-20T17:13:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866666+0000", 2024-04-20T17:13:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9147999999999997e-05 2024-04-20T17:13:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.876418+0000", 2024-04-20T17:13:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0097520650000000007 2024-04-20T17:13:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.876446+0000", 2024-04-20T17:13:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8068e-05 2024-04-20T17:13:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:207851 2.d e167/115 2:bc3064f0:::benchmark_data_smithi171_78918_object46857:head v 167'12788, mlcod=167'12785)", 2024-04-20T17:13:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:45.866896+0000", 2024-04-20T17:13:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 145.11001609600001, 2024-04-20T17:13:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0095667549999999997, 2024-04-20T17:13:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:11.030 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866896+0000", 2024-04-20T17:13:11.030 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.030 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.030 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.030 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:11.030 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866896+0000", 2024-04-20T17:13:11.030 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.030 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.030 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.031 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:11.031 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866897+0000", 2024-04-20T17:13:11.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.153e-06 2024-04-20T17:13:11.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.031 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:11.031 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866904+0000", 2024-04-20T17:13:11.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6050000000000003e-06 2024-04-20T17:13:11.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.031 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:11.031 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866915+0000", 2024-04-20T17:13:11.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0778000000000001e-05 2024-04-20T17:13:11.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.033 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:11.033 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866922+0000", 2024-04-20T17:13:11.033 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.464e-06 2024-04-20T17:13:11.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.033 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:11.033 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.870735+0000", 2024-04-20T17:13:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0038132550000000002 2024-04-20T17:13:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.870756+0000", 2024-04-20T17:13:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0865000000000001e-05 2024-04-20T17:13:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.876449+0000", 2024-04-20T17:13:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0056927790000000002 2024-04-20T17:13:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:11.035 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.876463+0000", 2024-04-20T17:13:11.035 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3856000000000001e-05 2024-04-20T17:13:11.035 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.035 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:11.035 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.035 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.035 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:208672 2.d e167/115 2:be8432c1:::benchmark_data_smithi171_78918_object47678:head v 167'12834, mlcod=167'12831)", 2024-04-20T17:13:11.035 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:46.774497+0000", 2024-04-20T17:13:11.035 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 144.20241525700001, 2024-04-20T17:13:11.035 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0092546139999999995, 2024-04-20T17:13:11.035 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:11.035 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:11.035 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:11.036 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:11.036 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.036 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:11.036 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:46.774497+0000", 2024-04-20T17:13:11.036 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.036 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.036 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.036 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:11.036 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:46.774497+0000", 2024-04-20T17:13:11.036 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.036 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.036 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.036 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:11.036 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:46.774498+0000", 2024-04-20T17:13:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2950000000000001e-06 2024-04-20T17:13:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:46.774505+0000", 2024-04-20T17:13:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5860000000000001e-06 2024-04-20T17:13:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:46.774516+0000", 2024-04-20T17:13:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0829000000000001e-05 2024-04-20T17:13:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:46.774522+0000", 2024-04-20T17:13:11.038 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.421e-06 2024-04-20T17:13:11.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.038 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.038 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:11.038 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:46.778316+0000", 2024-04-20T17:13:11.038 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003794296 2024-04-20T17:13:11.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.038 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.038 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:11.038 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:46.778336+0000", 2024-04-20T17:13:11.038 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9119000000000001e-05 2024-04-20T17:13:11.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.038 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.039 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:11.039 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:46.783708+0000", 2024-04-20T17:13:11.039 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.005372501 2024-04-20T17:13:11.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.039 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:11.039 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:46.783752+0000", 2024-04-20T17:13:11.039 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3566999999999998e-05 2024-04-20T17:13:11.039 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.039 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:11.039 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.039 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:209742 2.d e168/115 2:b2c27aa2:::benchmark_data_smithi171_78918_object48748:head v 168'12891, mlcod=168'12889)", 2024-04-20T17:13:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:47.949446+0000", 2024-04-20T17:13:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 143.02746679800001, 2024-04-20T17:13:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0097404459999999998, 2024-04-20T17:13:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949446+0000", 2024-04-20T17:13:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949446+0000", 2024-04-20T17:13:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.041 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.041 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.041 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:11.041 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949448+0000", 2024-04-20T17:13:11.041 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.531e-06 2024-04-20T17:13:11.041 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.041 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.041 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:11.041 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949457+0000", 2024-04-20T17:13:11.041 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1330000000000005e-06 2024-04-20T17:13:11.041 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.041 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.041 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:11.041 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949471+0000", 2024-04-20T17:13:11.042 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3478999999999999e-05 2024-04-20T17:13:11.042 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.042 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.042 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:11.042 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949480+0000", 2024-04-20T17:13:11.042 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8999999999999995e-06 2024-04-20T17:13:11.042 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.042 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.042 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:11.042 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949528+0000", 2024-04-20T17:13:11.042 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8916999999999999e-05 2024-04-20T17:13:11.042 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.042 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.042 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:11.043 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949547+0000", 2024-04-20T17:13:11.043 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8362000000000001e-05 2024-04-20T17:13:11.043 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.043 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.043 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:11.043 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.959142+0000", 2024-04-20T17:13:11.043 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0095948030000000007 2024-04-20T17:13:11.043 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.043 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.043 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:11.043 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.959186+0000", 2024-04-20T17:13:11.043 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4320999999999998e-05 2024-04-20T17:13:11.043 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.043 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:11.043 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.043 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.044 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.044 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:209749 2.d e168/115 2:b2c3330a:::benchmark_data_smithi171_78918_object48755:head v 168'12892, mlcod=168'12889)", 2024-04-20T17:13:11.044 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:47.949983+0000", 2024-04-20T17:13:11.044 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 143.02692925700001, 2024-04-20T17:13:11.044 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0092349100000000007, 2024-04-20T17:13:11.044 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:11.044 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:11.044 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:11.044 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:11.044 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.044 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:11.044 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949983+0000", 2024-04-20T17:13:11.044 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.044 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.045 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:11.045 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949983+0000", 2024-04-20T17:13:11.045 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.045 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:11.045 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949984+0000", 2024-04-20T17:13:11.045 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7100000000000001e-07 2024-04-20T17:13:11.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.045 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:11.045 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949988+0000", 2024-04-20T17:13:11.045 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5009999999999998e-06 2024-04-20T17:13:11.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.046 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.046 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:11.046 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949994+0000", 2024-04-20T17:13:11.046 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1680000000000001e-06 2024-04-20T17:13:11.046 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.046 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.046 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:11.046 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949998+0000", 2024-04-20T17:13:11.046 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.856e-06 2024-04-20T17:13:11.046 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.046 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.046 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:11.046 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.952247+0000", 2024-04-20T17:13:11.046 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022486509999999999 2024-04-20T17:13:11.047 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.047 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.047 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:11.047 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.952268+0000", 2024-04-20T17:13:11.047 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1228000000000001e-05 2024-04-20T17:13:11.047 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.047 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.047 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:11.047 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.959190+0000", 2024-04-20T17:13:11.047 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0069222399999999996 2024-04-20T17:13:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.959218+0000", 2024-04-20T17:13:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7495e-05 2024-04-20T17:13:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:210465 2.d e168/115 2:bf240863:::benchmark_data_smithi171_78918_object49471:head v 168'12947, mlcod=168'12944)", 2024-04-20T17:13:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:48.753722+0000", 2024-04-20T17:13:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 142.22319006699999, 2024-04-20T17:13:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.014363803, 2024-04-20T17:13:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:11.049 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:11.049 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:11.049 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:11.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.049 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:11.049 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753722+0000", 2024-04-20T17:13:11.049 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.049 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:11.049 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753722+0000", 2024-04-20T17:13:11.049 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753725+0000", 2024-04-20T17:13:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5110000000000002e-06 2024-04-20T17:13:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753737+0000", 2024-04-20T17:13:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1874e-05 2024-04-20T17:13:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753757+0000", 2024-04-20T17:13:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0670999999999999e-05 2024-04-20T17:13:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.051 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.051 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:11.051 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753768+0000", 2024-04-20T17:13:11.051 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0219e-05 2024-04-20T17:13:11.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.051 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.051 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:11.051 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753854+0000", 2024-04-20T17:13:11.051 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6948999999999997e-05 2024-04-20T17:13:11.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.051 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.051 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:11.051 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753888+0000", 2024-04-20T17:13:11.051 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3741000000000001e-05 2024-04-20T17:13:11.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.768045+0000", 2024-04-20T17:13:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.014157085 2024-04-20T17:13:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.768086+0000", 2024-04-20T17:13:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0753000000000001e-05 2024-04-20T17:13:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:210469 2.d e168/115 2:b66681fa:::benchmark_data_smithi171_78918_object49475:head v 168'12948, mlcod=168'12944)", 2024-04-20T17:13:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:48.753793+0000", 2024-04-20T17:13:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 142.22311969899999, 2024-04-20T17:13:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.014324059, 2024-04-20T17:13:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753793+0000", 2024-04-20T17:13:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753793+0000", 2024-04-20T17:13:11.054 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.054 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.054 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.054 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:11.054 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753793+0000", 2024-04-20T17:13:11.054 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3900000000000004e-07 2024-04-20T17:13:11.054 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.054 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.054 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:11.054 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753803+0000", 2024-04-20T17:13:11.054 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8130000000000007e-06 2024-04-20T17:13:11.054 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.054 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.054 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753810+0000", 2024-04-20T17:13:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6069999999999996e-06 2024-04-20T17:13:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753815+0000", 2024-04-20T17:13:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6030000000000004e-06 2024-04-20T17:13:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.757230+0000", 2024-04-20T17:13:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003414182 2024-04-20T17:13:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:11.056 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.757255+0000", 2024-04-20T17:13:11.056 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5324999999999999e-05 2024-04-20T17:13:11.056 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.056 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.056 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:11.056 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.768091+0000", 2024-04-20T17:13:11.056 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010835869999999999 2024-04-20T17:13:11.056 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.056 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.056 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:11.056 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.768117+0000", 2024-04-20T17:13:11.056 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5919999999999999e-05 2024-04-20T17:13:11.056 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.056 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:11.056 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:210471 2.d e168/115 2:b5294da4:::benchmark_data_smithi171_78918_object49477:head v 168'12949, mlcod=168'12944)", 2024-04-20T17:13:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:48.753843+0000", 2024-04-20T17:13:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 142.223069675, 2024-04-20T17:13:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.014299944, 2024-04-20T17:13:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753843+0000", 2024-04-20T17:13:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.058 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.058 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:11.058 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753843+0000", 2024-04-20T17:13:11.058 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.058 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.058 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.058 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:11.058 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753843+0000", 2024-04-20T17:13:11.058 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2200000000000004e-07 2024-04-20T17:13:11.058 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.058 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.058 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:11.058 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753848+0000", 2024-04-20T17:13:11.058 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.454e-06 2024-04-20T17:13:11.058 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753856+0000", 2024-04-20T17:13:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2069999999999995e-06 2024-04-20T17:13:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753861+0000", 2024-04-20T17:13:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1540000000000001e-06 2024-04-20T17:13:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.760194+0000", 2024-04-20T17:13:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0063330349999999999 2024-04-20T17:13:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.760219+0000", 2024-04-20T17:13:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4428999999999999e-05 2024-04-20T17:13:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.768119+0000", 2024-04-20T17:13:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0079001060000000005 2024-04-20T17:13:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.768143+0000", 2024-04-20T17:13:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3937000000000001e-05 2024-04-20T17:13:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:210474 2.d e168/115 2:b8832972:::benchmark_data_smithi171_78918_object49480:head v 168'12950, mlcod=168'12945)", 2024-04-20T17:13:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:48.758784+0000", 2024-04-20T17:13:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 142.21812789000001, 2024-04-20T17:13:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0094639479999999998, 2024-04-20T17:13:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.758784+0000", 2024-04-20T17:13:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.062 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.062 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.062 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:11.062 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.758784+0000", 2024-04-20T17:13:11.062 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:11.062 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.062 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.062 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:11.062 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.758786+0000", 2024-04-20T17:13:11.062 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8309999999999999e-06 2024-04-20T17:13:11.062 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.062 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.062 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:11.062 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.758797+0000", 2024-04-20T17:13:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1024e-05 2024-04-20T17:13:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.758808+0000", 2024-04-20T17:13:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0569e-05 2024-04-20T17:13:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.758816+0000", 2024-04-20T17:13:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3319999999999992e-06 2024-04-20T17:13:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.764056+0000", 2024-04-20T17:13:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0052398640000000003 2024-04-20T17:13:11.064 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.064 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.064 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:11.064 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.764082+0000", 2024-04-20T17:13:11.064 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5616999999999999e-05 2024-04-20T17:13:11.064 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.064 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.064 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:11.064 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.768223+0000", 2024-04-20T17:13:11.064 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0041416029999999998 2024-04-20T17:13:11.064 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:11.064 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:11.064 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:11.064 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.768248+0000", 2024-04-20T17:13:11.064 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5108000000000001e-05 2024-04-20T17:13:11.065 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.065 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:11.065 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.065 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:11.065 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:11.065 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:13:11.065 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:13:11.085 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:13:11.089 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:13:11.092 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:11.112 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:13:11.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:11.117 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:13:11.217 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:13:11.246 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:11.284 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:13:11.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:11.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:13:11.388 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:13:11.415 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:11.733 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-20T17:13:11.762 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:13:11.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:11.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:13:11.867 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:13:11.903 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:11.918 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:13:12.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:11 smithi106 ceph-mon[44423]: pgmap v621: 17 pgs: 1 active+recovering, 16 active+clean; 6.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 267 op/s; 10610/197492 objects misplaced (5.372%) 2024-04-20T17:13:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:11 smithi106 ceph-mon[44423]: osdmap e220: 12 total, 12 up, 12 in 2024-04-20T17:13:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:11 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:13:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:11 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:13:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:11 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1576937689' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:13:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:11 smithi106 ceph-mon[44423]: from='client.25073 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:13:12.028 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:13:12.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:11 smithi005 ceph-mon[39562]: pgmap v621: 17 pgs: 1 active+recovering, 16 active+clean; 6.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 267 op/s; 10610/197492 objects misplaced (5.372%) 2024-04-20T17:13:12.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:11 smithi005 ceph-mon[39562]: osdmap e220: 12 total, 12 up, 12 in 2024-04-20T17:13:12.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:11 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:13:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:11 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:13:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:11 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1576937689' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:13:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:11 smithi005 ceph-mon[39562]: from='client.25073 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:13:12.257 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:13:12.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:12.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:13:12.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:11 smithi171 ceph-mon[44189]: pgmap v621: 17 pgs: 1 active+recovering, 16 active+clean; 6.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 267 op/s; 10610/197492 objects misplaced (5.372%) 2024-04-20T17:13:12.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:11 smithi171 ceph-mon[44189]: osdmap e220: 12 total, 12 up, 12 in 2024-04-20T17:13:12.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:11 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:13:12.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:11 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:13:12.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:11 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1576937689' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:13:12.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:11 smithi171 ceph-mon[44189]: from='client.25073 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:13:12.359 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:13:12.387 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:12.407 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:13:12.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:12.411 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:13:12.512 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:13:12.544 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:12.802 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:13:12.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:12.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:13:12.906 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:13:12.935 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:13.058 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:13:13.060 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:12 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:13:13.060 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:12 smithi005 ceph-mon[39562]: from='client.25073 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:13:13.060 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:12 smithi005 ceph-mon[39562]: osdmap e221: 12 total, 12 up, 12 in 2024-04-20T17:13:13.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:13.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:13:13.162 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:13:13.186 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:12 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:13:13.186 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:12 smithi171 ceph-mon[44189]: from='client.25073 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:13:13.186 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:12 smithi171 ceph-mon[44189]: osdmap e221: 12 total, 12 up, 12 in 2024-04-20T17:13:13.188 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:13.206 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:13:13.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:13.210 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:13:13.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:12 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:13:13.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:12 smithi106 ceph-mon[44423]: from='client.25073 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:13:13.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:12 smithi106 ceph-mon[44423]: osdmap e221: 12 total, 12 up, 12 in 2024-04-20T17:13:13.311 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:13:13.340 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:13.362 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:13:13.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:13.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:13:13.466 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:13:13.493 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:13.518 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:13:13.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:13.522 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:13:13.623 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:13:13.650 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:13.672 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:13:13.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:13.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:13:13.777 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:13:13.805 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:13.824 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:13:13.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:13.827 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:13:13.928 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:13:13.958 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:13.990 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:13:13.990 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:13:13.990 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-04-20T17:13:13.990 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:13:14.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:13 smithi171 ceph-mon[44189]: pgmap v624: 17 pgs: 1 active+recovering, 16 active+clean; 6.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 217 op/s; 10610/196654 objects misplaced (5.395%) 2024-04-20T17:13:14.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:13 smithi171 ceph-mon[44189]: 2.6 scrub starts 2024-04-20T17:13:14.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:13 smithi171 ceph-mon[44189]: 2.6 scrub ok 2024-04-20T17:13:14.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:13 smithi005 ceph-mon[39562]: pgmap v624: 17 pgs: 1 active+recovering, 16 active+clean; 6.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 217 op/s; 10610/196654 objects misplaced (5.395%) 2024-04-20T17:13:14.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:13 smithi005 ceph-mon[39562]: 2.6 scrub starts 2024-04-20T17:13:14.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:13 smithi005 ceph-mon[39562]: 2.6 scrub ok 2024-04-20T17:13:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:13 smithi106 ceph-mon[44423]: pgmap v624: 17 pgs: 1 active+recovering, 16 active+clean; 6.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 217 op/s; 10610/196654 objects misplaced (5.395%) 2024-04-20T17:13:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:13 smithi106 ceph-mon[44423]: 2.6 scrub starts 2024-04-20T17:13:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:13 smithi106 ceph-mon[44423]: 2.6 scrub ok 2024-04-20T17:13:14.739 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-20T17:13:15.004 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:13:15.708 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:13:15.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:15.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:13:15.816 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:13:15.843 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:15.865 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:13:15.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:15.868 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:13:15.913 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:15 smithi106 ceph-mon[44423]: pgmap v625: 17 pgs: 1 active+recovering, 16 active+clean; 6.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 223 op/s; 10610/195238 objects misplaced (5.434%) 2024-04-20T17:13:15.913 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:15 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:13:15.913 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:15 smithi106 ceph-mon[44423]: osdmap e222: 12 total, 12 up, 12 in 2024-04-20T17:13:15.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:15 smithi005 ceph-mon[39562]: pgmap v625: 17 pgs: 1 active+recovering, 16 active+clean; 6.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 223 op/s; 10610/195238 objects misplaced (5.434%) 2024-04-20T17:13:15.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:15 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:13:15.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:15 smithi005 ceph-mon[39562]: osdmap e222: 12 total, 12 up, 12 in 2024-04-20T17:13:15.969 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:13:15.995 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:16.019 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:13:16.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:16.023 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:13:16.124 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:13:16.150 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:15 smithi171 ceph-mon[44189]: pgmap v625: 17 pgs: 1 active+recovering, 16 active+clean; 6.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 223 op/s; 10610/195238 objects misplaced (5.434%) 2024-04-20T17:13:16.151 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:15 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:13:16.151 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:15 smithi171 ceph-mon[44189]: osdmap e222: 12 total, 12 up, 12 in 2024-04-20T17:13:16.154 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:16.174 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:13:16.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:16.180 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:13:16.281 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:13:16.313 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:17.005 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:13:17.005 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:13:17.005 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-04-20T17:13:17.005 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-04-20T17:13:17.005 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:13:17.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:16 smithi106 ceph-mon[44423]: 2.9 scrub starts 2024-04-20T17:13:17.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:16 smithi106 ceph-mon[44423]: pgmap v627: 17 pgs: 1 active+recovering, 16 active+clean; 5.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 254 op/s; 10610/194434 objects misplaced (5.457%) 2024-04-20T17:13:17.056 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:13:17.056 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:13:17.057 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:13:17.057 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:13:17.057 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:13:17.062 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:13:17.069 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:13:17.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:17.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:13:17.108 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:13:17.129 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:13:17.149 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:13:17.172 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:13:17.186 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:13:17.196 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:16 smithi171 ceph-mon[44189]: 2.9 scrub starts 2024-04-20T17:13:17.196 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:16 smithi171 ceph-mon[44189]: pgmap v627: 17 pgs: 1 active+recovering, 16 active+clean; 5.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 254 op/s; 10610/194434 objects misplaced (5.457%) 2024-04-20T17:13:17.198 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:17.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:16 smithi005 ceph-mon[39562]: 2.9 scrub starts 2024-04-20T17:13:17.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:16 smithi005 ceph-mon[39562]: pgmap v627: 17 pgs: 1 active+recovering, 16 active+clean; 5.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 254 op/s; 10610/194434 objects misplaced (5.457%) 2024-04-20T17:13:17.217 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:13:17.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:17.220 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:13:17.222 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:13:17.244 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:13:17.246 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:13:17.246 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:13:17.246 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:13:17.246 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:13:17.246 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:13:17.247 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:13:17.247 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:13:17.247 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:13:17.247 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:13:17.247 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:13:17.247 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:13:17.247 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:13:17.321 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:13:17.353 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:17.554 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:13:17.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:17.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:13:17.658 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:13:17.688 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:17.801 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-20T17:13:17.923 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.4 to 1 (10000) 2024-04-20T17:13:18.078 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:13:18.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:17 smithi005 ceph-mon[39562]: 2.9 scrub ok 2024-04-20T17:13:18.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:17 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3465282527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:13:18.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:17 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2718715566' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-20T17:13:18.218 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:13:18.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:17 smithi106 ceph-mon[44423]: 2.9 scrub ok 2024-04-20T17:13:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:17 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3465282527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:13:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:17 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2718715566' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-20T17:13:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:17 smithi171 ceph-mon[44189]: 2.9 scrub ok 2024-04-20T17:13:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:17 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3465282527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:13:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:18.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:17 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2718715566' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-20T17:13:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:18 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2718715566' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-20T17:13:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:18 smithi005 ceph-mon[39562]: osdmap e223: 12 total, 12 up, 12 in 2024-04-20T17:13:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:18 smithi005 ceph-mon[39562]: pgmap v629: 17 pgs: 1 active+recovering, 16 active+clean; 5.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 300 op/s; 10610/193136 objects misplaced (5.494%) 2024-04-20T17:13:19.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:18 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2718715566' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-20T17:13:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:18 smithi106 ceph-mon[44423]: osdmap e223: 12 total, 12 up, 12 in 2024-04-20T17:13:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:18 smithi106 ceph-mon[44423]: pgmap v629: 17 pgs: 1 active+recovering, 16 active+clean; 5.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 300 op/s; 10610/193136 objects misplaced (5.494%) 2024-04-20T17:13:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:18 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2718715566' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-20T17:13:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:18 smithi171 ceph-mon[44189]: osdmap e223: 12 total, 12 up, 12 in 2024-04-20T17:13:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:18 smithi171 ceph-mon[44189]: pgmap v629: 17 pgs: 1 active+recovering, 16 active+clean; 5.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 300 op/s; 10610/193136 objects misplaced (5.494%) 2024-04-20T17:13:19.933 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 5 1 2024-04-20T17:13:19.939 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:13:19.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:19.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:13:20.047 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:13:20.067 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:13:20.068 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:13:20.076 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:13:20.081 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:20.102 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:13:20.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:20.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:13:20.207 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:13:20.236 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:20.391 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:13:20.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:20.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:13:20.496 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:13:20.528 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:20.823 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:13:20.871 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-20T17:13:21.143 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:13:21.181 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:13:21.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:21.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:13:21.185 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:21 smithi005 ceph-mon[39562]: 2.8 scrub starts 2024-04-20T17:13:21.185 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:21 smithi005 ceph-mon[39562]: pgmap v630: 17 pgs: 1 active+recovering, 16 active+clean; 5.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 251 op/s; 10610/192626 objects misplaced (5.508%) 2024-04-20T17:13:21.185 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:21 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:13:21.185 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:21 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:13:21.185 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:21 smithi005 ceph-mon[39562]: osdmap e224: 12 total, 12 up, 12 in 2024-04-20T17:13:21.221 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:13:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:21 smithi106 ceph-mon[44423]: 2.8 scrub starts 2024-04-20T17:13:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:21 smithi106 ceph-mon[44423]: pgmap v630: 17 pgs: 1 active+recovering, 16 active+clean; 5.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 251 op/s; 10610/192626 objects misplaced (5.508%) 2024-04-20T17:13:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:21 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:13:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:21 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:13:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:21 smithi106 ceph-mon[44423]: osdmap e224: 12 total, 12 up, 12 in 2024-04-20T17:13:21.285 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:13:21.312 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:21 smithi171 ceph-mon[44189]: 2.8 scrub starts 2024-04-20T17:13:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:21 smithi171 ceph-mon[44189]: pgmap v630: 17 pgs: 1 active+recovering, 16 active+clean; 5.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 251 op/s; 10610/192626 objects misplaced (5.508%) 2024-04-20T17:13:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:21 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:13:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:21 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:13:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:21 smithi171 ceph-mon[44189]: osdmap e224: 12 total, 12 up, 12 in 2024-04-20T17:13:21.461 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:13:21.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:21.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:13:21.565 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:13:21.594 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:22.042 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:13:22.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:22.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:13:22.147 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:13:22.174 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:22.214 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:13:22.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:22.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:13:22.323 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:13:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:22 smithi171 ceph-mon[44189]: 2.8 scrub ok 2024-04-20T17:13:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:22 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:13:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:22 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:13:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:22 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:13:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:22 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:13:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:22 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:13:22.350 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:22 smithi005 ceph-mon[39562]: 2.8 scrub ok 2024-04-20T17:13:22.350 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:22 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:13:22.350 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:22 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:13:22.350 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:22 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:13:22.350 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:22 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:13:22.351 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:22 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:13:22.353 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:22.405 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:13:22.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:22 smithi106 ceph-mon[44423]: 2.8 scrub ok 2024-04-20T17:13:22.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:22 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:13:22.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:22 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:13:22.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:22 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:13:22.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:22 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:13:22.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:22 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:13:23.195 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:13:23.196 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:13:23.225 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:23 smithi106 ceph-mon[44423]: pgmap v632: 17 pgs: 1 active+recovering, 16 active+clean; 5.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 247 op/s; 10610/191598 objects misplaced (5.538%) 2024-04-20T17:13:23.304 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:13:23.307 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:23 smithi005 ceph-mon[39562]: pgmap v632: 17 pgs: 1 active+recovering, 16 active+clean; 5.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 247 op/s; 10610/191598 objects misplaced (5.538%) 2024-04-20T17:13:23.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:23.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:13:23.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:23 smithi171 ceph-mon[44189]: pgmap v632: 17 pgs: 1 active+recovering, 16 active+clean; 5.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 247 op/s; 10610/191598 objects misplaced (5.538%) 2024-04-20T17:13:23.409 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:13:23.442 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:23.923 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-20T17:13:24.204 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:13:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:24 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:13:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:24 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:13:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:24 smithi005 ceph-mon[39562]: osdmap e225: 12 total, 12 up, 12 in 2024-04-20T17:13:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:24 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:13:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:24 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:13:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:24 smithi106 ceph-mon[44423]: osdmap e225: 12 total, 12 up, 12 in 2024-04-20T17:13:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:24 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:13:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:24 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:13:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:24 smithi171 ceph-mon[44189]: osdmap e225: 12 total, 12 up, 12 in 2024-04-20T17:13:24.894 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:13:24.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:24.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:13:25.000 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:13:25.032 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:25.085 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:13:25.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:25.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:13:25.097 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:13:25.189 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:13:25.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:25 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:13:25.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:25 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:13:25.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:25 smithi005 ceph-mon[39562]: pgmap v634: 17 pgs: 1 active+recovering, 16 active+clean; 5.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 301 op/s; 10610/189642 objects misplaced (5.595%) 2024-04-20T17:13:25.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:25 smithi005 ceph-mon[39562]: osdmap e226: 12 total, 12 up, 12 in 2024-04-20T17:13:25.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:25 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3751712688' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:13:25.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:25 smithi005 ceph-mon[39562]: from='client.25091 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:13:25.215 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:25.236 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:13:25.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:25.239 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:13:25.310 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:25 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:13:25.310 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:25 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:13:25.310 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:25 smithi106 ceph-mon[44423]: pgmap v634: 17 pgs: 1 active+recovering, 16 active+clean; 5.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 301 op/s; 10610/189642 objects misplaced (5.595%) 2024-04-20T17:13:25.310 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:25 smithi106 ceph-mon[44423]: osdmap e226: 12 total, 12 up, 12 in 2024-04-20T17:13:25.310 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:25 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3751712688' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:13:25.310 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:25 smithi106 ceph-mon[44423]: from='client.25091 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:13:25.340 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:13:25.366 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:25 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:13:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:25 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:13:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:25 smithi171 ceph-mon[44189]: pgmap v634: 17 pgs: 1 active+recovering, 16 active+clean; 5.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 301 op/s; 10610/189642 objects misplaced (5.595%) 2024-04-20T17:13:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:25 smithi171 ceph-mon[44189]: osdmap e226: 12 total, 12 up, 12 in 2024-04-20T17:13:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:25 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3751712688' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:13:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:25 smithi171 ceph-mon[44189]: from='client.25091 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:13:25.715 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:13:25.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:25.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:13:25.820 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:13:25.847 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:25.865 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:13:25.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:25.868 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:13:25.969 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:13:25.998 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:26.095 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:13:26.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:26.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:13:26.155 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.5 to 1 (10000) 2024-04-20T17:13:26.199 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:13:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:26 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:13:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:26 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:13:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:26 smithi005 ceph-mon[39562]: from='client.25091 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:13:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:26 smithi005 ceph-mon[39562]: osdmap e227: 12 total, 12 up, 12 in 2024-04-20T17:13:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:26 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/819463382' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:13:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:26 smithi005 ceph-mon[39562]: from='client.25097 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:13:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:26 smithi005 ceph-mon[39562]: from='client.25097 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-20T17:13:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:26 smithi005 ceph-mon[39562]: osdmap e228: 12 total, 12 up, 12 in 2024-04-20T17:13:26.229 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:13:26.229 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:13:26.230 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:26.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:26 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:13:26.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:26 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:13:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:26 smithi106 ceph-mon[44423]: from='client.25091 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:13:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:26 smithi106 ceph-mon[44423]: osdmap e227: 12 total, 12 up, 12 in 2024-04-20T17:13:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:26 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/819463382' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:13:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:26 smithi106 ceph-mon[44423]: from='client.25097 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:13:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:26 smithi106 ceph-mon[44423]: from='client.25097 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-20T17:13:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:26 smithi106 ceph-mon[44423]: osdmap e228: 12 total, 12 up, 12 in 2024-04-20T17:13:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:26 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:13:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:26 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:13:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:26 smithi171 ceph-mon[44189]: from='client.25091 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:13:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:26 smithi171 ceph-mon[44189]: osdmap e227: 12 total, 12 up, 12 in 2024-04-20T17:13:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:26 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/819463382' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:13:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:26 smithi171 ceph-mon[44189]: from='client.25097 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:13:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:26 smithi171 ceph-mon[44189]: from='client.25097 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-20T17:13:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:26 smithi171 ceph-mon[44189]: osdmap e228: 12 total, 12 up, 12 in 2024-04-20T17:13:26.880 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:13:26.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:26.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:13:26.899 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 6 1 2024-04-20T17:13:26.985 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:13:26.988 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-20T17:13:27.012 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:27.031 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:13:27.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:27.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:13:27.135 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:13:27.136 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:27 smithi005 ceph-mon[39562]: pgmap v638: 17 pgs: 1 active+recovering, 16 active+clean; 5.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 336 op/s; 10610/188904 objects misplaced (5.617%) 2024-04-20T17:13:27.136 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:27 smithi005 ceph-mon[39562]: 2.2 scrub starts 2024-04-20T17:13:27.161 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:27.180 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:13:27.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:27.183 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:13:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:27 smithi106 ceph-mon[44423]: pgmap v638: 17 pgs: 1 active+recovering, 16 active+clean; 5.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 336 op/s; 10610/188904 objects misplaced (5.617%) 2024-04-20T17:13:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:27 smithi106 ceph-mon[44423]: 2.2 scrub starts 2024-04-20T17:13:27.272 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:13:27.284 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:13:27.310 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:27.330 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:13:27.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:27.333 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:13:27.434 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:13:27.460 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:27.486 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:13:27.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:27.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:13:27.540 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:13:27.540 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:13:27.540 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:13:27.540 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:13:27.560 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:13:27.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:27 smithi171 ceph-mon[44189]: pgmap v638: 17 pgs: 1 active+recovering, 16 active+clean; 5.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 336 op/s; 10610/188904 objects misplaced (5.617%) 2024-04-20T17:13:27.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:27 smithi171 ceph-mon[44189]: 2.2 scrub starts 2024-04-20T17:13:27.587 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:13:27.590 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:13:27.609 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:13:27.622 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:27.631 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:13:27.653 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:13:27.672 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:13:27.693 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:13:27.718 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:13:27.719 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:13:27.719 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:13:27.719 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:13:27.719 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:13:27.719 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:13:27.719 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:13:27.719 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:13:27.720 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:13:27.720 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:13:27.720 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:13:27.720 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:13:27.720 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:13:28.173 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:13:28.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:28.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:13:28.191 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:13:28.278 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:13:28.305 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:28.323 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:13:28.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:28.326 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:13:28.427 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:13:28.451 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:28 smithi171 ceph-mon[44189]: 2.2 scrub ok 2024-04-20T17:13:28.451 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:28 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1502540601' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:13:28.451 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:28.451 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:28.451 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:28.451 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:28.452 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:28.453 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:28.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:28 smithi005 ceph-mon[39562]: 2.2 scrub ok 2024-04-20T17:13:28.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:28 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1502540601' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:13:28.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:28.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:28.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:28.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:28.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:28.471 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:13:28.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:28.474 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:13:28.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:28 smithi106 ceph-mon[44423]: 2.2 scrub ok 2024-04-20T17:13:28.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:28 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1502540601' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:13:28.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:28.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:28.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:28.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:28.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:28.575 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:13:28.609 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:29.260 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:13:29.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:29.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:13:29.364 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:13:29.390 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:29.390 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:29.390 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:29.390 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:29.391 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:29.391 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:29.391 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:29.391 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:29 smithi005 ceph-mon[39562]: pgmap v639: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 220 op/s; 10610/188194 objects misplaced (5.638%) 2024-04-20T17:13:29.393 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:29 smithi106 ceph-mon[44423]: pgmap v639: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 220 op/s; 10610/188194 objects misplaced (5.638%) 2024-04-20T17:13:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:29 smithi171 ceph-mon[44189]: pgmap v639: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 220 op/s; 10610/188194 objects misplaced (5.638%) 2024-04-20T17:13:30.186 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:13:30.242 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:13:30.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:30.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:13:30.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:30 smithi106 ceph-mon[44423]: 2.7 scrub starts 2024-04-20T17:13:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:30 smithi106 ceph-mon[44423]: 2.f scrub starts 2024-04-20T17:13:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:30 smithi106 ceph-mon[44423]: 2.7 scrub ok 2024-04-20T17:13:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:30 smithi106 ceph-mon[44423]: 2.f scrub ok 2024-04-20T17:13:30.346 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:13:30.377 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:30.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:30 smithi005 ceph-mon[39562]: 2.7 scrub starts 2024-04-20T17:13:30.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:30 smithi005 ceph-mon[39562]: 2.f scrub starts 2024-04-20T17:13:30.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:30 smithi005 ceph-mon[39562]: 2.7 scrub ok 2024-04-20T17:13:30.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:30 smithi005 ceph-mon[39562]: 2.f scrub ok 2024-04-20T17:13:30.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:30 smithi171 ceph-mon[44189]: 2.7 scrub starts 2024-04-20T17:13:30.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:30 smithi171 ceph-mon[44189]: 2.f scrub starts 2024-04-20T17:13:30.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:30 smithi171 ceph-mon[44189]: 2.7 scrub ok 2024-04-20T17:13:30.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:30 smithi171 ceph-mon[44189]: 2.f scrub ok 2024-04-20T17:13:30.631 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:13:30.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:30.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:13:30.734 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:13:30.761 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:30.782 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:13:30.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:30.786 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:13:30.881 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:13:30.887 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:13:30.924 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:30.944 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:13:30.944 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-04-20T17:13:30.944 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-04-20T17:13:30.944 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:13:30.944 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.944 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:197147 2.7 2:ea8daa9e:::benchmark_data_smithi171_78918_object36153:head [delete] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-04-20T17:13:30.945 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:33.934691+0000", 2024-04-20T17:13:30.945 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 177.00815415599999, 2024-04-20T17:13:30.945 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.023606318000000001, 2024-04-20T17:13:30.945 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:30.945 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:30.945 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:30.945 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:13:30.945 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:13:30.945 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:13:30.945 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 197147 2024-04-20T17:13:30.945 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.945 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:30.945 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.945 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:30.945 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.934691+0000", 2024-04-20T17:13:30.946 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:30.946 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.946 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.946 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:30.946 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.934691+0000", 2024-04-20T17:13:30.946 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:30.946 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.946 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.946 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:30.946 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.934692+0000", 2024-04-20T17:13:30.946 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1990000000000001e-06 2024-04-20T17:13:30.946 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.946 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.946 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:30.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.934697+0000", 2024-04-20T17:13:30.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1900000000000003e-06 2024-04-20T17:13:30.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.947 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.947 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:30.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.934706+0000", 2024-04-20T17:13:30.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7609999999999996e-06 2024-04-20T17:13:30.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.947 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.947 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:30.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.934711+0000", 2024-04-20T17:13:30.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9169999999999997e-06 2024-04-20T17:13:30.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.948 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:30.948 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.938530+0000", 2024-04-20T17:13:30.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0038189280000000001 2024-04-20T17:13:30.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.948 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:30.948 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.938597+0000", 2024-04-20T17:13:30.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6685000000000003e-05 2024-04-20T17:13:30.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.948 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:13:30.948 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.938660+0000", 2024-04-20T17:13:30.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3102000000000007e-05 2024-04-20T17:13:30.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.949 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.949 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:13:30.949 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.943755+0000", 2024-04-20T17:13:30.949 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0050954820000000001 2024-04-20T17:13:30.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.949 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.949 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:13:30.949 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.958220+0000", 2024-04-20T17:13:30.949 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.014465107 2024-04-20T17:13:30.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.949 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.949 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:30.949 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.958256+0000", 2024-04-20T17:13:30.950 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5411999999999999e-05 2024-04-20T17:13:30.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.950 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:30.950 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.958297+0000", 2024-04-20T17:13:30.950 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1535000000000001e-05 2024-04-20T17:13:30.950 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:30.950 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:30.950 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:30.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.950 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:197152 2.7 2:e790fa34:::benchmark_data_smithi171_78918_object36158:head [delete] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-04-20T17:13:30.950 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:33.936770+0000", 2024-04-20T17:13:30.950 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 177.006074896, 2024-04-20T17:13:30.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.021672428000000001, 2024-04-20T17:13:30.951 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:30.951 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:30.951 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:30.951 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:13:30.951 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:13:30.951 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:13:30.951 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 197152 2024-04-20T17:13:30.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.951 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:30.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:30.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.936770+0000", 2024-04-20T17:13:30.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:30.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.952 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.952 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:30.952 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.936770+0000", 2024-04-20T17:13:30.952 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:30.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.952 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.952 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:30.952 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.936772+0000", 2024-04-20T17:13:30.952 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.28e-06 2024-04-20T17:13:30.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.952 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.952 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:30.952 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.936776+0000", 2024-04-20T17:13:30.952 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.605e-06 2024-04-20T17:13:30.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.953 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:30.953 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.936788+0000", 2024-04-20T17:13:30.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1438e-05 2024-04-20T17:13:30.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.953 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:30.953 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.936793+0000", 2024-04-20T17:13:30.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7790000000000001e-06 2024-04-20T17:13:30.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.953 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:30.953 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.941066+0000", 2024-04-20T17:13:30.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.004272517 2024-04-20T17:13:30.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:30.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.941167+0000", 2024-04-20T17:13:30.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000101486 2024-04-20T17:13:30.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:13:30.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.941236+0000", 2024-04-20T17:13:30.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8788999999999995e-05 2024-04-20T17:13:30.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:13:30.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.943915+0000", 2024-04-20T17:13:30.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0026784880000000001 2024-04-20T17:13:30.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:13:30.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.958350+0000", 2024-04-20T17:13:30.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.014435784 2024-04-20T17:13:30.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:30.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.958402+0000", 2024-04-20T17:13:30.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1932999999999999e-05 2024-04-20T17:13:30.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:30.956 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.958443+0000", 2024-04-20T17:13:30.956 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0329e-05 2024-04-20T17:13:30.956 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:30.956 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:30.956 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:30.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.956 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.956 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:197920 2.7 2:e55163d9:::benchmark_data_smithi171_78918_object36926:head [delete] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-04-20T17:13:30.956 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:34.682047+0000", 2024-04-20T17:13:30.956 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 176.26079839900001, 2024-04-20T17:13:30.956 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.022660026, 2024-04-20T17:13:30.956 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:30.956 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:30.956 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:30.956 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:13:30.956 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:13:30.957 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:13:30.957 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 197920 2024-04-20T17:13:30.957 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.957 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:30.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.957 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:30.957 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.682047+0000", 2024-04-20T17:13:30.957 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:30.957 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.957 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:30.957 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.682047+0000", 2024-04-20T17:13:30.957 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:30.957 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.958 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:30.958 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.682048+0000", 2024-04-20T17:13:30.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.381e-06 2024-04-20T17:13:30.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.958 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:30.958 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.682064+0000", 2024-04-20T17:13:30.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6047000000000001e-05 2024-04-20T17:13:30.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.958 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:30.958 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.682074+0000", 2024-04-20T17:13:30.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0039e-05 2024-04-20T17:13:30.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:30.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.682082+0000", 2024-04-20T17:13:30.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8560000000000007e-06 2024-04-20T17:13:30.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:30.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.686324+0000", 2024-04-20T17:13:30.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0042415380000000004 2024-04-20T17:13:30.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:30.960 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.686460+0000", 2024-04-20T17:13:30.960 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000136618 2024-04-20T17:13:30.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.960 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.960 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:13:30.960 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.686596+0000", 2024-04-20T17:13:30.960 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013620200000000001 2024-04-20T17:13:30.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.960 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.960 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:13:30.960 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.691247+0000", 2024-04-20T17:13:30.960 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0046507700000000003 2024-04-20T17:13:30.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.960 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.961 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:13:30.961 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.704602+0000", 2024-04-20T17:13:30.961 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.013354672 2024-04-20T17:13:30.961 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.961 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.961 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:30.961 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.704658+0000", 2024-04-20T17:13:30.961 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5721999999999999e-05 2024-04-20T17:13:30.961 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.961 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.961 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:30.961 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.704707+0000", 2024-04-20T17:13:30.961 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9181000000000002e-05 2024-04-20T17:13:30.961 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:30.961 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:30.962 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:30.962 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.962 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.962 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:203733 2.7 2:e2dc7fc6:::benchmark_data_smithi171_78918_object42739:head [delete] snapc 0=[] ondisk+write+known_if_redirected e167)", 2024-04-20T17:13:30.962 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:41.382712+0000", 2024-04-20T17:13:30.962 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 169.56013270099999, 2024-04-20T17:13:30.962 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.024753645000000001, 2024-04-20T17:13:30.962 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:30.962 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:30.962 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:30.962 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:13:30.962 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:13:30.962 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:13:30.963 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 203733 2024-04-20T17:13:30.963 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.963 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:30.963 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.963 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:30.963 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.382712+0000", 2024-04-20T17:13:30.963 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:30.963 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.963 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.963 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:30.963 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.382712+0000", 2024-04-20T17:13:30.963 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:30.963 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.963 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.963 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:30.964 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.382715+0000", 2024-04-20T17:13:30.964 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2139999999999999e-06 2024-04-20T17:13:30.964 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.964 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.964 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:30.964 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.382722+0000", 2024-04-20T17:13:30.964 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9310000000000002e-06 2024-04-20T17:13:30.964 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.964 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.964 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:30.964 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.382736+0000", 2024-04-20T17:13:30.964 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4786999999999999e-05 2024-04-20T17:13:30.964 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.964 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.964 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:30.965 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.382747+0000", 2024-04-20T17:13:30.965 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0186999999999999e-05 2024-04-20T17:13:30.965 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.965 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.965 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:30.965 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.382815+0000", 2024-04-20T17:13:30.965 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8223000000000001e-05 2024-04-20T17:13:30.965 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.965 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.965 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:30.965 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.382955+0000", 2024-04-20T17:13:30.965 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013996599999999999 2024-04-20T17:13:30.965 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.965 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.966 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:13:30.966 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.383060+0000", 2024-04-20T17:13:30.966 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010482800000000001 2024-04-20T17:13:30.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.966 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.966 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:13:30.966 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.389788+0000", 2024-04-20T17:13:30.966 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.006728365 2024-04-20T17:13:30.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.966 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.966 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:13:30.966 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.407360+0000", 2024-04-20T17:13:30.966 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.017572042999999999 2024-04-20T17:13:30.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.966 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.967 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:30.967 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.407421+0000", 2024-04-20T17:13:30.967 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1248999999999996e-05 2024-04-20T17:13:30.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.967 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.967 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:30.967 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.407466+0000", 2024-04-20T17:13:30.967 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4851999999999997e-05 2024-04-20T17:13:30.967 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:30.967 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:30.967 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:30.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.967 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.967 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:203740 2.7 2:ee294609:::benchmark_data_smithi171_78918_object42746:head [delete] snapc 0=[] ondisk+write+known_if_redirected e167)", 2024-04-20T17:13:30.968 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:41.383679+0000", 2024-04-20T17:13:30.968 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 169.55916651800001, 2024-04-20T17:13:30.968 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.023936742, 2024-04-20T17:13:30.968 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:30.968 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:30.968 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:30.968 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:13:30.968 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:13:30.968 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:13:30.968 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 203740 2024-04-20T17:13:30.968 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.968 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:30.968 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.968 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:30.968 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.383679+0000", 2024-04-20T17:13:30.968 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:30.969 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.969 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.969 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:30.969 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.383679+0000", 2024-04-20T17:13:30.969 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:30.969 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.969 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.969 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:30.969 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.383680+0000", 2024-04-20T17:13:30.969 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5850000000000001e-06 2024-04-20T17:13:30.969 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.969 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.969 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:30.969 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.383685+0000", 2024-04-20T17:13:30.969 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7720000000000002e-06 2024-04-20T17:13:30.970 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.970 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.970 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:30.970 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.383699+0000", 2024-04-20T17:13:30.970 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3696999999999999e-05 2024-04-20T17:13:30.970 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.970 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.970 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:30.970 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.383707+0000", 2024-04-20T17:13:30.970 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2050000000000002e-06 2024-04-20T17:13:30.970 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.970 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.970 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:30.970 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.386373+0000", 2024-04-20T17:13:30.971 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0026656520000000001 2024-04-20T17:13:30.971 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.971 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.971 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:30.971 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.386490+0000", 2024-04-20T17:13:30.971 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000117297 2024-04-20T17:13:30.971 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.971 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.971 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:13:30.971 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.386589+0000", 2024-04-20T17:13:30.971 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8766999999999998e-05 2024-04-20T17:13:30.971 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.971 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.971 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:13:30.971 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.389920+0000", 2024-04-20T17:13:30.972 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003331582 2024-04-20T17:13:30.972 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.972 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.972 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:13:30.972 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.407515+0000", 2024-04-20T17:13:30.972 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.017594957000000001 2024-04-20T17:13:30.972 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.972 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.972 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:30.972 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.407574+0000", 2024-04-20T17:13:30.972 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9120000000000003e-05 2024-04-20T17:13:30.972 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.972 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.972 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:30.973 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.407615+0000", 2024-04-20T17:13:30.973 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1108e-05 2024-04-20T17:13:30.973 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:30.973 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:30.973 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:30.973 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.973 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.973 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:205043 2.7 2:ef4284f4:::benchmark_data_smithi171_78918_object44049:head [delete] snapc 0=[] ondisk+write+known_if_redirected e167)", 2024-04-20T17:13:30.973 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:42.749577+0000", 2024-04-20T17:13:30.973 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 168.19326835199999, 2024-04-20T17:13:30.973 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.023824217000000002, 2024-04-20T17:13:30.973 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:30.973 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:30.973 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:30.974 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:13:30.974 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:13:30.974 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:13:30.974 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 205043 2024-04-20T17:13:30.974 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.974 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:30.974 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.974 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:30.974 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:42.749577+0000", 2024-04-20T17:13:30.974 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:30.974 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.974 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.974 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:30.974 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:42.749577+0000", 2024-04-20T17:13:30.975 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:30.975 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.975 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.975 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:30.975 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:42.749578+0000", 2024-04-20T17:13:30.975 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.604e-06 2024-04-20T17:13:30.975 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.975 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.975 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:30.975 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:42.749584+0000", 2024-04-20T17:13:30.975 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1699999999999996e-06 2024-04-20T17:13:30.975 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.975 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.975 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:30.975 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:42.749594+0000", 2024-04-20T17:13:30.976 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0852e-05 2024-04-20T17:13:30.976 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.976 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.976 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:30.976 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:42.749604+0000", 2024-04-20T17:13:30.976 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4190000000000008e-06 2024-04-20T17:13:30.976 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.976 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.976 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:30.976 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:42.757543+0000", 2024-04-20T17:13:30.976 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0079390180000000008 2024-04-20T17:13:30.976 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.976 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.976 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:30.976 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:42.757669+0000", 2024-04-20T17:13:30.977 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012637799999999999 2024-04-20T17:13:30.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.977 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:13:30.977 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:42.757758+0000", 2024-04-20T17:13:30.977 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8449999999999995e-05 2024-04-20T17:13:30.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.977 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:13:30.977 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:42.761678+0000", 2024-04-20T17:13:30.977 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003920148 2024-04-20T17:13:30.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.977 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:13:30.977 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:42.773299+0000", 2024-04-20T17:13:30.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011621444999999999 2024-04-20T17:13:30.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.978 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:30.978 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:42.773356+0000", 2024-04-20T17:13:30.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6674e-05 2024-04-20T17:13:30.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.978 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:30.978 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:42.773401+0000", 2024-04-20T17:13:30.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5058999999999997e-05 2024-04-20T17:13:30.978 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:30.978 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:30.978 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:30.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.979 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:16901 2.1 e208/196 2:881f254a:::benchmark_data_smithi171_78918_object154004:head v 208'16974, mlcod=208'16964)", 2024-04-20T17:13:30.979 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:38.734500+0000", 2024-04-20T17:13:30.979 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 52.208345004000002, 2024-04-20T17:13:30.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.021894659, 2024-04-20T17:13:30.979 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:30.979 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:30.979 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:30.979 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:30.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.979 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:30.979 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.734500+0000", 2024-04-20T17:13:30.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:30.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:30.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.734500+0000", 2024-04-20T17:13:30.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:30.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:30.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.734501+0000", 2024-04-20T17:13:30.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.31e-06 2024-04-20T17:13:30.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:30.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.734508+0000", 2024-04-20T17:13:30.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9380000000000003e-06 2024-04-20T17:13:30.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:30.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.734517+0000", 2024-04-20T17:13:30.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9239999999999996e-06 2024-04-20T17:13:30.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:30.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.734524+0000", 2024-04-20T17:13:30.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0949999999999998e-06 2024-04-20T17:13:30.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:30.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.739316+0000", 2024-04-20T17:13:30.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0047910909999999999 2024-04-20T17:13:30.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.982 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:30.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.739338+0000", 2024-04-20T17:13:30.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.279e-05 2024-04-20T17:13:30.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.982 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:30.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.756351+0000", 2024-04-20T17:13:30.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.017012994 2024-04-20T17:13:30.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.982 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:30.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.756395+0000", 2024-04-20T17:13:30.983 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3516999999999997e-05 2024-04-20T17:13:30.983 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:30.983 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:30.983 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:30.983 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.983 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.983 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:16904 2.1 e208/196 2:88200d33:::benchmark_data_smithi171_78918_object94734:head v 208'16976, mlcod=208'16964)", 2024-04-20T17:13:30.983 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:38.739676+0000", 2024-04-20T17:13:30.983 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 52.203169363999997, 2024-04-20T17:13:30.983 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.026299711999999999, 2024-04-20T17:13:30.983 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:30.983 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:30.983 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:30.983 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:30.983 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.983 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:30.984 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.739676+0000", 2024-04-20T17:13:30.984 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:30.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.984 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:30.984 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.739676+0000", 2024-04-20T17:13:30.984 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:30.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.984 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:30.984 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.739677+0000", 2024-04-20T17:13:30.984 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.612e-06 2024-04-20T17:13:30.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:30.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.739685+0000", 2024-04-20T17:13:30.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8790000000000003e-06 2024-04-20T17:13:30.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:30.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.739697+0000", 2024-04-20T17:13:30.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1344e-05 2024-04-20T17:13:30.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:30.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.739705+0000", 2024-04-20T17:13:30.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9039999999999992e-06 2024-04-20T17:13:30.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.986 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:30.986 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.744115+0000", 2024-04-20T17:13:30.986 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0044101619999999996 2024-04-20T17:13:30.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.986 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.986 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:30.986 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.744137+0000", 2024-04-20T17:13:30.986 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2812000000000001e-05 2024-04-20T17:13:30.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.986 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.986 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:30.986 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.765928+0000", 2024-04-20T17:13:30.986 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.021790990999999999 2024-04-20T17:13:30.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.987 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.987 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:30.987 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.765975+0000", 2024-04-20T17:13:30.987 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7008000000000001e-05 2024-04-20T17:13:30.987 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:30.987 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:30.987 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:30.987 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.987 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.987 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:16905 2.1 e208/196 2:88224025:::benchmark_data_smithi171_78918_object72445:head v 208'16977, mlcod=208'16964)", 2024-04-20T17:13:30.987 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:38.742067+0000", 2024-04-20T17:13:30.987 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 52.200777725000002, 2024-04-20T17:13:30.987 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.023927125, 2024-04-20T17:13:30.987 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:30.987 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:30.988 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:30.988 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:30.988 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.988 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:30.988 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.742067+0000", 2024-04-20T17:13:30.988 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:30.988 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.988 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.988 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:30.988 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.742067+0000", 2024-04-20T17:13:30.988 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:30.988 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.988 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.988 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:30.989 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.742069+0000", 2024-04-20T17:13:30.989 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.573e-06 2024-04-20T17:13:30.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.989 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.989 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:30.989 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.742079+0000", 2024-04-20T17:13:30.989 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6609999999999995e-06 2024-04-20T17:13:30.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.989 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.989 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:30.989 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.742087+0000", 2024-04-20T17:13:30.989 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1589999999999993e-06 2024-04-20T17:13:30.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.989 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.990 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:30.990 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.742094+0000", 2024-04-20T17:13:30.990 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5650000000000004e-06 2024-04-20T17:13:30.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.990 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.990 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:30.990 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.746479+0000", 2024-04-20T17:13:30.990 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0043843149999999997 2024-04-20T17:13:30.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.990 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.990 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:30.990 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.746505+0000", 2024-04-20T17:13:30.990 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6585e-05 2024-04-20T17:13:30.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.991 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.991 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:30.991 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.765979+0000", 2024-04-20T17:13:30.991 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.019474207 2024-04-20T17:13:30.991 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.991 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.991 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:30.991 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.765995+0000", 2024-04-20T17:13:30.991 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.506e-05 2024-04-20T17:13:30.991 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:30.991 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:30.991 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:30.991 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.991 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.992 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:16907 2.1 e208/196 2:88241c2a:::benchmark_data_smithi171_78918_object149251:head v 208'16979, mlcod=208'16964)", 2024-04-20T17:13:30.992 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:38.746952+0000", 2024-04-20T17:13:30.992 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 52.195893456999997, 2024-04-20T17:13:30.992 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.025347308999999998, 2024-04-20T17:13:30.992 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:30.992 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:30.992 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:30.992 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:30.992 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.992 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:30.992 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.746952+0000", 2024-04-20T17:13:30.992 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:30.992 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.992 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.993 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:30.993 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.746952+0000", 2024-04-20T17:13:30.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:30.993 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.993 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:30.993 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.746953+0000", 2024-04-20T17:13:30.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6899999999999999e-06 2024-04-20T17:13:30.993 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.993 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:30.993 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.746961+0000", 2024-04-20T17:13:30.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4150000000000002e-06 2024-04-20T17:13:30.993 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.994 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:30.994 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.746971+0000", 2024-04-20T17:13:30.994 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0149999999999999e-05 2024-04-20T17:13:30.994 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.994 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.994 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:30.994 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.746977+0000", 2024-04-20T17:13:30.994 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3169999999999998e-06 2024-04-20T17:13:30.994 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.994 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.994 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:30.994 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.753997+0000", 2024-04-20T17:13:30.994 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0070197419999999998 2024-04-20T17:13:30.995 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.995 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.995 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:30.995 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.754023+0000", 2024-04-20T17:13:30.995 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6067999999999998e-05 2024-04-20T17:13:30.995 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.995 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.995 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:30.995 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.772265+0000", 2024-04-20T17:13:30.995 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.018241908000000001 2024-04-20T17:13:30.995 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.995 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.995 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:30.995 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.772299+0000", 2024-04-20T17:13:30.995 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4019000000000001e-05 2024-04-20T17:13:30.995 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:30.996 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:30.996 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:30.996 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.996 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.996 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:16908 2.1 e208/196 2:8824861e:::benchmark_data_smithi171_78918_object60058:head v 208'16980, mlcod=208'16964)", 2024-04-20T17:13:30.996 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:38.749127+0000", 2024-04-20T17:13:30.996 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 52.193717992000003, 2024-04-20T17:13:30.996 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.023192629999999999, 2024-04-20T17:13:30.996 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:30.996 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:30.996 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:30.996 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:30.996 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.996 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:30.997 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.749127+0000", 2024-04-20T17:13:30.997 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:30.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.997 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:30.997 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.749127+0000", 2024-04-20T17:13:30.997 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:30.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.997 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:30.997 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.749129+0000", 2024-04-20T17:13:30.997 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8059999999999999e-06 2024-04-20T17:13:30.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.997 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:30.997 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.749138+0000", 2024-04-20T17:13:30.997 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5660000000000007e-06 2024-04-20T17:13:30.998 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.998 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:30.998 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.749150+0000", 2024-04-20T17:13:30.998 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2497000000000001e-05 2024-04-20T17:13:30.998 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.998 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:30.998 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.749157+0000", 2024-04-20T17:13:30.998 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8589999999999999e-06 2024-04-20T17:13:30.998 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.998 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:30.998 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.756470+0000", 2024-04-20T17:13:30.999 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0073132900000000001 2024-04-20T17:13:30.999 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.999 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.999 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:30.999 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.756488+0000", 2024-04-20T17:13:30.999 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7677999999999999e-05 2024-04-20T17:13:30.999 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.999 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.999 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:30.999 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.772303+0000", 2024-04-20T17:13:30.999 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015815175000000001 2024-04-20T17:13:30.999 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:30.999 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:30.999 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:30.999 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.772320+0000", 2024-04-20T17:13:30.999 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6759e-05 2024-04-20T17:13:31.000 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:31.000 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:31.000 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:31.000 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.000 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.000 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:16910 2.1 e208/196 2:882582ca:::benchmark_data_smithi171_78918_object136366:head v 208'16982, mlcod=208'16964)", 2024-04-20T17:13:31.000 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:38.753684+0000", 2024-04-20T17:13:31.000 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 52.189161284999997, 2024-04-20T17:13:31.000 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.023972904, 2024-04-20T17:13:31.000 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:31.000 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:31.000 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:31.000 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:31.000 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.000 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:31.000 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.753684+0000", 2024-04-20T17:13:31.001 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:31.001 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.001 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.001 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:31.001 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.753684+0000", 2024-04-20T17:13:31.001 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:31.001 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.001 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.001 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:31.001 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.753685+0000", 2024-04-20T17:13:31.001 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5200000000000001e-06 2024-04-20T17:13:31.001 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.001 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.001 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:31.002 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.753692+0000", 2024-04-20T17:13:31.002 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0140000000000001e-06 2024-04-20T17:13:31.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.002 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:31.002 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.753703+0000", 2024-04-20T17:13:31.002 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0569e-05 2024-04-20T17:13:31.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.002 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:31.002 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.753709+0000", 2024-04-20T17:13:31.002 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7679999999999997e-06 2024-04-20T17:13:31.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.003 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:31.003 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.763427+0000", 2024-04-20T17:13:31.003 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0097184950000000006 2024-04-20T17:13:31.003 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.003 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:31.003 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.763451+0000", 2024-04-20T17:13:31.003 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3818000000000001e-05 2024-04-20T17:13:31.003 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.003 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:31.003 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.777613+0000", 2024-04-20T17:13:31.003 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.014161711 2024-04-20T17:13:31.004 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.004 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.004 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:31.004 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.777657+0000", 2024-04-20T17:13:31.004 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4008999999999998e-05 2024-04-20T17:13:31.004 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:31.004 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:31.004 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:31.004 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.004 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.004 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:16911 2.1 e208/196 2:88260f83:::benchmark_data_smithi171_78918_object56795:head v 208'16983, mlcod=208'16964)", 2024-04-20T17:13:31.004 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:38.755617+0000", 2024-04-20T17:13:31.004 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 52.187228484999999, 2024-04-20T17:13:31.004 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.022073652999999999, 2024-04-20T17:13:31.005 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:31.005 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:31.005 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:31.005 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:31.005 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.005 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:31.005 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.755617+0000", 2024-04-20T17:13:31.005 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:31.005 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.005 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.005 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:31.005 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.755617+0000", 2024-04-20T17:13:31.005 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:31.005 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.005 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.006 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:31.006 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.755618+0000", 2024-04-20T17:13:31.006 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.567e-06 2024-04-20T17:13:31.006 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.006 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.006 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:31.006 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.755627+0000", 2024-04-20T17:13:31.006 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.473e-06 2024-04-20T17:13:31.006 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.006 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.006 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:31.006 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.755641+0000", 2024-04-20T17:13:31.006 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4127999999999999e-05 2024-04-20T17:13:31.006 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.007 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.007 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:31.007 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.755649+0000", 2024-04-20T17:13:31.007 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7160000000000003e-06 2024-04-20T17:13:31.007 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.007 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.007 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:31.007 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.766042+0000", 2024-04-20T17:13:31.007 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010393528000000001 2024-04-20T17:13:31.007 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.007 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.007 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:31.007 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.766055+0000", 2024-04-20T17:13:31.007 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2850999999999999e-05 2024-04-20T17:13:31.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.008 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.008 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:31.008 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.777660+0000", 2024-04-20T17:13:31.008 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011605167 2024-04-20T17:13:31.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.008 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.008 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:31.008 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.777690+0000", 2024-04-20T17:13:31.008 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0222999999999998e-05 2024-04-20T17:13:31.008 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:31.008 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:31.008 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:31.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.008 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.009 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:16913 2.1 e208/196 2:8827185f:::benchmark_data_smithi171_78918_object97707:head v 208'16985, mlcod=208'16964)", 2024-04-20T17:13:31.009 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:38.759277+0000", 2024-04-20T17:13:31.009 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 52.183568373999996, 2024-04-20T17:13:31.009 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.023300471999999999, 2024-04-20T17:13:31.009 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:31.009 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:31.009 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:31.009 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:31.009 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.009 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:31.009 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.759277+0000", 2024-04-20T17:13:31.009 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:31.009 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.009 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.010 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:31.010 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.759277+0000", 2024-04-20T17:13:31.010 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:31.010 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.010 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.010 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:31.010 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.759278+0000", 2024-04-20T17:13:31.010 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.037e-06 2024-04-20T17:13:31.010 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.010 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.010 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:31.010 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.759283+0000", 2024-04-20T17:13:31.010 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3020000000000002e-06 2024-04-20T17:13:31.010 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.010 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.011 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:31.011 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.759291+0000", 2024-04-20T17:13:31.011 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0560000000000005e-06 2024-04-20T17:13:31.011 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.011 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.011 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:31.011 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.759314+0000", 2024-04-20T17:13:31.011 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2818000000000001e-05 2024-04-20T17:13:31.011 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.011 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.011 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:31.011 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.770149+0000", 2024-04-20T17:13:31.011 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010835299 2024-04-20T17:13:31.011 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.011 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.012 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:31.012 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.770166+0000", 2024-04-20T17:13:31.012 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7200000000000001e-05 2024-04-20T17:13:31.012 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.012 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.012 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:31.012 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.782553+0000", 2024-04-20T17:13:31.012 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012386359 2024-04-20T17:13:31.012 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.012 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.012 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:31.012 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.782577+0000", 2024-04-20T17:13:31.012 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4400999999999999e-05 2024-04-20T17:13:31.012 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:31.013 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:31.013 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:31.013 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.013 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.013 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:16914 2.1 e208/196 2:882743ef:::benchmark_data_smithi171_78918_object108384:head v 208'16986, mlcod=208'16964)", 2024-04-20T17:13:31.013 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:38.760999+0000", 2024-04-20T17:13:31.013 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 52.181846372000003, 2024-04-20T17:13:31.013 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.021603013000000001, 2024-04-20T17:13:31.013 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:31.013 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:31.013 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:31.013 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:31.013 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.013 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:31.013 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.760999+0000", 2024-04-20T17:13:31.014 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:31.014 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.014 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.014 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:31.014 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.760999+0000", 2024-04-20T17:13:31.014 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:31.014 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.014 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.014 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:31.014 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.761000+0000", 2024-04-20T17:13:31.014 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.082e-06 2024-04-20T17:13:31.014 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.014 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.015 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:31.015 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.761007+0000", 2024-04-20T17:13:31.015 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7710000000000001e-06 2024-04-20T17:13:31.015 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.015 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:31.015 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.761016+0000", 2024-04-20T17:13:31.015 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0089999999999996e-06 2024-04-20T17:13:31.015 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.015 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:31.015 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.761022+0000", 2024-04-20T17:13:31.015 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8989999999999998e-06 2024-04-20T17:13:31.015 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.016 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:31.016 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.772375+0000", 2024-04-20T17:13:31.016 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011353781 2024-04-20T17:13:31.016 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.016 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.016 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:31.016 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.772390+0000", 2024-04-20T17:13:31.016 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4823e-05 2024-04-20T17:13:31.016 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.016 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.016 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:31.016 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.782579+0000", 2024-04-20T17:13:31.016 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010189024 2024-04-20T17:13:31.016 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.017 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.017 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:31.017 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.782602+0000", 2024-04-20T17:13:31.017 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2623999999999999e-05 2024-04-20T17:13:31.017 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:31.017 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:31.017 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:31.017 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.017 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.017 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:17110 2.1 e209/196 2:889b19a7:::benchmark_data_smithi171_78918_object80717:head v 209'17168, mlcod=209'17148)", 2024-04-20T17:13:31.017 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:39.201487+0000", 2024-04-20T17:13:31.017 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 51.741358210000001, 2024-04-20T17:13:31.017 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.021433605000000001, 2024-04-20T17:13:31.017 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:31.017 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:31.017 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:31.018 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:31.018 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.018 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:31.018 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.201487+0000", 2024-04-20T17:13:31.018 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:31.018 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.018 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.018 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:31.018 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.201487+0000", 2024-04-20T17:13:31.018 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:31.018 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.018 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.018 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:31.018 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.201488+0000", 2024-04-20T17:13:31.018 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.505e-06 2024-04-20T17:13:31.019 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.019 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.019 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:31.019 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.201498+0000", 2024-04-20T17:13:31.019 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.645e-06 2024-04-20T17:13:31.019 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.019 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.019 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:31.019 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.201507+0000", 2024-04-20T17:13:31.019 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6419999999999996e-06 2024-04-20T17:13:31.019 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.019 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.019 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:31.019 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.201512+0000", 2024-04-20T17:13:31.020 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1569999999999999e-06 2024-04-20T17:13:31.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.020 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:31.020 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.206347+0000", 2024-04-20T17:13:31.020 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0048346259999999999 2024-04-20T17:13:31.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.020 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:31.020 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.206362+0000", 2024-04-20T17:13:31.020 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5221e-05 2024-04-20T17:13:31.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.020 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:31.020 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.222884+0000", 2024-04-20T17:13:31.021 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.016522326 2024-04-20T17:13:31.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.021 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:31.021 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.222921+0000", 2024-04-20T17:13:31.021 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6483000000000003e-05 2024-04-20T17:13:31.021 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:31.021 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:31.021 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:31.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.021 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:17112 2.1 e209/196 2:889c7af1:::benchmark_data_smithi171_78918_object66844:head v 209'17170, mlcod=209'17148)", 2024-04-20T17:13:31.021 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:39.204841+0000", 2024-04-20T17:13:31.021 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 51.738003872999997, 2024-04-20T17:13:31.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.02283437, 2024-04-20T17:13:31.022 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:31.022 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:31.022 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:31.022 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:31.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:31.022 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.204841+0000", 2024-04-20T17:13:31.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:31.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:31.022 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.204841+0000", 2024-04-20T17:13:31.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:31.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.023 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:31.023 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.204843+0000", 2024-04-20T17:13:31.023 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.488e-06 2024-04-20T17:13:31.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.023 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:31.023 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.204853+0000", 2024-04-20T17:13:31.023 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9520000000000006e-06 2024-04-20T17:13:31.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.023 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:31.023 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.204863+0000", 2024-04-20T17:13:31.023 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0762e-05 2024-04-20T17:13:31.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.024 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.024 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:31.024 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.204872+0000", 2024-04-20T17:13:31.024 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4379999999999995e-06 2024-04-20T17:13:31.024 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.024 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.024 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:31.024 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.211280+0000", 2024-04-20T17:13:31.024 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0064079460000000003 2024-04-20T17:13:31.024 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.024 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.024 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:31.024 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.211295+0000", 2024-04-20T17:13:31.024 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4778e-05 2024-04-20T17:13:31.024 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.024 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.025 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:31.025 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.227627+0000", 2024-04-20T17:13:31.025 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.016332276999999999 2024-04-20T17:13:31.025 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.025 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:31.025 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.227676+0000", 2024-04-20T17:13:31.025 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8729000000000001e-05 2024-04-20T17:13:31.025 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:31.025 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:31.025 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:31.025 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.025 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:17115 2.1 e209/196 2:889cf538:::benchmark_data_smithi171_78918_object138127:head v 209'17173, mlcod=209'17148)", 2024-04-20T17:13:31.025 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:39.209715+0000", 2024-04-20T17:13:31.026 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 51.733129910000002, 2024-04-20T17:13:31.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.02173806, 2024-04-20T17:13:31.026 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:31.026 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:31.026 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:31.026 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:31.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.026 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:31.026 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.209715+0000", 2024-04-20T17:13:31.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:31.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.026 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:31.026 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.209715+0000", 2024-04-20T17:13:31.027 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:31.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.027 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:31.027 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.209716+0000", 2024-04-20T17:13:31.027 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1969999999999999e-06 2024-04-20T17:13:31.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.027 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:31.027 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.209723+0000", 2024-04-20T17:13:31.027 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6839999999999999e-06 2024-04-20T17:13:31.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.027 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:31.027 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.209733+0000", 2024-04-20T17:13:31.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5089999999999999e-06 2024-04-20T17:13:31.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.028 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:31.028 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.209739+0000", 2024-04-20T17:13:31.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.066e-06 2024-04-20T17:13:31.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.028 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:31.028 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.220468+0000", 2024-04-20T17:13:31.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010729304 2024-04-20T17:13:31.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.028 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:31.029 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.220487+0000", 2024-04-20T17:13:31.029 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8558000000000001e-05 2024-04-20T17:13:31.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.029 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:31.029 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.231405+0000", 2024-04-20T17:13:31.029 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010918572 2024-04-20T17:13:31.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.029 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:31.029 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.231453+0000", 2024-04-20T17:13:31.029 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8170000000000001e-05 2024-04-20T17:13:31.029 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:31.029 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:31.030 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:31.030 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.030 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.030 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:19009 2.1 e212/196 2:8cb05c7f:::benchmark_data_smithi171_78918_object102033:head v 212'18978, mlcod=212'18965)", 2024-04-20T17:13:31.030 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:44.360334+0000", 2024-04-20T17:13:31.030 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 46.582510755999998, 2024-04-20T17:13:31.030 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.023791541999999999, 2024-04-20T17:13:31.030 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:31.030 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:31.030 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:31.030 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:31.030 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.030 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:31.030 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.360334+0000", 2024-04-20T17:13:31.030 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:31.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.031 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:31.031 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.360334+0000", 2024-04-20T17:13:31.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:31.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.031 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:31.031 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.360335+0000", 2024-04-20T17:13:31.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6100000000000002e-07 2024-04-20T17:13:31.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.031 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:31.031 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.360338+0000", 2024-04-20T17:13:31.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3380000000000001e-06 2024-04-20T17:13:31.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.032 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:31.032 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.360342+0000", 2024-04-20T17:13:31.032 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0910000000000003e-06 2024-04-20T17:13:31.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.032 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:31.032 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.360345+0000", 2024-04-20T17:13:31.032 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6850000000000001e-06 2024-04-20T17:13:31.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.032 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:31.032 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.362909+0000", 2024-04-20T17:13:31.033 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0025641840000000002 2024-04-20T17:13:31.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.033 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:31.033 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.362940+0000", 2024-04-20T17:13:31.033 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1402000000000002e-05 2024-04-20T17:13:31.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.033 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:31.033 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.384086+0000", 2024-04-20T17:13:31.033 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.021145265999999999 2024-04-20T17:13:31.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.033 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:31.033 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.384126+0000", 2024-04-20T17:13:31.033 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0215e-05 2024-04-20T17:13:31.034 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:31.034 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:31.034 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:31.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.034 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:19010 2.1 e212/196 2:8cb0f6dc:::benchmark_data_smithi171_78918_object135138:head v 212'18979, mlcod=212'18965)", 2024-04-20T17:13:31.034 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:44.362525+0000", 2024-04-20T17:13:31.034 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 46.580320538000002, 2024-04-20T17:13:31.034 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.021626044000000001, 2024-04-20T17:13:31.034 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:13:31.034 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:13:31.034 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:31.034 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:13:31.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.035 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:13:31.035 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.362525+0000", 2024-04-20T17:13:31.035 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:31.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.035 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.035 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:13:31.035 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.362525+0000", 2024-04-20T17:13:31.035 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:13:31.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.035 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.035 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:13:31.035 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.362526+0000", 2024-04-20T17:13:31.035 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5689999999999999e-06 2024-04-20T17:13:31.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.035 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.036 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:13:31.036 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.362536+0000", 2024-04-20T17:13:31.036 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0059e-05 2024-04-20T17:13:31.036 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.036 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.036 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:13:31.036 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.362546+0000", 2024-04-20T17:13:31.036 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5710000000000004e-06 2024-04-20T17:13:31.036 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.036 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.036 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:13:31.036 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.362553+0000", 2024-04-20T17:13:31.036 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8990000000000004e-06 2024-04-20T17:13:31.036 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.036 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.036 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:13:31.037 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.365804+0000", 2024-04-20T17:13:31.037 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00325115 2024-04-20T17:13:31.037 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.037 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:13:31.037 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.365830+0000", 2024-04-20T17:13:31.037 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6131000000000001e-05 2024-04-20T17:13:31.037 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.037 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:13:31.037 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.384130+0000", 2024-04-20T17:13:31.037 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.018299539 2024-04-20T17:13:31.037 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:13:31.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:13:31.037 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:13:31.037 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.384151+0000", 2024-04-20T17:13:31.038 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1126000000000001e-05 2024-04-20T17:13:31.038 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:31.038 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:31.038 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:31.038 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:13:31.038 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:13:31.038 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:13:31.276 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:13:31.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:31.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:13:31.281 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:13:31.380 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:13:31.410 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:31.453 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:13:31.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:31.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:13:31.557 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:13:31.583 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:31.601 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:13:31.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:31.604 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:13:31.702 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-20T17:13:31.704 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:13:31.731 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:31.750 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:13:31.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:31.753 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:13:31.854 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:13:31.880 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:31.900 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.6 to 1 (10000) 2024-04-20T17:13:31.900 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:13:31.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:31.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:13:31.967 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:13:32.005 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:13:32.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:31 smithi106 ceph-mon[44423]: pgmap v640: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 148 op/s; 10567/187858 objects misplaced (5.625%); 171 KiB/s, 2 objects/s recovering 2024-04-20T17:13:32.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:31 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2889532385' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-20T17:13:32.037 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:32.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:31 smithi005 ceph-mon[39562]: pgmap v640: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 148 op/s; 10567/187858 objects misplaced (5.625%); 171 KiB/s, 2 objects/s recovering 2024-04-20T17:13:32.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:31 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2889532385' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-20T17:13:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:31 smithi171 ceph-mon[44189]: pgmap v640: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 148 op/s; 10567/187858 objects misplaced (5.625%); 171 KiB/s, 2 objects/s recovering 2024-04-20T17:13:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:31 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2889532385' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-20T17:13:32.688 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:13:32.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:32.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:13:32.793 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:13:32.859 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:32 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2889532385' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-20T17:13:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:32 smithi005 ceph-mon[39562]: osdmap e229: 12 total, 12 up, 12 in 2024-04-20T17:13:33.243 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 7 1 2024-04-20T17:13:33.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:32 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2889532385' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-20T17:13:33.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:32 smithi106 ceph-mon[44423]: osdmap e229: 12 total, 12 up, 12 in 2024-04-20T17:13:33.306 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:13:33.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:33.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:13:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:32 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2889532385' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-20T17:13:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:32 smithi171 ceph-mon[44189]: osdmap e229: 12 total, 12 up, 12 in 2024-04-20T17:13:33.412 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:13:33.438 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:33.587 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:13:33.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:33.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:13:33.692 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:13:33.721 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:33.897 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:13:33.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:33.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:13:33.914 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:13:33.914 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:13:33.914 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-04-20T17:13:33.914 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:13:33.928 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:13:33.934 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:13:34.001 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:13:34.034 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:34.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:33 smithi005 ceph-mon[39562]: pgmap v642: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 158 op/s; 10567/187452 objects misplaced (5.637%); 148 KiB/s, 2 objects/s recovering 2024-04-20T17:13:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:33 smithi106 ceph-mon[44423]: pgmap v642: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 158 op/s; 10567/187452 objects misplaced (5.637%); 148 KiB/s, 2 objects/s recovering 2024-04-20T17:13:34.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:33 smithi171 ceph-mon[44189]: pgmap v642: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 158 op/s; 10567/187452 objects misplaced (5.637%); 148 KiB/s, 2 objects/s recovering 2024-04-20T17:13:34.635 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-20T17:13:34.916 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:13:34.941 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:13:35.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:34 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/884545164' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:13:35.214 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:34 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/884545164' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:13:35.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:34 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/884545164' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:13:36.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:35 smithi005 ceph-mon[39562]: pgmap v643: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 150 op/s; 10493/186592 objects misplaced (5.623%); 425 KiB/s, 6 objects/s recovering 2024-04-20T17:13:36.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:35 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:13:36.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:35 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/884545164' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:13:36.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:35 smithi005 ceph-mon[39562]: osdmap e230: 12 total, 12 up, 12 in 2024-04-20T17:13:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:35 smithi106 ceph-mon[44423]: pgmap v643: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 150 op/s; 10493/186592 objects misplaced (5.623%); 425 KiB/s, 6 objects/s recovering 2024-04-20T17:13:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:35 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:13:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:35 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/884545164' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:13:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:35 smithi106 ceph-mon[44423]: osdmap e230: 12 total, 12 up, 12 in 2024-04-20T17:13:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:35 smithi171 ceph-mon[44189]: pgmap v643: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 150 op/s; 10493/186592 objects misplaced (5.623%); 425 KiB/s, 6 objects/s recovering 2024-04-20T17:13:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:35 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:13:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:35 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/884545164' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:13:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:35 smithi171 ceph-mon[44189]: osdmap e230: 12 total, 12 up, 12 in 2024-04-20T17:13:36.386 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:13:36.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:36.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:13:36.492 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:13:36.521 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:36.644 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:13:36.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:36.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:13:36.749 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:13:36.776 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:36.804 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:13:36.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:36.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:13:36.908 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:13:36.936 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:36.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:36 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:13:36.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:36 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:13:36.969 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:13:36.969 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:13:36.969 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-04-20T17:13:36.969 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-04-20T17:13:36.970 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:13:36.975 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:13:36.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:36.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:13:37.079 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:13:37.110 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:37.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:36 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:13:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:36 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:13:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:36 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:13:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:36 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:13:37.343 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:13:37.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:37.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:13:37.365 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:13:37.385 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:13:37.408 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:13:37.435 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:13:37.448 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:13:37.464 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:13:37.475 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:37.485 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:13:37.494 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:13:37.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:37.497 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:13:37.505 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:13:37.525 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:13:37.547 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:13:37.570 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:13:37.591 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:13:37.597 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:13:37.616 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:13:37.618 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:13:37.618 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:13:37.618 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:13:37.618 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:13:37.618 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:13:37.618 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:13:37.619 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:13:37.619 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:13:37.619 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:13:37.619 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:13:37.619 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:13:37.619 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:13:37.633 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:37.663 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-20T17:13:37.941 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:13:37.984 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:13:37.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:37.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:13:38.088 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:13:38.122 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:37 smithi005 ceph-mon[39562]: pgmap v645: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 122 op/s; 10493/186228 objects misplaced (5.634%); 432 KiB/s, 6 objects/s recovering 2024-04-20T17:13:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:37 smithi005 ceph-mon[39562]: osdmap e231: 12 total, 12 up, 12 in 2024-04-20T17:13:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:37 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2927461936' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:13:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:38.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:38.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:38.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:38.221 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:13:38.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:38.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:13:38.230 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:37 smithi106 ceph-mon[44423]: pgmap v645: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 122 op/s; 10493/186228 objects misplaced (5.634%); 432 KiB/s, 6 objects/s recovering 2024-04-20T17:13:38.230 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:37 smithi106 ceph-mon[44423]: osdmap e231: 12 total, 12 up, 12 in 2024-04-20T17:13:38.231 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:37 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2927461936' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:13:38.231 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:38.231 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:38.231 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:38.231 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:38.232 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:38.232 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:38.232 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:38.326 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:13:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:37 smithi171 ceph-mon[44189]: pgmap v645: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 122 op/s; 10493/186228 objects misplaced (5.634%); 432 KiB/s, 6 objects/s recovering 2024-04-20T17:13:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:37 smithi171 ceph-mon[44189]: osdmap e231: 12 total, 12 up, 12 in 2024-04-20T17:13:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:37 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2927461936' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:13:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:38.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:38.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:38.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:38.355 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:38.377 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:13:38.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:38.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:13:38.485 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:13:38.513 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:38.763 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:13:38.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:38.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:13:38.868 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:13:38.894 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:39.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:39.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:39.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:39.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:39.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:39.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:38 smithi005 ceph-mon[39562]: pgmap v647: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 152 op/s; 10493/185660 objects misplaced (5.652%); 397 KiB/s, 6 objects/s recovering 2024-04-20T17:13:39.215 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:39.215 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:39.215 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:39.215 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:39.215 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:39.216 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:38 smithi106 ceph-mon[44423]: pgmap v647: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 152 op/s; 10493/185660 objects misplaced (5.652%); 397 KiB/s, 6 objects/s recovering 2024-04-20T17:13:39.243 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:13:39.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:39.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:13:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:38 smithi171 ceph-mon[44189]: pgmap v647: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 152 op/s; 10493/185660 objects misplaced (5.652%); 397 KiB/s, 6 objects/s recovering 2024-04-20T17:13:39.347 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:13:39.373 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:39.570 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:13:39.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:39.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:13:39.673 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:13:39.713 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:39.846 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:13:39.846 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:13:39.963 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:13:39.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:39.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:13:40.068 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:13:40.104 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:40.123 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:13:40.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:40.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:13:40.228 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:13:40.257 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:40.277 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:13:40.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:40.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:13:40.381 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:13:40.410 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:40.429 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:13:40.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:40.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:13:40.534 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:13:40.562 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:40.600 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:13:40.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:40.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:13:40.626 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-20T17:13:40.705 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:13:40.733 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:40.742 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:13:40.754 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:13:40.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:40.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:13:40.858 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:13:40.885 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:40.895 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:13:40.906 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:13:40.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:40.909 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:13:41.010 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:13:41.031 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:13:41.037 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:41.056 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:13:41.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:41.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:13:41.087 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:41 smithi005 ceph-mon[39562]: pgmap v648: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 130 op/s; 10420/185358 objects misplaced (5.622%); 584 KiB/s, 9 objects/s recovering 2024-04-20T17:13:41.087 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:41 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:13:41.087 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:41 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:13:41.087 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:41 smithi005 ceph-mon[39562]: osdmap e232: 12 total, 12 up, 12 in 2024-04-20T17:13:41.087 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:41 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2667368261' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:13:41.087 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:41 smithi005 ceph-mon[39562]: from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:13:41.089 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.7 to 1 (10000) 2024-04-20T17:13:41.160 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:13:41.187 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:41.206 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:13:41.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:41.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:13:41.309 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:13:41.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:41 smithi171 ceph-mon[44189]: pgmap v648: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 130 op/s; 10420/185358 objects misplaced (5.622%); 584 KiB/s, 9 objects/s recovering 2024-04-20T17:13:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:41 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:13:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:41 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:13:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:41 smithi171 ceph-mon[44189]: osdmap e232: 12 total, 12 up, 12 in 2024-04-20T17:13:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:41 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2667368261' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:13:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:41 smithi171 ceph-mon[44189]: from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:13:41.335 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:41.431 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:13:41.487 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:13:41.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:41.490 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:13:41.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:41 smithi106 ceph-mon[44423]: pgmap v648: 17 pgs: 1 active+recovering, 16 active+clean; 5.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 130 op/s; 10420/185358 objects misplaced (5.622%); 584 KiB/s, 9 objects/s recovering 2024-04-20T17:13:41.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:41 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:13:41.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:41 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:13:41.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:41 smithi106 ceph-mon[44423]: osdmap e232: 12 total, 12 up, 12 in 2024-04-20T17:13:41.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:41 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2667368261' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:13:41.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:41 smithi106 ceph-mon[44423]: from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:13:41.591 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:13:41.621 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:41.718 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:13:41.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:41.721 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:13:41.822 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:13:41.851 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:41.949 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:13:41.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:41.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:13:42.053 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:13:42.082 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:42.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:42 smithi005 ceph-mon[39562]: from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-20T17:13:42.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:42 smithi005 ceph-mon[39562]: osdmap e233: 12 total, 12 up, 12 in 2024-04-20T17:13:42.432 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:13:42.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:42.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:13:42.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:42 smithi106 ceph-mon[44423]: from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-20T17:13:42.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:42 smithi106 ceph-mon[44423]: osdmap e233: 12 total, 12 up, 12 in 2024-04-20T17:13:42.543 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:13:42.574 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:42.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:42 smithi171 ceph-mon[44189]: from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-20T17:13:42.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:42 smithi171 ceph-mon[44189]: osdmap e233: 12 total, 12 up, 12 in 2024-04-20T17:13:42.912 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 8 1 2024-04-20T17:13:42.924 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:13:42.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:42.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:13:43.028 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:13:43.058 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:43.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:43 smithi005 ceph-mon[39562]: pgmap v651: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 114 op/s; 10420/184858 objects misplaced (5.637%); 373 KiB/s, 5 objects/s recovering 2024-04-20T17:13:43.226 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:13:43.226 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:13:43.232 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:43 smithi171 ceph-mon[44189]: pgmap v651: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 114 op/s; 10420/184858 objects misplaced (5.637%); 373 KiB/s, 5 objects/s recovering 2024-04-20T17:13:43.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:43 smithi106 ceph-mon[44423]: pgmap v651: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 114 op/s; 10420/184858 objects misplaced (5.637%); 373 KiB/s, 5 objects/s recovering 2024-04-20T17:13:43.592 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:13:43.611 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:13:43.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:43.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:13:43.717 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:13:43.746 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:43.903 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:13:43.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:43.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:13:43.985 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-20T17:13:44.009 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:13:44.056 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:44.086 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:13:44.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:44.089 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:13:44.191 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:13:44.219 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:44.239 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:13:44.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:44.242 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:13:44.275 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:13:44.343 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:13:44.371 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:44.390 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:13:44.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:44.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:13:44.494 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:13:44.526 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:44.615 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:13:44.625 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:13:44.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:44.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:13:44.729 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:13:44.760 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:44.910 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:13:44.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:44.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:13:45.015 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:13:45.043 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:45.166 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:13:45.167 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:45 smithi171 ceph-mon[44189]: pgmap v652: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 154 op/s; 10347/184038 objects misplaced (5.622%); 649 KiB/s, 10 objects/s recovering 2024-04-20T17:13:45.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:45.168 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:13:45.186 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:45 smithi005 ceph-mon[39562]: pgmap v652: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 154 op/s; 10347/184038 objects misplaced (5.622%); 649 KiB/s, 10 objects/s recovering 2024-04-20T17:13:45.269 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:13:45.302 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:45.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:45 smithi106 ceph-mon[44423]: pgmap v652: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 154 op/s; 10347/184038 objects misplaced (5.622%); 649 KiB/s, 10 objects/s recovering 2024-04-20T17:13:45.903 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:13:46.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:46 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2910324355' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:13:46.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:46 smithi171 ceph-mon[44189]: from='client.25111 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:13:46.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:46 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:13:46.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:46 smithi171 ceph-mon[44189]: from='client.25111 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:13:46.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:46 smithi171 ceph-mon[44189]: osdmap e234: 12 total, 12 up, 12 in 2024-04-20T17:13:46.350 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:13:46.350 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:13:46.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:46 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2910324355' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:13:46.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:46 smithi005 ceph-mon[39562]: from='client.25111 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:13:46.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:46 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:13:46.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:46 smithi005 ceph-mon[39562]: from='client.25111 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:13:46.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:46 smithi005 ceph-mon[39562]: osdmap e234: 12 total, 12 up, 12 in 2024-04-20T17:13:46.456 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:13:46.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:46.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:13:46.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:46 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2910324355' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:13:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:46 smithi106 ceph-mon[44423]: from='client.25111 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:13:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:46 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:13:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:46 smithi106 ceph-mon[44423]: from='client.25111 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:13:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:46 smithi106 ceph-mon[44423]: osdmap e234: 12 total, 12 up, 12 in 2024-04-20T17:13:46.559 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:13:46.593 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:47.084 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-20T17:13:47.146 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:13:47.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:47.149 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:13:47.250 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:13:47.283 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:47.393 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:13:47.534 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:13:47.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:47.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:13:47.638 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:13:47.665 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:47.683 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:13:47.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:47.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:13:47.787 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:13:47.813 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:47.831 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:13:47.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:47.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:13:47.935 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:13:47.936 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:47 smithi171 ceph-mon[44189]: pgmap v654: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 139 op/s; 10347/183680 objects misplaced (5.633%); 395 KiB/s, 6 objects/s recovering 2024-04-20T17:13:47.936 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:47 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:13:47.936 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:47 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:13:47.936 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:47 smithi171 ceph-mon[44189]: osdmap e235: 12 total, 12 up, 12 in 2024-04-20T17:13:47.960 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:47 smithi106 ceph-mon[44423]: pgmap v654: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 139 op/s; 10347/183680 objects misplaced (5.633%); 395 KiB/s, 6 objects/s recovering 2024-04-20T17:13:47.960 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:47 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:13:47.960 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:47 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:13:47.960 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:47 smithi106 ceph-mon[44423]: osdmap e235: 12 total, 12 up, 12 in 2024-04-20T17:13:47.962 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:47.980 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:13:47.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:47.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:13:48.084 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:13:48.093 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:13:48.112 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:48.117 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:13:48.139 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:13:48.159 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:13:48.183 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:13:48.203 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:13:48.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:47 smithi005 ceph-mon[39562]: pgmap v654: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 139 op/s; 10347/183680 objects misplaced (5.633%); 395 KiB/s, 6 objects/s recovering 2024-04-20T17:13:48.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:47 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:13:48.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:47 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:13:48.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:47 smithi005 ceph-mon[39562]: osdmap e235: 12 total, 12 up, 12 in 2024-04-20T17:13:48.228 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:13:48.257 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:13:48.261 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:13:48.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:48.264 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:13:48.282 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:13:48.303 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:13:48.324 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:13:48.345 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:13:48.347 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:13:48.347 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:13:48.347 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:13:48.347 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:13:48.347 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:13:48.347 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:13:48.347 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:13:48.347 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:13:48.347 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:13:48.347 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:13:48.347 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:13:48.347 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:13:48.365 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:13:48.390 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:48.408 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:13:48.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:48.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:13:48.512 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:13:48.544 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:48.618 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:13:48.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:48.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:13:48.722 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:13:48.754 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:48.906 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.8 to 1 (10000) 2024-04-20T17:13:49.137 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:48 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2168548317' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:13:49.138 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.138 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.138 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.138 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.138 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.138 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.138 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.138 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.138 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.138 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.138 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:48 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2242134099' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-20T17:13:49.201 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:48 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2168548317' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:13:49.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:48 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2242134099' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-20T17:13:49.204 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:13:49.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:49.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:13:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:48 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2168548317' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:13:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:49.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:48 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2242134099' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-20T17:13:49.308 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:13:49.339 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:49.384 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:13:49.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:49.387 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:13:49.426 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:13:49.426 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:13:49.426 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:13:49.426 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:13:49.426 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.426 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:138161 2.5 2:a4f53bd8:::benchmark_data_smithi171_78918_object138160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-20T17:13:49.426 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:09:22.723185+0000", 2024-04-20T17:13:49.426 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 266.701003179, 2024-04-20T17:13:49.426 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.037369641000000002, 2024-04-20T17:13:49.426 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:13:49.427 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:13:49.427 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:49.427 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:13:49.427 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:13:49.427 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:13:49.427 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 138161 2024-04-20T17:13:49.427 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.427 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:13:49.427 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.427 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:13:49.427 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.723185+0000", 2024-04-20T17:13:49.427 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.427 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.427 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.427 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:13:49.428 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.723185+0000", 2024-04-20T17:13:49.428 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.428 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.428 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.428 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:13:49.428 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.723186+0000", 2024-04-20T17:13:49.428 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4000000000000001e-07 2024-04-20T17:13:49.428 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.428 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.428 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:13:49.428 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.723231+0000", 2024-04-20T17:13:49.428 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5262999999999997e-05 2024-04-20T17:13:49.428 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.428 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.429 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:13:49.429 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.723240+0000", 2024-04-20T17:13:49.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9649999999999997e-06 2024-04-20T17:13:49.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.429 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.429 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:13:49.429 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.723246+0000", 2024-04-20T17:13:49.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.207e-06 2024-04-20T17:13:49.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.429 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.429 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:13:49.429 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.727449+0000", 2024-04-20T17:13:49.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0042026570000000003 2024-04-20T17:13:49.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.430 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.430 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:13:49.430 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.727582+0000", 2024-04-20T17:13:49.430 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013308600000000001 2024-04-20T17:13:49.430 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.430 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.430 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:13:49.430 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.727720+0000", 2024-04-20T17:13:49.430 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000138348 2024-04-20T17:13:49.430 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.430 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.430 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:13:49.430 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.731368+0000", 2024-04-20T17:13:49.430 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0036483380000000001 2024-04-20T17:13:49.431 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.431 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.431 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:13:49.431 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.760485+0000", 2024-04-20T17:13:49.431 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.029116454 2024-04-20T17:13:49.431 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.431 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.431 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:13:49.431 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.760520+0000", 2024-04-20T17:13:49.431 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5349999999999999e-05 2024-04-20T17:13:49.431 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.431 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.431 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:13:49.431 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.760554+0000", 2024-04-20T17:13:49.432 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4233000000000002e-05 2024-04-20T17:13:49.432 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.432 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:13:49.432 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.432 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.432 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.432 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:195841 2.5 2:a970f203:::benchmark_data_smithi171_78918_object34847:head [delete] snapc 0=[] ondisk+write+known_if_redirected e163)", 2024-04-20T17:13:49.432 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:30.697234+0000", 2024-04-20T17:13:49.432 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 198.726953713, 2024-04-20T17:13:49.432 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.037855572999999997, 2024-04-20T17:13:49.432 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:13:49.432 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:13:49.432 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:49.432 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:13:49.432 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:13:49.433 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:13:49.433 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 195841 2024-04-20T17:13:49.433 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.433 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:13:49.433 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.433 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:13:49.433 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.697234+0000", 2024-04-20T17:13:49.433 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.433 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.433 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.433 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:13:49.433 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.697234+0000", 2024-04-20T17:13:49.433 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.434 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.434 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.434 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:13:49.434 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.697235+0000", 2024-04-20T17:13:49.434 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1939999999999999e-06 2024-04-20T17:13:49.434 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.434 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.434 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:13:49.434 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.697239+0000", 2024-04-20T17:13:49.434 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0589999999999996e-06 2024-04-20T17:13:49.434 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.434 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.434 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:13:49.434 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.697247+0000", 2024-04-20T17:13:49.435 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5909999999999998e-06 2024-04-20T17:13:49.435 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.435 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.435 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:13:49.435 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.697254+0000", 2024-04-20T17:13:49.435 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5760000000000002e-06 2024-04-20T17:13:49.435 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.435 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.435 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:13:49.435 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.701087+0000", 2024-04-20T17:13:49.435 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.003833192 2024-04-20T17:13:49.435 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.435 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.436 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:13:49.436 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.701179+0000", 2024-04-20T17:13:49.436 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2037e-05 2024-04-20T17:13:49.436 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.436 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.436 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:13:49.436 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.701266+0000", 2024-04-20T17:13:49.436 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6891999999999994e-05 2024-04-20T17:13:49.436 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.436 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.436 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:13:49.436 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.704805+0000", 2024-04-20T17:13:49.436 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0035387330000000001 2024-04-20T17:13:49.436 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.436 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.437 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:13:49.437 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.735014+0000", 2024-04-20T17:13:49.437 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030209192999999999 2024-04-20T17:13:49.437 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.437 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.437 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:13:49.437 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.735056+0000", 2024-04-20T17:13:49.437 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2469000000000003e-05 2024-04-20T17:13:49.437 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.437 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.437 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:13:49.437 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.735090+0000", 2024-04-20T17:13:49.437 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3637000000000003e-05 2024-04-20T17:13:49.438 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.438 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:13:49.438 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.438 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.438 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.438 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:202267 2.5 2:abe7d7a2:::benchmark_data_smithi171_78918_object41273:head [delete] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-04-20T17:13:49.438 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:39.807427+0000", 2024-04-20T17:13:49.438 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 189.61676112800001, 2024-04-20T17:13:49.438 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.038790863000000002, 2024-04-20T17:13:49.438 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:13:49.438 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:13:49.438 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:49.438 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:13:49.439 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:13:49.439 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:13:49.439 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 202267 2024-04-20T17:13:49.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.439 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:13:49.439 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.439 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:13:49.439 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.807427+0000", 2024-04-20T17:13:49.439 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.439 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.439 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:13:49.439 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.807427+0000", 2024-04-20T17:13:49.439 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.440 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.440 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:13:49.440 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.807429+0000", 2024-04-20T17:13:49.440 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0959999999999999e-06 2024-04-20T17:13:49.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.440 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.440 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:13:49.440 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.807435+0000", 2024-04-20T17:13:49.440 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9499999999999998e-06 2024-04-20T17:13:49.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.440 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.440 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:13:49.440 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.807447+0000", 2024-04-20T17:13:49.441 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1654000000000001e-05 2024-04-20T17:13:49.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.441 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:13:49.441 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.807454+0000", 2024-04-20T17:13:49.441 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2790000000000001e-06 2024-04-20T17:13:49.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.441 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:13:49.441 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.807508+0000", 2024-04-20T17:13:49.441 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4370999999999998e-05 2024-04-20T17:13:49.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.441 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:13:49.441 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.807648+0000", 2024-04-20T17:13:49.442 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000139734 2024-04-20T17:13:49.442 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.442 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.442 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:13:49.442 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.807766+0000", 2024-04-20T17:13:49.442 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000118279 2024-04-20T17:13:49.442 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.442 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.442 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:13:49.442 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.811542+0000", 2024-04-20T17:13:49.442 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0037755639999999999 2024-04-20T17:13:49.442 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.442 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.442 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:13:49.442 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.846127+0000", 2024-04-20T17:13:49.443 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.034585171999999997 2024-04-20T17:13:49.443 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.443 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.443 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:13:49.443 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.846164+0000", 2024-04-20T17:13:49.443 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6671999999999997e-05 2024-04-20T17:13:49.443 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.443 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.443 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:13:49.443 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.846218+0000", 2024-04-20T17:13:49.443 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4092000000000003e-05 2024-04-20T17:13:49.443 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.443 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:13:49.444 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.444 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.444 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.444 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:205502 2.b 2:da81e759:::benchmark_data_smithi171_78918_object44508:head [delete] snapc 0=[] ondisk+write+known_if_redirected e167)", 2024-04-20T17:13:49.444 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:43.228493+0000", 2024-04-20T17:13:49.444 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 186.19569468500001, 2024-04-20T17:13:49.444 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036099956000000002, 2024-04-20T17:13:49.444 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:13:49.444 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:13:49.444 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:49.444 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:13:49.444 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:13:49.444 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:13:49.444 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 205502 2024-04-20T17:13:49.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.445 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:13:49.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.445 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:13:49.445 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.228493+0000", 2024-04-20T17:13:49.445 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.445 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:13:49.445 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.228493+0000", 2024-04-20T17:13:49.445 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.446 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:13:49.446 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.228495+0000", 2024-04-20T17:13:49.446 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5370000000000001e-06 2024-04-20T17:13:49.446 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.446 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.446 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:13:49.446 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.228501+0000", 2024-04-20T17:13:49.446 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8939999999999998e-06 2024-04-20T17:13:49.446 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.446 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.446 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:13:49.446 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.228510+0000", 2024-04-20T17:13:49.446 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6509999999999996e-06 2024-04-20T17:13:49.446 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.447 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.447 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:13:49.447 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.228519+0000", 2024-04-20T17:13:49.447 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6600000000000001e-06 2024-04-20T17:13:49.447 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.447 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.447 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:13:49.447 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.245447+0000", 2024-04-20T17:13:49.447 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.016928144999999999 2024-04-20T17:13:49.447 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.447 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.447 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:13:49.448 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.245548+0000", 2024-04-20T17:13:49.448 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010089 2024-04-20T17:13:49.448 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.448 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.448 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:13:49.448 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.245644+0000", 2024-04-20T17:13:49.448 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6254000000000004e-05 2024-04-20T17:13:49.448 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.448 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.448 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:13:49.448 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.260040+0000", 2024-04-20T17:13:49.448 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014396153 2024-04-20T17:13:49.448 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.448 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.449 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:13:49.449 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.264494+0000", 2024-04-20T17:13:49.449 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0044534120000000003 2024-04-20T17:13:49.449 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.449 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.449 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:13:49.449 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.264544+0000", 2024-04-20T17:13:49.449 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0451000000000002e-05 2024-04-20T17:13:49.449 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.449 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.449 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:13:49.449 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.264593+0000", 2024-04-20T17:13:49.449 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8909000000000002e-05 2024-04-20T17:13:49.450 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.450 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:13:49.450 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.450 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.450 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.450 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:31842 2.6 e210/115 2:609476f8:::benchmark_data_smithi171_78918_object64933:head v 210'13806, mlcod=210'13795)", 2024-04-20T17:13:49.450 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.322569+0000", 2024-04-20T17:13:49.450 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 68.101618762000001, 2024-04-20T17:13:49.450 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.038410374999999997, 2024-04-20T17:13:49.450 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:13:49.450 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:13:49.450 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:49.450 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:13:49.450 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.451 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:13:49.451 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.322569+0000", 2024-04-20T17:13:49.451 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.451 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.451 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.451 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:13:49.451 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.322569+0000", 2024-04-20T17:13:49.451 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.451 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.451 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.451 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:13:49.451 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.322571+0000", 2024-04-20T17:13:49.452 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6929999999999999e-06 2024-04-20T17:13:49.452 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.452 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.452 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:13:49.452 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.322578+0000", 2024-04-20T17:13:49.452 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4370000000000001e-06 2024-04-20T17:13:49.452 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.452 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.452 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:13:49.452 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.322589+0000", 2024-04-20T17:13:49.452 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0548e-05 2024-04-20T17:13:49.452 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.453 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.453 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:13:49.453 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.322596+0000", 2024-04-20T17:13:49.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0090000000000001e-06 2024-04-20T17:13:49.453 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.453 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.453 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:13:49.453 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.334018+0000", 2024-04-20T17:13:49.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011421751000000001 2024-04-20T17:13:49.453 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.453 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.453 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:13:49.453 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.334039+0000", 2024-04-20T17:13:49.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1721000000000001e-05 2024-04-20T17:13:49.454 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.454 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.454 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:13:49.454 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.360942+0000", 2024-04-20T17:13:49.454 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026902463000000001 2024-04-20T17:13:49.454 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.454 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.454 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:13:49.454 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.360980+0000", 2024-04-20T17:13:49.454 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7753000000000003e-05 2024-04-20T17:13:49.454 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.454 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:13:49.454 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.455 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.455 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.455 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:31847 2.6 e210/115 2:6097b0e2:::benchmark_data_smithi171_78918_object71834:head v 210'13811, mlcod=210'13795)", 2024-04-20T17:13:49.455 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.341065+0000", 2024-04-20T17:13:49.455 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 68.083123017999995, 2024-04-20T17:13:49.455 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.038246251000000002, 2024-04-20T17:13:49.455 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:13:49.455 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:13:49.455 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:49.455 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:13:49.455 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.455 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:13:49.455 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.341065+0000", 2024-04-20T17:13:49.455 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.455 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.456 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.456 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:13:49.456 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.341065+0000", 2024-04-20T17:13:49.456 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.456 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.456 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.456 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:13:49.456 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.341066+0000", 2024-04-20T17:13:49.456 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5600000000000001e-06 2024-04-20T17:13:49.456 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.456 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.456 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:13:49.456 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.341075+0000", 2024-04-20T17:13:49.456 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2979999999999992e-06 2024-04-20T17:13:49.456 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.457 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:13:49.457 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.341091+0000", 2024-04-20T17:13:49.457 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5715000000000001e-05 2024-04-20T17:13:49.457 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.457 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:13:49.457 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.341102+0000", 2024-04-20T17:13:49.457 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1224e-05 2024-04-20T17:13:49.457 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.457 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:13:49.457 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.357362+0000", 2024-04-20T17:13:49.457 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.016260406000000002 2024-04-20T17:13:49.458 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.458 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.458 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:13:49.458 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.357386+0000", 2024-04-20T17:13:49.458 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3601999999999999e-05 2024-04-20T17:13:49.458 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.458 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.458 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:13:49.458 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.379270+0000", 2024-04-20T17:13:49.458 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021883969999999999 2024-04-20T17:13:49.458 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.458 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.458 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:13:49.458 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.379311+0000", 2024-04-20T17:13:49.458 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1476000000000001e-05 2024-04-20T17:13:49.459 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.459 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:13:49.459 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.459 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.459 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.459 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:31849 2.6 e210/115 2:609a0ddc:::benchmark_data_smithi171_78918_object159331:head v 210'13813, mlcod=210'13795)", 2024-04-20T17:13:49.459 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.347350+0000", 2024-04-20T17:13:49.459 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 68.076837458, 2024-04-20T17:13:49.459 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.039795965000000003, 2024-04-20T17:13:49.459 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:13:49.459 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:13:49.459 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:49.460 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:13:49.460 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.460 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:13:49.460 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.347350+0000", 2024-04-20T17:13:49.460 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.460 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.460 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.460 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:13:49.460 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.347350+0000", 2024-04-20T17:13:49.460 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.460 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.460 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.460 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:13:49.460 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.347352+0000", 2024-04-20T17:13:49.460 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4300000000000001e-06 2024-04-20T17:13:49.461 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.461 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.461 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:13:49.461 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.347361+0000", 2024-04-20T17:13:49.461 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3649999999999993e-06 2024-04-20T17:13:49.461 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.461 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.461 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:13:49.461 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.347371+0000", 2024-04-20T17:13:49.461 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7920000000000004e-06 2024-04-20T17:13:49.461 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.461 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.461 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:13:49.461 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.347378+0000", 2024-04-20T17:13:49.462 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7499999999999997e-06 2024-04-20T17:13:49.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.462 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:13:49.462 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.365885+0000", 2024-04-20T17:13:49.462 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018507132999999999 2024-04-20T17:13:49.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.462 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:13:49.462 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.365910+0000", 2024-04-20T17:13:49.462 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5338999999999999e-05 2024-04-20T17:13:49.463 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.463 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.463 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:13:49.463 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.387117+0000", 2024-04-20T17:13:49.463 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021207085000000001 2024-04-20T17:13:49.463 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.463 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.463 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:13:49.463 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.387146+0000", 2024-04-20T17:13:49.463 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9071e-05 2024-04-20T17:13:49.463 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.463 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:13:49.463 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.463 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.463 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.464 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:31850 2.6 e210/115 2:609a3392:::benchmark_data_smithi171_78918_object78297:head v 210'13814, mlcod=210'13795)", 2024-04-20T17:13:49.464 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.351057+0000", 2024-04-20T17:13:49.464 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 68.073131177999997, 2024-04-20T17:13:49.464 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036121317, 2024-04-20T17:13:49.464 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:13:49.464 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:13:49.464 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:49.464 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:13:49.464 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.464 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:13:49.464 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.351057+0000", 2024-04-20T17:13:49.464 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.464 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.464 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.464 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:13:49.465 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.351057+0000", 2024-04-20T17:13:49.465 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.465 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.465 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.465 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:13:49.465 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.351058+0000", 2024-04-20T17:13:49.465 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.652e-06 2024-04-20T17:13:49.465 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.465 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.465 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:13:49.465 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.351068+0000", 2024-04-20T17:13:49.465 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9760000000000007e-06 2024-04-20T17:13:49.465 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.465 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.465 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:13:49.466 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.351083+0000", 2024-04-20T17:13:49.466 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4934e-05 2024-04-20T17:13:49.466 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.466 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.466 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:13:49.466 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.351092+0000", 2024-04-20T17:13:49.466 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2069999999999995e-06 2024-04-20T17:13:49.466 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.466 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.466 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:13:49.466 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.370433+0000", 2024-04-20T17:13:49.466 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019341862000000001 2024-04-20T17:13:49.466 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.467 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.467 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:13:49.467 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.370455+0000", 2024-04-20T17:13:49.467 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1469000000000001e-05 2024-04-20T17:13:49.467 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.467 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.467 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:13:49.467 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.387150+0000", 2024-04-20T17:13:49.467 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.016695254999999999 2024-04-20T17:13:49.467 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.467 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.467 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:13:49.467 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.387178+0000", 2024-04-20T17:13:49.467 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7962000000000001e-05 2024-04-20T17:13:49.467 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.467 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:13:49.468 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.468 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.468 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.468 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:31851 2.6 e210/115 2:609a34ba:::benchmark_data_smithi171_78918_object151524:head v 210'13815, mlcod=210'13795)", 2024-04-20T17:13:49.468 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.355231+0000", 2024-04-20T17:13:49.468 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 68.068956475999997, 2024-04-20T17:13:49.468 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.045050494000000003, 2024-04-20T17:13:49.468 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:13:49.468 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:13:49.468 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:49.468 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:13:49.468 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.468 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:13:49.469 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.355231+0000", 2024-04-20T17:13:49.469 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.469 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.469 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.469 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:13:49.469 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.355231+0000", 2024-04-20T17:13:49.469 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.469 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.469 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.469 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:13:49.469 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.355233+0000", 2024-04-20T17:13:49.469 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.902e-06 2024-04-20T17:13:49.469 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.469 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.469 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:13:49.470 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.355241+0000", 2024-04-20T17:13:49.470 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8359999999999992e-06 2024-04-20T17:13:49.470 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.470 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.470 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:13:49.470 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.355254+0000", 2024-04-20T17:13:49.470 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2542e-05 2024-04-20T17:13:49.470 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.470 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.470 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:13:49.470 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.355263+0000", 2024-04-20T17:13:49.470 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.7790000000000001e-06 2024-04-20T17:13:49.470 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.471 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.471 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:13:49.471 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.374844+0000", 2024-04-20T17:13:49.471 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019581365999999999 2024-04-20T17:13:49.471 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.471 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.471 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:13:49.471 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.374869+0000", 2024-04-20T17:13:49.471 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5092e-05 2024-04-20T17:13:49.471 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.471 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.471 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:13:49.471 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.400241+0000", 2024-04-20T17:13:49.472 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.025372098999999999 2024-04-20T17:13:49.472 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.472 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.472 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:13:49.472 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.400282+0000", 2024-04-20T17:13:49.472 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0877999999999997e-05 2024-04-20T17:13:49.472 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.472 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:13:49.472 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.472 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.472 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.472 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:31852 2.6 e210/115 2:609bb098:::benchmark_data_smithi171_78918_object152231:head v 210'13816, mlcod=210'13795)", 2024-04-20T17:13:49.472 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.359184+0000", 2024-04-20T17:13:49.472 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 68.065004286999994, 2024-04-20T17:13:49.473 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.041126272999999998, 2024-04-20T17:13:49.473 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:13:49.473 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:13:49.473 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:49.473 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:13:49.473 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.473 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:13:49.473 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.359184+0000", 2024-04-20T17:13:49.473 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.473 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.473 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.473 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:13:49.473 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.359184+0000", 2024-04-20T17:13:49.473 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.474 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.474 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.474 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:13:49.474 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.359188+0000", 2024-04-20T17:13:49.474 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8890000000000004e-06 2024-04-20T17:13:49.474 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.474 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.474 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:13:49.474 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.359196+0000", 2024-04-20T17:13:49.474 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9529999999999999e-06 2024-04-20T17:13:49.474 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.474 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.474 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:13:49.474 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.359212+0000", 2024-04-20T17:13:49.474 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6718000000000002e-05 2024-04-20T17:13:49.475 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.475 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.475 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:13:49.475 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.359220+0000", 2024-04-20T17:13:49.475 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8420000000000005e-06 2024-04-20T17:13:49.475 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.475 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.475 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:13:49.475 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.379386+0000", 2024-04-20T17:13:49.475 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020166119999999999 2024-04-20T17:13:49.475 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.475 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.475 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:13:49.476 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.379414+0000", 2024-04-20T17:13:49.476 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8072000000000002e-05 2024-04-20T17:13:49.476 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.476 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.476 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:13:49.476 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.400286+0000", 2024-04-20T17:13:49.476 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020871654 2024-04-20T17:13:49.476 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.476 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.476 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:13:49.476 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.400310+0000", 2024-04-20T17:13:49.476 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4025000000000001e-05 2024-04-20T17:13:49.476 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.477 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:13:49.477 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.477 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.477 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.477 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:31853 2.6 e210/115 2:609c5ccc:::benchmark_data_smithi171_78918_object127897:head v 210'13817, mlcod=210'13795)", 2024-04-20T17:13:49.477 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.363464+0000", 2024-04-20T17:13:49.477 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 68.060723873000001, 2024-04-20T17:13:49.477 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.048461072000000001, 2024-04-20T17:13:49.477 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:13:49.477 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:13:49.477 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:49.477 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:13:49.477 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.477 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:13:49.477 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.363464+0000", 2024-04-20T17:13:49.478 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.478 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.478 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.478 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:13:49.478 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.363464+0000", 2024-04-20T17:13:49.478 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.478 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.478 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.478 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:13:49.478 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.363466+0000", 2024-04-20T17:13:49.478 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.818e-06 2024-04-20T17:13:49.478 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.478 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.479 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:13:49.479 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.363474+0000", 2024-04-20T17:13:49.479 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4919999999999993e-06 2024-04-20T17:13:49.479 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.479 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.479 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:13:49.479 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.363485+0000", 2024-04-20T17:13:49.479 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1046999999999999e-05 2024-04-20T17:13:49.479 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.479 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.479 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:13:49.479 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.363494+0000", 2024-04-20T17:13:49.479 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6130000000000004e-06 2024-04-20T17:13:49.480 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.480 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.480 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:13:49.480 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.383962+0000", 2024-04-20T17:13:49.480 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020467928999999999 2024-04-20T17:13:49.480 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.480 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.480 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:13:49.480 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.383979+0000", 2024-04-20T17:13:49.480 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7142e-05 2024-04-20T17:13:49.480 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.480 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.480 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:13:49.481 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.411881+0000", 2024-04-20T17:13:49.481 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.027902278999999999 2024-04-20T17:13:49.481 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.481 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.481 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:13:49.481 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.411925+0000", 2024-04-20T17:13:49.481 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3751999999999997e-05 2024-04-20T17:13:49.481 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.481 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:13:49.481 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.481 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.481 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.481 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:31854 2.6 e210/115 2:609ceadc:::benchmark_data_smithi171_78918_object160601:head v 210'13818, mlcod=210'13795)", 2024-04-20T17:13:49.481 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.367804+0000", 2024-04-20T17:13:49.481 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 68.056383788999995, 2024-04-20T17:13:49.482 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044152018000000001, 2024-04-20T17:13:49.482 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:13:49.482 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:13:49.482 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:49.482 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:13:49.482 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.482 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:13:49.482 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.367804+0000", 2024-04-20T17:13:49.482 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.482 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.482 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.482 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:13:49.482 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.367804+0000", 2024-04-20T17:13:49.483 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.483 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.483 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.483 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:13:49.483 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.367806+0000", 2024-04-20T17:13:49.483 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.516e-06 2024-04-20T17:13:49.483 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.483 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.483 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:13:49.483 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.367814+0000", 2024-04-20T17:13:49.483 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9419999999999995e-06 2024-04-20T17:13:49.483 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.484 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.484 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:13:49.484 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.367823+0000", 2024-04-20T17:13:49.484 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.8199999999999992e-06 2024-04-20T17:13:49.484 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.484 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.484 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:13:49.484 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.367831+0000", 2024-04-20T17:13:49.484 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.418e-06 2024-04-20T17:13:49.484 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.484 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.484 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:13:49.484 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.387226+0000", 2024-04-20T17:13:49.484 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019395580999999999 2024-04-20T17:13:49.485 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.485 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.485 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:13:49.485 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.387251+0000", 2024-04-20T17:13:49.485 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4610999999999999e-05 2024-04-20T17:13:49.485 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.485 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.485 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:13:49.485 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.411929+0000", 2024-04-20T17:13:49.485 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024677999999999999 2024-04-20T17:13:49.485 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.485 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.485 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:13:49.485 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.411956+0000", 2024-04-20T17:13:49.485 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7129999999999999e-05 2024-04-20T17:13:49.486 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.486 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:13:49.486 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.486 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.486 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.486 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:31855 2.6 e210/115 2:609d09da:::benchmark_data_smithi171_78918_object118890:head v 210'13819, mlcod=210'13795)", 2024-04-20T17:13:49.486 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.372439+0000", 2024-04-20T17:13:49.486 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 68.051748767000007, 2024-04-20T17:13:49.486 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.039545667, 2024-04-20T17:13:49.486 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:13:49.486 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:13:49.486 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:49.486 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:13:49.486 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.487 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:13:49.487 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.372439+0000", 2024-04-20T17:13:49.487 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.487 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.487 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.487 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:13:49.487 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.372439+0000", 2024-04-20T17:13:49.487 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.487 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.487 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.487 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:13:49.487 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.372441+0000", 2024-04-20T17:13:49.487 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4839999999999999e-06 2024-04-20T17:13:49.487 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.487 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.488 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:13:49.488 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.372449+0000", 2024-04-20T17:13:49.488 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9349999999999994e-06 2024-04-20T17:13:49.488 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.488 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.488 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:13:49.488 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.372456+0000", 2024-04-20T17:13:49.488 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1110000000000002e-06 2024-04-20T17:13:49.488 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.488 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.488 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:13:49.488 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.372461+0000", 2024-04-20T17:13:49.488 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.711e-06 2024-04-20T17:13:49.489 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.489 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.489 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:13:49.489 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.391672+0000", 2024-04-20T17:13:49.489 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019210820999999999 2024-04-20T17:13:49.489 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.489 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.489 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:13:49.489 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.391694+0000", 2024-04-20T17:13:49.489 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2045999999999999e-05 2024-04-20T17:13:49.489 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.489 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.489 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:13:49.489 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.411959+0000", 2024-04-20T17:13:49.489 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020264379999999999 2024-04-20T17:13:49.490 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.490 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.490 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:13:49.490 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.411985+0000", 2024-04-20T17:13:49.490 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6179000000000001e-05 2024-04-20T17:13:49.490 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.490 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:13:49.490 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.490 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.490 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.490 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:31984 2.6 e210/115 2:60e80bc9:::benchmark_data_smithi171_78918_object105296:head v 210'13948, mlcod=210'13930)", 2024-04-20T17:13:49.490 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.957574+0000", 2024-04-20T17:13:49.490 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 67.466614011000004, 2024-04-20T17:13:49.491 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.039531605999999997, 2024-04-20T17:13:49.491 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:13:49.491 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:13:49.491 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:49.491 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:13:49.491 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.491 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:13:49.491 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.957574+0000", 2024-04-20T17:13:49.491 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.491 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.491 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.491 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:13:49.491 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.957574+0000", 2024-04-20T17:13:49.491 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.491 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.492 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.492 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:13:49.492 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.957575+0000", 2024-04-20T17:13:49.492 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.252e-06 2024-04-20T17:13:49.492 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.492 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.492 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:13:49.492 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.957583+0000", 2024-04-20T17:13:49.492 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5510000000000002e-06 2024-04-20T17:13:49.492 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.492 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.492 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:13:49.492 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.957593+0000", 2024-04-20T17:13:49.492 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0091999999999999e-05 2024-04-20T17:13:49.493 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.493 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.493 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:13:49.493 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.957598+0000", 2024-04-20T17:13:49.493 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2780000000000001e-06 2024-04-20T17:13:49.493 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.493 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.493 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:13:49.493 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.964817+0000", 2024-04-20T17:13:49.493 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0072190850000000001 2024-04-20T17:13:49.493 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.493 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.493 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:13:49.494 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.964835+0000", 2024-04-20T17:13:49.494 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7546000000000001e-05 2024-04-20T17:13:49.494 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.494 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.494 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:13:49.494 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.997055+0000", 2024-04-20T17:13:49.494 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032220293999999997 2024-04-20T17:13:49.494 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.494 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.494 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:13:49.494 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.997106+0000", 2024-04-20T17:13:49.494 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0507999999999998e-05 2024-04-20T17:13:49.495 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.495 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:13:49.495 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.495 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.495 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.495 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:31985 2.6 e210/115 2:60e8f41b:::benchmark_data_smithi171_78918_object156733:head v 210'13949, mlcod=210'13930)", 2024-04-20T17:13:49.495 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.960543+0000", 2024-04-20T17:13:49.495 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 67.463645243000002, 2024-04-20T17:13:49.495 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036587569, 2024-04-20T17:13:49.495 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:13:49.495 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:13:49.495 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:49.495 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:13:49.496 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.496 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:13:49.496 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.960543+0000", 2024-04-20T17:13:49.496 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.496 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.496 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.496 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:13:49.496 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.960543+0000", 2024-04-20T17:13:49.496 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.496 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.496 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.496 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:13:49.496 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.960545+0000", 2024-04-20T17:13:49.497 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2809999999999998e-06 2024-04-20T17:13:49.497 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.497 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.497 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:13:49.497 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.960555+0000", 2024-04-20T17:13:49.497 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0059e-05 2024-04-20T17:13:49.497 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.497 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.497 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:13:49.497 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.960566+0000", 2024-04-20T17:13:49.497 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1197e-05 2024-04-20T17:13:49.497 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.497 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.497 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:13:49.498 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.960572+0000", 2024-04-20T17:13:49.498 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.852e-06 2024-04-20T17:13:49.498 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.498 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.498 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:13:49.498 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.968088+0000", 2024-04-20T17:13:49.498 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.007515585 2024-04-20T17:13:49.498 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.498 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.498 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:13:49.498 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.968104+0000", 2024-04-20T17:13:49.498 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5818e-05 2024-04-20T17:13:49.498 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.498 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.499 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:13:49.499 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.997109+0000", 2024-04-20T17:13:49.499 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.029005025 2024-04-20T17:13:49.499 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.499 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.499 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:13:49.499 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.997130+0000", 2024-04-20T17:13:49.499 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1752000000000001e-05 2024-04-20T17:13:49.499 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.499 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:13:49.499 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.499 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.499 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.499 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:31986 2.6 e210/115 2:60e97f5f:::benchmark_data_smithi171_78918_object86440:head v 210'13950, mlcod=210'13930)", 2024-04-20T17:13:49.500 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.963757+0000", 2024-04-20T17:13:49.500 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 67.460431056999994, 2024-04-20T17:13:49.500 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.039352508000000001, 2024-04-20T17:13:49.500 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:13:49.500 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:13:49.500 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:49.500 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:13:49.500 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.500 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:13:49.500 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.963757+0000", 2024-04-20T17:13:49.500 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.500 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.500 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.501 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:13:49.501 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.963757+0000", 2024-04-20T17:13:49.501 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.501 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.501 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.501 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:13:49.501 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.963758+0000", 2024-04-20T17:13:49.501 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.48e-06 2024-04-20T17:13:49.501 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.501 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.501 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:13:49.501 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.963765+0000", 2024-04-20T17:13:49.501 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.72e-06 2024-04-20T17:13:49.501 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.502 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.502 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:13:49.502 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.963775+0000", 2024-04-20T17:13:49.502 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7340000000000003e-06 2024-04-20T17:13:49.502 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.502 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.502 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:13:49.502 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.963780+0000", 2024-04-20T17:13:49.502 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2619999999999998e-06 2024-04-20T17:13:49.502 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.502 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.502 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:13:49.502 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.971061+0000", 2024-04-20T17:13:49.503 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0072804319999999999 2024-04-20T17:13:49.503 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.503 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.503 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:13:49.503 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.971085+0000", 2024-04-20T17:13:49.503 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4885999999999999e-05 2024-04-20T17:13:49.503 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.503 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.503 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:13:49.503 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:42.003079+0000", 2024-04-20T17:13:49.503 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.031993261000000002 2024-04-20T17:13:49.503 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.503 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.504 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:13:49.504 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:42.003109+0000", 2024-04-20T17:13:49.504 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0732999999999999e-05 2024-04-20T17:13:49.504 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.504 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:13:49.504 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.504 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.504 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.504 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:31987 2.6 e210/115 2:60ea5806:::benchmark_data_smithi171_78918_object109244:head v 210'13951, mlcod=210'13930)", 2024-04-20T17:13:49.504 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.966919+0000", 2024-04-20T17:13:49.504 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 67.457268561000006, 2024-04-20T17:13:49.504 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036210481000000003, 2024-04-20T17:13:49.504 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:13:49.504 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:13:49.504 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:49.505 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:13:49.505 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.505 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:13:49.505 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.966919+0000", 2024-04-20T17:13:49.505 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.505 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.505 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.505 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:13:49.505 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.966919+0000", 2024-04-20T17:13:49.505 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.505 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.505 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.505 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:13:49.505 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.966921+0000", 2024-04-20T17:13:49.506 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5850000000000001e-06 2024-04-20T17:13:49.506 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.506 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.506 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:13:49.506 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.966929+0000", 2024-04-20T17:13:49.506 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4959999999999996e-06 2024-04-20T17:13:49.506 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.506 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.506 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:13:49.506 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.966939+0000", 2024-04-20T17:13:49.506 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7389999999999994e-06 2024-04-20T17:13:49.506 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.506 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.506 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:13:49.506 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.966945+0000", 2024-04-20T17:13:49.507 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1519999999999997e-06 2024-04-20T17:13:49.507 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.507 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.507 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:13:49.507 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.974737+0000", 2024-04-20T17:13:49.507 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.007791877 2024-04-20T17:13:49.507 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.507 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.507 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:13:49.507 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.974752+0000", 2024-04-20T17:13:49.507 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4504999999999999e-05 2024-04-20T17:13:49.507 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.507 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.508 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:13:49.508 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:42.003113+0000", 2024-04-20T17:13:49.508 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028360811999999999 2024-04-20T17:13:49.508 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.508 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.508 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:13:49.508 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:42.003130+0000", 2024-04-20T17:13:49.508 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7314999999999999e-05 2024-04-20T17:13:49.508 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.508 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:13:49.508 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.508 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.508 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.508 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:32412 2.6 e211/115 2:61ea66cc:::benchmark_data_smithi171_78918_object99072:head v 211'14376, mlcod=211'14352)", 2024-04-20T17:13:49.508 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:43.845406+0000", 2024-04-20T17:13:49.509 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 65.578781805000006, 2024-04-20T17:13:49.509 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.037156029, 2024-04-20T17:13:49.509 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:13:49.509 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:13:49.509 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:49.509 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:13:49.509 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.509 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:13:49.509 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.845406+0000", 2024-04-20T17:13:49.509 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.509 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.509 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.509 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:13:49.509 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.845406+0000", 2024-04-20T17:13:49.510 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.510 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.510 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.510 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:13:49.510 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.845408+0000", 2024-04-20T17:13:49.510 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3099999999999999e-06 2024-04-20T17:13:49.510 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.510 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.510 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:13:49.510 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.845418+0000", 2024-04-20T17:13:49.510 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3440000000000007e-06 2024-04-20T17:13:49.510 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.510 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.510 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:13:49.513 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.845430+0000", 2024-04-20T17:13:49.513 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2690999999999999e-05 2024-04-20T17:13:49.513 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.513 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.514 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:13:49.514 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.845438+0000", 2024-04-20T17:13:49.514 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7589999999999997e-06 2024-04-20T17:13:49.514 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.514 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.514 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:13:49.514 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.860483+0000", 2024-04-20T17:13:49.514 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015045127 2024-04-20T17:13:49.514 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.514 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.514 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:13:49.514 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.860506+0000", 2024-04-20T17:13:49.514 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2527e-05 2024-04-20T17:13:49.514 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.514 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.515 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:13:49.515 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.882522+0000", 2024-04-20T17:13:49.515 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022016029999999999 2024-04-20T17:13:49.515 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.515 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.515 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:13:49.515 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.882562+0000", 2024-04-20T17:13:49.515 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0241000000000003e-05 2024-04-20T17:13:49.515 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.515 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:13:49.515 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.515 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.515 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.515 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:32728 2.6 e212/115 2:629fe820:::benchmark_data_smithi171_78918_object95717:head v 212'14692, mlcod=212'14670)", 2024-04-20T17:13:49.515 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:45.227687+0000", 2024-04-20T17:13:49.516 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 64.196500994000004, 2024-04-20T17:13:49.516 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.037695732000000003, 2024-04-20T17:13:49.516 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:13:49.516 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:13:49.516 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:49.516 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:13:49.516 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.516 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:13:49.516 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.227687+0000", 2024-04-20T17:13:49.516 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.516 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.516 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.516 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:13:49.517 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.227687+0000", 2024-04-20T17:13:49.517 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.517 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.517 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.517 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:13:49.517 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.227688+0000", 2024-04-20T17:13:49.517 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.64e-07 2024-04-20T17:13:49.517 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.517 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.517 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:13:49.517 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.227694+0000", 2024-04-20T17:13:49.517 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.755e-06 2024-04-20T17:13:49.517 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.517 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.518 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:13:49.518 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.227703+0000", 2024-04-20T17:13:49.518 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.499e-06 2024-04-20T17:13:49.518 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.518 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.518 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:13:49.518 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.227709+0000", 2024-04-20T17:13:49.518 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3399999999999997e-06 2024-04-20T17:13:49.518 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.518 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.518 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:13:49.518 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.238353+0000", 2024-04-20T17:13:49.519 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010644345 2024-04-20T17:13:49.519 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.519 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.519 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:13:49.519 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.238375+0000", 2024-04-20T17:13:49.519 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1691999999999998e-05 2024-04-20T17:13:49.519 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.519 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.519 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:13:49.519 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.265339+0000", 2024-04-20T17:13:49.519 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026964251000000002 2024-04-20T17:13:49.519 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.519 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.519 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:13:49.519 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.265383+0000", 2024-04-20T17:13:49.520 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3886e-05 2024-04-20T17:13:49.520 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.520 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:13:49.520 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.520 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.520 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.520 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:33314 2.6 e213/115 2:63f10b9a:::benchmark_data_smithi171_78918_object142360:head v 213'15278, mlcod=213'15258)", 2024-04-20T17:13:49.520 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:47.737543+0000", 2024-04-20T17:13:49.520 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 61.686645095000003, 2024-04-20T17:13:49.520 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.038417434, 2024-04-20T17:13:49.520 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:13:49.520 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:13:49.520 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:13:49.520 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:13:49.520 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.521 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:13:49.521 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.737543+0000", 2024-04-20T17:13:49.521 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.521 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.521 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.521 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:13:49.521 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.737543+0000", 2024-04-20T17:13:49.521 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:13:49.521 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.521 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.521 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:13:49.521 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.737544+0000", 2024-04-20T17:13:49.521 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3549999999999999e-06 2024-04-20T17:13:49.521 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.522 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.522 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:13:49.522 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.737552+0000", 2024-04-20T17:13:49.522 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1079999999999993e-06 2024-04-20T17:13:49.522 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.522 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.522 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:13:49.522 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.737563+0000", 2024-04-20T17:13:49.522 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0957e-05 2024-04-20T17:13:49.522 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.522 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.522 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:13:49.522 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.737571+0000", 2024-04-20T17:13:49.522 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9850000000000006e-06 2024-04-20T17:13:49.523 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.523 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.523 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:13:49.523 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.743454+0000", 2024-04-20T17:13:49.523 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0058832340000000002 2024-04-20T17:13:49.523 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.523 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.523 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:13:49.523 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.743475+0000", 2024-04-20T17:13:49.523 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0472999999999999e-05 2024-04-20T17:13:49.523 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.523 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.523 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:13:49.523 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.775922+0000", 2024-04-20T17:13:49.523 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032446572999999999 2024-04-20T17:13:49.523 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:13:49.524 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:13:49.524 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:13:49.524 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.775960+0000", 2024-04-20T17:13:49.524 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8748999999999998e-05 2024-04-20T17:13:49.524 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.524 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:13:49.524 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.524 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:13:49.524 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:13:49.524 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:13:49.524 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:13:49.554 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:50.012 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 9 1 2024-04-20T17:13:50.207 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:49 smithi171 ceph-mon[44189]: pgmap v656: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 130 op/s; 10347/183124 objects misplaced (5.650%); 341 KiB/s, 5 objects/s recovering 2024-04-20T17:13:50.207 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:50.207 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:49 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2242134099' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-20T17:13:50.207 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:49 smithi171 ceph-mon[44189]: osdmap e236: 12 total, 12 up, 12 in 2024-04-20T17:13:50.208 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:13:50.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:49 smithi005 ceph-mon[39562]: pgmap v656: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 130 op/s; 10347/183124 objects misplaced (5.650%); 341 KiB/s, 5 objects/s recovering 2024-04-20T17:13:50.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:50.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:49 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2242134099' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-20T17:13:50.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:49 smithi005 ceph-mon[39562]: osdmap e236: 12 total, 12 up, 12 in 2024-04-20T17:13:50.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:50.213 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:13:50.242 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-20T17:13:50.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:49 smithi106 ceph-mon[44423]: pgmap v656: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 130 op/s; 10347/183124 objects misplaced (5.650%); 341 KiB/s, 5 objects/s recovering 2024-04-20T17:13:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:49 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2242134099' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-20T17:13:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:49 smithi106 ceph-mon[44423]: osdmap e236: 12 total, 12 up, 12 in 2024-04-20T17:13:50.313 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:13:50.339 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:50.358 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:13:50.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:50.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:13:50.462 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:13:50.490 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:50.514 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:13:50.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:50.517 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:13:50.562 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:13:50.618 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:13:50.649 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:50.678 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:13:50.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:50.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:13:50.746 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:13:50.785 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:13:50.811 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:50.829 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:13:50.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:50.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:13:50.933 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:13:50.962 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:51.062 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:13:51.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:51.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:13:51.166 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:13:51.199 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:51.239 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:13:51.349 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:13:51.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:51.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:13:51.455 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:13:51.483 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:51.581 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:13:51.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:51.585 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:13:51.686 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:13:51.712 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:51.715 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:13:51.961 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:13:51.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:51.963 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:13:52.064 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:13:52.092 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:52.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:51 smithi005 ceph-mon[39562]: pgmap v658: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 103 op/s; 10273/182800 objects misplaced (5.620%); 384 KiB/s, 5 objects/s recovering 2024-04-20T17:13:52.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:51 smithi106 ceph-mon[44423]: pgmap v658: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 103 op/s; 10273/182800 objects misplaced (5.620%); 384 KiB/s, 5 objects/s recovering 2024-04-20T17:13:52.277 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:51 smithi171 ceph-mon[44189]: pgmap v658: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 103 op/s; 10273/182800 objects misplaced (5.620%); 384 KiB/s, 5 objects/s recovering 2024-04-20T17:13:52.289 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:13:52.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:52.293 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:13:52.394 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:13:52.422 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:52.442 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:13:52.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:52.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:13:52.536 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:13:52.536 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:13:52.536 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 0 2024-04-20T17:13:52.536 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:13:52.546 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:13:52.574 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:52.593 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:13:52.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:52.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:13:52.697 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:13:52.702 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:13:52.724 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:52.743 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:13:52.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:52.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:13:52.847 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:13:52.876 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:52.914 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:13:52.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:52.921 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:13:53.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:52 smithi106 ceph-mon[44423]: pgmap v659: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 128 op/s; 10273/182184 objects misplaced (5.639%); 376 KiB/s, 5 objects/s recovering 2024-04-20T17:13:53.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:52 smithi106 ceph-mon[44423]: osdmap e237: 12 total, 12 up, 12 in 2024-04-20T17:13:53.022 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:13:53.048 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:53.065 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:13:53.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:53.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:13:53.137 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:52 smithi005 ceph-mon[39562]: pgmap v659: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 128 op/s; 10273/182184 objects misplaced (5.639%); 376 KiB/s, 5 objects/s recovering 2024-04-20T17:13:53.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:52 smithi005 ceph-mon[39562]: osdmap e237: 12 total, 12 up, 12 in 2024-04-20T17:13:53.169 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:13:53.196 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:53.214 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:13:53.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:53.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:13:53.246 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:52 smithi171 ceph-mon[44189]: pgmap v659: 17 pgs: 1 active+recovering, 16 active+clean; 5.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 128 op/s; 10273/182184 objects misplaced (5.639%); 376 KiB/s, 5 objects/s recovering 2024-04-20T17:13:53.246 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:52 smithi171 ceph-mon[44189]: osdmap e237: 12 total, 12 up, 12 in 2024-04-20T17:13:53.273 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-20T17:13:53.318 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:13:53.344 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:53.362 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:13:53.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:53.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:13:53.466 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:13:53.501 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:53.558 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:13:53.973 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.9 to 1 (10000) 2024-04-20T17:13:54.010 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:13:54.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:53 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:13:54.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:53 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:13:54.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:53 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2139959944' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:13:54.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:53 smithi171 ceph-mon[44189]: from='client.25148 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:13:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:53 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:13:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:53 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:13:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:53 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2139959944' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:13:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:53 smithi106 ceph-mon[44423]: from='client.25148 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:13:54.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:53 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:13:54.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:53 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:13:54.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:53 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2139959944' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:13:54.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:53 smithi005 ceph-mon[39562]: from='client.25148 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:13:54.557 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:13:54.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:54.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:13:54.660 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:13:54.687 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:54.936 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:13:54.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:54.943 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:13:55.044 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:13:55.071 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:55.231 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:54 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:13:55.231 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:54 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:13:55.231 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:54 smithi005 ceph-mon[39562]: from='client.25148 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-20T17:13:55.232 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:54 smithi005 ceph-mon[39562]: osdmap e238: 12 total, 12 up, 12 in 2024-04-20T17:13:55.232 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:54 smithi005 ceph-mon[39562]: pgmap v662: 17 pgs: 1 active+recovering, 16 active+clean; 5.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 144 op/s; 10200/181392 objects misplaced (5.623%); 779 KiB/s, 12 objects/s recovering 2024-04-20T17:13:55.233 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 10 1 2024-04-20T17:13:55.250 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:54 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:13:55.250 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:54 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:13:55.250 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:54 smithi171 ceph-mon[44189]: from='client.25148 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-20T17:13:55.251 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:54 smithi171 ceph-mon[44189]: osdmap e238: 12 total, 12 up, 12 in 2024-04-20T17:13:55.251 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:54 smithi171 ceph-mon[44189]: pgmap v662: 17 pgs: 1 active+recovering, 16 active+clean; 5.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 144 op/s; 10200/181392 objects misplaced (5.623%); 779 KiB/s, 12 objects/s recovering 2024-04-20T17:13:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:54 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:13:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:54 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:13:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:54 smithi106 ceph-mon[44423]: from='client.25148 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-20T17:13:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:54 smithi106 ceph-mon[44423]: osdmap e238: 12 total, 12 up, 12 in 2024-04-20T17:13:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:54 smithi106 ceph-mon[44423]: pgmap v662: 17 pgs: 1 active+recovering, 16 active+clean; 5.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 144 op/s; 10200/181392 objects misplaced (5.623%); 779 KiB/s, 12 objects/s recovering 2024-04-20T17:13:55.269 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:13:55.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:55.272 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:13:55.373 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:13:55.407 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:55.562 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:13:55.562 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:13:55.562 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-04-20T17:13:55.562 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-04-20T17:13:55.562 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:13:55.760 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:13:55.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:55.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:13:55.864 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:13:55.893 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:55.913 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:13:55.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:55.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:13:56.017 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:13:56.048 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:56.377 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-20T17:13:56.398 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:13:56.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:56.400 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:13:56.501 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:13:56.527 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:56.545 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:13:56.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:56.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:13:56.649 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:13:56.677 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:13:56.677 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:56.700 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:13:56.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:56.706 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:13:56.807 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:13:56.836 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:57.127 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:57 smithi005 ceph-mon[39562]: pgmap v663: 17 pgs: 1 active+recovering, 16 active+clean; 5.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 120 op/s; 10200/181078 objects misplaced (5.633%); 333 KiB/s, 5 objects/s recovering 2024-04-20T17:13:57.138 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:13:57.206 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:57 smithi171 ceph-mon[44189]: pgmap v663: 17 pgs: 1 active+recovering, 16 active+clean; 5.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 120 op/s; 10200/181078 objects misplaced (5.633%); 333 KiB/s, 5 objects/s recovering 2024-04-20T17:13:57.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:57 smithi106 ceph-mon[44423]: pgmap v663: 17 pgs: 1 active+recovering, 16 active+clean; 5.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 120 op/s; 10200/181078 objects misplaced (5.633%); 333 KiB/s, 5 objects/s recovering 2024-04-20T17:13:57.879 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:13:57.879 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:13:57.879 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:13:57.879 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:13:57.879 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:13:57.880 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:13:57.880 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:13:57.880 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:13:57.880 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:13:57.890 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:13:57.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:57.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:13:57.909 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:13:57.932 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:13:57.957 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:13:57.959 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:13:57.959 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:13:57.959 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:13:57.959 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:13:57.959 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:13:57.959 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:13:57.959 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:13:57.959 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:13:57.959 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:13:57.959 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:13:57.960 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:13:57.960 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:13:57.995 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:13:58.021 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:58.039 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:13:58.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:58.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:13:58.081 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:13:58.142 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:13:58.169 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:58.187 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:13:58.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:58.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:13:58.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:58 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:13:58.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:58 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:13:58.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:58 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1406207920' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:13:58.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:58.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:58 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3729082868' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:13:58.290 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:13:58.314 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:58 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:13:58.315 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:58 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:13:58.315 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:58 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1406207920' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:13:58.315 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:58.315 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:58 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3729082868' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:13:58.317 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:58.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:58 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:13:58.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:58 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:13:58.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:58 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1406207920' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:13:58.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:58.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:58 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3729082868' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:13:58.335 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:13:58.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:58.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:13:58.438 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:13:58.467 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:58.714 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:13:58.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:58.718 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:13:58.719 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:13:58.719 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:13:58.819 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:13:58.845 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:58.864 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:13:58.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:58.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:13:58.968 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:13:58.998 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:59 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:13:59.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:59 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3729082868' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:13:59.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:59 smithi106 ceph-mon[44423]: osdmap e239: 12 total, 12 up, 12 in 2024-04-20T17:13:59.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:59 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:13:59.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:59 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:13:59.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:59 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:13:59.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:59 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:13:59.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:59 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:13:59.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:59 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:13:59.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:59 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:13:59.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:13:59 smithi106 ceph-mon[44423]: 2.e scrub starts 2024-04-20T17:13:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:59 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:13:59.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:59 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3729082868' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:13:59.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:59 smithi171 ceph-mon[44189]: osdmap e239: 12 total, 12 up, 12 in 2024-04-20T17:13:59.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:59 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:13:59.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:59 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:13:59.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:59 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:13:59.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:59 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:13:59.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:59 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:13:59.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:59 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:13:59.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:59 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:13:59.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:13:59 smithi171 ceph-mon[44189]: 2.e scrub starts 2024-04-20T17:13:59.382 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.383 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.383 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.383 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.383 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.383 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.383 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.383 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.383 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.383 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.383 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:13:59.383 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:59 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:13:59.383 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:59 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3729082868' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:13:59.383 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:59 smithi005 ceph-mon[39562]: osdmap e239: 12 total, 12 up, 12 in 2024-04-20T17:13:59.383 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:59 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:13:59.384 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:59 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:13:59.384 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:59 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:13:59.384 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:59 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:13:59.384 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:59 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:13:59.384 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:59 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:13:59.384 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:59 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:13:59.384 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:13:59 smithi005 ceph-mon[39562]: 2.e scrub starts 2024-04-20T17:13:59.521 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-20T17:13:59.547 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:13:59.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:13:59.550 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:13:59.651 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:13:59.683 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:13:59.813 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:14:00.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:00 smithi171 ceph-mon[44189]: 2.e scrub ok 2024-04-20T17:14:00.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:00 smithi171 ceph-mon[44189]: 2.1 scrub starts 2024-04-20T17:14:00.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:00 smithi005 ceph-mon[39562]: 2.e scrub ok 2024-04-20T17:14:00.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:00 smithi005 ceph-mon[39562]: 2.1 scrub starts 2024-04-20T17:14:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:00 smithi106 ceph-mon[44423]: 2.e scrub ok 2024-04-20T17:14:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:00 smithi106 ceph-mon[44423]: 2.1 scrub starts 2024-04-20T17:14:00.862 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:14:00.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:00.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:14:00.966 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:14:00.998 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:01.017 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:14:01.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:01.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:14:01.121 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:14:01.150 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:01.174 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:14:01.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:01.175 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:14:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:01 smithi005 ceph-mon[39562]: 2.1 scrub ok 2024-04-20T17:14:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:01 smithi005 ceph-mon[39562]: pgmap v666: 17 pgs: 1 active+recovering, 16 active+clean; 5.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 107 op/s; 10127/180174 objects misplaced (5.621%); 676 KiB/s, 10 objects/s recovering 2024-04-20T17:14:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:01 smithi005 ceph-mon[39562]: osdmap e240: 12 total, 12 up, 12 in 2024-04-20T17:14:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:01 smithi106 ceph-mon[44423]: 2.1 scrub ok 2024-04-20T17:14:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:01 smithi106 ceph-mon[44423]: pgmap v666: 17 pgs: 1 active+recovering, 16 active+clean; 5.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 107 op/s; 10127/180174 objects misplaced (5.621%); 676 KiB/s, 10 objects/s recovering 2024-04-20T17:14:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:01 smithi106 ceph-mon[44423]: osdmap e240: 12 total, 12 up, 12 in 2024-04-20T17:14:01.275 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:14:01.307 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:01.432 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:14:01.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:01.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:14:01.498 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:01 smithi171 ceph-mon[44189]: 2.1 scrub ok 2024-04-20T17:14:01.498 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:01 smithi171 ceph-mon[44189]: pgmap v666: 17 pgs: 1 active+recovering, 16 active+clean; 5.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 107 op/s; 10127/180174 objects misplaced (5.621%); 676 KiB/s, 10 objects/s recovering 2024-04-20T17:14:01.498 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:01 smithi171 ceph-mon[44189]: osdmap e240: 12 total, 12 up, 12 in 2024-04-20T17:14:01.536 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:14:01.573 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:01.651 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:14:01.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:01.655 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:14:01.756 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:14:01.788 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:01.801 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:14:01.801 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:14:01.811 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:14:01.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:01.814 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:14:01.864 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:14:01.915 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:14:01.941 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:01.960 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:14:01.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:01.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:14:02.063 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:14:02.098 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.10 to 1 (10000) 2024-04-20T17:14:02.098 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:02.127 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:14:02.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:02.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:14:02.163 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:14:02.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:02 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3107218352' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:14:02.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:02 smithi005 ceph-mon[39562]: from='client.25166 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:14:02.231 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:14:02.260 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:02.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:02 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3107218352' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:14:02.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:02 smithi106 ceph-mon[44423]: from='client.25166 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:14:02.316 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:14:02.388 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:14:02.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:02.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:14:02.493 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:14:02.506 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:02 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3107218352' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:14:02.506 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:02 smithi171 ceph-mon[44189]: from='client.25166 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:14:02.523 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:02.532 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-20T17:14:02.563 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:14:02.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:02.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:14:02.670 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:14:02.703 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:02.812 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:14:02.947 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 11 1 2024-04-20T17:14:03.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:03 smithi005 ceph-mon[39562]: 2.0 scrub starts 2024-04-20T17:14:03.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:03 smithi005 ceph-mon[39562]: 2.0 scrub ok 2024-04-20T17:14:03.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:03 smithi005 ceph-mon[39562]: pgmap v668: 17 pgs: 1 active+recovering, 16 active+clean; 5.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 114 op/s; 10127/179566 objects misplaced (5.640%); 304 KiB/s, 4 objects/s recovering 2024-04-20T17:14:03.184 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:03 smithi005 ceph-mon[39562]: from='client.25166 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-20T17:14:03.184 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:03 smithi005 ceph-mon[39562]: osdmap e241: 12 total, 12 up, 12 in 2024-04-20T17:14:03.184 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:03 smithi005 ceph-mon[39562]: 2.4 scrub starts 2024-04-20T17:14:03.253 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:14:03.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:03.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:14:03.357 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:14:03.387 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:03 smithi171 ceph-mon[44189]: 2.0 scrub starts 2024-04-20T17:14:03.387 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:03 smithi171 ceph-mon[44189]: 2.0 scrub ok 2024-04-20T17:14:03.387 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:03 smithi171 ceph-mon[44189]: pgmap v668: 17 pgs: 1 active+recovering, 16 active+clean; 5.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 114 op/s; 10127/179566 objects misplaced (5.640%); 304 KiB/s, 4 objects/s recovering 2024-04-20T17:14:03.387 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:03 smithi171 ceph-mon[44189]: from='client.25166 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-20T17:14:03.387 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:03 smithi171 ceph-mon[44189]: osdmap e241: 12 total, 12 up, 12 in 2024-04-20T17:14:03.387 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:03 smithi171 ceph-mon[44189]: 2.4 scrub starts 2024-04-20T17:14:03.390 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:03 smithi106 ceph-mon[44423]: 2.0 scrub starts 2024-04-20T17:14:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:03 smithi106 ceph-mon[44423]: 2.0 scrub ok 2024-04-20T17:14:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:03 smithi106 ceph-mon[44423]: pgmap v668: 17 pgs: 1 active+recovering, 16 active+clean; 5.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 114 op/s; 10127/179566 objects misplaced (5.640%); 304 KiB/s, 4 objects/s recovering 2024-04-20T17:14:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:03 smithi106 ceph-mon[44423]: from='client.25166 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-20T17:14:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:03 smithi106 ceph-mon[44423]: osdmap e241: 12 total, 12 up, 12 in 2024-04-20T17:14:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:03 smithi106 ceph-mon[44423]: 2.4 scrub starts 2024-04-20T17:14:03.759 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:14:03.796 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:14:04.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:04 smithi171 ceph-mon[44189]: 2.a scrub starts 2024-04-20T17:14:04.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:04 smithi171 ceph-mon[44189]: 2.4 scrub ok 2024-04-20T17:14:04.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:04 smithi171 ceph-mon[44189]: 2.5 scrub starts 2024-04-20T17:14:04.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:04 smithi171 ceph-mon[44189]: 2.a scrub ok 2024-04-20T17:14:04.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:04 smithi171 ceph-mon[44189]: 2.5 scrub ok 2024-04-20T17:14:04.343 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:14:04.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:04.346 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:14:04.404 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:04 smithi005 ceph-mon[39562]: 2.a scrub starts 2024-04-20T17:14:04.404 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:04 smithi005 ceph-mon[39562]: 2.4 scrub ok 2024-04-20T17:14:04.404 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:04 smithi005 ceph-mon[39562]: 2.5 scrub starts 2024-04-20T17:14:04.404 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:04 smithi005 ceph-mon[39562]: 2.a scrub ok 2024-04-20T17:14:04.404 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:04 smithi005 ceph-mon[39562]: 2.5 scrub ok 2024-04-20T17:14:04.447 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:14:04.474 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:04.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:04 smithi106 ceph-mon[44423]: 2.a scrub starts 2024-04-20T17:14:04.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:04 smithi106 ceph-mon[44423]: 2.4 scrub ok 2024-04-20T17:14:04.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:04 smithi106 ceph-mon[44423]: 2.5 scrub starts 2024-04-20T17:14:04.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:04 smithi106 ceph-mon[44423]: 2.a scrub ok 2024-04-20T17:14:04.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:04 smithi106 ceph-mon[44423]: 2.5 scrub ok 2024-04-20T17:14:04.622 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:14:04.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:04.626 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:14:04.727 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:14:04.755 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:04.775 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:14:04.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:04.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:14:04.879 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:14:04.911 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:04.924 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:14:04.924 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:14:05.062 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:14:05.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:05.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:14:05.165 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:14:05.191 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:05.210 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:14:05.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:05.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:14:05.314 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:14:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:05 smithi171 ceph-mon[44189]: 2.c scrub starts 2024-04-20T17:14:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:05 smithi171 ceph-mon[44189]: pgmap v670: 17 pgs: 1 active+recovering, 16 active+clean; 5.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 155 op/s; 10051/178690 objects misplaced (5.625%); 800 KiB/s, 12 objects/s recovering 2024-04-20T17:14:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:05 smithi171 ceph-mon[44189]: 2.c scrub ok 2024-04-20T17:14:05.340 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:05.353 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:05 smithi005 ceph-mon[39562]: 2.c scrub starts 2024-04-20T17:14:05.353 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:05 smithi005 ceph-mon[39562]: pgmap v670: 17 pgs: 1 active+recovering, 16 active+clean; 5.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 155 op/s; 10051/178690 objects misplaced (5.625%); 800 KiB/s, 12 objects/s recovering 2024-04-20T17:14:05.353 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:05 smithi005 ceph-mon[39562]: 2.c scrub ok 2024-04-20T17:14:05.359 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:14:05.362 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:05 smithi106 ceph-mon[44423]: 2.c scrub starts 2024-04-20T17:14:05.362 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:05 smithi106 ceph-mon[44423]: pgmap v670: 17 pgs: 1 active+recovering, 16 active+clean; 5.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 155 op/s; 10051/178690 objects misplaced (5.625%); 800 KiB/s, 12 objects/s recovering 2024-04-20T17:14:05.362 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:05 smithi106 ceph-mon[44423]: 2.c scrub ok 2024-04-20T17:14:05.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:05.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:14:05.465 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:14:05.494 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:05.807 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-20T17:14:06.071 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:14:06.348 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:14:06.350 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:06 smithi005 ceph-mon[39562]: 2.b scrub starts 2024-04-20T17:14:06.350 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:06 smithi005 ceph-mon[39562]: 2.b scrub ok 2024-04-20T17:14:06.350 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:06 smithi005 ceph-mon[39562]: osdmap e242: 12 total, 12 up, 12 in 2024-04-20T17:14:06.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:06.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:14:06.368 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:06 smithi171 ceph-mon[44189]: 2.b scrub starts 2024-04-20T17:14:06.368 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:06 smithi171 ceph-mon[44189]: 2.b scrub ok 2024-04-20T17:14:06.368 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:06 smithi171 ceph-mon[44189]: osdmap e242: 12 total, 12 up, 12 in 2024-04-20T17:14:06.452 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:14:06.479 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:06.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:06 smithi106 ceph-mon[44423]: 2.b scrub starts 2024-04-20T17:14:06.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:06 smithi106 ceph-mon[44423]: 2.b scrub ok 2024-04-20T17:14:06.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:06 smithi106 ceph-mon[44423]: osdmap e242: 12 total, 12 up, 12 in 2024-04-20T17:14:06.577 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:14:06.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:06.580 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:14:06.661 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:14:06.681 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:14:06.683 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:14:06.705 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:14:06.713 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:06.726 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:14:06.747 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:14:06.767 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:14:06.787 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:14:06.808 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:14:06.816 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:14:06.832 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:14:06.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:06.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:14:06.854 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:14:06.889 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:14:06.921 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:14:06.922 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:14:06.922 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:14:06.922 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:14:06.923 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:14:06.923 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:14:06.923 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:14:06.923 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:14:06.923 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:14:06.923 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:14:06.923 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:14:06.923 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:14:06.923 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:14:06.934 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:14:06.963 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:06.982 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:14:06.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:06.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:14:07.086 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:14:07.117 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:07.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:07 smithi106 ceph-mon[44423]: pgmap v672: 17 pgs: 1 active+recovering, 16 active+clean; 5.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 157 op/s; 10051/178286 objects misplaced (5.638%); 395 KiB/s, 6 objects/s recovering 2024-04-20T17:14:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:07 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3241769527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:14:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:07.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:07 smithi005 ceph-mon[39562]: pgmap v672: 17 pgs: 1 active+recovering, 16 active+clean; 5.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 157 op/s; 10051/178286 objects misplaced (5.638%); 395 KiB/s, 6 objects/s recovering 2024-04-20T17:14:07.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:07 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3241769527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:14:07.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:07.496 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:07 smithi171 ceph-mon[44189]: pgmap v672: 17 pgs: 1 active+recovering, 16 active+clean; 5.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 157 op/s; 10051/178286 objects misplaced (5.638%); 395 KiB/s, 6 objects/s recovering 2024-04-20T17:14:07.496 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:07 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3241769527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:14:07.496 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.215 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:14:08.215 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:14:08.215 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:14:08.216 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:14:08.216 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.216 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:194660 2.3 e163/150 2:c260d96c:::benchmark_data_smithi171_78918_object33666:head v 163'11958, mlcod=122'2109)", 2024-04-20T17:14:08.216 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:29.397421+0000", 2024-04-20T17:14:08.216 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 218.816414917, 2024-04-20T17:14:08.216 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.039947886000000002, 2024-04-20T17:14:08.216 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:14:08.216 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:14:08.216 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:08.216 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:14:08.216 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.216 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:14:08.216 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.397421+0000", 2024-04-20T17:14:08.216 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.217 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.217 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.217 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:14:08.217 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.397421+0000", 2024-04-20T17:14:08.217 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.217 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.217 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.217 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:14:08.217 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.397423+0000", 2024-04-20T17:14:08.217 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7630000000000001e-06 2024-04-20T17:14:08.217 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.217 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.217 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:14:08.217 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.397432+0000", 2024-04-20T17:14:08.217 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.541e-06 2024-04-20T17:14:08.218 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.218 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.218 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:14:08.218 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.397443+0000", 2024-04-20T17:14:08.218 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1776999999999999e-05 2024-04-20T17:14:08.218 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.218 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.218 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:14:08.218 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.397450+0000", 2024-04-20T17:14:08.218 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7179999999999999e-06 2024-04-20T17:14:08.218 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.218 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.218 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:14:08.218 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.404629+0000", 2024-04-20T17:14:08.219 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.007178551 2024-04-20T17:14:08.219 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.219 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.219 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:14:08.219 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.404652+0000", 2024-04-20T17:14:08.219 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3575e-05 2024-04-20T17:14:08.219 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.219 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.219 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:14:08.219 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.437330+0000", 2024-04-20T17:14:08.219 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032677465000000003 2024-04-20T17:14:08.219 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.219 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.219 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:14:08.220 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.437369+0000", 2024-04-20T17:14:08.220 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9496000000000003e-05 2024-04-20T17:14:08.220 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.220 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:14:08.220 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.220 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.220 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.220 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:194796 2.3 e163/150 2:c824a312:::benchmark_data_smithi171_78918_object33802:head v 163'11966, mlcod=122'2112)", 2024-04-20T17:14:08.220 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:29.533354+0000", 2024-04-20T17:14:08.220 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 218.680482329, 2024-04-20T17:14:08.220 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0406288, 2024-04-20T17:14:08.220 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:14:08.220 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:14:08.220 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:08.220 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:14:08.221 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.221 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:14:08.221 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.533354+0000", 2024-04-20T17:14:08.221 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.221 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.221 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.221 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:14:08.221 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.533354+0000", 2024-04-20T17:14:08.221 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.221 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.221 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.221 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:14:08.221 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.533355+0000", 2024-04-20T17:14:08.222 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3999999999999999e-06 2024-04-20T17:14:08.222 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.222 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.222 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:14:08.222 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.533367+0000", 2024-04-20T17:14:08.222 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1691e-05 2024-04-20T17:14:08.222 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.222 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.222 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:14:08.222 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.533377+0000", 2024-04-20T17:14:08.222 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.8199999999999992e-06 2024-04-20T17:14:08.222 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.222 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.222 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:14:08.222 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.533393+0000", 2024-04-20T17:14:08.223 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6192999999999999e-05 2024-04-20T17:14:08.223 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.223 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.223 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:14:08.223 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.542134+0000", 2024-04-20T17:14:08.223 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0087409270000000008 2024-04-20T17:14:08.223 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.223 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.223 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:14:08.223 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.542162+0000", 2024-04-20T17:14:08.223 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7766000000000001e-05 2024-04-20T17:14:08.223 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.223 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.223 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:14:08.224 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.573938+0000", 2024-04-20T17:14:08.224 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.031776739999999998 2024-04-20T17:14:08.224 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.224 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.224 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:14:08.224 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.573983+0000", 2024-04-20T17:14:08.224 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4263e-05 2024-04-20T17:14:08.224 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.224 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:14:08.224 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.224 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.224 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.224 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:194810 2.3 e163/150 2:c314b47a:::benchmark_data_smithi171_78918_object33816:head v 163'11968, mlcod=122'2112)", 2024-04-20T17:14:08.224 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:29.541537+0000", 2024-04-20T17:14:08.224 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 218.672298832, 2024-04-20T17:14:08.224 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.046523396000000002, 2024-04-20T17:14:08.225 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:14:08.225 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:14:08.225 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:08.225 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:14:08.225 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.225 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:14:08.225 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.541537+0000", 2024-04-20T17:14:08.225 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.225 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.225 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.225 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:14:08.225 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.541537+0000", 2024-04-20T17:14:08.225 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.225 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.225 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.226 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:14:08.226 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.541539+0000", 2024-04-20T17:14:08.226 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4300000000000001e-06 2024-04-20T17:14:08.226 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.226 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.226 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:14:08.226 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.541547+0000", 2024-04-20T17:14:08.226 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1049999999999995e-06 2024-04-20T17:14:08.226 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.226 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.226 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:14:08.226 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.541558+0000", 2024-04-20T17:14:08.226 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0915e-05 2024-04-20T17:14:08.226 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.227 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.227 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:14:08.227 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.541565+0000", 2024-04-20T17:14:08.227 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8469999999999998e-06 2024-04-20T17:14:08.227 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.227 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.227 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:14:08.227 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.557126+0000", 2024-04-20T17:14:08.227 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015561478 2024-04-20T17:14:08.227 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.227 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.227 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:14:08.227 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.557145+0000", 2024-04-20T17:14:08.227 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8454e-05 2024-04-20T17:14:08.227 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.228 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.228 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:14:08.228 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.588028+0000", 2024-04-20T17:14:08.228 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030883058000000001 2024-04-20T17:14:08.228 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.228 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.228 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:14:08.228 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.588061+0000", 2024-04-20T17:14:08.228 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3108999999999998e-05 2024-04-20T17:14:08.228 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.228 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:14:08.228 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.228 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.228 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.228 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:194816 2.3 e163/150 2:c4e181df:::benchmark_data_smithi171_78918_object33822:head v 163'11969, mlcod=122'2112)", 2024-04-20T17:14:08.229 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:29.545777+0000", 2024-04-20T17:14:08.229 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 218.668059384, 2024-04-20T17:14:08.229 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.042317029999999999, 2024-04-20T17:14:08.229 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:14:08.229 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:14:08.229 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:08.229 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:14:08.229 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.229 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:14:08.229 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.545777+0000", 2024-04-20T17:14:08.229 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.229 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.229 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.229 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:14:08.230 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.545777+0000", 2024-04-20T17:14:08.230 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.230 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.230 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.230 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:14:08.230 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.545779+0000", 2024-04-20T17:14:08.230 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.717e-06 2024-04-20T17:14:08.230 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.230 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.230 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:14:08.230 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.545789+0000", 2024-04-20T17:14:08.230 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0498e-05 2024-04-20T17:14:08.230 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.230 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.230 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:14:08.231 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.545803+0000", 2024-04-20T17:14:08.231 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3423e-05 2024-04-20T17:14:08.231 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.231 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.231 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:14:08.231 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.545810+0000", 2024-04-20T17:14:08.231 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8869999999999992e-06 2024-04-20T17:14:08.231 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.231 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.231 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:14:08.231 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.565547+0000", 2024-04-20T17:14:08.231 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019736581999999999 2024-04-20T17:14:08.231 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.231 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.232 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:14:08.232 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.565579+0000", 2024-04-20T17:14:08.232 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2079999999999998e-05 2024-04-20T17:14:08.232 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.232 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.232 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:14:08.232 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.588064+0000", 2024-04-20T17:14:08.232 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022484671000000001 2024-04-20T17:14:08.232 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.232 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.232 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:14:08.232 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.588094+0000", 2024-04-20T17:14:08.232 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0171999999999998e-05 2024-04-20T17:14:08.232 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.232 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:14:08.233 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.233 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.233 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.233 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:196209 2.3 e166/150 2:ca2be9e5:::benchmark_data_smithi171_78918_object35215:head v 166'12051, mlcod=122'2264)", 2024-04-20T17:14:08.233 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:33.094691+0000", 2024-04-20T17:14:08.233 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 215.119145062, 2024-04-20T17:14:08.233 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043283582000000001, 2024-04-20T17:14:08.233 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:14:08.233 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:14:08.233 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:08.233 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:14:08.233 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.233 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:14:08.233 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.094691+0000", 2024-04-20T17:14:08.233 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.234 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.234 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.234 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:14:08.234 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.094691+0000", 2024-04-20T17:14:08.234 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.234 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.234 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.234 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:14:08.234 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.094692+0000", 2024-04-20T17:14:08.234 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9599999999999999e-07 2024-04-20T17:14:08.234 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.234 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.234 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:14:08.235 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.094696+0000", 2024-04-20T17:14:08.235 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3649999999999997e-06 2024-04-20T17:14:08.235 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.235 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.235 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:14:08.235 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.094701+0000", 2024-04-20T17:14:08.235 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6419999999999998e-06 2024-04-20T17:14:08.235 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.235 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.235 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:14:08.235 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.094705+0000", 2024-04-20T17:14:08.235 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4819999999999996e-06 2024-04-20T17:14:08.235 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.235 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.235 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:14:08.236 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.103126+0000", 2024-04-20T17:14:08.236 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0084207180000000006 2024-04-20T17:14:08.236 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.236 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.236 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:14:08.236 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.103155+0000", 2024-04-20T17:14:08.236 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9300000000000001e-05 2024-04-20T17:14:08.236 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.236 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.236 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:14:08.236 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.137934+0000", 2024-04-20T17:14:08.236 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.034779139000000001 2024-04-20T17:14:08.236 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.236 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.236 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:14:08.237 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.137975+0000", 2024-04-20T17:14:08.237 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0439999999999999e-05 2024-04-20T17:14:08.237 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.237 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:14:08.237 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.237 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.237 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.237 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:196690 2.3 e166/150 2:cb6b9f79:::benchmark_data_smithi171_78918_object35696:head v 166'12075, mlcod=122'2273)", 2024-04-20T17:14:08.237 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:33.527490+0000", 2024-04-20T17:14:08.237 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 214.68634581699999, 2024-04-20T17:14:08.237 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043966988999999998, 2024-04-20T17:14:08.237 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:14:08.237 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:14:08.237 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:08.237 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:14:08.238 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.238 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:14:08.238 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.527490+0000", 2024-04-20T17:14:08.238 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.238 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.238 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.238 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:14:08.238 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.527490+0000", 2024-04-20T17:14:08.238 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.238 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.238 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.238 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:14:08.238 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.527491+0000", 2024-04-20T17:14:08.238 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4900000000000005e-07 2024-04-20T17:14:08.238 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.239 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.239 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:14:08.239 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.527496+0000", 2024-04-20T17:14:08.239 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1309999999999997e-06 2024-04-20T17:14:08.239 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.239 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.239 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:14:08.239 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.527503+0000", 2024-04-20T17:14:08.239 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9569999999999996e-06 2024-04-20T17:14:08.239 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.239 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.239 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:14:08.239 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.527508+0000", 2024-04-20T17:14:08.239 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.118e-06 2024-04-20T17:14:08.239 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.240 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.240 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:14:08.240 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.527920+0000", 2024-04-20T17:14:08.240 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00041207700000000001 2024-04-20T17:14:08.240 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.240 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.240 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:14:08.240 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.527944+0000", 2024-04-20T17:14:08.240 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3935e-05 2024-04-20T17:14:08.240 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.240 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.240 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:14:08.240 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.571413+0000", 2024-04-20T17:14:08.240 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043468967999999997 2024-04-20T17:14:08.240 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.241 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.241 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:14:08.241 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.571457+0000", 2024-04-20T17:14:08.241 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4054000000000002e-05 2024-04-20T17:14:08.241 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.241 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:14:08.241 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.241 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.241 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.241 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:197148 2.3 e166/150 2:cc8bb281:::benchmark_data_smithi171_78918_object36154:head v 166'12100, mlcod=122'2280)", 2024-04-20T17:14:08.241 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:33.965330+0000", 2024-04-20T17:14:08.241 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 214.24850636900001, 2024-04-20T17:14:08.241 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044780069999999998, 2024-04-20T17:14:08.241 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:14:08.241 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:14:08.242 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:08.242 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:14:08.242 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.242 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:14:08.242 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.965330+0000", 2024-04-20T17:14:08.242 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.242 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.242 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.242 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:14:08.242 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.965330+0000", 2024-04-20T17:14:08.242 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.242 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.242 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.242 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:14:08.242 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.965331+0000", 2024-04-20T17:14:08.243 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4640000000000001e-06 2024-04-20T17:14:08.243 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.243 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.243 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:14:08.243 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.965343+0000", 2024-04-20T17:14:08.243 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1589e-05 2024-04-20T17:14:08.243 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.243 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.243 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:14:08.243 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.965357+0000", 2024-04-20T17:14:08.243 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4555999999999999e-05 2024-04-20T17:14:08.243 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.243 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.243 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:14:08.244 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.965363+0000", 2024-04-20T17:14:08.244 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.648e-06 2024-04-20T17:14:08.244 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.244 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.244 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:14:08.244 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.965422+0000", 2024-04-20T17:14:08.244 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8631000000000002e-05 2024-04-20T17:14:08.244 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.244 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.244 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:14:08.244 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.965445+0000", 2024-04-20T17:14:08.244 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3706999999999999e-05 2024-04-20T17:14:08.244 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.244 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.244 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:14:08.245 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.010066+0000", 2024-04-20T17:14:08.245 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044620303 2024-04-20T17:14:08.245 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.245 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.245 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:14:08.245 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.010110+0000", 2024-04-20T17:14:08.245 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4172000000000003e-05 2024-04-20T17:14:08.245 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.245 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:14:08.245 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.245 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.245 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.245 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:197153 2.3 e166/150 2:cb452060:::benchmark_data_smithi171_78918_object36159:head v 166'12101, mlcod=122'2280)", 2024-04-20T17:14:08.245 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:33.968794+0000", 2024-04-20T17:14:08.245 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 214.245042576, 2024-04-20T17:14:08.246 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.041343306000000003, 2024-04-20T17:14:08.246 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:14:08.246 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:14:08.246 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:08.246 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:14:08.246 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.246 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:14:08.246 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.968794+0000", 2024-04-20T17:14:08.246 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.246 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.246 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.246 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:14:08.246 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.968794+0000", 2024-04-20T17:14:08.246 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.247 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.247 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.247 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:14:08.247 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.968800+0000", 2024-04-20T17:14:08.247 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6409999999999996e-06 2024-04-20T17:14:08.247 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.247 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.247 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:14:08.247 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.968817+0000", 2024-04-20T17:14:08.247 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6310000000000001e-05 2024-04-20T17:14:08.247 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.247 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.247 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:14:08.247 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.968833+0000", 2024-04-20T17:14:08.247 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6028999999999998e-05 2024-04-20T17:14:08.248 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.248 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.248 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:14:08.248 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.968844+0000", 2024-04-20T17:14:08.248 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1484999999999999e-05 2024-04-20T17:14:08.248 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.248 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.248 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:14:08.248 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.974323+0000", 2024-04-20T17:14:08.248 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0054788629999999996 2024-04-20T17:14:08.248 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.248 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.248 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:14:08.248 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.974345+0000", 2024-04-20T17:14:08.248 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1639000000000001e-05 2024-04-20T17:14:08.249 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.249 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.249 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:14:08.249 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.010113+0000", 2024-04-20T17:14:08.249 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.035768056999999999 2024-04-20T17:14:08.249 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.249 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.249 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:14:08.249 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.010137+0000", 2024-04-20T17:14:08.249 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4281999999999999e-05 2024-04-20T17:14:08.249 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.249 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:14:08.249 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.249 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.249 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.250 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:197280 2.3 e166/150 2:c3bfc80e:::benchmark_data_smithi171_78918_object36286:head v 166'12109, mlcod=123'2282)", 2024-04-20T17:14:08.250 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:34.080902+0000", 2024-04-20T17:14:08.250 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 214.13293376499999, 2024-04-20T17:14:08.250 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043969018999999998, 2024-04-20T17:14:08.250 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:14:08.250 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:14:08.250 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:08.250 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:14:08.250 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.250 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:14:08.250 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.080902+0000", 2024-04-20T17:14:08.250 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.250 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.250 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.251 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:14:08.251 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.080902+0000", 2024-04-20T17:14:08.251 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.251 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.251 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.251 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:14:08.251 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.080904+0000", 2024-04-20T17:14:08.251 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.096e-06 2024-04-20T17:14:08.251 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.251 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.251 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:14:08.251 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.080910+0000", 2024-04-20T17:14:08.251 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5239999999999997e-06 2024-04-20T17:14:08.251 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.252 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.252 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:14:08.252 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.080917+0000", 2024-04-20T17:14:08.252 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5409999999999997e-06 2024-04-20T17:14:08.252 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.252 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.252 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:14:08.252 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.080922+0000", 2024-04-20T17:14:08.252 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.989e-06 2024-04-20T17:14:08.252 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.252 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.252 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:14:08.252 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.088738+0000", 2024-04-20T17:14:08.252 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0078159309999999999 2024-04-20T17:14:08.252 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.252 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.253 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:14:08.253 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.088756+0000", 2024-04-20T17:14:08.253 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8604e-05 2024-04-20T17:14:08.253 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.253 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.253 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:14:08.253 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.124825+0000", 2024-04-20T17:14:08.253 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036068950000000002 2024-04-20T17:14:08.253 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.253 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.253 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:14:08.253 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.124872+0000", 2024-04-20T17:14:08.253 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6384000000000002e-05 2024-04-20T17:14:08.253 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.253 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:14:08.254 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.254 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.254 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.254 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:197284 2.3 e166/150 2:c6c18a95:::benchmark_data_smithi171_78918_object36290:head v 166'12110, mlcod=123'2282)", 2024-04-20T17:14:08.254 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:34.085104+0000", 2024-04-20T17:14:08.254 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 214.12873203000001, 2024-04-20T17:14:08.254 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.039799240999999999, 2024-04-20T17:14:08.254 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:14:08.254 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:14:08.254 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:08.254 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:14:08.254 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.254 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:14:08.254 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.085104+0000", 2024-04-20T17:14:08.254 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.255 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.255 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.255 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:14:08.255 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.085104+0000", 2024-04-20T17:14:08.255 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.255 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.255 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.255 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:14:08.255 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.085106+0000", 2024-04-20T17:14:08.255 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.657e-06 2024-04-20T17:14:08.255 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.255 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.255 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:14:08.255 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.085114+0000", 2024-04-20T17:14:08.256 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7559999999999999e-06 2024-04-20T17:14:08.256 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.256 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.256 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:14:08.256 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.085125+0000", 2024-04-20T17:14:08.256 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.119e-05 2024-04-20T17:14:08.256 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.256 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.256 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:14:08.256 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.085132+0000", 2024-04-20T17:14:08.256 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1339999999999998e-06 2024-04-20T17:14:08.256 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.256 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.256 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:14:08.256 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.098328+0000", 2024-04-20T17:14:08.257 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013196505000000001 2024-04-20T17:14:08.257 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.257 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.257 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:14:08.257 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.098352+0000", 2024-04-20T17:14:08.257 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3306e-05 2024-04-20T17:14:08.257 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.257 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.257 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:14:08.257 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.124874+0000", 2024-04-20T17:14:08.258 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026522343 2024-04-20T17:14:08.258 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.258 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.258 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:14:08.258 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.124903+0000", 2024-04-20T17:14:08.258 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9349999999999999e-05 2024-04-20T17:14:08.258 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.258 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:14:08.258 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.258 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.258 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.258 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:197285 2.3 e166/150 2:ce3b9f7d:::benchmark_data_smithi171_78918_object36291:head v 166'12111, mlcod=123'2282)", 2024-04-20T17:14:08.258 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:34.089569+0000", 2024-04-20T17:14:08.259 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 214.12426750899999, 2024-04-20T17:14:08.259 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.042953340999999999, 2024-04-20T17:14:08.259 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:14:08.259 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:14:08.259 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:08.259 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:14:08.259 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.259 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:14:08.259 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.089569+0000", 2024-04-20T17:14:08.259 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.259 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.259 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.259 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:14:08.259 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.089569+0000", 2024-04-20T17:14:08.259 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.260 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.260 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.260 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:14:08.260 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.089570+0000", 2024-04-20T17:14:08.260 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3740000000000001e-06 2024-04-20T17:14:08.260 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.260 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.260 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:14:08.260 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.089577+0000", 2024-04-20T17:14:08.260 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8700000000000003e-06 2024-04-20T17:14:08.260 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.260 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.260 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:14:08.260 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.089589+0000", 2024-04-20T17:14:08.260 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1571e-05 2024-04-20T17:14:08.261 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.261 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.261 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:14:08.261 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.089595+0000", 2024-04-20T17:14:08.261 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2079999999999997e-06 2024-04-20T17:14:08.261 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.261 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.261 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:14:08.261 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.107695+0000", 2024-04-20T17:14:08.261 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018100574000000001 2024-04-20T17:14:08.261 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.261 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.261 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:14:08.261 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.107723+0000", 2024-04-20T17:14:08.261 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7291999999999999e-05 2024-04-20T17:14:08.262 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.262 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.262 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:14:08.262 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.132481+0000", 2024-04-20T17:14:08.262 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024757923000000001 2024-04-20T17:14:08.262 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.262 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.262 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:14:08.262 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.132522+0000", 2024-04-20T17:14:08.262 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1529000000000002e-05 2024-04-20T17:14:08.262 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.262 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:14:08.262 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.262 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.263 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.263 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:197311 2.3 e166/150 2:c92347ee:::benchmark_data_smithi171_78918_object36317:head v 166'12113, mlcod=123'2282)", 2024-04-20T17:14:08.263 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:34.100868+0000", 2024-04-20T17:14:08.263 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 214.112968481, 2024-04-20T17:14:08.263 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.047080359000000002, 2024-04-20T17:14:08.263 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:14:08.263 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:14:08.263 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:08.263 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:14:08.263 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.263 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:14:08.263 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.100868+0000", 2024-04-20T17:14:08.263 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.263 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.263 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.264 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:14:08.264 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.100868+0000", 2024-04-20T17:14:08.264 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.264 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.264 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.264 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:14:08.264 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.100870+0000", 2024-04-20T17:14:08.264 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5359999999999999e-06 2024-04-20T17:14:08.264 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.264 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.264 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:14:08.264 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.100882+0000", 2024-04-20T17:14:08.264 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1579999999999999e-05 2024-04-20T17:14:08.264 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.264 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.265 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:14:08.265 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.100893+0000", 2024-04-20T17:14:08.265 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.118e-05 2024-04-20T17:14:08.265 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.265 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.265 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:14:08.265 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.100900+0000", 2024-04-20T17:14:08.265 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3989999999999998e-06 2024-04-20T17:14:08.265 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.265 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.265 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:14:08.265 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.124930+0000", 2024-04-20T17:14:08.265 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024029600000000002 2024-04-20T17:14:08.265 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.265 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.266 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:14:08.266 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.124952+0000", 2024-04-20T17:14:08.266 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2024000000000001e-05 2024-04-20T17:14:08.266 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.266 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.266 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:14:08.266 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.147909+0000", 2024-04-20T17:14:08.266 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022956826999999999 2024-04-20T17:14:08.266 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.266 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.266 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:14:08.266 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.147948+0000", 2024-04-20T17:14:08.266 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9212999999999999e-05 2024-04-20T17:14:08.266 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.266 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:14:08.267 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.267 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.267 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.267 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:197313 2.3 e166/150 2:ce42bf9c:::benchmark_data_smithi171_78918_object36319:head v 166'12114, mlcod=123'2282)", 2024-04-20T17:14:08.267 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:34.104580+0000", 2024-04-20T17:14:08.267 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 214.109255975, 2024-04-20T17:14:08.267 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043407387999999998, 2024-04-20T17:14:08.267 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:14:08.267 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:14:08.267 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:08.267 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:14:08.267 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.267 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:14:08.267 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.104580+0000", 2024-04-20T17:14:08.267 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.268 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.268 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.268 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:14:08.268 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.104580+0000", 2024-04-20T17:14:08.268 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.268 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.268 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.268 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:14:08.268 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.104581+0000", 2024-04-20T17:14:08.268 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.203e-06 2024-04-20T17:14:08.268 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.268 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.268 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:14:08.268 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.104588+0000", 2024-04-20T17:14:08.268 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.951e-06 2024-04-20T17:14:08.269 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.269 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.269 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:14:08.269 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.104597+0000", 2024-04-20T17:14:08.269 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6090000000000001e-06 2024-04-20T17:14:08.269 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.269 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.269 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:14:08.269 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.104605+0000", 2024-04-20T17:14:08.269 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9710000000000004e-06 2024-04-20T17:14:08.269 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.269 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.269 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:14:08.269 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.132598+0000", 2024-04-20T17:14:08.269 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.027992929999999999 2024-04-20T17:14:08.270 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.270 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.270 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:14:08.270 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.132623+0000", 2024-04-20T17:14:08.270 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.529e-05 2024-04-20T17:14:08.270 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.270 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.270 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:14:08.270 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.147957+0000", 2024-04-20T17:14:08.270 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015334129 2024-04-20T17:14:08.270 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.270 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.270 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:14:08.270 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.147988+0000", 2024-04-20T17:14:08.271 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0304999999999999e-05 2024-04-20T17:14:08.271 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.271 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:14:08.271 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.271 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.271 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.271 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:197988 2.3 e166/150 2:cafb7b2e:::benchmark_data_smithi171_78918_object36994:head v 166'12151, mlcod=123'2294)", 2024-04-20T17:14:08.271 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:34.765325+0000", 2024-04-20T17:14:08.271 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 213.448511564, 2024-04-20T17:14:08.271 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044294795999999997, 2024-04-20T17:14:08.271 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:14:08.271 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:14:08.271 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:08.272 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:14:08.272 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.272 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:14:08.272 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.765325+0000", 2024-04-20T17:14:08.272 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.272 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.272 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.272 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:14:08.272 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.765325+0000", 2024-04-20T17:14:08.272 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.272 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.272 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.272 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:14:08.272 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.765325+0000", 2024-04-20T17:14:08.272 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7100000000000001e-07 2024-04-20T17:14:08.273 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.273 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.273 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:14:08.273 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.765333+0000", 2024-04-20T17:14:08.273 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6040000000000004e-06 2024-04-20T17:14:08.273 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.273 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.273 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:14:08.273 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.765342+0000", 2024-04-20T17:14:08.273 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3600000000000002e-06 2024-04-20T17:14:08.273 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.273 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.273 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:14:08.274 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.765349+0000", 2024-04-20T17:14:08.274 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8349999999999998e-06 2024-04-20T17:14:08.274 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.274 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.274 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:14:08.274 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.775787+0000", 2024-04-20T17:14:08.274 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.01043725 2024-04-20T17:14:08.274 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.274 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.274 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:14:08.274 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.775802+0000", 2024-04-20T17:14:08.274 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5911000000000001e-05 2024-04-20T17:14:08.274 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.274 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.275 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:14:08.275 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.809585+0000", 2024-04-20T17:14:08.275 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.033782464999999998 2024-04-20T17:14:08.275 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.275 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.275 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:14:08.275 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.809619+0000", 2024-04-20T17:14:08.275 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4600000000000001e-05 2024-04-20T17:14:08.275 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.275 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:14:08.275 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.275 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.275 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.275 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:197989 2.3 e166/150 2:c00840d5:::benchmark_data_smithi171_78918_object36995:head v 166'12152, mlcod=123'2294)", 2024-04-20T17:14:08.275 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:34.769894+0000", 2024-04-20T17:14:08.276 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 213.443942577, 2024-04-20T17:14:08.276 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.039770080999999999, 2024-04-20T17:14:08.276 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:14:08.276 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:14:08.276 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:08.276 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:14:08.276 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.276 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:14:08.276 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.769894+0000", 2024-04-20T17:14:08.276 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.276 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.276 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.276 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:14:08.276 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.769894+0000", 2024-04-20T17:14:08.276 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.277 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.277 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.277 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:14:08.277 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.769895+0000", 2024-04-20T17:14:08.277 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7399999999999991e-07 2024-04-20T17:14:08.277 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.277 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.277 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:14:08.277 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.769901+0000", 2024-04-20T17:14:08.277 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3559999999999998e-06 2024-04-20T17:14:08.277 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.277 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.277 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:14:08.277 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.769909+0000", 2024-04-20T17:14:08.278 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6310000000000003e-06 2024-04-20T17:14:08.278 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.278 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.278 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:14:08.278 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.769916+0000", 2024-04-20T17:14:08.278 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5700000000000004e-06 2024-04-20T17:14:08.278 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.278 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.278 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:14:08.278 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.784392+0000", 2024-04-20T17:14:08.278 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014475449 2024-04-20T17:14:08.278 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.278 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.278 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:14:08.278 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.784419+0000", 2024-04-20T17:14:08.279 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7633000000000001e-05 2024-04-20T17:14:08.279 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.279 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.279 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:14:08.279 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.809623+0000", 2024-04-20T17:14:08.279 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.025203204999999999 2024-04-20T17:14:08.279 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.279 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.279 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:14:08.279 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.809664+0000", 2024-04-20T17:14:08.279 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1263000000000001e-05 2024-04-20T17:14:08.279 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.279 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:14:08.279 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.280 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.280 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.280 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:198153 2.3 e166/150 2:c93b993b:::benchmark_data_smithi171_78918_object37159:head v 166'12166, mlcod=123'2301)", 2024-04-20T17:14:08.280 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:35.037480+0000", 2024-04-20T17:14:08.280 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 213.17635586399999, 2024-04-20T17:14:08.280 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.041818424999999999, 2024-04-20T17:14:08.280 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:14:08.280 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:14:08.280 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:08.280 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:14:08.280 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.280 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:14:08.280 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.037480+0000", 2024-04-20T17:14:08.280 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.280 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.281 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.281 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:14:08.281 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.037480+0000", 2024-04-20T17:14:08.281 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.281 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.281 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.281 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:14:08.281 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.037482+0000", 2024-04-20T17:14:08.281 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.319e-06 2024-04-20T17:14:08.281 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.281 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.281 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:14:08.281 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.037488+0000", 2024-04-20T17:14:08.281 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8189999999999997e-06 2024-04-20T17:14:08.281 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.282 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.282 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:14:08.282 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.037496+0000", 2024-04-20T17:14:08.282 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.7329999999999992e-06 2024-04-20T17:14:08.282 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.282 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.282 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:14:08.282 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.037502+0000", 2024-04-20T17:14:08.282 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.818e-06 2024-04-20T17:14:08.282 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.282 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.282 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:14:08.282 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.048850+0000", 2024-04-20T17:14:08.282 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011348008999999999 2024-04-20T17:14:08.282 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.283 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.283 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:14:08.283 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.048868+0000", 2024-04-20T17:14:08.283 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8148e-05 2024-04-20T17:14:08.283 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.283 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.283 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:14:08.283 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.079255+0000", 2024-04-20T17:14:08.283 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030387038000000002 2024-04-20T17:14:08.283 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.283 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.283 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:14:08.283 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.079299+0000", 2024-04-20T17:14:08.283 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3541000000000002e-05 2024-04-20T17:14:08.284 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.284 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:14:08.284 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.284 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.284 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.284 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:200778 2.3 e166/150 2:ca9ecfbd:::benchmark_data_smithi171_78918_object39784:head v 166'12349, mlcod=124'2485)", 2024-04-20T17:14:08.284 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:38.271498+0000", 2024-04-20T17:14:08.284 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 209.94233798100001, 2024-04-20T17:14:08.284 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.040696903, 2024-04-20T17:14:08.284 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:14:08.284 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:14:08.284 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:08.284 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:14:08.284 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.284 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:14:08.285 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.271498+0000", 2024-04-20T17:14:08.285 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.285 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.285 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.285 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:14:08.285 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.271498+0000", 2024-04-20T17:14:08.285 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.285 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.285 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.285 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:14:08.285 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.271499+0000", 2024-04-20T17:14:08.285 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7599999999999996e-07 2024-04-20T17:14:08.285 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.285 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.285 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:14:08.286 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.271505+0000", 2024-04-20T17:14:08.286 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1859999999999997e-06 2024-04-20T17:14:08.286 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.286 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.286 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:14:08.286 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.271513+0000", 2024-04-20T17:14:08.286 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0269999999999995e-06 2024-04-20T17:14:08.286 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.286 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.286 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:14:08.286 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.271519+0000", 2024-04-20T17:14:08.286 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6060000000000002e-06 2024-04-20T17:14:08.286 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.286 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.286 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:14:08.287 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.279364+0000", 2024-04-20T17:14:08.287 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0078446799999999997 2024-04-20T17:14:08.287 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.287 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.287 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:14:08.287 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.279381+0000", 2024-04-20T17:14:08.287 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7142999999999998e-05 2024-04-20T17:14:08.287 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.287 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.287 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:14:08.287 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.312170+0000", 2024-04-20T17:14:08.287 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032789670999999999 2024-04-20T17:14:08.287 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.287 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.287 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:14:08.288 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.312195+0000", 2024-04-20T17:14:08.288 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4814e-05 2024-04-20T17:14:08.288 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.288 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:14:08.288 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.288 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.288 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.288 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:201134 2.3 e166/150 2:c429e5fa:::benchmark_data_smithi171_78918_object40140:head v 166'12367, mlcod=124'2502)", 2024-04-20T17:14:08.288 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:38.559710+0000", 2024-04-20T17:14:08.288 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 209.654126566, 2024-04-20T17:14:08.288 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.048219026999999998, 2024-04-20T17:14:08.288 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:14:08.288 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:14:08.288 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:08.288 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:14:08.289 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.289 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:14:08.289 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.559710+0000", 2024-04-20T17:14:08.289 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.289 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.289 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.289 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:14:08.289 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.559710+0000", 2024-04-20T17:14:08.289 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.289 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.289 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.289 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:14:08.289 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.559712+0000", 2024-04-20T17:14:08.289 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0679999999999999e-06 2024-04-20T17:14:08.289 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.290 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.290 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:14:08.290 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.559721+0000", 2024-04-20T17:14:08.290 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1379999999999996e-06 2024-04-20T17:14:08.290 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.290 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.290 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:14:08.290 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.559733+0000", 2024-04-20T17:14:08.290 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2199e-05 2024-04-20T17:14:08.290 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.290 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.290 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:14:08.290 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.559742+0000", 2024-04-20T17:14:08.290 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9740000000000008e-06 2024-04-20T17:14:08.290 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.291 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.291 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:14:08.291 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.575041+0000", 2024-04-20T17:14:08.291 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015299046 2024-04-20T17:14:08.291 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.291 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.291 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:14:08.291 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.575057+0000", 2024-04-20T17:14:08.291 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5900999999999999e-05 2024-04-20T17:14:08.291 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.291 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.291 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:14:08.291 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.607891+0000", 2024-04-20T17:14:08.291 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032834224000000002 2024-04-20T17:14:08.291 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.292 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.292 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:14:08.292 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.607929+0000", 2024-04-20T17:14:08.292 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7477e-05 2024-04-20T17:14:08.292 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.292 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:14:08.292 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.292 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.292 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.292 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:201680 2.3 e166/150 2:cf5ca5b2:::benchmark_data_smithi171_78918_object40686:head v 166'12411, mlcod=124'2544)", 2024-04-20T17:14:08.292 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:39.306758+0000", 2024-04-20T17:14:08.292 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 208.90707866299999, 2024-04-20T17:14:08.292 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.040304325000000002, 2024-04-20T17:14:08.292 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:14:08.292 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:14:08.292 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:08.293 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:14:08.293 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.293 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:14:08.293 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.306758+0000", 2024-04-20T17:14:08.293 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.293 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.293 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.293 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:14:08.293 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.306758+0000", 2024-04-20T17:14:08.293 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.293 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.293 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.293 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:14:08.293 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.306759+0000", 2024-04-20T17:14:08.293 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.141e-06 2024-04-20T17:14:08.294 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.294 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.294 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:14:08.294 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.306765+0000", 2024-04-20T17:14:08.294 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8889999999999999e-06 2024-04-20T17:14:08.294 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.294 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.294 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:14:08.294 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.306772+0000", 2024-04-20T17:14:08.294 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2599999999999999e-06 2024-04-20T17:14:08.294 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.294 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.294 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:14:08.294 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.306778+0000", 2024-04-20T17:14:08.294 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1890000000000004e-06 2024-04-20T17:14:08.295 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.295 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.295 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:14:08.295 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.306829+0000", 2024-04-20T17:14:08.295 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0553999999999997e-05 2024-04-20T17:14:08.295 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.295 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.295 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:14:08.295 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.306853+0000", 2024-04-20T17:14:08.295 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4615999999999999e-05 2024-04-20T17:14:08.295 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.295 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.296 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:14:08.296 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.347027+0000", 2024-04-20T17:14:08.296 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.040173826000000003 2024-04-20T17:14:08.296 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.296 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.296 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:14:08.296 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.347062+0000", 2024-04-20T17:14:08.296 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.485e-05 2024-04-20T17:14:08.296 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.296 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:14:08.296 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.296 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.296 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.297 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:203113 2.3 e166/150 2:c71a3208:::benchmark_data_smithi171_78918_object42119:head v 166'12491, mlcod=124'2628)", 2024-04-20T17:14:08.297 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:40.855113+0000", 2024-04-20T17:14:08.297 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 207.358723575, 2024-04-20T17:14:08.297 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044317189999999999, 2024-04-20T17:14:08.297 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:14:08.297 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:14:08.297 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:08.297 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:14:08.297 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.297 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:14:08.297 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.855113+0000", 2024-04-20T17:14:08.297 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.297 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.297 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.297 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:14:08.298 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.855113+0000", 2024-04-20T17:14:08.298 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:14:08.298 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.298 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.298 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:14:08.298 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.855114+0000", 2024-04-20T17:14:08.298 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.649e-06 2024-04-20T17:14:08.298 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.298 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.298 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:14:08.298 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.855121+0000", 2024-04-20T17:14:08.298 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.985e-06 2024-04-20T17:14:08.298 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.298 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.299 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:14:08.299 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.855131+0000", 2024-04-20T17:14:08.299 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7890000000000006e-06 2024-04-20T17:14:08.299 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.299 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.299 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:14:08.299 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.855139+0000", 2024-04-20T17:14:08.299 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8949999999999998e-06 2024-04-20T17:14:08.299 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.299 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.299 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:14:08.299 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.865644+0000", 2024-04-20T17:14:08.299 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010505393999999999 2024-04-20T17:14:08.299 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.299 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.299 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:14:08.300 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.865672+0000", 2024-04-20T17:14:08.300 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8022e-05 2024-04-20T17:14:08.300 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.300 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.300 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:14:08.300 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.899392+0000", 2024-04-20T17:14:08.300 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.033719879000000001 2024-04-20T17:14:08.300 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:14:08.300 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:14:08.300 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:14:08.300 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.899430+0000", 2024-04-20T17:14:08.300 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7577000000000003e-05 2024-04-20T17:14:08.300 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.300 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:14:08.300 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.301 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:14:08.301 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:14:08.301 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:14:08.301 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.301 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.301 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.302 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.302 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.302 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.302 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.302 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.302 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.302 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.302 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:08.933 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-20T17:14:09.098 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:14:09.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:09.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:14:09.202 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:14:09.204 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:14:09.229 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:09 smithi171 ceph-mon[44189]: pgmap v673: 17 pgs: 1 active+recovering, 16 active+clean; 5.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 137 op/s; 10051/177806 objects misplaced (5.653%); 332 KiB/s, 5 objects/s recovering 2024-04-20T17:14:09.229 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:09 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/337010565' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:14:09.230 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:09 smithi171 ceph-mon[44189]: from='client.25178 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:14:09.231 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:09.243 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.11 to 1 (10000) 2024-04-20T17:14:09.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:09 smithi005 ceph-mon[39562]: pgmap v673: 17 pgs: 1 active+recovering, 16 active+clean; 5.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 137 op/s; 10051/177806 objects misplaced (5.653%); 332 KiB/s, 5 objects/s recovering 2024-04-20T17:14:09.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:09 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/337010565' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:14:09.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:09 smithi005 ceph-mon[39562]: from='client.25178 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:14:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:09 smithi106 ceph-mon[44423]: pgmap v673: 17 pgs: 1 active+recovering, 16 active+clean; 5.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 137 op/s; 10051/177806 objects misplaced (5.653%); 332 KiB/s, 5 objects/s recovering 2024-04-20T17:14:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:09 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/337010565' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:14:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:09 smithi106 ceph-mon[44423]: from='client.25178 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:14:09.581 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:14:09.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:09.584 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:14:09.685 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:14:09.712 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:09.731 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:14:09.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:09.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:14:09.834 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:14:09.867 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:10.116 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:14:10.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:10.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:14:10.149 INFO:tasks.thrashosds.thrasher:test_map_discontinuity 2024-04-20T17:14:10.149 INFO:tasks.thrashosds.thrasher:Waiting for recovery 2024-04-20T17:14:10.149 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-20T17:14:10.149 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-20T17:14:10.222 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:14:10.262 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:10.421 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:14:10.424 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:10 smithi005 ceph-mon[39562]: from='client.25178 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-20T17:14:10.424 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:10 smithi005 ceph-mon[39562]: osdmap e243: 12 total, 12 up, 12 in 2024-04-20T17:14:10.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:10.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:14:10.476 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:10 smithi171 ceph-mon[44189]: from='client.25178 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-20T17:14:10.477 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:10 smithi171 ceph-mon[44189]: osdmap e243: 12 total, 12 up, 12 in 2024-04-20T17:14:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:10 smithi106 ceph-mon[44423]: from='client.25178 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-20T17:14:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:10 smithi106 ceph-mon[44423]: osdmap e243: 12 total, 12 up, 12 in 2024-04-20T17:14:10.526 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:14:10.556 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:10.755 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:14:10.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:10.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:14:10.860 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:14:10.865 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:14:10.887 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:10.905 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:14:10.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:10.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:14:11.009 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:14:11.037 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:11.056 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:14:11.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:11.059 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:14:11.060 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:14:11.159 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:14:11.189 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:11.191 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:14:11.192 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:14:11.192 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 0 2024-04-20T17:14:11.192 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:14:11.214 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:14:11.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:11.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:14:11.319 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:14:11.350 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:11.900 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:14:11.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:11.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:14:11.918 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:14:12.004 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:14:12.030 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:12.048 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:14:12.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:12.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:14:12.152 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:14:12.170 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:11 smithi171 ceph-mon[44189]: pgmap v675: 17 pgs: 1 active+recovering, 16 active+clean; 5.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 131 op/s; 9971/177488 objects misplaced (5.618%); 629 KiB/s, 9 objects/s recovering 2024-04-20T17:14:12.170 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:11 smithi171 ceph-mon[44189]: osdmap e244: 12 total, 12 up, 12 in 2024-04-20T17:14:12.170 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:11 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3775718099' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:14:12.191 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:12.206 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-20T17:14:12.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:11 smithi005 ceph-mon[39562]: pgmap v675: 17 pgs: 1 active+recovering, 16 active+clean; 5.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 131 op/s; 9971/177488 objects misplaced (5.618%); 629 KiB/s, 9 objects/s recovering 2024-04-20T17:14:12.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:11 smithi005 ceph-mon[39562]: osdmap e244: 12 total, 12 up, 12 in 2024-04-20T17:14:12.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:11 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3775718099' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:14:12.215 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:14:12.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:12.223 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:14:12.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:11 smithi106 ceph-mon[44423]: pgmap v675: 17 pgs: 1 active+recovering, 16 active+clean; 5.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 131 op/s; 9971/177488 objects misplaced (5.618%); 629 KiB/s, 9 objects/s recovering 2024-04-20T17:14:12.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:11 smithi106 ceph-mon[44423]: osdmap e244: 12 total, 12 up, 12 in 2024-04-20T17:14:12.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:11 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3775718099' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:14:12.324 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:14:12.355 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:12.522 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:14:13.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:12 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:14:13.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:12 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:14:13.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:12 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:14:13.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:12 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:14:13.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:12 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:14:13.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:12 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:14:13.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:12 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3775718099' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:14:13.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:12 smithi005 ceph-mon[39562]: osdmap e245: 12 total, 12 up, 12 in 2024-04-20T17:14:13.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:12 smithi005 ceph-mon[39562]: 2.d scrub starts 2024-04-20T17:14:13.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:12 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:14:13.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:12 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:14:13.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:12 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:14:13.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:12 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:14:13.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:12 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:14:13.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:12 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:14:13.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:12 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3775718099' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:14:13.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:12 smithi106 ceph-mon[44423]: osdmap e245: 12 total, 12 up, 12 in 2024-04-20T17:14:13.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:12 smithi106 ceph-mon[44423]: 2.d scrub starts 2024-04-20T17:14:13.309 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:14:13.309 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:14:13.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:13.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:14:13.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:12 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:14:13.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:12 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:14:13.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:12 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:14:13.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:12 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:14:13.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:12 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:14:13.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:12 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:14:13.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:12 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3775718099' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:14:13.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:12 smithi171 ceph-mon[44189]: osdmap e245: 12 total, 12 up, 12 in 2024-04-20T17:14:13.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:12 smithi171 ceph-mon[44189]: 2.d scrub starts 2024-04-20T17:14:13.416 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:14:13.443 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:13.461 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:14:13.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:13.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:14:13.564 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:14:13.592 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:14.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:13 smithi005 ceph-mon[39562]: pgmap v678: 17 pgs: 1 active+recovering, 16 active+clean; 5.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 103 op/s; 9971/177044 objects misplaced (5.632%); 437 KiB/s, 6 objects/s recovering 2024-04-20T17:14:14.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:13 smithi005 ceph-mon[39562]: 2.d scrub ok 2024-04-20T17:14:14.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:13 smithi106 ceph-mon[44423]: pgmap v678: 17 pgs: 1 active+recovering, 16 active+clean; 5.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 103 op/s; 9971/177044 objects misplaced (5.632%); 437 KiB/s, 6 objects/s recovering 2024-04-20T17:14:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:13 smithi106 ceph-mon[44423]: 2.d scrub ok 2024-04-20T17:14:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:13 smithi171 ceph-mon[44189]: pgmap v678: 17 pgs: 1 active+recovering, 16 active+clean; 5.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 103 op/s; 9971/177044 objects misplaced (5.632%); 437 KiB/s, 6 objects/s recovering 2024-04-20T17:14:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:13 smithi171 ceph-mon[44189]: 2.d scrub ok 2024-04-20T17:14:14.335 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:14:14.335 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":245,"fsid":"13c6e76c-ff37-11ee-bc93-c7b262605968","created":"2024-04-20T16:58:25.272557+0000","modified":"2024-04-20T17:14:11.910698+0000","last_up_change":"2024-04-20T17:12:21.882767+0000","last_in_change":"2024-04-20T17:05:42.849446+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T17:01:35.909493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2024-04-20T17:08:04.904546+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":171,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6801","nonce":3901484705}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6802","nonce":3901484705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6804","nonce":3901484705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6803","nonce":3901484705}]},"public_addr":"172.21.15.5:6801/3901484705","cluster_addr":"172.21.15.5:6802/3901484705","heartbeat_back_addr":"172.21.15.5:6804/3901484705","heartbeat_front_addr":"172.21.15.5:6803/3901484705","state":["exists","up"]},{"osd":1,"uuid":"aed0c8eb-0e81-4e85-ab8f-a60420060656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6805","nonce":3391893914}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6806","nonce":3391893914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6808","nonce":3391893914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6807","nonce":3391893914}]},"public_addr":"172.21.15.5:6805/3391893914","cluster_addr":"172.21.15.5:6806/3391893914","heartbeat_back_addr":"172.21.15.5:6808/3391893914","heartbeat_front_addr":"172.21.15.5:6807/3391893914","state":["exists","up"]},{"osd":2,"uuid":"f733a6a2-19c2-4c7a-af9c-04e282062bb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":20,"last_clean_end":169,"up_from":202,"up_thru":202,"down_at":170,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6809","nonce":2120275501}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6810","nonce":2120275501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6812","nonce":2120275501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6811","nonce":2120275501}]},"public_addr":"172.21.15.5:6809/2120275501","cluster_addr":"172.21.15.5:6810/2120275501","heartbeat_back_addr":"172.21.15.5:6812/2120275501","heartbeat_front_addr":"172.21.15.5:6811/2120275501","state":["exists","up"]},{"osd":3,"uuid":"8c92f1df-e298-4606-88ae-b8a8e79628a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6813","nonce":377652039}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6814","nonce":377652039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6816","nonce":377652039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6815","nonce":377652039}]},"public_addr":"172.21.15.5:6813/377652039","cluster_addr":"172.21.15.5:6814/377652039","heartbeat_back_addr":"172.21.15.5:6816/377652039","heartbeat_front_addr":"172.21.15.5:6815/377652039","state":["exists","up"]},{"osd":4,"uuid":"46df16f1-9607-4d7b-9be8-9d4236e160d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6800","nonce":1839524469}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6801","nonce":1839524469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6803","nonce":1839524469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6802","nonce":1839524469}]},"public_addr":"172.21.15.106:6800/1839524469","cluster_addr":"172.21.15.106:6801/1839524469","heartbeat_back_addr":"172.21.15.106:6803/1839524469","heartbeat_front_addr":"172.21.15.106:6802/1839524469","state":["exists","up"]},{"osd":5,"uuid":"6603231d-1e5c-4575-bfaa-2eeca2c12215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6804","nonce":2450315908}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6805","nonce":2450315908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6807","nonce":2450315908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6806","nonce":2450315908}]},"public_addr":"172.21.15.106:6804/2450315908","cluster_addr":"172.21.15.106:6805/2450315908","heartbeat_back_addr":"172.21.15.106:6807/2450315908","heartbeat_front_addr":"172.21.15.106:6806/2450315908","state":["exists","up"]},{"osd":6,"uuid":"783a55a6-f176-42be-8c8b-af8481d08dc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6808","nonce":2840237913}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6809","nonce":2840237913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6811","nonce":2840237913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6810","nonce":2840237913}]},"public_addr":"172.21.15.106:6808/2840237913","cluster_addr":"172.21.15.106:6809/2840237913","heartbeat_back_addr":"172.21.15.106:6811/2840237913","heartbeat_front_addr":"172.21.15.106:6810/2840237913","state":["exists","up"]},{"osd":7,"uuid":"b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6812","nonce":1105371883}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6813","nonce":1105371883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6815","nonce":1105371883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6814","nonce":1105371883}]},"public_addr":"172.21.15.106:6812/1105371883","cluster_addr":"172.21.15.106:6813/1105371883","heartbeat_back_addr":"172.21.15.106:6815/1105371883","heartbeat_front_addr":"172.21.15.106:6814/1105371883","state":["exists","up"]},{"osd":8,"uuid":"1cb232ad-1f59-4a0a-a215-e67419ad6476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6800","nonce":4123058942}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6801","nonce":4123058942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6803","nonce":4123058942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6802","nonce":4123058942}]},"public_addr":"172.21.15.171:6800/4123058942","cluster_addr":"172.21.15.171:6801/4123058942","heartbeat_back_addr":"172.21.15.171:6803/4123058942","heartbeat_front_addr":"172.21.15.171:6802/4123058942","state":["exists","up"]},{"osd":9,"uuid":"b6c17147-b3dc-475b-bb20-f9f712fb99e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6804","nonce":2877589636}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6805","nonce":2877589636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6807","nonce":2877589636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6806","nonce":2877589636}]},"public_addr":"172.21.15.171:6804/2877589636","cluster_addr":"172.21.15.171:6805/2877589636","heartbeat_back_addr":"172.21.15.171:6807/2877589636","heartbeat_front_addr":"172.21.15.171:6806/2877589636","state":["exists","up"]},{"osd":10,"uuid":"1adcdf55-f002-4a39-91e7-378e5fae849c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6808","nonce":181490621}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6809","nonce":181490621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6811","nonce":181490621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6810","nonce":181490621}]},"public_addr":"172.21.15.171:6808/181490621","cluster_addr":"172.21.15.171:6809/181490621","heartbeat_back_addr":"172.21.15.171:6811/181490621","heartbeat_front_addr":"172.21.15.171:6810/181490621","state":["exists","up"]},{"osd":11,"uuid":"737c5aec-694a-4968-819d-fccf22661f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":92,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6812","nonce":3373585701}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6813","nonce":3373585701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6815","nonce":3373585701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6814","nonce":3373585701}]},"public_addr":"172.21.15.171:6812/3373585701","cluster_addr":"172.21.15.171:6813/3373585701","heartbeat_back_addr":"172.21.15.171:6815/3373585701","heartbeat_front_addr":"172.21.15.171:6814/3373585701","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:13:40.031008+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:13:22.508752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:13:46.225864+0000","dead_epoch":169},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:13:57.358022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:13:23.862439+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:12:52.835307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:13:36.071047+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:14:11.377508+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:13:53.822088+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:14:11.660363+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:13:11.061560+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:13:52.662244+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.1","osds":[9,6]},{"pgid":"2.9","osds":[7,10]}],"pg_upmap_items":[{"pgid":"2.3","mappings":[{"from":2,"to":10},{"from":5,"to":0}]},{"pgid":"2.4","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.5:0/3881726425":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/338649532":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/1227843664":"2024-04-21T16:58:55.392162+0000","172.21.15.5:6800/1876064631":"2024-04-21T16:58:55.392162+0000","172.21.15.5:0/160689449":"2024-04-21T16:58:55.392162+0000","172.21.15.5:6800/3521715286":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/3841663462":"2024-04-21T16:58:55.392162+0000","172.21.15.5:0/687690919":"2024-04-21T16:59:25.808337+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T17:14:15.043 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:14:15.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:15.046 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:14:15.147 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:14:15.171 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:14 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/932388452' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:14:15.172 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:15.191 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:14:15.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:15.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:14:15.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:14 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/932388452' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:14:15.285 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:14 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/932388452' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:14:15.294 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:14:15.320 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:15.339 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:14:15.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:15.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:14:15.363 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:14:15.422 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-20T17:14:15.442 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:14:15.469 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:15.487 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:14:15.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:15.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:14:15.590 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:14:15.619 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:15.744 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:14:15.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:15.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:14:15.848 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:14:15.874 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:15.893 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:14:15.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:15.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:14:15.996 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:14:16.023 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:16.041 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:14:16.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:16.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:14:16.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:15 smithi171 ceph-mon[44189]: pgmap v679: 17 pgs: 1 active+recovering, 16 active+clean; 5.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 127 op/s; 9971/176272 objects misplaced (5.657%); 437 KiB/s, 6 objects/s recovering 2024-04-20T17:14:16.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:15 smithi171 ceph-mon[44189]: osdmap e246: 12 total, 12 up, 12 in 2024-04-20T17:14:16.144 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:14:16.173 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:16.192 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:14:16.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:16.195 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:14:16.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:15 smithi005 ceph-mon[39562]: pgmap v679: 17 pgs: 1 active+recovering, 16 active+clean; 5.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 127 op/s; 9971/176272 objects misplaced (5.657%); 437 KiB/s, 6 objects/s recovering 2024-04-20T17:14:16.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:15 smithi005 ceph-mon[39562]: osdmap e246: 12 total, 12 up, 12 in 2024-04-20T17:14:16.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:15 smithi106 ceph-mon[44423]: pgmap v679: 17 pgs: 1 active+recovering, 16 active+clean; 5.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 127 op/s; 9971/176272 objects misplaced (5.657%); 437 KiB/s, 6 objects/s recovering 2024-04-20T17:14:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:15 smithi106 ceph-mon[44423]: osdmap e246: 12 total, 12 up, 12 in 2024-04-20T17:14:16.296 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:14:16.322 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:16.340 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:14:16.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:16.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:14:16.397 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:14:16.444 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:14:16.470 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:16.488 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:14:16.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:16.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:14:16.591 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:14:16.621 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:16.641 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:14:16.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:16.644 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:14:16.745 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:14:16.776 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:16.778 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:14:16.778 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:14:16.778 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-04-20T17:14:16.778 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-04-20T17:14:16.778 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:14:16.903 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:14:16.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:16.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:14:16.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:16 smithi005 ceph-mon[39562]: pgmap v681: 17 pgs: 1 active+recovering, 16 active+clean; 5.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 148 op/s; 9893/175700 objects misplaced (5.631%); 405 KiB/s, 6 objects/s recovering 2024-04-20T17:14:17.008 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:14:17.036 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:17.060 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:14:17.061 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:16 smithi171 ceph-mon[44189]: pgmap v681: 17 pgs: 1 active+recovering, 16 active+clean; 5.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 148 op/s; 9893/175700 objects misplaced (5.631%); 405 KiB/s, 6 objects/s recovering 2024-04-20T17:14:17.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:17.063 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:14:17.164 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:14:17.197 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:17.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:16 smithi106 ceph-mon[44423]: pgmap v681: 17 pgs: 1 active+recovering, 16 active+clean; 5.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 148 op/s; 9893/175700 objects misplaced (5.631%); 405 KiB/s, 6 objects/s recovering 2024-04-20T17:14:17.447 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:14:17.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:17.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:14:17.550 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:14:17.579 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:17.594 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-20T17:14:17.609 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:14:17.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:17.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:14:17.730 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:14:17.758 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:17.864 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:14:18.538 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:14:18.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:18.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:14:18.642 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:14:18.674 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:18.698 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:14:18.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:18.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:14:18.807 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:14:18.837 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:18.981 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:14:18.981 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:14:18.981 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:14:18.981 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:14:18.982 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:14:18.982 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:14:18.982 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:14:18.982 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:14:18.982 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:14:18.982 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:14:18.982 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:14:18.982 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:14:18.982 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:14:18.983 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:14:18.983 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:14:18.983 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:14:18.983 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:14:18.983 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:14:18.983 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:14:18.983 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:14:18.983 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:14:18.983 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:14:18.983 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:14:18.983 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:14:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:18 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:14:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:18 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:14:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:18 smithi171 ceph-mon[44189]: osdmap e247: 12 total, 12 up, 12 in 2024-04-20T17:14:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:18 smithi171 ceph-mon[44189]: pgmap v683: 17 pgs: 1 active+recovering, 16 active+clean; 5.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 176 op/s; 9893/174982 objects misplaced (5.654%); 398 KiB/s, 6 objects/s recovering 2024-04-20T17:14:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:18 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/575646374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:14:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:19.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:19.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:19.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:19.186 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:14:19.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:19.189 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:14:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:18 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:14:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:18 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:14:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:18 smithi005 ceph-mon[39562]: osdmap e247: 12 total, 12 up, 12 in 2024-04-20T17:14:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:18 smithi005 ceph-mon[39562]: pgmap v683: 17 pgs: 1 active+recovering, 16 active+clean; 5.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 176 op/s; 9893/174982 objects misplaced (5.654%); 398 KiB/s, 6 objects/s recovering 2024-04-20T17:14:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:18 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/575646374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:14:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:19.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:19.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:19.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:19.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:18 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:14:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:18 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:14:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:18 smithi106 ceph-mon[44423]: osdmap e247: 12 total, 12 up, 12 in 2024-04-20T17:14:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:18 smithi106 ceph-mon[44423]: pgmap v683: 17 pgs: 1 active+recovering, 16 active+clean; 5.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 176 op/s; 9893/174982 objects misplaced (5.654%); 398 KiB/s, 6 objects/s recovering 2024-04-20T17:14:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:18 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/575646374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:14:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:19.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:19.290 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:14:19.316 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:19.334 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:14:19.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:19.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:14:19.438 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:14:19.464 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:19.483 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:14:19.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:19.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:14:19.587 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:14:19.616 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:19.636 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:14:19.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:19.639 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:14:19.740 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:14:19.768 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:19.773 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:14:19.773 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:14:19.792 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:14:19.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:19.795 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:14:19.896 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:14:19.924 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:19.936 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:14:20.073 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:14:20.074 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:20.074 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:20.074 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:20.074 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:20.074 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:20.074 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:19 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1284356304' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:14:20.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:20.075 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:14:20.176 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:14:20.201 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:20.201 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:20.202 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:20.202 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:20.202 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:20.202 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:19 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1284356304' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:14:20.203 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:19 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1284356304' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:14:20.222 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:14:20.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:20.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:14:20.326 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:14:20.352 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:20.370 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:14:20.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:20.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:14:20.474 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:14:20.501 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:20.518 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:14:20.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:20.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:14:20.593 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-20T17:14:20.622 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:14:20.650 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:20.668 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:14:20.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:20.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:14:20.772 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:14:20.802 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:20.825 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:14:20.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:20.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:14:20.901 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:14:20.931 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:14:20.956 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:20.974 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:14:20.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:20.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:14:21.078 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:14:21.104 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:21.123 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:14:21.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:21.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:14:21.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:20 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:14:21.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:20 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1284356304' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:14:21.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:20 smithi005 ceph-mon[39562]: osdmap e248: 12 total, 12 up, 12 in 2024-04-20T17:14:21.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:20 smithi005 ceph-mon[39562]: pgmap v685: 17 pgs: 1 active+recovering, 16 active+clean; 5.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 137 op/s; 9814/174616 objects misplaced (5.620%); 821 KiB/s, 12 objects/s recovering 2024-04-20T17:14:21.226 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:14:21.258 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:21.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:20 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:14:21.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:20 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1284356304' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:14:21.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:20 smithi106 ceph-mon[44423]: osdmap e248: 12 total, 12 up, 12 in 2024-04-20T17:14:21.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:20 smithi106 ceph-mon[44423]: pgmap v685: 17 pgs: 1 active+recovering, 16 active+clean; 5.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 137 op/s; 9814/174616 objects misplaced (5.620%); 821 KiB/s, 12 objects/s recovering 2024-04-20T17:14:21.279 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:14:21.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:21.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:14:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:20 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:14:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:20 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1284356304' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:14:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:20 smithi171 ceph-mon[44189]: osdmap e248: 12 total, 12 up, 12 in 2024-04-20T17:14:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:20 smithi171 ceph-mon[44189]: pgmap v685: 17 pgs: 1 active+recovering, 16 active+clean; 5.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 137 op/s; 9814/174616 objects misplaced (5.620%); 821 KiB/s, 12 objects/s recovering 2024-04-20T17:14:21.383 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:14:21.409 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:21.427 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:14:21.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:21.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:14:21.531 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:14:21.560 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:21.809 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:14:21.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:21.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:14:21.913 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:14:21.941 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:21.960 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:14:21.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:21.963 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:14:21.968 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:14:22.064 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:14:22.091 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:22.112 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:14:22.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:22.117 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:14:22.218 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:14:22.245 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:22.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:22.498 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:14:22.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:14:22.599 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:14:22.625 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:22.645 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:14:22.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:22.648 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:14:22.749 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:14:22.776 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:22.794 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:14:22.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:22.797 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:14:22.844 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:14:22.898 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:14:22.932 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:23.035 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:14:23.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:23.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:14:23.078 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:23 smithi005 ceph-mon[39562]: pgmap v686: 17 pgs: 1 active+recovering, 16 active+clean; 5.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 122 op/s; 9814/174200 objects misplaced (5.634%); 408 KiB/s, 6 objects/s recovering 2024-04-20T17:14:23.139 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:14:23.169 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:23.199 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:14:23.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:23.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:14:23.225 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:14:23.226 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:14:23.230 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:14:23.248 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:14:23.304 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:14:23.329 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:23 smithi171 ceph-mon[44189]: pgmap v686: 17 pgs: 1 active+recovering, 16 active+clean; 5.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 122 op/s; 9814/174200 objects misplaced (5.634%); 408 KiB/s, 6 objects/s recovering 2024-04-20T17:14:23.331 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:23.350 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:14:23.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:23.353 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:14:23.454 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:14:23.479 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:23 smithi106 ceph-mon[44423]: pgmap v686: 17 pgs: 1 active+recovering, 16 active+clean; 5.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 122 op/s; 9814/174200 objects misplaced (5.634%); 408 KiB/s, 6 objects/s recovering 2024-04-20T17:14:23.481 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:23.500 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:14:23.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:23.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:14:23.604 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:14:23.631 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:23.648 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:14:23.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:23.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:14:23.752 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:14:23.777 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:23.796 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:14:23.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:23.799 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:14:23.900 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:14:23.926 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:23.944 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:14:23.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:23.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:14:24.048 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:14:24.075 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:24.093 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:14:24.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:24.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:14:24.197 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:14:24.225 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:24.244 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:14:24.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:24.247 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:14:24.348 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:14:24.374 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:24.393 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:14:24.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:24.396 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:14:24.497 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:14:24.523 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:24.541 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:14:24.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:24.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:14:24.646 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:14:24.676 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:25.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:25 smithi171 ceph-mon[44189]: pgmap v687: 17 pgs: 1 active+recovering, 16 active+clean; 5.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 129 op/s; 9814/173626 objects misplaced (5.652%); 312 KiB/s, 4 objects/s recovering 2024-04-20T17:14:25.388 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:25 smithi005 ceph-mon[39562]: pgmap v687: 17 pgs: 1 active+recovering, 16 active+clean; 5.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 129 op/s; 9814/173626 objects misplaced (5.652%); 312 KiB/s, 4 objects/s recovering 2024-04-20T17:14:25.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:25 smithi106 ceph-mon[44423]: pgmap v687: 17 pgs: 1 active+recovering, 16 active+clean; 5.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 129 op/s; 9814/173626 objects misplaced (5.652%); 312 KiB/s, 4 objects/s recovering 2024-04-20T17:14:26.856 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:14:26.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:26.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:14:26.875 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-20T17:14:26.960 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:14:26.986 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:27.006 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:14:27.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:27.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:14:27.109 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:14:27.136 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:27.156 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:14:27.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:27.160 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:14:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:26 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:14:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:26 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:14:27.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:26 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:14:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:26 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:14:27.261 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:14:27.287 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:27.306 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:14:27.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:27.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:14:27.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:26 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:14:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:26 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:14:27.409 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:14:27.436 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:27.455 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:14:27.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:27.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:14:27.559 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:14:27.592 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:28.029 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:14:28.053 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:14:28.089 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:14:28.109 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:14:28.130 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:14:28.150 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:14:28.171 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:14:28.192 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:14:28.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:27 smithi005 ceph-mon[39562]: pgmap v688: 17 pgs: 1 active+recovering, 16 active+clean; 5.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 107 op/s; 9736/173236 objects misplaced (5.620%); 616 KiB/s, 9 objects/s recovering 2024-04-20T17:14:28.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:27 smithi005 ceph-mon[39562]: osdmap e249: 12 total, 12 up, 12 in 2024-04-20T17:14:28.213 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:14:28.245 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:14:28.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:27 smithi106 ceph-mon[44423]: pgmap v688: 17 pgs: 1 active+recovering, 16 active+clean; 5.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 107 op/s; 9736/173236 objects misplaced (5.620%); 616 KiB/s, 9 objects/s recovering 2024-04-20T17:14:28.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:27 smithi106 ceph-mon[44423]: osdmap e249: 12 total, 12 up, 12 in 2024-04-20T17:14:28.265 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:14:28.286 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:14:28.288 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:14:28.288 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:14:28.288 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:14:28.288 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:14:28.288 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:14:28.288 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:14:28.288 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:14:28.288 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:14:28.288 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:14:28.288 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:14:28.289 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:14:28.289 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:14:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:27 smithi171 ceph-mon[44189]: pgmap v688: 17 pgs: 1 active+recovering, 16 active+clean; 5.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 107 op/s; 9736/173236 objects misplaced (5.620%); 616 KiB/s, 9 objects/s recovering 2024-04-20T17:14:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:27 smithi171 ceph-mon[44189]: osdmap e249: 12 total, 12 up, 12 in 2024-04-20T17:14:28.343 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:14:28.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:28.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:14:28.448 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:14:28.474 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:28.492 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:14:28.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:28.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:14:28.596 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:14:28.622 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:28.637 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:14:28.641 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:14:28.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:28.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:14:28.744 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:14:28.771 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:28.790 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:14:28.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:28.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:14:28.893 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:14:28.927 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:28.927 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:14:28.954 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:14:28.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:28.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:14:29.058 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:14:29.090 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:29.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:28 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3726402768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:14:29.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:28 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3955759971' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:14:29.216 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:14:29.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:29.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:14:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:28 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3726402768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:14:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:28 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3955759971' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:14:29.320 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:14:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:28 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3726402768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:14:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:28 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3955759971' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:14:29.346 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:29.365 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:14:29.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:29.368 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:14:29.469 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:14:29.495 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:29.513 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:14:29.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:29.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:14:29.617 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:14:29.643 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:29.661 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:14:29.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:29.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:14:29.765 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:14:29.792 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:29.812 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:14:29.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:29.814 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:14:29.915 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:14:29.942 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:29.961 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:14:29.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:29.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:14:30.065 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:14:30.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:29 smithi171 ceph-mon[44189]: pgmap v690: 17 pgs: 1 active+recovering, 16 active+clean; 5.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 119 op/s; 9736/172684 objects misplaced (5.638%); 308 KiB/s, 4 objects/s recovering 2024-04-20T17:14:30.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:30.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:30.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:29 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:14:30.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:29 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3955759971' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:14:30.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:29 smithi171 ceph-mon[44189]: osdmap e250: 12 total, 12 up, 12 in 2024-04-20T17:14:30.096 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:29 smithi005 ceph-mon[39562]: pgmap v690: 17 pgs: 1 active+recovering, 16 active+clean; 5.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 119 op/s; 9736/172684 objects misplaced (5.638%); 308 KiB/s, 4 objects/s recovering 2024-04-20T17:14:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:29 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:14:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:29 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3955759971' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:14:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:29 smithi005 ceph-mon[39562]: osdmap e250: 12 total, 12 up, 12 in 2024-04-20T17:14:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:29 smithi106 ceph-mon[44423]: pgmap v690: 17 pgs: 1 active+recovering, 16 active+clean; 5.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 119 op/s; 9736/172684 objects misplaced (5.638%); 308 KiB/s, 4 objects/s recovering 2024-04-20T17:14:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:29 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:14:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:29 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3955759971' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:14:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:29 smithi106 ceph-mon[44423]: osdmap e250: 12 total, 12 up, 12 in 2024-04-20T17:14:30.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:30 smithi005 ceph-mon[39562]: pgmap v692: 17 pgs: 1 active+recovering, 16 active+clean; 5.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 115 op/s; 9655/172356 objects misplaced (5.602%); 640 KiB/s, 9 objects/s recovering 2024-04-20T17:14:31.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:30 smithi106 ceph-mon[44423]: pgmap v692: 17 pgs: 1 active+recovering, 16 active+clean; 5.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 115 op/s; 9655/172356 objects misplaced (5.602%); 640 KiB/s, 9 objects/s recovering 2024-04-20T17:14:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:30 smithi171 ceph-mon[44189]: pgmap v692: 17 pgs: 1 active+recovering, 16 active+clean; 5.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 115 op/s; 9655/172356 objects misplaced (5.602%); 640 KiB/s, 9 objects/s recovering 2024-04-20T17:14:32.148 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:14:32.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:32.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:14:32.155 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:14:32.155 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:14:32.155 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:14:32.155 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:14:32.155 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:14:32.253 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:14:32.281 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:32.301 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:14:32.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:32.305 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:14:32.406 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:14:32.432 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:32.452 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:14:32.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:32.454 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:14:32.457 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:14:32.555 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:14:32.586 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:32.606 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:14:32.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:32.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:14:32.711 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:14:32.738 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:32.757 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:14:32.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:32.760 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:14:32.861 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:14:32.887 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:32.905 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:14:32.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:32.908 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:14:32.959 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-20T17:14:32.972 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:14:33.009 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:14:33.040 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:33.061 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:14:33.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:33.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:14:33.165 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:14:33.194 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:33 smithi005 ceph-mon[39562]: pgmap v693: 17 pgs: 1 active+recovering, 16 active+clean; 5.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 112 op/s; 9655/171830 objects misplaced (5.619%); 640 KiB/s, 9 objects/s recovering 2024-04-20T17:14:33.196 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:33.299 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:14:33.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:33.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:14:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:33 smithi171 ceph-mon[44189]: pgmap v693: 17 pgs: 1 active+recovering, 16 active+clean; 5.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 112 op/s; 9655/171830 objects misplaced (5.619%); 640 KiB/s, 9 objects/s recovering 2024-04-20T17:14:33.403 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:14:33.436 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:33.499 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:14:33.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:33 smithi106 ceph-mon[44423]: pgmap v693: 17 pgs: 1 active+recovering, 16 active+clean; 5.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 112 op/s; 9655/171830 objects misplaced (5.619%); 640 KiB/s, 9 objects/s recovering 2024-04-20T17:14:33.535 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:14:33.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:33.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:14:33.640 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:14:33.673 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:33.875 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:14:33.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:33.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:14:33.882 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:14:33.983 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:14:34.011 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:34.030 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:14:34.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:34.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:14:34.134 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:14:34.160 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:34.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:34.213 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:14:34.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:14:34.314 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:14:34.345 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:35.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:35 smithi005 ceph-mon[39562]: pgmap v694: 17 pgs: 1 active+recovering, 16 active+clean; 5.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 133 op/s; 9655/171092 objects misplaced (5.643%); 328 KiB/s, 5 objects/s recovering 2024-04-20T17:14:35.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:35 smithi171 ceph-mon[44189]: pgmap v694: 17 pgs: 1 active+recovering, 16 active+clean; 5.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 133 op/s; 9655/171092 objects misplaced (5.643%); 328 KiB/s, 5 objects/s recovering 2024-04-20T17:14:35.422 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-20T17:14:35.423 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:14:35.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:35 smithi106 ceph-mon[44423]: pgmap v694: 17 pgs: 1 active+recovering, 16 active+clean; 5.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 133 op/s; 9655/171092 objects misplaced (5.643%); 328 KiB/s, 5 objects/s recovering 2024-04-20T17:14:35.824 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:14:36.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:36 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/855976232' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:14:36.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:36 smithi005 ceph-mon[39562]: from='client.25220 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:14:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:36 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/855976232' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:14:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:36 smithi171 ceph-mon[44189]: from='client.25220 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:14:36.429 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:14:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:36 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/855976232' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:14:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:36 smithi106 ceph-mon[44423]: from='client.25220 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:14:37.001 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:14:37.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:37.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:14:37.106 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:14:37.134 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:37.154 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:14:37.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:37.160 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:14:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:37 smithi005 ceph-mon[39562]: pgmap v695: 17 pgs: 1 active+recovering, 16 active+clean; 5.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 141 op/s; 9573/170650 objects misplaced (5.610%); 576 KiB/s, 9 objects/s recovering 2024-04-20T17:14:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:37 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:14:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:37 smithi005 ceph-mon[39562]: from='client.25220 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:14:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:37 smithi005 ceph-mon[39562]: osdmap e251: 12 total, 12 up, 12 in 2024-04-20T17:14:37.260 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:14:37.287 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:37.307 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:14:37.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:37.312 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:14:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:37 smithi171 ceph-mon[44189]: pgmap v695: 17 pgs: 1 active+recovering, 16 active+clean; 5.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 141 op/s; 9573/170650 objects misplaced (5.610%); 576 KiB/s, 9 objects/s recovering 2024-04-20T17:14:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:37 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:14:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:37 smithi171 ceph-mon[44189]: from='client.25220 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:14:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:37 smithi171 ceph-mon[44189]: osdmap e251: 12 total, 12 up, 12 in 2024-04-20T17:14:37.412 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:14:37.436 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:37 smithi106 ceph-mon[44423]: pgmap v695: 17 pgs: 1 active+recovering, 16 active+clean; 5.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 141 op/s; 9573/170650 objects misplaced (5.610%); 576 KiB/s, 9 objects/s recovering 2024-04-20T17:14:37.436 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:37 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:14:37.436 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:37 smithi106 ceph-mon[44423]: from='client.25220 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:14:37.437 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:37 smithi106 ceph-mon[44423]: osdmap e251: 12 total, 12 up, 12 in 2024-04-20T17:14:37.438 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:37.457 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:14:37.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:37.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:14:37.562 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:14:37.595 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:37.621 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:14:37.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:37.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:14:37.725 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:14:37.752 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:37.771 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:14:37.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:37.774 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:14:37.805 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:14:37.805 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:14:37.805 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:14:37.805 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:14:37.805 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.805 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212446 2.e 2:72d00b8c:::benchmark_data_smithi171_78918_object51452:head [delete] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:14:37.805 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897321+0000", 2024-04-20T17:14:37.805 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 134.527002356, 2024-04-20T17:14:37.805 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.67883147099999996, 2024-04-20T17:14:37.806 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:14:37.806 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:14:37.806 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:37.806 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:14:37.806 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:14:37.806 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:14:37.806 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212446 2024-04-20T17:14:37.806 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.806 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:14:37.806 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.806 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:14:37.806 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897321+0000", 2024-04-20T17:14:37.806 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.807 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.807 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.807 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:14:37.807 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897321+0000", 2024-04-20T17:14:37.807 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.807 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.807 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.808 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:14:37.808 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897323+0000", 2024-04-20T17:14:37.808 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.424e-06 2024-04-20T17:14:37.808 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.808 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.808 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:14:37.808 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897329+0000", 2024-04-20T17:14:37.808 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5559999999999998e-06 2024-04-20T17:14:37.808 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.808 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.808 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:14:37.809 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897338+0000", 2024-04-20T17:14:37.809 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0520000000000007e-06 2024-04-20T17:14:37.809 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.809 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.809 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:14:37.809 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897346+0000", 2024-04-20T17:14:37.809 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7619999999999995e-06 2024-04-20T17:14:37.809 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.809 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.809 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.809 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898452+0000", 2024-04-20T17:14:37.809 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0011067480000000001 2024-04-20T17:14:37.810 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.810 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.810 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:14:37.810 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898470+0000", 2024-04-20T17:14:37.810 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7864e-05 2024-04-20T17:14:37.810 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.810 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.810 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.810 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.559597+0000", 2024-04-20T17:14:37.810 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.66112692699999998 2024-04-20T17:14:37.810 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.811 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.811 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:14:37.811 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.559698+0000", 2024-04-20T17:14:37.811 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000101 2024-04-20T17:14:37.811 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.811 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.811 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:14:37.811 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.559788+0000", 2024-04-20T17:14:37.811 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9697999999999994e-05 2024-04-20T17:14:37.811 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.811 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.811 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:14:37.811 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.575978+0000", 2024-04-20T17:14:37.812 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.016189985 2024-04-20T17:14:37.812 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.812 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.812 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:14:37.812 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576031+0000", 2024-04-20T17:14:37.812 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3086e-05 2024-04-20T17:14:37.812 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.812 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.812 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:14:37.812 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576097+0000", 2024-04-20T17:14:37.812 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6303999999999994e-05 2024-04-20T17:14:37.812 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.812 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.812 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:14:37.813 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576152+0000", 2024-04-20T17:14:37.813 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5065000000000002e-05 2024-04-20T17:14:37.813 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.813 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:14:37.813 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.813 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.813 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.813 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212450 2.e 2:72ccd005:::benchmark_data_smithi171_78918_object51456:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:14:37.813 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897360+0000", 2024-04-20T17:14:37.813 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 134.52696273399999, 2024-04-20T17:14:37.813 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.67896632000000001, 2024-04-20T17:14:37.813 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:14:37.813 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:14:37.813 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:37.814 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:14:37.814 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:14:37.814 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:14:37.814 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212450 2024-04-20T17:14:37.814 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.814 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:14:37.814 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.814 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:14:37.814 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897360+0000", 2024-04-20T17:14:37.814 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.814 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.815 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.815 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:14:37.815 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897360+0000", 2024-04-20T17:14:37.815 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.815 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.815 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.815 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:14:37.815 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897361+0000", 2024-04-20T17:14:37.815 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2800000000000005e-07 2024-04-20T17:14:37.815 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.815 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.816 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:14:37.816 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897364+0000", 2024-04-20T17:14:37.816 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.977e-06 2024-04-20T17:14:37.816 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.816 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.816 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:14:37.816 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897370+0000", 2024-04-20T17:14:37.816 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0530000000000003e-06 2024-04-20T17:14:37.816 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.816 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.817 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:14:37.817 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897376+0000", 2024-04-20T17:14:37.817 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1419999999999998e-06 2024-04-20T17:14:37.817 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.817 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.817 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.817 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898518+0000", 2024-04-20T17:14:37.817 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0011413 2024-04-20T17:14:37.817 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.817 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.817 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:14:37.818 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898526+0000", 2024-04-20T17:14:37.818 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0150000000000003e-06 2024-04-20T17:14:37.818 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.818 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.818 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.818 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.563095+0000", 2024-04-20T17:14:37.818 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.664568838 2024-04-20T17:14:37.818 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.818 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.819 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:14:37.819 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.563218+0000", 2024-04-20T17:14:37.819 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000123051 2024-04-20T17:14:37.819 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.819 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.819 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:14:37.819 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.563315+0000", 2024-04-20T17:14:37.819 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6935000000000006e-05 2024-04-20T17:14:37.819 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.819 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.819 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:14:37.820 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576186+0000", 2024-04-20T17:14:37.820 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012871698000000001 2024-04-20T17:14:37.820 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.820 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.820 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:14:37.820 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576223+0000", 2024-04-20T17:14:37.820 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7048999999999997e-05 2024-04-20T17:14:37.820 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.820 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.820 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:14:37.820 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576275+0000", 2024-04-20T17:14:37.820 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1944000000000003e-05 2024-04-20T17:14:37.820 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.820 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.820 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:14:37.820 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576327+0000", 2024-04-20T17:14:37.821 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1390000000000001e-05 2024-04-20T17:14:37.821 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.821 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:14:37.821 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.821 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.821 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.821 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212454 2.e 2:7603a42b:::benchmark_data_smithi171_78918_object51460:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:14:37.821 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897383+0000", 2024-04-20T17:14:37.821 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 134.526940164, 2024-04-20T17:14:37.821 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.67906811499999997, 2024-04-20T17:14:37.821 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:14:37.821 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:14:37.821 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:37.821 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:14:37.822 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:14:37.822 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:14:37.822 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212454 2024-04-20T17:14:37.822 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.822 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:14:37.822 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.822 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:14:37.822 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897383+0000", 2024-04-20T17:14:37.822 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.822 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.822 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.822 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:14:37.822 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897383+0000", 2024-04-20T17:14:37.822 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.823 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.823 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.823 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:14:37.823 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897383+0000", 2024-04-20T17:14:37.823 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8599999999999998e-07 2024-04-20T17:14:37.823 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.823 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.823 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:14:37.823 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897385+0000", 2024-04-20T17:14:37.823 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.041e-06 2024-04-20T17:14:37.823 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.823 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.823 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:14:37.823 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897393+0000", 2024-04-20T17:14:37.824 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3730000000000004e-06 2024-04-20T17:14:37.824 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.824 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.824 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:14:37.824 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897396+0000", 2024-04-20T17:14:37.824 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2449999999999998e-06 2024-04-20T17:14:37.824 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.824 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.824 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.824 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898572+0000", 2024-04-20T17:14:37.824 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001176329 2024-04-20T17:14:37.824 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.824 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.824 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:14:37.824 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898581+0000", 2024-04-20T17:14:37.824 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6989999999999992e-06 2024-04-20T17:14:37.825 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.826 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.826 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.826 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.566135+0000", 2024-04-20T17:14:37.826 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.66755413200000002 2024-04-20T17:14:37.826 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.826 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.826 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:14:37.826 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.566275+0000", 2024-04-20T17:14:37.826 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014009600000000001 2024-04-20T17:14:37.826 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.826 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.826 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:14:37.827 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.566419+0000", 2024-04-20T17:14:37.827 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014339600000000001 2024-04-20T17:14:37.827 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.827 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.827 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:14:37.828 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576374+0000", 2024-04-20T17:14:37.828 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0099556760000000001 2024-04-20T17:14:37.828 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.828 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.828 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:14:37.828 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576383+0000", 2024-04-20T17:14:37.828 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0920000000000004e-06 2024-04-20T17:14:37.828 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.828 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.828 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:14:37.829 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576412+0000", 2024-04-20T17:14:37.829 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8642000000000001e-05 2024-04-20T17:14:37.829 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.829 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.829 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:14:37.829 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576451+0000", 2024-04-20T17:14:37.829 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8908000000000001e-05 2024-04-20T17:14:37.829 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.829 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:14:37.829 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.829 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.829 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.829 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212458 2.e 2:7672b154:::benchmark_data_smithi171_78918_object51464:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:14:37.829 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897400+0000", 2024-04-20T17:14:37.829 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 134.52692306599999, 2024-04-20T17:14:37.830 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.68587742500000004, 2024-04-20T17:14:37.830 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:14:37.830 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:14:37.830 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:37.830 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:14:37.830 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:14:37.830 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:14:37.830 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212458 2024-04-20T17:14:37.830 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.830 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:14:37.830 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.830 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:14:37.830 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897400+0000", 2024-04-20T17:14:37.830 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.831 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.831 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.831 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:14:37.831 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897400+0000", 2024-04-20T17:14:37.831 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.831 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.831 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.831 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:14:37.831 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897401+0000", 2024-04-20T17:14:37.831 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3000000000000001e-07 2024-04-20T17:14:37.831 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.831 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.831 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:14:37.831 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897403+0000", 2024-04-20T17:14:37.832 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.198e-06 2024-04-20T17:14:37.832 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.832 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.832 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:14:37.832 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897408+0000", 2024-04-20T17:14:37.832 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1109999999999999e-06 2024-04-20T17:14:37.832 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.832 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.832 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:14:37.832 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897414+0000", 2024-04-20T17:14:37.832 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7139999999999998e-06 2024-04-20T17:14:37.832 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.832 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.832 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.832 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898608+0000", 2024-04-20T17:14:37.833 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0011941160000000001 2024-04-20T17:14:37.833 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.833 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.833 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:14:37.833 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898614+0000", 2024-04-20T17:14:37.833 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7100000000000001e-06 2024-04-20T17:14:37.833 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.833 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.833 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.833 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.569428+0000", 2024-04-20T17:14:37.833 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.67081384399999999 2024-04-20T17:14:37.833 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.833 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.833 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:14:37.833 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.569562+0000", 2024-04-20T17:14:37.834 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000134055 2024-04-20T17:14:37.834 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.834 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.834 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:14:37.834 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.569692+0000", 2024-04-20T17:14:37.834 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012998099999999999 2024-04-20T17:14:37.834 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.834 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.834 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:14:37.834 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576458+0000", 2024-04-20T17:14:37.834 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0067652670000000002 2024-04-20T17:14:37.834 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.834 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.834 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:14:37.834 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.583158+0000", 2024-04-20T17:14:37.835 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0067007229999999996 2024-04-20T17:14:37.835 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.835 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.835 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:14:37.835 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.583225+0000", 2024-04-20T17:14:37.835 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6617000000000003e-05 2024-04-20T17:14:37.835 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.835 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.835 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:14:37.835 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.583277+0000", 2024-04-20T17:14:37.835 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2559000000000003e-05 2024-04-20T17:14:37.835 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.835 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:14:37.835 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.835 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.836 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.836 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212462 2.e 2:7f179a5c:::benchmark_data_smithi171_78918_object51468:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:14:37.836 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897424+0000", 2024-04-20T17:14:37.836 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 134.52689873, 2024-04-20T17:14:37.836 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.69171569799999999, 2024-04-20T17:14:37.836 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:14:37.836 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:14:37.836 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:37.836 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:14:37.836 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:14:37.836 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:14:37.836 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212462 2024-04-20T17:14:37.836 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.836 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:14:37.836 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.837 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:14:37.837 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897424+0000", 2024-04-20T17:14:37.837 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.837 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.837 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.837 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:14:37.837 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897424+0000", 2024-04-20T17:14:37.837 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.837 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.837 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.837 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:14:37.837 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897425+0000", 2024-04-20T17:14:37.837 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6199999999999997e-07 2024-04-20T17:14:37.837 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.838 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.838 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:14:37.838 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897428+0000", 2024-04-20T17:14:37.838 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1219999999999999e-06 2024-04-20T17:14:37.838 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.838 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.838 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:14:37.838 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897436+0000", 2024-04-20T17:14:37.838 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3270000000000001e-06 2024-04-20T17:14:37.838 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.838 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.838 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:14:37.838 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897442+0000", 2024-04-20T17:14:37.838 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6200000000000004e-06 2024-04-20T17:14:37.838 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.839 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.839 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.839 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898653+0000", 2024-04-20T17:14:37.839 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0012105550000000001 2024-04-20T17:14:37.839 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.839 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.839 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:14:37.839 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898660+0000", 2024-04-20T17:14:37.839 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1810000000000004e-06 2024-04-20T17:14:37.839 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.839 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.839 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.839 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.573380+0000", 2024-04-20T17:14:37.839 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.67471982699999999 2024-04-20T17:14:37.839 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.841 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.841 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:14:37.841 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.573481+0000", 2024-04-20T17:14:37.841 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010135499999999999 2024-04-20T17:14:37.841 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.841 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.841 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:14:37.841 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.573564+0000", 2024-04-20T17:14:37.841 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3258999999999997e-05 2024-04-20T17:14:37.842 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.842 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.842 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:14:37.842 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576463+0000", 2024-04-20T17:14:37.842 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0028989609999999998 2024-04-20T17:14:37.842 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.842 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.842 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:14:37.842 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.589024+0000", 2024-04-20T17:14:37.842 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012561249999999999 2024-04-20T17:14:37.842 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.842 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.842 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:14:37.842 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.589088+0000", 2024-04-20T17:14:37.843 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3398000000000005e-05 2024-04-20T17:14:37.843 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.843 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.843 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:14:37.843 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.589140+0000", 2024-04-20T17:14:37.843 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2181000000000001e-05 2024-04-20T17:14:37.843 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.843 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:14:37.843 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.843 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.843 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.843 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212469 2.e 2:7df2bad1:::benchmark_data_smithi171_78918_object51475:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:14:37.843 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897447+0000", 2024-04-20T17:14:37.843 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 134.526876536, 2024-04-20T17:14:37.843 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.71189628299999996, 2024-04-20T17:14:37.844 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:14:37.844 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:14:37.844 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:37.844 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:14:37.844 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:14:37.844 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:14:37.844 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212469 2024-04-20T17:14:37.844 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.844 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:14:37.844 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.844 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:14:37.844 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897447+0000", 2024-04-20T17:14:37.844 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.844 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.844 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.845 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:14:37.845 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897447+0000", 2024-04-20T17:14:37.845 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.845 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.845 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.845 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:14:37.845 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897447+0000", 2024-04-20T17:14:37.845 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.89e-07 2024-04-20T17:14:37.845 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.845 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.845 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:14:37.845 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897449+0000", 2024-04-20T17:14:37.845 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3530000000000002e-06 2024-04-20T17:14:37.845 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.845 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.846 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:14:37.846 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897455+0000", 2024-04-20T17:14:37.846 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2850000000000002e-06 2024-04-20T17:14:37.846 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.846 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.846 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:14:37.846 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897459+0000", 2024-04-20T17:14:37.846 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7129999999999996e-06 2024-04-20T17:14:37.846 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.846 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.846 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.846 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898683+0000", 2024-04-20T17:14:37.846 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0012234629999999999 2024-04-20T17:14:37.846 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.847 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.847 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:14:37.847 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898690+0000", 2024-04-20T17:14:37.847 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9419999999999998e-06 2024-04-20T17:14:37.847 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.847 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.847 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.847 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576505+0000", 2024-04-20T17:14:37.847 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.677814888 2024-04-20T17:14:37.847 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.847 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.847 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:14:37.847 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576644+0000", 2024-04-20T17:14:37.847 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013927000000000001 2024-04-20T17:14:37.847 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.848 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.848 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:14:37.848 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576783+0000", 2024-04-20T17:14:37.848 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013961799999999999 2024-04-20T17:14:37.848 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.848 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.848 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:14:37.848 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.586310+0000", 2024-04-20T17:14:37.848 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0095266459999999997 2024-04-20T17:14:37.848 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.848 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.848 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:14:37.848 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609211+0000", 2024-04-20T17:14:37.848 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.022901248999999999 2024-04-20T17:14:37.848 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.849 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.849 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:14:37.849 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609274+0000", 2024-04-20T17:14:37.849 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3045000000000004e-05 2024-04-20T17:14:37.849 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.849 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.849 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:14:37.849 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609343+0000", 2024-04-20T17:14:37.849 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8422000000000003e-05 2024-04-20T17:14:37.849 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.849 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:14:37.849 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.849 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.849 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.849 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212495 2.e 2:782e28d2:::benchmark_data_smithi171_78918_object51501:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:14:37.849 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897465+0000", 2024-04-20T17:14:37.850 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 134.526858285, 2024-04-20T17:14:37.850 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.71202960599999998, 2024-04-20T17:14:37.850 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:14:37.850 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:14:37.850 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:37.850 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:14:37.850 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:14:37.850 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:14:37.850 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212495 2024-04-20T17:14:37.850 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.850 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:14:37.850 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.850 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:14:37.850 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897465+0000", 2024-04-20T17:14:37.851 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.851 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.851 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.851 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:14:37.851 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897465+0000", 2024-04-20T17:14:37.851 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.851 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.851 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.851 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:14:37.851 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897465+0000", 2024-04-20T17:14:37.851 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.68e-07 2024-04-20T17:14:37.851 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.851 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.851 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:14:37.851 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897468+0000", 2024-04-20T17:14:37.852 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7800000000000001e-06 2024-04-20T17:14:37.852 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.852 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.852 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:14:37.852 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897475+0000", 2024-04-20T17:14:37.852 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1069999999999999e-06 2024-04-20T17:14:37.852 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.852 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.852 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:14:37.852 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897481+0000", 2024-04-20T17:14:37.852 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3539999999999999e-06 2024-04-20T17:14:37.852 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.852 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.852 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.852 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898736+0000", 2024-04-20T17:14:37.853 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001255812 2024-04-20T17:14:37.853 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.853 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.853 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:14:37.853 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898743+0000", 2024-04-20T17:14:37.853 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6830000000000002e-06 2024-04-20T17:14:37.853 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.853 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.853 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.853 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.580475+0000", 2024-04-20T17:14:37.853 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.68173151600000004 2024-04-20T17:14:37.853 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.853 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.853 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:14:37.853 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.580557+0000", 2024-04-20T17:14:37.854 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2826999999999998e-05 2024-04-20T17:14:37.854 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.854 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.854 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:14:37.854 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.580649+0000", 2024-04-20T17:14:37.854 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1611000000000001e-05 2024-04-20T17:14:37.854 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.854 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.854 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:14:37.854 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.589156+0000", 2024-04-20T17:14:37.854 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0085067349999999996 2024-04-20T17:14:37.854 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.854 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.854 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:14:37.854 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609403+0000", 2024-04-20T17:14:37.855 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.020247660000000001 2024-04-20T17:14:37.855 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.855 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.855 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:14:37.855 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609449+0000", 2024-04-20T17:14:37.855 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5649999999999998e-05 2024-04-20T17:14:37.855 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.855 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.855 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:14:37.855 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609494+0000", 2024-04-20T17:14:37.855 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5303000000000003e-05 2024-04-20T17:14:37.855 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.855 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:14:37.855 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.855 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.855 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.856 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212512 2.e 2:7c5add18:::benchmark_data_smithi171_78918_object51518:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:14:37.856 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897487+0000", 2024-04-20T17:14:37.856 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 134.52683634900001, 2024-04-20T17:14:37.856 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.71708918399999999, 2024-04-20T17:14:37.856 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:14:37.856 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:14:37.856 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:37.856 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:14:37.856 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:14:37.856 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:14:37.856 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212512 2024-04-20T17:14:37.856 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.856 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:14:37.856 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.856 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:14:37.857 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897487+0000", 2024-04-20T17:14:37.857 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.857 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.857 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.857 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:14:37.857 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897487+0000", 2024-04-20T17:14:37.857 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.857 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.857 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.857 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:14:37.857 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897487+0000", 2024-04-20T17:14:37.857 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9399999999999995e-07 2024-04-20T17:14:37.857 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.857 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.857 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:14:37.858 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897498+0000", 2024-04-20T17:14:37.858 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0565e-05 2024-04-20T17:14:37.858 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.858 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.858 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:14:37.858 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897503+0000", 2024-04-20T17:14:37.858 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6590000000000004e-06 2024-04-20T17:14:37.858 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.858 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.858 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:14:37.858 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897507+0000", 2024-04-20T17:14:37.858 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.805e-06 2024-04-20T17:14:37.858 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.858 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.858 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.859 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898819+0000", 2024-04-20T17:14:37.859 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00131193 2024-04-20T17:14:37.859 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.859 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.859 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:14:37.859 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898826+0000", 2024-04-20T17:14:37.859 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6760000000000001e-06 2024-04-20T17:14:37.859 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.859 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.859 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.859 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.583338+0000", 2024-04-20T17:14:37.859 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.68451254699999997 2024-04-20T17:14:37.859 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.859 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.859 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:14:37.859 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.583467+0000", 2024-04-20T17:14:37.860 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000129016 2024-04-20T17:14:37.860 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.860 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.860 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:14:37.860 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.583627+0000", 2024-04-20T17:14:37.860 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000159658 2024-04-20T17:14:37.860 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.860 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.860 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:14:37.860 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.591594+0000", 2024-04-20T17:14:37.860 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0079671770000000006 2024-04-20T17:14:37.860 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.860 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.860 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:14:37.860 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.614474+0000", 2024-04-20T17:14:37.861 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.022880211000000001 2024-04-20T17:14:37.861 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.861 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.861 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:14:37.861 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.614528+0000", 2024-04-20T17:14:37.861 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3721000000000003e-05 2024-04-20T17:14:37.861 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.861 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.861 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:14:37.861 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.614576+0000", 2024-04-20T17:14:37.861 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7725000000000001e-05 2024-04-20T17:14:37.861 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.861 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:14:37.861 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.861 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.861 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.862 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212527 2.e 2:73e27e1e:::benchmark_data_smithi171_78918_object51533:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:14:37.862 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897511+0000", 2024-04-20T17:14:37.862 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 134.52681192700001, 2024-04-20T17:14:37.862 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.71722206099999997, 2024-04-20T17:14:37.862 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:14:37.862 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:14:37.862 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:37.862 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:14:37.862 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:14:37.862 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:14:37.862 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212527 2024-04-20T17:14:37.862 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.862 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:14:37.862 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.862 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:14:37.862 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897511+0000", 2024-04-20T17:14:37.863 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.863 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.863 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.863 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:14:37.863 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897511+0000", 2024-04-20T17:14:37.863 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.863 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.863 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.863 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:14:37.863 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897512+0000", 2024-04-20T17:14:37.863 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3999999999999997e-07 2024-04-20T17:14:37.863 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.863 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.863 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:14:37.863 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897513+0000", 2024-04-20T17:14:37.864 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.302e-06 2024-04-20T17:14:37.864 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.864 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.864 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:14:37.864 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897516+0000", 2024-04-20T17:14:37.864 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4060000000000001e-06 2024-04-20T17:14:37.864 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.864 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.864 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:14:37.864 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897519+0000", 2024-04-20T17:14:37.864 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6840000000000001e-06 2024-04-20T17:14:37.864 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.864 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.864 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.864 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898869+0000", 2024-04-20T17:14:37.865 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0013499180000000001 2024-04-20T17:14:37.865 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.865 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.865 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:14:37.865 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898876+0000", 2024-04-20T17:14:37.865 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4490000000000002e-06 2024-04-20T17:14:37.865 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.865 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.865 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.865 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.586356+0000", 2024-04-20T17:14:37.865 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.68747954499999997 2024-04-20T17:14:37.865 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.865 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.865 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:14:37.865 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.586502+0000", 2024-04-20T17:14:37.865 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014652999999999999 2024-04-20T17:14:37.866 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.866 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.866 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:14:37.866 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.586643+0000", 2024-04-20T17:14:37.866 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014104100000000001 2024-04-20T17:14:37.866 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.866 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.866 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:14:37.866 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.591604+0000", 2024-04-20T17:14:37.866 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0049606090000000004 2024-04-20T17:14:37.866 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.866 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.866 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:14:37.866 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.614639+0000", 2024-04-20T17:14:37.866 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023034974 2024-04-20T17:14:37.867 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.867 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.867 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:14:37.867 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.614690+0000", 2024-04-20T17:14:37.867 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1134999999999997e-05 2024-04-20T17:14:37.867 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.867 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.867 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:14:37.867 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.614733+0000", 2024-04-20T17:14:37.867 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3127999999999998e-05 2024-04-20T17:14:37.867 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.867 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:14:37.867 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.867 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.867 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.867 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212531 2.e 2:7eb9e3fb:::benchmark_data_smithi171_78918_object51537:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:14:37.868 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897525+0000", 2024-04-20T17:14:37.868 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 134.52679814800001, 2024-04-20T17:14:37.868 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.71744083700000005, 2024-04-20T17:14:37.868 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:14:37.868 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:14:37.868 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:37.868 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:14:37.868 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:14:37.868 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:14:37.868 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212531 2024-04-20T17:14:37.868 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.868 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:14:37.868 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.868 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:14:37.868 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897525+0000", 2024-04-20T17:14:37.869 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.869 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.869 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.869 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:14:37.869 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897525+0000", 2024-04-20T17:14:37.869 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.869 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.869 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.869 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:14:37.869 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897525+0000", 2024-04-20T17:14:37.869 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5600000000000001e-07 2024-04-20T17:14:37.869 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.869 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.869 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:14:37.869 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897527+0000", 2024-04-20T17:14:37.870 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9410000000000001e-06 2024-04-20T17:14:37.870 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.870 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.870 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:14:37.870 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897534+0000", 2024-04-20T17:14:37.870 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1199999999999996e-06 2024-04-20T17:14:37.870 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.870 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.870 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:14:37.870 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897539+0000", 2024-04-20T17:14:37.870 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.921e-06 2024-04-20T17:14:37.870 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.870 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.870 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.870 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898909+0000", 2024-04-20T17:14:37.871 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001369706 2024-04-20T17:14:37.871 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.871 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.871 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:14:37.871 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898915+0000", 2024-04-20T17:14:37.871 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4160000000000001e-06 2024-04-20T17:14:37.871 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.871 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.871 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.871 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.589187+0000", 2024-04-20T17:14:37.871 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.69027111600000002 2024-04-20T17:14:37.871 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.871 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.871 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:14:37.871 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.589267+0000", 2024-04-20T17:14:37.871 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0399000000000001e-05 2024-04-20T17:14:37.872 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.872 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.872 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:14:37.872 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.589349+0000", 2024-04-20T17:14:37.872 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1904999999999996e-05 2024-04-20T17:14:37.872 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.872 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.872 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:14:37.872 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.600413+0000", 2024-04-20T17:14:37.872 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.01106378 2024-04-20T17:14:37.872 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.872 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.872 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:14:37.872 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.614882+0000", 2024-04-20T17:14:37.872 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014469556999999999 2024-04-20T17:14:37.872 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.873 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.873 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:14:37.873 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.614926+0000", 2024-04-20T17:14:37.873 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3649000000000002e-05 2024-04-20T17:14:37.873 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.873 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.873 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:14:37.873 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.614966+0000", 2024-04-20T17:14:37.873 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9971000000000001e-05 2024-04-20T17:14:37.873 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.873 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:14:37.873 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.873 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.873 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.873 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212541 2.e 2:7a53380b:::benchmark_data_smithi171_78918_object51547:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:14:37.874 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897543+0000", 2024-04-20T17:14:37.874 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 134.52677985099999, 2024-04-20T17:14:37.874 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.720308693, 2024-04-20T17:14:37.874 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:14:37.874 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:14:37.874 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:37.874 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:14:37.874 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:14:37.874 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:14:37.874 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212541 2024-04-20T17:14:37.874 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.874 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:14:37.874 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.874 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:14:37.874 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897543+0000", 2024-04-20T17:14:37.874 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.875 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.875 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.875 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:14:37.875 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897543+0000", 2024-04-20T17:14:37.875 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.875 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.875 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.875 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:14:37.875 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897544+0000", 2024-04-20T17:14:37.875 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5100000000000001e-07 2024-04-20T17:14:37.875 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.875 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.875 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:14:37.875 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897545+0000", 2024-04-20T17:14:37.875 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.412e-06 2024-04-20T17:14:37.876 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.876 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.876 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:14:37.876 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897548+0000", 2024-04-20T17:14:37.876 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.858e-06 2024-04-20T17:14:37.876 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.876 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.876 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:14:37.876 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897551+0000", 2024-04-20T17:14:37.876 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0189999999999998e-06 2024-04-20T17:14:37.876 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.876 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.877 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.877 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898951+0000", 2024-04-20T17:14:37.877 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0013997440000000001 2024-04-20T17:14:37.877 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.877 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.877 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:14:37.877 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898958+0000", 2024-04-20T17:14:37.877 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4580000000000004e-06 2024-04-20T17:14:37.877 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.877 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.877 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.877 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.591641+0000", 2024-04-20T17:14:37.877 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.69268292399999998 2024-04-20T17:14:37.877 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.878 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.878 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:14:37.878 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.591724+0000", 2024-04-20T17:14:37.878 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3249000000000002e-05 2024-04-20T17:14:37.878 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.878 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.878 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:14:37.878 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.591816+0000", 2024-04-20T17:14:37.878 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1774000000000006e-05 2024-04-20T17:14:37.878 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.878 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.878 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:14:37.878 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.600419+0000", 2024-04-20T17:14:37.878 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0086027299999999994 2024-04-20T17:14:37.879 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.879 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.879 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:14:37.879 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.617785+0000", 2024-04-20T17:14:37.879 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.017365914999999999 2024-04-20T17:14:37.879 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.879 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.879 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:14:37.879 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.617819+0000", 2024-04-20T17:14:37.879 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4743000000000003e-05 2024-04-20T17:14:37.879 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.879 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.879 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:14:37.879 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.617852+0000", 2024-04-20T17:14:37.879 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2515999999999998e-05 2024-04-20T17:14:37.880 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.880 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:14:37.880 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.880 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.880 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.880 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212573 2.e 2:73f6f3d3:::benchmark_data_smithi171_78918_object51579:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:14:37.880 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897554+0000", 2024-04-20T17:14:37.880 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 134.52676949799999, 2024-04-20T17:14:37.880 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.72045646900000004, 2024-04-20T17:14:37.880 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:14:37.880 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:14:37.880 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:37.880 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:14:37.880 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:14:37.880 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:14:37.881 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212573 2024-04-20T17:14:37.881 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.881 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:14:37.881 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.881 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:14:37.881 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897554+0000", 2024-04-20T17:14:37.881 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.881 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.881 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.881 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:14:37.881 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897554+0000", 2024-04-20T17:14:37.881 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.881 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.881 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.881 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:14:37.882 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897554+0000", 2024-04-20T17:14:37.882 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8900000000000001e-07 2024-04-20T17:14:37.882 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.882 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.882 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:14:37.882 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897555+0000", 2024-04-20T17:14:37.882 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.063e-06 2024-04-20T17:14:37.882 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.882 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.882 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:14:37.882 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897558+0000", 2024-04-20T17:14:37.882 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3400000000000002e-06 2024-04-20T17:14:37.882 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.882 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.882 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:14:37.883 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897566+0000", 2024-04-20T17:14:37.883 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6629999999999993e-06 2024-04-20T17:14:37.883 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.883 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.883 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.883 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898998+0000", 2024-04-20T17:14:37.883 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001432326 2024-04-20T17:14:37.883 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.883 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.883 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:14:37.883 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.899005+0000", 2024-04-20T17:14:37.883 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.737e-06 2024-04-20T17:14:37.883 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.883 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.883 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.884 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.594204+0000", 2024-04-20T17:14:37.884 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.69519878599999996 2024-04-20T17:14:37.884 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.884 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.884 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:14:37.884 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.594324+0000", 2024-04-20T17:14:37.884 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000119827 2024-04-20T17:14:37.884 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.884 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.884 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:14:37.884 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.594406+0000", 2024-04-20T17:14:37.884 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2727999999999998e-05 2024-04-20T17:14:37.884 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.884 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.884 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:14:37.885 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.605467+0000", 2024-04-20T17:14:37.885 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011060449999999999 2024-04-20T17:14:37.885 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.885 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.885 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:14:37.885 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.617903+0000", 2024-04-20T17:14:37.885 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012436361 2024-04-20T17:14:37.885 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.885 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.885 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:14:37.885 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.617964+0000", 2024-04-20T17:14:37.885 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0569999999999998e-05 2024-04-20T17:14:37.885 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.885 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.885 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:14:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.618010+0000", 2024-04-20T17:14:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6329000000000003e-05 2024-04-20T17:14:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:14:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212593 2.e 2:744300f2:::benchmark_data_smithi171_78918_object51599:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:14:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897569+0000", 2024-04-20T17:14:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 134.52675361300001, 2024-04-20T17:14:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.72464595399999998, 2024-04-20T17:14:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:14:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:14:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:14:37.887 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:14:37.887 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:14:37.887 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212593 2024-04-20T17:14:37.887 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.887 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:14:37.887 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.887 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:14:37.887 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897569+0000", 2024-04-20T17:14:37.887 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.887 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.887 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.887 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:14:37.887 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897569+0000", 2024-04-20T17:14:37.887 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.887 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.887 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.888 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:14:37.888 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897570+0000", 2024-04-20T17:14:37.888 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3999999999999997e-07 2024-04-20T17:14:37.888 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.888 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.888 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:14:37.888 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897571+0000", 2024-04-20T17:14:37.888 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.296e-06 2024-04-20T17:14:37.888 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.888 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.888 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:14:37.888 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897574+0000", 2024-04-20T17:14:37.888 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7970000000000001e-06 2024-04-20T17:14:37.888 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.888 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.889 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:14:37.889 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897577+0000", 2024-04-20T17:14:37.889 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8389999999999998e-06 2024-04-20T17:14:37.889 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.889 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.889 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.889 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.899045+0000", 2024-04-20T17:14:37.889 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0014678639999999999 2024-04-20T17:14:37.889 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.889 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.889 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:14:37.889 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.899052+0000", 2024-04-20T17:14:37.889 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3010000000000001e-06 2024-04-20T17:14:37.889 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.889 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.890 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.890 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.597702+0000", 2024-04-20T17:14:37.890 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.69865043599999999 2024-04-20T17:14:37.890 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.890 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.890 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:14:37.890 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.597847+0000", 2024-04-20T17:14:37.890 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000144291 2024-04-20T17:14:37.890 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.890 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.890 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:14:37.890 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.598003+0000", 2024-04-20T17:14:37.890 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000156615 2024-04-20T17:14:37.890 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.891 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.891 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:14:37.891 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609358+0000", 2024-04-20T17:14:37.891 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011354991 2024-04-20T17:14:37.891 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.891 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.891 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:14:37.891 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.622109+0000", 2024-04-20T17:14:37.891 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012751035000000001 2024-04-20T17:14:37.891 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.891 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.891 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:14:37.891 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.622164+0000", 2024-04-20T17:14:37.891 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4867000000000002e-05 2024-04-20T17:14:37.891 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.892 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.892 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:14:37.892 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.622215+0000", 2024-04-20T17:14:37.892 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1282000000000001e-05 2024-04-20T17:14:37.892 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.892 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:14:37.892 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.892 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.892 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.892 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212638 2.e 2:7e5004ea:::benchmark_data_smithi171_78918_object51644:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:14:37.892 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897579+0000", 2024-04-20T17:14:37.892 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 134.52674365600001, 2024-04-20T17:14:37.892 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.73145174000000002, 2024-04-20T17:14:37.892 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:14:37.892 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:14:37.893 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:37.893 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:14:37.893 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:14:37.893 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:14:37.893 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212638 2024-04-20T17:14:37.893 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.893 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:14:37.893 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.893 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:14:37.893 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897579+0000", 2024-04-20T17:14:37.893 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.893 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.893 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.893 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:14:37.893 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897579+0000", 2024-04-20T17:14:37.894 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.894 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.894 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.894 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:14:37.894 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897580+0000", 2024-04-20T17:14:37.894 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9900000000000002e-07 2024-04-20T17:14:37.894 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.894 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.894 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:14:37.894 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897581+0000", 2024-04-20T17:14:37.894 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0380000000000001e-06 2024-04-20T17:14:37.894 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.894 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.894 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:14:37.894 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897585+0000", 2024-04-20T17:14:37.895 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7479999999999999e-06 2024-04-20T17:14:37.895 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.895 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.895 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:14:37.895 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897594+0000", 2024-04-20T17:14:37.895 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7899999999999994e-06 2024-04-20T17:14:37.895 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.895 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.895 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.895 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.899085+0000", 2024-04-20T17:14:37.895 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0014909649999999999 2024-04-20T17:14:37.895 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.895 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.895 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:14:37.896 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.899091+0000", 2024-04-20T17:14:37.896 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8170000000000004e-06 2024-04-20T17:14:37.896 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.896 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.896 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.896 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.600462+0000", 2024-04-20T17:14:37.896 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.70137081800000001 2024-04-20T17:14:37.896 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.896 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.896 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:14:37.896 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.600548+0000", 2024-04-20T17:14:37.896 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5815000000000004e-05 2024-04-20T17:14:37.896 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.896 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.896 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:14:37.897 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.600638+0000", 2024-04-20T17:14:37.897 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9998000000000001e-05 2024-04-20T17:14:37.897 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.897 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.897 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:14:37.897 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609366+0000", 2024-04-20T17:14:37.897 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0087284849999999994 2024-04-20T17:14:37.897 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.897 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.897 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:14:37.897 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.628925+0000", 2024-04-20T17:14:37.897 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.019558467999999999 2024-04-20T17:14:37.897 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.897 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.897 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:14:37.898 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.628978+0000", 2024-04-20T17:14:37.898 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.329e-05 2024-04-20T17:14:37.898 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.898 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.898 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:14:37.898 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.629031+0000", 2024-04-20T17:14:37.898 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3208999999999998e-05 2024-04-20T17:14:37.898 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.898 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:14:37.898 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.898 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.898 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.898 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212664 2.e 2:78efb1e3:::benchmark_data_smithi171_78918_object51670:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:14:37.898 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897598+0000", 2024-04-20T17:14:37.898 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 134.52672519199999, 2024-04-20T17:14:37.899 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.73297929500000003, 2024-04-20T17:14:37.899 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:14:37.899 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:14:37.899 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:37.899 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:14:37.899 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:14:37.899 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:14:37.899 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212664 2024-04-20T17:14:37.899 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.899 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:14:37.899 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.899 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:14:37.899 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897598+0000", 2024-04-20T17:14:37.899 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.899 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.899 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.900 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:14:37.900 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897598+0000", 2024-04-20T17:14:37.900 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.900 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.900 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.900 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:14:37.900 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897598+0000", 2024-04-20T17:14:37.900 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6300000000000001e-07 2024-04-20T17:14:37.900 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.900 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.900 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:14:37.900 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897600+0000", 2024-04-20T17:14:37.900 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.451e-06 2024-04-20T17:14:37.900 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.901 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.901 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:14:37.901 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897603+0000", 2024-04-20T17:14:37.901 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9950000000000001e-06 2024-04-20T17:14:37.901 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.901 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.901 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:14:37.901 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897605+0000", 2024-04-20T17:14:37.901 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6400000000000001e-06 2024-04-20T17:14:37.901 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.901 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.901 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.901 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.899129+0000", 2024-04-20T17:14:37.901 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001523567 2024-04-20T17:14:37.901 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.902 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.902 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:14:37.902 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.899136+0000", 2024-04-20T17:14:37.902 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4880000000000001e-06 2024-04-20T17:14:37.902 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.902 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.902 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.902 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.602930+0000", 2024-04-20T17:14:37.902 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.703793365 2024-04-20T17:14:37.902 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.902 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.902 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:14:37.902 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.603007+0000", 2024-04-20T17:14:37.902 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7519000000000001e-05 2024-04-20T17:14:37.902 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.903 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.903 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:14:37.903 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.603094+0000", 2024-04-20T17:14:37.903 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6271000000000001e-05 2024-04-20T17:14:37.903 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.903 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.903 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:14:37.903 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.613185+0000", 2024-04-20T17:14:37.903 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010091126000000001 2024-04-20T17:14:37.903 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.903 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.903 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:14:37.903 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.630457+0000", 2024-04-20T17:14:37.903 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.017272526 2024-04-20T17:14:37.903 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.904 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.904 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:14:37.904 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.630522+0000", 2024-04-20T17:14:37.904 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4422000000000001e-05 2024-04-20T17:14:37.904 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.904 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.904 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:14:37.904 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.630577+0000", 2024-04-20T17:14:37.904 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5562000000000001e-05 2024-04-20T17:14:37.904 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.904 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:14:37.904 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.904 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.904 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.904 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212665 2.e 2:734f527b:::benchmark_data_smithi171_78918_object51671:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:14:37.904 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897609+0000", 2024-04-20T17:14:37.905 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 134.526713788, 2024-04-20T17:14:37.905 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.73311006000000001, 2024-04-20T17:14:37.905 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:14:37.905 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:14:37.905 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:37.905 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:14:37.905 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:14:37.905 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:14:37.905 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212665 2024-04-20T17:14:37.905 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.905 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:14:37.905 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.905 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:14:37.905 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897609+0000", 2024-04-20T17:14:37.905 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.906 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.906 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.906 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:14:37.906 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897609+0000", 2024-04-20T17:14:37.906 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.906 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.906 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.906 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:14:37.906 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897610+0000", 2024-04-20T17:14:37.906 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4400000000000001e-07 2024-04-20T17:14:37.906 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.906 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.906 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:14:37.906 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897612+0000", 2024-04-20T17:14:37.906 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.824e-06 2024-04-20T17:14:37.907 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.907 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.907 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:14:37.907 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897618+0000", 2024-04-20T17:14:37.907 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4330000000000001e-06 2024-04-20T17:14:37.907 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.907 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.907 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:14:37.907 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897621+0000", 2024-04-20T17:14:37.907 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9359999999999999e-06 2024-04-20T17:14:37.907 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.907 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.907 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.907 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.899161+0000", 2024-04-20T17:14:37.907 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001540198 2024-04-20T17:14:37.908 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.908 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.908 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:14:37.908 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.899169+0000", 2024-04-20T17:14:37.908 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4989999999999997e-06 2024-04-20T17:14:37.908 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.908 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.908 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.908 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.605497+0000", 2024-04-20T17:14:37.908 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.70632853799999995 2024-04-20T17:14:37.908 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.908 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.908 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:14:37.908 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.605602+0000", 2024-04-20T17:14:37.908 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000105147 2024-04-20T17:14:37.908 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.909 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.909 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:14:37.909 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.605676+0000", 2024-04-20T17:14:37.909 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4098999999999996e-05 2024-04-20T17:14:37.909 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.909 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.909 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:14:37.909 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.613193+0000", 2024-04-20T17:14:37.909 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0075168830000000002 2024-04-20T17:14:37.909 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.909 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.909 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:14:37.909 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.630633+0000", 2024-04-20T17:14:37.909 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.017440124000000001 2024-04-20T17:14:37.909 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.910 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.910 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:14:37.910 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.630681+0000", 2024-04-20T17:14:37.910 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7673999999999998e-05 2024-04-20T17:14:37.910 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.910 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.910 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:14:37.910 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.630719+0000", 2024-04-20T17:14:37.910 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8260999999999998e-05 2024-04-20T17:14:37.910 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.910 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:14:37.910 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.910 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.910 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.911 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212685 2.e 2:7acb7d43:::benchmark_data_smithi171_78918_object51691:head [delete] snapc 0=[] ondisk+write+known_if_redirected e203)", 2024-04-20T17:14:37.911 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:23.604266+0000", 2024-04-20T17:14:37.911 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 133.82005715599999, 2024-04-20T17:14:37.911 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.026602213, 2024-04-20T17:14:37.911 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:14:37.911 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:14:37.911 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:37.911 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:14:37.911 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:14:37.911 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:14:37.911 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212685 2024-04-20T17:14:37.911 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.911 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:14:37.911 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.911 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:14:37.911 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.604266+0000", 2024-04-20T17:14:37.912 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.912 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.912 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.912 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:14:37.912 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.604266+0000", 2024-04-20T17:14:37.912 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.912 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.912 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.912 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:14:37.912 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.604269+0000", 2024-04-20T17:14:37.912 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1379999999999999e-06 2024-04-20T17:14:37.912 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.912 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.912 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:14:37.912 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.604277+0000", 2024-04-20T17:14:37.913 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4209999999999995e-06 2024-04-20T17:14:37.913 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.913 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.913 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:14:37.913 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.604298+0000", 2024-04-20T17:14:37.913 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0041999999999999e-05 2024-04-20T17:14:37.913 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.913 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.913 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:14:37.913 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.604308+0000", 2024-04-20T17:14:37.913 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0392e-05 2024-04-20T17:14:37.913 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.913 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.913 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.913 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609525+0000", 2024-04-20T17:14:37.914 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0052173369999999998 2024-04-20T17:14:37.914 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.914 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.914 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:14:37.914 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609662+0000", 2024-04-20T17:14:37.914 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013646400000000001 2024-04-20T17:14:37.914 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.914 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.914 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:14:37.914 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609758+0000", 2024-04-20T17:14:37.914 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6510999999999998e-05 2024-04-20T17:14:37.914 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.914 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.914 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:14:37.914 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.613348+0000", 2024-04-20T17:14:37.915 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0035896919999999998 2024-04-20T17:14:37.915 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.915 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.915 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:14:37.915 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.630768+0000", 2024-04-20T17:14:37.915 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.017419752 2024-04-20T17:14:37.915 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.915 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.915 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:14:37.915 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.630825+0000", 2024-04-20T17:14:37.915 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6937000000000001e-05 2024-04-20T17:14:37.915 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.915 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.915 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:14:37.916 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.630868+0000", 2024-04-20T17:14:37.916 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3526999999999998e-05 2024-04-20T17:14:37.916 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.916 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:14:37.916 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.916 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.916 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.916 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:224633 2.e 2:74b55b25:::benchmark_data_smithi171_78918_object63639:head [delete] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-20T17:14:37.916 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:13.474420+0000", 2024-04-20T17:14:37.916 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 83.949902965000007, 2024-04-20T17:14:37.916 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014516803, 2024-04-20T17:14:37.916 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:14:37.916 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:14:37.916 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:37.916 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:14:37.917 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:14:37.917 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:14:37.917 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 224633 2024-04-20T17:14:37.917 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.917 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:14:37.917 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.917 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:14:37.917 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.474420+0000", 2024-04-20T17:14:37.917 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.917 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.917 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.917 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:14:37.917 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.474420+0000", 2024-04-20T17:14:37.917 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.917 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.918 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.918 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:14:37.918 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.474421+0000", 2024-04-20T17:14:37.918 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.142e-06 2024-04-20T17:14:37.918 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.918 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.918 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:14:37.918 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.474426+0000", 2024-04-20T17:14:37.918 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9189999999999998e-06 2024-04-20T17:14:37.918 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.918 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.918 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:14:37.918 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.474435+0000", 2024-04-20T17:14:37.918 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6589999999999996e-06 2024-04-20T17:14:37.919 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.919 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.919 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:14:37.919 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.474441+0000", 2024-04-20T17:14:37.919 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8279999999999999e-06 2024-04-20T17:14:37.919 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.919 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.919 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.919 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.476924+0000", 2024-04-20T17:14:37.919 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0024833659999999999 2024-04-20T17:14:37.919 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.919 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.919 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:14:37.919 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.477060+0000", 2024-04-20T17:14:37.919 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013633300000000001 2024-04-20T17:14:37.920 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.920 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.920 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:14:37.920 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.477195+0000", 2024-04-20T17:14:37.920 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013424800000000001 2024-04-20T17:14:37.920 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.920 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.920 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:14:37.920 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.485883+0000", 2024-04-20T17:14:37.920 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0086886370000000008 2024-04-20T17:14:37.920 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.920 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.920 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:14:37.920 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.488839+0000", 2024-04-20T17:14:37.920 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0029559109999999999 2024-04-20T17:14:37.921 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.921 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.921 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:14:37.921 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.488892+0000", 2024-04-20T17:14:37.921 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2732000000000003e-05 2024-04-20T17:14:37.921 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.921 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.921 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:14:37.921 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.488937+0000", 2024-04-20T17:14:37.921 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5028000000000003e-05 2024-04-20T17:14:37.921 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.921 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:14:37.921 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.921 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.921 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.922 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:224636 2.e 2:730710de:::benchmark_data_smithi171_78918_object63642:head [delete] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-20T17:14:37.922 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:13.479254+0000", 2024-04-20T17:14:37.922 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 83.945069235999995, 2024-04-20T17:14:37.922 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014410431, 2024-04-20T17:14:37.922 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:14:37.922 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:14:37.922 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:37.922 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:14:37.922 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:14:37.922 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:14:37.922 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 224636 2024-04-20T17:14:37.922 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.922 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:14:37.922 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.922 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:14:37.923 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.479254+0000", 2024-04-20T17:14:37.923 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.923 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.923 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.923 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:14:37.923 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.479254+0000", 2024-04-20T17:14:37.923 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.923 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.923 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.923 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:14:37.923 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.479256+0000", 2024-04-20T17:14:37.923 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.108e-06 2024-04-20T17:14:37.923 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.923 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.923 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:14:37.923 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.479261+0000", 2024-04-20T17:14:37.924 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3190000000000002e-06 2024-04-20T17:14:37.924 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.924 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.924 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:14:37.924 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.479272+0000", 2024-04-20T17:14:37.924 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0793e-05 2024-04-20T17:14:37.924 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.924 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.924 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:14:37.924 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.479293+0000", 2024-04-20T17:14:37.924 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0996e-05 2024-04-20T17:14:37.924 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.924 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.924 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.924 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.482880+0000", 2024-04-20T17:14:37.924 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003587186 2024-04-20T17:14:37.925 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.925 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.925 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:14:37.925 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.483029+0000", 2024-04-20T17:14:37.925 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014848300000000001 2024-04-20T17:14:37.925 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.925 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.925 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:14:37.925 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.483174+0000", 2024-04-20T17:14:37.925 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0001451 2024-04-20T17:14:37.925 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.925 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.925 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:14:37.925 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.485892+0000", 2024-04-20T17:14:37.925 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0027186369999999999 2024-04-20T17:14:37.925 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.926 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.926 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:14:37.926 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.493550+0000", 2024-04-20T17:14:37.926 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0076577839999999999 2024-04-20T17:14:37.926 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.926 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.926 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:14:37.926 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.493613+0000", 2024-04-20T17:14:37.926 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2966000000000007e-05 2024-04-20T17:14:37.926 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.926 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.926 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:14:37.926 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.493664+0000", 2024-04-20T17:14:37.926 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1059e-05 2024-04-20T17:14:37.926 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.927 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:14:37.927 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.927 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.927 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.927 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:228856 2.e 2:7f266611:::benchmark_data_smithi171_78918_object67862:head [delete] snapc 0=[] ondisk+write+known_if_redirected e229)", 2024-04-20T17:14:37.927 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:33.475269+0000", 2024-04-20T17:14:37.927 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 63.949054408000002, 2024-04-20T17:14:37.927 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013188525, 2024-04-20T17:14:37.927 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:14:37.927 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:14:37.927 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:14:37.927 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:14:37.927 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:14:37.927 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:14:37.927 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 228856 2024-04-20T17:14:37.928 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.928 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:14:37.928 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.928 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:14:37.928 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.475269+0000", 2024-04-20T17:14:37.928 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.928 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.928 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.928 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:14:37.928 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.475269+0000", 2024-04-20T17:14:37.928 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:14:37.928 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.928 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.928 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:14:37.928 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.475272+0000", 2024-04-20T17:14:37.928 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8099999999999999e-06 2024-04-20T17:14:37.929 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.929 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.929 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:14:37.929 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.475290+0000", 2024-04-20T17:14:37.929 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7537000000000002e-05 2024-04-20T17:14:37.929 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.929 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.929 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:14:37.929 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.475315+0000", 2024-04-20T17:14:37.929 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.529e-05 2024-04-20T17:14:37.929 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.929 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.929 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:14:37.929 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.475328+0000", 2024-04-20T17:14:37.929 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2449999999999999e-05 2024-04-20T17:14:37.929 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.930 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.930 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:14:37.930 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.475397+0000", 2024-04-20T17:14:37.930 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9575999999999999e-05 2024-04-20T17:14:37.930 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.930 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.930 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:14:37.930 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.475556+0000", 2024-04-20T17:14:37.930 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015880500000000001 2024-04-20T17:14:37.930 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.930 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.930 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:14:37.930 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.475706+0000", 2024-04-20T17:14:37.930 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000149942 2024-04-20T17:14:37.930 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.930 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.931 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:14:37.931 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.479629+0000", 2024-04-20T17:14:37.931 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0039224359999999996 2024-04-20T17:14:37.931 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.931 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.931 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:14:37.931 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.488372+0000", 2024-04-20T17:14:37.931 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0087433340000000002 2024-04-20T17:14:37.931 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.931 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.931 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:14:37.931 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.488424+0000", 2024-04-20T17:14:37.931 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2454000000000003e-05 2024-04-20T17:14:37.931 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:14:37.931 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:14:37.931 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:14:37.932 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.488457+0000", 2024-04-20T17:14:37.932 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2891000000000001e-05 2024-04-20T17:14:37.932 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.932 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:14:37.932 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.932 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:14:37.932 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:14:37.932 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:14:37.932 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:14:37.959 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:37.979 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:14:37.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:37.982 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:14:38.083 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:14:38.109 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:38.127 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:14:38.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:38.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:14:38.230 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:14:38.260 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:38.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:38 smithi005 ceph-mon[39562]: 2.7 scrub starts 2024-04-20T17:14:38.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:38 smithi005 ceph-mon[39562]: 2.7 scrub ok 2024-04-20T17:14:38.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:38 smithi106 ceph-mon[44423]: 2.7 scrub starts 2024-04-20T17:14:38.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:38 smithi106 ceph-mon[44423]: 2.7 scrub ok 2024-04-20T17:14:38.512 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:14:38.512 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:14:38.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:38.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:14:38.541 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-20T17:14:38.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:38 smithi171 ceph-mon[44189]: 2.7 scrub starts 2024-04-20T17:14:38.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:38 smithi171 ceph-mon[44189]: 2.7 scrub ok 2024-04-20T17:14:38.618 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:14:38.645 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:38.665 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:14:38.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:38.668 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:14:38.769 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:14:38.797 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:38.840 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:14:38.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:38.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:14:38.947 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:14:38.973 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:38.992 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:14:38.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:38.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:14:39.096 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:14:39.122 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:39.144 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:14:39.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:39.148 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:14:39.249 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:14:39.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:39 smithi106 ceph-mon[44423]: pgmap v697: 17 pgs: 1 active+recovering, 16 active+clean; 5.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 132 op/s; 9573/170020 objects misplaced (5.631%); 576 KiB/s, 9 objects/s recovering 2024-04-20T17:14:39.280 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:39.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:39 smithi005 ceph-mon[39562]: pgmap v697: 17 pgs: 1 active+recovering, 16 active+clean; 5.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 132 op/s; 9573/170020 objects misplaced (5.631%); 576 KiB/s, 9 objects/s recovering 2024-04-20T17:14:39.478 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:14:39.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:39.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:14:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:39 smithi171 ceph-mon[44189]: pgmap v697: 17 pgs: 1 active+recovering, 16 active+clean; 5.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 132 op/s; 9573/170020 objects misplaced (5.631%); 576 KiB/s, 9 objects/s recovering 2024-04-20T17:14:39.584 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:14:39.610 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:39.630 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:14:39.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:39.632 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:14:39.657 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:14:39.733 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:14:39.764 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:39.805 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:14:39.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:39.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:14:39.915 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:14:39.943 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:40.093 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:40 smithi005 ceph-mon[39562]: 2.f scrub starts 2024-04-20T17:14:40.094 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:40 smithi005 ceph-mon[39562]: 2.f scrub ok 2024-04-20T17:14:40.192 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:14:40.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:40.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:14:40.296 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:14:40.323 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:40.343 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:14:40.345 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:40 smithi171 ceph-mon[44189]: 2.f scrub starts 2024-04-20T17:14:40.345 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:40 smithi171 ceph-mon[44189]: 2.f scrub ok 2024-04-20T17:14:40.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:40.346 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:14:40.447 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:14:40.478 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:40.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:40 smithi106 ceph-mon[44423]: 2.f scrub starts 2024-04-20T17:14:40.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:40 smithi106 ceph-mon[44423]: 2.f scrub ok 2024-04-20T17:14:40.968 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:14:41.328 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:14:41.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:41.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:14:41.432 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:14:41.459 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:41.479 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:14:41.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:41.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:14:41.583 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:14:41.609 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:41.627 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:14:41.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:41.630 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:14:41.731 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:14:41.759 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:41.778 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:14:41.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:41.780 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:14:41.881 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:14:41.908 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:41.928 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:14:41.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:41.931 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:14:42.032 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:14:42.061 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:42.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:41 smithi005 ceph-mon[39562]: pgmap v698: 17 pgs: 1 active+recovering, 16 active+clean; 5.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 131 op/s; 9573/169732 objects misplaced (5.640%); 262 KiB/s, 4 objects/s recovering 2024-04-20T17:14:42.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:41 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:14:42.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:41 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:14:42.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:41 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:14:42.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:41 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:14:42.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:41 smithi005 ceph-mon[39562]: osdmap e252: 12 total, 12 up, 12 in 2024-04-20T17:14:42.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:41 smithi106 ceph-mon[44423]: pgmap v698: 17 pgs: 1 active+recovering, 16 active+clean; 5.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 131 op/s; 9573/169732 objects misplaced (5.640%); 262 KiB/s, 4 objects/s recovering 2024-04-20T17:14:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:41 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:14:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:41 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:14:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:41 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:14:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:41 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:14:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:41 smithi106 ceph-mon[44423]: osdmap e252: 12 total, 12 up, 12 in 2024-04-20T17:14:42.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:41 smithi171 ceph-mon[44189]: pgmap v698: 17 pgs: 1 active+recovering, 16 active+clean; 5.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 131 op/s; 9573/169732 objects misplaced (5.640%); 262 KiB/s, 4 objects/s recovering 2024-04-20T17:14:42.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:41 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:14:42.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:41 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:14:42.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:41 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:14:42.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:41 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:14:42.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:41 smithi171 ceph-mon[44189]: osdmap e252: 12 total, 12 up, 12 in 2024-04-20T17:14:42.785 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:14:43.276 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:14:43.279 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:14:44.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:43 smithi005 ceph-mon[39562]: pgmap v700: 17 pgs: 1 active+recovering, 16 active+clean; 5.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 128 op/s; 9493/169034 objects misplaced (5.616%); 648 KiB/s, 10 objects/s recovering 2024-04-20T17:14:44.242 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:14:44.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:44.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:14:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:43 smithi106 ceph-mon[44423]: pgmap v700: 17 pgs: 1 active+recovering, 16 active+clean; 5.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 128 op/s; 9493/169034 objects misplaced (5.616%); 648 KiB/s, 10 objects/s recovering 2024-04-20T17:14:44.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:43 smithi171 ceph-mon[44189]: pgmap v700: 17 pgs: 1 active+recovering, 16 active+clean; 5.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 128 op/s; 9493/169034 objects misplaced (5.616%); 648 KiB/s, 10 objects/s recovering 2024-04-20T17:14:44.347 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:14:44.373 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:44.392 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:14:44.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:44.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:14:44.498 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:14:44.524 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:44.543 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:14:44.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:44.546 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:14:44.566 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:14:44.599 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:14:44.624 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:14:44.646 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:14:44.648 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:14:44.675 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:44.683 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:14:44.696 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:14:44.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:44.700 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:14:44.706 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:14:44.734 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:14:44.763 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:14:44.787 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:14:44.801 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:14:44.820 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:14:44.844 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:44.852 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:14:44.880 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:14:44.882 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:14:44.882 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:14:44.882 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:14:44.883 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:14:44.883 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:14:44.883 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:14:44.883 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:14:44.883 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:14:44.883 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:14:44.883 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:14:44.883 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:14:44.883 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:14:45.201 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:44 smithi005 ceph-mon[39562]: from='client.15294 v1:172.21.15.5:0/585985636' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:14:45.201 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:44 smithi005 ceph-mon[39562]: 2.9 scrub starts 2024-04-20T17:14:45.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:44 smithi005 ceph-mon[39562]: 2.9 scrub ok 2024-04-20T17:14:45.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:44 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2106211650' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:14:45.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:44 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:45.203 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:14:45.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:45.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:14:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:44 smithi106 ceph-mon[44423]: from='client.15294 v1:172.21.15.5:0/585985636' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:14:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:44 smithi106 ceph-mon[44423]: 2.9 scrub starts 2024-04-20T17:14:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:44 smithi106 ceph-mon[44423]: 2.9 scrub ok 2024-04-20T17:14:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:44 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2106211650' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:14:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:44 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:45.308 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:14:45.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:44 smithi171 ceph-mon[44189]: from='client.15294 v1:172.21.15.5:0/585985636' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:14:45.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:44 smithi171 ceph-mon[44189]: 2.9 scrub starts 2024-04-20T17:14:45.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:44 smithi171 ceph-mon[44189]: 2.9 scrub ok 2024-04-20T17:14:45.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:44 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2106211650' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:14:45.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:44 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:45.334 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:45.352 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:14:45.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:45.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:14:45.456 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:14:45.482 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:45.500 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:14:45.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:45.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:14:45.604 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:14:45.630 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:45.648 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:14:45.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:45.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:14:45.752 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:14:45.778 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:45.796 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:14:45.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:45.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:14:45.899 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:14:45.928 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:45.947 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:14:45.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:45.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:14:46.051 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:14:46.078 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:46.095 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:14:46.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:46.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:14:46.198 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:14:46.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:45 smithi005 ceph-mon[39562]: pgmap v701: 17 pgs: 1 active+recovering, 16 active+clean; 5.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 144 op/s; 9493/168330 objects misplaced (5.640%); 320 KiB/s, 4 objects/s recovering 2024-04-20T17:14:46.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:45 smithi005 ceph-mon[39562]: 2.8 scrub starts 2024-04-20T17:14:46.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:45 smithi005 ceph-mon[39562]: 2.6 scrub starts 2024-04-20T17:14:46.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:45 smithi005 ceph-mon[39562]: 2.2 scrub starts 2024-04-20T17:14:46.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:45 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:14:46.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:45 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:14:46.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:45 smithi005 ceph-mon[39562]: 2.8 scrub ok 2024-04-20T17:14:46.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:45 smithi005 ceph-mon[39562]: osdmap e253: 12 total, 12 up, 12 in 2024-04-20T17:14:46.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:45 smithi005 ceph-mon[39562]: 2.6 scrub ok 2024-04-20T17:14:46.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:45 smithi005 ceph-mon[39562]: 2.2 scrub ok 2024-04-20T17:14:46.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:45 smithi005 ceph-mon[39562]: osdmap e254: 12 total, 12 up, 12 in 2024-04-20T17:14:46.222 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:45 smithi171 ceph-mon[44189]: pgmap v701: 17 pgs: 1 active+recovering, 16 active+clean; 5.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 144 op/s; 9493/168330 objects misplaced (5.640%); 320 KiB/s, 4 objects/s recovering 2024-04-20T17:14:46.222 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:45 smithi171 ceph-mon[44189]: 2.8 scrub starts 2024-04-20T17:14:46.222 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:45 smithi171 ceph-mon[44189]: 2.6 scrub starts 2024-04-20T17:14:46.222 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.222 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.222 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.222 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.222 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.222 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:45 smithi171 ceph-mon[44189]: 2.2 scrub starts 2024-04-20T17:14:46.223 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.223 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:45 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:14:46.223 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:45 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:14:46.223 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.223 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.223 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.223 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.223 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.223 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:45 smithi171 ceph-mon[44189]: 2.8 scrub ok 2024-04-20T17:14:46.223 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:45 smithi171 ceph-mon[44189]: osdmap e253: 12 total, 12 up, 12 in 2024-04-20T17:14:46.223 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:45 smithi171 ceph-mon[44189]: 2.6 scrub ok 2024-04-20T17:14:46.223 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:45 smithi171 ceph-mon[44189]: 2.2 scrub ok 2024-04-20T17:14:46.223 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:45 smithi171 ceph-mon[44189]: osdmap e254: 12 total, 12 up, 12 in 2024-04-20T17:14:46.224 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:46.242 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:14:46.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:46.245 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:14:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:45 smithi106 ceph-mon[44423]: pgmap v701: 17 pgs: 1 active+recovering, 16 active+clean; 5.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 144 op/s; 9493/168330 objects misplaced (5.640%); 320 KiB/s, 4 objects/s recovering 2024-04-20T17:14:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:45 smithi106 ceph-mon[44423]: 2.8 scrub starts 2024-04-20T17:14:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:45 smithi106 ceph-mon[44423]: 2.6 scrub starts 2024-04-20T17:14:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:45 smithi106 ceph-mon[44423]: 2.2 scrub starts 2024-04-20T17:14:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:45 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:14:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:45 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:14:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:46.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:45 smithi106 ceph-mon[44423]: 2.8 scrub ok 2024-04-20T17:14:46.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:45 smithi106 ceph-mon[44423]: osdmap e253: 12 total, 12 up, 12 in 2024-04-20T17:14:46.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:45 smithi106 ceph-mon[44423]: 2.6 scrub ok 2024-04-20T17:14:46.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:45 smithi106 ceph-mon[44423]: 2.2 scrub ok 2024-04-20T17:14:46.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:45 smithi106 ceph-mon[44423]: osdmap e254: 12 total, 12 up, 12 in 2024-04-20T17:14:46.345 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:14:46.372 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:46.391 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:14:46.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:46.394 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:14:46.469 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":700,"stamp":"2024-04-20T17:14:42.028319+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5539299896,"num_objects":84517,"num_object_clones":0,"num_object_copies":169034,"num_objects_missing_on_primary":4740,"num_objects_missing":4740,"num_objects_degraded":0,"num_objects_misplaced":9493,"num_objects_unfound":0,"num_objects_dirty":84517,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":237650,"num_write_kb":10306363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24471,"num_bytes_recovered":1605322880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160999,"ondisk_log_size":160999,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":38,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":12506212,"kb_used_data":11449908,"kb_used_omap":219,"kb_used_meta":1055972,"kb_avail":1112435612,"statfs":{"total":1151940427776,"available":1139134066688,"internally_reserved":0,"allocated":11724705792,"data_stored":11716390152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224949,"internal_metadata":1081315659},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2,1,0,0,3,4,6],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-67436544,"num_objects":-1029,"num_object_clones":0,"num_object_copies":-2058,"num_objects_missing_on_primary":-81,"num_objects_missing":-81,"num_objects_degraded":0,"num_objects_misplaced":-162,"num_objects_unfound":0,"num_objects_dirty":-1029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1029,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":5308416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001426"},"pg_stats":[{"pgid":"2.f","version":"251'14856","reported_seq":15532,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-20T17:14:38.099460+0000","last_change":"2024-04-20T17:13:29.675970+0000","last_active":"2024-04-20T17:14:38.099460+0000","last_peered":"2024-04-20T17:14:38.099460+0000","last_clean":"2024-04-20T17:14:38.099460+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:14:38.099460+0000","last_undegraded":"2024-04-20T17:14:38.099460+0000","last_fullsized":"2024-04-20T17:14:38.099460+0000","mapping_epoch":115,"log_start":"128'4800","ondisk_log_start":"128'4800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'14266","last_scrub_stamp":"2024-04-20T17:13:29.675870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:13:29.675870+0000","objects_scrubbed":5916,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349044736,"num_objects":5326,"num_object_clones":0,"num_object_copies":10652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14856,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"251'14799","reported_seq":26550,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-20T17:14:40.580915+0000","last_change":"2024-04-20T17:14:02.868984+0000","last_active":"2024-04-20T17:14:40.580915+0000","last_peered":"2024-04-20T17:14:40.580915+0000","last_clean":"2024-04-20T17:14:40.580915+0000","last_became_active":"2024-04-20T17:10:28.865670+0000","last_became_peered":"2024-04-20T17:10:28.865670+0000","last_unstale":"2024-04-20T17:14:40.580915+0000","last_undegraded":"2024-04-20T17:14:40.580915+0000","last_fullsized":"2024-04-20T17:14:40.580915+0000","mapping_epoch":162,"log_start":"127'4700","ondisk_log_start":"127'4700","created":115,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'14480","last_scrub_stamp":"2024-04-20T17:14:02.868843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:02.868843+0000","objects_scrubbed":5576,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:07.951248+0000","scrub_duration":456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344522752,"num_objects":5257,"num_object_clones":0,"num_object_copies":10514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14799,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5619,"num_bytes_recovered":368246784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"251'15089","reported_seq":15604,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-20T17:14:40.089639+0000","last_change":"2024-04-20T17:13:26.941209+0000","last_active":"2024-04-20T17:14:40.089639+0000","last_peered":"2024-04-20T17:14:40.089639+0000","last_clean":"2024-04-20T17:14:40.089639+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:14:40.089639+0000","last_undegraded":"2024-04-20T17:14:40.089639+0000","last_fullsized":"2024-04-20T17:14:40.089639+0000","mapping_epoch":115,"log_start":"128'5000","ondisk_log_start":"128'5000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'14487","last_scrub_stamp":"2024-04-20T17:13:26.941068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:13:26.941068+0000","objects_scrubbed":5996,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":506,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353435648,"num_objects":5393,"num_object_clones":0,"num_object_copies":10786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15089,"num_write_kb":655424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"251'21530","reported_seq":42854,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-20T17:14:40.615425+0000","last_change":"2024-04-20T17:13:59.403959+0000","last_active":"2024-04-20T17:14:40.615425+0000","last_peered":"2024-04-20T17:14:40.615425+0000","last_clean":"2024-04-20T17:14:40.615425+0000","last_became_active":"2024-04-20T17:12:08.085936+0000","last_became_peered":"2024-04-20T17:12:08.085936+0000","last_unstale":"2024-04-20T17:14:40.615425+0000","last_undegraded":"2024-04-20T17:14:40.615425+0000","last_fullsized":"2024-04-20T17:14:40.615425+0000","mapping_epoch":196,"log_start":"157'11468","ondisk_log_start":"157'11468","created":115,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'21205","last_scrub_stamp":"2024-04-20T17:13:59.403804+0000","last_deep_scrub":"213'20373","last_deep_scrub_stamp":"2024-04-20T17:12:48.443808+0000","last_clean_scrub_stamp":"2024-04-20T17:13:59.403804+0000","objects_scrubbed":5682,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:06.827884+0000","scrub_duration":478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351010816,"num_objects":5356,"num_object_clones":0,"num_object_copies":10712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14878,"num_write_kb":647488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":450166784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"251'14547","reported_seq":28884,"reported_epoch":251,"state":"active+recovering","last_fresh":"2024-04-20T17:14:39.989721+0000","last_change":"2024-04-20T17:12:26.166109+0000","last_active":"2024-04-20T17:14:39.989721+0000","last_peered":"2024-04-20T17:14:39.989721+0000","last_clean":"2024-04-20T17:09:35.882603+0000","last_became_active":"2024-04-20T17:10:54.470253+0000","last_became_peered":"2024-04-20T17:10:54.470253+0000","last_unstale":"2024-04-20T17:14:39.989721+0000","last_undegraded":"2024-04-20T17:14:39.989721+0000","last_fullsized":"2024-04-20T17:14:39.989721+0000","mapping_epoch":171,"log_start":"126'4500","ondisk_log_start":"126'4500","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339673088,"num_objects":5183,"num_object_clones":0,"num_object_copies":10366,"num_objects_missing_on_primary":4740,"num_objects_missing":4740,"num_objects_degraded":0,"num_objects_misplaced":9493,"num_objects_unfound":0,"num_objects_dirty":5183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14547,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4679,"num_bytes_recovered":306642944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"0,2,5","objects":13},{"shards":"0,10","objects":430},{"shards":"2,5","objects":4740}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":680,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-20T17:14:36.142817+0000","last_change":"2024-04-20T17:13:20.646446+0000","last_active":"2024-04-20T17:14:36.142817+0000","last_peered":"2024-04-20T17:14:36.142817+0000","last_clean":"2024-04-20T17:14:36.142817+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:14:36.142817+0000","last_undegraded":"2024-04-20T17:14:36.142817+0000","last_fullsized":"2024-04-20T17:14:36.142817+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:13:20.646341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:13:20.646341+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:14:37.378418+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"251'14901","reported_seq":15506,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-20T17:14:37.189275+0000","last_change":"2024-04-20T17:14:01.843280+0000","last_active":"2024-04-20T17:14:37.189275+0000","last_peered":"2024-04-20T17:14:37.189275+0000","last_clean":"2024-04-20T17:14:37.189275+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:14:37.189275+0000","last_undegraded":"2024-04-20T17:14:37.189275+0000","last_fullsized":"2024-04-20T17:14:37.189275+0000","mapping_epoch":115,"log_start":"128'4900","ondisk_log_start":"128'4900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'14629","last_scrub_stamp":"2024-04-20T17:14:01.843179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:01.843179+0000","objects_scrubbed":5626,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:24.160162+0000","scrub_duration":499,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350683136,"num_objects":5351,"num_object_clones":0,"num_object_copies":10702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14901,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"251'14993","reported_seq":15599,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-20T17:14:38.108136+0000","last_change":"2024-04-20T17:14:03.619324+0000","last_active":"2024-04-20T17:14:38.108136+0000","last_peered":"2024-04-20T17:14:38.108136+0000","last_clean":"2024-04-20T17:14:38.108136+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:14:38.108136+0000","last_undegraded":"2024-04-20T17:14:38.108136+0000","last_fullsized":"2024-04-20T17:14:38.108136+0000","mapping_epoch":115,"log_start":"128'4900","ondisk_log_start":"128'4900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'14687","last_scrub_stamp":"2024-04-20T17:14:03.619176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:03.619176+0000","objects_scrubbed":5637,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:17.353819+0000","scrub_duration":558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349372416,"num_objects":5331,"num_object_clones":0,"num_object_copies":10662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14993,"num_write_kb":650368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"252'18994","reported_seq":19695,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-04-20T17:14:41.727009+0000","last_change":"2024-04-20T17:13:12.813364+0000","last_active":"2024-04-20T17:14:41.727009+0000","last_peered":"2024-04-20T17:14:41.727009+0000","last_clean":"2024-04-20T17:14:41.727009+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:14:41.727009+0000","last_undegraded":"2024-04-20T17:14:41.727009+0000","last_fullsized":"2024-04-20T17:14:41.727009+0000","mapping_epoch":115,"log_start":"144'8902","ondisk_log_start":"144'8902","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'18183","last_scrub_stamp":"2024-04-20T17:13:12.813195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:13:12.813195+0000","objects_scrubbed":5972,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":588,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338100224,"num_objects":5159,"num_object_clones":0,"num_object_copies":10318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14883,"num_write_kb":641344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"251'14984","reported_seq":15648,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-20T17:14:37.447231+0000","last_change":"2024-04-20T17:14:37.076566+0000","last_active":"2024-04-20T17:14:37.447231+0000","last_peered":"2024-04-20T17:14:37.447231+0000","last_clean":"2024-04-20T17:14:37.447231+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:14:37.447231+0000","last_undegraded":"2024-04-20T17:14:37.447231+0000","last_fullsized":"2024-04-20T17:14:37.447231+0000","mapping_epoch":115,"log_start":"128'4900","ondisk_log_start":"128'4900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'14978","last_scrub_stamp":"2024-04-20T17:14:37.076332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:37.076332+0000","objects_scrubbed":5242,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:40.993014+0000","scrub_duration":400,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343146496,"num_objects":5236,"num_object_clones":0,"num_object_copies":10472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14984,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"251'14739","reported_seq":15270,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-20T17:14:37.421110+0000","last_change":"2024-04-20T17:13:20.234492+0000","last_active":"2024-04-20T17:14:37.421110+0000","last_peered":"2024-04-20T17:14:37.421110+0000","last_clean":"2024-04-20T17:14:37.421110+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:14:37.421110+0000","last_undegraded":"2024-04-20T17:14:37.421110+0000","last_fullsized":"2024-04-20T17:14:37.421110+0000","mapping_epoch":115,"log_start":"127'4700","ondisk_log_start":"127'4700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'14047","last_scrub_stamp":"2024-04-20T17:13:20.234393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:13:20.234393+0000","objects_scrubbed":6012,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:14:37.974568+0000","scrub_duration":621,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348192768,"num_objects":5313,"num_object_clones":0,"num_object_copies":10626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14739,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"251'14906","reported_seq":30044,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-20T17:14:37.498908+0000","last_change":"2024-04-20T17:13:16.153058+0000","last_active":"2024-04-20T17:14:37.498908+0000","last_peered":"2024-04-20T17:14:37.498908+0000","last_clean":"2024-04-20T17:14:37.498908+0000","last_became_active":"2024-04-20T17:10:55.931751+0000","last_became_peered":"2024-04-20T17:10:55.931751+0000","last_unstale":"2024-04-20T17:14:37.498908+0000","last_undegraded":"2024-04-20T17:14:37.498908+0000","last_fullsized":"2024-04-20T17:14:37.498908+0000","mapping_epoch":173,"log_start":"128'4900","ondisk_log_start":"128'4900","created":115,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'14137","last_scrub_stamp":"2024-04-20T17:13:16.152960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:13:16.152960+0000","objects_scrubbed":6129,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351010816,"num_objects":5356,"num_object_clones":0,"num_object_copies":10712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14906,"num_write_kb":648384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7296,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"251'15015","reported_seq":15617,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-20T17:14:37.442001+0000","last_change":"2024-04-20T17:14:03.230824+0000","last_active":"2024-04-20T17:14:37.442001+0000","last_peered":"2024-04-20T17:14:37.442001+0000","last_clean":"2024-04-20T17:14:37.442001+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:14:37.442001+0000","last_undegraded":"2024-04-20T17:14:37.442001+0000","last_fullsized":"2024-04-20T17:14:37.442001+0000","mapping_epoch":115,"log_start":"128'5000","ondisk_log_start":"128'5000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'14739","last_scrub_stamp":"2024-04-20T17:14:03.230712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:03.230712+0000","objects_scrubbed":5578,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:11.279553+0000","scrub_duration":546,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347406336,"num_objects":5301,"num_object_clones":0,"num_object_copies":10602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15015,"num_write_kb":650112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"251'14605","reported_seq":15206,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-20T17:14:38.705563+0000","last_change":"2024-04-20T17:14:05.574878+0000","last_active":"2024-04-20T17:14:38.705563+0000","last_peered":"2024-04-20T17:14:38.705563+0000","last_clean":"2024-04-20T17:14:38.705563+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:14:38.705563+0000","last_undegraded":"2024-04-20T17:14:38.705563+0000","last_fullsized":"2024-04-20T17:14:38.705563+0000","mapping_epoch":115,"log_start":"127'4600","ondisk_log_start":"127'4600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'14330","last_scrub_stamp":"2024-04-20T17:14:05.574772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:05.574772+0000","objects_scrubbed":5536,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:05.590167+0000","scrub_duration":494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344653824,"num_objects":5259,"num_object_clones":0,"num_object_copies":10518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14605,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"251'14877","reported_seq":15478,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-20T17:14:37.396145+0000","last_change":"2024-04-20T17:14:04.200256+0000","last_active":"2024-04-20T17:14:37.396145+0000","last_peered":"2024-04-20T17:14:37.396145+0000","last_clean":"2024-04-20T17:14:37.396145+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:14:37.396145+0000","last_undegraded":"2024-04-20T17:14:37.396145+0000","last_fullsized":"2024-04-20T17:14:37.396145+0000","mapping_epoch":115,"log_start":"127'4800","ondisk_log_start":"127'4800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'14590","last_scrub_stamp":"2024-04-20T17:14:04.200147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:04.200147+0000","objects_scrubbed":5516,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:21.444062+0000","scrub_duration":506,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342425600,"num_objects":5225,"num_object_clones":0,"num_object_copies":10450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14877,"num_write_kb":643264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"251'14673","reported_seq":15274,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-20T17:14:40.520771+0000","last_change":"2024-04-20T17:14:12.420994+0000","last_active":"2024-04-20T17:14:40.520771+0000","last_peered":"2024-04-20T17:14:40.520771+0000","last_clean":"2024-04-20T17:14:40.520771+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:14:40.520771+0000","last_undegraded":"2024-04-20T17:14:40.520771+0000","last_fullsized":"2024-04-20T17:14:40.520771+0000","mapping_epoch":115,"log_start":"126'4600","ondisk_log_start":"126'4600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'14443","last_scrub_stamp":"2024-04-20T17:14:12.420702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:12.420702+0000","objects_scrubbed":5431,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:19.358247+0000","scrub_duration":478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340525080,"num_objects":5197,"num_object_clones":0,"num_object_copies":10394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5197,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14673,"num_write_kb":635777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"251'14724","reported_seq":15191,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-04-20T17:14:40.576131+0000","last_change":"2024-04-20T17:13:58.682929+0000","last_active":"2024-04-20T17:14:40.576131+0000","last_peered":"2024-04-20T17:14:40.576131+0000","last_clean":"2024-04-20T17:14:40.576131+0000","last_became_active":"2024-04-20T17:12:23.515375+0000","last_became_peered":"2024-04-20T17:12:23.515375+0000","last_unstale":"2024-04-20T17:14:40.576131+0000","last_undegraded":"2024-04-20T17:14:40.576131+0000","last_fullsized":"2024-04-20T17:14:40.576131+0000","mapping_epoch":202,"log_start":"127'4700","ondisk_log_start":"127'4700","created":115,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'14359","last_scrub_stamp":"2024-04-20T17:13:58.682797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:13:58.682797+0000","objects_scrubbed":5638,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:14:59.113672+0000","scrub_duration":533,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345505792,"num_objects":5272,"num_object_clones":0,"num_object_copies":10544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14724,"num_write_kb":639872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":5538709528,"num_objects":84515,"num_object_clones":0,"num_object_copies":169030,"num_objects_missing_on_primary":4740,"num_objects_missing":4740,"num_objects_degraded":0,"num_objects_misplaced":9493,"num_objects_unfound":0,"num_objects_dirty":84515,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":237469,"num_write_kb":10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24463,"num_bytes_recovered":1603207168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11697201152,"data_stored":11697193008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},"log_size":160862,"ondisk_log_size":160862,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991339,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1084352,"kb_used_data":1012916,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92660800,"statfs":{"total":95995035648,"available":94884659200,"internally_reserved":0,"allocated":1037225984,"data_stored":1036531210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":73136580},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777252977,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1190964,"kb_used_data":1047340,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92554188,"statfs":{"total":95995035648,"available":94775488512,"internally_reserved":0,"allocated":1072476160,"data_stored":1071784938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":147061184},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612727,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1485560,"kb_used_data":1355824,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92259592,"statfs":{"total":95995035648,"available":94473822208,"internally_reserved":0,"allocated":1388363776,"data_stored":1387668502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":132839872},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762808956,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1455860,"kb_used_data":1352492,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92289292,"statfs":{"total":95995035648,"available":94504235008,"internally_reserved":0,"allocated":1384951808,"data_stored":1384260606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168705,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1090432,"kb_used_data":1027316,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92654720,"statfs":{"total":95995035648,"available":94878433280,"internally_reserved":0,"allocated":1051971584,"data_stored":1051276830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430341,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":763192,"kb_used_data":682028,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":92981960,"statfs":{"total":95995035648,"available":95213527040,"internally_reserved":0,"allocated":698396672,"data_stored":697705470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":83098054},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575008,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":777348,"kb_used_data":687864,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":92967804,"statfs":{"total":95995035648,"available":95199031296,"internally_reserved":0,"allocated":704372736,"data_stored":703675606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738533,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1148340,"kb_used_data":1040812,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92596812,"statfs":{"total":95995035648,"available":94819135488,"internally_reserved":0,"allocated":1065791488,"data_stored":1065100286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2,1,0,0,3,4,6],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393821,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":988020,"kb_used_data":943788,"kb_used_omap":202,"kb_used_meta":44021,"kb_avail":92757132,"statfs":{"total":95995035648,"available":94983303168,"internally_reserved":0,"allocated":966438912,"data_stored":965747710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":45077975},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182549,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":757108,"kb_used_data":677356,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":92988044,"statfs":{"total":95995035648,"available":95219757056,"internally_reserved":0,"allocated":693612544,"data_stored":692921342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":81656263},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986320,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":369912,"kb_used_data":334576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93375240,"statfs":{"total":95995035648,"available":95616245760,"internally_reserved":0,"allocated":342605824,"data_stored":341910550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790090,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1395124,"kb_used_data":1287596,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92350028,"statfs":{"total":95995035648,"available":94566428672,"internally_reserved":0,"allocated":1318498304,"data_stored":1317807102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1063649280,"data_stored":1063649280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":702545920,"data_stored":702545920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":962658304,"data_stored":962658304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":691011584,"data_stored":691011584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":340725760,"data_stored":340721688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1316356096,"data_stored":1316356096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":695861248,"data_stored":695861248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1049886720,"data_stored":1049886720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1382875136,"data_stored":1382875136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1385369600,"data_stored":1385365528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1070989312,"data_stored":1070989312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1035272192,"data_stored":1035272192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:14:46.470 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:14:46.495 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:14:46.522 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:46.541 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:14:46.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:46.544 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:14:46.645 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:14:46.671 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:46.689 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:14:46.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:46.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:14:46.793 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:14:46.824 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:46.982 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:14:46.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:46.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:14:47.086 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:14:47.113 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:47.132 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:14:47.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:47.135 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:14:47.159 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:46 smithi005 ceph-mon[39562]: pgmap v704: 17 pgs: 1 active+recovering, 16 active+clean; 5.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 151 op/s; 9411/167918 objects misplaced (5.605%); 853 KiB/s, 13 objects/s recovering 2024-04-20T17:14:47.236 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:14:47.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:46 smithi106 ceph-mon[44423]: pgmap v704: 17 pgs: 1 active+recovering, 16 active+clean; 5.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 151 op/s; 9411/167918 objects misplaced (5.605%); 853 KiB/s, 13 objects/s recovering 2024-04-20T17:14:47.263 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:47.282 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:14:47.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:47.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:14:47.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:46 smithi171 ceph-mon[44189]: pgmap v704: 17 pgs: 1 active+recovering, 16 active+clean; 5.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 151 op/s; 9411/167918 objects misplaced (5.605%); 853 KiB/s, 13 objects/s recovering 2024-04-20T17:14:47.385 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:14:47.415 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:47.943 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:14:48.102 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:14:48.102 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:14:48.125 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:47 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3983359377' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:14:48.125 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:47 smithi005 ceph-mon[39562]: from='client.25238 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:14:48.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:47 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3983359377' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:14:48.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:47 smithi106 ceph-mon[44423]: from='client.25238 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:14:48.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:47 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3983359377' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:14:48.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:47 smithi171 ceph-mon[44189]: from='client.25238 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:14:49.146 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:14:49.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:48 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:14:49.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:48 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:14:49.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:48 smithi005 ceph-mon[39562]: from='client.25238 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:14:49.151 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:48 smithi005 ceph-mon[39562]: osdmap e255: 12 total, 12 up, 12 in 2024-04-20T17:14:49.151 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:48 smithi005 ceph-mon[39562]: pgmap v706: 17 pgs: 1 active+recovering, 16 active+clean; 5.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 130 op/s; 9411/167464 objects misplaced (5.620%); 427 KiB/s, 6 objects/s recovering 2024-04-20T17:14:49.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:49.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:14:49.253 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:14:49.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:48 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:14:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:48 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:14:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:48 smithi106 ceph-mon[44423]: from='client.25238 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:14:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:48 smithi106 ceph-mon[44423]: osdmap e255: 12 total, 12 up, 12 in 2024-04-20T17:14:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:48 smithi106 ceph-mon[44423]: pgmap v706: 17 pgs: 1 active+recovering, 16 active+clean; 5.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 130 op/s; 9411/167464 objects misplaced (5.620%); 427 KiB/s, 6 objects/s recovering 2024-04-20T17:14:49.276 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:48 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:14:49.276 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:48 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:14:49.276 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:48 smithi171 ceph-mon[44189]: from='client.25238 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:14:49.276 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:48 smithi171 ceph-mon[44189]: osdmap e255: 12 total, 12 up, 12 in 2024-04-20T17:14:49.276 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:48 smithi171 ceph-mon[44189]: pgmap v706: 17 pgs: 1 active+recovering, 16 active+clean; 5.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 130 op/s; 9411/167464 objects misplaced (5.620%); 427 KiB/s, 6 objects/s recovering 2024-04-20T17:14:49.278 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:49.296 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:14:49.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:49.299 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:14:49.400 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:14:49.427 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:49.444 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:14:49.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:49.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:14:49.539 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:14:49.548 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:14:49.612 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:49.652 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:14:49.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:49.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:14:49.695 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-20T17:14:49.710 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:14:49.761 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:14:49.792 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:50.045 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:14:50.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:50.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:14:50.150 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:14:50.182 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:50.631 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:14:50.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:50.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:14:50.675 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:14:50.677 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:14:50.736 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:14:50.767 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:50.865 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:14:50.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:50.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:14:50.898 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:14:50.972 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:14:51.006 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:51.038 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:14:51.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:51.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:14:51.144 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:14:51.174 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:51.211 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:14:51.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:51.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:14:51.316 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:14:51.346 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:51.366 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:14:51.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:51.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:14:51.470 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:14:51.497 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:51.517 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:14:51.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:51.519 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:14:51.620 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:14:51.646 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:51.666 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:14:51.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:51.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:14:51.770 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:14:51.798 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:51.817 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:14:51.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:51.820 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:14:51.920 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:14:51.952 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:51.977 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:14:51.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:51.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:14:52.083 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:14:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:51 smithi171 ceph-mon[44189]: pgmap v707: 17 pgs: 1 active+recovering, 16 active+clean; 5.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 89 op/s; 9411/167250 objects misplaced (5.627%); 427 KiB/s, 6 objects/s recovering 2024-04-20T17:14:52.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:51 smithi171 ceph-mon[44189]: osdmap e256: 12 total, 12 up, 12 in 2024-04-20T17:14:52.114 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:52.167 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:51 smithi005 ceph-mon[39562]: pgmap v707: 17 pgs: 1 active+recovering, 16 active+clean; 5.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 89 op/s; 9411/167250 objects misplaced (5.627%); 427 KiB/s, 6 objects/s recovering 2024-04-20T17:14:52.167 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:51 smithi005 ceph-mon[39562]: osdmap e256: 12 total, 12 up, 12 in 2024-04-20T17:14:52.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:51 smithi106 ceph-mon[44423]: pgmap v707: 17 pgs: 1 active+recovering, 16 active+clean; 5.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 89 op/s; 9411/167250 objects misplaced (5.627%); 427 KiB/s, 6 objects/s recovering 2024-04-20T17:14:52.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:51 smithi106 ceph-mon[44423]: osdmap e256: 12 total, 12 up, 12 in 2024-04-20T17:14:53.412 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:14:53.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:53.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:14:53.516 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:14:53.543 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:53.562 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:14:53.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:53.565 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:14:53.666 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:14:53.692 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:53.713 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:14:53.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:53.717 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:14:53.818 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:14:53.846 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:53.864 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:14:53.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:53.867 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:14:53.968 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:14:53.994 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:54.012 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:14:54.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:54.015 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:14:54.115 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:14:54.142 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:53 smithi171 ceph-mon[44189]: pgmap v709: 17 pgs: 1 active+recovering, 16 active+clean; 5.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 93 op/s; 9331/166772 objects misplaced (5.595%); 429 KiB/s, 6 objects/s recovering 2024-04-20T17:14:54.144 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:54.165 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:14:54.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:54.168 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:14:54.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:53 smithi005 ceph-mon[39562]: pgmap v709: 17 pgs: 1 active+recovering, 16 active+clean; 5.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 93 op/s; 9331/166772 objects misplaced (5.595%); 429 KiB/s, 6 objects/s recovering 2024-04-20T17:14:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:53 smithi106 ceph-mon[44423]: pgmap v709: 17 pgs: 1 active+recovering, 16 active+clean; 5.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 93 op/s; 9331/166772 objects misplaced (5.595%); 429 KiB/s, 6 objects/s recovering 2024-04-20T17:14:54.268 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:14:54.299 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:54.353 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:14:54.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:54.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:14:54.459 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:14:54.486 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:54.504 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:14:54.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:54.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:14:54.607 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:14:54.639 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:54.801 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:14:54.803 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:14:55.188 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:14:55.191 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:54 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:14:55.191 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:54 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:14:55.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:55.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:14:55.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:54 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:14:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:54 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:14:55.293 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:14:55.326 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:55.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:54 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:14:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:54 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:14:55.924 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:55 smithi005 ceph-mon[39562]: pgmap v710: 17 pgs: 1 active+recovering, 16 active+clean; 5.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 118 op/s; 9331/166026 objects misplaced (5.620%); 328 KiB/s, 5 objects/s recovering 2024-04-20T17:14:55.924 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:55 smithi005 ceph-mon[39562]: from='client.15312 v1:172.21.15.5:0/1053101955' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:14:55.924 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:55 smithi005 ceph-mon[39562]: osdmap e257: 12 total, 12 up, 12 in 2024-04-20T17:14:55.924 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:55 smithi005 ceph-mon[39562]: osdmap e258: 12 total, 12 up, 12 in 2024-04-20T17:14:55.979 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:14:55.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:55.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:14:56.083 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:14:56.110 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:56.117 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:14:56.118 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:14:56.128 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:14:56.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:56.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:14:56.233 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:14:56.257 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:55 smithi106 ceph-mon[44423]: pgmap v710: 17 pgs: 1 active+recovering, 16 active+clean; 5.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 118 op/s; 9331/166026 objects misplaced (5.620%); 328 KiB/s, 5 objects/s recovering 2024-04-20T17:14:56.257 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:55 smithi106 ceph-mon[44423]: from='client.15312 v1:172.21.15.5:0/1053101955' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:14:56.257 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:55 smithi106 ceph-mon[44423]: osdmap e257: 12 total, 12 up, 12 in 2024-04-20T17:14:56.257 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:55 smithi106 ceph-mon[44423]: osdmap e258: 12 total, 12 up, 12 in 2024-04-20T17:14:56.259 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:56.278 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:14:56.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:56.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:14:56.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:55 smithi171 ceph-mon[44189]: pgmap v710: 17 pgs: 1 active+recovering, 16 active+clean; 5.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 118 op/s; 9331/166026 objects misplaced (5.620%); 328 KiB/s, 5 objects/s recovering 2024-04-20T17:14:56.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:55 smithi171 ceph-mon[44189]: from='client.15312 v1:172.21.15.5:0/1053101955' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:14:56.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:55 smithi171 ceph-mon[44189]: osdmap e257: 12 total, 12 up, 12 in 2024-04-20T17:14:56.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:55 smithi171 ceph-mon[44189]: osdmap e258: 12 total, 12 up, 12 in 2024-04-20T17:14:56.353 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:14:56.381 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:14:56.412 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:56.859 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":710,"stamp":"2024-04-20T17:14:54.030484+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5440733752,"num_objects":83013,"num_object_clones":0,"num_object_copies":166026,"num_objects_missing_on_primary":4659,"num_objects_missing":4659,"num_objects_degraded":0,"num_objects_misplaced":9331,"num_objects_unfound":0,"num_objects_dirty":83013,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":239154,"num_write_kb":10306363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24552,"num_bytes_recovered":1610631296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160903,"ondisk_log_size":160903,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":37,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":12295152,"kb_used_data":11243004,"kb_used_omap":219,"kb_used_meta":1051812,"kb_avail":1112646672,"statfs":{"total":1151940427776,"available":1139350192128,"internally_reserved":0,"allocated":11512836096,"data_stored":11504198794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224949,"internal_metadata":1077055819},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-61997056,"num_objects":-946,"num_object_clones":0,"num_object_copies":-1892,"num_objects_missing_on_primary":-39,"num_objects_missing":-39,"num_objects_degraded":0,"num_objects_misplaced":-80,"num_objects_unfound":0,"num_objects_dirty":-946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":946,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":2686976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001557"},"pg_stats":[{"pgid":"2.f","version":"256'14988","reported_seq":15736,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-04-20T17:14:53.717057+0000","last_change":"2024-04-20T17:14:39.601610+0000","last_active":"2024-04-20T17:14:53.717057+0000","last_peered":"2024-04-20T17:14:53.717057+0000","last_clean":"2024-04-20T17:14:53.717057+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:14:53.717057+0000","last_undegraded":"2024-04-20T17:14:53.717057+0000","last_fullsized":"2024-04-20T17:14:53.717057+0000","mapping_epoch":115,"log_start":"128'4900","ondisk_log_start":"128'4900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'14864","last_scrub_stamp":"2024-04-20T17:14:39.601506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:39.601506+0000","objects_scrubbed":5320,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:01.262639+0000","scrub_duration":467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340393984,"num_objects":5194,"num_object_clones":0,"num_object_copies":10388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14988,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"255'14866","reported_seq":26625,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-04-20T17:14:50.487842+0000","last_change":"2024-04-20T17:14:02.868984+0000","last_active":"2024-04-20T17:14:50.487842+0000","last_peered":"2024-04-20T17:14:50.487842+0000","last_clean":"2024-04-20T17:14:50.487842+0000","last_became_active":"2024-04-20T17:10:28.865670+0000","last_became_peered":"2024-04-20T17:10:28.865670+0000","last_unstale":"2024-04-20T17:14:50.487842+0000","last_undegraded":"2024-04-20T17:14:50.487842+0000","last_fullsized":"2024-04-20T17:14:50.487842+0000","mapping_epoch":162,"log_start":"128'4800","ondisk_log_start":"128'4800","created":115,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'14480","last_scrub_stamp":"2024-04-20T17:14:02.868843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:02.868843+0000","objects_scrubbed":5576,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:07.951248+0000","scrub_duration":456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340131840,"num_objects":5190,"num_object_clones":0,"num_object_copies":10380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14866,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5619,"num_bytes_recovered":368246784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"255'15157","reported_seq":15741,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-04-20T17:14:50.478738+0000","last_change":"2024-04-20T17:14:45.170265+0000","last_active":"2024-04-20T17:14:50.478738+0000","last_peered":"2024-04-20T17:14:50.478738+0000","last_clean":"2024-04-20T17:14:50.478738+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:14:50.478738+0000","last_undegraded":"2024-04-20T17:14:50.478738+0000","last_fullsized":"2024-04-20T17:14:50.478738+0000","mapping_epoch":115,"log_start":"128'5100","ondisk_log_start":"128'5100","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'15118","last_scrub_stamp":"2024-04-20T17:14:45.170147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:45.170147+0000","objects_scrubbed":5364,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:55.316236+0000","scrub_duration":445,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348979200,"num_objects":5325,"num_object_clones":0,"num_object_copies":10650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15157,"num_write_kb":655424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"255'21588","reported_seq":42920,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-04-20T17:14:50.699680+0000","last_change":"2024-04-20T17:13:59.403959+0000","last_active":"2024-04-20T17:14:50.699680+0000","last_peered":"2024-04-20T17:14:50.699680+0000","last_clean":"2024-04-20T17:14:50.699680+0000","last_became_active":"2024-04-20T17:12:08.085936+0000","last_became_peered":"2024-04-20T17:12:08.085936+0000","last_unstale":"2024-04-20T17:14:50.699680+0000","last_undegraded":"2024-04-20T17:14:50.699680+0000","last_fullsized":"2024-04-20T17:14:50.699680+0000","mapping_epoch":196,"log_start":"157'11568","ondisk_log_start":"157'11568","created":115,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'21205","last_scrub_stamp":"2024-04-20T17:13:59.403804+0000","last_deep_scrub":"213'20373","last_deep_scrub_stamp":"2024-04-20T17:12:48.443808+0000","last_clean_scrub_stamp":"2024-04-20T17:13:59.403804+0000","objects_scrubbed":5682,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:06.827884+0000","scrub_duration":478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347209728,"num_objects":5298,"num_object_clones":0,"num_object_copies":10596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14936,"num_write_kb":647488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":450166784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"255'14629","reported_seq":29217,"reported_epoch":255,"state":"active+recovering","last_fresh":"2024-04-20T17:14:50.140785+0000","last_change":"2024-04-20T17:12:26.166109+0000","last_active":"2024-04-20T17:14:50.140785+0000","last_peered":"2024-04-20T17:14:50.140785+0000","last_clean":"2024-04-20T17:09:35.882603+0000","last_became_active":"2024-04-20T17:10:54.470253+0000","last_became_peered":"2024-04-20T17:10:54.470253+0000","last_unstale":"2024-04-20T17:14:50.140785+0000","last_undegraded":"2024-04-20T17:14:50.140785+0000","last_fullsized":"2024-04-20T17:14:50.140785+0000","mapping_epoch":171,"log_start":"126'4600","ondisk_log_start":"126'4600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334299136,"num_objects":5101,"num_object_clones":0,"num_object_copies":10202,"num_objects_missing_on_primary":4659,"num_objects_missing":4659,"num_objects_degraded":0,"num_objects_misplaced":9331,"num_objects_unfound":0,"num_objects_dirty":5101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14629,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4760,"num_bytes_recovered":311951360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"0,2,5","objects":13},{"shards":"0,10","objects":429},{"shards":"2,5","objects":4659}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":698,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-04-20T17:14:51.140280+0000","last_change":"2024-04-20T17:14:40.440515+0000","last_active":"2024-04-20T17:14:51.140280+0000","last_peered":"2024-04-20T17:14:51.140280+0000","last_clean":"2024-04-20T17:14:51.140280+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:14:51.140280+0000","last_undegraded":"2024-04-20T17:14:51.140280+0000","last_fullsized":"2024-04-20T17:14:51.140280+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:14:40.440403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:14:40.440403+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:01.949231+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"256'15016","reported_seq":15631,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-04-20T17:14:52.543225+0000","last_change":"2024-04-20T17:14:01.843280+0000","last_active":"2024-04-20T17:14:52.543225+0000","last_peered":"2024-04-20T17:14:52.543225+0000","last_clean":"2024-04-20T17:14:52.543225+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:14:52.543225+0000","last_undegraded":"2024-04-20T17:14:52.543225+0000","last_fullsized":"2024-04-20T17:14:52.543225+0000","mapping_epoch":115,"log_start":"128'5000","ondisk_log_start":"128'5000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'14629","last_scrub_stamp":"2024-04-20T17:14:01.843179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:01.843179+0000","objects_scrubbed":5626,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:24.160162+0000","scrub_duration":499,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343146496,"num_objects":5236,"num_object_clones":0,"num_object_copies":10472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15016,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"256'15108","reported_seq":15724,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-04-20T17:14:53.248156+0000","last_change":"2024-04-20T17:14:03.619324+0000","last_active":"2024-04-20T17:14:53.248156+0000","last_peered":"2024-04-20T17:14:53.248156+0000","last_clean":"2024-04-20T17:14:53.248156+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:14:53.248156+0000","last_undegraded":"2024-04-20T17:14:53.248156+0000","last_fullsized":"2024-04-20T17:14:53.248156+0000","mapping_epoch":115,"log_start":"128'5100","ondisk_log_start":"128'5100","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'14687","last_scrub_stamp":"2024-04-20T17:14:03.619176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:03.619176+0000","objects_scrubbed":5637,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:17.353819+0000","scrub_duration":558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341835776,"num_objects":5216,"num_object_clones":0,"num_object_copies":10432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15108,"num_write_kb":650368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"256'19059","reported_seq":19827,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-04-20T17:14:51.717113+0000","last_change":"2024-04-20T17:14:45.127365+0000","last_active":"2024-04-20T17:14:51.717113+0000","last_peered":"2024-04-20T17:14:51.717113+0000","last_clean":"2024-04-20T17:14:51.717113+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:14:51.717113+0000","last_undegraded":"2024-04-20T17:14:51.717113+0000","last_fullsized":"2024-04-20T17:14:51.717113+0000","mapping_epoch":115,"log_start":"144'9002","ondisk_log_start":"144'9002","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'19011","last_scrub_stamp":"2024-04-20T17:14:45.127219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:45.127219+0000","objects_scrubbed":5143,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:07.550027+0000","scrub_duration":538,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333840384,"num_objects":5094,"num_object_clones":0,"num_object_copies":10188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14948,"num_write_kb":641344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"256'15094","reported_seq":15768,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-04-20T17:14:51.992044+0000","last_change":"2024-04-20T17:14:37.076566+0000","last_active":"2024-04-20T17:14:51.992044+0000","last_peered":"2024-04-20T17:14:51.992044+0000","last_clean":"2024-04-20T17:14:51.992044+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:14:51.992044+0000","last_undegraded":"2024-04-20T17:14:51.992044+0000","last_fullsized":"2024-04-20T17:14:51.992044+0000","mapping_epoch":115,"log_start":"128'5000","ondisk_log_start":"128'5000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'14978","last_scrub_stamp":"2024-04-20T17:14:37.076332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:37.076332+0000","objects_scrubbed":5242,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:40.993014+0000","scrub_duration":400,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335937536,"num_objects":5126,"num_object_clones":0,"num_object_copies":10252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15094,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"256'14866","reported_seq":15467,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-04-20T17:14:52.615348+0000","last_change":"2024-04-20T17:14:44.918151+0000","last_active":"2024-04-20T17:14:52.615348+0000","last_peered":"2024-04-20T17:14:52.615348+0000","last_clean":"2024-04-20T17:14:52.615348+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:14:52.615348+0000","last_undegraded":"2024-04-20T17:14:52.615348+0000","last_fullsized":"2024-04-20T17:14:52.615348+0000","mapping_epoch":115,"log_start":"128'4800","ondisk_log_start":"128'4800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'14799","last_scrub_stamp":"2024-04-20T17:14:44.917948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:44.917948+0000","objects_scrubbed":5254,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:56.572429+0000","scrub_duration":440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339869696,"num_objects":5186,"num_object_clones":0,"num_object_copies":10372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14866,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"256'15029","reported_seq":30239,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-04-20T17:14:52.647003+0000","last_change":"2024-04-20T17:14:43.945880+0000","last_active":"2024-04-20T17:14:52.647003+0000","last_peered":"2024-04-20T17:14:52.647003+0000","last_clean":"2024-04-20T17:14:52.647003+0000","last_became_active":"2024-04-20T17:10:55.931751+0000","last_became_peered":"2024-04-20T17:10:55.931751+0000","last_unstale":"2024-04-20T17:14:52.647003+0000","last_undegraded":"2024-04-20T17:14:52.647003+0000","last_fullsized":"2024-04-20T17:14:52.647003+0000","mapping_epoch":173,"log_start":"128'5000","ondisk_log_start":"128'5000","created":115,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'14953","last_scrub_stamp":"2024-04-20T17:14:43.945755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:43.945755+0000","objects_scrubbed":5309,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:57.154812+0000","scrub_duration":454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342949888,"num_objects":5233,"num_object_clones":0,"num_object_copies":10466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15029,"num_write_kb":648384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7296,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"256'15116","reported_seq":15728,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-04-20T17:14:51.778955+0000","last_change":"2024-04-20T17:14:03.230824+0000","last_active":"2024-04-20T17:14:51.778955+0000","last_peered":"2024-04-20T17:14:51.778955+0000","last_clean":"2024-04-20T17:14:51.778955+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:14:51.778955+0000","last_undegraded":"2024-04-20T17:14:51.778955+0000","last_fullsized":"2024-04-20T17:14:51.778955+0000","mapping_epoch":115,"log_start":"128'5100","ondisk_log_start":"128'5100","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'14739","last_scrub_stamp":"2024-04-20T17:14:03.230712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:03.230712+0000","objects_scrubbed":5578,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:11.279553+0000","scrub_duration":546,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340787200,"num_objects":5200,"num_object_clones":0,"num_object_copies":10400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15116,"num_write_kb":650112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"256'14718","reported_seq":15329,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-04-20T17:14:53.235851+0000","last_change":"2024-04-20T17:14:05.574878+0000","last_active":"2024-04-20T17:14:53.235851+0000","last_peered":"2024-04-20T17:14:53.235851+0000","last_clean":"2024-04-20T17:14:53.235851+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:14:53.235851+0000","last_undegraded":"2024-04-20T17:14:53.235851+0000","last_fullsized":"2024-04-20T17:14:53.235851+0000","mapping_epoch":115,"log_start":"128'4700","ondisk_log_start":"128'4700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'14330","last_scrub_stamp":"2024-04-20T17:14:05.574772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:05.574772+0000","objects_scrubbed":5536,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:05.590167+0000","scrub_duration":494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337248256,"num_objects":5146,"num_object_clones":0,"num_object_copies":10292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14718,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"256'14969","reported_seq":15581,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-04-20T17:14:52.564529+0000","last_change":"2024-04-20T17:14:04.200256+0000","last_active":"2024-04-20T17:14:52.564529+0000","last_peered":"2024-04-20T17:14:52.564529+0000","last_clean":"2024-04-20T17:14:52.564529+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:14:52.564529+0000","last_undegraded":"2024-04-20T17:14:52.564529+0000","last_fullsized":"2024-04-20T17:14:52.564529+0000","mapping_epoch":115,"log_start":"128'4900","ondisk_log_start":"128'4900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'14590","last_scrub_stamp":"2024-04-20T17:14:04.200147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:04.200147+0000","objects_scrubbed":5516,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:21.444062+0000","scrub_duration":506,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336396288,"num_objects":5133,"num_object_clones":0,"num_object_copies":10266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14969,"num_write_kb":643264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"255'14741","reported_seq":15351,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-04-20T17:14:50.645738+0000","last_change":"2024-04-20T17:14:12.420994+0000","last_active":"2024-04-20T17:14:50.645738+0000","last_peered":"2024-04-20T17:14:50.645738+0000","last_clean":"2024-04-20T17:14:50.645738+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:14:50.645738+0000","last_undegraded":"2024-04-20T17:14:50.645738+0000","last_fullsized":"2024-04-20T17:14:50.645738+0000","mapping_epoch":115,"log_start":"127'4700","ondisk_log_start":"127'4700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'14443","last_scrub_stamp":"2024-04-20T17:14:12.420702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:12.420702+0000","objects_scrubbed":5431,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:19.358247+0000","scrub_duration":478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336068632,"num_objects":5129,"num_object_clones":0,"num_object_copies":10258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5129,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14741,"num_write_kb":635777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"255'14792","reported_seq":15267,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-04-20T17:14:50.783810+0000","last_change":"2024-04-20T17:13:58.682929+0000","last_active":"2024-04-20T17:14:50.783810+0000","last_peered":"2024-04-20T17:14:50.783810+0000","last_clean":"2024-04-20T17:14:50.783810+0000","last_became_active":"2024-04-20T17:12:23.515375+0000","last_became_peered":"2024-04-20T17:12:23.515375+0000","last_unstale":"2024-04-20T17:14:50.783810+0000","last_undegraded":"2024-04-20T17:14:50.783810+0000","last_fullsized":"2024-04-20T17:14:50.783810+0000","mapping_epoch":202,"log_start":"127'4700","ondisk_log_start":"127'4700","created":115,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'14359","last_scrub_stamp":"2024-04-20T17:13:58.682797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:13:58.682797+0000","objects_scrubbed":5638,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:14:59.113672+0000","scrub_duration":533,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341049344,"num_objects":5204,"num_object_clones":0,"num_object_copies":10408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14792,"num_write_kb":639872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":5440143384,"num_objects":83011,"num_object_clones":0,"num_object_copies":166022,"num_objects_missing_on_primary":4659,"num_objects_missing":4659,"num_objects_degraded":0,"num_objects_misplaced":9331,"num_objects_unfound":0,"num_objects_dirty":83011,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":238973,"num_write_kb":10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24544,"num_bytes_recovered":1608515584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11487551488,"data_stored":11487543344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},"log_size":160766,"ondisk_log_size":160766,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991342,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1064828,"kb_used_data":993392,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92680324,"statfs":{"total":95995035648,"available":94904651776,"internally_reserved":0,"allocated":1017233408,"data_stored":1016508844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":73136580},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777252980,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1168944,"kb_used_data":1025320,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92576208,"statfs":{"total":95995035648,"available":94798036992,"internally_reserved":0,"allocated":1049927680,"data_stored":1049206668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":147061184},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612729,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1466988,"kb_used_data":1337248,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92278164,"statfs":{"total":95995035648,"available":94492839936,"internally_reserved":0,"allocated":1369341952,"data_stored":1368622866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":132839872},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762808959,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1430320,"kb_used_data":1322792,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92314832,"statfs":{"total":95995035648,"available":94530387968,"internally_reserved":0,"allocated":1354539008,"data_stored":1353818016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168708,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1067320,"kb_used_data":1004208,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92677832,"statfs":{"total":95995035648,"available":94902099968,"internally_reserved":0,"allocated":1028308992,"data_stored":1027584448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430344,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":749300,"kb_used_data":668136,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":92995852,"statfs":{"total":95995035648,"available":95227752448,"internally_reserved":0,"allocated":684171264,"data_stored":683450272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":83098054},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575010,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":761840,"kb_used_data":668200,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":92983312,"statfs":{"total":95995035648,"available":95214911488,"internally_reserved":0,"allocated":684236800,"data_stored":683515808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738535,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1140004,"kb_used_data":1028316,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92605148,"statfs":{"total":95995035648,"available":94827671552,"internally_reserved":0,"allocated":1052995584,"data_stored":1052280570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393823,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":962724,"kb_used_data":918492,"kb_used_omap":202,"kb_used_meta":44021,"kb_avail":92782428,"statfs":{"total":95995035648,"available":95009206272,"internally_reserved":0,"allocated":940535808,"data_stored":939820794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":45077975},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182551,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":731060,"kb_used_data":667944,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93014092,"statfs":{"total":95995035648,"available":95246430208,"internally_reserved":0,"allocated":683974656,"data_stored":683253664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986322,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":365228,"kb_used_data":329888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93379924,"statfs":{"total":95995035648,"available":95621042176,"internally_reserved":0,"allocated":337805312,"data_stored":337086226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790092,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1386596,"kb_used_data":1279068,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92358556,"statfs":{"total":95995035648,"available":94575161344,"internally_reserved":0,"allocated":1309765632,"data_stored":1309050618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1050804224,"data_stored":1050804224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":682164224,"data_stored":682164224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":938541056,"data_stored":938541056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":682491904,"data_stored":682491904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":336138240,"data_stored":336134168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1307508736,"data_stored":1307508736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":682229760,"data_stored":682229760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1025966080,"data_stored":1025966080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1352728576,"data_stored":1352728576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1366298624,"data_stored":1366294552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1047724032,"data_stored":1047724032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1014956032,"data_stored":1014956032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:14:56.861 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:14:56.861 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:14:56.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:56.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:14:56.965 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:14:56.994 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:57.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:56 smithi005 ceph-mon[39562]: pgmap v713: 17 pgs: 1 active+recovering, 16 active+clean; 5.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 142 op/s; 9248/165538 objects misplaced (5.587%); 874 KiB/s, 13 objects/s recovering 2024-04-20T17:14:57.198 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:14:57.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:57.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:14:57.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:56 smithi106 ceph-mon[44423]: pgmap v713: 17 pgs: 1 active+recovering, 16 active+clean; 5.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 142 op/s; 9248/165538 objects misplaced (5.587%); 874 KiB/s, 13 objects/s recovering 2024-04-20T17:14:57.304 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:14:57.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:56 smithi171 ceph-mon[44189]: pgmap v713: 17 pgs: 1 active+recovering, 16 active+clean; 5.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 142 op/s; 9248/165538 objects misplaced (5.587%); 874 KiB/s, 13 objects/s recovering 2024-04-20T17:14:57.337 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:57.489 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:14:57.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:57.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:14:57.596 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:14:57.623 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:57.641 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:14:57.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:57.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:14:57.744 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:14:57.771 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:57.790 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:14:57.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:57.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:14:57.893 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:14:57.919 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:57.938 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:14:57.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:57.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:14:58.041 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:14:58.073 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:58.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:57 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3791096551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:14:58.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:57 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3791096551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:14:58.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:57 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3791096551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:14:58.725 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:14:58.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:58.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:14:58.755 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-20T17:14:58.830 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:14:58.871 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:58.885 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:14:58.885 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:14:58.885 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:14:58.885 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:14:58.885 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:14:58.885 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:14:58.885 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:14:58.885 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:14:58.886 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:14:58.886 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:14:58.886 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:14:58.886 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:14:58.886 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:14:58.886 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:14:58.886 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:14:58.886 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:14:58.886 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:14:58.886 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:14:58.886 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:14:58.886 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:14:58.887 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:14:58.887 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:14:58.887 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:14:58.887 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:14:59.124 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:14:59.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:59.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:14:59.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:58 smithi005 ceph-mon[39562]: pgmap v714: 17 pgs: 1 active+recovering, 16 active+clean; 5.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 135 op/s; 9248/164912 objects misplaced (5.608%); 369 KiB/s, 5 objects/s recovering 2024-04-20T17:14:59.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:58 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:14:59.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:58 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:14:59.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.134 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.134 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.134 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:58 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:14:59.233 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:14:59.257 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.257 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.258 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.258 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.258 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.258 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.258 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.258 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:58 smithi171 ceph-mon[44189]: pgmap v714: 17 pgs: 1 active+recovering, 16 active+clean; 5.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 135 op/s; 9248/164912 objects misplaced (5.608%); 369 KiB/s, 5 objects/s recovering 2024-04-20T17:14:59.258 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.258 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.258 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:58 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:14:59.258 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:58 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:14:59.258 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.258 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.258 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.258 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:58 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:14:59.260 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:59.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:58 smithi106 ceph-mon[44423]: pgmap v714: 17 pgs: 1 active+recovering, 16 active+clean; 5.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 135 op/s; 9248/164912 objects misplaced (5.608%); 369 KiB/s, 5 objects/s recovering 2024-04-20T17:14:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:58 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:14:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:58 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:14:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:14:59.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:58 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:14:59.279 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:14:59.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:59.282 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:14:59.383 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:14:59.414 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:59.719 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:14:59.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:59.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:14:59.801 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:14:59.823 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:14:59.851 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:14:59.870 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:14:59.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:14:59.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:14:59.974 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:15:00.003 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:00.021 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:15:00.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:00.024 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:15:00.125 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:15:00.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:59 smithi005 ceph-mon[39562]: osdmap e259: 12 total, 12 up, 12 in 2024-04-20T17:15:00.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:59 smithi005 ceph-mon[39562]: 2.e scrub starts 2024-04-20T17:15:00.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:59 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:15:00.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:14:59 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:15:00.156 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:00.260 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:15:00.260 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:15:00.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:59 smithi106 ceph-mon[44423]: osdmap e259: 12 total, 12 up, 12 in 2024-04-20T17:15:00.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:59 smithi106 ceph-mon[44423]: 2.e scrub starts 2024-04-20T17:15:00.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:59 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:15:00.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:14:59 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:15:00.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:00.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:15:00.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:59 smithi171 ceph-mon[44189]: osdmap e259: 12 total, 12 up, 12 in 2024-04-20T17:15:00.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:59 smithi171 ceph-mon[44189]: 2.e scrub starts 2024-04-20T17:15:00.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:59 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:15:00.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:14:59 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:15:00.365 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:15:00.392 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:00.410 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:15:00.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:00.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:15:00.514 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:15:00.540 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:00.558 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:15:00.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:00.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:15:00.662 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:15:00.688 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:00.706 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:15:00.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:00.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:15:00.810 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:15:00.836 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:00.856 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:15:00.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:00.858 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:15:00.959 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:15:00.986 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:01.007 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:15:01.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:01.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:15:01.111 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:15:01.137 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:01.156 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:15:01.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:01.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:15:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:00 smithi005 ceph-mon[39562]: 2.e scrub ok 2024-04-20T17:15:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:00 smithi005 ceph-mon[39562]: pgmap v716: 17 pgs: 1 active+recovering, 16 active+clean; 5.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 113 op/s; 9248/164658 objects misplaced (5.616%); 437 KiB/s, 6 objects/s recovering 2024-04-20T17:15:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:00 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:15:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:00 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:15:01.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:00 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:15:01.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:00 smithi005 ceph-mon[39562]: osdmap e260: 12 total, 12 up, 12 in 2024-04-20T17:15:01.259 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:15:01.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:00 smithi106 ceph-mon[44423]: 2.e scrub ok 2024-04-20T17:15:01.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:00 smithi106 ceph-mon[44423]: pgmap v716: 17 pgs: 1 active+recovering, 16 active+clean; 5.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 113 op/s; 9248/164658 objects misplaced (5.616%); 437 KiB/s, 6 objects/s recovering 2024-04-20T17:15:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:00 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:15:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:00 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:15:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:00 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:15:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:00 smithi106 ceph-mon[44423]: osdmap e260: 12 total, 12 up, 12 in 2024-04-20T17:15:01.291 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:01.321 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:15:01.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:01.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:15:01.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:00 smithi171 ceph-mon[44189]: 2.e scrub ok 2024-04-20T17:15:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:00 smithi171 ceph-mon[44189]: pgmap v716: 17 pgs: 1 active+recovering, 16 active+clean; 5.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 113 op/s; 9248/164658 objects misplaced (5.616%); 437 KiB/s, 6 objects/s recovering 2024-04-20T17:15:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:00 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:15:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:00 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:15:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:00 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:15:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:00 smithi171 ceph-mon[44189]: osdmap e260: 12 total, 12 up, 12 in 2024-04-20T17:15:01.426 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:15:01.451 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:01.469 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:15:01.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:01.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:15:01.573 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:15:01.600 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:01.622 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:15:01.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:01.625 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:15:01.726 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:15:01.752 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:01.755 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:15:01.771 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:15:01.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:01.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:15:01.875 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:15:01.903 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:01.931 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:15:01.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:01.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:15:02.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:01 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:15:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:01 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:15:02.036 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:15:02.063 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:01 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:15:02.063 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:01 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:15:02.065 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:02.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:01 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:15:02.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:01 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:15:03.016 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:15:03.019 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:02 smithi005 ceph-mon[39562]: pgmap v718: 17 pgs: 1 active+recovering, 16 active+clean; 5.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 118 op/s; 9169/164088 objects misplaced (5.588%); 429 KiB/s, 6 objects/s recovering 2024-04-20T17:15:03.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:03.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:15:03.121 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:15:03.148 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:03.168 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:15:03.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:03.171 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:15:03.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:02 smithi106 ceph-mon[44423]: pgmap v718: 17 pgs: 1 active+recovering, 16 active+clean; 5.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 118 op/s; 9169/164088 objects misplaced (5.588%); 429 KiB/s, 6 objects/s recovering 2024-04-20T17:15:03.272 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:15:03.299 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:02 smithi171 ceph-mon[44189]: pgmap v718: 17 pgs: 1 active+recovering, 16 active+clean; 5.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 118 op/s; 9169/164088 objects misplaced (5.588%); 429 KiB/s, 6 objects/s recovering 2024-04-20T17:15:03.301 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:03.322 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:15:03.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:03.326 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:15:03.427 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:15:03.453 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:03.471 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:15:03.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:03.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:15:03.575 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:15:03.601 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:03.620 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:15:03.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:03.625 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:15:03.726 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:15:03.744 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:15:03.745 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:15:03.752 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:03.771 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:15:03.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:03.774 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:15:03.875 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:15:03.908 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:03.960 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:15:04.092 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:15:04.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:04.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:15:04.197 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:15:04.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:03 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/586102204' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:15:04.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:03 smithi005 ceph-mon[39562]: from='client.25256 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:15:04.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:03 smithi005 ceph-mon[39562]: from='client.15330 v1:172.21.15.5:0/3974880949' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:15:04.228 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:04.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:03 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/586102204' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:15:04.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:03 smithi106 ceph-mon[44423]: from='client.25256 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:15:04.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:03 smithi106 ceph-mon[44423]: from='client.15330 v1:172.21.15.5:0/3974880949' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:15:04.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:03 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/586102204' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:15:04.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:03 smithi171 ceph-mon[44189]: from='client.25256 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:15:04.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:03 smithi171 ceph-mon[44189]: from='client.15330 v1:172.21.15.5:0/3974880949' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:15:04.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:04 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:15:04.999 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:04 smithi005 ceph-mon[39562]: from='client.25256 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:15:04.999 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:04 smithi005 ceph-mon[39562]: osdmap e261: 12 total, 12 up, 12 in 2024-04-20T17:15:04.999 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:04 smithi005 ceph-mon[39562]: pgmap v720: 17 pgs: 1 active+recovering, 16 active+clean; 5.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 122 op/s; 9169/163438 objects misplaced (5.610%); 437 KiB/s, 6 objects/s recovering 2024-04-20T17:15:04.999 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:04 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:15:04.999 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:04 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:15:05.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:04 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:15:05.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:04 smithi106 ceph-mon[44423]: from='client.25256 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:15:05.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:04 smithi106 ceph-mon[44423]: osdmap e261: 12 total, 12 up, 12 in 2024-04-20T17:15:05.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:04 smithi106 ceph-mon[44423]: pgmap v720: 17 pgs: 1 active+recovering, 16 active+clean; 5.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 122 op/s; 9169/163438 objects misplaced (5.610%); 437 KiB/s, 6 objects/s recovering 2024-04-20T17:15:05.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:04 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:15:05.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:04 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:15:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:04 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:15:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:04 smithi171 ceph-mon[44189]: from='client.25256 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:15:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:04 smithi171 ceph-mon[44189]: osdmap e261: 12 total, 12 up, 12 in 2024-04-20T17:15:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:04 smithi171 ceph-mon[44189]: pgmap v720: 17 pgs: 1 active+recovering, 16 active+clean; 5.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 122 op/s; 9169/163438 objects misplaced (5.610%); 437 KiB/s, 6 objects/s recovering 2024-04-20T17:15:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:04 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:15:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:04 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:15:05.679 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:15:05.713 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":718,"stamp":"2024-04-20T17:15:02.032001+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5377229368,"num_objects":82044,"num_object_clones":0,"num_object_copies":164088,"num_objects_missing_on_primary":4579,"num_objects_missing":4579,"num_objects_degraded":0,"num_objects_misplaced":9169,"num_objects_unfound":0,"num_objects_dirty":82044,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":240123,"num_write_kb":10306363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24634,"num_bytes_recovered":1616005248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160772,"ondisk_log_size":160772,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":37,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":12176132,"kb_used_data":11107332,"kb_used_omap":219,"kb_used_meta":1068452,"kb_avail":1112765692,"statfs":{"total":1151940427776,"available":1139472068608,"internally_reserved":0,"allocated":11373907968,"data_stored":11365044552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224949,"internal_metadata":1094095179},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,1,0,1,2,4,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-47513600,"num_objects":-725,"num_object_clones":0,"num_object_copies":-1450,"num_objects_missing_on_primary":-38,"num_objects_missing":-38,"num_objects_degraded":0,"num_objects_misplaced":-79,"num_objects_unfound":0,"num_objects_dirty":-725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":725,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":2686976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.114479"},"pg_stats":[{"pgid":"2.f","version":"258'15034","reported_seq":15786,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-04-20T17:14:58.336152+0000","last_change":"2024-04-20T17:14:39.601610+0000","last_active":"2024-04-20T17:14:58.336152+0000","last_peered":"2024-04-20T17:14:58.336152+0000","last_clean":"2024-04-20T17:14:58.336152+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:14:58.336152+0000","last_undegraded":"2024-04-20T17:14:58.336152+0000","last_fullsized":"2024-04-20T17:14:58.336152+0000","mapping_epoch":115,"log_start":"128'5000","ondisk_log_start":"128'5000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'14864","last_scrub_stamp":"2024-04-20T17:14:39.601506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:39.601506+0000","objects_scrubbed":5320,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:01.262639+0000","scrub_duration":467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337379328,"num_objects":5148,"num_object_clones":0,"num_object_copies":10296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15034,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"259'14952","reported_seq":26719,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-04-20T17:15:00.323110+0000","last_change":"2024-04-20T17:14:02.868984+0000","last_active":"2024-04-20T17:15:00.323110+0000","last_peered":"2024-04-20T17:15:00.323110+0000","last_clean":"2024-04-20T17:15:00.323110+0000","last_became_active":"2024-04-20T17:10:28.865670+0000","last_became_peered":"2024-04-20T17:10:28.865670+0000","last_unstale":"2024-04-20T17:15:00.323110+0000","last_undegraded":"2024-04-20T17:15:00.323110+0000","last_fullsized":"2024-04-20T17:15:00.323110+0000","mapping_epoch":162,"log_start":"128'4900","ondisk_log_start":"128'4900","created":115,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'14480","last_scrub_stamp":"2024-04-20T17:14:02.868843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:02.868843+0000","objects_scrubbed":5576,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:07.951248+0000","scrub_duration":456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334495744,"num_objects":5104,"num_object_clones":0,"num_object_copies":10208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14952,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5619,"num_bytes_recovered":368246784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"259'15240","reported_seq":15832,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-04-20T17:15:00.286496+0000","last_change":"2024-04-20T17:14:45.170265+0000","last_active":"2024-04-20T17:15:00.286496+0000","last_peered":"2024-04-20T17:15:00.286496+0000","last_clean":"2024-04-20T17:15:00.286496+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:15:00.286496+0000","last_undegraded":"2024-04-20T17:15:00.286496+0000","last_fullsized":"2024-04-20T17:15:00.286496+0000","mapping_epoch":115,"log_start":"129'5200","ondisk_log_start":"129'5200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'15118","last_scrub_stamp":"2024-04-20T17:14:45.170147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:45.170147+0000","objects_scrubbed":5364,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:55.316236+0000","scrub_duration":445,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343539712,"num_objects":5242,"num_object_clones":0,"num_object_copies":10484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15240,"num_write_kb":655424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"259'21679","reported_seq":43019,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-04-20T17:15:00.291110+0000","last_change":"2024-04-20T17:13:59.403959+0000","last_active":"2024-04-20T17:15:00.291110+0000","last_peered":"2024-04-20T17:15:00.291110+0000","last_clean":"2024-04-20T17:15:00.291110+0000","last_became_active":"2024-04-20T17:12:08.085936+0000","last_became_peered":"2024-04-20T17:12:08.085936+0000","last_unstale":"2024-04-20T17:15:00.291110+0000","last_undegraded":"2024-04-20T17:15:00.291110+0000","last_fullsized":"2024-04-20T17:15:00.291110+0000","mapping_epoch":196,"log_start":"158'11668","ondisk_log_start":"158'11668","created":115,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'21205","last_scrub_stamp":"2024-04-20T17:13:59.403804+0000","last_deep_scrub":"213'20373","last_deep_scrub_stamp":"2024-04-20T17:12:48.443808+0000","last_clean_scrub_stamp":"2024-04-20T17:13:59.403804+0000","objects_scrubbed":5682,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:06.827884+0000","scrub_duration":478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341245952,"num_objects":5207,"num_object_clones":0,"num_object_copies":10414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15027,"num_write_kb":647488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":450166784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"259'14710","reported_seq":29549,"reported_epoch":259,"state":"active+recovering","last_fresh":"2024-04-20T17:15:00.317797+0000","last_change":"2024-04-20T17:12:26.166109+0000","last_active":"2024-04-20T17:15:00.317797+0000","last_peered":"2024-04-20T17:15:00.317797+0000","last_clean":"2024-04-20T17:09:35.882603+0000","last_became_active":"2024-04-20T17:10:54.470253+0000","last_became_peered":"2024-04-20T17:10:54.470253+0000","last_unstale":"2024-04-20T17:15:00.317797+0000","last_undegraded":"2024-04-20T17:15:00.317797+0000","last_fullsized":"2024-04-20T17:15:00.317797+0000","mapping_epoch":171,"log_start":"127'4700","ondisk_log_start":"127'4700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328990720,"num_objects":5020,"num_object_clones":0,"num_object_copies":10040,"num_objects_missing_on_primary":4579,"num_objects_missing":4579,"num_objects_degraded":0,"num_objects_misplaced":9169,"num_objects_unfound":0,"num_objects_dirty":5020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14710,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4842,"num_bytes_recovered":317325312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"0,2,5","objects":11},{"shards":"0,10","objects":430},{"shards":"2,5","objects":4579}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":702,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-04-20T17:14:55.927963+0000","last_change":"2024-04-20T17:14:40.440515+0000","last_active":"2024-04-20T17:14:55.927963+0000","last_peered":"2024-04-20T17:14:55.927963+0000","last_clean":"2024-04-20T17:14:55.927963+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:14:55.927963+0000","last_undegraded":"2024-04-20T17:14:55.927963+0000","last_fullsized":"2024-04-20T17:14:55.927963+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:14:40.440403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:14:40.440403+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:01.949231+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"258'15062","reported_seq":15681,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-04-20T17:14:57.605099+0000","last_change":"2024-04-20T17:14:01.843280+0000","last_active":"2024-04-20T17:14:57.605099+0000","last_peered":"2024-04-20T17:14:57.605099+0000","last_clean":"2024-04-20T17:14:57.605099+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:14:57.605099+0000","last_undegraded":"2024-04-20T17:14:57.605099+0000","last_fullsized":"2024-04-20T17:14:57.605099+0000","mapping_epoch":115,"log_start":"128'5000","ondisk_log_start":"128'5000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'14629","last_scrub_stamp":"2024-04-20T17:14:01.843179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:01.843179+0000","objects_scrubbed":5626,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:24.160162+0000","scrub_duration":499,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340131840,"num_objects":5190,"num_object_clones":0,"num_object_copies":10380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15062,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"258'15159","reported_seq":15779,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-04-20T17:14:58.521724+0000","last_change":"2024-04-20T17:14:03.619324+0000","last_active":"2024-04-20T17:14:58.521724+0000","last_peered":"2024-04-20T17:14:58.521724+0000","last_clean":"2024-04-20T17:14:58.521724+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:14:58.521724+0000","last_undegraded":"2024-04-20T17:14:58.521724+0000","last_fullsized":"2024-04-20T17:14:58.521724+0000","mapping_epoch":115,"log_start":"128'5100","ondisk_log_start":"128'5100","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'14687","last_scrub_stamp":"2024-04-20T17:14:03.619176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:03.619176+0000","objects_scrubbed":5637,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:17.353819+0000","scrub_duration":558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338493440,"num_objects":5165,"num_object_clones":0,"num_object_copies":10330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15159,"num_write_kb":650368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"260'19140","reported_seq":19916,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-04-20T17:15:01.677573+0000","last_change":"2024-04-20T17:14:45.127365+0000","last_active":"2024-04-20T17:15:01.677573+0000","last_peered":"2024-04-20T17:15:01.677573+0000","last_clean":"2024-04-20T17:15:01.677573+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:15:01.677573+0000","last_undegraded":"2024-04-20T17:15:01.677573+0000","last_fullsized":"2024-04-20T17:15:01.677573+0000","mapping_epoch":115,"log_start":"145'9102","ondisk_log_start":"145'9102","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'19011","last_scrub_stamp":"2024-04-20T17:14:45.127219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:45.127219+0000","objects_scrubbed":5143,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:07.550027+0000","scrub_duration":538,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328531968,"num_objects":5013,"num_object_clones":0,"num_object_copies":10026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15029,"num_write_kb":641344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"258'15143","reported_seq":15821,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-04-20T17:14:57.460189+0000","last_change":"2024-04-20T17:14:37.076566+0000","last_active":"2024-04-20T17:14:57.460189+0000","last_peered":"2024-04-20T17:14:57.460189+0000","last_clean":"2024-04-20T17:14:57.460189+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:14:57.460189+0000","last_undegraded":"2024-04-20T17:14:57.460189+0000","last_fullsized":"2024-04-20T17:14:57.460189+0000","mapping_epoch":115,"log_start":"128'5100","ondisk_log_start":"128'5100","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'14978","last_scrub_stamp":"2024-04-20T17:14:37.076332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:37.076332+0000","objects_scrubbed":5242,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:40.993014+0000","scrub_duration":400,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332726272,"num_objects":5077,"num_object_clones":0,"num_object_copies":10154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15143,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"258'14918","reported_seq":15523,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-04-20T17:14:57.515203+0000","last_change":"2024-04-20T17:14:44.918151+0000","last_active":"2024-04-20T17:14:57.515203+0000","last_peered":"2024-04-20T17:14:57.515203+0000","last_clean":"2024-04-20T17:14:57.515203+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:14:57.515203+0000","last_undegraded":"2024-04-20T17:14:57.515203+0000","last_fullsized":"2024-04-20T17:14:57.515203+0000","mapping_epoch":115,"log_start":"128'4900","ondisk_log_start":"128'4900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'14799","last_scrub_stamp":"2024-04-20T17:14:44.917948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:44.917948+0000","objects_scrubbed":5254,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:56.572429+0000","scrub_duration":440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336461824,"num_objects":5134,"num_object_clones":0,"num_object_copies":10268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14918,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"258'15078","reported_seq":30292,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-04-20T17:14:57.843956+0000","last_change":"2024-04-20T17:14:43.945880+0000","last_active":"2024-04-20T17:14:57.843956+0000","last_peered":"2024-04-20T17:14:57.843956+0000","last_clean":"2024-04-20T17:14:57.843956+0000","last_became_active":"2024-04-20T17:10:55.931751+0000","last_became_peered":"2024-04-20T17:10:55.931751+0000","last_unstale":"2024-04-20T17:14:57.843956+0000","last_undegraded":"2024-04-20T17:14:57.843956+0000","last_fullsized":"2024-04-20T17:14:57.843956+0000","mapping_epoch":173,"log_start":"128'5000","ondisk_log_start":"128'5000","created":115,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'14953","last_scrub_stamp":"2024-04-20T17:14:43.945755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:43.945755+0000","objects_scrubbed":5309,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:57.154812+0000","scrub_duration":454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339738624,"num_objects":5184,"num_object_clones":0,"num_object_copies":10368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15078,"num_write_kb":648384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7296,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"258'15159","reported_seq":15775,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-04-20T17:14:57.549805+0000","last_change":"2024-04-20T17:14:03.230824+0000","last_active":"2024-04-20T17:14:57.549805+0000","last_peered":"2024-04-20T17:14:57.549805+0000","last_clean":"2024-04-20T17:14:57.549805+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:14:57.549805+0000","last_undegraded":"2024-04-20T17:14:57.549805+0000","last_fullsized":"2024-04-20T17:14:57.549805+0000","mapping_epoch":115,"log_start":"128'5100","ondisk_log_start":"128'5100","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'14739","last_scrub_stamp":"2024-04-20T17:14:03.230712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:03.230712+0000","objects_scrubbed":5578,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:11.279553+0000","scrub_duration":546,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337969152,"num_objects":5157,"num_object_clones":0,"num_object_copies":10314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15159,"num_write_kb":650112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"258'14748","reported_seq":15363,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-04-20T17:14:58.378232+0000","last_change":"2024-04-20T17:14:05.574878+0000","last_active":"2024-04-20T17:14:58.378232+0000","last_peered":"2024-04-20T17:14:58.378232+0000","last_clean":"2024-04-20T17:14:58.378232+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:14:58.378232+0000","last_undegraded":"2024-04-20T17:14:58.378232+0000","last_fullsized":"2024-04-20T17:14:58.378232+0000","mapping_epoch":115,"log_start":"128'4700","ondisk_log_start":"128'4700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'14330","last_scrub_stamp":"2024-04-20T17:14:05.574772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:05.574772+0000","objects_scrubbed":5536,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:05.590167+0000","scrub_duration":494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335282176,"num_objects":5116,"num_object_clones":0,"num_object_copies":10232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14748,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"258'15010","reported_seq":15626,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-04-20T17:14:57.381119+0000","last_change":"2024-04-20T17:14:04.200256+0000","last_active":"2024-04-20T17:14:57.381119+0000","last_peered":"2024-04-20T17:14:57.381119+0000","last_clean":"2024-04-20T17:14:57.381119+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:14:57.381119+0000","last_undegraded":"2024-04-20T17:14:57.381119+0000","last_fullsized":"2024-04-20T17:14:57.381119+0000","mapping_epoch":115,"log_start":"128'5000","ondisk_log_start":"128'5000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'14590","last_scrub_stamp":"2024-04-20T17:14:04.200147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:04.200147+0000","objects_scrubbed":5516,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:21.444062+0000","scrub_duration":506,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333709312,"num_objects":5092,"num_object_clones":0,"num_object_copies":10184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15010,"num_write_kb":643264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"259'14810","reported_seq":15428,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-04-20T17:15:00.348253+0000","last_change":"2024-04-20T17:14:12.420994+0000","last_active":"2024-04-20T17:15:00.348253+0000","last_peered":"2024-04-20T17:15:00.348253+0000","last_clean":"2024-04-20T17:15:00.348253+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:15:00.348253+0000","last_undegraded":"2024-04-20T17:15:00.348253+0000","last_fullsized":"2024-04-20T17:15:00.348253+0000","mapping_epoch":115,"log_start":"128'4800","ondisk_log_start":"128'4800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'14443","last_scrub_stamp":"2024-04-20T17:14:12.420702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:12.420702+0000","objects_scrubbed":5431,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:19.358247+0000","scrub_duration":478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331546648,"num_objects":5060,"num_object_clones":0,"num_object_copies":10120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5060,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14810,"num_write_kb":635777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"259'14863","reported_seq":15405,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-04-20T17:15:00.422650+0000","last_change":"2024-04-20T17:14:59.605205+0000","last_active":"2024-04-20T17:15:00.422650+0000","last_peered":"2024-04-20T17:15:00.422650+0000","last_clean":"2024-04-20T17:15:00.422650+0000","last_became_active":"2024-04-20T17:12:23.515375+0000","last_became_peered":"2024-04-20T17:12:23.515375+0000","last_unstale":"2024-04-20T17:15:00.422650+0000","last_undegraded":"2024-04-20T17:15:00.422650+0000","last_fullsized":"2024-04-20T17:15:00.422650+0000","mapping_epoch":202,"log_start":"128'4800","ondisk_log_start":"128'4800","created":115,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'14856","last_scrub_stamp":"2024-04-20T17:14:59.605082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:59.605082+0000","objects_scrubbed":5143,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:21.549388+0000","scrub_duration":486,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336396288,"num_objects":5133,"num_object_clones":0,"num_object_copies":10266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14863,"num_write_kb":639872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":5376639000,"num_objects":82042,"num_object_clones":0,"num_object_copies":164084,"num_objects_missing_on_primary":4579,"num_objects_missing":4579,"num_objects_degraded":0,"num_objects_misplaced":9169,"num_objects_unfound":0,"num_objects_dirty":82042,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":239942,"num_write_kb":10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24626,"num_bytes_recovered":1613889536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11343896576,"data_stored":11343888432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},"log_size":160635,"ondisk_log_size":160635,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991343,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1061712,"kb_used_data":986120,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92683440,"statfs":{"total":95995035648,"available":94907842560,"internally_reserved":0,"allocated":1009786880,"data_stored":1009050410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":77396420},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777252981,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1163976,"kb_used_data":1016192,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92581176,"statfs":{"total":95995035648,"available":94803124224,"internally_reserved":0,"allocated":1040580608,"data_stored":1039847690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":151321024},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612731,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1445848,"kb_used_data":1316112,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92299304,"statfs":{"total":95995035648,"available":94514487296,"internally_reserved":0,"allocated":1347698688,"data_stored":1346955790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":132839872},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762808960,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1419848,"kb_used_data":1312320,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92325304,"statfs":{"total":95995035648,"available":94541111296,"internally_reserved":0,"allocated":1343815680,"data_stored":1343082782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168709,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1057940,"kb_used_data":994824,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92687212,"statfs":{"total":95995035648,"available":94911705088,"internally_reserved":0,"allocated":1018699776,"data_stored":1017963326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430345,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":743760,"kb_used_data":662592,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93001392,"statfs":{"total":95995035648,"available":95233425408,"internally_reserved":0,"allocated":678494208,"data_stored":677761310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":83098054},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575012,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":751460,"kb_used_data":657816,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":92993692,"statfs":{"total":95995035648,"available":95225540608,"internally_reserved":0,"allocated":673603584,"data_stored":672858830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738537,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1124440,"kb_used_data":1012748,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92620712,"statfs":{"total":95995035648,"available":94843609088,"internally_reserved":0,"allocated":1037053952,"data_stored":1036315126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,1,0,1,2,4,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":940824,"kb_used_data":896588,"kb_used_omap":202,"kb_used_meta":44021,"kb_avail":92804328,"statfs":{"total":95995035648,"available":95031631872,"internally_reserved":0,"allocated":918106112,"data_stored":917367286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":45077975},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182553,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":721572,"kb_used_data":658456,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93023580,"statfs":{"total":95995035648,"available":95256145920,"internally_reserved":0,"allocated":674258944,"data_stored":673514190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986324,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":364952,"kb_used_data":325456,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93380200,"statfs":{"total":95995035648,"available":95621324800,"internally_reserved":0,"allocated":333266944,"data_stored":332524046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790094,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1379800,"kb_used_data":1268108,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92365352,"statfs":{"total":95995035648,"available":94582120448,"internally_reserved":0,"allocated":1298542592,"data_stored":1297803766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1033568256,"data_stored":1033568256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":671088640,"data_stored":671088640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":914227200,"data_stored":914227200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":672399360,"data_stored":672399360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":331550720,"data_stored":331546648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1296433152,"data_stored":1296433152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":675676160,"data_stored":675676160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1016332288,"data_stored":1016332288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1341325312,"data_stored":1341325312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1344802816,"data_stored":1344798744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1039007744,"data_stored":1039007744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1007484928,"data_stored":1007484928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:15:05.716 INFO:tasks.cephadm.ceph_manager.ceph:PG 2.3 is not active+clean 2024-04-20T17:15:05.717 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '2.3', 'version': "259'14710", 'reported_seq': 29549, 'reported_epoch': 259, 'state': 'active+recovering', 'last_fresh': '2024-04-20T17:15:00.317797+0000', 'last_change': '2024-04-20T17:12:26.166109+0000', 'last_active': '2024-04-20T17:15:00.317797+0000', 'last_peered': '2024-04-20T17:15:00.317797+0000', 'last_clean': '2024-04-20T17:09:35.882603+0000', 'last_became_active': '2024-04-20T17:10:54.470253+0000', 'last_became_peered': '2024-04-20T17:10:54.470253+0000', 'last_unstale': '2024-04-20T17:15:00.317797+0000', 'last_undegraded': '2024-04-20T17:15:00.317797+0000', 'last_fullsized': '2024-04-20T17:15:00.317797+0000', 'mapping_epoch': 171, 'log_start': "127'4700", 'ondisk_log_start': "127'4700", 'created': 115, 'last_epoch_clean': 116, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-20T17:08:04.990639+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T17:08:04.990639+0000', 'last_clean_scrub_stamp': '2024-04-20T17:08:04.990639+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 328990720, 'num_objects': 5020, 'num_object_clones': 0, 'num_object_copies': 10040, 'num_objects_missing_on_primary': 4579, 'num_objects_missing': 4579, 'num_objects_degraded': 0, 'num_objects_misplaced': 9169, 'num_objects_unfound': 0, 'num_objects_dirty': 5020, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14710, 'num_write_kb': 631360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4842, 'num_bytes_recovered': 317325312, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 10], 'acting': [0, 10], 'avail_no_missing': ['2', '5'], 'object_location_counts': [{'shards': '0,2,5', 'objects': 11}, {'shards': '0,10', 'objects': 430}, {'shards': '2,5', 'objects': 4579}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-20T17:15:05.717 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph status --format=json 2024-04-20T17:15:06.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:05 smithi005 ceph-mon[39562]: osdmap e262: 12 total, 12 up, 12 in 2024-04-20T17:15:06.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:05 smithi106 ceph-mon[44423]: osdmap e262: 12 total, 12 up, 12 in 2024-04-20T17:15:06.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:05 smithi171 ceph-mon[44189]: osdmap e262: 12 total, 12 up, 12 in 2024-04-20T17:15:06.335 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:15:06.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:06.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:15:06.341 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:15:06.358 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:15:06.358 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:15:06.358 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:15:06.358 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:15:06.358 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.358 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:228278 2.3 2:cbde697d:::benchmark_data_smithi171_78918_object67284:head [delete] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-20T17:15:06.358 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:29.213212+0000", 2024-04-20T17:15:06.358 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 97.141196355000005, 2024-04-20T17:15:06.358 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4218705329999999, 2024-04-20T17:15:06.358 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:15:06.358 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:15:06.359 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:15:06.359 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:15:06.359 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:15:06.359 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:15:06.359 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 228278 2024-04-20T17:15:06.359 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.359 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:15:06.359 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.359 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:15:06.359 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.213212+0000", 2024-04-20T17:15:06.359 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.359 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.359 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.359 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:15:06.359 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.213212+0000", 2024-04-20T17:15:06.360 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.360 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.360 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.360 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:15:06.360 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.213215+0000", 2024-04-20T17:15:06.360 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4930000000000001e-06 2024-04-20T17:15:06.360 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.360 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.360 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:15:06.360 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.213222+0000", 2024-04-20T17:15:06.360 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2030000000000003e-06 2024-04-20T17:15:06.360 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.360 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.360 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:15:06.361 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.213235+0000", 2024-04-20T17:15:06.361 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2626000000000001e-05 2024-04-20T17:15:06.361 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.361 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.361 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:15:06.361 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.213243+0000", 2024-04-20T17:15:06.361 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4239999999999993e-06 2024-04-20T17:15:06.361 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.361 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.361 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.361 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.337241+0000", 2024-04-20T17:15:06.361 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.123998226 2024-04-20T17:15:06.361 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.361 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.361 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:15:06.362 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.344845+0000", 2024-04-20T17:15:06.362 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0076031060000000001 2024-04-20T17:15:06.362 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.362 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.362 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.362 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.387780+0000", 2024-04-20T17:15:06.362 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.042935177999999997 2024-04-20T17:15:06.362 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.362 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.362 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:15:06.362 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.392394+0000", 2024-04-20T17:15:06.362 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0046145170000000003 2024-04-20T17:15:06.362 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.362 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.363 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.363 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.508260+0000", 2024-04-20T17:15:06.363 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1158655610000001 2024-04-20T17:15:06.363 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.363 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.363 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:15:06.363 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.514617+0000", 2024-04-20T17:15:06.363 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0063572530000000002 2024-04-20T17:15:06.363 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.363 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.363 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:15:06.363 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.532374+0000", 2024-04-20T17:15:06.363 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.017756732000000001 2024-04-20T17:15:06.363 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.364 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.364 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:15:06.364 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.577271+0000", 2024-04-20T17:15:06.364 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.044896881999999999 2024-04-20T17:15:06.364 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.364 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.364 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:15:06.364 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.617897+0000", 2024-04-20T17:15:06.364 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.040626439 2024-04-20T17:15:06.364 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.364 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.364 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:15:06.364 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.621386+0000", 2024-04-20T17:15:06.364 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003488807 2024-04-20T17:15:06.364 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.365 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.365 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:15:06.365 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.635083+0000", 2024-04-20T17:15:06.365 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013697086000000001 2024-04-20T17:15:06.365 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.365 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:15:06.365 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.365 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.365 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.365 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:228280 2.3 2:c765d134:::benchmark_data_smithi171_78918_object67286:head [delete] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-20T17:15:06.365 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:29.220826+0000", 2024-04-20T17:15:06.365 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 97.133582454000006, 2024-04-20T17:15:06.365 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3460507879999999, 2024-04-20T17:15:06.365 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:15:06.365 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:15:06.366 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:15:06.366 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:15:06.366 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:15:06.366 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:15:06.366 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 228280 2024-04-20T17:15:06.366 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.366 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:15:06.366 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.366 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:15:06.366 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.220826+0000", 2024-04-20T17:15:06.366 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.366 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.366 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.366 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:15:06.367 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.220826+0000", 2024-04-20T17:15:06.367 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.367 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.367 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.367 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:15:06.367 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.220828+0000", 2024-04-20T17:15:06.367 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6610000000000001e-06 2024-04-20T17:15:06.367 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.367 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.367 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:15:06.367 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.220833+0000", 2024-04-20T17:15:06.367 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.668e-06 2024-04-20T17:15:06.367 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.367 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.367 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:15:06.367 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.220844+0000", 2024-04-20T17:15:06.368 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1131999999999999e-05 2024-04-20T17:15:06.368 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.368 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.368 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:15:06.368 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.220851+0000", 2024-04-20T17:15:06.368 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8299999999999998e-06 2024-04-20T17:15:06.368 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.368 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.368 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.368 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.345596+0000", 2024-04-20T17:15:06.368 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.124745306 2024-04-20T17:15:06.368 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.368 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.368 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:15:06.368 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.353177+0000", 2024-04-20T17:15:06.369 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.007581366 2024-04-20T17:15:06.369 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.369 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.369 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.369 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.429194+0000", 2024-04-20T17:15:06.369 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.076017086999999997 2024-04-20T17:15:06.369 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.369 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.369 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:15:06.369 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.433408+0000", 2024-04-20T17:15:06.369 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0042133099999999996 2024-04-20T17:15:06.370 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.370 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.370 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.370 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.445818+0000", 2024-04-20T17:15:06.370 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0124105569999999 2024-04-20T17:15:06.370 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.370 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.370 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:15:06.370 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.455999+0000", 2024-04-20T17:15:06.370 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010180751 2024-04-20T17:15:06.370 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.370 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.370 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:15:06.370 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.472009+0000", 2024-04-20T17:15:06.371 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.016010301000000001 2024-04-20T17:15:06.371 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.371 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.371 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:15:06.371 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.506770+0000", 2024-04-20T17:15:06.371 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.034761045999999997 2024-04-20T17:15:06.371 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.371 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.371 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:15:06.371 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.548721+0000", 2024-04-20T17:15:06.371 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.041950618000000002 2024-04-20T17:15:06.371 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.371 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.371 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:15:06.371 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.553159+0000", 2024-04-20T17:15:06.372 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0044375090000000001 2024-04-20T17:15:06.372 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.372 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.372 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:15:06.372 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.566877+0000", 2024-04-20T17:15:06.372 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013718645999999999 2024-04-20T17:15:06.372 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.372 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:15:06.372 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.372 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.372 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.372 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:228281 2.3 2:cb7bd631:::benchmark_data_smithi171_78918_object67287:head [delete] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-20T17:15:06.372 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:29.604753+0000", 2024-04-20T17:15:06.372 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 96.749656156, 2024-04-20T17:15:06.372 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3682788320000001, 2024-04-20T17:15:06.373 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:15:06.373 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:15:06.373 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:15:06.373 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:15:06.373 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:15:06.373 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:15:06.373 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 228281 2024-04-20T17:15:06.373 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.373 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:15:06.373 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.373 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:15:06.373 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.604753+0000", 2024-04-20T17:15:06.373 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.373 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.374 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.374 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:15:06.374 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.604753+0000", 2024-04-20T17:15:06.374 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.374 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.374 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.374 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:15:06.374 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.604755+0000", 2024-04-20T17:15:06.374 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3329999999999999e-06 2024-04-20T17:15:06.374 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.374 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.374 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:15:06.374 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.604762+0000", 2024-04-20T17:15:06.374 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5359999999999998e-06 2024-04-20T17:15:06.375 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.375 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.375 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:15:06.375 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.604774+0000", 2024-04-20T17:15:06.375 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2575000000000001e-05 2024-04-20T17:15:06.375 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.375 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.375 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:15:06.375 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.604783+0000", 2024-04-20T17:15:06.375 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5129999999999996e-06 2024-04-20T17:15:06.375 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.375 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.375 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.375 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.769321+0000", 2024-04-20T17:15:06.376 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.16453883899999999 2024-04-20T17:15:06.376 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.376 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.376 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:15:06.376 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.777087+0000", 2024-04-20T17:15:06.376 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0077658639999999999 2024-04-20T17:15:06.376 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.376 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.376 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.376 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.823978+0000", 2024-04-20T17:15:06.376 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.046890606000000001 2024-04-20T17:15:06.376 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.376 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.376 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:15:06.376 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.828156+0000", 2024-04-20T17:15:06.377 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041778960000000004 2024-04-20T17:15:06.377 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.377 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.377 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.377 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.801221+0000", 2024-04-20T17:15:06.377 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9730655019999999 2024-04-20T17:15:06.377 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.377 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.377 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:15:06.377 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.807293+0000", 2024-04-20T17:15:06.377 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0060714280000000002 2024-04-20T17:15:06.377 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.377 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.377 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:15:06.377 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.819374+0000", 2024-04-20T17:15:06.378 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012080771000000001 2024-04-20T17:15:06.378 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.378 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.378 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:15:06.378 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.856885+0000", 2024-04-20T17:15:06.378 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.037511272999999998 2024-04-20T17:15:06.378 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.378 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.378 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:15:06.378 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.954790+0000", 2024-04-20T17:15:06.378 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.097905345000000005 2024-04-20T17:15:06.378 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.378 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.378 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:15:06.378 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.959531+0000", 2024-04-20T17:15:06.379 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0047406000000000002 2024-04-20T17:15:06.379 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.379 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.379 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:15:06.379 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.973032+0000", 2024-04-20T17:15:06.379 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013500751 2024-04-20T17:15:06.379 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.379 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:15:06.379 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.379 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.379 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.379 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:228399 2.3 2:c70941a5:::benchmark_data_smithi171_78918_object67405:head [delete] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-20T17:15:06.379 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:30.170222+0000", 2024-04-20T17:15:06.379 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 96.184187284000004, 2024-04-20T17:15:06.379 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.59054261, 2024-04-20T17:15:06.380 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:15:06.380 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:15:06.380 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:15:06.380 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:15:06.380 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:15:06.380 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:15:06.380 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 228399 2024-04-20T17:15:06.380 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.380 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:15:06.380 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.380 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:15:06.380 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.170222+0000", 2024-04-20T17:15:06.380 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.380 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.380 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.381 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:15:06.381 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.170222+0000", 2024-04-20T17:15:06.381 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.381 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.381 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.381 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:15:06.381 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.170224+0000", 2024-04-20T17:15:06.381 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0729999999999999e-06 2024-04-20T17:15:06.381 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.381 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.381 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:15:06.381 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.170232+0000", 2024-04-20T17:15:06.381 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7320000000000004e-06 2024-04-20T17:15:06.381 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.381 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.382 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:15:06.382 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.170245+0000", 2024-04-20T17:15:06.382 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2333e-05 2024-04-20T17:15:06.382 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.382 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.382 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:15:06.382 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.170255+0000", 2024-04-20T17:15:06.382 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9860000000000006e-06 2024-04-20T17:15:06.382 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.382 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.382 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.382 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.384564+0000", 2024-04-20T17:15:06.382 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.21430969599999999 2024-04-20T17:15:06.382 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.382 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.383 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:15:06.383 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.392665+0000", 2024-04-20T17:15:06.383 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.008101044 2024-04-20T17:15:06.383 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.383 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.383 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.383 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.450812+0000", 2024-04-20T17:15:06.383 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.058146301999999997 2024-04-20T17:15:06.383 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.383 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.383 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:15:06.383 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.454910+0000", 2024-04-20T17:15:06.383 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0040980859999999999 2024-04-20T17:15:06.383 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.383 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.384 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.384 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.477906+0000", 2024-04-20T17:15:06.384 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.022996129 2024-04-20T17:15:06.384 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.384 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.384 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:15:06.384 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.486187+0000", 2024-04-20T17:15:06.384 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0082809719999999993 2024-04-20T17:15:06.384 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.384 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.384 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:15:06.384 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.501211+0000", 2024-04-20T17:15:06.384 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015024404 2024-04-20T17:15:06.384 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.385 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.385 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:15:06.385 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.542755+0000", 2024-04-20T17:15:06.385 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.041543941000000001 2024-04-20T17:15:06.385 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.385 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.385 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:15:06.385 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.743025+0000", 2024-04-20T17:15:06.385 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.20027012 2024-04-20T17:15:06.385 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.385 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.385 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:15:06.385 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.747199+0000", 2024-04-20T17:15:06.385 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.004173693 2024-04-20T17:15:06.385 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.386 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.386 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:15:06.386 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.760764+0000", 2024-04-20T17:15:06.386 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013565099000000001 2024-04-20T17:15:06.386 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.386 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:15:06.386 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.386 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.386 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.386 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:228401 2.3 2:cb66e8d6:::benchmark_data_smithi171_78918_object67407:head [delete] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-20T17:15:06.386 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:30.176585+0000", 2024-04-20T17:15:06.386 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 96.177823497999995, 2024-04-20T17:15:06.386 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6073272909999998, 2024-04-20T17:15:06.386 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:15:06.386 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:15:06.387 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:15:06.387 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:15:06.387 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:15:06.387 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:15:06.387 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 228401 2024-04-20T17:15:06.387 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.387 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:15:06.387 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.387 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:15:06.387 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.176585+0000", 2024-04-20T17:15:06.387 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.387 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.387 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.387 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:15:06.387 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.176585+0000", 2024-04-20T17:15:06.388 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.388 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.388 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.388 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:15:06.388 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.176587+0000", 2024-04-20T17:15:06.388 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7910000000000001e-06 2024-04-20T17:15:06.388 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.388 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.388 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:15:06.388 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.176594+0000", 2024-04-20T17:15:06.388 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5950000000000004e-06 2024-04-20T17:15:06.388 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.388 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.388 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:15:06.388 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.176605+0000", 2024-04-20T17:15:06.389 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1304000000000001e-05 2024-04-20T17:15:06.389 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.389 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.389 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:15:06.389 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.176613+0000", 2024-04-20T17:15:06.389 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5680000000000002e-06 2024-04-20T17:15:06.389 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.389 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.389 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.389 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.393900+0000", 2024-04-20T17:15:06.389 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.21728766199999999 2024-04-20T17:15:06.389 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.389 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.389 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:15:06.389 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.402263+0000", 2024-04-20T17:15:06.390 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0083628279999999992 2024-04-20T17:15:06.390 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.390 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.390 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.390 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.485909+0000", 2024-04-20T17:15:06.390 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.083645419999999998 2024-04-20T17:15:06.390 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.390 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.390 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:15:06.390 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.490020+0000", 2024-04-20T17:15:06.390 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041117300000000001 2024-04-20T17:15:06.390 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.390 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.390 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.391 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.589948+0000", 2024-04-20T17:15:06.391 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0999282500000001 2024-04-20T17:15:06.391 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.391 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.391 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:15:06.391 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.596087+0000", 2024-04-20T17:15:06.391 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0061383920000000003 2024-04-20T17:15:06.391 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.391 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.391 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:15:06.391 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.608189+0000", 2024-04-20T17:15:06.391 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012101835999999999 2024-04-20T17:15:06.391 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.391 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.392 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:15:06.392 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.764145+0000", 2024-04-20T17:15:06.392 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.15595646499999999 2024-04-20T17:15:06.392 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.392 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.392 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:15:06.392 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.766235+0000", 2024-04-20T17:15:06.392 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0020894360000000001 2024-04-20T17:15:06.392 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.392 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.392 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:15:06.392 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.770408+0000", 2024-04-20T17:15:06.392 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041729999999999996 2024-04-20T17:15:06.392 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.392 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.393 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:15:06.393 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.783913+0000", 2024-04-20T17:15:06.393 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013505013999999999 2024-04-20T17:15:06.393 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.393 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:15:06.393 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.393 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.393 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.393 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:228553 2.3 2:c472042b:::benchmark_data_smithi171_78918_object67559:head [delete] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-20T17:15:06.393 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:30.926355+0000", 2024-04-20T17:15:06.393 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 95.428054333000006, 2024-04-20T17:15:06.393 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.445260926, 2024-04-20T17:15:06.393 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:15:06.393 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:15:06.393 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:15:06.393 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:15:06.394 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:15:06.394 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:15:06.394 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 228553 2024-04-20T17:15:06.394 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.394 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:15:06.394 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.394 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:15:06.394 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.926355+0000", 2024-04-20T17:15:06.394 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.394 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.394 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.394 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:15:06.394 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.926355+0000", 2024-04-20T17:15:06.394 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.395 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.395 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.395 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:15:06.395 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.926357+0000", 2024-04-20T17:15:06.395 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7640000000000001e-06 2024-04-20T17:15:06.395 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.395 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.395 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:15:06.395 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.926369+0000", 2024-04-20T17:15:06.395 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1695000000000001e-05 2024-04-20T17:15:06.395 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.395 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.395 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:15:06.395 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.926381+0000", 2024-04-20T17:15:06.395 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1564e-05 2024-04-20T17:15:06.395 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.396 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.396 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:15:06.396 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.926390+0000", 2024-04-20T17:15:06.396 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7790000000000007e-06 2024-04-20T17:15:06.396 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.396 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.396 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.396 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.023105+0000", 2024-04-20T17:15:06.396 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.096714498999999995 2024-04-20T17:15:06.396 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.396 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.396 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:15:06.396 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.032045+0000", 2024-04-20T17:15:06.396 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00894 2024-04-20T17:15:06.396 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.396 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.397 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.397 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.098624+0000", 2024-04-20T17:15:06.397 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.066579360000000004 2024-04-20T17:15:06.397 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.397 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.397 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:15:06.397 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.102937+0000", 2024-04-20T17:15:06.397 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0043129550000000003 2024-04-20T17:15:06.397 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.397 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.397 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.397 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.009478+0000", 2024-04-20T17:15:06.397 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9065410709999999 2024-04-20T17:15:06.397 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.398 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.398 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:15:06.398 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.015563+0000", 2024-04-20T17:15:06.398 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.006085282 2024-04-20T17:15:06.398 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.398 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.398 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:15:06.398 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.027745+0000", 2024-04-20T17:15:06.398 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012181053000000001 2024-04-20T17:15:06.398 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.398 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.398 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:15:06.398 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.151400+0000", 2024-04-20T17:15:06.398 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.123655958 2024-04-20T17:15:06.399 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.399 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.399 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:15:06.399 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.350233+0000", 2024-04-20T17:15:06.399 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.19883220500000001 2024-04-20T17:15:06.399 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.399 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.399 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:15:06.399 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.355745+0000", 2024-04-20T17:15:06.399 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0055118099999999998 2024-04-20T17:15:06.399 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.399 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.399 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:15:06.399 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.371615+0000", 2024-04-20T17:15:06.399 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015870931000000001 2024-04-20T17:15:06.400 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.400 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:15:06.400 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.400 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.400 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.400 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:228579 2.3 2:c4136e50:::benchmark_data_smithi171_78918_object67585:head [delete] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-20T17:15:06.400 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:31.034417+0000", 2024-04-20T17:15:06.400 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 95.319991408999996, 2024-04-20T17:15:06.400 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3561227750000002, 2024-04-20T17:15:06.400 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:15:06.400 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:15:06.400 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:15:06.400 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:15:06.400 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:15:06.400 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:15:06.401 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 228579 2024-04-20T17:15:06.401 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.401 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:15:06.401 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.401 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:15:06.401 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.034417+0000", 2024-04-20T17:15:06.401 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.401 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.401 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.401 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:15:06.401 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.034417+0000", 2024-04-20T17:15:06.401 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.401 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.401 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.401 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:15:06.402 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.034420+0000", 2024-04-20T17:15:06.402 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.491e-06 2024-04-20T17:15:06.402 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.402 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.402 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:15:06.402 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.034427+0000", 2024-04-20T17:15:06.402 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1010000000000003e-06 2024-04-20T17:15:06.402 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.402 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.402 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:15:06.402 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.034439+0000", 2024-04-20T17:15:06.402 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1761e-05 2024-04-20T17:15:06.402 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.402 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.403 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:15:06.403 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.034448+0000", 2024-04-20T17:15:06.403 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0389999999999993e-06 2024-04-20T17:15:06.403 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.403 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.403 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.403 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.165259+0000", 2024-04-20T17:15:06.403 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.130811069 2024-04-20T17:15:06.403 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.403 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.403 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:15:06.403 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.172966+0000", 2024-04-20T17:15:06.403 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0077074969999999998 2024-04-20T17:15:06.403 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.403 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.404 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.404 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.243877+0000", 2024-04-20T17:15:06.404 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.070910794999999999 2024-04-20T17:15:06.404 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.404 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.404 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:15:06.404 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.248014+0000", 2024-04-20T17:15:06.404 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041367089999999997 2024-04-20T17:15:06.404 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.404 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.404 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.404 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.065976+0000", 2024-04-20T17:15:06.404 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.817961926 2024-04-20T17:15:06.404 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.404 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.405 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:15:06.405 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.072436+0000", 2024-04-20T17:15:06.405 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0064605950000000004 2024-04-20T17:15:06.405 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.405 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.405 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:15:06.405 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.084520+0000", 2024-04-20T17:15:06.405 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012083774 2024-04-20T17:15:06.405 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.405 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.405 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:15:06.405 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.154180+0000", 2024-04-20T17:15:06.405 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.06965971 2024-04-20T17:15:06.405 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.405 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.406 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:15:06.406 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.372323+0000", 2024-04-20T17:15:06.406 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.21814293600000001 2024-04-20T17:15:06.406 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.406 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.406 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:15:06.406 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.375811+0000", 2024-04-20T17:15:06.406 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0034884730000000002 2024-04-20T17:15:06.406 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.406 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.406 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:15:06.406 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.390540+0000", 2024-04-20T17:15:06.406 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014728899 2024-04-20T17:15:06.406 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.407 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:15:06.407 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.407 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.407 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.407 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:228600 2.3 2:c07eaf5a:::benchmark_data_smithi171_78918_object67606:head [delete] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-20T17:15:06.407 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:31.132443+0000", 2024-04-20T17:15:06.407 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 95.221966140000006, 2024-04-20T17:15:06.407 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2809894110000002, 2024-04-20T17:15:06.407 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:15:06.407 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:15:06.407 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:15:06.407 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:15:06.407 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:15:06.407 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:15:06.407 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 228600 2024-04-20T17:15:06.408 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.408 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:15:06.408 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.408 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:15:06.408 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.132443+0000", 2024-04-20T17:15:06.408 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.408 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.408 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.408 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:15:06.408 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.132443+0000", 2024-04-20T17:15:06.408 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.408 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.408 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.408 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:15:06.408 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.132445+0000", 2024-04-20T17:15:06.409 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3250000000000002e-06 2024-04-20T17:15:06.409 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.409 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.409 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:15:06.409 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.132452+0000", 2024-04-20T17:15:06.409 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3669999999999999e-06 2024-04-20T17:15:06.409 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.409 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.409 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:15:06.409 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.132467+0000", 2024-04-20T17:15:06.409 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4446e-05 2024-04-20T17:15:06.409 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.409 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.409 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:15:06.409 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.132477+0000", 2024-04-20T17:15:06.410 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0563e-05 2024-04-20T17:15:06.410 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.410 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.410 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.410 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.202711+0000", 2024-04-20T17:15:06.410 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.070233400000000001 2024-04-20T17:15:06.410 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.410 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.410 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:15:06.410 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.214357+0000", 2024-04-20T17:15:06.410 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011646409 2024-04-20T17:15:06.410 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.410 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.410 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.411 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.275337+0000", 2024-04-20T17:15:06.411 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.060979829999999999 2024-04-20T17:15:06.411 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.411 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.411 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:15:06.411 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.281042+0000", 2024-04-20T17:15:06.411 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0057048949999999998 2024-04-20T17:15:06.411 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.411 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.411 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.411 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.122534+0000", 2024-04-20T17:15:06.411 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.841492031 2024-04-20T17:15:06.411 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.411 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.411 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:15:06.412 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.128841+0000", 2024-04-20T17:15:06.412 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0063070879999999998 2024-04-20T17:15:06.412 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.412 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.412 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:15:06.412 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.141175+0000", 2024-04-20T17:15:06.412 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012334401 2024-04-20T17:15:06.412 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.412 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.412 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:15:06.412 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.156924+0000", 2024-04-20T17:15:06.412 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015748330000000001 2024-04-20T17:15:06.412 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.412 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.412 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:15:06.413 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.391193+0000", 2024-04-20T17:15:06.413 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.23426899000000001 2024-04-20T17:15:06.413 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.413 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.413 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:15:06.413 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.394659+0000", 2024-04-20T17:15:06.413 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0034663749999999998 2024-04-20T17:15:06.413 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.413 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.413 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:15:06.413 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.413432+0000", 2024-04-20T17:15:06.413 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.018772961000000001 2024-04-20T17:15:06.413 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.413 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:15:06.413 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.414 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.414 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.414 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:228644 2.3 2:c8d65292:::benchmark_data_smithi171_78918_object67650:head [delete] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-20T17:15:06.414 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:31.656045+0000", 2024-04-20T17:15:06.414 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 94.698363842000006, 2024-04-20T17:15:06.414 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.428389734, 2024-04-20T17:15:06.414 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:15:06.414 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:15:06.414 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:15:06.414 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:15:06.414 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:15:06.414 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:15:06.414 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 228644 2024-04-20T17:15:06.414 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.415 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:15:06.415 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.415 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:15:06.415 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.656045+0000", 2024-04-20T17:15:06.415 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.415 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.415 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.415 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:15:06.415 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.656045+0000", 2024-04-20T17:15:06.415 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.415 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.415 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.415 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:15:06.415 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.656048+0000", 2024-04-20T17:15:06.415 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6620000000000001e-06 2024-04-20T17:15:06.416 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.416 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.416 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:15:06.416 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.656054+0000", 2024-04-20T17:15:06.416 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8000000000000001e-06 2024-04-20T17:15:06.416 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.416 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.416 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:15:06.416 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.656067+0000", 2024-04-20T17:15:06.416 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2381999999999999e-05 2024-04-20T17:15:06.416 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.416 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.416 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:15:06.416 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.656076+0000", 2024-04-20T17:15:06.416 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6780000000000006e-06 2024-04-20T17:15:06.417 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.417 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.417 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.417 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.680283+0000", 2024-04-20T17:15:06.417 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.024207619 2024-04-20T17:15:06.417 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.417 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.417 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:15:06.417 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.687974+0000", 2024-04-20T17:15:06.417 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0076904490000000002 2024-04-20T17:15:06.417 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.417 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.417 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.417 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.747637+0000", 2024-04-20T17:15:06.417 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.059663348999999997 2024-04-20T17:15:06.418 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.418 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.418 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:15:06.418 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.751859+0000", 2024-04-20T17:15:06.418 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0042222179999999998 2024-04-20T17:15:06.418 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.418 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.418 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.418 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.969716+0000", 2024-04-20T17:15:06.418 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2178571630000001 2024-04-20T17:15:06.418 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.418 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.418 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:15:06.418 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.977217+0000", 2024-04-20T17:15:06.419 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0075008410000000003 2024-04-20T17:15:06.419 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.419 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.419 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:15:06.419 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.989454+0000", 2024-04-20T17:15:06.419 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012237125 2024-04-20T17:15:06.419 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.419 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.419 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:15:06.419 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:34.005410+0000", 2024-04-20T17:15:06.419 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015955666 2024-04-20T17:15:06.419 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.419 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.419 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:15:06.419 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:34.066358+0000", 2024-04-20T17:15:06.420 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.060947875999999998 2024-04-20T17:15:06.420 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.420 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.420 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:15:06.420 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:34.070486+0000", 2024-04-20T17:15:06.420 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041281529999999999 2024-04-20T17:15:06.420 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.420 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.420 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:15:06.420 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:34.084435+0000", 2024-04-20T17:15:06.420 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013948752999999999 2024-04-20T17:15:06.420 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.420 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:15:06.420 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.420 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.421 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.421 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:228734 2.3 2:c6201dcd:::benchmark_data_smithi171_78918_object67740:head [delete] snapc 0=[] ondisk+write+known_if_redirected e229)", 2024-04-20T17:15:06.421 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:32.283347+0000", 2024-04-20T17:15:06.421 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 94.071061892000003, 2024-04-20T17:15:06.421 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3561478459999998, 2024-04-20T17:15:06.421 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:15:06.421 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:15:06.421 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:15:06.421 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:15:06.421 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:15:06.421 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:15:06.421 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 228734 2024-04-20T17:15:06.421 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.421 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:15:06.421 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.422 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:15:06.422 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.283347+0000", 2024-04-20T17:15:06.422 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.422 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.422 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.422 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:15:06.422 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.283347+0000", 2024-04-20T17:15:06.422 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.422 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.422 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.422 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:15:06.422 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.283348+0000", 2024-04-20T17:15:06.422 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4950000000000001e-06 2024-04-20T17:15:06.422 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.422 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.423 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:15:06.423 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.283353+0000", 2024-04-20T17:15:06.423 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.006e-06 2024-04-20T17:15:06.423 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.423 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.423 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:15:06.423 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.283365+0000", 2024-04-20T17:15:06.423 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1686e-05 2024-04-20T17:15:06.423 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.423 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.423 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:15:06.423 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.283374+0000", 2024-04-20T17:15:06.423 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4239999999999993e-06 2024-04-20T17:15:06.423 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.424 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.424 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.424 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.315956+0000", 2024-04-20T17:15:06.424 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.032582425999999998 2024-04-20T17:15:06.424 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.424 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.424 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:15:06.424 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.324563+0000", 2024-04-20T17:15:06.424 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0086066839999999999 2024-04-20T17:15:06.424 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.424 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.424 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.424 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.414971+0000", 2024-04-20T17:15:06.424 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.090408295 2024-04-20T17:15:06.424 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.425 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.425 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:15:06.425 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.420910+0000", 2024-04-20T17:15:06.425 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0059390040000000003 2024-04-20T17:15:06.425 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.425 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.425 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.425 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:34.528387+0000", 2024-04-20T17:15:06.425 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1074769180000001 2024-04-20T17:15:06.425 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.425 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.425 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:15:06.425 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:34.534511+0000", 2024-04-20T17:15:06.425 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0061236490000000001 2024-04-20T17:15:06.425 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.426 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.426 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:15:06.426 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:34.546825+0000", 2024-04-20T17:15:06.426 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012314794 2024-04-20T17:15:06.426 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.426 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.426 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:15:06.426 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:34.614264+0000", 2024-04-20T17:15:06.426 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.067438940000000003 2024-04-20T17:15:06.426 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.426 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.426 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:15:06.426 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:34.620647+0000", 2024-04-20T17:15:06.426 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0063823370000000001 2024-04-20T17:15:06.427 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.427 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.427 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:15:06.427 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:34.624696+0000", 2024-04-20T17:15:06.427 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0040494559999999999 2024-04-20T17:15:06.427 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.427 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.427 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:15:06.427 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:34.639495+0000", 2024-04-20T17:15:06.427 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014798732 2024-04-20T17:15:06.427 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.427 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:15:06.427 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.427 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.427 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.428 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:228769 2.3 2:c47e88d9:::benchmark_data_smithi171_78918_object67775:head [delete] snapc 0=[] ondisk+write+known_if_redirected e229)", 2024-04-20T17:15:06.428 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:32.812925+0000", 2024-04-20T17:15:06.428 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 93.541483783999993, 2024-04-20T17:15:06.428 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3076056889999998, 2024-04-20T17:15:06.428 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:15:06.428 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:15:06.428 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:15:06.428 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:15:06.428 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:15:06.428 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:15:06.428 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 228769 2024-04-20T17:15:06.428 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.428 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:15:06.428 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.428 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:15:06.429 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.812925+0000", 2024-04-20T17:15:06.429 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.429 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.429 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.429 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:15:06.429 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.812925+0000", 2024-04-20T17:15:06.429 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.429 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.429 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.429 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:15:06.429 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.812928+0000", 2024-04-20T17:15:06.429 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.994e-06 2024-04-20T17:15:06.429 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.429 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.429 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:15:06.430 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.812937+0000", 2024-04-20T17:15:06.430 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0869999999999996e-06 2024-04-20T17:15:06.430 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.430 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.430 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:15:06.430 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.812950+0000", 2024-04-20T17:15:06.430 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3227999999999999e-05 2024-04-20T17:15:06.430 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.430 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.430 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:15:06.430 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.812960+0000", 2024-04-20T17:15:06.430 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4040000000000001e-06 2024-04-20T17:15:06.430 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.430 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.431 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.431 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.844995+0000", 2024-04-20T17:15:06.431 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.032035149999999998 2024-04-20T17:15:06.431 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.431 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.431 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:15:06.431 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.852543+0000", 2024-04-20T17:15:06.431 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.007547893 2024-04-20T17:15:06.431 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.431 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.431 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.431 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.953194+0000", 2024-04-20T17:15:06.431 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10065070499999999 2024-04-20T17:15:06.431 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.431 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.432 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:15:06.432 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.957382+0000", 2024-04-20T17:15:06.432 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041883729999999996 2024-04-20T17:15:06.432 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.432 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.432 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.432 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:35.055854+0000", 2024-04-20T17:15:06.432 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0984725329999998 2024-04-20T17:15:06.432 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.432 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.432 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:15:06.432 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:35.061870+0000", 2024-04-20T17:15:06.432 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0060152859999999999 2024-04-20T17:15:06.432 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.432 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.432 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:15:06.433 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:35.074203+0000", 2024-04-20T17:15:06.433 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012333191 2024-04-20T17:15:06.433 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.433 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.433 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:15:06.433 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:35.094118+0000", 2024-04-20T17:15:06.433 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.019914630999999999 2024-04-20T17:15:06.433 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.433 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.433 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:15:06.433 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:35.101255+0000", 2024-04-20T17:15:06.433 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0071371840000000004 2024-04-20T17:15:06.433 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.433 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.433 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:15:06.434 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:35.105408+0000", 2024-04-20T17:15:06.434 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.004153505 2024-04-20T17:15:06.434 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.434 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.434 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:15:06.434 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:35.120531+0000", 2024-04-20T17:15:06.434 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015122525 2024-04-20T17:15:06.434 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.434 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:15:06.434 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.434 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.434 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.434 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:228913 2.3 2:c8d17bf9:::benchmark_data_smithi171_78918_object67919:head [delete] snapc 0=[] ondisk+write+known_if_redirected e229)", 2024-04-20T17:15:06.434 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:33.652143+0000", 2024-04-20T17:15:06.434 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 92.702265999000005, 2024-04-20T17:15:06.435 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3437341059999999, 2024-04-20T17:15:06.435 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:15:06.435 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:15:06.435 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:15:06.435 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:15:06.435 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:15:06.435 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:15:06.435 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 228913 2024-04-20T17:15:06.435 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.435 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:15:06.435 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.435 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:15:06.435 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.652143+0000", 2024-04-20T17:15:06.435 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.435 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.436 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.436 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:15:06.436 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.652143+0000", 2024-04-20T17:15:06.436 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.436 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.436 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.436 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:15:06.436 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.652145+0000", 2024-04-20T17:15:06.436 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0360000000000001e-06 2024-04-20T17:15:06.436 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.436 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.436 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:15:06.436 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.652151+0000", 2024-04-20T17:15:06.436 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0440000000000001e-06 2024-04-20T17:15:06.437 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.437 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.437 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:15:06.437 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.652161+0000", 2024-04-20T17:15:06.437 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0419e-05 2024-04-20T17:15:06.437 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.437 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.437 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:15:06.437 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.652168+0000", 2024-04-20T17:15:06.437 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4749999999999998e-06 2024-04-20T17:15:06.437 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.437 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.437 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.437 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.751065+0000", 2024-04-20T17:15:06.438 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.098896703000000002 2024-04-20T17:15:06.438 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.438 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.438 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:15:06.438 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.759742+0000", 2024-04-20T17:15:06.438 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0086775240000000007 2024-04-20T17:15:06.438 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.438 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.438 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.438 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.829542+0000", 2024-04-20T17:15:06.438 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.069800143999999995 2024-04-20T17:15:06.438 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.438 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.438 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:15:06.439 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.834830+0000", 2024-04-20T17:15:06.439 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0052875709999999996 2024-04-20T17:15:06.439 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.439 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.439 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.439 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:35.831588+0000", 2024-04-20T17:15:06.439 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.996758094 2024-04-20T17:15:06.439 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.439 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.439 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:15:06.439 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:35.838563+0000", 2024-04-20T17:15:06.439 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0069750209999999997 2024-04-20T17:15:06.439 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.439 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.439 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:15:06.440 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:35.857019+0000", 2024-04-20T17:15:06.440 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.018455849 2024-04-20T17:15:06.440 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.440 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.440 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:15:06.440 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:35.873901+0000", 2024-04-20T17:15:06.440 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.016882766 2024-04-20T17:15:06.440 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.440 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.440 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:15:06.440 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:35.978024+0000", 2024-04-20T17:15:06.440 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.104122649 2024-04-20T17:15:06.440 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.440 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.440 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:15:06.441 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:35.982208+0000", 2024-04-20T17:15:06.441 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041836479999999999 2024-04-20T17:15:06.441 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.441 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.441 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:15:06.441 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:35.995877+0000", 2024-04-20T17:15:06.441 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013669163 2024-04-20T17:15:06.441 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.441 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:15:06.441 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.441 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.441 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.441 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:231279 2.3 2:c2fe291f:::benchmark_data_smithi171_78918_object70285:head [delete] snapc 0=[] ondisk+write+known_if_redirected e236)", 2024-04-20T17:15:06.441 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:52.016993+0000", 2024-04-20T17:15:06.442 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 74.337416305000005, 2024-04-20T17:15:06.442 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2714570009999999, 2024-04-20T17:15:06.442 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:15:06.442 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:15:06.442 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:15:06.442 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:15:06.442 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:15:06.442 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:15:06.442 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 231279 2024-04-20T17:15:06.442 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.442 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:15:06.442 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.442 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:15:06.442 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:52.016993+0000", 2024-04-20T17:15:06.442 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.443 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.443 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.443 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:15:06.443 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:52.016993+0000", 2024-04-20T17:15:06.443 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.443 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.443 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.443 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:15:06.443 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:52.016995+0000", 2024-04-20T17:15:06.443 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1270000000000001e-06 2024-04-20T17:15:06.443 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.443 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.443 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:15:06.443 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:52.017002+0000", 2024-04-20T17:15:06.444 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6410000000000002e-06 2024-04-20T17:15:06.444 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.444 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.444 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:15:06.444 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:52.017016+0000", 2024-04-20T17:15:06.444 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3497e-05 2024-04-20T17:15:06.444 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.444 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.444 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:15:06.444 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:52.017024+0000", 2024-04-20T17:15:06.444 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9820000000000008e-06 2024-04-20T17:15:06.444 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.444 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.444 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.444 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:52.159692+0000", 2024-04-20T17:15:06.445 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.14266816299999999 2024-04-20T17:15:06.445 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.445 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.445 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:15:06.445 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:52.167209+0000", 2024-04-20T17:15:06.445 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.007517429 2024-04-20T17:15:06.445 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.445 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.445 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.445 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:52.226262+0000", 2024-04-20T17:15:06.445 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.059052220000000002 2024-04-20T17:15:06.445 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.445 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.445 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:15:06.445 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:52.230323+0000", 2024-04-20T17:15:06.446 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0040616610000000003 2024-04-20T17:15:06.446 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.446 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.446 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.446 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:54.183729+0000", 2024-04-20T17:15:06.446 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.953405474 2024-04-20T17:15:06.446 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.446 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.446 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:15:06.446 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:54.189801+0000", 2024-04-20T17:15:06.446 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0060722179999999999 2024-04-20T17:15:06.446 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.446 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.446 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:15:06.447 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:54.201895+0000", 2024-04-20T17:15:06.447 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.01209437 2024-04-20T17:15:06.447 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.447 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.447 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:15:06.447 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:54.269225+0000", 2024-04-20T17:15:06.447 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.067329489000000006 2024-04-20T17:15:06.447 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.447 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.447 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:15:06.447 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:54.270554+0000", 2024-04-20T17:15:06.447 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001329305 2024-04-20T17:15:06.447 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.447 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.447 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:15:06.448 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:54.274637+0000", 2024-04-20T17:15:06.448 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.004082919 2024-04-20T17:15:06.448 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.448 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.448 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:15:06.448 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:54.288450+0000", 2024-04-20T17:15:06.448 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013812506 2024-04-20T17:15:06.448 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.448 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:15:06.448 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.448 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.448 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.448 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:231451 2.3 2:ce942c68:::benchmark_data_smithi171_78918_object70457:head [delete] snapc 0=[] ondisk+write+known_if_redirected e237)", 2024-04-20T17:15:06.448 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:53.608704+0000", 2024-04-20T17:15:06.448 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 72.745704379000003, 2024-04-20T17:15:06.448 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3313055249999999, 2024-04-20T17:15:06.449 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:15:06.449 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:15:06.449 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:15:06.449 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:15:06.449 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:15:06.449 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:15:06.449 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 231451 2024-04-20T17:15:06.449 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.449 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:15:06.449 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.449 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:15:06.449 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:53.608704+0000", 2024-04-20T17:15:06.449 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.449 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.450 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.450 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:15:06.450 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:53.608704+0000", 2024-04-20T17:15:06.450 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.450 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.450 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.450 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:15:06.450 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:53.608707+0000", 2024-04-20T17:15:06.450 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3130000000000001e-06 2024-04-20T17:15:06.450 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.450 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.450 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:15:06.450 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:53.608714+0000", 2024-04-20T17:15:06.450 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4420000000000001e-06 2024-04-20T17:15:06.450 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.451 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.451 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:15:06.451 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:53.608729+0000", 2024-04-20T17:15:06.451 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4284000000000001e-05 2024-04-20T17:15:06.451 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.451 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.451 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:15:06.451 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:53.608740+0000", 2024-04-20T17:15:06.451 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1148999999999999e-05 2024-04-20T17:15:06.451 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.451 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.451 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.451 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:53.679980+0000", 2024-04-20T17:15:06.451 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.071240205000000001 2024-04-20T17:15:06.451 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.452 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.452 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:15:06.452 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:53.692861+0000", 2024-04-20T17:15:06.452 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012881182 2024-04-20T17:15:06.452 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.452 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.452 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.452 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:53.787511+0000", 2024-04-20T17:15:06.452 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.094649464000000003 2024-04-20T17:15:06.452 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.452 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.452 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:15:06.452 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:53.791542+0000", 2024-04-20T17:15:06.452 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0040310779999999996 2024-04-20T17:15:06.452 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.453 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.453 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.453 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:55.833719+0000", 2024-04-20T17:15:06.453 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0421774479999999 2024-04-20T17:15:06.453 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.453 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.453 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:15:06.453 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:55.839813+0000", 2024-04-20T17:15:06.453 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0060943270000000001 2024-04-20T17:15:06.453 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.453 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.453 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:15:06.453 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:55.851947+0000", 2024-04-20T17:15:06.453 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.01213349 2024-04-20T17:15:06.453 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.454 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.454 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:15:06.454 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:55.919140+0000", 2024-04-20T17:15:06.454 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.067192937999999994 2024-04-20T17:15:06.454 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.454 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.454 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:15:06.454 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:55.920484+0000", 2024-04-20T17:15:06.454 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0013444329999999999 2024-04-20T17:15:06.454 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.454 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.454 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:15:06.454 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:55.924525+0000", 2024-04-20T17:15:06.454 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0040412649999999996 2024-04-20T17:15:06.455 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.455 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.455 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:15:06.455 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:55.940010+0000", 2024-04-20T17:15:06.455 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015484507 2024-04-20T17:15:06.455 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.455 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:15:06.455 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.455 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.455 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.455 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:233551 2.3 2:c717208a:::benchmark_data_smithi171_78918_object72557:head [delete] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-04-20T17:15:06.455 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:14:09.264468+0000", 2024-04-20T17:15:06.455 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 57.089940943000002, 2024-04-20T17:15:06.455 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3236778899999999, 2024-04-20T17:15:06.455 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:15:06.456 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:15:06.456 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:15:06.456 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:15:06.456 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:15:06.456 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:15:06.456 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 233551 2024-04-20T17:15:06.456 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.456 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:15:06.456 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.456 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:15:06.456 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.264468+0000", 2024-04-20T17:15:06.456 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.456 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.456 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.456 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:15:06.457 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.264468+0000", 2024-04-20T17:15:06.457 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.457 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.457 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.457 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:15:06.457 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.264470+0000", 2024-04-20T17:15:06.457 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5390000000000002e-06 2024-04-20T17:15:06.457 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.457 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.457 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:15:06.457 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.264478+0000", 2024-04-20T17:15:06.457 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.294e-06 2024-04-20T17:15:06.457 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.457 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.457 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:15:06.457 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.264491+0000", 2024-04-20T17:15:06.458 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3305999999999999e-05 2024-04-20T17:15:06.458 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.458 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.458 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:15:06.458 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.264502+0000", 2024-04-20T17:15:06.458 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1326e-05 2024-04-20T17:15:06.458 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.458 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.458 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.458 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.427957+0000", 2024-04-20T17:15:06.458 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.16345464500000001 2024-04-20T17:15:06.458 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.458 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.458 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:15:06.458 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.435153+0000", 2024-04-20T17:15:06.459 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.007196282 2024-04-20T17:15:06.459 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.459 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.459 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.459 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.490235+0000", 2024-04-20T17:15:06.459 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.055081443000000001 2024-04-20T17:15:06.459 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.459 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.459 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:15:06.459 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.494200+0000", 2024-04-20T17:15:06.459 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003965161 2024-04-20T17:15:06.459 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.459 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.459 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.459 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:11.427618+0000", 2024-04-20T17:15:06.460 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9334183870000001 2024-04-20T17:15:06.460 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.460 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.460 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:15:06.460 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:11.433616+0000", 2024-04-20T17:15:06.460 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0059977229999999999 2024-04-20T17:15:06.460 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.460 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.460 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:15:06.460 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:11.445335+0000", 2024-04-20T17:15:06.460 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011719283 2024-04-20T17:15:06.460 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.460 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.460 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:15:06.460 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:11.537472+0000", 2024-04-20T17:15:06.460 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.092136309 2024-04-20T17:15:06.461 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.461 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.461 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:15:06.461 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:11.570480+0000", 2024-04-20T17:15:06.461 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.033008570000000001 2024-04-20T17:15:06.461 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.461 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.461 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:15:06.461 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:11.575045+0000", 2024-04-20T17:15:06.461 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0045648030000000001 2024-04-20T17:15:06.461 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.461 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.461 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:15:06.461 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:11.588146+0000", 2024-04-20T17:15:06.462 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013100819 2024-04-20T17:15:06.462 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.462 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:15:06.462 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.462 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.462 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.462 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:233592 2.3 2:c6ee60e5:::benchmark_data_smithi171_78918_object72598:head [delete] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-04-20T17:15:06.462 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:14:09.873340+0000", 2024-04-20T17:15:06.462 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 56.481069003999998, 2024-04-20T17:15:06.462 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2808158880000002, 2024-04-20T17:15:06.462 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:15:06.462 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:15:06.462 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:15:06.462 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:15:06.462 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:15:06.463 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:15:06.463 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 233592 2024-04-20T17:15:06.463 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.463 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:15:06.463 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.463 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:15:06.463 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.873340+0000", 2024-04-20T17:15:06.463 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.463 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.463 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.463 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:15:06.463 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.873340+0000", 2024-04-20T17:15:06.463 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.463 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.464 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.464 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:15:06.464 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.873343+0000", 2024-04-20T17:15:06.464 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5429999999999998e-06 2024-04-20T17:15:06.464 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.464 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.464 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:15:06.464 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.873354+0000", 2024-04-20T17:15:06.464 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0825999999999999e-05 2024-04-20T17:15:06.464 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.464 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.464 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:15:06.464 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.873372+0000", 2024-04-20T17:15:06.464 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7694999999999999e-05 2024-04-20T17:15:06.464 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.465 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.465 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:15:06.465 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.873385+0000", 2024-04-20T17:15:06.465 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3079e-05 2024-04-20T17:15:06.465 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.465 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.465 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.465 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.893470+0000", 2024-04-20T17:15:06.465 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.020085408999999999 2024-04-20T17:15:06.465 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.465 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.465 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:15:06.465 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.900736+0000", 2024-04-20T17:15:06.465 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0072656650000000001 2024-04-20T17:15:06.465 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.466 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.466 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.466 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.962755+0000", 2024-04-20T17:15:06.466 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.062019078999999998 2024-04-20T17:15:06.466 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.466 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.466 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:15:06.466 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.966656+0000", 2024-04-20T17:15:06.466 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0039011599999999999 2024-04-20T17:15:06.466 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.466 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.466 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.466 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.099651+0000", 2024-04-20T17:15:06.466 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1329947539999998 2024-04-20T17:15:06.467 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.467 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.467 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:15:06.467 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.105504+0000", 2024-04-20T17:15:06.467 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0058531349999999998 2024-04-20T17:15:06.467 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.467 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.467 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:15:06.467 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.117342+0000", 2024-04-20T17:15:06.467 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011837627999999999 2024-04-20T17:15:06.467 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.467 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.467 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:15:06.467 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.133991+0000", 2024-04-20T17:15:06.467 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.016649125000000001 2024-04-20T17:15:06.468 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.468 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.468 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:15:06.468 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.136266+0000", 2024-04-20T17:15:06.468 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0022746450000000001 2024-04-20T17:15:06.468 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.468 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.468 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:15:06.468 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.140799+0000", 2024-04-20T17:15:06.468 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.004533863 2024-04-20T17:15:06.468 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.468 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.468 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:15:06.468 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.154156+0000", 2024-04-20T17:15:06.469 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013356282000000001 2024-04-20T17:15:06.469 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.469 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:15:06.469 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.469 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.469 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.469 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:233665 2.3 2:ce74c18b:::benchmark_data_smithi171_78918_object72671:head [delete] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-04-20T17:15:06.469 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:14:10.405351+0000", 2024-04-20T17:15:06.469 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 55.949058078, 2024-04-20T17:15:06.469 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.294950746, 2024-04-20T17:15:06.469 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:15:06.469 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:15:06.469 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:15:06.469 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:15:06.470 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:15:06.470 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:15:06.470 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 233665 2024-04-20T17:15:06.470 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.470 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:15:06.470 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.470 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:15:06.470 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.405351+0000", 2024-04-20T17:15:06.470 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.470 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.470 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.470 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:15:06.470 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.405351+0000", 2024-04-20T17:15:06.470 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.470 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.471 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.471 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:15:06.471 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.405354+0000", 2024-04-20T17:15:06.471 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8600000000000001e-06 2024-04-20T17:15:06.471 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.471 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.471 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:15:06.471 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.405367+0000", 2024-04-20T17:15:06.471 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3407000000000001e-05 2024-04-20T17:15:06.471 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.471 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.471 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:15:06.471 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.405380+0000", 2024-04-20T17:15:06.471 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2792999999999999e-05 2024-04-20T17:15:06.471 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.472 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.472 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:15:06.472 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.405389+0000", 2024-04-20T17:15:06.472 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2520000000000005e-06 2024-04-20T17:15:06.472 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.472 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.472 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.472 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.560324+0000", 2024-04-20T17:15:06.472 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.15493442399999999 2024-04-20T17:15:06.472 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.472 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.472 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:15:06.472 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.570571+0000", 2024-04-20T17:15:06.472 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010247473 2024-04-20T17:15:06.472 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.473 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.473 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.473 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.627623+0000", 2024-04-20T17:15:06.473 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057051727000000003 2024-04-20T17:15:06.473 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.473 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.473 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:15:06.473 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.631562+0000", 2024-04-20T17:15:06.473 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0039391640000000002 2024-04-20T17:15:06.473 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.473 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.473 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.473 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.562874+0000", 2024-04-20T17:15:06.473 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9313125119999999 2024-04-20T17:15:06.473 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.473 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.474 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:15:06.474 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.568721+0000", 2024-04-20T17:15:06.474 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0058464770000000001 2024-04-20T17:15:06.474 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.474 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.474 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:15:06.474 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.580398+0000", 2024-04-20T17:15:06.474 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011677481999999999 2024-04-20T17:15:06.474 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.474 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.474 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:15:06.474 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.598120+0000", 2024-04-20T17:15:06.474 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.017722090999999999 2024-04-20T17:15:06.474 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.475 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.475 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:15:06.475 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.683063+0000", 2024-04-20T17:15:06.475 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.084942387999999994 2024-04-20T17:15:06.475 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.475 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.475 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:15:06.475 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.686953+0000", 2024-04-20T17:15:06.475 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003890096 2024-04-20T17:15:06.475 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.475 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.475 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:15:06.475 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.700302+0000", 2024-04-20T17:15:06.475 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0133486 2024-04-20T17:15:06.475 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.476 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:15:06.476 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.476 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.476 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.476 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:233667 2.3 2:c866dd2f:::benchmark_data_smithi171_78918_object72673:head [delete] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-04-20T17:15:06.476 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:14:10.550057+0000", 2024-04-20T17:15:06.476 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 55.804352059999999, 2024-04-20T17:15:06.476 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3848929349999999, 2024-04-20T17:15:06.476 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:15:06.476 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:15:06.476 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:15:06.476 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:15:06.476 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:15:06.476 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:15:06.476 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 233667 2024-04-20T17:15:06.477 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.477 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:15:06.477 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.477 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:15:06.477 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.550057+0000", 2024-04-20T17:15:06.477 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.477 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.477 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.477 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:15:06.477 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.550057+0000", 2024-04-20T17:15:06.477 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.477 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.477 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.477 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:15:06.477 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.550061+0000", 2024-04-20T17:15:06.478 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3470000000000001e-06 2024-04-20T17:15:06.478 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.478 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.478 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:15:06.478 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.550074+0000", 2024-04-20T17:15:06.478 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3032e-05 2024-04-20T17:15:06.478 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.478 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.478 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:15:06.478 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.550089+0000", 2024-04-20T17:15:06.478 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5142e-05 2024-04-20T17:15:06.478 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.478 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.478 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:15:06.479 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.550100+0000", 2024-04-20T17:15:06.479 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0626e-05 2024-04-20T17:15:06.479 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.479 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.479 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.479 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.632247+0000", 2024-04-20T17:15:06.479 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.082146648000000003 2024-04-20T17:15:06.479 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.479 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.479 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:15:06.479 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.639860+0000", 2024-04-20T17:15:06.479 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0076133900000000003 2024-04-20T17:15:06.479 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.479 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.479 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.480 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.674992+0000", 2024-04-20T17:15:06.480 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.035132413000000001 2024-04-20T17:15:06.480 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.480 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.480 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:15:06.480 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.678995+0000", 2024-04-20T17:15:06.480 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0040029799999999997 2024-04-20T17:15:06.480 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.480 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.480 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.480 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.738116+0000", 2024-04-20T17:15:06.480 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.059120294 2024-04-20T17:15:06.480 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.480 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.480 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:15:06.481 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.746723+0000", 2024-04-20T17:15:06.481 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0086068800000000008 2024-04-20T17:15:06.481 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.481 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.481 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:15:06.481 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.763749+0000", 2024-04-20T17:15:06.481 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.017026935 2024-04-20T17:15:06.481 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.481 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.481 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:15:06.481 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.894660+0000", 2024-04-20T17:15:06.481 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.13091038299999999 2024-04-20T17:15:06.481 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.481 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.481 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:15:06.482 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.918102+0000", 2024-04-20T17:15:06.482 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023442187999999999 2024-04-20T17:15:06.482 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.482 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.482 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:15:06.482 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.921919+0000", 2024-04-20T17:15:06.482 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0038171429999999998 2024-04-20T17:15:06.482 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.482 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.482 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:15:06.482 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.934950+0000", 2024-04-20T17:15:06.482 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013030534 2024-04-20T17:15:06.482 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.482 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:15:06.483 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.483 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.483 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.483 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:233673 2.3 2:c40ab28f:::benchmark_data_smithi171_78918_object72679:head [delete] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-04-20T17:15:06.483 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:14:10.809835+0000", 2024-04-20T17:15:06.483 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 55.544573855000003, 2024-04-20T17:15:06.483 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2900486679999998, 2024-04-20T17:15:06.483 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:15:06.483 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:15:06.483 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:15:06.483 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:15:06.483 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:15:06.483 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:15:06.483 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 233673 2024-04-20T17:15:06.483 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.483 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:15:06.484 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.484 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:15:06.484 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.809835+0000", 2024-04-20T17:15:06.484 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.484 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.484 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.484 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:15:06.484 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.809835+0000", 2024-04-20T17:15:06.484 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.484 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.484 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.484 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:15:06.484 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.809838+0000", 2024-04-20T17:15:06.484 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9059999999999998e-06 2024-04-20T17:15:06.484 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.485 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.485 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:15:06.485 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.809847+0000", 2024-04-20T17:15:06.485 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7430000000000008e-06 2024-04-20T17:15:06.485 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.485 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.485 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:15:06.485 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.809861+0000", 2024-04-20T17:15:06.485 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4182000000000001e-05 2024-04-20T17:15:06.485 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.485 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.485 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:15:06.485 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.809873+0000", 2024-04-20T17:15:06.485 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1800000000000001e-05 2024-04-20T17:15:06.485 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.486 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.486 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.486 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.954883+0000", 2024-04-20T17:15:06.486 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.14501086199999999 2024-04-20T17:15:06.486 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.486 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.486 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:15:06.486 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.962267+0000", 2024-04-20T17:15:06.486 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0073832510000000004 2024-04-20T17:15:06.486 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.486 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.486 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.486 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:11.074968+0000", 2024-04-20T17:15:06.486 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.112701309 2024-04-20T17:15:06.486 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.487 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.487 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:15:06.487 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:11.079464+0000", 2024-04-20T17:15:06.487 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0044959429999999996 2024-04-20T17:15:06.487 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.487 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.487 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.487 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.813486+0000", 2024-04-20T17:15:06.487 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.73402235 2024-04-20T17:15:06.487 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.487 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.487 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:15:06.487 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.819390+0000", 2024-04-20T17:15:06.487 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0059034769999999999 2024-04-20T17:15:06.487 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.487 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.488 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:15:06.488 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.831066+0000", 2024-04-20T17:15:06.488 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011675792000000001 2024-04-20T17:15:06.488 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.488 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.488 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:15:06.488 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.897304+0000", 2024-04-20T17:15:06.488 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.066238067999999997 2024-04-20T17:15:06.488 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.488 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.488 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:15:06.488 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:13.082592+0000", 2024-04-20T17:15:06.488 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.18528828899999999 2024-04-20T17:15:06.488 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.488 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.489 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:15:06.489 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:13.086525+0000", 2024-04-20T17:15:06.489 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0039326359999999998 2024-04-20T17:15:06.489 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.489 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.489 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:15:06.489 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:13.099884+0000", 2024-04-20T17:15:06.489 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013359060000000001 2024-04-20T17:15:06.489 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.489 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:15:06.489 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.489 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.489 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.489 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:236418 2.3 2:c3a548fb:::benchmark_data_smithi171_78918_object75424:head [delete] snapc 0=[] ondisk+write+known_if_redirected e250)", 2024-04-20T17:15:06.489 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:14:31.937892+0000", 2024-04-20T17:15:06.490 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 34.416517034999998, 2024-04-20T17:15:06.490 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2949184520000001, 2024-04-20T17:15:06.490 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:15:06.490 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:15:06.490 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:15:06.490 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:15:06.490 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:15:06.490 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:15:06.490 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 236418 2024-04-20T17:15:06.490 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.490 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:15:06.490 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.490 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:15:06.490 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:31.937892+0000", 2024-04-20T17:15:06.490 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.491 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.491 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.491 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:15:06.491 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:31.937892+0000", 2024-04-20T17:15:06.491 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:15:06.491 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.491 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.491 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:15:06.491 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:31.937894+0000", 2024-04-20T17:15:06.491 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.52e-06 2024-04-20T17:15:06.491 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.491 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.491 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:15:06.491 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:31.937902+0000", 2024-04-20T17:15:06.491 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5159999999999997e-06 2024-04-20T17:15:06.492 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.492 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.492 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:15:06.492 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:31.937914+0000", 2024-04-20T17:15:06.492 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2568999999999999e-05 2024-04-20T17:15:06.492 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.492 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.492 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:15:06.492 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:31.937922+0000", 2024-04-20T17:15:06.492 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6729999999999992e-06 2024-04-20T17:15:06.492 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.492 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.492 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.492 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:31.957343+0000", 2024-04-20T17:15:06.492 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.019420957999999999 2024-04-20T17:15:06.493 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.493 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.493 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:15:06.493 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:31.964173+0000", 2024-04-20T17:15:06.493 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0068297610000000002 2024-04-20T17:15:06.493 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.493 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.493 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.493 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:32.049718+0000", 2024-04-20T17:15:06.493 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.085545429000000006 2024-04-20T17:15:06.493 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.493 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.493 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:15:06.493 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:32.053473+0000", 2024-04-20T17:15:06.493 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003754847 2024-04-20T17:15:06.493 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.494 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.494 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:15:06.494 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:34.119663+0000", 2024-04-20T17:15:06.494 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.066189456 2024-04-20T17:15:06.494 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.494 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.494 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:15:06.494 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:34.125255+0000", 2024-04-20T17:15:06.494 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.005592517 2024-04-20T17:15:06.494 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.494 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.494 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:15:06.494 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:34.137707+0000", 2024-04-20T17:15:06.494 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012452201 2024-04-20T17:15:06.494 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.495 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.495 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:15:06.495 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:34.204495+0000", 2024-04-20T17:15:06.495 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.066788058999999997 2024-04-20T17:15:06.495 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.495 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.495 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:15:06.495 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:34.210800+0000", 2024-04-20T17:15:06.495 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0063041970000000001 2024-04-20T17:15:06.495 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.495 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.495 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:15:06.495 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:34.214715+0000", 2024-04-20T17:15:06.495 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0039154510000000003 2024-04-20T17:15:06.495 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:15:06.495 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:15:06.496 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:15:06.496 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:34.232810+0000", 2024-04-20T17:15:06.496 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.018095297999999999 2024-04-20T17:15:06.496 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.496 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:15:06.496 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.496 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:15:06.496 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:15:06.496 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:15:06.496 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:15:06.532 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:06.608 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:15:06.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:06.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:15:06.712 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:15:06.739 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:06.757 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:15:06.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:06.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:15:06.861 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:15:06.887 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:06.906 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:15:06.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:06.909 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:15:07.009 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:15:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:06 smithi106 ceph-mon[44423]: pgmap v722: 17 pgs: 1 active+recovering, 16 active+clean; 5.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 148 op/s; 9085/162874 objects misplaced (5.578%); 875 KiB/s, 13 objects/s recovering 2024-04-20T17:15:07.037 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:07.056 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:15:07.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:07.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:15:07.160 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:15:07.184 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:06 smithi171 ceph-mon[44189]: pgmap v722: 17 pgs: 1 active+recovering, 16 active+clean; 5.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 148 op/s; 9085/162874 objects misplaced (5.578%); 875 KiB/s, 13 objects/s recovering 2024-04-20T17:15:07.186 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:07.204 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:15:07.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:07.207 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:15:07.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:06 smithi005 ceph-mon[39562]: pgmap v722: 17 pgs: 1 active+recovering, 16 active+clean; 5.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 148 op/s; 9085/162874 objects misplaced (5.578%); 875 KiB/s, 13 objects/s recovering 2024-04-20T17:15:07.239 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:15:07.307 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:15:07.338 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:07.367 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:15:07.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:07.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:15:07.471 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:15:07.498 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:07.516 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:15:07.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:07.519 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:15:07.620 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:15:07.647 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:07.667 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:15:07.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:07.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:15:07.770 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:15:07.798 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:07.818 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:15:07.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:07.823 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:15:07.924 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:15:07.954 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:08.261 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-20T17:15:08.703 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:15:08.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:08.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:15:08.809 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:15:08.836 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:08.855 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:15:08.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:08.858 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:15:08.959 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:15:08.988 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:09.316 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:09 smithi005 ceph-mon[39562]: pgmap v723: 17 pgs: 1 active+recovering, 16 active+clean; 5.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 134 op/s; 9085/162264 objects misplaced (5.599%); 369 KiB/s, 5 objects/s recovering 2024-04-20T17:15:09.340 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:15:09.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:09.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:15:09.444 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:15:09.472 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:09.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:09 smithi106 ceph-mon[44423]: pgmap v723: 17 pgs: 1 active+recovering, 16 active+clean; 5.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 134 op/s; 9085/162264 objects misplaced (5.599%); 369 KiB/s, 5 objects/s recovering 2024-04-20T17:15:09.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:09 smithi171 ceph-mon[44189]: pgmap v723: 17 pgs: 1 active+recovering, 16 active+clean; 5.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 134 op/s; 9085/162264 objects misplaced (5.599%); 369 KiB/s, 5 objects/s recovering 2024-04-20T17:15:09.956 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:15:09.958 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:15:10.222 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:15:10.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:10.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:15:10.327 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:15:10.355 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:10.373 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:15:10.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:10.376 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:15:10.476 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:15:10.506 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:10.523 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:15:10.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:10.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:15:10.627 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:15:10.660 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:10.734 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:15:10.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:10.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:15:10.839 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:15:10.865 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:10.884 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:15:10.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:10.886 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:15:10.987 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:15:11.013 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:11.032 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:15:11.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:11.035 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:15:11.136 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:15:11.165 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:11 smithi171 ceph-mon[44189]: pgmap v724: 17 pgs: 1 active+recovering, 16 active+clean; 4.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 129 op/s; 9085/162018 objects misplaced (5.607%); 328 KiB/s, 5 objects/s recovering 2024-04-20T17:15:11.342 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:11 smithi005 ceph-mon[39562]: pgmap v724: 17 pgs: 1 active+recovering, 16 active+clean; 4.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 129 op/s; 9085/162018 objects misplaced (5.607%); 328 KiB/s, 5 objects/s recovering 2024-04-20T17:15:11.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:11 smithi106 ceph-mon[44423]: pgmap v724: 17 pgs: 1 active+recovering, 16 active+clean; 4.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 129 op/s; 9085/162018 objects misplaced (5.607%); 328 KiB/s, 5 objects/s recovering 2024-04-20T17:15:11.715 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:15:11.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:11.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:15:11.819 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:15:11.855 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:12.050 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:15:12.050 INFO:teuthology.orchestra.run.smithi005.stdout:{"fsid":"13c6e76c-ff37-11ee-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":880,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":262,"num_osds":12,"num_up_osds":12,"osd_up_since":1713633141,"num_in_osds":12,"osd_in_since":1713632742,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovering","count":1}],"num_pgs":17,"num_pools":2,"num_objects":81009,"data_bytes":5309399608,"bytes_used":12347101184,"bytes_avail":1139593326592,"bytes_total":1151940427776,"misplaced_objects":9085,"misplaced_total":162018,"misplaced_ratio":0.056074016467306101,"recovering_objects_per_sec":5,"recovering_bytes_per_sec":335811,"recovering_keys_per_sec":0,"num_objects_recovered":41,"num_bytes_recovered":2686976,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":129},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.5:8443/"}},"servicemap":{"epoch":20,"modified":"2024-04-20T17:15:10.033503+0000","services":{"mgr":{"daemons":{"summary":"","y":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","10":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-20T17:15:12.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:12 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1673270430' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:15:12.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:12 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1673270430' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:15:12.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:12 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1673270430' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:15:12.804 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:15:12.810 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:15:12.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:12.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:15:12.827 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:15:12.846 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:15:12.867 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:15:12.888 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:15:12.911 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:15:12.915 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:15:12.931 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:15:12.941 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:12.955 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:15:12.959 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:15:12.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:12.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:15:12.975 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:15:12.995 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:15:13.015 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:15:13.036 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:15:13.037 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:15:13.038 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:15:13.038 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:15:13.038 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:15:13.038 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:15:13.038 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:15:13.038 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:15:13.038 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:15:13.038 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:15:13.038 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:15:13.038 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:15:13.038 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:15:13.062 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:15:13.088 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:13.107 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:15:13.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:13.110 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:15:13.211 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:15:13.235 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:13 smithi106 ceph-mon[44423]: pgmap v725: 17 pgs: 1 active+recovering, 16 active+clean; 4.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 127 op/s; 9002/161380 objects misplaced (5.578%); 650 KiB/s, 10 objects/s recovering 2024-04-20T17:15:13.236 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:13 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4259325661' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:15:13.236 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:13 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:13.237 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:13.255 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:15:13.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:13.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:15:13.359 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:15:13.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:13 smithi005 ceph-mon[39562]: pgmap v725: 17 pgs: 1 active+recovering, 16 active+clean; 4.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 127 op/s; 9002/161380 objects misplaced (5.578%); 650 KiB/s, 10 objects/s recovering 2024-04-20T17:15:13.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:13 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4259325661' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:15:13.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:13 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:13.390 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:13.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:13 smithi171 ceph-mon[44189]: pgmap v725: 17 pgs: 1 active+recovering, 16 active+clean; 4.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 127 op/s; 9002/161380 objects misplaced (5.578%); 650 KiB/s, 10 objects/s recovering 2024-04-20T17:15:13.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:13 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4259325661' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:15:13.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:13 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:13.814 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-20T17:15:13.814 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:15:14.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:15.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:15 smithi005 ceph-mon[39562]: pgmap v726: 17 pgs: 1 active+recovering, 16 active+clean; 4.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 142 op/s; 9002/160582 objects misplaced (5.606%); 525 KiB/s, 8 objects/s recovering 2024-04-20T17:15:15.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:15 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:15:15.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:15 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:15:15.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:15 smithi106 ceph-mon[44423]: pgmap v726: 17 pgs: 1 active+recovering, 16 active+clean; 4.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 142 op/s; 9002/160582 objects misplaced (5.606%); 525 KiB/s, 8 objects/s recovering 2024-04-20T17:15:15.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:15 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:15:15.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:15 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:15:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:15 smithi171 ceph-mon[44189]: pgmap v726: 17 pgs: 1 active+recovering, 16 active+clean; 4.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 142 op/s; 9002/160582 objects misplaced (5.606%); 525 KiB/s, 8 objects/s recovering 2024-04-20T17:15:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:15 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:15:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:15 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:15:16.059 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:15:16.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:16.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:15:16.165 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:15:16.195 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:16.215 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:15:16.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:16.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:15:16.319 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:15:16.345 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:16.360 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:15:16.360 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:15:16.363 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:15:16.365 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:16 smithi106 ceph-mon[44423]: osdmap e263: 12 total, 12 up, 12 in 2024-04-20T17:15:16.365 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:16 smithi106 ceph-mon[44423]: osdmap e264: 12 total, 12 up, 12 in 2024-04-20T17:15:16.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:16.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:15:16.379 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:16 smithi005 ceph-mon[39562]: osdmap e263: 12 total, 12 up, 12 in 2024-04-20T17:15:16.379 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:16 smithi005 ceph-mon[39562]: osdmap e264: 12 total, 12 up, 12 in 2024-04-20T17:15:16.401 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:15:16.467 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:15:16.500 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:16.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:16 smithi171 ceph-mon[44189]: osdmap e263: 12 total, 12 up, 12 in 2024-04-20T17:15:16.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:16 smithi171 ceph-mon[44189]: osdmap e264: 12 total, 12 up, 12 in 2024-04-20T17:15:16.653 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:15:16.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:16.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:15:16.757 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:15:16.788 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:16.914 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:15:16.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:16.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:15:17.019 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:15:17.048 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:17.075 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:15:17.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:17.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:15:17.183 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:15:17.210 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:17.229 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:15:17.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:17.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:15:17.333 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:15:17.365 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:17.370 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:15:17.448 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:15:17.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:17.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:15:17.552 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:15:17.592 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:18.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:17 smithi005 ceph-mon[39562]: pgmap v729: 17 pgs: 1 active+recovering, 16 active+clean; 4.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 133 op/s; 8914/160128 objects misplaced (5.567%); 688 KiB/s, 10 objects/s recovering 2024-04-20T17:15:18.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:17 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3650001782' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:15:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:17 smithi106 ceph-mon[44423]: pgmap v729: 17 pgs: 1 active+recovering, 16 active+clean; 4.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 133 op/s; 8914/160128 objects misplaced (5.567%); 688 KiB/s, 10 objects/s recovering 2024-04-20T17:15:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:17 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3650001782' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:15:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:17 smithi171 ceph-mon[44189]: pgmap v729: 17 pgs: 1 active+recovering, 16 active+clean; 4.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 133 op/s; 8914/160128 objects misplaced (5.567%); 688 KiB/s, 10 objects/s recovering 2024-04-20T17:15:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:17 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3650001782' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:15:18.449 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:15:18.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:18.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:15:18.553 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:15:18.583 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:18.584 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:15:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:18 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:15:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:18 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3650001782' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:15:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:18 smithi005 ceph-mon[39562]: osdmap e265: 12 total, 12 up, 12 in 2024-04-20T17:15:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:18 smithi005 ceph-mon[39562]: pgmap v731: 17 pgs: 1 active+recovering, 16 active+clean; 4.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 141 op/s; 8914/159680 objects misplaced (5.582%); 480 KiB/s, 7 objects/s recovering 2024-04-20T17:15:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:18 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:15:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:18 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3650001782' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:15:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:18 smithi106 ceph-mon[44423]: osdmap e265: 12 total, 12 up, 12 in 2024-04-20T17:15:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:18 smithi106 ceph-mon[44423]: pgmap v731: 17 pgs: 1 active+recovering, 16 active+clean; 4.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 141 op/s; 8914/159680 objects misplaced (5.582%); 480 KiB/s, 7 objects/s recovering 2024-04-20T17:15:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:18 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:15:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:18 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3650001782' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:15:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:18 smithi171 ceph-mon[44189]: osdmap e265: 12 total, 12 up, 12 in 2024-04-20T17:15:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:18 smithi171 ceph-mon[44189]: pgmap v731: 17 pgs: 1 active+recovering, 16 active+clean; 4.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 141 op/s; 8914/159680 objects misplaced (5.582%); 480 KiB/s, 7 objects/s recovering 2024-04-20T17:15:19.638 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:15:19.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:19.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:15:19.744 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:15:19.771 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:19.788 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:15:19.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:19.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:15:19.892 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:15:19.931 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:19 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:15:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:19 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:15:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:19 smithi005 ceph-mon[39562]: osdmap e266: 12 total, 12 up, 12 in 2024-04-20T17:15:20.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:19 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:15:20.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:19 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:15:20.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:19 smithi106 ceph-mon[44423]: osdmap e266: 12 total, 12 up, 12 in 2024-04-20T17:15:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:19 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:15:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:19 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:15:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:19 smithi171 ceph-mon[44189]: osdmap e266: 12 total, 12 up, 12 in 2024-04-20T17:15:20.506 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:15:20.507 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:15:20.882 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:15:20.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:20.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:15:20.986 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:15:21.012 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:21.031 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:15:21.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:21.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:15:21.054 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:15:21.091 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-20T17:15:21.134 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:15:21.161 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:21.179 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:15:21.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:21.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:15:21.283 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:15:21.310 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:21.328 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:15:21.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:21.331 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:15:21.432 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:15:21.465 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:21.659 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:15:21.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:21 smithi005 ceph-mon[39562]: pgmap v733: 17 pgs: 1 active+recovering, 16 active+clean; 4.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 108 op/s; 8914/159512 objects misplaced (5.588%); 584 KiB/s, 9 objects/s recovering 2024-04-20T17:15:21.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:21 smithi005 ceph-mon[39562]: from='client.25286 v1:172.21.15.5:0/2096073131' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:15:22.015 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:15:22.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:22.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:15:22.119 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:15:22.150 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:22.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:21 smithi106 ceph-mon[44423]: pgmap v733: 17 pgs: 1 active+recovering, 16 active+clean; 4.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 108 op/s; 8914/159512 objects misplaced (5.588%); 584 KiB/s, 9 objects/s recovering 2024-04-20T17:15:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:21 smithi106 ceph-mon[44423]: from='client.25286 v1:172.21.15.5:0/2096073131' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:15:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:21 smithi171 ceph-mon[44189]: pgmap v733: 17 pgs: 1 active+recovering, 16 active+clean; 4.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 108 op/s; 8914/159512 objects misplaced (5.588%); 584 KiB/s, 9 objects/s recovering 2024-04-20T17:15:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:21 smithi171 ceph-mon[44189]: from='client.25286 v1:172.21.15.5:0/2096073131' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:15:22.600 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:15:22.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:22.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:15:22.616 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":733,"stamp":"2024-04-20T17:15:20.035428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5227283000,"num_objects":79756,"num_object_clones":0,"num_object_copies":159512,"num_objects_missing_on_primary":4451,"num_objects_missing":4451,"num_objects_degraded":0,"num_objects_misplaced":8914,"num_objects_unfound":0,"num_objects_dirty":79756,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":242411,"num_write_kb":10306363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24761,"num_bytes_recovered":1624328320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161060,"ondisk_log_size":161060,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":37,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":11878880,"kb_used_data":10805912,"kb_used_omap":219,"kb_used_meta":1072612,"kb_avail":1113062944,"statfs":{"total":1151940427776,"available":1139776454656,"internally_reserved":0,"allocated":11065253888,"data_stored":11056063432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224949,"internal_metadata":1098355019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-35061760,"num_objects":-535,"num_object_clones":0,"num_object_copies":-1070,"num_objects_missing_on_primary":-43,"num_objects_missing":-43,"num_objects_degraded":0,"num_objects_misplaced":-88,"num_objects_unfound":0,"num_objects_dirty":-535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":535,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":2949120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.931525"},"pg_stats":[{"pgid":"2.f","version":"265'15170","reported_seq":15936,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-20T17:15:18.146443+0000","last_change":"2024-04-20T17:14:39.601610+0000","last_active":"2024-04-20T17:15:18.146443+0000","last_peered":"2024-04-20T17:15:18.146443+0000","last_clean":"2024-04-20T17:15:18.146443+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:15:18.146443+0000","last_undegraded":"2024-04-20T17:15:18.146443+0000","last_fullsized":"2024-04-20T17:15:18.146443+0000","mapping_epoch":115,"log_start":"129'5100","ondisk_log_start":"129'5100","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'14864","last_scrub_stamp":"2024-04-20T17:14:39.601506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:39.601506+0000","objects_scrubbed":5320,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:01.262639+0000","scrub_duration":467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328466432,"num_objects":5012,"num_object_clones":0,"num_object_copies":10024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15170,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"263'15085","reported_seq":26860,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-20T17:15:15.625471+0000","last_change":"2024-04-20T17:14:02.868984+0000","last_active":"2024-04-20T17:15:15.625471+0000","last_peered":"2024-04-20T17:15:15.625471+0000","last_clean":"2024-04-20T17:15:15.625471+0000","last_became_active":"2024-04-20T17:10:28.865670+0000","last_became_peered":"2024-04-20T17:10:28.865670+0000","last_unstale":"2024-04-20T17:15:15.625471+0000","last_undegraded":"2024-04-20T17:15:15.625471+0000","last_fullsized":"2024-04-20T17:15:15.625471+0000","mapping_epoch":162,"log_start":"128'5000","ondisk_log_start":"128'5000","created":115,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'14480","last_scrub_stamp":"2024-04-20T17:14:02.868843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:02.868843+0000","objects_scrubbed":5576,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:19.785254+0000","scrub_duration":456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325779456,"num_objects":4971,"num_object_clones":0,"num_object_copies":9942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15085,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5619,"num_bytes_recovered":368246784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"263'15380","reported_seq":15980,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-20T17:15:15.154214+0000","last_change":"2024-04-20T17:14:45.170265+0000","last_active":"2024-04-20T17:15:15.154214+0000","last_peered":"2024-04-20T17:15:15.154214+0000","last_clean":"2024-04-20T17:15:15.154214+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:15:15.154214+0000","last_undegraded":"2024-04-20T17:15:15.154214+0000","last_fullsized":"2024-04-20T17:15:15.154214+0000","mapping_epoch":115,"log_start":"129'5300","ondisk_log_start":"129'5300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'15118","last_scrub_stamp":"2024-04-20T17:14:45.170147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:45.170147+0000","objects_scrubbed":5364,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:55.316236+0000","scrub_duration":445,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334364672,"num_objects":5102,"num_object_clones":0,"num_object_copies":10204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15380,"num_write_kb":655424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"263'21806","reported_seq":43154,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-20T17:15:15.687283+0000","last_change":"2024-04-20T17:13:59.403959+0000","last_active":"2024-04-20T17:15:15.687283+0000","last_peered":"2024-04-20T17:15:15.687283+0000","last_clean":"2024-04-20T17:15:15.687283+0000","last_became_active":"2024-04-20T17:12:08.085936+0000","last_became_peered":"2024-04-20T17:12:08.085936+0000","last_unstale":"2024-04-20T17:15:15.687283+0000","last_undegraded":"2024-04-20T17:15:15.687283+0000","last_fullsized":"2024-04-20T17:15:15.687283+0000","mapping_epoch":196,"log_start":"159'11768","ondisk_log_start":"159'11768","created":115,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'21205","last_scrub_stamp":"2024-04-20T17:13:59.403804+0000","last_deep_scrub":"213'20373","last_deep_scrub_stamp":"2024-04-20T17:12:48.443808+0000","last_clean_scrub_stamp":"2024-04-20T17:13:59.403804+0000","objects_scrubbed":5682,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:18.863462+0000","scrub_duration":478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332922880,"num_objects":5080,"num_object_clones":0,"num_object_copies":10160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15154,"num_write_kb":647488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":450166784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"263'14838","reported_seq":30067,"reported_epoch":263,"state":"active+recovering","last_fresh":"2024-04-20T17:15:15.481338+0000","last_change":"2024-04-20T17:12:26.166109+0000","last_active":"2024-04-20T17:15:15.481338+0000","last_peered":"2024-04-20T17:15:15.481338+0000","last_clean":"2024-04-20T17:09:35.882603+0000","last_became_active":"2024-04-20T17:10:54.470253+0000","last_became_peered":"2024-04-20T17:10:54.470253+0000","last_unstale":"2024-04-20T17:15:15.481338+0000","last_undegraded":"2024-04-20T17:15:15.481338+0000","last_fullsized":"2024-04-20T17:15:15.481338+0000","mapping_epoch":171,"log_start":"128'4800","ondisk_log_start":"128'4800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320602112,"num_objects":4892,"num_object_clones":0,"num_object_copies":9784,"num_objects_missing_on_primary":4451,"num_objects_missing":4451,"num_objects_degraded":0,"num_objects_misplaced":8914,"num_objects_unfound":0,"num_objects_dirty":4892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14838,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4969,"num_bytes_recovered":325648384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"0,2,5","objects":12},{"shards":"0,10","objects":429},{"shards":"2,5","objects":4451}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":714,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-04-20T17:15:15.930348+0000","last_change":"2024-04-20T17:14:40.440515+0000","last_active":"2024-04-20T17:15:15.930348+0000","last_peered":"2024-04-20T17:15:15.930348+0000","last_clean":"2024-04-20T17:15:15.930348+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:15:15.930348+0000","last_undegraded":"2024-04-20T17:15:15.930348+0000","last_fullsized":"2024-04-20T17:15:15.930348+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:14:40.440403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:14:40.440403+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:01.949231+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"264'15224","reported_seq":15855,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-04-20T17:15:17.524617+0000","last_change":"2024-04-20T17:14:01.843280+0000","last_active":"2024-04-20T17:15:17.524617+0000","last_peered":"2024-04-20T17:15:17.524617+0000","last_clean":"2024-04-20T17:15:17.524617+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:15:17.524617+0000","last_undegraded":"2024-04-20T17:15:17.524617+0000","last_fullsized":"2024-04-20T17:15:17.524617+0000","mapping_epoch":115,"log_start":"129'5200","ondisk_log_start":"129'5200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'14629","last_scrub_stamp":"2024-04-20T17:14:01.843179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:01.843179+0000","objects_scrubbed":5626,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:24.160162+0000","scrub_duration":499,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329515008,"num_objects":5028,"num_object_clones":0,"num_object_copies":10056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15224,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"265'15317","reported_seq":15951,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-20T17:15:18.296065+0000","last_change":"2024-04-20T17:14:03.619324+0000","last_active":"2024-04-20T17:15:18.296065+0000","last_peered":"2024-04-20T17:15:18.296065+0000","last_clean":"2024-04-20T17:15:18.296065+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:15:18.296065+0000","last_undegraded":"2024-04-20T17:15:18.296065+0000","last_fullsized":"2024-04-20T17:15:18.296065+0000","mapping_epoch":115,"log_start":"129'5300","ondisk_log_start":"129'5300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'14687","last_scrub_stamp":"2024-04-20T17:14:03.619176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:03.619176+0000","objects_scrubbed":5637,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328138752,"num_objects":5007,"num_object_clones":0,"num_object_copies":10014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15317,"num_write_kb":650368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"264'19269","reported_seq":20054,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-04-20T17:15:16.518397+0000","last_change":"2024-04-20T17:14:45.127365+0000","last_active":"2024-04-20T17:15:16.518397+0000","last_peered":"2024-04-20T17:15:16.518397+0000","last_clean":"2024-04-20T17:15:16.518397+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:15:16.518397+0000","last_undegraded":"2024-04-20T17:15:16.518397+0000","last_fullsized":"2024-04-20T17:15:16.518397+0000","mapping_epoch":115,"log_start":"146'9202","ondisk_log_start":"146'9202","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'19011","last_scrub_stamp":"2024-04-20T17:14:45.127219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:45.127219+0000","objects_scrubbed":5143,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:07.550027+0000","scrub_duration":538,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320077824,"num_objects":4884,"num_object_clones":0,"num_object_copies":9768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15158,"num_write_kb":641344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"264'15298","reported_seq":15988,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-04-20T17:15:16.966436+0000","last_change":"2024-04-20T17:14:37.076566+0000","last_active":"2024-04-20T17:15:16.966436+0000","last_peered":"2024-04-20T17:15:16.966436+0000","last_clean":"2024-04-20T17:15:16.966436+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:15:16.966436+0000","last_undegraded":"2024-04-20T17:15:16.966436+0000","last_fullsized":"2024-04-20T17:15:16.966436+0000","mapping_epoch":115,"log_start":"129'5200","ondisk_log_start":"129'5200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'14978","last_scrub_stamp":"2024-04-20T17:14:37.076332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:37.076332+0000","objects_scrubbed":5242,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:40.993014+0000","scrub_duration":400,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322568192,"num_objects":4922,"num_object_clones":0,"num_object_copies":9844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15298,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"264'15073","reported_seq":15690,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-04-20T17:15:17.559841+0000","last_change":"2024-04-20T17:14:44.918151+0000","last_active":"2024-04-20T17:15:17.559841+0000","last_peered":"2024-04-20T17:15:17.559841+0000","last_clean":"2024-04-20T17:15:17.559841+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:15:17.559841+0000","last_undegraded":"2024-04-20T17:15:17.559841+0000","last_fullsized":"2024-04-20T17:15:17.559841+0000","mapping_epoch":115,"log_start":"128'5000","ondisk_log_start":"128'5000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'14799","last_scrub_stamp":"2024-04-20T17:14:44.917948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:44.917948+0000","objects_scrubbed":5254,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:56.572429+0000","scrub_duration":440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326303744,"num_objects":4979,"num_object_clones":0,"num_object_copies":9958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15073,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"264'15252","reported_seq":30478,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-04-20T17:15:17.592357+0000","last_change":"2024-04-20T17:14:43.945880+0000","last_active":"2024-04-20T17:15:17.592357+0000","last_peered":"2024-04-20T17:15:17.592357+0000","last_clean":"2024-04-20T17:15:17.592357+0000","last_became_active":"2024-04-20T17:10:55.931751+0000","last_became_peered":"2024-04-20T17:10:55.931751+0000","last_unstale":"2024-04-20T17:15:17.592357+0000","last_undegraded":"2024-04-20T17:15:17.592357+0000","last_fullsized":"2024-04-20T17:15:17.592357+0000","mapping_epoch":173,"log_start":"129'5200","ondisk_log_start":"129'5200","created":115,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'14953","last_scrub_stamp":"2024-04-20T17:14:43.945755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:43.945755+0000","objects_scrubbed":5309,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:57.154812+0000","scrub_duration":454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328335360,"num_objects":5010,"num_object_clones":0,"num_object_copies":10020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15252,"num_write_kb":648384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7296,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"264'15306","reported_seq":15934,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-04-20T17:15:17.520184+0000","last_change":"2024-04-20T17:14:03.230824+0000","last_active":"2024-04-20T17:15:17.520184+0000","last_peered":"2024-04-20T17:15:17.520184+0000","last_clean":"2024-04-20T17:15:17.520184+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:15:17.520184+0000","last_undegraded":"2024-04-20T17:15:17.520184+0000","last_fullsized":"2024-04-20T17:15:17.520184+0000","mapping_epoch":115,"log_start":"129'5300","ondisk_log_start":"129'5300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'14739","last_scrub_stamp":"2024-04-20T17:14:03.230712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:03.230712+0000","objects_scrubbed":5578,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:22.306554+0000","scrub_duration":546,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328335360,"num_objects":5010,"num_object_clones":0,"num_object_copies":10020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15306,"num_write_kb":650112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"265'14886","reported_seq":15516,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-04-20T17:15:18.703722+0000","last_change":"2024-04-20T17:14:05.574878+0000","last_active":"2024-04-20T17:15:18.703722+0000","last_peered":"2024-04-20T17:15:18.703722+0000","last_clean":"2024-04-20T17:15:18.703722+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:15:18.703722+0000","last_undegraded":"2024-04-20T17:15:18.703722+0000","last_fullsized":"2024-04-20T17:15:18.703722+0000","mapping_epoch":115,"log_start":"128'4800","ondisk_log_start":"128'4800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'14330","last_scrub_stamp":"2024-04-20T17:14:05.574772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:05.574772+0000","objects_scrubbed":5536,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326238208,"num_objects":4978,"num_object_clones":0,"num_object_copies":9956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14886,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"264'15159","reported_seq":15787,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-04-20T17:15:17.143305+0000","last_change":"2024-04-20T17:14:04.200256+0000","last_active":"2024-04-20T17:15:17.143305+0000","last_peered":"2024-04-20T17:15:17.143305+0000","last_clean":"2024-04-20T17:15:17.143305+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:15:17.143305+0000","last_undegraded":"2024-04-20T17:15:17.143305+0000","last_fullsized":"2024-04-20T17:15:17.143305+0000","mapping_epoch":115,"log_start":"128'5100","ondisk_log_start":"128'5100","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'14590","last_scrub_stamp":"2024-04-20T17:14:04.200147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:04.200147+0000","objects_scrubbed":5516,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:21.444062+0000","scrub_duration":506,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323944448,"num_objects":4943,"num_object_clones":0,"num_object_copies":9886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15159,"num_write_kb":643264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"263'14936","reported_seq":15562,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-20T17:15:15.658665+0000","last_change":"2024-04-20T17:14:12.420994+0000","last_active":"2024-04-20T17:15:15.658665+0000","last_peered":"2024-04-20T17:15:15.658665+0000","last_clean":"2024-04-20T17:15:15.658665+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:15:15.658665+0000","last_undegraded":"2024-04-20T17:15:15.658665+0000","last_fullsized":"2024-04-20T17:15:15.658665+0000","mapping_epoch":115,"log_start":"128'4900","ondisk_log_start":"128'4900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'14443","last_scrub_stamp":"2024-04-20T17:14:12.420702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:12.420702+0000","objects_scrubbed":5431,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:19.358247+0000","scrub_duration":478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323289112,"num_objects":4934,"num_object_clones":0,"num_object_copies":9868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4934,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14936,"num_write_kb":635777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"263'14994","reported_seq":15544,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-20T17:15:15.885468+0000","last_change":"2024-04-20T17:14:59.605205+0000","last_active":"2024-04-20T17:15:15.885468+0000","last_peered":"2024-04-20T17:15:15.885468+0000","last_clean":"2024-04-20T17:15:15.885468+0000","last_became_active":"2024-04-20T17:12:23.515375+0000","last_became_peered":"2024-04-20T17:12:23.515375+0000","last_unstale":"2024-04-20T17:15:15.885468+0000","last_undegraded":"2024-04-20T17:15:15.885468+0000","last_fullsized":"2024-04-20T17:15:15.885468+0000","mapping_epoch":202,"log_start":"128'4900","ondisk_log_start":"128'4900","created":115,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'14856","last_scrub_stamp":"2024-04-20T17:14:59.605082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:59.605082+0000","objects_scrubbed":5143,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:21.549388+0000","scrub_duration":486,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327811072,"num_objects":5002,"num_object_clones":0,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14994,"num_write_kb":639872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":5226692632,"num_objects":79754,"num_object_clones":0,"num_object_copies":159508,"num_objects_missing_on_primary":4451,"num_objects_missing":4451,"num_objects_degraded":0,"num_objects_misplaced":8914,"num_objects_unfound":0,"num_objects_dirty":79754,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":242230,"num_write_kb":10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24753,"num_bytes_recovered":1622212608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11037057024,"data_stored":11037048880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},"log_size":160923,"ondisk_log_size":160923,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991347,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1034016,"kb_used_data":958416,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92711136,"statfs":{"total":95995035648,"available":94936203264,"internally_reserved":0,"allocated":981417984,"data_stored":980645846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":77396420},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777252985,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1132896,"kb_used_data":985108,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92612256,"statfs":{"total":95995035648,"available":94834950144,"internally_reserved":0,"allocated":1008750592,"data_stored":1007976028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":151321024},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612734,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1402172,"kb_used_data":1284916,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":92342980,"statfs":{"total":95995035648,"available":94559211520,"internally_reserved":0,"allocated":1315753984,"data_stored":1314993252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":120060352},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762808964,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1385380,"kb_used_data":1273684,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92359772,"statfs":{"total":95995035648,"available":94576406528,"internally_reserved":0,"allocated":1304252416,"data_stored":1303477872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168713,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1030424,"kb_used_data":963152,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92714728,"statfs":{"total":95995035648,"available":94939881472,"internally_reserved":0,"allocated":986267648,"data_stored":985495530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430349,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":727000,"kb_used_data":641672,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93018152,"statfs":{"total":95995035648,"available":95250587648,"internally_reserved":0,"allocated":657072128,"data_stored":656303562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87357894},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575015,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":734164,"kb_used_data":640520,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93010988,"statfs":{"total":95995035648,"available":95243251712,"internally_reserved":0,"allocated":655892480,"data_stored":655123914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738540,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1102664,"kb_used_data":986812,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":92642488,"statfs":{"total":95995035648,"available":94865907712,"internally_reserved":0,"allocated":1010495488,"data_stored":1009732900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":118618565},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393828,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":903800,"kb_used_data":859568,"kb_used_omap":202,"kb_used_meta":44021,"kb_avail":92841352,"statfs":{"total":95995035648,"available":95069544448,"internally_reserved":0,"allocated":880197632,"data_stored":879440972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":45077975},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182556,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":706376,"kb_used_data":643260,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93038776,"statfs":{"total":95995035648,"available":95271706624,"internally_reserved":0,"allocated":658698240,"data_stored":657935652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986327,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":357308,"kb_used_data":317812,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93387844,"statfs":{"total":95995035648,"available":95629152256,"internally_reserved":0,"allocated":325439488,"data_stored":324678756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790097,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1362680,"kb_used_data":1250992,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92382472,"statfs":{"total":95995035648,"available":94599651328,"internally_reserved":0,"allocated":1281015808,"data_stored":1280259148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1008730112,"data_stored":1008730112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":654049280,"data_stored":654049280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":876150784,"data_stored":876150784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":656277504,"data_stored":656277504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":323686400,"data_stored":323682328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1278541824,"data_stored":1278541824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":655163392,"data_stored":655163392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":984154112,"data_stored":984154112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1302003712,"data_stored":1302003712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1312165888,"data_stored":1312161816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1007157248,"data_stored":1007157248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":978976768,"data_stored":978976768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:15:22.648 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:15:22.705 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:15:22.732 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:22.751 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:15:22.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:22.754 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:15:22.855 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:15:22.880 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:22.902 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:15:22.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:22.904 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:15:23.005 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:15:23.032 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:23.051 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:15:23.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:23.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:15:23.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:22 smithi171 ceph-mon[44189]: pgmap v734: 17 pgs: 1 active+recovering, 16 active+clean; 4.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 95 op/s; 8827/158956 objects misplaced (5.553%); 440 KiB/s, 6 objects/s recovering 2024-04-20T17:15:23.155 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:15:23.186 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:23.212 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:15:23.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:23.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:15:23.221 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:15:23.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:22 smithi106 ceph-mon[44423]: pgmap v734: 17 pgs: 1 active+recovering, 16 active+clean; 4.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 95 op/s; 8827/158956 objects misplaced (5.553%); 440 KiB/s, 6 objects/s recovering 2024-04-20T17:15:23.318 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:15:23.346 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:22 smithi005 ceph-mon[39562]: pgmap v734: 17 pgs: 1 active+recovering, 16 active+clean; 4.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 95 op/s; 8827/158956 objects misplaced (5.553%); 440 KiB/s, 6 objects/s recovering 2024-04-20T17:15:23.348 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:23.477 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:15:23.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:23.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:15:23.587 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:15:23.619 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:23.721 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:15:23.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:23.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:15:23.828 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:15:23.854 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:23.873 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:15:23.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:23.876 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:15:23.977 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:15:24.010 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:24.050 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:15:24.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:24.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:15:24.154 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:15:24.181 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:24.200 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:15:24.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:24.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:15:24.304 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:15:24.331 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:24.349 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:15:24.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:24.352 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:15:24.453 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:15:24.479 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:24.497 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:15:24.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:24.499 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:15:24.601 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:15:24.630 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:24.685 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:15:24.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:24.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:15:24.794 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:15:24.824 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:25.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:25 smithi005 ceph-mon[39562]: pgmap v735: 17 pgs: 1 active+recovering, 16 active+clean; 4.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 117 op/s; 8827/158246 objects misplaced (5.578%); 336 KiB/s, 5 objects/s recovering 2024-04-20T17:15:25.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:25 smithi106 ceph-mon[44423]: pgmap v735: 17 pgs: 1 active+recovering, 16 active+clean; 4.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 117 op/s; 8827/158246 objects misplaced (5.578%); 336 KiB/s, 5 objects/s recovering 2024-04-20T17:15:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:25 smithi171 ceph-mon[44189]: pgmap v735: 17 pgs: 1 active+recovering, 16 active+clean; 4.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 117 op/s; 8827/158246 objects misplaced (5.578%); 336 KiB/s, 5 objects/s recovering 2024-04-20T17:15:25.618 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:15:26.274 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:15:26.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:26.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:15:26.378 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:15:26.408 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:26.427 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:15:26.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:26.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:15:26.431 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:15:26.452 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:15:26.474 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:15:26.501 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:15:26.523 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:15:26.531 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:15:26.544 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:15:26.560 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:26.578 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:15:26.579 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:15:26.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:26.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:15:26.598 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:15:26.633 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:15:26.653 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:15:26.683 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:15:26.687 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:15:26.709 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:26.713 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:15:26.715 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:15:26.715 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:15:26.715 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:15:26.715 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:15:26.715 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:15:26.715 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:15:26.715 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:15:26.716 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:15:26.716 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:15:26.716 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:15:26.716 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:15:26.716 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:15:26.728 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:15:26.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:26.730 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:15:26.831 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:15:26.865 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:27.030 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:26 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/650714628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:15:27.031 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:26 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:27.031 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:26 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:27.031 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:26 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:27.031 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:26 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:27.031 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:26 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:26 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/650714628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:15:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:26 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:26 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:26 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:26 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:26 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:27.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:26 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/650714628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:15:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:26 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:26 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:26 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:26 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:26 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:27.771 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:15:27.783 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:15:27.783 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:15:27.783 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:15:27.783 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:15:27.783 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:15:27.940 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:15:28.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:27 smithi005 ceph-mon[39562]: pgmap v736: 17 pgs: 1 active+recovering, 16 active+clean; 4.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 115 op/s; 8739/157810 objects misplaced (5.538%); 687 KiB/s, 10 objects/s recovering 2024-04-20T17:15:28.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:28.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:27 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2781929943' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:15:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:27 smithi005 ceph-mon[39562]: from='client.25298 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:15:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:27 smithi106 ceph-mon[44423]: pgmap v736: 17 pgs: 1 active+recovering, 16 active+clean; 4.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 115 op/s; 8739/157810 objects misplaced (5.538%); 687 KiB/s, 10 objects/s recovering 2024-04-20T17:15:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:27 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2781929943' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:15:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:27 smithi106 ceph-mon[44423]: from='client.25298 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:15:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:27 smithi171 ceph-mon[44189]: pgmap v736: 17 pgs: 1 active+recovering, 16 active+clean; 4.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 115 op/s; 8739/157810 objects misplaced (5.538%); 687 KiB/s, 10 objects/s recovering 2024-04-20T17:15:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:27 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2781929943' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:15:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:27 smithi171 ceph-mon[44189]: from='client.25298 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:15:28.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:28 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:15:28.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:28 smithi005 ceph-mon[39562]: from='client.25298 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:15:28.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:28 smithi005 ceph-mon[39562]: osdmap e267: 12 total, 12 up, 12 in 2024-04-20T17:15:28.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:28 smithi005 ceph-mon[39562]: pgmap v738: 17 pgs: 1 active+recovering, 16 active+clean; 4.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 119 op/s; 8739/157330 objects misplaced (5.555%); 612 KiB/s, 9 objects/s recovering 2024-04-20T17:15:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:28 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:15:29.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:28 smithi106 ceph-mon[44423]: from='client.25298 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:15:29.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:28 smithi106 ceph-mon[44423]: osdmap e267: 12 total, 12 up, 12 in 2024-04-20T17:15:29.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:28 smithi106 ceph-mon[44423]: pgmap v738: 17 pgs: 1 active+recovering, 16 active+clean; 4.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 119 op/s; 8739/157330 objects misplaced (5.555%); 612 KiB/s, 9 objects/s recovering 2024-04-20T17:15:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:28 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:15:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:28 smithi171 ceph-mon[44189]: from='client.25298 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:15:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:28 smithi171 ceph-mon[44189]: osdmap e267: 12 total, 12 up, 12 in 2024-04-20T17:15:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:28 smithi171 ceph-mon[44189]: pgmap v738: 17 pgs: 1 active+recovering, 16 active+clean; 4.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 119 op/s; 8739/157330 objects misplaced (5.555%); 612 KiB/s, 9 objects/s recovering 2024-04-20T17:15:30.062 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-20T17:15:30.462 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:15:31.218 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:15:31.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:31.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:15:31.322 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:15:31.347 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:31 smithi171 ceph-mon[44189]: 2.1 scrub starts 2024-04-20T17:15:31.348 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:31 smithi171 ceph-mon[44189]: pgmap v739: 17 pgs: 1 active+recovering, 16 active+clean; 4.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 122 op/s; 8739/157058 objects misplaced (5.564%); 557 KiB/s, 8 objects/s recovering 2024-04-20T17:15:31.348 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:31 smithi171 ceph-mon[44189]: 2.a scrub starts 2024-04-20T17:15:31.348 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:31 smithi171 ceph-mon[44189]: osdmap e268: 12 total, 12 up, 12 in 2024-04-20T17:15:31.350 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:31.372 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:15:31.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:31.375 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:15:31.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:31 smithi005 ceph-mon[39562]: 2.1 scrub starts 2024-04-20T17:15:31.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:31 smithi005 ceph-mon[39562]: pgmap v739: 17 pgs: 1 active+recovering, 16 active+clean; 4.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 122 op/s; 8739/157058 objects misplaced (5.564%); 557 KiB/s, 8 objects/s recovering 2024-04-20T17:15:31.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:31 smithi005 ceph-mon[39562]: 2.a scrub starts 2024-04-20T17:15:31.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:31 smithi005 ceph-mon[39562]: osdmap e268: 12 total, 12 up, 12 in 2024-04-20T17:15:31.476 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:15:31.503 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:31.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:31 smithi106 ceph-mon[44423]: 2.1 scrub starts 2024-04-20T17:15:31.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:31 smithi106 ceph-mon[44423]: pgmap v739: 17 pgs: 1 active+recovering, 16 active+clean; 4.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 122 op/s; 8739/157058 objects misplaced (5.564%); 557 KiB/s, 8 objects/s recovering 2024-04-20T17:15:31.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:31 smithi106 ceph-mon[44423]: 2.a scrub starts 2024-04-20T17:15:31.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:31 smithi106 ceph-mon[44423]: osdmap e268: 12 total, 12 up, 12 in 2024-04-20T17:15:31.522 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:15:31.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:31.525 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:15:31.626 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:15:31.652 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:31.671 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:15:31.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:31.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:15:31.775 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:15:31.801 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:31.817 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:15:31.819 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:15:31.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:31.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:15:31.923 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:15:31.954 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:32.078 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:15:32.081 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:15:32.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:32.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:15:32.185 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:15:32.211 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:32.231 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:15:32.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:32.233 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:15:32.334 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:15:32.359 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:15:32.360 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:15:32.360 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: 2.1 scrub ok 2024-04-20T17:15:32.360 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: 2.a scrub ok 2024-04-20T17:15:32.360 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: 2.4 scrub starts 2024-04-20T17:15:32.360 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: 2.0 scrub starts 2024-04-20T17:15:32.360 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: 2.4 scrub ok 2024-04-20T17:15:32.360 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: 2.c scrub starts 2024-04-20T17:15:32.360 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=483310580,config=1460075751,health=1997216375,kv=2581766130}) 2024-04-20T17:15:32.360 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {kv=6,logm=94} crc {kv=3212602960,logm=3397044854}) 2024-04-20T17:15:32.360 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=181340831}) 2024-04-20T17:15:32.360 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=918940708}) 2024-04-20T17:15:32.360 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1676412796}) 2024-04-20T17:15:32.360 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4068704490}) 2024-04-20T17:15:32.361 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3845384983}) 2024-04-20T17:15:32.361 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2777548660}) 2024-04-20T17:15:32.361 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=852366183}) 2024-04-20T17:15:32.361 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1706958208}) 2024-04-20T17:15:32.361 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=70350257}) 2024-04-20T17:15:32.361 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2464845863}) 2024-04-20T17:15:32.361 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1576881599}) 2024-04-20T17:15:32.361 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=480755048}) 2024-04-20T17:15:32.361 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=886347351}) 2024-04-20T17:15:32.361 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1927660586}) 2024-04-20T17:15:32.361 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3018628498}) 2024-04-20T17:15:32.361 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1967430481}) 2024-04-20T17:15:32.361 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2692641017}) 2024-04-20T17:15:32.361 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2572291463}) 2024-04-20T17:15:32.361 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2158081595}) 2024-04-20T17:15:32.362 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=996984882}) 2024-04-20T17:15:32.362 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3884814478}) 2024-04-20T17:15:32.362 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4007929067}) 2024-04-20T17:15:32.362 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3366238757}) 2024-04-20T17:15:32.362 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2535365673}) 2024-04-20T17:15:32.362 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3745611497}) 2024-04-20T17:15:32.362 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3715548958}) 2024-04-20T17:15:32.362 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=229461521}) 2024-04-20T17:15:32.362 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2715346916}) 2024-04-20T17:15:32.362 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2462647314}) 2024-04-20T17:15:32.362 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=603020674}) 2024-04-20T17:15:32.362 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2410540540}) 2024-04-20T17:15:32.362 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3179857386}) 2024-04-20T17:15:32.362 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=893770348}) 2024-04-20T17:15:32.362 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=60,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=5} crc {logm=3275294448,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=2917645751,mon_config_key=3127254339}) 2024-04-20T17:15:32.363 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1918393755}) 2024-04-20T17:15:32.363 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2124202441}) 2024-04-20T17:15:32.363 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=85,monmap=5,osd_metadata=10} crc {mon_config_key=2334417039,monmap=717113318,osd_metadata=810191694}) 2024-04-20T17:15:32.363 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=2,osd_pg_creating=1,osdmap=97} crc {osd_metadata=2166464046,osd_pg_creating=2890833197,osdmap=4225587329}) 2024-04-20T17:15:32.363 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1125713236}) 2024-04-20T17:15:32.363 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=31} crc {osdmap=688026574}) 2024-04-20T17:15:32.363 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:32.379 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:15:32.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:32.381 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:15:32.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:15:32.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:15:32.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: 2.1 scrub ok 2024-04-20T17:15:32.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: 2.a scrub ok 2024-04-20T17:15:32.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: 2.4 scrub starts 2024-04-20T17:15:32.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: 2.0 scrub starts 2024-04-20T17:15:32.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: 2.4 scrub ok 2024-04-20T17:15:32.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: 2.c scrub starts 2024-04-20T17:15:32.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=483310580,config=1460075751,health=1997216375,kv=2581766130}) 2024-04-20T17:15:32.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {kv=6,logm=94} crc {kv=3212602960,logm=3397044854}) 2024-04-20T17:15:32.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=181340831}) 2024-04-20T17:15:32.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=918940708}) 2024-04-20T17:15:32.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1676412796}) 2024-04-20T17:15:32.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4068704490}) 2024-04-20T17:15:32.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3845384983}) 2024-04-20T17:15:32.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2777548660}) 2024-04-20T17:15:32.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=852366183}) 2024-04-20T17:15:32.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1706958208}) 2024-04-20T17:15:32.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=70350257}) 2024-04-20T17:15:32.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2464845863}) 2024-04-20T17:15:32.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1576881599}) 2024-04-20T17:15:32.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=480755048}) 2024-04-20T17:15:32.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=886347351}) 2024-04-20T17:15:32.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1927660586}) 2024-04-20T17:15:32.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3018628498}) 2024-04-20T17:15:32.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1967430481}) 2024-04-20T17:15:32.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2692641017}) 2024-04-20T17:15:32.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2572291463}) 2024-04-20T17:15:32.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2158081595}) 2024-04-20T17:15:32.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=996984882}) 2024-04-20T17:15:32.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3884814478}) 2024-04-20T17:15:32.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4007929067}) 2024-04-20T17:15:32.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3366238757}) 2024-04-20T17:15:32.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2535365673}) 2024-04-20T17:15:32.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3745611497}) 2024-04-20T17:15:32.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3715548958}) 2024-04-20T17:15:32.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=229461521}) 2024-04-20T17:15:32.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2715346916}) 2024-04-20T17:15:32.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2462647314}) 2024-04-20T17:15:32.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=603020674}) 2024-04-20T17:15:32.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2410540540}) 2024-04-20T17:15:32.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3179857386}) 2024-04-20T17:15:32.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=893770348}) 2024-04-20T17:15:32.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=60,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=5} crc {logm=3275294448,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=2917645751,mon_config_key=3127254339}) 2024-04-20T17:15:32.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1918393755}) 2024-04-20T17:15:32.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2124202441}) 2024-04-20T17:15:32.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=85,monmap=5,osd_metadata=10} crc {mon_config_key=2334417039,monmap=717113318,osd_metadata=810191694}) 2024-04-20T17:15:32.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=2,osd_pg_creating=1,osdmap=97} crc {osd_metadata=2166464046,osd_pg_creating=2890833197,osdmap=4225587329}) 2024-04-20T17:15:32.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1125713236}) 2024-04-20T17:15:32.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=31} crc {osdmap=688026574}) 2024-04-20T17:15:32.482 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:15:32.509 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:15:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:15:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: 2.1 scrub ok 2024-04-20T17:15:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: 2.a scrub ok 2024-04-20T17:15:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: 2.4 scrub starts 2024-04-20T17:15:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: 2.0 scrub starts 2024-04-20T17:15:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: 2.4 scrub ok 2024-04-20T17:15:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: 2.c scrub starts 2024-04-20T17:15:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=483310580,config=1460075751,health=1997216375,kv=2581766130}) 2024-04-20T17:15:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {kv=6,logm=94} crc {kv=3212602960,logm=3397044854}) 2024-04-20T17:15:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=181340831}) 2024-04-20T17:15:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=918940708}) 2024-04-20T17:15:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1676412796}) 2024-04-20T17:15:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4068704490}) 2024-04-20T17:15:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3845384983}) 2024-04-20T17:15:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2777548660}) 2024-04-20T17:15:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=852366183}) 2024-04-20T17:15:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1706958208}) 2024-04-20T17:15:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=70350257}) 2024-04-20T17:15:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2464845863}) 2024-04-20T17:15:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1576881599}) 2024-04-20T17:15:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=480755048}) 2024-04-20T17:15:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=886347351}) 2024-04-20T17:15:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1927660586}) 2024-04-20T17:15:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3018628498}) 2024-04-20T17:15:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1967430481}) 2024-04-20T17:15:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2692641017}) 2024-04-20T17:15:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2572291463}) 2024-04-20T17:15:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2158081595}) 2024-04-20T17:15:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=996984882}) 2024-04-20T17:15:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3884814478}) 2024-04-20T17:15:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4007929067}) 2024-04-20T17:15:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3366238757}) 2024-04-20T17:15:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2535365673}) 2024-04-20T17:15:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3745611497}) 2024-04-20T17:15:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3715548958}) 2024-04-20T17:15:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=229461521}) 2024-04-20T17:15:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2715346916}) 2024-04-20T17:15:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2462647314}) 2024-04-20T17:15:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=603020674}) 2024-04-20T17:15:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2410540540}) 2024-04-20T17:15:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3179857386}) 2024-04-20T17:15:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=893770348}) 2024-04-20T17:15:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=60,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=5} crc {logm=3275294448,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=2917645751,mon_config_key=3127254339}) 2024-04-20T17:15:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1918393755}) 2024-04-20T17:15:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2124202441}) 2024-04-20T17:15:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=85,monmap=5,osd_metadata=10} crc {mon_config_key=2334417039,monmap=717113318,osd_metadata=810191694}) 2024-04-20T17:15:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=2,osd_pg_creating=1,osdmap=97} crc {osd_metadata=2166464046,osd_pg_creating=2890833197,osdmap=4225587329}) 2024-04-20T17:15:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1125713236}) 2024-04-20T17:15:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=31} crc {osdmap=688026574}) 2024-04-20T17:15:32.530 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:15:32.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:32.536 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:15:32.637 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:15:32.667 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:32.747 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:15:32.749 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:15:33.204 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:15:33.318 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:15:33.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:33.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:15:33.423 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:15:33.448 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:33 smithi171 ceph-mon[44189]: 2.0 scrub ok 2024-04-20T17:15:33.448 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:33 smithi171 ceph-mon[44189]: 2.c scrub ok 2024-04-20T17:15:33.448 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:33 smithi171 ceph-mon[44189]: pgmap v741: 17 pgs: 1 active+recovering, 16 active+clean; 4.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 111 op/s; 8650/156458 objects misplaced (5.529%); 712 KiB/s, 11 objects/s recovering 2024-04-20T17:15:33.448 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:33 smithi171 ceph-mon[44189]: from='client.15372 v1:172.21.15.5:0/1845456896' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:15:33.450 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:33.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:33 smithi005 ceph-mon[39562]: 2.0 scrub ok 2024-04-20T17:15:33.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:33 smithi005 ceph-mon[39562]: 2.c scrub ok 2024-04-20T17:15:33.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:33 smithi005 ceph-mon[39562]: pgmap v741: 17 pgs: 1 active+recovering, 16 active+clean; 4.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 111 op/s; 8650/156458 objects misplaced (5.529%); 712 KiB/s, 11 objects/s recovering 2024-04-20T17:15:33.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:33 smithi005 ceph-mon[39562]: from='client.15372 v1:172.21.15.5:0/1845456896' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:15:33.468 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:15:33.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:33.470 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:15:33.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:33 smithi106 ceph-mon[44423]: 2.0 scrub ok 2024-04-20T17:15:33.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:33 smithi106 ceph-mon[44423]: 2.c scrub ok 2024-04-20T17:15:33.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:33 smithi106 ceph-mon[44423]: pgmap v741: 17 pgs: 1 active+recovering, 16 active+clean; 4.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 111 op/s; 8650/156458 objects misplaced (5.529%); 712 KiB/s, 11 objects/s recovering 2024-04-20T17:15:33.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:33 smithi106 ceph-mon[44423]: from='client.15372 v1:172.21.15.5:0/1845456896' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:15:33.571 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:15:33.601 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:34.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:34 smithi005 ceph-mon[39562]: 2.d scrub starts 2024-04-20T17:15:34.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:34 smithi106 ceph-mon[44423]: 2.d scrub starts 2024-04-20T17:15:34.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:34 smithi171 ceph-mon[44189]: 2.d scrub starts 2024-04-20T17:15:34.653 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:15:34.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:34.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:15:34.689 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":741,"stamp":"2024-04-20T17:15:32.037526+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5127209528,"num_objects":78229,"num_object_clones":0,"num_object_copies":156458,"num_objects_missing_on_primary":4318,"num_objects_missing":4318,"num_objects_degraded":0,"num_objects_misplaced":8650,"num_objects_unfound":0,"num_objects_dirty":78229,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":243938,"num_write_kb":10306363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24892,"num_bytes_recovered":1632913536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160987,"ondisk_log_size":160987,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":37,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":11674556,"kb_used_data":10594640,"kb_used_omap":219,"kb_used_meta":1079588,"kb_avail":1113267268,"statfs":{"total":1151940427776,"available":1139985682432,"internally_reserved":0,"allocated":10848911360,"data_stored":10839470678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224949,"internal_metadata":1105498443},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,0,0,2,2,2,3,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-58589184,"num_objects":-894,"num_object_clones":0,"num_object_copies":-1788,"num_objects_missing_on_primary":-88,"num_objects_missing":-88,"num_objects_degraded":0,"num_objects_misplaced":-177,"num_objects_unfound":0,"num_objects_dirty":-894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":894,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":5832704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001427"},"pg_stats":[{"pgid":"2.f","version":"267'15254","reported_seq":16024,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-04-20T17:15:28.678445+0000","last_change":"2024-04-20T17:14:39.601610+0000","last_active":"2024-04-20T17:15:28.678445+0000","last_peered":"2024-04-20T17:15:28.678445+0000","last_clean":"2024-04-20T17:15:28.678445+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:15:28.678445+0000","last_undegraded":"2024-04-20T17:15:28.678445+0000","last_fullsized":"2024-04-20T17:15:28.678445+0000","mapping_epoch":115,"log_start":"129'5200","ondisk_log_start":"129'5200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'14864","last_scrub_stamp":"2024-04-20T17:14:39.601506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:39.601506+0000","objects_scrubbed":5320,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:01.262639+0000","scrub_duration":467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322961408,"num_objects":4928,"num_object_clones":0,"num_object_copies":9856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15254,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"267'15197","reported_seq":26980,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-04-20T17:15:30.218065+0000","last_change":"2024-04-20T17:14:02.868984+0000","last_active":"2024-04-20T17:15:30.218065+0000","last_peered":"2024-04-20T17:15:30.218065+0000","last_clean":"2024-04-20T17:15:30.218065+0000","last_became_active":"2024-04-20T17:10:28.865670+0000","last_became_peered":"2024-04-20T17:10:28.865670+0000","last_unstale":"2024-04-20T17:15:30.218065+0000","last_undegraded":"2024-04-20T17:15:30.218065+0000","last_fullsized":"2024-04-20T17:15:30.218065+0000","mapping_epoch":162,"log_start":"129'5100","ondisk_log_start":"129'5100","created":115,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'14480","last_scrub_stamp":"2024-04-20T17:14:02.868843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:02.868843+0000","objects_scrubbed":5576,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:30.773008+0000","scrub_duration":456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318439424,"num_objects":4859,"num_object_clones":0,"num_object_copies":9718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15197,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5619,"num_bytes_recovered":368246784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"267'15505","reported_seq":16113,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-04-20T17:15:30.524995+0000","last_change":"2024-04-20T17:14:45.170265+0000","last_active":"2024-04-20T17:15:30.524995+0000","last_peered":"2024-04-20T17:15:30.524995+0000","last_clean":"2024-04-20T17:15:30.524995+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:15:30.524995+0000","last_undegraded":"2024-04-20T17:15:30.524995+0000","last_fullsized":"2024-04-20T17:15:30.524995+0000","mapping_epoch":115,"log_start":"129'5500","ondisk_log_start":"129'5500","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'15118","last_scrub_stamp":"2024-04-20T17:14:45.170147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:45.170147+0000","objects_scrubbed":5364,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:55.316236+0000","scrub_duration":445,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326172672,"num_objects":4977,"num_object_clones":0,"num_object_copies":9954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15505,"num_write_kb":655424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"267'21923","reported_seq":43336,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-04-20T17:15:30.287285+0000","last_change":"2024-04-20T17:15:30.287285+0000","last_active":"2024-04-20T17:15:30.287285+0000","last_peered":"2024-04-20T17:15:30.287285+0000","last_clean":"2024-04-20T17:15:30.287285+0000","last_became_active":"2024-04-20T17:12:08.085936+0000","last_became_peered":"2024-04-20T17:12:08.085936+0000","last_unstale":"2024-04-20T17:15:30.287285+0000","last_undegraded":"2024-04-20T17:15:30.287285+0000","last_fullsized":"2024-04-20T17:15:30.287285+0000","mapping_epoch":196,"log_start":"160'11868","ondisk_log_start":"160'11868","created":115,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'21922","last_scrub_stamp":"2024-04-20T17:15:30.287129+0000","last_deep_scrub":"213'20373","last_deep_scrub_stamp":"2024-04-20T17:12:48.443808+0000","last_clean_scrub_stamp":"2024-04-20T17:15:30.287129+0000","objects_scrubbed":4965,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:34.168023+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325255168,"num_objects":4963,"num_object_clones":0,"num_object_copies":9926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15271,"num_write_kb":647488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":450166784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"267'14969","reported_seq":30602,"reported_epoch":267,"state":"active+recovering","last_fresh":"2024-04-20T17:15:30.760590+0000","last_change":"2024-04-20T17:12:26.166109+0000","last_active":"2024-04-20T17:15:30.760590+0000","last_peered":"2024-04-20T17:15:30.760590+0000","last_clean":"2024-04-20T17:09:35.882603+0000","last_became_active":"2024-04-20T17:10:54.470253+0000","last_became_peered":"2024-04-20T17:10:54.470253+0000","last_unstale":"2024-04-20T17:15:30.760590+0000","last_undegraded":"2024-04-20T17:15:30.760590+0000","last_fullsized":"2024-04-20T17:15:30.760590+0000","mapping_epoch":171,"log_start":"128'4900","ondisk_log_start":"128'4900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312016896,"num_objects":4761,"num_object_clones":0,"num_object_copies":9522,"num_objects_missing_on_primary":4318,"num_objects_missing":4318,"num_objects_degraded":0,"num_objects_misplaced":8650,"num_objects_unfound":0,"num_objects_dirty":4761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14969,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5100,"num_bytes_recovered":334233600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"0,2,5","objects":14},{"shards":"0,10","objects":429},{"shards":"2,5","objects":4318}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":718,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-20T17:15:19.089347+0000","last_change":"2024-04-20T17:14:40.440515+0000","last_active":"2024-04-20T17:15:19.089347+0000","last_peered":"2024-04-20T17:15:19.089347+0000","last_clean":"2024-04-20T17:15:19.089347+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:15:19.089347+0000","last_undegraded":"2024-04-20T17:15:19.089347+0000","last_fullsized":"2024-04-20T17:15:19.089347+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:14:40.440403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:14:40.440403+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:01.949231+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"266'15299","reported_seq":15934,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-20T17:15:27.609897+0000","last_change":"2024-04-20T17:14:01.843280+0000","last_active":"2024-04-20T17:15:27.609897+0000","last_peered":"2024-04-20T17:15:27.609897+0000","last_clean":"2024-04-20T17:15:27.609897+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:15:27.609897+0000","last_undegraded":"2024-04-20T17:15:27.609897+0000","last_fullsized":"2024-04-20T17:15:27.609897+0000","mapping_epoch":115,"log_start":"129'5200","ondisk_log_start":"129'5200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'14629","last_scrub_stamp":"2024-04-20T17:14:01.843179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:01.843179+0000","objects_scrubbed":5626,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:29.835860+0000","scrub_duration":499,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324599808,"num_objects":4953,"num_object_clones":0,"num_object_copies":9906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15299,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"267'15407","reported_seq":16045,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-04-20T17:15:28.638299+0000","last_change":"2024-04-20T17:14:03.619324+0000","last_active":"2024-04-20T17:15:28.638299+0000","last_peered":"2024-04-20T17:15:28.638299+0000","last_clean":"2024-04-20T17:15:28.638299+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:15:28.638299+0000","last_undegraded":"2024-04-20T17:15:28.638299+0000","last_fullsized":"2024-04-20T17:15:28.638299+0000","mapping_epoch":115,"log_start":"129'5400","ondisk_log_start":"129'5400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'14687","last_scrub_stamp":"2024-04-20T17:14:03.619176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:03.619176+0000","objects_scrubbed":5637,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:31.290464+0000","scrub_duration":558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322240512,"num_objects":4917,"num_object_clones":0,"num_object_copies":9834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15407,"num_write_kb":650368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"268'19381","reported_seq":20174,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-20T17:15:30.987251+0000","last_change":"2024-04-20T17:14:45.127365+0000","last_active":"2024-04-20T17:15:30.987251+0000","last_peered":"2024-04-20T17:15:30.987251+0000","last_clean":"2024-04-20T17:15:30.987251+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:15:30.987251+0000","last_undegraded":"2024-04-20T17:15:30.987251+0000","last_fullsized":"2024-04-20T17:15:30.987251+0000","mapping_epoch":115,"log_start":"146'9302","ondisk_log_start":"146'9302","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'19011","last_scrub_stamp":"2024-04-20T17:14:45.127219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:45.127219+0000","objects_scrubbed":5143,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:07.550027+0000","scrub_duration":538,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312737792,"num_objects":4772,"num_object_clones":0,"num_object_copies":9544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15270,"num_write_kb":641344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"266'15359","reported_seq":16053,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-20T17:15:27.423763+0000","last_change":"2024-04-20T17:14:37.076566+0000","last_active":"2024-04-20T17:15:27.423763+0000","last_peered":"2024-04-20T17:15:27.423763+0000","last_clean":"2024-04-20T17:15:27.423763+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:15:27.423763+0000","last_undegraded":"2024-04-20T17:15:27.423763+0000","last_fullsized":"2024-04-20T17:15:27.423763+0000","mapping_epoch":115,"log_start":"129'5300","ondisk_log_start":"129'5300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'14978","last_scrub_stamp":"2024-04-20T17:14:37.076332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:37.076332+0000","objects_scrubbed":5242,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:40.993014+0000","scrub_duration":400,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318570496,"num_objects":4861,"num_object_clones":0,"num_object_copies":9722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15359,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"266'15149","reported_seq":15770,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-20T17:15:27.549272+0000","last_change":"2024-04-20T17:14:44.918151+0000","last_active":"2024-04-20T17:15:27.549272+0000","last_peered":"2024-04-20T17:15:27.549272+0000","last_clean":"2024-04-20T17:15:27.549272+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:15:27.549272+0000","last_undegraded":"2024-04-20T17:15:27.549272+0000","last_fullsized":"2024-04-20T17:15:27.549272+0000","mapping_epoch":115,"log_start":"129'5100","ondisk_log_start":"129'5100","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'14799","last_scrub_stamp":"2024-04-20T17:14:44.917948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:44.917948+0000","objects_scrubbed":5254,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:56.572429+0000","scrub_duration":440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321323008,"num_objects":4903,"num_object_clones":0,"num_object_copies":9806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15149,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"266'15325","reported_seq":30555,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-20T17:15:27.618350+0000","last_change":"2024-04-20T17:14:43.945880+0000","last_active":"2024-04-20T17:15:27.618350+0000","last_peered":"2024-04-20T17:15:27.618350+0000","last_clean":"2024-04-20T17:15:27.618350+0000","last_became_active":"2024-04-20T17:10:55.931751+0000","last_became_peered":"2024-04-20T17:10:55.931751+0000","last_unstale":"2024-04-20T17:15:27.618350+0000","last_undegraded":"2024-04-20T17:15:27.618350+0000","last_fullsized":"2024-04-20T17:15:27.618350+0000","mapping_epoch":173,"log_start":"129'5300","ondisk_log_start":"129'5300","created":115,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'14953","last_scrub_stamp":"2024-04-20T17:14:43.945755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:43.945755+0000","objects_scrubbed":5309,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:57.154812+0000","scrub_duration":454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323551232,"num_objects":4937,"num_object_clones":0,"num_object_copies":9874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15325,"num_write_kb":648384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7296,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"266'15376","reported_seq":16008,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-20T17:15:27.537388+0000","last_change":"2024-04-20T17:14:03.230824+0000","last_active":"2024-04-20T17:15:27.537388+0000","last_peered":"2024-04-20T17:15:27.537388+0000","last_clean":"2024-04-20T17:15:27.537388+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:15:27.537388+0000","last_undegraded":"2024-04-20T17:15:27.537388+0000","last_fullsized":"2024-04-20T17:15:27.537388+0000","mapping_epoch":115,"log_start":"129'5300","ondisk_log_start":"129'5300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'14739","last_scrub_stamp":"2024-04-20T17:14:03.230712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:03.230712+0000","objects_scrubbed":5578,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":546,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323747840,"num_objects":4940,"num_object_clones":0,"num_object_copies":9880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15376,"num_write_kb":650112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"267'14981","reported_seq":15615,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-04-20T17:15:28.652523+0000","last_change":"2024-04-20T17:14:05.574878+0000","last_active":"2024-04-20T17:15:28.652523+0000","last_peered":"2024-04-20T17:15:28.652523+0000","last_clean":"2024-04-20T17:15:28.652523+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:15:28.652523+0000","last_undegraded":"2024-04-20T17:15:28.652523+0000","last_fullsized":"2024-04-20T17:15:28.652523+0000","mapping_epoch":115,"log_start":"128'4900","ondisk_log_start":"128'4900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'14330","last_scrub_stamp":"2024-04-20T17:14:05.574772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:05.574772+0000","objects_scrubbed":5536,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:31.290429+0000","scrub_duration":494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320012288,"num_objects":4883,"num_object_clones":0,"num_object_copies":9766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14981,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"266'15230","reported_seq":15862,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-04-20T17:15:27.444803+0000","last_change":"2024-04-20T17:14:04.200256+0000","last_active":"2024-04-20T17:15:27.444803+0000","last_peered":"2024-04-20T17:15:27.444803+0000","last_clean":"2024-04-20T17:15:27.444803+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:15:27.444803+0000","last_undegraded":"2024-04-20T17:15:27.444803+0000","last_fullsized":"2024-04-20T17:15:27.444803+0000","mapping_epoch":115,"log_start":"129'5200","ondisk_log_start":"129'5200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'14590","last_scrub_stamp":"2024-04-20T17:14:04.200147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:04.200147+0000","objects_scrubbed":5516,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":506,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319291392,"num_objects":4872,"num_object_clones":0,"num_object_copies":9744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15230,"num_write_kb":643264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"267'15042","reported_seq":15676,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-04-20T17:15:30.379846+0000","last_change":"2024-04-20T17:14:12.420994+0000","last_active":"2024-04-20T17:15:30.379846+0000","last_peered":"2024-04-20T17:15:30.379846+0000","last_clean":"2024-04-20T17:15:30.379846+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:15:30.379846+0000","last_undegraded":"2024-04-20T17:15:30.379846+0000","last_fullsized":"2024-04-20T17:15:30.379846+0000","mapping_epoch":115,"log_start":"128'5000","ondisk_log_start":"128'5000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'14443","last_scrub_stamp":"2024-04-20T17:14:12.420702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:12.420702+0000","objects_scrubbed":5431,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316342296,"num_objects":4828,"num_object_clones":0,"num_object_copies":9656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4828,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15042,"num_write_kb":635777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"267'15123","reported_seq":15681,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-04-20T17:15:30.449458+0000","last_change":"2024-04-20T17:14:59.605205+0000","last_active":"2024-04-20T17:15:30.449458+0000","last_peered":"2024-04-20T17:15:30.449458+0000","last_clean":"2024-04-20T17:15:30.449458+0000","last_became_active":"2024-04-20T17:12:23.515375+0000","last_became_peered":"2024-04-20T17:12:23.515375+0000","last_unstale":"2024-04-20T17:15:30.449458+0000","last_undegraded":"2024-04-20T17:15:30.449458+0000","last_fullsized":"2024-04-20T17:15:30.449458+0000","mapping_epoch":202,"log_start":"129'5100","ondisk_log_start":"129'5100","created":115,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'14856","last_scrub_stamp":"2024-04-20T17:14:59.605082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:59.605082+0000","objects_scrubbed":5143,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:21.549388+0000","scrub_duration":486,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319356928,"num_objects":4873,"num_object_clones":0,"num_object_copies":9746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15123,"num_write_kb":639872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":5126619160,"num_objects":78227,"num_object_clones":0,"num_object_copies":156454,"num_objects_missing_on_primary":4318,"num_objects_missing":4318,"num_objects_degraded":0,"num_objects_misplaced":8650,"num_objects_unfound":0,"num_objects_dirty":78227,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":243757,"num_write_kb":10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24884,"num_bytes_recovered":1630797824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10816987136,"data_stored":10816978992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},"log_size":160850,"ondisk_log_size":160850,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991349,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1020592,"kb_used_data":945000,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92724560,"statfs":{"total":95995035648,"available":94949949440,"internally_reserved":0,"allocated":967680000,"data_stored":966895956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":77396420},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777252987,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1105720,"kb_used_data":970412,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":92639432,"statfs":{"total":95995035648,"available":94862778368,"internally_reserved":0,"allocated":993701888,"data_stored":992915418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":138541504},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612737,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1391224,"kb_used_data":1255920,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":92353928,"statfs":{"total":95995035648,"available":94570422272,"internally_reserved":0,"allocated":1286062080,"data_stored":1285271558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":138541504},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762808966,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1365304,"kb_used_data":1253612,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92379848,"statfs":{"total":95995035648,"available":94596964352,"internally_reserved":0,"allocated":1283698688,"data_stored":1282912238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168715,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1018096,"kb_used_data":950824,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92727056,"statfs":{"total":95995035648,"available":94952505344,"internally_reserved":0,"allocated":973643776,"data_stored":972859752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430351,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":718440,"kb_used_data":633120,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93026712,"statfs":{"total":95995035648,"available":95259353088,"internally_reserved":0,"allocated":648314880,"data_stored":647534408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87357894},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575018,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":719044,"kb_used_data":625400,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93026108,"statfs":{"total":95995035648,"available":95258734592,"internally_reserved":0,"allocated":640409600,"data_stored":639617222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738543,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1081588,"kb_used_data":964332,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":92663564,"statfs":{"total":95995035648,"available":94887489536,"internally_reserved":0,"allocated":987475968,"data_stored":986689518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2,2,2,3,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393831,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":866612,"kb_used_data":822380,"kb_used_omap":202,"kb_used_meta":44021,"kb_avail":92878540,"statfs":{"total":95995035648,"available":95107624960,"internally_reserved":0,"allocated":842117120,"data_stored":841330670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":45077975},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182559,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":690292,"kb_used_data":627180,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93054860,"statfs":{"total":95995035648,"available":95288176640,"internally_reserved":0,"allocated":642232320,"data_stored":641445870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986330,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":350328,"kb_used_data":310832,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93394824,"statfs":{"total":95995035648,"available":95636299776,"internally_reserved":0,"allocated":318291968,"data_stored":317501446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790100,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1347316,"kb_used_data":1235628,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92397836,"statfs":{"total":95995035648,"available":94615384064,"internally_reserved":0,"allocated":1265283072,"data_stored":1264496622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":985792512,"data_stored":985792512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":638517248,"data_stored":638517248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":838205440,"data_stored":838205440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":640417792,"data_stored":640417792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":316346368,"data_stored":316342296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1263009792,"data_stored":1263009792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":646184960,"data_stored":646184960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":969474048,"data_stored":969474048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1278869504,"data_stored":1278869504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1283395584,"data_stored":1283391512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":992018432,"data_stored":992018432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":964755456,"data_stored":964755456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:15:34.691 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:15:34.760 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:15:34.804 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:35 smithi005 ceph-mon[39562]: 2.d scrub ok 2024-04-20T17:15:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:35 smithi005 ceph-mon[39562]: pgmap v742: 17 pgs: 1 active+recovering, 16 active+clean; 4.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 127 op/s; 8650/155770 objects misplaced (5.553%); 352 KiB/s, 5 objects/s recovering 2024-04-20T17:15:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:35 smithi005 ceph-mon[39562]: 2.b scrub starts 2024-04-20T17:15:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:35 smithi005 ceph-mon[39562]: 2.b scrub ok 2024-04-20T17:15:35.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:35 smithi106 ceph-mon[44423]: 2.d scrub ok 2024-04-20T17:15:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:35 smithi106 ceph-mon[44423]: pgmap v742: 17 pgs: 1 active+recovering, 16 active+clean; 4.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 127 op/s; 8650/155770 objects misplaced (5.553%); 352 KiB/s, 5 objects/s recovering 2024-04-20T17:15:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:35 smithi106 ceph-mon[44423]: 2.b scrub starts 2024-04-20T17:15:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:35 smithi106 ceph-mon[44423]: 2.b scrub ok 2024-04-20T17:15:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:35 smithi171 ceph-mon[44189]: 2.d scrub ok 2024-04-20T17:15:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:35 smithi171 ceph-mon[44189]: pgmap v742: 17 pgs: 1 active+recovering, 16 active+clean; 4.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 127 op/s; 8650/155770 objects misplaced (5.553%); 352 KiB/s, 5 objects/s recovering 2024-04-20T17:15:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:35 smithi171 ceph-mon[44189]: 2.b scrub starts 2024-04-20T17:15:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:35 smithi171 ceph-mon[44189]: 2.b scrub ok 2024-04-20T17:15:35.839 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:15:35.859 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:15:35.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:35.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:15:35.878 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:15:35.965 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:15:35.996 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:36.021 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:15:36.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:36.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:15:36.128 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:15:36.165 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:36.398 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:36 smithi005 ceph-mon[39562]: 2.5 scrub starts 2024-04-20T17:15:36.399 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:36 smithi005 ceph-mon[39562]: 2.5 scrub ok 2024-04-20T17:15:36.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:36 smithi106 ceph-mon[44423]: 2.5 scrub starts 2024-04-20T17:15:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:36 smithi106 ceph-mon[44423]: 2.5 scrub ok 2024-04-20T17:15:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:36 smithi171 ceph-mon[44189]: 2.5 scrub starts 2024-04-20T17:15:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:36 smithi171 ceph-mon[44189]: 2.5 scrub ok 2024-04-20T17:15:37.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:37 smithi005 ceph-mon[39562]: pgmap v743: 17 pgs: 1 active+recovering, 16 active+clean; 4.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 125 op/s; 8560/155304 objects misplaced (5.512%); 711 KiB/s, 11 objects/s recovering 2024-04-20T17:15:37.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:37 smithi106 ceph-mon[44423]: pgmap v743: 17 pgs: 1 active+recovering, 16 active+clean; 4.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 125 op/s; 8560/155304 objects misplaced (5.512%); 711 KiB/s, 11 objects/s recovering 2024-04-20T17:15:37.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:37 smithi171 ceph-mon[44189]: pgmap v743: 17 pgs: 1 active+recovering, 16 active+clean; 4.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 125 op/s; 8560/155304 objects misplaced (5.512%); 711 KiB/s, 11 objects/s recovering 2024-04-20T17:15:38.228 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:15:38.228 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:15:38.228 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:15:38.228 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:15:38.247 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:15:38.267 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:15:38.288 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:15:38.308 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:15:38.329 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:15:38.338 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:15:38.338 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:15:38.350 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:15:38.371 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:15:38.392 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:15:38.394 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:15:38.394 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:15:38.394 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:15:38.394 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:15:38.394 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:15:38.394 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:15:38.394 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:15:38.394 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:15:38.394 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:15:38.394 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:15:38.395 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:15:38.395 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:15:38.416 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:38 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:15:38.417 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:38 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:15:38.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:38 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:15:38.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:38 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:15:38.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:38 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:15:38.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:38 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:15:39.420 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:15:39.422 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:39 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:15:39.422 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:39 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:15:39.422 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:39 smithi005 ceph-mon[39562]: pgmap v744: 17 pgs: 1 active+recovering, 16 active+clean; 4.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 132 op/s; 8560/154688 objects misplaced (5.534%); 576 KiB/s, 8 objects/s recovering 2024-04-20T17:15:39.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:39 smithi005 ceph-mon[39562]: osdmap e269: 12 total, 12 up, 12 in 2024-04-20T17:15:39.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:39 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/131470740' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:15:39.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:39.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:15:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:39 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:15:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:39 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:15:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:39 smithi106 ceph-mon[44423]: pgmap v744: 17 pgs: 1 active+recovering, 16 active+clean; 4.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 132 op/s; 8560/154688 objects misplaced (5.534%); 576 KiB/s, 8 objects/s recovering 2024-04-20T17:15:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:39 smithi106 ceph-mon[44423]: osdmap e269: 12 total, 12 up, 12 in 2024-04-20T17:15:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:39 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/131470740' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:15:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.525 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:15:39.556 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:39.581 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:15:39.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:39 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:15:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:39 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:15:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:39 smithi171 ceph-mon[44189]: pgmap v744: 17 pgs: 1 active+recovering, 16 active+clean; 4.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 132 op/s; 8560/154688 objects misplaced (5.534%); 576 KiB/s, 8 objects/s recovering 2024-04-20T17:15:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:39 smithi171 ceph-mon[44189]: osdmap e269: 12 total, 12 up, 12 in 2024-04-20T17:15:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:39 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/131470740' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:15:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:39.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:39.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:15:39.689 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:15:39.718 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:39.733 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:15:39.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:39.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:15:39.838 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:15:39.865 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:39.884 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:15:39.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:39.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:15:39.987 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:15:40.018 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:40.094 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:15:40.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:40.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:15:40.121 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-20T17:15:40.198 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:15:40.225 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:40.243 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:15:40.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:40.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:15:40.347 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:15:40.373 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:40.392 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:15:40.393 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:40.393 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:40.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:40.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:15:40.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:40.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:40.496 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:15:40.531 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:40.551 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:15:40.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:40.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:15:40.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:40.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:40.657 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:15:40.685 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:40.838 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:15:40.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:40.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:15:40.944 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:15:40.972 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:40.990 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:15:40.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:40.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:15:41.094 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:15:41.120 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:41.141 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:15:41.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:41.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:15:41.243 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:15:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:41 smithi106 ceph-mon[44423]: pgmap v746: 17 pgs: 1 active+recovering, 16 active+clean; 4.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 117 op/s; 8560/154392 objects misplaced (5.544%); 323 KiB/s, 5 objects/s recovering 2024-04-20T17:15:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:41 smithi106 ceph-mon[44423]: osdmap e270: 12 total, 12 up, 12 in 2024-04-20T17:15:41.276 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:41.430 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:41 smithi005 ceph-mon[39562]: pgmap v746: 17 pgs: 1 active+recovering, 16 active+clean; 4.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 117 op/s; 8560/154392 objects misplaced (5.544%); 323 KiB/s, 5 objects/s recovering 2024-04-20T17:15:41.430 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:41 smithi005 ceph-mon[39562]: osdmap e270: 12 total, 12 up, 12 in 2024-04-20T17:15:41.437 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:15:41.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:41.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:15:41.542 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:15:41.568 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:41.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:41 smithi171 ceph-mon[44189]: pgmap v746: 17 pgs: 1 active+recovering, 16 active+clean; 4.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 117 op/s; 8560/154392 objects misplaced (5.544%); 323 KiB/s, 5 objects/s recovering 2024-04-20T17:15:41.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:41 smithi171 ceph-mon[44189]: osdmap e270: 12 total, 12 up, 12 in 2024-04-20T17:15:41.587 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:15:41.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:41.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:15:41.691 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:15:41.719 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:41.741 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:15:41.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:41.744 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:15:41.845 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:15:41.873 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:41.897 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:15:41.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:41.902 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:15:42.003 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:15:42.030 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:42.048 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:15:42.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:42.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:15:42.132 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:15:42.151 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:15:42.160 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:42 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/737030917' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:15:42.161 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:42 smithi005 ceph-mon[39562]: 2.7 scrub starts 2024-04-20T17:15:42.186 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:42.435 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:15:42.437 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:15:42.483 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:15:42.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:42 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/737030917' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:15:42.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:42 smithi106 ceph-mon[44423]: 2.7 scrub starts 2024-04-20T17:15:42.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:42 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/737030917' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:15:42.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:42 smithi171 ceph-mon[44189]: 2.7 scrub starts 2024-04-20T17:15:42.739 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:15:42.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:42.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:15:42.844 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:15:42.870 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:42.889 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:15:42.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:42.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:15:42.993 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:15:43.020 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:43.038 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:15:43.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:43.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:15:43.142 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:15:43.168 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:43.186 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:15:43.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:43.189 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:15:43.290 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:15:43.316 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:43.335 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:15:43.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:43.337 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:15:43.438 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:15:43.461 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:15:43.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:43 smithi005 ceph-mon[39562]: 2.7 scrub ok 2024-04-20T17:15:43.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:43 smithi005 ceph-mon[39562]: pgmap v748: 17 pgs: 1 active+recovering, 16 active+clean; 4.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 129 op/s; 8471/153696 objects misplaced (5.512%); 720 KiB/s, 11 objects/s recovering 2024-04-20T17:15:43.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:43 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:15:43.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:43 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:15:43.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:43 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/737030917' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:15:43.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:43 smithi005 ceph-mon[39562]: osdmap e271: 12 total, 12 up, 12 in 2024-04-20T17:15:43.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:43 smithi005 ceph-mon[39562]: from='client.25294 v1:172.21.15.5:0/3376484914' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:15:43.462 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:43 smithi171 ceph-mon[44189]: 2.7 scrub ok 2024-04-20T17:15:43.463 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:43 smithi171 ceph-mon[44189]: pgmap v748: 17 pgs: 1 active+recovering, 16 active+clean; 4.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 129 op/s; 8471/153696 objects misplaced (5.512%); 720 KiB/s, 11 objects/s recovering 2024-04-20T17:15:43.463 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:43 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:15:43.463 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:43 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:15:43.463 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:43 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/737030917' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:15:43.463 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:43 smithi171 ceph-mon[44189]: osdmap e271: 12 total, 12 up, 12 in 2024-04-20T17:15:43.463 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:43 smithi171 ceph-mon[44189]: from='client.25294 v1:172.21.15.5:0/3376484914' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:15:43.464 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:43.482 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:15:43.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:43.485 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:15:43.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:43 smithi106 ceph-mon[44423]: 2.7 scrub ok 2024-04-20T17:15:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:43 smithi106 ceph-mon[44423]: pgmap v748: 17 pgs: 1 active+recovering, 16 active+clean; 4.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 129 op/s; 8471/153696 objects misplaced (5.512%); 720 KiB/s, 11 objects/s recovering 2024-04-20T17:15:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:43 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:15:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:43 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:15:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:43 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/737030917' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:15:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:43 smithi106 ceph-mon[44423]: osdmap e271: 12 total, 12 up, 12 in 2024-04-20T17:15:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:43 smithi106 ceph-mon[44423]: from='client.25294 v1:172.21.15.5:0/3376484914' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:15:43.586 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:15:43.600 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":749,"stamp":"2024-04-20T17:15:42.129320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5036704312,"num_objects":76848,"num_object_clones":0,"num_object_copies":153696,"num_objects_missing_on_primary":4229,"num_objects_missing":4229,"num_objects_degraded":0,"num_objects_misplaced":8471,"num_objects_unfound":0,"num_objects_dirty":76848,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":245319,"num_write_kb":10306363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24982,"num_bytes_recovered":1638811776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161068,"ondisk_log_size":161068,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":37,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":11503892,"kb_used_data":10411492,"kb_used_omap":219,"kb_used_meta":1092068,"kb_avail":1113437932,"statfs":{"total":1151940427776,"available":1140160442368,"internally_reserved":0,"allocated":10661367808,"data_stored":10651790982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224949,"internal_metadata":1118277963},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,6],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-52690944,"num_objects":-804,"num_object_clones":0,"num_object_copies":-1608,"num_objects_missing_on_primary":-45,"num_objects_missing":-45,"num_objects_degraded":0,"num_objects_misplaced":-89,"num_objects_unfound":0,"num_objects_dirty":-804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":804,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":2883584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.091074"},"pg_stats":[{"pgid":"2.f","version":"269'15341","reported_seq":16115,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-04-20T17:15:38.639583+0000","last_change":"2024-04-20T17:14:39.601610+0000","last_active":"2024-04-20T17:15:38.639583+0000","last_peered":"2024-04-20T17:15:38.639583+0000","last_clean":"2024-04-20T17:15:38.639583+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:15:38.639583+0000","last_undegraded":"2024-04-20T17:15:38.639583+0000","last_fullsized":"2024-04-20T17:15:38.639583+0000","mapping_epoch":115,"log_start":"129'5300","ondisk_log_start":"129'5300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'14864","last_scrub_stamp":"2024-04-20T17:14:39.601506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:39.601506+0000","objects_scrubbed":5320,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:01.262639+0000","scrub_duration":467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317259776,"num_objects":4841,"num_object_clones":0,"num_object_copies":9682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15341,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"269'15284","reported_seq":27127,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-04-20T17:15:40.444618+0000","last_change":"2024-04-20T17:15:31.328405+0000","last_active":"2024-04-20T17:15:40.444618+0000","last_peered":"2024-04-20T17:15:40.444618+0000","last_clean":"2024-04-20T17:15:40.444618+0000","last_became_active":"2024-04-20T17:10:28.865670+0000","last_became_peered":"2024-04-20T17:10:28.865670+0000","last_unstale":"2024-04-20T17:15:40.444618+0000","last_undegraded":"2024-04-20T17:15:40.444618+0000","last_fullsized":"2024-04-20T17:15:40.444618+0000","mapping_epoch":162,"log_start":"129'5200","ondisk_log_start":"129'5200","created":115,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15200","last_scrub_stamp":"2024-04-20T17:15:31.328301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:31.328301+0000","objects_scrubbed":4857,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:38.848343+0000","scrub_duration":424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312737792,"num_objects":4772,"num_object_clones":0,"num_object_copies":9544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15284,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5619,"num_bytes_recovered":368246784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"269'15597","reported_seq":16209,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-04-20T17:15:40.529749+0000","last_change":"2024-04-20T17:14:45.170265+0000","last_active":"2024-04-20T17:15:40.529749+0000","last_peered":"2024-04-20T17:15:40.529749+0000","last_clean":"2024-04-20T17:15:40.529749+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:15:40.529749+0000","last_undegraded":"2024-04-20T17:15:40.529749+0000","last_fullsized":"2024-04-20T17:15:40.529749+0000","mapping_epoch":115,"log_start":"129'5500","ondisk_log_start":"129'5500","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'15118","last_scrub_stamp":"2024-04-20T17:14:45.170147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:45.170147+0000","objects_scrubbed":5364,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:55.316236+0000","scrub_duration":445,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320143360,"num_objects":4885,"num_object_clones":0,"num_object_copies":9770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15597,"num_write_kb":655424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"269'22011","reported_seq":43428,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-04-20T17:15:40.596024+0000","last_change":"2024-04-20T17:15:30.287285+0000","last_active":"2024-04-20T17:15:40.596024+0000","last_peered":"2024-04-20T17:15:40.596024+0000","last_clean":"2024-04-20T17:15:40.596024+0000","last_became_active":"2024-04-20T17:12:08.085936+0000","last_became_peered":"2024-04-20T17:12:08.085936+0000","last_unstale":"2024-04-20T17:15:40.596024+0000","last_undegraded":"2024-04-20T17:15:40.596024+0000","last_fullsized":"2024-04-20T17:15:40.596024+0000","mapping_epoch":196,"log_start":"160'11968","ondisk_log_start":"160'11968","created":115,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'21922","last_scrub_stamp":"2024-04-20T17:15:30.287129+0000","last_deep_scrub":"213'20373","last_deep_scrub_stamp":"2024-04-20T17:12:48.443808+0000","last_clean_scrub_stamp":"2024-04-20T17:15:30.287129+0000","objects_scrubbed":4965,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:34.168023+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319488000,"num_objects":4875,"num_object_clones":0,"num_object_copies":9750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15359,"num_write_kb":647488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":450166784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"269'15059","reported_seq":30966,"reported_epoch":269,"state":"active+recovering","last_fresh":"2024-04-20T17:15:41.025629+0000","last_change":"2024-04-20T17:12:26.166109+0000","last_active":"2024-04-20T17:15:41.025629+0000","last_peered":"2024-04-20T17:15:41.025629+0000","last_clean":"2024-04-20T17:09:35.882603+0000","last_became_active":"2024-04-20T17:10:54.470253+0000","last_became_peered":"2024-04-20T17:10:54.470253+0000","last_unstale":"2024-04-20T17:15:41.025629+0000","last_undegraded":"2024-04-20T17:15:41.025629+0000","last_fullsized":"2024-04-20T17:15:41.025629+0000","mapping_epoch":171,"log_start":"129'5000","ondisk_log_start":"129'5000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306118656,"num_objects":4671,"num_object_clones":0,"num_object_copies":9342,"num_objects_missing_on_primary":4229,"num_objects_missing":4229,"num_objects_degraded":0,"num_objects_misplaced":8471,"num_objects_unfound":0,"num_objects_dirty":4671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15059,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5190,"num_bytes_recovered":340131840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"0,2,5","objects":13},{"shards":"0,10","objects":429},{"shards":"2,5","objects":4229}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":722,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-20T17:15:30.988588+0000","last_change":"2024-04-20T17:14:40.440515+0000","last_active":"2024-04-20T17:15:30.988588+0000","last_peered":"2024-04-20T17:15:30.988588+0000","last_clean":"2024-04-20T17:15:30.988588+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:15:30.988588+0000","last_undegraded":"2024-04-20T17:15:30.988588+0000","last_fullsized":"2024-04-20T17:15:30.988588+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:14:40.440403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:14:40.440403+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:01.949231+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"268'15386","reported_seq":16082,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-20T17:15:37.846585+0000","last_change":"2024-04-20T17:15:31.396169+0000","last_active":"2024-04-20T17:15:37.846585+0000","last_peered":"2024-04-20T17:15:37.846585+0000","last_clean":"2024-04-20T17:15:37.846585+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:15:37.846585+0000","last_undegraded":"2024-04-20T17:15:37.846585+0000","last_fullsized":"2024-04-20T17:15:37.846585+0000","mapping_epoch":115,"log_start":"129'5300","ondisk_log_start":"129'5300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15333","last_scrub_stamp":"2024-04-20T17:15:31.396069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:31.396069+0000","objects_scrubbed":4919,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:57.811864+0000","scrub_duration":440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318898176,"num_objects":4866,"num_object_clones":0,"num_object_copies":9732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15386,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"269'15487","reported_seq":16184,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-04-20T17:15:38.725133+0000","last_change":"2024-04-20T17:15:35.632022+0000","last_active":"2024-04-20T17:15:38.725133+0000","last_peered":"2024-04-20T17:15:38.725133+0000","last_clean":"2024-04-20T17:15:38.725133+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:15:38.725133+0000","last_undegraded":"2024-04-20T17:15:38.725133+0000","last_fullsized":"2024-04-20T17:15:38.725133+0000","mapping_epoch":115,"log_start":"129'5400","ondisk_log_start":"129'5400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15459","last_scrub_stamp":"2024-04-20T17:15:35.631842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:35.631842+0000","objects_scrubbed":4867,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:52.716132+0000","scrub_duration":426,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316997632,"num_objects":4837,"num_object_clones":0,"num_object_copies":9674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15487,"num_write_kb":650368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"270'19484","reported_seq":20281,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-04-20T17:15:41.903209+0000","last_change":"2024-04-20T17:14:45.127365+0000","last_active":"2024-04-20T17:15:41.903209+0000","last_peered":"2024-04-20T17:15:41.903209+0000","last_clean":"2024-04-20T17:15:41.903209+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:15:41.903209+0000","last_undegraded":"2024-04-20T17:15:41.903209+0000","last_fullsized":"2024-04-20T17:15:41.903209+0000","mapping_epoch":115,"log_start":"146'9402","ondisk_log_start":"146'9402","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'19011","last_scrub_stamp":"2024-04-20T17:14:45.127219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:45.127219+0000","objects_scrubbed":5143,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:07.550027+0000","scrub_duration":538,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305987584,"num_objects":4669,"num_object_clones":0,"num_object_copies":9338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15373,"num_write_kb":641344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"268'15434","reported_seq":16132,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-20T17:15:37.020791+0000","last_change":"2024-04-20T17:14:37.076566+0000","last_active":"2024-04-20T17:15:37.020791+0000","last_peered":"2024-04-20T17:15:37.020791+0000","last_clean":"2024-04-20T17:15:37.020791+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:15:37.020791+0000","last_undegraded":"2024-04-20T17:15:37.020791+0000","last_fullsized":"2024-04-20T17:15:37.020791+0000","mapping_epoch":115,"log_start":"129'5400","ondisk_log_start":"129'5400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'14978","last_scrub_stamp":"2024-04-20T17:14:37.076332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:37.076332+0000","objects_scrubbed":5242,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:40.993014+0000","scrub_duration":400,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313655296,"num_objects":4786,"num_object_clones":0,"num_object_copies":9572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15434,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"268'15224","reported_seq":15849,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-20T17:15:37.640495+0000","last_change":"2024-04-20T17:14:44.918151+0000","last_active":"2024-04-20T17:15:37.640495+0000","last_peered":"2024-04-20T17:15:37.640495+0000","last_clean":"2024-04-20T17:15:37.640495+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:15:37.640495+0000","last_undegraded":"2024-04-20T17:15:37.640495+0000","last_fullsized":"2024-04-20T17:15:37.640495+0000","mapping_epoch":115,"log_start":"129'5200","ondisk_log_start":"129'5200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'14799","last_scrub_stamp":"2024-04-20T17:14:44.917948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:44.917948+0000","objects_scrubbed":5254,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:56.572429+0000","scrub_duration":440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316407808,"num_objects":4828,"num_object_clones":0,"num_object_copies":9656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15224,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"268'15400","reported_seq":30634,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-20T17:15:37.721082+0000","last_change":"2024-04-20T17:14:43.945880+0000","last_active":"2024-04-20T17:15:37.721082+0000","last_peered":"2024-04-20T17:15:37.721082+0000","last_clean":"2024-04-20T17:15:37.721082+0000","last_became_active":"2024-04-20T17:10:55.931751+0000","last_became_peered":"2024-04-20T17:10:55.931751+0000","last_unstale":"2024-04-20T17:15:37.721082+0000","last_undegraded":"2024-04-20T17:15:37.721082+0000","last_fullsized":"2024-04-20T17:15:37.721082+0000","mapping_epoch":173,"log_start":"129'5300","ondisk_log_start":"129'5300","created":115,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'14953","last_scrub_stamp":"2024-04-20T17:14:43.945755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:43.945755+0000","objects_scrubbed":5309,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:15:57.154812+0000","scrub_duration":454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318636032,"num_objects":4862,"num_object_clones":0,"num_object_copies":9724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15400,"num_write_kb":648384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7296,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"268'15475","reported_seq":16168,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-20T17:15:37.046938+0000","last_change":"2024-04-20T17:15:30.785298+0000","last_active":"2024-04-20T17:15:37.046938+0000","last_peered":"2024-04-20T17:15:37.046938+0000","last_clean":"2024-04-20T17:15:37.046938+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:15:37.046938+0000","last_undegraded":"2024-04-20T17:15:37.046938+0000","last_fullsized":"2024-04-20T17:15:37.046938+0000","mapping_epoch":115,"log_start":"129'5400","ondisk_log_start":"129'5400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'15415","last_scrub_stamp":"2024-04-20T17:15:30.785135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:30.785135+0000","objects_scrubbed":4901,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:40.155273+0000","scrub_duration":453,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317259776,"num_objects":4841,"num_object_clones":0,"num_object_copies":9682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15475,"num_write_kb":650112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"269'15063","reported_seq":15757,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-04-20T17:15:38.684537+0000","last_change":"2024-04-20T17:15:34.656600+0000","last_active":"2024-04-20T17:15:38.684537+0000","last_peered":"2024-04-20T17:15:38.684537+0000","last_clean":"2024-04-20T17:15:38.684537+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:15:38.684537+0000","last_undegraded":"2024-04-20T17:15:38.684537+0000","last_fullsized":"2024-04-20T17:15:38.684537+0000","mapping_epoch":115,"log_start":"129'5000","ondisk_log_start":"129'5000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15023","last_scrub_stamp":"2024-04-20T17:15:34.656503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:34.656503+0000","objects_scrubbed":4842,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:51.197304+0000","scrub_duration":430,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314638336,"num_objects":4801,"num_object_clones":0,"num_object_copies":9602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15063,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"268'15315","reported_seq":16007,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-04-20T17:15:37.410300+0000","last_change":"2024-04-20T17:15:31.794340+0000","last_active":"2024-04-20T17:15:37.410300+0000","last_peered":"2024-04-20T17:15:37.410300+0000","last_clean":"2024-04-20T17:15:37.410300+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:15:37.410300+0000","last_undegraded":"2024-04-20T17:15:37.410300+0000","last_fullsized":"2024-04-20T17:15:37.410300+0000","mapping_epoch":115,"log_start":"129'5300","ondisk_log_start":"129'5300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15271","last_scrub_stamp":"2024-04-20T17:15:31.794156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:31.794156+0000","objects_scrubbed":4831,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:58.805155+0000","scrub_duration":443,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313720832,"num_objects":4787,"num_object_clones":0,"num_object_copies":9574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15315,"num_write_kb":643264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"269'15129","reported_seq":15823,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-04-20T17:15:40.432274+0000","last_change":"2024-04-20T17:15:33.298229+0000","last_active":"2024-04-20T17:15:40.432274+0000","last_peered":"2024-04-20T17:15:40.432274+0000","last_clean":"2024-04-20T17:15:40.432274+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:15:40.432274+0000","last_undegraded":"2024-04-20T17:15:40.432274+0000","last_fullsized":"2024-04-20T17:15:40.432274+0000","mapping_epoch":115,"log_start":"129'5100","ondisk_log_start":"129'5100","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15067","last_scrub_stamp":"2024-04-20T17:15:33.298103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:33.298103+0000","objects_scrubbed":4809,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:56.012242+0000","scrub_duration":458,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310640664,"num_objects":4741,"num_object_clones":0,"num_object_copies":9482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4741,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15129,"num_write_kb":635777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"269'15212","reported_seq":15774,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-04-20T17:15:40.567148+0000","last_change":"2024-04-20T17:14:59.605205+0000","last_active":"2024-04-20T17:15:40.567148+0000","last_peered":"2024-04-20T17:15:40.567148+0000","last_clean":"2024-04-20T17:15:40.567148+0000","last_became_active":"2024-04-20T17:12:23.515375+0000","last_became_peered":"2024-04-20T17:12:23.515375+0000","last_unstale":"2024-04-20T17:15:40.567148+0000","last_undegraded":"2024-04-20T17:15:40.567148+0000","last_fullsized":"2024-04-20T17:15:40.567148+0000","mapping_epoch":202,"log_start":"129'5200","ondisk_log_start":"129'5200","created":115,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'14856","last_scrub_stamp":"2024-04-20T17:14:59.605082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:59.605082+0000","objects_scrubbed":5143,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:21.549388+0000","scrub_duration":486,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313524224,"num_objects":4784,"num_object_clones":0,"num_object_copies":9568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15212,"num_write_kb":639872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":5036113944,"num_objects":76846,"num_object_clones":0,"num_object_copies":153692,"num_objects_missing_on_primary":4229,"num_objects_missing":4229,"num_objects_degraded":0,"num_objects_misplaced":8471,"num_objects_unfound":0,"num_objects_dirty":76846,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":245138,"num_write_kb":10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24974,"num_bytes_recovered":1636696064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10627457024,"data_stored":10627448880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},"log_size":160931,"ondisk_log_size":160931,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991351,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1002824,"kb_used_data":927232,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92742328,"statfs":{"total":95995035648,"available":94968143872,"internally_reserved":0,"allocated":949485568,"data_stored":948689618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":77396420},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777252989,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1090960,"kb_used_data":955652,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":92654192,"statfs":{"total":95995035648,"available":94877892608,"internally_reserved":0,"allocated":978587648,"data_stored":977789372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":138541504},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612739,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1374096,"kb_used_data":1234632,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92371056,"statfs":{"total":95995035648,"available":94587961344,"internally_reserved":0,"allocated":1264263168,"data_stored":1263460840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":142801344},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762808968,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1342656,"kb_used_data":1230968,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92402496,"statfs":{"total":95995035648,"available":94620155904,"internally_reserved":0,"allocated":1260511232,"data_stored":1259718854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168717,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1002312,"kb_used_data":935040,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92742840,"statfs":{"total":95995035648,"available":94968668160,"internally_reserved":0,"allocated":957480960,"data_stored":956685030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430353,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":708352,"kb_used_data":623032,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93036800,"statfs":{"total":95995035648,"available":95269683200,"internally_reserved":0,"allocated":637984768,"data_stored":637192390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87357894},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575020,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":706844,"kb_used_data":613200,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93038308,"statfs":{"total":95995035648,"available":95271227392,"internally_reserved":0,"allocated":627916800,"data_stored":627112566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738545,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1070092,"kb_used_data":948676,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":92675060,"statfs":{"total":95995035648,"available":94899261440,"internally_reserved":0,"allocated":971444224,"data_stored":970645968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,6],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393833,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":842128,"kb_used_data":797892,"kb_used_omap":202,"kb_used_meta":44021,"kb_avail":92903024,"statfs":{"total":95995035648,"available":95132696576,"internally_reserved":0,"allocated":817041408,"data_stored":816243152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":45077975},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182561,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":682444,"kb_used_data":615172,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93062708,"statfs":{"total":95995035648,"available":95296212992,"internally_reserved":0,"allocated":629936128,"data_stored":629137872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986332,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":344656,"kb_used_data":305160,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93400496,"statfs":{"total":95995035648,"available":95642107904,"internally_reserved":0,"allocated":312483840,"data_stored":311681512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790102,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1336528,"kb_used_data":1224836,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92408624,"statfs":{"total":95995035648,"available":94626430976,"internally_reserved":0,"allocated":1254232064,"data_stored":1253433808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":969408512,"data_stored":969408512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":625147904,"data_stored":625147904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":812711936,"data_stored":812711936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":628293632,"data_stored":628293632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":310710272,"data_stored":310706200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1251278848,"data_stored":1251278848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":635895808,"data_stored":635895808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":953876480,"data_stored":953876480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1257046016,"data_stored":1257046016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1259802624,"data_stored":1259798552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":976617472,"data_stored":976617472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":946667520,"data_stored":946667520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:15:43.601 INFO:tasks.cephadm.ceph_manager.ceph:PG 2.3 is not active+clean 2024-04-20T17:15:43.601 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '2.3', 'version': "269'15059", 'reported_seq': 30966, 'reported_epoch': 269, 'state': 'active+recovering', 'last_fresh': '2024-04-20T17:15:41.025629+0000', 'last_change': '2024-04-20T17:12:26.166109+0000', 'last_active': '2024-04-20T17:15:41.025629+0000', 'last_peered': '2024-04-20T17:15:41.025629+0000', 'last_clean': '2024-04-20T17:09:35.882603+0000', 'last_became_active': '2024-04-20T17:10:54.470253+0000', 'last_became_peered': '2024-04-20T17:10:54.470253+0000', 'last_unstale': '2024-04-20T17:15:41.025629+0000', 'last_undegraded': '2024-04-20T17:15:41.025629+0000', 'last_fullsized': '2024-04-20T17:15:41.025629+0000', 'mapping_epoch': 171, 'log_start': "129'5000", 'ondisk_log_start': "129'5000", 'created': 115, 'last_epoch_clean': 116, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-20T17:08:04.990639+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T17:08:04.990639+0000', 'last_clean_scrub_stamp': '2024-04-20T17:08:04.990639+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 306118656, 'num_objects': 4671, 'num_object_clones': 0, 'num_object_copies': 9342, 'num_objects_missing_on_primary': 4229, 'num_objects_missing': 4229, 'num_objects_degraded': 0, 'num_objects_misplaced': 8471, 'num_objects_unfound': 0, 'num_objects_dirty': 4671, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15059, 'num_write_kb': 631360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5190, 'num_bytes_recovered': 340131840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 10], 'acting': [0, 10], 'avail_no_missing': ['2', '5'], 'object_location_counts': [{'shards': '0,2,5', 'objects': 13}, {'shards': '0,10', 'objects': 429}, {'shards': '2,5', 'objects': 4229}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-20T17:15:43.602 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph status --format=json 2024-04-20T17:15:43.612 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:43.630 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:15:43.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:43.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:15:43.736 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:15:43.768 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:44.053 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:15:44.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:44.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:15:44.156 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:15:44.183 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:44.202 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:15:44.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:44.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:15:44.306 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:15:44.317 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:44 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:15:44.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:44 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:15:44.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:44 smithi005 ceph-mon[39562]: osdmap e272: 12 total, 12 up, 12 in 2024-04-20T17:15:44.338 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:44.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:44 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:15:44.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:44 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:15:44.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:44 smithi106 ceph-mon[44423]: osdmap e272: 12 total, 12 up, 12 in 2024-04-20T17:15:44.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:44 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:15:44.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:44 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:15:44.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:44 smithi171 ceph-mon[44189]: osdmap e272: 12 total, 12 up, 12 in 2024-04-20T17:15:44.687 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:15:44.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:44.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:15:44.791 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:15:44.817 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:44.835 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:15:44.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:44.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:15:44.939 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:15:44.966 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:44.984 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:15:44.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:44.987 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:15:45.088 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:15:45.117 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:45.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:45 smithi171 ceph-mon[44189]: pgmap v751: 17 pgs: 1 active+recovering, 16 active+clean; 4.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 158 op/s; 8471/152806 objects misplaced (5.544%); 476 KiB/s, 7 objects/s recovering 2024-04-20T17:15:45.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:45 smithi005 ceph-mon[39562]: pgmap v751: 17 pgs: 1 active+recovering, 16 active+clean; 4.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 158 op/s; 8471/152806 objects misplaced (5.544%); 476 KiB/s, 7 objects/s recovering 2024-04-20T17:15:45.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:45 smithi106 ceph-mon[44423]: pgmap v751: 17 pgs: 1 active+recovering, 16 active+clean; 4.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 158 op/s; 8471/152806 objects misplaced (5.544%); 476 KiB/s, 7 objects/s recovering 2024-04-20T17:15:45.669 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:15:45.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:45.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:15:45.769 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:15:45.769 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:15:45.773 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:15:45.800 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:45.819 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:15:45.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:45.823 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:15:45.924 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:15:45.952 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:45.972 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:15:45.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:45.975 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:15:46.076 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:15:46.108 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:46.307 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:15:46.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:46.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:15:46.411 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:15:46.437 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:46.455 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:15:46.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:46.458 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:15:46.559 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:15:46.591 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:46.630 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:15:46.631 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:15:46.715 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:15:47.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:46 smithi005 ceph-mon[39562]: pgmap v752: 17 pgs: 1 active+recovering, 16 active+clean; 4.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 175 op/s; 8471/152282 objects misplaced (5.563%); 469 KiB/s, 7 objects/s recovering 2024-04-20T17:15:47.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:46 smithi106 ceph-mon[44423]: pgmap v752: 17 pgs: 1 active+recovering, 16 active+clean; 4.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 175 op/s; 8471/152282 objects misplaced (5.563%); 469 KiB/s, 7 objects/s recovering 2024-04-20T17:15:47.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:46 smithi171 ceph-mon[44189]: pgmap v752: 17 pgs: 1 active+recovering, 16 active+clean; 4.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 175 op/s; 8471/152282 objects misplaced (5.563%); 469 KiB/s, 7 objects/s recovering 2024-04-20T17:15:47.345 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:15:47.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:47.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:15:47.452 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:15:47.481 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:48.332 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:15:48.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:48.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:15:48.436 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:15:48.467 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:48.719 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:15:48.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:48.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:15:48.824 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:15:48.850 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:48.868 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:15:48.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:48.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:15:48.972 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:15:49.002 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:49.023 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:15:49.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:49.026 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:15:49.127 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:15:49.152 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:49 smithi106 ceph-mon[44423]: pgmap v753: 17 pgs: 1 active+recovering, 16 active+clean; 4.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 171 op/s; 8380/151448 objects misplaced (5.533%); 801 KiB/s, 12 objects/s recovering 2024-04-20T17:15:49.153 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:49.172 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:15:49.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:49.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:15:49.276 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:15:49.307 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:49 smithi005 ceph-mon[39562]: pgmap v753: 17 pgs: 1 active+recovering, 16 active+clean; 4.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 171 op/s; 8380/151448 objects misplaced (5.533%); 801 KiB/s, 12 objects/s recovering 2024-04-20T17:15:49.311 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:49.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:49 smithi171 ceph-mon[44189]: pgmap v753: 17 pgs: 1 active+recovering, 16 active+clean; 4.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 171 op/s; 8380/151448 objects misplaced (5.533%); 801 KiB/s, 12 objects/s recovering 2024-04-20T17:15:49.663 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:15:49.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:49.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:15:49.676 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-20T17:15:49.768 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:15:49.795 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:49.813 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:15:49.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:49.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:15:49.917 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:15:49.943 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:49.961 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:15:49.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:49.964 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:15:50.065 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:15:50.092 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:50.112 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:15:50.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:50.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:15:50.215 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:15:50.246 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:50.327 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:15:50.327 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:15:50.327 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:15:50.327 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:15:50.328 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:15:50.328 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:15:50.328 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:15:50.328 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:15:50.328 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:15:50.328 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:15:50.328 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:15:50.328 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:15:50.328 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:15:50.328 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:15:50.328 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:15:50.328 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:15:50.328 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:15:50.329 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:15:50.329 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:15:50.329 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:15:50.329 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:15:50.329 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:15:50.329 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:15:50.329 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:15:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:50 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1151439860' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:15:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:50.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:50 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1151439860' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:15:50.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:50.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:50.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:50 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1151439860' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:15:50.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:50.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:50.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:50.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:50.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:50.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:50.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:50.778 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:15:51.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:51.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:51.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:51.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:51.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:51.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:51 smithi005 ceph-mon[39562]: pgmap v754: 17 pgs: 1 active+recovering, 16 active+clean; 4.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 162 op/s; 8380/151102 objects misplaced (5.546%); 360 KiB/s, 5 objects/s recovering 2024-04-20T17:15:51.423 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:15:51.428 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:15:51.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:51.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:15:51.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:51.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:51.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:51.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:51.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:51.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:51 smithi106 ceph-mon[44423]: pgmap v754: 17 pgs: 1 active+recovering, 16 active+clean; 4.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 162 op/s; 8380/151102 objects misplaced (5.546%); 360 KiB/s, 5 objects/s recovering 2024-04-20T17:15:51.533 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:15:51.564 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:51.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:51.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:51.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:51.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:51.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:15:51.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:51 smithi171 ceph-mon[44189]: pgmap v754: 17 pgs: 1 active+recovering, 16 active+clean; 4.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 162 op/s; 8380/151102 objects misplaced (5.546%); 360 KiB/s, 5 objects/s recovering 2024-04-20T17:15:51.664 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:15:51.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:51.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:15:51.769 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:15:51.794 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:51.813 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:15:51.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:51.817 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:15:51.919 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:15:51.945 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:51.965 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:15:51.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:51.968 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:15:52.069 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:15:52.095 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:52.115 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:15:52.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:52.117 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:15:52.219 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:15:52.255 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:15:52.612 INFO:teuthology.orchestra.run.smithi005.stdout:{"fsid":"13c6e76c-ff37-11ee-bc93-c7b262605968","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":920,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":272,"num_osds":12,"num_up_osds":12,"osd_up_since":1713633141,"num_in_osds":12,"osd_in_since":1713632742,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovering","count":1}],"num_pgs":17,"num_pools":2,"num_objects":75210,"data_bytes":4929356344,"bytes_used":11569094656,"bytes_avail":1140371333120,"bytes_total":1151940427776,"misplaced_objects":8287,"misplaced_total":150420,"misplaced_ratio":0.055092407924478136,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":601684,"recovering_keys_per_sec":0,"num_objects_recovered":91,"num_bytes_recovered":5963776,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":165},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.5:8443/"}},"servicemap":{"epoch":22,"modified":"2024-04-20T17:15:34.037967+0000","services":{"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-20T17:15:53.459 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:15:53.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:53 smithi005 ceph-mon[39562]: pgmap v755: 17 pgs: 1 active+recovering, 16 active+clean; 4.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 165 op/s; 8287/150420 objects misplaced (5.509%); 588 KiB/s, 9 objects/s recovering 2024-04-20T17:15:53.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:53 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3960917103' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:15:53.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:53 smithi106 ceph-mon[44423]: pgmap v755: 17 pgs: 1 active+recovering, 16 active+clean; 4.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 165 op/s; 8287/150420 objects misplaced (5.509%); 588 KiB/s, 9 objects/s recovering 2024-04-20T17:15:53.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:53 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3960917103' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:15:53.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:53 smithi171 ceph-mon[44189]: pgmap v755: 17 pgs: 1 active+recovering, 16 active+clean; 4.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 165 op/s; 8287/150420 objects misplaced (5.509%); 588 KiB/s, 9 objects/s recovering 2024-04-20T17:15:53.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:53 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3960917103' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:15:53.806 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:15:53.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:53.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:15:53.880 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-20T17:15:53.880 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:15:53.911 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:15:53.938 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:53.955 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:15:53.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:53.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:15:54.059 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:15:54.089 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:54.467 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:15:54.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:54.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:15:54.572 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:15:54.598 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:54.617 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:15:54.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:54.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:15:54.721 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:15:54.747 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:54.766 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:15:54.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:54.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:15:54.869 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:15:54.899 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:55.001 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:15:55.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:55.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:15:55.017 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:15:55.105 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:15:55.133 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:55.152 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:15:55.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:55.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:15:55.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:55 smithi005 ceph-mon[39562]: pgmap v756: 17 pgs: 1 active+recovering, 16 active+clean; 4.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 156 op/s; 8287/149670 objects misplaced (5.537%); 534 KiB/s, 8 objects/s recovering 2024-04-20T17:15:55.256 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:15:55.284 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:55.360 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:15:55.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:55.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:15:55.468 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:15:55.492 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:55 smithi171 ceph-mon[44189]: pgmap v756: 17 pgs: 1 active+recovering, 16 active+clean; 4.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 156 op/s; 8287/149670 objects misplaced (5.537%); 534 KiB/s, 8 objects/s recovering 2024-04-20T17:15:55.495 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:55.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:55 smithi106 ceph-mon[44423]: pgmap v756: 17 pgs: 1 active+recovering, 16 active+clean; 4.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 156 op/s; 8287/149670 objects misplaced (5.537%); 534 KiB/s, 8 objects/s recovering 2024-04-20T17:15:55.514 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:15:55.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:55.517 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:15:55.618 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:15:55.644 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:55.662 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:15:55.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:55.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:15:55.765 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:15:55.797 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:56.031 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:15:56.031 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:15:56.031 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:15:56.031 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:15:56.031 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:15:56.050 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:15:56.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:56.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:15:56.155 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:15:56.181 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:56.201 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:15:56.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:56.203 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:15:56.304 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:15:56.334 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:56.434 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:15:56.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:56.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:15:56.539 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:15:56.565 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:56.583 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:15:56.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:56.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:15:56.688 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:15:56.719 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:57.365 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:57 smithi005 ceph-mon[39562]: pgmap v757: 17 pgs: 1 active+recovering, 16 active+clean; 4.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 151 op/s; 8287/149166 objects misplaced (5.556%); 485 KiB/s, 7 objects/s recovering 2024-04-20T17:15:57.369 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:15:57.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:57.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:15:57.476 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:15:57.504 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:57.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:57 smithi106 ceph-mon[44423]: pgmap v757: 17 pgs: 1 active+recovering, 16 active+clean; 4.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 151 op/s; 8287/149166 objects misplaced (5.556%); 485 KiB/s, 7 objects/s recovering 2024-04-20T17:15:57.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:57 smithi171 ceph-mon[44189]: pgmap v757: 17 pgs: 1 active+recovering, 16 active+clean; 4.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 151 op/s; 8287/149166 objects misplaced (5.556%); 485 KiB/s, 7 objects/s recovering 2024-04-20T17:15:57.854 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:15:57.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:57.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:15:57.958 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:15:57.986 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:58.005 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:15:58.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:58.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:15:58.109 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:15:58.135 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:58.154 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:15:58.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:58.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:15:58.258 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:15:58.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:58 smithi106 ceph-mon[44423]: 2.2 scrub starts 2024-04-20T17:15:58.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:58 smithi106 ceph-mon[44423]: 2.2 scrub ok 2024-04-20T17:15:58.285 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:58.303 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:15:58.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:58.306 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:15:58.407 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:15:58.410 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:58 smithi005 ceph-mon[39562]: 2.2 scrub starts 2024-04-20T17:15:58.410 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:58 smithi005 ceph-mon[39562]: 2.2 scrub ok 2024-04-20T17:15:58.442 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:58.541 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:15:58.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:58.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:15:58.571 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-20T17:15:58.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:58 smithi171 ceph-mon[44189]: 2.2 scrub starts 2024-04-20T17:15:58.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:58 smithi171 ceph-mon[44189]: 2.2 scrub ok 2024-04-20T17:15:58.647 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:15:58.685 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:58.886 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:15:58.890 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:15:58.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:58.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:15:58.999 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:15:59.027 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:59.046 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:15:59.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:59.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:15:59.150 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:15:59.177 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:59.200 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:15:59.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:59.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:15:59.304 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:15:59.328 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:15:59 smithi106 ceph-mon[44423]: pgmap v758: 17 pgs: 1 active+recovering, 16 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 163 op/s; 8198/148346 objects misplaced (5.526%); 731 KiB/s, 11 objects/s recovering 2024-04-20T17:15:59.330 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:59.349 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:15:59.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:15:59.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:15:59.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:15:59 smithi005 ceph-mon[39562]: pgmap v758: 17 pgs: 1 active+recovering, 16 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 163 op/s; 8198/148346 objects misplaced (5.526%); 731 KiB/s, 11 objects/s recovering 2024-04-20T17:15:59.452 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:15:59.482 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:15:59.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:15:59 smithi171 ceph-mon[44189]: pgmap v758: 17 pgs: 1 active+recovering, 16 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 163 op/s; 8198/148346 objects misplaced (5.526%); 731 KiB/s, 11 objects/s recovering 2024-04-20T17:15:59.906 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:15:59.906 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:15:59.906 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:15:59.906 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:15:59.906 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:15:59.906 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:15:59.906 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:15:59.921 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:15:59.952 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:15:59.972 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:15:59.992 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:16:00.013 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:16:00.014 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:16:00.014 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:16:00.014 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:16:00.015 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:16:00.015 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:16:00.015 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:16:00.015 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:16:00.015 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:16:00.015 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:16:00.015 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:16:00.015 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:16:00.015 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:16:00.119 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:16:00.437 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:16:00.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:00.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:16:00.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:00 smithi005 ceph-mon[39562]: 2.8 scrub starts 2024-04-20T17:16:00.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:00 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2244690377' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:16:00.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:00 smithi005 ceph-mon[39562]: from='client.25337 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:16:00.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:00 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2409131935' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:16:00.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:00.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:00 smithi106 ceph-mon[44423]: 2.8 scrub starts 2024-04-20T17:16:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:00 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2244690377' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:16:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:00 smithi106 ceph-mon[44423]: from='client.25337 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:16:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:00 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2409131935' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:16:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:00.541 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:16:00.571 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:00.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:00 smithi171 ceph-mon[44189]: 2.8 scrub starts 2024-04-20T17:16:00.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:00 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2244690377' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:16:00.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:00 smithi171 ceph-mon[44189]: from='client.25337 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:16:00.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:00 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2409131935' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:16:00.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:00.652 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:16:00.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:00.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:16:00.758 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:16:00.785 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:00.803 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:16:00.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:00.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:16:00.908 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:16:00.939 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:01.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:01 smithi005 ceph-mon[39562]: 2.8 scrub ok 2024-04-20T17:16:01.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:01 smithi005 ceph-mon[39562]: pgmap v759: 17 pgs: 1 active+recovering, 16 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 142 op/s; 8198/148036 objects misplaced (5.538%); 491 KiB/s, 7 objects/s recovering 2024-04-20T17:16:01.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:01 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:16:01.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:01 smithi005 ceph-mon[39562]: from='client.25337 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:16:01.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:01 smithi005 ceph-mon[39562]: osdmap e273: 12 total, 12 up, 12 in 2024-04-20T17:16:01.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:01 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:16:01.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:01 smithi005 ceph-mon[39562]: osdmap e274: 12 total, 12 up, 12 in 2024-04-20T17:16:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:01 smithi106 ceph-mon[44423]: 2.8 scrub ok 2024-04-20T17:16:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:01 smithi106 ceph-mon[44423]: pgmap v759: 17 pgs: 1 active+recovering, 16 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 142 op/s; 8198/148036 objects misplaced (5.538%); 491 KiB/s, 7 objects/s recovering 2024-04-20T17:16:01.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:01 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:16:01.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:01 smithi106 ceph-mon[44423]: from='client.25337 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:16:01.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:01 smithi106 ceph-mon[44423]: osdmap e273: 12 total, 12 up, 12 in 2024-04-20T17:16:01.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:01 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:16:01.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:01 smithi106 ceph-mon[44423]: osdmap e274: 12 total, 12 up, 12 in 2024-04-20T17:16:01.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:01 smithi171 ceph-mon[44189]: 2.8 scrub ok 2024-04-20T17:16:01.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:01.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:01 smithi171 ceph-mon[44189]: pgmap v759: 17 pgs: 1 active+recovering, 16 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 142 op/s; 8198/148036 objects misplaced (5.538%); 491 KiB/s, 7 objects/s recovering 2024-04-20T17:16:01.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:01 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:16:01.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:01 smithi171 ceph-mon[44189]: from='client.25337 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:16:01.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:01 smithi171 ceph-mon[44189]: osdmap e273: 12 total, 12 up, 12 in 2024-04-20T17:16:01.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:01 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:16:01.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:01 smithi171 ceph-mon[44189]: osdmap e274: 12 total, 12 up, 12 in 2024-04-20T17:16:02.450 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:16:02.791 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:16:02.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:02.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:16:02.895 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:16:02.924 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:02 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:16:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:02 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:16:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:02 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:16:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:02 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:16:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:02 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:16:03.051 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:02 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:16:03.051 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:02 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:16:03.051 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:02 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:16:03.051 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:02 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:16:03.051 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:02 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:16:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:02 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:16:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:02 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:16:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:02 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:16:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:02 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:16:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:02 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:16:03.674 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:16:03.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:03.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:16:03.780 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:16:03.809 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:03.827 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:16:03.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:03.830 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:16:03.931 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:16:03.959 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:03 smithi005 ceph-mon[39562]: pgmap v762: 17 pgs: 1 active+recovering, 16 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 136 op/s; 8105/147486 objects misplaced (5.495%); 744 KiB/s, 11 objects/s recovering 2024-04-20T17:16:03.962 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:03.987 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:16:03.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:03.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:16:04.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:03 smithi106 ceph-mon[44423]: pgmap v762: 17 pgs: 1 active+recovering, 16 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 136 op/s; 8105/147486 objects misplaced (5.495%); 744 KiB/s, 11 objects/s recovering 2024-04-20T17:16:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:03 smithi171 ceph-mon[44189]: pgmap v762: 17 pgs: 1 active+recovering, 16 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 136 op/s; 8105/147486 objects misplaced (5.495%); 744 KiB/s, 11 objects/s recovering 2024-04-20T17:16:04.091 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:16:04.117 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:04.136 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:16:04.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:04.139 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:16:04.225 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:16:04.227 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:16:04.239 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:16:04.266 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:04.285 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:16:04.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:04.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:16:04.389 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:16:04.415 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:04.433 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:16:04.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:04.436 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:16:04.488 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:16:04.536 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:16:04.562 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:04.580 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:16:04.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:04.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:16:04.683 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:16:04.710 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:04.730 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:16:04.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:04.733 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:16:04.834 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:16:04.860 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:04 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:16:04.860 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:04 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:16:04.860 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:04 smithi005 ceph-mon[39562]: osdmap e275: 12 total, 12 up, 12 in 2024-04-20T17:16:04.863 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:04 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:16:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:04 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:16:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:04 smithi106 ceph-mon[44423]: osdmap e275: 12 total, 12 up, 12 in 2024-04-20T17:16:05.063 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:16:05.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:05.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:16:05.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:04 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:16:05.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:04 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:16:05.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:04 smithi171 ceph-mon[44189]: osdmap e275: 12 total, 12 up, 12 in 2024-04-20T17:16:05.170 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:16:05.197 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:05.216 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:16:05.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:05.219 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:16:05.320 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:16:05.350 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:05.371 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:16:05.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:05.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:16:05.475 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:16:05.505 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:05.823 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":764,"stamp":"2024-04-20T17:16:04.043325+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4813161016,"num_objects":73437,"num_object_clones":0,"num_object_copies":146874,"num_objects_missing_on_primary":4047,"num_objects_missing":4047,"num_objects_degraded":0,"num_objects_misplaced":8105,"num_objects_unfound":0,"num_objects_dirty":73437,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":248730,"num_write_kb":10306363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25166,"num_bytes_recovered":1650870400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160879,"ondisk_log_size":160879,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":37,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":11042604,"kb_used_data":9969580,"kb_used_omap":219,"kb_used_meta":1072676,"kb_avail":1113899220,"statfs":{"total":1151940427776,"available":1140632801280,"internally_reserved":0,"allocated":10208849920,"data_stored":10198951282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224949,"internal_metadata":1098420555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,0,0,1,0,3,5,7],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-48234496,"num_objects":-736,"num_object_clones":0,"num_object_copies":-1472,"num_objects_missing_on_primary":-46,"num_objects_missing":-46,"num_objects_degraded":0,"num_objects_misplaced":-93,"num_objects_unfound":0,"num_objects_dirty":-736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":736,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":3080192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000989"},"pg_stats":[{"pgid":"2.f","version":"274'15550","reported_seq":16334,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-04-20T17:16:03.247159+0000","last_change":"2024-04-20T17:14:39.601610+0000","last_active":"2024-04-20T17:16:03.247159+0000","last_peered":"2024-04-20T17:16:03.247159+0000","last_clean":"2024-04-20T17:16:03.247159+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:16:03.247159+0000","last_undegraded":"2024-04-20T17:16:03.247159+0000","last_fullsized":"2024-04-20T17:16:03.247159+0000","mapping_epoch":115,"log_start":"131'5500","ondisk_log_start":"131'5500","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'14864","last_scrub_stamp":"2024-04-20T17:14:39.601506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:39.601506+0000","objects_scrubbed":5320,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:07.266130+0000","scrub_duration":467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303562752,"num_objects":4632,"num_object_clones":0,"num_object_copies":9264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15550,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"273'15466","reported_seq":27317,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-04-20T17:16:00.231096+0000","last_change":"2024-04-20T17:15:31.328405+0000","last_active":"2024-04-20T17:16:00.231096+0000","last_peered":"2024-04-20T17:16:00.231096+0000","last_clean":"2024-04-20T17:16:00.231096+0000","last_became_active":"2024-04-20T17:10:28.865670+0000","last_became_peered":"2024-04-20T17:10:28.865670+0000","last_unstale":"2024-04-20T17:16:00.231096+0000","last_undegraded":"2024-04-20T17:16:00.231096+0000","last_fullsized":"2024-04-20T17:16:00.231096+0000","mapping_epoch":162,"log_start":"130'5400","ondisk_log_start":"130'5400","created":115,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15200","last_scrub_stamp":"2024-04-20T17:15:31.328301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:31.328301+0000","objects_scrubbed":4857,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:38.848343+0000","scrub_duration":424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300810240,"num_objects":4590,"num_object_clones":0,"num_object_copies":9180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15466,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5619,"num_bytes_recovered":368246784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"272'15798","reported_seq":16473,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-04-20T17:16:00.201423+0000","last_change":"2024-04-20T17:15:57.136522+0000","last_active":"2024-04-20T17:16:00.201423+0000","last_peered":"2024-04-20T17:16:00.201423+0000","last_clean":"2024-04-20T17:16:00.201423+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:16:00.201423+0000","last_undegraded":"2024-04-20T17:16:00.201423+0000","last_fullsized":"2024-04-20T17:16:00.201423+0000","mapping_epoch":115,"log_start":"131'5700","ondisk_log_start":"131'5700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'15780","last_scrub_stamp":"2024-04-20T17:15:57.136412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:57.136412+0000","objects_scrubbed":4702,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:10.130229+0000","scrub_duration":368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306970624,"num_objects":4684,"num_object_clones":0,"num_object_copies":9368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15798,"num_write_kb":655424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"273'22191","reported_seq":43616,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-04-20T17:16:00.339300+0000","last_change":"2024-04-20T17:15:30.287285+0000","last_active":"2024-04-20T17:16:00.339300+0000","last_peered":"2024-04-20T17:16:00.339300+0000","last_clean":"2024-04-20T17:16:00.339300+0000","last_became_active":"2024-04-20T17:12:08.085936+0000","last_became_peered":"2024-04-20T17:12:08.085936+0000","last_unstale":"2024-04-20T17:16:00.339300+0000","last_undegraded":"2024-04-20T17:16:00.339300+0000","last_fullsized":"2024-04-20T17:16:00.339300+0000","mapping_epoch":196,"log_start":"161'12168","ondisk_log_start":"161'12168","created":115,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'21922","last_scrub_stamp":"2024-04-20T17:15:30.287129+0000","last_deep_scrub":"213'20373","last_deep_scrub_stamp":"2024-04-20T17:12:48.443808+0000","last_clean_scrub_stamp":"2024-04-20T17:15:30.287129+0000","objects_scrubbed":4965,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:34.168023+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307691520,"num_objects":4695,"num_object_clones":0,"num_object_copies":9390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15539,"num_write_kb":647488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":450166784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"274'15243","reported_seq":31710,"reported_epoch":274,"state":"active+recovering","last_fresh":"2024-04-20T17:16:01.252802+0000","last_change":"2024-04-20T17:12:26.166109+0000","last_active":"2024-04-20T17:16:01.252802+0000","last_peered":"2024-04-20T17:16:01.252802+0000","last_clean":"2024-04-20T17:09:35.882603+0000","last_became_active":"2024-04-20T17:10:54.470253+0000","last_became_peered":"2024-04-20T17:10:54.470253+0000","last_unstale":"2024-04-20T17:16:01.252802+0000","last_undegraded":"2024-04-20T17:16:01.252802+0000","last_fullsized":"2024-04-20T17:16:01.252802+0000","mapping_epoch":171,"log_start":"129'5200","ondisk_log_start":"129'5200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294060032,"num_objects":4487,"num_object_clones":0,"num_object_copies":8974,"num_objects_missing_on_primary":4047,"num_objects_missing":4047,"num_objects_degraded":0,"num_objects_misplaced":8105,"num_objects_unfound":0,"num_objects_dirty":4487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15243,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5374,"num_bytes_recovered":352190464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"0,2,5","objects":11},{"shards":"0,10","objects":429},{"shards":"2,5","objects":4047}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":734,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-04-20T17:16:01.053241+0000","last_change":"2024-04-20T17:14:40.440515+0000","last_active":"2024-04-20T17:16:01.053241+0000","last_peered":"2024-04-20T17:16:01.053241+0000","last_clean":"2024-04-20T17:16:01.053241+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:16:01.053241+0000","last_undegraded":"2024-04-20T17:16:01.053241+0000","last_fullsized":"2024-04-20T17:16:01.053241+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:14:40.440403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:14:40.440403+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:01.949231+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"274'15623","reported_seq":16331,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-04-20T17:16:02.678607+0000","last_change":"2024-04-20T17:15:31.396169+0000","last_active":"2024-04-20T17:16:02.678607+0000","last_peered":"2024-04-20T17:16:02.678607+0000","last_clean":"2024-04-20T17:16:02.678607+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:16:02.678607+0000","last_undegraded":"2024-04-20T17:16:02.678607+0000","last_fullsized":"2024-04-20T17:16:02.678607+0000","mapping_epoch":115,"log_start":"131'5600","ondisk_log_start":"131'5600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15333","last_scrub_stamp":"2024-04-20T17:15:31.396069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:31.396069+0000","objects_scrubbed":4919,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:57.811864+0000","scrub_duration":440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303366144,"num_objects":4629,"num_object_clones":0,"num_object_copies":9258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15623,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"274'15722","reported_seq":16430,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-04-20T17:16:03.574144+0000","last_change":"2024-04-20T17:15:35.632022+0000","last_active":"2024-04-20T17:16:03.574144+0000","last_peered":"2024-04-20T17:16:03.574144+0000","last_clean":"2024-04-20T17:16:03.574144+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:16:03.574144+0000","last_undegraded":"2024-04-20T17:16:03.574144+0000","last_fullsized":"2024-04-20T17:16:03.574144+0000","mapping_epoch":115,"log_start":"131'5700","ondisk_log_start":"131'5700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15459","last_scrub_stamp":"2024-04-20T17:15:35.631842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:35.631842+0000","objects_scrubbed":4867,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:52.716132+0000","scrub_duration":426,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301596672,"num_objects":4602,"num_object_clones":0,"num_object_copies":9204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15722,"num_write_kb":650368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"274'19666","reported_seq":20471,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-04-20T17:16:01.664051+0000","last_change":"2024-04-20T17:14:45.127365+0000","last_active":"2024-04-20T17:16:01.664051+0000","last_peered":"2024-04-20T17:16:01.664051+0000","last_clean":"2024-04-20T17:16:01.664051+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:16:01.664051+0000","last_undegraded":"2024-04-20T17:16:01.664051+0000","last_fullsized":"2024-04-20T17:16:01.664051+0000","mapping_epoch":115,"log_start":"148'9602","ondisk_log_start":"148'9602","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'19011","last_scrub_stamp":"2024-04-20T17:14:45.127219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:45.127219+0000","objects_scrubbed":5143,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:07.550027+0000","scrub_duration":538,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294060032,"num_objects":4487,"num_object_clones":0,"num_object_copies":8974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15555,"num_write_kb":641344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"274'15664","reported_seq":16429,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-04-20T17:16:02.364550+0000","last_change":"2024-04-20T17:15:41.950854+0000","last_active":"2024-04-20T17:16:02.364550+0000","last_peered":"2024-04-20T17:16:02.364550+0000","last_clean":"2024-04-20T17:16:02.364550+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:16:02.364550+0000","last_undegraded":"2024-04-20T17:16:02.364550+0000","last_fullsized":"2024-04-20T17:16:02.364550+0000","mapping_epoch":115,"log_start":"130'5600","ondisk_log_start":"130'5600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'15491","last_scrub_stamp":"2024-04-20T17:15:41.950723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:41.950723+0000","objects_scrubbed":4733,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:58.122837+0000","scrub_duration":448,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298582016,"num_objects":4556,"num_object_clones":0,"num_object_copies":9112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15664,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"274'15475","reported_seq":16167,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-04-20T17:16:02.770052+0000","last_change":"2024-04-20T17:15:59.551771+0000","last_active":"2024-04-20T17:16:02.770052+0000","last_peered":"2024-04-20T17:16:02.770052+0000","last_clean":"2024-04-20T17:16:02.770052+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:16:02.770052+0000","last_undegraded":"2024-04-20T17:16:02.770052+0000","last_fullsized":"2024-04-20T17:16:02.770052+0000","mapping_epoch":115,"log_start":"130'5400","ondisk_log_start":"130'5400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'15438","last_scrub_stamp":"2024-04-20T17:15:59.551671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:59.551671+0000","objects_scrubbed":4614,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:10.078708+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299958272,"num_objects":4577,"num_object_clones":0,"num_object_copies":9154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15475,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"274'15630","reported_seq":30876,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-04-20T17:16:02.681727+0000","last_change":"2024-04-20T17:14:43.945880+0000","last_active":"2024-04-20T17:16:02.681727+0000","last_peered":"2024-04-20T17:16:02.681727+0000","last_clean":"2024-04-20T17:16:02.681727+0000","last_became_active":"2024-04-20T17:10:55.931751+0000","last_became_peered":"2024-04-20T17:10:55.931751+0000","last_unstale":"2024-04-20T17:16:02.681727+0000","last_undegraded":"2024-04-20T17:16:02.681727+0000","last_fullsized":"2024-04-20T17:16:02.681727+0000","mapping_epoch":173,"log_start":"131'5600","ondisk_log_start":"131'5600","created":115,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'14953","last_scrub_stamp":"2024-04-20T17:14:43.945755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:43.945755+0000","objects_scrubbed":5309,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303562752,"num_objects":4632,"num_object_clones":0,"num_object_copies":9264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15630,"num_write_kb":648384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7296,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"274'15717","reported_seq":16422,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-04-20T17:16:02.360448+0000","last_change":"2024-04-20T17:15:30.785298+0000","last_active":"2024-04-20T17:16:02.360448+0000","last_peered":"2024-04-20T17:16:02.360448+0000","last_clean":"2024-04-20T17:16:02.360448+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:16:02.360448+0000","last_undegraded":"2024-04-20T17:16:02.360448+0000","last_fullsized":"2024-04-20T17:16:02.360448+0000","mapping_epoch":115,"log_start":"131'5700","ondisk_log_start":"131'5700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'15415","last_scrub_stamp":"2024-04-20T17:15:30.785135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:30.785135+0000","objects_scrubbed":4901,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:40.155273+0000","scrub_duration":453,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301400064,"num_objects":4599,"num_object_clones":0,"num_object_copies":9198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15717,"num_write_kb":650112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"274'15294","reported_seq":15998,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-04-20T17:16:03.396519+0000","last_change":"2024-04-20T17:15:34.656600+0000","last_active":"2024-04-20T17:16:03.396519+0000","last_peered":"2024-04-20T17:16:03.396519+0000","last_clean":"2024-04-20T17:16:03.396519+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:16:03.396519+0000","last_undegraded":"2024-04-20T17:16:03.396519+0000","last_fullsized":"2024-04-20T17:16:03.396519+0000","mapping_epoch":115,"log_start":"129'5200","ondisk_log_start":"129'5200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15023","last_scrub_stamp":"2024-04-20T17:15:34.656503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:34.656503+0000","objects_scrubbed":4842,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:51.197304+0000","scrub_duration":430,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299499520,"num_objects":4570,"num_object_clones":0,"num_object_copies":9140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15294,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"274'15541","reported_seq":16245,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-04-20T17:16:02.398616+0000","last_change":"2024-04-20T17:15:31.794340+0000","last_active":"2024-04-20T17:16:02.398616+0000","last_peered":"2024-04-20T17:16:02.398616+0000","last_clean":"2024-04-20T17:16:02.398616+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:16:02.398616+0000","last_undegraded":"2024-04-20T17:16:02.398616+0000","last_fullsized":"2024-04-20T17:16:02.398616+0000","mapping_epoch":115,"log_start":"130'5500","ondisk_log_start":"130'5500","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15271","last_scrub_stamp":"2024-04-20T17:15:31.794156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:31.794156+0000","objects_scrubbed":4831,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:58.805155+0000","scrub_duration":443,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298909696,"num_objects":4561,"num_object_clones":0,"num_object_copies":9122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15541,"num_write_kb":643264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"272'15318","reported_seq":16020,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-04-20T17:16:00.172955+0000","last_change":"2024-04-20T17:15:33.298229+0000","last_active":"2024-04-20T17:16:00.172955+0000","last_peered":"2024-04-20T17:16:00.172955+0000","last_clean":"2024-04-20T17:16:00.172955+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:16:00.172955+0000","last_undegraded":"2024-04-20T17:16:00.172955+0000","last_fullsized":"2024-04-20T17:16:00.172955+0000","mapping_epoch":115,"log_start":"129'5300","ondisk_log_start":"129'5300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15067","last_scrub_stamp":"2024-04-20T17:15:33.298103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:33.298103+0000","objects_scrubbed":4809,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:56.012242+0000","scrub_duration":458,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298254360,"num_objects":4552,"num_object_clones":0,"num_object_copies":9104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4552,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15318,"num_write_kb":635777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"273'15414","reported_seq":15984,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-04-20T17:16:00.114000+0000","last_change":"2024-04-20T17:14:59.605205+0000","last_active":"2024-04-20T17:16:00.114000+0000","last_peered":"2024-04-20T17:16:00.114000+0000","last_clean":"2024-04-20T17:16:00.114000+0000","last_became_active":"2024-04-20T17:12:23.515375+0000","last_became_peered":"2024-04-20T17:12:23.515375+0000","last_unstale":"2024-04-20T17:16:00.114000+0000","last_undegraded":"2024-04-20T17:16:00.114000+0000","last_fullsized":"2024-04-20T17:16:00.114000+0000","mapping_epoch":202,"log_start":"130'5400","ondisk_log_start":"130'5400","created":115,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'14856","last_scrub_stamp":"2024-04-20T17:14:59.605082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:59.605082+0000","objects_scrubbed":5143,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:21.549388+0000","scrub_duration":486,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300285952,"num_objects":4582,"num_object_clones":0,"num_object_copies":9164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15414,"num_write_kb":639872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":4812570648,"num_objects":73435,"num_object_clones":0,"num_object_copies":146870,"num_objects_missing_on_primary":4047,"num_objects_missing":4047,"num_objects_degraded":0,"num_objects_misplaced":8105,"num_objects_unfound":0,"num_objects_dirty":73435,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":248549,"num_write_kb":10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25158,"num_bytes_recovered":1648754688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10174341120,"data_stored":10174332976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},"log_size":160742,"ondisk_log_size":160742,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991356,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":946456,"kb_used_data":883336,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92798696,"statfs":{"total":95995035648,"available":95025864704,"internally_reserved":0,"allocated":904536064,"data_stored":903704396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":64616900},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777252994,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1052680,"kb_used_data":913216,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92692472,"statfs":{"total":95995035648,"available":94917091328,"internally_reserved":0,"allocated":935133184,"data_stored":934305068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":142801344},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612743,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1326708,"kb_used_data":1187244,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92418444,"statfs":{"total":95995035648,"available":94636486656,"internally_reserved":0,"allocated":1215737856,"data_stored":1214917644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":142801344},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762808973,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1271564,"kb_used_data":1172352,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92473588,"statfs":{"total":95995035648,"available":94692954112,"internally_reserved":0,"allocated":1200488448,"data_stored":1199660352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168722,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":961424,"kb_used_data":889992,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92783728,"statfs":{"total":95995035648,"available":95010537472,"internally_reserved":0,"allocated":911351808,"data_stored":910520160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430358,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":679056,"kb_used_data":593728,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93066096,"statfs":{"total":95995035648,"available":95299682304,"internally_reserved":0,"allocated":607977472,"data_stored":607149376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87357894},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575024,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":669516,"kb_used_data":588352,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93075636,"statfs":{"total":95995035648,"available":95309451264,"internally_reserved":0,"allocated":602472448,"data_stored":601644352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738549,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1056700,"kb_used_data":913076,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92688452,"statfs":{"total":95995035648,"available":94912974848,"internally_reserved":0,"allocated":934989824,"data_stored":934167706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1,0,3,5,7],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393837,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":790208,"kb_used_data":745972,"kb_used_omap":202,"kb_used_meta":44021,"kb_avail":92954944,"statfs":{"total":95995035648,"available":95185862656,"internally_reserved":0,"allocated":763875328,"data_stored":763053210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":45077975},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":657280,"kb_used_data":590004,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93087872,"statfs":{"total":95995035648,"available":95321980928,"internally_reserved":0,"allocated":604164096,"data_stored":603341978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986336,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":332404,"kb_used_data":292908,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93412748,"statfs":{"total":95995035648,"available":95654653952,"internally_reserved":0,"allocated":299937792,"data_stored":299117580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790106,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1298608,"kb_used_data":1199400,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92446544,"statfs":{"total":95995035648,"available":94665261056,"internally_reserved":0,"allocated":1228185600,"data_stored":1227369460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":932118528,"data_stored":932118528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":600244224,"data_stored":600244224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":760152064,"data_stored":760152064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":602472448,"data_stored":602472448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":298258432,"data_stored":298254360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1226178560,"data_stored":1226178560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":605224960,"data_stored":605224960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":906887168,"data_stored":906887168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1197277184,"data_stored":1197277184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1213140992,"data_stored":1213136920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":931659776,"data_stored":931659776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":900726784,"data_stored":900726784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:16:05.834 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:16:05.855 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:16:05.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:05.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:16:05.960 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:16:05.989 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:05 smithi106 ceph-mon[44423]: pgmap v764: 17 pgs: 1 active+recovering, 16 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 122 op/s; 8105/146874 objects misplaced (5.518%); 501 KiB/s, 7 objects/s recovering 2024-04-20T17:16:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:05 smithi106 ceph-mon[44423]: from='client.15414 v1:172.21.15.5:0/1750878734' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:16:06.077 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:05 smithi005 ceph-mon[39562]: pgmap v764: 17 pgs: 1 active+recovering, 16 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 122 op/s; 8105/146874 objects misplaced (5.518%); 501 KiB/s, 7 objects/s recovering 2024-04-20T17:16:06.077 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:05 smithi005 ceph-mon[39562]: from='client.15414 v1:172.21.15.5:0/1750878734' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:16:06.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:05 smithi171 ceph-mon[44189]: pgmap v764: 17 pgs: 1 active+recovering, 16 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 122 op/s; 8105/146874 objects misplaced (5.518%); 501 KiB/s, 7 objects/s recovering 2024-04-20T17:16:06.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:05 smithi171 ceph-mon[44189]: from='client.15414 v1:172.21.15.5:0/1750878734' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:16:06.091 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:16:06.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:06.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:16:06.195 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:16:06.222 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:06.242 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:16:06.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:06.245 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:16:06.346 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:16:06.377 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:06.561 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:16:06.630 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:16:06.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:06.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:16:06.735 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:16:06.773 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:06.804 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:16:06.804 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:16:07.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:06 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:16:07.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:06 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:16:07.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:06 smithi005 ceph-mon[39562]: osdmap e276: 12 total, 12 up, 12 in 2024-04-20T17:16:07.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:06 smithi005 ceph-mon[39562]: pgmap v766: 17 pgs: 1 active+recovering, 16 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 132 op/s; 8105/146462 objects misplaced (5.534%); 507 KiB/s, 7 objects/s recovering 2024-04-20T17:16:07.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:06 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:16:07.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:06 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:16:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:06 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:16:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:06 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:16:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:06 smithi106 ceph-mon[44423]: osdmap e276: 12 total, 12 up, 12 in 2024-04-20T17:16:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:06 smithi106 ceph-mon[44423]: pgmap v766: 17 pgs: 1 active+recovering, 16 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 132 op/s; 8105/146462 objects misplaced (5.534%); 507 KiB/s, 7 objects/s recovering 2024-04-20T17:16:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:06 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:16:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:06 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:16:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:06 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:16:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:06 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:16:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:06 smithi171 ceph-mon[44189]: osdmap e276: 12 total, 12 up, 12 in 2024-04-20T17:16:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:06 smithi171 ceph-mon[44189]: pgmap v766: 17 pgs: 1 active+recovering, 16 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 132 op/s; 8105/146462 objects misplaced (5.534%); 507 KiB/s, 7 objects/s recovering 2024-04-20T17:16:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:06 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:16:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:06 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:16:07.426 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:16:07.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:07.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:16:07.530 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:16:07.558 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:07.577 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:16:07.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:07.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:16:07.681 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:16:07.710 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:07.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:07 smithi005 ceph-mon[39562]: osdmap e277: 12 total, 12 up, 12 in 2024-04-20T17:16:08.040 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-20T17:16:08.059 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:16:08.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:08.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:16:08.091 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:16:08.163 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:16:08.190 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:08.208 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:16:08.209 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:07 smithi171 ceph-mon[44189]: osdmap e277: 12 total, 12 up, 12 in 2024-04-20T17:16:08.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:08.211 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:16:08.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:07 smithi106 ceph-mon[44423]: osdmap e277: 12 total, 12 up, 12 in 2024-04-20T17:16:08.308 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:16:08.311 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:16:08.338 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:08.357 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:16:08.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:08.362 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:16:08.463 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:16:08.492 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:08.720 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:16:08.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:08.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:16:08.825 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:16:08.826 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:16:08.857 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:09.167 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:08 smithi106 ceph-mon[44423]: pgmap v768: 17 pgs: 1 active+recovering, 16 active+clean; 4.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 147 op/s; 8009/145716 objects misplaced (5.496%); 501 KiB/s, 7 objects/s recovering 2024-04-20T17:16:09.168 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:08 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:16:09.168 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:08 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:16:09.168 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:08 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:16:09.168 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:08 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:16:09.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:08 smithi005 ceph-mon[39562]: pgmap v768: 17 pgs: 1 active+recovering, 16 active+clean; 4.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 147 op/s; 8009/145716 objects misplaced (5.496%); 501 KiB/s, 7 objects/s recovering 2024-04-20T17:16:09.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:08 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:16:09.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:08 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:16:09.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:08 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:16:09.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:08 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:16:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:08 smithi171 ceph-mon[44189]: pgmap v768: 17 pgs: 1 active+recovering, 16 active+clean; 4.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 147 op/s; 8009/145716 objects misplaced (5.496%); 501 KiB/s, 7 objects/s recovering 2024-04-20T17:16:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:08 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:16:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:08 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:16:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:08 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:16:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:08 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:16:09.814 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:16:09.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:09.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:16:09.918 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:16:09.950 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:09.976 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:16:09.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:09.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:16:10.080 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:16:10.116 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:09 smithi005 ceph-mon[39562]: osdmap e278: 12 total, 12 up, 12 in 2024-04-20T17:16:10.118 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:10.140 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:16:10.142 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:16:10.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:10.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:16:10.164 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:16:10.184 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:16:10.205 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:16:10.226 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:16:10.245 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:16:10.246 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:16:10.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:09 smithi106 ceph-mon[44423]: osdmap e278: 12 total, 12 up, 12 in 2024-04-20T17:16:10.269 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:16:10.273 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:10.289 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:16:10.323 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:16:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:09 smithi171 ceph-mon[44189]: osdmap e278: 12 total, 12 up, 12 in 2024-04-20T17:16:10.349 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:16:10.376 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:16:10.398 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:16:10.400 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:16:10.400 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:16:10.400 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:16:10.400 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:16:10.400 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:16:10.400 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:16:10.400 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:16:10.401 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:16:10.401 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:16:10.401 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:16:10.401 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:16:10.401 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:16:10.472 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:16:10.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:10.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:16:10.577 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:16:10.607 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:10.747 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:16:10.747 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:16:10.747 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-04-20T17:16:10.747 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:16:10.760 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:16:10.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:10.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:16:10.864 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:16:10.893 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:11.060 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:16:11.094 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:16:11.097 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:10 smithi005 ceph-mon[39562]: pgmap v770: 17 pgs: 1 active+recovering, 16 active+clean; 4.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 118 op/s; 8009/145454 objects misplaced (5.506%); 501 KiB/s, 7 objects/s recovering 2024-04-20T17:16:11.097 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:10 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1119055036' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:16:11.097 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.097 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.097 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.097 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.097 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.097 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.097 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.097 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.098 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.098 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:11.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:16:11.199 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:16:11.228 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:10 smithi106 ceph-mon[44423]: pgmap v770: 17 pgs: 1 active+recovering, 16 active+clean; 4.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 118 op/s; 8009/145454 objects misplaced (5.506%); 501 KiB/s, 7 objects/s recovering 2024-04-20T17:16:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:10 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1119055036' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:16:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:10 smithi171 ceph-mon[44189]: pgmap v770: 17 pgs: 1 active+recovering, 16 active+clean; 4.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 118 op/s; 8009/145454 objects misplaced (5.506%); 501 KiB/s, 7 objects/s recovering 2024-04-20T17:16:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:10 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1119055036' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:16:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:11.386 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:16:11.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:11.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:16:11.452 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-20T17:16:11.490 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:16:11.518 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:11.537 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:16:11.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:11.541 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:16:11.642 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:16:11.668 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:11.689 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:16:11.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:11.692 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:16:11.724 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:16:11.792 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:16:11.818 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:11.837 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:16:11.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:11.839 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:16:11.940 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:16:11.968 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:11.988 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:16:11.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:11.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:16:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:12.091 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:16:12.118 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:12.170 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:16:12.171 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:12.172 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:12.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:12.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:16:12.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:12.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:12.273 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:16:12.300 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:12.497 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:16:12.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:12.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:16:12.600 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:16:12.631 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:12.831 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:16:12.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:12.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:16:12.935 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:16:12.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:12 smithi005 ceph-mon[39562]: pgmap v771: 17 pgs: 1 active+recovering, 16 active+clean; 4.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 134 op/s; 7915/144814 objects misplaced (5.466%); 996 KiB/s, 15 objects/s recovering 2024-04-20T17:16:12.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:12 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1774285879' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:16:12.965 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:12.966 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:16:13.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:12 smithi106 ceph-mon[44423]: pgmap v771: 17 pgs: 1 active+recovering, 16 active+clean; 4.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 134 op/s; 7915/144814 objects misplaced (5.466%); 996 KiB/s, 15 objects/s recovering 2024-04-20T17:16:13.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:12 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1774285879' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:16:13.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:12 smithi171 ceph-mon[44189]: pgmap v771: 17 pgs: 1 active+recovering, 16 active+clean; 4.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 134 op/s; 7915/144814 objects misplaced (5.466%); 996 KiB/s, 15 objects/s recovering 2024-04-20T17:16:13.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:12 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1774285879' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:16:13.594 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:16:13.594 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:16:13.594 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-04-20T17:16:13.595 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-04-20T17:16:13.595 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:16:14.018 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:16:14.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:14.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:16:14.124 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:16:14.152 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:14.172 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:16:14.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:14.175 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:16:14.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:13 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:16:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:13 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:16:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:13 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:16:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:13 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1774285879' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:16:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:13 smithi106 ceph-mon[44423]: osdmap e279: 12 total, 12 up, 12 in 2024-04-20T17:16:14.275 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:16:14.305 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:13 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:16:14.306 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:13 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:16:14.306 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:13 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:16:14.306 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:13 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1774285879' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:16:14.306 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:13 smithi005 ceph-mon[39562]: osdmap e279: 12 total, 12 up, 12 in 2024-04-20T17:16:14.307 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:13 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:16:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:13 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:16:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:13 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:16:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:13 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1774285879' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:16:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:13 smithi171 ceph-mon[44189]: osdmap e279: 12 total, 12 up, 12 in 2024-04-20T17:16:14.345 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-20T17:16:14.507 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:16:14.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:14.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:16:14.612 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:16:14.615 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:16:14.640 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:14.835 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:16:14.838 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:16:14.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:14.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:16:14.943 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:16:14.977 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:15.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:14 smithi005 ceph-mon[39562]: pgmap v773: 17 pgs: 1 active+recovering, 16 active+clean; 4.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 172 op/s; 7915/143920 objects misplaced (5.500%); 433 KiB/s, 6 objects/s recovering 2024-04-20T17:16:15.229 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:16:15.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:15.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:16:15.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:14 smithi106 ceph-mon[44423]: pgmap v773: 17 pgs: 1 active+recovering, 16 active+clean; 4.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 172 op/s; 7915/143920 objects misplaced (5.500%); 433 KiB/s, 6 objects/s recovering 2024-04-20T17:16:15.278 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:16:15.335 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:16:15.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:14 smithi171 ceph-mon[44189]: pgmap v773: 17 pgs: 1 active+recovering, 16 active+clean; 4.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 172 op/s; 7915/143920 objects misplaced (5.500%); 433 KiB/s, 6 objects/s recovering 2024-04-20T17:16:15.362 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:15.380 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:16:15.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:15.383 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:16:15.456 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:16:15.457 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:16:15.483 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:16:15.509 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:15.528 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:16:15.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:15.531 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:16:15.632 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:16:15.658 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:15.679 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:16:15.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:15.681 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:16:15.782 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:16:15.809 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:15.830 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:16:15.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:15.833 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:16:15.934 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:16:15.962 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:15.981 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:16:15.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:15.984 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:16:16.051 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:16:16.085 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:16:16.113 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:16.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:15 smithi106 ceph-mon[44423]: from='client.15429 v1:172.21.15.5:0/1078760689' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:16:16.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:15 smithi106 ceph-mon[44423]: osdmap e280: 12 total, 12 up, 12 in 2024-04-20T17:16:16.210 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:16:16.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:15 smithi005 ceph-mon[39562]: from='client.15429 v1:172.21.15.5:0/1078760689' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:16:16.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:15 smithi005 ceph-mon[39562]: osdmap e280: 12 total, 12 up, 12 in 2024-04-20T17:16:16.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:16.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:16:16.314 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:16:16.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:15 smithi171 ceph-mon[44189]: from='client.15429 v1:172.21.15.5:0/1078760689' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:16:16.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:15 smithi171 ceph-mon[44189]: osdmap e280: 12 total, 12 up, 12 in 2024-04-20T17:16:16.342 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:16.419 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:16:16.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:16.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:16:16.478 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:16:16.479 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-04-20T17:16:16.479 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-04-20T17:16:16.479 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:16:16.479 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.479 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:135900 2.d e142/115 2:b066403e:::benchmark_data_smithi171_78918_object135899:head v 142'8409, mlcod=142'8406)", 2024-04-20T17:16:16.479 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:09:21.017880+0000", 2024-04-20T17:16:16.479 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 415.458562594, 2024-04-20T17:16:16.479 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012669634000000001, 2024-04-20T17:16:16.479 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:16.479 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:16.479 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:16.479 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:16.479 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.480 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:16.480 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.017880+0000", 2024-04-20T17:16:16.480 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.480 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.480 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.480 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:16.480 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.017880+0000", 2024-04-20T17:16:16.480 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.480 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.480 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.480 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:16.480 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.017883+0000", 2024-04-20T17:16:16.480 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.196e-06 2024-04-20T17:16:16.480 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.480 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.481 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:16.481 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.017976+0000", 2024-04-20T17:16:16.481 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2625000000000002e-05 2024-04-20T17:16:16.481 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.481 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.481 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:16.481 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.017993+0000", 2024-04-20T17:16:16.481 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7181e-05 2024-04-20T17:16:16.481 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.481 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.481 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:16.481 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.018004+0000", 2024-04-20T17:16:16.481 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1474999999999999e-05 2024-04-20T17:16:16.481 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.481 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.482 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:16.482 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.018061+0000", 2024-04-20T17:16:16.482 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7043000000000003e-05 2024-04-20T17:16:16.482 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.482 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.482 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:16.482 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.018086+0000", 2024-04-20T17:16:16.482 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4368000000000001e-05 2024-04-20T17:16:16.482 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.482 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.482 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:16.482 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.030488+0000", 2024-04-20T17:16:16.483 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012402616999999999 2024-04-20T17:16:16.483 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.483 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.483 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:16.483 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:21.030549+0000", 2024-04-20T17:16:16.484 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1129000000000004e-05 2024-04-20T17:16:16.484 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.484 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:16.484 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.484 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.484 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.484 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:138919 2.d e143/115 2:b227306f:::benchmark_data_smithi171_78918_object138918:head v 143'8595, mlcod=143'8592)", 2024-04-20T17:16:16.485 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:09:23.338496+0000", 2024-04-20T17:16:16.485 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 413.137946038, 2024-04-20T17:16:16.485 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0098117770000000007, 2024-04-20T17:16:16.485 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:16.485 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:16.485 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:16.485 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:16.485 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.485 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:16.485 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338496+0000", 2024-04-20T17:16:16.485 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.485 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.485 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.485 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:16.486 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338496+0000", 2024-04-20T17:16:16.486 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.486 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.486 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.486 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:16.486 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338499+0000", 2024-04-20T17:16:16.486 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4700000000000001e-06 2024-04-20T17:16:16.486 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.486 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.486 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:16.486 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338647+0000", 2024-04-20T17:16:16.487 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014819599999999999 2024-04-20T17:16:16.487 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.487 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.487 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:16.487 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338663+0000", 2024-04-20T17:16:16.487 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6532e-05 2024-04-20T17:16:16.487 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.487 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.487 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:16.487 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338671+0000", 2024-04-20T17:16:16.487 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1779999999999997e-06 2024-04-20T17:16:16.487 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.487 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.488 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:16.488 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338719+0000", 2024-04-20T17:16:16.488 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8152000000000002e-05 2024-04-20T17:16:16.488 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.488 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.488 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:16.488 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338737+0000", 2024-04-20T17:16:16.488 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8117e-05 2024-04-20T17:16:16.488 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.488 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.488 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:16.488 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.348258+0000", 2024-04-20T17:16:16.489 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0095206400000000004 2024-04-20T17:16:16.489 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.489 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.489 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:16.489 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.348308+0000", 2024-04-20T17:16:16.489 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0491999999999997e-05 2024-04-20T17:16:16.489 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.489 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:16.489 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.489 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.489 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.489 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:138920 2.d e143/115 2:ba4e1b76:::benchmark_data_smithi171_78918_object138919:head v 143'8596, mlcod=143'8593)", 2024-04-20T17:16:16.489 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:09:23.338925+0000", 2024-04-20T17:16:16.490 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 413.13751727499999, 2024-04-20T17:16:16.490 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0094244870000000005, 2024-04-20T17:16:16.490 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:16.490 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:16.490 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:16.490 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:16.490 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.491 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:16.491 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338925+0000", 2024-04-20T17:16:16.491 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.491 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.491 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.491 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:16.491 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338925+0000", 2024-04-20T17:16:16.491 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.491 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.491 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.491 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:16.491 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.338926+0000", 2024-04-20T17:16:16.491 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6700000000000002e-07 2024-04-20T17:16:16.492 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.492 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.492 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:16.492 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.339107+0000", 2024-04-20T17:16:16.492 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000181397 2024-04-20T17:16:16.492 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.492 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.492 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:16.492 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.339116+0000", 2024-04-20T17:16:16.492 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9600000000000006e-06 2024-04-20T17:16:16.492 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.492 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.492 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:16.493 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.339123+0000", 2024-04-20T17:16:16.493 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1250000000000004e-06 2024-04-20T17:16:16.493 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.493 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.493 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:16.494 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.341580+0000", 2024-04-20T17:16:16.494 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0024563979999999998 2024-04-20T17:16:16.494 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.494 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.494 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:16.494 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.341606+0000", 2024-04-20T17:16:16.495 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5919999999999999e-05 2024-04-20T17:16:16.495 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.495 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.495 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:16.495 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.348316+0000", 2024-04-20T17:16:16.495 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0067097709999999998 2024-04-20T17:16:16.495 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.495 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.495 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:16.495 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:23.348350+0000", 2024-04-20T17:16:16.495 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3949000000000003e-05 2024-04-20T17:16:16.495 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.496 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:16.496 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.496 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.496 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.496 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:145540 2.d e144/115 2:bbadbea3:::benchmark_data_smithi171_78918_object145539:head v 144'9033, mlcod=144'9030)", 2024-04-20T17:16:16.496 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:09:28.610761+0000", 2024-04-20T17:16:16.496 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 407.86568148700002, 2024-04-20T17:16:16.496 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0095878439999999999, 2024-04-20T17:16:16.496 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:16.496 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:16.496 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:16.496 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:16.496 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.497 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:16.497 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.610761+0000", 2024-04-20T17:16:16.497 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.497 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.497 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.497 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:16.497 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.610761+0000", 2024-04-20T17:16:16.497 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.497 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.497 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.497 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:16.497 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.610763+0000", 2024-04-20T17:16:16.497 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8649999999999999e-06 2024-04-20T17:16:16.498 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.498 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.498 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:16.498 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.610874+0000", 2024-04-20T17:16:16.498 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000111185 2024-04-20T17:16:16.498 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.498 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.498 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:16.498 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.610886+0000", 2024-04-20T17:16:16.498 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2379e-05 2024-04-20T17:16:16.498 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.498 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.498 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:16.499 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.610895+0000", 2024-04-20T17:16:16.499 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3210000000000005e-06 2024-04-20T17:16:16.499 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.499 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.499 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:16.499 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.613942+0000", 2024-04-20T17:16:16.499 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003047892 2024-04-20T17:16:16.499 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.499 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.499 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:16.499 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.613966+0000", 2024-04-20T17:16:16.499 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4000999999999999e-05 2024-04-20T17:16:16.499 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.499 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.499 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:16.500 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.620302+0000", 2024-04-20T17:16:16.500 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0063351930000000002 2024-04-20T17:16:16.500 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.501 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.501 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:16.501 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:28.620349+0000", 2024-04-20T17:16:16.501 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7008000000000001e-05 2024-04-20T17:16:16.501 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.501 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:16.501 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.502 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.502 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.502 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:146284 2.d e145/115 2:ba16209d:::benchmark_data_smithi171_78918_object146283:head v 145'9080, mlcod=145'9077)", 2024-04-20T17:16:16.502 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:09:29.190321+0000", 2024-04-20T17:16:16.502 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 407.28612133199999, 2024-04-20T17:16:16.502 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.009650166, 2024-04-20T17:16:16.502 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:16.502 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:16.502 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:16.502 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:16.502 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.503 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:16.503 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.190321+0000", 2024-04-20T17:16:16.503 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.503 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.503 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.503 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:16.503 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.190321+0000", 2024-04-20T17:16:16.503 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.503 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.503 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.503 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:16.503 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.190322+0000", 2024-04-20T17:16:16.504 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2500000000000005e-07 2024-04-20T17:16:16.504 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.504 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.504 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:16.504 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.190371+0000", 2024-04-20T17:16:16.504 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9458e-05 2024-04-20T17:16:16.504 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.505 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.505 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:16.505 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.190376+0000", 2024-04-20T17:16:16.505 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5619999999999997e-06 2024-04-20T17:16:16.505 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.505 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.505 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:16.505 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.190379+0000", 2024-04-20T17:16:16.505 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.557e-06 2024-04-20T17:16:16.505 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.506 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.506 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:16.506 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.193010+0000", 2024-04-20T17:16:16.506 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0026303440000000002 2024-04-20T17:16:16.506 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.506 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.506 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:16.506 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.193033+0000", 2024-04-20T17:16:16.506 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3027000000000002e-05 2024-04-20T17:16:16.506 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.506 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.506 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:16.507 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.199918+0000", 2024-04-20T17:16:16.507 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0068856550000000001 2024-04-20T17:16:16.507 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.507 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.507 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:16.507 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:09:29.199971+0000", 2024-04-20T17:16:16.507 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2837999999999998e-05 2024-04-20T17:16:16.507 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.507 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:16.508 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.508 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.508 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.508 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:180296 2.d e156/115 2:b3863235:::benchmark_data_smithi171_78918_object19302:head v 156'11076, mlcod=156'11074)", 2024-04-20T17:16:16.508 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:11.606812+0000", 2024-04-20T17:16:16.508 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 364.86963054900002, 2024-04-20T17:16:16.509 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010515645000000001, 2024-04-20T17:16:16.509 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:16.509 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:16.509 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:16.509 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:16.509 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.509 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:16.510 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.606812+0000", 2024-04-20T17:16:16.510 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.510 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.511 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.511 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:16.511 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.606812+0000", 2024-04-20T17:16:16.511 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.511 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.511 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.511 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:16.511 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.606814+0000", 2024-04-20T17:16:16.511 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2819999999999999e-06 2024-04-20T17:16:16.511 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.511 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.512 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:16.512 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.606822+0000", 2024-04-20T17:16:16.512 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0700000000000007e-06 2024-04-20T17:16:16.512 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.512 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.512 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:16.512 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.606835+0000", 2024-04-20T17:16:16.512 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3154e-05 2024-04-20T17:16:16.512 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.512 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.512 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:16.512 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.606843+0000", 2024-04-20T17:16:16.512 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2290000000000003e-06 2024-04-20T17:16:16.512 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.512 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.513 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:16.513 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.606904+0000", 2024-04-20T17:16:16.513 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0714000000000002e-05 2024-04-20T17:16:16.513 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.513 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.513 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:16.513 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.606928+0000", 2024-04-20T17:16:16.513 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.336e-05 2024-04-20T17:16:16.513 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.513 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.513 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:16.513 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.617278+0000", 2024-04-20T17:16:16.513 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010350597 2024-04-20T17:16:16.513 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.513 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.513 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:16.514 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:11.617327+0000", 2024-04-20T17:16:16.514 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9239000000000001e-05 2024-04-20T17:16:16.514 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.514 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:16.514 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.514 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.514 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.514 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:202407 2.d e166/115 2:be74e115:::benchmark_data_smithi171_78918_object41413:head v 166'12446, mlcod=166'12442)", 2024-04-20T17:16:16.514 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:39.985793+0000", 2024-04-20T17:16:16.514 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 336.490649206, 2024-04-20T17:16:16.514 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0095545500000000002, 2024-04-20T17:16:16.514 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:16.514 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:16.514 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:16.514 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:16.515 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.515 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:16.515 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.985793+0000", 2024-04-20T17:16:16.515 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.515 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.515 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.515 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:16.515 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.985793+0000", 2024-04-20T17:16:16.515 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.515 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.515 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.515 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:16.515 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.985794+0000", 2024-04-20T17:16:16.515 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3e-06 2024-04-20T17:16:16.515 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.516 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.516 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:16.516 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.985801+0000", 2024-04-20T17:16:16.516 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1890000000000004e-06 2024-04-20T17:16:16.516 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.516 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.516 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:16.516 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.985811+0000", 2024-04-20T17:16:16.516 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.06e-05 2024-04-20T17:16:16.516 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.516 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.516 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:16.516 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.985819+0000", 2024-04-20T17:16:16.516 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6580000000000002e-06 2024-04-20T17:16:16.516 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.517 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.517 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:16.517 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.990292+0000", 2024-04-20T17:16:16.517 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0044734869999999999 2024-04-20T17:16:16.517 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.517 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.517 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:16.517 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.990308+0000", 2024-04-20T17:16:16.518 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5389000000000001e-05 2024-04-20T17:16:16.518 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.518 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.518 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:16.518 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.995301+0000", 2024-04-20T17:16:16.518 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0049930010000000004 2024-04-20T17:16:16.518 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.518 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.518 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:16.518 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.995348+0000", 2024-04-20T17:16:16.518 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6925999999999998e-05 2024-04-20T17:16:16.518 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.518 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:16.518 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.519 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.519 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.519 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:202710 2.d e166/115 2:b3a6797d:::benchmark_data_smithi171_78918_object41716:head v 166'12468, mlcod=166'12466)", 2024-04-20T17:16:16.519 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:40.303984+0000", 2024-04-20T17:16:16.519 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 336.17245858299998, 2024-04-20T17:16:16.519 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.013242804, 2024-04-20T17:16:16.519 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:16.519 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:16.519 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:16.519 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:16.519 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.519 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:16.519 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.303984+0000", 2024-04-20T17:16:16.519 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.519 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.520 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.520 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:16.520 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.303984+0000", 2024-04-20T17:16:16.520 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.520 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.520 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.520 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:16.520 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.303986+0000", 2024-04-20T17:16:16.520 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0099999999999998e-06 2024-04-20T17:16:16.520 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.520 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.521 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:16.521 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.303997+0000", 2024-04-20T17:16:16.521 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0852999999999999e-05 2024-04-20T17:16:16.521 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.521 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.521 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:16.521 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.304011+0000", 2024-04-20T17:16:16.522 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4304000000000001e-05 2024-04-20T17:16:16.522 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.522 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.522 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:16.522 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.304020+0000", 2024-04-20T17:16:16.522 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6449999999999994e-06 2024-04-20T17:16:16.522 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.522 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.522 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:16.522 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.304086+0000", 2024-04-20T17:16:16.522 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6268999999999999e-05 2024-04-20T17:16:16.522 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.522 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.522 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:16.522 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.304111+0000", 2024-04-20T17:16:16.523 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5480999999999999e-05 2024-04-20T17:16:16.523 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.523 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.523 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:16.523 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.317176+0000", 2024-04-20T17:16:16.523 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.013064697 2024-04-20T17:16:16.523 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.523 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.523 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:16.523 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.317227+0000", 2024-04-20T17:16:16.523 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0544999999999998e-05 2024-04-20T17:16:16.523 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.523 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:16.523 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.523 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.524 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.524 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:202714 2.d e166/115 2:b3a8ef48:::benchmark_data_smithi171_78918_object41720:head v 166'12469, mlcod=166'12466)", 2024-04-20T17:16:16.524 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:40.308710+0000", 2024-04-20T17:16:16.524 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 336.16773214400001, 2024-04-20T17:16:16.524 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012740461, 2024-04-20T17:16:16.524 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:16.524 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:16.524 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:16.524 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:16.524 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.524 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:16.524 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.308710+0000", 2024-04-20T17:16:16.524 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.524 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.524 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.525 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:16.525 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.308710+0000", 2024-04-20T17:16:16.525 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.525 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.525 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.525 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:16.525 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.308712+0000", 2024-04-20T17:16:16.525 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0590000000000001e-06 2024-04-20T17:16:16.525 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.525 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.525 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:16.525 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.308722+0000", 2024-04-20T17:16:16.525 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0236e-05 2024-04-20T17:16:16.525 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.525 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.526 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:16.526 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.308735+0000", 2024-04-20T17:16:16.526 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2743e-05 2024-04-20T17:16:16.526 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.526 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.526 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:16.526 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.308744+0000", 2024-04-20T17:16:16.526 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8529999999999998e-06 2024-04-20T17:16:16.526 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.526 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.526 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:16.526 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.308809+0000", 2024-04-20T17:16:16.526 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4975999999999996e-05 2024-04-20T17:16:16.526 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.526 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.527 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:16.527 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.308834+0000", 2024-04-20T17:16:16.527 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5329000000000001e-05 2024-04-20T17:16:16.527 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.527 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.527 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:16.527 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.321396+0000", 2024-04-20T17:16:16.527 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012561948 2024-04-20T17:16:16.527 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.527 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.527 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:16.527 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:40.321451+0000", 2024-04-20T17:16:16.527 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4317000000000002e-05 2024-04-20T17:16:16.527 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.527 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:16.527 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.528 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.528 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.528 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:203759 2.d e167/115 2:b448a1cf:::benchmark_data_smithi171_78918_object42765:head v 167'12540, mlcod=167'12537)", 2024-04-20T17:16:16.528 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:41.412120+0000", 2024-04-20T17:16:16.528 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 335.06432255499999, 2024-04-20T17:16:16.528 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010611162, 2024-04-20T17:16:16.528 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:16.528 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:16.528 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:16.528 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:16.528 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.528 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:16.528 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.412120+0000", 2024-04-20T17:16:16.528 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.528 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.529 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.529 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:16.529 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.412120+0000", 2024-04-20T17:16:16.529 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.529 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.529 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.529 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:16.529 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.412121+0000", 2024-04-20T17:16:16.529 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7000000000000003e-07 2024-04-20T17:16:16.529 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.529 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.529 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:16.529 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.412126+0000", 2024-04-20T17:16:16.529 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8629999999999999e-06 2024-04-20T17:16:16.529 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.530 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.530 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:16.530 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.412132+0000", 2024-04-20T17:16:16.530 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3770000000000001e-06 2024-04-20T17:16:16.530 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.530 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.530 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:16.530 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.412136+0000", 2024-04-20T17:16:16.530 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2030000000000002e-06 2024-04-20T17:16:16.530 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.530 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.530 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:16.530 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.415923+0000", 2024-04-20T17:16:16.530 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037866380000000002 2024-04-20T17:16:16.531 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.531 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.531 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:16.531 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.415956+0000", 2024-04-20T17:16:16.531 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3308e-05 2024-04-20T17:16:16.531 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.531 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.531 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:16.531 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.422682+0000", 2024-04-20T17:16:16.531 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0067258149999999996 2024-04-20T17:16:16.532 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.532 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.532 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:16.532 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.422731+0000", 2024-04-20T17:16:16.532 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8987999999999999e-05 2024-04-20T17:16:16.532 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.532 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:16.532 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.532 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.532 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.532 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:203788 2.d e167/115 2:b22759bd:::benchmark_data_smithi171_78918_object42794:head v 167'12544, mlcod=167'12540)", 2024-04-20T17:16:16.532 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:41.437431+0000", 2024-04-20T17:16:16.532 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 335.03901098, 2024-04-20T17:16:16.532 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.009289472, 2024-04-20T17:16:16.532 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:16.533 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:16.533 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:16.533 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:16.533 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.533 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:16.533 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.437431+0000", 2024-04-20T17:16:16.533 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.533 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.533 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.533 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:16.533 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.437431+0000", 2024-04-20T17:16:16.533 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.533 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.533 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.533 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:16.534 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.437432+0000", 2024-04-20T17:16:16.534 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0809999999999999e-06 2024-04-20T17:16:16.534 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.534 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.534 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:16.534 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.437437+0000", 2024-04-20T17:16:16.534 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8459999999999999e-06 2024-04-20T17:16:16.534 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.534 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.534 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:16.534 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.437445+0000", 2024-04-20T17:16:16.534 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7249999999999997e-06 2024-04-20T17:16:16.534 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.534 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.535 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:16.535 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.437450+0000", 2024-04-20T17:16:16.535 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9250000000000003e-06 2024-04-20T17:16:16.535 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.535 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.535 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:16.535 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.441273+0000", 2024-04-20T17:16:16.535 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003822607 2024-04-20T17:16:16.535 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.535 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.535 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:16.535 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.441298+0000", 2024-04-20T17:16:16.535 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5850000000000002e-05 2024-04-20T17:16:16.535 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.535 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.536 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:16.536 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.446674+0000", 2024-04-20T17:16:16.536 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0053756129999999996 2024-04-20T17:16:16.536 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.536 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.536 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:16.536 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.446721+0000", 2024-04-20T17:16:16.536 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6825e-05 2024-04-20T17:16:16.536 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.536 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:16.536 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.536 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.536 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.536 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:207844 2.d e167/115 2:b4c70255:::benchmark_data_smithi171_78918_object46850:head v 167'12787, mlcod=167'12784)", 2024-04-20T17:16:16.536 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:45.866401+0000", 2024-04-20T17:16:16.537 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 330.61004096400001, 2024-04-20T17:16:16.537 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010044907, 2024-04-20T17:16:16.537 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:16.537 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:16.537 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:16.537 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:16.537 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.537 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:16.537 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866401+0000", 2024-04-20T17:16:16.537 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.537 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.537 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.537 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:16.537 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866401+0000", 2024-04-20T17:16:16.537 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.538 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.538 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.538 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:16.538 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866405+0000", 2024-04-20T17:16:16.538 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.67e-06 2024-04-20T17:16:16.538 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.538 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.538 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:16.538 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866421+0000", 2024-04-20T17:16:16.538 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6113e-05 2024-04-20T17:16:16.538 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.538 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.538 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:16.538 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866450+0000", 2024-04-20T17:16:16.539 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8915999999999999e-05 2024-04-20T17:16:16.539 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.539 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.539 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:16.539 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866470+0000", 2024-04-20T17:16:16.539 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.959e-05 2024-04-20T17:16:16.539 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.539 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.539 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:16.539 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866607+0000", 2024-04-20T17:16:16.539 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013733699999999999 2024-04-20T17:16:16.539 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.539 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.539 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:16.539 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866666+0000", 2024-04-20T17:16:16.539 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9147999999999997e-05 2024-04-20T17:16:16.540 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.540 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.540 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:16.540 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.876418+0000", 2024-04-20T17:16:16.540 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0097520650000000007 2024-04-20T17:16:16.540 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.540 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.540 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:16.540 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.876446+0000", 2024-04-20T17:16:16.540 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8068e-05 2024-04-20T17:16:16.540 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.540 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:16.540 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.540 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.540 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.541 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:207851 2.d e167/115 2:bc3064f0:::benchmark_data_smithi171_78918_object46857:head v 167'12788, mlcod=167'12785)", 2024-04-20T17:16:16.541 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:45.866896+0000", 2024-04-20T17:16:16.541 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 330.60954609200002, 2024-04-20T17:16:16.541 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0095667549999999997, 2024-04-20T17:16:16.541 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:16.541 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:16.541 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:16.541 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:16.541 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.541 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:16.541 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866896+0000", 2024-04-20T17:16:16.541 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.541 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.542 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.542 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:16.542 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866896+0000", 2024-04-20T17:16:16.542 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.542 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.542 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.542 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:16.542 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866897+0000", 2024-04-20T17:16:16.542 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.153e-06 2024-04-20T17:16:16.542 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.542 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.543 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:16.543 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866904+0000", 2024-04-20T17:16:16.543 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6050000000000003e-06 2024-04-20T17:16:16.543 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.543 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.543 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:16.543 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866915+0000", 2024-04-20T17:16:16.543 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0778000000000001e-05 2024-04-20T17:16:16.543 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.543 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.543 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:16.543 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.866922+0000", 2024-04-20T17:16:16.543 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.464e-06 2024-04-20T17:16:16.543 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.543 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.544 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:16.544 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.870735+0000", 2024-04-20T17:16:16.544 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0038132550000000002 2024-04-20T17:16:16.544 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.544 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.544 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:16.544 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.870756+0000", 2024-04-20T17:16:16.544 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0865000000000001e-05 2024-04-20T17:16:16.544 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.544 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.544 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:16.544 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.876449+0000", 2024-04-20T17:16:16.544 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0056927790000000002 2024-04-20T17:16:16.544 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.545 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.545 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:16.545 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:45.876463+0000", 2024-04-20T17:16:16.545 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3856000000000001e-05 2024-04-20T17:16:16.545 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.545 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:16.545 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.545 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.545 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.545 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:208672 2.d e167/115 2:be8432c1:::benchmark_data_smithi171_78918_object47678:head v 167'12834, mlcod=167'12831)", 2024-04-20T17:16:16.545 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:46.774497+0000", 2024-04-20T17:16:16.545 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 329.70194525300002, 2024-04-20T17:16:16.545 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0092546139999999995, 2024-04-20T17:16:16.545 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:16.545 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:16.546 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:16.546 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:16.546 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.546 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:16.546 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:46.774497+0000", 2024-04-20T17:16:16.546 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.546 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.546 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.546 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:16.546 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:46.774497+0000", 2024-04-20T17:16:16.546 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.546 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.546 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.546 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:16.547 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:46.774498+0000", 2024-04-20T17:16:16.547 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2950000000000001e-06 2024-04-20T17:16:16.547 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.547 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.547 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:16.547 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:46.774505+0000", 2024-04-20T17:16:16.547 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5860000000000001e-06 2024-04-20T17:16:16.547 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.547 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.547 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:16.547 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:46.774516+0000", 2024-04-20T17:16:16.547 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0829000000000001e-05 2024-04-20T17:16:16.547 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.547 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.547 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:16.548 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:46.774522+0000", 2024-04-20T17:16:16.548 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.421e-06 2024-04-20T17:16:16.548 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.548 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.548 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:16.548 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:46.778316+0000", 2024-04-20T17:16:16.548 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003794296 2024-04-20T17:16:16.548 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.548 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.548 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:16.548 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:46.778336+0000", 2024-04-20T17:16:16.548 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9119000000000001e-05 2024-04-20T17:16:16.548 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.548 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.548 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:16.549 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:46.783708+0000", 2024-04-20T17:16:16.549 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.005372501 2024-04-20T17:16:16.549 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.549 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.549 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:16.549 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:46.783752+0000", 2024-04-20T17:16:16.549 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3566999999999998e-05 2024-04-20T17:16:16.549 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.549 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:16.549 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.549 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.549 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.549 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:209742 2.d e168/115 2:b2c27aa2:::benchmark_data_smithi171_78918_object48748:head v 168'12891, mlcod=168'12889)", 2024-04-20T17:16:16.549 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:47.949446+0000", 2024-04-20T17:16:16.549 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 328.52699679400001, 2024-04-20T17:16:16.550 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0097404459999999998, 2024-04-20T17:16:16.550 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:16.550 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:16.550 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:16.550 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:16.550 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.550 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:16.550 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949446+0000", 2024-04-20T17:16:16.550 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.550 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.550 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.550 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:16.550 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949446+0000", 2024-04-20T17:16:16.550 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.550 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.551 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.551 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:16.551 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949448+0000", 2024-04-20T17:16:16.551 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.531e-06 2024-04-20T17:16:16.551 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.551 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.551 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:16.551 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949457+0000", 2024-04-20T17:16:16.551 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1330000000000005e-06 2024-04-20T17:16:16.551 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.551 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.551 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:16.551 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949471+0000", 2024-04-20T17:16:16.551 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3478999999999999e-05 2024-04-20T17:16:16.552 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.552 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.552 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:16.552 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949480+0000", 2024-04-20T17:16:16.552 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8999999999999995e-06 2024-04-20T17:16:16.552 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.552 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.552 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:16.552 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949528+0000", 2024-04-20T17:16:16.552 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8916999999999999e-05 2024-04-20T17:16:16.552 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.553 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.553 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:16.553 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949547+0000", 2024-04-20T17:16:16.553 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8362000000000001e-05 2024-04-20T17:16:16.553 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.553 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.553 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:16.553 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.959142+0000", 2024-04-20T17:16:16.553 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0095948030000000007 2024-04-20T17:16:16.553 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.553 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.553 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:16.553 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.959186+0000", 2024-04-20T17:16:16.553 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4320999999999998e-05 2024-04-20T17:16:16.553 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.554 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:16.554 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.554 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.554 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.554 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:209749 2.d e168/115 2:b2c3330a:::benchmark_data_smithi171_78918_object48755:head v 168'12892, mlcod=168'12889)", 2024-04-20T17:16:16.554 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:47.949983+0000", 2024-04-20T17:16:16.554 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 328.52645925299998, 2024-04-20T17:16:16.554 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0092349100000000007, 2024-04-20T17:16:16.554 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:16.554 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:16.554 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:16.554 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:16.554 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.554 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:16.554 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949983+0000", 2024-04-20T17:16:16.555 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.555 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.555 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.555 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:16.555 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949983+0000", 2024-04-20T17:16:16.555 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.555 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.555 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.555 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:16.555 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949984+0000", 2024-04-20T17:16:16.555 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7100000000000001e-07 2024-04-20T17:16:16.555 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.555 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.555 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:16.555 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949988+0000", 2024-04-20T17:16:16.555 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5009999999999998e-06 2024-04-20T17:16:16.556 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.556 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.556 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:16.556 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949994+0000", 2024-04-20T17:16:16.556 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1680000000000001e-06 2024-04-20T17:16:16.556 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.556 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.556 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:16.556 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.949998+0000", 2024-04-20T17:16:16.556 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.856e-06 2024-04-20T17:16:16.556 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.556 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.556 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:16.556 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.952247+0000", 2024-04-20T17:16:16.556 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022486509999999999 2024-04-20T17:16:16.557 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.557 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.557 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:16.557 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.952268+0000", 2024-04-20T17:16:16.557 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1228000000000001e-05 2024-04-20T17:16:16.557 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.557 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.557 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:16.557 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.959190+0000", 2024-04-20T17:16:16.557 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0069222399999999996 2024-04-20T17:16:16.557 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.557 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.557 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:16.557 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:47.959218+0000", 2024-04-20T17:16:16.557 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7495e-05 2024-04-20T17:16:16.558 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.558 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:16.558 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.558 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.558 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.558 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:210465 2.d e168/115 2:bf240863:::benchmark_data_smithi171_78918_object49471:head v 168'12947, mlcod=168'12944)", 2024-04-20T17:16:16.558 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:48.753722+0000", 2024-04-20T17:16:16.558 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 327.722720063, 2024-04-20T17:16:16.558 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.014363803, 2024-04-20T17:16:16.558 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:16.558 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:16.558 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:16.558 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:16.558 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.559 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:16.559 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753722+0000", 2024-04-20T17:16:16.559 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.559 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.559 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.559 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:16.559 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753722+0000", 2024-04-20T17:16:16.559 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.559 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.559 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.559 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:16.559 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753725+0000", 2024-04-20T17:16:16.559 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5110000000000002e-06 2024-04-20T17:16:16.559 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.559 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.560 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:16.560 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753737+0000", 2024-04-20T17:16:16.560 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1874e-05 2024-04-20T17:16:16.560 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.560 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.560 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:16.560 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753757+0000", 2024-04-20T17:16:16.560 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0670999999999999e-05 2024-04-20T17:16:16.560 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.560 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.560 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:16.560 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753768+0000", 2024-04-20T17:16:16.560 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0219e-05 2024-04-20T17:16:16.560 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.560 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.561 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:16.561 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753854+0000", 2024-04-20T17:16:16.561 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6948999999999997e-05 2024-04-20T17:16:16.561 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.561 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.561 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:16.561 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753888+0000", 2024-04-20T17:16:16.561 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3741000000000001e-05 2024-04-20T17:16:16.561 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.561 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.561 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:16.561 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.768045+0000", 2024-04-20T17:16:16.561 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.014157085 2024-04-20T17:16:16.561 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.561 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.562 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:16.562 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.768086+0000", 2024-04-20T17:16:16.562 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0753000000000001e-05 2024-04-20T17:16:16.562 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.562 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:16.562 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.562 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.562 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.562 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:210469 2.d e168/115 2:b66681fa:::benchmark_data_smithi171_78918_object49475:head v 168'12948, mlcod=168'12944)", 2024-04-20T17:16:16.562 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:48.753793+0000", 2024-04-20T17:16:16.562 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 327.72264969499997, 2024-04-20T17:16:16.563 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.014324059, 2024-04-20T17:16:16.563 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:16.563 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:16.563 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:16.563 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:16.563 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.563 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:16.563 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753793+0000", 2024-04-20T17:16:16.563 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.563 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.563 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.563 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:16.563 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753793+0000", 2024-04-20T17:16:16.563 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.563 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.564 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.564 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:16.564 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753793+0000", 2024-04-20T17:16:16.564 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3900000000000004e-07 2024-04-20T17:16:16.564 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.564 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.564 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:16.564 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753803+0000", 2024-04-20T17:16:16.564 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8130000000000007e-06 2024-04-20T17:16:16.564 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.564 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.564 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:16.564 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753810+0000", 2024-04-20T17:16:16.564 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6069999999999996e-06 2024-04-20T17:16:16.565 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.565 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.565 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:16.565 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753815+0000", 2024-04-20T17:16:16.565 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6030000000000004e-06 2024-04-20T17:16:16.565 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.565 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.565 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:16.565 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.757230+0000", 2024-04-20T17:16:16.565 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003414182 2024-04-20T17:16:16.565 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.565 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.565 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:16.565 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.757255+0000", 2024-04-20T17:16:16.565 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5324999999999999e-05 2024-04-20T17:16:16.565 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.566 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.566 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:16.566 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.768091+0000", 2024-04-20T17:16:16.566 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010835869999999999 2024-04-20T17:16:16.566 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.566 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.566 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:16.566 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.768117+0000", 2024-04-20T17:16:16.566 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5919999999999999e-05 2024-04-20T17:16:16.566 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.566 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:16.566 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.566 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.566 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.567 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:210471 2.d e168/115 2:b5294da4:::benchmark_data_smithi171_78918_object49477:head v 168'12949, mlcod=168'12944)", 2024-04-20T17:16:16.567 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:48.753843+0000", 2024-04-20T17:16:16.567 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 327.72259967100001, 2024-04-20T17:16:16.567 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.014299944, 2024-04-20T17:16:16.567 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:16.567 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:16.567 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:16.567 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:16.567 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.567 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:16.567 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753843+0000", 2024-04-20T17:16:16.567 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.567 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.567 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.567 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:16.568 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753843+0000", 2024-04-20T17:16:16.568 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.568 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.568 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.568 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:16.568 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753843+0000", 2024-04-20T17:16:16.568 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2200000000000004e-07 2024-04-20T17:16:16.568 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.568 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.568 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:16.568 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753848+0000", 2024-04-20T17:16:16.568 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.454e-06 2024-04-20T17:16:16.568 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.568 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.568 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:16.568 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753856+0000", 2024-04-20T17:16:16.569 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2069999999999995e-06 2024-04-20T17:16:16.569 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.569 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.569 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:16.569 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.753861+0000", 2024-04-20T17:16:16.569 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1540000000000001e-06 2024-04-20T17:16:16.569 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.569 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.569 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:16.569 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.760194+0000", 2024-04-20T17:16:16.569 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0063330349999999999 2024-04-20T17:16:16.569 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.569 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.569 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:16.569 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.760219+0000", 2024-04-20T17:16:16.570 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4428999999999999e-05 2024-04-20T17:16:16.570 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.570 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.570 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:16.570 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.768119+0000", 2024-04-20T17:16:16.570 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0079001060000000005 2024-04-20T17:16:16.570 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.570 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.570 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:16.570 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.768143+0000", 2024-04-20T17:16:16.570 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3937000000000001e-05 2024-04-20T17:16:16.570 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.570 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:16.570 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.571 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.571 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.571 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:210474 2.d e168/115 2:b8832972:::benchmark_data_smithi171_78918_object49480:head v 168'12950, mlcod=168'12945)", 2024-04-20T17:16:16.571 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:48.758784+0000", 2024-04-20T17:16:16.571 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 327.71765788599998, 2024-04-20T17:16:16.571 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0094639479999999998, 2024-04-20T17:16:16.571 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:16.571 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:16.571 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:16.571 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:16.571 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.571 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:16.571 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.758784+0000", 2024-04-20T17:16:16.571 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.571 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.572 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.572 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:16.572 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.758784+0000", 2024-04-20T17:16:16.572 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:16.572 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.572 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.572 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:16.572 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.758786+0000", 2024-04-20T17:16:16.572 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8309999999999999e-06 2024-04-20T17:16:16.572 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.572 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.573 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:16.573 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.758797+0000", 2024-04-20T17:16:16.573 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1024e-05 2024-04-20T17:16:16.573 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.573 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.573 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:16.573 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.758808+0000", 2024-04-20T17:16:16.573 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0569e-05 2024-04-20T17:16:16.573 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.573 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.573 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:16.573 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.758816+0000", 2024-04-20T17:16:16.573 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3319999999999992e-06 2024-04-20T17:16:16.573 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.573 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.574 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:16.574 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.764056+0000", 2024-04-20T17:16:16.574 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0052398640000000003 2024-04-20T17:16:16.574 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.574 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.574 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:16.574 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.764082+0000", 2024-04-20T17:16:16.574 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5616999999999999e-05 2024-04-20T17:16:16.574 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.574 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.574 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:16.574 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.768223+0000", 2024-04-20T17:16:16.574 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0041416029999999998 2024-04-20T17:16:16.574 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:16.574 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:16.574 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:16.575 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:48.768248+0000", 2024-04-20T17:16:16.575 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5108000000000001e-05 2024-04-20T17:16:16.575 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.575 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:16.575 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.575 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:16.575 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:16.575 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:16:16.575 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:16:16.603 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:16.726 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:16:16.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:16.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:16:16.831 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:16:16.861 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:16.894 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:16:16.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:16.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:16:16.944 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:16:16.998 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:16:17.030 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:17.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:16 smithi005 ceph-mon[39562]: pgmap v775: 17 pgs: 1 active+recovering, 16 active+clean; 4.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 139 op/s; 7915/143472 objects misplaced (5.517%); 433 KiB/s, 6 objects/s recovering 2024-04-20T17:16:17.229 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:16 smithi106 ceph-mon[44423]: pgmap v775: 17 pgs: 1 active+recovering, 16 active+clean; 4.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 139 op/s; 7915/143472 objects misplaced (5.517%); 433 KiB/s, 6 objects/s recovering 2024-04-20T17:16:17.254 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-20T17:16:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:16 smithi171 ceph-mon[44189]: pgmap v775: 17 pgs: 1 active+recovering, 16 active+clean; 4.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 139 op/s; 7915/143472 objects misplaced (5.517%); 433 KiB/s, 6 objects/s recovering 2024-04-20T17:16:17.519 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:16:18.324 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":773,"stamp":"2024-04-20T17:16:14.045087+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4716364344,"num_objects":71960,"num_object_clones":0,"num_object_copies":143920,"num_objects_missing_on_primary":3952,"num_objects_missing":3952,"num_objects_degraded":0,"num_objects_misplaced":7915,"num_objects_unfound":0,"num_objects_dirty":71960,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":250207,"num_write_kb":10306363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25261,"num_bytes_recovered":1657096320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160856,"ondisk_log_size":160856,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":37,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":10861996,"kb_used_data":9772272,"kb_used_omap":219,"kb_used_meta":1089316,"kb_avail":1114079828,"statfs":{"total":1151940427776,"available":1140817743872,"internally_reserved":0,"allocated":10006806528,"data_stored":9996558288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224949,"internal_metadata":1115459915},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-80216064,"num_objects":-1224,"num_object_clones":0,"num_object_copies":-2448,"num_objects_missing_on_primary":-46,"num_objects_missing":-46,"num_objects_degraded":0,"num_objects_misplaced":-94,"num_objects_unfound":0,"num_objects_dirty":-1224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1224,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":3145728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.095017"},"pg_stats":[{"pgid":"2.f","version":"279'15651","reported_seq":16445,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-20T17:16:13.580221+0000","last_change":"2024-04-20T17:14:39.601610+0000","last_active":"2024-04-20T17:16:13.580221+0000","last_peered":"2024-04-20T17:16:13.580221+0000","last_clean":"2024-04-20T17:16:13.580221+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:16:13.580221+0000","last_undegraded":"2024-04-20T17:16:13.580221+0000","last_fullsized":"2024-04-20T17:16:13.580221+0000","mapping_epoch":115,"log_start":"131'5600","ondisk_log_start":"131'5600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'14864","last_scrub_stamp":"2024-04-20T17:14:39.601506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:39.601506+0000","objects_scrubbed":5320,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296943616,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15651,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"278'15556","reported_seq":27417,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-20T17:16:10.558355+0000","last_change":"2024-04-20T17:15:31.328405+0000","last_active":"2024-04-20T17:16:10.558355+0000","last_peered":"2024-04-20T17:16:10.558355+0000","last_clean":"2024-04-20T17:16:10.558355+0000","last_became_active":"2024-04-20T17:10:28.865670+0000","last_became_peered":"2024-04-20T17:10:28.865670+0000","last_unstale":"2024-04-20T17:16:10.558355+0000","last_undegraded":"2024-04-20T17:16:10.558355+0000","last_fullsized":"2024-04-20T17:16:10.558355+0000","mapping_epoch":162,"log_start":"131'5500","ondisk_log_start":"131'5500","created":115,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15200","last_scrub_stamp":"2024-04-20T17:15:31.328301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:31.328301+0000","objects_scrubbed":4857,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:38.848343+0000","scrub_duration":424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294912000,"num_objects":4500,"num_object_clones":0,"num_object_copies":9000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15556,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5619,"num_bytes_recovered":368246784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"278'15885","reported_seq":16570,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-20T17:16:10.472127+0000","last_change":"2024-04-20T17:15:57.136522+0000","last_active":"2024-04-20T17:16:10.472127+0000","last_peered":"2024-04-20T17:16:10.472127+0000","last_clean":"2024-04-20T17:16:10.472127+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:16:10.472127+0000","last_undegraded":"2024-04-20T17:16:10.472127+0000","last_fullsized":"2024-04-20T17:16:10.472127+0000","mapping_epoch":115,"log_start":"135'5800","ondisk_log_start":"135'5800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'15780","last_scrub_stamp":"2024-04-20T17:15:57.136412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:57.136412+0000","objects_scrubbed":4702,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:10.130229+0000","scrub_duration":368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301268992,"num_objects":4597,"num_object_clones":0,"num_object_copies":9194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15885,"num_write_kb":655424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"278'22281","reported_seq":43716,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-20T17:16:10.525982+0000","last_change":"2024-04-20T17:15:30.287285+0000","last_active":"2024-04-20T17:16:10.525982+0000","last_peered":"2024-04-20T17:16:10.525982+0000","last_clean":"2024-04-20T17:16:10.525982+0000","last_became_active":"2024-04-20T17:12:08.085936+0000","last_became_peered":"2024-04-20T17:12:08.085936+0000","last_unstale":"2024-04-20T17:16:10.525982+0000","last_undegraded":"2024-04-20T17:16:10.525982+0000","last_fullsized":"2024-04-20T17:16:10.525982+0000","mapping_epoch":196,"log_start":"163'12268","ondisk_log_start":"163'12268","created":115,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'21922","last_scrub_stamp":"2024-04-20T17:15:30.287129+0000","last_deep_scrub":"213'20373","last_deep_scrub_stamp":"2024-04-20T17:12:48.443808+0000","last_clean_scrub_stamp":"2024-04-20T17:15:30.287129+0000","objects_scrubbed":4965,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:34.168023+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301793280,"num_objects":4605,"num_object_clones":0,"num_object_copies":9210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15629,"num_write_kb":647488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":450166784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"278'15338","reported_seq":32098,"reported_epoch":278,"state":"active+recovering","last_fresh":"2024-04-20T17:16:11.378434+0000","last_change":"2024-04-20T17:12:26.166109+0000","last_active":"2024-04-20T17:16:11.378434+0000","last_peered":"2024-04-20T17:16:11.378434+0000","last_clean":"2024-04-20T17:09:35.882603+0000","last_became_active":"2024-04-20T17:10:54.470253+0000","last_became_peered":"2024-04-20T17:10:54.470253+0000","last_unstale":"2024-04-20T17:16:11.378434+0000","last_undegraded":"2024-04-20T17:16:11.378434+0000","last_fullsized":"2024-04-20T17:16:11.378434+0000","mapping_epoch":171,"log_start":"129'5300","ondisk_log_start":"129'5300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287834112,"num_objects":4392,"num_object_clones":0,"num_object_copies":8784,"num_objects_missing_on_primary":3952,"num_objects_missing":3952,"num_objects_degraded":0,"num_objects_misplaced":7915,"num_objects_unfound":0,"num_objects_dirty":4392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15338,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5469,"num_bytes_recovered":358416384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"0,2,5","objects":11},{"shards":"0,10","objects":429},{"shards":"2,5","objects":3952}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":742,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-20T17:16:08.959663+0000","last_change":"2024-04-20T17:14:40.440515+0000","last_active":"2024-04-20T17:16:08.959663+0000","last_peered":"2024-04-20T17:16:08.959663+0000","last_clean":"2024-04-20T17:16:08.959663+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:16:08.959663+0000","last_undegraded":"2024-04-20T17:16:08.959663+0000","last_fullsized":"2024-04-20T17:16:08.959663+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:14:40.440403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:14:40.440403+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:13.341773+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"278'15709","reported_seq":16426,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-20T17:16:12.211869+0000","last_change":"2024-04-20T17:15:31.396169+0000","last_active":"2024-04-20T17:16:12.211869+0000","last_peered":"2024-04-20T17:16:12.211869+0000","last_clean":"2024-04-20T17:16:12.211869+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:16:12.211869+0000","last_undegraded":"2024-04-20T17:16:12.211869+0000","last_fullsized":"2024-04-20T17:16:12.211869+0000","mapping_epoch":115,"log_start":"135'5700","ondisk_log_start":"135'5700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15333","last_scrub_stamp":"2024-04-20T17:15:31.396069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:31.396069+0000","objects_scrubbed":4919,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:57.811864+0000","scrub_duration":440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297730048,"num_objects":4543,"num_object_clones":0,"num_object_copies":9086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15709,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"279'15806","reported_seq":16524,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-20T17:16:13.542807+0000","last_change":"2024-04-20T17:15:35.632022+0000","last_active":"2024-04-20T17:16:13.542807+0000","last_peered":"2024-04-20T17:16:13.542807+0000","last_clean":"2024-04-20T17:16:13.542807+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:16:13.542807+0000","last_undegraded":"2024-04-20T17:16:13.542807+0000","last_fullsized":"2024-04-20T17:16:13.542807+0000","mapping_epoch":115,"log_start":"135'5800","ondisk_log_start":"135'5800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15459","last_scrub_stamp":"2024-04-20T17:15:35.631842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:35.631842+0000","objects_scrubbed":4867,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:52.716132+0000","scrub_duration":426,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296091648,"num_objects":4518,"num_object_clones":0,"num_object_copies":9036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15806,"num_write_kb":650368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"278'19758","reported_seq":20571,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-20T17:16:11.450285+0000","last_change":"2024-04-20T17:14:45.127365+0000","last_active":"2024-04-20T17:16:11.450285+0000","last_peered":"2024-04-20T17:16:11.450285+0000","last_clean":"2024-04-20T17:16:11.450285+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:16:11.450285+0000","last_undegraded":"2024-04-20T17:16:11.450285+0000","last_fullsized":"2024-04-20T17:16:11.450285+0000","mapping_epoch":115,"log_start":"151'9702","ondisk_log_start":"151'9702","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'19011","last_scrub_stamp":"2024-04-20T17:14:45.127219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:45.127219+0000","objects_scrubbed":5143,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:14.228306+0000","scrub_duration":538,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288030720,"num_objects":4395,"num_object_clones":0,"num_object_copies":8790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15647,"num_write_kb":641344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"278'15774","reported_seq":16547,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-20T17:16:12.238596+0000","last_change":"2024-04-20T17:15:41.950854+0000","last_active":"2024-04-20T17:16:12.238596+0000","last_peered":"2024-04-20T17:16:12.238596+0000","last_clean":"2024-04-20T17:16:12.238596+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:16:12.238596+0000","last_undegraded":"2024-04-20T17:16:12.238596+0000","last_fullsized":"2024-04-20T17:16:12.238596+0000","mapping_epoch":115,"log_start":"131'5700","ondisk_log_start":"131'5700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'15491","last_scrub_stamp":"2024-04-20T17:15:41.950723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:41.950723+0000","objects_scrubbed":4733,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:58.122837+0000","scrub_duration":448,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291373056,"num_objects":4446,"num_object_clones":0,"num_object_copies":8892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15774,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"278'15576","reported_seq":16276,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-20T17:16:12.693615+0000","last_change":"2024-04-20T17:15:59.551771+0000","last_active":"2024-04-20T17:16:12.693615+0000","last_peered":"2024-04-20T17:16:12.693615+0000","last_clean":"2024-04-20T17:16:12.693615+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:16:12.693615+0000","last_undegraded":"2024-04-20T17:16:12.693615+0000","last_fullsized":"2024-04-20T17:16:12.693615+0000","mapping_epoch":115,"log_start":"131'5500","ondisk_log_start":"131'5500","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'15438","last_scrub_stamp":"2024-04-20T17:15:59.551671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:59.551671+0000","objects_scrubbed":4614,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:10.078708+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293339136,"num_objects":4476,"num_object_clones":0,"num_object_copies":8952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15576,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"278'15737","reported_seq":30991,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-20T17:16:12.710696+0000","last_change":"2024-04-20T17:14:43.945880+0000","last_active":"2024-04-20T17:16:12.710696+0000","last_peered":"2024-04-20T17:16:12.710696+0000","last_clean":"2024-04-20T17:16:12.710696+0000","last_became_active":"2024-04-20T17:10:55.931751+0000","last_became_peered":"2024-04-20T17:10:55.931751+0000","last_unstale":"2024-04-20T17:16:12.710696+0000","last_undegraded":"2024-04-20T17:16:12.710696+0000","last_fullsized":"2024-04-20T17:16:12.710696+0000","mapping_epoch":173,"log_start":"132'5700","ondisk_log_start":"132'5700","created":115,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'14953","last_scrub_stamp":"2024-04-20T17:14:43.945755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:43.945755+0000","objects_scrubbed":5309,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:13.341753+0000","scrub_duration":454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296550400,"num_objects":4525,"num_object_clones":0,"num_object_copies":9050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15737,"num_write_kb":648384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7296,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"278'15810","reported_seq":16523,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-20T17:16:12.275735+0000","last_change":"2024-04-20T17:15:30.785298+0000","last_active":"2024-04-20T17:16:12.275735+0000","last_peered":"2024-04-20T17:16:12.275735+0000","last_clean":"2024-04-20T17:16:12.275735+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:16:12.275735+0000","last_undegraded":"2024-04-20T17:16:12.275735+0000","last_fullsized":"2024-04-20T17:16:12.275735+0000","mapping_epoch":115,"log_start":"135'5800","ondisk_log_start":"135'5800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'15415","last_scrub_stamp":"2024-04-20T17:15:30.785135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:30.785135+0000","objects_scrubbed":4901,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:40.155273+0000","scrub_duration":453,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295305216,"num_objects":4506,"num_object_clones":0,"num_object_copies":9012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15810,"num_write_kb":650112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"279'15378","reported_seq":16092,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-20T17:16:13.537975+0000","last_change":"2024-04-20T17:15:34.656600+0000","last_active":"2024-04-20T17:16:13.537975+0000","last_peered":"2024-04-20T17:16:13.537975+0000","last_clean":"2024-04-20T17:16:13.537975+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:16:13.537975+0000","last_undegraded":"2024-04-20T17:16:13.537975+0000","last_fullsized":"2024-04-20T17:16:13.537975+0000","mapping_epoch":115,"log_start":"130'5300","ondisk_log_start":"130'5300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15023","last_scrub_stamp":"2024-04-20T17:15:34.656503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:34.656503+0000","objects_scrubbed":4842,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:51.197304+0000","scrub_duration":430,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293994496,"num_objects":4486,"num_object_clones":0,"num_object_copies":8972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15378,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"278'15631","reported_seq":16343,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-20T17:16:12.271314+0000","last_change":"2024-04-20T17:15:31.794340+0000","last_active":"2024-04-20T17:16:12.271314+0000","last_peered":"2024-04-20T17:16:12.271314+0000","last_clean":"2024-04-20T17:16:12.271314+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:16:12.271314+0000","last_undegraded":"2024-04-20T17:16:12.271314+0000","last_fullsized":"2024-04-20T17:16:12.271314+0000","mapping_epoch":115,"log_start":"131'5600","ondisk_log_start":"131'5600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15271","last_scrub_stamp":"2024-04-20T17:15:31.794156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:31.794156+0000","objects_scrubbed":4831,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:58.805155+0000","scrub_duration":443,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293011456,"num_objects":4471,"num_object_clones":0,"num_object_copies":8942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15631,"num_write_kb":643264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"278'15406","reported_seq":16118,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-20T17:16:10.540802+0000","last_change":"2024-04-20T17:15:33.298229+0000","last_active":"2024-04-20T17:16:10.540802+0000","last_peered":"2024-04-20T17:16:10.540802+0000","last_clean":"2024-04-20T17:16:10.540802+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:16:10.540802+0000","last_undegraded":"2024-04-20T17:16:10.540802+0000","last_fullsized":"2024-04-20T17:16:10.540802+0000","mapping_epoch":115,"log_start":"130'5400","ondisk_log_start":"130'5400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15067","last_scrub_stamp":"2024-04-20T17:15:33.298103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:33.298103+0000","objects_scrubbed":4809,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:56.012242+0000","scrub_duration":458,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292487192,"num_objects":4464,"num_object_clones":0,"num_object_copies":8928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4464,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15406,"num_write_kb":635777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"278'15493","reported_seq":16073,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-04-20T17:16:10.797861+0000","last_change":"2024-04-20T17:14:59.605205+0000","last_active":"2024-04-20T17:16:10.797861+0000","last_peered":"2024-04-20T17:16:10.797861+0000","last_clean":"2024-04-20T17:16:10.797861+0000","last_became_active":"2024-04-20T17:12:23.515375+0000","last_became_peered":"2024-04-20T17:12:23.515375+0000","last_unstale":"2024-04-20T17:16:10.797861+0000","last_undegraded":"2024-04-20T17:16:10.797861+0000","last_fullsized":"2024-04-20T17:16:10.797861+0000","mapping_epoch":202,"log_start":"130'5400","ondisk_log_start":"130'5400","created":115,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'14856","last_scrub_stamp":"2024-04-20T17:14:59.605082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:59.605082+0000","objects_scrubbed":5143,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:21.549388+0000","scrub_duration":486,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295108608,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15493,"num_write_kb":639872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":4715773976,"num_objects":71958,"num_object_clones":0,"num_object_copies":143916,"num_objects_missing_on_primary":3952,"num_objects_missing":3952,"num_objects_degraded":0,"num_objects_misplaced":7915,"num_objects_unfound":0,"num_objects_dirty":71958,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":250026,"num_write_kb":10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25253,"num_bytes_recovered":1654980608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9972817920,"data_stored":9972809776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},"log_size":160719,"ondisk_log_size":160719,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991358,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":928256,"kb_used_data":865080,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92816896,"statfs":{"total":95995035648,"available":95044501504,"internally_reserved":0,"allocated":885841920,"data_stored":884986590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":64616900},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777252996,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1037256,"kb_used_data":893628,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92707896,"statfs":{"total":95995035648,"available":94932885504,"internally_reserved":0,"allocated":915075072,"data_stored":914217366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":147061184},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612745,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1305600,"kb_used_data":1161972,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92439552,"statfs":{"total":95995035648,"available":94658101248,"internally_reserved":0,"allocated":1189859328,"data_stored":1189003498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":147061184},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762808975,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1252616,"kb_used_data":1149244,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92492536,"statfs":{"total":95995035648,"available":94712356864,"internally_reserved":0,"allocated":1176825856,"data_stored":1175968170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168724,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":940992,"kb_used_data":869560,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92804160,"statfs":{"total":95995035648,"available":95031459840,"internally_reserved":0,"allocated":890429440,"data_stored":889574130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430360,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":670008,"kb_used_data":580528,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93075144,"statfs":{"total":95995035648,"available":95308947456,"internally_reserved":0,"allocated":594460672,"data_stored":593608914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91617734},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575026,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":658168,"kb_used_data":577008,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93086984,"statfs":{"total":95995035648,"available":95321071616,"internally_reserved":0,"allocated":590856192,"data_stored":590004434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738551,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1039992,"kb_used_data":896368,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92705160,"statfs":{"total":95995035648,"available":94930083840,"internally_reserved":0,"allocated":917880832,"data_stored":917029074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393839,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":766140,"kb_used_data":721904,"kb_used_omap":202,"kb_used_meta":44021,"kb_avail":92979012,"statfs":{"total":95995035648,"available":95210508288,"internally_reserved":0,"allocated":739229696,"data_stored":738377938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":45077975},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182567,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":647420,"kb_used_data":580144,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93097732,"statfs":{"total":95995035648,"available":95332077568,"internally_reserved":0,"allocated":594067456,"data_stored":593215698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986338,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":327168,"kb_used_data":287668,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93417984,"statfs":{"total":95995035648,"available":95660015616,"internally_reserved":0,"allocated":294572032,"data_stored":293716202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790108,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1288380,"kb_used_data":1189168,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92456772,"statfs":{"total":95995035648,"available":94675734528,"internally_reserved":0,"allocated":1217708032,"data_stored":1216856274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":914096128,"data_stored":914096128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":588447744,"data_stored":588447744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":735313920,"data_stored":735313920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":591069184,"data_stored":591069184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":292687872,"data_stored":292683800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1214578688,"data_stored":1214578688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":592248832,"data_stored":592248832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":887619584,"data_stored":887619584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1174274048,"data_stored":1174274048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1188106240,"data_stored":1188102168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":911540224,"data_stored":911540224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":882835456,"data_stored":882835456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:16:18.326 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:16:19.181 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:16:19.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:19.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:16:19.285 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:16:19.311 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:19.321 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:16:19.322 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:16:19.322 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-04-20T17:16:19.322 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:16:19.331 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:16:19.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:19.334 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:16:19.342 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:19 smithi106 ceph-mon[44423]: pgmap v776: 17 pgs: 1 active+recovering, 16 active+clean; 4.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 171 op/s; 7816/142710 objects misplaced (5.477%); 776 KiB/s, 12 objects/s recovering 2024-04-20T17:16:19.434 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:16:19.436 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:16:19.452 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:16:19.459 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:19 smithi171 ceph-mon[44189]: pgmap v776: 17 pgs: 1 active+recovering, 16 active+clean; 4.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 171 op/s; 7816/142710 objects misplaced (5.477%); 776 KiB/s, 12 objects/s recovering 2024-04-20T17:16:19.461 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:19.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:19 smithi005 ceph-mon[39562]: pgmap v776: 17 pgs: 1 active+recovering, 16 active+clean; 4.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 171 op/s; 7816/142710 objects misplaced (5.477%); 776 KiB/s, 12 objects/s recovering 2024-04-20T17:16:19.471 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:16:19.479 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:16:19.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:19.482 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:16:19.492 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:16:19.522 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:16:19.542 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:16:19.562 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:16:19.583 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:16:19.584 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:16:19.604 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:16:19.614 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:19.624 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:16:19.645 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:16:19.668 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:16:19.669 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:16:19.669 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:16:19.669 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:16:19.669 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:16:19.669 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:16:19.669 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:16:19.669 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:16:19.669 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:16:19.670 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:16:19.670 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:16:19.670 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:16:19.670 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:16:19.737 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:16:19.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:19.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:16:19.843 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:16:19.869 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:19.888 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:16:19.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:19.891 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:16:19.992 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:16:20.025 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:20.061 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-20T17:16:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:20 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1961824674' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:16:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:20.338 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:16:20.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:20 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1961824674' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:16:20.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:20.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:20.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:20.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:20.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:20.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:20.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:20 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1961824674' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:16:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:20.577 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:16:20.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:20.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:16:20.683 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:16:20.715 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:21.405 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:21.406 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:21.406 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:21.406 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:21.406 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:21.406 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:21 smithi005 ceph-mon[39562]: pgmap v777: 17 pgs: 1 active+recovering, 16 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 149 op/s; 7816/142426 objects misplaced (5.488%); 392 KiB/s, 6 objects/s recovering 2024-04-20T17:16:21.406 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:21 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:16:21.406 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:21 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:16:21.431 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:16:21.467 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:16:21.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:21.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:16:21.501 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:21.501 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:21.501 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:21.502 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:21.502 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:21.502 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:21 smithi106 ceph-mon[44423]: pgmap v777: 17 pgs: 1 active+recovering, 16 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 149 op/s; 7816/142426 objects misplaced (5.488%); 392 KiB/s, 6 objects/s recovering 2024-04-20T17:16:21.502 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:21 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:16:21.502 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:21 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:16:21.572 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:16:21.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:21.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:21.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:21.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:21.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:21.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:21 smithi171 ceph-mon[44189]: pgmap v777: 17 pgs: 1 active+recovering, 16 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 149 op/s; 7816/142426 objects misplaced (5.488%); 392 KiB/s, 6 objects/s recovering 2024-04-20T17:16:21.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:21 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:16:21.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:21 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:16:21.599 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:21.847 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:16:21.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:21.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:16:21.951 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:16:21.980 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:22.002 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:16:22.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:22.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:16:22.106 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:16:22.123 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:16:22.128 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:16:22.129 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:16:22.133 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:22.153 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:16:22.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:22.158 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:16:22.259 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:16:22.286 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:22.306 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:16:22.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:22.310 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:16:22.411 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:16:22.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:22 smithi005 ceph-mon[39562]: osdmap e281: 12 total, 12 up, 12 in 2024-04-20T17:16:22.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:22 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4274136167' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:16:22.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:22 smithi005 ceph-mon[39562]: from='client.25376 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:16:22.442 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:22.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:22 smithi106 ceph-mon[44423]: osdmap e281: 12 total, 12 up, 12 in 2024-04-20T17:16:22.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:22 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4274136167' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:16:22.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:22 smithi106 ceph-mon[44423]: from='client.25376 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:16:22.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:22 smithi171 ceph-mon[44189]: osdmap e281: 12 total, 12 up, 12 in 2024-04-20T17:16:22.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:22 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4274136167' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:16:22.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:22 smithi171 ceph-mon[44189]: from='client.25376 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:16:22.594 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:16:22.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:22.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:16:22.698 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:16:22.724 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:22.743 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:16:22.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:22.745 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:16:22.844 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-20T17:16:22.846 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:16:22.873 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:22.892 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:16:22.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:22.895 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:16:22.996 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:16:23.022 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:23.040 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:16:23.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:23.043 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:16:23.114 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:16:23.144 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:16:23.179 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:23.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:23 smithi171 ceph-mon[44189]: pgmap v779: 17 pgs: 1 active+recovering, 16 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 129 op/s; 7714/141850 objects misplaced (5.438%); 792 KiB/s, 12 objects/s recovering 2024-04-20T17:16:23.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:23 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:16:23.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:23 smithi171 ceph-mon[44189]: from='client.25376 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:16:23.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:23 smithi171 ceph-mon[44189]: osdmap e282: 12 total, 12 up, 12 in 2024-04-20T17:16:23.381 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:23 smithi106 ceph-mon[44423]: pgmap v779: 17 pgs: 1 active+recovering, 16 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 129 op/s; 7714/141850 objects misplaced (5.438%); 792 KiB/s, 12 objects/s recovering 2024-04-20T17:16:23.382 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:23 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:16:23.382 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:23 smithi106 ceph-mon[44423]: from='client.25376 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:16:23.382 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:23 smithi106 ceph-mon[44423]: osdmap e282: 12 total, 12 up, 12 in 2024-04-20T17:16:23.441 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:16:23.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:23 smithi005 ceph-mon[39562]: pgmap v779: 17 pgs: 1 active+recovering, 16 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 129 op/s; 7714/141850 objects misplaced (5.438%); 792 KiB/s, 12 objects/s recovering 2024-04-20T17:16:23.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:23 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:16:23.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:23 smithi005 ceph-mon[39562]: from='client.25376 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:16:23.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:23 smithi005 ceph-mon[39562]: osdmap e282: 12 total, 12 up, 12 in 2024-04-20T17:16:24.033 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:16:24.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:24.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:16:24.138 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:16:24.173 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:24.311 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:24 smithi005 ceph-mon[39562]: 2.e scrub starts 2024-04-20T17:16:24.322 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:16:24.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:24.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:16:24.375 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:24 smithi106 ceph-mon[44423]: 2.e scrub starts 2024-04-20T17:16:24.430 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:16:24.461 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:24 smithi171 ceph-mon[44189]: 2.e scrub starts 2024-04-20T17:16:24.659 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:16:24.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:24.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:16:24.741 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:16:24.765 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:16:24.800 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:24.880 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:16:24.884 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:16:24.985 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:16:24.985 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:16:25.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:25 smithi106 ceph-mon[44423]: 2.e scrub ok 2024-04-20T17:16:25.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:25 smithi106 ceph-mon[44423]: pgmap v781: 17 pgs: 1 active+recovering, 16 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 153 op/s; 7714/141008 objects misplaced (5.471%); 792 KiB/s, 12 objects/s recovering 2024-04-20T17:16:25.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:25 smithi106 ceph-mon[44423]: 2.f scrub starts 2024-04-20T17:16:25.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:25 smithi106 ceph-mon[44423]: from='client.15447 v1:172.21.15.5:0/1356850361' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:16:25.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:25 smithi005 ceph-mon[39562]: 2.e scrub ok 2024-04-20T17:16:25.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:25 smithi005 ceph-mon[39562]: pgmap v781: 17 pgs: 1 active+recovering, 16 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 153 op/s; 7714/141008 objects misplaced (5.471%); 792 KiB/s, 12 objects/s recovering 2024-04-20T17:16:25.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:25 smithi005 ceph-mon[39562]: 2.f scrub starts 2024-04-20T17:16:25.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:25 smithi005 ceph-mon[39562]: from='client.15447 v1:172.21.15.5:0/1356850361' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:16:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:25 smithi171 ceph-mon[44189]: 2.e scrub ok 2024-04-20T17:16:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:25 smithi171 ceph-mon[44189]: pgmap v781: 17 pgs: 1 active+recovering, 16 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 153 op/s; 7714/141008 objects misplaced (5.471%); 792 KiB/s, 12 objects/s recovering 2024-04-20T17:16:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:25 smithi171 ceph-mon[44189]: 2.f scrub starts 2024-04-20T17:16:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:25 smithi171 ceph-mon[44189]: from='client.15447 v1:172.21.15.5:0/1356850361' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:16:25.698 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-20T17:16:25.809 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":781,"stamp":"2024-04-20T17:16:24.046913+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4620943928,"num_objects":70504,"num_object_clones":0,"num_object_copies":141008,"num_objects_missing_on_primary":3850,"num_objects_missing":3850,"num_objects_degraded":0,"num_objects_misplaced":7714,"num_objects_unfound":0,"num_objects_dirty":70504,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":251663,"num_write_kb":10306363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25360,"num_bytes_recovered":1663584384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161112,"ondisk_log_size":161112,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":10713292,"kb_used_data":9601432,"kb_used_omap":219,"kb_used_meta":1111524,"kb_avail":1114228532,"statfs":{"total":1151940427776,"available":1140970016768,"internally_reserved":0,"allocated":9831866368,"data_stored":9821439588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224949,"internal_metadata":1138200907},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,4,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-80740352,"num_objects":-1232,"num_object_clones":0,"num_object_copies":-2464,"num_objects_missing_on_primary":-102,"num_objects_missing":-102,"num_objects_degraded":0,"num_objects_misplaced":-201,"num_objects_unfound":0,"num_objects_dirty":-1232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1232,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":6488064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001415"},"pg_stats":[{"pgid":"2.f","version":"282'15755","reported_seq":16555,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-04-20T17:16:23.334253+0000","last_change":"2024-04-20T17:14:39.601610+0000","last_active":"2024-04-20T17:16:23.334253+0000","last_peered":"2024-04-20T17:16:23.334253+0000","last_clean":"2024-04-20T17:16:23.334253+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:16:23.334253+0000","last_undegraded":"2024-04-20T17:16:23.334253+0000","last_fullsized":"2024-04-20T17:16:23.334253+0000","mapping_epoch":115,"log_start":"135'5700","ondisk_log_start":"135'5700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'14864","last_scrub_stamp":"2024-04-20T17:14:39.601506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:39.601506+0000","objects_scrubbed":5320,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:24.420300+0000","scrub_duration":467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290127872,"num_objects":4427,"num_object_clones":0,"num_object_copies":8854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15755,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"280'15645","reported_seq":27510,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-04-20T17:16:20.610794+0000","last_change":"2024-04-20T17:15:31.328405+0000","last_active":"2024-04-20T17:16:20.610794+0000","last_peered":"2024-04-20T17:16:20.610794+0000","last_clean":"2024-04-20T17:16:20.610794+0000","last_became_active":"2024-04-20T17:10:28.865670+0000","last_became_peered":"2024-04-20T17:10:28.865670+0000","last_unstale":"2024-04-20T17:16:20.610794+0000","last_undegraded":"2024-04-20T17:16:20.610794+0000","last_fullsized":"2024-04-20T17:16:20.610794+0000","mapping_epoch":162,"log_start":"131'5600","ondisk_log_start":"131'5600","created":115,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15200","last_scrub_stamp":"2024-04-20T17:15:31.328301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:31.328301+0000","objects_scrubbed":4857,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:38.848343+0000","scrub_duration":424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289079296,"num_objects":4411,"num_object_clones":0,"num_object_copies":8822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15645,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5619,"num_bytes_recovered":368246784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"280'15983","reported_seq":16672,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-04-20T17:16:20.655619+0000","last_change":"2024-04-20T17:15:57.136522+0000","last_active":"2024-04-20T17:16:20.655619+0000","last_peered":"2024-04-20T17:16:20.655619+0000","last_clean":"2024-04-20T17:16:20.655619+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:16:20.655619+0000","last_undegraded":"2024-04-20T17:16:20.655619+0000","last_fullsized":"2024-04-20T17:16:20.655619+0000","mapping_epoch":115,"log_start":"135'5900","ondisk_log_start":"135'5900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'15780","last_scrub_stamp":"2024-04-20T17:15:57.136412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:57.136412+0000","objects_scrubbed":4702,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:10.130229+0000","scrub_duration":368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294846464,"num_objects":4499,"num_object_clones":0,"num_object_copies":8998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15983,"num_write_kb":655424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"280'22350","reported_seq":43789,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-04-20T17:16:20.530524+0000","last_change":"2024-04-20T17:15:30.287285+0000","last_active":"2024-04-20T17:16:20.530524+0000","last_peered":"2024-04-20T17:16:20.530524+0000","last_clean":"2024-04-20T17:16:20.530524+0000","last_became_active":"2024-04-20T17:12:08.085936+0000","last_became_peered":"2024-04-20T17:12:08.085936+0000","last_unstale":"2024-04-20T17:16:20.530524+0000","last_undegraded":"2024-04-20T17:16:20.530524+0000","last_fullsized":"2024-04-20T17:16:20.530524+0000","mapping_epoch":196,"log_start":"163'12268","ondisk_log_start":"163'12268","created":115,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'21922","last_scrub_stamp":"2024-04-20T17:15:30.287129+0000","last_deep_scrub":"213'20373","last_deep_scrub_stamp":"2024-04-20T17:12:48.443808+0000","last_clean_scrub_stamp":"2024-04-20T17:15:30.287129+0000","objects_scrubbed":4965,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:34.168023+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297271296,"num_objects":4536,"num_object_clones":0,"num_object_copies":9072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15698,"num_write_kb":647488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":450166784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"281'15436","reported_seq":32502,"reported_epoch":281,"state":"active+recovering","last_fresh":"2024-04-20T17:16:21.551948+0000","last_change":"2024-04-20T17:12:26.166109+0000","last_active":"2024-04-20T17:16:21.551948+0000","last_peered":"2024-04-20T17:16:21.551948+0000","last_clean":"2024-04-20T17:09:35.882603+0000","last_became_active":"2024-04-20T17:10:54.470253+0000","last_became_peered":"2024-04-20T17:10:54.470253+0000","last_unstale":"2024-04-20T17:16:21.551948+0000","last_undegraded":"2024-04-20T17:16:21.551948+0000","last_fullsized":"2024-04-20T17:16:21.551948+0000","mapping_epoch":171,"log_start":"130'5400","ondisk_log_start":"130'5400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281411584,"num_objects":4294,"num_object_clones":0,"num_object_copies":8588,"num_objects_missing_on_primary":3850,"num_objects_missing":3850,"num_objects_degraded":0,"num_objects_misplaced":7714,"num_objects_unfound":0,"num_objects_dirty":4294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15436,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5568,"num_bytes_recovered":364904448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"0,2,5","objects":14},{"shards":"0,10","objects":430},{"shards":"2,5","objects":3850}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":751,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-04-20T17:16:22.139859+0000","last_change":"2024-04-20T17:14:40.440515+0000","last_active":"2024-04-20T17:16:22.139859+0000","last_peered":"2024-04-20T17:16:22.139859+0000","last_clean":"2024-04-20T17:16:22.139859+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:16:22.139859+0000","last_undegraded":"2024-04-20T17:16:22.139859+0000","last_fullsized":"2024-04-20T17:16:22.139859+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:14:40.440403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:14:40.440403+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:27.104523+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"282'15790","reported_seq":16515,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-04-20T17:16:22.461941+0000","last_change":"2024-04-20T17:15:31.396169+0000","last_active":"2024-04-20T17:16:22.461941+0000","last_peered":"2024-04-20T17:16:22.461941+0000","last_clean":"2024-04-20T17:16:22.461941+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:16:22.461941+0000","last_undegraded":"2024-04-20T17:16:22.461941+0000","last_fullsized":"2024-04-20T17:16:22.461941+0000","mapping_epoch":115,"log_start":"135'5700","ondisk_log_start":"135'5700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15333","last_scrub_stamp":"2024-04-20T17:15:31.396069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:31.396069+0000","objects_scrubbed":4919,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:57.811864+0000","scrub_duration":440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292421632,"num_objects":4462,"num_object_clones":0,"num_object_copies":8924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15790,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"282'15919","reported_seq":16643,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-04-20T17:16:23.340623+0000","last_change":"2024-04-20T17:15:35.632022+0000","last_active":"2024-04-20T17:16:23.340623+0000","last_peered":"2024-04-20T17:16:23.340623+0000","last_clean":"2024-04-20T17:16:23.340623+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:16:23.340623+0000","last_undegraded":"2024-04-20T17:16:23.340623+0000","last_fullsized":"2024-04-20T17:16:23.340623+0000","mapping_epoch":115,"log_start":"135'5900","ondisk_log_start":"135'5900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15459","last_scrub_stamp":"2024-04-20T17:15:35.631842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:35.631842+0000","objects_scrubbed":4867,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:52.716132+0000","scrub_duration":426,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288686080,"num_objects":4405,"num_object_clones":0,"num_object_copies":8810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":650368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"281'19847","reported_seq":20666,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-04-20T17:16:21.888249+0000","last_change":"2024-04-20T17:14:45.127365+0000","last_active":"2024-04-20T17:16:21.888249+0000","last_peered":"2024-04-20T17:16:21.888249+0000","last_clean":"2024-04-20T17:16:21.888249+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:16:21.888249+0000","last_undegraded":"2024-04-20T17:16:21.888249+0000","last_fullsized":"2024-04-20T17:16:21.888249+0000","mapping_epoch":115,"log_start":"152'9802","ondisk_log_start":"152'9802","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'19011","last_scrub_stamp":"2024-04-20T17:14:45.127219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:45.127219+0000","objects_scrubbed":5143,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:24.221415+0000","scrub_duration":538,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282198016,"num_objects":4306,"num_object_clones":0,"num_object_copies":8612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15736,"num_write_kb":641344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"282'15857","reported_seq":16638,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-04-20T17:16:22.308978+0000","last_change":"2024-04-20T17:15:41.950854+0000","last_active":"2024-04-20T17:16:22.308978+0000","last_peered":"2024-04-20T17:16:22.308978+0000","last_clean":"2024-04-20T17:16:22.308978+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:16:22.308978+0000","last_undegraded":"2024-04-20T17:16:22.308978+0000","last_fullsized":"2024-04-20T17:16:22.308978+0000","mapping_epoch":115,"log_start":"135'5800","ondisk_log_start":"135'5800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'15491","last_scrub_stamp":"2024-04-20T17:15:41.950723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:41.950723+0000","objects_scrubbed":4733,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:58.122837+0000","scrub_duration":448,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285933568,"num_objects":4363,"num_object_clones":0,"num_object_copies":8726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15857,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"282'15669","reported_seq":16377,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-04-20T17:16:22.549293+0000","last_change":"2024-04-20T17:15:59.551771+0000","last_active":"2024-04-20T17:16:22.549293+0000","last_peered":"2024-04-20T17:16:22.549293+0000","last_clean":"2024-04-20T17:16:22.549293+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:16:22.549293+0000","last_undegraded":"2024-04-20T17:16:22.549293+0000","last_fullsized":"2024-04-20T17:16:22.549293+0000","mapping_epoch":115,"log_start":"131'5600","ondisk_log_start":"131'5600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'15438","last_scrub_stamp":"2024-04-20T17:15:59.551671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:59.551671+0000","objects_scrubbed":4614,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:10.078708+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287244288,"num_objects":4383,"num_object_clones":0,"num_object_copies":8766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15669,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"282'15835","reported_seq":31097,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-04-20T17:16:22.505043+0000","last_change":"2024-04-20T17:14:43.945880+0000","last_active":"2024-04-20T17:16:22.505043+0000","last_peered":"2024-04-20T17:16:22.505043+0000","last_clean":"2024-04-20T17:16:22.505043+0000","last_became_active":"2024-04-20T17:10:55.931751+0000","last_became_peered":"2024-04-20T17:10:55.931751+0000","last_unstale":"2024-04-20T17:16:22.505043+0000","last_undegraded":"2024-04-20T17:16:22.505043+0000","last_fullsized":"2024-04-20T17:16:22.505043+0000","mapping_epoch":173,"log_start":"135'5800","ondisk_log_start":"135'5800","created":115,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'14953","last_scrub_stamp":"2024-04-20T17:14:43.945755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:43.945755+0000","objects_scrubbed":5309,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:27.104503+0000","scrub_duration":454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290127872,"num_objects":4427,"num_object_clones":0,"num_object_copies":8854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15835,"num_write_kb":648384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7296,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"282'15895","reported_seq":16616,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-04-20T17:16:22.527255+0000","last_change":"2024-04-20T17:15:30.785298+0000","last_active":"2024-04-20T17:16:22.527255+0000","last_peered":"2024-04-20T17:16:22.527255+0000","last_clean":"2024-04-20T17:16:22.527255+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:16:22.527255+0000","last_undegraded":"2024-04-20T17:16:22.527255+0000","last_fullsized":"2024-04-20T17:16:22.527255+0000","mapping_epoch":115,"log_start":"135'5800","ondisk_log_start":"135'5800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'15415","last_scrub_stamp":"2024-04-20T17:15:30.785135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:30.785135+0000","objects_scrubbed":4901,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:40.155273+0000","scrub_duration":453,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289734656,"num_objects":4421,"num_object_clones":0,"num_object_copies":8842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15895,"num_write_kb":650112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"282'15465","reported_seq":16185,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-04-20T17:16:23.350805+0000","last_change":"2024-04-20T17:15:34.656600+0000","last_active":"2024-04-20T17:16:23.350805+0000","last_peered":"2024-04-20T17:16:23.350805+0000","last_clean":"2024-04-20T17:16:23.350805+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:16:23.350805+0000","last_undegraded":"2024-04-20T17:16:23.350805+0000","last_fullsized":"2024-04-20T17:16:23.350805+0000","mapping_epoch":115,"log_start":"131'5400","ondisk_log_start":"131'5400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15023","last_scrub_stamp":"2024-04-20T17:15:34.656503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:34.656503+0000","objects_scrubbed":4842,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:51.197304+0000","scrub_duration":430,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288292864,"num_objects":4399,"num_object_clones":0,"num_object_copies":8798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15465,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"282'15739","reported_seq":16459,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-04-20T17:16:22.340217+0000","last_change":"2024-04-20T17:15:31.794340+0000","last_active":"2024-04-20T17:16:22.340217+0000","last_peered":"2024-04-20T17:16:22.340217+0000","last_clean":"2024-04-20T17:16:22.340217+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:16:22.340217+0000","last_undegraded":"2024-04-20T17:16:22.340217+0000","last_fullsized":"2024-04-20T17:16:22.340217+0000","mapping_epoch":115,"log_start":"134'5700","ondisk_log_start":"134'5700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15271","last_scrub_stamp":"2024-04-20T17:15:31.794156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:31.794156+0000","objects_scrubbed":4831,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:58.805155+0000","scrub_duration":443,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285933568,"num_objects":4363,"num_object_clones":0,"num_object_copies":8726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15739,"num_write_kb":643264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"280'15495","reported_seq":16211,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-04-20T17:16:20.439263+0000","last_change":"2024-04-20T17:15:33.298229+0000","last_active":"2024-04-20T17:16:20.439263+0000","last_peered":"2024-04-20T17:16:20.439263+0000","last_clean":"2024-04-20T17:16:20.439263+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:16:20.439263+0000","last_undegraded":"2024-04-20T17:16:20.439263+0000","last_fullsized":"2024-04-20T17:16:20.439263+0000","mapping_epoch":115,"log_start":"130'5400","ondisk_log_start":"130'5400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15067","last_scrub_stamp":"2024-04-20T17:15:33.298103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:33.298103+0000","objects_scrubbed":4809,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:56.012242+0000","scrub_duration":458,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286654488,"num_objects":4375,"num_object_clones":0,"num_object_copies":8750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4375,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15495,"num_write_kb":635777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"280'15565","reported_seq":16149,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-04-20T17:16:20.726878+0000","last_change":"2024-04-20T17:14:59.605205+0000","last_active":"2024-04-20T17:16:20.726878+0000","last_peered":"2024-04-20T17:16:20.726878+0000","last_clean":"2024-04-20T17:16:20.726878+0000","last_became_active":"2024-04-20T17:12:23.515375+0000","last_became_peered":"2024-04-20T17:12:23.515375+0000","last_unstale":"2024-04-20T17:16:20.726878+0000","last_undegraded":"2024-04-20T17:16:20.726878+0000","last_fullsized":"2024-04-20T17:16:20.726878+0000","mapping_epoch":202,"log_start":"131'5500","ondisk_log_start":"131'5500","created":115,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'14856","last_scrub_stamp":"2024-04-20T17:14:59.605082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:14:59.605082+0000","objects_scrubbed":5143,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:21.549388+0000","scrub_duration":486,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290390016,"num_objects":4431,"num_object_clones":0,"num_object_copies":8862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15565,"num_write_kb":639872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":4620353560,"num_objects":70502,"num_object_clones":0,"num_object_copies":141004,"num_objects_missing_on_primary":3850,"num_objects_missing":3850,"num_objects_degraded":0,"num_objects_misplaced":7714,"num_objects_unfound":0,"num_objects_dirty":70502,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":251482,"num_write_kb":10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25352,"num_bytes_recovered":1661468672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9795739648,"data_stored":9795731504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},"log_size":160975,"ondisk_log_size":160975,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991360,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":914984,"kb_used_data":847708,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92830168,"statfs":{"total":95995035648,"available":95058092032,"internally_reserved":0,"allocated":868052992,"data_stored":867179828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":68876740},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777252998,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1019884,"kb_used_data":876256,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92725268,"statfs":{"total":95995035648,"available":94950674432,"internally_reserved":0,"allocated":897286144,"data_stored":896410554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":147061184},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612747,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1283860,"kb_used_data":1140236,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92461292,"statfs":{"total":95995035648,"available":94680363008,"internally_reserved":0,"allocated":1167601664,"data_stored":1166733928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":147061184},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762808977,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1226284,"kb_used_data":1122912,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92518868,"statfs":{"total":95995035648,"available":94739320832,"internally_reserved":0,"allocated":1149861888,"data_stored":1148986318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168726,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":924712,"kb_used_data":853276,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92820440,"statfs":{"total":95995035648,"available":95048130560,"internally_reserved":0,"allocated":873754624,"data_stored":872881480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430362,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":678812,"kb_used_data":571284,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93066340,"statfs":{"total":95995035648,"available":95299932160,"internally_reserved":0,"allocated":584994816,"data_stored":584125224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575028,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":646880,"kb_used_data":565716,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93098272,"statfs":{"total":95995035648,"available":95332630528,"internally_reserved":0,"allocated":579293184,"data_stored":578423592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738553,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1021200,"kb_used_data":877576,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92723952,"statfs":{"total":95995035648,"available":94949326848,"internally_reserved":0,"allocated":898637824,"data_stored":897774160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,4,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393841,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":759888,"kb_used_data":715656,"kb_used_omap":202,"kb_used_meta":44021,"kb_avail":92985264,"statfs":{"total":95995035648,"available":95216910336,"internally_reserved":0,"allocated":732831744,"data_stored":731968080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":45077975},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":638160,"kb_used_data":570888,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93106992,"statfs":{"total":95995035648,"available":95341559808,"internally_reserved":0,"allocated":584589312,"data_stored":583725648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986340,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":321748,"kb_used_data":282252,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93423404,"statfs":{"total":95995035648,"available":95665565696,"internally_reserved":0,"allocated":289026048,"data_stored":288158312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790110,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1276880,"kb_used_data":1177672,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92468272,"statfs":{"total":95995035648,"available":94687510528,"internally_reserved":0,"allocated":1205936128,"data_stored":1205072464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":895025152,"data_stored":895025152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":576323584,"data_stored":576323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":730529792,"data_stored":730529792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":581369856,"data_stored":581369856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":286724096,"data_stored":286720024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1202323456,"data_stored":1202323456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":581566464,"data_stored":581566464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":869793792,"data_stored":869793792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1147994112,"data_stored":1147994112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1165758464,"data_stored":1165754392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":893845504,"data_stored":893845504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":864485376,"data_stored":864485376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:16:25.811 INFO:tasks.cephadm.ceph_manager.ceph:PG 2.3 is not active+clean 2024-04-20T17:16:25.811 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '2.3', 'version': "281'15436", 'reported_seq': 32502, 'reported_epoch': 281, 'state': 'active+recovering', 'last_fresh': '2024-04-20T17:16:21.551948+0000', 'last_change': '2024-04-20T17:12:26.166109+0000', 'last_active': '2024-04-20T17:16:21.551948+0000', 'last_peered': '2024-04-20T17:16:21.551948+0000', 'last_clean': '2024-04-20T17:09:35.882603+0000', 'last_became_active': '2024-04-20T17:10:54.470253+0000', 'last_became_peered': '2024-04-20T17:10:54.470253+0000', 'last_unstale': '2024-04-20T17:16:21.551948+0000', 'last_undegraded': '2024-04-20T17:16:21.551948+0000', 'last_fullsized': '2024-04-20T17:16:21.551948+0000', 'mapping_epoch': 171, 'log_start': "130'5400", 'ondisk_log_start': "130'5400", 'created': 115, 'last_epoch_clean': 116, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-20T17:08:04.990639+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T17:08:04.990639+0000', 'last_clean_scrub_stamp': '2024-04-20T17:08:04.990639+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 281411584, 'num_objects': 4294, 'num_object_clones': 0, 'num_object_copies': 8588, 'num_objects_missing_on_primary': 3850, 'num_objects_missing': 3850, 'num_objects_degraded': 0, 'num_objects_misplaced': 7714, 'num_objects_unfound': 0, 'num_objects_dirty': 4294, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15436, 'num_write_kb': 631360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5568, 'num_bytes_recovered': 364904448, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 10], 'acting': [0, 10], 'avail_no_missing': ['2', '5'], 'object_location_counts': [{'shards': '0,2,5', 'objects': 14}, {'shards': '0,10', 'objects': 430}, {'shards': '2,5', 'objects': 3850}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-20T17:16:25.811 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph status --format=json 2024-04-20T17:16:25.961 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:16:26.013 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:16:26.352 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:16:26.353 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:26 smithi005 ceph-mon[39562]: 2.f scrub ok 2024-04-20T17:16:26.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:26.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:16:26.455 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:16:26.482 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:26 smithi106 ceph-mon[44423]: 2.f scrub ok 2024-04-20T17:16:26.485 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:26.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:26 smithi171 ceph-mon[44189]: 2.f scrub ok 2024-04-20T17:16:26.834 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:16:26.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:26.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:16:26.896 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:16:26.938 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:16:26.965 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:26.988 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:16:26.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:26.995 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:16:27.096 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:16:27.124 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:27.145 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:16:27.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:27.148 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:16:27.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:27 smithi005 ceph-mon[39562]: 2.6 scrub starts 2024-04-20T17:16:27.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:27 smithi005 ceph-mon[39562]: 2.6 scrub ok 2024-04-20T17:16:27.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:27 smithi005 ceph-mon[39562]: pgmap v782: 17 pgs: 1 active+recovering, 16 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 140 op/s; 7714/140466 objects misplaced (5.492%); 400 KiB/s, 6 objects/s recovering 2024-04-20T17:16:27.249 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:16:27.282 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:27 smithi171 ceph-mon[44189]: 2.6 scrub starts 2024-04-20T17:16:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:27 smithi171 ceph-mon[44189]: 2.6 scrub ok 2024-04-20T17:16:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:27 smithi171 ceph-mon[44189]: pgmap v782: 17 pgs: 1 active+recovering, 16 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 140 op/s; 7714/140466 objects misplaced (5.492%); 400 KiB/s, 6 objects/s recovering 2024-04-20T17:16:27.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:27 smithi106 ceph-mon[44423]: 2.6 scrub starts 2024-04-20T17:16:27.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:27 smithi106 ceph-mon[44423]: 2.6 scrub ok 2024-04-20T17:16:27.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:27 smithi106 ceph-mon[44423]: pgmap v782: 17 pgs: 1 active+recovering, 16 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 140 op/s; 7714/140466 objects misplaced (5.492%); 400 KiB/s, 6 objects/s recovering 2024-04-20T17:16:27.820 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:16:27.821 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:16:28.645 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-20T17:16:28.925 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:16:29.219 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:16:29.219 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:16:29.219 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:16:29.219 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:16:29.219 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:16:29.220 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:16:29.220 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:16:29.220 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:16:29.220 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:16:29.220 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:16:29.220 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:16:29.220 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:16:29.220 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:16:29.220 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:16:29.220 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:16:29.220 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:16:29.220 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:16:29.220 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:16:29.221 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:16:29.221 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:16:29.221 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:16:29.221 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:16:29.221 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:16:29.221 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:16:29.431 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:29 smithi106 ceph-mon[44423]: pgmap v783: 17 pgs: 1 active+recovering, 16 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 174 op/s; 7616/139640 objects misplaced (5.454%); 800 KiB/s, 12 objects/s recovering 2024-04-20T17:16:29.431 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:29 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3998834827' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:16:29.431 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:29.431 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:29.432 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:29.432 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:29.432 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:29.432 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:29.432 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:29.432 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:16:29.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:29 smithi005 ceph-mon[39562]: pgmap v783: 17 pgs: 1 active+recovering, 16 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 174 op/s; 7616/139640 objects misplaced (5.454%); 800 KiB/s, 12 objects/s recovering 2024-04-20T17:16:29.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:29 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3998834827' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:16:29.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:29.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:29.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:29.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:29.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:29.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:29.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:29.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:29.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:16:29.537 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:16:29.569 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:29 smithi171 ceph-mon[44189]: pgmap v783: 17 pgs: 1 active+recovering, 16 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 174 op/s; 7616/139640 objects misplaced (5.454%); 800 KiB/s, 12 objects/s recovering 2024-04-20T17:16:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:29 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3998834827' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:16:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:29.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:29.921 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:16:30.185 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:30.185 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:30.185 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:30.185 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:30.185 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:30.498 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:30.498 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:30.498 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:30.498 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:30.498 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:30.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:30.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:30.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:30.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:30.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:30.766 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:16:30.766 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:16:30.766 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-04-20T17:16:30.766 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-04-20T17:16:30.767 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:16:31.139 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:16:31.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:31 smithi005 ceph-mon[39562]: pgmap v784: 17 pgs: 1 active+recovering, 16 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 150 op/s; 7616/139354 objects misplaced (5.465%); 716 KiB/s, 11 objects/s recovering 2024-04-20T17:16:31.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:31 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1623589756' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:16:31.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:31 smithi005 ceph-mon[39562]: from='client.25363 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:16:31.507 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:31 smithi106 ceph-mon[44423]: pgmap v784: 17 pgs: 1 active+recovering, 16 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 150 op/s; 7616/139354 objects misplaced (5.465%); 716 KiB/s, 11 objects/s recovering 2024-04-20T17:16:31.507 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:31 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1623589756' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:16:31.507 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:31 smithi106 ceph-mon[44423]: from='client.25363 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:16:31.532 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-20T17:16:31.549 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:16:31.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:31.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:16:31.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:31 smithi171 ceph-mon[44189]: pgmap v784: 17 pgs: 1 active+recovering, 16 active+clean; 4.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 150 op/s; 7616/139354 objects misplaced (5.465%); 716 KiB/s, 11 objects/s recovering 2024-04-20T17:16:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:31 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1623589756' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:16:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:31 smithi171 ceph-mon[44189]: from='client.25363 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:16:31.654 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:16:31.681 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:31.702 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:16:31.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:31.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:16:31.802 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:16:31.805 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:16:31.833 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:31.949 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:16:31.955 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:16:31.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:31.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:16:32.061 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:16:32.096 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:32.445 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:16:32.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:32 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:16:32.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:32 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:16:32.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:32 smithi005 ceph-mon[39562]: from='client.25363 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:16:32.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:32 smithi005 ceph-mon[39562]: osdmap e283: 12 total, 12 up, 12 in 2024-04-20T17:16:32.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:32 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:16:32.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:32 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:16:32.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:32.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:16:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:32 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:16:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:32 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:16:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:32 smithi106 ceph-mon[44423]: from='client.25363 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:16:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:32 smithi106 ceph-mon[44423]: osdmap e283: 12 total, 12 up, 12 in 2024-04-20T17:16:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:32 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:16:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:32 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:16:32.550 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:16:32.580 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:32.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:32 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:16:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:32 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:16:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:32 smithi171 ceph-mon[44189]: from='client.25363 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:16:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:32 smithi171 ceph-mon[44189]: osdmap e283: 12 total, 12 up, 12 in 2024-04-20T17:16:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:32 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:16:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:32 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:16:32.619 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:16:32.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:32.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:16:32.724 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:16:32.755 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:33.067 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:16:33.067 INFO:teuthology.orchestra.run.smithi005.stdout:{"fsid":"13c6e76c-ff37-11ee-bc93-c7b262605968","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":961,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":284,"num_osds":12,"num_up_osds":12,"osd_up_since":1713633141,"num_in_osds":12,"osd_in_since":1713632742,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovering","count":1}],"num_pgs":17,"num_pools":2,"num_objects":69386,"data_bytes":4547674680,"bytes_used":10842628096,"bytes_avail":1141097799680,"bytes_total":1151940427776,"misplaced_objects":7512,"misplaced_total":138772,"misplaced_ratio":0.05413195745539446,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":666366,"recovering_keys_per_sec":0,"num_objects_recovered":101,"num_bytes_recovered":6619136,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":154},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.5:8443/"}},"servicemap":{"epoch":23,"modified":"2024-04-20T17:16:24.047017+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-20T17:16:33.108 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:16:33.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:33.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:16:33.215 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:16:33.245 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:33.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:33 smithi005 ceph-mon[39562]: pgmap v786: 17 pgs: 1 active+recovering, 16 active+clean; 4.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 154 op/s; 7512/138772 objects misplaced (5.413%); 651 KiB/s, 10 objects/s recovering 2024-04-20T17:16:33.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:33 smithi005 ceph-mon[39562]: osdmap e284: 12 total, 12 up, 12 in 2024-04-20T17:16:33.424 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:33 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/266378132' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:16:33.439 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:33 smithi106 ceph-mon[44423]: pgmap v786: 17 pgs: 1 active+recovering, 16 active+clean; 4.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 154 op/s; 7512/138772 objects misplaced (5.413%); 651 KiB/s, 10 objects/s recovering 2024-04-20T17:16:33.440 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:33 smithi106 ceph-mon[44423]: osdmap e284: 12 total, 12 up, 12 in 2024-04-20T17:16:33.440 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:33 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/266378132' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:16:33.443 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:16:33.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:33.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:16:33.547 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:16:33.578 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:33.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:33 smithi171 ceph-mon[44189]: pgmap v786: 17 pgs: 1 active+recovering, 16 active+clean; 4.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 154 op/s; 7512/138772 objects misplaced (5.413%); 651 KiB/s, 10 objects/s recovering 2024-04-20T17:16:33.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:33 smithi171 ceph-mon[44189]: osdmap e284: 12 total, 12 up, 12 in 2024-04-20T17:16:33.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:33 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/266378132' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:16:33.709 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:16:33.709 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-04-20T17:16:33.709 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-04-20T17:16:33.709 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:16:33.710 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.710 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:197147 2.7 2:ea8daa9e:::benchmark_data_smithi171_78918_object36153:head [delete] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-04-20T17:16:33.710 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:33.934691+0000", 2024-04-20T17:16:33.710 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 359.77218678600002, 2024-04-20T17:16:33.710 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.023606318000000001, 2024-04-20T17:16:33.710 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:33.710 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:33.710 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:33.711 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:16:33.711 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:16:33.711 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:16:33.711 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 197147 2024-04-20T17:16:33.711 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.711 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:33.711 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.711 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:33.712 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.934691+0000", 2024-04-20T17:16:33.712 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.712 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.712 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.712 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:33.712 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.934691+0000", 2024-04-20T17:16:33.712 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.712 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.713 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.713 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:33.713 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.934692+0000", 2024-04-20T17:16:33.713 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1990000000000001e-06 2024-04-20T17:16:33.713 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.713 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.713 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:33.713 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.934697+0000", 2024-04-20T17:16:33.714 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1900000000000003e-06 2024-04-20T17:16:33.714 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.714 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.714 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:33.714 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.934706+0000", 2024-04-20T17:16:33.714 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7609999999999996e-06 2024-04-20T17:16:33.714 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.715 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.715 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:33.715 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.934711+0000", 2024-04-20T17:16:33.715 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9169999999999997e-06 2024-04-20T17:16:33.715 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.715 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.715 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:33.715 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.938530+0000", 2024-04-20T17:16:33.716 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0038189280000000001 2024-04-20T17:16:33.716 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.716 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.716 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:33.716 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.938597+0000", 2024-04-20T17:16:33.716 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6685000000000003e-05 2024-04-20T17:16:33.716 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.716 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.717 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:16:33.717 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.938660+0000", 2024-04-20T17:16:33.717 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3102000000000007e-05 2024-04-20T17:16:33.717 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.717 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.717 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:16:33.717 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.943755+0000", 2024-04-20T17:16:33.717 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0050954820000000001 2024-04-20T17:16:33.718 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.718 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.718 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:16:33.718 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.958220+0000", 2024-04-20T17:16:33.718 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.014465107 2024-04-20T17:16:33.718 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.718 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:33.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.958256+0000", 2024-04-20T17:16:33.719 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5411999999999999e-05 2024-04-20T17:16:33.719 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.719 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:33.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.958297+0000", 2024-04-20T17:16:33.719 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1535000000000001e-05 2024-04-20T17:16:33.719 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.720 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:33.720 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.720 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.720 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.720 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:197152 2.7 2:e790fa34:::benchmark_data_smithi171_78918_object36158:head [delete] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-04-20T17:16:33.720 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:33.936770+0000", 2024-04-20T17:16:33.720 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 359.770107526, 2024-04-20T17:16:33.721 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.021672428000000001, 2024-04-20T17:16:33.721 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:33.721 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:33.721 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:33.721 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:16:33.721 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:16:33.721 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:16:33.721 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 197152 2024-04-20T17:16:33.721 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.722 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:33.722 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.722 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:33.722 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.936770+0000", 2024-04-20T17:16:33.722 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.722 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.722 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.722 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:33.723 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.936770+0000", 2024-04-20T17:16:33.723 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.723 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.723 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.723 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:33.723 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.936772+0000", 2024-04-20T17:16:33.723 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.28e-06 2024-04-20T17:16:33.723 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.724 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.724 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:33.724 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.936776+0000", 2024-04-20T17:16:33.724 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.605e-06 2024-04-20T17:16:33.724 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.724 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.724 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:33.724 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.936788+0000", 2024-04-20T17:16:33.725 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1438e-05 2024-04-20T17:16:33.725 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.725 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.725 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:33.725 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.936793+0000", 2024-04-20T17:16:33.725 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7790000000000001e-06 2024-04-20T17:16:33.725 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.725 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.726 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:33.726 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.941066+0000", 2024-04-20T17:16:33.726 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.004272517 2024-04-20T17:16:33.726 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.726 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.726 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:33.726 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.941167+0000", 2024-04-20T17:16:33.726 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000101486 2024-04-20T17:16:33.727 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.727 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.727 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:16:33.727 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.941236+0000", 2024-04-20T17:16:33.727 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8788999999999995e-05 2024-04-20T17:16:33.727 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.727 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.727 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:16:33.727 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.943915+0000", 2024-04-20T17:16:33.728 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0026784880000000001 2024-04-20T17:16:33.728 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.728 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.728 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:16:33.728 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.958350+0000", 2024-04-20T17:16:33.728 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.014435784 2024-04-20T17:16:33.728 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.728 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.729 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:33.729 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.958402+0000", 2024-04-20T17:16:33.729 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1932999999999999e-05 2024-04-20T17:16:33.729 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.729 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.729 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:33.729 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.958443+0000", 2024-04-20T17:16:33.730 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0329e-05 2024-04-20T17:16:33.730 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.730 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:33.730 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.730 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.730 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.730 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:197920 2.7 2:e55163d9:::benchmark_data_smithi171_78918_object36926:head [delete] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-04-20T17:16:33.730 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:34.682047+0000", 2024-04-20T17:16:33.730 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 359.02483102899998, 2024-04-20T17:16:33.731 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.022660026, 2024-04-20T17:16:33.731 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:33.731 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:33.731 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:33.731 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:16:33.731 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:16:33.731 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:16:33.731 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 197920 2024-04-20T17:16:33.732 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.732 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:33.732 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.732 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:33.732 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.682047+0000", 2024-04-20T17:16:33.732 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.732 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.732 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.732 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:33.733 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.682047+0000", 2024-04-20T17:16:33.733 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.733 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.733 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.733 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:33.733 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.682048+0000", 2024-04-20T17:16:33.733 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.381e-06 2024-04-20T17:16:33.733 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.734 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.734 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:33.734 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.682064+0000", 2024-04-20T17:16:33.734 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6047000000000001e-05 2024-04-20T17:16:33.734 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.734 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.735 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:33.735 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.682074+0000", 2024-04-20T17:16:33.735 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0039e-05 2024-04-20T17:16:33.735 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.735 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.735 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:33.735 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.682082+0000", 2024-04-20T17:16:33.735 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8560000000000007e-06 2024-04-20T17:16:33.736 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.736 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.736 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:33.736 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.686324+0000", 2024-04-20T17:16:33.736 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0042415380000000004 2024-04-20T17:16:33.736 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.736 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.736 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:33.737 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.686460+0000", 2024-04-20T17:16:33.737 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000136618 2024-04-20T17:16:33.737 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.737 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.737 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:16:33.737 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.686596+0000", 2024-04-20T17:16:33.738 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013620200000000001 2024-04-20T17:16:33.738 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.738 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.738 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:16:33.738 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.691247+0000", 2024-04-20T17:16:33.738 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0046507700000000003 2024-04-20T17:16:33.738 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.738 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.739 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:16:33.739 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.704602+0000", 2024-04-20T17:16:33.739 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.013354672 2024-04-20T17:16:33.739 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.739 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.739 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:33.739 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.704658+0000", 2024-04-20T17:16:33.740 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5721999999999999e-05 2024-04-20T17:16:33.740 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.740 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.740 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:33.740 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.704707+0000", 2024-04-20T17:16:33.740 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9181000000000002e-05 2024-04-20T17:16:33.740 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.740 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:33.741 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.741 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.741 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.741 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:203733 2.7 2:e2dc7fc6:::benchmark_data_smithi171_78918_object42739:head [delete] snapc 0=[] ondisk+write+known_if_redirected e167)", 2024-04-20T17:16:33.741 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:41.382712+0000", 2024-04-20T17:16:33.741 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 352.32416533100002, 2024-04-20T17:16:33.741 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.024753645000000001, 2024-04-20T17:16:33.741 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:33.742 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:33.742 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:33.742 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:16:33.742 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:16:33.742 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:16:33.742 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 203733 2024-04-20T17:16:33.742 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.743 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:33.743 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.743 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:33.743 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.382712+0000", 2024-04-20T17:16:33.743 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.743 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.743 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.743 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:33.744 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.382712+0000", 2024-04-20T17:16:33.744 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.744 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.744 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.744 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:33.744 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.382715+0000", 2024-04-20T17:16:33.744 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2139999999999999e-06 2024-04-20T17:16:33.744 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.745 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.745 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:33.745 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.382722+0000", 2024-04-20T17:16:33.745 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9310000000000002e-06 2024-04-20T17:16:33.745 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.745 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.745 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:33.745 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.382736+0000", 2024-04-20T17:16:33.746 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4786999999999999e-05 2024-04-20T17:16:33.746 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.746 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.746 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:33.746 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.382747+0000", 2024-04-20T17:16:33.746 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0186999999999999e-05 2024-04-20T17:16:33.746 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.746 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.747 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:33.747 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.382815+0000", 2024-04-20T17:16:33.747 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8223000000000001e-05 2024-04-20T17:16:33.747 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.747 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.747 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:33.747 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.382955+0000", 2024-04-20T17:16:33.747 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013996599999999999 2024-04-20T17:16:33.748 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.748 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.748 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:16:33.748 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.383060+0000", 2024-04-20T17:16:33.748 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010482800000000001 2024-04-20T17:16:33.748 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.748 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.748 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:16:33.749 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.389788+0000", 2024-04-20T17:16:33.749 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.006728365 2024-04-20T17:16:33.749 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.749 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.749 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:16:33.749 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.407360+0000", 2024-04-20T17:16:33.749 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.017572042999999999 2024-04-20T17:16:33.749 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.750 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.750 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:33.750 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.407421+0000", 2024-04-20T17:16:33.750 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1248999999999996e-05 2024-04-20T17:16:33.750 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.750 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.750 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:33.751 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.407466+0000", 2024-04-20T17:16:33.751 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4851999999999997e-05 2024-04-20T17:16:33.751 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.751 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:33.751 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.751 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.751 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.751 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:203740 2.7 2:ee294609:::benchmark_data_smithi171_78918_object42746:head [delete] snapc 0=[] ondisk+write+known_if_redirected e167)", 2024-04-20T17:16:33.751 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:41.383679+0000", 2024-04-20T17:16:33.751 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 352.32319914800001, 2024-04-20T17:16:33.751 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.023936742, 2024-04-20T17:16:33.751 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:33.751 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:33.751 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:33.752 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:16:33.752 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:16:33.752 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:16:33.752 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 203740 2024-04-20T17:16:33.752 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.752 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:33.752 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.752 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:33.752 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.383679+0000", 2024-04-20T17:16:33.752 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.752 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.752 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.752 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:33.752 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.383679+0000", 2024-04-20T17:16:33.752 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.753 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.753 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.753 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:33.753 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.383680+0000", 2024-04-20T17:16:33.753 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5850000000000001e-06 2024-04-20T17:16:33.753 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.753 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.753 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:33.753 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.383685+0000", 2024-04-20T17:16:33.753 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7720000000000002e-06 2024-04-20T17:16:33.753 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.753 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.753 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:33.753 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.383699+0000", 2024-04-20T17:16:33.754 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3696999999999999e-05 2024-04-20T17:16:33.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.754 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:33.754 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.383707+0000", 2024-04-20T17:16:33.754 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2050000000000002e-06 2024-04-20T17:16:33.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.754 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:33.754 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.386373+0000", 2024-04-20T17:16:33.754 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0026656520000000001 2024-04-20T17:16:33.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.754 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:33.755 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.386490+0000", 2024-04-20T17:16:33.755 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000117297 2024-04-20T17:16:33.755 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.755 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.755 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:16:33.755 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.386589+0000", 2024-04-20T17:16:33.755 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8766999999999998e-05 2024-04-20T17:16:33.755 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.755 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.755 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:16:33.755 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.389920+0000", 2024-04-20T17:16:33.755 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003331582 2024-04-20T17:16:33.755 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.756 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.756 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:16:33.756 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.407515+0000", 2024-04-20T17:16:33.756 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.017594957000000001 2024-04-20T17:16:33.756 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.756 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.756 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:33.756 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.407574+0000", 2024-04-20T17:16:33.756 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9120000000000003e-05 2024-04-20T17:16:33.756 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.756 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.756 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:33.756 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:41.407615+0000", 2024-04-20T17:16:33.756 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1108e-05 2024-04-20T17:16:33.756 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.757 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:33.757 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.757 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.757 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.757 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:205043 2.7 2:ef4284f4:::benchmark_data_smithi171_78918_object44049:head [delete] snapc 0=[] ondisk+write+known_if_redirected e167)", 2024-04-20T17:16:33.757 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:42.749577+0000", 2024-04-20T17:16:33.757 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 350.95730098199999, 2024-04-20T17:16:33.757 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.023824217000000002, 2024-04-20T17:16:33.757 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:33.757 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:33.757 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:33.757 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:16:33.757 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:16:33.757 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:16:33.758 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 205043 2024-04-20T17:16:33.758 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.758 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:33.758 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.758 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:33.758 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:42.749577+0000", 2024-04-20T17:16:33.758 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.758 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.758 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.758 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:33.758 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:42.749577+0000", 2024-04-20T17:16:33.758 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.758 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.758 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.759 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:33.759 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:42.749578+0000", 2024-04-20T17:16:33.759 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.604e-06 2024-04-20T17:16:33.759 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.759 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.759 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:33.759 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:42.749584+0000", 2024-04-20T17:16:33.759 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1699999999999996e-06 2024-04-20T17:16:33.759 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.759 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.759 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:33.759 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:42.749594+0000", 2024-04-20T17:16:33.759 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0852e-05 2024-04-20T17:16:33.759 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.760 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.760 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:33.760 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:42.749604+0000", 2024-04-20T17:16:33.760 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4190000000000008e-06 2024-04-20T17:16:33.760 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.760 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.760 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:33.760 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:42.757543+0000", 2024-04-20T17:16:33.760 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0079390180000000008 2024-04-20T17:16:33.760 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.760 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.760 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:33.760 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:42.757669+0000", 2024-04-20T17:16:33.760 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012637799999999999 2024-04-20T17:16:33.760 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:16:33.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:42.757758+0000", 2024-04-20T17:16:33.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8449999999999995e-05 2024-04-20T17:16:33.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:16:33.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:42.761678+0000", 2024-04-20T17:16:33.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003920148 2024-04-20T17:16:33.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:16:33.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:42.773299+0000", 2024-04-20T17:16:33.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011621444999999999 2024-04-20T17:16:33.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:33.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:42.773356+0000", 2024-04-20T17:16:33.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6674e-05 2024-04-20T17:16:33.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:33.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:42.773401+0000", 2024-04-20T17:16:33.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5058999999999997e-05 2024-04-20T17:16:33.762 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.762 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:33.762 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.763 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.763 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:16901 2.1 e208/196 2:881f254a:::benchmark_data_smithi171_78918_object154004:head v 208'16974, mlcod=208'16964)", 2024-04-20T17:16:33.763 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:38.734500+0000", 2024-04-20T17:16:33.763 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 234.972377634, 2024-04-20T17:16:33.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.021894659, 2024-04-20T17:16:33.763 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:33.763 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:33.763 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:33.763 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:33.763 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.763 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:33.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.734500+0000", 2024-04-20T17:16:33.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.764 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.764 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.764 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:33.764 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.734500+0000", 2024-04-20T17:16:33.764 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.764 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.764 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.764 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:33.764 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.734501+0000", 2024-04-20T17:16:33.764 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.31e-06 2024-04-20T17:16:33.764 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.764 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.764 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:33.764 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.734508+0000", 2024-04-20T17:16:33.764 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9380000000000003e-06 2024-04-20T17:16:33.765 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.765 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.765 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:33.765 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.734517+0000", 2024-04-20T17:16:33.765 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9239999999999996e-06 2024-04-20T17:16:33.765 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.765 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.765 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:33.765 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.734524+0000", 2024-04-20T17:16:33.765 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0949999999999998e-06 2024-04-20T17:16:33.765 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.765 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.765 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:33.765 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.739316+0000", 2024-04-20T17:16:33.765 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0047910909999999999 2024-04-20T17:16:33.766 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.766 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.766 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:33.766 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.739338+0000", 2024-04-20T17:16:33.766 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.279e-05 2024-04-20T17:16:33.766 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.766 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.766 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:33.766 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.756351+0000", 2024-04-20T17:16:33.766 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.017012994 2024-04-20T17:16:33.766 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.766 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.766 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:33.766 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.756395+0000", 2024-04-20T17:16:33.767 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3516999999999997e-05 2024-04-20T17:16:33.767 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.767 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:33.767 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.767 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.767 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.767 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:16904 2.1 e208/196 2:88200d33:::benchmark_data_smithi171_78918_object94734:head v 208'16976, mlcod=208'16964)", 2024-04-20T17:16:33.767 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:38.739676+0000", 2024-04-20T17:16:33.767 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 234.96720199399999, 2024-04-20T17:16:33.767 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.026299711999999999, 2024-04-20T17:16:33.767 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:33.767 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:33.767 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:33.767 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:33.767 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.768 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:33.768 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.739676+0000", 2024-04-20T17:16:33.768 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.768 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.768 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.768 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:33.768 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.739676+0000", 2024-04-20T17:16:33.768 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.768 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.768 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.768 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:33.768 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.739677+0000", 2024-04-20T17:16:33.768 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.612e-06 2024-04-20T17:16:33.768 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.768 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.769 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:33.769 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.739685+0000", 2024-04-20T17:16:33.769 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8790000000000003e-06 2024-04-20T17:16:33.769 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.769 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.769 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:33.769 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.739697+0000", 2024-04-20T17:16:33.769 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1344e-05 2024-04-20T17:16:33.769 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.769 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.769 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:33.769 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.739705+0000", 2024-04-20T17:16:33.769 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9039999999999992e-06 2024-04-20T17:16:33.770 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.770 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.770 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:33.770 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.744115+0000", 2024-04-20T17:16:33.770 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0044101619999999996 2024-04-20T17:16:33.770 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.770 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.770 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:33.770 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.744137+0000", 2024-04-20T17:16:33.770 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2812000000000001e-05 2024-04-20T17:16:33.770 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.770 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.770 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:33.770 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.765928+0000", 2024-04-20T17:16:33.770 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.021790990999999999 2024-04-20T17:16:33.771 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.771 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.771 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:33.771 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.765975+0000", 2024-04-20T17:16:33.771 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7008000000000001e-05 2024-04-20T17:16:33.771 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.771 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:33.771 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.771 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.771 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.771 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:16905 2.1 e208/196 2:88224025:::benchmark_data_smithi171_78918_object72445:head v 208'16977, mlcod=208'16964)", 2024-04-20T17:16:33.771 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:38.742067+0000", 2024-04-20T17:16:33.771 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 234.964810355, 2024-04-20T17:16:33.771 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.023927125, 2024-04-20T17:16:33.772 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:33.772 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:33.772 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:33.772 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:33.772 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.772 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:33.772 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.742067+0000", 2024-04-20T17:16:33.772 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.772 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.772 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.772 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:33.772 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.742067+0000", 2024-04-20T17:16:33.772 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.772 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.773 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.773 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:33.773 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.742069+0000", 2024-04-20T17:16:33.773 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.573e-06 2024-04-20T17:16:33.773 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.773 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.773 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:33.773 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.742079+0000", 2024-04-20T17:16:33.773 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6609999999999995e-06 2024-04-20T17:16:33.773 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.773 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.773 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:33.773 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.742087+0000", 2024-04-20T17:16:33.773 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1589999999999993e-06 2024-04-20T17:16:33.773 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.774 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.774 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:33.774 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.742094+0000", 2024-04-20T17:16:33.774 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5650000000000004e-06 2024-04-20T17:16:33.774 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.774 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.774 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:33.774 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.746479+0000", 2024-04-20T17:16:33.774 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0043843149999999997 2024-04-20T17:16:33.774 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.774 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.774 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:33.775 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.746505+0000", 2024-04-20T17:16:33.775 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6585e-05 2024-04-20T17:16:33.775 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.775 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.775 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:33.775 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.765979+0000", 2024-04-20T17:16:33.775 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.019474207 2024-04-20T17:16:33.775 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.775 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.775 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:33.775 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.765995+0000", 2024-04-20T17:16:33.775 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.506e-05 2024-04-20T17:16:33.775 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.775 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:33.776 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.776 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.776 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.776 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:16907 2.1 e208/196 2:88241c2a:::benchmark_data_smithi171_78918_object149251:head v 208'16979, mlcod=208'16964)", 2024-04-20T17:16:33.776 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:38.746952+0000", 2024-04-20T17:16:33.776 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 234.95992608700001, 2024-04-20T17:16:33.776 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.025347308999999998, 2024-04-20T17:16:33.776 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:33.776 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:33.776 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:33.776 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:33.776 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.776 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:33.776 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.746952+0000", 2024-04-20T17:16:33.776 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.777 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.777 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.777 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:33.777 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.746952+0000", 2024-04-20T17:16:33.777 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.777 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.777 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.777 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:33.777 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.746953+0000", 2024-04-20T17:16:33.777 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6899999999999999e-06 2024-04-20T17:16:33.777 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.777 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.777 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:33.777 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.746961+0000", 2024-04-20T17:16:33.778 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4150000000000002e-06 2024-04-20T17:16:33.778 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.778 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.778 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:33.778 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.746971+0000", 2024-04-20T17:16:33.778 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0149999999999999e-05 2024-04-20T17:16:33.778 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.778 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.778 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:33.778 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.746977+0000", 2024-04-20T17:16:33.778 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3169999999999998e-06 2024-04-20T17:16:33.778 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.778 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.778 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:33.779 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.753997+0000", 2024-04-20T17:16:33.779 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0070197419999999998 2024-04-20T17:16:33.779 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.779 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.779 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:33.779 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.754023+0000", 2024-04-20T17:16:33.779 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6067999999999998e-05 2024-04-20T17:16:33.779 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.779 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.779 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:33.779 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.772265+0000", 2024-04-20T17:16:33.779 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.018241908000000001 2024-04-20T17:16:33.779 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.779 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.779 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:33.780 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.772299+0000", 2024-04-20T17:16:33.780 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4019000000000001e-05 2024-04-20T17:16:33.780 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.780 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:33.780 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.780 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.780 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.780 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:16908 2.1 e208/196 2:8824861e:::benchmark_data_smithi171_78918_object60058:head v 208'16980, mlcod=208'16964)", 2024-04-20T17:16:33.780 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:38.749127+0000", 2024-04-20T17:16:33.780 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 234.95775062199999, 2024-04-20T17:16:33.780 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.023192629999999999, 2024-04-20T17:16:33.780 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:33.780 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:33.780 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:33.780 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:33.781 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.781 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:33.781 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.749127+0000", 2024-04-20T17:16:33.781 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.781 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.781 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.781 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:33.781 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.749127+0000", 2024-04-20T17:16:33.781 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.781 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.781 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.781 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:33.781 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.749129+0000", 2024-04-20T17:16:33.781 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8059999999999999e-06 2024-04-20T17:16:33.781 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.782 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.782 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:33.782 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.749138+0000", 2024-04-20T17:16:33.782 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5660000000000007e-06 2024-04-20T17:16:33.782 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.782 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.782 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:33.782 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.749150+0000", 2024-04-20T17:16:33.782 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2497000000000001e-05 2024-04-20T17:16:33.782 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.782 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.782 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:33.782 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.749157+0000", 2024-04-20T17:16:33.782 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8589999999999999e-06 2024-04-20T17:16:33.783 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.783 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.783 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:33.783 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.756470+0000", 2024-04-20T17:16:33.783 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0073132900000000001 2024-04-20T17:16:33.783 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.783 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.783 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:33.783 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.756488+0000", 2024-04-20T17:16:33.783 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7677999999999999e-05 2024-04-20T17:16:33.783 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.783 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.783 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:33.783 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.772303+0000", 2024-04-20T17:16:33.784 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015815175000000001 2024-04-20T17:16:33.784 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.784 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.784 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:33.784 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.772320+0000", 2024-04-20T17:16:33.784 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6759e-05 2024-04-20T17:16:33.784 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.784 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:33.784 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.784 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.784 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.784 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:16910 2.1 e208/196 2:882582ca:::benchmark_data_smithi171_78918_object136366:head v 208'16982, mlcod=208'16964)", 2024-04-20T17:16:33.784 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:38.753684+0000", 2024-04-20T17:16:33.784 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 234.95319391499999, 2024-04-20T17:16:33.785 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.023972904, 2024-04-20T17:16:33.785 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:33.785 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:33.785 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:33.785 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:33.785 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.785 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:33.785 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.753684+0000", 2024-04-20T17:16:33.785 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.785 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.785 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.785 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:33.785 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.753684+0000", 2024-04-20T17:16:33.785 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.786 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.786 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.786 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:33.786 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.753685+0000", 2024-04-20T17:16:33.786 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5200000000000001e-06 2024-04-20T17:16:33.786 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.786 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.786 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:33.786 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.753692+0000", 2024-04-20T17:16:33.786 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0140000000000001e-06 2024-04-20T17:16:33.786 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.786 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.786 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:33.786 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.753703+0000", 2024-04-20T17:16:33.787 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0569e-05 2024-04-20T17:16:33.787 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.787 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.787 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:33.787 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.753709+0000", 2024-04-20T17:16:33.787 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7679999999999997e-06 2024-04-20T17:16:33.787 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.787 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.787 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:33.787 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.763427+0000", 2024-04-20T17:16:33.787 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0097184950000000006 2024-04-20T17:16:33.787 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.787 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.787 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:33.788 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.763451+0000", 2024-04-20T17:16:33.788 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3818000000000001e-05 2024-04-20T17:16:33.788 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.788 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.788 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:33.788 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.777613+0000", 2024-04-20T17:16:33.788 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.014161711 2024-04-20T17:16:33.788 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.788 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.788 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:33.788 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.777657+0000", 2024-04-20T17:16:33.788 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4008999999999998e-05 2024-04-20T17:16:33.788 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.788 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:33.789 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.789 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.789 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.789 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:16911 2.1 e208/196 2:88260f83:::benchmark_data_smithi171_78918_object56795:head v 208'16983, mlcod=208'16964)", 2024-04-20T17:16:33.789 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:38.755617+0000", 2024-04-20T17:16:33.789 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 234.95126111499999, 2024-04-20T17:16:33.789 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.022073652999999999, 2024-04-20T17:16:33.789 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:33.789 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:33.789 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:33.789 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:33.789 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.789 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:33.789 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.755617+0000", 2024-04-20T17:16:33.789 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.790 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.790 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.790 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:33.790 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.755617+0000", 2024-04-20T17:16:33.790 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.790 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.790 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.790 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:33.790 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.755618+0000", 2024-04-20T17:16:33.790 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.567e-06 2024-04-20T17:16:33.790 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.790 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.790 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:33.790 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.755627+0000", 2024-04-20T17:16:33.791 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.473e-06 2024-04-20T17:16:33.791 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.791 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.791 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:33.791 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.755641+0000", 2024-04-20T17:16:33.791 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4127999999999999e-05 2024-04-20T17:16:33.791 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.791 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.791 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:33.791 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.755649+0000", 2024-04-20T17:16:33.791 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7160000000000003e-06 2024-04-20T17:16:33.791 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.791 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.791 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:33.792 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.766042+0000", 2024-04-20T17:16:33.792 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010393528000000001 2024-04-20T17:16:33.792 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.792 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.792 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:33.792 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.766055+0000", 2024-04-20T17:16:33.792 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2850999999999999e-05 2024-04-20T17:16:33.792 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.792 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.792 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:33.792 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.777660+0000", 2024-04-20T17:16:33.792 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011605167 2024-04-20T17:16:33.792 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.792 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.792 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:33.793 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.777690+0000", 2024-04-20T17:16:33.793 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0222999999999998e-05 2024-04-20T17:16:33.793 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.793 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:33.793 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.793 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.793 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.793 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:16913 2.1 e208/196 2:8827185f:::benchmark_data_smithi171_78918_object97707:head v 208'16985, mlcod=208'16964)", 2024-04-20T17:16:33.793 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:38.759277+0000", 2024-04-20T17:16:33.793 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 234.94760100400001, 2024-04-20T17:16:33.793 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.023300471999999999, 2024-04-20T17:16:33.793 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:33.793 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:33.793 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:33.794 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:33.794 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.794 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:33.794 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.759277+0000", 2024-04-20T17:16:33.794 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.794 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.794 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.794 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:33.794 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.759277+0000", 2024-04-20T17:16:33.794 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.794 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.794 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.794 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:33.794 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.759278+0000", 2024-04-20T17:16:33.794 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.037e-06 2024-04-20T17:16:33.795 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.795 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.795 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:33.795 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.759283+0000", 2024-04-20T17:16:33.795 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3020000000000002e-06 2024-04-20T17:16:33.795 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.795 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.795 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:33.795 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.759291+0000", 2024-04-20T17:16:33.795 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0560000000000005e-06 2024-04-20T17:16:33.795 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.795 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.795 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:33.795 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.759314+0000", 2024-04-20T17:16:33.795 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2818000000000001e-05 2024-04-20T17:16:33.796 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.796 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.796 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:33.796 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.770149+0000", 2024-04-20T17:16:33.796 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010835299 2024-04-20T17:16:33.796 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.796 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.796 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:33.796 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.770166+0000", 2024-04-20T17:16:33.796 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7200000000000001e-05 2024-04-20T17:16:33.796 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.796 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.796 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:33.796 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.782553+0000", 2024-04-20T17:16:33.797 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012386359 2024-04-20T17:16:33.797 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.797 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.797 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:33.797 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.782577+0000", 2024-04-20T17:16:33.797 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4400999999999999e-05 2024-04-20T17:16:33.797 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.797 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:33.797 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.797 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.797 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.797 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:16914 2.1 e208/196 2:882743ef:::benchmark_data_smithi171_78918_object108384:head v 208'16986, mlcod=208'16964)", 2024-04-20T17:16:33.797 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:38.760999+0000", 2024-04-20T17:16:33.798 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 234.945879002, 2024-04-20T17:16:33.798 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.021603013000000001, 2024-04-20T17:16:33.798 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:33.798 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:33.798 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:33.798 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:33.798 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.798 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:33.798 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.760999+0000", 2024-04-20T17:16:33.798 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.798 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.798 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.798 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:33.798 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.760999+0000", 2024-04-20T17:16:33.799 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.799 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.799 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.799 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:33.799 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.761000+0000", 2024-04-20T17:16:33.799 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.082e-06 2024-04-20T17:16:33.799 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.799 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.799 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:33.799 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.761007+0000", 2024-04-20T17:16:33.799 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7710000000000001e-06 2024-04-20T17:16:33.799 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.799 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.799 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:33.799 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.761016+0000", 2024-04-20T17:16:33.800 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0089999999999996e-06 2024-04-20T17:16:33.800 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.800 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.800 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:33.800 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.761022+0000", 2024-04-20T17:16:33.800 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8989999999999998e-06 2024-04-20T17:16:33.800 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.800 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.800 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:33.800 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.772375+0000", 2024-04-20T17:16:33.800 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011353781 2024-04-20T17:16:33.800 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.800 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.800 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:33.801 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.772390+0000", 2024-04-20T17:16:33.801 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4823e-05 2024-04-20T17:16:33.801 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.801 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.801 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:33.801 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.782579+0000", 2024-04-20T17:16:33.801 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010189024 2024-04-20T17:16:33.801 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.801 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.801 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:33.801 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:38.782602+0000", 2024-04-20T17:16:33.801 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2623999999999999e-05 2024-04-20T17:16:33.801 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.801 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:33.801 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.802 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.802 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.802 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:17110 2.1 e209/196 2:889b19a7:::benchmark_data_smithi171_78918_object80717:head v 209'17168, mlcod=209'17148)", 2024-04-20T17:16:33.802 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:39.201487+0000", 2024-04-20T17:16:33.802 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 234.50539083999999, 2024-04-20T17:16:33.802 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.021433605000000001, 2024-04-20T17:16:33.802 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:33.802 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:33.802 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:33.802 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:33.802 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.802 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:33.802 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.201487+0000", 2024-04-20T17:16:33.802 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.803 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.803 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.803 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:33.803 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.201487+0000", 2024-04-20T17:16:33.803 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.803 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.803 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.803 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:33.803 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.201488+0000", 2024-04-20T17:16:33.803 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.505e-06 2024-04-20T17:16:33.803 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.803 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.803 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:33.803 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.201498+0000", 2024-04-20T17:16:33.804 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.645e-06 2024-04-20T17:16:33.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.804 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.804 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:33.804 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.201507+0000", 2024-04-20T17:16:33.804 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6419999999999996e-06 2024-04-20T17:16:33.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.804 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.804 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:33.804 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.201512+0000", 2024-04-20T17:16:33.804 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1569999999999999e-06 2024-04-20T17:16:33.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.804 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.804 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:33.805 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.206347+0000", 2024-04-20T17:16:33.805 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0048346259999999999 2024-04-20T17:16:33.805 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.805 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.805 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:33.805 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.206362+0000", 2024-04-20T17:16:33.805 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5221e-05 2024-04-20T17:16:33.805 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.805 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.805 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:33.805 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.222884+0000", 2024-04-20T17:16:33.805 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.016522326 2024-04-20T17:16:33.805 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.805 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.805 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:33.806 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.222921+0000", 2024-04-20T17:16:33.806 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6483000000000003e-05 2024-04-20T17:16:33.806 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.806 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:33.806 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.806 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.806 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.806 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:17112 2.1 e209/196 2:889c7af1:::benchmark_data_smithi171_78918_object66844:head v 209'17170, mlcod=209'17148)", 2024-04-20T17:16:33.806 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:39.204841+0000", 2024-04-20T17:16:33.806 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 234.502036503, 2024-04-20T17:16:33.806 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.02283437, 2024-04-20T17:16:33.806 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:33.806 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:33.806 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:33.807 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:33.807 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.807 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:33.807 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.204841+0000", 2024-04-20T17:16:33.807 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.807 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.807 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.807 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:33.807 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.204841+0000", 2024-04-20T17:16:33.807 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.807 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.807 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.807 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:33.807 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.204843+0000", 2024-04-20T17:16:33.808 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.488e-06 2024-04-20T17:16:33.808 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.808 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.808 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:33.808 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.204853+0000", 2024-04-20T17:16:33.808 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9520000000000006e-06 2024-04-20T17:16:33.808 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.808 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.808 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:33.808 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.204863+0000", 2024-04-20T17:16:33.808 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0762e-05 2024-04-20T17:16:33.808 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.808 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.808 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:33.808 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.204872+0000", 2024-04-20T17:16:33.808 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4379999999999995e-06 2024-04-20T17:16:33.809 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.809 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.809 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:33.809 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.211280+0000", 2024-04-20T17:16:33.809 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0064079460000000003 2024-04-20T17:16:33.809 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.809 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.809 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:33.809 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.211295+0000", 2024-04-20T17:16:33.809 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4778e-05 2024-04-20T17:16:33.809 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.809 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.809 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:33.809 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.227627+0000", 2024-04-20T17:16:33.810 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.016332276999999999 2024-04-20T17:16:33.810 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.810 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.810 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:33.810 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.227676+0000", 2024-04-20T17:16:33.810 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8729000000000001e-05 2024-04-20T17:16:33.810 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.810 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:33.810 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.810 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.810 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.810 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:17115 2.1 e209/196 2:889cf538:::benchmark_data_smithi171_78918_object138127:head v 209'17173, mlcod=209'17148)", 2024-04-20T17:16:33.810 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:39.209715+0000", 2024-04-20T17:16:33.810 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 234.49716254000001, 2024-04-20T17:16:33.811 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.02173806, 2024-04-20T17:16:33.811 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:33.811 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:33.811 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:33.811 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:33.811 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.811 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:33.811 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.209715+0000", 2024-04-20T17:16:33.811 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.811 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.811 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.811 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:33.811 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.209715+0000", 2024-04-20T17:16:33.811 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.811 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.812 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.812 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:33.812 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.209716+0000", 2024-04-20T17:16:33.812 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1969999999999999e-06 2024-04-20T17:16:33.812 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.812 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.812 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:33.812 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.209723+0000", 2024-04-20T17:16:33.812 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6839999999999999e-06 2024-04-20T17:16:33.812 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.812 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.812 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:33.812 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.209733+0000", 2024-04-20T17:16:33.812 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5089999999999999e-06 2024-04-20T17:16:33.813 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.813 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.813 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:33.813 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.209739+0000", 2024-04-20T17:16:33.813 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.066e-06 2024-04-20T17:16:33.813 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.813 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.813 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:33.813 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.220468+0000", 2024-04-20T17:16:33.813 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010729304 2024-04-20T17:16:33.813 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.813 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.813 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:33.813 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.220487+0000", 2024-04-20T17:16:33.813 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8558000000000001e-05 2024-04-20T17:16:33.814 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.814 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.814 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:33.814 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.231405+0000", 2024-04-20T17:16:33.814 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010918572 2024-04-20T17:16:33.814 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.814 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.814 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:33.814 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:39.231453+0000", 2024-04-20T17:16:33.814 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8170000000000001e-05 2024-04-20T17:16:33.814 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.814 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:33.814 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.814 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.815 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.815 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:19009 2.1 e212/196 2:8cb05c7f:::benchmark_data_smithi171_78918_object102033:head v 212'18978, mlcod=212'18965)", 2024-04-20T17:16:33.815 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:44.360334+0000", 2024-04-20T17:16:33.815 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 229.34654338600001, 2024-04-20T17:16:33.815 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.023791541999999999, 2024-04-20T17:16:33.815 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:33.815 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:33.815 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:33.815 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:33.815 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.815 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:33.815 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.360334+0000", 2024-04-20T17:16:33.815 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.815 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.816 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.816 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:33.816 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.360334+0000", 2024-04-20T17:16:33.816 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.816 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.816 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.816 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:33.816 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.360335+0000", 2024-04-20T17:16:33.816 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6100000000000002e-07 2024-04-20T17:16:33.816 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.816 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.816 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:33.816 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.360338+0000", 2024-04-20T17:16:33.816 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3380000000000001e-06 2024-04-20T17:16:33.817 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.817 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.817 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:33.817 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.360342+0000", 2024-04-20T17:16:33.817 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0910000000000003e-06 2024-04-20T17:16:33.817 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.817 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.817 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:33.817 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.360345+0000", 2024-04-20T17:16:33.817 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6850000000000001e-06 2024-04-20T17:16:33.817 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.817 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.817 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:33.817 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.362909+0000", 2024-04-20T17:16:33.818 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0025641840000000002 2024-04-20T17:16:33.818 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.818 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.818 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:33.818 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.362940+0000", 2024-04-20T17:16:33.818 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1402000000000002e-05 2024-04-20T17:16:33.818 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.818 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.818 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:33.818 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.384086+0000", 2024-04-20T17:16:33.818 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.021145265999999999 2024-04-20T17:16:33.818 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.818 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.818 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:33.819 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.384126+0000", 2024-04-20T17:16:33.819 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0215e-05 2024-04-20T17:16:33.819 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.819 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:33.819 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.819 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.819 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.819 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.9.0:19010 2.1 e212/196 2:8cb0f6dc:::benchmark_data_smithi171_78918_object135138:head v 212'18979, mlcod=212'18965)", 2024-04-20T17:16:33.819 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:12:44.362525+0000", 2024-04-20T17:16:33.819 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 229.344353168, 2024-04-20T17:16:33.819 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.021626044000000001, 2024-04-20T17:16:33.819 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:16:33.819 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:16:33.819 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:33.819 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:16:33.820 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.820 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:16:33.820 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.362525+0000", 2024-04-20T17:16:33.820 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.820 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.820 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.820 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:16:33.820 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.362525+0000", 2024-04-20T17:16:33.820 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:16:33.820 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.820 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.820 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:16:33.820 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.362526+0000", 2024-04-20T17:16:33.820 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5689999999999999e-06 2024-04-20T17:16:33.820 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.821 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.821 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:16:33.821 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.362536+0000", 2024-04-20T17:16:33.821 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0059e-05 2024-04-20T17:16:33.821 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.821 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.821 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:16:33.821 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.362546+0000", 2024-04-20T17:16:33.821 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5710000000000004e-06 2024-04-20T17:16:33.821 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.821 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.821 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:16:33.821 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.362553+0000", 2024-04-20T17:16:33.821 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8990000000000004e-06 2024-04-20T17:16:33.821 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.822 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.822 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:16:33.822 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.365804+0000", 2024-04-20T17:16:33.822 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00325115 2024-04-20T17:16:33.822 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.822 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.822 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:16:33.822 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.365830+0000", 2024-04-20T17:16:33.822 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6131000000000001e-05 2024-04-20T17:16:33.822 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.822 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.822 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:16:33.822 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.384130+0000", 2024-04-20T17:16:33.822 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.018299539 2024-04-20T17:16:33.823 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:16:33.823 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:16:33.823 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:16:33.823 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:12:44.384151+0000", 2024-04-20T17:16:33.823 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1126000000000001e-05 2024-04-20T17:16:33.823 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.823 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:33.823 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.823 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:16:33.823 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:16:33.823 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:16:33.929 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:16:33.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:33.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:16:33.963 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-20T17:16:33.964 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:16:33.968 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:16:34.033 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:16:34.062 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:34.496 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-20T17:16:34.510 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:16:34.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:34.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:16:34.614 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:16:34.640 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:34.659 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:16:34.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:34.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:16:34.762 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:16:34.790 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:34.801 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:16:34.815 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:16:34.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:34.820 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:16:34.921 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:16:34.949 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:35.099 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:16:35.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:35.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:16:35.147 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:16:35.203 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:16:35.227 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:16:35.230 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:35.249 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:16:35.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:35.251 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:16:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:35 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:16:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:35 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:16:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:35 smithi106 ceph-mon[44423]: pgmap v788: 17 pgs: 1 active+recovering, 16 active+clean; 4.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 156 op/s; 7512/137954 objects misplaced (5.445%); 808 KiB/s, 12 objects/s recovering 2024-04-20T17:16:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:35 smithi106 ceph-mon[44423]: 2.9 scrub starts 2024-04-20T17:16:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:35 smithi106 ceph-mon[44423]: osdmap e285: 12 total, 12 up, 12 in 2024-04-20T17:16:35.352 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:16:35.388 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:35 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:16:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:35 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:16:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:35 smithi005 ceph-mon[39562]: pgmap v788: 17 pgs: 1 active+recovering, 16 active+clean; 4.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 156 op/s; 7512/137954 objects misplaced (5.445%); 808 KiB/s, 12 objects/s recovering 2024-04-20T17:16:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:35 smithi005 ceph-mon[39562]: 2.9 scrub starts 2024-04-20T17:16:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:35 smithi005 ceph-mon[39562]: osdmap e285: 12 total, 12 up, 12 in 2024-04-20T17:16:35.493 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:16:35.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:35.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:16:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:35 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:16:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:35 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:16:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:35 smithi171 ceph-mon[44189]: pgmap v788: 17 pgs: 1 active+recovering, 16 active+clean; 4.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 156 op/s; 7512/137954 objects misplaced (5.445%); 808 KiB/s, 12 objects/s recovering 2024-04-20T17:16:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:35 smithi171 ceph-mon[44189]: 2.9 scrub starts 2024-04-20T17:16:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:35 smithi171 ceph-mon[44189]: osdmap e285: 12 total, 12 up, 12 in 2024-04-20T17:16:35.599 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:16:35.628 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:35.680 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:16:35.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:35.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:16:35.784 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:16:35.810 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:35.830 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:16:35.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:35.834 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:16:35.934 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:16:35.961 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:35.981 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:16:35.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:35.984 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:16:36.085 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:16:36.115 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:36.137 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:16:36.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:36.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:16:36.241 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:16:36.268 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:36.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:36 smithi005 ceph-mon[39562]: 2.9 scrub ok 2024-04-20T17:16:36.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:36 smithi005 ceph-mon[39562]: 2.1 scrub starts 2024-04-20T17:16:36.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:36 smithi005 ceph-mon[39562]: 2.1 scrub ok 2024-04-20T17:16:36.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:36 smithi005 ceph-mon[39562]: osdmap e286: 12 total, 12 up, 12 in 2024-04-20T17:16:36.462 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:36 smithi106 ceph-mon[44423]: 2.9 scrub ok 2024-04-20T17:16:36.462 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:36 smithi106 ceph-mon[44423]: 2.1 scrub starts 2024-04-20T17:16:36.463 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:36 smithi106 ceph-mon[44423]: 2.1 scrub ok 2024-04-20T17:16:36.463 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:36 smithi106 ceph-mon[44423]: osdmap e286: 12 total, 12 up, 12 in 2024-04-20T17:16:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:36 smithi171 ceph-mon[44189]: 2.9 scrub ok 2024-04-20T17:16:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:36 smithi171 ceph-mon[44189]: 2.1 scrub starts 2024-04-20T17:16:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:36 smithi171 ceph-mon[44189]: 2.1 scrub ok 2024-04-20T17:16:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:36 smithi171 ceph-mon[44189]: osdmap e286: 12 total, 12 up, 12 in 2024-04-20T17:16:36.596 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:16:36.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:36.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:16:36.702 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:16:36.729 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:36.748 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:16:36.748 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:16:36.748 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:16:36.748 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-04-20T17:16:36.748 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:16:36.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:36.750 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:16:36.851 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:16:36.883 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:37.417 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:37 smithi005 ceph-mon[39562]: pgmap v791: 17 pgs: 1 active+recovering, 16 active+clean; 4.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 149 op/s; 7512/137474 objects misplaced (5.464%); 664 KiB/s, 10 objects/s recovering 2024-04-20T17:16:37.480 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:37 smithi106 ceph-mon[44423]: pgmap v791: 17 pgs: 1 active+recovering, 16 active+clean; 4.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 149 op/s; 7512/137474 objects misplaced (5.464%); 664 KiB/s, 10 objects/s recovering 2024-04-20T17:16:37.505 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-20T17:16:37.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:37 smithi171 ceph-mon[44189]: pgmap v791: 17 pgs: 1 active+recovering, 16 active+clean; 4.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 149 op/s; 7512/137474 objects misplaced (5.464%); 664 KiB/s, 10 objects/s recovering 2024-04-20T17:16:37.733 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:16:37.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:37.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:16:37.751 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:16:37.770 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:16:37.771 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:16:37.794 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:16:37.815 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:16:37.838 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:16:37.841 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:16:37.870 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:16:37.871 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:37.903 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:16:37.928 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:16:37.948 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:16:37.968 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:16:37.989 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:16:38.010 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:16:38.014 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:16:38.014 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:16:38.014 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:16:38.014 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:16:38.014 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:16:38.014 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:16:38.014 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:16:38.014 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:16:38.014 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:16:38.015 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:16:38.015 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:16:38.015 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:16:38.022 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:16:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:38.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:16:38.130 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:16:38.159 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:38 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3583780503' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:16:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:38.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:38 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3583780503' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:16:38.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:38.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:38 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3583780503' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:16:38.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:38.606 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:16:38.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:38.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:16:38.710 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:16:38.738 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:39.152 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.152 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.152 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.201 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:16:39.374 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:16:39.376 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:16:39.402 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.402 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.402 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.402 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.402 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.402 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.402 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.402 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.403 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:39 smithi005 ceph-mon[39562]: pgmap v792: 17 pgs: 1 active+recovering, 16 active+clean; 4.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 182 op/s; 7412/136578 objects misplaced (5.427%); 544 KiB/s, 8 objects/s recovering 2024-04-20T17:16:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:39 smithi106 ceph-mon[44423]: pgmap v792: 17 pgs: 1 active+recovering, 16 active+clean; 4.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 182 op/s; 7412/136578 objects misplaced (5.427%); 544 KiB/s, 8 objects/s recovering 2024-04-20T17:16:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:39.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:39 smithi171 ceph-mon[44189]: pgmap v792: 17 pgs: 1 active+recovering, 16 active+clean; 4.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 182 op/s; 7412/136578 objects misplaced (5.427%); 544 KiB/s, 8 objects/s recovering 2024-04-20T17:16:39.686 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:16:39.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:39.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:16:39.791 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:16:39.824 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:40.055 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:16:40.055 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:16:40.281 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:16:40.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:40.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:16:40.385 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:16:40.412 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:40 smithi005 ceph-mon[39562]: from='client.15477 v1:172.21.15.5:0/2164448828' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:16:40.414 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:40.489 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:40 smithi106 ceph-mon[44423]: from='client.15477 v1:172.21.15.5:0/2164448828' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:16:40.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:40 smithi171 ceph-mon[44189]: from='client.15477 v1:172.21.15.5:0/2164448828' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:16:40.837 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-20T17:16:40.859 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:16:40.965 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:16:40.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:40.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:16:40.990 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":792,"stamp":"2024-04-20T17:16:38.050484+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4475781688,"num_objects":68289,"num_object_clones":0,"num_object_copies":136578,"num_objects_missing_on_primary":3700,"num_objects_missing":3700,"num_objects_degraded":0,"num_objects_misplaced":7412,"num_objects_unfound":0,"num_objects_dirty":68289,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":253878,"num_write_kb":10306363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25512,"num_bytes_recovered":1673545856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160927,"ondisk_log_size":160927,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":10480580,"kb_used_data":9339564,"kb_used_omap":219,"kb_used_meta":1140644,"kb_avail":1114461244,"statfs":{"total":1151940427776,"available":1141208313856,"internally_reserved":0,"allocated":9563713536,"data_stored":9552959366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224949,"internal_metadata":1168019787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,0,1,1,1,3,4,6],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-71892992,"num_objects":-1097,"num_object_clones":0,"num_object_copies":-2194,"num_objects_missing_on_primary":-49,"num_objects_missing":-49,"num_objects_degraded":0,"num_objects_misplaced":-100,"num_objects_unfound":0,"num_objects_dirty":-1097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1097,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":3342336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001065"},"pg_stats":[{"pgid":"2.f","version":"284'15843","reported_seq":16699,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-04-20T17:16:32.853537+0000","last_change":"2024-04-20T17:16:24.907036+0000","last_active":"2024-04-20T17:16:32.853537+0000","last_peered":"2024-04-20T17:16:32.853537+0000","last_clean":"2024-04-20T17:16:32.853537+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:16:32.853537+0000","last_undegraded":"2024-04-20T17:16:32.853537+0000","last_fullsized":"2024-04-20T17:16:32.853537+0000","mapping_epoch":115,"log_start":"135'5800","ondisk_log_start":"135'5800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'15773","last_scrub_stamp":"2024-04-20T17:16:24.906931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:24.906931+0000","objects_scrubbed":4413,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:47.687304+0000","scrub_duration":421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284360704,"num_objects":4339,"num_object_clones":0,"num_object_copies":8678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15843,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"285'15786","reported_seq":27661,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-04-20T17:16:35.584002+0000","last_change":"2024-04-20T17:15:31.328405+0000","last_active":"2024-04-20T17:16:35.584002+0000","last_peered":"2024-04-20T17:16:35.584002+0000","last_clean":"2024-04-20T17:16:35.584002+0000","last_became_active":"2024-04-20T17:10:28.865670+0000","last_became_peered":"2024-04-20T17:10:28.865670+0000","last_unstale":"2024-04-20T17:16:35.584002+0000","last_undegraded":"2024-04-20T17:16:35.584002+0000","last_fullsized":"2024-04-20T17:16:35.584002+0000","mapping_epoch":162,"log_start":"135'5700","ondisk_log_start":"135'5700","created":115,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15200","last_scrub_stamp":"2024-04-20T17:15:31.328301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:31.328301+0000","objects_scrubbed":4857,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:38.848343+0000","scrub_duration":424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279838720,"num_objects":4270,"num_object_clones":0,"num_object_copies":8540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15786,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5619,"num_bytes_recovered":368246784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"285'16141","reported_seq":16841,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-04-20T17:16:35.574512+0000","last_change":"2024-04-20T17:15:57.136522+0000","last_active":"2024-04-20T17:16:35.574512+0000","last_peered":"2024-04-20T17:16:35.574512+0000","last_clean":"2024-04-20T17:16:35.574512+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:16:35.574512+0000","last_undegraded":"2024-04-20T17:16:35.574512+0000","last_fullsized":"2024-04-20T17:16:35.574512+0000","mapping_epoch":115,"log_start":"135'6100","ondisk_log_start":"135'6100","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'15780","last_scrub_stamp":"2024-04-20T17:15:57.136412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:57.136412+0000","objects_scrubbed":4702,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:10.130229+0000","scrub_duration":368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284491776,"num_objects":4341,"num_object_clones":0,"num_object_copies":8682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16141,"num_write_kb":655424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"285'22497","reported_seq":43997,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-04-20T17:16:35.593557+0000","last_change":"2024-04-20T17:16:35.136358+0000","last_active":"2024-04-20T17:16:35.593557+0000","last_peered":"2024-04-20T17:16:35.593557+0000","last_clean":"2024-04-20T17:16:35.593557+0000","last_became_active":"2024-04-20T17:12:08.085936+0000","last_became_peered":"2024-04-20T17:12:08.085936+0000","last_unstale":"2024-04-20T17:16:35.593557+0000","last_undegraded":"2024-04-20T17:16:35.593557+0000","last_fullsized":"2024-04-20T17:16:35.593557+0000","mapping_epoch":196,"log_start":"166'12468","ondisk_log_start":"166'12468","created":115,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'22494","last_scrub_stamp":"2024-04-20T17:16:35.136177+0000","last_deep_scrub":"213'20373","last_deep_scrub_stamp":"2024-04-20T17:12:48.443808+0000","last_clean_scrub_stamp":"2024-04-20T17:16:35.136177+0000","objects_scrubbed":4393,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:40.491841+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287637504,"num_objects":4389,"num_object_clones":0,"num_object_copies":8778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15845,"num_write_kb":647488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":450166784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"286'15589","reported_seq":33119,"reported_epoch":286,"state":"active+recovering","last_fresh":"2024-04-20T17:16:36.812556+0000","last_change":"2024-04-20T17:12:26.166109+0000","last_active":"2024-04-20T17:16:36.812556+0000","last_peered":"2024-04-20T17:16:36.812556+0000","last_clean":"2024-04-20T17:09:35.882603+0000","last_became_active":"2024-04-20T17:10:54.470253+0000","last_became_peered":"2024-04-20T17:10:54.470253+0000","last_unstale":"2024-04-20T17:16:36.812556+0000","last_undegraded":"2024-04-20T17:16:36.812556+0000","last_fullsized":"2024-04-20T17:16:36.812556+0000","mapping_epoch":171,"log_start":"131'5500","ondisk_log_start":"131'5500","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271384576,"num_objects":4141,"num_object_clones":0,"num_object_copies":8282,"num_objects_missing_on_primary":3700,"num_objects_missing":3700,"num_objects_degraded":0,"num_objects_misplaced":7412,"num_objects_unfound":0,"num_objects_dirty":4141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15589,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5720,"num_bytes_recovered":374865920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"0,2,5","objects":12},{"shards":"0,10","objects":429},{"shards":"2,5","objects":3700}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":759,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-20T17:16:35.955450+0000","last_change":"2024-04-20T17:14:40.440515+0000","last_active":"2024-04-20T17:16:35.955450+0000","last_peered":"2024-04-20T17:16:35.955450+0000","last_clean":"2024-04-20T17:16:35.955450+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:16:35.955450+0000","last_undegraded":"2024-04-20T17:16:35.955450+0000","last_fullsized":"2024-04-20T17:16:35.955450+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:14:40.440403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:14:40.440403+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"286'15932","reported_seq":16665,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-20T17:16:37.814785+0000","last_change":"2024-04-20T17:15:31.396169+0000","last_active":"2024-04-20T17:16:37.814785+0000","last_peered":"2024-04-20T17:16:37.814785+0000","last_clean":"2024-04-20T17:16:37.814785+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:16:37.814785+0000","last_undegraded":"2024-04-20T17:16:37.814785+0000","last_fullsized":"2024-04-20T17:16:37.814785+0000","mapping_epoch":115,"log_start":"135'5900","ondisk_log_start":"135'5900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15333","last_scrub_stamp":"2024-04-20T17:15:31.396069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:31.396069+0000","objects_scrubbed":4919,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:57.811864+0000","scrub_duration":440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283115520,"num_objects":4320,"num_object_clones":0,"num_object_copies":8640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15932,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"284'16026","reported_seq":16755,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-04-20T17:16:33.362321+0000","last_change":"2024-04-20T17:15:35.632022+0000","last_active":"2024-04-20T17:16:33.362321+0000","last_peered":"2024-04-20T17:16:33.362321+0000","last_clean":"2024-04-20T17:16:33.362321+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:16:33.362321+0000","last_undegraded":"2024-04-20T17:16:33.362321+0000","last_fullsized":"2024-04-20T17:16:33.362321+0000","mapping_epoch":115,"log_start":"135'6000","ondisk_log_start":"135'6000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15459","last_scrub_stamp":"2024-04-20T17:15:35.631842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:35.631842+0000","objects_scrubbed":4867,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:52.716132+0000","scrub_duration":426,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281673728,"num_objects":4298,"num_object_clones":0,"num_object_copies":8596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16026,"num_write_kb":650368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"286'19984","reported_seq":20863,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-20T17:16:36.218361+0000","last_change":"2024-04-20T17:16:25.742067+0000","last_active":"2024-04-20T17:16:36.218361+0000","last_peered":"2024-04-20T17:16:36.218361+0000","last_clean":"2024-04-20T17:16:36.218361+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:16:36.218361+0000","last_undegraded":"2024-04-20T17:16:36.218361+0000","last_fullsized":"2024-04-20T17:16:36.218361+0000","mapping_epoch":115,"log_start":"152'9902","ondisk_log_start":"152'9902","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'19891","last_scrub_stamp":"2024-04-20T17:16:25.741880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:25.741880+0000","objects_scrubbed":4262,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:39.901935+0000","scrub_duration":378,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273219584,"num_objects":4169,"num_object_clones":0,"num_object_copies":8338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15873,"num_write_kb":641344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"286'16004","reported_seq":16793,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-20T17:16:37.400789+0000","last_change":"2024-04-20T17:15:41.950854+0000","last_active":"2024-04-20T17:16:37.400789+0000","last_peered":"2024-04-20T17:16:37.400789+0000","last_clean":"2024-04-20T17:16:37.400789+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:16:37.400789+0000","last_undegraded":"2024-04-20T17:16:37.400789+0000","last_fullsized":"2024-04-20T17:16:37.400789+0000","mapping_epoch":115,"log_start":"135'6000","ondisk_log_start":"135'6000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'15491","last_scrub_stamp":"2024-04-20T17:15:41.950723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:41.950723+0000","objects_scrubbed":4733,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:58.122837+0000","scrub_duration":448,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276299776,"num_objects":4216,"num_object_clones":0,"num_object_copies":8432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16004,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"286'15829","reported_seq":16545,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-20T17:16:37.637758+0000","last_change":"2024-04-20T17:15:59.551771+0000","last_active":"2024-04-20T17:16:37.637758+0000","last_peered":"2024-04-20T17:16:37.637758+0000","last_clean":"2024-04-20T17:16:37.637758+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:16:37.637758+0000","last_undegraded":"2024-04-20T17:16:37.637758+0000","last_fullsized":"2024-04-20T17:16:37.637758+0000","mapping_epoch":115,"log_start":"135'5800","ondisk_log_start":"135'5800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'15438","last_scrub_stamp":"2024-04-20T17:15:59.551671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:59.551671+0000","objects_scrubbed":4614,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:10.078708+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276758528,"num_objects":4223,"num_object_clones":0,"num_object_copies":8446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15829,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"286'15980","reported_seq":31301,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-20T17:16:37.796889+0000","last_change":"2024-04-20T17:16:34.421711+0000","last_active":"2024-04-20T17:16:37.796889+0000","last_peered":"2024-04-20T17:16:37.796889+0000","last_clean":"2024-04-20T17:16:37.796889+0000","last_became_active":"2024-04-20T17:10:55.931751+0000","last_became_peered":"2024-04-20T17:10:55.931751+0000","last_unstale":"2024-04-20T17:16:37.796889+0000","last_undegraded":"2024-04-20T17:16:37.796889+0000","last_fullsized":"2024-04-20T17:16:37.796889+0000","mapping_epoch":173,"log_start":"135'5900","ondisk_log_start":"135'5900","created":115,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'15941","last_scrub_stamp":"2024-04-20T17:16:34.421587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:34.421587+0000","objects_scrubbed":4322,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:43.906503+0000","scrub_duration":366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280625152,"num_objects":4282,"num_object_clones":0,"num_object_copies":8564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15980,"num_write_kb":648384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7296,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"286'16062","reported_seq":16791,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-20T17:16:37.370369+0000","last_change":"2024-04-20T17:15:30.785298+0000","last_active":"2024-04-20T17:16:37.370369+0000","last_peered":"2024-04-20T17:16:37.370369+0000","last_clean":"2024-04-20T17:16:37.370369+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:16:37.370369+0000","last_undegraded":"2024-04-20T17:16:37.370369+0000","last_fullsized":"2024-04-20T17:16:37.370369+0000","mapping_epoch":115,"log_start":"135'6000","ondisk_log_start":"135'6000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"267'15415","last_scrub_stamp":"2024-04-20T17:15:30.785135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:30.785135+0000","objects_scrubbed":4901,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:40.155273+0000","scrub_duration":453,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278790144,"num_objects":4254,"num_object_clones":0,"num_object_copies":8508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16062,"num_write_kb":650112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"284'15556","reported_seq":16280,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-04-20T17:16:33.584376+0000","last_change":"2024-04-20T17:15:34.656600+0000","last_active":"2024-04-20T17:16:33.584376+0000","last_peered":"2024-04-20T17:16:33.584376+0000","last_clean":"2024-04-20T17:16:33.584376+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:16:33.584376+0000","last_undegraded":"2024-04-20T17:16:33.584376+0000","last_fullsized":"2024-04-20T17:16:33.584376+0000","mapping_epoch":115,"log_start":"131'5500","ondisk_log_start":"131'5500","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15023","last_scrub_stamp":"2024-04-20T17:15:34.656503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:34.656503+0000","objects_scrubbed":4842,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:51.197304+0000","scrub_duration":430,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282329088,"num_objects":4308,"num_object_clones":0,"num_object_copies":8616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15556,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"286'15899","reported_seq":16627,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-04-20T17:16:37.234451+0000","last_change":"2024-04-20T17:15:31.794340+0000","last_active":"2024-04-20T17:16:37.234451+0000","last_peered":"2024-04-20T17:16:37.234451+0000","last_clean":"2024-04-20T17:16:37.234451+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:16:37.234451+0000","last_undegraded":"2024-04-20T17:16:37.234451+0000","last_fullsized":"2024-04-20T17:16:37.234451+0000","mapping_epoch":115,"log_start":"135'5800","ondisk_log_start":"135'5800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15271","last_scrub_stamp":"2024-04-20T17:15:31.794156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:31.794156+0000","objects_scrubbed":4831,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:58.805155+0000","scrub_duration":443,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275447808,"num_objects":4203,"num_object_clones":0,"num_object_copies":8406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15899,"num_write_kb":643264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"285'15627","reported_seq":16353,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-04-20T17:16:34.978890+0000","last_change":"2024-04-20T17:15:33.298229+0000","last_active":"2024-04-20T17:16:34.978890+0000","last_peered":"2024-04-20T17:16:34.978890+0000","last_clean":"2024-04-20T17:16:34.978890+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:16:34.978890+0000","last_undegraded":"2024-04-20T17:16:34.978890+0000","last_fullsized":"2024-04-20T17:16:34.978890+0000","mapping_epoch":115,"log_start":"134'5600","ondisk_log_start":"134'5600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15067","last_scrub_stamp":"2024-04-20T17:15:33.298103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:33.298103+0000","objects_scrubbed":4809,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:56.012242+0000","scrub_duration":458,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278003736,"num_objects":4243,"num_object_clones":0,"num_object_copies":8486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4243,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15627,"num_write_kb":635777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"285'15705","reported_seq":16351,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-04-20T17:16:35.862433+0000","last_change":"2024-04-20T17:16:23.593408+0000","last_active":"2024-04-20T17:16:35.862433+0000","last_peered":"2024-04-20T17:16:35.862433+0000","last_clean":"2024-04-20T17:16:35.862433+0000","last_became_active":"2024-04-20T17:12:23.515375+0000","last_became_peered":"2024-04-20T17:12:23.515375+0000","last_unstale":"2024-04-20T17:16:35.862433+0000","last_undegraded":"2024-04-20T17:16:35.862433+0000","last_fullsized":"2024-04-20T17:16:35.862433+0000","mapping_epoch":202,"log_start":"135'5700","ondisk_log_start":"135'5700","created":115,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'15586","last_scrub_stamp":"2024-04-20T17:16:23.593300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:23.593300+0000","objects_scrubbed":4410,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:33.363583+0000","scrub_duration":446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281214976,"num_objects":4291,"num_object_clones":0,"num_object_copies":8582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15705,"num_write_kb":639872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":4475191320,"num_objects":68287,"num_object_clones":0,"num_object_copies":136574,"num_objects_missing_on_primary":3700,"num_objects_missing":3700,"num_objects_degraded":0,"num_objects_misplaced":7412,"num_objects_unfound":0,"num_objects_dirty":68287,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":253697,"num_write_kb":10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25504,"num_bytes_recovered":1671430144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9526583296,"data_stored":9526575152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},"log_size":160790,"ondisk_log_size":160790,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991363,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":884836,"kb_used_data":817560,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92860316,"statfs":{"total":95995035648,"available":95088963584,"internally_reserved":0,"allocated":837181440,"data_stored":836278486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":68876740},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253000,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1006728,"kb_used_data":858936,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92738424,"statfs":{"total":95995035648,"available":94964146176,"internally_reserved":0,"allocated":879550464,"data_stored":878662968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":151321024},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612750,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1252500,"kb_used_data":1104712,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92492652,"statfs":{"total":95995035648,"available":94712475648,"internally_reserved":0,"allocated":1131225088,"data_stored":1130327612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":151321024},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762808979,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1203336,"kb_used_data":1099960,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92541816,"statfs":{"total":95995035648,"available":94762819584,"internally_reserved":0,"allocated":1126359040,"data_stored":1125471564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168729,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":901092,"kb_used_data":825496,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92844060,"statfs":{"total":95995035648,"available":95072317440,"internally_reserved":0,"allocated":845307904,"data_stored":844404970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430365,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":659808,"kb_used_data":552272,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93085344,"statfs":{"total":95995035648,"available":95319392256,"internally_reserved":0,"allocated":565526528,"data_stored":564627146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575031,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":631392,"kb_used_data":546064,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93113760,"statfs":{"total":95995035648,"available":95348490240,"internally_reserved":0,"allocated":559169536,"data_stored":558270154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738556,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":992728,"kb_used_data":844944,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92752424,"statfs":{"total":95995035648,"available":94978482176,"internally_reserved":0,"allocated":865222656,"data_stored":864323274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,1,1,3,4,6],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393844,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":755340,"kb_used_data":706948,"kb_used_omap":202,"kb_used_meta":48181,"kb_avail":92989812,"statfs":{"total":95995035648,"available":95221567488,"internally_reserved":0,"allocated":723914752,"data_stored":723021348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":49337815},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182572,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":622608,"kb_used_data":551172,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93122544,"statfs":{"total":95995035648,"available":95357485056,"internally_reserved":0,"allocated":564400128,"data_stored":563506724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986343,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312984,"kb_used_data":273480,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93432168,"statfs":{"total":95995035648,"available":95674540032,"internally_reserved":0,"allocated":280043520,"data_stored":279146044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790113,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1257228,"kb_used_data":1158020,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92487924,"statfs":{"total":95995035648,"available":94707634176,"internally_reserved":0,"allocated":1185812480,"data_stored":1184919076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":862912512,"data_stored":862912512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":556990464,"data_stored":556990464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":721420288,"data_stored":721420288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":560660480,"data_stored":560660480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":278007808,"data_stored":278003736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1182728192,"data_stored":1182728192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":562429952,"data_stored":562429952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":840433664,"data_stored":840433664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1124073472,"data_stored":1124073472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1126502400,"data_stored":1126498328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":876609536,"data_stored":876609536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":833814528,"data_stored":833814528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:16:41.070 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:16:41.095 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:41.107 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:16:41.114 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:16:41.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:41.117 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:16:41.218 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:16:41.245 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:41.432 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:41 smithi106 ceph-mon[44423]: 2.4 scrub starts 2024-04-20T17:16:41.432 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:41 smithi106 ceph-mon[44423]: pgmap v793: 17 pgs: 1 active+recovering, 16 active+clean; 4.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 160 op/s; 7412/136226 objects misplaced (5.441%); 413 KiB/s, 6 objects/s recovering 2024-04-20T17:16:41.432 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:41 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:16:41.432 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:41 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:16:41.432 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:41 smithi106 ceph-mon[44423]: 2.4 scrub ok 2024-04-20T17:16:41.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:41 smithi005 ceph-mon[39562]: 2.4 scrub starts 2024-04-20T17:16:41.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:41 smithi005 ceph-mon[39562]: pgmap v793: 17 pgs: 1 active+recovering, 16 active+clean; 4.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 160 op/s; 7412/136226 objects misplaced (5.441%); 413 KiB/s, 6 objects/s recovering 2024-04-20T17:16:41.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:41 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:16:41.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:41 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:16:41.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:41 smithi005 ceph-mon[39562]: 2.4 scrub ok 2024-04-20T17:16:41.444 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:16:41.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:41.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:16:41.517 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:16:41.549 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:16:41.582 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:41.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:41 smithi171 ceph-mon[44189]: 2.4 scrub starts 2024-04-20T17:16:41.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:41 smithi171 ceph-mon[44189]: pgmap v793: 17 pgs: 1 active+recovering, 16 active+clean; 4.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 160 op/s; 7412/136226 objects misplaced (5.441%); 413 KiB/s, 6 objects/s recovering 2024-04-20T17:16:41.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:41 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:16:41.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:41 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:16:41.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:41 smithi171 ceph-mon[44189]: 2.4 scrub ok 2024-04-20T17:16:41.831 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:16:41.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:41.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:16:41.935 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:16:41.963 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:41.987 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:16:41.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:41.989 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:16:42.090 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:16:42.119 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:42.368 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:16:42.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:42.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:16:42.472 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:16:42.499 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:42.655 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:16:42.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:42.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:16:42.763 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:16:42.795 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:42.951 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:16:42.951 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:16:43.045 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:16:43.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:43.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:16:43.149 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:16:43.169 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:16:43.178 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:43.203 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:16:43.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:43.206 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:16:43.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:43 smithi106 ceph-mon[44423]: pgmap v794: 17 pgs: 1 active+recovering, 16 active+clean; 4.1 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 150 op/s; 7306/135550 objects misplaced (5.390%); 824 KiB/s, 12 objects/s recovering 2024-04-20T17:16:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:43 smithi106 ceph-mon[44423]: 2.a scrub starts 2024-04-20T17:16:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:43 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3480075452' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:16:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:43 smithi106 ceph-mon[44423]: 2.a scrub ok 2024-04-20T17:16:43.307 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:16:43.337 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:43.357 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:16:43.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:43.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:16:43.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:43 smithi005 ceph-mon[39562]: pgmap v794: 17 pgs: 1 active+recovering, 16 active+clean; 4.1 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 150 op/s; 7306/135550 objects misplaced (5.390%); 824 KiB/s, 12 objects/s recovering 2024-04-20T17:16:43.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:43 smithi005 ceph-mon[39562]: 2.a scrub starts 2024-04-20T17:16:43.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:43 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3480075452' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:16:43.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:43 smithi005 ceph-mon[39562]: 2.a scrub ok 2024-04-20T17:16:43.471 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:16:43.495 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:43 smithi171 ceph-mon[44189]: pgmap v794: 17 pgs: 1 active+recovering, 16 active+clean; 4.1 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 150 op/s; 7306/135550 objects misplaced (5.390%); 824 KiB/s, 12 objects/s recovering 2024-04-20T17:16:43.495 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:43 smithi171 ceph-mon[44189]: 2.a scrub starts 2024-04-20T17:16:43.495 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:43 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3480075452' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:16:43.495 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:43 smithi171 ceph-mon[44189]: 2.a scrub ok 2024-04-20T17:16:43.497 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:43.517 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:16:43.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:43.521 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:16:43.622 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:16:43.648 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:43.666 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:16:43.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:43.670 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:16:43.713 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-20T17:16:43.771 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:16:43.799 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:43.823 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:16:43.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:43.826 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:16:43.927 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:16:43.953 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:43.972 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:16:43.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:43.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:16:43.992 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:16:43.993 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:16:44.077 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:16:44.105 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:44 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:16:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:44 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3480075452' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:16:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:44 smithi106 ceph-mon[44423]: osdmap e287: 12 total, 12 up, 12 in 2024-04-20T17:16:44.305 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:16:44.306 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:44 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:16:44.306 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:44 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3480075452' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:16:44.306 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:44 smithi005 ceph-mon[39562]: osdmap e287: 12 total, 12 up, 12 in 2024-04-20T17:16:44.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:44.308 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:16:44.409 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:16:44.440 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:44.516 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:44 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:16:44.516 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:44 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3480075452' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:16:44.517 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:44 smithi171 ceph-mon[44189]: osdmap e287: 12 total, 12 up, 12 in 2024-04-20T17:16:44.584 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:16:44.590 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:16:44.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:44.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:16:44.606 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:16:44.631 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:16:44.664 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:16:44.684 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:16:44.697 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:16:44.705 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:16:44.726 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:44.727 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:16:44.750 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:16:44.770 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:16:44.790 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:16:44.810 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:16:44.830 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:16:44.831 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:16:44.831 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:16:44.831 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:16:44.832 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:16:44.832 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:16:44.832 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:16:44.832 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:16:44.832 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:16:44.832 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:16:44.832 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:16:44.832 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:16:44.832 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:16:45.176 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:16:45.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:45.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:16:45.280 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:16:45.311 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:45.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:45 smithi171 ceph-mon[44189]: pgmap v796: 17 pgs: 1 active+recovering, 16 active+clean; 4.1 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 161 op/s; 7306/134850 objects misplaced (5.418%); 813 KiB/s, 12 objects/s recovering 2024-04-20T17:16:45.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:45 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2662987695' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:16:45.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:45.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:45.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:45 smithi005 ceph-mon[39562]: pgmap v796: 17 pgs: 1 active+recovering, 16 active+clean; 4.1 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 161 op/s; 7306/134850 objects misplaced (5.418%); 813 KiB/s, 12 objects/s recovering 2024-04-20T17:16:45.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:45 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2662987695' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:16:45.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:45.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:45 smithi106 ceph-mon[44423]: pgmap v796: 17 pgs: 1 active+recovering, 16 active+clean; 4.1 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 161 op/s; 7306/134850 objects misplaced (5.418%); 813 KiB/s, 12 objects/s recovering 2024-04-20T17:16:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:45 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2662987695' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:16:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:45.542 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:16:45.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:45.546 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:16:45.647 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:16:45.674 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:45.693 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:16:45.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:45.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:16:45.798 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:16:45.829 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:45.920 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:16:45.932 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:16:45.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:45.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:16:46.039 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:16:46.066 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:46.085 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:16:46.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:46.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:16:46.189 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:16:46.198 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:16:46.198 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:16:46.198 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 0 2024-04-20T17:16:46.198 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:16:46.217 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:46.243 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:16:46.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:46.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:16:46.347 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:16:46.378 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:46.450 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:16:46.452 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.452 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.452 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.452 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.452 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.452 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.452 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.452 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.452 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.453 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.453 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:46 smithi171 ceph-mon[44189]: osdmap e288: 12 total, 12 up, 12 in 2024-04-20T17:16:46.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:46.454 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:16:46.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:46 smithi005 ceph-mon[39562]: osdmap e288: 12 total, 12 up, 12 in 2024-04-20T17:16:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:46.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:46 smithi106 ceph-mon[44423]: osdmap e288: 12 total, 12 up, 12 in 2024-04-20T17:16:46.554 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:16:46.593 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:46.799 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:16:46.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:46.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:16:46.903 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:16:46.932 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:46.958 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-20T17:16:47.189 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:16:47.242 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:16:47.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:47 smithi171 ceph-mon[44189]: pgmap v798: 17 pgs: 1 active+recovering, 16 active+clean; 4.1 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 138 op/s; 7306/134364 objects misplaced (5.437%); 416 KiB/s, 6 objects/s recovering 2024-04-20T17:16:47.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:47 smithi005 ceph-mon[39562]: pgmap v798: 17 pgs: 1 active+recovering, 16 active+clean; 4.1 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 138 op/s; 7306/134364 objects misplaced (5.437%); 416 KiB/s, 6 objects/s recovering 2024-04-20T17:16:47.483 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:16:47.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:47.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:16:47.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:47 smithi106 ceph-mon[44423]: pgmap v798: 17 pgs: 1 active+recovering, 16 active+clean; 4.1 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 138 op/s; 7306/134364 objects misplaced (5.437%); 416 KiB/s, 6 objects/s recovering 2024-04-20T17:16:47.519 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:16:47.589 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:16:47.621 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:47.642 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:16:47.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:47.646 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:16:47.747 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:16:47.775 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:47.814 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:16:47.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:47.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:16:47.919 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:16:47.951 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:47.965 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:16:48.025 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:16:48.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:48.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:16:48.130 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:16:48.157 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:48.213 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:16:48.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:48.217 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:16:48.318 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:16:48.352 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:48.419 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:16:48.421 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:16:48.431 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:16:48.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:48.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:16:48.536 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:16:48.562 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:48.582 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:16:48.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:48.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:16:48.687 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:16:48.713 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:48.733 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:16:48.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:48.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:16:48.836 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:16:48.867 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:49.260 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:16:49.260 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:16:49.260 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-04-20T17:16:49.260 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-04-20T17:16:49.260 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:16:49.281 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:49 smithi171 ceph-mon[44189]: pgmap v799: 17 pgs: 1 active+recovering, 16 active+clean; 4.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 175 op/s; 7201/133410 objects misplaced (5.398%); 840 KiB/s, 13 objects/s recovering 2024-04-20T17:16:49.281 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:49 smithi171 ceph-mon[44189]: from='client.25396 v1:172.21.15.5:0/4259391224' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:16:49.451 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:49 smithi005 ceph-mon[39562]: pgmap v799: 17 pgs: 1 active+recovering, 16 active+clean; 4.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 175 op/s; 7201/133410 objects misplaced (5.398%); 840 KiB/s, 13 objects/s recovering 2024-04-20T17:16:49.451 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:49 smithi005 ceph-mon[39562]: from='client.25396 v1:172.21.15.5:0/4259391224' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:16:49.484 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":799,"stamp":"2024-04-20T17:16:48.052337+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4371972664,"num_objects":66705,"num_object_clones":0,"num_object_copies":133410,"num_objects_missing_on_primary":3594,"num_objects_missing":3594,"num_objects_degraded":0,"num_objects_misplaced":7201,"num_objects_unfound":0,"num_objects_dirty":66705,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":255462,"num_write_kb":10306363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25617,"num_bytes_recovered":1680427136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160811,"ondisk_log_size":160811,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":10292832,"kb_used_data":9155940,"kb_used_omap":219,"kb_used_meta":1136484,"kb_avail":1114648992,"statfs":{"total":1151940427776,"available":1141400567808,"internally_reserved":0,"allocated":9375682560,"data_stored":9364773062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224949,"internal_metadata":1163759947},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,0,0,1,1,2,2,5,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-92274688,"num_objects":-1408,"num_object_clones":0,"num_object_copies":-2816,"num_objects_missing_on_primary":-106,"num_objects_missing":-106,"num_objects_degraded":0,"num_objects_misplaced":-211,"num_objects_unfound":0,"num_objects_dirty":-1408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":6881280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001492"},"pg_stats":[{"pgid":"2.f","version":"287'15954","reported_seq":16816,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-04-20T17:16:43.569566+0000","last_change":"2024-04-20T17:16:24.907036+0000","last_active":"2024-04-20T17:16:43.569566+0000","last_peered":"2024-04-20T17:16:43.569566+0000","last_clean":"2024-04-20T17:16:43.569566+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:16:43.569566+0000","last_undegraded":"2024-04-20T17:16:43.569566+0000","last_fullsized":"2024-04-20T17:16:43.569566+0000","mapping_epoch":115,"log_start":"135'5900","ondisk_log_start":"135'5900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'15773","last_scrub_stamp":"2024-04-20T17:16:24.906931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:24.906931+0000","objects_scrubbed":4413,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:47.687304+0000","scrub_duration":421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277086208,"num_objects":4228,"num_object_clones":0,"num_object_copies":8456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15954,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"287'15874","reported_seq":27804,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-04-20T17:16:45.213439+0000","last_change":"2024-04-20T17:16:40.290499+0000","last_active":"2024-04-20T17:16:45.213439+0000","last_peered":"2024-04-20T17:16:45.213439+0000","last_clean":"2024-04-20T17:16:45.213439+0000","last_became_active":"2024-04-20T17:10:28.865670+0000","last_became_peered":"2024-04-20T17:10:28.865670+0000","last_unstale":"2024-04-20T17:16:45.213439+0000","last_undegraded":"2024-04-20T17:16:45.213439+0000","last_fullsized":"2024-04-20T17:16:45.213439+0000","mapping_epoch":162,"log_start":"135'5800","ondisk_log_start":"135'5800","created":115,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'15835","last_scrub_stamp":"2024-04-20T17:16:40.290402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:40.290402+0000","objects_scrubbed":4222,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:55.604081+0000","scrub_duration":371,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274071552,"num_objects":4182,"num_object_clones":0,"num_object_copies":8364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15874,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5619,"num_bytes_recovered":368246784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"287'16234","reported_seq":16938,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-04-20T17:16:45.366724+0000","last_change":"2024-04-20T17:15:57.136522+0000","last_active":"2024-04-20T17:16:45.366724+0000","last_peered":"2024-04-20T17:16:45.366724+0000","last_clean":"2024-04-20T17:16:45.366724+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:16:45.366724+0000","last_undegraded":"2024-04-20T17:16:45.366724+0000","last_fullsized":"2024-04-20T17:16:45.366724+0000","mapping_epoch":115,"log_start":"135'6200","ondisk_log_start":"135'6200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'15780","last_scrub_stamp":"2024-04-20T17:15:57.136412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:57.136412+0000","objects_scrubbed":4702,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:10.130229+0000","scrub_duration":368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278396928,"num_objects":4248,"num_object_clones":0,"num_object_copies":8496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16234,"num_write_kb":655424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"287'22593","reported_seq":44098,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-04-20T17:16:45.439378+0000","last_change":"2024-04-20T17:16:35.136358+0000","last_active":"2024-04-20T17:16:45.439378+0000","last_peered":"2024-04-20T17:16:45.439378+0000","last_clean":"2024-04-20T17:16:45.439378+0000","last_became_active":"2024-04-20T17:12:08.085936+0000","last_became_peered":"2024-04-20T17:12:08.085936+0000","last_unstale":"2024-04-20T17:16:45.439378+0000","last_undegraded":"2024-04-20T17:16:45.439378+0000","last_fullsized":"2024-04-20T17:16:45.439378+0000","mapping_epoch":196,"log_start":"166'12568","ondisk_log_start":"166'12568","created":115,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'22494","last_scrub_stamp":"2024-04-20T17:16:35.136177+0000","last_deep_scrub":"213'20373","last_deep_scrub_stamp":"2024-04-20T17:12:48.443808+0000","last_clean_scrub_stamp":"2024-04-20T17:16:35.136177+0000","objects_scrubbed":4393,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:40.491841+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281346048,"num_objects":4293,"num_object_clones":0,"num_object_copies":8586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15941,"num_write_kb":647488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":450166784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"288'15693","reported_seq":33542,"reported_epoch":288,"state":"active+recovering","last_fresh":"2024-04-20T17:16:46.962202+0000","last_change":"2024-04-20T17:12:26.166109+0000","last_active":"2024-04-20T17:16:46.962202+0000","last_peered":"2024-04-20T17:16:46.962202+0000","last_clean":"2024-04-20T17:09:35.882603+0000","last_became_active":"2024-04-20T17:10:54.470253+0000","last_became_peered":"2024-04-20T17:10:54.470253+0000","last_unstale":"2024-04-20T17:16:46.962202+0000","last_undegraded":"2024-04-20T17:16:46.962202+0000","last_fullsized":"2024-04-20T17:16:46.962202+0000","mapping_epoch":171,"log_start":"134'5600","ondisk_log_start":"134'5600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264568832,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":3594,"num_objects_missing":3594,"num_objects_degraded":0,"num_objects_misplaced":7201,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15693,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5825,"num_bytes_recovered":381747200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"0,2,5","objects":13},{"shards":"0,10","objects":430},{"shards":"2,5","objects":3594}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":771,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-20T17:16:45.957909+0000","last_change":"2024-04-20T17:16:40.132382+0000","last_active":"2024-04-20T17:16:45.957909+0000","last_peered":"2024-04-20T17:16:45.957909+0000","last_clean":"2024-04-20T17:16:45.957909+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:16:45.957909+0000","last_undegraded":"2024-04-20T17:16:45.957909+0000","last_fullsized":"2024-04-20T17:16:45.957909+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:16:40.132254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:16:40.132254+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:54.442752+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"288'16025","reported_seq":16762,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-20T17:16:47.705232+0000","last_change":"2024-04-20T17:15:31.396169+0000","last_active":"2024-04-20T17:16:47.705232+0000","last_peered":"2024-04-20T17:16:47.705232+0000","last_clean":"2024-04-20T17:16:47.705232+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:16:47.705232+0000","last_undegraded":"2024-04-20T17:16:47.705232+0000","last_fullsized":"2024-04-20T17:16:47.705232+0000","mapping_epoch":115,"log_start":"135'6000","ondisk_log_start":"135'6000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15333","last_scrub_stamp":"2024-04-20T17:15:31.396069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:31.396069+0000","objects_scrubbed":4919,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:57.811864+0000","scrub_duration":440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277020672,"num_objects":4227,"num_object_clones":0,"num_object_copies":8454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16025,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"287'16126","reported_seq":16861,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-04-20T17:16:43.505864+0000","last_change":"2024-04-20T17:15:35.632022+0000","last_active":"2024-04-20T17:16:43.505864+0000","last_peered":"2024-04-20T17:16:43.505864+0000","last_clean":"2024-04-20T17:16:43.505864+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:16:43.505864+0000","last_undegraded":"2024-04-20T17:16:43.505864+0000","last_fullsized":"2024-04-20T17:16:43.505864+0000","mapping_epoch":115,"log_start":"135'6100","ondisk_log_start":"135'6100","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15459","last_scrub_stamp":"2024-04-20T17:15:35.631842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:35.631842+0000","objects_scrubbed":4867,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:52.716132+0000","scrub_duration":426,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275120128,"num_objects":4198,"num_object_clones":0,"num_object_copies":8396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16126,"num_write_kb":650368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"288'20070","reported_seq":20953,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-20T17:16:46.750292+0000","last_change":"2024-04-20T17:16:25.742067+0000","last_active":"2024-04-20T17:16:46.750292+0000","last_peered":"2024-04-20T17:16:46.750292+0000","last_clean":"2024-04-20T17:16:46.750292+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:16:46.750292+0000","last_undegraded":"2024-04-20T17:16:46.750292+0000","last_fullsized":"2024-04-20T17:16:46.750292+0000","mapping_epoch":115,"log_start":"152'10002","ondisk_log_start":"152'10002","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'19891","last_scrub_stamp":"2024-04-20T17:16:25.741880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:25.741880+0000","objects_scrubbed":4262,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:39.901935+0000","scrub_duration":378,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267583488,"num_objects":4083,"num_object_clones":0,"num_object_copies":8166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15959,"num_write_kb":641344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"288'16125","reported_seq":16917,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-20T17:16:47.478088+0000","last_change":"2024-04-20T17:15:41.950854+0000","last_active":"2024-04-20T17:16:47.478088+0000","last_peered":"2024-04-20T17:16:47.478088+0000","last_clean":"2024-04-20T17:16:47.478088+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:16:47.478088+0000","last_undegraded":"2024-04-20T17:16:47.478088+0000","last_fullsized":"2024-04-20T17:16:47.478088+0000","mapping_epoch":115,"log_start":"135'6100","ondisk_log_start":"135'6100","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'15491","last_scrub_stamp":"2024-04-20T17:15:41.950723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:41.950723+0000","objects_scrubbed":4733,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:58.122837+0000","scrub_duration":448,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268369920,"num_objects":4095,"num_object_clones":0,"num_object_copies":8190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16125,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"288'15933","reported_seq":16653,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-20T17:16:47.654856+0000","last_change":"2024-04-20T17:15:59.551771+0000","last_active":"2024-04-20T17:16:47.654856+0000","last_peered":"2024-04-20T17:16:47.654856+0000","last_clean":"2024-04-20T17:16:47.654856+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:16:47.654856+0000","last_undegraded":"2024-04-20T17:16:47.654856+0000","last_fullsized":"2024-04-20T17:16:47.654856+0000","mapping_epoch":115,"log_start":"135'5900","ondisk_log_start":"135'5900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'15438","last_scrub_stamp":"2024-04-20T17:15:59.551671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:59.551671+0000","objects_scrubbed":4614,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:10.078708+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269942784,"num_objects":4119,"num_object_clones":0,"num_object_copies":8238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15933,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"288'16069","reported_seq":31394,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-20T17:16:47.803903+0000","last_change":"2024-04-20T17:16:34.421711+0000","last_active":"2024-04-20T17:16:47.803903+0000","last_peered":"2024-04-20T17:16:47.803903+0000","last_clean":"2024-04-20T17:16:47.803903+0000","last_became_active":"2024-04-20T17:10:55.931751+0000","last_became_peered":"2024-04-20T17:10:55.931751+0000","last_unstale":"2024-04-20T17:16:47.803903+0000","last_undegraded":"2024-04-20T17:16:47.803903+0000","last_fullsized":"2024-04-20T17:16:47.803903+0000","mapping_epoch":173,"log_start":"135'6000","ondisk_log_start":"135'6000","created":115,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'15941","last_scrub_stamp":"2024-04-20T17:16:34.421587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:34.421587+0000","objects_scrubbed":4322,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:43.906503+0000","scrub_duration":366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274792448,"num_objects":4193,"num_object_clones":0,"num_object_copies":8386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16069,"num_write_kb":648384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7296,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"288'16159","reported_seq":16943,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-20T17:16:47.587436+0000","last_change":"2024-04-20T17:16:42.499318+0000","last_active":"2024-04-20T17:16:47.587436+0000","last_peered":"2024-04-20T17:16:47.587436+0000","last_clean":"2024-04-20T17:16:47.587436+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:16:47.587436+0000","last_undegraded":"2024-04-20T17:16:47.587436+0000","last_fullsized":"2024-04-20T17:16:47.587436+0000","mapping_epoch":115,"log_start":"135'6100","ondisk_log_start":"135'6100","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'16092","last_scrub_stamp":"2024-04-20T17:16:42.499193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:42.499193+0000","objects_scrubbed":4224,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:04.645993+0000","scrub_duration":355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272433152,"num_objects":4157,"num_object_clones":0,"num_object_copies":8314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16159,"num_write_kb":650112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"287'15647","reported_seq":16377,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-04-20T17:16:43.559576+0000","last_change":"2024-04-20T17:15:34.656600+0000","last_active":"2024-04-20T17:16:43.559576+0000","last_peered":"2024-04-20T17:16:43.559576+0000","last_clean":"2024-04-20T17:16:43.559576+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:16:43.559576+0000","last_undegraded":"2024-04-20T17:16:43.559576+0000","last_fullsized":"2024-04-20T17:16:43.559576+0000","mapping_epoch":115,"log_start":"135'5600","ondisk_log_start":"135'5600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15023","last_scrub_stamp":"2024-04-20T17:15:34.656503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:34.656503+0000","objects_scrubbed":4842,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:51.197304+0000","scrub_duration":430,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276365312,"num_objects":4217,"num_object_clones":0,"num_object_copies":8434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15647,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"288'16003","reported_seq":16735,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-20T17:16:47.558787+0000","last_change":"2024-04-20T17:15:31.794340+0000","last_active":"2024-04-20T17:16:47.558787+0000","last_peered":"2024-04-20T17:16:47.558787+0000","last_clean":"2024-04-20T17:16:47.558787+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:16:47.558787+0000","last_undegraded":"2024-04-20T17:16:47.558787+0000","last_fullsized":"2024-04-20T17:16:47.558787+0000","mapping_epoch":115,"log_start":"135'6000","ondisk_log_start":"135'6000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15271","last_scrub_stamp":"2024-04-20T17:15:31.794156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:31.794156+0000","objects_scrubbed":4831,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:58.805155+0000","scrub_duration":443,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268632064,"num_objects":4099,"num_object_clones":0,"num_object_copies":8198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16003,"num_write_kb":643264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"287'15730","reported_seq":16460,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-04-20T17:16:45.395463+0000","last_change":"2024-04-20T17:15:33.298229+0000","last_active":"2024-04-20T17:16:45.395463+0000","last_peered":"2024-04-20T17:16:45.395463+0000","last_clean":"2024-04-20T17:16:45.395463+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:16:45.395463+0000","last_undegraded":"2024-04-20T17:16:45.395463+0000","last_fullsized":"2024-04-20T17:16:45.395463+0000","mapping_epoch":115,"log_start":"135'5700","ondisk_log_start":"135'5700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15067","last_scrub_stamp":"2024-04-20T17:15:33.298103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:33.298103+0000","objects_scrubbed":4809,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:56.012242+0000","scrub_duration":458,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271253528,"num_objects":4140,"num_object_clones":0,"num_object_copies":8280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4140,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15730,"num_write_kb":635777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"287'15809","reported_seq":16459,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-04-20T17:16:45.863656+0000","last_change":"2024-04-20T17:16:23.593408+0000","last_active":"2024-04-20T17:16:45.863656+0000","last_peered":"2024-04-20T17:16:45.863656+0000","last_clean":"2024-04-20T17:16:45.863656+0000","last_became_active":"2024-04-20T17:12:23.515375+0000","last_became_peered":"2024-04-20T17:12:23.515375+0000","last_unstale":"2024-04-20T17:16:45.863656+0000","last_undegraded":"2024-04-20T17:16:45.863656+0000","last_fullsized":"2024-04-20T17:16:45.863656+0000","mapping_epoch":202,"log_start":"135'5800","ondisk_log_start":"135'5800","created":115,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'15586","last_scrub_stamp":"2024-04-20T17:16:23.593300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:23.593300+0000","objects_scrubbed":4410,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:33.363583+0000","scrub_duration":446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274399232,"num_objects":4187,"num_object_clones":0,"num_object_copies":8374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15809,"num_write_kb":639872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":4371382296,"num_objects":66703,"num_object_clones":0,"num_object_copies":133406,"num_objects_missing_on_primary":3594,"num_objects_missing":3594,"num_objects_degraded":0,"num_objects_misplaced":7201,"num_objects_unfound":0,"num_objects_dirty":66703,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":255281,"num_write_kb":10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25609,"num_bytes_recovered":1678311424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9334366208,"data_stored":9334358064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},"log_size":160674,"ondisk_log_size":160674,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991365,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":866876,"kb_used_data":799600,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92878276,"statfs":{"total":95995035648,"available":95107354624,"internally_reserved":0,"allocated":818790400,"data_stored":817875490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":68876740},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253002,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":988392,"kb_used_data":840604,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92756760,"statfs":{"total":95995035648,"available":94982922240,"internally_reserved":0,"allocated":860778496,"data_stored":859873116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":151321024},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612752,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1227116,"kb_used_data":1079328,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92518036,"statfs":{"total":95995035648,"available":94738468864,"internally_reserved":0,"allocated":1105231872,"data_stored":1104322440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":151321024},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762808981,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1181672,"kb_used_data":1074140,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92563480,"statfs":{"total":95995035648,"available":94785003520,"internally_reserved":0,"allocated":1099919360,"data_stored":1099014000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168731,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":882488,"kb_used_data":806896,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92862664,"statfs":{"total":95995035648,"available":95091367936,"internally_reserved":0,"allocated":826261504,"data_stored":825346614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430367,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":646772,"kb_used_data":539240,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93098380,"statfs":{"total":95995035648,"available":95332741120,"internally_reserved":0,"allocated":552181760,"data_stored":551270422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575033,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":620660,"kb_used_data":535336,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93124492,"statfs":{"total":95995035648,"available":95359479808,"internally_reserved":0,"allocated":548184064,"data_stored":547272726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738558,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":962996,"kb_used_data":827688,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":92782156,"statfs":{"total":95995035648,"available":95008927744,"internally_reserved":0,"allocated":847552512,"data_stored":846641174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,0,1,1,2,2,5,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393846,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":748772,"kb_used_data":700380,"kb_used_omap":202,"kb_used_meta":48181,"kb_avail":92996380,"statfs":{"total":95995035648,"available":95228293120,"internally_reserved":0,"allocated":717189120,"data_stored":716283760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":49337815},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182574,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":610148,"kb_used_data":538652,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93135004,"statfs":{"total":95995035648,"available":95370244096,"internally_reserved":0,"allocated":551579648,"data_stored":550674288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986345,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307176,"kb_used_data":267680,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93437976,"statfs":{"total":95995035648,"available":95680487424,"internally_reserved":0,"allocated":274104320,"data_stored":273194888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790115,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1249764,"kb_used_data":1146396,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92495388,"statfs":{"total":95995035648,"available":94715277312,"internally_reserved":0,"allocated":1173909504,"data_stored":1173004144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":844365824,"data_stored":844365824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":545521664,"data_stored":545521664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":714604544,"data_stored":714604544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":548208640,"data_stored":548208640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":271257600,"data_stored":271253528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1170866176,"data_stored":1170866176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":548077568,"data_stored":548077568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":821755904,"data_stored":821755904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1097859072,"data_stored":1097859072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1100156928,"data_stored":1100152856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":857407488,"data_stored":857407488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":814284800,"data_stored":814284800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:16:49.485 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:16:49.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:49 smithi106 ceph-mon[44423]: pgmap v799: 17 pgs: 1 active+recovering, 16 active+clean; 4.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 175 op/s; 7201/133410 objects misplaced (5.398%); 840 KiB/s, 13 objects/s recovering 2024-04-20T17:16:49.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:49 smithi106 ceph-mon[44423]: from='client.25396 v1:172.21.15.5:0/4259391224' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:16:50.042 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-20T17:16:50.314 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:16:51.006 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:16:51.018 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:16:51.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:51.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:16:51.126 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:16:51.154 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:51.175 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:16:51.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:51.178 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:16:51.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:51 smithi005 ceph-mon[39562]: pgmap v800: 17 pgs: 1 active+recovering, 16 active+clean; 4.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 154 op/s; 7201/133078 objects misplaced (5.411%); 424 KiB/s, 6 objects/s recovering 2024-04-20T17:16:51.279 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:16:51.313 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:51.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:51 smithi106 ceph-mon[44423]: pgmap v800: 17 pgs: 1 active+recovering, 16 active+clean; 4.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 154 op/s; 7201/133078 objects misplaced (5.411%); 424 KiB/s, 6 objects/s recovering 2024-04-20T17:16:51.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:51 smithi171 ceph-mon[44189]: pgmap v800: 17 pgs: 1 active+recovering, 16 active+clean; 4.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 154 op/s; 7201/133078 objects misplaced (5.411%); 424 KiB/s, 6 objects/s recovering 2024-04-20T17:16:52.133 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:16:52.165 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:16:52.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:52.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:16:52.271 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:16:52.303 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:52.387 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:16:52.387 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:16:52.387 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:16:52.387 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:16:52.387 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.388 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:138161 2.5 2:a4f53bd8:::benchmark_data_smithi171_78918_object138160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-04-20T17:16:52.388 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:09:22.723185+0000", 2024-04-20T17:16:52.388 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 449.66215145000001, 2024-04-20T17:16:52.388 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.037369641000000002, 2024-04-20T17:16:52.388 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:16:52.388 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:16:52.388 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:52.388 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:16:52.388 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:16:52.388 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:16:52.388 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 138161 2024-04-20T17:16:52.388 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.389 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:16:52.389 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.389 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:16:52.389 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.723185+0000", 2024-04-20T17:16:52.389 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.389 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.389 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.389 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:16:52.389 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.723185+0000", 2024-04-20T17:16:52.389 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.389 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.389 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.389 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:16:52.389 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.723186+0000", 2024-04-20T17:16:52.390 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4000000000000001e-07 2024-04-20T17:16:52.390 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.390 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.390 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:16:52.390 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.723231+0000", 2024-04-20T17:16:52.390 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5262999999999997e-05 2024-04-20T17:16:52.390 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.390 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.390 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:16:52.390 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.723240+0000", 2024-04-20T17:16:52.390 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9649999999999997e-06 2024-04-20T17:16:52.390 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.391 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.391 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:16:52.391 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.723246+0000", 2024-04-20T17:16:52.391 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.207e-06 2024-04-20T17:16:52.391 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.391 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.391 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:16:52.391 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.727449+0000", 2024-04-20T17:16:52.391 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0042026570000000003 2024-04-20T17:16:52.391 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.391 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.391 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:16:52.391 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.727582+0000", 2024-04-20T17:16:52.391 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013308600000000001 2024-04-20T17:16:52.392 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.392 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.392 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:16:52.392 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.727720+0000", 2024-04-20T17:16:52.392 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000138348 2024-04-20T17:16:52.392 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.392 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.392 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:16:52.392 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.731368+0000", 2024-04-20T17:16:52.392 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0036483380000000001 2024-04-20T17:16:52.392 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.392 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.392 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:16:52.392 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.760485+0000", 2024-04-20T17:16:52.392 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.029116454 2024-04-20T17:16:52.393 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.393 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.393 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:16:52.393 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.760520+0000", 2024-04-20T17:16:52.393 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5349999999999999e-05 2024-04-20T17:16:52.393 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.393 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.393 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:16:52.393 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:09:22.760554+0000", 2024-04-20T17:16:52.393 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4233000000000002e-05 2024-04-20T17:16:52.393 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.393 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:16:52.393 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.393 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.394 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.394 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:195841 2.5 2:a970f203:::benchmark_data_smithi171_78918_object34847:head [delete] snapc 0=[] ondisk+write+known_if_redirected e163)", 2024-04-20T17:16:52.394 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:30.697234+0000", 2024-04-20T17:16:52.394 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 381.68810198400001, 2024-04-20T17:16:52.394 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.037855572999999997, 2024-04-20T17:16:52.394 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:16:52.394 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:16:52.394 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:52.394 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:16:52.394 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:16:52.394 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:16:52.394 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 195841 2024-04-20T17:16:52.394 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.394 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:16:52.394 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.395 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:16:52.395 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.697234+0000", 2024-04-20T17:16:52.395 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.395 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.395 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.395 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:16:52.395 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.697234+0000", 2024-04-20T17:16:52.395 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.395 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.395 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.395 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:16:52.395 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.697235+0000", 2024-04-20T17:16:52.395 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1939999999999999e-06 2024-04-20T17:16:52.395 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.396 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.396 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:16:52.396 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.697239+0000", 2024-04-20T17:16:52.396 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0589999999999996e-06 2024-04-20T17:16:52.396 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.396 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.396 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:16:52.396 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.697247+0000", 2024-04-20T17:16:52.396 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5909999999999998e-06 2024-04-20T17:16:52.396 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.396 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.396 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:16:52.396 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.697254+0000", 2024-04-20T17:16:52.396 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5760000000000002e-06 2024-04-20T17:16:52.396 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.397 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.397 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:16:52.397 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.701087+0000", 2024-04-20T17:16:52.397 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.003833192 2024-04-20T17:16:52.397 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.397 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.397 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:16:52.397 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.701179+0000", 2024-04-20T17:16:52.397 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2037e-05 2024-04-20T17:16:52.397 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.397 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.397 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:16:52.397 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.701266+0000", 2024-04-20T17:16:52.397 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6891999999999994e-05 2024-04-20T17:16:52.397 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.398 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.398 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:16:52.398 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.704805+0000", 2024-04-20T17:16:52.398 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0035387330000000001 2024-04-20T17:16:52.398 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.398 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.398 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:16:52.398 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.735014+0000", 2024-04-20T17:16:52.398 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030209192999999999 2024-04-20T17:16:52.398 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.398 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.398 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:16:52.398 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.735056+0000", 2024-04-20T17:16:52.398 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2469000000000003e-05 2024-04-20T17:16:52.399 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.399 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.399 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:16:52.399 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:30.735090+0000", 2024-04-20T17:16:52.399 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3637000000000003e-05 2024-04-20T17:16:52.399 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.399 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:16:52.399 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.399 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.399 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.399 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:202267 2.5 2:abe7d7a2:::benchmark_data_smithi171_78918_object41273:head [delete] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-04-20T17:16:52.399 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:39.807427+0000", 2024-04-20T17:16:52.399 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 372.57790939900002, 2024-04-20T17:16:52.399 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.038790863000000002, 2024-04-20T17:16:52.399 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:16:52.400 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:16:52.400 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:52.400 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:16:52.400 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:16:52.400 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:16:52.400 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 202267 2024-04-20T17:16:52.400 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.400 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:16:52.400 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.400 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:16:52.400 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.807427+0000", 2024-04-20T17:16:52.400 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.400 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.400 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.400 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:16:52.400 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.807427+0000", 2024-04-20T17:16:52.401 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.401 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.401 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.401 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:16:52.401 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.807429+0000", 2024-04-20T17:16:52.401 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0959999999999999e-06 2024-04-20T17:16:52.401 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.401 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.401 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:16:52.401 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.807435+0000", 2024-04-20T17:16:52.401 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9499999999999998e-06 2024-04-20T17:16:52.401 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.401 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.401 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:16:52.401 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.807447+0000", 2024-04-20T17:16:52.402 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1654000000000001e-05 2024-04-20T17:16:52.402 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.402 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.402 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:16:52.402 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.807454+0000", 2024-04-20T17:16:52.402 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2790000000000001e-06 2024-04-20T17:16:52.402 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.402 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.402 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:16:52.402 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.807508+0000", 2024-04-20T17:16:52.402 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4370999999999998e-05 2024-04-20T17:16:52.402 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.402 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.402 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:16:52.403 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.807648+0000", 2024-04-20T17:16:52.403 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000139734 2024-04-20T17:16:52.403 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.403 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.403 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:16:52.403 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.807766+0000", 2024-04-20T17:16:52.403 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000118279 2024-04-20T17:16:52.403 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.403 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.403 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:16:52.403 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.811542+0000", 2024-04-20T17:16:52.403 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0037755639999999999 2024-04-20T17:16:52.403 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.403 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.403 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:16:52.404 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.846127+0000", 2024-04-20T17:16:52.404 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.034585171999999997 2024-04-20T17:16:52.404 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.404 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.404 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:16:52.404 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.846164+0000", 2024-04-20T17:16:52.404 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6671999999999997e-05 2024-04-20T17:16:52.404 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.404 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.404 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:16:52.404 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.846218+0000", 2024-04-20T17:16:52.404 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4092000000000003e-05 2024-04-20T17:16:52.404 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.404 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:16:52.404 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.405 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.405 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.405 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:205502 2.b 2:da81e759:::benchmark_data_smithi171_78918_object44508:head [delete] snapc 0=[] ondisk+write+known_if_redirected e167)", 2024-04-20T17:16:52.405 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:43.228493+0000", 2024-04-20T17:16:52.405 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 369.15684295599999, 2024-04-20T17:16:52.405 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036099956000000002, 2024-04-20T17:16:52.405 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:16:52.405 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:16:52.405 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:52.405 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:16:52.405 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:16:52.405 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:16:52.405 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 205502 2024-04-20T17:16:52.405 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.406 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:16:52.406 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.406 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:16:52.406 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.228493+0000", 2024-04-20T17:16:52.406 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.406 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.406 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.406 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:16:52.406 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.228493+0000", 2024-04-20T17:16:52.406 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.406 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.406 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.406 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:16:52.406 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.228495+0000", 2024-04-20T17:16:52.407 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5370000000000001e-06 2024-04-20T17:16:52.407 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.407 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.407 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:16:52.407 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.228501+0000", 2024-04-20T17:16:52.407 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8939999999999998e-06 2024-04-20T17:16:52.407 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.407 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.407 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:16:52.407 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.228510+0000", 2024-04-20T17:16:52.407 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6509999999999996e-06 2024-04-20T17:16:52.407 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.407 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.407 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:16:52.408 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.228519+0000", 2024-04-20T17:16:52.408 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6600000000000001e-06 2024-04-20T17:16:52.408 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.408 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.408 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:16:52.408 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.245447+0000", 2024-04-20T17:16:52.408 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.016928144999999999 2024-04-20T17:16:52.408 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.408 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.408 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:16:52.408 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.245548+0000", 2024-04-20T17:16:52.408 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010089 2024-04-20T17:16:52.408 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.408 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.408 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:16:52.409 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.245644+0000", 2024-04-20T17:16:52.409 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6254000000000004e-05 2024-04-20T17:16:52.409 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.409 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.409 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:16:52.409 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.260040+0000", 2024-04-20T17:16:52.409 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014396153 2024-04-20T17:16:52.409 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.409 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.409 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:16:52.409 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.264494+0000", 2024-04-20T17:16:52.409 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0044534120000000003 2024-04-20T17:16:52.409 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.409 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.409 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:16:52.410 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.264544+0000", 2024-04-20T17:16:52.410 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0451000000000002e-05 2024-04-20T17:16:52.410 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.410 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.410 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:16:52.410 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:43.264593+0000", 2024-04-20T17:16:52.410 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8909000000000002e-05 2024-04-20T17:16:52.410 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.410 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:16:52.410 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.410 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.410 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.410 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:31842 2.6 e210/115 2:609476f8:::benchmark_data_smithi171_78918_object64933:head v 210'13806, mlcod=210'13795)", 2024-04-20T17:16:52.410 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.322569+0000", 2024-04-20T17:16:52.410 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 251.062767033, 2024-04-20T17:16:52.411 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.038410374999999997, 2024-04-20T17:16:52.411 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:16:52.411 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:16:52.411 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:52.411 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:16:52.411 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.411 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:16:52.411 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.322569+0000", 2024-04-20T17:16:52.411 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.411 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.411 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.411 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:16:52.411 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.322569+0000", 2024-04-20T17:16:52.411 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.411 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.412 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.412 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:16:52.412 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.322571+0000", 2024-04-20T17:16:52.412 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6929999999999999e-06 2024-04-20T17:16:52.412 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.412 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.412 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:16:52.412 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.322578+0000", 2024-04-20T17:16:52.412 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4370000000000001e-06 2024-04-20T17:16:52.412 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.412 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.412 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:16:52.412 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.322589+0000", 2024-04-20T17:16:52.412 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0548e-05 2024-04-20T17:16:52.412 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.413 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.413 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:16:52.413 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.322596+0000", 2024-04-20T17:16:52.413 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0090000000000001e-06 2024-04-20T17:16:52.413 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.413 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.413 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:16:52.413 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.334018+0000", 2024-04-20T17:16:52.413 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011421751000000001 2024-04-20T17:16:52.413 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.413 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.413 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:16:52.413 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.334039+0000", 2024-04-20T17:16:52.413 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1721000000000001e-05 2024-04-20T17:16:52.413 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.413 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.414 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:16:52.414 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.360942+0000", 2024-04-20T17:16:52.414 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026902463000000001 2024-04-20T17:16:52.414 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.414 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.414 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:16:52.414 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.360980+0000", 2024-04-20T17:16:52.414 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7753000000000003e-05 2024-04-20T17:16:52.414 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.414 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:16:52.414 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.414 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.414 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.415 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:31847 2.6 e210/115 2:6097b0e2:::benchmark_data_smithi171_78918_object71834:head v 210'13811, mlcod=210'13795)", 2024-04-20T17:16:52.415 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.341065+0000", 2024-04-20T17:16:52.415 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 251.04427128899999, 2024-04-20T17:16:52.415 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.038246251000000002, 2024-04-20T17:16:52.415 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:16:52.415 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:16:52.415 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:52.415 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:16:52.415 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.415 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:16:52.415 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.341065+0000", 2024-04-20T17:16:52.415 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.415 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.415 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.415 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:16:52.416 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.341065+0000", 2024-04-20T17:16:52.416 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.416 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.416 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.416 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:16:52.416 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.341066+0000", 2024-04-20T17:16:52.416 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5600000000000001e-06 2024-04-20T17:16:52.416 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.416 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.416 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:16:52.416 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.341075+0000", 2024-04-20T17:16:52.416 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2979999999999992e-06 2024-04-20T17:16:52.416 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.416 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.416 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:16:52.417 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.341091+0000", 2024-04-20T17:16:52.417 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5715000000000001e-05 2024-04-20T17:16:52.417 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.417 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.417 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:16:52.417 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.341102+0000", 2024-04-20T17:16:52.417 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1224e-05 2024-04-20T17:16:52.417 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.417 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.417 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:16:52.417 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.357362+0000", 2024-04-20T17:16:52.417 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.016260406000000002 2024-04-20T17:16:52.417 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.417 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.418 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:16:52.418 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.357386+0000", 2024-04-20T17:16:52.418 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3601999999999999e-05 2024-04-20T17:16:52.418 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.418 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.418 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:16:52.418 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.379270+0000", 2024-04-20T17:16:52.418 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021883969999999999 2024-04-20T17:16:52.418 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.418 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.418 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:16:52.418 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.379311+0000", 2024-04-20T17:16:52.418 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1476000000000001e-05 2024-04-20T17:16:52.418 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.419 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:16:52.419 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.419 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.419 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.419 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:31849 2.6 e210/115 2:609a0ddc:::benchmark_data_smithi171_78918_object159331:head v 210'13813, mlcod=210'13795)", 2024-04-20T17:16:52.419 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.347350+0000", 2024-04-20T17:16:52.419 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 251.03798572900001, 2024-04-20T17:16:52.419 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.039795965000000003, 2024-04-20T17:16:52.419 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:16:52.419 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:16:52.419 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:52.419 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:16:52.419 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.419 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:16:52.419 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.347350+0000", 2024-04-20T17:16:52.420 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.420 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.420 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.420 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:16:52.420 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.347350+0000", 2024-04-20T17:16:52.420 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.420 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.420 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.420 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:16:52.420 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.347352+0000", 2024-04-20T17:16:52.420 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4300000000000001e-06 2024-04-20T17:16:52.420 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.420 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.420 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:16:52.420 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.347361+0000", 2024-04-20T17:16:52.421 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3649999999999993e-06 2024-04-20T17:16:52.421 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.421 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.421 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:16:52.421 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.347371+0000", 2024-04-20T17:16:52.421 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7920000000000004e-06 2024-04-20T17:16:52.421 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.421 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.421 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:16:52.421 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.347378+0000", 2024-04-20T17:16:52.421 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7499999999999997e-06 2024-04-20T17:16:52.421 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.421 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.421 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:16:52.421 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.365885+0000", 2024-04-20T17:16:52.422 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018507132999999999 2024-04-20T17:16:52.422 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.422 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.422 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:16:52.422 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.365910+0000", 2024-04-20T17:16:52.422 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5338999999999999e-05 2024-04-20T17:16:52.422 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.422 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.422 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:16:52.422 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.387117+0000", 2024-04-20T17:16:52.422 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021207085000000001 2024-04-20T17:16:52.422 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.422 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.422 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:16:52.423 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.387146+0000", 2024-04-20T17:16:52.423 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9071e-05 2024-04-20T17:16:52.423 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.423 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:16:52.423 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.423 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.423 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.423 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:31850 2.6 e210/115 2:609a3392:::benchmark_data_smithi171_78918_object78297:head v 210'13814, mlcod=210'13795)", 2024-04-20T17:16:52.423 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.351057+0000", 2024-04-20T17:16:52.423 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 251.034279449, 2024-04-20T17:16:52.423 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036121317, 2024-04-20T17:16:52.423 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:16:52.423 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:16:52.423 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:52.423 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:16:52.424 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.424 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:16:52.424 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.351057+0000", 2024-04-20T17:16:52.424 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.424 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.424 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.424 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:16:52.424 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.351057+0000", 2024-04-20T17:16:52.424 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.424 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.424 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.424 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:16:52.424 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.351058+0000", 2024-04-20T17:16:52.424 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.652e-06 2024-04-20T17:16:52.424 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.425 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.425 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:16:52.425 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.351068+0000", 2024-04-20T17:16:52.425 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9760000000000007e-06 2024-04-20T17:16:52.425 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.425 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.425 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:16:52.425 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.351083+0000", 2024-04-20T17:16:52.425 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4934e-05 2024-04-20T17:16:52.425 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.425 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.425 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:16:52.425 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.351092+0000", 2024-04-20T17:16:52.425 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2069999999999995e-06 2024-04-20T17:16:52.425 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.425 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.426 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:16:52.426 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.370433+0000", 2024-04-20T17:16:52.426 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019341862000000001 2024-04-20T17:16:52.426 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.426 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.426 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:16:52.426 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.370455+0000", 2024-04-20T17:16:52.426 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1469000000000001e-05 2024-04-20T17:16:52.426 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.426 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.426 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:16:52.426 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.387150+0000", 2024-04-20T17:16:52.426 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.016695254999999999 2024-04-20T17:16:52.426 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.426 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.427 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:16:52.427 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.387178+0000", 2024-04-20T17:16:52.427 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7962000000000001e-05 2024-04-20T17:16:52.427 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.427 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:16:52.427 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.427 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.427 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.427 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:31851 2.6 e210/115 2:609a34ba:::benchmark_data_smithi171_78918_object151524:head v 210'13815, mlcod=210'13795)", 2024-04-20T17:16:52.427 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.355231+0000", 2024-04-20T17:16:52.427 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 251.030104747, 2024-04-20T17:16:52.427 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.045050494000000003, 2024-04-20T17:16:52.427 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:16:52.427 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:16:52.427 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:52.428 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:16:52.428 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.428 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:16:52.428 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.355231+0000", 2024-04-20T17:16:52.428 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.428 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.428 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.428 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:16:52.428 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.355231+0000", 2024-04-20T17:16:52.428 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.428 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.428 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.428 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:16:52.428 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.355233+0000", 2024-04-20T17:16:52.428 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.902e-06 2024-04-20T17:16:52.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.429 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.429 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:16:52.429 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.355241+0000", 2024-04-20T17:16:52.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8359999999999992e-06 2024-04-20T17:16:52.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.429 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.429 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:16:52.429 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.355254+0000", 2024-04-20T17:16:52.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2542e-05 2024-04-20T17:16:52.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.429 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.429 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:16:52.429 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.355263+0000", 2024-04-20T17:16:52.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.7790000000000001e-06 2024-04-20T17:16:52.430 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.430 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.430 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:16:52.430 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.374844+0000", 2024-04-20T17:16:52.430 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019581365999999999 2024-04-20T17:16:52.430 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.430 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.430 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:16:52.430 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.374869+0000", 2024-04-20T17:16:52.430 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5092e-05 2024-04-20T17:16:52.430 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.430 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.430 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:16:52.430 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.400241+0000", 2024-04-20T17:16:52.431 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.025372098999999999 2024-04-20T17:16:52.431 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.431 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.431 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:16:52.431 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.400282+0000", 2024-04-20T17:16:52.431 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0877999999999997e-05 2024-04-20T17:16:52.431 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.431 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:16:52.431 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.431 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.431 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.431 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:31852 2.6 e210/115 2:609bb098:::benchmark_data_smithi171_78918_object152231:head v 210'13816, mlcod=210'13795)", 2024-04-20T17:16:52.431 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.359184+0000", 2024-04-20T17:16:52.431 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 251.02615255800001, 2024-04-20T17:16:52.431 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.041126272999999998, 2024-04-20T17:16:52.432 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:16:52.432 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:16:52.432 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:52.432 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:16:52.432 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.432 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:16:52.432 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.359184+0000", 2024-04-20T17:16:52.432 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.432 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.432 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.432 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:16:52.432 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.359184+0000", 2024-04-20T17:16:52.432 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.432 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.432 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.433 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:16:52.433 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.359188+0000", 2024-04-20T17:16:52.433 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8890000000000004e-06 2024-04-20T17:16:52.433 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.433 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.433 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:16:52.433 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.359196+0000", 2024-04-20T17:16:52.433 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9529999999999999e-06 2024-04-20T17:16:52.433 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.433 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.433 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:16:52.433 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.359212+0000", 2024-04-20T17:16:52.433 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6718000000000002e-05 2024-04-20T17:16:52.433 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.434 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.434 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:16:52.434 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.359220+0000", 2024-04-20T17:16:52.434 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8420000000000005e-06 2024-04-20T17:16:52.434 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.434 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.434 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:16:52.434 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.379386+0000", 2024-04-20T17:16:52.434 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020166119999999999 2024-04-20T17:16:52.434 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.434 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.434 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:16:52.434 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.379414+0000", 2024-04-20T17:16:52.434 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8072000000000002e-05 2024-04-20T17:16:52.434 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.435 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.435 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:16:52.435 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.400286+0000", 2024-04-20T17:16:52.435 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020871654 2024-04-20T17:16:52.435 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.435 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.435 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:16:52.435 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.400310+0000", 2024-04-20T17:16:52.435 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4025000000000001e-05 2024-04-20T17:16:52.435 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.435 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:16:52.435 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.435 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.435 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.435 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:31853 2.6 e210/115 2:609c5ccc:::benchmark_data_smithi171_78918_object127897:head v 210'13817, mlcod=210'13795)", 2024-04-20T17:16:52.436 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.363464+0000", 2024-04-20T17:16:52.436 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 251.02187214400001, 2024-04-20T17:16:52.436 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.048461072000000001, 2024-04-20T17:16:52.436 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:16:52.436 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:16:52.436 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:52.436 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:16:52.436 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.436 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:16:52.436 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.363464+0000", 2024-04-20T17:16:52.436 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.436 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.436 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.436 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:16:52.436 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.363464+0000", 2024-04-20T17:16:52.437 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.437 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.437 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.437 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:16:52.437 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.363466+0000", 2024-04-20T17:16:52.437 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.818e-06 2024-04-20T17:16:52.437 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.437 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.437 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:16:52.437 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.363474+0000", 2024-04-20T17:16:52.437 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4919999999999993e-06 2024-04-20T17:16:52.437 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.437 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.437 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:16:52.437 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.363485+0000", 2024-04-20T17:16:52.438 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1046999999999999e-05 2024-04-20T17:16:52.438 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.438 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.438 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:16:52.438 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.363494+0000", 2024-04-20T17:16:52.438 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6130000000000004e-06 2024-04-20T17:16:52.438 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.438 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.438 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:16:52.438 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.383962+0000", 2024-04-20T17:16:52.438 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020467928999999999 2024-04-20T17:16:52.438 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.438 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.438 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:16:52.438 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.383979+0000", 2024-04-20T17:16:52.439 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7142e-05 2024-04-20T17:16:52.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.439 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.439 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:16:52.439 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.411881+0000", 2024-04-20T17:16:52.439 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.027902278999999999 2024-04-20T17:16:52.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.439 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.439 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:16:52.439 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.411925+0000", 2024-04-20T17:16:52.439 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3751999999999997e-05 2024-04-20T17:16:52.439 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.439 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:16:52.439 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.439 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.440 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:31854 2.6 e210/115 2:609ceadc:::benchmark_data_smithi171_78918_object160601:head v 210'13818, mlcod=210'13795)", 2024-04-20T17:16:52.440 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.367804+0000", 2024-04-20T17:16:52.440 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 251.01753206000001, 2024-04-20T17:16:52.440 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044152018000000001, 2024-04-20T17:16:52.440 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:16:52.440 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:16:52.440 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:52.440 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:16:52.440 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.440 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:16:52.440 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.367804+0000", 2024-04-20T17:16:52.440 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.440 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.441 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:16:52.441 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.367804+0000", 2024-04-20T17:16:52.441 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.441 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:16:52.441 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.367806+0000", 2024-04-20T17:16:52.441 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.516e-06 2024-04-20T17:16:52.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.441 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:16:52.441 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.367814+0000", 2024-04-20T17:16:52.441 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9419999999999995e-06 2024-04-20T17:16:52.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.441 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:16:52.442 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.367823+0000", 2024-04-20T17:16:52.442 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.8199999999999992e-06 2024-04-20T17:16:52.442 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.442 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.442 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:16:52.442 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.367831+0000", 2024-04-20T17:16:52.442 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.418e-06 2024-04-20T17:16:52.442 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.442 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.442 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:16:52.442 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.387226+0000", 2024-04-20T17:16:52.442 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019395580999999999 2024-04-20T17:16:52.442 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.442 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.443 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:16:52.443 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.387251+0000", 2024-04-20T17:16:52.443 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4610999999999999e-05 2024-04-20T17:16:52.443 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.443 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.443 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:16:52.443 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.411929+0000", 2024-04-20T17:16:52.443 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024677999999999999 2024-04-20T17:16:52.443 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.443 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.443 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:16:52.443 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.411956+0000", 2024-04-20T17:16:52.443 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7129999999999999e-05 2024-04-20T17:16:52.443 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.443 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:16:52.444 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.444 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.444 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.444 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:31855 2.6 e210/115 2:609d09da:::benchmark_data_smithi171_78918_object118890:head v 210'13819, mlcod=210'13795)", 2024-04-20T17:16:52.444 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.372439+0000", 2024-04-20T17:16:52.444 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 251.01289703800001, 2024-04-20T17:16:52.444 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.039545667, 2024-04-20T17:16:52.444 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:16:52.444 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:16:52.444 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:52.444 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:16:52.444 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.444 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:16:52.444 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.372439+0000", 2024-04-20T17:16:52.444 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.445 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:16:52.445 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.372439+0000", 2024-04-20T17:16:52.445 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.445 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:16:52.445 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.372441+0000", 2024-04-20T17:16:52.445 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4839999999999999e-06 2024-04-20T17:16:52.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.445 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:16:52.445 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.372449+0000", 2024-04-20T17:16:52.445 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9349999999999994e-06 2024-04-20T17:16:52.446 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.446 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.446 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:16:52.446 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.372456+0000", 2024-04-20T17:16:52.446 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1110000000000002e-06 2024-04-20T17:16:52.446 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.446 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.446 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:16:52.446 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.372461+0000", 2024-04-20T17:16:52.446 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.711e-06 2024-04-20T17:16:52.446 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.446 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.446 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:16:52.446 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.391672+0000", 2024-04-20T17:16:52.447 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019210820999999999 2024-04-20T17:16:52.447 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.447 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.447 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:16:52.447 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.391694+0000", 2024-04-20T17:16:52.447 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2045999999999999e-05 2024-04-20T17:16:52.447 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.447 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.447 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:16:52.447 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.411959+0000", 2024-04-20T17:16:52.447 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020264379999999999 2024-04-20T17:16:52.447 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.447 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.447 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:16:52.447 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.411985+0000", 2024-04-20T17:16:52.448 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6179000000000001e-05 2024-04-20T17:16:52.448 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.448 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:16:52.448 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.448 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.448 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.448 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:31984 2.6 e210/115 2:60e80bc9:::benchmark_data_smithi171_78918_object105296:head v 210'13948, mlcod=210'13930)", 2024-04-20T17:16:52.448 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.957574+0000", 2024-04-20T17:16:52.448 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 250.427762282, 2024-04-20T17:16:52.448 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.039531605999999997, 2024-04-20T17:16:52.448 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:16:52.448 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:16:52.448 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:52.448 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:16:52.448 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.449 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:16:52.449 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.957574+0000", 2024-04-20T17:16:52.449 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.449 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.449 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.449 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:16:52.449 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.957574+0000", 2024-04-20T17:16:52.449 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.449 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.449 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.449 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:16:52.449 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.957575+0000", 2024-04-20T17:16:52.449 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.252e-06 2024-04-20T17:16:52.449 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.449 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.450 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:16:52.450 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.957583+0000", 2024-04-20T17:16:52.450 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5510000000000002e-06 2024-04-20T17:16:52.450 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.450 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.450 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:16:52.450 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.957593+0000", 2024-04-20T17:16:52.450 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0091999999999999e-05 2024-04-20T17:16:52.450 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.450 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.450 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:16:52.450 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.957598+0000", 2024-04-20T17:16:52.450 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2780000000000001e-06 2024-04-20T17:16:52.450 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.450 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.451 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:16:52.451 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.964817+0000", 2024-04-20T17:16:52.451 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0072190850000000001 2024-04-20T17:16:52.451 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.451 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.451 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:16:52.451 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.964835+0000", 2024-04-20T17:16:52.451 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7546000000000001e-05 2024-04-20T17:16:52.451 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.451 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.451 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:16:52.451 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.997055+0000", 2024-04-20T17:16:52.451 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032220293999999997 2024-04-20T17:16:52.451 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.451 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.452 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:16:52.452 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.997106+0000", 2024-04-20T17:16:52.452 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0507999999999998e-05 2024-04-20T17:16:52.452 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.452 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:16:52.452 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.452 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.452 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.452 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:31985 2.6 e210/115 2:60e8f41b:::benchmark_data_smithi171_78918_object156733:head v 210'13949, mlcod=210'13930)", 2024-04-20T17:16:52.452 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.960543+0000", 2024-04-20T17:16:52.452 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 250.42479351399999, 2024-04-20T17:16:52.452 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036587569, 2024-04-20T17:16:52.452 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:16:52.452 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:16:52.452 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:52.453 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:16:52.453 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.453 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:16:52.453 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.960543+0000", 2024-04-20T17:16:52.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.453 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.453 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.453 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:16:52.453 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.960543+0000", 2024-04-20T17:16:52.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.453 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.453 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.453 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:16:52.453 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.960545+0000", 2024-04-20T17:16:52.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2809999999999998e-06 2024-04-20T17:16:52.454 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.454 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.454 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:16:52.454 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.960555+0000", 2024-04-20T17:16:52.454 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0059e-05 2024-04-20T17:16:52.454 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.454 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.454 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:16:52.454 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.960566+0000", 2024-04-20T17:16:52.454 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1197e-05 2024-04-20T17:16:52.454 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.454 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.454 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:16:52.454 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.960572+0000", 2024-04-20T17:16:52.455 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.852e-06 2024-04-20T17:16:52.455 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.455 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.455 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:16:52.455 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.968088+0000", 2024-04-20T17:16:52.455 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.007515585 2024-04-20T17:16:52.455 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.455 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.455 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:16:52.455 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.968104+0000", 2024-04-20T17:16:52.455 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5818e-05 2024-04-20T17:16:52.455 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.455 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.455 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:16:52.455 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.997109+0000", 2024-04-20T17:16:52.456 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.029005025 2024-04-20T17:16:52.456 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.456 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.456 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:16:52.456 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.997130+0000", 2024-04-20T17:16:52.456 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1752000000000001e-05 2024-04-20T17:16:52.456 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.456 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:16:52.456 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.456 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.456 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.456 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:31986 2.6 e210/115 2:60e97f5f:::benchmark_data_smithi171_78918_object86440:head v 210'13950, mlcod=210'13930)", 2024-04-20T17:16:52.456 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.963757+0000", 2024-04-20T17:16:52.456 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 250.42157932800001, 2024-04-20T17:16:52.456 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.039352508000000001, 2024-04-20T17:16:52.457 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:16:52.457 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:16:52.457 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:52.457 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:16:52.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.457 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:16:52.457 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.963757+0000", 2024-04-20T17:16:52.457 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.457 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.457 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:16:52.457 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.963757+0000", 2024-04-20T17:16:52.457 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.457 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.458 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:16:52.458 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.963758+0000", 2024-04-20T17:16:52.458 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.48e-06 2024-04-20T17:16:52.458 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.458 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.458 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:16:52.458 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.963765+0000", 2024-04-20T17:16:52.458 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.72e-06 2024-04-20T17:16:52.458 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.458 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.458 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:16:52.458 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.963775+0000", 2024-04-20T17:16:52.458 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7340000000000003e-06 2024-04-20T17:16:52.458 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.459 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.459 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:16:52.459 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.963780+0000", 2024-04-20T17:16:52.459 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2619999999999998e-06 2024-04-20T17:16:52.459 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.459 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.459 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:16:52.459 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.971061+0000", 2024-04-20T17:16:52.459 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0072804319999999999 2024-04-20T17:16:52.459 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.459 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.459 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:16:52.459 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.971085+0000", 2024-04-20T17:16:52.459 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4885999999999999e-05 2024-04-20T17:16:52.459 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.460 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.460 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:16:52.460 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:42.003079+0000", 2024-04-20T17:16:52.460 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.031993261000000002 2024-04-20T17:16:52.460 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.460 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.460 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:16:52.460 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:42.003109+0000", 2024-04-20T17:16:52.460 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0732999999999999e-05 2024-04-20T17:16:52.460 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.460 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:16:52.460 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.460 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.460 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.460 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:31987 2.6 e210/115 2:60ea5806:::benchmark_data_smithi171_78918_object109244:head v 210'13951, mlcod=210'13930)", 2024-04-20T17:16:52.461 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.966919+0000", 2024-04-20T17:16:52.461 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 250.41841683199999, 2024-04-20T17:16:52.461 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036210481000000003, 2024-04-20T17:16:52.461 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:16:52.461 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:16:52.461 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:52.461 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:16:52.461 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.461 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:16:52.461 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.966919+0000", 2024-04-20T17:16:52.461 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.461 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.461 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.461 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:16:52.461 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.966919+0000", 2024-04-20T17:16:52.461 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.462 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:16:52.462 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.966921+0000", 2024-04-20T17:16:52.462 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5850000000000001e-06 2024-04-20T17:16:52.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.462 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:16:52.462 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.966929+0000", 2024-04-20T17:16:52.462 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4959999999999996e-06 2024-04-20T17:16:52.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.462 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:16:52.462 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.966939+0000", 2024-04-20T17:16:52.463 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7389999999999994e-06 2024-04-20T17:16:52.463 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.463 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.463 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:16:52.463 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.966945+0000", 2024-04-20T17:16:52.463 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1519999999999997e-06 2024-04-20T17:16:52.463 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.463 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.463 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:16:52.463 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.974737+0000", 2024-04-20T17:16:52.463 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.007791877 2024-04-20T17:16:52.463 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.463 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.463 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:16:52.463 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.974752+0000", 2024-04-20T17:16:52.463 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4504999999999999e-05 2024-04-20T17:16:52.464 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.464 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.464 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:16:52.464 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:42.003113+0000", 2024-04-20T17:16:52.464 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028360811999999999 2024-04-20T17:16:52.464 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.464 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.464 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:16:52.464 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:42.003130+0000", 2024-04-20T17:16:52.464 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7314999999999999e-05 2024-04-20T17:16:52.464 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.464 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:16:52.464 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.464 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.464 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.465 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:32412 2.6 e211/115 2:61ea66cc:::benchmark_data_smithi171_78918_object99072:head v 211'14376, mlcod=211'14352)", 2024-04-20T17:16:52.465 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:43.845406+0000", 2024-04-20T17:16:52.465 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 248.53993007599999, 2024-04-20T17:16:52.465 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.037156029, 2024-04-20T17:16:52.465 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:16:52.465 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:16:52.465 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:52.465 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:16:52.465 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.465 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:16:52.465 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.845406+0000", 2024-04-20T17:16:52.465 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.465 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.465 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.465 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:16:52.466 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.845406+0000", 2024-04-20T17:16:52.466 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.466 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.466 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.466 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:16:52.466 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.845408+0000", 2024-04-20T17:16:52.466 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3099999999999999e-06 2024-04-20T17:16:52.466 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.466 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.466 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:16:52.466 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.845418+0000", 2024-04-20T17:16:52.466 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3440000000000007e-06 2024-04-20T17:16:52.466 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.466 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.467 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:16:52.467 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.845430+0000", 2024-04-20T17:16:52.467 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2690999999999999e-05 2024-04-20T17:16:52.467 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.467 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.467 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:16:52.467 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.845438+0000", 2024-04-20T17:16:52.467 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7589999999999997e-06 2024-04-20T17:16:52.467 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.467 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.467 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:16:52.467 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.860483+0000", 2024-04-20T17:16:52.467 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015045127 2024-04-20T17:16:52.467 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.467 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.468 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:16:52.468 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.860506+0000", 2024-04-20T17:16:52.468 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2527e-05 2024-04-20T17:16:52.468 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.468 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.468 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:16:52.468 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.882522+0000", 2024-04-20T17:16:52.468 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022016029999999999 2024-04-20T17:16:52.468 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.468 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.468 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:16:52.468 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.882562+0000", 2024-04-20T17:16:52.468 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0241000000000003e-05 2024-04-20T17:16:52.468 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.468 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:16:52.469 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.469 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.469 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.469 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:32728 2.6 e212/115 2:629fe820:::benchmark_data_smithi171_78918_object95717:head v 212'14692, mlcod=212'14670)", 2024-04-20T17:16:52.469 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:45.227687+0000", 2024-04-20T17:16:52.469 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 247.157649265, 2024-04-20T17:16:52.469 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.037695732000000003, 2024-04-20T17:16:52.469 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:16:52.469 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:16:52.469 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:52.469 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:16:52.469 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.469 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:16:52.469 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.227687+0000", 2024-04-20T17:16:52.469 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.470 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.470 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.470 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:16:52.470 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.227687+0000", 2024-04-20T17:16:52.470 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.470 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.470 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.470 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:16:52.470 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.227688+0000", 2024-04-20T17:16:52.470 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.64e-07 2024-04-20T17:16:52.470 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.470 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.470 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:16:52.470 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.227694+0000", 2024-04-20T17:16:52.470 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.755e-06 2024-04-20T17:16:52.471 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.471 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.471 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:16:52.471 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.227703+0000", 2024-04-20T17:16:52.471 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.499e-06 2024-04-20T17:16:52.471 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.471 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.471 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:16:52.471 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.227709+0000", 2024-04-20T17:16:52.471 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3399999999999997e-06 2024-04-20T17:16:52.471 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.471 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.471 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:16:52.471 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.238353+0000", 2024-04-20T17:16:52.471 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010644345 2024-04-20T17:16:52.472 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.472 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.472 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:16:52.472 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.238375+0000", 2024-04-20T17:16:52.472 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1691999999999998e-05 2024-04-20T17:16:52.472 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.472 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.472 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:16:52.472 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.265339+0000", 2024-04-20T17:16:52.472 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026964251000000002 2024-04-20T17:16:52.472 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.472 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.472 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:16:52.472 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.265383+0000", 2024-04-20T17:16:52.472 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3886e-05 2024-04-20T17:16:52.473 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.473 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:16:52.473 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.473 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.473 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.473 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.1.0:33314 2.6 e213/115 2:63f10b9a:::benchmark_data_smithi171_78918_object142360:head v 213'15278, mlcod=213'15258)", 2024-04-20T17:16:52.473 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:47.737543+0000", 2024-04-20T17:16:52.473 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 244.647793366, 2024-04-20T17:16:52.473 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.038417434, 2024-04-20T17:16:52.473 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:16:52.473 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:16:52.473 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:16:52.473 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:16:52.473 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.473 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:16:52.474 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.737543+0000", 2024-04-20T17:16:52.474 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.474 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.474 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.474 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:16:52.474 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.737543+0000", 2024-04-20T17:16:52.474 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:16:52.474 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.474 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.474 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:16:52.474 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.737544+0000", 2024-04-20T17:16:52.474 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3549999999999999e-06 2024-04-20T17:16:52.474 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.474 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.474 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:16:52.475 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.737552+0000", 2024-04-20T17:16:52.475 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1079999999999993e-06 2024-04-20T17:16:52.475 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.475 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.475 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:16:52.475 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.737563+0000", 2024-04-20T17:16:52.475 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0957e-05 2024-04-20T17:16:52.475 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.475 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.475 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:16:52.475 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.737571+0000", 2024-04-20T17:16:52.475 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9850000000000006e-06 2024-04-20T17:16:52.475 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.475 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.475 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:16:52.476 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.743454+0000", 2024-04-20T17:16:52.476 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0058832340000000002 2024-04-20T17:16:52.476 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.476 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.476 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:16:52.476 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.743475+0000", 2024-04-20T17:16:52.476 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0472999999999999e-05 2024-04-20T17:16:52.476 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.476 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.476 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:16:52.476 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.775922+0000", 2024-04-20T17:16:52.476 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032446572999999999 2024-04-20T17:16:52.476 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:16:52.476 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:16:52.476 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:16:52.476 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.775960+0000", 2024-04-20T17:16:52.477 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8748999999999998e-05 2024-04-20T17:16:52.477 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.477 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:16:52.477 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.477 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:16:52.477 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:16:52.477 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:16:53.058 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:16:53.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:53.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:16:53.163 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:16:53.195 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:53.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:53 smithi005 ceph-mon[39562]: pgmap v801: 17 pgs: 1 active+recovering, 16 active+clean; 4.0 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 149 op/s; 7201/132440 objects misplaced (5.437%); 382 KiB/s, 5 objects/s recovering 2024-04-20T17:16:53.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:53 smithi106 ceph-mon[44423]: pgmap v801: 17 pgs: 1 active+recovering, 16 active+clean; 4.0 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 149 op/s; 7201/132440 objects misplaced (5.437%); 382 KiB/s, 5 objects/s recovering 2024-04-20T17:16:53.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:53 smithi171 ceph-mon[44189]: pgmap v801: 17 pgs: 1 active+recovering, 16 active+clean; 4.0 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 149 op/s; 7201/132440 objects misplaced (5.437%); 382 KiB/s, 5 objects/s recovering 2024-04-20T17:16:54.146 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:16:54.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:54.155 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:16:54.182 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:16:54.257 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:16:54.289 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:54.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:54 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2313990691' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:16:54.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:54 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2313990691' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:16:54.540 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:16:54.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:54.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:16:54.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:54 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2313990691' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:16:54.650 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:16:54.677 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:54.695 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:16:54.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:54.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:16:54.801 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:16:54.827 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:54.845 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:16:54.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:54.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:16:54.949 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:16:54.978 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:55.177 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-20T17:16:55.178 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:55 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:16:55.178 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:55 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:16:55.178 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:55 smithi005 ceph-mon[39562]: pgmap v802: 17 pgs: 1 active+recovering, 16 active+clean; 4.0 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 175 op/s; 7091/131346 objects misplaced (5.399%); 691 KiB/s, 10 objects/s recovering 2024-04-20T17:16:55.178 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:55 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:16:55.178 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:55 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2313990691' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:16:55.178 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:55 smithi005 ceph-mon[39562]: osdmap e289: 12 total, 12 up, 12 in 2024-04-20T17:16:55.178 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:55 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2998919149' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:16:55.179 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:16:55.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:55.183 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:16:55.192 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:16:55.210 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:16:55.233 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:16:55.253 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:16:55.274 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:16:55.284 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:16:55.294 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:16:55.315 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:55.315 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:16:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:55 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:16:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:55 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:16:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:55 smithi171 ceph-mon[44189]: pgmap v802: 17 pgs: 1 active+recovering, 16 active+clean; 4.0 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 175 op/s; 7091/131346 objects misplaced (5.399%); 691 KiB/s, 10 objects/s recovering 2024-04-20T17:16:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:55 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:16:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:55 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2313990691' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:16:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:55 smithi171 ceph-mon[44189]: osdmap e289: 12 total, 12 up, 12 in 2024-04-20T17:16:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:55 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2998919149' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:16:55.337 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:16:55.338 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:16:55.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:55.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:16:55.369 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:16:55.389 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:16:55.411 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:16:55.431 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:16:55.433 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:16:55.433 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:16:55.433 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:16:55.433 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:16:55.433 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:16:55.433 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:16:55.433 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:16:55.433 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:16:55.433 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:16:55.433 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:16:55.433 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:16:55.433 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:16:55.445 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:16:55.459 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:16:55.484 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:55 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:16:55.484 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:55 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:16:55.484 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:55 smithi106 ceph-mon[44423]: pgmap v802: 17 pgs: 1 active+recovering, 16 active+clean; 4.0 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 175 op/s; 7091/131346 objects misplaced (5.399%); 691 KiB/s, 10 objects/s recovering 2024-04-20T17:16:55.484 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:55 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:16:55.485 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:55 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2313990691' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:16:55.485 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:55 smithi106 ceph-mon[44423]: osdmap e289: 12 total, 12 up, 12 in 2024-04-20T17:16:55.485 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:55 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2998919149' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:16:55.486 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:55.505 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:16:55.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:55.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:16:55.608 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:16:55.635 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:55.758 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:16:55.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:55.760 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:16:55.862 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:16:55.888 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:55.909 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:16:55.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:55.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:16:56.012 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:16:56.043 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:56.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:56.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:56.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:56.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:56.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:56.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:56.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:56.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:56.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:56 smithi171 ceph-mon[44189]: osdmap e290: 12 total, 12 up, 12 in 2024-04-20T17:16:56.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:56.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:56.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:56.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:56.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:56.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:56.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:56.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:56.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:56 smithi005 ceph-mon[39562]: osdmap e290: 12 total, 12 up, 12 in 2024-04-20T17:16:56.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:56.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:56.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:56.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:56.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:56.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:56.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:56.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:56.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:56 smithi106 ceph-mon[44423]: osdmap e290: 12 total, 12 up, 12 in 2024-04-20T17:16:56.824 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:16:56.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:56.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:16:56.941 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:16:56.974 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:57.129 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:16:57.133 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:16:57.225 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:16:57.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:57.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:16:57.332 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:16:57.360 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:57.379 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:16:57.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:57.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:16:57.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:57.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:57.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:57.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:57.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:57 smithi005 ceph-mon[39562]: pgmap v805: 17 pgs: 1 active+recovering, 16 active+clean; 4.0 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 166 op/s; 7091/130752 objects misplaced (5.423%); 440 KiB/s, 6 objects/s recovering 2024-04-20T17:16:57.483 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:16:57.507 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:57.507 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:57.507 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:57.507 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:57.507 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:57 smithi106 ceph-mon[44423]: pgmap v805: 17 pgs: 1 active+recovering, 16 active+clean; 4.0 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 166 op/s; 7091/130752 objects misplaced (5.423%); 440 KiB/s, 6 objects/s recovering 2024-04-20T17:16:57.508 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:57.529 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:16:57.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:57.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:16:57.548 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:16:57.548 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:16:57.570 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:57.570 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:57.570 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:57.570 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:16:57.570 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:57 smithi171 ceph-mon[44189]: pgmap v805: 17 pgs: 1 active+recovering, 16 active+clean; 4.0 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 166 op/s; 7091/130752 objects misplaced (5.423%); 440 KiB/s, 6 objects/s recovering 2024-04-20T17:16:57.634 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:16:57.662 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:57.681 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:16:57.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:57.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:16:57.787 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:16:57.813 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:57.832 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:16:57.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:57.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:16:57.935 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:16:57.964 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:57.984 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:16:57.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:57.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:16:58.006 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":805,"stamp":"2024-04-20T17:16:56.053678+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4284875320,"num_objects":65376,"num_object_clones":0,"num_object_copies":130752,"num_objects_missing_on_primary":3539,"num_objects_missing":3539,"num_objects_degraded":0,"num_objects_misplaced":7091,"num_objects_unfound":0,"num_objects_dirty":65376,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":256791,"num_write_kb":10306363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25672,"num_bytes_recovered":1684031616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160940,"ondisk_log_size":160940,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":10135912,"kb_used_data":8982452,"kb_used_omap":219,"kb_used_meta":1153124,"kb_avail":1114805912,"statfs":{"total":1151940427776,"available":1141561253888,"internally_reserved":0,"allocated":9198030848,"data_stored":9187049864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224949,"internal_metadata":1180799307},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,0,1,0,4,0,2,7,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-87097344,"num_objects":-1329,"num_object_clones":0,"num_object_copies":-2658,"num_objects_missing_on_primary":-55,"num_objects_missing":-55,"num_objects_degraded":0,"num_objects_misplaced":-110,"num_objects_unfound":0,"num_objects_dirty":-1329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1329,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":3604480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001341"},"pg_stats":[{"pgid":"2.f","version":"288'16081","reported_seq":16945,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-20T17:16:53.389140+0000","last_change":"2024-04-20T17:16:24.907036+0000","last_active":"2024-04-20T17:16:53.389140+0000","last_peered":"2024-04-20T17:16:53.389140+0000","last_clean":"2024-04-20T17:16:53.389140+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:16:53.389140+0000","last_undegraded":"2024-04-20T17:16:53.389140+0000","last_fullsized":"2024-04-20T17:16:53.389140+0000","mapping_epoch":115,"log_start":"135'6000","ondisk_log_start":"135'6000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'15773","last_scrub_stamp":"2024-04-20T17:16:24.906931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:24.906931+0000","objects_scrubbed":4413,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:47.687304+0000","scrub_duration":421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268763136,"num_objects":4101,"num_object_clones":0,"num_object_copies":8202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16081,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"289'15986","reported_seq":27920,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-20T17:16:55.237464+0000","last_change":"2024-04-20T17:16:40.290499+0000","last_active":"2024-04-20T17:16:55.237464+0000","last_peered":"2024-04-20T17:16:55.237464+0000","last_clean":"2024-04-20T17:16:55.237464+0000","last_became_active":"2024-04-20T17:10:28.865670+0000","last_became_peered":"2024-04-20T17:10:28.865670+0000","last_unstale":"2024-04-20T17:16:55.237464+0000","last_undegraded":"2024-04-20T17:16:55.237464+0000","last_fullsized":"2024-04-20T17:16:55.237464+0000","mapping_epoch":162,"log_start":"135'5900","ondisk_log_start":"135'5900","created":115,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'15835","last_scrub_stamp":"2024-04-20T17:16:40.290402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:40.290402+0000","objects_scrubbed":4222,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:55.604081+0000","scrub_duration":371,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266731520,"num_objects":4070,"num_object_clones":0,"num_object_copies":8140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15986,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5619,"num_bytes_recovered":368246784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"289'16341","reported_seq":17049,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-20T17:16:55.179124+0000","last_change":"2024-04-20T17:15:57.136522+0000","last_active":"2024-04-20T17:16:55.179124+0000","last_peered":"2024-04-20T17:16:55.179124+0000","last_clean":"2024-04-20T17:16:55.179124+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:16:55.179124+0000","last_undegraded":"2024-04-20T17:16:55.179124+0000","last_fullsized":"2024-04-20T17:16:55.179124+0000","mapping_epoch":115,"log_start":"135'6300","ondisk_log_start":"135'6300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'15780","last_scrub_stamp":"2024-04-20T17:15:57.136412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:57.136412+0000","objects_scrubbed":4702,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:10.130229+0000","scrub_duration":368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271384576,"num_objects":4141,"num_object_clones":0,"num_object_copies":8282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16341,"num_write_kb":655424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"289'22709","reported_seq":44218,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-20T17:16:55.683226+0000","last_change":"2024-04-20T17:16:35.136358+0000","last_active":"2024-04-20T17:16:55.683226+0000","last_peered":"2024-04-20T17:16:55.683226+0000","last_clean":"2024-04-20T17:16:55.683226+0000","last_became_active":"2024-04-20T17:12:08.085936+0000","last_became_peered":"2024-04-20T17:12:08.085936+0000","last_unstale":"2024-04-20T17:16:55.683226+0000","last_undegraded":"2024-04-20T17:16:55.683226+0000","last_fullsized":"2024-04-20T17:16:55.683226+0000","mapping_epoch":196,"log_start":"166'12668","ondisk_log_start":"166'12668","created":115,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'22494","last_scrub_stamp":"2024-04-20T17:16:35.136177+0000","last_deep_scrub":"213'20373","last_deep_scrub_stamp":"2024-04-20T17:12:48.443808+0000","last_clean_scrub_stamp":"2024-04-20T17:16:35.136177+0000","objects_scrubbed":4393,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:40.491841+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273743872,"num_objects":4177,"num_object_clones":0,"num_object_copies":8354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16057,"num_write_kb":647488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":450166784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"288'15748","reported_seq":33762,"reported_epoch":288,"state":"active+recovering","last_fresh":"2024-04-20T17:16:52.130854+0000","last_change":"2024-04-20T17:12:26.166109+0000","last_active":"2024-04-20T17:16:52.130854+0000","last_peered":"2024-04-20T17:16:52.130854+0000","last_clean":"2024-04-20T17:09:35.882603+0000","last_became_active":"2024-04-20T17:10:54.470253+0000","last_became_peered":"2024-04-20T17:10:54.470253+0000","last_unstale":"2024-04-20T17:16:52.130854+0000","last_undegraded":"2024-04-20T17:16:52.130854+0000","last_fullsized":"2024-04-20T17:16:52.130854+0000","mapping_epoch":171,"log_start":"135'5700","ondisk_log_start":"135'5700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260964352,"num_objects":3982,"num_object_clones":0,"num_object_copies":7964,"num_objects_missing_on_primary":3539,"num_objects_missing":3539,"num_objects_degraded":0,"num_objects_misplaced":7091,"num_objects_unfound":0,"num_objects_dirty":3982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15748,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5880,"num_bytes_recovered":385351680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"0,2,5","objects":13},{"shards":"0,10","objects":430},{"shards":"2,5","objects":3539}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":771,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-20T17:16:45.957909+0000","last_change":"2024-04-20T17:16:40.132382+0000","last_active":"2024-04-20T17:16:45.957909+0000","last_peered":"2024-04-20T17:16:45.957909+0000","last_clean":"2024-04-20T17:16:45.957909+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:16:45.957909+0000","last_undegraded":"2024-04-20T17:16:45.957909+0000","last_fullsized":"2024-04-20T17:16:45.957909+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:16:40.132254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:16:40.132254+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:54.442752+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"288'16079","reported_seq":16816,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-20T17:16:52.751567+0000","last_change":"2024-04-20T17:15:31.396169+0000","last_active":"2024-04-20T17:16:52.751567+0000","last_peered":"2024-04-20T17:16:52.751567+0000","last_clean":"2024-04-20T17:16:52.751567+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:16:52.751567+0000","last_undegraded":"2024-04-20T17:16:52.751567+0000","last_fullsized":"2024-04-20T17:16:52.751567+0000","mapping_epoch":115,"log_start":"135'6000","ondisk_log_start":"135'6000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15333","last_scrub_stamp":"2024-04-20T17:15:31.396069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:31.396069+0000","objects_scrubbed":4919,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:57.811864+0000","scrub_duration":440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273481728,"num_objects":4173,"num_object_clones":0,"num_object_copies":8346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16079,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"288'16246","reported_seq":16983,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-20T17:16:53.437620+0000","last_change":"2024-04-20T17:15:35.632022+0000","last_active":"2024-04-20T17:16:53.437620+0000","last_peered":"2024-04-20T17:16:53.437620+0000","last_clean":"2024-04-20T17:16:53.437620+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:16:53.437620+0000","last_undegraded":"2024-04-20T17:16:53.437620+0000","last_fullsized":"2024-04-20T17:16:53.437620+0000","mapping_epoch":115,"log_start":"135'6200","ondisk_log_start":"135'6200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15459","last_scrub_stamp":"2024-04-20T17:15:35.631842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:35.631842+0000","objects_scrubbed":4867,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":426,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267255808,"num_objects":4078,"num_object_clones":0,"num_object_copies":8156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16246,"num_write_kb":650368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"288'20130","reported_seq":21013,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-20T17:16:51.738473+0000","last_change":"2024-04-20T17:16:25.742067+0000","last_active":"2024-04-20T17:16:51.738473+0000","last_peered":"2024-04-20T17:16:51.738473+0000","last_clean":"2024-04-20T17:16:51.738473+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:16:51.738473+0000","last_undegraded":"2024-04-20T17:16:51.738473+0000","last_fullsized":"2024-04-20T17:16:51.738473+0000","mapping_epoch":115,"log_start":"152'10102","ondisk_log_start":"152'10102","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'19891","last_scrub_stamp":"2024-04-20T17:16:25.741880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:25.741880+0000","objects_scrubbed":4262,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:39.901935+0000","scrub_duration":378,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263651328,"num_objects":4023,"num_object_clones":0,"num_object_copies":8046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16019,"num_write_kb":641344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"288'16176","reported_seq":16970,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-20T17:16:52.377969+0000","last_change":"2024-04-20T17:15:41.950854+0000","last_active":"2024-04-20T17:16:52.377969+0000","last_peered":"2024-04-20T17:16:52.377969+0000","last_clean":"2024-04-20T17:16:52.377969+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:16:52.377969+0000","last_undegraded":"2024-04-20T17:16:52.377969+0000","last_fullsized":"2024-04-20T17:16:52.377969+0000","mapping_epoch":115,"log_start":"135'6100","ondisk_log_start":"135'6100","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'15491","last_scrub_stamp":"2024-04-20T17:15:41.950723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:41.950723+0000","objects_scrubbed":4733,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:58.122837+0000","scrub_duration":448,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265027584,"num_objects":4044,"num_object_clones":0,"num_object_copies":8088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16176,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"288'15989","reported_seq":16709,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-20T17:16:52.760359+0000","last_change":"2024-04-20T17:15:59.551771+0000","last_active":"2024-04-20T17:16:52.760359+0000","last_peered":"2024-04-20T17:16:52.760359+0000","last_clean":"2024-04-20T17:16:52.760359+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:16:52.760359+0000","last_undegraded":"2024-04-20T17:16:52.760359+0000","last_fullsized":"2024-04-20T17:16:52.760359+0000","mapping_epoch":115,"log_start":"135'5900","ondisk_log_start":"135'5900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'15438","last_scrub_stamp":"2024-04-20T17:15:59.551671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:59.551671+0000","objects_scrubbed":4614,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:10.078708+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266272768,"num_objects":4063,"num_object_clones":0,"num_object_copies":8126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15989,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"288'16113","reported_seq":31438,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-20T17:16:52.744069+0000","last_change":"2024-04-20T17:16:34.421711+0000","last_active":"2024-04-20T17:16:52.744069+0000","last_peered":"2024-04-20T17:16:52.744069+0000","last_clean":"2024-04-20T17:16:52.744069+0000","last_became_active":"2024-04-20T17:10:55.931751+0000","last_became_peered":"2024-04-20T17:10:55.931751+0000","last_unstale":"2024-04-20T17:16:52.744069+0000","last_undegraded":"2024-04-20T17:16:52.744069+0000","last_fullsized":"2024-04-20T17:16:52.744069+0000","mapping_epoch":173,"log_start":"135'6100","ondisk_log_start":"135'6100","created":115,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'15941","last_scrub_stamp":"2024-04-20T17:16:34.421587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:34.421587+0000","objects_scrubbed":4322,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:43.906503+0000","scrub_duration":366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271908864,"num_objects":4149,"num_object_clones":0,"num_object_copies":8298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16113,"num_write_kb":648384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7296,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"288'16219","reported_seq":17003,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-20T17:16:52.287988+0000","last_change":"2024-04-20T17:16:42.499318+0000","last_active":"2024-04-20T17:16:52.287988+0000","last_peered":"2024-04-20T17:16:52.287988+0000","last_clean":"2024-04-20T17:16:52.287988+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:16:52.287988+0000","last_undegraded":"2024-04-20T17:16:52.287988+0000","last_fullsized":"2024-04-20T17:16:52.287988+0000","mapping_epoch":115,"log_start":"135'6200","ondisk_log_start":"135'6200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'16092","last_scrub_stamp":"2024-04-20T17:16:42.499193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:42.499193+0000","objects_scrubbed":4224,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:04.645993+0000","scrub_duration":355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268500992,"num_objects":4097,"num_object_clones":0,"num_object_copies":8194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16219,"num_write_kb":650112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"288'15750","reported_seq":16482,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-20T17:16:53.276828+0000","last_change":"2024-04-20T17:15:34.656600+0000","last_active":"2024-04-20T17:16:53.276828+0000","last_peered":"2024-04-20T17:16:53.276828+0000","last_clean":"2024-04-20T17:16:53.276828+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:16:53.276828+0000","last_undegraded":"2024-04-20T17:16:53.276828+0000","last_fullsized":"2024-04-20T17:16:53.276828+0000","mapping_epoch":115,"log_start":"135'5700","ondisk_log_start":"135'5700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15023","last_scrub_stamp":"2024-04-20T17:15:34.656503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:34.656503+0000","objects_scrubbed":4842,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:56.614124+0000","scrub_duration":430,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269615104,"num_objects":4114,"num_object_clones":0,"num_object_copies":8228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15750,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"288'16046","reported_seq":16778,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-04-20T17:16:52.254882+0000","last_change":"2024-04-20T17:15:31.794340+0000","last_active":"2024-04-20T17:16:52.254882+0000","last_peered":"2024-04-20T17:16:52.254882+0000","last_clean":"2024-04-20T17:16:52.254882+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:16:52.254882+0000","last_undegraded":"2024-04-20T17:16:52.254882+0000","last_fullsized":"2024-04-20T17:16:52.254882+0000","mapping_epoch":115,"log_start":"135'6000","ondisk_log_start":"135'6000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15271","last_scrub_stamp":"2024-04-20T17:15:31.794156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:31.794156+0000","objects_scrubbed":4831,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:58.805155+0000","scrub_duration":443,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265814016,"num_objects":4056,"num_object_clones":0,"num_object_copies":8112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16046,"num_write_kb":643264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"289'15847","reported_seq":16581,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-20T17:16:55.725058+0000","last_change":"2024-04-20T17:15:33.298229+0000","last_active":"2024-04-20T17:16:55.725058+0000","last_peered":"2024-04-20T17:16:55.725058+0000","last_clean":"2024-04-20T17:16:55.725058+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:16:55.725058+0000","last_undegraded":"2024-04-20T17:16:55.725058+0000","last_fullsized":"2024-04-20T17:16:55.725058+0000","mapping_epoch":115,"log_start":"135'5800","ondisk_log_start":"135'5800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15067","last_scrub_stamp":"2024-04-20T17:15:33.298103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:33.298103+0000","objects_scrubbed":4809,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:16:56.012242+0000","scrub_duration":458,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263585816,"num_objects":4023,"num_object_clones":0,"num_object_copies":8046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4023,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15847,"num_write_kb":635777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"289'15913","reported_seq":16568,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-04-20T17:16:55.866385+0000","last_change":"2024-04-20T17:16:23.593408+0000","last_active":"2024-04-20T17:16:55.866385+0000","last_peered":"2024-04-20T17:16:55.866385+0000","last_clean":"2024-04-20T17:16:55.866385+0000","last_became_active":"2024-04-20T17:12:23.515375+0000","last_became_peered":"2024-04-20T17:12:23.515375+0000","last_unstale":"2024-04-20T17:16:55.866385+0000","last_undegraded":"2024-04-20T17:16:55.866385+0000","last_fullsized":"2024-04-20T17:16:55.866385+0000","mapping_epoch":202,"log_start":"135'5900","ondisk_log_start":"135'5900","created":115,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'15586","last_scrub_stamp":"2024-04-20T17:16:23.593300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:23.593300+0000","objects_scrubbed":4410,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:33.363583+0000","scrub_duration":446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267583488,"num_objects":4083,"num_object_clones":0,"num_object_copies":8166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15913,"num_write_kb":639872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":4284284952,"num_objects":65374,"num_object_clones":0,"num_object_copies":130748,"num_objects_missing_on_primary":3539,"num_objects_missing":3539,"num_objects_degraded":0,"num_objects_misplaced":7091,"num_objects_unfound":0,"num_objects_dirty":65374,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":256610,"num_write_kb":10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25664,"num_bytes_recovered":1681915904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9161940992,"data_stored":9161932848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},"log_size":160803,"ondisk_log_size":160803,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991366,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":860024,"kb_used_data":788592,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92885128,"statfs":{"total":95995035648,"available":95114371072,"internally_reserved":0,"allocated":807518208,"data_stored":806603298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":73136580},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253004,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":970992,"kb_used_data":819048,"kb_used_omap":1,"kb_used_meta":151934,"kb_avail":92774160,"statfs":{"total":95995035648,"available":95000739840,"internally_reserved":0,"allocated":838705152,"data_stored":837793794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":155580864},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612754,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1198084,"kb_used_data":1046136,"kb_used_omap":1,"kb_used_meta":151934,"kb_avail":92547068,"statfs":{"total":95995035648,"available":94768197632,"internally_reserved":0,"allocated":1071243264,"data_stored":1070321926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":155580864},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762808983,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1152560,"kb_used_data":1045032,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92592592,"statfs":{"total":95995035648,"available":94814814208,"internally_reserved":0,"allocated":1070112768,"data_stored":1069201430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168732,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":872248,"kb_used_data":796656,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92872904,"statfs":{"total":95995035648,"available":95101853696,"internally_reserved":0,"allocated":815775744,"data_stored":814860854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430368,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":638896,"kb_used_data":531368,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93106256,"statfs":{"total":95995035648,"available":95340806144,"internally_reserved":0,"allocated":544120832,"data_stored":543209494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575034,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":614256,"kb_used_data":528936,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93130896,"statfs":{"total":95995035648,"available":95366037504,"internally_reserved":0,"allocated":541630464,"data_stored":540719126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738559,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":952496,"kb_used_data":817192,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":92792656,"statfs":{"total":95995035648,"available":95019679744,"internally_reserved":0,"allocated":836804608,"data_stored":835893270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,0,4,0,2,7,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393848,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":741760,"kb_used_data":693364,"kb_used_omap":202,"kb_used_meta":48181,"kb_avail":93003392,"statfs":{"total":95995035648,"available":95235473408,"internally_reserved":0,"allocated":710004736,"data_stored":709087470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":49337815},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182576,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":596096,"kb_used_data":524660,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93149056,"statfs":{"total":95995035648,"available":95384633344,"internally_reserved":0,"allocated":537251840,"data_stored":536334574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986347,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303364,"kb_used_data":259704,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":93441788,"statfs":{"total":95995035648,"available":95684390912,"internally_reserved":0,"allocated":265936896,"data_stored":265015558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790117,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1235136,"kb_used_data":1131764,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92510016,"statfs":{"total":95995035648,"available":94730256384,"internally_reserved":0,"allocated":1158926336,"data_stored":1158009070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":832831488,"data_stored":832831488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":537526272,"data_stored":537526272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":707788800,"data_stored":707788800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":534249472,"data_stored":534249472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":263786496,"data_stored":263782424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1156448256,"data_stored":1156448256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":540737536,"data_stored":540737536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":811663360,"data_stored":811663360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1068236800,"data_stored":1068236800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1068699648,"data_stored":1068695576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":835518464,"data_stored":835518464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":804454400,"data_stored":804454400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:16:58.007 INFO:tasks.cephadm.ceph_manager.ceph:PG 2.3 is not active+clean 2024-04-20T17:16:58.007 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '2.3', 'version': "288'15748", 'reported_seq': 33762, 'reported_epoch': 288, 'state': 'active+recovering', 'last_fresh': '2024-04-20T17:16:52.130854+0000', 'last_change': '2024-04-20T17:12:26.166109+0000', 'last_active': '2024-04-20T17:16:52.130854+0000', 'last_peered': '2024-04-20T17:16:52.130854+0000', 'last_clean': '2024-04-20T17:09:35.882603+0000', 'last_became_active': '2024-04-20T17:10:54.470253+0000', 'last_became_peered': '2024-04-20T17:10:54.470253+0000', 'last_unstale': '2024-04-20T17:16:52.130854+0000', 'last_undegraded': '2024-04-20T17:16:52.130854+0000', 'last_fullsized': '2024-04-20T17:16:52.130854+0000', 'mapping_epoch': 171, 'log_start': "135'5700", 'ondisk_log_start': "135'5700", 'created': 115, 'last_epoch_clean': 116, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-20T17:08:04.990639+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T17:08:04.990639+0000', 'last_clean_scrub_stamp': '2024-04-20T17:08:04.990639+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 260964352, 'num_objects': 3982, 'num_object_clones': 0, 'num_object_copies': 7964, 'num_objects_missing_on_primary': 3539, 'num_objects_missing': 3539, 'num_objects_degraded': 0, 'num_objects_misplaced': 7091, 'num_objects_unfound': 0, 'num_objects_dirty': 3982, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15748, 'num_write_kb': 631360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5880, 'num_bytes_recovered': 385351680, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 10], 'acting': [0, 10], 'avail_no_missing': ['2', '5'], 'object_location_counts': [{'shards': '0,2,5', 'objects': 13}, {'shards': '0,10', 'objects': 430}, {'shards': '2,5', 'objects': 3539}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-20T17:16:58.008 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph status --format=json 2024-04-20T17:16:58.069 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:16:58.088 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:16:58.115 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:58.133 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:16:58.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:58.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:16:58.237 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:16:58.263 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:58.281 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:16:58.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:58.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:16:58.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:58 smithi171 ceph-mon[44189]: from='client.25445 v1:172.21.15.5:0/2580092067' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:16:58.367 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-20T17:16:58.368 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:16:58.385 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:16:58.416 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:58 smithi005 ceph-mon[39562]: from='client.25445 v1:172.21.15.5:0/2580092067' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:16:58.418 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:58.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:58 smithi106 ceph-mon[44423]: from='client.25445 v1:172.21.15.5:0/2580092067' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:16:58.639 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:16:58.650 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:16:58.668 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:16:58.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:58.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:16:58.777 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:16:58.807 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:59.062 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:16:59.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:59.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:16:59.166 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:16:59.195 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:16:59 smithi005 ceph-mon[39562]: pgmap v806: 17 pgs: 1 active+recovering, 16 active+clean; 4.0 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 205 op/s; 6984/129786 objects misplaced (5.381%); 872 KiB/s, 13 objects/s recovering 2024-04-20T17:16:59.263 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:16:59.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:16:59 smithi106 ceph-mon[44423]: pgmap v806: 17 pgs: 1 active+recovering, 16 active+clean; 4.0 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 205 op/s; 6984/129786 objects misplaced (5.381%); 872 KiB/s, 13 objects/s recovering 2024-04-20T17:16:59.549 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:16:59.551 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:16:59 smithi171 ceph-mon[44189]: pgmap v806: 17 pgs: 1 active+recovering, 16 active+clean; 4.0 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 205 op/s; 6984/129786 objects misplaced (5.381%); 872 KiB/s, 13 objects/s recovering 2024-04-20T17:16:59.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:59.554 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:16:59.655 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:16:59.681 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:59.700 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:16:59.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:59.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:16:59.804 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:16:59.833 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:16:59.851 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:16:59.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:16:59.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:16:59.955 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:16:59.983 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:00.312 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:17:00.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:00.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:17:00.415 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:17:00.447 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:00.657 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:17:00.658 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:17:00.898 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:17:00.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:00.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:17:00.938 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:17:01.003 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:17:01.031 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:01.106 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:17:01.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:01.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:17:01.211 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:17:01.243 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:01 smithi171 ceph-mon[44189]: pgmap v807: 17 pgs: 1 active+recovering, 16 active+clean; 4.0 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 186 op/s; 6984/129450 objects misplaced (5.395%); 872 KiB/s, 13 objects/s recovering 2024-04-20T17:17:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:01 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:17:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:01 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:17:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:01 smithi171 ceph-mon[44189]: osdmap e291: 12 total, 12 up, 12 in 2024-04-20T17:17:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:01 smithi171 ceph-mon[44189]: osdmap e292: 12 total, 12 up, 12 in 2024-04-20T17:17:01.367 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:17:01.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:01.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:17:01.446 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-20T17:17:01.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:01 smithi005 ceph-mon[39562]: pgmap v807: 17 pgs: 1 active+recovering, 16 active+clean; 4.0 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 186 op/s; 6984/129450 objects misplaced (5.395%); 872 KiB/s, 13 objects/s recovering 2024-04-20T17:17:01.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:01 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:17:01.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:01 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:17:01.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:01 smithi005 ceph-mon[39562]: osdmap e291: 12 total, 12 up, 12 in 2024-04-20T17:17:01.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:01 smithi005 ceph-mon[39562]: osdmap e292: 12 total, 12 up, 12 in 2024-04-20T17:17:01.471 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:17:01.501 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:01.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:01 smithi106 ceph-mon[44423]: pgmap v807: 17 pgs: 1 active+recovering, 16 active+clean; 4.0 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 186 op/s; 6984/129450 objects misplaced (5.395%); 872 KiB/s, 13 objects/s recovering 2024-04-20T17:17:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:01 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:17:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:01 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:17:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:01 smithi106 ceph-mon[44423]: osdmap e291: 12 total, 12 up, 12 in 2024-04-20T17:17:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:01 smithi106 ceph-mon[44423]: osdmap e292: 12 total, 12 up, 12 in 2024-04-20T17:17:01.533 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:17:01.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:01.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:17:01.638 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:17:01.665 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:01.683 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:17:01.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:01.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:17:01.724 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:17:01.787 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:17:01.816 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:01.891 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:17:01.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:01.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:17:01.995 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:17:02.029 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:02.784 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:17:02.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:02.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:17:02.890 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:17:02.916 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:02.935 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:17:02.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:02.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:17:03.038 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:17:03.068 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:03.169 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:17:03.170 INFO:teuthology.orchestra.run.smithi005.stdout:{"fsid":"13c6e76c-ff37-11ee-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":991,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":292,"num_osds":12,"num_up_osds":12,"osd_up_since":1713633141,"num_in_osds":12,"osd_in_since":1713632742,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovering","count":1}],"num_pgs":17,"num_pools":2,"num_objects":64372,"data_bytes":4219077176,"bytes_used":10248437760,"bytes_avail":1141691990016,"bytes_total":1151940427776,"misplaced_objects":6984,"misplaced_total":128744,"misplaced_ratio":0.05424718821848009,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":579697,"recovering_keys_per_sec":0,"num_objects_recovered":54,"num_bytes_recovered":3538944,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":164},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.5:8443/"}},"servicemap":{"epoch":25,"modified":"2024-04-20T17:16:50.052718+0000","services":{}},"progress_events":{}} 2024-04-20T17:17:03.191 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:17:03.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:03.194 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:17:03.296 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:17:03.328 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:03.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:03 smithi005 ceph-mon[39562]: pgmap v810: 17 pgs: 1 active+recovering, 16 active+clean; 3.9 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 164 op/s; 6984/128744 objects misplaced (5.425%); 566 KiB/s, 8 objects/s recovering 2024-04-20T17:17:03.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:03 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:17:03.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:03 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:17:03.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:03 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:17:03.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:03 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:17:03.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:03 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1598186896' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:17:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:03 smithi106 ceph-mon[44423]: pgmap v810: 17 pgs: 1 active+recovering, 16 active+clean; 3.9 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 164 op/s; 6984/128744 objects misplaced (5.425%); 566 KiB/s, 8 objects/s recovering 2024-04-20T17:17:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:03 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:17:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:03 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:17:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:03 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:17:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:03 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:17:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:03 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1598186896' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:17:03.523 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:03 smithi171 ceph-mon[44189]: pgmap v810: 17 pgs: 1 active+recovering, 16 active+clean; 3.9 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 164 op/s; 6984/128744 objects misplaced (5.425%); 566 KiB/s, 8 objects/s recovering 2024-04-20T17:17:03.523 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:03 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:17:03.523 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:03 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:17:03.523 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:03 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:17:03.523 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:03 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:17:03.523 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:03 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1598186896' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:17:03.830 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:17:03.830 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:17:03.880 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:17:03.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:03.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:17:03.985 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:17:04.014 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:04.165 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:17:04.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:04.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:17:04.203 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:17:04.270 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:17:04.306 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:04.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:04 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3051653680' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:17:04.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:04 smithi005 ceph-mon[39562]: from='client.25426 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:17:04.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:04 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3051653680' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:17:04.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:04 smithi106 ceph-mon[44423]: from='client.25426 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:17:04.577 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:04 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3051653680' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:17:04.577 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:04 smithi171 ceph-mon[44189]: from='client.25426 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:17:04.610 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-20T17:17:04.652 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:17:04.652 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:17:04.652 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:17:04.652 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:17:04.652 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:17:04.652 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:17:04.652 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:17:04.652 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:17:04.660 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:17:04.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:04.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:17:04.667 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:17:04.688 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:17:04.710 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:17:04.732 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:17:04.733 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:17:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:17:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:17:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:17:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:17:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:17:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:17:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:17:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:17:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:17:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:17:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:17:04.765 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:17:04.796 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:04.919 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:17:05.443 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:05 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:17:05.443 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:05 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:17:05.443 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:05 smithi171 ceph-mon[44189]: pgmap v811: 17 pgs: 1 active+recovering, 16 active+clean; 3.9 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 191 op/s; 6875/127686 objects misplaced (5.384%); 864 KiB/s, 13 objects/s recovering 2024-04-20T17:17:05.443 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:05 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:17:05.443 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:05 smithi171 ceph-mon[44189]: from='client.25426 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:17:05.444 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:05 smithi171 ceph-mon[44189]: osdmap e293: 12 total, 12 up, 12 in 2024-04-20T17:17:05.444 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:05 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3001048287' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:17:05.444 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:05.444 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:05.444 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:05.444 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:05.444 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:05.444 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:05.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:05 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:17:05.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:05 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:17:05.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:05 smithi005 ceph-mon[39562]: pgmap v811: 17 pgs: 1 active+recovering, 16 active+clean; 3.9 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 191 op/s; 6875/127686 objects misplaced (5.384%); 864 KiB/s, 13 objects/s recovering 2024-04-20T17:17:05.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:05 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:17:05.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:05 smithi005 ceph-mon[39562]: from='client.25426 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:17:05.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:05 smithi005 ceph-mon[39562]: osdmap e293: 12 total, 12 up, 12 in 2024-04-20T17:17:05.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:05 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3001048287' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:17:05.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:05.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:05.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:05.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:05.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:05.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:05 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:17:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:05 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:17:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:05 smithi106 ceph-mon[44423]: pgmap v811: 17 pgs: 1 active+recovering, 16 active+clean; 3.9 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 191 op/s; 6875/127686 objects misplaced (5.384%); 864 KiB/s, 13 objects/s recovering 2024-04-20T17:17:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:05 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:17:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:05 smithi106 ceph-mon[44423]: from='client.25426 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:17:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:05 smithi106 ceph-mon[44423]: osdmap e293: 12 total, 12 up, 12 in 2024-04-20T17:17:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:05 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3001048287' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:17:05.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:05.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:05.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:05.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:05.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:05.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:05.545 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:17:05.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:05.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:17:05.654 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:17:05.681 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:05.700 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:17:05.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:05.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:17:05.804 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:17:05.831 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:05.849 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:17:05.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:05.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:17:05.895 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-20T17:17:05.895 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:17:05.953 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:17:05.990 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:06.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:06.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:06.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:06 smithi005 ceph-mon[39562]: 2.d scrub starts 2024-04-20T17:17:06.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:06.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:06.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:06.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:06.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:06 smithi005 ceph-mon[39562]: 2.d scrub ok 2024-04-20T17:17:06.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:06 smithi005 ceph-mon[39562]: osdmap e294: 12 total, 12 up, 12 in 2024-04-20T17:17:06.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:06.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:06.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:06 smithi106 ceph-mon[44423]: 2.d scrub starts 2024-04-20T17:17:06.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:06.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:06.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:06.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:06.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:06 smithi106 ceph-mon[44423]: 2.d scrub ok 2024-04-20T17:17:06.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:06 smithi106 ceph-mon[44423]: osdmap e294: 12 total, 12 up, 12 in 2024-04-20T17:17:06.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:06.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:06.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:06 smithi171 ceph-mon[44189]: 2.d scrub starts 2024-04-20T17:17:06.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:06.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:06.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:06.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:06.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:06 smithi171 ceph-mon[44189]: 2.d scrub ok 2024-04-20T17:17:06.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:06 smithi171 ceph-mon[44189]: osdmap e294: 12 total, 12 up, 12 in 2024-04-20T17:17:06.984 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:17:06.984 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:17:06.984 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 0 2024-04-20T17:17:06.984 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:17:07.047 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:17:07.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:07.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:17:07.107 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:17:07.152 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:17:07.184 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:07.226 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:17:07.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:07.230 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:17:07.331 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:17:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:07 smithi171 ceph-mon[44189]: pgmap v814: 17 pgs: 1 active+recovering, 16 active+clean; 3.9 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 198 op/s; 6875/127120 objects misplaced (5.408%); 589 KiB/s, 9 objects/s recovering 2024-04-20T17:17:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:07 smithi171 ceph-mon[44189]: 2.b scrub starts 2024-04-20T17:17:07.357 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:07.377 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:17:07.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:07.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:17:07.398 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:07 smithi005 ceph-mon[39562]: pgmap v814: 17 pgs: 1 active+recovering, 16 active+clean; 3.9 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 198 op/s; 6875/127120 objects misplaced (5.408%); 589 KiB/s, 9 objects/s recovering 2024-04-20T17:17:07.398 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:07 smithi005 ceph-mon[39562]: 2.b scrub starts 2024-04-20T17:17:07.429 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:17:07.483 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:17:07.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:07 smithi106 ceph-mon[44423]: pgmap v814: 17 pgs: 1 active+recovering, 16 active+clean; 3.9 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 198 op/s; 6875/127120 objects misplaced (5.408%); 589 KiB/s, 9 objects/s recovering 2024-04-20T17:17:07.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:07 smithi106 ceph-mon[44423]: 2.b scrub starts 2024-04-20T17:17:07.511 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:07.759 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:17:07.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:07.773 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:17:07.784 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-20T17:17:07.873 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:17:07.903 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:08.052 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:17:08.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:08.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:17:08.125 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:17:08.156 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:17:08.182 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:08.202 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:17:08.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:08.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:17:08.305 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:17:08.331 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:08.350 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:17:08.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:08.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:17:08.453 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:17:08.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:08 smithi005 ceph-mon[39562]: 2.b scrub ok 2024-04-20T17:17:08.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:08 smithi005 ceph-mon[39562]: 2.c scrub starts 2024-04-20T17:17:08.483 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:08 smithi171 ceph-mon[44189]: 2.b scrub ok 2024-04-20T17:17:08.483 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:08 smithi171 ceph-mon[44189]: 2.c scrub starts 2024-04-20T17:17:08.490 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:08.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:08 smithi106 ceph-mon[44423]: 2.b scrub ok 2024-04-20T17:17:08.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:08 smithi106 ceph-mon[44423]: 2.c scrub starts 2024-04-20T17:17:08.515 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:17:08.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:08.518 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:17:08.620 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:17:08.651 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:08.752 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:17:08.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:08.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:17:08.857 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:17:08.883 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:08.902 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:17:08.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:08.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:17:09.005 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:17:09.033 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:09.051 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:17:09.067 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:17:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:09 smithi171 ceph-mon[44189]: 2.7 scrub starts 2024-04-20T17:17:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:09 smithi171 ceph-mon[44189]: 2.c scrub ok 2024-04-20T17:17:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:09 smithi171 ceph-mon[44189]: 2.7 scrub ok 2024-04-20T17:17:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:09 smithi171 ceph-mon[44189]: pgmap v815: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.9 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 185 op/s; 6765/126232 objects misplaced (5.359%); 991 KiB/s, 15 objects/s recovering 2024-04-20T17:17:09.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:09 smithi005 ceph-mon[39562]: 2.7 scrub starts 2024-04-20T17:17:09.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:09 smithi005 ceph-mon[39562]: 2.c scrub ok 2024-04-20T17:17:09.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:09 smithi005 ceph-mon[39562]: 2.7 scrub ok 2024-04-20T17:17:09.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:09 smithi005 ceph-mon[39562]: pgmap v815: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.9 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 185 op/s; 6765/126232 objects misplaced (5.359%); 991 KiB/s, 15 objects/s recovering 2024-04-20T17:17:09.481 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:17:09.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:09.486 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:17:09.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:09 smithi106 ceph-mon[44423]: 2.7 scrub starts 2024-04-20T17:17:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:09 smithi106 ceph-mon[44423]: 2.c scrub ok 2024-04-20T17:17:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:09 smithi106 ceph-mon[44423]: 2.7 scrub ok 2024-04-20T17:17:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:09 smithi106 ceph-mon[44423]: pgmap v815: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.9 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 185 op/s; 6765/126232 objects misplaced (5.359%); 991 KiB/s, 15 objects/s recovering 2024-04-20T17:17:09.587 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:17:09.598 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:17:09.602 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:17:09.613 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:09.631 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:17:09.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:09.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:17:09.735 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:17:09.765 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:09.914 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:17:09.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:09.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:17:10.017 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:17:10.047 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:10.139 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:17:10.140 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:17:10.140 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-04-20T17:17:10.140 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-04-20T17:17:10.140 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:17:10.209 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:10 smithi005 ceph-mon[39562]: from='client.25469 v1:172.21.15.5:0/3000823866' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:17:10.209 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:10 smithi005 ceph-mon[39562]: 2.0 scrub starts 2024-04-20T17:17:10.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:10 smithi106 ceph-mon[44423]: from='client.25469 v1:172.21.15.5:0/3000823866' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:17:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:10 smithi106 ceph-mon[44423]: 2.0 scrub starts 2024-04-20T17:17:10.541 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:10 smithi171 ceph-mon[44189]: from='client.25469 v1:172.21.15.5:0/3000823866' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:17:10.541 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:10 smithi171 ceph-mon[44189]: 2.0 scrub starts 2024-04-20T17:17:11.010 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-20T17:17:11.307 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:17:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:11 smithi171 ceph-mon[44189]: pgmap v816: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.8 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 179 op/s; 6765/125866 objects misplaced (5.375%); 880 KiB/s, 13 objects/s recovering 2024-04-20T17:17:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:11 smithi171 ceph-mon[44189]: 2.0 scrub ok 2024-04-20T17:17:11.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:11 smithi005 ceph-mon[39562]: pgmap v816: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.8 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 179 op/s; 6765/125866 objects misplaced (5.375%); 880 KiB/s, 13 objects/s recovering 2024-04-20T17:17:11.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:11 smithi005 ceph-mon[39562]: 2.0 scrub ok 2024-04-20T17:17:11.490 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":815,"stamp":"2024-04-20T17:17:08.055846+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4136763960,"num_objects":63116,"num_object_clones":0,"num_object_copies":126232,"num_objects_missing_on_primary":3377,"num_objects_missing":3377,"num_objects_degraded":0,"num_objects_misplaced":6765,"num_objects_unfound":0,"num_objects_dirty":63116,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":259051,"num_write_kb":10306363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25836,"num_bytes_recovered":1694779520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":9842100,"kb_used_data":8720428,"kb_used_omap":219,"kb_used_meta":1121252,"kb_avail":1115099724,"statfs":{"total":1151940427776,"available":1141862117376,"internally_reserved":0,"allocated":8929718272,"data_stored":8918392014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224949,"internal_metadata":1148162379},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,2,0,0,2,3,5,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-86310912,"num_objects":-1317,"num_object_clones":0,"num_object_copies":-2634,"num_objects_missing_on_primary":-109,"num_objects_missing":-109,"num_objects_degraded":0,"num_objects_misplaced":-219,"num_objects_unfound":0,"num_objects_dirty":-1317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1317,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":7208960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.103384"},"pg_stats":[{"pgid":"2.f","version":"292'16203","reported_seq":17075,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-04-20T17:17:03.635302+0000","last_change":"2024-04-20T17:16:24.907036+0000","last_active":"2024-04-20T17:17:03.635302+0000","last_peered":"2024-04-20T17:17:03.635302+0000","last_clean":"2024-04-20T17:17:03.635302+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:17:03.635302+0000","last_undegraded":"2024-04-20T17:17:03.635302+0000","last_fullsized":"2024-04-20T17:17:03.635302+0000","mapping_epoch":115,"log_start":"135'6200","ondisk_log_start":"135'6200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'15773","last_scrub_stamp":"2024-04-20T17:16:24.906931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:24.906931+0000","objects_scrubbed":4413,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:47.687304+0000","scrub_duration":421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260767744,"num_objects":3979,"num_object_clones":0,"num_object_copies":7958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16203,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"293'16087","reported_seq":28029,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-20T17:17:05.615923+0000","last_change":"2024-04-20T17:16:40.290499+0000","last_active":"2024-04-20T17:17:05.615923+0000","last_peered":"2024-04-20T17:17:05.615923+0000","last_clean":"2024-04-20T17:17:05.615923+0000","last_became_active":"2024-04-20T17:10:28.865670+0000","last_became_peered":"2024-04-20T17:10:28.865670+0000","last_unstale":"2024-04-20T17:17:05.615923+0000","last_undegraded":"2024-04-20T17:17:05.615923+0000","last_fullsized":"2024-04-20T17:17:05.615923+0000","mapping_epoch":162,"log_start":"135'6000","ondisk_log_start":"135'6000","created":115,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'15835","last_scrub_stamp":"2024-04-20T17:16:40.290402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:40.290402+0000","objects_scrubbed":4222,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:55.604081+0000","scrub_duration":371,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260112384,"num_objects":3969,"num_object_clones":0,"num_object_copies":7938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16087,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5619,"num_bytes_recovered":368246784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"293'16470","reported_seq":17186,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-20T17:17:05.435832+0000","last_change":"2024-04-20T17:15:57.136522+0000","last_active":"2024-04-20T17:17:05.435832+0000","last_peered":"2024-04-20T17:17:05.435832+0000","last_clean":"2024-04-20T17:17:05.435832+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:17:05.435832+0000","last_undegraded":"2024-04-20T17:17:05.435832+0000","last_fullsized":"2024-04-20T17:17:05.435832+0000","mapping_epoch":115,"log_start":"136'6400","ondisk_log_start":"136'6400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'15780","last_scrub_stamp":"2024-04-20T17:15:57.136412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:57.136412+0000","objects_scrubbed":4702,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:10.130229+0000","scrub_duration":368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262930432,"num_objects":4012,"num_object_clones":0,"num_object_copies":8024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16470,"num_write_kb":655424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"293'22828","reported_seq":44345,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-20T17:17:05.704132+0000","last_change":"2024-04-20T17:16:35.136358+0000","last_active":"2024-04-20T17:17:05.704132+0000","last_peered":"2024-04-20T17:17:05.704132+0000","last_clean":"2024-04-20T17:17:05.704132+0000","last_became_active":"2024-04-20T17:12:08.085936+0000","last_became_peered":"2024-04-20T17:12:08.085936+0000","last_unstale":"2024-04-20T17:17:05.704132+0000","last_undegraded":"2024-04-20T17:17:05.704132+0000","last_fullsized":"2024-04-20T17:17:05.704132+0000","mapping_epoch":196,"log_start":"166'12768","ondisk_log_start":"166'12768","created":115,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'22494","last_scrub_stamp":"2024-04-20T17:16:35.136177+0000","last_deep_scrub":"213'20373","last_deep_scrub_stamp":"2024-04-20T17:12:48.443808+0000","last_clean_scrub_stamp":"2024-04-20T17:16:35.136177+0000","objects_scrubbed":4393,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:40.491841+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265945088,"num_objects":4058,"num_object_clones":0,"num_object_copies":8116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16176,"num_write_kb":647488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":450166784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"294'15912","reported_seq":34429,"reported_epoch":294,"state":"active+recovering","last_fresh":"2024-04-20T17:17:07.326409+0000","last_change":"2024-04-20T17:12:26.166109+0000","last_active":"2024-04-20T17:17:07.326409+0000","last_peered":"2024-04-20T17:17:07.326409+0000","last_clean":"2024-04-20T17:09:35.882603+0000","last_became_active":"2024-04-20T17:10:54.470253+0000","last_became_peered":"2024-04-20T17:10:54.470253+0000","last_unstale":"2024-04-20T17:17:07.326409+0000","last_undegraded":"2024-04-20T17:17:07.326409+0000","last_fullsized":"2024-04-20T17:17:07.326409+0000","mapping_epoch":171,"log_start":"135'5900","ondisk_log_start":"135'5900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250216448,"num_objects":3818,"num_object_clones":0,"num_object_copies":7636,"num_objects_missing_on_primary":3377,"num_objects_missing":3377,"num_objects_degraded":0,"num_objects_misplaced":6765,"num_objects_unfound":0,"num_objects_dirty":3818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15912,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6044,"num_bytes_recovered":396099584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"0,2,5","objects":11},{"shards":"0,10","objects":430},{"shards":"2,5","objects":3377}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":783,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-20T17:17:06.482378+0000","last_change":"2024-04-20T17:16:40.132382+0000","last_active":"2024-04-20T17:17:06.482378+0000","last_peered":"2024-04-20T17:17:06.482378+0000","last_clean":"2024-04-20T17:17:06.482378+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:17:06.482378+0000","last_undegraded":"2024-04-20T17:17:06.482378+0000","last_fullsized":"2024-04-20T17:17:06.482378+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:16:40.132254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:16:40.132254+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:54.442752+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"294'16262","reported_seq":17011,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-20T17:17:07.851133+0000","last_change":"2024-04-20T17:15:31.396169+0000","last_active":"2024-04-20T17:17:07.851133+0000","last_peered":"2024-04-20T17:17:07.851133+0000","last_clean":"2024-04-20T17:17:07.851133+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:17:07.851133+0000","last_undegraded":"2024-04-20T17:17:07.851133+0000","last_fullsized":"2024-04-20T17:17:07.851133+0000","mapping_epoch":115,"log_start":"135'6200","ondisk_log_start":"135'6200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15333","last_scrub_stamp":"2024-04-20T17:15:31.396069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:31.396069+0000","objects_scrubbed":4919,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:08.119506+0000","scrub_duration":440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261488640,"num_objects":3990,"num_object_clones":0,"num_object_copies":7980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16262,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"292'16346","reported_seq":17091,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-04-20T17:17:03.601117+0000","last_change":"2024-04-20T17:15:35.632022+0000","last_active":"2024-04-20T17:17:03.601117+0000","last_peered":"2024-04-20T17:17:03.601117+0000","last_clean":"2024-04-20T17:17:03.601117+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:17:03.601117+0000","last_undegraded":"2024-04-20T17:17:03.601117+0000","last_fullsized":"2024-04-20T17:17:03.601117+0000","mapping_epoch":115,"log_start":"136'6300","ondisk_log_start":"136'6300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15459","last_scrub_stamp":"2024-04-20T17:15:35.631842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:35.631842+0000","objects_scrubbed":4867,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:06.555706+0000","scrub_duration":426,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260702208,"num_objects":3978,"num_object_clones":0,"num_object_copies":7956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16346,"num_write_kb":650368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"294'20304","reported_seq":21199,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-20T17:17:06.651529+0000","last_change":"2024-04-20T17:16:25.742067+0000","last_active":"2024-04-20T17:17:06.651529+0000","last_peered":"2024-04-20T17:17:06.651529+0000","last_clean":"2024-04-20T17:17:06.651529+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:17:06.651529+0000","last_undegraded":"2024-04-20T17:17:06.651529+0000","last_fullsized":"2024-04-20T17:17:06.651529+0000","mapping_epoch":115,"log_start":"152'10302","ondisk_log_start":"152'10302","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'19891","last_scrub_stamp":"2024-04-20T17:16:25.741880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:25.741880+0000","objects_scrubbed":4262,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:39.901935+0000","scrub_duration":378,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252248064,"num_objects":3849,"num_object_clones":0,"num_object_copies":7698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":641344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"294'16353","reported_seq":17159,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-20T17:17:07.234263+0000","last_change":"2024-04-20T17:15:41.950854+0000","last_active":"2024-04-20T17:17:07.234263+0000","last_peered":"2024-04-20T17:17:07.234263+0000","last_clean":"2024-04-20T17:17:07.234263+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:17:07.234263+0000","last_undegraded":"2024-04-20T17:17:07.234263+0000","last_fullsized":"2024-04-20T17:17:07.234263+0000","mapping_epoch":115,"log_start":"135'6300","ondisk_log_start":"135'6300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'15491","last_scrub_stamp":"2024-04-20T17:15:41.950723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:41.950723+0000","objects_scrubbed":4733,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":448,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253427712,"num_objects":3867,"num_object_clones":0,"num_object_copies":7734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16353,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"294'16150","reported_seq":16882,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-20T17:17:07.819576+0000","last_change":"2024-04-20T17:15:59.551771+0000","last_active":"2024-04-20T17:17:07.819576+0000","last_peered":"2024-04-20T17:17:07.819576+0000","last_clean":"2024-04-20T17:17:07.819576+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:17:07.819576+0000","last_undegraded":"2024-04-20T17:17:07.819576+0000","last_fullsized":"2024-04-20T17:17:07.819576+0000","mapping_epoch":115,"log_start":"135'6100","ondisk_log_start":"135'6100","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'15438","last_scrub_stamp":"2024-04-20T17:15:59.551671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:59.551671+0000","objects_scrubbed":4614,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:10.078708+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255721472,"num_objects":3902,"num_object_clones":0,"num_object_copies":7804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16150,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"294'16283","reported_seq":31621,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-20T17:17:07.837505+0000","last_change":"2024-04-20T17:16:34.421711+0000","last_active":"2024-04-20T17:17:07.837505+0000","last_peered":"2024-04-20T17:17:07.837505+0000","last_clean":"2024-04-20T17:17:07.837505+0000","last_became_active":"2024-04-20T17:10:55.931751+0000","last_became_peered":"2024-04-20T17:10:55.931751+0000","last_unstale":"2024-04-20T17:17:07.837505+0000","last_undegraded":"2024-04-20T17:17:07.837505+0000","last_fullsized":"2024-04-20T17:17:07.837505+0000","mapping_epoch":173,"log_start":"135'6200","ondisk_log_start":"135'6200","created":115,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'15941","last_scrub_stamp":"2024-04-20T17:16:34.421587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:34.421587+0000","objects_scrubbed":4322,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:43.906503+0000","scrub_duration":366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260767744,"num_objects":3979,"num_object_clones":0,"num_object_copies":7958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16283,"num_write_kb":648384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7296,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"294'16366","reported_seq":17162,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-20T17:17:07.239773+0000","last_change":"2024-04-20T17:16:42.499318+0000","last_active":"2024-04-20T17:17:07.239773+0000","last_peered":"2024-04-20T17:17:07.239773+0000","last_clean":"2024-04-20T17:17:07.239773+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:17:07.239773+0000","last_undegraded":"2024-04-20T17:17:07.239773+0000","last_fullsized":"2024-04-20T17:17:07.239773+0000","mapping_epoch":115,"log_start":"136'6300","ondisk_log_start":"136'6300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'16092","last_scrub_stamp":"2024-04-20T17:16:42.499193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:42.499193+0000","objects_scrubbed":4224,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:04.645993+0000","scrub_duration":355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258867200,"num_objects":3950,"num_object_clones":0,"num_object_copies":7900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16366,"num_write_kb":650112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"292'15851","reported_seq":16591,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-04-20T17:17:03.621273+0000","last_change":"2024-04-20T17:15:34.656600+0000","last_active":"2024-04-20T17:17:03.621273+0000","last_peered":"2024-04-20T17:17:03.621273+0000","last_clean":"2024-04-20T17:17:03.621273+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:17:03.621273+0000","last_undegraded":"2024-04-20T17:17:03.621273+0000","last_fullsized":"2024-04-20T17:17:03.621273+0000","mapping_epoch":115,"log_start":"135'5800","ondisk_log_start":"135'5800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15023","last_scrub_stamp":"2024-04-20T17:15:34.656503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:34.656503+0000","objects_scrubbed":4842,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":430,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262995968,"num_objects":4013,"num_object_clones":0,"num_object_copies":8026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15851,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"294'16232","reported_seq":16989,"reported_epoch":294,"state":"active+clean+scrubbing","last_fresh":"2024-04-20T17:17:07.586683+0000","last_change":"2024-04-20T17:17:07.511942+0000","last_active":"2024-04-20T17:17:07.586683+0000","last_peered":"2024-04-20T17:17:07.586683+0000","last_clean":"2024-04-20T17:17:07.586683+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:17:07.586683+0000","last_undegraded":"2024-04-20T17:17:07.586683+0000","last_fullsized":"2024-04-20T17:17:07.586683+0000","mapping_epoch":115,"log_start":"135'6200","ondisk_log_start":"135'6200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15271","last_scrub_stamp":"2024-04-20T17:15:31.794156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:31.794156+0000","objects_scrubbed":899,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":443,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253624320,"num_objects":3870,"num_object_clones":0,"num_object_copies":7740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":643264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"293'15954","reported_seq":16744,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-20T17:17:05.740734+0000","last_change":"2024-04-20T17:17:04.896764+0000","last_active":"2024-04-20T17:17:05.740734+0000","last_peered":"2024-04-20T17:17:05.740734+0000","last_clean":"2024-04-20T17:17:05.740734+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:17:05.740734+0000","last_undegraded":"2024-04-20T17:17:05.740734+0000","last_fullsized":"2024-04-20T17:17:05.740734+0000","mapping_epoch":115,"log_start":"135'5900","ondisk_log_start":"135'5900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"293'15941","last_scrub_stamp":"2024-04-20T17:17:04.896640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:04.896640+0000","objects_scrubbed":3929,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:16.518914+0000","scrub_duration":355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256573464,"num_objects":3916,"num_object_clones":0,"num_object_copies":7832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3916,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15954,"num_write_kb":635777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"293'16032","reported_seq":16695,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-20T17:17:05.610696+0000","last_change":"2024-04-20T17:16:23.593408+0000","last_active":"2024-04-20T17:17:05.610696+0000","last_peered":"2024-04-20T17:17:05.610696+0000","last_clean":"2024-04-20T17:17:05.610696+0000","last_became_active":"2024-04-20T17:12:23.515375+0000","last_became_peered":"2024-04-20T17:12:23.515375+0000","last_unstale":"2024-04-20T17:17:05.610696+0000","last_undegraded":"2024-04-20T17:17:05.610696+0000","last_fullsized":"2024-04-20T17:17:05.610696+0000","mapping_epoch":202,"log_start":"135'6000","ondisk_log_start":"135'6000","created":115,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'15586","last_scrub_stamp":"2024-04-20T17:16:23.593300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:23.593300+0000","objects_scrubbed":4410,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:33.363583+0000","scrub_duration":446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259784704,"num_objects":3964,"num_object_clones":0,"num_object_copies":7928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16032,"num_write_kb":639872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":4136173592,"num_objects":63114,"num_object_clones":0,"num_object_copies":126228,"num_objects_missing_on_primary":3377,"num_objects_missing":3377,"num_objects_degraded":0,"num_objects_misplaced":6765,"num_objects_unfound":0,"num_objects_dirty":63114,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":258870,"num_write_kb":10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25828,"num_bytes_recovered":1692663808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8890228736,"data_stored":8890220592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},"log_size":160763,"ondisk_log_size":160763,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991369,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":828740,"kb_used_data":757304,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92916412,"statfs":{"total":95995035648,"available":95146405888,"internally_reserved":0,"allocated":775479296,"data_stored":774528668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":73136580},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253006,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":931360,"kb_used_data":796056,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":92813792,"statfs":{"total":95995035648,"available":95041323008,"internally_reserved":0,"allocated":815161344,"data_stored":814226174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":138541504},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612756,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1152244,"kb_used_data":1016936,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":92592908,"statfs":{"total":95995035648,"available":94815137792,"internally_reserved":0,"allocated":1041342464,"data_stored":1040397314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":138541504},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762808985,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1128868,"kb_used_data":1017176,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92616284,"statfs":{"total":95995035648,"available":94839074816,"internally_reserved":0,"allocated":1041588224,"data_stored":1040653074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168735,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":827524,"kb_used_data":764344,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92917628,"statfs":{"total":95995035648,"available":95147651072,"internally_reserved":0,"allocated":782688256,"data_stored":781737648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":621112,"kb_used_data":509424,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93124040,"statfs":{"total":95995035648,"available":95359016960,"internally_reserved":0,"allocated":521650176,"data_stored":520703120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575037,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":608640,"kb_used_data":505264,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93136512,"statfs":{"total":95995035648,"available":95371788288,"internally_reserved":0,"allocated":517390336,"data_stored":516443280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738562,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":925440,"kb_used_data":785968,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92819712,"statfs":{"total":95995035648,"available":95047385088,"internally_reserved":0,"allocated":804831232,"data_stored":803884176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,2,0,0,2,3,5,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393850,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":734188,"kb_used_data":685796,"kb_used_omap":202,"kb_used_meta":48181,"kb_avail":93010964,"statfs":{"total":95995035648,"available":95243227136,"internally_reserved":0,"allocated":702255104,"data_stored":701314026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":49337815},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182578,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":581936,"kb_used_data":510500,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93163216,"statfs":{"total":95995035648,"available":95399133184,"internally_reserved":0,"allocated":522752000,"data_stored":521810922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986349,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":280308,"kb_used_data":253288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93464844,"statfs":{"total":95995035648,"available":95708000256,"internally_reserved":0,"allocated":259366912,"data_stored":258421762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790119,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1221740,"kb_used_data":1118372,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92523412,"statfs":{"total":95995035648,"available":94743973888,"internally_reserved":0,"allocated":1145212928,"data_stored":1144271850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":799604736,"data_stored":799604736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":514195456,"data_stored":514195456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":699990016,"data_stored":699990016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":519569408,"data_stored":519569408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":256774144,"data_stored":256770072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1141440512,"data_stored":1141440512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":518586368,"data_stored":518586368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":777977856,"data_stored":777977856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1038483456,"data_stored":1038483456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1038880768,"data_stored":1038876696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":811859968,"data_stored":811859968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":772866048,"data_stored":772866048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:17:11.500 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:17:11.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:11.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:17:11.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:11 smithi106 ceph-mon[44423]: pgmap v816: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.8 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 179 op/s; 6765/125866 objects misplaced (5.375%); 880 KiB/s, 13 objects/s recovering 2024-04-20T17:17:11.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:11 smithi106 ceph-mon[44423]: 2.0 scrub ok 2024-04-20T17:17:11.547 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:17:11.604 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:17:11.637 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:11.786 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:17:11.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:11.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:17:11.891 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:17:11.920 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:12.118 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:17:12.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:12.121 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:17:12.222 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:17:12.248 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:12.698 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:17:12.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:12.704 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:17:12.805 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:17:12.832 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:13.031 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:17:13.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:13.034 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:17:13.135 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:17:13.161 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:13.179 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:17:13.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:13.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:17:13.221 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:17:13.282 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:17:13.307 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:17:13.307 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:17:13.307 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:17:13.307 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:17:13.308 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.308 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:194660 2.3 e163/150 2:c260d96c:::benchmark_data_smithi171_78918_object33666:head v 163'11958, mlcod=122'2109)", 2024-04-20T17:17:13.308 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:29.397421+0000", 2024-04-20T17:17:13.308 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 403.90630987600002, 2024-04-20T17:17:13.308 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.039947886000000002, 2024-04-20T17:17:13.308 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:17:13.308 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:17:13.308 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:13.308 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:17:13.308 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.308 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:17:13.308 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.397421+0000", 2024-04-20T17:17:13.308 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.308 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.309 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.309 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:17:13.309 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.397421+0000", 2024-04-20T17:17:13.309 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.309 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.309 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.309 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:17:13.309 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.397423+0000", 2024-04-20T17:17:13.309 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7630000000000001e-06 2024-04-20T17:17:13.309 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.309 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.309 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:17:13.309 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.397432+0000", 2024-04-20T17:17:13.309 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.541e-06 2024-04-20T17:17:13.310 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.310 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.310 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:17:13.310 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.397443+0000", 2024-04-20T17:17:13.310 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1776999999999999e-05 2024-04-20T17:17:13.310 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.310 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.310 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:17:13.310 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.397450+0000", 2024-04-20T17:17:13.310 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7179999999999999e-06 2024-04-20T17:17:13.310 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.310 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.310 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:17:13.311 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.404629+0000", 2024-04-20T17:17:13.311 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.007178551 2024-04-20T17:17:13.311 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.311 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.311 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:17:13.311 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.404652+0000", 2024-04-20T17:17:13.311 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3575e-05 2024-04-20T17:17:13.311 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.311 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.311 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:17:13.311 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.437330+0000", 2024-04-20T17:17:13.311 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032677465000000003 2024-04-20T17:17:13.311 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.312 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.312 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:17:13.312 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.437369+0000", 2024-04-20T17:17:13.312 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9496000000000003e-05 2024-04-20T17:17:13.312 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.312 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:17:13.312 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.312 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.312 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.312 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:194796 2.3 e163/150 2:c824a312:::benchmark_data_smithi171_78918_object33802:head v 163'11966, mlcod=122'2112)", 2024-04-20T17:17:13.312 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:29.533354+0000", 2024-04-20T17:17:13.312 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 403.77037728800002, 2024-04-20T17:17:13.312 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0406288, 2024-04-20T17:17:13.313 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:17:13.313 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:17:13.313 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:13.313 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:17:13.313 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.313 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:17:13.313 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.533354+0000", 2024-04-20T17:17:13.313 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.313 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.313 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.313 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:17:13.313 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.533354+0000", 2024-04-20T17:17:13.313 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.314 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.314 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.314 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:17:13.314 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.533355+0000", 2024-04-20T17:17:13.314 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3999999999999999e-06 2024-04-20T17:17:13.314 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.314 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.314 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:17:13.314 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.533367+0000", 2024-04-20T17:17:13.314 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1691e-05 2024-04-20T17:17:13.314 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.314 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.314 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:17:13.314 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.533377+0000", 2024-04-20T17:17:13.314 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.8199999999999992e-06 2024-04-20T17:17:13.315 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.315 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.315 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:17:13.315 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.533393+0000", 2024-04-20T17:17:13.315 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6192999999999999e-05 2024-04-20T17:17:13.315 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.315 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.315 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:17:13.315 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.542134+0000", 2024-04-20T17:17:13.315 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0087409270000000008 2024-04-20T17:17:13.315 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.315 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.315 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:17:13.315 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.542162+0000", 2024-04-20T17:17:13.315 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7766000000000001e-05 2024-04-20T17:17:13.316 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.316 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.316 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:17:13.316 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.573938+0000", 2024-04-20T17:17:13.316 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.031776739999999998 2024-04-20T17:17:13.316 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.316 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.316 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:17:13.316 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.573983+0000", 2024-04-20T17:17:13.316 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4263e-05 2024-04-20T17:17:13.316 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.316 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:17:13.316 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.316 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.316 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.316 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:194810 2.3 e163/150 2:c314b47a:::benchmark_data_smithi171_78918_object33816:head v 163'11968, mlcod=122'2112)", 2024-04-20T17:17:13.317 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:29.541537+0000", 2024-04-20T17:17:13.317 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 403.76219379100002, 2024-04-20T17:17:13.317 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.046523396000000002, 2024-04-20T17:17:13.317 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:17:13.317 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:17:13.317 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:13.317 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:17:13.317 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.317 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:17:13.317 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.541537+0000", 2024-04-20T17:17:13.317 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.317 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.317 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.317 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:17:13.317 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.541537+0000", 2024-04-20T17:17:13.317 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.318 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.318 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.318 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:17:13.318 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.541539+0000", 2024-04-20T17:17:13.318 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4300000000000001e-06 2024-04-20T17:17:13.318 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.318 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.318 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:17:13.318 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.541547+0000", 2024-04-20T17:17:13.318 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1049999999999995e-06 2024-04-20T17:17:13.318 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.318 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.318 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:17:13.319 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.541558+0000", 2024-04-20T17:17:13.319 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0915e-05 2024-04-20T17:17:13.319 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.319 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.319 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:17:13.319 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.541565+0000", 2024-04-20T17:17:13.319 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8469999999999998e-06 2024-04-20T17:17:13.319 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.319 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.319 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:17:13.319 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.557126+0000", 2024-04-20T17:17:13.319 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015561478 2024-04-20T17:17:13.319 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.319 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.319 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:17:13.319 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.557145+0000", 2024-04-20T17:17:13.320 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8454e-05 2024-04-20T17:17:13.320 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.320 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.320 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:17:13.320 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.588028+0000", 2024-04-20T17:17:13.320 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030883058000000001 2024-04-20T17:17:13.320 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.320 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.320 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:17:13.320 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.588061+0000", 2024-04-20T17:17:13.320 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3108999999999998e-05 2024-04-20T17:17:13.320 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.320 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:17:13.320 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.320 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.320 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.321 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:194816 2.3 e163/150 2:c4e181df:::benchmark_data_smithi171_78918_object33822:head v 163'11969, mlcod=122'2112)", 2024-04-20T17:17:13.321 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:29.545777+0000", 2024-04-20T17:17:13.321 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 403.75795434299999, 2024-04-20T17:17:13.321 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.042317029999999999, 2024-04-20T17:17:13.321 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:17:13.321 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:17:13.321 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:13.321 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:17:13.321 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.321 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:17:13.321 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.545777+0000", 2024-04-20T17:17:13.321 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.321 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.321 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.321 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:17:13.322 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.545777+0000", 2024-04-20T17:17:13.322 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.322 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.322 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.322 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:17:13.322 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.545779+0000", 2024-04-20T17:17:13.322 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.717e-06 2024-04-20T17:17:13.322 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.322 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.322 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:17:13.322 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.545789+0000", 2024-04-20T17:17:13.322 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0498e-05 2024-04-20T17:17:13.322 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.322 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.323 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:17:13.323 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.545803+0000", 2024-04-20T17:17:13.323 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3423e-05 2024-04-20T17:17:13.323 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.323 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.323 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:17:13.323 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.545810+0000", 2024-04-20T17:17:13.323 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8869999999999992e-06 2024-04-20T17:17:13.323 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.323 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.323 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:17:13.323 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.565547+0000", 2024-04-20T17:17:13.323 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019736581999999999 2024-04-20T17:17:13.323 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.324 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.324 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:17:13.324 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.565579+0000", 2024-04-20T17:17:13.324 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2079999999999998e-05 2024-04-20T17:17:13.324 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.324 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.324 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:17:13.324 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.588064+0000", 2024-04-20T17:17:13.324 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022484671000000001 2024-04-20T17:17:13.324 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.324 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.324 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:17:13.324 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:29.588094+0000", 2024-04-20T17:17:13.324 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0171999999999998e-05 2024-04-20T17:17:13.325 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.325 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:17:13.325 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.325 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.325 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.325 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:196209 2.3 e166/150 2:ca2be9e5:::benchmark_data_smithi171_78918_object35215:head v 166'12051, mlcod=122'2264)", 2024-04-20T17:17:13.325 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:33.094691+0000", 2024-04-20T17:17:13.325 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 400.20904002100002, 2024-04-20T17:17:13.325 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043283582000000001, 2024-04-20T17:17:13.325 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:17:13.325 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:17:13.325 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:13.325 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:17:13.325 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.326 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:17:13.326 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.094691+0000", 2024-04-20T17:17:13.326 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.326 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.326 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.326 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:17:13.326 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.094691+0000", 2024-04-20T17:17:13.326 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.326 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.326 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.326 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:17:13.326 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.094692+0000", 2024-04-20T17:17:13.326 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9599999999999999e-07 2024-04-20T17:17:13.326 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.327 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.327 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:17:13.327 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.094696+0000", 2024-04-20T17:17:13.327 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3649999999999997e-06 2024-04-20T17:17:13.327 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.327 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.327 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:17:13.327 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.094701+0000", 2024-04-20T17:17:13.327 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6419999999999998e-06 2024-04-20T17:17:13.327 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.327 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.327 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:17:13.327 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.094705+0000", 2024-04-20T17:17:13.327 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4819999999999996e-06 2024-04-20T17:17:13.328 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.328 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.328 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:17:13.328 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.103126+0000", 2024-04-20T17:17:13.328 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0084207180000000006 2024-04-20T17:17:13.328 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.328 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.328 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:17:13.328 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.103155+0000", 2024-04-20T17:17:13.328 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9300000000000001e-05 2024-04-20T17:17:13.328 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.328 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.328 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:17:13.328 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.137934+0000", 2024-04-20T17:17:13.328 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.034779139000000001 2024-04-20T17:17:13.329 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.329 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.329 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:17:13.329 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.137975+0000", 2024-04-20T17:17:13.329 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0439999999999999e-05 2024-04-20T17:17:13.329 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.329 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:17:13.329 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.329 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.329 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.329 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:196690 2.3 e166/150 2:cb6b9f79:::benchmark_data_smithi171_78918_object35696:head v 166'12075, mlcod=122'2273)", 2024-04-20T17:17:13.329 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:33.527490+0000", 2024-04-20T17:17:13.329 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 399.77624077600001, 2024-04-20T17:17:13.329 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043966988999999998, 2024-04-20T17:17:13.329 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:17:13.329 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:17:13.330 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:13.330 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:17:13.330 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.330 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:17:13.330 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.527490+0000", 2024-04-20T17:17:13.330 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.330 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.330 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.330 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:17:13.330 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.527490+0000", 2024-04-20T17:17:13.330 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.330 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.330 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.331 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:17:13.331 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.527491+0000", 2024-04-20T17:17:13.331 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4900000000000005e-07 2024-04-20T17:17:13.331 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.331 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.331 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:17:13.331 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.527496+0000", 2024-04-20T17:17:13.331 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1309999999999997e-06 2024-04-20T17:17:13.331 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.331 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.331 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:17:13.331 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.527503+0000", 2024-04-20T17:17:13.331 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9569999999999996e-06 2024-04-20T17:17:13.331 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.332 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.332 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:17:13.332 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.527508+0000", 2024-04-20T17:17:13.332 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.118e-06 2024-04-20T17:17:13.332 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.332 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.332 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:17:13.332 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.527920+0000", 2024-04-20T17:17:13.332 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00041207700000000001 2024-04-20T17:17:13.332 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.332 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.332 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:17:13.332 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.527944+0000", 2024-04-20T17:17:13.332 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3935e-05 2024-04-20T17:17:13.333 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.333 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.333 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:17:13.333 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.571413+0000", 2024-04-20T17:17:13.333 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043468967999999997 2024-04-20T17:17:13.333 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.333 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.333 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:17:13.333 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.571457+0000", 2024-04-20T17:17:13.333 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4054000000000002e-05 2024-04-20T17:17:13.333 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.333 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:17:13.333 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.333 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.333 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.334 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:197148 2.3 e166/150 2:cc8bb281:::benchmark_data_smithi171_78918_object36154:head v 166'12100, mlcod=122'2280)", 2024-04-20T17:17:13.334 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:33.965330+0000", 2024-04-20T17:17:13.334 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 399.33840132799997, 2024-04-20T17:17:13.334 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044780069999999998, 2024-04-20T17:17:13.334 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:17:13.334 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:17:13.334 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:13.334 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:17:13.334 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.334 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:17:13.334 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.965330+0000", 2024-04-20T17:17:13.334 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.334 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.334 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.335 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:17:13.335 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.965330+0000", 2024-04-20T17:17:13.335 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.335 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.335 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.335 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:17:13.335 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.965331+0000", 2024-04-20T17:17:13.335 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4640000000000001e-06 2024-04-20T17:17:13.335 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.335 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.335 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:17:13.335 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.965343+0000", 2024-04-20T17:17:13.335 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1589e-05 2024-04-20T17:17:13.335 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.336 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.336 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:17:13.336 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.965357+0000", 2024-04-20T17:17:13.336 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4555999999999999e-05 2024-04-20T17:17:13.336 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.336 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.336 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:17:13.336 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.965363+0000", 2024-04-20T17:17:13.336 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.648e-06 2024-04-20T17:17:13.336 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.336 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.336 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:17:13.336 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.965422+0000", 2024-04-20T17:17:13.336 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8631000000000002e-05 2024-04-20T17:17:13.336 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.337 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.337 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:17:13.337 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.965445+0000", 2024-04-20T17:17:13.337 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3706999999999999e-05 2024-04-20T17:17:13.337 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.337 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.337 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:17:13.337 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.010066+0000", 2024-04-20T17:17:13.337 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044620303 2024-04-20T17:17:13.337 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.337 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.337 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:17:13.337 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.010110+0000", 2024-04-20T17:17:13.338 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4172000000000003e-05 2024-04-20T17:17:13.338 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.338 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:17:13.338 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.338 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.338 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.338 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:197153 2.3 e166/150 2:cb452060:::benchmark_data_smithi171_78918_object36159:head v 166'12101, mlcod=122'2280)", 2024-04-20T17:17:13.338 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:33.968794+0000", 2024-04-20T17:17:13.338 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 399.33493753499999, 2024-04-20T17:17:13.338 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.041343306000000003, 2024-04-20T17:17:13.338 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:17:13.338 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:17:13.338 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:13.338 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:17:13.339 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.339 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:17:13.339 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.968794+0000", 2024-04-20T17:17:13.339 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.339 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.339 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.339 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:17:13.339 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.968794+0000", 2024-04-20T17:17:13.339 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.339 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.339 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.339 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:17:13.339 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.968800+0000", 2024-04-20T17:17:13.339 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6409999999999996e-06 2024-04-20T17:17:13.340 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.340 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.340 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:17:13.340 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.968817+0000", 2024-04-20T17:17:13.340 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6310000000000001e-05 2024-04-20T17:17:13.340 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.340 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.340 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:17:13.340 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.968833+0000", 2024-04-20T17:17:13.340 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6028999999999998e-05 2024-04-20T17:17:13.340 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.340 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.340 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:17:13.340 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.968844+0000", 2024-04-20T17:17:13.341 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1484999999999999e-05 2024-04-20T17:17:13.341 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.341 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.341 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:17:13.341 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.974323+0000", 2024-04-20T17:17:13.341 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0054788629999999996 2024-04-20T17:17:13.341 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.341 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.341 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:17:13.341 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:33.974345+0000", 2024-04-20T17:17:13.341 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1639000000000001e-05 2024-04-20T17:17:13.341 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.341 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.342 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:17:13.342 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.010113+0000", 2024-04-20T17:17:13.342 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.035768056999999999 2024-04-20T17:17:13.342 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.342 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.342 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:17:13.342 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.010137+0000", 2024-04-20T17:17:13.342 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4281999999999999e-05 2024-04-20T17:17:13.342 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.342 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:17:13.342 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.342 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.342 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.343 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:197280 2.3 e166/150 2:c3bfc80e:::benchmark_data_smithi171_78918_object36286:head v 166'12109, mlcod=123'2282)", 2024-04-20T17:17:13.343 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:34.080902+0000", 2024-04-20T17:17:13.343 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 399.22282872400001, 2024-04-20T17:17:13.343 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043969018999999998, 2024-04-20T17:17:13.343 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:17:13.343 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:17:13.343 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:13.343 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:17:13.343 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.343 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:17:13.343 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.080902+0000", 2024-04-20T17:17:13.343 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.343 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.343 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.344 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:17:13.344 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.080902+0000", 2024-04-20T17:17:13.344 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.344 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.344 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.344 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:17:13.344 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.080904+0000", 2024-04-20T17:17:13.344 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.096e-06 2024-04-20T17:17:13.344 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.344 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.344 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:17:13.344 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.080910+0000", 2024-04-20T17:17:13.344 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5239999999999997e-06 2024-04-20T17:17:13.344 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.344 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.345 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:17:13.345 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.080917+0000", 2024-04-20T17:17:13.345 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5409999999999997e-06 2024-04-20T17:17:13.345 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.345 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.345 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:17:13.345 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.080922+0000", 2024-04-20T17:17:13.345 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.989e-06 2024-04-20T17:17:13.345 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.345 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.345 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:17:13.345 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.088738+0000", 2024-04-20T17:17:13.345 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0078159309999999999 2024-04-20T17:17:13.345 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.345 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.346 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:17:13.346 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.088756+0000", 2024-04-20T17:17:13.346 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8604e-05 2024-04-20T17:17:13.346 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.346 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.346 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:17:13.346 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.124825+0000", 2024-04-20T17:17:13.346 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036068950000000002 2024-04-20T17:17:13.346 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.346 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.346 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:17:13.346 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.124872+0000", 2024-04-20T17:17:13.346 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6384000000000002e-05 2024-04-20T17:17:13.346 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.347 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:17:13.347 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.347 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.347 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.347 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:197284 2.3 e166/150 2:c6c18a95:::benchmark_data_smithi171_78918_object36290:head v 166'12110, mlcod=123'2282)", 2024-04-20T17:17:13.347 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:34.085104+0000", 2024-04-20T17:17:13.347 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 399.21862698899997, 2024-04-20T17:17:13.347 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.039799240999999999, 2024-04-20T17:17:13.347 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:17:13.347 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:17:13.347 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:13.347 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:17:13.347 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.347 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:17:13.347 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.085104+0000", 2024-04-20T17:17:13.347 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.348 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.348 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.348 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:17:13.348 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.085104+0000", 2024-04-20T17:17:13.348 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.348 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.348 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.348 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:17:13.348 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.085106+0000", 2024-04-20T17:17:13.348 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.657e-06 2024-04-20T17:17:13.348 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.348 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.348 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:17:13.348 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.085114+0000", 2024-04-20T17:17:13.348 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7559999999999999e-06 2024-04-20T17:17:13.348 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.349 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.349 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:17:13.349 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.085125+0000", 2024-04-20T17:17:13.349 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.119e-05 2024-04-20T17:17:13.349 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.349 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.349 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:17:13.349 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.085132+0000", 2024-04-20T17:17:13.349 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1339999999999998e-06 2024-04-20T17:17:13.349 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.349 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.349 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:17:13.349 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.098328+0000", 2024-04-20T17:17:13.349 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013196505000000001 2024-04-20T17:17:13.349 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.350 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.350 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:17:13.350 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.098352+0000", 2024-04-20T17:17:13.350 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3306e-05 2024-04-20T17:17:13.350 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.350 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.350 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:17:13.350 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.124874+0000", 2024-04-20T17:17:13.350 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026522343 2024-04-20T17:17:13.350 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.350 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.350 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:17:13.351 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.124903+0000", 2024-04-20T17:17:13.351 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9349999999999999e-05 2024-04-20T17:17:13.351 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.351 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:17:13.351 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.351 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.351 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.351 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:197285 2.3 e166/150 2:ce3b9f7d:::benchmark_data_smithi171_78918_object36291:head v 166'12111, mlcod=123'2282)", 2024-04-20T17:17:13.351 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:34.089569+0000", 2024-04-20T17:17:13.351 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 399.21416246799998, 2024-04-20T17:17:13.351 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.042953340999999999, 2024-04-20T17:17:13.351 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:17:13.351 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:17:13.351 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:13.351 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:17:13.352 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.352 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:17:13.352 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.089569+0000", 2024-04-20T17:17:13.352 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.352 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.352 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.352 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:17:13.352 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.089569+0000", 2024-04-20T17:17:13.352 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.352 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.352 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.352 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:17:13.352 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.089570+0000", 2024-04-20T17:17:13.352 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3740000000000001e-06 2024-04-20T17:17:13.352 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.353 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.353 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:17:13.353 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.089577+0000", 2024-04-20T17:17:13.353 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8700000000000003e-06 2024-04-20T17:17:13.353 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.353 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.353 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:17:13.353 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.089589+0000", 2024-04-20T17:17:13.353 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1571e-05 2024-04-20T17:17:13.353 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.353 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.353 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:17:13.353 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.089595+0000", 2024-04-20T17:17:13.353 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2079999999999997e-06 2024-04-20T17:17:13.353 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.354 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.354 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:17:13.354 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.107695+0000", 2024-04-20T17:17:13.354 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018100574000000001 2024-04-20T17:17:13.354 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.354 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.354 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:17:13.354 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.107723+0000", 2024-04-20T17:17:13.354 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7291999999999999e-05 2024-04-20T17:17:13.354 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.354 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.354 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:17:13.354 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.132481+0000", 2024-04-20T17:17:13.355 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024757923000000001 2024-04-20T17:17:13.355 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.355 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.355 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:17:13.355 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.132522+0000", 2024-04-20T17:17:13.355 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1529000000000002e-05 2024-04-20T17:17:13.355 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.355 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:17:13.355 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.355 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.355 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.355 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:197311 2.3 e166/150 2:c92347ee:::benchmark_data_smithi171_78918_object36317:head v 166'12113, mlcod=123'2282)", 2024-04-20T17:17:13.355 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:34.100868+0000", 2024-04-20T17:17:13.355 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 399.20286343999999, 2024-04-20T17:17:13.356 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.047080359000000002, 2024-04-20T17:17:13.356 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:17:13.356 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:17:13.356 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:13.356 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:17:13.356 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.356 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:17:13.356 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.100868+0000", 2024-04-20T17:17:13.356 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.356 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.356 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.356 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:17:13.356 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.100868+0000", 2024-04-20T17:17:13.356 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.356 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.357 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.357 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:17:13.357 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.100870+0000", 2024-04-20T17:17:13.357 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5359999999999999e-06 2024-04-20T17:17:13.357 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.357 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.357 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:17:13.357 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.100882+0000", 2024-04-20T17:17:13.357 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1579999999999999e-05 2024-04-20T17:17:13.357 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.357 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.357 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:17:13.357 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.100893+0000", 2024-04-20T17:17:13.357 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.118e-05 2024-04-20T17:17:13.357 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.358 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.358 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:17:13.358 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.100900+0000", 2024-04-20T17:17:13.358 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3989999999999998e-06 2024-04-20T17:17:13.358 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.358 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.358 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:17:13.358 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.124930+0000", 2024-04-20T17:17:13.358 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024029600000000002 2024-04-20T17:17:13.358 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.358 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.358 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:17:13.358 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.124952+0000", 2024-04-20T17:17:13.358 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2024000000000001e-05 2024-04-20T17:17:13.358 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.359 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.359 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:17:13.359 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.147909+0000", 2024-04-20T17:17:13.359 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022956826999999999 2024-04-20T17:17:13.359 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.359 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.359 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:17:13.359 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.147948+0000", 2024-04-20T17:17:13.359 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9212999999999999e-05 2024-04-20T17:17:13.359 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.359 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:17:13.359 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.359 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.359 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.360 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:197313 2.3 e166/150 2:ce42bf9c:::benchmark_data_smithi171_78918_object36319:head v 166'12114, mlcod=123'2282)", 2024-04-20T17:17:13.360 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:34.104580+0000", 2024-04-20T17:17:13.360 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 399.19915093399999, 2024-04-20T17:17:13.360 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043407387999999998, 2024-04-20T17:17:13.360 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:17:13.360 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:17:13.360 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:13.360 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:17:13.360 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.360 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:17:13.360 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.104580+0000", 2024-04-20T17:17:13.360 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.360 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.360 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.360 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:17:13.361 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.104580+0000", 2024-04-20T17:17:13.361 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.361 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.361 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.361 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:17:13.361 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.104581+0000", 2024-04-20T17:17:13.361 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.203e-06 2024-04-20T17:17:13.361 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.361 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.361 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:17:13.361 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.104588+0000", 2024-04-20T17:17:13.361 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.951e-06 2024-04-20T17:17:13.361 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.361 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.361 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:17:13.362 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.104597+0000", 2024-04-20T17:17:13.362 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6090000000000001e-06 2024-04-20T17:17:13.362 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.362 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.362 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:17:13.362 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.104605+0000", 2024-04-20T17:17:13.362 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9710000000000004e-06 2024-04-20T17:17:13.362 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.362 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.362 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:17:13.362 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.132598+0000", 2024-04-20T17:17:13.362 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.027992929999999999 2024-04-20T17:17:13.362 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.363 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.363 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:17:13.363 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.132623+0000", 2024-04-20T17:17:13.363 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.529e-05 2024-04-20T17:17:13.363 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.363 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.363 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:17:13.363 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.147957+0000", 2024-04-20T17:17:13.363 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015334129 2024-04-20T17:17:13.363 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.363 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.363 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:17:13.363 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.147988+0000", 2024-04-20T17:17:13.363 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0304999999999999e-05 2024-04-20T17:17:13.363 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.364 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:17:13.364 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.364 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.364 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.364 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:197988 2.3 e166/150 2:cafb7b2e:::benchmark_data_smithi171_78918_object36994:head v 166'12151, mlcod=123'2294)", 2024-04-20T17:17:13.364 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:34.765325+0000", 2024-04-20T17:17:13.364 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 398.53840652299999, 2024-04-20T17:17:13.364 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044294795999999997, 2024-04-20T17:17:13.364 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:17:13.364 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:17:13.364 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:13.364 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:17:13.364 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.364 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:17:13.365 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.765325+0000", 2024-04-20T17:17:13.365 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.365 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.365 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.365 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:17:13.365 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.765325+0000", 2024-04-20T17:17:13.365 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.365 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.365 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.365 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:17:13.365 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.765325+0000", 2024-04-20T17:17:13.365 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7100000000000001e-07 2024-04-20T17:17:13.365 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.365 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.366 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:17:13.366 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.765333+0000", 2024-04-20T17:17:13.366 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6040000000000004e-06 2024-04-20T17:17:13.366 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.366 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.366 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:17:13.366 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.765342+0000", 2024-04-20T17:17:13.366 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3600000000000002e-06 2024-04-20T17:17:13.366 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.366 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.366 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:17:13.366 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.765349+0000", 2024-04-20T17:17:13.366 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8349999999999998e-06 2024-04-20T17:17:13.367 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.367 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.367 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:17:13.367 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.775787+0000", 2024-04-20T17:17:13.367 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.01043725 2024-04-20T17:17:13.367 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.367 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.367 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:17:13.367 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.775802+0000", 2024-04-20T17:17:13.367 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5911000000000001e-05 2024-04-20T17:17:13.367 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.367 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.367 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:17:13.367 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.809585+0000", 2024-04-20T17:17:13.368 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.033782464999999998 2024-04-20T17:17:13.368 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.368 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.368 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:17:13.368 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.809619+0000", 2024-04-20T17:17:13.368 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4600000000000001e-05 2024-04-20T17:17:13.368 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.368 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:17:13.368 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.368 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.368 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.368 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:197989 2.3 e166/150 2:c00840d5:::benchmark_data_smithi171_78918_object36995:head v 166'12152, mlcod=123'2294)", 2024-04-20T17:17:13.368 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:34.769894+0000", 2024-04-20T17:17:13.368 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 398.53383753600002, 2024-04-20T17:17:13.369 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.039770080999999999, 2024-04-20T17:17:13.369 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:17:13.369 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:17:13.369 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:13.369 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:17:13.369 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.369 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:17:13.369 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.769894+0000", 2024-04-20T17:17:13.369 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.369 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.369 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.369 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:17:13.369 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.769894+0000", 2024-04-20T17:17:13.369 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.369 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.370 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.370 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:17:13.370 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.769895+0000", 2024-04-20T17:17:13.370 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7399999999999991e-07 2024-04-20T17:17:13.370 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.370 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.370 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:17:13.370 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.769901+0000", 2024-04-20T17:17:13.370 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3559999999999998e-06 2024-04-20T17:17:13.370 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.370 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.370 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:17:13.370 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.769909+0000", 2024-04-20T17:17:13.371 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6310000000000003e-06 2024-04-20T17:17:13.371 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.371 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.371 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:17:13.371 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.769916+0000", 2024-04-20T17:17:13.371 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5700000000000004e-06 2024-04-20T17:17:13.371 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.371 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.371 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:17:13.371 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.784392+0000", 2024-04-20T17:17:13.371 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014475449 2024-04-20T17:17:13.371 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.371 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.371 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:17:13.371 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.784419+0000", 2024-04-20T17:17:13.372 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7633000000000001e-05 2024-04-20T17:17:13.372 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.372 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.372 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:17:13.372 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.809623+0000", 2024-04-20T17:17:13.372 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.025203204999999999 2024-04-20T17:17:13.372 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.372 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.372 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:17:13.372 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:34.809664+0000", 2024-04-20T17:17:13.372 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1263000000000001e-05 2024-04-20T17:17:13.372 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.372 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:17:13.372 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.372 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.373 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.373 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:198153 2.3 e166/150 2:c93b993b:::benchmark_data_smithi171_78918_object37159:head v 166'12166, mlcod=123'2301)", 2024-04-20T17:17:13.373 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:35.037480+0000", 2024-04-20T17:17:13.373 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 398.26625082300001, 2024-04-20T17:17:13.373 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.041818424999999999, 2024-04-20T17:17:13.373 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:17:13.373 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:17:13.373 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:13.373 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:17:13.373 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.373 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:17:13.373 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.037480+0000", 2024-04-20T17:17:13.373 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.373 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.373 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.374 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:17:13.374 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.037480+0000", 2024-04-20T17:17:13.374 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.374 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.374 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.374 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:17:13.374 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.037482+0000", 2024-04-20T17:17:13.374 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.319e-06 2024-04-20T17:17:13.374 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.374 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.374 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:17:13.374 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.037488+0000", 2024-04-20T17:17:13.374 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8189999999999997e-06 2024-04-20T17:17:13.374 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.374 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.375 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:17:13.375 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.037496+0000", 2024-04-20T17:17:13.375 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.7329999999999992e-06 2024-04-20T17:17:13.375 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.375 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.375 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:17:13.375 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.037502+0000", 2024-04-20T17:17:13.375 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.818e-06 2024-04-20T17:17:13.375 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.375 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.375 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:17:13.375 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.048850+0000", 2024-04-20T17:17:13.375 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011348008999999999 2024-04-20T17:17:13.375 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.376 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.376 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:17:13.376 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.048868+0000", 2024-04-20T17:17:13.376 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8148e-05 2024-04-20T17:17:13.376 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.376 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.376 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:17:13.376 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.079255+0000", 2024-04-20T17:17:13.376 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030387038000000002 2024-04-20T17:17:13.376 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.376 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.376 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:17:13.376 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:35.079299+0000", 2024-04-20T17:17:13.376 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3541000000000002e-05 2024-04-20T17:17:13.376 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.377 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:17:13.377 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.377 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.377 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.377 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:200778 2.3 e166/150 2:ca9ecfbd:::benchmark_data_smithi171_78918_object39784:head v 166'12349, mlcod=124'2485)", 2024-04-20T17:17:13.377 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:38.271498+0000", 2024-04-20T17:17:13.377 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 395.03223293999997, 2024-04-20T17:17:13.377 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.040696903, 2024-04-20T17:17:13.377 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:17:13.377 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:17:13.377 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:13.377 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:17:13.377 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.377 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:17:13.377 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.271498+0000", 2024-04-20T17:17:13.377 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.378 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.378 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.378 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:17:13.378 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.271498+0000", 2024-04-20T17:17:13.378 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.378 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.378 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.378 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:17:13.378 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.271499+0000", 2024-04-20T17:17:13.378 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7599999999999996e-07 2024-04-20T17:17:13.378 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.378 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.378 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:17:13.378 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.271505+0000", 2024-04-20T17:17:13.379 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1859999999999997e-06 2024-04-20T17:17:13.379 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.379 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.379 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:17:13.379 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.271513+0000", 2024-04-20T17:17:13.379 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0269999999999995e-06 2024-04-20T17:17:13.379 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.379 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.379 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:17:13.379 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.271519+0000", 2024-04-20T17:17:13.379 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6060000000000002e-06 2024-04-20T17:17:13.379 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.379 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.379 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:17:13.380 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.279364+0000", 2024-04-20T17:17:13.380 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0078446799999999997 2024-04-20T17:17:13.380 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.380 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.380 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:17:13.380 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.279381+0000", 2024-04-20T17:17:13.380 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7142999999999998e-05 2024-04-20T17:17:13.380 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.380 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.380 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:17:13.380 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.312170+0000", 2024-04-20T17:17:13.380 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032789670999999999 2024-04-20T17:17:13.380 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.380 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.380 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:17:13.381 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.312195+0000", 2024-04-20T17:17:13.381 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4814e-05 2024-04-20T17:17:13.381 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.381 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:17:13.381 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.381 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.381 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.381 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:201134 2.3 e166/150 2:c429e5fa:::benchmark_data_smithi171_78918_object40140:head v 166'12367, mlcod=124'2502)", 2024-04-20T17:17:13.381 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:38.559710+0000", 2024-04-20T17:17:13.381 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 394.74402152499999, 2024-04-20T17:17:13.381 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.048219026999999998, 2024-04-20T17:17:13.381 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:17:13.381 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:17:13.381 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:13.382 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:17:13.382 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.382 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:17:13.382 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.559710+0000", 2024-04-20T17:17:13.382 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.382 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.382 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.382 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:17:13.382 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.559710+0000", 2024-04-20T17:17:13.382 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.382 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.382 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.382 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:17:13.383 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.559712+0000", 2024-04-20T17:17:13.383 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0679999999999999e-06 2024-04-20T17:17:13.383 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.383 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.383 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:17:13.383 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.559721+0000", 2024-04-20T17:17:13.383 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1379999999999996e-06 2024-04-20T17:17:13.383 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.383 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.383 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:17:13.383 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.559733+0000", 2024-04-20T17:17:13.383 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2199e-05 2024-04-20T17:17:13.383 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.383 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.383 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:17:13.384 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.559742+0000", 2024-04-20T17:17:13.384 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9740000000000008e-06 2024-04-20T17:17:13.384 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.384 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.384 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:17:13.384 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.575041+0000", 2024-04-20T17:17:13.384 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015299046 2024-04-20T17:17:13.384 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.384 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.384 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:17:13.384 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.575057+0000", 2024-04-20T17:17:13.384 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5900999999999999e-05 2024-04-20T17:17:13.384 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.384 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.385 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:17:13.385 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.607891+0000", 2024-04-20T17:17:13.385 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032834224000000002 2024-04-20T17:17:13.385 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.385 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.385 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:17:13.385 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:38.607929+0000", 2024-04-20T17:17:13.385 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7477e-05 2024-04-20T17:17:13.385 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.385 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:17:13.385 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.385 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.385 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.385 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:201680 2.3 e166/150 2:cf5ca5b2:::benchmark_data_smithi171_78918_object40686:head v 166'12411, mlcod=124'2544)", 2024-04-20T17:17:13.385 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:39.306758+0000", 2024-04-20T17:17:13.386 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 393.99697362199998, 2024-04-20T17:17:13.386 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.040304325000000002, 2024-04-20T17:17:13.386 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:17:13.386 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:17:13.386 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:13.386 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:17:13.386 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.386 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:17:13.386 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.306758+0000", 2024-04-20T17:17:13.386 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.386 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.386 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.386 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:17:13.386 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.306758+0000", 2024-04-20T17:17:13.387 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.387 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.387 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.387 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:17:13.387 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.306759+0000", 2024-04-20T17:17:13.387 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.141e-06 2024-04-20T17:17:13.387 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.387 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.387 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:17:13.387 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.306765+0000", 2024-04-20T17:17:13.387 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8889999999999999e-06 2024-04-20T17:17:13.387 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.387 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.387 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:17:13.387 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.306772+0000", 2024-04-20T17:17:13.388 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2599999999999999e-06 2024-04-20T17:17:13.388 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.388 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.388 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:17:13.388 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.306778+0000", 2024-04-20T17:17:13.388 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1890000000000004e-06 2024-04-20T17:17:13.388 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.388 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.388 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:17:13.388 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.306829+0000", 2024-04-20T17:17:13.388 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0553999999999997e-05 2024-04-20T17:17:13.388 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.388 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.388 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:17:13.388 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.306853+0000", 2024-04-20T17:17:13.389 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4615999999999999e-05 2024-04-20T17:17:13.389 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.389 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.389 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:17:13.389 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.347027+0000", 2024-04-20T17:17:13.389 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.040173826000000003 2024-04-20T17:17:13.389 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.389 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.389 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:17:13.389 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:39.347062+0000", 2024-04-20T17:17:13.389 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.485e-05 2024-04-20T17:17:13.389 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.389 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:17:13.389 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.390 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.390 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.390 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24736.0:203113 2.3 e166/150 2:c71a3208:::benchmark_data_smithi171_78918_object42119:head v 166'12491, mlcod=124'2628)", 2024-04-20T17:17:13.390 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:10:40.855113+0000", 2024-04-20T17:17:13.390 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 392.44861853399999, 2024-04-20T17:17:13.390 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044317189999999999, 2024-04-20T17:17:13.390 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:17:13.390 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:17:13.390 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:13.390 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:17:13.390 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.390 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:17:13.390 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.855113+0000", 2024-04-20T17:17:13.390 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.390 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.391 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.391 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:17:13.391 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.855113+0000", 2024-04-20T17:17:13.391 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:17:13.391 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.391 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.391 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:17:13.391 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.855114+0000", 2024-04-20T17:17:13.391 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.649e-06 2024-04-20T17:17:13.391 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.391 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.391 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:17:13.391 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.855121+0000", 2024-04-20T17:17:13.391 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.985e-06 2024-04-20T17:17:13.392 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.392 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.392 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:17:13.392 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.855131+0000", 2024-04-20T17:17:13.392 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7890000000000006e-06 2024-04-20T17:17:13.392 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.392 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.392 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:17:13.392 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.855139+0000", 2024-04-20T17:17:13.392 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8949999999999998e-06 2024-04-20T17:17:13.392 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.392 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.392 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:17:13.392 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.865644+0000", 2024-04-20T17:17:13.393 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010505393999999999 2024-04-20T17:17:13.393 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.393 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.393 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:17:13.393 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.865672+0000", 2024-04-20T17:17:13.393 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8022e-05 2024-04-20T17:17:13.393 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.393 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.393 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:17:13.393 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.899392+0000", 2024-04-20T17:17:13.393 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.033719879000000001 2024-04-20T17:17:13.393 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:17:13.393 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:17:13.393 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:17:13.393 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:10:40.899430+0000", 2024-04-20T17:17:13.393 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7577000000000003e-05 2024-04-20T17:17:13.394 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.394 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:17:13.394 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.394 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:17:13.394 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:17:13.394 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:17:13.395 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:13 smithi171 ceph-mon[44189]: 2.2 scrub starts 2024-04-20T17:17:13.395 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:13 smithi171 ceph-mon[44189]: pgmap v817: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.8 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 159 op/s; 6765/125130 objects misplaced (5.406%); 448 KiB/s, 6 objects/s recovering 2024-04-20T17:17:13.395 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:13 smithi171 ceph-mon[44189]: 2.2 scrub ok 2024-04-20T17:17:13.395 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:13 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/876068983' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:17:13.395 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:13 smithi171 ceph-mon[44189]: from='client.25475 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:17:13.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:13.396 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:13.396 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:17:13.396 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:17:13.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:13 smithi005 ceph-mon[39562]: 2.2 scrub starts 2024-04-20T17:17:13.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:13 smithi005 ceph-mon[39562]: pgmap v817: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.8 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 159 op/s; 6765/125130 objects misplaced (5.406%); 448 KiB/s, 6 objects/s recovering 2024-04-20T17:17:13.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:13 smithi005 ceph-mon[39562]: 2.2 scrub ok 2024-04-20T17:17:13.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:13 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/876068983' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:17:13.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:13 smithi005 ceph-mon[39562]: from='client.25475 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:17:13.497 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:17:13.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:13 smithi106 ceph-mon[44423]: 2.2 scrub starts 2024-04-20T17:17:13.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:13 smithi106 ceph-mon[44423]: pgmap v817: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.8 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 159 op/s; 6765/125130 objects misplaced (5.406%); 448 KiB/s, 6 objects/s recovering 2024-04-20T17:17:13.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:13 smithi106 ceph-mon[44423]: 2.2 scrub ok 2024-04-20T17:17:13.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:13 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/876068983' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:17:13.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:13 smithi106 ceph-mon[44423]: from='client.25475 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:17:13.532 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:13.571 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:17:13.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:13.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:17:13.680 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:17:13.710 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:14.093 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-20T17:17:14.160 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:17:14.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:14.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:17:14.264 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:17:14.292 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:14.314 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:17:14.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:14.319 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:17:14.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:14 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:17:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:14 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:17:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:14 smithi171 ceph-mon[44189]: from='client.25475 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:17:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:14 smithi171 ceph-mon[44189]: osdmap e295: 12 total, 12 up, 12 in 2024-04-20T17:17:14.375 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:17:14.419 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:17:14.446 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:14.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:14 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:17:14.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:14 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:17:14.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:14 smithi005 ceph-mon[39562]: from='client.25475 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:17:14.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:14 smithi005 ceph-mon[39562]: osdmap e295: 12 total, 12 up, 12 in 2024-04-20T17:17:14.464 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:17:14.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:14.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:17:14.492 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:17:14.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:14 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:17:14.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:14 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:17:14.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:14 smithi106 ceph-mon[44423]: from='client.25475 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:17:14.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:14 smithi106 ceph-mon[44423]: osdmap e295: 12 total, 12 up, 12 in 2024-04-20T17:17:14.568 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:17:14.594 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:14.613 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:17:14.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:14.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:17:14.717 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:17:14.743 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:14.762 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:17:14.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:14.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:17:14.866 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:17:14.898 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:14.972 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:17:14.998 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:17:15.023 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:17:15.033 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:17:15.062 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:17:15.084 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:17:15.111 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:17:15.137 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:17:15.165 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:17:15.192 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:17:15.212 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:17:15.234 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:17:15.254 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:17:15.256 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:17:15.256 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:17:15.256 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:17:15.256 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:17:15.256 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:17:15.256 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:17:15.256 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:17:15.256 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:17:15.257 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:17:15.257 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:17:15.257 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:17:15.257 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:17:15.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:15 smithi171 ceph-mon[44189]: pgmap v819: 17 pgs: 1 active+recovering, 16 active+clean; 3.8 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 198 op/s; 6651/123898 objects misplaced (5.368%); 877 KiB/s, 13 objects/s recovering 2024-04-20T17:17:15.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:15 smithi171 ceph-mon[44189]: 2.5 scrub starts 2024-04-20T17:17:15.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:15 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/804624042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:17:15.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:15.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:15.351 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:17:15.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:15.354 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:17:15.455 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:17:15.480 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:15 smithi106 ceph-mon[44423]: pgmap v819: 17 pgs: 1 active+recovering, 16 active+clean; 3.8 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 198 op/s; 6651/123898 objects misplaced (5.368%); 877 KiB/s, 13 objects/s recovering 2024-04-20T17:17:15.480 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:15 smithi106 ceph-mon[44423]: 2.5 scrub starts 2024-04-20T17:17:15.480 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:15 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/804624042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:17:15.480 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:15.480 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:15.481 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:15.501 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:17:15.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:15.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:17:15.605 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:17:15.633 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:15.668 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:15 smithi005 ceph-mon[39562]: pgmap v819: 17 pgs: 1 active+recovering, 16 active+clean; 3.8 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 198 op/s; 6651/123898 objects misplaced (5.368%); 877 KiB/s, 13 objects/s recovering 2024-04-20T17:17:15.668 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:15 smithi005 ceph-mon[39562]: 2.5 scrub starts 2024-04-20T17:17:15.669 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:15 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/804624042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:17:15.669 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:15.669 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:15.883 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:17:15.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:15.886 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:17:15.986 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:17:16.014 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:16.032 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:17:16.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:16.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:17:16.136 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:17:16.166 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:16.186 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:17:16.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:16.189 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:17:16.289 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:17:16.321 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:16.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:16 smithi005 ceph-mon[39562]: 2.5 scrub ok 2024-04-20T17:17:16.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:16 smithi005 ceph-mon[39562]: osdmap e296: 12 total, 12 up, 12 in 2024-04-20T17:17:16.481 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:17:16.481 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:17:16.504 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.504 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.504 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.504 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.504 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:16 smithi171 ceph-mon[44189]: 2.5 scrub ok 2024-04-20T17:17:16.504 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.504 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.505 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.505 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.505 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.505 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.505 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:16 smithi171 ceph-mon[44189]: osdmap e296: 12 total, 12 up, 12 in 2024-04-20T17:17:16.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:16 smithi106 ceph-mon[44423]: 2.5 scrub ok 2024-04-20T17:17:16.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:16.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:16 smithi106 ceph-mon[44423]: osdmap e296: 12 total, 12 up, 12 in 2024-04-20T17:17:16.672 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:17:16.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:16.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:17:16.732 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:17:16.776 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:17:16.807 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:16.827 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:17:16.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:16.830 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:17:16.931 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:17:16.958 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:16.977 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:17:16.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:16.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:17:17.081 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:17:17.108 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:17.126 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:17:17.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:17.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:17:17.230 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:17:17.259 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:17.331 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:17:17.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:17.334 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:17:17.356 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-20T17:17:17.380 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:17:17.435 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:17:17.465 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:17.484 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:17:17.486 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:17 smithi171 ceph-mon[44189]: pgmap v821: 17 pgs: 1 active+recovering, 16 active+clean; 3.8 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 182 op/s; 6651/123316 objects misplaced (5.393%); 440 KiB/s, 6 objects/s recovering 2024-04-20T17:17:17.486 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:17 smithi171 ceph-mon[44189]: 2.8 scrub starts 2024-04-20T17:17:17.486 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:17 smithi171 ceph-mon[44189]: 2.8 scrub ok 2024-04-20T17:17:17.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:17.487 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:17:17.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:17 smithi106 ceph-mon[44423]: pgmap v821: 17 pgs: 1 active+recovering, 16 active+clean; 3.8 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 182 op/s; 6651/123316 objects misplaced (5.393%); 440 KiB/s, 6 objects/s recovering 2024-04-20T17:17:17.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:17 smithi106 ceph-mon[44423]: 2.8 scrub starts 2024-04-20T17:17:17.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:17 smithi106 ceph-mon[44423]: 2.8 scrub ok 2024-04-20T17:17:17.587 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:17:17.615 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:17.634 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:17:17.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:17.638 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:17:17.674 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:17:17.698 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:17 smithi005 ceph-mon[39562]: pgmap v821: 17 pgs: 1 active+recovering, 16 active+clean; 3.8 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 182 op/s; 6651/123316 objects misplaced (5.393%); 440 KiB/s, 6 objects/s recovering 2024-04-20T17:17:17.698 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:17 smithi005 ceph-mon[39562]: 2.8 scrub starts 2024-04-20T17:17:17.698 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:17 smithi005 ceph-mon[39562]: 2.8 scrub ok 2024-04-20T17:17:17.738 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:17:17.767 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:17.969 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:17:17.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:17.971 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:17:18.072 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:17:18.104 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:18.220 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:17:18.638 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:17:18.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:18.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:17:18.740 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:17:18.746 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:17:18.746 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:17:18.767 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:18.785 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:17:18.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:18.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:17:18.889 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:17:18.917 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:19.116 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:17:19.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:19.120 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:17:19.221 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:17:19.249 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:19.430 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:19 smithi171 ceph-mon[44189]: pgmap v822: 17 pgs: 1 active+recovering, 16 active+clean; 3.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 216 op/s; 6534/122400 objects misplaced (5.338%); 912 KiB/s, 14 objects/s recovering 2024-04-20T17:17:19.430 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:19 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:17:19.430 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:19 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:17:19.430 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:19 smithi171 ceph-mon[44189]: osdmap e297: 12 total, 12 up, 12 in 2024-04-20T17:17:19.430 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:19 smithi171 ceph-mon[44189]: from='client.25487 v1:172.21.15.5:0/1586121552' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:17:19.430 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:19 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:17:19.430 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:19 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:17:19.449 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:17:19.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:19.452 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:17:19.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:19 smithi005 ceph-mon[39562]: pgmap v822: 17 pgs: 1 active+recovering, 16 active+clean; 3.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 216 op/s; 6534/122400 objects misplaced (5.338%); 912 KiB/s, 14 objects/s recovering 2024-04-20T17:17:19.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:19 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:17:19.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:19 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:17:19.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:19 smithi005 ceph-mon[39562]: osdmap e297: 12 total, 12 up, 12 in 2024-04-20T17:17:19.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:19 smithi005 ceph-mon[39562]: from='client.25487 v1:172.21.15.5:0/1586121552' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:17:19.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:19 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:17:19.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:19 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:17:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:19 smithi106 ceph-mon[44423]: pgmap v822: 17 pgs: 1 active+recovering, 16 active+clean; 3.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 216 op/s; 6534/122400 objects misplaced (5.338%); 912 KiB/s, 14 objects/s recovering 2024-04-20T17:17:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:19 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:17:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:19 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:17:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:19 smithi106 ceph-mon[44423]: osdmap e297: 12 total, 12 up, 12 in 2024-04-20T17:17:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:19 smithi106 ceph-mon[44423]: from='client.25487 v1:172.21.15.5:0/1586121552' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:17:19.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:19 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:17:19.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:19 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:17:19.526 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":823,"stamp":"2024-04-20T17:17:18.237092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4011196984,"num_objects":61200,"num_object_clones":0,"num_object_copies":122400,"num_objects_missing_on_primary":3260,"num_objects_missing":3260,"num_objects_degraded":0,"num_objects_misplaced":6534,"num_objects_unfound":0,"num_objects_dirty":61200,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":260967,"num_write_kb":10306363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25950,"num_bytes_recovered":1702250624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161116,"ondisk_log_size":161116,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":9618856,"kb_used_data":8484784,"kb_used_omap":219,"kb_used_meta":1133732,"kb_avail":1115322968,"statfs":{"total":1151940427776,"available":1142090719232,"internally_reserved":0,"allocated":8688418816,"data_stored":8676931252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224949,"internal_metadata":1160941899},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,2,0,0,0,0,0,3,5,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-89456640,"num_objects":-1365,"num_object_clones":0,"num_object_copies":-2730,"num_objects_missing_on_primary":-117,"num_objects_missing":-117,"num_objects_degraded":0,"num_objects_misplaced":-231,"num_objects_unfound":0,"num_objects_dirty":-1365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1365,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":7471104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.180472"},"pg_stats":[{"pgid":"2.f","version":"295'16326","reported_seq":17205,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-04-20T17:17:13.214777+0000","last_change":"2024-04-20T17:16:24.907036+0000","last_active":"2024-04-20T17:17:13.214777+0000","last_peered":"2024-04-20T17:17:13.214777+0000","last_clean":"2024-04-20T17:17:13.214777+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:17:13.214777+0000","last_undegraded":"2024-04-20T17:17:13.214777+0000","last_fullsized":"2024-04-20T17:17:13.214777+0000","mapping_epoch":115,"log_start":"136'6300","ondisk_log_start":"136'6300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'15773","last_scrub_stamp":"2024-04-20T17:16:24.906931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:24.906931+0000","objects_scrubbed":4413,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:47.687304+0000","scrub_duration":421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252706816,"num_objects":3856,"num_object_clones":0,"num_object_copies":7712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16326,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"295'16201","reported_seq":28147,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-04-20T17:17:15.615689+0000","last_change":"2024-04-20T17:16:40.290499+0000","last_active":"2024-04-20T17:17:15.615689+0000","last_peered":"2024-04-20T17:17:15.615689+0000","last_clean":"2024-04-20T17:17:15.615689+0000","last_became_active":"2024-04-20T17:10:28.865670+0000","last_became_peered":"2024-04-20T17:10:28.865670+0000","last_unstale":"2024-04-20T17:17:15.615689+0000","last_undegraded":"2024-04-20T17:17:15.615689+0000","last_fullsized":"2024-04-20T17:17:15.615689+0000","mapping_epoch":162,"log_start":"136'6200","ondisk_log_start":"136'6200","created":115,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'15835","last_scrub_stamp":"2024-04-20T17:16:40.290402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:40.290402+0000","objects_scrubbed":4222,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:55.604081+0000","scrub_duration":371,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252641280,"num_objects":3855,"num_object_clones":0,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16201,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5619,"num_bytes_recovered":368246784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"295'16591","reported_seq":17358,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-04-20T17:17:15.528505+0000","last_change":"2024-04-20T17:17:12.348489+0000","last_active":"2024-04-20T17:17:15.528505+0000","last_peered":"2024-04-20T17:17:15.528505+0000","last_clean":"2024-04-20T17:17:15.528505+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:17:15.528505+0000","last_undegraded":"2024-04-20T17:17:15.528505+0000","last_fullsized":"2024-04-20T17:17:15.528505+0000","mapping_epoch":115,"log_start":"136'6500","ondisk_log_start":"136'6500","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'16554","last_scrub_stamp":"2024-04-20T17:17:12.348391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:12.348391+0000","objects_scrubbed":3930,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:33.813428+0000","scrub_duration":337,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255000576,"num_objects":3891,"num_object_clones":0,"num_object_copies":7782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16591,"num_write_kb":655424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"295'22928","reported_seq":44449,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-04-20T17:17:15.567170+0000","last_change":"2024-04-20T17:16:35.136358+0000","last_active":"2024-04-20T17:17:15.567170+0000","last_peered":"2024-04-20T17:17:15.567170+0000","last_clean":"2024-04-20T17:17:15.567170+0000","last_became_active":"2024-04-20T17:12:08.085936+0000","last_became_peered":"2024-04-20T17:12:08.085936+0000","last_unstale":"2024-04-20T17:17:15.567170+0000","last_undegraded":"2024-04-20T17:17:15.567170+0000","last_fullsized":"2024-04-20T17:17:15.567170+0000","mapping_epoch":196,"log_start":"167'12868","ondisk_log_start":"167'12868","created":115,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'22494","last_scrub_stamp":"2024-04-20T17:16:35.136177+0000","last_deep_scrub":"213'20373","last_deep_scrub_stamp":"2024-04-20T17:12:48.443808+0000","last_clean_scrub_stamp":"2024-04-20T17:16:35.136177+0000","objects_scrubbed":4393,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:40.491841+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259391488,"num_objects":3958,"num_object_clones":0,"num_object_copies":7916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16276,"num_write_kb":647488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":450166784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"296'16027","reported_seq":34894,"reported_epoch":296,"state":"active+recovering","last_fresh":"2024-04-20T17:17:17.515573+0000","last_change":"2024-04-20T17:12:26.166109+0000","last_active":"2024-04-20T17:17:17.515573+0000","last_peered":"2024-04-20T17:17:17.515573+0000","last_clean":"2024-04-20T17:09:35.882603+0000","last_became_active":"2024-04-20T17:10:54.470253+0000","last_became_peered":"2024-04-20T17:10:54.470253+0000","last_unstale":"2024-04-20T17:17:17.515573+0000","last_undegraded":"2024-04-20T17:17:17.515573+0000","last_fullsized":"2024-04-20T17:17:17.515573+0000","mapping_epoch":171,"log_start":"135'6000","ondisk_log_start":"135'6000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242679808,"num_objects":3703,"num_object_clones":0,"num_object_copies":7406,"num_objects_missing_on_primary":3260,"num_objects_missing":3260,"num_objects_degraded":0,"num_objects_misplaced":6534,"num_objects_unfound":0,"num_objects_dirty":3703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16027,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6158,"num_bytes_recovered":403570688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"0,2,5","objects":14},{"shards":"0,10","objects":429},{"shards":"2,5","objects":3260}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":787,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-04-20T17:17:16.007226+0000","last_change":"2024-04-20T17:16:40.132382+0000","last_active":"2024-04-20T17:17:16.007226+0000","last_peered":"2024-04-20T17:17:16.007226+0000","last_clean":"2024-04-20T17:17:16.007226+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:17:16.007226+0000","last_undegraded":"2024-04-20T17:17:16.007226+0000","last_fullsized":"2024-04-20T17:17:16.007226+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:16:40.132254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:16:40.132254+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:54.442752+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"296'16391","reported_seq":17191,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-04-20T17:17:17.827360+0000","last_change":"2024-04-20T17:17:10.481774+0000","last_active":"2024-04-20T17:17:17.827360+0000","last_peered":"2024-04-20T17:17:17.827360+0000","last_clean":"2024-04-20T17:17:17.827360+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:17:17.827360+0000","last_undegraded":"2024-04-20T17:17:17.827360+0000","last_fullsized":"2024-04-20T17:17:17.827360+0000","mapping_epoch":115,"log_start":"136'6300","ondisk_log_start":"136'6300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'16300","last_scrub_stamp":"2024-04-20T17:17:10.481666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:10.481666+0000","objects_scrubbed":3956,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:18.079313+0000","scrub_duration":337,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253034496,"num_objects":3861,"num_object_clones":0,"num_object_copies":7722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16391,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"295'16460","reported_seq":17211,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-04-20T17:17:13.706108+0000","last_change":"2024-04-20T17:15:35.632022+0000","last_active":"2024-04-20T17:17:13.706108+0000","last_peered":"2024-04-20T17:17:13.706108+0000","last_clean":"2024-04-20T17:17:13.706108+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:17:13.706108+0000","last_undegraded":"2024-04-20T17:17:13.706108+0000","last_fullsized":"2024-04-20T17:17:13.706108+0000","mapping_epoch":115,"log_start":"136'6400","ondisk_log_start":"136'6400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'15459","last_scrub_stamp":"2024-04-20T17:15:35.631842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:15:35.631842+0000","objects_scrubbed":4867,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":426,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253231104,"num_objects":3864,"num_object_clones":0,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16460,"num_write_kb":650368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"296'20433","reported_seq":21333,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-04-20T17:17:16.865859+0000","last_change":"2024-04-20T17:16:25.742067+0000","last_active":"2024-04-20T17:17:16.865859+0000","last_peered":"2024-04-20T17:17:16.865859+0000","last_clean":"2024-04-20T17:17:16.865859+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:17:16.865859+0000","last_undegraded":"2024-04-20T17:17:16.865859+0000","last_fullsized":"2024-04-20T17:17:16.865859+0000","mapping_epoch":115,"log_start":"154'10402","ondisk_log_start":"154'10402","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'19891","last_scrub_stamp":"2024-04-20T17:16:25.741880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:25.741880+0000","objects_scrubbed":4262,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:39.901935+0000","scrub_duration":378,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243793920,"num_objects":3720,"num_object_clones":0,"num_object_copies":7440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16322,"num_write_kb":641344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"296'16475","reported_seq":17331,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-04-20T17:17:17.232557+0000","last_change":"2024-04-20T17:17:08.025358+0000","last_active":"2024-04-20T17:17:17.232557+0000","last_peered":"2024-04-20T17:17:17.232557+0000","last_clean":"2024-04-20T17:17:17.232557+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:17:17.232557+0000","last_undegraded":"2024-04-20T17:17:17.232557+0000","last_fullsized":"2024-04-20T17:17:17.232557+0000","mapping_epoch":115,"log_start":"136'6400","ondisk_log_start":"136'6400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'16358","last_scrub_stamp":"2024-04-20T17:17:08.025214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:08.025214+0000","objects_scrubbed":3866,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:25.817318+0000","scrub_duration":359,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245432320,"num_objects":3745,"num_object_clones":0,"num_object_copies":7490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16475,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"296'16273","reported_seq":17054,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-04-20T17:17:17.765735+0000","last_change":"2024-04-20T17:17:16.522161+0000","last_active":"2024-04-20T17:17:17.765735+0000","last_peered":"2024-04-20T17:17:17.765735+0000","last_clean":"2024-04-20T17:17:17.765735+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:17:17.765735+0000","last_undegraded":"2024-04-20T17:17:17.765735+0000","last_fullsized":"2024-04-20T17:17:17.765735+0000","mapping_epoch":115,"log_start":"136'6200","ondisk_log_start":"136'6200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'16253","last_scrub_stamp":"2024-04-20T17:17:16.522032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:16.522032+0000","objects_scrubbed":3799,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:29.552786+0000","scrub_duration":315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247660544,"num_objects":3779,"num_object_clones":0,"num_object_copies":7558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16273,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"296'16400","reported_seq":31742,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-04-20T17:17:17.853038+0000","last_change":"2024-04-20T17:16:34.421711+0000","last_active":"2024-04-20T17:17:17.853038+0000","last_peered":"2024-04-20T17:17:17.853038+0000","last_clean":"2024-04-20T17:17:17.853038+0000","last_became_active":"2024-04-20T17:10:55.931751+0000","last_became_peered":"2024-04-20T17:10:55.931751+0000","last_unstale":"2024-04-20T17:17:17.853038+0000","last_undegraded":"2024-04-20T17:17:17.853038+0000","last_fullsized":"2024-04-20T17:17:17.853038+0000","mapping_epoch":173,"log_start":"136'6300","ondisk_log_start":"136'6300","created":115,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'15941","last_scrub_stamp":"2024-04-20T17:16:34.421587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:34.421587+0000","objects_scrubbed":4322,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:43.906503+0000","scrub_duration":366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253100032,"num_objects":3862,"num_object_clones":0,"num_object_copies":7724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16400,"num_write_kb":648384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7296,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"296'16485","reported_seq":17285,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-04-20T17:17:17.143081+0000","last_change":"2024-04-20T17:16:42.499318+0000","last_active":"2024-04-20T17:17:17.143081+0000","last_peered":"2024-04-20T17:17:17.143081+0000","last_clean":"2024-04-20T17:17:17.143081+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:17:17.143081+0000","last_undegraded":"2024-04-20T17:17:17.143081+0000","last_fullsized":"2024-04-20T17:17:17.143081+0000","mapping_epoch":115,"log_start":"136'6400","ondisk_log_start":"136'6400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'16092","last_scrub_stamp":"2024-04-20T17:16:42.499193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:42.499193+0000","objects_scrubbed":4224,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:04.645993+0000","scrub_duration":355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251068416,"num_objects":3831,"num_object_clones":0,"num_object_copies":7662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16485,"num_write_kb":650112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"295'15969","reported_seq":16762,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-04-20T17:17:13.214631+0000","last_change":"2024-04-20T17:17:06.994658+0000","last_active":"2024-04-20T17:17:13.214631+0000","last_peered":"2024-04-20T17:17:13.214631+0000","last_clean":"2024-04-20T17:17:13.214631+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:17:13.214631+0000","last_undegraded":"2024-04-20T17:17:13.214631+0000","last_fullsized":"2024-04-20T17:17:13.214631+0000","mapping_epoch":115,"log_start":"135'5900","ondisk_log_start":"135'5900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'15884","last_scrub_stamp":"2024-04-20T17:17:06.994455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:06.994455+0000","objects_scrubbed":3982,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:21.957254+0000","scrub_duration":397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255262720,"num_objects":3895,"num_object_clones":0,"num_object_copies":7790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15969,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"296'16348","reported_seq":17143,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-04-20T17:17:17.535724+0000","last_change":"2024-04-20T17:17:07.882718+0000","last_active":"2024-04-20T17:17:17.535724+0000","last_peered":"2024-04-20T17:17:17.535724+0000","last_clean":"2024-04-20T17:17:17.535724+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:17:17.535724+0000","last_undegraded":"2024-04-20T17:17:17.535724+0000","last_fullsized":"2024-04-20T17:17:17.535724+0000","mapping_epoch":115,"log_start":"136'6300","ondisk_log_start":"136'6300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'16236","last_scrub_stamp":"2024-04-20T17:17:07.882596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:07.882596+0000","objects_scrubbed":3868,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:32.661323+0000","scrub_duration":370,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246022144,"num_objects":3754,"num_object_clones":0,"num_object_copies":7508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16348,"num_write_kb":643264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"295'16087","reported_seq":16881,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-04-20T17:17:15.656109+0000","last_change":"2024-04-20T17:17:04.896764+0000","last_active":"2024-04-20T17:17:15.656109+0000","last_peered":"2024-04-20T17:17:15.656109+0000","last_clean":"2024-04-20T17:17:15.656109+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:17:15.656109+0000","last_undegraded":"2024-04-20T17:17:15.656109+0000","last_fullsized":"2024-04-20T17:17:15.656109+0000","mapping_epoch":115,"log_start":"135'6000","ondisk_log_start":"135'6000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"293'15941","last_scrub_stamp":"2024-04-20T17:17:04.896640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:04.896640+0000","objects_scrubbed":3929,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:16.518914+0000","scrub_duration":355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247857176,"num_objects":3783,"num_object_clones":0,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16087,"num_write_kb":635777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"295'16155","reported_seq":16822,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-04-20T17:17:15.607379+0000","last_change":"2024-04-20T17:16:23.593408+0000","last_active":"2024-04-20T17:17:15.607379+0000","last_peered":"2024-04-20T17:17:15.607379+0000","last_clean":"2024-04-20T17:17:15.607379+0000","last_became_active":"2024-04-20T17:12:23.515375+0000","last_became_peered":"2024-04-20T17:12:23.515375+0000","last_unstale":"2024-04-20T17:17:15.607379+0000","last_undegraded":"2024-04-20T17:17:15.607379+0000","last_fullsized":"2024-04-20T17:17:15.607379+0000","mapping_epoch":202,"log_start":"135'6100","ondisk_log_start":"135'6100","created":115,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'15586","last_scrub_stamp":"2024-04-20T17:16:23.593300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:23.593300+0000","objects_scrubbed":4410,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:33.363583+0000","scrub_duration":446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251723776,"num_objects":3841,"num_object_clones":0,"num_object_copies":7682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16155,"num_write_kb":639872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":4010606616,"num_objects":61198,"num_object_clones":0,"num_object_copies":122396,"num_objects_missing_on_primary":3260,"num_objects_missing":3260,"num_objects_degraded":0,"num_objects_misplaced":6534,"num_objects_unfound":0,"num_objects_dirty":61198,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":260786,"num_write_kb":10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25942,"num_bytes_recovered":1700134912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8653971456,"data_stored":8653963312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},"log_size":160979,"ondisk_log_size":160979,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991371,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":805528,"kb_used_data":734096,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92939624,"statfs":{"total":95995035648,"available":95170174976,"internally_reserved":0,"allocated":751714304,"data_stored":750751720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":73136580},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253008,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":913160,"kb_used_data":773692,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92831992,"statfs":{"total":95995035648,"available":95059959808,"internally_reserved":0,"allocated":792260608,"data_stored":791307554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":142801344},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612758,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1125640,"kb_used_data":986176,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92619512,"statfs":{"total":95995035648,"available":94842380288,"internally_reserved":0,"allocated":1009844224,"data_stored":1008887118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":142801344},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762808987,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1097928,"kb_used_data":986236,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92647224,"statfs":{"total":95995035648,"available":94870757376,"internally_reserved":0,"allocated":1009905664,"data_stored":1008952630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168737,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":802968,"kb_used_data":739856,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92942184,"statfs":{"total":95995035648,"available":95172796416,"internally_reserved":0,"allocated":757612544,"data_stored":756649980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":607316,"kb_used_data":495624,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93137836,"statfs":{"total":95995035648,"available":95373144064,"internally_reserved":0,"allocated":507518976,"data_stored":506559964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575039,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":592340,"kb_used_data":488968,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93152812,"statfs":{"total":95995035648,"available":95388479488,"internally_reserved":0,"allocated":500703232,"data_stored":499744220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738564,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":903636,"kb_used_data":760008,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92841516,"statfs":{"total":95995035648,"available":95069712384,"internally_reserved":0,"allocated":778248192,"data_stored":777289180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,3,5,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":726468,"kb_used_data":678076,"kb_used_omap":202,"kb_used_meta":48181,"kb_avail":93018684,"statfs":{"total":95995035648,"available":95251132416,"internally_reserved":0,"allocated":694349824,"data_stored":693396790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":49337815},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182580,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":565460,"kb_used_data":494024,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93179692,"statfs":{"total":95995035648,"available":95416004608,"internally_reserved":0,"allocated":505880576,"data_stored":504921564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986351,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":271816,"kb_used_data":244800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93473336,"statfs":{"total":95995035648,"available":95716696064,"internally_reserved":0,"allocated":250675200,"data_stored":249718094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790121,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1206596,"kb_used_data":1103228,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92538556,"statfs":{"total":95995035648,"available":94759481344,"internally_reserved":0,"allocated":1129705472,"data_stored":1128752438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":776142848,"data_stored":776142848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":498008064,"data_stored":498008064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":691929088,"data_stored":691929088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":503840768,"data_stored":503840768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":248254464,"data_stored":248250392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1126039552,"data_stored":1126039552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":503513088,"data_stored":503513088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":753795072,"data_stored":753795072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1007681536,"data_stored":1007681536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1006637056,"data_stored":1006632984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":789118976,"data_stored":789118976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":749010944,"data_stored":749010944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:17:19.527 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:17:19.553 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:17:19.581 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:19.619 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:17:19.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:19.623 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:17:19.724 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:17:19.751 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:19.765 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:17:19.765 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:17:19.769 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:17:19.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:19.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:17:19.873 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:17:19.903 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:20.002 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:17:20.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:20.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:17:20.049 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:17:20.091 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:17:20.111 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:17:20.140 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:20 smithi106 ceph-mon[44423]: osdmap e298: 12 total, 12 up, 12 in 2024-04-20T17:17:20.566 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:20 smithi171 ceph-mon[44189]: osdmap e298: 12 total, 12 up, 12 in 2024-04-20T17:17:20.589 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:17:20.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:20.592 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:17:20.595 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-20T17:17:20.677 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:20 smithi005 ceph-mon[39562]: osdmap e298: 12 total, 12 up, 12 in 2024-04-20T17:17:20.693 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:17:20.720 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:20.739 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:17:20.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:20.741 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:17:20.842 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:17:20.870 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:20.890 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:17:20.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:20.894 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:17:20.907 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:17:20.994 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:17:21.021 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:21.040 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:17:21.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:21.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:17:21.147 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:17:21.175 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:21.228 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:17:21.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:21.230 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:17:21.332 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:17:21.357 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:21 smithi106 ceph-mon[44423]: pgmap v825: 17 pgs: 1 active+recovering, 16 active+clean; 3.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 159 op/s; 6534/121986 objects misplaced (5.356%); 629 KiB/s, 9 objects/s recovering 2024-04-20T17:17:21.359 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:21.378 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:17:21.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:21.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:17:21.461 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:21 smithi171 ceph-mon[44189]: pgmap v825: 17 pgs: 1 active+recovering, 16 active+clean; 3.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 159 op/s; 6534/121986 objects misplaced (5.356%); 629 KiB/s, 9 objects/s recovering 2024-04-20T17:17:21.482 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:17:21.514 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:21 smithi005 ceph-mon[39562]: pgmap v825: 17 pgs: 1 active+recovering, 16 active+clean; 3.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 159 op/s; 6534/121986 objects misplaced (5.356%); 629 KiB/s, 9 objects/s recovering 2024-04-20T17:17:21.515 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:21.646 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:17:21.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:21.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:17:21.750 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:17:21.784 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:21.843 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:17:21.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:21.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:17:21.962 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:17:21.992 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:22.257 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:17:22.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:22 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4123830727' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:17:22.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:22 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4123830727' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:17:22.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:22 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4123830727' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:17:22.941 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:17:22.942 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:17:23.242 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:17:23.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:23.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:17:23.347 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:17:23.377 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:23.508 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:23 smithi005 ceph-mon[39562]: pgmap v826: 17 pgs: 1 active+recovering, 16 active+clean; 3.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 158 op/s; 6534/121380 objects misplaced (5.383%); 619 KiB/s, 9 objects/s recovering 2024-04-20T17:17:23.508 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:23 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:17:23.508 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:23 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4123830727' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:17:23.508 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:23 smithi005 ceph-mon[39562]: osdmap e299: 12 total, 12 up, 12 in 2024-04-20T17:17:23.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:23 smithi106 ceph-mon[44423]: pgmap v826: 17 pgs: 1 active+recovering, 16 active+clean; 3.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 158 op/s; 6534/121380 objects misplaced (5.383%); 619 KiB/s, 9 objects/s recovering 2024-04-20T17:17:23.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:23 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:17:23.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:23 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4123830727' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:17:23.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:23 smithi106 ceph-mon[44423]: osdmap e299: 12 total, 12 up, 12 in 2024-04-20T17:17:23.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:23 smithi171 ceph-mon[44189]: pgmap v826: 17 pgs: 1 active+recovering, 16 active+clean; 3.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 158 op/s; 6534/121380 objects misplaced (5.383%); 619 KiB/s, 9 objects/s recovering 2024-04-20T17:17:23.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:23 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:17:23.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:23 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4123830727' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:17:23.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:23 smithi171 ceph-mon[44189]: osdmap e299: 12 total, 12 up, 12 in 2024-04-20T17:17:23.669 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-20T17:17:23.726 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:17:23.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:23.729 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:17:23.830 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:17:23.860 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:17:23.860 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:23.890 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:17:23.895 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:17:23.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:23.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:17:23.921 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:17:23.953 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:17:23.973 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:17:23.994 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:17:24.001 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:17:24.022 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:17:24.030 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:24.046 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:17:24.068 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:17:24.089 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:17:24.110 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:17:24.131 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:17:24.132 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:17:24.132 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:17:24.132 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:17:24.133 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:17:24.133 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:17:24.133 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:17:24.133 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:17:24.133 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:17:24.133 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:17:24.133 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:17:24.133 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:17:24.133 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:17:24.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:24 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3888319718' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:17:24.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:24 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:24.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:24 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:24.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:24 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:24.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:24 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:24.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:24 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:24.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:24 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3888319718' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:17:24.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:24 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:24.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:24 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:24.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:24 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:24.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:24 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:24.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:24 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:24 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3888319718' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:17:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:24 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:24 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:24 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:24 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:24 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:24.984 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:17:24.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:24.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:17:25.090 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:17:25.118 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:25.137 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:17:25.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:25.140 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:17:25.241 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:17:25.271 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:25 smithi106 ceph-mon[44423]: pgmap v828: 17 pgs: 1 active+recovering, 16 active+clean; 3.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 178 op/s; 6417/120252 objects misplaced (5.336%); 619 KiB/s, 9 objects/s recovering 2024-04-20T17:17:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:25 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:17:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:25 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:17:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:25 smithi171 ceph-mon[44189]: pgmap v828: 17 pgs: 1 active+recovering, 16 active+clean; 3.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 178 op/s; 6417/120252 objects misplaced (5.336%); 619 KiB/s, 9 objects/s recovering 2024-04-20T17:17:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:25 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:17:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:25 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:17:25.621 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:17:25.622 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:25.622 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:25.622 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:25.622 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:25.622 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:25 smithi005 ceph-mon[39562]: pgmap v828: 17 pgs: 1 active+recovering, 16 active+clean; 3.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 178 op/s; 6417/120252 objects misplaced (5.336%); 619 KiB/s, 9 objects/s recovering 2024-04-20T17:17:25.623 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:25.623 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:25.623 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:25.623 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:25 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:17:25.623 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:25 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:17:25.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:25.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:17:25.725 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:17:25.752 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:25.771 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:17:25.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:25.774 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:17:25.875 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:17:25.901 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:25.920 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:17:25.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:25.926 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:17:26.030 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:17:26.061 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:26.084 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:17:26.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:26.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:17:26.191 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:17:26.219 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:26.242 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:17:26.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:26.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:17:26.346 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:17:26.378 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:26.402 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:17:26.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:26.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:17:26.507 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:17:26.536 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:26 smithi005 ceph-mon[39562]: osdmap e300: 12 total, 12 up, 12 in 2024-04-20T17:17:26.539 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:26.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:26 smithi171 ceph-mon[44189]: osdmap e300: 12 total, 12 up, 12 in 2024-04-20T17:17:26.627 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:17:26.628 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:17:26.718 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:17:26.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:26 smithi106 ceph-mon[44423]: osdmap e300: 12 total, 12 up, 12 in 2024-04-20T17:17:27.095 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:17:27.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:27.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:17:27.208 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:17:27.236 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:27.258 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:17:27.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:27.262 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:17:27.363 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:17:27.389 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:27.408 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:17:27.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:27.411 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:17:27.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:27 smithi005 ceph-mon[39562]: pgmap v830: 17 pgs: 1 active+recovering, 16 active+clean; 3.7 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 159 op/s; 6417/119806 objects misplaced (5.356%); 545 KiB/s, 8 objects/s recovering 2024-04-20T17:17:27.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:27 smithi005 ceph-mon[39562]: from='client.25474 v1:172.21.15.5:0/31715030' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:17:27.512 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:17:27.542 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:27.568 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:17:27.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:27.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:17:27.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:27 smithi171 ceph-mon[44189]: pgmap v830: 17 pgs: 1 active+recovering, 16 active+clean; 3.7 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 159 op/s; 6417/119806 objects misplaced (5.356%); 545 KiB/s, 8 objects/s recovering 2024-04-20T17:17:27.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:27 smithi171 ceph-mon[44189]: from='client.25474 v1:172.21.15.5:0/31715030' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:17:27.589 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":830,"stamp":"2024-04-20T17:17:26.059051+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3926196792,"num_objects":59903,"num_object_clones":0,"num_object_copies":119806,"num_objects_missing_on_primary":3201,"num_objects_missing":3201,"num_objects_degraded":0,"num_objects_misplaced":6417,"num_objects_unfound":0,"num_objects_dirty":59903,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":262264,"num_write_kb":10306363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26008,"num_bytes_recovered":1706051712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161213,"ondisk_log_size":161213,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":9475224,"kb_used_data":8324444,"kb_used_omap":219,"kb_used_meta":1150372,"kb_avail":1115466600,"statfs":{"total":1151940427776,"available":1142237798400,"internally_reserved":0,"allocated":8524230656,"data_stored":8512499294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224949,"internal_metadata":1177981259},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-71434240,"num_objects":-1090,"num_object_clones":0,"num_object_copies":-2180,"num_objects_missing_on_primary":-59,"num_objects_missing":-59,"num_objects_degraded":0,"num_objects_misplaced":-117,"num_objects_unfound":0,"num_objects_dirty":-1090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1090,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":3801088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.813638"},"pg_stats":[{"pgid":"2.f","version":"299'16448","reported_seq":17335,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-04-20T17:17:23.433249+0000","last_change":"2024-04-20T17:16:24.907036+0000","last_active":"2024-04-20T17:17:23.433249+0000","last_peered":"2024-04-20T17:17:23.433249+0000","last_clean":"2024-04-20T17:17:23.433249+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:17:23.433249+0000","last_undegraded":"2024-04-20T17:17:23.433249+0000","last_fullsized":"2024-04-20T17:17:23.433249+0000","mapping_epoch":115,"log_start":"136'6400","ondisk_log_start":"136'6400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'15773","last_scrub_stamp":"2024-04-20T17:16:24.906931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:24.906931+0000","objects_scrubbed":4413,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:47.687304+0000","scrub_duration":421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244711424,"num_objects":3734,"num_object_clones":0,"num_object_copies":7468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16448,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"300'16277","reported_seq":28233,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-20T17:17:25.262373+0000","last_change":"2024-04-20T17:16:40.290499+0000","last_active":"2024-04-20T17:17:25.262373+0000","last_peered":"2024-04-20T17:17:25.262373+0000","last_clean":"2024-04-20T17:17:25.262373+0000","last_became_active":"2024-04-20T17:10:28.865670+0000","last_became_peered":"2024-04-20T17:10:28.865670+0000","last_unstale":"2024-04-20T17:17:25.262373+0000","last_undegraded":"2024-04-20T17:17:25.262373+0000","last_fullsized":"2024-04-20T17:17:25.262373+0000","mapping_epoch":162,"log_start":"136'6200","ondisk_log_start":"136'6200","created":115,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'15835","last_scrub_stamp":"2024-04-20T17:16:40.290402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:40.290402+0000","objects_scrubbed":4222,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:55.604081+0000","scrub_duration":371,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247660544,"num_objects":3779,"num_object_clones":0,"num_object_copies":7558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16277,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5619,"num_bytes_recovered":368246784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"300'16694","reported_seq":17471,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-20T17:17:25.262351+0000","last_change":"2024-04-20T17:17:12.348489+0000","last_active":"2024-04-20T17:17:25.262351+0000","last_peered":"2024-04-20T17:17:25.262351+0000","last_clean":"2024-04-20T17:17:25.262351+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:17:25.262351+0000","last_undegraded":"2024-04-20T17:17:25.262351+0000","last_fullsized":"2024-04-20T17:17:25.262351+0000","mapping_epoch":115,"log_start":"136'6600","ondisk_log_start":"136'6600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'16554","last_scrub_stamp":"2024-04-20T17:17:12.348391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:12.348391+0000","objects_scrubbed":3930,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:33.813428+0000","scrub_duration":337,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248250368,"num_objects":3788,"num_object_clones":0,"num_object_copies":7576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16694,"num_write_kb":655424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"300'23031","reported_seq":44562,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-20T17:17:25.706476+0000","last_change":"2024-04-20T17:16:35.136358+0000","last_active":"2024-04-20T17:17:25.706476+0000","last_peered":"2024-04-20T17:17:25.706476+0000","last_clean":"2024-04-20T17:17:25.706476+0000","last_became_active":"2024-04-20T17:12:08.085936+0000","last_became_peered":"2024-04-20T17:12:08.085936+0000","last_unstale":"2024-04-20T17:17:25.706476+0000","last_undegraded":"2024-04-20T17:17:25.706476+0000","last_fullsized":"2024-04-20T17:17:25.706476+0000","mapping_epoch":196,"log_start":"167'12968","ondisk_log_start":"167'12968","created":115,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'22494","last_scrub_stamp":"2024-04-20T17:16:35.136177+0000","last_deep_scrub":"213'20373","last_deep_scrub_stamp":"2024-04-20T17:12:48.443808+0000","last_clean_scrub_stamp":"2024-04-20T17:16:35.136177+0000","objects_scrubbed":4393,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:40.491841+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252641280,"num_objects":3855,"num_object_clones":0,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16379,"num_write_kb":647488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":450166784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"299'16085","reported_seq":35132,"reported_epoch":299,"state":"active+recovering","last_fresh":"2024-04-20T17:17:22.540518+0000","last_change":"2024-04-20T17:12:26.166109+0000","last_active":"2024-04-20T17:17:22.540518+0000","last_peered":"2024-04-20T17:17:22.540518+0000","last_clean":"2024-04-20T17:09:35.882603+0000","last_became_active":"2024-04-20T17:10:54.470253+0000","last_became_peered":"2024-04-20T17:10:54.470253+0000","last_unstale":"2024-04-20T17:17:22.540518+0000","last_undegraded":"2024-04-20T17:17:22.540518+0000","last_fullsized":"2024-04-20T17:17:22.540518+0000","mapping_epoch":171,"log_start":"135'6000","ondisk_log_start":"135'6000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238878720,"num_objects":3645,"num_object_clones":0,"num_object_copies":7290,"num_objects_missing_on_primary":3201,"num_objects_missing":3201,"num_objects_degraded":0,"num_objects_misplaced":6417,"num_objects_unfound":0,"num_objects_dirty":3645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16085,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6216,"num_bytes_recovered":407371776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"0,2,5","objects":15},{"shards":"0,10","objects":429},{"shards":"2,5","objects":3201}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":793,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-04-20T17:17:22.367784+0000","last_change":"2024-04-20T17:16:40.132382+0000","last_active":"2024-04-20T17:17:22.367784+0000","last_peered":"2024-04-20T17:17:22.367784+0000","last_clean":"2024-04-20T17:17:22.367784+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:17:22.367784+0000","last_undegraded":"2024-04-20T17:17:22.367784+0000","last_fullsized":"2024-04-20T17:17:22.367784+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:16:40.132254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:16:40.132254+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:54.442752+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"299'16458","reported_seq":17264,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-04-20T17:17:22.812790+0000","last_change":"2024-04-20T17:17:10.481774+0000","last_active":"2024-04-20T17:17:22.812790+0000","last_peered":"2024-04-20T17:17:22.812790+0000","last_clean":"2024-04-20T17:17:22.812790+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:17:22.812790+0000","last_undegraded":"2024-04-20T17:17:22.812790+0000","last_fullsized":"2024-04-20T17:17:22.812790+0000","mapping_epoch":115,"log_start":"136'6400","ondisk_log_start":"136'6400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'16300","last_scrub_stamp":"2024-04-20T17:17:10.481666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:10.481666+0000","objects_scrubbed":3956,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:18.079313+0000","scrub_duration":337,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248643584,"num_objects":3794,"num_object_clones":0,"num_object_copies":7588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16458,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"299'16597","reported_seq":17402,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-04-20T17:17:23.432159+0000","last_change":"2024-04-20T17:17:15.112506+0000","last_active":"2024-04-20T17:17:23.432159+0000","last_peered":"2024-04-20T17:17:23.432159+0000","last_clean":"2024-04-20T17:17:23.432159+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:17:23.432159+0000","last_undegraded":"2024-04-20T17:17:23.432159+0000","last_fullsized":"2024-04-20T17:17:23.432159+0000","mapping_epoch":115,"log_start":"136'6500","ondisk_log_start":"136'6500","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'16484","last_scrub_stamp":"2024-04-20T17:17:15.112253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:15.112253+0000","objects_scrubbed":3848,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:45.026807+0000","scrub_duration":457,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244252672,"num_objects":3727,"num_object_clones":0,"num_object_copies":7454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16597,"num_write_kb":650368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"298'20487","reported_seq":21391,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-04-20T17:17:21.837758+0000","last_change":"2024-04-20T17:16:25.742067+0000","last_active":"2024-04-20T17:17:21.837758+0000","last_peered":"2024-04-20T17:17:21.837758+0000","last_clean":"2024-04-20T17:17:21.837758+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:17:21.837758+0000","last_undegraded":"2024-04-20T17:17:21.837758+0000","last_fullsized":"2024-04-20T17:17:21.837758+0000","mapping_epoch":115,"log_start":"154'10402","ondisk_log_start":"154'10402","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'19891","last_scrub_stamp":"2024-04-20T17:16:25.741880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:25.741880+0000","objects_scrubbed":4262,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:39.901935+0000","scrub_duration":378,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240254976,"num_objects":3666,"num_object_clones":0,"num_object_copies":7332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16376,"num_write_kb":641344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"299'16522","reported_seq":17384,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-04-20T17:17:22.393445+0000","last_change":"2024-04-20T17:17:08.025358+0000","last_active":"2024-04-20T17:17:22.393445+0000","last_peered":"2024-04-20T17:17:22.393445+0000","last_clean":"2024-04-20T17:17:22.393445+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:17:22.393445+0000","last_undegraded":"2024-04-20T17:17:22.393445+0000","last_fullsized":"2024-04-20T17:17:22.393445+0000","mapping_epoch":115,"log_start":"136'6500","ondisk_log_start":"136'6500","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'16358","last_scrub_stamp":"2024-04-20T17:17:08.025214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:08.025214+0000","objects_scrubbed":3866,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:25.817318+0000","scrub_duration":359,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242352128,"num_objects":3698,"num_object_clones":0,"num_object_copies":7396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16522,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"299'16330","reported_seq":17117,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-04-20T17:17:22.774818+0000","last_change":"2024-04-20T17:17:16.522161+0000","last_active":"2024-04-20T17:17:22.774818+0000","last_peered":"2024-04-20T17:17:22.774818+0000","last_clean":"2024-04-20T17:17:22.774818+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:17:22.774818+0000","last_undegraded":"2024-04-20T17:17:22.774818+0000","last_fullsized":"2024-04-20T17:17:22.774818+0000","mapping_epoch":115,"log_start":"136'6300","ondisk_log_start":"136'6300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'16253","last_scrub_stamp":"2024-04-20T17:17:16.522032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:16.522032+0000","objects_scrubbed":3799,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:29.552786+0000","scrub_duration":315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243924992,"num_objects":3722,"num_object_clones":0,"num_object_copies":7444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16330,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"299'16465","reported_seq":31813,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-04-20T17:17:22.767762+0000","last_change":"2024-04-20T17:16:34.421711+0000","last_active":"2024-04-20T17:17:22.767762+0000","last_peered":"2024-04-20T17:17:22.767762+0000","last_clean":"2024-04-20T17:17:22.767762+0000","last_became_active":"2024-04-20T17:10:55.931751+0000","last_became_peered":"2024-04-20T17:10:55.931751+0000","last_unstale":"2024-04-20T17:17:22.767762+0000","last_undegraded":"2024-04-20T17:17:22.767762+0000","last_fullsized":"2024-04-20T17:17:22.767762+0000","mapping_epoch":173,"log_start":"136'6400","ondisk_log_start":"136'6400","created":115,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'15941","last_scrub_stamp":"2024-04-20T17:16:34.421587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:34.421587+0000","objects_scrubbed":4322,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:43.906503+0000","scrub_duration":366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248840192,"num_objects":3797,"num_object_clones":0,"num_object_copies":7594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16465,"num_write_kb":648384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7296,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"299'16545","reported_seq":17351,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-04-20T17:17:22.467479+0000","last_change":"2024-04-20T17:16:42.499318+0000","last_active":"2024-04-20T17:17:22.467479+0000","last_peered":"2024-04-20T17:17:22.467479+0000","last_clean":"2024-04-20T17:17:22.467479+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:17:22.467479+0000","last_undegraded":"2024-04-20T17:17:22.467479+0000","last_fullsized":"2024-04-20T17:17:22.467479+0000","mapping_epoch":115,"log_start":"136'6500","ondisk_log_start":"136'6500","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'16092","last_scrub_stamp":"2024-04-20T17:16:42.499193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:42.499193+0000","objects_scrubbed":4224,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:04.645993+0000","scrub_duration":355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247136256,"num_objects":3771,"num_object_clones":0,"num_object_copies":7542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16545,"num_write_kb":650112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"299'16078","reported_seq":16878,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-04-20T17:17:23.750326+0000","last_change":"2024-04-20T17:17:06.994658+0000","last_active":"2024-04-20T17:17:23.750326+0000","last_peered":"2024-04-20T17:17:23.750326+0000","last_clean":"2024-04-20T17:17:23.750326+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:17:23.750326+0000","last_undegraded":"2024-04-20T17:17:23.750326+0000","last_fullsized":"2024-04-20T17:17:23.750326+0000","mapping_epoch":115,"log_start":"135'6000","ondisk_log_start":"135'6000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'15884","last_scrub_stamp":"2024-04-20T17:17:06.994455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:06.994455+0000","objects_scrubbed":3982,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:21.957254+0000","scrub_duration":397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248119296,"num_objects":3786,"num_object_clones":0,"num_object_copies":7572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16078,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"299'16397","reported_seq":17198,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-04-20T17:17:22.467459+0000","last_change":"2024-04-20T17:17:07.882718+0000","last_active":"2024-04-20T17:17:22.467459+0000","last_peered":"2024-04-20T17:17:22.467459+0000","last_clean":"2024-04-20T17:17:22.467459+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:17:22.467459+0000","last_undegraded":"2024-04-20T17:17:22.467459+0000","last_fullsized":"2024-04-20T17:17:22.467459+0000","mapping_epoch":115,"log_start":"136'6300","ondisk_log_start":"136'6300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'16236","last_scrub_stamp":"2024-04-20T17:17:07.882596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:07.882596+0000","objects_scrubbed":3868,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:32.661323+0000","scrub_duration":370,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242810880,"num_objects":3705,"num_object_clones":0,"num_object_copies":7410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16397,"num_write_kb":643264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"300'16182","reported_seq":16986,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-20T17:17:25.475397+0000","last_change":"2024-04-20T17:17:04.896764+0000","last_active":"2024-04-20T17:17:25.475397+0000","last_peered":"2024-04-20T17:17:25.475397+0000","last_clean":"2024-04-20T17:17:25.475397+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:17:25.475397+0000","last_undegraded":"2024-04-20T17:17:25.475397+0000","last_fullsized":"2024-04-20T17:17:25.475397+0000","mapping_epoch":115,"log_start":"135'6100","ondisk_log_start":"135'6100","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"293'15941","last_scrub_stamp":"2024-04-20T17:17:04.896640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:04.896640+0000","objects_scrubbed":3929,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:16.518914+0000","scrub_duration":355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241631256,"num_objects":3688,"num_object_clones":0,"num_object_copies":7376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3688,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16182,"num_write_kb":635777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"300'16250","reported_seq":16927,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-20T17:17:25.803971+0000","last_change":"2024-04-20T17:16:23.593408+0000","last_active":"2024-04-20T17:17:25.803971+0000","last_peered":"2024-04-20T17:17:25.803971+0000","last_clean":"2024-04-20T17:17:25.803971+0000","last_became_active":"2024-04-20T17:12:23.515375+0000","last_became_peered":"2024-04-20T17:12:23.515375+0000","last_unstale":"2024-04-20T17:17:25.803971+0000","last_undegraded":"2024-04-20T17:17:25.803971+0000","last_fullsized":"2024-04-20T17:17:25.803971+0000","mapping_epoch":202,"log_start":"136'6200","ondisk_log_start":"136'6200","created":115,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'15586","last_scrub_stamp":"2024-04-20T17:16:23.593300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:23.593300+0000","objects_scrubbed":4410,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:33.363583+0000","scrub_duration":446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245497856,"num_objects":3746,"num_object_clones":0,"num_object_copies":7492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16250,"num_write_kb":639872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":3925606424,"num_objects":59901,"num_object_clones":0,"num_object_copies":119802,"num_objects_missing_on_primary":3201,"num_objects_missing":3201,"num_objects_degraded":0,"num_objects_misplaced":6417,"num_objects_unfound":0,"num_objects_dirty":59901,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":262083,"num_write_kb":10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26000,"num_bytes_recovered":1703936000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8487837696,"data_stored":8487829552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},"log_size":161076,"ondisk_log_size":161076,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991372,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":802160,"kb_used_data":726568,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92942992,"statfs":{"total":95995035648,"available":95173623808,"internally_reserved":0,"allocated":744005632,"data_stored":743031192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":77396420},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253010,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":890424,"kb_used_data":750956,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92854728,"statfs":{"total":95995035648,"available":95083241472,"internally_reserved":0,"allocated":768978944,"data_stored":768002078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":142801344},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612760,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1099720,"kb_used_data":960252,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92645432,"statfs":{"total":95995035648,"available":94868922368,"internally_reserved":0,"allocated":983298048,"data_stored":982311202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":142801344},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762808989,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1073016,"kb_used_data":957164,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":92672136,"statfs":{"total":95995035648,"available":94896267264,"internally_reserved":0,"allocated":980135936,"data_stored":979159090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":118618565},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168738,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":791792,"kb_used_data":728616,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92953360,"statfs":{"total":95995035648,"available":95184240640,"internally_reserved":0,"allocated":746102784,"data_stored":745128364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":599032,"kb_used_data":487340,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93146120,"statfs":{"total":95995035648,"available":95381626880,"internally_reserved":0,"allocated":499036160,"data_stored":498059314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575040,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":590124,"kb_used_data":482592,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93155028,"statfs":{"total":95995035648,"available":95390748672,"internally_reserved":0,"allocated":494174208,"data_stored":493203340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738565,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":893496,"kb_used_data":749868,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92851656,"statfs":{"total":95995035648,"available":95080095744,"internally_reserved":0,"allocated":767864832,"data_stored":766887986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":720436,"kb_used_data":672044,"kb_used_omap":202,"kb_used_meta":48181,"kb_avail":93024716,"statfs":{"total":95995035648,"available":95257309184,"internally_reserved":0,"allocated":688173056,"data_stored":687196210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":49337815},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182582,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":555844,"kb_used_data":480248,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93189308,"statfs":{"total":95995035648,"available":95425851392,"internally_reserved":0,"allocated":491773952,"data_stored":490791178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986353,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":265400,"kb_used_data":238384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93479752,"statfs":{"total":95995035648,"available":95723266048,"internally_reserved":0,"allocated":244105216,"data_stored":243124298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790123,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1193780,"kb_used_data":1090412,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92551372,"statfs":{"total":95995035648,"available":94772604928,"internally_reserved":0,"allocated":1116581888,"data_stored":1115605042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":765722624,"data_stored":765722624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":491061248,"data_stored":491061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":685703168,"data_stored":685703168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":489750528,"data_stored":489750528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":241700864,"data_stored":241696792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1114308608,"data_stored":1114308608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":496893952,"data_stored":496893952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":741408768,"data_stored":741408768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":976224256,"data_stored":976224256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":980815872,"data_stored":980811800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":765591552,"data_stored":765591552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":738656256,"data_stored":738656256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:17:27.590 INFO:tasks.cephadm.ceph_manager.ceph:PG 2.3 is not active+clean 2024-04-20T17:17:27.591 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '2.3', 'version': "299'16085", 'reported_seq': 35132, 'reported_epoch': 299, 'state': 'active+recovering', 'last_fresh': '2024-04-20T17:17:22.540518+0000', 'last_change': '2024-04-20T17:12:26.166109+0000', 'last_active': '2024-04-20T17:17:22.540518+0000', 'last_peered': '2024-04-20T17:17:22.540518+0000', 'last_clean': '2024-04-20T17:09:35.882603+0000', 'last_became_active': '2024-04-20T17:10:54.470253+0000', 'last_became_peered': '2024-04-20T17:10:54.470253+0000', 'last_unstale': '2024-04-20T17:17:22.540518+0000', 'last_undegraded': '2024-04-20T17:17:22.540518+0000', 'last_fullsized': '2024-04-20T17:17:22.540518+0000', 'mapping_epoch': 171, 'log_start': "135'6000", 'ondisk_log_start': "135'6000", 'created': 115, 'last_epoch_clean': 116, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-20T17:08:04.990639+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T17:08:04.990639+0000', 'last_clean_scrub_stamp': '2024-04-20T17:08:04.990639+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 238878720, 'num_objects': 3645, 'num_object_clones': 0, 'num_object_copies': 7290, 'num_objects_missing_on_primary': 3201, 'num_objects_missing': 3201, 'num_objects_degraded': 0, 'num_objects_misplaced': 6417, 'num_objects_unfound': 0, 'num_objects_dirty': 3645, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16085, 'num_write_kb': 631360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6216, 'num_bytes_recovered': 407371776, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 10], 'acting': [0, 10], 'avail_no_missing': ['2', '5'], 'object_location_counts': [{'shards': '0,2,5', 'objects': 15}, {'shards': '0,10', 'objects': 429}, {'shards': '2,5', 'objects': 3201}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-20T17:17:27.591 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph status --format=json 2024-04-20T17:17:27.658 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:17:27.677 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:17:27.701 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:27 smithi106 ceph-mon[44423]: pgmap v830: 17 pgs: 1 active+recovering, 16 active+clean; 3.7 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 159 op/s; 6417/119806 objects misplaced (5.356%); 545 KiB/s, 8 objects/s recovering 2024-04-20T17:17:27.701 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:27 smithi106 ceph-mon[44423]: from='client.25474 v1:172.21.15.5:0/31715030' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:17:27.703 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:27.724 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:17:27.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:27.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:17:27.828 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:17:27.861 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:28.069 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:17:28.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:28.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:17:28.173 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:17:28.200 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:28.218 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:17:28.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:28.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:17:28.321 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:17:28.356 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:28.378 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:17:28.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:28.383 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:17:28.484 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:17:28.514 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:28.715 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:17:28.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:28.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:17:28.748 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:17:28.754 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:17:28.757 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:17:28.819 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:17:28.845 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:28.863 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:17:28.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:28.866 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:17:28.967 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:17:28.994 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:29.012 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:17:29.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:29.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:17:29.116 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:17:29.147 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:29.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:29 smithi005 ceph-mon[39562]: pgmap v831: 17 pgs: 1 active+recovering, 16 active+clean; 3.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 184 op/s; 6297/119030 objects misplaced (5.290%); 960 KiB/s, 14 objects/s recovering 2024-04-20T17:17:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:29 smithi171 ceph-mon[44189]: pgmap v831: 17 pgs: 1 active+recovering, 16 active+clean; 3.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 184 op/s; 6297/119030 objects misplaced (5.290%); 960 KiB/s, 14 objects/s recovering 2024-04-20T17:17:29.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:29 smithi106 ceph-mon[44423]: pgmap v831: 17 pgs: 1 active+recovering, 16 active+clean; 3.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 184 op/s; 6297/119030 objects misplaced (5.290%); 960 KiB/s, 14 objects/s recovering 2024-04-20T17:17:30.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:30 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:17:30.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:30 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:17:30.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:30 smithi005 ceph-mon[39562]: osdmap e301: 12 total, 12 up, 12 in 2024-04-20T17:17:30.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:30 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:17:30.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:30 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:17:30.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:30 smithi171 ceph-mon[44189]: osdmap e301: 12 total, 12 up, 12 in 2024-04-20T17:17:30.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:30 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:17:30.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:30 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:17:30.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:30 smithi106 ceph-mon[44423]: osdmap e301: 12 total, 12 up, 12 in 2024-04-20T17:17:31.030 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:17:31.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:31.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:17:31.064 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:17:31.136 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:17:31.162 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:31.180 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:17:31.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:31.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:17:31.283 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:17:31.310 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:31 smithi005 ceph-mon[39562]: pgmap v833: 17 pgs: 1 active+recovering, 16 active+clean; 3.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 173 op/s; 6297/118670 objects misplaced (5.306%); 984 KiB/s, 15 objects/s recovering 2024-04-20T17:17:31.310 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:31 smithi005 ceph-mon[39562]: osdmap e302: 12 total, 12 up, 12 in 2024-04-20T17:17:31.312 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:31.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:31 smithi106 ceph-mon[44423]: pgmap v833: 17 pgs: 1 active+recovering, 16 active+clean; 3.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 173 op/s; 6297/118670 objects misplaced (5.306%); 984 KiB/s, 15 objects/s recovering 2024-04-20T17:17:31.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:31 smithi106 ceph-mon[44423]: osdmap e302: 12 total, 12 up, 12 in 2024-04-20T17:17:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:31 smithi171 ceph-mon[44189]: pgmap v833: 17 pgs: 1 active+recovering, 16 active+clean; 3.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 173 op/s; 6297/118670 objects misplaced (5.306%); 984 KiB/s, 15 objects/s recovering 2024-04-20T17:17:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:31 smithi171 ceph-mon[44189]: osdmap e302: 12 total, 12 up, 12 in 2024-04-20T17:17:32.168 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:17:32.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:32.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:17:32.277 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:17:32.303 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:32.321 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:17:32.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:32.324 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:17:32.425 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:17:32.452 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:32.471 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:17:32.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:32.474 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:17:32.575 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:17:32.601 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:32.620 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:17:32.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:32.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:17:32.726 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:17:32.752 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:32.773 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:17:32.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:32.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:17:32.879 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:17:32.911 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:33.329 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:17:33.329 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:17:33.329 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-04-20T17:17:33.329 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:17:33.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:33 smithi171 ceph-mon[44189]: pgmap v835: 17 pgs: 1 active+recovering, 16 active+clean; 3.6 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 166 op/s; 6297/117992 objects misplaced (5.337%); 584 KiB/s, 9 objects/s recovering 2024-04-20T17:17:33.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:33 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:17:33.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:33 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:17:33.588 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:33 smithi005 ceph-mon[39562]: pgmap v835: 17 pgs: 1 active+recovering, 16 active+clean; 3.6 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 166 op/s; 6297/117992 objects misplaced (5.337%); 584 KiB/s, 9 objects/s recovering 2024-04-20T17:17:33.588 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:33 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:17:33.588 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:33 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:17:33.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:33 smithi106 ceph-mon[44423]: pgmap v835: 17 pgs: 1 active+recovering, 16 active+clean; 3.6 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 166 op/s; 6297/117992 objects misplaced (5.337%); 584 KiB/s, 9 objects/s recovering 2024-04-20T17:17:33.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:33 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:17:33.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:33 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:17:33.963 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:17:33.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:33.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:17:34.068 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:17:34.103 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:34.286 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:34 smithi005 ceph-mon[39562]: osdmap e303: 12 total, 12 up, 12 in 2024-04-20T17:17:34.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:34 smithi171 ceph-mon[44189]: osdmap e303: 12 total, 12 up, 12 in 2024-04-20T17:17:34.647 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-20T17:17:34.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:34 smithi106 ceph-mon[44423]: osdmap e303: 12 total, 12 up, 12 in 2024-04-20T17:17:34.855 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:17:34.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:34.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:17:34.885 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:17:34.905 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:17:34.926 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:17:34.949 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:17:34.961 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:17:34.969 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:17:34.987 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:34.989 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:17:35.006 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:17:35.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:35.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:17:35.012 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:17:35.034 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:17:35.054 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:17:35.074 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:17:35.094 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:17:35.109 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:17:35.117 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:17:35.119 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:17:35.119 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:17:35.119 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:17:35.119 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:17:35.119 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:17:35.119 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:17:35.119 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:17:35.119 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:17:35.119 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:17:35.119 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:17:35.119 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:17:35.119 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:17:35.134 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:35.152 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:17:35.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:35.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:17:35.250 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:17:35.250 INFO:teuthology.orchestra.run.smithi005.stdout:{"fsid":"13c6e76c-ff37-11ee-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1022,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":303,"num_osds":12,"num_up_osds":12,"osd_up_since":1713633141,"num_in_osds":12,"osd_in_since":1713632742,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovering","count":1}],"num_pgs":17,"num_pools":2,"num_objects":58427,"data_bytes":3829465656,"bytes_used":9519935488,"bytes_avail":1142420492288,"bytes_total":1151940427776,"misplaced_objects":6177,"misplaced_total":116854,"misplaced_ratio":0.052860834887979873,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":644333,"recovering_keys_per_sec":0,"num_objects_recovered":59,"num_bytes_recovered":3866624,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":181},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.5:8443/"}},"servicemap":{"epoch":25,"modified":"2024-04-20T17:16:50.052718+0000","services":{}},"progress_events":{}} 2024-04-20T17:17:35.256 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:17:35.282 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:35.300 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:17:35.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:35.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:17:35.404 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:17:35.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:35 smithi005 ceph-mon[39562]: pgmap v837: 17 pgs: 1 active+recovering, 16 active+clean; 3.6 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 181 op/s; 6177/116854 objects misplaced (5.286%); 629 KiB/s, 9 objects/s recovering 2024-04-20T17:17:35.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:35 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:17:35.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:35 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:17:35.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:35 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2507799984' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:17:35.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:35 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3796290132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:17:35.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:35.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:35.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:35.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:35.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:35.434 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:35 smithi171 ceph-mon[44189]: pgmap v837: 17 pgs: 1 active+recovering, 16 active+clean; 3.6 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 181 op/s; 6177/116854 objects misplaced (5.286%); 629 KiB/s, 9 objects/s recovering 2024-04-20T17:17:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:35 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:17:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:35 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:17:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:35 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2507799984' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:17:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:35 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3796290132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:17:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:35.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:35.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:35.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:35 smithi106 ceph-mon[44423]: pgmap v837: 17 pgs: 1 active+recovering, 16 active+clean; 3.6 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 181 op/s; 6177/116854 objects misplaced (5.286%); 629 KiB/s, 9 objects/s recovering 2024-04-20T17:17:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:35 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:17:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:35 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:17:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:35 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2507799984' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:17:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:35 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3796290132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:17:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:35.982 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-20T17:17:35.982 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:17:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:36 smithi171 ceph-mon[44189]: osdmap e304: 12 total, 12 up, 12 in 2024-04-20T17:17:36.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:36.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:36.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:36.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:36.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:36.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:36.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:36.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:36 smithi005 ceph-mon[39562]: osdmap e304: 12 total, 12 up, 12 in 2024-04-20T17:17:36.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:36.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:36.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:36.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:36.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:36.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:36.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:36.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:36 smithi106 ceph-mon[44423]: osdmap e304: 12 total, 12 up, 12 in 2024-04-20T17:17:37.287 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:17:37.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:37.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:17:37.392 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:17:37.422 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:37.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:37 smithi171 ceph-mon[44189]: pgmap v839: 17 pgs: 1 active+recovering, 16 active+clean; 3.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 199 op/s; 6177/116276 objects misplaced (5.312%); 629 KiB/s, 9 objects/s recovering 2024-04-20T17:17:37.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:37 smithi005 ceph-mon[39562]: pgmap v839: 17 pgs: 1 active+recovering, 16 active+clean; 3.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 199 op/s; 6177/116276 objects misplaced (5.312%); 629 KiB/s, 9 objects/s recovering 2024-04-20T17:17:37.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:37 smithi106 ceph-mon[44423]: pgmap v839: 17 pgs: 1 active+recovering, 16 active+clean; 3.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 199 op/s; 6177/116276 objects misplaced (5.312%); 629 KiB/s, 9 objects/s recovering 2024-04-20T17:17:37.870 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:17:37.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:37.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:17:37.973 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:17:37.978 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:17:38.004 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:38.023 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:17:38.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:38.026 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:17:38.127 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:17:38.153 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:38.173 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:17:38.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:38.176 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:17:38.277 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:17:38.304 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:38.324 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:17:38.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:38.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:17:38.427 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:17:38.454 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:38.473 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:17:38.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:38.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:17:38.577 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:17:38.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:38 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:17:38.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:38 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:17:38.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:38 smithi171 ceph-mon[44189]: osdmap e305: 12 total, 12 up, 12 in 2024-04-20T17:17:38.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:38 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:17:38.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:38 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:17:38.608 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:38 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:17:38.608 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:38 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:17:38.608 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:38 smithi005 ceph-mon[39562]: osdmap e305: 12 total, 12 up, 12 in 2024-04-20T17:17:38.609 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:38 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:17:38.609 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:38 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:17:38.612 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:38.686 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:17:38.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:38.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:17:38.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:38 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:17:38.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:38 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:17:38.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:38 smithi106 ceph-mon[44423]: osdmap e305: 12 total, 12 up, 12 in 2024-04-20T17:17:38.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:38 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:17:38.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:38 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:17:38.792 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:17:38.795 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:17:38.821 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:39.062 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:17:39.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:39.064 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:17:39.064 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:17:39.164 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:17:39.191 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:39.212 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:17:39.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:39.214 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:17:39.315 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:17:39.331 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:17:39.349 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:39 smithi005 ceph-mon[39562]: pgmap v841: 17 pgs: 1 active+recovering, 16 active+clean; 3.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 217 op/s; 6051/115380 objects misplaced (5.244%); 1.3 MiB/s, 20 objects/s recovering 2024-04-20T17:17:39.349 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:39 smithi005 ceph-mon[39562]: osdmap e306: 12 total, 12 up, 12 in 2024-04-20T17:17:39.349 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:39 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/165472883' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:17:39.349 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:39 smithi005 ceph-mon[39562]: from='client.25492 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:17:39.352 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:39 smithi171 ceph-mon[44189]: pgmap v841: 17 pgs: 1 active+recovering, 16 active+clean; 3.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 217 op/s; 6051/115380 objects misplaced (5.244%); 1.3 MiB/s, 20 objects/s recovering 2024-04-20T17:17:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:39 smithi171 ceph-mon[44189]: osdmap e306: 12 total, 12 up, 12 in 2024-04-20T17:17:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:39 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/165472883' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:17:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:39 smithi171 ceph-mon[44189]: from='client.25492 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:17:39.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:39 smithi106 ceph-mon[44423]: pgmap v841: 17 pgs: 1 active+recovering, 16 active+clean; 3.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 217 op/s; 6051/115380 objects misplaced (5.244%); 1.3 MiB/s, 20 objects/s recovering 2024-04-20T17:17:39.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:39 smithi106 ceph-mon[44423]: osdmap e306: 12 total, 12 up, 12 in 2024-04-20T17:17:39.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:39 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/165472883' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:17:39.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:39 smithi106 ceph-mon[44423]: from='client.25492 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:17:40.210 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:17:40.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:40.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:17:40.315 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:17:40.340 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:40.358 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:17:40.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:40.361 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:17:40.462 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:17:40.490 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:40 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:17:40.490 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:40 smithi005 ceph-mon[39562]: from='client.25492 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:17:40.490 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:40 smithi005 ceph-mon[39562]: osdmap e307: 12 total, 12 up, 12 in 2024-04-20T17:17:40.492 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:40.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:40 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:17:40.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:40 smithi171 ceph-mon[44189]: from='client.25492 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:17:40.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:40 smithi171 ceph-mon[44189]: osdmap e307: 12 total, 12 up, 12 in 2024-04-20T17:17:40.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:40 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:17:40.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:40 smithi106 ceph-mon[44423]: from='client.25492 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:17:40.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:40 smithi106 ceph-mon[44423]: osdmap e307: 12 total, 12 up, 12 in 2024-04-20T17:17:40.844 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:17:40.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:40.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:17:40.948 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:17:40.977 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:40.995 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:17:40.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:40.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:17:41.100 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:17:41.126 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:41.144 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:17:41.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:41.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:17:41.248 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:17:41.280 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:41.442 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:41 smithi005 ceph-mon[39562]: pgmap v844: 17 pgs: 1 active+recovering, 16 active+clean; 3.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 191 op/s; 6051/115028 objects misplaced (5.260%); 846 KiB/s, 13 objects/s recovering 2024-04-20T17:17:41.442 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:41 smithi005 ceph-mon[39562]: osdmap e308: 12 total, 12 up, 12 in 2024-04-20T17:17:41.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:41 smithi171 ceph-mon[44189]: pgmap v844: 17 pgs: 1 active+recovering, 16 active+clean; 3.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 191 op/s; 6051/115028 objects misplaced (5.260%); 846 KiB/s, 13 objects/s recovering 2024-04-20T17:17:41.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:41 smithi171 ceph-mon[44189]: osdmap e308: 12 total, 12 up, 12 in 2024-04-20T17:17:41.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:41 smithi106 ceph-mon[44423]: pgmap v844: 17 pgs: 1 active+recovering, 16 active+clean; 3.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 191 op/s; 6051/115028 objects misplaced (5.260%); 846 KiB/s, 13 objects/s recovering 2024-04-20T17:17:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:41 smithi106 ceph-mon[44423]: osdmap e308: 12 total, 12 up, 12 in 2024-04-20T17:17:41.830 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:17:41.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:41.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:17:41.845 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:17:41.933 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:17:41.960 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:41.979 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:17:41.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:41.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:17:42.082 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:17:42.108 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:42.127 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:17:42.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:42.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:17:42.231 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:17:42.266 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:42.421 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:17:42.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:42.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:17:42.525 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:17:42.556 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:42.688 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:17:42.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:42.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:17:42.793 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:17:42.819 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:42.837 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:17:42.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:42.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:17:42.942 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:17:42.968 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:42.987 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:17:42.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:42.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:17:43.090 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:17:43.118 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:43.135 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:17:43.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:43.138 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:17:43.239 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:17:43.267 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:43.291 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:17:43.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:43.295 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:17:43.322 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:43 smithi005 ceph-mon[39562]: pgmap v846: 17 pgs: 1 active+recovering, 16 active+clean; 3.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 182 op/s; 6051/114434 objects misplaced (5.288%); 847 KiB/s, 13 objects/s recovering 2024-04-20T17:17:43.395 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:17:43.420 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:43 smithi171 ceph-mon[44189]: pgmap v846: 17 pgs: 1 active+recovering, 16 active+clean; 3.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 182 op/s; 6051/114434 objects misplaced (5.288%); 847 KiB/s, 13 objects/s recovering 2024-04-20T17:17:43.422 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:43.441 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:17:43.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:43.444 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:17:43.469 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:17:43.544 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:17:43.574 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:43.620 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:17:43.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:43.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:17:43.727 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:17:43.753 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:43.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:43 smithi106 ceph-mon[44423]: pgmap v846: 17 pgs: 1 active+recovering, 16 active+clean; 3.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 182 op/s; 6051/114434 objects misplaced (5.288%); 847 KiB/s, 13 objects/s recovering 2024-04-20T17:17:43.772 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:17:43.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:43.774 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:17:43.792 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:17:43.792 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:17:43.875 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:17:43.902 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:43.921 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:17:43.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:43.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:17:44.023 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:17:44.049 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:44.067 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:17:44.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:44.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:17:44.171 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:17:44.196 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:44.215 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:17:44.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:44.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:17:44.319 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:17:44.351 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:44.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:44 smithi005 ceph-mon[39562]: from='client.25498 v1:172.21.15.5:0/329011959' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:17:44.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:44 smithi106 ceph-mon[44423]: from='client.25498 v1:172.21.15.5:0/329011959' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:17:44.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:44 smithi171 ceph-mon[44189]: from='client.25498 v1:172.21.15.5:0/329011959' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:17:44.604 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:17:44.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:44.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:17:44.627 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":846,"stamp":"2024-04-20T17:17:42.061960+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3750167096,"num_objects":57217,"num_object_clones":0,"num_object_copies":114434,"num_objects_missing_on_primary":3019,"num_objects_missing":3019,"num_objects_degraded":0,"num_objects_misplaced":6051,"num_objects_unfound":0,"num_objects_dirty":57217,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":264950,"num_write_kb":10306363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26192,"num_bytes_recovered":1718110336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160799,"ondisk_log_size":160799,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":9164948,"kb_used_data":8010048,"kb_used_omap":219,"kb_used_meta":1154532,"kb_avail":1115776876,"statfs":{"total":1151940427776,"available":1142555521024,"internally_reserved":0,"allocated":8202289152,"data_stored":8190064666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224949,"internal_metadata":1182241099},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,13,1,0,0,1,1,3,6,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-57016320,"num_objects":-870,"num_object_clones":0,"num_object_copies":-1740,"num_objects_missing_on_primary":-63,"num_objects_missing":-63,"num_objects_degraded":0,"num_objects_misplaced":-126,"num_objects_unfound":0,"num_objects_dirty":-870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":870,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":4128768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.758132"},"pg_stats":[{"pgid":"2.f","version":"306'16623","reported_seq":17524,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-04-20T17:17:38.639559+0000","last_change":"2024-04-20T17:16:24.907036+0000","last_active":"2024-04-20T17:17:38.639559+0000","last_peered":"2024-04-20T17:17:38.639559+0000","last_clean":"2024-04-20T17:17:38.639559+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:17:38.639559+0000","last_undegraded":"2024-04-20T17:17:38.639559+0000","last_fullsized":"2024-04-20T17:17:38.639559+0000","mapping_epoch":115,"log_start":"136'6600","ondisk_log_start":"136'6600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'15773","last_scrub_stamp":"2024-04-20T17:16:24.906931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:24.906931+0000","objects_scrubbed":4413,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:47.687304+0000","scrub_duration":421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233242624,"num_objects":3559,"num_object_clones":0,"num_object_copies":7118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16623,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"307'16440","reported_seq":28410,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-04-20T17:17:40.523072+0000","last_change":"2024-04-20T17:16:40.290499+0000","last_active":"2024-04-20T17:17:40.523072+0000","last_peered":"2024-04-20T17:17:40.523072+0000","last_clean":"2024-04-20T17:17:40.523072+0000","last_became_active":"2024-04-20T17:10:28.865670+0000","last_became_peered":"2024-04-20T17:10:28.865670+0000","last_unstale":"2024-04-20T17:17:40.523072+0000","last_undegraded":"2024-04-20T17:17:40.523072+0000","last_fullsized":"2024-04-20T17:17:40.523072+0000","mapping_epoch":162,"log_start":"136'6400","ondisk_log_start":"136'6400","created":115,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'15835","last_scrub_stamp":"2024-04-20T17:16:40.290402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:40.290402+0000","objects_scrubbed":4222,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:55.604081+0000","scrub_duration":371,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236978176,"num_objects":3616,"num_object_clones":0,"num_object_copies":7232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16440,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5619,"num_bytes_recovered":368246784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"307'16838","reported_seq":17629,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-04-20T17:17:40.508779+0000","last_change":"2024-04-20T17:17:12.348489+0000","last_active":"2024-04-20T17:17:40.508779+0000","last_peered":"2024-04-20T17:17:40.508779+0000","last_clean":"2024-04-20T17:17:40.508779+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:17:40.508779+0000","last_undegraded":"2024-04-20T17:17:40.508779+0000","last_fullsized":"2024-04-20T17:17:40.508779+0000","mapping_epoch":115,"log_start":"136'6800","ondisk_log_start":"136'6800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'16554","last_scrub_stamp":"2024-04-20T17:17:12.348391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:12.348391+0000","objects_scrubbed":3930,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:33.813428+0000","scrub_duration":337,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238813184,"num_objects":3644,"num_object_clones":0,"num_object_copies":7288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16838,"num_write_kb":655424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"307'23205","reported_seq":44750,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-04-20T17:17:40.268400+0000","last_change":"2024-04-20T17:16:35.136358+0000","last_active":"2024-04-20T17:17:40.268400+0000","last_peered":"2024-04-20T17:17:40.268400+0000","last_clean":"2024-04-20T17:17:40.268400+0000","last_became_active":"2024-04-20T17:12:08.085936+0000","last_became_peered":"2024-04-20T17:12:08.085936+0000","last_unstale":"2024-04-20T17:17:40.268400+0000","last_undegraded":"2024-04-20T17:17:40.268400+0000","last_fullsized":"2024-04-20T17:17:40.268400+0000","mapping_epoch":196,"log_start":"167'13168","ondisk_log_start":"167'13168","created":115,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'22494","last_scrub_stamp":"2024-04-20T17:16:35.136177+0000","last_deep_scrub":"213'20373","last_deep_scrub_stamp":"2024-04-20T17:12:48.443808+0000","last_clean_scrub_stamp":"2024-04-20T17:16:35.136177+0000","objects_scrubbed":4393,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241238016,"num_objects":3681,"num_object_clones":0,"num_object_copies":7362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16553,"num_write_kb":647488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":450166784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"305'16268","reported_seq":35877,"reported_epoch":305,"state":"active+recovering","last_fresh":"2024-04-20T17:17:37.861550+0000","last_change":"2024-04-20T17:12:26.166109+0000","last_active":"2024-04-20T17:17:37.861550+0000","last_peered":"2024-04-20T17:17:37.861550+0000","last_clean":"2024-04-20T17:09:35.882603+0000","last_became_active":"2024-04-20T17:10:54.470253+0000","last_became_peered":"2024-04-20T17:10:54.470253+0000","last_unstale":"2024-04-20T17:17:37.861550+0000","last_undegraded":"2024-04-20T17:17:37.861550+0000","last_fullsized":"2024-04-20T17:17:37.861550+0000","mapping_epoch":171,"log_start":"136'6200","ondisk_log_start":"136'6200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226885632,"num_objects":3462,"num_object_clones":0,"num_object_copies":6924,"num_objects_missing_on_primary":3019,"num_objects_missing":3019,"num_objects_degraded":0,"num_objects_misplaced":6051,"num_objects_unfound":0,"num_objects_dirty":3462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16268,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6400,"num_bytes_recovered":419430400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"0,2,5","objects":13},{"shards":"0,10","objects":430},{"shards":"2,5","objects":3019}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":805,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-04-20T17:17:37.363345+0000","last_change":"2024-04-20T17:16:40.132382+0000","last_active":"2024-04-20T17:17:37.363345+0000","last_peered":"2024-04-20T17:17:37.363345+0000","last_clean":"2024-04-20T17:17:37.363345+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:17:37.363345+0000","last_undegraded":"2024-04-20T17:17:37.363345+0000","last_fullsized":"2024-04-20T17:17:37.363345+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:16:40.132254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:16:40.132254+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:54.442752+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"305'16620","reported_seq":17438,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-04-20T17:17:37.654219+0000","last_change":"2024-04-20T17:17:10.481774+0000","last_active":"2024-04-20T17:17:37.654219+0000","last_peered":"2024-04-20T17:17:37.654219+0000","last_clean":"2024-04-20T17:17:37.654219+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:17:37.654219+0000","last_undegraded":"2024-04-20T17:17:37.654219+0000","last_fullsized":"2024-04-20T17:17:37.654219+0000","mapping_epoch":115,"log_start":"136'6600","ondisk_log_start":"136'6600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'16300","last_scrub_stamp":"2024-04-20T17:17:10.481666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:10.481666+0000","objects_scrubbed":3956,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:18.079313+0000","scrub_duration":337,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238026752,"num_objects":3632,"num_object_clones":0,"num_object_copies":7264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16620,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"306'16767","reported_seq":17586,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-04-20T17:17:38.563072+0000","last_change":"2024-04-20T17:17:15.112506+0000","last_active":"2024-04-20T17:17:38.563072+0000","last_peered":"2024-04-20T17:17:38.563072+0000","last_clean":"2024-04-20T17:17:38.563072+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:17:38.563072+0000","last_undegraded":"2024-04-20T17:17:38.563072+0000","last_fullsized":"2024-04-20T17:17:38.563072+0000","mapping_epoch":115,"log_start":"136'6700","ondisk_log_start":"136'6700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'16484","last_scrub_stamp":"2024-04-20T17:17:15.112253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:15.112253+0000","objects_scrubbed":3848,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:45.026807+0000","scrub_duration":457,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233111552,"num_objects":3557,"num_object_clones":0,"num_object_copies":7114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16767,"num_write_kb":650368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"308'20715","reported_seq":21639,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-04-20T17:17:41.944896+0000","last_change":"2024-04-20T17:16:25.742067+0000","last_active":"2024-04-20T17:17:41.944896+0000","last_peered":"2024-04-20T17:17:41.944896+0000","last_clean":"2024-04-20T17:17:41.944896+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:17:41.944896+0000","last_undegraded":"2024-04-20T17:17:41.944896+0000","last_fullsized":"2024-04-20T17:17:41.944896+0000","mapping_epoch":115,"log_start":"154'10702","ondisk_log_start":"154'10702","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'19891","last_scrub_stamp":"2024-04-20T17:16:25.741880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:25.741880+0000","objects_scrubbed":4262,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":378,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225312768,"num_objects":3438,"num_object_clones":0,"num_object_copies":6876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16604,"num_write_kb":641344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"305'16658","reported_seq":17532,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-04-20T17:17:37.460549+0000","last_change":"2024-04-20T17:17:08.025358+0000","last_active":"2024-04-20T17:17:37.460549+0000","last_peered":"2024-04-20T17:17:37.460549+0000","last_clean":"2024-04-20T17:17:37.460549+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:17:37.460549+0000","last_undegraded":"2024-04-20T17:17:37.460549+0000","last_fullsized":"2024-04-20T17:17:37.460549+0000","mapping_epoch":115,"log_start":"136'6600","ondisk_log_start":"136'6600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'16358","last_scrub_stamp":"2024-04-20T17:17:08.025214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:08.025214+0000","objects_scrubbed":3866,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:25.817318+0000","scrub_duration":359,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233439232,"num_objects":3562,"num_object_clones":0,"num_object_copies":7124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16658,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"305'16478","reported_seq":17277,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-04-20T17:17:37.708134+0000","last_change":"2024-04-20T17:17:16.522161+0000","last_active":"2024-04-20T17:17:37.708134+0000","last_peered":"2024-04-20T17:17:37.708134+0000","last_clean":"2024-04-20T17:17:37.708134+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:17:37.708134+0000","last_undegraded":"2024-04-20T17:17:37.708134+0000","last_fullsized":"2024-04-20T17:17:37.708134+0000","mapping_epoch":115,"log_start":"136'6400","ondisk_log_start":"136'6400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'16253","last_scrub_stamp":"2024-04-20T17:17:16.522032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:16.522032+0000","objects_scrubbed":3799,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:29.552786+0000","scrub_duration":315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234225664,"num_objects":3574,"num_object_clones":0,"num_object_copies":7148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16478,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"305'16627","reported_seq":31987,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-04-20T17:17:37.695112+0000","last_change":"2024-04-20T17:16:34.421711+0000","last_active":"2024-04-20T17:17:37.695112+0000","last_peered":"2024-04-20T17:17:37.695112+0000","last_clean":"2024-04-20T17:17:37.695112+0000","last_became_active":"2024-04-20T17:10:55.931751+0000","last_became_peered":"2024-04-20T17:10:55.931751+0000","last_unstale":"2024-04-20T17:17:37.695112+0000","last_undegraded":"2024-04-20T17:17:37.695112+0000","last_fullsized":"2024-04-20T17:17:37.695112+0000","mapping_epoch":173,"log_start":"136'6600","ondisk_log_start":"136'6600","created":115,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'15941","last_scrub_stamp":"2024-04-20T17:16:34.421587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:34.421587+0000","objects_scrubbed":4322,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:43.906503+0000","scrub_duration":366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238223360,"num_objects":3635,"num_object_clones":0,"num_object_copies":7270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16627,"num_write_kb":648384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7296,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"305'16728","reported_seq":17546,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-04-20T17:17:37.451535+0000","last_change":"2024-04-20T17:16:42.499318+0000","last_active":"2024-04-20T17:17:37.451535+0000","last_peered":"2024-04-20T17:17:37.451535+0000","last_clean":"2024-04-20T17:17:37.451535+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:17:37.451535+0000","last_undegraded":"2024-04-20T17:17:37.451535+0000","last_fullsized":"2024-04-20T17:17:37.451535+0000","mapping_epoch":115,"log_start":"136'6700","ondisk_log_start":"136'6700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'16092","last_scrub_stamp":"2024-04-20T17:16:42.499193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:42.499193+0000","objects_scrubbed":4224,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:04.645993+0000","scrub_duration":355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235143168,"num_objects":3588,"num_object_clones":0,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16728,"num_write_kb":650112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"306'16253","reported_seq":17069,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-04-20T17:17:38.606584+0000","last_change":"2024-04-20T17:17:06.994658+0000","last_active":"2024-04-20T17:17:38.606584+0000","last_peered":"2024-04-20T17:17:38.606584+0000","last_clean":"2024-04-20T17:17:38.606584+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:17:38.606584+0000","last_undegraded":"2024-04-20T17:17:38.606584+0000","last_fullsized":"2024-04-20T17:17:38.606584+0000","mapping_epoch":115,"log_start":"136'6200","ondisk_log_start":"136'6200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'15884","last_scrub_stamp":"2024-04-20T17:17:06.994455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:06.994455+0000","objects_scrubbed":3982,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:21.957254+0000","scrub_duration":397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236650496,"num_objects":3611,"num_object_clones":0,"num_object_copies":7222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16253,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"305'16560","reported_seq":17373,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-04-20T17:17:37.353391+0000","last_change":"2024-04-20T17:17:07.882718+0000","last_active":"2024-04-20T17:17:37.353391+0000","last_peered":"2024-04-20T17:17:37.353391+0000","last_clean":"2024-04-20T17:17:37.353391+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:17:37.353391+0000","last_undegraded":"2024-04-20T17:17:37.353391+0000","last_fullsized":"2024-04-20T17:17:37.353391+0000","mapping_epoch":115,"log_start":"136'6500","ondisk_log_start":"136'6500","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'16236","last_scrub_stamp":"2024-04-20T17:17:07.882596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:07.882596+0000","objects_scrubbed":3868,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:32.661323+0000","scrub_duration":370,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232128512,"num_objects":3542,"num_object_clones":0,"num_object_copies":7084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16560,"num_write_kb":643264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"307'16338","reported_seq":17156,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-04-20T17:17:40.547076+0000","last_change":"2024-04-20T17:17:04.896764+0000","last_active":"2024-04-20T17:17:40.547076+0000","last_peered":"2024-04-20T17:17:40.547076+0000","last_clean":"2024-04-20T17:17:40.547076+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:17:40.547076+0000","last_undegraded":"2024-04-20T17:17:40.547076+0000","last_fullsized":"2024-04-20T17:17:40.547076+0000","mapping_epoch":115,"log_start":"136'6300","ondisk_log_start":"136'6300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"293'15941","last_scrub_stamp":"2024-04-20T17:17:04.896640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:04.896640+0000","objects_scrubbed":3929,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:16.518914+0000","scrub_duration":355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231407640,"num_objects":3532,"num_object_clones":0,"num_object_copies":7064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3532,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16338,"num_write_kb":635777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"307'16414","reported_seq":17105,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-04-20T17:17:40.533052+0000","last_change":"2024-04-20T17:16:23.593408+0000","last_active":"2024-04-20T17:17:40.533052+0000","last_peered":"2024-04-20T17:17:40.533052+0000","last_clean":"2024-04-20T17:17:40.533052+0000","last_became_active":"2024-04-20T17:12:23.515375+0000","last_became_peered":"2024-04-20T17:12:23.515375+0000","last_unstale":"2024-04-20T17:17:40.533052+0000","last_undegraded":"2024-04-20T17:17:40.533052+0000","last_fullsized":"2024-04-20T17:17:40.533052+0000","mapping_epoch":202,"log_start":"136'6400","ondisk_log_start":"136'6400","created":115,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'15586","last_scrub_stamp":"2024-04-20T17:16:23.593300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:23.593300+0000","objects_scrubbed":4410,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234749952,"num_objects":3582,"num_object_clones":0,"num_object_copies":7164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16414,"num_write_kb":639872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":3749576728,"num_objects":57215,"num_object_clones":0,"num_object_copies":114430,"num_objects_missing_on_primary":3019,"num_objects_missing":3019,"num_objects_degraded":0,"num_objects_misplaced":6051,"num_objects_unfound":0,"num_objects_dirty":57215,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":264769,"num_write_kb":10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26184,"num_bytes_recovered":1715994624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8160157696,"data_stored":8160149552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},"log_size":160662,"ondisk_log_size":160662,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991375,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":769148,"kb_used_data":693552,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92976004,"statfs":{"total":95995035648,"available":95207428096,"internally_reserved":0,"allocated":710197248,"data_stored":709187140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":77396420},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253013,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":865344,"kb_used_data":721716,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92879808,"statfs":{"total":95995035648,"available":95108923392,"internally_reserved":0,"allocated":739037184,"data_stored":738024700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":147061184},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612763,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1062428,"kb_used_data":918800,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92682724,"statfs":{"total":95995035648,"available":94907109376,"internally_reserved":0,"allocated":940851200,"data_stored":939822758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":147061184},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762808992,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1012416,"kb_used_data":913204,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92732736,"statfs":{"total":95995035648,"available":94958321664,"internally_reserved":0,"allocated":935120896,"data_stored":934108432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168741,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":765372,"kb_used_data":698096,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92979780,"statfs":{"total":95995035648,"available":95211294720,"internally_reserved":0,"allocated":714850304,"data_stored":713840216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430377,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":581168,"kb_used_data":469480,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93163984,"statfs":{"total":95995035648,"available":95399919616,"internally_reserved":0,"allocated":480747520,"data_stored":479740984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575044,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":562660,"kb_used_data":455128,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93182492,"statfs":{"total":95995035648,"available":95418871808,"internally_reserved":0,"allocated":466051072,"data_stored":465020724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738568,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":866944,"kb_used_data":719156,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92878208,"statfs":{"total":95995035648,"available":95107284992,"internally_reserved":0,"allocated":736415744,"data_stored":735403280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,13,1,0,0,1,1,3,6,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393857,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":709788,"kb_used_data":661388,"kb_used_omap":202,"kb_used_meta":48181,"kb_avail":93035364,"statfs":{"total":95995035648,"available":95268212736,"internally_reserved":0,"allocated":677261312,"data_stored":676236942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":49337815},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182585,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":535320,"kb_used_data":459724,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93209832,"statfs":{"total":95995035648,"available":95446867968,"internally_reserved":0,"allocated":470757376,"data_stored":469733006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986356,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":255452,"kb_used_data":228432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93489700,"statfs":{"total":95995035648,"available":95733452800,"internally_reserved":0,"allocated":233914368,"data_stored":232885926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790126,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1178908,"kb_used_data":1071372,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92566244,"statfs":{"total":95995035648,"available":94787833856,"internally_reserved":0,"allocated":1097084928,"data_stored":1096060558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":733282304,"data_stored":733282304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":462749696,"data_stored":462749696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":674955264,"data_stored":674955264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":467992576,"data_stored":467992576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":231608320,"data_stored":231604248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1094189056,"data_stored":1094189056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":476643328,"data_stored":476643328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":710475776,"data_stored":710475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":930086912,"data_stored":930086912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":937562112,"data_stored":937558040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":735903744,"data_stored":735903744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":704708608,"data_stored":704708608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:17:44.708 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:17:44.738 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:44.839 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:17:44.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:44.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:17:44.945 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:17:44.973 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:45.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:45 smithi005 ceph-mon[39562]: pgmap v847: 17 pgs: 1 active+recovering, 16 active+clean; 3.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 167 op/s; 5929/113364 objects misplaced (5.230%); 661 KiB/s, 10 objects/s recovering 2024-04-20T17:17:45.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:45 smithi171 ceph-mon[44189]: pgmap v847: 17 pgs: 1 active+recovering, 16 active+clean; 3.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 167 op/s; 5929/113364 objects misplaced (5.230%); 661 KiB/s, 10 objects/s recovering 2024-04-20T17:17:45.623 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:17:45.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:45.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:17:45.727 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:17:45.750 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:45 smithi106 ceph-mon[44423]: pgmap v847: 17 pgs: 1 active+recovering, 16 active+clean; 3.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 167 op/s; 5929/113364 objects misplaced (5.230%); 661 KiB/s, 10 objects/s recovering 2024-04-20T17:17:45.752 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:45.771 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:17:45.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:45.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:17:45.875 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:17:45.901 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:45.921 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:17:45.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:45.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:17:46.023 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:17:46.050 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:46.069 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:17:46.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:46.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:17:46.171 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:17:46.202 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:46.378 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:17:46.399 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:17:46.419 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:17:46.440 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:17:46.460 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:17:46.483 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:17:46.503 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:17:46.515 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:17:46.515 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:17:46.524 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:17:46.547 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:17:46.568 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:17:46.588 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:17:46.607 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:17:46.609 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:17:46.609 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:17:46.609 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:17:46.609 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:17:46.609 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:17:46.609 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:17:46.609 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:17:46.609 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:17:46.609 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:17:46.609 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:17:46.610 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:17:46.610 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:17:46.753 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:17:46.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:46.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:17:46.857 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:17:46.883 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:46.902 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:17:46.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:46.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:17:47.006 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:17:47.036 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:47.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:46 smithi106 ceph-mon[44423]: pgmap v848: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 183 op/s; 5929/112536 objects misplaced (5.269%); 512 KiB/s, 7 objects/s recovering 2024-04-20T17:17:47.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:46 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2602016994' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:17:47.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:47.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:47.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:47.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:47.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:47.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:47.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:47.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:46 smithi171 ceph-mon[44189]: pgmap v848: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 183 op/s; 5929/112536 objects misplaced (5.269%); 512 KiB/s, 7 objects/s recovering 2024-04-20T17:17:47.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:46 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2602016994' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:17:47.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:47.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:47.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:47.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:47.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:47.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:47.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:47.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:46 smithi005 ceph-mon[39562]: pgmap v848: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 183 op/s; 5929/112536 objects misplaced (5.269%); 512 KiB/s, 7 objects/s recovering 2024-04-20T17:17:47.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:46 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2602016994' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:17:47.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:47.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:47.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:47.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:47.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:47.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:47.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:47.489 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:17:47.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:47.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:17:47.593 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:17:47.620 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:47.629 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:17:47.639 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:17:47.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:47.642 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:17:47.744 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:17:47.770 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:47.791 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:17:47.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:47.793 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:17:47.894 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:17:47.920 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:47.939 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:17:47.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:47.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:17:48.039 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-20T17:17:48.043 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:17:48.070 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:48.090 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:17:48.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:48.093 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:17:48.093 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:17:48.103 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:17:48.194 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:17:48.225 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:48.225 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:48.225 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:48.225 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:48.225 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:48.226 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:48.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:48.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:48.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:48.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:48.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:48.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:48.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:48.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:48.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:48.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:17:48.431 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:17:48.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:48.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:17:48.535 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:17:48.569 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:49.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:48 smithi005 ceph-mon[39562]: pgmap v849: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 201 op/s; 5799/111512 objects misplaced (5.200%); 922 KiB/s, 14 objects/s recovering 2024-04-20T17:17:49.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:48 smithi106 ceph-mon[44423]: pgmap v849: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 201 op/s; 5799/111512 objects misplaced (5.200%); 922 KiB/s, 14 objects/s recovering 2024-04-20T17:17:49.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:48 smithi171 ceph-mon[44189]: pgmap v849: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 201 op/s; 5799/111512 objects misplaced (5.200%); 922 KiB/s, 14 objects/s recovering 2024-04-20T17:17:49.517 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:17:49.724 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:17:51.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:51 smithi005 ceph-mon[39562]: pgmap v850: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 196 op/s; 5799/111090 objects misplaced (5.220%); 806 KiB/s, 12 objects/s recovering 2024-04-20T17:17:51.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:51 smithi106 ceph-mon[44423]: pgmap v850: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 196 op/s; 5799/111090 objects misplaced (5.220%); 806 KiB/s, 12 objects/s recovering 2024-04-20T17:17:51.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:51 smithi171 ceph-mon[44189]: pgmap v850: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 196 op/s; 5799/111090 objects misplaced (5.220%); 806 KiB/s, 12 objects/s recovering 2024-04-20T17:17:52.124 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:17:52.349 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:17:52.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:52.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:17:52.454 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:17:52.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:52 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/46400696' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:17:52.485 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:52.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:52 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/46400696' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:17:52.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:52 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/46400696' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:17:52.588 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:17:52.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:52.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:17:52.693 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:17:52.719 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:52.737 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:17:52.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:52.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:17:52.841 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:17:52.870 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:52.891 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:17:52.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:52.895 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:17:52.996 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:17:53.027 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:53.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:53 smithi005 ceph-mon[39562]: pgmap v851: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 196 op/s; 5799/110186 objects misplaced (5.263%); 727 KiB/s, 11 objects/s recovering 2024-04-20T17:17:53.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:53 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:17:53.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:53 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/46400696' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:17:53.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:53 smithi005 ceph-mon[39562]: osdmap e309: 12 total, 12 up, 12 in 2024-04-20T17:17:53.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:53 smithi106 ceph-mon[44423]: pgmap v851: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 196 op/s; 5799/110186 objects misplaced (5.263%); 727 KiB/s, 11 objects/s recovering 2024-04-20T17:17:53.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:53 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:17:53.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:53 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/46400696' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:17:53.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:53 smithi106 ceph-mon[44423]: osdmap e309: 12 total, 12 up, 12 in 2024-04-20T17:17:53.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:53 smithi171 ceph-mon[44189]: pgmap v851: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 196 op/s; 5799/110186 objects misplaced (5.263%); 727 KiB/s, 11 objects/s recovering 2024-04-20T17:17:53.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:53 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:17:53.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:53 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/46400696' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:17:53.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:53 smithi171 ceph-mon[44189]: osdmap e309: 12 total, 12 up, 12 in 2024-04-20T17:17:54.078 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:17:54.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:54.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:17:54.185 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:17:54.214 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:54.242 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:17:54.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:54.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:17:54.346 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:17:54.372 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:54.391 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:17:54.392 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:54 smithi106 ceph-mon[44423]: 2.1 scrub starts 2024-04-20T17:17:54.392 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:54 smithi106 ceph-mon[44423]: 2.1 scrub ok 2024-04-20T17:17:54.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:54.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:17:54.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:54 smithi005 ceph-mon[39562]: 2.1 scrub starts 2024-04-20T17:17:54.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:54 smithi005 ceph-mon[39562]: 2.1 scrub ok 2024-04-20T17:17:54.494 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:17:54.521 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:54.539 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:17:54.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:54.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:17:54.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:54 smithi171 ceph-mon[44189]: 2.1 scrub starts 2024-04-20T17:17:54.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:54 smithi171 ceph-mon[44189]: 2.1 scrub ok 2024-04-20T17:17:54.606 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:17:54.609 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:17:54.642 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:17:54.671 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:54.692 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:17:54.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:54.696 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:17:54.797 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:17:54.828 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:55.080 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:17:55.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:55.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:17:55.184 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:17:55.211 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:55.232 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:17:55.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:55.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:17:55.337 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:17:55.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:55 smithi005 ceph-mon[39562]: pgmap v853: 17 pgs: 1 active+recovering, 16 active+clean; 3.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 231 op/s; 5671/108732 objects misplaced (5.216%); 825 KiB/s, 12 objects/s recovering 2024-04-20T17:17:55.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:55 smithi005 ceph-mon[39562]: from='client.25544 v1:172.21.15.5:0/974571169' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:17:55.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:55 smithi005 ceph-mon[39562]: 2.f scrub starts 2024-04-20T17:17:55.367 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:55.390 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:17:55.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:55.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:17:55.498 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:17:55.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:55 smithi106 ceph-mon[44423]: pgmap v853: 17 pgs: 1 active+recovering, 16 active+clean; 3.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 231 op/s; 5671/108732 objects misplaced (5.216%); 825 KiB/s, 12 objects/s recovering 2024-04-20T17:17:55.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:55 smithi106 ceph-mon[44423]: from='client.25544 v1:172.21.15.5:0/974571169' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:17:55.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:55 smithi106 ceph-mon[44423]: 2.f scrub starts 2024-04-20T17:17:55.523 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:55 smithi171 ceph-mon[44189]: pgmap v853: 17 pgs: 1 active+recovering, 16 active+clean; 3.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 231 op/s; 5671/108732 objects misplaced (5.216%); 825 KiB/s, 12 objects/s recovering 2024-04-20T17:17:55.523 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:55 smithi171 ceph-mon[44189]: from='client.25544 v1:172.21.15.5:0/974571169' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:17:55.523 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:55 smithi171 ceph-mon[44189]: 2.f scrub starts 2024-04-20T17:17:55.525 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:55.543 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:17:55.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:55.546 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:17:55.647 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:17:55.674 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:55.695 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:17:55.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:55.698 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:17:55.799 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:17:55.828 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:55.848 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:17:55.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:55.851 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:17:55.952 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:17:55.990 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:56.010 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:17:56.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:56.015 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:17:56.117 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:17:56.144 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:56.163 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:17:56.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:56.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:17:56.266 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:17:56.292 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:56.311 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:17:56.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:56.314 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:17:56.336 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":853,"stamp":"2024-04-20T17:17:54.064247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3563323960,"num_objects":54366,"num_object_clones":0,"num_object_copies":108732,"num_objects_missing_on_primary":2830,"num_objects_missing":2830,"num_objects_degraded":0,"num_objects_misplaced":5671,"num_objects_unfound":0,"num_objects_dirty":54366,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":267801,"num_write_kb":10306363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26383,"num_bytes_recovered":1730627712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160850,"ondisk_log_size":160850,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":8853844,"kb_used_data":7672616,"kb_used_omap":219,"kb_used_meta":1180900,"kb_avail":1116087980,"statfs":{"total":1151940427776,"available":1142874091520,"internally_reserved":0,"allocated":7856758784,"data_stored":7844355408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224949,"internal_metadata":1209241931},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1,1,1,1,5,6,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-151781376,"num_objects":-2316,"num_object_clones":0,"num_object_copies":-4632,"num_objects_missing_on_primary":-129,"num_objects_missing":-129,"num_objects_degraded":0,"num_objects_misplaced":-258,"num_objects_unfound":0,"num_objects_dirty":-2316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2316,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":8454144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001907"},"pg_stats":[{"pgid":"2.f","version":"309'16802","reported_seq":17709,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-04-20T17:17:53.546541+0000","last_change":"2024-04-20T17:16:24.907036+0000","last_active":"2024-04-20T17:17:53.546541+0000","last_peered":"2024-04-20T17:17:53.546541+0000","last_clean":"2024-04-20T17:17:53.546541+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:17:53.546541+0000","last_undegraded":"2024-04-20T17:17:53.546541+0000","last_fullsized":"2024-04-20T17:17:53.546541+0000","mapping_epoch":115,"log_start":"136'6800","ondisk_log_start":"136'6800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'15773","last_scrub_stamp":"2024-04-20T17:16:24.906931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:24.906931+0000","objects_scrubbed":4413,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221511680,"num_objects":3380,"num_object_clones":0,"num_object_copies":6760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16802,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"308'16604","reported_seq":28576,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-04-20T17:17:50.434002+0000","last_change":"2024-04-20T17:16:40.290499+0000","last_active":"2024-04-20T17:17:50.434002+0000","last_peered":"2024-04-20T17:17:50.434002+0000","last_clean":"2024-04-20T17:17:50.434002+0000","last_became_active":"2024-04-20T17:10:28.865670+0000","last_became_peered":"2024-04-20T17:10:28.865670+0000","last_unstale":"2024-04-20T17:17:50.434002+0000","last_undegraded":"2024-04-20T17:17:50.434002+0000","last_fullsized":"2024-04-20T17:17:50.434002+0000","mapping_epoch":162,"log_start":"136'6600","ondisk_log_start":"136'6600","created":115,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'15835","last_scrub_stamp":"2024-04-20T17:16:40.290402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:40.290402+0000","objects_scrubbed":4222,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:55.604081+0000","scrub_duration":371,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226230272,"num_objects":3452,"num_object_clones":0,"num_object_copies":6904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16604,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5619,"num_bytes_recovered":368246784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"308'17013","reported_seq":17806,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-04-20T17:17:50.370034+0000","last_change":"2024-04-20T17:17:12.348489+0000","last_active":"2024-04-20T17:17:50.370034+0000","last_peered":"2024-04-20T17:17:50.370034+0000","last_clean":"2024-04-20T17:17:50.370034+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:17:50.370034+0000","last_undegraded":"2024-04-20T17:17:50.370034+0000","last_fullsized":"2024-04-20T17:17:50.370034+0000","mapping_epoch":115,"log_start":"136'7000","ondisk_log_start":"136'7000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'16554","last_scrub_stamp":"2024-04-20T17:17:12.348391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:12.348391+0000","objects_scrubbed":3930,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:33.813428+0000","scrub_duration":337,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227344384,"num_objects":3469,"num_object_clones":0,"num_object_copies":6938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17013,"num_write_kb":655424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"308'23364","reported_seq":44911,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-04-20T17:17:50.675802+0000","last_change":"2024-04-20T17:16:35.136358+0000","last_active":"2024-04-20T17:17:50.675802+0000","last_peered":"2024-04-20T17:17:50.675802+0000","last_clean":"2024-04-20T17:17:50.675802+0000","last_became_active":"2024-04-20T17:12:08.085936+0000","last_became_peered":"2024-04-20T17:12:08.085936+0000","last_unstale":"2024-04-20T17:17:50.675802+0000","last_undegraded":"2024-04-20T17:17:50.675802+0000","last_fullsized":"2024-04-20T17:17:50.675802+0000","mapping_epoch":196,"log_start":"168'13268","ondisk_log_start":"168'13268","created":115,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'22494","last_scrub_stamp":"2024-04-20T17:16:35.136177+0000","last_deep_scrub":"213'20373","last_deep_scrub_stamp":"2024-04-20T17:12:48.443808+0000","last_clean_scrub_stamp":"2024-04-20T17:16:35.136177+0000","objects_scrubbed":4393,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230817792,"num_objects":3522,"num_object_clones":0,"num_object_copies":7044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16712,"num_write_kb":647488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":450166784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"309'16460","reported_seq":36649,"reported_epoch":309,"state":"active+recovering","last_fresh":"2024-04-20T17:17:52.995833+0000","last_change":"2024-04-20T17:12:26.166109+0000","last_active":"2024-04-20T17:17:52.995833+0000","last_peered":"2024-04-20T17:17:52.995833+0000","last_clean":"2024-04-20T17:09:35.882603+0000","last_became_active":"2024-04-20T17:10:54.470253+0000","last_became_peered":"2024-04-20T17:10:54.470253+0000","last_unstale":"2024-04-20T17:17:52.995833+0000","last_undegraded":"2024-04-20T17:17:52.995833+0000","last_fullsized":"2024-04-20T17:17:52.995833+0000","mapping_epoch":171,"log_start":"136'6400","ondisk_log_start":"136'6400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214302720,"num_objects":3270,"num_object_clones":0,"num_object_copies":6540,"num_objects_missing_on_primary":2830,"num_objects_missing":2830,"num_objects_degraded":0,"num_objects_misplaced":5671,"num_objects_unfound":0,"num_objects_dirty":3270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16460,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6591,"num_bytes_recovered":431947776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"0,2,5","objects":11},{"shards":"0,10","objects":429},{"shards":"2,5","objects":2830}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":813,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-04-20T17:17:52.167492+0000","last_change":"2024-04-20T17:16:40.132382+0000","last_active":"2024-04-20T17:17:52.167492+0000","last_peered":"2024-04-20T17:17:52.167492+0000","last_clean":"2024-04-20T17:17:52.167492+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:17:52.167492+0000","last_undegraded":"2024-04-20T17:17:52.167492+0000","last_fullsized":"2024-04-20T17:17:52.167492+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:16:40.132254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:16:40.132254+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:54.442752+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"309'16806","reported_seq":17632,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-04-20T17:17:52.638813+0000","last_change":"2024-04-20T17:17:10.481774+0000","last_active":"2024-04-20T17:17:52.638813+0000","last_peered":"2024-04-20T17:17:52.638813+0000","last_clean":"2024-04-20T17:17:52.638813+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:17:52.638813+0000","last_undegraded":"2024-04-20T17:17:52.638813+0000","last_fullsized":"2024-04-20T17:17:52.638813+0000","mapping_epoch":115,"log_start":"136'6800","ondisk_log_start":"136'6800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'16300","last_scrub_stamp":"2024-04-20T17:17:10.481666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:10.481666+0000","objects_scrubbed":3956,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:18.079313+0000","scrub_duration":337,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225837056,"num_objects":3446,"num_object_clones":0,"num_object_copies":6892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16806,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"309'16954","reported_seq":17779,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-04-20T17:17:53.587045+0000","last_change":"2024-04-20T17:17:15.112506+0000","last_active":"2024-04-20T17:17:53.587045+0000","last_peered":"2024-04-20T17:17:53.587045+0000","last_clean":"2024-04-20T17:17:53.587045+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:17:53.587045+0000","last_undegraded":"2024-04-20T17:17:53.587045+0000","last_fullsized":"2024-04-20T17:17:53.587045+0000","mapping_epoch":115,"log_start":"136'6900","ondisk_log_start":"136'6900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'16484","last_scrub_stamp":"2024-04-20T17:17:15.112253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:15.112253+0000","objects_scrubbed":3848,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:45.026807+0000","scrub_duration":457,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220856320,"num_objects":3370,"num_object_clones":0,"num_object_copies":6740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16954,"num_write_kb":650368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"308'20848","reported_seq":21772,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-04-20T17:17:51.948268+0000","last_change":"2024-04-20T17:16:25.742067+0000","last_active":"2024-04-20T17:17:51.948268+0000","last_peered":"2024-04-20T17:17:51.948268+0000","last_clean":"2024-04-20T17:17:51.948268+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:17:51.948268+0000","last_undegraded":"2024-04-20T17:17:51.948268+0000","last_fullsized":"2024-04-20T17:17:51.948268+0000","mapping_epoch":115,"log_start":"154'10802","ondisk_log_start":"154'10802","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'19891","last_scrub_stamp":"2024-04-20T17:16:25.741880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:25.741880+0000","objects_scrubbed":4262,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:55.327619+0000","scrub_duration":378,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216596480,"num_objects":3305,"num_object_clones":0,"num_object_copies":6610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16737,"num_write_kb":641344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"309'16848","reported_seq":17730,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-04-20T17:17:52.457633+0000","last_change":"2024-04-20T17:17:08.025358+0000","last_active":"2024-04-20T17:17:52.457633+0000","last_peered":"2024-04-20T17:17:52.457633+0000","last_clean":"2024-04-20T17:17:52.457633+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:17:52.457633+0000","last_undegraded":"2024-04-20T17:17:52.457633+0000","last_fullsized":"2024-04-20T17:17:52.457633+0000","mapping_epoch":115,"log_start":"136'6800","ondisk_log_start":"136'6800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'16358","last_scrub_stamp":"2024-04-20T17:17:08.025214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:08.025214+0000","objects_scrubbed":3866,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:25.817318+0000","scrub_duration":359,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220987392,"num_objects":3372,"num_object_clones":0,"num_object_copies":6744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16848,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"309'16662","reported_seq":17469,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-04-20T17:17:52.610167+0000","last_change":"2024-04-20T17:17:16.522161+0000","last_active":"2024-04-20T17:17:52.610167+0000","last_peered":"2024-04-20T17:17:52.610167+0000","last_clean":"2024-04-20T17:17:52.610167+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:17:52.610167+0000","last_undegraded":"2024-04-20T17:17:52.610167+0000","last_fullsized":"2024-04-20T17:17:52.610167+0000","mapping_epoch":115,"log_start":"136'6600","ondisk_log_start":"136'6600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'16253","last_scrub_stamp":"2024-04-20T17:17:16.522032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:16.522032+0000","objects_scrubbed":3799,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:29.552786+0000","scrub_duration":315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222167040,"num_objects":3390,"num_object_clones":0,"num_object_copies":6780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16662,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"309'16837","reported_seq":32205,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-04-20T17:17:52.817108+0000","last_change":"2024-04-20T17:16:34.421711+0000","last_active":"2024-04-20T17:17:52.817108+0000","last_peered":"2024-04-20T17:17:52.817108+0000","last_clean":"2024-04-20T17:17:52.817108+0000","last_became_active":"2024-04-20T17:10:55.931751+0000","last_became_peered":"2024-04-20T17:10:55.931751+0000","last_unstale":"2024-04-20T17:17:52.817108+0000","last_undegraded":"2024-04-20T17:17:52.817108+0000","last_fullsized":"2024-04-20T17:17:52.817108+0000","mapping_epoch":173,"log_start":"136'6800","ondisk_log_start":"136'6800","created":115,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'15941","last_scrub_stamp":"2024-04-20T17:16:34.421587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:34.421587+0000","objects_scrubbed":4322,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:17:54.245981+0000","scrub_duration":366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224460800,"num_objects":3425,"num_object_clones":0,"num_object_copies":6850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16837,"num_write_kb":648384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7296,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"309'16928","reported_seq":17754,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-04-20T17:17:52.507750+0000","last_change":"2024-04-20T17:16:42.499318+0000","last_active":"2024-04-20T17:17:52.507750+0000","last_peered":"2024-04-20T17:17:52.507750+0000","last_clean":"2024-04-20T17:17:52.507750+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:17:52.507750+0000","last_undegraded":"2024-04-20T17:17:52.507750+0000","last_fullsized":"2024-04-20T17:17:52.507750+0000","mapping_epoch":115,"log_start":"136'6900","ondisk_log_start":"136'6900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'16092","last_scrub_stamp":"2024-04-20T17:16:42.499193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:42.499193+0000","objects_scrubbed":4224,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:04.645993+0000","scrub_duration":355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222035968,"num_objects":3388,"num_object_clones":0,"num_object_copies":6776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16928,"num_write_kb":650112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"309'16459","reported_seq":17281,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-04-20T17:17:53.480583+0000","last_change":"2024-04-20T17:17:06.994658+0000","last_active":"2024-04-20T17:17:53.480583+0000","last_peered":"2024-04-20T17:17:53.480583+0000","last_clean":"2024-04-20T17:17:53.480583+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:17:53.480583+0000","last_undegraded":"2024-04-20T17:17:53.480583+0000","last_fullsized":"2024-04-20T17:17:53.480583+0000","mapping_epoch":115,"log_start":"136'6400","ondisk_log_start":"136'6400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'15884","last_scrub_stamp":"2024-04-20T17:17:06.994455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:06.994455+0000","objects_scrubbed":3982,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:21.957254+0000","scrub_duration":397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223150080,"num_objects":3405,"num_object_clones":0,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16459,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"309'16751","reported_seq":17572,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-04-20T17:17:52.520152+0000","last_change":"2024-04-20T17:17:07.882718+0000","last_active":"2024-04-20T17:17:52.520152+0000","last_peered":"2024-04-20T17:17:52.520152+0000","last_clean":"2024-04-20T17:17:52.520152+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:17:52.520152+0000","last_undegraded":"2024-04-20T17:17:52.520152+0000","last_fullsized":"2024-04-20T17:17:52.520152+0000","mapping_epoch":115,"log_start":"136'6700","ondisk_log_start":"136'6700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'16236","last_scrub_stamp":"2024-04-20T17:17:07.882596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:07.882596+0000","objects_scrubbed":3868,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:32.661323+0000","scrub_duration":370,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219611136,"num_objects":3351,"num_object_clones":0,"num_object_copies":6702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16751,"num_write_kb":643264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"308'16470","reported_seq":17290,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-04-20T17:17:50.709144+0000","last_change":"2024-04-20T17:17:04.896764+0000","last_active":"2024-04-20T17:17:50.709144+0000","last_peered":"2024-04-20T17:17:50.709144+0000","last_clean":"2024-04-20T17:17:50.709144+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:17:50.709144+0000","last_undegraded":"2024-04-20T17:17:50.709144+0000","last_fullsized":"2024-04-20T17:17:50.709144+0000","mapping_epoch":115,"log_start":"136'6400","ondisk_log_start":"136'6400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"293'15941","last_scrub_stamp":"2024-04-20T17:17:04.896640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:04.896640+0000","objects_scrubbed":3929,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:16.518914+0000","scrub_duration":355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222756888,"num_objects":3400,"num_object_clones":0,"num_object_copies":6800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3400,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16470,"num_write_kb":635777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"308'16577","reported_seq":17270,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-04-20T17:17:50.861966+0000","last_change":"2024-04-20T17:16:23.593408+0000","last_active":"2024-04-20T17:17:50.861966+0000","last_peered":"2024-04-20T17:17:50.861966+0000","last_clean":"2024-04-20T17:17:50.861966+0000","last_became_active":"2024-04-20T17:12:23.515375+0000","last_became_peered":"2024-04-20T17:12:23.515375+0000","last_unstale":"2024-04-20T17:17:50.861966+0000","last_undegraded":"2024-04-20T17:17:50.861966+0000","last_fullsized":"2024-04-20T17:17:50.861966+0000","mapping_epoch":202,"log_start":"136'6500","ondisk_log_start":"136'6500","created":115,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'15586","last_scrub_stamp":"2024-04-20T17:16:23.593300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:23.593300+0000","objects_scrubbed":4410,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224067584,"num_objects":3419,"num_object_clones":0,"num_object_copies":6838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16577,"num_write_kb":639872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":3562733592,"num_objects":54364,"num_object_clones":0,"num_object_copies":108728,"num_objects_missing_on_primary":2830,"num_objects_missing":2830,"num_objects_degraded":0,"num_objects_misplaced":5671,"num_objects_unfound":0,"num_objects_dirty":54364,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":267620,"num_write_kb":10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26375,"num_bytes_recovered":1728512000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7810064384,"data_stored":7810056240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},"log_size":160713,"ondisk_log_size":160713,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991378,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":734696,"kb_used_data":654944,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93010456,"statfs":{"total":95995035648,"available":95242706944,"internally_reserved":0,"allocated":670662656,"data_stored":669628736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":81656260},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253016,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":850092,"kb_used_data":684260,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":92895060,"statfs":{"total":95995035648,"available":95124541440,"internally_reserved":0,"allocated":700682240,"data_stored":699645894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":169802176},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612765,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1029540,"kb_used_data":885916,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92715612,"statfs":{"total":95995035648,"available":94940786688,"internally_reserved":0,"allocated":907177984,"data_stored":906143564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":147061184},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762808995,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":965936,"kb_used_data":862564,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92779216,"statfs":{"total":95995035648,"available":95005917184,"internally_reserved":0,"allocated":883265536,"data_stored":882229210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168744,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":732792,"kb_used_data":661356,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93012360,"statfs":{"total":95995035648,"available":95244656640,"internally_reserved":0,"allocated":677228544,"data_stored":676188666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430380,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":542752,"kb_used_data":443544,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93202400,"statfs":{"total":95995035648,"available":95439257600,"internally_reserved":0,"allocated":454189056,"data_stored":453158708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101579206},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575046,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":543520,"kb_used_data":435992,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93201632,"statfs":{"total":95995035648,"available":95438471168,"internally_reserved":0,"allocated":446455808,"data_stored":445425460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738571,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":832944,"kb_used_data":680996,"kb_used_omap":1,"kb_used_meta":151934,"kb_avail":92912208,"statfs":{"total":95995035648,"available":95142100992,"internally_reserved":0,"allocated":697339904,"data_stored":696303578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":155580869},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1,1,1,5,6,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393859,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":699872,"kb_used_data":651480,"kb_used_omap":202,"kb_used_meta":48181,"kb_avail":93045280,"statfs":{"total":95995035648,"available":95278366720,"internally_reserved":0,"allocated":667115520,"data_stored":666085172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":49337815},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182587,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":516640,"kb_used_data":441048,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93228512,"statfs":{"total":95995035648,"available":95465996288,"internally_reserved":0,"allocated":451633152,"data_stored":450602804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986358,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":247524,"kb_used_data":220508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93497628,"statfs":{"total":95995035648,"available":95741571072,"internally_reserved":0,"allocated":225800192,"data_stored":224765772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790128,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1157536,"kb_used_data":1050008,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92587616,"statfs":{"total":95995035648,"available":94809718784,"internally_reserved":0,"allocated":1075208192,"data_stored":1074177844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":693239808,"data_stored":693239808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":442695680,"data_stored":442695680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":664272896,"data_stored":664272896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":447938560,"data_stored":447938560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":222957568,"data_stored":222953496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1071972352,"data_stored":1071972352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":450625536,"data_stored":450625536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":672333824,"data_stored":672333824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":881131520,"data_stored":881131520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":901517312,"data_stored":901513240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":695926784,"data_stored":695926784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":665452544,"data_stored":665452544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:17:56.338 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:17:56.414 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:17:56.439 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:56 smithi171 ceph-mon[44189]: 2.f scrub ok 2024-04-20T17:17:56.440 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:56 smithi171 ceph-mon[44189]: osdmap e310: 12 total, 12 up, 12 in 2024-04-20T17:17:56.441 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:56.461 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:17:56.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:56 smithi005 ceph-mon[39562]: 2.f scrub ok 2024-04-20T17:17:56.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:56 smithi005 ceph-mon[39562]: osdmap e310: 12 total, 12 up, 12 in 2024-04-20T17:17:56.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:56.463 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:17:56.491 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:17:56.491 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:17:56.492 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:17:56.492 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:17:56.492 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.492 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212446 2.e 2:72d00b8c:::benchmark_data_smithi171_78918_object51452:head [delete] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:17:56.492 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897321+0000", 2024-04-20T17:17:56.492 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 332.90711246699999, 2024-04-20T17:17:56.492 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.67883147099999996, 2024-04-20T17:17:56.492 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:17:56.492 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:17:56.492 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:56.492 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:17:56.492 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:17:56.492 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:17:56.492 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212446 2024-04-20T17:17:56.492 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.493 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:17:56.493 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.493 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:17:56.493 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897321+0000", 2024-04-20T17:17:56.493 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.493 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.493 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.493 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:17:56.493 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897321+0000", 2024-04-20T17:17:56.493 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.493 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.493 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.493 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:17:56.493 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897323+0000", 2024-04-20T17:17:56.494 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.424e-06 2024-04-20T17:17:56.494 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.494 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.494 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:17:56.494 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897329+0000", 2024-04-20T17:17:56.494 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5559999999999998e-06 2024-04-20T17:17:56.494 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.494 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.494 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:17:56.494 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897338+0000", 2024-04-20T17:17:56.494 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0520000000000007e-06 2024-04-20T17:17:56.494 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.494 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.494 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:17:56.495 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897346+0000", 2024-04-20T17:17:56.495 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7619999999999995e-06 2024-04-20T17:17:56.495 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.495 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.495 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.495 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898452+0000", 2024-04-20T17:17:56.495 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0011067480000000001 2024-04-20T17:17:56.495 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.495 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.495 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:17:56.495 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898470+0000", 2024-04-20T17:17:56.495 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7864e-05 2024-04-20T17:17:56.495 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.495 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.496 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.496 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.559597+0000", 2024-04-20T17:17:56.496 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.66112692699999998 2024-04-20T17:17:56.496 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.496 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.496 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:17:56.496 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.559698+0000", 2024-04-20T17:17:56.496 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000101 2024-04-20T17:17:56.496 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.496 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.496 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:17:56.496 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.559788+0000", 2024-04-20T17:17:56.496 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9697999999999994e-05 2024-04-20T17:17:56.497 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.497 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.497 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:17:56.497 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.575978+0000", 2024-04-20T17:17:56.497 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.016189985 2024-04-20T17:17:56.497 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.497 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.497 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:17:56.497 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576031+0000", 2024-04-20T17:17:56.497 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3086e-05 2024-04-20T17:17:56.497 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.497 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.497 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:17:56.498 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576097+0000", 2024-04-20T17:17:56.498 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6303999999999994e-05 2024-04-20T17:17:56.498 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.498 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.498 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:17:56.498 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576152+0000", 2024-04-20T17:17:56.498 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5065000000000002e-05 2024-04-20T17:17:56.498 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.498 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:17:56.498 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.498 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.498 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.498 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212450 2.e 2:72ccd005:::benchmark_data_smithi171_78918_object51456:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:17:56.498 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897360+0000", 2024-04-20T17:17:56.498 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 332.90707284500002, 2024-04-20T17:17:56.499 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.67896632000000001, 2024-04-20T17:17:56.499 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:17:56.499 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:17:56.499 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:56.499 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:17:56.499 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:17:56.499 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:17:56.499 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212450 2024-04-20T17:17:56.499 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.499 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:17:56.499 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.499 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:17:56.499 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897360+0000", 2024-04-20T17:17:56.499 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.500 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.500 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.500 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:17:56.500 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897360+0000", 2024-04-20T17:17:56.500 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.500 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.500 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.500 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:17:56.500 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897361+0000", 2024-04-20T17:17:56.500 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2800000000000005e-07 2024-04-20T17:17:56.500 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.500 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.501 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:17:56.501 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897364+0000", 2024-04-20T17:17:56.501 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.977e-06 2024-04-20T17:17:56.501 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.501 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.501 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:17:56.501 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897370+0000", 2024-04-20T17:17:56.501 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0530000000000003e-06 2024-04-20T17:17:56.501 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.501 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.501 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:17:56.501 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897376+0000", 2024-04-20T17:17:56.501 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1419999999999998e-06 2024-04-20T17:17:56.501 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.501 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.502 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.502 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898518+0000", 2024-04-20T17:17:56.502 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0011413 2024-04-20T17:17:56.502 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.502 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.502 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:17:56.502 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898526+0000", 2024-04-20T17:17:56.502 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0150000000000003e-06 2024-04-20T17:17:56.502 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.502 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.502 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.502 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.563095+0000", 2024-04-20T17:17:56.502 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.664568838 2024-04-20T17:17:56.502 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.503 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.503 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:17:56.503 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.563218+0000", 2024-04-20T17:17:56.503 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000123051 2024-04-20T17:17:56.503 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.503 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.503 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:17:56.503 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.563315+0000", 2024-04-20T17:17:56.503 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6935000000000006e-05 2024-04-20T17:17:56.503 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.503 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.503 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:17:56.503 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576186+0000", 2024-04-20T17:17:56.503 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012871698000000001 2024-04-20T17:17:56.504 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.504 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.504 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:17:56.504 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576223+0000", 2024-04-20T17:17:56.504 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7048999999999997e-05 2024-04-20T17:17:56.504 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.504 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.504 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:17:56.504 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576275+0000", 2024-04-20T17:17:56.504 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1944000000000003e-05 2024-04-20T17:17:56.504 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.504 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.504 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:17:56.504 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576327+0000", 2024-04-20T17:17:56.504 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1390000000000001e-05 2024-04-20T17:17:56.505 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.505 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:17:56.505 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.505 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.505 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.505 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212454 2.e 2:7603a42b:::benchmark_data_smithi171_78918_object51460:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:17:56.505 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897383+0000", 2024-04-20T17:17:56.505 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 332.90705027500002, 2024-04-20T17:17:56.505 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.67906811499999997, 2024-04-20T17:17:56.505 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:17:56.505 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:17:56.505 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:56.505 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:17:56.505 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:17:56.505 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:17:56.506 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212454 2024-04-20T17:17:56.506 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.506 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:17:56.506 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.506 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:17:56.506 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897383+0000", 2024-04-20T17:17:56.506 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.506 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.506 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.506 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:17:56.506 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897383+0000", 2024-04-20T17:17:56.506 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.506 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.506 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.507 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:17:56.507 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897383+0000", 2024-04-20T17:17:56.507 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8599999999999998e-07 2024-04-20T17:17:56.507 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.507 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.507 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:17:56.507 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897385+0000", 2024-04-20T17:17:56.507 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.041e-06 2024-04-20T17:17:56.507 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.507 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.507 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:17:56.507 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897393+0000", 2024-04-20T17:17:56.507 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3730000000000004e-06 2024-04-20T17:17:56.507 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.507 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.507 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:17:56.508 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897396+0000", 2024-04-20T17:17:56.508 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2449999999999998e-06 2024-04-20T17:17:56.508 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.508 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.508 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.508 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898572+0000", 2024-04-20T17:17:56.508 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001176329 2024-04-20T17:17:56.508 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.508 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.508 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:17:56.508 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898581+0000", 2024-04-20T17:17:56.508 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6989999999999992e-06 2024-04-20T17:17:56.508 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.508 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.508 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.509 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.566135+0000", 2024-04-20T17:17:56.509 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.66755413200000002 2024-04-20T17:17:56.509 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.509 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.509 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:17:56.509 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.566275+0000", 2024-04-20T17:17:56.509 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014009600000000001 2024-04-20T17:17:56.509 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.509 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.509 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:17:56.509 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.566419+0000", 2024-04-20T17:17:56.509 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014339600000000001 2024-04-20T17:17:56.509 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.509 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.509 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:17:56.509 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576374+0000", 2024-04-20T17:17:56.510 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0099556760000000001 2024-04-20T17:17:56.510 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.510 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.510 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:17:56.510 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576383+0000", 2024-04-20T17:17:56.510 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0920000000000004e-06 2024-04-20T17:17:56.510 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.510 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.510 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:17:56.510 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576412+0000", 2024-04-20T17:17:56.510 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8642000000000001e-05 2024-04-20T17:17:56.510 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.510 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.510 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:17:56.511 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576451+0000", 2024-04-20T17:17:56.511 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8908000000000001e-05 2024-04-20T17:17:56.511 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.511 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:17:56.511 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.511 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.511 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.511 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212458 2.e 2:7672b154:::benchmark_data_smithi171_78918_object51464:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:17:56.511 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897400+0000", 2024-04-20T17:17:56.511 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 332.90703317700002, 2024-04-20T17:17:56.511 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.68587742500000004, 2024-04-20T17:17:56.511 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:17:56.511 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:17:56.511 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:56.511 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:17:56.512 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:17:56.512 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:17:56.512 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212458 2024-04-20T17:17:56.512 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.512 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:17:56.512 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.512 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:17:56.512 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897400+0000", 2024-04-20T17:17:56.512 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.512 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.512 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.512 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:17:56.512 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897400+0000", 2024-04-20T17:17:56.512 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.512 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.512 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.513 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:17:56.513 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897401+0000", 2024-04-20T17:17:56.513 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3000000000000001e-07 2024-04-20T17:17:56.513 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.513 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.513 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:17:56.513 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897403+0000", 2024-04-20T17:17:56.513 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.198e-06 2024-04-20T17:17:56.513 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.513 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.513 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:17:56.513 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897408+0000", 2024-04-20T17:17:56.513 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1109999999999999e-06 2024-04-20T17:17:56.513 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.513 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.514 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:17:56.514 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897414+0000", 2024-04-20T17:17:56.514 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7139999999999998e-06 2024-04-20T17:17:56.514 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.514 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.514 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.514 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898608+0000", 2024-04-20T17:17:56.514 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0011941160000000001 2024-04-20T17:17:56.514 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.514 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.514 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:17:56.514 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898614+0000", 2024-04-20T17:17:56.514 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7100000000000001e-06 2024-04-20T17:17:56.514 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.515 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.515 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.515 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.569428+0000", 2024-04-20T17:17:56.515 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.67081384399999999 2024-04-20T17:17:56.515 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.515 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.515 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:17:56.515 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.569562+0000", 2024-04-20T17:17:56.515 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000134055 2024-04-20T17:17:56.515 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.515 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.515 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:17:56.515 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.569692+0000", 2024-04-20T17:17:56.515 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012998099999999999 2024-04-20T17:17:56.516 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.516 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.516 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:17:56.516 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576458+0000", 2024-04-20T17:17:56.516 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0067652670000000002 2024-04-20T17:17:56.516 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.516 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.516 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:17:56.516 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.583158+0000", 2024-04-20T17:17:56.516 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0067007229999999996 2024-04-20T17:17:56.516 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.516 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.516 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:17:56.516 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.583225+0000", 2024-04-20T17:17:56.516 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6617000000000003e-05 2024-04-20T17:17:56.516 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.517 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.517 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:17:56.517 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.583277+0000", 2024-04-20T17:17:56.517 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2559000000000003e-05 2024-04-20T17:17:56.517 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.517 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:17:56.517 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.517 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.517 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.517 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212462 2.e 2:7f179a5c:::benchmark_data_smithi171_78918_object51468:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:17:56.517 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897424+0000", 2024-04-20T17:17:56.517 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 332.90700884099999, 2024-04-20T17:17:56.517 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.69171569799999999, 2024-04-20T17:17:56.517 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:17:56.517 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:17:56.517 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:56.518 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:17:56.518 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:17:56.518 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:17:56.518 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212462 2024-04-20T17:17:56.518 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.518 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:17:56.518 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.518 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:17:56.518 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897424+0000", 2024-04-20T17:17:56.518 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.518 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.518 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.518 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:17:56.518 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897424+0000", 2024-04-20T17:17:56.519 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.519 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.519 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.519 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:17:56.519 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897425+0000", 2024-04-20T17:17:56.519 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6199999999999997e-07 2024-04-20T17:17:56.519 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.519 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.519 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:17:56.519 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897428+0000", 2024-04-20T17:17:56.519 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1219999999999999e-06 2024-04-20T17:17:56.519 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.519 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.519 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:17:56.519 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897436+0000", 2024-04-20T17:17:56.520 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3270000000000001e-06 2024-04-20T17:17:56.520 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.520 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.520 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:17:56.520 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897442+0000", 2024-04-20T17:17:56.520 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6200000000000004e-06 2024-04-20T17:17:56.520 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.520 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.520 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.520 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898653+0000", 2024-04-20T17:17:56.520 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0012105550000000001 2024-04-20T17:17:56.520 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.520 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.520 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:17:56.520 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898660+0000", 2024-04-20T17:17:56.521 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1810000000000004e-06 2024-04-20T17:17:56.521 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.521 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.521 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.521 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.573380+0000", 2024-04-20T17:17:56.521 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.67471982699999999 2024-04-20T17:17:56.521 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.521 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.521 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:17:56.521 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.573481+0000", 2024-04-20T17:17:56.521 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010135499999999999 2024-04-20T17:17:56.521 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.521 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.521 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:17:56.521 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.573564+0000", 2024-04-20T17:17:56.522 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3258999999999997e-05 2024-04-20T17:17:56.522 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.522 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.522 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:17:56.522 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576463+0000", 2024-04-20T17:17:56.522 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0028989609999999998 2024-04-20T17:17:56.522 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.522 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.522 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:17:56.522 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.589024+0000", 2024-04-20T17:17:56.522 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012561249999999999 2024-04-20T17:17:56.522 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.522 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.522 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:17:56.522 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.589088+0000", 2024-04-20T17:17:56.523 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3398000000000005e-05 2024-04-20T17:17:56.523 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.523 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.523 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:17:56.523 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.589140+0000", 2024-04-20T17:17:56.523 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2181000000000001e-05 2024-04-20T17:17:56.523 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.523 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:17:56.523 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.523 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.523 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.523 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212469 2.e 2:7df2bad1:::benchmark_data_smithi171_78918_object51475:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:17:56.524 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897447+0000", 2024-04-20T17:17:56.524 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 332.906986647, 2024-04-20T17:17:56.524 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.71189628299999996, 2024-04-20T17:17:56.524 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:17:56.524 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:17:56.524 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:56.524 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:17:56.524 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:17:56.524 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:17:56.524 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212469 2024-04-20T17:17:56.524 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.524 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:17:56.524 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.524 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:17:56.524 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897447+0000", 2024-04-20T17:17:56.525 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.525 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.525 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.525 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:17:56.525 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897447+0000", 2024-04-20T17:17:56.525 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.525 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.525 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.525 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:17:56.525 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897447+0000", 2024-04-20T17:17:56.525 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.89e-07 2024-04-20T17:17:56.525 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.525 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.525 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:17:56.525 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897449+0000", 2024-04-20T17:17:56.526 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3530000000000002e-06 2024-04-20T17:17:56.526 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.526 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.526 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:17:56.526 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897455+0000", 2024-04-20T17:17:56.526 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2850000000000002e-06 2024-04-20T17:17:56.526 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.526 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.526 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:17:56.526 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897459+0000", 2024-04-20T17:17:56.526 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7129999999999996e-06 2024-04-20T17:17:56.526 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.526 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.526 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.526 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898683+0000", 2024-04-20T17:17:56.527 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0012234629999999999 2024-04-20T17:17:56.527 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.527 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.527 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:17:56.527 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898690+0000", 2024-04-20T17:17:56.527 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9419999999999998e-06 2024-04-20T17:17:56.527 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.527 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.527 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.527 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576505+0000", 2024-04-20T17:17:56.527 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.677814888 2024-04-20T17:17:56.527 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.527 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.527 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:17:56.528 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576644+0000", 2024-04-20T17:17:56.528 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013927000000000001 2024-04-20T17:17:56.528 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.528 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.528 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:17:56.528 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576783+0000", 2024-04-20T17:17:56.528 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013961799999999999 2024-04-20T17:17:56.528 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.528 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.528 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:17:56.528 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.586310+0000", 2024-04-20T17:17:56.528 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0095266459999999997 2024-04-20T17:17:56.528 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.528 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.528 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:17:56.528 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609211+0000", 2024-04-20T17:17:56.529 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.022901248999999999 2024-04-20T17:17:56.529 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.529 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.529 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:17:56.529 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609274+0000", 2024-04-20T17:17:56.529 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3045000000000004e-05 2024-04-20T17:17:56.529 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.529 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.529 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:17:56.529 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609343+0000", 2024-04-20T17:17:56.529 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8422000000000003e-05 2024-04-20T17:17:56.529 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.529 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:17:56.529 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.529 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.530 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.530 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212495 2.e 2:782e28d2:::benchmark_data_smithi171_78918_object51501:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:17:56.530 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897465+0000", 2024-04-20T17:17:56.530 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 332.90696839600002, 2024-04-20T17:17:56.530 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.71202960599999998, 2024-04-20T17:17:56.530 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:17:56.530 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:17:56.530 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:56.530 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:17:56.530 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:17:56.530 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:17:56.530 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212495 2024-04-20T17:17:56.530 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.531 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:17:56.531 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.531 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:17:56.531 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897465+0000", 2024-04-20T17:17:56.531 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.531 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.531 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.531 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:17:56.531 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897465+0000", 2024-04-20T17:17:56.531 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.531 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.531 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.531 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:17:56.531 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897465+0000", 2024-04-20T17:17:56.532 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.68e-07 2024-04-20T17:17:56.532 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.532 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.532 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:17:56.532 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897468+0000", 2024-04-20T17:17:56.532 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7800000000000001e-06 2024-04-20T17:17:56.532 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.532 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.532 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:17:56.532 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897475+0000", 2024-04-20T17:17:56.532 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1069999999999999e-06 2024-04-20T17:17:56.532 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.532 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.533 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:17:56.533 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897481+0000", 2024-04-20T17:17:56.533 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3539999999999999e-06 2024-04-20T17:17:56.533 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.533 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.533 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.533 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898736+0000", 2024-04-20T17:17:56.533 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001255812 2024-04-20T17:17:56.533 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.533 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.533 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:17:56.533 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898743+0000", 2024-04-20T17:17:56.533 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6830000000000002e-06 2024-04-20T17:17:56.533 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.534 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.534 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.534 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.580475+0000", 2024-04-20T17:17:56.534 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.68173151600000004 2024-04-20T17:17:56.534 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.534 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.534 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:17:56.534 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.580557+0000", 2024-04-20T17:17:56.534 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2826999999999998e-05 2024-04-20T17:17:56.534 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.534 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.534 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:17:56.534 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.580649+0000", 2024-04-20T17:17:56.534 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1611000000000001e-05 2024-04-20T17:17:56.534 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.535 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.535 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:17:56.535 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.589156+0000", 2024-04-20T17:17:56.535 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0085067349999999996 2024-04-20T17:17:56.535 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.535 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.535 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:17:56.535 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609403+0000", 2024-04-20T17:17:56.535 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.020247660000000001 2024-04-20T17:17:56.535 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.535 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.535 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:17:56.535 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609449+0000", 2024-04-20T17:17:56.535 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5649999999999998e-05 2024-04-20T17:17:56.535 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.536 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.536 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:17:56.536 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609494+0000", 2024-04-20T17:17:56.536 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5303000000000003e-05 2024-04-20T17:17:56.536 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.536 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:17:56.536 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.536 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.536 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.536 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212512 2.e 2:7c5add18:::benchmark_data_smithi171_78918_object51518:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:17:56.536 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897487+0000", 2024-04-20T17:17:56.536 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 332.90694645999997, 2024-04-20T17:17:56.536 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.71708918399999999, 2024-04-20T17:17:56.536 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:17:56.536 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:17:56.536 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:56.537 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:17:56.537 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:17:56.537 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:17:56.537 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212512 2024-04-20T17:17:56.537 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.537 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:17:56.537 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.537 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:17:56.537 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897487+0000", 2024-04-20T17:17:56.537 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.537 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.537 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.537 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:17:56.537 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897487+0000", 2024-04-20T17:17:56.538 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.538 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.538 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.538 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:17:56.538 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897487+0000", 2024-04-20T17:17:56.538 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9399999999999995e-07 2024-04-20T17:17:56.538 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.538 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.538 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:17:56.538 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897498+0000", 2024-04-20T17:17:56.538 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0565e-05 2024-04-20T17:17:56.538 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.538 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.538 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:17:56.539 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897503+0000", 2024-04-20T17:17:56.539 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6590000000000004e-06 2024-04-20T17:17:56.539 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.539 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.539 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:17:56.539 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897507+0000", 2024-04-20T17:17:56.539 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.805e-06 2024-04-20T17:17:56.539 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.539 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.539 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.539 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898819+0000", 2024-04-20T17:17:56.539 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00131193 2024-04-20T17:17:56.539 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.539 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.539 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:17:56.540 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898826+0000", 2024-04-20T17:17:56.540 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6760000000000001e-06 2024-04-20T17:17:56.540 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.540 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.540 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.540 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.583338+0000", 2024-04-20T17:17:56.540 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.68451254699999997 2024-04-20T17:17:56.540 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.540 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.540 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:17:56.540 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.583467+0000", 2024-04-20T17:17:56.540 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000129016 2024-04-20T17:17:56.540 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.540 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.540 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:17:56.540 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.583627+0000", 2024-04-20T17:17:56.541 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000159658 2024-04-20T17:17:56.541 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.541 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.541 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:17:56.541 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.591594+0000", 2024-04-20T17:17:56.541 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0079671770000000006 2024-04-20T17:17:56.541 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.541 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.541 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:17:56.541 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.614474+0000", 2024-04-20T17:17:56.541 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.022880211000000001 2024-04-20T17:17:56.541 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.541 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.541 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:17:56.541 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.614528+0000", 2024-04-20T17:17:56.542 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3721000000000003e-05 2024-04-20T17:17:56.542 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.542 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.542 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:17:56.542 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.614576+0000", 2024-04-20T17:17:56.542 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7725000000000001e-05 2024-04-20T17:17:56.542 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.542 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:17:56.542 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.542 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.542 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.542 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212527 2.e 2:73e27e1e:::benchmark_data_smithi171_78918_object51533:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:17:56.542 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897511+0000", 2024-04-20T17:17:56.542 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 332.906922038, 2024-04-20T17:17:56.542 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.71722206099999997, 2024-04-20T17:17:56.543 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:17:56.543 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:17:56.543 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:56.543 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:17:56.543 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:17:56.543 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:17:56.543 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212527 2024-04-20T17:17:56.543 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.543 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:17:56.543 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.543 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:17:56.543 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897511+0000", 2024-04-20T17:17:56.543 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.543 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.543 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.543 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:17:56.544 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897511+0000", 2024-04-20T17:17:56.544 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.544 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.544 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.544 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:17:56.544 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897512+0000", 2024-04-20T17:17:56.544 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3999999999999997e-07 2024-04-20T17:17:56.544 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.544 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.544 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:17:56.544 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897513+0000", 2024-04-20T17:17:56.544 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.302e-06 2024-04-20T17:17:56.545 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.545 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.545 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:17:56.545 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897516+0000", 2024-04-20T17:17:56.545 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4060000000000001e-06 2024-04-20T17:17:56.545 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.545 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.545 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:17:56.545 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897519+0000", 2024-04-20T17:17:56.545 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6840000000000001e-06 2024-04-20T17:17:56.545 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.545 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.545 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.545 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898869+0000", 2024-04-20T17:17:56.545 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0013499180000000001 2024-04-20T17:17:56.546 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.546 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.546 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:17:56.546 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898876+0000", 2024-04-20T17:17:56.546 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4490000000000002e-06 2024-04-20T17:17:56.546 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.546 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.546 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.546 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.586356+0000", 2024-04-20T17:17:56.546 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.68747954499999997 2024-04-20T17:17:56.546 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.546 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.546 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:17:56.546 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.586502+0000", 2024-04-20T17:17:56.547 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014652999999999999 2024-04-20T17:17:56.547 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.547 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.547 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:17:56.547 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.586643+0000", 2024-04-20T17:17:56.547 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014104100000000001 2024-04-20T17:17:56.547 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.547 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.547 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:17:56.547 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.591604+0000", 2024-04-20T17:17:56.547 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0049606090000000004 2024-04-20T17:17:56.547 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.547 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.547 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:17:56.547 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.614639+0000", 2024-04-20T17:17:56.548 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023034974 2024-04-20T17:17:56.548 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.548 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.548 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:17:56.548 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.614690+0000", 2024-04-20T17:17:56.548 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1134999999999997e-05 2024-04-20T17:17:56.548 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.548 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.548 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:17:56.548 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.614733+0000", 2024-04-20T17:17:56.548 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3127999999999998e-05 2024-04-20T17:17:56.548 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.548 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:17:56.548 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.548 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.549 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.549 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212531 2.e 2:7eb9e3fb:::benchmark_data_smithi171_78918_object51537:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:17:56.549 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897525+0000", 2024-04-20T17:17:56.549 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 332.90690825899998, 2024-04-20T17:17:56.549 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.71744083700000005, 2024-04-20T17:17:56.549 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:17:56.549 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:17:56.549 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:56.549 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:17:56.549 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:17:56.549 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:17:56.549 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212531 2024-04-20T17:17:56.549 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.549 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:17:56.549 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.550 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:17:56.550 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897525+0000", 2024-04-20T17:17:56.550 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.550 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.550 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.550 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:17:56.550 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897525+0000", 2024-04-20T17:17:56.550 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.550 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.550 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.550 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:17:56.550 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897525+0000", 2024-04-20T17:17:56.550 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5600000000000001e-07 2024-04-20T17:17:56.551 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.551 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.551 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:17:56.551 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897527+0000", 2024-04-20T17:17:56.551 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9410000000000001e-06 2024-04-20T17:17:56.551 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.551 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.551 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:17:56.551 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897534+0000", 2024-04-20T17:17:56.551 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1199999999999996e-06 2024-04-20T17:17:56.551 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.551 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.551 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:17:56.551 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897539+0000", 2024-04-20T17:17:56.551 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.921e-06 2024-04-20T17:17:56.551 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.552 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.552 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.552 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898909+0000", 2024-04-20T17:17:56.552 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001369706 2024-04-20T17:17:56.552 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.552 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.552 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:17:56.552 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898915+0000", 2024-04-20T17:17:56.552 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4160000000000001e-06 2024-04-20T17:17:56.552 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.552 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.552 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.552 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.589187+0000", 2024-04-20T17:17:56.552 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.69027111600000002 2024-04-20T17:17:56.552 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.553 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.553 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:17:56.553 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.589267+0000", 2024-04-20T17:17:56.553 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0399000000000001e-05 2024-04-20T17:17:56.553 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.553 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.553 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:17:56.553 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.589349+0000", 2024-04-20T17:17:56.553 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1904999999999996e-05 2024-04-20T17:17:56.553 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.553 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.553 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:17:56.553 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.600413+0000", 2024-04-20T17:17:56.553 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.01106378 2024-04-20T17:17:56.553 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.553 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.553 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:17:56.554 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.614882+0000", 2024-04-20T17:17:56.554 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014469556999999999 2024-04-20T17:17:56.554 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.554 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.554 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:17:56.554 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.614926+0000", 2024-04-20T17:17:56.554 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3649000000000002e-05 2024-04-20T17:17:56.554 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.554 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.554 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:17:56.554 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.614966+0000", 2024-04-20T17:17:56.554 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9971000000000001e-05 2024-04-20T17:17:56.554 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.554 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:17:56.555 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.555 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.555 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.555 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212541 2.e 2:7a53380b:::benchmark_data_smithi171_78918_object51547:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:17:56.555 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897543+0000", 2024-04-20T17:17:56.555 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 332.90688996199998, 2024-04-20T17:17:56.555 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.720308693, 2024-04-20T17:17:56.555 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:17:56.555 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:17:56.555 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:56.555 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:17:56.555 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:17:56.555 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:17:56.555 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212541 2024-04-20T17:17:56.555 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.556 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:17:56.556 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.556 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:17:56.556 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897543+0000", 2024-04-20T17:17:56.556 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.556 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.556 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.556 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:17:56.556 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897543+0000", 2024-04-20T17:17:56.556 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.556 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.556 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.556 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:17:56.556 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897544+0000", 2024-04-20T17:17:56.556 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5100000000000001e-07 2024-04-20T17:17:56.557 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.557 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.557 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:17:56.557 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897545+0000", 2024-04-20T17:17:56.557 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.412e-06 2024-04-20T17:17:56.557 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.557 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.557 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:17:56.557 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897548+0000", 2024-04-20T17:17:56.557 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.858e-06 2024-04-20T17:17:56.557 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.557 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.557 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:17:56.557 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897551+0000", 2024-04-20T17:17:56.557 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0189999999999998e-06 2024-04-20T17:17:56.558 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.558 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.558 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.558 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898951+0000", 2024-04-20T17:17:56.558 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0013997440000000001 2024-04-20T17:17:56.558 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.558 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.558 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:17:56.558 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898958+0000", 2024-04-20T17:17:56.558 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4580000000000004e-06 2024-04-20T17:17:56.558 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.558 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.558 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.558 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.591641+0000", 2024-04-20T17:17:56.559 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.69268292399999998 2024-04-20T17:17:56.559 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.559 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.559 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:17:56.559 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.591724+0000", 2024-04-20T17:17:56.559 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3249000000000002e-05 2024-04-20T17:17:56.559 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.559 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.559 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:17:56.559 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.591816+0000", 2024-04-20T17:17:56.559 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1774000000000006e-05 2024-04-20T17:17:56.559 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.559 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.559 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:17:56.559 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.600419+0000", 2024-04-20T17:17:56.560 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0086027299999999994 2024-04-20T17:17:56.560 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.560 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.560 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:17:56.560 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.617785+0000", 2024-04-20T17:17:56.560 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.017365914999999999 2024-04-20T17:17:56.560 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.560 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.560 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:17:56.560 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.617819+0000", 2024-04-20T17:17:56.560 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4743000000000003e-05 2024-04-20T17:17:56.560 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.560 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.560 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:17:56.560 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.617852+0000", 2024-04-20T17:17:56.560 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2515999999999998e-05 2024-04-20T17:17:56.561 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.561 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:17:56.561 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.561 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.561 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.561 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212573 2.e 2:73f6f3d3:::benchmark_data_smithi171_78918_object51579:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:17:56.561 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897554+0000", 2024-04-20T17:17:56.561 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 332.90687960899999, 2024-04-20T17:17:56.561 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.72045646900000004, 2024-04-20T17:17:56.561 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:17:56.561 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:17:56.561 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:56.561 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:17:56.561 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:17:56.561 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:17:56.562 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212573 2024-04-20T17:17:56.562 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.562 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:17:56.562 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.562 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:17:56.562 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897554+0000", 2024-04-20T17:17:56.562 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.562 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.562 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.562 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:17:56.562 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897554+0000", 2024-04-20T17:17:56.562 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.562 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.562 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.563 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:17:56.563 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897554+0000", 2024-04-20T17:17:56.563 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8900000000000001e-07 2024-04-20T17:17:56.563 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.563 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.563 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:17:56.563 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897555+0000", 2024-04-20T17:17:56.563 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.063e-06 2024-04-20T17:17:56.563 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.563 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.563 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:17:56.563 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897558+0000", 2024-04-20T17:17:56.563 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3400000000000002e-06 2024-04-20T17:17:56.563 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.564 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.564 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:17:56.564 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897566+0000", 2024-04-20T17:17:56.564 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6629999999999993e-06 2024-04-20T17:17:56.564 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.564 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.564 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.564 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898998+0000", 2024-04-20T17:17:56.564 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001432326 2024-04-20T17:17:56.564 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.564 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.564 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:17:56.564 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.899005+0000", 2024-04-20T17:17:56.564 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.737e-06 2024-04-20T17:17:56.565 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.565 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.565 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.565 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.594204+0000", 2024-04-20T17:17:56.565 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.69519878599999996 2024-04-20T17:17:56.565 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.565 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.565 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:17:56.565 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.594324+0000", 2024-04-20T17:17:56.565 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000119827 2024-04-20T17:17:56.565 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.565 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.565 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:17:56.565 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.594406+0000", 2024-04-20T17:17:56.565 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2727999999999998e-05 2024-04-20T17:17:56.566 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.566 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.566 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:17:56.566 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.605467+0000", 2024-04-20T17:17:56.566 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011060449999999999 2024-04-20T17:17:56.566 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.566 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.566 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:17:56.566 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.617903+0000", 2024-04-20T17:17:56.566 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012436361 2024-04-20T17:17:56.566 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.566 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.566 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:17:56.566 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.617964+0000", 2024-04-20T17:17:56.566 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0569999999999998e-05 2024-04-20T17:17:56.567 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.567 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.567 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:17:56.567 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.618010+0000", 2024-04-20T17:17:56.567 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6329000000000003e-05 2024-04-20T17:17:56.567 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.567 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:17:56.567 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.567 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.567 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.567 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212593 2.e 2:744300f2:::benchmark_data_smithi171_78918_object51599:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:17:56.567 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897569+0000", 2024-04-20T17:17:56.567 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 332.906863724, 2024-04-20T17:17:56.567 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.72464595399999998, 2024-04-20T17:17:56.567 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:17:56.567 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:17:56.568 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:56.568 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:17:56.568 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:17:56.568 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:17:56.568 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212593 2024-04-20T17:17:56.568 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.568 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:17:56.568 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.568 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:17:56.568 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897569+0000", 2024-04-20T17:17:56.568 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.568 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.568 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.569 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:17:56.569 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897569+0000", 2024-04-20T17:17:56.569 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.569 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:17:56.569 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897570+0000", 2024-04-20T17:17:56.569 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3999999999999997e-07 2024-04-20T17:17:56.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.569 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:17:56.569 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897571+0000", 2024-04-20T17:17:56.569 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.296e-06 2024-04-20T17:17:56.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.570 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:17:56.570 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897574+0000", 2024-04-20T17:17:56.570 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7970000000000001e-06 2024-04-20T17:17:56.570 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.570 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.570 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:17:56.570 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897577+0000", 2024-04-20T17:17:56.570 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8389999999999998e-06 2024-04-20T17:17:56.570 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.570 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.570 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.570 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.899045+0000", 2024-04-20T17:17:56.570 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0014678639999999999 2024-04-20T17:17:56.570 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.571 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.571 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:17:56.571 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.899052+0000", 2024-04-20T17:17:56.571 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3010000000000001e-06 2024-04-20T17:17:56.571 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.571 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.571 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.571 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.597702+0000", 2024-04-20T17:17:56.571 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.69865043599999999 2024-04-20T17:17:56.571 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.571 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.571 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:17:56.571 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.597847+0000", 2024-04-20T17:17:56.571 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000144291 2024-04-20T17:17:56.572 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.572 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:17:56.572 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.598003+0000", 2024-04-20T17:17:56.572 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000156615 2024-04-20T17:17:56.572 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.572 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:17:56.572 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609358+0000", 2024-04-20T17:17:56.572 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011354991 2024-04-20T17:17:56.572 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.572 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:17:56.572 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.622109+0000", 2024-04-20T17:17:56.572 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012751035000000001 2024-04-20T17:17:56.573 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.573 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.573 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:17:56.573 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.622164+0000", 2024-04-20T17:17:56.573 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4867000000000002e-05 2024-04-20T17:17:56.573 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.573 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.573 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:17:56.573 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.622215+0000", 2024-04-20T17:17:56.573 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1282000000000001e-05 2024-04-20T17:17:56.573 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.573 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:17:56.573 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.573 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.574 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.574 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212638 2.e 2:7e5004ea:::benchmark_data_smithi171_78918_object51644:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:17:56.574 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897579+0000", 2024-04-20T17:17:56.574 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 332.90685376699997, 2024-04-20T17:17:56.574 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.73145174000000002, 2024-04-20T17:17:56.574 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:17:56.574 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:17:56.574 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:56.574 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:17:56.574 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:17:56.574 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:17:56.574 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212638 2024-04-20T17:17:56.574 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.575 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:17:56.575 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.575 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:17:56.575 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897579+0000", 2024-04-20T17:17:56.575 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.575 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.575 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.575 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:17:56.575 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897579+0000", 2024-04-20T17:17:56.575 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.575 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.575 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.575 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:17:56.575 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897580+0000", 2024-04-20T17:17:56.575 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9900000000000002e-07 2024-04-20T17:17:56.576 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.576 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.576 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:17:56.576 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897581+0000", 2024-04-20T17:17:56.576 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0380000000000001e-06 2024-04-20T17:17:56.576 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.576 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.576 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:17:56.576 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897585+0000", 2024-04-20T17:17:56.576 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7479999999999999e-06 2024-04-20T17:17:56.576 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.576 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.576 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:17:56.576 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897594+0000", 2024-04-20T17:17:56.577 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7899999999999994e-06 2024-04-20T17:17:56.577 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.577 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.577 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.577 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.899085+0000", 2024-04-20T17:17:56.577 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0014909649999999999 2024-04-20T17:17:56.577 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.577 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.577 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:17:56.577 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.899091+0000", 2024-04-20T17:17:56.577 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8170000000000004e-06 2024-04-20T17:17:56.577 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.577 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.577 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.577 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.600462+0000", 2024-04-20T17:17:56.578 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.70137081800000001 2024-04-20T17:17:56.578 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.578 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.578 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:17:56.578 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.600548+0000", 2024-04-20T17:17:56.578 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5815000000000004e-05 2024-04-20T17:17:56.578 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.578 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.578 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:17:56.578 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.600638+0000", 2024-04-20T17:17:56.578 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9998000000000001e-05 2024-04-20T17:17:56.578 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.578 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.579 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:17:56.579 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609366+0000", 2024-04-20T17:17:56.579 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0087284849999999994 2024-04-20T17:17:56.579 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.579 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.579 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:17:56.579 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.628925+0000", 2024-04-20T17:17:56.579 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.019558467999999999 2024-04-20T17:17:56.579 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.579 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.579 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:17:56.579 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.628978+0000", 2024-04-20T17:17:56.579 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.329e-05 2024-04-20T17:17:56.579 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.580 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.580 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:17:56.580 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.629031+0000", 2024-04-20T17:17:56.580 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3208999999999998e-05 2024-04-20T17:17:56.580 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.580 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:17:56.580 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.580 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.580 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.580 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212664 2.e 2:78efb1e3:::benchmark_data_smithi171_78918_object51670:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:17:56.580 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897598+0000", 2024-04-20T17:17:56.580 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 332.90683530299998, 2024-04-20T17:17:56.580 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.73297929500000003, 2024-04-20T17:17:56.580 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:17:56.580 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:17:56.581 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:56.581 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:17:56.581 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:17:56.581 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:17:56.581 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212664 2024-04-20T17:17:56.581 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.581 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:17:56.581 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.581 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:17:56.581 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897598+0000", 2024-04-20T17:17:56.581 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.581 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.581 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.581 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:17:56.581 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897598+0000", 2024-04-20T17:17:56.582 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.582 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.582 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.582 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:17:56.582 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897598+0000", 2024-04-20T17:17:56.582 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6300000000000001e-07 2024-04-20T17:17:56.582 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.582 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.582 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:17:56.582 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897600+0000", 2024-04-20T17:17:56.582 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.451e-06 2024-04-20T17:17:56.582 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.582 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.582 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:17:56.582 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897603+0000", 2024-04-20T17:17:56.583 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9950000000000001e-06 2024-04-20T17:17:56.583 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.583 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.583 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:17:56.583 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897605+0000", 2024-04-20T17:17:56.583 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6400000000000001e-06 2024-04-20T17:17:56.583 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.583 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.583 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.583 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.899129+0000", 2024-04-20T17:17:56.583 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001523567 2024-04-20T17:17:56.583 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.583 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.583 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:17:56.583 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.899136+0000", 2024-04-20T17:17:56.584 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4880000000000001e-06 2024-04-20T17:17:56.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.584 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.584 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.602930+0000", 2024-04-20T17:17:56.584 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.703793365 2024-04-20T17:17:56.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.584 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:17:56.584 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.603007+0000", 2024-04-20T17:17:56.584 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7519000000000001e-05 2024-04-20T17:17:56.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.584 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:17:56.584 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.603094+0000", 2024-04-20T17:17:56.584 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6271000000000001e-05 2024-04-20T17:17:56.585 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.585 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.585 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:17:56.585 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.613185+0000", 2024-04-20T17:17:56.585 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010091126000000001 2024-04-20T17:17:56.585 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.585 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.585 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:17:56.585 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.630457+0000", 2024-04-20T17:17:56.585 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.017272526 2024-04-20T17:17:56.585 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.585 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.585 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:17:56.585 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.630522+0000", 2024-04-20T17:17:56.585 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4422000000000001e-05 2024-04-20T17:17:56.586 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.586 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.586 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:17:56.586 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.630577+0000", 2024-04-20T17:17:56.586 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5562000000000001e-05 2024-04-20T17:17:56.586 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.586 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:17:56.586 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.586 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.586 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.586 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212665 2.e 2:734f527b:::benchmark_data_smithi171_78918_object51671:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:17:56.586 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897609+0000", 2024-04-20T17:17:56.586 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 332.90682389900002, 2024-04-20T17:17:56.586 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.73311006000000001, 2024-04-20T17:17:56.587 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:17:56.587 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:17:56.587 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:56.587 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:17:56.587 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:17:56.587 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:17:56.587 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212665 2024-04-20T17:17:56.587 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.587 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:17:56.587 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.587 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:17:56.587 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897609+0000", 2024-04-20T17:17:56.587 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.587 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.587 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.587 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:17:56.589 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897609+0000", 2024-04-20T17:17:56.589 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.589 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.590 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.590 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:17:56.590 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897610+0000", 2024-04-20T17:17:56.590 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4400000000000001e-07 2024-04-20T17:17:56.590 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.590 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.590 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:17:56.590 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897612+0000", 2024-04-20T17:17:56.590 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.824e-06 2024-04-20T17:17:56.590 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.590 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.590 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:17:56.590 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897618+0000", 2024-04-20T17:17:56.590 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4330000000000001e-06 2024-04-20T17:17:56.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.591 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.591 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:17:56.591 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897621+0000", 2024-04-20T17:17:56.591 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9359999999999999e-06 2024-04-20T17:17:56.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.591 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.591 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.591 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.899161+0000", 2024-04-20T17:17:56.591 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001540198 2024-04-20T17:17:56.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.591 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.591 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:17:56.591 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.899169+0000", 2024-04-20T17:17:56.591 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4989999999999997e-06 2024-04-20T17:17:56.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.592 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.592 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.592 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.605497+0000", 2024-04-20T17:17:56.592 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.70632853799999995 2024-04-20T17:17:56.592 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.592 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.592 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:17:56.592 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.605602+0000", 2024-04-20T17:17:56.592 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000105147 2024-04-20T17:17:56.592 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.592 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.592 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:17:56.592 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.605676+0000", 2024-04-20T17:17:56.592 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4098999999999996e-05 2024-04-20T17:17:56.592 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.592 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.593 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:17:56.593 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.613193+0000", 2024-04-20T17:17:56.593 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0075168830000000002 2024-04-20T17:17:56.593 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.593 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.593 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:17:56.593 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.630633+0000", 2024-04-20T17:17:56.593 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.017440124000000001 2024-04-20T17:17:56.593 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.593 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.593 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:17:56.593 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.630681+0000", 2024-04-20T17:17:56.593 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7673999999999998e-05 2024-04-20T17:17:56.593 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.593 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.594 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:17:56.594 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.630719+0000", 2024-04-20T17:17:56.594 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8260999999999998e-05 2024-04-20T17:17:56.594 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.594 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:17:56.594 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.594 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.594 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.594 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212685 2.e 2:7acb7d43:::benchmark_data_smithi171_78918_object51691:head [delete] snapc 0=[] ondisk+write+known_if_redirected e203)", 2024-04-20T17:17:56.594 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:23.604266+0000", 2024-04-20T17:17:56.594 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 332.20016726699998, 2024-04-20T17:17:56.594 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.026602213, 2024-04-20T17:17:56.594 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:17:56.594 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:17:56.595 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:56.595 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:17:56.595 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:17:56.595 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:17:56.595 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212685 2024-04-20T17:17:56.595 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.595 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:17:56.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.595 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:17:56.595 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.604266+0000", 2024-04-20T17:17:56.595 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.595 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.595 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:17:56.595 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.604266+0000", 2024-04-20T17:17:56.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.596 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:17:56.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.604269+0000", 2024-04-20T17:17:56.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1379999999999999e-06 2024-04-20T17:17:56.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.596 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:17:56.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.604277+0000", 2024-04-20T17:17:56.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4209999999999995e-06 2024-04-20T17:17:56.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.596 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:17:56.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.604298+0000", 2024-04-20T17:17:56.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0041999999999999e-05 2024-04-20T17:17:56.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.597 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:17:56.597 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.604308+0000", 2024-04-20T17:17:56.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0392e-05 2024-04-20T17:17:56.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.597 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.597 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609525+0000", 2024-04-20T17:17:56.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0052173369999999998 2024-04-20T17:17:56.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.597 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:17:56.597 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609662+0000", 2024-04-20T17:17:56.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013646400000000001 2024-04-20T17:17:56.598 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.598 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.598 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:17:56.598 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609758+0000", 2024-04-20T17:17:56.598 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6510999999999998e-05 2024-04-20T17:17:56.598 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.598 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.598 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:17:56.598 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.613348+0000", 2024-04-20T17:17:56.598 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0035896919999999998 2024-04-20T17:17:56.598 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.598 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.598 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:17:56.598 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.630768+0000", 2024-04-20T17:17:56.599 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.017419752 2024-04-20T17:17:56.599 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.599 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.599 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:17:56.599 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.630825+0000", 2024-04-20T17:17:56.599 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6937000000000001e-05 2024-04-20T17:17:56.599 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.599 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.599 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:17:56.599 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.630868+0000", 2024-04-20T17:17:56.599 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3526999999999998e-05 2024-04-20T17:17:56.599 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.599 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:17:56.599 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.599 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.599 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.600 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:224633 2.e 2:74b55b25:::benchmark_data_smithi171_78918_object63639:head [delete] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-20T17:17:56.600 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:13.474420+0000", 2024-04-20T17:17:56.600 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 282.330013076, 2024-04-20T17:17:56.600 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014516803, 2024-04-20T17:17:56.600 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:17:56.600 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:17:56.600 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:56.600 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:17:56.600 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:17:56.600 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:17:56.600 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 224633 2024-04-20T17:17:56.600 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.600 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:17:56.600 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.600 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:17:56.600 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.474420+0000", 2024-04-20T17:17:56.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:17:56.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.474420+0000", 2024-04-20T17:17:56.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:17:56.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.474421+0000", 2024-04-20T17:17:56.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.142e-06 2024-04-20T17:17:56.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:17:56.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.474426+0000", 2024-04-20T17:17:56.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9189999999999998e-06 2024-04-20T17:17:56.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:17:56.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.474435+0000", 2024-04-20T17:17:56.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6589999999999996e-06 2024-04-20T17:17:56.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:17:56.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.474441+0000", 2024-04-20T17:17:56.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8279999999999999e-06 2024-04-20T17:17:56.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.603 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.476924+0000", 2024-04-20T17:17:56.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0024833659999999999 2024-04-20T17:17:56.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.603 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:17:56.603 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.477060+0000", 2024-04-20T17:17:56.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013633300000000001 2024-04-20T17:17:56.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.603 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:17:56.603 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.477195+0000", 2024-04-20T17:17:56.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013424800000000001 2024-04-20T17:17:56.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.604 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:17:56.604 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.485883+0000", 2024-04-20T17:17:56.604 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0086886370000000008 2024-04-20T17:17:56.604 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.604 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:17:56.604 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.488839+0000", 2024-04-20T17:17:56.604 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0029559109999999999 2024-04-20T17:17:56.604 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.604 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:17:56.604 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.488892+0000", 2024-04-20T17:17:56.604 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2732000000000003e-05 2024-04-20T17:17:56.604 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.605 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:17:56.605 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.488937+0000", 2024-04-20T17:17:56.605 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5028000000000003e-05 2024-04-20T17:17:56.605 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.605 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:17:56.605 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.605 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.605 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.605 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:224636 2.e 2:730710de:::benchmark_data_smithi171_78918_object63642:head [delete] snapc 0=[] ondisk+write+known_if_redirected e221)", 2024-04-20T17:17:56.605 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:13.479254+0000", 2024-04-20T17:17:56.605 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 282.32517934700002, 2024-04-20T17:17:56.605 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014410431, 2024-04-20T17:17:56.605 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:17:56.605 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:17:56.605 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:56.606 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:17:56.606 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:17:56.606 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:17:56.606 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 224636 2024-04-20T17:17:56.606 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.606 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:17:56.606 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.606 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:17:56.606 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.479254+0000", 2024-04-20T17:17:56.606 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.606 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.606 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.606 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:17:56.606 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.479254+0000", 2024-04-20T17:17:56.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.607 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:17:56.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.479256+0000", 2024-04-20T17:17:56.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.108e-06 2024-04-20T17:17:56.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.607 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:17:56.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.479261+0000", 2024-04-20T17:17:56.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3190000000000002e-06 2024-04-20T17:17:56.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.607 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:17:56.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.479272+0000", 2024-04-20T17:17:56.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0793e-05 2024-04-20T17:17:56.608 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.608 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.608 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:17:56.608 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.479293+0000", 2024-04-20T17:17:56.608 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0996e-05 2024-04-20T17:17:56.608 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.608 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.608 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.608 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.482880+0000", 2024-04-20T17:17:56.608 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003587186 2024-04-20T17:17:56.608 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.608 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.608 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:17:56.608 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.483029+0000", 2024-04-20T17:17:56.608 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014848300000000001 2024-04-20T17:17:56.608 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.608 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.609 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:17:56.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.483174+0000", 2024-04-20T17:17:56.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0001451 2024-04-20T17:17:56.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.609 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:17:56.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.485892+0000", 2024-04-20T17:17:56.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0027186369999999999 2024-04-20T17:17:56.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.609 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:17:56.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.493550+0000", 2024-04-20T17:17:56.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0076577839999999999 2024-04-20T17:17:56.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.609 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:17:56.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.493613+0000", 2024-04-20T17:17:56.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2966000000000007e-05 2024-04-20T17:17:56.610 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.610 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.610 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:17:56.610 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:13.493664+0000", 2024-04-20T17:17:56.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1059e-05 2024-04-20T17:17:56.610 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.610 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:17:56.610 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.610 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.610 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.610 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:268397 2.e 2:793bae04:::benchmark_data_smithi171_78918_object107403:head [delete] snapc 0=[] ondisk+write+known_if_redirected e309)", 2024-04-20T17:17:56.610 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:17:55.525797+0000", 2024-04-20T17:17:56.610 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.27863618099999998, 2024-04-20T17:17:56.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.016801780999999998, 2024-04-20T17:17:56.611 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:17:56.611 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:17:56.611 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:17:56.611 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:17:56.611 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:17:56.611 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:17:56.611 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 268397 2024-04-20T17:17:56.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.611 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:17:56.611 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.611 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:17:56.611 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.525797+0000", 2024-04-20T17:17:56.611 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.611 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.611 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:17:56.611 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.525797+0000", 2024-04-20T17:17:56.612 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:17:56.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.612 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:17:56.612 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.525800+0000", 2024-04-20T17:17:56.612 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4769999999999999e-06 2024-04-20T17:17:56.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.612 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:17:56.612 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.525811+0000", 2024-04-20T17:17:56.612 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0243e-05 2024-04-20T17:17:56.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.612 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:17:56.612 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.525836+0000", 2024-04-20T17:17:56.612 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5052999999999999e-05 2024-04-20T17:17:56.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.613 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.613 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:17:56.613 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.525846+0000", 2024-04-20T17:17:56.613 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0328e-05 2024-04-20T17:17:56.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.613 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.613 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:17:56.613 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.525908+0000", 2024-04-20T17:17:56.613 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2057000000000006e-05 2024-04-20T17:17:56.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.613 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.613 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:17:56.613 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.526045+0000", 2024-04-20T17:17:56.613 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000137279 2024-04-20T17:17:56.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.614 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:17:56.614 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.526164+0000", 2024-04-20T17:17:56.614 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011880899999999999 2024-04-20T17:17:56.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.614 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:17:56.614 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.530321+0000", 2024-04-20T17:17:56.614 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041566429999999998 2024-04-20T17:17:56.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.614 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:17:56.614 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.542463+0000", 2024-04-20T17:17:56.614 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012141684999999999 2024-04-20T17:17:56.615 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.615 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:17:56.615 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.542534+0000", 2024-04-20T17:17:56.615 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1855999999999998e-05 2024-04-20T17:17:56.615 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:17:56.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:17:56.615 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:17:56.615 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.542599+0000", 2024-04-20T17:17:56.615 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4350999999999994e-05 2024-04-20T17:17:56.615 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.615 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:17:56.615 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.615 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:17:56.615 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:17:56.616 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:17:56.616 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:17:56.617 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:56 smithi106 ceph-mon[44423]: 2.f scrub ok 2024-04-20T17:17:56.617 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:56 smithi106 ceph-mon[44423]: osdmap e310: 12 total, 12 up, 12 in 2024-04-20T17:17:56.644 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:56.719 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:17:56.726 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:17:56.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:56.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:17:56.828 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:17:56.854 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:56.873 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:17:56.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:56.876 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:17:56.977 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:17:57.003 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:57.023 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:17:57.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:57.026 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:17:57.127 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:17:57.153 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:57.173 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:17:57.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:57.176 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:17:57.241 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-20T17:17:57.277 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:17:57.306 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:57.328 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:17:57.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:57.331 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:17:57.336 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:17:57.432 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:17:57.456 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:57 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:17:57.456 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:57 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:17:57.456 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:57 smithi171 ceph-mon[44189]: 2.6 scrub starts 2024-04-20T17:17:57.456 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:57 smithi171 ceph-mon[44189]: 2.6 scrub ok 2024-04-20T17:17:57.456 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:57 smithi171 ceph-mon[44189]: pgmap v855: 17 pgs: 1 active+recovering, 16 active+clean; 3.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 217 op/s; 5671/108026 objects misplaced (5.250%); 520 KiB/s, 8 objects/s recovering 2024-04-20T17:17:57.458 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:57.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:57 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:17:57.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:57 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:17:57.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:57 smithi005 ceph-mon[39562]: 2.6 scrub starts 2024-04-20T17:17:57.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:57 smithi005 ceph-mon[39562]: 2.6 scrub ok 2024-04-20T17:17:57.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:57 smithi005 ceph-mon[39562]: pgmap v855: 17 pgs: 1 active+recovering, 16 active+clean; 3.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 217 op/s; 5671/108026 objects misplaced (5.250%); 520 KiB/s, 8 objects/s recovering 2024-04-20T17:17:57.477 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:17:57.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:57.479 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:17:57.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:57 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:17:57.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:57 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:17:57.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:57 smithi106 ceph-mon[44423]: 2.6 scrub starts 2024-04-20T17:17:57.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:57 smithi106 ceph-mon[44423]: 2.6 scrub ok 2024-04-20T17:17:57.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:57 smithi106 ceph-mon[44423]: pgmap v855: 17 pgs: 1 active+recovering, 16 active+clean; 3.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 217 op/s; 5671/108026 objects misplaced (5.250%); 520 KiB/s, 8 objects/s recovering 2024-04-20T17:17:57.569 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:17:57.580 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:17:57.656 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:57.680 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:17:57.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:57.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:17:57.786 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:17:57.817 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:57.966 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:17:57.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:57.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:17:58.070 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:17:58.098 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:58.231 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:17:58.235 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:58 smithi005 ceph-mon[39562]: 2.9 scrub starts 2024-04-20T17:17:58.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:58.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:17:58.338 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:17:58.365 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:58.385 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:17:58.386 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:58 smithi171 ceph-mon[44189]: 2.9 scrub starts 2024-04-20T17:17:58.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:58.388 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:17:58.489 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:17:58.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:58 smithi106 ceph-mon[44423]: 2.9 scrub starts 2024-04-20T17:17:58.515 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:58.534 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:17:58.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:58.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:17:58.637 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:17:58.664 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:58.683 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:17:58.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:58.685 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:17:58.786 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:17:58.815 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:58.835 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:17:58.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:58.838 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:17:58.939 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:17:58.966 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:58.985 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:17:58.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:17:58.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:17:59.091 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:17:59.122 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:17:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:59 smithi106 ceph-mon[44423]: 2.9 scrub ok 2024-04-20T17:17:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:17:59 smithi106 ceph-mon[44423]: pgmap v856: 17 pgs: 1 active+recovering, 16 active+clean; 3.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 257 op/s; 5671/106966 objects misplaced (5.302%); 520 KiB/s, 8 objects/s recovering 2024-04-20T17:17:59.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:59 smithi005 ceph-mon[39562]: 2.9 scrub ok 2024-04-20T17:17:59.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:17:59 smithi005 ceph-mon[39562]: pgmap v856: 17 pgs: 1 active+recovering, 16 active+clean; 3.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 257 op/s; 5671/106966 objects misplaced (5.302%); 520 KiB/s, 8 objects/s recovering 2024-04-20T17:17:59.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:59 smithi171 ceph-mon[44189]: 2.9 scrub ok 2024-04-20T17:17:59.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:17:59 smithi171 ceph-mon[44189]: pgmap v856: 17 pgs: 1 active+recovering, 16 active+clean; 3.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 257 op/s; 5671/106966 objects misplaced (5.302%); 520 KiB/s, 8 objects/s recovering 2024-04-20T17:18:00.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:00 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:18:00.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:00 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:18:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:00 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:18:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:00 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:18:00.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:00 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:18:00.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:00 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:18:00.774 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:18:00.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:00.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:18:00.879 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:18:00.905 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:00.923 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:18:00.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:00.926 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:18:01.027 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:18:01.054 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:01.069 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:18:01.073 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:18:01.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:01.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:18:01.091 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:18:01.120 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:18:01.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:01 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:18:01.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:01 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:18:01.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:01 smithi005 ceph-mon[39562]: pgmap v857: 17 pgs: 1 active+recovering, 16 active+clean; 3.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 237 op/s; 5537/106386 objects misplaced (5.205%); 1.0 MiB/s, 16 objects/s recovering 2024-04-20T17:18:01.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:01 smithi005 ceph-mon[39562]: osdmap e311: 12 total, 12 up, 12 in 2024-04-20T17:18:01.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:01 smithi005 ceph-mon[39562]: osdmap e312: 12 total, 12 up, 12 in 2024-04-20T17:18:01.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:01 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1145943707' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:18:01.142 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:18:01.177 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:18:01.178 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:18:01.201 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:18:01.202 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:01 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:18:01.202 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:01 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:18:01.202 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:01 smithi171 ceph-mon[44189]: pgmap v857: 17 pgs: 1 active+recovering, 16 active+clean; 3.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 237 op/s; 5537/106386 objects misplaced (5.205%); 1.0 MiB/s, 16 objects/s recovering 2024-04-20T17:18:01.202 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:01 smithi171 ceph-mon[44189]: osdmap e311: 12 total, 12 up, 12 in 2024-04-20T17:18:01.202 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:01 smithi171 ceph-mon[44189]: osdmap e312: 12 total, 12 up, 12 in 2024-04-20T17:18:01.202 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:01 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1145943707' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:18:01.203 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:01.221 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:18:01.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:01.224 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:18:01.226 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:18:01.255 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:18:01.281 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:18:01.305 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:18:01.325 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:18:01.327 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:18:01.350 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:01 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:18:01.350 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:01 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:18:01.350 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:01 smithi106 ceph-mon[44423]: pgmap v857: 17 pgs: 1 active+recovering, 16 active+clean; 3.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 237 op/s; 5537/106386 objects misplaced (5.205%); 1.0 MiB/s, 16 objects/s recovering 2024-04-20T17:18:01.350 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:01 smithi106 ceph-mon[44423]: osdmap e311: 12 total, 12 up, 12 in 2024-04-20T17:18:01.350 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:01 smithi106 ceph-mon[44423]: osdmap e312: 12 total, 12 up, 12 in 2024-04-20T17:18:01.350 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:01 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1145943707' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:18:01.350 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:18:01.351 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:18:01.351 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:18:01.351 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:18:01.351 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:18:01.351 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:18:01.351 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:18:01.351 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:18:01.352 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:18:01.352 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:18:01.352 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:18:01.352 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:18:01.352 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:18:01.352 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:01.371 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:18:01.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:01.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:18:01.381 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:18:01.474 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:18:01.501 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:01.519 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:18:01.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:01.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:18:01.616 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:18:01.618 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:18:01.623 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:18:01.656 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:18:01.668 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:02.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:02 smithi005 ceph-mon[39562]: from='client.25528 v1:172.21.15.5:0/263530902' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:18:02.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:02 smithi106 ceph-mon[44423]: from='client.25528 v1:172.21.15.5:0/263530902' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:18:02.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:02.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:02 smithi171 ceph-mon[44189]: from='client.25528 v1:172.21.15.5:0/263530902' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:18:02.857 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":859,"stamp":"2024-04-20T17:18:00.970003+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3462267448,"num_objects":52824,"num_object_clones":0,"num_object_copies":105648,"num_objects_missing_on_primary":2761,"num_objects_missing":2761,"num_objects_degraded":0,"num_objects_misplaced":5537,"num_objects_unfound":0,"num_objects_dirty":52824,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":269343,"num_write_kb":10306363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26448,"num_bytes_recovered":1734887552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160792,"ondisk_log_size":160792,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":8649920,"kb_used_data":7489484,"kb_used_omap":219,"kb_used_meta":1160100,"kb_avail":1116291904,"statfs":{"total":1151940427776,"available":1143082909696,"internally_reserved":0,"allocated":7669231616,"data_stored":7656715208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224949,"internal_metadata":1187942731},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,4,1,0,1,0,0,2,7,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-101056512,"num_objects":-1542,"num_object_clones":0,"num_object_copies":-3084,"num_objects_missing_on_primary":-69,"num_objects_missing":-69,"num_objects_degraded":0,"num_objects_misplaced":-134,"num_objects_unfound":0,"num_objects_dirty":-1542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1542,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":4259840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.905755"},"pg_stats":[{"pgid":"2.f","version":"310'16877","reported_seq":17827,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-20T17:17:58.265500+0000","last_change":"2024-04-20T17:17:55.070885+0000","last_active":"2024-04-20T17:17:58.265500+0000","last_peered":"2024-04-20T17:17:58.265500+0000","last_clean":"2024-04-20T17:17:58.265500+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:17:58.265500+0000","last_undegraded":"2024-04-20T17:17:58.265500+0000","last_fullsized":"2024-04-20T17:17:58.265500+0000","mapping_epoch":115,"log_start":"136'6800","ondisk_log_start":"136'6800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'16821","last_scrub_stamp":"2024-04-20T17:17:55.070768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:55.070768+0000","objects_scrubbed":3362,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:57.945378+0000","scrub_duration":356,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216596480,"num_objects":3305,"num_object_clones":0,"num_object_copies":6610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16877,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"311'16772","reported_seq":28750,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-20T17:18:00.337688+0000","last_change":"2024-04-20T17:16:40.290499+0000","last_active":"2024-04-20T17:18:00.337688+0000","last_peered":"2024-04-20T17:18:00.337688+0000","last_clean":"2024-04-20T17:18:00.337688+0000","last_became_active":"2024-04-20T17:10:28.865670+0000","last_became_peered":"2024-04-20T17:10:28.865670+0000","last_unstale":"2024-04-20T17:18:00.337688+0000","last_undegraded":"2024-04-20T17:18:00.337688+0000","last_fullsized":"2024-04-20T17:18:00.337688+0000","mapping_epoch":162,"log_start":"136'6700","ondisk_log_start":"136'6700","created":115,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'15835","last_scrub_stamp":"2024-04-20T17:16:40.290402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:40.290402+0000","objects_scrubbed":4222,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":371,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215220224,"num_objects":3284,"num_object_clones":0,"num_object_copies":6568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16772,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5619,"num_bytes_recovered":368246784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"311'17139","reported_seq":17938,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-20T17:18:00.361228+0000","last_change":"2024-04-20T17:17:12.348489+0000","last_active":"2024-04-20T17:18:00.361228+0000","last_peered":"2024-04-20T17:18:00.361228+0000","last_clean":"2024-04-20T17:18:00.361228+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:18:00.361228+0000","last_undegraded":"2024-04-20T17:18:00.361228+0000","last_fullsized":"2024-04-20T17:18:00.361228+0000","mapping_epoch":115,"log_start":"136'7100","ondisk_log_start":"136'7100","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'16554","last_scrub_stamp":"2024-04-20T17:17:12.348391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:12.348391+0000","objects_scrubbed":3930,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:33.813428+0000","scrub_duration":337,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219086848,"num_objects":3343,"num_object_clones":0,"num_object_copies":6686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17139,"num_write_kb":655424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"311'23494","reported_seq":45089,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-20T17:18:00.143499+0000","last_change":"2024-04-20T17:17:53.095109+0000","last_active":"2024-04-20T17:18:00.143499+0000","last_peered":"2024-04-20T17:18:00.143499+0000","last_clean":"2024-04-20T17:18:00.143499+0000","last_became_active":"2024-04-20T17:12:08.085936+0000","last_became_peered":"2024-04-20T17:12:08.085936+0000","last_unstale":"2024-04-20T17:18:00.143499+0000","last_undegraded":"2024-04-20T17:18:00.143499+0000","last_fullsized":"2024-04-20T17:18:00.143499+0000","mapping_epoch":196,"log_start":"206'13468","ondisk_log_start":"206'13468","created":115,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'23389","last_scrub_stamp":"2024-04-20T17:17:53.094985+0000","last_deep_scrub":"213'20373","last_deep_scrub_stamp":"2024-04-20T17:12:48.443808+0000","last_clean_scrub_stamp":"2024-04-20T17:17:53.094985+0000","objects_scrubbed":3498,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:20.542939+0000","scrub_duration":306,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222298112,"num_objects":3392,"num_object_clones":0,"num_object_copies":6784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16842,"num_write_kb":647488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":450166784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"310'16524","reported_seq":36914,"reported_epoch":310,"state":"active+recovering","last_fresh":"2024-04-20T17:17:58.067575+0000","last_change":"2024-04-20T17:12:26.166109+0000","last_active":"2024-04-20T17:17:58.067575+0000","last_peered":"2024-04-20T17:17:58.067575+0000","last_clean":"2024-04-20T17:09:35.882603+0000","last_became_active":"2024-04-20T17:10:54.470253+0000","last_became_peered":"2024-04-20T17:10:54.470253+0000","last_unstale":"2024-04-20T17:17:58.067575+0000","last_undegraded":"2024-04-20T17:17:58.067575+0000","last_fullsized":"2024-04-20T17:17:58.067575+0000","mapping_epoch":171,"log_start":"136'6500","ondisk_log_start":"136'6500","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210108416,"num_objects":3206,"num_object_clones":0,"num_object_copies":6412,"num_objects_missing_on_primary":2761,"num_objects_missing":2761,"num_objects_degraded":0,"num_objects_misplaced":5537,"num_objects_unfound":0,"num_objects_dirty":3206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16524,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6656,"num_bytes_recovered":436207616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"0,2,5","objects":15},{"shards":"0,10","objects":430},{"shards":"2,5","objects":2761}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":815,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-20T17:17:55.973647+0000","last_change":"2024-04-20T17:16:40.132382+0000","last_active":"2024-04-20T17:17:55.973647+0000","last_peered":"2024-04-20T17:17:55.973647+0000","last_clean":"2024-04-20T17:17:55.973647+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:17:55.973647+0000","last_undegraded":"2024-04-20T17:17:55.973647+0000","last_fullsized":"2024-04-20T17:17:55.973647+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:16:40.132254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:16:40.132254+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"310'16887","reported_seq":17715,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-20T17:17:57.426787+0000","last_change":"2024-04-20T17:17:10.481774+0000","last_active":"2024-04-20T17:17:57.426787+0000","last_peered":"2024-04-20T17:17:57.426787+0000","last_clean":"2024-04-20T17:17:57.426787+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:17:57.426787+0000","last_undegraded":"2024-04-20T17:17:57.426787+0000","last_fullsized":"2024-04-20T17:17:57.426787+0000","mapping_epoch":115,"log_start":"136'6800","ondisk_log_start":"136'6800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'16300","last_scrub_stamp":"2024-04-20T17:17:10.481666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:10.481666+0000","objects_scrubbed":3956,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:18.079313+0000","scrub_duration":337,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220528640,"num_objects":3365,"num_object_clones":0,"num_object_copies":6730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16887,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"310'17031","reported_seq":17858,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-20T17:17:58.373213+0000","last_change":"2024-04-20T17:17:15.112506+0000","last_active":"2024-04-20T17:17:58.373213+0000","last_peered":"2024-04-20T17:17:58.373213+0000","last_clean":"2024-04-20T17:17:58.373213+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:17:58.373213+0000","last_undegraded":"2024-04-20T17:17:58.373213+0000","last_fullsized":"2024-04-20T17:17:58.373213+0000","mapping_epoch":115,"log_start":"136'7000","ondisk_log_start":"136'7000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'16484","last_scrub_stamp":"2024-04-20T17:17:15.112253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:15.112253+0000","objects_scrubbed":3848,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:45.026807+0000","scrub_duration":457,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215810048,"num_objects":3293,"num_object_clones":0,"num_object_copies":6586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17031,"num_write_kb":650368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"310'20900","reported_seq":21868,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-20T17:17:56.755210+0000","last_change":"2024-04-20T17:17:55.709069+0000","last_active":"2024-04-20T17:17:56.755210+0000","last_peered":"2024-04-20T17:17:56.755210+0000","last_clean":"2024-04-20T17:17:56.755210+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:17:56.755210+0000","last_undegraded":"2024-04-20T17:17:56.755210+0000","last_fullsized":"2024-04-20T17:17:56.755210+0000","mapping_epoch":115,"log_start":"154'10802","ondisk_log_start":"154'10802","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'20878","last_scrub_stamp":"2024-04-20T17:17:55.708833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:55.708833+0000","objects_scrubbed":3275,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:11.407326+0000","scrub_duration":320,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213188608,"num_objects":3253,"num_object_clones":0,"num_object_copies":6506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16789,"num_write_kb":641344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"310'16918","reported_seq":17802,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-20T17:17:57.299124+0000","last_change":"2024-04-20T17:17:08.025358+0000","last_active":"2024-04-20T17:17:57.299124+0000","last_peered":"2024-04-20T17:17:57.299124+0000","last_clean":"2024-04-20T17:17:57.299124+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:17:57.299124+0000","last_undegraded":"2024-04-20T17:17:57.299124+0000","last_fullsized":"2024-04-20T17:17:57.299124+0000","mapping_epoch":115,"log_start":"136'6900","ondisk_log_start":"136'6900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'16358","last_scrub_stamp":"2024-04-20T17:17:08.025214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:08.025214+0000","objects_scrubbed":3866,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:25.817318+0000","scrub_duration":359,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216399872,"num_objects":3302,"num_object_clones":0,"num_object_copies":6604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16918,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"310'16750","reported_seq":17560,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-20T17:17:57.900655+0000","last_change":"2024-04-20T17:17:16.522161+0000","last_active":"2024-04-20T17:17:57.900655+0000","last_peered":"2024-04-20T17:17:57.900655+0000","last_clean":"2024-04-20T17:17:57.900655+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:17:57.900655+0000","last_undegraded":"2024-04-20T17:17:57.900655+0000","last_fullsized":"2024-04-20T17:17:57.900655+0000","mapping_epoch":115,"log_start":"136'6700","ondisk_log_start":"136'6700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'16253","last_scrub_stamp":"2024-04-20T17:17:16.522032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:16.522032+0000","objects_scrubbed":3799,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:29.552786+0000","scrub_duration":315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216399872,"num_objects":3302,"num_object_clones":0,"num_object_copies":6604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16750,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"310'16919","reported_seq":32330,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-20T17:17:57.771338+0000","last_change":"2024-04-20T17:17:57.501217+0000","last_active":"2024-04-20T17:17:57.771338+0000","last_peered":"2024-04-20T17:17:57.771338+0000","last_clean":"2024-04-20T17:17:57.771338+0000","last_became_active":"2024-04-20T17:10:55.931751+0000","last_became_peered":"2024-04-20T17:10:55.931751+0000","last_unstale":"2024-04-20T17:17:57.771338+0000","last_undegraded":"2024-04-20T17:17:57.771338+0000","last_fullsized":"2024-04-20T17:17:57.771338+0000","mapping_epoch":173,"log_start":"136'6900","ondisk_log_start":"136'6900","created":115,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'16916","last_scrub_stamp":"2024-04-20T17:17:57.501089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:57.501089+0000","objects_scrubbed":3350,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:26.178414+0000","scrub_duration":318,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219086848,"num_objects":3343,"num_object_clones":0,"num_object_copies":6686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16919,"num_write_kb":648384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7296,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"310'17010","reported_seq":17838,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-20T17:17:57.371208+0000","last_change":"2024-04-20T17:16:42.499318+0000","last_active":"2024-04-20T17:17:57.371208+0000","last_peered":"2024-04-20T17:17:57.371208+0000","last_clean":"2024-04-20T17:17:57.371208+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:17:57.371208+0000","last_undegraded":"2024-04-20T17:17:57.371208+0000","last_fullsized":"2024-04-20T17:17:57.371208+0000","mapping_epoch":115,"log_start":"136'7000","ondisk_log_start":"136'7000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'16092","last_scrub_stamp":"2024-04-20T17:16:42.499193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:42.499193+0000","objects_scrubbed":4224,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:04.645993+0000","scrub_duration":355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216662016,"num_objects":3306,"num_object_clones":0,"num_object_copies":6612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17010,"num_write_kb":650112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"310'16533","reported_seq":17357,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-20T17:17:58.492682+0000","last_change":"2024-04-20T17:17:06.994658+0000","last_active":"2024-04-20T17:17:58.492682+0000","last_peered":"2024-04-20T17:17:58.492682+0000","last_clean":"2024-04-20T17:17:58.492682+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:17:58.492682+0000","last_undegraded":"2024-04-20T17:17:58.492682+0000","last_fullsized":"2024-04-20T17:17:58.492682+0000","mapping_epoch":115,"log_start":"136'6500","ondisk_log_start":"136'6500","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'15884","last_scrub_stamp":"2024-04-20T17:17:06.994455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:06.994455+0000","objects_scrubbed":3982,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:21.957254+0000","scrub_duration":397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218300416,"num_objects":3331,"num_object_clones":0,"num_object_copies":6662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16533,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"310'16826","reported_seq":17649,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-04-20T17:17:57.371667+0000","last_change":"2024-04-20T17:17:07.882718+0000","last_active":"2024-04-20T17:17:57.371667+0000","last_peered":"2024-04-20T17:17:57.371667+0000","last_clean":"2024-04-20T17:17:57.371667+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:17:57.371667+0000","last_undegraded":"2024-04-20T17:17:57.371667+0000","last_fullsized":"2024-04-20T17:17:57.371667+0000","mapping_epoch":115,"log_start":"136'6800","ondisk_log_start":"136'6800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'16236","last_scrub_stamp":"2024-04-20T17:17:07.882596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:07.882596+0000","objects_scrubbed":3868,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:32.661323+0000","scrub_duration":370,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214695936,"num_objects":3276,"num_object_clones":0,"num_object_copies":6552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16826,"num_write_kb":643264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"311'16628","reported_seq":17454,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-20T17:18:00.369847+0000","last_change":"2024-04-20T17:17:04.896764+0000","last_active":"2024-04-20T17:18:00.369847+0000","last_peered":"2024-04-20T17:18:00.369847+0000","last_clean":"2024-04-20T17:18:00.369847+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:18:00.369847+0000","last_undegraded":"2024-04-20T17:18:00.369847+0000","last_fullsized":"2024-04-20T17:18:00.369847+0000","mapping_epoch":115,"log_start":"136'6600","ondisk_log_start":"136'6600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"293'15941","last_scrub_stamp":"2024-04-20T17:17:04.896640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:04.896640+0000","objects_scrubbed":3929,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:16.518914+0000","scrub_duration":355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212402200,"num_objects":3242,"num_object_clones":0,"num_object_copies":6484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3242,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16628,"num_write_kb":635777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"311'16717","reported_seq":17416,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-04-20T17:18:00.840762+0000","last_change":"2024-04-20T17:16:23.593408+0000","last_active":"2024-04-20T17:18:00.840762+0000","last_peered":"2024-04-20T17:18:00.840762+0000","last_clean":"2024-04-20T17:18:00.840762+0000","last_became_active":"2024-04-20T17:12:23.515375+0000","last_became_peered":"2024-04-20T17:12:23.515375+0000","last_unstale":"2024-04-20T17:18:00.840762+0000","last_undegraded":"2024-04-20T17:18:00.840762+0000","last_fullsized":"2024-04-20T17:18:00.840762+0000","mapping_epoch":202,"log_start":"136'6700","ondisk_log_start":"136'6700","created":115,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'15586","last_scrub_stamp":"2024-04-20T17:16:23.593300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:16:23.593300+0000","objects_scrubbed":4410,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214892544,"num_objects":3279,"num_object_clones":0,"num_object_copies":6558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16717,"num_write_kb":639872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":3461677080,"num_objects":52822,"num_object_clones":0,"num_object_copies":105644,"num_objects_missing_on_primary":2761,"num_objects_missing":2761,"num_objects_degraded":0,"num_objects_misplaced":5537,"num_objects_unfound":0,"num_objects_dirty":52822,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":269162,"num_write_kb":10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26440,"num_bytes_recovered":1732771840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7626104832,"data_stored":7626096688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},"log_size":160655,"ondisk_log_size":160655,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991379,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":702336,"kb_used_data":639224,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93042816,"statfs":{"total":95995035648,"available":95275843584,"internally_reserved":0,"allocated":654565376,"data_stored":653519550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":64616900},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253017,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":839164,"kb_used_data":669168,"kb_used_omap":1,"kb_used_meta":169982,"kb_avail":92905988,"statfs":{"total":95995035648,"available":95135731712,"internally_reserved":0,"allocated":685228032,"data_stored":684185758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":174062016},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612767,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":995388,"kb_used_data":847604,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92749764,"statfs":{"total":95995035648,"available":94975758336,"internally_reserved":0,"allocated":867946496,"data_stored":866900170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":151321024},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762808996,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":947704,"kb_used_data":844336,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92797448,"statfs":{"total":95995035648,"available":95024586752,"internally_reserved":0,"allocated":864600064,"data_stored":863557810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168745,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":716740,"kb_used_data":645304,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93028412,"statfs":{"total":95995035648,"available":95261093888,"internally_reserved":0,"allocated":660791296,"data_stored":659745490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430381,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":533948,"kb_used_data":434736,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93211204,"statfs":{"total":95995035648,"available":95448272896,"internally_reserved":0,"allocated":445169664,"data_stored":444127410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101579206},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575047,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":535036,"kb_used_data":427504,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93210116,"statfs":{"total":95995035648,"available":95447158784,"internally_reserved":0,"allocated":437764096,"data_stored":436721842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738572,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":799868,"kb_used_data":664560,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":92945284,"statfs":{"total":95995035648,"available":95175970816,"internally_reserved":0,"allocated":680509440,"data_stored":679467186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,0,1,0,0,2,7,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393861,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":695544,"kb_used_data":642992,"kb_used_omap":202,"kb_used_meta":52341,"kb_avail":93049608,"statfs":{"total":95995035648,"available":95282798592,"internally_reserved":0,"allocated":658423808,"data_stored":657381554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":53597655},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182588,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":507372,"kb_used_data":431780,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93237780,"statfs":{"total":95995035648,"available":95475486720,"internally_reserved":0,"allocated":442142720,"data_stored":441106394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986360,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":237564,"kb_used_data":210548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93507588,"statfs":{"total":95995035648,"available":95751770112,"internally_reserved":0,"allocated":215601152,"data_stored":214554826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790130,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1139256,"kb_used_data":1031728,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92605896,"statfs":{"total":95995035648,"available":94828437504,"internally_reserved":0,"allocated":1056489472,"data_stored":1055447218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":677642240,"data_stored":677642240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":435093504,"data_stored":435093504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":655097856,"data_stored":655097856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":438239232,"data_stored":438239232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":212471808,"data_stored":212467736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1052704768,"data_stored":1052704768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":441778176,"data_stored":441778176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":656015360,"data_stored":656015360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":862191616,"data_stored":862191616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":862523392,"data_stored":862519320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":681705472,"data_stored":681705472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":650641408,"data_stored":650641408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:18:02.858 INFO:tasks.cephadm.ceph_manager.ceph:PG 2.3 is not active+clean 2024-04-20T17:18:02.858 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '2.3', 'version': "310'16524", 'reported_seq': 36914, 'reported_epoch': 310, 'state': 'active+recovering', 'last_fresh': '2024-04-20T17:17:58.067575+0000', 'last_change': '2024-04-20T17:12:26.166109+0000', 'last_active': '2024-04-20T17:17:58.067575+0000', 'last_peered': '2024-04-20T17:17:58.067575+0000', 'last_clean': '2024-04-20T17:09:35.882603+0000', 'last_became_active': '2024-04-20T17:10:54.470253+0000', 'last_became_peered': '2024-04-20T17:10:54.470253+0000', 'last_unstale': '2024-04-20T17:17:58.067575+0000', 'last_undegraded': '2024-04-20T17:17:58.067575+0000', 'last_fullsized': '2024-04-20T17:17:58.067575+0000', 'mapping_epoch': 171, 'log_start': "136'6500", 'ondisk_log_start': "136'6500", 'created': 115, 'last_epoch_clean': 116, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-20T17:08:04.990639+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T17:08:04.990639+0000', 'last_clean_scrub_stamp': '2024-04-20T17:08:04.990639+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 210108416, 'num_objects': 3206, 'num_object_clones': 0, 'num_object_copies': 6412, 'num_objects_missing_on_primary': 2761, 'num_objects_missing': 2761, 'num_objects_degraded': 0, 'num_objects_misplaced': 5537, 'num_objects_unfound': 0, 'num_objects_dirty': 3206, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16524, 'num_write_kb': 631360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6656, 'num_bytes_recovered': 436207616, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 10], 'acting': [0, 10], 'avail_no_missing': ['2', '5'], 'object_location_counts': [{'shards': '0,2,5', 'objects': 15}, {'shards': '0,10', 'objects': 430}, {'shards': '2,5', 'objects': 2761}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-20T17:18:02.859 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph status --format=json 2024-04-20T17:18:03.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:03.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:03.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:03 smithi005 ceph-mon[39562]: pgmap v860: 17 pgs: 1 active+recovering, 16 active+clean; 3.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 206 op/s; 5537/105504 objects misplaced (5.248%); 682 KiB/s, 10 objects/s recovering 2024-04-20T17:18:03.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:03 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:18:03.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:03.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:03 smithi106 ceph-mon[44423]: pgmap v860: 17 pgs: 1 active+recovering, 16 active+clean; 3.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 206 op/s; 5537/105504 objects misplaced (5.248%); 682 KiB/s, 10 objects/s recovering 2024-04-20T17:18:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:03 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:18:03.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:03.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:03.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:03 smithi171 ceph-mon[44189]: pgmap v860: 17 pgs: 1 active+recovering, 16 active+clean; 3.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 206 op/s; 5537/105504 objects misplaced (5.248%); 682 KiB/s, 10 objects/s recovering 2024-04-20T17:18:03.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:03 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:18:04.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:04 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:18:04.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:04 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:18:04.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:04 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:18:04.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:04 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:18:04.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:04 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:18:04.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:04 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:18:04.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:04 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:18:04.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:04 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:18:04.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:04 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:18:05.017 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:18:05.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:05.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:18:05.131 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:18:05.158 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:05.176 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:18:05.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:05.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:18:05.280 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:18:05.307 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:05.327 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:18:05.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:05.330 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:18:05.431 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:18:05.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:05 smithi005 ceph-mon[39562]: pgmap v861: 17 pgs: 1 active+recovering, 16 active+clean; 3.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 240 op/s; 5403/104178 objects misplaced (5.186%); 1.0 MiB/s, 16 objects/s recovering 2024-04-20T17:18:05.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:05 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:18:05.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:05 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:18:05.464 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:05.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:05 smithi106 ceph-mon[44423]: pgmap v861: 17 pgs: 1 active+recovering, 16 active+clean; 3.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 240 op/s; 5403/104178 objects misplaced (5.186%); 1.0 MiB/s, 16 objects/s recovering 2024-04-20T17:18:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:05 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:18:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:05 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:18:05.520 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:18:05.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:05.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:18:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:05 smithi171 ceph-mon[44189]: pgmap v861: 17 pgs: 1 active+recovering, 16 active+clean; 3.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 240 op/s; 5403/104178 objects misplaced (5.186%); 1.0 MiB/s, 16 objects/s recovering 2024-04-20T17:18:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:05 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:18:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:05 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:18:05.625 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:18:05.651 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:05.670 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:18:05.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:05.673 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:18:05.774 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:18:05.810 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:06.298 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:18:06.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:06 smithi005 ceph-mon[39562]: osdmap e313: 12 total, 12 up, 12 in 2024-04-20T17:18:06.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:06 smithi005 ceph-mon[39562]: osdmap e314: 12 total, 12 up, 12 in 2024-04-20T17:18:06.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:06 smithi106 ceph-mon[44423]: osdmap e313: 12 total, 12 up, 12 in 2024-04-20T17:18:06.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:06 smithi106 ceph-mon[44423]: osdmap e314: 12 total, 12 up, 12 in 2024-04-20T17:18:06.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:06 smithi171 ceph-mon[44189]: osdmap e313: 12 total, 12 up, 12 in 2024-04-20T17:18:06.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:06 smithi171 ceph-mon[44189]: osdmap e314: 12 total, 12 up, 12 in 2024-04-20T17:18:06.655 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:18:06.663 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:18:06.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:06.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:18:06.768 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:18:06.795 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:06.813 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:18:06.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:06.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:18:06.917 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:18:06.953 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:06.982 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:18:06.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:06.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:18:07.085 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:18:07.111 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:07.129 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:18:07.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:07.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:18:07.166 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:18:07.199 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:07 smithi005 ceph-mon[39562]: 2.a scrub starts 2024-04-20T17:18:07.199 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:07 smithi005 ceph-mon[39562]: pgmap v864: 17 pgs: 1 active+recovering, 16 active+clean; 3.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 242 op/s; 5403/103508 objects misplaced (5.220%); 734 KiB/s, 11 objects/s recovering 2024-04-20T17:18:07.199 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:07 smithi005 ceph-mon[39562]: 2.a scrub ok 2024-04-20T17:18:07.199 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:07 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1029768049' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:18:07.199 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:07 smithi005 ceph-mon[39562]: from='client.25562 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:18:07.232 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:18:07.259 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:07.279 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:18:07.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:07.284 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:18:07.302 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:18:07.302 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:18:07.384 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:18:07.408 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:07 smithi106 ceph-mon[44423]: 2.a scrub starts 2024-04-20T17:18:07.408 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:07 smithi106 ceph-mon[44423]: pgmap v864: 17 pgs: 1 active+recovering, 16 active+clean; 3.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 242 op/s; 5403/103508 objects misplaced (5.220%); 734 KiB/s, 11 objects/s recovering 2024-04-20T17:18:07.408 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:07 smithi106 ceph-mon[44423]: 2.a scrub ok 2024-04-20T17:18:07.408 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:07 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1029768049' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:18:07.408 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:07 smithi106 ceph-mon[44423]: from='client.25562 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:18:07.410 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:07.428 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:18:07.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:07.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:18:07.532 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:18:07.558 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:07.577 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:18:07.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:07.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:18:07.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:07 smithi171 ceph-mon[44189]: 2.a scrub starts 2024-04-20T17:18:07.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:07 smithi171 ceph-mon[44189]: pgmap v864: 17 pgs: 1 active+recovering, 16 active+clean; 3.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 242 op/s; 5403/103508 objects misplaced (5.220%); 734 KiB/s, 11 objects/s recovering 2024-04-20T17:18:07.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:07 smithi171 ceph-mon[44189]: 2.a scrub ok 2024-04-20T17:18:07.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:07 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1029768049' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:18:07.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:07 smithi171 ceph-mon[44189]: from='client.25562 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:18:07.681 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:18:07.707 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:07.725 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:18:07.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:07.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:18:07.828 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:18:07.830 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:18:07.855 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:07.873 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:18:07.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:07.876 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:18:07.977 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:18:08.002 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:08.022 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:18:08.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:08.025 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:18:08.126 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:18:08.152 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:08.173 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:18:08.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:08.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:18:08.278 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:18:08.303 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:08.322 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:18:08.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:08.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:18:08.425 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:18:08.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:08 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:18:08.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:08 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:18:08.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:08 smithi005 ceph-mon[39562]: from='client.25562 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:18:08.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:08 smithi005 ceph-mon[39562]: osdmap e315: 12 total, 12 up, 12 in 2024-04-20T17:18:08.455 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:08.508 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:18:08.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:08 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:18:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:08 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:18:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:08 smithi106 ceph-mon[44423]: from='client.25562 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:18:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:08 smithi106 ceph-mon[44423]: osdmap e315: 12 total, 12 up, 12 in 2024-04-20T17:18:08.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:08.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:18:08.537 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-20T17:18:08.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:08 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:18:08.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:08 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:18:08.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:08 smithi171 ceph-mon[44189]: from='client.25562 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:18:08.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:08 smithi171 ceph-mon[44189]: osdmap e315: 12 total, 12 up, 12 in 2024-04-20T17:18:08.612 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:18:08.639 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:08.658 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:18:08.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:08.663 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:18:08.764 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:18:08.790 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:08.810 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:18:08.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:08.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:18:08.914 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:18:08.940 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:08.959 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:18:08.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:08.961 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:18:09.062 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:18:09.089 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:09.110 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:18:09.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:09.113 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:18:09.214 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:18:09.240 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:09 smithi171 ceph-mon[44189]: 2.4 scrub starts 2024-04-20T17:18:09.240 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:09 smithi171 ceph-mon[44189]: pgmap v866: 17 pgs: 1 active+recovering, 16 active+clean; 3.1 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 241 op/s; 5403/102600 objects misplaced (5.266%); 725 KiB/s, 11 objects/s recovering 2024-04-20T17:18:09.240 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:09 smithi171 ceph-mon[44189]: 2.4 scrub ok 2024-04-20T17:18:09.243 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:09.264 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:18:09.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:09.267 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:18:09.368 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:18:09.394 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:09.412 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:18:09.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:09.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:18:09.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:09 smithi005 ceph-mon[39562]: 2.4 scrub starts 2024-04-20T17:18:09.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:09 smithi005 ceph-mon[39562]: pgmap v866: 17 pgs: 1 active+recovering, 16 active+clean; 3.1 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 241 op/s; 5403/102600 objects misplaced (5.266%); 725 KiB/s, 11 objects/s recovering 2024-04-20T17:18:09.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:09 smithi005 ceph-mon[39562]: 2.4 scrub ok 2024-04-20T17:18:09.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:09 smithi106 ceph-mon[44423]: 2.4 scrub starts 2024-04-20T17:18:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:09 smithi106 ceph-mon[44423]: pgmap v866: 17 pgs: 1 active+recovering, 16 active+clean; 3.1 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 241 op/s; 5403/102600 objects misplaced (5.266%); 725 KiB/s, 11 objects/s recovering 2024-04-20T17:18:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:09 smithi106 ceph-mon[44423]: 2.4 scrub ok 2024-04-20T17:18:09.518 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:18:09.544 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:09.567 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:18:09.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:09.570 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:18:09.671 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:18:09.698 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:09.718 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:18:09.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:09.720 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:18:09.821 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:18:09.847 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:09.866 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:18:09.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:09.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:18:09.970 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:18:10.004 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:11.014 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:18:11.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:11 smithi005 ceph-mon[39562]: pgmap v867: 17 pgs: 1 active+recovering, 16 active+clean; 3.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 173 op/s; 5266/102092 objects misplaced (5.158%); 747 KiB/s, 11 objects/s recovering 2024-04-20T17:18:11.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:11 smithi005 ceph-mon[39562]: osdmap e316: 12 total, 12 up, 12 in 2024-04-20T17:18:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:11 smithi106 ceph-mon[44423]: pgmap v867: 17 pgs: 1 active+recovering, 16 active+clean; 3.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 173 op/s; 5266/102092 objects misplaced (5.158%); 747 KiB/s, 11 objects/s recovering 2024-04-20T17:18:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:11 smithi106 ceph-mon[44423]: osdmap e316: 12 total, 12 up, 12 in 2024-04-20T17:18:11.556 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:18:11.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:11.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:18:11.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:11 smithi171 ceph-mon[44189]: pgmap v867: 17 pgs: 1 active+recovering, 16 active+clean; 3.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 173 op/s; 5266/102092 objects misplaced (5.158%); 747 KiB/s, 11 objects/s recovering 2024-04-20T17:18:11.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:11 smithi171 ceph-mon[44189]: osdmap e316: 12 total, 12 up, 12 in 2024-04-20T17:18:11.661 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:18:11.687 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:11.705 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:18:11.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:11.708 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:18:11.747 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:18:11.809 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:18:11.838 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:11.861 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:18:11.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:11.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:18:11.966 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:18:11.999 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:13.013 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:18:13.032 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:18:13.055 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:18:13.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:13.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:18:13.061 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:18:13.085 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:18:13.105 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:18:13.126 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:18:13.150 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:18:13.160 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:18:13.174 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:18:13.188 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:13.196 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:18:13.208 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:18:13.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:13.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:18:13.230 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:18:13.256 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:18:13.278 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:18:13.280 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:18:13.280 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:18:13.280 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:18:13.280 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:18:13.280 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:18:13.280 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:18:13.280 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:18:13.280 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:18:13.281 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:18:13.281 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:18:13.281 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:18:13.281 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:18:13.311 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:18:13.324 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:13 smithi005 ceph-mon[39562]: pgmap v869: 17 pgs: 1 active+recovering, 16 active+clean; 3.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 188 op/s; 5266/101210 objects misplaced (5.203%); 735 KiB/s, 11 objects/s recovering 2024-04-20T17:18:13.324 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:13 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/847500961' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:18:13.337 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:13.356 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:18:13.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:13.359 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:18:13.460 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:18:13.490 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:13.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:13 smithi106 ceph-mon[44423]: pgmap v869: 17 pgs: 1 active+recovering, 16 active+clean; 3.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 188 op/s; 5266/101210 objects misplaced (5.203%); 735 KiB/s, 11 objects/s recovering 2024-04-20T17:18:13.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:13 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/847500961' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:18:13.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:13 smithi171 ceph-mon[44189]: pgmap v869: 17 pgs: 1 active+recovering, 16 active+clean; 3.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 188 op/s; 5266/101210 objects misplaced (5.203%); 735 KiB/s, 11 objects/s recovering 2024-04-20T17:18:13.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:13 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/847500961' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:18:14.244 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:18:14.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:14.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:18:14.350 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:18:14.379 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:14.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.530 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:18:14.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:14.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:18:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:14.634 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:18:14.662 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:15.189 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:18:15.190 INFO:teuthology.orchestra.run.smithi005.stdout:{"fsid":"13c6e76c-ff37-11ee-bc93-c7b262605968","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1063,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":316,"num_osds":12,"num_up_osds":12,"osd_up_since":1713633141,"num_in_osds":12,"osd_in_since":1713632742,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovering","count":1}],"num_pgs":17,"num_pools":2,"num_objects":49872,"data_bytes":3268805176,"bytes_used":8485011456,"bytes_avail":1143455416320,"bytes_total":1151940427776,"misplaced_objects":5123,"misplaced_total":99744,"misplaced_ratio":0.051361485402630726,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":1146682,"recovering_keys_per_sec":0,"num_objects_recovered":140,"num_bytes_recovered":9175040,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":235},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.5:8443/"}},"servicemap":{"epoch":25,"modified":"2024-04-20T17:16:50.052718+0000","services":{}},"progress_events":{}} 2024-04-20T17:18:15.212 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:18:15.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:15.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:18:15.322 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:18:15.349 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:15.368 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:18:15.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:15.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:18:15.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:15.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:15 smithi005 ceph-mon[39562]: pgmap v870: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 235 op/s; 5123/99744 objects misplaced (5.136%); 1.1 MiB/s, 17 objects/s recovering 2024-04-20T17:18:15.472 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:18:15.497 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:15.497 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:15 smithi171 ceph-mon[44189]: pgmap v870: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 235 op/s; 5123/99744 objects misplaced (5.136%); 1.1 MiB/s, 17 objects/s recovering 2024-04-20T17:18:15.499 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:15.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:15.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:15 smithi106 ceph-mon[44423]: pgmap v870: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 235 op/s; 5123/99744 objects misplaced (5.136%); 1.1 MiB/s, 17 objects/s recovering 2024-04-20T17:18:15.519 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:18:15.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:15.524 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:18:15.625 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:18:15.651 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:15.669 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:18:15.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:15.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:18:15.773 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:18:15.799 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:15.817 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:18:15.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:15.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:18:15.920 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:18:15.952 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:16.100 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:18:16.107 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:18:16.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:16.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:18:16.211 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:18:16.241 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:16.370 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:18:16.372 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:16 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2220156577' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:18:16.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:16.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:18:16.475 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:18:16.505 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:16.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:16 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2220156577' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:18:16.528 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:18:16.528 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:18:16.528 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:18:16.528 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:18:16.528 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:18:16.528 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:18:16.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:16.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:18:16.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:16 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2220156577' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:18:16.632 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:18:16.662 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:16.993 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-20T17:18:16.993 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:18:17.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:17 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:18:17.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:17 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:18:17.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:17 smithi005 ceph-mon[39562]: pgmap v871: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 238 op/s; 5123/99104 objects misplaced (5.169%); 1005 KiB/s, 15 objects/s recovering 2024-04-20T17:18:17.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:17 smithi005 ceph-mon[39562]: osdmap e317: 12 total, 12 up, 12 in 2024-04-20T17:18:17.312 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:18:17.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:17.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:18:17.417 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:18:17.441 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:17 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:18:17.442 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:17 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:18:17.442 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:17 smithi106 ceph-mon[44423]: pgmap v871: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 238 op/s; 5123/99104 objects misplaced (5.169%); 1005 KiB/s, 15 objects/s recovering 2024-04-20T17:18:17.442 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:17 smithi106 ceph-mon[44423]: osdmap e317: 12 total, 12 up, 12 in 2024-04-20T17:18:17.443 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:17.462 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:18:17.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:17.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:18:17.566 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:18:17.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:17 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:18:17.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:17 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:18:17.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:17 smithi171 ceph-mon[44189]: pgmap v871: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 238 op/s; 5123/99104 objects misplaced (5.169%); 1005 KiB/s, 15 objects/s recovering 2024-04-20T17:18:17.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:17 smithi171 ceph-mon[44189]: osdmap e317: 12 total, 12 up, 12 in 2024-04-20T17:18:17.598 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:17.612 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:18:18.339 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-20T17:18:18.350 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:18:18.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:18.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:18:18.454 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:18:18.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:18 smithi005 ceph-mon[39562]: 2.e scrub starts 2024-04-20T17:18:18.480 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:18 smithi171 ceph-mon[44189]: 2.e scrub starts 2024-04-20T17:18:18.482 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:18.501 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:18:18.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:18.503 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:18:18.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:18 smithi106 ceph-mon[44423]: 2.e scrub starts 2024-04-20T17:18:18.604 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:18:18.632 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:18.705 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:18:18.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:18.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:18:18.740 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:18:18.740 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:18:18.809 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:18:18.837 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:19 smithi005 ceph-mon[39562]: 2.e scrub ok 2024-04-20T17:18:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:19 smithi005 ceph-mon[39562]: pgmap v873: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 242 op/s; 5123/98212 objects misplaced (5.216%); 560 KiB/s, 8 objects/s recovering 2024-04-20T17:18:19.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:19 smithi106 ceph-mon[44423]: 2.e scrub ok 2024-04-20T17:18:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:19 smithi106 ceph-mon[44423]: pgmap v873: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 242 op/s; 5123/98212 objects misplaced (5.216%); 560 KiB/s, 8 objects/s recovering 2024-04-20T17:18:19.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:19 smithi171 ceph-mon[44189]: 2.e scrub ok 2024-04-20T17:18:19.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:19 smithi171 ceph-mon[44189]: pgmap v873: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 242 op/s; 5123/98212 objects misplaced (5.216%); 560 KiB/s, 8 objects/s recovering 2024-04-20T17:18:19.686 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:18:19.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:19.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:18:19.792 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:18:19.818 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:19.839 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:18:19.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:19.842 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:18:19.943 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:18:19.970 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:19.989 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:18:19.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:19.993 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:18:20.094 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:18:20.121 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:20.140 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:18:20.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:20.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:18:20.243 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:18:20.273 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:20.527 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:18:20.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:20.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:18:20.632 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:18:20.659 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:20.683 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:18:20.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:20.688 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:18:20.789 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:18:20.824 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:20.847 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:18:20.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:20.851 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:18:20.955 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:18:20.982 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:21.004 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:18:21.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:21.007 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:18:21.108 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:18:21.135 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:21.156 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:18:21.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:21.164 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:18:21.265 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:18:21.291 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:21.310 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:18:21.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:21.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:18:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:21 smithi171 ceph-mon[44189]: 2.0 deep-scrub starts 2024-04-20T17:18:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:21 smithi171 ceph-mon[44189]: pgmap v874: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 202 op/s; 4980/97664 objects misplaced (5.099%); 1006 KiB/s, 15 objects/s recovering 2024-04-20T17:18:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:21 smithi171 ceph-mon[44189]: osdmap e318: 12 total, 12 up, 12 in 2024-04-20T17:18:21.389 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:18:21.413 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:18:21.451 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:21 smithi005 ceph-mon[39562]: 2.0 deep-scrub starts 2024-04-20T17:18:21.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:21 smithi005 ceph-mon[39562]: pgmap v874: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 202 op/s; 4980/97664 objects misplaced (5.099%); 1006 KiB/s, 15 objects/s recovering 2024-04-20T17:18:21.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:21 smithi005 ceph-mon[39562]: osdmap e318: 12 total, 12 up, 12 in 2024-04-20T17:18:21.454 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:21.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:21 smithi106 ceph-mon[44423]: 2.0 deep-scrub starts 2024-04-20T17:18:21.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:21 smithi106 ceph-mon[44423]: pgmap v874: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 202 op/s; 4980/97664 objects misplaced (5.099%); 1006 KiB/s, 15 objects/s recovering 2024-04-20T17:18:21.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:21 smithi106 ceph-mon[44423]: osdmap e318: 12 total, 12 up, 12 in 2024-04-20T17:18:21.705 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:18:21.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:21.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:18:21.810 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:18:21.840 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:21.863 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:18:21.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:21.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:18:21.970 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:18:21.996 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:22.017 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:18:22.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:22.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:18:22.121 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:18:22.148 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:22.167 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:18:22.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:22.170 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:18:22.205 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:18:22.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:22 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3967023238' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:18:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:22 smithi106 ceph-mon[44423]: from='client.25552 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:18:22.270 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:18:22.302 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:22.347 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:18:22.370 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:18:22.396 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:18:22.419 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:18:22.444 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:18:22.454 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:18:22.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:22 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3967023238' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:18:22.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:22 smithi005 ceph-mon[39562]: from='client.25552 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:18:22.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:22.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:18:22.467 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:18:22.489 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:18:22.509 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:18:22.536 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:18:22.556 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:18:22.558 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:18:22.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:22 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3967023238' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:18:22.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:22 smithi171 ceph-mon[44189]: from='client.25552 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:18:22.584 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:18:22.589 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:22.615 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:18:22.617 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:18:22.617 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:18:22.617 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:18:22.617 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:18:22.617 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:18:22.617 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:18:22.617 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:18:22.618 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:18:22.618 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:18:22.618 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:18:22.618 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:18:22.618 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:18:23.372 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:18:23.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:23.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:18:23.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:23 smithi005 ceph-mon[39562]: pgmap v876: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 185 op/s; 4980/96782 objects misplaced (5.146%); 584 KiB/s, 9 objects/s recovering 2024-04-20T17:18:23.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:23 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:18:23.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:23 smithi005 ceph-mon[39562]: from='client.25552 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:18:23.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:23 smithi005 ceph-mon[39562]: osdmap e319: 12 total, 12 up, 12 in 2024-04-20T17:18:23.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:23 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1567117901' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:18:23.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:23.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:23.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:23.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:23.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:23.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:23.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:23.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:23.476 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:18:23.500 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:23 smithi106 ceph-mon[44423]: pgmap v876: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 185 op/s; 4980/96782 objects misplaced (5.146%); 584 KiB/s, 9 objects/s recovering 2024-04-20T17:18:23.500 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:23 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:18:23.500 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:23 smithi106 ceph-mon[44423]: from='client.25552 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:18:23.500 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:23 smithi106 ceph-mon[44423]: osdmap e319: 12 total, 12 up, 12 in 2024-04-20T17:18:23.500 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:23 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1567117901' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:18:23.500 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:23.500 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:23.500 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:23.500 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:23.500 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:23.500 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:23.500 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:23.500 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:23.502 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:23.521 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:18:23.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:23.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:18:23.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:23 smithi171 ceph-mon[44189]: pgmap v876: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 185 op/s; 4980/96782 objects misplaced (5.146%); 584 KiB/s, 9 objects/s recovering 2024-04-20T17:18:23.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:23 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:18:23.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:23 smithi171 ceph-mon[44189]: from='client.25552 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:18:23.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:23 smithi171 ceph-mon[44189]: osdmap e319: 12 total, 12 up, 12 in 2024-04-20T17:18:23.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:23 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1567117901' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:18:23.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:23.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:23.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:23.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:23.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:23.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:23.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:23.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:23.624 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:18:23.655 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:23.675 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:18:23.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:23.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:18:23.779 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:18:23.805 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:23.824 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:18:23.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:23.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:18:23.929 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:18:23.961 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:24 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:24 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:24 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:24 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:24.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:24 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:18:24.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:24 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:18:24.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:24 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:24.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:24 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:24.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:24 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:24.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:24 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:24.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:24 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:18:24.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:24 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:18:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:24 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:24 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:24 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:24 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:24.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:24 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:18:24.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:24 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:18:25.383 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:18:25.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:25 smithi005 ceph-mon[39562]: pgmap v878: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 2.9 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 272 op/s; 4829/95236 objects misplaced (5.071%); 1.3 MiB/s, 21 objects/s recovering 2024-04-20T17:18:25.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:25 smithi005 ceph-mon[39562]: osdmap e320: 12 total, 12 up, 12 in 2024-04-20T17:18:25.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:25 smithi106 ceph-mon[44423]: pgmap v878: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 2.9 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 272 op/s; 4829/95236 objects misplaced (5.071%); 1.3 MiB/s, 21 objects/s recovering 2024-04-20T17:18:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:25 smithi106 ceph-mon[44423]: osdmap e320: 12 total, 12 up, 12 in 2024-04-20T17:18:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:25 smithi171 ceph-mon[44189]: pgmap v878: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 2.9 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 272 op/s; 4829/95236 objects misplaced (5.071%); 1.3 MiB/s, 21 objects/s recovering 2024-04-20T17:18:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:25 smithi171 ceph-mon[44189]: osdmap e320: 12 total, 12 up, 12 in 2024-04-20T17:18:26.520 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:18:27.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:26 smithi106 ceph-mon[44423]: pgmap v880: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 2.9 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 265 op/s; 4829/94474 objects misplaced (5.111%); 811 KiB/s, 12 objects/s recovering 2024-04-20T17:18:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:26 smithi171 ceph-mon[44189]: pgmap v880: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 2.9 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 265 op/s; 4829/94474 objects misplaced (5.111%); 811 KiB/s, 12 objects/s recovering 2024-04-20T17:18:27.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:26 smithi005 ceph-mon[39562]: pgmap v880: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 2.9 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 265 op/s; 4829/94474 objects misplaced (5.111%); 811 KiB/s, 12 objects/s recovering 2024-04-20T17:18:27.516 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:18:27.752 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:18:27.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:27.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:18:27.775 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:18:27.775 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:18:27.856 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:18:27.884 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:27.904 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:18:27.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:27.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:18:28.009 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:18:28.036 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:28.040 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:27 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:18:28.040 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:27 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:18:28.040 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:27 smithi005 ceph-mon[39562]: osdmap e321: 12 total, 12 up, 12 in 2024-04-20T17:18:28.057 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:18:28.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:28.060 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:18:28.161 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:18:28.193 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:28.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:27 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:18:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:27 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:18:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:27 smithi106 ceph-mon[44423]: osdmap e321: 12 total, 12 up, 12 in 2024-04-20T17:18:28.296 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:18:28.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:28.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:18:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:27 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:18:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:27 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:18:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:27 smithi171 ceph-mon[44189]: osdmap e321: 12 total, 12 up, 12 in 2024-04-20T17:18:28.402 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:18:28.429 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:28.454 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:18:28.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:28.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:18:28.557 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:18:28.584 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:28.604 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:18:28.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:28.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:18:28.708 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:18:28.734 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:28.755 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:18:28.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:28.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:18:28.859 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:18:28.886 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:28.904 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:18:28.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:28.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:18:28.964 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-20T17:18:29.008 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:18:29.035 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:29.055 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:18:29.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:29.057 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:18:29.158 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:18:29.184 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:29.204 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:18:29.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:29.207 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:18:29.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:28 smithi005 ceph-mon[39562]: pgmap v882: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 2.9 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 270 op/s; 4829/93538 objects misplaced (5.163%); 811 KiB/s, 12 objects/s recovering 2024-04-20T17:18:29.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:28 smithi106 ceph-mon[44423]: pgmap v882: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 2.9 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 270 op/s; 4829/93538 objects misplaced (5.163%); 811 KiB/s, 12 objects/s recovering 2024-04-20T17:18:29.308 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:18:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:28 smithi171 ceph-mon[44189]: pgmap v882: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 2.9 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 270 op/s; 4829/93538 objects misplaced (5.163%); 811 KiB/s, 12 objects/s recovering 2024-04-20T17:18:29.337 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:29.391 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:18:29.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:29.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:18:29.496 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:18:29.522 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:29.545 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:18:29.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:29.548 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:18:29.648 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:18:29.675 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:29.696 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:18:29.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:29.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:18:29.750 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:18:29.753 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:18:29.799 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:18:29.826 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:29.845 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:18:29.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:29.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:18:29.949 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:18:29.977 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:29.996 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:18:29.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:30.000 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:18:30.101 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:18:30.129 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:30.147 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:18:30.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:30.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:18:30.251 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:18:30.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:29 smithi106 ceph-mon[44423]: 2.0 deep-scrub ok 2024-04-20T17:18:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:29 smithi106 ceph-mon[44423]: from='client.15654 v1:172.21.15.5:0/3939723130' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:18:30.280 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:29 smithi005 ceph-mon[39562]: 2.0 deep-scrub ok 2024-04-20T17:18:30.280 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:29 smithi005 ceph-mon[39562]: from='client.15654 v1:172.21.15.5:0/3939723130' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:18:30.283 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:30.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:29 smithi171 ceph-mon[44189]: 2.0 deep-scrub ok 2024-04-20T17:18:30.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:29 smithi171 ceph-mon[44189]: from='client.15654 v1:172.21.15.5:0/3939723130' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:18:30.941 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:18:30.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:30.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:18:30.963 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:18:31.046 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:18:31.077 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:31.207 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:30 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:18:31.207 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:30 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:18:31.207 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:30 smithi005 ceph-mon[39562]: osdmap e322: 12 total, 12 up, 12 in 2024-04-20T17:18:31.207 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:30 smithi005 ceph-mon[39562]: pgmap v884: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 2.8 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 191 op/s; 4675/92938 objects misplaced (5.030%); 810 KiB/s, 12 objects/s recovering 2024-04-20T17:18:31.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:30 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:18:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:30 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:18:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:30 smithi106 ceph-mon[44423]: osdmap e322: 12 total, 12 up, 12 in 2024-04-20T17:18:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:30 smithi106 ceph-mon[44423]: pgmap v884: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 2.8 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 191 op/s; 4675/92938 objects misplaced (5.030%); 810 KiB/s, 12 objects/s recovering 2024-04-20T17:18:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:30 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:18:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:30 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:18:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:30 smithi171 ceph-mon[44189]: osdmap e322: 12 total, 12 up, 12 in 2024-04-20T17:18:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:30 smithi171 ceph-mon[44189]: pgmap v884: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 2.8 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 191 op/s; 4675/92938 objects misplaced (5.030%); 810 KiB/s, 12 objects/s recovering 2024-04-20T17:18:32.091 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:18:32.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:32 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2313538952' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:18:32.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:32 smithi005 ceph-mon[39562]: from='client.25570 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:18:32.434 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:18:32.438 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:18:32.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:32.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:18:32.475 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":882,"stamp":"2024-04-20T17:18:28.070242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3065446968,"num_objects":46769,"num_object_clones":0,"num_object_copies":93538,"num_objects_missing_on_primary":2410,"num_objects_missing":2410,"num_objects_degraded":0,"num_objects_misplaced":4829,"num_objects_unfound":0,"num_objects_dirty":46769,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":275398,"num_write_kb":10306363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26805,"num_bytes_recovered":1758283904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161021,"ondisk_log_size":161021,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":7931648,"kb_used_data":6765624,"kb_used_omap":219,"kb_used_meta":1165668,"kb_avail":1117010176,"statfs":{"total":1151940427776,"available":1143818420224,"internally_reserved":0,"allocated":6927998976,"data_stored":6914761680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224949,"internal_metadata":1193644363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,8,1,2,3,4,4,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-106299392,"num_objects":-1622,"num_object_clones":0,"num_object_copies":-3244,"num_objects_missing_on_primary":-75,"num_objects_missing":-75,"num_objects_degraded":0,"num_objects_misplaced":-151,"num_objects_unfound":0,"num_objects_dirty":-1622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1622,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":4980736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001136"},"pg_stats":[{"pgid":"2.f","version":"319'17247","reported_seq":18215,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-04-20T17:18:23.410368+0000","last_change":"2024-04-20T17:17:55.070885+0000","last_active":"2024-04-20T17:18:23.410368+0000","last_peered":"2024-04-20T17:18:23.410368+0000","last_clean":"2024-04-20T17:18:23.410368+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:18:23.410368+0000","last_undegraded":"2024-04-20T17:18:23.410368+0000","last_fullsized":"2024-04-20T17:18:23.410368+0000","mapping_epoch":115,"log_start":"137'7200","ondisk_log_start":"137'7200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'16821","last_scrub_stamp":"2024-04-20T17:17:55.070768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:55.070768+0000","objects_scrubbed":3362,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:57.945378+0000","scrub_duration":356,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192348160,"num_objects":2935,"num_object_clones":0,"num_object_copies":5870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17247,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"320'17118","reported_seq":29153,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-04-20T17:18:25.378735+0000","last_change":"2024-04-20T17:18:08.306341+0000","last_active":"2024-04-20T17:18:25.378735+0000","last_peered":"2024-04-20T17:18:25.378735+0000","last_clean":"2024-04-20T17:18:25.378735+0000","last_became_active":"2024-04-20T17:10:28.865670+0000","last_became_peered":"2024-04-20T17:10:28.865670+0000","last_unstale":"2024-04-20T17:18:25.378735+0000","last_undegraded":"2024-04-20T17:18:25.378735+0000","last_fullsized":"2024-04-20T17:18:25.378735+0000","mapping_epoch":162,"log_start":"137'7100","ondisk_log_start":"137'7100","created":115,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'16860","last_scrub_stamp":"2024-04-20T17:18:08.306156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:08.306156+0000","objects_scrubbed":3197,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:34.841351+0000","scrub_duration":281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192544768,"num_objects":2938,"num_object_clones":0,"num_object_copies":5876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17118,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5619,"num_bytes_recovered":368246784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"320'17483","reported_seq":18300,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-04-20T17:18:25.378068+0000","last_change":"2024-04-20T17:17:12.348489+0000","last_active":"2024-04-20T17:18:25.378068+0000","last_peered":"2024-04-20T17:18:25.378068+0000","last_clean":"2024-04-20T17:18:25.378068+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:18:25.378068+0000","last_undegraded":"2024-04-20T17:18:25.378068+0000","last_fullsized":"2024-04-20T17:18:25.378068+0000","mapping_epoch":115,"log_start":"138'7400","ondisk_log_start":"138'7400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'16554","last_scrub_stamp":"2024-04-20T17:17:12.348391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:12.348391+0000","objects_scrubbed":3930,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:33.813428+0000","scrub_duration":337,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196542464,"num_objects":2999,"num_object_clones":0,"num_object_copies":5998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17483,"num_write_kb":655424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"320'23887","reported_seq":45500,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-04-20T17:18:25.664856+0000","last_change":"2024-04-20T17:17:53.095109+0000","last_active":"2024-04-20T17:18:25.664856+0000","last_peered":"2024-04-20T17:18:25.664856+0000","last_clean":"2024-04-20T17:18:25.664856+0000","last_became_active":"2024-04-20T17:12:08.085936+0000","last_became_peered":"2024-04-20T17:12:08.085936+0000","last_unstale":"2024-04-20T17:18:25.664856+0000","last_undegraded":"2024-04-20T17:18:25.664856+0000","last_fullsized":"2024-04-20T17:18:25.664856+0000","mapping_epoch":196,"log_start":"207'13868","ondisk_log_start":"207'13868","created":115,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'23389","last_scrub_stamp":"2024-04-20T17:17:53.094985+0000","last_deep_scrub":"213'20373","last_deep_scrub_stamp":"2024-04-20T17:12:48.443808+0000","last_clean_scrub_stamp":"2024-04-20T17:17:53.094985+0000","objects_scrubbed":3498,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:20.542939+0000","scrub_duration":306,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196542464,"num_objects":2999,"num_object_clones":0,"num_object_copies":5998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17235,"num_write_kb":647488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":450166784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"319'16882","reported_seq":38355,"reported_epoch":319,"state":"active+recovering","last_fresh":"2024-04-20T17:18:23.372178+0000","last_change":"2024-04-20T17:12:26.166109+0000","last_active":"2024-04-20T17:18:23.372178+0000","last_peered":"2024-04-20T17:18:23.372178+0000","last_clean":"2024-04-20T17:09:35.882603+0000","last_became_active":"2024-04-20T17:10:54.470253+0000","last_became_peered":"2024-04-20T17:10:54.470253+0000","last_unstale":"2024-04-20T17:18:23.372178+0000","last_undegraded":"2024-04-20T17:18:23.372178+0000","last_fullsized":"2024-04-20T17:18:23.372178+0000","mapping_epoch":171,"log_start":"136'6800","ondisk_log_start":"136'6800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186646528,"num_objects":2848,"num_object_clones":0,"num_object_copies":5696,"num_objects_missing_on_primary":2410,"num_objects_missing":2410,"num_objects_degraded":0,"num_objects_misplaced":4829,"num_objects_unfound":0,"num_objects_dirty":2848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16882,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7013,"num_bytes_recovered":459603968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"0,2,5","objects":9},{"shards":"0,10","objects":429},{"shards":"2,5","objects":2410}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":846,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-20T17:18:27.000256+0000","last_change":"2024-04-20T17:17:58.199584+0000","last_active":"2024-04-20T17:18:27.000256+0000","last_peered":"2024-04-20T17:18:27.000256+0000","last_clean":"2024-04-20T17:18:27.000256+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:18:27.000256+0000","last_undegraded":"2024-04-20T17:18:27.000256+0000","last_fullsized":"2024-04-20T17:18:27.000256+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:17:58.199502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:17:58.199502+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:16.100652+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"321'20074","reported_seq":21016,"reported_epoch":321,"state":"active+clean+scrubbing+deep","last_fresh":"2024-04-20T17:18:27.877890+0000","last_change":"2024-04-20T17:18:19.267298+0000","last_active":"2024-04-20T17:18:27.877890+0000","last_peered":"2024-04-20T17:18:27.877890+0000","last_clean":"2024-04-20T17:18:27.877890+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:18:27.877890+0000","last_undegraded":"2024-04-20T17:18:27.877890+0000","last_fullsized":"2024-04-20T17:18:27.877890+0000","mapping_epoch":115,"log_start":"152'10000","ondisk_log_start":"152'10000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'16300","last_scrub_stamp":"2024-04-20T17:17:10.481666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:10.481666+0000","objects_scrubbed":2774,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"deep scrubbing for 8s","scrub_duration":337,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193462272,"num_objects":2952,"num_object_clones":0,"num_object_copies":5904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17300,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"319'17400","reported_seq":18245,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-04-20T17:18:23.715217+0000","last_change":"2024-04-20T17:17:15.112506+0000","last_active":"2024-04-20T17:18:23.715217+0000","last_peered":"2024-04-20T17:18:23.715217+0000","last_clean":"2024-04-20T17:18:23.715217+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:18:23.715217+0000","last_undegraded":"2024-04-20T17:18:23.715217+0000","last_fullsized":"2024-04-20T17:18:23.715217+0000","mapping_epoch":115,"log_start":"137'7300","ondisk_log_start":"137'7300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'16484","last_scrub_stamp":"2024-04-20T17:17:15.112253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:15.112253+0000","objects_scrubbed":3848,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:45.026807+0000","scrub_duration":457,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191627264,"num_objects":2924,"num_object_clones":0,"num_object_copies":5848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17400,"num_write_kb":650368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"320'21308","reported_seq":22296,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-04-20T17:18:26.859814+0000","last_change":"2024-04-20T17:17:55.709069+0000","last_active":"2024-04-20T17:18:26.859814+0000","last_peered":"2024-04-20T17:18:26.859814+0000","last_clean":"2024-04-20T17:18:26.859814+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:18:26.859814+0000","last_undegraded":"2024-04-20T17:18:26.859814+0000","last_fullsized":"2024-04-20T17:18:26.859814+0000","mapping_epoch":115,"log_start":"156'11302","ondisk_log_start":"156'11302","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'20878","last_scrub_stamp":"2024-04-20T17:17:55.708833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:55.708833+0000","objects_scrubbed":3275,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:11.407326+0000","scrub_duration":320,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186449920,"num_objects":2845,"num_object_clones":0,"num_object_copies":5690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17197,"num_write_kb":641344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"321'17330","reported_seq":18236,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-20T17:18:27.382157+0000","last_change":"2024-04-20T17:17:08.025358+0000","last_active":"2024-04-20T17:18:27.382157+0000","last_peered":"2024-04-20T17:18:27.382157+0000","last_clean":"2024-04-20T17:18:27.382157+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:18:27.382157+0000","last_undegraded":"2024-04-20T17:18:27.382157+0000","last_fullsized":"2024-04-20T17:18:27.382157+0000","mapping_epoch":115,"log_start":"138'7300","ondisk_log_start":"138'7300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'16358","last_scrub_stamp":"2024-04-20T17:17:08.025214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:08.025214+0000","objects_scrubbed":3866,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:32.370368+0000","scrub_duration":359,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189399040,"num_objects":2890,"num_object_clones":0,"num_object_copies":5780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17330,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"321'17160","reported_seq":17992,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-20T17:18:27.865281+0000","last_change":"2024-04-20T17:17:16.522161+0000","last_active":"2024-04-20T17:18:27.865281+0000","last_peered":"2024-04-20T17:18:27.865281+0000","last_clean":"2024-04-20T17:18:27.865281+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:18:27.865281+0000","last_undegraded":"2024-04-20T17:18:27.865281+0000","last_fullsized":"2024-04-20T17:18:27.865281+0000","mapping_epoch":115,"log_start":"137'7100","ondisk_log_start":"137'7100","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'16253","last_scrub_stamp":"2024-04-20T17:17:16.522032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:16.522032+0000","objects_scrubbed":3799,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:29.552786+0000","scrub_duration":315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189530112,"num_objects":2892,"num_object_clones":0,"num_object_copies":5784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17160,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"321'17348","reported_seq":32781,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-20T17:18:27.798676+0000","last_change":"2024-04-20T17:17:57.501217+0000","last_active":"2024-04-20T17:18:27.798676+0000","last_peered":"2024-04-20T17:18:27.798676+0000","last_clean":"2024-04-20T17:18:27.798676+0000","last_became_active":"2024-04-20T17:10:55.931751+0000","last_became_peered":"2024-04-20T17:10:55.931751+0000","last_unstale":"2024-04-20T17:18:27.798676+0000","last_undegraded":"2024-04-20T17:18:27.798676+0000","last_fullsized":"2024-04-20T17:18:27.798676+0000","mapping_epoch":173,"log_start":"138'7300","ondisk_log_start":"138'7300","created":115,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'16916","last_scrub_stamp":"2024-04-20T17:17:57.501089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:57.501089+0000","objects_scrubbed":3350,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:26.178414+0000","scrub_duration":318,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190971904,"num_objects":2914,"num_object_clones":0,"num_object_copies":5828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17348,"num_write_kb":648384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7296,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"321'17384","reported_seq":18273,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-20T17:18:27.234385+0000","last_change":"2024-04-20T17:18:06.314140+0000","last_active":"2024-04-20T17:18:27.234385+0000","last_peered":"2024-04-20T17:18:27.234385+0000","last_clean":"2024-04-20T17:18:27.234385+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:18:27.234385+0000","last_undegraded":"2024-04-20T17:18:27.234385+0000","last_fullsized":"2024-04-20T17:18:27.234385+0000","mapping_epoch":115,"log_start":"138'7300","ondisk_log_start":"138'7300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"314'17127","last_scrub_stamp":"2024-04-20T17:18:06.313989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:06.313989+0000","objects_scrubbed":3191,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:11.886992+0000","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192151552,"num_objects":2932,"num_object_clones":0,"num_object_copies":5864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17384,"num_write_kb":650112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"319'16884","reported_seq":17726,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-04-20T17:18:23.729657+0000","last_change":"2024-04-20T17:17:06.994658+0000","last_active":"2024-04-20T17:18:23.729657+0000","last_peered":"2024-04-20T17:18:23.729657+0000","last_clean":"2024-04-20T17:18:23.729657+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:18:23.729657+0000","last_undegraded":"2024-04-20T17:18:23.729657+0000","last_fullsized":"2024-04-20T17:18:23.729657+0000","mapping_epoch":115,"log_start":"136'6800","ondisk_log_start":"136'6800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'15884","last_scrub_stamp":"2024-04-20T17:17:06.994455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:06.994455+0000","objects_scrubbed":3982,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195297280,"num_objects":2980,"num_object_clones":0,"num_object_copies":5960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16884,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"321'17214","reported_seq":18059,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-20T17:18:27.042921+0000","last_change":"2024-04-20T17:17:07.882718+0000","last_active":"2024-04-20T17:18:27.042921+0000","last_peered":"2024-04-20T17:18:27.042921+0000","last_clean":"2024-04-20T17:18:27.042921+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:18:27.042921+0000","last_undegraded":"2024-04-20T17:18:27.042921+0000","last_fullsized":"2024-04-20T17:18:27.042921+0000","mapping_epoch":115,"log_start":"138'7200","ondisk_log_start":"138'7200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'16236","last_scrub_stamp":"2024-04-20T17:17:07.882596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:07.882596+0000","objects_scrubbed":3868,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:32.661323+0000","scrub_duration":370,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189267968,"num_objects":2888,"num_object_clones":0,"num_object_copies":5776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17214,"num_write_kb":643264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"320'16990","reported_seq":17834,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-04-20T17:18:25.575687+0000","last_change":"2024-04-20T17:17:04.896764+0000","last_active":"2024-04-20T17:18:25.575687+0000","last_peered":"2024-04-20T17:18:25.575687+0000","last_clean":"2024-04-20T17:18:25.575687+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:18:25.575687+0000","last_undegraded":"2024-04-20T17:18:25.575687+0000","last_fullsized":"2024-04-20T17:18:25.575687+0000","mapping_epoch":115,"log_start":"136'6900","ondisk_log_start":"136'6900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"293'15941","last_scrub_stamp":"2024-04-20T17:17:04.896640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:04.896640+0000","objects_scrubbed":3929,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188678168,"num_objects":2880,"num_object_clones":0,"num_object_copies":5760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2880,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16990,"num_write_kb":635777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"320'17045","reported_seq":17800,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-04-20T17:18:25.729596+0000","last_change":"2024-04-20T17:18:17.426012+0000","last_active":"2024-04-20T17:18:25.729596+0000","last_peered":"2024-04-20T17:18:25.729596+0000","last_clean":"2024-04-20T17:18:25.729596+0000","last_became_active":"2024-04-20T17:12:23.515375+0000","last_became_peered":"2024-04-20T17:12:23.515375+0000","last_unstale":"2024-04-20T17:18:25.729596+0000","last_undegraded":"2024-04-20T17:18:25.729596+0000","last_fullsized":"2024-04-20T17:18:25.729596+0000","mapping_epoch":202,"log_start":"136'7000","ondisk_log_start":"136'7000","created":115,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'16928","last_scrub_stamp":"2024-04-20T17:18:17.425878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:17.425878+0000","objects_scrubbed":3071,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:47.218525+0000","scrub_duration":353,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193396736,"num_objects":2951,"num_object_clones":0,"num_object_copies":5902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17045,"num_write_kb":639872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":3064856600,"num_objects":46767,"num_object_clones":0,"num_object_copies":93534,"num_objects_missing_on_primary":2410,"num_objects_missing":2410,"num_objects_degraded":0,"num_objects_misplaced":4829,"num_objects_unfound":0,"num_objects_dirty":46767,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":275217,"num_write_kb":10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26797,"num_bytes_recovered":1756168192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6884564992,"data_stored":6884556848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},"log_size":160884,"ondisk_log_size":160884,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991385,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":630216,"kb_used_data":562940,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93114936,"statfs":{"total":95995035648,"available":95349694464,"internally_reserved":0,"allocated":576450560,"data_stored":575339226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":68876740},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253022,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":781988,"kb_used_data":602268,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":92963164,"statfs":{"total":95995035648,"available":95194279936,"internally_reserved":0,"allocated":616722432,"data_stored":615626506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":184023488},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612772,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":886200,"kb_used_data":750892,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":92858952,"statfs":{"total":95995035648,"available":95087566848,"internally_reserved":0,"allocated":768913408,"data_stored":767807502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":138541504},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809001,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":868968,"kb_used_data":757276,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92876184,"statfs":{"total":95995035648,"available":95105212416,"internally_reserved":0,"allocated":775450624,"data_stored":774354718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168751,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":635464,"kb_used_data":564028,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93109688,"statfs":{"total":95995035648,"available":95344320512,"internally_reserved":0,"allocated":577564672,"data_stored":576453358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430387,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":481088,"kb_used_data":377716,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93264064,"statfs":{"total":95995035648,"available":95502401536,"internally_reserved":0,"allocated":386781184,"data_stored":385673422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575053,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":488240,"kb_used_data":376552,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93256912,"statfs":{"total":95995035648,"available":95495077888,"internally_reserved":0,"allocated":385589248,"data_stored":384487414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738577,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":741992,"kb_used_data":598364,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93003160,"statfs":{"total":95995035648,"available":95235235840,"internally_reserved":0,"allocated":612724736,"data_stored":611628830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,1,2,3,4,4,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393866,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":657652,"kb_used_data":621736,"kb_used_omap":202,"kb_used_meta":35701,"kb_avail":93087500,"statfs":{"total":95995035648,"available":95321600000,"internally_reserved":0,"allocated":636657664,"data_stored":635555830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":36558295},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182594,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":442612,"kb_used_data":379496,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93302540,"statfs":{"total":95995035648,"available":95541800960,"internally_reserved":0,"allocated":388603904,"data_stored":387502070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986365,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":218552,"kb_used_data":187372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93526600,"statfs":{"total":95995035648,"available":95771238400,"internally_reserved":0,"allocated":191868928,"data_stored":190763022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790135,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1098676,"kb_used_data":986984,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92646476,"statfs":{"total":95995035648,"available":94869991424,"internally_reserved":0,"allocated":1010671616,"data_stored":1009569782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":608043008,"data_stored":608043008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":381681664,"data_stored":381681664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":633602048,"data_stored":633602048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":386269184,"data_stored":386269184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":188878848,"data_stored":188874776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1007484928,"data_stored":1007484928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":384368640,"data_stored":384368640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":573964288,"data_stored":573964288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":768999424,"data_stored":768999424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":766513152,"data_stored":766509080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":611713024,"data_stored":611713024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":573046784,"data_stored":573046784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:18:32.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:32 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2313538952' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:18:32.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:32 smithi106 ceph-mon[44423]: from='client.25570 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:18:32.540 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:18:32.571 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:32 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2313538952' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:18:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:32 smithi171 ceph-mon[44189]: from='client.25570 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:18:32.921 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:18:32.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:32.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:18:33.024 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:18:33.051 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:33.070 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:18:33.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:33.072 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:18:33.173 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:18:33.204 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:33 smithi005 ceph-mon[39562]: pgmap v885: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 2.8 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 203 op/s; 4675/92034 objects misplaced (5.080%); 618 KiB/s, 9 objects/s recovering 2024-04-20T17:18:33.204 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:33 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:18:33.204 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:33 smithi005 ceph-mon[39562]: from='client.25570 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:18:33.204 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:33 smithi005 ceph-mon[39562]: osdmap e323: 12 total, 12 up, 12 in 2024-04-20T17:18:33.206 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:33.239 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:18:33.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:33.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:18:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:33 smithi171 ceph-mon[44189]: pgmap v885: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 2.8 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 203 op/s; 4675/92034 objects misplaced (5.080%); 618 KiB/s, 9 objects/s recovering 2024-04-20T17:18:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:33 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:18:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:33 smithi171 ceph-mon[44189]: from='client.25570 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:18:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:33 smithi171 ceph-mon[44189]: osdmap e323: 12 total, 12 up, 12 in 2024-04-20T17:18:33.350 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:18:33.375 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:33 smithi106 ceph-mon[44423]: pgmap v885: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 2.8 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 203 op/s; 4675/92034 objects misplaced (5.080%); 618 KiB/s, 9 objects/s recovering 2024-04-20T17:18:33.375 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:33 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:18:33.376 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:33 smithi106 ceph-mon[44423]: from='client.25570 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:18:33.376 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:33 smithi106 ceph-mon[44423]: osdmap e323: 12 total, 12 up, 12 in 2024-04-20T17:18:33.378 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:33.398 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:18:33.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:33.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:18:33.505 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:18:33.533 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:33.555 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:18:33.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:33.559 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:18:33.660 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:18:33.688 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:33.716 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:18:33.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:33.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:18:33.821 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:18:33.847 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:33.865 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:18:33.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:33.868 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:18:33.969 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:18:34.000 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:34.450 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:18:34.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:34.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:18:34.557 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:18:34.588 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:35 smithi005 ceph-mon[39562]: pgmap v887: 17 pgs: 1 active+recovering, 16 active+clean; 2.8 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 281 op/s; 4518/90478 objects misplaced (4.993%); 1.4 MiB/s, 21 objects/s recovering 2024-04-20T17:18:35.478 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:18:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:35 smithi106 ceph-mon[44423]: pgmap v887: 17 pgs: 1 active+recovering, 16 active+clean; 2.8 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 281 op/s; 4518/90478 objects misplaced (4.993%); 1.4 MiB/s, 21 objects/s recovering 2024-04-20T17:18:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:35 smithi171 ceph-mon[44189]: pgmap v887: 17 pgs: 1 active+recovering, 16 active+clean; 2.8 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 281 op/s; 4518/90478 objects misplaced (4.993%); 1.4 MiB/s, 21 objects/s recovering 2024-04-20T17:18:35.674 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:18:35.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:35.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:18:35.779 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:18:35.804 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:35.822 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:18:35.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:35.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:18:35.926 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:18:35.952 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:35.971 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:18:35.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:35.974 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:18:36.075 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:18:36.102 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:36.125 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:18:36.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:36.128 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:18:36.229 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:18:36.256 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:36.274 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:18:36.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:36.277 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:18:36.378 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:18:36.407 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:36.460 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:18:36.507 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:18:36.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:36.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:18:36.611 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:18:36.639 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:36.658 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:18:36.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:36.661 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:18:36.762 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:18:36.787 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:36.805 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:18:36.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:36.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:18:36.909 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:18:36.934 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:36.954 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:18:36.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:36.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:18:37.060 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:18:37.086 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:37.105 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:18:37.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:37.108 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:18:37.158 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:18:37.158 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:18:37.163 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:18:37.185 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:18:37.206 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:18:37.208 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:18:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:36 smithi005 ceph-mon[39562]: osdmap e324: 12 total, 12 up, 12 in 2024-04-20T17:18:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:36 smithi005 ceph-mon[39562]: pgmap v889: 17 pgs: 1 active+recovering, 16 active+clean; 2.7 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 278 op/s; 4518/89556 objects misplaced (5.045%); 831 KiB/s, 12 objects/s recovering 2024-04-20T17:18:37.230 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:18:37.233 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:36 smithi106 ceph-mon[44423]: osdmap e324: 12 total, 12 up, 12 in 2024-04-20T17:18:37.234 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:36 smithi106 ceph-mon[44423]: pgmap v889: 17 pgs: 1 active+recovering, 16 active+clean; 2.7 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 278 op/s; 4518/89556 objects misplaced (5.045%); 831 KiB/s, 12 objects/s recovering 2024-04-20T17:18:37.236 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:37.253 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:18:37.255 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:18:37.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:37.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:18:37.276 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:18:37.311 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:18:37.333 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:18:37.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:36 smithi171 ceph-mon[44189]: osdmap e324: 12 total, 12 up, 12 in 2024-04-20T17:18:37.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:36 smithi171 ceph-mon[44189]: pgmap v889: 17 pgs: 1 active+recovering, 16 active+clean; 2.7 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 278 op/s; 4518/89556 objects misplaced (5.045%); 831 KiB/s, 12 objects/s recovering 2024-04-20T17:18:37.356 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:18:37.359 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:18:37.390 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:37.391 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:18:37.393 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:18:37.393 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:18:37.393 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:18:37.394 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:18:37.394 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:18:37.394 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:18:37.394 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:18:37.394 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:18:37.394 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:18:37.394 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:18:37.394 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:18:37.394 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:18:37.414 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:18:37.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:37.417 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:18:37.518 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:18:37.534 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:18:37.534 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:18:37.535 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:18:37.535 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:278317 2.2 e324/115 2:42564a0d:::benchmark_data_smithi171_78918_object117323:head v 324'17688, mlcod=324'17686)", 2024-04-20T17:18:37.535 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:18:37.532802+0000", 2024-04-20T17:18:37.535 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.002401257, 2024-04-20T17:18:37.535 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002433167, 2024-04-20T17:18:37.535 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:18:37.535 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:18:37.535 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:18:37.535 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:18:37.535 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:18:37.535 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:18:37.535 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:37.532802+0000", 2024-04-20T17:18:37.535 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:18:37.535 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:18:37.535 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:18:37.536 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:18:37.536 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:37.532802+0000", 2024-04-20T17:18:37.536 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:18:37.536 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:18:37.536 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:18:37.536 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:18:37.536 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:37.532804+0000", 2024-04-20T17:18:37.536 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.199e-06 2024-04-20T17:18:37.536 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:18:37.536 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:18:37.536 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:18:37.536 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:37.532814+0000", 2024-04-20T17:18:37.536 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8269999999999993e-06 2024-04-20T17:18:37.536 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:18:37.536 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:18:37.536 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:18:37.536 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:37.532827+0000", 2024-04-20T17:18:37.537 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3645000000000001e-05 2024-04-20T17:18:37.537 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:18:37.537 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:18:37.537 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:18:37.537 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:37.532835+0000", 2024-04-20T17:18:37.537 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0900000000000005e-06 2024-04-20T17:18:37.537 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:18:37.537 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:18:37.537 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:18:37.537 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:37.532900+0000", 2024-04-20T17:18:37.537 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4799000000000001e-05 2024-04-20T17:18:37.537 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:18:37.537 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:18:37.537 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:18:37.537 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:37.532918+0000", 2024-04-20T17:18:37.537 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8230999999999999e-05 2024-04-20T17:18:37.537 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:18:37.538 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:18:37.538 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:18:37.538 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:18:37.538 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:18:37.538 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 1 2024-04-20T17:18:37.538 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:18:37.546 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:37.568 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:18:37.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:37.571 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:18:37.672 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:18:37.699 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:37.724 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:18:37.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:37.726 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:18:37.827 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:18:37.861 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:37.958 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:18:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:37 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3677777261' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:18:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:37 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3677777261' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:18:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:37 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3677777261' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:18:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:38.915 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:18:38.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:38.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:18:38.940 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:18:39.019 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:18:39.045 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:39.062 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:18:39.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:39.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:18:39.166 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:18:39.199 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:39.199 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:39.199 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:38 smithi005 ceph-mon[39562]: pgmap v890: 17 pgs: 1 active+recovering, 16 active+clean; 2.7 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 296 op/s; 4518/88192 objects misplaced (5.123%); 632 KiB/s, 9 objects/s recovering 2024-04-20T17:18:39.201 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:39.232 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:18:39.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:39.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:18:39.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:39.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:39.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:38 smithi106 ceph-mon[44423]: pgmap v890: 17 pgs: 1 active+recovering, 16 active+clean; 2.7 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 296 op/s; 4518/88192 objects misplaced (5.123%); 632 KiB/s, 9 objects/s recovering 2024-04-20T17:18:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:38 smithi171 ceph-mon[44189]: pgmap v890: 17 pgs: 1 active+recovering, 16 active+clean; 2.7 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 296 op/s; 4518/88192 objects misplaced (5.123%); 632 KiB/s, 9 objects/s recovering 2024-04-20T17:18:39.338 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:18:39.364 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:39.384 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:18:39.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:39.387 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:18:39.487 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:18:39.513 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:39.534 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:18:39.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:39.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:18:39.638 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:18:39.666 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:39.687 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:18:39.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:39.690 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:18:39.791 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:18:39.817 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:39.834 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:18:39.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:39.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:18:39.938 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:18:39.964 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:39.983 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:18:39.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:39.985 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:18:40.085 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-20T17:18:40.085 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:18:40.120 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:40.626 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:18:41.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:41 smithi005 ceph-mon[39562]: pgmap v891: 17 pgs: 1 active+recovering, 16 active+clean; 2.7 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 287 op/s; 4356/87438 objects misplaced (4.982%); 1.2 MiB/s, 19 objects/s recovering 2024-04-20T17:18:41.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:41 smithi106 ceph-mon[44423]: pgmap v891: 17 pgs: 1 active+recovering, 16 active+clean; 2.7 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 287 op/s; 4356/87438 objects misplaced (4.982%); 1.2 MiB/s, 19 objects/s recovering 2024-04-20T17:18:41.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:41 smithi171 ceph-mon[44189]: pgmap v891: 17 pgs: 1 active+recovering, 16 active+clean; 2.7 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 287 op/s; 4356/87438 objects misplaced (4.982%); 1.2 MiB/s, 19 objects/s recovering 2024-04-20T17:18:42.058 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:18:42.473 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:18:42.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:42.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:18:42.577 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:18:42.608 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:42.736 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:18:42.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:42.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:18:42.841 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:18:42.866 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:42.885 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:18:42.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:42.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:18:42.988 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:18:43.021 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:43.106 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:18:43.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:43.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:18:43.210 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:18:43.229 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:18:43.236 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:43.255 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:18:43.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:43.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:18:43.272 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:18:43.275 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:18:43.300 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:43 smithi005 ceph-mon[39562]: pgmap v892: 17 pgs: 1 active+recovering, 16 active+clean; 2.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 286 op/s; 4356/86310 objects misplaced (5.047%); 1013 KiB/s, 15 objects/s recovering 2024-04-20T17:18:43.358 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:18:43.382 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:43 smithi171 ceph-mon[44189]: pgmap v892: 17 pgs: 1 active+recovering, 16 active+clean; 2.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 286 op/s; 4356/86310 objects misplaced (5.047%); 1013 KiB/s, 15 objects/s recovering 2024-04-20T17:18:43.384 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:43.402 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:18:43.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:43.405 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:18:43.506 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:18:43.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:43 smithi106 ceph-mon[44423]: pgmap v892: 17 pgs: 1 active+recovering, 16 active+clean; 2.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 286 op/s; 4356/86310 objects misplaced (5.047%); 1013 KiB/s, 15 objects/s recovering 2024-04-20T17:18:43.532 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:43.551 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:18:43.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:43.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:18:43.655 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:18:43.687 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:44.038 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:18:44.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:44.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:18:44.143 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:18:44.148 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:18:44.170 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:44.191 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:18:44.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:44.195 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:18:44.296 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:18:44.323 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:44.343 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:18:44.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:44.349 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:18:44.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:44 smithi005 ceph-mon[39562]: from='client.25610 v1:172.21.15.5:0/3432498534' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:18:44.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:44 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1247347140' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:18:44.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:44 smithi005 ceph-mon[39562]: from='client.25616 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:18:44.450 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:18:44.481 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:44.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:44 smithi106 ceph-mon[44423]: from='client.25610 v1:172.21.15.5:0/3432498534' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:18:44.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:44 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1247347140' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:18:44.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:44 smithi106 ceph-mon[44423]: from='client.25616 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:18:44.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:44 smithi171 ceph-mon[44189]: from='client.25610 v1:172.21.15.5:0/3432498534' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:18:44.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:44 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1247347140' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:18:44.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:44 smithi171 ceph-mon[44189]: from='client.25616 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:18:44.691 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:18:44.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:44.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:18:44.795 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:18:44.821 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:44.840 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:18:44.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:44.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:18:44.943 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:18:44.971 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:45.220 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:18:45.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:45 smithi005 ceph-mon[39562]: pgmap v893: 17 pgs: 1 active+recovering, 16 active+clean; 2.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 290 op/s; 4188/84660 objects misplaced (4.947%); 1.0 MiB/s, 16 objects/s recovering 2024-04-20T17:18:45.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:45 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:18:45.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:45 smithi005 ceph-mon[39562]: from='client.25616 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:18:45.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:45 smithi005 ceph-mon[39562]: osdmap e325: 12 total, 12 up, 12 in 2024-04-20T17:18:45.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:45.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:18:45.324 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:18:45.352 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:45.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:45 smithi106 ceph-mon[44423]: pgmap v893: 17 pgs: 1 active+recovering, 16 active+clean; 2.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 290 op/s; 4188/84660 objects misplaced (4.947%); 1.0 MiB/s, 16 objects/s recovering 2024-04-20T17:18:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:45 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:18:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:45 smithi106 ceph-mon[44423]: from='client.25616 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:18:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:45 smithi106 ceph-mon[44423]: osdmap e325: 12 total, 12 up, 12 in 2024-04-20T17:18:45.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:45 smithi171 ceph-mon[44189]: pgmap v893: 17 pgs: 1 active+recovering, 16 active+clean; 2.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 290 op/s; 4188/84660 objects misplaced (4.947%); 1.0 MiB/s, 16 objects/s recovering 2024-04-20T17:18:45.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:45 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:18:45.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:45 smithi171 ceph-mon[44189]: from='client.25616 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:18:45.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:45 smithi171 ceph-mon[44189]: osdmap e325: 12 total, 12 up, 12 in 2024-04-20T17:18:45.682 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":892,"stamp":"2024-04-20T17:18:42.072682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2828599864,"num_objects":43155,"num_object_clones":0,"num_object_copies":86310,"num_objects_missing_on_primary":2171,"num_objects_missing":2171,"num_objects_degraded":0,"num_objects_misplaced":4356,"num_objects_unfound":0,"num_objects_dirty":43155,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":279012,"num_write_kb":10306363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27039,"num_bytes_recovered":1773619328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160967,"ondisk_log_size":160967,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":7520096,"kb_used_data":6327736,"kb_used_omap":219,"kb_used_meta":1192036,"kb_avail":1117421728,"statfs":{"total":1151940427776,"available":1144239849472,"internally_reserved":0,"allocated":6479601664,"data_stored":6466078624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224949,"internal_metadata":1220645195},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,0,0,1,2,4,7,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-187564032,"num_objects":-2862,"num_object_clones":0,"num_object_copies":-5724,"num_objects_missing_on_primary":-161,"num_objects_missing":-161,"num_objects_degraded":0,"num_objects_misplaced":-319,"num_objects_unfound":0,"num_objects_dirty":-2862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2862,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":10354688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.987036"},"pg_stats":[{"pgid":"2.f","version":"324'17487","reported_seq":18465,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-04-20T17:18:38.422793+0000","last_change":"2024-04-20T17:17:55.070885+0000","last_active":"2024-04-20T17:18:38.422793+0000","last_peered":"2024-04-20T17:18:38.422793+0000","last_clean":"2024-04-20T17:18:38.422793+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:18:38.422793+0000","last_undegraded":"2024-04-20T17:18:38.422793+0000","last_fullsized":"2024-04-20T17:18:38.422793+0000","mapping_epoch":115,"log_start":"138'7400","ondisk_log_start":"138'7400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'16821","last_scrub_stamp":"2024-04-20T17:17:55.070768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:55.070768+0000","objects_scrubbed":3362,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:57.945378+0000","scrub_duration":356,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176619520,"num_objects":2695,"num_object_clones":0,"num_object_copies":5390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17487,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"324'17368","reported_seq":29411,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-04-20T17:18:40.391146+0000","last_change":"2024-04-20T17:18:08.306341+0000","last_active":"2024-04-20T17:18:40.391146+0000","last_peered":"2024-04-20T17:18:40.391146+0000","last_clean":"2024-04-20T17:18:40.391146+0000","last_became_active":"2024-04-20T17:10:28.865670+0000","last_became_peered":"2024-04-20T17:10:28.865670+0000","last_unstale":"2024-04-20T17:18:40.391146+0000","last_undegraded":"2024-04-20T17:18:40.391146+0000","last_fullsized":"2024-04-20T17:18:40.391146+0000","mapping_epoch":162,"log_start":"138'7300","ondisk_log_start":"138'7300","created":115,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'16860","last_scrub_stamp":"2024-04-20T17:18:08.306156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:08.306156+0000","objects_scrubbed":3197,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:34.841351+0000","scrub_duration":281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176160768,"num_objects":2688,"num_object_clones":0,"num_object_copies":5376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17368,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5619,"num_bytes_recovered":368246784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"324'17740","reported_seq":18566,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-04-20T17:18:40.344097+0000","last_change":"2024-04-20T17:17:12.348489+0000","last_active":"2024-04-20T17:18:40.344097+0000","last_peered":"2024-04-20T17:18:40.344097+0000","last_clean":"2024-04-20T17:18:40.344097+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:18:40.344097+0000","last_undegraded":"2024-04-20T17:18:40.344097+0000","last_fullsized":"2024-04-20T17:18:40.344097+0000","mapping_epoch":115,"log_start":"139'7700","ondisk_log_start":"139'7700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'16554","last_scrub_stamp":"2024-04-20T17:17:12.348391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:12.348391+0000","objects_scrubbed":3930,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:41.093694+0000","scrub_duration":337,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179699712,"num_objects":2742,"num_object_clones":0,"num_object_copies":5484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17740,"num_write_kb":655424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"324'24135","reported_seq":45756,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-04-20T17:18:40.551442+0000","last_change":"2024-04-20T17:17:53.095109+0000","last_active":"2024-04-20T17:18:40.551442+0000","last_peered":"2024-04-20T17:18:40.551442+0000","last_clean":"2024-04-20T17:18:40.551442+0000","last_became_active":"2024-04-20T17:12:08.085936+0000","last_became_peered":"2024-04-20T17:12:08.085936+0000","last_unstale":"2024-04-20T17:18:40.551442+0000","last_undegraded":"2024-04-20T17:18:40.551442+0000","last_fullsized":"2024-04-20T17:18:40.551442+0000","mapping_epoch":196,"log_start":"207'14068","ondisk_log_start":"207'14068","created":115,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'23389","last_scrub_stamp":"2024-04-20T17:17:53.094985+0000","last_deep_scrub":"213'20373","last_deep_scrub_stamp":"2024-04-20T17:12:48.443808+0000","last_clean_scrub_stamp":"2024-04-20T17:17:53.094985+0000","objects_scrubbed":3498,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:20.542939+0000","scrub_duration":306,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180289536,"num_objects":2751,"num_object_clones":0,"num_object_copies":5502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17483,"num_write_kb":647488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":450166784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"324'17116","reported_seq":39306,"reported_epoch":324,"state":"active+recovering","last_fresh":"2024-04-20T17:18:38.553866+0000","last_change":"2024-04-20T17:12:26.166109+0000","last_active":"2024-04-20T17:18:38.553866+0000","last_peered":"2024-04-20T17:18:38.553866+0000","last_clean":"2024-04-20T17:09:35.882603+0000","last_became_active":"2024-04-20T17:10:54.470253+0000","last_became_peered":"2024-04-20T17:10:54.470253+0000","last_unstale":"2024-04-20T17:18:38.553866+0000","last_undegraded":"2024-04-20T17:18:38.553866+0000","last_fullsized":"2024-04-20T17:18:38.553866+0000","mapping_epoch":171,"log_start":"137'7100","ondisk_log_start":"137'7100","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171311104,"num_objects":2614,"num_object_clones":0,"num_object_copies":5228,"num_objects_missing_on_primary":2171,"num_objects_missing":2171,"num_objects_degraded":0,"num_objects_misplaced":4356,"num_objects_unfound":0,"num_objects_dirty":2614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17116,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7247,"num_bytes_recovered":474939392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"0,2,5","objects":14},{"shards":"0,10","objects":429},{"shards":"2,5","objects":2171}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":852,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-04-20T17:18:36.015491+0000","last_change":"2024-04-20T17:17:58.199584+0000","last_active":"2024-04-20T17:18:36.015491+0000","last_peered":"2024-04-20T17:18:36.015491+0000","last_clean":"2024-04-20T17:18:36.015491+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:18:36.015491+0000","last_undegraded":"2024-04-20T17:18:36.015491+0000","last_fullsized":"2024-04-20T17:18:36.015491+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:17:58.199502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:17:58.199502+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:16.100652+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"324'20501","reported_seq":21487,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-04-20T17:18:37.908142+0000","last_change":"2024-04-20T17:18:28.829916+0000","last_active":"2024-04-20T17:18:37.908142+0000","last_peered":"2024-04-20T17:18:37.908142+0000","last_clean":"2024-04-20T17:18:37.908142+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:18:37.908142+0000","last_undegraded":"2024-04-20T17:18:37.908142+0000","last_fullsized":"2024-04-20T17:18:37.908142+0000","mapping_epoch":115,"log_start":"153'10500","ondisk_log_start":"153'10500","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'20320","last_scrub_stamp":"2024-04-20T17:18:28.829770+0000","last_deep_scrub":"321'20320","last_deep_scrub_stamp":"2024-04-20T17:18:28.829770+0000","last_clean_scrub_stamp":"2024-04-20T17:18:28.829770+0000","objects_scrubbed":3006,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":10,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:40.227886+0000","scrub_duration":9561,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180682752,"num_objects":2757,"num_object_clones":0,"num_object_copies":5514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17495,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"324'17669","reported_seq":18525,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-04-20T17:18:38.592794+0000","last_change":"2024-04-20T17:17:15.112506+0000","last_active":"2024-04-20T17:18:38.592794+0000","last_peered":"2024-04-20T17:18:38.592794+0000","last_clean":"2024-04-20T17:18:38.592794+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:18:38.592794+0000","last_undegraded":"2024-04-20T17:18:38.592794+0000","last_fullsized":"2024-04-20T17:18:38.592794+0000","mapping_epoch":115,"log_start":"139'7600","ondisk_log_start":"139'7600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'16484","last_scrub_stamp":"2024-04-20T17:17:15.112253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:15.112253+0000","objects_scrubbed":3848,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:45.026807+0000","scrub_duration":457,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173998080,"num_objects":2655,"num_object_clones":0,"num_object_copies":5310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17669,"num_write_kb":650368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"324'21581","reported_seq":22577,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-04-20T17:18:41.970579+0000","last_change":"2024-04-20T17:17:55.709069+0000","last_active":"2024-04-20T17:18:41.970579+0000","last_peered":"2024-04-20T17:18:41.970579+0000","last_clean":"2024-04-20T17:18:41.970579+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:18:41.970579+0000","last_undegraded":"2024-04-20T17:18:41.970579+0000","last_fullsized":"2024-04-20T17:18:41.970579+0000","mapping_epoch":115,"log_start":"157'11502","ondisk_log_start":"157'11502","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'20878","last_scrub_stamp":"2024-04-20T17:17:55.708833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:55.708833+0000","objects_scrubbed":3275,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:11.407326+0000","scrub_duration":320,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168558592,"num_objects":2572,"num_object_clones":0,"num_object_copies":5144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17470,"num_write_kb":641344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"324'17498","reported_seq":18409,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-04-20T17:18:37.493292+0000","last_change":"2024-04-20T17:17:08.025358+0000","last_active":"2024-04-20T17:18:37.493292+0000","last_peered":"2024-04-20T17:18:37.493292+0000","last_clean":"2024-04-20T17:18:37.493292+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:18:37.493292+0000","last_undegraded":"2024-04-20T17:18:37.493292+0000","last_fullsized":"2024-04-20T17:18:37.493292+0000","mapping_epoch":115,"log_start":"138'7400","ondisk_log_start":"138'7400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'16358","last_scrub_stamp":"2024-04-20T17:17:08.025214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:08.025214+0000","objects_scrubbed":3866,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":359,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178388992,"num_objects":2722,"num_object_clones":0,"num_object_copies":5444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17498,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"324'17331","reported_seq":18169,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-04-20T17:18:37.831024+0000","last_change":"2024-04-20T17:17:16.522161+0000","last_active":"2024-04-20T17:18:37.831024+0000","last_peered":"2024-04-20T17:18:37.831024+0000","last_clean":"2024-04-20T17:18:37.831024+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:18:37.831024+0000","last_undegraded":"2024-04-20T17:18:37.831024+0000","last_fullsized":"2024-04-20T17:18:37.831024+0000","mapping_epoch":115,"log_start":"138'7300","ondisk_log_start":"138'7300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'16253","last_scrub_stamp":"2024-04-20T17:17:16.522032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:16.522032+0000","objects_scrubbed":3799,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:38.339395+0000","scrub_duration":315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178323456,"num_objects":2721,"num_object_clones":0,"num_object_copies":5442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17331,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"324'17541","reported_seq":32980,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-04-20T17:18:37.866241+0000","last_change":"2024-04-20T17:17:57.501217+0000","last_active":"2024-04-20T17:18:37.866241+0000","last_peered":"2024-04-20T17:18:37.866241+0000","last_clean":"2024-04-20T17:18:37.866241+0000","last_became_active":"2024-04-20T17:10:55.931751+0000","last_became_peered":"2024-04-20T17:10:55.931751+0000","last_unstale":"2024-04-20T17:18:37.866241+0000","last_undegraded":"2024-04-20T17:18:37.866241+0000","last_fullsized":"2024-04-20T17:18:37.866241+0000","mapping_epoch":173,"log_start":"139'7500","ondisk_log_start":"139'7500","created":115,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'16916","last_scrub_stamp":"2024-04-20T17:17:57.501089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:57.501089+0000","objects_scrubbed":3350,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:26.178414+0000","scrub_duration":318,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178323456,"num_objects":2721,"num_object_clones":0,"num_object_copies":5442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17541,"num_write_kb":648384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7296,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"324'17559","reported_seq":18454,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-04-20T17:18:37.526867+0000","last_change":"2024-04-20T17:18:06.314140+0000","last_active":"2024-04-20T17:18:37.526867+0000","last_peered":"2024-04-20T17:18:37.526867+0000","last_clean":"2024-04-20T17:18:37.526867+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:18:37.526867+0000","last_undegraded":"2024-04-20T17:18:37.526867+0000","last_fullsized":"2024-04-20T17:18:37.526867+0000","mapping_epoch":115,"log_start":"139'7500","ondisk_log_start":"139'7500","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"314'17127","last_scrub_stamp":"2024-04-20T17:18:06.313989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:06.313989+0000","objects_scrubbed":3191,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:11.886992+0000","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180682752,"num_objects":2757,"num_object_clones":0,"num_object_copies":5514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17559,"num_write_kb":650112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"324'17143","reported_seq":17995,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-04-20T17:18:38.561129+0000","last_change":"2024-04-20T17:17:06.994658+0000","last_active":"2024-04-20T17:18:38.561129+0000","last_peered":"2024-04-20T17:18:38.561129+0000","last_clean":"2024-04-20T17:18:38.561129+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:18:38.561129+0000","last_undegraded":"2024-04-20T17:18:38.561129+0000","last_fullsized":"2024-04-20T17:18:38.561129+0000","mapping_epoch":115,"log_start":"138'7100","ondisk_log_start":"138'7100","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'15884","last_scrub_stamp":"2024-04-20T17:17:06.994455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:06.994455+0000","objects_scrubbed":3982,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178323456,"num_objects":2721,"num_object_clones":0,"num_object_copies":5442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17143,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"324'17362","reported_seq":18213,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-04-20T17:18:37.532306+0000","last_change":"2024-04-20T17:17:07.882718+0000","last_active":"2024-04-20T17:18:37.532306+0000","last_peered":"2024-04-20T17:18:37.532306+0000","last_clean":"2024-04-20T17:18:37.532306+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:18:37.532306+0000","last_undegraded":"2024-04-20T17:18:37.532306+0000","last_fullsized":"2024-04-20T17:18:37.532306+0000","mapping_epoch":115,"log_start":"138'7300","ondisk_log_start":"138'7300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'16236","last_scrub_stamp":"2024-04-20T17:17:07.882596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:07.882596+0000","objects_scrubbed":3868,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:39.264672+0000","scrub_duration":370,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179568640,"num_objects":2740,"num_object_clones":0,"num_object_copies":5480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17362,"num_write_kb":643264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"324'17254","reported_seq":18106,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-04-20T17:18:40.514299+0000","last_change":"2024-04-20T17:17:04.896764+0000","last_active":"2024-04-20T17:18:40.514299+0000","last_peered":"2024-04-20T17:18:40.514299+0000","last_clean":"2024-04-20T17:18:40.514299+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:18:40.514299+0000","last_undegraded":"2024-04-20T17:18:40.514299+0000","last_fullsized":"2024-04-20T17:18:40.514299+0000","mapping_epoch":115,"log_start":"138'7200","ondisk_log_start":"138'7200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"293'15941","last_scrub_stamp":"2024-04-20T17:17:04.896640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:04.896640+0000","objects_scrubbed":3929,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:18:41.908547+0000","scrub_duration":355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171376664,"num_objects":2616,"num_object_clones":0,"num_object_copies":5232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2616,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":17254,"num_write_kb":635777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"324'17315","reported_seq":18078,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-04-20T17:18:40.746665+0000","last_change":"2024-04-20T17:18:17.426012+0000","last_active":"2024-04-20T17:18:40.746665+0000","last_peered":"2024-04-20T17:18:40.746665+0000","last_clean":"2024-04-20T17:18:40.746665+0000","last_became_active":"2024-04-20T17:12:23.515375+0000","last_became_peered":"2024-04-20T17:12:23.515375+0000","last_unstale":"2024-04-20T17:18:40.746665+0000","last_undegraded":"2024-04-20T17:18:40.746665+0000","last_fullsized":"2024-04-20T17:18:40.746665+0000","mapping_epoch":202,"log_start":"138'7300","ondisk_log_start":"138'7300","created":115,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'16928","last_scrub_stamp":"2024-04-20T17:18:17.425878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:17.425878+0000","objects_scrubbed":3071,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:47.218525+0000","scrub_duration":353,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175702016,"num_objects":2681,"num_object_clones":0,"num_object_copies":5362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17315,"num_write_kb":639872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":2828009496,"num_objects":43153,"num_object_clones":0,"num_object_copies":86306,"num_objects_missing_on_primary":2171,"num_objects_missing":2171,"num_objects_degraded":0,"num_objects_misplaced":4356,"num_objects_unfound":0,"num_objects_dirty":43153,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":278831,"num_write_kb":10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27031,"num_bytes_recovered":1771503616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6425092096,"data_stored":6425083952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},"log_size":160830,"ondisk_log_size":160830,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991387,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":604204,"kb_used_data":532768,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93140948,"statfs":{"total":95995035648,"available":95376330752,"internally_reserved":0,"allocated":545554432,"data_stored":544425214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":73136580},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253025,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":728416,"kb_used_data":552856,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93016736,"statfs":{"total":95995035648,"available":95249137664,"internally_reserved":0,"allocated":566124544,"data_stored":564998878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":179763648},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612775,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":831908,"kb_used_data":692444,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92913244,"statfs":{"total":95995035648,"available":95143161856,"internally_reserved":0,"allocated":709062656,"data_stored":707932938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":142801344},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809004,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":790112,"kb_used_data":690904,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92955040,"statfs":{"total":95995035648,"available":95185960960,"internally_reserved":0,"allocated":707485696,"data_stored":706360050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168753,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":620588,"kb_used_data":531104,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93124564,"statfs":{"total":95995035648,"available":95359553536,"internally_reserved":0,"allocated":543850496,"data_stored":542721298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":91617738},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430389,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":459808,"kb_used_data":356440,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93285344,"statfs":{"total":95995035648,"available":95524192256,"internally_reserved":0,"allocated":364994560,"data_stored":363868914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575056,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":458336,"kb_used_data":342488,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":93286816,"statfs":{"total":95995035648,"available":95525699584,"internally_reserved":0,"allocated":350707712,"data_stored":349582066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":118618565},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738580,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":699296,"kb_used_data":551512,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93045856,"statfs":{"total":95995035648,"available":95278956544,"internally_reserved":0,"allocated":564748288,"data_stored":563622642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,2,4,7,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393869,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":641056,"kb_used_data":605144,"kb_used_omap":202,"kb_used_meta":35701,"kb_avail":93104096,"statfs":{"total":95995035648,"available":95338594304,"internally_reserved":0,"allocated":619667456,"data_stored":618541810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":36558295},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182597,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":414048,"kb_used_data":346776,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93331104,"statfs":{"total":95995035648,"available":95571050496,"internally_reserved":0,"allocated":355098624,"data_stored":353972978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986368,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":201892,"kb_used_data":170716,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93543260,"statfs":{"total":95995035648,"available":95788298240,"internally_reserved":0,"allocated":174813184,"data_stored":173683466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790138,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1070432,"kb_used_data":954584,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":92674720,"statfs":{"total":95995035648,"available":94898913280,"internally_reserved":0,"allocated":977494016,"data_stored":976368370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":118618565},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":559349760,"data_stored":559349760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":346030080,"data_stored":346030080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":615907328,"data_stored":615907328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":352321536,"data_stored":352321536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":171446272,"data_stored":171442200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":974192640,"data_stored":974192640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":362086400,"data_stored":362086400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":537329664,"data_stored":537329664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":701825024,"data_stored":701825024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":702877696,"data_stored":702873624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":561577984,"data_stored":561577984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":540147712,"data_stored":540147712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:18:45.684 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:18:46.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:46 smithi005 ceph-mon[39562]: 2.7 scrub starts 2024-04-20T17:18:46.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:46 smithi005 ceph-mon[39562]: osdmap e326: 12 total, 12 up, 12 in 2024-04-20T17:18:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:46 smithi106 ceph-mon[44423]: 2.7 scrub starts 2024-04-20T17:18:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:46 smithi106 ceph-mon[44423]: osdmap e326: 12 total, 12 up, 12 in 2024-04-20T17:18:46.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:46 smithi171 ceph-mon[44189]: 2.7 scrub starts 2024-04-20T17:18:46.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:46 smithi171 ceph-mon[44189]: osdmap e326: 12 total, 12 up, 12 in 2024-04-20T17:18:46.602 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:18:46.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:46.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:18:46.708 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:18:46.734 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:46.753 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:18:46.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:46.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:18:46.857 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:18:46.883 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:46.901 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:18:46.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:46.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:18:47.005 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:18:47.037 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:47.190 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:18:47.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:47.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:18:47.294 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:18:47.321 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:47.341 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:18:47.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:47.347 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:18:47.448 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:18:47.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:47 smithi005 ceph-mon[39562]: 2.8 scrub starts 2024-04-20T17:18:47.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:47 smithi005 ceph-mon[39562]: 2.7 scrub ok 2024-04-20T17:18:47.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:47 smithi005 ceph-mon[39562]: 2.8 scrub ok 2024-04-20T17:18:47.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:47 smithi005 ceph-mon[39562]: pgmap v896: 17 pgs: 1 active+recovering, 16 active+clean; 2.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 281 op/s; 4188/83694 objects misplaced (5.004%); 1.3 MiB/s, 20 objects/s recovering 2024-04-20T17:18:47.477 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:47.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:47 smithi106 ceph-mon[44423]: 2.8 scrub starts 2024-04-20T17:18:47.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:47 smithi106 ceph-mon[44423]: 2.7 scrub ok 2024-04-20T17:18:47.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:47 smithi106 ceph-mon[44423]: 2.8 scrub ok 2024-04-20T17:18:47.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:47 smithi106 ceph-mon[44423]: pgmap v896: 17 pgs: 1 active+recovering, 16 active+clean; 2.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 281 op/s; 4188/83694 objects misplaced (5.004%); 1.3 MiB/s, 20 objects/s recovering 2024-04-20T17:18:47.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:47 smithi171 ceph-mon[44189]: 2.8 scrub starts 2024-04-20T17:18:47.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:47 smithi171 ceph-mon[44189]: 2.7 scrub ok 2024-04-20T17:18:47.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:47 smithi171 ceph-mon[44189]: 2.8 scrub ok 2024-04-20T17:18:47.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:47 smithi171 ceph-mon[44189]: pgmap v896: 17 pgs: 1 active+recovering, 16 active+clean; 2.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 281 op/s; 4188/83694 objects misplaced (5.004%); 1.3 MiB/s, 20 objects/s recovering 2024-04-20T17:18:47.679 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:18:47.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:47.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:18:47.784 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:18:47.810 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:47.828 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:18:47.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:47.831 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:18:47.932 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:18:47.958 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:47.977 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:18:47.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:47.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:18:48.081 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:18:48.108 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:48.125 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:18:48.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:48.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:18:48.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:48 smithi005 ceph-mon[39562]: 2.b scrub starts 2024-04-20T17:18:48.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:48 smithi005 ceph-mon[39562]: 2.b scrub ok 2024-04-20T17:18:48.229 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:18:48.256 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:48.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:48 smithi106 ceph-mon[44423]: 2.b scrub starts 2024-04-20T17:18:48.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:48 smithi106 ceph-mon[44423]: 2.b scrub ok 2024-04-20T17:18:48.267 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:18:48.267 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:18:48.267 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:18:48.267 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:18:48.267 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:18:48.275 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:18:48.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:48.279 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:18:48.380 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:18:48.406 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:48.424 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:18:48.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:48.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:18:48.527 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:18:48.567 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:48.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:48 smithi171 ceph-mon[44189]: 2.b scrub starts 2024-04-20T17:18:48.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:48 smithi171 ceph-mon[44189]: 2.b scrub ok 2024-04-20T17:18:48.919 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:18:48.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:48.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:18:49.027 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:18:49.053 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:49.072 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:18:49.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:49.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:18:49.176 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:18:49.182 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:18:49.182 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:18:49.182 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:18:49.182 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:18:49.182 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:18:49.185 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:18:49.205 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:49 smithi005 ceph-mon[39562]: pgmap v897: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 308 op/s; 4188/82494 objects misplaced (5.077%); 688 KiB/s, 10 objects/s recovering 2024-04-20T17:18:49.205 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:49 smithi005 ceph-mon[39562]: 2.c scrub starts 2024-04-20T17:18:49.205 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:49 smithi005 ceph-mon[39562]: 2.c scrub ok 2024-04-20T17:18:49.205 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:49 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3269690243' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:18:49.207 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:49.208 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:18:49.234 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:18:49.255 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:18:49.277 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:18:49.298 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:18:49.318 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:18:49.319 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:18:49.319 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:18:49.319 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:18:49.319 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:18:49.320 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:18:49.320 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:18:49.320 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:18:49.320 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:18:49.320 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:18:49.320 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:18:49.320 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:18:49.320 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:18:49.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:49 smithi106 ceph-mon[44423]: pgmap v897: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 308 op/s; 4188/82494 objects misplaced (5.077%); 688 KiB/s, 10 objects/s recovering 2024-04-20T17:18:49.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:49 smithi106 ceph-mon[44423]: 2.c scrub starts 2024-04-20T17:18:49.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:49 smithi106 ceph-mon[44423]: 2.c scrub ok 2024-04-20T17:18:49.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:49 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3269690243' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:18:49.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:49 smithi171 ceph-mon[44189]: pgmap v897: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 308 op/s; 4188/82494 objects misplaced (5.077%); 688 KiB/s, 10 objects/s recovering 2024-04-20T17:18:49.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:49 smithi171 ceph-mon[44189]: 2.c scrub starts 2024-04-20T17:18:49.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:49 smithi171 ceph-mon[44189]: 2.c scrub ok 2024-04-20T17:18:49.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:49 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3269690243' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:18:49.680 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:18:49.705 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-20T17:18:49.929 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:18:49.966 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:18:49.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:49.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:18:50.076 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:18:50.104 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:50.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:50 smithi005 ceph-mon[39562]: 2.5 scrub starts 2024-04-20T17:18:50.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:50 smithi005 ceph-mon[39562]: 2.2 scrub starts 2024-04-20T17:18:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:50 smithi005 ceph-mon[39562]: 2.5 scrub ok 2024-04-20T17:18:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:50 smithi005 ceph-mon[39562]: 2.2 scrub ok 2024-04-20T17:18:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:50 smithi106 ceph-mon[44423]: 2.5 scrub starts 2024-04-20T17:18:50.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:50 smithi106 ceph-mon[44423]: 2.2 scrub starts 2024-04-20T17:18:50.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:50 smithi106 ceph-mon[44423]: 2.5 scrub ok 2024-04-20T17:18:50.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:50 smithi106 ceph-mon[44423]: 2.2 scrub ok 2024-04-20T17:18:50.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.538 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:18:50.554 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:18:50.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:50.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:18:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:50 smithi171 ceph-mon[44189]: 2.5 scrub starts 2024-04-20T17:18:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:50 smithi171 ceph-mon[44189]: 2.2 scrub starts 2024-04-20T17:18:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:50 smithi171 ceph-mon[44189]: 2.5 scrub ok 2024-04-20T17:18:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:50 smithi171 ceph-mon[44189]: 2.2 scrub ok 2024-04-20T17:18:50.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:50.662 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:18:50.692 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:50.766 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:18:50.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:50.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:18:50.872 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:18:50.904 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:51.105 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:18:51.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:51.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:18:51.210 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:18:51.238 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:51.256 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:18:51.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:51.259 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:18:51.360 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:18:51.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:51.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:51.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:51 smithi005 ceph-mon[39562]: pgmap v898: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 289 op/s; 4015/81670 objects misplaced (4.916%); 1.4 MiB/s, 21 objects/s recovering 2024-04-20T17:18:51.386 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:51.407 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:18:51.412 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:51.413 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:51.413 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:51 smithi171 ceph-mon[44189]: pgmap v898: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 289 op/s; 4015/81670 objects misplaced (4.916%); 1.4 MiB/s, 21 objects/s recovering 2024-04-20T17:18:51.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:51.414 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:18:51.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:51.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:18:51.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:51 smithi106 ceph-mon[44423]: pgmap v898: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 289 op/s; 4015/81670 objects misplaced (4.916%); 1.4 MiB/s, 21 objects/s recovering 2024-04-20T17:18:51.514 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:18:51.540 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:51.559 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:18:51.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:51.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:18:51.663 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:18:51.695 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:51.798 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:18:51.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:51.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:18:51.902 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:18:51.930 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:51.948 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:18:51.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:51.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:18:52.052 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:18:52.079 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:52.100 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:18:52.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:52.106 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:18:52.207 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:18:52.234 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:52.254 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:18:52.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:52.259 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:18:52.360 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:18:52.386 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:52.405 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:18:52.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:52.409 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:18:52.510 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:18:52.537 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:52.540 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:18:52.787 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:18:52.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:52.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:18:52.842 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:18:52.895 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:18:52.922 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:52.940 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:18:52.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:52.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:18:53.044 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:18:53.071 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:53.089 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:18:53.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:53.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:18:53.192 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:18:53.222 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:53.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:53 smithi106 ceph-mon[44423]: 2.d scrub starts 2024-04-20T17:18:53.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:53 smithi106 ceph-mon[44423]: pgmap v899: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 254 op/s; 4015/80592 objects misplaced (4.982%); 712 KiB/s, 11 objects/s recovering 2024-04-20T17:18:53.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:53 smithi005 ceph-mon[39562]: 2.d scrub starts 2024-04-20T17:18:53.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:53 smithi005 ceph-mon[39562]: pgmap v899: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 254 op/s; 4015/80592 objects misplaced (4.982%); 712 KiB/s, 11 objects/s recovering 2024-04-20T17:18:53.473 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:18:53.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:53.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:18:53.579 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:18:53.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:53 smithi171 ceph-mon[44189]: 2.d scrub starts 2024-04-20T17:18:53.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:53 smithi171 ceph-mon[44189]: pgmap v899: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 254 op/s; 4015/80592 objects misplaced (4.982%); 712 KiB/s, 11 objects/s recovering 2024-04-20T17:18:53.605 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:53.624 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:18:53.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:53.626 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:18:53.727 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:18:53.754 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:53.773 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:18:53.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:53.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:18:53.876 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:18:53.902 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:53.922 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:18:53.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:53.925 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:18:54.026 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:18:54.060 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:54.184 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:18:54.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:54.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:18:54.191 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:18:54.288 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:18:54.322 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:54.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:54 smithi005 ceph-mon[39562]: 2.d scrub ok 2024-04-20T17:18:54.472 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:18:54.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:54.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:18:54.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:54 smithi106 ceph-mon[44423]: 2.d scrub ok 2024-04-20T17:18:54.577 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:18:54.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:54 smithi171 ceph-mon[44189]: 2.d scrub ok 2024-04-20T17:18:54.604 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:54.622 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:18:54.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:54.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:18:54.726 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:18:54.759 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:55.159 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:55 smithi005 ceph-mon[39562]: pgmap v900: 17 pgs: 1 active+recovering, 16 active+clean; 2.4 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 285 op/s; 3836/78978 objects misplaced (4.857%); 1.1 MiB/s, 17 objects/s recovering 2024-04-20T17:18:55.159 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:55 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2814419322' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:18:55.170 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:18:55.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:55 smithi106 ceph-mon[44423]: pgmap v900: 17 pgs: 1 active+recovering, 16 active+clean; 2.4 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 285 op/s; 3836/78978 objects misplaced (4.857%); 1.1 MiB/s, 17 objects/s recovering 2024-04-20T17:18:55.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:55 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2814419322' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:18:55.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:55 smithi171 ceph-mon[44189]: pgmap v900: 17 pgs: 1 active+recovering, 16 active+clean; 2.4 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 285 op/s; 3836/78978 objects misplaced (4.857%); 1.1 MiB/s, 17 objects/s recovering 2024-04-20T17:18:55.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:55 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2814419322' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:18:56.160 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:56 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:18:56.160 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:56 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:18:56.160 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:56 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2814419322' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:18:56.160 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:56 smithi005 ceph-mon[39562]: osdmap e327: 12 total, 12 up, 12 in 2024-04-20T17:18:56.160 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:56 smithi005 ceph-mon[39562]: osdmap e328: 12 total, 12 up, 12 in 2024-04-20T17:18:56.441 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:18:56.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:56.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:18:56.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:56 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:18:56.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:56 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:18:56.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:56 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2814419322' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:18:56.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:56 smithi106 ceph-mon[44423]: osdmap e327: 12 total, 12 up, 12 in 2024-04-20T17:18:56.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:56 smithi106 ceph-mon[44423]: osdmap e328: 12 total, 12 up, 12 in 2024-04-20T17:18:56.550 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:18:56.576 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:56 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:18:56.576 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:56 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:18:56.576 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:56 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2814419322' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:18:56.576 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:56 smithi171 ceph-mon[44189]: osdmap e327: 12 total, 12 up, 12 in 2024-04-20T17:18:56.576 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:56 smithi171 ceph-mon[44189]: osdmap e328: 12 total, 12 up, 12 in 2024-04-20T17:18:56.578 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:56.598 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:18:56.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:56.601 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:18:56.702 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:18:56.734 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:57.086 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:18:57.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:57.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:18:57.193 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:18:57.220 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:57.240 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:18:57.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:57.244 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:18:57.345 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:18:57.373 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:57.384 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:57 smithi005 ceph-mon[39562]: pgmap v903: 17 pgs: 1 active+recovering, 16 active+clean; 2.4 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 272 op/s; 3836/78134 objects misplaced (4.910%); 1.4 MiB/s, 21 objects/s recovering 2024-04-20T17:18:57.392 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:18:57.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:57.395 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:18:57.496 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:18:57.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:57 smithi106 ceph-mon[44423]: pgmap v903: 17 pgs: 1 active+recovering, 16 active+clean; 2.4 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 272 op/s; 3836/78134 objects misplaced (4.910%); 1.4 MiB/s, 21 objects/s recovering 2024-04-20T17:18:57.520 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:57 smithi171 ceph-mon[44189]: pgmap v903: 17 pgs: 1 active+recovering, 16 active+clean; 2.4 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 272 op/s; 3836/78134 objects misplaced (4.910%); 1.4 MiB/s, 21 objects/s recovering 2024-04-20T17:18:57.522 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:57.541 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:18:57.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:57.544 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:18:57.645 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:18:57.671 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:57.691 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:18:57.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:57.694 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:18:57.794 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:18:57.830 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:58.983 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:18:58.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:58.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:18:59.096 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:18:59.123 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:59.141 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:18:59.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:59.144 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:18:59.244 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:18:59.278 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:59.364 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:18:59.365 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:18:59.383 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:18:59.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:59.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:18:59.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:18:59 smithi005 ceph-mon[39562]: pgmap v904: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 302 op/s; 3836/76826 objects misplaced (4.993%); 688 KiB/s, 10 objects/s recovering 2024-04-20T17:18:59.487 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:18:59.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:18:59 smithi106 ceph-mon[44423]: pgmap v904: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 302 op/s; 3836/76826 objects misplaced (4.993%); 688 KiB/s, 10 objects/s recovering 2024-04-20T17:18:59.511 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:18:59 smithi171 ceph-mon[44189]: pgmap v904: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 302 op/s; 3836/76826 objects misplaced (4.993%); 688 KiB/s, 10 objects/s recovering 2024-04-20T17:18:59.513 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:59.533 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:18:59.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:59.536 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:18:59.638 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:18:59.664 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:59.682 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:18:59.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:59.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:18:59.786 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:18:59.812 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:59.830 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:18:59.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:59.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:18:59.934 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:18:59.960 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:18:59.984 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:18:59.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:18:59.987 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:19:00.078 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:19:00.079 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:19:00.079 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:19:00.079 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:19:00.079 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.079 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:216588 2.6 2:65396415:::benchmark_data_smithi171_78918_object55594:head [delete] snapc 0=[] ondisk+write+known_if_redirected e210)", 2024-04-20T17:19:00.079 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:40.388336+0000", 2024-04-20T17:19:00.079 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 379.68721588099999, 2024-04-20T17:19:00.079 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.107803945, 2024-04-20T17:19:00.079 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:00.079 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:00.079 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:00.079 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:19:00.080 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:19:00.080 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:00.080 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 216588 2024-04-20T17:19:00.080 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.080 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:00.080 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.080 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:00.080 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.388336+0000", 2024-04-20T17:19:00.080 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.080 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.080 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.080 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:00.080 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.388336+0000", 2024-04-20T17:19:00.080 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.081 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.081 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.081 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:00.081 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.388338+0000", 2024-04-20T17:19:00.081 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.779e-06 2024-04-20T17:19:00.081 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.081 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.081 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:00.081 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.388343+0000", 2024-04-20T17:19:00.081 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5539999999999999e-06 2024-04-20T17:19:00.081 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.081 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.081 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:00.082 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.388353+0000", 2024-04-20T17:19:00.082 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0502999999999999e-05 2024-04-20T17:19:00.082 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.082 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.082 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:00.082 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.388360+0000", 2024-04-20T17:19:00.082 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2810000000000003e-06 2024-04-20T17:19:00.082 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.083 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.083 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:00.083 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.388403+0000", 2024-04-20T17:19:00.083 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2366000000000001e-05 2024-04-20T17:19:00.083 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.083 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.083 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:00.083 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.388481+0000", 2024-04-20T17:19:00.083 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8220000000000007e-05 2024-04-20T17:19:00.083 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.083 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.083 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:19:00.083 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.388547+0000", 2024-04-20T17:19:00.084 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5841999999999997e-05 2024-04-20T17:19:00.084 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.084 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.084 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:19:00.084 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.393289+0000", 2024-04-20T17:19:00.084 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0047422339999999997 2024-04-20T17:19:00.084 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.084 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.084 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:00.084 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.496033+0000", 2024-04-20T17:19:00.084 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.102744271 2024-04-20T17:19:00.084 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.084 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.085 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:00.085 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.496078+0000", 2024-04-20T17:19:00.085 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5105999999999998e-05 2024-04-20T17:19:00.085 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.085 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.085 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:00.085 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.496140+0000", 2024-04-20T17:19:00.085 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1789000000000001e-05 2024-04-20T17:19:00.085 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.085 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:00.085 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.085 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.085 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.085 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(osd.1.0:31745 2.6 e210/115 ondisk, result = 0)", 2024-04-20T17:19:00.086 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:40.881641+0000", 2024-04-20T17:19:00.086 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 379.19391113199998, 2024-04-20T17:19:00.086 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10404883400000001, 2024-04-20T17:19:00.086 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:00.086 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:00.086 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:19:00.086 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:00.086 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.086 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:00.086 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.881641+0000", 2024-04-20T17:19:00.086 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.086 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.086 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.087 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:00.087 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.881641+0000", 2024-04-20T17:19:00.087 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.087 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.087 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.087 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:00.087 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.881642+0000", 2024-04-20T17:19:00.087 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.034e-06 2024-04-20T17:19:00.087 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.087 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.087 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:00.087 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.881646+0000", 2024-04-20T17:19:00.087 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0690000000000003e-06 2024-04-20T17:19:00.087 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.088 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.088 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:00.088 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.881655+0000", 2024-04-20T17:19:00.088 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8270000000000004e-06 2024-04-20T17:19:00.088 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.088 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.088 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:00.088 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.881661+0000", 2024-04-20T17:19:00.088 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5629999999999999e-06 2024-04-20T17:19:00.088 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.088 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.088 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:00.088 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.985621+0000", 2024-04-20T17:19:00.088 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10396008 2024-04-20T17:19:00.088 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.089 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.089 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:00.089 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.985628+0000", 2024-04-20T17:19:00.089 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4939999999999997e-06 2024-04-20T17:19:00.089 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.089 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.089 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:00.089 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.985690+0000", 2024-04-20T17:19:00.089 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1766999999999993e-05 2024-04-20T17:19:00.089 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.089 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:00.089 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.089 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.089 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.090 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:216773 2.6 2:6f744e0b:::benchmark_data_smithi171_78918_object55779:head [delete] snapc 0=[] ondisk+write+known_if_redirected e210)", 2024-04-20T17:19:00.090 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:40.890292+0000", 2024-04-20T17:19:00.090 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 379.18526066099997, 2024-04-20T17:19:00.090 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10639662900000001, 2024-04-20T17:19:00.090 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:00.090 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:00.090 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:00.090 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:19:00.090 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:19:00.090 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:00.090 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 216773 2024-04-20T17:19:00.090 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.090 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:00.090 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.091 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:00.091 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.890292+0000", 2024-04-20T17:19:00.091 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.091 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.091 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.091 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:00.092 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.890292+0000", 2024-04-20T17:19:00.092 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.092 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.092 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.092 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:00.092 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.890293+0000", 2024-04-20T17:19:00.092 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5340000000000001e-06 2024-04-20T17:19:00.092 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.092 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.093 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:00.093 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.890298+0000", 2024-04-20T17:19:00.093 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.101e-06 2024-04-20T17:19:00.093 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.093 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.093 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:00.093 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.890308+0000", 2024-04-20T17:19:00.093 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8579999999999995e-06 2024-04-20T17:19:00.093 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.093 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.093 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:00.093 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.890314+0000", 2024-04-20T17:19:00.093 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6890000000000001e-06 2024-04-20T17:19:00.093 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.093 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.094 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:00.094 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.987519+0000", 2024-04-20T17:19:00.094 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.097205285000000002 2024-04-20T17:19:00.094 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.094 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.094 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:00.094 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.987620+0000", 2024-04-20T17:19:00.094 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000100648 2024-04-20T17:19:00.094 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.094 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.094 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:19:00.094 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.987697+0000", 2024-04-20T17:19:00.094 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7645000000000006e-05 2024-04-20T17:19:00.095 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.095 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.095 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:00.095 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.996506+0000", 2024-04-20T17:19:00.095 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0088082349999999993 2024-04-20T17:19:00.095 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.095 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.095 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:19:00.095 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.996589+0000", 2024-04-20T17:19:00.095 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3683000000000005e-05 2024-04-20T17:19:00.095 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.095 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.095 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:00.096 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.996640+0000", 2024-04-20T17:19:00.096 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0555e-05 2024-04-20T17:19:00.096 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.096 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.096 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:00.096 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.996688+0000", 2024-04-20T17:19:00.096 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8396000000000002e-05 2024-04-20T17:19:00.096 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.097 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:00.097 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.097 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.097 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.097 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:216942 2.6 2:6cfbbda6:::benchmark_data_smithi171_78918_object55948:head [delete] snapc 0=[] ondisk+write+known_if_redirected e210)", 2024-04-20T17:19:00.097 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:41.297347+0000", 2024-04-20T17:19:00.097 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 378.77820549500001, 2024-04-20T17:19:00.097 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.11543711399999999, 2024-04-20T17:19:00.097 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:00.097 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:00.097 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:00.097 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:19:00.097 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:19:00.097 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:00.098 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 216942 2024-04-20T17:19:00.098 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.098 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:00.098 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.098 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:00.098 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.297347+0000", 2024-04-20T17:19:00.098 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.098 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.098 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.098 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:00.098 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.297347+0000", 2024-04-20T17:19:00.098 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.098 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.099 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.099 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:00.099 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.297348+0000", 2024-04-20T17:19:00.099 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7540000000000001e-06 2024-04-20T17:19:00.099 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.099 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.099 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:00.099 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.297355+0000", 2024-04-20T17:19:00.099 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0929999999999999e-06 2024-04-20T17:19:00.099 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.099 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.099 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:00.099 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.297366+0000", 2024-04-20T17:19:00.099 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1318000000000001e-05 2024-04-20T17:19:00.100 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.100 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.100 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:00.100 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.297374+0000", 2024-04-20T17:19:00.100 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4020000000000002e-06 2024-04-20T17:19:00.100 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.100 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.100 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:00.100 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.384301+0000", 2024-04-20T17:19:00.100 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.086926952000000002 2024-04-20T17:19:00.100 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.100 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.100 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:00.100 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.384455+0000", 2024-04-20T17:19:00.101 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015342900000000001 2024-04-20T17:19:00.101 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.101 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.101 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:19:00.101 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.384607+0000", 2024-04-20T17:19:00.101 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000152406 2024-04-20T17:19:00.101 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.101 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.101 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:19:00.101 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.388368+0000", 2024-04-20T17:19:00.101 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003760939 2024-04-20T17:19:00.101 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.101 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.101 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:00.102 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.412667+0000", 2024-04-20T17:19:00.102 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.024299438999999999 2024-04-20T17:19:00.102 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.102 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.102 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:00.102 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.412728+0000", 2024-04-20T17:19:00.102 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0936000000000001e-05 2024-04-20T17:19:00.102 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.102 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.102 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:00.102 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.412784+0000", 2024-04-20T17:19:00.102 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5445999999999997e-05 2024-04-20T17:19:00.102 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.102 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:00.102 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.103 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.103 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.103 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:217108 2.6 2:6f890090:::benchmark_data_smithi171_78918_object56114:head [delete] snapc 0=[] ondisk+write+known_if_redirected e210)", 2024-04-20T17:19:00.103 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:41.907680+0000", 2024-04-20T17:19:00.103 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 378.167872402, 2024-04-20T17:19:00.103 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10201656200000001, 2024-04-20T17:19:00.103 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:00.103 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:00.103 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:00.103 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:19:00.103 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:19:00.103 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:00.103 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 217108 2024-04-20T17:19:00.103 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.104 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:00.104 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.104 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:00.104 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.907680+0000", 2024-04-20T17:19:00.104 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.104 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.104 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.104 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:00.104 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.907680+0000", 2024-04-20T17:19:00.104 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.104 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.104 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.104 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:00.104 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.907683+0000", 2024-04-20T17:19:00.104 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2650000000000001e-06 2024-04-20T17:19:00.105 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.105 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.105 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:00.105 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.907693+0000", 2024-04-20T17:19:00.105 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0362e-05 2024-04-20T17:19:00.105 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.105 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.105 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:00.105 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.907714+0000", 2024-04-20T17:19:00.105 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0568999999999999e-05 2024-04-20T17:19:00.105 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.105 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.105 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:00.105 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.907726+0000", 2024-04-20T17:19:00.105 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2153e-05 2024-04-20T17:19:00.106 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.106 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.106 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:00.106 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.986670+0000", 2024-04-20T17:19:00.106 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.078944241999999998 2024-04-20T17:19:00.106 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.106 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.106 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:00.106 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.986761+0000", 2024-04-20T17:19:00.106 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0179999999999994e-05 2024-04-20T17:19:00.106 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.106 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.106 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:19:00.106 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.986840+0000", 2024-04-20T17:19:00.106 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9021000000000001e-05 2024-04-20T17:19:00.107 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.107 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.107 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:19:00.107 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.998662+0000", 2024-04-20T17:19:00.107 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011822882999999999 2024-04-20T17:19:00.107 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.107 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.107 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:00.107 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.009567+0000", 2024-04-20T17:19:00.107 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010904528 2024-04-20T17:19:00.107 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.107 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.107 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:00.107 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.009632+0000", 2024-04-20T17:19:00.107 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5217999999999998e-05 2024-04-20T17:19:00.108 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.108 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.108 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:00.108 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.009696+0000", 2024-04-20T17:19:00.108 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4140999999999995e-05 2024-04-20T17:19:00.108 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.108 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:00.108 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.108 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.108 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.108 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:217163 2.6 2:61348ded:::benchmark_data_smithi171_78918_object56169:head [delete] snapc 0=[] ondisk+write+known_if_redirected e210)", 2024-04-20T17:19:00.108 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:42.127376+0000", 2024-04-20T17:19:00.108 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 377.94817576899999, 2024-04-20T17:19:00.108 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.106929598, 2024-04-20T17:19:00.109 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:00.109 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:00.109 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:00.109 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:19:00.109 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:19:00.109 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:00.109 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 217163 2024-04-20T17:19:00.109 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.109 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:00.109 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.109 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:00.109 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.127376+0000", 2024-04-20T17:19:00.109 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.109 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.109 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.110 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:00.110 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.127376+0000", 2024-04-20T17:19:00.110 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.110 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.110 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.110 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:00.110 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.127379+0000", 2024-04-20T17:19:00.110 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6220000000000001e-06 2024-04-20T17:19:00.110 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.110 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.110 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:00.110 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.127386+0000", 2024-04-20T17:19:00.110 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7889999999999997e-06 2024-04-20T17:19:00.110 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.111 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.111 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:00.111 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.127404+0000", 2024-04-20T17:19:00.111 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8179e-05 2024-04-20T17:19:00.111 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.111 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.111 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:00.111 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.127414+0000", 2024-04-20T17:19:00.111 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0122000000000001e-05 2024-04-20T17:19:00.111 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.111 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.111 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:00.111 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.127490+0000", 2024-04-20T17:19:00.112 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5359999999999997e-05 2024-04-20T17:19:00.112 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.112 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.112 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:00.112 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.127622+0000", 2024-04-20T17:19:00.112 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0001325 2024-04-20T17:19:00.112 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.112 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.112 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:19:00.112 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.127751+0000", 2024-04-20T17:19:00.112 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012849500000000001 2024-04-20T17:19:00.112 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.112 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.112 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:19:00.112 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.234161+0000", 2024-04-20T17:19:00.113 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.106410616 2024-04-20T17:19:00.113 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.113 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.113 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:00.113 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.234211+0000", 2024-04-20T17:19:00.113 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9472999999999999e-05 2024-04-20T17:19:00.113 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.113 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.113 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:00.113 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.234246+0000", 2024-04-20T17:19:00.113 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5463999999999998e-05 2024-04-20T17:19:00.113 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.113 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.113 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:00.114 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.234306+0000", 2024-04-20T17:19:00.114 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9978000000000001e-05 2024-04-20T17:19:00.114 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.114 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:00.114 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.114 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.114 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.114 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(client.24736.0:217163 2.6 e210/115 ondisk, result = 0)", 2024-04-20T17:19:00.114 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:42.133593+0000", 2024-04-20T17:19:00.114 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 377.941958774, 2024-04-20T17:19:00.114 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.100718011, 2024-04-20T17:19:00.114 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:00.114 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:00.114 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:19:00.115 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:00.115 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.115 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:00.115 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.133593+0000", 2024-04-20T17:19:00.115 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.115 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.115 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.115 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:00.115 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.133593+0000", 2024-04-20T17:19:00.115 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.115 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.115 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.115 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:00.115 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.133595+0000", 2024-04-20T17:19:00.116 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2309999999999999e-06 2024-04-20T17:19:00.116 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.116 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.116 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:00.116 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.133599+0000", 2024-04-20T17:19:00.116 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7350000000000004e-06 2024-04-20T17:19:00.116 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.116 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.116 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:00.116 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.133610+0000", 2024-04-20T17:19:00.116 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0481e-05 2024-04-20T17:19:00.116 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.116 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.116 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:00.116 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.133617+0000", 2024-04-20T17:19:00.117 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6710000000000002e-06 2024-04-20T17:19:00.117 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.117 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.117 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:00.117 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.234199+0000", 2024-04-20T17:19:00.117 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10058223099999999 2024-04-20T17:19:00.117 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.117 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.117 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:00.117 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.234207+0000", 2024-04-20T17:19:00.117 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8339999999999999e-06 2024-04-20T17:19:00.117 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.117 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.118 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:00.118 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.234311+0000", 2024-04-20T17:19:00.118 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010482800000000001 2024-04-20T17:19:00.118 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.118 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:00.118 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.118 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.118 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.118 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:217217 2.6 2:6881c357:::benchmark_data_smithi171_78918_object56223:head [delete] snapc 0=[] ondisk+write+known_if_redirected e210)", 2024-04-20T17:19:00.118 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:42.357612+0000", 2024-04-20T17:19:00.118 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 377.71794029900002, 2024-04-20T17:19:00.118 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.11809499399999999, 2024-04-20T17:19:00.118 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:00.118 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:00.119 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:00.119 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:19:00.119 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:19:00.119 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:00.119 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 217217 2024-04-20T17:19:00.119 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.119 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:00.119 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.119 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:00.119 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.357612+0000", 2024-04-20T17:19:00.119 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.119 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.119 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.119 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:00.119 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.357612+0000", 2024-04-20T17:19:00.120 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.120 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.120 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.120 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:00.120 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.357614+0000", 2024-04-20T17:19:00.120 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0169999999999999e-06 2024-04-20T17:19:00.120 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.120 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.120 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:00.120 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.357619+0000", 2024-04-20T17:19:00.120 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9459999999999997e-06 2024-04-20T17:19:00.120 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.120 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.120 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:00.121 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.357628+0000", 2024-04-20T17:19:00.121 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6999999999999997e-06 2024-04-20T17:19:00.121 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.121 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.121 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:00.121 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.357637+0000", 2024-04-20T17:19:00.121 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0799999999999995e-06 2024-04-20T17:19:00.121 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.121 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.121 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:00.121 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.359614+0000", 2024-04-20T17:19:00.121 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0019772430000000001 2024-04-20T17:19:00.121 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.121 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.122 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:00.122 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.359706+0000", 2024-04-20T17:19:00.122 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1989000000000003e-05 2024-04-20T17:19:00.122 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.122 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.122 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:19:00.122 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.359783+0000", 2024-04-20T17:19:00.122 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6790000000000002e-05 2024-04-20T17:19:00.122 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.122 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.122 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:19:00.122 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.363334+0000", 2024-04-20T17:19:00.122 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003551008 2024-04-20T17:19:00.122 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.123 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.123 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:00.123 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.475601+0000", 2024-04-20T17:19:00.123 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.112267324 2024-04-20T17:19:00.123 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.123 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.123 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:00.123 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.475642+0000", 2024-04-20T17:19:00.123 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0760000000000003e-05 2024-04-20T17:19:00.123 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.123 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.123 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:00.123 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.475707+0000", 2024-04-20T17:19:00.123 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5136999999999997e-05 2024-04-20T17:19:00.124 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.124 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:00.124 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.124 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.124 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.124 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(client.24736.0:217217 2.6 e210/115 ondisk, result = 0)", 2024-04-20T17:19:00.124 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:42.372331+0000", 2024-04-20T17:19:00.124 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 377.70322076299999, 2024-04-20T17:19:00.124 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.103382027, 2024-04-20T17:19:00.124 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:00.124 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:00.124 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:19:00.124 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:00.124 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.124 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:00.125 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.372331+0000", 2024-04-20T17:19:00.125 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.125 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.125 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.125 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:00.125 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.372331+0000", 2024-04-20T17:19:00.125 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.125 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.125 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.125 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:00.125 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.372334+0000", 2024-04-20T17:19:00.125 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.92e-06 2024-04-20T17:19:00.125 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.125 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.125 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:00.126 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.372344+0000", 2024-04-20T17:19:00.126 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2499999999999995e-06 2024-04-20T17:19:00.126 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.126 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.126 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:00.126 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.372356+0000", 2024-04-20T17:19:00.126 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2478e-05 2024-04-20T17:19:00.126 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.126 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.126 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:00.126 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.372369+0000", 2024-04-20T17:19:00.126 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.292e-05 2024-04-20T17:19:00.126 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.126 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.127 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:00.127 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.475587+0000", 2024-04-20T17:19:00.127 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.103218184 2024-04-20T17:19:00.127 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.127 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.127 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:00.127 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.475596+0000", 2024-04-20T17:19:00.127 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3990000000000004e-06 2024-04-20T17:19:00.127 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.127 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.127 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:00.127 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.475713+0000", 2024-04-20T17:19:00.127 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011787599999999999 2024-04-20T17:19:00.127 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.128 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:00.128 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.128 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.128 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.128 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(osd.1.0:32102 2.6 e210/115 ondisk, result = 0)", 2024-04-20T17:19:00.128 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:42.500776+0000", 2024-04-20T17:19:00.128 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 377.57477668299998, 2024-04-20T17:19:00.128 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.100849886, 2024-04-20T17:19:00.128 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:00.128 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:00.128 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:19:00.128 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:00.128 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.128 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:00.128 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.500776+0000", 2024-04-20T17:19:00.129 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.129 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.129 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.129 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:00.129 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.500776+0000", 2024-04-20T17:19:00.129 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.129 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.129 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.130 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:00.130 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.500778+0000", 2024-04-20T17:19:00.130 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2359999999999999e-06 2024-04-20T17:19:00.130 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.130 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.130 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:00.130 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.500786+0000", 2024-04-20T17:19:00.130 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3280000000000006e-06 2024-04-20T17:19:00.130 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.130 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.130 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:00.130 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.500799+0000", 2024-04-20T17:19:00.130 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3223e-05 2024-04-20T17:19:00.131 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.131 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.131 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:00.131 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.500808+0000", 2024-04-20T17:19:00.131 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.098e-06 2024-04-20T17:19:00.131 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.131 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.131 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:00.131 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.601598+0000", 2024-04-20T17:19:00.131 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.100790038 2024-04-20T17:19:00.131 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.131 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.131 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:00.131 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.601608+0000", 2024-04-20T17:19:00.132 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8349999999999999e-06 2024-04-20T17:19:00.132 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.132 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.132 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:00.132 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.601625+0000", 2024-04-20T17:19:00.132 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7127999999999999e-05 2024-04-20T17:19:00.132 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.132 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:00.132 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.132 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.132 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.132 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(osd.1.0:32103 2.6 e210/115 ondisk, result = 0)", 2024-04-20T17:19:00.132 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:42.500909+0000", 2024-04-20T17:19:00.133 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 377.57464349999998, 2024-04-20T17:19:00.133 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.100980444, 2024-04-20T17:19:00.133 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:00.133 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:00.133 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:19:00.133 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:00.133 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.133 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:00.133 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.500909+0000", 2024-04-20T17:19:00.133 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.133 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.133 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.133 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:00.133 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.500909+0000", 2024-04-20T17:19:00.133 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.134 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.134 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.134 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:00.134 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.500909+0000", 2024-04-20T17:19:00.134 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6700000000000003e-07 2024-04-20T17:19:00.134 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.134 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.134 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:00.134 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.500913+0000", 2024-04-20T17:19:00.134 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.366e-06 2024-04-20T17:19:00.134 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.134 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.134 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:00.135 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.500921+0000", 2024-04-20T17:19:00.135 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5879999999999998e-06 2024-04-20T17:19:00.135 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.135 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.135 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:00.135 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.500929+0000", 2024-04-20T17:19:00.135 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4869999999999996e-06 2024-04-20T17:19:00.135 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.135 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.135 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:00.135 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.601650+0000", 2024-04-20T17:19:00.135 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10072130899999999 2024-04-20T17:19:00.135 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.136 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.136 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:00.136 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.601655+0000", 2024-04-20T17:19:00.136 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8709999999999996e-06 2024-04-20T17:19:00.136 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.136 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.136 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:00.136 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.601889+0000", 2024-04-20T17:19:00.136 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00023405600000000001 2024-04-20T17:19:00.136 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.136 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:00.136 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.136 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.136 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.136 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:217254 2.6 2:67989c67:::benchmark_data_smithi171_78918_object56260:head [delete] snapc 0=[] ondisk+write+known_if_redirected e210)", 2024-04-20T17:19:00.137 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:42.610585+0000", 2024-04-20T17:19:00.137 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 377.46496764699998, 2024-04-20T17:19:00.137 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.111531326, 2024-04-20T17:19:00.137 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:00.137 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:00.137 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:00.137 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:19:00.137 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:19:00.137 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:00.137 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 217254 2024-04-20T17:19:00.137 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.137 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:00.138 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.138 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:00.138 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.610585+0000", 2024-04-20T17:19:00.138 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.138 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.138 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.138 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:00.138 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.610585+0000", 2024-04-20T17:19:00.138 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.138 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.138 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.139 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:00.139 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.610586+0000", 2024-04-20T17:19:00.139 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8700000000000001e-06 2024-04-20T17:19:00.139 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.139 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.139 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:00.139 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.610592+0000", 2024-04-20T17:19:00.139 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1889999999999998e-06 2024-04-20T17:19:00.139 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.139 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.139 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:00.139 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.610601+0000", 2024-04-20T17:19:00.139 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6029999999999994e-06 2024-04-20T17:19:00.139 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.140 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.140 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:00.140 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.610608+0000", 2024-04-20T17:19:00.140 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6980000000000001e-06 2024-04-20T17:19:00.140 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.140 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.140 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:00.140 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.610641+0000", 2024-04-20T17:19:00.140 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3491000000000002e-05 2024-04-20T17:19:00.140 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.140 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.140 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:00.140 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.610729+0000", 2024-04-20T17:19:00.140 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7793999999999993e-05 2024-04-20T17:19:00.141 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.141 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.141 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:19:00.141 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.610803+0000", 2024-04-20T17:19:00.141 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3292000000000002e-05 2024-04-20T17:19:00.141 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.141 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.141 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:19:00.141 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.614316+0000", 2024-04-20T17:19:00.141 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0035132879999999998 2024-04-20T17:19:00.141 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.141 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.141 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:00.142 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.722032+0000", 2024-04-20T17:19:00.142 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.107716066 2024-04-20T17:19:00.142 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.142 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.142 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:00.142 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.722070+0000", 2024-04-20T17:19:00.142 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8568000000000001e-05 2024-04-20T17:19:00.142 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.142 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.142 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:00.142 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.722116+0000", 2024-04-20T17:19:00.143 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5466999999999997e-05 2024-04-20T17:19:00.143 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.143 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:00.143 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.143 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.143 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.143 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:217333 2.6 2:6fe5f61e:::benchmark_data_smithi171_78918_object56339:head [delete] snapc 0=[] ondisk+write+known_if_redirected e211)", 2024-04-20T17:19:00.143 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:43.301777+0000", 2024-04-20T17:19:00.143 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 376.77377530500002, 2024-04-20T17:19:00.143 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.104629244, 2024-04-20T17:19:00.143 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:00.143 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:00.143 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:00.144 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:19:00.144 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:19:00.144 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:00.144 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 217333 2024-04-20T17:19:00.144 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.144 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:00.144 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.144 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:00.144 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:43.301777+0000", 2024-04-20T17:19:00.144 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.144 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.144 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.144 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:00.144 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:43.301777+0000", 2024-04-20T17:19:00.145 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.145 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.145 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.145 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:00.145 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:43.301778+0000", 2024-04-20T17:19:00.145 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4440000000000001e-06 2024-04-20T17:19:00.145 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.145 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.145 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:00.145 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:43.301782+0000", 2024-04-20T17:19:00.145 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1280000000000001e-06 2024-04-20T17:19:00.145 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.145 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.145 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:00.146 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:43.301792+0000", 2024-04-20T17:19:00.146 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2569999999999996e-06 2024-04-20T17:19:00.146 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.146 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.146 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:00.146 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:43.301798+0000", 2024-04-20T17:19:00.146 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1340000000000001e-06 2024-04-20T17:19:00.146 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.146 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.146 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:00.146 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:43.301827+0000", 2024-04-20T17:19:00.146 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9444000000000001e-05 2024-04-20T17:19:00.146 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.146 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.147 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:00.147 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:43.301911+0000", 2024-04-20T17:19:00.147 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3218000000000002e-05 2024-04-20T17:19:00.147 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.147 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.147 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:19:00.147 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:43.301987+0000", 2024-04-20T17:19:00.147 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6551999999999995e-05 2024-04-20T17:19:00.147 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.147 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.147 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:19:00.147 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:43.305370+0000", 2024-04-20T17:19:00.147 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0033824860000000001 2024-04-20T17:19:00.148 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.148 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.148 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:00.148 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:43.406310+0000", 2024-04-20T17:19:00.148 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10094076 2024-04-20T17:19:00.148 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.148 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.148 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:00.148 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:43.406347+0000", 2024-04-20T17:19:00.148 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6730000000000002e-05 2024-04-20T17:19:00.148 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.148 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.148 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:00.148 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:43.406406+0000", 2024-04-20T17:19:00.149 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9091000000000001e-05 2024-04-20T17:19:00.149 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.149 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:00.149 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.149 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.149 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.149 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:217607 2.6 2:699b2bdd:::benchmark_data_smithi171_78918_object56613:head [delete] snapc 0=[] ondisk+write+known_if_redirected e212)", 2024-04-20T17:19:00.149 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:44.238355+0000", 2024-04-20T17:19:00.149 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 375.83719722799998, 2024-04-20T17:19:00.149 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.112049101, 2024-04-20T17:19:00.149 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:00.149 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:00.149 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:00.150 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:19:00.150 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:19:00.150 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:00.150 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 217607 2024-04-20T17:19:00.150 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.150 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:00.150 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.150 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:00.150 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:44.238355+0000", 2024-04-20T17:19:00.150 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.151 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.151 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.151 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:00.151 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:44.238355+0000", 2024-04-20T17:19:00.151 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.151 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.151 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.151 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:00.151 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:44.238357+0000", 2024-04-20T17:19:00.151 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.249e-06 2024-04-20T17:19:00.151 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.151 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.151 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:00.151 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:44.238363+0000", 2024-04-20T17:19:00.152 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.592e-06 2024-04-20T17:19:00.152 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.152 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.152 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:00.152 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:44.238373+0000", 2024-04-20T17:19:00.152 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0083e-05 2024-04-20T17:19:00.152 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.152 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.152 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:00.152 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:44.238381+0000", 2024-04-20T17:19:00.152 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2150000000000001e-06 2024-04-20T17:19:00.152 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.153 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.153 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:00.153 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:44.327777+0000", 2024-04-20T17:19:00.153 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.089396327999999997 2024-04-20T17:19:00.153 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.153 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.153 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:00.153 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:44.327869+0000", 2024-04-20T17:19:00.153 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1675000000000006e-05 2024-04-20T17:19:00.153 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.153 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.153 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:19:00.153 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:44.327950+0000", 2024-04-20T17:19:00.153 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0946999999999996e-05 2024-04-20T17:19:00.153 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.154 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.154 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:19:00.154 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:44.331381+0000", 2024-04-20T17:19:00.154 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0034311739999999999 2024-04-20T17:19:00.154 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.154 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.154 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:00.154 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:44.350269+0000", 2024-04-20T17:19:00.154 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.018888139000000002 2024-04-20T17:19:00.154 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.154 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.154 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:00.154 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:44.350342+0000", 2024-04-20T17:19:00.154 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2323999999999994e-05 2024-04-20T17:19:00.155 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.155 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.155 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:00.155 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:44.350404+0000", 2024-04-20T17:19:00.155 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2374999999999998e-05 2024-04-20T17:19:00.155 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.155 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:00.155 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.155 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.155 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.155 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:218037 2.6 2:6deb4aaa:::benchmark_data_smithi171_78918_object57043:head [delete] snapc 0=[] ondisk+write+known_if_redirected e212)", 2024-04-20T17:19:00.155 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:46.049475+0000", 2024-04-20T17:19:00.156 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 374.02607720100002, 2024-04-20T17:19:00.156 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10739074899999999, 2024-04-20T17:19:00.156 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:00.156 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:00.156 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:00.156 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:19:00.156 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:19:00.156 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:00.156 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 218037 2024-04-20T17:19:00.156 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.156 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:00.156 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.156 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:00.156 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.049475+0000", 2024-04-20T17:19:00.156 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.157 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.157 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.157 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:00.157 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.049475+0000", 2024-04-20T17:19:00.157 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.157 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.157 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.157 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:00.157 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.049478+0000", 2024-04-20T17:19:00.157 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.914e-06 2024-04-20T17:19:00.157 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.157 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.157 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:00.157 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.049488+0000", 2024-04-20T17:19:00.157 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8810000000000008e-06 2024-04-20T17:19:00.157 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.158 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.158 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:00.158 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.049510+0000", 2024-04-20T17:19:00.158 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2116e-05 2024-04-20T17:19:00.158 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.158 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.158 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:00.158 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.049521+0000", 2024-04-20T17:19:00.158 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1358e-05 2024-04-20T17:19:00.158 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.158 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.158 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:00.158 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.049584+0000", 2024-04-20T17:19:00.158 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3163999999999994e-05 2024-04-20T17:19:00.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.159 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.159 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:00.159 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.049740+0000", 2024-04-20T17:19:00.159 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015564100000000001 2024-04-20T17:19:00.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.159 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.159 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:19:00.159 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.049877+0000", 2024-04-20T17:19:00.159 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013700999999999999 2024-04-20T17:19:00.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.159 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.159 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:19:00.159 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.054646+0000", 2024-04-20T17:19:00.159 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0047689020000000002 2024-04-20T17:19:00.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.160 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.160 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:00.160 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.156779+0000", 2024-04-20T17:19:00.160 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.102132614 2024-04-20T17:19:00.160 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.160 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.160 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:00.160 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.156815+0000", 2024-04-20T17:19:00.160 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6798000000000002e-05 2024-04-20T17:19:00.160 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.160 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.160 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:00.160 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.156866+0000", 2024-04-20T17:19:00.160 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0351e-05 2024-04-20T17:19:00.161 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.161 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:00.161 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.161 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.161 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.161 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:218053 2.6 2:6c335b6c:::benchmark_data_smithi171_78918_object57059:head [delete] snapc 0=[] ondisk+write+known_if_redirected e212)", 2024-04-20T17:19:00.161 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:46.175572+0000", 2024-04-20T17:19:00.161 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 373.89998017800002, 2024-04-20T17:19:00.161 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10556868, 2024-04-20T17:19:00.161 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:00.161 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:00.161 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:00.161 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:19:00.162 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:19:00.162 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:00.162 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 218053 2024-04-20T17:19:00.162 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.162 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:00.162 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.162 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:00.162 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.175572+0000", 2024-04-20T17:19:00.162 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.162 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.162 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.162 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:00.162 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.175572+0000", 2024-04-20T17:19:00.162 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.163 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.163 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.163 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:00.163 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.175575+0000", 2024-04-20T17:19:00.163 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9359999999999999e-06 2024-04-20T17:19:00.163 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.163 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.163 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:00.163 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.175583+0000", 2024-04-20T17:19:00.163 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0479999999999999e-06 2024-04-20T17:19:00.163 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.163 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.163 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:00.163 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.175598+0000", 2024-04-20T17:19:00.164 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4854999999999999e-05 2024-04-20T17:19:00.164 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.164 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.164 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:00.164 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.175610+0000", 2024-04-20T17:19:00.164 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1688000000000001e-05 2024-04-20T17:19:00.164 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.164 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.164 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:00.164 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.269744+0000", 2024-04-20T17:19:00.164 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.094134526999999996 2024-04-20T17:19:00.164 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.164 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.164 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:00.164 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.269830+0000", 2024-04-20T17:19:00.165 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6122e-05 2024-04-20T17:19:00.165 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.165 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.165 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:19:00.165 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.269904+0000", 2024-04-20T17:19:00.165 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3610999999999997e-05 2024-04-20T17:19:00.165 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.165 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.165 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:19:00.165 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.280955+0000", 2024-04-20T17:19:00.165 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011051518999999999 2024-04-20T17:19:00.165 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.165 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.165 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:00.166 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.281015+0000", 2024-04-20T17:19:00.166 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9401999999999998e-05 2024-04-20T17:19:00.166 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.166 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.166 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:00.166 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.281077+0000", 2024-04-20T17:19:00.166 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2491999999999997e-05 2024-04-20T17:19:00.166 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.166 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.166 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:00.166 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.281141+0000", 2024-04-20T17:19:00.166 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3479999999999995e-05 2024-04-20T17:19:00.166 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.166 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:00.167 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.167 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.167 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.167 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:218337 2.6 2:66af67bd:::benchmark_data_smithi171_78918_object57343:head [delete] snapc 0=[] ondisk+write+known_if_redirected e213)", 2024-04-20T17:19:00.167 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:47.343662+0000", 2024-04-20T17:19:00.167 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 372.73189027500001, 2024-04-20T17:19:00.167 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10596135800000001, 2024-04-20T17:19:00.167 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:00.167 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:00.167 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:00.167 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:19:00.167 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:19:00.167 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:00.167 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 218337 2024-04-20T17:19:00.167 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.168 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:00.168 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.168 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:00.168 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:47.343662+0000", 2024-04-20T17:19:00.168 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.168 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.168 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.168 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:00.168 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:47.343662+0000", 2024-04-20T17:19:00.168 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.168 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.168 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.168 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:00.169 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:47.343665+0000", 2024-04-20T17:19:00.169 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8080000000000001e-06 2024-04-20T17:19:00.169 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.169 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.169 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:00.169 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:47.343673+0000", 2024-04-20T17:19:00.169 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1009999999999992e-06 2024-04-20T17:19:00.169 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.169 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.169 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:00.169 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:47.343685+0000", 2024-04-20T17:19:00.169 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2547e-05 2024-04-20T17:19:00.169 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.169 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.170 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:00.170 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:47.343701+0000", 2024-04-20T17:19:00.170 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5223e-05 2024-04-20T17:19:00.170 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.170 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.170 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:00.170 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:47.433742+0000", 2024-04-20T17:19:00.170 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.090041287999999997 2024-04-20T17:19:00.170 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.170 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.170 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:00.170 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:47.433834+0000", 2024-04-20T17:19:00.170 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2311000000000004e-05 2024-04-20T17:19:00.170 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.171 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.171 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:19:00.171 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:47.433930+0000", 2024-04-20T17:19:00.171 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6169999999999996e-05 2024-04-20T17:19:00.171 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.171 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.171 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:19:00.171 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:47.437577+0000", 2024-04-20T17:19:00.171 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0036469300000000001 2024-04-20T17:19:00.171 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.171 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.171 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:00.171 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:47.449548+0000", 2024-04-20T17:19:00.171 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011970824999999999 2024-04-20T17:19:00.171 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.172 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.172 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:00.172 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:47.449585+0000", 2024-04-20T17:19:00.172 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6810999999999997e-05 2024-04-20T17:19:00.172 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.172 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.172 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:00.172 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:47.449623+0000", 2024-04-20T17:19:00.172 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8343999999999997e-05 2024-04-20T17:19:00.172 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.172 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:00.172 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.172 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.172 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.172 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:218701 2.6 2:6d43b903:::benchmark_data_smithi171_78918_object57707:head [delete] snapc 0=[] ondisk+write+known_if_redirected e213)", 2024-04-20T17:19:00.173 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:49.344998+0000", 2024-04-20T17:19:00.173 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 370.73055433000002, 2024-04-20T17:19:00.173 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10736288099999999, 2024-04-20T17:19:00.173 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:00.173 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:00.173 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:00.173 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:19:00.173 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:19:00.173 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:00.173 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 218701 2024-04-20T17:19:00.173 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.173 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:00.173 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.173 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:00.174 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.344998+0000", 2024-04-20T17:19:00.174 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.174 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.174 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.174 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:00.174 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.344998+0000", 2024-04-20T17:19:00.174 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.174 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.174 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.174 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:00.174 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.345000+0000", 2024-04-20T17:19:00.174 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1730000000000002e-06 2024-04-20T17:19:00.174 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.174 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.175 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:00.175 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.345006+0000", 2024-04-20T17:19:00.175 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1489999999999999e-06 2024-04-20T17:19:00.175 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.175 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.175 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:00.175 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.345017+0000", 2024-04-20T17:19:00.175 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0341e-05 2024-04-20T17:19:00.175 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.175 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.175 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:00.175 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.345024+0000", 2024-04-20T17:19:00.175 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3699999999999997e-06 2024-04-20T17:19:00.175 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.175 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.176 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:00.176 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.444845+0000", 2024-04-20T17:19:00.176 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.099821550999999994 2024-04-20T17:19:00.176 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.176 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.176 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:00.176 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.444949+0000", 2024-04-20T17:19:00.176 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010375200000000001 2024-04-20T17:19:00.176 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.176 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.176 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:19:00.176 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.445038+0000", 2024-04-20T17:19:00.176 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8702999999999994e-05 2024-04-20T17:19:00.176 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.176 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.177 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:19:00.177 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.449444+0000", 2024-04-20T17:19:00.177 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0044062459999999999 2024-04-20T17:19:00.177 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.177 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.177 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:00.177 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.452220+0000", 2024-04-20T17:19:00.177 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0027756500000000002 2024-04-20T17:19:00.177 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.177 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.177 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:00.177 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.452303+0000", 2024-04-20T17:19:00.177 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2978000000000004e-05 2024-04-20T17:19:00.177 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.177 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.178 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:00.178 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.452361+0000", 2024-04-20T17:19:00.178 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7967999999999998e-05 2024-04-20T17:19:00.178 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.178 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:00.178 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.178 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.178 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.178 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(osd.1.0:33705 2.6 e213/115 ondisk, result = 0)", 2024-04-20T17:19:00.178 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:49.349222+0000", 2024-04-20T17:19:00.178 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 370.72633037999998, 2024-04-20T17:19:00.178 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10013370100000001, 2024-04-20T17:19:00.178 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:00.179 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:00.179 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:19:00.179 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:00.179 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.179 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:00.179 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.349222+0000", 2024-04-20T17:19:00.179 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.179 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.179 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.179 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:00.179 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.349222+0000", 2024-04-20T17:19:00.179 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.179 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.179 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.180 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:00.180 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.349224+0000", 2024-04-20T17:19:00.180 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.674e-06 2024-04-20T17:19:00.180 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.180 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.180 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:00.180 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.349230+0000", 2024-04-20T17:19:00.180 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4099999999999996e-06 2024-04-20T17:19:00.180 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.180 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.180 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:00.180 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.349240+0000", 2024-04-20T17:19:00.180 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6150000000000003e-06 2024-04-20T17:19:00.180 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.181 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.181 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:00.181 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.349248+0000", 2024-04-20T17:19:00.181 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.755e-06 2024-04-20T17:19:00.181 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.181 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.181 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:00.181 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.449096+0000", 2024-04-20T17:19:00.181 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.099847801999999999 2024-04-20T17:19:00.181 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.181 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.181 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:00.181 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.449107+0000", 2024-04-20T17:19:00.181 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1037e-05 2024-04-20T17:19:00.182 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.182 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.182 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:00.182 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.449356+0000", 2024-04-20T17:19:00.182 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00024840799999999998 2024-04-20T17:19:00.182 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.182 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:00.182 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.182 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.182 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.182 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(osd.1.0:33759 2.6 e213/115 ondisk, result = 0)", 2024-04-20T17:19:00.182 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:49.593394+0000", 2024-04-20T17:19:00.182 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 370.48215870899998, 2024-04-20T17:19:00.183 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10355750800000001, 2024-04-20T17:19:00.183 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:00.183 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:00.183 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:19:00.183 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:00.183 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.183 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:00.183 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.593394+0000", 2024-04-20T17:19:00.183 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.183 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.183 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.183 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:00.183 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.593394+0000", 2024-04-20T17:19:00.183 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:00.184 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.184 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.184 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:00.184 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.593395+0000", 2024-04-20T17:19:00.184 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.209e-06 2024-04-20T17:19:00.184 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.184 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.184 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:00.184 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.593399+0000", 2024-04-20T17:19:00.184 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2939999999999999e-06 2024-04-20T17:19:00.184 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.184 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.184 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:00.184 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.593408+0000", 2024-04-20T17:19:00.184 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5359999999999992e-06 2024-04-20T17:19:00.185 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.185 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.185 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:00.185 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.593414+0000", 2024-04-20T17:19:00.185 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3749999999999999e-06 2024-04-20T17:19:00.185 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.185 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.185 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:00.185 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.696853+0000", 2024-04-20T17:19:00.185 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10343883800000001 2024-04-20T17:19:00.185 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.185 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.185 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:00.185 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.696862+0000", 2024-04-20T17:19:00.186 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.789e-06 2024-04-20T17:19:00.186 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:00.186 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:00.186 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:00.186 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.696951+0000", 2024-04-20T17:19:00.186 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9467000000000002e-05 2024-04-20T17:19:00.186 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.186 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:00.186 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.186 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:00.186 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:00.186 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:19:00.187 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:19:00.211 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:00 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:19:00.211 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:00 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:19:00.211 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:00 smithi171 ceph-mon[44189]: osdmap e329: 12 total, 12 up, 12 in 2024-04-20T17:19:00.211 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:00 smithi171 ceph-mon[44189]: from='client.25634 v1:172.21.15.5:0/1961608057' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:19:00.211 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:00 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1639958236' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:19:00.213 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:00.233 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:19:00.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:00.236 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:19:00.300 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:19:00.301 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:19:00.301 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:19:00.301 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:19:00.301 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:19:00.301 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:19:00.301 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:19:00.301 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:19:00.301 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:19:00.301 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:19:00.301 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:19:00.301 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:19:00.302 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:19:00.302 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:19:00.302 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:19:00.302 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:19:00.302 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:19:00.302 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:19:00.302 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:19:00.302 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:19:00.302 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:19:00.302 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:19:00.302 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:19:00.302 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:19:00.337 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:19:00.363 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:00 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:19:00.363 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:00 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:19:00.364 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:00 smithi005 ceph-mon[39562]: osdmap e329: 12 total, 12 up, 12 in 2024-04-20T17:19:00.364 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:00 smithi005 ceph-mon[39562]: from='client.25634 v1:172.21.15.5:0/1961608057' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:19:00.364 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:00 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1639958236' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:19:00.366 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:00.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:00 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:19:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:00 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:19:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:00 smithi106 ceph-mon[44423]: osdmap e329: 12 total, 12 up, 12 in 2024-04-20T17:19:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:00 smithi106 ceph-mon[44423]: from='client.25634 v1:172.21.15.5:0/1961608057' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:19:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:00 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1639958236' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:19:00.917 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:19:00.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:00.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:19:00.936 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":905,"stamp":"2024-04-20T17:18:59.167183+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2493383224,"num_objects":38040,"num_object_clones":0,"num_object_copies":76080,"num_objects_missing_on_primary":1819,"num_objects_missing":1819,"num_objects_degraded":0,"num_objects_misplaced":3650,"num_objects_unfound":0,"num_objects_dirty":38040,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":284127,"num_write_kb":10306363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27393,"num_bytes_recovered":1796819072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160882,"ondisk_log_size":160882,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":6922812,"kb_used_data":5720704,"kb_used_omap":219,"kb_used_meta":1201764,"kb_avail":1118019012,"statfs":{"total":1151940427776,"available":1144851468288,"internally_reserved":0,"allocated":5858000896,"data_stored":5844250692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224949,"internal_metadata":1230606667},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-147849216,"num_objects":-2256,"num_object_clones":0,"num_object_copies":-4512,"num_objects_missing_on_primary":-186,"num_objects_missing":-186,"num_objects_degraded":0,"num_objects_misplaced":-365,"num_objects_unfound":0,"num_objects_dirty":-2256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2256,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":179,"num_bytes_recovered":11730944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.092671"},"pg_stats":[{"pgid":"2.f","version":"328'17837","reported_seq":18823,"reported_epoch":328,"state":"active+clean","last_fresh":"2024-04-20T17:18:58.733405+0000","last_change":"2024-04-20T17:17:55.070885+0000","last_active":"2024-04-20T17:18:58.733405+0000","last_peered":"2024-04-20T17:18:58.733405+0000","last_clean":"2024-04-20T17:18:58.733405+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:18:58.733405+0000","last_undegraded":"2024-04-20T17:18:58.733405+0000","last_fullsized":"2024-04-20T17:18:58.733405+0000","mapping_epoch":115,"log_start":"141'7800","ondisk_log_start":"141'7800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'16821","last_scrub_stamp":"2024-04-20T17:17:55.070768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:55.070768+0000","objects_scrubbed":3362,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":356,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153681920,"num_objects":2345,"num_object_clones":0,"num_object_copies":4690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17837,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"327'17620","reported_seq":29670,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-04-20T17:18:55.527602+0000","last_change":"2024-04-20T17:18:08.306341+0000","last_active":"2024-04-20T17:18:55.527602+0000","last_peered":"2024-04-20T17:18:55.527602+0000","last_clean":"2024-04-20T17:18:55.527602+0000","last_became_active":"2024-04-20T17:10:28.865670+0000","last_became_peered":"2024-04-20T17:10:28.865670+0000","last_unstale":"2024-04-20T17:18:55.527602+0000","last_undegraded":"2024-04-20T17:18:55.527602+0000","last_fullsized":"2024-04-20T17:18:55.527602+0000","mapping_epoch":162,"log_start":"140'7600","ondisk_log_start":"140'7600","created":115,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'16860","last_scrub_stamp":"2024-04-20T17:18:08.306156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:08.306156+0000","objects_scrubbed":3197,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:34.841351+0000","scrub_duration":281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159645696,"num_objects":2436,"num_object_clones":0,"num_object_copies":4872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17620,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5619,"num_bytes_recovered":368246784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"327'18006","reported_seq":18871,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-04-20T17:18:55.411394+0000","last_change":"2024-04-20T17:18:49.298437+0000","last_active":"2024-04-20T17:18:55.411394+0000","last_peered":"2024-04-20T17:18:55.411394+0000","last_clean":"2024-04-20T17:18:55.411394+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:18:55.411394+0000","last_undegraded":"2024-04-20T17:18:55.411394+0000","last_fullsized":"2024-04-20T17:18:55.411394+0000","mapping_epoch":115,"log_start":"141'8000","ondisk_log_start":"141'8000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'17915","last_scrub_stamp":"2024-04-20T17:18:49.298307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:49.298307+0000","objects_scrubbed":2567,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:03.315951+0000","scrub_duration":225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162267136,"num_objects":2476,"num_object_clones":0,"num_object_copies":4952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18006,"num_write_kb":655424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"327'24393","reported_seq":46021,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-04-20T17:18:55.658020+0000","last_change":"2024-04-20T17:17:53.095109+0000","last_active":"2024-04-20T17:18:55.658020+0000","last_peered":"2024-04-20T17:18:55.658020+0000","last_clean":"2024-04-20T17:18:55.658020+0000","last_became_active":"2024-04-20T17:12:08.085936+0000","last_became_peered":"2024-04-20T17:12:08.085936+0000","last_unstale":"2024-04-20T17:18:55.658020+0000","last_undegraded":"2024-04-20T17:18:55.658020+0000","last_fullsized":"2024-04-20T17:18:55.658020+0000","mapping_epoch":196,"log_start":"207'14368","ondisk_log_start":"207'14368","created":115,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'23389","last_scrub_stamp":"2024-04-20T17:17:53.094985+0000","last_deep_scrub":"213'20373","last_deep_scrub_stamp":"2024-04-20T17:12:48.443808+0000","last_clean_scrub_stamp":"2024-04-20T17:17:53.094985+0000","objects_scrubbed":3498,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:20.542939+0000","scrub_duration":306,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163381248,"num_objects":2493,"num_object_clones":0,"num_object_copies":4986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17741,"num_write_kb":647488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":450166784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"328'17469","reported_seq":40726,"reported_epoch":328,"state":"active+recovering","last_fresh":"2024-04-20T17:18:58.796010+0000","last_change":"2024-04-20T17:12:26.166109+0000","last_active":"2024-04-20T17:18:58.796010+0000","last_peered":"2024-04-20T17:18:58.796010+0000","last_clean":"2024-04-20T17:09:35.882603+0000","last_became_active":"2024-04-20T17:10:54.470253+0000","last_became_peered":"2024-04-20T17:10:54.470253+0000","last_unstale":"2024-04-20T17:18:58.796010+0000","last_undegraded":"2024-04-20T17:18:58.796010+0000","last_fullsized":"2024-04-20T17:18:58.796010+0000","mapping_epoch":171,"log_start":"139'7400","ondisk_log_start":"139'7400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148176896,"num_objects":2261,"num_object_clones":0,"num_object_copies":4522,"num_objects_missing_on_primary":1819,"num_objects_missing":1819,"num_objects_degraded":0,"num_objects_misplaced":3650,"num_objects_unfound":0,"num_objects_dirty":2261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17469,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7601,"num_bytes_recovered":498139136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"0,2,5","objects":12},{"shards":"0,10","objects":430},{"shards":"2,5","objects":1819}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":860,"reported_epoch":328,"state":"active+clean","last_fresh":"2024-04-20T17:18:56.009589+0000","last_change":"2024-04-20T17:17:58.199584+0000","last_active":"2024-04-20T17:18:56.009589+0000","last_peered":"2024-04-20T17:18:56.009589+0000","last_clean":"2024-04-20T17:18:56.009589+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:18:56.009589+0000","last_undegraded":"2024-04-20T17:18:56.009589+0000","last_fullsized":"2024-04-20T17:18:56.009589+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:17:58.199502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:17:58.199502+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:16.100652+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"328'20870","reported_seq":21864,"reported_epoch":328,"state":"active+clean","last_fresh":"2024-04-20T17:18:57.925655+0000","last_change":"2024-04-20T17:18:28.829916+0000","last_active":"2024-04-20T17:18:57.925655+0000","last_peered":"2024-04-20T17:18:57.925655+0000","last_clean":"2024-04-20T17:18:57.925655+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:18:57.925655+0000","last_undegraded":"2024-04-20T17:18:57.925655+0000","last_fullsized":"2024-04-20T17:18:57.925655+0000","mapping_epoch":115,"log_start":"154'10800","ondisk_log_start":"154'10800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'20320","last_scrub_stamp":"2024-04-20T17:18:28.829770+0000","last_deep_scrub":"321'20320","last_deep_scrub_stamp":"2024-04-20T17:18:28.829770+0000","last_clean_scrub_stamp":"2024-04-20T17:18:28.829770+0000","objects_scrubbed":3006,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":10,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:40.227886+0000","scrub_duration":9561,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156499968,"num_objects":2388,"num_object_clones":0,"num_object_copies":4776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17864,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"328'17993","reported_seq":18890,"reported_epoch":328,"state":"active+clean","last_fresh":"2024-04-20T17:18:58.716742+0000","last_change":"2024-04-20T17:18:49.142560+0000","last_active":"2024-04-20T17:18:58.716742+0000","last_peered":"2024-04-20T17:18:58.716742+0000","last_clean":"2024-04-20T17:18:58.716742+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:18:58.716742+0000","last_undegraded":"2024-04-20T17:18:58.716742+0000","last_fullsized":"2024-04-20T17:18:58.716742+0000","mapping_epoch":115,"log_start":"141'7900","ondisk_log_start":"141'7900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'17862","last_scrub_stamp":"2024-04-20T17:18:49.142459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:49.142459+0000","objects_scrubbed":2464,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:07.477325+0000","scrub_duration":245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152764416,"num_objects":2331,"num_object_clones":0,"num_object_copies":4662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17993,"num_write_kb":650368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"328'21829","reported_seq":22833,"reported_epoch":328,"state":"active+clean","last_fresh":"2024-04-20T17:18:56.923521+0000","last_change":"2024-04-20T17:17:55.709069+0000","last_active":"2024-04-20T17:18:56.923521+0000","last_peered":"2024-04-20T17:18:56.923521+0000","last_clean":"2024-04-20T17:18:56.923521+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:18:56.923521+0000","last_undegraded":"2024-04-20T17:18:56.923521+0000","last_fullsized":"2024-04-20T17:18:56.923521+0000","mapping_epoch":115,"log_start":"160'11802","ondisk_log_start":"160'11802","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'20878","last_scrub_stamp":"2024-04-20T17:17:55.708833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:55.708833+0000","objects_scrubbed":3275,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:11.407326+0000","scrub_duration":320,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152305664,"num_objects":2324,"num_object_clones":0,"num_object_copies":4648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17718,"num_write_kb":641344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"328'17858","reported_seq":18813,"reported_epoch":328,"state":"active+clean","last_fresh":"2024-04-20T17:18:57.228496+0000","last_change":"2024-04-20T17:18:45.620454+0000","last_active":"2024-04-20T17:18:57.228496+0000","last_peered":"2024-04-20T17:18:57.228496+0000","last_clean":"2024-04-20T17:18:57.228496+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:18:57.228496+0000","last_undegraded":"2024-04-20T17:18:57.228496+0000","last_fullsized":"2024-04-20T17:18:57.228496+0000","mapping_epoch":115,"log_start":"140'7800","ondisk_log_start":"140'7800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'17642","last_scrub_stamp":"2024-04-20T17:18:45.620321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:45.620321+0000","objects_scrubbed":2579,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:50.964012+0000","scrub_duration":253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154796032,"num_objects":2362,"num_object_clones":0,"num_object_copies":4724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17858,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"328'17654","reported_seq":18533,"reported_epoch":328,"state":"active+clean","last_fresh":"2024-04-20T17:18:57.906899+0000","last_change":"2024-04-20T17:18:45.702943+0000","last_active":"2024-04-20T17:18:57.906899+0000","last_peered":"2024-04-20T17:18:57.906899+0000","last_clean":"2024-04-20T17:18:57.906899+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:18:57.906899+0000","last_undegraded":"2024-04-20T17:18:57.906899+0000","last_fullsized":"2024-04-20T17:18:57.906899+0000","mapping_epoch":115,"log_start":"140'7600","ondisk_log_start":"140'7600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'17455","last_scrub_stamp":"2024-04-20T17:18:45.702769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:45.702769+0000","objects_scrubbed":2599,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:54.717372+0000","scrub_duration":222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157155328,"num_objects":2398,"num_object_clones":0,"num_object_copies":4796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17654,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"328'17874","reported_seq":33321,"reported_epoch":328,"state":"active+clean","last_fresh":"2024-04-20T17:18:57.872932+0000","last_change":"2024-04-20T17:17:57.501217+0000","last_active":"2024-04-20T17:18:57.872932+0000","last_peered":"2024-04-20T17:18:57.872932+0000","last_clean":"2024-04-20T17:18:57.872932+0000","last_became_active":"2024-04-20T17:10:55.931751+0000","last_became_peered":"2024-04-20T17:10:55.931751+0000","last_unstale":"2024-04-20T17:18:57.872932+0000","last_undegraded":"2024-04-20T17:18:57.872932+0000","last_fullsized":"2024-04-20T17:18:57.872932+0000","mapping_epoch":173,"log_start":"141'7800","ondisk_log_start":"141'7800","created":115,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'16916","last_scrub_stamp":"2024-04-20T17:17:57.501089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:57.501089+0000","objects_scrubbed":3350,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:26.178414+0000","scrub_duration":318,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156499968,"num_objects":2388,"num_object_clones":0,"num_object_copies":4776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17874,"num_write_kb":648384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7296,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"328'17937","reported_seq":18841,"reported_epoch":328,"state":"active+clean","last_fresh":"2024-04-20T17:18:57.496740+0000","last_change":"2024-04-20T17:18:06.314140+0000","last_active":"2024-04-20T17:18:57.496740+0000","last_peered":"2024-04-20T17:18:57.496740+0000","last_clean":"2024-04-20T17:18:57.496740+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:18:57.496740+0000","last_undegraded":"2024-04-20T17:18:57.496740+0000","last_fullsized":"2024-04-20T17:18:57.496740+0000","mapping_epoch":115,"log_start":"141'7900","ondisk_log_start":"141'7900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"314'17127","last_scrub_stamp":"2024-04-20T17:18:06.313989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:06.313989+0000","objects_scrubbed":3191,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:11.886992+0000","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155910144,"num_objects":2379,"num_object_clones":0,"num_object_copies":4758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17937,"num_write_kb":650112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"328'17494","reported_seq":18387,"reported_epoch":328,"state":"active+clean","last_fresh":"2024-04-20T17:18:58.737758+0000","last_change":"2024-04-20T17:18:47.241194+0000","last_active":"2024-04-20T17:18:58.737758+0000","last_peered":"2024-04-20T17:18:58.737758+0000","last_clean":"2024-04-20T17:18:58.737758+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:18:58.737758+0000","last_undegraded":"2024-04-20T17:18:58.737758+0000","last_fullsized":"2024-04-20T17:18:58.737758+0000","mapping_epoch":115,"log_start":"139'7400","ondisk_log_start":"139'7400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'17283","last_scrub_stamp":"2024-04-20T17:18:47.241064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:47.241064+0000","objects_scrubbed":2584,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:10.974140+0000","scrub_duration":266,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155320320,"num_objects":2370,"num_object_clones":0,"num_object_copies":4740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17494,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"328'17722","reported_seq":18615,"reported_epoch":328,"state":"active+clean","last_fresh":"2024-04-20T17:18:57.529048+0000","last_change":"2024-04-20T17:18:48.599315+0000","last_active":"2024-04-20T17:18:57.529048+0000","last_peered":"2024-04-20T17:18:57.529048+0000","last_clean":"2024-04-20T17:18:57.529048+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:18:57.529048+0000","last_undegraded":"2024-04-20T17:18:57.529048+0000","last_fullsized":"2024-04-20T17:18:57.529048+0000","mapping_epoch":115,"log_start":"140'7700","ondisk_log_start":"140'7700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'17566","last_scrub_stamp":"2024-04-20T17:18:48.599209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:48.599209+0000","objects_scrubbed":2538,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:05.548836+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155975680,"num_objects":2380,"num_object_clones":0,"num_object_copies":4760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17722,"num_write_kb":643264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"327'17537","reported_seq":18427,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-04-20T17:18:55.651209+0000","last_change":"2024-04-20T17:18:52.214316+0000","last_active":"2024-04-20T17:18:55.651209+0000","last_peered":"2024-04-20T17:18:55.651209+0000","last_clean":"2024-04-20T17:18:55.651209+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:18:55.651209+0000","last_undegraded":"2024-04-20T17:18:55.651209+0000","last_fullsized":"2024-04-20T17:18:55.651209+0000","mapping_epoch":115,"log_start":"139'7500","ondisk_log_start":"139'7500","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'17470","last_scrub_stamp":"2024-04-20T17:18:52.214208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:52.214208+0000","objects_scrubbed":2403,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:55.920246+0000","scrub_duration":228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152829976,"num_objects":2333,"num_object_clones":0,"num_object_copies":4666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2333,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":17537,"num_write_kb":635777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"327'17622","reported_seq":18391,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-04-20T17:18:55.886157+0000","last_change":"2024-04-20T17:18:17.426012+0000","last_active":"2024-04-20T17:18:55.886157+0000","last_peered":"2024-04-20T17:18:55.886157+0000","last_clean":"2024-04-20T17:18:55.886157+0000","last_became_active":"2024-04-20T17:12:23.515375+0000","last_became_peered":"2024-04-20T17:12:23.515375+0000","last_unstale":"2024-04-20T17:18:55.886157+0000","last_undegraded":"2024-04-20T17:18:55.886157+0000","last_fullsized":"2024-04-20T17:18:55.886157+0000","mapping_epoch":202,"log_start":"139'7600","ondisk_log_start":"139'7600","created":115,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'16928","last_scrub_stamp":"2024-04-20T17:18:17.425878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:17.425878+0000","objects_scrubbed":3071,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:47.218525+0000","scrub_duration":353,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155582464,"num_objects":2374,"num_object_clones":0,"num_object_copies":4748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17622,"num_write_kb":639872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":2492792856,"num_objects":38038,"num_object_clones":0,"num_object_copies":76076,"num_objects_missing_on_primary":1819,"num_objects_missing":1819,"num_objects_degraded":0,"num_objects_misplaced":3650,"num_objects_unfound":0,"num_objects_dirty":38038,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":283946,"num_write_kb":10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27385,"num_bytes_recovered":1794703360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5797584896,"data_stored":5797576752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},"log_size":160745,"ondisk_log_size":160745,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991391,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":536920,"kb_used_data":461328,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93208232,"statfs":{"total":95995035648,"available":95445229568,"internally_reserved":0,"allocated":472399872,"data_stored":471246742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":77396420},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253029,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":668564,"kb_used_data":484680,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93076588,"statfs":{"total":95995035648,"available":95310426112,"internally_reserved":0,"allocated":496312320,"data_stored":495162742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":188283328},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612778,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":770236,"kb_used_data":626612,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92974916,"statfs":{"total":95995035648,"available":95206313984,"internally_reserved":0,"allocated":641650688,"data_stored":640509014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":147061184},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809008,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":708632,"kb_used_data":605256,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93036520,"statfs":{"total":95995035648,"available":95269396480,"internally_reserved":0,"allocated":619782144,"data_stored":618632586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168757,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":576024,"kb_used_data":464336,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93169128,"statfs":{"total":95995035648,"available":95405187072,"internally_reserved":0,"allocated":475480064,"data_stored":474326954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":114358730},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430393,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":418648,"kb_used_data":311112,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93326504,"statfs":{"total":95995035648,"available":95566340096,"internally_reserved":0,"allocated":318578688,"data_stored":317429130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575059,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":408920,"kb_used_data":309704,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93336232,"statfs":{"total":95995035648,"available":95576301568,"internally_reserved":0,"allocated":317136896,"data_stored":315987338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738584,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":624080,"kb_used_data":484616,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93121072,"statfs":{"total":95995035648,"available":95355977728,"internally_reserved":0,"allocated":496246784,"data_stored":495097226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393872,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":625592,"kb_used_data":585520,"kb_used_omap":202,"kb_used_meta":39861,"kb_avail":93119560,"statfs":{"total":95995035648,"available":95354429440,"internally_reserved":0,"allocated":599572480,"data_stored":598434878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":40818135},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182600,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":379512,"kb_used_data":312240,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93365640,"statfs":{"total":95995035648,"available":95606415360,"internally_reserved":0,"allocated":319733760,"data_stored":318596158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986371,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":184700,"kb_used_data":153524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93560452,"statfs":{"total":95995035648,"available":95805902848,"internally_reserved":0,"allocated":157208576,"data_stored":156066902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790141,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1020984,"kb_used_data":921776,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92724168,"statfs":{"total":95995035648,"available":94949548032,"internally_reserved":0,"allocated":943898624,"data_stored":942761022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":490995712,"data_stored":490995712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":312147968,"data_stored":312147968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":595787776,"data_stored":595787776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":314245120,"data_stored":314245120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":153096192,"data_stored":153092120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":940310528,"data_stored":940310528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":316145664,"data_stored":316145664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":470155264,"data_stored":470155264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":611975168,"data_stored":611975168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":633016320,"data_stored":633012248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":490733568,"data_stored":490733568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":468975616,"data_stored":468975616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:19:00.937 INFO:tasks.cephadm.ceph_manager.ceph:PG 2.3 is not active+clean 2024-04-20T17:19:00.937 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '2.3', 'version': "328'17469", 'reported_seq': 40726, 'reported_epoch': 328, 'state': 'active+recovering', 'last_fresh': '2024-04-20T17:18:58.796010+0000', 'last_change': '2024-04-20T17:12:26.166109+0000', 'last_active': '2024-04-20T17:18:58.796010+0000', 'last_peered': '2024-04-20T17:18:58.796010+0000', 'last_clean': '2024-04-20T17:09:35.882603+0000', 'last_became_active': '2024-04-20T17:10:54.470253+0000', 'last_became_peered': '2024-04-20T17:10:54.470253+0000', 'last_unstale': '2024-04-20T17:18:58.796010+0000', 'last_undegraded': '2024-04-20T17:18:58.796010+0000', 'last_fullsized': '2024-04-20T17:18:58.796010+0000', 'mapping_epoch': 171, 'log_start': "139'7400", 'ondisk_log_start': "139'7400", 'created': 115, 'last_epoch_clean': 116, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-20T17:08:04.990639+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T17:08:04.990639+0000', 'last_clean_scrub_stamp': '2024-04-20T17:08:04.990639+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 148176896, 'num_objects': 2261, 'num_object_clones': 0, 'num_object_copies': 4522, 'num_objects_missing_on_primary': 1819, 'num_objects_missing': 1819, 'num_objects_degraded': 0, 'num_objects_misplaced': 3650, 'num_objects_unfound': 0, 'num_objects_dirty': 2261, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17469, 'num_write_kb': 631360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7601, 'num_bytes_recovered': 498139136, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 10], 'acting': [0, 10], 'avail_no_missing': ['2', '5'], 'object_location_counts': [{'shards': '0,2,5', 'objects': 12}, {'shards': '0,10', 'objects': 430}, {'shards': '2,5', 'objects': 1819}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-20T17:19:00.937 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph status --format=json 2024-04-20T17:19:01.012 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:19:01.026 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:19:01.053 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:01.072 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:19:01.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:01.075 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:19:01.175 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.175 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.175 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.175 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.175 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.175 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.175 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.175 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:01 smithi005 ceph-mon[39562]: pgmap v906: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 241 op/s; 3650/76080 objects misplaced (4.798%); 992 KiB/s, 15 objects/s recovering 2024-04-20T17:19:01.175 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.175 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.175 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.176 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.176 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.176 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:01 smithi005 ceph-mon[39562]: osdmap e330: 12 total, 12 up, 12 in 2024-04-20T17:19:01.176 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:19:01.202 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:01.221 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:19:01.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:01.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:19:01.325 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:19:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:01 smithi171 ceph-mon[44189]: pgmap v906: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 241 op/s; 3650/76080 objects misplaced (4.798%); 992 KiB/s, 15 objects/s recovering 2024-04-20T17:19:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:01 smithi171 ceph-mon[44189]: osdmap e330: 12 total, 12 up, 12 in 2024-04-20T17:19:01.351 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:01.370 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:19:01.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:01.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:19:01.474 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:19:01.500 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:01.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:01 smithi106 ceph-mon[44423]: pgmap v906: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 241 op/s; 3650/76080 objects misplaced (4.798%); 992 KiB/s, 15 objects/s recovering 2024-04-20T17:19:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:01 smithi106 ceph-mon[44423]: osdmap e330: 12 total, 12 up, 12 in 2024-04-20T17:19:01.518 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:19:01.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:01.521 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:19:01.622 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:19:01.651 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:02.404 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:19:02.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:02.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:19:02.420 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-20T17:19:02.448 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:19:02.455 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:19:02.509 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:19:02.542 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:03.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:03 smithi005 ceph-mon[39562]: pgmap v908: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 250 op/s; 3650/75084 objects misplaced (4.861%); 977 KiB/s, 15 objects/s recovering 2024-04-20T17:19:03.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:03 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:19:03.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:03 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:19:03.502 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:19:03.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:03 smithi106 ceph-mon[44423]: pgmap v908: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 250 op/s; 3650/75084 objects misplaced (4.861%); 977 KiB/s, 15 objects/s recovering 2024-04-20T17:19:03.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:03 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:19:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:03 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:19:03.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:03 smithi171 ceph-mon[44189]: pgmap v908: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 250 op/s; 3650/75084 objects misplaced (4.861%); 977 KiB/s, 15 objects/s recovering 2024-04-20T17:19:03.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:03 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:19:03.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:03 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:19:03.696 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:19:03.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:03.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:19:03.801 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:19:03.828 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:03.846 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:19:03.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:03.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:19:03.950 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:19:03.976 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:03.998 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:19:04.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:04.000 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:19:04.102 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:19:04.131 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:04.155 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:19:04.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:04.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:19:04.187 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:04 smithi005 ceph-mon[39562]: 2.f scrub starts 2024-04-20T17:19:04.187 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:04 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:19:04.187 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:04 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:19:04.187 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:04 smithi005 ceph-mon[39562]: osdmap e331: 12 total, 12 up, 12 in 2024-04-20T17:19:04.187 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:04 smithi005 ceph-mon[39562]: 2.f scrub ok 2024-04-20T17:19:04.187 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:04 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:19:04.259 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:19:04.298 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:04.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:04 smithi171 ceph-mon[44189]: 2.f scrub starts 2024-04-20T17:19:04.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:04 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:19:04.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:04 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:19:04.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:04 smithi171 ceph-mon[44189]: osdmap e331: 12 total, 12 up, 12 in 2024-04-20T17:19:04.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:04 smithi171 ceph-mon[44189]: 2.f scrub ok 2024-04-20T17:19:04.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:04 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:19:04.376 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:19:04.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:04.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:19:04.484 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:19:04.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:04 smithi106 ceph-mon[44423]: 2.f scrub starts 2024-04-20T17:19:04.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:04 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:19:04.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:04 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:19:04.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:04 smithi106 ceph-mon[44423]: osdmap e331: 12 total, 12 up, 12 in 2024-04-20T17:19:04.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:04 smithi106 ceph-mon[44423]: 2.f scrub ok 2024-04-20T17:19:04.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:04 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:19:04.522 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:05.075 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:19:05.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:05.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:19:05.187 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:19:05.219 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:05.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:05 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:19:05.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:05 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:19:05.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:05 smithi005 ceph-mon[39562]: pgmap v910: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 293 op/s; 3458/73306 objects misplaced (4.717%); 2.0 MiB/s, 31 objects/s recovering 2024-04-20T17:19:05.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:05 smithi005 ceph-mon[39562]: osdmap e332: 12 total, 12 up, 12 in 2024-04-20T17:19:05.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:05 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:19:05.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:05 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:19:05.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:05 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:19:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:05 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:19:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:05 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:19:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:05 smithi106 ceph-mon[44423]: pgmap v910: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 293 op/s; 3458/73306 objects misplaced (4.717%); 2.0 MiB/s, 31 objects/s recovering 2024-04-20T17:19:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:05 smithi106 ceph-mon[44423]: osdmap e332: 12 total, 12 up, 12 in 2024-04-20T17:19:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:05 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:19:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:05 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:19:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:05 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:19:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:05 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:19:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:05 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:19:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:05 smithi171 ceph-mon[44189]: pgmap v910: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 293 op/s; 3458/73306 objects misplaced (4.717%); 2.0 MiB/s, 31 objects/s recovering 2024-04-20T17:19:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:05 smithi171 ceph-mon[44189]: osdmap e332: 12 total, 12 up, 12 in 2024-04-20T17:19:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:05 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:19:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:05 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:19:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:05 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:19:05.763 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:19:05.769 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:19:05.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:05.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:19:05.879 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:19:05.911 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:06.013 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:19:06.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:06.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:19:06.117 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:19:06.147 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:06.276 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:19:06.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:06.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:19:06.381 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:19:06.413 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:06.567 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:19:06.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:06.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:19:06.672 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:19:06.703 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:07.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:06 smithi106 ceph-mon[44423]: pgmap v912: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 313 op/s; 3458/72314 objects misplaced (4.782%); 1013 KiB/s, 15 objects/s recovering 2024-04-20T17:19:07.310 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:19:07.310 INFO:teuthology.orchestra.run.smithi005.stdout:{"fsid":"13c6e76c-ff37-11ee-bc93-c7b262605968","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1115,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":333,"num_osds":12,"num_up_osds":12,"osd_up_since":1713633141,"num_in_osds":12,"osd_in_since":1713632742,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovering","count":1}],"num_pgs":17,"num_pools":2,"num_objects":36157,"data_bytes":2369978936,"bytes_used":6851551232,"bytes_avail":1145088876544,"bytes_total":1151940427776,"misplaced_objects":3458,"misplaced_total":72314,"misplaced_ratio":0.047819232790331047,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":1037481,"recovering_keys_per_sec":0,"num_objects_recovered":95,"num_bytes_recovered":6225920,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":313},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.5:8443/"}},"servicemap":{"epoch":25,"modified":"2024-04-20T17:16:50.052718+0000","services":{}},"progress_events":{}} 2024-04-20T17:19:07.333 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:06 smithi005 ceph-mon[39562]: pgmap v912: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 313 op/s; 3458/72314 objects misplaced (4.782%); 1013 KiB/s, 15 objects/s recovering 2024-04-20T17:19:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:06 smithi171 ceph-mon[44189]: pgmap v912: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 313 op/s; 3458/72314 objects misplaced (4.782%); 1013 KiB/s, 15 objects/s recovering 2024-04-20T17:19:07.657 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:19:07.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:07.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:19:07.761 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:19:07.792 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:08.004 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:19:08.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:07 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:19:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:07 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:19:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:07 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:19:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:07 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:19:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:07 smithi106 ceph-mon[44423]: osdmap e333: 12 total, 12 up, 12 in 2024-04-20T17:19:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:07 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3657692044' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:19:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:07 smithi106 ceph-mon[44423]: from='client.25646 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:19:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:07 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1298915351' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:19:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:07 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:19:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:07 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:19:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:07 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:19:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:07 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:19:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:07 smithi171 ceph-mon[44189]: osdmap e333: 12 total, 12 up, 12 in 2024-04-20T17:19:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:07 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3657692044' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:19:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:07 smithi171 ceph-mon[44189]: from='client.25646 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:19:08.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:07 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1298915351' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:19:08.392 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:07 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:19:08.392 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:07 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:19:08.392 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:07 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:19:08.392 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:07 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:19:08.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:07 smithi005 ceph-mon[39562]: osdmap e333: 12 total, 12 up, 12 in 2024-04-20T17:19:08.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:07 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3657692044' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:19:08.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:07 smithi005 ceph-mon[39562]: from='client.25646 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:19:08.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:07 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1298915351' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:19:08.836 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:19:08.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:08.854 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:19:08.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:19:08.955 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:19:08.982 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:09.001 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:19:09.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:09.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:19:09.108 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:19:09.137 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:08 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:19:09.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:08 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:19:09.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:08 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:19:09.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:08 smithi005 ceph-mon[39562]: from='client.25646 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:19:09.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:08 smithi005 ceph-mon[39562]: osdmap e334: 12 total, 12 up, 12 in 2024-04-20T17:19:09.139 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:08 smithi005 ceph-mon[39562]: pgmap v915: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 326 op/s; 3458/70788 objects misplaced (4.885%); 1.2 MiB/s, 19 objects/s recovering 2024-04-20T17:19:09.139 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:08 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:19:09.139 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:08 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:19:09.140 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:08 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:19:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:08 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:19:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:08 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:19:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:08 smithi106 ceph-mon[44423]: from='client.25646 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:19:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:08 smithi106 ceph-mon[44423]: osdmap e334: 12 total, 12 up, 12 in 2024-04-20T17:19:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:08 smithi106 ceph-mon[44423]: pgmap v915: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 326 op/s; 3458/70788 objects misplaced (4.885%); 1.2 MiB/s, 19 objects/s recovering 2024-04-20T17:19:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:08 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:19:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:08 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:19:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:08 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:19:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:08 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:19:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:08 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:19:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:08 smithi171 ceph-mon[44189]: from='client.25646 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:19:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:08 smithi171 ceph-mon[44189]: osdmap e334: 12 total, 12 up, 12 in 2024-04-20T17:19:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:08 smithi171 ceph-mon[44189]: pgmap v915: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 326 op/s; 3458/70788 objects misplaced (4.885%); 1.2 MiB/s, 19 objects/s recovering 2024-04-20T17:19:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:08 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:19:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:08 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:19:10.003 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:10 smithi005 ceph-mon[39562]: osdmap e335: 12 total, 12 up, 12 in 2024-04-20T17:19:10.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:10 smithi106 ceph-mon[44423]: osdmap e335: 12 total, 12 up, 12 in 2024-04-20T17:19:10.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:10 smithi171 ceph-mon[44189]: osdmap e335: 12 total, 12 up, 12 in 2024-04-20T17:19:10.690 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:19:10.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:10.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:19:10.733 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-20T17:19:10.734 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:19:10.797 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:19:10.854 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:10.960 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:19:10.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:10.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:19:11.065 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:19:11.091 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:19:11.091 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:19:11.091 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-04-20T17:19:11.091 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:19:11.093 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:11.112 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:19:11.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:11.116 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:19:11.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:11 smithi005 ceph-mon[39562]: pgmap v917: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 289 op/s; 3262/69902 objects misplaced (4.667%); 1.0 MiB/s, 16 objects/s recovering 2024-04-20T17:19:11.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:11 smithi005 ceph-mon[39562]: osdmap e336: 12 total, 12 up, 12 in 2024-04-20T17:19:11.217 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:19:11.250 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:11 smithi171 ceph-mon[44189]: pgmap v917: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 289 op/s; 3262/69902 objects misplaced (4.667%); 1.0 MiB/s, 16 objects/s recovering 2024-04-20T17:19:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:11 smithi171 ceph-mon[44189]: osdmap e336: 12 total, 12 up, 12 in 2024-04-20T17:19:11.348 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:19:11.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:11.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:19:11.453 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:19:11.481 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:11.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:11 smithi106 ceph-mon[44423]: pgmap v917: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 289 op/s; 3262/69902 objects misplaced (4.667%); 1.0 MiB/s, 16 objects/s recovering 2024-04-20T17:19:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:11 smithi106 ceph-mon[44423]: osdmap e336: 12 total, 12 up, 12 in 2024-04-20T17:19:11.555 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:19:11.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:11.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:19:11.659 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:19:11.689 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:11.783 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:19:11.940 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:19:11.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:11.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:19:12.046 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:19:12.072 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:12.093 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:19:12.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:12.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:19:12.196 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:19:12.225 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:12.570 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-20T17:19:12.576 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:19:12.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:12.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:19:12.583 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:19:12.682 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:19:12.715 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:13.162 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:19:13.381 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:13 smithi005 ceph-mon[39562]: pgmap v919: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 364 op/s; 3262/68608 objects misplaced (4.755%); 1.2 MiB/s, 19 objects/s recovering 2024-04-20T17:19:13.398 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:19:13.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:13 smithi106 ceph-mon[44423]: pgmap v919: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 364 op/s; 3262/68608 objects misplaced (4.755%); 1.2 MiB/s, 19 objects/s recovering 2024-04-20T17:19:13.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:13 smithi171 ceph-mon[44189]: pgmap v919: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 364 op/s; 3262/68608 objects misplaced (4.755%); 1.2 MiB/s, 19 objects/s recovering 2024-04-20T17:19:13.670 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:19:13.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:13.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:19:13.777 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:19:13.808 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:15.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:15 smithi005 ceph-mon[39562]: pgmap v920: 17 pgs: 1 active+recovering, 16 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 351 op/s; 3055/66508 objects misplaced (4.593%); 2.1 MiB/s, 33 objects/s recovering 2024-04-20T17:19:15.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:15 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2216704118' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:19:15.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:15.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:15.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:15.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:15.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:15.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:15.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:15.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:15 smithi106 ceph-mon[44423]: pgmap v920: 17 pgs: 1 active+recovering, 16 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 351 op/s; 3055/66508 objects misplaced (4.593%); 2.1 MiB/s, 33 objects/s recovering 2024-04-20T17:19:15.515 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:15 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2216704118' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:19:15.515 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:15.515 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:15.515 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:15.515 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:15.515 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:15.515 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:15.515 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:15 smithi171 ceph-mon[44189]: pgmap v920: 17 pgs: 1 active+recovering, 16 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 351 op/s; 3055/66508 objects misplaced (4.593%); 2.1 MiB/s, 33 objects/s recovering 2024-04-20T17:19:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:15 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2216704118' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:19:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:15.843 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:19:15.844 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:19:15.844 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:19:15.844 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:19:15.844 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:19:15.844 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:19:15.844 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:19:15.844 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:19:15.844 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:19:15.844 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:19:15.844 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:19:15.844 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:19:15.844 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:19:15.845 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:19:15.845 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:19:15.845 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:19:15.845 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:19:15.845 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:19:15.845 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:19:15.845 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:19:15.845 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:19:15.845 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:19:15.845 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:19:15.845 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:19:16.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:16.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:16.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:16.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:16.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:16.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:16.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:16.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:16.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:16.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:16.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:16.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:16.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:16.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:16.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:16.766 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:19:16.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:16.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:19:16.872 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:19:16.899 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:16.919 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:19:16.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:16.922 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:19:17.023 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:19:17.049 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:17.069 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:19:17.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:17.071 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:19:17.172 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:19:17.197 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:17 smithi171 ceph-mon[44189]: pgmap v921: 17 pgs: 1 active+recovering, 16 active+clean; 2.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 339 op/s; 3055/65362 objects misplaced (4.674%); 1.6 MiB/s, 25 objects/s recovering 2024-04-20T17:19:17.199 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:17.219 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:19:17.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:17.222 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:19:17.272 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:17 smithi005 ceph-mon[39562]: pgmap v921: 17 pgs: 1 active+recovering, 16 active+clean; 2.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 339 op/s; 3055/65362 objects misplaced (4.674%); 1.6 MiB/s, 25 objects/s recovering 2024-04-20T17:19:17.322 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:19:17.349 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:17.368 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:19:17.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:17.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:19:17.474 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:19:17.500 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:17.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:17 smithi106 ceph-mon[44423]: pgmap v921: 17 pgs: 1 active+recovering, 16 active+clean; 2.0 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 339 op/s; 3055/65362 objects misplaced (4.674%); 1.6 MiB/s, 25 objects/s recovering 2024-04-20T17:19:17.522 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:19:17.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:17.525 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:19:17.626 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:19:17.653 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:17.675 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:19:17.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:17.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:19:17.779 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:19:17.817 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:17.857 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:19:17.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:17.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:19:17.961 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:19:17.988 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:18.000 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:19:18.000 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:19:18.009 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:19:18.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:18.012 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:19:18.113 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:19:18.139 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:18 smithi005 ceph-mon[39562]: from='client.15723 v1:172.21.15.5:0/3149792766' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:19:18.144 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:18.244 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:19:18.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:18.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:19:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:18 smithi171 ceph-mon[44189]: from='client.15723 v1:172.21.15.5:0/3149792766' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:19:18.347 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:19:18.375 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:18.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:18 smithi106 ceph-mon[44423]: from='client.15723 v1:172.21.15.5:0/3149792766' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:19:19.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:19 smithi005 ceph-mon[39562]: pgmap v922: 17 pgs: 1 active+recovering, 16 active+clean; 1.9 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 336 op/s; 3055/63790 objects misplaced (4.789%); 740 KiB/s, 11 objects/s recovering 2024-04-20T17:19:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:19 smithi106 ceph-mon[44423]: pgmap v922: 17 pgs: 1 active+recovering, 16 active+clean; 1.9 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 336 op/s; 3055/63790 objects misplaced (4.789%); 740 KiB/s, 11 objects/s recovering 2024-04-20T17:19:19.526 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:19:19.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:19.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:19:19.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:19 smithi171 ceph-mon[44189]: pgmap v922: 17 pgs: 1 active+recovering, 16 active+clean; 1.9 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 336 op/s; 3055/63790 objects misplaced (4.789%); 740 KiB/s, 11 objects/s recovering 2024-04-20T17:19:19.632 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:19:19.667 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:20.338 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:19:20.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:20.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:19:20.442 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:19:20.469 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:20.488 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:19:20.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:20.490 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:19:20.592 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:19:20.625 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:20.876 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:19:20.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:20.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:19:20.982 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:19:21.008 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:21.028 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:19:21.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:21.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:19:21.131 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:19:21.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:21 smithi005 ceph-mon[39562]: pgmap v923: 17 pgs: 1 active+recovering, 16 active+clean; 1.9 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 350 op/s; 2832/62884 objects misplaced (4.504%); 1.3 MiB/s, 21 objects/s recovering 2024-04-20T17:19:21.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:21 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1696202074' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:19:21.152 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:19:21.159 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:21.177 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:19:21.177 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:19:21.178 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:19:21.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:21.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:19:21.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:21 smithi106 ceph-mon[44423]: pgmap v923: 17 pgs: 1 active+recovering, 16 active+clean; 1.9 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 350 op/s; 2832/62884 objects misplaced (4.504%); 1.3 MiB/s, 21 objects/s recovering 2024-04-20T17:19:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:21 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1696202074' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:19:21.286 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:19:21.312 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:21.332 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:19:21.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:21.335 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:19:21.436 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:19:21.468 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:21.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:21 smithi171 ceph-mon[44189]: pgmap v923: 17 pgs: 1 active+recovering, 16 active+clean; 1.9 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 350 op/s; 2832/62884 objects misplaced (4.504%); 1.3 MiB/s, 21 objects/s recovering 2024-04-20T17:19:21.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:21 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1696202074' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:19:21.718 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:19:21.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:21.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:19:21.823 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:19:21.850 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:21.872 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:19:21.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:21.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:19:21.976 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:19:22.006 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:22.312 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:19:22.324 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:22 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:19:22.324 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:22 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1696202074' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:19:22.324 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:22 smithi005 ceph-mon[39562]: osdmap e337: 12 total, 12 up, 12 in 2024-04-20T17:19:22.361 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:19:22.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:22.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:19:22.368 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":921,"stamp":"2024-04-20T17:19:16.078688+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2142175800,"num_objects":32681,"num_object_clones":0,"num_object_copies":65362,"num_objects_missing_on_primary":1522,"num_objects_missing":1522,"num_objects_degraded":0,"num_objects_misplaced":3055,"num_objects_unfound":0,"num_objects_dirty":32681,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":289486,"num_write_kb":10306363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27691,"num_bytes_recovered":1816348800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160941,"ondisk_log_size":160941,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":6275172,"kb_used_data":5073080,"kb_used_omap":219,"kb_used_meta":1201764,"kb_avail":1118666652,"statfs":{"total":1151940427776,"available":1145514651648,"internally_reserved":0,"allocated":5194833920,"data_stored":5180454848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224949,"internal_metadata":1230606667},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,1,0,2,4,6,1,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-177799168,"num_objects":-2713,"num_object_clones":0,"num_object_copies":-5426,"num_objects_missing_on_primary":-200,"num_objects_missing":-200,"num_objects_degraded":0,"num_objects_misplaced":-403,"num_objects_unfound":0,"num_objects_dirty":-2713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2713,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":13303808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001341"},"pg_stats":[{"pgid":"2.f","version":"336'18144","reported_seq":19176,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-04-20T17:19:13.718093+0000","last_change":"2024-04-20T17:19:03.232229+0000","last_active":"2024-04-20T17:19:13.718093+0000","last_peered":"2024-04-20T17:19:13.718093+0000","last_clean":"2024-04-20T17:19:13.718093+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:19:13.718093+0000","last_undegraded":"2024-04-20T17:19:13.718093+0000","last_fullsized":"2024-04-20T17:19:13.718093+0000","mapping_epoch":115,"log_start":"142'8100","ondisk_log_start":"142'8100","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'17904","last_scrub_stamp":"2024-04-20T17:19:03.232003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:19:03.232003+0000","objects_scrubbed":2282,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:09.723163+0000","scrub_duration":268,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133562368,"num_objects":2038,"num_object_clones":0,"num_object_copies":4076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18144,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"336'18076","reported_seq":30144,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-04-20T17:19:15.484285+0000","last_change":"2024-04-20T17:18:08.306341+0000","last_active":"2024-04-20T17:19:15.484285+0000","last_peered":"2024-04-20T17:19:15.484285+0000","last_clean":"2024-04-20T17:19:15.484285+0000","last_became_active":"2024-04-20T17:10:28.865670+0000","last_became_peered":"2024-04-20T17:10:28.865670+0000","last_unstale":"2024-04-20T17:19:15.484285+0000","last_undegraded":"2024-04-20T17:19:15.484285+0000","last_fullsized":"2024-04-20T17:19:15.484285+0000","mapping_epoch":162,"log_start":"142'8000","ondisk_log_start":"142'8000","created":115,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'16860","last_scrub_stamp":"2024-04-20T17:18:08.306156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:08.306156+0000","objects_scrubbed":3197,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:34.841351+0000","scrub_duration":281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129761280,"num_objects":1980,"num_object_clones":0,"num_object_copies":3960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18076,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5619,"num_bytes_recovered":368246784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"336'18404","reported_seq":19287,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-04-20T17:19:15.536461+0000","last_change":"2024-04-20T17:18:49.298437+0000","last_active":"2024-04-20T17:19:15.536461+0000","last_peered":"2024-04-20T17:19:15.536461+0000","last_clean":"2024-04-20T17:19:15.536461+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:19:15.536461+0000","last_undegraded":"2024-04-20T17:19:15.536461+0000","last_fullsized":"2024-04-20T17:19:15.536461+0000","mapping_epoch":115,"log_start":"142'8400","ondisk_log_start":"142'8400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'17915","last_scrub_stamp":"2024-04-20T17:18:49.298307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:49.298307+0000","objects_scrubbed":2567,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:03.315951+0000","scrub_duration":225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136183808,"num_objects":2078,"num_object_clones":0,"num_object_copies":4156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18404,"num_write_kb":655424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"336'24795","reported_seq":46441,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-04-20T17:19:15.578339+0000","last_change":"2024-04-20T17:17:53.095109+0000","last_active":"2024-04-20T17:19:15.578339+0000","last_peered":"2024-04-20T17:19:15.578339+0000","last_clean":"2024-04-20T17:19:15.578339+0000","last_became_active":"2024-04-20T17:12:08.085936+0000","last_became_peered":"2024-04-20T17:12:08.085936+0000","last_unstale":"2024-04-20T17:19:15.578339+0000","last_undegraded":"2024-04-20T17:19:15.578339+0000","last_fullsized":"2024-04-20T17:19:15.578339+0000","mapping_epoch":196,"log_start":"207'14768","ondisk_log_start":"207'14768","created":115,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'23389","last_scrub_stamp":"2024-04-20T17:17:53.094985+0000","last_deep_scrub":"213'20373","last_deep_scrub_stamp":"2024-04-20T17:12:48.443808+0000","last_clean_scrub_stamp":"2024-04-20T17:17:53.094985+0000","objects_scrubbed":3498,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:20.542939+0000","scrub_duration":306,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137035776,"num_objects":2091,"num_object_clones":0,"num_object_copies":4182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18143,"num_write_kb":647488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":450166784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"336'17767","reported_seq":41932,"reported_epoch":336,"state":"active+recovering","last_fresh":"2024-04-20T17:19:13.879763+0000","last_change":"2024-04-20T17:12:26.166109+0000","last_active":"2024-04-20T17:19:13.879763+0000","last_peered":"2024-04-20T17:19:13.879763+0000","last_clean":"2024-04-20T17:09:35.882603+0000","last_became_active":"2024-04-20T17:10:54.470253+0000","last_became_peered":"2024-04-20T17:10:54.470253+0000","last_unstale":"2024-04-20T17:19:13.879763+0000","last_undegraded":"2024-04-20T17:19:13.879763+0000","last_fullsized":"2024-04-20T17:19:13.879763+0000","mapping_epoch":171,"log_start":"141'7700","ondisk_log_start":"141'7700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128647168,"num_objects":1963,"num_object_clones":0,"num_object_copies":3926,"num_objects_missing_on_primary":1522,"num_objects_missing":1522,"num_objects_degraded":0,"num_objects_misplaced":3055,"num_objects_unfound":0,"num_objects_dirty":1963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17767,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7899,"num_bytes_recovered":517668864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"0,2,5","objects":11},{"shards":"0,10","objects":430},{"shards":"2,5","objects":1522}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":876,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-04-20T17:19:11.037043+0000","last_change":"2024-04-20T17:17:58.199584+0000","last_active":"2024-04-20T17:19:11.037043+0000","last_peered":"2024-04-20T17:19:11.037043+0000","last_clean":"2024-04-20T17:19:11.037043+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:19:11.037043+0000","last_undegraded":"2024-04-20T17:19:11.037043+0000","last_fullsized":"2024-04-20T17:19:11.037043+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:17:58.199502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:17:58.199502+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:16.100652+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"336'21183","reported_seq":22193,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-04-20T17:19:12.891691+0000","last_change":"2024-04-20T17:18:28.829916+0000","last_active":"2024-04-20T17:19:12.891691+0000","last_peered":"2024-04-20T17:19:12.891691+0000","last_clean":"2024-04-20T17:19:12.891691+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:19:12.891691+0000","last_undegraded":"2024-04-20T17:19:12.891691+0000","last_fullsized":"2024-04-20T17:19:12.891691+0000","mapping_epoch":115,"log_start":"154'11100","ondisk_log_start":"154'11100","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'20320","last_scrub_stamp":"2024-04-20T17:18:28.829770+0000","last_deep_scrub":"321'20320","last_deep_scrub_stamp":"2024-04-20T17:18:28.829770+0000","last_clean_scrub_stamp":"2024-04-20T17:18:28.829770+0000","objects_scrubbed":3006,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":10,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:40.227886+0000","scrub_duration":9561,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135987200,"num_objects":2075,"num_object_clones":0,"num_object_copies":4150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18177,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"336'18316","reported_seq":19229,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-04-20T17:19:13.733364+0000","last_change":"2024-04-20T17:18:49.142560+0000","last_active":"2024-04-20T17:19:13.733364+0000","last_peered":"2024-04-20T17:19:13.733364+0000","last_clean":"2024-04-20T17:19:13.733364+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:19:13.733364+0000","last_undegraded":"2024-04-20T17:19:13.733364+0000","last_fullsized":"2024-04-20T17:19:13.733364+0000","mapping_epoch":115,"log_start":"142'8300","ondisk_log_start":"142'8300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'17862","last_scrub_stamp":"2024-04-20T17:18:49.142459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:49.142459+0000","objects_scrubbed":2464,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:07.477325+0000","scrub_duration":245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131596288,"num_objects":2008,"num_object_clones":0,"num_object_copies":4016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18316,"num_write_kb":650368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"336'22110","reported_seq":23130,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-04-20T17:19:11.906286+0000","last_change":"2024-04-20T17:17:55.709069+0000","last_active":"2024-04-20T17:19:11.906286+0000","last_peered":"2024-04-20T17:19:11.906286+0000","last_clean":"2024-04-20T17:19:11.906286+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:19:11.906286+0000","last_undegraded":"2024-04-20T17:19:11.906286+0000","last_fullsized":"2024-04-20T17:19:11.906286+0000","mapping_epoch":115,"log_start":"161'12102","ondisk_log_start":"161'12102","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'20878","last_scrub_stamp":"2024-04-20T17:17:55.708833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:55.708833+0000","objects_scrubbed":3275,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:16.450254+0000","scrub_duration":320,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133890048,"num_objects":2043,"num_object_clones":0,"num_object_copies":4086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17999,"num_write_kb":641344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"336'18141","reported_seq":19112,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-04-20T17:19:12.356545+0000","last_change":"2024-04-20T17:18:45.620454+0000","last_active":"2024-04-20T17:19:12.356545+0000","last_peered":"2024-04-20T17:19:12.356545+0000","last_clean":"2024-04-20T17:19:12.356545+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:19:12.356545+0000","last_undegraded":"2024-04-20T17:19:12.356545+0000","last_fullsized":"2024-04-20T17:19:12.356545+0000","mapping_epoch":115,"log_start":"142'8100","ondisk_log_start":"142'8100","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'17642","last_scrub_stamp":"2024-04-20T17:18:45.620321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:45.620321+0000","objects_scrubbed":2579,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:50.964012+0000","scrub_duration":253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136249344,"num_objects":2079,"num_object_clones":0,"num_object_copies":4158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18141,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"336'17962","reported_seq":18857,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-04-20T17:19:12.909057+0000","last_change":"2024-04-20T17:18:45.702943+0000","last_active":"2024-04-20T17:19:12.909057+0000","last_peered":"2024-04-20T17:19:12.909057+0000","last_clean":"2024-04-20T17:19:12.909057+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:19:12.909057+0000","last_undegraded":"2024-04-20T17:19:12.909057+0000","last_fullsized":"2024-04-20T17:19:12.909057+0000","mapping_epoch":115,"log_start":"142'7900","ondisk_log_start":"142'7900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'17455","last_scrub_stamp":"2024-04-20T17:18:45.702769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:45.702769+0000","objects_scrubbed":2599,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:54.717372+0000","scrub_duration":222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136970240,"num_objects":2090,"num_object_clones":0,"num_object_copies":4180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17962,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"336'18190","reported_seq":33654,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-04-20T17:19:12.884126+0000","last_change":"2024-04-20T17:17:57.501217+0000","last_active":"2024-04-20T17:19:12.884126+0000","last_peered":"2024-04-20T17:19:12.884126+0000","last_clean":"2024-04-20T17:19:12.884126+0000","last_became_active":"2024-04-20T17:10:55.931751+0000","last_became_peered":"2024-04-20T17:10:55.931751+0000","last_unstale":"2024-04-20T17:19:12.884126+0000","last_undegraded":"2024-04-20T17:19:12.884126+0000","last_fullsized":"2024-04-20T17:19:12.884126+0000","mapping_epoch":173,"log_start":"142'8100","ondisk_log_start":"142'8100","created":115,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'16916","last_scrub_stamp":"2024-04-20T17:17:57.501089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:57.501089+0000","objects_scrubbed":3350,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:26.178414+0000","scrub_duration":318,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135790592,"num_objects":2072,"num_object_clones":0,"num_object_copies":4144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18190,"num_write_kb":648384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7296,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"336'18252","reported_seq":19172,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-04-20T17:19:12.525464+0000","last_change":"2024-04-20T17:18:06.314140+0000","last_active":"2024-04-20T17:19:12.525464+0000","last_peered":"2024-04-20T17:19:12.525464+0000","last_clean":"2024-04-20T17:19:12.525464+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:19:12.525464+0000","last_undegraded":"2024-04-20T17:19:12.525464+0000","last_fullsized":"2024-04-20T17:19:12.525464+0000","mapping_epoch":115,"log_start":"142'8200","ondisk_log_start":"142'8200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"314'17127","last_scrub_stamp":"2024-04-20T17:18:06.313989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:06.313989+0000","objects_scrubbed":3191,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:17.300375+0000","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135266304,"num_objects":2064,"num_object_clones":0,"num_object_copies":4128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18252,"num_write_kb":650112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"336'17786","reported_seq":18695,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-04-20T17:19:13.687805+0000","last_change":"2024-04-20T17:18:47.241194+0000","last_active":"2024-04-20T17:19:13.687805+0000","last_peered":"2024-04-20T17:19:13.687805+0000","last_clean":"2024-04-20T17:19:13.687805+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:19:13.687805+0000","last_undegraded":"2024-04-20T17:19:13.687805+0000","last_fullsized":"2024-04-20T17:19:13.687805+0000","mapping_epoch":115,"log_start":"141'7700","ondisk_log_start":"141'7700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'17283","last_scrub_stamp":"2024-04-20T17:18:47.241064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:47.241064+0000","objects_scrubbed":2584,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:10.974140+0000","scrub_duration":266,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136183808,"num_objects":2078,"num_object_clones":0,"num_object_copies":4156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17786,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"336'17998","reported_seq":18908,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-04-20T17:19:12.438520+0000","last_change":"2024-04-20T17:18:48.599315+0000","last_active":"2024-04-20T17:19:12.438520+0000","last_peered":"2024-04-20T17:19:12.438520+0000","last_clean":"2024-04-20T17:19:12.438520+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:19:12.438520+0000","last_undegraded":"2024-04-20T17:19:12.438520+0000","last_fullsized":"2024-04-20T17:19:12.438520+0000","mapping_epoch":115,"log_start":"142'7900","ondisk_log_start":"142'7900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'17566","last_scrub_stamp":"2024-04-20T17:18:48.599209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:48.599209+0000","objects_scrubbed":2538,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:05.548836+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137887744,"num_objects":2104,"num_object_clones":0,"num_object_copies":4208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17998,"num_write_kb":643264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"336'17930","reported_seq":18838,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-04-20T17:19:15.545788+0000","last_change":"2024-04-20T17:18:52.214316+0000","last_active":"2024-04-20T17:19:15.545788+0000","last_peered":"2024-04-20T17:19:15.545788+0000","last_clean":"2024-04-20T17:19:15.545788+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:19:15.545788+0000","last_undegraded":"2024-04-20T17:19:15.545788+0000","last_fullsized":"2024-04-20T17:19:15.545788+0000","mapping_epoch":115,"log_start":"142'7900","ondisk_log_start":"142'7900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'17470","last_scrub_stamp":"2024-04-20T17:18:52.214208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:52.214208+0000","objects_scrubbed":2403,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:55.920246+0000","scrub_duration":228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127074328,"num_objects":1940,"num_object_clones":0,"num_object_copies":3880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1940,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":17930,"num_write_kb":635777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"336'18020","reported_seq":18808,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-04-20T17:19:15.855910+0000","last_change":"2024-04-20T17:18:17.426012+0000","last_active":"2024-04-20T17:19:15.855910+0000","last_peered":"2024-04-20T17:19:15.855910+0000","last_clean":"2024-04-20T17:19:15.855910+0000","last_became_active":"2024-04-20T17:12:23.515375+0000","last_became_peered":"2024-04-20T17:12:23.515375+0000","last_unstale":"2024-04-20T17:19:15.855910+0000","last_undegraded":"2024-04-20T17:19:15.855910+0000","last_fullsized":"2024-04-20T17:19:15.855910+0000","mapping_epoch":202,"log_start":"142'8000","ondisk_log_start":"142'8000","created":115,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'16928","last_scrub_stamp":"2024-04-20T17:18:17.425878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:17.425878+0000","objects_scrubbed":3071,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:47.218525+0000","scrub_duration":353,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129499136,"num_objects":1976,"num_object_clones":0,"num_object_copies":3952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18020,"num_write_kb":639872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":2141585432,"num_objects":32679,"num_object_clones":0,"num_object_copies":65358,"num_objects_missing_on_primary":1522,"num_objects_missing":1522,"num_objects_degraded":0,"num_objects_misplaced":3055,"num_objects_unfound":0,"num_objects_dirty":32679,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":289305,"num_write_kb":10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27683,"num_bytes_recovered":1814233088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5135147008,"data_stored":5135138864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},"log_size":160804,"ondisk_log_size":160804,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991394,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":485752,"kb_used_data":406000,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93259400,"statfs":{"total":95995035648,"available":95497625600,"internally_reserved":0,"allocated":415744000,"data_stored":414543446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":81656260},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253032,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":603316,"kb_used_data":427752,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93141836,"statfs":{"total":95995035648,"available":95377240064,"internally_reserved":0,"allocated":438018048,"data_stored":436821046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":179763648},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612782,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":657652,"kb_used_data":522348,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":93087500,"statfs":{"total":95995035648,"available":95321600000,"internally_reserved":0,"allocated":534884352,"data_stored":533683298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":138541504},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809011,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":634992,"kb_used_data":527464,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93110160,"statfs":{"total":95995035648,"available":95344803840,"internally_reserved":0,"allocated":540123136,"data_stored":538926154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168760,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":504760,"kb_used_data":405552,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93240392,"statfs":{"total":95995035648,"available":95478161408,"internally_reserved":0,"allocated":415285248,"data_stored":414084714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":101579210},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430396,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":382708,"kb_used_data":275176,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93362444,"statfs":{"total":95995035648,"available":95603142656,"internally_reserved":0,"allocated":281780224,"data_stored":280583242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575062,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":376564,"kb_used_data":273192,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93368588,"statfs":{"total":95995035648,"available":95609434112,"internally_reserved":0,"allocated":279748608,"data_stored":278551626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738587,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":568240,"kb_used_data":424616,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93176912,"statfs":{"total":95995035648,"available":95413157888,"internally_reserved":0,"allocated":434806784,"data_stored":433609802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1,0,2,4,6,1,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393876,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":599024,"kb_used_data":558952,"kb_used_omap":202,"kb_used_meta":39861,"kb_avail":93146128,"statfs":{"total":95995035648,"available":95381635072,"internally_reserved":0,"allocated":572366848,"data_stored":571169866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":40818135},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182604,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":328240,"kb_used_data":256808,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93416912,"statfs":{"total":95995035648,"available":95658917888,"internally_reserved":0,"allocated":262971392,"data_stored":261774410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986375,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":163188,"kb_used_data":127852,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93581964,"statfs":{"total":95995035648,"available":95827931136,"internally_reserved":0,"allocated":130920448,"data_stored":129719394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790145,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":970736,"kb_used_data":867368,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92774416,"statfs":{"total":95995035648,"available":95001001984,"internally_reserved":0,"allocated":888184832,"data_stored":886987850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":428146688,"data_stored":428146688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":276365312,"data_stored":276365312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":569638912,"data_stored":569638912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":258080768,"data_stored":258080768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":127209472,"data_stored":127205400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":884342784,"data_stored":884342784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":278855680,"data_stored":278855680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":408748032,"data_stored":408748032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532611072,"data_stored":532611072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":527765504,"data_stored":527761432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":432209920,"data_stored":432209920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":411172864,"data_stored":411172864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:19:22.467 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:19:22.495 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:22.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:22 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:19:22.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:22 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1696202074' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:19:22.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:22 smithi106 ceph-mon[44423]: osdmap e337: 12 total, 12 up, 12 in 2024-04-20T17:19:22.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:22 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:19:22.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:22 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1696202074' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:19:22.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:22 smithi171 ceph-mon[44189]: osdmap e337: 12 total, 12 up, 12 in 2024-04-20T17:19:23.010 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-20T17:19:23.046 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:19:23.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:23.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:19:23.149 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:19:23.177 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:23.196 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:19:23.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:23.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:19:23.300 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:19:23.326 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:23.345 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:19:23.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:23.347 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:19:23.448 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:19:23.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:23 smithi005 ceph-mon[39562]: pgmap v925: 17 pgs: 1 active+recovering, 16 active+clean; 1.9 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 358 op/s; 2832/61428 objects misplaced (4.610%); 1.3 MiB/s, 21 objects/s recovering 2024-04-20T17:19:23.479 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:23.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:23 smithi106 ceph-mon[44423]: pgmap v925: 17 pgs: 1 active+recovering, 16 active+clean; 1.9 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 358 op/s; 2832/61428 objects misplaced (4.610%); 1.3 MiB/s, 21 objects/s recovering 2024-04-20T17:19:23.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:23 smithi171 ceph-mon[44189]: pgmap v925: 17 pgs: 1 active+recovering, 16 active+clean; 1.9 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 358 op/s; 2832/61428 objects misplaced (4.610%); 1.3 MiB/s, 21 objects/s recovering 2024-04-20T17:19:23.604 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:19:23.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:23.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:19:23.640 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:19:23.708 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:19:23.736 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:23.761 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:19:23.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:23.766 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:19:23.868 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:19:23.896 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:23.923 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:19:23.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:23.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:19:24.028 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:19:24.054 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:24.073 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:19:24.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:24.075 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:19:24.176 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:19:24.205 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:24.223 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:19:24.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:24.226 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:19:24.327 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:19:24.353 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:24.373 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:19:24.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:24.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:19:24.476 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:19:24.503 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:24.524 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:19:24.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:24.530 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:19:24.587 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:19:24.631 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:19:24.659 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:24.679 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:19:24.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:24.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:19:24.782 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:19:24.808 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:24.827 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:19:24.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:24.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:19:24.930 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:19:24.958 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:24.978 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:19:24.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:24.984 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:19:25.085 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:19:25.111 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:25.130 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:19:25.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:25.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:19:25.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:25 smithi005 ceph-mon[39562]: pgmap v926: 17 pgs: 1 active+recovering, 16 active+clean; 1.8 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 377 op/s; 2601/58950 objects misplaced (4.412%); 1.4 MiB/s, 22 objects/s recovering 2024-04-20T17:19:25.233 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:19:25.265 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:25.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:25 smithi171 ceph-mon[44189]: pgmap v926: 17 pgs: 1 active+recovering, 16 active+clean; 1.8 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 377 op/s; 2601/58950 objects misplaced (4.412%); 1.4 MiB/s, 22 objects/s recovering 2024-04-20T17:19:25.370 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:19:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:25 smithi106 ceph-mon[44423]: pgmap v926: 17 pgs: 1 active+recovering, 16 active+clean; 1.8 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 377 op/s; 2601/58950 objects misplaced (4.412%); 1.4 MiB/s, 22 objects/s recovering 2024-04-20T17:19:25.620 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:19:25.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:25.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:19:25.633 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:19:25.726 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:19:25.752 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:25.773 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:19:25.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:25.776 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:19:25.877 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:19:25.903 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:25.921 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:19:25.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:25.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:19:26.025 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:19:26.051 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:26.073 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:19:26.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:26.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:19:26.177 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:19:26.206 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:26.230 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:19:26.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:26.235 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:19:26.336 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:19:26.363 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:26.388 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:19:26.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:26.392 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:19:26.493 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:19:26.520 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:26.539 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:19:26.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:26.541 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:19:26.642 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:19:26.669 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:26.692 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:19:26.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:26.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:19:26.797 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:19:26.798 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:19:26.822 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:19:26.830 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:26.851 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:19:26.871 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:19:26.901 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:19:26.923 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:19:26.943 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:19:26.964 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:19:26.984 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:19:27.006 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:19:27.026 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:19:27.046 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:19:27.047 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:19:27.047 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:19:27.048 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:19:27.048 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:19:27.048 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:19:27.048 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:19:27.048 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:19:27.048 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:19:27.048 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:19:27.048 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:19:27.048 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:19:27.048 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:19:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:26 smithi005 ceph-mon[39562]: osdmap e338: 12 total, 12 up, 12 in 2024-04-20T17:19:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:26 smithi005 ceph-mon[39562]: pgmap v928: 17 pgs: 1 active+recovering, 16 active+clean; 1.8 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 391 op/s; 2601/57530 objects misplaced (4.521%); 1.8 MiB/s, 28 objects/s recovering 2024-04-20T17:19:27.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:26 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/403289980' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:19:27.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:26 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:27.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:26 smithi106 ceph-mon[44423]: osdmap e338: 12 total, 12 up, 12 in 2024-04-20T17:19:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:26 smithi106 ceph-mon[44423]: pgmap v928: 17 pgs: 1 active+recovering, 16 active+clean; 1.8 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 391 op/s; 2601/57530 objects misplaced (4.521%); 1.8 MiB/s, 28 objects/s recovering 2024-04-20T17:19:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:26 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/403289980' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:19:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:26 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:26 smithi171 ceph-mon[44189]: osdmap e338: 12 total, 12 up, 12 in 2024-04-20T17:19:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:26 smithi171 ceph-mon[44189]: pgmap v928: 17 pgs: 1 active+recovering, 16 active+clean; 1.8 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 391 op/s; 2601/57530 objects misplaced (4.521%); 1.8 MiB/s, 28 objects/s recovering 2024-04-20T17:19:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:26 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/403289980' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:19:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:26 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:27.433 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:19:27.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:27.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:19:27.536 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:19:27.563 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:27.585 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:19:27.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:27.587 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:19:27.689 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:19:27.704 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:19:27.715 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:27.736 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:19:27.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:27.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:19:27.749 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:19:27.839 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:19:27.873 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:28.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:28.536 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:19:28.537 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:19:28.537 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:19:28.537 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:19:28.537 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.537 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:176488 2.1 e154/115 2:82dc00e5:::benchmark_data_smithi171_78918_object15494:head v 154'11121, mlcod=154'11117)", 2024-04-20T17:19:28.537 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:10:06.997836+0000", 2024-04-20T17:19:28.537 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 561.53767209900002, 2024-04-20T17:19:28.537 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.024910439999999999, 2024-04-20T17:19:28.537 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:28.537 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:28.537 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:28.538 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:28.538 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.538 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:28.538 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:06.997836+0000", 2024-04-20T17:19:28.538 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.538 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.538 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.538 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:28.538 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:06.997836+0000", 2024-04-20T17:19:28.538 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.538 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.538 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.538 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:28.538 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:06.997837+0000", 2024-04-20T17:19:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2300000000000001e-07 2024-04-20T17:19:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:06.997845+0000", 2024-04-20T17:19:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5280000000000003e-06 2024-04-20T17:19:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:06.997852+0000", 2024-04-20T17:19:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7460000000000003e-06 2024-04-20T17:19:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:28.539 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:06.997858+0000", 2024-04-20T17:19:28.540 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6509999999999998e-06 2024-04-20T17:19:28.540 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.540 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.540 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:28.540 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.007320+0000", 2024-04-20T17:19:28.540 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0094623309999999992 2024-04-20T17:19:28.540 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.540 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.540 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.007343+0000", 2024-04-20T17:19:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3438000000000002e-05 2024-04-20T17:19:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.022698+0000", 2024-04-20T17:19:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015354892 2024-04-20T17:19:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.022747+0000", 2024-04-20T17:19:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8331000000000002e-05 2024-04-20T17:19:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:28.541 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.542 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.542 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.542 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:176512 2.e e154/115 2:7d9dff84:::benchmark_data_smithi171_78918_object15518:head v 154'10943, mlcod=154'10939)", 2024-04-20T17:19:28.542 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:10:07.032787+0000", 2024-04-20T17:19:28.542 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 561.50272108800004, 2024-04-20T17:19:28.542 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.027008193999999999, 2024-04-20T17:19:28.542 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:28.542 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:28.542 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:28.542 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:28.542 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.542 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:28.542 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.032787+0000", 2024-04-20T17:19:28.543 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.543 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.543 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.543 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:28.543 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.032787+0000", 2024-04-20T17:19:28.543 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.543 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.544 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.544 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:28.544 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.032788+0000", 2024-04-20T17:19:28.544 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1900000000000002e-07 2024-04-20T17:19:28.544 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.544 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.544 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:28.545 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.032792+0000", 2024-04-20T17:19:28.545 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5889999999999996e-06 2024-04-20T17:19:28.545 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.545 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.545 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:28.545 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.032797+0000", 2024-04-20T17:19:28.545 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9309999999999999e-06 2024-04-20T17:19:28.545 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.545 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.545 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:28.546 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.032802+0000", 2024-04-20T17:19:28.546 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4170000000000002e-06 2024-04-20T17:19:28.546 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.546 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.546 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:28.546 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.037454+0000", 2024-04-20T17:19:28.546 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0046517939999999999 2024-04-20T17:19:28.546 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.546 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.546 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:28.546 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.037474+0000", 2024-04-20T17:19:28.546 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0689000000000001e-05 2024-04-20T17:19:28.547 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.547 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.547 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:28.547 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.059757+0000", 2024-04-20T17:19:28.547 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.022282340000000001 2024-04-20T17:19:28.547 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.547 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.547 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:28.547 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.059795+0000", 2024-04-20T17:19:28.547 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8815e-05 2024-04-20T17:19:28.547 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.547 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:28.548 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.548 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.548 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.548 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:176689 2.a e154/115 2:540e9f45:::benchmark_data_smithi171_78918_object15695:head v 154'11117, mlcod=154'11114)", 2024-04-20T17:19:28.548 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:10:07.233150+0000", 2024-04-20T17:19:28.548 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 561.30235864700001, 2024-04-20T17:19:28.548 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.028013902, 2024-04-20T17:19:28.548 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:28.548 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:28.548 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:28.548 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:28.548 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.548 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:28.549 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.233150+0000", 2024-04-20T17:19:28.549 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.549 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.549 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.549 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:28.549 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.233150+0000", 2024-04-20T17:19:28.549 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.549 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.549 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.549 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:28.549 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.233151+0000", 2024-04-20T17:19:28.549 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4699999999999999e-06 2024-04-20T17:19:28.549 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.549 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.550 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:28.550 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.233159+0000", 2024-04-20T17:19:28.550 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6009999999999997e-06 2024-04-20T17:19:28.550 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.550 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.550 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:28.550 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.233172+0000", 2024-04-20T17:19:28.550 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3044000000000001e-05 2024-04-20T17:19:28.550 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.550 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.550 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:28.550 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.233179+0000", 2024-04-20T17:19:28.550 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.374e-06 2024-04-20T17:19:28.550 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.237355+0000", 2024-04-20T17:19:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041753650000000003 2024-04-20T17:19:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.237378+0000", 2024-04-20T17:19:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3003999999999999e-05 2024-04-20T17:19:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.261098+0000", 2024-04-20T17:19:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023720448000000002 2024-04-20T17:19:28.551 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.552 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.552 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:28.552 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.261164+0000", 2024-04-20T17:19:28.552 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5596e-05 2024-04-20T17:19:28.552 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.552 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:28.552 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.552 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.552 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.552 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:176738 2.a e154/115 2:5f9d34d3:::benchmark_data_smithi171_78918_object15744:head v 154'11124, mlcod=154'11120)", 2024-04-20T17:19:28.552 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:10:07.303815+0000", 2024-04-20T17:19:28.552 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 561.23169308399997, 2024-04-20T17:19:28.552 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.030124618999999998, 2024-04-20T17:19:28.553 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:28.553 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:28.553 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:28.553 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:28.553 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.553 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:28.553 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.303815+0000", 2024-04-20T17:19:28.553 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.553 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.553 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.553 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:28.553 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.303815+0000", 2024-04-20T17:19:28.553 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.553 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.553 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.554 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:28.554 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.303819+0000", 2024-04-20T17:19:28.554 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.422e-06 2024-04-20T17:19:28.554 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.554 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.554 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:28.554 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.303834+0000", 2024-04-20T17:19:28.554 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5792000000000001e-05 2024-04-20T17:19:28.554 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.554 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.554 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:28.554 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.303857+0000", 2024-04-20T17:19:28.554 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2504e-05 2024-04-20T17:19:28.555 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.555 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.555 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:28.555 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.303869+0000", 2024-04-20T17:19:28.555 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.241e-05 2024-04-20T17:19:28.555 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.555 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.555 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:28.555 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.303935+0000", 2024-04-20T17:19:28.555 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5839000000000004e-05 2024-04-20T17:19:28.555 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.555 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.303960+0000", 2024-04-20T17:19:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.512e-05 2024-04-20T17:19:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.333895+0000", 2024-04-20T17:19:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.029934444000000001 2024-04-20T17:19:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:07.333940+0000", 2024-04-20T17:19:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5087999999999999e-05 2024-04-20T17:19:28.556 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.557 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:28.557 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.557 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.557 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.557 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:181903 2.1 e157/115 2:8dddee52:::benchmark_data_smithi171_78918_object20909:head v 157'11471, mlcod=157'11468)", 2024-04-20T17:19:28.557 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:10:13.469412+0000", 2024-04-20T17:19:28.557 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 555.06609645100002, 2024-04-20T17:19:28.557 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.025238851, 2024-04-20T17:19:28.557 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:28.557 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:28.557 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:28.557 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:28.557 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.558 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:28.558 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.469412+0000", 2024-04-20T17:19:28.558 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.558 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.558 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.558 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:28.558 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.469412+0000", 2024-04-20T17:19:28.558 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.558 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.558 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.558 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:28.558 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.469414+0000", 2024-04-20T17:19:28.558 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.858e-06 2024-04-20T17:19:28.558 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.559 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.559 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:28.559 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.469423+0000", 2024-04-20T17:19:28.559 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.414e-06 2024-04-20T17:19:28.559 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.559 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.559 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:28.559 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.469439+0000", 2024-04-20T17:19:28.559 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5747e-05 2024-04-20T17:19:28.559 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.559 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.559 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:28.559 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.469447+0000", 2024-04-20T17:19:28.559 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1599999999999998e-06 2024-04-20T17:19:28.559 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.559 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.469509+0000", 2024-04-20T17:19:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2337999999999998e-05 2024-04-20T17:19:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.469538+0000", 2024-04-20T17:19:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8221e-05 2024-04-20T17:19:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.494609+0000", 2024-04-20T17:19:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.025071586999999999 2024-04-20T17:19:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.494651+0000", 2024-04-20T17:19:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1526000000000002e-05 2024-04-20T17:19:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:181959 2.e e157/115 2:7dbaa849:::benchmark_data_smithi171_78918_object20965:head v 157'11248, mlcod=157'11245)", 2024-04-20T17:19:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:10:13.529815+0000", 2024-04-20T17:19:28.561 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 555.00569376099997, 2024-04-20T17:19:28.562 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.025783212999999999, 2024-04-20T17:19:28.562 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:28.562 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:28.562 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:28.562 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:28.562 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.562 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:28.562 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.529815+0000", 2024-04-20T17:19:28.562 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.562 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.562 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.562 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:28.563 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.529815+0000", 2024-04-20T17:19:28.563 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.563 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.563 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.563 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:28.563 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.529815+0000", 2024-04-20T17:19:28.563 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5499999999999998e-07 2024-04-20T17:19:28.563 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.563 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.563 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:28.563 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.529818+0000", 2024-04-20T17:19:28.563 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.219e-06 2024-04-20T17:19:28.564 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.564 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.564 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:28.564 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.529822+0000", 2024-04-20T17:19:28.564 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.286e-06 2024-04-20T17:19:28.564 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.564 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.564 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:28.564 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.529824+0000", 2024-04-20T17:19:28.564 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6539999999999999e-06 2024-04-20T17:19:28.564 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.564 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.564 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:28.564 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.533535+0000", 2024-04-20T17:19:28.565 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0037106130000000002 2024-04-20T17:19:28.565 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.565 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.565 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:28.565 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.533551+0000", 2024-04-20T17:19:28.565 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6075000000000001e-05 2024-04-20T17:19:28.565 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.565 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.565 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:28.565 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.555543+0000", 2024-04-20T17:19:28.565 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.021991739999999999 2024-04-20T17:19:28.565 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.566 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.566 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:28.566 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.555598+0000", 2024-04-20T17:19:28.566 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5170999999999997e-05 2024-04-20T17:19:28.566 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.566 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:28.566 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.566 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.566 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.566 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:182176 2.e e157/115 2:73571c13:::benchmark_data_smithi171_78918_object21182:head v 157'11257, mlcod=157'11255)", 2024-04-20T17:19:28.566 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:10:13.763760+0000", 2024-04-20T17:19:28.566 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 554.77174880099994, 2024-04-20T17:19:28.566 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.026084005, 2024-04-20T17:19:28.567 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:28.567 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:28.567 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:28.567 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:28.567 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.567 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:28.567 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.763760+0000", 2024-04-20T17:19:28.567 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.567 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.567 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.567 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:28.567 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.763760+0000", 2024-04-20T17:19:28.567 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.568 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.568 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.568 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:28.568 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.763763+0000", 2024-04-20T17:19:28.568 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6349999999999999e-06 2024-04-20T17:19:28.568 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.568 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.568 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:28.568 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.763777+0000", 2024-04-20T17:19:28.569 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3906e-05 2024-04-20T17:19:28.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.569 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:28.569 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.763798+0000", 2024-04-20T17:19:28.569 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1026000000000002e-05 2024-04-20T17:19:28.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.569 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:28.569 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.763808+0000", 2024-04-20T17:19:28.569 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0344e-05 2024-04-20T17:19:28.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.570 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:28.570 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.763935+0000", 2024-04-20T17:19:28.570 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000126573 2024-04-20T17:19:28.570 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.570 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.570 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:28.570 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.763956+0000", 2024-04-20T17:19:28.570 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.109e-05 2024-04-20T17:19:28.570 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.570 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.570 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:28.570 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.789794+0000", 2024-04-20T17:19:28.570 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.025837802999999999 2024-04-20T17:19:28.570 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.571 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.571 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:28.571 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.789844+0000", 2024-04-20T17:19:28.571 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9628e-05 2024-04-20T17:19:28.571 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.571 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:28.571 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.571 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.571 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.571 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:182179 2.1 e157/115 2:8e4016a1:::benchmark_data_smithi171_78918_object21185:head v 157'11491, mlcod=157'11489)", 2024-04-20T17:19:28.571 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:10:13.763839+0000", 2024-04-20T17:19:28.571 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 554.77166965699996, 2024-04-20T17:19:28.571 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.026592022, 2024-04-20T17:19:28.571 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:28.572 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:28.572 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:28.572 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:28.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.572 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:28.572 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.763839+0000", 2024-04-20T17:19:28.572 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.572 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.572 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:28.572 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.763839+0000", 2024-04-20T17:19:28.572 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.572 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.573 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:28.573 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.763839+0000", 2024-04-20T17:19:28.573 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9299999999999997e-07 2024-04-20T17:19:28.573 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.573 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.573 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:28.573 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.763845+0000", 2024-04-20T17:19:28.573 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8590000000000001e-06 2024-04-20T17:19:28.573 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.573 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.573 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:28.573 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.763852+0000", 2024-04-20T17:19:28.573 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7309999999999996e-06 2024-04-20T17:19:28.573 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.574 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.574 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:28.574 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.763858+0000", 2024-04-20T17:19:28.574 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5119999999999999e-06 2024-04-20T17:19:28.574 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.574 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.574 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:28.574 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.769984+0000", 2024-04-20T17:19:28.574 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0061264429999999996 2024-04-20T17:19:28.574 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.574 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.574 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:28.574 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.770007+0000", 2024-04-20T17:19:28.575 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3166000000000001e-05 2024-04-20T17:19:28.575 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.575 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.575 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:28.575 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.790394+0000", 2024-04-20T17:19:28.575 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.020386721 2024-04-20T17:19:28.575 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.575 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.575 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:28.575 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:13.790431+0000", 2024-04-20T17:19:28.575 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6797e-05 2024-04-20T17:19:28.575 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.575 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:28.575 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.576 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.576 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.576 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:183149 2.a e157/115 2:55a70d02:::benchmark_data_smithi171_78918_object22155:head v 157'11530, mlcod=157'11528)", 2024-04-20T17:19:28.576 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:10:14.933320+0000", 2024-04-20T17:19:28.576 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 553.60218846500004, 2024-04-20T17:19:28.576 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.027285667999999999, 2024-04-20T17:19:28.576 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:28.576 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:28.576 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:28.576 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:28.576 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.576 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:28.576 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:14.933320+0000", 2024-04-20T17:19:28.576 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.576 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.577 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.577 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:28.577 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:14.933320+0000", 2024-04-20T17:19:28.577 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.577 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.577 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.577 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:28.577 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:14.933323+0000", 2024-04-20T17:19:28.577 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.88e-06 2024-04-20T17:19:28.577 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.577 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.577 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:28.577 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:14.933338+0000", 2024-04-20T17:19:28.577 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5642e-05 2024-04-20T17:19:28.577 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.577 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.578 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:28.578 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:14.933358+0000", 2024-04-20T17:19:28.578 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0015999999999999e-05 2024-04-20T17:19:28.578 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.578 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.578 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:28.578 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:14.933368+0000", 2024-04-20T17:19:28.578 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4809999999999995e-06 2024-04-20T17:19:28.578 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.578 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.578 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:28.579 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:14.935210+0000", 2024-04-20T17:19:28.579 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0018418320000000001 2024-04-20T17:19:28.579 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.579 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.579 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:28.579 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:14.935234+0000", 2024-04-20T17:19:28.579 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4011000000000001e-05 2024-04-20T17:19:28.579 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.579 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.579 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:28.579 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:14.960561+0000", 2024-04-20T17:19:28.579 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.025327354999999999 2024-04-20T17:19:28.579 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.579 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.580 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:28.580 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:14.960606+0000", 2024-04-20T17:19:28.580 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4450999999999998e-05 2024-04-20T17:19:28.580 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.580 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:28.580 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.580 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.580 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.580 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:194984 2.a e163/115 2:5aa3402e:::benchmark_data_smithi171_78918_object33990:head v 163'12350, mlcod=163'12348)", 2024-04-20T17:19:28.580 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:10:29.704552+0000", 2024-04-20T17:19:28.581 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 538.83095641399996, 2024-04-20T17:19:28.581 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.02526685, 2024-04-20T17:19:28.581 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:28.581 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:28.581 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:28.581 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:28.581 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.581 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:28.581 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:29.704552+0000", 2024-04-20T17:19:28.581 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.581 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.581 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.582 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:28.582 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:29.704552+0000", 2024-04-20T17:19:28.582 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.582 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.582 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.582 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:28.582 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:29.704554+0000", 2024-04-20T17:19:28.582 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7570000000000001e-06 2024-04-20T17:19:28.582 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.582 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.582 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:28.582 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:29.704563+0000", 2024-04-20T17:19:28.582 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8910000000000001e-06 2024-04-20T17:19:28.583 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.583 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.583 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:28.583 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:29.704580+0000", 2024-04-20T17:19:28.583 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7116999999999999e-05 2024-04-20T17:19:28.583 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.583 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.583 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:28.583 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:29.704590+0000", 2024-04-20T17:19:28.583 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8530000000000004e-06 2024-04-20T17:19:28.583 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.583 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.583 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:28.583 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:29.704691+0000", 2024-04-20T17:19:28.584 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000101438 2024-04-20T17:19:28.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.584 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:28.584 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:29.704713+0000", 2024-04-20T17:19:28.584 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1675000000000002e-05 2024-04-20T17:19:28.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.584 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:28.584 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:29.729765+0000", 2024-04-20T17:19:28.584 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.025052680000000001 2024-04-20T17:19:28.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.584 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:28.585 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:29.729819+0000", 2024-04-20T17:19:28.585 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3439000000000001e-05 2024-04-20T17:19:28.585 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.585 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:28.585 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.585 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.585 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.585 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:200750 2.1 e166/115 2:8b1e1ec1:::benchmark_data_smithi171_78918_object39756:head v 166'12647, mlcod=166'12645)", 2024-04-20T17:19:28.585 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:10:38.092482+0000", 2024-04-20T17:19:28.585 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 530.44302682399996, 2024-04-20T17:19:28.586 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.025730521999999999, 2024-04-20T17:19:28.586 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:28.586 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:28.586 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:28.586 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:28.586 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.586 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:28.586 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:38.092482+0000", 2024-04-20T17:19:28.586 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.586 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.586 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.586 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:28.586 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:38.092482+0000", 2024-04-20T17:19:28.586 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.587 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.587 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.587 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:28.587 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:38.092484+0000", 2024-04-20T17:19:28.587 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6240000000000002e-06 2024-04-20T17:19:28.587 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.587 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.587 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:28.587 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:38.092495+0000", 2024-04-20T17:19:28.587 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0475000000000001e-05 2024-04-20T17:19:28.587 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.587 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.587 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:28.588 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:38.092512+0000", 2024-04-20T17:19:28.588 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7275e-05 2024-04-20T17:19:28.588 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.588 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.588 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:28.588 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:38.092522+0000", 2024-04-20T17:19:28.588 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0175e-05 2024-04-20T17:19:28.588 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.588 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.588 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:28.588 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:38.092639+0000", 2024-04-20T17:19:28.588 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011742499999999999 2024-04-20T17:19:28.588 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.588 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.589 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:28.589 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:38.092664+0000", 2024-04-20T17:19:28.589 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4630999999999998e-05 2024-04-20T17:19:28.589 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.589 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.589 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:28.589 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:38.118150+0000", 2024-04-20T17:19:28.589 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.025485826 2024-04-20T17:19:28.589 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.589 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.589 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:28.589 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:38.118212+0000", 2024-04-20T17:19:28.589 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2090999999999999e-05 2024-04-20T17:19:28.589 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.589 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:28.590 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.590 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.590 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.590 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:201913 2.1 e166/115 2:8860f8e7:::benchmark_data_smithi171_78918_object40919:head v 166'12729, mlcod=166'12724)", 2024-04-20T17:19:28.590 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:10:39.456828+0000", 2024-04-20T17:19:28.590 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 529.07868057899998, 2024-04-20T17:19:28.590 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.027632008999999999, 2024-04-20T17:19:28.590 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:28.590 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:28.590 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:28.590 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:28.590 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.590 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:28.591 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:39.456828+0000", 2024-04-20T17:19:28.591 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.591 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.591 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:28.591 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:39.456828+0000", 2024-04-20T17:19:28.591 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.591 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.591 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:28.591 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:39.456830+0000", 2024-04-20T17:19:28.591 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.875e-06 2024-04-20T17:19:28.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.591 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.592 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:28.592 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:39.456837+0000", 2024-04-20T17:19:28.592 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.554e-06 2024-04-20T17:19:28.592 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.592 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.592 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:28.592 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:39.456850+0000", 2024-04-20T17:19:28.592 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.274e-05 2024-04-20T17:19:28.592 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.592 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.592 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:28.592 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:39.456858+0000", 2024-04-20T17:19:28.592 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4179999999999997e-06 2024-04-20T17:19:28.592 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.592 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.593 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:28.593 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:39.456929+0000", 2024-04-20T17:19:28.593 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0542000000000003e-05 2024-04-20T17:19:28.593 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.593 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.593 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:28.593 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:39.456953+0000", 2024-04-20T17:19:28.593 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4385000000000001e-05 2024-04-20T17:19:28.593 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.593 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.593 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:28.593 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:39.484406+0000", 2024-04-20T17:19:28.593 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.027453011999999999 2024-04-20T17:19:28.593 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.594 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.594 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:28.594 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:39.484460+0000", 2024-04-20T17:19:28.594 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3483000000000003e-05 2024-04-20T17:19:28.594 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.594 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:28.594 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.594 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.594 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.594 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:201920 2.e e166/115 2:71c24ed5:::benchmark_data_smithi171_78918_object40926:head v 166'12462, mlcod=166'12460)", 2024-04-20T17:19:28.594 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:10:39.462488+0000", 2024-04-20T17:19:28.594 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 529.07302015799996, 2024-04-20T17:19:28.594 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.029213514999999999, 2024-04-20T17:19:28.595 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:28.595 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:28.595 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:28.595 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:28.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.595 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:28.595 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:39.462488+0000", 2024-04-20T17:19:28.595 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.595 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.595 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:28.595 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:39.462488+0000", 2024-04-20T17:19:28.595 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.596 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:28.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:39.462490+0000", 2024-04-20T17:19:28.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7069999999999999e-06 2024-04-20T17:19:28.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.596 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:28.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:39.462497+0000", 2024-04-20T17:19:28.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2599999999999999e-06 2024-04-20T17:19:28.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.596 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:28.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:39.462510+0000", 2024-04-20T17:19:28.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2387e-05 2024-04-20T17:19:28.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.597 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:28.597 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:39.462517+0000", 2024-04-20T17:19:28.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5220000000000001e-06 2024-04-20T17:19:28.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.597 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:28.597 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:39.473555+0000", 2024-04-20T17:19:28.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011037987000000001 2024-04-20T17:19:28.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.597 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:28.598 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:39.473579+0000", 2024-04-20T17:19:28.598 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3550999999999999e-05 2024-04-20T17:19:28.598 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.598 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.598 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:28.598 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:39.491662+0000", 2024-04-20T17:19:28.598 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.018083098999999998 2024-04-20T17:19:28.598 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.598 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.598 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:28.598 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:10:39.491702+0000", 2024-04-20T17:19:28.598 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0002000000000001e-05 2024-04-20T17:19:28.598 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.598 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:28.599 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.599 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.599 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.599 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:285969 2.e e332/202 2:7bd8d0f8:::benchmark_data_smithi171_78918_object124975:head v 332'17775, mlcod=332'17772)", 2024-04-20T17:19:28.599 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:19:04.367056+0000", 2024-04-20T17:19:28.599 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 24.168452568999999, 2024-04-20T17:19:28.599 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011130431, 2024-04-20T17:19:28.599 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:28.599 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:28.599 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:28.599 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:28.599 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.599 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:28.600 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:04.367056+0000", 2024-04-20T17:19:28.600 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.600 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.600 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.600 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:28.600 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:04.367056+0000", 2024-04-20T17:19:28.600 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.600 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.600 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.600 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:28.600 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:04.367057+0000", 2024-04-20T17:19:28.600 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6029999999999999e-06 2024-04-20T17:19:28.600 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.600 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.600 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:28.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:04.367066+0000", 2024-04-20T17:19:28.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8140000000000007e-06 2024-04-20T17:19:28.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:28.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:04.367079+0000", 2024-04-20T17:19:28.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2914e-05 2024-04-20T17:19:28.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:28.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:04.367086+0000", 2024-04-20T17:19:28.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1609999999999997e-06 2024-04-20T17:19:28.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:28.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:04.367143+0000", 2024-04-20T17:19:28.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6895999999999999e-05 2024-04-20T17:19:28.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:28.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:04.367168+0000", 2024-04-20T17:19:28.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4909000000000002e-05 2024-04-20T17:19:28.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:28.603 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:04.378127+0000", 2024-04-20T17:19:28.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010958745000000001 2024-04-20T17:19:28.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.603 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:28.603 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:04.378186+0000", 2024-04-20T17:19:28.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9388999999999997e-05 2024-04-20T17:19:28.603 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.603 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:28.603 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.604 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:288840 2.a e336/115 2:52f2dc04:::benchmark_data_smithi171_78918_object127846:head v 336'18255, mlcod=336'18253)", 2024-04-20T17:19:28.604 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:19:12.719252+0000", 2024-04-20T17:19:28.604 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 15.816255923, 2024-04-20T17:19:28.604 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010489844999999999, 2024-04-20T17:19:28.604 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:28.604 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:28.604 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:28.604 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:28.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.604 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:28.604 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:12.719252+0000", 2024-04-20T17:19:28.604 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.604 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.605 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.605 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:28.605 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:12.719252+0000", 2024-04-20T17:19:28.605 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.605 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.605 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.605 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:28.605 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:12.719256+0000", 2024-04-20T17:19:28.605 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6669999999999998e-06 2024-04-20T17:19:28.605 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.605 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.605 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:28.605 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:12.719277+0000", 2024-04-20T17:19:28.605 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0837999999999999e-05 2024-04-20T17:19:28.606 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.606 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.606 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:28.606 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:12.719302+0000", 2024-04-20T17:19:28.606 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5004e-05 2024-04-20T17:19:28.606 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.606 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.606 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:28.606 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:12.719317+0000", 2024-04-20T17:19:28.606 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5095e-05 2024-04-20T17:19:28.606 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.606 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.606 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:28.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:12.719401+0000", 2024-04-20T17:19:28.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3702000000000006e-05 2024-04-20T17:19:28.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.607 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:28.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:12.719431+0000", 2024-04-20T17:19:28.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0746e-05 2024-04-20T17:19:28.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.607 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:28.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:12.729673+0000", 2024-04-20T17:19:28.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010241422 2024-04-20T17:19:28.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.608 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:28.608 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:12.729742+0000", 2024-04-20T17:19:28.608 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9370999999999997e-05 2024-04-20T17:19:28.608 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.608 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:28.608 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.608 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.608 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.608 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:288982 2.a e336/115 2:588e27ae:::benchmark_data_smithi171_78918_object127988:head v 336'18271, mlcod=336'18269)", 2024-04-20T17:19:28.608 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:19:13.151434+0000", 2024-04-20T17:19:28.608 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 15.384074331000001, 2024-04-20T17:19:28.608 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011516406, 2024-04-20T17:19:28.608 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:28.608 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:28.608 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:28.609 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:28.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.609 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:28.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:13.151434+0000", 2024-04-20T17:19:28.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.609 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:28.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:13.151434+0000", 2024-04-20T17:19:28.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.609 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:28.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:13.151438+0000", 2024-04-20T17:19:28.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6839999999999998e-06 2024-04-20T17:19:28.610 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.610 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.610 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:28.610 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:13.151457+0000", 2024-04-20T17:19:28.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8828e-05 2024-04-20T17:19:28.610 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.610 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.610 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:28.610 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:13.151485+0000", 2024-04-20T17:19:28.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8419e-05 2024-04-20T17:19:28.610 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.610 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.610 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:28.610 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:13.151500+0000", 2024-04-20T17:19:28.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4705000000000001e-05 2024-04-20T17:19:28.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.611 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.611 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:28.611 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:13.152331+0000", 2024-04-20T17:19:28.611 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00083109899999999997 2024-04-20T17:19:28.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.611 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.611 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:28.611 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:13.152361+0000", 2024-04-20T17:19:28.611 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0451999999999999e-05 2024-04-20T17:19:28.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.611 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.611 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:28.612 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:13.162888+0000", 2024-04-20T17:19:28.612 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.01052634 2024-04-20T17:19:28.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.612 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:28.612 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:13.162950+0000", 2024-04-20T17:19:28.612 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2879000000000006e-05 2024-04-20T17:19:28.612 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.612 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:28.612 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.612 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:294675 2.e e338/202 2:7030205d:::benchmark_data_smithi171_78918_object133681:head v 338'18319, mlcod=338'18317)", 2024-04-20T17:19:28.613 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:19:27.239049+0000", 2024-04-20T17:19:28.613 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 1.2964593259999999, 2024-04-20T17:19:28.613 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0088207070000000005, 2024-04-20T17:19:28.613 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:28.613 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:28.613 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:28.613 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:28.613 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.613 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:28.613 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.239049+0000", 2024-04-20T17:19:28.613 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.613 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.613 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:28.614 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.239049+0000", 2024-04-20T17:19:28.614 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.614 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:28.614 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.239051+0000", 2024-04-20T17:19:28.614 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3999999999999999e-06 2024-04-20T17:19:28.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.614 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:28.614 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.239062+0000", 2024-04-20T17:19:28.614 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0641e-05 2024-04-20T17:19:28.615 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.615 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:28.615 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.239078+0000", 2024-04-20T17:19:28.615 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6174000000000001e-05 2024-04-20T17:19:28.615 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.615 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:28.615 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.239087+0000", 2024-04-20T17:19:28.615 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5650000000000002e-06 2024-04-20T17:19:28.615 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.615 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:28.615 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.239166+0000", 2024-04-20T17:19:28.616 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9354e-05 2024-04-20T17:19:28.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.616 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:28.616 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.239190+0000", 2024-04-20T17:19:28.616 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3346e-05 2024-04-20T17:19:28.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.616 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:28.616 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.247824+0000", 2024-04-20T17:19:28.616 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0086346349999999999 2024-04-20T17:19:28.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.616 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:28.616 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.247870+0000", 2024-04-20T17:19:28.617 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5592e-05 2024-04-20T17:19:28.617 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.617 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:28.617 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.617 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.617 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:294932 2.a e338/115 2:55efd8c8:::benchmark_data_smithi171_78918_object133938:head v 338'18644, mlcod=338'18642)", 2024-04-20T17:19:28.617 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:19:27.710115+0000", 2024-04-20T17:19:28.617 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.82539319200000005, 2024-04-20T17:19:28.617 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0098857939999999998, 2024-04-20T17:19:28.617 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:28.617 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:28.617 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:28.618 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:28.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.618 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:28.618 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.710115+0000", 2024-04-20T17:19:28.618 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.618 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.618 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:28.618 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.710115+0000", 2024-04-20T17:19:28.618 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.618 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.618 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:28.618 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.710117+0000", 2024-04-20T17:19:28.618 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0320000000000002e-06 2024-04-20T17:19:28.619 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.619 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:28.619 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.710130+0000", 2024-04-20T17:19:28.619 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2387e-05 2024-04-20T17:19:28.619 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.619 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:28.619 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.710138+0000", 2024-04-20T17:19:28.619 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5599999999999994e-06 2024-04-20T17:19:28.619 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.619 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:28.619 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.710145+0000", 2024-04-20T17:19:28.619 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.888e-06 2024-04-20T17:19:28.619 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.620 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.620 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:28.620 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.710206+0000", 2024-04-20T17:19:28.620 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0785000000000002e-05 2024-04-20T17:19:28.620 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.620 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.620 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:28.620 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.710230+0000", 2024-04-20T17:19:28.620 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3798000000000002e-05 2024-04-20T17:19:28.620 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.620 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.620 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:28.620 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.719936+0000", 2024-04-20T17:19:28.620 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0097062009999999994 2024-04-20T17:19:28.620 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.620 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.621 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:28.621 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.720001+0000", 2024-04-20T17:19:28.621 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5142999999999996e-05 2024-04-20T17:19:28.621 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.621 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:28.621 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.621 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.621 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.621 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:294945 2.a e338/115 2:562bc5c1:::benchmark_data_smithi171_78918_object133951:head v 338'18647, mlcod=338'18644)", 2024-04-20T17:19:28.621 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:19:27.730328+0000", 2024-04-20T17:19:28.621 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.80518055899999996, 2024-04-20T17:19:28.621 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.01500256, 2024-04-20T17:19:28.621 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:28.621 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:28.621 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:28.621 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:28.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.622 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:28.622 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.730328+0000", 2024-04-20T17:19:28.622 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.622 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:28.622 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.730328+0000", 2024-04-20T17:19:28.622 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.622 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:28.622 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.730328+0000", 2024-04-20T17:19:28.622 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6499999999999999e-07 2024-04-20T17:19:28.623 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.623 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:28.623 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.730334+0000", 2024-04-20T17:19:28.623 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2839999999999997e-06 2024-04-20T17:19:28.623 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.623 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:28.623 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.730340+0000", 2024-04-20T17:19:28.623 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9460000000000003e-06 2024-04-20T17:19:28.623 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.623 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:28.623 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.730346+0000", 2024-04-20T17:19:28.623 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0499999999999997e-06 2024-04-20T17:19:28.624 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.624 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.624 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:28.624 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.736145+0000", 2024-04-20T17:19:28.624 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0057990120000000001 2024-04-20T17:19:28.624 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.624 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.624 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:28.624 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.736163+0000", 2024-04-20T17:19:28.624 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8385000000000001e-05 2024-04-20T17:19:28.624 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.624 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.624 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:28.624 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.745263+0000", 2024-04-20T17:19:28.625 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0090996219999999999 2024-04-20T17:19:28.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.625 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.625 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:28.625 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.745330+0000", 2024-04-20T17:19:28.625 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7595999999999995e-05 2024-04-20T17:19:28.625 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.625 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:28.625 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.625 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.625 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.24736.0:294996 2.e e338/202 2:7c381111:::benchmark_data_smithi171_78918_object134002:head v 338'18341, mlcod=338'18339)", 2024-04-20T17:19:28.625 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:19:27.800541+0000", 2024-04-20T17:19:28.625 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.73496718100000002, 2024-04-20T17:19:28.626 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0084381209999999998, 2024-04-20T17:19:28.626 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:19:28.626 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:19:28.626 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:28.626 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:19:28.626 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.626 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:19:28.626 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.800541+0000", 2024-04-20T17:19:28.626 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.626 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.626 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.626 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:19:28.626 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.800541+0000", 2024-04-20T17:19:28.626 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:19:28.627 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.627 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.627 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:19:28.627 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.800544+0000", 2024-04-20T17:19:28.627 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4679999999999999e-06 2024-04-20T17:19:28.627 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.627 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.627 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:19:28.627 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.800552+0000", 2024-04-20T17:19:28.627 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3440000000000001e-06 2024-04-20T17:19:28.627 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.627 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.627 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:19:28.627 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.800566+0000", 2024-04-20T17:19:28.628 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4317e-05 2024-04-20T17:19:28.628 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.628 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.628 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:19:28.628 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.800576+0000", 2024-04-20T17:19:28.628 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7180000000000008e-06 2024-04-20T17:19:28.628 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.628 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.628 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:19:28.628 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.800658+0000", 2024-04-20T17:19:28.628 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2312999999999996e-05 2024-04-20T17:19:28.628 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.628 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.628 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:19:28.628 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.800682+0000", 2024-04-20T17:19:28.629 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3595e-05 2024-04-20T17:19:28.629 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.629 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.629 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:19:28.629 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.808931+0000", 2024-04-20T17:19:28.629 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0082488700000000002 2024-04-20T17:19:28.629 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:19:28.629 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:19:28.629 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:19:28.629 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:27.808979+0000", 2024-04-20T17:19:28.629 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8495999999999998e-05 2024-04-20T17:19:28.629 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.629 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:28.629 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.629 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:19:28.629 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:19:28.630 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:19:29.025 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:19:29.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:29.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:19:29.132 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:19:29.158 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:29.177 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:19:29.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:29.179 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:19:29.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:29 smithi005 ceph-mon[39562]: pgmap v929: 17 pgs: 1 active+recovering, 16 active+clean; 1.7 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 460 op/s; 2601/55510 objects misplaced (4.686%); 952 KiB/s, 14 objects/s recovering 2024-04-20T17:19:29.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:29 smithi106 ceph-mon[44423]: pgmap v929: 17 pgs: 1 active+recovering, 16 active+clean; 1.7 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 460 op/s; 2601/55510 objects misplaced (4.686%); 952 KiB/s, 14 objects/s recovering 2024-04-20T17:19:29.280 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:19:29.315 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:29.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:29 smithi171 ceph-mon[44189]: pgmap v929: 17 pgs: 1 active+recovering, 16 active+clean; 1.7 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 460 op/s; 2601/55510 objects misplaced (4.686%); 952 KiB/s, 14 objects/s recovering 2024-04-20T17:19:29.449 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:19:29.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:29.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:19:29.557 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:19:29.583 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:29.603 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:19:29.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:29.606 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:19:29.707 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:19:29.734 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:29.756 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:19:29.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:29.759 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:19:29.859 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:19:29.886 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:29.905 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:19:29.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:29.907 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:19:30.008 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:19:30.035 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:30.053 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:19:30.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:30.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:19:30.157 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:19:30.184 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:30.202 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:19:30.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:30.204 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:19:30.305 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:19:30.335 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:30.837 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:19:30.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:30.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:19:30.942 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:19:30.969 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:30.988 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:19:30.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:30.994 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:19:31.095 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:19:31.126 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:31 smithi171 ceph-mon[44189]: pgmap v930: 17 pgs: 1 active+recovering, 16 active+clean; 1.7 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 405 op/s; 2351/54434 objects misplaced (4.319%); 1.7 MiB/s, 27 objects/s recovering 2024-04-20T17:19:31.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:31 smithi005 ceph-mon[39562]: pgmap v930: 17 pgs: 1 active+recovering, 16 active+clean; 1.7 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 405 op/s; 2351/54434 objects misplaced (4.319%); 1.7 MiB/s, 27 objects/s recovering 2024-04-20T17:19:31.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:31 smithi106 ceph-mon[44423]: pgmap v930: 17 pgs: 1 active+recovering, 16 active+clean; 1.7 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 405 op/s; 2351/54434 objects misplaced (4.319%); 1.7 MiB/s, 27 objects/s recovering 2024-04-20T17:19:32.283 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:19:33.137 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:33 smithi005 ceph-mon[39562]: pgmap v931: 17 pgs: 1 active+recovering, 16 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 424 op/s; 2351/52932 objects misplaced (4.442%); 1.5 MiB/s, 24 objects/s recovering 2024-04-20T17:19:33.383 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:19:33.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:33.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:19:33.410 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-20T17:19:33.434 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:33 smithi106 ceph-mon[44423]: pgmap v931: 17 pgs: 1 active+recovering, 16 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 424 op/s; 2351/52932 objects misplaced (4.442%); 1.5 MiB/s, 24 objects/s recovering 2024-04-20T17:19:33.489 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:19:33.514 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:33 smithi171 ceph-mon[44189]: pgmap v931: 17 pgs: 1 active+recovering, 16 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 424 op/s; 2351/52932 objects misplaced (4.442%); 1.5 MiB/s, 24 objects/s recovering 2024-04-20T17:19:33.516 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:33.537 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:19:33.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:33.539 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:19:33.640 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:19:33.667 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:33.682 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:19:33.689 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:19:33.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:33.692 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:19:33.793 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:19:33.821 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:33.949 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:19:33.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:33.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:19:34.053 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:19:34.083 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:34.170 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:19:34.208 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:19:34.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:34.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:19:34.312 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:19:34.340 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:34.359 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:19:34.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:34.361 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:19:34.462 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:19:34.491 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:34.546 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:19:34.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:34.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:19:34.643 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:19:34.647 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:19:34.651 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:19:34.679 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:34.702 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:19:34.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:34.706 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:19:34.807 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:19:34.838 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:34.962 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:19:34.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:34.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:19:35.066 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:19:35.096 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:35.155 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:19:35.195 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:19:35.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:35.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:19:35.300 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:19:35.326 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:35.352 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:19:35.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:35.358 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:19:35.459 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:19:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:35 smithi005 ceph-mon[39562]: pgmap v932: 17 pgs: 1 active+recovering, 16 active+clean; 1.5 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 436 op/s; 2071/50222 objects misplaced (4.124%); 1.6 MiB/s, 26 objects/s recovering 2024-04-20T17:19:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:35 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1953951370' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:19:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:35 smithi005 ceph-mon[39562]: from='client.25682 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:19:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:35 smithi005 ceph-mon[39562]: from='client.25688 v1:172.21.15.5:0/3999325557' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:19:35.485 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:35 smithi106 ceph-mon[44423]: pgmap v932: 17 pgs: 1 active+recovering, 16 active+clean; 1.5 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 436 op/s; 2071/50222 objects misplaced (4.124%); 1.6 MiB/s, 26 objects/s recovering 2024-04-20T17:19:35.485 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:35 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1953951370' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:19:35.485 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:35 smithi106 ceph-mon[44423]: from='client.25682 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:19:35.485 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:35 smithi106 ceph-mon[44423]: from='client.25688 v1:172.21.15.5:0/3999325557' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:19:35.487 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:35.541 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:19:35.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:35.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:19:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:35 smithi171 ceph-mon[44189]: pgmap v932: 17 pgs: 1 active+recovering, 16 active+clean; 1.5 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 436 op/s; 2071/50222 objects misplaced (4.124%); 1.6 MiB/s, 26 objects/s recovering 2024-04-20T17:19:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:35 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1953951370' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:19:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:35 smithi171 ceph-mon[44189]: from='client.25682 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:19:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:35 smithi171 ceph-mon[44189]: from='client.25688 v1:172.21.15.5:0/3999325557' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:19:35.647 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:19:35.674 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:35.694 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:19:35.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:35.697 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:19:35.798 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:19:35.827 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:35.838 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:19:35.838 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:19:35.838 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-04-20T17:19:35.838 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:19:35.858 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:19:35.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:35.865 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:19:35.966 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:19:36.001 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:36.356 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:19:36.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:36.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:19:36.460 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:19:36.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:36 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:19:36.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:36 smithi005 ceph-mon[39562]: from='client.25682 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:19:36.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:36 smithi005 ceph-mon[39562]: osdmap e339: 12 total, 12 up, 12 in 2024-04-20T17:19:36.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:36 smithi005 ceph-mon[39562]: osdmap e340: 12 total, 12 up, 12 in 2024-04-20T17:19:36.495 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:36 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:19:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:36 smithi106 ceph-mon[44423]: from='client.25682 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:19:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:36 smithi106 ceph-mon[44423]: osdmap e339: 12 total, 12 up, 12 in 2024-04-20T17:19:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:36 smithi106 ceph-mon[44423]: osdmap e340: 12 total, 12 up, 12 in 2024-04-20T17:19:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:36 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:19:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:36 smithi171 ceph-mon[44189]: from='client.25682 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:19:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:36 smithi171 ceph-mon[44189]: osdmap e339: 12 total, 12 up, 12 in 2024-04-20T17:19:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:36 smithi171 ceph-mon[44189]: osdmap e340: 12 total, 12 up, 12 in 2024-04-20T17:19:36.596 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-20T17:19:36.890 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:19:37.203 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":932,"stamp":"2024-04-20T17:19:34.082170+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1646068280,"num_objects":25111,"num_object_clones":0,"num_object_copies":50222,"num_objects_missing_on_primary":1031,"num_objects_missing":1031,"num_objects_degraded":0,"num_objects_misplaced":2071,"num_objects_unfound":0,"num_objects_dirty":25111,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":297056,"num_write_kb":10306363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28184,"num_bytes_recovered":1848658048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161011,"ondisk_log_size":161011,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":5384888,"kb_used_data":4170264,"kb_used_omap":219,"kb_used_meta":1214244,"kb_avail":1119556936,"statfs":{"total":1151940427776,"available":1146426302464,"internally_reserved":0,"allocated":4270350336,"data_stored":4255827792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224949,"internal_metadata":1243386187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,2,0,2,3,5,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-285999104,"num_objects":-4364,"num_object_clones":0,"num_object_copies":-8728,"num_objects_missing_on_primary":-266,"num_objects_missing":-266,"num_objects_degraded":0,"num_objects_misplaced":-530,"num_objects_unfound":0,"num_objects_dirty":-4364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4364,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":17301504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001912"},"pg_stats":[{"pgid":"2.f","version":"338'18673","reported_seq":19710,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-20T17:19:33.753415+0000","last_change":"2024-04-20T17:19:03.232229+0000","last_active":"2024-04-20T17:19:33.753415+0000","last_peered":"2024-04-20T17:19:33.753415+0000","last_clean":"2024-04-20T17:19:33.753415+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:19:33.753415+0000","last_undegraded":"2024-04-20T17:19:33.753415+0000","last_fullsized":"2024-04-20T17:19:33.753415+0000","mapping_epoch":115,"log_start":"142'8600","ondisk_log_start":"142'8600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'17904","last_scrub_stamp":"2024-04-20T17:19:03.232003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:19:03.232003+0000","objects_scrubbed":2282,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:09.723163+0000","scrub_duration":268,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98893824,"num_objects":1509,"num_object_clones":0,"num_object_copies":3018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18673,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"338'18433","reported_seq":30505,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-20T17:19:30.479651+0000","last_change":"2024-04-20T17:18:08.306341+0000","last_active":"2024-04-20T17:19:30.479651+0000","last_peered":"2024-04-20T17:19:30.479651+0000","last_clean":"2024-04-20T17:19:30.479651+0000","last_became_active":"2024-04-20T17:10:28.865670+0000","last_became_peered":"2024-04-20T17:10:28.865670+0000","last_unstale":"2024-04-20T17:19:30.479651+0000","last_undegraded":"2024-04-20T17:19:30.479651+0000","last_fullsized":"2024-04-20T17:19:30.479651+0000","mapping_epoch":162,"log_start":"142'8400","ondisk_log_start":"142'8400","created":115,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'16860","last_scrub_stamp":"2024-04-20T17:18:08.306156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:08.306156+0000","objects_scrubbed":3197,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:34.841351+0000","scrub_duration":281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106364928,"num_objects":1623,"num_object_clones":0,"num_object_copies":3246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18433,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5619,"num_bytes_recovered":368246784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"338'18790","reported_seq":19677,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-20T17:19:30.492904+0000","last_change":"2024-04-20T17:18:49.298437+0000","last_active":"2024-04-20T17:19:30.492904+0000","last_peered":"2024-04-20T17:19:30.492904+0000","last_clean":"2024-04-20T17:19:30.492904+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:19:30.492904+0000","last_undegraded":"2024-04-20T17:19:30.492904+0000","last_fullsized":"2024-04-20T17:19:30.492904+0000","mapping_epoch":115,"log_start":"142'8700","ondisk_log_start":"142'8700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'17915","last_scrub_stamp":"2024-04-20T17:18:49.298307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:49.298307+0000","objects_scrubbed":2567,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:03.315951+0000","scrub_duration":225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110886912,"num_objects":1692,"num_object_clones":0,"num_object_copies":3384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18790,"num_write_kb":655424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"338'25221","reported_seq":46871,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-20T17:19:30.688847+0000","last_change":"2024-04-20T17:17:53.095109+0000","last_active":"2024-04-20T17:19:30.688847+0000","last_peered":"2024-04-20T17:19:30.688847+0000","last_clean":"2024-04-20T17:19:30.688847+0000","last_became_active":"2024-04-20T17:12:08.085936+0000","last_became_peered":"2024-04-20T17:12:08.085936+0000","last_unstale":"2024-04-20T17:19:30.688847+0000","last_undegraded":"2024-04-20T17:19:30.688847+0000","last_fullsized":"2024-04-20T17:19:30.688847+0000","mapping_epoch":196,"log_start":"207'15168","ondisk_log_start":"207'15168","created":115,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'23389","last_scrub_stamp":"2024-04-20T17:17:53.094985+0000","last_deep_scrub":"213'20373","last_deep_scrub_stamp":"2024-04-20T17:12:48.443808+0000","last_clean_scrub_stamp":"2024-04-20T17:17:53.094985+0000","objects_scrubbed":3498,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:30.911551+0000","scrub_duration":306,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109117440,"num_objects":1665,"num_object_clones":0,"num_object_copies":3330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18569,"num_write_kb":647488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":450166784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"338'18261","reported_seq":43910,"reported_epoch":338,"state":"active+recovering","last_fresh":"2024-04-20T17:19:33.947004+0000","last_change":"2024-04-20T17:12:26.166109+0000","last_active":"2024-04-20T17:19:33.947004+0000","last_peered":"2024-04-20T17:19:33.947004+0000","last_clean":"2024-04-20T17:09:35.882603+0000","last_became_active":"2024-04-20T17:10:54.470253+0000","last_became_peered":"2024-04-20T17:10:54.470253+0000","last_unstale":"2024-04-20T17:19:33.947004+0000","last_undegraded":"2024-04-20T17:19:33.947004+0000","last_fullsized":"2024-04-20T17:19:33.947004+0000","mapping_epoch":171,"log_start":"142'8200","ondisk_log_start":"142'8200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96272384,"num_objects":1469,"num_object_clones":0,"num_object_copies":2938,"num_objects_missing_on_primary":1031,"num_objects_missing":1031,"num_objects_degraded":0,"num_objects_misplaced":2071,"num_objects_unfound":0,"num_objects_dirty":1469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18261,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8392,"num_bytes_recovered":549978112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"0,2,5","objects":9},{"shards":"0,10","objects":429},{"shards":"2,5","objects":1031}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":880,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-20T17:19:26.002889+0000","last_change":"2024-04-20T17:17:58.199584+0000","last_active":"2024-04-20T17:19:26.002889+0000","last_peered":"2024-04-20T17:19:26.002889+0000","last_clean":"2024-04-20T17:19:26.002889+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:19:26.002889+0000","last_undegraded":"2024-04-20T17:19:26.002889+0000","last_fullsized":"2024-04-20T17:19:26.002889+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:17:58.199502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:17:58.199502+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:30.136711+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"338'21694","reported_seq":22708,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-20T17:19:32.595972+0000","last_change":"2024-04-20T17:18:28.829916+0000","last_active":"2024-04-20T17:19:32.595972+0000","last_peered":"2024-04-20T17:19:32.595972+0000","last_clean":"2024-04-20T17:19:32.595972+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:19:32.595972+0000","last_undegraded":"2024-04-20T17:19:32.595972+0000","last_fullsized":"2024-04-20T17:19:32.595972+0000","mapping_epoch":115,"log_start":"158'11600","ondisk_log_start":"158'11600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'20320","last_scrub_stamp":"2024-04-20T17:18:28.829770+0000","last_deep_scrub":"321'20320","last_deep_scrub_stamp":"2024-04-20T17:18:28.829770+0000","last_clean_scrub_stamp":"2024-04-20T17:18:28.829770+0000","objects_scrubbed":3006,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":10,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:40.227886+0000","scrub_duration":9561,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102498304,"num_objects":1564,"num_object_clones":0,"num_object_copies":3128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18688,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"338'18820","reported_seq":19737,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-20T17:19:33.788452+0000","last_change":"2024-04-20T17:18:49.142560+0000","last_active":"2024-04-20T17:19:33.788452+0000","last_peered":"2024-04-20T17:19:33.788452+0000","last_clean":"2024-04-20T17:19:33.788452+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:19:33.788452+0000","last_undegraded":"2024-04-20T17:19:33.788452+0000","last_fullsized":"2024-04-20T17:19:33.788452+0000","mapping_epoch":115,"log_start":"143'8800","ondisk_log_start":"143'8800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'17862","last_scrub_stamp":"2024-04-20T17:18:49.142459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:49.142459+0000","objects_scrubbed":2464,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:07.477325+0000","scrub_duration":245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98566144,"num_objects":1504,"num_object_clones":0,"num_object_copies":3008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18820,"num_write_kb":650368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"338'22615","reported_seq":23640,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-20T17:19:31.989219+0000","last_change":"2024-04-20T17:17:55.709069+0000","last_active":"2024-04-20T17:19:31.989219+0000","last_peered":"2024-04-20T17:19:31.989219+0000","last_clean":"2024-04-20T17:19:31.989219+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:19:31.989219+0000","last_undegraded":"2024-04-20T17:19:31.989219+0000","last_fullsized":"2024-04-20T17:19:31.989219+0000","mapping_epoch":115,"log_start":"166'12602","ondisk_log_start":"166'12602","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"309'20878","last_scrub_stamp":"2024-04-20T17:17:55.708833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:55.708833+0000","objects_scrubbed":3275,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:32.440334+0000","scrub_duration":320,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100794368,"num_objects":1538,"num_object_clones":0,"num_object_copies":3076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18504,"num_write_kb":641344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"338'18631","reported_seq":19606,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-20T17:19:32.043671+0000","last_change":"2024-04-20T17:18:45.620454+0000","last_active":"2024-04-20T17:19:32.043671+0000","last_peered":"2024-04-20T17:19:32.043671+0000","last_clean":"2024-04-20T17:19:32.043671+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:19:32.043671+0000","last_undegraded":"2024-04-20T17:19:32.043671+0000","last_fullsized":"2024-04-20T17:19:32.043671+0000","mapping_epoch":115,"log_start":"142'8600","ondisk_log_start":"142'8600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'17642","last_scrub_stamp":"2024-04-20T17:18:45.620321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:45.620321+0000","objects_scrubbed":2579,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:50.964012+0000","scrub_duration":253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104136704,"num_objects":1589,"num_object_clones":0,"num_object_copies":3178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18631,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"338'18480","reported_seq":19379,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-20T17:19:32.704345+0000","last_change":"2024-04-20T17:18:45.702943+0000","last_active":"2024-04-20T17:19:32.704345+0000","last_peered":"2024-04-20T17:19:32.704345+0000","last_clean":"2024-04-20T17:19:32.704345+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:19:32.704345+0000","last_undegraded":"2024-04-20T17:19:32.704345+0000","last_fullsized":"2024-04-20T17:19:32.704345+0000","mapping_epoch":115,"log_start":"142'8400","ondisk_log_start":"142'8400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'17455","last_scrub_stamp":"2024-04-20T17:18:45.702769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:45.702769+0000","objects_scrubbed":2599,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:54.717372+0000","scrub_duration":222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103022592,"num_objects":1572,"num_object_clones":0,"num_object_copies":3144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18480,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"338'18717","reported_seq":34185,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-20T17:19:32.615587+0000","last_change":"2024-04-20T17:17:57.501217+0000","last_active":"2024-04-20T17:19:32.615587+0000","last_peered":"2024-04-20T17:19:32.615587+0000","last_clean":"2024-04-20T17:19:32.615587+0000","last_became_active":"2024-04-20T17:10:55.931751+0000","last_became_peered":"2024-04-20T17:10:55.931751+0000","last_unstale":"2024-04-20T17:19:32.615587+0000","last_undegraded":"2024-04-20T17:19:32.615587+0000","last_fullsized":"2024-04-20T17:19:32.615587+0000","mapping_epoch":173,"log_start":"142'8700","ondisk_log_start":"142'8700","created":115,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'16916","last_scrub_stamp":"2024-04-20T17:17:57.501089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:57.501089+0000","objects_scrubbed":3350,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":318,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101253120,"num_objects":1545,"num_object_clones":0,"num_object_copies":3090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18717,"num_write_kb":648384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7296,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"338'18786","reported_seq":19710,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-20T17:19:32.580181+0000","last_change":"2024-04-20T17:18:06.314140+0000","last_active":"2024-04-20T17:19:32.580181+0000","last_peered":"2024-04-20T17:19:32.580181+0000","last_clean":"2024-04-20T17:19:32.580181+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:19:32.580181+0000","last_undegraded":"2024-04-20T17:19:32.580181+0000","last_fullsized":"2024-04-20T17:19:32.580181+0000","mapping_epoch":115,"log_start":"142'8700","ondisk_log_start":"142'8700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"314'17127","last_scrub_stamp":"2024-04-20T17:18:06.313989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:06.313989+0000","objects_scrubbed":3191,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:37.449169+0000","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100270080,"num_objects":1530,"num_object_clones":0,"num_object_copies":3060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18786,"num_write_kb":650112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"338'18320","reported_seq":19233,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-20T17:19:33.716771+0000","last_change":"2024-04-20T17:18:47.241194+0000","last_active":"2024-04-20T17:19:33.716771+0000","last_peered":"2024-04-20T17:19:33.716771+0000","last_clean":"2024-04-20T17:19:33.716771+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:19:33.716771+0000","last_undegraded":"2024-04-20T17:19:33.716771+0000","last_fullsized":"2024-04-20T17:19:33.716771+0000","mapping_epoch":115,"log_start":"142'8300","ondisk_log_start":"142'8300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'17283","last_scrub_stamp":"2024-04-20T17:18:47.241064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:47.241064+0000","objects_scrubbed":2584,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:10.974140+0000","scrub_duration":266,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101187584,"num_objects":1544,"num_object_clones":0,"num_object_copies":3088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18320,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"338'18497","reported_seq":19411,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-20T17:19:32.565248+0000","last_change":"2024-04-20T17:18:48.599315+0000","last_active":"2024-04-20T17:19:32.565248+0000","last_peered":"2024-04-20T17:19:32.565248+0000","last_clean":"2024-04-20T17:19:32.565248+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:19:32.565248+0000","last_undegraded":"2024-04-20T17:19:32.565248+0000","last_fullsized":"2024-04-20T17:19:32.565248+0000","mapping_epoch":115,"log_start":"142'8400","ondisk_log_start":"142'8400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'17566","last_scrub_stamp":"2024-04-20T17:18:48.599209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:48.599209+0000","objects_scrubbed":2538,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:05.548836+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105185280,"num_objects":1605,"num_object_clones":0,"num_object_copies":3210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18497,"num_write_kb":643264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"338'18298","reported_seq":19211,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-20T17:19:30.676214+0000","last_change":"2024-04-20T17:18:52.214316+0000","last_active":"2024-04-20T17:19:30.676214+0000","last_peered":"2024-04-20T17:19:30.676214+0000","last_clean":"2024-04-20T17:19:30.676214+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:19:30.676214+0000","last_undegraded":"2024-04-20T17:19:30.676214+0000","last_fullsized":"2024-04-20T17:19:30.676214+0000","mapping_epoch":115,"log_start":"142'8200","ondisk_log_start":"142'8200","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'17470","last_scrub_stamp":"2024-04-20T17:18:52.214208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:52.214208+0000","objects_scrubbed":2403,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:55.920246+0000","scrub_duration":228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102957080,"num_objects":1572,"num_object_clones":0,"num_object_copies":3144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1572,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18298,"num_write_kb":635777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"338'18408","reported_seq":19200,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-20T17:19:30.753253+0000","last_change":"2024-04-20T17:18:17.426012+0000","last_active":"2024-04-20T17:19:30.753253+0000","last_peered":"2024-04-20T17:19:30.753253+0000","last_clean":"2024-04-20T17:19:30.753253+0000","last_became_active":"2024-04-20T17:12:23.515375+0000","last_became_peered":"2024-04-20T17:12:23.515375+0000","last_unstale":"2024-04-20T17:19:30.753253+0000","last_undegraded":"2024-04-20T17:19:30.753253+0000","last_fullsized":"2024-04-20T17:19:30.753253+0000","mapping_epoch":202,"log_start":"142'8400","ondisk_log_start":"142'8400","created":115,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'16928","last_scrub_stamp":"2024-04-20T17:18:17.425878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:17.425878+0000","objects_scrubbed":3071,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:47.218525+0000","scrub_duration":353,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104071168,"num_objects":1588,"num_object_clones":0,"num_object_copies":3176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18408,"num_write_kb":639872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":1645477912,"num_objects":25109,"num_object_clones":0,"num_object_copies":50218,"num_objects_missing_on_primary":1031,"num_objects_missing":1031,"num_objects_degraded":0,"num_objects_misplaced":2071,"num_objects_unfound":0,"num_objects_dirty":25109,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":296875,"num_write_kb":10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28176,"num_bytes_recovered":1846542336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4201062400,"data_stored":4201054256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},"log_size":160874,"ondisk_log_size":160874,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991398,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":372368,"kb_used_data":305096,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93372784,"statfs":{"total":95995035648,"available":95613730816,"internally_reserved":0,"allocated":312418304,"data_stored":311205794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":68876740},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253036,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":511816,"kb_used_data":327936,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93233336,"statfs":{"total":95995035648,"available":95470936064,"internally_reserved":0,"allocated":335806464,"data_stored":334597506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":188283328},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612785,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":561036,"kb_used_data":421572,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93184116,"statfs":{"total":95995035648,"available":95420534784,"internally_reserved":0,"allocated":431689728,"data_stored":430476718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":142801344},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809015,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":495880,"kb_used_data":396672,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93249272,"statfs":{"total":95995035648,"available":95487254528,"internally_reserved":0,"allocated":406192128,"data_stored":404983190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168764,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":412240,"kb_used_data":304712,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93332912,"statfs":{"total":95995035648,"available":95572901888,"internally_reserved":0,"allocated":312025088,"data_stored":310812598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430400,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":323336,"kb_used_data":211648,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93421816,"statfs":{"total":95995035648,"available":95663939584,"internally_reserved":0,"allocated":216727552,"data_stored":215518614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575066,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317064,"kb_used_data":209536,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93428088,"statfs":{"total":95995035648,"available":95670362112,"internally_reserved":0,"allocated":214564864,"data_stored":213355926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738591,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":467528,"kb_used_data":328000,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93277624,"statfs":{"total":95995035648,"available":95516286976,"internally_reserved":0,"allocated":335872000,"data_stored":334663062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,0,2,3,5,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393879,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":574408,"kb_used_data":534336,"kb_used_omap":202,"kb_used_meta":39861,"kb_avail":93170744,"statfs":{"total":95995035648,"available":95406841856,"internally_reserved":0,"allocated":547160064,"data_stored":545951126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":40818135},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182607,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":282312,"kb_used_data":206720,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93462840,"statfs":{"total":95995035648,"available":95705948160,"internally_reserved":0,"allocated":211681280,"data_stored":210472342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986378,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":139596,"kb_used_data":104260,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93605556,"statfs":{"total":95995035648,"available":95852089344,"internally_reserved":0,"allocated":106762240,"data_stored":105549230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790148,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":927304,"kb_used_data":819776,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92817848,"statfs":{"total":95995035648,"available":95045476352,"internally_reserved":0,"allocated":839450624,"data_stored":838241686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":329187328,"data_stored":329187328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208797696,"data_stored":208797696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":544210944,"data_stored":544210944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":207028224,"data_stored":207028224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":103092224,"data_stored":103088152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":835649536,"data_stored":835649536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":210698240,"data_stored":210698240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":306774016,"data_stored":306774016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":396492800,"data_stored":396492800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":423694336,"data_stored":423690264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":327024640,"data_stored":327024640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":308412416,"data_stored":308412416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:19:37.206 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:19:37.434 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:37 smithi106 ceph-mon[44423]: pgmap v935: 17 pgs: 1 active+recovering, 16 active+clean; 1.5 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 415 op/s; 2071/48862 objects misplaced (4.238%); 2.1 MiB/s, 32 objects/s recovering 2024-04-20T17:19:37.434 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:37 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:19:37.434 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:37 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:19:37.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:37 smithi005 ceph-mon[39562]: pgmap v935: 17 pgs: 1 active+recovering, 16 active+clean; 1.5 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 415 op/s; 2071/48862 objects misplaced (4.238%); 2.1 MiB/s, 32 objects/s recovering 2024-04-20T17:19:37.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:37 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:19:37.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:37 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:19:37.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:37 smithi171 ceph-mon[44189]: pgmap v935: 17 pgs: 1 active+recovering, 16 active+clean; 1.5 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 415 op/s; 2071/48862 objects misplaced (4.238%); 2.1 MiB/s, 32 objects/s recovering 2024-04-20T17:19:37.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:37 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:19:37.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:37 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:19:38.638 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:19:38.652 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:19:38.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:38.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:19:38.660 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:19:38.690 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:19:38.713 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:19:38.734 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:19:38.735 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:19:38.756 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:19:38.757 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:19:38.780 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:19:38.785 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:38.803 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:19:38.806 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:19:38.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:38.809 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:19:38.825 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:19:38.852 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:19:38.864 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:19:38.864 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:19:38.874 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:19:38.903 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:19:38.904 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:19:38.905 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:19:38.905 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:19:38.905 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:19:38.905 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:19:38.905 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:19:38.905 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:19:38.905 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:19:38.905 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:19:38.905 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:19:38.905 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:19:38.905 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:19:38.910 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:19:38.940 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:39.146 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:39 smithi106 ceph-mon[44423]: 2.6 scrub starts 2024-04-20T17:19:39.147 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:39 smithi106 ceph-mon[44423]: 2.6 scrub ok 2024-04-20T17:19:39.147 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:39 smithi106 ceph-mon[44423]: pgmap v936: 17 pgs: 1 active+recovering, 16 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 490 op/s; 2071/46584 objects misplaced (4.446%); 1.1 MiB/s, 17 objects/s recovering 2024-04-20T17:19:39.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:39 smithi005 ceph-mon[39562]: 2.6 scrub starts 2024-04-20T17:19:39.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:39 smithi005 ceph-mon[39562]: 2.6 scrub ok 2024-04-20T17:19:39.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:39 smithi005 ceph-mon[39562]: pgmap v936: 17 pgs: 1 active+recovering, 16 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 490 op/s; 2071/46584 objects misplaced (4.446%); 1.1 MiB/s, 17 objects/s recovering 2024-04-20T17:19:39.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:39 smithi005 ceph-mon[39562]: 2.9 scrub starts 2024-04-20T17:19:39.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:39 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3109271884' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:19:39.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:39.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:39 smithi106 ceph-mon[44423]: 2.9 scrub starts 2024-04-20T17:19:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:39 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3109271884' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:19:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:39 smithi171 ceph-mon[44189]: 2.6 scrub starts 2024-04-20T17:19:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:39 smithi171 ceph-mon[44189]: 2.6 scrub ok 2024-04-20T17:19:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:39 smithi171 ceph-mon[44189]: pgmap v936: 17 pgs: 1 active+recovering, 16 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 490 op/s; 2071/46584 objects misplaced (4.446%); 1.1 MiB/s, 17 objects/s recovering 2024-04-20T17:19:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:39 smithi171 ceph-mon[44189]: 2.9 scrub starts 2024-04-20T17:19:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:39 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3109271884' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:19:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:39.593 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:19:39.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:39.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:19:39.603 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-20T17:19:39.698 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:19:39.727 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:39.748 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:19:39.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:39.753 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:19:39.853 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:19:39.888 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:39.924 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:19:39.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:39.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:19:39.929 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:19:40.029 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:19:40.043 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:19:40.055 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:40.203 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:19:40.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:40.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:19:40.307 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:19:40.332 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:40 smithi106 ceph-mon[44423]: 2.9 scrub ok 2024-04-20T17:19:40.333 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.333 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.333 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.333 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.333 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.333 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.333 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.333 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.333 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.333 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.333 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.333 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:40 smithi106 ceph-mon[44423]: 2.1 scrub starts 2024-04-20T17:19:40.333 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:40 smithi106 ceph-mon[44423]: 2.1 scrub ok 2024-04-20T17:19:40.333 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:40 smithi106 ceph-mon[44423]: 2.a scrub starts 2024-04-20T17:19:40.334 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:40 smithi106 ceph-mon[44423]: 2.a scrub ok 2024-04-20T17:19:40.334 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:40.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:40 smithi005 ceph-mon[39562]: 2.9 scrub ok 2024-04-20T17:19:40.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:40 smithi005 ceph-mon[39562]: 2.1 scrub starts 2024-04-20T17:19:40.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:40 smithi005 ceph-mon[39562]: 2.1 scrub ok 2024-04-20T17:19:40.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:40 smithi005 ceph-mon[39562]: 2.a scrub starts 2024-04-20T17:19:40.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:40 smithi005 ceph-mon[39562]: 2.a scrub ok 2024-04-20T17:19:40.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:40 smithi171 ceph-mon[44189]: 2.9 scrub ok 2024-04-20T17:19:40.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:40.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:40 smithi171 ceph-mon[44189]: 2.1 scrub starts 2024-04-20T17:19:40.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:40 smithi171 ceph-mon[44189]: 2.1 scrub ok 2024-04-20T17:19:40.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:40 smithi171 ceph-mon[44189]: 2.a scrub starts 2024-04-20T17:19:40.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:40 smithi171 ceph-mon[44189]: 2.a scrub ok 2024-04-20T17:19:40.785 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:19:40.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:40.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:19:40.889 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:19:40.926 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:40.951 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:19:40.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:40.955 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:19:41.056 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:19:41.084 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:41.314 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:19:41.332 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:19:41.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:41 smithi171 ceph-mon[44189]: pgmap v937: 17 pgs: 1 active+recovering, 16 active+clean; 1.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 469 op/s; 1772/45418 objects misplaced (3.902%); 2.3 MiB/s, 36 objects/s recovering 2024-04-20T17:19:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:41 smithi171 ceph-mon[44189]: 2.4 scrub starts 2024-04-20T17:19:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:41 smithi171 ceph-mon[44189]: 2.4 scrub ok 2024-04-20T17:19:41.334 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:41 smithi106 ceph-mon[44423]: pgmap v937: 17 pgs: 1 active+recovering, 16 active+clean; 1.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 469 op/s; 1772/45418 objects misplaced (3.902%); 2.3 MiB/s, 36 objects/s recovering 2024-04-20T17:19:41.335 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:41 smithi106 ceph-mon[44423]: 2.4 scrub starts 2024-04-20T17:19:41.335 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:41 smithi106 ceph-mon[44423]: 2.4 scrub ok 2024-04-20T17:19:41.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:41.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:19:41.437 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:19:41.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:41 smithi005 ceph-mon[39562]: pgmap v937: 17 pgs: 1 active+recovering, 16 active+clean; 1.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 469 op/s; 1772/45418 objects misplaced (3.902%); 2.3 MiB/s, 36 objects/s recovering 2024-04-20T17:19:41.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:41 smithi005 ceph-mon[39562]: 2.4 scrub starts 2024-04-20T17:19:41.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:41 smithi005 ceph-mon[39562]: 2.4 scrub ok 2024-04-20T17:19:41.463 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:41.664 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:19:41.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:41.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:19:41.768 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:19:41.801 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:41.837 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:19:41.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:41.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:19:41.929 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:19:41.929 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:19:41.943 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:19:41.969 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:19:41.970 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:42.174 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:19:42.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:42.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:19:42.251 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:19:42.253 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:19:42.296 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:19:42.331 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:42.354 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:19:42.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:42.357 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:19:42.458 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:19:42.485 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:42.504 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:19:42.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:42.506 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:19:42.607 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:19:42.643 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:42.666 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:19:42.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:42.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:19:42.670 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-20T17:19:42.770 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:19:42.801 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:42.821 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:19:42.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:42.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:19:42.926 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:19:42.959 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:43.009 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:19:43.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:43 smithi005 ceph-mon[39562]: pgmap v938: 17 pgs: 1 active+recovering, 16 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 428 op/s; 1772/43360 objects misplaced (4.087%); 1.2 MiB/s, 18 objects/s recovering 2024-04-20T17:19:43.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:43 smithi005 ceph-mon[39562]: from='client.15756 v1:172.21.15.5:0/2821703941' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:19:43.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:43 smithi106 ceph-mon[44423]: pgmap v938: 17 pgs: 1 active+recovering, 16 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 428 op/s; 1772/43360 objects misplaced (4.087%); 1.2 MiB/s, 18 objects/s recovering 2024-04-20T17:19:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:43 smithi106 ceph-mon[44423]: from='client.15756 v1:172.21.15.5:0/2821703941' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:19:43.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:43 smithi171 ceph-mon[44189]: pgmap v938: 17 pgs: 1 active+recovering, 16 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 428 op/s; 1772/43360 objects misplaced (4.087%); 1.2 MiB/s, 18 objects/s recovering 2024-04-20T17:19:43.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:43 smithi171 ceph-mon[44189]: from='client.15756 v1:172.21.15.5:0/2821703941' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:19:44.042 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:19:44.042 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:19:44.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:44.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:19:44.076 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":938,"stamp":"2024-04-20T17:19:42.083594+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1421214264,"num_objects":21680,"num_object_clones":0,"num_object_copies":43360,"num_objects_missing_on_primary":883,"num_objects_missing":883,"num_objects_degraded":0,"num_objects_misplaced":1772,"num_objects_unfound":0,"num_objects_dirty":21680,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":300487,"num_write_kb":10306363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28335,"num_bytes_recovered":1858553984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160942,"ondisk_log_size":160942,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":4955052,"kb_used_data":3744632,"kb_used_omap":219,"kb_used_meta":1210084,"kb_avail":1119986772,"statfs":{"total":1151940427776,"available":1146866454528,"internally_reserved":0,"allocated":3834503168,"data_stored":3819837152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224949,"internal_metadata":1239126347},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,6,1,1,2,1,3,2,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-224854016,"num_objects":-3431,"num_object_clones":0,"num_object_copies":-6862,"num_objects_missing_on_primary":-148,"num_objects_missing":-148,"num_objects_degraded":0,"num_objects_misplaced":-299,"num_objects_unfound":0,"num_objects_dirty":-3431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3431,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":9895936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001423"},"pg_stats":[{"pgid":"2.f","version":"340'18817","reported_seq":19858,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-04-20T17:19:38.480450+0000","last_change":"2024-04-20T17:19:03.232229+0000","last_active":"2024-04-20T17:19:38.480450+0000","last_peered":"2024-04-20T17:19:38.480450+0000","last_clean":"2024-04-20T17:19:38.480450+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:19:38.480450+0000","last_undegraded":"2024-04-20T17:19:38.480450+0000","last_fullsized":"2024-04-20T17:19:38.480450+0000","mapping_epoch":115,"log_start":"143'8800","ondisk_log_start":"143'8800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'17904","last_scrub_stamp":"2024-04-20T17:19:03.232003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:19:03.232003+0000","objects_scrubbed":2282,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:09.723163+0000","scrub_duration":268,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89456640,"num_objects":1365,"num_object_clones":0,"num_object_copies":2730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18817,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"340'18755","reported_seq":30852,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-04-20T17:19:40.558714+0000","last_change":"2024-04-20T17:19:40.244380+0000","last_active":"2024-04-20T17:19:40.558714+0000","last_peered":"2024-04-20T17:19:40.558714+0000","last_clean":"2024-04-20T17:19:40.558714+0000","last_became_active":"2024-04-20T17:10:28.865670+0000","last_became_peered":"2024-04-20T17:10:28.865670+0000","last_unstale":"2024-04-20T17:19:40.558714+0000","last_undegraded":"2024-04-20T17:19:40.558714+0000","last_fullsized":"2024-04-20T17:19:40.558714+0000","mapping_epoch":162,"log_start":"143'8700","ondisk_log_start":"143'8700","created":115,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'18745","last_scrub_stamp":"2024-04-20T17:19:40.244244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:19:40.244244+0000","objects_scrubbed":1311,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:59.798440+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85262336,"num_objects":1301,"num_object_clones":0,"num_object_copies":2602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18755,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5619,"num_bytes_recovered":368246784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"340'19087","reported_seq":19978,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-04-20T17:19:40.539392+0000","last_change":"2024-04-20T17:18:49.298437+0000","last_active":"2024-04-20T17:19:40.539392+0000","last_peered":"2024-04-20T17:19:40.539392+0000","last_clean":"2024-04-20T17:19:40.539392+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:19:40.539392+0000","last_undegraded":"2024-04-20T17:19:40.539392+0000","last_fullsized":"2024-04-20T17:19:40.539392+0000","mapping_epoch":115,"log_start":"144'9000","ondisk_log_start":"144'9000","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'17915","last_scrub_stamp":"2024-04-20T17:18:49.298307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:49.298307+0000","objects_scrubbed":2567,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:03.315951+0000","scrub_duration":225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91422720,"num_objects":1395,"num_object_clones":0,"num_object_copies":2790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19087,"num_write_kb":655424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"340'25529","reported_seq":47205,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-04-20T17:19:40.742332+0000","last_change":"2024-04-20T17:19:39.130990+0000","last_active":"2024-04-20T17:19:40.742332+0000","last_peered":"2024-04-20T17:19:40.742332+0000","last_clean":"2024-04-20T17:19:40.742332+0000","last_became_active":"2024-04-20T17:12:08.085936+0000","last_became_peered":"2024-04-20T17:12:08.085936+0000","last_unstale":"2024-04-20T17:19:40.742332+0000","last_undegraded":"2024-04-20T17:19:40.742332+0000","last_fullsized":"2024-04-20T17:19:40.742332+0000","mapping_epoch":196,"log_start":"207'15468","ondisk_log_start":"207'15468","created":115,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'25482","last_scrub_stamp":"2024-04-20T17:19:39.130906+0000","last_deep_scrub":"213'20373","last_deep_scrub_stamp":"2024-04-20T17:12:48.443808+0000","last_clean_scrub_stamp":"2024-04-20T17:19:39.130906+0000","objects_scrubbed":1406,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:52.548198+0000","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88932352,"num_objects":1357,"num_object_clones":0,"num_object_copies":2714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18877,"num_write_kb":647488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":450166784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"340'18411","reported_seq":44508,"reported_epoch":340,"state":"active+recovering","last_fresh":"2024-04-20T17:19:38.978636+0000","last_change":"2024-04-20T17:12:26.166109+0000","last_active":"2024-04-20T17:19:38.978636+0000","last_peered":"2024-04-20T17:19:38.978636+0000","last_clean":"2024-04-20T17:09:35.882603+0000","last_became_active":"2024-04-20T17:10:54.470253+0000","last_became_peered":"2024-04-20T17:10:54.470253+0000","last_unstale":"2024-04-20T17:19:38.978636+0000","last_undegraded":"2024-04-20T17:19:38.978636+0000","last_fullsized":"2024-04-20T17:19:38.978636+0000","mapping_epoch":171,"log_start":"142'8400","ondisk_log_start":"142'8400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86441984,"num_objects":1319,"num_object_clones":0,"num_object_copies":2638,"num_objects_missing_on_primary":883,"num_objects_missing":883,"num_objects_degraded":0,"num_objects_misplaced":1772,"num_objects_unfound":0,"num_objects_dirty":1319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18411,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8543,"num_bytes_recovered":559874048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"0,2,5","objects":6},{"shards":"0,10","objects":430},{"shards":"2,5","objects":883}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":892,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-04-20T17:19:36.190237+0000","last_change":"2024-04-20T17:19:36.190286+0000","last_active":"2024-04-20T17:19:36.190237+0000","last_peered":"2024-04-20T17:19:36.190237+0000","last_clean":"2024-04-20T17:19:36.190237+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:19:36.190237+0000","last_undegraded":"2024-04-20T17:19:36.190237+0000","last_fullsized":"2024-04-20T17:19:36.190237+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:19:36.190151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:19:36.190151+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:40.598718+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"340'21858","reported_seq":22876,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-04-20T17:19:37.626642+0000","last_change":"2024-04-20T17:18:28.829916+0000","last_active":"2024-04-20T17:19:37.626642+0000","last_peered":"2024-04-20T17:19:37.626642+0000","last_clean":"2024-04-20T17:19:37.626642+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:19:37.626642+0000","last_undegraded":"2024-04-20T17:19:37.626642+0000","last_fullsized":"2024-04-20T17:19:37.626642+0000","mapping_epoch":115,"log_start":"160'11800","ondisk_log_start":"160'11800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'20320","last_scrub_stamp":"2024-04-20T17:18:28.829770+0000","last_deep_scrub":"321'20320","last_deep_scrub_stamp":"2024-04-20T17:18:28.829770+0000","last_clean_scrub_stamp":"2024-04-20T17:18:28.829770+0000","objects_scrubbed":3006,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":10,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:40.227886+0000","scrub_duration":9561,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91750400,"num_objects":1400,"num_object_clones":0,"num_object_copies":2800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18852,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"340'18958","reported_seq":19879,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-04-20T17:19:38.413353+0000","last_change":"2024-04-20T17:18:49.142560+0000","last_active":"2024-04-20T17:19:38.413353+0000","last_peered":"2024-04-20T17:19:38.413353+0000","last_clean":"2024-04-20T17:19:38.413353+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:19:38.413353+0000","last_undegraded":"2024-04-20T17:19:38.413353+0000","last_fullsized":"2024-04-20T17:19:38.413353+0000","mapping_epoch":115,"log_start":"143'8900","ondisk_log_start":"143'8900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'17862","last_scrub_stamp":"2024-04-20T17:18:49.142459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:49.142459+0000","objects_scrubbed":2464,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:07.477325+0000","scrub_duration":245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89522176,"num_objects":1366,"num_object_clones":0,"num_object_copies":2732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18958,"num_write_kb":650368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"340'22920","reported_seq":23970,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-04-20T17:19:41.886029+0000","last_change":"2024-04-20T17:19:37.659813+0000","last_active":"2024-04-20T17:19:41.886029+0000","last_peered":"2024-04-20T17:19:41.886029+0000","last_clean":"2024-04-20T17:19:41.886029+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:19:41.886029+0000","last_undegraded":"2024-04-20T17:19:41.886029+0000","last_fullsized":"2024-04-20T17:19:41.886029+0000","mapping_epoch":115,"log_start":"167'12902","ondisk_log_start":"167'12902","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'22771","last_scrub_stamp":"2024-04-20T17:19:37.659563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:19:37.659563+0000","objects_scrubbed":1384,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:39.997966+0000","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80805888,"num_objects":1233,"num_object_clones":0,"num_object_copies":2466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18809,"num_write_kb":641344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"340'18809","reported_seq":19788,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-04-20T17:19:37.349670+0000","last_change":"2024-04-20T17:18:45.620454+0000","last_active":"2024-04-20T17:19:37.349670+0000","last_peered":"2024-04-20T17:19:37.349670+0000","last_clean":"2024-04-20T17:19:37.349670+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:19:37.349670+0000","last_undegraded":"2024-04-20T17:19:37.349670+0000","last_fullsized":"2024-04-20T17:19:37.349670+0000","mapping_epoch":115,"log_start":"143'8800","ondisk_log_start":"143'8800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'17642","last_scrub_stamp":"2024-04-20T17:18:45.620321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:45.620321+0000","objects_scrubbed":2579,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:50.964012+0000","scrub_duration":253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92471296,"num_objects":1411,"num_object_clones":0,"num_object_copies":2822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18809,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"340'18660","reported_seq":19563,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-04-20T17:19:37.699094+0000","last_change":"2024-04-20T17:18:45.702943+0000","last_active":"2024-04-20T17:19:37.699094+0000","last_peered":"2024-04-20T17:19:37.699094+0000","last_clean":"2024-04-20T17:19:37.699094+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:19:37.699094+0000","last_undegraded":"2024-04-20T17:19:37.699094+0000","last_fullsized":"2024-04-20T17:19:37.699094+0000","mapping_epoch":115,"log_start":"142'8600","ondisk_log_start":"142'8600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'17455","last_scrub_stamp":"2024-04-20T17:18:45.702769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:45.702769+0000","objects_scrubbed":2599,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:54.717372+0000","scrub_duration":222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91226112,"num_objects":1392,"num_object_clones":0,"num_object_copies":2784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18660,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"340'18891","reported_seq":34363,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-04-20T17:19:37.703411+0000","last_change":"2024-04-20T17:17:57.501217+0000","last_active":"2024-04-20T17:19:37.703411+0000","last_peered":"2024-04-20T17:19:37.703411+0000","last_clean":"2024-04-20T17:19:37.703411+0000","last_became_active":"2024-04-20T17:10:55.931751+0000","last_became_peered":"2024-04-20T17:10:55.931751+0000","last_unstale":"2024-04-20T17:19:37.703411+0000","last_undegraded":"2024-04-20T17:19:37.703411+0000","last_fullsized":"2024-04-20T17:19:37.703411+0000","mapping_epoch":173,"log_start":"143'8800","ondisk_log_start":"143'8800","created":115,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'16916","last_scrub_stamp":"2024-04-20T17:17:57.501089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:17:57.501089+0000","objects_scrubbed":3350,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":318,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89849856,"num_objects":1371,"num_object_clones":0,"num_object_copies":2742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18891,"num_write_kb":648384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7296,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"340'18933","reported_seq":19861,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-04-20T17:19:37.569180+0000","last_change":"2024-04-20T17:18:06.314140+0000","last_active":"2024-04-20T17:19:37.569180+0000","last_peered":"2024-04-20T17:19:37.569180+0000","last_clean":"2024-04-20T17:19:37.569180+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:19:37.569180+0000","last_undegraded":"2024-04-20T17:19:37.569180+0000","last_fullsized":"2024-04-20T17:19:37.569180+0000","mapping_epoch":115,"log_start":"143'8900","ondisk_log_start":"143'8900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"314'17127","last_scrub_stamp":"2024-04-20T17:18:06.313989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:06.313989+0000","objects_scrubbed":3191,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90636288,"num_objects":1383,"num_object_clones":0,"num_object_copies":2766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18933,"num_write_kb":650112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"340'18471","reported_seq":19388,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-04-20T17:19:38.508696+0000","last_change":"2024-04-20T17:18:47.241194+0000","last_active":"2024-04-20T17:19:38.508696+0000","last_peered":"2024-04-20T17:19:38.508696+0000","last_clean":"2024-04-20T17:19:38.508696+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:19:38.508696+0000","last_undegraded":"2024-04-20T17:19:38.508696+0000","last_fullsized":"2024-04-20T17:19:38.508696+0000","mapping_epoch":115,"log_start":"142'8400","ondisk_log_start":"142'8400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'17283","last_scrub_stamp":"2024-04-20T17:18:47.241064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:47.241064+0000","objects_scrubbed":2584,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:10.974140+0000","scrub_duration":266,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91291648,"num_objects":1393,"num_object_clones":0,"num_object_copies":2786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18471,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"340'18670","reported_seq":19588,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-04-20T17:19:37.554410+0000","last_change":"2024-04-20T17:18:48.599315+0000","last_active":"2024-04-20T17:19:37.554410+0000","last_peered":"2024-04-20T17:19:37.554410+0000","last_clean":"2024-04-20T17:19:37.554410+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:19:37.554410+0000","last_undegraded":"2024-04-20T17:19:37.554410+0000","last_fullsized":"2024-04-20T17:19:37.554410+0000","mapping_epoch":115,"log_start":"142'8600","ondisk_log_start":"142'8600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'17566","last_scrub_stamp":"2024-04-20T17:18:48.599209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:48.599209+0000","objects_scrubbed":2538,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:05.548836+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93847552,"num_objects":1432,"num_object_clones":0,"num_object_copies":2864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18670,"num_write_kb":643264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"340'18613","reported_seq":19530,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-04-20T17:19:40.747584+0000","last_change":"2024-04-20T17:18:52.214316+0000","last_active":"2024-04-20T17:19:40.747584+0000","last_peered":"2024-04-20T17:19:40.747584+0000","last_clean":"2024-04-20T17:19:40.747584+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:19:40.747584+0000","last_undegraded":"2024-04-20T17:19:40.747584+0000","last_fullsized":"2024-04-20T17:19:40.747584+0000","mapping_epoch":115,"log_start":"143'8600","ondisk_log_start":"143'8600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'17470","last_scrub_stamp":"2024-04-20T17:18:52.214208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:52.214208+0000","objects_scrubbed":2403,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:55.920246+0000","scrub_duration":228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82313240,"num_objects":1257,"num_object_clones":0,"num_object_copies":2514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1257,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18613,"num_write_kb":635777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"340'18693","reported_seq":19489,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-04-20T17:19:40.860730+0000","last_change":"2024-04-20T17:18:17.426012+0000","last_active":"2024-04-20T17:19:40.860730+0000","last_peered":"2024-04-20T17:19:40.860730+0000","last_clean":"2024-04-20T17:19:40.860730+0000","last_became_active":"2024-04-20T17:12:23.515375+0000","last_became_peered":"2024-04-20T17:12:23.515375+0000","last_unstale":"2024-04-20T17:19:40.860730+0000","last_undegraded":"2024-04-20T17:19:40.860730+0000","last_fullsized":"2024-04-20T17:19:40.860730+0000","mapping_epoch":202,"log_start":"142'8600","ondisk_log_start":"142'8600","created":115,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'16928","last_scrub_stamp":"2024-04-20T17:18:17.425878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:17.425878+0000","objects_scrubbed":3071,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:47.218525+0000","scrub_duration":353,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85393408,"num_objects":1303,"num_object_clones":0,"num_object_copies":2606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18693,"num_write_kb":639872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":1420623896,"num_objects":21678,"num_object_clones":0,"num_object_copies":43356,"num_objects_missing_on_primary":883,"num_objects_missing":883,"num_objects_degraded":0,"num_objects_misplaced":1772,"num_objects_unfound":0,"num_objects_dirty":21678,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":300306,"num_write_kb":10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28327,"num_bytes_recovered":1856438272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3770294272,"data_stored":3770286128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},"log_size":160805,"ondisk_log_size":160805,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991399,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":348140,"kb_used_data":276704,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93397012,"statfs":{"total":95995035648,"available":95638540288,"internally_reserved":0,"allocated":283344896,"data_stored":282120430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":73136580},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253037,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":459808,"kb_used_data":288408,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93285344,"statfs":{"total":95995035648,"available":95524192256,"internally_reserved":0,"allocated":295329792,"data_stored":294108878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":175503808},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612787,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":489444,"kb_used_data":341660,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93255708,"statfs":{"total":95995035648,"available":95493844992,"internally_reserved":0,"allocated":349859840,"data_stored":348634874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":151321024},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809016,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":459232,"kb_used_data":355864,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93285920,"statfs":{"total":95995035648,"available":95524782080,"internally_reserved":0,"allocated":364404736,"data_stored":363183842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168765,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":381936,"kb_used_data":274400,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93363216,"statfs":{"total":95995035648,"available":95603933184,"internally_reserved":0,"allocated":280985600,"data_stored":279761154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430401,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":290216,"kb_used_data":191000,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93454936,"statfs":{"total":95995035648,"available":95697854464,"internally_reserved":0,"allocated":195584000,"data_stored":194363106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101579206},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575068,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":276576,"kb_used_data":169048,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93468576,"statfs":{"total":95995035648,"available":95711821824,"internally_reserved":0,"allocated":173105152,"data_stored":171884258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738592,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":435488,"kb_used_data":296024,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93309664,"statfs":{"total":95995035648,"available":95549095936,"internally_reserved":0,"allocated":303128576,"data_stored":301907682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,0,2,1,3,2,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393881,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":560480,"kb_used_data":516248,"kb_used_omap":202,"kb_used_meta":44021,"kb_avail":93184672,"statfs":{"total":95995035648,"available":95421104128,"internally_reserved":0,"allocated":528637952,"data_stored":527417058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":45077975},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182609,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":245216,"kb_used_data":169624,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93499936,"statfs":{"total":95995035648,"available":95743934464,"internally_reserved":0,"allocated":173694976,"data_stored":172474082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986380,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":120612,"kb_used_data":85276,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93624540,"statfs":{"total":95995035648,"available":95871528960,"internally_reserved":0,"allocated":87322624,"data_stored":86097658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790151,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":887904,"kb_used_data":780376,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92857248,"statfs":{"total":95995035648,"available":95085821952,"internally_reserved":0,"allocated":799105024,"data_stored":797884130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":297140224,"data_stored":297140224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":169082880,"data_stored":169082880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":525598720,"data_stored":525598720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":169279488,"data_stored":169279488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":82972672,"data_stored":82968600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":794951680,"data_stored":794951680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":188481536,"data_stored":188481536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":272826368,"data_stored":272826368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":358612992,"data_stored":358612992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":340660224,"data_stored":340656152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":292683776,"data_stored":292683776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":278003712,"data_stored":278003712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:19:44.078 INFO:tasks.cephadm.ceph_manager.ceph:PG 2.3 is not active+clean 2024-04-20T17:19:44.078 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '2.3', 'version': "340'18411", 'reported_seq': 44508, 'reported_epoch': 340, 'state': 'active+recovering', 'last_fresh': '2024-04-20T17:19:38.978636+0000', 'last_change': '2024-04-20T17:12:26.166109+0000', 'last_active': '2024-04-20T17:19:38.978636+0000', 'last_peered': '2024-04-20T17:19:38.978636+0000', 'last_clean': '2024-04-20T17:09:35.882603+0000', 'last_became_active': '2024-04-20T17:10:54.470253+0000', 'last_became_peered': '2024-04-20T17:10:54.470253+0000', 'last_unstale': '2024-04-20T17:19:38.978636+0000', 'last_undegraded': '2024-04-20T17:19:38.978636+0000', 'last_fullsized': '2024-04-20T17:19:38.978636+0000', 'mapping_epoch': 171, 'log_start': "142'8400", 'ondisk_log_start': "142'8400", 'created': 115, 'last_epoch_clean': 116, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-20T17:08:04.990639+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T17:08:04.990639+0000', 'last_clean_scrub_stamp': '2024-04-20T17:08:04.990639+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 86441984, 'num_objects': 1319, 'num_object_clones': 0, 'num_object_copies': 2638, 'num_objects_missing_on_primary': 883, 'num_objects_missing': 883, 'num_objects_degraded': 0, 'num_objects_misplaced': 1772, 'num_objects_unfound': 0, 'num_objects_dirty': 1319, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18411, 'num_write_kb': 631360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8543, 'num_bytes_recovered': 559874048, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 10], 'acting': [0, 10], 'avail_no_missing': ['2', '5'], 'object_location_counts': [{'shards': '0,2,5', 'objects': 6}, {'shards': '0,10', 'objects': 430}, {'shards': '2,5', 'objects': 883}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-20T17:19:44.078 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph status --format=json 2024-04-20T17:19:44.149 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:19:44.194 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:44.396 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:44 smithi106 ceph-mon[44423]: 2.0 scrub starts 2024-04-20T17:19:44.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:44 smithi005 ceph-mon[39562]: 2.0 scrub starts 2024-04-20T17:19:44.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:44 smithi171 ceph-mon[44189]: 2.0 scrub starts 2024-04-20T17:19:45.037 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:19:45.037 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:19:45.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:45 smithi005 ceph-mon[39562]: 2.0 scrub ok 2024-04-20T17:19:45.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:45 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:19:45.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:45 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:19:45.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:45 smithi005 ceph-mon[39562]: pgmap v939: 17 pgs: 1 active+recovering, 16 active+clean; 1.2 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 575 op/s; 1435/39932 objects misplaced (3.594%); 2.2 MiB/s, 35 objects/s recovering 2024-04-20T17:19:45.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:45 smithi005 ceph-mon[39562]: osdmap e341: 12 total, 12 up, 12 in 2024-04-20T17:19:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:45 smithi106 ceph-mon[44423]: 2.0 scrub ok 2024-04-20T17:19:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:45 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:19:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:45 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:19:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:45 smithi106 ceph-mon[44423]: pgmap v939: 17 pgs: 1 active+recovering, 16 active+clean; 1.2 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 575 op/s; 1435/39932 objects misplaced (3.594%); 2.2 MiB/s, 35 objects/s recovering 2024-04-20T17:19:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:45 smithi106 ceph-mon[44423]: osdmap e341: 12 total, 12 up, 12 in 2024-04-20T17:19:45.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:45 smithi171 ceph-mon[44189]: 2.0 scrub ok 2024-04-20T17:19:45.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:45 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:19:45.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:45 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:19:45.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:45 smithi171 ceph-mon[44189]: pgmap v939: 17 pgs: 1 active+recovering, 16 active+clean; 1.2 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 575 op/s; 1435/39932 objects misplaced (3.594%); 2.2 MiB/s, 35 objects/s recovering 2024-04-20T17:19:45.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:45 smithi171 ceph-mon[44189]: osdmap e341: 12 total, 12 up, 12 in 2024-04-20T17:19:45.916 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-20T17:19:46.161 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:46 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:19:46.221 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:19:46.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:46 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:19:46.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:46 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:19:46.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:46 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4027547319' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:19:46.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:46 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:19:46.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:46 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:19:46.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:46 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4027547319' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:19:46.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:46 smithi005 ceph-mon[39562]: osdmap e342: 12 total, 12 up, 12 in 2024-04-20T17:19:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:46 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:19:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:46 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4027547319' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:19:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:46 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:19:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:46 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:19:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:46 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4027547319' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:19:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:46 smithi106 ceph-mon[44423]: osdmap e342: 12 total, 12 up, 12 in 2024-04-20T17:19:46.551 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:19:46.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:46.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:19:46.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:46 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:19:46.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:46 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:19:46.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:46 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4027547319' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:19:46.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:46 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:19:46.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:46 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:19:46.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:46 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4027547319' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:19:46.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:46 smithi171 ceph-mon[44189]: osdmap e342: 12 total, 12 up, 12 in 2024-04-20T17:19:46.656 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:19:46.706 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:46.716 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:19:46.738 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:19:46.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:46.742 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:19:46.843 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:19:46.878 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:46.938 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:19:46.939 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:19:46.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:46.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:19:47.051 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:19:47.082 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:47.137 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:19:47.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:47.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:19:47.161 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:47 smithi005 ceph-mon[39562]: pgmap v942: 17 pgs: 1 active+recovering, 16 active+clean; 1.2 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 542 op/s; 1435/37904 objects misplaced (3.786%); 2.5 MiB/s, 39 objects/s recovering 2024-04-20T17:19:47.243 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:19:47.278 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:47.304 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:19:47.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:47.307 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:19:47.408 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:19:47.435 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:47 smithi106 ceph-mon[44423]: pgmap v942: 17 pgs: 1 active+recovering, 16 active+clean; 1.2 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 542 op/s; 1435/37904 objects misplaced (3.786%); 2.5 MiB/s, 39 objects/s recovering 2024-04-20T17:19:47.438 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:47.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:47 smithi171 ceph-mon[44189]: pgmap v942: 17 pgs: 1 active+recovering, 16 active+clean; 1.2 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 542 op/s; 1435/37904 objects misplaced (3.786%); 2.5 MiB/s, 39 objects/s recovering 2024-04-20T17:19:47.638 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:19:47.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:47.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:19:47.744 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:19:47.777 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:47.801 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:19:47.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:47.804 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:19:47.905 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:19:47.934 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:47.955 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:19:47.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:47.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:19:48.060 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:19:48.101 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:48.119 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:19:48.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:48.121 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:19:48.222 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:19:48.223 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:19:48.223 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:19:48.264 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:48.360 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:19:48.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:48.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:19:48.465 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:19:48.500 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:48.524 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:19:48.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:48.527 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:19:48.628 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:19:48.657 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:48.973 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-20T17:19:49.005 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:19:49.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:49.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:19:49.108 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:19:49.142 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:49.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:49 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:19:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:49 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:19:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:49 smithi106 ceph-mon[44423]: 2.e scrub starts 2024-04-20T17:19:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:49 smithi106 ceph-mon[44423]: 2.e scrub ok 2024-04-20T17:19:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:49 smithi106 ceph-mon[44423]: pgmap v943: 17 pgs: 1 active+recovering, 16 active+clean; 1.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 654 op/s; 1435/34950 objects misplaced (4.106%); 1.3 MiB/s, 20 objects/s recovering 2024-04-20T17:19:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:49 smithi106 ceph-mon[44423]: osdmap e343: 12 total, 12 up, 12 in 2024-04-20T17:19:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:49 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2929607620' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:19:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:49.302 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:19:49.431 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:19:49.431 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:19:49.432 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:19:49.432 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:19:49.432 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:19:49.432 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:19:49.432 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:19:49.432 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:19:49.432 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:19:49.432 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:19:49.432 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:19:49.432 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:19:49.432 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:19:49.432 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:19:49.433 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:19:49.433 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:19:49.433 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:19:49.433 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:19:49.433 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:19:49.433 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:19:49.433 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:19:49.433 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:19:49.433 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:19:49.433 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:19:49.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:49 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:19:49.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:49 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:19:49.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:49 smithi005 ceph-mon[39562]: 2.e scrub starts 2024-04-20T17:19:49.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:49 smithi005 ceph-mon[39562]: 2.e scrub ok 2024-04-20T17:19:49.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:49 smithi005 ceph-mon[39562]: pgmap v943: 17 pgs: 1 active+recovering, 16 active+clean; 1.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 654 op/s; 1435/34950 objects misplaced (4.106%); 1.3 MiB/s, 20 objects/s recovering 2024-04-20T17:19:49.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:49 smithi005 ceph-mon[39562]: osdmap e343: 12 total, 12 up, 12 in 2024-04-20T17:19:49.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:49 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2929607620' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:19:49.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:49.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:49.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:49.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:49.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:49.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:49 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:19:49.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:49 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:19:49.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:49 smithi171 ceph-mon[44189]: 2.e scrub starts 2024-04-20T17:19:49.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:49 smithi171 ceph-mon[44189]: 2.e scrub ok 2024-04-20T17:19:49.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:49 smithi171 ceph-mon[44189]: pgmap v943: 17 pgs: 1 active+recovering, 16 active+clean; 1.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 654 op/s; 1435/34950 objects misplaced (4.106%); 1.3 MiB/s, 20 objects/s recovering 2024-04-20T17:19:49.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:49 smithi171 ceph-mon[44189]: osdmap e343: 12 total, 12 up, 12 in 2024-04-20T17:19:49.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:49 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2929607620' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:19:49.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:49.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:49.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:49.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:49.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:49.694 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:19:49.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:49.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:19:49.785 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:19:49.798 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:19:49.829 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:49.856 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:19:49.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:49.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:19:49.960 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:19:49.993 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:50.249 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:19:50.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:50.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:19:50.353 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:19:50.384 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:50.406 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:19:50.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:50.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:19:50.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:50 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:19:50.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:50 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:19:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:50 smithi005 ceph-mon[39562]: osdmap e344: 12 total, 12 up, 12 in 2024-04-20T17:19:50.510 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:19:50.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:50 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:19:50.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:50 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:19:50.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:50.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:50.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:50.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:50.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:50.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:50.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:50.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:50 smithi106 ceph-mon[44423]: osdmap e344: 12 total, 12 up, 12 in 2024-04-20T17:19:50.541 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:50.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:50 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:19:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:50 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:19:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:19:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:50 smithi171 ceph-mon[44189]: osdmap e344: 12 total, 12 up, 12 in 2024-04-20T17:19:51.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:51 smithi005 ceph-mon[39562]: pgmap v946: 17 pgs: 1 active+recovering, 16 active+clean; 1.0 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 565 op/s; 1041/33234 objects misplaced (3.132%); 2.1 MiB/s, 33 objects/s recovering 2024-04-20T17:19:51.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:51 smithi106 ceph-mon[44423]: pgmap v946: 17 pgs: 1 active+recovering, 16 active+clean; 1.0 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 565 op/s; 1041/33234 objects misplaced (3.132%); 2.1 MiB/s, 33 objects/s recovering 2024-04-20T17:19:51.592 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:51 smithi171 ceph-mon[44189]: pgmap v946: 17 pgs: 1 active+recovering, 16 active+clean; 1.0 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 565 op/s; 1041/33234 objects misplaced (3.132%); 2.1 MiB/s, 33 objects/s recovering 2024-04-20T17:19:51.708 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:19:52.024 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:19:52.024 INFO:teuthology.orchestra.run.smithi005.stdout:{"fsid":"13c6e76c-ff37-11ee-bc93-c7b262605968","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1160,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":344,"num_osds":12,"num_up_osds":12,"osd_up_since":1713633141,"num_in_osds":12,"osd_in_since":1713632742,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovering","count":1}],"num_pgs":17,"num_pools":2,"num_objects":16617,"data_bytes":1089405496,"bytes_used":4560048128,"bytes_avail":1147380379648,"bytes_total":1151940427776,"misplaced_objects":1041,"misplaced_total":33234,"misplaced_ratio":0.031323343563820182,"recovering_objects_per_sec":33,"recovering_bytes_per_sec":2178548,"recovering_keys_per_sec":0,"num_objects_recovered":197,"num_bytes_recovered":12910592,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":565},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.5:8443/"}},"servicemap":{"epoch":28,"modified":"2024-04-20T17:19:48.085806+0000","services":{"osd":{"daemons":{"summary":"","5":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"7":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"8":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-20T17:19:52.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:52 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1298748325' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:19:52.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:52 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1298748325' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:19:52.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:52 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1298748325' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:19:52.692 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:19:52.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:52.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:19:52.798 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:19:52.836 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:52.862 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:19:52.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:52.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:19:52.969 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:19:52.998 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:53.100 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:19:53.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:53.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:19:53.185 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:53 smithi106 ceph-mon[44423]: pgmap v947: 17 pgs: 1 active+recovering, 16 active+clean; 946 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 627 op/s; 1041/30262 objects misplaced (3.440%); 2.0 MiB/s, 32 objects/s recovering 2024-04-20T17:19:53.186 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:53 smithi005 ceph-mon[39562]: pgmap v947: 17 pgs: 1 active+recovering, 16 active+clean; 946 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 627 op/s; 1041/30262 objects misplaced (3.440%); 2.0 MiB/s, 32 objects/s recovering 2024-04-20T17:19:53.204 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:19:53.238 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:53.259 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:19:53.259 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-04-20T17:19:53.259 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-04-20T17:19:53.260 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:19:53.260 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.260 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:194018 2.4 2:27a9c215:::benchmark_data_smithi171_78918_object33024:head [delete] snapc 0=[] ondisk+write+known_if_redirected e163)", 2024-04-20T17:19:53.260 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:28.864443+0000", 2024-04-20T17:19:53.260 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 564.39234837200001, 2024-04-20T17:19:53.260 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.062669097000000007, 2024-04-20T17:19:53.260 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:19:53.260 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:19:53.260 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:53.260 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:19:53.260 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:19:53.260 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:53.260 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 194018 2024-04-20T17:19:53.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.261 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:19:53.261 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.261 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:19:53.261 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864443+0000", 2024-04-20T17:19:53.261 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.261 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.261 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:19:53.261 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864443+0000", 2024-04-20T17:19:53.261 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.261 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.261 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:19:53.262 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864443+0000", 2024-04-20T17:19:53.262 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6899999999999998e-07 2024-04-20T17:19:53.262 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.262 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:19:53.262 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864445+0000", 2024-04-20T17:19:53.262 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.632e-06 2024-04-20T17:19:53.262 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.262 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:19:53.262 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864448+0000", 2024-04-20T17:19:53.262 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1300000000000001e-06 2024-04-20T17:19:53.262 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.263 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:19:53.263 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864452+0000", 2024-04-20T17:19:53.263 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1649999999999999e-06 2024-04-20T17:19:53.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.263 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.263 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864569+0000", 2024-04-20T17:19:53.263 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000117321 2024-04-20T17:19:53.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.263 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:19:53.263 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864572+0000", 2024-04-20T17:19:53.263 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1209999999999998e-06 2024-04-20T17:19:53.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.264 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.264 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.893144+0000", 2024-04-20T17:19:53.264 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.028571745999999999 2024-04-20T17:19:53.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.264 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:19:53.264 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.893260+0000", 2024-04-20T17:19:53.264 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000115869 2024-04-20T17:19:53.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.264 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:19:53.264 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.893380+0000", 2024-04-20T17:19:53.265 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000119837 2024-04-20T17:19:53.265 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.265 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.265 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:19:53.265 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.904596+0000", 2024-04-20T17:19:53.265 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011216271 2024-04-20T17:19:53.265 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.265 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.265 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:53.265 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.926863+0000", 2024-04-20T17:19:53.265 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.022266828999999998 2024-04-20T17:19:53.265 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.265 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.265 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:19:53.266 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.927080+0000", 2024-04-20T17:19:53.266 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00021687 2024-04-20T17:19:53.266 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.266 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.266 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:19:53.266 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.927112+0000", 2024-04-20T17:19:53.266 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1937000000000003e-05 2024-04-20T17:19:53.266 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.266 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:19:53.266 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.266 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.266 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.266 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:194042 2.4 2:22fc9479:::benchmark_data_smithi171_78918_object33048:head [delete] snapc 0=[] ondisk+write+known_if_redirected e163)", 2024-04-20T17:19:53.266 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:28.864458+0000", 2024-04-20T17:19:53.267 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 564.39233339299994, 2024-04-20T17:19:53.267 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.062804993000000003, 2024-04-20T17:19:53.267 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:19:53.267 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:19:53.267 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:53.267 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:19:53.267 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:19:53.267 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:53.267 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 194042 2024-04-20T17:19:53.267 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.267 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:19:53.267 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.267 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:19:53.267 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864458+0000", 2024-04-20T17:19:53.267 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.268 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.268 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.268 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:19:53.268 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864458+0000", 2024-04-20T17:19:53.268 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.268 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.268 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.268 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:19:53.268 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864458+0000", 2024-04-20T17:19:53.268 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5200000000000002e-07 2024-04-20T17:19:53.268 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.268 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.268 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:19:53.268 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864460+0000", 2024-04-20T17:19:53.268 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0880000000000002e-06 2024-04-20T17:19:53.269 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.269 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.269 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:19:53.269 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864465+0000", 2024-04-20T17:19:53.269 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5480000000000003e-06 2024-04-20T17:19:53.269 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.269 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.269 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:19:53.269 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864469+0000", 2024-04-20T17:19:53.269 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6140000000000002e-06 2024-04-20T17:19:53.269 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.269 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.269 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.269 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864602+0000", 2024-04-20T17:19:53.269 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000132342 2024-04-20T17:19:53.270 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.270 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.270 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:19:53.270 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864608+0000", 2024-04-20T17:19:53.270 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3459999999999999e-06 2024-04-20T17:19:53.270 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.270 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.270 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.270 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.895141+0000", 2024-04-20T17:19:53.270 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.030533385 2024-04-20T17:19:53.270 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.270 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.270 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:19:53.270 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.895235+0000", 2024-04-20T17:19:53.270 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3633000000000003e-05 2024-04-20T17:19:53.271 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.271 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.271 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:19:53.271 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.895303+0000", 2024-04-20T17:19:53.271 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7793000000000006e-05 2024-04-20T17:19:53.271 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.271 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.271 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:19:53.271 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.904603+0000", 2024-04-20T17:19:53.271 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0093000429999999992 2024-04-20T17:19:53.271 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.271 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.271 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:53.271 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.927163+0000", 2024-04-20T17:19:53.271 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.022559795000000001 2024-04-20T17:19:53.272 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.272 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.272 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:19:53.272 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.927218+0000", 2024-04-20T17:19:53.272 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5537e-05 2024-04-20T17:19:53.272 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.272 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.272 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:19:53.272 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.927263+0000", 2024-04-20T17:19:53.272 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4416999999999998e-05 2024-04-20T17:19:53.272 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.272 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:19:53.272 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.272 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.272 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.272 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:194094 2.4 2:2b689924:::benchmark_data_smithi171_78918_object33100:head [delete] snapc 0=[] ondisk+write+known_if_redirected e163)", 2024-04-20T17:19:53.273 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:28.864480+0000", 2024-04-20T17:19:53.273 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 564.39231129200004, 2024-04-20T17:19:53.273 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.067369730000000003, 2024-04-20T17:19:53.273 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:19:53.273 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:19:53.273 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:53.273 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:19:53.273 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:19:53.273 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:53.273 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 194094 2024-04-20T17:19:53.273 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.274 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:19:53.274 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.274 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:19:53.274 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864480+0000", 2024-04-20T17:19:53.274 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.274 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.274 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.274 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:19:53.274 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864480+0000", 2024-04-20T17:19:53.274 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.274 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.274 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.274 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:19:53.274 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864480+0000", 2024-04-20T17:19:53.275 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6499999999999999e-07 2024-04-20T17:19:53.275 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.275 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:19:53.275 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864483+0000", 2024-04-20T17:19:53.275 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.306e-06 2024-04-20T17:19:53.275 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.275 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:19:53.275 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864487+0000", 2024-04-20T17:19:53.275 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7149999999999997e-06 2024-04-20T17:19:53.275 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.276 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:19:53.276 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864493+0000", 2024-04-20T17:19:53.276 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5090000000000001e-06 2024-04-20T17:19:53.276 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.276 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.276 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.276 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864631+0000", 2024-04-20T17:19:53.276 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013836400000000001 2024-04-20T17:19:53.276 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.276 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.276 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:19:53.276 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864634+0000", 2024-04-20T17:19:53.276 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1829999999999998e-06 2024-04-20T17:19:53.277 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.277 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.277 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.277 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.897021+0000", 2024-04-20T17:19:53.277 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.032386854999999999 2024-04-20T17:19:53.277 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.277 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.277 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:19:53.277 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.897088+0000", 2024-04-20T17:19:53.277 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6322000000000006e-05 2024-04-20T17:19:53.277 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.277 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.277 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:19:53.277 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.897152+0000", 2024-04-20T17:19:53.278 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4330000000000002e-05 2024-04-20T17:19:53.278 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.278 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.278 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:19:53.278 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.908260+0000", 2024-04-20T17:19:53.278 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011107911999999999 2024-04-20T17:19:53.278 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.278 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.278 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:53.278 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.931751+0000", 2024-04-20T17:19:53.278 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.023491452999999999 2024-04-20T17:19:53.279 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.279 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.279 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:19:53.279 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.931806+0000", 2024-04-20T17:19:53.279 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4557e-05 2024-04-20T17:19:53.279 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.279 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.279 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:19:53.279 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.931849+0000", 2024-04-20T17:19:53.279 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3559000000000001e-05 2024-04-20T17:19:53.279 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.279 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:19:53.279 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.280 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.280 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:194109 2.4 2:221bd37d:::benchmark_data_smithi171_78918_object33115:head [delete] snapc 0=[] ondisk+write+known_if_redirected e163)", 2024-04-20T17:19:53.280 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:28.864504+0000", 2024-04-20T17:19:53.280 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 564.392286883, 2024-04-20T17:19:53.280 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.067444737000000005, 2024-04-20T17:19:53.280 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:19:53.280 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:19:53.280 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:53.280 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:19:53.280 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:19:53.280 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:53.280 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 194109 2024-04-20T17:19:53.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.281 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:19:53.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.281 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:19:53.281 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864504+0000", 2024-04-20T17:19:53.281 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.281 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.281 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:19:53.281 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864504+0000", 2024-04-20T17:19:53.281 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.281 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.281 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:19:53.282 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864504+0000", 2024-04-20T17:19:53.282 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4700000000000002e-07 2024-04-20T17:19:53.282 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.282 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.282 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:19:53.282 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864506+0000", 2024-04-20T17:19:53.282 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6249999999999999e-06 2024-04-20T17:19:53.282 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.282 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.282 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:19:53.282 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864510+0000", 2024-04-20T17:19:53.282 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7440000000000001e-06 2024-04-20T17:19:53.282 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.283 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.283 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:19:53.283 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864513+0000", 2024-04-20T17:19:53.283 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2799999999999999e-06 2024-04-20T17:19:53.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.283 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.283 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.283 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864659+0000", 2024-04-20T17:19:53.283 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014595999999999999 2024-04-20T17:19:53.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.283 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.283 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:19:53.283 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864663+0000", 2024-04-20T17:19:53.283 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6109999999999998e-06 2024-04-20T17:19:53.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.284 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.284 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.284 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.899017+0000", 2024-04-20T17:19:53.284 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.034354572 2024-04-20T17:19:53.284 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.284 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.284 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:19:53.284 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.899105+0000", 2024-04-20T17:19:53.284 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7593e-05 2024-04-20T17:19:53.284 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.284 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.284 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:19:53.284 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.899199+0000", 2024-04-20T17:19:53.284 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3713000000000002e-05 2024-04-20T17:19:53.284 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.285 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.285 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:19:53.285 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.908284+0000", 2024-04-20T17:19:53.285 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0090854170000000001 2024-04-20T17:19:53.285 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.285 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.285 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:53.285 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.931883+0000", 2024-04-20T17:19:53.285 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.023599196999999999 2024-04-20T17:19:53.285 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.285 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.285 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:19:53.285 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.931912+0000", 2024-04-20T17:19:53.285 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8374e-05 2024-04-20T17:19:53.285 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.286 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.286 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:19:53.286 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.931949+0000", 2024-04-20T17:19:53.286 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7304000000000001e-05 2024-04-20T17:19:53.286 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.286 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:19:53.286 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.286 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.286 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.286 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:194115 2.4 2:2492cd19:::benchmark_data_smithi171_78918_object33121:head [delete] snapc 0=[] ondisk+write+known_if_redirected e163)", 2024-04-20T17:19:53.286 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:28.864518+0000", 2024-04-20T17:19:53.286 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 564.39227255399999, 2024-04-20T17:19:53.286 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.072661744, 2024-04-20T17:19:53.286 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:19:53.286 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:19:53.287 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:53.287 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:19:53.287 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:19:53.287 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:53.287 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 194115 2024-04-20T17:19:53.287 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.287 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:19:53.287 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.287 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:19:53.287 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864518+0000", 2024-04-20T17:19:53.287 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.287 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.287 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.288 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:19:53.288 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864518+0000", 2024-04-20T17:19:53.288 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.288 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.288 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.288 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:19:53.288 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864519+0000", 2024-04-20T17:19:53.288 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.51e-07 2024-04-20T17:19:53.288 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.288 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.288 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:19:53.288 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864521+0000", 2024-04-20T17:19:53.288 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9379999999999999e-06 2024-04-20T17:19:53.288 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.288 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.289 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:19:53.289 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864525+0000", 2024-04-20T17:19:53.289 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5909999999999998e-06 2024-04-20T17:19:53.289 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.289 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.289 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:19:53.289 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864530+0000", 2024-04-20T17:19:53.289 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8969999999999999e-06 2024-04-20T17:19:53.289 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.289 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.289 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.289 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864681+0000", 2024-04-20T17:19:53.289 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015036199999999999 2024-04-20T17:19:53.290 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.290 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.290 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:19:53.290 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864684+0000", 2024-04-20T17:19:53.290 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1669999999999999e-06 2024-04-20T17:19:53.290 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.290 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.290 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.290 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.900866+0000", 2024-04-20T17:19:53.290 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.036182178000000002 2024-04-20T17:19:53.290 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.290 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.290 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:19:53.290 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.900959+0000", 2024-04-20T17:19:53.291 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3410000000000002e-05 2024-04-20T17:19:53.291 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.291 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.291 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:19:53.291 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.901038+0000", 2024-04-20T17:19:53.291 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9041000000000004e-05 2024-04-20T17:19:53.291 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.291 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.291 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:19:53.291 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.908290+0000", 2024-04-20T17:19:53.291 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0072511820000000001 2024-04-20T17:19:53.291 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.291 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.291 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:53.292 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.937092+0000", 2024-04-20T17:19:53.292 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.028801941000000001 2024-04-20T17:19:53.292 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.292 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.292 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:19:53.292 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.937148+0000", 2024-04-20T17:19:53.292 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6276000000000001e-05 2024-04-20T17:19:53.292 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.292 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.292 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:19:53.292 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.937180+0000", 2024-04-20T17:19:53.292 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2310000000000001e-05 2024-04-20T17:19:53.292 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.292 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:19:53.292 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.293 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.293 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.293 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:194120 2.4 2:24f46cbd:::benchmark_data_smithi171_78918_object33126:head [delete] snapc 0=[] ondisk+write+known_if_redirected e163)", 2024-04-20T17:19:53.293 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:28.864535+0000", 2024-04-20T17:19:53.293 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 564.39225569500002, 2024-04-20T17:19:53.293 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.081574351000000003, 2024-04-20T17:19:53.293 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:19:53.293 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:19:53.293 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:53.293 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:19:53.293 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:19:53.293 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:53.293 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 194120 2024-04-20T17:19:53.293 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.294 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:19:53.294 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.294 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:19:53.294 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864535+0000", 2024-04-20T17:19:53.294 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.294 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.294 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.294 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:19:53.294 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864535+0000", 2024-04-20T17:19:53.294 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.294 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.294 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.294 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:19:53.294 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864536+0000", 2024-04-20T17:19:53.295 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3700000000000001e-07 2024-04-20T17:19:53.295 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.295 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.295 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:19:53.295 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864538+0000", 2024-04-20T17:19:53.295 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.294e-06 2024-04-20T17:19:53.295 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.295 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.295 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:19:53.295 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864542+0000", 2024-04-20T17:19:53.295 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4390000000000002e-06 2024-04-20T17:19:53.295 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.295 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.295 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:19:53.296 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864547+0000", 2024-04-20T17:19:53.296 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7180000000000004e-06 2024-04-20T17:19:53.296 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.296 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.296 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.296 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864703+0000", 2024-04-20T17:19:53.296 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015556899999999999 2024-04-20T17:19:53.296 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.296 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.296 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:19:53.296 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864706+0000", 2024-04-20T17:19:53.296 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1030000000000002e-06 2024-04-20T17:19:53.296 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.296 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.296 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.297 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.902671+0000", 2024-04-20T17:19:53.297 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.037965153000000001 2024-04-20T17:19:53.297 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.297 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.297 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:19:53.297 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.902734+0000", 2024-04-20T17:19:53.297 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2800999999999997e-05 2024-04-20T17:19:53.297 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.297 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.297 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:19:53.297 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.902800+0000", 2024-04-20T17:19:53.297 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5881000000000002e-05 2024-04-20T17:19:53.297 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.298 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.298 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:19:53.298 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.908296+0000", 2024-04-20T17:19:53.298 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0054963160000000002 2024-04-20T17:19:53.298 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.298 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.298 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:53.298 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.946010+0000", 2024-04-20T17:19:53.298 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.037714152000000001 2024-04-20T17:19:53.298 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.298 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.298 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:19:53.298 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.946064+0000", 2024-04-20T17:19:53.298 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3724999999999998e-05 2024-04-20T17:19:53.299 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.299 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.299 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:19:53.299 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.946110+0000", 2024-04-20T17:19:53.299 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5763000000000002e-05 2024-04-20T17:19:53.299 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.299 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:19:53.299 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.299 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.299 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.299 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:194128 2.4 2:25cb3f7f:::benchmark_data_smithi171_78918_object33134:head [delete] snapc 0=[] ondisk+write+known_if_redirected e163)", 2024-04-20T17:19:53.299 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:28.864566+0000", 2024-04-20T17:19:53.299 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 564.39222515200004, 2024-04-20T17:19:53.300 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.083445169999999999, 2024-04-20T17:19:53.300 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:19:53.300 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:19:53.300 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:53.300 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:19:53.300 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:19:53.300 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:53.300 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 194128 2024-04-20T17:19:53.300 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.300 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:19:53.300 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.300 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:19:53.300 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864566+0000", 2024-04-20T17:19:53.300 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.301 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.301 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.301 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:19:53.301 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864566+0000", 2024-04-20T17:19:53.301 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.301 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.301 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.301 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:19:53.301 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864566+0000", 2024-04-20T17:19:53.301 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5499999999999999e-07 2024-04-20T17:19:53.301 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.301 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.301 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:19:53.301 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864568+0000", 2024-04-20T17:19:53.302 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6080000000000001e-06 2024-04-20T17:19:53.302 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.302 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.302 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:19:53.302 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864571+0000", 2024-04-20T17:19:53.302 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0450000000000001e-06 2024-04-20T17:19:53.302 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.302 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.302 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:19:53.302 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864574+0000", 2024-04-20T17:19:53.302 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5470000000000001e-06 2024-04-20T17:19:53.302 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.302 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.303 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.303 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864727+0000", 2024-04-20T17:19:53.303 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015271600000000001 2024-04-20T17:19:53.303 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.303 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.303 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:19:53.303 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864730+0000", 2024-04-20T17:19:53.303 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0750000000000002e-06 2024-04-20T17:19:53.303 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.303 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.303 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.303 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.904621+0000", 2024-04-20T17:19:53.303 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.039891031 2024-04-20T17:19:53.304 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.304 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.304 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:19:53.304 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.904687+0000", 2024-04-20T17:19:53.304 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5309999999999996e-05 2024-04-20T17:19:53.304 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.304 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.304 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:19:53.304 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.904758+0000", 2024-04-20T17:19:53.304 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0996999999999998e-05 2024-04-20T17:19:53.304 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.304 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.304 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:19:53.304 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.912312+0000", 2024-04-20T17:19:53.305 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0075546420000000003 2024-04-20T17:19:53.305 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.305 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.305 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:53.305 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.947961+0000", 2024-04-20T17:19:53.305 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.035649040999999999 2024-04-20T17:19:53.305 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.305 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.305 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:19:53.305 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.947988+0000", 2024-04-20T17:19:53.305 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6741999999999999e-05 2024-04-20T17:19:53.305 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.305 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.305 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:19:53.306 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.948011+0000", 2024-04-20T17:19:53.306 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3061000000000002e-05 2024-04-20T17:19:53.306 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.306 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:19:53.306 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.306 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.306 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.306 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:194135 2.4 2:2e111232:::benchmark_data_smithi171_78918_object33141:head [delete] snapc 0=[] ondisk+write+known_if_redirected e163)", 2024-04-20T17:19:53.306 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:28.864580+0000", 2024-04-20T17:19:53.306 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 564.39221129299995, 2024-04-20T17:19:53.306 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.095628708000000007, 2024-04-20T17:19:53.306 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:19:53.306 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:19:53.307 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:53.307 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:19:53.307 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:19:53.307 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:53.307 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 194135 2024-04-20T17:19:53.307 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.307 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:19:53.307 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.307 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:19:53.307 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864580+0000", 2024-04-20T17:19:53.307 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.307 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.307 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.307 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:19:53.308 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864580+0000", 2024-04-20T17:19:53.308 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.308 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.308 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.308 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:19:53.308 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864580+0000", 2024-04-20T17:19:53.308 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4499999999999997e-07 2024-04-20T17:19:53.308 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.308 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.308 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:19:53.308 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864582+0000", 2024-04-20T17:19:53.308 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.933e-06 2024-04-20T17:19:53.308 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.308 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.309 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:19:53.309 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864591+0000", 2024-04-20T17:19:53.309 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9209999999999998e-06 2024-04-20T17:19:53.309 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.309 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.309 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:19:53.309 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864597+0000", 2024-04-20T17:19:53.309 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8810000000000001e-06 2024-04-20T17:19:53.309 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.309 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.309 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.309 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864749+0000", 2024-04-20T17:19:53.309 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000152072 2024-04-20T17:19:53.309 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.310 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.310 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:19:53.310 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864752+0000", 2024-04-20T17:19:53.310 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0429999999999999e-06 2024-04-20T17:19:53.310 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.310 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.310 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.310 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.906387+0000", 2024-04-20T17:19:53.310 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.041634717000000002 2024-04-20T17:19:53.310 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.310 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.310 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:19:53.310 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.906498+0000", 2024-04-20T17:19:53.310 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000111109 2024-04-20T17:19:53.311 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.311 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.311 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:19:53.311 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.906614+0000", 2024-04-20T17:19:53.311 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000116396 2024-04-20T17:19:53.311 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.311 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.311 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:19:53.311 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.912573+0000", 2024-04-20T17:19:53.311 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0059592669999999999 2024-04-20T17:19:53.311 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.311 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.311 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:53.311 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.960103+0000", 2024-04-20T17:19:53.312 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.047529505 2024-04-20T17:19:53.312 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.312 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.312 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:19:53.312 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.960159+0000", 2024-04-20T17:19:53.312 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6390000000000001e-05 2024-04-20T17:19:53.312 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.312 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.312 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:19:53.312 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.960208+0000", 2024-04-20T17:19:53.312 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9029000000000001e-05 2024-04-20T17:19:53.312 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.312 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:19:53.312 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.313 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.313 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.313 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:194155 2.4 2:2056e495:::benchmark_data_smithi171_78918_object33161:head [delete] snapc 0=[] ondisk+write+known_if_redirected e163)", 2024-04-20T17:19:53.313 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:28.864604+0000", 2024-04-20T17:19:53.313 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 564.39218662300004, 2024-04-20T17:19:53.313 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.099538138999999998, 2024-04-20T17:19:53.313 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:19:53.313 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:19:53.313 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:53.313 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:19:53.313 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:19:53.313 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:53.314 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 194155 2024-04-20T17:19:53.314 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.314 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:19:53.314 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.314 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:19:53.314 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864604+0000", 2024-04-20T17:19:53.314 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.314 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.314 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.314 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:19:53.314 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864604+0000", 2024-04-20T17:19:53.314 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.314 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.314 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.315 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:19:53.315 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864605+0000", 2024-04-20T17:19:53.315 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5499999999999998e-07 2024-04-20T17:19:53.315 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.315 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.315 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:19:53.315 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864607+0000", 2024-04-20T17:19:53.315 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2129999999999998e-06 2024-04-20T17:19:53.315 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.315 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.315 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:19:53.315 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864612+0000", 2024-04-20T17:19:53.315 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6820000000000002e-06 2024-04-20T17:19:53.316 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.316 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.316 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:19:53.316 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864617+0000", 2024-04-20T17:19:53.316 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7890000000000002e-06 2024-04-20T17:19:53.316 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.316 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.316 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.316 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864771+0000", 2024-04-20T17:19:53.316 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015395200000000001 2024-04-20T17:19:53.316 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.316 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.316 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:19:53.316 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864774+0000", 2024-04-20T17:19:53.317 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0350000000000002e-06 2024-04-20T17:19:53.317 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.317 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.317 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.317 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.908334+0000", 2024-04-20T17:19:53.317 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.04356061 2024-04-20T17:19:53.317 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.317 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.317 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:19:53.317 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.908408+0000", 2024-04-20T17:19:53.317 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4017999999999995e-05 2024-04-20T17:19:53.317 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.317 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.317 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:19:53.318 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.908483+0000", 2024-04-20T17:19:53.318 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5022000000000001e-05 2024-04-20T17:19:53.318 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.318 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.318 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:19:53.318 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.912581+0000", 2024-04-20T17:19:53.318 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0040974380000000001 2024-04-20T17:19:53.318 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.318 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.318 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:53.318 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.964041+0000", 2024-04-20T17:19:53.318 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.051460469000000002 2024-04-20T17:19:53.318 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.318 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.319 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:19:53.319 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.964098+0000", 2024-04-20T17:19:53.319 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6424000000000001e-05 2024-04-20T17:19:53.319 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.319 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.319 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:19:53.319 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.964143+0000", 2024-04-20T17:19:53.319 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4932000000000003e-05 2024-04-20T17:19:53.319 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.319 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:19:53.319 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.320 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.320 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.320 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:194181 2.4 2:216b496b:::benchmark_data_smithi171_78918_object33187:head [delete] snapc 0=[] ondisk+write+known_if_redirected e163)", 2024-04-20T17:19:53.320 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:28.864628+0000", 2024-04-20T17:19:53.320 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 564.39216281400002, 2024-04-20T17:19:53.320 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.099622924000000002, 2024-04-20T17:19:53.320 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:19:53.320 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:19:53.320 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:53.320 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:19:53.320 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:19:53.320 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:53.320 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 194181 2024-04-20T17:19:53.320 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.321 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:19:53.321 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.321 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:19:53.321 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864628+0000", 2024-04-20T17:19:53.321 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.321 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.321 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.321 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:19:53.321 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864628+0000", 2024-04-20T17:19:53.321 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.321 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.321 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.321 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:19:53.321 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864629+0000", 2024-04-20T17:19:53.321 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8599999999999999e-07 2024-04-20T17:19:53.322 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.322 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.322 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:19:53.322 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864630+0000", 2024-04-20T17:19:53.322 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.409e-06 2024-04-20T17:19:53.322 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.322 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.322 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:19:53.322 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864633+0000", 2024-04-20T17:19:53.322 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.016e-06 2024-04-20T17:19:53.322 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.322 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.322 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:19:53.323 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864636+0000", 2024-04-20T17:19:53.323 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.495e-06 2024-04-20T17:19:53.323 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.323 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.323 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.323 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864785+0000", 2024-04-20T17:19:53.323 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014829900000000001 2024-04-20T17:19:53.323 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.323 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.323 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:19:53.323 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.864788+0000", 2024-04-20T17:19:53.323 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0120000000000001e-06 2024-04-20T17:19:53.323 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.324 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.324 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.324 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.910423+0000", 2024-04-20T17:19:53.324 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.045635697000000003 2024-04-20T17:19:53.324 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.324 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.324 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:19:53.324 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.910531+0000", 2024-04-20T17:19:53.324 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000107558 2024-04-20T17:19:53.324 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.324 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.324 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:19:53.324 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.910652+0000", 2024-04-20T17:19:53.325 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000120567 2024-04-20T17:19:53.325 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.325 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.325 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:19:53.325 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.914643+0000", 2024-04-20T17:19:53.325 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0039910190000000002 2024-04-20T17:19:53.325 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.325 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.325 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:53.325 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.964187+0000", 2024-04-20T17:19:53.325 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.049544461999999997 2024-04-20T17:19:53.325 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.325 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.325 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:19:53.326 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.964222+0000", 2024-04-20T17:19:53.326 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4786000000000002e-05 2024-04-20T17:19:53.326 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.326 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.326 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:19:53.326 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.964251+0000", 2024-04-20T17:19:53.326 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9218e-05 2024-04-20T17:19:53.326 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.326 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:19:53.326 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.326 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.326 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.326 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:194184 2.4 2:2b9fb9ba:::benchmark_data_smithi171_78918_object33190:head [delete] snapc 0=[] ondisk+write+known_if_redirected e163)", 2024-04-20T17:19:53.326 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:28.901968+0000", 2024-04-20T17:19:53.327 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 564.35482330000002, 2024-04-20T17:19:53.327 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.062420349999999999, 2024-04-20T17:19:53.327 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:19:53.327 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:19:53.327 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:53.327 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:19:53.327 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:19:53.327 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:53.327 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 194184 2024-04-20T17:19:53.327 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.327 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:19:53.328 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.328 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:19:53.328 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.901968+0000", 2024-04-20T17:19:53.328 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.328 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.328 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.328 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:19:53.328 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.901968+0000", 2024-04-20T17:19:53.328 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.328 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.328 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.328 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:19:53.328 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.901970+0000", 2024-04-20T17:19:53.328 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8309999999999999e-06 2024-04-20T17:19:53.328 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.328 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.329 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:19:53.329 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.901975+0000", 2024-04-20T17:19:53.329 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0329999999999999e-06 2024-04-20T17:19:53.329 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.329 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.329 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:19:53.329 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.901985+0000", 2024-04-20T17:19:53.329 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0841e-05 2024-04-20T17:19:53.329 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.329 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.329 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:19:53.329 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.901992+0000", 2024-04-20T17:19:53.329 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4019999999999999e-06 2024-04-20T17:19:53.330 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.330 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.330 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.330 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.912601+0000", 2024-04-20T17:19:53.330 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010609142 2024-04-20T17:19:53.330 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.330 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.330 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:19:53.330 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.912677+0000", 2024-04-20T17:19:53.330 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6341999999999995e-05 2024-04-20T17:19:53.330 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.330 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.331 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:19:53.331 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.912741+0000", 2024-04-20T17:19:53.331 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4208000000000006e-05 2024-04-20T17:19:53.331 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.331 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.331 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:19:53.331 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.916428+0000", 2024-04-20T17:19:53.331 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0036868030000000002 2024-04-20T17:19:53.331 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.331 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.331 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:53.331 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.964297+0000", 2024-04-20T17:19:53.331 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.047868850999999997 2024-04-20T17:19:53.332 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.332 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.332 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:19:53.332 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.964346+0000", 2024-04-20T17:19:53.332 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8999999999999998e-05 2024-04-20T17:19:53.332 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.332 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.332 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:19:53.332 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:28.964388+0000", 2024-04-20T17:19:53.332 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1897000000000002e-05 2024-04-20T17:19:53.332 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.332 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:19:53.332 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.333 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.333 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.333 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:197556 2.2 2:433198d9:::benchmark_data_smithi171_78918_object36562:head [delete] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-04-20T17:19:53.333 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:34.332716+0000", 2024-04-20T17:19:53.333 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 558.92407473799994, 2024-04-20T17:19:53.333 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.068642177999999998, 2024-04-20T17:19:53.333 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:19:53.333 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:19:53.333 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:53.333 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:19:53.333 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:19:53.333 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:53.333 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 197556 2024-04-20T17:19:53.334 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.334 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:19:53.334 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.334 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:19:53.334 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.332716+0000", 2024-04-20T17:19:53.334 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.334 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.334 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.334 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:19:53.334 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.332716+0000", 2024-04-20T17:19:53.334 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.334 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.334 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.334 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:19:53.334 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.332718+0000", 2024-04-20T17:19:53.335 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0169999999999999e-06 2024-04-20T17:19:53.335 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.335 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.335 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:19:53.335 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.332724+0000", 2024-04-20T17:19:53.335 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8000000000000004e-06 2024-04-20T17:19:53.335 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.335 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.335 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:19:53.335 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.332733+0000", 2024-04-20T17:19:53.335 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8829999999999995e-06 2024-04-20T17:19:53.335 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.335 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.336 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:19:53.336 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.332739+0000", 2024-04-20T17:19:53.336 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1510000000000001e-06 2024-04-20T17:19:53.336 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.336 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.336 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.336 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.332784+0000", 2024-04-20T17:19:53.336 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4787999999999999e-05 2024-04-20T17:19:53.336 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.336 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.336 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:19:53.336 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.332862+0000", 2024-04-20T17:19:53.336 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7624000000000001e-05 2024-04-20T17:19:53.336 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.337 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.337 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:19:53.337 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.332927+0000", 2024-04-20T17:19:53.337 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5748999999999997e-05 2024-04-20T17:19:53.337 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.337 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.337 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:19:53.337 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.336407+0000", 2024-04-20T17:19:53.337 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0034794909999999999 2024-04-20T17:19:53.337 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.337 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.337 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:53.337 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.401233+0000", 2024-04-20T17:19:53.338 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.064826243000000006 2024-04-20T17:19:53.338 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.338 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.338 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:19:53.338 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.401296+0000", 2024-04-20T17:19:53.338 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2558000000000007e-05 2024-04-20T17:19:53.338 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.338 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.338 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:19:53.338 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.401358+0000", 2024-04-20T17:19:53.338 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2873999999999995e-05 2024-04-20T17:19:53.338 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.338 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:19:53.339 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.339 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.339 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.339 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:197569 2.2 2:470ccb18:::benchmark_data_smithi171_78918_object36575:head [delete] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-04-20T17:19:53.339 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:34.343803+0000", 2024-04-20T17:19:53.339 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 558.91298844899995, 2024-04-20T17:19:53.339 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.057676193000000001, 2024-04-20T17:19:53.339 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:19:53.339 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:19:53.339 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:53.339 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:19:53.339 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:19:53.339 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:53.340 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 197569 2024-04-20T17:19:53.340 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.340 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:19:53.340 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.340 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:19:53.340 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.343803+0000", 2024-04-20T17:19:53.340 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.340 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.340 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.340 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:19:53.340 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.343803+0000", 2024-04-20T17:19:53.340 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.340 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.341 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.341 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:19:53.341 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.343804+0000", 2024-04-20T17:19:53.341 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8929999999999999e-06 2024-04-20T17:19:53.341 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.341 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.341 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:19:53.341 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.343809+0000", 2024-04-20T17:19:53.341 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7389999999999999e-06 2024-04-20T17:19:53.341 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.341 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.341 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:19:53.341 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.343820+0000", 2024-04-20T17:19:53.341 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0526999999999999e-05 2024-04-20T17:19:53.342 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.342 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.342 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:19:53.342 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.343828+0000", 2024-04-20T17:19:53.342 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3240000000000003e-06 2024-04-20T17:19:53.342 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.342 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.342 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.342 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.343887+0000", 2024-04-20T17:19:53.342 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.944e-05 2024-04-20T17:19:53.342 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.342 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.342 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:19:53.342 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344033+0000", 2024-04-20T17:19:53.342 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014574299999999999 2024-04-20T17:19:53.343 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.343 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.343 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:19:53.343 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344172+0000", 2024-04-20T17:19:53.343 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000139219 2024-04-20T17:19:53.343 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.343 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.343 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:19:53.343 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.354796+0000", 2024-04-20T17:19:53.343 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010623169 2024-04-20T17:19:53.343 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.343 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.343 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:53.343 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.401402+0000", 2024-04-20T17:19:53.343 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.046606389999999998 2024-04-20T17:19:53.344 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.344 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.344 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:19:53.344 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.401432+0000", 2024-04-20T17:19:53.344 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9845e-05 2024-04-20T17:19:53.344 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.344 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.344 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:19:53.344 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.401479+0000", 2024-04-20T17:19:53.344 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6903999999999997e-05 2024-04-20T17:19:53.344 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.344 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:19:53.344 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.344 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.344 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.345 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:197573 2.2 2:4b63adf6:::benchmark_data_smithi171_78918_object36579:head [delete] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-04-20T17:19:53.345 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:34.344613+0000", 2024-04-20T17:19:53.345 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 558.91217807600003, 2024-04-20T17:19:53.345 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.057018594999999998, 2024-04-20T17:19:53.345 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:19:53.345 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:19:53.345 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:53.345 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:19:53.345 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:19:53.345 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:53.345 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 197573 2024-04-20T17:19:53.345 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.345 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:19:53.346 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.346 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:19:53.346 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344613+0000", 2024-04-20T17:19:53.346 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.346 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.346 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.346 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:19:53.346 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344613+0000", 2024-04-20T17:19:53.346 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.346 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.346 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.346 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:19:53.346 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344614+0000", 2024-04-20T17:19:53.346 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4670000000000001e-06 2024-04-20T17:19:53.347 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.347 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.347 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:19:53.347 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344619+0000", 2024-04-20T17:19:53.347 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.933e-06 2024-04-20T17:19:53.347 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.347 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.347 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:19:53.347 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344631+0000", 2024-04-20T17:19:53.347 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2108e-05 2024-04-20T17:19:53.347 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.347 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.347 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:19:53.347 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344639+0000", 2024-04-20T17:19:53.348 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3250000000000002e-06 2024-04-20T17:19:53.348 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.348 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.348 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.348 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.347516+0000", 2024-04-20T17:19:53.348 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0028768959999999999 2024-04-20T17:19:53.348 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.348 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.348 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:19:53.348 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.347642+0000", 2024-04-20T17:19:53.348 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012597199999999999 2024-04-20T17:19:53.348 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.348 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.349 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:19:53.349 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.347773+0000", 2024-04-20T17:19:53.349 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013104600000000001 2024-04-20T17:19:53.349 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.349 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.349 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:19:53.349 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.354799+0000", 2024-04-20T17:19:53.349 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0070264330000000003 2024-04-20T17:19:53.349 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.349 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.349 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:53.349 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.401527+0000", 2024-04-20T17:19:53.349 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.046728262999999999 2024-04-20T17:19:53.349 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.350 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.350 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:19:53.350 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.401578+0000", 2024-04-20T17:19:53.350 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0299000000000001e-05 2024-04-20T17:19:53.350 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.350 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.350 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:19:53.350 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.401632+0000", 2024-04-20T17:19:53.350 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3853000000000001e-05 2024-04-20T17:19:53.350 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.350 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:19:53.350 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.351 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.351 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.351 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:197574 2.2 2:453bbb18:::benchmark_data_smithi171_78918_object36580:head [delete] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-04-20T17:19:53.351 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:34.344795+0000", 2024-04-20T17:19:53.351 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 558.911995488, 2024-04-20T17:19:53.351 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.056943622999999999, 2024-04-20T17:19:53.351 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:19:53.351 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:19:53.351 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:53.351 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:19:53.351 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:19:53.351 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:53.351 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 197574 2024-04-20T17:19:53.352 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.352 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:19:53.352 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.352 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:19:53.352 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344795+0000", 2024-04-20T17:19:53.352 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.352 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.352 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.352 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:19:53.352 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344795+0000", 2024-04-20T17:19:53.352 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.352 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.352 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.352 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:19:53.353 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344797+0000", 2024-04-20T17:19:53.353 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5e-06 2024-04-20T17:19:53.353 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.353 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.353 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:19:53.353 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344802+0000", 2024-04-20T17:19:53.353 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9350000000000002e-06 2024-04-20T17:19:53.353 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.353 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.353 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:19:53.353 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344814+0000", 2024-04-20T17:19:53.353 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2300000000000001e-05 2024-04-20T17:19:53.353 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.354 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.354 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:19:53.354 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.344822+0000", 2024-04-20T17:19:53.354 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1519999999999992e-06 2024-04-20T17:19:53.354 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.354 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.354 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.354 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.351164+0000", 2024-04-20T17:19:53.354 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0063413799999999998 2024-04-20T17:19:53.354 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.354 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.354 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:19:53.354 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.351280+0000", 2024-04-20T17:19:53.354 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011615700000000001 2024-04-20T17:19:53.354 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.355 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.355 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:19:53.355 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.351359+0000", 2024-04-20T17:19:53.355 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8974000000000007e-05 2024-04-20T17:19:53.355 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.355 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.355 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:19:53.355 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.354931+0000", 2024-04-20T17:19:53.355 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0035719570000000002 2024-04-20T17:19:53.355 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.355 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.355 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:53.355 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.401670+0000", 2024-04-20T17:19:53.355 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.046739347000000001 2024-04-20T17:19:53.355 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.356 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.356 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:19:53.356 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.401699+0000", 2024-04-20T17:19:53.356 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9179999999999998e-05 2024-04-20T17:19:53.356 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.356 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.356 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:19:53.356 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:34.401739+0000", 2024-04-20T17:19:53.356 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9740999999999998e-05 2024-04-20T17:19:53.356 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.356 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:19:53.356 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.356 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.356 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.356 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:202243 2.4 2:2e1fd5a1:::benchmark_data_smithi171_78918_object41249:head [delete] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-04-20T17:19:53.357 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:39.784900+0000", 2024-04-20T17:19:53.357 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 553.47189111299997, 2024-04-20T17:19:53.357 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.062568198000000005, 2024-04-20T17:19:53.357 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:19:53.357 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:19:53.357 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:53.357 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:19:53.357 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:19:53.357 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:53.357 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 202243 2024-04-20T17:19:53.357 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.357 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:19:53.357 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.357 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:19:53.358 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.784900+0000", 2024-04-20T17:19:53.358 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.358 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.358 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.358 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:19:53.358 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.784900+0000", 2024-04-20T17:19:53.358 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.358 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.358 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.358 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:19:53.358 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.784903+0000", 2024-04-20T17:19:53.358 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.633e-06 2024-04-20T17:19:53.358 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.358 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.359 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:19:53.359 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.784908+0000", 2024-04-20T17:19:53.359 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5679999999999999e-06 2024-04-20T17:19:53.359 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.359 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.359 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:19:53.359 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.784919+0000", 2024-04-20T17:19:53.359 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0583e-05 2024-04-20T17:19:53.359 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.359 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.359 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:19:53.359 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.784926+0000", 2024-04-20T17:19:53.359 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.515e-06 2024-04-20T17:19:53.359 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.360 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.360 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.360 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.785293+0000", 2024-04-20T17:19:53.360 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000366898 2024-04-20T17:19:53.360 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.360 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.360 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:19:53.360 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.785405+0000", 2024-04-20T17:19:53.360 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000111912 2024-04-20T17:19:53.360 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.360 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.360 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:19:53.360 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.785512+0000", 2024-04-20T17:19:53.360 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010692900000000001 2024-04-20T17:19:53.360 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.361 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.361 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:19:53.361 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.789865+0000", 2024-04-20T17:19:53.361 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0043533360000000002 2024-04-20T17:19:53.361 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.361 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.361 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:53.361 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.847336+0000", 2024-04-20T17:19:53.361 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.057471181000000003 2024-04-20T17:19:53.361 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.361 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.361 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:19:53.361 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.847403+0000", 2024-04-20T17:19:53.361 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6317999999999998e-05 2024-04-20T17:19:53.361 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.362 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.362 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:19:53.362 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.847468+0000", 2024-04-20T17:19:53.362 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5325000000000002e-05 2024-04-20T17:19:53.362 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.362 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:19:53.362 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.362 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.362 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.362 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:202253 2.4 2:2c7ba7dc:::benchmark_data_smithi171_78918_object41259:head [delete] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-04-20T17:19:53.362 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:39.790264+0000", 2024-04-20T17:19:53.362 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 553.46652736700003, 2024-04-20T17:19:53.362 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.057390243000000001, 2024-04-20T17:19:53.363 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:19:53.363 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:19:53.363 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:53.363 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:19:53.363 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:19:53.363 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:53.363 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 202253 2024-04-20T17:19:53.363 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.363 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:19:53.363 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.363 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:19:53.363 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.790264+0000", 2024-04-20T17:19:53.363 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.363 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.363 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.364 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:19:53.364 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.790264+0000", 2024-04-20T17:19:53.364 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.364 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.364 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.364 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:19:53.364 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.790265+0000", 2024-04-20T17:19:53.364 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7379999999999999e-06 2024-04-20T17:19:53.364 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.364 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.364 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:19:53.364 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.790270+0000", 2024-04-20T17:19:53.365 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7570000000000004e-06 2024-04-20T17:19:53.365 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.365 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.365 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:19:53.365 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.790278+0000", 2024-04-20T17:19:53.365 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2649999999999996e-06 2024-04-20T17:19:53.365 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.365 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.365 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:19:53.365 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.790284+0000", 2024-04-20T17:19:53.365 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2299999999999999e-06 2024-04-20T17:19:53.365 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.365 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.365 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.366 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.790490+0000", 2024-04-20T17:19:53.366 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00020615999999999999 2024-04-20T17:19:53.366 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.366 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.366 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:19:53.366 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.790561+0000", 2024-04-20T17:19:53.366 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1202e-05 2024-04-20T17:19:53.366 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.366 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.366 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:19:53.366 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.790627+0000", 2024-04-20T17:19:53.366 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6366999999999997e-05 2024-04-20T17:19:53.366 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.366 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.367 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:19:53.367 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.793569+0000", 2024-04-20T17:19:53.367 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0029416799999999999 2024-04-20T17:19:53.367 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.367 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.367 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:53.367 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.847528+0000", 2024-04-20T17:19:53.367 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.053958675999999997 2024-04-20T17:19:53.367 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.367 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.367 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:19:53.367 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.847587+0000", 2024-04-20T17:19:53.367 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8881000000000001e-05 2024-04-20T17:19:53.367 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.368 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.368 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:19:53.368 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:39.847654+0000", 2024-04-20T17:19:53.368 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7286999999999995e-05 2024-04-20T17:19:53.368 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.368 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:19:53.368 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.368 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.368 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.368 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:308750 2.2 2:4bf72ae4:::benchmark_data_smithi171_78918_object147756:head [delete] snapc 0=[] ondisk+write+known_if_redirected e344)", 2024-04-20T17:19:53.368 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:19:51.558943+0000", 2024-04-20T17:19:53.368 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 1.6978478990000001, 2024-04-20T17:19:53.368 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.027705099, 2024-04-20T17:19:53.368 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:19:53.368 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:19:53.369 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:53.369 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:19:53.369 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:19:53.369 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:53.369 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 308750 2024-04-20T17:19:53.369 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.369 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:19:53.369 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.369 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:19:53.369 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.558943+0000", 2024-04-20T17:19:53.369 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.369 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.369 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.369 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:19:53.369 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.558943+0000", 2024-04-20T17:19:53.370 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.370 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.370 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.370 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:19:53.370 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.558944+0000", 2024-04-20T17:19:53.370 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8400000000000002e-07 2024-04-20T17:19:53.370 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.370 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.370 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:19:53.370 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.558948+0000", 2024-04-20T17:19:53.370 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6009999999999999e-06 2024-04-20T17:19:53.370 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.370 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.371 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:19:53.371 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.558955+0000", 2024-04-20T17:19:53.371 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9380000000000003e-06 2024-04-20T17:19:53.371 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.371 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.371 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:19:53.371 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.558961+0000", 2024-04-20T17:19:53.371 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1290000000000001e-06 2024-04-20T17:19:53.371 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.371 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.371 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.371 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.563073+0000", 2024-04-20T17:19:53.371 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0041125420000000003 2024-04-20T17:19:53.372 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.372 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.372 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:19:53.372 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.563201+0000", 2024-04-20T17:19:53.372 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0001281 2024-04-20T17:19:53.372 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.372 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.372 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:19:53.372 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.563332+0000", 2024-04-20T17:19:53.372 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013049000000000001 2024-04-20T17:19:53.372 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.372 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.372 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:19:53.372 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.567174+0000", 2024-04-20T17:19:53.372 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0038423580000000001 2024-04-20T17:19:53.373 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.373 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.373 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:53.373 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.586583+0000", 2024-04-20T17:19:53.373 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.019408280999999999 2024-04-20T17:19:53.373 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.373 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.373 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:19:53.373 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.586619+0000", 2024-04-20T17:19:53.373 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6547000000000001e-05 2024-04-20T17:19:53.373 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.373 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.373 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:19:53.373 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.586648+0000", 2024-04-20T17:19:53.373 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9128999999999998e-05 2024-04-20T17:19:53.374 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.374 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:19:53.374 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.374 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.374 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.374 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:309137 2.4 2:24bacc14:::benchmark_data_smithi171_78918_object148143:head [delete] snapc 0=[] ondisk+write+known_if_redirected e344)", 2024-04-20T17:19:53.374 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:19:51.953471+0000", 2024-04-20T17:19:53.374 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 1.303320483, 2024-04-20T17:19:53.374 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.032431807, 2024-04-20T17:19:53.374 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:19:53.374 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:19:53.374 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:19:53.374 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:19:53.375 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:19:53.375 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:53.375 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 309137 2024-04-20T17:19:53.375 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.375 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:19:53.375 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.375 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:19:53.375 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.953471+0000", 2024-04-20T17:19:53.375 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.375 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.375 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.375 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:19:53.375 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.953471+0000", 2024-04-20T17:19:53.376 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.376 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.376 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.376 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:19:53.376 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.953472+0000", 2024-04-20T17:19:53.376 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6869999999999999e-06 2024-04-20T17:19:53.376 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.376 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.376 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:19:53.376 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.953477+0000", 2024-04-20T17:19:53.376 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4070000000000003e-06 2024-04-20T17:19:53.376 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.376 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.376 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:19:53.376 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.953486+0000", 2024-04-20T17:19:53.377 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9649999999999997e-06 2024-04-20T17:19:53.377 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.377 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.377 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:19:53.377 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.953491+0000", 2024-04-20T17:19:53.377 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4369999999999998e-06 2024-04-20T17:19:53.377 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.377 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.377 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.377 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.953541+0000", 2024-04-20T17:19:53.377 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0046999999999997e-05 2024-04-20T17:19:53.377 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.377 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.377 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:19:53.378 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.953652+0000", 2024-04-20T17:19:53.378 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0001108 2024-04-20T17:19:53.378 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.378 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.378 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:19:53.378 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.953737+0000", 2024-04-20T17:19:53.378 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5277000000000004e-05 2024-04-20T17:19:53.378 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.378 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.378 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:19:53.378 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.957257+0000", 2024-04-20T17:19:53.378 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0035201189999999999 2024-04-20T17:19:53.378 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.378 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.378 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:19:53.379 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.958719+0000", 2024-04-20T17:19:53.379 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001461878 2024-04-20T17:19:53.379 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.379 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.379 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:19:53.379 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.985857+0000", 2024-04-20T17:19:53.379 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.027137840999999999 2024-04-20T17:19:53.379 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.379 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.379 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:19:53.379 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.985902+0000", 2024-04-20T17:19:53.379 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5349000000000002e-05 2024-04-20T17:19:53.379 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.380 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:19:53.380 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.380 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.380 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.380 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.24736.0:309137 2.4 e344/162 ondisk, result = 0)", 2024-04-20T17:19:53.380 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:19:51.958638+0000", 2024-04-20T17:19:53.380 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 1.2981527129999999, 2024-04-20T17:19:53.380 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.027271780999999998, 2024-04-20T17:19:53.380 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:19:53.380 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:19:53.380 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-04-20T17:19:53.380 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:19:53.380 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.380 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:19:53.381 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.958638+0000", 2024-04-20T17:19:53.381 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.381 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.381 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.381 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:19:53.381 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.958638+0000", 2024-04-20T17:19:53.381 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:53.381 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.381 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.381 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:19:53.381 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.958640+0000", 2024-04-20T17:19:53.381 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.494e-06 2024-04-20T17:19:53.381 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.381 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.381 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:19:53.382 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.958644+0000", 2024-04-20T17:19:53.382 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.617e-06 2024-04-20T17:19:53.382 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.382 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.382 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:19:53.382 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.958670+0000", 2024-04-20T17:19:53.382 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5394000000000001e-05 2024-04-20T17:19:53.382 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.382 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.382 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:19:53.382 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.958675+0000", 2024-04-20T17:19:53.382 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9559999999999996e-06 2024-04-20T17:19:53.382 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.383 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.383 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:53.383 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.958711+0000", 2024-04-20T17:19:53.383 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6458000000000002e-05 2024-04-20T17:19:53.383 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.383 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.383 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:19:53.383 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.958716+0000", 2024-04-20T17:19:53.383 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8940000000000001e-06 2024-04-20T17:19:53.383 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:53.383 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:53.383 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:19:53.383 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.985910+0000", 2024-04-20T17:19:53.383 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.027193967999999999 2024-04-20T17:19:53.383 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.384 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:19:53.384 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.384 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:53.384 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:19:53.384 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:19:53.384 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:19:53.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:53 smithi171 ceph-mon[44189]: pgmap v947: 17 pgs: 1 active+recovering, 16 active+clean; 946 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 627 op/s; 1041/30262 objects misplaced (3.440%); 2.0 MiB/s, 32 objects/s recovering 2024-04-20T17:19:53.589 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:19:53.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:53.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:19:53.610 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-20T17:19:53.610 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:19:53.694 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:19:53.726 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:53.750 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:19:53.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:53.754 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:19:53.856 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:19:53.898 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:53.996 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-20T17:19:54.274 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:19:54.286 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:19:54.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:54.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:19:54.294 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:19:54.307 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:19:54.394 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:19:54.447 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:54.473 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:19:54.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:54.481 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:19:54.581 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:19:54.619 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:54.645 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:19:54.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:54.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:19:54.750 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:19:54.787 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:54.814 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:19:54.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:54.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:19:54.921 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:19:54.955 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:55.156 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:19:55.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:55.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:19:55.262 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:19:55.312 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:19:55.450 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:55 smithi005 ceph-mon[39562]: pgmap v948: 17 pgs: 1 active+recovering, 16 active+clean; 790 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 789 op/s; 544/25272 objects misplaced (2.153%); 3.5 MiB/s, 56 objects/s recovering 2024-04-20T17:19:55.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:55.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:19:55.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:55 smithi106 ceph-mon[44423]: pgmap v948: 17 pgs: 1 active+recovering, 16 active+clean; 790 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 789 op/s; 544/25272 objects misplaced (2.153%); 3.5 MiB/s, 56 objects/s recovering 2024-04-20T17:19:55.553 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:19:55.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:55 smithi171 ceph-mon[44189]: pgmap v948: 17 pgs: 1 active+recovering, 16 active+clean; 790 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 789 op/s; 544/25272 objects misplaced (2.153%); 3.5 MiB/s, 56 objects/s recovering 2024-04-20T17:19:55.591 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:55.707 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:19:55.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:55.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:19:55.814 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:19:55.850 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:56.188 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:56 smithi005 ceph-mon[39562]: 2.7 scrub starts 2024-04-20T17:19:56.189 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:56 smithi005 ceph-mon[39562]: 2.7 scrub ok 2024-04-20T17:19:56.189 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:56 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3662844011' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:19:56.197 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:19:56.246 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:19:56.246 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:19:56.246 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:56.246 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24736.0:313073 2.1 e345/196 2:8197cd1c:::benchmark_data_smithi171_78918_object152079:head v 345'26298, mlcod=344'26296)", 2024-04-20T17:19:56.246 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:19:56.231688+0000", 2024-04-20T17:19:56.246 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0020826590000000002, 2024-04-20T17:19:56.246 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0021067389999999998, 2024-04-20T17:19:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:19:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:19:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-04-20T17:19:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:19:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:19:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:56.231688+0000", 2024-04-20T17:19:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:19:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:56.231688+0000", 2024-04-20T17:19:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:56.248 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:19:56.248 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:56.231690+0000", 2024-04-20T17:19:56.248 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.686e-06 2024-04-20T17:19:56.248 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:56.248 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:56.248 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:19:56.248 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:56.231698+0000", 2024-04-20T17:19:56.248 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0039999999999999e-06 2024-04-20T17:19:56.248 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:56.248 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:56.248 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:19:56.248 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:56.231709+0000", 2024-04-20T17:19:56.249 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.065e-05 2024-04-20T17:19:56.249 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:56.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:56.249 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:19:56.249 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:56.231714+0000", 2024-04-20T17:19:56.249 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8370000000000002e-06 2024-04-20T17:19:56.249 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:56.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:56.249 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:56.249 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:56.231768+0000", 2024-04-20T17:19:56.249 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3427999999999997e-05 2024-04-20T17:19:56.249 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:56.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:56.249 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:19:56.250 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:56.231792+0000", 2024-04-20T17:19:56.250 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3702000000000001e-05 2024-04-20T17:19:56.250 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:56.250 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:19:56.250 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:56.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:56.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:56.250 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:313075 2.7 2:e38fca43:::benchmark_data_smithi171_78918_object152081:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-04-20T17:19:56.250 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:19:56.232452+0000", 2024-04-20T17:19:56.250 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.00131865, 2024-04-20T17:19:56.250 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013844090000000001, 2024-04-20T17:19:56.250 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:19:56.250 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:19:56.251 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:19:56.251 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:19:56.251 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:19:56.251 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:56.251 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 313075 2024-04-20T17:19:56.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:56.251 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:19:56.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:56.251 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:19:56.251 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:56.232452+0000", 2024-04-20T17:19:56.251 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:56.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:56.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:56.252 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:19:56.252 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:56.232452+0000", 2024-04-20T17:19:56.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:56.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:56.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:56.252 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:19:56.252 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:56.232455+0000", 2024-04-20T17:19:56.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3329999999999999e-06 2024-04-20T17:19:56.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:56.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:56.252 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:19:56.252 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:56.232461+0000", 2024-04-20T17:19:56.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3160000000000002e-06 2024-04-20T17:19:56.253 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:56.253 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:56.253 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:19:56.253 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:56.232474+0000", 2024-04-20T17:19:56.253 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3363000000000001e-05 2024-04-20T17:19:56.253 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:56.253 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:56.253 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:19:56.253 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:56.232483+0000", 2024-04-20T17:19:56.253 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5969999999999992e-06 2024-04-20T17:19:56.253 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:56.253 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:56.253 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:19:56.253 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:56.232549+0000", 2024-04-20T17:19:56.254 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6204000000000005e-05 2024-04-20T17:19:56.254 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:56.254 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:56.254 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:19:56.254 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:56.232661+0000", 2024-04-20T17:19:56.254 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000111835 2024-04-20T17:19:56.254 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:56.254 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:56.254 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:19:56.254 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:56.232741+0000", 2024-04-20T17:19:56.254 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0526999999999997e-05 2024-04-20T17:19:56.254 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:56.254 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:19:56.254 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:56.255 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:56.255 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:56.255 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:313079 2.7 2.8e0dbe77 (undecoded) ondisk+write+known_if_redirected e345)", 2024-04-20T17:19:56.255 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:19:56.232636+0000", 2024-04-20T17:19:56.255 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.00113463, 2024-04-20T17:19:56.255 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0012425789999999999, 2024-04-20T17:19:56.255 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:19:56.255 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:19:56.255 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "queued for pg", 2024-04-20T17:19:56.255 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:19:56.255 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:19:56.255 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:19:56.255 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 313079 2024-04-20T17:19:56.255 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:56.256 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:19:56.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:56.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:19:56.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:56.232636+0000", 2024-04-20T17:19:56.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:56.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:56.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:56.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:19:56.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:56.232636+0000", 2024-04-20T17:19:56.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:19:56.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:56.257 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:56.257 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:19:56.257 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:56.232638+0000", 2024-04-20T17:19:56.257 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4589999999999999e-06 2024-04-20T17:19:56.257 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:56.257 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:56.257 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:19:56.257 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:56.232643+0000", 2024-04-20T17:19:56.257 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4580000000000001e-06 2024-04-20T17:19:56.257 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:56.257 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:56.257 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:19:56.257 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:56.232652+0000", 2024-04-20T17:19:56.257 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1330000000000005e-06 2024-04-20T17:19:56.257 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:19:56.258 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:19:56.258 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:19:56.258 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:56.232659+0000", 2024-04-20T17:19:56.258 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0890000000000002e-06 2024-04-20T17:19:56.258 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:56.258 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:19:56.258 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:56.258 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:19:56.258 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-04-20T17:19:56.258 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 3 2024-04-20T17:19:56.258 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:19:56.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:56 smithi106 ceph-mon[44423]: 2.7 scrub starts 2024-04-20T17:19:56.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:56 smithi106 ceph-mon[44423]: 2.7 scrub ok 2024-04-20T17:19:56.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:56 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3662844011' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:19:56.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:56 smithi171 ceph-mon[44189]: 2.7 scrub starts 2024-04-20T17:19:56.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:56 smithi171 ceph-mon[44189]: 2.7 scrub ok 2024-04-20T17:19:56.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:56 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3662844011' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:19:56.963 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-20T17:19:57.192 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:57 smithi005 ceph-mon[39562]: pgmap v949: 17 pgs: 1 active+recovering, 16 active+clean; 703 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 778 op/s; 544/22492 objects misplaced (2.419%); 3.5 MiB/s, 56 objects/s recovering 2024-04-20T17:19:57.192 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:57 smithi005 ceph-mon[39562]: 2.d scrub starts 2024-04-20T17:19:57.192 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:57 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:19:57.192 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:57 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3662844011' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:19:57.192 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:57 smithi005 ceph-mon[39562]: osdmap e345: 12 total, 12 up, 12 in 2024-04-20T17:19:57.300 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:19:57.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:57 smithi106 ceph-mon[44423]: pgmap v949: 17 pgs: 1 active+recovering, 16 active+clean; 703 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 778 op/s; 544/22492 objects misplaced (2.419%); 3.5 MiB/s, 56 objects/s recovering 2024-04-20T17:19:57.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:57 smithi106 ceph-mon[44423]: 2.d scrub starts 2024-04-20T17:19:57.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:57 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:19:57.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:57 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3662844011' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:19:57.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:57 smithi106 ceph-mon[44423]: osdmap e345: 12 total, 12 up, 12 in 2024-04-20T17:19:57.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:57 smithi171 ceph-mon[44189]: pgmap v949: 17 pgs: 1 active+recovering, 16 active+clean; 703 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 778 op/s; 544/22492 objects misplaced (2.419%); 3.5 MiB/s, 56 objects/s recovering 2024-04-20T17:19:57.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:57 smithi171 ceph-mon[44189]: 2.d scrub starts 2024-04-20T17:19:57.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:57 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:19:57.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:57 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3662844011' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:19:57.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:57 smithi171 ceph-mon[44189]: osdmap e345: 12 total, 12 up, 12 in 2024-04-20T17:19:58.005 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:19:58.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:58.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:19:58.110 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:19:58.161 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:58.192 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:19:58.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:58.200 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:19:58.301 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:19:58.341 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:58.369 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:19:58.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:58.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:19:58.474 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:19:58.505 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:58.707 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:19:58.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:58.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:19:58.811 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:19:58.861 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:58.899 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:19:58.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:58.905 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:19:59.006 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:19:59.047 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:59.208 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:19:59 smithi171 ceph-mon[44189]: pgmap v951: 17 pgs: 1 active+recovering, 16 active+clean; 569 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s; 544/18194 objects misplaced (2.990%); 1.8 MiB/s, 28 objects/s recovering 2024-04-20T17:19:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:19:59 smithi106 ceph-mon[44423]: pgmap v951: 17 pgs: 1 active+recovering, 16 active+clean; 569 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s; 544/18194 objects misplaced (2.990%); 1.8 MiB/s, 28 objects/s recovering 2024-04-20T17:19:59.264 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:19:59.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:19:59.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:19:59.290 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:19:59.308 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:19:59.309 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:19:59.309 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-04-20T17:19:59.309 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-04-20T17:19:59.309 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:19:59.317 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:19:59.349 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:19:59.369 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:19:59.379 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:19:59.403 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:19:59.409 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:19:59.435 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:19:59.462 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:19:59.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:19:59 smithi005 ceph-mon[39562]: pgmap v951: 17 pgs: 1 active+recovering, 16 active+clean; 569 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s; 544/18194 objects misplaced (2.990%); 1.8 MiB/s, 28 objects/s recovering 2024-04-20T17:19:59.493 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:19:59.494 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:19:59.535 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:19:59.570 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:19:59.605 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:19:59.638 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:19:59.639 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:19:59.639 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:19:59.640 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:19:59.640 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:19:59.640 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:19:59.640 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:19:59.640 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:19:59.640 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:19:59.640 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:19:59.640 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:19:59.641 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:19:59.641 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:20:00.068 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:20:00.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:00.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:20:00.077 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-20T17:20:00.173 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:20:00.231 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:00.232 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:20:00.237 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:20:00.455 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:20:00.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:00 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1151124874' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:20:00.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:00.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:00.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:00.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:00.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:00.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:00.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:00.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:00.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:00 smithi005 ceph-mon[39562]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-04-20T17:20:00.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:00 smithi005 ceph-mon[39562]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-04-20T17:20:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:00 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1151124874' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:20:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:00 smithi106 ceph-mon[44423]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-04-20T17:20:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:00 smithi106 ceph-mon[44423]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-04-20T17:20:00.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:00 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1151124874' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:20:00.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:00.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:00.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:00.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:00.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:00.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:00.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:00.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:00.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:00 smithi171 ceph-mon[44189]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-04-20T17:20:00.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:00 smithi171 ceph-mon[44189]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-04-20T17:20:00.892 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:20:00.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:00.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:20:00.999 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:20:01.042 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:01.395 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:20:01.397 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:01.397 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:01.397 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:01.397 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:01.397 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:01 smithi106 ceph-mon[44423]: pgmap v952: 17 pgs: 17 active+clean; 485 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 885 op/s; 3.3 MiB/s, 52 objects/s recovering 2024-04-20T17:20:01.397 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:01 smithi106 ceph-mon[44423]: from='client.25693 v1:172.21.15.5:0/2765431947' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:20:01.398 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:01 smithi106 ceph-mon[44423]: osdmap e346: 12 total, 12 up, 12 in 2024-04-20T17:20:01.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:01.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:20:01.499 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:20:01.538 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:01.564 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:20:01.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:01.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:20:01.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:01.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:01.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:01.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:01.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:01 smithi171 ceph-mon[44189]: pgmap v952: 17 pgs: 17 active+clean; 485 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 885 op/s; 3.3 MiB/s, 52 objects/s recovering 2024-04-20T17:20:01.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:01 smithi171 ceph-mon[44189]: from='client.25693 v1:172.21.15.5:0/2765431947' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:20:01.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:01 smithi171 ceph-mon[44189]: osdmap e346: 12 total, 12 up, 12 in 2024-04-20T17:20:01.613 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:01.613 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:01.613 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:01.613 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:01.613 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:01 smithi005 ceph-mon[39562]: pgmap v952: 17 pgs: 17 active+clean; 485 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 885 op/s; 3.3 MiB/s, 52 objects/s recovering 2024-04-20T17:20:01.613 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:01 smithi005 ceph-mon[39562]: from='client.25693 v1:172.21.15.5:0/2765431947' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:20:01.613 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:01 smithi005 ceph-mon[39562]: osdmap e346: 12 total, 12 up, 12 in 2024-04-20T17:20:01.648 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":952,"stamp":"2024-04-20T17:20:00.089234+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":508756536,"num_objects":7757,"num_object_clones":0,"num_object_copies":15514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7757,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":314410,"num_write_kb":10306363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29224,"num_bytes_recovered":1916815488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160965,"ondisk_log_size":160965,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":3354552,"kb_used_data":2096844,"kb_used_omap":219,"kb_used_meta":1257316,"kb_avail":1121587272,"statfs":{"total":1151940427776,"available":1148505366528,"internally_reserved":0,"allocated":2147168256,"data_stored":2132175850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224949,"internal_metadata":1287491915},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,3,9,9],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-580648960,"num_objects":-8860,"num_object_clones":0,"num_object_copies":-17720,"num_objects_missing_on_primary":-516,"num_objects_missing":-516,"num_objects_degraded":0,"num_objects_misplaced":-1041,"num_objects_unfound":0,"num_objects_dirty":-8860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8860,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":525,"num_bytes_recovered":34406400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003109"},"pg_stats":[{"pgid":"2.f","version":"345'19791","reported_seq":20842,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-04-20T17:19:58.554798+0000","last_change":"2024-04-20T17:19:03.232229+0000","last_active":"2024-04-20T17:19:58.554798+0000","last_peered":"2024-04-20T17:19:58.554798+0000","last_clean":"2024-04-20T17:19:58.554798+0000","last_became_active":"2024-04-20T17:08:06.005877+0000","last_became_peered":"2024-04-20T17:08:06.005877+0000","last_unstale":"2024-04-20T17:19:58.554798+0000","last_undegraded":"2024-04-20T17:19:58.554798+0000","last_fullsized":"2024-04-20T17:19:58.554798+0000","mapping_epoch":115,"log_start":"151'9700","ondisk_log_start":"151'9700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'17904","last_scrub_stamp":"2024-04-20T17:19:03.232003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:19:03.232003+0000","objects_scrubbed":2282,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:09.723163+0000","scrub_duration":268,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624576,"num_objects":391,"num_object_clones":0,"num_object_copies":782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19791,"num_write_kb":645824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"344'19472","reported_seq":31577,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-04-20T17:19:55.527877+0000","last_change":"2024-04-20T17:19:40.244380+0000","last_active":"2024-04-20T17:19:55.527877+0000","last_peered":"2024-04-20T17:19:55.527877+0000","last_clean":"2024-04-20T17:19:55.527877+0000","last_became_active":"2024-04-20T17:10:28.865670+0000","last_became_peered":"2024-04-20T17:10:28.865670+0000","last_unstale":"2024-04-20T17:19:55.527877+0000","last_undegraded":"2024-04-20T17:19:55.527877+0000","last_fullsized":"2024-04-20T17:19:55.527877+0000","mapping_epoch":162,"log_start":"146'9400","ondisk_log_start":"146'9400","created":115,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'18745","last_scrub_stamp":"2024-04-20T17:19:40.244244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:19:40.244244+0000","objects_scrubbed":1311,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:59.798440+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38273024,"num_objects":584,"num_object_clones":0,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19472,"num_write_kb":641792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5619,"num_bytes_recovered":368246784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"344'19835","reported_seq":20734,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-04-20T17:19:55.522834+0000","last_change":"2024-04-20T17:18:49.298437+0000","last_active":"2024-04-20T17:19:55.522834+0000","last_peered":"2024-04-20T17:19:55.522834+0000","last_clean":"2024-04-20T17:19:55.522834+0000","last_became_active":"2024-04-20T17:08:06.013348+0000","last_became_peered":"2024-04-20T17:08:06.013348+0000","last_unstale":"2024-04-20T17:19:55.522834+0000","last_undegraded":"2024-04-20T17:19:55.522834+0000","last_fullsized":"2024-04-20T17:19:55.522834+0000","mapping_epoch":115,"log_start":"149'9800","ondisk_log_start":"149'9800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'17915","last_scrub_stamp":"2024-04-20T17:18:49.298307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:49.298307+0000","objects_scrubbed":2567,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:03.315951+0000","scrub_duration":225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42401792,"num_objects":647,"num_object_clones":0,"num_object_copies":1294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19835,"num_write_kb":655424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"344'26270","reported_seq":47954,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-04-20T17:19:55.765809+0000","last_change":"2024-04-20T17:19:39.130990+0000","last_active":"2024-04-20T17:19:55.765809+0000","last_peered":"2024-04-20T17:19:55.765809+0000","last_clean":"2024-04-20T17:19:55.765809+0000","last_became_active":"2024-04-20T17:12:08.085936+0000","last_became_peered":"2024-04-20T17:12:08.085936+0000","last_unstale":"2024-04-20T17:19:55.765809+0000","last_undegraded":"2024-04-20T17:19:55.765809+0000","last_fullsized":"2024-04-20T17:19:55.765809+0000","mapping_epoch":196,"log_start":"208'16268","ondisk_log_start":"208'16268","created":115,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'25482","last_scrub_stamp":"2024-04-20T17:19:39.130906+0000","last_deep_scrub":"213'20373","last_deep_scrub_stamp":"2024-04-20T17:12:48.443808+0000","last_clean_scrub_stamp":"2024-04-20T17:19:39.130906+0000","objects_scrubbed":1406,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:52.548198+0000","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40370176,"num_objects":616,"num_object_clones":0,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19618,"num_write_kb":647488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":450166784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"345'19354","reported_seq":48131,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-04-20T17:19:59.056083+0000","last_change":"2024-04-20T17:19:58.249351+0000","last_active":"2024-04-20T17:19:59.056083+0000","last_peered":"2024-04-20T17:19:59.056083+0000","last_clean":"2024-04-20T17:19:59.056083+0000","last_became_active":"2024-04-20T17:10:54.470253+0000","last_became_peered":"2024-04-20T17:10:54.470253+0000","last_unstale":"2024-04-20T17:19:59.056083+0000","last_undegraded":"2024-04-20T17:19:59.056083+0000","last_fullsized":"2024-04-20T17:19:59.056083+0000","mapping_epoch":171,"log_start":"148'9300","ondisk_log_start":"148'9300","created":115,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:08:04.990639+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24641536,"num_objects":376,"num_object_clones":0,"num_object_copies":752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19354,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9432,"num_bytes_recovered":618135552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":902,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-04-20T17:19:56.202163+0000","last_change":"2024-04-20T17:19:36.190286+0000","last_active":"2024-04-20T17:19:56.202163+0000","last_peered":"2024-04-20T17:19:56.202163+0000","last_clean":"2024-04-20T17:19:56.202163+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:19:56.202163+0000","last_undegraded":"2024-04-20T17:19:56.202163+0000","last_fullsized":"2024-04-20T17:19:56.202163+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:19:36.190151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:19:36.190151+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:40.598718+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"345'22819","reported_seq":23867,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-04-20T17:19:57.744678+0000","last_change":"2024-04-20T17:19:43.219161+0000","last_active":"2024-04-20T17:19:57.744678+0000","last_peered":"2024-04-20T17:19:57.744678+0000","last_clean":"2024-04-20T17:19:57.744678+0000","last_became_active":"2024-04-20T17:08:06.044634+0000","last_became_peered":"2024-04-20T17:08:06.044634+0000","last_unstale":"2024-04-20T17:19:57.744678+0000","last_undegraded":"2024-04-20T17:19:57.744678+0000","last_fullsized":"2024-04-20T17:19:57.744678+0000","mapping_epoch":115,"log_start":"167'12800","ondisk_log_start":"167'12800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'22048","last_scrub_stamp":"2024-04-20T17:19:43.219059+0000","last_deep_scrub":"321'20320","last_deep_scrub_stamp":"2024-04-20T17:18:28.829770+0000","last_clean_scrub_stamp":"2024-04-20T17:19:43.219059+0000","objects_scrubbed":1215,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:59.081422+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28770304,"num_objects":439,"num_object_clones":0,"num_object_copies":878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19813,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"345'19922","reported_seq":20853,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-04-20T17:19:58.535624+0000","last_change":"2024-04-20T17:18:49.142560+0000","last_active":"2024-04-20T17:19:58.535624+0000","last_peered":"2024-04-20T17:19:58.535624+0000","last_clean":"2024-04-20T17:19:58.535624+0000","last_became_active":"2024-04-20T17:08:06.003050+0000","last_became_peered":"2024-04-20T17:08:06.003050+0000","last_unstale":"2024-04-20T17:19:58.535624+0000","last_undegraded":"2024-04-20T17:19:58.535624+0000","last_fullsized":"2024-04-20T17:19:58.535624+0000","mapping_epoch":115,"log_start":"152'9900","ondisk_log_start":"152'9900","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'17862","last_scrub_stamp":"2024-04-20T17:18:49.142459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:49.142459+0000","objects_scrubbed":2464,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:07.477325+0000","scrub_duration":245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26345472,"num_objects":402,"num_object_clones":0,"num_object_copies":804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19922,"num_write_kb":650368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"345'23677","reported_seq":24737,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-04-20T17:19:56.989662+0000","last_change":"2024-04-20T17:19:37.659813+0000","last_active":"2024-04-20T17:19:56.989662+0000","last_peered":"2024-04-20T17:19:56.989662+0000","last_clean":"2024-04-20T17:19:56.989662+0000","last_became_active":"2024-04-20T17:08:06.009378+0000","last_became_peered":"2024-04-20T17:08:06.009378+0000","last_unstale":"2024-04-20T17:19:56.989662+0000","last_undegraded":"2024-04-20T17:19:56.989662+0000","last_fullsized":"2024-04-20T17:19:56.989662+0000","mapping_epoch":115,"log_start":"210'13602","ondisk_log_start":"210'13602","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'22771","last_scrub_stamp":"2024-04-20T17:19:37.659563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:19:37.659563+0000","objects_scrubbed":1384,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:39.997966+0000","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31195136,"num_objects":476,"num_object_clones":0,"num_object_copies":952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19566,"num_write_kb":641344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8],"acting":[1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"345'19788","reported_seq":20790,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-04-20T17:19:57.322846+0000","last_change":"2024-04-20T17:19:54.949485+0000","last_active":"2024-04-20T17:19:57.322846+0000","last_peered":"2024-04-20T17:19:57.322846+0000","last_clean":"2024-04-20T17:19:57.322846+0000","last_became_active":"2024-04-20T17:08:06.044737+0000","last_became_peered":"2024-04-20T17:08:06.044737+0000","last_unstale":"2024-04-20T17:19:57.322846+0000","last_undegraded":"2024-04-20T17:19:57.322846+0000","last_fullsized":"2024-04-20T17:19:57.322846+0000","mapping_epoch":115,"log_start":"148'9700","ondisk_log_start":"148'9700","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'19623","last_scrub_stamp":"2024-04-20T17:19:54.949237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:19:54.949237+0000","objects_scrubbed":598,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:13.307535+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28311552,"num_objects":432,"num_object_clones":0,"num_object_copies":864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19788,"num_write_kb":647040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"345'19599","reported_seq":20512,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-04-20T17:19:57.748354+0000","last_change":"2024-04-20T17:18:45.702943+0000","last_active":"2024-04-20T17:19:57.748354+0000","last_peered":"2024-04-20T17:19:57.748354+0000","last_clean":"2024-04-20T17:19:57.748354+0000","last_became_active":"2024-04-20T17:08:06.005788+0000","last_became_peered":"2024-04-20T17:08:06.005788+0000","last_unstale":"2024-04-20T17:19:57.748354+0000","last_undegraded":"2024-04-20T17:19:57.748354+0000","last_fullsized":"2024-04-20T17:19:57.748354+0000","mapping_epoch":115,"log_start":"148'9500","ondisk_log_start":"148'9500","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'17455","last_scrub_stamp":"2024-04-20T17:18:45.702769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:45.702769+0000","objects_scrubbed":2599,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:02.040514+0000","scrub_duration":222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29687808,"num_objects":453,"num_object_clones":0,"num_object_copies":906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19599,"num_write_kb":641664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"345'19798","reported_seq":35301,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-04-20T17:19:57.760996+0000","last_change":"2024-04-20T17:19:38.250604+0000","last_active":"2024-04-20T17:19:57.760996+0000","last_peered":"2024-04-20T17:19:57.760996+0000","last_clean":"2024-04-20T17:19:57.760996+0000","last_became_active":"2024-04-20T17:10:55.931751+0000","last_became_peered":"2024-04-20T17:10:55.931751+0000","last_unstale":"2024-04-20T17:19:57.760996+0000","last_undegraded":"2024-04-20T17:19:57.760996+0000","last_fullsized":"2024-04-20T17:19:57.760996+0000","mapping_epoch":173,"log_start":"149'9700","ondisk_log_start":"149'9700","created":115,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'18906","last_scrub_stamp":"2024-04-20T17:19:38.250494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:19:38.250494+0000","objects_scrubbed":1356,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:46.992923+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30408704,"num_objects":464,"num_object_clones":0,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19798,"num_write_kb":648384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7296,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"345'19862","reported_seq":20821,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-04-20T17:19:57.564161+0000","last_change":"2024-04-20T17:19:39.448326+0000","last_active":"2024-04-20T17:19:57.564161+0000","last_peered":"2024-04-20T17:19:57.564161+0000","last_clean":"2024-04-20T17:19:57.564161+0000","last_became_active":"2024-04-20T17:08:06.027950+0000","last_became_peered":"2024-04-20T17:08:06.027950+0000","last_unstale":"2024-04-20T17:19:57.564161+0000","last_undegraded":"2024-04-20T17:19:57.564161+0000","last_fullsized":"2024-04-20T17:19:57.564161+0000","mapping_epoch":115,"log_start":"151'9800","ondisk_log_start":"151'9800","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'18990","last_scrub_stamp":"2024-04-20T17:19:39.448230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:19:39.448230+0000","objects_scrubbed":1326,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:40.978614+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29753344,"num_objects":454,"num_object_clones":0,"num_object_copies":908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19862,"num_write_kb":650112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3],"acting":[11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"345'19464","reported_seq":20391,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-04-20T17:19:58.550855+0000","last_change":"2024-04-20T17:18:47.241194+0000","last_active":"2024-04-20T17:19:58.550855+0000","last_peered":"2024-04-20T17:19:58.550855+0000","last_clean":"2024-04-20T17:19:58.550855+0000","last_became_active":"2024-04-20T17:08:06.005515+0000","last_became_peered":"2024-04-20T17:08:06.005515+0000","last_unstale":"2024-04-20T17:19:58.550855+0000","last_undegraded":"2024-04-20T17:19:58.550855+0000","last_fullsized":"2024-04-20T17:19:58.550855+0000","mapping_epoch":115,"log_start":"148'9400","ondisk_log_start":"148'9400","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'17283","last_scrub_stamp":"2024-04-20T17:18:47.241064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:47.241064+0000","objects_scrubbed":2584,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:10.974140+0000","scrub_duration":266,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26214400,"num_objects":400,"num_object_clones":0,"num_object_copies":800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19464,"num_write_kb":635648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"345'19613","reported_seq":20541,"reported_epoch":345,"state":"active+clean","last_fresh":"2024-04-20T17:19:57.610289+0000","last_change":"2024-04-20T17:18:48.599315+0000","last_active":"2024-04-20T17:19:57.610289+0000","last_peered":"2024-04-20T17:19:57.610289+0000","last_clean":"2024-04-20T17:19:57.610289+0000","last_became_active":"2024-04-20T17:08:06.006691+0000","last_became_peered":"2024-04-20T17:08:06.006691+0000","last_unstale":"2024-04-20T17:19:57.610289+0000","last_undegraded":"2024-04-20T17:19:57.610289+0000","last_fullsized":"2024-04-20T17:19:57.610289+0000","mapping_epoch":115,"log_start":"148'9600","ondisk_log_start":"148'9600","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'17566","last_scrub_stamp":"2024-04-20T17:18:48.599209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:48.599209+0000","objects_scrubbed":2538,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:05.548836+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32047104,"num_objects":489,"num_object_clones":0,"num_object_copies":978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19613,"num_write_kb":643264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"344'19328","reported_seq":20253,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-04-20T17:19:55.779536+0000","last_change":"2024-04-20T17:18:52.214316+0000","last_active":"2024-04-20T17:19:55.779536+0000","last_peered":"2024-04-20T17:19:55.779536+0000","last_clean":"2024-04-20T17:19:55.779536+0000","last_became_active":"2024-04-20T17:08:06.044585+0000","last_became_peered":"2024-04-20T17:08:06.044585+0000","last_unstale":"2024-04-20T17:19:55.779536+0000","last_undegraded":"2024-04-20T17:19:55.779536+0000","last_fullsized":"2024-04-20T17:19:55.779536+0000","mapping_epoch":115,"log_start":"146'9300","ondisk_log_start":"146'9300","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'17470","last_scrub_stamp":"2024-04-20T17:18:52.214208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:18:52.214208+0000","objects_scrubbed":2403,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:19:55.920246+0000","scrub_duration":228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35455000,"num_objects":542,"num_object_clones":0,"num_object_copies":1084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":542,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":19328,"num_write_kb":635777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"344'19406","reported_seq":20228,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-04-20T17:19:55.926117+0000","last_change":"2024-04-20T17:19:47.732773+0000","last_active":"2024-04-20T17:19:55.926117+0000","last_peered":"2024-04-20T17:19:55.926117+0000","last_clean":"2024-04-20T17:19:55.926117+0000","last_became_active":"2024-04-20T17:12:23.515375+0000","last_became_peered":"2024-04-20T17:12:23.515375+0000","last_unstale":"2024-04-20T17:19:55.926117+0000","last_undegraded":"2024-04-20T17:19:55.926117+0000","last_fullsized":"2024-04-20T17:19:55.926117+0000","mapping_epoch":202,"log_start":"147'9400","ondisk_log_start":"147'9400","created":115,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'18971","last_scrub_stamp":"2024-04-20T17:19:47.732623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:08:04.990639+0000","last_clean_scrub_stamp":"2024-04-20T17:19:47.732623+0000","objects_scrubbed":1028,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:04.187243+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38666240,"num_objects":590,"num_object_clones":0,"num_object_copies":1180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19406,"num_write_kb":639872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":508166168,"num_objects":7755,"num_object_clones":0,"num_object_copies":15510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7755,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":314229,"num_write_kb":10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29216,"num_bytes_recovered":1914699776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2046697472,"data_stored":2046689328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},"log_size":160828,"ondisk_log_size":160828,"up":32,"acting":32,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991403,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":198564,"kb_used_data":99292,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93546588,"statfs":{"total":95995035648,"available":95791706112,"internally_reserved":0,"allocated":101675008,"data_stored":100420852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":101579204},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253041,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":273116,"kb_used_data":97556,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93472036,"statfs":{"total":95995035648,"available":95715364864,"internally_reserved":0,"allocated":99897344,"data_stored":98646740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":179763648},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612790,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":300180,"kb_used_data":160716,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93444972,"statfs":{"total":95995035648,"available":95687651328,"internally_reserved":0,"allocated":164573184,"data_stored":163324506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":142801344},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809020,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":215516,"kb_used_data":112148,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93529636,"statfs":{"total":95995035648,"available":95774347264,"internally_reserved":0,"allocated":114839552,"data_stored":113588968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168769,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":202408,"kb_used_data":99036,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93542744,"statfs":{"total":95995035648,"available":95787769856,"internally_reserved":0,"allocated":101412864,"data_stored":100158728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":105839050},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430405,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":173856,"kb_used_data":66324,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93571296,"statfs":{"total":95995035648,"available":95817007104,"internally_reserved":0,"allocated":67915776,"data_stored":66665192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575071,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":169248,"kb_used_data":70036,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93575904,"statfs":{"total":95995035648,"available":95821725696,"internally_reserved":0,"allocated":71716864,"data_stored":70466280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738596,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":290972,"kb_used_data":111252,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93454180,"statfs":{"total":95995035648,"available":95697080320,"internally_reserved":0,"allocated":113922048,"data_stored":112671464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393884,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":514960,"kb_used_data":470728,"kb_used_omap":202,"kb_used_meta":44021,"kb_avail":93230192,"statfs":{"total":95995035648,"available":95467716608,"internally_reserved":0,"allocated":482025472,"data_stored":480780866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":45077975},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182612,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":148176,"kb_used_data":80904,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93596976,"statfs":{"total":95995035648,"available":95843303424,"internally_reserved":0,"allocated":82845696,"data_stored":81601090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986383,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80020,"kb_used_data":40524,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93665132,"statfs":{"total":95995035648,"available":95913095168,"internally_reserved":0,"allocated":41496576,"data_stored":40247898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790154,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":787536,"kb_used_data":688328,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92957616,"statfs":{"total":95995035648,"available":95188598784,"internally_reserved":0,"allocated":704847872,"data_stored":703603266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":96534528,"data_stored":96534528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":66584576,"data_stored":66584576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":478871552,"data_stored":478871552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202991,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":76611584,"data_stored":76611584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36376576,"data_stored":36372504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":699072512,"data_stored":699072512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":61210624,"data_stored":61210624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":88735744,"data_stored":88735744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":103743488,"data_stored":103743488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":151719936,"data_stored":151715864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":94437376,"data_stored":94437376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":92798976,"data_stored":92798976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:20:01.655 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:20:01.676 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:20:01.730 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:01.758 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:20:01.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:01.765 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:20:01.866 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:20:01.905 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:01.928 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:20:01.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:01.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:20:02.032 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:20:02.072 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:02.102 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:20:02.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:02.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:20:02.212 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:20:02.246 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:02.603 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:20:02.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:02.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:20:02.662 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:20:02.662 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:20:02.707 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:20:02.739 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:02.771 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:20:02.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:02.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:20:02.875 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:20:02.908 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:02.911 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:20:03.275 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:20:03.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:03.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:20:03.313 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:20:03.380 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:20:03.415 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:03.440 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:03 smithi106 ceph-mon[44423]: pgmap v954: 17 pgs: 17 active+clean; 350 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 880 op/s; 2.1 MiB/s, 33 objects/s recovering 2024-04-20T17:20:03.473 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-20T17:20:03.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:03 smithi171 ceph-mon[44189]: pgmap v954: 17 pgs: 17 active+clean; 350 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 880 op/s; 2.1 MiB/s, 33 objects/s recovering 2024-04-20T17:20:03.593 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:03 smithi005 ceph-mon[39562]: pgmap v954: 17 pgs: 17 active+clean; 350 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 880 op/s; 2.1 MiB/s, 33 objects/s recovering 2024-04-20T17:20:03.801 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:20:03.876 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:20:03.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:03.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:20:03.983 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:20:04.012 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:04.070 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:20:04.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:04.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:20:04.098 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Removed 160991 objects 2024-04-20T17:20:04.098 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Clean up completed and total clean up time :616.644 2024-04-20T17:20:04.140 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-04-20T17:20:04.141 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-04-20T17:20:04.174 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:20:04.214 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:04.274 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:20:04.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:04.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:20:04.378 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:20:04.406 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:04.602 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:20:04.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:04.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:20:04.651 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:20:04.706 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:20:04.733 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:04.806 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:20:04.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:04.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:20:04.912 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:20:04.941 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:04.961 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:20:04.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:04.964 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:20:05.065 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:20:05.092 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:05.110 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:20:05.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:05.113 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:20:05.214 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:20:05.241 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:05.243 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:20:05.260 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:20:05.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:05.263 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:20:05.366 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:20:05.393 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:05.487 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:05 smithi106 ceph-mon[44423]: pgmap v955: 17 pgs: 17 active+clean; 127 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 2.1 MiB/s, 33 objects/s recovering 2024-04-20T17:20:05.487 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:05 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/686222554' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:20:05.488 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:05 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:20:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:05 smithi171 ceph-mon[44189]: pgmap v955: 17 pgs: 17 active+clean; 127 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 2.1 MiB/s, 33 objects/s recovering 2024-04-20T17:20:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:05 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/686222554' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:20:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:05 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:20:05.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:05 smithi005 ceph-mon[39562]: pgmap v955: 17 pgs: 17 active+clean; 127 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 2.1 MiB/s, 33 objects/s recovering 2024-04-20T17:20:05.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:05 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/686222554' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:20:05.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:05 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:20:05.754 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:20:05.754 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:20:05.943 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:20:05.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:05.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:20:06.049 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:20:06.078 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:06.098 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:20:06.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:06.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:20:06.203 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:20:06.234 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:06.255 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:20:06.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:06.259 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:20:06.360 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:20:06.390 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:06.410 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:20:06.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:06.413 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:20:06.429 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:06 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:20:06.429 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:06 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/686222554' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:20:06.429 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:06 smithi106 ceph-mon[44423]: osdmap e347: 12 total, 12 up, 12 in 2024-04-20T17:20:06.430 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:06 smithi106 ceph-mon[44423]: osdmap e348: 12 total, 12 up, 12 in 2024-04-20T17:20:06.455 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-20T17:20:06.514 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:20:06.533 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:06 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:20:06.533 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:06 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/686222554' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:20:06.533 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:06 smithi005 ceph-mon[39562]: osdmap e347: 12 total, 12 up, 12 in 2024-04-20T17:20:06.533 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:06 smithi005 ceph-mon[39562]: osdmap e348: 12 total, 12 up, 12 in 2024-04-20T17:20:06.550 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:06.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:06 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:20:06.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:06 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/686222554' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:20:06.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:06 smithi171 ceph-mon[44189]: osdmap e347: 12 total, 12 up, 12 in 2024-04-20T17:20:06.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:06 smithi171 ceph-mon[44189]: osdmap e348: 12 total, 12 up, 12 in 2024-04-20T17:20:06.714 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:20:06.804 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:20:06.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:06.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:20:06.909 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:20:06.941 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:06.961 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:20:06.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:06.964 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:20:07.065 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:20:07.102 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:07.456 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:20:07.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:07 smithi005 ceph-mon[39562]: pgmap v958: 17 pgs: 17 active+clean; 55 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:20:07.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:07.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:20:07.474 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:20:07.493 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:20:07.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:07 smithi106 ceph-mon[44423]: pgmap v958: 17 pgs: 17 active+clean; 55 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:20:07.512 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:20:07.532 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:20:07.553 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:20:07.564 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:20:07.576 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:20:07.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:07 smithi171 ceph-mon[44189]: pgmap v958: 17 pgs: 17 active+clean; 55 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:20:07.593 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:07.597 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:20:07.620 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:20:07.640 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:20:07.660 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:20:07.681 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:20:07.701 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:20:07.702 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:20:07.702 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:20:07.703 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:20:07.703 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:20:07.703 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:20:07.703 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:20:07.703 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:20:07.703 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:20:07.703 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:20:07.703 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:20:07.703 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:20:07.703 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:20:08.276 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:20:08.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:08 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1751375472' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:20:08.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:08 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:20:08.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:08 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:20:08.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:08 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1751375472' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:20:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:08 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:20:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:08 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:20:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.522 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:20:08.522 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:20:08.522 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-04-20T17:20:08.522 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-04-20T17:20:08.522 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:20:08.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:08 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1751375472' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:20:08.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:08 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:20:08.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:08 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:20:08.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:08.644 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:20:08.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:08.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:20:08.748 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:20:08.780 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:08.801 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:20:08.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:08.805 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:20:08.905 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:20:08.933 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:09.185 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:20:09.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:09.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:20:09.203 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-20T17:20:09.288 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:20:09.318 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:09.321 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:20:09.501 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:20:09.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:09 smithi106 ceph-mon[44423]: pgmap v959: 17 pgs: 17 active+clean; 7.6 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 821 op/s 2024-04-20T17:20:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:09 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:20:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:09 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:20:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:09 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:20:09.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:09.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:09.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:09 smithi171 ceph-mon[44189]: pgmap v959: 17 pgs: 17 active+clean; 7.6 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 821 op/s 2024-04-20T17:20:09.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:09 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:20:09.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:09 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:20:09.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:09 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:20:09.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:09.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:09.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:09 smithi005 ceph-mon[39562]: pgmap v959: 17 pgs: 17 active+clean; 7.6 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 821 op/s 2024-04-20T17:20:09.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:09 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:20:09.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:09 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:20:09.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:09 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:20:10.097 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:20:10.169 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:20:10.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:10.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:20:10.273 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:20:10.301 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:10.430 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:20:10.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:10 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:20:10.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:10 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:20:10.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:10.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:20:10.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:10 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:20:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:10 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:20:10.535 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:20:10.563 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:10 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:20:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:10 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:20:11.111 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:20:11.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:11.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:20:11.215 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:20:11.242 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:20:11.244 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:11.264 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:20:11.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:11.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:20:11.368 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:20:11.388 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:20:11.388 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-04-20T17:20:11.388 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-04-20T17:20:11.388 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:20:11.388 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.388 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:196122 2.9 2:933c82c9:::benchmark_data_smithi171_78918_object35128:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165)", 2024-04-20T17:20:11.388 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:32.895950+0000", 2024-04-20T17:20:11.388 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 578.490411634, 2024-04-20T17:20:11.389 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.167013678, 2024-04-20T17:20:11.389 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:20:11.389 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:20:11.389 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:20:11.389 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:20:11.389 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:20:11.389 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:20:11.389 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 196122 2024-04-20T17:20:11.389 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.389 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:20:11.389 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.389 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:20:11.389 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.895950+0000", 2024-04-20T17:20:11.389 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.390 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.390 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.390 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:20:11.390 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.895950+0000", 2024-04-20T17:20:11.390 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.390 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.390 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.390 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:20:11.390 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.895951+0000", 2024-04-20T17:20:11.390 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5000000000000001e-07 2024-04-20T17:20:11.390 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.390 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.390 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:20:11.391 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.895954+0000", 2024-04-20T17:20:11.391 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1669999999999999e-06 2024-04-20T17:20:11.391 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.391 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.391 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:20:11.391 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.895970+0000", 2024-04-20T17:20:11.391 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5940999999999999e-05 2024-04-20T17:20:11.391 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.391 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.391 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:20:11.391 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.895975+0000", 2024-04-20T17:20:11.391 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4100000000000001e-06 2024-04-20T17:20:11.391 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.392 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.392 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:20:11.392 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896153+0000", 2024-04-20T17:20:11.392 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00017863500000000001 2024-04-20T17:20:11.392 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.392 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.392 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:20:11.392 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896162+0000", 2024-04-20T17:20:11.392 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7839999999999992e-06 2024-04-20T17:20:11.392 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.392 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.392 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:20:11.392 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.028626+0000", 2024-04-20T17:20:11.392 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.132463993 2024-04-20T17:20:11.393 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.393 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.393 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:20:11.393 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.028695+0000", 2024-04-20T17:20:11.393 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8398999999999993e-05 2024-04-20T17:20:11.393 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.393 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.393 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,10", 2024-04-20T17:20:11.393 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.028768+0000", 2024-04-20T17:20:11.393 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3794000000000005e-05 2024-04-20T17:20:11.393 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.393 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.393 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:20:11.393 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.031138+0000", 2024-04-20T17:20:11.394 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002370002 2024-04-20T17:20:11.394 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.394 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.394 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:11.394 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.037438+0000", 2024-04-20T17:20:11.394 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.006300089 2024-04-20T17:20:11.394 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.394 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.394 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:11.394 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.062845+0000", 2024-04-20T17:20:11.394 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.025406423000000001 2024-04-20T17:20:11.394 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.394 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.395 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:20:11.395 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.062885+0000", 2024-04-20T17:20:11.395 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0365999999999999e-05 2024-04-20T17:20:11.395 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.395 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.395 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:20:11.395 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.062964+0000", 2024-04-20T17:20:11.395 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8825000000000005e-05 2024-04-20T17:20:11.395 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.395 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:20:11.395 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.395 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.395 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.395 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:196127 2.9 2:989d3d5e:::benchmark_data_smithi171_78918_object35133:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165)", 2024-04-20T17:20:11.396 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:32.895984+0000", 2024-04-20T17:20:11.396 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 578.49037775399995, 2024-04-20T17:20:11.396 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.167111072, 2024-04-20T17:20:11.396 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:20:11.396 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:20:11.396 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:20:11.396 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:20:11.396 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:20:11.396 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:20:11.396 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 196127 2024-04-20T17:20:11.396 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.396 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:20:11.396 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.396 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:20:11.396 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.895984+0000", 2024-04-20T17:20:11.397 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.397 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.397 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.397 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:20:11.397 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.895984+0000", 2024-04-20T17:20:11.397 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.397 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.397 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.397 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:20:11.397 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.895985+0000", 2024-04-20T17:20:11.397 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7100000000000001e-07 2024-04-20T17:20:11.397 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.397 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.397 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:20:11.398 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.895988+0000", 2024-04-20T17:20:11.398 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.779e-06 2024-04-20T17:20:11.398 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.398 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.398 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:20:11.398 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.895995+0000", 2024-04-20T17:20:11.398 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9779999999999999e-06 2024-04-20T17:20:11.398 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.398 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.398 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:20:11.398 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896001+0000", 2024-04-20T17:20:11.398 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3210000000000001e-06 2024-04-20T17:20:11.398 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.398 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.398 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:20:11.399 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896187+0000", 2024-04-20T17:20:11.399 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000186085 2024-04-20T17:20:11.399 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.399 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.399 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:20:11.399 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896200+0000", 2024-04-20T17:20:11.399 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3358e-05 2024-04-20T17:20:11.399 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.399 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.399 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:20:11.399 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.031176+0000", 2024-04-20T17:20:11.399 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13497598299999999 2024-04-20T17:20:11.399 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.400 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.400 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:20:11.400 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.031291+0000", 2024-04-20T17:20:11.400 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011480399999999999 2024-04-20T17:20:11.400 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.400 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.400 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,10", 2024-04-20T17:20:11.400 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.031405+0000", 2024-04-20T17:20:11.400 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0001142 2024-04-20T17:20:11.400 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.400 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.400 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:20:11.400 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.037407+0000", 2024-04-20T17:20:11.400 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.006001246 2024-04-20T17:20:11.401 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.401 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.401 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:11.401 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.047687+0000", 2024-04-20T17:20:11.401 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010279806000000001 2024-04-20T17:20:11.401 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.401 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.401 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:11.401 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.062994+0000", 2024-04-20T17:20:11.401 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015307359 2024-04-20T17:20:11.401 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.401 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.401 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:20:11.401 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.063038+0000", 2024-04-20T17:20:11.401 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4335999999999997e-05 2024-04-20T17:20:11.402 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.402 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.402 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:20:11.402 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.063095+0000", 2024-04-20T17:20:11.402 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7145999999999998e-05 2024-04-20T17:20:11.402 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.402 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:20:11.402 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.402 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.402 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.402 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:196138 2.9 2:98f63a91:::benchmark_data_smithi171_78918_object35144:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165)", 2024-04-20T17:20:11.402 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:32.896007+0000", 2024-04-20T17:20:11.402 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 578.49035492600001, 2024-04-20T17:20:11.402 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.16730315900000001, 2024-04-20T17:20:11.403 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:20:11.403 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:20:11.403 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:20:11.403 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:20:11.403 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:20:11.403 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:20:11.403 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 196138 2024-04-20T17:20:11.403 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.403 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:20:11.403 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.403 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:20:11.403 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896007+0000", 2024-04-20T17:20:11.403 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.403 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.404 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.404 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:20:11.404 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896007+0000", 2024-04-20T17:20:11.404 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.404 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.404 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.404 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:20:11.404 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896008+0000", 2024-04-20T17:20:11.404 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7000000000000005e-07 2024-04-20T17:20:11.404 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.404 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.404 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:20:11.404 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896010+0000", 2024-04-20T17:20:11.405 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.261e-06 2024-04-20T17:20:11.405 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.405 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.405 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:20:11.405 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896015+0000", 2024-04-20T17:20:11.405 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9679999999999997e-06 2024-04-20T17:20:11.405 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.405 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.405 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:20:11.405 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896020+0000", 2024-04-20T17:20:11.405 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6439999999999999e-06 2024-04-20T17:20:11.405 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.405 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.405 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:20:11.406 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896222+0000", 2024-04-20T17:20:11.406 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00020292300000000001 2024-04-20T17:20:11.406 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.406 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.406 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:20:11.406 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896226+0000", 2024-04-20T17:20:11.406 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7009999999999998e-06 2024-04-20T17:20:11.406 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.406 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.406 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:20:11.406 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.034517+0000", 2024-04-20T17:20:11.406 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13829066100000001 2024-04-20T17:20:11.406 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.407 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.407 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:20:11.407 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.034595+0000", 2024-04-20T17:20:11.407 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7904000000000005e-05 2024-04-20T17:20:11.407 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.407 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.407 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,10", 2024-04-20T17:20:11.407 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.034674+0000", 2024-04-20T17:20:11.407 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9190999999999994e-05 2024-04-20T17:20:11.407 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.407 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.407 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:20:11.407 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.047647+0000", 2024-04-20T17:20:11.407 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012973531 2024-04-20T17:20:11.408 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.408 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.408 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:11.408 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.047728+0000", 2024-04-20T17:20:11.408 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0043000000000006e-05 2024-04-20T17:20:11.408 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.408 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.408 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:11.408 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.063145+0000", 2024-04-20T17:20:11.408 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015417481 2024-04-20T17:20:11.408 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.408 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.408 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:20:11.408 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.063205+0000", 2024-04-20T17:20:11.408 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0275999999999997e-05 2024-04-20T17:20:11.409 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.409 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.409 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:20:11.409 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.063310+0000", 2024-04-20T17:20:11.409 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000105005 2024-04-20T17:20:11.409 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.409 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:20:11.409 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.409 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.409 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.409 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:196154 2.9 2:93e7ef58:::benchmark_data_smithi171_78918_object35160:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165)", 2024-04-20T17:20:11.409 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:32.896058+0000", 2024-04-20T17:20:11.409 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 578.49030370699995, 2024-04-20T17:20:11.409 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.16739100000000001, 2024-04-20T17:20:11.409 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:20:11.410 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:20:11.410 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:20:11.410 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:20:11.410 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:20:11.410 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:20:11.410 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 196154 2024-04-20T17:20:11.410 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.410 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:20:11.410 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.410 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:20:11.410 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896058+0000", 2024-04-20T17:20:11.410 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.411 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.411 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.411 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:20:11.411 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896058+0000", 2024-04-20T17:20:11.411 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.411 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.411 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.411 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:20:11.411 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896059+0000", 2024-04-20T17:20:11.411 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4099999999999998e-07 2024-04-20T17:20:11.411 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.411 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.411 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:20:11.411 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896062+0000", 2024-04-20T17:20:11.412 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.971e-06 2024-04-20T17:20:11.412 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.412 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.412 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:20:11.412 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896070+0000", 2024-04-20T17:20:11.412 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2209999999999997e-06 2024-04-20T17:20:11.412 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.412 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.412 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:20:11.412 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896077+0000", 2024-04-20T17:20:11.412 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3910000000000003e-06 2024-04-20T17:20:11.412 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.412 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.412 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:20:11.412 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896243+0000", 2024-04-20T17:20:11.413 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000166071 2024-04-20T17:20:11.413 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.413 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.413 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:20:11.413 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896246+0000", 2024-04-20T17:20:11.413 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3890000000000001e-06 2024-04-20T17:20:11.413 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.413 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.413 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:20:11.413 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.037468+0000", 2024-04-20T17:20:11.413 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14122193 2024-04-20T17:20:11.413 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.413 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.414 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:20:11.414 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.037543+0000", 2024-04-20T17:20:11.414 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4926999999999996e-05 2024-04-20T17:20:11.414 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.414 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.414 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,10", 2024-04-20T17:20:11.414 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.037620+0000", 2024-04-20T17:20:11.414 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7600000000000002e-05 2024-04-20T17:20:11.414 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.414 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.414 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:20:11.414 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.047739+0000", 2024-04-20T17:20:11.414 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010118257 2024-04-20T17:20:11.414 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.414 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.415 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:11.415 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.047781+0000", 2024-04-20T17:20:11.415 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2079000000000001e-05 2024-04-20T17:20:11.415 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.415 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.415 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:11.415 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.063360+0000", 2024-04-20T17:20:11.415 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015579025 2024-04-20T17:20:11.415 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.415 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.415 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:20:11.415 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.063394+0000", 2024-04-20T17:20:11.415 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4093e-05 2024-04-20T17:20:11.415 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.415 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.416 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:20:11.416 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.063449+0000", 2024-04-20T17:20:11.416 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5405000000000002e-05 2024-04-20T17:20:11.416 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.416 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:20:11.416 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.416 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.416 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.416 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:196168 2.9 2:9c851961:::benchmark_data_smithi171_78918_object35174:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165)", 2024-04-20T17:20:11.416 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:10:32.896084+0000", 2024-04-20T17:20:11.416 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 578.49027773499995, 2024-04-20T17:20:11.416 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.16757227899999999, 2024-04-20T17:20:11.416 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:20:11.417 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:20:11.417 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:20:11.417 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:20:11.417 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:20:11.417 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:20:11.417 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 196168 2024-04-20T17:20:11.417 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.417 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:20:11.417 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.417 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:20:11.417 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896084+0000", 2024-04-20T17:20:11.417 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.417 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.417 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.418 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:20:11.418 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896084+0000", 2024-04-20T17:20:11.418 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.418 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.418 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.418 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:20:11.418 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896085+0000", 2024-04-20T17:20:11.418 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2499999999999995e-07 2024-04-20T17:20:11.418 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.418 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.418 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:20:11.418 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896087+0000", 2024-04-20T17:20:11.418 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.351e-06 2024-04-20T17:20:11.419 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.419 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.419 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:20:11.419 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896093+0000", 2024-04-20T17:20:11.419 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4020000000000001e-06 2024-04-20T17:20:11.419 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.419 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.419 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:20:11.419 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896097+0000", 2024-04-20T17:20:11.419 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8600000000000001e-06 2024-04-20T17:20:11.419 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.419 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.419 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:20:11.419 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896265+0000", 2024-04-20T17:20:11.420 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000167267 2024-04-20T17:20:11.420 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.420 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.420 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:20:11.420 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:32.896269+0000", 2024-04-20T17:20:11.420 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3490000000000002e-06 2024-04-20T17:20:11.420 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.420 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.420 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:20:11.420 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.040193+0000", 2024-04-20T17:20:11.420 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14392408100000001 2024-04-20T17:20:11.420 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.420 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.421 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:20:11.421 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.040286+0000", 2024-04-20T17:20:11.421 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2927e-05 2024-04-20T17:20:11.421 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.421 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.421 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,10", 2024-04-20T17:20:11.421 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.040384+0000", 2024-04-20T17:20:11.421 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8128000000000006e-05 2024-04-20T17:20:11.421 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.421 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.421 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:20:11.421 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.047746+0000", 2024-04-20T17:20:11.421 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0073614850000000001 2024-04-20T17:20:11.421 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.421 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.422 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:11.422 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.047820+0000", 2024-04-20T17:20:11.422 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3875999999999995e-05 2024-04-20T17:20:11.422 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.422 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.422 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:11.422 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.063498+0000", 2024-04-20T17:20:11.422 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015678820999999999 2024-04-20T17:20:11.422 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.422 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.422 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:20:11.422 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.063556+0000", 2024-04-20T17:20:11.422 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7506000000000001e-05 2024-04-20T17:20:11.422 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.423 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.423 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:20:11.423 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:10:33.063657+0000", 2024-04-20T17:20:11.423 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000100701 2024-04-20T17:20:11.423 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.423 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:20:11.423 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.423 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.423 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.423 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:296439 2.0 2:0c3ed833:::benchmark_data_smithi171_78918_object135445:head [delete] snapc 0=[] ondisk+write+known_if_redirected e338)", 2024-04-20T17:20:11.423 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:19:31.268219+0000", 2024-04-20T17:20:11.423 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 40.118143238999998, 2024-04-20T17:20:11.423 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21360322700000001, 2024-04-20T17:20:11.423 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:20:11.424 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:20:11.424 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:20:11.424 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:20:11.424 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:20:11.424 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:20:11.424 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 296439 2024-04-20T17:20:11.424 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.424 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:20:11.424 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.424 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:20:11.424 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.268219+0000", 2024-04-20T17:20:11.424 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.424 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.424 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.424 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:20:11.425 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.268219+0000", 2024-04-20T17:20:11.425 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.425 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.425 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.425 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:20:11.425 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.268220+0000", 2024-04-20T17:20:11.425 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.319e-06 2024-04-20T17:20:11.425 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.425 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.425 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:20:11.425 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.268224+0000", 2024-04-20T17:20:11.425 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1110000000000001e-06 2024-04-20T17:20:11.425 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.425 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.426 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:20:11.426 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.268234+0000", 2024-04-20T17:20:11.426 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0020000000000001e-05 2024-04-20T17:20:11.426 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.426 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.426 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:20:11.426 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.268240+0000", 2024-04-20T17:20:11.426 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2160000000000003e-06 2024-04-20T17:20:11.426 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.426 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.426 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:20:11.426 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.269497+0000", 2024-04-20T17:20:11.426 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0012564939999999999 2024-04-20T17:20:11.426 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.427 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.427 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:20:11.427 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.269574+0000", 2024-04-20T17:20:11.427 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6921000000000004e-05 2024-04-20T17:20:11.427 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.427 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.427 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:20:11.427 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.269666+0000", 2024-04-20T17:20:11.427 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2298000000000003e-05 2024-04-20T17:20:11.427 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.427 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.427 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:20:11.427 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.272721+0000", 2024-04-20T17:20:11.427 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003054611 2024-04-20T17:20:11.428 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.428 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.428 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:11.428 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.481756+0000", 2024-04-20T17:20:11.428 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.209035421 2024-04-20T17:20:11.428 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.428 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.428 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:20:11.429 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.481792+0000", 2024-04-20T17:20:11.429 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5277999999999997e-05 2024-04-20T17:20:11.429 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.429 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.429 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:20:11.429 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.481822+0000", 2024-04-20T17:20:11.429 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0537999999999998e-05 2024-04-20T17:20:11.429 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.429 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:20:11.429 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.429 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.429 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.429 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:296446 2.9 2:99908dcc:::benchmark_data_smithi171_78918_object135452:head [delete] snapc 0=[] ondisk+write+known_if_redirected e338)", 2024-04-20T17:20:11.429 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:19:31.274054+0000", 2024-04-20T17:20:11.430 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 40.112307989000001, 2024-04-20T17:20:11.430 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20894923900000001, 2024-04-20T17:20:11.430 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:20:11.430 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:20:11.430 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:20:11.430 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:20:11.430 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:20:11.430 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:20:11.430 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 296446 2024-04-20T17:20:11.430 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.430 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:20:11.430 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.430 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:20:11.430 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.274054+0000", 2024-04-20T17:20:11.430 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.431 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.431 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.431 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:20:11.431 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.274054+0000", 2024-04-20T17:20:11.431 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.431 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.431 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.431 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:20:11.431 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.274058+0000", 2024-04-20T17:20:11.431 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8829999999999999e-06 2024-04-20T17:20:11.431 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.431 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.431 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:20:11.431 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.274070+0000", 2024-04-20T17:20:11.432 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2257e-05 2024-04-20T17:20:11.432 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.432 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.432 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:20:11.432 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.274102+0000", 2024-04-20T17:20:11.432 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2218999999999998e-05 2024-04-20T17:20:11.432 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.432 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.432 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:20:11.432 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.274124+0000", 2024-04-20T17:20:11.432 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1798999999999999e-05 2024-04-20T17:20:11.432 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.432 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.432 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:20:11.432 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.274237+0000", 2024-04-20T17:20:11.433 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000112489 2024-04-20T17:20:11.433 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.433 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.433 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:20:11.433 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.274353+0000", 2024-04-20T17:20:11.433 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011646600000000001 2024-04-20T17:20:11.433 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.433 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.433 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:20:11.433 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.274483+0000", 2024-04-20T17:20:11.433 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013020999999999999 2024-04-20T17:20:11.433 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.433 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.433 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:20:11.434 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.278114+0000", 2024-04-20T17:20:11.434 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003631085 2024-04-20T17:20:11.434 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.434 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.434 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:11.434 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.482903+0000", 2024-04-20T17:20:11.434 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20478887500000001 2024-04-20T17:20:11.434 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.434 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.434 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:20:11.434 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.482958+0000", 2024-04-20T17:20:11.434 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4292000000000001e-05 2024-04-20T17:20:11.434 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.434 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.435 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:20:11.435 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.483003+0000", 2024-04-20T17:20:11.435 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5664000000000002e-05 2024-04-20T17:20:11.435 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.435 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:20:11.435 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.435 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.435 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.435 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:296469 2.9 2:90451252:::benchmark_data_smithi171_78918_object135475:head [delete] snapc 0=[] ondisk+write+known_if_redirected e338)", 2024-04-20T17:20:11.435 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:19:31.309320+0000", 2024-04-20T17:20:11.435 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 40.077041727999998, 2024-04-20T17:20:11.435 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.173782625, 2024-04-20T17:20:11.435 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:20:11.435 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:20:11.436 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:20:11.436 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:20:11.436 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:20:11.436 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:20:11.436 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 296469 2024-04-20T17:20:11.436 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.436 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:20:11.436 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.436 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:20:11.436 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.309320+0000", 2024-04-20T17:20:11.436 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.436 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.436 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.437 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:20:11.437 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.309320+0000", 2024-04-20T17:20:11.437 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.437 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.437 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.437 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:20:11.437 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.309324+0000", 2024-04-20T17:20:11.437 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3730000000000001e-06 2024-04-20T17:20:11.437 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.437 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.437 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:20:11.437 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.309352+0000", 2024-04-20T17:20:11.437 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7936000000000002e-05 2024-04-20T17:20:11.437 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.438 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.438 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:20:11.438 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.309377+0000", 2024-04-20T17:20:11.438 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5545999999999999e-05 2024-04-20T17:20:11.438 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.438 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.438 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:20:11.438 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.309392+0000", 2024-04-20T17:20:11.438 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5006e-05 2024-04-20T17:20:11.438 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.438 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.439 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:20:11.439 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.309537+0000", 2024-04-20T17:20:11.439 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014458099999999999 2024-04-20T17:20:11.439 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.439 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.439 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:20:11.439 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.309761+0000", 2024-04-20T17:20:11.439 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00022380999999999999 2024-04-20T17:20:11.439 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.439 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.439 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:20:11.439 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.309894+0000", 2024-04-20T17:20:11.439 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013309800000000001 2024-04-20T17:20:11.440 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.440 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.440 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:20:11.440 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.317804+0000", 2024-04-20T17:20:11.440 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0079105890000000009 2024-04-20T17:20:11.440 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.440 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.440 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:11.440 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.483047+0000", 2024-04-20T17:20:11.440 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.165242948 2024-04-20T17:20:11.440 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.440 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.440 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:20:11.440 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.483078+0000", 2024-04-20T17:20:11.440 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0616e-05 2024-04-20T17:20:11.440 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.441 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.441 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:20:11.441 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.483103+0000", 2024-04-20T17:20:11.441 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5122000000000001e-05 2024-04-20T17:20:11.441 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.441 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:20:11.441 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.441 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.441 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.441 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:296470 2.9 2:94565716:::benchmark_data_smithi171_78918_object135476:head [delete] snapc 0=[] ondisk+write+known_if_redirected e338)", 2024-04-20T17:20:11.441 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:19:31.309425+0000", 2024-04-20T17:20:11.441 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 40.076937119999997, 2024-04-20T17:20:11.441 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.173793381, 2024-04-20T17:20:11.441 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:20:11.441 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:20:11.442 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:20:11.442 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:20:11.442 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:20:11.442 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:20:11.442 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 296470 2024-04-20T17:20:11.442 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.442 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:20:11.442 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.442 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:20:11.442 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.309425+0000", 2024-04-20T17:20:11.442 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.442 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.442 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.442 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:20:11.442 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.309425+0000", 2024-04-20T17:20:11.443 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.443 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.443 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.443 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:20:11.443 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.309426+0000", 2024-04-20T17:20:11.443 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3629999999999999e-06 2024-04-20T17:20:11.443 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.443 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.443 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:20:11.443 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.309431+0000", 2024-04-20T17:20:11.443 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9930000000000003e-06 2024-04-20T17:20:11.443 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.443 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.443 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:20:11.443 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.309442+0000", 2024-04-20T17:20:11.444 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.025e-05 2024-04-20T17:20:11.444 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.444 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.444 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:20:11.444 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.309450+0000", 2024-04-20T17:20:11.444 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4530000000000002e-06 2024-04-20T17:20:11.444 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.444 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.444 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:20:11.444 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.313377+0000", 2024-04-20T17:20:11.444 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003926756 2024-04-20T17:20:11.444 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.444 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.444 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:20:11.445 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.313489+0000", 2024-04-20T17:20:11.445 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000111894 2024-04-20T17:20:11.445 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.445 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.445 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:20:11.445 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.313604+0000", 2024-04-20T17:20:11.445 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000115796 2024-04-20T17:20:11.445 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.445 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.445 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:20:11.445 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.317960+0000", 2024-04-20T17:20:11.445 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.004355683 2024-04-20T17:20:11.445 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.445 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.446 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:11.446 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.483146+0000", 2024-04-20T17:20:11.446 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.16518603800000001 2024-04-20T17:20:11.446 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.446 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.446 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:20:11.446 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.483192+0000", 2024-04-20T17:20:11.446 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5475000000000001e-05 2024-04-20T17:20:11.446 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.446 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.446 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:20:11.446 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.483218+0000", 2024-04-20T17:20:11.446 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6679999999999999e-05 2024-04-20T17:20:11.447 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.447 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:20:11.447 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.447 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.447 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.447 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:296473 2.0 2:0e4d736e:::benchmark_data_smithi171_78918_object135479:head [delete] snapc 0=[] ondisk+write+known_if_redirected e338)", 2024-04-20T17:20:11.447 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:19:31.316896+0000", 2024-04-20T17:20:11.447 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 40.069465942999997, 2024-04-20T17:20:11.447 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.16501105999999999, 2024-04-20T17:20:11.447 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:20:11.447 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:20:11.447 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:20:11.447 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:20:11.447 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:20:11.447 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:20:11.448 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 296473 2024-04-20T17:20:11.448 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.448 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:20:11.448 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.448 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:20:11.448 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.316896+0000", 2024-04-20T17:20:11.448 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.448 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.448 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.448 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:20:11.448 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.316896+0000", 2024-04-20T17:20:11.448 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.448 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.448 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.448 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:20:11.449 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.316898+0000", 2024-04-20T17:20:11.449 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.097e-06 2024-04-20T17:20:11.449 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.449 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.449 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:20:11.449 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.316905+0000", 2024-04-20T17:20:11.449 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6220000000000003e-06 2024-04-20T17:20:11.449 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.449 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.449 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:20:11.449 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.316917+0000", 2024-04-20T17:20:11.449 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1905999999999999e-05 2024-04-20T17:20:11.449 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.450 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.450 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:20:11.450 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.316924+0000", 2024-04-20T17:20:11.450 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4939999999999997e-06 2024-04-20T17:20:11.450 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.450 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.450 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:20:11.450 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.316987+0000", 2024-04-20T17:20:11.450 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3245999999999997e-05 2024-04-20T17:20:11.450 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.450 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.450 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:20:11.450 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.317072+0000", 2024-04-20T17:20:11.451 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4202000000000005e-05 2024-04-20T17:20:11.451 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.451 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.451 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:20:11.451 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.317144+0000", 2024-04-20T17:20:11.451 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2255999999999994e-05 2024-04-20T17:20:11.451 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.451 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.451 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:20:11.451 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.320859+0000", 2024-04-20T17:20:11.451 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037154559999999998 2024-04-20T17:20:11.451 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.451 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.451 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:11.451 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.481853+0000", 2024-04-20T17:20:11.452 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.16099369299999999 2024-04-20T17:20:11.452 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.452 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.452 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:20:11.452 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.481883+0000", 2024-04-20T17:20:11.452 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9910999999999999e-05 2024-04-20T17:20:11.452 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.452 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.452 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:20:11.452 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.481907+0000", 2024-04-20T17:20:11.452 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4176999999999999e-05 2024-04-20T17:20:11.452 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.453 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:20:11.453 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.453 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.453 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.453 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:296477 2.0 2:02f5a685:::benchmark_data_smithi171_78918_object135483:head [delete] snapc 0=[] ondisk+write+known_if_redirected e338)", 2024-04-20T17:20:11.453 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:19:31.330353+0000", 2024-04-20T17:20:11.453 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 40.056009062000001, 2024-04-20T17:20:11.453 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.15456261299999999, 2024-04-20T17:20:11.453 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:20:11.453 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:20:11.453 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:20:11.453 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:20:11.453 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:20:11.453 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:20:11.454 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 296477 2024-04-20T17:20:11.454 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.454 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:20:11.454 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.454 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:20:11.454 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.330353+0000", 2024-04-20T17:20:11.454 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.454 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.454 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.454 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:20:11.454 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.330353+0000", 2024-04-20T17:20:11.454 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.454 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.454 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.455 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:20:11.455 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.330354+0000", 2024-04-20T17:20:11.455 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2780000000000001e-06 2024-04-20T17:20:11.455 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.455 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.455 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:20:11.455 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.330359+0000", 2024-04-20T17:20:11.455 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.301e-06 2024-04-20T17:20:11.455 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.455 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.455 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:20:11.455 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.330368+0000", 2024-04-20T17:20:11.455 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3540000000000006e-06 2024-04-20T17:20:11.455 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.456 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.456 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:20:11.456 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.330373+0000", 2024-04-20T17:20:11.456 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2039999999999996e-06 2024-04-20T17:20:11.456 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.456 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.456 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:20:11.456 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.330458+0000", 2024-04-20T17:20:11.456 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4501999999999998e-05 2024-04-20T17:20:11.456 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.456 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.456 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:20:11.456 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.330535+0000", 2024-04-20T17:20:11.457 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7043000000000001e-05 2024-04-20T17:20:11.457 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.457 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.457 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:20:11.457 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.330602+0000", 2024-04-20T17:20:11.457 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7628999999999999e-05 2024-04-20T17:20:11.457 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.457 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.457 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:20:11.457 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.334144+0000", 2024-04-20T17:20:11.457 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0035412460000000001 2024-04-20T17:20:11.457 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.457 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.457 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:11.458 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.484820+0000", 2024-04-20T17:20:11.458 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.15067651700000001 2024-04-20T17:20:11.458 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.458 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.458 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:20:11.458 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.484872+0000", 2024-04-20T17:20:11.458 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2160999999999997e-05 2024-04-20T17:20:11.458 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.458 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.458 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:20:11.458 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.484916+0000", 2024-04-20T17:20:11.458 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3377999999999997e-05 2024-04-20T17:20:11.458 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.459 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:20:11.459 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.459 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.459 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.459 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:296479 2.0 2:0af10367:::benchmark_data_smithi171_78918_object135485:head [delete] snapc 0=[] ondisk+write+known_if_redirected e338)", 2024-04-20T17:20:11.459 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:19:31.433968+0000", 2024-04-20T17:20:11.459 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 39.952394183000003, 2024-04-20T17:20:11.459 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.051092179000000001, 2024-04-20T17:20:11.459 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:20:11.459 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:20:11.459 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:20:11.459 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:20:11.459 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:20:11.459 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:20:11.460 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 296479 2024-04-20T17:20:11.460 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.460 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:20:11.460 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.460 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:20:11.460 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.433968+0000", 2024-04-20T17:20:11.460 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.460 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.460 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.460 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:20:11.460 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.433968+0000", 2024-04-20T17:20:11.460 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.460 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.460 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.460 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:20:11.461 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.433987+0000", 2024-04-20T17:20:11.461 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8689999999999999e-05 2024-04-20T17:20:11.461 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.461 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.461 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:20:11.461 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.433991+0000", 2024-04-20T17:20:11.461 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3089999999999997e-06 2024-04-20T17:20:11.461 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.461 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.461 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:20:11.461 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.434000+0000", 2024-04-20T17:20:11.461 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4420000000000004e-06 2024-04-20T17:20:11.461 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.461 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.462 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:20:11.462 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.434006+0000", 2024-04-20T17:20:11.462 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3900000000000001e-06 2024-04-20T17:20:11.462 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.462 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.462 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:20:11.462 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.434090+0000", 2024-04-20T17:20:11.462 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4759000000000006e-05 2024-04-20T17:20:11.462 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.462 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.462 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:20:11.462 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.434167+0000", 2024-04-20T17:20:11.462 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6153000000000001e-05 2024-04-20T17:20:11.462 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.463 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.463 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:20:11.463 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.434237+0000", 2024-04-20T17:20:11.463 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0310999999999998e-05 2024-04-20T17:20:11.463 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.463 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.463 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:20:11.463 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.436945+0000", 2024-04-20T17:20:11.463 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0027083010000000002 2024-04-20T17:20:11.463 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.463 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.463 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:11.463 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.484966+0000", 2024-04-20T17:20:11.463 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.048020793999999999 2024-04-20T17:20:11.464 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.464 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.464 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:20:11.464 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.485019+0000", 2024-04-20T17:20:11.464 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3452000000000003e-05 2024-04-20T17:20:11.464 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.464 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.464 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:20:11.464 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.485060+0000", 2024-04-20T17:20:11.464 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0577999999999997e-05 2024-04-20T17:20:11.464 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.464 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:20:11.464 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.464 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.464 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.465 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:296482 2.9 2:93a0443b:::benchmark_data_smithi171_78918_object135488:head [delete] snapc 0=[] ondisk+write+known_if_redirected e338)", 2024-04-20T17:20:11.465 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:19:31.445897+0000", 2024-04-20T17:20:11.465 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 39.940465256000003, 2024-04-20T17:20:11.465 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.037448162, 2024-04-20T17:20:11.465 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:20:11.465 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:20:11.465 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:20:11.465 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:20:11.465 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:20:11.465 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:20:11.465 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 296482 2024-04-20T17:20:11.465 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.465 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:20:11.466 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.466 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:20:11.466 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.445897+0000", 2024-04-20T17:20:11.466 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.466 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.466 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.466 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:20:11.466 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.445897+0000", 2024-04-20T17:20:11.466 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.466 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.466 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.466 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:20:11.466 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.445905+0000", 2024-04-20T17:20:11.466 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0830000000000004e-06 2024-04-20T17:20:11.467 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.467 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.467 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:20:11.467 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.445916+0000", 2024-04-20T17:20:11.467 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0643e-05 2024-04-20T17:20:11.467 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.467 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.467 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:20:11.467 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.445963+0000", 2024-04-20T17:20:11.467 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6989e-05 2024-04-20T17:20:11.467 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.467 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.467 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:20:11.467 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.445983+0000", 2024-04-20T17:20:11.468 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0400000000000001e-05 2024-04-20T17:20:11.468 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.468 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.468 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:20:11.468 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.446126+0000", 2024-04-20T17:20:11.468 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000143127 2024-04-20T17:20:11.468 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.468 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.468 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:20:11.468 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.446307+0000", 2024-04-20T17:20:11.468 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00018072300000000001 2024-04-20T17:20:11.468 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.468 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.468 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:20:11.469 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.446424+0000", 2024-04-20T17:20:11.469 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000117024 2024-04-20T17:20:11.469 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.469 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.469 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:20:11.469 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.449896+0000", 2024-04-20T17:20:11.469 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0034723620000000001 2024-04-20T17:20:11.469 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.469 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.469 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:11.469 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.483259+0000", 2024-04-20T17:20:11.469 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.033362356000000003 2024-04-20T17:20:11.469 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.469 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.470 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:20:11.470 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.483308+0000", 2024-04-20T17:20:11.470 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9356e-05 2024-04-20T17:20:11.470 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.470 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.470 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:20:11.470 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:31.483345+0000", 2024-04-20T17:20:11.470 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7098999999999998e-05 2024-04-20T17:20:11.470 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.470 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:20:11.470 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.470 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.470 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.470 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:308578 2.0 2:09204347:::benchmark_data_smithi171_78918_object147584:head [delete] snapc 0=[] ondisk+write+known_if_redirected e344)", 2024-04-20T17:20:11.471 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:19:51.385980+0000", 2024-04-20T17:20:11.471 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 20.000382347999999, 2024-04-20T17:20:11.471 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.030924884999999999, 2024-04-20T17:20:11.471 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:20:11.471 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:20:11.471 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:20:11.471 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:20:11.471 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:20:11.471 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:20:11.471 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 308578 2024-04-20T17:20:11.471 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.471 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:20:11.471 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.471 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:20:11.472 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.385980+0000", 2024-04-20T17:20:11.472 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.472 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.472 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.472 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:20:11.472 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.385980+0000", 2024-04-20T17:20:11.472 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.472 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.472 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.472 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:20:11.472 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.385981+0000", 2024-04-20T17:20:11.472 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4959999999999999e-06 2024-04-20T17:20:11.472 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.472 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.472 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:20:11.472 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.385984+0000", 2024-04-20T17:20:11.473 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2320000000000001e-06 2024-04-20T17:20:11.473 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.473 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.473 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:20:11.473 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.385991+0000", 2024-04-20T17:20:11.473 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.601e-06 2024-04-20T17:20:11.473 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.473 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.473 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:20:11.473 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.385995+0000", 2024-04-20T17:20:11.473 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.296e-06 2024-04-20T17:20:11.473 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.473 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.473 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:20:11.473 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.386026+0000", 2024-04-20T17:20:11.473 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0871999999999999e-05 2024-04-20T17:20:11.474 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.474 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.474 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:20:11.474 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.386102+0000", 2024-04-20T17:20:11.474 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5870000000000004e-05 2024-04-20T17:20:11.474 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.474 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.474 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:20:11.474 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.386167+0000", 2024-04-20T17:20:11.474 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5215000000000005e-05 2024-04-20T17:20:11.474 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.474 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.475 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:20:11.475 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.416688+0000", 2024-04-20T17:20:11.475 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.030521073999999999 2024-04-20T17:20:11.475 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.475 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.475 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:11.475 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.416808+0000", 2024-04-20T17:20:11.475 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011972 2024-04-20T17:20:11.475 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.475 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.475 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:20:11.475 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.416861+0000", 2024-04-20T17:20:11.475 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2457999999999998e-05 2024-04-20T17:20:11.475 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.476 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.476 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:20:11.476 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.416905+0000", 2024-04-20T17:20:11.476 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4051000000000002e-05 2024-04-20T17:20:11.476 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.476 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:20:11.476 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.476 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.476 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.476 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:308579 2.9 2:99c478bc:::benchmark_data_smithi171_78918_object147585:head [delete] snapc 0=[] ondisk+write+known_if_redirected e344)", 2024-04-20T17:20:11.476 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:19:51.388451+0000", 2024-04-20T17:20:11.477 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 19.997911465000001, 2024-04-20T17:20:11.477 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.033042228999999999, 2024-04-20T17:20:11.477 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:20:11.477 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:20:11.477 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:20:11.477 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:20:11.477 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:20:11.477 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:20:11.477 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 308579 2024-04-20T17:20:11.477 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.477 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:20:11.477 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.477 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:20:11.478 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.388451+0000", 2024-04-20T17:20:11.478 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.478 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.478 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.478 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:20:11.478 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.388451+0000", 2024-04-20T17:20:11.478 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.478 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.478 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.478 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:20:11.478 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.388452+0000", 2024-04-20T17:20:11.478 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2410000000000001e-06 2024-04-20T17:20:11.478 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.478 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.479 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:20:11.479 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.388456+0000", 2024-04-20T17:20:11.479 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.211e-06 2024-04-20T17:20:11.479 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.479 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.479 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:20:11.479 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.388464+0000", 2024-04-20T17:20:11.479 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1769999999999998e-06 2024-04-20T17:20:11.479 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.479 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.479 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:20:11.479 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.388469+0000", 2024-04-20T17:20:11.479 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2020000000000003e-06 2024-04-20T17:20:11.480 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.480 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.480 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:20:11.480 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.388510+0000", 2024-04-20T17:20:11.480 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0649000000000003e-05 2024-04-20T17:20:11.480 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.480 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.480 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:20:11.480 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.388582+0000", 2024-04-20T17:20:11.480 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1629000000000001e-05 2024-04-20T17:20:11.480 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.480 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.480 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:20:11.480 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.388645+0000", 2024-04-20T17:20:11.481 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3106000000000002e-05 2024-04-20T17:20:11.481 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.481 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.481 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:11.481 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.421256+0000", 2024-04-20T17:20:11.481 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.032611543 2024-04-20T17:20:11.481 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.481 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.481 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:20:11.481 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.421398+0000", 2024-04-20T17:20:11.481 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014171299999999999 2024-04-20T17:20:11.481 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.481 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.481 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:20:11.481 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.421446+0000", 2024-04-20T17:20:11.482 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8328999999999998e-05 2024-04-20T17:20:11.482 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.482 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.482 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:20:11.482 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:51.421493+0000", 2024-04-20T17:20:11.482 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6428999999999999e-05 2024-04-20T17:20:11.482 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.482 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:20:11.482 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.482 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.482 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.482 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:311804 2.8 2:1d94bd01:::benchmark_data_smithi171_78918_object150810:head [delete] snapc 0=[] ondisk+write+known_if_redirected e344)", 2024-04-20T17:20:11.482 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:19:54.968446+0000", 2024-04-20T17:20:11.482 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 16.417916524999999, 2024-04-20T17:20:11.482 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.029438519999999999, 2024-04-20T17:20:11.483 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:20:11.483 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:20:11.483 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:20:11.483 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:20:11.483 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:20:11.483 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:20:11.483 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 311804 2024-04-20T17:20:11.483 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.483 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:20:11.483 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.483 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:20:11.483 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:54.968446+0000", 2024-04-20T17:20:11.483 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.483 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.483 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.483 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:20:11.484 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:54.968446+0000", 2024-04-20T17:20:11.484 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.484 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.484 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.484 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:20:11.484 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:54.968447+0000", 2024-04-20T17:20:11.484 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.781e-06 2024-04-20T17:20:11.484 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.484 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.484 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:20:11.484 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:54.968453+0000", 2024-04-20T17:20:11.484 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8880000000000002e-06 2024-04-20T17:20:11.484 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.484 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.485 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:20:11.485 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:54.968464+0000", 2024-04-20T17:20:11.485 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0834e-05 2024-04-20T17:20:11.485 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.485 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.485 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:20:11.485 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:54.968471+0000", 2024-04-20T17:20:11.485 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6969999999999996e-06 2024-04-20T17:20:11.485 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.485 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.485 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:20:11.485 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:54.979227+0000", 2024-04-20T17:20:11.485 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010756623 2024-04-20T17:20:11.485 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.486 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.486 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:20:11.486 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:54.979338+0000", 2024-04-20T17:20:11.486 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000110912 2024-04-20T17:20:11.486 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.486 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.486 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 9", 2024-04-20T17:20:11.486 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:54.979452+0000", 2024-04-20T17:20:11.486 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011423 2024-04-20T17:20:11.486 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.486 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.486 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:11.486 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:54.993799+0000", 2024-04-20T17:20:11.486 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.014346978 2024-04-20T17:20:11.486 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.487 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.487 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:20:11.487 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:54.997815+0000", 2024-04-20T17:20:11.487 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0040153050000000003 2024-04-20T17:20:11.487 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.487 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.487 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:20:11.487 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:54.997844+0000", 2024-04-20T17:20:11.487 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9403e-05 2024-04-20T17:20:11.487 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.487 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.487 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:20:11.487 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:54.997884+0000", 2024-04-20T17:20:11.487 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9869000000000001e-05 2024-04-20T17:20:11.487 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.488 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:20:11.488 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.488 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.488 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.488 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:315648 2.9 2:9b191409:::benchmark_data_smithi171_78918_object154654:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-04-20T17:20:11.488 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:19:58.555455+0000", 2024-04-20T17:20:11.488 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 12.830906904000001, 2024-04-20T17:20:11.488 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.03140681, 2024-04-20T17:20:11.488 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:20:11.488 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:20:11.488 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:20:11.488 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:20:11.488 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:20:11.488 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:20:11.488 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 315648 2024-04-20T17:20:11.488 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.489 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:20:11.489 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.489 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:20:11.489 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.555455+0000", 2024-04-20T17:20:11.489 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.489 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.489 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.489 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:20:11.489 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.555455+0000", 2024-04-20T17:20:11.489 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.489 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.489 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.489 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:20:11.489 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.555456+0000", 2024-04-20T17:20:11.489 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4299999999999998e-07 2024-04-20T17:20:11.490 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.490 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.490 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:20:11.490 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.555458+0000", 2024-04-20T17:20:11.490 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.092e-06 2024-04-20T17:20:11.490 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.490 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.490 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:20:11.490 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.555462+0000", 2024-04-20T17:20:11.490 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2849999999999996e-06 2024-04-20T17:20:11.490 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.490 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.490 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:20:11.490 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.555465+0000", 2024-04-20T17:20:11.490 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2200000000000001e-06 2024-04-20T17:20:11.491 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.491 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.491 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:20:11.491 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.559689+0000", 2024-04-20T17:20:11.491 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0042239679999999998 2024-04-20T17:20:11.491 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.491 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.491 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:20:11.491 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.559777+0000", 2024-04-20T17:20:11.491 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8279e-05 2024-04-20T17:20:11.491 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.491 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.491 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:20:11.491 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.559853+0000", 2024-04-20T17:20:11.491 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5109000000000002e-05 2024-04-20T17:20:11.492 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.492 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.492 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:20:11.492 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.563261+0000", 2024-04-20T17:20:11.492 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003408576 2024-04-20T17:20:11.492 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.492 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.492 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:11.492 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.586778+0000", 2024-04-20T17:20:11.492 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.023517318999999998 2024-04-20T17:20:11.492 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.492 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.492 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:20:11.493 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.586826+0000", 2024-04-20T17:20:11.493 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7620000000000001e-05 2024-04-20T17:20:11.493 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.493 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.493 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:20:11.493 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.586862+0000", 2024-04-20T17:20:11.493 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5899000000000003e-05 2024-04-20T17:20:11.493 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.493 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:20:11.493 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.493 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.493 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.493 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:315783 2.0 2:0cde8c3c:::benchmark_data_smithi171_78918_object154789:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-04-20T17:20:11.494 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:19:58.671859+0000", 2024-04-20T17:20:11.494 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 12.714503434999999, 2024-04-20T17:20:11.494 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.030862891, 2024-04-20T17:20:11.494 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:20:11.494 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:20:11.494 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:20:11.494 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:20:11.494 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:20:11.494 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:20:11.494 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 315783 2024-04-20T17:20:11.494 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.494 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:20:11.494 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.495 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:20:11.495 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.671859+0000", 2024-04-20T17:20:11.495 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.495 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.495 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.495 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:20:11.495 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.671859+0000", 2024-04-20T17:20:11.495 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.495 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.495 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.495 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:20:11.495 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.671860+0000", 2024-04-20T17:20:11.495 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5230000000000001e-06 2024-04-20T17:20:11.495 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.495 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.496 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:20:11.496 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.671865+0000", 2024-04-20T17:20:11.496 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.916e-06 2024-04-20T17:20:11.496 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.496 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.496 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:20:11.496 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.671875+0000", 2024-04-20T17:20:11.496 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0151e-05 2024-04-20T17:20:11.496 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.496 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.496 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:20:11.496 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.671882+0000", 2024-04-20T17:20:11.496 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8669999999999996e-06 2024-04-20T17:20:11.496 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.496 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.496 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:20:11.497 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.679875+0000", 2024-04-20T17:20:11.497 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.007993188 2024-04-20T17:20:11.497 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.497 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.497 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:20:11.497 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.680007+0000", 2024-04-20T17:20:11.497 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013206700000000001 2024-04-20T17:20:11.497 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.497 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.497 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:20:11.497 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.680131+0000", 2024-04-20T17:20:11.497 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012318499999999999 2024-04-20T17:20:11.497 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.497 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.497 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:20:11.498 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.689756+0000", 2024-04-20T17:20:11.498 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0096255690000000005 2024-04-20T17:20:11.498 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.498 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.498 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:11.498 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.702630+0000", 2024-04-20T17:20:11.498 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012874128 2024-04-20T17:20:11.498 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.498 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.498 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:20:11.498 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.702671+0000", 2024-04-20T17:20:11.498 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0735999999999998e-05 2024-04-20T17:20:11.498 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.498 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.498 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:20:11.499 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.702722+0000", 2024-04-20T17:20:11.499 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0560999999999999e-05 2024-04-20T17:20:11.499 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.499 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:20:11.499 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.499 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.499 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.499 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:315880 2.0 2:0342f9ca:::benchmark_data_smithi171_78918_object154886:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-04-20T17:20:11.499 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:19:58.751890+0000", 2024-04-20T17:20:11.499 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 12.634472332, 2024-04-20T17:20:11.499 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.030040837000000001, 2024-04-20T17:20:11.499 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:20:11.499 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:20:11.499 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:20:11.500 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:20:11.500 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:20:11.500 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:20:11.500 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 315880 2024-04-20T17:20:11.500 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.500 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:20:11.500 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.500 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:20:11.500 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.751890+0000", 2024-04-20T17:20:11.500 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.500 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.500 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.500 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:20:11.501 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.751890+0000", 2024-04-20T17:20:11.501 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.501 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.501 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.501 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:20:11.501 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.751890+0000", 2024-04-20T17:20:11.501 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4999999999999998e-07 2024-04-20T17:20:11.501 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.501 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.501 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:20:11.501 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.751892+0000", 2024-04-20T17:20:11.501 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0020000000000001e-06 2024-04-20T17:20:11.501 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.501 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.502 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:20:11.502 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.751898+0000", 2024-04-20T17:20:11.502 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3480000000000003e-06 2024-04-20T17:20:11.502 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.502 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.502 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:20:11.502 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.751902+0000", 2024-04-20T17:20:11.502 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4970000000000003e-06 2024-04-20T17:20:11.502 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.502 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.502 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:20:11.502 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.763442+0000", 2024-04-20T17:20:11.502 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011540135 2024-04-20T17:20:11.502 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.503 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.503 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:20:11.503 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.763533+0000", 2024-04-20T17:20:11.503 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1248000000000005e-05 2024-04-20T17:20:11.503 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.503 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.503 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:20:11.503 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.763604+0000", 2024-04-20T17:20:11.503 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0112999999999998e-05 2024-04-20T17:20:11.503 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.503 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.503 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:20:11.503 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.776334+0000", 2024-04-20T17:20:11.503 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012730142999999999 2024-04-20T17:20:11.504 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.504 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.504 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:11.504 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.781830+0000", 2024-04-20T17:20:11.504 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0054964840000000003 2024-04-20T17:20:11.504 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.504 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.504 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:20:11.504 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.781874+0000", 2024-04-20T17:20:11.504 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3665999999999998e-05 2024-04-20T17:20:11.504 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.504 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.504 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:20:11.504 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.781931+0000", 2024-04-20T17:20:11.504 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6750999999999999e-05 2024-04-20T17:20:11.505 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.505 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:20:11.505 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.505 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.505 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.505 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24736.0:315883 2.0 2:0b6ae6c1:::benchmark_data_smithi171_78918_object154889:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-04-20T17:20:11.505 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:19:58.751907+0000", 2024-04-20T17:20:11.505 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 12.634455384000001, 2024-04-20T17:20:11.505 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.030165013000000001, 2024-04-20T17:20:11.505 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:20:11.505 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:20:11.505 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:20:11.505 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:20:11.505 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24736", 2024-04-20T17:20:11.506 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:20:11.506 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 315883 2024-04-20T17:20:11.506 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.506 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:20:11.506 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.506 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:20:11.506 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.751907+0000", 2024-04-20T17:20:11.506 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.506 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.506 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.506 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:20:11.506 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.751907+0000", 2024-04-20T17:20:11.506 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:20:11.507 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.507 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.507 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:20:11.507 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.751907+0000", 2024-04-20T17:20:11.507 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.58e-07 2024-04-20T17:20:11.507 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.507 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.507 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:20:11.507 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.751909+0000", 2024-04-20T17:20:11.507 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2010000000000002e-06 2024-04-20T17:20:11.507 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.507 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.507 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:20:11.507 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.751915+0000", 2024-04-20T17:20:11.508 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3550000000000004e-06 2024-04-20T17:20:11.508 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.508 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.508 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:20:11.508 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.751919+0000", 2024-04-20T17:20:11.508 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2560000000000004e-06 2024-04-20T17:20:11.508 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.508 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.508 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:20:11.508 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.767861+0000", 2024-04-20T17:20:11.508 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015942331000000001 2024-04-20T17:20:11.508 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.508 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.508 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:20:11.508 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.767996+0000", 2024-04-20T17:20:11.508 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013452700000000001 2024-04-20T17:20:11.509 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.509 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.509 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:20:11.509 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.768118+0000", 2024-04-20T17:20:11.509 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012213299999999999 2024-04-20T17:20:11.509 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.509 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.509 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:20:11.509 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.781277+0000", 2024-04-20T17:20:11.509 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.013158596999999999 2024-04-20T17:20:11.509 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.509 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.509 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:11.509 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.781983+0000", 2024-04-20T17:20:11.509 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00070633899999999995 2024-04-20T17:20:11.509 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.510 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.510 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:20:11.510 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.782035+0000", 2024-04-20T17:20:11.510 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1767000000000001e-05 2024-04-20T17:20:11.510 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:20:11.510 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:20:11.510 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:20:11.510 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:19:58.782072+0000", 2024-04-20T17:20:11.510 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7048999999999997e-05 2024-04-20T17:20:11.510 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.510 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:20:11.510 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.510 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:20:11.510 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:20:11.510 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:20:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:11 smithi106 ceph-mon[44423]: pgmap v960: 17 pgs: 17 active+clean; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 698 op/s 2024-04-20T17:20:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:11 smithi106 ceph-mon[44423]: osdmap e349: 12 total, 12 up, 12 in 2024-04-20T17:20:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:11 smithi106 ceph-mon[44423]: osdmap e350: 12 total, 12 up, 12 in 2024-04-20T17:20:11.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:11.512 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:11.512 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:20:11.513 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:20:11.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:11 smithi171 ceph-mon[44189]: pgmap v960: 17 pgs: 17 active+clean; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 698 op/s 2024-04-20T17:20:11.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:11 smithi171 ceph-mon[44189]: osdmap e349: 12 total, 12 up, 12 in 2024-04-20T17:20:11.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:11 smithi171 ceph-mon[44189]: osdmap e350: 12 total, 12 up, 12 in 2024-04-20T17:20:11.613 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:20:11.643 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:11.663 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:20:11.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:11.667 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:20:11.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:11 smithi005 ceph-mon[39562]: pgmap v960: 17 pgs: 17 active+clean; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 698 op/s 2024-04-20T17:20:11.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:11 smithi005 ceph-mon[39562]: osdmap e349: 12 total, 12 up, 12 in 2024-04-20T17:20:11.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:11 smithi005 ceph-mon[39562]: osdmap e350: 12 total, 12 up, 12 in 2024-04-20T17:20:11.767 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:20:11.795 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:12.083 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-20T17:20:12.143 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:20:12.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:12.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:20:12.167 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:20:12.210 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:20:12.246 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:20:12.260 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'unique_pool_0' removed 2024-04-20T17:20:12.280 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:12.370 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:20:12.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:12 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/914641779' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-20T17:20:12.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:12 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/914641779' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-20T17:20:12.529 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:20:12.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:12.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:20:12.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:12 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/914641779' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-20T17:20:12.634 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:20:12.663 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:12.890 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:20:12.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:12.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:20:12.998 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:20:13.025 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:13.043 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:20:13.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:13.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:20:13.147 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:20:13.175 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:13.353 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:20:13.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:13.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:20:13.382 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-20T17:20:13.382 INFO:tasks.radosbench:Beginning radosbench... 2024-04-20T17:20:13.383 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-04-20T17:20:13.383 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-04-20T17:20:13.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:13 smithi005 ceph-mon[39562]: pgmap v963: 17 pgs: 17 active+clean; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 143 op/s 2024-04-20T17:20:13.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:13 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/914641779' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-04-20T17:20:13.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:13 smithi005 ceph-mon[39562]: osdmap e351: 12 total, 12 up, 12 in 2024-04-20T17:20:13.456 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:20:13.489 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:13.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:13 smithi106 ceph-mon[44423]: pgmap v963: 17 pgs: 17 active+clean; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 143 op/s 2024-04-20T17:20:13.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:13 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/914641779' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-04-20T17:20:13.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:13 smithi106 ceph-mon[44423]: osdmap e351: 12 total, 12 up, 12 in 2024-04-20T17:20:13.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:13 smithi171 ceph-mon[44189]: pgmap v963: 17 pgs: 17 active+clean; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 143 op/s 2024-04-20T17:20:13.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:13 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/914641779' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-04-20T17:20:13.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:13 smithi171 ceph-mon[44189]: osdmap e351: 12 total, 12 up, 12 in 2024-04-20T17:20:13.838 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:20:13.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:13.841 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:20:13.942 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:20:13.975 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:13.998 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:20:14.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:14.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:20:14.102 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:20:14.130 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:14.154 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:20:14.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:14.157 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:20:14.258 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:20:14.285 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:14.304 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:20:14.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:14.307 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:20:14.407 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:20:14.435 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:14.439 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:20:14.439 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:20:14.439 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 0 2024-04-20T17:20:14.439 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:20:14.456 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:20:14.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:14.459 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:20:14.560 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:20:14.593 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:14.614 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:20:14.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:14.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:20:14.718 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:20:14.751 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:14.781 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:20:14.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:14.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:20:14.891 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:20:14.924 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:14.946 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:20:14.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:14.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:20:15.049 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:20:15.078 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:15.151 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:20:15.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:15.155 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:20:15.176 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-20T17:20:15.255 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:20:15.288 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:15.309 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:20:15.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:15.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:20:15.413 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:20:15.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:15 smithi005 ceph-mon[39562]: pgmap v965: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:20:15.450 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:15.476 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:20:15.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:15 smithi106 ceph-mon[44423]: pgmap v965: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:20:15.576 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:20:15.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:15.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:20:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:15 smithi171 ceph-mon[44189]: pgmap v965: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:20:15.682 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:20:15.713 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:15.734 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:20:15.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:15.737 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:20:15.838 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:20:15.868 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:16.071 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:20:16.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:16.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:20:16.175 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:20:16.203 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:16.926 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:20:16.927 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:20:17.019 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:20:17.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:17 smithi171 ceph-mon[44189]: osdmap e352: 12 total, 12 up, 12 in 2024-04-20T17:20:17.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:17 smithi171 ceph-mon[44189]: pgmap v967: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:20:17.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:17 smithi171 ceph-mon[44189]: from='client.15807 v1:172.21.15.5:0/169561922' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:20:17.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:17 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1664506738' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:20:17.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:17 smithi106 ceph-mon[44423]: osdmap e352: 12 total, 12 up, 12 in 2024-04-20T17:20:17.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:17 smithi106 ceph-mon[44423]: pgmap v967: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:20:17.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:17 smithi106 ceph-mon[44423]: from='client.15807 v1:172.21.15.5:0/169561922' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:20:17.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:17 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1664506738' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:20:17.351 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:20:17.418 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:20:17.418 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:20:17.419 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-04-20T17:20:17.419 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-04-20T17:20:17.419 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:20:17.430 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:17 smithi005 ceph-mon[39562]: osdmap e352: 12 total, 12 up, 12 in 2024-04-20T17:20:17.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:17 smithi005 ceph-mon[39562]: pgmap v967: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:20:17.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:17 smithi005 ceph-mon[39562]: from='client.15807 v1:172.21.15.5:0/169561922' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:20:17.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:17 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1664506738' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:20:17.431 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:20:17.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:17.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:20:17.544 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:20:17.577 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:17.599 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:20:17.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:17.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:20:17.703 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:20:17.737 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:17.897 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":967,"stamp":"2024-04-20T17:20:16.092494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":6,"kb":1124941824,"kb_used":2393116,"kb_used_data":1000824,"kb_used_omap":219,"kb_used_meta":1391908,"kb_avail":1122548708,"statfs":{"total":1151940427776,"available":1149489876992,"internally_reserved":0,"allocated":1024843776,"data_stored":1009402524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224949,"internal_metadata":1425314123},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":62,"apply_latency_ms":62,"commit_latency_ns":62000000,"apply_latency_ns":62000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-321984,"num_write_kb":-10303425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-29216,"num_bytes_recovered":-1914699776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.842702"},"pg_stats":[{"pgid":"1.0","version":"15'137","reported_seq":911,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-04-20T17:20:11.473179+0000","last_change":"2024-04-20T17:19:36.190286+0000","last_active":"2024-04-20T17:20:11.473179+0000","last_peered":"2024-04-20T17:20:11.473179+0000","last_clean":"2024-04-20T17:20:11.473179+0000","last_became_active":"2024-04-20T17:06:04.391416+0000","last_became_peered":"2024-04-20T17:06:04.391416+0000","last_unstale":"2024-04-20T17:20:11.473179+0000","last_undegraded":"2024-04-20T17:20:11.473179+0000","last_fullsized":"2024-04-20T17:20:11.473179+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:19:36.190151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:19:36.190151+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:20:40.598718+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991406,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":110628,"kb_used_data":3096,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93634524,"statfs":{"total":95995035648,"available":95881752576,"internally_reserved":0,"allocated":3170304,"data_stored":1886308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":110098884},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253044,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":218412,"kb_used_data":2592,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93526740,"statfs":{"total":95995035648,"available":95771381760,"internally_reserved":0,"allocated":2654208,"data_stored":1363642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":220985792},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612794,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":150380,"kb_used_data":2592,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93594772,"statfs":{"total":95995035648,"available":95841046528,"internally_reserved":0,"allocated":2654208,"data_stored":1363662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":151321024},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809023,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":128180,"kb_used_data":2592,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93616972,"statfs":{"total":95995035648,"available":95863779328,"internally_reserved":0,"allocated":2654208,"data_stored":1363662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168772,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":114788,"kb_used_data":3096,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93630364,"statfs":{"total":95995035648,"available":95877492736,"internally_reserved":0,"allocated":3170304,"data_stored":1886328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":114358730},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430408,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":114256,"kb_used_data":2564,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93630896,"statfs":{"total":95995035648,"available":95878037504,"internally_reserved":0,"allocated":2625536,"data_stored":1351090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575074,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":105936,"kb_used_data":2564,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93639216,"statfs":{"total":95995035648,"available":95886557184,"internally_reserved":0,"allocated":2625536,"data_stored":1351090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738599,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":186484,"kb_used_data":2592,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93558668,"statfs":{"total":95995035648,"available":95804076032,"internally_reserved":0,"allocated":2654208,"data_stored":1363662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":448368,"kb_used_data":399968,"kb_used_omap":202,"kb_used_meta":48181,"kb_avail":93296784,"statfs":{"total":95995035648,"available":95535906816,"internally_reserved":0,"allocated":409567232,"data_stored":408276686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":49337815},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182616,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":74028,"kb_used_data":2592,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93671124,"statfs":{"total":95995035648,"available":95919230976,"internally_reserved":0,"allocated":2654208,"data_stored":1363662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986387,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":64300,"kb_used_data":2592,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":93680852,"statfs":{"total":95995035648,"available":95929192448,"internally_reserved":0,"allocated":2654208,"data_stored":1363662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790158,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":677356,"kb_used_data":573984,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93067796,"statfs":{"total":95995035648,"available":95301423104,"internally_reserved":0,"allocated":587759616,"data_stored":586469070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:20:17.898 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-20T17:20:18.088 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:20:18.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:18.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:20:18.145 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:18 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:20:18.145 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:18 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1664506738' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:20:18.145 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:18 smithi171 ceph-mon[44189]: osdmap e353: 12 total, 12 up, 12 in 2024-04-20T17:20:18.173 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-20T17:20:18.193 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:20:18.225 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:18.247 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:20:18.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:18.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:20:18.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:18 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:20:18.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:18 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1664506738' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:20:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:18 smithi106 ceph-mon[44423]: osdmap e353: 12 total, 12 up, 12 in 2024-04-20T17:20:18.351 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:20:18.352 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:18 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:20:18.352 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:18 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1664506738' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:20:18.352 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:18 smithi005 ceph-mon[39562]: osdmap e353: 12 total, 12 up, 12 in 2024-04-20T17:20:18.382 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:18.403 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:20:18.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:18.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:20:18.439 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:20:18.507 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:20:18.539 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:18.746 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:20:18.746 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:20:18.747 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:20:18.747 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:20:18.747 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:20:18.747 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:20:18.766 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:20:18.786 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:20:18.806 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:20:18.826 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:20:18.846 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:20:18.866 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:20:18.868 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:20:18.868 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:20:18.868 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:20:18.868 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:20:18.868 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:20:18.868 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:20:18.868 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:20:18.868 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:20:18.868 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:20:18.868 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:20:18.868 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:20:18.868 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:20:19.193 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:19 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:20:19.193 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:19 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:20:19.193 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:19 smithi005 ceph-mon[39562]: pgmap v969: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:20:19.193 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:19 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1749647910' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:20:19.193 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:19.193 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:19.193 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:20:19.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:19.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:20:19.300 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:20:19.330 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:19 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:20:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:19 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:20:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:19 smithi171 ceph-mon[44189]: pgmap v969: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:20:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:19 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1749647910' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:20:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:19 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:20:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:19 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:20:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:19 smithi106 ceph-mon[44423]: pgmap v969: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:20:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:19 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1749647910' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:20:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:20 smithi171 ceph-mon[44189]: osdmap e354: 12 total, 12 up, 12 in 2024-04-20T17:20:20.443 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:20:20.443 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:20:20.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:20 smithi005 ceph-mon[39562]: osdmap e354: 12 total, 12 up, 12 in 2024-04-20T17:20:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:20 smithi106 ceph-mon[44423]: osdmap e354: 12 total, 12 up, 12 in 2024-04-20T17:20:20.580 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:20:20.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:20.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:20:20.685 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:20:20.715 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:20.737 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:20:20.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:20.739 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:20:20.840 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:20:20.873 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:21.083 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:20:21.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:21.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:20:21.186 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:20:21.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:21 smithi005 ceph-mon[39562]: pgmap v971: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:20:21.213 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:21.251 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:20:21.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:21.255 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:20:21.273 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-20T17:20:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:21 smithi171 ceph-mon[44189]: pgmap v971: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:20:21.356 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:20:21.386 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:21 smithi106 ceph-mon[44423]: pgmap v971: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:20:21.389 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:21.411 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:20:21.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:21.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:20:21.515 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:20:21.544 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:21.565 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:20:21.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:21.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:20:21.573 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:20:21.668 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:20:21.695 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:21.894 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:20:21.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:21.896 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:20:21.997 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:20:22.031 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'unique_pool_1' created 2024-04-20T17:20:22.032 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:22.053 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:20:22.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:22.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:20:22.157 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:20:22.189 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:22 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:20:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:22 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:20:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:22 smithi171 ceph-mon[44189]: osdmap e355: 12 total, 12 up, 12 in 2024-04-20T17:20:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:22 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1155670007' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-04-20T17:20:22.437 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:20:22.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:22.441 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:20:22.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:22 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:20:22.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:22 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:20:22.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:22 smithi005 ceph-mon[39562]: osdmap e355: 12 total, 12 up, 12 in 2024-04-20T17:20:22.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:22 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1155670007' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-04-20T17:20:22.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:22 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:20:22.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:22 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:20:22.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:22 smithi106 ceph-mon[44423]: osdmap e355: 12 total, 12 up, 12 in 2024-04-20T17:20:22.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:22 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1155670007' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-04-20T17:20:22.541 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:20:22.569 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:22.589 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:20:22.918 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:20:22.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:22.921 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:20:23.022 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:20:23.051 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:23.069 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:20:23.075 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-04-20T17:20:23.108 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:20:23.197 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:23 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1155670007' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-04-20T17:20:23.197 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:23 smithi171 ceph-mon[44189]: osdmap e356: 12 total, 12 up, 12 in 2024-04-20T17:20:23.197 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:23 smithi171 ceph-mon[44189]: pgmap v974: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:20:23.197 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:23 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:20:23.198 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:23 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:20:23.199 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:20:23.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:23.202 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:20:23.303 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:20:23.333 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:23 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1155670007' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-04-20T17:20:23.333 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:23 smithi106 ceph-mon[44423]: osdmap e356: 12 total, 12 up, 12 in 2024-04-20T17:20:23.333 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:23 smithi106 ceph-mon[44423]: pgmap v974: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:20:23.333 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:23 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:20:23.333 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:23 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:20:23.335 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:23.357 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:20:23.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:23.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:20:23.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:23 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1155670007' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-04-20T17:20:23.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:23 smithi005 ceph-mon[39562]: osdmap e356: 12 total, 12 up, 12 in 2024-04-20T17:20:23.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:23 smithi005 ceph-mon[39562]: pgmap v974: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:20:23.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:23 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:20:23.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:23 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:20:23.448 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:20:23.448 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:20:23.461 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:20:23.490 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:23.638 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:20:23.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:23.641 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:20:23.742 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:20:23.772 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:24.073 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-20T17:20:24.332 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:20:24.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:24 smithi171 ceph-mon[44189]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-20T17:20:24.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:24 smithi171 ceph-mon[44189]: osdmap e357: 12 total, 12 up, 12 in 2024-04-20T17:20:24.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:24 smithi005 ceph-mon[39562]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-20T17:20:24.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:24 smithi005 ceph-mon[39562]: osdmap e357: 12 total, 12 up, 12 in 2024-04-20T17:20:24.504 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:20:24.505 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:20:24.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:24 smithi106 ceph-mon[44423]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-20T17:20:24.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:24 smithi106 ceph-mon[44423]: osdmap e357: 12 total, 12 up, 12 in 2024-04-20T17:20:24.520 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:20:24.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:24.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:20:24.627 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:20:24.660 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:24.682 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:20:24.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:24.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:20:24.785 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:20:24.814 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:24.851 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:20:24.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:24.854 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:20:24.955 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:20:24.988 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:25.010 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:20:25.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:25.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:20:25.114 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:20:25.143 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:25 smithi005 ceph-mon[39562]: osdmap e358: 12 total, 12 up, 12 in 2024-04-20T17:20:25.143 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:25 smithi005 ceph-mon[39562]: pgmap v977: 17 pgs: 5 creating+peering, 4 active+clean, 8 unknown; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:20:25.146 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:25.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:25 smithi106 ceph-mon[44423]: osdmap e358: 12 total, 12 up, 12 in 2024-04-20T17:20:25.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:25 smithi106 ceph-mon[44423]: pgmap v977: 17 pgs: 5 creating+peering, 4 active+clean, 8 unknown; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:20:25.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:25 smithi171 ceph-mon[44189]: osdmap e358: 12 total, 12 up, 12 in 2024-04-20T17:20:25.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:25 smithi171 ceph-mon[44189]: pgmap v977: 17 pgs: 5 creating+peering, 4 active+clean, 8 unknown; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:20:26.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:26 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:20:26.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:26 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:20:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:26 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:20:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:26 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:20:26.352 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:20:26.353 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:20:26.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:26 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:20:26.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:26 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:20:27.092 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-20T17:20:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:27 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:20:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:27 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:20:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:27 smithi005 ceph-mon[39562]: osdmap e359: 12 total, 12 up, 12 in 2024-04-20T17:20:27.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:27 smithi005 ceph-mon[39562]: pgmap v979: 17 pgs: 5 creating+peering, 10 active+clean, 2 unknown; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:20:27.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:27 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4118681891' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:20:27.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:27.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:27 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:20:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:27 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:20:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:27 smithi171 ceph-mon[44189]: osdmap e359: 12 total, 12 up, 12 in 2024-04-20T17:20:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:27 smithi171 ceph-mon[44189]: pgmap v979: 17 pgs: 5 creating+peering, 10 active+clean, 2 unknown; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:20:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:27 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4118681891' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:20:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:27.357 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:20:27.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:27 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:20:27.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:27 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:20:27.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:27 smithi106 ceph-mon[44423]: osdmap e359: 12 total, 12 up, 12 in 2024-04-20T17:20:27.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:27 smithi106 ceph-mon[44423]: pgmap v979: 17 pgs: 5 creating+peering, 10 active+clean, 2 unknown; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:20:27.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:27 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4118681891' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:20:27.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:27.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:27.527 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:20:27.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:27.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:20:27.633 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:20:27.665 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:27.687 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:20:27.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:27.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:20:27.776 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:20:27.776 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:20:27.776 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:20:27.776 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:20:27.777 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:20:27.777 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:20:27.777 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:20:27.777 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:20:27.777 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:20:27.777 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:20:27.777 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:20:27.777 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:20:27.777 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:20:27.777 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:20:27.777 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:20:27.777 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:20:27.777 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:20:27.777 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:20:27.778 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:20:27.778 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:20:27.778 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:20:27.778 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:20:27.778 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:20:27.778 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:20:27.791 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:20:27.820 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:27.839 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:20:27.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:27.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:20:27.943 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:20:27.971 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:27.990 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:20:27.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:27.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:20:28.094 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:20:28.126 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:28.246 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:20:28.248 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.248 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.248 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.248 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.248 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.248 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.248 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.248 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.248 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.248 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:28.249 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:20:28.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.350 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:20:28.379 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:28.399 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:20:28.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:28.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:20:28.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:28.502 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:20:28.531 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:28.550 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:20:28.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:28.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:20:28.653 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:20:28.684 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:29.061 INFO:teuthology.orchestra.run.smithi005.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-04-20T17:20:29.129 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:29 smithi171 ceph-mon[44189]: pgmap v980: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:20:29.129 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:29 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2913849864' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:20:29.415 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:20:29.415 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:20:29.415 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:20:29.415 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:20:29.415 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.415 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop_reply(osd.9.0:17489 2.1 e210/196 ondisk, result = 0)", 2024-04-20T17:20:29.415 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:40.112229+0000", 2024-04-20T17:20:29.415 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 469.30110105400001, 2024-04-20T17:20:29.415 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.074599223000000006, 2024-04-20T17:20:29.415 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:20:29.416 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:20:29.416 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:20:29.416 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:20:29.416 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.416 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:20:29.416 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:40.112229+0000", 2024-04-20T17:20:29.416 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.416 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.416 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.416 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:20:29.416 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:40.112229+0000", 2024-04-20T17:20:29.416 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.416 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.416 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.416 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:20:29.417 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:40.112230+0000", 2024-04-20T17:20:29.417 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.051e-06 2024-04-20T17:20:29.417 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.417 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.417 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:20:29.417 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:40.112235+0000", 2024-04-20T17:20:29.417 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4660000000000001e-06 2024-04-20T17:20:29.417 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.417 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.417 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:20:29.417 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:40.112242+0000", 2024-04-20T17:20:29.417 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8430000000000004e-06 2024-04-20T17:20:29.417 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.417 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.418 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:20:29.418 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:40.112246+0000", 2024-04-20T17:20:29.418 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5800000000000002e-06 2024-04-20T17:20:29.418 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.418 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.418 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:20:29.418 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:40.186800+0000", 2024-04-20T17:20:29.418 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.074553845999999993 2024-04-20T17:20:29.418 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.418 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.418 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:20:29.418 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:40.186811+0000", 2024-04-20T17:20:29.418 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0766e-05 2024-04-20T17:20:29.418 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.419 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.419 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:20:29.419 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:40.186829+0000", 2024-04-20T17:20:29.419 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7671000000000001e-05 2024-04-20T17:20:29.419 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.419 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:20:29.419 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.419 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.419 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.419 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop_reply(osd.9.0:17490 2.1 e210/196 ondisk, result = 0)", 2024-04-20T17:20:29.419 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:40.114672+0000", 2024-04-20T17:20:29.419 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 469.29865798100002, 2024-04-20T17:20:29.419 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.072210796999999993, 2024-04-20T17:20:29.419 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:20:29.419 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:20:29.420 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:20:29.420 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:20:29.420 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.420 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:20:29.420 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:40.114672+0000", 2024-04-20T17:20:29.420 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.420 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.420 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.420 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:20:29.420 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:40.114672+0000", 2024-04-20T17:20:29.420 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.420 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.420 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.420 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:20:29.421 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:40.114674+0000", 2024-04-20T17:20:29.421 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6139999999999999e-06 2024-04-20T17:20:29.421 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.421 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.421 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:20:29.421 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:40.114682+0000", 2024-04-20T17:20:29.421 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4540000000000007e-06 2024-04-20T17:20:29.421 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.421 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.421 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:20:29.421 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:40.114695+0000", 2024-04-20T17:20:29.421 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2313999999999999e-05 2024-04-20T17:20:29.422 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.422 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.422 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:20:29.422 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:40.114702+0000", 2024-04-20T17:20:29.422 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3420000000000002e-06 2024-04-20T17:20:29.422 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.422 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.422 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:20:29.422 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:40.186867+0000", 2024-04-20T17:20:29.422 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.072164942999999995 2024-04-20T17:20:29.422 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.422 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.422 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:20:29.423 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:40.186873+0000", 2024-04-20T17:20:29.423 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.699e-06 2024-04-20T17:20:29.423 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.423 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.423 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:20:29.423 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:40.186883+0000", 2024-04-20T17:20:29.423 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0431000000000001e-05 2024-04-20T17:20:29.423 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.423 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:20:29.423 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.424 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.424 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.424 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop_reply(osd.9.0:17832 2.1 e210/196 ondisk, result = 0)", 2024-04-20T17:20:29.424 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.055399+0000", 2024-04-20T17:20:29.424 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 468.35793178900002, 2024-04-20T17:20:29.424 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.076129015999999994, 2024-04-20T17:20:29.424 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:20:29.424 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:20:29.424 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:20:29.424 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:20:29.424 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.424 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:20:29.424 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.055399+0000", 2024-04-20T17:20:29.424 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.425 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.425 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.425 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:20:29.425 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.055399+0000", 2024-04-20T17:20:29.425 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.425 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.425 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.425 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:20:29.425 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.055400+0000", 2024-04-20T17:20:29.425 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.761e-06 2024-04-20T17:20:29.425 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.425 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.425 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:20:29.425 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.055408+0000", 2024-04-20T17:20:29.425 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8590000000000005e-06 2024-04-20T17:20:29.426 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.426 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.426 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:20:29.426 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.055417+0000", 2024-04-20T17:20:29.426 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.7530000000000007e-06 2024-04-20T17:20:29.426 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.426 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.426 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:20:29.426 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.055423+0000", 2024-04-20T17:20:29.426 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.122e-06 2024-04-20T17:20:29.426 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.426 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.426 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:20:29.426 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.131510+0000", 2024-04-20T17:20:29.427 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.076087023000000004 2024-04-20T17:20:29.427 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.427 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.427 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:20:29.427 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.131515+0000", 2024-04-20T17:20:29.427 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.259e-06 2024-04-20T17:20:29.427 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.427 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.427 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:20:29.427 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.131528+0000", 2024-04-20T17:20:29.427 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2238999999999999e-05 2024-04-20T17:20:29.427 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.427 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:20:29.427 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.427 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.428 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.428 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop_reply(osd.9.0:17833 2.1 e210/196 ondisk, result = 0)", 2024-04-20T17:20:29.428 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.059852+0000", 2024-04-20T17:20:29.428 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 468.35347819100002, 2024-04-20T17:20:29.428 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.071719485999999999, 2024-04-20T17:20:29.428 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:20:29.428 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:20:29.428 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:20:29.428 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:20:29.428 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.428 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:20:29.428 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.059852+0000", 2024-04-20T17:20:29.428 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.428 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.428 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.428 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:20:29.429 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.059852+0000", 2024-04-20T17:20:29.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.429 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.429 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:20:29.429 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.059854+0000", 2024-04-20T17:20:29.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7859999999999999e-06 2024-04-20T17:20:29.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.429 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.429 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:20:29.429 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.059860+0000", 2024-04-20T17:20:29.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.835e-06 2024-04-20T17:20:29.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.429 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.430 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:20:29.430 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.059869+0000", 2024-04-20T17:20:29.430 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.4900000000000006e-06 2024-04-20T17:20:29.430 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.430 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.430 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:20:29.430 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.059876+0000", 2024-04-20T17:20:29.430 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.443e-06 2024-04-20T17:20:29.430 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.430 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.430 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:20:29.430 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.131554+0000", 2024-04-20T17:20:29.430 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.071678521999999995 2024-04-20T17:20:29.430 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.430 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.431 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:20:29.431 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.131559+0000", 2024-04-20T17:20:29.431 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6349999999999997e-06 2024-04-20T17:20:29.431 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.431 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.431 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:20:29.431 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.131572+0000", 2024-04-20T17:20:29.431 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2775000000000001e-05 2024-04-20T17:20:29.431 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.431 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:20:29.431 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.431 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.431 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.431 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop_reply(osd.9.0:17888 2.1 e210/196 ondisk, result = 0)", 2024-04-20T17:20:29.432 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.238325+0000", 2024-04-20T17:20:29.432 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 468.17500579099999, 2024-04-20T17:20:29.432 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.077219381000000004, 2024-04-20T17:20:29.432 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:20:29.432 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:20:29.432 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:20:29.432 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:20:29.432 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.432 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:20:29.432 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.238325+0000", 2024-04-20T17:20:29.432 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.432 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.432 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.432 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:20:29.432 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.238325+0000", 2024-04-20T17:20:29.433 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.433 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.433 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.433 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:20:29.433 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.238326+0000", 2024-04-20T17:20:29.433 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.435e-06 2024-04-20T17:20:29.433 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.433 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.433 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:20:29.433 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.238333+0000", 2024-04-20T17:20:29.433 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9840000000000004e-06 2024-04-20T17:20:29.433 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.433 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.433 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:20:29.433 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.238341+0000", 2024-04-20T17:20:29.434 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0669999999999992e-06 2024-04-20T17:20:29.434 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.434 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.434 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:20:29.434 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.238347+0000", 2024-04-20T17:20:29.434 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1829999999999999e-06 2024-04-20T17:20:29.434 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.434 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.434 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:20:29.434 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.315522+0000", 2024-04-20T17:20:29.434 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.077174712000000006 2024-04-20T17:20:29.434 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.434 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.434 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:20:29.435 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.315530+0000", 2024-04-20T17:20:29.435 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5390000000000002e-06 2024-04-20T17:20:29.435 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.435 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.435 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:20:29.435 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.315544+0000", 2024-04-20T17:20:29.435 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4460999999999999e-05 2024-04-20T17:20:29.435 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.435 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:20:29.435 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.435 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.435 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.435 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop_reply(osd.9.0:17889 2.1 e210/196 ondisk, result = 0)", 2024-04-20T17:20:29.435 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.241379+0000", 2024-04-20T17:20:29.435 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 468.17195119000002, 2024-04-20T17:20:29.436 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.074201934999999997, 2024-04-20T17:20:29.436 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:20:29.436 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:20:29.436 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:20:29.436 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:20:29.436 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.436 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:20:29.436 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.241379+0000", 2024-04-20T17:20:29.436 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.436 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.436 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.436 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:20:29.436 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.241379+0000", 2024-04-20T17:20:29.436 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.436 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.437 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.437 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:20:29.437 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.241381+0000", 2024-04-20T17:20:29.437 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3510000000000001e-06 2024-04-20T17:20:29.437 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.437 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.437 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:20:29.437 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.241386+0000", 2024-04-20T17:20:29.437 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8980000000000001e-06 2024-04-20T17:20:29.437 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.437 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.437 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:20:29.437 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.241394+0000", 2024-04-20T17:20:29.437 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3030000000000002e-06 2024-04-20T17:20:29.437 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.438 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.438 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:20:29.438 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.241399+0000", 2024-04-20T17:20:29.438 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8160000000000001e-06 2024-04-20T17:20:29.438 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.438 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.438 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:20:29.438 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.315568+0000", 2024-04-20T17:20:29.438 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.074169297999999995 2024-04-20T17:20:29.438 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.438 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.438 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:20:29.438 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.315572+0000", 2024-04-20T17:20:29.439 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5839999999999996e-06 2024-04-20T17:20:29.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.439 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.439 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:20:29.439 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.315581+0000", 2024-04-20T17:20:29.439 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6850000000000007e-06 2024-04-20T17:20:29.439 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.439 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:20:29.439 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.439 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.439 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop_reply(osd.9.0:17946 2.1 e210/196 ondisk, result = 0)", 2024-04-20T17:20:29.439 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.428639+0000", 2024-04-20T17:20:29.439 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 467.98469179, 2024-04-20T17:20:29.439 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.078332470000000001, 2024-04-20T17:20:29.440 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:20:29.440 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:20:29.440 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:20:29.440 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:20:29.440 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.440 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:20:29.440 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.428639+0000", 2024-04-20T17:20:29.440 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.440 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.440 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:20:29.440 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.428639+0000", 2024-04-20T17:20:29.440 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.440 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.441 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:20:29.441 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.428640+0000", 2024-04-20T17:20:29.441 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3740000000000001e-06 2024-04-20T17:20:29.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.441 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:20:29.441 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.428648+0000", 2024-04-20T17:20:29.441 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5970000000000003e-06 2024-04-20T17:20:29.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.441 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:20:29.441 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.428656+0000", 2024-04-20T17:20:29.441 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4230000000000005e-06 2024-04-20T17:20:29.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.441 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:20:29.442 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.428661+0000", 2024-04-20T17:20:29.442 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0409999999999997e-06 2024-04-20T17:20:29.442 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.442 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.442 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:20:29.442 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.506956+0000", 2024-04-20T17:20:29.442 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.078294927 2024-04-20T17:20:29.442 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.442 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.442 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:20:29.442 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.506961+0000", 2024-04-20T17:20:29.442 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3580000000000002e-06 2024-04-20T17:20:29.442 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.442 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.442 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:20:29.443 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.506971+0000", 2024-04-20T17:20:29.443 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7499999999999998e-06 2024-04-20T17:20:29.443 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.443 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:20:29.443 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.443 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.443 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.443 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop_reply(osd.9.0:17948 2.1 e210/196 ondisk, result = 0)", 2024-04-20T17:20:29.443 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.431625+0000", 2024-04-20T17:20:29.443 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 467.98170584899998, 2024-04-20T17:20:29.443 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.075695629, 2024-04-20T17:20:29.443 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:20:29.443 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:20:29.443 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:20:29.444 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:20:29.444 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.444 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:20:29.444 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.431625+0000", 2024-04-20T17:20:29.444 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.444 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.444 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.444 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:20:29.444 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.431625+0000", 2024-04-20T17:20:29.444 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.444 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.444 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.444 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:20:29.444 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.431626+0000", 2024-04-20T17:20:29.445 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3740000000000001e-06 2024-04-20T17:20:29.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.445 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:20:29.445 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.431634+0000", 2024-04-20T17:20:29.445 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9049999999999997e-06 2024-04-20T17:20:29.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.445 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:20:29.445 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.431644+0000", 2024-04-20T17:20:29.445 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0394000000000001e-05 2024-04-20T17:20:29.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.445 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:20:29.445 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.431650+0000", 2024-04-20T17:20:29.446 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5269999999999998e-06 2024-04-20T17:20:29.446 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.446 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.446 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:20:29.446 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.507000+0000", 2024-04-20T17:20:29.446 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.075350544000000005 2024-04-20T17:20:29.446 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.446 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.446 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:20:29.446 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.507008+0000", 2024-04-20T17:20:29.446 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9330000000000001e-06 2024-04-20T17:20:29.446 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.446 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.446 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:20:29.447 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.507320+0000", 2024-04-20T17:20:29.447 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000311952 2024-04-20T17:20:29.447 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.447 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:20:29.447 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.447 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.447 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.447 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop_reply(osd.9.0:17949 2.1 e210/196 ondisk, result = 0)", 2024-04-20T17:20:29.447 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.435105+0000", 2024-04-20T17:20:29.447 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 467.97822557299997, 2024-04-20T17:20:29.447 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.072280062000000006, 2024-04-20T17:20:29.447 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:20:29.447 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:20:29.447 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:20:29.447 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:20:29.447 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.448 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:20:29.448 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.435105+0000", 2024-04-20T17:20:29.448 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.448 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.448 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.448 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:20:29.448 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.435105+0000", 2024-04-20T17:20:29.448 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.448 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.448 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.448 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:20:29.448 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.435106+0000", 2024-04-20T17:20:29.448 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.133e-06 2024-04-20T17:20:29.448 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.448 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.448 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:20:29.449 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.435113+0000", 2024-04-20T17:20:29.449 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9090000000000003e-06 2024-04-20T17:20:29.449 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.449 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.449 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:20:29.449 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.435121+0000", 2024-04-20T17:20:29.449 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5709999999999998e-06 2024-04-20T17:20:29.449 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.449 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.449 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:20:29.449 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.435128+0000", 2024-04-20T17:20:29.449 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7479999999999996e-06 2024-04-20T17:20:29.449 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.449 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.450 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:20:29.450 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.507333+0000", 2024-04-20T17:20:29.450 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.072204711000000005 2024-04-20T17:20:29.450 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.450 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.450 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:20:29.450 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.507337+0000", 2024-04-20T17:20:29.450 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.873e-06 2024-04-20T17:20:29.450 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.450 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.450 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:20:29.450 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.507385+0000", 2024-04-20T17:20:29.450 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8117e-05 2024-04-20T17:20:29.450 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.450 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:20:29.451 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.451 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.451 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.451 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop_reply(osd.9.0:18002 2.1 e210/196 ondisk, result = 0)", 2024-04-20T17:20:29.451 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:41.607603+0000", 2024-04-20T17:20:29.451 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 467.80572753600001, 2024-04-20T17:20:29.451 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.075973654000000002, 2024-04-20T17:20:29.451 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:20:29.451 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:20:29.451 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:20:29.451 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:20:29.451 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.451 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:20:29.451 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.607603+0000", 2024-04-20T17:20:29.451 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.452 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.452 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.452 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:20:29.452 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.607603+0000", 2024-04-20T17:20:29.452 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.452 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.452 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.452 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:20:29.452 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.607605+0000", 2024-04-20T17:20:29.452 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9230000000000001e-06 2024-04-20T17:20:29.452 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.452 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.452 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:20:29.452 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.607614+0000", 2024-04-20T17:20:29.452 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5780000000000005e-06 2024-04-20T17:20:29.453 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.453 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.453 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:20:29.453 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.607629+0000", 2024-04-20T17:20:29.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4610000000000001e-05 2024-04-20T17:20:29.453 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.453 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.453 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:20:29.453 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.607637+0000", 2024-04-20T17:20:29.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.3760000000000008e-06 2024-04-20T17:20:29.453 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.453 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.453 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:20:29.453 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.683440+0000", 2024-04-20T17:20:29.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.075802639000000005 2024-04-20T17:20:29.453 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.454 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.454 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:20:29.454 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.683447+0000", 2024-04-20T17:20:29.454 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1400000000000002e-06 2024-04-20T17:20:29.454 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.454 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.454 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:20:29.454 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:41.683577+0000", 2024-04-20T17:20:29.454 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000129388 2024-04-20T17:20:29.454 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.454 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:20:29.454 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.454 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.454 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.454 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:217356 2.1 2:874fa609:::benchmark_data_smithi171_78918_object56362:head [delete] snapc 0=[] ondisk+write+known_if_redirected e211)", 2024-04-20T17:20:29.455 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:43.370229+0000", 2024-04-20T17:20:29.455 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 466.043101654, 2024-04-20T17:20:29.455 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.075470378000000005, 2024-04-20T17:20:29.455 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:20:29.455 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:20:29.455 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:20:29.455 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:20:29.455 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:20:29.455 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:20:29.455 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 217356 2024-04-20T17:20:29.455 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.455 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:20:29.455 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.455 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:20:29.455 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.370229+0000", 2024-04-20T17:20:29.456 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.456 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.456 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.456 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:20:29.456 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.370229+0000", 2024-04-20T17:20:29.456 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.456 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.456 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.456 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:20:29.456 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.370231+0000", 2024-04-20T17:20:29.456 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.154e-06 2024-04-20T17:20:29.456 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.456 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.456 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:20:29.456 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.370237+0000", 2024-04-20T17:20:29.457 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.212e-06 2024-04-20T17:20:29.457 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.457 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:20:29.457 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.370249+0000", 2024-04-20T17:20:29.457 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.221e-05 2024-04-20T17:20:29.457 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.457 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:20:29.457 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.370256+0000", 2024-04-20T17:20:29.457 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1149999999999996e-06 2024-04-20T17:20:29.457 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.457 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:20:29.458 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.438241+0000", 2024-04-20T17:20:29.458 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.067984471000000005 2024-04-20T17:20:29.458 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.458 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.458 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:20:29.458 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.438329+0000", 2024-04-20T17:20:29.458 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8096999999999994e-05 2024-04-20T17:20:29.458 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.458 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.458 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 6", 2024-04-20T17:20:29.458 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.438422+0000", 2024-04-20T17:20:29.458 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3395999999999998e-05 2024-04-20T17:20:29.458 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.458 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.459 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:29.459 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.443322+0000", 2024-04-20T17:20:29.459 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0048996730000000002 2024-04-20T17:20:29.459 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.459 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.459 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:20:29.459 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.445587+0000", 2024-04-20T17:20:29.459 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.002265249 2024-04-20T17:20:29.459 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.459 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.459 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:20:29.459 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.445638+0000", 2024-04-20T17:20:29.459 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0247000000000002e-05 2024-04-20T17:20:29.459 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.460 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.460 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:20:29.460 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.445699+0000", 2024-04-20T17:20:29.460 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1554000000000001e-05 2024-04-20T17:20:29.460 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.460 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:20:29.460 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.460 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.460 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.460 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:217527 2.1 2:80cdc6c2:::benchmark_data_smithi171_78918_object56533:head [delete] snapc 0=[] ondisk+write+known_if_redirected e211)", 2024-04-20T17:20:29.460 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:43.918312+0000", 2024-04-20T17:20:29.460 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 465.495018845, 2024-04-20T17:20:29.460 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.080793121999999995, 2024-04-20T17:20:29.460 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:20:29.461 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:20:29.461 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:20:29.461 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:20:29.461 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:20:29.461 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:20:29.461 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 217527 2024-04-20T17:20:29.461 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.461 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:20:29.461 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.461 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:20:29.461 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.918312+0000", 2024-04-20T17:20:29.461 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.461 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.461 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.461 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:20:29.462 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.918312+0000", 2024-04-20T17:20:29.462 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.462 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:20:29.462 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.918314+0000", 2024-04-20T17:20:29.462 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1500000000000002e-06 2024-04-20T17:20:29.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.462 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:20:29.462 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.918320+0000", 2024-04-20T17:20:29.462 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1700000000000002e-06 2024-04-20T17:20:29.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.463 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:20:29.463 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.918332+0000", 2024-04-20T17:20:29.463 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2482e-05 2024-04-20T17:20:29.463 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.463 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.463 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:20:29.463 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.918341+0000", 2024-04-20T17:20:29.463 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8990000000000007e-06 2024-04-20T17:20:29.463 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.463 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.463 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:20:29.463 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.923804+0000", 2024-04-20T17:20:29.463 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0054630039999999996 2024-04-20T17:20:29.463 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.463 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.464 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:20:29.464 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.923930+0000", 2024-04-20T17:20:29.464 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000125549 2024-04-20T17:20:29.464 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.464 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.464 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 6", 2024-04-20T17:20:29.464 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.924033+0000", 2024-04-20T17:20:29.464 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010332600000000001 2024-04-20T17:20:29.464 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.464 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.464 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:20:29.464 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.996198+0000", 2024-04-20T17:20:29.464 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.072165354000000001 2024-04-20T17:20:29.464 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.464 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.465 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:29.465 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.999001+0000", 2024-04-20T17:20:29.465 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.002802311 2024-04-20T17:20:29.465 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.465 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.465 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:20:29.465 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.999051+0000", 2024-04-20T17:20:29.465 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0074000000000002e-05 2024-04-20T17:20:29.465 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.465 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.465 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:20:29.465 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.999105+0000", 2024-04-20T17:20:29.465 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3803e-05 2024-04-20T17:20:29.465 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.465 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:20:29.466 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.466 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.466 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.466 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:217529 2.1 2:860e0fee:::benchmark_data_smithi171_78918_object56535:head [delete] snapc 0=[] ondisk+write+known_if_redirected e212)", 2024-04-20T17:20:29.466 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:43.923227+0000", 2024-04-20T17:20:29.466 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 465.49010356899998, 2024-04-20T17:20:29.466 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.078287889999999999, 2024-04-20T17:20:29.466 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:20:29.466 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:20:29.466 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:20:29.466 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:20:29.466 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:20:29.466 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:20:29.466 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 217529 2024-04-20T17:20:29.467 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.467 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:20:29.467 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.467 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:20:29.467 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.923227+0000", 2024-04-20T17:20:29.467 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.467 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.467 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.467 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:20:29.467 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.923227+0000", 2024-04-20T17:20:29.467 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.467 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.467 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.467 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:20:29.467 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.923229+0000", 2024-04-20T17:20:29.468 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2970000000000002e-06 2024-04-20T17:20:29.468 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.468 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.468 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:20:29.468 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.923236+0000", 2024-04-20T17:20:29.468 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5529999999999998e-06 2024-04-20T17:20:29.468 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.468 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.468 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:20:29.468 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.923247+0000", 2024-04-20T17:20:29.468 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.146e-05 2024-04-20T17:20:29.468 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.468 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.468 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:20:29.468 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.923255+0000", 2024-04-20T17:20:29.469 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4900000000000003e-06 2024-04-20T17:20:29.469 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.469 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.469 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:20:29.469 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.926619+0000", 2024-04-20T17:20:29.469 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.003364279 2024-04-20T17:20:29.469 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.469 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.469 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-04-20T17:20:29.469 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.926623+0000", 2024-04-20T17:20:29.469 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7749999999999999e-06 2024-04-20T17:20:29.469 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.469 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.469 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:20:29.469 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.996447+0000", 2024-04-20T17:20:29.470 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.069824356000000004 2024-04-20T17:20:29.470 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.470 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.470 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:20:29.470 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.996528+0000", 2024-04-20T17:20:29.470 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1433000000000005e-05 2024-04-20T17:20:29.470 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.470 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.470 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 6", 2024-04-20T17:20:29.470 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.996607+0000", 2024-04-20T17:20:29.470 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8930000000000005e-05 2024-04-20T17:20:29.470 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.470 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.471 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:20:29.471 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:43.999352+0000", 2024-04-20T17:20:29.471 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0027443250000000001 2024-04-20T17:20:29.471 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.471 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.471 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:29.471 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:44.001435+0000", 2024-04-20T17:20:29.471 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.002083222 2024-04-20T17:20:29.471 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.471 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.471 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:20:29.471 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:44.001465+0000", 2024-04-20T17:20:29.471 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0463999999999998e-05 2024-04-20T17:20:29.471 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.472 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.472 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:20:29.472 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:44.001515+0000", 2024-04-20T17:20:29.472 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9305999999999999e-05 2024-04-20T17:20:29.472 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.472 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:20:29.472 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.472 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.472 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.472 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:217871 2.1 2:8370647d:::benchmark_data_smithi171_78918_object56877:head [delete] snapc 0=[] ondisk+write+known_if_redirected e212)", 2024-04-20T17:20:29.472 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:45.208318+0000", 2024-04-20T17:20:29.472 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 464.20501272199999, 2024-04-20T17:20:29.472 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.074389018000000001, 2024-04-20T17:20:29.472 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:20:29.472 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:20:29.473 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:20:29.473 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:20:29.473 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:20:29.473 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:20:29.473 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 217871 2024-04-20T17:20:29.473 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.473 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:20:29.473 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.473 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:20:29.473 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.208318+0000", 2024-04-20T17:20:29.473 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.473 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.473 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.473 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:20:29.474 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.208318+0000", 2024-04-20T17:20:29.474 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.474 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.474 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.474 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:20:29.474 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.208320+0000", 2024-04-20T17:20:29.474 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8759999999999999e-06 2024-04-20T17:20:29.474 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.474 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.474 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:20:29.474 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.208325+0000", 2024-04-20T17:20:29.474 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4530000000000001e-06 2024-04-20T17:20:29.474 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.474 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.475 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:20:29.475 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.208334+0000", 2024-04-20T17:20:29.475 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.4779999999999997e-06 2024-04-20T17:20:29.475 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.475 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.475 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:20:29.475 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.208342+0000", 2024-04-20T17:20:29.475 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6290000000000001e-06 2024-04-20T17:20:29.475 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.475 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.475 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:20:29.475 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.208692+0000", 2024-04-20T17:20:29.475 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00035009800000000002 2024-04-20T17:20:29.475 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.476 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.476 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:20:29.476 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.208812+0000", 2024-04-20T17:20:29.476 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000119752 2024-04-20T17:20:29.476 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.476 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.476 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 6", 2024-04-20T17:20:29.476 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.208942+0000", 2024-04-20T17:20:29.476 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012996899999999999 2024-04-20T17:20:29.476 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.476 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.476 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:29.476 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.282548+0000", 2024-04-20T17:20:29.476 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.073605947000000005 2024-04-20T17:20:29.476 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.477 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.477 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:20:29.477 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.282623+0000", 2024-04-20T17:20:29.477 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5060000000000003e-05 2024-04-20T17:20:29.477 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.477 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.477 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:20:29.477 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.282652+0000", 2024-04-20T17:20:29.477 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9532000000000001e-05 2024-04-20T17:20:29.477 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.477 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.477 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:20:29.477 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.282707+0000", 2024-04-20T17:20:29.477 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4224000000000001e-05 2024-04-20T17:20:29.478 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.478 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:20:29.478 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.478 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.478 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.478 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop_reply(osd.9.0:19543 2.1 e212/196 ondisk, result = 0)", 2024-04-20T17:20:29.478 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:45.874492+0000", 2024-04-20T17:20:29.478 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 463.53883826800001, 2024-04-20T17:20:29.478 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.076683425999999999, 2024-04-20T17:20:29.478 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:20:29.478 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:20:29.478 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:20:29.478 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:20:29.478 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.478 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:20:29.479 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.874492+0000", 2024-04-20T17:20:29.479 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.479 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.479 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.479 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:20:29.479 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.874492+0000", 2024-04-20T17:20:29.479 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.479 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.479 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.479 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:20:29.479 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.874494+0000", 2024-04-20T17:20:29.479 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2060000000000001e-06 2024-04-20T17:20:29.479 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.480 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.480 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:20:29.480 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.874500+0000", 2024-04-20T17:20:29.480 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.156e-06 2024-04-20T17:20:29.480 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.480 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.480 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:20:29.480 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.874512+0000", 2024-04-20T17:20:29.480 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1399999999999999e-05 2024-04-20T17:20:29.480 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.480 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.480 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:20:29.480 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.874520+0000", 2024-04-20T17:20:29.480 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5949999999999999e-06 2024-04-20T17:20:29.480 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.481 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.481 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:20:29.481 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.951150+0000", 2024-04-20T17:20:29.481 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.076629261000000004 2024-04-20T17:20:29.481 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.481 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.481 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:20:29.481 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.951159+0000", 2024-04-20T17:20:29.481 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.4399999999999994e-06 2024-04-20T17:20:29.481 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.481 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.481 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:20:29.481 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.951176+0000", 2024-04-20T17:20:29.481 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6368e-05 2024-04-20T17:20:29.482 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.482 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:20:29.482 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.482 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.482 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.482 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop_reply(osd.9.0:19544 2.1 e212/196 ondisk, result = 0)", 2024-04-20T17:20:29.482 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:45.876949+0000", 2024-04-20T17:20:29.482 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 463.53638093900003, 2024-04-20T17:20:29.482 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.074484244000000005, 2024-04-20T17:20:29.482 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:20:29.482 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:20:29.482 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:20:29.482 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:20:29.482 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.482 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:20:29.483 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.876949+0000", 2024-04-20T17:20:29.483 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.483 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.483 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.483 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:20:29.483 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.876949+0000", 2024-04-20T17:20:29.483 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.483 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.483 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.483 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:20:29.483 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.876951+0000", 2024-04-20T17:20:29.483 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7209999999999999e-06 2024-04-20T17:20:29.483 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.483 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.484 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:20:29.484 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.876956+0000", 2024-04-20T17:20:29.484 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7620000000000003e-06 2024-04-20T17:20:29.484 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.484 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.484 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:20:29.484 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.876963+0000", 2024-04-20T17:20:29.484 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4900000000000003e-06 2024-04-20T17:20:29.484 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.484 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.484 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:20:29.484 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.876971+0000", 2024-04-20T17:20:29.484 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9200000000000004e-06 2024-04-20T17:20:29.484 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.485 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.485 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:20:29.485 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.951212+0000", 2024-04-20T17:20:29.485 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.074240904999999996 2024-04-20T17:20:29.485 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.485 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.485 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:20:29.485 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.951217+0000", 2024-04-20T17:20:29.485 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2039999999999996e-06 2024-04-20T17:20:29.485 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.485 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.485 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:20:29.485 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.951434+0000", 2024-04-20T17:20:29.485 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00021624200000000001 2024-04-20T17:20:29.485 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.485 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:20:29.486 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.486 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.486 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.486 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop_reply(osd.9.0:19545 2.1 e212/196 ondisk, result = 0)", 2024-04-20T17:20:29.486 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:45.878905+0000", 2024-04-20T17:20:29.486 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 463.53442528599999, 2024-04-20T17:20:29.486 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.072651151999999997, 2024-04-20T17:20:29.486 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:20:29.486 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:20:29.486 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:20:29.486 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:20:29.486 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.486 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:20:29.487 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.878905+0000", 2024-04-20T17:20:29.487 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.487 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.487 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.487 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:20:29.487 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.878905+0000", 2024-04-20T17:20:29.487 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.487 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.487 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.487 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:20:29.487 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.878906+0000", 2024-04-20T17:20:29.487 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.314e-06 2024-04-20T17:20:29.487 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.487 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.487 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:20:29.488 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.878911+0000", 2024-04-20T17:20:29.488 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6500000000000004e-06 2024-04-20T17:20:29.488 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.488 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.488 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:20:29.488 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.878921+0000", 2024-04-20T17:20:29.488 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.8689999999999999e-06 2024-04-20T17:20:29.488 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.488 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.488 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:20:29.488 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.878926+0000", 2024-04-20T17:20:29.488 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2850000000000002e-06 2024-04-20T17:20:29.488 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.488 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.488 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:20:29.489 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.951464+0000", 2024-04-20T17:20:29.489 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.072538048999999993 2024-04-20T17:20:29.489 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.489 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.489 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:20:29.489 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.951472+0000", 2024-04-20T17:20:29.489 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3030000000000002e-06 2024-04-20T17:20:29.489 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.489 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.489 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:20:29.489 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:45.951556+0000", 2024-04-20T17:20:29.489 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4682e-05 2024-04-20T17:20:29.489 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.489 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:20:29.490 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.490 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.490 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.490 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:218294 2.1 2:8c4f0610:::benchmark_data_smithi171_78918_object57300:head [delete] snapc 0=[] ondisk+write+known_if_redirected e213)", 2024-04-20T17:20:29.490 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:47.133760+0000", 2024-04-20T17:20:29.490 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 462.27956988400001, 2024-04-20T17:20:29.490 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.076450596999999995, 2024-04-20T17:20:29.490 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:20:29.490 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:20:29.490 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:20:29.490 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:20:29.490 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:20:29.490 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:20:29.490 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 218294 2024-04-20T17:20:29.490 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.491 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:20:29.491 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.491 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:20:29.491 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.133760+0000", 2024-04-20T17:20:29.491 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.491 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.491 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.491 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:20:29.491 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.133760+0000", 2024-04-20T17:20:29.491 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.491 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.491 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.491 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:20:29.491 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.133762+0000", 2024-04-20T17:20:29.491 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8470000000000001e-06 2024-04-20T17:20:29.492 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.492 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.492 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:20:29.492 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.133767+0000", 2024-04-20T17:20:29.492 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7840000000000003e-06 2024-04-20T17:20:29.492 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.492 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.492 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:20:29.492 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.133777+0000", 2024-04-20T17:20:29.492 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0152999999999999e-05 2024-04-20T17:20:29.492 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.492 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.492 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:20:29.492 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.133785+0000", 2024-04-20T17:20:29.492 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5120000000000002e-06 2024-04-20T17:20:29.493 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.493 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.493 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:20:29.493 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.139132+0000", 2024-04-20T17:20:29.493 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0053474999999999998 2024-04-20T17:20:29.493 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.493 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.493 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:20:29.493 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.139231+0000", 2024-04-20T17:20:29.493 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.8550999999999999e-05 2024-04-20T17:20:29.493 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.493 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.493 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 6", 2024-04-20T17:20:29.493 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.139307+0000", 2024-04-20T17:20:29.493 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5888000000000003e-05 2024-04-20T17:20:29.494 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.494 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.494 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:20:29.494 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.210074+0000", 2024-04-20T17:20:29.494 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.070767457000000006 2024-04-20T17:20:29.494 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.494 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.494 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:29.494 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.210113+0000", 2024-04-20T17:20:29.494 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9292000000000003e-05 2024-04-20T17:20:29.494 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.494 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.494 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:20:29.494 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.210147+0000", 2024-04-20T17:20:29.494 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3698999999999997e-05 2024-04-20T17:20:29.495 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.495 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.495 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:20:29.495 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:47.210211+0000", 2024-04-20T17:20:29.495 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3913999999999998e-05 2024-04-20T17:20:29.495 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.495 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:20:29.495 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.495 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.495 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.495 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:218486 2.1 2:8516dbaf:::benchmark_data_smithi171_78918_object57492:head [delete] snapc 0=[] ondisk+write+known_if_redirected e213)", 2024-04-20T17:20:29.495 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:48.067279+0000", 2024-04-20T17:20:29.495 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 461.34605108699998, 2024-04-20T17:20:29.495 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.073711417000000001, 2024-04-20T17:20:29.495 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:20:29.496 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:20:29.496 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:20:29.496 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:20:29.496 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:20:29.496 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:20:29.496 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 218486 2024-04-20T17:20:29.496 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.496 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:20:29.496 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.496 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:20:29.496 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:48.067279+0000", 2024-04-20T17:20:29.496 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.496 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.496 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.496 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:20:29.496 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:48.067279+0000", 2024-04-20T17:20:29.497 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.497 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.497 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.497 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:20:29.497 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:48.067281+0000", 2024-04-20T17:20:29.497 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0959999999999999e-06 2024-04-20T17:20:29.497 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.497 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.497 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:20:29.497 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:48.067288+0000", 2024-04-20T17:20:29.497 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3040000000000001e-06 2024-04-20T17:20:29.497 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.497 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.497 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:20:29.497 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:48.067301+0000", 2024-04-20T17:20:29.498 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.31e-05 2024-04-20T17:20:29.498 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.498 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.498 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:20:29.498 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:48.067309+0000", 2024-04-20T17:20:29.498 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8320000000000006e-06 2024-04-20T17:20:29.498 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.498 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.498 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:20:29.498 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:48.068707+0000", 2024-04-20T17:20:29.498 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001397899 2024-04-20T17:20:29.498 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.498 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.498 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:20:29.499 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:48.068794+0000", 2024-04-20T17:20:29.499 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.7582000000000003e-05 2024-04-20T17:20:29.499 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.499 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.499 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 6", 2024-04-20T17:20:29.499 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:48.068889+0000", 2024-04-20T17:20:29.499 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5359999999999995e-05 2024-04-20T17:20:29.499 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.499 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.499 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:20:29.499 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:48.140818+0000", 2024-04-20T17:20:29.499 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.071928400000000003 2024-04-20T17:20:29.499 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.499 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.499 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:29.500 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:48.140871+0000", 2024-04-20T17:20:29.500 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3029000000000003e-05 2024-04-20T17:20:29.500 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.500 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.500 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:20:29.500 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:48.140928+0000", 2024-04-20T17:20:29.500 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7568000000000002e-05 2024-04-20T17:20:29.500 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.500 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.500 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:20:29.500 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:48.140991+0000", 2024-04-20T17:20:29.500 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2247000000000002e-05 2024-04-20T17:20:29.500 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.500 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:20:29.500 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.501 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.501 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.501 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:218492 2.1 2:82d5c701:::benchmark_data_smithi171_78918_object57498:head [delete] snapc 0=[] ondisk+write+known_if_redirected e213)", 2024-04-20T17:20:29.501 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:12:48.077146+0000", 2024-04-20T17:20:29.501 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 461.33618480000001, 2024-04-20T17:20:29.501 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.072876473999999997, 2024-04-20T17:20:29.501 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:20:29.501 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:20:29.501 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:20:29.501 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:20:29.501 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:20:29.501 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:20:29.501 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 218492 2024-04-20T17:20:29.501 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.501 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:20:29.502 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.502 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:20:29.502 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:48.077146+0000", 2024-04-20T17:20:29.502 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.502 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.502 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.502 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:20:29.502 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:48.077146+0000", 2024-04-20T17:20:29.502 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:20:29.502 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.502 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.502 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:20:29.502 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:48.077148+0000", 2024-04-20T17:20:29.503 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.396e-06 2024-04-20T17:20:29.503 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.503 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.503 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:20:29.503 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:48.077154+0000", 2024-04-20T17:20:29.503 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.251e-06 2024-04-20T17:20:29.503 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.503 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.503 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:20:29.503 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:48.077167+0000", 2024-04-20T17:20:29.503 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2527000000000001e-05 2024-04-20T17:20:29.503 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.503 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.503 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:20:29.503 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:48.077174+0000", 2024-04-20T17:20:29.504 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.289e-06 2024-04-20T17:20:29.504 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.504 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.504 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:20:29.504 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:48.142635+0000", 2024-04-20T17:20:29.504 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.06546109 2024-04-20T17:20:29.504 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.504 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.504 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:20:29.504 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:48.142714+0000", 2024-04-20T17:20:29.504 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8464e-05 2024-04-20T17:20:29.504 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.504 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.504 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 6", 2024-04-20T17:20:29.504 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:48.142783+0000", 2024-04-20T17:20:29.505 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9468000000000006e-05 2024-04-20T17:20:29.505 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.505 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.505 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:20:29.505 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:48.149894+0000", 2024-04-20T17:20:29.505 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0071105309999999998 2024-04-20T17:20:29.505 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.505 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.505 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:20:29.505 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:48.149909+0000", 2024-04-20T17:20:29.505 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5075999999999999e-05 2024-04-20T17:20:29.505 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.505 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.505 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:20:29.505 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:48.149961+0000", 2024-04-20T17:20:29.506 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2076000000000001e-05 2024-04-20T17:20:29.506 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:20:29.506 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:20:29.506 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:20:29.506 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:12:48.150022+0000", 2024-04-20T17:20:29.506 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1305999999999999e-05 2024-04-20T17:20:29.506 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.506 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:20:29.506 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.506 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:20:29.506 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:20:29.506 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:20:29.507 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:29 smithi005 ceph-mon[39562]: pgmap v980: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:20:29.507 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:29 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2913849864' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:20:29.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:29 smithi106 ceph-mon[44423]: pgmap v980: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:20:29.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:29 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2913849864' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:20:30.034 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:20:30.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:30.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:20:30.139 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:20:30.169 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:30.192 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-20T17:20:30.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:30 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2913849864' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-20T17:20:30.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:30 smithi171 ceph-mon[44189]: osdmap e360: 12 total, 12 up, 12 in 2024-04-20T17:20:30.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:30 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:20:30.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:30 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:20:30.456 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:20:30.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:30 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2913849864' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-20T17:20:30.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:30 smithi005 ceph-mon[39562]: osdmap e360: 12 total, 12 up, 12 in 2024-04-20T17:20:30.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:30 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:20:30.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:30 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:20:30.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:30 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2913849864' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-20T17:20:30.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:30 smithi106 ceph-mon[44423]: osdmap e360: 12 total, 12 up, 12 in 2024-04-20T17:20:30.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:30 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:20:30.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:30 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:20:30.718 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:20:30.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:30.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:20:30.822 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:20:30.854 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:30.875 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:20:30.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:30.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:20:30.979 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:20:31.010 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:31.031 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:20:31.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:31.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:20:31.072 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:20:31.135 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:20:31.163 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:31.166 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:20:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:31 smithi171 ceph-mon[44189]: osdmap e361: 12 total, 12 up, 12 in 2024-04-20T17:20:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:31 smithi171 ceph-mon[44189]: pgmap v983: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:20:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:31 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1319786866' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:20:31.363 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:20:31.365 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:31 smithi005 ceph-mon[39562]: osdmap e361: 12 total, 12 up, 12 in 2024-04-20T17:20:31.365 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:31 smithi005 ceph-mon[39562]: pgmap v983: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:20:31.365 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:31 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1319786866' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:20:31.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:31.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:20:31.467 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:20:31.497 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:31.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:31 smithi106 ceph-mon[44423]: osdmap e361: 12 total, 12 up, 12 in 2024-04-20T17:20:31.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:31 smithi106 ceph-mon[44423]: pgmap v983: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:20:31.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:31 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1319786866' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:20:31.686 DEBUG:teuthology.orchestra.run.smithi171:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-04-20T17:20:31.689 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-20T17:20:31.689 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-20T17:20:31.745 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:20:31.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:31.748 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:20:31.790 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:hints = 1 2024-04-20T17:20:31.790 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-20T17:20:31.790 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Object prefix: benchmark_data_smithi171_110765 2024-04-20T17:20:31.790 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:20:31.790 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 0 0 0 0 0 0 - 0 2024-04-20T17:20:31.849 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:20:31.898 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:31.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:31.939 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:20:31.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:20:31.995 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:20:32.041 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:20:32.077 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:32.104 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:20:32.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:32.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:20:32.208 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:20:32.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-20T17:20:32.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:20:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1319786866' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:20:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: osdmap e362: 12 total, 12 up, 12 in 2024-04-20T17:20:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=3640712970,config=1460075751,health=3897446751,kv=931226538}) 2024-04-20T17:20:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {kv=5,logm=95} crc {kv=571464332,logm=1991913918}) 2024-04-20T17:20:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=887203525}) 2024-04-20T17:20:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2340427505}) 2024-04-20T17:20:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3659073397}) 2024-04-20T17:20:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4290301312}) 2024-04-20T17:20:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1473538472}) 2024-04-20T17:20:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2612976217}) 2024-04-20T17:20:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3186309417}) 2024-04-20T17:20:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=533910587}) 2024-04-20T17:20:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=963120067}) 2024-04-20T17:20:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2957720207}) 2024-04-20T17:20:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=100713154}) 2024-04-20T17:20:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3890318679}) 2024-04-20T17:20:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=263405216}) 2024-04-20T17:20:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=859103554}) 2024-04-20T17:20:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3581387597}) 2024-04-20T17:20:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3766785077}) 2024-04-20T17:20:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=940264748}) 2024-04-20T17:20:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=563483318}) 2024-04-20T17:20:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4252570098}) 2024-04-20T17:20:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3103627407}) 2024-04-20T17:20:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1152425576}) 2024-04-20T17:20:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2875921274}) 2024-04-20T17:20:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2709233972}) 2024-04-20T17:20:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2394141698}) 2024-04-20T17:20:32.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2535438873}) 2024-04-20T17:20:32.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1135384172}) 2024-04-20T17:20:32.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1776805465}) 2024-04-20T17:20:32.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=17450602}) 2024-04-20T17:20:32.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=773719723}) 2024-04-20T17:20:32.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2702381215}) 2024-04-20T17:20:32.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1799982686}) 2024-04-20T17:20:32.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3971553314}) 2024-04-20T17:20:32.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2631835817}) 2024-04-20T17:20:32.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3602080934}) 2024-04-20T17:20:32.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1957298077}) 2024-04-20T17:20:32.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=795783997}) 2024-04-20T17:20:32.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1661234810}) 2024-04-20T17:20:32.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1955793180}) 2024-04-20T17:20:32.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2064598678}) 2024-04-20T17:20:32.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1927754497}) 2024-04-20T17:20:32.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1635062026}) 2024-04-20T17:20:32.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=634710715}) 2024-04-20T17:20:32.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=895410904}) 2024-04-20T17:20:32.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=59,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=6} crc {logm=351161986,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=2569096738,mon_config_key=4121640298}) 2024-04-20T17:20:32.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=522052159}) 2024-04-20T17:20:32.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1240130756}) 2024-04-20T17:20:32.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=84,monmap=5,osd_metadata=11} crc {mon_config_key=1402996739,monmap=717113318,osd_metadata=3419993334}) 2024-04-20T17:20:32.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=1,osd_pg_creating=1,osdmap=98} crc {osd_metadata=391414970,osd_pg_creating=343897251,osdmap=49590541}) 2024-04-20T17:20:32.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=23} crc {osdmap=3775054600}) 2024-04-20T17:20:32.253 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:32.277 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:20:32.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:32.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:20:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-20T17:20:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:20:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1319786866' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:20:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: osdmap e362: 12 total, 12 up, 12 in 2024-04-20T17:20:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=3640712970,config=1460075751,health=3897446751,kv=931226538}) 2024-04-20T17:20:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {kv=5,logm=95} crc {kv=571464332,logm=1991913918}) 2024-04-20T17:20:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=887203525}) 2024-04-20T17:20:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2340427505}) 2024-04-20T17:20:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3659073397}) 2024-04-20T17:20:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4290301312}) 2024-04-20T17:20:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1473538472}) 2024-04-20T17:20:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2612976217}) 2024-04-20T17:20:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3186309417}) 2024-04-20T17:20:32.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=533910587}) 2024-04-20T17:20:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=963120067}) 2024-04-20T17:20:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2957720207}) 2024-04-20T17:20:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=100713154}) 2024-04-20T17:20:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3890318679}) 2024-04-20T17:20:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=263405216}) 2024-04-20T17:20:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=859103554}) 2024-04-20T17:20:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3581387597}) 2024-04-20T17:20:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3766785077}) 2024-04-20T17:20:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=940264748}) 2024-04-20T17:20:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=563483318}) 2024-04-20T17:20:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4252570098}) 2024-04-20T17:20:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3103627407}) 2024-04-20T17:20:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1152425576}) 2024-04-20T17:20:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2875921274}) 2024-04-20T17:20:32.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2709233972}) 2024-04-20T17:20:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2394141698}) 2024-04-20T17:20:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2535438873}) 2024-04-20T17:20:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1135384172}) 2024-04-20T17:20:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1776805465}) 2024-04-20T17:20:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=17450602}) 2024-04-20T17:20:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=773719723}) 2024-04-20T17:20:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2702381215}) 2024-04-20T17:20:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1799982686}) 2024-04-20T17:20:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3971553314}) 2024-04-20T17:20:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2631835817}) 2024-04-20T17:20:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3602080934}) 2024-04-20T17:20:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1957298077}) 2024-04-20T17:20:32.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=795783997}) 2024-04-20T17:20:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1661234810}) 2024-04-20T17:20:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1955793180}) 2024-04-20T17:20:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2064598678}) 2024-04-20T17:20:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1927754497}) 2024-04-20T17:20:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1635062026}) 2024-04-20T17:20:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=634710715}) 2024-04-20T17:20:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=895410904}) 2024-04-20T17:20:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=59,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=6} crc {logm=351161986,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=2569096738,mon_config_key=4121640298}) 2024-04-20T17:20:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=522052159}) 2024-04-20T17:20:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1240130756}) 2024-04-20T17:20:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=84,monmap=5,osd_metadata=11} crc {mon_config_key=1402996739,monmap=717113318,osd_metadata=3419993334}) 2024-04-20T17:20:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=1,osd_pg_creating=1,osdmap=98} crc {osd_metadata=391414970,osd_pg_creating=343897251,osdmap=49590541}) 2024-04-20T17:20:32.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=23} crc {osdmap=3775054600}) 2024-04-20T17:20:32.381 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:20:32.419 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:20:32.419 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:20:32.419 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 0 2024-04-20T17:20:32.419 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:20:32.426 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-20T17:20:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:20:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1319786866' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:20:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: osdmap e362: 12 total, 12 up, 12 in 2024-04-20T17:20:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=3640712970,config=1460075751,health=3897446751,kv=931226538}) 2024-04-20T17:20:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {kv=5,logm=95} crc {kv=571464332,logm=1991913918}) 2024-04-20T17:20:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=887203525}) 2024-04-20T17:20:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2340427505}) 2024-04-20T17:20:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3659073397}) 2024-04-20T17:20:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4290301312}) 2024-04-20T17:20:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1473538472}) 2024-04-20T17:20:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2612976217}) 2024-04-20T17:20:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3186309417}) 2024-04-20T17:20:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=533910587}) 2024-04-20T17:20:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=963120067}) 2024-04-20T17:20:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2957720207}) 2024-04-20T17:20:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=100713154}) 2024-04-20T17:20:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3890318679}) 2024-04-20T17:20:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=263405216}) 2024-04-20T17:20:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=859103554}) 2024-04-20T17:20:32.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3581387597}) 2024-04-20T17:20:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3766785077}) 2024-04-20T17:20:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=940264748}) 2024-04-20T17:20:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=563483318}) 2024-04-20T17:20:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4252570098}) 2024-04-20T17:20:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3103627407}) 2024-04-20T17:20:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1152425576}) 2024-04-20T17:20:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2875921274}) 2024-04-20T17:20:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2709233972}) 2024-04-20T17:20:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2394141698}) 2024-04-20T17:20:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2535438873}) 2024-04-20T17:20:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1135384172}) 2024-04-20T17:20:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1776805465}) 2024-04-20T17:20:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=17450602}) 2024-04-20T17:20:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=773719723}) 2024-04-20T17:20:32.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2702381215}) 2024-04-20T17:20:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1799982686}) 2024-04-20T17:20:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3971553314}) 2024-04-20T17:20:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2631835817}) 2024-04-20T17:20:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3602080934}) 2024-04-20T17:20:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1957298077}) 2024-04-20T17:20:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=795783997}) 2024-04-20T17:20:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1661234810}) 2024-04-20T17:20:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1955793180}) 2024-04-20T17:20:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2064598678}) 2024-04-20T17:20:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1927754497}) 2024-04-20T17:20:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1635062026}) 2024-04-20T17:20:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=634710715}) 2024-04-20T17:20:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=895410904}) 2024-04-20T17:20:32.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=59,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=6} crc {logm=351161986,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=2569096738,mon_config_key=4121640298}) 2024-04-20T17:20:32.515 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=522052159}) 2024-04-20T17:20:32.515 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1240130756}) 2024-04-20T17:20:32.515 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=84,monmap=5,osd_metadata=11} crc {mon_config_key=1402996739,monmap=717113318,osd_metadata=3419993334}) 2024-04-20T17:20:32.515 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=1,osd_pg_creating=1,osdmap=98} crc {osd_metadata=391414970,osd_pg_creating=343897251,osdmap=49590541}) 2024-04-20T17:20:32.515 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=23} crc {osdmap=3775054600}) 2024-04-20T17:20:32.536 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:20:32.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:32.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:20:32.640 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:20:32.682 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:32.732 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:20:32.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:32.736 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:20:32.790 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 1 16 5873 5857 366.044 366.062 0.00206967 0.00272671 2024-04-20T17:20:32.836 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:20:32.875 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:32.900 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:20:32.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:32.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:20:33.005 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:20:33.048 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:33.092 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:20:33.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:33.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:20:33.201 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:20:33.241 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:33.249 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-20T17:20:33.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:33 smithi106 ceph-mon[44423]: pgmap v985: 17 pgs: 17 active+clean; 5.1 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s wr, 11 op/s 2024-04-20T17:20:33.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:33 smithi171 ceph-mon[44189]: pgmap v985: 17 pgs: 17 active+clean; 5.1 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s wr, 11 op/s 2024-04-20T17:20:33.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:33 smithi005 ceph-mon[39562]: pgmap v985: 17 pgs: 17 active+clean; 5.1 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s wr, 11 op/s 2024-04-20T17:20:33.668 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:20:33.700 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:20:33.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:33.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:20:33.790 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 2 16 11290 11274 352.29 338.562 0.00295257 0.00283399 2024-04-20T17:20:33.805 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:20:33.845 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:33.940 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:20:33.974 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:20:33.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:33.978 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:20:34.079 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:20:34.117 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:34.145 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:20:34.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:34.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:20:34.249 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:20:34.303 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:34.334 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:20:34.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:34.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:20:34.381 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:20:34.400 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:20:34.420 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:20:34.440 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:20:34.452 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:20:34.479 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:34.481 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:20:34.503 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:20:34.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:34.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:20:34.508 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:20:34.542 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:20:34.581 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:20:34.608 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:20:34.615 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:20:34.649 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:20:34.650 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:34.680 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:20:34.717 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:20:34.752 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:20:34.762 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:20:34.762 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:20:34.762 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:20:34.762 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:20:34.762 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:20:34.762 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:20:34.763 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:20:34.763 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:20:34.763 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:20:34.763 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:20:34.763 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:20:34.763 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:20:34.790 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 3 16 15733 15717 327.415 277.688 0.00609604 0.00305204 2024-04-20T17:20:35.207 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:20:35.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:35.211 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:20:35.311 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:20:35.354 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:35 smithi005 ceph-mon[39562]: pgmap v986: 17 pgs: 17 active+clean; 264 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 701 op/s 2024-04-20T17:20:35.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:35 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1213994223' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:20:35.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:35.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:35.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:35.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:35.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:35.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:35.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:35 smithi106 ceph-mon[44423]: pgmap v986: 17 pgs: 17 active+clean; 264 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 701 op/s 2024-04-20T17:20:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:35 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1213994223' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:20:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:35 smithi171 ceph-mon[44189]: pgmap v986: 17 pgs: 17 active+clean; 264 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 701 op/s 2024-04-20T17:20:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:35 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1213994223' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:20:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:35.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:35.790 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 4 16 20457 20441 319.365 295.25 0.00217598 0.00312885 2024-04-20T17:20:36.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:36.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:36.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:36.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:36.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:36.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:36.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:36 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:20:36.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:36 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:20:36.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:36 smithi005 ceph-mon[39562]: osdmap e363: 12 total, 12 up, 12 in 2024-04-20T17:20:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:36 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:20:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:36 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:20:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:36 smithi106 ceph-mon[44423]: osdmap e363: 12 total, 12 up, 12 in 2024-04-20T17:20:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:36 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:20:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:36 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:20:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:36 smithi171 ceph-mon[44189]: osdmap e363: 12 total, 12 up, 12 in 2024-04-20T17:20:36.790 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 5 16 24304 24288 303.573 240.438 0.00153816 0.00327746 2024-04-20T17:20:37.010 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:20:37.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:37.016 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:20:37.117 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:20:37.157 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:37.216 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:20:37.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:37.220 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:20:37.321 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:20:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:37 smithi171 ceph-mon[44189]: pgmap v988: 17 pgs: 17 active+clean; 730 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 121 MiB/s wr, 1.93k op/s 2024-04-20T17:20:37.375 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:37 smithi005 ceph-mon[39562]: pgmap v988: 17 pgs: 17 active+clean; 730 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 121 MiB/s wr, 1.93k op/s 2024-04-20T17:20:37.379 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:37.393 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:20:37.393 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:20:37.410 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:20:37.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:37.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:20:37.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:37 smithi106 ceph-mon[44423]: pgmap v988: 17 pgs: 17 active+clean; 730 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 121 MiB/s wr, 1.93k op/s 2024-04-20T17:20:37.525 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:20:37.559 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:37.591 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:20:37.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:37.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:20:37.676 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:20:37.696 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:20:37.734 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:37.758 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:20:37.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:37.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:20:37.791 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 6 16 28117 28101 292.689 238.312 0.00639777 0.00341487 2024-04-20T17:20:37.862 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:20:37.899 INFO:tasks.thrashosds.thrasher:Recovered, killing an osd 2024-04-20T17:20:37.899 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2] 2024-04-20T17:20:37.899 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-20T17:20:37.900 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.7 2024-04-20T17:20:37.916 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:37.949 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:20:37.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:37.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:20:38.054 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:20:38.093 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:38.110 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:20:38.196 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:20:38.235 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:20:37 smithi106 systemd[1]: Stopping Ceph osd.7 for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:20:38.235 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:38 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4030513257' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:20:38.235 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:38 smithi106 ceph-mon[44423]: from='client.25811 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:20:38.249 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:20:38.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:38.254 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:20:38.278 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-20T17:20:38.352 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:38 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4030513257' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:20:38.353 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:38 smithi171 ceph-mon[44189]: from='client.25811 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:20:38.354 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:20:38.409 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:38.441 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:20:38.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:38.447 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:20:38.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:38 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4030513257' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:20:38.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:38 smithi005 ceph-mon[39562]: from='client.25811 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:20:38.511 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:20:38 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[73608]: 2024-04-20T17:20:38.234+0000 7fc50279c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T17:20:38.511 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:20:38 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[73608]: 2024-04-20T17:20:38.234+0000 7fc50279c700 -1 osd.7 364 *** Got signal Terminated *** 2024-04-20T17:20:38.511 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:20:38 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[73608]: 2024-04-20T17:20:38.234+0000 7fc50279c700 -1 osd.7 364 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T17:20:38.548 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:20:38.596 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:38.623 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:20:38.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:38.627 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:20:38.714 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:20:38.728 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:20:38.767 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:38.791 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 7 16 31602 31586 281.99 217.812 0.00522566 0.00354429 2024-04-20T17:20:38.796 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:20:38.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:38.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:20:38.902 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:20:38.937 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:39.021 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:20:39.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:39.024 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:20:39.125 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:20:39.157 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:39.187 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:20:39.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:39.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:20:39.273 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:39 smithi171 ceph-mon[44189]: pgmap v989: 17 pgs: 17 active+clean; 1.3 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 170 MiB/s wr, 2.72k op/s 2024-04-20T17:20:39.274 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:39 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:20:39.274 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:39 smithi171 ceph-mon[44189]: from='client.25811 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:20:39.274 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:39 smithi171 ceph-mon[44189]: osdmap e364: 12 total, 12 up, 12 in 2024-04-20T17:20:39.274 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:39 smithi171 ceph-mon[44189]: osd.7 marked itself down and dead 2024-04-20T17:20:39.289 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:20:39.315 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:39 smithi005 ceph-mon[39562]: pgmap v989: 17 pgs: 17 active+clean; 1.3 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 170 MiB/s wr, 2.72k op/s 2024-04-20T17:20:39.316 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:39 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:20:39.316 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:39 smithi005 ceph-mon[39562]: from='client.25811 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:20:39.316 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:39 smithi005 ceph-mon[39562]: osdmap e364: 12 total, 12 up, 12 in 2024-04-20T17:20:39.316 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:39 smithi005 ceph-mon[39562]: osd.7 marked itself down and dead 2024-04-20T17:20:39.318 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:39.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:39 smithi106 ceph-mon[44423]: pgmap v989: 17 pgs: 17 active+clean; 1.3 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 170 MiB/s wr, 2.72k op/s 2024-04-20T17:20:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:39 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:20:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:39 smithi106 ceph-mon[44423]: from='client.25811 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:20:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:39 smithi106 ceph-mon[44423]: osdmap e364: 12 total, 12 up, 12 in 2024-04-20T17:20:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:39 smithi106 ceph-mon[44423]: osd.7 marked itself down and dead 2024-04-20T17:20:39.768 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:20:39.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:39.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:20:39.791 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 8 16 33111 33095 258.529 94.3125 0.00171601 0.00358137 2024-04-20T17:20:39.871 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:20:39.902 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:40.150 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:20:40.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:40.154 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:20:40.255 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:20:40.281 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:40.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:40 smithi005 ceph-mon[39562]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T17:20:40.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:40 smithi005 ceph-mon[39562]: osdmap e365: 12 total, 11 up, 12 in 2024-04-20T17:20:40.464 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:40 smithi171 ceph-mon[44189]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T17:20:40.464 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:40 smithi171 ceph-mon[44189]: osdmap e365: 12 total, 11 up, 12 in 2024-04-20T17:20:40.479 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:20:40.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:40.482 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:20:40.510 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:20:40 smithi106 bash[113791]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7 2024-04-20T17:20:40.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:40 smithi106 ceph-mon[44423]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T17:20:40.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:40 smithi106 ceph-mon[44423]: osdmap e365: 12 total, 11 up, 12 in 2024-04-20T17:20:40.582 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:20:40.611 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:40.781 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:20:40.781 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:20:40.791 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 9 16 33111 33095 229.802 0 - 0.00358137 2024-04-20T17:20:41.390 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:41 smithi005 ceph-mon[39562]: pgmap v992: 17 pgs: 3 stale+active+clean, 14 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 222 MiB/s wr, 3.56k op/s 2024-04-20T17:20:41.391 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:41 smithi005 ceph-mon[39562]: osdmap e366: 12 total, 11 up, 12 in 2024-04-20T17:20:41.391 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:41 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1722672773' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:20:41.508 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:41 smithi171 ceph-mon[44189]: pgmap v992: 17 pgs: 3 stale+active+clean, 14 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 222 MiB/s wr, 3.56k op/s 2024-04-20T17:20:41.508 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:41 smithi171 ceph-mon[44189]: osdmap e366: 12 total, 11 up, 12 in 2024-04-20T17:20:41.508 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:41 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1722672773' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:20:41.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:41 smithi106 ceph-mon[44423]: pgmap v992: 17 pgs: 3 stale+active+clean, 14 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 222 MiB/s wr, 3.56k op/s 2024-04-20T17:20:41.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:41 smithi106 ceph-mon[44423]: osdmap e366: 12 total, 11 up, 12 in 2024-04-20T17:20:41.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:41 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1722672773' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:20:41.535 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-20T17:20:41.544 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:20:41.544 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:20:41.544 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:20:41.544 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:20:41.544 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:20:41.544 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:20:41.544 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:20:41.544 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:20:41.544 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:20:41.544 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:20:41.544 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:20:41.545 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:20:41.545 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:20:41.545 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:20:41.545 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:20:41.545 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:20:41.545 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:20:41.545 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:20:41.545 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:20:41.545 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:20:41.545 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:20:41.545 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:20:41.545 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:20:41.589 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:20:41.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:41.592 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:20:41.693 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:20:41.721 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:41.791 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 10 16 33111 33095 206.822 0 - 0.00358137 2024-04-20T17:20:41.794 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:20:41.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:41.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:20:41.898 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:20:41.929 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:41.948 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:20:41.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:41.951 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:20:41.982 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.7.service' 2024-04-20T17:20:42.030 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T17:20:42.030 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-20T17:20:42.031 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd down 7 2024-04-20T17:20:42.052 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:20:42.100 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.405 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.405 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.405 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.405 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.406 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.406 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.406 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.406 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.406 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.406 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.406 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.431 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:20:42.432 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:20:42.449 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:20:42.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:42.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:20:42.486 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:20:42.496 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:20:42.552 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:20:42.583 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:42.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:42.604 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:20:42.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:42.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:20:42.708 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:20:42.742 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:42.764 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:20:42.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:42.767 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:20:42.791 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 11 16 33111 33095 188.019 0 - 0.00358137 2024-04-20T17:20:42.868 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:20:42.897 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:43.020 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:20:43.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:43.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:20:43.124 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:20:43.157 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:43.177 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:20:43.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:43.180 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:20:43.281 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:20:43.307 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:43 smithi005 ceph-mon[39562]: pgmap v994: 17 pgs: 1 undersized+degraded+peered, 2 stale+active+clean, 14 active+clean; 1.9 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 203 MiB/s wr, 3.24k op/s; 2083/62244 objects degraded (3.347%) 2024-04-20T17:20:43.307 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:43 smithi005 ceph-mon[39562]: Health check failed: Degraded data redundancy: 2083/62244 objects degraded (3.347%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:20:43.309 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:43.417 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:20:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:43 smithi106 ceph-mon[44423]: pgmap v994: 17 pgs: 1 undersized+degraded+peered, 2 stale+active+clean, 14 active+clean; 1.9 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 203 MiB/s wr, 3.24k op/s; 2083/62244 objects degraded (3.347%) 2024-04-20T17:20:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:43 smithi106 ceph-mon[44423]: Health check failed: Degraded data redundancy: 2083/62244 objects degraded (3.347%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:20:43.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:43 smithi171 ceph-mon[44189]: pgmap v994: 17 pgs: 1 undersized+degraded+peered, 2 stale+active+clean, 14 active+clean; 1.9 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 203 MiB/s wr, 3.24k op/s; 2083/62244 objects degraded (3.347%) 2024-04-20T17:20:43.607 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:43 smithi171 ceph-mon[44189]: Health check failed: Degraded data redundancy: 2083/62244 objects degraded (3.347%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:20:43.659 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:20:43.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:43.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:20:43.763 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:20:43.791 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 12 16 33111 33095 172.351 0 - 0.00358137 2024-04-20T17:20:43.794 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:43.816 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:20:43.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:43.819 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:20:43.921 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:20:43.951 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:43.972 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:20:43.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:43.975 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:20:44.076 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:20:44.106 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:44.229 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:20:44.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:44.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:20:44.333 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:20:44.366 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:44.387 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:20:44.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:44.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:20:44.417 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:44 smithi005 ceph-mon[39562]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-20T17:20:44.433 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:20:44.491 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:20:44.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:44 smithi106 ceph-mon[44423]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-20T17:20:44.521 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:44 smithi171 ceph-mon[44189]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-20T17:20:44.523 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:44.544 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:20:44.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:44.547 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:20:44.649 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:20:44.680 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:44.791 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 13 16 33111 33095 159.093 0 - 0.00358137 2024-04-20T17:20:44.827 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:20:44.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:44.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:20:44.859 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:20:44.932 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:20:44.961 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:45.135 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:20:45.409 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:20:45.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:45.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:20:45.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:45 smithi106 ceph-mon[44423]: pgmap v995: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 2.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 118 MiB/s wr, 1.89k op/s; 4156/66206 objects degraded (6.277%) 2024-04-20T17:20:45.513 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:20:45.526 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:45 smithi005 ceph-mon[39562]: pgmap v995: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 2.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 118 MiB/s wr, 1.89k op/s; 4156/66206 objects degraded (6.277%) 2024-04-20T17:20:45.544 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:45 smithi171 ceph-mon[44189]: pgmap v995: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 2.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 118 MiB/s wr, 1.89k op/s; 4156/66206 objects degraded (6.277%) 2024-04-20T17:20:45.545 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:45.568 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:20:45.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:45.571 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:20:45.672 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:20:45.702 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:45.792 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 14 16 33111 33095 147.729 0 - 0.00358137 2024-04-20T17:20:46.063 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:46 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mon-a[39539]: 2024-04-20T17:20:46.036+0000 7f73b92a1700 -1 mon.a@0(leader).osd e366 definitely_dead 0 2024-04-20T17:20:46.546 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7 is already down. 2024-04-20T17:20:46.792 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 15 16 33111 33095 137.88 0 - 0.00358137 2024-04-20T17:20:46.853 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:20:46.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:46.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:20:46.958 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:20:46.988 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:47.108 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:47 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3184168081' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["7"]}]: dispatch 2024-04-20T17:20:47.109 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:47 smithi005 ceph-mon[39562]: pgmap v996: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 2.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s; 4156/66206 objects degraded (6.277%) 2024-04-20T17:20:47.112 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:20:47.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:47.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:20:47.222 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:20:47.255 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:47.277 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:20:47.278 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:47 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3184168081' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["7"]}]: dispatch 2024-04-20T17:20:47.279 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:47 smithi106 ceph-mon[44423]: pgmap v996: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 2.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s; 4156/66206 objects degraded (6.277%) 2024-04-20T17:20:47.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:47.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:20:47.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:47 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3184168081' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["7"]}]: dispatch 2024-04-20T17:20:47.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:47 smithi171 ceph-mon[44189]: pgmap v996: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 2.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s; 4156/66206 objects degraded (6.277%) 2024-04-20T17:20:47.380 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:20:47.414 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:47.434 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:20:47.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:47.437 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:20:47.538 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:20:47.562 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-20T17:20:47.562 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd out 7 2024-04-20T17:20:47.569 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:47.792 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 16 16 33111 33095 129.262 0 - 0.00358137 2024-04-20T17:20:47.821 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:20:47.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:47.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:20:47.925 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:20:47.958 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:47.978 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:20:47.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:47.982 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:20:48.083 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:20:48.113 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:48.211 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:20:48.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:48.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:20:48.315 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:20:48.347 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:48.368 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:20:48.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:48.372 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:20:48.473 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:20:48.503 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:48.524 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:20:48.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:48.527 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:20:48.628 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:20:48.656 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:48.694 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:20:48.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:48.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:20:48.792 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 17 16 33111 33095 121.658 0 - 0.00358137 2024-04-20T17:20:48.798 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:20:48.825 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:48.853 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:20:48.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:48.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:20:48.962 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:20:48.995 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:49.016 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:20:49.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:49.020 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:20:49.121 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:20:49.149 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:49.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:49 smithi171 ceph-mon[44189]: pgmap v997: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 2.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 845 op/s; 4156/66206 objects degraded (6.277%) 2024-04-20T17:20:49.381 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:20:49.398 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:20:49.400 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:49 smithi005 ceph-mon[39562]: pgmap v997: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 2.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 845 op/s; 4156/66206 objects degraded (6.277%) 2024-04-20T17:20:49.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:49.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:20:49.502 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:20:49.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:49 smithi106 ceph-mon[44423]: pgmap v997: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 2.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 845 op/s; 4156/66206 objects degraded (6.277%) 2024-04-20T17:20:49.535 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:49.556 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:20:49.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:49.559 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:20:49.660 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:20:49.689 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:49.710 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:20:49.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:49.713 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:20:49.792 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 18 16 33111 33095 114.9 0 - 0.00358137 2024-04-20T17:20:49.814 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:20:49.842 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:49.861 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:20:49.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:49.864 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:20:49.965 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:20:49.993 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:50.012 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:20:50.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:50.015 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:20:50.116 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:20:50.150 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:50.171 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:20:50.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:50.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:20:50.276 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:20:50.307 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:50.327 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:20:50.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:50.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:20:50.431 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:20:50.459 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:50.478 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:20:50.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:50.480 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:20:50.581 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:20:50.616 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:50.673 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:20:50.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:50.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:20:50.777 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:20:50.792 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 19 16 33111 33095 108.852 0 - 0.00358137 2024-04-20T17:20:50.807 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:50.932 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:20:50.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:50.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:20:51.037 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:20:51.065 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:51.085 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:20:51.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:51.088 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:20:51.189 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:20:51.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:51 smithi005 ceph-mon[39562]: pgmap v998: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 2.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 753 op/s; 4156/66206 objects degraded (6.277%) 2024-04-20T17:20:51.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:51 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 4156/66206 objects degraded (6.277%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:20:51.215 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:51.235 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:20:51.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:51.238 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:20:51.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:51 smithi171 ceph-mon[44189]: pgmap v998: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 2.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 753 op/s; 4156/66206 objects degraded (6.277%) 2024-04-20T17:20:51.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:51 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 4156/66206 objects degraded (6.277%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:20:51.339 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:20:51.365 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:51.383 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:20:51.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:51.387 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:20:51.488 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:20:51.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:51 smithi106 ceph-mon[44423]: pgmap v998: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 2.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 753 op/s; 4156/66206 objects degraded (6.277%) 2024-04-20T17:20:51.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:51 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 4156/66206 objects degraded (6.277%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:20:51.518 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:51.792 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:20:51.793659 min lat: 0.000946083 max lat: 0.0749339 avg lat: 0.00358137 2024-04-20T17:20:51.792 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:20:51.793 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 20 16 33111 33095 103.41 0 - 0.00358137 2024-04-20T17:20:52.000 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:20:52.000 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:20:52.001 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:20:52.001 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:20:52.001 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:20:52.001 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:20:52.001 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:20:52.001 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:20:52.001 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:20:52.002 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:20:52.002 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:20:52.002 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:20:52.002 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:20:52.002 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:20:52.002 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:20:52.002 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:20:52.003 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:20:52.003 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:20:52.003 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:20:52.003 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:20:52.003 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:20:52.003 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:20:52.003 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:20:52.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:52 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/751415576' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:20:52.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:52.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:52.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:52.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:52.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:52.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:52.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:52.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:52 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/751415576' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:20:52.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:52.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:52.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:52.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:52.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:52.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:52.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:52.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:52 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/751415576' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:20:52.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:52.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:52.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:52.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:52.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:52.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:52.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:52.792 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 21 16 33111 33095 98.4856 0 - 0.00358137 2024-04-20T17:20:52.976 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:20:52.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:52.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:20:53.081 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:20:53.114 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:53.135 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:20:53.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:53.138 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:20:53.168 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:20:53.239 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:20:53.269 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:53.290 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:20:53.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:53.293 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:20:53.394 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:20:53.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:53 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:53.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:53 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:53.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:53 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:53.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:53 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:53.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:53 smithi005 ceph-mon[39562]: pgmap v999: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 2.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 633 op/s; 4156/66206 objects degraded (6.277%) 2024-04-20T17:20:53.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:53 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/98296105' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:20:53.423 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:53.442 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:20:53.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:53.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:20:53.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:53 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:53.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:53 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:53.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:53 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:53.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:53 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:53.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:53 smithi106 ceph-mon[44423]: pgmap v999: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 2.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 633 op/s; 4156/66206 objects degraded (6.277%) 2024-04-20T17:20:53.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:53 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/98296105' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:20:53.546 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:20:53.575 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:53 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:53.575 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:53 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:53.575 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:53 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:53.575 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:53 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:20:53.575 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:53 smithi171 ceph-mon[44189]: pgmap v999: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 2.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 633 op/s; 4156/66206 objects degraded (6.277%) 2024-04-20T17:20:53.575 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:53 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/98296105' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:20:53.577 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:53.598 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:20:53.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:53.601 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:20:53.702 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:20:53.735 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:53.756 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:20:53.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:53.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:20:53.793 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 22 16 33111 33095 94.0088 0 - 0.00358137 2024-04-20T17:20:53.861 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:20:53.891 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:53.909 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:20:53.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:53.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:20:54.013 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:20:54.049 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:54.055 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:20:54.056 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:20:54.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:54 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mon-a[39539]: 2024-04-20T17:20:54.104+0000 7f73b92a1700 -1 mon.a@0(leader).osd e367 definitely_dead 0 2024-04-20T17:20:54.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:54 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:20:54.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:54 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/98296105' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:20:54.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:54 smithi005 ceph-mon[39562]: osdmap e367: 12 total, 11 up, 12 in 2024-04-20T17:20:54.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:54 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3327918582' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-04-20T17:20:54.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:54 smithi005 ceph-mon[39562]: Client client.admin marked osd.7 out, after it was down for 14 seconds 2024-04-20T17:20:54.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:54 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:20:54.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:54 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/98296105' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:20:54.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:54 smithi106 ceph-mon[44423]: osdmap e367: 12 total, 11 up, 12 in 2024-04-20T17:20:54.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:54 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3327918582' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-04-20T17:20:54.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:54 smithi106 ceph-mon[44423]: Client client.admin marked osd.7 out, after it was down for 14 seconds 2024-04-20T17:20:54.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:54 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:20:54.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:54 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/98296105' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:20:54.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:54 smithi171 ceph-mon[44189]: osdmap e367: 12 total, 11 up, 12 in 2024-04-20T17:20:54.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:54 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3327918582' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-04-20T17:20:54.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:54 smithi171 ceph-mon[44189]: Client client.admin marked osd.7 out, after it was down for 14 seconds 2024-04-20T17:20:54.793 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 23 16 33111 33095 89.9214 0 - 0.00358137 2024-04-20T17:20:54.909 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:20:54.910 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:20:55.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:55 smithi005 ceph-mon[39562]: pgmap v1001: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 2.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4156/66206 objects degraded (6.277%) 2024-04-20T17:20:55.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:55 smithi005 ceph-mon[39562]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T17:20:55.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:55 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3327918582' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-04-20T17:20:55.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:55 smithi005 ceph-mon[39562]: osdmap e368: 12 total, 11 up, 11 in 2024-04-20T17:20:55.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:55 smithi106 ceph-mon[44423]: pgmap v1001: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 2.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4156/66206 objects degraded (6.277%) 2024-04-20T17:20:55.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:55 smithi106 ceph-mon[44423]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T17:20:55.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:55 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3327918582' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-04-20T17:20:55.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:55 smithi106 ceph-mon[44423]: osdmap e368: 12 total, 11 up, 11 in 2024-04-20T17:20:55.540 INFO:teuthology.orchestra.run.smithi005.stderr:marked out osd.7. 2024-04-20T17:20:55.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:55 smithi171 ceph-mon[44189]: pgmap v1001: 17 pgs: 3 undersized+degraded+peered, 14 active+clean; 2.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4156/66206 objects degraded (6.277%) 2024-04-20T17:20:55.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:55 smithi171 ceph-mon[44189]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T17:20:55.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:55 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3327918582' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-04-20T17:20:55.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:55 smithi171 ceph-mon[44189]: osdmap e368: 12 total, 11 up, 11 in 2024-04-20T17:20:55.793 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 24 16 34398 34382 89.526 5.02734 0.0040688 0.0111671 2024-04-20T17:20:56.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:56 smithi005 ceph-mon[39562]: osdmap e369: 12 total, 11 up, 11 in 2024-04-20T17:20:56.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:56 smithi005 ceph-mon[39562]: osdmap e370: 12 total, 11 up, 11 in 2024-04-20T17:20:56.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:56 smithi106 ceph-mon[44423]: osdmap e369: 12 total, 11 up, 11 in 2024-04-20T17:20:56.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:56 smithi106 ceph-mon[44423]: osdmap e370: 12 total, 11 up, 11 in 2024-04-20T17:20:56.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:56 smithi171 ceph-mon[44189]: osdmap e369: 12 total, 11 up, 11 in 2024-04-20T17:20:56.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:56 smithi171 ceph-mon[44189]: osdmap e370: 12 total, 11 up, 11 in 2024-04-20T17:20:56.793 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 25 16 37046 37030 92.5644 165.5 0.00356794 0.0108003 2024-04-20T17:20:56.910 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:20:57.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:57 smithi005 ceph-mon[39562]: pgmap v1005: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 2.0 GiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 7.2 MiB/s wr, 115 op/s; 4125/67132 objects degraded (6.145%); 496 KiB/s, 7 objects/s recovering 2024-04-20T17:20:57.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:57 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 4125/67132 objects degraded (6.145%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:20:57.447 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:20:57.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:57 smithi106 ceph-mon[44423]: pgmap v1005: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 2.0 GiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 7.2 MiB/s wr, 115 op/s; 4125/67132 objects degraded (6.145%); 496 KiB/s, 7 objects/s recovering 2024-04-20T17:20:57.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:57 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 4125/67132 objects degraded (6.145%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:20:57.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:57 smithi171 ceph-mon[44189]: pgmap v1005: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 2.0 GiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 7.2 MiB/s wr, 115 op/s; 4125/67132 objects degraded (6.145%); 496 KiB/s, 7 objects/s recovering 2024-04-20T17:20:57.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:57 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 4125/67132 objects degraded (6.145%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:20:57.793 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 26 16 39303 39287 94.4292 141.062 0.00171097 0.0105714 2024-04-20T17:20:58.439 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:20:58.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:58.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:20:58.492 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:58 smithi005 ceph-mon[39562]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-20T17:20:58.493 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.7 2024-04-20T17:20:58.494 INFO:tasks.thrashosds.thrasher:remote for osd 7 is ubuntu@smithi106.front.sepia.ceph.com 2024-04-20T17:20:58.494 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.7 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op list-pgs 2024-04-20T17:20:58.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:58 smithi106 ceph-mon[44423]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-20T17:20:58.544 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:20:58.578 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:58.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:58 smithi171 ceph-mon[44189]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-20T17:20:58.604 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:20:58.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:58.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:20:58.710 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:20:58.752 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:58.757 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.7/config 2024-04-20T17:20:58.794 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 27 15 41636 41621 96.3319 145.875 0.00748297 0.0103786 2024-04-20T17:20:59.015 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:20:59.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:59.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:20:59.065 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-20T17:20:59.075 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:20:59.123 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:20:59.137 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:20:59.161 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:59.269 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:20:59.271 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:20:59 smithi106 ceph-mon[44423]: pgmap v1006: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 2.2 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 32 MiB/s wr, 511 op/s; 3912/71262 objects degraded (5.490%); 3.2 MiB/s, 49 objects/s recovering 2024-04-20T17:20:59.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:59.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:20:59.373 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:20:59.395 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:20:59 smithi005 ceph-mon[39562]: pgmap v1006: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 2.2 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 32 MiB/s wr, 511 op/s; 3912/71262 objects degraded (5.490%); 3.2 MiB/s, 49 objects/s recovering 2024-04-20T17:20:59.415 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:59.447 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:20:59.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:59.451 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:20:59.552 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:20:59.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:20:59 smithi171 ceph-mon[44189]: pgmap v1006: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 2.2 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 32 MiB/s wr, 511 op/s; 3912/71262 objects degraded (5.490%); 3.2 MiB/s, 49 objects/s recovering 2024-04-20T17:20:59.599 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:59.630 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:20:59.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:59.637 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:20:59.738 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:20:59.777 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:59.794 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 28 16 44027 44011 98.2258 149.375 0.0072311 0.0101782 2024-04-20T17:20:59.808 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:20:59.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:59.812 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:20:59.913 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:20:59.965 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:20:59.986 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:20:59.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:20:59.989 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:21:00.090 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:21:00.132 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:00.158 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:21:00.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:00.161 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:21:00.172 INFO:teuthology.orchestra.run.smithi106.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:21:00.262 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:21:00.309 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:00.347 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:21:00.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:00.353 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:21:00.454 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:21:00.482 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:00 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:21:00.482 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:00 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:21:00.498 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:00.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:00 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:21:00.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:00 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:21:00.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:00 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:21:00.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:00 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:21:00.794 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 29 16 46447 46431 100.054 151.25 0.0078027 0.00999314 2024-04-20T17:21:01.261 INFO:teuthology.orchestra.run.smithi106.stdout:3.f 2024-04-20T17:21:01.261 INFO:teuthology.orchestra.run.smithi106.stdout:1.0 2024-04-20T17:21:01.261 INFO:teuthology.orchestra.run.smithi106.stdout:3.d 2024-04-20T17:21:01.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:01 smithi106 ceph-mon[44423]: pgmap v1007: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 2.3 GiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 44 MiB/s wr, 701 op/s; 3912/74620 objects degraded (5.243%); 2.6 MiB/s, 40 objects/s recovering 2024-04-20T17:21:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:01 smithi106 ceph-mon[44423]: osdmap e371: 12 total, 11 up, 11 in 2024-04-20T17:21:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:01 smithi106 ceph-mon[44423]: osdmap e372: 12 total, 11 up, 11 in 2024-04-20T17:21:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:01 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:21:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:01 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:21:01.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:01 smithi171 ceph-mon[44189]: pgmap v1007: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 2.3 GiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 44 MiB/s wr, 701 op/s; 3912/74620 objects degraded (5.243%); 2.6 MiB/s, 40 objects/s recovering 2024-04-20T17:21:01.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:01 smithi171 ceph-mon[44189]: osdmap e371: 12 total, 11 up, 11 in 2024-04-20T17:21:01.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:01 smithi171 ceph-mon[44189]: osdmap e372: 12 total, 11 up, 11 in 2024-04-20T17:21:01.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:01 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:21:01.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:01 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:21:01.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:01 smithi005 ceph-mon[39562]: pgmap v1007: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 2.3 GiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 44 MiB/s wr, 701 op/s; 3912/74620 objects degraded (5.243%); 2.6 MiB/s, 40 objects/s recovering 2024-04-20T17:21:01.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:01 smithi005 ceph-mon[39562]: osdmap e371: 12 total, 11 up, 11 in 2024-04-20T17:21:01.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:01 smithi005 ceph-mon[39562]: osdmap e372: 12 total, 11 up, 11 in 2024-04-20T17:21:01.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:01 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:21:01.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:01 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:21:01.794 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 30 16 48889 48873 101.806 152.625 0.0170096 0.00982021 2024-04-20T17:21:01.969 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:21:01.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:01.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:21:01.991 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:21:02.073 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:21:02.112 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:02.317 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:21:02.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:02.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:21:02.340 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.7 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op export --pgid 1.0 --file /var/log/ceph/exp.1.0.7 2024-04-20T17:21:02.421 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:21:02.460 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:02.612 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.7/config 2024-04-20T17:21:02.794 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 31 16 51153 51137 103.086 141.5 0.0141937 0.00969871 2024-04-20T17:21:03.115 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:03 smithi106 ceph-mon[44423]: osdmap e373: 12 total, 11 up, 11 in 2024-04-20T17:21:03.115 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:03 smithi106 ceph-mon[44423]: pgmap v1011: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 2.6 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 102 MiB/s wr, 1.64k op/s; 3361/86768 objects degraded (3.874%); 8.0 MiB/s, 127 objects/s recovering 2024-04-20T17:21:03.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:03 smithi171 ceph-mon[44189]: osdmap e373: 12 total, 11 up, 11 in 2024-04-20T17:21:03.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:03 smithi171 ceph-mon[44189]: pgmap v1011: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 2.6 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 102 MiB/s wr, 1.64k op/s; 3361/86768 objects degraded (3.874%); 8.0 MiB/s, 127 objects/s recovering 2024-04-20T17:21:03.436 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:21:03.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:03 smithi005 ceph-mon[39562]: osdmap e373: 12 total, 11 up, 11 in 2024-04-20T17:21:03.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:03 smithi005 ceph-mon[39562]: pgmap v1011: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 2.6 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 102 MiB/s wr, 1.64k op/s; 3361/86768 objects degraded (3.874%); 8.0 MiB/s, 127 objects/s recovering 2024-04-20T17:21:03.469 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:21:03.503 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:21:03.535 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:21:03.567 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:21:03.598 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:21:03.629 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:21:03.670 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:21:03.695 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:21:03.733 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:21:03.780 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:21:03.781 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:21:03.781 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:21:03.781 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:21:03.781 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:21:03.781 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:21:03.781 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:21:03.781 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:21:03.781 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:21:03.781 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:21:03.781 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:21:03.781 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:21:03.782 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:21:03.794 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 32 16 53438 53422 104.327 142.812 0.00662855 0.0095832 2024-04-20T17:21:04.010 INFO:teuthology.orchestra.run.smithi106.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:21:04.053 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:04 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 3361/86768 objects degraded (3.874%), 2 pgs degraded (PG_DEGRADED) 2024-04-20T17:21:04.053 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:04 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1554915887' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:21:04.053 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:04 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:04.054 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:04 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:04.054 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:04 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:04.054 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:04 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:04.054 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:04 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:04.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:04 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 3361/86768 objects degraded (3.874%), 2 pgs degraded (PG_DEGRADED) 2024-04-20T17:21:04.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:04 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1554915887' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:21:04.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:04 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:04.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:04 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:04.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:04 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:04.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:04 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:04.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:04 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:04.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:04 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 3361/86768 objects degraded (3.874%), 2 pgs degraded (PG_DEGRADED) 2024-04-20T17:21:04.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:04 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1554915887' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:21:04.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:04 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:04.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:04 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:04.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:04 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:04.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:04 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:04.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:04 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:04.794 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 33 16 55699 55683 105.447 141.312 0.00573134 0.00948165 2024-04-20T17:21:05.069 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:21:05.098 INFO:teuthology.orchestra.run.smithi106.stderr:Exporting 1.0 info 1.0( v 15'137 (0'0,15'137] local-lis/les=93/94 n=2 ec=14/14 lis/c=93/93 les/c/f=94/94/0 sis=93) 2024-04-20T17:21:05.112 INFO:teuthology.orchestra.run.smithi106.stderr:Read #1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head# 2024-04-20T17:21:05.112 INFO:teuthology.orchestra.run.smithi106.stderr:Read #1:30306672:devicehealth::main.db.0000000000000000:head# 2024-04-20T17:21:05.113 INFO:teuthology.orchestra.run.smithi106.stderr:Export successful 2024-04-20T17:21:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:05 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4195384472' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:21:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:05 smithi171 ceph-mon[44189]: pgmap v1012: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 3.0 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 141 MiB/s wr, 2.25k op/s; 2889/98302 objects degraded (2.939%); 11 MiB/s, 170 objects/s recovering 2024-04-20T17:21:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:05 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:21:05.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:05 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:21:05.365 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:21:05.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:05.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:21:05.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:05.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:05.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:05.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:05.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:05.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:05.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:05 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4195384472' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:21:05.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:05 smithi005 ceph-mon[39562]: pgmap v1012: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 3.0 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 141 MiB/s wr, 2.25k op/s; 2889/98302 objects degraded (2.939%); 11 MiB/s, 170 objects/s recovering 2024-04-20T17:21:05.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:05 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:21:05.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:05 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:21:05.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:05.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:05.403 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:05.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:05.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:05.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:05.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:05 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4195384472' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:21:05.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:05 smithi106 ceph-mon[44423]: pgmap v1012: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 3.0 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 141 MiB/s wr, 2.25k op/s; 2889/98302 objects degraded (2.939%); 11 MiB/s, 170 objects/s recovering 2024-04-20T17:21:05.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:05 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:21:05.404 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:05 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:21:05.469 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:21:05.513 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:05.549 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:21:05.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:05.553 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:21:05.654 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:21:05.700 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:05.729 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:21:05.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:05.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:21:05.794 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 34 16 57853 57837 106.305 134.625 0.00365434 0.00940461 2024-04-20T17:21:05.834 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:21:05.882 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:05.902 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:21:05.903 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:21:05.903 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:21:05.903 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:21:05.903 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:21:05.909 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:21:05.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:05.913 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:21:06.014 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:21:06.061 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:06.154 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:06 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:21:06.155 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:06 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4195384472' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:21:06.155 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:06 smithi106 ceph-mon[44423]: osdmap e374: 12 total, 11 up, 11 in 2024-04-20T17:21:06.155 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:06 smithi106 ceph-mon[44423]: osdmap e375: 12 total, 11 up, 11 in 2024-04-20T17:21:06.166 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:21:06.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:06.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:21:06.191 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.7 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --force --op remove --pgid 1.0 2024-04-20T17:21:06.270 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:21:06.310 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:06 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:21:06.310 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:06 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4195384472' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:21:06.310 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:06 smithi005 ceph-mon[39562]: osdmap e374: 12 total, 11 up, 11 in 2024-04-20T17:21:06.310 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:06 smithi005 ceph-mon[39562]: osdmap e375: 12 total, 11 up, 11 in 2024-04-20T17:21:06.315 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:06.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:06 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:21:06.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:06 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4195384472' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:21:06.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:06 smithi171 ceph-mon[44189]: osdmap e374: 12 total, 11 up, 11 in 2024-04-20T17:21:06.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:06 smithi171 ceph-mon[44189]: osdmap e375: 12 total, 11 up, 11 in 2024-04-20T17:21:06.508 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.7/config 2024-04-20T17:21:06.571 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:21:06.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:06.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:21:06.678 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:21:06.722 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:06.750 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:21:06.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:06.756 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:21:06.794 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 35 16 60020 60004 107.137 135.438 0.00868874 0.00933233 2024-04-20T17:21:06.857 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:21:06.896 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:06.924 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:21:06.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:06.927 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:21:07.028 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:21:07.076 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:07.078 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:07 smithi106 ceph-mon[44423]: pgmap v1015: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 3.3 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 143 MiB/s wr, 2.28k op/s; 2316/108506 objects degraded (2.134%); 13 MiB/s, 205 objects/s recovering 2024-04-20T17:21:07.110 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:21:07.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:07.114 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:21:07.171 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:21:07.171 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:21:07.215 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:21:07.270 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:07.303 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:21:07.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:07.307 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:21:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:07 smithi171 ceph-mon[44189]: pgmap v1015: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 3.3 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 143 MiB/s wr, 2.28k op/s; 2316/108506 objects degraded (2.134%); 13 MiB/s, 205 objects/s recovering 2024-04-20T17:21:07.408 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:21:07.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:07 smithi005 ceph-mon[39562]: pgmap v1015: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 3.3 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 143 MiB/s wr, 2.28k op/s; 2316/108506 objects degraded (2.134%); 13 MiB/s, 205 objects/s recovering 2024-04-20T17:21:07.466 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:07.499 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:21:07.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:07.503 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:21:07.604 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:21:07.644 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:07.671 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:21:07.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:07.675 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:21:07.776 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:21:07.794 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 36 16 61989 61973 107.579 123.062 0.0131515 0.00929325 2024-04-20T17:21:07.813 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:08.067 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:21:08.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:08.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:21:08.102 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-20T17:21:08.174 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:21:08.211 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:08.241 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:21:08.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:08.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:21:08.291 INFO:teuthology.orchestra.run.smithi106.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:21:08.347 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:21:08.393 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:08.436 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:21:08.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:08.439 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:21:08.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:08 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 1813/116500 objects degraded (1.556%), 2 pgs degraded (PG_DEGRADED) 2024-04-20T17:21:08.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:08 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 1813/116500 objects degraded (1.556%), 2 pgs degraded (PG_DEGRADED) 2024-04-20T17:21:08.527 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:21:08.540 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:21:08.581 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:08.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:08 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 1813/116500 objects degraded (1.556%), 2 pgs degraded (PG_DEGRADED) 2024-04-20T17:21:08.591 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:21:08.794 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 37 16 64038 64022 108.132 128.062 0.00920904 0.00924625 2024-04-20T17:21:08.843 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:21:08.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:08.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:21:08.951 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:21:09.000 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:09.134 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:21:09.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:09.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:21:09.172 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:21:09.240 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:21:09.275 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:09.301 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:21:09.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:09.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:21:09.409 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:21:09.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:09 smithi005 ceph-mon[39562]: pgmap v1016: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 3.6 GiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 153 MiB/s wr, 2.45k op/s; 1813/116500 objects degraded (1.556%); 16 MiB/s, 255 objects/s recovering 2024-04-20T17:21:09.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:09 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:21:09.471 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:09.507 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:21:09.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:09.510 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:21:09.510 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:09 smithi171 ceph-mon[44189]: pgmap v1016: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 3.6 GiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 153 MiB/s wr, 2.45k op/s; 1813/116500 objects degraded (1.556%); 16 MiB/s, 255 objects/s recovering 2024-04-20T17:21:09.510 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:09 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:21:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:09 smithi106 ceph-mon[44423]: pgmap v1016: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 3.6 GiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 153 MiB/s wr, 2.45k op/s; 1813/116500 objects degraded (1.556%); 16 MiB/s, 255 objects/s recovering 2024-04-20T17:21:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:09 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:21:09.554 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:21:09.611 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:21:09.654 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:09.684 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:21:09.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:09.688 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:21:09.789 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:21:09.797 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 38 16 65778 65762 108.144 108.75 0.0124655 0.00924409 2024-04-20T17:21:09.833 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:09.859 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:21:09.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:09.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:21:09.963 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:21:10.002 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:10.039 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:21:10.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:10.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:21:10.134 INFO:teuthology.orchestra.run.smithi106.stdout: marking collection for removal 2024-04-20T17:21:10.135 INFO:teuthology.orchestra.run.smithi106.stdout:setting '_remove' omap key 2024-04-20T17:21:10.138 INFO:teuthology.orchestra.run.smithi106.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-04-20T17:21:10.143 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:21:10.147 INFO:teuthology.orchestra.run.smithi106.stdout:Remove successful 2024-04-20T17:21:10.174 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:10.196 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:21:10.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:10.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:21:10.268 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:21:10.300 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:21:10.344 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:10.366 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:21:10.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:10.369 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:21:10.470 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:21:10.493 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:10 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:21:10.493 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:10 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:21:10.493 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:10 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:21:10.493 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:10 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:21:10.494 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:10 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:21:10.494 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:10 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:21:10.494 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:10 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:21:10.518 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:10.543 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:21:10.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:10.548 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:21:10.649 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:21:10.704 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:10.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:10 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:21:10.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:10 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:21:10.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:10 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:21:10.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:10 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:21:10.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:10 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:21:10.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:10 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:21:10.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:10 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:21:10.796 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 39 16 67694 67678 108.441 119.75 0.00796749 0.00922031 2024-04-20T17:21:10.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:10 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:21:10.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:10 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:21:10.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:10 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:21:10.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:10 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:21:10.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:10 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:21:10.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:10 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:21:10.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:10 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:21:11.460 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:21:11.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:11.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:21:11.506 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.7 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.1.0.7 2024-04-20T17:21:11.564 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:21:11.606 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:11.632 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:21:11.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:11.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:21:11.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:11 smithi005 ceph-mon[39562]: pgmap v1017: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 3.7 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 132 MiB/s wr, 2.11k op/s; 1813/120528 objects degraded (1.504%); 12 MiB/s, 193 objects/s recovering 2024-04-20T17:21:11.739 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:21:11.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:11 smithi106 ceph-mon[44423]: pgmap v1017: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 3.7 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 132 MiB/s wr, 2.11k op/s; 1813/120528 objects degraded (1.504%); 12 MiB/s, 193 objects/s recovering 2024-04-20T17:21:11.781 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:11.789 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.7/config 2024-04-20T17:21:11.796 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:21:11.797570 min lat: 0.000946083 max lat: 16.1837 avg lat: 0.00917926 2024-04-20T17:21:11.796 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:21:11.797 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 40 16 69734 69718 108.917 127.5 0.00832424 0.00917926 2024-04-20T17:21:11.809 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:21:11.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:11.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:21:11.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:11 smithi171 ceph-mon[44189]: pgmap v1017: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 3.7 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 132 MiB/s wr, 2.11k op/s; 1813/120528 objects degraded (1.504%); 12 MiB/s, 193 objects/s recovering 2024-04-20T17:21:11.915 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:21:11.961 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:12.283 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:21:12.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:12.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:21:12.392 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:21:12.431 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:12.509 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:21:12.509 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:21:12.509 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:21:12.510 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:21:12.510 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.510 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212446 2.e 2:72d00b8c:::benchmark_data_smithi171_78918_object51452:head [delete] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:21:12.510 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897321+0000", 2024-04-20T17:21:12.510 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 529.00019646099997, 2024-04-20T17:21:12.510 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.67883147099999996, 2024-04-20T17:21:12.511 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:12.511 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:12.511 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:12.511 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:12.511 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:12.511 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:12.512 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212446 2024-04-20T17:21:12.512 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.512 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:12.512 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.512 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:12.512 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897321+0000", 2024-04-20T17:21:12.513 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.513 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.513 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.513 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:12.513 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897321+0000", 2024-04-20T17:21:12.514 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.514 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.514 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.514 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:12.514 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897323+0000", 2024-04-20T17:21:12.514 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.424e-06 2024-04-20T17:21:12.515 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.515 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.515 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:12.515 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897329+0000", 2024-04-20T17:21:12.515 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5559999999999998e-06 2024-04-20T17:21:12.516 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.516 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.516 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:12.516 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897338+0000", 2024-04-20T17:21:12.516 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0520000000000007e-06 2024-04-20T17:21:12.516 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.516 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.517 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:12.517 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897346+0000", 2024-04-20T17:21:12.517 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7619999999999995e-06 2024-04-20T17:21:12.517 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.517 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.517 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.518 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898452+0000", 2024-04-20T17:21:12.518 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0011067480000000001 2024-04-20T17:21:12.518 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.518 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.518 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:21:12.519 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898470+0000", 2024-04-20T17:21:12.519 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7864e-05 2024-04-20T17:21:12.519 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.519 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.519 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.519 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.559597+0000", 2024-04-20T17:21:12.520 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.66112692699999998 2024-04-20T17:21:12.520 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.520 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.520 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:12.520 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.559698+0000", 2024-04-20T17:21:12.521 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000101 2024-04-20T17:21:12.521 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.521 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.521 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:21:12.521 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.559788+0000", 2024-04-20T17:21:12.521 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9697999999999994e-05 2024-04-20T17:21:12.522 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.522 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.522 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:12.522 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.575978+0000", 2024-04-20T17:21:12.522 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.016189985 2024-04-20T17:21:12.522 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.523 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.523 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:12.523 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576031+0000", 2024-04-20T17:21:12.523 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3086e-05 2024-04-20T17:21:12.523 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.523 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.524 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:12.524 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576097+0000", 2024-04-20T17:21:12.524 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6303999999999994e-05 2024-04-20T17:21:12.524 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.524 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.525 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:12.525 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576152+0000", 2024-04-20T17:21:12.525 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5065000000000002e-05 2024-04-20T17:21:12.525 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.525 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:12.525 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.526 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.526 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.526 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212450 2.e 2:72ccd005:::benchmark_data_smithi171_78918_object51456:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:21:12.526 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897360+0000", 2024-04-20T17:21:12.526 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 529.00015683900006, 2024-04-20T17:21:12.526 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.67896632000000001, 2024-04-20T17:21:12.527 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:12.527 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:12.527 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:12.527 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:12.527 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:12.528 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:12.528 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212450 2024-04-20T17:21:12.528 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.528 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:12.528 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.528 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:12.529 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897360+0000", 2024-04-20T17:21:12.529 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.529 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.529 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.529 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:12.529 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897360+0000", 2024-04-20T17:21:12.530 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.530 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.530 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.530 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:12.530 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897361+0000", 2024-04-20T17:21:12.530 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2800000000000005e-07 2024-04-20T17:21:12.531 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.531 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.531 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:12.531 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897364+0000", 2024-04-20T17:21:12.531 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.977e-06 2024-04-20T17:21:12.531 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.532 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.532 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:12.532 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897370+0000", 2024-04-20T17:21:12.532 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0530000000000003e-06 2024-04-20T17:21:12.532 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.532 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.533 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:12.533 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897376+0000", 2024-04-20T17:21:12.533 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1419999999999998e-06 2024-04-20T17:21:12.533 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.533 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.533 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.534 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898518+0000", 2024-04-20T17:21:12.534 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0011413 2024-04-20T17:21:12.534 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.534 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.534 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:21:12.535 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898526+0000", 2024-04-20T17:21:12.535 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0150000000000003e-06 2024-04-20T17:21:12.535 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.535 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.535 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.535 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.563095+0000", 2024-04-20T17:21:12.536 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.664568838 2024-04-20T17:21:12.536 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.536 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.536 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:12.536 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.563218+0000", 2024-04-20T17:21:12.536 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000123051 2024-04-20T17:21:12.537 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.537 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.537 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:21:12.537 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.563315+0000", 2024-04-20T17:21:12.537 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6935000000000006e-05 2024-04-20T17:21:12.537 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.538 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.538 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:12.538 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576186+0000", 2024-04-20T17:21:12.538 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012871698000000001 2024-04-20T17:21:12.538 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.538 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.539 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:12.539 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576223+0000", 2024-04-20T17:21:12.539 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7048999999999997e-05 2024-04-20T17:21:12.539 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.539 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.539 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:12.540 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576275+0000", 2024-04-20T17:21:12.540 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1944000000000003e-05 2024-04-20T17:21:12.540 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.540 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.540 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:12.540 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576327+0000", 2024-04-20T17:21:12.541 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1390000000000001e-05 2024-04-20T17:21:12.541 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.541 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:12.541 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.541 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.541 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.542 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212454 2.e 2:7603a42b:::benchmark_data_smithi171_78918_object51460:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:21:12.542 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897383+0000", 2024-04-20T17:21:12.542 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 529.000134269, 2024-04-20T17:21:12.542 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.67906811499999997, 2024-04-20T17:21:12.542 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:12.543 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:12.543 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:12.543 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:12.543 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:12.543 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:12.544 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212454 2024-04-20T17:21:12.544 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.544 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:12.544 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.544 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:12.544 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897383+0000", 2024-04-20T17:21:12.545 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.545 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.545 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.545 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:12.545 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897383+0000", 2024-04-20T17:21:12.545 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.546 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.546 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.546 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:12.546 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897383+0000", 2024-04-20T17:21:12.546 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8599999999999998e-07 2024-04-20T17:21:12.547 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.547 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.547 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:12.547 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897385+0000", 2024-04-20T17:21:12.547 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.041e-06 2024-04-20T17:21:12.547 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.548 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.548 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:12.548 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897393+0000", 2024-04-20T17:21:12.548 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3730000000000004e-06 2024-04-20T17:21:12.548 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.548 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.549 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:12.549 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897396+0000", 2024-04-20T17:21:12.549 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2449999999999998e-06 2024-04-20T17:21:12.549 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.549 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.550 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.550 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898572+0000", 2024-04-20T17:21:12.550 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001176329 2024-04-20T17:21:12.550 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.550 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.550 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:21:12.551 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898581+0000", 2024-04-20T17:21:12.551 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6989999999999992e-06 2024-04-20T17:21:12.551 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.551 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.551 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.552 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.566135+0000", 2024-04-20T17:21:12.552 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.66755413200000002 2024-04-20T17:21:12.552 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.552 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.552 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:12.552 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.566275+0000", 2024-04-20T17:21:12.553 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014009600000000001 2024-04-20T17:21:12.553 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.553 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.553 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:21:12.553 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.566419+0000", 2024-04-20T17:21:12.554 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014339600000000001 2024-04-20T17:21:12.554 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.554 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.554 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:12.554 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576374+0000", 2024-04-20T17:21:12.554 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0099556760000000001 2024-04-20T17:21:12.555 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.555 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.555 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:12.555 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576383+0000", 2024-04-20T17:21:12.555 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0920000000000004e-06 2024-04-20T17:21:12.556 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.556 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.556 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:12.556 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576412+0000", 2024-04-20T17:21:12.556 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8642000000000001e-05 2024-04-20T17:21:12.556 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.557 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.557 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:12.557 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576451+0000", 2024-04-20T17:21:12.557 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8908000000000001e-05 2024-04-20T17:21:12.557 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.557 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:12.558 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.558 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.558 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.558 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212458 2.e 2:7672b154:::benchmark_data_smithi171_78918_object51464:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:21:12.558 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897400+0000", 2024-04-20T17:21:12.558 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 529.00011717100006, 2024-04-20T17:21:12.559 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.68587742500000004, 2024-04-20T17:21:12.559 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:12.559 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:12.559 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:12.559 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:12.559 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:12.560 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:12.560 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212458 2024-04-20T17:21:12.560 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.560 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:12.560 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.560 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:12.561 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897400+0000", 2024-04-20T17:21:12.561 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.561 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.561 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.561 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:12.562 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897400+0000", 2024-04-20T17:21:12.562 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.562 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.562 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.562 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:12.562 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897401+0000", 2024-04-20T17:21:12.563 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3000000000000001e-07 2024-04-20T17:21:12.563 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.563 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.563 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:12.563 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897403+0000", 2024-04-20T17:21:12.564 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.198e-06 2024-04-20T17:21:12.564 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.564 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.564 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:12.564 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897408+0000", 2024-04-20T17:21:12.564 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1109999999999999e-06 2024-04-20T17:21:12.565 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.565 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.565 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:12.565 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897414+0000", 2024-04-20T17:21:12.565 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7139999999999998e-06 2024-04-20T17:21:12.565 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.566 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.566 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.566 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898608+0000", 2024-04-20T17:21:12.566 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0011941160000000001 2024-04-20T17:21:12.566 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.567 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.567 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:21:12.567 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898614+0000", 2024-04-20T17:21:12.567 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7100000000000001e-06 2024-04-20T17:21:12.567 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.567 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.568 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.568 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.569428+0000", 2024-04-20T17:21:12.568 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.67081384399999999 2024-04-20T17:21:12.568 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.568 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.569 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:12.569 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.569562+0000", 2024-04-20T17:21:12.569 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000134055 2024-04-20T17:21:12.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.569 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:21:12.569 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.569692+0000", 2024-04-20T17:21:12.570 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012998099999999999 2024-04-20T17:21:12.570 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.570 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.570 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:12.570 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576458+0000", 2024-04-20T17:21:12.571 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0067652670000000002 2024-04-20T17:21:12.571 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.571 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.571 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:12.571 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.583158+0000", 2024-04-20T17:21:12.571 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0067007229999999996 2024-04-20T17:21:12.572 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.572 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:12.572 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.583225+0000", 2024-04-20T17:21:12.572 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6617000000000003e-05 2024-04-20T17:21:12.572 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.573 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.573 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:12.573 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.583277+0000", 2024-04-20T17:21:12.573 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2559000000000003e-05 2024-04-20T17:21:12.573 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.573 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:12.574 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.574 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.574 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.574 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212462 2.e 2:7f179a5c:::benchmark_data_smithi171_78918_object51468:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:21:12.574 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897424+0000", 2024-04-20T17:21:12.574 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 529.00009283500003, 2024-04-20T17:21:12.575 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.69171569799999999, 2024-04-20T17:21:12.575 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:12.575 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:12.575 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:12.575 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:12.575 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:12.576 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:12.576 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212462 2024-04-20T17:21:12.576 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.576 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:12.576 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.577 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:12.577 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897424+0000", 2024-04-20T17:21:12.577 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.577 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.577 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.577 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:12.578 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897424+0000", 2024-04-20T17:21:12.578 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.578 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.578 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.578 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:12.578 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897425+0000", 2024-04-20T17:21:12.579 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6199999999999997e-07 2024-04-20T17:21:12.579 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.579 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.579 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:12.579 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897428+0000", 2024-04-20T17:21:12.580 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1219999999999999e-06 2024-04-20T17:21:12.580 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.580 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.580 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:12.580 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897436+0000", 2024-04-20T17:21:12.580 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3270000000000001e-06 2024-04-20T17:21:12.581 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.581 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.581 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:12.581 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897442+0000", 2024-04-20T17:21:12.581 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6200000000000004e-06 2024-04-20T17:21:12.581 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.582 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.582 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.582 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898653+0000", 2024-04-20T17:21:12.582 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0012105550000000001 2024-04-20T17:21:12.582 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.583 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.583 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:21:12.583 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898660+0000", 2024-04-20T17:21:12.583 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1810000000000004e-06 2024-04-20T17:21:12.583 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.583 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.583 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.583 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.573380+0000", 2024-04-20T17:21:12.583 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.67471982699999999 2024-04-20T17:21:12.583 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.583 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.583 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:12.583 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.573481+0000", 2024-04-20T17:21:12.583 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010135499999999999 2024-04-20T17:21:12.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.584 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:21:12.584 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.573564+0000", 2024-04-20T17:21:12.584 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3258999999999997e-05 2024-04-20T17:21:12.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.584 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:12.584 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576463+0000", 2024-04-20T17:21:12.584 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0028989609999999998 2024-04-20T17:21:12.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.584 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:12.584 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.589024+0000", 2024-04-20T17:21:12.584 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012561249999999999 2024-04-20T17:21:12.585 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.585 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.585 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:12.585 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.589088+0000", 2024-04-20T17:21:12.585 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3398000000000005e-05 2024-04-20T17:21:12.585 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.585 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.585 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:12.585 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.589140+0000", 2024-04-20T17:21:12.585 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2181000000000001e-05 2024-04-20T17:21:12.585 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.585 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:12.585 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.585 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.585 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.586 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212469 2.e 2:7df2bad1:::benchmark_data_smithi171_78918_object51475:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:21:12.586 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897447+0000", 2024-04-20T17:21:12.586 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 529.00007064099998, 2024-04-20T17:21:12.586 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.71189628299999996, 2024-04-20T17:21:12.586 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:12.586 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:12.586 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:12.586 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:12.586 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:12.586 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:12.586 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212469 2024-04-20T17:21:12.586 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.586 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:12.586 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.587 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:12.587 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897447+0000", 2024-04-20T17:21:12.587 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.587 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.587 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.587 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:12.587 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897447+0000", 2024-04-20T17:21:12.587 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.587 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.587 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.587 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:12.587 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897447+0000", 2024-04-20T17:21:12.587 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.89e-07 2024-04-20T17:21:12.587 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.587 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.588 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:12.588 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897449+0000", 2024-04-20T17:21:12.588 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3530000000000002e-06 2024-04-20T17:21:12.588 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.588 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.588 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:12.588 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897455+0000", 2024-04-20T17:21:12.588 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2850000000000002e-06 2024-04-20T17:21:12.588 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.588 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.588 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:12.588 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897459+0000", 2024-04-20T17:21:12.588 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7129999999999996e-06 2024-04-20T17:21:12.588 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.588 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.589 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.589 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898683+0000", 2024-04-20T17:21:12.589 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0012234629999999999 2024-04-20T17:21:12.589 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.589 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.589 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:21:12.589 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898690+0000", 2024-04-20T17:21:12.589 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9419999999999998e-06 2024-04-20T17:21:12.589 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.589 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.589 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.589 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576505+0000", 2024-04-20T17:21:12.589 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.677814888 2024-04-20T17:21:12.589 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.589 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.590 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:12.590 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576644+0000", 2024-04-20T17:21:12.590 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013927000000000001 2024-04-20T17:21:12.590 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.590 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.590 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:21:12.590 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.576783+0000", 2024-04-20T17:21:12.590 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013961799999999999 2024-04-20T17:21:12.590 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.590 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.590 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:12.590 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.586310+0000", 2024-04-20T17:21:12.590 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0095266459999999997 2024-04-20T17:21:12.590 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.591 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.591 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:12.591 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609211+0000", 2024-04-20T17:21:12.591 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.022901248999999999 2024-04-20T17:21:12.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.591 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.591 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:12.591 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609274+0000", 2024-04-20T17:21:12.591 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3045000000000004e-05 2024-04-20T17:21:12.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.591 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.591 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:12.591 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609343+0000", 2024-04-20T17:21:12.591 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8422000000000003e-05 2024-04-20T17:21:12.591 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.592 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:12.592 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.592 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.592 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.592 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212495 2.e 2:782e28d2:::benchmark_data_smithi171_78918_object51501:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:21:12.592 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897465+0000", 2024-04-20T17:21:12.592 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 529.00005238999995, 2024-04-20T17:21:12.592 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.71202960599999998, 2024-04-20T17:21:12.592 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:12.592 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:12.592 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:12.592 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:12.592 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:12.592 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:12.592 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212495 2024-04-20T17:21:12.592 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.593 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:12.593 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.593 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:12.593 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897465+0000", 2024-04-20T17:21:12.593 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.593 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.593 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.593 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:12.593 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897465+0000", 2024-04-20T17:21:12.593 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.593 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.593 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.593 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:12.593 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897465+0000", 2024-04-20T17:21:12.594 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.68e-07 2024-04-20T17:21:12.594 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.594 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.594 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:12.594 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897468+0000", 2024-04-20T17:21:12.594 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7800000000000001e-06 2024-04-20T17:21:12.594 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.594 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.594 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:12.594 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897475+0000", 2024-04-20T17:21:12.594 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1069999999999999e-06 2024-04-20T17:21:12.594 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.594 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.594 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:12.594 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897481+0000", 2024-04-20T17:21:12.595 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3539999999999999e-06 2024-04-20T17:21:12.595 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.595 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.595 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898736+0000", 2024-04-20T17:21:12.595 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001255812 2024-04-20T17:21:12.595 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.595 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:21:12.595 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898743+0000", 2024-04-20T17:21:12.595 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6830000000000002e-06 2024-04-20T17:21:12.595 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.595 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.595 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.580475+0000", 2024-04-20T17:21:12.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.68173151600000004 2024-04-20T17:21:12.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.596 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:12.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.580557+0000", 2024-04-20T17:21:12.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2826999999999998e-05 2024-04-20T17:21:12.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.596 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:21:12.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.580649+0000", 2024-04-20T17:21:12.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1611000000000001e-05 2024-04-20T17:21:12.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.596 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:12.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.589156+0000", 2024-04-20T17:21:12.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0085067349999999996 2024-04-20T17:21:12.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.597 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:12.597 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609403+0000", 2024-04-20T17:21:12.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.020247660000000001 2024-04-20T17:21:12.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.597 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:12.597 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609449+0000", 2024-04-20T17:21:12.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5649999999999998e-05 2024-04-20T17:21:12.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.597 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:12.598 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609494+0000", 2024-04-20T17:21:12.598 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5303000000000003e-05 2024-04-20T17:21:12.598 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.598 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:12.598 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.598 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.598 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.598 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212512 2.e 2:7c5add18:::benchmark_data_smithi171_78918_object51518:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:21:12.598 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897487+0000", 2024-04-20T17:21:12.598 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 529.00003045400001, 2024-04-20T17:21:12.598 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.71708918399999999, 2024-04-20T17:21:12.598 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:12.598 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:12.598 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:12.598 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:12.599 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:12.599 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:12.599 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212512 2024-04-20T17:21:12.599 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.599 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:12.599 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.599 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:12.599 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897487+0000", 2024-04-20T17:21:12.599 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.599 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.599 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.599 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:12.599 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897487+0000", 2024-04-20T17:21:12.599 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.599 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.600 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.600 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:12.600 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897487+0000", 2024-04-20T17:21:12.600 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9399999999999995e-07 2024-04-20T17:21:12.600 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.600 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.600 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:12.600 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897498+0000", 2024-04-20T17:21:12.600 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0565e-05 2024-04-20T17:21:12.600 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.600 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.600 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:12.600 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897503+0000", 2024-04-20T17:21:12.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6590000000000004e-06 2024-04-20T17:21:12.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:12.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897507+0000", 2024-04-20T17:21:12.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.805e-06 2024-04-20T17:21:12.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898819+0000", 2024-04-20T17:21:12.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00131193 2024-04-20T17:21:12.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:21:12.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898826+0000", 2024-04-20T17:21:12.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6760000000000001e-06 2024-04-20T17:21:12.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.583338+0000", 2024-04-20T17:21:12.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.68451254699999997 2024-04-20T17:21:12.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:12.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.583467+0000", 2024-04-20T17:21:12.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000129016 2024-04-20T17:21:12.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.603 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:21:12.603 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.583627+0000", 2024-04-20T17:21:12.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000159658 2024-04-20T17:21:12.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.603 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:12.603 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.591594+0000", 2024-04-20T17:21:12.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0079671770000000006 2024-04-20T17:21:12.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.603 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:12.603 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.614474+0000", 2024-04-20T17:21:12.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.022880211000000001 2024-04-20T17:21:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.614528+0000", 2024-04-20T17:21:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3721000000000003e-05 2024-04-20T17:21:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.614576+0000", 2024-04-20T17:21:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7725000000000001e-05 2024-04-20T17:21:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.604 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212527 2.e 2:73e27e1e:::benchmark_data_smithi171_78918_object51533:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:21:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897511+0000", 2024-04-20T17:21:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 529.00000603199999, 2024-04-20T17:21:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.71722206099999997, 2024-04-20T17:21:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212527 2024-04-20T17:21:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:12.605 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897511+0000", 2024-04-20T17:21:12.606 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.606 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.606 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.606 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:12.606 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897511+0000", 2024-04-20T17:21:12.606 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.606 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.606 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.606 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:12.606 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897512+0000", 2024-04-20T17:21:12.606 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3999999999999997e-07 2024-04-20T17:21:12.606 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.606 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.607 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:12.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897513+0000", 2024-04-20T17:21:12.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.302e-06 2024-04-20T17:21:12.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.607 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:12.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897516+0000", 2024-04-20T17:21:12.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4060000000000001e-06 2024-04-20T17:21:12.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.607 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:12.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897519+0000", 2024-04-20T17:21:12.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6840000000000001e-06 2024-04-20T17:21:12.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.608 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.608 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898869+0000", 2024-04-20T17:21:12.608 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0013499180000000001 2024-04-20T17:21:12.608 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.608 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.608 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:21:12.608 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898876+0000", 2024-04-20T17:21:12.608 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4490000000000002e-06 2024-04-20T17:21:12.608 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.608 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.608 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.608 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.586356+0000", 2024-04-20T17:21:12.608 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.68747954499999997 2024-04-20T17:21:12.608 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.608 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.609 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:12.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.586502+0000", 2024-04-20T17:21:12.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014652999999999999 2024-04-20T17:21:12.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.609 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:21:12.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.586643+0000", 2024-04-20T17:21:12.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014104100000000001 2024-04-20T17:21:12.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.609 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:12.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.591604+0000", 2024-04-20T17:21:12.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0049606090000000004 2024-04-20T17:21:12.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.610 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.610 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:12.610 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.614639+0000", 2024-04-20T17:21:12.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023034974 2024-04-20T17:21:12.610 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.610 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.610 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:12.610 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.614690+0000", 2024-04-20T17:21:12.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1134999999999997e-05 2024-04-20T17:21:12.610 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.610 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.610 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:12.610 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.614733+0000", 2024-04-20T17:21:12.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3127999999999998e-05 2024-04-20T17:21:12.611 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.611 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:12.611 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.611 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.611 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212531 2.e 2:7eb9e3fb:::benchmark_data_smithi171_78918_object51537:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:21:12.611 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897525+0000", 2024-04-20T17:21:12.611 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 528.99999225299996, 2024-04-20T17:21:12.611 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.71744083700000005, 2024-04-20T17:21:12.611 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:12.611 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:12.611 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:12.611 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:12.611 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:12.611 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:12.612 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212531 2024-04-20T17:21:12.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.612 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:12.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.612 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:12.612 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897525+0000", 2024-04-20T17:21:12.612 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.612 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:12.612 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897525+0000", 2024-04-20T17:21:12.612 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.613 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:12.613 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897525+0000", 2024-04-20T17:21:12.613 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5600000000000001e-07 2024-04-20T17:21:12.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.613 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.613 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:12.613 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897527+0000", 2024-04-20T17:21:12.613 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9410000000000001e-06 2024-04-20T17:21:12.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.613 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.613 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:12.613 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897534+0000", 2024-04-20T17:21:12.613 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1199999999999996e-06 2024-04-20T17:21:12.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.613 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.614 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:12.614 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897539+0000", 2024-04-20T17:21:12.614 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.921e-06 2024-04-20T17:21:12.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.614 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.614 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898909+0000", 2024-04-20T17:21:12.614 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001369706 2024-04-20T17:21:12.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.614 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:21:12.614 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898915+0000", 2024-04-20T17:21:12.614 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4160000000000001e-06 2024-04-20T17:21:12.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.615 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.615 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.589187+0000", 2024-04-20T17:21:12.615 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.69027111600000002 2024-04-20T17:21:12.615 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.615 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:12.615 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.589267+0000", 2024-04-20T17:21:12.615 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0399000000000001e-05 2024-04-20T17:21:12.615 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.615 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:21:12.615 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.589349+0000", 2024-04-20T17:21:12.615 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1904999999999996e-05 2024-04-20T17:21:12.615 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.616 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:12.616 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.600413+0000", 2024-04-20T17:21:12.616 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.01106378 2024-04-20T17:21:12.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.616 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:12.616 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.614882+0000", 2024-04-20T17:21:12.616 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014469556999999999 2024-04-20T17:21:12.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.616 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:12.616 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.614926+0000", 2024-04-20T17:21:12.616 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3649000000000002e-05 2024-04-20T17:21:12.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.617 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.617 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:12.617 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.614966+0000", 2024-04-20T17:21:12.617 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9971000000000001e-05 2024-04-20T17:21:12.617 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.617 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:12.617 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.617 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.617 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212541 2.e 2:7a53380b:::benchmark_data_smithi171_78918_object51547:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:21:12.617 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897543+0000", 2024-04-20T17:21:12.617 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 528.99997395599996, 2024-04-20T17:21:12.617 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.720308693, 2024-04-20T17:21:12.617 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:12.618 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:12.618 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:12.618 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:12.618 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:12.618 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:12.618 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212541 2024-04-20T17:21:12.618 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.618 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:12.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.618 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:12.618 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897543+0000", 2024-04-20T17:21:12.618 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.618 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.618 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:12.619 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897543+0000", 2024-04-20T17:21:12.619 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.619 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.619 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:12.619 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897544+0000", 2024-04-20T17:21:12.619 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5100000000000001e-07 2024-04-20T17:21:12.619 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.619 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:12.619 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897545+0000", 2024-04-20T17:21:12.619 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.412e-06 2024-04-20T17:21:12.619 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.619 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:12.620 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897548+0000", 2024-04-20T17:21:12.620 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.858e-06 2024-04-20T17:21:12.620 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.620 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.620 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:12.620 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897551+0000", 2024-04-20T17:21:12.620 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0189999999999998e-06 2024-04-20T17:21:12.620 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.620 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.620 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.620 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898951+0000", 2024-04-20T17:21:12.620 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0013997440000000001 2024-04-20T17:21:12.620 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.620 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.620 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:21:12.621 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898958+0000", 2024-04-20T17:21:12.621 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4580000000000004e-06 2024-04-20T17:21:12.621 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.621 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.621 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.621 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.591641+0000", 2024-04-20T17:21:12.621 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.69268292399999998 2024-04-20T17:21:12.621 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.621 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.621 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:12.621 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.591724+0000", 2024-04-20T17:21:12.621 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3249000000000002e-05 2024-04-20T17:21:12.621 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.621 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.621 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:21:12.622 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.591816+0000", 2024-04-20T17:21:12.622 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1774000000000006e-05 2024-04-20T17:21:12.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.622 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:12.622 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.600419+0000", 2024-04-20T17:21:12.622 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0086027299999999994 2024-04-20T17:21:12.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.622 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:12.622 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.617785+0000", 2024-04-20T17:21:12.622 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.017365914999999999 2024-04-20T17:21:12.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.623 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:12.623 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.617819+0000", 2024-04-20T17:21:12.623 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4743000000000003e-05 2024-04-20T17:21:12.623 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.623 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:12.623 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.617852+0000", 2024-04-20T17:21:12.623 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2515999999999998e-05 2024-04-20T17:21:12.623 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.623 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:12.623 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.623 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.623 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212573 2.e 2:73f6f3d3:::benchmark_data_smithi171_78918_object51579:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:21:12.624 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897554+0000", 2024-04-20T17:21:12.624 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 528.99996360299997, 2024-04-20T17:21:12.624 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.72045646900000004, 2024-04-20T17:21:12.624 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:12.624 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:12.624 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:12.624 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:12.624 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:12.624 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:12.624 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212573 2024-04-20T17:21:12.624 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.624 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:12.624 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.624 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:12.624 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897554+0000", 2024-04-20T17:21:12.625 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.625 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.625 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:12.625 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897554+0000", 2024-04-20T17:21:12.625 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.625 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.625 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:12.625 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897554+0000", 2024-04-20T17:21:12.625 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8900000000000001e-07 2024-04-20T17:21:12.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.625 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.625 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:12.626 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897555+0000", 2024-04-20T17:21:12.626 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.063e-06 2024-04-20T17:21:12.626 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.626 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.626 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:12.626 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897558+0000", 2024-04-20T17:21:12.626 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3400000000000002e-06 2024-04-20T17:21:12.626 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.626 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.626 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:12.626 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897566+0000", 2024-04-20T17:21:12.626 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6629999999999993e-06 2024-04-20T17:21:12.626 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.626 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.626 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.627 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.898998+0000", 2024-04-20T17:21:12.627 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001432326 2024-04-20T17:21:12.627 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.627 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.627 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:21:12.627 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.899005+0000", 2024-04-20T17:21:12.627 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.737e-06 2024-04-20T17:21:12.627 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.627 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.627 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.627 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.594204+0000", 2024-04-20T17:21:12.627 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.69519878599999996 2024-04-20T17:21:12.627 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.627 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.628 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:12.628 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.594324+0000", 2024-04-20T17:21:12.628 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000119827 2024-04-20T17:21:12.628 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.628 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.628 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:21:12.628 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.594406+0000", 2024-04-20T17:21:12.628 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2727999999999998e-05 2024-04-20T17:21:12.628 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.628 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.628 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:12.628 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.605467+0000", 2024-04-20T17:21:12.628 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011060449999999999 2024-04-20T17:21:12.628 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.628 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.629 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:12.629 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.617903+0000", 2024-04-20T17:21:12.629 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012436361 2024-04-20T17:21:12.629 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.629 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.629 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:12.629 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.617964+0000", 2024-04-20T17:21:12.629 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0569999999999998e-05 2024-04-20T17:21:12.629 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.629 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.629 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:12.629 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.618010+0000", 2024-04-20T17:21:12.629 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6329000000000003e-05 2024-04-20T17:21:12.629 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.629 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:12.630 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.630 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.630 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.630 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212593 2.e 2:744300f2:::benchmark_data_smithi171_78918_object51599:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:21:12.630 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897569+0000", 2024-04-20T17:21:12.630 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 528.99994771800004, 2024-04-20T17:21:12.630 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.72464595399999998, 2024-04-20T17:21:12.630 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:12.630 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:12.630 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:12.630 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:12.630 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:12.630 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:12.630 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212593 2024-04-20T17:21:12.630 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.631 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:12.631 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.631 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:12.631 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897569+0000", 2024-04-20T17:21:12.631 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.631 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.631 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.631 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:12.631 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897569+0000", 2024-04-20T17:21:12.631 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.631 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.631 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.631 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:12.631 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897570+0000", 2024-04-20T17:21:12.631 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3999999999999997e-07 2024-04-20T17:21:12.632 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.632 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.632 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:12.632 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897571+0000", 2024-04-20T17:21:12.632 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.296e-06 2024-04-20T17:21:12.632 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.632 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.632 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:12.632 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897574+0000", 2024-04-20T17:21:12.632 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7970000000000001e-06 2024-04-20T17:21:12.632 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.632 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.632 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:12.632 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897577+0000", 2024-04-20T17:21:12.632 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8389999999999998e-06 2024-04-20T17:21:12.633 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.633 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.633 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.633 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.899045+0000", 2024-04-20T17:21:12.633 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0014678639999999999 2024-04-20T17:21:12.633 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.633 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.633 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:21:12.633 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.899052+0000", 2024-04-20T17:21:12.633 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3010000000000001e-06 2024-04-20T17:21:12.633 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.633 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.633 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.633 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.597702+0000", 2024-04-20T17:21:12.633 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.69865043599999999 2024-04-20T17:21:12.634 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.634 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.634 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:12.634 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.597847+0000", 2024-04-20T17:21:12.634 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000144291 2024-04-20T17:21:12.634 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.634 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.634 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:21:12.634 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.598003+0000", 2024-04-20T17:21:12.634 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000156615 2024-04-20T17:21:12.634 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.634 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.634 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:12.634 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609358+0000", 2024-04-20T17:21:12.635 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011354991 2024-04-20T17:21:12.635 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.635 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.635 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:12.635 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.622109+0000", 2024-04-20T17:21:12.635 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012751035000000001 2024-04-20T17:21:12.635 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.635 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.635 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:12.635 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.622164+0000", 2024-04-20T17:21:12.635 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4867000000000002e-05 2024-04-20T17:21:12.635 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.635 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.635 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:12.636 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.622215+0000", 2024-04-20T17:21:12.636 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1282000000000001e-05 2024-04-20T17:21:12.636 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.636 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:12.636 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.636 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.636 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.636 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212638 2.e 2:7e5004ea:::benchmark_data_smithi171_78918_object51644:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:21:12.636 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897579+0000", 2024-04-20T17:21:12.636 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 528.99993776099996, 2024-04-20T17:21:12.636 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.73145174000000002, 2024-04-20T17:21:12.636 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:12.636 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:12.636 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:12.636 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:12.637 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:12.637 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:12.637 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212638 2024-04-20T17:21:12.637 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.637 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:12.637 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.637 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:12.637 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897579+0000", 2024-04-20T17:21:12.637 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.637 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.637 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.637 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:12.637 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897579+0000", 2024-04-20T17:21:12.637 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.638 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.638 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.638 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:12.638 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897580+0000", 2024-04-20T17:21:12.638 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9900000000000002e-07 2024-04-20T17:21:12.638 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.638 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.638 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:12.638 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897581+0000", 2024-04-20T17:21:12.638 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0380000000000001e-06 2024-04-20T17:21:12.638 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.638 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.638 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:12.638 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897585+0000", 2024-04-20T17:21:12.638 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7479999999999999e-06 2024-04-20T17:21:12.639 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.639 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.639 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:12.639 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897594+0000", 2024-04-20T17:21:12.639 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7899999999999994e-06 2024-04-20T17:21:12.639 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.639 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.639 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.639 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.899085+0000", 2024-04-20T17:21:12.639 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0014909649999999999 2024-04-20T17:21:12.639 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.639 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.639 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:21:12.639 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.899091+0000", 2024-04-20T17:21:12.640 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8170000000000004e-06 2024-04-20T17:21:12.640 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.640 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.640 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.640 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.600462+0000", 2024-04-20T17:21:12.640 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.70137081800000001 2024-04-20T17:21:12.640 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.640 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.640 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:12.640 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.600548+0000", 2024-04-20T17:21:12.640 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5815000000000004e-05 2024-04-20T17:21:12.640 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.640 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.640 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:21:12.640 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.600638+0000", 2024-04-20T17:21:12.640 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9998000000000001e-05 2024-04-20T17:21:12.641 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.641 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.641 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:12.641 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609366+0000", 2024-04-20T17:21:12.641 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0087284849999999994 2024-04-20T17:21:12.641 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.641 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.641 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:12.641 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.628925+0000", 2024-04-20T17:21:12.641 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.019558467999999999 2024-04-20T17:21:12.641 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.642 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.642 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:12.642 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.628978+0000", 2024-04-20T17:21:12.642 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.329e-05 2024-04-20T17:21:12.642 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.642 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.642 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:12.642 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.629031+0000", 2024-04-20T17:21:12.642 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3208999999999998e-05 2024-04-20T17:21:12.642 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.643 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:12.643 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.643 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.643 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.643 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212664 2.e 2:78efb1e3:::benchmark_data_smithi171_78918_object51670:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:21:12.643 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897598+0000", 2024-04-20T17:21:12.643 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 528.99991929700002, 2024-04-20T17:21:12.643 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.73297929500000003, 2024-04-20T17:21:12.643 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:12.643 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:12.643 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:12.643 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:12.643 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:12.643 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:12.644 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212664 2024-04-20T17:21:12.644 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.644 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:12.644 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.644 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:12.644 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897598+0000", 2024-04-20T17:21:12.644 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.644 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.644 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.644 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:12.644 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897598+0000", 2024-04-20T17:21:12.644 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.644 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.644 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.644 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:12.645 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897598+0000", 2024-04-20T17:21:12.645 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6300000000000001e-07 2024-04-20T17:21:12.645 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.645 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.645 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:12.645 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897600+0000", 2024-04-20T17:21:12.645 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.451e-06 2024-04-20T17:21:12.645 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.645 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.645 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:12.645 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897603+0000", 2024-04-20T17:21:12.645 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9950000000000001e-06 2024-04-20T17:21:12.645 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.645 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.646 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:12.646 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897605+0000", 2024-04-20T17:21:12.646 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6400000000000001e-06 2024-04-20T17:21:12.646 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.646 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.646 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.646 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.899129+0000", 2024-04-20T17:21:12.646 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001523567 2024-04-20T17:21:12.646 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.646 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.646 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:21:12.646 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.899136+0000", 2024-04-20T17:21:12.646 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4880000000000001e-06 2024-04-20T17:21:12.647 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.647 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.647 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.647 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.602930+0000", 2024-04-20T17:21:12.647 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.703793365 2024-04-20T17:21:12.647 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.647 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.647 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:12.647 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.603007+0000", 2024-04-20T17:21:12.647 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7519000000000001e-05 2024-04-20T17:21:12.647 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.647 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.647 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:21:12.647 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.603094+0000", 2024-04-20T17:21:12.648 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6271000000000001e-05 2024-04-20T17:21:12.648 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.648 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.648 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:12.648 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.613185+0000", 2024-04-20T17:21:12.648 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010091126000000001 2024-04-20T17:21:12.648 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.648 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.648 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:12.648 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.630457+0000", 2024-04-20T17:21:12.648 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.017272526 2024-04-20T17:21:12.648 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.649 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.649 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:12.649 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.630522+0000", 2024-04-20T17:21:12.649 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4422000000000001e-05 2024-04-20T17:21:12.649 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.649 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.649 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:12.649 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.630577+0000", 2024-04-20T17:21:12.649 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5562000000000001e-05 2024-04-20T17:21:12.649 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.649 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:12.649 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.649 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.649 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.649 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212665 2.e 2:734f527b:::benchmark_data_smithi171_78918_object51671:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e202)", 2024-04-20T17:21:12.650 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:22.897609+0000", 2024-04-20T17:21:12.650 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 528.999907893, 2024-04-20T17:21:12.650 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.73311006000000001, 2024-04-20T17:21:12.650 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:12.650 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:12.650 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:12.650 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:12.650 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:12.650 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:12.650 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212665 2024-04-20T17:21:12.650 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.650 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:12.650 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.651 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:12.651 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897609+0000", 2024-04-20T17:21:12.651 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.651 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.651 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.651 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:12.651 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897609+0000", 2024-04-20T17:21:12.651 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.651 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.651 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.651 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:12.651 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897610+0000", 2024-04-20T17:21:12.651 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4400000000000001e-07 2024-04-20T17:21:12.652 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.652 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.652 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:12.652 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897612+0000", 2024-04-20T17:21:12.652 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.824e-06 2024-04-20T17:21:12.652 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.652 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.652 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:12.653 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897618+0000", 2024-04-20T17:21:12.653 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4330000000000001e-06 2024-04-20T17:21:12.653 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.653 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.653 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:12.653 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.897621+0000", 2024-04-20T17:21:12.653 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9359999999999999e-06 2024-04-20T17:21:12.653 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.653 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.653 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.653 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.899161+0000", 2024-04-20T17:21:12.654 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001540198 2024-04-20T17:21:12.654 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.654 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.654 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:21:12.654 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:22.899169+0000", 2024-04-20T17:21:12.654 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4989999999999997e-06 2024-04-20T17:21:12.654 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.654 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.654 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.655 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.605497+0000", 2024-04-20T17:21:12.655 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.70632853799999995 2024-04-20T17:21:12.655 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.655 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.655 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:12.655 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.605602+0000", 2024-04-20T17:21:12.656 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000105147 2024-04-20T17:21:12.656 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.656 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.656 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:21:12.656 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.605676+0000", 2024-04-20T17:21:12.656 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4098999999999996e-05 2024-04-20T17:21:12.657 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.657 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.657 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:12.657 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.613193+0000", 2024-04-20T17:21:12.657 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0075168830000000002 2024-04-20T17:21:12.657 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.657 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.658 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:12.658 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.630633+0000", 2024-04-20T17:21:12.658 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.017440124000000001 2024-04-20T17:21:12.658 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.658 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.658 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:12.659 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.630681+0000", 2024-04-20T17:21:12.659 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7673999999999998e-05 2024-04-20T17:21:12.659 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.659 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.659 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:12.659 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.630719+0000", 2024-04-20T17:21:12.659 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8260999999999998e-05 2024-04-20T17:21:12.660 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.660 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:12.660 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.660 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.660 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.660 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:212685 2.e 2:7acb7d43:::benchmark_data_smithi171_78918_object51691:head [delete] snapc 0=[] ondisk+write+known_if_redirected e203)", 2024-04-20T17:21:12.660 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:23.604266+0000", 2024-04-20T17:21:12.661 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 528.29325126100002, 2024-04-20T17:21:12.661 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.026602213, 2024-04-20T17:21:12.661 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:12.661 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:12.661 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:12.661 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:12.661 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:12.662 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:12.662 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 212685 2024-04-20T17:21:12.662 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.662 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:12.662 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.662 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:12.663 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.604266+0000", 2024-04-20T17:21:12.663 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.663 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.663 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.663 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:12.663 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.604266+0000", 2024-04-20T17:21:12.664 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.664 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.664 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.664 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:12.664 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.604269+0000", 2024-04-20T17:21:12.664 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1379999999999999e-06 2024-04-20T17:21:12.665 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.665 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.665 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:12.665 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.604277+0000", 2024-04-20T17:21:12.665 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4209999999999995e-06 2024-04-20T17:21:12.665 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.665 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.666 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:12.666 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.604298+0000", 2024-04-20T17:21:12.666 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0041999999999999e-05 2024-04-20T17:21:12.666 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.666 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.666 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:12.667 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.604308+0000", 2024-04-20T17:21:12.667 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0392e-05 2024-04-20T17:21:12.667 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.667 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.667 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.667 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609525+0000", 2024-04-20T17:21:12.667 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0052173369999999998 2024-04-20T17:21:12.668 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.668 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.668 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:12.668 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609662+0000", 2024-04-20T17:21:12.668 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013646400000000001 2024-04-20T17:21:12.668 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.669 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.669 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:21:12.669 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.609758+0000", 2024-04-20T17:21:12.669 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6510999999999998e-05 2024-04-20T17:21:12.669 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.669 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.670 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:12.670 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.613348+0000", 2024-04-20T17:21:12.670 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0035896919999999998 2024-04-20T17:21:12.670 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.670 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.671 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:12.671 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.630768+0000", 2024-04-20T17:21:12.671 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.017419752 2024-04-20T17:21:12.671 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.671 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.671 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:12.672 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.630825+0000", 2024-04-20T17:21:12.672 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6937000000000001e-05 2024-04-20T17:21:12.672 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.672 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.672 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:12.672 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:23.630868+0000", 2024-04-20T17:21:12.672 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3526999999999998e-05 2024-04-20T17:21:12.673 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.673 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:12.673 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.673 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.673 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.673 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:308765 2.e 2:738ce44e:::benchmark_data_smithi171_78918_object147771:head [delete] snapc 0=[] ondisk+write+known_if_redirected e344)", 2024-04-20T17:21:12.674 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:19:51.570715+0000", 2024-04-20T17:21:12.674 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 80.326802448999999, 2024-04-20T17:21:12.674 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023924622999999999, 2024-04-20T17:21:12.674 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:12.674 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:12.674 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:12.675 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:12.675 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:12.675 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:12.675 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 308765 2024-04-20T17:21:12.675 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.675 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:12.676 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.676 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:12.676 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:51.570715+0000", 2024-04-20T17:21:12.676 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.676 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.676 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.676 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:12.677 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:51.570715+0000", 2024-04-20T17:21:12.677 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.677 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.677 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.677 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:12.677 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:51.570716+0000", 2024-04-20T17:21:12.677 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6270000000000001e-06 2024-04-20T17:21:12.677 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.677 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.677 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:12.677 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:51.570725+0000", 2024-04-20T17:21:12.677 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1920000000000005e-06 2024-04-20T17:21:12.677 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.677 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.678 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:12.678 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:51.570732+0000", 2024-04-20T17:21:12.678 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7729999999999999e-06 2024-04-20T17:21:12.678 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.678 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.678 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:12.678 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:51.570739+0000", 2024-04-20T17:21:12.678 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8739999999999997e-06 2024-04-20T17:21:12.678 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.678 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.678 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.678 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:51.575294+0000", 2024-04-20T17:21:12.678 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0045547879999999997 2024-04-20T17:21:12.678 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.679 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.679 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:12.679 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:51.575426+0000", 2024-04-20T17:21:12.679 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013234299999999999 2024-04-20T17:21:12.679 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.679 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.679 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:21:12.679 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:51.575561+0000", 2024-04-20T17:21:12.679 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000134311 2024-04-20T17:21:12.679 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.679 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.679 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:12.679 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:51.580076+0000", 2024-04-20T17:21:12.679 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0045157239999999996 2024-04-20T17:21:12.680 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.680 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.680 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:12.680 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:51.594535+0000", 2024-04-20T17:21:12.680 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014459062 2024-04-20T17:21:12.680 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.680 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.680 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:12.680 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:51.594594+0000", 2024-04-20T17:21:12.680 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8882000000000003e-05 2024-04-20T17:21:12.680 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.680 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.680 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:12.680 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:51.594639+0000", 2024-04-20T17:21:12.681 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5046999999999997e-05 2024-04-20T17:21:12.681 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.681 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:12.681 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.681 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.681 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.681 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:314359 2.e 2:767b09fa:::benchmark_data_smithi171_78918_object153365:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-04-20T17:21:12.681 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:19:57.382167+0000", 2024-04-20T17:21:12.681 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 74.515350247000001, 2024-04-20T17:21:12.681 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023609794, 2024-04-20T17:21:12.681 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:12.681 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:12.681 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:12.681 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:12.681 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:12.682 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:12.682 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 314359 2024-04-20T17:21:12.682 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.682 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:12.682 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.682 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:12.682 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:57.382167+0000", 2024-04-20T17:21:12.682 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.682 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.682 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.682 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:12.682 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:57.382167+0000", 2024-04-20T17:21:12.682 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.682 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.683 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.683 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:12.683 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:57.382167+0000", 2024-04-20T17:21:12.683 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3099999999999998e-07 2024-04-20T17:21:12.683 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.683 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.683 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:12.683 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:57.382170+0000", 2024-04-20T17:21:12.683 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1289999999999999e-06 2024-04-20T17:21:12.683 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.683 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.683 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:12.683 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:57.382173+0000", 2024-04-20T17:21:12.683 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.405e-06 2024-04-20T17:21:12.684 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.684 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.684 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:12.684 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:57.382176+0000", 2024-04-20T17:21:12.684 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0939999999999999e-06 2024-04-20T17:21:12.684 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.684 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.684 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.684 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:57.388848+0000", 2024-04-20T17:21:12.684 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0066718560000000003 2024-04-20T17:21:12.684 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.684 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.684 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:12.684 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:57.388956+0000", 2024-04-20T17:21:12.684 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010835 2024-04-20T17:21:12.685 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.685 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.685 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:21:12.685 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:57.389099+0000", 2024-04-20T17:21:12.685 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014256699999999999 2024-04-20T17:21:12.685 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.685 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.685 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:12.685 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:57.401002+0000", 2024-04-20T17:21:12.685 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011903508 2024-04-20T17:21:12.685 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.685 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.685 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:12.685 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:57.405680+0000", 2024-04-20T17:21:12.685 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0046773270000000002 2024-04-20T17:21:12.686 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.686 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.686 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:12.686 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:57.405737+0000", 2024-04-20T17:21:12.686 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7723999999999998e-05 2024-04-20T17:21:12.686 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.686 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.686 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:12.686 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:57.405777+0000", 2024-04-20T17:21:12.686 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9303e-05 2024-04-20T17:21:12.686 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.686 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:12.686 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.687 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.687 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.687 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:316290 2.e 2:779e6ef9:::benchmark_data_smithi171_78918_object155296:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-04-20T17:21:12.687 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:19:59.122503+0000", 2024-04-20T17:21:12.687 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 72.775014244000005, 2024-04-20T17:21:12.687 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.025463489999999998, 2024-04-20T17:21:12.687 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:12.687 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:12.687 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:12.687 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:12.687 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:12.687 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:12.687 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 316290 2024-04-20T17:21:12.687 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.687 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:12.688 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.688 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:12.688 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:59.122503+0000", 2024-04-20T17:21:12.688 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.688 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.688 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.688 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:12.688 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:59.122503+0000", 2024-04-20T17:21:12.688 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:12.688 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.688 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.688 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:12.688 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:59.122505+0000", 2024-04-20T17:21:12.688 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.776e-06 2024-04-20T17:21:12.688 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.689 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.689 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:12.689 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:59.122512+0000", 2024-04-20T17:21:12.689 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6620000000000005e-06 2024-04-20T17:21:12.689 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.689 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.689 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:12.689 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:59.122524+0000", 2024-04-20T17:21:12.689 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1279e-05 2024-04-20T17:21:12.689 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.689 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.689 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:12.689 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:59.122529+0000", 2024-04-20T17:21:12.689 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6849999999999998e-06 2024-04-20T17:21:12.689 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.690 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.690 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:12.690 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:59.130584+0000", 2024-04-20T17:21:12.690 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0080544569999999992 2024-04-20T17:21:12.690 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.690 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.690 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:12.690 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:59.130668+0000", 2024-04-20T17:21:12.690 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3753999999999998e-05 2024-04-20T17:21:12.690 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.690 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.690 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:21:12.690 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:59.130759+0000", 2024-04-20T17:21:12.690 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1246e-05 2024-04-20T17:21:12.690 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.691 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.691 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:12.691 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:59.139726+0000", 2024-04-20T17:21:12.691 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.008967315 2024-04-20T17:21:12.691 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.691 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.691 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:12.691 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:59.147857+0000", 2024-04-20T17:21:12.691 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0081306039999999996 2024-04-20T17:21:12.691 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.691 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.691 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:12.691 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:59.147917+0000", 2024-04-20T17:21:12.691 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0236999999999999e-05 2024-04-20T17:21:12.691 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:12.691 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:12.692 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:12.692 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:19:59.147966+0000", 2024-04-20T17:21:12.692 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9475000000000003e-05 2024-04-20T17:21:12.692 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.692 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:12.692 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.692 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:12.692 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:12.692 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:21:12.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:12.693 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:21:12.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:21:12.794 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:21:12.796 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 41 16 71782 71766 109.383 128 0.00730685 0.00914083 2024-04-20T17:21:12.845 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:13.516 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-20T17:21:13.796 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 42 16 73939 73923 109.988 134.812 0.00443813 0.00909004 2024-04-20T17:21:13.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:13 smithi171 ceph-mon[44189]: pgmap v1018: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 4.0 GiB data, 9.9 GiB used, 973 GiB / 983 GiB avail; 125 MiB/s wr, 2.00k op/s; 1152/130264 objects degraded (0.884%); 14 MiB/s, 217 objects/s recovering 2024-04-20T17:21:13.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:13 smithi005 ceph-mon[39562]: pgmap v1018: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 4.0 GiB data, 9.9 GiB used, 973 GiB / 983 GiB avail; 125 MiB/s wr, 2.00k op/s; 1152/130264 objects degraded (0.884%); 14 MiB/s, 217 objects/s recovering 2024-04-20T17:21:14.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:13 smithi106 ceph-mon[44423]: pgmap v1018: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 4.0 GiB data, 9.9 GiB used, 973 GiB / 983 GiB avail; 125 MiB/s wr, 2.00k op/s; 1152/130264 objects degraded (0.884%); 14 MiB/s, 217 objects/s recovering 2024-04-20T17:21:14.106 INFO:teuthology.orchestra.run.smithi106.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:21:14.797 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 43 16 75983 75967 110.401 127.75 0.00677145 0.00905626 2024-04-20T17:21:14.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:14 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 563/140004 objects degraded (0.402%), 2 pgs degraded (PG_DEGRADED) 2024-04-20T17:21:14.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:14 smithi005 ceph-mon[39562]: osdmap e376: 12 total, 11 up, 11 in 2024-04-20T17:21:15.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:14 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 563/140004 objects degraded (0.402%), 2 pgs degraded (PG_DEGRADED) 2024-04-20T17:21:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:14 smithi106 ceph-mon[44423]: osdmap e376: 12 total, 11 up, 11 in 2024-04-20T17:21:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:14 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 563/140004 objects degraded (0.402%), 2 pgs degraded (PG_DEGRADED) 2024-04-20T17:21:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:14 smithi171 ceph-mon[44189]: osdmap e376: 12 total, 11 up, 11 in 2024-04-20T17:21:15.796 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 44 16 78034 78018 110.804 128.188 0.0066658 0.00902385 2024-04-20T17:21:15.805 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:21:15.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:15.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:21:15.912 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:21:15.939 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:15 smithi005 ceph-mon[39562]: pgmap v1019: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 4.3 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 144 MiB/s wr, 2.30k op/s; 563/140004 objects degraded (0.402%); 16 MiB/s, 257 objects/s recovering 2024-04-20T17:21:15.939 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:15 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:21:15.939 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:15 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:21:15.940 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:15 smithi106 ceph-mon[44423]: pgmap v1019: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 4.3 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 144 MiB/s wr, 2.30k op/s; 563/140004 objects degraded (0.402%); 16 MiB/s, 257 objects/s recovering 2024-04-20T17:21:15.940 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:15 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:21:15.940 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:15 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:21:15.943 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:16.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:15 smithi171 ceph-mon[44189]: pgmap v1019: 17 pgs: 2 active+recovering+degraded, 15 active+clean; 4.3 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 144 MiB/s wr, 2.30k op/s; 563/140004 objects degraded (0.402%); 16 MiB/s, 257 objects/s recovering 2024-04-20T17:21:16.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:15 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:21:16.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:15 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:21:16.600 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:21:16.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:16.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:21:16.704 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:21:16.743 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:16.770 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:21:16.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:16.774 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:21:16.797 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 45 16 79959 79943 111.016 120.312 0.0225467 0.00900599 2024-04-20T17:21:16.875 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:21:16.933 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:17.082 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:21:17.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:17.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:21:17.157 INFO:teuthology.orchestra.run.smithi106.stdout:get_pg_num_history pg_num_history pg_num_history(e364 pg_nums {1={14=1},2={115=16},3={356=16}} deleted_pools 351,2) 2024-04-20T17:21:17.157 INFO:teuthology.orchestra.run.smithi106.stdout:Importing pgid 1.0 2024-04-20T17:21:17.157 INFO:teuthology.orchestra.run.smithi106.stdout:Write #1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head# 2024-04-20T17:21:17.157 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:21:17.158 INFO:teuthology.orchestra.run.smithi106.stdout:Write #1:30306672:devicehealth::main.db.0000000000000000:head# 2024-04-20T17:21:17.158 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:21:17.158 INFO:teuthology.orchestra.run.smithi106.stdout:write_pg epoch 340 info 1.0( v 15'137 (0'0,15'137] local-lis/les=93/94 n=2 ec=14/14 lis/c=93/93 les/c/f=94/94/0 sis=93) 2024-04-20T17:21:17.162 INFO:teuthology.orchestra.run.smithi106.stdout:Import successful 2024-04-20T17:21:17.193 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:21:17.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:17 smithi005 ceph-mon[39562]: osdmap e377: 12 total, 11 up, 11 in 2024-04-20T17:21:17.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:17 smithi005 ceph-mon[39562]: pgmap v1022: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 4.6 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 127 MiB/s wr, 2.04k op/s; 296/149082 objects degraded (0.199%); 12 MiB/s, 189 objects/s recovering 2024-04-20T17:21:17.234 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:17.262 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:21:17.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:17.267 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:21:17.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:17 smithi171 ceph-mon[44189]: osdmap e377: 12 total, 11 up, 11 in 2024-04-20T17:21:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:17 smithi171 ceph-mon[44189]: pgmap v1022: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 4.6 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 127 MiB/s wr, 2.04k op/s; 296/149082 objects degraded (0.199%); 12 MiB/s, 189 objects/s recovering 2024-04-20T17:21:17.368 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:21:17.421 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:17.464 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:21:17.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:17.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:21:17.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:17 smithi106 ceph-mon[44423]: osdmap e377: 12 total, 11 up, 11 in 2024-04-20T17:21:17.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:17 smithi106 ceph-mon[44423]: pgmap v1022: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 4.6 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 127 MiB/s wr, 2.04k op/s; 296/149082 objects degraded (0.199%); 12 MiB/s, 189 objects/s recovering 2024-04-20T17:21:17.514 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:21:17.522 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:21:17.551 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:21:17.571 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:21:17.588 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:21:17.616 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:17.641 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:21:17.680 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:21:17.709 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:21:17.743 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:21:17.781 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:21:17.797 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 46 16 81781 81765 111.077 113.875 0.00975226 0.0090012 2024-04-20T17:21:17.819 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:21:17.859 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:21:17.900 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:21:17.900 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:21:17.900 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:21:17.900 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:21:17.901 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:21:17.901 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:21:17.901 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:21:17.901 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:21:17.901 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:21:17.901 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:21:17.901 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:21:17.901 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:21:17.901 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:21:17.901 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:21:17.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:17.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:21:18.004 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:21:18.044 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:18.060 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:21:18.102 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:21:18.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:18.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:21:18.210 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:21:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:18 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2899096522' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:21:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:18 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:21:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:18 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:21:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:18 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3918888464' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:21:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:18.243 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:18.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:18 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2899096522' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:21:18.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:18 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:21:18.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:18 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:21:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:18 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3918888464' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:21:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:18.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:18 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2899096522' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:21:18.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:18 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:21:18.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:18 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:21:18.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:18 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3918888464' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:21:18.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:18.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:18.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:18.586 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/exp.1.0.7 2024-04-20T17:21:18.598 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:21:18.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:18.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:21:18.632 INFO:tasks.thrashosds.thrasher:Waiting for clean again 2024-04-20T17:21:18.632 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-20T17:21:18.633 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:21:18.703 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:21:18.763 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:18.797 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 47 16 83754 83738 111.338 123.312 0.00290617 0.0089793 2024-04-20T17:21:19.047 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:21:19.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:19.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:21:19.154 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:21:19.196 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:19.225 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:21:19.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:19.229 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:21:19.326 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:19.326 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:19.326 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:19.326 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:19.326 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:19.326 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:19.326 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:19.326 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:19.326 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:19 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:21:19.327 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:19 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2899096522' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:21:19.327 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:19 smithi005 ceph-mon[39562]: osdmap e378: 12 total, 11 up, 11 in 2024-04-20T17:21:19.327 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:19 smithi005 ceph-mon[39562]: pgmap v1024: 17 pgs: 17 active+clean; 4.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 138 MiB/s wr, 2.20k op/s; 12 MiB/s, 191 objects/s recovering 2024-04-20T17:21:19.330 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:21:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:19 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:21:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:19 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2899096522' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:21:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:19 smithi171 ceph-mon[44189]: osdmap e378: 12 total, 11 up, 11 in 2024-04-20T17:21:19.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:19 smithi171 ceph-mon[44189]: pgmap v1024: 17 pgs: 17 active+clean; 4.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 138 MiB/s wr, 2.20k op/s; 12 MiB/s, 191 objects/s recovering 2024-04-20T17:21:19.378 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:19.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:19 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:21:19.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:19 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2899096522' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:21:19.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:19 smithi106 ceph-mon[44423]: osdmap e378: 12 total, 11 up, 11 in 2024-04-20T17:21:19.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:19 smithi106 ceph-mon[44423]: pgmap v1024: 17 pgs: 17 active+clean; 4.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 138 MiB/s wr, 2.20k op/s; 12 MiB/s, 191 objects/s recovering 2024-04-20T17:21:19.747 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:21:19.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:19.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:21:19.797 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 48 16 85713 85697 111.569 122.438 0.0126224 0.00896162 2024-04-20T17:21:19.852 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:21:19.897 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:19.924 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:21:19.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:19.930 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:21:20.031 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:21:20.074 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:20.099 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:21:20.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:20.104 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:21:20.205 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:21:20.257 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:20.290 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:21:20.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:20.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:21:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:20 smithi171 ceph-mon[44189]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 296/149082 objects degraded (0.199%), 1 pg degraded) 2024-04-20T17:21:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:20 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:21:20.395 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:21:20.398 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:20 smithi005 ceph-mon[39562]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 296/149082 objects degraded (0.199%), 1 pg degraded) 2024-04-20T17:21:20.398 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:20 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:21:20.447 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:20.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:20 smithi106 ceph-mon[44423]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 296/149082 objects degraded (0.199%), 1 pg degraded) 2024-04-20T17:21:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:20 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:21:20.797 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 49 16 87646 87630 111.757 120.812 0.00847591 0.00894652 2024-04-20T17:21:21.065 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:21:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:21 smithi171 ceph-mon[44189]: pgmap v1025: 17 pgs: 17 active+clean; 4.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 107 MiB/s wr, 1.70k op/s; 5.9 MiB/s, 93 objects/s recovering 2024-04-20T17:21:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:21 smithi171 ceph-mon[44189]: osdmap e379: 12 total, 11 up, 11 in 2024-04-20T17:21:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:21 smithi005 ceph-mon[39562]: pgmap v1025: 17 pgs: 17 active+clean; 4.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 107 MiB/s wr, 1.70k op/s; 5.9 MiB/s, 93 objects/s recovering 2024-04-20T17:21:21.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:21 smithi005 ceph-mon[39562]: osdmap e379: 12 total, 11 up, 11 in 2024-04-20T17:21:21.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:21 smithi106 ceph-mon[44423]: pgmap v1025: 17 pgs: 17 active+clean; 4.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 107 MiB/s wr, 1.70k op/s; 5.9 MiB/s, 93 objects/s recovering 2024-04-20T17:21:21.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:21 smithi106 ceph-mon[44423]: osdmap e379: 12 total, 11 up, 11 in 2024-04-20T17:21:21.798 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 50 16 89565 89549 111.919 119.938 0.00760203 0.00893372 2024-04-20T17:21:22.118 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:21:22.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:22.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:21:22.171 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:21:22.172 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:21:22.178 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:21:22.227 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:21:22.280 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:22.341 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:21:22.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:22.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:21:22.447 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:21:22.488 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:22.515 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:21:22.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:22.520 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:21:22.621 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:21:22.666 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:22.798 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 51 16 91412 91396 111.988 115.438 0.0180758 0.00892819 2024-04-20T17:21:23.199 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:21:23.199 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:21:23.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:23 smithi005 ceph-mon[39562]: pgmap v1027: 17 pgs: 17 active+clean; 5.2 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 108 MiB/s wr, 1.73k op/s; 3.0 MiB/s, 48 objects/s recovering 2024-04-20T17:21:23.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:23 smithi106 ceph-mon[44423]: pgmap v1027: 17 pgs: 17 active+clean; 5.2 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 108 MiB/s wr, 1.73k op/s; 3.0 MiB/s, 48 objects/s recovering 2024-04-20T17:21:23.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:23 smithi171 ceph-mon[44189]: pgmap v1027: 17 pgs: 17 active+clean; 5.2 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 108 MiB/s wr, 1.73k op/s; 3.0 MiB/s, 48 objects/s recovering 2024-04-20T17:21:23.798 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 52 15 93255 93240 112.049 115.25 0.0124794 0.00892321 2024-04-20T17:21:24.069 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:21:24.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:24.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:21:24.091 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-20T17:21:24.173 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:21:24.173 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:21:24.203 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:24.224 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:21:24.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:24.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:21:24.328 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:21:24.364 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:24.384 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:21:24.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:24.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:21:24.488 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:21:24.527 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:24.802 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 53 15 95004 94989 111.991 109.312 0.00970674 0.00892785 2024-04-20T17:21:25.093 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:21:25.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:25.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:21:25.114 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:21:25.189 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:21:25.203 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:21:25.205 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:21:25.269 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:25.329 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:21:25.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:25.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:21:25.338 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:25 smithi005 ceph-mon[39562]: pgmap v1028: 17 pgs: 17 active+clean; 5.5 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 119 MiB/s wr, 1.91k op/s; 2.3 MiB/s, 36 objects/s recovering 2024-04-20T17:21:25.439 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:21:25.474 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:25 smithi171 ceph-mon[44189]: pgmap v1028: 17 pgs: 17 active+clean; 5.5 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 119 MiB/s wr, 1.91k op/s; 2.3 MiB/s, 36 objects/s recovering 2024-04-20T17:21:25.482 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:25.508 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:21:25.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:25 smithi106 ceph-mon[44423]: pgmap v1028: 17 pgs: 17 active+clean; 5.5 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 119 MiB/s wr, 1.91k op/s; 2.3 MiB/s, 36 objects/s recovering 2024-04-20T17:21:25.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:25.511 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:21:25.612 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:21:25.648 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:25.675 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:21:25.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:25.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:21:25.779 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:21:25.802 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 54 16 96703 96687 111.882 106.125 0.0137818 0.00893636 2024-04-20T17:21:25.817 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:25.843 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:21:25.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:25.847 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:21:25.948 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:21:25.987 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:26.009 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:21:26.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:26.015 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:21:26.115 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:21:26.163 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:26.411 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:26 smithi005 ceph-mon[39562]: 3.5 scrub starts 2024-04-20T17:21:26.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:26 smithi106 ceph-mon[44423]: 3.5 scrub starts 2024-04-20T17:21:26.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:26 smithi171 ceph-mon[44189]: 3.5 scrub starts 2024-04-20T17:21:26.802 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 55 16 98439 98423 111.82 108.5 0.00330073 0.00894123 2024-04-20T17:21:27.196 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:27 smithi005 ceph-mon[39562]: pgmap v1029: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 120 MiB/s wr, 1.91k op/s 2024-04-20T17:21:27.196 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:27 smithi005 ceph-mon[39562]: 3.5 scrub ok 2024-04-20T17:21:27.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:27 smithi106 ceph-mon[44423]: pgmap v1029: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 120 MiB/s wr, 1.91k op/s 2024-04-20T17:21:27.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:27 smithi106 ceph-mon[44423]: 3.5 scrub ok 2024-04-20T17:21:27.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:27 smithi171 ceph-mon[44189]: pgmap v1029: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 120 MiB/s wr, 1.91k op/s 2024-04-20T17:21:27.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:27 smithi171 ceph-mon[44189]: 3.5 scrub ok 2024-04-20T17:21:27.802 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 56 16 100258 100242 111.854 113.688 0.00535503 0.00893894 2024-04-20T17:21:28.802 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 57 16 101997 101981 111.798 108.688 0.00479327 0.00894318 2024-04-20T17:21:29.015 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:21:29.035 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:21:29.037 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:21:29.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:29.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:21:29.145 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:21:29.184 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:29.216 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:21:29.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:29.220 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:21:29.320 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:21:29.358 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:29.396 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:21:29.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:29.406 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:21:29.414 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:21:29.414 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:21:29.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:29 smithi005 ceph-mon[39562]: pgmap v1030: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-04-20T17:21:29.507 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:21:29.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:29 smithi106 ceph-mon[44423]: pgmap v1030: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-04-20T17:21:29.544 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:29 smithi171 ceph-mon[44189]: pgmap v1030: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-04-20T17:21:29.544 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:29.575 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:21:29.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:29.579 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:21:29.680 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:21:29.745 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:29.802 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 58 16 103467 103451 111.454 91.875 0.00553644 0.00896969 2024-04-20T17:21:29.814 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:21:29.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:29.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:21:29.928 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:21:29.956 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:29.977 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:21:29.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:29.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:21:30.081 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:21:30.123 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:30.153 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:21:30.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:30.157 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:21:30.211 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:30 smithi106 ceph-mon[44423]: from='client.25822 v1:172.21.15.5:0/3783397295' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:21:30.211 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:30 smithi106 ceph-mon[44423]: 3.1 scrub starts 2024-04-20T17:21:30.212 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:30 smithi106 ceph-mon[44423]: 3.f deep-scrub starts 2024-04-20T17:21:30.257 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:21:30.295 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:30.323 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:21:30.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:30.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:21:30.428 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:21:30.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:30 smithi005 ceph-mon[39562]: from='client.25822 v1:172.21.15.5:0/3783397295' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:21:30.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:30 smithi005 ceph-mon[39562]: 3.1 scrub starts 2024-04-20T17:21:30.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:30 smithi005 ceph-mon[39562]: 3.f deep-scrub starts 2024-04-20T17:21:30.464 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:30 smithi171 ceph-mon[44189]: from='client.25822 v1:172.21.15.5:0/3783397295' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:21:30.464 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:30 smithi171 ceph-mon[44189]: 3.1 scrub starts 2024-04-20T17:21:30.464 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:30 smithi171 ceph-mon[44189]: 3.f deep-scrub starts 2024-04-20T17:21:30.468 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:30.495 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:21:30.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:30.498 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:21:30.599 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:21:30.642 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:30.664 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:21:30.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:30.667 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:21:30.768 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:21:30.798 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:30.802 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 59 16 104816 104800 110.994 84.3125 0.0064622 0.00900691 2024-04-20T17:21:30.821 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:21:30.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:30.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:21:30.924 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:21:30.958 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:30.984 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:21:30.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:30.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:21:31.087 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:21:31.115 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:31.136 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:21:31.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:31.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:21:31.240 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:21:31.274 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:31.300 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:21:31.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:31.309 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:21:31.410 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:21:31.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:31 smithi005 ceph-mon[39562]: pgmap v1031: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-04-20T17:21:31.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:31 smithi005 ceph-mon[39562]: 3.d scrub starts 2024-04-20T17:21:31.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:31 smithi005 ceph-mon[39562]: 3.1 scrub ok 2024-04-20T17:21:31.462 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:31.502 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:21:31.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:31.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:21:31.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:31 smithi106 ceph-mon[44423]: pgmap v1031: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-04-20T17:21:31.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:31 smithi106 ceph-mon[44423]: 3.d scrub starts 2024-04-20T17:21:31.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:31 smithi106 ceph-mon[44423]: 3.1 scrub ok 2024-04-20T17:21:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:31 smithi171 ceph-mon[44189]: pgmap v1031: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-04-20T17:21:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:31 smithi171 ceph-mon[44189]: 3.d scrub starts 2024-04-20T17:21:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:31 smithi171 ceph-mon[44189]: 3.1 scrub ok 2024-04-20T17:21:31.605 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:21:31.649 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:31.676 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:21:31.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:31.680 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:21:31.781 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:21:31.802 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:21:31.803540 min lat: 0.000946083 max lat: 16.1837 avg lat: 0.00904283 2024-04-20T17:21:31.802 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:21:31.802 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 60 16 106170 106154 110.555 84.625 0.0101727 0.00904283 2024-04-20T17:21:31.831 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:32.153 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:21:32.153 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:21:32.153 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:21:32.169 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:21:32.200 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:21:32.230 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:21:32.256 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:21:32.296 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:21:32.332 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:21:32.364 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:21:32.396 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:21:32.397 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:21:32.397 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:21:32.397 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:21:32.397 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:21:32.397 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:21:32.397 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:21:32.397 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:21:32.397 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:21:32.397 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:21:32.397 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:21:32.397 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:21:32.397 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:21:32.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:32 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:21:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:32 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:21:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:32 smithi106 ceph-mon[44423]: 3.d scrub ok 2024-04-20T17:21:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:32 smithi106 ceph-mon[44423]: osdmap e380: 12 total, 11 up, 11 in 2024-04-20T17:21:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:32 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3235266761' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:21:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:32 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:21:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:32 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:21:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:32 smithi171 ceph-mon[44189]: 3.d scrub ok 2024-04-20T17:21:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:32 smithi171 ceph-mon[44189]: osdmap e380: 12 total, 11 up, 11 in 2024-04-20T17:21:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:32 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3235266761' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:21:32.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:32 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:21:32.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:32 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:21:32.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:32 smithi005 ceph-mon[39562]: 3.d scrub ok 2024-04-20T17:21:32.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:32 smithi005 ceph-mon[39562]: osdmap e380: 12 total, 11 up, 11 in 2024-04-20T17:21:32.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:32 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3235266761' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:21:32.802 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 61 16 107542 107526 110.148 85.75 0.00972942 0.00907729 2024-04-20T17:21:33.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:33 smithi106 ceph-mon[44423]: pgmap v1033: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.3 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-04-20T17:21:33.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:33 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:21:33.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:33 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:21:33.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:33 smithi171 ceph-mon[44189]: pgmap v1033: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.3 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-04-20T17:21:33.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:33 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:21:33.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:33 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:21:33.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:33 smithi005 ceph-mon[39562]: pgmap v1033: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.3 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-04-20T17:21:33.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:33.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:33 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:21:33.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:33 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:21:33.802 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 62 16 108943 108927 109.784 87.5625 0.00356041 0.00910625 2024-04-20T17:21:34.165 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:21:34.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:34.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:21:34.270 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:21:34.305 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:34.330 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:21:34.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:34.332 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:21:34.433 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:21:34.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:34.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:34 smithi005 ceph-mon[39562]: osdmap e381: 12 total, 11 up, 11 in 2024-04-20T17:21:34.464 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:34.484 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:21:34.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:34.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:21:34.489 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:34.490 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:34 smithi106 ceph-mon[44423]: osdmap e381: 12 total, 11 up, 11 in 2024-04-20T17:21:34.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:34.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:34 smithi171 ceph-mon[44189]: osdmap e381: 12 total, 11 up, 11 in 2024-04-20T17:21:34.589 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:21:34.633 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:34.655 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:21:34.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:34.662 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:21:34.763 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:21:34.800 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:34.802 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 63 16 110251 110235 109.338 81.75 0.033582 0.0091442 2024-04-20T17:21:35.343 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-20T17:21:35.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:35 smithi106 ceph-mon[44423]: pgmap v1035: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 6.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-04-20T17:21:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:35 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:21:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:35 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:21:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:35 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2308896732' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:21:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:35 smithi171 ceph-mon[44189]: pgmap v1035: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 6.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-04-20T17:21:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:35 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:21:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:35 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:21:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:35 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2308896732' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:21:35.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:35 smithi005 ceph-mon[39562]: pgmap v1035: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 6.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-04-20T17:21:35.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:35 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:21:35.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:35 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:21:35.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:35 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2308896732' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:21:35.805 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 64 16 111584 111568 108.932 83.3125 0.0137875 0.00917738 2024-04-20T17:21:36.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:36 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:21:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:36 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2308896732' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:21:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:36 smithi106 ceph-mon[44423]: osdmap e382: 12 total, 11 up, 11 in 2024-04-20T17:21:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:36 smithi106 ceph-mon[44423]: osdmap e383: 12 total, 11 up, 11 in 2024-04-20T17:21:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:36 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:21:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:36 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2308896732' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:21:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:36 smithi171 ceph-mon[44189]: osdmap e382: 12 total, 11 up, 11 in 2024-04-20T17:21:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:36 smithi171 ceph-mon[44189]: osdmap e383: 12 total, 11 up, 11 in 2024-04-20T17:21:36.609 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:21:36.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:36 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:21:36.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:36 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2308896732' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:21:36.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:36 smithi005 ceph-mon[39562]: osdmap e382: 12 total, 11 up, 11 in 2024-04-20T17:21:36.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:36 smithi005 ceph-mon[39562]: osdmap e383: 12 total, 11 up, 11 in 2024-04-20T17:21:36.803 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 65 16 112938 112922 108.558 84.625 0.00765665 0.00920646 2024-04-20T17:21:37.464 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:21:37.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:37.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:21:37.482 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":1030,"stamp":"2024-04-20T17:21:28.113805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6362432032,"num_objects":97076,"num_object_clones":0,"num_object_copies":194152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97076,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":97255,"num_write_kb":6215674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4162,"num_bytes_recovered":274352256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97211,"ondisk_log_size":97211,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":14542496,"kb_used_data":13021764,"kb_used_omap":218,"kb_used_meta":1520421,"kb_avail":1016654176,"statfs":{"total":1055945392128,"available":1041053876224,"internally_reserved":0,"allocated":13334286336,"data_stored":13329131610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":223359,"internal_metadata":1556912001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1227948032,"num_objects":18737,"num_object_clones":0,"num_object_copies":37474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18737,"num_write_kb":1199168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002638"},"pg_stats":[{"pgid":"3.f","version":"379'6163","reported_seq":10366,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-20T17:21:27.457370+0000","last_change":"2024-04-20T17:21:14.733804+0000","last_active":"2024-04-20T17:21:27.457370+0000","last_peered":"2024-04-20T17:21:27.457370+0000","last_clean":"2024-04-20T17:21:27.457370+0000","last_became_active":"2024-04-20T17:20:55.357479+0000","last_became_peered":"2024-04-20T17:20:55.357479+0000","last_unstale":"2024-04-20T17:21:27.457370+0000","last_undegraded":"2024-04-20T17:21:27.457370+0000","last_fullsized":"2024-04-20T17:21:27.457370+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":0,"log_size":6163,"log_dups_size":0,"ondisk_log_size":6163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:28.962635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403898368,"num_objects":6163,"num_object_clones":0,"num_object_copies":12326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6163,"num_write_kb":394432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2071,"num_bytes_recovered":135725056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"379'6271","reported_seq":6325,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-20T17:21:27.467040+0000","last_change":"2024-04-20T17:20:23.037437+0000","last_active":"2024-04-20T17:21:27.467040+0000","last_peered":"2024-04-20T17:21:27.467040+0000","last_clean":"2024-04-20T17:21:27.467040+0000","last_became_active":"2024-04-20T17:20:23.037195+0000","last_became_peered":"2024-04-20T17:20:23.037195+0000","last_unstale":"2024-04-20T17:21:27.467040+0000","last_undegraded":"2024-04-20T17:21:27.467040+0000","last_fullsized":"2024-04-20T17:21:27.467040+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":0,"log_size":6271,"log_dups_size":0,"ondisk_log_size":6271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:48.986247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410976256,"num_objects":6271,"num_object_clones":0,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6271,"num_write_kb":401344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"379'6073","reported_seq":10299,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-20T17:21:25.574764+0000","last_change":"2024-04-20T17:21:12.333550+0000","last_active":"2024-04-20T17:21:25.574764+0000","last_peered":"2024-04-20T17:21:25.574764+0000","last_clean":"2024-04-20T17:21:25.574764+0000","last_became_active":"2024-04-20T17:20:55.283498+0000","last_became_peered":"2024-04-20T17:20:55.283498+0000","last_unstale":"2024-04-20T17:21:25.574764+0000","last_undegraded":"2024-04-20T17:21:25.574764+0000","last_fullsized":"2024-04-20T17:21:25.574764+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":0,"log_size":6073,"log_dups_size":0,"ondisk_log_size":6073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:29.357976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398000128,"num_objects":6073,"num_object_clones":0,"num_object_copies":12146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6073,"num_write_kb":388672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2083,"num_bytes_recovered":136511488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"379'5887","reported_seq":5941,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-20T17:21:25.569117+0000","last_change":"2024-04-20T17:20:23.038299+0000","last_active":"2024-04-20T17:21:25.569117+0000","last_peered":"2024-04-20T17:21:25.569117+0000","last_clean":"2024-04-20T17:21:25.569117+0000","last_became_active":"2024-04-20T17:20:23.038035+0000","last_became_peered":"2024-04-20T17:20:23.038035+0000","last_unstale":"2024-04-20T17:21:25.569117+0000","last_undegraded":"2024-04-20T17:21:25.569117+0000","last_fullsized":"2024-04-20T17:21:25.569117+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":0,"log_size":5887,"log_dups_size":0,"ondisk_log_size":5887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:36.944139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385810432,"num_objects":5887,"num_object_clones":0,"num_object_copies":11774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5887,"num_write_kb":376768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"379'5924","reported_seq":5978,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-20T17:21:25.948544+0000","last_change":"2024-04-20T17:20:23.038256+0000","last_active":"2024-04-20T17:21:25.948544+0000","last_peered":"2024-04-20T17:21:25.948544+0000","last_clean":"2024-04-20T17:21:25.948544+0000","last_became_active":"2024-04-20T17:20:23.037709+0000","last_became_peered":"2024-04-20T17:20:23.037709+0000","last_unstale":"2024-04-20T17:21:25.948544+0000","last_undegraded":"2024-04-20T17:21:25.948544+0000","last_fullsized":"2024-04-20T17:21:25.948544+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":0,"log_size":5924,"log_dups_size":0,"ondisk_log_size":5924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:30.822265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388235264,"num_objects":5924,"num_object_clones":0,"num_object_copies":11848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5924,"num_write_kb":379136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"379'6266","reported_seq":6320,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-20T17:21:27.390322+0000","last_change":"2024-04-20T17:20:23.037647+0000","last_active":"2024-04-20T17:21:27.390322+0000","last_peered":"2024-04-20T17:21:27.390322+0000","last_clean":"2024-04-20T17:21:27.390322+0000","last_became_active":"2024-04-20T17:20:23.037291+0000","last_became_peered":"2024-04-20T17:20:23.037291+0000","last_unstale":"2024-04-20T17:21:27.390322+0000","last_undegraded":"2024-04-20T17:21:27.390322+0000","last_fullsized":"2024-04-20T17:21:27.390322+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":0,"log_size":6266,"log_dups_size":0,"ondisk_log_size":6266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:36.974547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410648576,"num_objects":6266,"num_object_clones":0,"num_object_copies":12532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6266,"num_write_kb":401024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"379'5816","reported_seq":5871,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-20T17:21:23.587545+0000","last_change":"2024-04-20T17:20:23.036662+0000","last_active":"2024-04-20T17:21:23.587545+0000","last_peered":"2024-04-20T17:21:23.587545+0000","last_clean":"2024-04-20T17:21:23.587545+0000","last_became_active":"2024-04-20T17:20:23.036396+0000","last_became_peered":"2024-04-20T17:20:23.036396+0000","last_unstale":"2024-04-20T17:21:23.587545+0000","last_undegraded":"2024-04-20T17:21:23.587545+0000","last_fullsized":"2024-04-20T17:21:23.587545+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":0,"log_size":5816,"log_dups_size":0,"ondisk_log_size":5816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:50.594173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381157376,"num_objects":5816,"num_object_clones":0,"num_object_copies":11632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5816,"num_write_kb":372224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"379'6273","reported_seq":6327,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-20T17:21:27.497321+0000","last_change":"2024-04-20T17:20:23.038640+0000","last_active":"2024-04-20T17:21:27.497321+0000","last_peered":"2024-04-20T17:21:27.497321+0000","last_clean":"2024-04-20T17:21:27.497321+0000","last_became_active":"2024-04-20T17:20:23.038395+0000","last_became_peered":"2024-04-20T17:20:23.038395+0000","last_unstale":"2024-04-20T17:21:27.497321+0000","last_undegraded":"2024-04-20T17:21:27.497321+0000","last_fullsized":"2024-04-20T17:21:27.497321+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":0,"log_size":6273,"log_dups_size":0,"ondisk_log_size":6273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:43.786722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411107328,"num_objects":6273,"num_object_clones":0,"num_object_copies":12546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6273,"num_write_kb":401472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"379'6188","reported_seq":6242,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-20T17:21:25.934333+0000","last_change":"2024-04-20T17:20:23.036646+0000","last_active":"2024-04-20T17:21:25.934333+0000","last_peered":"2024-04-20T17:21:25.934333+0000","last_clean":"2024-04-20T17:21:25.934333+0000","last_became_active":"2024-04-20T17:20:23.036358+0000","last_became_peered":"2024-04-20T17:20:23.036358+0000","last_unstale":"2024-04-20T17:21:25.934333+0000","last_undegraded":"2024-04-20T17:21:25.934333+0000","last_fullsized":"2024-04-20T17:21:25.934333+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":0,"log_size":6188,"log_dups_size":0,"ondisk_log_size":6188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:39.266775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405536768,"num_objects":6188,"num_object_clones":0,"num_object_copies":12376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6188,"num_write_kb":396032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"379'5899","reported_seq":5953,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-20T17:21:24.066228+0000","last_change":"2024-04-20T17:20:23.041005+0000","last_active":"2024-04-20T17:21:24.066228+0000","last_peered":"2024-04-20T17:21:24.066228+0000","last_clean":"2024-04-20T17:21:24.066228+0000","last_became_active":"2024-04-20T17:20:23.040736+0000","last_became_peered":"2024-04-20T17:20:23.040736+0000","last_unstale":"2024-04-20T17:21:24.066228+0000","last_undegraded":"2024-04-20T17:21:24.066228+0000","last_fullsized":"2024-04-20T17:21:24.066228+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":0,"log_size":5899,"log_dups_size":0,"ondisk_log_size":5899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:48.465285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386596864,"num_objects":5899,"num_object_clones":0,"num_object_copies":11798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5899,"num_write_kb":377536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"379'6063","reported_seq":6144,"reported_epoch":379,"state":"active+clean+scrubbing","last_fresh":"2024-04-20T17:21:25.581783+0000","last_change":"2024-04-20T17:21:25.132218+0000","last_active":"2024-04-20T17:21:25.581783+0000","last_peered":"2024-04-20T17:21:25.581783+0000","last_clean":"2024-04-20T17:21:25.581783+0000","last_became_active":"2024-04-20T17:20:23.039242+0000","last_became_peered":"2024-04-20T17:20:23.039242+0000","last_unstale":"2024-04-20T17:21:25.581783+0000","last_undegraded":"2024-04-20T17:21:25.581783+0000","last_fullsized":"2024-04-20T17:21:25.581783+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":2224,"log_size":6063,"log_dups_size":0,"ondisk_log_size":6063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397344768,"num_objects":6063,"num_object_clones":0,"num_object_copies":12126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6063,"num_write_kb":388032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"379'6183","reported_seq":6238,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-20T17:21:27.013397+0000","last_change":"2024-04-20T17:20:23.038535+0000","last_active":"2024-04-20T17:21:27.013397+0000","last_peered":"2024-04-20T17:21:27.013397+0000","last_clean":"2024-04-20T17:21:27.013397+0000","last_became_active":"2024-04-20T17:20:23.038263+0000","last_became_peered":"2024-04-20T17:20:23.038263+0000","last_unstale":"2024-04-20T17:21:27.013397+0000","last_undegraded":"2024-04-20T17:21:27.013397+0000","last_fullsized":"2024-04-20T17:21:27.013397+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":0,"log_size":6183,"log_dups_size":0,"ondisk_log_size":6183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:47.816939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405209088,"num_objects":6183,"num_object_clones":0,"num_object_copies":12366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6183,"num_write_kb":395712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"379'5960","reported_seq":6014,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-20T17:21:25.537359+0000","last_change":"2024-04-20T17:20:23.039442+0000","last_active":"2024-04-20T17:21:25.537359+0000","last_peered":"2024-04-20T17:21:25.537359+0000","last_clean":"2024-04-20T17:21:25.537359+0000","last_became_active":"2024-04-20T17:20:23.039051+0000","last_became_peered":"2024-04-20T17:20:23.039051+0000","last_unstale":"2024-04-20T17:21:25.537359+0000","last_undegraded":"2024-04-20T17:21:25.537359+0000","last_fullsized":"2024-04-20T17:21:25.537359+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":0,"log_size":5960,"log_dups_size":0,"ondisk_log_size":5960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:48.296438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390594560,"num_objects":5960,"num_object_clones":0,"num_object_copies":11920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5960,"num_write_kb":381440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"379'5949","reported_seq":6003,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-20T17:21:24.066027+0000","last_change":"2024-04-20T17:20:23.037066+0000","last_active":"2024-04-20T17:21:24.066027+0000","last_peered":"2024-04-20T17:21:24.066027+0000","last_clean":"2024-04-20T17:21:24.066027+0000","last_became_active":"2024-04-20T17:20:23.036876+0000","last_became_peered":"2024-04-20T17:20:23.036876+0000","last_unstale":"2024-04-20T17:21:24.066027+0000","last_undegraded":"2024-04-20T17:21:24.066027+0000","last_fullsized":"2024-04-20T17:21:24.066027+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":0,"log_size":5949,"log_dups_size":0,"ondisk_log_size":5949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:28.550226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389873664,"num_objects":5949,"num_object_clones":0,"num_object_copies":11898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5949,"num_write_kb":380736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"379'6165","reported_seq":6219,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-20T17:21:27.482654+0000","last_change":"2024-04-20T17:20:23.171337+0000","last_active":"2024-04-20T17:21:27.482654+0000","last_peered":"2024-04-20T17:21:27.482654+0000","last_clean":"2024-04-20T17:21:27.482654+0000","last_became_active":"2024-04-20T17:20:23.171030+0000","last_became_peered":"2024-04-20T17:20:23.171030+0000","last_unstale":"2024-04-20T17:21:27.482654+0000","last_undegraded":"2024-04-20T17:21:27.482654+0000","last_fullsized":"2024-04-20T17:21:27.482654+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":0,"log_size":6165,"log_dups_size":0,"ondisk_log_size":6165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:46.281016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404029440,"num_objects":6165,"num_object_clones":0,"num_object_copies":12330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6165,"num_write_kb":394560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"379'5994","reported_seq":6048,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-20T17:21:25.939636+0000","last_change":"2024-04-20T17:20:23.171325+0000","last_active":"2024-04-20T17:21:25.939636+0000","last_peered":"2024-04-20T17:21:25.939636+0000","last_clean":"2024-04-20T17:21:25.939636+0000","last_became_active":"2024-04-20T17:20:23.171078+0000","last_became_peered":"2024-04-20T17:20:23.171078+0000","last_unstale":"2024-04-20T17:21:25.939636+0000","last_undegraded":"2024-04-20T17:21:25.939636+0000","last_fullsized":"2024-04-20T17:21:25.939636+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":0,"log_size":5994,"log_dups_size":0,"ondisk_log_size":5994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:45.457170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392822784,"num_objects":5994,"num_object_clones":0,"num_object_copies":11988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5994,"num_write_kb":383616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":360,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-04-20T17:21:21.032511+0000","last_change":"2024-04-20T17:21:17.342231+0000","last_active":"2024-04-20T17:21:21.032511+0000","last_peered":"2024-04-20T17:21:21.032511+0000","last_clean":"2024-04-20T17:21:21.032511+0000","last_became_active":"2024-04-20T17:20:55.304416+0000","last_became_peered":"2024-04-20T17:20:55.304416+0000","last_unstale":"2024-04-20T17:21:21.032511+0000","last_undegraded":"2024-04-20T17:21:21.032511+0000","last_fullsized":"2024-04-20T17:21:21.032511+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:21:17.342091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:21:17.342091+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:22:19.858669+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":6361841664,"num_objects":97074,"num_object_clones":0,"num_object_copies":194148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97074,"num_write_kb":6212736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4154,"num_bytes_recovered":272236544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12943491072,"data_stored":12943491072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97074,"ondisk_log_size":97074,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991421,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2942764,"kb_used_data":2785252,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":90802388,"statfs":{"total":95995035648,"available":92981645312,"internally_reserved":0,"allocated":2852098048,"data_stored":2851626514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":161282500},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253059,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":216744,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93528408,"statfs":{"total":95995035648,"available":95773089792,"internally_reserved":0,"allocated":946176,"data_stored":478194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":220985792},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612808,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1333348,"kb_used_data":1145308,"kb_used_omap":1,"kb_used_meta":188030,"kb_avail":92411804,"statfs":{"total":95995035648,"available":94629687296,"internally_reserved":0,"allocated":1172795392,"data_stored":1172327430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":192543168},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809037,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":854436,"kb_used_data":733020,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":92890716,"statfs":{"total":95995035648,"available":95120093184,"internally_reserved":0,"allocated":750612480,"data_stored":750144518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,2,3,4,5,6,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430423,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":505260,"kb_used_data":401892,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93239892,"statfs":{"total":95995035648,"available":95477649408,"internally_reserved":0,"allocated":411537408,"data_stored":411065894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,1,2,3,4,5,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575089,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1307044,"kb_used_data":1185628,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":92438108,"statfs":{"total":95995035648,"available":94656622592,"internally_reserved":0,"allocated":1214083072,"data_stored":1213615110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738613,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1693092,"kb_used_data":1473116,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":92052060,"statfs":{"total":95995035648,"available":94261309440,"internally_reserved":0,"allocated":1508470784,"data_stored":1508002822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312292,"kb_used_data":263900,"kb_used_omap":202,"kb_used_meta":48181,"kb_avail":93432860,"statfs":{"total":95995035648,"available":95675248640,"internally_reserved":0,"allocated":270233600,"data_stored":269765638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":49337815},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182630,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2032804,"kb_used_data":1903068,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":91712348,"statfs":{"total":95995035648,"available":93913444352,"internally_reserved":0,"allocated":1948741632,"data_stored":1948273670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":132839879},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986401,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1619044,"kb_used_data":1547612,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92126108,"statfs":{"total":95995035648,"available":94337134592,"internally_reserved":0,"allocated":1584754688,"data_stored":1584286726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790172,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1725668,"kb_used_data":1582044,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92019484,"statfs":{"total":95995035648,"available":94227951616,"internally_reserved":0,"allocated":1620013056,"data_stored":1619545094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1533345792,"data_stored":1533345792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1220149248,"data_stored":1220149248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1974796288,"data_stored":1974796288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1584070656,"data_stored":1584070656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1181351936,"data_stored":1181351936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":410648576,"data_stored":410648576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":229638144,"data_stored":229638144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":757268480,"data_stored":757268480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1184628736,"data_stored":1184628736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2867593216,"data_stored":2867593216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:21:37.483 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:21:37.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:37 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:21:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:37 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:21:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:37 smithi106 ceph-mon[44423]: pgmap v1038: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 6.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-04-20T17:21:37.569 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:21:37.572 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:37 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:21:37.572 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:37 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:21:37.572 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:37 smithi005 ceph-mon[39562]: pgmap v1038: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 6.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-04-20T17:21:37.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:37 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:21:37.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:37 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:21:37.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:37 smithi171 ceph-mon[44189]: pgmap v1038: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 6.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-04-20T17:21:37.615 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:37.803 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 66 16 114199 114183 108.107 78.8125 0.0056491 0.00924785 2024-04-20T17:21:38.165 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:21:38.166 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:21:38.685 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:21:38.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:38.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:21:38.703 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:21:38.775 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:21:38.794 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:21:38.803 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 67 16 115433 115417 107.645 77.125 0.00669326 0.0092835 2024-04-20T17:21:38.821 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:38.844 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:21:38.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:38.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:21:38.949 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:21:38.979 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:39.003 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:21:39.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:39.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:21:39.111 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:21:39.156 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:39.237 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:21:39.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:39.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:21:39.343 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:21:39.372 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:39.399 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:21:39.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:39.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:21:39.503 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:21:39.544 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:39 smithi171 ceph-mon[44189]: pgmap v1039: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 6.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 95 MiB/s wr, 1.53k op/s 2024-04-20T17:21:39.544 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:39.572 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:21:39.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:39.578 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:21:39.679 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:21:39.708 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:39 smithi106 ceph-mon[44423]: pgmap v1039: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 6.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 95 MiB/s wr, 1.53k op/s 2024-04-20T17:21:39.710 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:39.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:39 smithi005 ceph-mon[39562]: pgmap v1039: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 6.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 95 MiB/s wr, 1.53k op/s 2024-04-20T17:21:39.736 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:21:39.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:39.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:21:39.803 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 68 16 116660 116644 107.189 76.6875 0.00734672 0.00932693 2024-04-20T17:21:39.840 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:21:39.867 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:39.893 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:21:39.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:39.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:21:39.996 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:21:40.054 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:40.078 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:21:40.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:40.085 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:21:40.167 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:21:40.185 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:21:40.220 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:40.252 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:21:40.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:40.256 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:21:40.357 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:21:40.402 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:40.433 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:21:40.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:40.443 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:21:40.544 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:21:40.579 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:40.695 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:21:40.803 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 69 16 117827 117811 106.693 72.9375 0.108757 0.00936834 2024-04-20T17:21:41.269 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:41 smithi005 ceph-mon[39562]: pgmap v1040: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 6.9 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-20T17:21:41.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:41 smithi171 ceph-mon[44189]: pgmap v1040: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 6.9 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-20T17:21:41.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:41 smithi106 ceph-mon[44423]: pgmap v1040: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 6.9 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-20T17:21:41.803 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 70 16 119050 119034 106.261 76.4375 0.0096005 0.00940786 2024-04-20T17:21:42.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:42 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:21:42.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:42 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:21:42.486 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:21:42.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:42.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:21:42.515 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:21:42.515 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:21:42.515 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:21:42.515 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:21:42.515 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.515 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:228278 2.3 2:cbde697d:::benchmark_data_smithi171_78918_object67284:head [delete] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-20T17:21:42.515 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:29.213212+0000", 2024-04-20T17:21:42.515 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 493.29703848100002, 2024-04-20T17:21:42.515 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4218705329999999, 2024-04-20T17:21:42.515 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:42.515 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:42.516 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:42.516 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:42.516 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:42.516 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:42.516 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 228278 2024-04-20T17:21:42.516 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.516 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:42.516 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.516 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:42.516 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.213212+0000", 2024-04-20T17:21:42.516 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.516 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.516 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.516 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:42.517 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.213212+0000", 2024-04-20T17:21:42.517 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.517 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.517 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.517 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:42.517 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.213215+0000", 2024-04-20T17:21:42.517 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4930000000000001e-06 2024-04-20T17:21:42.517 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.517 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.517 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:42.517 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.213222+0000", 2024-04-20T17:21:42.517 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2030000000000003e-06 2024-04-20T17:21:42.517 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.518 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.518 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:42.518 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.213235+0000", 2024-04-20T17:21:42.518 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2626000000000001e-05 2024-04-20T17:21:42.518 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.518 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.518 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:42.518 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.213243+0000", 2024-04-20T17:21:42.518 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4239999999999993e-06 2024-04-20T17:21:42.518 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.518 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.518 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.518 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.337241+0000", 2024-04-20T17:21:42.518 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.123998226 2024-04-20T17:21:42.519 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.519 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.519 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:21:42.519 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.344845+0000", 2024-04-20T17:21:42.519 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0076031060000000001 2024-04-20T17:21:42.519 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.519 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.519 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.519 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.387780+0000", 2024-04-20T17:21:42.519 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.042935177999999997 2024-04-20T17:21:42.519 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.520 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.520 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:21:42.520 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.392394+0000", 2024-04-20T17:21:42.520 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0046145170000000003 2024-04-20T17:21:42.520 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.520 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.520 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.520 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.508260+0000", 2024-04-20T17:21:42.520 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1158655610000001 2024-04-20T17:21:42.520 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.520 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.521 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:42.521 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.514617+0000", 2024-04-20T17:21:42.521 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0063572530000000002 2024-04-20T17:21:42.521 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.521 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.521 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:21:42.521 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.532374+0000", 2024-04-20T17:21:42.521 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.017756732000000001 2024-04-20T17:21:42.521 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.521 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.521 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:42.521 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.577271+0000", 2024-04-20T17:21:42.522 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.044896881999999999 2024-04-20T17:21:42.522 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.522 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.522 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:42.522 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.617897+0000", 2024-04-20T17:21:42.522 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.040626439 2024-04-20T17:21:42.522 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.522 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.522 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:42.522 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.621386+0000", 2024-04-20T17:21:42.522 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003488807 2024-04-20T17:21:42.522 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.522 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.522 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.635083+0000", 2024-04-20T17:21:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013697086000000001 2024-04-20T17:21:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:228280 2.3 2:c765d134:::benchmark_data_smithi171_78918_object67286:head [delete] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-20T17:21:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:29.220826+0000", 2024-04-20T17:21:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 493.28942458, 2024-04-20T17:21:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3460507879999999, 2024-04-20T17:21:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:42.524 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:42.524 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:42.524 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:42.524 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 228280 2024-04-20T17:21:42.524 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.524 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:42.524 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.524 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:42.524 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.220826+0000", 2024-04-20T17:21:42.524 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.524 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.524 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.524 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.220826+0000", 2024-04-20T17:21:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.220828+0000", 2024-04-20T17:21:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6610000000000001e-06 2024-04-20T17:21:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.220833+0000", 2024-04-20T17:21:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.668e-06 2024-04-20T17:21:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:42.526 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.220844+0000", 2024-04-20T17:21:42.526 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1131999999999999e-05 2024-04-20T17:21:42.526 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.526 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.526 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:42.526 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.220851+0000", 2024-04-20T17:21:42.526 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8299999999999998e-06 2024-04-20T17:21:42.526 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.526 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.526 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.526 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.345596+0000", 2024-04-20T17:21:42.526 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.124745306 2024-04-20T17:21:42.526 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.526 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:21:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.353177+0000", 2024-04-20T17:21:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.007581366 2024-04-20T17:21:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.429194+0000", 2024-04-20T17:21:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.076017086999999997 2024-04-20T17:21:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:21:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.433408+0000", 2024-04-20T17:21:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0042133099999999996 2024-04-20T17:21:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.445818+0000", 2024-04-20T17:21:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0124105569999999 2024-04-20T17:21:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.455999+0000", 2024-04-20T17:21:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010180751 2024-04-20T17:21:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:21:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.472009+0000", 2024-04-20T17:21:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.016010301000000001 2024-04-20T17:21:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.506770+0000", 2024-04-20T17:21:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.034761045999999997 2024-04-20T17:21:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.548721+0000", 2024-04-20T17:21:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.041950618000000002 2024-04-20T17:21:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.553159+0000", 2024-04-20T17:21:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0044375090000000001 2024-04-20T17:21:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.530 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:42.530 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.566877+0000", 2024-04-20T17:21:42.530 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013718645999999999 2024-04-20T17:21:42.530 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.530 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:42.530 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.530 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.530 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.530 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:228281 2.3 2:cb7bd631:::benchmark_data_smithi171_78918_object67287:head [delete] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-20T17:21:42.530 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:29.604753+0000", 2024-04-20T17:21:42.530 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 492.905498282, 2024-04-20T17:21:42.530 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3682788320000001, 2024-04-20T17:21:42.530 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 228281 2024-04-20T17:21:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.604753+0000", 2024-04-20T17:21:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.604753+0000", 2024-04-20T17:21:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.604755+0000", 2024-04-20T17:21:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3329999999999999e-06 2024-04-20T17:21:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.604762+0000", 2024-04-20T17:21:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5359999999999998e-06 2024-04-20T17:21:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.604774+0000", 2024-04-20T17:21:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2575000000000001e-05 2024-04-20T17:21:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.604783+0000", 2024-04-20T17:21:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5129999999999996e-06 2024-04-20T17:21:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.769321+0000", 2024-04-20T17:21:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.16453883899999999 2024-04-20T17:21:42.534 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.534 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.534 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:21:42.534 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.777087+0000", 2024-04-20T17:21:42.534 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0077658639999999999 2024-04-20T17:21:42.534 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.534 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.534 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.534 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.823978+0000", 2024-04-20T17:21:42.534 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.046890606000000001 2024-04-20T17:21:42.534 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.534 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.534 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:21:42.534 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:29.828156+0000", 2024-04-20T17:21:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041778960000000004 2024-04-20T17:21:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.801221+0000", 2024-04-20T17:21:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9730655019999999 2024-04-20T17:21:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.807293+0000", 2024-04-20T17:21:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0060714280000000002 2024-04-20T17:21:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:21:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.819374+0000", 2024-04-20T17:21:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012080771000000001 2024-04-20T17:21:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.856885+0000", 2024-04-20T17:21:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.037511272999999998 2024-04-20T17:21:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.954790+0000", 2024-04-20T17:21:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.097905345000000005 2024-04-20T17:21:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:42.537 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.959531+0000", 2024-04-20T17:21:42.537 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0047406000000000002 2024-04-20T17:21:42.537 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.537 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.537 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:42.537 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.973032+0000", 2024-04-20T17:21:42.537 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013500751 2024-04-20T17:21:42.537 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.537 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:42.537 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.537 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.538 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.538 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:228399 2.3 2:c70941a5:::benchmark_data_smithi171_78918_object67405:head [delete] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-20T17:21:42.538 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:30.170222+0000", 2024-04-20T17:21:42.538 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 492.34002941, 2024-04-20T17:21:42.538 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.59054261, 2024-04-20T17:21:42.538 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:42.538 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:42.538 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:42.538 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:42.538 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:42.538 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:42.538 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 228399 2024-04-20T17:21:42.538 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.538 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:42.539 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.539 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:42.539 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.170222+0000", 2024-04-20T17:21:42.539 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.539 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.539 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.539 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:42.539 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.170222+0000", 2024-04-20T17:21:42.539 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.539 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.539 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.539 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:42.539 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.170224+0000", 2024-04-20T17:21:42.539 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0729999999999999e-06 2024-04-20T17:21:42.539 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.539 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.539 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:42.540 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.170232+0000", 2024-04-20T17:21:42.540 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7320000000000004e-06 2024-04-20T17:21:42.540 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.540 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.540 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:42.540 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.170245+0000", 2024-04-20T17:21:42.540 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2333e-05 2024-04-20T17:21:42.540 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.540 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.540 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:42.540 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.170255+0000", 2024-04-20T17:21:42.540 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9860000000000006e-06 2024-04-20T17:21:42.540 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.540 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.540 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.540 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.384564+0000", 2024-04-20T17:21:42.541 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.21430969599999999 2024-04-20T17:21:42.541 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.541 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.541 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:21:42.541 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.392665+0000", 2024-04-20T17:21:42.541 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.008101044 2024-04-20T17:21:42.541 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.541 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.541 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.541 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.450812+0000", 2024-04-20T17:21:42.541 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.058146301999999997 2024-04-20T17:21:42.541 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.541 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.541 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:21:42.541 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.454910+0000", 2024-04-20T17:21:42.542 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0040980859999999999 2024-04-20T17:21:42.542 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.542 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.542 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.542 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.477906+0000", 2024-04-20T17:21:42.542 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.022996129 2024-04-20T17:21:42.542 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.542 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.542 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:42.542 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.486187+0000", 2024-04-20T17:21:42.542 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0082809719999999993 2024-04-20T17:21:42.542 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.542 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.543 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:21:42.543 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.501211+0000", 2024-04-20T17:21:42.543 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015024404 2024-04-20T17:21:42.543 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.543 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.543 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:42.543 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.542755+0000", 2024-04-20T17:21:42.543 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.041543941000000001 2024-04-20T17:21:42.543 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.543 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.543 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:42.543 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.743025+0000", 2024-04-20T17:21:42.543 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.20027012 2024-04-20T17:21:42.543 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.544 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.544 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:42.544 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.747199+0000", 2024-04-20T17:21:42.544 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.004173693 2024-04-20T17:21:42.544 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.544 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.544 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:42.544 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.760764+0000", 2024-04-20T17:21:42.544 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013565099000000001 2024-04-20T17:21:42.544 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.544 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:42.544 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.544 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.545 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.545 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:228401 2.3 2:cb66e8d6:::benchmark_data_smithi171_78918_object67407:head [delete] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-20T17:21:42.545 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:30.176585+0000", 2024-04-20T17:21:42.545 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 492.33366562399999, 2024-04-20T17:21:42.545 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6073272909999998, 2024-04-20T17:21:42.545 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:42.545 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:42.545 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:42.545 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:42.545 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:42.545 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:42.545 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 228401 2024-04-20T17:21:42.546 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.546 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:42.546 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.546 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:42.546 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.176585+0000", 2024-04-20T17:21:42.546 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.546 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.546 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.546 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:42.546 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.176585+0000", 2024-04-20T17:21:42.546 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.546 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.547 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.547 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:42.547 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.176587+0000", 2024-04-20T17:21:42.547 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7910000000000001e-06 2024-04-20T17:21:42.547 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.547 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.547 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:42.547 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.176594+0000", 2024-04-20T17:21:42.547 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5950000000000004e-06 2024-04-20T17:21:42.547 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.547 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.547 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:42.547 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.176605+0000", 2024-04-20T17:21:42.547 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1304000000000001e-05 2024-04-20T17:21:42.548 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.548 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.548 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:42.548 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.176613+0000", 2024-04-20T17:21:42.548 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5680000000000002e-06 2024-04-20T17:21:42.548 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.548 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.548 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.548 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.393900+0000", 2024-04-20T17:21:42.548 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.21728766199999999 2024-04-20T17:21:42.548 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.548 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.548 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:21:42.548 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.402263+0000", 2024-04-20T17:21:42.549 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0083628279999999992 2024-04-20T17:21:42.549 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.549 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.549 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.549 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.485909+0000", 2024-04-20T17:21:42.549 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.083645419999999998 2024-04-20T17:21:42.549 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.549 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.549 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:21:42.549 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.490020+0000", 2024-04-20T17:21:42.549 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041117300000000001 2024-04-20T17:21:42.549 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.549 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.549 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.549 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.589948+0000", 2024-04-20T17:21:42.550 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0999282500000001 2024-04-20T17:21:42.550 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.550 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.550 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:42.550 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.596087+0000", 2024-04-20T17:21:42.550 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0061383920000000003 2024-04-20T17:21:42.550 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.550 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.550 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:21:42.550 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.608189+0000", 2024-04-20T17:21:42.550 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012101835999999999 2024-04-20T17:21:42.550 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.550 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.551 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:42.551 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.764145+0000", 2024-04-20T17:21:42.551 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.15595646499999999 2024-04-20T17:21:42.551 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.551 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.551 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:42.551 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.766235+0000", 2024-04-20T17:21:42.551 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0020894360000000001 2024-04-20T17:21:42.551 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.551 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.551 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:42.551 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.770408+0000", 2024-04-20T17:21:42.551 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041729999999999996 2024-04-20T17:21:42.551 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.552 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.552 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:42.552 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.783913+0000", 2024-04-20T17:21:42.552 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013505013999999999 2024-04-20T17:21:42.552 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.552 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:42.552 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.552 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.552 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.552 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:228553 2.3 2:c472042b:::benchmark_data_smithi171_78918_object67559:head [delete] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-20T17:21:42.552 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:30.926355+0000", 2024-04-20T17:21:42.552 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 491.58389645900002, 2024-04-20T17:21:42.552 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.445260926, 2024-04-20T17:21:42.552 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:42.553 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:42.553 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:42.553 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:42.553 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:42.553 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:42.553 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 228553 2024-04-20T17:21:42.553 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.553 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:42.553 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.553 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:42.553 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.926355+0000", 2024-04-20T17:21:42.554 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.554 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.554 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.554 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:42.554 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.926355+0000", 2024-04-20T17:21:42.554 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.555 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.555 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.555 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:42.555 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.926357+0000", 2024-04-20T17:21:42.555 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7640000000000001e-06 2024-04-20T17:21:42.555 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.555 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.555 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:42.555 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.926369+0000", 2024-04-20T17:21:42.555 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1695000000000001e-05 2024-04-20T17:21:42.555 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.555 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.555 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:42.555 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.926381+0000", 2024-04-20T17:21:42.555 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1564e-05 2024-04-20T17:21:42.556 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.556 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.556 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:42.556 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:30.926390+0000", 2024-04-20T17:21:42.556 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7790000000000007e-06 2024-04-20T17:21:42.556 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.556 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.556 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.556 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.023105+0000", 2024-04-20T17:21:42.556 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.096714498999999995 2024-04-20T17:21:42.556 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.556 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.556 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:21:42.557 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.032045+0000", 2024-04-20T17:21:42.557 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00894 2024-04-20T17:21:42.557 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.557 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.557 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.557 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.098624+0000", 2024-04-20T17:21:42.557 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.066579360000000004 2024-04-20T17:21:42.557 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.557 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.557 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:21:42.557 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.102937+0000", 2024-04-20T17:21:42.557 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0043129550000000003 2024-04-20T17:21:42.557 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.557 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.558 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.558 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.009478+0000", 2024-04-20T17:21:42.558 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9065410709999999 2024-04-20T17:21:42.558 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.558 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.558 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:42.558 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.015563+0000", 2024-04-20T17:21:42.558 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.006085282 2024-04-20T17:21:42.558 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.558 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.558 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:21:42.558 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.027745+0000", 2024-04-20T17:21:42.558 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012181053000000001 2024-04-20T17:21:42.559 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.559 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.559 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:42.559 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.151400+0000", 2024-04-20T17:21:42.559 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.123655958 2024-04-20T17:21:42.559 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.559 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.559 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:42.559 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.350233+0000", 2024-04-20T17:21:42.559 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.19883220500000001 2024-04-20T17:21:42.559 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.559 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.559 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:42.559 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.355745+0000", 2024-04-20T17:21:42.560 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0055118099999999998 2024-04-20T17:21:42.560 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.560 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.560 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:42.560 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.371615+0000", 2024-04-20T17:21:42.560 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015870931000000001 2024-04-20T17:21:42.560 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.560 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:42.560 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.560 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.560 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.560 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:228579 2.3 2:c4136e50:::benchmark_data_smithi171_78918_object67585:head [delete] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-20T17:21:42.560 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:31.034417+0000", 2024-04-20T17:21:42.560 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 491.47583353499999, 2024-04-20T17:21:42.560 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3561227750000002, 2024-04-20T17:21:42.561 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:42.561 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:42.561 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:42.561 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:42.561 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:42.561 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:42.561 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 228579 2024-04-20T17:21:42.561 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.561 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:42.561 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.561 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:42.561 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.034417+0000", 2024-04-20T17:21:42.561 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.561 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.561 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.561 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:42.562 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.034417+0000", 2024-04-20T17:21:42.562 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.562 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.562 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.562 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:42.562 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.034420+0000", 2024-04-20T17:21:42.562 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.491e-06 2024-04-20T17:21:42.562 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.562 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.562 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:42.562 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.034427+0000", 2024-04-20T17:21:42.562 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1010000000000003e-06 2024-04-20T17:21:42.562 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.562 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.563 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:42.563 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.034439+0000", 2024-04-20T17:21:42.563 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1761e-05 2024-04-20T17:21:42.563 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.563 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.563 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:42.563 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.034448+0000", 2024-04-20T17:21:42.563 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0389999999999993e-06 2024-04-20T17:21:42.563 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.563 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.563 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.563 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.165259+0000", 2024-04-20T17:21:42.563 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.130811069 2024-04-20T17:21:42.563 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.563 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.564 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:21:42.564 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.172966+0000", 2024-04-20T17:21:42.564 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0077074969999999998 2024-04-20T17:21:42.564 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.564 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.564 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.564 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.243877+0000", 2024-04-20T17:21:42.564 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.070910794999999999 2024-04-20T17:21:42.564 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.564 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.564 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:21:42.564 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.248014+0000", 2024-04-20T17:21:42.564 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041367089999999997 2024-04-20T17:21:42.564 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.564 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.565 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.565 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.065976+0000", 2024-04-20T17:21:42.565 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.817961926 2024-04-20T17:21:42.565 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.565 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.565 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:42.565 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.072436+0000", 2024-04-20T17:21:42.565 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0064605950000000004 2024-04-20T17:21:42.565 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.565 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.565 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:21:42.565 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.084520+0000", 2024-04-20T17:21:42.565 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012083774 2024-04-20T17:21:42.565 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.566 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.566 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:42.566 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.154180+0000", 2024-04-20T17:21:42.566 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.06965971 2024-04-20T17:21:42.566 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.566 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.566 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:42.566 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.372323+0000", 2024-04-20T17:21:42.566 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.21814293600000001 2024-04-20T17:21:42.566 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.566 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.566 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:42.566 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.375811+0000", 2024-04-20T17:21:42.566 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0034884730000000002 2024-04-20T17:21:42.566 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.567 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.567 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:42.567 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.390540+0000", 2024-04-20T17:21:42.567 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014728899 2024-04-20T17:21:42.567 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.567 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:42.567 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.567 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.567 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.567 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:228644 2.3 2:c8d65292:::benchmark_data_smithi171_78918_object67650:head [delete] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-20T17:21:42.567 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:31.656045+0000", 2024-04-20T17:21:42.567 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 490.85420596799997, 2024-04-20T17:21:42.567 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.428389734, 2024-04-20T17:21:42.567 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:42.567 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:42.567 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:42.568 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:42.568 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:42.568 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:42.568 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 228644 2024-04-20T17:21:42.568 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.568 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:42.568 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.568 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:42.568 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.656045+0000", 2024-04-20T17:21:42.568 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.568 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.568 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.568 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:42.568 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.656045+0000", 2024-04-20T17:21:42.568 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.568 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.569 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:42.569 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.656048+0000", 2024-04-20T17:21:42.569 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6620000000000001e-06 2024-04-20T17:21:42.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.569 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:42.569 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.656054+0000", 2024-04-20T17:21:42.569 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8000000000000001e-06 2024-04-20T17:21:42.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.569 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:42.569 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.656067+0000", 2024-04-20T17:21:42.569 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2381999999999999e-05 2024-04-20T17:21:42.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.569 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:42.570 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.656076+0000", 2024-04-20T17:21:42.570 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6780000000000006e-06 2024-04-20T17:21:42.570 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.570 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.570 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.570 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.680283+0000", 2024-04-20T17:21:42.570 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.024207619 2024-04-20T17:21:42.570 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.570 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.570 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:21:42.570 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.687974+0000", 2024-04-20T17:21:42.570 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0076904490000000002 2024-04-20T17:21:42.570 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.570 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.571 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.571 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.747637+0000", 2024-04-20T17:21:42.571 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.059663348999999997 2024-04-20T17:21:42.571 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.571 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.571 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:21:42.571 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:31.751859+0000", 2024-04-20T17:21:42.571 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0042222179999999998 2024-04-20T17:21:42.571 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.571 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.571 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.571 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.969716+0000", 2024-04-20T17:21:42.571 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2178571630000001 2024-04-20T17:21:42.571 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.572 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:42.572 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.977217+0000", 2024-04-20T17:21:42.572 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0075008410000000003 2024-04-20T17:21:42.572 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.572 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:21:42.572 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.989454+0000", 2024-04-20T17:21:42.572 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012237125 2024-04-20T17:21:42.572 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.572 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:42.572 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:34.005410+0000", 2024-04-20T17:21:42.572 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015955666 2024-04-20T17:21:42.573 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.573 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.573 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:42.573 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:34.066358+0000", 2024-04-20T17:21:42.573 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.060947875999999998 2024-04-20T17:21:42.573 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.573 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.573 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:42.573 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:34.070486+0000", 2024-04-20T17:21:42.573 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041281529999999999 2024-04-20T17:21:42.573 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.573 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.573 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:42.573 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:34.084435+0000", 2024-04-20T17:21:42.574 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013948752999999999 2024-04-20T17:21:42.574 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.574 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:42.574 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.574 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.574 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.574 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:228734 2.3 2:c6201dcd:::benchmark_data_smithi171_78918_object67740:head [delete] snapc 0=[] ondisk+write+known_if_redirected e229)", 2024-04-20T17:21:42.574 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:32.283347+0000", 2024-04-20T17:21:42.574 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 490.22690401800003, 2024-04-20T17:21:42.574 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3561478459999998, 2024-04-20T17:21:42.574 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:42.574 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:42.574 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:42.574 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:42.575 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:42.575 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:42.575 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 228734 2024-04-20T17:21:42.575 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.575 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:42.575 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.575 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:42.575 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.283347+0000", 2024-04-20T17:21:42.575 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.575 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.575 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.575 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:42.575 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.283347+0000", 2024-04-20T17:21:42.575 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.575 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.576 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.576 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:42.576 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.283348+0000", 2024-04-20T17:21:42.576 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4950000000000001e-06 2024-04-20T17:21:42.576 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.576 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.576 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:42.576 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.283353+0000", 2024-04-20T17:21:42.576 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.006e-06 2024-04-20T17:21:42.576 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.576 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.576 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:42.576 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.283365+0000", 2024-04-20T17:21:42.576 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1686e-05 2024-04-20T17:21:42.577 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.577 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.577 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:42.577 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.283374+0000", 2024-04-20T17:21:42.577 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4239999999999993e-06 2024-04-20T17:21:42.577 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.577 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.577 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.577 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.315956+0000", 2024-04-20T17:21:42.577 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.032582425999999998 2024-04-20T17:21:42.577 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.577 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.577 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:21:42.577 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.324563+0000", 2024-04-20T17:21:42.577 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0086066839999999999 2024-04-20T17:21:42.578 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.578 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.578 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.578 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.414971+0000", 2024-04-20T17:21:42.578 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.090408295 2024-04-20T17:21:42.578 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.578 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.578 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:21:42.578 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:32.420910+0000", 2024-04-20T17:21:42.578 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0059390040000000003 2024-04-20T17:21:42.578 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.578 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.578 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.578 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:34.528387+0000", 2024-04-20T17:21:42.579 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1074769180000001 2024-04-20T17:21:42.579 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.579 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.579 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:42.579 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:34.534511+0000", 2024-04-20T17:21:42.579 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0061236490000000001 2024-04-20T17:21:42.579 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.579 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.579 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:21:42.579 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:34.546825+0000", 2024-04-20T17:21:42.579 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012314794 2024-04-20T17:21:42.579 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.579 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.579 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:42.579 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:34.614264+0000", 2024-04-20T17:21:42.581 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.067438940000000003 2024-04-20T17:21:42.582 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.582 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.582 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:42.582 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:34.620647+0000", 2024-04-20T17:21:42.582 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0063823370000000001 2024-04-20T17:21:42.582 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.582 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.582 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:42.582 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:34.624696+0000", 2024-04-20T17:21:42.582 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0040494559999999999 2024-04-20T17:21:42.582 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.582 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.582 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:42.582 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:34.639495+0000", 2024-04-20T17:21:42.583 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014798732 2024-04-20T17:21:42.583 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.583 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:42.583 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.583 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.583 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.583 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:228913 2.3 2:c8d17bf9:::benchmark_data_smithi171_78918_object67919:head [delete] snapc 0=[] ondisk+write+known_if_redirected e229)", 2024-04-20T17:21:42.583 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:33.652143+0000", 2024-04-20T17:21:42.583 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 488.858108125, 2024-04-20T17:21:42.583 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3437341059999999, 2024-04-20T17:21:42.583 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:42.583 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:42.583 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:42.583 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:42.584 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:42.584 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:42.584 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 228913 2024-04-20T17:21:42.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.584 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:42.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.584 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:42.584 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.652143+0000", 2024-04-20T17:21:42.584 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.584 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:42.584 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.652143+0000", 2024-04-20T17:21:42.584 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.585 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:42.585 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.652145+0000", 2024-04-20T17:21:42.585 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0360000000000001e-06 2024-04-20T17:21:42.585 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.585 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.585 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:42.585 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.652151+0000", 2024-04-20T17:21:42.585 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0440000000000001e-06 2024-04-20T17:21:42.585 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.585 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.585 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:42.585 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.652161+0000", 2024-04-20T17:21:42.585 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0419e-05 2024-04-20T17:21:42.585 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.585 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.586 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:42.586 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.652168+0000", 2024-04-20T17:21:42.586 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4749999999999998e-06 2024-04-20T17:21:42.586 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.586 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.586 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.586 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.751065+0000", 2024-04-20T17:21:42.586 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.098896703000000002 2024-04-20T17:21:42.586 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.586 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.586 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:21:42.586 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.759742+0000", 2024-04-20T17:21:42.586 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0086775240000000007 2024-04-20T17:21:42.586 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.587 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.587 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.587 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.829542+0000", 2024-04-20T17:21:42.587 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.069800143999999995 2024-04-20T17:21:42.587 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.587 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.587 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:21:42.587 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:33.834830+0000", 2024-04-20T17:21:42.587 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0052875709999999996 2024-04-20T17:21:42.587 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.587 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.587 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.587 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:35.831588+0000", 2024-04-20T17:21:42.587 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.996758094 2024-04-20T17:21:42.587 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.588 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.588 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:42.588 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:35.838563+0000", 2024-04-20T17:21:42.588 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0069750209999999997 2024-04-20T17:21:42.588 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.588 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.588 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:21:42.588 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:35.857019+0000", 2024-04-20T17:21:42.588 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.018455849 2024-04-20T17:21:42.588 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.588 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.588 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:42.588 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:35.873901+0000", 2024-04-20T17:21:42.588 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.016882766 2024-04-20T17:21:42.588 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.589 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.589 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:42.589 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:35.978024+0000", 2024-04-20T17:21:42.589 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.104122649 2024-04-20T17:21:42.589 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.589 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.589 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:42.589 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:35.982208+0000", 2024-04-20T17:21:42.589 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041836479999999999 2024-04-20T17:21:42.589 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.589 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.589 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:42.589 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:35.995877+0000", 2024-04-20T17:21:42.589 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013669163 2024-04-20T17:21:42.590 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.590 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:42.590 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.590 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.590 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.590 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:231451 2.3 2:ce942c68:::benchmark_data_smithi171_78918_object70457:head [delete] snapc 0=[] ondisk+write+known_if_redirected e237)", 2024-04-20T17:21:42.590 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:13:53.608704+0000", 2024-04-20T17:21:42.590 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 468.901546505, 2024-04-20T17:21:42.590 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3313055249999999, 2024-04-20T17:21:42.590 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:42.590 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:42.590 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:42.590 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:42.590 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:42.591 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:42.591 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 231451 2024-04-20T17:21:42.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.591 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:42.591 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.591 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:42.591 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:53.608704+0000", 2024-04-20T17:21:42.591 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.591 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.591 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:42.591 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:53.608704+0000", 2024-04-20T17:21:42.591 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.591 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.591 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:42.592 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:53.608707+0000", 2024-04-20T17:21:42.592 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3130000000000001e-06 2024-04-20T17:21:42.592 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.592 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.592 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:42.592 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:53.608714+0000", 2024-04-20T17:21:42.592 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4420000000000001e-06 2024-04-20T17:21:42.592 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.592 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.592 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:42.592 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:53.608729+0000", 2024-04-20T17:21:42.592 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4284000000000001e-05 2024-04-20T17:21:42.592 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.592 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.592 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:42.593 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:53.608740+0000", 2024-04-20T17:21:42.593 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1148999999999999e-05 2024-04-20T17:21:42.593 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.593 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.593 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.593 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:53.679980+0000", 2024-04-20T17:21:42.593 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.071240205000000001 2024-04-20T17:21:42.593 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.593 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.593 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:21:42.593 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:53.692861+0000", 2024-04-20T17:21:42.593 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012881182 2024-04-20T17:21:42.593 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.593 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.594 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.594 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:53.787511+0000", 2024-04-20T17:21:42.594 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.094649464000000003 2024-04-20T17:21:42.594 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.594 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.594 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:21:42.594 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:53.791542+0000", 2024-04-20T17:21:42.594 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0040310779999999996 2024-04-20T17:21:42.594 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.594 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.594 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.594 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:55.833719+0000", 2024-04-20T17:21:42.594 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0421774479999999 2024-04-20T17:21:42.595 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.595 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:42.595 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:55.839813+0000", 2024-04-20T17:21:42.595 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0060943270000000001 2024-04-20T17:21:42.595 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.595 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:21:42.595 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:55.851947+0000", 2024-04-20T17:21:42.595 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.01213349 2024-04-20T17:21:42.595 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.595 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:42.595 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:55.919140+0000", 2024-04-20T17:21:42.595 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.067192937999999994 2024-04-20T17:21:42.595 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:42.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:55.920484+0000", 2024-04-20T17:21:42.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0013444329999999999 2024-04-20T17:21:42.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.596 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:42.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:55.924525+0000", 2024-04-20T17:21:42.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0040412649999999996 2024-04-20T17:21:42.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.596 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:42.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:13:55.940010+0000", 2024-04-20T17:21:42.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015484507 2024-04-20T17:21:42.596 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.596 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:42.596 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.597 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:233551 2.3 2:c717208a:::benchmark_data_smithi171_78918_object72557:head [delete] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-04-20T17:21:42.597 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:14:09.264468+0000", 2024-04-20T17:21:42.597 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 453.24578306900003, 2024-04-20T17:21:42.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3236778899999999, 2024-04-20T17:21:42.597 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:42.597 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:42.597 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:42.597 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:42.597 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:42.597 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:42.597 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 233551 2024-04-20T17:21:42.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.597 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:42.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.598 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:42.598 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.264468+0000", 2024-04-20T17:21:42.598 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.598 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.598 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.598 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:42.598 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.264468+0000", 2024-04-20T17:21:42.598 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.598 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.598 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.598 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:42.598 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.264470+0000", 2024-04-20T17:21:42.598 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5390000000000002e-06 2024-04-20T17:21:42.599 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.599 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.599 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:42.599 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.264478+0000", 2024-04-20T17:21:42.599 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.294e-06 2024-04-20T17:21:42.599 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.599 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.599 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:42.599 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.264491+0000", 2024-04-20T17:21:42.599 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3305999999999999e-05 2024-04-20T17:21:42.599 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.599 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.599 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:42.599 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.264502+0000", 2024-04-20T17:21:42.599 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1326e-05 2024-04-20T17:21:42.600 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.600 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.600 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.600 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.427957+0000", 2024-04-20T17:21:42.600 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.16345464500000001 2024-04-20T17:21:42.600 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.600 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.600 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:21:42.600 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.435153+0000", 2024-04-20T17:21:42.600 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.007196282 2024-04-20T17:21:42.600 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.600 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.600 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.490235+0000", 2024-04-20T17:21:42.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.055081443000000001 2024-04-20T17:21:42.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:21:42.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:09.494200+0000", 2024-04-20T17:21:42.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003965161 2024-04-20T17:21:42.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:11.427618+0000", 2024-04-20T17:21:42.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9334183870000001 2024-04-20T17:21:42.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:42.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:11.433616+0000", 2024-04-20T17:21:42.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0059977229999999999 2024-04-20T17:21:42.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:21:42.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:11.445335+0000", 2024-04-20T17:21:42.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011719283 2024-04-20T17:21:42.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:42.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:11.537472+0000", 2024-04-20T17:21:42.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.092136309 2024-04-20T17:21:42.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.603 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:42.603 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:11.570480+0000", 2024-04-20T17:21:42.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.033008570000000001 2024-04-20T17:21:42.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.603 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:42.603 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:11.575045+0000", 2024-04-20T17:21:42.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0045648030000000001 2024-04-20T17:21:42.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.603 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:42.603 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:11.588146+0000", 2024-04-20T17:21:42.604 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013100819 2024-04-20T17:21:42.604 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.604 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:42.604 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.604 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.604 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:233667 2.3 2:c866dd2f:::benchmark_data_smithi171_78918_object72673:head [delete] snapc 0=[] ondisk+write+known_if_redirected e243)", 2024-04-20T17:21:42.604 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:14:10.550057+0000", 2024-04-20T17:21:42.604 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 451.96019418600002, 2024-04-20T17:21:42.604 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3848929349999999, 2024-04-20T17:21:42.604 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:42.604 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:42.604 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:42.604 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:42.605 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:42.605 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:42.605 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 233667 2024-04-20T17:21:42.605 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.605 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:42.605 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.605 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:42.605 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.550057+0000", 2024-04-20T17:21:42.605 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.605 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.605 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.605 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:42.605 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.550057+0000", 2024-04-20T17:21:42.605 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.606 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.606 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.606 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:42.606 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.550061+0000", 2024-04-20T17:21:42.606 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3470000000000001e-06 2024-04-20T17:21:42.606 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.606 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.606 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:42.606 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.550074+0000", 2024-04-20T17:21:42.606 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3032e-05 2024-04-20T17:21:42.606 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.606 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.606 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:42.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.550089+0000", 2024-04-20T17:21:42.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5142e-05 2024-04-20T17:21:42.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.607 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:42.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.550100+0000", 2024-04-20T17:21:42.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0626e-05 2024-04-20T17:21:42.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.607 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.632247+0000", 2024-04-20T17:21:42.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.082146648000000003 2024-04-20T17:21:42.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.607 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:21:42.608 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.639860+0000", 2024-04-20T17:21:42.608 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0076133900000000003 2024-04-20T17:21:42.608 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.608 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.608 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.608 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.674992+0000", 2024-04-20T17:21:42.608 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.035132413000000001 2024-04-20T17:21:42.608 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.608 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.608 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:21:42.608 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:10.678995+0000", 2024-04-20T17:21:42.608 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0040029799999999997 2024-04-20T17:21:42.608 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.608 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.609 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.738116+0000", 2024-04-20T17:21:42.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.059120294 2024-04-20T17:21:42.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.609 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:42.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.746723+0000", 2024-04-20T17:21:42.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0086068800000000008 2024-04-20T17:21:42.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.609 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:21:42.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.763749+0000", 2024-04-20T17:21:42.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.017026935 2024-04-20T17:21:42.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.610 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.610 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:42.610 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.894660+0000", 2024-04-20T17:21:42.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.13091038299999999 2024-04-20T17:21:42.610 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.610 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.610 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:42.610 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.918102+0000", 2024-04-20T17:21:42.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023442187999999999 2024-04-20T17:21:42.610 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.610 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.610 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:42.610 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.921919+0000", 2024-04-20T17:21:42.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0038171429999999998 2024-04-20T17:21:42.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.611 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.611 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:42.611 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:14:12.934950+0000", 2024-04-20T17:21:42.611 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013030534 2024-04-20T17:21:42.611 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.611 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:42.611 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.611 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.611 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:259121 2.3 2:c91e897c:::benchmark_data_smithi171_78918_object98127:head [delete] snapc 0=[] ondisk+write+known_if_redirected e294)", 2024-04-20T17:21:42.611 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:17:07.741988+0000", 2024-04-20T17:21:42.611 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 274.76826337300002, 2024-04-20T17:21:42.611 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.645843406, 2024-04-20T17:21:42.611 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:42.612 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:42.612 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:42.612 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:42.612 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:42.612 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:42.612 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 259121 2024-04-20T17:21:42.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.612 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:42.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.612 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:42.612 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:07.741988+0000", 2024-04-20T17:21:42.612 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.613 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:42.613 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:07.741988+0000", 2024-04-20T17:21:42.613 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.613 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.613 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:42.613 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:07.741990+0000", 2024-04-20T17:21:42.613 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.08e-06 2024-04-20T17:21:42.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.613 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.613 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:42.613 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:07.741995+0000", 2024-04-20T17:21:42.613 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.789e-06 2024-04-20T17:21:42.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.614 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:42.614 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:07.742007+0000", 2024-04-20T17:21:42.614 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1728e-05 2024-04-20T17:21:42.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.614 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:42.614 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:07.742013+0000", 2024-04-20T17:21:42.614 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0680000000000002e-06 2024-04-20T17:21:42.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.614 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.614 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:07.826185+0000", 2024-04-20T17:21:42.615 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.084171678999999999 2024-04-20T17:21:42.615 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.615 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:21:42.615 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:07.828716+0000", 2024-04-20T17:21:42.615 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00253126 2024-04-20T17:21:42.615 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.615 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.615 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:08.499283+0000", 2024-04-20T17:21:42.615 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.67056691899999998 2024-04-20T17:21:42.615 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.615 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:21:42.616 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:08.501962+0000", 2024-04-20T17:21:42.616 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0026792109999999999 2024-04-20T17:21:42.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.616 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.616 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:10.238389+0000", 2024-04-20T17:21:42.616 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.736427071 2024-04-20T17:21:42.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.616 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:42.616 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:10.242452+0000", 2024-04-20T17:21:42.616 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0040623830000000001 2024-04-20T17:21:42.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.617 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:21:42.617 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:10.250550+0000", 2024-04-20T17:21:42.617 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0080983040000000006 2024-04-20T17:21:42.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.617 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.617 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:42.617 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:10.295991+0000", 2024-04-20T17:21:42.617 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.045440734000000003 2024-04-20T17:21:42.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.617 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.617 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:42.617 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:10.375642+0000", 2024-04-20T17:21:42.617 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.079650913000000004 2024-04-20T17:21:42.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.618 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:42.618 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:10.378793+0000", 2024-04-20T17:21:42.618 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003151526 2024-04-20T17:21:42.618 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.618 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:42.618 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:10.387831+0000", 2024-04-20T17:21:42.618 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0090377410000000002 2024-04-20T17:21:42.618 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.618 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:42.618 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.618 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.619 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:259416 2.3 2:c3b5e54f:::benchmark_data_smithi171_78918_object98422:head [delete] snapc 0=[] ondisk+write+known_if_redirected e294)", 2024-04-20T17:21:42.619 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:17:09.086846+0000", 2024-04-20T17:21:42.619 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 273.42340513400001, 2024-04-20T17:21:42.619 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.90832223, 2024-04-20T17:21:42.619 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:42.619 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:42.619 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:42.619 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:42.619 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:42.619 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:42.619 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 259416 2024-04-20T17:21:42.619 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.619 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:42.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.620 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:42.620 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:09.086846+0000", 2024-04-20T17:21:42.620 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.620 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.620 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.620 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:42.620 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:09.086846+0000", 2024-04-20T17:21:42.620 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.620 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.620 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.620 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:42.620 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:09.086850+0000", 2024-04-20T17:21:42.620 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.087e-06 2024-04-20T17:21:42.620 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.621 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.621 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:42.621 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:09.086865+0000", 2024-04-20T17:21:42.621 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5051999999999999e-05 2024-04-20T17:21:42.621 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.621 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.621 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:42.621 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:09.086892+0000", 2024-04-20T17:21:42.621 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7324000000000001e-05 2024-04-20T17:21:42.621 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.621 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.621 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:42.621 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:09.086908+0000", 2024-04-20T17:21:42.621 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5880999999999999e-05 2024-04-20T17:21:42.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.622 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.622 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:09.110572+0000", 2024-04-20T17:21:42.622 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.02366383 2024-04-20T17:21:42.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.622 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:21:42.622 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:09.112793+0000", 2024-04-20T17:21:42.622 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0022206690000000002 2024-04-20T17:21:42.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.622 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.623 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:09.770052+0000", 2024-04-20T17:21:42.623 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.65725883699999998 2024-04-20T17:21:42.623 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.623 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:21:42.623 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:09.772834+0000", 2024-04-20T17:21:42.623 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0027826449999999998 2024-04-20T17:21:42.623 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.623 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.623 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:11.923958+0000", 2024-04-20T17:21:42.623 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.151123519 2024-04-20T17:21:42.623 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.624 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:42.624 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:11.927994+0000", 2024-04-20T17:21:42.624 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0040365269999999998 2024-04-20T17:21:42.624 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.624 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.624 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:21:42.624 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:11.936064+0000", 2024-04-20T17:21:42.624 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0080701729999999999 2024-04-20T17:21:42.624 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.624 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.624 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:42.624 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:11.961695+0000", 2024-04-20T17:21:42.624 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.025630868000000001 2024-04-20T17:21:42.624 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.624 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.625 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:42.625 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:11.982001+0000", 2024-04-20T17:21:42.625 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.020305349 2024-04-20T17:21:42.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.625 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.625 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:42.625 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:11.986291+0000", 2024-04-20T17:21:42.625 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0042906259999999996 2024-04-20T17:21:42.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.625 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.625 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:42.625 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:11.995168+0000", 2024-04-20T17:21:42.625 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0088768430000000006 2024-04-20T17:21:42.625 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.625 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:42.626 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.626 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.626 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.626 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:263462 2.3 2:cae5d6f3:::benchmark_data_smithi171_78918_object102468:head [delete] snapc 0=[] ondisk+write+known_if_redirected e302)", 2024-04-20T17:21:42.626 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:17:31.688766+0000", 2024-04-20T17:21:42.626 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 250.82148485100001, 2024-04-20T17:21:42.626 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.724801673, 2024-04-20T17:21:42.626 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:42.626 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:42.626 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-04-20T17:21:42.626 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:42.626 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:42.626 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:42.627 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 263462 2024-04-20T17:21:42.627 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.627 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:42.627 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.627 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:42.627 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:31.688766+0000", 2024-04-20T17:21:42.627 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.627 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.627 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.627 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:42.627 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:31.688766+0000", 2024-04-20T17:21:42.627 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.627 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.627 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.628 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:42.628 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:31.688768+0000", 2024-04-20T17:21:42.628 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.773e-06 2024-04-20T17:21:42.628 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.628 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.628 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:42.628 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:31.688773+0000", 2024-04-20T17:21:42.628 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0669999999999999e-06 2024-04-20T17:21:42.628 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.628 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.628 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:42.628 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:31.688785+0000", 2024-04-20T17:21:42.628 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1792e-05 2024-04-20T17:21:42.628 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.629 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.629 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:42.629 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:31.688792+0000", 2024-04-20T17:21:42.629 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2010000000000002e-06 2024-04-20T17:21:42.629 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.629 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.629 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.629 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:31.835481+0000", 2024-04-20T17:21:42.629 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.146689184 2024-04-20T17:21:42.629 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.629 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.629 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:21:42.629 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:31.837531+0000", 2024-04-20T17:21:42.629 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002049993 2024-04-20T17:21:42.629 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.630 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.630 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.630 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:32.495598+0000", 2024-04-20T17:21:42.630 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.65806703499999997 2024-04-20T17:21:42.630 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.630 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.630 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:21:42.630 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:32.498093+0000", 2024-04-20T17:21:42.630 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0024949080000000001 2024-04-20T17:21:42.630 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.630 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.630 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.630 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:34.413013+0000", 2024-04-20T17:21:42.631 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9149201200000001 2024-04-20T17:21:42.631 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.631 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.631 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:42.631 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:34.413568+0000", 2024-04-20T17:21:42.631 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00055460000000000004 2024-04-20T17:21:42.631 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.631 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:42.631 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.631 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.631 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.631 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:267267 2.3 2:c16bfb3f:::benchmark_data_smithi171_78918_object106273:head [delete] snapc 0=[] ondisk+write+known_if_redirected e308)", 2024-04-20T17:21:42.631 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:17:50.686207+0000", 2024-04-20T17:21:42.631 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 231.82404347900001, 2024-04-20T17:21:42.631 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7237566850000001, 2024-04-20T17:21:42.631 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:42.632 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:42.632 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:42.632 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:42.632 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:42.632 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:42.632 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 267267 2024-04-20T17:21:42.632 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.632 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:42.632 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.632 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:42.632 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:50.686207+0000", 2024-04-20T17:21:42.632 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.632 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.632 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.632 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:42.632 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:50.686207+0000", 2024-04-20T17:21:42.633 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.633 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.633 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.633 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:42.633 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:50.686210+0000", 2024-04-20T17:21:42.633 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5739999999999998e-06 2024-04-20T17:21:42.633 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.633 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.633 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:42.633 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:50.686219+0000", 2024-04-20T17:21:42.633 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6610000000000006e-06 2024-04-20T17:21:42.633 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.633 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.633 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:42.633 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:50.686231+0000", 2024-04-20T17:21:42.634 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2765999999999999e-05 2024-04-20T17:21:42.634 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.634 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.634 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:42.634 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:50.686242+0000", 2024-04-20T17:21:42.634 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0593e-05 2024-04-20T17:21:42.634 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.634 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.634 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.634 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:50.693321+0000", 2024-04-20T17:21:42.634 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0070788379999999996 2024-04-20T17:21:42.634 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.634 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.635 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:21:42.635 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:50.695239+0000", 2024-04-20T17:21:42.635 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001918171 2024-04-20T17:21:42.635 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.635 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.635 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.635 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:51.493583+0000", 2024-04-20T17:21:42.635 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.79834359300000002 2024-04-20T17:21:42.635 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.635 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.635 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:21:42.635 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:51.496785+0000", 2024-04-20T17:21:42.635 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0032021380000000002 2024-04-20T17:21:42.635 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.635 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.635 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.636 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:53.274012+0000", 2024-04-20T17:21:42.636 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.777227224 2024-04-20T17:21:42.636 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.636 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.636 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:42.636 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:53.277485+0000", 2024-04-20T17:21:42.636 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003472485 2024-04-20T17:21:42.636 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.636 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.636 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:21:42.636 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:53.284302+0000", 2024-04-20T17:21:42.636 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0068175969999999999 2024-04-20T17:21:42.636 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.636 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.636 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:42.637 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:53.291173+0000", 2024-04-20T17:21:42.637 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0068707500000000001 2024-04-20T17:21:42.637 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.637 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.637 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:42.637 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:53.400486+0000", 2024-04-20T17:21:42.637 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.109313438 2024-04-20T17:21:42.637 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.637 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.637 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:42.637 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:53.402384+0000", 2024-04-20T17:21:42.637 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0018981479999999999 2024-04-20T17:21:42.637 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.637 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.637 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:42.637 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:53.409964+0000", 2024-04-20T17:21:42.637 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0075797089999999996 2024-04-20T17:21:42.638 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.638 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:42.638 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.638 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.638 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.638 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:268280 2.3 2:c0f96bdd:::benchmark_data_smithi171_78918_object107286:head [delete] snapc 0=[] ondisk+write+known_if_redirected e309)", 2024-04-20T17:21:42.638 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:17:55.131150+0000", 2024-04-20T17:21:42.638 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 227.37910143299999, 2024-04-20T17:21:42.638 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.634024288, 2024-04-20T17:21:42.638 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:42.638 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:42.638 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:42.638 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:42.638 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:42.638 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:42.639 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 268280 2024-04-20T17:21:42.639 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.639 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:42.639 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.639 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:42.639 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.131150+0000", 2024-04-20T17:21:42.639 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.639 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.639 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.639 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:42.639 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.131150+0000", 2024-04-20T17:21:42.639 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.639 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.639 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.639 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:42.639 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.131152+0000", 2024-04-20T17:21:42.640 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1179999999999999e-06 2024-04-20T17:21:42.640 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.640 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.640 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:42.640 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.131158+0000", 2024-04-20T17:21:42.640 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7150000000000001e-06 2024-04-20T17:21:42.640 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.640 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.640 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:42.640 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.131170+0000", 2024-04-20T17:21:42.640 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1433999999999999e-05 2024-04-20T17:21:42.640 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.640 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.640 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:42.640 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.131179+0000", 2024-04-20T17:21:42.640 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0420000000000008e-06 2024-04-20T17:21:42.641 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.641 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.641 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.641 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.185255+0000", 2024-04-20T17:21:42.641 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.054076235 2024-04-20T17:21:42.641 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.641 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.641 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:21:42.641 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.187152+0000", 2024-04-20T17:21:42.641 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001896539 2024-04-20T17:21:42.641 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.641 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.641 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.641 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.925874+0000", 2024-04-20T17:21:42.641 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.73872241400000005 2024-04-20T17:21:42.642 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.642 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.642 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:21:42.642 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.928102+0000", 2024-04-20T17:21:42.642 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0022278860000000001 2024-04-20T17:21:42.642 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.642 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.642 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.642 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:57.681606+0000", 2024-04-20T17:21:42.642 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.753504239 2024-04-20T17:21:42.642 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.642 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.642 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:42.642 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:57.685368+0000", 2024-04-20T17:21:42.642 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0037615859999999999 2024-04-20T17:21:42.643 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.643 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.643 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:21:42.643 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:57.693441+0000", 2024-04-20T17:21:42.643 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0080737340000000008 2024-04-20T17:21:42.643 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.643 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.643 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:42.643 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:57.734945+0000", 2024-04-20T17:21:42.643 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.041503962999999998 2024-04-20T17:21:42.643 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.643 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.643 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:42.643 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:57.755109+0000", 2024-04-20T17:21:42.643 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.020163945999999999 2024-04-20T17:21:42.644 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.644 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.644 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:42.644 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:57.757740+0000", 2024-04-20T17:21:42.644 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002630453 2024-04-20T17:21:42.644 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.644 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.644 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:42.644 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:57.765174+0000", 2024-04-20T17:21:42.644 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0074339840000000003 2024-04-20T17:21:42.644 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.644 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:42.644 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.644 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.644 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.644 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:272485 2.3 2:cf8b6f46:::benchmark_data_smithi171_78918_object111491:head [delete] snapc 0=[] ondisk+write+known_if_redirected e316)", 2024-04-20T17:21:42.645 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:18:13.931178+0000", 2024-04-20T17:21:42.645 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 208.579073247, 2024-04-20T17:21:42.645 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5463639260000002, 2024-04-20T17:21:42.645 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:42.645 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:42.645 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:42.645 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:42.645 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:42.645 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:42.645 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 272485 2024-04-20T17:21:42.645 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.645 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:42.645 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.645 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:42.645 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:13.931178+0000", 2024-04-20T17:21:42.646 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.646 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.646 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.646 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:42.646 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:13.931178+0000", 2024-04-20T17:21:42.646 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.646 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.646 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.646 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:42.646 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:13.931180+0000", 2024-04-20T17:21:42.646 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0200000000000001e-06 2024-04-20T17:21:42.646 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.646 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.647 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:42.647 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:13.931186+0000", 2024-04-20T17:21:42.647 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5209999999999999e-06 2024-04-20T17:21:42.647 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.647 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.647 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:42.647 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:13.931198+0000", 2024-04-20T17:21:42.647 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2126999999999999e-05 2024-04-20T17:21:42.647 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.647 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.647 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:42.647 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:13.931205+0000", 2024-04-20T17:21:42.647 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0010000000000004e-06 2024-04-20T17:21:42.647 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.647 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.648 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.648 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:14.024529+0000", 2024-04-20T17:21:42.648 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.093323992999999994 2024-04-20T17:21:42.648 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.648 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.648 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:21:42.648 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:14.026231+0000", 2024-04-20T17:21:42.648 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0017012640000000001 2024-04-20T17:21:42.648 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.648 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.648 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.648 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:14.688291+0000", 2024-04-20T17:21:42.648 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.66206054700000005 2024-04-20T17:21:42.648 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.648 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.648 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:21:42.648 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:14.690674+0000", 2024-04-20T17:21:42.649 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002383282 2024-04-20T17:21:42.649 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.649 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.649 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.649 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:16.448551+0000", 2024-04-20T17:21:42.649 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7578762220000002 2024-04-20T17:21:42.649 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.649 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.649 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:42.649 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:16.451664+0000", 2024-04-20T17:21:42.649 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003113476 2024-04-20T17:21:42.649 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.649 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.649 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:21:42.649 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:16.457746+0000", 2024-04-20T17:21:42.649 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0060814440000000001 2024-04-20T17:21:42.650 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.650 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.650 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:42.650 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:16.468555+0000", 2024-04-20T17:21:42.650 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010809440999999999 2024-04-20T17:21:42.650 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.650 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.650 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:42.650 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:16.469204+0000", 2024-04-20T17:21:42.650 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00064895199999999995 2024-04-20T17:21:42.650 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.650 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.650 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:42.650 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:16.470865+0000", 2024-04-20T17:21:42.651 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00166113 2024-04-20T17:21:42.651 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.651 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.651 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:42.651 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:16.477542+0000", 2024-04-20T17:21:42.651 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0066765059999999996 2024-04-20T17:21:42.651 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.651 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:42.651 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.651 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.651 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.651 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:273306 2.3 2:c7ad72a8:::benchmark_data_smithi171_78918_object112312:head [delete] snapc 0=[] ondisk+write+known_if_redirected e317)", 2024-04-20T17:21:42.651 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:18:17.740700+0000", 2024-04-20T17:21:42.651 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 204.76955111300001, 2024-04-20T17:21:42.651 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7316630169999998, 2024-04-20T17:21:42.652 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:21:42.652 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:21:42.652 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:21:42.652 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:21:42.652 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:21:42.652 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:21:42.652 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 273306 2024-04-20T17:21:42.652 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.652 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:21:42.652 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.652 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:21:42.652 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:17.740700+0000", 2024-04-20T17:21:42.652 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.652 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.652 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.653 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:21:42.653 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:17.740700+0000", 2024-04-20T17:21:42.653 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:21:42.653 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.653 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.653 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:21:42.653 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:17.740703+0000", 2024-04-20T17:21:42.653 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3699999999999999e-06 2024-04-20T17:21:42.653 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.653 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.653 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:21:42.653 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:17.740713+0000", 2024-04-20T17:21:42.653 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7049999999999994e-06 2024-04-20T17:21:42.653 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.653 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.654 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:21:42.654 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:17.740727+0000", 2024-04-20T17:21:42.654 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3828999999999999e-05 2024-04-20T17:21:42.654 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.654 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.654 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:21:42.654 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:17.740736+0000", 2024-04-20T17:21:42.654 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.3209999999999994e-06 2024-04-20T17:21:42.654 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.654 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.654 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.654 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:17.819955+0000", 2024-04-20T17:21:42.654 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.079219081999999996 2024-04-20T17:21:42.654 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.655 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.655 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:21:42.655 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:17.822169+0000", 2024-04-20T17:21:42.655 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0022142680000000001 2024-04-20T17:21:42.655 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.655 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.655 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.655 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:18.772171+0000", 2024-04-20T17:21:42.655 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.95000131600000004 2024-04-20T17:21:42.655 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.655 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.655 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:21:42.655 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:18.774199+0000", 2024-04-20T17:21:42.655 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0020284610000000001 2024-04-20T17:21:42.656 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.656 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.656 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:21:42.656 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:20.414977+0000", 2024-04-20T17:21:42.656 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6407782499999999 2024-04-20T17:21:42.656 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.656 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.656 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:21:42.656 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:20.417976+0000", 2024-04-20T17:21:42.656 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002998213 2024-04-20T17:21:42.656 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.656 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.656 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:21:42.656 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:20.423980+0000", 2024-04-20T17:21:42.656 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.006004634 2024-04-20T17:21:42.657 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.657 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.657 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:21:42.657 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:20.460896+0000", 2024-04-20T17:21:42.657 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.036915765000000003 2024-04-20T17:21:42.657 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.657 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.657 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:21:42.657 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:20.462179+0000", 2024-04-20T17:21:42.657 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0012827940000000001 2024-04-20T17:21:42.657 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.657 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.657 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:21:42.657 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:20.464524+0000", 2024-04-20T17:21:42.657 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0023448779999999999 2024-04-20T17:21:42.657 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:21:42.658 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:21:42.658 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:21:42.658 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:20.472363+0000", 2024-04-20T17:21:42.658 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0078391309999999992 2024-04-20T17:21:42.658 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.658 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:42.658 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.658 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:21:42.658 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:21:42.658 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:21:42.658 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:21:42.659 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:42 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:21:42.659 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:42 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:21:42.710 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:42.747 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:21:42.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:42.753 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:21:42.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:42 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:21:42.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:42 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:21:42.803 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 71 16 120230 120214 105.803 73.75 0.015645 0.00944828 2024-04-20T17:21:42.854 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:21:42.909 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:43.548 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:43 smithi005 ceph-mon[39562]: pgmap v1041: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-04-20T17:21:43.548 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:43 smithi005 ceph-mon[39562]: osdmap e384: 12 total, 11 up, 11 in 2024-04-20T17:21:43.549 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-20T17:21:43.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:43 smithi171 ceph-mon[44189]: pgmap v1041: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-04-20T17:21:43.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:43 smithi171 ceph-mon[44189]: osdmap e384: 12 total, 11 up, 11 in 2024-04-20T17:21:43.710 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:21:43.711 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:21:43.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:43 smithi106 ceph-mon[44423]: pgmap v1041: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-04-20T17:21:43.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:43 smithi106 ceph-mon[44423]: osdmap e384: 12 total, 11 up, 11 in 2024-04-20T17:21:43.803 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 72 16 121459 121443 105.4 76.8125 0.00487621 0.00948585 2024-04-20T17:21:43.976 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:21:43.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:43.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:21:44.019 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:21:44.081 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:21:44.118 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:44.293 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:44 smithi005 ceph-mon[39562]: from='client.25883 v1:172.21.15.5:0/1305441283' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:21:44.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:44 smithi171 ceph-mon[44189]: from='client.25883 v1:172.21.15.5:0/1305441283' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:21:44.588 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:21:44.594 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:21:44.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:44.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:21:44.625 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":1042,"stamp":"2024-04-20T17:21:42.282356+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7620330016,"num_objects":116270,"num_object_clones":0,"num_object_copies":232540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":116270,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":116449,"num_write_kb":7444090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4162,"num_bytes_recovered":274352256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117379,"ondisk_log_size":117379,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":17072372,"kb_used_data":15496084,"kb_used_omap":218,"kb_used_meta":1575973,"kb_avail":1014124300,"statfs":{"total":1055945392128,"available":1038463283200,"internally_reserved":0,"allocated":15867990016,"data_stored":15862569002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":223359,"internal_metadata":1613797249},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,4,8,7,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":648019968,"num_objects":9888,"num_object_clones":0,"num_object_copies":19776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9888,"num_write_kb":632832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.022549"},"pg_stats":[{"pgid":"3.f","version":"383'8047","reported_seq":12338,"reported_epoch":383,"state":"active+clean+scrubbing+deep","last_fresh":"2024-04-20T17:21:37.597245+0000","last_change":"2024-04-20T17:21:29.186805+0000","last_active":"2024-04-20T17:21:37.597245+0000","last_peered":"2024-04-20T17:21:37.597245+0000","last_clean":"2024-04-20T17:21:37.597245+0000","last_became_active":"2024-04-20T17:20:55.357479+0000","last_became_peered":"2024-04-20T17:20:55.357479+0000","last_unstale":"2024-04-20T17:21:37.597245+0000","last_undegraded":"2024-04-20T17:21:37.597245+0000","last_fullsized":"2024-04-20T17:21:37.597245+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":2111,"log_size":8047,"log_dups_size":0,"ondisk_log_size":8047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 8s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":463536128,"num_objects":7073,"num_object_clones":0,"num_object_copies":14146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7073,"num_write_kb":452672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2071,"num_bytes_recovered":135725056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"383'7133","reported_seq":7195,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-04-20T17:21:37.599654+0000","last_change":"2024-04-20T17:20:23.037437+0000","last_active":"2024-04-20T17:21:37.599654+0000","last_peered":"2024-04-20T17:21:37.599654+0000","last_clean":"2024-04-20T17:21:37.599654+0000","last_became_active":"2024-04-20T17:20:23.037195+0000","last_became_peered":"2024-04-20T17:20:23.037195+0000","last_unstale":"2024-04-20T17:21:37.599654+0000","last_undegraded":"2024-04-20T17:21:37.599654+0000","last_fullsized":"2024-04-20T17:21:37.599654+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":0,"log_size":7133,"log_dups_size":0,"ondisk_log_size":7133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:48.986247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":467468288,"num_objects":7133,"num_object_clones":0,"num_object_copies":14266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7133,"num_write_kb":456512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"383'7342","reported_seq":11651,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-04-20T17:21:40.577257+0000","last_change":"2024-04-20T17:21:31.018241+0000","last_active":"2024-04-20T17:21:40.577257+0000","last_peered":"2024-04-20T17:21:40.577257+0000","last_clean":"2024-04-20T17:21:40.577257+0000","last_became_active":"2024-04-20T17:20:55.283498+0000","last_became_peered":"2024-04-20T17:20:55.283498+0000","last_unstale":"2024-04-20T17:21:40.577257+0000","last_undegraded":"2024-04-20T17:21:40.577257+0000","last_fullsized":"2024-04-20T17:21:40.577257+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"379'6609","last_scrub_stamp":"2024-04-20T17:21:31.018148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:21:31.018148+0000","objects_scrubbed":6588,"log_size":7342,"log_dups_size":0,"ondisk_log_size":7342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:22:54.932611+0000","scrub_duration":849,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481165312,"num_objects":7342,"num_object_clones":0,"num_object_copies":14684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7342,"num_write_kb":469888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2083,"num_bytes_recovered":136511488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"383'7244","reported_seq":7306,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-04-20T17:21:40.555670+0000","last_change":"2024-04-20T17:20:23.038299+0000","last_active":"2024-04-20T17:21:40.555670+0000","last_peered":"2024-04-20T17:21:40.555670+0000","last_clean":"2024-04-20T17:21:40.555670+0000","last_became_active":"2024-04-20T17:20:23.038035+0000","last_became_peered":"2024-04-20T17:20:23.038035+0000","last_unstale":"2024-04-20T17:21:40.555670+0000","last_undegraded":"2024-04-20T17:21:40.555670+0000","last_fullsized":"2024-04-20T17:21:40.555670+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":0,"log_size":7244,"log_dups_size":0,"ondisk_log_size":7244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:42.290085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474742784,"num_objects":7244,"num_object_clones":0,"num_object_copies":14488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7244,"num_write_kb":463616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"383'7211","reported_seq":7273,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-04-20T17:21:40.942554+0000","last_change":"2024-04-20T17:20:23.038256+0000","last_active":"2024-04-20T17:21:40.942554+0000","last_peered":"2024-04-20T17:21:40.942554+0000","last_clean":"2024-04-20T17:21:40.942554+0000","last_became_active":"2024-04-20T17:20:23.037709+0000","last_became_peered":"2024-04-20T17:20:23.037709+0000","last_unstale":"2024-04-20T17:21:40.942554+0000","last_undegraded":"2024-04-20T17:21:40.942554+0000","last_fullsized":"2024-04-20T17:21:40.942554+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":0,"log_size":7211,"log_dups_size":0,"ondisk_log_size":7211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:41.131412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472580096,"num_objects":7211,"num_object_clones":0,"num_object_copies":14422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7211,"num_write_kb":461504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"383'7177","reported_seq":7239,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-04-20T17:21:37.372533+0000","last_change":"2024-04-20T17:20:23.037647+0000","last_active":"2024-04-20T17:21:37.372533+0000","last_peered":"2024-04-20T17:21:37.372533+0000","last_clean":"2024-04-20T17:21:37.372533+0000","last_became_active":"2024-04-20T17:20:23.037291+0000","last_became_peered":"2024-04-20T17:20:23.037291+0000","last_unstale":"2024-04-20T17:21:37.372533+0000","last_undegraded":"2024-04-20T17:21:37.372533+0000","last_fullsized":"2024-04-20T17:21:37.372533+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":0,"log_size":7177,"log_dups_size":0,"ondisk_log_size":7177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470351872,"num_objects":7177,"num_object_clones":0,"num_object_copies":14354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7177,"num_write_kb":459328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"383'7304","reported_seq":7367,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-04-20T17:21:38.577920+0000","last_change":"2024-04-20T17:20:23.036662+0000","last_active":"2024-04-20T17:21:38.577920+0000","last_peered":"2024-04-20T17:21:38.577920+0000","last_clean":"2024-04-20T17:21:38.577920+0000","last_became_active":"2024-04-20T17:20:23.036396+0000","last_became_peered":"2024-04-20T17:20:23.036396+0000","last_unstale":"2024-04-20T17:21:38.577920+0000","last_undegraded":"2024-04-20T17:21:38.577920+0000","last_fullsized":"2024-04-20T17:21:38.577920+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":0,"log_size":7304,"log_dups_size":0,"ondisk_log_size":7304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:50.594173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":478674944,"num_objects":7304,"num_object_clones":0,"num_object_copies":14608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7304,"num_write_kb":467456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"383'7218","reported_seq":7278,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-04-20T17:21:37.588175+0000","last_change":"2024-04-20T17:20:23.038640+0000","last_active":"2024-04-20T17:21:37.588175+0000","last_peered":"2024-04-20T17:21:37.588175+0000","last_clean":"2024-04-20T17:21:37.588175+0000","last_became_active":"2024-04-20T17:20:23.038395+0000","last_became_peered":"2024-04-20T17:20:23.038395+0000","last_unstale":"2024-04-20T17:21:37.588175+0000","last_undegraded":"2024-04-20T17:21:37.588175+0000","last_fullsized":"2024-04-20T17:21:37.588175+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":0,"log_size":7218,"log_dups_size":0,"ondisk_log_size":7218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:43.786722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473038848,"num_objects":7218,"num_object_clones":0,"num_object_copies":14436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7218,"num_write_kb":461952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"383'7560","reported_seq":7622,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-04-20T17:21:40.958722+0000","last_change":"2024-04-20T17:20:23.036646+0000","last_active":"2024-04-20T17:21:40.958722+0000","last_peered":"2024-04-20T17:21:40.958722+0000","last_clean":"2024-04-20T17:21:40.958722+0000","last_became_active":"2024-04-20T17:20:23.036358+0000","last_became_peered":"2024-04-20T17:20:23.036358+0000","last_unstale":"2024-04-20T17:21:40.958722+0000","last_undegraded":"2024-04-20T17:21:40.958722+0000","last_fullsized":"2024-04-20T17:21:40.958722+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":0,"log_size":7560,"log_dups_size":0,"ondisk_log_size":7560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:45.137835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":495452160,"num_objects":7560,"num_object_clones":0,"num_object_copies":15120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7560,"num_write_kb":483840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"383'7188","reported_seq":7250,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-04-20T17:21:39.037641+0000","last_change":"2024-04-20T17:20:23.041005+0000","last_active":"2024-04-20T17:21:39.037641+0000","last_peered":"2024-04-20T17:21:39.037641+0000","last_clean":"2024-04-20T17:21:39.037641+0000","last_became_active":"2024-04-20T17:20:23.040736+0000","last_became_peered":"2024-04-20T17:20:23.040736+0000","last_unstale":"2024-04-20T17:21:39.037641+0000","last_undegraded":"2024-04-20T17:21:39.037641+0000","last_fullsized":"2024-04-20T17:21:39.037641+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":0,"log_size":7188,"log_dups_size":0,"ondisk_log_size":7188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:48.465285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471072768,"num_objects":7188,"num_object_clones":0,"num_object_copies":14376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7188,"num_write_kb":460032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"383'7374","reported_seq":7507,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-04-20T17:21:40.587152+0000","last_change":"2024-04-20T17:21:26.191112+0000","last_active":"2024-04-20T17:21:40.587152+0000","last_peered":"2024-04-20T17:21:40.587152+0000","last_clean":"2024-04-20T17:21:40.587152+0000","last_became_active":"2024-04-20T17:20:23.039242+0000","last_became_peered":"2024-04-20T17:20:23.039242+0000","last_unstale":"2024-04-20T17:21:40.587152+0000","last_undegraded":"2024-04-20T17:21:40.587152+0000","last_fullsized":"2024-04-20T17:21:40.587152+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"379'6117","last_scrub_stamp":"2024-04-20T17:21:26.190958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:21:26.190958+0000","objects_scrubbed":6072,"log_size":7374,"log_dups_size":0,"ondisk_log_size":7374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:22:28.459099+0000","scrub_duration":1057,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483262464,"num_objects":7374,"num_object_clones":0,"num_object_copies":14748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7374,"num_write_kb":471936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"383'7465","reported_seq":7528,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-04-20T17:21:41.973345+0000","last_change":"2024-04-20T17:20:23.038535+0000","last_active":"2024-04-20T17:21:41.973345+0000","last_peered":"2024-04-20T17:21:41.973345+0000","last_clean":"2024-04-20T17:21:41.973345+0000","last_became_active":"2024-04-20T17:20:23.038263+0000","last_became_peered":"2024-04-20T17:20:23.038263+0000","last_unstale":"2024-04-20T17:21:41.973345+0000","last_undegraded":"2024-04-20T17:21:41.973345+0000","last_fullsized":"2024-04-20T17:21:41.973345+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":0,"log_size":7465,"log_dups_size":0,"ondisk_log_size":7465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:47.816939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489226240,"num_objects":7465,"num_object_clones":0,"num_object_copies":14930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7465,"num_write_kb":477760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"383'7290","reported_seq":7352,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-04-20T17:21:40.532796+0000","last_change":"2024-04-20T17:20:23.039442+0000","last_active":"2024-04-20T17:21:40.532796+0000","last_peered":"2024-04-20T17:21:40.532796+0000","last_clean":"2024-04-20T17:21:40.532796+0000","last_became_active":"2024-04-20T17:20:23.039051+0000","last_became_peered":"2024-04-20T17:20:23.039051+0000","last_unstale":"2024-04-20T17:21:40.532796+0000","last_undegraded":"2024-04-20T17:21:40.532796+0000","last_fullsized":"2024-04-20T17:21:40.532796+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":0,"log_size":7290,"log_dups_size":0,"ondisk_log_size":7290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:48.296438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477757440,"num_objects":7290,"num_object_clones":0,"num_object_copies":14580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7290,"num_write_kb":466560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"383'7339","reported_seq":7474,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-04-20T17:21:39.037470+0000","last_change":"2024-04-20T17:21:30.216370+0000","last_active":"2024-04-20T17:21:39.037470+0000","last_peered":"2024-04-20T17:21:39.037470+0000","last_clean":"2024-04-20T17:21:39.037470+0000","last_became_active":"2024-04-20T17:20:23.036876+0000","last_became_peered":"2024-04-20T17:20:23.036876+0000","last_unstale":"2024-04-20T17:21:39.037470+0000","last_undegraded":"2024-04-20T17:21:39.037470+0000","last_fullsized":"2024-04-20T17:21:39.037470+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"379'6596","last_scrub_stamp":"2024-04-20T17:21:30.216031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:21:30.216031+0000","objects_scrubbed":6549,"log_size":7339,"log_dups_size":0,"ondisk_log_size":7339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:22:31.848948+0000","scrub_duration":1023,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":480968704,"num_objects":7339,"num_object_clones":0,"num_object_copies":14678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7339,"num_write_kb":469696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"383'7056","reported_seq":7118,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-04-20T17:21:37.574646+0000","last_change":"2024-04-20T17:20:23.171337+0000","last_active":"2024-04-20T17:21:37.574646+0000","last_peered":"2024-04-20T17:21:37.574646+0000","last_clean":"2024-04-20T17:21:37.574646+0000","last_became_active":"2024-04-20T17:20:23.171030+0000","last_became_peered":"2024-04-20T17:20:23.171030+0000","last_unstale":"2024-04-20T17:21:37.574646+0000","last_undegraded":"2024-04-20T17:21:37.574646+0000","last_fullsized":"2024-04-20T17:21:37.574646+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":0,"log_size":7056,"log_dups_size":0,"ondisk_log_size":7056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:46.281016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462422016,"num_objects":7056,"num_object_clones":0,"num_object_copies":14112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7056,"num_write_kb":451584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"383'7294","reported_seq":7355,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-04-20T17:21:40.954496+0000","last_change":"2024-04-20T17:20:23.171325+0000","last_active":"2024-04-20T17:21:40.954496+0000","last_peered":"2024-04-20T17:21:40.954496+0000","last_clean":"2024-04-20T17:21:40.954496+0000","last_became_active":"2024-04-20T17:20:23.171078+0000","last_became_peered":"2024-04-20T17:20:23.171078+0000","last_unstale":"2024-04-20T17:21:40.954496+0000","last_undegraded":"2024-04-20T17:21:40.954496+0000","last_fullsized":"2024-04-20T17:21:40.954496+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:20:22.020114+0000","objects_scrubbed":0,"log_size":7294,"log_dups_size":0,"ondisk_log_size":7294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:21:45.457170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":478019584,"num_objects":7294,"num_object_clones":0,"num_object_copies":14588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7294,"num_write_kb":466816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":368,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-04-20T17:21:36.040647+0000","last_change":"2024-04-20T17:21:17.342231+0000","last_active":"2024-04-20T17:21:36.040647+0000","last_peered":"2024-04-20T17:21:36.040647+0000","last_clean":"2024-04-20T17:21:36.040647+0000","last_became_active":"2024-04-20T17:20:55.304416+0000","last_became_peered":"2024-04-20T17:20:55.304416+0000","last_unstale":"2024-04-20T17:21:36.040647+0000","last_undegraded":"2024-04-20T17:21:36.040647+0000","last_fullsized":"2024-04-20T17:21:36.040647+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:21:17.342091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:21:17.342091+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:22:19.858669+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":7619739648,"num_objects":116268,"num_object_clones":0,"num_object_copies":232536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":116268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116268,"num_write_kb":7441152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4154,"num_bytes_recovered":272236544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15505817600,"data_stored":15505817600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117242,"ondisk_log_size":117242,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991423,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3359964,"kb_used_data":3194132,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":90385188,"statfs":{"total":95995035648,"available":92554432512,"internally_reserved":0,"allocated":3270791168,"data_stored":3270295426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":169802180},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253062,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":216796,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93528356,"statfs":{"total":95995035648,"available":95773036544,"internally_reserved":0,"allocated":995328,"data_stored":503138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":220985792},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612811,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1578516,"kb_used_data":1394636,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92166636,"statfs":{"total":95995035648,"available":94378635264,"internally_reserved":0,"allocated":1428107264,"data_stored":1427615094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":188283328},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809040,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1057428,"kb_used_data":922124,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":92687724,"statfs":{"total":95995035648,"available":94912229376,"internally_reserved":0,"allocated":944254976,"data_stored":943762806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,1,2,3,4,5,6,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430425,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":567772,"kb_used_data":460244,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93177380,"statfs":{"total":95995035648,"available":95413637120,"internally_reserved":0,"allocated":471289856,"data_stored":470794134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[0,1,2,3,4,5,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575092,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1571028,"kb_used_data":1431564,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92174124,"statfs":{"total":95995035648,"available":94386302976,"internally_reserved":0,"allocated":1465921536,"data_stored":1465429366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738616,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2033620,"kb_used_data":1826124,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":91711532,"statfs":{"total":95995035648,"available":93912608768,"internally_reserved":0,"allocated":1869950976,"data_stored":1869458806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":212466117},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":283540,"kb_used_data":235148,"kb_used_omap":202,"kb_used_meta":48181,"kb_avail":93461612,"statfs":{"total":95995035648,"available":95704690688,"internally_reserved":0,"allocated":240791552,"data_stored":240299382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":49337815},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182633,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2467412,"kb_used_data":2332108,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":91277740,"statfs":{"total":95995035648,"available":93468405760,"internally_reserved":0,"allocated":2388078592,"data_stored":2387586422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":138541511},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986404,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1981844,"kb_used_data":1888204,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":91763308,"statfs":{"total":95995035648,"available":93965627392,"internally_reserved":0,"allocated":1933520896,"data_stored":1933028726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790175,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1954452,"kb_used_data":1810828,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":91790700,"statfs":{"total":95995035648,"available":93993676800,"internally_reserved":0,"allocated":1854287872,"data_stored":1853795702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1886650368,"data_stored":1886650368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1470693376,"data_stored":1470693376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2408054784,"data_stored":2408054784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1933443072,"data_stored":1933443072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1439236096,"data_stored":1439236096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":470417408,"data_stored":470417408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":229638144,"data_stored":229638144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":947388416,"data_stored":947388416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1436024832,"data_stored":1436024832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3284271104,"data_stored":3284271104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:21:44.630 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-20T17:21:44.631 INFO:tasks.thrashosds.thrasher:Waiting for trim 2024-04-20T17:21:44.702 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:21:44.729 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:44 smithi106 ceph-mon[44423]: from='client.25883 v1:172.21.15.5:0/1305441283' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:21:44.732 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:44.754 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:21:44.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:44.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:21:44.803 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 73 16 122615 122599 104.946 72.25 0.0046021 0.00952609 2024-04-20T17:21:44.861 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:21:44.889 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:44.913 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:21:44.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:44.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:21:45.019 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:21:45.188 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:45.211 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:21:45.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:45.220 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:21:45.321 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:21:45.360 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:45.387 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:21:45.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:45.396 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:21:45.497 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:21:45.524 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:45.543 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:45 smithi005 ceph-mon[39562]: pgmap v1043: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.3 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-20T17:21:45.544 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:21:45.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:45.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:21:45.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:45 smithi171 ceph-mon[44189]: pgmap v1043: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.3 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-20T17:21:45.647 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:21:45.687 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:45.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:45 smithi106 ceph-mon[44423]: pgmap v1043: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.3 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-20T17:21:45.812 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 74 16 123765 123749 104.487 71.875 0.0069887 0.00956892 2024-04-20T17:21:45.895 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:21:45.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:45.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:21:46.002 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:21:46.034 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:46.058 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:21:46.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:46.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:21:46.164 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:21:46.198 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:46.261 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:21:46.266 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:21:46.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:46.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:21:46.372 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:21:46.408 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:46.432 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:21:46.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:46.438 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:21:46.539 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:21:46.569 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:46.590 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:21:46.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:46.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:21:46.694 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:21:46.739 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:46.812 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 75 16 124922 124906 104.058 72.3125 0.00423113 0.00960695 2024-04-20T17:21:46.992 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:21:46.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:46.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:21:47.096 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:21:47.125 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:47.147 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:21:47.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:47.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:21:47.251 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:21:47.282 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:47 smithi005 ceph-mon[39562]: osdmap e385: 12 total, 11 up, 11 in 2024-04-20T17:21:47.282 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:47 smithi005 ceph-mon[39562]: pgmap v1045: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-04-20T17:21:47.284 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:47 smithi171 ceph-mon[44189]: osdmap e385: 12 total, 11 up, 11 in 2024-04-20T17:21:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:47 smithi171 ceph-mon[44189]: pgmap v1045: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-04-20T17:21:47.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:47 smithi106 ceph-mon[44423]: osdmap e385: 12 total, 11 up, 11 in 2024-04-20T17:21:47.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:47 smithi106 ceph-mon[44423]: pgmap v1045: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-04-20T17:21:47.816 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 76 16 126113 126097 103.663 74.4375 0.00914654 0.00964494 2024-04-20T17:21:48.816 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 77 16 127252 127236 103.242 71.1875 0.00792036 0.00968463 2024-04-20T17:21:49.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:49 smithi005 ceph-mon[39562]: pgmap v1046: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.6 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-20T17:21:49.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:49 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:21:49.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:49 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:21:49.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:49 smithi106 ceph-mon[44423]: pgmap v1046: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.6 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-20T17:21:49.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:49 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:21:49.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:49 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:21:49.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:49 smithi171 ceph-mon[44189]: pgmap v1046: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.6 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-20T17:21:49.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:49 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:21:49.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:49 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:21:49.816 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 78 16 128360 128344 102.806 69.25 0.00451645 0.00972295 2024-04-20T17:21:50.328 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:50 smithi005 ceph-mon[39562]: osdmap e386: 12 total, 11 up, 11 in 2024-04-20T17:21:50.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:50 smithi106 ceph-mon[44423]: osdmap e386: 12 total, 11 up, 11 in 2024-04-20T17:21:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:50 smithi171 ceph-mon[44189]: osdmap e386: 12 total, 11 up, 11 in 2024-04-20T17:21:50.816 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 79 16 129484 129468 102.394 70.25 0.00549904 0.00976421 2024-04-20T17:21:50.858 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:21:50.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:50.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:21:50.908 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:21:50.936 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:21:50.962 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:21:50.964 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:21:50.992 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:50.993 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:21:51.012 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:21:51.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:51.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:21:51.026 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:21:51.049 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:21:51.073 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:21:51.105 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:21:51.116 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:21:51.132 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:21:51.145 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:51.158 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:21:51.166 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:21:51.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:51.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:21:51.186 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:21:51.187 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:21:51.187 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:21:51.187 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:21:51.187 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:21:51.187 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:21:51.187 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:21:51.187 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:21:51.187 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:21:51.187 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:21:51.187 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:21:51.187 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:21:51.187 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:21:51.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:51 smithi106 ceph-mon[44423]: pgmap v1048: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.6 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-20T17:21:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:51 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4109672518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:21:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:51 smithi106 ceph-mon[44423]: osdmap e387: 12 total, 11 up, 11 in 2024-04-20T17:21:51.270 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:21:51.315 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:51.343 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:21:51.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:51.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:21:51.456 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:21:51.469 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:51 smithi005 ceph-mon[39562]: pgmap v1048: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.6 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-20T17:21:51.469 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:51 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4109672518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:21:51.469 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:51.469 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:51 smithi005 ceph-mon[39562]: osdmap e387: 12 total, 11 up, 11 in 2024-04-20T17:21:51.506 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:51.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:51 smithi171 ceph-mon[44189]: pgmap v1048: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.6 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-20T17:21:51.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:51 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4109672518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:21:51.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:51.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:51 smithi171 ceph-mon[44189]: osdmap e387: 12 total, 11 up, 11 in 2024-04-20T17:21:51.816 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:21:51.817269 min lat: 0.000946083 max lat: 16.1837 avg lat: 0.00980269 2024-04-20T17:21:51.816 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:21:51.816 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 80 16 130604 130588 101.989 70 0.00615159 0.00980269 2024-04-20T17:21:51.866 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:21:51.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:51.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:21:51.972 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:21:52.002 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:52.028 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:21:52.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:52.033 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:21:52.134 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:21:52.180 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:52.265 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:21:52.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:52.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:21:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.382 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:21:52.409 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:52.428 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:21:52.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:52.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:21:52.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:21:52.533 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:21:52.564 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:52.589 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:21:52.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:52.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:21:52.626 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:21:52.627 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:21:52.692 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:21:52.728 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:52.754 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:21:52.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:52.760 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:21:52.817 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 81 16 131644 131628 101.532 65 0.00888198 0.00984392 2024-04-20T17:21:52.860 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:21:52.906 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:52.940 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:21:52.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:52.947 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:21:53.048 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:21:53.097 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:53.125 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:21:53.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:53.130 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:21:53.218 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:21:53.231 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:21:53.245 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:53 smithi005 ceph-mon[39562]: pgmap v1050: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-04-20T17:21:53.245 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:53 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1106220463' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:21:53.245 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:53 smithi005 ceph-mon[39562]: from='client.25895 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:21:53.272 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:53.286 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:21:53.286 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:21:53.302 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:21:53.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:53.308 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:21:53.409 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:21:53.441 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:53.458 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:21:53.460 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:53 smithi106 ceph-mon[44423]: pgmap v1050: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-04-20T17:21:53.460 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:53 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1106220463' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:21:53.460 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:53 smithi106 ceph-mon[44423]: from='client.25895 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:21:53.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:53.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:21:53.562 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:21:53.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:53 smithi171 ceph-mon[44189]: pgmap v1050: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-04-20T17:21:53.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:53 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1106220463' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:21:53.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:53 smithi171 ceph-mon[44189]: from='client.25895 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:21:53.599 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:53.626 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:21:53.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:53.631 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:21:53.732 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:21:53.800 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:53.816 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 82 16 132859 132843 101.22 75.9375 0.0138865 0.00987819 2024-04-20T17:21:53.838 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:21:53.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:53.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:21:53.956 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:21:53.988 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:54.019 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:21:54.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:54.026 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:21:54.127 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:21:54.180 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:54.287 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:21:54.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:54.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:21:54.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:54 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:21:54.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:54 smithi171 ceph-mon[44189]: from='client.25895 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:21:54.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:54 smithi171 ceph-mon[44189]: osdmap e388: 12 total, 11 up, 11 in 2024-04-20T17:21:54.398 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:21:54.437 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:54.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:54 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:21:54.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:54 smithi005 ceph-mon[39562]: from='client.25895 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:21:54.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:54 smithi005 ceph-mon[39562]: osdmap e388: 12 total, 11 up, 11 in 2024-04-20T17:21:54.465 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:21:54.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:54.470 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:21:54.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:54 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:21:54.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:54 smithi106 ceph-mon[44423]: from='client.25895 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:21:54.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:54 smithi106 ceph-mon[44423]: osdmap e388: 12 total, 11 up, 11 in 2024-04-20T17:21:54.571 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:21:54.605 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:54.628 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:21:54.635 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:21:54.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:54.638 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:21:54.739 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:21:54.767 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:54.787 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:21:54.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:54.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:21:54.816 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 83 16 134207 134191 101.016 84.25 0.0135859 0.00989797 2024-04-20T17:21:54.891 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:21:54.927 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:54.957 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:21:54.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:54.961 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:21:55.062 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:21:55.096 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:55.119 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:21:55.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:55.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:21:55.223 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:21:55.273 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:55.296 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:21:55.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:55.300 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:21:55.401 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:21:55.435 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:55.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:55 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:21:55.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:55 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:21:55.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:55 smithi106 ceph-mon[44423]: pgmap v1052: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-04-20T17:21:55.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:55 smithi106 ceph-mon[44423]: osdmap e389: 12 total, 11 up, 11 in 2024-04-20T17:21:55.578 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:21:55.579 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:55 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:21:55.579 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:55 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:21:55.579 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:55 smithi005 ceph-mon[39562]: pgmap v1052: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-04-20T17:21:55.579 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:55 smithi005 ceph-mon[39562]: osdmap e389: 12 total, 11 up, 11 in 2024-04-20T17:21:55.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:55.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:21:55.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:55 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:21:55.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:55 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:21:55.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:55 smithi171 ceph-mon[44189]: pgmap v1052: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-04-20T17:21:55.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:55 smithi171 ceph-mon[44189]: osdmap e389: 12 total, 11 up, 11 in 2024-04-20T17:21:55.620 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-20T17:21:55.681 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:21:55.710 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:55.733 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:21:55.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:55.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:21:55.769 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:21:55.816 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 84 16 135558 135542 100.818 84.4375 0.0156188 0.0099171 2024-04-20T17:21:55.838 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:21:55.885 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:56.058 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:21:56.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:56.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:21:56.168 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:21:56.224 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:56.318 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:21:56.366 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:21:56.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:56.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:21:56.472 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:21:56.504 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:56.530 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:21:56.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:56.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:21:56.634 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:21:56.682 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:56.721 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:21:56.735 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:21:56.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:56.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:21:56.816 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 85 16 136825 136809 100.564 79.1875 0.0116286 0.00994241 2024-04-20T17:21:56.838 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:21:56.881 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:57.322 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:21:57.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:57 smithi171 ceph-mon[44189]: pgmap v1054: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-04-20T17:21:57.449 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:21:57.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:57 smithi005 ceph-mon[39562]: pgmap v1054: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-04-20T17:21:57.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:57.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:21:57.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:57 smithi106 ceph-mon[44423]: pgmap v1054: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-04-20T17:21:57.558 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:21:57.613 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:57.816 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 86 16 138150 138134 100.357 82.8125 0.00499761 0.00996287 2024-04-20T17:21:57.824 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:21:57.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:57.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:21:57.933 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:21:57.959 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:57.984 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:21:57.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:57.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:21:58.091 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:21:58.120 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:58.143 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:21:58.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:58.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:21:58.248 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:21:58.297 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:58.817 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 87 16 139469 139453 100.151 82.4375 0.00694367 0.00998364 2024-04-20T17:21:58.859 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:21:58.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:58.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:21:58.966 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:21:58.998 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:59.022 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:21:59.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:59.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:21:59.069 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:21:59.109 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:21:59.125 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:21:59.150 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:21:59.152 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:59.177 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:21:59.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:59.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:21:59.186 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:21:59.219 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:21:59.248 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:21:59.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:59 smithi106 ceph-mon[44423]: pgmap v1055: 17 pgs: 17 active+clean; 8.3 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-20T17:21:59.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:21:59 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2273251881' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:21:59.276 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:21:59.281 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:21:59.309 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:21:59.313 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:59.335 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:21:59.352 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:21:59.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:59.357 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:21:59.363 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:21:59.407 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:21:59.407 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:21:59.407 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:21:59.407 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:21:59.407 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:21:59.407 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:21:59.407 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:21:59.408 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:21:59.408 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:21:59.408 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:21:59.408 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:21:59.408 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:21:59.408 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:21:59.457 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:21:59.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:59 smithi005 ceph-mon[39562]: pgmap v1055: 17 pgs: 17 active+clean; 8.3 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-20T17:21:59.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:21:59 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2273251881' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:21:59.485 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:59.509 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:21:59.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:59.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:21:59.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:59 smithi171 ceph-mon[44189]: pgmap v1055: 17 pgs: 17 active+clean; 8.3 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-20T17:21:59.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:21:59 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2273251881' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:21:59.613 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:21:59.646 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:59.665 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:21:59.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:59.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:21:59.769 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:21:59.805 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:21:59.817 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 88 16 140740 140724 99.9158 79.4375 0.00879731 0.0100065 2024-04-20T17:21:59.831 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:21:59.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:21:59.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:21:59.940 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:21:59.985 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:00.016 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:22:00.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:00.019 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:22:00.120 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:22:00.170 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:00.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.393 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:22:00.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:00.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:22:00.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.424 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.424 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.424 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.424 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.424 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.424 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.424 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.499 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:22:00.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:00.565 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:00.817 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 89 16 142006 141990 99.6822 79.125 0.0116033 0.0100303 2024-04-20T17:22:01.137 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:22:01.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:01.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:22:01.242 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:22:01.245 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:22:01.245 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:22:01.245 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:22:01.245 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:22:01.246 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:22:01.282 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:01.313 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:22:01.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:01.315 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:22:01.416 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:22:01.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:01.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:01 smithi005 ceph-mon[39562]: pgmap v1056: 17 pgs: 17 active+clean; 8.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-20T17:22:01.462 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:01.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:01 smithi106 ceph-mon[44423]: pgmap v1056: 17 pgs: 17 active+clean; 8.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-20T17:22:01.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:01.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:01 smithi171 ceph-mon[44189]: pgmap v1056: 17 pgs: 17 active+clean; 8.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-20T17:22:01.817 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Total time run: 90.0088 2024-04-20T17:22:01.817 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Total writes made: 143212 2024-04-20T17:22:01.817 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Write size: 65536 2024-04-20T17:22:01.817 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Object size: 65536 2024-04-20T17:22:01.817 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Bandwidth (MB/sec): 99.443 2024-04-20T17:22:01.817 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Stddev Bandwidth: 72.2 2024-04-20T17:22:01.817 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Max bandwidth (MB/sec): 366.062 2024-04-20T17:22:01.817 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Min bandwidth (MB/sec): 0 2024-04-20T17:22:01.817 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Average IOPS: 1591 2024-04-20T17:22:01.818 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Stddev IOPS: 1155.21 2024-04-20T17:22:01.818 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Max IOPS: 5857 2024-04-20T17:22:01.818 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Min IOPS: 0 2024-04-20T17:22:01.818 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Average Latency(s): 0.0100551 2024-04-20T17:22:01.818 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Stddev Latency(s): 0.170646 2024-04-20T17:22:01.818 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Max latency(s): 16.1837 2024-04-20T17:22:01.818 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Min latency(s): 0.000946083 2024-04-20T17:22:01.824 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Cleaning up (deleting benchmark objects) 2024-04-20T17:22:02.040 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:22:02.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:02.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:22:02.078 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:22:02.079 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:22:02.148 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:22:02.190 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:02.227 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:22:02.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:02.234 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:22:02.236 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:22:02.335 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:22:02.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:02 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3469789134' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:22:02.381 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:02.404 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:22:02.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:02.411 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:22:02.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:02 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3469789134' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:22:02.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:02 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3469789134' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:22:02.512 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:22:02.555 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:02.597 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:22:02.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:02.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:22:02.627 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-20T17:22:02.712 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:22:02.739 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:02.760 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:22:02.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:02.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:22:02.867 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:22:02.950 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:02.990 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:22:03.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:03.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:22:03.113 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:22:03.159 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:03.189 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:22:03.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:03.191 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:22:03.293 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:22:03.342 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:03.372 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:22:03.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:03.380 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:22:03.480 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:22:03.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:03 smithi106 ceph-mon[44423]: pgmap v1057: 17 pgs: 17 active+clean; 8.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-20T17:22:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:03 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:22:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:03 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3469789134' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:22:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:03 smithi106 ceph-mon[44423]: osdmap e390: 12 total, 11 up, 11 in 2024-04-20T17:22:03.513 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:03 smithi171 ceph-mon[44189]: pgmap v1057: 17 pgs: 17 active+clean; 8.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-20T17:22:03.513 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:03 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:22:03.513 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:03 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3469789134' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:22:03.513 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:03 smithi171 ceph-mon[44189]: osdmap e390: 12 total, 11 up, 11 in 2024-04-20T17:22:03.515 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:03.545 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:22:03.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:03.551 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:22:03.552 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:22:03.652 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:22:03.684 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:03.708 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:22:03.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:03.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:22:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:03 smithi005 ceph-mon[39562]: pgmap v1057: 17 pgs: 17 active+clean; 8.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-20T17:22:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:03 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:22:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:03 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3469789134' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:22:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:03 smithi005 ceph-mon[39562]: osdmap e390: 12 total, 11 up, 11 in 2024-04-20T17:22:03.811 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:22:03.839 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:03.865 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:22:03.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:03.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:22:03.969 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:22:04.020 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:04.052 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:22:04.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:04.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:22:04.080 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:22:04.156 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:22:04.183 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:04.204 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:22:04.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:04.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:22:04.308 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:22:04.349 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:04.391 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:22:04.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:04.394 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:22:04.495 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:22:04.539 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:04.573 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:22:04.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:04.579 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:22:04.679 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:22:04.710 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:04.729 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:22:04.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:04.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:22:04.833 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:22:04.883 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:04.924 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:22:04.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:04.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:22:05.030 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:22:05.061 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:05.084 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:22:05.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:05.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:22:05.188 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:22:05.231 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:05.257 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:22:05.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:05.259 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:22:05.360 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:22:05.397 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:05.421 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:22:05.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:05.425 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:22:05.472 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:22:05.510 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:22:05.563 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:22:05.564 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:05 smithi106 ceph-mon[44423]: pgmap v1059: 17 pgs: 17 active+clean; 8.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 75 MiB/s wr, 1.29k op/s 2024-04-20T17:22:05.564 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:05 smithi106 ceph-mon[44423]: 3.f scrub starts 2024-04-20T17:22:05.564 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:05 smithi005 ceph-mon[39562]: pgmap v1059: 17 pgs: 17 active+clean; 8.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 75 MiB/s wr, 1.29k op/s 2024-04-20T17:22:05.565 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:05 smithi005 ceph-mon[39562]: 3.f scrub starts 2024-04-20T17:22:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:05 smithi171 ceph-mon[44189]: pgmap v1059: 17 pgs: 17 active+clean; 8.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 75 MiB/s wr, 1.29k op/s 2024-04-20T17:22:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:05 smithi171 ceph-mon[44189]: 3.f scrub starts 2024-04-20T17:22:05.596 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:22:05.597 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:22:05.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:22:05.597 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:22:05.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.597 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:216588 2.6 2:65396415:::benchmark_data_smithi171_78918_object55594:head [delete] snapc 0=[] ondisk+write+known_if_redirected e210)", 2024-04-20T17:22:05.597 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:40.388336+0000", 2024-04-20T17:22:05.597 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 565.20163102699996, 2024-04-20T17:22:05.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.107803945, 2024-04-20T17:22:05.597 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:22:05.598 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:22:05.598 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:05.598 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:22:05.598 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:22:05.598 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:22:05.598 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 216588 2024-04-20T17:22:05.598 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.598 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:22:05.598 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.598 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:22:05.598 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.388336+0000", 2024-04-20T17:22:05.598 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.599 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.599 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.599 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:22:05.599 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.388336+0000", 2024-04-20T17:22:05.599 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.599 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.599 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.599 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:22:05.599 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.388338+0000", 2024-04-20T17:22:05.599 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.779e-06 2024-04-20T17:22:05.599 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.599 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.600 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:22:05.600 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.388343+0000", 2024-04-20T17:22:05.600 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5539999999999999e-06 2024-04-20T17:22:05.600 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.600 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.600 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:22:05.600 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.388353+0000", 2024-04-20T17:22:05.600 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0502999999999999e-05 2024-04-20T17:22:05.600 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.600 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.600 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:22:05.600 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.388360+0000", 2024-04-20T17:22:05.600 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2810000000000003e-06 2024-04-20T17:22:05.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:22:05.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.388403+0000", 2024-04-20T17:22:05.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2366000000000001e-05 2024-04-20T17:22:05.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:22:05.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.388481+0000", 2024-04-20T17:22:05.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8220000000000007e-05 2024-04-20T17:22:05.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:22:05.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.388547+0000", 2024-04-20T17:22:05.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5841999999999997e-05 2024-04-20T17:22:05.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:22:05.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.393289+0000", 2024-04-20T17:22:05.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0047422339999999997 2024-04-20T17:22:05.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:05.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.496033+0000", 2024-04-20T17:22:05.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.102744271 2024-04-20T17:22:05.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.603 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:22:05.603 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.496078+0000", 2024-04-20T17:22:05.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5105999999999998e-05 2024-04-20T17:22:05.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.603 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:22:05.603 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.496140+0000", 2024-04-20T17:22:05.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1789000000000001e-05 2024-04-20T17:22:05.603 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.603 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:05.603 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.603 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(osd.1.0:31745 2.6 e210/115 ondisk, result = 0)", 2024-04-20T17:22:05.604 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:40.881641+0000", 2024-04-20T17:22:05.604 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 564.70832627799996, 2024-04-20T17:22:05.604 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10404883400000001, 2024-04-20T17:22:05.604 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:22:05.604 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:22:05.604 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:22:05.604 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:22:05.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.604 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:22:05.604 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.881641+0000", 2024-04-20T17:22:05.604 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.604 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.604 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:22:05.604 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.881641+0000", 2024-04-20T17:22:05.605 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.605 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.605 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.605 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:22:05.605 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.881642+0000", 2024-04-20T17:22:05.605 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.034e-06 2024-04-20T17:22:05.605 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.605 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.605 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:22:05.605 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.881646+0000", 2024-04-20T17:22:05.605 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0690000000000003e-06 2024-04-20T17:22:05.605 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.605 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.605 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:22:05.606 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.881655+0000", 2024-04-20T17:22:05.606 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8270000000000004e-06 2024-04-20T17:22:05.606 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.606 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.606 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:22:05.606 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.881661+0000", 2024-04-20T17:22:05.606 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5629999999999999e-06 2024-04-20T17:22:05.606 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.606 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.606 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:22:05.606 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.985621+0000", 2024-04-20T17:22:05.606 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10396008 2024-04-20T17:22:05.606 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.606 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.607 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:22:05.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.985628+0000", 2024-04-20T17:22:05.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4939999999999997e-06 2024-04-20T17:22:05.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.607 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:22:05.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.985690+0000", 2024-04-20T17:22:05.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1766999999999993e-05 2024-04-20T17:22:05.607 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.607 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:05.607 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.607 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:216773 2.6 2:6f744e0b:::benchmark_data_smithi171_78918_object55779:head [delete] snapc 0=[] ondisk+write+known_if_redirected e210)", 2024-04-20T17:22:05.608 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:40.890292+0000", 2024-04-20T17:22:05.608 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 564.69967580699995, 2024-04-20T17:22:05.608 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10639662900000001, 2024-04-20T17:22:05.608 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:22:05.608 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:22:05.608 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:05.608 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:22:05.608 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:22:05.608 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:22:05.608 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 216773 2024-04-20T17:22:05.608 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.608 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:22:05.608 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.608 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:22:05.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.890292+0000", 2024-04-20T17:22:05.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.609 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:22:05.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.890292+0000", 2024-04-20T17:22:05.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.609 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:22:05.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.890293+0000", 2024-04-20T17:22:05.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5340000000000001e-06 2024-04-20T17:22:05.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.610 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.610 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:22:05.610 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.890298+0000", 2024-04-20T17:22:05.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.101e-06 2024-04-20T17:22:05.610 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.610 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.610 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:22:05.610 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.890308+0000", 2024-04-20T17:22:05.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8579999999999995e-06 2024-04-20T17:22:05.610 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.610 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.610 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:22:05.610 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.890314+0000", 2024-04-20T17:22:05.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6890000000000001e-06 2024-04-20T17:22:05.610 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.611 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.611 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:22:05.611 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.987519+0000", 2024-04-20T17:22:05.611 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.097205285000000002 2024-04-20T17:22:05.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.611 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.611 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:22:05.611 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.987620+0000", 2024-04-20T17:22:05.611 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000100648 2024-04-20T17:22:05.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.611 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.611 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:22:05.611 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.987697+0000", 2024-04-20T17:22:05.612 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7645000000000006e-05 2024-04-20T17:22:05.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.612 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:05.612 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.996506+0000", 2024-04-20T17:22:05.612 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0088082349999999993 2024-04-20T17:22:05.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.612 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:22:05.612 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.996589+0000", 2024-04-20T17:22:05.612 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3683000000000005e-05 2024-04-20T17:22:05.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.612 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:22:05.613 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.996640+0000", 2024-04-20T17:22:05.613 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0555e-05 2024-04-20T17:22:05.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.613 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.613 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:22:05.613 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:40.996688+0000", 2024-04-20T17:22:05.613 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8396000000000002e-05 2024-04-20T17:22:05.613 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.613 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:05.613 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.614 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:216942 2.6 2:6cfbbda6:::benchmark_data_smithi171_78918_object55948:head [delete] snapc 0=[] ondisk+write+known_if_redirected e210)", 2024-04-20T17:22:05.614 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:41.297347+0000", 2024-04-20T17:22:05.614 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 564.29262064099998, 2024-04-20T17:22:05.614 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.11543711399999999, 2024-04-20T17:22:05.614 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:22:05.614 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:22:05.614 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:05.614 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:22:05.614 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:22:05.614 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:22:05.614 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 216942 2024-04-20T17:22:05.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.614 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:22:05.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.615 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:22:05.615 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.297347+0000", 2024-04-20T17:22:05.615 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.615 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.615 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:22:05.615 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.297347+0000", 2024-04-20T17:22:05.615 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.615 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.615 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:22:05.615 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.297348+0000", 2024-04-20T17:22:05.615 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7540000000000001e-06 2024-04-20T17:22:05.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.616 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:22:05.616 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.297355+0000", 2024-04-20T17:22:05.616 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0929999999999999e-06 2024-04-20T17:22:05.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.616 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:22:05.616 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.297366+0000", 2024-04-20T17:22:05.616 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1318000000000001e-05 2024-04-20T17:22:05.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.616 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:22:05.616 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.297374+0000", 2024-04-20T17:22:05.617 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4020000000000002e-06 2024-04-20T17:22:05.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.617 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.617 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:22:05.617 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.384301+0000", 2024-04-20T17:22:05.617 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.086926952000000002 2024-04-20T17:22:05.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.617 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.617 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:22:05.617 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.384455+0000", 2024-04-20T17:22:05.617 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015342900000000001 2024-04-20T17:22:05.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.617 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.617 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:22:05.618 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.384607+0000", 2024-04-20T17:22:05.618 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000152406 2024-04-20T17:22:05.618 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.618 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:22:05.618 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.388368+0000", 2024-04-20T17:22:05.618 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003760939 2024-04-20T17:22:05.618 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.618 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:05.618 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.412667+0000", 2024-04-20T17:22:05.618 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.024299438999999999 2024-04-20T17:22:05.618 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.619 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:22:05.619 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.412728+0000", 2024-04-20T17:22:05.619 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0936000000000001e-05 2024-04-20T17:22:05.619 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.619 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:22:05.619 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.412784+0000", 2024-04-20T17:22:05.619 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5445999999999997e-05 2024-04-20T17:22:05.619 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.619 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:05.619 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.619 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.619 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:217108 2.6 2:6f890090:::benchmark_data_smithi171_78918_object56114:head [delete] snapc 0=[] ondisk+write+known_if_redirected e210)", 2024-04-20T17:22:05.620 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:41.907680+0000", 2024-04-20T17:22:05.620 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 563.68228754799998, 2024-04-20T17:22:05.620 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10201656200000001, 2024-04-20T17:22:05.620 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:22:05.620 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:22:05.620 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:05.620 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:22:05.620 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:22:05.620 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:22:05.620 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 217108 2024-04-20T17:22:05.620 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.620 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:22:05.620 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.620 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:22:05.621 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.907680+0000", 2024-04-20T17:22:05.621 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.621 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.621 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.621 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:22:05.621 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.907680+0000", 2024-04-20T17:22:05.621 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.621 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.621 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.621 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:22:05.621 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.907683+0000", 2024-04-20T17:22:05.621 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2650000000000001e-06 2024-04-20T17:22:05.621 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.621 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.621 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:22:05.622 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.907693+0000", 2024-04-20T17:22:05.622 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0362e-05 2024-04-20T17:22:05.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.622 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:22:05.622 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.907714+0000", 2024-04-20T17:22:05.622 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0568999999999999e-05 2024-04-20T17:22:05.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.622 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:22:05.622 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.907726+0000", 2024-04-20T17:22:05.622 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2153e-05 2024-04-20T17:22:05.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.623 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:22:05.623 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.986670+0000", 2024-04-20T17:22:05.623 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.078944241999999998 2024-04-20T17:22:05.623 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.623 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:22:05.623 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.986761+0000", 2024-04-20T17:22:05.623 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0179999999999994e-05 2024-04-20T17:22:05.623 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.624 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.624 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:22:05.624 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.986840+0000", 2024-04-20T17:22:05.624 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9021000000000001e-05 2024-04-20T17:22:05.624 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.624 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.624 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:22:05.624 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:41.998662+0000", 2024-04-20T17:22:05.625 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011822882999999999 2024-04-20T17:22:05.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.625 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.625 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:05.625 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.009567+0000", 2024-04-20T17:22:05.625 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010904528 2024-04-20T17:22:05.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.625 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.625 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:22:05.625 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.009632+0000", 2024-04-20T17:22:05.625 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5217999999999998e-05 2024-04-20T17:22:05.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.625 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.626 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:22:05.626 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.009696+0000", 2024-04-20T17:22:05.626 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4140999999999995e-05 2024-04-20T17:22:05.626 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.626 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:05.626 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.626 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.626 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.626 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:217163 2.6 2:61348ded:::benchmark_data_smithi171_78918_object56169:head [delete] snapc 0=[] ondisk+write+known_if_redirected e210)", 2024-04-20T17:22:05.626 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:42.127376+0000", 2024-04-20T17:22:05.626 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 563.46259091499996, 2024-04-20T17:22:05.626 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.106929598, 2024-04-20T17:22:05.626 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:22:05.627 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:22:05.627 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:05.627 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:22:05.627 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:22:05.627 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:22:05.627 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 217163 2024-04-20T17:22:05.627 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.627 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:22:05.627 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.627 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:22:05.627 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.127376+0000", 2024-04-20T17:22:05.627 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.628 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.628 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.628 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:22:05.628 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.127376+0000", 2024-04-20T17:22:05.628 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.628 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.628 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.628 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:22:05.628 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.127379+0000", 2024-04-20T17:22:05.628 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6220000000000001e-06 2024-04-20T17:22:05.628 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.628 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.628 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:22:05.628 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.127386+0000", 2024-04-20T17:22:05.629 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7889999999999997e-06 2024-04-20T17:22:05.629 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.629 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.629 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:22:05.629 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.127404+0000", 2024-04-20T17:22:05.629 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8179e-05 2024-04-20T17:22:05.629 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.629 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.629 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:22:05.629 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.127414+0000", 2024-04-20T17:22:05.629 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0122000000000001e-05 2024-04-20T17:22:05.629 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.630 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.630 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:22:05.630 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.127490+0000", 2024-04-20T17:22:05.630 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5359999999999997e-05 2024-04-20T17:22:05.630 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.630 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.630 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:22:05.630 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.127622+0000", 2024-04-20T17:22:05.630 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0001325 2024-04-20T17:22:05.630 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.630 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.630 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:22:05.630 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.127751+0000", 2024-04-20T17:22:05.630 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012849500000000001 2024-04-20T17:22:05.631 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.631 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.631 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:22:05.631 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.234161+0000", 2024-04-20T17:22:05.631 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.106410616 2024-04-20T17:22:05.631 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.631 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.631 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:05.631 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.234211+0000", 2024-04-20T17:22:05.631 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9472999999999999e-05 2024-04-20T17:22:05.631 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.631 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.631 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:22:05.632 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.234246+0000", 2024-04-20T17:22:05.632 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5463999999999998e-05 2024-04-20T17:22:05.632 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.632 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.632 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:22:05.632 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.234306+0000", 2024-04-20T17:22:05.632 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9978000000000001e-05 2024-04-20T17:22:05.632 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.632 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:05.632 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.632 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.632 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.632 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(client.24736.0:217163 2.6 e210/115 ondisk, result = 0)", 2024-04-20T17:22:05.632 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:42.133593+0000", 2024-04-20T17:22:05.632 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 563.45637392000003, 2024-04-20T17:22:05.633 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.100718011, 2024-04-20T17:22:05.633 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:22:05.633 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:22:05.633 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:22:05.633 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:22:05.633 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.633 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:22:05.633 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.133593+0000", 2024-04-20T17:22:05.633 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.633 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.633 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.633 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:22:05.633 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.133593+0000", 2024-04-20T17:22:05.633 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.633 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.634 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.634 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:22:05.634 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.133595+0000", 2024-04-20T17:22:05.634 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2309999999999999e-06 2024-04-20T17:22:05.634 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.634 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.634 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:22:05.634 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.133599+0000", 2024-04-20T17:22:05.634 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7350000000000004e-06 2024-04-20T17:22:05.634 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.635 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.635 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:22:05.635 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.133610+0000", 2024-04-20T17:22:05.635 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0481e-05 2024-04-20T17:22:05.635 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.635 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.635 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:22:05.635 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.133617+0000", 2024-04-20T17:22:05.635 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6710000000000002e-06 2024-04-20T17:22:05.635 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.635 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.635 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:22:05.636 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.234199+0000", 2024-04-20T17:22:05.636 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10058223099999999 2024-04-20T17:22:05.636 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.636 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.636 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:22:05.636 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.234207+0000", 2024-04-20T17:22:05.636 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8339999999999999e-06 2024-04-20T17:22:05.636 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.636 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.636 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:22:05.636 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.234311+0000", 2024-04-20T17:22:05.636 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010482800000000001 2024-04-20T17:22:05.636 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.636 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:05.636 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.637 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.637 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.637 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:217217 2.6 2:6881c357:::benchmark_data_smithi171_78918_object56223:head [delete] snapc 0=[] ondisk+write+known_if_redirected e210)", 2024-04-20T17:22:05.637 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:42.357612+0000", 2024-04-20T17:22:05.637 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 563.23235544500005, 2024-04-20T17:22:05.637 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.11809499399999999, 2024-04-20T17:22:05.637 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:22:05.637 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:22:05.637 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:05.637 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:22:05.637 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:22:05.637 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:22:05.637 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 217217 2024-04-20T17:22:05.638 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.638 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:22:05.638 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.638 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:22:05.638 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.357612+0000", 2024-04-20T17:22:05.638 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.638 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.638 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.638 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:22:05.638 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.357612+0000", 2024-04-20T17:22:05.638 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.639 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.639 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.639 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:22:05.639 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.357614+0000", 2024-04-20T17:22:05.639 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0169999999999999e-06 2024-04-20T17:22:05.639 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.639 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.639 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:22:05.639 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.357619+0000", 2024-04-20T17:22:05.639 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9459999999999997e-06 2024-04-20T17:22:05.639 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.639 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.639 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:22:05.640 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.357628+0000", 2024-04-20T17:22:05.640 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6999999999999997e-06 2024-04-20T17:22:05.640 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.640 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.640 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:22:05.640 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.357637+0000", 2024-04-20T17:22:05.640 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0799999999999995e-06 2024-04-20T17:22:05.640 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.640 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.640 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:22:05.640 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.359614+0000", 2024-04-20T17:22:05.640 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0019772430000000001 2024-04-20T17:22:05.640 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.640 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.641 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:22:05.641 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.359706+0000", 2024-04-20T17:22:05.641 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1989000000000003e-05 2024-04-20T17:22:05.641 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.641 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.641 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:22:05.641 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.359783+0000", 2024-04-20T17:22:05.641 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6790000000000002e-05 2024-04-20T17:22:05.641 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.641 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.641 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:22:05.641 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.363334+0000", 2024-04-20T17:22:05.641 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003551008 2024-04-20T17:22:05.642 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.642 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.642 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:05.642 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.475601+0000", 2024-04-20T17:22:05.642 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.112267324 2024-04-20T17:22:05.642 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.642 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.642 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:22:05.642 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.475642+0000", 2024-04-20T17:22:05.643 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0760000000000003e-05 2024-04-20T17:22:05.643 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.643 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.643 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:22:05.643 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.475707+0000", 2024-04-20T17:22:05.643 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5136999999999997e-05 2024-04-20T17:22:05.643 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.643 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:05.643 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.643 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.643 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.643 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(client.24736.0:217217 2.6 e210/115 ondisk, result = 0)", 2024-04-20T17:22:05.643 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:42.372331+0000", 2024-04-20T17:22:05.644 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 563.21763590900002, 2024-04-20T17:22:05.644 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.103382027, 2024-04-20T17:22:05.644 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:22:05.644 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:22:05.644 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:22:05.644 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:22:05.644 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.644 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:22:05.644 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.372331+0000", 2024-04-20T17:22:05.644 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.644 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.644 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.644 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:22:05.645 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.372331+0000", 2024-04-20T17:22:05.645 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.645 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.645 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.645 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:22:05.645 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.372334+0000", 2024-04-20T17:22:05.645 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.92e-06 2024-04-20T17:22:05.645 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.645 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.645 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:22:05.645 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.372344+0000", 2024-04-20T17:22:05.646 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2499999999999995e-06 2024-04-20T17:22:05.646 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.646 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.646 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:22:05.646 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.372356+0000", 2024-04-20T17:22:05.646 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2478e-05 2024-04-20T17:22:05.646 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.646 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.646 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:22:05.646 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.372369+0000", 2024-04-20T17:22:05.646 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.292e-05 2024-04-20T17:22:05.646 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.646 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.646 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:22:05.647 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.475587+0000", 2024-04-20T17:22:05.647 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.103218184 2024-04-20T17:22:05.647 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.647 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.647 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:22:05.647 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.475596+0000", 2024-04-20T17:22:05.647 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3990000000000004e-06 2024-04-20T17:22:05.647 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.647 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.647 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:22:05.647 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.475713+0000", 2024-04-20T17:22:05.647 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011787599999999999 2024-04-20T17:22:05.647 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.647 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:05.648 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.648 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.648 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.648 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(osd.1.0:32102 2.6 e210/115 ondisk, result = 0)", 2024-04-20T17:22:05.648 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:42.500776+0000", 2024-04-20T17:22:05.648 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 563.08919182900001, 2024-04-20T17:22:05.648 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.100849886, 2024-04-20T17:22:05.648 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:22:05.648 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:22:05.648 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:22:05.648 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:22:05.648 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.648 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:22:05.648 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.500776+0000", 2024-04-20T17:22:05.648 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.649 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.649 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.649 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:22:05.649 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.500776+0000", 2024-04-20T17:22:05.649 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.649 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.649 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.649 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:22:05.649 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.500778+0000", 2024-04-20T17:22:05.649 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2359999999999999e-06 2024-04-20T17:22:05.649 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.649 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.649 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:22:05.650 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.500786+0000", 2024-04-20T17:22:05.650 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3280000000000006e-06 2024-04-20T17:22:05.650 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.650 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.650 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:22:05.650 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.500799+0000", 2024-04-20T17:22:05.650 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3223e-05 2024-04-20T17:22:05.650 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.650 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.650 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:22:05.650 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.500808+0000", 2024-04-20T17:22:05.651 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.098e-06 2024-04-20T17:22:05.651 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.651 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.651 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:22:05.651 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.601598+0000", 2024-04-20T17:22:05.651 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.100790038 2024-04-20T17:22:05.651 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.651 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.651 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:22:05.651 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.601608+0000", 2024-04-20T17:22:05.651 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8349999999999999e-06 2024-04-20T17:22:05.651 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.651 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.652 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:22:05.652 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.601625+0000", 2024-04-20T17:22:05.652 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7127999999999999e-05 2024-04-20T17:22:05.652 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.652 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:05.652 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.652 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.652 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.652 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(osd.1.0:32103 2.6 e210/115 ondisk, result = 0)", 2024-04-20T17:22:05.652 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:42.500909+0000", 2024-04-20T17:22:05.652 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 563.08905864600001, 2024-04-20T17:22:05.652 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.100980444, 2024-04-20T17:22:05.652 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:22:05.652 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:22:05.653 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:22:05.653 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:22:05.653 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.653 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:22:05.653 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.500909+0000", 2024-04-20T17:22:05.653 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.653 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.653 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.653 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:22:05.653 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.500909+0000", 2024-04-20T17:22:05.653 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.653 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.653 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.653 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:22:05.653 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.500909+0000", 2024-04-20T17:22:05.654 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6700000000000003e-07 2024-04-20T17:22:05.654 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.654 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.654 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:22:05.654 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.500913+0000", 2024-04-20T17:22:05.654 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.366e-06 2024-04-20T17:22:05.654 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.654 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.654 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:22:05.654 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.500921+0000", 2024-04-20T17:22:05.654 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5879999999999998e-06 2024-04-20T17:22:05.654 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.654 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.655 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:22:05.655 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.500929+0000", 2024-04-20T17:22:05.655 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4869999999999996e-06 2024-04-20T17:22:05.655 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.655 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.655 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:22:05.655 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.601650+0000", 2024-04-20T17:22:05.655 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10072130899999999 2024-04-20T17:22:05.655 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.655 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.655 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:22:05.655 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.601655+0000", 2024-04-20T17:22:05.655 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8709999999999996e-06 2024-04-20T17:22:05.655 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.656 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.656 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:22:05.656 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.601889+0000", 2024-04-20T17:22:05.656 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00023405600000000001 2024-04-20T17:22:05.656 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.656 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:05.656 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.656 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.656 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.657 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:217254 2.6 2:67989c67:::benchmark_data_smithi171_78918_object56260:head [delete] snapc 0=[] ondisk+write+known_if_redirected e210)", 2024-04-20T17:22:05.657 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:42.610585+0000", 2024-04-20T17:22:05.657 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 562.97938279300001, 2024-04-20T17:22:05.658 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.111531326, 2024-04-20T17:22:05.658 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:22:05.658 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:22:05.658 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:05.658 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:22:05.658 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:22:05.658 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:22:05.658 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 217254 2024-04-20T17:22:05.658 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.658 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:22:05.658 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.658 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:22:05.658 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.610585+0000", 2024-04-20T17:22:05.659 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.659 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.659 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.659 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:22:05.659 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.610585+0000", 2024-04-20T17:22:05.659 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.659 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.659 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.659 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:22:05.659 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.610586+0000", 2024-04-20T17:22:05.659 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8700000000000001e-06 2024-04-20T17:22:05.659 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.659 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.659 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:22:05.660 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.610592+0000", 2024-04-20T17:22:05.660 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1889999999999998e-06 2024-04-20T17:22:05.660 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.660 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.660 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:22:05.660 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.610601+0000", 2024-04-20T17:22:05.660 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6029999999999994e-06 2024-04-20T17:22:05.660 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.660 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.660 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:22:05.660 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.610608+0000", 2024-04-20T17:22:05.660 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6980000000000001e-06 2024-04-20T17:22:05.660 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.660 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.661 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:22:05.661 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.610641+0000", 2024-04-20T17:22:05.661 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3491000000000002e-05 2024-04-20T17:22:05.661 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.661 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.661 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:22:05.661 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.610729+0000", 2024-04-20T17:22:05.661 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7793999999999993e-05 2024-04-20T17:22:05.661 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.661 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.661 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:22:05.661 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.610803+0000", 2024-04-20T17:22:05.661 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3292000000000002e-05 2024-04-20T17:22:05.661 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.661 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.662 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:22:05.662 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.614316+0000", 2024-04-20T17:22:05.662 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0035132879999999998 2024-04-20T17:22:05.662 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.662 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.662 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:05.662 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.722032+0000", 2024-04-20T17:22:05.662 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.107716066 2024-04-20T17:22:05.662 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.662 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.662 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:22:05.662 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.722070+0000", 2024-04-20T17:22:05.662 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8568000000000001e-05 2024-04-20T17:22:05.663 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.663 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.663 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:22:05.663 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:42.722116+0000", 2024-04-20T17:22:05.663 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5466999999999997e-05 2024-04-20T17:22:05.663 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.663 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:05.663 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.663 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.663 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.663 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:217333 2.6 2:6fe5f61e:::benchmark_data_smithi171_78918_object56339:head [delete] snapc 0=[] ondisk+write+known_if_redirected e211)", 2024-04-20T17:22:05.663 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:43.301777+0000", 2024-04-20T17:22:05.664 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 562.28819045099999, 2024-04-20T17:22:05.664 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.104629244, 2024-04-20T17:22:05.664 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:22:05.664 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:22:05.664 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:05.664 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:22:05.664 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:22:05.664 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:22:05.664 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 217333 2024-04-20T17:22:05.664 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.664 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:22:05.664 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.664 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:22:05.665 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:43.301777+0000", 2024-04-20T17:22:05.665 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.665 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.665 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.665 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:22:05.665 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:43.301777+0000", 2024-04-20T17:22:05.665 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.665 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.665 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.665 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:22:05.665 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:43.301778+0000", 2024-04-20T17:22:05.665 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4440000000000001e-06 2024-04-20T17:22:05.666 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.666 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.666 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:22:05.666 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:43.301782+0000", 2024-04-20T17:22:05.666 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1280000000000001e-06 2024-04-20T17:22:05.666 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.666 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.666 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:22:05.666 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:43.301792+0000", 2024-04-20T17:22:05.666 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2569999999999996e-06 2024-04-20T17:22:05.666 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.666 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.666 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:22:05.667 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:43.301798+0000", 2024-04-20T17:22:05.667 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1340000000000001e-06 2024-04-20T17:22:05.667 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.667 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.667 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:22:05.667 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:43.301827+0000", 2024-04-20T17:22:05.667 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9444000000000001e-05 2024-04-20T17:22:05.667 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.667 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.667 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:22:05.667 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:43.301911+0000", 2024-04-20T17:22:05.667 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3218000000000002e-05 2024-04-20T17:22:05.668 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.668 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.668 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:22:05.668 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:43.301987+0000", 2024-04-20T17:22:05.668 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6551999999999995e-05 2024-04-20T17:22:05.668 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.668 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.668 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:22:05.668 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:43.305370+0000", 2024-04-20T17:22:05.668 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0033824860000000001 2024-04-20T17:22:05.668 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.668 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.668 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:05.669 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:43.406310+0000", 2024-04-20T17:22:05.669 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10094076 2024-04-20T17:22:05.669 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.669 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.669 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:22:05.669 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:43.406347+0000", 2024-04-20T17:22:05.669 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6730000000000002e-05 2024-04-20T17:22:05.669 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.669 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.669 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:22:05.669 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:43.406406+0000", 2024-04-20T17:22:05.669 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9091000000000001e-05 2024-04-20T17:22:05.669 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.669 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:05.669 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.670 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.670 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.670 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:217607 2.6 2:699b2bdd:::benchmark_data_smithi171_78918_object56613:head [delete] snapc 0=[] ondisk+write+known_if_redirected e212)", 2024-04-20T17:22:05.670 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:44.238355+0000", 2024-04-20T17:22:05.670 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 561.35161237399996, 2024-04-20T17:22:05.670 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.112049101, 2024-04-20T17:22:05.670 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:22:05.670 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:22:05.670 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:05.670 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:22:05.670 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:22:05.670 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:22:05.670 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 217607 2024-04-20T17:22:05.670 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.671 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:22:05.671 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.671 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:22:05.671 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:44.238355+0000", 2024-04-20T17:22:05.671 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.671 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.671 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.671 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:22:05.671 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:44.238355+0000", 2024-04-20T17:22:05.671 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.671 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.671 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.671 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:22:05.671 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:44.238357+0000", 2024-04-20T17:22:05.672 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.249e-06 2024-04-20T17:22:05.672 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.672 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.672 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:22:05.672 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:44.238363+0000", 2024-04-20T17:22:05.672 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.592e-06 2024-04-20T17:22:05.672 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.672 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.672 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:22:05.672 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:44.238373+0000", 2024-04-20T17:22:05.672 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0083e-05 2024-04-20T17:22:05.672 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.672 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.672 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:22:05.673 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:44.238381+0000", 2024-04-20T17:22:05.673 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2150000000000001e-06 2024-04-20T17:22:05.673 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.673 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.673 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:22:05.673 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:44.327777+0000", 2024-04-20T17:22:05.673 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.089396327999999997 2024-04-20T17:22:05.673 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.673 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.673 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:22:05.674 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:44.327869+0000", 2024-04-20T17:22:05.674 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1675000000000006e-05 2024-04-20T17:22:05.674 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.674 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.674 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:22:05.674 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:44.327950+0000", 2024-04-20T17:22:05.674 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0946999999999996e-05 2024-04-20T17:22:05.674 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.674 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.674 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:22:05.674 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:44.331381+0000", 2024-04-20T17:22:05.674 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0034311739999999999 2024-04-20T17:22:05.675 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.675 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.675 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:05.675 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:44.350269+0000", 2024-04-20T17:22:05.675 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.018888139000000002 2024-04-20T17:22:05.675 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.675 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.675 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:22:05.675 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:44.350342+0000", 2024-04-20T17:22:05.675 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2323999999999994e-05 2024-04-20T17:22:05.675 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.675 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.675 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:22:05.676 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:44.350404+0000", 2024-04-20T17:22:05.676 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2374999999999998e-05 2024-04-20T17:22:05.676 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.676 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:05.676 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.676 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.676 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.676 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:218037 2.6 2:6deb4aaa:::benchmark_data_smithi171_78918_object57043:head [delete] snapc 0=[] ondisk+write+known_if_redirected e212)", 2024-04-20T17:22:05.676 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:46.049475+0000", 2024-04-20T17:22:05.676 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 559.54049234700005, 2024-04-20T17:22:05.676 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10739074899999999, 2024-04-20T17:22:05.676 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:22:05.677 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:22:05.677 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:05.677 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:22:05.677 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:22:05.677 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:22:05.677 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 218037 2024-04-20T17:22:05.677 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.677 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:22:05.677 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.677 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:22:05.678 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.049475+0000", 2024-04-20T17:22:05.678 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.678 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.678 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.678 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:22:05.678 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.049475+0000", 2024-04-20T17:22:05.678 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.678 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.678 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.678 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:22:05.678 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.049478+0000", 2024-04-20T17:22:05.678 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.914e-06 2024-04-20T17:22:05.678 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.679 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.679 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:22:05.679 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.049488+0000", 2024-04-20T17:22:05.679 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8810000000000008e-06 2024-04-20T17:22:05.679 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.681 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.681 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:22:05.681 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.049510+0000", 2024-04-20T17:22:05.681 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2116e-05 2024-04-20T17:22:05.681 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.681 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.681 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:22:05.681 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.049521+0000", 2024-04-20T17:22:05.682 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1358e-05 2024-04-20T17:22:05.682 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.682 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.682 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:22:05.682 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.049584+0000", 2024-04-20T17:22:05.682 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3163999999999994e-05 2024-04-20T17:22:05.682 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.682 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.682 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:22:05.682 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.049740+0000", 2024-04-20T17:22:05.682 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015564100000000001 2024-04-20T17:22:05.682 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.683 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.683 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:22:05.683 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.049877+0000", 2024-04-20T17:22:05.683 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013700999999999999 2024-04-20T17:22:05.683 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.683 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.683 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:22:05.683 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.054646+0000", 2024-04-20T17:22:05.683 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0047689020000000002 2024-04-20T17:22:05.683 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.683 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.683 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:05.683 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.156779+0000", 2024-04-20T17:22:05.683 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.102132614 2024-04-20T17:22:05.684 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.684 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.684 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:22:05.684 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.156815+0000", 2024-04-20T17:22:05.684 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6798000000000002e-05 2024-04-20T17:22:05.684 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.684 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.684 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:22:05.685 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.156866+0000", 2024-04-20T17:22:05.685 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0351e-05 2024-04-20T17:22:05.685 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.685 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:05.685 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.685 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.685 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.685 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:218053 2.6 2:6c335b6c:::benchmark_data_smithi171_78918_object57059:head [delete] snapc 0=[] ondisk+write+known_if_redirected e212)", 2024-04-20T17:22:05.685 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:46.175572+0000", 2024-04-20T17:22:05.685 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 559.414395324, 2024-04-20T17:22:05.685 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10556868, 2024-04-20T17:22:05.686 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:22:05.686 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:22:05.686 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:05.686 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:22:05.686 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:22:05.686 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:22:05.686 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 218053 2024-04-20T17:22:05.686 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.686 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:22:05.686 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.686 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:22:05.686 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.175572+0000", 2024-04-20T17:22:05.687 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.687 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.687 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.687 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:22:05.687 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.175572+0000", 2024-04-20T17:22:05.687 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.687 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.687 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.687 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:22:05.687 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.175575+0000", 2024-04-20T17:22:05.687 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9359999999999999e-06 2024-04-20T17:22:05.687 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.687 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.687 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:22:05.688 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.175583+0000", 2024-04-20T17:22:05.688 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0479999999999999e-06 2024-04-20T17:22:05.688 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.688 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.688 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:22:05.688 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.175598+0000", 2024-04-20T17:22:05.688 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4854999999999999e-05 2024-04-20T17:22:05.688 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.688 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.688 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:22:05.688 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.175610+0000", 2024-04-20T17:22:05.688 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1688000000000001e-05 2024-04-20T17:22:05.688 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.688 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.688 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:22:05.689 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.269744+0000", 2024-04-20T17:22:05.689 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.094134526999999996 2024-04-20T17:22:05.689 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.689 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.689 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:22:05.689 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.269830+0000", 2024-04-20T17:22:05.689 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6122e-05 2024-04-20T17:22:05.689 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.689 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.689 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:22:05.689 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.269904+0000", 2024-04-20T17:22:05.689 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3610999999999997e-05 2024-04-20T17:22:05.690 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.690 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.690 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:22:05.690 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.280955+0000", 2024-04-20T17:22:05.690 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011051518999999999 2024-04-20T17:22:05.690 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.690 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.690 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:05.690 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.281015+0000", 2024-04-20T17:22:05.690 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9401999999999998e-05 2024-04-20T17:22:05.690 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.691 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.691 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:22:05.691 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.281077+0000", 2024-04-20T17:22:05.691 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2491999999999997e-05 2024-04-20T17:22:05.691 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.691 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.691 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:22:05.691 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:46.281141+0000", 2024-04-20T17:22:05.691 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3479999999999995e-05 2024-04-20T17:22:05.691 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.691 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:05.691 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.691 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.691 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.691 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:218337 2.6 2:66af67bd:::benchmark_data_smithi171_78918_object57343:head [delete] snapc 0=[] ondisk+write+known_if_redirected e213)", 2024-04-20T17:22:05.692 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:47.343662+0000", 2024-04-20T17:22:05.692 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 558.24630542099999, 2024-04-20T17:22:05.692 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10596135800000001, 2024-04-20T17:22:05.692 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:22:05.692 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:22:05.692 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:05.692 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:22:05.692 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:22:05.692 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:22:05.692 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 218337 2024-04-20T17:22:05.692 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.692 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:22:05.693 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.693 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:22:05.693 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:47.343662+0000", 2024-04-20T17:22:05.693 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.693 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.693 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.693 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:22:05.693 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:47.343662+0000", 2024-04-20T17:22:05.694 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.694 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.694 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.694 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:22:05.694 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:47.343665+0000", 2024-04-20T17:22:05.694 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8080000000000001e-06 2024-04-20T17:22:05.694 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.694 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.694 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:22:05.694 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:47.343673+0000", 2024-04-20T17:22:05.694 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1009999999999992e-06 2024-04-20T17:22:05.694 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.694 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.695 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:22:05.695 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:47.343685+0000", 2024-04-20T17:22:05.695 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2547e-05 2024-04-20T17:22:05.695 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.695 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.695 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:22:05.695 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:47.343701+0000", 2024-04-20T17:22:05.695 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5223e-05 2024-04-20T17:22:05.695 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.695 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.695 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:22:05.695 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:47.433742+0000", 2024-04-20T17:22:05.695 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.090041287999999997 2024-04-20T17:22:05.695 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.696 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.696 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:22:05.696 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:47.433834+0000", 2024-04-20T17:22:05.696 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2311000000000004e-05 2024-04-20T17:22:05.696 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.696 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.696 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:22:05.696 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:47.433930+0000", 2024-04-20T17:22:05.696 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6169999999999996e-05 2024-04-20T17:22:05.696 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.696 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.696 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:22:05.696 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:47.437577+0000", 2024-04-20T17:22:05.696 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0036469300000000001 2024-04-20T17:22:05.696 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.697 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.697 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:05.697 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:47.449548+0000", 2024-04-20T17:22:05.697 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011970824999999999 2024-04-20T17:22:05.697 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.697 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.697 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:22:05.697 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:47.449585+0000", 2024-04-20T17:22:05.697 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6810999999999997e-05 2024-04-20T17:22:05.697 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.697 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.697 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:22:05.697 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:47.449623+0000", 2024-04-20T17:22:05.697 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8343999999999997e-05 2024-04-20T17:22:05.698 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.698 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:05.698 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.698 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.698 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.698 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:218701 2.6 2:6d43b903:::benchmark_data_smithi171_78918_object57707:head [delete] snapc 0=[] ondisk+write+known_if_redirected e213)", 2024-04-20T17:22:05.698 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:49.344998+0000", 2024-04-20T17:22:05.698 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 556.24496947600005, 2024-04-20T17:22:05.698 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10736288099999999, 2024-04-20T17:22:05.698 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:22:05.698 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:22:05.698 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:05.698 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:22:05.699 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:22:05.699 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:22:05.699 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 218701 2024-04-20T17:22:05.699 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.699 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:22:05.699 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.699 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:22:05.699 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.344998+0000", 2024-04-20T17:22:05.699 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.699 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.699 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.699 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:22:05.699 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.344998+0000", 2024-04-20T17:22:05.699 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.699 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.700 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.700 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:22:05.700 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.345000+0000", 2024-04-20T17:22:05.700 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1730000000000002e-06 2024-04-20T17:22:05.700 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.700 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.700 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:22:05.700 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.345006+0000", 2024-04-20T17:22:05.700 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1489999999999999e-06 2024-04-20T17:22:05.700 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.700 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.700 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:22:05.700 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.345017+0000", 2024-04-20T17:22:05.700 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0341e-05 2024-04-20T17:22:05.701 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.701 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.701 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:22:05.701 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.345024+0000", 2024-04-20T17:22:05.701 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3699999999999997e-06 2024-04-20T17:22:05.701 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.701 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.701 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:22:05.701 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.444845+0000", 2024-04-20T17:22:05.701 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.099821550999999994 2024-04-20T17:22:05.701 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.701 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.701 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:22:05.702 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.444949+0000", 2024-04-20T17:22:05.702 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010375200000000001 2024-04-20T17:22:05.702 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.702 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.702 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:22:05.702 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.445038+0000", 2024-04-20T17:22:05.702 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8702999999999994e-05 2024-04-20T17:22:05.702 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.702 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.702 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:22:05.702 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.449444+0000", 2024-04-20T17:22:05.702 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0044062459999999999 2024-04-20T17:22:05.702 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.703 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.703 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:05.703 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.452220+0000", 2024-04-20T17:22:05.703 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0027756500000000002 2024-04-20T17:22:05.703 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.703 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.703 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:22:05.703 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.452303+0000", 2024-04-20T17:22:05.703 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2978000000000004e-05 2024-04-20T17:22:05.703 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.703 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.703 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:22:05.703 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.452361+0000", 2024-04-20T17:22:05.703 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7967999999999998e-05 2024-04-20T17:22:05.704 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.704 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:05.704 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.704 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.704 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.704 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(osd.1.0:33705 2.6 e213/115 ondisk, result = 0)", 2024-04-20T17:22:05.704 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:49.349222+0000", 2024-04-20T17:22:05.704 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 556.24074552599996, 2024-04-20T17:22:05.704 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10013370100000001, 2024-04-20T17:22:05.704 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:22:05.704 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:22:05.704 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:22:05.704 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:22:05.705 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.705 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:22:05.705 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.349222+0000", 2024-04-20T17:22:05.705 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.705 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.705 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.705 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:22:05.705 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.349222+0000", 2024-04-20T17:22:05.705 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.705 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.705 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.705 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:22:05.705 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.349224+0000", 2024-04-20T17:22:05.705 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.674e-06 2024-04-20T17:22:05.706 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.706 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.706 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:22:05.706 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.349230+0000", 2024-04-20T17:22:05.706 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4099999999999996e-06 2024-04-20T17:22:05.706 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.706 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.706 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:22:05.706 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.349240+0000", 2024-04-20T17:22:05.706 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6150000000000003e-06 2024-04-20T17:22:05.706 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.706 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.706 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:22:05.706 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.349248+0000", 2024-04-20T17:22:05.707 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.755e-06 2024-04-20T17:22:05.707 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.707 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.707 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:22:05.707 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.449096+0000", 2024-04-20T17:22:05.707 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.099847801999999999 2024-04-20T17:22:05.707 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.707 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.707 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:22:05.707 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.449107+0000", 2024-04-20T17:22:05.707 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1037e-05 2024-04-20T17:22:05.707 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.707 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.707 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:22:05.708 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.449356+0000", 2024-04-20T17:22:05.708 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00024840799999999998 2024-04-20T17:22:05.708 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.708 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:05.708 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.708 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.708 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.708 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(osd.1.0:33759 2.6 e213/115 ondisk, result = 0)", 2024-04-20T17:22:05.708 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:12:49.593394+0000", 2024-04-20T17:22:05.708 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 555.99657385499995, 2024-04-20T17:22:05.708 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10355750800000001, 2024-04-20T17:22:05.708 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:22:05.708 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:22:05.709 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:22:05.709 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:22:05.709 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.709 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:22:05.709 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.593394+0000", 2024-04-20T17:22:05.709 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.709 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.709 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.709 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:22:05.709 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.593394+0000", 2024-04-20T17:22:05.709 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:22:05.709 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.709 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.709 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:22:05.709 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.593395+0000", 2024-04-20T17:22:05.710 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.209e-06 2024-04-20T17:22:05.710 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.710 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.710 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:22:05.710 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.593399+0000", 2024-04-20T17:22:05.710 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2939999999999999e-06 2024-04-20T17:22:05.710 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.710 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.710 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:22:05.710 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.593408+0000", 2024-04-20T17:22:05.710 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5359999999999992e-06 2024-04-20T17:22:05.710 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.710 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.710 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:22:05.711 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.593414+0000", 2024-04-20T17:22:05.711 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3749999999999999e-06 2024-04-20T17:22:05.711 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.711 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.711 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:22:05.711 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.696853+0000", 2024-04-20T17:22:05.711 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10343883800000001 2024-04-20T17:22:05.711 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.711 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.711 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:22:05.711 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.696862+0000", 2024-04-20T17:22:05.711 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.789e-06 2024-04-20T17:22:05.711 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:05.711 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:05.711 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:22:05.711 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:12:49.696951+0000", 2024-04-20T17:22:05.712 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9467000000000002e-05 2024-04-20T17:22:05.712 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.712 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:05.712 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.712 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:05.712 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:05.712 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:22:05.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:05.713 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:05.713 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:22:05.713 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:22:05.814 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:22:05.846 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:05.868 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:22:05.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:05.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:22:05.972 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:22:06.002 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:06.025 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:22:06.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:06.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:22:06.130 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:22:06.178 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:06.650 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:22:06.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:06.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:22:06.664 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-20T17:22:06.758 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:22:06.826 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:06.929 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:22:06.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:06.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:22:07.035 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:22:07.076 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:07.104 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:22:07.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:07.108 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:22:07.209 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:22:07.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:07 smithi005 ceph-mon[39562]: 3.3 scrub starts 2024-04-20T17:22:07.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:07 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:22:07.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:07 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:22:07.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:07 smithi005 ceph-mon[39562]: pgmap v1060: 17 pgs: 17 active+clean; 8.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 102 B/s rd, 60 MiB/s wr, 1.28k op/s 2024-04-20T17:22:07.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:07 smithi005 ceph-mon[39562]: 3.f scrub ok 2024-04-20T17:22:07.253 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:07 smithi171 ceph-mon[44189]: 3.3 scrub starts 2024-04-20T17:22:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:07 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:22:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:07 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:22:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:07 smithi171 ceph-mon[44189]: pgmap v1060: 17 pgs: 17 active+clean; 8.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 102 B/s rd, 60 MiB/s wr, 1.28k op/s 2024-04-20T17:22:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:07 smithi171 ceph-mon[44189]: 3.f scrub ok 2024-04-20T17:22:07.490 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:22:07.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:07.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:22:07.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:07 smithi106 ceph-mon[44423]: 3.3 scrub starts 2024-04-20T17:22:07.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:07 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:22:07.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:07 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:22:07.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:07 smithi106 ceph-mon[44423]: pgmap v1060: 17 pgs: 17 active+clean; 8.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 102 B/s rd, 60 MiB/s wr, 1.28k op/s 2024-04-20T17:22:07.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:07 smithi106 ceph-mon[44423]: 3.f scrub ok 2024-04-20T17:22:07.579 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:22:07.601 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:22:07.637 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:07.662 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:22:07.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:07.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:22:07.769 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:22:07.816 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:07.841 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:22:07.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:07.849 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:22:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:22:07.949 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:22:07.986 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:08.011 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:22:08.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:08.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:22:08.117 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:22:08.149 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:08.173 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:22:08.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:08.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:22:08.193 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:08 smithi005 ceph-mon[39562]: 3.4 scrub starts 2024-04-20T17:22:08.193 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:08 smithi005 ceph-mon[39562]: 3.b scrub starts 2024-04-20T17:22:08.193 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:08 smithi005 ceph-mon[39562]: osdmap e391: 12 total, 11 up, 11 in 2024-04-20T17:22:08.193 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:08 smithi005 ceph-mon[39562]: 3.a scrub starts 2024-04-20T17:22:08.193 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:08 smithi005 ceph-mon[39562]: 3.3 scrub ok 2024-04-20T17:22:08.193 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:08 smithi005 ceph-mon[39562]: 3.c scrub starts 2024-04-20T17:22:08.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:08 smithi106 ceph-mon[44423]: 3.4 scrub starts 2024-04-20T17:22:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:08 smithi106 ceph-mon[44423]: 3.b scrub starts 2024-04-20T17:22:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:08 smithi106 ceph-mon[44423]: osdmap e391: 12 total, 11 up, 11 in 2024-04-20T17:22:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:08 smithi106 ceph-mon[44423]: 3.a scrub starts 2024-04-20T17:22:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:08 smithi106 ceph-mon[44423]: 3.3 scrub ok 2024-04-20T17:22:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:08 smithi106 ceph-mon[44423]: 3.c scrub starts 2024-04-20T17:22:08.277 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:22:08.312 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:08 smithi171 ceph-mon[44189]: 3.4 scrub starts 2024-04-20T17:22:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:08 smithi171 ceph-mon[44189]: 3.b scrub starts 2024-04-20T17:22:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:08 smithi171 ceph-mon[44189]: osdmap e391: 12 total, 11 up, 11 in 2024-04-20T17:22:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:08 smithi171 ceph-mon[44189]: 3.a scrub starts 2024-04-20T17:22:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:08 smithi171 ceph-mon[44189]: 3.3 scrub ok 2024-04-20T17:22:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:08 smithi171 ceph-mon[44189]: 3.c scrub starts 2024-04-20T17:22:08.338 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:22:08.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:08.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:22:08.443 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:22:08.485 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:22:08.488 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:08.512 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:22:08.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:08.515 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:22:08.532 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:22:08.577 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:22:08.616 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:22:08.625 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:22:08.645 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:08.669 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:22:08.674 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:22:08.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:08.677 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:22:08.696 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:22:08.739 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:22:08.778 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:22:08.804 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:22:08.805 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:08.826 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:22:08.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:08.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:22:08.841 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:22:08.883 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:22:08.920 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:22:08.921 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:22:08.921 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:22:08.921 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:22:08.921 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:22:08.921 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:22:08.921 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:22:08.921 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:22:08.921 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:22:08.921 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:22:08.921 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:22:08.921 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:22:08.921 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:22:08.933 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:22:08.966 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:08.992 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:22:08.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:08.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:22:09.096 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:22:09.132 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:09.156 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:22:09.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:09.157 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:22:09.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:09 smithi005 ceph-mon[39562]: 3.7 scrub starts 2024-04-20T17:22:09.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:09 smithi005 ceph-mon[39562]: pgmap v1062: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.4 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 127 B/s rd, 48 MiB/s wr, 1.44k op/s 2024-04-20T17:22:09.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:09 smithi005 ceph-mon[39562]: 3.4 scrub ok 2024-04-20T17:22:09.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:09 smithi005 ceph-mon[39562]: 3.8 scrub starts 2024-04-20T17:22:09.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:09 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3251638774' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:22:09.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:09.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:09 smithi005 ceph-mon[39562]: 3.a scrub ok 2024-04-20T17:22:09.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:09.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:09.259 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:22:09.293 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:09.323 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:22:09.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:09.331 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:22:09.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:09 smithi171 ceph-mon[44189]: 3.7 scrub starts 2024-04-20T17:22:09.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:09 smithi171 ceph-mon[44189]: pgmap v1062: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.4 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 127 B/s rd, 48 MiB/s wr, 1.44k op/s 2024-04-20T17:22:09.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:09 smithi171 ceph-mon[44189]: 3.4 scrub ok 2024-04-20T17:22:09.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:09 smithi171 ceph-mon[44189]: 3.8 scrub starts 2024-04-20T17:22:09.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:09 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3251638774' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:22:09.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:09.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:09 smithi171 ceph-mon[44189]: 3.a scrub ok 2024-04-20T17:22:09.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:09.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:09.432 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:22:09.487 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:09.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:09 smithi106 ceph-mon[44423]: 3.7 scrub starts 2024-04-20T17:22:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:09 smithi106 ceph-mon[44423]: pgmap v1062: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.4 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 127 B/s rd, 48 MiB/s wr, 1.44k op/s 2024-04-20T17:22:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:09 smithi106 ceph-mon[44423]: 3.4 scrub ok 2024-04-20T17:22:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:09 smithi106 ceph-mon[44423]: 3.8 scrub starts 2024-04-20T17:22:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:09 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3251638774' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:22:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:09 smithi106 ceph-mon[44423]: 3.a scrub ok 2024-04-20T17:22:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:10.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:10.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:10.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:10.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:10.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:10.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:10.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:10.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:10.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:10 smithi005 ceph-mon[39562]: 3.c scrub ok 2024-04-20T17:22:10.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:10 smithi005 ceph-mon[39562]: 3.2 scrub starts 2024-04-20T17:22:10.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:10 smithi005 ceph-mon[39562]: 3.b scrub ok 2024-04-20T17:22:10.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:10 smithi106 ceph-mon[44423]: 3.c scrub ok 2024-04-20T17:22:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:10 smithi106 ceph-mon[44423]: 3.2 scrub starts 2024-04-20T17:22:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:10 smithi106 ceph-mon[44423]: 3.b scrub ok 2024-04-20T17:22:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:10 smithi171 ceph-mon[44189]: 3.c scrub ok 2024-04-20T17:22:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:10 smithi171 ceph-mon[44189]: 3.2 scrub starts 2024-04-20T17:22:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:10 smithi171 ceph-mon[44189]: 3.b scrub ok 2024-04-20T17:22:10.835 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:22:10.835 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:22:11.092 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:11 smithi005 ceph-mon[39562]: pgmap v1063: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.3 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 127 B/s rd, 22 MiB/s wr, 1.17k op/s 2024-04-20T17:22:11.092 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:11 smithi005 ceph-mon[39562]: 3.6 scrub starts 2024-04-20T17:22:11.092 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:11 smithi005 ceph-mon[39562]: 3.7 scrub ok 2024-04-20T17:22:11.092 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:11 smithi005 ceph-mon[39562]: 3.8 scrub ok 2024-04-20T17:22:11.093 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:11 smithi005 ceph-mon[39562]: 3.9 scrub starts 2024-04-20T17:22:11.093 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:11 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:22:11.093 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:11 smithi005 ceph-mon[39562]: 3.2 scrub ok 2024-04-20T17:22:11.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:11 smithi106 ceph-mon[44423]: pgmap v1063: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.3 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 127 B/s rd, 22 MiB/s wr, 1.17k op/s 2024-04-20T17:22:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:11 smithi106 ceph-mon[44423]: 3.6 scrub starts 2024-04-20T17:22:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:11 smithi106 ceph-mon[44423]: 3.7 scrub ok 2024-04-20T17:22:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:11 smithi106 ceph-mon[44423]: 3.8 scrub ok 2024-04-20T17:22:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:11 smithi106 ceph-mon[44423]: 3.9 scrub starts 2024-04-20T17:22:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:11 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:22:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:11 smithi106 ceph-mon[44423]: 3.2 scrub ok 2024-04-20T17:22:11.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:11 smithi171 ceph-mon[44189]: pgmap v1063: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.3 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 127 B/s rd, 22 MiB/s wr, 1.17k op/s 2024-04-20T17:22:11.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:11 smithi171 ceph-mon[44189]: 3.6 scrub starts 2024-04-20T17:22:11.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:11 smithi171 ceph-mon[44189]: 3.7 scrub ok 2024-04-20T17:22:11.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:11 smithi171 ceph-mon[44189]: 3.8 scrub ok 2024-04-20T17:22:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:11 smithi171 ceph-mon[44189]: 3.9 scrub starts 2024-04-20T17:22:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:11 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:22:11.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:11 smithi171 ceph-mon[44189]: 3.2 scrub ok 2024-04-20T17:22:12.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:12 smithi005 ceph-mon[39562]: 3.0 scrub starts 2024-04-20T17:22:12.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:12 smithi005 ceph-mon[39562]: 3.6 scrub ok 2024-04-20T17:22:12.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:12 smithi005 ceph-mon[39562]: 3.9 scrub ok 2024-04-20T17:22:12.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:12 smithi106 ceph-mon[44423]: 3.0 scrub starts 2024-04-20T17:22:12.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:12 smithi106 ceph-mon[44423]: 3.6 scrub ok 2024-04-20T17:22:12.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:12 smithi106 ceph-mon[44423]: 3.9 scrub ok 2024-04-20T17:22:12.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:12 smithi171 ceph-mon[44189]: 3.0 scrub starts 2024-04-20T17:22:12.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:12 smithi171 ceph-mon[44189]: 3.6 scrub ok 2024-04-20T17:22:12.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:12 smithi171 ceph-mon[44189]: 3.9 scrub ok 2024-04-20T17:22:12.836 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:22:13.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:13 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:22:13.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:13 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:22:13.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:13 smithi005 ceph-mon[39562]: pgmap v1064: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.2 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 103 B/s rd, 18 MiB/s wr, 1.22k op/s 2024-04-20T17:22:13.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:13 smithi005 ceph-mon[39562]: 3.e scrub starts 2024-04-20T17:22:13.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:13 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:22:13.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:13 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:22:13.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:13 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:22:13.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:13 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:22:13.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:13 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:22:13.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:13 smithi106 ceph-mon[44423]: pgmap v1064: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.2 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 103 B/s rd, 18 MiB/s wr, 1.22k op/s 2024-04-20T17:22:13.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:13 smithi106 ceph-mon[44423]: 3.e scrub starts 2024-04-20T17:22:13.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:13 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:22:13.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:13 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:22:13.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:13 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:22:13.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:13 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:22:13.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:13 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:22:13.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:13 smithi171 ceph-mon[44189]: pgmap v1064: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 8.2 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 103 B/s rd, 18 MiB/s wr, 1.22k op/s 2024-04-20T17:22:13.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:13 smithi171 ceph-mon[44189]: 3.e scrub starts 2024-04-20T17:22:13.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:13 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:22:13.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:13 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:22:13.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:13 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:22:13.711 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:22:13.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:13.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:22:13.816 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:22:13.872 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:14.247 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:22:14.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:14.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:22:14.351 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:22:14.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:14 smithi005 ceph-mon[39562]: osdmap e392: 12 total, 11 up, 11 in 2024-04-20T17:22:14.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:14 smithi005 ceph-mon[39562]: 3.0 scrub ok 2024-04-20T17:22:14.410 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:14.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:14 smithi106 ceph-mon[44423]: osdmap e392: 12 total, 11 up, 11 in 2024-04-20T17:22:14.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:14 smithi106 ceph-mon[44423]: 3.0 scrub ok 2024-04-20T17:22:14.524 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:22:14.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:14.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:22:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:14 smithi171 ceph-mon[44189]: osdmap e392: 12 total, 11 up, 11 in 2024-04-20T17:22:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:14 smithi171 ceph-mon[44189]: 3.0 scrub ok 2024-04-20T17:22:14.627 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:22:14.690 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:14.746 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:22:14.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:14.754 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:22:14.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:22:14.830 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:22:14.830 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:22:14.855 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:22:14.931 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:15.189 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:22:15.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:15.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:22:15.293 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:22:15.319 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:15.341 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:22:15.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:15.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:22:15.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:15 smithi005 ceph-mon[39562]: 3.e scrub ok 2024-04-20T17:22:15.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:15 smithi005 ceph-mon[39562]: pgmap v1066: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-20T17:22:15.445 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:22:15.470 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:15 smithi106 ceph-mon[44423]: 3.e scrub ok 2024-04-20T17:22:15.470 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:15 smithi106 ceph-mon[44423]: pgmap v1066: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-20T17:22:15.473 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:15.499 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:22:15.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:15.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:22:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:15 smithi171 ceph-mon[44189]: 3.e scrub ok 2024-04-20T17:22:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:15 smithi171 ceph-mon[44189]: pgmap v1066: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 8.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-20T17:22:15.603 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:22:15.635 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:15.657 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:22:15.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:15.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:22:15.761 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:22:15.799 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:15.824 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:22:15.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:15.830 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:22:15.930 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:22:15.958 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:15.976 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:22:15.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:15.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:22:16.080 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:22:16.129 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:16.171 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:22:16.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:16 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4012890620' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:22:16.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:16 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4012890620' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:22:16.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:16 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4012890620' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:22:16.607 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-20T17:22:16.808 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:22:16.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:16.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:22:16.916 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:22:16.956 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:16.987 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:22:16.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:16.994 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:22:17.094 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:22:17.134 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:17.280 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:22:17.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:17.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:22:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:17 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:22:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:17 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:22:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:17 smithi171 ceph-mon[44189]: pgmap v1067: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:22:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:17 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4012890620' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:22:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:17 smithi171 ceph-mon[44189]: osdmap e393: 12 total, 11 up, 11 in 2024-04-20T17:22:17.387 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:22:17.424 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:17.457 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:22:17.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:17 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:22:17.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:17 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:22:17.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:17 smithi005 ceph-mon[39562]: pgmap v1067: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:22:17.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:17 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4012890620' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:22:17.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:17 smithi005 ceph-mon[39562]: osdmap e393: 12 total, 11 up, 11 in 2024-04-20T17:22:17.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:17.467 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:22:17.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:17 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:22:17.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:17 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:22:17.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:17 smithi106 ceph-mon[44423]: pgmap v1067: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:22:17.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:17 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4012890620' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:22:17.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:17 smithi106 ceph-mon[44423]: osdmap e393: 12 total, 11 up, 11 in 2024-04-20T17:22:17.567 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:22:17.597 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:17.628 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:22:17.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:17.630 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:22:17.731 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:22:17.769 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:17.879 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:22:17.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:17.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:22:17.987 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:22:18.023 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:18.052 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:22:18.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:18.058 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:22:18.142 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:22:18.158 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:22:18.199 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:18.227 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:22:18.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:18.232 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:22:18.262 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:22:18.297 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:22:18.328 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:22:18.333 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:22:18.358 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:22:18.374 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:18.389 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:22:18.400 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:22:18.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:18.407 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:22:18.426 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:22:18.447 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:22:18.471 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:22:18.506 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:22:18.508 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:22:18.534 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:18.536 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:22:18.564 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:22:18.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:18.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:22:18.570 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:22:18.571 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:22:18.571 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:22:18.571 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:22:18.571 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:22:18.571 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:22:18.571 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:22:18.571 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:22:18.571 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:22:18.571 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:22:18.571 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:22:18.571 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:22:18.571 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:22:18.668 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:22:18.706 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:18.729 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:22:18.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:18.736 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:22:18.837 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:22:18.879 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:18.913 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:22:18.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:18.917 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:22:19.018 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:22:19.079 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:19.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:19 smithi005 ceph-mon[39562]: pgmap v1069: 17 pgs: 17 active+clean; 7.7 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-20T17:22:19.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:19 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/624804517' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:22:19.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:19.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:19.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:19.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:19.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:19.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:19.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:19 smithi106 ceph-mon[44423]: pgmap v1069: 17 pgs: 17 active+clean; 7.7 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-20T17:22:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:19 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/624804517' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:22:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:19.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:19 smithi171 ceph-mon[44189]: pgmap v1069: 17 pgs: 17 active+clean; 7.7 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-20T17:22:19.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:19 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/624804517' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:22:19.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:19.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:19.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:19.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:19.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:19.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:19.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:19.640 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:22:20.061 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:22:20.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:20.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:22:20.170 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:22:20.218 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:20.248 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:22:20.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:20.253 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:22:20.354 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:22:20.384 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:20.384 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:20.384 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:20.384 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:20.386 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:20.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:21.461 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:22:21.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:21 smithi005 ceph-mon[39562]: pgmap v1070: 17 pgs: 17 active+clean; 7.7 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s 2024-04-20T17:22:21.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:21.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:22:21.494 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:22:21.495 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:22:21.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:21 smithi106 ceph-mon[44423]: pgmap v1070: 17 pgs: 17 active+clean; 7.7 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s 2024-04-20T17:22:21.571 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:22:21.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:21 smithi171 ceph-mon[44189]: pgmap v1070: 17 pgs: 17 active+clean; 7.7 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s 2024-04-20T17:22:21.614 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:21.897 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:22:21.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:21.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:22:21.968 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:22:22.001 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:22:22.031 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:22.053 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:22:22.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:22.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:22:22.158 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:22:22.186 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:22.206 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:22:22.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:22.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:22:22.314 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:22:22.360 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:22.391 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:22:22.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:22.401 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:22:22.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:22 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:22:22.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:22 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:22:22.501 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:22:22.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:22 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:22:22.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:22 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:22:22.560 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:22.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:22 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:22:22.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:22 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:22:22.615 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:22:22.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:22.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:22:22.722 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:22:22.756 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:22.788 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:22:22.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:22.793 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:22:22.807 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:22:22.808 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:22:22.808 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:22:22.808 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:22:22.808 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:22:22.895 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:22:22.971 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:23.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:23 smithi005 ceph-mon[39562]: pgmap v1071: 17 pgs: 17 active+clean; 7.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-20T17:22:23.497 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:22:23.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:23 smithi106 ceph-mon[44423]: pgmap v1071: 17 pgs: 17 active+clean; 7.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-20T17:22:23.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:23 smithi171 ceph-mon[44189]: pgmap v1071: 17 pgs: 17 active+clean; 7.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-20T17:22:23.925 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:22:23.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:23.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:22:24.038 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:22:24.092 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:24.185 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:22:24.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:24.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:22:24.292 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:22:24.354 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:24.441 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:22:24.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:24.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:22:24.451 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:24 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:22:24.451 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:24 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:22:24.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:24 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:22:24.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:24 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:22:24.547 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:22:24.578 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:24 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:22:24.578 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:24 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:22:24.580 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:24.621 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:22:24.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:24.624 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:22:24.631 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2024-04-20T17:22:24.631 INFO:tasks.cephadm.osd.7:Restarting osd.7 (starting--it wasn't running)... 2024-04-20T17:22:24.632 DEBUG:teuthology.orchestra.run.smithi106:osd.7> sudo journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.7.service 2024-04-20T17:22:24.634 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl start ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.7 2024-04-20T17:22:24.663 INFO:journalctl@ceph.osd.7.smithi106.stdout:-- Logs begin at Sat 2024-04-20 16:47:44 UTC. -- 2024-04-20T17:22:24.725 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:22:24.788 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:24.954 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:22:24.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:24.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:22:24.993 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-20T17:22:25.004 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:22:25.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:22:24 smithi106 systemd[1]: Starting Ceph osd.7 for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:22:25.067 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:22:25.103 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:25.127 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:22:25.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:25.131 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:22:25.232 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:22:25.248 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:22:25.261 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:25.298 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:22:25.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:25.304 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:22:25.325 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:25 smithi005 ceph-mon[39562]: pgmap v1072: 17 pgs: 17 active+clean; 7.3 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-20T17:22:25.326 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:25 smithi005 ceph-mon[39562]: osdmap e394: 12 total, 11 up, 11 in 2024-04-20T17:22:25.326 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:25 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2074003348' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:22:25.404 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:22:25.440 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:25 smithi171 ceph-mon[44189]: pgmap v1072: 17 pgs: 17 active+clean; 7.3 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-20T17:22:25.440 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:25 smithi171 ceph-mon[44189]: osdmap e394: 12 total, 11 up, 11 in 2024-04-20T17:22:25.440 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:25 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2074003348' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:22:25.443 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:25.469 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:22:25.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:25.472 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:22:25.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:25 smithi106 ceph-mon[44423]: pgmap v1072: 17 pgs: 17 active+clean; 7.3 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-20T17:22:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:25 smithi106 ceph-mon[44423]: osdmap e394: 12 total, 11 up, 11 in 2024-04-20T17:22:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:25 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2074003348' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:22:25.573 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:22:25.619 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:26.200 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:22:25 smithi106 bash[118936]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T17:22:26.200 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:22:25 smithi106 bash[118936]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-20T17:22:26.200 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:22:25 smithi106 bash[118936]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-20T17:22:26.200 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:22:25 smithi106 bash[118936]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-20T17:22:26.200 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:22:25 smithi106 bash[118936]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-20T17:22:26.200 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:22:25 smithi106 bash[118936]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-04-20T17:22:26.201 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:22:25 smithi106 bash[118936]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-20T17:22:26.201 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:22:25 smithi106 bash[118936]: --> ceph-volume raw activate successful for osd ID: 7 2024-04-20T17:22:26.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:26 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:22:26.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:26 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2074003348' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:22:26.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:26 smithi005 ceph-mon[39562]: osdmap e395: 12 total, 11 up, 11 in 2024-04-20T17:22:26.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:26 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:22:26.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:26 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2074003348' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:22:26.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:26 smithi106 ceph-mon[44423]: osdmap e395: 12 total, 11 up, 11 in 2024-04-20T17:22:26.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:26 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:22:26.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:26 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2074003348' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:22:26.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:26 smithi171 ceph-mon[44189]: osdmap e395: 12 total, 11 up, 11 in 2024-04-20T17:22:27.361 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-20T17:22:27.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:27 smithi005 ceph-mon[39562]: pgmap v1075: 17 pgs: 17 active+clean; 7.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:22:27.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:27 smithi106 ceph-mon[44423]: pgmap v1075: 17 pgs: 17 active+clean; 7.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:22:27.511 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:22:27 smithi106 bash[119185]: 5d305b184f3834d759df9ebb95c4bb15657f9d8ff0b28f598fc8d2adbe17ecc4 2024-04-20T17:22:27.511 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:22:27 smithi106 systemd[1]: Started Ceph osd.7 for 13c6e76c-ff37-11ee-bc93-c7b262605968. 2024-04-20T17:22:27.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:27 smithi171 ceph-mon[44189]: pgmap v1075: 17 pgs: 17 active+clean; 7.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:22:27.705 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:22:28.046 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:22:28.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:28.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:22:28.105 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:22:28.158 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:22:28.213 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:28.261 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:22:27 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[119199]: 2024-04-20T17:22:27.810+0000 7faee5e747c0 -1 Falling back to public interface 2024-04-20T17:22:28.937 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:22:28.981 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:22:29.023 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:22:29.058 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:22:29.084 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:22:29.122 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:22:29.152 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:22:29.186 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:22:29.217 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:22:29.252 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:22:29.277 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:22:29.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:29.283 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:29 smithi005 ceph-mon[39562]: pgmap v1076: 17 pgs: 17 active+clean; 7.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.37k op/s 2024-04-20T17:22:29.283 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:29 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2173834118' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:22:29.283 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:29.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:22:29.290 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:22:29.290 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:22:29.290 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:22:29.290 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:22:29.290 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:22:29.290 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:22:29.290 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:22:29.290 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:22:29.291 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:22:29.291 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:22:29.291 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:22:29.291 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:22:29.291 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:22:29.385 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:22:29.414 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:29 smithi106 ceph-mon[44423]: pgmap v1076: 17 pgs: 17 active+clean; 7.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.37k op/s 2024-04-20T17:22:29.415 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:29 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2173834118' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:22:29.415 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:29.418 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:29.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:29.455 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:22:29.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:22:29.556 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:22:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:29 smithi171 ceph-mon[44189]: pgmap v1076: 17 pgs: 17 active+clean; 7.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.37k op/s 2024-04-20T17:22:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:29 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2173834118' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:22:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:29.589 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:29.612 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:22:29.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:29.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:22:29.716 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:22:29.761 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:29.762 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-04-20T17:22:29.762 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-04-20T17:22:29.762 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-04-20T17:22:29.762 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-04-20T17:22:29.763 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-04-20T17:22:29.763 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-04-20T17:22:29.763 INFO:teuthology.orchestra.run.smithi106.stderr:config diff 2024-04-20T17:22:29.763 INFO:teuthology.orchestra.run.smithi106.stderr:config diff get 2024-04-20T17:22:29.763 INFO:teuthology.orchestra.run.smithi106.stderr:config get 2024-04-20T17:22:29.763 INFO:teuthology.orchestra.run.smithi106.stderr:config help [] 2024-04-20T17:22:29.763 INFO:teuthology.orchestra.run.smithi106.stderr:config set ... 2024-04-20T17:22:29.763 INFO:teuthology.orchestra.run.smithi106.stderr:admin_socket: invalid command 2024-04-20T17:22:29.958 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:22:30.030 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:22:30.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:30.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:22:30.143 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:22:30.179 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:30.457 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:30 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:22:30.457 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:30 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:22:30.457 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.457 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.457 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.457 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.457 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.457 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.458 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.458 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.458 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:30 smithi106 ceph-mon[44423]: osdmap e396: 12 total, 11 up, 11 in 2024-04-20T17:22:30.458 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.458 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:30 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:22:30.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:30 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:22:30.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:30 smithi005 ceph-mon[39562]: osdmap e396: 12 total, 11 up, 11 in 2024-04-20T17:22:30.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.487 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-20T17:22:30.488 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-04-20T17:22:30.537 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:22:30.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:30.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:22:30.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:30 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:22:30.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:30 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:22:30.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:30 smithi171 ceph-mon[44189]: osdmap e396: 12 total, 11 up, 11 in 2024-04-20T17:22:30.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:30.598 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:22:30.598 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:22:30.639 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:22:30.680 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:30.701 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:22:30.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:30.708 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:22:30.761 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:22:30 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[119199]: 2024-04-20T17:22:30.627+0000 7faee5e747c0 -1 osd.7 364 log_to_monitors true 2024-04-20T17:22:30.808 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:22:30.868 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:30.904 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:22:30.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:30.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:22:31.009 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:22:31.039 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:31.074 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:22:31.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:31.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:22:31.180 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:22:31.187 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:22:31.209 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:31.218 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:22:31.218 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:22:31.229 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:22:31.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:31.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:22:31.333 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:22:31.379 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:31.507 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:31 smithi005 ceph-mon[39562]: pgmap v1078: 17 pgs: 17 active+clean; 6.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-20T17:22:31.507 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:31 smithi005 ceph-mon[39562]: from='osd.7 v1:172.21.15.106:6812/1997086583' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-20T17:22:31.507 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:31 smithi005 ceph-mon[39562]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-20T17:22:31.510 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:22:31 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[119199]: 2024-04-20T17:22:31.256+0000 7faed32c4700 -1 osd.7 364 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T17:22:31.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:31 smithi106 ceph-mon[44423]: pgmap v1078: 17 pgs: 17 active+clean; 6.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-20T17:22:31.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:31 smithi106 ceph-mon[44423]: from='osd.7 v1:172.21.15.106:6812/1997086583' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-20T17:22:31.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:31 smithi106 ceph-mon[44423]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-20T17:22:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:31 smithi171 ceph-mon[44189]: pgmap v1078: 17 pgs: 17 active+clean; 6.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-20T17:22:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:31 smithi171 ceph-mon[44189]: from='osd.7 v1:172.21.15.106:6812/1997086583' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-20T17:22:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:31 smithi171 ceph-mon[44189]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-20T17:22:32.109 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:22:32.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:32.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:22:32.215 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:22:32.253 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:32.285 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:22:32.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:32.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:22:32.390 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:22:32.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:32 smithi005 ceph-mon[39562]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-20T17:22:32.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:32 smithi005 ceph-mon[39562]: osdmap e397: 12 total, 11 up, 11 in 2024-04-20T17:22:32.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:32 smithi005 ceph-mon[39562]: from='osd.7 v1:172.21.15.106:6812/1997086583' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:22:32.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:32 smithi005 ceph-mon[39562]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:22:32.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-20T17:22:32.449 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:32.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:32 smithi106 ceph-mon[44423]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-20T17:22:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:32 smithi106 ceph-mon[44423]: osdmap e397: 12 total, 11 up, 11 in 2024-04-20T17:22:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:32 smithi106 ceph-mon[44423]: from='osd.7 v1:172.21.15.106:6812/1997086583' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:22:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:32 smithi106 ceph-mon[44423]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:22:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-20T17:22:32.582 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:22:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:32 smithi171 ceph-mon[44189]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-20T17:22:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:32 smithi171 ceph-mon[44189]: osdmap e397: 12 total, 11 up, 11 in 2024-04-20T17:22:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:32 smithi171 ceph-mon[44189]: from='osd.7 v1:172.21.15.106:6812/1997086583' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:22:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:32 smithi171 ceph-mon[44189]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:22:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-20T17:22:32.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:32.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:22:32.599 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:22:32.688 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:22:32.739 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:33.115 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:22:33.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:33.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:22:33.161 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-20T17:22:33.224 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:22:33.259 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:33.283 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:22:33.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:33.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:22:33.382 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:33 smithi005 ceph-mon[39562]: pgmap v1080: 17 pgs: 17 active+clean; 6.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.31k op/s 2024-04-20T17:22:33.382 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:33 smithi005 ceph-mon[39562]: osd.7 v1:172.21.15.106:6812/1997086583 boot 2024-04-20T17:22:33.382 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:33 smithi005 ceph-mon[39562]: osdmap e398: 12 total, 12 up, 11 in 2024-04-20T17:22:33.383 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:33 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-20T17:22:33.388 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:22:33.420 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:33.448 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:22:33.454 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:33 smithi106 ceph-mon[44423]: pgmap v1080: 17 pgs: 17 active+clean; 6.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.31k op/s 2024-04-20T17:22:33.454 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:33 smithi106 ceph-mon[44423]: osd.7 v1:172.21.15.106:6812/1997086583 boot 2024-04-20T17:22:33.454 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:33 smithi106 ceph-mon[44423]: osdmap e398: 12 total, 12 up, 11 in 2024-04-20T17:22:33.454 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:33 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-20T17:22:33.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:33.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:22:33.522 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:22:33.556 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:22:33.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:33 smithi171 ceph-mon[44189]: pgmap v1080: 17 pgs: 17 active+clean; 6.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.31k op/s 2024-04-20T17:22:33.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:33 smithi171 ceph-mon[44189]: osd.7 v1:172.21.15.106:6812/1997086583 boot 2024-04-20T17:22:33.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:33 smithi171 ceph-mon[44189]: osdmap e398: 12 total, 12 up, 11 in 2024-04-20T17:22:33.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:33 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-20T17:22:33.586 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:33.620 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:22:33.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:33.625 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:22:33.725 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:22:33.776 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:33.824 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:22:33.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:33.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:22:33.866 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:22:33.933 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:22:33.974 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:34.030 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:22:34.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:34.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:22:34.140 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:22:34.199 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:34.234 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:22:34.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:34.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:22:34.300 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:22:34.324 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:34.324 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:34 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:22:34.324 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:34 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:22:34.324 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:34 smithi005 ceph-mon[39562]: osdmap e399: 12 total, 12 up, 11 in 2024-04-20T17:22:34.324 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:34 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3310619604' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:22:34.324 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:34 smithi005 ceph-mon[39562]: from='client.25873 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:22:34.341 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:22:34.375 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:34.400 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:22:34.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:34.407 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:22:34.508 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:22:34.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:34.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:34 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:22:34.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:34 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:22:34.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:34 smithi106 ceph-mon[44423]: osdmap e399: 12 total, 12 up, 11 in 2024-04-20T17:22:34.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:34 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3310619604' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:22:34.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:34 smithi106 ceph-mon[44423]: from='client.25873 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:22:34.537 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:34.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:34.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:34 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:22:34.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:34 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:22:34.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:34 smithi171 ceph-mon[44189]: osdmap e399: 12 total, 12 up, 11 in 2024-04-20T17:22:34.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:34 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3310619604' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:22:34.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:34 smithi171 ceph-mon[44189]: from='client.25873 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:22:35.001 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:22:35.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:35.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:22:35.108 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:22:35.142 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:35.249 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:22:35.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:35.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:22:35.353 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:22:35.382 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:35 smithi005 ceph-mon[39562]: pgmap v1083: 17 pgs: 17 active+clean; 6.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-20T17:22:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:35 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:22:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:35 smithi005 ceph-mon[39562]: from='client.25873 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:22:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:35 smithi005 ceph-mon[39562]: osdmap e400: 12 total, 12 up, 11 in 2024-04-20T17:22:35.488 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-20T17:22:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:35 smithi106 ceph-mon[44423]: pgmap v1083: 17 pgs: 17 active+clean; 6.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-20T17:22:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:35 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:22:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:35 smithi106 ceph-mon[44423]: from='client.25873 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:22:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:35 smithi106 ceph-mon[44423]: osdmap e400: 12 total, 12 up, 11 in 2024-04-20T17:22:35.555 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:22:35.555 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:22:35.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:35 smithi171 ceph-mon[44189]: pgmap v1083: 17 pgs: 17 active+clean; 6.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-20T17:22:35.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:35 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:22:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:35 smithi171 ceph-mon[44189]: from='client.25873 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:22:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:35 smithi171 ceph-mon[44189]: osdmap e400: 12 total, 12 up, 11 in 2024-04-20T17:22:35.590 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:22:35.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:35.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:22:35.696 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:22:35.724 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:35.829 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:22:35.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:35.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:22:35.933 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:22:35.972 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:36.004 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:22:36.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:36.010 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:22:36.111 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:22:36.157 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:36.266 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:22:36.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:36.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:22:36.379 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-20T17:22:36.411 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:22:36.456 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:36.465 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:22:36.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:36.598 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:22:36.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:22:36.655 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:22:36.693 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:22:36.699 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:22:36.732 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:22:36.736 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:36.754 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:22:36.764 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:22:36.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:36.768 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:22:36.777 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:22:36.811 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:22:36.846 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:22:36.869 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:22:36.878 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:22:36.913 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:22:36.934 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:36.955 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:22:36.968 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:22:36.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:36.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:22:37.000 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:22:37.035 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:22:37.039 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:22:37.039 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:22:37.039 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:22:37.039 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:22:37.039 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:22:37.039 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:22:37.039 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:22:37.040 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-04-20T17:22:37.040 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:22:37.040 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:22:37.040 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:22:37.040 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:22:37.077 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:22:37.138 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:37 smithi005 ceph-mon[39562]: pgmap v1085: 17 pgs: 17 active+clean; 6.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-20T17:22:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:37 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/219862941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:22:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:37.277 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:22:37.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:37.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:22:37.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:37 smithi171 ceph-mon[44189]: pgmap v1085: 17 pgs: 17 active+clean; 6.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-20T17:22:37.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:37 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/219862941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:22:37.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:37.385 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:22:37.423 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:37.449 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:22:37.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:37.456 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:22:37.473 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:37 smithi106 ceph-mon[44423]: pgmap v1085: 17 pgs: 17 active+clean; 6.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-20T17:22:37.473 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:37 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/219862941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:22:37.473 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:37.557 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:22:37.611 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:37.831 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:22:37.837 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:22:37.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:37.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:22:37.942 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:22:37.992 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:38.014 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:22:38.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:38.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:22:38.062 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:22:38.119 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.119 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.119 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.119 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.119 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.119 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.119 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.120 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.120 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.120 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.122 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:22:38.179 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.179 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.179 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.179 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.179 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.179 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.179 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.179 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.179 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.179 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.182 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:38.220 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:22:38.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:38.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:22:38.325 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:22:38.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:38.377 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:38.411 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:22:38.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:38.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:22:38.484 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:22:38.519 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:22:38.557 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:38.917 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:22:38.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:38.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:22:39.020 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:22:39.076 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:39.104 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:22:39.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:39.113 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:22:39.214 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:22:39.251 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:39.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:39 smithi106 ceph-mon[44423]: pgmap v1086: 17 pgs: 17 active+clean; 6.3 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.33k op/s 2024-04-20T17:22:39.291 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:22:39.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:39.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:22:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:39 smithi171 ceph-mon[44189]: pgmap v1086: 17 pgs: 17 active+clean; 6.3 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.33k op/s 2024-04-20T17:22:39.398 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:22:39.418 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:39 smithi005 ceph-mon[39562]: pgmap v1086: 17 pgs: 17 active+clean; 6.3 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.33k op/s 2024-04-20T17:22:39.435 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:39.569 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:22:39.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:39.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:22:39.674 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:22:39.723 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:39.752 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:22:39.753 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:22:39.753 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-04-20T17:22:39.753 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:22:39.806 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:22:39.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:39.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:22:39.865 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:22:39.911 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:22:39.942 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:40.450 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:22:40.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:40.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:22:40.496 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:22:40.496 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-04-20T17:22:40.496 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-20T17:22:40.496 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-20T17:22:40.497 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-04-20T17:22:40.502 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:22:40.555 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:22:40.586 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:40.742 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:22:40.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:40.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:22:40.846 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:22:40.879 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:40.906 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:22:40.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:40.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:22:41.010 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:22:41.065 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:41.066 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:22:41.066 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:22:41.066 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-04-20T17:22:41.066 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-04-20T17:22:41.066 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:22:41.096 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:22:41.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:41.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:22:41.202 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:22:41.226 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:22:41.246 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:41.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:41 smithi005 ceph-mon[39562]: pgmap v1087: 17 pgs: 17 active+clean; 6.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-20T17:22:41.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:41 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4222095647' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:22:41.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:41 smithi005 ceph-mon[39562]: from='client.25937 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:22:41.481 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:41 smithi106 ceph-mon[44423]: pgmap v1087: 17 pgs: 17 active+clean; 6.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-20T17:22:41.482 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:41 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4222095647' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:22:41.482 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:41 smithi106 ceph-mon[44423]: from='client.25937 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:22:41.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:41 smithi171 ceph-mon[44189]: pgmap v1087: 17 pgs: 17 active+clean; 6.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-20T17:22:41.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:41 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4222095647' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:22:41.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:41 smithi171 ceph-mon[44189]: from='client.25937 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:22:41.613 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:22:41.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:41.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:22:41.734 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:22:41.770 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:41.792 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:22:41.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:41.797 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:22:41.818 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-20T17:22:41.898 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:22:41.930 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:41.963 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:22:41.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:41.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:22:42.068 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:22:42.108 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:42.150 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:22:42.156 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:22:42.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:42.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:22:42.259 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:22:42.293 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:42.328 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:22:42.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:42.331 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:22:42.432 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:22:42.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:42 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:22:42.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:42 smithi005 ceph-mon[39562]: from='client.25937 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:22:42.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:42 smithi005 ceph-mon[39562]: osdmap e401: 12 total, 12 up, 11 in 2024-04-20T17:22:42.465 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:42.480 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:42 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:22:42.480 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:42 smithi106 ceph-mon[44423]: from='client.25937 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:22:42.480 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:42 smithi106 ceph-mon[44423]: osdmap e401: 12 total, 12 up, 11 in 2024-04-20T17:22:42.487 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:22:42.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:42 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:22:42.488 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:42 smithi171 ceph-mon[44189]: from='client.25937 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:22:42.489 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:42 smithi171 ceph-mon[44189]: osdmap e401: 12 total, 12 up, 11 in 2024-04-20T17:22:42.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:42.491 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:22:42.594 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:22:42.625 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:42.664 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:22:42.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:42.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:22:42.701 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:22:42.770 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:22:42.809 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:43.244 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:43 smithi005 ceph-mon[39562]: pgmap v1089: 17 pgs: 17 active+clean; 6.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:22:43.448 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:43 smithi106 ceph-mon[44423]: pgmap v1089: 17 pgs: 17 active+clean; 6.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:22:43.468 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:22:43.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:43.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:22:43.490 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:22:43.524 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:22:43.558 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:22:43.579 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:22:43.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:43 smithi171 ceph-mon[44189]: pgmap v1089: 17 pgs: 17 active+clean; 6.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:22:43.595 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:22:43.629 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:22:43.647 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:43.658 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:22:43.703 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:22:43.739 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:22:43.773 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:22:43.804 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:22:43.830 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:22:43.858 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:22:43.870 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:22:43.870 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:22:43.870 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:22:43.870 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:22:43.870 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:22:43.870 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:22:43.870 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:22:43.870 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:22:43.871 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:22:43.871 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:22:43.871 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:22:43.871 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:22:44.114 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:22:44.115 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:22:44.433 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:22:44.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:44.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:22:44.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:44 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1688488458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:22:44.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:44 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:44.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:44 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:44.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:44 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:44.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:44 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:44.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:44 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:44.539 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:22:44.573 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:44.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:44 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1688488458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:22:44.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:44 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:44.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:44 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:44.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:44 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:44.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:44 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:44.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:44 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:44.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:44 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1688488458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:22:44.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:44 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:44.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:44 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:44.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:44 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:44.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:44 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:44.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:44 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:44.741 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:22:45.418 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:45.418 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:45.418 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:45.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:45.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:45.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:45.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:45.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:45 smithi005 ceph-mon[39562]: pgmap v1090: 17 pgs: 17 active+clean; 5.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:22:45.482 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:45.482 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:45.482 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:45.482 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:45.482 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:45.483 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:45.483 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:45.483 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:45 smithi106 ceph-mon[44423]: pgmap v1090: 17 pgs: 17 active+clean; 5.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:22:45.515 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-20T17:22:45.524 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:22:45.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:45.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:22:45.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:45.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:45.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:45.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:45.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:45.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:45 smithi171 ceph-mon[44189]: pgmap v1090: 17 pgs: 17 active+clean; 5.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:22:45.631 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:22:45.662 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:45.686 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:22:45.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:45.690 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:22:45.738 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:22:45.791 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:22:45.799 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:22:45.799 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-20T17:22:45.799 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:22:45.823 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:45.843 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:22:45.848 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:22:45.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:45.850 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:22:45.951 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:22:45.990 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:46.013 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:22:46.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:46.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:22:46.121 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:22:46.161 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:46.185 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:22:46.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:46.190 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:22:46.291 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:22:46.338 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:46.367 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:22:46.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:46.370 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:22:46.471 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:22:46.533 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:46.806 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:22:46.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:46.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:22:46.916 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:22:46.949 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:47.092 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:47 smithi005 ceph-mon[39562]: pgmap v1091: 17 pgs: 17 active+clean; 5.7 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-20T17:22:47.163 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:22:47.257 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:47 smithi106 ceph-mon[44423]: pgmap v1091: 17 pgs: 17 active+clean; 5.7 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-20T17:22:47.312 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:22:47.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:47.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:22:47.416 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:22:47.455 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:47 smithi171 ceph-mon[44189]: pgmap v1091: 17 pgs: 17 active+clean; 5.7 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-20T17:22:47.457 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:47.488 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:22:47.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:47.493 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:22:47.594 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:22:47.640 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:47.739 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:22:47.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:47.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:22:47.846 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:22:47.900 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:22:47.900 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:22:47.910 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:48.040 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:22:48.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:48.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:22:48.152 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:22:48.190 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:48 smithi106 ceph-mon[44423]: 3.5 scrub starts 2024-04-20T17:22:48.193 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:48.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:48 smithi005 ceph-mon[39562]: 3.5 scrub starts 2024-04-20T17:22:48.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:48 smithi171 ceph-mon[44189]: 3.5 scrub starts 2024-04-20T17:22:48.650 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:22:48.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:48.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:22:48.676 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-20T17:22:48.759 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:22:48.797 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:48.829 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:22:48.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:48.834 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:22:48.936 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:22:48.985 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:49.039 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:22:49.171 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:22:49.173 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:22:49.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:49.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:22:49.224 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:22:49.279 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:22:49.326 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:49.349 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:22:49.354 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:49 smithi171 ceph-mon[44189]: pgmap v1092: 17 pgs: 17 active+clean; 5.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-20T17:22:49.354 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:49 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1216005318' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:22:49.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:49.356 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:22:49.457 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:22:49.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:49 smithi005 ceph-mon[39562]: pgmap v1092: 17 pgs: 17 active+clean; 5.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-20T17:22:49.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:49 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1216005318' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:22:49.507 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:49.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:49 smithi106 ceph-mon[44423]: pgmap v1092: 17 pgs: 17 active+clean; 5.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-20T17:22:49.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:49 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1216005318' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:22:49.539 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:22:49.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:49.543 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:22:49.644 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:22:49.682 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:49.714 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:22:49.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:49.722 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:22:49.823 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:22:49.858 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:49.882 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:22:49.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:49.890 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:22:49.991 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:22:50.041 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:50.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:50 smithi106 ceph-mon[44423]: 3.5 scrub ok 2024-04-20T17:22:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:50 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:22:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:50 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:22:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:50 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1216005318' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:22:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:50 smithi106 ceph-mon[44423]: osdmap e402: 12 total, 12 up, 11 in 2024-04-20T17:22:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:50 smithi005 ceph-mon[39562]: 3.5 scrub ok 2024-04-20T17:22:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:50 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:22:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:50 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:22:50.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:50 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1216005318' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:22:50.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:50 smithi005 ceph-mon[39562]: osdmap e402: 12 total, 12 up, 11 in 2024-04-20T17:22:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:50 smithi171 ceph-mon[44189]: 3.5 scrub ok 2024-04-20T17:22:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:50 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:22:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:50 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:22:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:50 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1216005318' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:22:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:50 smithi171 ceph-mon[44189]: osdmap e402: 12 total, 12 up, 11 in 2024-04-20T17:22:51.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:51 smithi005 ceph-mon[39562]: pgmap v1094: 17 pgs: 17 active+clean; 5.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s 2024-04-20T17:22:51.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:51 smithi005 ceph-mon[39562]: 3.1 scrub starts 2024-04-20T17:22:51.414 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:22:51.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:51.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:22:51.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:51 smithi106 ceph-mon[44423]: pgmap v1094: 17 pgs: 17 active+clean; 5.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s 2024-04-20T17:22:51.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:51 smithi106 ceph-mon[44423]: 3.1 scrub starts 2024-04-20T17:22:51.521 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:22:51.564 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:51.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:51 smithi171 ceph-mon[44189]: pgmap v1094: 17 pgs: 17 active+clean; 5.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s 2024-04-20T17:22:51.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:51 smithi171 ceph-mon[44189]: 3.1 scrub starts 2024-04-20T17:22:51.828 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11] out_osds: [7] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 8, 9, 10, 11, 2, 7] 2024-04-20T17:22:51.828 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:22:51.828 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-20T17:22:51.828 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-20T17:22:52.420 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:22:52.421 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:52 smithi106 ceph-mon[44423]: 3.1 scrub ok 2024-04-20T17:22:52.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:52.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:22:52.451 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:22:52.451 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:22:52.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:52 smithi005 ceph-mon[39562]: 3.1 scrub ok 2024-04-20T17:22:52.524 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:22:52.556 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:52.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:52 smithi171 ceph-mon[44189]: 3.1 scrub ok 2024-04-20T17:22:53.124 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:53 smithi005 ceph-mon[39562]: pgmap v1095: 17 pgs: 17 active+clean; 5.3 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-20T17:22:53.138 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:22:53.213 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:22:53.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:53.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:22:53.217 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:22:53.241 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-20T17:22:53.253 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:22:53.259 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:22:53.287 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:22:53.318 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:22:53.325 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:22:53.354 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:22:53.359 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:53.387 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:22:53.388 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:22:53.390 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:53 smithi171 ceph-mon[44189]: pgmap v1095: 17 pgs: 17 active+clean; 5.3 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-20T17:22:53.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:53.391 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:22:53.428 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:22:53.471 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:22:53.492 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:22:53.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:53 smithi106 ceph-mon[44423]: pgmap v1095: 17 pgs: 17 active+clean; 5.3 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-20T17:22:53.523 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:53.524 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:22:53.545 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:22:53.548 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:22:53.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:53.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:22:53.596 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:22:53.613 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:22:53.624 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:22:53.636 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:22:53.637 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:22:53.637 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:22:53.637 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:22:53.637 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:22:53.637 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:22:53.637 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:22:53.638 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:22:53.638 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:22:53.638 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:22:53.638 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:22:53.638 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:22:53.654 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:22:53.706 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:54.142 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:54 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2135727427' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:22:54.142 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.142 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.142 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.142 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.142 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.142 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.143 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.143 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.143 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.143 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:54 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2135727427' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:22:54.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:54 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2135727427' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:22:54.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:54.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:55.235 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:55.235 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:55.235 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:55 smithi106 ceph-mon[44423]: pgmap v1096: 17 pgs: 17 active+clean; 5.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-20T17:22:55.279 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:55.280 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:55.280 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:55 smithi005 ceph-mon[39562]: pgmap v1096: 17 pgs: 17 active+clean; 5.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-20T17:22:55.322 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:22:55.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:55.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:22:55.417 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:22:55.426 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:22:55.479 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:55.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:55.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:22:55.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:55 smithi171 ceph-mon[44189]: pgmap v1096: 17 pgs: 17 active+clean; 5.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-20T17:22:55.591 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:22:55.592 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-04-20T17:22:55.592 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-04-20T17:22:55.592 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:22:55.592 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.592 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:53797 3.d 3:b6be2254:::benchmark_data_smithi171_110765_object53796:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e373)", 2024-04-20T17:22:55.592 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:21:03.956457+0000", 2024-04-20T17:22:55.592 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 111.632977858, 2024-04-20T17:22:55.592 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.052490740000000001, 2024-04-20T17:22:55.592 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:22:55.592 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:22:55.592 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:55.592 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:22:55.592 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:22:55.593 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:22:55.593 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 53797 2024-04-20T17:22:55.593 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.593 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:22:55.593 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.593 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:22:55.593 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.956457+0000", 2024-04-20T17:22:55.593 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.593 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.593 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.593 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:22:55.593 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.956457+0000", 2024-04-20T17:22:55.593 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.593 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.594 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.594 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:22:55.594 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.956458+0000", 2024-04-20T17:22:55.594 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6899999999999996e-07 2024-04-20T17:22:55.594 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.594 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.594 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:22:55.594 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.956491+0000", 2024-04-20T17:22:55.594 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3043999999999997e-05 2024-04-20T17:22:55.594 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.594 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.594 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:22:55.594 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.956498+0000", 2024-04-20T17:22:55.595 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4049999999999997e-06 2024-04-20T17:22:55.595 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.595 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.595 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:22:55.595 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.956503+0000", 2024-04-20T17:22:55.595 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6899999999999997e-06 2024-04-20T17:22:55.595 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.595 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.595 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:22:55.595 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.958493+0000", 2024-04-20T17:22:55.596 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001989631 2024-04-20T17:22:55.596 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.596 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.596 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:22:55.596 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.958636+0000", 2024-04-20T17:22:55.596 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014356699999999999 2024-04-20T17:22:55.596 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.596 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.596 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 9", 2024-04-20T17:22:55.596 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.958791+0000", 2024-04-20T17:22:55.596 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000154592 2024-04-20T17:22:55.596 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.596 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.597 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:22:55.597 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.965345+0000", 2024-04-20T17:22:55.597 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0065539040000000002 2024-04-20T17:22:55.597 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.597 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.597 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:55.597 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:04.008380+0000", 2024-04-20T17:22:55.597 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.043034977000000002 2024-04-20T17:22:55.597 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.597 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.597 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:22:55.597 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:04.008417+0000", 2024-04-20T17:22:55.597 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7557e-05 2024-04-20T17:22:55.598 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.598 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.598 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:22:55.598 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:04.008948+0000", 2024-04-20T17:22:55.598 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00053060399999999995 2024-04-20T17:22:55.598 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.598 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:22:55.598 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.598 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.598 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.598 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:53800 3.d 3:b6019a0d:::benchmark_data_smithi171_110765_object53799:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e373)", 2024-04-20T17:22:55.598 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:21:03.956575+0000", 2024-04-20T17:22:55.598 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 111.63285975700001, 2024-04-20T17:22:55.598 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.054550105000000002, 2024-04-20T17:22:55.599 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:22:55.599 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:22:55.599 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:55.599 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:22:55.599 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:22:55.599 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:22:55.599 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 53800 2024-04-20T17:22:55.599 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.599 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:22:55.599 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.599 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:22:55.599 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.956575+0000", 2024-04-20T17:22:55.599 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.599 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.600 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.600 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:22:55.600 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.956575+0000", 2024-04-20T17:22:55.600 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.600 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.600 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.600 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:22:55.600 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.956576+0000", 2024-04-20T17:22:55.600 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0299999999999998e-07 2024-04-20T17:22:55.600 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.600 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.600 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:22:55.600 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.956699+0000", 2024-04-20T17:22:55.600 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012293499999999999 2024-04-20T17:22:55.601 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.601 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.601 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:22:55.601 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.956709+0000", 2024-04-20T17:22:55.601 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0108999999999999e-05 2024-04-20T17:22:55.601 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.601 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.601 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:22:55.601 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.956717+0000", 2024-04-20T17:22:55.601 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7239999999999992e-06 2024-04-20T17:22:55.601 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.601 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.601 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:22:55.601 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.960498+0000", 2024-04-20T17:22:55.601 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037809380000000002 2024-04-20T17:22:55.601 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.602 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.602 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:22:55.602 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.960641+0000", 2024-04-20T17:22:55.602 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014307 2024-04-20T17:22:55.602 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.602 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.602 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 9", 2024-04-20T17:22:55.602 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.960781+0000", 2024-04-20T17:22:55.602 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013990899999999999 2024-04-20T17:22:55.602 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.602 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.602 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:22:55.602 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.965352+0000", 2024-04-20T17:22:55.602 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045707370000000001 2024-04-20T17:22:55.603 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.603 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.603 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:55.603 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:04.010358+0000", 2024-04-20T17:22:55.603 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.045006127999999999 2024-04-20T17:22:55.603 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.603 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.603 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:22:55.603 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:04.010419+0000", 2024-04-20T17:22:55.603 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1402999999999995e-05 2024-04-20T17:22:55.603 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.603 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.603 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:22:55.603 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:04.011126+0000", 2024-04-20T17:22:55.603 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00070644900000000005 2024-04-20T17:22:55.604 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.604 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:22:55.604 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.604 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.604 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.604 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:53806 3.d 3:b34088a7:::benchmark_data_smithi171_110765_object53805:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e373)", 2024-04-20T17:22:55.604 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:21:03.959348+0000", 2024-04-20T17:22:55.604 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 111.63008726699999, 2024-04-20T17:22:55.604 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.053004308, 2024-04-20T17:22:55.604 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:22:55.604 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:22:55.604 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:55.604 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:22:55.604 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:22:55.604 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:22:55.604 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 53806 2024-04-20T17:22:55.605 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.605 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:22:55.605 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.605 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:22:55.605 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.959348+0000", 2024-04-20T17:22:55.605 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.605 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.605 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.605 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:22:55.605 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.959348+0000", 2024-04-20T17:22:55.605 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.605 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.605 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.605 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:22:55.605 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.959350+0000", 2024-04-20T17:22:55.606 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7039999999999999e-06 2024-04-20T17:22:55.606 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.606 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.606 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:22:55.606 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.959411+0000", 2024-04-20T17:22:55.606 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1217999999999996e-05 2024-04-20T17:22:55.606 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.606 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.606 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:22:55.606 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.959420+0000", 2024-04-20T17:22:55.606 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1379999999999996e-06 2024-04-20T17:22:55.606 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.606 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.606 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:22:55.606 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.959427+0000", 2024-04-20T17:22:55.607 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1130000000000003e-06 2024-04-20T17:22:55.607 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.607 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.607 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:22:55.607 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.963426+0000", 2024-04-20T17:22:55.607 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0039989609999999997 2024-04-20T17:22:55.607 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.607 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.607 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:22:55.607 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.963569+0000", 2024-04-20T17:22:55.607 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014339499999999999 2024-04-20T17:22:55.607 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.607 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.607 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 9", 2024-04-20T17:22:55.607 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.963666+0000", 2024-04-20T17:22:55.608 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6352000000000002e-05 2024-04-20T17:22:55.608 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.608 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.608 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:22:55.608 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:03.970579+0000", 2024-04-20T17:22:55.608 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0069136579999999996 2024-04-20T17:22:55.608 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.608 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.608 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:55.608 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:04.011846+0000", 2024-04-20T17:22:55.609 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.041266290999999997 2024-04-20T17:22:55.609 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.609 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.609 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:22:55.609 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:04.011880+0000", 2024-04-20T17:22:55.609 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4598999999999999e-05 2024-04-20T17:22:55.609 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.609 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.609 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:22:55.609 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:04.012352+0000", 2024-04-20T17:22:55.609 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00047187900000000003 2024-04-20T17:22:55.609 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.609 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:22:55.609 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.610 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.610 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.610 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:76064 3.d 3:b6b4f865:::benchmark_data_smithi171_110765_object76063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e376)", 2024-04-20T17:22:55.610 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:21:14.836144+0000", 2024-04-20T17:22:55.610 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 100.75329115, 2024-04-20T17:22:55.610 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.080439939000000002, 2024-04-20T17:22:55.610 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:22:55.610 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:22:55.610 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:55.610 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:22:55.610 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:22:55.610 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:22:55.610 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 76064 2024-04-20T17:22:55.611 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.611 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:22:55.611 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.611 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:22:55.611 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.836144+0000", 2024-04-20T17:22:55.611 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.611 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.611 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.611 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:22:55.611 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.836144+0000", 2024-04-20T17:22:55.611 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.611 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.611 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.611 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:22:55.612 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.836146+0000", 2024-04-20T17:22:55.612 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8160000000000001e-06 2024-04-20T17:22:55.612 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.612 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.612 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:22:55.612 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.836206+0000", 2024-04-20T17:22:55.612 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0059000000000002e-05 2024-04-20T17:22:55.612 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.612 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.612 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:22:55.612 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.836219+0000", 2024-04-20T17:22:55.612 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3066e-05 2024-04-20T17:22:55.612 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.612 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.613 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:22:55.613 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.836229+0000", 2024-04-20T17:22:55.613 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9070000000000002e-06 2024-04-20T17:22:55.613 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.613 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.613 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:22:55.613 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.838903+0000", 2024-04-20T17:22:55.613 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0026737129999999999 2024-04-20T17:22:55.613 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.613 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.613 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:22:55.613 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.838990+0000", 2024-04-20T17:22:55.614 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7775999999999994e-05 2024-04-20T17:22:55.614 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.614 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.614 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 9", 2024-04-20T17:22:55.614 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.839083+0000", 2024-04-20T17:22:55.614 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2918999999999996e-05 2024-04-20T17:22:55.614 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.614 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.614 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:22:55.614 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.840814+0000", 2024-04-20T17:22:55.614 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017302800000000001 2024-04-20T17:22:55.614 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.614 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.615 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:55.615 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.842878+0000", 2024-04-20T17:22:55.615 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020643010000000002 2024-04-20T17:22:55.615 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.615 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.615 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:22:55.615 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.842910+0000", 2024-04-20T17:22:55.615 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1702000000000002e-05 2024-04-20T17:22:55.616 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.616 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.616 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:22:55.616 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.916584+0000", 2024-04-20T17:22:55.616 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.073674400000000001 2024-04-20T17:22:55.616 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.616 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:22:55.616 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.616 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.616 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.616 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:76070 3.5 3:a971a0ed:::benchmark_data_smithi171_110765_object76069:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e376)", 2024-04-20T17:22:55.616 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:21:14.841177+0000", 2024-04-20T17:22:55.616 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 100.748258343, 2024-04-20T17:22:55.616 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.075482621, 2024-04-20T17:22:55.616 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:22:55.617 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:22:55.617 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:55.617 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:22:55.617 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:22:55.617 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:22:55.617 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 76070 2024-04-20T17:22:55.617 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.617 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:22:55.617 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.617 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:22:55.617 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.841177+0000", 2024-04-20T17:22:55.617 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.617 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.617 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.617 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:22:55.618 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.841177+0000", 2024-04-20T17:22:55.618 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.618 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.618 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.618 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:22:55.618 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.841197+0000", 2024-04-20T17:22:55.618 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0610000000000001e-05 2024-04-20T17:22:55.618 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.618 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.618 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:22:55.618 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.841261+0000", 2024-04-20T17:22:55.618 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3734999999999999e-05 2024-04-20T17:22:55.618 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.619 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.619 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:22:55.619 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.841281+0000", 2024-04-20T17:22:55.619 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9533000000000001e-05 2024-04-20T17:22:55.619 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.619 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.619 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:22:55.619 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.841294+0000", 2024-04-20T17:22:55.619 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3746e-05 2024-04-20T17:22:55.619 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.619 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.619 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:22:55.619 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.841376+0000", 2024-04-20T17:22:55.619 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1998999999999999e-05 2024-04-20T17:22:55.620 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.620 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.620 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:22:55.620 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.841548+0000", 2024-04-20T17:22:55.620 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00017164900000000001 2024-04-20T17:22:55.620 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.620 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.620 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:22:55.620 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.841672+0000", 2024-04-20T17:22:55.620 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000124296 2024-04-20T17:22:55.620 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.620 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.620 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:55.620 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.916568+0000", 2024-04-20T17:22:55.621 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.074895257000000007 2024-04-20T17:22:55.621 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.621 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.621 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:22:55.621 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.916597+0000", 2024-04-20T17:22:55.621 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9692e-05 2024-04-20T17:22:55.621 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.621 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.621 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:22:55.621 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.916635+0000", 2024-04-20T17:22:55.621 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7759000000000002e-05 2024-04-20T17:22:55.621 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.621 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.621 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:22:55.621 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.916659+0000", 2024-04-20T17:22:55.622 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4345000000000002e-05 2024-04-20T17:22:55.622 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.622 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:22:55.622 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.622 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.622 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.622 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.25802.0:76064 3.d e376/368 ondisk, result = 0)", 2024-04-20T17:22:55.622 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:21:14.842812+0000", 2024-04-20T17:22:55.622 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 100.74662302900001, 2024-04-20T17:22:55.622 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.073779119000000004, 2024-04-20T17:22:55.622 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:22:55.623 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:22:55.623 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-04-20T17:22:55.623 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:22:55.623 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.623 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:22:55.623 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.842812+0000", 2024-04-20T17:22:55.623 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.623 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.623 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.623 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:22:55.623 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.842812+0000", 2024-04-20T17:22:55.623 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.623 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.623 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.623 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:22:55.624 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.842813+0000", 2024-04-20T17:22:55.624 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.139e-06 2024-04-20T17:22:55.624 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.624 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.624 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:22:55.624 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.842820+0000", 2024-04-20T17:22:55.624 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4219999999999997e-06 2024-04-20T17:22:55.624 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.624 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.624 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:22:55.624 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.842828+0000", 2024-04-20T17:22:55.625 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6300000000000004e-06 2024-04-20T17:22:55.625 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.625 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.625 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:22:55.625 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.842835+0000", 2024-04-20T17:22:55.625 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2369999999999997e-06 2024-04-20T17:22:55.625 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.625 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.625 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:22:55.625 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.842870+0000", 2024-04-20T17:22:55.625 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5061999999999997e-05 2024-04-20T17:22:55.626 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.626 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.626 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:22:55.626 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.842875+0000", 2024-04-20T17:22:55.626 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0869999999999997e-06 2024-04-20T17:22:55.626 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.626 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.626 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:22:55.627 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.916591+0000", 2024-04-20T17:22:55.627 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.073716541999999996 2024-04-20T17:22:55.627 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.627 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:22:55.627 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.627 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.627 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.627 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:76078 3.5 3:a7301fe7:::benchmark_data_smithi171_110765_object76077:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e376)", 2024-04-20T17:22:55.627 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:21:14.844370+0000", 2024-04-20T17:22:55.628 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 100.74506476099999, 2024-04-20T17:22:55.628 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.082155157000000006, 2024-04-20T17:22:55.628 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:22:55.628 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:22:55.628 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:55.628 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:22:55.629 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:22:55.629 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:22:55.629 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 76078 2024-04-20T17:22:55.629 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.629 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:22:55.629 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.630 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:22:55.630 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.844370+0000", 2024-04-20T17:22:55.630 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.630 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.630 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.630 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:22:55.631 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.844370+0000", 2024-04-20T17:22:55.631 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.631 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.631 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.631 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:22:55.631 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.844372+0000", 2024-04-20T17:22:55.632 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4589999999999999e-06 2024-04-20T17:22:55.632 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.632 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.632 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:22:55.632 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.844456+0000", 2024-04-20T17:22:55.632 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3951999999999999e-05 2024-04-20T17:22:55.632 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.632 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.632 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:22:55.632 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.916593+0000", 2024-04-20T17:22:55.633 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.072136860999999997 2024-04-20T17:22:55.633 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.633 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.633 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:22:55.633 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.916601+0000", 2024-04-20T17:22:55.633 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5820000000000002e-06 2024-04-20T17:22:55.633 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.633 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.633 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:22:55.633 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.916683+0000", 2024-04-20T17:22:55.633 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1392999999999998e-05 2024-04-20T17:22:55.633 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.633 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.633 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:22:55.633 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.916788+0000", 2024-04-20T17:22:55.634 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000105654 2024-04-20T17:22:55.634 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.634 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.634 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:22:55.634 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.916929+0000", 2024-04-20T17:22:55.634 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014039400000000001 2024-04-20T17:22:55.634 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.634 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.634 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:55.634 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.924119+0000", 2024-04-20T17:22:55.634 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0071906490000000003 2024-04-20T17:22:55.634 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.634 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.634 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:22:55.635 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.926423+0000", 2024-04-20T17:22:55.635 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002303238 2024-04-20T17:22:55.635 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.635 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.635 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:22:55.635 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.926487+0000", 2024-04-20T17:22:55.635 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4335e-05 2024-04-20T17:22:55.635 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.635 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.635 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:22:55.635 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.926526+0000", 2024-04-20T17:22:55.635 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8640000000000003e-05 2024-04-20T17:22:55.635 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.635 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:22:55.635 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.636 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.636 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.636 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.25802.0:76070 3.5 e376/356 ondisk, result = 0)", 2024-04-20T17:22:55.636 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:21:14.845687+0000", 2024-04-20T17:22:55.636 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 100.74374850700001, 2024-04-20T17:22:55.636 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.070888386999999997, 2024-04-20T17:22:55.636 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:22:55.636 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:22:55.636 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-04-20T17:22:55.636 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:22:55.636 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.636 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:22:55.636 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.845687+0000", 2024-04-20T17:22:55.636 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.636 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.637 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.637 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:22:55.637 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.845687+0000", 2024-04-20T17:22:55.637 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.637 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.637 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.637 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:22:55.637 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.845688+0000", 2024-04-20T17:22:55.637 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5910000000000001e-06 2024-04-20T17:22:55.637 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.637 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.637 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:22:55.638 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.845693+0000", 2024-04-20T17:22:55.638 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.865e-06 2024-04-20T17:22:55.638 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.638 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.638 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:22:55.638 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.916473+0000", 2024-04-20T17:22:55.638 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.070779971999999997 2024-04-20T17:22:55.638 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.638 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.638 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:22:55.638 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.916485+0000", 2024-04-20T17:22:55.639 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1643e-05 2024-04-20T17:22:55.639 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.639 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.639 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:22:55.639 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.916554+0000", 2024-04-20T17:22:55.639 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9078000000000005e-05 2024-04-20T17:22:55.639 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.639 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.639 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:22:55.639 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.916562+0000", 2024-04-20T17:22:55.639 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0900000000000005e-06 2024-04-20T17:22:55.639 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.639 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.639 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:22:55.639 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.916575+0000", 2024-04-20T17:22:55.640 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3148e-05 2024-04-20T17:22:55.640 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.640 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:22:55.640 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.640 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.640 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.640 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:126473 3.c 3:31907acb:::benchmark_data_smithi171_110765_object126472:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-04-20T17:22:55.640 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:21:48.119484+0000", 2024-04-20T17:22:55.640 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 67.469951159999994, 2024-04-20T17:22:55.640 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.067303548000000005, 2024-04-20T17:22:55.640 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:22:55.640 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:22:55.640 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:55.640 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:22:55.641 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:22:55.641 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:22:55.641 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 126473 2024-04-20T17:22:55.641 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.641 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:22:55.641 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.641 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:22:55.641 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.119484+0000", 2024-04-20T17:22:55.641 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.641 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.641 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.641 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:22:55.642 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.119484+0000", 2024-04-20T17:22:55.642 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.642 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.642 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.642 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:22:55.642 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.119486+0000", 2024-04-20T17:22:55.642 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.885e-06 2024-04-20T17:22:55.642 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.642 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.642 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:22:55.642 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.119552+0000", 2024-04-20T17:22:55.642 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6544000000000005e-05 2024-04-20T17:22:55.642 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.642 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.643 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:22:55.643 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.119562+0000", 2024-04-20T17:22:55.643 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1849999999999993e-06 2024-04-20T17:22:55.643 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.643 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.643 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:22:55.643 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.119567+0000", 2024-04-20T17:22:55.643 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7620000000000001e-06 2024-04-20T17:22:55.644 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.644 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.644 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:22:55.644 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.119599+0000", 2024-04-20T17:22:55.644 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1952999999999997e-05 2024-04-20T17:22:55.644 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.644 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.644 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:22:55.645 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.119689+0000", 2024-04-20T17:22:55.645 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9598000000000005e-05 2024-04-20T17:22:55.645 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.645 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.645 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:22:55.645 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.119782+0000", 2024-04-20T17:22:55.645 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2859e-05 2024-04-20T17:22:55.645 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.645 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.645 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:22:55.646 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.122385+0000", 2024-04-20T17:22:55.646 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0026029149999999999 2024-04-20T17:22:55.646 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.646 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.646 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:55.646 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.186699+0000", 2024-04-20T17:22:55.646 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.064314647000000003 2024-04-20T17:22:55.646 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.646 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.646 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:22:55.646 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.186740+0000", 2024-04-20T17:22:55.646 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0262000000000002e-05 2024-04-20T17:22:55.646 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.646 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.647 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:22:55.647 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.186788+0000", 2024-04-20T17:22:55.647 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7938000000000001e-05 2024-04-20T17:22:55.647 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.647 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:22:55.647 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.647 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.647 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.647 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:126478 3.5 3:a1a37c79:::benchmark_data_smithi171_110765_object126477:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-04-20T17:22:55.647 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:21:48.120019+0000", 2024-04-20T17:22:55.647 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 67.469416089999996, 2024-04-20T17:22:55.647 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.070689770999999998, 2024-04-20T17:22:55.647 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:22:55.647 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:22:55.648 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:55.648 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:22:55.648 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:22:55.648 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:22:55.648 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 126478 2024-04-20T17:22:55.648 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.648 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:22:55.648 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.648 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:22:55.648 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.120019+0000", 2024-04-20T17:22:55.648 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.648 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.648 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.648 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:22:55.648 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.120019+0000", 2024-04-20T17:22:55.648 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.649 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.649 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.649 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:22:55.649 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.120020+0000", 2024-04-20T17:22:55.649 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6899999999999997e-07 2024-04-20T17:22:55.649 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.649 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.649 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:22:55.649 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.120109+0000", 2024-04-20T17:22:55.649 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9215999999999994e-05 2024-04-20T17:22:55.649 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.649 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.649 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:22:55.649 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.120117+0000", 2024-04-20T17:22:55.649 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4689999999999997e-06 2024-04-20T17:22:55.650 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.650 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.650 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:22:55.650 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.120121+0000", 2024-04-20T17:22:55.650 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6430000000000001e-06 2024-04-20T17:22:55.650 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.650 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.650 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:22:55.650 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.120160+0000", 2024-04-20T17:22:55.650 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9446000000000001e-05 2024-04-20T17:22:55.650 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.651 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.651 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:22:55.651 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.120284+0000", 2024-04-20T17:22:55.651 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000123165 2024-04-20T17:22:55.651 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.651 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.651 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:22:55.651 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.120452+0000", 2024-04-20T17:22:55.651 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016852699999999999 2024-04-20T17:22:55.651 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.651 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.654 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:22:55.654 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.123227+0000", 2024-04-20T17:22:55.654 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0027753230000000001 2024-04-20T17:22:55.654 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.654 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.654 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:55.654 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.190629+0000", 2024-04-20T17:22:55.654 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.067401208000000004 2024-04-20T17:22:55.654 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.654 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.655 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:22:55.655 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.190666+0000", 2024-04-20T17:22:55.655 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7483e-05 2024-04-20T17:22:55.655 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.655 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.655 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:22:55.655 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.190709+0000", 2024-04-20T17:22:55.655 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2722000000000002e-05 2024-04-20T17:22:55.655 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.655 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:22:55.655 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.655 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.655 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.656 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:126482 3.5 3:abb52e33:::benchmark_data_smithi171_110765_object126481:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-04-20T17:22:55.656 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:21:48.125755+0000", 2024-04-20T17:22:55.656 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 67.463680384, 2024-04-20T17:22:55.656 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.065105607999999995, 2024-04-20T17:22:55.656 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:22:55.656 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:22:55.656 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:55.656 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:22:55.657 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:22:55.657 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:22:55.657 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 126482 2024-04-20T17:22:55.657 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.657 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:22:55.657 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.657 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:22:55.658 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.125755+0000", 2024-04-20T17:22:55.658 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.658 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.658 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.658 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:22:55.658 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.125755+0000", 2024-04-20T17:22:55.658 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.658 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.658 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.658 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:22:55.658 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.125756+0000", 2024-04-20T17:22:55.658 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.643e-06 2024-04-20T17:22:55.658 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.659 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.659 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:22:55.659 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.125822+0000", 2024-04-20T17:22:55.659 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5308999999999994e-05 2024-04-20T17:22:55.659 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.659 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.659 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:22:55.659 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.125832+0000", 2024-04-20T17:22:55.659 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0298e-05 2024-04-20T17:22:55.659 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.659 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.659 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:22:55.659 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.125838+0000", 2024-04-20T17:22:55.659 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6629999999999998e-06 2024-04-20T17:22:55.659 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.660 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.660 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:22:55.660 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.125888+0000", 2024-04-20T17:22:55.660 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9820000000000001e-05 2024-04-20T17:22:55.660 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.660 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.660 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:22:55.660 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.126022+0000", 2024-04-20T17:22:55.660 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000134165 2024-04-20T17:22:55.660 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.660 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.660 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:22:55.660 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.126158+0000", 2024-04-20T17:22:55.660 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013643699999999999 2024-04-20T17:22:55.661 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.661 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.661 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:22:55.661 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.129617+0000", 2024-04-20T17:22:55.661 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0034583679999999999 2024-04-20T17:22:55.661 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.661 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.661 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:55.661 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.190759+0000", 2024-04-20T17:22:55.662 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.061142657000000003 2024-04-20T17:22:55.662 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.662 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.662 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:22:55.662 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.190817+0000", 2024-04-20T17:22:55.662 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7431e-05 2024-04-20T17:22:55.662 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.662 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.663 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:22:55.663 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.190860+0000", 2024-04-20T17:22:55.664 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3816999999999997e-05 2024-04-20T17:22:55.664 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.664 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:22:55.664 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.664 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.664 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.664 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:126486 3.c 3:35a3f2f0:::benchmark_data_smithi171_110765_object126485:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-04-20T17:22:55.664 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:21:48.128837+0000", 2024-04-20T17:22:55.664 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 67.460598047000005, 2024-04-20T17:22:55.664 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.058089779000000001, 2024-04-20T17:22:55.664 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:22:55.664 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:22:55.665 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:55.665 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:22:55.665 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:22:55.665 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:22:55.665 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 126486 2024-04-20T17:22:55.665 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.665 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:22:55.665 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.665 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:22:55.665 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.128837+0000", 2024-04-20T17:22:55.665 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.665 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.665 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.665 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:22:55.666 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.128837+0000", 2024-04-20T17:22:55.666 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.666 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.666 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.666 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:22:55.666 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.128839+0000", 2024-04-20T17:22:55.666 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5629999999999999e-06 2024-04-20T17:22:55.666 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.666 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.666 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:22:55.666 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.128928+0000", 2024-04-20T17:22:55.666 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8933999999999999e-05 2024-04-20T17:22:55.666 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.666 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.666 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:22:55.667 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.128943+0000", 2024-04-20T17:22:55.667 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5469e-05 2024-04-20T17:22:55.667 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.667 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.667 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:22:55.667 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.128949+0000", 2024-04-20T17:22:55.667 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0510000000000002e-06 2024-04-20T17:22:55.667 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.667 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.667 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:22:55.667 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.128994+0000", 2024-04-20T17:22:55.667 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5287000000000002e-05 2024-04-20T17:22:55.667 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.667 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.667 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:22:55.668 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.129082+0000", 2024-04-20T17:22:55.668 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7601000000000004e-05 2024-04-20T17:22:55.668 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.668 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.668 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:22:55.668 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.129180+0000", 2024-04-20T17:22:55.668 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8311000000000001e-05 2024-04-20T17:22:55.668 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.668 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.668 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:22:55.668 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.131881+0000", 2024-04-20T17:22:55.668 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0027004820000000001 2024-04-20T17:22:55.668 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.668 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.668 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:55.669 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.186834+0000", 2024-04-20T17:22:55.669 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.054952666999999997 2024-04-20T17:22:55.669 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.669 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.669 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:22:55.669 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.186887+0000", 2024-04-20T17:22:55.669 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3567999999999999e-05 2024-04-20T17:22:55.669 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.669 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.669 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:22:55.669 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.186927+0000", 2024-04-20T17:22:55.669 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9845999999999998e-05 2024-04-20T17:22:55.669 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.669 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:22:55.669 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.670 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.670 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.670 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:126487 3.5 3:a9010720:::benchmark_data_smithi171_110765_object126486:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-04-20T17:22:55.670 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:21:48.130169+0000", 2024-04-20T17:22:55.670 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 67.459266228999994, 2024-04-20T17:22:55.670 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.060942059, 2024-04-20T17:22:55.670 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:22:55.670 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:22:55.670 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:55.670 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:22:55.670 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:22:55.670 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:22:55.670 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 126487 2024-04-20T17:22:55.670 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.670 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:22:55.671 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.671 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:22:55.671 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.130169+0000", 2024-04-20T17:22:55.671 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.671 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.671 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.671 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:22:55.671 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.130169+0000", 2024-04-20T17:22:55.671 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.671 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.671 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.671 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:22:55.671 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.130170+0000", 2024-04-20T17:22:55.671 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.159e-06 2024-04-20T17:22:55.671 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.672 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.672 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:22:55.672 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.130248+0000", 2024-04-20T17:22:55.672 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7705000000000002e-05 2024-04-20T17:22:55.672 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.672 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.672 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:22:55.672 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.130255+0000", 2024-04-20T17:22:55.672 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6340000000000001e-06 2024-04-20T17:22:55.672 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.672 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.672 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:22:55.672 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.130260+0000", 2024-04-20T17:22:55.672 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5909999999999998e-06 2024-04-20T17:22:55.672 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.673 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.673 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:22:55.673 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.130308+0000", 2024-04-20T17:22:55.673 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8338999999999999e-05 2024-04-20T17:22:55.673 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.673 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.673 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:22:55.673 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.130428+0000", 2024-04-20T17:22:55.673 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011962499999999999 2024-04-20T17:22:55.673 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.673 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.673 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:22:55.674 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.130586+0000", 2024-04-20T17:22:55.674 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015836100000000001 2024-04-20T17:22:55.674 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.674 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.674 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:22:55.674 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.133126+0000", 2024-04-20T17:22:55.674 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0025395399999999999 2024-04-20T17:22:55.674 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.674 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.674 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:55.674 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.191052+0000", 2024-04-20T17:22:55.674 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.057925695999999999 2024-04-20T17:22:55.674 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.674 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.675 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:22:55.675 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.191085+0000", 2024-04-20T17:22:55.675 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3688e-05 2024-04-20T17:22:55.675 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.675 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.675 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:22:55.675 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.191111+0000", 2024-04-20T17:22:55.675 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5721e-05 2024-04-20T17:22:55.675 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.675 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:22:55.675 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.675 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.675 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.675 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:126491 3.c 3:354a27a8:::benchmark_data_smithi171_110765_object126490:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-04-20T17:22:55.675 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:21:48.134844+0000", 2024-04-20T17:22:55.676 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 67.454591093000005, 2024-04-20T17:22:55.676 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.056435148999999997, 2024-04-20T17:22:55.676 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:22:55.676 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:22:55.676 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:55.676 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:22:55.676 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:22:55.676 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:22:55.676 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 126491 2024-04-20T17:22:55.676 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.676 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:22:55.676 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.676 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:22:55.676 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.134844+0000", 2024-04-20T17:22:55.676 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.677 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.677 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.677 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:22:55.677 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.134844+0000", 2024-04-20T17:22:55.677 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.677 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.677 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.677 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:22:55.677 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.134846+0000", 2024-04-20T17:22:55.677 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6190000000000001e-06 2024-04-20T17:22:55.677 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.677 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.677 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:22:55.677 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.134913+0000", 2024-04-20T17:22:55.677 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6982000000000003e-05 2024-04-20T17:22:55.677 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.678 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.678 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:22:55.678 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.134923+0000", 2024-04-20T17:22:55.678 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0468e-05 2024-04-20T17:22:55.678 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.678 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.678 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:22:55.678 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.134930+0000", 2024-04-20T17:22:55.678 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8179999999999998e-06 2024-04-20T17:22:55.678 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.678 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.678 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:22:55.678 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.134976+0000", 2024-04-20T17:22:55.678 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6230999999999998e-05 2024-04-20T17:22:55.678 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.679 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.679 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:22:55.679 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.135053+0000", 2024-04-20T17:22:55.679 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6796000000000001e-05 2024-04-20T17:22:55.679 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.679 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.679 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:22:55.679 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.135140+0000", 2024-04-20T17:22:55.679 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7099000000000001e-05 2024-04-20T17:22:55.679 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.679 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.679 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:22:55.679 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.137279+0000", 2024-04-20T17:22:55.679 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00213919 2024-04-20T17:22:55.679 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.680 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.680 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:55.680 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.191226+0000", 2024-04-20T17:22:55.680 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.053946406000000002 2024-04-20T17:22:55.680 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.680 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.680 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:22:55.680 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.191257+0000", 2024-04-20T17:22:55.680 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1763000000000001e-05 2024-04-20T17:22:55.680 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.680 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.680 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:22:55.680 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.191279+0000", 2024-04-20T17:22:55.680 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1776999999999998e-05 2024-04-20T17:22:55.680 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.680 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:22:55.681 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.681 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.681 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.681 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:126492 3.5 3:acb22ae8:::benchmark_data_smithi171_110765_object126491:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-04-20T17:22:55.681 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:21:48.139173+0000", 2024-04-20T17:22:55.681 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 67.450262041000002, 2024-04-20T17:22:55.681 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.052067747999999997, 2024-04-20T17:22:55.681 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:22:55.681 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:22:55.681 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:55.681 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:22:55.681 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:22:55.681 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:22:55.681 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 126492 2024-04-20T17:22:55.681 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.682 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:22:55.682 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.682 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:22:55.682 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.139173+0000", 2024-04-20T17:22:55.682 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.682 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.682 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.682 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:22:55.682 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.139173+0000", 2024-04-20T17:22:55.682 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.682 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.682 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.682 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:22:55.682 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.139174+0000", 2024-04-20T17:22:55.682 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.316e-06 2024-04-20T17:22:55.683 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.683 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.683 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:22:55.683 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.139282+0000", 2024-04-20T17:22:55.683 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000107397 2024-04-20T17:22:55.683 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.683 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.683 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:22:55.683 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.139292+0000", 2024-04-20T17:22:55.683 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7850000000000003e-06 2024-04-20T17:22:55.683 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.683 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.683 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:22:55.683 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.139297+0000", 2024-04-20T17:22:55.683 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8100000000000003e-06 2024-04-20T17:22:55.683 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.684 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.684 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:22:55.684 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.139367+0000", 2024-04-20T17:22:55.684 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9060000000000006e-05 2024-04-20T17:22:55.684 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.684 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.684 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:22:55.684 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.139494+0000", 2024-04-20T17:22:55.684 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000127191 2024-04-20T17:22:55.684 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.684 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.684 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:22:55.684 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.139649+0000", 2024-04-20T17:22:55.684 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015578099999999999 2024-04-20T17:22:55.685 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.685 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.685 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:22:55.685 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.142078+0000", 2024-04-20T17:22:55.685 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0024281379999999998 2024-04-20T17:22:55.685 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.685 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.685 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:55.685 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.191151+0000", 2024-04-20T17:22:55.685 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.049072920999999999 2024-04-20T17:22:55.686 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.686 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.686 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:22:55.686 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.191202+0000", 2024-04-20T17:22:55.686 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0977000000000003e-05 2024-04-20T17:22:55.686 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.686 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.686 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:22:55.686 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:48.191241+0000", 2024-04-20T17:22:55.686 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9372000000000002e-05 2024-04-20T17:22:55.686 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.686 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:22:55.686 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.686 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.687 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.687 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:164929 3.5 3:a92cd60e:::benchmark_data_smithi171_110765_object21714:head [delete] snapc 0=[] ondisk+write+known_if_redirected e393)", 2024-04-20T17:22:55.687 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:22:20.512580+0000", 2024-04-20T17:22:55.687 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 35.076855410999997, 2024-04-20T17:22:55.687 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.066581024000000003, 2024-04-20T17:22:55.687 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:22:55.687 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:22:55.687 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:55.687 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:22:55.687 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:22:55.687 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:22:55.687 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 164929 2024-04-20T17:22:55.687 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.687 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:22:55.687 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.688 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:22:55.688 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.512580+0000", 2024-04-20T17:22:55.688 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.688 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.688 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.688 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:22:55.688 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.512580+0000", 2024-04-20T17:22:55.688 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.688 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.688 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.688 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:22:55.688 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.512581+0000", 2024-04-20T17:22:55.688 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7039999999999999e-06 2024-04-20T17:22:55.688 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.688 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.689 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:22:55.689 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.512585+0000", 2024-04-20T17:22:55.689 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9689999999999996e-06 2024-04-20T17:22:55.689 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.689 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.689 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:22:55.689 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.512595+0000", 2024-04-20T17:22:55.689 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3980000000000005e-06 2024-04-20T17:22:55.689 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.689 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.689 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:22:55.689 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.512602+0000", 2024-04-20T17:22:55.689 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1130000000000003e-06 2024-04-20T17:22:55.689 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.689 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.690 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:22:55.690 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.512636+0000", 2024-04-20T17:22:55.690 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4107999999999999e-05 2024-04-20T17:22:55.690 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.690 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.690 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:22:55.690 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.512750+0000", 2024-04-20T17:22:55.690 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000113647 2024-04-20T17:22:55.690 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.690 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.690 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:22:55.690 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.512820+0000", 2024-04-20T17:22:55.690 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0297999999999997e-05 2024-04-20T17:22:55.690 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.690 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.691 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:22:55.691 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.520710+0000", 2024-04-20T17:22:55.691 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.007890312 2024-04-20T17:22:55.691 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.691 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.691 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:55.691 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.579081+0000", 2024-04-20T17:22:55.691 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.058370730000000003 2024-04-20T17:22:55.691 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.691 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.691 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:22:55.691 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.579117+0000", 2024-04-20T17:22:55.691 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5713999999999997e-05 2024-04-20T17:22:55.691 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.691 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.692 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:22:55.692 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.579161+0000", 2024-04-20T17:22:55.692 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4030999999999999e-05 2024-04-20T17:22:55.692 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.692 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:22:55.692 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.692 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.692 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.692 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:164932 3.5 3:ad6ea9c8:::benchmark_data_smithi171_110765_object21717:head [delete] snapc 0=[] ondisk+write+known_if_redirected e393)", 2024-04-20T17:22:55.692 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:22:20.512887+0000", 2024-04-20T17:22:55.692 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 35.076547826999999, 2024-04-20T17:22:55.692 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.066381430000000005, 2024-04-20T17:22:55.692 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:22:55.692 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:22:55.692 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:55.692 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:22:55.693 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:22:55.693 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:22:55.693 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 164932 2024-04-20T17:22:55.693 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.693 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:22:55.693 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.693 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:22:55.693 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.512887+0000", 2024-04-20T17:22:55.693 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.693 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.693 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.693 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:22:55.693 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.512887+0000", 2024-04-20T17:22:55.693 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.693 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.694 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.694 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:22:55.694 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.512889+0000", 2024-04-20T17:22:55.694 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4330000000000001e-06 2024-04-20T17:22:55.694 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.694 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.694 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:22:55.694 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.512894+0000", 2024-04-20T17:22:55.694 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0089999999999998e-06 2024-04-20T17:22:55.694 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.694 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.694 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:22:55.694 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.512904+0000", 2024-04-20T17:22:55.694 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0069e-05 2024-04-20T17:22:55.694 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.695 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.695 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:22:55.695 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.512912+0000", 2024-04-20T17:22:55.695 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0209999999999999e-06 2024-04-20T17:22:55.695 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.695 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.695 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:22:55.695 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.517135+0000", 2024-04-20T17:22:55.695 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.004223157 2024-04-20T17:22:55.695 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.695 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.695 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:22:55.695 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.517222+0000", 2024-04-20T17:22:55.695 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6971999999999993e-05 2024-04-20T17:22:55.695 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.696 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.696 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:22:55.696 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.517297+0000", 2024-04-20T17:22:55.696 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5463000000000006e-05 2024-04-20T17:22:55.696 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.696 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.696 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:22:55.696 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.520828+0000", 2024-04-20T17:22:55.696 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0035303690000000002 2024-04-20T17:22:55.696 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.696 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.696 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:55.696 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.579202+0000", 2024-04-20T17:22:55.696 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.058373787000000003 2024-04-20T17:22:55.696 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.696 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.697 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:22:55.697 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.579236+0000", 2024-04-20T17:22:55.697 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4708e-05 2024-04-20T17:22:55.697 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.697 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.697 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:22:55.697 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.579269+0000", 2024-04-20T17:22:55.697 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2441999999999999e-05 2024-04-20T17:22:55.697 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.697 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:22:55.697 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.697 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.697 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.697 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:165132 3.c 3:3c5b4786:::benchmark_data_smithi171_110765_object21917:head [delete] snapc 0=[] ondisk+write+known_if_redirected e393)", 2024-04-20T17:22:55.698 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:22:20.694058+0000", 2024-04-20T17:22:55.698 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 34.895376835999997, 2024-04-20T17:22:55.698 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.052275645000000003, 2024-04-20T17:22:55.698 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:22:55.698 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:22:55.698 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:55.698 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:22:55.698 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:22:55.698 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:22:55.698 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 165132 2024-04-20T17:22:55.698 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.698 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:22:55.698 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.698 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:22:55.699 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.694058+0000", 2024-04-20T17:22:55.699 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.699 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.699 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.699 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:22:55.699 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.694058+0000", 2024-04-20T17:22:55.699 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.699 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.699 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.699 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:22:55.699 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.694059+0000", 2024-04-20T17:22:55.699 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6600000000000005e-07 2024-04-20T17:22:55.699 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.699 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.699 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:22:55.700 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.694062+0000", 2024-04-20T17:22:55.700 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8770000000000002e-06 2024-04-20T17:22:55.700 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.700 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.700 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:22:55.700 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.694068+0000", 2024-04-20T17:22:55.700 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7119999999999997e-06 2024-04-20T17:22:55.700 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.700 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.700 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:22:55.700 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.694073+0000", 2024-04-20T17:22:55.701 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6339999999999998e-06 2024-04-20T17:22:55.701 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.701 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.701 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:22:55.701 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.694110+0000", 2024-04-20T17:22:55.701 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6300000000000001e-05 2024-04-20T17:22:55.701 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.701 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.701 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:22:55.701 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.694209+0000", 2024-04-20T17:22:55.701 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9142999999999996e-05 2024-04-20T17:22:55.701 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.701 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.701 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:22:55.702 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.694326+0000", 2024-04-20T17:22:55.702 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000117617 2024-04-20T17:22:55.702 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.702 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.702 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:22:55.702 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.702233+0000", 2024-04-20T17:22:55.702 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0079061329999999992 2024-04-20T17:22:55.702 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.702 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.702 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:55.702 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.746239+0000", 2024-04-20T17:22:55.702 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.044006383000000003 2024-04-20T17:22:55.702 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.702 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.703 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:22:55.703 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.746291+0000", 2024-04-20T17:22:55.703 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1711e-05 2024-04-20T17:22:55.703 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.703 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.703 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:22:55.703 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.746334+0000", 2024-04-20T17:22:55.703 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3269000000000002e-05 2024-04-20T17:22:55.703 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.703 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:22:55.703 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.703 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.703 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.703 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:165133 3.c 3:36d027a3:::benchmark_data_smithi171_110765_object21918:head [delete] snapc 0=[] ondisk+write+known_if_redirected e393)", 2024-04-20T17:22:55.703 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:22:20.694085+0000", 2024-04-20T17:22:55.704 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 34.895350037999997, 2024-04-20T17:22:55.704 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.052364537000000003, 2024-04-20T17:22:55.704 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:22:55.704 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:22:55.704 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:55.704 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:22:55.704 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:22:55.704 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:22:55.704 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 165133 2024-04-20T17:22:55.704 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.704 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:22:55.704 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.704 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:22:55.704 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.694085+0000", 2024-04-20T17:22:55.704 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.706 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.706 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.706 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:22:55.706 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.694085+0000", 2024-04-20T17:22:55.706 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.706 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.706 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.706 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:22:55.706 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.694086+0000", 2024-04-20T17:22:55.706 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7599999999999997e-07 2024-04-20T17:22:55.706 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.707 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.707 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:22:55.707 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.694088+0000", 2024-04-20T17:22:55.707 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3089999999999998e-06 2024-04-20T17:22:55.707 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.708 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.708 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:22:55.708 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.694094+0000", 2024-04-20T17:22:55.708 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9560000000000002e-06 2024-04-20T17:22:55.708 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.708 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.709 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:22:55.709 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.694099+0000", 2024-04-20T17:22:55.709 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3929999999999999e-06 2024-04-20T17:22:55.709 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.709 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.709 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:22:55.709 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.698720+0000", 2024-04-20T17:22:55.709 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0046208350000000002 2024-04-20T17:22:55.709 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.709 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.709 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:22:55.709 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.698836+0000", 2024-04-20T17:22:55.709 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011519999999999999 2024-04-20T17:22:55.710 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.710 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.710 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:22:55.710 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.698919+0000", 2024-04-20T17:22:55.710 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3857999999999996e-05 2024-04-20T17:22:55.710 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.710 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.710 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:22:55.710 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.702386+0000", 2024-04-20T17:22:55.711 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0034662149999999999 2024-04-20T17:22:55.711 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.711 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.711 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:55.711 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.746372+0000", 2024-04-20T17:22:55.711 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.043986745000000001 2024-04-20T17:22:55.711 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.711 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.711 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:22:55.711 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.746415+0000", 2024-04-20T17:22:55.711 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3031000000000002e-05 2024-04-20T17:22:55.711 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.711 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.712 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:22:55.712 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.746450+0000", 2024-04-20T17:22:55.712 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4319000000000001e-05 2024-04-20T17:22:55.712 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.712 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:22:55.712 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.712 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.712 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.712 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:175240 3.5 3:abf02d2e:::benchmark_data_smithi171_110765_object32025:head [delete] snapc 0=[] ondisk+write+known_if_redirected e395)", 2024-04-20T17:22:55.712 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:22:29.008485+0000", 2024-04-20T17:22:55.712 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 26.580949718999999, 2024-04-20T17:22:55.712 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.048681110999999999, 2024-04-20T17:22:55.712 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:22:55.714 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:22:55.714 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:22:55.714 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:22:55.714 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:22:55.715 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:22:55.715 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 175240 2024-04-20T17:22:55.715 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.715 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:22:55.715 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.715 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:22:55.716 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:29.008485+0000", 2024-04-20T17:22:55.716 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.716 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.716 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.716 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:22:55.716 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:29.008485+0000", 2024-04-20T17:22:55.716 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:55.716 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.716 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.717 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:22:55.717 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:29.008486+0000", 2024-04-20T17:22:55.717 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2200000000000004e-07 2024-04-20T17:22:55.717 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.717 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.717 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:22:55.717 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:29.008488+0000", 2024-04-20T17:22:55.717 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2670000000000001e-06 2024-04-20T17:22:55.717 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.717 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.717 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:22:55.717 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:29.008493+0000", 2024-04-20T17:22:55.717 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0610000000000003e-06 2024-04-20T17:22:55.717 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.717 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.718 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:22:55.718 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:29.008497+0000", 2024-04-20T17:22:55.718 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5410000000000001e-06 2024-04-20T17:22:55.718 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.718 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.718 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:22:55.718 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:29.020253+0000", 2024-04-20T17:22:55.718 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011755928000000001 2024-04-20T17:22:55.718 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.718 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.718 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:22:55.718 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:29.020378+0000", 2024-04-20T17:22:55.718 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012474599999999999 2024-04-20T17:22:55.719 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.719 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:22:55.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:29.020498+0000", 2024-04-20T17:22:55.719 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000120158 2024-04-20T17:22:55.719 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.719 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:22:55.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:29.029692+0000", 2024-04-20T17:22:55.719 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0091945399999999993 2024-04-20T17:22:55.719 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.719 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:22:55.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:29.057101+0000", 2024-04-20T17:22:55.719 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.027408397000000001 2024-04-20T17:22:55.719 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.720 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.720 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:22:55.720 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:29.057136+0000", 2024-04-20T17:22:55.720 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5438999999999997e-05 2024-04-20T17:22:55.720 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:55.720 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:55.720 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:22:55.720 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:29.057167+0000", 2024-04-20T17:22:55.720 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0412e-05 2024-04-20T17:22:55.720 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.720 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:22:55.720 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.720 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:55.720 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:22:55.720 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:22:56.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:56 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:22:56.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:56 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:22:56.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:56 smithi106 ceph-mon[44423]: osdmap e403: 12 total, 12 up, 11 in 2024-04-20T17:22:56.388 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-20T17:22:56.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:56 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:22:56.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:56 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:22:56.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:56 smithi005 ceph-mon[39562]: osdmap e403: 12 total, 12 up, 11 in 2024-04-20T17:22:56.566 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:22:56.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:56 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:22:56.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:56 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:22:56.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:56 smithi171 ceph-mon[44189]: osdmap e403: 12 total, 12 up, 11 in 2024-04-20T17:22:56.710 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:22:57.144 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:22:57.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:57.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:22:57.249 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:57 smithi106 ceph-mon[44423]: pgmap v1098: 17 pgs: 17 active+clean; 4.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-20T17:22:57.249 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:57 smithi106 ceph-mon[44423]: 3.d scrub starts 2024-04-20T17:22:57.256 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:22:57.292 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:57.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:57 smithi005 ceph-mon[39562]: pgmap v1098: 17 pgs: 17 active+clean; 4.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-20T17:22:57.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:57 smithi005 ceph-mon[39562]: 3.d scrub starts 2024-04-20T17:22:57.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:57 smithi171 ceph-mon[44189]: pgmap v1098: 17 pgs: 17 active+clean; 4.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-20T17:22:57.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:57 smithi171 ceph-mon[44189]: 3.d scrub starts 2024-04-20T17:22:57.648 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:22:57.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:57.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:22:57.753 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:22:57.792 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:57.824 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:22:57.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:57.827 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:22:57.840 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:22:57.841 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:22:57.841 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 403, 2024-04-20T17:22:57.841 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "13c6e76c-ff37-11ee-bc93-c7b262605968", 2024-04-20T17:22:57.841 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-04-20T16:58:25.272557+0000", 2024-04-20T17:22:57.841 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-04-20T17:22:55.123915+0000", 2024-04-20T17:22:57.841 INFO:teuthology.orchestra.run.smithi005.stdout: "last_up_change": "2024-04-20T17:22:32.243867+0000", 2024-04-20T17:22:57.841 INFO:teuthology.orchestra.run.smithi005.stdout: "last_in_change": "2024-04-20T17:20:54.169566+0000", 2024-04-20T17:22:57.842 INFO:teuthology.orchestra.run.smithi005.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-20T17:22:57.842 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_num": 5799936, 2024-04-20T17:22:57.842 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_set": [ 2024-04-20T17:22:57.842 INFO:teuthology.orchestra.run.smithi005.stdout: "pglog_hardlimit", 2024-04-20T17:22:57.842 INFO:teuthology.orchestra.run.smithi005.stdout: "purged_snapdirs", 2024-04-20T17:22:57.843 INFO:teuthology.orchestra.run.smithi005.stdout: "recovery_deletes", 2024-04-20T17:22:57.843 INFO:teuthology.orchestra.run.smithi005.stdout: "sortbitwise" 2024-04-20T17:22:57.843 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:22:57.843 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_version": 29, 2024-04-20T17:22:57.843 INFO:teuthology.orchestra.run.smithi005.stdout: "full_ratio": 0.94999998807907104, 2024-04-20T17:22:57.843 INFO:teuthology.orchestra.run.smithi005.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-20T17:22:57.843 INFO:teuthology.orchestra.run.smithi005.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-20T17:22:57.844 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_snapshot": "", 2024-04-20T17:22:57.844 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_max": 3, 2024-04-20T17:22:57.844 INFO:teuthology.orchestra.run.smithi005.stdout: "max_osd": 12, 2024-04-20T17:22:57.844 INFO:teuthology.orchestra.run.smithi005.stdout: "require_min_compat_client": "luminous", 2024-04-20T17:22:57.844 INFO:teuthology.orchestra.run.smithi005.stdout: "min_compat_client": "jewel", 2024-04-20T17:22:57.844 INFO:teuthology.orchestra.run.smithi005.stdout: "require_osd_release": "squid", 2024-04-20T17:22:57.844 INFO:teuthology.orchestra.run.smithi005.stdout: "allow_crimson": false, 2024-04-20T17:22:57.845 INFO:teuthology.orchestra.run.smithi005.stdout: "pools": [ 2024-04-20T17:22:57.845 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.845 INFO:teuthology.orchestra.run.smithi005.stdout: "pool": 1, 2024-04-20T17:22:57.845 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_name": ".mgr", 2024-04-20T17:22:57.845 INFO:teuthology.orchestra.run.smithi005.stdout: "create_time": "2024-04-20T17:01:35.909493+0000", 2024-04-20T17:22:57.845 INFO:teuthology.orchestra.run.smithi005.stdout: "flags": 1, 2024-04-20T17:22:57.846 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_names": "hashpspool", 2024-04-20T17:22:57.846 INFO:teuthology.orchestra.run.smithi005.stdout: "type": 1, 2024-04-20T17:22:57.846 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 2, 2024-04-20T17:22:57.846 INFO:teuthology.orchestra.run.smithi005.stdout: "min_size": 2, 2024-04-20T17:22:57.846 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_rule": 0, 2024-04-20T17:22:57.846 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_count": 0, 2024-04-20T17:22:57.847 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_target": 0, 2024-04-20T17:22:57.847 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T17:22:57.847 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T17:22:57.847 INFO:teuthology.orchestra.run.smithi005.stdout: "object_hash": 2, 2024-04-20T17:22:57.847 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_autoscale_mode": "off", 2024-04-20T17:22:57.847 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num": 1, 2024-04-20T17:22:57.848 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num": 1, 2024-04-20T17:22:57.848 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num_target": 1, 2024-04-20T17:22:57.848 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_target": 1, 2024-04-20T17:22:57.848 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_pending": 1, 2024-04-20T17:22:57.848 INFO:teuthology.orchestra.run.smithi005.stdout: "last_pg_merge_meta": { 2024-04-20T17:22:57.848 INFO:teuthology.orchestra.run.smithi005.stdout: "source_pgid": "0.0", 2024-04-20T17:22:57.848 INFO:teuthology.orchestra.run.smithi005.stdout: "ready_epoch": 0, 2024-04-20T17:22:57.849 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_started": 0, 2024-04-20T17:22:57.849 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_clean": 0, 2024-04-20T17:22:57.849 INFO:teuthology.orchestra.run.smithi005.stdout: "source_version": "0'0", 2024-04-20T17:22:57.849 INFO:teuthology.orchestra.run.smithi005.stdout: "target_version": "0'0" 2024-04-20T17:22:57.849 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.849 INFO:teuthology.orchestra.run.smithi005.stdout: "last_change": "16", 2024-04-20T17:22:57.850 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend": "0", 2024-04-20T17:22:57.850 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-20T17:22:57.850 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_preluminous": "0", 2024-04-20T17:22:57.850 INFO:teuthology.orchestra.run.smithi005.stdout: "auid": 0, 2024-04-20T17:22:57.850 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_mode": "selfmanaged", 2024-04-20T17:22:57.850 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_seq": 0, 2024-04-20T17:22:57.851 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_epoch": 0, 2024-04-20T17:22:57.851 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_snaps": [], 2024-04-20T17:22:57.851 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_snaps": "[]", 2024-04-20T17:22:57.851 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_bytes": 0, 2024-04-20T17:22:57.851 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_objects": 0, 2024-04-20T17:22:57.851 INFO:teuthology.orchestra.run.smithi005.stdout: "tiers": [], 2024-04-20T17:22:57.852 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_of": -1, 2024-04-20T17:22:57.852 INFO:teuthology.orchestra.run.smithi005.stdout: "read_tier": -1, 2024-04-20T17:22:57.852 INFO:teuthology.orchestra.run.smithi005.stdout: "write_tier": -1, 2024-04-20T17:22:57.852 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_mode": "none", 2024-04-20T17:22:57.852 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_bytes": 0, 2024-04-20T17:22:57.852 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_objects": 0, 2024-04-20T17:22:57.853 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T17:22:57.853 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T17:22:57.853 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T17:22:57.853 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_flush_age": 0, 2024-04-20T17:22:57.853 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_evict_age": 0, 2024-04-20T17:22:57.853 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_profile": "", 2024-04-20T17:22:57.854 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_params": { 2024-04-20T17:22:57.854 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "none" 2024-04-20T17:22:57.854 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.854 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_period": 0, 2024-04-20T17:22:57.854 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_count": 0, 2024-04-20T17:22:57.854 INFO:teuthology.orchestra.run.smithi005.stdout: "use_gmt_hitset": true, 2024-04-20T17:22:57.855 INFO:teuthology.orchestra.run.smithi005.stdout: "min_read_recency_for_promote": 0, 2024-04-20T17:22:57.855 INFO:teuthology.orchestra.run.smithi005.stdout: "min_write_recency_for_promote": 0, 2024-04-20T17:22:57.855 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T17:22:57.855 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_search_last_n": 0, 2024-04-20T17:22:57.855 INFO:teuthology.orchestra.run.smithi005.stdout: "grade_table": [], 2024-04-20T17:22:57.855 INFO:teuthology.orchestra.run.smithi005.stdout: "stripe_width": 0, 2024-04-20T17:22:57.856 INFO:teuthology.orchestra.run.smithi005.stdout: "expected_num_objects": 0, 2024-04-20T17:22:57.856 INFO:teuthology.orchestra.run.smithi005.stdout: "fast_read": false, 2024-04-20T17:22:57.856 INFO:teuthology.orchestra.run.smithi005.stdout: "options": { 2024-04-20T17:22:57.856 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_max": 32, 2024-04-20T17:22:57.856 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_min": 1 2024-04-20T17:22:57.856 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.857 INFO:teuthology.orchestra.run.smithi005.stdout: "application_metadata": { 2024-04-20T17:22:57.857 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": {} 2024-04-20T17:22:57.857 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.857 INFO:teuthology.orchestra.run.smithi005.stdout: "read_balance": { 2024-04-20T17:22:57.857 INFO:teuthology.orchestra.run.smithi005.stdout: "score_type": "Fair distribution", 2024-04-20T17:22:57.857 INFO:teuthology.orchestra.run.smithi005.stdout: "score_acting": 11.760000228881836, 2024-04-20T17:22:57.857 INFO:teuthology.orchestra.run.smithi005.stdout: "score_stable": 11.760000228881836, 2024-04-20T17:22:57.858 INFO:teuthology.orchestra.run.smithi005.stdout: "optimal_score": 0.17000000178813934, 2024-04-20T17:22:57.858 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_acting": 2, 2024-04-20T17:22:57.858 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_stable": 2, 2024-04-20T17:22:57.858 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-04-20T17:22:57.858 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity": 1, 2024-04-20T17:22:57.858 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity_weighted": 1 2024-04-20T17:22:57.859 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.859 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.859 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.859 INFO:teuthology.orchestra.run.smithi005.stdout: "pool": 3, 2024-04-20T17:22:57.859 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_name": "unique_pool_1", 2024-04-20T17:22:57.859 INFO:teuthology.orchestra.run.smithi005.stdout: "create_time": "2024-04-20T17:20:21.928963+0000", 2024-04-20T17:22:57.860 INFO:teuthology.orchestra.run.smithi005.stdout: "flags": 1, 2024-04-20T17:22:57.860 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_names": "hashpspool", 2024-04-20T17:22:57.860 INFO:teuthology.orchestra.run.smithi005.stdout: "type": 1, 2024-04-20T17:22:57.860 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 2, 2024-04-20T17:22:57.860 INFO:teuthology.orchestra.run.smithi005.stdout: "min_size": 2, 2024-04-20T17:22:57.860 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_rule": 0, 2024-04-20T17:22:57.861 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_count": 0, 2024-04-20T17:22:57.861 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_target": 0, 2024-04-20T17:22:57.861 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T17:22:57.861 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T17:22:57.861 INFO:teuthology.orchestra.run.smithi005.stdout: "object_hash": 2, 2024-04-20T17:22:57.861 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_autoscale_mode": "off", 2024-04-20T17:22:57.861 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num": 16, 2024-04-20T17:22:57.862 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num": 16, 2024-04-20T17:22:57.862 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num_target": 16, 2024-04-20T17:22:57.862 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_target": 16, 2024-04-20T17:22:57.862 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_pending": 16, 2024-04-20T17:22:57.862 INFO:teuthology.orchestra.run.smithi005.stdout: "last_pg_merge_meta": { 2024-04-20T17:22:57.862 INFO:teuthology.orchestra.run.smithi005.stdout: "source_pgid": "0.0", 2024-04-20T17:22:57.862 INFO:teuthology.orchestra.run.smithi005.stdout: "ready_epoch": 0, 2024-04-20T17:22:57.862 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_started": 0, 2024-04-20T17:22:57.862 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_clean": 0, 2024-04-20T17:22:57.863 INFO:teuthology.orchestra.run.smithi005.stdout: "source_version": "0'0", 2024-04-20T17:22:57.863 INFO:teuthology.orchestra.run.smithi005.stdout: "target_version": "0'0" 2024-04-20T17:22:57.863 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.863 INFO:teuthology.orchestra.run.smithi005.stdout: "last_change": "360", 2024-04-20T17:22:57.863 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend": "0", 2024-04-20T17:22:57.863 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-20T17:22:57.863 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_preluminous": "0", 2024-04-20T17:22:57.863 INFO:teuthology.orchestra.run.smithi005.stdout: "auid": 0, 2024-04-20T17:22:57.863 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_mode": "selfmanaged", 2024-04-20T17:22:57.863 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_seq": 0, 2024-04-20T17:22:57.863 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_epoch": 0, 2024-04-20T17:22:57.863 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_snaps": [], 2024-04-20T17:22:57.863 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_snaps": "[]", 2024-04-20T17:22:57.863 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_bytes": 0, 2024-04-20T17:22:57.863 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_objects": 0, 2024-04-20T17:22:57.864 INFO:teuthology.orchestra.run.smithi005.stdout: "tiers": [], 2024-04-20T17:22:57.864 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_of": -1, 2024-04-20T17:22:57.864 INFO:teuthology.orchestra.run.smithi005.stdout: "read_tier": -1, 2024-04-20T17:22:57.864 INFO:teuthology.orchestra.run.smithi005.stdout: "write_tier": -1, 2024-04-20T17:22:57.864 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_mode": "none", 2024-04-20T17:22:57.864 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_bytes": 0, 2024-04-20T17:22:57.864 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_objects": 0, 2024-04-20T17:22:57.864 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T17:22:57.864 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T17:22:57.864 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T17:22:57.864 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_flush_age": 0, 2024-04-20T17:22:57.864 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_evict_age": 0, 2024-04-20T17:22:57.864 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_profile": "", 2024-04-20T17:22:57.864 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_params": { 2024-04-20T17:22:57.865 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "none" 2024-04-20T17:22:57.865 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.865 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_period": 0, 2024-04-20T17:22:57.865 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_count": 0, 2024-04-20T17:22:57.865 INFO:teuthology.orchestra.run.smithi005.stdout: "use_gmt_hitset": true, 2024-04-20T17:22:57.865 INFO:teuthology.orchestra.run.smithi005.stdout: "min_read_recency_for_promote": 0, 2024-04-20T17:22:57.865 INFO:teuthology.orchestra.run.smithi005.stdout: "min_write_recency_for_promote": 0, 2024-04-20T17:22:57.865 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T17:22:57.865 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_search_last_n": 0, 2024-04-20T17:22:57.865 INFO:teuthology.orchestra.run.smithi005.stdout: "grade_table": [], 2024-04-20T17:22:57.865 INFO:teuthology.orchestra.run.smithi005.stdout: "stripe_width": 0, 2024-04-20T17:22:57.865 INFO:teuthology.orchestra.run.smithi005.stdout: "expected_num_objects": 0, 2024-04-20T17:22:57.865 INFO:teuthology.orchestra.run.smithi005.stdout: "fast_read": false, 2024-04-20T17:22:57.865 INFO:teuthology.orchestra.run.smithi005.stdout: "options": {}, 2024-04-20T17:22:57.866 INFO:teuthology.orchestra.run.smithi005.stdout: "application_metadata": { 2024-04-20T17:22:57.866 INFO:teuthology.orchestra.run.smithi005.stdout: "rados": {} 2024-04-20T17:22:57.866 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.866 INFO:teuthology.orchestra.run.smithi005.stdout: "read_balance": { 2024-04-20T17:22:57.866 INFO:teuthology.orchestra.run.smithi005.stdout: "score_type": "Fair distribution", 2024-04-20T17:22:57.866 INFO:teuthology.orchestra.run.smithi005.stdout: "score_acting": 3, 2024-04-20T17:22:57.866 INFO:teuthology.orchestra.run.smithi005.stdout: "score_stable": 3, 2024-04-20T17:22:57.866 INFO:teuthology.orchestra.run.smithi005.stdout: "optimal_score": 0.75, 2024-04-20T17:22:57.866 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_acting": 2.25, 2024-04-20T17:22:57.866 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_stable": 2.25, 2024-04-20T17:22:57.866 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-04-20T17:22:57.866 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity": 1, 2024-04-20T17:22:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity_weighted": 1 2024-04-20T17:22:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:22:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: "osds": [ 2024-04-20T17:22:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 0, 2024-04-20T17:22:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b", 2024-04-20T17:22:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:22:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:22:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:22:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:22:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:22:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:22:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 8, 2024-04-20T17:22:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 356, 2024-04-20T17:22:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:22:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:22:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:22:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6801", 2024-04-20T17:22:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:22:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:22:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6802", 2024-04-20T17:22:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:22:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:22:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6804", 2024-04-20T17:22:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:22:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:22:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6803", 2024-04-20T17:22:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:22:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6801/3901484705", 2024-04-20T17:22:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6802/3901484705", 2024-04-20T17:22:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6804/3901484705", 2024-04-20T17:22:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6803/3901484705", 2024-04-20T17:22:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:22:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:22:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:22:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 1, 2024-04-20T17:22:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "aed0c8eb-0e81-4e85-ab8f-a60420060656", 2024-04-20T17:22:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:22:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:22:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:22:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:22:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:22:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:22:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 13, 2024-04-20T17:22:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 356, 2024-04-20T17:22:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:22:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:22:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:22:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6805", 2024-04-20T17:22:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3391893914 2024-04-20T17:22:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:22:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6806", 2024-04-20T17:22:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3391893914 2024-04-20T17:22:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:22:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6808", 2024-04-20T17:22:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3391893914 2024-04-20T17:22:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:22:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6807", 2024-04-20T17:22:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3391893914 2024-04-20T17:22:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6805/3391893914", 2024-04-20T17:22:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6806/3391893914", 2024-04-20T17:22:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6808/3391893914", 2024-04-20T17:22:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6807/3391893914", 2024-04-20T17:22:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:22:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:22:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:22:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 2, 2024-04-20T17:22:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "f733a6a2-19c2-4c7a-af9c-04e282062bb7", 2024-04-20T17:22:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:22:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:22:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:22:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:22:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 20, 2024-04-20T17:22:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 169, 2024-04-20T17:22:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 202, 2024-04-20T17:22:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 202, 2024-04-20T17:22:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 170, 2024-04-20T17:22:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:22:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:22:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6809", 2024-04-20T17:22:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2120275501 2024-04-20T17:22:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:22:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6810", 2024-04-20T17:22:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2120275501 2024-04-20T17:22:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:22:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6812", 2024-04-20T17:22:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2120275501 2024-04-20T17:22:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:22:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6811", 2024-04-20T17:22:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2120275501 2024-04-20T17:22:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6809/2120275501", 2024-04-20T17:22:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6810/2120275501", 2024-04-20T17:22:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6812/2120275501", 2024-04-20T17:22:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6811/2120275501", 2024-04-20T17:22:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:22:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:22:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:22:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 3, 2024-04-20T17:22:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "8c92f1df-e298-4606-88ae-b8a8e79628a1", 2024-04-20T17:22:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:22:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:22:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:22:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:22:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:22:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:22:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 25, 2024-04-20T17:22:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 356, 2024-04-20T17:22:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:22:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:22:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:22:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6813", 2024-04-20T17:22:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 377652039 2024-04-20T17:22:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:22:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6814", 2024-04-20T17:22:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 377652039 2024-04-20T17:22:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:22:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6816", 2024-04-20T17:22:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 377652039 2024-04-20T17:22:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:22:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6815", 2024-04-20T17:22:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 377652039 2024-04-20T17:22:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6813/377652039", 2024-04-20T17:22:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6814/377652039", 2024-04-20T17:22:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6816/377652039", 2024-04-20T17:22:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6815/377652039", 2024-04-20T17:22:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:22:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:22:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:22:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 4, 2024-04-20T17:22:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "46df16f1-9607-4d7b-9be8-9d4236e160d1", 2024-04-20T17:22:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:22:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:22:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:22:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:22:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:22:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:22:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 31, 2024-04-20T17:22:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 356, 2024-04-20T17:22:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:22:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:22:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:22:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6800", 2024-04-20T17:22:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:22:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:22:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6801", 2024-04-20T17:22:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:22:57.885 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.885 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.885 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.885 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:22:57.885 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.885 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.885 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.885 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6803", 2024-04-20T17:22:57.885 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:22:57.885 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.885 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.885 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.885 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:22:57.885 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.885 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.886 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.886 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6802", 2024-04-20T17:22:57.886 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:22:57.886 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.886 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.886 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.886 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6800/1839524469", 2024-04-20T17:22:57.886 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6801/1839524469", 2024-04-20T17:22:57.886 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6803/1839524469", 2024-04-20T17:22:57.886 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6802/1839524469", 2024-04-20T17:22:57.886 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:22:57.886 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:22:57.886 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:22:57.886 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.886 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.887 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.887 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 5, 2024-04-20T17:22:57.887 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "6603231d-1e5c-4575-bfaa-2eeca2c12215", 2024-04-20T17:22:57.887 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:22:57.887 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:22:57.887 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:22:57.887 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:22:57.887 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:22:57.887 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:22:57.887 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 37, 2024-04-20T17:22:57.887 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 368, 2024-04-20T17:22:57.887 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:22:57.887 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:22:57.887 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:22:57.888 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.888 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.888 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.888 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6804", 2024-04-20T17:22:57.888 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:22:57.888 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.888 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.888 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.888 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:22:57.888 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.888 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.888 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.888 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6805", 2024-04-20T17:22:57.888 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:22:57.889 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.889 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.889 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.889 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:22:57.889 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.889 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.889 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.889 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6807", 2024-04-20T17:22:57.889 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:22:57.889 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.889 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.889 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.889 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:22:57.889 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.889 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.890 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.890 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6806", 2024-04-20T17:22:57.890 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:22:57.890 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.890 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.890 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.890 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6804/2450315908", 2024-04-20T17:22:57.890 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6805/2450315908", 2024-04-20T17:22:57.890 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6807/2450315908", 2024-04-20T17:22:57.890 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6806/2450315908", 2024-04-20T17:22:57.890 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:22:57.890 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:22:57.890 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:22:57.891 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.891 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.891 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.891 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 6, 2024-04-20T17:22:57.891 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "783a55a6-f176-42be-8c8b-af8481d08dc9", 2024-04-20T17:22:57.891 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:22:57.891 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:22:57.891 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:22:57.891 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:22:57.891 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:22:57.891 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:22:57.891 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 48, 2024-04-20T17:22:57.891 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 356, 2024-04-20T17:22:57.892 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:22:57.892 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:22:57.892 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:22:57.892 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.892 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.892 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.892 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6808", 2024-04-20T17:22:57.892 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2840237913 2024-04-20T17:22:57.892 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.892 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.892 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.892 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:22:57.892 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.892 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.892 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.893 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6809", 2024-04-20T17:22:57.893 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2840237913 2024-04-20T17:22:57.893 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.893 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.893 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.893 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:22:57.893 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.893 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.893 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.893 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6811", 2024-04-20T17:22:57.893 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2840237913 2024-04-20T17:22:57.893 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.893 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.893 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.893 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:22:57.894 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.894 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.894 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.894 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6810", 2024-04-20T17:22:57.894 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2840237913 2024-04-20T17:22:57.894 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.894 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.894 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.894 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6808/2840237913", 2024-04-20T17:22:57.894 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6809/2840237913", 2024-04-20T17:22:57.894 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6811/2840237913", 2024-04-20T17:22:57.894 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6810/2840237913", 2024-04-20T17:22:57.894 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:22:57.895 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:22:57.895 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:22:57.895 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.895 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.895 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.895 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 7, 2024-04-20T17:22:57.895 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4", 2024-04-20T17:22:57.895 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:22:57.895 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 0, 2024-04-20T17:22:57.895 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-04-20T17:22:57.895 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:22:57.895 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 56, 2024-04-20T17:22:57.895 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 364, 2024-04-20T17:22:57.895 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 398, 2024-04-20T17:22:57.896 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 356, 2024-04-20T17:22:57.896 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 365, 2024-04-20T17:22:57.896 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:22:57.896 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:22:57.896 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.896 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.896 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.896 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6812", 2024-04-20T17:22:57.896 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1997086583 2024-04-20T17:22:57.896 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.896 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.896 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.896 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:22:57.896 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.897 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.897 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.897 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6813", 2024-04-20T17:22:57.897 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1997086583 2024-04-20T17:22:57.897 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.897 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.897 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.897 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:22:57.897 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.897 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.897 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.897 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6815", 2024-04-20T17:22:57.897 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1997086583 2024-04-20T17:22:57.897 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.897 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.898 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.898 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:22:57.898 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.898 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.898 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.898 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6814", 2024-04-20T17:22:57.898 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1997086583 2024-04-20T17:22:57.898 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.898 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.898 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.898 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6812/1997086583", 2024-04-20T17:22:57.898 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6813/1997086583", 2024-04-20T17:22:57.898 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6815/1997086583", 2024-04-20T17:22:57.899 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6814/1997086583", 2024-04-20T17:22:57.899 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:22:57.899 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:22:57.899 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:22:57.899 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.899 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.899 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.899 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 8, 2024-04-20T17:22:57.899 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "1cb232ad-1f59-4a0a-a215-e67419ad6476", 2024-04-20T17:22:57.899 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:22:57.899 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:22:57.899 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:22:57.899 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:22:57.899 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:22:57.899 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:22:57.900 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 67, 2024-04-20T17:22:57.900 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 356, 2024-04-20T17:22:57.900 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:22:57.900 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:22:57.900 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:22:57.900 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.900 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.900 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.900 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6800", 2024-04-20T17:22:57.900 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:22:57.900 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.900 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.900 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.900 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:22:57.900 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.901 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.901 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.901 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6801", 2024-04-20T17:22:57.901 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:22:57.901 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.901 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.901 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.901 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:22:57.901 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.901 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.901 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.901 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6803", 2024-04-20T17:22:57.901 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:22:57.901 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.901 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.901 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.902 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:22:57.902 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.902 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.902 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.902 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6802", 2024-04-20T17:22:57.902 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:22:57.902 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.902 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.902 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.902 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6800/4123058942", 2024-04-20T17:22:57.902 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6801/4123058942", 2024-04-20T17:22:57.902 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6803/4123058942", 2024-04-20T17:22:57.902 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6802/4123058942", 2024-04-20T17:22:57.903 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:22:57.903 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:22:57.903 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:22:57.903 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.903 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.903 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.903 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 9, 2024-04-20T17:22:57.903 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "b6c17147-b3dc-475b-bb20-f9f712fb99e8", 2024-04-20T17:22:57.903 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:22:57.903 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:22:57.903 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:22:57.903 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:22:57.903 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:22:57.903 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:22:57.904 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 73, 2024-04-20T17:22:57.904 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 196, 2024-04-20T17:22:57.904 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:22:57.904 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:22:57.904 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:22:57.904 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.904 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.904 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.904 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6804", 2024-04-20T17:22:57.904 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:22:57.904 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.904 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.904 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.905 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:22:57.905 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.905 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.905 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.905 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6805", 2024-04-20T17:22:57.905 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:22:57.905 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.905 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.905 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.905 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:22:57.905 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.905 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.905 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.905 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6807", 2024-04-20T17:22:57.905 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:22:57.906 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.906 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.906 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.906 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:22:57.906 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.906 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.906 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.906 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6806", 2024-04-20T17:22:57.906 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:22:57.906 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.906 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.906 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.906 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6804/2877589636", 2024-04-20T17:22:57.907 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6805/2877589636", 2024-04-20T17:22:57.907 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6807/2877589636", 2024-04-20T17:22:57.907 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6806/2877589636", 2024-04-20T17:22:57.907 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:22:57.907 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:22:57.907 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:22:57.907 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.907 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.907 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.907 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 10, 2024-04-20T17:22:57.907 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "1adcdf55-f002-4a39-91e7-378e5fae849c", 2024-04-20T17:22:57.907 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:22:57.907 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:22:57.907 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:22:57.908 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:22:57.908 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:22:57.908 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:22:57.908 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 84, 2024-04-20T17:22:57.908 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 0, 2024-04-20T17:22:57.908 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:22:57.908 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:22:57.908 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:22:57.908 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.908 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.908 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.908 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6808", 2024-04-20T17:22:57.908 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:22:57.908 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.909 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.909 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.909 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:22:57.909 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.909 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.909 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.909 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6809", 2024-04-20T17:22:57.909 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:22:57.909 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.909 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.910 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.910 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:22:57.910 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.910 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.910 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.910 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6811", 2024-04-20T17:22:57.910 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:22:57.910 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.910 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.910 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.910 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:22:57.910 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.910 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.910 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.911 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6810", 2024-04-20T17:22:57.911 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:22:57.911 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.911 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.911 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.911 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6808/181490621", 2024-04-20T17:22:57.911 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6809/181490621", 2024-04-20T17:22:57.911 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6811/181490621", 2024-04-20T17:22:57.911 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6810/181490621", 2024-04-20T17:22:57.911 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:22:57.911 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:22:57.911 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:22:57.911 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.911 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.912 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.912 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 11, 2024-04-20T17:22:57.912 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "737c5aec-694a-4968-819d-fccf22661f4f", 2024-04-20T17:22:57.912 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:22:57.912 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:22:57.912 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:22:57.912 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:22:57.912 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:22:57.912 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:22:57.912 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 92, 2024-04-20T17:22:57.912 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 368, 2024-04-20T17:22:57.912 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:22:57.912 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:22:57.913 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:22:57.913 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.913 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.913 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.913 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6812", 2024-04-20T17:22:57.913 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:22:57.913 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.913 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.913 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.913 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:22:57.913 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.913 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.913 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.913 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6813", 2024-04-20T17:22:57.913 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:22:57.914 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.914 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.914 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.914 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:22:57.914 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.914 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.914 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.914 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6815", 2024-04-20T17:22:57.914 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:22:57.914 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.914 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.914 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.914 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:22:57.914 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:22:57.915 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.915 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:22:57.915 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6814", 2024-04-20T17:22:57.915 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:22:57.915 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.915 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.915 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.915 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6812/3373585701", 2024-04-20T17:22:57.915 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6813/3373585701", 2024-04-20T17:22:57.915 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6815/3373585701", 2024-04-20T17:22:57.915 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6814/3373585701", 2024-04-20T17:22:57.915 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:22:57.915 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:22:57.915 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:22:57.916 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:22:57.916 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.916 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:22:57.916 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_xinfo": [ 2024-04-20T17:22:57.916 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.916 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 0, 2024-04-20T17:22:57.916 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:22:57.916 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:22:57.916 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:22:57.916 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:22:57.916 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:22:57.916 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:22:23.272339+0000", 2024-04-20T17:22:57.916 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:22:57.916 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.917 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.917 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 1, 2024-04-20T17:22:57.917 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:22:57.917 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:22:57.917 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:22:57.917 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:22:57.917 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:22:57.917 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:22:28.598520+0000", 2024-04-20T17:22:57.917 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:22:57.917 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.917 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.917 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 2, 2024-04-20T17:22:57.917 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:22:57.917 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:22:57.917 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:22:57.918 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:22:57.918 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:22:57.918 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:22:06.082223+0000", 2024-04-20T17:22:57.918 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 169 2024-04-20T17:22:57.918 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.918 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.918 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 3, 2024-04-20T17:22:57.918 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:22:57.918 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:22:57.918 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:22:57.918 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:22:57.918 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:22:57.918 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:21:35.821054+0000", 2024-04-20T17:22:57.919 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:22:57.919 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.919 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.919 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 4, 2024-04-20T17:22:57.919 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:22:57.919 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:22:57.919 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:22:57.919 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:22:57.919 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:22:57.919 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:21:41.306009+0000", 2024-04-20T17:22:57.919 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:22:57.919 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.919 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.919 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 5, 2024-04-20T17:22:57.920 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:22:57.920 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:22:57.920 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:22:57.920 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:22:57.920 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:22:57.920 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:21:53.903106+0000", 2024-04-20T17:22:57.920 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:22:57.920 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.920 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.920 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 6, 2024-04-20T17:22:57.920 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:22:57.920 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:22:57.920 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:22:57.920 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:22:57.921 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:22:57.921 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:21:32.433976+0000", 2024-04-20T17:22:57.921 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:22:57.921 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.921 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.921 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 7, 2024-04-20T17:22:57.921 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:22:57.921 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:22:57.921 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:22:57.921 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:22:57.921 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 65536, 2024-04-20T17:22:57.921 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:22:31.594263+0000", 2024-04-20T17:22:57.921 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 364 2024-04-20T17:22:57.921 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.922 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.922 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 8, 2024-04-20T17:22:57.922 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:22:57.922 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:22:57.922 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:22:57.922 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:22:57.922 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:22:57.922 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:21:30.887647+0000", 2024-04-20T17:22:57.922 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:22:57.922 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.922 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.922 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 9, 2024-04-20T17:22:57.922 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:22:57.923 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:22:57.923 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:22:57.923 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:22:57.923 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:22:57.923 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:22:12.126357+0000", 2024-04-20T17:22:57.923 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:22:57.923 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.923 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.923 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 10, 2024-04-20T17:22:57.923 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:22:57.923 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:22:57.923 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:22:57.923 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:22:57.923 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:22:57.924 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:21:48.423230+0000", 2024-04-20T17:22:57.924 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:22:57.924 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.924 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:22:57.924 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 11, 2024-04-20T17:22:57.924 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:22:57.924 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:22:57.924 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:22:57.924 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:22:57.924 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:22:57.924 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:22:54.922801+0000", 2024-04-20T17:22:57.924 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:22:57.924 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.924 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:22:57.925 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_upmap": [], 2024-04-20T17:22:57.925 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_upmap_items": [], 2024-04-20T17:22:57.925 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_upmap_primaries": [], 2024-04-20T17:22:57.925 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_temp": [], 2024-04-20T17:22:57.925 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_temp": [], 2024-04-20T17:22:57.925 INFO:teuthology.orchestra.run.smithi005.stdout: "blocklist": { 2024-04-20T17:22:57.925 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/3881726425": "2024-04-21T16:59:25.808337+0000", 2024-04-20T17:22:57.925 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/338649532": "2024-04-21T16:59:25.808337+0000", 2024-04-20T17:22:57.925 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/1227843664": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:22:57.925 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:6800/1876064631": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:22:57.925 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/160689449": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:22:57.925 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:6800/3521715286": "2024-04-21T16:59:25.808337+0000", 2024-04-20T17:22:57.925 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/3841663462": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:22:57.925 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/687690919": "2024-04-21T16:59:25.808337+0000" 2024-04-20T17:22:57.926 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.926 INFO:teuthology.orchestra.run.smithi005.stdout: "range_blocklist": {}, 2024-04-20T17:22:57.926 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_profiles": { 2024-04-20T17:22:57.926 INFO:teuthology.orchestra.run.smithi005.stdout: "default": { 2024-04-20T17:22:57.926 INFO:teuthology.orchestra.run.smithi005.stdout: "crush-failure-domain": "osd", 2024-04-20T17:22:57.926 INFO:teuthology.orchestra.run.smithi005.stdout: "k": "2", 2024-04-20T17:22:57.926 INFO:teuthology.orchestra.run.smithi005.stdout: "m": "1", 2024-04-20T17:22:57.926 INFO:teuthology.orchestra.run.smithi005.stdout: "plugin": "jerasure", 2024-04-20T17:22:57.926 INFO:teuthology.orchestra.run.smithi005.stdout: "technique": "reed_sol_van" 2024-04-20T17:22:57.926 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.926 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:22:57.926 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_snaps_queue": [], 2024-04-20T17:22:57.926 INFO:teuthology.orchestra.run.smithi005.stdout: "new_removed_snaps": [], 2024-04-20T17:22:57.926 INFO:teuthology.orchestra.run.smithi005.stdout: "new_purged_snaps": [], 2024-04-20T17:22:57.927 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_node_flags": {}, 2024-04-20T17:22:57.927 INFO:teuthology.orchestra.run.smithi005.stdout: "device_class_flags": {}, 2024-04-20T17:22:57.927 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": { 2024-04-20T17:22:57.927 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode_enabled": false, 2024-04-20T17:22:57.927 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_bucket_count": 0, 2024-04-20T17:22:57.927 INFO:teuthology.orchestra.run.smithi005.stdout: "degraded_stretch_mode": 0, 2024-04-20T17:22:57.927 INFO:teuthology.orchestra.run.smithi005.stdout: "recovering_stretch_mode": 0, 2024-04-20T17:22:57.927 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode_bucket": 0 2024-04-20T17:22:57.927 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:22:57.927 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:22:57.928 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:22:57.957 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:58.155 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:22:58.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:58.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:22:58.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:58 smithi005 ceph-mon[39562]: 3.d scrub ok 2024-04-20T17:22:58.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:58 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2852331249' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-20T17:22:58.259 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:22:58.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:58 smithi106 ceph-mon[44423]: 3.d scrub ok 2024-04-20T17:22:58.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:58 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2852331249' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-20T17:22:58.325 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:58.356 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:22:58.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:58.361 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:22:58.462 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:22:58.495 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:58 smithi171 ceph-mon[44189]: 3.d scrub ok 2024-04-20T17:22:58.495 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:58 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2852331249' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-20T17:22:58.497 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:58.533 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:22:58.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:58.539 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:22:58.640 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:22:58.681 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:58.707 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:22:58.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:58.712 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:22:58.801 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:22:58.801 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:22:58.802 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:58.802 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:211004 3.a 3:537899b6:::benchmark_data_smithi171_110765_object67789:head [delete] snapc 0=[] ondisk+write+known_if_redirected e403)", 2024-04-20T17:22:58.802 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:22:58.796472+0000", 2024-04-20T17:22:58.802 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0039951969999999998, 2024-04-20T17:22:58.802 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0040287719999999999, 2024-04-20T17:22:58.802 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:22:58.802 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:22:58.802 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:22:58.802 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:22:58.802 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:22:58.802 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:22:58.802 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 211004 2024-04-20T17:22:58.802 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:58.803 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:22:58.803 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:58.803 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:22:58.803 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:58.796472+0000", 2024-04-20T17:22:58.803 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:58.803 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:58.803 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:58.803 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:22:58.803 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:58.796472+0000", 2024-04-20T17:22:58.803 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:58.803 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:58.803 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:58.803 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:22:58.803 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:58.796475+0000", 2024-04-20T17:22:58.803 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.021e-06 2024-04-20T17:22:58.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:58.804 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:58.804 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:22:58.804 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:58.796483+0000", 2024-04-20T17:22:58.804 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2160000000000006e-06 2024-04-20T17:22:58.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:58.804 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:58.804 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:22:58.804 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:58.796501+0000", 2024-04-20T17:22:58.804 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8576e-05 2024-04-20T17:22:58.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:58.804 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:58.805 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:22:58.805 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:58.796511+0000", 2024-04-20T17:22:58.805 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0013999999999999e-05 2024-04-20T17:22:58.805 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:58.805 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:58.805 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:22:58.805 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:58.796584+0000", 2024-04-20T17:22:58.805 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2883e-05 2024-04-20T17:22:58.805 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:58.805 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:58.805 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:22:58.805 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:58.796723+0000", 2024-04-20T17:22:58.805 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013881399999999999 2024-04-20T17:22:58.805 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:58.805 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:58.806 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 4", 2024-04-20T17:22:58.806 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:58.796848+0000", 2024-04-20T17:22:58.806 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012483899999999999 2024-04-20T17:22:58.806 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:58.806 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:22:58.806 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:58.806 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:58.806 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:58.806 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:211008 3.a 3.954d79ba (undecoded) ondisk+write+known_if_redirected e403)", 2024-04-20T17:22:58.806 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:22:58.796747+0000", 2024-04-20T17:22:58.806 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0037203359999999999, 2024-04-20T17:22:58.806 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0038435140000000001, 2024-04-20T17:22:58.806 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:22:58.807 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:22:58.807 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "queued for pg", 2024-04-20T17:22:58.807 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:22:58.807 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:22:58.807 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:22:58.807 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 211008 2024-04-20T17:22:58.807 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:58.807 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:22:58.807 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:58.807 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:22:58.807 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:58.796747+0000", 2024-04-20T17:22:58.807 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:58.807 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:58.807 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:58.808 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:22:58.808 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:58.796747+0000", 2024-04-20T17:22:58.808 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:22:58.808 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:58.808 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:58.808 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:22:58.808 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:58.796748+0000", 2024-04-20T17:22:58.808 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1540000000000001e-06 2024-04-20T17:22:58.808 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:58.808 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:58.808 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:22:58.808 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:58.796752+0000", 2024-04-20T17:22:58.808 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.515e-06 2024-04-20T17:22:58.808 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:58.809 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:58.809 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:22:58.809 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:58.796761+0000", 2024-04-20T17:22:58.809 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7329999999999992e-06 2024-04-20T17:22:58.809 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:22:58.809 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:22:58.809 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:22:58.809 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:58.796767+0000", 2024-04-20T17:22:58.809 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5039999999999999e-06 2024-04-20T17:22:58.809 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:58.809 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:22:58.809 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:58.809 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:22:58.809 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-04-20T17:22:58.809 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 2 2024-04-20T17:22:58.809 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:22:58.813 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:22:58.846 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:58.970 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:22:59.107 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:22:59.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:59.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:22:59.212 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:22:59.249 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:59.371 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:22:59 smithi005 ceph-mon[39562]: pgmap v1099: 17 pgs: 17 active+clean; 4.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-20T17:22:59.407 INFO:tasks.thrashosds.thrasher:j is {'epoch': 403, 'fsid': '13c6e76c-ff37-11ee-bc93-c7b262605968', 'created': '2024-04-20T16:58:25.272557+0000', 'modified': '2024-04-20T17:22:55.123915+0000', 'last_up_change': '2024-04-20T17:22:32.243867+0000', 'last_in_change': '2024-04-20T17:20:54.169566+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 29, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-20T17:01:35.909493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.760000228881836, 'score_stable': 11.760000228881836, 'optimal_score': 0.17000000178813934, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'unique_pool_1', 'create_time': '2024-04-20T17:20:21.928963+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '360', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 0.75, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 356, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6801', 'nonce': 3901484705}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6802', 'nonce': 3901484705}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6804', 'nonce': 3901484705}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6803', 'nonce': 3901484705}]}, 'public_addr': '172.21.15.5:6801/3901484705', 'cluster_addr': '172.21.15.5:6802/3901484705', 'heartbeat_back_addr': '172.21.15.5:6804/3901484705', 'heartbeat_front_addr': '172.21.15.5:6803/3901484705', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'aed0c8eb-0e81-4e85-ab8f-a60420060656', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 356, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6805', 'nonce': 3391893914}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6806', 'nonce': 3391893914}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6808', 'nonce': 3391893914}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6807', 'nonce': 3391893914}]}, 'public_addr': '172.21.15.5:6805/3391893914', 'cluster_addr': '172.21.15.5:6806/3391893914', 'heartbeat_back_addr': '172.21.15.5:6808/3391893914', 'heartbeat_front_addr': '172.21.15.5:6807/3391893914', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f733a6a2-19c2-4c7a-af9c-04e282062bb7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 20, 'last_clean_end': 169, 'up_from': 202, 'up_thru': 202, 'down_at': 170, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6809', 'nonce': 2120275501}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6810', 'nonce': 2120275501}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6812', 'nonce': 2120275501}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6811', 'nonce': 2120275501}]}, 'public_addr': '172.21.15.5:6809/2120275501', 'cluster_addr': '172.21.15.5:6810/2120275501', 'heartbeat_back_addr': '172.21.15.5:6812/2120275501', 'heartbeat_front_addr': '172.21.15.5:6811/2120275501', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '8c92f1df-e298-4606-88ae-b8a8e79628a1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 356, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6813', 'nonce': 377652039}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6814', 'nonce': 377652039}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6816', 'nonce': 377652039}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6815', 'nonce': 377652039}]}, 'public_addr': '172.21.15.5:6813/377652039', 'cluster_addr': '172.21.15.5:6814/377652039', 'heartbeat_back_addr': '172.21.15.5:6816/377652039', 'heartbeat_front_addr': '172.21.15.5:6815/377652039', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '46df16f1-9607-4d7b-9be8-9d4236e160d1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 356, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6800', 'nonce': 1839524469}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6801', 'nonce': 1839524469}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6803', 'nonce': 1839524469}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6802', 'nonce': 1839524469}]}, 'public_addr': '172.21.15.106:6800/1839524469', 'cluster_addr': '172.21.15.106:6801/1839524469', 'heartbeat_back_addr': '172.21.15.106:6803/1839524469', 'heartbeat_front_addr': '172.21.15.106:6802/1839524469', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6603231d-1e5c-4575-bfaa-2eeca2c12215', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 368, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6804', 'nonce': 2450315908}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6805', 'nonce': 2450315908}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6807', 'nonce': 2450315908}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6806', 'nonce': 2450315908}]}, 'public_addr': '172.21.15.106:6804/2450315908', 'cluster_addr': '172.21.15.106:6805/2450315908', 'heartbeat_back_addr': '172.21.15.106:6807/2450315908', 'heartbeat_front_addr': '172.21.15.106:6806/2450315908', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '783a55a6-f176-42be-8c8b-af8481d08dc9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 356, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6808', 'nonce': 2840237913}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6809', 'nonce': 2840237913}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6811', 'nonce': 2840237913}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6810', 'nonce': 2840237913}]}, 'public_addr': '172.21.15.106:6808/2840237913', 'cluster_addr': '172.21.15.106:6809/2840237913', 'heartbeat_back_addr': '172.21.15.106:6811/2840237913', 'heartbeat_front_addr': '172.21.15.106:6810/2840237913', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 56, 'last_clean_end': 364, 'up_from': 398, 'up_thru': 356, 'down_at': 365, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6812', 'nonce': 1997086583}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6813', 'nonce': 1997086583}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6815', 'nonce': 1997086583}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6814', 'nonce': 1997086583}]}, 'public_addr': '172.21.15.106:6812/1997086583', 'cluster_addr': '172.21.15.106:6813/1997086583', 'heartbeat_back_addr': '172.21.15.106:6815/1997086583', 'heartbeat_front_addr': '172.21.15.106:6814/1997086583', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '1cb232ad-1f59-4a0a-a215-e67419ad6476', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 67, 'up_thru': 356, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6800', 'nonce': 4123058942}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6801', 'nonce': 4123058942}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6803', 'nonce': 4123058942}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6802', 'nonce': 4123058942}]}, 'public_addr': '172.21.15.171:6800/4123058942', 'cluster_addr': '172.21.15.171:6801/4123058942', 'heartbeat_back_addr': '172.21.15.171:6803/4123058942', 'heartbeat_front_addr': '172.21.15.171:6802/4123058942', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'b6c17147-b3dc-475b-bb20-f9f712fb99e8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 73, 'up_thru': 196, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6804', 'nonce': 2877589636}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6805', 'nonce': 2877589636}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6807', 'nonce': 2877589636}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6806', 'nonce': 2877589636}]}, 'public_addr': '172.21.15.171:6804/2877589636', 'cluster_addr': '172.21.15.171:6805/2877589636', 'heartbeat_back_addr': '172.21.15.171:6807/2877589636', 'heartbeat_front_addr': '172.21.15.171:6806/2877589636', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '1adcdf55-f002-4a39-91e7-378e5fae849c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 84, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6808', 'nonce': 181490621}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6809', 'nonce': 181490621}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6811', 'nonce': 181490621}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6810', 'nonce': 181490621}]}, 'public_addr': '172.21.15.171:6808/181490621', 'cluster_addr': '172.21.15.171:6809/181490621', 'heartbeat_back_addr': '172.21.15.171:6811/181490621', 'heartbeat_front_addr': '172.21.15.171:6810/181490621', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '737c5aec-694a-4968-819d-fccf22661f4f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 92, 'up_thru': 368, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6812', 'nonce': 3373585701}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6813', 'nonce': 3373585701}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6815', 'nonce': 3373585701}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6814', 'nonce': 3373585701}]}, 'public_addr': '172.21.15.171:6812/3373585701', 'cluster_addr': '172.21.15.171:6813/3373585701', 'heartbeat_back_addr': '172.21.15.171:6815/3373585701', 'heartbeat_front_addr': '172.21.15.171:6814/3373585701', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:22:23.272339+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:22:28.598520+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:22:06.082223+0000', 'dead_epoch': 169}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:21:35.821054+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:21:41.306009+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:21:53.903106+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:21:32.433976+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-20T17:22:31.594263+0000', 'dead_epoch': 364}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:21:30.887647+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:22:12.126357+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:21:48.423230+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:22:54.922801+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.5:0/3881726425': '2024-04-21T16:59:25.808337+0000', '172.21.15.5:0/338649532': '2024-04-21T16:59:25.808337+0000', '172.21.15.5:0/1227843664': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:6800/1876064631': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:0/160689449': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:6800/3521715286': '2024-04-21T16:59:25.808337+0000', '172.21.15.5:0/3841663462': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:0/687690919': '2024-04-21T16:59:25.808337+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-20T17:22:59.407 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-04-20T17:22:59.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:22:59 smithi106 ceph-mon[44423]: pgmap v1099: 17 pgs: 17 active+clean; 4.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-20T17:22:59.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:22:59 smithi171 ceph-mon[44189]: pgmap v1099: 17 pgs: 17 active+clean; 4.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-20T17:22:59.604 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:22:59.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:59.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:22:59.640 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-20T17:22:59.715 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:22:59.750 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:22:59.781 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:22:59.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:22:59.786 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:22:59.886 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:22:59.952 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:00.007 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:23:00.014 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:23:00.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:00.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:23:00.121 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:23:00.165 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:00.203 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:23:00.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:00.210 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:23:00.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:00 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:23:00.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:00 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:23:00.311 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:23:00.349 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:00.390 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:23:00.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:00.396 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:23:00.497 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:23:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:00 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:23:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:00 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:23:00.542 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:00.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:00 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:23:00.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:00 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:23:00.801 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:23:00.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:00.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:23:00.904 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:23:00.948 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:00.975 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:23:00.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:00.980 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:23:01.081 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:23:01.122 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:01.147 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:23:01.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:01.155 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:23:01.186 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:01 smithi005 ceph-mon[39562]: pgmap v1100: 17 pgs: 17 active+clean; 4.7 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-20T17:23:01.186 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:01 smithi005 ceph-mon[39562]: osdmap e404: 12 total, 12 up, 11 in 2024-04-20T17:23:01.256 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:23:01.324 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:01.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:01 smithi171 ceph-mon[44189]: pgmap v1100: 17 pgs: 17 active+clean; 4.7 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-20T17:23:01.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:01 smithi171 ceph-mon[44189]: osdmap e404: 12 total, 12 up, 11 in 2024-04-20T17:23:01.455 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:01 smithi106 ceph-mon[44423]: pgmap v1100: 17 pgs: 17 active+clean; 4.7 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-20T17:23:01.455 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:01 smithi106 ceph-mon[44423]: osdmap e404: 12 total, 12 up, 11 in 2024-04-20T17:23:01.585 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:23:01.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:01.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:23:01.625 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:23:01.654 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:23:01.684 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:23:01.690 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:23:01.738 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:23:01.757 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:01.770 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:23:01.803 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:23:01.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:01.920 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:23:01.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:23:01.921 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:23:01.921 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:23:01.921 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:23:01.932 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:23:01.963 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:23:01.995 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:23:01.997 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:23:01.997 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:23:01.997 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:23:01.997 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:23:01.997 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:23:01.997 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:23:01.998 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:23:01.998 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:23:01.998 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:23:01.998 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:23:01.998 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:23:01.998 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:23:02.022 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:23:02.064 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:02.091 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:23:02.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:02.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:23:02.094 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:23:02.094 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:23:02.194 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:23:02.211 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:23:02.236 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:02.269 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:23:02.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:02.279 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:23:02.380 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:23:02.423 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:02.450 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:23:02.452 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:02 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:23:02.452 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:02 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:23:02.452 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:02 smithi171 ceph-mon[44189]: osdmap e405: 12 total, 12 up, 11 in 2024-04-20T17:23:02.452 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:02 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3084135145' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:23:02.452 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:02 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:23:02.452 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:02 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:23:02.452 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:02 smithi171 ceph-mon[44189]: from='client.25955 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:23:02.452 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:02 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2465018662' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:23:02.453 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:02.453 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:02.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:02.454 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:23:02.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:02 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:23:02.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:02 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:23:02.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:02 smithi005 ceph-mon[39562]: osdmap e405: 12 total, 12 up, 11 in 2024-04-20T17:23:02.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:02 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3084135145' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:23:02.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:02 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:23:02.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:02 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:23:02.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:02 smithi005 ceph-mon[39562]: from='client.25955 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:23:02.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:02 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2465018662' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:23:02.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:02.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:02.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:02 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:23:02.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:02 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:23:02.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:02 smithi106 ceph-mon[44423]: osdmap e405: 12 total, 12 up, 11 in 2024-04-20T17:23:02.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:02 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3084135145' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:23:02.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:02 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:23:02.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:02 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:23:02.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:02 smithi106 ceph-mon[44423]: from='client.25955 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:23:02.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:02 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2465018662' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:23:02.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:02.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:02.554 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:23:02.595 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:02.629 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:23:02.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:02.633 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:23:02.734 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:23:02.774 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:02.809 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:23:02.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:02.812 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:23:02.898 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-20T17:23:02.913 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:23:02.964 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:03.048 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:23:03.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:03.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:23:03.152 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:23:03.192 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:03.229 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.229 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.229 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.230 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.230 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.230 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.230 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.230 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.230 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.230 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.230 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:03 smithi005 ceph-mon[39562]: pgmap v1103: 17 pgs: 17 active+clean; 4.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-20T17:23:03.230 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:03 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:23:03.230 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:03 smithi005 ceph-mon[39562]: from='client.25955 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:23:03.230 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:03 smithi005 ceph-mon[39562]: osdmap e406: 12 total, 12 up, 11 in 2024-04-20T17:23:03.231 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:23:03.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:03.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:23:03.243 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:23:03.338 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:23:03.392 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:03.430 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:23:03.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:03.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:23:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:03 smithi106 ceph-mon[44423]: pgmap v1103: 17 pgs: 17 active+clean; 4.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-20T17:23:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:03 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:23:03.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:03 smithi106 ceph-mon[44423]: from='client.25955 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:23:03.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:03 smithi106 ceph-mon[44423]: osdmap e406: 12 total, 12 up, 11 in 2024-04-20T17:23:03.534 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:23:03.575 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:03.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:03.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:03 smithi171 ceph-mon[44189]: pgmap v1103: 17 pgs: 17 active+clean; 4.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-20T17:23:03.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:03 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:23:03.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:03 smithi171 ceph-mon[44189]: from='client.25955 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:23:03.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:03 smithi171 ceph-mon[44189]: osdmap e406: 12 total, 12 up, 11 in 2024-04-20T17:23:03.599 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:23:03.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:03.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:23:03.707 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:23:03.741 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:04.190 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:23:04.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:04.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:23:04.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:04 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:23:04.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:04 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:23:04.295 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:23:04.348 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:04.377 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:23:04.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:04.386 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:23:04.408 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11] out_osds: [7] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 8, 9, 10, 11, 2, 7] 2024-04-20T17:23:04.408 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:23:04.408 INFO:tasks.thrashosds.thrasher:inject_pause on osd.10 2024-04-20T17:23:04.408 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-20T17:23:04.408 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-20T17:23:04.409 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 config set filestore_inject_stall 3 2024-04-20T17:23:04.487 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:23:04.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:04 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:23:04.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:04 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:23:04.539 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:04 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:23:04.539 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:04 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:23:04.543 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:04.566 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:23:04.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:04.571 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:23:04.672 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:23:04.714 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:04.758 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:23:04.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:04.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:23:04.810 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:23:04.863 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:23:04.909 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:04.939 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:23:04.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:04.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:23:05.048 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:23:05.085 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:05.117 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:23:05.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:05.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:23:05.222 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:23:05.259 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:05.334 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:23:05.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:05.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:23:05.344 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:05 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:23:05.344 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:05 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:23:05.344 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:05 smithi106 ceph-mon[44423]: pgmap v1105: 17 pgs: 17 active+clean; 4.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-20T17:23:05.344 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:05 smithi106 ceph-mon[44423]: osdmap e407: 12 total, 12 up, 11 in 2024-04-20T17:23:05.363 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:23:05.363 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:23:05.443 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:23:05.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:05 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:23:05.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:05 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:23:05.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:05 smithi005 ceph-mon[39562]: pgmap v1105: 17 pgs: 17 active+clean; 4.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-20T17:23:05.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:05 smithi005 ceph-mon[39562]: osdmap e407: 12 total, 12 up, 11 in 2024-04-20T17:23:05.473 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:05.478 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:05 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:23:05.478 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:05 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:23:05.478 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:05 smithi171 ceph-mon[44189]: pgmap v1105: 17 pgs: 17 active+clean; 4.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-20T17:23:05.478 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:05 smithi171 ceph-mon[44189]: osdmap e407: 12 total, 12 up, 11 in 2024-04-20T17:23:05.621 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:23:05.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:05.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:23:05.725 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:23:05.757 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:05.777 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:23:05.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:05.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:23:05.889 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:23:05.932 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:05.980 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:23:05.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:05.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:23:06.002 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:23:06.091 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:23:06.129 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-20T17:23:06.148 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:06.188 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:23:06.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:06.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:23:06.293 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:23:06.322 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:06.346 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:23:06.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:06.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:23:06.409 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:23:06.451 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:23:06.452 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:23:06.522 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:06.634 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:23:06.796 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:23:06.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:06.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:23:06.907 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:23:06.935 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:06.957 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:23:06.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:06.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:23:07.061 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:23:07.112 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:07.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:07 smithi106 ceph-mon[44423]: pgmap v1107: 17 pgs: 17 active+clean; 4.2 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.44k op/s 2024-04-20T17:23:07.274 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:23:07.274 INFO:teuthology.orchestra.run.smithi171.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-20T17:23:07.274 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:23:07.295 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:07 smithi171 ceph-mon[44189]: pgmap v1107: 17 pgs: 17 active+clean; 4.2 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.44k op/s 2024-04-20T17:23:07.370 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:23:07.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:07.381 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:07 smithi005 ceph-mon[39562]: pgmap v1107: 17 pgs: 17 active+clean; 4.2 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.44k op/s 2024-04-20T17:23:07.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:23:07.482 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:23:07.521 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:07.880 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:23:07.887 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:23:07.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:07.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:23:08.001 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:23:08.047 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:09.215 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:09 smithi171 ceph-mon[44189]: pgmap v1108: 17 pgs: 17 active+clean; 4.1 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:23:09.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:09 smithi005 ceph-mon[39562]: pgmap v1108: 17 pgs: 17 active+clean; 4.1 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:23:09.509 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:23:09.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:09 smithi106 ceph-mon[44423]: pgmap v1108: 17 pgs: 17 active+clean; 4.1 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:23:09.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:09.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:23:09.613 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:23:09.658 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:09.817 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:23:09.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:09.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:23:09.923 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:23:09.956 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:09.981 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:23:09.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:09.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:23:10.086 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:23:10.115 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:10.134 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:23:10.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:10.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:23:10.200 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:23:10.229 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:23:10.235 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:10 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:23:10.235 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:10 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:23:10.235 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:10 smithi106 ceph-mon[44423]: osdmap e408: 12 total, 12 up, 11 in 2024-04-20T17:23:10.235 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:10 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2420745205' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:23:10.235 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:10 smithi106 ceph-mon[44423]: from='client.25967 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:23:10.235 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:10 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3137410232' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:23:10.241 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:23:10.257 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:23:10.264 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:23:10.269 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:10.291 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:23:10.292 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:23:10.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:10.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:23:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:10 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:23:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:10 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:23:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:10 smithi171 ceph-mon[44189]: osdmap e408: 12 total, 12 up, 11 in 2024-04-20T17:23:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:10 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2420745205' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:23:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:10 smithi171 ceph-mon[44189]: from='client.25967 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:23:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:10 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3137410232' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:23:10.335 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:23:10.365 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:23:10.365 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:23:10.371 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:23:10.395 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:23:10.420 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:23:10.441 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:10.450 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:23:10.479 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:23:10.505 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:23:10.537 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:23:10.577 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:23:10.580 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:23:10.580 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:23:10.580 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:23:10.580 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:23:10.580 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:23:10.580 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:23:10.580 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:23:10.580 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:23:10.580 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:23:10.580 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:23:10.580 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:23:10.580 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:23:10.609 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:10 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:23:10.609 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:10 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:23:10.609 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:10 smithi005 ceph-mon[39562]: osdmap e408: 12 total, 12 up, 11 in 2024-04-20T17:23:10.609 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:10 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2420745205' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:23:10.609 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:10 smithi005 ceph-mon[39562]: from='client.25967 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:23:10.609 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:10 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3137410232' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:23:10.664 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:23:10.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:10.670 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:23:10.771 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:23:10.810 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:10.838 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:23:10.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:10.850 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:23:10.951 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:23:10.990 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:11.251 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:23:11.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:11.257 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:23:11.268 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-20T17:23:11.358 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:23:11.412 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:11.447 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:23:11.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:11.452 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:23:11.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:11 smithi005 ceph-mon[39562]: pgmap v1110: 17 pgs: 17 active+clean; 4.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-20T17:23:11.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:11 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:23:11.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:11 smithi005 ceph-mon[39562]: from='client.25967 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:23:11.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:11 smithi005 ceph-mon[39562]: osdmap e409: 12 total, 12 up, 11 in 2024-04-20T17:23:11.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:11 smithi106 ceph-mon[44423]: pgmap v1110: 17 pgs: 17 active+clean; 4.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-20T17:23:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:11 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:23:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:11 smithi106 ceph-mon[44423]: from='client.25967 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:23:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:11 smithi106 ceph-mon[44423]: osdmap e409: 12 total, 12 up, 11 in 2024-04-20T17:23:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.553 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:23:11.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:11 smithi171 ceph-mon[44189]: pgmap v1110: 17 pgs: 17 active+clean; 4.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-20T17:23:11.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:11 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:23:11.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:11 smithi171 ceph-mon[44189]: from='client.25967 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:23:11.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:11 smithi171 ceph-mon[44189]: osdmap e409: 12 total, 12 up, 11 in 2024-04-20T17:23:11.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:11.594 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:11.720 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:23:11.867 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:23:11.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:11.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:23:11.974 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:23:12.001 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:12.028 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:23:12.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:12.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:23:12.133 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:23:12.172 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:12.209 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:23:12.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:12.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:23:12.317 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:23:12.329 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:12.329 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:12.329 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:23:12.329 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:23:12.329 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:23:12.329 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:23:12.329 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:23:12.329 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:23:12.330 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:23:12.366 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:12.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:12.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:12.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:23:12.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:23:12.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:23:12.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:23:12.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:23:12.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:23:12.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:23:12.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:12.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:12.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:23:12.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:23:12.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:23:12.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:23:12.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:23:12.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:23:12.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:23:12.728 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:23:12.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:12.732 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:23:12.833 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:23:12.874 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:13.150 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:23:13.240 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:23:13.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:13.249 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:23:13.350 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:23:13.397 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:13.543 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:23:13.651 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:23:13.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:13.656 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:23:13.757 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:23:13.787 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:13.821 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:23:13.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:13.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:23:13.893 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:23:13.925 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:23:13.953 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:13.975 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:23:13.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:13.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:23:14.000 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11] out_osds: [7] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 8, 9, 10, 11, 2, 7] 2024-04-20T17:23:14.000 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:23:14.000 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-04-20T17:23:14.000 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd in 7 2024-04-20T17:23:14.033 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:23:14.033 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:23:14.033 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-04-20T17:23:14.033 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-04-20T17:23:14.033 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:23:14.079 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:23:14.123 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:14.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:13 smithi005 ceph-mon[39562]: pgmap v1112: 17 pgs: 17 active+clean; 3.8 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-20T17:23:14.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:13 smithi106 ceph-mon[44423]: pgmap v1112: 17 pgs: 17 active+clean; 3.8 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-20T17:23:14.327 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:23:14.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:14.334 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:23:14.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:13 smithi171 ceph-mon[44189]: pgmap v1112: 17 pgs: 17 active+clean; 3.8 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-20T17:23:14.435 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:23:14.471 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:14.892 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-20T17:23:14.910 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:23:14.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:14.915 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:23:15.016 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:23:15.077 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:23:15.086 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:23:15.102 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:15.138 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:23:15.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:15.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:23:15.244 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:23:15.258 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:23:15.270 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:15.291 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:23:15.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:15.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:23:15.396 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:23:15.452 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:15.584 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:23:15.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:15.588 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:23:15.689 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:23:15.732 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:16.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:15 smithi005 ceph-mon[39562]: pgmap v1113: 17 pgs: 17 active+clean; 3.7 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-20T17:23:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:15 smithi106 ceph-mon[44423]: pgmap v1113: 17 pgs: 17 active+clean; 3.7 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-20T17:23:16.294 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:23:16.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:16.299 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:23:16.299 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:15 smithi171 ceph-mon[44189]: pgmap v1113: 17 pgs: 17 active+clean; 3.7 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-20T17:23:16.400 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:23:16.430 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:16.451 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:23:16.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:16.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:23:16.555 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:23:16.583 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:16.611 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:23:16.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:16.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:23:16.716 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:23:16.748 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:16.767 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:23:16.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:16.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:23:16.874 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:23:16.903 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:16.930 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:23:16.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:16.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:23:17.036 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:23:17.080 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:17.240 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:23:17.242 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:17 smithi171 ceph-mon[44189]: pgmap v1114: 17 pgs: 17 active+clean; 3.5 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:23:17.242 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:17 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2080545358' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:23:17.243 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:17 smithi171 ceph-mon[44189]: from='client.25979 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:23:17.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:17.246 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:23:17.347 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:23:17.373 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:17 smithi106 ceph-mon[44423]: pgmap v1114: 17 pgs: 17 active+clean; 3.5 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:23:17.373 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:17 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2080545358' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:23:17.373 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:17 smithi106 ceph-mon[44423]: from='client.25979 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:23:17.375 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:17.396 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:23:17.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:17.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:23:17.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:17 smithi005 ceph-mon[39562]: pgmap v1114: 17 pgs: 17 active+clean; 3.5 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:23:17.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:17 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2080545358' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:23:17.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:17 smithi005 ceph-mon[39562]: from='client.25979 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:23:17.500 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:23:17.527 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:17.549 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:23:17.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:17.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:23:17.632 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:23:17.632 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:23:17.632 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:23:17.632 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:23:17.632 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.632 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:315959 2.b 2:da63b980:::benchmark_data_smithi171_78918_object154965:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-04-20T17:23:17.632 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:19:58.819465+0000", 2024-04-20T17:23:17.633 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 198.806791503, 2024-04-20T17:23:17.633 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.041140903, 2024-04-20T17:23:17.633 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:17.633 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:17.633 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:17.633 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:17.633 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:23:17.633 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:23:17.633 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 315959 2024-04-20T17:23:17.633 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.633 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:17.633 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.633 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:17.633 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:19:58.819465+0000", 2024-04-20T17:23:17.634 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.634 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.634 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.634 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:17.634 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:19:58.819465+0000", 2024-04-20T17:23:17.634 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.634 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.634 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.634 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:17.634 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:19:58.819468+0000", 2024-04-20T17:23:17.634 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0769999999999999e-06 2024-04-20T17:23:17.634 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.634 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.635 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:17.635 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:19:58.819476+0000", 2024-04-20T17:23:17.635 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9570000000000002e-06 2024-04-20T17:23:17.635 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.635 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.635 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:17.635 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:19:58.819490+0000", 2024-04-20T17:23:17.635 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3268000000000001e-05 2024-04-20T17:23:17.635 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.635 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.635 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:17.635 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:19:58.819498+0000", 2024-04-20T17:23:17.635 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8540000000000003e-06 2024-04-20T17:23:17.635 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.636 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.636 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:17.636 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:19:58.820995+0000", 2024-04-20T17:23:17.636 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0014966859999999999 2024-04-20T17:23:17.636 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.636 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.636 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:17.636 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:19:58.821114+0000", 2024-04-20T17:23:17.636 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011890100000000001 2024-04-20T17:23:17.636 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.636 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.636 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:23:17.636 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:19:58.821244+0000", 2024-04-20T17:23:17.636 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012984699999999999 2024-04-20T17:23:17.637 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.637 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.637 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:17.637 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:19:58.835174+0000", 2024-04-20T17:23:17.637 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013930563 2024-04-20T17:23:17.637 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.637 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.637 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:17.637 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:19:58.860517+0000", 2024-04-20T17:23:17.637 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.025342301000000001 2024-04-20T17:23:17.637 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.637 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.637 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:17.637 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:19:58.860566+0000", 2024-04-20T17:23:17.637 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8902e-05 2024-04-20T17:23:17.638 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.638 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.638 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:17.638 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:19:58.860606+0000", 2024-04-20T17:23:17.638 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0547000000000003e-05 2024-04-20T17:23:17.638 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.638 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:17.638 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.638 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.638 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.638 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:60286 3.9 3:9dc2e748:::benchmark_data_smithi171_110765_object60285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e375)", 2024-04-20T17:23:17.638 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:06.920871+0000", 2024-04-20T17:23:17.638 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 130.705385401, 2024-04-20T17:23:17.639 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.040852670000000001, 2024-04-20T17:23:17.639 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:17.639 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:17.639 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:17.639 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:17.639 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:17.639 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:17.639 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 60286 2024-04-20T17:23:17.639 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.639 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:17.639 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.639 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:17.639 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.920871+0000", 2024-04-20T17:23:17.639 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.639 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.640 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.640 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:17.640 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.920871+0000", 2024-04-20T17:23:17.640 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.640 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.640 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.640 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:17.640 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.920874+0000", 2024-04-20T17:23:17.640 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9799999999999998e-06 2024-04-20T17:23:17.640 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.640 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.640 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:17.640 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.920914+0000", 2024-04-20T17:23:17.641 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9759999999999999e-05 2024-04-20T17:23:17.641 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.641 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.641 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:17.641 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.920933+0000", 2024-04-20T17:23:17.641 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8845999999999999e-05 2024-04-20T17:23:17.641 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.641 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.641 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:17.641 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.920947+0000", 2024-04-20T17:23:17.641 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3783e-05 2024-04-20T17:23:17.641 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.642 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.642 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:17.642 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.921001+0000", 2024-04-20T17:23:17.642 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3726e-05 2024-04-20T17:23:17.642 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.642 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.642 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:17.642 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.921095+0000", 2024-04-20T17:23:17.642 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.4557999999999998e-05 2024-04-20T17:23:17.642 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.642 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.642 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:23:17.642 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.921189+0000", 2024-04-20T17:23:17.643 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3690999999999995e-05 2024-04-20T17:23:17.643 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.643 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.643 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:17.643 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.922912+0000", 2024-04-20T17:23:17.643 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0017236930000000001 2024-04-20T17:23:17.643 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.643 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.643 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:17.643 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.961608+0000", 2024-04-20T17:23:17.643 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.038695653000000003 2024-04-20T17:23:17.643 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.643 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.643 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:17.644 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.961672+0000", 2024-04-20T17:23:17.644 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4323000000000001e-05 2024-04-20T17:23:17.644 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.644 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.644 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:17.644 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.961724+0000", 2024-04-20T17:23:17.644 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1656999999999997e-05 2024-04-20T17:23:17.644 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.644 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:17.644 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.644 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.644 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.644 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:60331 3.9 3:963a7ec6:::benchmark_data_smithi171_110765_object60330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e375)", 2024-04-20T17:23:17.644 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:06.963321+0000", 2024-04-20T17:23:17.644 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 130.66293576300001, 2024-04-20T17:23:17.645 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.04670614, 2024-04-20T17:23:17.645 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:17.645 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:17.645 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:17.645 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:17.645 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:17.645 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:17.645 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 60331 2024-04-20T17:23:17.645 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.645 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:17.645 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.645 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:17.645 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.963321+0000", 2024-04-20T17:23:17.645 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.645 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.646 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.646 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:17.646 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.963321+0000", 2024-04-20T17:23:17.646 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.646 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.646 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.646 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:17.646 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.963324+0000", 2024-04-20T17:23:17.646 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.774e-06 2024-04-20T17:23:17.646 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.646 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.646 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:17.646 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.963348+0000", 2024-04-20T17:23:17.647 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3700999999999999e-05 2024-04-20T17:23:17.647 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.647 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.647 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:17.647 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.963360+0000", 2024-04-20T17:23:17.647 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.22e-05 2024-04-20T17:23:17.647 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.647 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.647 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:17.647 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.963368+0000", 2024-04-20T17:23:17.647 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1939999999999998e-06 2024-04-20T17:23:17.647 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.647 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.647 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:17.648 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.963423+0000", 2024-04-20T17:23:17.648 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5374999999999998e-05 2024-04-20T17:23:17.648 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.648 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.648 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:17.648 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.963547+0000", 2024-04-20T17:23:17.648 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012394199999999999 2024-04-20T17:23:17.648 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.648 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.648 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:23:17.648 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.963680+0000", 2024-04-20T17:23:17.648 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000132402 2024-04-20T17:23:17.648 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.648 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.648 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:17.649 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.966954+0000", 2024-04-20T17:23:17.649 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0032742380000000001 2024-04-20T17:23:17.649 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.649 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.649 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:17.649 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:07.009928+0000", 2024-04-20T17:23:17.649 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.042973650000000002 2024-04-20T17:23:17.649 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.649 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.649 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:17.649 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:07.009981+0000", 2024-04-20T17:23:17.649 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3270999999999999e-05 2024-04-20T17:23:17.649 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.649 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.649 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:17.649 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:07.010027+0000", 2024-04-20T17:23:17.650 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6393000000000001e-05 2024-04-20T17:23:17.650 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.650 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:17.650 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.650 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.650 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.650 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:60333 3.9 3:908c9452:::benchmark_data_smithi171_110765_object60332:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e375)", 2024-04-20T17:23:17.650 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:06.964890+0000", 2024-04-20T17:23:17.650 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 130.66136701799999, 2024-04-20T17:23:17.650 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.046167593999999999, 2024-04-20T17:23:17.650 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:17.650 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:17.650 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:17.650 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:17.651 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:17.651 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:17.651 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 60333 2024-04-20T17:23:17.651 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.651 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:17.651 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.651 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:17.651 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.964890+0000", 2024-04-20T17:23:17.651 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.651 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.651 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.651 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:17.651 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.964890+0000", 2024-04-20T17:23:17.651 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.652 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.652 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.652 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:17.652 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.964891+0000", 2024-04-20T17:23:17.652 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6670000000000001e-06 2024-04-20T17:23:17.652 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.652 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.652 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:17.652 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.964911+0000", 2024-04-20T17:23:17.652 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9799000000000001e-05 2024-04-20T17:23:17.652 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.652 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.652 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:17.652 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.964921+0000", 2024-04-20T17:23:17.652 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7529999999999996e-06 2024-04-20T17:23:17.653 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.653 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.653 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:17.653 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.964927+0000", 2024-04-20T17:23:17.653 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8109999999999999e-06 2024-04-20T17:23:17.653 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.653 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.653 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:17.653 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.965350+0000", 2024-04-20T17:23:17.653 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00042355300000000002 2024-04-20T17:23:17.653 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.653 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.653 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:17.653 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.965484+0000", 2024-04-20T17:23:17.653 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013349600000000001 2024-04-20T17:23:17.654 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.654 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.654 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:23:17.654 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.965632+0000", 2024-04-20T17:23:17.654 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0001482 2024-04-20T17:23:17.654 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.654 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.654 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:17.654 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.967100+0000", 2024-04-20T17:23:17.654 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0014678200000000001 2024-04-20T17:23:17.654 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.654 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.654 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:17.655 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:07.010972+0000", 2024-04-20T17:23:17.655 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043871996000000003 2024-04-20T17:23:17.655 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.655 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.655 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:17.655 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:07.011021+0000", 2024-04-20T17:23:17.655 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9329000000000002e-05 2024-04-20T17:23:17.655 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.655 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.655 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:17.655 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:07.011057+0000", 2024-04-20T17:23:17.655 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6170000000000001e-05 2024-04-20T17:23:17.655 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.655 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:17.655 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.655 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.656 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.656 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:112591 3.f e383/368 3:fc32f703:::benchmark_data_smithi171_110765_object112590:head v 383'7835, mlcod=383'7830)", 2024-04-20T17:23:17.656 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:36.569016+0000", 2024-04-20T17:23:17.656 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 101.057241249, 2024-04-20T17:23:17.656 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.039064569, 2024-04-20T17:23:17.656 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:17.656 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:17.656 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:17.656 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:17.656 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.656 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:17.656 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:36.569016+0000", 2024-04-20T17:23:17.656 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.656 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.656 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.656 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:17.657 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:36.569016+0000", 2024-04-20T17:23:17.657 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.657 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.657 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.657 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:17.657 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:36.569017+0000", 2024-04-20T17:23:17.657 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.341e-06 2024-04-20T17:23:17.657 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.657 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.657 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:17.657 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:36.569040+0000", 2024-04-20T17:23:17.657 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3309999999999999e-05 2024-04-20T17:23:17.657 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.657 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.657 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:17.657 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:36.569051+0000", 2024-04-20T17:23:17.658 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0514e-05 2024-04-20T17:23:17.658 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.658 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.658 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:17.658 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:36.569058+0000", 2024-04-20T17:23:17.658 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0820000000000001e-06 2024-04-20T17:23:17.658 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.658 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.658 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:17.658 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:36.569110+0000", 2024-04-20T17:23:17.658 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1941000000000003e-05 2024-04-20T17:23:17.658 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.658 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.658 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:17.659 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:36.569128+0000", 2024-04-20T17:23:17.659 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7788999999999999e-05 2024-04-20T17:23:17.659 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.659 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.659 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:17.659 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:36.608037+0000", 2024-04-20T17:23:17.659 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.038909060000000002 2024-04-20T17:23:17.659 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.659 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.659 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:17.659 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:36.608080+0000", 2024-04-20T17:23:17.659 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3532000000000003e-05 2024-04-20T17:23:17.659 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.660 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:17.660 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.660 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.660 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.660 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:120099 3.f e384/368 3:f273bf0a:::benchmark_data_smithi171_110765_object120098:head v 384'8997, mlcod=384'8994)", 2024-04-20T17:23:17.660 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:42.702388+0000", 2024-04-20T17:23:17.660 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 94.923869178999993, 2024-04-20T17:23:17.660 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.053590042999999997, 2024-04-20T17:23:17.660 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:17.660 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:17.660 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:17.660 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:17.660 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.660 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:17.660 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.702388+0000", 2024-04-20T17:23:17.661 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.661 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.661 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.661 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:17.661 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.702388+0000", 2024-04-20T17:23:17.661 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.661 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.661 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.661 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:17.661 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.702388+0000", 2024-04-20T17:23:17.661 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1699999999999998e-07 2024-04-20T17:23:17.661 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.661 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.661 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:17.661 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.702416+0000", 2024-04-20T17:23:17.661 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8048e-05 2024-04-20T17:23:17.662 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.662 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.662 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:17.662 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.702423+0000", 2024-04-20T17:23:17.662 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.289e-06 2024-04-20T17:23:17.662 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.662 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.662 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:17.662 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.702428+0000", 2024-04-20T17:23:17.662 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9520000000000002e-06 2024-04-20T17:23:17.662 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.662 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.662 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:17.662 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.714772+0000", 2024-04-20T17:23:17.663 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.012343597 2024-04-20T17:23:17.663 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.663 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.663 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:17.663 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.714787+0000", 2024-04-20T17:23:17.663 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4874e-05 2024-04-20T17:23:17.663 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.663 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.663 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:17.663 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.755938+0000", 2024-04-20T17:23:17.663 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.041150682000000001 2024-04-20T17:23:17.663 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.663 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.663 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:17.664 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.755978+0000", 2024-04-20T17:23:17.664 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0083999999999998e-05 2024-04-20T17:23:17.664 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.664 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:17.664 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.664 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.664 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.664 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:120120 3.f e384/368 3:fc235aad:::benchmark_data_smithi171_110765_object120119:head v 384'8998, mlcod=384'8995)", 2024-04-20T17:23:17.664 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:42.708550+0000", 2024-04-20T17:23:17.664 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 94.917706542000005, 2024-04-20T17:23:17.664 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.047461543000000002, 2024-04-20T17:23:17.664 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:17.664 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:17.664 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:17.664 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:17.665 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.665 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:17.665 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.708550+0000", 2024-04-20T17:23:17.665 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.665 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.665 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.665 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:17.665 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.708550+0000", 2024-04-20T17:23:17.665 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.665 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.665 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.665 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:17.665 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.708553+0000", 2024-04-20T17:23:17.665 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.491e-06 2024-04-20T17:23:17.665 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.666 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.666 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:17.666 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.708582+0000", 2024-04-20T17:23:17.666 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9521000000000001e-05 2024-04-20T17:23:17.666 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.666 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.666 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:17.666 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.708600+0000", 2024-04-20T17:23:17.666 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7703e-05 2024-04-20T17:23:17.666 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.666 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.666 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:17.666 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.708609+0000", 2024-04-20T17:23:17.666 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6899999999999998e-06 2024-04-20T17:23:17.667 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.667 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.667 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:17.667 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.719516+0000", 2024-04-20T17:23:17.667 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010907005000000001 2024-04-20T17:23:17.667 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.667 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.667 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:17.667 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.719540+0000", 2024-04-20T17:23:17.667 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4678e-05 2024-04-20T17:23:17.667 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.667 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.667 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:17.667 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.755983+0000", 2024-04-20T17:23:17.667 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036442863999999998 2024-04-20T17:23:17.668 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.668 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.668 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:17.668 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.756012+0000", 2024-04-20T17:23:17.668 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8591000000000001e-05 2024-04-20T17:23:17.668 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.668 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:17.668 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.668 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.668 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.668 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:120133 3.f e384/368 3:f541769c:::benchmark_data_smithi171_110765_object120132:head v 384'8999, mlcod=384'8995)", 2024-04-20T17:23:17.668 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:42.711517+0000", 2024-04-20T17:23:17.668 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 94.914740046000006, 2024-04-20T17:23:17.668 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044524068, 2024-04-20T17:23:17.669 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:17.669 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:17.669 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:17.669 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:17.669 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.669 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:17.669 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.711517+0000", 2024-04-20T17:23:17.669 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.669 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.669 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.669 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:17.669 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.711517+0000", 2024-04-20T17:23:17.669 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.669 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.669 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.670 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:17.670 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.711519+0000", 2024-04-20T17:23:17.670 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5550000000000001e-06 2024-04-20T17:23:17.670 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.670 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.670 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:17.670 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.711547+0000", 2024-04-20T17:23:17.670 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7535999999999999e-05 2024-04-20T17:23:17.670 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.670 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.670 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:17.670 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.711563+0000", 2024-04-20T17:23:17.670 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.624e-05 2024-04-20T17:23:17.670 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.671 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.671 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:17.671 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.711571+0000", 2024-04-20T17:23:17.671 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2130000000000008e-06 2024-04-20T17:23:17.671 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.671 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.671 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:17.671 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.724096+0000", 2024-04-20T17:23:17.671 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.012524926 2024-04-20T17:23:17.671 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.671 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.671 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:17.671 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.724120+0000", 2024-04-20T17:23:17.671 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3462e-05 2024-04-20T17:23:17.671 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.671 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.672 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:17.672 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.756014+0000", 2024-04-20T17:23:17.672 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.031894543999999997 2024-04-20T17:23:17.672 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.672 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.672 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:17.672 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.756041+0000", 2024-04-20T17:23:17.672 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6591999999999999e-05 2024-04-20T17:23:17.672 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.672 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:17.672 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.672 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.672 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.672 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:191644 3.9 3:9e908130:::benchmark_data_smithi171_110765_object48429:head [delete] snapc 0=[] ondisk+write+known_if_redirected e401)", 2024-04-20T17:23:17.672 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:22:42.582140+0000", 2024-04-20T17:23:17.673 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 35.044116402999997, 2024-04-20T17:23:17.673 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.035539554000000001, 2024-04-20T17:23:17.673 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:17.673 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:17.673 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:17.673 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:17.673 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:17.673 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:17.673 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 191644 2024-04-20T17:23:17.673 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.673 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:17.673 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.673 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:17.673 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:42.582140+0000", 2024-04-20T17:23:17.673 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.674 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.674 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.674 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:17.674 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:42.582140+0000", 2024-04-20T17:23:17.674 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.674 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.674 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.674 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:17.674 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:42.582142+0000", 2024-04-20T17:23:17.674 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.263e-06 2024-04-20T17:23:17.674 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.674 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.674 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:17.674 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:42.582146+0000", 2024-04-20T17:23:17.675 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6140000000000002e-06 2024-04-20T17:23:17.675 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.675 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.675 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:17.675 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:42.582155+0000", 2024-04-20T17:23:17.675 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6459999999999999e-06 2024-04-20T17:23:17.675 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.675 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.675 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:17.675 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:42.582161+0000", 2024-04-20T17:23:17.675 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4189999999999999e-06 2024-04-20T17:23:17.675 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.675 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.675 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:17.675 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:42.593244+0000", 2024-04-20T17:23:17.676 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011083005 2024-04-20T17:23:17.676 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.676 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.676 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:17.676 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:42.593359+0000", 2024-04-20T17:23:17.676 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011455099999999999 2024-04-20T17:23:17.676 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.676 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.676 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:23:17.676 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:42.593467+0000", 2024-04-20T17:23:17.676 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010849300000000001 2024-04-20T17:23:17.676 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.676 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.676 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:17.676 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:42.598130+0000", 2024-04-20T17:23:17.677 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0046630600000000001 2024-04-20T17:23:17.677 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.677 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.677 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:17.677 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:42.617601+0000", 2024-04-20T17:23:17.677 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019470489000000001 2024-04-20T17:23:17.677 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.677 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.677 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:17.677 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:42.617644+0000", 2024-04-20T17:23:17.677 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3242999999999999e-05 2024-04-20T17:23:17.677 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.677 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.677 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:17.678 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:42.617680+0000", 2024-04-20T17:23:17.678 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5771e-05 2024-04-20T17:23:17.678 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.678 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:17.678 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.678 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.678 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.678 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:210693 3.9 3:902702a5:::benchmark_data_smithi171_110765_object67478:head [delete] snapc 0=[] ondisk+write+known_if_redirected e403)", 2024-04-20T17:23:17.678 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:22:58.540883+0000", 2024-04-20T17:23:17.678 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 19.085373524000001, 2024-04-20T17:23:17.678 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.037391440999999997, 2024-04-20T17:23:17.678 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:17.678 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:17.678 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:17.678 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:17.679 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:17.679 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:17.679 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 210693 2024-04-20T17:23:17.679 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.679 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:17.679 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.679 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:17.679 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.540883+0000", 2024-04-20T17:23:17.679 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.679 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.679 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.679 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:17.679 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.540883+0000", 2024-04-20T17:23:17.679 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.679 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.680 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.680 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:17.680 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.540884+0000", 2024-04-20T17:23:17.680 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.085e-06 2024-04-20T17:23:17.680 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.680 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.680 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:17.680 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.540889+0000", 2024-04-20T17:23:17.680 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6009999999999997e-06 2024-04-20T17:23:17.680 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.680 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.680 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:17.680 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.540897+0000", 2024-04-20T17:23:17.680 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6359999999999994e-06 2024-04-20T17:23:17.681 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.681 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.681 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:17.681 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.540903+0000", 2024-04-20T17:23:17.681 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1380000000000004e-06 2024-04-20T17:23:17.681 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.681 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.681 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:17.681 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.545864+0000", 2024-04-20T17:23:17.681 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.004961043 2024-04-20T17:23:17.681 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.681 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.681 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:17.681 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.545995+0000", 2024-04-20T17:23:17.681 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013084300000000001 2024-04-20T17:23:17.682 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.682 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.682 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:23:17.682 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.546090+0000", 2024-04-20T17:23:17.682 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5228000000000004e-05 2024-04-20T17:23:17.682 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.682 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.682 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:17.682 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.567599+0000", 2024-04-20T17:23:17.682 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021508754000000001 2024-04-20T17:23:17.682 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.682 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.682 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:17.683 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.578169+0000", 2024-04-20T17:23:17.683 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010570527 2024-04-20T17:23:17.683 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.683 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.683 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:17.683 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.578232+0000", 2024-04-20T17:23:17.683 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2778000000000001e-05 2024-04-20T17:23:17.683 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.683 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.683 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:17.683 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.578275+0000", 2024-04-20T17:23:17.684 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2808000000000001e-05 2024-04-20T17:23:17.684 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.684 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:17.684 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.684 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.684 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.684 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:210694 3.9 3:9610da34:::benchmark_data_smithi171_110765_object67479:head [delete] snapc 0=[] ondisk+write+known_if_redirected e403)", 2024-04-20T17:23:17.684 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:22:58.546105+0000", 2024-04-20T17:23:17.684 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 19.080151347000001, 2024-04-20T17:23:17.684 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032271053000000001, 2024-04-20T17:23:17.684 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:17.684 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:17.684 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:17.684 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:17.684 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:17.685 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:17.685 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 210694 2024-04-20T17:23:17.685 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.685 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:17.685 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.685 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:17.685 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.546105+0000", 2024-04-20T17:23:17.685 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.685 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.685 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.685 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:17.685 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.546105+0000", 2024-04-20T17:23:17.685 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.685 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.685 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.686 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:17.686 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.546107+0000", 2024-04-20T17:23:17.686 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9970000000000001e-06 2024-04-20T17:23:17.686 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.686 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.686 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:17.686 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.546114+0000", 2024-04-20T17:23:17.686 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2929999999999997e-06 2024-04-20T17:23:17.686 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.686 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.686 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:17.686 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.546125+0000", 2024-04-20T17:23:17.686 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1052e-05 2024-04-20T17:23:17.686 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.687 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.687 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:17.687 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.546132+0000", 2024-04-20T17:23:17.687 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3780000000000003e-06 2024-04-20T17:23:17.687 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.687 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.687 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:17.687 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.549869+0000", 2024-04-20T17:23:17.687 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0037363689999999998 2024-04-20T17:23:17.687 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.687 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.687 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:17.687 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.549988+0000", 2024-04-20T17:23:17.687 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000119695 2024-04-20T17:23:17.688 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.688 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.688 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:23:17.688 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.550103+0000", 2024-04-20T17:23:17.688 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011469800000000001 2024-04-20T17:23:17.688 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.688 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.688 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:17.688 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.567605+0000", 2024-04-20T17:23:17.688 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.017501814000000001 2024-04-20T17:23:17.688 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.688 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.688 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:17.688 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.578311+0000", 2024-04-20T17:23:17.689 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.01070631 2024-04-20T17:23:17.689 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.689 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.689 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:17.689 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.578353+0000", 2024-04-20T17:23:17.689 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1668000000000002e-05 2024-04-20T17:23:17.689 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.689 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.689 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:17.689 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.578377+0000", 2024-04-20T17:23:17.689 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3779000000000001e-05 2024-04-20T17:23:17.689 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.689 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:17.689 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.689 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.689 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.690 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:210695 3.9 3:9b7ce700:::benchmark_data_smithi171_110765_object67480:head [delete] snapc 0=[] ondisk+write+known_if_redirected e403)", 2024-04-20T17:23:17.690 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:22:58.546144+0000", 2024-04-20T17:23:17.690 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 19.080113022999999, 2024-04-20T17:23:17.690 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032366497000000001, 2024-04-20T17:23:17.690 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:17.690 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:17.690 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:17.690 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:17.690 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:17.690 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:17.690 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 210695 2024-04-20T17:23:17.690 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.690 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:17.690 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.691 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:17.691 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.546144+0000", 2024-04-20T17:23:17.691 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.691 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.691 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.691 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:17.691 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.546144+0000", 2024-04-20T17:23:17.691 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.691 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.691 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.691 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:17.691 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.546144+0000", 2024-04-20T17:23:17.691 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6899999999999997e-07 2024-04-20T17:23:17.691 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.691 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.692 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:17.692 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.546147+0000", 2024-04-20T17:23:17.692 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8720000000000002e-06 2024-04-20T17:23:17.692 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.692 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.692 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:17.692 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.546153+0000", 2024-04-20T17:23:17.692 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5439999999999998e-06 2024-04-20T17:23:17.692 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.692 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.692 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:17.692 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.546159+0000", 2024-04-20T17:23:17.692 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7470000000000002e-06 2024-04-20T17:23:17.692 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.692 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.693 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:17.693 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.554769+0000", 2024-04-20T17:23:17.693 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0086102469999999997 2024-04-20T17:23:17.693 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.693 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.693 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:17.693 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.554898+0000", 2024-04-20T17:23:17.693 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012942499999999999 2024-04-20T17:23:17.693 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.693 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.693 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:23:17.693 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.555022+0000", 2024-04-20T17:23:17.693 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000123688 2024-04-20T17:23:17.694 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.694 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.694 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:17.694 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.567611+0000", 2024-04-20T17:23:17.694 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.012588564999999999 2024-04-20T17:23:17.694 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.694 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.694 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:17.694 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.578428+0000", 2024-04-20T17:23:17.694 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010817887 2024-04-20T17:23:17.694 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.694 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.694 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:17.694 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.578472+0000", 2024-04-20T17:23:17.694 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3325999999999998e-05 2024-04-20T17:23:17.695 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.695 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.695 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:17.695 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.578510+0000", 2024-04-20T17:23:17.695 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8526999999999999e-05 2024-04-20T17:23:17.695 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.695 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:17.695 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.695 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.695 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.695 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:225886 3.9 3:933d6dbe:::benchmark_data_smithi171_110765_object82671:head [delete] snapc 0=[] ondisk+write+known_if_redirected e409)", 2024-04-20T17:23:17.695 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:11.347080+0000", 2024-04-20T17:23:17.695 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 6.2791771379999997, 2024-04-20T17:23:17.695 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.033507204999999998, 2024-04-20T17:23:17.695 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:17.696 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:17.696 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:17.696 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:17.696 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:17.696 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:17.696 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 225886 2024-04-20T17:23:17.696 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.696 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:17.696 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.696 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:17.696 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:11.347080+0000", 2024-04-20T17:23:17.696 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.696 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.696 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.696 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:17.697 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:11.347080+0000", 2024-04-20T17:23:17.697 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.697 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.697 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.697 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:17.697 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:11.347082+0000", 2024-04-20T17:23:17.697 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1519999999999999e-06 2024-04-20T17:23:17.697 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.697 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.697 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:17.697 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:11.347088+0000", 2024-04-20T17:23:17.697 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6089999999999997e-06 2024-04-20T17:23:17.697 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.697 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.697 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:17.698 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:11.347105+0000", 2024-04-20T17:23:17.698 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6772000000000001e-05 2024-04-20T17:23:17.698 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.698 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.698 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:17.698 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:11.347114+0000", 2024-04-20T17:23:17.698 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.064e-06 2024-04-20T17:23:17.698 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.698 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.698 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:17.698 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:11.355834+0000", 2024-04-20T17:23:17.699 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0087200590000000005 2024-04-20T17:23:17.699 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.699 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.699 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:17.699 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:11.355970+0000", 2024-04-20T17:23:17.699 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013526099999999999 2024-04-20T17:23:17.699 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.699 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.699 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:23:17.699 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:11.356092+0000", 2024-04-20T17:23:17.699 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000122886 2024-04-20T17:23:17.699 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.699 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.699 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:17.699 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:11.376543+0000", 2024-04-20T17:23:17.700 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020450471000000001 2024-04-20T17:23:17.700 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.700 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.700 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:17.700 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:11.380476+0000", 2024-04-20T17:23:17.700 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0039333279999999998 2024-04-20T17:23:17.700 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.700 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.700 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:17.700 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:11.380535+0000", 2024-04-20T17:23:17.700 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8777999999999999e-05 2024-04-20T17:23:17.700 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.700 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.700 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:17.700 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:11.380587+0000", 2024-04-20T17:23:17.701 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1824999999999999e-05 2024-04-20T17:23:17.701 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.701 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:17.701 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.701 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.701 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.701 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:231538 3.9 3:96e48ba5:::benchmark_data_smithi171_110765_object88323:head [delete] snapc 0=[] ondisk+write+known_if_redirected e409)", 2024-04-20T17:23:17.701 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:16.289746+0000", 2024-04-20T17:23:17.701 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 1.3365105779999999, 2024-04-20T17:23:17.701 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.050879008000000003, 2024-04-20T17:23:17.701 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:17.701 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:17.701 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:17.701 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:17.701 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:17.702 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:17.702 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 231538 2024-04-20T17:23:17.702 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.702 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:17.702 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.702 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:17.702 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.289746+0000", 2024-04-20T17:23:17.702 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.702 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.702 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.702 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:17.702 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.289746+0000", 2024-04-20T17:23:17.702 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.702 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.703 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.703 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:17.703 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.289748+0000", 2024-04-20T17:23:17.703 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.361e-06 2024-04-20T17:23:17.703 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.703 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.703 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:17.703 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.289752+0000", 2024-04-20T17:23:17.703 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8149999999999996e-06 2024-04-20T17:23:17.703 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.703 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.703 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:17.703 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.289762+0000", 2024-04-20T17:23:17.703 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6150000000000003e-06 2024-04-20T17:23:17.703 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.703 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.704 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:17.704 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.289768+0000", 2024-04-20T17:23:17.704 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5759999999999996e-06 2024-04-20T17:23:17.704 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.704 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.704 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:17.704 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.293676+0000", 2024-04-20T17:23:17.704 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0039085839999999997 2024-04-20T17:23:17.704 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.704 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.704 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:17.704 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.293791+0000", 2024-04-20T17:23:17.704 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000115176 2024-04-20T17:23:17.704 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.704 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.705 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:23:17.705 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.293904+0000", 2024-04-20T17:23:17.705 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000113126 2024-04-20T17:23:17.705 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.705 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.705 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:17.705 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.304147+0000", 2024-04-20T17:23:17.705 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010242922 2024-04-20T17:23:17.705 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.705 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.705 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:17.706 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.340523+0000", 2024-04-20T17:23:17.706 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036375788999999999 2024-04-20T17:23:17.706 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.706 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.706 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:17.706 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.340578+0000", 2024-04-20T17:23:17.706 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4753999999999998e-05 2024-04-20T17:23:17.706 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.706 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.706 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:17.706 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.340625+0000", 2024-04-20T17:23:17.706 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7290000000000003e-05 2024-04-20T17:23:17.706 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.707 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:17.707 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.707 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.707 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.707 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:231545 3.9 3:9aa16b0e:::benchmark_data_smithi171_110765_object88330:head [delete] snapc 0=[] ondisk+write+known_if_redirected e409)", 2024-04-20T17:23:17.707 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:16.295242+0000", 2024-04-20T17:23:17.707 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 1.3310147190000001, 2024-04-20T17:23:17.707 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.045491156999999997, 2024-04-20T17:23:17.707 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:17.707 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:17.707 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:17.707 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:17.707 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:17.707 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:17.707 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 231545 2024-04-20T17:23:17.708 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.708 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:17.708 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.708 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:17.708 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.295242+0000", 2024-04-20T17:23:17.708 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.708 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.708 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.708 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:17.708 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.295242+0000", 2024-04-20T17:23:17.708 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.708 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.708 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.708 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:17.708 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.295244+0000", 2024-04-20T17:23:17.708 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1730000000000002e-06 2024-04-20T17:23:17.709 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.709 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.709 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:17.709 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.295251+0000", 2024-04-20T17:23:17.709 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0929999999999997e-06 2024-04-20T17:23:17.709 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.709 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.709 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:17.709 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.295263+0000", 2024-04-20T17:23:17.709 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1837e-05 2024-04-20T17:23:17.709 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.709 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.709 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:17.709 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.295271+0000", 2024-04-20T17:23:17.709 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7780000000000007e-06 2024-04-20T17:23:17.710 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.710 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.710 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:17.710 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.298979+0000", 2024-04-20T17:23:17.710 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0037076349999999999 2024-04-20T17:23:17.710 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.710 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.710 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:17.710 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.299088+0000", 2024-04-20T17:23:17.710 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000109747 2024-04-20T17:23:17.710 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.710 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.710 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:23:17.710 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.299192+0000", 2024-04-20T17:23:17.710 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010357 2024-04-20T17:23:17.711 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.711 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.711 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:17.711 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.304200+0000", 2024-04-20T17:23:17.711 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0050079340000000003 2024-04-20T17:23:17.711 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.711 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.711 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:17.711 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.340663+0000", 2024-04-20T17:23:17.711 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036463348999999999 2024-04-20T17:23:17.711 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.711 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.711 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:17.711 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.340702+0000", 2024-04-20T17:23:17.711 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8930000000000002e-05 2024-04-20T17:23:17.712 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.712 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.712 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:17.712 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.340733+0000", 2024-04-20T17:23:17.712 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1111000000000001e-05 2024-04-20T17:23:17.712 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.712 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:17.712 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.712 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.712 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.712 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:231549 3.9 3:9046f163:::benchmark_data_smithi171_110765_object88334:head [delete] snapc 0=[] ondisk+write+known_if_redirected e409)", 2024-04-20T17:23:17.712 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:16.304490+0000", 2024-04-20T17:23:17.712 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 1.321766483, 2024-04-20T17:23:17.712 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036344163999999998, 2024-04-20T17:23:17.712 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:17.713 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:17.713 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:17.713 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:17.713 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:17.713 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:17.713 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 231549 2024-04-20T17:23:17.713 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.713 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:17.713 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.713 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:17.713 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.304490+0000", 2024-04-20T17:23:17.713 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.713 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.713 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.713 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:17.714 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.304490+0000", 2024-04-20T17:23:17.714 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.714 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.714 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.714 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:17.714 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.304492+0000", 2024-04-20T17:23:17.714 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8449999999999999e-06 2024-04-20T17:23:17.714 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.714 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.714 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:17.714 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.304497+0000", 2024-04-20T17:23:17.714 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5279999999999997e-06 2024-04-20T17:23:17.714 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.714 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.715 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:17.715 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.304511+0000", 2024-04-20T17:23:17.715 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4399000000000001e-05 2024-04-20T17:23:17.715 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.715 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.715 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:17.715 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.304519+0000", 2024-04-20T17:23:17.715 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6120000000000001e-06 2024-04-20T17:23:17.715 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.715 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.715 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:17.715 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.304577+0000", 2024-04-20T17:23:17.715 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7800999999999998e-05 2024-04-20T17:23:17.715 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.715 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.716 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:17.716 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.304709+0000", 2024-04-20T17:23:17.716 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000132008 2024-04-20T17:23:17.716 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.716 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.716 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:23:17.716 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.304830+0000", 2024-04-20T17:23:17.717 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000121975 2024-04-20T17:23:17.717 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.717 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.717 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:17.717 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.313535+0000", 2024-04-20T17:23:17.717 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0087049169999999995 2024-04-20T17:23:17.717 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.717 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.717 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:17.717 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.340767+0000", 2024-04-20T17:23:17.717 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.027231397000000001 2024-04-20T17:23:17.717 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.717 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.717 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:17.717 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.340805+0000", 2024-04-20T17:23:17.718 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7706000000000001e-05 2024-04-20T17:23:17.718 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.718 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.718 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:17.718 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.340834+0000", 2024-04-20T17:23:17.718 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9975999999999999e-05 2024-04-20T17:23:17.718 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.718 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:17.718 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.718 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.718 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.718 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:231551 3.9 3:901c5336:::benchmark_data_smithi171_110765_object88336:head [delete] snapc 0=[] ondisk+write+known_if_redirected e409)", 2024-04-20T17:23:17.718 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:16.304535+0000", 2024-04-20T17:23:17.718 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 1.321721929, 2024-04-20T17:23:17.719 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036400728, 2024-04-20T17:23:17.719 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:17.719 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:17.719 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:17.719 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:17.719 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:17.719 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:17.719 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 231551 2024-04-20T17:23:17.719 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.719 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:17.719 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.719 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:17.719 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.304535+0000", 2024-04-20T17:23:17.719 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.719 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.719 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.720 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:17.720 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.304535+0000", 2024-04-20T17:23:17.720 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.720 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.720 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.720 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:17.720 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.304536+0000", 2024-04-20T17:23:17.720 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6000000000000003e-07 2024-04-20T17:23:17.720 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.720 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.720 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:17.720 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.304539+0000", 2024-04-20T17:23:17.720 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1250000000000001e-06 2024-04-20T17:23:17.720 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.720 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.721 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:17.721 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.304545+0000", 2024-04-20T17:23:17.721 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8300000000000001e-06 2024-04-20T17:23:17.721 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.721 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.721 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:17.721 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.304550+0000", 2024-04-20T17:23:17.721 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1649999999999997e-06 2024-04-20T17:23:17.721 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.721 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.721 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:17.721 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.309648+0000", 2024-04-20T17:23:17.721 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.005097776 2024-04-20T17:23:17.721 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.721 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.722 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:17.722 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.309783+0000", 2024-04-20T17:23:17.722 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013521 2024-04-20T17:23:17.722 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.722 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.722 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:23:17.722 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.309902+0000", 2024-04-20T17:23:17.722 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011974899999999999 2024-04-20T17:23:17.722 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.722 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.722 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:17.722 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.317186+0000", 2024-04-20T17:23:17.722 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0072837809999999996 2024-04-20T17:23:17.722 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.723 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.723 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:17.723 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.340868+0000", 2024-04-20T17:23:17.723 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.023682031999999999 2024-04-20T17:23:17.723 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.723 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.723 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:17.723 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.340905+0000", 2024-04-20T17:23:17.723 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7007e-05 2024-04-20T17:23:17.723 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.723 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.723 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:17.723 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.340936+0000", 2024-04-20T17:23:17.723 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0292999999999999e-05 2024-04-20T17:23:17.723 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.723 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:17.724 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.724 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.724 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.724 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:231555 3.9 3:9c48800a:::benchmark_data_smithi171_110765_object88340:head [delete] snapc 0=[] ondisk+write+known_if_redirected e409)", 2024-04-20T17:23:17.724 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:16.304621+0000", 2024-04-20T17:23:17.724 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 1.3216361779999999, 2024-04-20T17:23:17.724 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036486337000000001, 2024-04-20T17:23:17.724 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:17.724 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:17.724 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:17.724 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:17.724 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:17.725 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:17.725 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 231555 2024-04-20T17:23:17.725 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.725 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:17.725 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.725 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:17.725 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.304621+0000", 2024-04-20T17:23:17.725 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.725 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.725 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.725 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:17.725 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.304621+0000", 2024-04-20T17:23:17.725 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.725 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.725 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.726 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:17.726 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.304622+0000", 2024-04-20T17:23:17.726 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.8700000000000004e-07 2024-04-20T17:23:17.726 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.726 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.726 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:17.726 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.304626+0000", 2024-04-20T17:23:17.726 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0090000000000001e-06 2024-04-20T17:23:17.726 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.726 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.726 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:17.726 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.304632+0000", 2024-04-20T17:23:17.726 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7329999999999997e-06 2024-04-20T17:23:17.726 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.727 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.727 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:17.727 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.304638+0000", 2024-04-20T17:23:17.727 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0979999999999999e-06 2024-04-20T17:23:17.727 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.727 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.727 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:17.727 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.313567+0000", 2024-04-20T17:23:17.727 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0089286420000000005 2024-04-20T17:23:17.727 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.727 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.727 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:17.727 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.313697+0000", 2024-04-20T17:23:17.727 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013032300000000001 2024-04-20T17:23:17.728 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.728 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.728 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:23:17.728 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.313824+0000", 2024-04-20T17:23:17.728 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012699900000000001 2024-04-20T17:23:17.728 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.728 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.728 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:17.728 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.317278+0000", 2024-04-20T17:23:17.728 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0034534639999999998 2024-04-20T17:23:17.728 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.728 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.728 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:17.728 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.341029+0000", 2024-04-20T17:23:17.728 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.023750896000000001 2024-04-20T17:23:17.729 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.729 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.729 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:17.729 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.341073+0000", 2024-04-20T17:23:17.729 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4564000000000002e-05 2024-04-20T17:23:17.729 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.729 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.729 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:17.729 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.341107+0000", 2024-04-20T17:23:17.729 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3621999999999998e-05 2024-04-20T17:23:17.729 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.729 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:17.729 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.729 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.729 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.730 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:232473 3.9 3:9e76efc6:::benchmark_data_smithi171_110765_object89258:head [delete] snapc 0=[] ondisk+write+known_if_redirected e409)", 2024-04-20T17:23:17.730 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:17.048770+0000", 2024-04-20T17:23:17.730 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.57748713699999998, 2024-04-20T17:23:17.730 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.033452337999999998, 2024-04-20T17:23:17.730 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:17.730 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:17.730 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:17.730 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:17.730 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:17.730 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:17.730 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 232473 2024-04-20T17:23:17.730 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.730 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:17.730 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.731 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:17.731 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.048770+0000", 2024-04-20T17:23:17.731 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.731 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.731 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.731 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:17.731 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.048770+0000", 2024-04-20T17:23:17.731 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.731 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.731 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.731 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:17.731 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.048772+0000", 2024-04-20T17:23:17.731 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.497e-06 2024-04-20T17:23:17.731 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.731 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.732 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:17.732 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.048779+0000", 2024-04-20T17:23:17.732 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2830000000000004e-06 2024-04-20T17:23:17.732 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.732 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.732 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:17.732 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.048794+0000", 2024-04-20T17:23:17.732 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.467e-05 2024-04-20T17:23:17.732 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.732 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.732 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:17.732 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.048803+0000", 2024-04-20T17:23:17.732 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.636e-06 2024-04-20T17:23:17.732 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.732 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.733 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:17.733 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.060967+0000", 2024-04-20T17:23:17.733 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.012164002 2024-04-20T17:23:17.733 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.733 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.733 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:17.733 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.061098+0000", 2024-04-20T17:23:17.733 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000131137 2024-04-20T17:23:17.733 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.733 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.733 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:23:17.733 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.061222+0000", 2024-04-20T17:23:17.733 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000123714 2024-04-20T17:23:17.733 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.733 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.734 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:17.734 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.071604+0000", 2024-04-20T17:23:17.734 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010382087999999999 2024-04-20T17:23:17.734 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.734 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.734 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:17.734 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.082117+0000", 2024-04-20T17:23:17.734 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010513402 2024-04-20T17:23:17.734 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.734 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.734 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:17.734 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.082170+0000", 2024-04-20T17:23:17.734 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3022999999999997e-05 2024-04-20T17:23:17.734 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.735 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.735 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:17.735 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.082222+0000", 2024-04-20T17:23:17.735 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1885999999999998e-05 2024-04-20T17:23:17.735 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.735 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:17.735 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.735 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.735 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.735 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:232859 3.9 3:901cda7d:::benchmark_data_smithi171_110765_object89644:head [delete] snapc 0=[] ondisk+write+known_if_redirected e410)", 2024-04-20T17:23:17.735 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:17.407013+0000", 2024-04-20T17:23:17.735 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.21924375300000001, 2024-04-20T17:23:17.735 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032293095000000001, 2024-04-20T17:23:17.735 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:17.735 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:17.735 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:17.736 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:17.736 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:17.736 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:17.736 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 232859 2024-04-20T17:23:17.736 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.736 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:17.736 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.736 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:17.736 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.407013+0000", 2024-04-20T17:23:17.736 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.736 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.736 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.736 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:17.736 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.407013+0000", 2024-04-20T17:23:17.736 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:17.737 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.737 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.737 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:17.737 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.407015+0000", 2024-04-20T17:23:17.737 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2280000000000001e-06 2024-04-20T17:23:17.737 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.737 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.737 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:17.737 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.407022+0000", 2024-04-20T17:23:17.737 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6379999999999998e-06 2024-04-20T17:23:17.737 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.737 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.737 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:17.737 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.407033+0000", 2024-04-20T17:23:17.737 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0895e-05 2024-04-20T17:23:17.737 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.738 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.738 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:17.738 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.407042+0000", 2024-04-20T17:23:17.738 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.7849999999999997e-06 2024-04-20T17:23:17.738 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.738 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.738 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:17.738 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.414903+0000", 2024-04-20T17:23:17.738 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0078618309999999997 2024-04-20T17:23:17.738 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.738 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.738 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:17.738 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.415038+0000", 2024-04-20T17:23:17.738 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013485099999999999 2024-04-20T17:23:17.738 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.739 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.739 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:23:17.739 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.415800+0000", 2024-04-20T17:23:17.739 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000761485 2024-04-20T17:23:17.739 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.739 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.739 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:17.739 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.421991+0000", 2024-04-20T17:23:17.739 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0061916230000000003 2024-04-20T17:23:17.739 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.739 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.739 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:17.739 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.439207+0000", 2024-04-20T17:23:17.739 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.017216003000000001 2024-04-20T17:23:17.739 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.740 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.740 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:17.740 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.439260+0000", 2024-04-20T17:23:17.740 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2593000000000003e-05 2024-04-20T17:23:17.740 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:17.740 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:17.740 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:17.740 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.439306+0000", 2024-04-20T17:23:17.740 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6162999999999998e-05 2024-04-20T17:23:17.740 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.740 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:17.740 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.740 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:17.740 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:17.740 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:23:17.741 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:23:17.769 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:17.791 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:23:17.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:17.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:23:17.895 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:23:17.947 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:17.971 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:23:18.107 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:18 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:23:18.107 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:18 smithi106 ceph-mon[44423]: from='client.25979 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:23:18.107 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:18 smithi106 ceph-mon[44423]: osdmap e410: 12 total, 12 up, 11 in 2024-04-20T17:23:18.107 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:18 smithi106 ceph-mon[44423]: 3.9 scrub starts 2024-04-20T17:23:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:18 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:23:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:18 smithi005 ceph-mon[39562]: from='client.25979 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:23:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:18 smithi005 ceph-mon[39562]: osdmap e410: 12 total, 12 up, 11 in 2024-04-20T17:23:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:18 smithi005 ceph-mon[39562]: 3.9 scrub starts 2024-04-20T17:23:18.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:18 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:23:18.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:18 smithi171 ceph-mon[44189]: from='client.25979 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:23:18.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:18 smithi171 ceph-mon[44189]: osdmap e410: 12 total, 12 up, 11 in 2024-04-20T17:23:18.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:18 smithi171 ceph-mon[44189]: 3.9 scrub starts 2024-04-20T17:23:18.479 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-20T17:23:18.886 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:23:19.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:19 smithi005 ceph-mon[39562]: 3.9 scrub ok 2024-04-20T17:23:19.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:19 smithi005 ceph-mon[39562]: pgmap v1116: 17 pgs: 17 active+clean; 3.4 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-20T17:23:19.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:19 smithi005 ceph-mon[39562]: 3.6 scrub starts 2024-04-20T17:23:19.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:19 smithi005 ceph-mon[39562]: 3.6 scrub ok 2024-04-20T17:23:19.219 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:23:19.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:19.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:23:19.326 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:23:19.386 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:19.446 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:23:19.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:19.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:23:19.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:19 smithi106 ceph-mon[44423]: 3.9 scrub ok 2024-04-20T17:23:19.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:19 smithi106 ceph-mon[44423]: pgmap v1116: 17 pgs: 17 active+clean; 3.4 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-20T17:23:19.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:19 smithi106 ceph-mon[44423]: 3.6 scrub starts 2024-04-20T17:23:19.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:19 smithi106 ceph-mon[44423]: 3.6 scrub ok 2024-04-20T17:23:19.535 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:19 smithi171 ceph-mon[44189]: 3.9 scrub ok 2024-04-20T17:23:19.535 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:19 smithi171 ceph-mon[44189]: pgmap v1116: 17 pgs: 17 active+clean; 3.4 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-20T17:23:19.535 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:19 smithi171 ceph-mon[44189]: 3.6 scrub starts 2024-04-20T17:23:19.535 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:19 smithi171 ceph-mon[44189]: 3.6 scrub ok 2024-04-20T17:23:19.552 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:23:19.582 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:19.605 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:23:19.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:19.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:23:19.708 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:23:19.743 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:19.771 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:23:19.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:19.777 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:23:19.847 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:23:19.876 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:23:19.878 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:23:19.903 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:23:19.909 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:19.937 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:23:19.977 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:23:20.023 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:23:20.054 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:23:20.088 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:23:20.110 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:23:20.133 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:19 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mon-a[39539]: 2024-04-20T17:23:19.861+0000 7f73b92a1700 -1 mon.a@0(leader).osd e410 definitely_dead 0 2024-04-20T17:23:20.138 INFO:teuthology.orchestra.run.smithi005.stderr:marked in osd.7. 2024-04-20T17:23:20.149 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:23:20.174 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:23:20.199 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:23:20.200 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:23:20.200 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:23:20.201 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:23:20.201 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:23:20.201 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:23:20.201 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:23:20.201 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:23:20.201 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:23:20.201 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:23:20.201 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:23:20.201 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:23:20.201 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:23:20.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:20 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4046924135' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:23:20.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:20.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:20 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/903632339' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-04-20T17:23:20.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:20 smithi005 ceph-mon[39562]: from='client.25985 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-04-20T17:23:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:20 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4046924135' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:23:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:20 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/903632339' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-04-20T17:23:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:20 smithi106 ceph-mon[44423]: from='client.25985 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-04-20T17:23:20.527 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:20 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4046924135' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:23:20.527 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:20.527 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:20 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/903632339' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-04-20T17:23:20.527 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:20 smithi171 ceph-mon[44189]: from='client.25985 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-04-20T17:23:20.566 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:23:20.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:20.569 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:23:20.670 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:23:20.698 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:20.716 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:23:20.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:20.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:23:20.820 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:23:20.853 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:20.929 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:23:20.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:20.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:23:20.949 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-04-20T17:23:20.969 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:23:21.033 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:23:21.063 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:21.162 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:23:21.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:21.166 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:23:21.183 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:23:21.183 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:23:21.267 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:23:21.295 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:21.333 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:23:21.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:21.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:23:21.437 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:23:21.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:21 smithi005 ceph-mon[39562]: from='client.25985 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-04-20T17:23:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:21 smithi005 ceph-mon[39562]: osdmap e411: 12 total, 12 up, 12 in 2024-04-20T17:23:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:21 smithi005 ceph-mon[39562]: pgmap v1118: 17 pgs: 17 active+clean; 3.3 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.08k op/s 2024-04-20T17:23:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:21 smithi005 ceph-mon[39562]: osdmap e412: 12 total, 12 up, 12 in 2024-04-20T17:23:21.464 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:21.497 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.497 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.497 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.497 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.498 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.498 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.498 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.498 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.498 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:21 smithi171 ceph-mon[44189]: from='client.25985 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-04-20T17:23:21.498 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:21 smithi171 ceph-mon[44189]: osdmap e411: 12 total, 12 up, 12 in 2024-04-20T17:23:21.498 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.498 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:21 smithi171 ceph-mon[44189]: pgmap v1118: 17 pgs: 17 active+clean; 3.3 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.08k op/s 2024-04-20T17:23:21.498 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.498 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.498 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:21 smithi171 ceph-mon[44189]: osdmap e412: 12 total, 12 up, 12 in 2024-04-20T17:23:21.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:21 smithi106 ceph-mon[44423]: from='client.25985 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-04-20T17:23:21.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:21 smithi106 ceph-mon[44423]: osdmap e411: 12 total, 12 up, 12 in 2024-04-20T17:23:21.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:21 smithi106 ceph-mon[44423]: pgmap v1118: 17 pgs: 17 active+clean; 3.3 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.08k op/s 2024-04-20T17:23:21.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:21.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:21 smithi106 ceph-mon[44423]: osdmap e412: 12 total, 12 up, 12 in 2024-04-20T17:23:21.564 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:23:21.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:21.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:23:21.603 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:23:21.668 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:23:21.698 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:21.897 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:23:21.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:21.900 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:23:21.906 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-20T17:23:22.001 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:23:22.027 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:22.045 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:23:22.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:22.049 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:23:22.150 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:23:22.177 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:22.197 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:23:22.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:22.200 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:23:22.218 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:23:22.301 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:23:22.334 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:22.355 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:23:22.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:22.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:23:22.459 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:23:22.486 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:22.508 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:23:22.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:22.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:23:22.613 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:23:22.655 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:23.220 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:23:23.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:23.224 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:23:23.324 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:23:23.356 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:23.359 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:23 smithi106 ceph-mon[44423]: 3.f starting backfill to osd.7 from (0'0,0'0] MAX to 410'17238 2024-04-20T17:23:23.359 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:23 smithi106 ceph-mon[44423]: osdmap e413: 12 total, 12 up, 12 in 2024-04-20T17:23:23.359 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:23 smithi106 ceph-mon[44423]: 3.d starting backfill to osd.7 from (0'0,0'0] MAX to 410'14763 2024-04-20T17:23:23.359 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:23 smithi106 ceph-mon[44423]: pgmap v1121: 17 pgs: 17 active+clean; 3.2 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s 2024-04-20T17:23:23.368 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:23 smithi171 ceph-mon[44189]: 3.f starting backfill to osd.7 from (0'0,0'0] MAX to 410'17238 2024-04-20T17:23:23.368 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:23 smithi171 ceph-mon[44189]: osdmap e413: 12 total, 12 up, 12 in 2024-04-20T17:23:23.368 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:23 smithi171 ceph-mon[44189]: 3.d starting backfill to osd.7 from (0'0,0'0] MAX to 410'14763 2024-04-20T17:23:23.368 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:23 smithi171 ceph-mon[44189]: pgmap v1121: 17 pgs: 17 active+clean; 3.2 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s 2024-04-20T17:23:23.386 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:23:23.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:23.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:23:23.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:23 smithi005 ceph-mon[39562]: 3.f starting backfill to osd.7 from (0'0,0'0] MAX to 410'17238 2024-04-20T17:23:23.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:23 smithi005 ceph-mon[39562]: osdmap e413: 12 total, 12 up, 12 in 2024-04-20T17:23:23.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:23 smithi005 ceph-mon[39562]: 3.d starting backfill to osd.7 from (0'0,0'0] MAX to 410'14763 2024-04-20T17:23:23.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:23 smithi005 ceph-mon[39562]: pgmap v1121: 17 pgs: 17 active+clean; 3.2 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s 2024-04-20T17:23:23.491 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:23:23.533 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:23.598 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:23:23.757 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:23:23.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:23.763 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:23:23.864 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:23:23.918 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:23.962 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:23:23.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:23.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:23:24.062 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:23:24.068 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:23:24.101 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:24.124 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:23:24.130 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:23:24.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:24.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:23:24.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:24 smithi005 ceph-mon[39562]: osdmap e414: 12 total, 12 up, 12 in 2024-04-20T17:23:24.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:24 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/671585362' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:23:24.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:24 smithi005 ceph-mon[39562]: from='client.25997 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:23:24.238 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:23:24.284 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:24.311 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:23:24.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:24.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:23:24.415 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:23:24.451 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:24.455 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:24 smithi106 ceph-mon[44423]: osdmap e414: 12 total, 12 up, 12 in 2024-04-20T17:23:24.455 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:24 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/671585362' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:23:24.455 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:24 smithi106 ceph-mon[44423]: from='client.25997 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:23:24.482 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:23:24.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:24.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:23:24.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:24 smithi171 ceph-mon[44189]: osdmap e414: 12 total, 12 up, 12 in 2024-04-20T17:23:24.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:24 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/671585362' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:23:24.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:24 smithi171 ceph-mon[44189]: from='client.25997 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:23:24.587 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:23:24.621 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:24.648 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:23:24.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:24.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:23:24.753 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:23:24.802 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:25.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:25 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:23:25.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:25 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:23:25.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:25 smithi005 ceph-mon[39562]: from='client.25997 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:23:25.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:25 smithi005 ceph-mon[39562]: osdmap e415: 12 total, 12 up, 12 in 2024-04-20T17:23:25.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:25 smithi005 ceph-mon[39562]: pgmap v1124: 17 pgs: 1 active+remapped+backfilling, 1 active+remapped, 15 active+clean; 3.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 826 op/s; 2869/101230 objects misplaced (2.834%); 3.5 MiB/s, 56 objects/s recovering 2024-04-20T17:23:25.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:25 smithi005 ceph-mon[39562]: 3.3 scrub starts 2024-04-20T17:23:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:25 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:23:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:25 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:23:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:25 smithi106 ceph-mon[44423]: from='client.25997 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:23:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:25 smithi106 ceph-mon[44423]: osdmap e415: 12 total, 12 up, 12 in 2024-04-20T17:23:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:25 smithi106 ceph-mon[44423]: pgmap v1124: 17 pgs: 1 active+remapped+backfilling, 1 active+remapped, 15 active+clean; 3.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 826 op/s; 2869/101230 objects misplaced (2.834%); 3.5 MiB/s, 56 objects/s recovering 2024-04-20T17:23:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:25 smithi106 ceph-mon[44423]: 3.3 scrub starts 2024-04-20T17:23:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:25 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:23:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:25 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:23:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:25 smithi171 ceph-mon[44189]: from='client.25997 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:23:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:25 smithi171 ceph-mon[44189]: osdmap e415: 12 total, 12 up, 12 in 2024-04-20T17:23:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:25 smithi171 ceph-mon[44189]: pgmap v1124: 17 pgs: 1 active+remapped+backfilling, 1 active+remapped, 15 active+clean; 3.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 826 op/s; 2869/101230 objects misplaced (2.834%); 3.5 MiB/s, 56 objects/s recovering 2024-04-20T17:23:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:25 smithi171 ceph-mon[44189]: 3.3 scrub starts 2024-04-20T17:23:25.759 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:23:25.759 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:23:25.950 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 8, 9, 10, 11, 2, 7] 2024-04-20T17:23:25.950 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:23:25.950 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-20T17:23:25.950 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.000000 2024-04-20T17:23:25.951 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd primary-affinity 5 0 2024-04-20T17:23:26.109 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:26 smithi005 ceph-mon[39562]: 3.4 scrub starts 2024-04-20T17:23:26.109 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:26 smithi005 ceph-mon[39562]: 3.3 scrub ok 2024-04-20T17:23:26.109 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:26 smithi005 ceph-mon[39562]: osdmap e416: 12 total, 12 up, 12 in 2024-04-20T17:23:26.517 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:26 smithi106 ceph-mon[44423]: 3.4 scrub starts 2024-04-20T17:23:26.517 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:26 smithi106 ceph-mon[44423]: 3.3 scrub ok 2024-04-20T17:23:26.517 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:26 smithi106 ceph-mon[44423]: osdmap e416: 12 total, 12 up, 12 in 2024-04-20T17:23:26.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:26 smithi171 ceph-mon[44189]: 3.4 scrub starts 2024-04-20T17:23:26.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:26 smithi171 ceph-mon[44189]: 3.3 scrub ok 2024-04-20T17:23:26.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:26 smithi171 ceph-mon[44189]: osdmap e416: 12 total, 12 up, 12 in 2024-04-20T17:23:27.414 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:27 smithi005 ceph-mon[39562]: 3.4 scrub ok 2024-04-20T17:23:27.531 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:27 smithi005 ceph-mon[39562]: pgmap v1126: 17 pgs: 2 active+remapped+backfilling, 1 active+remapped, 14 active+clean; 3.0 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 898 op/s; 4977/96796 objects misplaced (5.142%); 16 MiB/s, 250 objects/s recovering 2024-04-20T17:23:27.532 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:27 smithi106 ceph-mon[44423]: 3.4 scrub ok 2024-04-20T17:23:27.532 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:27 smithi106 ceph-mon[44423]: pgmap v1126: 17 pgs: 2 active+remapped+backfilling, 1 active+remapped, 14 active+clean; 3.0 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 898 op/s; 4977/96796 objects misplaced (5.142%); 16 MiB/s, 250 objects/s recovering 2024-04-20T17:23:27.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:27 smithi171 ceph-mon[44189]: 3.4 scrub ok 2024-04-20T17:23:27.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:27 smithi171 ceph-mon[44189]: pgmap v1126: 17 pgs: 2 active+remapped+backfilling, 1 active+remapped, 14 active+clean; 3.0 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 898 op/s; 4977/96796 objects misplaced (5.142%); 16 MiB/s, 250 objects/s recovering 2024-04-20T17:23:27.678 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:23:27.739 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:23:27.776 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:23:27.805 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:23:27.834 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:23:27.879 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:23:27.904 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:23:27.941 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:23:27.973 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:23:28.004 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:23:28.036 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:23:28.075 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:23:28.105 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:23:28.108 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:23:28.108 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:23:28.109 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:23:28.109 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:23:28.109 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:23:28.109 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:23:28.109 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:23:28.109 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:23:28.109 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:23:28.109 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:23:28.109 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:23:28.109 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:23:28.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:28 smithi005 ceph-mon[39562]: 3.8 scrub starts 2024-04-20T17:23:28.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:28 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1817907506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:23:28.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:28.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:28 smithi106 ceph-mon[44423]: 3.8 scrub starts 2024-04-20T17:23:28.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:28 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1817907506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:23:28.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:28.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:28 smithi171 ceph-mon[44189]: 3.8 scrub starts 2024-04-20T17:23:28.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:28 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1817907506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:23:28.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.171 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:23:29.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:29 smithi005 ceph-mon[39562]: 3.8 scrub ok 2024-04-20T17:23:29.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:29 smithi005 ceph-mon[39562]: pgmap v1127: 17 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 14 active+clean; 2.9 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 851 op/s; 4977/93486 objects misplaced (5.324%); 11 MiB/s, 170 objects/s recovering 2024-04-20T17:23:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:29 smithi106 ceph-mon[44423]: 3.8 scrub ok 2024-04-20T17:23:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:29 smithi106 ceph-mon[44423]: pgmap v1127: 17 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 14 active+clean; 2.9 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 851 op/s; 4977/93486 objects misplaced (5.324%); 11 MiB/s, 170 objects/s recovering 2024-04-20T17:23:29.579 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.579 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.579 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.579 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.579 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.579 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.580 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.580 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.580 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.580 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.580 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:29.580 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:29 smithi171 ceph-mon[44189]: 3.8 scrub ok 2024-04-20T17:23:29.580 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:29 smithi171 ceph-mon[44189]: pgmap v1127: 17 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 14 active+clean; 2.9 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 851 op/s; 4977/93486 objects misplaced (5.324%); 11 MiB/s, 170 objects/s recovering 2024-04-20T17:23:30.012 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-20T17:23:30.063 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:23:30.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:30.069 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:23:30.169 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:23:30.203 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:30.233 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:23:30.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:30.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:23:30.343 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:23:30.383 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:30.405 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:23:30.409 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:23:30.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:30.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:23:30.514 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:23:30.552 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:30.568 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:23:30.574 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:23:30.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:30.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:23:30.677 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:23:30.721 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:30.749 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:23:30.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:30.757 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:23:30.858 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:23:30.896 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:30.918 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:23:30.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:30.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:23:31.022 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:23:31.059 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:31.081 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:23:31.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:31.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:23:31.091 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.5 primary-affinity to 0 (802) 2024-04-20T17:23:31.185 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:23:31.218 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:31.220 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:31 smithi106 ceph-mon[44423]: pgmap v1128: 17 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 14 active+clean; 2.8 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 805 op/s; 3638/90782 objects misplaced (4.007%); 19 MiB/s, 297 objects/s recovering 2024-04-20T17:23:31.220 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:31 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3738440738' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-04-20T17:23:31.220 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:31 smithi106 ceph-mon[44423]: from='client.26009 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-04-20T17:23:31.220 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:31 smithi106 ceph-mon[44423]: from='client.26009 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-04-20T17:23:31.220 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:31 smithi106 ceph-mon[44423]: osdmap e417: 12 total, 12 up, 12 in 2024-04-20T17:23:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:31 smithi171 ceph-mon[44189]: pgmap v1128: 17 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 14 active+clean; 2.8 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 805 op/s; 3638/90782 objects misplaced (4.007%); 19 MiB/s, 297 objects/s recovering 2024-04-20T17:23:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:31 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3738440738' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-04-20T17:23:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:31 smithi171 ceph-mon[44189]: from='client.26009 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-04-20T17:23:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:31 smithi171 ceph-mon[44189]: from='client.26009 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-04-20T17:23:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:31 smithi171 ceph-mon[44189]: osdmap e417: 12 total, 12 up, 12 in 2024-04-20T17:23:31.340 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:23:31.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:31 smithi005 ceph-mon[39562]: pgmap v1128: 17 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 14 active+clean; 2.8 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 805 op/s; 3638/90782 objects misplaced (4.007%); 19 MiB/s, 297 objects/s recovering 2024-04-20T17:23:31.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:31 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3738440738' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-04-20T17:23:31.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:31 smithi005 ceph-mon[39562]: from='client.26009 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-04-20T17:23:31.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:31 smithi005 ceph-mon[39562]: from='client.26009 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-04-20T17:23:31.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:31 smithi005 ceph-mon[39562]: osdmap e417: 12 total, 12 up, 12 in 2024-04-20T17:23:32.617 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:23:32.618 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:23:32.618 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:23:32.618 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:23:32.618 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.618 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:206257 3.d e402/368 3:bd97f094:::benchmark_data_smithi171_110765_object63042:head v 402'12918, mlcod=402'12916)", 2024-04-20T17:23:32.618 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:22:54.824250+0000", 2024-04-20T17:23:32.618 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 37.780488290000001, 2024-04-20T17:23:32.618 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021553322999999999, 2024-04-20T17:23:32.621 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:32.622 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:32.622 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:32.622 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:32.622 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.622 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:32.622 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.824250+0000", 2024-04-20T17:23:32.622 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.622 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.623 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.623 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:32.623 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.824250+0000", 2024-04-20T17:23:32.623 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.623 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.623 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.623 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:32.623 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.824251+0000", 2024-04-20T17:23:32.623 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.389e-06 2024-04-20T17:23:32.623 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.623 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.623 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:32.623 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.824259+0000", 2024-04-20T17:23:32.624 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5850000000000002e-06 2024-04-20T17:23:32.624 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.624 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.624 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:32.624 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.824270+0000", 2024-04-20T17:23:32.624 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.065e-05 2024-04-20T17:23:32.625 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.625 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.625 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:32.625 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.824275+0000", 2024-04-20T17:23:32.625 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6280000000000002e-06 2024-04-20T17:23:32.625 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.625 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.625 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:32.625 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.824334+0000", 2024-04-20T17:23:32.625 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8335999999999998e-05 2024-04-20T17:23:32.625 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.625 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.625 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:32.626 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.824358+0000", 2024-04-20T17:23:32.626 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4273e-05 2024-04-20T17:23:32.626 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.626 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.626 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:32.626 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.845766+0000", 2024-04-20T17:23:32.626 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021407698999999999 2024-04-20T17:23:32.626 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.626 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.626 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:32.626 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.845803+0000", 2024-04-20T17:23:32.627 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7762999999999997e-05 2024-04-20T17:23:32.627 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.627 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:32.627 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.627 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.627 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.627 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:206263 3.d e402/368 3:b0606613:::benchmark_data_smithi171_110765_object63048:head v 402'12919, mlcod=402'12916)", 2024-04-20T17:23:32.627 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:22:54.824293+0000", 2024-04-20T17:23:32.627 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 37.780445086999997, 2024-04-20T17:23:32.627 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021543014999999999, 2024-04-20T17:23:32.627 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:32.627 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:32.627 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:32.628 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:32.628 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.628 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:32.628 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.824293+0000", 2024-04-20T17:23:32.628 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.628 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.628 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.628 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:32.628 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.824293+0000", 2024-04-20T17:23:32.628 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.628 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.628 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.628 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:32.629 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.824294+0000", 2024-04-20T17:23:32.629 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0699999999999997e-07 2024-04-20T17:23:32.629 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.629 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.629 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:32.629 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.824299+0000", 2024-04-20T17:23:32.629 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2290000000000002e-06 2024-04-20T17:23:32.629 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.629 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.629 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:32.629 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.824305+0000", 2024-04-20T17:23:32.629 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5230000000000003e-06 2024-04-20T17:23:32.630 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.630 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.630 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:32.630 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.824309+0000", 2024-04-20T17:23:32.630 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8559999999999998e-06 2024-04-20T17:23:32.630 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.630 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.630 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:32.630 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.828992+0000", 2024-04-20T17:23:32.630 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0046829569999999997 2024-04-20T17:23:32.630 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.630 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.630 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:32.630 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.829030+0000", 2024-04-20T17:23:32.631 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7568999999999999e-05 2024-04-20T17:23:32.631 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.631 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.631 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:32.631 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.845810+0000", 2024-04-20T17:23:32.631 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.016779577 2024-04-20T17:23:32.631 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.631 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.631 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:32.631 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.845836+0000", 2024-04-20T17:23:32.631 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6797000000000001e-05 2024-04-20T17:23:32.631 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.631 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:32.631 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.632 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.632 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.632 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:206267 3.d e402/368 3:b393411c:::benchmark_data_smithi171_110765_object63052:head v 402'12920, mlcod=402'12916)", 2024-04-20T17:23:32.632 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:22:54.826982+0000", 2024-04-20T17:23:32.632 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 37.777756255, 2024-04-20T17:23:32.632 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.023434604000000001, 2024-04-20T17:23:32.632 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:32.632 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:32.632 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:32.632 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:32.632 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.632 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:32.632 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.826982+0000", 2024-04-20T17:23:32.632 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.633 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.633 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.633 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:32.633 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.826982+0000", 2024-04-20T17:23:32.633 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.633 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.633 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.633 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:32.633 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.826984+0000", 2024-04-20T17:23:32.633 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8789999999999999e-06 2024-04-20T17:23:32.633 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.633 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.633 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:32.633 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.826994+0000", 2024-04-20T17:23:32.634 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0101999999999999e-05 2024-04-20T17:23:32.634 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.634 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.634 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:32.634 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.827005+0000", 2024-04-20T17:23:32.634 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0868e-05 2024-04-20T17:23:32.634 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.634 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.634 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:32.634 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.827013+0000", 2024-04-20T17:23:32.634 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1480000000000006e-06 2024-04-20T17:23:32.634 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.634 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.634 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:32.634 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.833552+0000", 2024-04-20T17:23:32.635 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0065387359999999999 2024-04-20T17:23:32.635 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.635 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.635 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:32.635 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.833574+0000", 2024-04-20T17:23:32.635 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2531000000000002e-05 2024-04-20T17:23:32.635 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.635 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.635 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:32.635 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.850381+0000", 2024-04-20T17:23:32.635 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.016806320999999999 2024-04-20T17:23:32.635 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.635 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.635 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:32.635 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:54.850417+0000", 2024-04-20T17:23:32.636 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6019000000000002e-05 2024-04-20T17:23:32.636 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.636 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:32.636 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.636 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.636 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.636 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:208038 3.d e403/368 3:b71f10ce:::benchmark_data_smithi171_110765_object64823:head v 403'13044, mlcod=403'13039)", 2024-04-20T17:23:32.636 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:22:56.270367+0000", 2024-04-20T17:23:32.636 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 36.334371175000001, 2024-04-20T17:23:32.636 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022439994000000001, 2024-04-20T17:23:32.636 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:32.637 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:32.637 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:32.637 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:32.637 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.637 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:32.637 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:56.270367+0000", 2024-04-20T17:23:32.637 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.637 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.637 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.637 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:32.637 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:56.270367+0000", 2024-04-20T17:23:32.637 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.637 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.637 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.637 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:32.638 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:56.270368+0000", 2024-04-20T17:23:32.638 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.218e-06 2024-04-20T17:23:32.638 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.638 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.638 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:32.638 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:56.270375+0000", 2024-04-20T17:23:32.638 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9229999999999996e-06 2024-04-20T17:23:32.638 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.638 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.638 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:32.638 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:56.270385+0000", 2024-04-20T17:23:32.638 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9890000000000004e-06 2024-04-20T17:23:32.638 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.638 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.639 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:32.639 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:56.270392+0000", 2024-04-20T17:23:32.639 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9949999999999999e-06 2024-04-20T17:23:32.639 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.639 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.639 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:32.639 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:56.277093+0000", 2024-04-20T17:23:32.639 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0067003310000000003 2024-04-20T17:23:32.639 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.639 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.640 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:32.640 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:56.277113+0000", 2024-04-20T17:23:32.640 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0798999999999999e-05 2024-04-20T17:23:32.640 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.640 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.640 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:32.640 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:56.292764+0000", 2024-04-20T17:23:32.640 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015650872 2024-04-20T17:23:32.640 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.640 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.640 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:32.640 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:56.292807+0000", 2024-04-20T17:23:32.641 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2867000000000001e-05 2024-04-20T17:23:32.641 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.641 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:32.641 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.641 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.641 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.641 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:209901 3.d e403/368 3:bfd84229:::benchmark_data_smithi171_110765_object66686:head v 403'13146, mlcod=403'13143)", 2024-04-20T17:23:32.641 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:22:57.886635+0000", 2024-04-20T17:23:32.641 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 34.718103022000001, 2024-04-20T17:23:32.641 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022200475000000001, 2024-04-20T17:23:32.641 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:32.641 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:32.641 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:32.641 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:32.642 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.642 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:32.642 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:57.886635+0000", 2024-04-20T17:23:32.642 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.642 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.642 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.642 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:32.642 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:57.886635+0000", 2024-04-20T17:23:32.642 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.642 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.642 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.643 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:32.643 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:57.886637+0000", 2024-04-20T17:23:32.643 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1739999999999999e-06 2024-04-20T17:23:32.643 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.643 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.643 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:32.643 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:57.886644+0000", 2024-04-20T17:23:32.643 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5000000000000002e-06 2024-04-20T17:23:32.643 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.643 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.644 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:32.644 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:57.886652+0000", 2024-04-20T17:23:32.644 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2190000000000004e-06 2024-04-20T17:23:32.645 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.645 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.645 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:32.645 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:57.886659+0000", 2024-04-20T17:23:32.645 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7599999999999997e-06 2024-04-20T17:23:32.645 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.645 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.645 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:32.645 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:57.886712+0000", 2024-04-20T17:23:32.645 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2991999999999997e-05 2024-04-20T17:23:32.645 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.645 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.646 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:32.646 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:57.886732+0000", 2024-04-20T17:23:32.646 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0333e-05 2024-04-20T17:23:32.646 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.646 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.646 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:32.646 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:57.908788+0000", 2024-04-20T17:23:32.646 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022056163 2024-04-20T17:23:32.646 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.646 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.647 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:32.647 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:57.908836+0000", 2024-04-20T17:23:32.647 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7333999999999998e-05 2024-04-20T17:23:32.647 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.647 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:32.647 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.647 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.647 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.648 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:210685 3.0 e403/356 3:09e3c6a9:::benchmark_data_smithi171_110765_object67470:head v 403'13102, mlcod=403'13098)", 2024-04-20T17:23:32.648 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:22:58.540314+0000", 2024-04-20T17:23:32.648 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 34.064424514000002, 2024-04-20T17:23:32.648 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020588840000000001, 2024-04-20T17:23:32.648 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:32.648 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:32.648 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:32.648 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:32.648 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.648 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:32.648 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.540314+0000", 2024-04-20T17:23:32.649 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.649 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.649 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.649 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:32.649 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.540314+0000", 2024-04-20T17:23:32.649 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.649 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.649 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.649 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:32.649 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.540314+0000", 2024-04-20T17:23:32.649 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0999999999999999e-07 2024-04-20T17:23:32.649 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.649 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.650 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:32.650 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.540319+0000", 2024-04-20T17:23:32.650 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4469999999999999e-06 2024-04-20T17:23:32.650 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.650 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.650 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:32.650 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.540323+0000", 2024-04-20T17:23:32.650 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6079999999999998e-06 2024-04-20T17:23:32.650 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.650 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.650 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:32.650 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.540328+0000", 2024-04-20T17:23:32.651 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4390000000000002e-06 2024-04-20T17:23:32.651 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.651 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.651 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:32.651 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.549929+0000", 2024-04-20T17:23:32.651 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0096012690000000008 2024-04-20T17:23:32.651 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.651 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.651 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:32.651 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.549952+0000", 2024-04-20T17:23:32.651 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2863000000000001e-05 2024-04-20T17:23:32.652 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.652 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.652 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:32.652 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.560857+0000", 2024-04-20T17:23:32.652 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010905214999999999 2024-04-20T17:23:32.652 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.652 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.652 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:32.652 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:58.560903+0000", 2024-04-20T17:23:32.652 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5488999999999998e-05 2024-04-20T17:23:32.652 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.652 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:32.653 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.653 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.653 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.653 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:211635 3.d e403/368 3:b3c2171b:::benchmark_data_smithi171_110765_object68420:head v 403'13251, mlcod=403'13249)", 2024-04-20T17:23:32.653 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:22:59.295963+0000", 2024-04-20T17:23:32.653 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 33.308775830999998, 2024-04-20T17:23:32.653 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021022299000000001, 2024-04-20T17:23:32.654 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:32.654 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:32.654 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:32.654 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:32.654 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.654 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:32.654 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:59.295963+0000", 2024-04-20T17:23:32.654 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.654 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.654 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.655 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:32.655 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:59.295963+0000", 2024-04-20T17:23:32.655 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.655 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.655 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.655 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:32.655 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:59.295964+0000", 2024-04-20T17:23:32.655 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9230000000000001e-06 2024-04-20T17:23:32.655 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.655 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.655 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:32.655 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:59.295974+0000", 2024-04-20T17:23:32.656 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1509999999999993e-06 2024-04-20T17:23:32.656 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.656 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.656 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:32.656 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:59.295986+0000", 2024-04-20T17:23:32.656 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2836999999999999e-05 2024-04-20T17:23:32.656 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.656 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.656 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:32.656 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:59.295994+0000", 2024-04-20T17:23:32.656 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3379999999999999e-06 2024-04-20T17:23:32.656 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.657 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.657 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:32.657 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:59.296064+0000", 2024-04-20T17:23:32.657 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9772999999999998e-05 2024-04-20T17:23:32.657 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.657 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.657 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:32.657 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:59.296087+0000", 2024-04-20T17:23:32.657 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3244999999999999e-05 2024-04-20T17:23:32.657 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.657 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.657 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:32.657 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:59.316953+0000", 2024-04-20T17:23:32.657 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020866177 2024-04-20T17:23:32.658 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.658 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.658 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:32.658 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:22:59.316985+0000", 2024-04-20T17:23:32.658 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1854999999999999e-05 2024-04-20T17:23:32.658 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.658 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:32.658 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.658 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.658 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.658 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:212922 3.d e404/368 3:b89ef344:::benchmark_data_smithi171_110765_object69707:head v 404'13350, mlcod=404'13347)", 2024-04-20T17:23:32.658 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:00.372321+0000", 2024-04-20T17:23:32.659 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 32.232417853000001, 2024-04-20T17:23:32.659 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020571612, 2024-04-20T17:23:32.659 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:32.659 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:32.659 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:32.659 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:32.659 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.659 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:32.659 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:00.372321+0000", 2024-04-20T17:23:32.659 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.659 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.659 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.659 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:32.660 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:00.372321+0000", 2024-04-20T17:23:32.660 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.660 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.660 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.660 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:32.660 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:00.372321+0000", 2024-04-20T17:23:32.660 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9400000000000005e-07 2024-04-20T17:23:32.660 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.660 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.660 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:32.660 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:00.372326+0000", 2024-04-20T17:23:32.660 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7360000000000001e-06 2024-04-20T17:23:32.660 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.660 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.661 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:32.661 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:00.372331+0000", 2024-04-20T17:23:32.661 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0799999999999996e-06 2024-04-20T17:23:32.661 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.661 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.661 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:32.661 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:00.372336+0000", 2024-04-20T17:23:32.661 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4940000000000003e-06 2024-04-20T17:23:32.661 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.661 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.661 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:32.662 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:00.379128+0000", 2024-04-20T17:23:32.662 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0067920599999999999 2024-04-20T17:23:32.662 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.662 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.662 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:32.662 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:00.379154+0000", 2024-04-20T17:23:32.662 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5622e-05 2024-04-20T17:23:32.662 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.662 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.662 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:32.662 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:00.392852+0000", 2024-04-20T17:23:32.662 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013697526999999999 2024-04-20T17:23:32.663 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.663 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.663 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:32.663 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:00.392892+0000", 2024-04-20T17:23:32.663 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0499e-05 2024-04-20T17:23:32.663 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.663 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:32.663 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.663 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.663 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.663 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:227249 3.2 e409/356 3:42e7caa9:::benchmark_data_smithi171_110765_object84034:head v 409'14057, mlcod=409'14054)", 2024-04-20T17:23:32.663 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:12.569486+0000", 2024-04-20T17:23:32.663 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 20.035252580000002, 2024-04-20T17:23:32.663 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020961588, 2024-04-20T17:23:32.664 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:32.664 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:32.664 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:32.664 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:32.664 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.664 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:32.664 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:12.569486+0000", 2024-04-20T17:23:32.664 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.664 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.664 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.664 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:32.664 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:12.569486+0000", 2024-04-20T17:23:32.664 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.665 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.665 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.665 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:32.665 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:12.569486+0000", 2024-04-20T17:23:32.665 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1799999999999995e-07 2024-04-20T17:23:32.665 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.665 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.665 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:32.665 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:12.569491+0000", 2024-04-20T17:23:32.665 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9570000000000001e-06 2024-04-20T17:23:32.665 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.665 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.665 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:32.666 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:12.569503+0000", 2024-04-20T17:23:32.666 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1622e-05 2024-04-20T17:23:32.666 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.666 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.666 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:32.666 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:12.569508+0000", 2024-04-20T17:23:32.666 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4990000000000002e-06 2024-04-20T17:23:32.666 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.666 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.666 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:32.666 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:12.574137+0000", 2024-04-20T17:23:32.666 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0046282169999999996 2024-04-20T17:23:32.667 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.667 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.667 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:32.667 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:12.574159+0000", 2024-04-20T17:23:32.667 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2634999999999999e-05 2024-04-20T17:23:32.667 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.667 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.667 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:32.667 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:12.590405+0000", 2024-04-20T17:23:32.667 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.016245701000000001 2024-04-20T17:23:32.667 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.667 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.667 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:32.668 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:12.590447+0000", 2024-04-20T17:23:32.668 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2438999999999998e-05 2024-04-20T17:23:32.668 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.668 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:32.668 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.668 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.668 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.668 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:227639 3.d e409/368 3:bb05d80c:::benchmark_data_smithi171_110765_object84424:head v 409'14237, mlcod=409'14234)", 2024-04-20T17:23:32.668 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:12.885357+0000", 2024-04-20T17:23:32.668 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 19.719381746, 2024-04-20T17:23:32.668 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021647244, 2024-04-20T17:23:32.668 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:32.668 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:32.668 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:32.669 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:32.669 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.669 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:32.669 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:12.885357+0000", 2024-04-20T17:23:32.669 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.669 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.669 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.669 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:32.669 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:12.885357+0000", 2024-04-20T17:23:32.669 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.669 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.669 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.669 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:32.669 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:12.885359+0000", 2024-04-20T17:23:32.670 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.599e-06 2024-04-20T17:23:32.670 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.670 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.670 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:32.670 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:12.885372+0000", 2024-04-20T17:23:32.670 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2361000000000001e-05 2024-04-20T17:23:32.670 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.670 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.670 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:32.670 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:12.885385+0000", 2024-04-20T17:23:32.670 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3747000000000001e-05 2024-04-20T17:23:32.670 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.670 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.671 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:32.671 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:12.885396+0000", 2024-04-20T17:23:32.671 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0426e-05 2024-04-20T17:23:32.671 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.671 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.671 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:32.671 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:12.885453+0000", 2024-04-20T17:23:32.671 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6947999999999998e-05 2024-04-20T17:23:32.671 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.671 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.671 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:32.671 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:12.885471+0000", 2024-04-20T17:23:32.671 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8261e-05 2024-04-20T17:23:32.672 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.672 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.672 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:32.672 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:12.906957+0000", 2024-04-20T17:23:32.672 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021485605000000001 2024-04-20T17:23:32.672 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.672 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.672 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:32.672 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:12.907004+0000", 2024-04-20T17:23:32.672 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7296999999999998e-05 2024-04-20T17:23:32.672 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.672 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:32.672 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.672 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.673 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.673 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:227999 3.d e409/368 3:b91c7e6a:::benchmark_data_smithi171_110765_object84784:head v 409'14265, mlcod=409'14262)", 2024-04-20T17:23:32.673 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:13.174541+0000", 2024-04-20T17:23:32.673 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 19.430197136, 2024-04-20T17:23:32.673 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021139306, 2024-04-20T17:23:32.673 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:32.673 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:32.673 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:32.673 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:32.673 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.673 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:32.673 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:13.174541+0000", 2024-04-20T17:23:32.673 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.674 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.674 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.674 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:32.674 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:13.174541+0000", 2024-04-20T17:23:32.674 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.674 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.674 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.674 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:32.674 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:13.174543+0000", 2024-04-20T17:23:32.674 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.55e-06 2024-04-20T17:23:32.674 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.674 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.674 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:32.675 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:13.174551+0000", 2024-04-20T17:23:32.675 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9689999999999994e-06 2024-04-20T17:23:32.675 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.675 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.675 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:32.675 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:13.174562+0000", 2024-04-20T17:23:32.675 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1127999999999999e-05 2024-04-20T17:23:32.675 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.675 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.675 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:32.675 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:13.174568+0000", 2024-04-20T17:23:32.675 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7849999999999997e-06 2024-04-20T17:23:32.676 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.676 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.676 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:32.676 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:13.176770+0000", 2024-04-20T17:23:32.676 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0022018680000000001 2024-04-20T17:23:32.676 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.676 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.676 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:32.676 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:13.176788+0000", 2024-04-20T17:23:32.676 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8882e-05 2024-04-20T17:23:32.676 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.676 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.676 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:32.676 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:13.195639+0000", 2024-04-20T17:23:32.677 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018850636 2024-04-20T17:23:32.677 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.677 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.677 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:32.677 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:13.195681+0000", 2024-04-20T17:23:32.677 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1488e-05 2024-04-20T17:23:32.677 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.677 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:32.677 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.677 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.677 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.677 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:228007 3.d e409/368 3:bb67d5b4:::benchmark_data_smithi171_110765_object84792:head v 409'14266, mlcod=409'14262)", 2024-04-20T17:23:32.677 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:13.177393+0000", 2024-04-20T17:23:32.677 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 19.427345603999999, 2024-04-20T17:23:32.678 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021883307000000001, 2024-04-20T17:23:32.678 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:32.678 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:32.678 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:32.678 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:32.678 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.678 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:32.678 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:13.177393+0000", 2024-04-20T17:23:32.678 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.678 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.678 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.678 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:32.678 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:13.177393+0000", 2024-04-20T17:23:32.679 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.679 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.679 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.679 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:32.679 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:13.177394+0000", 2024-04-20T17:23:32.679 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.389e-06 2024-04-20T17:23:32.679 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.679 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.679 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:32.679 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:13.177402+0000", 2024-04-20T17:23:32.679 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1270000000000003e-06 2024-04-20T17:23:32.679 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.680 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.680 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:32.680 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:13.177413+0000", 2024-04-20T17:23:32.680 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1178e-05 2024-04-20T17:23:32.680 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.680 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.680 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:32.680 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:13.177421+0000", 2024-04-20T17:23:32.680 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4440000000000002e-06 2024-04-20T17:23:32.680 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.680 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.680 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:32.681 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:13.181104+0000", 2024-04-20T17:23:32.681 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0036834260000000001 2024-04-20T17:23:32.681 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.681 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.681 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:32.681 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:13.181126+0000", 2024-04-20T17:23:32.681 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1469000000000001e-05 2024-04-20T17:23:32.681 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.681 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.681 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:32.681 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:13.199232+0000", 2024-04-20T17:23:32.681 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018105729000000001 2024-04-20T17:23:32.681 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.682 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.682 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:32.682 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:13.199276+0000", 2024-04-20T17:23:32.682 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4545000000000001e-05 2024-04-20T17:23:32.682 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.682 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:32.682 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.682 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.682 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.682 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:230368 3.d e409/368 3:b913a7c6:::benchmark_data_smithi171_110765_object87153:head v 409'14421, mlcod=409'14417)", 2024-04-20T17:23:32.682 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:15.247217+0000", 2024-04-20T17:23:32.682 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 17.357521661, 2024-04-20T17:23:32.682 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028183840000000002, 2024-04-20T17:23:32.682 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:32.682 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:32.683 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:32.683 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:32.683 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.683 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:32.683 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.247217+0000", 2024-04-20T17:23:32.683 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.683 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.683 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.683 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:32.684 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.247217+0000", 2024-04-20T17:23:32.684 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.684 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.684 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.684 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:32.684 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.247218+0000", 2024-04-20T17:23:32.684 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.508e-06 2024-04-20T17:23:32.684 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.684 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.684 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:32.684 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.247227+0000", 2024-04-20T17:23:32.684 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6759999999999996e-06 2024-04-20T17:23:32.685 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.686 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.686 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:32.686 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.247237+0000", 2024-04-20T17:23:32.686 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0093e-05 2024-04-20T17:23:32.686 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.686 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.686 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:32.686 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.247244+0000", 2024-04-20T17:23:32.686 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6810000000000001e-06 2024-04-20T17:23:32.686 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.686 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.686 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:32.687 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.254031+0000", 2024-04-20T17:23:32.687 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0067877809999999997 2024-04-20T17:23:32.687 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.687 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.687 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:32.687 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.254051+0000", 2024-04-20T17:23:32.687 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9945e-05 2024-04-20T17:23:32.687 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.687 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.687 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:32.687 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.275368+0000", 2024-04-20T17:23:32.687 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021316693000000001 2024-04-20T17:23:32.687 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.688 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.688 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:32.688 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.275401+0000", 2024-04-20T17:23:32.688 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2462999999999997e-05 2024-04-20T17:23:32.688 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.688 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:32.688 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.688 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.688 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.688 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:230378 3.d e409/368 3:bc451203:::benchmark_data_smithi171_110765_object87163:head v 409'14422, mlcod=409'14417)", 2024-04-20T17:23:32.688 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:15.253230+0000", 2024-04-20T17:23:32.689 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 17.351508248999998, 2024-04-20T17:23:32.689 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022200645000000001, 2024-04-20T17:23:32.689 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:32.689 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:32.689 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:32.689 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:32.689 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.689 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:32.689 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.253230+0000", 2024-04-20T17:23:32.689 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.689 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.690 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.690 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:32.690 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.253230+0000", 2024-04-20T17:23:32.690 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.690 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.690 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.690 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:32.690 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.253232+0000", 2024-04-20T17:23:32.690 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0099999999999998e-06 2024-04-20T17:23:32.690 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.690 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.691 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:32.691 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.253241+0000", 2024-04-20T17:23:32.691 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0780000000000002e-06 2024-04-20T17:23:32.691 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.691 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.691 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:32.691 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.253254+0000", 2024-04-20T17:23:32.691 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3010000000000001e-05 2024-04-20T17:23:32.691 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.691 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.691 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:32.692 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.253264+0000", 2024-04-20T17:23:32.692 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3680000000000008e-06 2024-04-20T17:23:32.692 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.692 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.692 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:32.692 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.258707+0000", 2024-04-20T17:23:32.692 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0054434979999999997 2024-04-20T17:23:32.692 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.692 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.692 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:32.692 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.258726+0000", 2024-04-20T17:23:32.692 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8736000000000002e-05 2024-04-20T17:23:32.692 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.692 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.693 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:32.693 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.275405+0000", 2024-04-20T17:23:32.693 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.016679315 2024-04-20T17:23:32.693 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.693 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.693 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:32.693 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.275431+0000", 2024-04-20T17:23:32.693 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.563e-05 2024-04-20T17:23:32.693 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.693 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:32.693 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.693 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.693 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.694 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:230402 3.0 e409/356 3:08eb9126:::benchmark_data_smithi171_110765_object87187:head v 409'14340, mlcod=409'14336)", 2024-04-20T17:23:32.694 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:15.296430+0000", 2024-04-20T17:23:32.694 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 17.308308186000001, 2024-04-20T17:23:32.694 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.023634346000000001, 2024-04-20T17:23:32.694 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:32.694 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:32.694 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:32.694 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:32.694 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.694 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:32.694 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.296430+0000", 2024-04-20T17:23:32.695 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.695 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.695 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.695 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:32.695 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.296430+0000", 2024-04-20T17:23:32.695 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.695 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.695 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.695 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:32.695 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.296431+0000", 2024-04-20T17:23:32.695 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0900000000000001e-07 2024-04-20T17:23:32.696 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.696 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.696 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:32.696 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.296436+0000", 2024-04-20T17:23:32.696 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0520000000000001e-06 2024-04-20T17:23:32.696 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.696 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.696 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:32.696 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.296442+0000", 2024-04-20T17:23:32.697 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8420000000000001e-06 2024-04-20T17:23:32.697 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.697 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.697 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:32.697 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.296447+0000", 2024-04-20T17:23:32.697 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0050000000000004e-06 2024-04-20T17:23:32.697 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.697 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.697 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:32.697 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.300947+0000", 2024-04-20T17:23:32.697 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0045000919999999998 2024-04-20T17:23:32.697 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.698 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.698 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:32.698 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.300968+0000", 2024-04-20T17:23:32.698 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0874999999999999e-05 2024-04-20T17:23:32.698 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.698 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.698 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:32.698 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.320022+0000", 2024-04-20T17:23:32.698 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019054508000000001 2024-04-20T17:23:32.698 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.698 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.699 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:32.699 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.320065+0000", 2024-04-20T17:23:32.699 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2262999999999998e-05 2024-04-20T17:23:32.699 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.699 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:32.699 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.699 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.699 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.700 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:230403 3.0 e409/356 3:0f9cfb88:::benchmark_data_smithi171_110765_object87188:head v 409'14341, mlcod=409'14337)", 2024-04-20T17:23:32.700 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:15.299216+0000", 2024-04-20T17:23:32.700 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 17.305522261, 2024-04-20T17:23:32.700 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020875375000000002, 2024-04-20T17:23:32.700 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:32.700 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:32.700 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:32.700 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:32.700 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.700 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:32.700 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.299216+0000", 2024-04-20T17:23:32.701 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.701 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.701 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.701 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:32.701 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.299216+0000", 2024-04-20T17:23:32.701 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.701 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.701 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.701 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:32.701 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.299218+0000", 2024-04-20T17:23:32.701 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4950000000000001e-06 2024-04-20T17:23:32.702 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.702 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.702 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:32.702 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.299226+0000", 2024-04-20T17:23:32.702 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1510000000000004e-06 2024-04-20T17:23:32.702 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.702 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.702 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:32.702 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.299240+0000", 2024-04-20T17:23:32.702 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.467e-05 2024-04-20T17:23:32.702 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.703 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.703 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:32.703 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.299257+0000", 2024-04-20T17:23:32.703 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6932e-05 2024-04-20T17:23:32.703 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.703 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.703 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:32.703 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.306549+0000", 2024-04-20T17:23:32.703 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0072911900000000003 2024-04-20T17:23:32.703 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.704 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.704 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:32.704 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.306575+0000", 2024-04-20T17:23:32.704 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6767e-05 2024-04-20T17:23:32.704 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.704 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.704 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:32.704 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.320069+0000", 2024-04-20T17:23:32.704 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013493450000000001 2024-04-20T17:23:32.704 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.705 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.705 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:32.705 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.320091+0000", 2024-04-20T17:23:32.705 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2719999999999999e-05 2024-04-20T17:23:32.705 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.705 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:32.705 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.705 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.705 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.705 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:230411 3.0 e409/356 3:027016c7:::benchmark_data_smithi171_110765_object87196:head v 409'14342, mlcod=409'14337)", 2024-04-20T17:23:32.705 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:15.304534+0000", 2024-04-20T17:23:32.706 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 17.300204394000001, 2024-04-20T17:23:32.706 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024824156999999999, 2024-04-20T17:23:32.706 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:32.706 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:32.706 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:32.706 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:32.706 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.706 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:32.707 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.304534+0000", 2024-04-20T17:23:32.707 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.707 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.707 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.707 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:32.707 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.304534+0000", 2024-04-20T17:23:32.707 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.707 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.708 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.708 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:32.708 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.304536+0000", 2024-04-20T17:23:32.708 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.294e-06 2024-04-20T17:23:32.708 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.708 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.708 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:32.708 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.304546+0000", 2024-04-20T17:23:32.708 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0025e-05 2024-04-20T17:23:32.708 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.708 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.709 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:32.709 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.304562+0000", 2024-04-20T17:23:32.709 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5534999999999999e-05 2024-04-20T17:23:32.709 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.709 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.709 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:32.709 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.304571+0000", 2024-04-20T17:23:32.709 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5829999999999996e-06 2024-04-20T17:23:32.709 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.709 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.709 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:32.709 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.310990+0000", 2024-04-20T17:23:32.710 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0064184419999999999 2024-04-20T17:23:32.710 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.710 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.710 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:32.710 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.311013+0000", 2024-04-20T17:23:32.710 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3455000000000002e-05 2024-04-20T17:23:32.710 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.710 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.710 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:32.710 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.329306+0000", 2024-04-20T17:23:32.710 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018292776 2024-04-20T17:23:32.711 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.711 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.711 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:32.711 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:15.329358+0000", 2024-04-20T17:23:32.711 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2046999999999998e-05 2024-04-20T17:23:32.711 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.711 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:32.711 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.711 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.711 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.711 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:232858 3.0 e410/356 3:039f72b0:::benchmark_data_smithi171_110765_object89643:head v 410'14499, mlcod=410'14496)", 2024-04-20T17:23:32.712 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:17.407399+0000", 2024-04-20T17:23:32.712 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 15.197339278999999, 2024-04-20T17:23:32.712 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024443360000000001, 2024-04-20T17:23:32.712 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:32.712 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:32.712 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:32.712 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:32.712 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.712 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:32.712 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.407399+0000", 2024-04-20T17:23:32.712 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.712 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.713 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.713 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:32.713 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.407399+0000", 2024-04-20T17:23:32.713 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.713 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.713 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.713 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:32.713 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.407401+0000", 2024-04-20T17:23:32.713 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.99e-06 2024-04-20T17:23:32.714 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.714 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.714 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:32.714 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.407409+0000", 2024-04-20T17:23:32.714 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2199999999999992e-06 2024-04-20T17:23:32.714 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.714 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.714 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:32.714 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.407418+0000", 2024-04-20T17:23:32.714 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5499999999999995e-06 2024-04-20T17:23:32.714 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.714 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.714 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:32.715 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.407424+0000", 2024-04-20T17:23:32.715 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9270000000000002e-06 2024-04-20T17:23:32.715 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.715 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.715 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:32.715 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.409848+0000", 2024-04-20T17:23:32.715 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0024246300000000001 2024-04-20T17:23:32.715 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.715 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.715 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:32.715 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.409865+0000", 2024-04-20T17:23:32.716 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6469000000000001e-05 2024-04-20T17:23:32.716 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.716 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.716 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:32.716 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.431794+0000", 2024-04-20T17:23:32.716 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021929298999999999 2024-04-20T17:23:32.716 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.716 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.716 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:32.716 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.431842+0000", 2024-04-20T17:23:32.716 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8275000000000001e-05 2024-04-20T17:23:32.717 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.717 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:32.717 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.717 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.717 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.717 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:232878 3.d e410/368 3:bf38ffef:::benchmark_data_smithi171_110765_object89663:head v 410'14572, mlcod=410'14568)", 2024-04-20T17:23:32.717 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:17.423151+0000", 2024-04-20T17:23:32.717 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 15.181586977, 2024-04-20T17:23:32.717 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024012424000000001, 2024-04-20T17:23:32.717 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:32.717 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:32.717 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:32.718 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:32.718 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.718 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:32.718 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.423151+0000", 2024-04-20T17:23:32.718 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.718 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.718 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.718 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:32.718 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.423151+0000", 2024-04-20T17:23:32.718 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.718 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.718 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.719 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:32.719 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.423153+0000", 2024-04-20T17:23:32.719 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.61e-06 2024-04-20T17:23:32.719 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.719 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.719 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:32.719 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.423162+0000", 2024-04-20T17:23:32.719 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6959999999999994e-06 2024-04-20T17:23:32.719 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.719 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.719 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:32.719 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.423174+0000", 2024-04-20T17:23:32.719 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2621e-05 2024-04-20T17:23:32.720 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.720 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.720 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:32.720 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.423182+0000", 2024-04-20T17:23:32.720 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5120000000000002e-06 2024-04-20T17:23:32.720 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.720 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.720 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:32.720 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.424310+0000", 2024-04-20T17:23:32.721 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001127769 2024-04-20T17:23:32.721 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.721 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.721 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:32.721 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.424325+0000", 2024-04-20T17:23:32.721 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5023e-05 2024-04-20T17:23:32.721 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.721 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.721 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:32.722 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.447126+0000", 2024-04-20T17:23:32.722 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022801040000000002 2024-04-20T17:23:32.722 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.722 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.722 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:32.722 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.447164+0000", 2024-04-20T17:23:32.722 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8152999999999999e-05 2024-04-20T17:23:32.722 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.722 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:32.722 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.722 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.722 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.722 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:234327 3.0 e410/356 3:0d0d7eb7:::benchmark_data_smithi171_110765_object91112:head v 410'14604, mlcod=410'14599)", 2024-04-20T17:23:32.723 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:18.731228+0000", 2024-04-20T17:23:32.723 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 13.8735099, 2024-04-20T17:23:32.723 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021697732000000001, 2024-04-20T17:23:32.723 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:32.723 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:32.723 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:32.723 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:32.723 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.723 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:32.723 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:18.731228+0000", 2024-04-20T17:23:32.724 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.724 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.724 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.724 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:32.724 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:18.731228+0000", 2024-04-20T17:23:32.724 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:32.724 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.724 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.724 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:32.724 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:18.731230+0000", 2024-04-20T17:23:32.724 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.463e-06 2024-04-20T17:23:32.725 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.725 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.725 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:32.725 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:18.731237+0000", 2024-04-20T17:23:32.725 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9970000000000001e-06 2024-04-20T17:23:32.725 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.725 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.725 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:32.726 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:18.731245+0000", 2024-04-20T17:23:32.726 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0390000000000005e-06 2024-04-20T17:23:32.726 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.726 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.726 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:32.726 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:18.731250+0000", 2024-04-20T17:23:32.726 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7419999999999997e-06 2024-04-20T17:23:32.726 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.726 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.726 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:32.727 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:18.734839+0000", 2024-04-20T17:23:32.727 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0035888360000000002 2024-04-20T17:23:32.727 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.727 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.727 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:32.727 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:18.734862+0000", 2024-04-20T17:23:32.727 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3836999999999999e-05 2024-04-20T17:23:32.727 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.728 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.728 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:32.728 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:18.752886+0000", 2024-04-20T17:23:32.728 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018023299 2024-04-20T17:23:32.728 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:32.728 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:32.728 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:32.728 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:18.752926+0000", 2024-04-20T17:23:32.728 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0519000000000003e-05 2024-04-20T17:23:32.728 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.729 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:32.729 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.729 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:32.729 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:32.729 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:23:32.771 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:23:32.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:32.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:23:32.875 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:23:32.905 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:33 smithi171 ceph-mon[44189]: osdmap e418: 12 total, 12 up, 12 in 2024-04-20T17:23:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:33 smithi171 ceph-mon[44189]: pgmap v1131: 17 pgs: 2 peering, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 12 active+clean; 2.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 2101/85334 objects misplaced (2.462%); 25 MiB/s, 404 objects/s recovering 2024-04-20T17:23:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:33 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:23:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:33 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:23:33.417 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-20T17:23:33.432 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:23:33.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:33.439 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:23:33.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:33 smithi005 ceph-mon[39562]: osdmap e418: 12 total, 12 up, 12 in 2024-04-20T17:23:33.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:33 smithi005 ceph-mon[39562]: pgmap v1131: 17 pgs: 2 peering, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 12 active+clean; 2.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 2101/85334 objects misplaced (2.462%); 25 MiB/s, 404 objects/s recovering 2024-04-20T17:23:33.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:33 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:23:33.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:33 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:23:33.468 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:23:33.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:33 smithi106 ceph-mon[44423]: osdmap e418: 12 total, 12 up, 12 in 2024-04-20T17:23:33.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:33 smithi106 ceph-mon[44423]: pgmap v1131: 17 pgs: 2 peering, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 12 active+clean; 2.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 2101/85334 objects misplaced (2.462%); 25 MiB/s, 404 objects/s recovering 2024-04-20T17:23:33.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:33 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:23:33.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:33 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:23:33.540 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:23:33.568 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:33.595 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:23:33.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:33.598 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:23:33.699 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:23:33.713 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:23:33.727 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:33.746 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:23:33.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:33.750 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:23:33.851 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:23:33.883 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:34.314 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:34 smithi171 ceph-mon[44189]: osdmap e419: 12 total, 12 up, 12 in 2024-04-20T17:23:34.314 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:34 smithi171 ceph-mon[44189]: 3.c scrub starts 2024-04-20T17:23:34.314 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:34 smithi171 ceph-mon[44189]: 3.c scrub ok 2024-04-20T17:23:34.332 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:23:34.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:34.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:23:34.438 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:23:34.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:34 smithi005 ceph-mon[39562]: osdmap e419: 12 total, 12 up, 12 in 2024-04-20T17:23:34.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:34 smithi005 ceph-mon[39562]: 3.c scrub starts 2024-04-20T17:23:34.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:34 smithi005 ceph-mon[39562]: 3.c scrub ok 2024-04-20T17:23:34.485 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:34.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:34 smithi106 ceph-mon[44423]: osdmap e419: 12 total, 12 up, 12 in 2024-04-20T17:23:34.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:34 smithi106 ceph-mon[44423]: 3.c scrub starts 2024-04-20T17:23:34.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:34 smithi106 ceph-mon[44423]: 3.c scrub ok 2024-04-20T17:23:34.692 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:23:34.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:34.695 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:23:34.796 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:23:34.835 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:34.861 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:23:34.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:34.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:23:34.965 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:23:34.995 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:35.110 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:35 smithi106 ceph-mon[44423]: osdmap e420: 12 total, 12 up, 12 in 2024-04-20T17:23:35.110 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:35 smithi106 ceph-mon[44423]: pgmap v1134: 17 pgs: 3 peering, 1 active+remapped+backfilling, 13 active+clean; 2.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 984 op/s; 1530/82906 objects misplaced (1.845%); 30 MiB/s, 485 objects/s recovering 2024-04-20T17:23:35.110 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:35 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4165262851' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:23:35.110 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:35 smithi106 ceph-mon[44423]: from='client.25945 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:23:35.123 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:23:35.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:35 smithi005 ceph-mon[39562]: osdmap e420: 12 total, 12 up, 12 in 2024-04-20T17:23:35.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:35 smithi005 ceph-mon[39562]: pgmap v1134: 17 pgs: 3 peering, 1 active+remapped+backfilling, 13 active+clean; 2.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 984 op/s; 1530/82906 objects misplaced (1.845%); 30 MiB/s, 485 objects/s recovering 2024-04-20T17:23:35.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:35 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4165262851' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:23:35.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:35 smithi005 ceph-mon[39562]: from='client.25945 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:23:35.259 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:35 smithi171 ceph-mon[44189]: osdmap e420: 12 total, 12 up, 12 in 2024-04-20T17:23:35.259 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:35 smithi171 ceph-mon[44189]: pgmap v1134: 17 pgs: 3 peering, 1 active+remapped+backfilling, 13 active+clean; 2.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 984 op/s; 1530/82906 objects misplaced (1.845%); 30 MiB/s, 485 objects/s recovering 2024-04-20T17:23:35.259 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:35 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4165262851' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:23:35.260 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:35 smithi171 ceph-mon[44189]: from='client.25945 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:23:35.349 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:23:35.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:35.352 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:23:35.453 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:23:35.497 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:35.629 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:23:35.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:35.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:23:35.742 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:23:35.759 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:23:35.772 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:35.788 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:23:35.795 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:23:35.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:35.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:23:35.823 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:23:35.860 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:23:35.880 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:23:35.902 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:23:35.904 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:23:35.940 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:23:35.949 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:35.974 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:23:35.993 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:23:35.994 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:23:35.994 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 0 2024-04-20T17:23:35.994 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:23:36.008 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:23:36.044 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:23:36.081 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:23:36.084 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:23:36.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:36.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:23:36.120 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:36 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:23:36.120 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:36 smithi106 ceph-mon[44423]: from='client.25945 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:23:36.120 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:36 smithi106 ceph-mon[44423]: osdmap e421: 12 total, 12 up, 12 in 2024-04-20T17:23:36.121 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:36 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:23:36.121 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:36 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:23:36.121 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:36 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2739188591' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:23:36.121 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:36.125 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:23:36.137 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:23:36.137 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:23:36.138 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:23:36.138 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:23:36.138 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:23:36.138 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:23:36.138 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:23:36.138 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:23:36.139 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:23:36.139 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:23:36.139 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:23:36.199 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:23:36.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:36 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:23:36.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:36 smithi005 ceph-mon[39562]: from='client.25945 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:23:36.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:36 smithi005 ceph-mon[39562]: osdmap e421: 12 total, 12 up, 12 in 2024-04-20T17:23:36.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:36 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:23:36.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:36 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:23:36.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:36 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2739188591' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:23:36.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:36.227 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:36.250 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:23:36.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:36.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:23:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:36 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:23:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:36 smithi171 ceph-mon[44189]: from='client.25945 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:23:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:36 smithi171 ceph-mon[44189]: osdmap e421: 12 total, 12 up, 12 in 2024-04-20T17:23:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:36 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:23:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:36 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:23:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:36 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2739188591' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:23:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:36.360 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:23:36.390 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:36.416 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:23:36.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:36.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:23:36.521 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:23:36.557 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:36.818 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:23:36.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:36.823 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:23:36.856 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-20T17:23:36.925 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:23:36.985 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:37.016 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:23:37.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:37.020 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:23:37.121 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:23:37.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:37 smithi005 ceph-mon[39562]: osdmap e422: 12 total, 12 up, 12 in 2024-04-20T17:23:37.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:37 smithi005 ceph-mon[39562]: pgmap v1137: 17 pgs: 1 peering, 1 active+remapped+backfilling, 15 active+clean; 2.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 664 op/s; 1530/80016 objects misplaced (1.912%); 8.8 MiB/s, 140 objects/s recovering 2024-04-20T17:23:37.162 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:37.195 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:23:37.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:37.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:23:37.271 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:23:37.302 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:23:37.348 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:37.381 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:23:37.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:37.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:23:37.489 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:23:37.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:37 smithi106 ceph-mon[44423]: osdmap e422: 12 total, 12 up, 12 in 2024-04-20T17:23:37.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:37 smithi106 ceph-mon[44423]: pgmap v1137: 17 pgs: 1 peering, 1 active+remapped+backfilling, 15 active+clean; 2.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 664 op/s; 1530/80016 objects misplaced (1.912%); 8.8 MiB/s, 140 objects/s recovering 2024-04-20T17:23:37.527 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:37.553 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:23:37.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:37.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:23:37.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:37.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:37 smithi171 ceph-mon[44189]: osdmap e422: 12 total, 12 up, 12 in 2024-04-20T17:23:37.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:37 smithi171 ceph-mon[44189]: pgmap v1137: 17 pgs: 1 peering, 1 active+remapped+backfilling, 15 active+clean; 2.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 664 op/s; 1530/80016 objects misplaced (1.912%); 8.8 MiB/s, 140 objects/s recovering 2024-04-20T17:23:37.656 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:23:37.709 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:37.736 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:23:37.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:37.746 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:23:37.847 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:23:37.877 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:37.900 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:23:37.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:37.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:23:38.007 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:23:38.034 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:38.061 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:23:38.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:38.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:23:38.117 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:23:38.165 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:23:38.208 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:38.213 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 8, 9, 10, 11, 2, 7] 2024-04-20T17:23:38.213 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:23:38.214 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-20T17:23:38.214 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-20T17:23:38.287 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:23:38.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:38.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:23:38.395 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:23:38.425 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:38.593 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:23:38.831 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:23:38.935 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:23:38.981 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:23:38.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:38.984 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:23:39.085 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:23:39.121 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:39.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:39 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:23:39.151 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:39 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:23:39.151 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:39 smithi005 ceph-mon[39562]: osdmap e423: 12 total, 12 up, 12 in 2024-04-20T17:23:39.151 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:39 smithi005 ceph-mon[39562]: pgmap v1139: 17 pgs: 17 active+clean; 2.3 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 876 op/s; 22 MiB/s, 353 objects/s recovering 2024-04-20T17:23:39.323 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:23:39.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:39.331 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:39 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:23:39.331 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:39 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:23:39.331 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:39 smithi171 ceph-mon[44189]: osdmap e423: 12 total, 12 up, 12 in 2024-04-20T17:23:39.331 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:39 smithi171 ceph-mon[44189]: pgmap v1139: 17 pgs: 17 active+clean; 2.3 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 876 op/s; 22 MiB/s, 353 objects/s recovering 2024-04-20T17:23:39.332 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:23:39.433 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:23:39.459 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:39 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:23:39.459 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:39 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:23:39.459 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:39 smithi106 ceph-mon[44423]: osdmap e423: 12 total, 12 up, 12 in 2024-04-20T17:23:39.459 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:39 smithi106 ceph-mon[44423]: pgmap v1139: 17 pgs: 17 active+clean; 2.3 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 876 op/s; 22 MiB/s, 353 objects/s recovering 2024-04-20T17:23:39.461 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:39.488 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:23:39.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:39.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:23:39.595 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:23:39.642 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:39.747 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:23:39.747 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:23:39.797 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:23:39.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:39.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:23:39.903 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:23:39.955 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:40.331 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:23:40.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:40.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:23:40.436 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:23:40.471 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:40.493 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:23:40.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:40.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:23:40.517 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-20T17:23:40.598 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:23:40.637 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:40.886 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:23:41.359 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:41 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:23:41.359 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:41 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:23:41.359 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:41 smithi005 ceph-mon[39562]: osdmap e424: 12 total, 12 up, 12 in 2024-04-20T17:23:41.359 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:41 smithi005 ceph-mon[39562]: pgmap v1141: 17 pgs: 17 active+clean; 2.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 857 op/s; 18 MiB/s, 284 objects/s recovering 2024-04-20T17:23:41.480 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:41 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:23:41.481 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:41 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:23:41.481 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:41 smithi171 ceph-mon[44189]: osdmap e424: 12 total, 12 up, 12 in 2024-04-20T17:23:41.481 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:41 smithi171 ceph-mon[44189]: pgmap v1141: 17 pgs: 17 active+clean; 2.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 857 op/s; 18 MiB/s, 284 objects/s recovering 2024-04-20T17:23:41.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:41 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:23:41.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:41 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:23:41.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:41 smithi106 ceph-mon[44423]: osdmap e424: 12 total, 12 up, 12 in 2024-04-20T17:23:41.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:41 smithi106 ceph-mon[44423]: pgmap v1141: 17 pgs: 17 active+clean; 2.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 857 op/s; 18 MiB/s, 284 objects/s recovering 2024-04-20T17:23:41.799 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:23:41.804 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:23:41.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:41.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:23:41.910 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:23:41.989 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:42.049 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:23:42.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:42.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:23:42.160 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:23:42.195 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:42.226 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:23:42.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:42.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:23:42.293 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:23:42.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:42 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4266817967' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:23:42.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:42 smithi171 ceph-mon[44189]: from='client.26024 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:23:42.339 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:23:42.385 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:42.426 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:23:42.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:42.432 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:23:42.461 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:23:42.461 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:23:42.461 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 425, 2024-04-20T17:23:42.461 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "13c6e76c-ff37-11ee-bc93-c7b262605968", 2024-04-20T17:23:42.461 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-04-20T16:58:25.272557+0000", 2024-04-20T17:23:42.461 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-04-20T17:23:42.259320+0000", 2024-04-20T17:23:42.462 INFO:teuthology.orchestra.run.smithi005.stdout: "last_up_change": "2024-04-20T17:22:32.243867+0000", 2024-04-20T17:23:42.462 INFO:teuthology.orchestra.run.smithi005.stdout: "last_in_change": "2024-04-20T17:23:20.118061+0000", 2024-04-20T17:23:42.462 INFO:teuthology.orchestra.run.smithi005.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-20T17:23:42.462 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_num": 5806080, 2024-04-20T17:23:42.462 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_set": [ 2024-04-20T17:23:42.462 INFO:teuthology.orchestra.run.smithi005.stdout: "nodeep-scrub", 2024-04-20T17:23:42.462 INFO:teuthology.orchestra.run.smithi005.stdout: "noscrub", 2024-04-20T17:23:42.462 INFO:teuthology.orchestra.run.smithi005.stdout: "pglog_hardlimit", 2024-04-20T17:23:42.462 INFO:teuthology.orchestra.run.smithi005.stdout: "purged_snapdirs", 2024-04-20T17:23:42.462 INFO:teuthology.orchestra.run.smithi005.stdout: "recovery_deletes", 2024-04-20T17:23:42.462 INFO:teuthology.orchestra.run.smithi005.stdout: "sortbitwise" 2024-04-20T17:23:42.462 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:23:42.463 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_version": 29, 2024-04-20T17:23:42.463 INFO:teuthology.orchestra.run.smithi005.stdout: "full_ratio": 0.94999998807907104, 2024-04-20T17:23:42.463 INFO:teuthology.orchestra.run.smithi005.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-20T17:23:42.463 INFO:teuthology.orchestra.run.smithi005.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-20T17:23:42.463 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_snapshot": "", 2024-04-20T17:23:42.463 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_max": 3, 2024-04-20T17:23:42.463 INFO:teuthology.orchestra.run.smithi005.stdout: "max_osd": 12, 2024-04-20T17:23:42.463 INFO:teuthology.orchestra.run.smithi005.stdout: "require_min_compat_client": "luminous", 2024-04-20T17:23:42.463 INFO:teuthology.orchestra.run.smithi005.stdout: "min_compat_client": "jewel", 2024-04-20T17:23:42.463 INFO:teuthology.orchestra.run.smithi005.stdout: "require_osd_release": "squid", 2024-04-20T17:23:42.463 INFO:teuthology.orchestra.run.smithi005.stdout: "allow_crimson": false, 2024-04-20T17:23:42.463 INFO:teuthology.orchestra.run.smithi005.stdout: "pools": [ 2024-04-20T17:23:42.463 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.463 INFO:teuthology.orchestra.run.smithi005.stdout: "pool": 1, 2024-04-20T17:23:42.464 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_name": ".mgr", 2024-04-20T17:23:42.464 INFO:teuthology.orchestra.run.smithi005.stdout: "create_time": "2024-04-20T17:01:35.909493+0000", 2024-04-20T17:23:42.464 INFO:teuthology.orchestra.run.smithi005.stdout: "flags": 1, 2024-04-20T17:23:42.464 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_names": "hashpspool", 2024-04-20T17:23:42.464 INFO:teuthology.orchestra.run.smithi005.stdout: "type": 1, 2024-04-20T17:23:42.464 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 2, 2024-04-20T17:23:42.464 INFO:teuthology.orchestra.run.smithi005.stdout: "min_size": 2, 2024-04-20T17:23:42.464 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_rule": 0, 2024-04-20T17:23:42.464 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_count": 0, 2024-04-20T17:23:42.464 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_target": 0, 2024-04-20T17:23:42.464 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T17:23:42.464 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T17:23:42.464 INFO:teuthology.orchestra.run.smithi005.stdout: "object_hash": 2, 2024-04-20T17:23:42.464 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_autoscale_mode": "off", 2024-04-20T17:23:42.464 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num": 1, 2024-04-20T17:23:42.465 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num": 1, 2024-04-20T17:23:42.465 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num_target": 1, 2024-04-20T17:23:42.465 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_target": 1, 2024-04-20T17:23:42.465 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_pending": 1, 2024-04-20T17:23:42.465 INFO:teuthology.orchestra.run.smithi005.stdout: "last_pg_merge_meta": { 2024-04-20T17:23:42.465 INFO:teuthology.orchestra.run.smithi005.stdout: "source_pgid": "0.0", 2024-04-20T17:23:42.465 INFO:teuthology.orchestra.run.smithi005.stdout: "ready_epoch": 0, 2024-04-20T17:23:42.465 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_started": 0, 2024-04-20T17:23:42.465 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_clean": 0, 2024-04-20T17:23:42.465 INFO:teuthology.orchestra.run.smithi005.stdout: "source_version": "0'0", 2024-04-20T17:23:42.465 INFO:teuthology.orchestra.run.smithi005.stdout: "target_version": "0'0" 2024-04-20T17:23:42.465 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.465 INFO:teuthology.orchestra.run.smithi005.stdout: "last_change": "16", 2024-04-20T17:23:42.465 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend": "0", 2024-04-20T17:23:42.466 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-20T17:23:42.466 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_preluminous": "0", 2024-04-20T17:23:42.466 INFO:teuthology.orchestra.run.smithi005.stdout: "auid": 0, 2024-04-20T17:23:42.466 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_mode": "selfmanaged", 2024-04-20T17:23:42.466 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_seq": 0, 2024-04-20T17:23:42.466 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_epoch": 0, 2024-04-20T17:23:42.466 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_snaps": [], 2024-04-20T17:23:42.466 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_snaps": "[]", 2024-04-20T17:23:42.466 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_bytes": 0, 2024-04-20T17:23:42.466 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_objects": 0, 2024-04-20T17:23:42.466 INFO:teuthology.orchestra.run.smithi005.stdout: "tiers": [], 2024-04-20T17:23:42.466 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_of": -1, 2024-04-20T17:23:42.466 INFO:teuthology.orchestra.run.smithi005.stdout: "read_tier": -1, 2024-04-20T17:23:42.466 INFO:teuthology.orchestra.run.smithi005.stdout: "write_tier": -1, 2024-04-20T17:23:42.467 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_mode": "none", 2024-04-20T17:23:42.467 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_bytes": 0, 2024-04-20T17:23:42.467 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_objects": 0, 2024-04-20T17:23:42.467 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T17:23:42.467 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T17:23:42.467 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T17:23:42.467 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_flush_age": 0, 2024-04-20T17:23:42.467 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_evict_age": 0, 2024-04-20T17:23:42.467 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_profile": "", 2024-04-20T17:23:42.467 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_params": { 2024-04-20T17:23:42.467 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "none" 2024-04-20T17:23:42.467 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.467 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_period": 0, 2024-04-20T17:23:42.467 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_count": 0, 2024-04-20T17:23:42.468 INFO:teuthology.orchestra.run.smithi005.stdout: "use_gmt_hitset": true, 2024-04-20T17:23:42.468 INFO:teuthology.orchestra.run.smithi005.stdout: "min_read_recency_for_promote": 0, 2024-04-20T17:23:42.468 INFO:teuthology.orchestra.run.smithi005.stdout: "min_write_recency_for_promote": 0, 2024-04-20T17:23:42.468 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T17:23:42.468 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_search_last_n": 0, 2024-04-20T17:23:42.468 INFO:teuthology.orchestra.run.smithi005.stdout: "grade_table": [], 2024-04-20T17:23:42.468 INFO:teuthology.orchestra.run.smithi005.stdout: "stripe_width": 0, 2024-04-20T17:23:42.468 INFO:teuthology.orchestra.run.smithi005.stdout: "expected_num_objects": 0, 2024-04-20T17:23:42.468 INFO:teuthology.orchestra.run.smithi005.stdout: "fast_read": false, 2024-04-20T17:23:42.468 INFO:teuthology.orchestra.run.smithi005.stdout: "options": { 2024-04-20T17:23:42.468 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_max": 32, 2024-04-20T17:23:42.468 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_min": 1 2024-04-20T17:23:42.468 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.468 INFO:teuthology.orchestra.run.smithi005.stdout: "application_metadata": { 2024-04-20T17:23:42.468 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": {} 2024-04-20T17:23:42.469 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.469 INFO:teuthology.orchestra.run.smithi005.stdout: "read_balance": { 2024-04-20T17:23:42.469 INFO:teuthology.orchestra.run.smithi005.stdout: "score_type": "Fair distribution", 2024-04-20T17:23:42.469 INFO:teuthology.orchestra.run.smithi005.stdout: "score_acting": 11.109999656677246, 2024-04-20T17:23:42.469 INFO:teuthology.orchestra.run.smithi005.stdout: "score_stable": 11.109999656677246, 2024-04-20T17:23:42.469 INFO:teuthology.orchestra.run.smithi005.stdout: "optimal_score": 0.18000000715255737, 2024-04-20T17:23:42.469 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_acting": 2, 2024-04-20T17:23:42.469 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_stable": 2, 2024-04-20T17:23:42.469 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity_weighted": 0.91666656732559204, 2024-04-20T17:23:42.469 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity": 1, 2024-04-20T17:23:42.469 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-20T17:23:42.469 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.469 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.469 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.470 INFO:teuthology.orchestra.run.smithi005.stdout: "pool": 3, 2024-04-20T17:23:42.470 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_name": "unique_pool_1", 2024-04-20T17:23:42.470 INFO:teuthology.orchestra.run.smithi005.stdout: "create_time": "2024-04-20T17:20:21.928963+0000", 2024-04-20T17:23:42.470 INFO:teuthology.orchestra.run.smithi005.stdout: "flags": 1, 2024-04-20T17:23:42.470 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_names": "hashpspool", 2024-04-20T17:23:42.470 INFO:teuthology.orchestra.run.smithi005.stdout: "type": 1, 2024-04-20T17:23:42.470 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 2, 2024-04-20T17:23:42.470 INFO:teuthology.orchestra.run.smithi005.stdout: "min_size": 2, 2024-04-20T17:23:42.470 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_rule": 0, 2024-04-20T17:23:42.470 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_count": 0, 2024-04-20T17:23:42.470 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_target": 0, 2024-04-20T17:23:42.470 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T17:23:42.470 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T17:23:42.470 INFO:teuthology.orchestra.run.smithi005.stdout: "object_hash": 2, 2024-04-20T17:23:42.471 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_autoscale_mode": "off", 2024-04-20T17:23:42.471 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num": 16, 2024-04-20T17:23:42.471 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num": 16, 2024-04-20T17:23:42.471 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num_target": 16, 2024-04-20T17:23:42.471 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_target": 16, 2024-04-20T17:23:42.471 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_pending": 16, 2024-04-20T17:23:42.471 INFO:teuthology.orchestra.run.smithi005.stdout: "last_pg_merge_meta": { 2024-04-20T17:23:42.471 INFO:teuthology.orchestra.run.smithi005.stdout: "source_pgid": "0.0", 2024-04-20T17:23:42.471 INFO:teuthology.orchestra.run.smithi005.stdout: "ready_epoch": 0, 2024-04-20T17:23:42.471 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_started": 0, 2024-04-20T17:23:42.471 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_clean": 0, 2024-04-20T17:23:42.471 INFO:teuthology.orchestra.run.smithi005.stdout: "source_version": "0'0", 2024-04-20T17:23:42.471 INFO:teuthology.orchestra.run.smithi005.stdout: "target_version": "0'0" 2024-04-20T17:23:42.471 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.471 INFO:teuthology.orchestra.run.smithi005.stdout: "last_change": "360", 2024-04-20T17:23:42.472 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend": "0", 2024-04-20T17:23:42.472 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-20T17:23:42.472 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_preluminous": "0", 2024-04-20T17:23:42.472 INFO:teuthology.orchestra.run.smithi005.stdout: "auid": 0, 2024-04-20T17:23:42.472 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_mode": "selfmanaged", 2024-04-20T17:23:42.472 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_seq": 0, 2024-04-20T17:23:42.472 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_epoch": 0, 2024-04-20T17:23:42.472 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_snaps": [], 2024-04-20T17:23:42.472 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_snaps": "[]", 2024-04-20T17:23:42.472 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_bytes": 0, 2024-04-20T17:23:42.472 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_objects": 0, 2024-04-20T17:23:42.472 INFO:teuthology.orchestra.run.smithi005.stdout: "tiers": [], 2024-04-20T17:23:42.472 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_of": -1, 2024-04-20T17:23:42.472 INFO:teuthology.orchestra.run.smithi005.stdout: "read_tier": -1, 2024-04-20T17:23:42.472 INFO:teuthology.orchestra.run.smithi005.stdout: "write_tier": -1, 2024-04-20T17:23:42.473 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_mode": "none", 2024-04-20T17:23:42.473 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_bytes": 0, 2024-04-20T17:23:42.473 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_objects": 0, 2024-04-20T17:23:42.473 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T17:23:42.473 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T17:23:42.473 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T17:23:42.473 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_flush_age": 0, 2024-04-20T17:23:42.473 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_evict_age": 0, 2024-04-20T17:23:42.473 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_profile": "", 2024-04-20T17:23:42.473 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_params": { 2024-04-20T17:23:42.473 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "none" 2024-04-20T17:23:42.473 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.473 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_period": 0, 2024-04-20T17:23:42.473 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_count": 0, 2024-04-20T17:23:42.474 INFO:teuthology.orchestra.run.smithi005.stdout: "use_gmt_hitset": true, 2024-04-20T17:23:42.474 INFO:teuthology.orchestra.run.smithi005.stdout: "min_read_recency_for_promote": 0, 2024-04-20T17:23:42.474 INFO:teuthology.orchestra.run.smithi005.stdout: "min_write_recency_for_promote": 0, 2024-04-20T17:23:42.474 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T17:23:42.474 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_search_last_n": 0, 2024-04-20T17:23:42.474 INFO:teuthology.orchestra.run.smithi005.stdout: "grade_table": [], 2024-04-20T17:23:42.474 INFO:teuthology.orchestra.run.smithi005.stdout: "stripe_width": 0, 2024-04-20T17:23:42.474 INFO:teuthology.orchestra.run.smithi005.stdout: "expected_num_objects": 0, 2024-04-20T17:23:42.474 INFO:teuthology.orchestra.run.smithi005.stdout: "fast_read": false, 2024-04-20T17:23:42.474 INFO:teuthology.orchestra.run.smithi005.stdout: "options": {}, 2024-04-20T17:23:42.474 INFO:teuthology.orchestra.run.smithi005.stdout: "application_metadata": { 2024-04-20T17:23:42.474 INFO:teuthology.orchestra.run.smithi005.stdout: "rados": {} 2024-04-20T17:23:42.474 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.474 INFO:teuthology.orchestra.run.smithi005.stdout: "read_balance": { 2024-04-20T17:23:42.475 INFO:teuthology.orchestra.run.smithi005.stdout: "score_type": "Fair distribution", 2024-04-20T17:23:42.475 INFO:teuthology.orchestra.run.smithi005.stdout: "score_acting": 3.440000057220459, 2024-04-20T17:23:42.475 INFO:teuthology.orchestra.run.smithi005.stdout: "score_stable": 3.440000057220459, 2024-04-20T17:23:42.475 INFO:teuthology.orchestra.run.smithi005.stdout: "optimal_score": 0.9100000262260437, 2024-04-20T17:23:42.475 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_acting": 3.130000114440918, 2024-04-20T17:23:42.475 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_stable": 3.130000114440918, 2024-04-20T17:23:42.475 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity_weighted": 0.91666656732559204, 2024-04-20T17:23:42.475 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity": 1, 2024-04-20T17:23:42.475 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-20T17:23:42.475 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.475 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.475 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:23:42.475 INFO:teuthology.orchestra.run.smithi005.stdout: "osds": [ 2024-04-20T17:23:42.475 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.476 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 0, 2024-04-20T17:23:42.476 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b", 2024-04-20T17:23:42.476 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:23:42.476 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:23:42.476 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:23:42.476 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:23:42.476 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:23:42.476 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:23:42.476 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 8, 2024-04-20T17:23:42.476 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 356, 2024-04-20T17:23:42.476 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:23:42.476 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:23:42.476 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:23:42.476 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.477 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.477 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.477 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6801", 2024-04-20T17:23:42.477 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:23:42.477 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.477 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.477 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.477 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:23:42.477 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.477 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.477 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.477 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6802", 2024-04-20T17:23:42.477 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:23:42.477 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.477 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.478 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.478 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:23:42.478 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.478 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.478 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.478 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6804", 2024-04-20T17:23:42.478 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:23:42.478 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.478 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.478 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.478 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:23:42.478 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.479 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.479 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.479 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6803", 2024-04-20T17:23:42.479 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:23:42.479 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.479 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.479 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.479 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6801/3901484705", 2024-04-20T17:23:42.479 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6802/3901484705", 2024-04-20T17:23:42.479 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6804/3901484705", 2024-04-20T17:23:42.479 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6803/3901484705", 2024-04-20T17:23:42.479 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:23:42.479 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:23:42.479 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:23:42.480 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.480 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.480 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.480 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 1, 2024-04-20T17:23:42.480 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "aed0c8eb-0e81-4e85-ab8f-a60420060656", 2024-04-20T17:23:42.480 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:23:42.480 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:23:42.480 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:23:42.480 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:23:42.480 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:23:42.480 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:23:42.480 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 13, 2024-04-20T17:23:42.480 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 356, 2024-04-20T17:23:42.480 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:23:42.480 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:23:42.481 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:23:42.481 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.481 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.481 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.481 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6805", 2024-04-20T17:23:42.481 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3391893914 2024-04-20T17:23:42.481 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.481 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.481 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.481 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:23:42.481 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.481 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.481 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.481 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6806", 2024-04-20T17:23:42.482 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3391893914 2024-04-20T17:23:42.482 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.482 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.482 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.482 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:23:42.482 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.482 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.482 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.482 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6808", 2024-04-20T17:23:42.482 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3391893914 2024-04-20T17:23:42.482 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.482 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.482 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.482 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:23:42.483 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.483 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.483 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.483 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6807", 2024-04-20T17:23:42.483 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3391893914 2024-04-20T17:23:42.483 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.483 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.483 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.483 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6805/3391893914", 2024-04-20T17:23:42.483 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6806/3391893914", 2024-04-20T17:23:42.483 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6808/3391893914", 2024-04-20T17:23:42.483 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6807/3391893914", 2024-04-20T17:23:42.483 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:23:42.483 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:23:42.484 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:23:42.484 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.484 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.484 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.484 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 2, 2024-04-20T17:23:42.484 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "f733a6a2-19c2-4c7a-af9c-04e282062bb7", 2024-04-20T17:23:42.484 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:23:42.484 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:23:42.484 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:23:42.484 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:23:42.484 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 20, 2024-04-20T17:23:42.484 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 169, 2024-04-20T17:23:42.484 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 202, 2024-04-20T17:23:42.485 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 202, 2024-04-20T17:23:42.485 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 170, 2024-04-20T17:23:42.485 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:23:42.485 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:23:42.485 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.485 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.485 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.485 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6809", 2024-04-20T17:23:42.485 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2120275501 2024-04-20T17:23:42.485 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.485 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.485 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.485 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:23:42.485 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.486 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.486 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.486 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6810", 2024-04-20T17:23:42.486 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2120275501 2024-04-20T17:23:42.486 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.486 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.486 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.486 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:23:42.486 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.486 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.486 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.486 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6812", 2024-04-20T17:23:42.486 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2120275501 2024-04-20T17:23:42.486 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.487 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.487 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.487 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:23:42.487 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.487 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.487 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.487 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6811", 2024-04-20T17:23:42.487 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2120275501 2024-04-20T17:23:42.487 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.487 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.487 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.487 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6809/2120275501", 2024-04-20T17:23:42.487 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6810/2120275501", 2024-04-20T17:23:42.487 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6812/2120275501", 2024-04-20T17:23:42.488 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6811/2120275501", 2024-04-20T17:23:42.488 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:23:42.488 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:23:42.488 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:23:42.488 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.488 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.488 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.488 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 3, 2024-04-20T17:23:42.488 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "8c92f1df-e298-4606-88ae-b8a8e79628a1", 2024-04-20T17:23:42.488 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:23:42.488 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:23:42.488 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:23:42.488 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:23:42.488 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:23:42.489 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:23:42.489 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 25, 2024-04-20T17:23:42.489 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 417, 2024-04-20T17:23:42.489 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:23:42.489 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:23:42.489 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:23:42.489 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.489 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.489 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.489 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6813", 2024-04-20T17:23:42.489 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 377652039 2024-04-20T17:23:42.489 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.489 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.489 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.490 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:23:42.490 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.490 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.490 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.490 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6814", 2024-04-20T17:23:42.490 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 377652039 2024-04-20T17:23:42.490 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.490 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.490 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.490 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:23:42.490 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.490 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.490 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.490 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6816", 2024-04-20T17:23:42.491 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 377652039 2024-04-20T17:23:42.491 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.491 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.491 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.491 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:23:42.491 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.491 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.491 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.491 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6815", 2024-04-20T17:23:42.491 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 377652039 2024-04-20T17:23:42.491 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.491 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.491 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.491 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6813/377652039", 2024-04-20T17:23:42.492 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6814/377652039", 2024-04-20T17:23:42.492 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6816/377652039", 2024-04-20T17:23:42.492 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6815/377652039", 2024-04-20T17:23:42.492 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:23:42.492 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:23:42.492 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:23:42.492 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.492 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.492 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.492 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 4, 2024-04-20T17:23:42.492 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "46df16f1-9607-4d7b-9be8-9d4236e160d1", 2024-04-20T17:23:42.492 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:23:42.492 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:23:42.492 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:23:42.493 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:23:42.493 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:23:42.493 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:23:42.493 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 31, 2024-04-20T17:23:42.493 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 356, 2024-04-20T17:23:42.493 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:23:42.493 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:23:42.493 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:23:42.493 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.493 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.493 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.493 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6800", 2024-04-20T17:23:42.493 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:23:42.493 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.493 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.494 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.494 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:23:42.494 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.494 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.494 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.494 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6801", 2024-04-20T17:23:42.494 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:23:42.494 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.494 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.494 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.494 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:23:42.494 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.494 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.495 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.495 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6803", 2024-04-20T17:23:42.495 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:23:42.495 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.495 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.495 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.495 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:23:42.495 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.495 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.495 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.495 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6802", 2024-04-20T17:23:42.495 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:23:42.495 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.495 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.496 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.496 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6800/1839524469", 2024-04-20T17:23:42.496 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6801/1839524469", 2024-04-20T17:23:42.496 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6803/1839524469", 2024-04-20T17:23:42.496 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6802/1839524469", 2024-04-20T17:23:42.496 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:23:42.496 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:23:42.496 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:23:42.496 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.496 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.496 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.496 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 5, 2024-04-20T17:23:42.496 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "6603231d-1e5c-4575-bfaa-2eeca2c12215", 2024-04-20T17:23:42.496 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:23:42.497 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:23:42.497 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:23:42.497 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 0, 2024-04-20T17:23:42.497 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:23:42.497 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:23:42.497 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 37, 2024-04-20T17:23:42.497 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 417, 2024-04-20T17:23:42.497 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:23:42.497 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:23:42.497 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:23:42.497 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.497 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.497 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.497 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6804", 2024-04-20T17:23:42.497 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:23:42.498 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.498 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.498 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.498 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:23:42.498 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.498 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.498 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.498 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6805", 2024-04-20T17:23:42.498 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:23:42.498 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.498 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.498 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.498 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:23:42.499 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.499 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.499 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.499 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6807", 2024-04-20T17:23:42.499 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:23:42.499 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.499 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.499 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.499 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:23:42.499 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.499 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.499 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.499 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6806", 2024-04-20T17:23:42.499 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:23:42.500 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.500 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.500 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.501 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6804/2450315908", 2024-04-20T17:23:42.501 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6805/2450315908", 2024-04-20T17:23:42.501 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6807/2450315908", 2024-04-20T17:23:42.501 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6806/2450315908", 2024-04-20T17:23:42.501 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:23:42.501 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:23:42.501 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:23:42.501 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.501 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.501 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.501 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 6, 2024-04-20T17:23:42.501 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "783a55a6-f176-42be-8c8b-af8481d08dc9", 2024-04-20T17:23:42.501 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:23:42.501 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:23:42.502 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:23:42.502 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:23:42.502 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:23:42.502 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:23:42.502 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 48, 2024-04-20T17:23:42.502 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 413, 2024-04-20T17:23:42.502 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:23:42.502 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:23:42.502 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:23:42.502 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.502 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.502 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.502 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6808", 2024-04-20T17:23:42.503 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2840237913 2024-04-20T17:23:42.503 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.503 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.503 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.503 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:23:42.503 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.503 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.503 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.503 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6809", 2024-04-20T17:23:42.503 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2840237913 2024-04-20T17:23:42.503 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.503 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.503 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.503 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:23:42.504 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.504 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.504 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.504 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6811", 2024-04-20T17:23:42.504 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2840237913 2024-04-20T17:23:42.504 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.504 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.504 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.504 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:23:42.504 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.504 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.504 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.504 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6810", 2024-04-20T17:23:42.504 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2840237913 2024-04-20T17:23:42.505 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.505 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.505 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.505 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6808/2840237913", 2024-04-20T17:23:42.505 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6809/2840237913", 2024-04-20T17:23:42.505 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6811/2840237913", 2024-04-20T17:23:42.505 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6810/2840237913", 2024-04-20T17:23:42.505 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:23:42.505 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:23:42.505 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:23:42.505 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.505 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.505 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.506 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 7, 2024-04-20T17:23:42.506 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4", 2024-04-20T17:23:42.506 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:23:42.506 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:23:42.506 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:23:42.506 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:23:42.506 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 56, 2024-04-20T17:23:42.506 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 364, 2024-04-20T17:23:42.506 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 398, 2024-04-20T17:23:42.506 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 419, 2024-04-20T17:23:42.506 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 365, 2024-04-20T17:23:42.506 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:23:42.506 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:23:42.507 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.507 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.507 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.507 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6812", 2024-04-20T17:23:42.507 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1997086583 2024-04-20T17:23:42.507 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.507 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.507 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.507 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:23:42.507 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.507 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.507 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.507 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6813", 2024-04-20T17:23:42.507 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1997086583 2024-04-20T17:23:42.508 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.508 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.508 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.508 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:23:42.508 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.508 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.508 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.508 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6815", 2024-04-20T17:23:42.508 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1997086583 2024-04-20T17:23:42.508 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.508 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.508 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.508 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:23:42.509 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.509 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.509 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.509 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6814", 2024-04-20T17:23:42.509 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1997086583 2024-04-20T17:23:42.509 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.509 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.509 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.509 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6812/1997086583", 2024-04-20T17:23:42.509 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6813/1997086583", 2024-04-20T17:23:42.509 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6815/1997086583", 2024-04-20T17:23:42.509 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6814/1997086583", 2024-04-20T17:23:42.509 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:23:42.509 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:23:42.510 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:23:42.510 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.510 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.510 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.510 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 8, 2024-04-20T17:23:42.510 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "1cb232ad-1f59-4a0a-a215-e67419ad6476", 2024-04-20T17:23:42.510 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:23:42.510 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:23:42.510 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:23:42.510 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:23:42.510 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:23:42.510 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:23:42.510 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 67, 2024-04-20T17:23:42.510 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 418, 2024-04-20T17:23:42.511 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:23:42.511 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:23:42.511 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:23:42.511 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.511 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.511 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.511 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6800", 2024-04-20T17:23:42.511 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:23:42.511 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.511 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.511 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.511 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:23:42.511 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.511 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.512 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.512 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6801", 2024-04-20T17:23:42.512 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:23:42.512 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.512 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.512 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.512 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:23:42.512 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.512 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.512 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.512 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6803", 2024-04-20T17:23:42.512 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:23:42.512 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.513 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.513 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.513 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:23:42.513 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.513 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.513 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.513 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6802", 2024-04-20T17:23:42.513 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:23:42.513 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.513 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.513 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.513 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6800/4123058942", 2024-04-20T17:23:42.513 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6801/4123058942", 2024-04-20T17:23:42.513 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6803/4123058942", 2024-04-20T17:23:42.514 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6802/4123058942", 2024-04-20T17:23:42.514 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:23:42.514 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:23:42.514 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:23:42.514 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.514 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.514 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.514 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 9, 2024-04-20T17:23:42.514 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "b6c17147-b3dc-475b-bb20-f9f712fb99e8", 2024-04-20T17:23:42.514 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:23:42.514 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:23:42.514 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:23:42.514 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:23:42.514 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:23:42.514 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:23:42.515 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 73, 2024-04-20T17:23:42.515 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 196, 2024-04-20T17:23:42.515 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:23:42.515 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:23:42.515 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:23:42.515 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.515 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.515 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.515 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6804", 2024-04-20T17:23:42.515 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:23:42.515 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.515 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.515 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.515 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:23:42.516 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.516 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.516 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.516 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6805", 2024-04-20T17:23:42.516 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:23:42.516 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.516 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.516 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.516 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:23:42.516 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.516 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.516 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.516 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6807", 2024-04-20T17:23:42.517 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:23:42.517 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.517 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.517 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.517 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:23:42.517 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.517 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.517 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.517 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6806", 2024-04-20T17:23:42.517 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:23:42.517 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.517 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.517 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.517 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6804/2877589636", 2024-04-20T17:23:42.518 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6805/2877589636", 2024-04-20T17:23:42.518 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6807/2877589636", 2024-04-20T17:23:42.518 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6806/2877589636", 2024-04-20T17:23:42.518 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:23:42.518 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:23:42.518 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:23:42.518 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.518 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.518 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.518 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 10, 2024-04-20T17:23:42.518 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "1adcdf55-f002-4a39-91e7-378e5fae849c", 2024-04-20T17:23:42.518 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:23:42.518 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:23:42.519 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:23:42.519 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:23:42.519 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:23:42.519 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:23:42.519 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 84, 2024-04-20T17:23:42.519 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 0, 2024-04-20T17:23:42.519 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:23:42.519 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:23:42.519 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:23:42.519 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.519 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.519 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.519 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6808", 2024-04-20T17:23:42.519 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:23:42.520 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.520 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.520 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.520 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:23:42.520 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.520 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.520 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.520 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6809", 2024-04-20T17:23:42.520 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:23:42.520 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.520 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.520 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.520 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:23:42.521 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.521 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.521 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.521 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6811", 2024-04-20T17:23:42.521 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:23:42.521 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.521 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.521 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.521 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:23:42.521 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.521 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.521 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.521 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6810", 2024-04-20T17:23:42.521 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:23:42.522 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.522 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.522 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.522 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6808/181490621", 2024-04-20T17:23:42.522 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6809/181490621", 2024-04-20T17:23:42.522 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6811/181490621", 2024-04-20T17:23:42.522 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6810/181490621", 2024-04-20T17:23:42.522 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:23:42.522 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:23:42.522 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:23:42.522 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.522 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 11, 2024-04-20T17:23:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "737c5aec-694a-4968-819d-fccf22661f4f", 2024-04-20T17:23:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:23:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:23:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:23:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:23:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:23:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:23:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 92, 2024-04-20T17:23:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 368, 2024-04-20T17:23:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:23:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:23:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:23:42.523 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.524 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.524 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.524 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6812", 2024-04-20T17:23:42.524 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:23:42.524 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.524 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.524 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.524 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:23:42.524 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.524 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.524 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.524 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6813", 2024-04-20T17:23:42.524 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:23:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:23:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6815", 2024-04-20T17:23:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:23:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:23:42.525 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:23:42.526 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.526 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:23:42.526 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6814", 2024-04-20T17:23:42.526 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:23:42.526 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.526 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.526 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.526 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6812/3373585701", 2024-04-20T17:23:42.526 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6813/3373585701", 2024-04-20T17:23:42.526 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6815/3373585701", 2024-04-20T17:23:42.526 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6814/3373585701", 2024-04-20T17:23:42.526 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:23:42.526 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:23:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:23:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:23:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:23:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_xinfo": [ 2024-04-20T17:23:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 0, 2024-04-20T17:23:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:23:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:23:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:23:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:23:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:23:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:23:32.179740+0000", 2024-04-20T17:23:42.527 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:23:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 1, 2024-04-20T17:23:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:23:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:23:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:23:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:23:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:23:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:22:28.598520+0000", 2024-04-20T17:23:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:23:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 2, 2024-04-20T17:23:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:23:42.528 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:23:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:23:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:23:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:23:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:23:35.173618+0000", 2024-04-20T17:23:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 169 2024-04-20T17:23:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 3, 2024-04-20T17:23:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:23:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:23:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:23:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:23:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:23:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:23:03.632098+0000", 2024-04-20T17:23:42.529 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:23:42.530 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.530 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.530 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 4, 2024-04-20T17:23:42.530 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:23:42.530 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:23:42.530 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:23:42.530 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:23:42.530 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:23:42.530 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:23:01.455397+0000", 2024-04-20T17:23:42.530 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:23:42.530 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.530 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.530 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 5, 2024-04-20T17:23:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:23:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:23:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:23:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:23:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:23:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:23:08.762511+0000", 2024-04-20T17:23:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:23:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 6, 2024-04-20T17:23:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:23:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:23:42.531 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:23:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:23:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:23:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:23:00.700887+0000", 2024-04-20T17:23:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:23:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 7, 2024-04-20T17:23:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:23:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:23:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:23:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:23:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:23:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:23:37.791872+0000", 2024-04-20T17:23:42.532 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 364 2024-04-20T17:23:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 8, 2024-04-20T17:23:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:23:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:23:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:23:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:23:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:23:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:22:59.284699+0000", 2024-04-20T17:23:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:23:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 9, 2024-04-20T17:23:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:23:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:23:42.533 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:23:42.534 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:23:42.534 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:23:42.534 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:23:39.981739+0000", 2024-04-20T17:23:42.534 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:23:42.534 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.534 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.534 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 10, 2024-04-20T17:23:42.534 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:23:42.534 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:23:42.534 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:23:42.534 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:23:42.534 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:23:42.534 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:23:03.728991+0000", 2024-04-20T17:23:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:23:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:23:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 11, 2024-04-20T17:23:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:23:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:23:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:23:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:23:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:23:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:22:54.922801+0000", 2024-04-20T17:23:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:23:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:23:42.535 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_upmap": [], 2024-04-20T17:23:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_upmap_items": [], 2024-04-20T17:23:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_upmap_primaries": [], 2024-04-20T17:23:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_temp": [], 2024-04-20T17:23:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_temp": [], 2024-04-20T17:23:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: "blocklist": { 2024-04-20T17:23:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/3881726425": "2024-04-21T16:59:25.808337+0000", 2024-04-20T17:23:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/338649532": "2024-04-21T16:59:25.808337+0000", 2024-04-20T17:23:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/1227843664": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:23:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:6800/1876064631": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:23:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/160689449": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:23:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:6800/3521715286": "2024-04-21T16:59:25.808337+0000", 2024-04-20T17:23:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/3841663462": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:23:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/687690919": "2024-04-21T16:59:25.808337+0000" 2024-04-20T17:23:42.536 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.537 INFO:teuthology.orchestra.run.smithi005.stdout: "range_blocklist": {}, 2024-04-20T17:23:42.537 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_profiles": { 2024-04-20T17:23:42.537 INFO:teuthology.orchestra.run.smithi005.stdout: "default": { 2024-04-20T17:23:42.537 INFO:teuthology.orchestra.run.smithi005.stdout: "crush-failure-domain": "osd", 2024-04-20T17:23:42.537 INFO:teuthology.orchestra.run.smithi005.stdout: "k": "2", 2024-04-20T17:23:42.537 INFO:teuthology.orchestra.run.smithi005.stdout: "m": "1", 2024-04-20T17:23:42.537 INFO:teuthology.orchestra.run.smithi005.stdout: "plugin": "jerasure", 2024-04-20T17:23:42.537 INFO:teuthology.orchestra.run.smithi005.stdout: "technique": "reed_sol_van" 2024-04-20T17:23:42.537 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.537 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:23:42.537 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_snaps_queue": [], 2024-04-20T17:23:42.537 INFO:teuthology.orchestra.run.smithi005.stdout: "new_removed_snaps": [], 2024-04-20T17:23:42.537 INFO:teuthology.orchestra.run.smithi005.stdout: "new_purged_snaps": [], 2024-04-20T17:23:42.537 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_node_flags": {}, 2024-04-20T17:23:42.537 INFO:teuthology.orchestra.run.smithi005.stdout: "device_class_flags": {}, 2024-04-20T17:23:42.538 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": { 2024-04-20T17:23:42.538 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode_enabled": false, 2024-04-20T17:23:42.538 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_bucket_count": 0, 2024-04-20T17:23:42.538 INFO:teuthology.orchestra.run.smithi005.stdout: "degraded_stretch_mode": 0, 2024-04-20T17:23:42.538 INFO:teuthology.orchestra.run.smithi005.stdout: "recovering_stretch_mode": 0, 2024-04-20T17:23:42.538 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode_bucket": 0 2024-04-20T17:23:42.538 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:23:42.538 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:23:42.538 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:23:42.539 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:42 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4266817967' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:23:42.539 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:42 smithi106 ceph-mon[44423]: from='client.26024 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:23:42.605 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:42 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4266817967' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:23:42.605 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:42 smithi005 ceph-mon[39562]: from='client.26024 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:23:42.608 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:42.793 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:23:42.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:42.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:23:42.902 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:23:42.936 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:43.070 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:23:43.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:43.073 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:23:43.116 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:23:43.116 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:23:43.173 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:23:43.202 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:43.227 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:23:43.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:43.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:23:43.331 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:23:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:43 smithi171 ceph-mon[44189]: pgmap v1142: 17 pgs: 17 active+clean; 2.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 970 op/s; 15 MiB/s, 237 objects/s recovering 2024-04-20T17:23:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:43 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:23:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:43 smithi171 ceph-mon[44189]: from='client.26024 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:23:43.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:43 smithi171 ceph-mon[44189]: osdmap e425: 12 total, 12 up, 12 in 2024-04-20T17:23:43.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:43 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4172115684' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-20T17:23:43.400 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:43.408 INFO:tasks.thrashosds.thrasher:j is {'epoch': 425, 'fsid': '13c6e76c-ff37-11ee-bc93-c7b262605968', 'created': '2024-04-20T16:58:25.272557+0000', 'modified': '2024-04-20T17:23:42.259320+0000', 'last_up_change': '2024-04-20T17:22:32.243867+0000', 'last_in_change': '2024-04-20T17:23:20.118061+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 29, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-20T17:01:35.909493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.109999656677246, 'score_stable': 11.109999656677246, 'optimal_score': 0.18000000715255737, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.916666567325592, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 3, 'pool_name': 'unique_pool_1', 'create_time': '2024-04-20T17:20:21.928963+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '360', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.440000057220459, 'score_stable': 3.440000057220459, 'optimal_score': 0.9100000262260437, 'raw_score_acting': 3.130000114440918, 'raw_score_stable': 3.130000114440918, 'primary_affinity_weighted': 0.916666567325592, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': 'ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 356, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6801', 'nonce': 3901484705}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6802', 'nonce': 3901484705}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6804', 'nonce': 3901484705}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6803', 'nonce': 3901484705}]}, 'public_addr': '172.21.15.5:6801/3901484705', 'cluster_addr': '172.21.15.5:6802/3901484705', 'heartbeat_back_addr': '172.21.15.5:6804/3901484705', 'heartbeat_front_addr': '172.21.15.5:6803/3901484705', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'aed0c8eb-0e81-4e85-ab8f-a60420060656', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 356, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6805', 'nonce': 3391893914}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6806', 'nonce': 3391893914}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6808', 'nonce': 3391893914}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6807', 'nonce': 3391893914}]}, 'public_addr': '172.21.15.5:6805/3391893914', 'cluster_addr': '172.21.15.5:6806/3391893914', 'heartbeat_back_addr': '172.21.15.5:6808/3391893914', 'heartbeat_front_addr': '172.21.15.5:6807/3391893914', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f733a6a2-19c2-4c7a-af9c-04e282062bb7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 20, 'last_clean_end': 169, 'up_from': 202, 'up_thru': 202, 'down_at': 170, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6809', 'nonce': 2120275501}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6810', 'nonce': 2120275501}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6812', 'nonce': 2120275501}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6811', 'nonce': 2120275501}]}, 'public_addr': '172.21.15.5:6809/2120275501', 'cluster_addr': '172.21.15.5:6810/2120275501', 'heartbeat_back_addr': '172.21.15.5:6812/2120275501', 'heartbeat_front_addr': '172.21.15.5:6811/2120275501', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '8c92f1df-e298-4606-88ae-b8a8e79628a1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 417, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6813', 'nonce': 377652039}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6814', 'nonce': 377652039}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6816', 'nonce': 377652039}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6815', 'nonce': 377652039}]}, 'public_addr': '172.21.15.5:6813/377652039', 'cluster_addr': '172.21.15.5:6814/377652039', 'heartbeat_back_addr': '172.21.15.5:6816/377652039', 'heartbeat_front_addr': '172.21.15.5:6815/377652039', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '46df16f1-9607-4d7b-9be8-9d4236e160d1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 356, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6800', 'nonce': 1839524469}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6801', 'nonce': 1839524469}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6803', 'nonce': 1839524469}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6802', 'nonce': 1839524469}]}, 'public_addr': '172.21.15.106:6800/1839524469', 'cluster_addr': '172.21.15.106:6801/1839524469', 'heartbeat_back_addr': '172.21.15.106:6803/1839524469', 'heartbeat_front_addr': '172.21.15.106:6802/1839524469', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6603231d-1e5c-4575-bfaa-2eeca2c12215', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 417, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6804', 'nonce': 2450315908}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6805', 'nonce': 2450315908}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6807', 'nonce': 2450315908}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6806', 'nonce': 2450315908}]}, 'public_addr': '172.21.15.106:6804/2450315908', 'cluster_addr': '172.21.15.106:6805/2450315908', 'heartbeat_back_addr': '172.21.15.106:6807/2450315908', 'heartbeat_front_addr': '172.21.15.106:6806/2450315908', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '783a55a6-f176-42be-8c8b-af8481d08dc9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 413, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6808', 'nonce': 2840237913}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6809', 'nonce': 2840237913}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6811', 'nonce': 2840237913}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6810', 'nonce': 2840237913}]}, 'public_addr': '172.21.15.106:6808/2840237913', 'cluster_addr': '172.21.15.106:6809/2840237913', 'heartbeat_back_addr': '172.21.15.106:6811/2840237913', 'heartbeat_front_addr': '172.21.15.106:6810/2840237913', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 56, 'last_clean_end': 364, 'up_from': 398, 'up_thru': 419, 'down_at': 365, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6812', 'nonce': 1997086583}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6813', 'nonce': 1997086583}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6815', 'nonce': 1997086583}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6814', 'nonce': 1997086583}]}, 'public_addr': '172.21.15.106:6812/1997086583', 'cluster_addr': '172.21.15.106:6813/1997086583', 'heartbeat_back_addr': '172.21.15.106:6815/1997086583', 'heartbeat_front_addr': '172.21.15.106:6814/1997086583', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '1cb232ad-1f59-4a0a-a215-e67419ad6476', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 67, 'up_thru': 418, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6800', 'nonce': 4123058942}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6801', 'nonce': 4123058942}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6803', 'nonce': 4123058942}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6802', 'nonce': 4123058942}]}, 'public_addr': '172.21.15.171:6800/4123058942', 'cluster_addr': '172.21.15.171:6801/4123058942', 'heartbeat_back_addr': '172.21.15.171:6803/4123058942', 'heartbeat_front_addr': '172.21.15.171:6802/4123058942', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'b6c17147-b3dc-475b-bb20-f9f712fb99e8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 73, 'up_thru': 196, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6804', 'nonce': 2877589636}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6805', 'nonce': 2877589636}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6807', 'nonce': 2877589636}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6806', 'nonce': 2877589636}]}, 'public_addr': '172.21.15.171:6804/2877589636', 'cluster_addr': '172.21.15.171:6805/2877589636', 'heartbeat_back_addr': '172.21.15.171:6807/2877589636', 'heartbeat_front_addr': '172.21.15.171:6806/2877589636', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '1adcdf55-f002-4a39-91e7-378e5fae849c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 84, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6808', 'nonce': 181490621}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6809', 'nonce': 181490621}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6811', 'nonce': 181490621}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6810', 'nonce': 181490621}]}, 'public_addr': '172.21.15.171:6808/181490621', 'cluster_addr': '172.21.15.171:6809/181490621', 'heartbeat_back_addr': '172.21.15.171:6811/181490621', 'heartbeat_front_addr': '172.21.15.171:6810/181490621', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '737c5aec-694a-4968-819d-fccf22661f4f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 92, 'up_thru': 368, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6812', 'nonce': 3373585701}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6813', 'nonce': 3373585701}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6815', 'nonce': 3373585701}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6814', 'nonce': 3373585701}]}, 'public_addr': '172.21.15.171:6812/3373585701', 'cluster_addr': '172.21.15.171:6813/3373585701', 'heartbeat_back_addr': '172.21.15.171:6815/3373585701', 'heartbeat_front_addr': '172.21.15.171:6814/3373585701', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:23:32.179740+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:22:28.598520+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:23:35.173618+0000', 'dead_epoch': 169}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:23:03.632098+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:23:01.455397+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:23:08.762511+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:23:00.700887+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:23:37.791872+0000', 'dead_epoch': 364}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:22:59.284699+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:23:39.981739+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:23:03.728991+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:22:54.922801+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.5:0/3881726425': '2024-04-21T16:59:25.808337+0000', '172.21.15.5:0/338649532': '2024-04-21T16:59:25.808337+0000', '172.21.15.5:0/1227843664': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:6800/1876064631': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:0/160689449': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:6800/3521715286': '2024-04-21T16:59:25.808337+0000', '172.21.15.5:0/3841663462': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:0/687690919': '2024-04-21T16:59:25.808337+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-20T17:23:43.408 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:23:43.472 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:23:43.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:43.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:23:43.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:43 smithi106 ceph-mon[44423]: pgmap v1142: 17 pgs: 17 active+clean; 2.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 970 op/s; 15 MiB/s, 237 objects/s recovering 2024-04-20T17:23:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:43 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:23:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:43 smithi106 ceph-mon[44423]: from='client.26024 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:23:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:43 smithi106 ceph-mon[44423]: osdmap e425: 12 total, 12 up, 12 in 2024-04-20T17:23:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:43 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4172115684' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-20T17:23:43.592 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:23:43.625 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:43.643 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:23:43.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:43.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:23:43.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:43 smithi005 ceph-mon[39562]: pgmap v1142: 17 pgs: 17 active+clean; 2.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 970 op/s; 15 MiB/s, 237 objects/s recovering 2024-04-20T17:23:43.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:43 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:23:43.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:43 smithi005 ceph-mon[39562]: from='client.26024 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:23:43.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:43 smithi005 ceph-mon[39562]: osdmap e425: 12 total, 12 up, 12 in 2024-04-20T17:23:43.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:43 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4172115684' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-20T17:23:43.746 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:23:43.781 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:43.818 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:23:43.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:43.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:23:43.923 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:23:43.957 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:43.985 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:23:43.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:43.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:23:43.989 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-20T17:23:44.089 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:23:44.148 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:44.189 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:23:44.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:44.192 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:23:44.293 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:23:44.350 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:44.371 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:23:44.882 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:23:44.927 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:23:44.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:44.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:23:45.036 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:23:45.072 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:45.326 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:23:45.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:45.331 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:23:45.432 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:23:45.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:45 smithi005 ceph-mon[39562]: pgmap v1144: 17 pgs: 17 active+clean; 1.9 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-04-20T17:23:45.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:45 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:23:45.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:45 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:23:45.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:45 smithi005 ceph-mon[39562]: osdmap e426: 12 total, 12 up, 12 in 2024-04-20T17:23:45.460 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:45.483 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:23:45.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:45.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:23:45.535 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:45 smithi171 ceph-mon[44189]: pgmap v1144: 17 pgs: 17 active+clean; 1.9 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-04-20T17:23:45.535 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:45 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:23:45.535 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:45 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:23:45.535 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:45 smithi171 ceph-mon[44189]: osdmap e426: 12 total, 12 up, 12 in 2024-04-20T17:23:45.587 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:23:45.651 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:45 smithi106 ceph-mon[44423]: pgmap v1144: 17 pgs: 17 active+clean; 1.9 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-04-20T17:23:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:45 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:23:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:45 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:23:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:45 smithi106 ceph-mon[44423]: osdmap e426: 12 total, 12 up, 12 in 2024-04-20T17:23:45.857 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:23:46.004 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:23:46.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:46.008 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:23:46.109 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:23:46.201 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:46.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:46 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3644240611' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:23:46.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:46.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:46 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3644240611' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:23:46.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:46.641 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:23:46.641 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:23:46.641 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:23:46.641 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:23:46.641 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:23:46.642 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:23:46.642 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:23:46.642 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:23:46.642 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:23:46.642 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:23:46.642 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:23:46.642 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:23:46.642 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:23:46.642 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:23:46.642 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:23:46.642 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:23:46.643 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:23:46.643 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:23:46.643 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:23:46.643 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:23:46.643 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:23:46.643 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:23:46.643 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:23:46.643 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:23:46.702 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:23:46.703 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:23:46.703 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.703 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:260747 3.e 3:7c4c3586:::benchmark_data_smithi171_110765_object117532:head [delete] snapc 0=[] ondisk+write+known_if_redirected e426)", 2024-04-20T17:23:46.703 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:46.689678+0000", 2024-04-20T17:23:46.703 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.0082085620000000008, 2024-04-20T17:23:46.703 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.008238914, 2024-04-20T17:23:46.703 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:46.703 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:46.703 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:23:46.703 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:46.703 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:46.703 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:46.704 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 260747 2024-04-20T17:23:46.704 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.704 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:46.704 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.704 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:46.704 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.689678+0000", 2024-04-20T17:23:46.704 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:46.704 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.704 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.704 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:46.704 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.689678+0000", 2024-04-20T17:23:46.704 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:46.704 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.705 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.705 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:46.705 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.689680+0000", 2024-04-20T17:23:46.705 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2359999999999999e-06 2024-04-20T17:23:46.705 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.705 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.705 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:46.705 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.689686+0000", 2024-04-20T17:23:46.705 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9000000000000003e-06 2024-04-20T17:23:46.705 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.705 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.705 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:46.705 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.689700+0000", 2024-04-20T17:23:46.705 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3973e-05 2024-04-20T17:23:46.706 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.706 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.706 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:46.706 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.689709+0000", 2024-04-20T17:23:46.706 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0070000000000003e-06 2024-04-20T17:23:46.706 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.706 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.706 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:46.706 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.689784+0000", 2024-04-20T17:23:46.706 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5028000000000001e-05 2024-04-20T17:23:46.706 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.706 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.706 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:46.706 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.689921+0000", 2024-04-20T17:23:46.707 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013731 2024-04-20T17:23:46.707 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.707 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.707 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4", 2024-04-20T17:23:46.707 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.690048+0000", 2024-04-20T17:23:46.707 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000126864 2024-04-20T17:23:46.707 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.707 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.707 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:46.707 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.695210+0000", 2024-04-20T17:23:46.707 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0051622109999999999 2024-04-20T17:23:46.707 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:46.707 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:46.707 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:46.707 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.707 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.708 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:260751 3.8 3:1e4d0bcf:::benchmark_data_smithi171_110765_object117536:head [delete] snapc 0=[] ondisk+write+known_if_redirected e426)", 2024-04-20T17:23:46.708 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:46.691874+0000", 2024-04-20T17:23:46.708 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.0060126099999999998, 2024-04-20T17:23:46.708 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0061350959999999996, 2024-04-20T17:23:46.708 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:46.708 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:46.708 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:23:46.708 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:46.708 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:46.708 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:46.708 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 260751 2024-04-20T17:23:46.708 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.708 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:46.709 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.709 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:46.709 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.691874+0000", 2024-04-20T17:23:46.709 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:46.709 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.709 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.709 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:46.709 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.691874+0000", 2024-04-20T17:23:46.709 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:46.709 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.709 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.709 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:46.709 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.691875+0000", 2024-04-20T17:23:46.710 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4589999999999999e-06 2024-04-20T17:23:46.710 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.710 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.710 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:46.710 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.691881+0000", 2024-04-20T17:23:46.710 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7479999999999999e-06 2024-04-20T17:23:46.710 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.710 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.710 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:46.710 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.691894+0000", 2024-04-20T17:23:46.710 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2687000000000001e-05 2024-04-20T17:23:46.710 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.710 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.710 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:46.711 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.691902+0000", 2024-04-20T17:23:46.711 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2849999999999995e-06 2024-04-20T17:23:46.711 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.711 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.711 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:46.711 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.691966+0000", 2024-04-20T17:23:46.711 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4447000000000001e-05 2024-04-20T17:23:46.711 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.711 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.711 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:46.711 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.692078+0000", 2024-04-20T17:23:46.711 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000111785 2024-04-20T17:23:46.711 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.711 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.712 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:23:46.712 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.692185+0000", 2024-04-20T17:23:46.712 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010708 2024-04-20T17:23:46.712 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:46.712 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:46.712 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:46.712 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.712 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.712 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:260754 3.8 3:1e3b1a3d:::benchmark_data_smithi171_110765_object117539:head [delete] snapc 0=[] ondisk+write+known_if_redirected e426)", 2024-04-20T17:23:46.712 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:46.692103+0000", 2024-04-20T17:23:46.712 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.0057831779999999999, 2024-04-20T17:23:46.712 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0059790980000000004, 2024-04-20T17:23:46.712 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:46.712 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:46.712 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:23:46.712 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:46.713 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:46.713 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:46.713 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 260754 2024-04-20T17:23:46.713 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.713 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:46.713 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.713 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:46.713 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.692103+0000", 2024-04-20T17:23:46.713 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:46.713 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.713 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.713 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:46.713 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.692103+0000", 2024-04-20T17:23:46.713 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:46.713 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.714 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.714 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:46.714 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.692104+0000", 2024-04-20T17:23:46.714 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.362e-06 2024-04-20T17:23:46.714 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.714 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.714 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:46.714 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.692109+0000", 2024-04-20T17:23:46.714 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7520000000000004e-06 2024-04-20T17:23:46.714 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.714 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.714 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:46.714 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.692119+0000", 2024-04-20T17:23:46.714 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.4579999999999999e-06 2024-04-20T17:23:46.714 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.715 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.715 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:46.715 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.692126+0000", 2024-04-20T17:23:46.715 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2509999999999997e-06 2024-04-20T17:23:46.715 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.715 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.715 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:46.715 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.696489+0000", 2024-04-20T17:23:46.715 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0043628690000000001 2024-04-20T17:23:46.715 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.715 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.715 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:46.715 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.696614+0000", 2024-04-20T17:23:46.715 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000125208 2024-04-20T17:23:46.715 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.715 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.716 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:23:46.716 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.696735+0000", 2024-04-20T17:23:46.716 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012136199999999999 2024-04-20T17:23:46.716 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:46.716 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:46.716 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:46.716 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.716 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.716 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:260755 3.7 e426/356 3:e1ad1945:::benchmark_data_smithi171_110765_object117540:head v 426'16777, mlcod=426'16775)", 2024-04-20T17:23:46.716 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:46.693276+0000", 2024-04-20T17:23:46.716 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.0046103170000000001, 2024-04-20T17:23:46.716 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0048793129999999997, 2024-04-20T17:23:46.716 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:46.716 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:46.716 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:23:46.716 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:46.716 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.717 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:46.717 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.693276+0000", 2024-04-20T17:23:46.717 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:46.717 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.717 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.717 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:46.717 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.693276+0000", 2024-04-20T17:23:46.717 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:46.717 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.717 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.717 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:46.717 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.693277+0000", 2024-04-20T17:23:46.717 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6599999999999995e-07 2024-04-20T17:23:46.717 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.717 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.718 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:46.718 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.693282+0000", 2024-04-20T17:23:46.718 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8559999999999998e-06 2024-04-20T17:23:46.718 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.718 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.718 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:46.718 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.693287+0000", 2024-04-20T17:23:46.718 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.519e-06 2024-04-20T17:23:46.718 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.718 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.718 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:46.718 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.693292+0000", 2024-04-20T17:23:46.718 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1599999999999997e-06 2024-04-20T17:23:46.718 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.719 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.719 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:46.719 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.693333+0000", 2024-04-20T17:23:46.719 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1048999999999999e-05 2024-04-20T17:23:46.719 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.719 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.719 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:46.719 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.693354+0000", 2024-04-20T17:23:46.719 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1095999999999999e-05 2024-04-20T17:23:46.719 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:46.719 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:46.719 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:46.719 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.719 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.720 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:260756 3.f e426/419 3:f1181fc9:::benchmark_data_smithi171_110765_object117541:head v 426'18795, mlcod=426'18793)", 2024-04-20T17:23:46.720 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:46.693307+0000", 2024-04-20T17:23:46.720 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.0045792719999999997, 2024-04-20T17:23:46.720 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.004919858, 2024-04-20T17:23:46.720 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:46.720 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:46.720 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:23:46.720 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:46.720 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.720 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:46.720 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.693307+0000", 2024-04-20T17:23:46.720 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:46.721 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.721 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.721 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:46.721 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.693307+0000", 2024-04-20T17:23:46.721 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:46.721 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.721 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.721 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:46.721 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.693308+0000", 2024-04-20T17:23:46.721 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1500000000000004e-07 2024-04-20T17:23:46.721 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.721 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.721 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:46.722 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.693312+0000", 2024-04-20T17:23:46.722 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6269999999999999e-06 2024-04-20T17:23:46.722 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.722 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.722 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:46.722 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.693318+0000", 2024-04-20T17:23:46.722 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4199999999999998e-06 2024-04-20T17:23:46.722 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.722 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.722 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:46.722 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.693322+0000", 2024-04-20T17:23:46.722 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4700000000000004e-06 2024-04-20T17:23:46.722 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.723 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.723 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:46.723 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.695247+0000", 2024-04-20T17:23:46.723 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0019252939999999999 2024-04-20T17:23:46.723 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.723 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.723 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:46.723 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.695267+0000", 2024-04-20T17:23:46.723 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9811000000000001e-05 2024-04-20T17:23:46.723 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:46.723 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:46.723 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:46.723 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.723 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.723 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:260749 3.4 e426/356 3:2d8defe7:::benchmark_data_smithi171_110765_object117534:head v 426'16258, mlcod=426'16254)", 2024-04-20T17:23:46.724 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:46.697369+0000", 2024-04-20T17:23:46.724 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.000517648, 2024-04-20T17:23:46.724 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00091666100000000004, 2024-04-20T17:23:46.724 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:46.724 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:46.724 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:23:46.724 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:46.724 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.724 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:46.724 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697369+0000", 2024-04-20T17:23:46.724 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:46.724 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.724 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.725 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:46.725 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697369+0000", 2024-04-20T17:23:46.725 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:46.725 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.725 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.725 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:46.725 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697370+0000", 2024-04-20T17:23:46.725 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5340000000000001e-06 2024-04-20T17:23:46.725 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.725 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.725 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:46.725 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697380+0000", 2024-04-20T17:23:46.725 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0332999999999999e-05 2024-04-20T17:23:46.725 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.726 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.726 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:46.726 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697391+0000", 2024-04-20T17:23:46.726 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0688e-05 2024-04-20T17:23:46.726 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.726 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.726 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:46.726 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697398+0000", 2024-04-20T17:23:46.726 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3250000000000002e-06 2024-04-20T17:23:46.726 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.726 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.726 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:46.726 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697508+0000", 2024-04-20T17:23:46.727 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000109984 2024-04-20T17:23:46.727 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.727 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.727 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:46.727 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697532+0000", 2024-04-20T17:23:46.727 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3657000000000001e-05 2024-04-20T17:23:46.727 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:46.727 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:46.727 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:46.727 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.727 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.727 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:260750 3.4 e426/356)", 2024-04-20T17:23:46.727 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:46.697423+0000", 2024-04-20T17:23:46.727 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.00046304799999999998, 2024-04-20T17:23:46.728 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00091619200000000003, 2024-04-20T17:23:46.728 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:46.728 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:46.728 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "queued for pg", 2024-04-20T17:23:46.728 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:46.728 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.728 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:46.728 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697423+0000", 2024-04-20T17:23:46.728 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:46.728 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.728 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.728 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:46.728 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697423+0000", 2024-04-20T17:23:46.728 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:46.729 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.729 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.729 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:46.729 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697424+0000", 2024-04-20T17:23:46.729 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6700000000000003e-07 2024-04-20T17:23:46.729 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.729 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.729 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:46.729 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697429+0000", 2024-04-20T17:23:46.729 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8709999999999996e-06 2024-04-20T17:23:46.729 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.729 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.729 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:46.730 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697435+0000", 2024-04-20T17:23:46.730 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0990000000000004e-06 2024-04-20T17:23:46.730 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.730 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.730 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:46.730 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697440+0000", 2024-04-20T17:23:46.730 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2390000000000002e-06 2024-04-20T17:23:46.730 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:46.730 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:46.730 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:46.730 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.730 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.730 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:260759 3.e 3:72cdf903:::benchmark_data_smithi171_110765_object117544:head [delete] snapc 0=[] ondisk+write+known_if_redirected e426)", 2024-04-20T17:23:46.730 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:46.697560+0000", 2024-04-20T17:23:46.731 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.00032598300000000001, 2024-04-20T17:23:46.731 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000830702, 2024-04-20T17:23:46.731 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:46.731 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:46.731 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:23:46.731 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:46.731 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:46.731 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:46.731 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 260759 2024-04-20T17:23:46.731 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.731 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:46.732 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.732 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:46.732 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697560+0000", 2024-04-20T17:23:46.732 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:46.732 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.732 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.732 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:46.732 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697560+0000", 2024-04-20T17:23:46.732 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:46.732 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.732 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.732 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:46.732 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697563+0000", 2024-04-20T17:23:46.733 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2560000000000001e-06 2024-04-20T17:23:46.733 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.733 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.733 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:46.733 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697569+0000", 2024-04-20T17:23:46.733 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2759999999999997e-06 2024-04-20T17:23:46.733 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.733 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.733 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:46.733 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697585+0000", 2024-04-20T17:23:46.733 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6058000000000001e-05 2024-04-20T17:23:46.733 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.733 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.733 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:46.734 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697593+0000", 2024-04-20T17:23:46.734 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5180000000000004e-06 2024-04-20T17:23:46.734 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.734 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.734 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:46.734 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697650+0000", 2024-04-20T17:23:46.734 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6422000000000003e-05 2024-04-20T17:23:46.734 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.734 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.734 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:46.734 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697776+0000", 2024-04-20T17:23:46.734 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000126024 2024-04-20T17:23:46.734 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.734 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.734 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4", 2024-04-20T17:23:46.735 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697896+0000", 2024-04-20T17:23:46.735 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011975499999999999 2024-04-20T17:23:46.735 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:46.735 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:46.735 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:46.735 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.735 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.735 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop_reply(client.25802.0:260747 3.e e426/356)", 2024-04-20T17:23:46.735 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:46.697922+0000", 2024-04-20T17:23:46.735 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 4294967295.9999638, 2024-04-20T17:23:46.735 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00053791899999999998, 2024-04-20T17:23:46.735 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:46.735 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:46.735 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "queued for pg", 2024-04-20T17:23:46.735 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:46.736 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.736 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:46.736 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697922+0000", 2024-04-20T17:23:46.736 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:46.736 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.736 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.736 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:46.736 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697922+0000", 2024-04-20T17:23:46.736 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:46.736 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.736 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.736 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:46.736 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697924+0000", 2024-04-20T17:23:46.737 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.331e-06 2024-04-20T17:23:46.737 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.737 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.737 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:46.737 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697932+0000", 2024-04-20T17:23:46.737 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0479999999999999e-06 2024-04-20T17:23:46.737 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.737 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.737 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:46.737 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697940+0000", 2024-04-20T17:23:46.737 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.551e-06 2024-04-20T17:23:46.737 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:46.737 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:46.737 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:46.737 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:46.697947+0000", 2024-04-20T17:23:46.738 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8220000000000001e-06 2024-04-20T17:23:46.738 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:46.738 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:46.738 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:46.738 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:46.738 INFO:teuthology.orchestra.run.smithi171.stdout: ], 2024-04-20T17:23:46.738 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 9 2024-04-20T17:23:46.738 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:23:46.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:46 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3644240611' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:23:46.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.282 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.282 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.282 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.282 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.282 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.283 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.283 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.283 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.283 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:47 smithi005 ceph-mon[39562]: pgmap v1146: 17 pgs: 17 active+clean; 1.7 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.36k op/s 2024-04-20T17:23:47.283 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.283 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.283 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.529 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.529 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.530 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.530 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.530 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.530 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.530 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.530 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.530 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:47 smithi171 ceph-mon[44189]: pgmap v1146: 17 pgs: 17 active+clean; 1.7 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.36k op/s 2024-04-20T17:23:47.530 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.530 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.530 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.563 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-20T17:23:47.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:47.583 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:23:47.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:23:47.684 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:23:47.711 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.711 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.711 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.711 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.711 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.711 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.712 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.712 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.712 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:47 smithi106 ceph-mon[44423]: pgmap v1146: 17 pgs: 17 active+clean; 1.7 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.36k op/s 2024-04-20T17:23:47.712 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.712 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.712 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:47.713 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:47.740 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:23:47.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:47.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:23:47.847 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:23:47.860 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:23:47.878 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:47.899 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:23:47.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:47.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:23:47.921 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:23:48.008 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:23:48.043 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:48.229 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:23:48.236 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:23:48.246 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:23:48.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:48.251 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:23:48.354 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:23:48.393 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:48.547 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:23:48.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:48.554 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:23:48.655 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:23:48.689 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:48.943 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:23:48.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:48.947 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:23:49.048 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:23:49.095 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:49.372 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:23:49.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:49 smithi005 ceph-mon[39562]: pgmap v1147: 17 pgs: 17 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.35k op/s 2024-04-20T17:23:49.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:49 smithi005 ceph-mon[39562]: from='client.26039 v1:172.21.15.5:0/1813519813' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:23:49.477 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:23:49.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:49.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:23:49.508 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":1147,"stamp":"2024-04-20T17:23:48.174164+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1704657464,"num_objects":26005,"num_object_clones":0,"num_object_copies":52010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26005,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":260604,"num_write_kb":9168507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9649,"num_bytes_recovered":633882776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160973,"ondisk_log_size":160973,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1124941824,"kb_used":5761280,"kb_used_data":3726084,"kb_used_omap":224,"kb_used_meta":2034847,"kb_avail":1119180544,"statfs":{"total":1151940427776,"available":1146040877056,"internally_reserved":0,"allocated":3815510016,"data_stored":3811329576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":230126,"internal_metadata":2083683602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-707198976,"num_objects":-10791,"num_object_clones":0,"num_object_copies":-21582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10791,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005526"},"pg_stats":[{"pgid":"3.f","version":"426'18833","reported_seq":28924,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-04-20T17:23:47.217268+0000","last_change":"2024-04-20T17:23:34.149828+0000","last_active":"2024-04-20T17:23:47.217268+0000","last_peered":"2024-04-20T17:23:47.217268+0000","last_clean":"2024-04-20T17:23:47.217268+0000","last_became_active":"2024-04-20T17:23:34.107587+0000","last_became_peered":"2024-04-20T17:23:34.107587+0000","last_unstale":"2024-04-20T17:23:47.217268+0000","last_undegraded":"2024-04-20T17:23:47.217268+0000","last_fullsized":"2024-04-20T17:23:47.217268+0000","mapping_epoch":419,"log_start":"383'8800","ondisk_log_start":"383'8800","created":356,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'11850","last_scrub_stamp":"2024-04-20T17:22:06.236565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:22:06.236565+0000","objects_scrubbed":8637,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:23:47.770615+0000","scrub_duration":1904,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103546880,"num_objects":1580,"num_object_clones":0,"num_object_copies":3160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16220,"num_write_kb":569600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4818,"num_bytes_recovered":315752448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"426'16361","reported_seq":16598,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-04-20T17:23:48.052990+0000","last_change":"2024-04-20T17:22:13.829497+0000","last_active":"2024-04-20T17:23:48.052990+0000","last_peered":"2024-04-20T17:23:48.052990+0000","last_clean":"2024-04-20T17:23:48.052990+0000","last_became_active":"2024-04-20T17:20:23.037195+0000","last_became_peered":"2024-04-20T17:20:23.037195+0000","last_unstale":"2024-04-20T17:23:48.052990+0000","last_undegraded":"2024-04-20T17:23:48.052990+0000","last_fullsized":"2024-04-20T17:23:48.052990+0000","mapping_epoch":356,"log_start":"379'6300","ondisk_log_start":"379'6300","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"392'9774","last_scrub_stamp":"2024-04-20T17:22:13.829328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:22:13.829328+0000","objects_scrubbed":8074,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1538,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94437376,"num_objects":1441,"num_object_clones":0,"num_object_copies":2882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16361,"num_write_kb":569664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4],"acting":[11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"426'16309","reported_seq":26250,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-04-20T17:23:47.249212+0000","last_change":"2024-04-20T17:23:33.152328+0000","last_active":"2024-04-20T17:23:47.249212+0000","last_peered":"2024-04-20T17:23:47.249212+0000","last_clean":"2024-04-20T17:23:47.249212+0000","last_became_active":"2024-04-20T17:23:33.104412+0000","last_became_peered":"2024-04-20T17:23:33.104412+0000","last_unstale":"2024-04-20T17:23:47.249212+0000","last_undegraded":"2024-04-20T17:23:47.249212+0000","last_fullsized":"2024-04-20T17:23:47.249212+0000","mapping_epoch":418,"log_start":"379'6300","ondisk_log_start":"379'6300","created":356,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'13123","last_scrub_stamp":"2024-04-20T17:22:57.535030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:22:57.535030+0000","objects_scrubbed":4766,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:24:03.103493+0000","scrub_duration":863,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101842968,"num_objects":1555,"num_object_clones":0,"num_object_copies":3110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1555,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16309,"num_write_kb":571585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4823,"num_bytes_recovered":316014616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"426'16057","reported_seq":16335,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-04-20T17:23:46.084419+0000","last_change":"2024-04-20T17:23:33.729171+0000","last_active":"2024-04-20T17:23:46.084419+0000","last_peered":"2024-04-20T17:23:46.084419+0000","last_clean":"2024-04-20T17:23:46.084419+0000","last_became_active":"2024-04-20T17:23:32.101322+0000","last_became_peered":"2024-04-20T17:23:32.101322+0000","last_unstale":"2024-04-20T17:23:46.084419+0000","last_undegraded":"2024-04-20T17:23:46.084419+0000","last_fullsized":"2024-04-20T17:23:46.084419+0000","mapping_epoch":417,"log_start":"379'6000","ondisk_log_start":"379'6000","created":356,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'15170","last_scrub_stamp":"2024-04-20T17:23:33.728787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:23:33.728787+0000","objects_scrubbed":2554,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:25:02.789437+0000","scrub_duration":228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109248512,"num_objects":1667,"num_object_clones":0,"num_object_copies":3334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16057,"num_write_kb":567168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"426'15967","reported_seq":16208,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-04-20T17:23:46.093830+0000","last_change":"2024-04-20T17:22:09.124841+0000","last_active":"2024-04-20T17:23:46.093830+0000","last_peered":"2024-04-20T17:23:46.093830+0000","last_clean":"2024-04-20T17:23:46.093830+0000","last_became_active":"2024-04-20T17:20:23.037709+0000","last_became_peered":"2024-04-20T17:20:23.037709+0000","last_unstale":"2024-04-20T17:23:46.093830+0000","last_undegraded":"2024-04-20T17:23:46.093830+0000","last_fullsized":"2024-04-20T17:23:46.093830+0000","mapping_epoch":356,"log_start":"379'5900","ondisk_log_start":"379'5900","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'9364","last_scrub_stamp":"2024-04-20T17:22:09.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:22:09.124631+0000","objects_scrubbed":8352,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110034944,"num_objects":1679,"num_object_clones":0,"num_object_copies":3358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15967,"num_write_kb":564672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"426'16494","reported_seq":16736,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-04-20T17:23:47.443377+0000","last_change":"2024-04-20T17:22:08.541264+0000","last_active":"2024-04-20T17:23:47.443377+0000","last_peered":"2024-04-20T17:23:47.443377+0000","last_clean":"2024-04-20T17:23:47.443377+0000","last_became_active":"2024-04-20T17:20:23.037291+0000","last_became_peered":"2024-04-20T17:20:23.037291+0000","last_unstale":"2024-04-20T17:23:47.443377+0000","last_undegraded":"2024-04-20T17:23:47.443377+0000","last_fullsized":"2024-04-20T17:23:47.443377+0000","mapping_epoch":356,"log_start":"379'6400","ondisk_log_start":"379'6400","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'9567","last_scrub_stamp":"2024-04-20T17:22:08.541140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:22:08.541140+0000","objects_scrubbed":8508,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:23:50.980256+0000","scrub_duration":1298,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100532224,"num_objects":1534,"num_object_clones":0,"num_object_copies":3068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16494,"num_write_kb":576896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"425'16238","reported_seq":16520,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-04-20T17:23:43.623800+0000","last_change":"2024-04-20T17:23:18.055677+0000","last_active":"2024-04-20T17:23:43.623800+0000","last_peered":"2024-04-20T17:23:43.623800+0000","last_clean":"2024-04-20T17:23:43.623800+0000","last_became_active":"2024-04-20T17:20:23.036396+0000","last_became_peered":"2024-04-20T17:20:23.036396+0000","last_unstale":"2024-04-20T17:23:43.623800+0000","last_undegraded":"2024-04-20T17:23:43.623800+0000","last_fullsized":"2024-04-20T17:23:43.623800+0000","mapping_epoch":356,"log_start":"379'6200","ondisk_log_start":"379'6200","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'14645","last_scrub_stamp":"2024-04-20T17:23:18.055407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:23:18.055407+0000","objects_scrubbed":3410,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:24:41.943429+0000","scrub_duration":726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117178368,"num_objects":1788,"num_object_clones":0,"num_object_copies":3576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16238,"num_write_kb":576832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"426'16687","reported_seq":16967,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-04-20T17:23:48.008447+0000","last_change":"2024-04-20T17:23:28.128364+0000","last_active":"2024-04-20T17:23:48.008447+0000","last_peered":"2024-04-20T17:23:48.008447+0000","last_clean":"2024-04-20T17:23:48.008447+0000","last_became_active":"2024-04-20T17:20:23.038395+0000","last_became_peered":"2024-04-20T17:20:23.038395+0000","last_unstale":"2024-04-20T17:23:48.008447+0000","last_undegraded":"2024-04-20T17:23:48.008447+0000","last_fullsized":"2024-04-20T17:23:48.008447+0000","mapping_epoch":356,"log_start":"379'6600","ondisk_log_start":"379'6600","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'15310","last_scrub_stamp":"2024-04-20T17:23:28.128169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:23:28.128169+0000","objects_scrubbed":2904,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:24:45.092149+0000","scrub_duration":698,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99024896,"num_objects":1511,"num_object_clones":0,"num_object_copies":3022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16687,"num_write_kb":582336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"426'16727","reported_seq":16970,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-04-20T17:23:46.098804+0000","last_change":"2024-04-20T17:22:10.216053+0000","last_active":"2024-04-20T17:23:46.098804+0000","last_peered":"2024-04-20T17:23:46.098804+0000","last_clean":"2024-04-20T17:23:46.098804+0000","last_became_active":"2024-04-20T17:20:23.036358+0000","last_became_peered":"2024-04-20T17:20:23.036358+0000","last_unstale":"2024-04-20T17:23:46.098804+0000","last_undegraded":"2024-04-20T17:23:46.098804+0000","last_fullsized":"2024-04-20T17:23:46.098804+0000","mapping_epoch":356,"log_start":"379'6700","ondisk_log_start":"379'6700","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'9861","last_scrub_stamp":"2024-04-20T17:22:10.215968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:22:10.215968+0000","objects_scrubbed":8703,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115539968,"num_objects":1763,"num_object_clones":0,"num_object_copies":3526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16727,"num_write_kb":591680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"425'16005","reported_seq":16281,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-04-20T17:23:44.248743+0000","last_change":"2024-04-20T17:23:18.728174+0000","last_active":"2024-04-20T17:23:44.248743+0000","last_peered":"2024-04-20T17:23:44.248743+0000","last_clean":"2024-04-20T17:23:44.248743+0000","last_became_active":"2024-04-20T17:20:23.040736+0000","last_became_peered":"2024-04-20T17:20:23.040736+0000","last_unstale":"2024-04-20T17:23:44.248743+0000","last_undegraded":"2024-04-20T17:23:44.248743+0000","last_fullsized":"2024-04-20T17:23:44.248743+0000","mapping_epoch":356,"log_start":"379'6000","ondisk_log_start":"379'6000","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'14623","last_scrub_stamp":"2024-04-20T17:23:18.728047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:23:18.728047+0000","objects_scrubbed":3163,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:24:41.684816+0000","scrub_duration":561,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115277824,"num_objects":1759,"num_object_clones":0,"num_object_copies":3518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16005,"num_write_kb":568448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"426'16253","reported_seq":16541,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-04-20T17:23:46.137132+0000","last_change":"2024-04-20T17:23:32.136397+0000","last_active":"2024-04-20T17:23:46.137132+0000","last_peered":"2024-04-20T17:23:46.137132+0000","last_clean":"2024-04-20T17:23:46.137132+0000","last_became_active":"2024-04-20T17:23:32.101496+0000","last_became_peered":"2024-04-20T17:23:32.101496+0000","last_unstale":"2024-04-20T17:23:46.137132+0000","last_undegraded":"2024-04-20T17:23:46.137132+0000","last_fullsized":"2024-04-20T17:23:46.137132+0000","mapping_epoch":417,"log_start":"379'6200","ondisk_log_start":"379'6200","created":356,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'12465","last_scrub_stamp":"2024-04-20T17:22:48.748392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:22:48.748392+0000","objects_scrubbed":5431,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:24:08.748763+0000","scrub_duration":1137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104792064,"num_objects":1599,"num_object_clones":0,"num_object_copies":3198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16253,"num_write_kb":571264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"426'16293","reported_seq":16573,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-04-20T17:23:47.128482+0000","last_change":"2024-04-20T17:23:25.335537+0000","last_active":"2024-04-20T17:23:47.128482+0000","last_peered":"2024-04-20T17:23:47.128482+0000","last_clean":"2024-04-20T17:23:47.128482+0000","last_became_active":"2024-04-20T17:20:23.038263+0000","last_became_peered":"2024-04-20T17:20:23.038263+0000","last_unstale":"2024-04-20T17:23:47.128482+0000","last_undegraded":"2024-04-20T17:23:47.128482+0000","last_fullsized":"2024-04-20T17:23:47.128482+0000","mapping_epoch":356,"log_start":"379'6200","ondisk_log_start":"379'6200","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'14897","last_scrub_stamp":"2024-04-20T17:23:25.335360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:23:25.335360+0000","objects_scrubbed":2944,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:24:53.977629+0000","scrub_duration":723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99811328,"num_objects":1523,"num_object_clones":0,"num_object_copies":3046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16293,"num_write_kb":570112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"426'16065","reported_seq":16345,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-04-20T17:23:45.630654+0000","last_change":"2024-04-20T17:23:24.940515+0000","last_active":"2024-04-20T17:23:45.630654+0000","last_peered":"2024-04-20T17:23:45.630654+0000","last_clean":"2024-04-20T17:23:45.630654+0000","last_became_active":"2024-04-20T17:20:23.039051+0000","last_became_peered":"2024-04-20T17:20:23.039051+0000","last_unstale":"2024-04-20T17:23:45.630654+0000","last_undegraded":"2024-04-20T17:23:45.630654+0000","last_fullsized":"2024-04-20T17:23:45.630654+0000","mapping_epoch":356,"log_start":"379'6000","ondisk_log_start":"379'6000","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'14766","last_scrub_stamp":"2024-04-20T17:23:24.940393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:23:24.940393+0000","objects_scrubbed":2947,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:24:35.854075+0000","scrub_duration":491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107413504,"num_objects":1639,"num_object_clones":0,"num_object_copies":3278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16065,"num_write_kb":566528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"425'16346","reported_seq":16627,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-04-20T17:23:44.253544+0000","last_change":"2024-04-20T17:22:51.294771+0000","last_active":"2024-04-20T17:23:44.253544+0000","last_peered":"2024-04-20T17:23:44.253544+0000","last_clean":"2024-04-20T17:23:44.253544+0000","last_became_active":"2024-04-20T17:20:23.036876+0000","last_became_peered":"2024-04-20T17:20:23.036876+0000","last_unstale":"2024-04-20T17:23:44.253544+0000","last_undegraded":"2024-04-20T17:23:44.253544+0000","last_fullsized":"2024-04-20T17:23:44.253544+0000","mapping_epoch":356,"log_start":"379'6300","ondisk_log_start":"379'6300","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'12840","last_scrub_stamp":"2024-04-20T17:22:51.294500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:22:51.294500+0000","objects_scrubbed":5339,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:24:13.942709+0000","scrub_duration":1103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117833728,"num_objects":1798,"num_object_clones":0,"num_object_copies":3596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16346,"num_write_kb":580608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"426'16331","reported_seq":16569,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-04-20T17:23:48.042656+0000","last_change":"2024-04-20T17:22:13.270649+0000","last_active":"2024-04-20T17:23:48.042656+0000","last_peered":"2024-04-20T17:23:48.042656+0000","last_clean":"2024-04-20T17:23:48.042656+0000","last_became_active":"2024-04-20T17:20:23.171030+0000","last_became_peered":"2024-04-20T17:20:23.171030+0000","last_unstale":"2024-04-20T17:23:48.042656+0000","last_undegraded":"2024-04-20T17:23:48.042656+0000","last_fullsized":"2024-04-20T17:23:48.042656+0000","mapping_epoch":356,"log_start":"379'6300","ondisk_log_start":"379'6300","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"392'9773","last_scrub_stamp":"2024-04-20T17:22:13.270414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:22:13.270414+0000","objects_scrubbed":8172,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101646336,"num_objects":1551,"num_object_clones":0,"num_object_copies":3102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16331,"num_write_kb":572224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"426'16070","reported_seq":16310,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-04-20T17:23:46.106189+0000","last_change":"2024-04-20T17:22:10.697405+0000","last_active":"2024-04-20T17:23:46.106189+0000","last_peered":"2024-04-20T17:23:46.106189+0000","last_clean":"2024-04-20T17:23:46.106189+0000","last_became_active":"2024-04-20T17:20:23.171078+0000","last_became_peered":"2024-04-20T17:20:23.171078+0000","last_unstale":"2024-04-20T17:23:46.106189+0000","last_undegraded":"2024-04-20T17:23:46.106189+0000","last_fullsized":"2024-04-20T17:23:46.106189+0000","mapping_epoch":356,"log_start":"379'6000","ondisk_log_start":"379'6000","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'9457","last_scrub_stamp":"2024-04-20T17:22:10.697220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:20:22.020114+0000","last_clean_scrub_stamp":"2024-04-20T17:22:10.697220+0000","objects_scrubbed":8285,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1613,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105906176,"num_objects":1616,"num_object_clones":0,"num_object_copies":3232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16070,"num_write_kb":565952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":354,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-04-20T17:23:44.289315+0000","last_change":"2024-04-20T17:23:24.109535+0000","last_active":"2024-04-20T17:23:44.289315+0000","last_peered":"2024-04-20T17:23:44.289315+0000","last_clean":"2024-04-20T17:23:44.289315+0000","last_became_active":"2024-04-20T17:23:24.109202+0000","last_became_peered":"2024-04-20T17:23:24.109202+0000","last_unstale":"2024-04-20T17:23:44.289315+0000","last_undegraded":"2024-04-20T17:23:44.289315+0000","last_fullsized":"2024-04-20T17:23:44.289315+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:22:21.398925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:22:21.398925+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:23:47.770634+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1704067096,"num_objects":26003,"num_object_clones":0,"num_object_copies":52006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26003,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":260423,"num_write_kb":9165569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9641,"num_bytes_recovered":631767064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3560775680,"data_stored":3560767536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160836,"ondisk_log_size":160836,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991449,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":988220,"kb_used_data":696052,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":92756932,"statfs":{"total":95995035648,"available":94983098368,"internally_reserved":0,"allocated":712757248,"data_stored":712405079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":299170244},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253087,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":216492,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93528660,"statfs":{"total":95995035648,"available":95773347840,"internally_reserved":0,"allocated":688128,"data_stored":345515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":220985792},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612837,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":428536,"kb_used_data":222444,"kb_used_omap":1,"kb_used_meta":206078,"kb_avail":93316616,"statfs":{"total":95995035648,"available":95556214784,"internally_reserved":0,"allocated":227782656,"data_stored":227434059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":211024320},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809067,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":423084,"kb_used_data":261408,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93322068,"statfs":{"total":95995035648,"available":95561797632,"internally_reserved":0,"allocated":267681792,"data_stored":267339199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":398,"seq":1709396983824,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":263296,"kb_used_data":206968,"kb_used_omap":6,"kb_used_meta":56313,"kb_avail":93481856,"statfs":{"total":95995035648,"available":95725420544,"internally_reserved":0,"allocated":211935232,"data_stored":211579011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6767,"internal_metadata":57664913},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430451,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":230708,"kb_used_data":100972,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93514444,"statfs":{"total":95995035648,"available":95758790656,"internally_reserved":0,"allocated":103395328,"data_stored":103046731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575117,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":483380,"kb_used_data":307820,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93261772,"statfs":{"total":95995035648,"available":95500054528,"internally_reserved":0,"allocated":315207680,"data_stored":314859083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738641,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":732012,"kb_used_data":457888,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":93013140,"statfs":{"total":95995035648,"available":95245455360,"internally_reserved":0,"allocated":468877312,"data_stored":468534719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":280689093},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393931,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":49080,"kb_used_data":684,"kb_used_omap":202,"kb_used_meta":48181,"kb_avail":93696072,"statfs":{"total":95995035648,"available":95944777728,"internally_reserved":0,"allocated":700416,"data_stored":351819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":49337815},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182658,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":728504,"kb_used_data":548780,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93016648,"statfs":{"total":95995035648,"available":95249047552,"internally_reserved":0,"allocated":561950720,"data_stored":561602123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":184023495},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986429,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":538424,"kb_used_data":430892,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93206728,"statfs":{"total":95995035648,"available":95443689472,"internally_reserved":0,"allocated":441233408,"data_stored":440884811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790200,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":679544,"kb_used_data":491504,"kb_used_omap":1,"kb_used_meta":188030,"kb_avail":93065608,"statfs":{"total":95995035648,"available":95299182592,"internally_reserved":0,"allocated":503300096,"data_stored":502947427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":192543173},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":465502208,"data_stored":465502208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":305856512,"data_stored":305856512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":545456128,"data_stored":545456128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":432930816,"data_stored":432930816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":328208384,"data_stored":328204312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":100466688,"data_stored":100466688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":205328384,"data_stored":205324312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":265617408,"data_stored":265617408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":220594176,"data_stored":220594176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":690814976,"data_stored":690814976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:23:49.510 INFO:tasks.thrashosds.thrasher:Setting 3.c to [3, 11] 2024-04-20T17:23:49.510 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.c', '3', '11'] 2024-04-20T17:23:49.510 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pg-upmap 3.c 3 11 2024-04-20T17:23:49.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:49 smithi171 ceph-mon[44189]: pgmap v1147: 17 pgs: 17 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.35k op/s 2024-04-20T17:23:49.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:49 smithi171 ceph-mon[44189]: from='client.26039 v1:172.21.15.5:0/1813519813' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:23:49.585 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:23:49.664 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:49.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:49 smithi106 ceph-mon[44423]: pgmap v1147: 17 pgs: 17 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.35k op/s 2024-04-20T17:23:49.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:49 smithi106 ceph-mon[44423]: from='client.26039 v1:172.21.15.5:0/1813519813' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:23:49.919 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:23:49.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:49.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:23:50.028 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:23:50.057 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:23:50.058 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:50.082 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:23:50.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:50.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:23:50.188 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:23:50.221 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:50.293 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:23:50.474 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:23:50.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:50.477 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:23:50.578 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:23:50.619 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:50.639 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:23:50.640 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:23:50.640 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-04-20T17:23:50.640 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-04-20T17:23:50.640 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:23:50.644 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:23:50.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:50.647 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:23:50.748 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:23:50.779 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:50.802 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:23:50.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:50.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:23:50.906 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:23:50.939 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:50.964 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:23:50.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:50.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:23:51.069 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:23:51.124 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:51.322 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:23:51.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:51 smithi171 ceph-mon[44189]: pgmap v1148: 17 pgs: 17 active+clean; 1.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:23:51.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:51 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/701172308' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:23:51.357 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:51 smithi005 ceph-mon[39562]: pgmap v1148: 17 pgs: 17 active+clean; 1.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:23:51.357 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:51 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/701172308' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:23:51.480 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-20T17:23:51.502 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:23:51.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:51.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:23:51.608 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:23:51.638 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:51 smithi106 ceph-mon[44423]: pgmap v1148: 17 pgs: 17 active+clean; 1.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:23:51.638 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:51 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/701172308' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:23:51.640 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:51.665 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:23:51.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:51.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:23:51.773 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:23:51.813 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:51.841 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:23:51.914 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:23:51.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:51.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:23:52.025 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:23:52.061 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:52.085 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:23:52.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:52.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:23:52.192 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:23:52.239 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:52.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:52 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:23:52.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:52 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/701172308' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:23:52.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:52 smithi106 ceph-mon[44423]: osdmap e427: 12 total, 12 up, 12 in 2024-04-20T17:23:52.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:52 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:23:52.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:52 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/701172308' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:23:52.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:52 smithi171 ceph-mon[44189]: osdmap e427: 12 total, 12 up, 12 in 2024-04-20T17:23:52.675 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:52 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:23:52.675 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:52 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/701172308' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:23:52.675 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:52 smithi005 ceph-mon[39562]: osdmap e427: 12 total, 12 up, 12 in 2024-04-20T17:23:53.573 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:53 smithi005 ceph-mon[39562]: pgmap v1150: 17 pgs: 17 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-04-20T17:23:53.573 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:53 smithi005 ceph-mon[39562]: 3.0 scrub starts 2024-04-20T17:23:53.573 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:53 smithi005 ceph-mon[39562]: 3.0 scrub ok 2024-04-20T17:23:53.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:53 smithi171 ceph-mon[44189]: pgmap v1150: 17 pgs: 17 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-04-20T17:23:53.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:53 smithi171 ceph-mon[44189]: 3.0 scrub starts 2024-04-20T17:23:53.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:53 smithi171 ceph-mon[44189]: 3.0 scrub ok 2024-04-20T17:23:53.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:53 smithi106 ceph-mon[44423]: pgmap v1150: 17 pgs: 17 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-04-20T17:23:53.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:53 smithi106 ceph-mon[44423]: 3.0 scrub starts 2024-04-20T17:23:53.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:53 smithi106 ceph-mon[44423]: 3.0 scrub ok 2024-04-20T17:23:53.810 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:23:53.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:53.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:23:53.921 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:23:53.923 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:23:53.923 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:23:53.923 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:23:53.923 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:23:53.923 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.923 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:122903 3.f 3:f1e79158:::benchmark_data_smithi171_110765_object122902:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e384)", 2024-04-20T17:23:53.923 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:45.056333+0000", 2024-04-20T17:23:53.924 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 128.86058633499999, 2024-04-20T17:23:53.924 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.122128339, 2024-04-20T17:23:53.924 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:53.924 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:53.924 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:53.924 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:53.924 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:53.924 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:53.924 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 122903 2024-04-20T17:23:53.924 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.924 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:53.924 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.924 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:53.924 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.056333+0000", 2024-04-20T17:23:53.925 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.925 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.925 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.925 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:53.925 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.056333+0000", 2024-04-20T17:23:53.925 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.925 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.925 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.925 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:53.925 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.056335+0000", 2024-04-20T17:23:53.925 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9489999999999999e-06 2024-04-20T17:23:53.925 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.925 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.925 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:53.925 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.056364+0000", 2024-04-20T17:23:53.926 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8530999999999998e-05 2024-04-20T17:23:53.926 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.926 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.926 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:53.926 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.056373+0000", 2024-04-20T17:23:53.926 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2189999999999993e-06 2024-04-20T17:23:53.926 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.926 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.926 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:53.926 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.056380+0000", 2024-04-20T17:23:53.926 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4389999999999997e-06 2024-04-20T17:23:53.926 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.926 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.927 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:53.927 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.088954+0000", 2024-04-20T17:23:53.927 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032574918000000001 2024-04-20T17:23:53.927 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.927 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.927 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:53.927 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.089066+0000", 2024-04-20T17:23:53.927 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000111227 2024-04-20T17:23:53.927 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.927 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.927 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:23:53.927 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.089201+0000", 2024-04-20T17:23:53.927 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000135728 2024-04-20T17:23:53.927 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.928 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.928 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:53.928 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.178342+0000", 2024-04-20T17:23:53.928 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.089140237999999997 2024-04-20T17:23:53.928 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.928 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.928 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:53.928 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.178368+0000", 2024-04-20T17:23:53.928 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6364e-05 2024-04-20T17:23:53.928 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.928 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.928 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:53.929 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.178405+0000", 2024-04-20T17:23:53.929 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7054999999999997e-05 2024-04-20T17:23:53.929 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.929 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.929 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:53.929 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.178462+0000", 2024-04-20T17:23:53.929 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6671e-05 2024-04-20T17:23:53.929 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:53.929 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:53.929 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:53.929 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.929 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.929 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:122913 3.f 3:fd8eb2c6:::benchmark_data_smithi171_110765_object122912:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e384)", 2024-04-20T17:23:53.929 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:45.066520+0000", 2024-04-20T17:23:53.930 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 128.85039983300001, 2024-04-20T17:23:53.930 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.112070714, 2024-04-20T17:23:53.930 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:53.930 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:53.930 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:53.930 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:53.930 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:53.930 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:53.930 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 122913 2024-04-20T17:23:53.930 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.930 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:53.930 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.930 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:53.931 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.066520+0000", 2024-04-20T17:23:53.931 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.931 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.931 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.931 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:53.931 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.066520+0000", 2024-04-20T17:23:53.931 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.931 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.931 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.931 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:53.931 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.066522+0000", 2024-04-20T17:23:53.931 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8169999999999999e-06 2024-04-20T17:23:53.931 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.931 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.932 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:53.932 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.066539+0000", 2024-04-20T17:23:53.932 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7280000000000001e-05 2024-04-20T17:23:53.932 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.932 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.932 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:53.932 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.066550+0000", 2024-04-20T17:23:53.932 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0885e-05 2024-04-20T17:23:53.932 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.932 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.932 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:53.932 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.066557+0000", 2024-04-20T17:23:53.932 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6540000000000002e-06 2024-04-20T17:23:53.932 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.932 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.933 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:53.933 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.093712+0000", 2024-04-20T17:23:53.933 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.027155371000000001 2024-04-20T17:23:53.933 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.933 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.933 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:53.933 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.093839+0000", 2024-04-20T17:23:53.933 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012661500000000001 2024-04-20T17:23:53.933 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.933 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.933 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:23:53.934 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.093976+0000", 2024-04-20T17:23:53.934 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000137768 2024-04-20T17:23:53.934 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.934 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.934 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:53.934 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.178477+0000", 2024-04-20T17:23:53.934 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.084500637000000003 2024-04-20T17:23:53.934 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.934 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.934 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:53.934 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.178513+0000", 2024-04-20T17:23:53.934 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5651000000000001e-05 2024-04-20T17:23:53.934 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.934 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.935 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:53.935 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.178554+0000", 2024-04-20T17:23:53.935 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1165000000000003e-05 2024-04-20T17:23:53.935 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.935 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.935 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:53.935 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.178591+0000", 2024-04-20T17:23:53.935 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6871e-05 2024-04-20T17:23:53.935 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:53.935 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:53.935 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:53.935 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.935 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.935 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:125317 3.f 3:f831b376:::benchmark_data_smithi171_110765_object125316:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-04-20T17:23:53.935 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:47.140133+0000", 2024-04-20T17:23:53.936 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 126.77678691, 2024-04-20T17:23:53.936 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.114916223, 2024-04-20T17:23:53.936 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:53.936 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:53.936 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:53.936 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:53.936 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:53.936 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:53.936 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 125317 2024-04-20T17:23:53.936 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.936 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:53.936 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.936 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:53.936 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.140133+0000", 2024-04-20T17:23:53.936 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.936 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.937 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.937 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:53.937 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.140133+0000", 2024-04-20T17:23:53.937 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.937 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.937 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.937 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:53.937 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.140135+0000", 2024-04-20T17:23:53.937 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.249e-06 2024-04-20T17:23:53.937 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.937 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.937 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:53.937 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.140159+0000", 2024-04-20T17:23:53.937 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3754e-05 2024-04-20T17:23:53.937 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.938 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.938 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:53.938 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.140172+0000", 2024-04-20T17:23:53.938 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2694000000000001e-05 2024-04-20T17:23:53.938 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.938 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.938 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:53.938 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.140179+0000", 2024-04-20T17:23:53.938 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6930000000000007e-06 2024-04-20T17:23:53.938 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.938 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.938 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:53.938 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.169519+0000", 2024-04-20T17:23:53.938 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.029340226000000001 2024-04-20T17:23:53.939 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.939 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.939 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:53.939 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.169636+0000", 2024-04-20T17:23:53.939 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000116581 2024-04-20T17:23:53.939 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.939 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.939 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:23:53.939 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.169734+0000", 2024-04-20T17:23:53.939 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7849000000000005e-05 2024-04-20T17:23:53.939 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.939 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.939 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:53.939 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.178181+0000", 2024-04-20T17:23:53.939 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0084470499999999994 2024-04-20T17:23:53.940 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.940 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.940 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:53.940 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.254967+0000", 2024-04-20T17:23:53.940 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.076786222000000001 2024-04-20T17:23:53.940 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.940 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.940 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:53.940 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.255005+0000", 2024-04-20T17:23:53.940 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8105000000000002e-05 2024-04-20T17:23:53.940 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.940 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.940 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:53.940 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.255049+0000", 2024-04-20T17:23:53.940 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3800000000000001e-05 2024-04-20T17:23:53.941 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:53.941 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:53.941 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:53.941 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.941 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.941 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:125325 3.f 3:f3424bd2:::benchmark_data_smithi171_110765_object125324:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-04-20T17:23:53.941 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:47.144790+0000", 2024-04-20T17:23:53.941 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 126.772129564, 2024-04-20T17:23:53.941 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.114843684, 2024-04-20T17:23:53.941 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:53.941 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:53.941 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:53.941 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:53.941 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:53.941 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:53.941 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 125325 2024-04-20T17:23:53.942 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.942 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:53.942 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.942 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:53.942 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.144790+0000", 2024-04-20T17:23:53.942 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.942 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.942 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.942 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:53.942 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.144790+0000", 2024-04-20T17:23:53.942 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.942 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.942 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.943 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:53.943 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.144792+0000", 2024-04-20T17:23:53.943 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4890000000000001e-06 2024-04-20T17:23:53.943 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.943 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.943 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:53.943 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.144812+0000", 2024-04-20T17:23:53.943 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0718999999999999e-05 2024-04-20T17:23:53.943 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.943 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.943 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:53.943 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.144821+0000", 2024-04-20T17:23:53.943 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1669999999999999e-06 2024-04-20T17:23:53.943 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.944 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.944 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:53.944 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.144828+0000", 2024-04-20T17:23:53.944 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.576e-06 2024-04-20T17:23:53.944 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.944 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.944 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:53.944 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.173481+0000", 2024-04-20T17:23:53.944 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028652777000000001 2024-04-20T17:23:53.944 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.944 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.944 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:53.944 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.173568+0000", 2024-04-20T17:23:53.944 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6958000000000003e-05 2024-04-20T17:23:53.945 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.945 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.945 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:23:53.945 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.173679+0000", 2024-04-20T17:23:53.945 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000110905 2024-04-20T17:23:53.945 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.945 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.945 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:53.945 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.178409+0000", 2024-04-20T17:23:53.945 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0047305000000000003 2024-04-20T17:23:53.945 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.945 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.945 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:53.945 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.259555+0000", 2024-04-20T17:23:53.945 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.081145834999999999 2024-04-20T17:23:53.946 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.946 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.946 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:53.946 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.259591+0000", 2024-04-20T17:23:53.946 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5908999999999998e-05 2024-04-20T17:23:53.946 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.946 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.946 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:53.946 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.259634+0000", 2024-04-20T17:23:53.946 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2849000000000003e-05 2024-04-20T17:23:53.946 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:53.946 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:53.946 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:53.946 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.947 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.947 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:125888 3.f 3:fdac0d8e:::benchmark_data_smithi171_110765_object125887:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-04-20T17:23:53.947 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:47.621625+0000", 2024-04-20T17:23:53.947 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 126.295294759, 2024-04-20T17:23:53.947 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.120494553, 2024-04-20T17:23:53.947 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:53.947 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:53.947 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:53.947 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:53.947 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:53.947 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:53.947 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 125888 2024-04-20T17:23:53.947 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.947 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:53.947 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.948 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:53.948 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.621625+0000", 2024-04-20T17:23:53.948 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.948 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.948 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.948 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:53.948 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.621625+0000", 2024-04-20T17:23:53.948 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.948 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.948 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.948 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:53.948 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.621627+0000", 2024-04-20T17:23:53.948 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.663e-06 2024-04-20T17:23:53.948 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.948 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.949 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:53.949 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.621654+0000", 2024-04-20T17:23:53.949 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7107e-05 2024-04-20T17:23:53.949 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.949 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.949 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:53.949 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.621664+0000", 2024-04-20T17:23:53.949 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0658e-05 2024-04-20T17:23:53.949 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.949 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.949 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:53.949 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.621675+0000", 2024-04-20T17:23:53.949 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0447e-05 2024-04-20T17:23:53.949 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.949 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.950 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:53.950 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.721943+0000", 2024-04-20T17:23:53.950 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.100268565 2024-04-20T17:23:53.950 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.950 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.950 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:53.950 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.722065+0000", 2024-04-20T17:23:53.950 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000121941 2024-04-20T17:23:53.950 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.950 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.950 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:23:53.950 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.722151+0000", 2024-04-20T17:23:53.950 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5230999999999998e-05 2024-04-20T17:23:53.950 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.951 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.951 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:53.951 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.729406+0000", 2024-04-20T17:23:53.951 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0072549449999999996 2024-04-20T17:23:53.951 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.951 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.951 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:53.951 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.742037+0000", 2024-04-20T17:23:53.951 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.012631008000000001 2024-04-20T17:23:53.951 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.951 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.951 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:53.951 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.742072+0000", 2024-04-20T17:23:53.951 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5394e-05 2024-04-20T17:23:53.952 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.952 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.952 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:53.952 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.742120+0000", 2024-04-20T17:23:53.952 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7593999999999999e-05 2024-04-20T17:23:53.952 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:53.952 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:53.952 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:53.952 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.952 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.952 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:125895 3.f 3:fcf33e0f:::benchmark_data_smithi171_110765_object125894:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-04-20T17:23:53.952 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:47.626801+0000", 2024-04-20T17:23:53.952 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 126.290118643, 2024-04-20T17:23:53.952 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.11956959, 2024-04-20T17:23:53.952 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:53.953 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:53.953 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:53.953 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:53.953 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:53.953 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:53.953 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 125895 2024-04-20T17:23:53.953 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.953 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:53.953 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.953 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:53.953 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.626801+0000", 2024-04-20T17:23:53.953 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.953 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.953 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.953 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:53.954 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.626801+0000", 2024-04-20T17:23:53.954 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.954 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.954 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.954 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:53.954 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.626803+0000", 2024-04-20T17:23:53.954 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3880000000000001e-06 2024-04-20T17:23:53.954 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.954 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.954 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:53.954 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.626823+0000", 2024-04-20T17:23:53.954 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0369000000000001e-05 2024-04-20T17:23:53.954 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.954 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.955 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:53.955 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.626833+0000", 2024-04-20T17:23:53.955 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7699999999999996e-06 2024-04-20T17:23:53.955 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.955 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.955 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:53.955 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.626840+0000", 2024-04-20T17:23:53.955 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5089999999999996e-06 2024-04-20T17:23:53.955 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.955 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.955 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:53.955 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.732496+0000", 2024-04-20T17:23:53.955 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.105655913 2024-04-20T17:23:53.955 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.955 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.956 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:53.956 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.732647+0000", 2024-04-20T17:23:53.956 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015076 2024-04-20T17:23:53.956 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.956 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.956 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:23:53.956 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.732751+0000", 2024-04-20T17:23:53.956 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000104319 2024-04-20T17:23:53.956 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.956 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.956 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:53.956 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.746213+0000", 2024-04-20T17:23:53.956 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013462081000000001 2024-04-20T17:23:53.956 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.956 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.957 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:53.957 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.746260+0000", 2024-04-20T17:23:53.957 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6394000000000003e-05 2024-04-20T17:23:53.957 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.957 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.957 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:53.957 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.746313+0000", 2024-04-20T17:23:53.957 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3733999999999997e-05 2024-04-20T17:23:53.957 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.957 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.957 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:53.957 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.746371+0000", 2024-04-20T17:23:53.957 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7352999999999998e-05 2024-04-20T17:23:53.957 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:53.957 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:53.958 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:53.958 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.958 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.958 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:125904 3.f 3:f05d6ad0:::benchmark_data_smithi171_110765_object125903:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-04-20T17:23:53.958 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:47.635351+0000", 2024-04-20T17:23:53.958 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 126.281568894, 2024-04-20T17:23:53.958 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.115516498, 2024-04-20T17:23:53.958 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:53.958 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:53.958 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:53.958 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:53.958 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:53.958 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:53.958 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 125904 2024-04-20T17:23:53.959 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.959 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:53.959 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.959 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:53.959 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.635351+0000", 2024-04-20T17:23:53.959 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.959 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.959 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.959 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:53.959 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.635351+0000", 2024-04-20T17:23:53.959 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.959 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.959 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.959 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:53.959 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.635353+0000", 2024-04-20T17:23:53.960 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.75e-06 2024-04-20T17:23:53.960 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.960 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.960 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:53.960 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.635374+0000", 2024-04-20T17:23:53.960 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1145000000000002e-05 2024-04-20T17:23:53.960 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.960 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.960 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:53.960 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.635383+0000", 2024-04-20T17:23:53.960 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.4439999999999997e-06 2024-04-20T17:23:53.960 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.960 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.960 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:53.960 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.635389+0000", 2024-04-20T17:23:53.961 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.28e-06 2024-04-20T17:23:53.961 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.961 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.961 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:53.961 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.737510+0000", 2024-04-20T17:23:53.961 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.102120723 2024-04-20T17:23:53.961 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.961 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.961 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:53.961 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.737638+0000", 2024-04-20T17:23:53.961 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000127491 2024-04-20T17:23:53.961 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.961 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.961 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:23:53.961 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.737790+0000", 2024-04-20T17:23:53.962 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000152051 2024-04-20T17:23:53.962 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.962 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.962 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:53.962 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.746429+0000", 2024-04-20T17:23:53.962 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0086396300000000006 2024-04-20T17:23:53.962 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.962 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.962 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:53.962 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.750746+0000", 2024-04-20T17:23:53.963 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00431636 2024-04-20T17:23:53.963 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.963 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.963 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:53.963 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.750813+0000", 2024-04-20T17:23:53.963 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7374999999999998e-05 2024-04-20T17:23:53.963 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.963 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.963 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:53.963 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.750867+0000", 2024-04-20T17:23:53.963 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4249000000000002e-05 2024-04-20T17:23:53.963 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:53.963 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:53.963 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:53.964 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.964 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.964 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:125905 3.f 3:fffcbe7e:::benchmark_data_smithi171_110765_object125904:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-04-20T17:23:53.964 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:47.635413+0000", 2024-04-20T17:23:53.964 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 126.28150653900001, 2024-04-20T17:23:53.964 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.120174664, 2024-04-20T17:23:53.964 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:53.964 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:53.964 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:53.964 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:53.964 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:53.964 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:53.964 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 125905 2024-04-20T17:23:53.964 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.965 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:53.965 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.965 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:53.965 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.635413+0000", 2024-04-20T17:23:53.965 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.965 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.965 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.965 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:53.965 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.635413+0000", 2024-04-20T17:23:53.965 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.965 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.965 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.965 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:53.965 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.635414+0000", 2024-04-20T17:23:53.966 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5300000000000004e-07 2024-04-20T17:23:53.966 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.966 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.966 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:53.966 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.635436+0000", 2024-04-20T17:23:53.966 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2535999999999999e-05 2024-04-20T17:23:53.966 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.966 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.966 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:53.966 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.635441+0000", 2024-04-20T17:23:53.966 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5619999999999997e-06 2024-04-20T17:23:53.966 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.966 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.966 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:53.967 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.635445+0000", 2024-04-20T17:23:53.967 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7129999999999999e-06 2024-04-20T17:23:53.967 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.967 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.967 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:53.967 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.742154+0000", 2024-04-20T17:23:53.967 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.10670937 2024-04-20T17:23:53.967 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.967 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.967 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:53.967 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.742286+0000", 2024-04-20T17:23:53.967 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013218099999999999 2024-04-20T17:23:53.967 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.967 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.967 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:23:53.968 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.742436+0000", 2024-04-20T17:23:53.968 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000150118 2024-04-20T17:23:53.968 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.968 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.968 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:53.968 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.755452+0000", 2024-04-20T17:23:53.968 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013015633 2024-04-20T17:23:53.968 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.968 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.968 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:53.968 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.755494+0000", 2024-04-20T17:23:53.968 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1928999999999998e-05 2024-04-20T17:23:53.968 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.968 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.968 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:53.968 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.755534+0000", 2024-04-20T17:23:53.969 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9906e-05 2024-04-20T17:23:53.969 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.969 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.969 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:53.969 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.755588+0000", 2024-04-20T17:23:53.969 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4063000000000001e-05 2024-04-20T17:23:53.969 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:53.969 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:53.969 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:53.969 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.969 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.969 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:127050 3.f 3:f82f11a4:::benchmark_data_smithi171_110765_object127049:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-04-20T17:23:53.969 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:48.625659+0000", 2024-04-20T17:23:53.969 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 125.291260884, 2024-04-20T17:23:53.969 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.111995974, 2024-04-20T17:23:53.970 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:53.970 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:53.970 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:53.970 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:53.970 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:53.970 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:53.970 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 127050 2024-04-20T17:23:53.970 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.970 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:53.970 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.970 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:53.970 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.625659+0000", 2024-04-20T17:23:53.970 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.970 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.971 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.971 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:53.971 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.625659+0000", 2024-04-20T17:23:53.971 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.971 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.971 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.971 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:53.971 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.625661+0000", 2024-04-20T17:23:53.971 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3889999999999999e-06 2024-04-20T17:23:53.971 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.971 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.971 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:53.971 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.625687+0000", 2024-04-20T17:23:53.971 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5440999999999999e-05 2024-04-20T17:23:53.971 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.972 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.972 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:53.972 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.625701+0000", 2024-04-20T17:23:53.972 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3918e-05 2024-04-20T17:23:53.972 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.972 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.972 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:53.972 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.625710+0000", 2024-04-20T17:23:53.972 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0499999999999997e-06 2024-04-20T17:23:53.972 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.972 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.972 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:53.972 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.717453+0000", 2024-04-20T17:23:53.972 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.091743016999999996 2024-04-20T17:23:53.972 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.972 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.973 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:53.973 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.717540+0000", 2024-04-20T17:23:53.973 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.7035999999999998e-05 2024-04-20T17:23:53.973 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.973 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.973 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:23:53.973 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.717637+0000", 2024-04-20T17:23:53.973 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7759999999999999e-05 2024-04-20T17:23:53.973 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.973 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.973 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:53.973 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.722730+0000", 2024-04-20T17:23:53.973 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0050926249999999999 2024-04-20T17:23:53.973 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.973 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.973 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:53.973 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.737565+0000", 2024-04-20T17:23:53.974 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014835193 2024-04-20T17:23:53.974 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.974 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.974 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:53.974 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.737602+0000", 2024-04-20T17:23:53.974 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6949999999999997e-05 2024-04-20T17:23:53.974 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.974 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.974 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:53.974 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.737655+0000", 2024-04-20T17:23:53.974 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2595e-05 2024-04-20T17:23:53.974 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:53.974 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:53.974 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:53.974 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.975 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.975 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:127064 3.f 3:f4e625c7:::benchmark_data_smithi171_110765_object127063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-04-20T17:23:53.975 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:48.644193+0000", 2024-04-20T17:23:53.975 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 125.272726717, 2024-04-20T17:23:53.975 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.11587829500000001, 2024-04-20T17:23:53.975 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:53.975 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:53.975 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:53.975 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:53.975 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:53.975 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:53.975 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 127064 2024-04-20T17:23:53.975 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.975 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:53.975 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.976 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:53.976 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.644193+0000", 2024-04-20T17:23:53.976 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.976 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.976 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.976 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:53.976 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.644193+0000", 2024-04-20T17:23:53.976 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.976 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.976 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.976 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:53.976 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.644195+0000", 2024-04-20T17:23:53.976 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.73e-06 2024-04-20T17:23:53.976 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.976 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.977 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:53.977 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.644219+0000", 2024-04-20T17:23:53.977 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4630999999999998e-05 2024-04-20T17:23:53.977 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.977 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.977 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:53.977 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.644229+0000", 2024-04-20T17:23:53.977 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.4210000000000001e-06 2024-04-20T17:23:53.977 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.977 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.977 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:53.977 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.644234+0000", 2024-04-20T17:23:53.977 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1730000000000003e-06 2024-04-20T17:23:53.977 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.977 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.978 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:53.978 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.737687+0000", 2024-04-20T17:23:53.978 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.093452885999999999 2024-04-20T17:23:53.978 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.978 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.978 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:53.978 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.737839+0000", 2024-04-20T17:23:53.978 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015240499999999999 2024-04-20T17:23:53.978 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.978 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.978 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:23:53.978 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.737958+0000", 2024-04-20T17:23:53.978 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011915 2024-04-20T17:23:53.979 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.979 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.979 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:53.979 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.759902+0000", 2024-04-20T17:23:53.979 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021943451999999999 2024-04-20T17:23:53.979 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.979 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.979 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:53.979 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.759956+0000", 2024-04-20T17:23:53.979 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3739000000000002e-05 2024-04-20T17:23:53.979 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.979 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.979 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:53.979 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.760015+0000", 2024-04-20T17:23:53.979 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9653e-05 2024-04-20T17:23:53.980 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.980 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.980 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:53.980 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.760071+0000", 2024-04-20T17:23:53.980 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6054999999999998e-05 2024-04-20T17:23:53.980 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:53.980 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:53.980 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:53.980 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.980 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.980 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:127070 3.f 3:fdee2685:::benchmark_data_smithi171_110765_object127069:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-04-20T17:23:53.980 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:48.650189+0000", 2024-04-20T17:23:53.980 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 125.266730911, 2024-04-20T17:23:53.980 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.110068943, 2024-04-20T17:23:53.981 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:53.981 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:53.981 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:53.981 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:53.981 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:53.981 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:53.981 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 127070 2024-04-20T17:23:53.981 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.981 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:53.981 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.981 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:53.981 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.650189+0000", 2024-04-20T17:23:53.981 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.981 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.981 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.982 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:53.982 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.650189+0000", 2024-04-20T17:23:53.982 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.982 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.982 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.982 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:53.982 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.650192+0000", 2024-04-20T17:23:53.982 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7769999999999998e-06 2024-04-20T17:23:53.982 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.982 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.982 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:53.982 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.650218+0000", 2024-04-20T17:23:53.982 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6804e-05 2024-04-20T17:23:53.982 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.982 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.983 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:53.983 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.650229+0000", 2024-04-20T17:23:53.983 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0062e-05 2024-04-20T17:23:53.983 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.983 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.983 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:53.983 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.650236+0000", 2024-04-20T17:23:53.983 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0439999999999998e-06 2024-04-20T17:23:53.983 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.983 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.983 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:53.983 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.741633+0000", 2024-04-20T17:23:53.983 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.091397253999999997 2024-04-20T17:23:53.983 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.984 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.984 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:53.984 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.741777+0000", 2024-04-20T17:23:53.984 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014397100000000001 2024-04-20T17:23:53.984 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.984 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.984 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:23:53.984 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.741939+0000", 2024-04-20T17:23:53.984 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00016262700000000001 2024-04-20T17:23:53.984 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.984 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.984 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:53.984 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.760139+0000", 2024-04-20T17:23:53.984 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018199692999999999 2024-04-20T17:23:53.984 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.984 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.985 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:53.985 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.760156+0000", 2024-04-20T17:23:53.985 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6486000000000001e-05 2024-04-20T17:23:53.985 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.985 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.985 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:53.985 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.760209+0000", 2024-04-20T17:23:53.985 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3915999999999997e-05 2024-04-20T17:23:53.985 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.985 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.985 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:53.985 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:48.760258+0000", 2024-04-20T17:23:53.985 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8309000000000001e-05 2024-04-20T17:23:53.985 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:53.986 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:53.986 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:53.986 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.986 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.986 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:128128 3.f 3:f9dd52b4:::benchmark_data_smithi171_110765_object128127:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e386)", 2024-04-20T17:23:53.986 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:49.599278+0000", 2024-04-20T17:23:53.986 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 124.31764211399999, 2024-04-20T17:23:53.986 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.119170786, 2024-04-20T17:23:53.986 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:53.986 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:53.986 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:53.986 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:53.986 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:53.986 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:53.987 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 128128 2024-04-20T17:23:53.987 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.987 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:53.987 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.987 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:53.987 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:49.599278+0000", 2024-04-20T17:23:53.987 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.987 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.987 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.987 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:53.987 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:49.599278+0000", 2024-04-20T17:23:53.987 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.987 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.987 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.987 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:53.988 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:49.599279+0000", 2024-04-20T17:23:53.988 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.77e-06 2024-04-20T17:23:53.988 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.988 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.988 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:53.988 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:49.599320+0000", 2024-04-20T17:23:53.988 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0363e-05 2024-04-20T17:23:53.988 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.988 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.988 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:53.988 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:49.599334+0000", 2024-04-20T17:23:53.988 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4525999999999999e-05 2024-04-20T17:23:53.988 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.988 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.988 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:53.988 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:49.599342+0000", 2024-04-20T17:23:53.989 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0209999999999999e-06 2024-04-20T17:23:53.989 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.989 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.989 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:53.989 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:49.691164+0000", 2024-04-20T17:23:53.989 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.091822068000000007 2024-04-20T17:23:53.989 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.989 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.989 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:53.989 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:49.691257+0000", 2024-04-20T17:23:53.989 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2304000000000003e-05 2024-04-20T17:23:53.989 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.989 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.989 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:23:53.990 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:49.691346+0000", 2024-04-20T17:23:53.990 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9041000000000003e-05 2024-04-20T17:23:53.990 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.990 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.990 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:53.990 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:49.709059+0000", 2024-04-20T17:23:53.990 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.017713459000000001 2024-04-20T17:23:53.990 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.990 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.990 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:53.990 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:49.718358+0000", 2024-04-20T17:23:53.990 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0092983110000000001 2024-04-20T17:23:53.990 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.990 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.991 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:53.991 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:49.718394+0000", 2024-04-20T17:23:53.991 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6358e-05 2024-04-20T17:23:53.991 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.991 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.991 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:53.991 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:49.718448+0000", 2024-04-20T17:23:53.991 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4564999999999997e-05 2024-04-20T17:23:53.991 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:53.991 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:53.991 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:53.991 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.991 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.991 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:128130 3.f 3:ffbb54e4:::benchmark_data_smithi171_110765_object128129:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e386)", 2024-04-20T17:23:53.991 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:49.599355+0000", 2024-04-20T17:23:53.992 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 124.317564598, 2024-04-20T17:23:53.992 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.119190534, 2024-04-20T17:23:53.992 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:53.992 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:53.992 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:53.992 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:53.992 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:53.992 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:53.992 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 128130 2024-04-20T17:23:53.992 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.992 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:53.992 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.992 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:53.992 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:49.599355+0000", 2024-04-20T17:23:53.992 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.993 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.993 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.993 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:53.993 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:49.599355+0000", 2024-04-20T17:23:53.993 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.993 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.993 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.993 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:53.993 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:49.599356+0000", 2024-04-20T17:23:53.993 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1699999999999997e-07 2024-04-20T17:23:53.993 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.993 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.993 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:53.993 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:49.599378+0000", 2024-04-20T17:23:53.993 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2229e-05 2024-04-20T17:23:53.994 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.994 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.994 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:53.994 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:49.599392+0000", 2024-04-20T17:23:53.994 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3635000000000001e-05 2024-04-20T17:23:53.994 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.994 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.994 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:53.994 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:49.599402+0000", 2024-04-20T17:23:53.994 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0491e-05 2024-04-20T17:23:53.994 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.994 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.994 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:53.994 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:49.694956+0000", 2024-04-20T17:23:53.994 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.095553542000000005 2024-04-20T17:23:53.995 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.995 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.995 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:53.995 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:49.695088+0000", 2024-04-20T17:23:53.995 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013210399999999999 2024-04-20T17:23:53.995 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.995 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.995 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:23:53.995 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:49.695260+0000", 2024-04-20T17:23:53.995 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00017160799999999999 2024-04-20T17:23:53.995 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.995 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.995 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:53.995 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:49.709105+0000", 2024-04-20T17:23:53.995 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013845259 2024-04-20T17:23:53.996 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.996 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.996 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:53.996 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:49.718458+0000", 2024-04-20T17:23:53.996 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0093526700000000004 2024-04-20T17:23:53.996 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.996 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.996 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:53.996 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:49.718504+0000", 2024-04-20T17:23:53.996 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5892e-05 2024-04-20T17:23:53.996 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.996 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.996 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:53.996 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:49.718546+0000", 2024-04-20T17:23:53.997 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2187000000000001e-05 2024-04-20T17:23:53.997 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:53.997 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:53.997 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:53.997 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.997 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.997 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:128838 3.f 3:f8a295a4:::benchmark_data_smithi171_110765_object128837:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e386)", 2024-04-20T17:23:53.997 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:50.222504+0000", 2024-04-20T17:23:53.997 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 123.694415799, 2024-04-20T17:23:53.997 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.11502439, 2024-04-20T17:23:53.997 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:53.997 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:53.997 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:53.997 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:53.997 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:53.998 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:53.998 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 128838 2024-04-20T17:23:53.998 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.998 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:53.998 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.998 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:53.998 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.222504+0000", 2024-04-20T17:23:53.998 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.998 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.998 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.998 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:53.998 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.222504+0000", 2024-04-20T17:23:53.998 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:53.998 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.998 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.999 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:53.999 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.222505+0000", 2024-04-20T17:23:53.999 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9100000000000002e-07 2024-04-20T17:23:53.999 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.999 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.999 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:53.999 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.222530+0000", 2024-04-20T17:23:53.999 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5049999999999999e-05 2024-04-20T17:23:53.999 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.999 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:53.999 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:53.999 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.222537+0000", 2024-04-20T17:23:53.999 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.815e-06 2024-04-20T17:23:53.999 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:53.999 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.000 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:54.000 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.222543+0000", 2024-04-20T17:23:54.000 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1129999999999997e-06 2024-04-20T17:23:54.000 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.000 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.000 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:54.000 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.317829+0000", 2024-04-20T17:23:54.000 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.095286313999999997 2024-04-20T17:23:54.000 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.000 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.000 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:54.000 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.317955+0000", 2024-04-20T17:23:54.000 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012579600000000001 2024-04-20T17:23:54.000 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.000 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.001 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:23:54.001 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.318105+0000", 2024-04-20T17:23:54.001 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000150316 2024-04-20T17:23:54.001 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.001 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.001 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:54.001 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.332270+0000", 2024-04-20T17:23:54.001 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014164510999999999 2024-04-20T17:23:54.001 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.001 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.001 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:54.001 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.337440+0000", 2024-04-20T17:23:54.001 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0051701330000000004 2024-04-20T17:23:54.001 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.001 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.002 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:54.002 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.337472+0000", 2024-04-20T17:23:54.002 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2008999999999998e-05 2024-04-20T17:23:54.002 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.002 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.002 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:54.002 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.337528+0000", 2024-04-20T17:23:54.002 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6441999999999999e-05 2024-04-20T17:23:54.002 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:54.002 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:54.002 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:54.002 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.002 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.002 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:128839 3.f 3:fc3296f2:::benchmark_data_smithi171_110765_object128838:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e386)", 2024-04-20T17:23:54.003 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:50.222552+0000", 2024-04-20T17:23:54.003 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 123.69436758400001, 2024-04-20T17:23:54.003 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.11506727899999999, 2024-04-20T17:23:54.003 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:54.003 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:54.003 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:54.003 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:54.003 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:54.003 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:54.003 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 128839 2024-04-20T17:23:54.003 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.003 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:54.003 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.003 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:54.003 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.222552+0000", 2024-04-20T17:23:54.003 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:54.004 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.004 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.004 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:54.004 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.222552+0000", 2024-04-20T17:23:54.004 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:54.004 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.004 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.004 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:54.004 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.222553+0000", 2024-04-20T17:23:54.004 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9100000000000003e-07 2024-04-20T17:23:54.004 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.004 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.004 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:54.004 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.222575+0000", 2024-04-20T17:23:54.004 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1715000000000001e-05 2024-04-20T17:23:54.005 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.005 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.005 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:54.005 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.222581+0000", 2024-04-20T17:23:54.005 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2960000000000004e-06 2024-04-20T17:23:54.005 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.005 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.005 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:54.005 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.222585+0000", 2024-04-20T17:23:54.005 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4089999999999996e-06 2024-04-20T17:23:54.005 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.005 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.005 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:54.005 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.322540+0000", 2024-04-20T17:23:54.006 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.099954735000000003 2024-04-20T17:23:54.006 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.006 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.006 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:54.006 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.322653+0000", 2024-04-20T17:23:54.006 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011262800000000001 2024-04-20T17:23:54.006 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.006 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.006 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:23:54.006 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.322760+0000", 2024-04-20T17:23:54.006 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000106934 2024-04-20T17:23:54.006 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.006 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.006 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:54.006 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.332308+0000", 2024-04-20T17:23:54.007 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0095484820000000005 2024-04-20T17:23:54.007 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.007 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.007 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:54.007 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.337538+0000", 2024-04-20T17:23:54.007 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.005230401 2024-04-20T17:23:54.007 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.007 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.007 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:54.007 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.337570+0000", 2024-04-20T17:23:54.007 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1687999999999999e-05 2024-04-20T17:23:54.007 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.007 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.007 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:54.007 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.337619+0000", 2024-04-20T17:23:54.008 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9299999999999999e-05 2024-04-20T17:23:54.008 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:54.008 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:54.008 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:54.008 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.008 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.008 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:128840 3.f 3:f279fd3c:::benchmark_data_smithi171_110765_object128839:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e386)", 2024-04-20T17:23:54.008 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:50.224347+0000", 2024-04-20T17:23:54.008 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 123.692573154, 2024-04-20T17:23:54.008 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.11841188, 2024-04-20T17:23:54.008 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:54.008 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:54.008 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:54.008 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:54.008 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:54.009 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:54.009 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 128840 2024-04-20T17:23:54.009 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.009 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:54.009 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.009 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:54.009 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.224347+0000", 2024-04-20T17:23:54.009 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:54.009 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.009 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.009 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:54.009 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.224347+0000", 2024-04-20T17:23:54.009 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:54.009 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.009 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.010 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:54.010 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.224348+0000", 2024-04-20T17:23:54.010 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.779e-06 2024-04-20T17:23:54.010 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.010 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.010 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:54.010 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.224370+0000", 2024-04-20T17:23:54.010 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1818999999999999e-05 2024-04-20T17:23:54.010 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.010 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.010 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:54.010 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.224386+0000", 2024-04-20T17:23:54.010 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5495999999999999e-05 2024-04-20T17:23:54.010 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.011 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.011 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:54.011 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.224393+0000", 2024-04-20T17:23:54.011 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7540000000000006e-06 2024-04-20T17:23:54.011 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.011 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.011 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:54.011 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.327264+0000", 2024-04-20T17:23:54.011 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.102870324 2024-04-20T17:23:54.011 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.011 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.011 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:54.011 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.327426+0000", 2024-04-20T17:23:54.011 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00016247399999999999 2024-04-20T17:23:54.011 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.011 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.012 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:23:54.012 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.327568+0000", 2024-04-20T17:23:54.012 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014180900000000001 2024-04-20T17:23:54.012 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.012 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.012 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:54.012 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.342586+0000", 2024-04-20T17:23:54.012 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015018337 2024-04-20T17:23:54.012 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.012 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.012 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:54.012 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.342631+0000", 2024-04-20T17:23:54.012 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4484999999999998e-05 2024-04-20T17:23:54.012 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.013 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.013 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:54.013 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.342699+0000", 2024-04-20T17:23:54.013 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7929999999999995e-05 2024-04-20T17:23:54.013 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.013 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.013 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:54.013 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.342758+0000", 2024-04-20T17:23:54.013 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9673000000000003e-05 2024-04-20T17:23:54.013 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:54.013 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:54.013 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:54.013 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.013 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.013 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:128845 3.f 3:fc3ad6e5:::benchmark_data_smithi171_110765_object128844:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e386)", 2024-04-20T17:23:54.014 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:50.224571+0000", 2024-04-20T17:23:54.014 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 123.692348983, 2024-04-20T17:23:54.014 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.118340996, 2024-04-20T17:23:54.014 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:54.014 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:54.014 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:54.014 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:54.014 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:54.014 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:54.014 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 128845 2024-04-20T17:23:54.014 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.014 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:54.014 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.014 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:54.014 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.224571+0000", 2024-04-20T17:23:54.015 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:54.015 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.015 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.015 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:54.015 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.224571+0000", 2024-04-20T17:23:54.015 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:54.015 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.015 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.015 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:54.015 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.224572+0000", 2024-04-20T17:23:54.015 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6799999999999999e-07 2024-04-20T17:23:54.015 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.015 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.015 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:54.015 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.224590+0000", 2024-04-20T17:23:54.016 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8807000000000001e-05 2024-04-20T17:23:54.016 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.016 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.016 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:54.016 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.224596+0000", 2024-04-20T17:23:54.016 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9869999999999996e-06 2024-04-20T17:23:54.016 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.016 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.016 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:54.016 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.224602+0000", 2024-04-20T17:23:54.016 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3260000000000003e-06 2024-04-20T17:23:54.016 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.016 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.016 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:54.016 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.332352+0000", 2024-04-20T17:23:54.017 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.107750464 2024-04-20T17:23:54.017 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.017 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.017 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:54.017 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.332492+0000", 2024-04-20T17:23:54.017 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013986599999999999 2024-04-20T17:23:54.017 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.017 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.017 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:23:54.017 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.332660+0000", 2024-04-20T17:23:54.017 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000168297 2024-04-20T17:23:54.017 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.017 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.017 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:54.017 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.342777+0000", 2024-04-20T17:23:54.018 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010117107 2024-04-20T17:23:54.018 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.018 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.018 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:54.018 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.342809+0000", 2024-04-20T17:23:54.018 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1102000000000001e-05 2024-04-20T17:23:54.018 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.018 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.018 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:54.018 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.342859+0000", 2024-04-20T17:23:54.018 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0108000000000002e-05 2024-04-20T17:23:54.018 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.018 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.019 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:54.019 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.342912+0000", 2024-04-20T17:23:54.019 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3164000000000001e-05 2024-04-20T17:23:54.019 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:54.019 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:54.019 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:54.019 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.019 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.019 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:128848 3.f 3:f33753fe:::benchmark_data_smithi171_110765_object128847:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e386)", 2024-04-20T17:23:54.019 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:50.228826+0000", 2024-04-20T17:23:54.019 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 123.688093595, 2024-04-20T17:23:54.019 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.118193164, 2024-04-20T17:23:54.019 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:54.019 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:54.020 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:54.020 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:54.020 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:54.020 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:54.020 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 128848 2024-04-20T17:23:54.020 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.020 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:54.020 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.020 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:54.020 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.228826+0000", 2024-04-20T17:23:54.020 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:54.020 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.020 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.020 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:54.020 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.228826+0000", 2024-04-20T17:23:54.020 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:54.021 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.021 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.021 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:54.021 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.228829+0000", 2024-04-20T17:23:54.021 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1549999999999999e-06 2024-04-20T17:23:54.021 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.021 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.021 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:54.021 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.228856+0000", 2024-04-20T17:23:54.021 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6200999999999999e-05 2024-04-20T17:23:54.021 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.021 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.021 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:54.021 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.228868+0000", 2024-04-20T17:23:54.021 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2265e-05 2024-04-20T17:23:54.022 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.022 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.022 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:54.022 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.228877+0000", 2024-04-20T17:23:54.022 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.7139999999999999e-06 2024-04-20T17:23:54.022 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.022 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.022 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:54.022 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.337673+0000", 2024-04-20T17:23:54.022 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.10879638 2024-04-20T17:23:54.022 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.022 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.022 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:54.022 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.337792+0000", 2024-04-20T17:23:54.022 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011956900000000001 2024-04-20T17:23:54.023 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.023 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.023 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:23:54.023 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.337962+0000", 2024-04-20T17:23:54.023 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000169087 2024-04-20T17:23:54.023 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.023 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.023 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:54.023 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.342953+0000", 2024-04-20T17:23:54.023 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0049910609999999998 2024-04-20T17:23:54.023 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.023 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.023 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:54.023 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.346908+0000", 2024-04-20T17:23:54.023 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0039549019999999997 2024-04-20T17:23:54.024 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.024 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.024 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:54.024 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.346959+0000", 2024-04-20T17:23:54.024 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1548000000000002e-05 2024-04-20T17:23:54.024 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.024 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.024 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:54.024 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.347019+0000", 2024-04-20T17:23:54.024 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0282000000000003e-05 2024-04-20T17:23:54.024 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:54.024 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:54.024 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:54.024 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.024 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.024 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:128849 3.f 3:f595832e:::benchmark_data_smithi171_110765_object128848:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e386)", 2024-04-20T17:23:54.025 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:50.229225+0000", 2024-04-20T17:23:54.025 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 123.68769487, 2024-04-20T17:23:54.025 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.122216725, 2024-04-20T17:23:54.025 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:54.025 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:54.025 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:54.025 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:54.025 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:54.025 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:54.025 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 128849 2024-04-20T17:23:54.025 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.025 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:54.025 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.025 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:54.025 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.229225+0000", 2024-04-20T17:23:54.026 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:54.026 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.026 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.026 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:54.026 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.229225+0000", 2024-04-20T17:23:54.026 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:54.026 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.026 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.026 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:54.026 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.229226+0000", 2024-04-20T17:23:54.026 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3880000000000001e-06 2024-04-20T17:23:54.026 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.026 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.026 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:54.027 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.229246+0000", 2024-04-20T17:23:54.027 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9684e-05 2024-04-20T17:23:54.027 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.027 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.027 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:54.027 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.229254+0000", 2024-04-20T17:23:54.028 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9370000000000004e-06 2024-04-20T17:23:54.028 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.028 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.028 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:54.028 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.229259+0000", 2024-04-20T17:23:54.028 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9400000000000001e-06 2024-04-20T17:23:54.029 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.029 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.029 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:54.029 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.342986+0000", 2024-04-20T17:23:54.029 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.11372665 2024-04-20T17:23:54.029 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.029 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.029 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:54.029 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.343141+0000", 2024-04-20T17:23:54.029 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015515900000000001 2024-04-20T17:23:54.029 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.029 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.029 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:23:54.029 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.343265+0000", 2024-04-20T17:23:54.030 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012449299999999999 2024-04-20T17:23:54.030 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.030 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.030 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:54.030 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.351255+0000", 2024-04-20T17:23:54.030 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0079898290000000004 2024-04-20T17:23:54.030 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.030 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.030 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:54.030 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.351316+0000", 2024-04-20T17:23:54.030 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1113999999999998e-05 2024-04-20T17:23:54.030 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.030 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.031 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:54.031 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.351370+0000", 2024-04-20T17:23:54.031 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3705000000000002e-05 2024-04-20T17:23:54.031 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.031 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.031 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:54.031 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.351442+0000", 2024-04-20T17:23:54.031 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1826e-05 2024-04-20T17:23:54.031 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:54.031 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:54.031 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:54.031 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.031 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.031 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:128864 3.f 3:fd203149:::benchmark_data_smithi171_110765_object128863:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e386)", 2024-04-20T17:23:54.031 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:50.242354+0000", 2024-04-20T17:23:54.031 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 123.67456552, 2024-04-20T17:23:54.032 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.11400644, 2024-04-20T17:23:54.032 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:23:54.032 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:23:54.032 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:23:54.032 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:23:54.032 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:23:54.032 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:23:54.032 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 128864 2024-04-20T17:23:54.032 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.032 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:23:54.032 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.032 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:23:54.032 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.242354+0000", 2024-04-20T17:23:54.032 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:54.032 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.033 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.033 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:23:54.033 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.242354+0000", 2024-04-20T17:23:54.033 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:23:54.033 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.033 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.033 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:23:54.033 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.242357+0000", 2024-04-20T17:23:54.033 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3980000000000002e-06 2024-04-20T17:23:54.033 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.033 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.033 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:23:54.033 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.242380+0000", 2024-04-20T17:23:54.033 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3564e-05 2024-04-20T17:23:54.033 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.034 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.034 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:23:54.034 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.242394+0000", 2024-04-20T17:23:54.034 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3433e-05 2024-04-20T17:23:54.034 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.034 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.034 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:23:54.034 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.242402+0000", 2024-04-20T17:23:54.034 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8539999999999997e-06 2024-04-20T17:23:54.034 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.034 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.034 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:23:54.034 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.347068+0000", 2024-04-20T17:23:54.034 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.10466623799999999 2024-04-20T17:23:54.034 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.035 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.035 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:23:54.035 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.347189+0000", 2024-04-20T17:23:54.035 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000120876 2024-04-20T17:23:54.035 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.035 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.035 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:23:54.035 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.347353+0000", 2024-04-20T17:23:54.035 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000164854 2024-04-20T17:23:54.035 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.035 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.035 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:23:54.035 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.351478+0000", 2024-04-20T17:23:54.035 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0041248409999999998 2024-04-20T17:23:54.035 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.036 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.036 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:23:54.036 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.356249+0000", 2024-04-20T17:23:54.036 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0047706780000000004 2024-04-20T17:23:54.036 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.036 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.036 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:23:54.036 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.356286+0000", 2024-04-20T17:23:54.036 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7329999999999997e-05 2024-04-20T17:23:54.036 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:23:54.036 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:23:54.036 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:23:54.036 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.356361+0000", 2024-04-20T17:23:54.036 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4374000000000003e-05 2024-04-20T17:23:54.036 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:54.036 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:54.037 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:54.037 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:23:54.037 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:23:54.037 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:23:54.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:54.038 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:54.038 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:23:54.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:23:54.139 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:23:54.213 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:54.253 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:23:54.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:54.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:23:54.359 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:23:54.390 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:54.416 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:23:54.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:54.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:23:54.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:54 smithi005 ceph-mon[39562]: 3.e scrub starts 2024-04-20T17:23:54.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:54 smithi005 ceph-mon[39562]: 3.e scrub ok 2024-04-20T17:23:54.520 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:23:54.551 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:54.566 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:23:54.608 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:23:54.648 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:23:54.675 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:23:54.713 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:23:54.747 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:23:54.753 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:54 smithi171 ceph-mon[44189]: 3.e scrub starts 2024-04-20T17:23:54.753 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:54 smithi171 ceph-mon[44189]: 3.e scrub ok 2024-04-20T17:23:54.754 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:23:54.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:54.758 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:23:54.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:54 smithi106 ceph-mon[44423]: 3.e scrub starts 2024-04-20T17:23:54.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:54 smithi106 ceph-mon[44423]: 3.e scrub ok 2024-04-20T17:23:54.777 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:23:54.798 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-20T17:23:54.814 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:23:54.848 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:23:54.859 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:23:54.900 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:23:54.908 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:54.934 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:23:54.952 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:23:54.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:54.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:23:54.991 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:23:55.001 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:23:55.001 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:23:55.001 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:23:55.001 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:23:55.001 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:23:55.001 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:23:55.001 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:23:55.001 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:23:55.001 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:23:55.002 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:23:55.002 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:23:55.002 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:23:55.063 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:23:55.091 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:55.115 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:23:55.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:55.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:23:55.222 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:23:55.258 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:55.285 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:23:55.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:55.289 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:23:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:set 3.c pg_upmap mapping to [3,11] 2024-04-20T17:23:55.389 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:23:55.438 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:55.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:55 smithi171 ceph-mon[44189]: pgmap v1151: 17 pgs: 17 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:23:55.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:55 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4270202538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:23:55.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:55 smithi171 ceph-mon[44189]: 3.2 scrub starts 2024-04-20T17:23:55.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:55.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:55.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:55.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:55 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4170920677' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.c", "id": [3, 11]}]: dispatch 2024-04-20T17:23:55.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:55 smithi171 ceph-mon[44189]: from='client.25981 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.c", "id": [3, 11]}]: dispatch 2024-04-20T17:23:55.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:55 smithi171 ceph-mon[44189]: 3.2 scrub ok 2024-04-20T17:23:55.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:55.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:55.609 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:23:55.641 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:55 smithi005 ceph-mon[39562]: pgmap v1151: 17 pgs: 17 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:23:55.641 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:55 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4270202538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:23:55.641 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:55 smithi005 ceph-mon[39562]: 3.2 scrub starts 2024-04-20T17:23:55.641 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:55.641 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:55.641 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:55.641 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:55 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4170920677' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.c", "id": [3, 11]}]: dispatch 2024-04-20T17:23:55.641 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:55 smithi005 ceph-mon[39562]: from='client.25981 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.c", "id": [3, 11]}]: dispatch 2024-04-20T17:23:55.642 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:55 smithi005 ceph-mon[39562]: 3.2 scrub ok 2024-04-20T17:23:55.642 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:55.642 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:55.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:55 smithi106 ceph-mon[44423]: pgmap v1151: 17 pgs: 17 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:23:55.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:55 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4270202538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:23:55.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:55 smithi106 ceph-mon[44423]: 3.2 scrub starts 2024-04-20T17:23:55.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:55.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:55.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:55.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:55 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4170920677' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.c", "id": [3, 11]}]: dispatch 2024-04-20T17:23:55.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:55 smithi106 ceph-mon[44423]: from='client.25981 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.c", "id": [3, 11]}]: dispatch 2024-04-20T17:23:55.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:55 smithi106 ceph-mon[44423]: 3.2 scrub ok 2024-04-20T17:23:55.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:55.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:55.897 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:23:55.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:55.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:23:56.001 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:23:56.028 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:56.101 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:23:56.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:56.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:23:56.132 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:23:56.132 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:23:56.206 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:23:56.234 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:56.253 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:23:56.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:56.256 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:23:56.357 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:23:56.384 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:56.402 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:23:56.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:56.405 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:23:56.506 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:23:56.530 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:56.530 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:56.530 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:56.530 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:56.531 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:56.531 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:56.531 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:56.531 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:56 smithi171 ceph-mon[44189]: from='client.25981 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.c", "id": [3, 11]}]': finished 2024-04-20T17:23:56.531 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:56 smithi171 ceph-mon[44189]: osdmap e428: 12 total, 12 up, 12 in 2024-04-20T17:23:56.531 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:56 smithi171 ceph-mon[44189]: 3.f scrub starts 2024-04-20T17:23:56.531 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:56 smithi171 ceph-mon[44189]: osdmap e429: 12 total, 12 up, 12 in 2024-04-20T17:23:56.532 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:56.550 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:23:56.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:56.554 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:23:56.655 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:23:56.683 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:56.683 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:56.683 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:56.684 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:56.684 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:56.684 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:56.684 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:56.684 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:56 smithi005 ceph-mon[39562]: from='client.25981 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.c", "id": [3, 11]}]': finished 2024-04-20T17:23:56.684 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:56 smithi005 ceph-mon[39562]: osdmap e428: 12 total, 12 up, 12 in 2024-04-20T17:23:56.684 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:56 smithi005 ceph-mon[39562]: 3.f scrub starts 2024-04-20T17:23:56.684 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:56 smithi005 ceph-mon[39562]: osdmap e429: 12 total, 12 up, 12 in 2024-04-20T17:23:56.685 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:56.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:56.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:56.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:56.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:56.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:56.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:56.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:23:56.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:56 smithi106 ceph-mon[44423]: from='client.25981 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.c", "id": [3, 11]}]': finished 2024-04-20T17:23:56.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:56 smithi106 ceph-mon[44423]: osdmap e428: 12 total, 12 up, 12 in 2024-04-20T17:23:56.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:56 smithi106 ceph-mon[44423]: 3.f scrub starts 2024-04-20T17:23:56.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:56 smithi106 ceph-mon[44423]: osdmap e429: 12 total, 12 up, 12 in 2024-04-20T17:23:57.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:57 smithi005 ceph-mon[39562]: 3.f scrub ok 2024-04-20T17:23:57.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:57 smithi005 ceph-mon[39562]: pgmap v1154: 17 pgs: 1 remapped+peering, 16 active+clean; 1.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.32k op/s 2024-04-20T17:23:57.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:57 smithi005 ceph-mon[39562]: 3.a scrub starts 2024-04-20T17:23:57.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:57 smithi005 ceph-mon[39562]: 3.a scrub ok 2024-04-20T17:23:57.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:57 smithi005 ceph-mon[39562]: osdmap e430: 12 total, 12 up, 12 in 2024-04-20T17:23:57.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:57 smithi106 ceph-mon[44423]: 3.f scrub ok 2024-04-20T17:23:57.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:57 smithi106 ceph-mon[44423]: pgmap v1154: 17 pgs: 1 remapped+peering, 16 active+clean; 1.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.32k op/s 2024-04-20T17:23:57.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:57 smithi106 ceph-mon[44423]: 3.a scrub starts 2024-04-20T17:23:57.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:57 smithi106 ceph-mon[44423]: 3.a scrub ok 2024-04-20T17:23:57.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:57 smithi106 ceph-mon[44423]: osdmap e430: 12 total, 12 up, 12 in 2024-04-20T17:23:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:57 smithi171 ceph-mon[44189]: 3.f scrub ok 2024-04-20T17:23:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:57 smithi171 ceph-mon[44189]: pgmap v1154: 17 pgs: 1 remapped+peering, 16 active+clean; 1.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.32k op/s 2024-04-20T17:23:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:57 smithi171 ceph-mon[44189]: 3.a scrub starts 2024-04-20T17:23:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:57 smithi171 ceph-mon[44189]: 3.a scrub ok 2024-04-20T17:23:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:57 smithi171 ceph-mon[44189]: osdmap e430: 12 total, 12 up, 12 in 2024-04-20T17:23:57.882 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:23:58.239 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:23:58.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:58.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:23:58.346 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:23:58.374 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:58.396 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:23:58.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:58.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:23:58.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:58 smithi005 ceph-mon[39562]: 3.c starting backfill to osd.11 from (0'0,0'0] MAX to 427'16741 2024-04-20T17:23:58.502 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:23:58.531 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:58.558 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:23:58.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:58.565 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:23:58.666 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:23:58.692 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:58 smithi106 ceph-mon[44423]: 3.c starting backfill to osd.11 from (0'0,0'0] MAX to 427'16741 2024-04-20T17:23:58.694 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:58.723 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:23:58.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:23:58.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:23:58.829 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:23:58.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:58 smithi171 ceph-mon[44189]: 3.c starting backfill to osd.11 from (0'0,0'0] MAX to 427'16741 2024-04-20T17:23:58.865 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:23:59.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:59 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:23:59.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:59 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:23:59.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:23:59 smithi005 ceph-mon[39562]: pgmap v1156: 17 pgs: 1 remapped+peering, 16 active+clean; 960 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-04-20T17:23:59.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:59 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:23:59.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:59 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:23:59.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:23:59 smithi106 ceph-mon[44423]: pgmap v1156: 17 pgs: 1 remapped+peering, 16 active+clean; 960 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-04-20T17:23:59.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:59 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:23:59.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:59 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:23:59.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:23:59 smithi171 ceph-mon[44189]: pgmap v1156: 17 pgs: 1 remapped+peering, 16 active+clean; 960 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-04-20T17:24:01.231 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:24:01.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:01.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:24:01.340 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:24:01.376 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:01.396 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:24:01.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:01.402 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:24:01.484 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 8, 9, 10, 11, 2, 7] 2024-04-20T17:24:01.485 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:24:01.485 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-20T17:24:01.485 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.031379 2024-04-20T17:24:01.485 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd primary-affinity 1 0.03137910853457204 2024-04-20T17:24:01.486 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:01 smithi005 ceph-mon[39562]: pgmap v1157: 17 pgs: 1 remapped+peering, 16 active+clean; 918 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 845 op/s 2024-04-20T17:24:01.503 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:24:01.590 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:01.754 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:24:01.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:01 smithi106 ceph-mon[44423]: pgmap v1157: 17 pgs: 1 remapped+peering, 16 active+clean; 918 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 845 op/s 2024-04-20T17:24:01.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:01.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:24:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:01 smithi171 ceph-mon[44189]: pgmap v1157: 17 pgs: 1 remapped+peering, 16 active+clean; 918 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 845 op/s 2024-04-20T17:24:01.864 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:24:01.949 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:02.091 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:24:02.091 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:24:02.102 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:24:02.413 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:02 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:24:02.413 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:02 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:24:02.645 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:24:02.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:02.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:24:02.753 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:24:02.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:02 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:24:02.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:02 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:24:02.784 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:02.808 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:24:02.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:02.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:24:02.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:02 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:24:02.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:02 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:24:02.918 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:24:02.947 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:02.972 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:24:02.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:02.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:24:03.082 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:24:03.122 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:03.149 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:24:03.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:03.156 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:24:03.256 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:24:03.333 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:03.437 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:24:03.705 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:24:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:03 smithi005 ceph-mon[39562]: pgmap v1158: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 803 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 904 op/s; 217/25680 objects misplaced (0.845%); 5.3 MiB/s, 84 objects/s recovering 2024-04-20T17:24:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:03 smithi005 ceph-mon[39562]: osdmap e431: 12 total, 12 up, 12 in 2024-04-20T17:24:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:03 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/679899739' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:24:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:03 smithi005 ceph-mon[39562]: 3.b scrub starts 2024-04-20T17:24:03.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:03.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:24:03.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:03 smithi106 ceph-mon[44423]: pgmap v1158: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 803 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 904 op/s; 217/25680 objects misplaced (0.845%); 5.3 MiB/s, 84 objects/s recovering 2024-04-20T17:24:03.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:03 smithi106 ceph-mon[44423]: osdmap e431: 12 total, 12 up, 12 in 2024-04-20T17:24:03.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:03 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/679899739' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:24:03.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:03 smithi106 ceph-mon[44423]: 3.b scrub starts 2024-04-20T17:24:03.808 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:24:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:03 smithi171 ceph-mon[44189]: pgmap v1158: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 803 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 904 op/s; 217/25680 objects misplaced (0.845%); 5.3 MiB/s, 84 objects/s recovering 2024-04-20T17:24:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:03 smithi171 ceph-mon[44189]: osdmap e431: 12 total, 12 up, 12 in 2024-04-20T17:24:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:03 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/679899739' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:24:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:03 smithi171 ceph-mon[44189]: 3.b scrub starts 2024-04-20T17:24:03.836 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:04.038 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:24:04.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:04.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:24:04.047 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-20T17:24:04.143 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:24:04.175 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:04.194 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:24:04.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:04.197 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:24:04.299 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:24:04.327 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:04.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:04 smithi171 ceph-mon[44189]: 3.b scrub ok 2024-04-20T17:24:04.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:04 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:24:04.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:04 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:24:04.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:04 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:24:04.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:04 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:24:04.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:04 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/679899739' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:24:04.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:04 smithi171 ceph-mon[44189]: osdmap e432: 12 total, 12 up, 12 in 2024-04-20T17:24:04.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:04 smithi171 ceph-mon[44189]: 3.7 scrub starts 2024-04-20T17:24:04.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:04 smithi171 ceph-mon[44189]: 3.7 scrub ok 2024-04-20T17:24:04.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:04 smithi005 ceph-mon[39562]: 3.b scrub ok 2024-04-20T17:24:04.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:04 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:24:04.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:04 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:24:04.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:04 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:24:04.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:04 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:24:04.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:04 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/679899739' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:24:04.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:04 smithi005 ceph-mon[39562]: osdmap e432: 12 total, 12 up, 12 in 2024-04-20T17:24:04.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:04 smithi005 ceph-mon[39562]: 3.7 scrub starts 2024-04-20T17:24:04.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:04 smithi005 ceph-mon[39562]: 3.7 scrub ok 2024-04-20T17:24:04.759 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:24:04.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:04 smithi106 ceph-mon[44423]: 3.b scrub ok 2024-04-20T17:24:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:04 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:24:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:04 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:24:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:04 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:24:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:04 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:24:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:04 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/679899739' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:24:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:04 smithi106 ceph-mon[44423]: osdmap e432: 12 total, 12 up, 12 in 2024-04-20T17:24:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:04 smithi106 ceph-mon[44423]: 3.7 scrub starts 2024-04-20T17:24:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:04 smithi106 ceph-mon[44423]: 3.7 scrub ok 2024-04-20T17:24:04.790 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:24:04.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:04.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:24:04.897 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:24:04.933 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:04.965 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:24:04.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:04.972 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:24:05.073 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:24:05.102 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:05.132 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:24:05.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:05.137 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:24:05.239 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:24:05.275 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:05.301 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:24:05.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:05.304 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:24:05.405 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:24:05.434 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:05.468 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:24:05.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:05.471 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:24:05.572 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:24:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:05 smithi171 ceph-mon[44189]: 3.d scrub starts 2024-04-20T17:24:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:05 smithi171 ceph-mon[44189]: 3.d scrub ok 2024-04-20T17:24:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:05 smithi171 ceph-mon[44189]: pgmap v1161: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 651 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 931 op/s; 217/20820 objects misplaced (1.042%); 5.1 MiB/s, 81 objects/s recovering 2024-04-20T17:24:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:05 smithi171 ceph-mon[44189]: osdmap e433: 12 total, 12 up, 12 in 2024-04-20T17:24:05.586 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:24:05.615 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:24:05.616 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:05.650 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:24:05.658 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:24:05.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:05.670 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:24:05.679 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:24:05.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:05 smithi005 ceph-mon[39562]: 3.d scrub starts 2024-04-20T17:24:05.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:05 smithi005 ceph-mon[39562]: 3.d scrub ok 2024-04-20T17:24:05.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:05 smithi005 ceph-mon[39562]: pgmap v1161: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 651 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 931 op/s; 217/20820 objects misplaced (1.042%); 5.1 MiB/s, 81 objects/s recovering 2024-04-20T17:24:05.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:05 smithi005 ceph-mon[39562]: osdmap e433: 12 total, 12 up, 12 in 2024-04-20T17:24:05.720 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:24:05.755 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:24:05.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:05 smithi106 ceph-mon[44423]: 3.d scrub starts 2024-04-20T17:24:05.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:05 smithi106 ceph-mon[44423]: 3.d scrub ok 2024-04-20T17:24:05.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:05 smithi106 ceph-mon[44423]: pgmap v1161: 17 pgs: 1 active+remapped+backfilling, 16 active+clean; 651 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 931 op/s; 217/20820 objects misplaced (1.042%); 5.1 MiB/s, 81 objects/s recovering 2024-04-20T17:24:05.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:05 smithi106 ceph-mon[44423]: osdmap e433: 12 total, 12 up, 12 in 2024-04-20T17:24:05.771 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:24:05.792 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:24:05.821 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:05.824 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:24:05.868 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:24:05.903 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:24:05.937 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:24:05.945 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:24:05.945 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:24:05.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:05.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:24:05.991 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:24:05.994 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:24:05.994 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:24:05.994 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:24:05.994 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:24:05.994 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:24:05.994 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:24:05.994 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:24:05.994 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:24:05.995 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:24:05.995 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:24:05.995 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:24:05.995 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:24:06.055 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:24:06.082 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:06.102 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:24:06.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:06.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:24:06.209 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:24:06.249 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:06.271 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:24:06.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:06.274 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:24:06.375 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:24:06.422 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:06.451 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:24:06.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:06.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:24:06.570 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:24:06.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:06 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3486142005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:24:06.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:06.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:06.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:06.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:06.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:06.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:06.628 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:06.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:06 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3486142005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:24:06.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:06.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:06.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:06.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:06.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:06.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:06.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:06 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3486142005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:24:06.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:06.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:06.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:06.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:06.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:06.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:06.787 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:24:07.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:07.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:07.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:07.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:07.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:07.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:07.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:07 smithi005 ceph-mon[39562]: pgmap v1163: 17 pgs: 17 active+clean; 498 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 8.1 MiB/s, 129 objects/s recovering 2024-04-20T17:24:07.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:07.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:07.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:07.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:07.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:07.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:07.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:07 smithi106 ceph-mon[44423]: pgmap v1163: 17 pgs: 17 active+clean; 498 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 8.1 MiB/s, 129 objects/s recovering 2024-04-20T17:24:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:07 smithi171 ceph-mon[44189]: pgmap v1163: 17 pgs: 17 active+clean; 498 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 8.1 MiB/s, 129 objects/s recovering 2024-04-20T17:24:09.133 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:24:09.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:09.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:24:09.249 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:24:09.289 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:09.404 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:24:09.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:09.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:24:09.511 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:24:09.546 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:09.574 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:24:09.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:09.577 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:24:09.678 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:24:09.711 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:09 smithi106 ceph-mon[44423]: pgmap v1164: 17 pgs: 17 active+clean; 363 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 2.1 MiB/s, 33 objects/s recovering 2024-04-20T17:24:09.711 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:09 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:24:09.711 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:09 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:24:09.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:09 smithi005 ceph-mon[39562]: pgmap v1164: 17 pgs: 17 active+clean; 363 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 2.1 MiB/s, 33 objects/s recovering 2024-04-20T17:24:09.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:09 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:24:09.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:09 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:24:09.713 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:09.746 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:24:09.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:09.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:24:09.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:09 smithi171 ceph-mon[44189]: pgmap v1164: 17 pgs: 17 active+clean; 363 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 2.1 MiB/s, 33 objects/s recovering 2024-04-20T17:24:09.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:09 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:24:09.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:09 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:24:09.849 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:24:09.881 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:09.911 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:24:09.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:09.914 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:24:10.015 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:24:10.048 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:10.079 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:24:10.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:10.086 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:24:10.187 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:24:10.232 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:10.268 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:24:10.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:10.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:24:10.377 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:24:10.406 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:10.433 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:24:10.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:10.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:24:10.471 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.1 primary-affinity to 0.0313791 (820562) 2024-04-20T17:24:10.538 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:24:10.565 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:10.586 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:24:10.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:10.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:24:10.690 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:24:10.719 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:10.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:10 smithi106 ceph-mon[44423]: osdmap e434: 12 total, 12 up, 12 in 2024-04-20T17:24:10.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:10 smithi106 ceph-mon[44423]: 3.5 scrub starts 2024-04-20T17:24:10.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:10 smithi106 ceph-mon[44423]: 3.5 scrub ok 2024-04-20T17:24:10.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:10 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/799367711' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.03137910853457204}]: dispatch 2024-04-20T17:24:10.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:10 smithi106 ceph-mon[44423]: from='client.25996 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.03137910853457204}]: dispatch 2024-04-20T17:24:10.821 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:24:10.823 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:10 smithi005 ceph-mon[39562]: osdmap e434: 12 total, 12 up, 12 in 2024-04-20T17:24:10.823 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:10 smithi005 ceph-mon[39562]: 3.5 scrub starts 2024-04-20T17:24:10.823 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:10 smithi005 ceph-mon[39562]: 3.5 scrub ok 2024-04-20T17:24:10.823 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:10 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/799367711' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.03137910853457204}]: dispatch 2024-04-20T17:24:10.823 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:10 smithi005 ceph-mon[39562]: from='client.25996 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.03137910853457204}]: dispatch 2024-04-20T17:24:10.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:10.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:24:10.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:10 smithi171 ceph-mon[44189]: osdmap e434: 12 total, 12 up, 12 in 2024-04-20T17:24:10.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:10 smithi171 ceph-mon[44189]: 3.5 scrub starts 2024-04-20T17:24:10.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:10 smithi171 ceph-mon[44189]: 3.5 scrub ok 2024-04-20T17:24:10.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:10 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/799367711' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.03137910853457204}]: dispatch 2024-04-20T17:24:10.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:10 smithi171 ceph-mon[44189]: from='client.25996 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.03137910853457204}]: dispatch 2024-04-20T17:24:10.925 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:24:10.951 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:10.968 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:24:10.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:10.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:24:11.072 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:24:11.099 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:11.117 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:24:11.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:11.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:24:11.220 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:24:11.246 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:11.265 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:24:11.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:11.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:24:11.368 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:24:11.397 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:11 smithi106 ceph-mon[44423]: pgmap v1166: 17 pgs: 17 active+clean; 300 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 880 op/s; 1.9 MiB/s, 29 objects/s recovering 2024-04-20T17:24:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:11 smithi106 ceph-mon[44423]: from='client.25996 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.03137910853457204}]': finished 2024-04-20T17:24:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:11 smithi106 ceph-mon[44423]: osdmap e435: 12 total, 12 up, 12 in 2024-04-20T17:24:11.545 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:24:11.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:11.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:24:11.650 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:24:11.692 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:11 smithi005 ceph-mon[39562]: pgmap v1166: 17 pgs: 17 active+clean; 300 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 880 op/s; 1.9 MiB/s, 29 objects/s recovering 2024-04-20T17:24:11.692 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:11 smithi005 ceph-mon[39562]: from='client.25996 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.03137910853457204}]': finished 2024-04-20T17:24:11.692 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:11 smithi005 ceph-mon[39562]: osdmap e435: 12 total, 12 up, 12 in 2024-04-20T17:24:11.695 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:11.810 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:24:11.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:11.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:24:11.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:11 smithi171 ceph-mon[44189]: pgmap v1166: 17 pgs: 17 active+clean; 300 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 880 op/s; 1.9 MiB/s, 29 objects/s recovering 2024-04-20T17:24:11.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:11 smithi171 ceph-mon[44189]: from='client.25996 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.03137910853457204}]': finished 2024-04-20T17:24:11.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:11 smithi171 ceph-mon[44189]: osdmap e435: 12 total, 12 up, 12 in 2024-04-20T17:24:11.916 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:24:11.956 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:11.984 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:24:11.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:11.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:24:12.088 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:24:12.093 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:24:12.123 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Removed 143212 objects 2024-04-20T17:24:12.124 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Clean up completed and total clean up time :130.299 2024-04-20T17:24:12.163 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:12.191 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:24:12.191 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:24:12.191 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:24:12.191 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:24:12.191 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:24:12.485 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:24:12.748 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:12 smithi005 ceph-mon[39562]: osdmap e436: 12 total, 12 up, 12 in 2024-04-20T17:24:12.748 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:24:12.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:12 smithi106 ceph-mon[44423]: osdmap e436: 12 total, 12 up, 12 in 2024-04-20T17:24:12.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:24:12.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:12 smithi171 ceph-mon[44189]: osdmap e436: 12 total, 12 up, 12 in 2024-04-20T17:24:12.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:24:13.072 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-20T17:24:13.118 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:24:13.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:13.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:24:13.222 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:24:13.254 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:13.275 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:24:13.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:13.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:24:13.380 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:24:13.409 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:13.429 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:24:13.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:13.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:24:13.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:13 smithi106 ceph-mon[44423]: pgmap v1169: 17 pgs: 17 active+clean; 169 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 878 op/s 2024-04-20T17:24:13.532 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:24:13.566 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:13.588 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:24:13.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:13.591 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:24:13.692 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:24:13.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:13 smithi005 ceph-mon[39562]: pgmap v1169: 17 pgs: 17 active+clean; 169 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 878 op/s 2024-04-20T17:24:13.721 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:13.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:13 smithi171 ceph-mon[44189]: pgmap v1169: 17 pgs: 17 active+clean; 169 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 878 op/s 2024-04-20T17:24:14.398 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:24:15.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:15 smithi005 ceph-mon[39562]: pgmap v1170: 17 pgs: 17 active+clean; 35 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 875 op/s 2024-04-20T17:24:15.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:15 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:24:15.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:15 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:24:15.620 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:15 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:24:15.620 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:15 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:24:15.620 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:15 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:24:15.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:15 smithi106 ceph-mon[44423]: pgmap v1170: 17 pgs: 17 active+clean; 35 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 875 op/s 2024-04-20T17:24:15.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:15 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:24:15.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:15 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:24:15.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:15 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:24:15.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:15 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:24:15.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:15 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:24:15.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:15 smithi171 ceph-mon[44189]: pgmap v1170: 17 pgs: 17 active+clean; 35 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 875 op/s 2024-04-20T17:24:15.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:15 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:24:15.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:15 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:24:15.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:15 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:24:15.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:15 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:24:15.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:15 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:24:15.955 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:24:15.972 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:24:15.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:15.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:24:16.077 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:24:16.111 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:16.210 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:24:16.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:16.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:24:16.315 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:24:16.345 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:16 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:24:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:16 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:24:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:16 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:24:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:16 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:24:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:16 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:24:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:16 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:24:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:16 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:24:16.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:16 smithi106 ceph-mon[44423]: 3.1 scrub starts 2024-04-20T17:24:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:16 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:24:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:16 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:24:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:16 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:24:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:16 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:24:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:16 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:24:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:16 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:24:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:16 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:24:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:16 smithi171 ceph-mon[44189]: 3.1 scrub starts 2024-04-20T17:24:16.894 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:24:16.896 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:16 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:24:16.896 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:16 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:24:16.896 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:16 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:24:16.896 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:16 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:24:16.896 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:16 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:24:16.896 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:16 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:24:16.896 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:16 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:24:16.896 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:16 smithi005 ceph-mon[39562]: 3.1 scrub starts 2024-04-20T17:24:16.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:16.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:24:16.998 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:24:17.031 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:17.052 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:24:17.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:17.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:24:17.159 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:24:17.192 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:17.213 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:24:17.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:17.216 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:24:17.317 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:24:17.345 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:17.366 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:24:17.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:17.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:24:17.435 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-04-20T17:24:17.435 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-04-20T17:24:17.469 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:24:17.501 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:17.517 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 8, 9, 10, 11, 2, 7] 2024-04-20T17:24:17.517 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 0 1 2024-04-20T17:24:17.523 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:24:17.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:17.526 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:24:17.627 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:24:17.656 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:17 smithi005 ceph-mon[39562]: 3.1 scrub ok 2024-04-20T17:24:17.657 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:17 smithi005 ceph-mon[39562]: pgmap v1171: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 711 op/s 2024-04-20T17:24:17.657 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:17 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2431212889' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:24:17.660 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:17.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:17 smithi106 ceph-mon[44423]: 3.1 scrub ok 2024-04-20T17:24:17.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:17 smithi106 ceph-mon[44423]: pgmap v1171: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 711 op/s 2024-04-20T17:24:17.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:17 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2431212889' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:24:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:17 smithi171 ceph-mon[44189]: 3.1 scrub ok 2024-04-20T17:24:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:17 smithi171 ceph-mon[44189]: pgmap v1171: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 711 op/s 2024-04-20T17:24:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:17 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2431212889' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:24:18.015 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:24:18.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:18.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:24:18.105 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:24:18.121 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:24:18.150 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:18.160 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:24:18.160 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:24:18.160 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:24:18.160 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:24:18.161 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:24:18.161 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:24:18.161 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:24:18.161 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:24:18.161 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:24:18.161 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:24:18.161 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:24:18.161 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:24:18.161 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:24:18.161 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:24:18.161 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:24:18.161 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:24:18.162 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:24:18.162 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:24:18.162 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:24:18.162 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:24:18.162 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:24:18.162 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:24:18.162 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:24:18.162 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:24:18.348 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:24:18.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:18.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:24:18.453 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:24:18.486 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:18.508 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:24:18.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:18.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:24:18.612 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:24:18.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:18 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2030313852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:24:18.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:18.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:18.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:18.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:18.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:18.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:18.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:18 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:24:18.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:18 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2431212889' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:24:18.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:18 smithi005 ceph-mon[39562]: osdmap e437: 12 total, 12 up, 12 in 2024-04-20T17:24:18.642 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:18.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:18 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2030313852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:24:18.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:18.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:18.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:18.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:18.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:18.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:18.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:18 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:24:18.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:18 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2431212889' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:24:18.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:18 smithi106 ceph-mon[44423]: osdmap e437: 12 total, 12 up, 12 in 2024-04-20T17:24:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:18 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2030313852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:24:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:18 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:24:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:18 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2431212889' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:24:18.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:18 smithi171 ceph-mon[44189]: osdmap e437: 12 total, 12 up, 12 in 2024-04-20T17:24:18.840 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:24:18.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:18.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:24:18.945 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:24:18.978 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:18.999 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:24:19.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:19.002 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:24:19.104 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:24:19.134 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:19.154 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:24:19.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:19.157 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:24:19.258 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:24:19.296 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:19.419 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:24:19.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:19.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:24:19.534 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:24:19.563 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:19.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:19.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:19.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:19.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:19.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:19.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:19.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:19 smithi005 ceph-mon[39562]: pgmap v1173: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 619 op/s 2024-04-20T17:24:19.711 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:24:19.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:19.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:24:19.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:19.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:19.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:19.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:19.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:19.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:19.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:19 smithi106 ceph-mon[44423]: pgmap v1173: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 619 op/s 2024-04-20T17:24:19.815 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:24:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:19 smithi171 ceph-mon[44189]: pgmap v1173: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 619 op/s 2024-04-20T17:24:19.843 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:19.942 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:24:19.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:19.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:24:19.956 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:24:19.956 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:24:19.976 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:24:19.976 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:24:20.046 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:24:20.078 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:20.099 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:24:20.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:20.102 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:24:20.203 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:24:20.236 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:20.258 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:24:20.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:20.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:24:20.362 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:24:20.393 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:20.415 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:24:20.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:20.418 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:24:20.519 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:24:20.549 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:20.569 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:24:20.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:20.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:24:20.672 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:24:20.701 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:20.720 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:24:20.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:20.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:24:20.823 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:24:20.856 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:20.858 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:24:20.877 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:24:20.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:20.880 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:24:20.981 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:24:21.010 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:21.028 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:24:21.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:21.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:24:21.132 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:24:21.160 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:21.177 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:24:21.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:21.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:24:21.281 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:24:21.307 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:21.325 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:24:21.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:21.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:24:21.429 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:24:21.461 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:21.481 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:24:21.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:21.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:24:21.586 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:24:21.613 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:21.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:21 smithi005 ceph-mon[39562]: pgmap v1174: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 309 op/s 2024-04-20T17:24:21.732 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:24:21.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:21 smithi106 ceph-mon[44423]: pgmap v1174: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 309 op/s 2024-04-20T17:24:21.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:21 smithi171 ceph-mon[44189]: pgmap v1174: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 309 op/s 2024-04-20T17:24:22.363 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:24:22.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:22.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:24:22.467 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:24:22.500 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:22.524 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:24:22.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:22.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:24:22.627 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:24:22.656 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:22.675 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:24:22.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:22.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:24:22.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:22 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:24:22.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:22 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:24:22.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:22 smithi005 ceph-mon[39562]: osdmap e438: 12 total, 12 up, 12 in 2024-04-20T17:24:22.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:22 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:24:22.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:22 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:24:22.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:22 smithi106 ceph-mon[44423]: osdmap e438: 12 total, 12 up, 12 in 2024-04-20T17:24:22.779 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:24:22.807 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:22.827 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-20T17:24:22.827 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:24:22.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:22.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:24:22.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:22 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:24:22.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:22 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:24:22.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:22 smithi171 ceph-mon[44189]: osdmap e438: 12 total, 12 up, 12 in 2024-04-20T17:24:22.930 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:24:22.956 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:22.974 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:24:22.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:22.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:24:23.078 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:24:23.091 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:24:23.111 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:23.289 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:24:23.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:23.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:24:23.393 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:24:23.425 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:23.446 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:24:23.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:23.449 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:24:23.550 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:24:23.580 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:23.600 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:24:23.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:23.603 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:24:23.630 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:23 smithi106 ceph-mon[44423]: pgmap v1176: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 68 op/s 2024-04-20T17:24:23.704 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:24:23.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:23 smithi005 ceph-mon[39562]: pgmap v1176: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 68 op/s 2024-04-20T17:24:23.730 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:23 smithi171 ceph-mon[44189]: pgmap v1176: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 68 op/s 2024-04-20T17:24:23.732 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:23.751 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:24:23.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:23.754 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:24:23.851 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:24:23.851 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:24:23.855 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:24:23.885 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:24.233 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:24:24.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:24.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:24:24.337 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:24:24.368 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:24.389 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:24:24.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:24.392 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:24:24.493 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:24:24.514 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'unique_pool_1' removed 2024-04-20T17:24:24.526 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.0 to 1 (10000) 2024-04-20T17:24:24.537 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:24.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:24 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2563759465' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:24:24.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:24 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3514204254' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-20T17:24:24.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:24 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2563759465' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:24:24.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:24 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3514204254' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-20T17:24:24.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:24 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2563759465' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:24:24.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:24 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3514204254' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-20T17:24:25.089 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:24:25.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:25.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:24:25.193 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:24:25.221 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:25.241 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:24:25.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:25.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:24:25.273 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:24:25.273 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:24:25.345 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:24:25.372 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:25.494 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:24:25.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:25.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:24:25.598 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:24:25.622 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:25 smithi106 ceph-mon[44423]: pgmap v1177: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:25.622 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:25 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2563759465' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-20T17:24:25.622 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:25 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3514204254' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-04-20T17:24:25.622 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:25 smithi106 ceph-mon[44423]: osdmap e439: 12 total, 12 up, 12 in 2024-04-20T17:24:25.624 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:25.661 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:24:25.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:25.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:24:25.769 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:24:25.796 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:25.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:25 smithi171 ceph-mon[44189]: pgmap v1177: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:25.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:25 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2563759465' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-20T17:24:25.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:25 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3514204254' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-04-20T17:24:25.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:25 smithi171 ceph-mon[44189]: osdmap e439: 12 total, 12 up, 12 in 2024-04-20T17:24:25.886 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:25 smithi005 ceph-mon[39562]: pgmap v1177: 17 pgs: 17 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:25.886 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:25 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2563759465' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-20T17:24:25.887 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:25 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3514204254' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-04-20T17:24:25.887 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:25 smithi005 ceph-mon[39562]: osdmap e439: 12 total, 12 up, 12 in 2024-04-20T17:24:25.994 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:24:25.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:25.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:24:26.006 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-20T17:24:26.097 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:24:26.124 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:26.144 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:24:26.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:26.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:24:26.248 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:24:26.278 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:26.317 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:24:26.466 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 1 1 2024-04-20T17:24:26.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:26 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:24:26.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:26 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:24:26.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:26 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:24:26.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:26 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:24:26.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:26 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:24:26.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:26 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:24:26.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:26 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:24:26.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:26 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:24:26.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:26 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:24:26.802 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:26 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:24:26.803 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:26 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:24:26.803 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:26 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:24:26.803 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:26 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:24:26.803 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:26 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:24:26.803 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:26 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:24:26.803 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:26 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:24:26.803 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:26 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:24:26.803 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:26 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:24:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:26 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:24:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:26 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:24:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:26 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:24:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:26 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:24:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:26 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:24:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:26 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:24:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:26 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:24:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:26 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:24:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:26 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:24:27.576 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:27 smithi106 ceph-mon[44423]: pgmap v1179: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:27.576 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:27 smithi106 ceph-mon[44423]: osdmap e440: 12 total, 12 up, 12 in 2024-04-20T17:24:27.576 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:27 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1919046509' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:24:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:27 smithi171 ceph-mon[44189]: pgmap v1179: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:27 smithi171 ceph-mon[44189]: osdmap e440: 12 total, 12 up, 12 in 2024-04-20T17:24:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:27 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1919046509' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:24:27.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:27 smithi005 ceph-mon[39562]: pgmap v1179: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:27.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:27 smithi005 ceph-mon[39562]: osdmap e440: 12 total, 12 up, 12 in 2024-04-20T17:24:27.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:27 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1919046509' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:24:28.019 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-20T17:24:28.020 INFO:tasks.radosbench:Beginning radosbench... 2024-04-20T17:24:28.020 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-04-20T17:24:28.020 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-04-20T17:24:28.073 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:24:28.074 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:24:28.074 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:24:28.074 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:24:28.074 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:24:28.074 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:24:28.074 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:24:28.074 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:24:28.074 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:24:28.074 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:24:28.074 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:24:28.074 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:24:28.074 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:24:28.075 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:24:28.075 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:24:28.075 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:24:28.075 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:24:28.075 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:24:28.075 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:24:28.075 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:24:28.075 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:24:28.075 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:24:28.075 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:24:28.075 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:24:28.148 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:24:28.148 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:24:28.148 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-04-20T17:24:28.149 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-04-20T17:24:28.149 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:24:28.432 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:24:28.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:28.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:24:28.536 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:24:28.537 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:24:28.566 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:28.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:28 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/738393139' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:24:28.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:28 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/738393139' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:24:28.892 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.892 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.893 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.893 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.893 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.893 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.893 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.893 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.893 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.893 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.893 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:28.893 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:28 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/738393139' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:24:28.915 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:24:28.918 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-20T17:24:28.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:28.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:24:29.012 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:24:29.020 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:24:29.046 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:29.068 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:24:29.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:29.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:24:29.172 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:24:29.203 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:29.224 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:24:29.226 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:24:29.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:29.227 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:24:29.328 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:24:29.358 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:29.685 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:24:29.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:29.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:29 smithi171 ceph-mon[44189]: pgmap v1181: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:29.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:29 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:24:29.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:29 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/738393139' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:24:29.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:29 smithi171 ceph-mon[44189]: osdmap e441: 12 total, 12 up, 12 in 2024-04-20T17:24:29.885 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:29.885 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:29 smithi005 ceph-mon[39562]: pgmap v1181: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:29.885 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:29 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:24:29.886 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:29 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/738393139' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:24:29.886 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:29 smithi005 ceph-mon[39562]: osdmap e441: 12 total, 12 up, 12 in 2024-04-20T17:24:29.982 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:29.982 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:29 smithi106 ceph-mon[44423]: pgmap v1181: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:29.982 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:29 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:24:29.982 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:29 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/738393139' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:24:29.982 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:29 smithi106 ceph-mon[44423]: osdmap e441: 12 total, 12 up, 12 in 2024-04-20T17:24:30.107 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:24:30.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:30.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:24:30.211 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:24:30.243 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:30.264 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:24:30.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:30.267 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:24:30.368 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:24:30.399 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:30.648 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:24:30.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:30.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:24:30.753 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:24:30.784 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:30.806 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:24:30.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:30.809 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:24:30.820 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:24:30.910 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:24:30.940 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:30.960 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:24:30.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:30.963 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:24:31.064 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:24:31.092 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:31.112 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:24:31.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:31.115 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:24:31.216 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:24:31.244 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:31.263 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:24:31.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:31.266 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:24:31.368 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:24:31.368 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-04-20T17:24:31.368 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-04-20T17:24:31.368 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:24:31.368 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.368 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245460 3.d 3:bba062b0:::benchmark_data_smithi171_110765_object102245:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:24:31.368 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103564+0000", 2024-04-20T17:24:31.368 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 58.262203317000001, 2024-04-20T17:24:31.369 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.086211963000000003, 2024-04-20T17:24:31.369 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:24:31.369 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:24:31.369 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:31.369 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:24:31.369 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:24:31.369 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:31.369 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245460 2024-04-20T17:24:31.369 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.369 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:24:31.369 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.369 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:24:31.369 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103564+0000", 2024-04-20T17:24:31.369 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.370 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.370 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.370 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:24:31.370 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103564+0000", 2024-04-20T17:24:31.370 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.370 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.370 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.370 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:24:31.370 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103565+0000", 2024-04-20T17:24:31.370 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2999999999999999e-07 2024-04-20T17:24:31.370 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.370 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.370 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:24:31.371 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103568+0000", 2024-04-20T17:24:31.371 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9699999999999999e-06 2024-04-20T17:24:31.371 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.371 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.371 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:24:31.371 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103573+0000", 2024-04-20T17:24:31.371 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4220000000000002e-06 2024-04-20T17:24:31.371 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.371 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.371 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:24:31.371 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103577+0000", 2024-04-20T17:24:31.371 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.34e-06 2024-04-20T17:24:31.371 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.371 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.372 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.372 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103883+0000", 2024-04-20T17:24:31.372 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00030601700000000001 2024-04-20T17:24:31.372 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.372 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.372 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:24:31.372 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103890+0000", 2024-04-20T17:24:31.372 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4189999999999996e-06 2024-04-20T17:24:31.372 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.372 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.372 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.372 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.157260+0000", 2024-04-20T17:24:31.372 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.053369403000000003 2024-04-20T17:24:31.372 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.373 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.373 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:24:31.373 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.157381+0000", 2024-04-20T17:24:31.373 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000120776 2024-04-20T17:24:31.373 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.373 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.373 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:24:31.373 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.157507+0000", 2024-04-20T17:24:31.373 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012656000000000001 2024-04-20T17:24:31.373 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.373 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.373 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:24:31.373 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.165767+0000", 2024-04-20T17:24:31.373 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0082595310000000005 2024-04-20T17:24:31.374 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.374 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.374 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:31.374 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189715+0000", 2024-04-20T17:24:31.374 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.023948542999999999 2024-04-20T17:24:31.374 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.374 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.374 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:24:31.374 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189746+0000", 2024-04-20T17:24:31.374 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0939000000000003e-05 2024-04-20T17:24:31.374 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.374 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.374 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:24:31.374 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189776+0000", 2024-04-20T17:24:31.375 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0113000000000001e-05 2024-04-20T17:24:31.375 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.375 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:24:31.375 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.375 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.375 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.375 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245510 3.d 3:b21de460:::benchmark_data_smithi171_110765_object102295:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:24:31.375 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103589+0000", 2024-04-20T17:24:31.375 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 58.262178693000003, 2024-04-20T17:24:31.375 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.086330094999999996, 2024-04-20T17:24:31.375 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:24:31.375 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:24:31.375 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:31.375 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:24:31.376 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:24:31.376 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:31.376 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245510 2024-04-20T17:24:31.376 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.376 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:24:31.376 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.376 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:24:31.376 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103589+0000", 2024-04-20T17:24:31.376 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.376 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.376 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.376 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:24:31.376 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103589+0000", 2024-04-20T17:24:31.376 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.376 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.377 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.377 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:24:31.377 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103590+0000", 2024-04-20T17:24:31.377 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5400000000000001e-07 2024-04-20T17:24:31.377 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.377 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.377 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:24:31.377 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103592+0000", 2024-04-20T17:24:31.377 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.83e-06 2024-04-20T17:24:31.377 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.377 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.377 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:24:31.377 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103600+0000", 2024-04-20T17:24:31.377 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3139999999999998e-06 2024-04-20T17:24:31.378 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.378 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.378 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:24:31.378 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103605+0000", 2024-04-20T17:24:31.378 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.429e-06 2024-04-20T17:24:31.378 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.378 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.378 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.378 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103931+0000", 2024-04-20T17:24:31.378 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00032633599999999999 2024-04-20T17:24:31.378 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.378 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.378 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:24:31.378 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103944+0000", 2024-04-20T17:24:31.378 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.307e-05 2024-04-20T17:24:31.379 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.379 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.379 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.379 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.159475+0000", 2024-04-20T17:24:31.379 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.055531006000000001 2024-04-20T17:24:31.379 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.379 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.379 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:24:31.379 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.159584+0000", 2024-04-20T17:24:31.379 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010849399999999999 2024-04-20T17:24:31.379 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.379 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.379 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:24:31.379 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.159707+0000", 2024-04-20T17:24:31.379 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012344499999999999 2024-04-20T17:24:31.380 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.380 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.380 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:24:31.380 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.171026+0000", 2024-04-20T17:24:31.380 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.01131892 2024-04-20T17:24:31.380 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.380 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.380 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:31.380 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189843+0000", 2024-04-20T17:24:31.380 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.018816330999999999 2024-04-20T17:24:31.380 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.380 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.380 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:24:31.380 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189889+0000", 2024-04-20T17:24:31.380 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6647999999999998e-05 2024-04-20T17:24:31.381 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.381 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.381 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:24:31.381 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189919+0000", 2024-04-20T17:24:31.381 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9717999999999999e-05 2024-04-20T17:24:31.381 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.381 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:24:31.381 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.381 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.381 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.381 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245523 3.d 3:bc29c29a:::benchmark_data_smithi171_110765_object102308:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:24:31.381 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103692+0000", 2024-04-20T17:24:31.381 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 58.262075418999999, 2024-04-20T17:24:31.381 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.086309881000000005, 2024-04-20T17:24:31.382 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:24:31.382 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:24:31.382 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:31.382 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:24:31.382 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:24:31.382 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:31.382 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245523 2024-04-20T17:24:31.382 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.382 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:24:31.382 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.382 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:24:31.382 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103692+0000", 2024-04-20T17:24:31.383 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.383 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.383 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.383 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:24:31.383 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103692+0000", 2024-04-20T17:24:31.383 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.383 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.383 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.383 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:24:31.383 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103693+0000", 2024-04-20T17:24:31.383 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1710000000000001e-06 2024-04-20T17:24:31.383 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.383 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.383 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:24:31.384 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103699+0000", 2024-04-20T17:24:31.384 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3900000000000001e-06 2024-04-20T17:24:31.384 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.384 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.384 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:24:31.384 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103709+0000", 2024-04-20T17:24:31.384 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9119999999999993e-06 2024-04-20T17:24:31.384 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.384 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.384 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:24:31.384 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103716+0000", 2024-04-20T17:24:31.384 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0260000000000002e-06 2024-04-20T17:24:31.384 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.384 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.384 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.385 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104008+0000", 2024-04-20T17:24:31.385 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00029273899999999998 2024-04-20T17:24:31.385 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.385 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.385 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:24:31.385 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104018+0000", 2024-04-20T17:24:31.385 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5540000000000003e-06 2024-04-20T17:24:31.385 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.385 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.385 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.385 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.161784+0000", 2024-04-20T17:24:31.386 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.057766296000000002 2024-04-20T17:24:31.386 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.386 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.386 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:24:31.386 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.161875+0000", 2024-04-20T17:24:31.386 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0490999999999998e-05 2024-04-20T17:24:31.386 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.386 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.386 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:24:31.386 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.161953+0000", 2024-04-20T17:24:31.386 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8370999999999999e-05 2024-04-20T17:24:31.386 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.386 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.386 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:24:31.387 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.171034+0000", 2024-04-20T17:24:31.387 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0090805629999999998 2024-04-20T17:24:31.387 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.387 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.387 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:31.387 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189952+0000", 2024-04-20T17:24:31.387 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.018918249000000002 2024-04-20T17:24:31.387 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.387 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.387 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:24:31.387 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189980+0000", 2024-04-20T17:24:31.387 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8469000000000001e-05 2024-04-20T17:24:31.387 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.387 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.387 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:24:31.388 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.190002+0000", 2024-04-20T17:24:31.388 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1650000000000001e-05 2024-04-20T17:24:31.388 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.388 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:24:31.388 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.388 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.388 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.388 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245527 3.d 3:bef30612:::benchmark_data_smithi171_110765_object102312:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:24:31.388 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103727+0000", 2024-04-20T17:24:31.388 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 58.262041050999997, 2024-04-20T17:24:31.388 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.090154580999999998, 2024-04-20T17:24:31.388 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:24:31.388 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:24:31.388 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:31.388 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:24:31.389 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:24:31.389 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:31.389 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245527 2024-04-20T17:24:31.389 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.389 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:24:31.389 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.389 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:24:31.389 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103727+0000", 2024-04-20T17:24:31.389 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.389 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.389 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.389 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:24:31.389 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103727+0000", 2024-04-20T17:24:31.389 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.389 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.390 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.390 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:24:31.390 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103727+0000", 2024-04-20T17:24:31.390 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6400000000000001e-07 2024-04-20T17:24:31.390 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.390 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.390 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:24:31.390 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103731+0000", 2024-04-20T17:24:31.390 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7589999999999999e-06 2024-04-20T17:24:31.390 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.390 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.390 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:24:31.390 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103737+0000", 2024-04-20T17:24:31.390 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.207e-06 2024-04-20T17:24:31.391 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.391 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.391 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:24:31.391 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103743+0000", 2024-04-20T17:24:31.391 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9209999999999997e-06 2024-04-20T17:24:31.391 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.391 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.391 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.391 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104112+0000", 2024-04-20T17:24:31.391 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00036842100000000002 2024-04-20T17:24:31.391 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.391 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.391 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:24:31.391 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104116+0000", 2024-04-20T17:24:31.391 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2930000000000002e-06 2024-04-20T17:24:31.392 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.392 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.392 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.392 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.163799+0000", 2024-04-20T17:24:31.392 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.059683369999999999 2024-04-20T17:24:31.392 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.392 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.392 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:24:31.392 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.163872+0000", 2024-04-20T17:24:31.392 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2502999999999993e-05 2024-04-20T17:24:31.392 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.392 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.392 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:24:31.392 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.163953+0000", 2024-04-20T17:24:31.392 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0758000000000002e-05 2024-04-20T17:24:31.393 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.393 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.393 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:24:31.393 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.176271+0000", 2024-04-20T17:24:31.393 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012318610000000001 2024-04-20T17:24:31.393 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.393 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.393 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:31.393 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.193822+0000", 2024-04-20T17:24:31.393 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.017550603000000001 2024-04-20T17:24:31.393 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.393 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.393 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:24:31.393 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.193856+0000", 2024-04-20T17:24:31.393 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4239000000000002e-05 2024-04-20T17:24:31.394 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.394 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.394 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:24:31.394 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.193881+0000", 2024-04-20T17:24:31.394 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5133000000000001e-05 2024-04-20T17:24:31.394 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.394 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:24:31.394 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.394 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.394 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.394 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245531 3.d 3:ba69417f:::benchmark_data_smithi171_110765_object102316:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:24:31.394 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103783+0000", 2024-04-20T17:24:31.394 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 58.261984871000003, 2024-04-20T17:24:31.394 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.090206175, 2024-04-20T17:24:31.394 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:24:31.395 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:24:31.395 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:31.395 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:24:31.395 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:24:31.395 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:31.395 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245531 2024-04-20T17:24:31.395 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.395 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:24:31.395 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.395 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:24:31.395 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103783+0000", 2024-04-20T17:24:31.395 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.395 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.395 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.395 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:24:31.396 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103783+0000", 2024-04-20T17:24:31.396 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.396 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.396 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.396 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:24:31.396 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103785+0000", 2024-04-20T17:24:31.396 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.809e-06 2024-04-20T17:24:31.396 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.396 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.396 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:24:31.396 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103802+0000", 2024-04-20T17:24:31.396 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7075000000000002e-05 2024-04-20T17:24:31.396 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.396 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.397 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:24:31.397 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103815+0000", 2024-04-20T17:24:31.397 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3019e-05 2024-04-20T17:24:31.397 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.397 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.397 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:24:31.397 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103825+0000", 2024-04-20T17:24:31.397 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8309999999999996e-06 2024-04-20T17:24:31.397 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.397 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.397 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.397 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104133+0000", 2024-04-20T17:24:31.397 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00030879900000000001 2024-04-20T17:24:31.397 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.397 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.398 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:24:31.398 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104136+0000", 2024-04-20T17:24:31.398 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0910000000000001e-06 2024-04-20T17:24:31.398 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.398 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.398 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.398 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.165809+0000", 2024-04-20T17:24:31.398 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.061672957 2024-04-20T17:24:31.398 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.398 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.398 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:24:31.398 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.165889+0000", 2024-04-20T17:24:31.398 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9381000000000005e-05 2024-04-20T17:24:31.398 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.399 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.399 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:24:31.399 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.165967+0000", 2024-04-20T17:24:31.399 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7849999999999995e-05 2024-04-20T17:24:31.399 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.399 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.399 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:24:31.399 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.176279+0000", 2024-04-20T17:24:31.399 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010312702999999999 2024-04-20T17:24:31.399 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.399 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.399 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:31.399 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.193918+0000", 2024-04-20T17:24:31.399 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.017638628999999999 2024-04-20T17:24:31.400 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.400 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.400 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:24:31.400 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.193960+0000", 2024-04-20T17:24:31.400 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1746000000000003e-05 2024-04-20T17:24:31.400 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.400 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.400 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:24:31.400 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.193989+0000", 2024-04-20T17:24:31.400 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9285000000000002e-05 2024-04-20T17:24:31.400 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.400 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:24:31.400 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.400 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.400 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.401 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245546 3.d 3:b08c00b6:::benchmark_data_smithi171_110765_object102331:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:24:31.401 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103839+0000", 2024-04-20T17:24:31.401 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 58.261928159999997, 2024-04-20T17:24:31.401 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.097118706999999999, 2024-04-20T17:24:31.401 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:24:31.401 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:24:31.401 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:31.401 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:24:31.401 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:24:31.401 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:31.401 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245546 2024-04-20T17:24:31.401 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.401 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:24:31.401 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.401 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:24:31.402 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103839+0000", 2024-04-20T17:24:31.402 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.402 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.402 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.402 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:24:31.402 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103839+0000", 2024-04-20T17:24:31.402 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.402 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.402 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.402 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:24:31.402 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103840+0000", 2024-04-20T17:24:31.402 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1800000000000004e-07 2024-04-20T17:24:31.402 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.402 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.403 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:24:31.403 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103845+0000", 2024-04-20T17:24:31.403 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3000000000000003e-06 2024-04-20T17:24:31.403 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.403 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.403 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:24:31.403 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103853+0000", 2024-04-20T17:24:31.403 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9799999999999998e-06 2024-04-20T17:24:31.403 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.403 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.403 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:24:31.403 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103859+0000", 2024-04-20T17:24:31.403 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7020000000000004e-06 2024-04-20T17:24:31.403 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.403 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.404 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.404 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104182+0000", 2024-04-20T17:24:31.404 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00032271300000000003 2024-04-20T17:24:31.404 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.404 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.404 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:24:31.404 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104201+0000", 2024-04-20T17:24:31.404 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8547999999999999e-05 2024-04-20T17:24:31.404 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.404 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.404 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.404 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.167891+0000", 2024-04-20T17:24:31.404 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.063690467000000001 2024-04-20T17:24:31.404 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.404 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.405 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:24:31.405 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.168022+0000", 2024-04-20T17:24:31.405 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013114499999999999 2024-04-20T17:24:31.405 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.405 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.405 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:24:31.405 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.168147+0000", 2024-04-20T17:24:31.405 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012454299999999999 2024-04-20T17:24:31.405 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.405 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.405 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:24:31.405 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.181127+0000", 2024-04-20T17:24:31.405 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012980302000000001 2024-04-20T17:24:31.405 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.405 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.406 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:31.406 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.200861+0000", 2024-04-20T17:24:31.406 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.019734266 2024-04-20T17:24:31.406 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.406 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.406 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:24:31.406 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.200910+0000", 2024-04-20T17:24:31.406 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8294999999999998e-05 2024-04-20T17:24:31.406 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.406 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.406 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:24:31.406 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.200958+0000", 2024-04-20T17:24:31.406 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8528e-05 2024-04-20T17:24:31.406 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.406 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:24:31.407 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.407 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.407 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.407 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245552 3.d 3:b5dda02d:::benchmark_data_smithi171_110765_object102337:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:24:31.407 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103886+0000", 2024-04-20T17:24:31.407 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 58.261881311000003, 2024-04-20T17:24:31.407 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.097162344999999997, 2024-04-20T17:24:31.407 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:24:31.407 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:24:31.407 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:31.407 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:24:31.407 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:24:31.407 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:31.407 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245552 2024-04-20T17:24:31.407 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.408 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:24:31.408 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.408 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:24:31.408 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103886+0000", 2024-04-20T17:24:31.408 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.408 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.408 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.408 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:24:31.408 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103886+0000", 2024-04-20T17:24:31.408 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.408 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.408 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.408 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:24:31.408 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103887+0000", 2024-04-20T17:24:31.409 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2799999999999995e-07 2024-04-20T17:24:31.409 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.409 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.409 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:24:31.409 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103890+0000", 2024-04-20T17:24:31.409 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.084e-06 2024-04-20T17:24:31.409 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.409 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.409 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:24:31.409 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103896+0000", 2024-04-20T17:24:31.409 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3750000000000002e-06 2024-04-20T17:24:31.409 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.409 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.409 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:24:31.409 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103900+0000", 2024-04-20T17:24:31.410 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1679999999999997e-06 2024-04-20T17:24:31.410 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.410 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.410 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.410 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104230+0000", 2024-04-20T17:24:31.410 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00033005300000000002 2024-04-20T17:24:31.410 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.410 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.410 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:24:31.410 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104233+0000", 2024-04-20T17:24:31.410 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2260000000000001e-06 2024-04-20T17:24:31.410 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.410 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.410 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.411 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.171080+0000", 2024-04-20T17:24:31.411 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.066846953000000001 2024-04-20T17:24:31.411 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.411 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.411 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:24:31.411 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.171210+0000", 2024-04-20T17:24:31.411 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000130376 2024-04-20T17:24:31.411 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.411 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.411 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:24:31.411 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.171343+0000", 2024-04-20T17:24:31.411 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013265799999999999 2024-04-20T17:24:31.411 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.411 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.412 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:24:31.412 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.181134+0000", 2024-04-20T17:24:31.412 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0097907150000000002 2024-04-20T17:24:31.412 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.412 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.412 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:31.412 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.200992+0000", 2024-04-20T17:24:31.412 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.019858058000000001 2024-04-20T17:24:31.412 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.412 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.412 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:24:31.412 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.201024+0000", 2024-04-20T17:24:31.412 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1914e-05 2024-04-20T17:24:31.412 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.412 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.413 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:24:31.413 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.201049+0000", 2024-04-20T17:24:31.413 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5037000000000001e-05 2024-04-20T17:24:31.413 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.413 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:24:31.413 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.413 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.413 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.413 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245601 3.d 3:be8206df:::benchmark_data_smithi171_110765_object102386:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:24:31.413 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103906+0000", 2024-04-20T17:24:31.413 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 58.261861545000002, 2024-04-20T17:24:31.413 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.10597208700000001, 2024-04-20T17:24:31.413 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:24:31.413 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:24:31.413 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:31.414 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:24:31.414 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:24:31.414 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:31.414 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245601 2024-04-20T17:24:31.414 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.414 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:24:31.414 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.414 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:24:31.414 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103906+0000", 2024-04-20T17:24:31.414 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.414 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.414 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.414 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:24:31.414 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103906+0000", 2024-04-20T17:24:31.415 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.415 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.415 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.415 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:24:31.415 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103906+0000", 2024-04-20T17:24:31.415 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6100000000000002e-07 2024-04-20T17:24:31.415 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.415 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.415 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:24:31.415 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103908+0000", 2024-04-20T17:24:31.415 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7439999999999999e-06 2024-04-20T17:24:31.415 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.415 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.415 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:24:31.415 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103911+0000", 2024-04-20T17:24:31.416 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.954e-06 2024-04-20T17:24:31.416 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.416 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.416 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:24:31.416 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103915+0000", 2024-04-20T17:24:31.416 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9450000000000003e-06 2024-04-20T17:24:31.416 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.416 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.416 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.416 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.174025+0000", 2024-04-20T17:24:31.416 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.070109677999999995 2024-04-20T17:24:31.416 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.416 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.416 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:24:31.416 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.174130+0000", 2024-04-20T17:24:31.417 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010515599999999999 2024-04-20T17:24:31.417 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.417 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.417 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:24:31.417 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.174204+0000", 2024-04-20T17:24:31.417 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4097000000000006e-05 2024-04-20T17:24:31.417 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.417 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.417 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:24:31.417 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.181429+0000", 2024-04-20T17:24:31.417 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0072249250000000001 2024-04-20T17:24:31.417 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.417 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.417 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:31.417 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.209781+0000", 2024-04-20T17:24:31.418 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.028351694 2024-04-20T17:24:31.418 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.418 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.418 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:24:31.418 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.209834+0000", 2024-04-20T17:24:31.418 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3309000000000001e-05 2024-04-20T17:24:31.418 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.418 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.418 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:24:31.418 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.209878+0000", 2024-04-20T17:24:31.418 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4224000000000002e-05 2024-04-20T17:24:31.418 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.418 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:24:31.418 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.419 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.419 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.419 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245611 3.d 3:b89d9251:::benchmark_data_smithi171_110765_object102396:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:24:31.419 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103921+0000", 2024-04-20T17:24:31.419 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 58.261846353999999, 2024-04-20T17:24:31.419 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.106082866, 2024-04-20T17:24:31.419 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:24:31.419 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:24:31.419 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:31.419 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:24:31.419 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:24:31.419 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:31.419 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245611 2024-04-20T17:24:31.419 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.419 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:24:31.420 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.420 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:24:31.420 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103921+0000", 2024-04-20T17:24:31.420 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.420 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.420 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.420 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:24:31.420 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103921+0000", 2024-04-20T17:24:31.420 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.420 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.420 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.420 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:24:31.420 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103922+0000", 2024-04-20T17:24:31.420 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1999999999999999e-07 2024-04-20T17:24:31.420 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.421 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.421 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:24:31.421 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103926+0000", 2024-04-20T17:24:31.421 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3630000000000004e-06 2024-04-20T17:24:31.421 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.421 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.421 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:24:31.421 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103932+0000", 2024-04-20T17:24:31.421 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8259999999999998e-06 2024-04-20T17:24:31.421 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.421 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.421 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:24:31.421 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103948+0000", 2024-04-20T17:24:31.421 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6240999999999999e-05 2024-04-20T17:24:31.421 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.422 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.422 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.422 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.176327+0000", 2024-04-20T17:24:31.422 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.072379053999999998 2024-04-20T17:24:31.422 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.422 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.422 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:24:31.422 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.176443+0000", 2024-04-20T17:24:31.422 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011536600000000001 2024-04-20T17:24:31.422 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.422 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.422 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:24:31.422 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.176568+0000", 2024-04-20T17:24:31.422 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000125184 2024-04-20T17:24:31.423 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.423 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.423 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:24:31.423 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.181432+0000", 2024-04-20T17:24:31.423 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0048639720000000003 2024-04-20T17:24:31.423 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.423 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.423 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:31.423 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.209927+0000", 2024-04-20T17:24:31.423 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.028494789999999999 2024-04-20T17:24:31.423 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.423 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.423 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:24:31.423 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.209969+0000", 2024-04-20T17:24:31.424 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1773000000000001e-05 2024-04-20T17:24:31.424 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.424 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.424 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:24:31.424 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.210004+0000", 2024-04-20T17:24:31.424 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5577000000000002e-05 2024-04-20T17:24:31.424 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.424 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:24:31.424 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.424 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.424 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.424 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245620 3.d 3:be474e21:::benchmark_data_smithi171_110765_object102405:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:24:31.424 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103957+0000", 2024-04-20T17:24:31.424 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 58.261810355999998, 2024-04-20T17:24:31.424 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.112691712, 2024-04-20T17:24:31.425 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:24:31.425 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:24:31.425 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:31.425 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:24:31.425 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:24:31.425 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:31.425 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245620 2024-04-20T17:24:31.425 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.425 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:24:31.425 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.425 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:24:31.425 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103957+0000", 2024-04-20T17:24:31.425 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.425 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.425 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.426 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:24:31.426 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103957+0000", 2024-04-20T17:24:31.426 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.426 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.426 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.426 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:24:31.426 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103958+0000", 2024-04-20T17:24:31.426 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1299999999999999e-07 2024-04-20T17:24:31.426 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.426 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.426 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:24:31.426 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103962+0000", 2024-04-20T17:24:31.426 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6119999999999999e-06 2024-04-20T17:24:31.426 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.427 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.427 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:24:31.427 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103967+0000", 2024-04-20T17:24:31.427 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7679999999999997e-06 2024-04-20T17:24:31.427 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.427 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.427 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:24:31.427 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103971+0000", 2024-04-20T17:24:31.427 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6100000000000002e-06 2024-04-20T17:24:31.427 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.427 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.427 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.427 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.178796+0000", 2024-04-20T17:24:31.427 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.074824511999999996 2024-04-20T17:24:31.427 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.428 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.428 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:24:31.428 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.178903+0000", 2024-04-20T17:24:31.428 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000107731 2024-04-20T17:24:31.428 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.428 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.428 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:24:31.428 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.179004+0000", 2024-04-20T17:24:31.428 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000101139 2024-04-20T17:24:31.428 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.428 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.428 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:24:31.428 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.187634+0000", 2024-04-20T17:24:31.428 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0086293210000000006 2024-04-20T17:24:31.429 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.429 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.429 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:31.429 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.216547+0000", 2024-04-20T17:24:31.429 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.028912817 2024-04-20T17:24:31.429 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.429 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.429 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:24:31.429 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.216604+0000", 2024-04-20T17:24:31.429 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7009999999999998e-05 2024-04-20T17:24:31.429 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.429 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.429 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:24:31.429 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.216649+0000", 2024-04-20T17:24:31.430 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5379e-05 2024-04-20T17:24:31.430 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.430 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:24:31.430 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.430 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.430 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.430 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245622 3.d 3:b177e667:::benchmark_data_smithi171_110765_object102407:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:24:31.430 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103975+0000", 2024-04-20T17:24:31.430 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 58.261792247000002, 2024-04-20T17:24:31.430 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.112781889, 2024-04-20T17:24:31.430 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:24:31.430 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:24:31.430 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:31.430 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:24:31.430 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:24:31.431 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:31.431 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245622 2024-04-20T17:24:31.431 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.431 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:24:31.431 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.431 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:24:31.431 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103975+0000", 2024-04-20T17:24:31.431 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.431 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.431 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.431 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:24:31.431 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103975+0000", 2024-04-20T17:24:31.431 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.431 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.431 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.432 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:24:31.432 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103976+0000", 2024-04-20T17:24:31.432 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9000000000000002e-07 2024-04-20T17:24:31.432 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.432 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.432 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:24:31.432 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103978+0000", 2024-04-20T17:24:31.432 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.821e-06 2024-04-20T17:24:31.432 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.432 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.432 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:24:31.432 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103981+0000", 2024-04-20T17:24:31.432 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0350000000000002e-06 2024-04-20T17:24:31.432 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.432 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.433 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:24:31.433 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103984+0000", 2024-04-20T17:24:31.433 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0089999999999999e-06 2024-04-20T17:24:31.433 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.433 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.433 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.433 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.181477+0000", 2024-04-20T17:24:31.433 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.077492970999999994 2024-04-20T17:24:31.433 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.433 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.433 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:24:31.433 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.181593+0000", 2024-04-20T17:24:31.433 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000115979 2024-04-20T17:24:31.433 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.433 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.434 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:24:31.434 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.181711+0000", 2024-04-20T17:24:31.434 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000118245 2024-04-20T17:24:31.434 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.434 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.434 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:24:31.434 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189686+0000", 2024-04-20T17:24:31.434 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0079754270000000002 2024-04-20T17:24:31.434 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.434 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.434 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:31.434 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.216686+0000", 2024-04-20T17:24:31.434 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.026999433 2024-04-20T17:24:31.434 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.435 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.435 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:24:31.435 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.216718+0000", 2024-04-20T17:24:31.435 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2570000000000002e-05 2024-04-20T17:24:31.435 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.435 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.435 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:24:31.435 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.216757+0000", 2024-04-20T17:24:31.435 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9008999999999999e-05 2024-04-20T17:24:31.435 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.435 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:24:31.435 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.435 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.435 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.435 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245623 3.d 3:becddf3a:::benchmark_data_smithi171_110765_object102408:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:24:31.436 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103997+0000", 2024-04-20T17:24:31.436 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 58.261771107999998, 2024-04-20T17:24:31.436 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.11723939999999999, 2024-04-20T17:24:31.436 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:24:31.436 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:24:31.436 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:31.436 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:24:31.436 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:24:31.436 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:31.436 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245623 2024-04-20T17:24:31.436 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.436 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:24:31.436 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.436 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:24:31.436 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103997+0000", 2024-04-20T17:24:31.437 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.437 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.437 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.437 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:24:31.437 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103997+0000", 2024-04-20T17:24:31.437 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.437 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.437 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.437 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:24:31.437 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103997+0000", 2024-04-20T17:24:31.437 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8099999999999998e-07 2024-04-20T17:24:31.437 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.437 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.437 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:24:31.437 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103999+0000", 2024-04-20T17:24:31.438 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6929999999999999e-06 2024-04-20T17:24:31.438 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.438 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.438 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:24:31.438 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104008+0000", 2024-04-20T17:24:31.438 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0310000000000004e-06 2024-04-20T17:24:31.438 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.438 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.438 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:24:31.438 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104011+0000", 2024-04-20T17:24:31.438 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8060000000000001e-06 2024-04-20T17:24:31.438 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.438 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.438 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.439 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.185112+0000", 2024-04-20T17:24:31.439 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.081100835999999996 2024-04-20T17:24:31.439 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.439 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.439 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:24:31.439 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.185198+0000", 2024-04-20T17:24:31.439 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6045999999999996e-05 2024-04-20T17:24:31.439 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.439 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.439 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:24:31.439 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.185274+0000", 2024-04-20T17:24:31.439 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6091e-05 2024-04-20T17:24:31.439 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.439 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.439 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:24:31.440 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189787+0000", 2024-04-20T17:24:31.440 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045125410000000001 2024-04-20T17:24:31.440 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.440 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.440 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:31.440 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.221137+0000", 2024-04-20T17:24:31.440 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.031350204 2024-04-20T17:24:31.440 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.440 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.440 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:24:31.440 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.221185+0000", 2024-04-20T17:24:31.440 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7410000000000002e-05 2024-04-20T17:24:31.440 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.440 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.441 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:24:31.441 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.221236+0000", 2024-04-20T17:24:31.441 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1360999999999998e-05 2024-04-20T17:24:31.441 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.441 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:24:31.441 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.441 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.441 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.441 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245630 3.d 3:b005673c:::benchmark_data_smithi171_110765_object102415:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:24:31.441 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.104016+0000", 2024-04-20T17:24:31.441 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 58.261751916999998, 2024-04-20T17:24:31.441 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.117355654, 2024-04-20T17:24:31.441 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:24:31.441 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:24:31.442 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:31.442 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:24:31.442 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:24:31.442 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:31.442 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245630 2024-04-20T17:24:31.442 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.442 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:24:31.442 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.442 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:24:31.442 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104016+0000", 2024-04-20T17:24:31.442 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.442 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.442 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.442 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:24:31.442 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104016+0000", 2024-04-20T17:24:31.443 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.443 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.443 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.443 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:24:31.443 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104016+0000", 2024-04-20T17:24:31.443 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5100000000000001e-07 2024-04-20T17:24:31.443 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.443 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.443 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:24:31.443 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104018+0000", 2024-04-20T17:24:31.443 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2409999999999998e-06 2024-04-20T17:24:31.443 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.443 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.443 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:24:31.443 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104022+0000", 2024-04-20T17:24:31.444 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8340000000000001e-06 2024-04-20T17:24:31.444 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.444 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.444 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:24:31.444 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104027+0000", 2024-04-20T17:24:31.444 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.369e-06 2024-04-20T17:24:31.444 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.444 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.444 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.444 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.187654+0000", 2024-04-20T17:24:31.444 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.083627286999999995 2024-04-20T17:24:31.444 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.444 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.444 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:24:31.444 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.187717+0000", 2024-04-20T17:24:31.445 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3417999999999995e-05 2024-04-20T17:24:31.445 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.445 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.445 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:24:31.445 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.187783+0000", 2024-04-20T17:24:31.445 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5574000000000006e-05 2024-04-20T17:24:31.445 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.445 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.445 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:24:31.445 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.190010+0000", 2024-04-20T17:24:31.445 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022267150000000002 2024-04-20T17:24:31.445 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.445 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.445 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:31.445 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.221276+0000", 2024-04-20T17:24:31.446 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.031266250000000002 2024-04-20T17:24:31.446 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.446 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.446 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:24:31.446 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.221332+0000", 2024-04-20T17:24:31.446 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6298999999999997e-05 2024-04-20T17:24:31.446 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.446 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.446 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:24:31.446 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.221371+0000", 2024-04-20T17:24:31.446 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9316000000000001e-05 2024-04-20T17:24:31.446 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.446 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:24:31.446 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.446 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.447 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.447 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245803 3.f 3:fc19cf94:::benchmark_data_smithi171_110765_object102588:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:24:31.447 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:34.105692+0000", 2024-04-20T17:24:31.447 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 57.260075942999997, 2024-04-20T17:24:31.447 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.081553284000000004, 2024-04-20T17:24:31.447 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:24:31.447 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:24:31.447 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:31.447 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:24:31.447 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:24:31.447 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:31.447 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245803 2024-04-20T17:24:31.447 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.447 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:24:31.447 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.448 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:24:31.448 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105692+0000", 2024-04-20T17:24:31.448 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.448 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.448 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.448 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:24:31.448 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105692+0000", 2024-04-20T17:24:31.448 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.448 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.448 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.448 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:24:31.448 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105692+0000", 2024-04-20T17:24:31.448 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2199999999999999e-07 2024-04-20T17:24:31.448 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.449 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.449 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:24:31.449 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105694+0000", 2024-04-20T17:24:31.449 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.593e-06 2024-04-20T17:24:31.449 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.449 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.449 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:24:31.449 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105697+0000", 2024-04-20T17:24:31.449 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3060000000000002e-06 2024-04-20T17:24:31.449 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.449 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.449 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:24:31.449 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105700+0000", 2024-04-20T17:24:31.449 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8430000000000001e-06 2024-04-20T17:24:31.449 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.450 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.450 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.450 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105794+0000", 2024-04-20T17:24:31.450 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3948000000000002e-05 2024-04-20T17:24:31.450 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.450 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.450 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:24:31.450 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105797+0000", 2024-04-20T17:24:31.450 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6370000000000001e-06 2024-04-20T17:24:31.450 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.450 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.450 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.450 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.163315+0000", 2024-04-20T17:24:31.450 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.057517243000000003 2024-04-20T17:24:31.450 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.451 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.451 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:24:31.451 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.163373+0000", 2024-04-20T17:24:31.451 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8399000000000001e-05 2024-04-20T17:24:31.451 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.451 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.451 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 11", 2024-04-20T17:24:31.451 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.163436+0000", 2024-04-20T17:24:31.451 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2589999999999995e-05 2024-04-20T17:24:31.451 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.451 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.451 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:24:31.451 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.172663+0000", 2024-04-20T17:24:31.451 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0092275810000000003 2024-04-20T17:24:31.452 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.452 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.452 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:31.452 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.187151+0000", 2024-04-20T17:24:31.452 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.014487635 2024-04-20T17:24:31.452 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.452 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.452 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:24:31.452 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.187204+0000", 2024-04-20T17:24:31.452 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2793000000000001e-05 2024-04-20T17:24:31.452 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.452 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.452 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:24:31.452 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.187245+0000", 2024-04-20T17:24:31.452 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1294000000000001e-05 2024-04-20T17:24:31.453 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.453 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:24:31.453 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.453 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.453 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.453 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245820 3.f 3:f94b0a75:::benchmark_data_smithi171_110765_object102605:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:24:31.453 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:34.105704+0000", 2024-04-20T17:24:31.453 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 57.260064092, 2024-04-20T17:24:31.453 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.084848861999999997, 2024-04-20T17:24:31.453 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:24:31.453 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:24:31.453 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:31.453 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:24:31.453 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:24:31.454 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:31.454 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245820 2024-04-20T17:24:31.454 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.454 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:24:31.454 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.454 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:24:31.454 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105704+0000", 2024-04-20T17:24:31.454 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.454 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.454 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.454 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:24:31.454 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105704+0000", 2024-04-20T17:24:31.454 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.454 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.455 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.455 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:24:31.455 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105704+0000", 2024-04-20T17:24:31.455 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1e-07 2024-04-20T17:24:31.455 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.455 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.455 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:24:31.455 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105710+0000", 2024-04-20T17:24:31.455 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6890000000000001e-06 2024-04-20T17:24:31.455 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.455 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.455 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:24:31.455 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105713+0000", 2024-04-20T17:24:31.455 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.061e-06 2024-04-20T17:24:31.455 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.456 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.456 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:24:31.456 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105716+0000", 2024-04-20T17:24:31.456 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7780000000000001e-06 2024-04-20T17:24:31.456 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.456 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.456 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.456 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105816+0000", 2024-04-20T17:24:31.456 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9381000000000003e-05 2024-04-20T17:24:31.456 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.456 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.456 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:24:31.456 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105819+0000", 2024-04-20T17:24:31.456 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9689999999999999e-06 2024-04-20T17:24:31.456 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.457 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.457 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.457 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.165004+0000", 2024-04-20T17:24:31.457 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.059185298999999997 2024-04-20T17:24:31.457 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.457 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.457 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:24:31.457 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.165065+0000", 2024-04-20T17:24:31.457 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1136000000000006e-05 2024-04-20T17:24:31.457 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.457 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.457 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 11", 2024-04-20T17:24:31.457 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.165126+0000", 2024-04-20T17:24:31.457 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0863999999999999e-05 2024-04-20T17:24:31.457 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.458 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.458 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:24:31.458 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.172669+0000", 2024-04-20T17:24:31.458 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.007543396 2024-04-20T17:24:31.458 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.458 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.458 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:31.458 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.190456+0000", 2024-04-20T17:24:31.458 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.017786309 2024-04-20T17:24:31.458 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.458 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.458 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:24:31.458 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.190507+0000", 2024-04-20T17:24:31.458 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1102e-05 2024-04-20T17:24:31.459 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.459 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.459 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:24:31.459 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.190552+0000", 2024-04-20T17:24:31.459 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5568000000000001e-05 2024-04-20T17:24:31.459 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.459 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:24:31.459 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.459 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.459 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.459 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245821 3.f 3:f7671ae3:::benchmark_data_smithi171_110765_object102606:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:24:31.459 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:34.105727+0000", 2024-04-20T17:24:31.459 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 57.260040365999998, 2024-04-20T17:24:31.459 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.084930124999999995, 2024-04-20T17:24:31.459 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:24:31.459 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:24:31.460 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:31.460 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:24:31.460 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:24:31.460 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:31.460 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245821 2024-04-20T17:24:31.460 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.460 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:24:31.460 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.460 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:24:31.460 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105727+0000", 2024-04-20T17:24:31.460 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.460 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.460 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.460 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:24:31.461 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105727+0000", 2024-04-20T17:24:31.461 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.461 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.461 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.461 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:24:31.461 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105728+0000", 2024-04-20T17:24:31.461 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1699999999999999e-07 2024-04-20T17:24:31.461 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.461 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.461 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:24:31.461 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105729+0000", 2024-04-20T17:24:31.461 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4640000000000001e-06 2024-04-20T17:24:31.461 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.461 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.462 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:24:31.462 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105733+0000", 2024-04-20T17:24:31.462 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5860000000000001e-06 2024-04-20T17:24:31.462 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.462 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.462 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:24:31.462 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105735+0000", 2024-04-20T17:24:31.462 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7829999999999999e-06 2024-04-20T17:24:31.462 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.462 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.462 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.462 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105836+0000", 2024-04-20T17:24:31.462 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000100574 2024-04-20T17:24:31.462 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.462 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.463 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:24:31.463 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105841+0000", 2024-04-20T17:24:31.463 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.729e-06 2024-04-20T17:24:31.463 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.463 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.463 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.463 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.166821+0000", 2024-04-20T17:24:31.463 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.060979836000000003 2024-04-20T17:24:31.463 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.463 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.463 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:24:31.463 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.166913+0000", 2024-04-20T17:24:31.463 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2724000000000002e-05 2024-04-20T17:24:31.463 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.463 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.464 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 11", 2024-04-20T17:24:31.464 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.167000+0000", 2024-04-20T17:24:31.464 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6802e-05 2024-04-20T17:24:31.464 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.464 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.464 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:24:31.464 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.174876+0000", 2024-04-20T17:24:31.464 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0078754649999999999 2024-04-20T17:24:31.464 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.464 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.464 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:31.464 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.190587+0000", 2024-04-20T17:24:31.464 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015711573 2024-04-20T17:24:31.464 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.465 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.465 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:24:31.465 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.190626+0000", 2024-04-20T17:24:31.465 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8909999999999998e-05 2024-04-20T17:24:31.465 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.465 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.465 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:24:31.465 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.190657+0000", 2024-04-20T17:24:31.465 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1362000000000002e-05 2024-04-20T17:24:31.465 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.465 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:24:31.465 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.465 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.465 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.465 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245828 3.f 3:f231c1d2:::benchmark_data_smithi171_110765_object102613:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:24:31.466 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:34.105739+0000", 2024-04-20T17:24:31.466 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 57.260028441999999, 2024-04-20T17:24:31.466 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.085013469999999994, 2024-04-20T17:24:31.466 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:24:31.466 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:24:31.466 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:31.466 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:24:31.466 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:24:31.466 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:31.466 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245828 2024-04-20T17:24:31.466 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.466 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:24:31.466 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.466 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:24:31.466 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105739+0000", 2024-04-20T17:24:31.467 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.467 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.467 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.467 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:24:31.467 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105739+0000", 2024-04-20T17:24:31.467 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.467 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.467 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.467 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:24:31.467 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105740+0000", 2024-04-20T17:24:31.467 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1e-07 2024-04-20T17:24:31.467 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.467 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.467 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:24:31.468 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105741+0000", 2024-04-20T17:24:31.468 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.221e-06 2024-04-20T17:24:31.468 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.468 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.468 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:24:31.468 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105744+0000", 2024-04-20T17:24:31.468 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.892e-06 2024-04-20T17:24:31.468 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.468 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.468 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:24:31.468 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105746+0000", 2024-04-20T17:24:31.468 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6840000000000001e-06 2024-04-20T17:24:31.468 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.468 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.468 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.469 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105858+0000", 2024-04-20T17:24:31.469 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011199199999999999 2024-04-20T17:24:31.469 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.469 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.469 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:24:31.469 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105861+0000", 2024-04-20T17:24:31.469 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9440000000000001e-06 2024-04-20T17:24:31.469 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.469 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.469 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.469 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.168822+0000", 2024-04-20T17:24:31.469 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.062960753999999994 2024-04-20T17:24:31.469 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.469 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.469 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:24:31.470 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.168895+0000", 2024-04-20T17:24:31.470 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2817000000000004e-05 2024-04-20T17:24:31.470 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.470 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.470 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 11", 2024-04-20T17:24:31.470 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.168969+0000", 2024-04-20T17:24:31.470 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4091000000000006e-05 2024-04-20T17:24:31.470 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.470 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.470 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:24:31.470 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.174881+0000", 2024-04-20T17:24:31.470 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0059125870000000004 2024-04-20T17:24:31.470 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.470 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.470 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:31.471 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.190687+0000", 2024-04-20T17:24:31.471 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015805475999999999 2024-04-20T17:24:31.471 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.471 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.471 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:24:31.471 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.190724+0000", 2024-04-20T17:24:31.471 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6738e-05 2024-04-20T17:24:31.471 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.471 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.471 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:24:31.471 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.190753+0000", 2024-04-20T17:24:31.471 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8963999999999999e-05 2024-04-20T17:24:31.471 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.471 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:24:31.471 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.472 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.472 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.472 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245831 3.f 3:fd5d268b:::benchmark_data_smithi171_110765_object102616:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:24:31.472 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:34.105750+0000", 2024-04-20T17:24:31.472 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 57.260017814000001, 2024-04-20T17:24:31.472 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.090378368000000001, 2024-04-20T17:24:31.472 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:24:31.472 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:24:31.472 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:31.472 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:24:31.472 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:24:31.472 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:31.472 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245831 2024-04-20T17:24:31.472 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.472 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:24:31.473 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.473 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:24:31.473 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105750+0000", 2024-04-20T17:24:31.473 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.473 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.473 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.473 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:24:31.473 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105750+0000", 2024-04-20T17:24:31.473 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.473 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.473 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.473 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:24:31.473 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105750+0000", 2024-04-20T17:24:31.473 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7000000000000001e-07 2024-04-20T17:24:31.474 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.474 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.474 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:24:31.474 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105751+0000", 2024-04-20T17:24:31.474 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.091e-06 2024-04-20T17:24:31.474 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.474 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.474 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:24:31.474 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105754+0000", 2024-04-20T17:24:31.474 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.559e-06 2024-04-20T17:24:31.474 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.474 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.474 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:24:31.474 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105757+0000", 2024-04-20T17:24:31.474 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.819e-06 2024-04-20T17:24:31.475 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.475 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.475 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.475 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105878+0000", 2024-04-20T17:24:31.475 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000121839 2024-04-20T17:24:31.475 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.475 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.475 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:24:31.475 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105882+0000", 2024-04-20T17:24:31.475 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.224e-06 2024-04-20T17:24:31.475 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.475 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.475 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.475 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.170692+0000", 2024-04-20T17:24:31.475 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.064810693000000003 2024-04-20T17:24:31.476 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.476 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.476 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:24:31.476 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.170783+0000", 2024-04-20T17:24:31.476 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0681999999999997e-05 2024-04-20T17:24:31.476 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.476 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.476 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 11", 2024-04-20T17:24:31.476 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.170858+0000", 2024-04-20T17:24:31.476 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5333999999999994e-05 2024-04-20T17:24:31.476 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.476 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.476 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:24:31.476 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.175098+0000", 2024-04-20T17:24:31.477 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.004239278 2024-04-20T17:24:31.477 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.477 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.477 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:31.477 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.196021+0000", 2024-04-20T17:24:31.477 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.020923529 2024-04-20T17:24:31.477 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.477 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.477 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:24:31.477 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.196081+0000", 2024-04-20T17:24:31.477 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0188e-05 2024-04-20T17:24:31.477 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.477 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.477 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:24:31.477 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.196128+0000", 2024-04-20T17:24:31.478 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6862e-05 2024-04-20T17:24:31.478 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.478 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:24:31.478 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.478 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.478 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.478 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245857 3.f 3:f1991b6e:::benchmark_data_smithi171_110765_object102642:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:24:31.478 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:34.105770+0000", 2024-04-20T17:24:31.478 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 57.259997284999997, 2024-04-20T17:24:31.478 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.091072343, 2024-04-20T17:24:31.478 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:24:31.478 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:24:31.478 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:31.478 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:24:31.478 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:24:31.479 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:31.479 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245857 2024-04-20T17:24:31.479 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.479 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:24:31.479 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.479 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:24:31.479 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105770+0000", 2024-04-20T17:24:31.479 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.479 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.479 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.479 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:24:31.479 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105770+0000", 2024-04-20T17:24:31.479 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.479 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.480 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.480 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:24:31.480 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105771+0000", 2024-04-20T17:24:31.480 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.84e-07 2024-04-20T17:24:31.480 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.480 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.480 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:24:31.480 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105772+0000", 2024-04-20T17:24:31.480 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3939999999999999e-06 2024-04-20T17:24:31.480 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.480 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.480 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:24:31.480 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105776+0000", 2024-04-20T17:24:31.480 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8550000000000004e-06 2024-04-20T17:24:31.480 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.481 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.481 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:24:31.481 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105779+0000", 2024-04-20T17:24:31.481 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.768e-06 2024-04-20T17:24:31.481 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.481 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.481 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.481 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105899+0000", 2024-04-20T17:24:31.481 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012002600000000001 2024-04-20T17:24:31.481 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.481 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.481 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:24:31.481 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105902+0000", 2024-04-20T17:24:31.481 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9079999999999999e-06 2024-04-20T17:24:31.482 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.482 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.482 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.482 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.172714+0000", 2024-04-20T17:24:31.482 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.066812508000000007 2024-04-20T17:24:31.482 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.482 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.482 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:24:31.482 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.172829+0000", 2024-04-20T17:24:31.482 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000114436 2024-04-20T17:24:31.482 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.482 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.482 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 11", 2024-04-20T17:24:31.482 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.172951+0000", 2024-04-20T17:24:31.482 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012268899999999999 2024-04-20T17:24:31.483 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.483 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.483 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:24:31.483 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.177495+0000", 2024-04-20T17:24:31.483 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045431789999999996 2024-04-20T17:24:31.483 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.483 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.483 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:31.483 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.196749+0000", 2024-04-20T17:24:31.483 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.019254060999999999 2024-04-20T17:24:31.483 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.483 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.483 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:24:31.483 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.196800+0000", 2024-04-20T17:24:31.483 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1209999999999999e-05 2024-04-20T17:24:31.484 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.484 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.484 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:24:31.484 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.196843+0000", 2024-04-20T17:24:31.484 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2925e-05 2024-04-20T17:24:31.484 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.484 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:24:31.484 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.484 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.484 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.484 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245891 3.f 3:fe0f9ce5:::benchmark_data_smithi171_110765_object102676:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:24:31.484 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:34.105792+0000", 2024-04-20T17:24:31.484 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 57.259975984999997, 2024-04-20T17:24:31.484 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.091162981000000004, 2024-04-20T17:24:31.484 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:24:31.485 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:24:31.485 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:31.485 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:24:31.485 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:24:31.485 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:31.485 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245891 2024-04-20T17:24:31.485 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.485 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:24:31.485 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.485 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:24:31.485 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105792+0000", 2024-04-20T17:24:31.485 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.485 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.485 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.485 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:24:31.485 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105792+0000", 2024-04-20T17:24:31.486 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:24:31.486 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.486 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.486 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:24:31.486 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105792+0000", 2024-04-20T17:24:31.486 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9400000000000001e-07 2024-04-20T17:24:31.486 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.486 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.486 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:24:31.486 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105793+0000", 2024-04-20T17:24:31.486 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.398e-06 2024-04-20T17:24:31.486 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.486 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.486 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:24:31.487 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105796+0000", 2024-04-20T17:24:31.487 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.959e-06 2024-04-20T17:24:31.487 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.487 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.487 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:24:31.487 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105799+0000", 2024-04-20T17:24:31.487 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7719999999999999e-06 2024-04-20T17:24:31.487 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.487 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.487 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.487 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105913+0000", 2024-04-20T17:24:31.487 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011384599999999999 2024-04-20T17:24:31.487 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.487 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.487 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:24:31.487 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105916+0000", 2024-04-20T17:24:31.488 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0110000000000001e-06 2024-04-20T17:24:31.488 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.488 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.488 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:24:31.488 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.175112+0000", 2024-04-20T17:24:31.488 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.069195997999999995 2024-04-20T17:24:31.488 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.488 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.488 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:24:31.488 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.175190+0000", 2024-04-20T17:24:31.488 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7616999999999999e-05 2024-04-20T17:24:31.488 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.488 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.488 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 11", 2024-04-20T17:24:31.489 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.175261+0000", 2024-04-20T17:24:31.489 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1484999999999997e-05 2024-04-20T17:24:31.489 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.489 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.489 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:24:31.489 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.177849+0000", 2024-04-20T17:24:31.489 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0025879589999999999 2024-04-20T17:24:31.489 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.489 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.489 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:31.489 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.196883+0000", 2024-04-20T17:24:31.489 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.019034253000000001 2024-04-20T17:24:31.489 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.489 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.489 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:24:31.489 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.196916+0000", 2024-04-20T17:24:31.490 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2586999999999998e-05 2024-04-20T17:24:31.490 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:24:31.490 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:24:31.490 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:24:31.490 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.196955+0000", 2024-04-20T17:24:31.490 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8801999999999999e-05 2024-04-20T17:24:31.490 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.490 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:24:31.490 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.490 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:24:31.490 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:24:31.490 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:24:31.491 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:24:31.517 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:31.536 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:24:31.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:31.539 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:24:31.640 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:24:31.669 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:31.698 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:24:31.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:31 smithi106 ceph-mon[44423]: pgmap v1183: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:31.794 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:24:31.800 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:31 smithi005 ceph-mon[39562]: pgmap v1183: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:31.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:31.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:24:31.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:31 smithi171 ceph-mon[44189]: pgmap v1183: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:31.902 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:24:31.932 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:32.182 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:24:32.184 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-20T17:24:32.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:32.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:24:32.243 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:24:32.286 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:24:32.315 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:32.334 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:24:32.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:32.338 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:24:32.439 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:24:32.467 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:32.485 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:24:32.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:32.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:24:32.547 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.1 to 1 (10000) 2024-04-20T17:24:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:32 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:24:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:32 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:24:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:32 smithi171 ceph-mon[44189]: osdmap e442: 12 total, 12 up, 12 in 2024-04-20T17:24:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:32 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/661173973' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:24:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:32 smithi171 ceph-mon[44189]: from='client.26038 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:24:32.589 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:24:32.617 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:32.636 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:24:32.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:32.639 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:24:32.740 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:24:32.741 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:32 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:24:32.741 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:32 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:24:32.742 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:32 smithi005 ceph-mon[39562]: osdmap e442: 12 total, 12 up, 12 in 2024-04-20T17:24:32.742 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:32 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/661173973' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:24:32.742 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:32 smithi005 ceph-mon[39562]: from='client.26038 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:24:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:32 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:24:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:32 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:24:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:32 smithi106 ceph-mon[44423]: osdmap e442: 12 total, 12 up, 12 in 2024-04-20T17:24:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:32 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/661173973' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:24:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:32 smithi106 ceph-mon[44423]: from='client.26038 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:24:32.767 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:32.786 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:24:32.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:32.788 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:24:32.889 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:24:32.916 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:32.935 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:24:32.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:32.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:24:33.040 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:24:33.069 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:33.485 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 2 1 2024-04-20T17:24:33.519 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:24:33.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:33.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:24:33.623 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:24:33.656 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:33.677 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:24:33.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:33.680 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:24:33.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:33 smithi005 ceph-mon[39562]: pgmap v1185: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:33.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:33 smithi005 ceph-mon[39562]: from='client.26038 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-20T17:24:33.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:33 smithi005 ceph-mon[39562]: osdmap e443: 12 total, 12 up, 12 in 2024-04-20T17:24:33.780 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:24:33.811 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:33 smithi106 ceph-mon[44423]: pgmap v1185: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:33.811 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:33 smithi106 ceph-mon[44423]: from='client.26038 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-20T17:24:33.811 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:33 smithi106 ceph-mon[44423]: osdmap e443: 12 total, 12 up, 12 in 2024-04-20T17:24:33.813 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:33.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:33 smithi171 ceph-mon[44189]: pgmap v1185: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:33.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:33 smithi171 ceph-mon[44189]: from='client.26038 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-20T17:24:33.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:33 smithi171 ceph-mon[44189]: osdmap e443: 12 total, 12 up, 12 in 2024-04-20T17:24:33.834 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:24:33.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:33.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:24:33.939 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:24:33.968 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:33.988 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:24:33.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:33.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:24:34.092 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:24:34.122 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:34.220 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:24:34.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:34.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:24:34.325 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:24:34.352 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:34.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:34 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:24:34.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:34 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:24:34.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:34 smithi171 ceph-mon[44189]: osdmap e444: 12 total, 12 up, 12 in 2024-04-20T17:24:34.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:34 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:24:34.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:34 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:24:34.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:34 smithi005 ceph-mon[39562]: osdmap e444: 12 total, 12 up, 12 in 2024-04-20T17:24:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:34 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:24:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:34 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:24:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:34 smithi106 ceph-mon[44423]: osdmap e444: 12 total, 12 up, 12 in 2024-04-20T17:24:35.234 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:24:35.275 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:24:35.276 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:24:35.502 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:24:35.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:35.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:24:35.565 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'unique_pool_2' created 2024-04-20T17:24:35.609 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:24:35.641 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:35.663 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:24:35.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:35.666 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:24:35.767 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:24:35.773 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:35 smithi005 ceph-mon[39562]: pgmap v1188: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:35.773 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:35 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3891565634' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-04-20T17:24:35.796 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:35.817 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:24:35.818 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:35 smithi171 ceph-mon[44189]: pgmap v1188: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:35.818 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:35 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3891565634' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-04-20T17:24:35.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:35.820 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:24:35.921 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:24:35.949 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:35.968 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:24:35.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:35.971 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:24:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:35 smithi106 ceph-mon[44423]: pgmap v1188: 1 pgs: 1 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:35 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3891565634' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-04-20T17:24:36.071 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:24:36.083 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:24:36.102 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:36.105 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:24:36.127 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:24:36.146 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:24:36.166 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:24:36.187 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:24:36.207 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:24:36.227 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:24:36.247 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:24:36.267 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:24:36.288 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:24:36.309 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:24:36.310 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:24:36.310 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:24:36.310 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:24:36.310 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:24:36.310 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:24:36.310 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:24:36.311 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:24:36.311 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:24:36.311 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:24:36.311 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:24:36.311 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:24:36.311 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:24:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:36 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3891565634' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-04-20T17:24:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:36 smithi171 ceph-mon[44189]: osdmap e445: 12 total, 12 up, 12 in 2024-04-20T17:24:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:36 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3303587288' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:24:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:36.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:36 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3891565634' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-04-20T17:24:36.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:36 smithi005 ceph-mon[39562]: osdmap e445: 12 total, 12 up, 12 in 2024-04-20T17:24:36.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:36 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3303587288' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:24:36.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:36.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:36.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:36.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:36 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3891565634' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-04-20T17:24:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:36 smithi106 ceph-mon[44423]: osdmap e445: 12 total, 12 up, 12 in 2024-04-20T17:24:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:36 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3303587288' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:24:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:37.330 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-04-20T17:24:37.353 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:24:37.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:37.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:24:37.458 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:24:37.487 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:37.825 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:37.825 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:37.825 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:37.825 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:37.825 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:37 smithi005 ceph-mon[39562]: pgmap v1190: 17 pgs: 5 creating+peering, 11 unknown, 1 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:37.826 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:37.826 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:37.826 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:37.826 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:37.826 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:37 smithi005 ceph-mon[39562]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-20T17:24:37.826 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:37 smithi005 ceph-mon[39562]: osdmap e446: 12 total, 12 up, 12 in 2024-04-20T17:24:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:37 smithi171 ceph-mon[44189]: pgmap v1190: 17 pgs: 5 creating+peering, 11 unknown, 1 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:37.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:37.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:37.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:37.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:37 smithi171 ceph-mon[44189]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-20T17:24:37.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:37 smithi171 ceph-mon[44189]: osdmap e446: 12 total, 12 up, 12 in 2024-04-20T17:24:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:38.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:38.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:38.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:37 smithi106 ceph-mon[44423]: pgmap v1190: 17 pgs: 5 creating+peering, 11 unknown, 1 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:38.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:38.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:38.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:38.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:38.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:37 smithi106 ceph-mon[44423]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-20T17:24:38.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:37 smithi106 ceph-mon[44423]: osdmap e446: 12 total, 12 up, 12 in 2024-04-20T17:24:38.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:38 smithi005 ceph-mon[39562]: osdmap e447: 12 total, 12 up, 12 in 2024-04-20T17:24:38.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:38 smithi171 ceph-mon[44189]: osdmap e447: 12 total, 12 up, 12 in 2024-04-20T17:24:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:38 smithi106 ceph-mon[44423]: osdmap e447: 12 total, 12 up, 12 in 2024-04-20T17:24:39.339 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:24:39.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:39.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:24:39.442 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:24:39.475 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:39.496 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:24:39.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:39.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:24:39.600 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:24:39.631 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:39.651 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:24:39.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:39.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:24:39.756 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:24:39.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:39 smithi106 ceph-mon[44423]: pgmap v1193: 17 pgs: 7 active+clean, 5 creating+peering, 5 unknown; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:39.791 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:39.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:39 smithi171 ceph-mon[44189]: pgmap v1193: 17 pgs: 7 active+clean, 5 creating+peering, 5 unknown; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:39.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:39 smithi005 ceph-mon[39562]: pgmap v1193: 17 pgs: 7 active+clean, 5 creating+peering, 5 unknown; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:40.037 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:24:40.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:40.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:24:40.154 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:24:40.183 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:40.592 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:24:40.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:40 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4068800362' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:24:40.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:40 smithi171 ceph-mon[44189]: from='client.26126 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:24:40.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:40 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4068800362' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:24:40.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:40 smithi005 ceph-mon[39562]: from='client.26126 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:24:41.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:40 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4068800362' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:24:41.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:40 smithi106 ceph-mon[44423]: from='client.26126 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:24:41.307 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:24:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:41 smithi171 ceph-mon[44189]: pgmap v1194: 17 pgs: 9 active+clean, 5 creating+peering, 3 unknown; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:41 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:24:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:41 smithi171 ceph-mon[44189]: from='client.26126 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:24:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:41 smithi171 ceph-mon[44189]: osdmap e448: 12 total, 12 up, 12 in 2024-04-20T17:24:41.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:41 smithi171 ceph-mon[44189]: osdmap e449: 12 total, 12 up, 12 in 2024-04-20T17:24:41.874 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:41 smithi005 ceph-mon[39562]: pgmap v1194: 17 pgs: 9 active+clean, 5 creating+peering, 3 unknown; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:41.875 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:41 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:24:41.875 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:41 smithi005 ceph-mon[39562]: from='client.26126 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:24:41.875 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:41 smithi005 ceph-mon[39562]: osdmap e448: 12 total, 12 up, 12 in 2024-04-20T17:24:41.875 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:41 smithi005 ceph-mon[39562]: osdmap e449: 12 total, 12 up, 12 in 2024-04-20T17:24:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:41 smithi106 ceph-mon[44423]: pgmap v1194: 17 pgs: 9 active+clean, 5 creating+peering, 3 unknown; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:41 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:24:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:41 smithi106 ceph-mon[44423]: from='client.26126 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:24:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:41 smithi106 ceph-mon[44423]: osdmap e448: 12 total, 12 up, 12 in 2024-04-20T17:24:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:41 smithi106 ceph-mon[44423]: osdmap e449: 12 total, 12 up, 12 in 2024-04-20T17:24:42.634 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:24:42.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:42.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:24:42.681 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:24:42.681 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:24:42.692 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:24:42.738 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:24:42.769 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:43.471 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-20T17:24:43.561 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:24:43.597 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.2 to 1 (10000) 2024-04-20T17:24:43.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:43 smithi005 ceph-mon[39562]: pgmap v1197: 17 pgs: 17 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:43.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:43 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3762389832' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-20T17:24:44.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:43 smithi106 ceph-mon[44423]: pgmap v1197: 17 pgs: 17 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:44.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:43 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3762389832' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-20T17:24:44.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:43 smithi171 ceph-mon[44189]: pgmap v1197: 17 pgs: 17 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:44.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:43 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3762389832' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-20T17:24:44.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:44 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:24:44.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:44 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:24:44.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:44 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3762389832' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-20T17:24:44.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:44 smithi005 ceph-mon[39562]: osdmap e450: 12 total, 12 up, 12 in 2024-04-20T17:24:44.648 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:24:44.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:44.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:24:44.668 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:24:44.752 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:24:44.784 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:44.806 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:24:44.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:44.809 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:24:44.910 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:24:44.943 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:45.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:44 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:24:45.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:44 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:24:45.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:44 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3762389832' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-20T17:24:45.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:44 smithi106 ceph-mon[44423]: osdmap e450: 12 total, 12 up, 12 in 2024-04-20T17:24:45.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:44 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:24:45.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:44 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:24:45.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:44 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3762389832' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-20T17:24:45.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:44 smithi171 ceph-mon[44189]: osdmap e450: 12 total, 12 up, 12 in 2024-04-20T17:24:45.792 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:24:45.793 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 3 1 2024-04-20T17:24:45.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:45.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:24:45.862 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:45 smithi005 ceph-mon[39562]: pgmap v1199: 17 pgs: 17 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:45.897 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:24:45.927 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:45 smithi171 ceph-mon[44189]: pgmap v1199: 17 pgs: 17 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:45.930 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:45.951 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:24:45.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:45.955 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:24:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:45 smithi106 ceph-mon[44423]: pgmap v1199: 17 pgs: 17 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:46.056 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:24:46.089 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:46.110 INFO:teuthology.orchestra.run.smithi005.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-04-20T17:24:46.111 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:24:46.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:46.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:24:46.217 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:24:46.246 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:46.265 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:24:46.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:46.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:24:46.369 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:24:46.397 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:46.416 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:24:46.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:46.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:24:46.520 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:24:46.552 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:46.573 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:24:46.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:46.576 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:24:46.677 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:24:46.704 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:46.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:46 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/821221598' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:24:46.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:46 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/821221598' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-20T17:24:46.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:46 smithi106 ceph-mon[44423]: osdmap e451: 12 total, 12 up, 12 in 2024-04-20T17:24:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:46 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/821221598' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:24:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:46 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/821221598' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-20T17:24:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:46 smithi171 ceph-mon[44189]: osdmap e451: 12 total, 12 up, 12 in 2024-04-20T17:24:46.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:46 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/821221598' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:24:46.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:46 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/821221598' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-20T17:24:46.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:46 smithi005 ceph-mon[39562]: osdmap e451: 12 total, 12 up, 12 in 2024-04-20T17:24:47.322 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:24:47.361 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:24:47.554 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:24:47.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:47.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:24:47.659 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:24:47.689 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:47.812 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:24:47.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:47.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:24:47.862 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:47 smithi005 ceph-mon[39562]: pgmap v1201: 17 pgs: 17 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:47.863 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:47 smithi005 ceph-mon[39562]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-20T17:24:47.916 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:24:47.946 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:47 smithi171 ceph-mon[44189]: pgmap v1201: 17 pgs: 17 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:47.947 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:47 smithi171 ceph-mon[44189]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-20T17:24:47.949 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:47.970 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:24:47.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:47.973 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:24:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:47 smithi106 ceph-mon[44423]: pgmap v1201: 17 pgs: 17 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:47 smithi106 ceph-mon[44423]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-20T17:24:48.074 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:24:48.102 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:48.363 DEBUG:teuthology.orchestra.run.smithi171:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-04-20T17:24:48.365 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-20T17:24:48.365 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-20T17:24:48.462 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:hints = 1 2024-04-20T17:24:48.463 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-20T17:24:48.463 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Object prefix: benchmark_data_smithi171_122772 2024-04-20T17:24:48.463 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:24:48.463 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 0 0 0 0 0 0 - 0 2024-04-20T17:24:49.462 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 1 16 6142 6126 382.856 382.875 0.00199785 0.0026059 2024-04-20T17:24:49.609 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:49 smithi005 ceph-mon[39562]: pgmap v1202: 17 pgs: 17 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:50.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:49 smithi106 ceph-mon[44423]: pgmap v1202: 17 pgs: 17 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:50.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:49 smithi171 ceph-mon[44189]: pgmap v1202: 17 pgs: 17 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:24:50.353 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:24:50.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:50.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:24:50.459 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:24:50.462 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 2 15 11349 11334 354.163 325.5 0.00328364 0.00281963 2024-04-20T17:24:50.497 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:50.520 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:24:50.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:50.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:24:50.625 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:24:50.670 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:50.695 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:24:50.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:50.698 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:24:50.776 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:24:50.776 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:24:50.799 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:24:50.855 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:50.923 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:24:50.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:50.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:24:51.031 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:24:51.032 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:24:51.087 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:51.129 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:24:51.131 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:24:51.163 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:24:51.196 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:24:51.233 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:24:51.265 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:24:51.295 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:24:51.328 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:24:51.359 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:24:51.394 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:24:51.433 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:24:51.463 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 3 16 15808 15792 328.976 278.625 0.0027436 0.00303733 2024-04-20T17:24:51.466 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:24:51.500 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:24:51.502 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:24:51.503 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:24:51.503 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:24:51.503 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:24:51.503 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:24:51.503 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:24:51.503 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:24:51.503 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:24:51.503 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:24:51.503 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:24:51.503 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:24:51.503 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:24:51.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:51 smithi005 ceph-mon[39562]: pgmap v1203: 17 pgs: 17 active+clean; 35 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s wr, 69 op/s 2024-04-20T17:24:51.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:51 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/149628641' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:24:51.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:51 smithi005 ceph-mon[39562]: from='client.26150 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:24:51.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:51 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2199772492' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:24:51.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:51 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:24:51.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:51 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:24:51.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:51 smithi005 ceph-mon[39562]: from='client.26150 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:24:51.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:51 smithi005 ceph-mon[39562]: osdmap e452: 12 total, 12 up, 12 in 2024-04-20T17:24:51.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:51.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:51.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:51.746 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-20T17:24:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:51 smithi106 ceph-mon[44423]: pgmap v1203: 17 pgs: 17 active+clean; 35 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s wr, 69 op/s 2024-04-20T17:24:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:51 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/149628641' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:24:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:51 smithi106 ceph-mon[44423]: from='client.26150 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:24:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:51 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2199772492' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:24:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:51 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:24:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:51 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:24:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:51 smithi106 ceph-mon[44423]: from='client.26150 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:24:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:51 smithi106 ceph-mon[44423]: osdmap e452: 12 total, 12 up, 12 in 2024-04-20T17:24:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:51 smithi171 ceph-mon[44189]: pgmap v1203: 17 pgs: 17 active+clean; 35 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s wr, 69 op/s 2024-04-20T17:24:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:51 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/149628641' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:24:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:51 smithi171 ceph-mon[44189]: from='client.26150 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:24:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:51 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2199772492' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:24:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:51 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:24:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:51 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:24:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:51 smithi171 ceph-mon[44189]: from='client.26150 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:24:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:51 smithi171 ceph-mon[44189]: osdmap e452: 12 total, 12 up, 12 in 2024-04-20T17:24:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:52.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:52.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:52.148 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:24:52.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:52.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:24:52.252 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:24:52.303 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:52.365 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:24:52.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:52.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:24:52.463 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 4 16 20265 20249 316.367 278.562 0.00354793 0.00315864 2024-04-20T17:24:52.469 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:24:52.510 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:52.880 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:52.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:52.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:52.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:52.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:52.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:52.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:52.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:52.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:52.899 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:24:52.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:52.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:24:53.003 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:24:53.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:53.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:53.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:53.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:53.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:53.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:53.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:53.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:53.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:53.034 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:53.035 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:53.035 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:53.035 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:53.035 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:53.035 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:53.035 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:53.035 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:53.035 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:24:53.037 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:53.063 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:24:53.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:53.069 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:24:53.170 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:24:53.204 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:53.230 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:24:53.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:53.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:24:53.336 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:24:53.373 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:53.398 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:24:53.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:53.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:24:53.463 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 5 16 23744 23728 296.577 217.438 0.00242604 0.00336939 2024-04-20T17:24:53.504 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:24:53.540 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:53.566 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:24:53.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:53.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:24:53.670 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:24:53.711 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:53.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:53 smithi005 ceph-mon[39562]: pgmap v1205: 17 pgs: 17 active+clean; 451 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 900 op/s 2024-04-20T17:24:53.737 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:24:53.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:53.740 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:24:53.841 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:24:53.872 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:53 smithi106 ceph-mon[44423]: pgmap v1205: 17 pgs: 17 active+clean; 451 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 900 op/s 2024-04-20T17:24:53.875 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:53.901 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:24:53.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:53.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:24:54.006 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:24:54.045 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:54.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:53 smithi171 ceph-mon[44189]: pgmap v1205: 17 pgs: 17 active+clean; 451 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 900 op/s 2024-04-20T17:24:54.202 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:24:54.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:54.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:24:54.237 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:24:54.317 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:24:54.358 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:54.388 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:24:54.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:54.392 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:24:54.463 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 6 16 27696 27680 288.311 247 0.00268911 0.00346656 2024-04-20T17:24:54.493 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:24:54.540 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:54.906 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:24:55.206 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:24:55.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:55.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:24:55.308 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:24:55.352 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:55.378 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:24:55.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:55.383 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:24:55.463 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 7 16 31384 31368 280.049 230.5 0.00257606 0.00356829 2024-04-20T17:24:55.484 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:24:55.527 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:55.557 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:24:55.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:55.560 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:24:55.661 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:24:55.707 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:55.763 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:24:55.788 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:24:55.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:55.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:24:55.904 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:24:55.936 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:55 smithi106 ceph-mon[44423]: pgmap v1206: 17 pgs: 17 active+clean; 1.0 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 129 MiB/s wr, 2.06k op/s 2024-04-20T17:24:55.939 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:55.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:55 smithi005 ceph-mon[39562]: pgmap v1206: 17 pgs: 17 active+clean; 1.0 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 129 MiB/s wr, 2.06k op/s 2024-04-20T17:24:55.966 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:24:55.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:55.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:24:56.070 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:24:56.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:55 smithi171 ceph-mon[44189]: pgmap v1206: 17 pgs: 17 active+clean; 1.0 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 129 MiB/s wr, 2.06k op/s 2024-04-20T17:24:56.099 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:24:56.132 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:56.463 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 8 16 34538 34522 269.679 197.125 0.00479817 0.00370671 2024-04-20T17:24:57.157 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.3 to 1 (10000) 2024-04-20T17:24:57.187 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:24:57.187 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:24:57.187 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:24:57.187 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:24:57.187 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.187 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:259121 2.3 2:c91e897c:::benchmark_data_smithi171_78918_object98127:head [delete] snapc 0=[] ondisk+write+known_if_redirected e294)", 2024-04-20T17:24:57.187 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:17:07.741988+0000", 2024-04-20T17:24:57.188 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 469.44179560200001, 2024-04-20T17:24:57.188 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.645843406, 2024-04-20T17:24:57.188 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:24:57.188 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:24:57.188 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:57.188 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:24:57.188 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:24:57.188 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:24:57.188 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 259121 2024-04-20T17:24:57.188 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.189 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:24:57.189 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.189 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:24:57.189 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:07.741988+0000", 2024-04-20T17:24:57.189 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.189 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.189 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.189 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:24:57.189 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:07.741988+0000", 2024-04-20T17:24:57.189 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.189 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.189 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.190 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:24:57.190 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:07.741990+0000", 2024-04-20T17:24:57.190 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.08e-06 2024-04-20T17:24:57.190 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.190 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.190 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:24:57.190 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:07.741995+0000", 2024-04-20T17:24:57.190 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.789e-06 2024-04-20T17:24:57.190 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.190 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.190 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:24:57.191 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:07.742007+0000", 2024-04-20T17:24:57.191 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1728e-05 2024-04-20T17:24:57.191 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.191 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.191 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:24:57.191 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:07.742013+0000", 2024-04-20T17:24:57.191 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0680000000000002e-06 2024-04-20T17:24:57.191 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.191 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.191 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.192 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:07.826185+0000", 2024-04-20T17:24:57.192 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.084171678999999999 2024-04-20T17:24:57.192 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.192 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.192 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:24:57.192 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:07.828716+0000", 2024-04-20T17:24:57.192 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00253126 2024-04-20T17:24:57.192 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.192 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.192 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.192 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:08.499283+0000", 2024-04-20T17:24:57.193 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.67056691899999998 2024-04-20T17:24:57.193 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.193 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.193 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:24:57.193 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:08.501962+0000", 2024-04-20T17:24:57.193 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0026792109999999999 2024-04-20T17:24:57.193 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.193 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.193 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.193 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:10.238389+0000", 2024-04-20T17:24:57.194 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.736427071 2024-04-20T17:24:57.194 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.194 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.194 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:24:57.194 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:10.242452+0000", 2024-04-20T17:24:57.194 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0040623830000000001 2024-04-20T17:24:57.194 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.194 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.194 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:24:57.194 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:10.250550+0000", 2024-04-20T17:24:57.194 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0080983040000000006 2024-04-20T17:24:57.194 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.194 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.194 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:24:57.195 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:10.295991+0000", 2024-04-20T17:24:57.195 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.045440734000000003 2024-04-20T17:24:57.195 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.195 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.195 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:57.195 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:10.375642+0000", 2024-04-20T17:24:57.195 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.079650913000000004 2024-04-20T17:24:57.195 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.195 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.195 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:24:57.195 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:10.378793+0000", 2024-04-20T17:24:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003151526 2024-04-20T17:24:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:24:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:10.387831+0000", 2024-04-20T17:24:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0090377410000000002 2024-04-20T17:24:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:24:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:259416 2.3 2:c3b5e54f:::benchmark_data_smithi171_78918_object98422:head [delete] snapc 0=[] ondisk+write+known_if_redirected e294)", 2024-04-20T17:24:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:17:09.086846+0000", 2024-04-20T17:24:57.196 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 468.096937363, 2024-04-20T17:24:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.90832223, 2024-04-20T17:24:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:24:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:24:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:24:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:24:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:24:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 259416 2024-04-20T17:24:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:24:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:24:57.197 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:09.086846+0000", 2024-04-20T17:24:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:24:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:09.086846+0000", 2024-04-20T17:24:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:24:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:09.086850+0000", 2024-04-20T17:24:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.087e-06 2024-04-20T17:24:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.198 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:24:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:09.086865+0000", 2024-04-20T17:24:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5051999999999999e-05 2024-04-20T17:24:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:24:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:09.086892+0000", 2024-04-20T17:24:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7324000000000001e-05 2024-04-20T17:24:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:24:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:09.086908+0000", 2024-04-20T17:24:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5880999999999999e-05 2024-04-20T17:24:57.199 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:09.110572+0000", 2024-04-20T17:24:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.02366383 2024-04-20T17:24:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:24:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:09.112793+0000", 2024-04-20T17:24:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0022206690000000002 2024-04-20T17:24:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:09.770052+0000", 2024-04-20T17:24:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.65725883699999998 2024-04-20T17:24:57.200 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:24:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:09.772834+0000", 2024-04-20T17:24:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0027826449999999998 2024-04-20T17:24:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:11.923958+0000", 2024-04-20T17:24:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.151123519 2024-04-20T17:24:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:24:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:11.927994+0000", 2024-04-20T17:24:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0040365269999999998 2024-04-20T17:24:57.201 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:24:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:11.936064+0000", 2024-04-20T17:24:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0080701729999999999 2024-04-20T17:24:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:24:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:11.961695+0000", 2024-04-20T17:24:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.025630868000000001 2024-04-20T17:24:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:11.982001+0000", 2024-04-20T17:24:57.202 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.020305349 2024-04-20T17:24:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:24:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:11.986291+0000", 2024-04-20T17:24:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0042906259999999996 2024-04-20T17:24:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:24:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:11.995168+0000", 2024-04-20T17:24:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0088768430000000006 2024-04-20T17:24:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:24:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.203 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:263462 2.3 2:cae5d6f3:::benchmark_data_smithi171_78918_object102468:head [delete] snapc 0=[] ondisk+write+known_if_redirected e302)", 2024-04-20T17:24:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:17:31.688766+0000", 2024-04-20T17:24:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 445.49501708000003, 2024-04-20T17:24:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.724801673, 2024-04-20T17:24:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:24:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:24:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-04-20T17:24:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:24:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:24:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:24:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 263462 2024-04-20T17:24:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:24:57.204 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:24:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:31.688766+0000", 2024-04-20T17:24:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:24:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:31.688766+0000", 2024-04-20T17:24:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:24:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:31.688768+0000", 2024-04-20T17:24:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.773e-06 2024-04-20T17:24:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.205 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.206 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:24:57.206 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:31.688773+0000", 2024-04-20T17:24:57.206 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0669999999999999e-06 2024-04-20T17:24:57.206 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.206 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.206 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:24:57.206 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:31.688785+0000", 2024-04-20T17:24:57.206 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1792e-05 2024-04-20T17:24:57.206 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.206 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.206 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:24:57.206 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:31.688792+0000", 2024-04-20T17:24:57.206 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2010000000000002e-06 2024-04-20T17:24:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:31.835481+0000", 2024-04-20T17:24:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.146689184 2024-04-20T17:24:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:24:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:31.837531+0000", 2024-04-20T17:24:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002049993 2024-04-20T17:24:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.207 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:32.495598+0000", 2024-04-20T17:24:57.208 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.65806703499999997 2024-04-20T17:24:57.208 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.208 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.208 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:24:57.208 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:32.498093+0000", 2024-04-20T17:24:57.208 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0024949080000000001 2024-04-20T17:24:57.208 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.208 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.208 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.208 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:34.413013+0000", 2024-04-20T17:24:57.208 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9149201200000001 2024-04-20T17:24:57.208 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.208 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.208 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:24:57.208 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:34.413568+0000", 2024-04-20T17:24:57.209 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00055460000000000004 2024-04-20T17:24:57.209 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.209 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:24:57.209 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.209 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.209 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.209 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:267267 2.3 2:c16bfb3f:::benchmark_data_smithi171_78918_object106273:head [delete] snapc 0=[] ondisk+write+known_if_redirected e308)", 2024-04-20T17:24:57.209 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:17:50.686207+0000", 2024-04-20T17:24:57.209 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 426.497575708, 2024-04-20T17:24:57.209 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7237566850000001, 2024-04-20T17:24:57.209 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:24:57.209 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:24:57.209 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:57.209 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:24:57.210 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:24:57.210 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:24:57.210 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 267267 2024-04-20T17:24:57.210 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.210 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:24:57.210 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.210 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:24:57.210 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:50.686207+0000", 2024-04-20T17:24:57.210 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.210 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.210 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.210 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:24:57.210 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:50.686207+0000", 2024-04-20T17:24:57.210 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.211 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.211 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.211 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:24:57.211 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:50.686210+0000", 2024-04-20T17:24:57.211 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5739999999999998e-06 2024-04-20T17:24:57.211 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.211 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.211 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:24:57.211 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:50.686219+0000", 2024-04-20T17:24:57.211 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6610000000000006e-06 2024-04-20T17:24:57.211 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.211 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.211 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:24:57.211 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:50.686231+0000", 2024-04-20T17:24:57.212 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2765999999999999e-05 2024-04-20T17:24:57.212 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.212 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.212 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:24:57.212 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:50.686242+0000", 2024-04-20T17:24:57.212 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0593e-05 2024-04-20T17:24:57.212 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.212 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.212 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.212 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:50.693321+0000", 2024-04-20T17:24:57.212 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0070788379999999996 2024-04-20T17:24:57.212 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.212 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.212 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:24:57.212 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:50.695239+0000", 2024-04-20T17:24:57.213 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001918171 2024-04-20T17:24:57.213 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.213 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.213 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.213 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:51.493583+0000", 2024-04-20T17:24:57.213 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.79834359300000002 2024-04-20T17:24:57.213 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.213 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.213 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:24:57.213 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:51.496785+0000", 2024-04-20T17:24:57.213 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0032021380000000002 2024-04-20T17:24:57.213 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.213 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.213 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.213 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:53.274012+0000", 2024-04-20T17:24:57.214 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.777227224 2024-04-20T17:24:57.214 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.214 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.214 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:24:57.214 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:53.277485+0000", 2024-04-20T17:24:57.214 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003472485 2024-04-20T17:24:57.214 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.214 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.214 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:24:57.214 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:53.284302+0000", 2024-04-20T17:24:57.214 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0068175969999999999 2024-04-20T17:24:57.214 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.214 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.214 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:57.215 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:53.291173+0000", 2024-04-20T17:24:57.215 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0068707500000000001 2024-04-20T17:24:57.215 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.215 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.215 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:24:57.215 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:53.400486+0000", 2024-04-20T17:24:57.215 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.109313438 2024-04-20T17:24:57.215 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.215 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.215 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:24:57.215 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:53.402384+0000", 2024-04-20T17:24:57.216 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0018981479999999999 2024-04-20T17:24:57.216 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.216 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.216 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:24:57.216 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:53.409964+0000", 2024-04-20T17:24:57.216 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0075797089999999996 2024-04-20T17:24:57.216 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.216 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:24:57.217 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.217 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.217 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.217 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:268280 2.3 2:c0f96bdd:::benchmark_data_smithi171_78918_object107286:head [delete] snapc 0=[] ondisk+write+known_if_redirected e309)", 2024-04-20T17:24:57.217 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:17:55.131150+0000", 2024-04-20T17:24:57.217 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 422.05263366200001, 2024-04-20T17:24:57.217 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.634024288, 2024-04-20T17:24:57.217 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:24:57.217 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:24:57.217 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:57.217 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:24:57.217 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:24:57.217 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:24:57.217 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 268280 2024-04-20T17:24:57.218 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.218 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:24:57.218 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.218 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:24:57.218 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.131150+0000", 2024-04-20T17:24:57.218 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.218 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.218 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.218 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:24:57.218 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.131150+0000", 2024-04-20T17:24:57.219 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.219 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.219 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.219 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:24:57.219 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.131152+0000", 2024-04-20T17:24:57.219 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1179999999999999e-06 2024-04-20T17:24:57.219 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.219 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.219 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:24:57.219 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.131158+0000", 2024-04-20T17:24:57.219 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7150000000000001e-06 2024-04-20T17:24:57.219 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.219 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.220 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:24:57.220 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.131170+0000", 2024-04-20T17:24:57.220 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1433999999999999e-05 2024-04-20T17:24:57.220 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.220 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.220 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:24:57.220 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.131179+0000", 2024-04-20T17:24:57.220 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0420000000000008e-06 2024-04-20T17:24:57.220 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.220 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.220 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.220 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.185255+0000", 2024-04-20T17:24:57.221 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.054076235 2024-04-20T17:24:57.221 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.221 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.221 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:24:57.221 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.187152+0000", 2024-04-20T17:24:57.221 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001896539 2024-04-20T17:24:57.221 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.221 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.221 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.221 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.925874+0000", 2024-04-20T17:24:57.221 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.73872241400000005 2024-04-20T17:24:57.221 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.222 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.222 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:24:57.222 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.928102+0000", 2024-04-20T17:24:57.222 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0022278860000000001 2024-04-20T17:24:57.222 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.222 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.222 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.222 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:57.681606+0000", 2024-04-20T17:24:57.222 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.753504239 2024-04-20T17:24:57.223 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.223 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.223 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:24:57.223 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:57.685368+0000", 2024-04-20T17:24:57.223 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0037615859999999999 2024-04-20T17:24:57.223 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.223 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.223 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:24:57.223 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:57.693441+0000", 2024-04-20T17:24:57.223 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0080737340000000008 2024-04-20T17:24:57.223 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.223 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.223 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:24:57.224 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:57.734945+0000", 2024-04-20T17:24:57.224 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.041503962999999998 2024-04-20T17:24:57.224 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.224 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.224 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:57.224 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:57.755109+0000", 2024-04-20T17:24:57.224 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.020163945999999999 2024-04-20T17:24:57.224 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.224 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.225 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:24:57.225 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:57.757740+0000", 2024-04-20T17:24:57.225 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002630453 2024-04-20T17:24:57.225 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.225 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.225 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:24:57.225 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:57.765174+0000", 2024-04-20T17:24:57.225 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0074339840000000003 2024-04-20T17:24:57.225 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.225 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:24:57.225 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.226 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.226 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.226 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:272485 2.3 2:cf8b6f46:::benchmark_data_smithi171_78918_object111491:head [delete] snapc 0=[] ondisk+write+known_if_redirected e316)", 2024-04-20T17:24:57.226 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:18:13.931178+0000", 2024-04-20T17:24:57.226 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 403.25260547599999, 2024-04-20T17:24:57.226 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5463639260000002, 2024-04-20T17:24:57.226 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:24:57.226 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:24:57.226 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:57.227 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:24:57.227 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:24:57.227 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:24:57.227 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 272485 2024-04-20T17:24:57.227 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.227 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:24:57.227 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.227 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:24:57.227 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:13.931178+0000", 2024-04-20T17:24:57.227 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.227 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.227 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.227 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:24:57.227 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:13.931178+0000", 2024-04-20T17:24:57.228 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.228 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.228 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.228 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:24:57.228 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:13.931180+0000", 2024-04-20T17:24:57.228 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0200000000000001e-06 2024-04-20T17:24:57.228 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.228 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.228 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:24:57.228 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:13.931186+0000", 2024-04-20T17:24:57.228 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5209999999999999e-06 2024-04-20T17:24:57.228 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.228 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.228 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:24:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:13.931198+0000", 2024-04-20T17:24:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2126999999999999e-05 2024-04-20T17:24:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:24:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:13.931205+0000", 2024-04-20T17:24:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0010000000000004e-06 2024-04-20T17:24:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:14.024529+0000", 2024-04-20T17:24:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.093323992999999994 2024-04-20T17:24:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.229 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.230 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:24:57.230 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:14.026231+0000", 2024-04-20T17:24:57.230 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0017012640000000001 2024-04-20T17:24:57.230 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.230 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.230 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.230 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:14.688291+0000", 2024-04-20T17:24:57.230 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.66206054700000005 2024-04-20T17:24:57.230 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.230 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.230 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:24:57.230 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:14.690674+0000", 2024-04-20T17:24:57.230 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002383282 2024-04-20T17:24:57.230 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:16.448551+0000", 2024-04-20T17:24:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7578762220000002 2024-04-20T17:24:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:24:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:16.451664+0000", 2024-04-20T17:24:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003113476 2024-04-20T17:24:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:24:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:16.457746+0000", 2024-04-20T17:24:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0060814440000000001 2024-04-20T17:24:57.231 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.232 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.232 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:57.232 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:16.468555+0000", 2024-04-20T17:24:57.232 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010809440999999999 2024-04-20T17:24:57.232 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.232 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.232 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:24:57.232 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:16.469204+0000", 2024-04-20T17:24:57.232 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00064895199999999995 2024-04-20T17:24:57.232 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.232 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.232 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:24:57.232 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:16.470865+0000", 2024-04-20T17:24:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00166113 2024-04-20T17:24:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:24:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:16.477542+0000", 2024-04-20T17:24:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0066765059999999996 2024-04-20T17:24:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:24:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.233 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:273306 2.3 2:c7ad72a8:::benchmark_data_smithi171_78918_object112312:head [delete] snapc 0=[] ondisk+write+known_if_redirected e317)", 2024-04-20T17:24:57.234 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:18:17.740700+0000", 2024-04-20T17:24:57.234 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 399.44308334200002, 2024-04-20T17:24:57.234 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7316630169999998, 2024-04-20T17:24:57.234 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:24:57.234 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:24:57.234 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:57.234 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:24:57.234 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:24:57.234 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:24:57.234 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 273306 2024-04-20T17:24:57.234 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.234 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:24:57.235 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.235 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:24:57.235 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:17.740700+0000", 2024-04-20T17:24:57.235 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.235 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.235 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.235 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:24:57.235 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:17.740700+0000", 2024-04-20T17:24:57.235 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.235 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.235 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.235 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:24:57.235 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:17.740703+0000", 2024-04-20T17:24:57.236 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3699999999999999e-06 2024-04-20T17:24:57.236 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.236 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.236 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:24:57.236 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:17.740713+0000", 2024-04-20T17:24:57.236 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7049999999999994e-06 2024-04-20T17:24:57.236 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.236 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.236 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:24:57.236 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:17.740727+0000", 2024-04-20T17:24:57.236 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3828999999999999e-05 2024-04-20T17:24:57.236 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.236 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.237 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:24:57.237 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:17.740736+0000", 2024-04-20T17:24:57.237 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.3209999999999994e-06 2024-04-20T17:24:57.237 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.237 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.237 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.237 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:17.819955+0000", 2024-04-20T17:24:57.237 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.079219081999999996 2024-04-20T17:24:57.237 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.237 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.237 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:24:57.237 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:17.822169+0000", 2024-04-20T17:24:57.237 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0022142680000000001 2024-04-20T17:24:57.237 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.238 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.238 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.238 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:18.772171+0000", 2024-04-20T17:24:57.238 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.95000131600000004 2024-04-20T17:24:57.238 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.238 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.238 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:24:57.238 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:18.774199+0000", 2024-04-20T17:24:57.238 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0020284610000000001 2024-04-20T17:24:57.238 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.238 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.238 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:20.414977+0000", 2024-04-20T17:24:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6407782499999999 2024-04-20T17:24:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:24:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:20.417976+0000", 2024-04-20T17:24:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002998213 2024-04-20T17:24:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:24:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:20.423980+0000", 2024-04-20T17:24:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.006004634 2024-04-20T17:24:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.239 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:24:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:20.460896+0000", 2024-04-20T17:24:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.036915765000000003 2024-04-20T17:24:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:20.462179+0000", 2024-04-20T17:24:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0012827940000000001 2024-04-20T17:24:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:24:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:20.464524+0000", 2024-04-20T17:24:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0023448779999999999 2024-04-20T17:24:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.240 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.241 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:24:57.241 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:20.472363+0000", 2024-04-20T17:24:57.241 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0078391309999999992 2024-04-20T17:24:57.241 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.241 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:24:57.241 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.241 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.241 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.241 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.25802.0:260071 3.6 3:69d4640c:::benchmark_data_smithi171_110765_object116856:head [delete] snapc 0=[] ondisk+write+known_if_redirected e426)", 2024-04-20T17:24:57.241 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:23:46.061347+0000", 2024-04-20T17:24:57.241 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 71.122435748000001, 2024-04-20T17:24:57.241 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.054681311000000003, 2024-04-20T17:24:57.241 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:24:57.241 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:24:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:24:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.25802", 2024-04-20T17:24:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 260071 2024-04-20T17:24:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:24:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:24:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.061347+0000", 2024-04-20T17:24:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:24:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.061347+0000", 2024-04-20T17:24:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:24:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.061349+0000", 2024-04-20T17:24:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7570000000000001e-06 2024-04-20T17:24:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:24:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.061354+0000", 2024-04-20T17:24:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0329999999999999e-06 2024-04-20T17:24:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:24:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.061366+0000", 2024-04-20T17:24:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1362999999999999e-05 2024-04-20T17:24:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:24:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.061374+0000", 2024-04-20T17:24:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1960000000000008e-06 2024-04-20T17:24:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.064580+0000", 2024-04-20T17:24:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003205889 2024-04-20T17:24:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:24:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.064690+0000", 2024-04-20T17:24:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011050800000000001 2024-04-20T17:24:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:24:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.064770+0000", 2024-04-20T17:24:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9760999999999997e-05 2024-04-20T17:24:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:24:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.069597+0000", 2024-04-20T17:24:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0048274950000000002 2024-04-20T17:24:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.115919+0000", 2024-04-20T17:24:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.046321946000000003 2024-04-20T17:24:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:24:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.115977+0000", 2024-04-20T17:24:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7856000000000003e-05 2024-04-20T17:24:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:24:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.116029+0000", 2024-04-20T17:24:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1507e-05 2024-04-20T17:24:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:24:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.25802.0:260081 3.6 3:6d07392e:::benchmark_data_smithi171_110765_object116866:head [delete] snapc 0=[] ondisk+write+known_if_redirected e426)", 2024-04-20T17:24:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:23:46.081651+0000", 2024-04-20T17:24:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 71.102132533000002, 2024-04-20T17:24:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.050428216999999997, 2024-04-20T17:24:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:24:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:24:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:24:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.25802", 2024-04-20T17:24:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 260081 2024-04-20T17:24:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:24:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:24:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.081651+0000", 2024-04-20T17:24:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:24:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.081651+0000", 2024-04-20T17:24:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:24:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.081652+0000", 2024-04-20T17:24:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.519e-06 2024-04-20T17:24:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:24:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.081657+0000", 2024-04-20T17:24:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9840000000000001e-06 2024-04-20T17:24:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:24:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.081665+0000", 2024-04-20T17:24:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3059999999999998e-06 2024-04-20T17:24:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:24:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.081671+0000", 2024-04-20T17:24:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.959e-06 2024-04-20T17:24:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.081955+0000", 2024-04-20T17:24:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00028391000000000001 2024-04-20T17:24:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:24:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.082081+0000", 2024-04-20T17:24:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012533299999999999 2024-04-20T17:24:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:24:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.082199+0000", 2024-04-20T17:24:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011801700000000001 2024-04-20T17:24:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.116077+0000", 2024-04-20T17:24:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.033877917 2024-04-20T17:24:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:24:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.131983+0000", 2024-04-20T17:24:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015906057000000001 2024-04-20T17:24:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:24:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.132031+0000", 2024-04-20T17:24:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7871000000000003e-05 2024-04-20T17:24:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:24:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.132079+0000", 2024-04-20T17:24:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8344000000000003e-05 2024-04-20T17:24:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:24:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.25802.0:260085 3.6 3:644ff935:::benchmark_data_smithi171_110765_object116870:head [delete] snapc 0=[] ondisk+write+known_if_redirected e426)", 2024-04-20T17:24:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:23:46.081688+0000", 2024-04-20T17:24:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 71.102094950999998, 2024-04-20T17:24:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.050464499000000003, 2024-04-20T17:24:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:24:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:24:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:24:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.25802", 2024-04-20T17:24:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 260085 2024-04-20T17:24:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:24:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:24:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.081688+0000", 2024-04-20T17:24:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.255 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:24:57.255 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.081688+0000", 2024-04-20T17:24:57.255 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.255 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.255 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.255 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:24:57.255 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.081689+0000", 2024-04-20T17:24:57.255 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1799999999999995e-07 2024-04-20T17:24:57.255 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.255 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.255 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:24:57.255 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.081692+0000", 2024-04-20T17:24:57.256 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6630000000000002e-06 2024-04-20T17:24:57.256 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.256 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.256 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:24:57.256 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.081697+0000", 2024-04-20T17:24:57.256 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3240000000000002e-06 2024-04-20T17:24:57.256 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.256 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.256 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:24:57.256 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.081703+0000", 2024-04-20T17:24:57.256 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7069999999999997e-06 2024-04-20T17:24:57.256 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.257 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.257 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.257 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.087471+0000", 2024-04-20T17:24:57.257 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00576839 2024-04-20T17:24:57.257 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.257 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.257 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:24:57.257 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.087590+0000", 2024-04-20T17:24:57.257 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000118669 2024-04-20T17:24:57.257 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.257 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.257 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:24:57.258 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.087711+0000", 2024-04-20T17:24:57.258 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000121517 2024-04-20T17:24:57.258 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.258 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.258 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:57.258 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.131892+0000", 2024-04-20T17:24:57.258 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.044181311000000001 2024-04-20T17:24:57.258 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.258 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.258 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:24:57.258 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.132086+0000", 2024-04-20T17:24:57.258 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00019324399999999999 2024-04-20T17:24:57.258 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.259 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.259 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:24:57.259 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.132121+0000", 2024-04-20T17:24:57.259 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5395999999999998e-05 2024-04-20T17:24:57.259 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.259 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.259 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:24:57.259 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.132153+0000", 2024-04-20T17:24:57.259 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1659999999999998e-05 2024-04-20T17:24:57.259 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.260 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:24:57.260 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.260 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.260 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.260 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.25802.0:260088 3.6 3:6d7aad92:::benchmark_data_smithi171_110765_object116873:head [delete] snapc 0=[] ondisk+write+known_if_redirected e426)", 2024-04-20T17:24:57.260 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:23:46.081708+0000", 2024-04-20T17:24:57.260 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 71.102074715000001, 2024-04-20T17:24:57.260 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.050515259999999999, 2024-04-20T17:24:57.260 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:24:57.260 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:24:57.260 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:57.261 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:24:57.261 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.25802", 2024-04-20T17:24:57.261 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:57.261 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 260088 2024-04-20T17:24:57.261 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.261 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:24:57.261 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.261 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:24:57.261 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.081708+0000", 2024-04-20T17:24:57.261 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.262 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.262 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.262 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:24:57.262 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.081708+0000", 2024-04-20T17:24:57.262 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.262 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.262 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.262 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:24:57.263 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.081709+0000", 2024-04-20T17:24:57.263 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9599999999999999e-07 2024-04-20T17:24:57.263 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.263 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.263 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:24:57.263 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.081712+0000", 2024-04-20T17:24:57.263 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.678e-06 2024-04-20T17:24:57.263 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.263 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.263 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:24:57.263 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.081716+0000", 2024-04-20T17:24:57.263 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6600000000000003e-06 2024-04-20T17:24:57.263 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.263 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.264 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:24:57.264 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.081721+0000", 2024-04-20T17:24:57.264 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5449999999999997e-06 2024-04-20T17:24:57.264 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.264 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.264 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.264 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.092500+0000", 2024-04-20T17:24:57.264 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010779627 2024-04-20T17:24:57.264 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.264 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.264 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:24:57.264 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.092633+0000", 2024-04-20T17:24:57.264 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000132686 2024-04-20T17:24:57.264 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.265 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.265 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:24:57.265 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.092759+0000", 2024-04-20T17:24:57.265 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012627100000000001 2024-04-20T17:24:57.265 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.265 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.265 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:57.265 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.131931+0000", 2024-04-20T17:24:57.265 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.039171507000000001 2024-04-20T17:24:57.265 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.265 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.265 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:24:57.265 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.132158+0000", 2024-04-20T17:24:57.266 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00022706199999999999 2024-04-20T17:24:57.266 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.266 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.266 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:24:57.266 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.132193+0000", 2024-04-20T17:24:57.266 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4770000000000001e-05 2024-04-20T17:24:57.266 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.266 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.266 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:24:57.266 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.132224+0000", 2024-04-20T17:24:57.266 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0957999999999997e-05 2024-04-20T17:24:57.266 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.266 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:24:57.267 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.267 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.267 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.267 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.25802.0:260090 3.6 3:65a8f38d:::benchmark_data_smithi171_110765_object116875:head [delete] snapc 0=[] ondisk+write+known_if_redirected e426)", 2024-04-20T17:24:57.267 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:23:46.088485+0000", 2024-04-20T17:24:57.267 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 71.095298389999996, 2024-04-20T17:24:57.267 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.049244243, 2024-04-20T17:24:57.267 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:24:57.268 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:24:57.268 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:57.268 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:24:57.268 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.25802", 2024-04-20T17:24:57.268 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:57.268 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 260090 2024-04-20T17:24:57.268 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.268 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:24:57.268 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.268 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:24:57.268 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.088485+0000", 2024-04-20T17:24:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:24:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.088485+0000", 2024-04-20T17:24:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:24:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.088485+0000", 2024-04-20T17:24:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1999999999999999e-07 2024-04-20T17:24:57.269 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.270 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.270 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:24:57.270 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.088489+0000", 2024-04-20T17:24:57.270 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0730000000000001e-06 2024-04-20T17:24:57.270 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.270 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.270 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:24:57.270 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.088494+0000", 2024-04-20T17:24:57.270 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.356e-06 2024-04-20T17:24:57.270 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.270 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.270 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:24:57.270 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.088499+0000", 2024-04-20T17:24:57.270 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8849999999999998e-06 2024-04-20T17:24:57.271 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.271 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.271 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.271 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.103052+0000", 2024-04-20T17:24:57.271 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014553089999999999 2024-04-20T17:24:57.271 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.271 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.271 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:24:57.271 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.103203+0000", 2024-04-20T17:24:57.271 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000150846 2024-04-20T17:24:57.271 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.271 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.271 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:24:57.271 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.103353+0000", 2024-04-20T17:24:57.271 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000150543 2024-04-20T17:24:57.272 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.272 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.272 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:24:57.272 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.132308+0000", 2024-04-20T17:24:57.272 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.028954855000000002 2024-04-20T17:24:57.272 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.272 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.272 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:57.272 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.137627+0000", 2024-04-20T17:24:57.272 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0053184499999999997 2024-04-20T17:24:57.272 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.272 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.272 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:24:57.272 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.137683+0000", 2024-04-20T17:24:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6450000000000003e-05 2024-04-20T17:24:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:24:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.137729+0000", 2024-04-20T17:24:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6075000000000002e-05 2024-04-20T17:24:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:24:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.25802.0:260093 3.6 3:6d8aed7d:::benchmark_data_smithi171_110765_object116878:head [delete] snapc 0=[] ondisk+write+known_if_redirected e426)", 2024-04-20T17:24:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:23:46.088505+0000", 2024-04-20T17:24:57.273 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 71.095278695999994, 2024-04-20T17:24:57.274 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.049339701999999999, 2024-04-20T17:24:57.274 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:24:57.274 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:24:57.274 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:57.274 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:24:57.274 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.25802", 2024-04-20T17:24:57.274 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:57.274 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 260093 2024-04-20T17:24:57.274 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.274 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:24:57.274 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.274 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:24:57.274 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.088505+0000", 2024-04-20T17:24:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:24:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.088505+0000", 2024-04-20T17:24:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:24:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.088505+0000", 2024-04-20T17:24:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2900000000000004e-07 2024-04-20T17:24:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:24:57.275 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.088508+0000", 2024-04-20T17:24:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.61e-06 2024-04-20T17:24:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:24:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.088512+0000", 2024-04-20T17:24:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4730000000000002e-06 2024-04-20T17:24:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:24:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.088517+0000", 2024-04-20T17:24:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6859999999999997e-06 2024-04-20T17:24:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.276 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.111018+0000", 2024-04-20T17:24:57.277 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.022501264 2024-04-20T17:24:57.277 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.277 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.277 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:24:57.277 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.111113+0000", 2024-04-20T17:24:57.277 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.5389000000000004e-05 2024-04-20T17:24:57.277 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.277 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.277 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:24:57.277 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.111203+0000", 2024-04-20T17:24:57.277 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9429e-05 2024-04-20T17:24:57.277 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.277 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:24:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.132314+0000", 2024-04-20T17:24:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.021110776000000001 2024-04-20T17:24:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.137768+0000", 2024-04-20T17:24:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0054539979999999998 2024-04-20T17:24:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:24:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.137807+0000", 2024-04-20T17:24:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9480000000000001e-05 2024-04-20T17:24:57.278 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.279 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.279 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:24:57.279 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.137844+0000", 2024-04-20T17:24:57.279 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7067999999999998e-05 2024-04-20T17:24:57.279 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.279 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:24:57.279 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.279 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.279 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.279 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.25802.0:260106 3.6 3:6765069f:::benchmark_data_smithi171_110765_object116891:head [delete] snapc 0=[] ondisk+write+known_if_redirected e426)", 2024-04-20T17:24:57.279 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:23:46.099238+0000", 2024-04-20T17:24:57.279 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 71.084544871999995, 2024-04-20T17:24:57.279 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.045631457, 2024-04-20T17:24:57.279 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:24:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:24:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:24:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.25802", 2024-04-20T17:24:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 260106 2024-04-20T17:24:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:24:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:24:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.099238+0000", 2024-04-20T17:24:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.280 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.281 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:24:57.281 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.099238+0000", 2024-04-20T17:24:57.281 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.281 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.281 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.281 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:24:57.281 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.099239+0000", 2024-04-20T17:24:57.281 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0500000000000002e-07 2024-04-20T17:24:57.281 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.281 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.281 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:24:57.281 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.099242+0000", 2024-04-20T17:24:57.281 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3419999999999999e-06 2024-04-20T17:24:57.281 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.282 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.282 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:24:57.282 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.099251+0000", 2024-04-20T17:24:57.282 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0330000000000008e-06 2024-04-20T17:24:57.282 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.282 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.282 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:24:57.282 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.099256+0000", 2024-04-20T17:24:57.282 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4240000000000001e-06 2024-04-20T17:24:57.282 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.282 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.282 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.282 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.121138+0000", 2024-04-20T17:24:57.282 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.021882133000000002 2024-04-20T17:24:57.283 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.283 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:24:57.283 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.121259+0000", 2024-04-20T17:24:57.283 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000120702 2024-04-20T17:24:57.283 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.283 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:24:57.283 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.121389+0000", 2024-04-20T17:24:57.283 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013047300000000001 2024-04-20T17:24:57.283 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.283 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:24:57.284 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.137093+0000", 2024-04-20T17:24:57.284 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015704116000000001 2024-04-20T17:24:57.284 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.284 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.284 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:57.284 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.144759+0000", 2024-04-20T17:24:57.284 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0076659199999999997 2024-04-20T17:24:57.284 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.284 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.284 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:24:57.284 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.144821+0000", 2024-04-20T17:24:57.284 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1781999999999999e-05 2024-04-20T17:24:57.284 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.284 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.285 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:24:57.285 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:46.144870+0000", 2024-04-20T17:24:57.285 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8727000000000003e-05 2024-04-20T17:24:57.285 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.285 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:24:57.285 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.285 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.285 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.285 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.25802.0:269938 3.3 e427/356 3:c4c7647c:::benchmark_data_smithi171_110765_object126723:head v 427'16670, mlcod=427'16665)", 2024-04-20T17:24:57.285 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:23:54.437822+0000", 2024-04-20T17:24:57.286 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 62.745961178999998, 2024-04-20T17:24:57.286 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.045254892999999997, 2024-04-20T17:24:57.286 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:24:57.286 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:24:57.286 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:57.286 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:24:57.286 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.286 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:24:57.286 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:54.437822+0000", 2024-04-20T17:24:57.286 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.286 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.286 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.286 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:24:57.286 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:54.437822+0000", 2024-04-20T17:24:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:24:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:54.437824+0000", 2024-04-20T17:24:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8530000000000001e-06 2024-04-20T17:24:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:24:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:54.437833+0000", 2024-04-20T17:24:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2e-06 2024-04-20T17:24:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:24:57.287 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:54.437844+0000", 2024-04-20T17:24:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.102e-05 2024-04-20T17:24:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:24:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:54.437853+0000", 2024-04-20T17:24:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8910000000000001e-06 2024-04-20T17:24:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:54.448904+0000", 2024-04-20T17:24:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011050677 2024-04-20T17:24:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:24:57.288 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:54.448923+0000", 2024-04-20T17:24:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9468999999999999e-05 2024-04-20T17:24:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:24:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:54.483019+0000", 2024-04-20T17:24:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.034095731999999997 2024-04-20T17:24:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:24:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:54.483077+0000", 2024-04-20T17:24:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8050999999999997e-05 2024-04-20T17:24:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:24:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.290 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.290 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.25802.0:279875 3.3 e433/356 3:ca72fdb9:::benchmark_data_smithi171_110765_object136660:head v 433'17325, mlcod=433'17322)", 2024-04-20T17:24:57.290 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:24:05.826043+0000", 2024-04-20T17:24:57.290 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 51.357739866999999, 2024-04-20T17:24:57.290 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.047089842, 2024-04-20T17:24:57.290 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:24:57.290 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:24:57.290 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:57.290 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:24:57.290 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.290 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:24:57.290 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.826043+0000", 2024-04-20T17:24:57.290 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.290 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:24:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.826043+0000", 2024-04-20T17:24:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:24:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.826044+0000", 2024-04-20T17:24:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3e-07 2024-04-20T17:24:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:24:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.826051+0000", 2024-04-20T17:24:57.291 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4459999999999998e-06 2024-04-20T17:24:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:24:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.826057+0000", 2024-04-20T17:24:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6279999999999999e-06 2024-04-20T17:24:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:24:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.826063+0000", 2024-04-20T17:24:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6300000000000003e-06 2024-04-20T17:24:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.831626+0000", 2024-04-20T17:24:57.292 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0055627489999999996 2024-04-20T17:24:57.293 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.293 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.293 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:24:57.293 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.831649+0000", 2024-04-20T17:24:57.293 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3842999999999999e-05 2024-04-20T17:24:57.293 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.293 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.293 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:24:57.293 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.873090+0000", 2024-04-20T17:24:57.293 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.041441089 2024-04-20T17:24:57.293 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.293 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.293 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:24:57.293 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.873133+0000", 2024-04-20T17:24:57.294 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2726999999999999e-05 2024-04-20T17:24:57.294 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.294 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:24:57.294 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.294 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.294 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.294 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.25802.0:279870 3.b e433/356 3:d4358dcf:::benchmark_data_smithi171_110765_object136655:head v 433'17244, mlcod=433'17242)", 2024-04-20T17:24:57.294 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:24:05.826542+0000", 2024-04-20T17:24:57.294 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 51.357241457000001, 2024-04-20T17:24:57.294 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.046625753999999998, 2024-04-20T17:24:57.294 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:24:57.294 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:24:57.294 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:24:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:24:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.826542+0000", 2024-04-20T17:24:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:24:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.826542+0000", 2024-04-20T17:24:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:24:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.826543+0000", 2024-04-20T17:24:57.295 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7260000000000001e-06 2024-04-20T17:24:57.296 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.296 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:24:57.296 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.826552+0000", 2024-04-20T17:24:57.296 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.67e-06 2024-04-20T17:24:57.296 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.296 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:24:57.296 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.826569+0000", 2024-04-20T17:24:57.296 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7333000000000002e-05 2024-04-20T17:24:57.296 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.296 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:24:57.296 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.826577+0000", 2024-04-20T17:24:57.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5129999999999999e-06 2024-04-20T17:24:57.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.297 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.297 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.297 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.836620+0000", 2024-04-20T17:24:57.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010043345 2024-04-20T17:24:57.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.297 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.297 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:24:57.297 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.836642+0000", 2024-04-20T17:24:57.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1248000000000001e-05 2024-04-20T17:24:57.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.297 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.297 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:24:57.298 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.873139+0000", 2024-04-20T17:24:57.298 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.036497400999999999 2024-04-20T17:24:57.298 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.298 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.298 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:24:57.298 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.873167+0000", 2024-04-20T17:24:57.298 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8518e-05 2024-04-20T17:24:57.298 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.298 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:24:57.298 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.298 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.298 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.299 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.25802.0:279878 3.3 e433/356 3:c6c4d07a:::benchmark_data_smithi171_110765_object136663:head v 433'17326, mlcod=433'17322)", 2024-04-20T17:24:57.299 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:24:05.827702+0000", 2024-04-20T17:24:57.299 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 51.356081645000003, 2024-04-20T17:24:57.299 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.045488835999999998, 2024-04-20T17:24:57.299 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:24:57.299 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:24:57.299 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:57.299 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:24:57.299 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.299 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:24:57.299 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.827702+0000", 2024-04-20T17:24:57.299 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.299 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.299 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.300 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:24:57.300 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.827702+0000", 2024-04-20T17:24:57.300 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.300 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.300 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:24:57.300 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.827704+0000", 2024-04-20T17:24:57.300 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1409999999999999e-06 2024-04-20T17:24:57.300 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.300 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:24:57.300 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.827712+0000", 2024-04-20T17:24:57.300 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.422e-06 2024-04-20T17:24:57.300 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.301 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.301 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:24:57.301 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.827729+0000", 2024-04-20T17:24:57.301 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7193999999999998e-05 2024-04-20T17:24:57.301 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.301 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.301 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:24:57.301 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.827739+0000", 2024-04-20T17:24:57.301 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.3910000000000004e-06 2024-04-20T17:24:57.301 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.301 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.301 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.301 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.836820+0000", 2024-04-20T17:24:57.301 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0090812870000000004 2024-04-20T17:24:57.301 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.302 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.302 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:24:57.302 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.836839+0000", 2024-04-20T17:24:57.302 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8598e-05 2024-04-20T17:24:57.302 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.302 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.302 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:24:57.302 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.873169+0000", 2024-04-20T17:24:57.302 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.036330877999999997 2024-04-20T17:24:57.302 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.302 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.302 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:24:57.302 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:05.873190+0000", 2024-04-20T17:24:57.303 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0925e-05 2024-04-20T17:24:57.303 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.303 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:24:57.303 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.303 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.303 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.303 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.25802.0:284452 3.1 3:88d9a731:::benchmark_data_smithi171_110765_object141237:head [delete] snapc 0=[] ondisk+write+known_if_redirected e434)", 2024-04-20T17:24:57.303 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:24:09.575545+0000", 2024-04-20T17:24:57.303 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 47.608238147999998, 2024-04-20T17:24:57.303 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.034649923999999999, 2024-04-20T17:24:57.303 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:24:57.303 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:24:57.304 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:57.304 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:24:57.304 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.25802", 2024-04-20T17:24:57.304 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:24:57.304 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 284452 2024-04-20T17:24:57.304 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.304 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:24:57.304 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.304 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:24:57.304 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:09.575545+0000", 2024-04-20T17:24:57.304 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.304 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.304 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.305 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:24:57.305 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:09.575545+0000", 2024-04-20T17:24:57.305 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.305 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.305 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:24:57.305 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:09.575545+0000", 2024-04-20T17:24:57.305 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3000000000000002e-07 2024-04-20T17:24:57.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.305 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.305 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:24:57.305 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:09.575547+0000", 2024-04-20T17:24:57.305 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9420000000000002e-06 2024-04-20T17:24:57.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.306 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:24:57.306 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:09.575550+0000", 2024-04-20T17:24:57.306 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.898e-06 2024-04-20T17:24:57.306 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.306 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:24:57.306 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:09.575553+0000", 2024-04-20T17:24:57.306 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7240000000000001e-06 2024-04-20T17:24:57.306 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.306 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.306 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:09.589619+0000", 2024-04-20T17:24:57.306 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014066468 2024-04-20T17:24:57.307 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.307 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.307 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:24:57.307 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:09.589729+0000", 2024-04-20T17:24:57.307 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000110041 2024-04-20T17:24:57.307 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.307 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.307 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 4", 2024-04-20T17:24:57.307 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:09.589838+0000", 2024-04-20T17:24:57.307 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000108291 2024-04-20T17:24:57.307 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.307 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.307 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:24:57.308 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:09.604274+0000", 2024-04-20T17:24:57.308 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014436681999999999 2024-04-20T17:24:57.308 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.308 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.308 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:24:57.308 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:09.610103+0000", 2024-04-20T17:24:57.308 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0058283939999999998 2024-04-20T17:24:57.308 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.308 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.308 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:24:57.308 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:09.610153+0000", 2024-04-20T17:24:57.308 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9948999999999999e-05 2024-04-20T17:24:57.308 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.308 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.309 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:24:57.309 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:09.610195+0000", 2024-04-20T17:24:57.309 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2205e-05 2024-04-20T17:24:57.309 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.309 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:24:57.309 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.309 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.309 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.309 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.25802.0:286093 3.b e436/356 3:d74670af:::benchmark_data_smithi171_110765_object142878:head v 436'17630, mlcod=436'17628)", 2024-04-20T17:24:57.309 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:24:11.837884+0000", 2024-04-20T17:24:57.309 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 45.345898794999997, 2024-04-20T17:24:57.309 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.035743883999999997, 2024-04-20T17:24:57.309 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:24:57.309 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:24:57.310 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:24:57.310 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:24:57.310 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.310 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:24:57.310 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:11.837884+0000", 2024-04-20T17:24:57.310 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.310 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.310 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.310 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:24:57.310 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:11.837884+0000", 2024-04-20T17:24:57.310 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:24:57.310 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.310 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.311 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:24:57.311 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:11.837885+0000", 2024-04-20T17:24:57.311 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.051e-06 2024-04-20T17:24:57.311 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.311 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.311 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:24:57.311 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:11.837893+0000", 2024-04-20T17:24:57.311 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8909999999999995e-06 2024-04-20T17:24:57.311 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.311 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.311 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:24:57.311 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:11.837901+0000", 2024-04-20T17:24:57.311 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4699999999999996e-06 2024-04-20T17:24:57.311 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.312 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.312 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:24:57.312 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:11.837907+0000", 2024-04-20T17:24:57.312 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.122e-06 2024-04-20T17:24:57.312 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.312 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.312 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:24:57.312 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:11.838111+0000", 2024-04-20T17:24:57.312 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000203718 2024-04-20T17:24:57.312 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.312 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.312 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:24:57.312 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:11.838129+0000", 2024-04-20T17:24:57.313 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8787000000000002e-05 2024-04-20T17:24:57.313 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.313 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.313 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:24:57.313 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:11.873583+0000", 2024-04-20T17:24:57.313 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.035453747000000001 2024-04-20T17:24:57.313 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:24:57.313 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:24:57.313 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:24:57.313 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:24:11.873628+0000", 2024-04-20T17:24:57.313 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5098000000000001e-05 2024-04-20T17:24:57.313 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.313 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:24:57.313 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.313 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:24:57.314 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:24:57.314 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:24:57.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:57 smithi005 ceph-mon[39562]: osdmap e453: 12 total, 12 up, 12 in 2024-04-20T17:24:57.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:57 smithi005 ceph-mon[39562]: pgmap v1208: 17 pgs: 17 active+clean; 1.7 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 216 MiB/s wr, 3.46k op/s 2024-04-20T17:24:57.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:57 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3775249667' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:24:57.464 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 9 16 37854 37838 262.728 207.25 0.00494149 0.00380403 2024-04-20T17:24:57.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:57 smithi106 ceph-mon[44423]: osdmap e453: 12 total, 12 up, 12 in 2024-04-20T17:24:57.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:57 smithi106 ceph-mon[44423]: pgmap v1208: 17 pgs: 17 active+clean; 1.7 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 216 MiB/s wr, 3.46k op/s 2024-04-20T17:24:57.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:57 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3775249667' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:24:57.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:57 smithi171 ceph-mon[44189]: osdmap e453: 12 total, 12 up, 12 in 2024-04-20T17:24:57.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:57 smithi171 ceph-mon[44189]: pgmap v1208: 17 pgs: 17 active+clean; 1.7 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 216 MiB/s wr, 3.46k op/s 2024-04-20T17:24:57.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:57 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3775249667' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:24:57.728 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:24:57.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:57.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:24:57.834 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:24:57.882 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:58.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:58 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:24:58.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:58 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:24:58.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:58 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3775249667' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-20T17:24:58.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:58 smithi005 ceph-mon[39562]: osdmap e454: 12 total, 12 up, 12 in 2024-04-20T17:24:58.446 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:24:58.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:58.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:24:58.464 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 10 16 41235 41219 257.583 211.312 0.00263233 0.00388059 2024-04-20T17:24:58.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:58 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:24:58.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:58 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:24:58.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:58 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3775249667' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-20T17:24:58.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:58 smithi106 ceph-mon[44423]: osdmap e454: 12 total, 12 up, 12 in 2024-04-20T17:24:58.552 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:24:58.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:58 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:24:58.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:58 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:24:58.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:58 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3775249667' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-20T17:24:58.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:58 smithi171 ceph-mon[44189]: osdmap e454: 12 total, 12 up, 12 in 2024-04-20T17:24:58.586 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:58.612 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:24:58.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:58.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:24:58.717 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:24:58.763 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:58.985 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 4 1 2024-04-20T17:24:59.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:24:59 smithi005 ceph-mon[39562]: pgmap v1210: 17 pgs: 17 active+clean; 2.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 256 MiB/s wr, 4.10k op/s 2024-04-20T17:24:59.464 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 11 16 44329 44313 251.745 193.375 0.00504649 0.00397068 2024-04-20T17:24:59.502 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:24:59.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:24:59 smithi106 ceph-mon[44423]: pgmap v1210: 17 pgs: 17 active+clean; 2.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 256 MiB/s wr, 4.10k op/s 2024-04-20T17:24:59.529 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:24:59.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:59.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:24:59.549 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-20T17:24:59.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:24:59 smithi171 ceph-mon[44189]: pgmap v1210: 17 pgs: 17 active+clean; 2.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 256 MiB/s wr, 4.10k op/s 2024-04-20T17:24:59.635 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:24:59.674 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:59.698 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:24:59.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:59.701 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:24:59.803 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:24:59.843 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:24:59.871 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:24:59.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:24:59.875 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:24:59.976 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:25:00.028 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:00.112 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:25:00.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:00.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:25:00.219 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:25:00.273 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:00.309 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:25:00.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:00.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:25:00.419 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:25:00.458 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:00.464 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 12 16 47065 47049 245.015 171 0.00405532 0.00407961 2024-04-20T17:25:00.482 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:25:00.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:00.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:25:00.588 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:25:00.633 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:00.658 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:25:00.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:00.661 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:25:00.763 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:25:00.804 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:01.178 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:25:01.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:01.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:25:01.234 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:25:01.267 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:25:01.290 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:25:01.293 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:25:01.326 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:01.361 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:25:01.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:01.363 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:25:01.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:01 smithi005 ceph-mon[39562]: pgmap v1211: 17 pgs: 17 active+clean; 2.2 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 229 MiB/s wr, 3.67k op/s 2024-04-20T17:25:01.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:01 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2882420752' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:25:01.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:01 smithi005 ceph-mon[39562]: from='client.26089 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:25:01.463 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:25:01.464 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 13 16 49874 49858 239.672 175.562 0.00324118 0.00417059 2024-04-20T17:25:01.503 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:01.528 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:25:01.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:01.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:25:01.532 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:01 smithi106 ceph-mon[44423]: pgmap v1211: 17 pgs: 17 active+clean; 2.2 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 229 MiB/s wr, 3.67k op/s 2024-04-20T17:25:01.532 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:01 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2882420752' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:25:01.532 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:01 smithi106 ceph-mon[44423]: from='client.26089 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:25:01.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:01 smithi171 ceph-mon[44189]: pgmap v1211: 17 pgs: 17 active+clean; 2.2 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 229 MiB/s wr, 3.67k op/s 2024-04-20T17:25:01.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:01 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2882420752' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:25:01.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:01 smithi171 ceph-mon[44189]: from='client.26089 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:25:01.633 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:25:01.678 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:01.719 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:25:01.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:01.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:25:01.828 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:25:01.868 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:01.895 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:25:01.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:01.899 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:25:02.000 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:25:02.040 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:02.060 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:25:02.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:02.063 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:25:02.164 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:25:02.197 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:02.221 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:25:02.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:02.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:25:02.326 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:25:02.358 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:02.386 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:25:02.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:02.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:25:02.464 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 14 16 52531 52515 234.413 166.062 0.00779224 0.0042646 2024-04-20T17:25:02.491 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:25:02.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:02 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:25:02.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:02 smithi106 ceph-mon[44423]: from='client.26089 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:25:02.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:02 smithi106 ceph-mon[44423]: osdmap e455: 12 total, 12 up, 12 in 2024-04-20T17:25:02.525 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:02.554 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:25:02.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:02.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:25:02.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:02 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:25:02.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:02 smithi171 ceph-mon[44189]: from='client.26089 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:25:02.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:02 smithi171 ceph-mon[44189]: osdmap e455: 12 total, 12 up, 12 in 2024-04-20T17:25:02.661 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:25:02.702 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:02.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:02 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:25:02.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:02 smithi005 ceph-mon[39562]: from='client.26089 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:25:02.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:02 smithi005 ceph-mon[39562]: osdmap e455: 12 total, 12 up, 12 in 2024-04-20T17:25:02.734 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:25:02.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:02.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:25:02.838 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:25:02.879 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:02.906 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:25:02.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:02.909 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:25:03.010 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:25:03.050 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:03.279 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:03 smithi171 ceph-mon[44189]: pgmap v1213: 17 pgs: 17 active+clean; 2.7 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 196 MiB/s wr, 3.13k op/s 2024-04-20T17:25:03.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:03 smithi005 ceph-mon[39562]: pgmap v1213: 17 pgs: 17 active+clean; 2.7 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 196 MiB/s wr, 3.13k op/s 2024-04-20T17:25:03.464 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 15 16 55338 55322 230.481 175.438 0.00328175 0.00433719 2024-04-20T17:25:03.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:03 smithi106 ceph-mon[44423]: pgmap v1213: 17 pgs: 17 active+clean; 2.7 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 196 MiB/s wr, 3.13k op/s 2024-04-20T17:25:04.464 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 16 16 57410 57394 224.169 129.5 0.00236121 0.0044592 2024-04-20T17:25:04.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:04 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:25:04.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:04 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:25:04.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:04 smithi171 ceph-mon[44189]: osdmap e456: 12 total, 12 up, 12 in 2024-04-20T17:25:04.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:04 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:25:04.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:04 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:25:04.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:04 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:25:04.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:04 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:25:04.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:04 smithi005 ceph-mon[39562]: osdmap e456: 12 total, 12 up, 12 in 2024-04-20T17:25:04.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:04 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:25:04.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:04 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:25:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:04 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:25:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:04 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:25:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:04 smithi106 ceph-mon[44423]: osdmap e456: 12 total, 12 up, 12 in 2024-04-20T17:25:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:04 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:25:04.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:04 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:25:05.010 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:25:05.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:05.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:25:05.114 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:25:05.147 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:05.172 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:25:05.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:05.177 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:25:05.278 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:25:05.289 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:05 smithi005 ceph-mon[39562]: pgmap v1215: 17 pgs: 17 active+clean; 3.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 202 MiB/s wr, 3.24k op/s 2024-04-20T17:25:05.289 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:05 smithi005 ceph-mon[39562]: osdmap e457: 12 total, 12 up, 12 in 2024-04-20T17:25:05.320 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:05.465 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 17 15 59814 59799 219.823 150.312 0.0056851 0.00454741 2024-04-20T17:25:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:05 smithi171 ceph-mon[44189]: pgmap v1215: 17 pgs: 17 active+clean; 3.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 202 MiB/s wr, 3.24k op/s 2024-04-20T17:25:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:05 smithi171 ceph-mon[44189]: osdmap e457: 12 total, 12 up, 12 in 2024-04-20T17:25:05.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:05 smithi106 ceph-mon[44423]: pgmap v1215: 17 pgs: 17 active+clean; 3.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 202 MiB/s wr, 3.24k op/s 2024-04-20T17:25:05.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:05 smithi106 ceph-mon[44423]: osdmap e457: 12 total, 12 up, 12 in 2024-04-20T17:25:06.465 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 18 16 62446 62430 216.745 164.438 0.0027034 0.00461274 2024-04-20T17:25:07.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:07 smithi005 ceph-mon[39562]: osdmap e458: 12 total, 12 up, 12 in 2024-04-20T17:25:07.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:07 smithi005 ceph-mon[39562]: pgmap v1218: 17 pgs: 17 active+clean; 3.4 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 149 MiB/s wr, 2.39k op/s 2024-04-20T17:25:07.465 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 19 16 64920 64904 213.475 154.625 0.00629227 0.00468296 2024-04-20T17:25:07.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:07 smithi106 ceph-mon[44423]: osdmap e458: 12 total, 12 up, 12 in 2024-04-20T17:25:07.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:07 smithi106 ceph-mon[44423]: pgmap v1218: 17 pgs: 17 active+clean; 3.4 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 149 MiB/s wr, 2.39k op/s 2024-04-20T17:25:07.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:07 smithi171 ceph-mon[44189]: osdmap e458: 12 total, 12 up, 12 in 2024-04-20T17:25:07.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:07 smithi171 ceph-mon[44189]: pgmap v1218: 17 pgs: 17 active+clean; 3.4 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 149 MiB/s wr, 2.39k op/s 2024-04-20T17:25:07.983 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:25:07.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:07.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:25:08.098 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:25:08.137 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:08.361 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:25:08.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:08.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:25:08.465 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:25:08.466040 min lat: 0.00113144 max lat: 0.214111 avg lat: 0.00475601 2024-04-20T17:25:08.465 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:25:08.465 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 20 15 67293 67278 210.22 148.375 0.00572965 0.00475601 2024-04-20T17:25:08.469 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:25:08.502 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:08.530 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:25:08.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:08.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:25:08.638 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:25:08.671 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:08.690 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:25:08.692 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:25:08.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:08.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:25:08.729 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:25:08.760 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:25:08.798 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:25:08.803 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:25:08.831 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:25:08.835 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:08.860 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:25:08.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:08.863 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:25:08.863 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:25:08.893 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:25:08.927 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:25:08.960 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:25:08.964 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:25:09.001 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:25:09.002 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:09.026 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:25:09.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:09.032 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:25:09.033 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:25:09.068 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:25:09.077 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:25:09.077 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:25:09.077 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:25:09.077 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:25:09.077 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:25:09.077 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:25:09.078 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:25:09.078 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:25:09.078 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:25:09.078 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:25:09.078 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:25:09.078 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:25:09.133 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:25:09.166 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:09.194 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:25:09.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:09.199 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:25:09.300 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:25:09.335 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:09.366 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:25:09.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:09.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:25:09.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:09 smithi005 ceph-mon[39562]: pgmap v1219: 17 pgs: 17 active+clean; 3.8 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 187 MiB/s wr, 2.98k op/s 2024-04-20T17:25:09.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:09 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4237350996' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:25:09.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:09.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:09.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:09.465 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 21 16 69443 69427 206.604 134.312 0.00462774 0.00483869 2024-04-20T17:25:09.471 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:25:09.503 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:09 smithi171 ceph-mon[44189]: pgmap v1219: 17 pgs: 17 active+clean; 3.8 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 187 MiB/s wr, 2.98k op/s 2024-04-20T17:25:09.504 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:09 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4237350996' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:25:09.504 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:09.504 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:09.504 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:09.506 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:09.532 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:25:09.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:09.536 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:25:09.568 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:25:09.637 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:25:09.675 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:09.700 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:25:09.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:09.703 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:25:09.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:09 smithi106 ceph-mon[44423]: pgmap v1219: 17 pgs: 17 active+clean; 3.8 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 187 MiB/s wr, 2.98k op/s 2024-04-20T17:25:09.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:09 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4237350996' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:25:09.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:09.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:09.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:09.805 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:25:09.865 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:10.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:10.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:10.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:10.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:10.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:10.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:10.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:10.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:10.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:10.465 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 22 16 71668 71652 203.534 139.062 0.00389424 0.00491146 2024-04-20T17:25:10.542 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:25:10.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:10.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:25:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:10.649 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:25:10.678 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:10.678 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:10.678 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:10.678 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:10.678 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:10.678 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:10.678 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:10.678 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:10.679 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:10.680 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:10.710 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:25:10.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:10.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:25:10.816 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:25:10.870 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:11.213 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:25:11.245 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:25:11.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:11.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:25:11.269 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:25:11.270 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:25:11.361 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:25:11.401 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:11.427 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:25:11.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:11.430 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:25:11.465 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 23 16 73768 73752 200.391 131.25 0.00564676 0.00498908 2024-04-20T17:25:11.515 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:11 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:25:11.515 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:11 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:25:11.515 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:11 smithi005 ceph-mon[39562]: pgmap v1220: 17 pgs: 17 active+clean; 3.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 127 MiB/s wr, 2.03k op/s 2024-04-20T17:25:11.516 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:11 smithi005 ceph-mon[39562]: osdmap e459: 12 total, 12 up, 12 in 2024-04-20T17:25:11.516 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:11 smithi005 ceph-mon[39562]: osdmap e460: 12 total, 12 up, 12 in 2024-04-20T17:25:11.531 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:25:11.563 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:11 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:25:11.563 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:11 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:25:11.563 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:11 smithi106 ceph-mon[44423]: pgmap v1220: 17 pgs: 17 active+clean; 3.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 127 MiB/s wr, 2.03k op/s 2024-04-20T17:25:11.563 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:11 smithi106 ceph-mon[44423]: osdmap e459: 12 total, 12 up, 12 in 2024-04-20T17:25:11.563 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:11 smithi106 ceph-mon[44423]: osdmap e460: 12 total, 12 up, 12 in 2024-04-20T17:25:11.565 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:11.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:11 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:25:11.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:11 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:25:11.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:11 smithi171 ceph-mon[44189]: pgmap v1220: 17 pgs: 17 active+clean; 3.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 127 MiB/s wr, 2.03k op/s 2024-04-20T17:25:11.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:11 smithi171 ceph-mon[44189]: osdmap e459: 12 total, 12 up, 12 in 2024-04-20T17:25:11.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:11 smithi171 ceph-mon[44189]: osdmap e460: 12 total, 12 up, 12 in 2024-04-20T17:25:11.589 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:25:11.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:11.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:25:11.696 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:25:11.728 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:11.756 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:25:11.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:11.757 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:25:11.858 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:25:11.926 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:12.284 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.4 to 1 (10000) 2024-04-20T17:25:12.465 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 24 16 75757 75741 197.221 124.312 0.00933775 0.00506937 2024-04-20T17:25:12.550 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:25:12.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:12 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1803308265' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-20T17:25:12.594 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:12 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1803308265' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-20T17:25:12.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:12 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1803308265' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-20T17:25:13.465 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 25 15 77991 77976 194.919 139.688 0.0118054 0.00512902 2024-04-20T17:25:13.539 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:13 smithi005 ceph-mon[39562]: pgmap v1223: 17 pgs: 17 active+clean; 4.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 139 MiB/s wr, 2.22k op/s 2024-04-20T17:25:13.539 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:13 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1803308265' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-20T17:25:13.539 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:13 smithi005 ceph-mon[39562]: osdmap e461: 12 total, 12 up, 12 in 2024-04-20T17:25:13.540 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 5 1 2024-04-20T17:25:13.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:13 smithi171 ceph-mon[44189]: pgmap v1223: 17 pgs: 17 active+clean; 4.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 139 MiB/s wr, 2.22k op/s 2024-04-20T17:25:13.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:13 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1803308265' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-20T17:25:13.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:13 smithi171 ceph-mon[44189]: osdmap e461: 12 total, 12 up, 12 in 2024-04-20T17:25:13.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:13 smithi106 ceph-mon[44423]: pgmap v1223: 17 pgs: 17 active+clean; 4.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 139 MiB/s wr, 2.22k op/s 2024-04-20T17:25:13.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:13 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1803308265' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-20T17:25:13.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:13 smithi106 ceph-mon[44423]: osdmap e461: 12 total, 12 up, 12 in 2024-04-20T17:25:13.934 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:25:13.937 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-20T17:25:13.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:13.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:25:14.040 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:25:14.073 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:14.095 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:25:14.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:14.101 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:25:14.202 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:25:14.235 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:14.260 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:25:14.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:14.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:25:14.364 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:25:14.414 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:14.465 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 26 16 79968 79952 192.172 123.5 0.00636295 0.00520153 2024-04-20T17:25:14.519 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:25:14.524 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:25:14.535 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:25:14.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:14.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:25:14.593 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:25:14.644 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:25:14.689 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:14.821 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:25:14.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:14.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:25:14.932 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:25:14.970 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:14.995 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:25:14.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:14.999 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:25:15.100 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:25:15.148 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:15.301 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:25:15.455 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:25:15.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:15.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:25:15.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:15 smithi005 ceph-mon[39562]: pgmap v1225: 17 pgs: 17 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 126 MiB/s wr, 2.01k op/s 2024-04-20T17:25:15.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:15 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3315533103' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:25:15.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:15 smithi005 ceph-mon[39562]: from='client.26174 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:25:15.465 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 27 16 81886 81870 189.494 119.875 0.00788706 0.00527557 2024-04-20T17:25:15.561 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:25:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:15 smithi171 ceph-mon[44189]: pgmap v1225: 17 pgs: 17 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 126 MiB/s wr, 2.01k op/s 2024-04-20T17:25:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:15 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3315533103' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:25:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:15 smithi171 ceph-mon[44189]: from='client.26174 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:25:15.597 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:15.624 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:25:15.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:15.628 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:25:15.730 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:25:15.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:15 smithi106 ceph-mon[44423]: pgmap v1225: 17 pgs: 17 active+clean; 4.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 126 MiB/s wr, 2.01k op/s 2024-04-20T17:25:15.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:15 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3315533103' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:25:15.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:15 smithi106 ceph-mon[44423]: from='client.26174 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:25:15.764 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:15.789 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:25:15.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:15.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:25:15.894 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:25:15.929 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:15.956 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:25:15.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:15.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:25:16.060 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:25:16.109 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:16.130 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:25:16.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:16.134 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:25:16.234 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:25:16.308 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:16.317 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:16 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:25:16.318 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:16 smithi171 ceph-mon[44189]: from='client.26174 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:25:16.318 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:16 smithi171 ceph-mon[44189]: osdmap e462: 12 total, 12 up, 12 in 2024-04-20T17:25:16.466 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 28 16 83945 83929 187.32 128.688 0.00314016 0.00533731 2024-04-20T17:25:16.471 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:25:16.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:16.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:25:16.509 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:16 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:25:16.509 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:16 smithi005 ceph-mon[39562]: from='client.26174 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:25:16.509 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:16 smithi005 ceph-mon[39562]: osdmap e462: 12 total, 12 up, 12 in 2024-04-20T17:25:16.577 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:25:16.612 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:16.647 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:25:16.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:16.651 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:25:16.752 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:25:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:16 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:25:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:16 smithi106 ceph-mon[44423]: from='client.26174 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:25:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:16 smithi106 ceph-mon[44423]: osdmap e462: 12 total, 12 up, 12 in 2024-04-20T17:25:16.807 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:16.839 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:25:16.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:16.844 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:25:16.945 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:25:16.985 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:17.015 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:25:17.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:17.017 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:25:17.118 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:25:17.156 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:17.183 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:25:17.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:17.187 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:25:17.288 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:25:17.334 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:17.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:17 smithi005 ceph-mon[39562]: pgmap v1227: 17 pgs: 17 active+clean; 4.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 172 MiB/s wr, 2.74k op/s 2024-04-20T17:25:17.466 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 29 16 85993 85977 185.274 128 0.00639901 0.00539588 2024-04-20T17:25:17.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:17 smithi171 ceph-mon[44189]: pgmap v1227: 17 pgs: 17 active+clean; 4.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 172 MiB/s wr, 2.74k op/s 2024-04-20T17:25:17.598 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:25:17.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:17.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:25:17.708 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:25:17.758 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:17.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:17 smithi106 ceph-mon[44423]: pgmap v1227: 17 pgs: 17 active+clean; 4.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 172 MiB/s wr, 2.74k op/s 2024-04-20T17:25:18.289 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:25:18.466 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 30 16 87970 87954 183.217 123.562 0.00515551 0.00545602 2024-04-20T17:25:19.466 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 31 16 89877 89861 181.151 119.188 0.00339423 0.0055188 2024-04-20T17:25:19.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:19 smithi005 ceph-mon[39562]: pgmap v1228: 17 pgs: 17 active+clean; 5.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 138 MiB/s wr, 2.20k op/s 2024-04-20T17:25:19.730 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:25:19.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:19.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:25:19.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:19 smithi106 ceph-mon[44423]: pgmap v1228: 17 pgs: 17 active+clean; 5.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 138 MiB/s wr, 2.20k op/s 2024-04-20T17:25:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:19 smithi171 ceph-mon[44189]: pgmap v1228: 17 pgs: 17 active+clean; 5.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 138 MiB/s wr, 2.20k op/s 2024-04-20T17:25:19.844 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:25:19.897 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:19.929 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:25:19.929 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:25:19.929 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:25:19.929 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:25:19.929 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:25:19.929 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:25:19.929 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:25:19.929 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:25:19.929 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:25:19.929 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:25:19.929 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:25:19.931 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:25:19.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:19.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:25:19.936 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:25:19.943 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:25:19.943 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:25:19.943 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:25:19.943 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:25:19.943 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:25:19.943 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:25:19.943 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:25:19.943 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:25:19.943 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:25:19.943 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:25:19.944 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:25:19.944 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:25:20.035 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:25:20.069 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:20.099 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:25:20.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:20.107 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:25:20.208 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:25:20.258 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:20.351 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.5 to 1 (10000) 2024-04-20T17:25:20.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:20 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1213802953' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:25:20.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:20.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:20 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/928114305' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:25:20.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:20 smithi005 ceph-mon[39562]: from='client.26180 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:25:20.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:20.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:20.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:20.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:20.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:20.466 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 32 15 91718 91703 179.088 115.125 0.00494733 0.00558199 2024-04-20T17:25:20.709 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:25:20.709 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:25:20.709 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:25:20.709 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:25:20.709 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:25:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:20 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1213802953' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:25:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:20 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/928114305' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:25:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:20 smithi106 ceph-mon[44423]: from='client.26180 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:25:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:20.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:20 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1213802953' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:25:20.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:20.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:20 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/928114305' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:25:20.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:20 smithi171 ceph-mon[44189]: from='client.26180 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:25:20.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:20.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:20.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:20.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:20.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:21.349 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 6 1 2024-04-20T17:25:21.466 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 33 16 93631 93615 177.282 119.5 0.00719704 0.00563907 2024-04-20T17:25:21.527 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:25:21.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:21.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:25:21.631 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:25:21.662 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:21.662 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:21.662 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:21.663 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:21.663 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:21.663 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:21 smithi171 ceph-mon[44189]: pgmap v1229: 17 pgs: 17 active+clean; 5.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 121 MiB/s wr, 1.93k op/s 2024-04-20T17:25:21.663 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:21 smithi171 ceph-mon[44189]: from='client.26180 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-20T17:25:21.663 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:21 smithi171 ceph-mon[44189]: osdmap e463: 12 total, 12 up, 12 in 2024-04-20T17:25:21.663 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:21 smithi171 ceph-mon[44189]: osdmap e464: 12 total, 12 up, 12 in 2024-04-20T17:25:21.664 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:21.675 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:21.675 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:21.675 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:21.676 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:21.676 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:21.676 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:21 smithi005 ceph-mon[39562]: pgmap v1229: 17 pgs: 17 active+clean; 5.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 121 MiB/s wr, 1.93k op/s 2024-04-20T17:25:21.676 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:21 smithi005 ceph-mon[39562]: from='client.26180 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-20T17:25:21.676 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:21 smithi005 ceph-mon[39562]: osdmap e463: 12 total, 12 up, 12 in 2024-04-20T17:25:21.676 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:21 smithi005 ceph-mon[39562]: osdmap e464: 12 total, 12 up, 12 in 2024-04-20T17:25:21.692 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:25:21.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:21.697 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:25:21.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:21.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:21.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:21.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:21.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:21.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:21 smithi106 ceph-mon[44423]: pgmap v1229: 17 pgs: 17 active+clean; 5.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 121 MiB/s wr, 1.93k op/s 2024-04-20T17:25:21.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:21 smithi106 ceph-mon[44423]: from='client.26180 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-20T17:25:21.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:21 smithi106 ceph-mon[44423]: osdmap e463: 12 total, 12 up, 12 in 2024-04-20T17:25:21.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:21 smithi106 ceph-mon[44423]: osdmap e464: 12 total, 12 up, 12 in 2024-04-20T17:25:21.798 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:25:21.835 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:21.863 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:25:21.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:21.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:25:21.969 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:25:22.038 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-20T17:25:22.060 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:22.084 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:25:22.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:22.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:25:22.195 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:25:22.258 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:22.467 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 34 16 95368 95352 175.256 108.562 0.00428285 0.00570447 2024-04-20T17:25:22.633 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:25:22.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:22.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:25:22.670 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:25:22.692 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:25:22.738 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:25:22.746 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:25:22.777 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:22.802 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:25:22.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:22.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:25:22.906 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:25:22.941 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:22.967 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:25:22.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:22.970 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:25:23.071 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:25:23.119 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:23.258 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:25:23.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:23.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:25:23.364 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:25:23.400 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:23.423 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:25:23.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:23.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:25:23.467 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 35 15 97158 97143 173.447 111.938 0.00464443 0.00576416 2024-04-20T17:25:23.527 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:25:23.567 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:23 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:25:23.567 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:23 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:25:23.567 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:23 smithi005 ceph-mon[39562]: pgmap v1232: 17 pgs: 17 active+clean; 5.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s wr, 2.17k op/s 2024-04-20T17:25:23.567 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:23 smithi005 ceph-mon[39562]: osdmap e465: 12 total, 12 up, 12 in 2024-04-20T17:25:23.568 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:23.696 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:25:23.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:23.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:25:23.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:23 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:25:23.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:23 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:25:23.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:23 smithi106 ceph-mon[44423]: pgmap v1232: 17 pgs: 17 active+clean; 5.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s wr, 2.17k op/s 2024-04-20T17:25:23.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:23 smithi106 ceph-mon[44423]: osdmap e465: 12 total, 12 up, 12 in 2024-04-20T17:25:23.802 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:25:23.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:23 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:25:23.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:23 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:25:23.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:23 smithi171 ceph-mon[44189]: pgmap v1232: 17 pgs: 17 active+clean; 5.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s wr, 2.17k op/s 2024-04-20T17:25:23.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:23 smithi171 ceph-mon[44189]: osdmap e465: 12 total, 12 up, 12 in 2024-04-20T17:25:23.839 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:23.866 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:25:23.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:23.875 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:25:23.976 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:25:24.009 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:24.036 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:25:24.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:24.040 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:25:24.142 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:25:24.184 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:24.221 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:25:24.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:24.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:25:24.331 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:25:24.365 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:24.394 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:25:24.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:24.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:25:24.467 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 36 16 99036 99020 171.888 117.312 0.0051059 0.00581643 2024-04-20T17:25:24.498 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:25:24.537 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:24.566 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:25:24.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:24.570 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:25:24.671 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:25:24.707 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:25:24.711 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:24.742 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:25:24.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:24.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:25:24.847 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:25:24.886 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:24.912 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:25:24.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:24.918 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:25:25.019 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:25:25.053 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:25.080 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:25:25.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:25.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:25:25.184 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:25:25.231 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:25.273 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:25:25.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:25.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:25:25.382 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:25:25.431 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:25.467 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 37 16 100759 100743 170.152 107.688 0.00989294 0.00587559 2024-04-20T17:25:25.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:25 smithi005 ceph-mon[39562]: pgmap v1234: 17 pgs: 17 active+clean; 5.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.70k op/s 2024-04-20T17:25:25.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:25 smithi106 ceph-mon[44423]: pgmap v1234: 17 pgs: 17 active+clean; 5.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.70k op/s 2024-04-20T17:25:25.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:25 smithi171 ceph-mon[44189]: pgmap v1234: 17 pgs: 17 active+clean; 5.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.70k op/s 2024-04-20T17:25:26.076 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:25:26.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:26.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:25:26.146 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.6 to 1 (10000) 2024-04-20T17:25:26.182 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:25:26.245 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:26.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:26 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2520313833' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-20T17:25:26.370 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:26 smithi005 ceph-mon[39562]: from='client.26189 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-20T17:25:26.370 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:26 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:25:26.370 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:26 smithi005 ceph-mon[39562]: from='client.26189 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-20T17:25:26.370 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:26 smithi005 ceph-mon[39562]: osdmap e466: 12 total, 12 up, 12 in 2024-04-20T17:25:26.467 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 38 16 102381 102365 168.342 101.375 0.00964534 0.00593892 2024-04-20T17:25:26.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:26 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2520313833' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-20T17:25:26.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:26 smithi106 ceph-mon[44423]: from='client.26189 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-20T17:25:26.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:26 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:25:26.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:26 smithi106 ceph-mon[44423]: from='client.26189 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-20T17:25:26.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:26 smithi106 ceph-mon[44423]: osdmap e466: 12 total, 12 up, 12 in 2024-04-20T17:25:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:26 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2520313833' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-20T17:25:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:26 smithi171 ceph-mon[44189]: from='client.26189 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-20T17:25:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:26 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:25:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:26 smithi171 ceph-mon[44189]: from='client.26189 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-20T17:25:26.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:26 smithi171 ceph-mon[44189]: osdmap e466: 12 total, 12 up, 12 in 2024-04-20T17:25:27.467 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 39 16 104063 104047 166.721 105.125 0.00784812 0.00599682 2024-04-20T17:25:27.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:27 smithi005 ceph-mon[39562]: pgmap v1236: 17 pgs: 17 active+clean; 6.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 137 MiB/s wr, 2.19k op/s 2024-04-20T17:25:27.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:27 smithi106 ceph-mon[44423]: pgmap v1236: 17 pgs: 17 active+clean; 6.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 137 MiB/s wr, 2.19k op/s 2024-04-20T17:25:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:27 smithi171 ceph-mon[44189]: pgmap v1236: 17 pgs: 17 active+clean; 6.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 137 MiB/s wr, 2.19k op/s 2024-04-20T17:25:27.882 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 8 1 2024-04-20T17:25:28.124 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:25:28.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:28.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:25:28.229 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:25:28.270 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:28.303 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:25:28.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:28.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:25:28.413 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:25:28.444 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:28.468 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:25:28.468851 min lat: 0.00113144 max lat: 0.214111 avg lat: 0.00605262 2024-04-20T17:25:28.468 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:25:28.468 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 40 16 105735 105719 165.165 104.5 0.021376 0.00605262 2024-04-20T17:25:28.477 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:25:28.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:28.480 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:25:28.581 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:25:28.615 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:28.642 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:25:28.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:28.645 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:25:28.746 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:25:28.776 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:28.800 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:25:28.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:28.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:25:28.904 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:25:28.933 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:25:28.933 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:28.956 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:25:28.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:28.964 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:25:28.988 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:25:28.988 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:25:28.988 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:25:28.988 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:25:28.988 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:28.988 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.25802.0:243442 3.4 3:25fd1bc9:::benchmark_data_smithi171_110765_object100227:head [delete] snapc 0=[] ondisk+write+known_if_redirected e416)", 2024-04-20T17:25:28.988 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:23:29.372681+0000", 2024-04-20T17:25:28.989 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 119.60732574799999, 2024-04-20T17:25:28.989 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.096654121999999995, 2024-04-20T17:25:28.989 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:25:28.989 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:25:28.989 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:25:28.989 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:25:28.989 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.25802", 2024-04-20T17:25:28.989 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:25:28.990 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 243442 2024-04-20T17:25:28.990 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:28.990 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:25:28.990 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:28.990 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:25:28.990 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.372681+0000", 2024-04-20T17:25:28.990 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:28.990 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:28.990 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:28.990 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:25:28.990 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.372681+0000", 2024-04-20T17:25:28.990 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:28.991 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:28.991 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:28.991 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:25:28.991 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.372681+0000", 2024-04-20T17:25:28.991 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8400000000000004e-07 2024-04-20T17:25:28.991 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:28.991 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:28.991 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:25:28.991 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.372685+0000", 2024-04-20T17:25:28.991 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6030000000000001e-06 2024-04-20T17:25:28.991 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:28.991 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:28.991 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:25:28.991 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.372690+0000", 2024-04-20T17:25:28.992 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3279999999999996e-06 2024-04-20T17:25:28.992 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:28.992 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:28.992 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:25:28.992 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.372695+0000", 2024-04-20T17:25:28.992 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0259999999999998e-06 2024-04-20T17:25:28.992 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:28.992 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:28.993 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:25:28.993 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.377789+0000", 2024-04-20T17:25:28.993 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0050939519999999997 2024-04-20T17:25:28.993 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:28.993 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:28.993 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:25:28.993 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.377904+0000", 2024-04-20T17:25:28.993 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011461800000000001 2024-04-20T17:25:28.993 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:28.993 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:28.993 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 11", 2024-04-20T17:25:28.994 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.378011+0000", 2024-04-20T17:25:28.994 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010702800000000001 2024-04-20T17:25:28.994 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:28.994 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:28.994 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:25:28.994 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.386721+0000", 2024-04-20T17:25:28.994 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0087100849999999994 2024-04-20T17:25:28.994 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:28.994 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:28.994 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:25:28.994 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.469245+0000", 2024-04-20T17:25:28.994 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.082523795999999996 2024-04-20T17:25:28.994 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:28.995 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:28.995 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:25:28.995 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.469296+0000", 2024-04-20T17:25:28.995 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1668000000000001e-05 2024-04-20T17:25:28.995 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:28.995 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:28.995 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:25:28.995 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.469335+0000", 2024-04-20T17:25:28.995 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8334000000000003e-05 2024-04-20T17:25:28.995 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:28.995 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:25:28.996 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:28.996 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:28.996 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:28.996 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.25802.0:243452 3.4 3:2fa8b7ed:::benchmark_data_smithi171_110765_object100237:head [delete] snapc 0=[] ondisk+write+known_if_redirected e416)", 2024-04-20T17:25:28.996 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:23:29.382317+0000", 2024-04-20T17:25:28.996 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 119.597689555, 2024-04-20T17:25:28.996 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.087112933000000004, 2024-04-20T17:25:28.996 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:25:28.996 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:25:28.996 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:25:28.996 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:25:28.996 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.25802", 2024-04-20T17:25:28.996 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:25:28.997 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 243452 2024-04-20T17:25:28.997 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:28.997 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:25:28.997 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:28.997 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:25:28.997 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.382317+0000", 2024-04-20T17:25:28.997 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:28.997 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:28.997 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:28.997 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:25:28.997 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.382317+0000", 2024-04-20T17:25:28.997 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:28.997 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:28.997 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:28.997 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:25:28.998 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.382320+0000", 2024-04-20T17:25:28.998 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6519999999999999e-06 2024-04-20T17:25:28.998 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:28.998 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:28.998 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:25:28.998 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.382330+0000", 2024-04-20T17:25:28.998 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6029999999999994e-06 2024-04-20T17:25:28.998 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:28.998 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:28.998 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:25:28.998 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.382347+0000", 2024-04-20T17:25:28.998 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7016000000000001e-05 2024-04-20T17:25:28.999 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:28.999 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:28.999 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:25:28.999 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.382358+0000", 2024-04-20T17:25:28.999 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1229999999999999e-05 2024-04-20T17:25:28.999 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:28.999 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:28.999 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:25:28.999 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.382646+0000", 2024-04-20T17:25:28.999 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00028770999999999999 2024-04-20T17:25:28.999 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:28.999 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:28.999 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:25:28.999 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.382719+0000", 2024-04-20T17:25:28.999 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2669000000000005e-05 2024-04-20T17:25:29.000 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.000 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.000 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 11", 2024-04-20T17:25:29.000 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.382784+0000", 2024-04-20T17:25:29.000 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5502999999999999e-05 2024-04-20T17:25:29.000 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.000 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.000 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:25:29.000 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.386973+0000", 2024-04-20T17:25:29.000 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041887950000000004 2024-04-20T17:25:29.000 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.000 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.000 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:25:29.000 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.469369+0000", 2024-04-20T17:25:29.000 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.082395741999999994 2024-04-20T17:25:29.001 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.001 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.001 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:25:29.001 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.469402+0000", 2024-04-20T17:25:29.001 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3065000000000003e-05 2024-04-20T17:25:29.001 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.001 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.001 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:25:29.001 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.469430+0000", 2024-04-20T17:25:29.001 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7948000000000001e-05 2024-04-20T17:25:29.001 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.001 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:25:29.001 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.001 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.002 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.002 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:51496 4.6 e455/445 4:65c60a47:::benchmark_data_smithi171_122772_object51495:head v 455'3348, mlcod=455'3345)", 2024-04-20T17:25:29.002 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:02.037814+0000", 2024-04-20T17:25:29.002 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 26.942192048999999, 2024-04-20T17:25:29.002 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.086449278000000004, 2024-04-20T17:25:29.002 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:25:29.002 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:25:29.002 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:25:29.002 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:25:29.002 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.002 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:25:29.002 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.037814+0000", 2024-04-20T17:25:29.002 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.002 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.002 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.003 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:25:29.003 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.037814+0000", 2024-04-20T17:25:29.003 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.003 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.003 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.003 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:25:29.003 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.037816+0000", 2024-04-20T17:25:29.003 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4360000000000001e-06 2024-04-20T17:25:29.003 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.003 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.003 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:25:29.003 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.037861+0000", 2024-04-20T17:25:29.003 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4929000000000003e-05 2024-04-20T17:25:29.003 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.004 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.004 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:25:29.004 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.037870+0000", 2024-04-20T17:25:29.004 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9430000000000006e-06 2024-04-20T17:25:29.004 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.004 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.004 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:25:29.004 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.037876+0000", 2024-04-20T17:25:29.004 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4409999999999998e-06 2024-04-20T17:25:29.004 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.004 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.004 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:25:29.004 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.038362+0000", 2024-04-20T17:25:29.004 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00048568999999999999 2024-04-20T17:25:29.004 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.005 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.005 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:25:29.005 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.038381+0000", 2024-04-20T17:25:29.005 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9242999999999999e-05 2024-04-20T17:25:29.005 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.005 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.005 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:25:29.005 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.039822+0000", 2024-04-20T17:25:29.005 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001441178 2024-04-20T17:25:29.005 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.005 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.005 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:25:29.005 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.124264+0000", 2024-04-20T17:25:29.006 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.084441418000000004 2024-04-20T17:25:29.006 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.006 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:25:29.006 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.006 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.006 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.006 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:51502 4.b e455/445 4:d57cdef5:::benchmark_data_smithi171_122772_object51501:head v 455'3266, mlcod=455'3264)", 2024-04-20T17:25:29.006 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:02.038477+0000", 2024-04-20T17:25:29.006 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 26.941529522, 2024-04-20T17:25:29.006 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.088332016999999999, 2024-04-20T17:25:29.006 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:25:29.006 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:25:29.006 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:25:29.007 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:25:29.007 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.007 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:25:29.007 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.038477+0000", 2024-04-20T17:25:29.007 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.007 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.007 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.007 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:25:29.007 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.038477+0000", 2024-04-20T17:25:29.007 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.007 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.007 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.008 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:25:29.008 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.038478+0000", 2024-04-20T17:25:29.008 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4580000000000001e-06 2024-04-20T17:25:29.008 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.008 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.008 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:25:29.008 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.038504+0000", 2024-04-20T17:25:29.008 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5624000000000001e-05 2024-04-20T17:25:29.008 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.008 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.008 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:25:29.008 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.038513+0000", 2024-04-20T17:25:29.008 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9490000000000002e-06 2024-04-20T17:25:29.009 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.009 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.009 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:25:29.009 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.038520+0000", 2024-04-20T17:25:29.009 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9959999999999996e-06 2024-04-20T17:25:29.009 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.009 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.009 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:25:29.009 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.038563+0000", 2024-04-20T17:25:29.009 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3368000000000002e-05 2024-04-20T17:25:29.009 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.010 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.010 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:25:29.010 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.038581+0000", 2024-04-20T17:25:29.010 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7524999999999999e-05 2024-04-20T17:25:29.010 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.010 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.010 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:25:29.010 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.126754+0000", 2024-04-20T17:25:29.010 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.088172871 2024-04-20T17:25:29.010 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.010 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.010 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:25:29.011 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.126809+0000", 2024-04-20T17:25:29.011 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5226000000000003e-05 2024-04-20T17:25:29.011 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.011 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:25:29.011 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.011 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.011 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.011 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:51504 4.6 e455/445 4:66a041f5:::benchmark_data_smithi171_122772_object51503:head v 455'3349, mlcod=455'3345)", 2024-04-20T17:25:29.011 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:02.039938+0000", 2024-04-20T17:25:29.011 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 26.940068032999999, 2024-04-20T17:25:29.011 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.089091318000000003, 2024-04-20T17:25:29.011 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:25:29.012 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:25:29.013 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:25:29.013 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:25:29.013 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.013 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:25:29.013 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.039938+0000", 2024-04-20T17:25:29.013 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.013 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.013 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.013 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:25:29.014 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.039938+0000", 2024-04-20T17:25:29.014 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.014 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.014 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.014 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:25:29.014 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.039940+0000", 2024-04-20T17:25:29.014 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.415e-06 2024-04-20T17:25:29.014 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.014 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.014 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:25:29.014 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.040076+0000", 2024-04-20T17:25:29.014 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013595400000000001 2024-04-20T17:25:29.014 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.014 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.015 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:25:29.015 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.124224+0000", 2024-04-20T17:25:29.015 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.084147949999999999 2024-04-20T17:25:29.015 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.015 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.015 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:25:29.015 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.124237+0000", 2024-04-20T17:25:29.015 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3183999999999999e-05 2024-04-20T17:25:29.015 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.015 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.015 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:25:29.015 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.124308+0000", 2024-04-20T17:25:29.015 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1657999999999997e-05 2024-04-20T17:25:29.015 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.015 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.016 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:25:29.016 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.124331+0000", 2024-04-20T17:25:29.016 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2918999999999999e-05 2024-04-20T17:25:29.016 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.016 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.016 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:25:29.016 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.129000+0000", 2024-04-20T17:25:29.016 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0046690960000000002 2024-04-20T17:25:29.016 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.016 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.017 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:25:29.017 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.129030+0000", 2024-04-20T17:25:29.017 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9142e-05 2024-04-20T17:25:29.017 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.017 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:25:29.017 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.017 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.017 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.017 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:51513 4.a e455/445 4:5c308534:::benchmark_data_smithi171_122772_object51512:head v 455'3138, mlcod=455'3136)", 2024-04-20T17:25:29.017 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:02.045064+0000", 2024-04-20T17:25:29.017 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 26.934942358000001, 2024-04-20T17:25:29.017 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.084044983000000004, 2024-04-20T17:25:29.017 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:25:29.017 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:25:29.018 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:25:29.018 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:25:29.018 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.018 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:25:29.018 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.045064+0000", 2024-04-20T17:25:29.018 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.018 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.018 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.018 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:25:29.018 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.045064+0000", 2024-04-20T17:25:29.018 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.019 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.019 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.019 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:25:29.019 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.045065+0000", 2024-04-20T17:25:29.019 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0869999999999999e-06 2024-04-20T17:25:29.019 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.019 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.020 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:25:29.020 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.045118+0000", 2024-04-20T17:25:29.020 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2933999999999998e-05 2024-04-20T17:25:29.020 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.020 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.020 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:25:29.020 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.124563+0000", 2024-04-20T17:25:29.020 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.079445509999999997 2024-04-20T17:25:29.020 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.020 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.020 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:25:29.020 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.124571+0000", 2024-04-20T17:25:29.020 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2699999999999999e-06 2024-04-20T17:25:29.020 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.021 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.021 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:25:29.021 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.126109+0000", 2024-04-20T17:25:29.021 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0015385139999999999 2024-04-20T17:25:29.021 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.021 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.021 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:25:29.021 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.126131+0000", 2024-04-20T17:25:29.021 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1313000000000001e-05 2024-04-20T17:25:29.021 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.021 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.021 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:25:29.021 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.129062+0000", 2024-04-20T17:25:29.022 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0029310629999999998 2024-04-20T17:25:29.022 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.022 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.022 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:25:29.022 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.129109+0000", 2024-04-20T17:25:29.022 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7292000000000001e-05 2024-04-20T17:25:29.022 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.022 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:25:29.022 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.022 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.023 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.023 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:51519 4.4 e455/445 4:2ff1b437:::benchmark_data_smithi171_122772_object51518:head v 455'3185, mlcod=455'3183)", 2024-04-20T17:25:29.023 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:02.045175+0000", 2024-04-20T17:25:29.023 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 26.934831533000001, 2024-04-20T17:25:29.023 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.083446963999999998, 2024-04-20T17:25:29.023 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:25:29.023 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:25:29.023 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:25:29.023 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:25:29.023 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.023 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:25:29.023 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.045175+0000", 2024-04-20T17:25:29.023 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.023 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.023 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.024 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:25:29.024 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.045175+0000", 2024-04-20T17:25:29.024 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.024 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.024 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.024 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:25:29.024 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.045176+0000", 2024-04-20T17:25:29.024 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2100000000000001e-06 2024-04-20T17:25:29.024 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.024 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.024 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:25:29.024 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.045196+0000", 2024-04-20T17:25:29.024 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9738e-05 2024-04-20T17:25:29.025 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.025 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.025 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:25:29.025 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.124330+0000", 2024-04-20T17:25:29.025 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.079134322000000007 2024-04-20T17:25:29.025 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.025 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.025 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:25:29.025 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.124340+0000", 2024-04-20T17:25:29.025 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0456e-05 2024-04-20T17:25:29.025 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.025 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.025 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:25:29.025 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.124975+0000", 2024-04-20T17:25:29.026 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00063459699999999998 2024-04-20T17:25:29.026 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.026 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.026 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:25:29.026 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.124995+0000", 2024-04-20T17:25:29.026 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0346000000000001e-05 2024-04-20T17:25:29.026 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.026 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.026 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:25:29.026 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.128577+0000", 2024-04-20T17:25:29.026 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0035820359999999998 2024-04-20T17:25:29.027 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.027 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.027 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:25:29.027 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.128622+0000", 2024-04-20T17:25:29.027 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4258999999999998e-05 2024-04-20T17:25:29.027 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.027 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:25:29.027 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.027 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.027 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.027 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:57132 4.9 e457/445 4:9332f97e:::benchmark_data_smithi171_122772_object57131:head v 457'3545, mlcod=457'3542)", 2024-04-20T17:25:29.027 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:04.312793+0000", 2024-04-20T17:25:29.027 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 24.667213151999999, 2024-04-20T17:25:29.027 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057318102000000003, 2024-04-20T17:25:29.028 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:25:29.028 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:25:29.028 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:25:29.028 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:25:29.028 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.028 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:25:29.028 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.312793+0000", 2024-04-20T17:25:29.028 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.028 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.028 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.028 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:25:29.028 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.312793+0000", 2024-04-20T17:25:29.028 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.029 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.029 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.029 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:25:29.029 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.312794+0000", 2024-04-20T17:25:29.029 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.19e-06 2024-04-20T17:25:29.029 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.029 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.029 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:25:29.029 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.312911+0000", 2024-04-20T17:25:29.030 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000116208 2024-04-20T17:25:29.030 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.030 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.030 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:25:29.030 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.312921+0000", 2024-04-20T17:25:29.030 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.009e-05 2024-04-20T17:25:29.030 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.030 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.031 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:25:29.031 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.312928+0000", 2024-04-20T17:25:29.031 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3130000000000001e-06 2024-04-20T17:25:29.031 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.031 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.031 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:25:29.031 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.312962+0000", 2024-04-20T17:25:29.031 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4037e-05 2024-04-20T17:25:29.031 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.031 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.031 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:25:29.032 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.312973+0000", 2024-04-20T17:25:29.032 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1129e-05 2024-04-20T17:25:29.032 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.032 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.032 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:25:29.032 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.370081+0000", 2024-04-20T17:25:29.032 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057107509000000001 2024-04-20T17:25:29.033 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.033 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.033 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:25:29.033 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.370111+0000", 2024-04-20T17:25:29.033 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0626000000000001e-05 2024-04-20T17:25:29.033 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.033 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:25:29.033 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.033 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.033 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.033 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:57135 4.6 e457/445 4:6fcc02b2:::benchmark_data_smithi171_122772_object57134:head v 457'3686, mlcod=457'3684)", 2024-04-20T17:25:29.033 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:04.313169+0000", 2024-04-20T17:25:29.033 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 24.666837322999999, 2024-04-20T17:25:29.033 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.05722853, 2024-04-20T17:25:29.034 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:25:29.034 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:25:29.034 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:25:29.034 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:25:29.034 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.034 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:25:29.034 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.313169+0000", 2024-04-20T17:25:29.034 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.034 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.034 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.034 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:25:29.034 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.313169+0000", 2024-04-20T17:25:29.034 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.034 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.035 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.035 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:25:29.035 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.313170+0000", 2024-04-20T17:25:29.035 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4099999999999997e-07 2024-04-20T17:25:29.035 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.035 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.035 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:25:29.035 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.313232+0000", 2024-04-20T17:25:29.035 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1742000000000006e-05 2024-04-20T17:25:29.035 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.035 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.035 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:25:29.035 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.313238+0000", 2024-04-20T17:25:29.035 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.302e-06 2024-04-20T17:25:29.036 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.036 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.036 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:25:29.036 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.313242+0000", 2024-04-20T17:25:29.036 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9770000000000002e-06 2024-04-20T17:25:29.036 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.036 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.036 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:25:29.036 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.313299+0000", 2024-04-20T17:25:29.036 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.664e-05 2024-04-20T17:25:29.036 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.036 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.037 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:25:29.037 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.313317+0000", 2024-04-20T17:25:29.037 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8841000000000001e-05 2024-04-20T17:25:29.037 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.037 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.037 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:25:29.037 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.370365+0000", 2024-04-20T17:25:29.037 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057047489999999999 2024-04-20T17:25:29.038 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.038 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.038 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:25:29.038 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.370397+0000", 2024-04-20T17:25:29.038 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2597e-05 2024-04-20T17:25:29.038 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.038 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:25:29.038 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.038 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.038 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.038 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:57137 4.f e457/445 4:f70469a5:::benchmark_data_smithi171_122772_object57136:head v 457'3576, mlcod=457'3574)", 2024-04-20T17:25:29.038 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:04.313573+0000", 2024-04-20T17:25:29.038 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 24.666433336000001, 2024-04-20T17:25:29.039 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057404561999999999, 2024-04-20T17:25:29.039 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:25:29.039 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:25:29.039 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:25:29.039 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:25:29.039 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.039 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:25:29.039 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.313573+0000", 2024-04-20T17:25:29.039 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.039 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.039 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.039 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:25:29.039 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.313573+0000", 2024-04-20T17:25:29.040 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.040 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.040 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.040 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:25:29.040 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.313574+0000", 2024-04-20T17:25:29.040 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.319e-06 2024-04-20T17:25:29.040 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.040 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.040 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:25:29.040 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.313596+0000", 2024-04-20T17:25:29.040 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1600999999999999e-05 2024-04-20T17:25:29.040 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.041 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.041 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:25:29.041 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.365609+0000", 2024-04-20T17:25:29.041 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.052013082000000002 2024-04-20T17:25:29.041 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.041 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.041 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:25:29.041 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.365621+0000", 2024-04-20T17:25:29.041 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1661000000000001e-05 2024-04-20T17:25:29.041 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.041 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.041 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:25:29.041 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.365933+0000", 2024-04-20T17:25:29.042 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00031213500000000001 2024-04-20T17:25:29.042 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.042 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.042 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:25:29.042 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.365959+0000", 2024-04-20T17:25:29.042 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6234999999999999e-05 2024-04-20T17:25:29.042 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.042 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.043 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:25:29.043 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.370947+0000", 2024-04-20T17:25:29.043 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0049884020000000003 2024-04-20T17:25:29.043 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.043 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.043 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:25:29.043 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.370977+0000", 2024-04-20T17:25:29.043 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0127000000000001e-05 2024-04-20T17:25:29.043 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.043 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:25:29.043 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.043 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.043 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.043 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:57138 4.6 e457/445 4:694e5cfc:::benchmark_data_smithi171_122772_object57137:head v 457'3687, mlcod=457'3684)", 2024-04-20T17:25:29.044 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:04.314954+0000", 2024-04-20T17:25:29.044 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 24.665052588000002, 2024-04-20T17:25:29.044 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.055459545999999998, 2024-04-20T17:25:29.044 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:25:29.044 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:25:29.044 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:25:29.044 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:25:29.044 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.044 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:25:29.044 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.314954+0000", 2024-04-20T17:25:29.044 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.044 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.044 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.044 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:25:29.045 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.314954+0000", 2024-04-20T17:25:29.045 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.045 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.045 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.045 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:25:29.045 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.314958+0000", 2024-04-20T17:25:29.045 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0210000000000001e-06 2024-04-20T17:25:29.045 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.045 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.045 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:25:29.045 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.315156+0000", 2024-04-20T17:25:29.045 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00019783800000000001 2024-04-20T17:25:29.045 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.045 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.045 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:25:29.046 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.365441+0000", 2024-04-20T17:25:29.046 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.050285564999999997 2024-04-20T17:25:29.046 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.046 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.046 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:25:29.046 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.365455+0000", 2024-04-20T17:25:29.046 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.341e-05 2024-04-20T17:25:29.046 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.046 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.046 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:25:29.046 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.367225+0000", 2024-04-20T17:25:29.046 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0017709480000000001 2024-04-20T17:25:29.046 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.046 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.046 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:25:29.047 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.367243+0000", 2024-04-20T17:25:29.047 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7790000000000001e-05 2024-04-20T17:25:29.047 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.047 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.047 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:25:29.047 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.370400+0000", 2024-04-20T17:25:29.047 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003156524 2024-04-20T17:25:29.047 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.047 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.047 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:25:29.047 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.370413+0000", 2024-04-20T17:25:29.047 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.345e-05 2024-04-20T17:25:29.047 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.048 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:25:29.048 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.048 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.048 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.048 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:57143 4.a e457/445 4:5de4257f:::benchmark_data_smithi171_122772_object57142:head v 457'3489, mlcod=457'3487)", 2024-04-20T17:25:29.048 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:04.316057+0000", 2024-04-20T17:25:29.048 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 24.663948971, 2024-04-20T17:25:29.048 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.053247836999999999, 2024-04-20T17:25:29.048 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:25:29.048 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:25:29.048 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:25:29.048 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:25:29.048 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.048 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:25:29.048 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.316057+0000", 2024-04-20T17:25:29.049 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.049 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.049 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.049 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:25:29.049 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.316057+0000", 2024-04-20T17:25:29.049 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.049 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.049 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.049 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:25:29.049 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.316059+0000", 2024-04-20T17:25:29.049 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.249e-06 2024-04-20T17:25:29.049 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.049 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.049 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:25:29.050 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.316093+0000", 2024-04-20T17:25:29.050 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4483000000000002e-05 2024-04-20T17:25:29.050 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.050 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.050 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:25:29.050 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.365821+0000", 2024-04-20T17:25:29.050 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.049727510000000003 2024-04-20T17:25:29.050 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.050 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.050 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:25:29.050 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.365829+0000", 2024-04-20T17:25:29.050 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0700000000000007e-06 2024-04-20T17:25:29.050 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.050 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.050 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:25:29.051 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.366056+0000", 2024-04-20T17:25:29.051 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00022766600000000001 2024-04-20T17:25:29.051 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.051 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.051 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:25:29.051 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.366076+0000", 2024-04-20T17:25:29.051 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9817999999999999e-05 2024-04-20T17:25:29.051 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.051 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.051 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:25:29.051 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.369259+0000", 2024-04-20T17:25:29.051 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0031825590000000002 2024-04-20T17:25:29.051 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.051 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.051 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:25:29.052 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.369305+0000", 2024-04-20T17:25:29.052 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6482e-05 2024-04-20T17:25:29.052 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.052 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:25:29.052 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.052 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.052 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.052 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:63536 4.9 e458/445 4:938d0a2f:::benchmark_data_smithi171_122772_object63535:head v 458'3950, mlcod=458'3948)", 2024-04-20T17:25:29.052 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:06.886335+0000", 2024-04-20T17:25:29.052 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 22.093671138000001, 2024-04-20T17:25:29.052 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.052807923999999999, 2024-04-20T17:25:29.052 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:25:29.052 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:25:29.052 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:25:29.053 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:25:29.053 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.053 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:25:29.053 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.886335+0000", 2024-04-20T17:25:29.053 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.053 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.053 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.053 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:25:29.054 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.886335+0000", 2024-04-20T17:25:29.054 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.054 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.054 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.054 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:25:29.054 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.886336+0000", 2024-04-20T17:25:29.054 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0589999999999999e-06 2024-04-20T17:25:29.054 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.054 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.054 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:25:29.054 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.886367+0000", 2024-04-20T17:25:29.054 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0939000000000003e-05 2024-04-20T17:25:29.054 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.054 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.055 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:25:29.055 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.886377+0000", 2024-04-20T17:25:29.055 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.5319999999999995e-06 2024-04-20T17:25:29.055 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.055 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.055 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:25:29.055 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.886384+0000", 2024-04-20T17:25:29.055 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9870000000000002e-06 2024-04-20T17:25:29.055 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.055 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.055 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:25:29.055 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.886429+0000", 2024-04-20T17:25:29.055 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5246e-05 2024-04-20T17:25:29.055 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.056 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.056 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:25:29.056 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.886447+0000", 2024-04-20T17:25:29.056 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8043999999999999e-05 2024-04-20T17:25:29.056 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.056 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.056 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:25:29.056 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.892745+0000", 2024-04-20T17:25:29.056 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0062984089999999996 2024-04-20T17:25:29.056 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.056 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.056 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:25:29.056 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.939143+0000", 2024-04-20T17:25:29.056 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.046397708000000003 2024-04-20T17:25:29.056 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.057 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:25:29.057 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.057 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.057 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.057 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:63538 4.9 e458/445 4:9a173ad5:::benchmark_data_smithi171_122772_object63537:head v 458'3951, mlcod=458'3948)", 2024-04-20T17:25:29.057 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:06.887349+0000", 2024-04-20T17:25:29.057 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 22.092656855000001, 2024-04-20T17:25:29.057 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.051833481000000001, 2024-04-20T17:25:29.057 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:25:29.057 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:25:29.057 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:25:29.057 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:25:29.057 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.057 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:25:29.058 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.887349+0000", 2024-04-20T17:25:29.058 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.058 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.058 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.058 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:25:29.058 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.887349+0000", 2024-04-20T17:25:29.058 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.058 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.058 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.058 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:25:29.058 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.887351+0000", 2024-04-20T17:25:29.059 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.358e-06 2024-04-20T17:25:29.059 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.059 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.059 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:25:29.059 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.887468+0000", 2024-04-20T17:25:29.059 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011736899999999999 2024-04-20T17:25:29.059 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.059 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.059 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:25:29.059 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.887476+0000", 2024-04-20T17:25:29.060 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8490000000000006e-06 2024-04-20T17:25:29.060 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.060 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.060 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:25:29.060 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.887483+0000", 2024-04-20T17:25:29.060 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7299999999999999e-06 2024-04-20T17:25:29.060 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.060 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.060 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:25:29.060 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888251+0000", 2024-04-20T17:25:29.060 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00076797900000000001 2024-04-20T17:25:29.060 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.060 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.060 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:25:29.061 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888269+0000", 2024-04-20T17:25:29.061 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.857e-05 2024-04-20T17:25:29.061 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.061 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.061 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:25:29.061 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.939148+0000", 2024-04-20T17:25:29.061 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.050878707000000002 2024-04-20T17:25:29.061 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.061 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.061 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:25:29.061 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.939183+0000", 2024-04-20T17:25:29.061 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4919000000000002e-05 2024-04-20T17:25:29.061 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.061 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:25:29.061 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.062 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.062 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.062 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:63532 4.7 e458/445 4:edb4ebc6:::benchmark_data_smithi171_122772_object63531:head v 458'3994, mlcod=458'3991)", 2024-04-20T17:25:29.062 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:06.888255+0000", 2024-04-20T17:25:29.062 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 22.091751001999999, 2024-04-20T17:25:29.062 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.050897286999999999, 2024-04-20T17:25:29.062 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:25:29.062 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:25:29.062 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:25:29.062 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:25:29.062 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.062 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:25:29.062 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888255+0000", 2024-04-20T17:25:29.062 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.062 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.063 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.063 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:25:29.063 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888255+0000", 2024-04-20T17:25:29.063 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.063 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.063 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.063 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:25:29.063 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888257+0000", 2024-04-20T17:25:29.064 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.378e-06 2024-04-20T17:25:29.064 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.064 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.064 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:25:29.064 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888299+0000", 2024-04-20T17:25:29.064 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2379999999999997e-05 2024-04-20T17:25:29.064 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.064 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.064 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:25:29.064 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888307+0000", 2024-04-20T17:25:29.064 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4670000000000004e-06 2024-04-20T17:25:29.064 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.064 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.064 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:25:29.065 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888314+0000", 2024-04-20T17:25:29.065 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4320000000000004e-06 2024-04-20T17:25:29.065 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.065 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.065 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:25:29.065 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888360+0000", 2024-04-20T17:25:29.065 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6332000000000003e-05 2024-04-20T17:25:29.065 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.065 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.065 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:25:29.065 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888378+0000", 2024-04-20T17:25:29.065 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7855000000000001e-05 2024-04-20T17:25:29.065 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.065 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.065 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:25:29.065 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.892776+0000", 2024-04-20T17:25:29.066 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0043978389999999997 2024-04-20T17:25:29.066 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.066 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.066 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:25:29.066 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.939153+0000", 2024-04-20T17:25:29.066 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.046376604000000002 2024-04-20T17:25:29.066 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.066 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:25:29.066 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.066 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.066 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.066 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:63535 4.4 e458/445 4:2d7f2e3e:::benchmark_data_smithi171_122772_object63534:head v 458'3947, mlcod=458'3944)", 2024-04-20T17:25:29.066 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:06.888425+0000", 2024-04-20T17:25:29.067 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 22.091580884999999, 2024-04-20T17:25:29.067 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.050783572999999999, 2024-04-20T17:25:29.067 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:25:29.067 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:25:29.067 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:25:29.067 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:25:29.067 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.067 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:25:29.067 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888425+0000", 2024-04-20T17:25:29.067 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.068 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.068 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.068 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:25:29.068 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888425+0000", 2024-04-20T17:25:29.068 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.068 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.068 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.068 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:25:29.068 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888427+0000", 2024-04-20T17:25:29.068 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.801e-06 2024-04-20T17:25:29.068 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.068 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.068 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:25:29.068 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888453+0000", 2024-04-20T17:25:29.069 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5899999999999999e-05 2024-04-20T17:25:29.069 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.069 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.069 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:25:29.069 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888461+0000", 2024-04-20T17:25:29.069 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0199999999999994e-06 2024-04-20T17:25:29.069 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.069 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.069 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:25:29.069 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888466+0000", 2024-04-20T17:25:29.069 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7310000000000001e-06 2024-04-20T17:25:29.069 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.069 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.069 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:25:29.069 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888509+0000", 2024-04-20T17:25:29.070 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3022000000000002e-05 2024-04-20T17:25:29.070 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.070 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.070 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:25:29.070 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888526+0000", 2024-04-20T17:25:29.070 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7391999999999999e-05 2024-04-20T17:25:29.070 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.070 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.070 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:25:29.070 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.892770+0000", 2024-04-20T17:25:29.070 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0042435390000000002 2024-04-20T17:25:29.074 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.074 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.074 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:25:29.075 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.939209+0000", 2024-04-20T17:25:29.075 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.046439168000000003 2024-04-20T17:25:29.075 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.075 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:25:29.075 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.075 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.075 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.075 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:63529 4.f e458/445 4:ffb27176:::benchmark_data_smithi171_122772_object63528:head v 458'3985, mlcod=458'3982)", 2024-04-20T17:25:29.075 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:06.888492+0000", 2024-04-20T17:25:29.075 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 22.091514416999999, 2024-04-20T17:25:29.075 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.050709013999999997, 2024-04-20T17:25:29.075 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:25:29.075 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:25:29.075 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:25:29.076 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:25:29.076 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.076 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:25:29.076 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888492+0000", 2024-04-20T17:25:29.076 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.076 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.076 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.076 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:25:29.076 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888492+0000", 2024-04-20T17:25:29.076 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.076 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.076 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.076 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:25:29.076 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888493+0000", 2024-04-20T17:25:29.076 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2529999999999999e-06 2024-04-20T17:25:29.077 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.077 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.077 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:25:29.077 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888514+0000", 2024-04-20T17:25:29.077 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1087e-05 2024-04-20T17:25:29.077 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.077 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.077 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:25:29.077 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888523+0000", 2024-04-20T17:25:29.077 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6859999999999995e-06 2024-04-20T17:25:29.077 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.077 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.077 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:25:29.077 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888529+0000", 2024-04-20T17:25:29.078 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.46e-06 2024-04-20T17:25:29.078 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.078 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.078 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:25:29.078 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888571+0000", 2024-04-20T17:25:29.078 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1171000000000003e-05 2024-04-20T17:25:29.078 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.078 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.078 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:25:29.078 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888588+0000", 2024-04-20T17:25:29.078 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7728e-05 2024-04-20T17:25:29.078 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.078 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.078 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:25:29.078 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.939157+0000", 2024-04-20T17:25:29.079 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.050568861999999999 2024-04-20T17:25:29.079 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.079 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.079 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:25:29.079 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.939201+0000", 2024-04-20T17:25:29.079 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3767000000000003e-05 2024-04-20T17:25:29.079 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.079 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:25:29.079 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.079 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.079 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.079 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:63537 4.7 e458/445 4:e12db56f:::benchmark_data_smithi171_122772_object63536:head v 458'3995, mlcod=458'3991)", 2024-04-20T17:25:29.079 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:06.888644+0000", 2024-04-20T17:25:29.079 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 22.091362679, 2024-04-20T17:25:29.079 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.050585249999999998, 2024-04-20T17:25:29.080 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:25:29.080 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:25:29.080 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:25:29.080 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:25:29.080 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.080 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:25:29.080 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888644+0000", 2024-04-20T17:25:29.080 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.080 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.080 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.080 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:25:29.080 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888644+0000", 2024-04-20T17:25:29.080 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.080 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.081 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.081 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:25:29.081 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888645+0000", 2024-04-20T17:25:29.081 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0559999999999999e-06 2024-04-20T17:25:29.081 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.081 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.081 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:25:29.081 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888741+0000", 2024-04-20T17:25:29.081 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6401000000000001e-05 2024-04-20T17:25:29.081 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.081 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.081 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:25:29.081 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888750+0000", 2024-04-20T17:25:29.081 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4129999999999995e-06 2024-04-20T17:25:29.081 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.081 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.082 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:25:29.082 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.888757+0000", 2024-04-20T17:25:29.082 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1870000000000002e-06 2024-04-20T17:25:29.082 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.082 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.082 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:25:29.082 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.890354+0000", 2024-04-20T17:25:29.082 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001596863 2024-04-20T17:25:29.082 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.082 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.082 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:25:29.082 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.890373+0000", 2024-04-20T17:25:29.082 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9485000000000001e-05 2024-04-20T17:25:29.082 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.083 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.083 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:25:29.083 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.939204+0000", 2024-04-20T17:25:29.083 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.048831002999999998 2024-04-20T17:25:29.083 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.083 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.083 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:25:29.083 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.939229+0000", 2024-04-20T17:25:29.083 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4842e-05 2024-04-20T17:25:29.083 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.083 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:25:29.083 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.083 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.083 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.083 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:63534 4.f e458/445 4:f3c91de0:::benchmark_data_smithi171_122772_object63533:head v 458'3986, mlcod=458'3982)", 2024-04-20T17:25:29.084 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:06.889168+0000", 2024-04-20T17:25:29.084 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 22.090837772, 2024-04-20T17:25:29.084 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.050086145999999998, 2024-04-20T17:25:29.084 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:25:29.084 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:25:29.084 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:25:29.084 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:25:29.084 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.084 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:25:29.084 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.889168+0000", 2024-04-20T17:25:29.084 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.085 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.085 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.085 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:25:29.085 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.889168+0000", 2024-04-20T17:25:29.085 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.085 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.085 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.085 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:25:29.085 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.889170+0000", 2024-04-20T17:25:29.085 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.17e-06 2024-04-20T17:25:29.085 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.085 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.086 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:25:29.086 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.889191+0000", 2024-04-20T17:25:29.086 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0888999999999999e-05 2024-04-20T17:25:29.086 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.086 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.086 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:25:29.086 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.889200+0000", 2024-04-20T17:25:29.086 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.003e-06 2024-04-20T17:25:29.086 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.086 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.086 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:25:29.086 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.889206+0000", 2024-04-20T17:25:29.086 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2519999999999996e-06 2024-04-20T17:25:29.086 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.086 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.087 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:25:29.087 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.890436+0000", 2024-04-20T17:25:29.087 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001230512 2024-04-20T17:25:29.087 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.087 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.087 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:25:29.087 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.890459+0000", 2024-04-20T17:25:29.087 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2442999999999998e-05 2024-04-20T17:25:29.087 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.087 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.087 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:25:29.087 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.939231+0000", 2024-04-20T17:25:29.087 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.048772678999999999 2024-04-20T17:25:29.087 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.088 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.088 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:25:29.088 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.939255+0000", 2024-04-20T17:25:29.088 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3198e-05 2024-04-20T17:25:29.088 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.088 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:25:29.088 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.088 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.088 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.088 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:63542 4.a e458/445 4:509f6063:::benchmark_data_smithi171_122772_object63541:head v 458'3862, mlcod=458'3860)", 2024-04-20T17:25:29.088 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:06.890770+0000", 2024-04-20T17:25:29.088 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 22.089236208999999, 2024-04-20T17:25:29.088 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.049154188000000001, 2024-04-20T17:25:29.088 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:25:29.088 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:25:29.089 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:25:29.089 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:25:29.089 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.089 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:25:29.089 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.890770+0000", 2024-04-20T17:25:29.089 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.089 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.089 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.089 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:25:29.089 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.890770+0000", 2024-04-20T17:25:29.089 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:25:29.089 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.089 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.089 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:25:29.090 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.890772+0000", 2024-04-20T17:25:29.090 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.488e-06 2024-04-20T17:25:29.090 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.090 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.090 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:25:29.090 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.890830+0000", 2024-04-20T17:25:29.090 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8282000000000002e-05 2024-04-20T17:25:29.090 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.090 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.090 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:25:29.090 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.890840+0000", 2024-04-20T17:25:29.090 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9299999999999998e-06 2024-04-20T17:25:29.091 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.091 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.091 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:25:29.091 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.890848+0000", 2024-04-20T17:25:29.091 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8890000000000002e-06 2024-04-20T17:25:29.091 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.091 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.091 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:25:29.091 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.890926+0000", 2024-04-20T17:25:29.091 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8238000000000005e-05 2024-04-20T17:25:29.091 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.091 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.091 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:25:29.091 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.890946+0000", 2024-04-20T17:25:29.091 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.018e-05 2024-04-20T17:25:29.092 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.092 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.092 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:25:29.092 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.892976+0000", 2024-04-20T17:25:29.092 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0020297589999999999 2024-04-20T17:25:29.092 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:25:29.092 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:25:29.092 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:25:29.092 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.939924+0000", 2024-04-20T17:25:29.092 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.046948421999999997 2024-04-20T17:25:29.092 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.092 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:25:29.092 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.092 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:25:29.092 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:25:29.092 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:25:29.093 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:25:29.094 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:28 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:25:29.094 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:28 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:25:29.094 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:28 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:25:29.094 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:28 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:25:29.094 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:28 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:25:29.094 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:28 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:25:29.094 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:28 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:25:29.126 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:28 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:25:29.127 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:28 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:25:29.127 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:28 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:25:29.127 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:28 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:25:29.127 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:28 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:25:29.127 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:28 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:25:29.127 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:28 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:25:29.127 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:29.151 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:25:29.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:29.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:25:29.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:28 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:25:29.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:28 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:25:29.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:28 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:25:29.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:28 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:25:29.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:28 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:25:29.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:28 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:25:29.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:28 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:25:29.255 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:25:29.289 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:29.320 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:25:29.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:29.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:25:29.431 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:25:29.461 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:25:29.468 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 41 16 107293 107277 163.511 97.375 0.0130901 0.00611394 2024-04-20T17:25:29.469 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:29.501 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:25:29.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:29.506 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:25:29.607 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:25:29.646 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:29.671 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:25:29.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:29.677 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:25:29.778 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:25:29.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:29 smithi171 ceph-mon[44189]: pgmap v1237: 17 pgs: 17 active+clean; 6.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 121 MiB/s wr, 1.93k op/s 2024-04-20T17:25:29.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:29 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3811794250' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:25:29.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:29 smithi171 ceph-mon[44189]: from='client.26119 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:25:29.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:29 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:25:29.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:29 smithi171 ceph-mon[44189]: from='client.26119 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:25:29.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:29 smithi171 ceph-mon[44189]: osdmap e467: 12 total, 12 up, 12 in 2024-04-20T17:25:29.814 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:29.837 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:25:29.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:29.841 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:25:29.941 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:25:29.979 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:30.019 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:25:30.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:30.023 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:25:30.124 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:25:30.185 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:29 smithi005 ceph-mon[39562]: pgmap v1237: 17 pgs: 17 active+clean; 6.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 121 MiB/s wr, 1.93k op/s 2024-04-20T17:25:30.185 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:29 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3811794250' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:25:30.185 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:29 smithi005 ceph-mon[39562]: from='client.26119 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:25:30.186 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:29 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:25:30.186 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:29 smithi005 ceph-mon[39562]: from='client.26119 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:25:30.186 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:29 smithi005 ceph-mon[39562]: osdmap e467: 12 total, 12 up, 12 in 2024-04-20T17:25:30.189 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:29 smithi106 ceph-mon[44423]: pgmap v1237: 17 pgs: 17 active+clean; 6.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 121 MiB/s wr, 1.93k op/s 2024-04-20T17:25:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:29 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3811794250' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:25:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:29 smithi106 ceph-mon[44423]: from='client.26119 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:25:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:29 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:25:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:29 smithi106 ceph-mon[44423]: from='client.26119 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:25:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:29 smithi106 ceph-mon[44423]: osdmap e467: 12 total, 12 up, 12 in 2024-04-20T17:25:30.451 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:25:30.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:30.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:25:30.461 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-20T17:25:30.468 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 42 16 109005 108989 162.166 107 0.00901375 0.00616505 2024-04-20T17:25:30.559 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:25:30.638 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:30.696 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:25:30.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:30.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:25:30.807 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:25:30.840 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:30.869 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:25:30.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:30.872 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:25:30.974 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:25:31.026 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:31.143 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:25:31.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:31.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:25:31.261 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:25:31.296 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:31.329 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:25:31.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:31.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:25:31.438 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:25:31.468 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 43 16 110627 110611 160.752 101.375 0.00874829 0.00621926 2024-04-20T17:25:31.472 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:31.512 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:25:31.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:31.523 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:25:31.624 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:25:31.663 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:31.691 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:25:31.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:31.694 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:25:31.744 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:25:31.795 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:25:31.834 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:25:31.867 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:31.923 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:25:31.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:31.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:25:32.026 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:25:32.054 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:31 smithi005 ceph-mon[39562]: pgmap v1239: 17 pgs: 17 active+clean; 6.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 1.71k op/s 2024-04-20T17:25:32.054 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:31 smithi005 ceph-mon[39562]: osdmap e468: 12 total, 12 up, 12 in 2024-04-20T17:25:32.057 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:32.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:31 smithi171 ceph-mon[44189]: pgmap v1239: 17 pgs: 17 active+clean; 6.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 1.71k op/s 2024-04-20T17:25:32.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:31 smithi171 ceph-mon[44189]: osdmap e468: 12 total, 12 up, 12 in 2024-04-20T17:25:32.085 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:25:32.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:32.091 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:25:32.192 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:25:32.224 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:32.255 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:25:32.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:32.259 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:25:32.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:31 smithi106 ceph-mon[44423]: pgmap v1239: 17 pgs: 17 active+clean; 6.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 1.71k op/s 2024-04-20T17:25:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:31 smithi106 ceph-mon[44423]: osdmap e468: 12 total, 12 up, 12 in 2024-04-20T17:25:32.359 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:25:32.436 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:32.462 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:25:32.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:32.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:25:32.468 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 44 16 112219 112203 159.36 99.5 0.00954971 0.00627362 2024-04-20T17:25:32.566 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:25:32.602 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:32.629 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:25:32.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:32.632 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:25:32.733 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:25:32.776 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:32.806 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:25:32.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:32.811 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:25:32.912 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:25:32.943 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:25:32.943 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:25:32.943 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=2814999290,config=1460075751,health=1997216375,kv=4064000511}) 2024-04-20T17:25:32.943 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4205797453}) 2024-04-20T17:25:32.943 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4057315742}) 2024-04-20T17:25:32.943 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3171700004}) 2024-04-20T17:25:32.943 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1542984154}) 2024-04-20T17:25:32.943 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2731952862}) 2024-04-20T17:25:32.943 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3116032445}) 2024-04-20T17:25:32.944 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=736994500}) 2024-04-20T17:25:32.944 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=424198993}) 2024-04-20T17:25:32.944 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3633015325}) 2024-04-20T17:25:32.944 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2756545193}) 2024-04-20T17:25:32.944 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2353682880}) 2024-04-20T17:25:32.944 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3520192789}) 2024-04-20T17:25:32.944 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1371592320}) 2024-04-20T17:25:32.944 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1908124822}) 2024-04-20T17:25:32.944 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3475870644}) 2024-04-20T17:25:32.944 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1586003688}) 2024-04-20T17:25:32.944 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=494785912}) 2024-04-20T17:25:32.944 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2489078941}) 2024-04-20T17:25:32.944 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1902528043}) 2024-04-20T17:25:32.944 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=989173195}) 2024-04-20T17:25:32.945 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3095696304}) 2024-04-20T17:25:32.945 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=955334209}) 2024-04-20T17:25:32.945 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=868081055}) 2024-04-20T17:25:32.945 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4129100311}) 2024-04-20T17:25:32.945 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3774417330}) 2024-04-20T17:25:32.945 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3513069639}) 2024-04-20T17:25:32.945 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2256552334}) 2024-04-20T17:25:32.945 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3773222806}) 2024-04-20T17:25:32.945 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3162875977}) 2024-04-20T17:25:32.945 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=560768723}) 2024-04-20T17:25:32.945 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1445691944}) 2024-04-20T17:25:32.945 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=528107312}) 2024-04-20T17:25:32.945 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2885423832}) 2024-04-20T17:25:32.946 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1090995966}) 2024-04-20T17:25:32.946 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=144576810}) 2024-04-20T17:25:32.946 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4149142266}) 2024-04-20T17:25:32.946 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4041335277}) 2024-04-20T17:25:32.946 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4276030502}) 2024-04-20T17:25:32.946 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1277436686}) 2024-04-20T17:25:32.946 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=617026455}) 2024-04-20T17:25:32.946 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=697954303}) 2024-04-20T17:25:32.946 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2392268187}) 2024-04-20T17:25:32.946 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1674574877}) 2024-04-20T17:25:32.947 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=228972859}) 2024-04-20T17:25:32.947 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1638486260}) 2024-04-20T17:25:32.947 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3850040146}) 2024-04-20T17:25:32.950 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3592054272}) 2024-04-20T17:25:32.950 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1809226083}) 2024-04-20T17:25:32.950 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2918497377}) 2024-04-20T17:25:32.950 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3963552585}) 2024-04-20T17:25:32.950 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=533087145}) 2024-04-20T17:25:32.951 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=841868992}) 2024-04-20T17:25:32.951 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2284971}) 2024-04-20T17:25:32.951 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3786753652}) 2024-04-20T17:25:32.951 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=110151907}) 2024-04-20T17:25:32.951 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=2,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=63} crc {logm=1465928792,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=1838383097,mon_config_key=1570156382}) 2024-04-20T17:25:32.951 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3865000865}) 2024-04-20T17:25:32.951 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=570385374}) 2024-04-20T17:25:32.951 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=27,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=55} crc {mon_config_key=2368208904,monmap=717113318,osd_metadata=2016089214,osd_pg_creating=3592309798,osdmap=994340108}) 2024-04-20T17:25:32.951 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=566897761}) 2024-04-20T17:25:32.951 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=4} crc {osdmap=3186609539}) 2024-04-20T17:25:32.951 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:32.974 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:25:32.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:32.978 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:25:33.079 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:25:33.109 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:25:33.109 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:25:33.110 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=2814999290,config=1460075751,health=1997216375,kv=4064000511}) 2024-04-20T17:25:33.110 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4205797453}) 2024-04-20T17:25:33.110 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4057315742}) 2024-04-20T17:25:33.110 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3171700004}) 2024-04-20T17:25:33.110 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1542984154}) 2024-04-20T17:25:33.110 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2731952862}) 2024-04-20T17:25:33.110 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3116032445}) 2024-04-20T17:25:33.110 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=736994500}) 2024-04-20T17:25:33.110 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=424198993}) 2024-04-20T17:25:33.110 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3633015325}) 2024-04-20T17:25:33.110 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2756545193}) 2024-04-20T17:25:33.110 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2353682880}) 2024-04-20T17:25:33.111 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3520192789}) 2024-04-20T17:25:33.111 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1371592320}) 2024-04-20T17:25:33.111 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1908124822}) 2024-04-20T17:25:33.111 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3475870644}) 2024-04-20T17:25:33.111 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1586003688}) 2024-04-20T17:25:33.111 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=494785912}) 2024-04-20T17:25:33.111 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2489078941}) 2024-04-20T17:25:33.111 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1902528043}) 2024-04-20T17:25:33.111 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=989173195}) 2024-04-20T17:25:33.111 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3095696304}) 2024-04-20T17:25:33.111 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=955334209}) 2024-04-20T17:25:33.111 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=868081055}) 2024-04-20T17:25:33.111 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4129100311}) 2024-04-20T17:25:33.111 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3774417330}) 2024-04-20T17:25:33.112 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3513069639}) 2024-04-20T17:25:33.112 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2256552334}) 2024-04-20T17:25:33.112 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3773222806}) 2024-04-20T17:25:33.112 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3162875977}) 2024-04-20T17:25:33.112 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=560768723}) 2024-04-20T17:25:33.112 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1445691944}) 2024-04-20T17:25:33.112 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=528107312}) 2024-04-20T17:25:33.112 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2885423832}) 2024-04-20T17:25:33.112 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1090995966}) 2024-04-20T17:25:33.112 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=144576810}) 2024-04-20T17:25:33.112 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4149142266}) 2024-04-20T17:25:33.112 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4041335277}) 2024-04-20T17:25:33.112 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4276030502}) 2024-04-20T17:25:33.112 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1277436686}) 2024-04-20T17:25:33.113 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=617026455}) 2024-04-20T17:25:33.113 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=697954303}) 2024-04-20T17:25:33.113 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2392268187}) 2024-04-20T17:25:33.113 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1674574877}) 2024-04-20T17:25:33.113 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=228972859}) 2024-04-20T17:25:33.113 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1638486260}) 2024-04-20T17:25:33.113 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3850040146}) 2024-04-20T17:25:33.113 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3592054272}) 2024-04-20T17:25:33.113 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1809226083}) 2024-04-20T17:25:33.113 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2918497377}) 2024-04-20T17:25:33.113 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3963552585}) 2024-04-20T17:25:33.113 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=533087145}) 2024-04-20T17:25:33.113 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=841868992}) 2024-04-20T17:25:33.114 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2284971}) 2024-04-20T17:25:33.114 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3786753652}) 2024-04-20T17:25:33.114 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=110151907}) 2024-04-20T17:25:33.114 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=2,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=63} crc {logm=1465928792,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=1838383097,mon_config_key=1570156382}) 2024-04-20T17:25:33.114 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3865000865}) 2024-04-20T17:25:33.114 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=570385374}) 2024-04-20T17:25:33.114 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=27,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=55} crc {mon_config_key=2368208904,monmap=717113318,osd_metadata=2016089214,osd_pg_creating=3592309798,osdmap=994340108}) 2024-04-20T17:25:33.114 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=566897761}) 2024-04-20T17:25:33.114 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=4} crc {osdmap=3186609539}) 2024-04-20T17:25:33.115 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:33.140 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:25:33.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:33.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:25:33.188 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:25:33.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:25:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:25:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=2814999290,config=1460075751,health=1997216375,kv=4064000511}) 2024-04-20T17:25:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4205797453}) 2024-04-20T17:25:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4057315742}) 2024-04-20T17:25:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3171700004}) 2024-04-20T17:25:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1542984154}) 2024-04-20T17:25:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2731952862}) 2024-04-20T17:25:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3116032445}) 2024-04-20T17:25:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=736994500}) 2024-04-20T17:25:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=424198993}) 2024-04-20T17:25:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3633015325}) 2024-04-20T17:25:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2756545193}) 2024-04-20T17:25:33.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2353682880}) 2024-04-20T17:25:33.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3520192789}) 2024-04-20T17:25:33.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1371592320}) 2024-04-20T17:25:33.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1908124822}) 2024-04-20T17:25:33.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3475870644}) 2024-04-20T17:25:33.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1586003688}) 2024-04-20T17:25:33.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=494785912}) 2024-04-20T17:25:33.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2489078941}) 2024-04-20T17:25:33.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1902528043}) 2024-04-20T17:25:33.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=989173195}) 2024-04-20T17:25:33.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3095696304}) 2024-04-20T17:25:33.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=955334209}) 2024-04-20T17:25:33.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=868081055}) 2024-04-20T17:25:33.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4129100311}) 2024-04-20T17:25:33.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3774417330}) 2024-04-20T17:25:33.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3513069639}) 2024-04-20T17:25:33.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2256552334}) 2024-04-20T17:25:33.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3773222806}) 2024-04-20T17:25:33.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3162875977}) 2024-04-20T17:25:33.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=560768723}) 2024-04-20T17:25:33.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1445691944}) 2024-04-20T17:25:33.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=528107312}) 2024-04-20T17:25:33.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2885423832}) 2024-04-20T17:25:33.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1090995966}) 2024-04-20T17:25:33.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=144576810}) 2024-04-20T17:25:33.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4149142266}) 2024-04-20T17:25:33.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4041335277}) 2024-04-20T17:25:33.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4276030502}) 2024-04-20T17:25:33.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1277436686}) 2024-04-20T17:25:33.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=617026455}) 2024-04-20T17:25:33.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=697954303}) 2024-04-20T17:25:33.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2392268187}) 2024-04-20T17:25:33.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1674574877}) 2024-04-20T17:25:33.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=228972859}) 2024-04-20T17:25:33.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1638486260}) 2024-04-20T17:25:33.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3850040146}) 2024-04-20T17:25:33.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3592054272}) 2024-04-20T17:25:33.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1809226083}) 2024-04-20T17:25:33.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2918497377}) 2024-04-20T17:25:33.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3963552585}) 2024-04-20T17:25:33.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=533087145}) 2024-04-20T17:25:33.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=841868992}) 2024-04-20T17:25:33.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2284971}) 2024-04-20T17:25:33.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3786753652}) 2024-04-20T17:25:33.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=110151907}) 2024-04-20T17:25:33.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=2,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=63} crc {logm=1465928792,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=1838383097,mon_config_key=1570156382}) 2024-04-20T17:25:33.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3865000865}) 2024-04-20T17:25:33.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=570385374}) 2024-04-20T17:25:33.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=27,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=55} crc {mon_config_key=2368208904,monmap=717113318,osd_metadata=2016089214,osd_pg_creating=3592309798,osdmap=994340108}) 2024-04-20T17:25:33.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=566897761}) 2024-04-20T17:25:33.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=4} crc {osdmap=3186609539}) 2024-04-20T17:25:33.244 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:25:33.281 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:33.305 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:25:33.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:33.308 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:25:33.409 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:25:33.445 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:33.470 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:25:33.470 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 45 16 113809 113793 158.02 99.375 0.00910511 0.00632677 2024-04-20T17:25:33.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:33.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:25:33.573 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:25:33.602 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:33.634 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:25:33.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:33.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:25:33.738 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:25:33.829 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:33.890 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:25:33.931 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:25:33.960 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:25:33.993 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:25:34.020 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:25:34.055 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:25:34.090 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:25:34.129 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:25:34.136 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:33 smithi005 ceph-mon[39562]: pgmap v1241: 17 pgs: 17 active+clean; 6.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.56k op/s 2024-04-20T17:25:34.167 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:25:34.207 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:25:34.238 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:25:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:33 smithi106 ceph-mon[44423]: pgmap v1241: 17 pgs: 17 active+clean; 6.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.56k op/s 2024-04-20T17:25:34.279 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:25:34.290 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:25:34.290 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:25:34.290 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:25:34.290 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:25:34.291 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:25:34.291 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:25:34.291 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:25:34.291 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:25:34.291 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:25:34.291 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:25:34.291 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:25:34.291 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:25:34.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:33 smithi171 ceph-mon[44189]: pgmap v1241: 17 pgs: 17 active+clean; 6.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.56k op/s 2024-04-20T17:25:34.470 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 46 16 115354 115338 156.683 96.5625 0.0119919 0.006381 2024-04-20T17:25:35.187 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:34 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1760749145' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:25:35.187 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.187 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.187 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.187 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.187 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.187 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.187 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.187 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.188 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.188 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:34 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1760749145' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:25:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:34 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1760749145' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:25:35.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.470 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 47 16 116976 116960 155.506 101.375 0.0100888 0.00642952 2024-04-20T17:25:35.865 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.8 to 1 (10000) 2024-04-20T17:25:35.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:35.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:35 smithi005 ceph-mon[39562]: pgmap v1242: 17 pgs: 17 active+clean; 6.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.63k op/s 2024-04-20T17:25:35.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:35 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/821562526' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-20T17:25:35.895 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:35 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:25:35.895 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:35 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:25:36.102 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:25:36.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:36.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:25:36.206 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:25:36.257 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:36.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:35 smithi106 ceph-mon[44423]: pgmap v1242: 17 pgs: 17 active+clean; 6.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.63k op/s 2024-04-20T17:25:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:35 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/821562526' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-20T17:25:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:35 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:25:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:35 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:25:36.291 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:25:36.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:36.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:25:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:35 smithi171 ceph-mon[44189]: pgmap v1242: 17 pgs: 17 active+clean; 6.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.63k op/s 2024-04-20T17:25:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:35 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/821562526' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-20T17:25:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:35 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:25:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:35 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:25:36.402 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:25:36.442 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:36.470 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 48 16 118536 118520 154.298 97.5 0.0112582 0.00647966 2024-04-20T17:25:36.471 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:25:36.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:36.475 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:25:36.576 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:25:36.613 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:36.631 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:25:36.635 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:25:36.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:36.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:25:36.682 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 9 1 2024-04-20T17:25:36.739 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:25:36.811 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:36.852 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:25:36.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:36.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:25:36.923 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:36 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/821562526' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-20T17:25:36.923 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:36 smithi005 ceph-mon[39562]: osdmap e469: 12 total, 12 up, 12 in 2024-04-20T17:25:36.924 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:36 smithi005 ceph-mon[39562]: osdmap e470: 12 total, 12 up, 12 in 2024-04-20T17:25:36.971 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:25:37.020 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:37.067 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:25:37.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:37.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:25:37.173 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:25:37.249 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:36 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/821562526' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-20T17:25:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:36 smithi106 ceph-mon[44423]: osdmap e469: 12 total, 12 up, 12 in 2024-04-20T17:25:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:36 smithi106 ceph-mon[44423]: osdmap e470: 12 total, 12 up, 12 in 2024-04-20T17:25:37.281 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:25:37.281 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:25:37.283 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:25:37.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:37.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:25:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:36 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/821562526' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-20T17:25:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:36 smithi171 ceph-mon[44189]: osdmap e469: 12 total, 12 up, 12 in 2024-04-20T17:25:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:36 smithi171 ceph-mon[44189]: osdmap e470: 12 total, 12 up, 12 in 2024-04-20T17:25:37.389 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:25:37.427 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:37.458 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:25:37.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:37.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:25:37.470 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 49 16 119993 119977 153.007 91.0625 0.00422502 0.00653421 2024-04-20T17:25:37.561 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:25:37.619 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:38.189 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:25:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:37 smithi005 ceph-mon[39562]: pgmap v1245: 17 pgs: 17 active+clean; 7.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 116 MiB/s wr, 1.86k op/s 2024-04-20T17:25:38.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:37 smithi106 ceph-mon[44423]: pgmap v1245: 17 pgs: 17 active+clean; 7.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 116 MiB/s wr, 1.86k op/s 2024-04-20T17:25:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:37 smithi171 ceph-mon[44189]: pgmap v1245: 17 pgs: 17 active+clean; 7.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 116 MiB/s wr, 1.86k op/s 2024-04-20T17:25:38.470 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 50 16 121567 121551 151.914 98.375 0.0221678 0.00658111 2024-04-20T17:25:38.890 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:25:38.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:38.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:25:38.999 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:25:39.045 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:39.474 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 51 16 123083 123067 150.783 94.75 0.00448724 0.00663063 2024-04-20T17:25:40.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:39 smithi005 ceph-mon[39562]: pgmap v1246: 17 pgs: 17 active+clean; 7.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.70k op/s 2024-04-20T17:25:40.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:39 smithi106 ceph-mon[44423]: pgmap v1246: 17 pgs: 17 active+clean; 7.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.70k op/s 2024-04-20T17:25:40.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:39 smithi171 ceph-mon[44189]: pgmap v1246: 17 pgs: 17 active+clean; 7.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.70k op/s 2024-04-20T17:25:40.474 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 52 16 124701 124685 149.828 101.125 0.0218082 0.00667262 2024-04-20T17:25:41.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:40 smithi005 ceph-mon[39562]: 4.9 scrub starts 2024-04-20T17:25:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:40 smithi106 ceph-mon[44423]: 4.9 scrub starts 2024-04-20T17:25:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:40 smithi171 ceph-mon[44189]: 4.9 scrub starts 2024-04-20T17:25:41.474 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 53 16 126155 126139 148.716 90.875 0.0121894 0.00672313 2024-04-20T17:25:42.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:41 smithi005 ceph-mon[39562]: pgmap v1247: 17 pgs: 17 active+clean; 7.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-20T17:25:42.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:41 smithi005 ceph-mon[39562]: 4.8 scrub starts 2024-04-20T17:25:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:41 smithi106 ceph-mon[44423]: pgmap v1247: 17 pgs: 17 active+clean; 7.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-20T17:25:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:41 smithi106 ceph-mon[44423]: 4.8 scrub starts 2024-04-20T17:25:42.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:41 smithi171 ceph-mon[44189]: pgmap v1247: 17 pgs: 17 active+clean; 7.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-20T17:25:42.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:41 smithi171 ceph-mon[44189]: 4.8 scrub starts 2024-04-20T17:25:42.475 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 54 16 127462 127446 147.473 81.6875 0.00967763 0.0067796 2024-04-20T17:25:43.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:43 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:25:43.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:43 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:25:43.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:43 smithi005 ceph-mon[39562]: osdmap e471: 12 total, 12 up, 12 in 2024-04-20T17:25:43.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:43 smithi005 ceph-mon[39562]: pgmap v1249: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.61k op/s 2024-04-20T17:25:43.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:43 smithi005 ceph-mon[39562]: 4.9 scrub ok 2024-04-20T17:25:43.475 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 55 16 128969 128953 146.504 94.1875 0.0040544 0.00682445 2024-04-20T17:25:43.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:43 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:25:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:43 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:25:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:43 smithi106 ceph-mon[44423]: osdmap e471: 12 total, 12 up, 12 in 2024-04-20T17:25:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:43 smithi106 ceph-mon[44423]: pgmap v1249: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.61k op/s 2024-04-20T17:25:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:43 smithi106 ceph-mon[44423]: 4.9 scrub ok 2024-04-20T17:25:43.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:43 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:25:43.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:43 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:25:43.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:43 smithi171 ceph-mon[44189]: osdmap e471: 12 total, 12 up, 12 in 2024-04-20T17:25:43.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:43 smithi171 ceph-mon[44189]: pgmap v1249: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.61k op/s 2024-04-20T17:25:43.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:43 smithi171 ceph-mon[44189]: 4.9 scrub ok 2024-04-20T17:25:44.273 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:25:44.323 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-20T17:25:44.345 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:25:44.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:44 smithi005 ceph-mon[39562]: 4.8 scrub ok 2024-04-20T17:25:44.475 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 56 16 130457 130441 145.548 93 0.00578103 0.00686907 2024-04-20T17:25:44.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:44 smithi106 ceph-mon[44423]: 4.8 scrub ok 2024-04-20T17:25:44.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:44 smithi171 ceph-mon[44189]: 4.8 scrub ok 2024-04-20T17:25:45.468 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:45 smithi005 ceph-mon[39562]: pgmap v1250: 17 pgs: 17 active+clean; 7.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.48k op/s 2024-04-20T17:25:45.476 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 57 16 131945 131929 144.626 93 0.00699948 0.00691351 2024-04-20T17:25:45.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:45 smithi106 ceph-mon[44423]: pgmap v1250: 17 pgs: 17 active+clean; 7.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.48k op/s 2024-04-20T17:25:45.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:45 smithi171 ceph-mon[44189]: pgmap v1250: 17 pgs: 17 active+clean; 7.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.48k op/s 2024-04-20T17:25:45.849 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:25:45.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:45.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:25:45.954 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:25:45.994 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:46.026 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:25:46.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:46.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:25:46.134 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:25:46.193 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:46 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:25:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:46 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:25:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:46 smithi106 ceph-mon[44423]: osdmap e472: 12 total, 12 up, 12 in 2024-04-20T17:25:46.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:46 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:25:46.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:46 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:25:46.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:46 smithi005 ceph-mon[39562]: osdmap e472: 12 total, 12 up, 12 in 2024-04-20T17:25:46.476 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 58 16 133403 133387 143.704 91.125 0.00797569 0.00695733 2024-04-20T17:25:46.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:46 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:25:46.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:46 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:25:46.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:46 smithi171 ceph-mon[44189]: osdmap e472: 12 total, 12 up, 12 in 2024-04-20T17:25:46.761 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:25:46.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:46.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:25:46.867 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:25:46.909 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:46.930 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:25:46.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:46.935 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:25:47.036 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:25:47.066 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:47.093 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:25:47.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:47.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:25:47.174 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:47 smithi171 ceph-mon[44189]: 4.a scrub starts 2024-04-20T17:25:47.175 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:47 smithi171 ceph-mon[44189]: pgmap v1252: 17 pgs: 17 active+clean; 7.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-04-20T17:25:47.198 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:25:47.276 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:47.317 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:25:47.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:47.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:25:47.430 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:25:47.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:47 smithi005 ceph-mon[39562]: 4.a scrub starts 2024-04-20T17:25:47.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:47 smithi005 ceph-mon[39562]: pgmap v1252: 17 pgs: 17 active+clean; 7.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-04-20T17:25:47.476 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 59 16 134703 134687 142.645 81.25 0.00428971 0.00700881 2024-04-20T17:25:47.494 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:47.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:47 smithi106 ceph-mon[44423]: 4.a scrub starts 2024-04-20T17:25:47.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:47 smithi106 ceph-mon[44423]: pgmap v1252: 17 pgs: 17 active+clean; 7.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-04-20T17:25:47.834 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:25:47.841 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:25:47.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:47.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:25:47.952 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:25:47.982 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:48.006 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:25:48.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:48.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:25:48.110 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:25:48.146 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:48.172 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:25:48.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:48.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:25:48.206 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:25:48.208 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.9 to 1 (10000) 2024-04-20T17:25:48.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:48 smithi106 ceph-mon[44423]: 4.a scrub ok 2024-04-20T17:25:48.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:48 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2613568255' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:25:48.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:48 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2163416932' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:25:48.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:48 smithi106 ceph-mon[44423]: from='client.26213 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:25:48.279 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:25:48.336 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:48.358 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:25:48.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:48.362 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:25:48.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:48 smithi005 ceph-mon[39562]: 4.a scrub ok 2024-04-20T17:25:48.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:48 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2613568255' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:25:48.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:48 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2163416932' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:25:48.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:48 smithi005 ceph-mon[39562]: from='client.26213 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:25:48.462 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:25:48.481 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:25:48.476817 min lat: 0.00113144 max lat: 0.214111 avg lat: 0.00705735 2024-04-20T17:25:48.481 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:25:48.481 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 60 16 136025 136009 141.645 82.625 0.0742308 0.00705735 2024-04-20T17:25:48.577 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:48.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:48 smithi171 ceph-mon[44189]: 4.a scrub ok 2024-04-20T17:25:48.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:48 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2613568255' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:25:48.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:48 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2163416932' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:25:48.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:48 smithi171 ceph-mon[44189]: from='client.26213 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:25:48.830 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:25:48.865 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:25:48.903 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:25:48.935 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:25:48.964 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:25:49.004 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:25:49.032 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:25:49.071 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:25:49.101 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:25:49.136 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:25:49.178 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:25:49.216 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:25:49.217 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:25:49.218 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:25:49.218 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:25:49.218 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:25:49.218 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:25:49.218 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:25:49.218 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:25:49.218 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:25:49.219 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:25:49.222 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:25:49.222 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:25:49.223 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:25:49.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:49 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:25:49.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:49 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:25:49.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:49 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:25:49.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:49 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:25:49.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:49 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2613568255' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:25:49.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:49 smithi005 ceph-mon[39562]: from='client.26213 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-20T17:25:49.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:49 smithi005 ceph-mon[39562]: osdmap e473: 12 total, 12 up, 12 in 2024-04-20T17:25:49.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:49 smithi005 ceph-mon[39562]: pgmap v1254: 17 pgs: 17 active+clean; 8.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s wr, 1.66k op/s 2024-04-20T17:25:49.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:49 smithi005 ceph-mon[39562]: 4.1 scrub starts 2024-04-20T17:25:49.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:49 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/618725270' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:25:49.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:49.476 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 61 16 137346 137330 140.677 82.5625 0.013843 0.00710672 2024-04-20T17:25:49.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:49 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:25:49.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:49 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:25:49.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:49 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:25:49.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:49 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:25:49.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:49 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2613568255' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:25:49.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:49 smithi106 ceph-mon[44423]: from='client.26213 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-20T17:25:49.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:49 smithi106 ceph-mon[44423]: osdmap e473: 12 total, 12 up, 12 in 2024-04-20T17:25:49.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:49 smithi106 ceph-mon[44423]: pgmap v1254: 17 pgs: 17 active+clean; 8.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s wr, 1.66k op/s 2024-04-20T17:25:49.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:49 smithi106 ceph-mon[44423]: 4.1 scrub starts 2024-04-20T17:25:49.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:49 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/618725270' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:25:49.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:49.531 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:25:49.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:49.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:25:49.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:49 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:25:49.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:49 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:25:49.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:49 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:25:49.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:49 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:25:49.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:49 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2613568255' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:25:49.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:49 smithi171 ceph-mon[44189]: from='client.26213 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-20T17:25:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:49 smithi171 ceph-mon[44189]: osdmap e473: 12 total, 12 up, 12 in 2024-04-20T17:25:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:49 smithi171 ceph-mon[44189]: pgmap v1254: 17 pgs: 17 active+clean; 8.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s wr, 1.66k op/s 2024-04-20T17:25:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:49 smithi171 ceph-mon[44189]: 4.1 scrub starts 2024-04-20T17:25:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:49 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/618725270' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:25:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:49.636 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:25:49.679 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:49.713 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:25:49.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:49.717 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:25:49.818 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:25:49.865 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:49.938 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:25:49.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:49.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:25:50.043 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:25:50.074 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:50.096 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:25:50.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:50.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:25:50.201 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:25:50.239 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:50.272 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:25:50.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:50.275 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:25:50.376 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:25:50.408 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:50.408 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.408 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.408 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.408 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.408 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.409 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.409 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.409 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.409 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.409 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.409 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.433 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:25:50.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:50.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:25:50.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.476 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 62 16 138722 138706 139.795 86 0.00569451 0.00715206 2024-04-20T17:25:50.536 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:25:50.569 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:50.580 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 10 1 2024-04-20T17:25:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:50.593 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:25:50.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:50.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:25:50.701 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:25:50.735 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:50.761 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:25:50.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:50.764 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:25:50.865 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:25:50.938 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:51.106 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:25:51.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:51.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:25:51.216 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:25:51.270 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:51.379 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:25:51.384 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:51 smithi005 ceph-mon[39562]: pgmap v1255: 17 pgs: 17 active+clean; 8.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-20T17:25:51.384 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:51 smithi005 ceph-mon[39562]: 4.1 scrub ok 2024-04-20T17:25:51.384 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:51 smithi005 ceph-mon[39562]: osdmap e474: 12 total, 12 up, 12 in 2024-04-20T17:25:51.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:51.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:25:51.476 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 63 16 140049 140033 138.892 82.9375 0.0119389 0.00719841 2024-04-20T17:25:51.486 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:25:51.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:51 smithi106 ceph-mon[44423]: pgmap v1255: 17 pgs: 17 active+clean; 8.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-20T17:25:51.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:51 smithi106 ceph-mon[44423]: 4.1 scrub ok 2024-04-20T17:25:51.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:51 smithi106 ceph-mon[44423]: osdmap e474: 12 total, 12 up, 12 in 2024-04-20T17:25:51.534 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:51.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:51 smithi171 ceph-mon[44189]: pgmap v1255: 17 pgs: 17 active+clean; 8.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-20T17:25:51.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:51 smithi171 ceph-mon[44189]: 4.1 scrub ok 2024-04-20T17:25:51.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:51 smithi171 ceph-mon[44189]: osdmap e474: 12 total, 12 up, 12 in 2024-04-20T17:25:51.587 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:25:51.587 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:25:51.587 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:25:51.588 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:25:51.588 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:25:52.476 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 64 16 141435 141419 138.076 86.625 0.00537113 0.007241 2024-04-20T17:25:52.810 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:25:52.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:52.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:25:52.846 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:25:52.860 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-20T17:25:52.895 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:25:52.916 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:25:52.953 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:52.980 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:25:52.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:52.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:25:53.086 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:25:53.123 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:53.148 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:25:53.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:53.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:25:53.257 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:25:53.349 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:53.402 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:25:53.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:53.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:25:53.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:53 smithi005 ceph-mon[39562]: pgmap v1257: 17 pgs: 17 active+clean; 8.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-04-20T17:25:53.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:53 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:25:53.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:53 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:25:53.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:53 smithi005 ceph-mon[39562]: osdmap e475: 12 total, 12 up, 12 in 2024-04-20T17:25:53.476 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 65 16 142589 142573 137.061 72.125 0.0136361 0.00729422 2024-04-20T17:25:53.506 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:25:53.538 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:53.560 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:25:53.565 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:53 smithi106 ceph-mon[44423]: pgmap v1257: 17 pgs: 17 active+clean; 8.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-04-20T17:25:53.565 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:53 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:25:53.565 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:53 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:25:53.565 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:53 smithi106 ceph-mon[44423]: osdmap e475: 12 total, 12 up, 12 in 2024-04-20T17:25:53.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:53.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:25:53.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:53 smithi171 ceph-mon[44189]: pgmap v1257: 17 pgs: 17 active+clean; 8.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-04-20T17:25:53.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:53 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:25:53.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:53 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:25:53.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:53 smithi171 ceph-mon[44189]: osdmap e475: 12 total, 12 up, 12 in 2024-04-20T17:25:53.666 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:25:53.733 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:53.855 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:25:53.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:53.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:25:53.967 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:25:54.012 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:54.265 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:25:54.347 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:25:54.371 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:25:54.371 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:25:54.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:54.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:25:54.476 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 66 16 143676 143660 136.014 67.9375 0.0122403 0.00734958 2024-04-20T17:25:54.483 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:25:54.532 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:54.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:54 smithi171 ceph-mon[44189]: 4.2 deep-scrub starts 2024-04-20T17:25:54.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:54 smithi171 ceph-mon[44189]: 4.0 scrub starts 2024-04-20T17:25:54.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:54 smithi005 ceph-mon[39562]: 4.2 deep-scrub starts 2024-04-20T17:25:54.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:54 smithi005 ceph-mon[39562]: 4.0 scrub starts 2024-04-20T17:25:54.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:54 smithi106 ceph-mon[44423]: 4.2 deep-scrub starts 2024-04-20T17:25:54.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:54 smithi106 ceph-mon[44423]: 4.0 scrub starts 2024-04-20T17:25:54.765 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:25:54.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:54.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:25:54.869 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:25:54.901 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:54.926 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:25:54.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:54.932 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:25:55.033 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:25:55.066 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:55.095 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:25:55.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:55.102 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:25:55.203 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:25:55.240 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:55.265 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:25:55.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:55.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:25:55.305 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:55 smithi171 ceph-mon[44189]: 4.d scrub starts 2024-04-20T17:25:55.305 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:55 smithi171 ceph-mon[44189]: pgmap v1259: 17 pgs: 17 active+clean; 8.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-20T17:25:55.305 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:55 smithi171 ceph-mon[44189]: 4.0 scrub ok 2024-04-20T17:25:55.370 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:25:55.402 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:55 smithi005 ceph-mon[39562]: 4.d scrub starts 2024-04-20T17:25:55.402 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:55 smithi005 ceph-mon[39562]: pgmap v1259: 17 pgs: 17 active+clean; 8.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-20T17:25:55.402 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:55 smithi005 ceph-mon[39562]: 4.0 scrub ok 2024-04-20T17:25:55.404 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:55.476 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 67 16 144735 144719 134.972 66.1875 0.00933056 0.00740674 2024-04-20T17:25:55.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:55 smithi106 ceph-mon[44423]: 4.d scrub starts 2024-04-20T17:25:55.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:55 smithi106 ceph-mon[44423]: pgmap v1259: 17 pgs: 17 active+clean; 8.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-20T17:25:55.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:55 smithi106 ceph-mon[44423]: 4.0 scrub ok 2024-04-20T17:25:55.560 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:25:55.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:55.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:25:55.664 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:25:55.704 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:55.735 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:25:55.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:55.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:25:55.844 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:25:55.874 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:55.905 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:25:55.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:55.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:25:56.010 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:25:56.055 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:56.476 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 68 16 145923 145907 134.079 74.25 0.0149106 0.00745689 2024-04-20T17:25:56.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:56 smithi171 ceph-mon[44189]: 4.d scrub ok 2024-04-20T17:25:56.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:56 smithi171 ceph-mon[44189]: osdmap e476: 12 total, 12 up, 12 in 2024-04-20T17:25:56.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:56 smithi005 ceph-mon[39562]: 4.d scrub ok 2024-04-20T17:25:56.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:56 smithi005 ceph-mon[39562]: osdmap e476: 12 total, 12 up, 12 in 2024-04-20T17:25:56.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:56 smithi106 ceph-mon[44423]: 4.d scrub ok 2024-04-20T17:25:56.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:56 smithi106 ceph-mon[44423]: osdmap e476: 12 total, 12 up, 12 in 2024-04-20T17:25:56.930 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:25:56.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:56.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:25:57.035 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:25:57.082 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:57.170 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.10 to 1 (10000) 2024-04-20T17:25:57.236 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:25:57.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:57.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:25:57.344 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:25:57.389 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:57.422 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:25:57.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:57.432 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:25:57.459 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:25:57.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:57 smithi005 ceph-mon[39562]: pgmap v1261: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.48k op/s 2024-04-20T17:25:57.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:57 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1359691081' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:25:57.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:57 smithi005 ceph-mon[39562]: from='client.26140 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:25:57.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:57 smithi005 ceph-mon[39562]: from='client.26140 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-20T17:25:57.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:57 smithi005 ceph-mon[39562]: osdmap e477: 12 total, 12 up, 12 in 2024-04-20T17:25:57.476 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 69 16 147096 147080 133.198 73.3125 0.00837082 0.00750618 2024-04-20T17:25:57.534 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:25:57.570 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:57.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:57 smithi171 ceph-mon[44189]: pgmap v1261: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.48k op/s 2024-04-20T17:25:57.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:57 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1359691081' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:25:57.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:57 smithi171 ceph-mon[44189]: from='client.26140 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:25:57.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:57 smithi171 ceph-mon[44189]: from='client.26140 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-20T17:25:57.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:57 smithi171 ceph-mon[44189]: osdmap e477: 12 total, 12 up, 12 in 2024-04-20T17:25:57.599 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:25:57.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:57.607 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:25:57.708 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:25:57.746 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:57.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:57 smithi106 ceph-mon[44423]: pgmap v1261: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.48k op/s 2024-04-20T17:25:57.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:57 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1359691081' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:25:57.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:57 smithi106 ceph-mon[44423]: from='client.26140 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:25:57.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:57 smithi106 ceph-mon[44423]: from='client.26140 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-20T17:25:57.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:57 smithi106 ceph-mon[44423]: osdmap e477: 12 total, 12 up, 12 in 2024-04-20T17:25:57.782 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:25:57.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:57.784 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:25:57.886 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:25:57.918 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:57.945 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:25:57.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:57.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:25:58.031 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:25:58.051 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:25:58.058 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:25:58.081 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:25:58.082 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:58.106 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:25:58.108 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:25:58.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:58.116 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:25:58.130 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:25:58.166 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:25:58.198 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:25:58.217 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:25:58.228 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:25:58.267 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:25:58.294 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:58.298 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 11 1 2024-04-20T17:25:58.300 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:25:58.331 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:25:58.341 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:25:58.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:58.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:25:58.381 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:25:58.391 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:25:58.391 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:25:58.391 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:25:58.391 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:25:58.391 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:25:58.392 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:25:58.392 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:25:58.392 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:25:58.392 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:25:58.392 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:25:58.392 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:25:58.392 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:25:58.444 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:25:58.477 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 70 16 148233 148217 132.31 71.0625 0.00573723 0.00755637 2024-04-20T17:25:58.483 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:58.505 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:25:58.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:58.512 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:25:58.613 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:25:58.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:58 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2030665990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:25:58.668 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:58.668 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:58.670 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:58.751 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:25:58.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:58.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:25:58.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:58 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2030665990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:25:58.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:58.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:58.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:58 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2030665990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:25:58.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:58.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:58.857 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:25:58.895 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:58.921 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:25:58.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:58.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:25:59.028 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:25:59.082 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:59.274 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:25:59.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:59.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:25:59.356 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.356 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.356 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.356 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.356 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.356 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.356 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:59 smithi005 ceph-mon[39562]: pgmap v1263: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-04-20T17:25:59.356 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.356 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.357 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.357 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.357 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:25:59 smithi005 ceph-mon[39562]: 4.6 scrub starts 2024-04-20T17:25:59.390 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:25:59.412 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:25:59.413 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:25:59.428 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:59.457 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:25:59.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:25:59.462 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:25:59.477 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 71 16 149281 149265 131.369 65.5 0.0228709 0.00761065 2024-04-20T17:25:59.563 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:25:59.617 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:25:59.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:59 smithi106 ceph-mon[44423]: pgmap v1263: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-04-20T17:25:59.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:25:59 smithi106 ceph-mon[44423]: 4.6 scrub starts 2024-04-20T17:25:59.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:59 smithi171 ceph-mon[44189]: pgmap v1263: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-04-20T17:25:59.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:25:59.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:25:59 smithi171 ceph-mon[44189]: 4.6 scrub starts 2024-04-20T17:26:00.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:00 smithi005 ceph-mon[39562]: 4.4 scrub starts 2024-04-20T17:26:00.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:00 smithi005 ceph-mon[39562]: 4.6 scrub ok 2024-04-20T17:26:00.476 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 72 16 150337 150321 130.461 66 0.00570111 0.00766138 2024-04-20T17:26:00.594 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:26:00.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:00.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:26:00.620 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-20T17:26:00.648 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:00 smithi106 ceph-mon[44423]: 4.4 scrub starts 2024-04-20T17:26:00.648 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:00 smithi106 ceph-mon[44423]: 4.6 scrub ok 2024-04-20T17:26:00.700 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:26:00.729 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:00 smithi171 ceph-mon[44189]: 4.4 scrub starts 2024-04-20T17:26:00.729 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:00 smithi171 ceph-mon[44189]: 4.6 scrub ok 2024-04-20T17:26:00.732 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:00.760 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:26:00.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:00.765 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:26:00.866 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:26:00.905 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:00.984 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:26:01.062 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:26:01.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:01.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:26:01.167 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:26:01.215 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:01.252 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:26:01.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:01.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:26:01.357 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:26:01.401 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:01.438 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:26:01.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:01.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:26:01.477 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 73 16 151420 151404 129.602 67.6875 0.0188749 0.00771356 2024-04-20T17:26:01.496 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:26:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:01 smithi106 ceph-mon[44423]: pgmap v1264: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-20T17:26:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:01 smithi106 ceph-mon[44423]: osdmap e478: 12 total, 12 up, 12 in 2024-04-20T17:26:01.542 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:26:01.579 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:01.610 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:26:01.613 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:01 smithi171 ceph-mon[44189]: pgmap v1264: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-20T17:26:01.613 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:01 smithi171 ceph-mon[44189]: osdmap e478: 12 total, 12 up, 12 in 2024-04-20T17:26:01.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:01.615 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:26:01.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:01 smithi005 ceph-mon[39562]: pgmap v1264: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-20T17:26:01.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:01 smithi005 ceph-mon[39562]: osdmap e478: 12 total, 12 up, 12 in 2024-04-20T17:26:01.716 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:26:01.795 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:02.284 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:26:02.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:02.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:26:02.370 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:02 smithi106 ceph-mon[44423]: 4.4 scrub ok 2024-04-20T17:26:02.371 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:02 smithi106 ceph-mon[44423]: 4.c scrub starts 2024-04-20T17:26:02.371 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:02 smithi106 ceph-mon[44423]: 4.b scrub starts 2024-04-20T17:26:02.372 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:02 smithi005 ceph-mon[39562]: 4.4 scrub ok 2024-04-20T17:26:02.372 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:02 smithi005 ceph-mon[39562]: 4.c scrub starts 2024-04-20T17:26:02.372 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:02 smithi005 ceph-mon[39562]: 4.b scrub starts 2024-04-20T17:26:02.395 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:26:02.435 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:02.467 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:26:02.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:02.476 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:26:02.477 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 74 16 152410 152394 128.686 61.875 0.00939124 0.00776917 2024-04-20T17:26:02.578 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:26:02.632 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:02.700 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:26:02.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:02.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:26:02.805 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:26:02.832 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:02 smithi171 ceph-mon[44189]: 4.4 scrub ok 2024-04-20T17:26:02.832 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:02 smithi171 ceph-mon[44189]: 4.c scrub starts 2024-04-20T17:26:02.832 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:02 smithi171 ceph-mon[44189]: 4.b scrub starts 2024-04-20T17:26:02.834 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:02.860 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:26:02.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:02.865 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:26:02.966 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:26:03.002 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:03.088 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:26:03.217 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:26:03.217 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:26:03.223 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:26:03.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:03.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:26:03.329 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:26:03.371 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:03.384 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:26:03.405 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:26:03.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:03.408 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:26:03.477 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 75 16 153338 153322 127.744 58 0.018746 0.00782503 2024-04-20T17:26:03.509 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:26:03.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:03 smithi106 ceph-mon[44423]: 4.e scrub starts 2024-04-20T17:26:03.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:03 smithi106 ceph-mon[44423]: pgmap v1266: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-04-20T17:26:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:03 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2061878445' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:26:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:03 smithi106 ceph-mon[44423]: from='client.26152 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:26:03.549 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:03.578 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:26:03.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:03.583 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:26:03.684 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:26:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:03 smithi005 ceph-mon[39562]: 4.e scrub starts 2024-04-20T17:26:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:03 smithi005 ceph-mon[39562]: pgmap v1266: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-04-20T17:26:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:03 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2061878445' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:26:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:03 smithi005 ceph-mon[39562]: from='client.26152 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:26:03.716 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:03 smithi171 ceph-mon[44189]: 4.e scrub starts 2024-04-20T17:26:03.716 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:03 smithi171 ceph-mon[44189]: pgmap v1266: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-04-20T17:26:03.716 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:03 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2061878445' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:26:03.716 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:03 smithi171 ceph-mon[44189]: from='client.26152 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:26:03.719 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:03.748 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:26:03.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:03.756 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:26:03.856 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:26:03.904 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:03.930 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:26:03.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:03.936 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:26:04.037 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:26:04.100 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:04.156 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-20T17:26:04.189 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:26:04.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:04.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:26:04.306 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:26:04.348 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:04.372 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:26:04.378 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:04 smithi106 ceph-mon[44423]: 4.c scrub ok 2024-04-20T17:26:04.378 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:04 smithi106 ceph-mon[44423]: 4.7 scrub starts 2024-04-20T17:26:04.379 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:04 smithi106 ceph-mon[44423]: 4.b scrub ok 2024-04-20T17:26:04.379 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:04 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:26:04.379 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:04 smithi106 ceph-mon[44423]: from='client.26152 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:26:04.379 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:04 smithi106 ceph-mon[44423]: osdmap e479: 12 total, 12 up, 12 in 2024-04-20T17:26:04.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:04.380 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:26:04.477 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 76 16 154427 154411 126.959 68.0625 0.00508429 0.00787387 2024-04-20T17:26:04.481 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:26:04.513 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:26:04.528 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:04.631 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:26:04.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:04.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:26:04.647 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:04 smithi005 ceph-mon[39562]: 4.c scrub ok 2024-04-20T17:26:04.647 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:04 smithi005 ceph-mon[39562]: 4.7 scrub starts 2024-04-20T17:26:04.648 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:04 smithi005 ceph-mon[39562]: 4.b scrub ok 2024-04-20T17:26:04.648 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:04 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:26:04.648 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:04 smithi005 ceph-mon[39562]: from='client.26152 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:26:04.648 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:04 smithi005 ceph-mon[39562]: osdmap e479: 12 total, 12 up, 12 in 2024-04-20T17:26:04.724 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:26:04.747 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:26:04.802 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:04.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:04 smithi171 ceph-mon[44189]: 4.c scrub ok 2024-04-20T17:26:04.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:04 smithi171 ceph-mon[44189]: 4.7 scrub starts 2024-04-20T17:26:04.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:04 smithi171 ceph-mon[44189]: 4.b scrub ok 2024-04-20T17:26:04.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:04 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:26:04.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:04 smithi171 ceph-mon[44189]: from='client.26152 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:26:04.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:04 smithi171 ceph-mon[44189]: osdmap e479: 12 total, 12 up, 12 in 2024-04-20T17:26:05.068 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:26:05.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:05.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:26:05.177 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:26:05.235 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:05.303 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:26:05.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:05.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:26:05.399 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:05 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:26:05.399 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:05 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:26:05.399 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:05 smithi106 ceph-mon[44423]: pgmap v1268: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-20T17:26:05.399 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:05 smithi106 ceph-mon[44423]: osdmap e480: 12 total, 12 up, 12 in 2024-04-20T17:26:05.399 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:05 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1057489350' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:26:05.400 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:05 smithi106 ceph-mon[44423]: from='client.26158 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:26:05.400 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.11 to 1 (10000) 2024-04-20T17:26:05.407 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:26:05.448 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:05.474 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:26:05.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:05.477 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:26:05.477 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 77 16 155541 155525 126.214 69.625 0.00762731 0.00792169 2024-04-20T17:26:05.578 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:26:05.627 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:05.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:05 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:26:05.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:05 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:26:05.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:05 smithi005 ceph-mon[39562]: pgmap v1268: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-20T17:26:05.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:05 smithi005 ceph-mon[39562]: osdmap e480: 12 total, 12 up, 12 in 2024-04-20T17:26:05.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:05 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1057489350' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:26:05.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:05 smithi005 ceph-mon[39562]: from='client.26158 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:26:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:05 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:26:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:05 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:26:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:05 smithi171 ceph-mon[44189]: pgmap v1268: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-20T17:26:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:05 smithi171 ceph-mon[44189]: osdmap e480: 12 total, 12 up, 12 in 2024-04-20T17:26:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:05 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1057489350' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:26:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:05 smithi171 ceph-mon[44189]: from='client.26158 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:26:06.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:06 smithi005 ceph-mon[39562]: from='client.26158 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-20T17:26:06.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:06 smithi005 ceph-mon[39562]: osdmap e481: 12 total, 12 up, 12 in 2024-04-20T17:26:06.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:06 smithi005 ceph-mon[39562]: osdmap e482: 12 total, 12 up, 12 in 2024-04-20T17:26:06.477 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 78 16 156674 156658 125.504 70.8125 0.00547071 0.00796525 2024-04-20T17:26:06.657 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:26:06.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:06 smithi106 ceph-mon[44423]: from='client.26158 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-20T17:26:06.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:06 smithi106 ceph-mon[44423]: osdmap e481: 12 total, 12 up, 12 in 2024-04-20T17:26:06.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:06 smithi106 ceph-mon[44423]: osdmap e482: 12 total, 12 up, 12 in 2024-04-20T17:26:06.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:06 smithi171 ceph-mon[44189]: from='client.26158 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-20T17:26:06.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:06 smithi171 ceph-mon[44189]: osdmap e481: 12 total, 12 up, 12 in 2024-04-20T17:26:06.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:06 smithi171 ceph-mon[44189]: osdmap e482: 12 total, 12 up, 12 in 2024-04-20T17:26:06.935 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 7 1 2024-04-20T17:26:07.477 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 79 15 157679 157664 124.711 62.875 0.0146436 0.00801705 2024-04-20T17:26:07.579 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:26:07.592 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:26:07.595 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:26:07.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:07.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:26:07.613 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:26:07.655 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:26:07.689 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:26:07.697 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:26:07.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:07 smithi005 ceph-mon[39562]: pgmap v1272: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.13k op/s 2024-04-20T17:26:07.719 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:26:07.724 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:07 smithi106 ceph-mon[44423]: pgmap v1272: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.13k op/s 2024-04-20T17:26:07.728 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:07.750 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:26:07.781 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:26:07.808 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:26:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:07 smithi171 ceph-mon[44189]: pgmap v1272: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.13k op/s 2024-04-20T17:26:07.835 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:26:07.863 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:26:07.894 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:26:07.922 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:26:07.933 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:26:07.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:07.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:26:07.949 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:26:07.955 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:26:07.955 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:26:07.955 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:26:07.956 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:26:07.956 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:26:07.956 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:26:07.956 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:26:07.956 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:26:07.956 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:26:07.956 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:26:07.956 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:26:07.956 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:26:08.040 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:26:08.081 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:08.142 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:26:08.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:08.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:26:08.246 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:26:08.288 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:08.306 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:26:08.306 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:26:08.306 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-04-20T17:26:08.306 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-04-20T17:26:08.306 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:26:08.320 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:26:08.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:08.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:26:08.428 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:26:08.474 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:08.477 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:26:08.478532 min lat: 0.00113144 max lat: 0.214111 avg lat: 0.00806773 2024-04-20T17:26:08.477 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:26:08.477 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 80 16 158674 158658 123.929 62.125 0.0121132 0.00806773 2024-04-20T17:26:08.509 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:26:08.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:08.518 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:26:08.529 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:08 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2901652949' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:26:08.530 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:08.530 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:08.530 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:08.530 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:08.618 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:26:08.658 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:08 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2901652949' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:26:08.658 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:08.658 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:08.658 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:08.658 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:08.661 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:08.690 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:26:08.694 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:08 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2901652949' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:26:08.694 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:08.694 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:08.694 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:08.694 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:08.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:08.696 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:26:08.797 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:26:08.841 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:09.165 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-20T17:26:09.194 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:26:09.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:09.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:26:09.300 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:26:09.342 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:09.377 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:26:09.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:09.381 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:26:09.409 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:09.409 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:09.409 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:09.409 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:09.409 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:09.409 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:09.409 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:09.409 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:09.409 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:09 smithi106 ceph-mon[44423]: pgmap v1273: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s 2024-04-20T17:26:09.478 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 81 16 159739 159723 123.22 66.5625 0.0695932 0.00811292 2024-04-20T17:26:09.482 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:26:09.514 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:09.529 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:26:09.539 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:26:09.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:09.542 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:26:09.643 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:26:09.678 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:09.678 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:09.678 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:09.679 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:09.679 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:09.679 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:09.679 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:09.679 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:09.679 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:09 smithi171 ceph-mon[44189]: pgmap v1273: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s 2024-04-20T17:26:09.681 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:09.709 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:26:09.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:09.713 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:26:09.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:09.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:09.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:09.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:09.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:09.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:09.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:09.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:09.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:09 smithi005 ceph-mon[39562]: pgmap v1273: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s 2024-04-20T17:26:09.814 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:26:09.855 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:09.882 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:26:09.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:09.885 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:26:09.986 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:26:10.033 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:10.065 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:26:10.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:10.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:26:10.171 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:26:10.208 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:10.233 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:26:10.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:10.240 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:26:10.341 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:26:10.384 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:10.414 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:26:10.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:10.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:26:10.478 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 82 16 160827 160811 122.546 68 0.0134523 0.00815852 2024-04-20T17:26:10.523 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:26:10.558 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:10 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:26:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:10 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:26:10.614 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:26:10.623 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:10 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:26:10.623 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:10 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:26:10.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:10.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:26:10.726 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:26:10.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:10 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:26:10.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:10 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:26:10.782 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:10.808 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:26:10.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:10.816 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:26:10.917 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:26:10.958 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:11.478 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 83 16 161871 161855 121.856 65.25 0.00769813 0.00820462 2024-04-20T17:26:11.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:11 smithi005 ceph-mon[39562]: pgmap v1274: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 937 op/s 2024-04-20T17:26:11.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:11 smithi005 ceph-mon[39562]: osdmap e483: 12 total, 12 up, 12 in 2024-04-20T17:26:11.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:11 smithi005 ceph-mon[39562]: osdmap e484: 12 total, 12 up, 12 in 2024-04-20T17:26:11.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:11 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/814407778' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:26:11.622 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:26:11.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:11.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:26:11.660 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:26:11.660 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-04-20T17:26:11.661 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-04-20T17:26:11.661 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:26:11.661 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.661 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25802.0:76085 3.e e376/356 3:7c846ccd:::benchmark_data_smithi171_110765_object76084:head v 376'4766, mlcod=376'4762)", 2024-04-20T17:26:11.661 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:21:14.852164+0000", 2024-04-20T17:26:11.661 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 296.80432328400002, 2024-04-20T17:26:11.661 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.070277086000000002, 2024-04-20T17:26:11.661 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:26:11.661 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:26:11.661 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:11.661 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:26:11.661 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.661 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:26:11.661 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.852164+0000", 2024-04-20T17:26:11.661 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.662 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.662 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.662 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:26:11.662 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.852164+0000", 2024-04-20T17:26:11.662 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.662 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.662 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.662 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:26:11.662 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.852170+0000", 2024-04-20T17:26:11.662 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8109999999999999e-06 2024-04-20T17:26:11.662 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.662 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.662 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:26:11.662 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.852231+0000", 2024-04-20T17:26:11.663 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0825999999999997e-05 2024-04-20T17:26:11.663 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.663 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.663 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:26:11.663 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.852239+0000", 2024-04-20T17:26:11.663 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2109999999999998e-06 2024-04-20T17:26:11.663 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.663 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.663 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:26:11.663 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.852244+0000", 2024-04-20T17:26:11.663 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5639999999999998e-06 2024-04-20T17:26:11.663 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.663 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.663 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:26:11.663 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.852294+0000", 2024-04-20T17:26:11.664 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0555e-05 2024-04-20T17:26:11.664 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.664 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.664 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:26:11.664 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.852313+0000", 2024-04-20T17:26:11.664 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8260000000000001e-05 2024-04-20T17:26:11.664 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.664 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.664 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:26:11.664 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.922364+0000", 2024-04-20T17:26:11.664 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.070051641999999997 2024-04-20T17:26:11.664 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.664 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.664 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:26:11.664 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.922441+0000", 2024-04-20T17:26:11.665 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7217000000000003e-05 2024-04-20T17:26:11.665 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.665 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:26:11.665 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.665 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.665 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.665 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:76104 3.3 3:cd5093fe:::benchmark_data_smithi171_110765_object76103:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e376)", 2024-04-20T17:26:11.665 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:21:14.856450+0000", 2024-04-20T17:26:11.665 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 296.80003750100002, 2024-04-20T17:26:11.665 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.065203257000000001, 2024-04-20T17:26:11.665 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:26:11.665 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:26:11.665 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:11.665 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:26:11.666 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:26:11.666 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:26:11.666 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 76104 2024-04-20T17:26:11.666 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.666 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:26:11.666 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.666 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:26:11.666 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.856450+0000", 2024-04-20T17:26:11.666 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.666 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.666 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.666 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:26:11.666 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.856450+0000", 2024-04-20T17:26:11.666 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.667 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.667 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.667 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:26:11.667 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.856452+0000", 2024-04-20T17:26:11.667 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9089999999999998e-06 2024-04-20T17:26:11.667 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.667 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.667 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:26:11.667 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.856514+0000", 2024-04-20T17:26:11.667 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1940999999999995e-05 2024-04-20T17:26:11.667 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.667 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.667 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:26:11.667 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.916467+0000", 2024-04-20T17:26:11.667 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.059952892000000001 2024-04-20T17:26:11.668 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.668 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.668 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:26:11.668 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.916479+0000", 2024-04-20T17:26:11.668 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1929e-05 2024-04-20T17:26:11.668 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.668 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.668 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:26:11.668 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.916536+0000", 2024-04-20T17:26:11.668 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7404999999999997e-05 2024-04-20T17:26:11.668 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.668 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.668 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:26:11.668 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.916650+0000", 2024-04-20T17:26:11.668 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000113651 2024-04-20T17:26:11.669 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.669 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.669 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:26:11.669 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.916731+0000", 2024-04-20T17:26:11.669 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1389000000000003e-05 2024-04-20T17:26:11.669 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.669 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.669 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:26:11.669 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.921388+0000", 2024-04-20T17:26:11.669 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0046563059999999998 2024-04-20T17:26:11.669 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.669 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.669 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:11.669 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.921552+0000", 2024-04-20T17:26:11.669 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016445699999999999 2024-04-20T17:26:11.670 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.670 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.670 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:26:11.670 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.921607+0000", 2024-04-20T17:26:11.670 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4771999999999997e-05 2024-04-20T17:26:11.670 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.670 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.670 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:26:11.670 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.921653+0000", 2024-04-20T17:26:11.670 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6606000000000001e-05 2024-04-20T17:26:11.670 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.670 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:26:11.670 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.670 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.670 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.671 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25802.0:76091 3.e e376/356 3:702bd02d:::benchmark_data_smithi171_110765_object76090:head v 376'4767, mlcod=376'4763)", 2024-04-20T17:26:11.671 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:21:14.856508+0000", 2024-04-20T17:26:11.671 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 296.79997925399999, 2024-04-20T17:26:11.671 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.065967812000000001, 2024-04-20T17:26:11.671 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:26:11.671 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:26:11.671 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:11.671 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:26:11.671 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.671 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:26:11.671 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.856508+0000", 2024-04-20T17:26:11.671 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.671 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.671 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.671 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:26:11.672 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.856508+0000", 2024-04-20T17:26:11.672 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.672 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.672 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.672 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:26:11.672 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.856510+0000", 2024-04-20T17:26:11.672 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2669999999999999e-06 2024-04-20T17:26:11.672 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.672 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.672 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:26:11.672 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.856573+0000", 2024-04-20T17:26:11.672 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2977000000000004e-05 2024-04-20T17:26:11.672 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.672 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.672 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:26:11.672 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.917423+0000", 2024-04-20T17:26:11.673 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.060850278000000001 2024-04-20T17:26:11.673 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.673 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.673 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:26:11.673 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.917433+0000", 2024-04-20T17:26:11.673 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0159e-05 2024-04-20T17:26:11.673 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.673 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.673 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:26:11.673 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.918653+0000", 2024-04-20T17:26:11.673 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00121962 2024-04-20T17:26:11.673 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.673 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.673 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:26:11.673 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.918673+0000", 2024-04-20T17:26:11.674 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.003e-05 2024-04-20T17:26:11.674 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.674 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.674 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:26:11.674 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.922445+0000", 2024-04-20T17:26:11.674 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037718320000000001 2024-04-20T17:26:11.674 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.674 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.674 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:26:11.674 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:21:14.922476+0000", 2024-04-20T17:26:11.674 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1649000000000001e-05 2024-04-20T17:26:11.674 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.674 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:26:11.674 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.675 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.675 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.675 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:164759 3.3 3:c1715c23:::benchmark_data_smithi171_110765_object21544:head [delete] snapc 0=[] ondisk+write+known_if_redirected e393)", 2024-04-20T17:26:11.675 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:22:20.361595+0000", 2024-04-20T17:26:11.675 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 231.29489275899999, 2024-04-20T17:26:11.675 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.055921910999999998, 2024-04-20T17:26:11.675 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:26:11.675 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:26:11.675 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:11.675 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:26:11.675 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:26:11.675 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:26:11.675 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 164759 2024-04-20T17:26:11.675 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.676 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:26:11.676 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.676 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:26:11.676 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.361595+0000", 2024-04-20T17:26:11.676 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.676 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.676 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.676 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:26:11.676 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.361595+0000", 2024-04-20T17:26:11.676 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.676 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.676 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.676 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:26:11.676 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.361597+0000", 2024-04-20T17:26:11.676 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.838e-06 2024-04-20T17:26:11.677 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.677 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.677 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:26:11.677 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.361602+0000", 2024-04-20T17:26:11.677 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7960000000000001e-06 2024-04-20T17:26:11.677 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.677 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.677 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:26:11.677 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.361612+0000", 2024-04-20T17:26:11.677 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.132e-06 2024-04-20T17:26:11.677 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.677 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.677 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:26:11.677 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.361617+0000", 2024-04-20T17:26:11.677 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.874e-06 2024-04-20T17:26:11.678 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.678 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.678 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:26:11.678 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.361674+0000", 2024-04-20T17:26:11.678 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6490000000000003e-05 2024-04-20T17:26:11.678 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.678 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.678 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:26:11.678 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.361785+0000", 2024-04-20T17:26:11.678 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000111494 2024-04-20T17:26:11.678 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.678 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.678 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:26:11.678 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.361900+0000", 2024-04-20T17:26:11.678 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011406899999999999 2024-04-20T17:26:11.679 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.679 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.679 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:26:11.679 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.370403+0000", 2024-04-20T17:26:11.679 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0085035909999999996 2024-04-20T17:26:11.679 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.679 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.679 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:11.679 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.417418+0000", 2024-04-20T17:26:11.679 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.047014528999999999 2024-04-20T17:26:11.679 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.679 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.679 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:26:11.679 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.417472+0000", 2024-04-20T17:26:11.679 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4088999999999997e-05 2024-04-20T17:26:11.680 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.680 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.680 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:26:11.680 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.417517+0000", 2024-04-20T17:26:11.680 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5009000000000002e-05 2024-04-20T17:26:11.680 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.680 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:26:11.680 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.680 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.680 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.680 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:164770 3.3 3:c7d5b1bb:::benchmark_data_smithi171_110765_object21555:head [delete] snapc 0=[] ondisk+write+known_if_redirected e393)", 2024-04-20T17:26:11.680 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:22:20.371577+0000", 2024-04-20T17:26:11.680 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 231.284910787, 2024-04-20T17:26:11.680 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.058087044999999997, 2024-04-20T17:26:11.680 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:26:11.681 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:26:11.681 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:11.681 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:26:11.681 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:26:11.681 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:26:11.681 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 164770 2024-04-20T17:26:11.681 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.681 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:26:11.681 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.681 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:26:11.681 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.371577+0000", 2024-04-20T17:26:11.681 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.681 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.681 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.681 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:26:11.682 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.371577+0000", 2024-04-20T17:26:11.682 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.682 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.682 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.682 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:26:11.682 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.371579+0000", 2024-04-20T17:26:11.682 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8789999999999999e-06 2024-04-20T17:26:11.682 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.682 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.682 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:26:11.682 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.371584+0000", 2024-04-20T17:26:11.682 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3809999999999998e-06 2024-04-20T17:26:11.682 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.682 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.683 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:26:11.683 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.371595+0000", 2024-04-20T17:26:11.683 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1100999999999999e-05 2024-04-20T17:26:11.683 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.683 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.683 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:26:11.683 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.371602+0000", 2024-04-20T17:26:11.683 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0600000000000002e-06 2024-04-20T17:26:11.683 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.683 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.683 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:26:11.683 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.373811+0000", 2024-04-20T17:26:11.683 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022089789999999998 2024-04-20T17:26:11.684 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.684 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.684 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:26:11.684 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.373942+0000", 2024-04-20T17:26:11.684 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013118 2024-04-20T17:26:11.684 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.684 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.684 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:26:11.684 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.374068+0000", 2024-04-20T17:26:11.684 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012536099999999999 2024-04-20T17:26:11.684 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.684 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.684 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:26:11.684 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.379040+0000", 2024-04-20T17:26:11.684 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.004971998 2024-04-20T17:26:11.685 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.685 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.685 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:11.685 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.429564+0000", 2024-04-20T17:26:11.685 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.050524544999999997 2024-04-20T17:26:11.685 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.685 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.685 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:26:11.685 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.429618+0000", 2024-04-20T17:26:11.685 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3520999999999998e-05 2024-04-20T17:26:11.685 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.685 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.685 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:26:11.685 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.429664+0000", 2024-04-20T17:26:11.685 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.604e-05 2024-04-20T17:26:11.686 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.686 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:26:11.686 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.686 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.686 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.686 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:164771 3.3 3:c84e3de8:::benchmark_data_smithi171_110765_object21556:head [delete] snapc 0=[] ondisk+write+known_if_redirected e393)", 2024-04-20T17:26:11.686 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:22:20.371771+0000", 2024-04-20T17:26:11.686 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 231.28471662600001, 2024-04-20T17:26:11.686 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.058036722999999998, 2024-04-20T17:26:11.686 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:26:11.686 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:26:11.687 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:11.687 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:26:11.687 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:26:11.687 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:26:11.687 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 164771 2024-04-20T17:26:11.687 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.687 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:26:11.687 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.687 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:26:11.687 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.371771+0000", 2024-04-20T17:26:11.687 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.687 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.687 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.687 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:26:11.688 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.371771+0000", 2024-04-20T17:26:11.688 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.688 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.688 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.688 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:26:11.688 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.371772+0000", 2024-04-20T17:26:11.688 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.071e-06 2024-04-20T17:26:11.688 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.688 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.688 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:26:11.688 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.371776+0000", 2024-04-20T17:26:11.688 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4829999999999999e-06 2024-04-20T17:26:11.688 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.688 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.688 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:26:11.689 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.371783+0000", 2024-04-20T17:26:11.689 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3000000000000004e-06 2024-04-20T17:26:11.689 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.689 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.689 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:26:11.689 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.371788+0000", 2024-04-20T17:26:11.689 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3430000000000003e-06 2024-04-20T17:26:11.689 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.689 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.689 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:26:11.689 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.376884+0000", 2024-04-20T17:26:11.689 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0050953839999999997 2024-04-20T17:26:11.689 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.689 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.689 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:26:11.690 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.377004+0000", 2024-04-20T17:26:11.690 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000120436 2024-04-20T17:26:11.690 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.690 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.690 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:26:11.690 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.377125+0000", 2024-04-20T17:26:11.690 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000120794 2024-04-20T17:26:11.690 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.690 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.690 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:26:11.690 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.379143+0000", 2024-04-20T17:26:11.690 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020177680000000001 2024-04-20T17:26:11.690 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.690 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.691 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:11.691 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.429710+0000", 2024-04-20T17:26:11.691 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.050567237000000001 2024-04-20T17:26:11.691 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.691 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.691 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:26:11.691 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.429763+0000", 2024-04-20T17:26:11.691 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3476000000000001e-05 2024-04-20T17:26:11.691 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.691 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.691 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:26:11.691 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.429808+0000", 2024-04-20T17:26:11.691 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4431000000000002e-05 2024-04-20T17:26:11.691 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.691 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:26:11.692 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.692 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.692 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.692 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:230195 3.3 3:cac27cb4:::benchmark_data_smithi171_110765_object86980:head [delete] snapc 0=[] ondisk+write+known_if_redirected e409)", 2024-04-20T17:26:11.692 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:15.050603+0000", 2024-04-20T17:26:11.692 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 176.60588498999999, 2024-04-20T17:26:11.692 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.070679689000000004, 2024-04-20T17:26:11.692 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:26:11.692 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:26:11.692 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:11.692 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:26:11.692 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:26:11.692 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:26:11.692 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 230195 2024-04-20T17:26:11.692 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.693 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:26:11.693 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.693 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:26:11.693 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.050603+0000", 2024-04-20T17:26:11.693 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.693 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.693 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.693 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:26:11.693 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.050603+0000", 2024-04-20T17:26:11.693 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.693 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.693 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.693 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:26:11.693 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.050604+0000", 2024-04-20T17:26:11.693 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.48e-06 2024-04-20T17:26:11.694 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.694 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.694 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:26:11.694 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.050609+0000", 2024-04-20T17:26:11.694 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7389999999999999e-06 2024-04-20T17:26:11.694 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.694 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.694 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:26:11.694 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.050619+0000", 2024-04-20T17:26:11.694 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7610000000000002e-06 2024-04-20T17:26:11.694 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.694 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.694 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:26:11.694 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.050627+0000", 2024-04-20T17:26:11.694 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3799999999999994e-06 2024-04-20T17:26:11.695 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.695 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.695 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:26:11.695 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.050676+0000", 2024-04-20T17:26:11.695 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9363000000000002e-05 2024-04-20T17:26:11.695 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.695 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.695 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:26:11.695 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.050787+0000", 2024-04-20T17:26:11.695 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011053 2024-04-20T17:26:11.695 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.695 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.695 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:26:11.695 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.050906+0000", 2024-04-20T17:26:11.695 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000118736 2024-04-20T17:26:11.696 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.696 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.696 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:26:11.696 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.054523+0000", 2024-04-20T17:26:11.696 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0036171269999999999 2024-04-20T17:26:11.696 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.696 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.696 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:11.696 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.121187+0000", 2024-04-20T17:26:11.696 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.066664243999999998 2024-04-20T17:26:11.696 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.696 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.696 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:26:11.696 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.121220+0000", 2024-04-20T17:26:11.696 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3359999999999999e-05 2024-04-20T17:26:11.697 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.697 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.697 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:26:11.697 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.121282+0000", 2024-04-20T17:26:11.697 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1969000000000002e-05 2024-04-20T17:26:11.697 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.697 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:26:11.697 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.697 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.697 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.697 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:230197 3.3 3:cb24cc1d:::benchmark_data_smithi171_110765_object86982:head [delete] snapc 0=[] ondisk+write+known_if_redirected e409)", 2024-04-20T17:26:11.697 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:15.054810+0000", 2024-04-20T17:26:11.697 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 176.60167750599999, 2024-04-20T17:26:11.697 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.066635147000000006, 2024-04-20T17:26:11.697 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:26:11.698 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:26:11.698 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:11.698 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:26:11.698 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:26:11.698 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:26:11.698 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 230197 2024-04-20T17:26:11.698 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.698 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:26:11.698 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.698 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:26:11.698 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.054810+0000", 2024-04-20T17:26:11.698 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.698 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.698 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.698 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:26:11.698 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.054810+0000", 2024-04-20T17:26:11.699 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.699 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.699 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.699 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:26:11.699 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.054812+0000", 2024-04-20T17:26:11.699 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8959999999999999e-06 2024-04-20T17:26:11.699 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.699 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.699 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:26:11.699 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.054818+0000", 2024-04-20T17:26:11.699 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9050000000000002e-06 2024-04-20T17:26:11.699 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.699 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.699 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:26:11.699 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.054828+0000", 2024-04-20T17:26:11.700 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0519999999999999e-05 2024-04-20T17:26:11.700 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.700 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.700 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:26:11.700 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.054836+0000", 2024-04-20T17:26:11.700 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6680000000000001e-06 2024-04-20T17:26:11.700 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.700 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.700 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:26:11.700 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.054882+0000", 2024-04-20T17:26:11.700 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6198000000000001e-05 2024-04-20T17:26:11.700 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.700 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.700 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:26:11.701 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.055021+0000", 2024-04-20T17:26:11.701 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013839000000000001 2024-04-20T17:26:11.701 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.701 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.701 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:26:11.701 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.055147+0000", 2024-04-20T17:26:11.701 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012633300000000001 2024-04-20T17:26:11.701 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.701 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.701 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:26:11.701 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.058294+0000", 2024-04-20T17:26:11.701 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00314694 2024-04-20T17:26:11.701 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.701 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.701 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:11.702 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.121349+0000", 2024-04-20T17:26:11.702 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.063054889000000003 2024-04-20T17:26:11.702 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.702 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.702 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:26:11.702 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.121402+0000", 2024-04-20T17:26:11.702 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2719999999999997e-05 2024-04-20T17:26:11.702 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.702 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.702 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:26:11.702 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.121445+0000", 2024-04-20T17:26:11.702 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3687999999999999e-05 2024-04-20T17:26:11.702 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.702 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:26:11.702 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.703 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.703 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.703 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:279875 3.3 3:ca72fdb9:::benchmark_data_smithi171_110765_object136660:head [delete] snapc 0=[] ondisk+write+known_if_redirected e433)", 2024-04-20T17:26:11.703 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:24:05.819548+0000", 2024-04-20T17:26:11.703 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 125.836939523, 2024-04-20T17:26:11.703 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.058138337999999998, 2024-04-20T17:26:11.703 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:26:11.703 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:26:11.703 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:11.703 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:26:11.703 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:26:11.703 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:26:11.703 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 279875 2024-04-20T17:26:11.703 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.704 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:26:11.704 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.704 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:26:11.704 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:24:05.819548+0000", 2024-04-20T17:26:11.704 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.704 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.704 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.704 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:26:11.704 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:24:05.819548+0000", 2024-04-20T17:26:11.704 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.704 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.704 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.704 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:26:11.704 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:24:05.819549+0000", 2024-04-20T17:26:11.704 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7000000000000003e-07 2024-04-20T17:26:11.705 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.705 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.705 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:26:11.705 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:24:05.819552+0000", 2024-04-20T17:26:11.705 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9639999999999999e-06 2024-04-20T17:26:11.705 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.705 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.705 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:26:11.705 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:24:05.819556+0000", 2024-04-20T17:26:11.705 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.065e-06 2024-04-20T17:26:11.705 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.705 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.705 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:26:11.705 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:24:05.819560+0000", 2024-04-20T17:26:11.706 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7330000000000001e-06 2024-04-20T17:26:11.706 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.706 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.706 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:26:11.706 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:24:05.823294+0000", 2024-04-20T17:26:11.706 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037338359999999999 2024-04-20T17:26:11.706 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.706 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.706 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:26:11.706 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:24:05.823423+0000", 2024-04-20T17:26:11.706 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012909400000000001 2024-04-20T17:26:11.706 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.706 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.706 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:26:11.706 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:24:05.823547+0000", 2024-04-20T17:26:11.707 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000124051 2024-04-20T17:26:11.707 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.707 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.707 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:26:11.707 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:24:05.841165+0000", 2024-04-20T17:26:11.707 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.017618156999999999 2024-04-20T17:26:11.707 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.707 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.707 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:11.707 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:24:05.877588+0000", 2024-04-20T17:26:11.707 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.036423171999999997 2024-04-20T17:26:11.707 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.707 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.707 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:26:11.708 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:24:05.877639+0000", 2024-04-20T17:26:11.708 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0992999999999998e-05 2024-04-20T17:26:11.708 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.708 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.708 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:26:11.708 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:24:05.877686+0000", 2024-04-20T17:26:11.708 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7403e-05 2024-04-20T17:26:11.708 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.708 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:26:11.708 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.708 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.708 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.708 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26074.0:57710 4.c 4:3b7a6ad0:::benchmark_data_smithi171_122772_object57709:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e457)", 2024-04-20T17:26:11.708 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:25:04.576003+0000", 2024-04-20T17:26:11.709 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 67.080484620999997, 2024-04-20T17:26:11.709 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.068512081000000002, 2024-04-20T17:26:11.709 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:26:11.709 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:26:11.709 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:11.709 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:26:11.709 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26074", 2024-04-20T17:26:11.709 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:26:11.709 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 57710 2024-04-20T17:26:11.709 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.709 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:26:11.709 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.709 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:26:11.709 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.576003+0000", 2024-04-20T17:26:11.709 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.710 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.710 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.710 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:26:11.710 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.576003+0000", 2024-04-20T17:26:11.710 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.710 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.710 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.710 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:26:11.710 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.576005+0000", 2024-04-20T17:26:11.710 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.249e-06 2024-04-20T17:26:11.710 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.710 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.711 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:26:11.711 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.576054+0000", 2024-04-20T17:26:11.711 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8282000000000003e-05 2024-04-20T17:26:11.711 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.711 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.711 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:26:11.711 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.576066+0000", 2024-04-20T17:26:11.711 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2549e-05 2024-04-20T17:26:11.711 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.711 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.711 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:26:11.711 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.576073+0000", 2024-04-20T17:26:11.711 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1409999999999999e-06 2024-04-20T17:26:11.711 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.712 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.712 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:26:11.712 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.576131+0000", 2024-04-20T17:26:11.712 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7941e-05 2024-04-20T17:26:11.712 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.712 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.712 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:26:11.712 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.576299+0000", 2024-04-20T17:26:11.712 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016742499999999999 2024-04-20T17:26:11.712 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.712 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.712 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:26:11.712 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.576454+0000", 2024-04-20T17:26:11.712 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000155437 2024-04-20T17:26:11.712 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.712 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.713 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:26:11.713 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.577978+0000", 2024-04-20T17:26:11.713 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001523592 2024-04-20T17:26:11.713 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.713 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.713 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:11.713 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.644404+0000", 2024-04-20T17:26:11.713 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.066426764999999999 2024-04-20T17:26:11.713 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.713 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.713 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:26:11.713 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.644460+0000", 2024-04-20T17:26:11.713 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6072000000000001e-05 2024-04-20T17:26:11.713 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.714 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.714 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:26:11.714 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.644515+0000", 2024-04-20T17:26:11.714 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4627999999999999e-05 2024-04-20T17:26:11.714 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.714 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:26:11.714 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.714 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.714 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.714 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26074.0:57713 4.c 4:38f8b18d:::benchmark_data_smithi171_122772_object57712:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e457)", 2024-04-20T17:26:11.714 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:25:04.578710+0000", 2024-04-20T17:26:11.714 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 67.077777753000007, 2024-04-20T17:26:11.714 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.065931321000000001, 2024-04-20T17:26:11.714 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:26:11.715 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:26:11.715 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:11.715 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:26:11.715 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26074", 2024-04-20T17:26:11.715 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:26:11.715 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 57713 2024-04-20T17:26:11.715 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.715 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:26:11.715 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.715 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:26:11.715 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578710+0000", 2024-04-20T17:26:11.715 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.715 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.715 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.716 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:26:11.716 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578710+0000", 2024-04-20T17:26:11.716 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.716 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.716 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.716 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:26:11.716 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578712+0000", 2024-04-20T17:26:11.716 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6700000000000001e-06 2024-04-20T17:26:11.716 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.716 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.716 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:26:11.716 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578769+0000", 2024-04-20T17:26:11.716 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7652999999999998e-05 2024-04-20T17:26:11.716 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.716 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.717 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:26:11.717 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578779+0000", 2024-04-20T17:26:11.717 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0049e-05 2024-04-20T17:26:11.717 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.717 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.717 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:26:11.717 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578785+0000", 2024-04-20T17:26:11.717 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.852e-06 2024-04-20T17:26:11.717 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.717 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.717 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:26:11.717 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578830+0000", 2024-04-20T17:26:11.717 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5249e-05 2024-04-20T17:26:11.717 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.717 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.717 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:26:11.718 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578927+0000", 2024-04-20T17:26:11.718 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6475e-05 2024-04-20T17:26:11.718 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.718 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.718 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:26:11.718 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.579034+0000", 2024-04-20T17:26:11.718 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010687399999999999 2024-04-20T17:26:11.718 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.718 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.718 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:26:11.718 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.581249+0000", 2024-04-20T17:26:11.718 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022152859999999999 2024-04-20T17:26:11.718 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.718 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:11.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.644557+0000", 2024-04-20T17:26:11.719 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.063307873000000001 2024-04-20T17:26:11.719 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.719 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:26:11.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.644598+0000", 2024-04-20T17:26:11.719 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1362999999999997e-05 2024-04-20T17:26:11.719 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.719 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:26:11.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.644641+0000", 2024-04-20T17:26:11.719 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2976999999999999e-05 2024-04-20T17:26:11.719 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.719 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:26:11.719 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.720 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.720 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.720 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26074.0:57714 4.c 4:386280b9:::benchmark_data_smithi171_122772_object57713:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e457)", 2024-04-20T17:26:11.720 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:25:04.578796+0000", 2024-04-20T17:26:11.720 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 67.077691822999995, 2024-04-20T17:26:11.720 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.066014529000000002, 2024-04-20T17:26:11.720 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:26:11.720 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:26:11.720 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:11.720 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:26:11.720 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26074", 2024-04-20T17:26:11.720 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:26:11.720 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 57714 2024-04-20T17:26:11.720 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.721 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:26:11.721 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.721 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:26:11.721 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578796+0000", 2024-04-20T17:26:11.721 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.721 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.721 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.721 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:26:11.721 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578796+0000", 2024-04-20T17:26:11.721 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.721 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.721 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.721 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:26:11.721 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578797+0000", 2024-04-20T17:26:11.722 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4399999999999999e-07 2024-04-20T17:26:11.722 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.722 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.722 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:26:11.722 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578820+0000", 2024-04-20T17:26:11.722 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3425999999999999e-05 2024-04-20T17:26:11.722 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.722 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.722 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:26:11.722 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578824+0000", 2024-04-20T17:26:11.722 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5229999999999997e-06 2024-04-20T17:26:11.722 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.722 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.722 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:26:11.722 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578830+0000", 2024-04-20T17:26:11.723 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2580000000000003e-06 2024-04-20T17:26:11.723 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.723 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.723 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:26:11.723 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.580253+0000", 2024-04-20T17:26:11.723 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0014233679999999999 2024-04-20T17:26:11.723 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.723 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.723 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:26:11.723 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.580361+0000", 2024-04-20T17:26:11.723 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010793699999999999 2024-04-20T17:26:11.723 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.723 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.723 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:26:11.723 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.580465+0000", 2024-04-20T17:26:11.724 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000104218 2024-04-20T17:26:11.724 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.724 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.724 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:26:11.724 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.581337+0000", 2024-04-20T17:26:11.724 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00087222799999999996 2024-04-20T17:26:11.724 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.724 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.724 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:11.724 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.644712+0000", 2024-04-20T17:26:11.724 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.063374377999999995 2024-04-20T17:26:11.724 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.724 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.724 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:26:11.724 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.644766+0000", 2024-04-20T17:26:11.724 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4304e-05 2024-04-20T17:26:11.725 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.725 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.725 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:26:11.725 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.644810+0000", 2024-04-20T17:26:11.725 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4144999999999998e-05 2024-04-20T17:26:11.725 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.725 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:26:11.725 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.725 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.725 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.725 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26074.0:67472 4.e 4:717ad074:::benchmark_data_smithi171_122772_object67471:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e458)", 2024-04-20T17:26:11.725 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:25:08.545538+0000", 2024-04-20T17:26:11.725 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 63.110950027999998, 2024-04-20T17:26:11.725 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.066188420999999997, 2024-04-20T17:26:11.725 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:26:11.725 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:26:11.726 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:11.726 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:26:11.726 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26074", 2024-04-20T17:26:11.726 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:26:11.726 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67472 2024-04-20T17:26:11.726 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.726 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:26:11.726 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.726 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:26:11.726 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.545538+0000", 2024-04-20T17:26:11.726 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.726 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.726 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.726 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:26:11.727 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.545538+0000", 2024-04-20T17:26:11.727 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.727 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.727 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.727 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:26:11.727 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.545538+0000", 2024-04-20T17:26:11.727 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1299999999999998e-07 2024-04-20T17:26:11.727 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.727 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.727 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:26:11.727 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.545721+0000", 2024-04-20T17:26:11.727 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000182504 2024-04-20T17:26:11.727 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.727 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.727 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:26:11.728 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.545728+0000", 2024-04-20T17:26:11.728 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0489999999999998e-06 2024-04-20T17:26:11.728 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.728 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.728 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:26:11.728 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.545734+0000", 2024-04-20T17:26:11.728 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6550000000000001e-06 2024-04-20T17:26:11.728 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.728 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.728 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:26:11.728 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.548549+0000", 2024-04-20T17:26:11.728 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0028149730000000001 2024-04-20T17:26:11.728 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.728 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.728 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:26:11.729 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.548628+0000", 2024-04-20T17:26:11.729 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9009000000000002e-05 2024-04-20T17:26:11.729 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.729 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.729 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:26:11.729 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.548705+0000", 2024-04-20T17:26:11.729 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7286999999999994e-05 2024-04-20T17:26:11.729 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.729 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.729 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:26:11.729 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.551339+0000", 2024-04-20T17:26:11.729 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0026335859999999998 2024-04-20T17:26:11.729 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.729 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.729 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:11.730 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611636+0000", 2024-04-20T17:26:11.730 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.060297852999999998 2024-04-20T17:26:11.730 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.730 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.730 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:26:11.730 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611696+0000", 2024-04-20T17:26:11.730 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9827000000000002e-05 2024-04-20T17:26:11.730 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.730 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.730 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:26:11.730 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611726+0000", 2024-04-20T17:26:11.730 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9765e-05 2024-04-20T17:26:11.730 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.730 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:26:11.730 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.731 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.731 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.731 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26074.0:67478 4.e 4:74b2a46e:::benchmark_data_smithi171_122772_object67477:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e458)", 2024-04-20T17:26:11.731 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:25:08.548203+0000", 2024-04-20T17:26:11.731 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 63.108284961999999, 2024-04-20T17:26:11.731 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.063669150999999993, 2024-04-20T17:26:11.731 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:26:11.731 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:26:11.731 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:11.731 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:26:11.731 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26074", 2024-04-20T17:26:11.731 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:26:11.731 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67478 2024-04-20T17:26:11.731 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.731 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:26:11.732 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.732 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:26:11.732 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.548203+0000", 2024-04-20T17:26:11.732 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.732 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.732 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.732 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:26:11.732 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.548203+0000", 2024-04-20T17:26:11.732 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.732 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.732 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.732 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:26:11.732 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.548204+0000", 2024-04-20T17:26:11.732 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.502e-06 2024-04-20T17:26:11.732 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.733 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.733 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:26:11.733 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.548262+0000", 2024-04-20T17:26:11.733 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8094000000000003e-05 2024-04-20T17:26:11.733 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.733 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.733 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:26:11.733 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.548271+0000", 2024-04-20T17:26:11.733 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.303e-06 2024-04-20T17:26:11.733 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.733 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.733 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:26:11.733 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.548276+0000", 2024-04-20T17:26:11.733 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7309999999999998e-06 2024-04-20T17:26:11.733 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.734 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.734 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:26:11.734 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.549768+0000", 2024-04-20T17:26:11.734 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0014920210000000001 2024-04-20T17:26:11.734 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.734 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.734 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:26:11.734 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.549870+0000", 2024-04-20T17:26:11.734 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000102002 2024-04-20T17:26:11.734 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.734 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.734 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:26:11.734 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.549998+0000", 2024-04-20T17:26:11.734 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000128007 2024-04-20T17:26:11.734 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.735 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.735 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:26:11.735 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.551461+0000", 2024-04-20T17:26:11.735 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001463015 2024-04-20T17:26:11.735 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.735 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.735 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:11.735 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611818+0000", 2024-04-20T17:26:11.735 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.060356685 2024-04-20T17:26:11.735 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.735 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.735 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:26:11.735 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611851+0000", 2024-04-20T17:26:11.735 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3114000000000002e-05 2024-04-20T17:26:11.735 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.736 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.736 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:26:11.736 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611872+0000", 2024-04-20T17:26:11.736 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0676999999999999e-05 2024-04-20T17:26:11.736 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.736 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:26:11.736 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.736 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.736 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.736 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26074.0:67486 4.e 4:7a975fb4:::benchmark_data_smithi171_122772_object67485:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e458)", 2024-04-20T17:26:11.736 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:25:08.551436+0000", 2024-04-20T17:26:11.736 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 63.105051969000002, 2024-04-20T17:26:11.736 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.060534031000000002, 2024-04-20T17:26:11.736 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:26:11.736 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:26:11.736 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:11.737 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:26:11.737 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26074", 2024-04-20T17:26:11.737 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:26:11.737 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67486 2024-04-20T17:26:11.737 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.737 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:26:11.737 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.737 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:26:11.737 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.551436+0000", 2024-04-20T17:26:11.737 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.737 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.737 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.737 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:26:11.737 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.551436+0000", 2024-04-20T17:26:11.737 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.738 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.738 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.738 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:26:11.738 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.551437+0000", 2024-04-20T17:26:11.738 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4699999999999999e-06 2024-04-20T17:26:11.738 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.738 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.738 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:26:11.738 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.551506+0000", 2024-04-20T17:26:11.738 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9208000000000006e-05 2024-04-20T17:26:11.738 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.738 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.738 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:26:11.738 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.551516+0000", 2024-04-20T17:26:11.738 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0173999999999999e-05 2024-04-20T17:26:11.739 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.739 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.739 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:26:11.739 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.551524+0000", 2024-04-20T17:26:11.739 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3159999999999999e-06 2024-04-20T17:26:11.739 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.739 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.739 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:26:11.739 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.551564+0000", 2024-04-20T17:26:11.739 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0593999999999998e-05 2024-04-20T17:26:11.739 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.739 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.739 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:26:11.739 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.551655+0000", 2024-04-20T17:26:11.739 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0486000000000001e-05 2024-04-20T17:26:11.740 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.740 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.740 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:26:11.740 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.551735+0000", 2024-04-20T17:26:11.740 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0178000000000004e-05 2024-04-20T17:26:11.740 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.740 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.740 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:26:11.740 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553066+0000", 2024-04-20T17:26:11.740 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001330721 2024-04-20T17:26:11.740 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.740 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.740 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:11.740 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611896+0000", 2024-04-20T17:26:11.740 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.058829748000000001 2024-04-20T17:26:11.741 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.741 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.741 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:26:11.741 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611945+0000", 2024-04-20T17:26:11.741 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9048000000000002e-05 2024-04-20T17:26:11.741 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.741 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.741 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:26:11.741 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611970+0000", 2024-04-20T17:26:11.741 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5088000000000001e-05 2024-04-20T17:26:11.741 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.741 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:26:11.741 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.741 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.742 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.742 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26074.0:67489 4.e 4:7f789828:::benchmark_data_smithi171_122772_object67488:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e458)", 2024-04-20T17:26:11.742 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:25:08.553188+0000", 2024-04-20T17:26:11.742 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 63.103299294999999, 2024-04-20T17:26:11.742 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.058877112000000002, 2024-04-20T17:26:11.742 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:26:11.742 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:26:11.742 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:11.742 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:26:11.742 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26074", 2024-04-20T17:26:11.742 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:26:11.742 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67489 2024-04-20T17:26:11.742 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.742 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:26:11.742 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.743 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:26:11.743 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553188+0000", 2024-04-20T17:26:11.743 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.743 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.743 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.743 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:26:11.743 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553188+0000", 2024-04-20T17:26:11.743 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.743 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.743 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.743 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:26:11.743 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553190+0000", 2024-04-20T17:26:11.743 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.578e-06 2024-04-20T17:26:11.743 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.743 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.744 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:26:11.744 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553282+0000", 2024-04-20T17:26:11.744 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2596999999999995e-05 2024-04-20T17:26:11.744 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.744 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.744 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:26:11.744 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553293+0000", 2024-04-20T17:26:11.744 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0372e-05 2024-04-20T17:26:11.744 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.744 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.744 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:26:11.744 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553306+0000", 2024-04-20T17:26:11.744 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2894000000000001e-05 2024-04-20T17:26:11.744 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.744 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.745 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:26:11.745 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553349+0000", 2024-04-20T17:26:11.745 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3517999999999999e-05 2024-04-20T17:26:11.745 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.745 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.745 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:26:11.745 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553442+0000", 2024-04-20T17:26:11.745 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2981999999999998e-05 2024-04-20T17:26:11.745 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.745 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.745 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:26:11.745 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553542+0000", 2024-04-20T17:26:11.745 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9889000000000005e-05 2024-04-20T17:26:11.745 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.745 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.746 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:26:11.746 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611740+0000", 2024-04-20T17:26:11.746 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.058197394 2024-04-20T17:26:11.746 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.746 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.746 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:11.746 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.612014+0000", 2024-04-20T17:26:11.746 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00027397700000000001 2024-04-20T17:26:11.746 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.746 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.746 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:26:11.746 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.612040+0000", 2024-04-20T17:26:11.746 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6957e-05 2024-04-20T17:26:11.746 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.747 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.747 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:26:11.747 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.612065+0000", 2024-04-20T17:26:11.747 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4953999999999999e-05 2024-04-20T17:26:11.747 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.747 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:26:11.747 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.747 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.747 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.747 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26074.0:67492 4.c 4:3e44fca8:::benchmark_data_smithi171_122772_object67491:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e458)", 2024-04-20T17:26:11.747 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:25:08.553598+0000", 2024-04-20T17:26:11.747 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 63.102889632, 2024-04-20T17:26:11.747 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.058295435999999999, 2024-04-20T17:26:11.747 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:26:11.748 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:26:11.748 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:11.748 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:26:11.748 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26074", 2024-04-20T17:26:11.748 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:26:11.748 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67492 2024-04-20T17:26:11.748 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.748 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:26:11.748 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.748 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:26:11.748 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553598+0000", 2024-04-20T17:26:11.748 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.748 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.748 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.748 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:26:11.748 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553598+0000", 2024-04-20T17:26:11.749 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.749 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.749 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.749 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:26:11.749 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553600+0000", 2024-04-20T17:26:11.749 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5349999999999999e-06 2024-04-20T17:26:11.749 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.749 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.749 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:26:11.749 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553655+0000", 2024-04-20T17:26:11.749 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5262e-05 2024-04-20T17:26:11.749 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.749 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.749 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:26:11.749 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553664+0000", 2024-04-20T17:26:11.749 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7630000000000006e-06 2024-04-20T17:26:11.750 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.750 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.750 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:26:11.750 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553670+0000", 2024-04-20T17:26:11.750 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.934e-06 2024-04-20T17:26:11.750 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.750 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.750 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:26:11.750 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553714+0000", 2024-04-20T17:26:11.750 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3523999999999999e-05 2024-04-20T17:26:11.750 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.750 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.750 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:26:11.750 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553787+0000", 2024-04-20T17:26:11.751 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2996000000000004e-05 2024-04-20T17:26:11.751 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.751 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.751 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:26:11.751 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553880+0000", 2024-04-20T17:26:11.751 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3465e-05 2024-04-20T17:26:11.751 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.751 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.751 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:26:11.751 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611756+0000", 2024-04-20T17:26:11.751 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.057875886000000001 2024-04-20T17:26:11.751 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.751 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.751 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:11.752 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611802+0000", 2024-04-20T17:26:11.752 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5352999999999998e-05 2024-04-20T17:26:11.752 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.752 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.752 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:26:11.752 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611853+0000", 2024-04-20T17:26:11.752 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1350000000000001e-05 2024-04-20T17:26:11.752 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.752 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.752 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:26:11.752 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611893+0000", 2024-04-20T17:26:11.752 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0367999999999997e-05 2024-04-20T17:26:11.752 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.752 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:26:11.753 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.753 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.753 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.753 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26074.0:67493 4.e 4:7c90e581:::benchmark_data_smithi171_122772_object67492:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e458)", 2024-04-20T17:26:11.753 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:25:08.553815+0000", 2024-04-20T17:26:11.753 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 63.102673062000001, 2024-04-20T17:26:11.753 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.058399287000000001, 2024-04-20T17:26:11.753 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:26:11.753 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:26:11.753 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:11.753 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:26:11.753 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26074", 2024-04-20T17:26:11.753 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:26:11.753 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67493 2024-04-20T17:26:11.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.754 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:26:11.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.754 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:26:11.754 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553815+0000", 2024-04-20T17:26:11.754 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.754 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:26:11.754 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553815+0000", 2024-04-20T17:26:11.754 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.754 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:26:11.754 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553816+0000", 2024-04-20T17:26:11.755 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1829999999999999e-06 2024-04-20T17:26:11.755 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.755 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.755 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:26:11.755 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553890+0000", 2024-04-20T17:26:11.755 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4521e-05 2024-04-20T17:26:11.755 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.755 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.755 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:26:11.755 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553897+0000", 2024-04-20T17:26:11.755 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0770000000000002e-06 2024-04-20T17:26:11.755 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.755 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.755 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:26:11.755 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553903+0000", 2024-04-20T17:26:11.756 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.784e-06 2024-04-20T17:26:11.756 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.756 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.756 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:26:11.756 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.555067+0000", 2024-04-20T17:26:11.756 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001163772 2024-04-20T17:26:11.756 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.756 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.756 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:26:11.756 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.555152+0000", 2024-04-20T17:26:11.756 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5455000000000001e-05 2024-04-20T17:26:11.756 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.756 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.756 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:26:11.756 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.555229+0000", 2024-04-20T17:26:11.757 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7152999999999998e-05 2024-04-20T17:26:11.757 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.757 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.757 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:26:11.757 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611786+0000", 2024-04-20T17:26:11.757 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.056556228 2024-04-20T17:26:11.757 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.757 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.757 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:11.757 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.612111+0000", 2024-04-20T17:26:11.757 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00032532900000000002 2024-04-20T17:26:11.757 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.757 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.757 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:26:11.757 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.612193+0000", 2024-04-20T17:26:11.758 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2028999999999997e-05 2024-04-20T17:26:11.758 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.758 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.758 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:26:11.758 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.612214+0000", 2024-04-20T17:26:11.758 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0755999999999999e-05 2024-04-20T17:26:11.758 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.758 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:26:11.758 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.758 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.758 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.758 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26074.0:67494 4.c 4:305ad90d:::benchmark_data_smithi171_122772_object67493:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e458)", 2024-04-20T17:26:11.758 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:25:08.553916+0000", 2024-04-20T17:26:11.758 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 63.102571902000001, 2024-04-20T17:26:11.759 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.058192206000000003, 2024-04-20T17:26:11.759 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:26:11.759 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:26:11.759 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:11.759 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:26:11.759 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26074", 2024-04-20T17:26:11.759 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:26:11.759 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67494 2024-04-20T17:26:11.759 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.759 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:26:11.759 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.759 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:26:11.759 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553916+0000", 2024-04-20T17:26:11.759 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.759 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.760 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.760 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:26:11.760 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553916+0000", 2024-04-20T17:26:11.760 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.760 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.760 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.760 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:26:11.760 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553916+0000", 2024-04-20T17:26:11.760 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3200000000000004e-07 2024-04-20T17:26:11.760 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.760 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.760 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:26:11.760 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.554003+0000", 2024-04-20T17:26:11.760 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6440999999999995e-05 2024-04-20T17:26:11.760 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.760 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:26:11.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.554016+0000", 2024-04-20T17:26:11.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3345e-05 2024-04-20T17:26:11.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:26:11.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.554022+0000", 2024-04-20T17:26:11.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1580000000000002e-06 2024-04-20T17:26:11.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:26:11.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.555541+0000", 2024-04-20T17:26:11.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0015190970000000001 2024-04-20T17:26:11.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:26:11.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.555658+0000", 2024-04-20T17:26:11.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000116316 2024-04-20T17:26:11.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:26:11.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.555776+0000", 2024-04-20T17:26:11.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000117861 2024-04-20T17:26:11.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:26:11.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611763+0000", 2024-04-20T17:26:11.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.055987396000000002 2024-04-20T17:26:11.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:11.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611967+0000", 2024-04-20T17:26:11.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00020356899999999999 2024-04-20T17:26:11.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.763 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.763 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:26:11.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.612034+0000", 2024-04-20T17:26:11.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7879000000000005e-05 2024-04-20T17:26:11.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.763 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.763 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:26:11.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.612108+0000", 2024-04-20T17:26:11.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3411999999999994e-05 2024-04-20T17:26:11.763 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.763 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:26:11.763 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.764 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.764 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.764 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26074.0:67498 4.d 4:b63a32dc:::benchmark_data_smithi171_122772_object67497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e458)", 2024-04-20T17:26:11.764 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:25:08.557009+0000", 2024-04-20T17:26:11.764 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 63.099478515999998, 2024-04-20T17:26:11.764 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.059470428999999998, 2024-04-20T17:26:11.764 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:26:11.764 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:26:11.764 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:11.764 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:26:11.764 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26074", 2024-04-20T17:26:11.764 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:26:11.764 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67498 2024-04-20T17:26:11.764 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.765 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:26:11.765 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.765 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:26:11.765 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.557009+0000", 2024-04-20T17:26:11.765 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.765 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.765 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.765 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:26:11.765 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.557009+0000", 2024-04-20T17:26:11.765 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:11.765 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.765 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.765 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:26:11.765 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.557010+0000", 2024-04-20T17:26:11.765 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3459999999999999e-06 2024-04-20T17:26:11.765 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.766 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.766 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:26:11.766 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.557120+0000", 2024-04-20T17:26:11.766 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010925999999999999 2024-04-20T17:26:11.766 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.766 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.766 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:26:11.766 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611595+0000", 2024-04-20T17:26:11.766 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.054474886 2024-04-20T17:26:11.766 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.766 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.766 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:26:11.766 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611608+0000", 2024-04-20T17:26:11.766 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3089e-05 2024-04-20T17:26:11.767 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.767 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.767 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:26:11.767 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611663+0000", 2024-04-20T17:26:11.767 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5305e-05 2024-04-20T17:26:11.767 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.767 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.767 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:26:11.767 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611814+0000", 2024-04-20T17:26:11.767 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015124600000000001 2024-04-20T17:26:11.767 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.767 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.767 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 9", 2024-04-20T17:26:11.767 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611994+0000", 2024-04-20T17:26:11.767 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00018020699999999999 2024-04-20T17:26:11.767 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.768 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.768 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:26:11.768 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.616331+0000", 2024-04-20T17:26:11.768 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.004336749 2024-04-20T17:26:11.768 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.768 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.768 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:11.768 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.616375+0000", 2024-04-20T17:26:11.768 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3829000000000003e-05 2024-04-20T17:26:11.768 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.768 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.768 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:26:11.768 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.616432+0000", 2024-04-20T17:26:11.768 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7243000000000001e-05 2024-04-20T17:26:11.768 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:11.769 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:11.769 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:26:11.769 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.616480+0000", 2024-04-20T17:26:11.769 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7268999999999998e-05 2024-04-20T17:26:11.769 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.769 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:26:11.769 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.769 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:11.769 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:26:11.769 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:26:11.769 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:26:11.770 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:11 smithi106 ceph-mon[44423]: pgmap v1274: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 937 op/s 2024-04-20T17:26:11.771 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:11 smithi106 ceph-mon[44423]: osdmap e483: 12 total, 12 up, 12 in 2024-04-20T17:26:11.771 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:11 smithi106 ceph-mon[44423]: osdmap e484: 12 total, 12 up, 12 in 2024-04-20T17:26:11.771 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:11 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/814407778' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:26:11.811 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:11.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:11 smithi171 ceph-mon[44189]: pgmap v1274: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 937 op/s 2024-04-20T17:26:11.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:11 smithi171 ceph-mon[44189]: osdmap e483: 12 total, 12 up, 12 in 2024-04-20T17:26:11.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:11 smithi171 ceph-mon[44189]: osdmap e484: 12 total, 12 up, 12 in 2024-04-20T17:26:11.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:11 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/814407778' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:26:11.879 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:26:11.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:11.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:26:11.983 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:26:12.022 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:12.046 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:26:12.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:12.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:26:12.150 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:26:12.158 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:26:12.187 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:12.423 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:12 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:26:12.423 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:12 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:26:12.423 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:12 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:26:12.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:12 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:26:12.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:12 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:26:12.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:12 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:26:12.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:12 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/814407778' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:26:12.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:12 smithi005 ceph-mon[39562]: osdmap e485: 12 total, 12 up, 12 in 2024-04-20T17:26:12.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:12 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3509483356' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:26:12.478 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 84 16 162850 162834 121.134 61.1875 0.00523157 0.00825379 2024-04-20T17:26:12.756 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:12 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/814407778' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:26:12.756 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:12 smithi106 ceph-mon[44423]: osdmap e485: 12 total, 12 up, 12 in 2024-04-20T17:26:12.756 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:12 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3509483356' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:26:12.796 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-20T17:26:12.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:12 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:26:12.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:12 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:26:12.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:12 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:26:12.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:12 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/814407778' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:26:12.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:12 smithi171 ceph-mon[44189]: osdmap e485: 12 total, 12 up, 12 in 2024-04-20T17:26:12.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:12 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3509483356' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:26:12.847 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:26:12.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:12.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:26:12.952 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:26:13.015 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:13.050 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:26:13.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:13.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:26:13.152 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:26:13.178 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.7 to 1 (10000) 2024-04-20T17:26:13.197 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:26:13.216 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:13.380 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:26:13.478 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 85 16 164096 164080 120.625 77.875 0.0172517 0.00828893 2024-04-20T17:26:13.490 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:26:13.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:13.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:26:13.594 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:26:13.627 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:13.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:13 smithi005 ceph-mon[39562]: pgmap v1278: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 990 op/s 2024-04-20T17:26:13.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:13 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3509483356' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-20T17:26:13.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:13 smithi005 ceph-mon[39562]: osdmap e486: 12 total, 12 up, 12 in 2024-04-20T17:26:13.734 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:13 smithi106 ceph-mon[44423]: pgmap v1278: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 990 op/s 2024-04-20T17:26:13.734 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:13 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3509483356' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-20T17:26:13.734 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:13 smithi106 ceph-mon[44423]: osdmap e486: 12 total, 12 up, 12 in 2024-04-20T17:26:13.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:13 smithi171 ceph-mon[44189]: pgmap v1278: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 990 op/s 2024-04-20T17:26:13.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:13 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3509483356' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-20T17:26:13.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:13 smithi171 ceph-mon[44189]: osdmap e486: 12 total, 12 up, 12 in 2024-04-20T17:26:13.983 INFO:tasks.thrashosds.thrasher:test_map_discontinuity 2024-04-20T17:26:13.984 INFO:tasks.thrashosds.thrasher:Waiting for recovery 2024-04-20T17:26:13.984 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-20T17:26:13.984 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-20T17:26:14.056 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:26:14.144 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:26:14.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:14.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:26:14.250 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:26:14.294 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:14.321 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:26:14.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:14.324 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:26:14.425 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:26:14.465 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:14.478 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 86 16 165298 165282 120.096 75.125 0.0100297 0.00832571 2024-04-20T17:26:14.636 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:26:14.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:14.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:26:14.745 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:26:14.782 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:14.813 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:26:14.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:14.821 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:26:14.922 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:26:14.956 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:14.993 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:26:14.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:14.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:26:15.032 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:26:15.099 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:26:15.159 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:15.358 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:26:15.358 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:26:15.358 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-04-20T17:26:15.358 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:26:15.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:15 smithi005 ceph-mon[39562]: pgmap v1280: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-04-20T17:26:15.478 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 87 16 166485 166469 119.568 74.1875 0.015752 0.00836187 2024-04-20T17:26:15.530 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:26:15.543 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:26:15.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:15.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:26:15.609 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:26:15.649 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:26:15.701 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:15.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:15 smithi106 ceph-mon[44423]: pgmap v1280: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-04-20T17:26:15.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:15 smithi171 ceph-mon[44189]: pgmap v1280: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 9.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-04-20T17:26:16.124 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-20T17:26:16.478 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 88 16 167712 167696 119.081 76.6875 0.0127336 0.00839634 2024-04-20T17:26:16.519 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:26:17.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:17 smithi005 ceph-mon[39562]: pgmap v1281: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-20T17:26:17.478 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 89 16 168986 168970 118.637 79.625 0.0101181 0.00842724 2024-04-20T17:26:17.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:17 smithi106 ceph-mon[44423]: pgmap v1281: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-20T17:26:17.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:17 smithi171 ceph-mon[44189]: pgmap v1281: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-20T17:26:18.055 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:26:18.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:18.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:26:18.169 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:26:18.209 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:18.478 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Total time run: 90.009 2024-04-20T17:26:18.479 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Total writes made: 170189 2024-04-20T17:26:18.479 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Write size: 65536 2024-04-20T17:26:18.479 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Object size: 65536 2024-04-20T17:26:18.479 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Bandwidth (MB/sec): 118.175 2024-04-20T17:26:18.479 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Stddev Bandwidth: 61.2185 2024-04-20T17:26:18.479 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Max bandwidth (MB/sec): 382.875 2024-04-20T17:26:18.479 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Min bandwidth (MB/sec): 58 2024-04-20T17:26:18.479 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Average IOPS: 1890 2024-04-20T17:26:18.479 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Stddev IOPS: 979.496 2024-04-20T17:26:18.479 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Max IOPS: 6126 2024-04-20T17:26:18.479 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Min IOPS: 928 2024-04-20T17:26:18.479 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Average Latency(s): 0.00846139 2024-04-20T17:26:18.479 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Stddev Latency(s): 0.00840963 2024-04-20T17:26:18.480 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Max latency(s): 0.228747 2024-04-20T17:26:18.480 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Min latency(s): 0.00113144 2024-04-20T17:26:18.486 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Cleaning up (deleting benchmark objects) 2024-04-20T17:26:19.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:19 smithi005 ceph-mon[39562]: pgmap v1282: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s 2024-04-20T17:26:19.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:19 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2402002617' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:26:19.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:19 smithi106 ceph-mon[44423]: pgmap v1282: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s 2024-04-20T17:26:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:19 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2402002617' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:26:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:19.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:19 smithi171 ceph-mon[44189]: pgmap v1282: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s 2024-04-20T17:26:19.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:19 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2402002617' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:26:19.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:20.904 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:26:20.904 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:26:20.904 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:26:20.904 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:26:20.904 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:26:20.904 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:26:20.904 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:26:20.904 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:26:20.904 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:26:20.904 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:26:20.904 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:26:20.905 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:26:20.905 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:26:20.905 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:26:20.905 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:26:20.905 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:26:20.905 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:26:20.905 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:26:20.905 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:26:20.905 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:26:20.905 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:26:20.905 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:26:20.905 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:26:20.906 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:26:21.518 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:21 smithi005 ceph-mon[39562]: pgmap v1283: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.07k op/s 2024-04-20T17:26:21.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:21 smithi171 ceph-mon[44189]: pgmap v1283: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.07k op/s 2024-04-20T17:26:21.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:21 smithi106 ceph-mon[44423]: pgmap v1283: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.07k op/s 2024-04-20T17:26:22.066 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:26:22.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:22.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:26:22.154 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:26:22.154 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":486,"fsid":"13c6e76c-ff37-11ee-bc93-c7b262605968","created":"2024-04-20T16:58:25.272557+0000","modified":"2024-04-20T17:26:13.153831+0000","last_up_change":"2024-04-20T17:22:32.243867+0000","last_in_change":"2024-04-20T17:23:20.118061+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":29,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T17:01:35.909493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.109999656677246,"score_stable":11.109999656677246,"optimal_score":0.18000000715255737,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.83594763278961182,"average_primary_affinity":0.9100000262260437,"average_primary_affinity_weighted":0.8399999737739563}},{"pool":4,"pool_name":"unique_pool_2","create_time":"2024-04-20T17:24:35.220785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"451","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.75,"score_stable":2.75,"optimal_score":1,"raw_score_acting":2.75,"raw_score_stable":2.75,"primary_affinity_weighted":0.83594763278961182,"average_primary_affinity":0.9100000262260437,"average_primary_affinity_weighted":0.8399999737739563}}],"osds":[{"osd":0,"uuid":"ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":445,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6801","nonce":3901484705}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6802","nonce":3901484705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6804","nonce":3901484705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6803","nonce":3901484705}]},"public_addr":"172.21.15.5:6801/3901484705","cluster_addr":"172.21.15.5:6802/3901484705","heartbeat_back_addr":"172.21.15.5:6804/3901484705","heartbeat_front_addr":"172.21.15.5:6803/3901484705","state":["exists","up"]},{"osd":1,"uuid":"aed0c8eb-0e81-4e85-ab8f-a60420060656","up":1,"in":1,"weight":1,"primary_affinity":0.0313720703125,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":356,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6805","nonce":3391893914}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6806","nonce":3391893914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6808","nonce":3391893914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6807","nonce":3391893914}]},"public_addr":"172.21.15.5:6805/3391893914","cluster_addr":"172.21.15.5:6806/3391893914","heartbeat_back_addr":"172.21.15.5:6808/3391893914","heartbeat_front_addr":"172.21.15.5:6807/3391893914","state":["exists","up"]},{"osd":2,"uuid":"f733a6a2-19c2-4c7a-af9c-04e282062bb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":20,"last_clean_end":169,"up_from":202,"up_thru":445,"down_at":170,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6809","nonce":2120275501}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6810","nonce":2120275501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6812","nonce":2120275501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6811","nonce":2120275501}]},"public_addr":"172.21.15.5:6809/2120275501","cluster_addr":"172.21.15.5:6810/2120275501","heartbeat_back_addr":"172.21.15.5:6812/2120275501","heartbeat_front_addr":"172.21.15.5:6811/2120275501","state":["exists","up"]},{"osd":3,"uuid":"8c92f1df-e298-4606-88ae-b8a8e79628a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":445,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6813","nonce":377652039}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6814","nonce":377652039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6816","nonce":377652039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6815","nonce":377652039}]},"public_addr":"172.21.15.5:6813/377652039","cluster_addr":"172.21.15.5:6814/377652039","heartbeat_back_addr":"172.21.15.5:6816/377652039","heartbeat_front_addr":"172.21.15.5:6815/377652039","state":["exists","up"]},{"osd":4,"uuid":"46df16f1-9607-4d7b-9be8-9d4236e160d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":445,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6800","nonce":1839524469}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6801","nonce":1839524469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6803","nonce":1839524469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6802","nonce":1839524469}]},"public_addr":"172.21.15.106:6800/1839524469","cluster_addr":"172.21.15.106:6801/1839524469","heartbeat_back_addr":"172.21.15.106:6803/1839524469","heartbeat_front_addr":"172.21.15.106:6802/1839524469","state":["exists","up"]},{"osd":5,"uuid":"6603231d-1e5c-4575-bfaa-2eeca2c12215","up":1,"in":1,"weight":1,"primary_affinity":0,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":417,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6804","nonce":2450315908}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6805","nonce":2450315908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6807","nonce":2450315908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6806","nonce":2450315908}]},"public_addr":"172.21.15.106:6804/2450315908","cluster_addr":"172.21.15.106:6805/2450315908","heartbeat_back_addr":"172.21.15.106:6807/2450315908","heartbeat_front_addr":"172.21.15.106:6806/2450315908","state":["exists","up"]},{"osd":6,"uuid":"783a55a6-f176-42be-8c8b-af8481d08dc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":445,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6808","nonce":2840237913}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6809","nonce":2840237913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6811","nonce":2840237913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6810","nonce":2840237913}]},"public_addr":"172.21.15.106:6808/2840237913","cluster_addr":"172.21.15.106:6809/2840237913","heartbeat_back_addr":"172.21.15.106:6811/2840237913","heartbeat_front_addr":"172.21.15.106:6810/2840237913","state":["exists","up"]},{"osd":7,"uuid":"b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":56,"last_clean_end":364,"up_from":398,"up_thru":419,"down_at":365,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6812","nonce":1997086583}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6813","nonce":1997086583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6815","nonce":1997086583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6814","nonce":1997086583}]},"public_addr":"172.21.15.106:6812/1997086583","cluster_addr":"172.21.15.106:6813/1997086583","heartbeat_back_addr":"172.21.15.106:6815/1997086583","heartbeat_front_addr":"172.21.15.106:6814/1997086583","state":["exists","up"]},{"osd":8,"uuid":"1cb232ad-1f59-4a0a-a215-e67419ad6476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":418,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6800","nonce":4123058942}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6801","nonce":4123058942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6803","nonce":4123058942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6802","nonce":4123058942}]},"public_addr":"172.21.15.171:6800/4123058942","cluster_addr":"172.21.15.171:6801/4123058942","heartbeat_back_addr":"172.21.15.171:6803/4123058942","heartbeat_front_addr":"172.21.15.171:6802/4123058942","state":["exists","up"]},{"osd":9,"uuid":"b6c17147-b3dc-475b-bb20-f9f712fb99e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":445,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6804","nonce":2877589636}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6805","nonce":2877589636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6807","nonce":2877589636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6806","nonce":2877589636}]},"public_addr":"172.21.15.171:6804/2877589636","cluster_addr":"172.21.15.171:6805/2877589636","heartbeat_back_addr":"172.21.15.171:6807/2877589636","heartbeat_front_addr":"172.21.15.171:6806/2877589636","state":["exists","up"]},{"osd":10,"uuid":"1adcdf55-f002-4a39-91e7-378e5fae849c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":445,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6808","nonce":181490621}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6809","nonce":181490621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6811","nonce":181490621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6810","nonce":181490621}]},"public_addr":"172.21.15.171:6808/181490621","cluster_addr":"172.21.15.171:6809/181490621","heartbeat_back_addr":"172.21.15.171:6811/181490621","heartbeat_front_addr":"172.21.15.171:6810/181490621","state":["exists","up"]},{"osd":11,"uuid":"737c5aec-694a-4968-819d-fccf22661f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":92,"up_thru":445,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6812","nonce":3373585701}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6813","nonce":3373585701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6815","nonce":3373585701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6814","nonce":3373585701}]},"public_addr":"172.21.15.171:6812/3373585701","cluster_addr":"172.21.15.171:6813/3373585701","heartbeat_back_addr":"172.21.15.171:6815/3373585701","heartbeat_front_addr":"172.21.15.171:6814/3373585701","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:26:11.146767+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:25:03.313859+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:26:03.715568+0000","dead_epoch":169},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:25:45.413690+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:25:28.665537+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:25:52.229705+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:25:41.848101+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:26:09.588660+0000","dead_epoch":364},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:25:31.031658+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:25:10.027088+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:25:47.603085+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:25:21.957370+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.5:0/3881726425":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/338649532":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/1227843664":"2024-04-21T16:58:55.392162+0000","172.21.15.5:6800/1876064631":"2024-04-21T16:58:55.392162+0000","172.21.15.5:0/160689449":"2024-04-21T16:58:55.392162+0000","172.21.15.5:6800/3521715286":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/3841663462":"2024-04-21T16:58:55.392162+0000","172.21.15.5:0/687690919":"2024-04-21T16:59:25.808337+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T17:26:22.170 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:26:22.216 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:22.301 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:22 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2323504457' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:26:22.351 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:26:22.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:22.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:26:22.455 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:26:22.485 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:22.518 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:26:22.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:22.520 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:26:22.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:22 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2323504457' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:26:22.622 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:26:22.692 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:22.701 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:26:22.702 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:26:22.727 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:22 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2323504457' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:26:22.761 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:26:22.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:22.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:26:22.872 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:26:22.915 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:23.058 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:26:23.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:23.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:26:23.112 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-20T17:26:23.165 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:26:23.218 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:23.260 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:26:23.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:23.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:26:23.322 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:26:23.367 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:26:23.412 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:23.442 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:23 smithi106 ceph-mon[44423]: pgmap v1284: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 63 MiB/s wr, 1.18k op/s 2024-04-20T17:26:23.443 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:23 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1425031153' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:26:23.443 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:23 smithi106 ceph-mon[44423]: from='client.26188 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:26:23.482 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-20T17:26:23.540 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:26:23.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:23.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:26:23.557 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:26:23.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:23 smithi171 ceph-mon[44189]: pgmap v1284: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 63 MiB/s wr, 1.18k op/s 2024-04-20T17:26:23.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:23 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1425031153' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:26:23.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:23 smithi171 ceph-mon[44189]: from='client.26188 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:26:23.592 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:23 smithi005 ceph-mon[39562]: pgmap v1284: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 63 MiB/s wr, 1.18k op/s 2024-04-20T17:26:23.593 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:23 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1425031153' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:26:23.593 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:23 smithi005 ceph-mon[39562]: from='client.26188 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:26:23.646 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:26:23.695 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:23.721 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:26:23.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:23.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:26:23.831 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:26:23.853 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:26:23.881 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:24.097 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:26:24.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:24.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:26:24.208 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:26:24.210 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:26:24.249 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:24.363 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:26:24.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:24.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:26:24.392 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:24 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:26:24.393 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:24 smithi106 ceph-mon[44423]: from='client.26188 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:26:24.393 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:24 smithi106 ceph-mon[44423]: osdmap e487: 12 total, 12 up, 12 in 2024-04-20T17:26:24.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:24 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:26:24.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:24 smithi005 ceph-mon[39562]: from='client.26188 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:26:24.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:24 smithi005 ceph-mon[39562]: osdmap e487: 12 total, 12 up, 12 in 2024-04-20T17:26:24.473 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:26:24.531 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:24.565 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:26:24.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:24.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:26:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:24 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:26:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:24 smithi171 ceph-mon[44189]: from='client.26188 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:26:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:24 smithi171 ceph-mon[44189]: osdmap e487: 12 total, 12 up, 12 in 2024-04-20T17:26:24.669 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:26:24.702 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:24.736 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:26:24.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:24.741 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:26:24.842 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:26:24.879 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:25.343 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:26:25.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:25.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:26:25.451 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:26:25.475 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:25 smithi005 ceph-mon[39562]: pgmap v1286: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 50 MiB/s wr, 1.23k op/s 2024-04-20T17:26:25.475 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:25 smithi005 ceph-mon[39562]: 4.2 scrub starts 2024-04-20T17:26:25.494 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:25.547 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:26:25.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:25.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:26:25.656 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:25 smithi106 ceph-mon[44423]: pgmap v1286: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 50 MiB/s wr, 1.23k op/s 2024-04-20T17:26:25.656 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:25 smithi106 ceph-mon[44423]: 4.2 scrub starts 2024-04-20T17:26:25.656 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:26:25.710 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:25.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:25 smithi171 ceph-mon[44189]: pgmap v1286: 17 pgs: 17 active+clean; 10 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 50 MiB/s wr, 1.23k op/s 2024-04-20T17:26:25.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:25 smithi171 ceph-mon[44189]: 4.2 scrub starts 2024-04-20T17:26:25.851 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:26:25.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:25.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:26:25.957 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:26:25.994 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:26.011 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:26:26.011 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:26:26.022 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:26:26.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:26.025 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:26:26.125 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:26:26.145 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:26:26.186 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:26.239 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:26:26.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:26.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:26:26.335 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:26:26.355 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:26:26.380 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:26:26.415 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:26.429 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:26:26.450 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:26:26.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:26.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:26:26.491 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:26:26.522 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:26:26.557 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:26:26.561 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:26:26.588 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:26.600 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:26:26.609 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:26:26.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:26.612 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:26:26.645 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:26:26.652 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:26:26.682 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:26:26.713 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:26:26.716 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:26:26.741 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:26:26.749 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-20T17:26:26.757 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:26.766 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:26:26.781 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:26:26.781 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:26:26.781 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:26:26.781 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:26:26.781 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:26:26.781 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:26:26.781 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:26:26.781 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:26:26.781 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:26:26.781 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:26:26.782 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:26:26.782 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:26:26.784 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:26:26.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:26.787 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:26:26.889 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:26:26.928 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:26.965 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:26:26.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:26.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:26:27.073 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:26:27.105 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:26:27.118 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:27.236 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:26:27.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:27.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:26:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:27 smithi106 ceph-mon[44423]: pgmap v1287: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 10 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 34 MiB/s wr, 1.25k op/s 2024-04-20T17:26:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:27 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3363430701' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:26:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:27.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:27.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:27.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:27.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:27.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:27.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:27.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:27.345 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:26:27.387 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:27.415 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:26:27.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:27.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:26:27.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:27 smithi005 ceph-mon[39562]: pgmap v1287: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 10 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 34 MiB/s wr, 1.25k op/s 2024-04-20T17:26:27.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:27 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3363430701' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:26:27.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:27.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:27.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:27.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:27.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:27.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:27.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:27.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:27.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:27.523 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:26:27.561 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:27 smithi171 ceph-mon[44189]: pgmap v1287: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 10 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 34 MiB/s wr, 1.25k op/s 2024-04-20T17:26:27.561 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:27 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3363430701' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:26:27.561 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:27.561 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:27.561 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:27.561 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:27.561 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:27.561 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:27.561 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:27.561 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:27.561 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:27.562 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:27.588 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:26:27.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:27.595 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:26:27.696 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:26:27.739 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:28.198 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:26:28.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:28.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:26:28.307 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:26:28.360 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:28.396 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:26:28.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:28.401 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:26:28.462 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:28.463 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:28 smithi106 ceph-mon[44423]: 4.2 scrub ok 2024-04-20T17:26:28.463 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:28.463 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:28.463 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:28 smithi106 ceph-mon[44423]: 4.7 scrub starts 2024-04-20T17:26:28.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:28.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:28 smithi005 ceph-mon[39562]: 4.2 scrub ok 2024-04-20T17:26:28.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:28.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:28.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:28 smithi005 ceph-mon[39562]: 4.7 scrub starts 2024-04-20T17:26:28.502 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:26:28.545 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:28.545 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:28 smithi171 ceph-mon[44189]: 4.2 scrub ok 2024-04-20T17:26:28.545 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:28.545 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:28.545 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:28 smithi171 ceph-mon[44189]: 4.7 scrub starts 2024-04-20T17:26:28.547 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:28.581 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:26:28.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:28.585 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:26:28.686 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:26:28.730 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:28.803 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:26:28.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:28.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:26:28.911 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:26:28.951 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:28.992 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:26:28.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:28.995 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:26:29.097 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:26:29.146 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:29.206 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:26:29.206 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:26:29.206 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.206 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26074.0:183300 4.a 4:55fdbfac:::benchmark_data_smithi171_122772_object13108:head [delete] snapc 0=[] ondisk+write+known_if_redirected e487)", 2024-04-20T17:26:29.206 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:26:29.188774+0000", 2024-04-20T17:26:29.206 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.015606781, 2024-04-20T17:26:29.206 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015638894, 2024-04-20T17:26:29.206 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:26:29.207 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:26:29.207 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:26:29.207 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:26:29.207 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26074", 2024-04-20T17:26:29.207 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:26:29.207 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 183300 2024-04-20T17:26:29.207 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.207 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:26:29.207 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.207 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:26:29.207 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.188774+0000", 2024-04-20T17:26:29.207 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:29.207 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.207 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.208 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:26:29.208 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.188774+0000", 2024-04-20T17:26:29.208 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:29.208 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.208 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.208 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:26:29.208 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.188777+0000", 2024-04-20T17:26:29.208 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.751e-06 2024-04-20T17:26:29.208 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.208 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.208 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:26:29.208 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.188788+0000", 2024-04-20T17:26:29.208 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1129e-05 2024-04-20T17:26:29.208 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.209 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.209 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:26:29.209 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.188799+0000", 2024-04-20T17:26:29.209 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1353e-05 2024-04-20T17:26:29.209 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.209 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.209 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:26:29.209 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.188809+0000", 2024-04-20T17:26:29.209 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5899999999999997e-06 2024-04-20T17:26:29.209 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.209 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.209 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:26:29.209 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.188894+0000", 2024-04-20T17:26:29.210 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4754999999999997e-05 2024-04-20T17:26:29.210 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.210 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:26:29.210 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.189030+0000", 2024-04-20T17:26:29.210 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013602899999999999 2024-04-20T17:26:29.210 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.210 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:26:29.210 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.189148+0000", 2024-04-20T17:26:29.210 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000118507 2024-04-20T17:26:29.210 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:29.210 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:26:29.211 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:29.211 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.211 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.211 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26074.0:183301 4.a 4:589ce9c2:::benchmark_data_smithi171_122772_object13109:head [delete] snapc 0=[] ondisk+write+known_if_redirected e487)", 2024-04-20T17:26:29.211 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:26:29.188832+0000", 2024-04-20T17:26:29.211 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.015549393, 2024-04-20T17:26:29.211 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015667719, 2024-04-20T17:26:29.211 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:26:29.211 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:26:29.211 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:26:29.211 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:26:29.211 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26074", 2024-04-20T17:26:29.211 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:26:29.212 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 183301 2024-04-20T17:26:29.212 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.212 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:26:29.212 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.212 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:26:29.212 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.188832+0000", 2024-04-20T17:26:29.213 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:29.213 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.213 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.213 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:26:29.213 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.188832+0000", 2024-04-20T17:26:29.213 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:29.213 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.213 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.213 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:26:29.213 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.188832+0000", 2024-04-20T17:26:29.213 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2099999999999995e-07 2024-04-20T17:26:29.213 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.213 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.213 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:26:29.213 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.188836+0000", 2024-04-20T17:26:29.214 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5190000000000001e-06 2024-04-20T17:26:29.214 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.214 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.214 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:26:29.214 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.188842+0000", 2024-04-20T17:26:29.214 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1290000000000001e-06 2024-04-20T17:26:29.214 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.214 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.214 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:26:29.214 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.188848+0000", 2024-04-20T17:26:29.214 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6030000000000004e-06 2024-04-20T17:26:29.214 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.214 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.214 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:26:29.215 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.194086+0000", 2024-04-20T17:26:29.215 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0052384679999999996 2024-04-20T17:26:29.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.215 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.215 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:26:29.215 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.194208+0000", 2024-04-20T17:26:29.215 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012144 2024-04-20T17:26:29.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.215 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.215 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:26:29.215 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.194322+0000", 2024-04-20T17:26:29.215 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000114584 2024-04-20T17:26:29.215 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:29.215 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:26:29.216 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:29.216 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.216 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.216 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.26074.0:183300 4.a e487/445)", 2024-04-20T17:26:29.216 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:26:29.201617+0000", 2024-04-20T17:26:29.216 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.002764079, 2024-04-20T17:26:29.216 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0029556999999999999, 2024-04-20T17:26:29.216 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:26:29.216 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:26:29.216 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "queued for pg", 2024-04-20T17:26:29.216 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:26:29.216 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.216 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:26:29.216 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.201617+0000", 2024-04-20T17:26:29.216 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:29.217 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.217 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.217 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:26:29.217 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.201617+0000", 2024-04-20T17:26:29.217 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:29.217 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.217 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.217 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:26:29.217 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.201618+0000", 2024-04-20T17:26:29.217 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.33e-06 2024-04-20T17:26:29.217 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.217 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.217 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:26:29.217 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.201624+0000", 2024-04-20T17:26:29.217 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8259999999999998e-06 2024-04-20T17:26:29.218 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.218 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:26:29.218 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.201634+0000", 2024-04-20T17:26:29.218 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0087e-05 2024-04-20T17:26:29.218 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.218 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:26:29.218 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.201641+0000", 2024-04-20T17:26:29.218 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.754e-06 2024-04-20T17:26:29.218 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:29.218 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:26:29.218 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:29.218 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.219 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.26074.0:183301 4.a e487/445)", 2024-04-20T17:26:29.219 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:26:29.201751+0000", 2024-04-20T17:26:29.219 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.002629447, 2024-04-20T17:26:29.219 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0028943369999999999, 2024-04-20T17:26:29.219 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:26:29.219 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:26:29.219 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "queued for pg", 2024-04-20T17:26:29.219 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:26:29.219 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.219 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:26:29.219 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.201751+0000", 2024-04-20T17:26:29.219 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:29.219 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.219 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.219 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:26:29.220 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.201751+0000", 2024-04-20T17:26:29.220 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:26:29.220 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.220 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.220 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:26:29.220 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.201753+0000", 2024-04-20T17:26:29.220 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0440000000000001e-06 2024-04-20T17:26:29.220 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.220 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.220 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:26:29.220 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.201756+0000", 2024-04-20T17:26:29.220 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4010000000000001e-06 2024-04-20T17:26:29.220 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.220 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.220 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:26:29.221 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.201762+0000", 2024-04-20T17:26:29.221 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8680000000000004e-06 2024-04-20T17:26:29.221 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:26:29.221 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:26:29.221 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:26:29.221 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:26:29.201767+0000", 2024-04-20T17:26:29.221 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.916e-06 2024-04-20T17:26:29.221 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:29.221 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:26:29.221 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:29.221 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:26:29.221 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-04-20T17:26:29.221 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 4 2024-04-20T17:26:29.221 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:26:29.236 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:26:29.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:29.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:26:29.344 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:26:29.377 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:29.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:29 smithi005 ceph-mon[39562]: 4.e scrub starts 2024-04-20T17:26:29.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:29 smithi005 ceph-mon[39562]: pgmap v1288: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 15 MiB/s wr, 1.25k op/s 2024-04-20T17:26:29.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:29 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:26:29.483 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:26:29.485 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:29 smithi171 ceph-mon[44189]: 4.e scrub starts 2024-04-20T17:26:29.485 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:29 smithi171 ceph-mon[44189]: pgmap v1288: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 15 MiB/s wr, 1.25k op/s 2024-04-20T17:26:29.486 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:29 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:26:29.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:29.486 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:26:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:29 smithi106 ceph-mon[44423]: 4.e scrub starts 2024-04-20T17:26:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:29 smithi106 ceph-mon[44423]: pgmap v1288: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 15 MiB/s wr, 1.25k op/s 2024-04-20T17:26:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:29 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:26:29.587 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:26:29.642 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:29.674 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:26:29.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:29.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:26:29.778 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:26:29.825 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:29.954 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:26:29.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:29.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:26:29.986 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-20T17:26:30.064 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:26:30.095 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:30.128 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:26:30.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:30.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:26:30.195 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:26:30.229 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:30 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:26:30.229 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:30 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:26:30.229 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:30 smithi106 ceph-mon[44423]: 4.5 scrub starts 2024-04-20T17:26:30.229 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:30 smithi106 ceph-mon[44423]: 4.7 scrub ok 2024-04-20T17:26:30.229 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:30 smithi106 ceph-mon[44423]: osdmap e488: 12 total, 12 up, 12 in 2024-04-20T17:26:30.229 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:30 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4072032166' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:26:30.229 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:30 smithi106 ceph-mon[44423]: from='client.26282 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:26:30.229 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:26:30.238 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:26:30.281 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:30.306 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:26:30.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:30.311 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:26:30.412 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:26:30.420 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:26:30.446 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:30.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:30 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:26:30.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:30 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:26:30.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:30 smithi005 ceph-mon[39562]: 4.5 scrub starts 2024-04-20T17:26:30.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:30 smithi005 ceph-mon[39562]: 4.7 scrub ok 2024-04-20T17:26:30.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:30 smithi005 ceph-mon[39562]: osdmap e488: 12 total, 12 up, 12 in 2024-04-20T17:26:30.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:30 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4072032166' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:26:30.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:30 smithi005 ceph-mon[39562]: from='client.26282 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:26:30.472 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:26:30.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:30.480 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:30 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:26:30.480 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:30 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:26:30.480 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:30 smithi171 ceph-mon[44189]: 4.5 scrub starts 2024-04-20T17:26:30.480 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:30 smithi171 ceph-mon[44189]: 4.7 scrub ok 2024-04-20T17:26:30.481 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:30 smithi171 ceph-mon[44189]: osdmap e488: 12 total, 12 up, 12 in 2024-04-20T17:26:30.481 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:30 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4072032166' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:26:30.481 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:30 smithi171 ceph-mon[44189]: from='client.26282 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:26:30.481 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:26:30.582 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:26:30.636 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:30.703 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:26:30.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:30.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:26:30.812 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:26:30.844 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:31.112 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:26:31.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:31.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:26:31.147 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:26:31.215 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:26:31.257 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:31 smithi106 ceph-mon[44423]: 4.e scrub ok 2024-04-20T17:26:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:31 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:26:31.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:31 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:26:31.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:31 smithi106 ceph-mon[44423]: from='client.26282 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:26:31.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:31 smithi106 ceph-mon[44423]: osdmap e489: 12 total, 12 up, 12 in 2024-04-20T17:26:31.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:31 smithi106 ceph-mon[44423]: pgmap v1291: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s 2024-04-20T17:26:31.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:31 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:26:31.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:31 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:26:31.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:31 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:26:31.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:31 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:26:31.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:31 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:26:31.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:31 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:26:31.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:31 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:26:31.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:31 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:26:31.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:31 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:26:31.287 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:26:31.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:31.293 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:26:31.394 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:26:31.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:31 smithi005 ceph-mon[39562]: 4.e scrub ok 2024-04-20T17:26:31.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:31 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:26:31.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:31 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:26:31.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:31 smithi005 ceph-mon[39562]: from='client.26282 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:26:31.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:31 smithi005 ceph-mon[39562]: osdmap e489: 12 total, 12 up, 12 in 2024-04-20T17:26:31.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:31 smithi005 ceph-mon[39562]: pgmap v1291: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s 2024-04-20T17:26:31.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:31 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:26:31.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:31 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:26:31.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:31 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:26:31.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:31 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:26:31.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:31 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:26:31.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:31 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:26:31.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:31 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:26:31.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:31 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:26:31.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:31 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:26:31.446 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:31.469 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:26:31.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:31.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:26:31.574 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:26:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:31 smithi171 ceph-mon[44189]: 4.e scrub ok 2024-04-20T17:26:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:31 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:26:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:31 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:26:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:31 smithi171 ceph-mon[44189]: from='client.26282 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:26:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:31 smithi171 ceph-mon[44189]: osdmap e489: 12 total, 12 up, 12 in 2024-04-20T17:26:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:31 smithi171 ceph-mon[44189]: pgmap v1291: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s 2024-04-20T17:26:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:31 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:26:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:31 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:26:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:31 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:26:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:31 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:26:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:31 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:26:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:31 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:26:31.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:31 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:26:31.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:31 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:26:31.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:31 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:26:31.604 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:31.871 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:26:31.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:31.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:26:31.976 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:26:32.014 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:32.037 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:26:32.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:32.042 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:26:32.143 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:26:32.174 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:32.288 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:26:32.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:32.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:26:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:32 smithi171 ceph-mon[44189]: 4.5 scrub ok 2024-04-20T17:26:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:32 smithi171 ceph-mon[44189]: osdmap e490: 12 total, 12 up, 12 in 2024-04-20T17:26:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:32 smithi171 ceph-mon[44189]: 4.3 scrub starts 2024-04-20T17:26:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:32 smithi171 ceph-mon[44189]: 4.f scrub starts 2024-04-20T17:26:32.394 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:26:32.422 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:32 smithi106 ceph-mon[44423]: 4.5 scrub ok 2024-04-20T17:26:32.423 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:32 smithi106 ceph-mon[44423]: osdmap e490: 12 total, 12 up, 12 in 2024-04-20T17:26:32.423 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:32 smithi106 ceph-mon[44423]: 4.3 scrub starts 2024-04-20T17:26:32.423 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:32 smithi106 ceph-mon[44423]: 4.f scrub starts 2024-04-20T17:26:32.424 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:32.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:32 smithi005 ceph-mon[39562]: 4.5 scrub ok 2024-04-20T17:26:32.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:32 smithi005 ceph-mon[39562]: osdmap e490: 12 total, 12 up, 12 in 2024-04-20T17:26:32.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:32 smithi005 ceph-mon[39562]: 4.3 scrub starts 2024-04-20T17:26:32.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:32 smithi005 ceph-mon[39562]: 4.f scrub starts 2024-04-20T17:26:32.467 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:26:32.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:32.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:26:32.572 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:26:32.610 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:32.618 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:26:32.618 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:26:32.618 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-04-20T17:26:32.618 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-04-20T17:26:32.618 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:26:32.638 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:26:32.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:32.644 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:26:32.745 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:26:32.782 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:32.819 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:26:32.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:32.825 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:26:32.926 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:26:32.969 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:33.062 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:26:33.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:33.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:26:33.135 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:26:33.167 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:26:33.208 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:33.295 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:26:33.342 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:26:33.353 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:33 smithi106 ceph-mon[44423]: pgmap v1293: 17 pgs: 17 active+clean; 9.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s 2024-04-20T17:26:33.353 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:33 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/593246270' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:26:33.362 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:26:33.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:33.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:26:33.388 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-20T17:26:33.392 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:26:33.444 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:26:33.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:33 smithi005 ceph-mon[39562]: pgmap v1293: 17 pgs: 17 active+clean; 9.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s 2024-04-20T17:26:33.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:33 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/593246270' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:26:33.469 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:26:33.489 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:26:33.498 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:33.523 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:26:33.524 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:26:33.526 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:33 smithi171 ceph-mon[44189]: pgmap v1293: 17 pgs: 17 active+clean; 9.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s 2024-04-20T17:26:33.526 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:33 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/593246270' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:26:33.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:33.528 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:26:33.562 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:26:33.599 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:26:33.610 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:26:33.628 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:26:33.638 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:26:33.662 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:33.685 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:26:33.688 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:26:33.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:33.689 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:26:33.721 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:26:33.757 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:26:33.763 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:26:33.763 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:26:33.763 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:26:33.763 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:26:33.764 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:26:33.764 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:26:33.764 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:26:33.764 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:26:33.764 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:26:33.764 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:26:33.764 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:26:33.764 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:26:33.766 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:26:33.790 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:26:33.823 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:33.846 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:26:33.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:33.851 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:26:33.952 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:26:33.994 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:34.037 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:26:34.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:34.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:26:34.145 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:26:34.184 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:34.213 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:26:34.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:34.217 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:26:34.318 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:26:34.362 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:34.382 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:26:34.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:34.388 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:26:34.489 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:26:34.536 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:34 smithi005 ceph-mon[39562]: 4.3 scrub ok 2024-04-20T17:26:34.536 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.536 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.536 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.536 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.536 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.536 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.536 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.536 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.536 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.536 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.537 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:34.574 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:26:34.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:34 smithi171 ceph-mon[44189]: 4.3 scrub ok 2024-04-20T17:26:34.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:34.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:26:34.697 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:26:34.741 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:34 smithi106 ceph-mon[44423]: 4.3 scrub ok 2024-04-20T17:26:34.741 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.742 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.742 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.742 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.742 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.742 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.742 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.742 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.742 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.742 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:34.744 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:35.205 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:26:35.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:35.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:26:35.312 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:26:35.346 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:35.369 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:26:35.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:35.372 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:26:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:35 smithi005 ceph-mon[39562]: 4.f scrub ok 2024-04-20T17:26:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:35 smithi005 ceph-mon[39562]: pgmap v1294: 17 pgs: 17 active+clean; 9.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-04-20T17:26:35.473 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:26:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:35 smithi106 ceph-mon[44423]: 4.f scrub ok 2024-04-20T17:26:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:35 smithi106 ceph-mon[44423]: pgmap v1294: 17 pgs: 17 active+clean; 9.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-04-20T17:26:35.512 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:35.536 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:26:35.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:35.543 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:26:35.543 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:35.543 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:35.543 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:35 smithi171 ceph-mon[44189]: 4.f scrub ok 2024-04-20T17:26:35.543 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:35 smithi171 ceph-mon[44189]: pgmap v1294: 17 pgs: 17 active+clean; 9.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-04-20T17:26:35.643 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:26:35.681 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:35.794 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:26:35.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:35.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:26:35.858 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:26:35.859 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:26:35.898 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:26:35.959 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:35.987 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:26:35.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:35.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:26:36.093 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:26:36.156 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:36.272 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:26:36.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:36.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:26:36.377 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:26:36.415 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:36.618 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:26:36.671 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-20T17:26:36.672 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:26:36.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:36.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:26:36.775 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:26:36.827 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:36.854 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:26:36.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:36.857 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:26:36.958 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:26:37.007 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:37.022 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:26:37.036 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:26:37.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:37.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:26:37.046 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:26:37.146 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:26:37.187 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:37.208 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:26:37.333 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:37 smithi005 ceph-mon[39562]: pgmap v1295: 17 pgs: 17 active+clean; 9.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:26:37.333 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:37 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2397704541' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:26:37.426 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:37 smithi171 ceph-mon[44189]: pgmap v1295: 17 pgs: 17 active+clean; 9.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:26:37.426 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:37 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2397704541' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:26:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:37 smithi106 ceph-mon[44423]: pgmap v1295: 17 pgs: 17 active+clean; 9.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:26:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:37 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2397704541' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:26:38.092 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:26:38.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:38.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:26:38.198 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:26:38.251 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:38.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:38 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:26:38.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:38 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2397704541' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:26:38.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:38 smithi005 ceph-mon[39562]: osdmap e491: 12 total, 12 up, 12 in 2024-04-20T17:26:38.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:38 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:26:38.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:38 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2397704541' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:26:38.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:38 smithi106 ceph-mon[44423]: osdmap e491: 12 total, 12 up, 12 in 2024-04-20T17:26:38.528 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:26:38.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:38.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:26:38.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:38 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:26:38.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:38 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2397704541' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:26:38.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:38 smithi171 ceph-mon[44189]: osdmap e491: 12 total, 12 up, 12 in 2024-04-20T17:26:38.635 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:26:38.681 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:39.325 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:26:39.326 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:26:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:39 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:26:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:39 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:26:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:39 smithi171 ceph-mon[44189]: osdmap e492: 12 total, 12 up, 12 in 2024-04-20T17:26:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:39 smithi171 ceph-mon[44189]: pgmap v1298: 17 pgs: 17 active+clean; 9.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.28k op/s 2024-04-20T17:26:39.403 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:39 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:26:39.403 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:39 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:26:39.403 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:39 smithi005 ceph-mon[39562]: osdmap e492: 12 total, 12 up, 12 in 2024-04-20T17:26:39.403 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:39 smithi005 ceph-mon[39562]: pgmap v1298: 17 pgs: 17 active+clean; 9.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.28k op/s 2024-04-20T17:26:39.438 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:26:39.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:39.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:26:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:39 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:26:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:39 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:26:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:39 smithi106 ceph-mon[44423]: osdmap e492: 12 total, 12 up, 12 in 2024-04-20T17:26:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:39 smithi106 ceph-mon[44423]: pgmap v1298: 17 pgs: 17 active+clean; 9.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.28k op/s 2024-04-20T17:26:39.540 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:26:39.576 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:39.642 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:26:39.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:39.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:26:39.757 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:26:39.807 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:39.853 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:26:39.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:39.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:26:39.959 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:26:39.991 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:40.016 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:26:40.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:40.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:26:40.124 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:26:40.147 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:26:40.165 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:40.173 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:26:40.195 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-20T17:26:40.201 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:26:40.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:40.206 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:26:40.207 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:26:40.234 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:26:40.258 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:26:40.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:40 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1072202982' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:26:40.286 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:26:40.307 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:26:40.320 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:26:40.348 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:40.353 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:26:40.377 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:26:40.381 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:26:40.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:40.384 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:26:40.417 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:26:40.454 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:26:40.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:40 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1072202982' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:26:40.485 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:26:40.490 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:26:40.493 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:26:40.493 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:26:40.493 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:26:40.493 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:26:40.493 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:26:40.493 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:26:40.493 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:26:40.493 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:26:40.493 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:26:40.494 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:26:40.494 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:26:40.494 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:26:40.517 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:40.547 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:26:40.549 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:26:40.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:40.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:26:40.574 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:26:40.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:40 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1072202982' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:26:40.651 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:26:40.696 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:40.901 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:26:40.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:40.906 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:26:41.007 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:26:41.060 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:41.172 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:26:41.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:41.181 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:26:41.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:41 smithi005 ceph-mon[39562]: pgmap v1299: 17 pgs: 17 active+clean; 9.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s 2024-04-20T17:26:41.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.282 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:26:41.314 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.314 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.314 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.314 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:41 smithi171 ceph-mon[44189]: pgmap v1299: 17 pgs: 17 active+clean; 9.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s 2024-04-20T17:26:41.314 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.314 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.315 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.315 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.315 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.315 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.315 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.315 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.317 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:41.427 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:26:41.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:41 smithi106 ceph-mon[44423]: pgmap v1299: 17 pgs: 17 active+clean; 9.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s 2024-04-20T17:26:41.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:41.529 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:26:41.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:41.533 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:26:41.634 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:26:41.664 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:41.743 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:26:41.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:41.748 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:26:41.850 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:26:41.901 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:41.967 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:26:41.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:41.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:26:42.077 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:26:42.110 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:42.141 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:26:42.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:42.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:26:42.245 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:26:42.287 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:42.291 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:42.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:42.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:42.855 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:26:42.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:42.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:26:42.962 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:26:43.006 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:43.049 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:26:43.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:43.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:26:43.113 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-20T17:26:43.114 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:26:43.159 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:26:43.221 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:43 smithi171 ceph-mon[44189]: pgmap v1300: 17 pgs: 17 active+clean; 8.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-04-20T17:26:43.230 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:43.279 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:26:43.280 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:26:43.280 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-04-20T17:26:43.280 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-04-20T17:26:43.280 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:26:43.321 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:26:43.455 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:26:43.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:43.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:26:43.473 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:43 smithi005 ceph-mon[39562]: pgmap v1300: 17 pgs: 17 active+clean; 8.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-04-20T17:26:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:43 smithi106 ceph-mon[44423]: pgmap v1300: 17 pgs: 17 active+clean; 8.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-04-20T17:26:43.564 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:26:43.597 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:43.604 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:26:44.111 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-20T17:26:44.136 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:26:44.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:44.140 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:26:44.241 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:26:44.244 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:26:44.289 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:44.318 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:26:44.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:44.322 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:26:44.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:44 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1906781299' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:26:44.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:44 smithi171 ceph-mon[44189]: from='client.26221 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:26:44.423 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:26:44.452 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:44.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:44 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1906781299' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:26:44.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:44 smithi005 ceph-mon[39562]: from='client.26221 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:26:44.476 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:26:44.479 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:44 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1906781299' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:26:44.479 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:44 smithi106 ceph-mon[44423]: from='client.26221 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:26:44.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:44.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:26:44.532 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:26:44.581 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:26:44.622 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:45.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:45 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:26:45.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:45 smithi106 ceph-mon[44423]: pgmap v1301: 17 pgs: 17 active+clean; 8.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:26:45.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:45 smithi106 ceph-mon[44423]: from='client.26221 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:26:45.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:45 smithi106 ceph-mon[44423]: osdmap e493: 12 total, 12 up, 12 in 2024-04-20T17:26:45.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:45 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:26:45.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:45 smithi171 ceph-mon[44189]: pgmap v1301: 17 pgs: 17 active+clean; 8.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:26:45.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:45 smithi171 ceph-mon[44189]: from='client.26221 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:26:45.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:45 smithi171 ceph-mon[44189]: osdmap e493: 12 total, 12 up, 12 in 2024-04-20T17:26:45.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:45 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:26:45.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:45 smithi005 ceph-mon[39562]: pgmap v1301: 17 pgs: 17 active+clean; 8.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:26:45.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:45 smithi005 ceph-mon[39562]: from='client.26221 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:26:45.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:45 smithi005 ceph-mon[39562]: osdmap e493: 12 total, 12 up, 12 in 2024-04-20T17:26:46.790 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:26:46.791 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:26:46.791 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:26:46.791 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:26:46.791 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.791 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24736.0:315959 2.b 2:da63b980:::benchmark_data_smithi171_78918_object154965:head [delete] snapc 0=[] ondisk+write+known_if_redirected e345)", 2024-04-20T17:26:46.791 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:19:58.819465+0000", 2024-04-20T17:26:46.791 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 407.96772484100001, 2024-04-20T17:26:46.791 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.041140903, 2024-04-20T17:26:46.791 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:26:46.791 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:26:46.791 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:46.792 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:26:46.792 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24736", 2024-04-20T17:26:46.792 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:26:46.792 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 315959 2024-04-20T17:26:46.792 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.792 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:26:46.792 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.792 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:26:46.792 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:19:58.819465+0000", 2024-04-20T17:26:46.792 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.792 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.792 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.792 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:26:46.792 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:19:58.819465+0000", 2024-04-20T17:26:46.792 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.793 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.793 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.793 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:26:46.793 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:19:58.819468+0000", 2024-04-20T17:26:46.793 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0769999999999999e-06 2024-04-20T17:26:46.793 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.793 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.793 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:26:46.793 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:19:58.819476+0000", 2024-04-20T17:26:46.793 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9570000000000002e-06 2024-04-20T17:26:46.793 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.793 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.793 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:26:46.793 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:19:58.819490+0000", 2024-04-20T17:26:46.794 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3268000000000001e-05 2024-04-20T17:26:46.794 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.794 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.794 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:26:46.794 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:19:58.819498+0000", 2024-04-20T17:26:46.794 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8540000000000003e-06 2024-04-20T17:26:46.794 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.794 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.794 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:26:46.794 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:19:58.820995+0000", 2024-04-20T17:26:46.794 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0014966859999999999 2024-04-20T17:26:46.794 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.794 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.795 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:26:46.795 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:19:58.821114+0000", 2024-04-20T17:26:46.795 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011890100000000001 2024-04-20T17:26:46.795 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.795 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.795 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:26:46.795 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:19:58.821244+0000", 2024-04-20T17:26:46.795 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012984699999999999 2024-04-20T17:26:46.795 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.795 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.795 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:26:46.795 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:19:58.835174+0000", 2024-04-20T17:26:46.795 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013930563 2024-04-20T17:26:46.796 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.796 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.796 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:46.796 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:19:58.860517+0000", 2024-04-20T17:26:46.796 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.025342301000000001 2024-04-20T17:26:46.796 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.796 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.796 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:26:46.796 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:19:58.860566+0000", 2024-04-20T17:26:46.797 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8902e-05 2024-04-20T17:26:46.797 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.797 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.797 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:26:46.797 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:19:58.860606+0000", 2024-04-20T17:26:46.797 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0547000000000003e-05 2024-04-20T17:26:46.797 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.797 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:26:46.797 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.797 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.797 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.797 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:60286 3.9 3:9dc2e748:::benchmark_data_smithi171_110765_object60285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e375)", 2024-04-20T17:26:46.797 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:06.920871+0000", 2024-04-20T17:26:46.798 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 339.86631873900001, 2024-04-20T17:26:46.798 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.040852670000000001, 2024-04-20T17:26:46.798 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:26:46.798 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:26:46.798 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:46.798 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:26:46.798 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:26:46.798 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:26:46.798 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 60286 2024-04-20T17:26:46.798 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.798 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:26:46.798 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.798 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:26:46.799 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.920871+0000", 2024-04-20T17:26:46.799 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.799 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.799 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.799 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:26:46.799 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.920871+0000", 2024-04-20T17:26:46.799 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.799 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.799 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.799 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:26:46.799 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.920874+0000", 2024-04-20T17:26:46.799 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9799999999999998e-06 2024-04-20T17:26:46.799 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.799 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.799 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:26:46.800 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.920914+0000", 2024-04-20T17:26:46.800 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9759999999999999e-05 2024-04-20T17:26:46.800 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.800 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.800 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:26:46.800 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.920933+0000", 2024-04-20T17:26:46.800 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8845999999999999e-05 2024-04-20T17:26:46.800 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.800 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.801 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:26:46.801 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.920947+0000", 2024-04-20T17:26:46.801 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3783e-05 2024-04-20T17:26:46.801 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.801 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.801 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:26:46.801 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.921001+0000", 2024-04-20T17:26:46.801 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3726e-05 2024-04-20T17:26:46.801 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.801 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.801 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:26:46.801 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.921095+0000", 2024-04-20T17:26:46.801 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.4557999999999998e-05 2024-04-20T17:26:46.801 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.801 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.802 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:26:46.802 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.921189+0000", 2024-04-20T17:26:46.802 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3690999999999995e-05 2024-04-20T17:26:46.802 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.802 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.802 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:26:46.802 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.922912+0000", 2024-04-20T17:26:46.802 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0017236930000000001 2024-04-20T17:26:46.802 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.802 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.802 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:46.802 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.961608+0000", 2024-04-20T17:26:46.802 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.038695653000000003 2024-04-20T17:26:46.803 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.803 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.803 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:26:46.803 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.961672+0000", 2024-04-20T17:26:46.803 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4323000000000001e-05 2024-04-20T17:26:46.803 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.803 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.803 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:26:46.803 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.961724+0000", 2024-04-20T17:26:46.803 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1656999999999997e-05 2024-04-20T17:26:46.803 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.803 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:26:46.803 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.803 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.803 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.804 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:60331 3.9 3:963a7ec6:::benchmark_data_smithi171_110765_object60330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e375)", 2024-04-20T17:26:46.804 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:06.963321+0000", 2024-04-20T17:26:46.804 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 339.82386910100001, 2024-04-20T17:26:46.804 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.04670614, 2024-04-20T17:26:46.804 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:26:46.804 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:26:46.804 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:46.804 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:26:46.804 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:26:46.804 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:26:46.804 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 60331 2024-04-20T17:26:46.804 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.804 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:26:46.804 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.805 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:26:46.805 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.963321+0000", 2024-04-20T17:26:46.805 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.805 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.805 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.805 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:26:46.805 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.963321+0000", 2024-04-20T17:26:46.805 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.805 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.805 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.805 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:26:46.805 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.963324+0000", 2024-04-20T17:26:46.805 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.774e-06 2024-04-20T17:26:46.806 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.806 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.806 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:26:46.806 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.963348+0000", 2024-04-20T17:26:46.806 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3700999999999999e-05 2024-04-20T17:26:46.806 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.806 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.806 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:26:46.806 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.963360+0000", 2024-04-20T17:26:46.806 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.22e-05 2024-04-20T17:26:46.806 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.806 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.806 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:26:46.806 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.963368+0000", 2024-04-20T17:26:46.807 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1939999999999998e-06 2024-04-20T17:26:46.807 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.807 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.807 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:26:46.807 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.963423+0000", 2024-04-20T17:26:46.807 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5374999999999998e-05 2024-04-20T17:26:46.808 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.808 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.808 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:26:46.808 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.963547+0000", 2024-04-20T17:26:46.808 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012394199999999999 2024-04-20T17:26:46.808 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.808 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.809 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:26:46.809 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.963680+0000", 2024-04-20T17:26:46.809 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000132402 2024-04-20T17:26:46.809 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.809 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.809 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:26:46.809 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.966954+0000", 2024-04-20T17:26:46.809 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0032742380000000001 2024-04-20T17:26:46.809 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.809 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.809 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:46.809 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:07.009928+0000", 2024-04-20T17:26:46.810 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.042973650000000002 2024-04-20T17:26:46.810 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.810 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.810 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:26:46.810 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:07.009981+0000", 2024-04-20T17:26:46.810 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3270999999999999e-05 2024-04-20T17:26:46.810 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.811 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.811 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:26:46.811 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:07.010027+0000", 2024-04-20T17:26:46.811 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6393000000000001e-05 2024-04-20T17:26:46.811 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.811 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:26:46.811 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.811 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.811 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.811 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:60333 3.9 3:908c9452:::benchmark_data_smithi171_110765_object60332:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e375)", 2024-04-20T17:26:46.811 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:06.964890+0000", 2024-04-20T17:26:46.812 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 339.82230035600003, 2024-04-20T17:26:46.812 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.046167593999999999, 2024-04-20T17:26:46.812 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:26:46.812 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:26:46.812 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:46.812 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:26:46.812 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:26:46.812 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:26:46.812 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 60333 2024-04-20T17:26:46.812 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.812 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:26:46.812 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.812 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:26:46.812 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.964890+0000", 2024-04-20T17:26:46.813 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.813 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.813 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.813 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:26:46.813 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.964890+0000", 2024-04-20T17:26:46.813 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.813 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.813 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.813 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:26:46.813 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.964891+0000", 2024-04-20T17:26:46.813 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6670000000000001e-06 2024-04-20T17:26:46.813 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.813 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.813 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:26:46.814 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.964911+0000", 2024-04-20T17:26:46.814 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9799000000000001e-05 2024-04-20T17:26:46.814 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.814 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.814 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:26:46.814 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.964921+0000", 2024-04-20T17:26:46.814 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7529999999999996e-06 2024-04-20T17:26:46.814 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.814 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.814 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:26:46.814 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.964927+0000", 2024-04-20T17:26:46.814 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8109999999999999e-06 2024-04-20T17:26:46.814 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.815 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.815 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:26:46.815 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.965350+0000", 2024-04-20T17:26:46.815 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00042355300000000002 2024-04-20T17:26:46.815 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.815 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.815 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:26:46.815 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.965484+0000", 2024-04-20T17:26:46.815 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013349600000000001 2024-04-20T17:26:46.815 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.815 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.815 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:26:46.815 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.965632+0000", 2024-04-20T17:26:46.815 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0001482 2024-04-20T17:26:46.815 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.816 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.816 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:26:46.816 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:06.967100+0000", 2024-04-20T17:26:46.816 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0014678200000000001 2024-04-20T17:26:46.816 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.816 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.816 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:46.816 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:07.010972+0000", 2024-04-20T17:26:46.816 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043871996000000003 2024-04-20T17:26:46.816 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.816 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.816 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:26:46.816 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:07.011021+0000", 2024-04-20T17:26:46.816 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9329000000000002e-05 2024-04-20T17:26:46.816 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.816 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.817 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:26:46.817 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:07.011057+0000", 2024-04-20T17:26:46.817 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6170000000000001e-05 2024-04-20T17:26:46.817 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.817 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:26:46.817 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.817 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.817 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.817 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:112591 3.f e383/368 3:fc32f703:::benchmark_data_smithi171_110765_object112590:head v 383'7835, mlcod=383'7830)", 2024-04-20T17:26:46.817 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:36.569016+0000", 2024-04-20T17:26:46.817 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 310.21817458700002, 2024-04-20T17:26:46.817 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.039064569, 2024-04-20T17:26:46.817 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:26:46.817 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:26:46.817 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:46.818 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:26:46.818 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.818 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:26:46.818 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:36.569016+0000", 2024-04-20T17:26:46.818 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.818 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.818 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.818 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:26:46.818 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:36.569016+0000", 2024-04-20T17:26:46.818 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.818 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.818 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.818 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:26:46.819 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:36.569017+0000", 2024-04-20T17:26:46.819 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.341e-06 2024-04-20T17:26:46.819 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.819 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.819 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:26:46.819 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:36.569040+0000", 2024-04-20T17:26:46.819 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3309999999999999e-05 2024-04-20T17:26:46.819 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.819 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.819 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:26:46.819 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:36.569051+0000", 2024-04-20T17:26:46.819 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0514e-05 2024-04-20T17:26:46.819 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.819 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.819 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:26:46.820 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:36.569058+0000", 2024-04-20T17:26:46.820 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0820000000000001e-06 2024-04-20T17:26:46.820 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.820 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.820 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:26:46.820 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:36.569110+0000", 2024-04-20T17:26:46.820 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1941000000000003e-05 2024-04-20T17:26:46.820 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.820 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.820 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:26:46.820 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:36.569128+0000", 2024-04-20T17:26:46.820 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7788999999999999e-05 2024-04-20T17:26:46.820 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.820 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.820 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:26:46.821 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:36.608037+0000", 2024-04-20T17:26:46.821 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.038909060000000002 2024-04-20T17:26:46.821 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.821 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.821 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:26:46.821 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:36.608080+0000", 2024-04-20T17:26:46.821 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3532000000000003e-05 2024-04-20T17:26:46.821 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.821 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:26:46.821 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.821 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.821 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.821 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:120099 3.f e384/368 3:f273bf0a:::benchmark_data_smithi171_110765_object120098:head v 384'8997, mlcod=384'8994)", 2024-04-20T17:26:46.821 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:42.702388+0000", 2024-04-20T17:26:46.822 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 304.08480251700001, 2024-04-20T17:26:46.822 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.053590042999999997, 2024-04-20T17:26:46.822 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:26:46.822 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:26:46.822 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:46.822 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:26:46.822 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.822 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:26:46.822 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.702388+0000", 2024-04-20T17:26:46.822 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.822 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.822 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.822 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:26:46.823 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.702388+0000", 2024-04-20T17:26:46.823 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.823 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.823 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.823 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:26:46.823 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.702388+0000", 2024-04-20T17:26:46.823 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1699999999999998e-07 2024-04-20T17:26:46.823 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.823 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.823 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:26:46.823 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.702416+0000", 2024-04-20T17:26:46.823 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8048e-05 2024-04-20T17:26:46.823 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.823 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.823 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:26:46.824 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.702423+0000", 2024-04-20T17:26:46.824 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.289e-06 2024-04-20T17:26:46.824 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.824 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.824 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:26:46.824 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.702428+0000", 2024-04-20T17:26:46.824 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9520000000000002e-06 2024-04-20T17:26:46.824 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.824 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.824 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:26:46.824 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.714772+0000", 2024-04-20T17:26:46.824 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.012343597 2024-04-20T17:26:46.824 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.824 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.825 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:26:46.825 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.714787+0000", 2024-04-20T17:26:46.825 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4874e-05 2024-04-20T17:26:46.825 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.825 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.825 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:26:46.825 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.755938+0000", 2024-04-20T17:26:46.825 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.041150682000000001 2024-04-20T17:26:46.825 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.825 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.825 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:26:46.825 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.755978+0000", 2024-04-20T17:26:46.825 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0083999999999998e-05 2024-04-20T17:26:46.825 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.826 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:26:46.826 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.826 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.826 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.826 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:120120 3.f e384/368 3:fc235aad:::benchmark_data_smithi171_110765_object120119:head v 384'8998, mlcod=384'8995)", 2024-04-20T17:26:46.826 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:42.708550+0000", 2024-04-20T17:26:46.826 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 304.07863988000003, 2024-04-20T17:26:46.826 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.047461543000000002, 2024-04-20T17:26:46.826 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:26:46.826 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:26:46.826 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:46.826 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:26:46.826 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.826 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:26:46.827 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.708550+0000", 2024-04-20T17:26:46.827 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.827 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.827 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.827 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:26:46.827 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.708550+0000", 2024-04-20T17:26:46.827 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.827 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.827 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.827 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:26:46.827 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.708553+0000", 2024-04-20T17:26:46.827 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.491e-06 2024-04-20T17:26:46.827 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.828 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.828 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:26:46.828 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.708582+0000", 2024-04-20T17:26:46.828 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9521000000000001e-05 2024-04-20T17:26:46.828 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.828 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.828 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:26:46.829 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.708600+0000", 2024-04-20T17:26:46.829 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7703e-05 2024-04-20T17:26:46.829 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.829 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.829 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:26:46.829 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.708609+0000", 2024-04-20T17:26:46.829 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6899999999999998e-06 2024-04-20T17:26:46.829 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.830 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.830 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:26:46.830 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.719516+0000", 2024-04-20T17:26:46.830 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010907005000000001 2024-04-20T17:26:46.830 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.830 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.830 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:26:46.831 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.719540+0000", 2024-04-20T17:26:46.831 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4678e-05 2024-04-20T17:26:46.831 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.831 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.831 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:26:46.831 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.755983+0000", 2024-04-20T17:26:46.832 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036442863999999998 2024-04-20T17:26:46.832 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.832 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.832 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:26:46.832 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.756012+0000", 2024-04-20T17:26:46.832 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8591000000000001e-05 2024-04-20T17:26:46.832 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.833 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:26:46.833 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.833 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.833 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.833 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.25802.0:120133 3.f e384/368 3:f541769c:::benchmark_data_smithi171_110765_object120132:head v 384'8999, mlcod=384'8995)", 2024-04-20T17:26:46.834 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:42.711517+0000", 2024-04-20T17:26:46.834 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 304.07567338400003, 2024-04-20T17:26:46.834 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044524068, 2024-04-20T17:26:46.834 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:26:46.834 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:26:46.834 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:46.834 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:26:46.835 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.835 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:26:46.835 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.711517+0000", 2024-04-20T17:26:46.835 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.835 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.835 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.835 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:26:46.836 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.711517+0000", 2024-04-20T17:26:46.836 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.836 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.836 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.836 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:26:46.836 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.711519+0000", 2024-04-20T17:26:46.837 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5550000000000001e-06 2024-04-20T17:26:46.837 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.837 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.837 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:26:46.837 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.711547+0000", 2024-04-20T17:26:46.838 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7535999999999999e-05 2024-04-20T17:26:46.838 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.838 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.838 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:26:46.838 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.711563+0000", 2024-04-20T17:26:46.838 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.624e-05 2024-04-20T17:26:46.839 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.839 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.839 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:26:46.839 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.711571+0000", 2024-04-20T17:26:46.839 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2130000000000008e-06 2024-04-20T17:26:46.839 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.839 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.839 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:26:46.839 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.724096+0000", 2024-04-20T17:26:46.839 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.012524926 2024-04-20T17:26:46.839 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.839 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.840 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:26:46.840 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.724120+0000", 2024-04-20T17:26:46.840 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3462e-05 2024-04-20T17:26:46.840 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.840 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.840 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:26:46.840 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.756014+0000", 2024-04-20T17:26:46.840 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.031894543999999997 2024-04-20T17:26:46.840 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.840 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.840 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:26:46.840 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:42.756041+0000", 2024-04-20T17:26:46.840 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6591999999999999e-05 2024-04-20T17:26:46.840 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.841 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:26:46.841 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.841 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.841 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.841 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:231538 3.9 3:96e48ba5:::benchmark_data_smithi171_110765_object88323:head [delete] snapc 0=[] ondisk+write+known_if_redirected e409)", 2024-04-20T17:26:46.841 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:16.289746+0000", 2024-04-20T17:26:46.841 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 210.49744391600001, 2024-04-20T17:26:46.841 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.050879008000000003, 2024-04-20T17:26:46.841 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:26:46.841 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:26:46.841 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:46.841 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:26:46.841 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:26:46.841 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:26:46.841 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 231538 2024-04-20T17:26:46.842 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.842 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:26:46.842 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.842 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:26:46.842 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.289746+0000", 2024-04-20T17:26:46.842 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.842 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.842 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.842 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:26:46.842 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.289746+0000", 2024-04-20T17:26:46.842 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.842 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.842 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.843 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:26:46.843 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.289748+0000", 2024-04-20T17:26:46.843 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.361e-06 2024-04-20T17:26:46.843 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.843 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.843 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:26:46.843 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.289752+0000", 2024-04-20T17:26:46.843 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8149999999999996e-06 2024-04-20T17:26:46.843 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.843 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.843 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:26:46.843 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.289762+0000", 2024-04-20T17:26:46.843 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6150000000000003e-06 2024-04-20T17:26:46.843 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.844 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.844 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:26:46.844 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.289768+0000", 2024-04-20T17:26:46.844 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5759999999999996e-06 2024-04-20T17:26:46.844 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.844 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.844 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:26:46.844 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.293676+0000", 2024-04-20T17:26:46.844 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0039085839999999997 2024-04-20T17:26:46.844 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.844 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.844 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:26:46.844 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.293791+0000", 2024-04-20T17:26:46.844 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000115176 2024-04-20T17:26:46.845 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.845 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.845 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:26:46.845 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.293904+0000", 2024-04-20T17:26:46.845 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000113126 2024-04-20T17:26:46.845 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.845 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.845 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:26:46.845 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.304147+0000", 2024-04-20T17:26:46.845 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010242922 2024-04-20T17:26:46.845 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.845 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.845 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:46.845 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.340523+0000", 2024-04-20T17:26:46.845 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036375788999999999 2024-04-20T17:26:46.846 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.846 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.846 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:26:46.846 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.340578+0000", 2024-04-20T17:26:46.846 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4753999999999998e-05 2024-04-20T17:26:46.846 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.846 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.846 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:26:46.846 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.340625+0000", 2024-04-20T17:26:46.846 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7290000000000003e-05 2024-04-20T17:26:46.846 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.846 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:26:46.846 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.847 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.847 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.847 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:231545 3.9 3:9aa16b0e:::benchmark_data_smithi171_110765_object88330:head [delete] snapc 0=[] ondisk+write+known_if_redirected e409)", 2024-04-20T17:26:46.847 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:16.295242+0000", 2024-04-20T17:26:46.847 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 210.491948057, 2024-04-20T17:26:46.847 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.045491156999999997, 2024-04-20T17:26:46.847 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:26:46.847 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:26:46.847 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:46.847 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:26:46.847 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:26:46.847 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:26:46.847 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 231545 2024-04-20T17:26:46.847 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.848 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:26:46.848 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.848 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:26:46.848 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.295242+0000", 2024-04-20T17:26:46.848 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.848 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.848 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.848 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:26:46.848 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.295242+0000", 2024-04-20T17:26:46.848 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.848 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.848 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.848 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:26:46.848 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.295244+0000", 2024-04-20T17:26:46.848 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1730000000000002e-06 2024-04-20T17:26:46.849 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.849 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.849 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:26:46.849 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.295251+0000", 2024-04-20T17:26:46.849 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0929999999999997e-06 2024-04-20T17:26:46.849 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.849 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.849 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:26:46.849 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.295263+0000", 2024-04-20T17:26:46.850 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1837e-05 2024-04-20T17:26:46.850 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.850 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.850 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:26:46.850 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.295271+0000", 2024-04-20T17:26:46.850 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7780000000000007e-06 2024-04-20T17:26:46.850 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.850 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.850 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:26:46.850 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.298979+0000", 2024-04-20T17:26:46.850 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0037076349999999999 2024-04-20T17:26:46.850 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.850 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.851 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:26:46.851 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.299088+0000", 2024-04-20T17:26:46.851 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000109747 2024-04-20T17:26:46.851 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.851 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.851 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:26:46.851 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.299192+0000", 2024-04-20T17:26:46.851 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010357 2024-04-20T17:26:46.851 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.851 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.851 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:26:46.852 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.304200+0000", 2024-04-20T17:26:46.852 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0050079340000000003 2024-04-20T17:26:46.852 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.852 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.852 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:46.852 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.340663+0000", 2024-04-20T17:26:46.852 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036463348999999999 2024-04-20T17:26:46.852 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.852 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.852 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:26:46.852 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.340702+0000", 2024-04-20T17:26:46.852 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8930000000000002e-05 2024-04-20T17:26:46.852 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.852 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.852 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:26:46.853 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:16.340733+0000", 2024-04-20T17:26:46.853 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1111000000000001e-05 2024-04-20T17:26:46.853 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.853 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:26:46.853 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.853 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.853 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.853 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:233189 3.9 3:93526fd3:::benchmark_data_smithi171_110765_object89974:head [delete] snapc 0=[] ondisk+write+known_if_redirected e410)", 2024-04-20T17:26:46.853 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:17.715749+0000", 2024-04-20T17:26:46.853 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 209.071441343, 2024-04-20T17:26:46.853 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.038314657000000002, 2024-04-20T17:26:46.853 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:26:46.854 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:26:46.854 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:46.854 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:26:46.854 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:26:46.854 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:26:46.854 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233189 2024-04-20T17:26:46.854 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.854 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:26:46.854 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.854 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:26:46.854 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.715749+0000", 2024-04-20T17:26:46.854 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.854 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.855 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.855 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:26:46.855 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.715749+0000", 2024-04-20T17:26:46.855 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.855 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.855 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.855 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:26:46.855 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.715751+0000", 2024-04-20T17:26:46.855 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1500000000000002e-06 2024-04-20T17:26:46.855 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.855 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.855 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:26:46.855 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.715757+0000", 2024-04-20T17:26:46.855 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9020000000000004e-06 2024-04-20T17:26:46.856 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.856 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.856 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:26:46.856 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.715766+0000", 2024-04-20T17:26:46.856 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3570000000000004e-06 2024-04-20T17:26:46.856 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.856 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.856 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:26:46.856 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.715773+0000", 2024-04-20T17:26:46.856 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7320000000000001e-06 2024-04-20T17:26:46.856 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.856 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.856 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:26:46.857 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.724072+0000", 2024-04-20T17:26:46.857 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0082987919999999993 2024-04-20T17:26:46.857 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.857 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.857 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:26:46.857 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.724178+0000", 2024-04-20T17:26:46.857 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000106456 2024-04-20T17:26:46.857 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.857 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.857 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:26:46.857 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.724298+0000", 2024-04-20T17:26:46.857 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000119401 2024-04-20T17:26:46.857 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.857 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.857 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:26:46.858 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.744272+0000", 2024-04-20T17:26:46.858 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019974582000000001 2024-04-20T17:26:46.858 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.858 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.858 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:46.858 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.753944+0000", 2024-04-20T17:26:46.858 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0096715810000000003 2024-04-20T17:26:46.858 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.858 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.858 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:26:46.858 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.753999+0000", 2024-04-20T17:26:46.858 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5336e-05 2024-04-20T17:26:46.858 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.859 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.859 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:26:46.859 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:17.754063+0000", 2024-04-20T17:26:46.859 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4368000000000004e-05 2024-04-20T17:26:46.859 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.859 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:26:46.859 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.859 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.859 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.859 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:242591 3.9 3:947730f9:::benchmark_data_smithi171_110765_object99376:head [delete] snapc 0=[] ondisk+write+known_if_redirected e416)", 2024-04-20T17:26:46.859 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:28.605493+0000", 2024-04-20T17:26:46.859 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 198.1816972, 2024-04-20T17:26:46.859 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043905926999999997, 2024-04-20T17:26:46.860 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:26:46.860 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:26:46.860 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:46.860 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:26:46.860 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:26:46.860 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:26:46.860 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 242591 2024-04-20T17:26:46.860 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.860 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:26:46.860 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.860 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:26:46.860 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.605493+0000", 2024-04-20T17:26:46.861 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.861 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.861 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.861 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:26:46.861 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.605493+0000", 2024-04-20T17:26:46.861 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.861 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.861 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.861 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:26:46.861 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.605495+0000", 2024-04-20T17:26:46.861 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3149999999999999e-06 2024-04-20T17:26:46.861 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.861 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.861 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:26:46.862 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.605502+0000", 2024-04-20T17:26:46.862 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9519999999999997e-06 2024-04-20T17:26:46.862 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.862 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.862 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:26:46.862 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.637474+0000", 2024-04-20T17:26:46.862 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.031971989999999999 2024-04-20T17:26:46.862 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.862 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.862 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:26:46.862 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.637490+0000", 2024-04-20T17:26:46.862 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5370999999999999e-05 2024-04-20T17:26:46.862 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.863 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.863 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:26:46.863 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.637664+0000", 2024-04-20T17:26:46.863 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000174346 2024-04-20T17:26:46.863 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.863 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.863 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:26:46.863 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.638254+0000", 2024-04-20T17:26:46.863 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00059011700000000003 2024-04-20T17:26:46.863 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.863 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.863 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:26:46.864 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.638359+0000", 2024-04-20T17:26:46.864 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010539099999999999 2024-04-20T17:26:46.864 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.864 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.864 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:26:46.864 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.648280+0000", 2024-04-20T17:26:46.864 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0099209239999999994 2024-04-20T17:26:46.864 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.864 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.865 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:46.865 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.649297+0000", 2024-04-20T17:26:46.865 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00101651 2024-04-20T17:26:46.865 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.865 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.865 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:26:46.865 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.649349+0000", 2024-04-20T17:26:46.866 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2379000000000001e-05 2024-04-20T17:26:46.866 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.866 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.866 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:26:46.866 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.649399+0000", 2024-04-20T17:26:46.866 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9632000000000002e-05 2024-04-20T17:26:46.866 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.866 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:26:46.867 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.867 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.867 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.867 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:242621 3.f 3:f6fd0383:::benchmark_data_smithi171_110765_object99406:head [delete] snapc 0=[] ondisk+write+known_if_redirected e416)", 2024-04-20T17:26:46.867 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:28.656335+0000", 2024-04-20T17:26:46.867 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 198.13085560600001, 2024-04-20T17:26:46.867 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.048888699000000001, 2024-04-20T17:26:46.867 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:26:46.867 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:26:46.867 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:46.867 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:26:46.867 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:26:46.868 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:26:46.868 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 242621 2024-04-20T17:26:46.868 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.868 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:26:46.868 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.868 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:26:46.869 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.656335+0000", 2024-04-20T17:26:46.869 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.869 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.869 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.869 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:26:46.869 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.656335+0000", 2024-04-20T17:26:46.869 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.869 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.870 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.870 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:26:46.870 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.656336+0000", 2024-04-20T17:26:46.870 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.916e-06 2024-04-20T17:26:46.870 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.870 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.870 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:26:46.871 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.656342+0000", 2024-04-20T17:26:46.871 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6330000000000001e-06 2024-04-20T17:26:46.871 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.871 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.871 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:26:46.871 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.656354+0000", 2024-04-20T17:26:46.871 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1531e-05 2024-04-20T17:26:46.871 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.871 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.871 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:26:46.871 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.656361+0000", 2024-04-20T17:26:46.871 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4950000000000002e-06 2024-04-20T17:26:46.872 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.872 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.872 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:26:46.872 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.656410+0000", 2024-04-20T17:26:46.872 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8446000000000003e-05 2024-04-20T17:26:46.872 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.872 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.872 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:26:46.872 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.656546+0000", 2024-04-20T17:26:46.872 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013692700000000001 2024-04-20T17:26:46.872 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.872 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.872 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7,11", 2024-04-20T17:26:46.872 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.656692+0000", 2024-04-20T17:26:46.873 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014555699999999999 2024-04-20T17:26:46.873 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.873 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.873 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:46.873 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.664251+0000", 2024-04-20T17:26:46.873 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0075594149999999999 2024-04-20T17:26:46.873 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.873 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.873 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:26:46.873 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.665032+0000", 2024-04-20T17:26:46.873 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00078029100000000001 2024-04-20T17:26:46.873 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.873 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.873 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:46.874 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.705032+0000", 2024-04-20T17:26:46.874 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.040000307999999998 2024-04-20T17:26:46.874 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.874 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.874 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:26:46.874 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.705098+0000", 2024-04-20T17:26:46.874 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5494999999999995e-05 2024-04-20T17:26:46.874 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.874 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.874 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:26:46.874 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.705223+0000", 2024-04-20T17:26:46.874 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000125685 2024-04-20T17:26:46.875 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.875 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:26:46.875 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.875 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.875 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.875 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:242624 3.f 3:f3655b59:::benchmark_data_smithi171_110765_object99409:head [delete] snapc 0=[] ondisk+write+known_if_redirected e416)", 2024-04-20T17:26:46.875 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:28.657404+0000", 2024-04-20T17:26:46.875 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 198.12978629700001, 2024-04-20T17:26:46.875 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.048069296999999997, 2024-04-20T17:26:46.875 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:26:46.875 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:26:46.875 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:46.875 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:26:46.875 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:26:46.875 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:26:46.876 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 242624 2024-04-20T17:26:46.876 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.876 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:26:46.876 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.876 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:26:46.876 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.657404+0000", 2024-04-20T17:26:46.876 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.876 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.876 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.876 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:26:46.876 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.657404+0000", 2024-04-20T17:26:46.876 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.876 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.877 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.877 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:26:46.877 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.657405+0000", 2024-04-20T17:26:46.877 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.049e-06 2024-04-20T17:26:46.877 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.877 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.877 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:26:46.877 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.657409+0000", 2024-04-20T17:26:46.877 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9489999999999998e-06 2024-04-20T17:26:46.877 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.877 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.877 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:26:46.878 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.657417+0000", 2024-04-20T17:26:46.878 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1510000000000004e-06 2024-04-20T17:26:46.878 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.878 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.878 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:26:46.878 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.657423+0000", 2024-04-20T17:26:46.878 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3380000000000001e-06 2024-04-20T17:26:46.878 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.879 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.879 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:26:46.879 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.660448+0000", 2024-04-20T17:26:46.879 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.003024592 2024-04-20T17:26:46.879 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.879 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.879 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:26:46.879 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.660576+0000", 2024-04-20T17:26:46.879 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000128434 2024-04-20T17:26:46.879 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.879 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.879 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7,11", 2024-04-20T17:26:46.879 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.660698+0000", 2024-04-20T17:26:46.879 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0001214 2024-04-20T17:26:46.880 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.880 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.880 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:26:46.880 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.665039+0000", 2024-04-20T17:26:46.880 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0043414960000000002 2024-04-20T17:26:46.880 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.880 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.880 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:46.880 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.665083+0000", 2024-04-20T17:26:46.880 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4014999999999998e-05 2024-04-20T17:26:46.880 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.880 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.880 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:46.880 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.705290+0000", 2024-04-20T17:26:46.881 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.040207159999999999 2024-04-20T17:26:46.881 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.881 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.881 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:26:46.881 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.705355+0000", 2024-04-20T17:26:46.881 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4677000000000004e-05 2024-04-20T17:26:46.881 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.881 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.881 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:26:46.881 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.705473+0000", 2024-04-20T17:26:46.881 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011803600000000001 2024-04-20T17:26:46.881 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.881 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:26:46.881 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.882 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.882 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.882 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:242791 3.9 3:9bda8523:::benchmark_data_smithi171_110765_object99576:head [delete] snapc 0=[] ondisk+write+known_if_redirected e416)", 2024-04-20T17:26:46.882 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:28.815327+0000", 2024-04-20T17:26:46.882 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 197.971863329, 2024-04-20T17:26:46.882 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.040144943000000002, 2024-04-20T17:26:46.882 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:26:46.882 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:26:46.882 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:46.882 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:26:46.882 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:26:46.882 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:26:46.882 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 242791 2024-04-20T17:26:46.883 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.883 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:26:46.883 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.883 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:26:46.883 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.815327+0000", 2024-04-20T17:26:46.883 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.883 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.883 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.883 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:26:46.883 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.815327+0000", 2024-04-20T17:26:46.884 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.884 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.884 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.884 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:26:46.884 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.815327+0000", 2024-04-20T17:26:46.884 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8400000000000004e-07 2024-04-20T17:26:46.884 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.885 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.885 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:26:46.885 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.815330+0000", 2024-04-20T17:26:46.885 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3190000000000002e-06 2024-04-20T17:26:46.885 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.885 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.885 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:26:46.885 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.815334+0000", 2024-04-20T17:26:46.885 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3329999999999998e-06 2024-04-20T17:26:46.886 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.886 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.886 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:26:46.886 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.815339+0000", 2024-04-20T17:26:46.886 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9060000000000001e-06 2024-04-20T17:26:46.886 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.886 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.886 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:26:46.886 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.824615+0000", 2024-04-20T17:26:46.887 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0092760970000000005 2024-04-20T17:26:46.887 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.887 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.887 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:26:46.887 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.824741+0000", 2024-04-20T17:26:46.887 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012594099999999999 2024-04-20T17:26:46.887 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.887 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.887 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:26:46.887 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.824872+0000", 2024-04-20T17:26:46.887 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000131304 2024-04-20T17:26:46.887 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.888 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.888 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:26:46.888 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.844032+0000", 2024-04-20T17:26:46.888 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019159733000000002 2024-04-20T17:26:46.888 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.888 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.888 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:46.888 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.855407+0000", 2024-04-20T17:26:46.888 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011375175 2024-04-20T17:26:46.888 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.888 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.888 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:26:46.889 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.855447+0000", 2024-04-20T17:26:46.889 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9552999999999999e-05 2024-04-20T17:26:46.889 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.889 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.889 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:26:46.889 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.855472+0000", 2024-04-20T17:26:46.889 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4998e-05 2024-04-20T17:26:46.889 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.889 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:26:46.889 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.889 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.889 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.889 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:242797 3.9 3:9a0ea06c:::benchmark_data_smithi171_110765_object99582:head [delete] snapc 0=[] ondisk+write+known_if_redirected e416)", 2024-04-20T17:26:46.890 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:28.817815+0000", 2024-04-20T17:26:46.890 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 197.969375482, 2024-04-20T17:26:46.890 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.037773329000000001, 2024-04-20T17:26:46.890 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:26:46.890 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:26:46.890 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:46.890 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:26:46.890 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:26:46.891 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:26:46.891 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 242797 2024-04-20T17:26:46.891 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.891 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:26:46.891 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.891 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:26:46.892 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.817815+0000", 2024-04-20T17:26:46.892 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.892 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.892 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.892 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:26:46.892 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.817815+0000", 2024-04-20T17:26:46.893 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.893 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.893 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.893 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:26:46.893 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.817816+0000", 2024-04-20T17:26:46.893 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.714e-06 2024-04-20T17:26:46.893 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.894 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.894 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:26:46.894 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.817822+0000", 2024-04-20T17:26:46.894 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1359999999999996e-06 2024-04-20T17:26:46.894 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.894 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.895 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:26:46.895 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.817836+0000", 2024-04-20T17:26:46.895 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4235e-05 2024-04-20T17:26:46.895 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.895 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.895 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:26:46.896 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.817843+0000", 2024-04-20T17:26:46.896 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1539999999999996e-06 2024-04-20T17:26:46.896 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.896 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.896 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:26:46.896 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.829039+0000", 2024-04-20T17:26:46.896 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011196284000000001 2024-04-20T17:26:46.897 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.897 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.897 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:26:46.897 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.829152+0000", 2024-04-20T17:26:46.897 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000112619 2024-04-20T17:26:46.897 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.898 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.898 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:26:46.898 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.829265+0000", 2024-04-20T17:26:46.898 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011323399999999999 2024-04-20T17:26:46.898 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.898 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.899 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:26:46.899 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.848239+0000", 2024-04-20T17:26:46.899 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018973898999999999 2024-04-20T17:26:46.899 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.899 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.899 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:46.900 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.855514+0000", 2024-04-20T17:26:46.900 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0072750510000000003 2024-04-20T17:26:46.900 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.900 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.901 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:26:46.901 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.855561+0000", 2024-04-20T17:26:46.901 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6826000000000002e-05 2024-04-20T17:26:46.901 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.901 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.901 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:26:46.902 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:28.855588+0000", 2024-04-20T17:26:46.902 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7177000000000001e-05 2024-04-20T17:26:46.902 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.902 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:26:46.902 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.902 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.902 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.902 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:243458 3.f 3:fecbe48e:::benchmark_data_smithi171_110765_object100243:head [delete] snapc 0=[] ondisk+write+known_if_redirected e416)", 2024-04-20T17:26:46.903 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:29.382642+0000", 2024-04-20T17:26:46.903 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 197.40454814099999, 2024-04-20T17:26:46.903 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.10183466400000001, 2024-04-20T17:26:46.903 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:26:46.903 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:26:46.903 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:46.903 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:26:46.903 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:26:46.903 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:26:46.903 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 243458 2024-04-20T17:26:46.903 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.903 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:26:46.903 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.903 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:26:46.903 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.382642+0000", 2024-04-20T17:26:46.904 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.904 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.904 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.904 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:26:46.904 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.382642+0000", 2024-04-20T17:26:46.904 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.904 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.904 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.904 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:26:46.904 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.382644+0000", 2024-04-20T17:26:46.904 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2019999999999998e-06 2024-04-20T17:26:46.904 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.904 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.904 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:26:46.905 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.382651+0000", 2024-04-20T17:26:46.905 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6669999999999999e-06 2024-04-20T17:26:46.905 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.905 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.905 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:26:46.905 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.382662+0000", 2024-04-20T17:26:46.905 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1272e-05 2024-04-20T17:26:46.905 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.905 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.905 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:26:46.905 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.382670+0000", 2024-04-20T17:26:46.905 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0059999999999992e-06 2024-04-20T17:26:46.905 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.905 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.905 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:26:46.906 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.382728+0000", 2024-04-20T17:26:46.906 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7772000000000002e-05 2024-04-20T17:26:46.906 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.906 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.906 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:26:46.906 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.382809+0000", 2024-04-20T17:26:46.906 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0984999999999998e-05 2024-04-20T17:26:46.907 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.907 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.907 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7,11", 2024-04-20T17:26:46.907 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.382921+0000", 2024-04-20T17:26:46.908 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011195 2024-04-20T17:26:46.908 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.908 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.908 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:46.908 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.387691+0000", 2024-04-20T17:26:46.908 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0047698660000000002 2024-04-20T17:26:46.909 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.909 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.909 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:26:46.909 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.388778+0000", 2024-04-20T17:26:46.909 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001087342 2024-04-20T17:26:46.909 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.909 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.909 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:46.910 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.484307+0000", 2024-04-20T17:26:46.910 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.095529426000000001 2024-04-20T17:26:46.910 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.910 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.910 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:26:46.910 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.484371+0000", 2024-04-20T17:26:46.910 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3324999999999994e-05 2024-04-20T17:26:46.910 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.910 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.910 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:26:46.910 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.484477+0000", 2024-04-20T17:26:46.910 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000105851 2024-04-20T17:26:46.910 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.910 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:26:46.911 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.911 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.911 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.911 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:243465 3.f 3:fa13a8ae:::benchmark_data_smithi171_110765_object100250:head [delete] snapc 0=[] ondisk+write+known_if_redirected e416)", 2024-04-20T17:26:46.911 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:29.388454+0000", 2024-04-20T17:26:46.911 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 197.398736211, 2024-04-20T17:26:46.911 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.096277862000000006, 2024-04-20T17:26:46.911 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:26:46.911 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:26:46.911 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:46.911 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:26:46.912 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:26:46.912 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:26:46.912 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 243465 2024-04-20T17:26:46.912 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.912 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:26:46.912 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.912 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:26:46.913 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.388454+0000", 2024-04-20T17:26:46.913 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.913 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.913 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.913 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:26:46.913 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.388454+0000", 2024-04-20T17:26:46.913 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.913 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.913 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.913 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:26:46.914 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.388457+0000", 2024-04-20T17:26:46.914 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6989999999999999e-06 2024-04-20T17:26:46.914 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.914 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.914 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:26:46.914 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.388463+0000", 2024-04-20T17:26:46.915 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9889999999999997e-06 2024-04-20T17:26:46.915 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.915 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.915 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:26:46.915 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.388473+0000", 2024-04-20T17:26:46.915 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0841e-05 2024-04-20T17:26:46.916 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.916 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.916 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:26:46.916 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.388480+0000", 2024-04-20T17:26:46.916 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6239999999999996e-06 2024-04-20T17:26:46.916 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.916 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.916 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:26:46.916 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.388798+0000", 2024-04-20T17:26:46.916 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00031772700000000002 2024-04-20T17:26:46.916 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.916 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.917 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:26:46.917 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.388875+0000", 2024-04-20T17:26:46.917 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.737e-05 2024-04-20T17:26:46.917 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.917 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.917 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7,11", 2024-04-20T17:26:46.917 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.388949+0000", 2024-04-20T17:26:46.917 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3825999999999994e-05 2024-04-20T17:26:46.917 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.917 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.917 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:46.917 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.392749+0000", 2024-04-20T17:26:46.917 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0037998509999999999 2024-04-20T17:26:46.917 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.917 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.918 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:26:46.918 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.392991+0000", 2024-04-20T17:26:46.918 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000242513 2024-04-20T17:26:46.918 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.918 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.918 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:46.918 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.484566+0000", 2024-04-20T17:26:46.918 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.091574401999999999 2024-04-20T17:26:46.918 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.918 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.918 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:26:46.918 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.484624+0000", 2024-04-20T17:26:46.918 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8301999999999998e-05 2024-04-20T17:26:46.919 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.919 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.919 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:26:46.919 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.484732+0000", 2024-04-20T17:26:46.919 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000107718 2024-04-20T17:26:46.919 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.919 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:26:46.919 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.919 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.919 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.919 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "MOSDPGPushReply(3.f 416/412 [PushReplyOp(3:f88e7e1e:::benchmark_data_smithi171_110765_object139150:head)])", 2024-04-20T17:26:46.919 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:29.395683+0000", 2024-04-20T17:26:46.919 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 197.39150716099999, 2024-04-20T17:26:46.920 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.073417189999999993, 2024-04-20T17:26:46.920 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:26:46.920 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:26:46.920 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "reached pg", 2024-04-20T17:26:46.920 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:26:46.921 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.921 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:26:46.921 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.395683+0000", 2024-04-20T17:26:46.921 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.921 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.921 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.922 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:26:46.922 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.395683+0000", 2024-04-20T17:26:46.922 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.922 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.922 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.922 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:26:46.923 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.395683+0000", 2024-04-20T17:26:46.923 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4700000000000002e-07 2024-04-20T17:26:46.923 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.923 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.923 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:26:46.923 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.395686+0000", 2024-04-20T17:26:46.923 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.542e-06 2024-04-20T17:26:46.923 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.923 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.923 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:26:46.924 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.395690+0000", 2024-04-20T17:26:46.924 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1470000000000003e-06 2024-04-20T17:26:46.924 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.924 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.924 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:26:46.924 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.395695+0000", 2024-04-20T17:26:46.925 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8130000000000003e-06 2024-04-20T17:26:46.925 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.925 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.925 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:26:46.925 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.395748+0000", 2024-04-20T17:26:46.925 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2976000000000002e-05 2024-04-20T17:26:46.926 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.926 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.926 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:26:46.926 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.469100+0000", 2024-04-20T17:26:46.926 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.073352365000000003 2024-04-20T17:26:46.927 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.927 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:26:46.927 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.927 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.927 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.927 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:243476 3.9 3:975d1f76:::benchmark_data_smithi171_110765_object100261:head [delete] snapc 0=[] ondisk+write+known_if_redirected e416)", 2024-04-20T17:26:46.928 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:23:29.403586+0000", 2024-04-20T17:26:46.928 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 197.383604182, 2024-04-20T17:26:46.928 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.085046994000000001, 2024-04-20T17:26:46.928 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:26:46.928 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:26:46.928 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:26:46.929 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:26:46.929 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:26:46.929 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:26:46.929 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 243476 2024-04-20T17:26:46.929 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.929 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:26:46.929 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.930 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:26:46.930 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.403586+0000", 2024-04-20T17:26:46.930 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.930 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.930 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.930 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:26:46.931 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.403586+0000", 2024-04-20T17:26:46.931 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:46.931 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.931 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.931 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:26:46.931 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.403589+0000", 2024-04-20T17:26:46.932 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6280000000000001e-06 2024-04-20T17:26:46.932 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.932 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.932 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:26:46.932 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.403596+0000", 2024-04-20T17:26:46.932 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2099999999999996e-06 2024-04-20T17:26:46.932 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.932 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.933 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:26:46.933 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.468881+0000", 2024-04-20T17:26:46.933 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.065285683999999997 2024-04-20T17:26:46.933 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.933 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.933 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:26:46.933 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.468893+0000", 2024-04-20T17:26:46.933 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1759999999999999e-05 2024-04-20T17:26:46.933 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.933 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.933 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:26:46.933 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.468983+0000", 2024-04-20T17:26:46.933 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9797999999999996e-05 2024-04-20T17:26:46.933 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.934 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.934 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:26:46.934 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.469119+0000", 2024-04-20T17:26:46.934 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000136179 2024-04-20T17:26:46.934 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.934 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.934 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 11", 2024-04-20T17:26:46.934 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.469250+0000", 2024-04-20T17:26:46.934 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013118599999999999 2024-04-20T17:26:46.934 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.934 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.934 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:26:46.934 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.488522+0000", 2024-04-20T17:26:46.934 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019271782000000001 2024-04-20T17:26:46.934 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.935 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.935 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:26:46.935 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.488540+0000", 2024-04-20T17:26:46.935 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7527999999999999e-05 2024-04-20T17:26:46.935 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.935 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.935 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:26:46.935 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.488592+0000", 2024-04-20T17:26:46.935 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2274999999999997e-05 2024-04-20T17:26:46.935 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:46.935 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:46.935 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:26:46.935 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:23:29.488633+0000", 2024-04-20T17:26:46.935 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0964000000000003e-05 2024-04-20T17:26:46.936 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.936 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:26:46.936 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.936 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:46.936 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:26:46.936 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:26:47.199 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:26:47.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:47.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:26:47.312 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:26:47.348 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:47.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:47 smithi005 ceph-mon[39562]: pgmap v1303: 17 pgs: 17 active+clean; 8.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.37k op/s 2024-04-20T17:26:47.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:47 smithi106 ceph-mon[44423]: pgmap v1303: 17 pgs: 17 active+clean; 8.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.37k op/s 2024-04-20T17:26:47.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:47 smithi171 ceph-mon[44189]: pgmap v1303: 17 pgs: 17 active+clean; 8.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.37k op/s 2024-04-20T17:26:47.599 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:26:47.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:47.608 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:26:47.642 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-20T17:26:47.709 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:26:47.738 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:47.761 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:26:47.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:47.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:26:47.866 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:26:47.896 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:47.920 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:26:47.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:47.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:26:48.024 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:26:48.057 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:48.062 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:26:49.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:49 smithi171 ceph-mon[44189]: pgmap v1304: 17 pgs: 17 active+clean; 8.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-04-20T17:26:49.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:49 smithi005 ceph-mon[39562]: pgmap v1304: 17 pgs: 17 active+clean; 8.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-04-20T17:26:49.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:49 smithi106 ceph-mon[44423]: pgmap v1304: 17 pgs: 17 active+clean; 8.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-04-20T17:26:50.561 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:26:50.561 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:26:50.561 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:50.561 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.26074.0:209660 4.d e493/445 4:b6c3a4cc:::benchmark_data_smithi171_122772_object39468:head v 493'12983, mlcod=493'12981)", 2024-04-20T17:26:50.562 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:50.560159+0000", 2024-04-20T17:26:50.562 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.00028791100000000002, 2024-04-20T17:26:50.562 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00032683899999999999, 2024-04-20T17:26:50.562 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:26:50.562 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:26:50.562 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:26:50.562 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:26:50.562 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:50.562 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:26:50.562 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:50.560159+0000", 2024-04-20T17:26:50.562 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:50.562 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:50.562 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:50.562 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:26:50.563 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:50.560159+0000", 2024-04-20T17:26:50.563 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:26:50.563 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:50.563 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:50.563 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:26:50.563 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:50.560161+0000", 2024-04-20T17:26:50.563 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6419999999999999e-06 2024-04-20T17:26:50.563 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:50.563 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:50.563 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:26:50.563 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:50.560169+0000", 2024-04-20T17:26:50.563 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1790000000000008e-06 2024-04-20T17:26:50.563 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:50.563 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:50.563 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:26:50.564 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:50.560182+0000", 2024-04-20T17:26:50.564 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2276000000000001e-05 2024-04-20T17:26:50.564 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:50.564 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:50.564 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:26:50.564 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:50.560189+0000", 2024-04-20T17:26:50.564 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5920000000000003e-06 2024-04-20T17:26:50.564 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:50.564 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:50.564 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:26:50.564 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:50.560259+0000", 2024-04-20T17:26:50.564 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0154999999999995e-05 2024-04-20T17:26:50.564 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:26:50.564 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:26:50.564 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:26:50.565 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:50.560282+0000", 2024-04-20T17:26:50.565 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3189000000000001e-05 2024-04-20T17:26:50.565 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:50.565 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:26:50.565 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:50.565 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:26:50.565 INFO:teuthology.orchestra.run.smithi171.stdout: ], 2024-04-20T17:26:50.565 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 1 2024-04-20T17:26:50.566 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:26:51.291 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:51 smithi171 ceph-mon[44189]: pgmap v1305: 17 pgs: 17 active+clean; 8.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-04-20T17:26:51.375 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-20T17:26:51.438 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:26:51.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:51.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:26:51.544 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:26:51.576 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:51 smithi106 ceph-mon[44423]: pgmap v1305: 17 pgs: 17 active+clean; 8.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-04-20T17:26:51.578 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:51.611 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:26:51.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:51.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:26:51.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:51 smithi005 ceph-mon[39562]: pgmap v1305: 17 pgs: 17 active+clean; 8.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-04-20T17:26:51.717 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:26:51.746 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:51.757 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:26:51.770 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:26:51.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:51.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:26:51.874 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:26:51.914 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:51.942 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:26:51.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:51.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:26:52.050 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:26:52.081 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:52.105 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:26:52.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:52.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:26:52.211 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:26:52.247 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:52.462 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:26:52.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:52.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:26:52.572 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:26:52.604 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:52.631 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:26:52.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:52.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:26:52.734 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:26:52.735 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:26:52.772 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:52.792 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:26:52.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:52.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:26:52.898 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:26:52.935 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:53.093 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:26:53.101 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:26:53.297 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:53 smithi005 ceph-mon[39562]: pgmap v1306: 17 pgs: 17 active+clean; 8.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s 2024-04-20T17:26:53.297 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:53 smithi005 ceph-mon[39562]: from='client.26227 v1:172.21.15.5:0/2720907515' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:26:53.491 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:26:53.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:53.494 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:26:53.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:53 smithi171 ceph-mon[44189]: pgmap v1306: 17 pgs: 17 active+clean; 8.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s 2024-04-20T17:26:53.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:53 smithi171 ceph-mon[44189]: from='client.26227 v1:172.21.15.5:0/2720907515' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:26:53.594 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:26:53.637 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:53.719 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:26:53.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:53.723 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:26:53.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:53 smithi106 ceph-mon[44423]: pgmap v1306: 17 pgs: 17 active+clean; 8.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s 2024-04-20T17:26:53.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:53 smithi106 ceph-mon[44423]: from='client.26227 v1:172.21.15.5:0/2720907515' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:26:53.823 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:26:54.070 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:54.196 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:26:54.278 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:26:54.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:54.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:26:54.287 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:26:54.287 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:26:54.287 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-04-20T17:26:54.287 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-04-20T17:26:54.287 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:26:54.385 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:26:54.422 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:54.580 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:26:54.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:54.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:54 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:26:54.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:54 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:26:54.585 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:26:54.686 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:26:54.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:54 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:26:54.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:54 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:26:54.714 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:54 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:26:54.714 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:54 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:26:54.716 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:54.743 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:26:54.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:54.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:26:54.790 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":1306,"stamp":"2024-04-20T17:26:52.230118+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8638825016,"num_objects":131812,"num_object_clones":0,"num_object_copies":263624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131812,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":208751,"num_write_kb":10895035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160816,"ondisk_log_size":160816,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":11,"kb":1124941824,"kb_used":19776988,"kb_used_data":17045880,"kb_used_omap":224,"kb_used_meta":2730463,"kb_avail":1105164836,"statfs":{"total":1151940427776,"available":1131688792064,"internally_reserved":0,"allocated":17454981120,"data_stored":17450574636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":230126,"internal_metadata":2795994386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,9,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-831127552,"num_objects":-12682,"num_object_clones":0,"num_object_copies":-25364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12682,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001929"},"pg_stats":[{"pgid":"4.f","version":"493'13214","reported_seq":13424,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-04-20T17:26:51.315880+0000","last_change":"2024-04-20T17:26:34.139365+0000","last_active":"2024-04-20T17:26:51.315880+0000","last_peered":"2024-04-20T17:26:51.315880+0000","last_clean":"2024-04-20T17:26:51.315880+0000","last_became_active":"2024-04-20T17:24:36.727388+0000","last_became_peered":"2024-04-20T17:24:36.727388+0000","last_unstale":"2024-04-20T17:26:51.315880+0000","last_undegraded":"2024-04-20T17:26:51.315880+0000","last_fullsized":"2024-04-20T17:26:51.315880+0000","mapping_epoch":445,"log_start":"455'3200","ondisk_log_start":"455'3200","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'11858","last_scrub_stamp":"2024-04-20T17:26:34.139195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:34.139195+0000","objects_scrubbed":9615,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:51.741322+0000","scrub_duration":2286,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535035904,"num_objects":8164,"num_object_clones":0,"num_object_copies":16328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13214,"num_write_kb":684096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"493'12958","reported_seq":13259,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-04-20T17:26:50.755528+0000","last_change":"2024-04-20T17:26:29.940846+0000","last_active":"2024-04-20T17:26:50.755528+0000","last_peered":"2024-04-20T17:26:50.755528+0000","last_clean":"2024-04-20T17:26:50.755528+0000","last_became_active":"2024-04-20T17:24:36.573304+0000","last_became_peered":"2024-04-20T17:24:36.573304+0000","last_unstale":"2024-04-20T17:26:50.755528+0000","last_undegraded":"2024-04-20T17:26:50.755528+0000","last_fullsized":"2024-04-20T17:26:50.755528+0000","mapping_epoch":445,"log_start":"454'2900","ondisk_log_start":"454'2900","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'11392","last_scrub_stamp":"2024-04-20T17:26:29.940585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:29.940585+0000","objects_scrubbed":9772,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:57.030101+0000","scrub_duration":1744,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":534249472,"num_objects":8152,"num_object_clones":0,"num_object_copies":16304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12958,"num_write_kb":675520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"493'13001","reported_seq":13204,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-04-20T17:26:50.750233+0000","last_change":"2024-04-20T17:25:55.471139+0000","last_active":"2024-04-20T17:26:50.750233+0000","last_peered":"2024-04-20T17:26:50.750233+0000","last_clean":"2024-04-20T17:26:50.750233+0000","last_became_active":"2024-04-20T17:24:36.573474+0000","last_became_peered":"2024-04-20T17:24:36.573474+0000","last_unstale":"2024-04-20T17:26:50.750233+0000","last_undegraded":"2024-04-20T17:26:50.750233+0000","last_fullsized":"2024-04-20T17:26:50.750233+0000","mapping_epoch":445,"log_start":"454'3000","ondisk_log_start":"454'3000","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'8994","last_scrub_stamp":"2024-04-20T17:25:55.470885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:25:55.470885+0000","objects_scrubbed":8940,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:01.834108+0000","scrub_duration":1381,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536477720,"num_objects":8187,"num_object_clones":0,"num_object_copies":16374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8187,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13001,"num_write_kb":677953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"493'13161","reported_seq":13369,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-04-20T17:26:50.715855+0000","last_change":"2024-04-20T17:26:02.737108+0000","last_active":"2024-04-20T17:26:50.715855+0000","last_peered":"2024-04-20T17:26:50.715855+0000","last_clean":"2024-04-20T17:26:50.715855+0000","last_became_active":"2024-04-20T17:24:36.572684+0000","last_became_peered":"2024-04-20T17:24:36.572684+0000","last_unstale":"2024-04-20T17:26:50.715855+0000","last_undegraded":"2024-04-20T17:26:50.715855+0000","last_fullsized":"2024-04-20T17:26:50.715855+0000","mapping_epoch":445,"log_start":"455'3100","ondisk_log_start":"455'3100","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'9546","last_scrub_stamp":"2024-04-20T17:26:02.736996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:02.736996+0000","objects_scrubbed":9482,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:03.229190+0000","scrub_duration":1696,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":534577152,"num_objects":8157,"num_object_clones":0,"num_object_copies":16314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13161,"num_write_kb":682176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"493'13163","reported_seq":13371,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-04-20T17:26:49.305442+0000","last_change":"2024-04-20T17:26:03.163791+0000","last_active":"2024-04-20T17:26:49.305442+0000","last_peered":"2024-04-20T17:26:49.305442+0000","last_clean":"2024-04-20T17:26:49.305442+0000","last_became_active":"2024-04-20T17:24:36.726966+0000","last_became_peered":"2024-04-20T17:24:36.726966+0000","last_unstale":"2024-04-20T17:26:49.305442+0000","last_undegraded":"2024-04-20T17:26:49.305442+0000","last_fullsized":"2024-04-20T17:26:49.305442+0000","mapping_epoch":445,"log_start":"454'3100","ondisk_log_start":"454'3100","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'9675","last_scrub_stamp":"2024-04-20T17:26:03.163686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:03.163686+0000","objects_scrubbed":9632,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:06.504359+0000","scrub_duration":1737,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":546897920,"num_objects":8345,"num_object_clones":0,"num_object_copies":16690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13163,"num_write_kb":688256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"493'12770","reported_seq":12965,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-04-20T17:26:47.490482+0000","last_change":"2024-04-20T17:25:47.229572+0000","last_active":"2024-04-20T17:26:47.490482+0000","last_peered":"2024-04-20T17:26:47.490482+0000","last_clean":"2024-04-20T17:26:47.490482+0000","last_became_active":"2024-04-20T17:24:36.727448+0000","last_became_peered":"2024-04-20T17:24:36.727448+0000","last_unstale":"2024-04-20T17:26:47.490482+0000","last_undegraded":"2024-04-20T17:26:47.490482+0000","last_fullsized":"2024-04-20T17:26:47.490482+0000","mapping_epoch":445,"log_start":"454'2700","ondisk_log_start":"454'2700","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'8313","last_scrub_stamp":"2024-04-20T17:25:47.229419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:25:47.229419+0000","objects_scrubbed":8261,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:15.544542+0000","scrub_duration":1387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":547225600,"num_objects":8350,"num_object_clones":0,"num_object_copies":16700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12770,"num_write_kb":675840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"493'12933","reported_seq":13122,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-04-20T17:26:48.174049+0000","last_change":"2024-04-20T17:25:42.475029+0000","last_active":"2024-04-20T17:26:48.174049+0000","last_peered":"2024-04-20T17:26:48.174049+0000","last_clean":"2024-04-20T17:26:48.174049+0000","last_became_active":"2024-04-20T17:24:36.727451+0000","last_became_peered":"2024-04-20T17:24:36.727451+0000","last_unstale":"2024-04-20T17:26:48.174049+0000","last_undegraded":"2024-04-20T17:26:48.174049+0000","last_fullsized":"2024-04-20T17:26:48.174049+0000","mapping_epoch":445,"log_start":"454'2900","ondisk_log_start":"454'2900","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"471'7991","last_scrub_stamp":"2024-04-20T17:25:42.474753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:25:42.474753+0000","objects_scrubbed":7878,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:01.483744+0000","scrub_duration":2083,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":546373632,"num_objects":8337,"num_object_clones":0,"num_object_copies":16674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12933,"num_write_kb":680640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"493'12839","reported_seq":13032,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-04-20T17:26:48.197254+0000","last_change":"2024-04-20T17:25:43.026635+0000","last_active":"2024-04-20T17:26:48.197254+0000","last_peered":"2024-04-20T17:26:48.197254+0000","last_clean":"2024-04-20T17:26:48.197254+0000","last_became_active":"2024-04-20T17:24:36.895302+0000","last_became_peered":"2024-04-20T17:24:36.895302+0000","last_unstale":"2024-04-20T17:26:48.197254+0000","last_undegraded":"2024-04-20T17:26:48.197254+0000","last_fullsized":"2024-04-20T17:26:48.197254+0000","mapping_epoch":445,"log_start":"454'2800","ondisk_log_start":"454'2800","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"471'7970","last_scrub_stamp":"2024-04-20T17:25:43.026438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:25:43.026438+0000","objects_scrubbed":7867,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:26:50.674000+0000","scrub_duration":1624,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":547684352,"num_objects":8357,"num_object_clones":0,"num_object_copies":16714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12839,"num_write_kb":678272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"493'13031","reported_seq":13275,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-04-20T17:26:49.075274+0000","last_change":"2024-04-20T17:26:28.991603+0000","last_active":"2024-04-20T17:26:49.075274+0000","last_peered":"2024-04-20T17:26:49.075274+0000","last_clean":"2024-04-20T17:26:49.075274+0000","last_became_active":"2024-04-20T17:24:36.727119+0000","last_became_peered":"2024-04-20T17:24:36.727119+0000","last_unstale":"2024-04-20T17:26:49.075274+0000","last_undegraded":"2024-04-20T17:26:49.075274+0000","last_fullsized":"2024-04-20T17:26:49.075274+0000","mapping_epoch":445,"log_start":"454'3000","ondisk_log_start":"454'3000","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"487'11515","last_scrub_stamp":"2024-04-20T17:26:28.991420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:28.991420+0000","objects_scrubbed":9918,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:47.134221+0000","scrub_duration":2074,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":545718272,"num_objects":8327,"num_object_clones":0,"num_object_copies":16654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13031,"num_write_kb":683456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"493'13431","reported_seq":13640,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-04-20T17:26:51.032458+0000","last_change":"2024-04-20T17:26:00.309772+0000","last_active":"2024-04-20T17:26:51.032458+0000","last_peered":"2024-04-20T17:26:51.032458+0000","last_clean":"2024-04-20T17:26:51.032458+0000","last_became_active":"2024-04-20T17:24:36.727225+0000","last_became_peered":"2024-04-20T17:24:36.727225+0000","last_unstale":"2024-04-20T17:26:51.032458+0000","last_undegraded":"2024-04-20T17:26:51.032458+0000","last_fullsized":"2024-04-20T17:26:51.032458+0000","mapping_epoch":445,"log_start":"455'3400","ondisk_log_start":"455'3400","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'9588","last_scrub_stamp":"2024-04-20T17:26:00.309610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:00.309610+0000","objects_scrubbed":9527,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:02.777067+0000","scrub_duration":1737,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537198592,"num_objects":8197,"num_object_clones":0,"num_object_copies":16394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13431,"num_write_kb":692096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"493'12939","reported_seq":13144,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-04-20T17:26:50.981482+0000","last_change":"2024-04-20T17:26:00.587646+0000","last_active":"2024-04-20T17:26:50.981482+0000","last_peered":"2024-04-20T17:26:50.981482+0000","last_clean":"2024-04-20T17:26:50.981482+0000","last_became_active":"2024-04-20T17:24:36.727028+0000","last_became_peered":"2024-04-20T17:24:36.727028+0000","last_unstale":"2024-04-20T17:26:50.981482+0000","last_undegraded":"2024-04-20T17:26:50.981482+0000","last_fullsized":"2024-04-20T17:26:50.981482+0000","mapping_epoch":445,"log_start":"454'2900","ondisk_log_start":"454'2900","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'9204","last_scrub_stamp":"2024-04-20T17:26:00.587445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:00.587445+0000","objects_scrubbed":9139,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:00.762089+0000","scrub_duration":1490,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":523567104,"num_objects":7989,"num_object_clones":0,"num_object_copies":15978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12939,"num_write_kb":669696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"493'12860","reported_seq":13068,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-04-20T17:26:48.172027+0000","last_change":"2024-04-20T17:26:32.955835+0000","last_active":"2024-04-20T17:26:48.172027+0000","last_peered":"2024-04-20T17:26:48.172027+0000","last_clean":"2024-04-20T17:26:48.172027+0000","last_became_active":"2024-04-20T17:24:36.895348+0000","last_became_peered":"2024-04-20T17:24:36.895348+0000","last_unstale":"2024-04-20T17:26:48.172027+0000","last_undegraded":"2024-04-20T17:26:48.172027+0000","last_fullsized":"2024-04-20T17:26:48.172027+0000","mapping_epoch":445,"log_start":"454'2800","ondisk_log_start":"454'2800","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'11582","last_scrub_stamp":"2024-04-20T17:26:32.955650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:32.955650+0000","objects_scrubbed":9541,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:49.509455+0000","scrub_duration":1635,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538312704,"num_objects":8214,"num_object_clones":0,"num_object_copies":16428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12860,"num_write_kb":674368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"493'15938","reported_seq":16310,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-04-20T17:26:51.023496+0000","last_change":"2024-04-20T17:26:26.494294+0000","last_active":"2024-04-20T17:26:51.023496+0000","last_peered":"2024-04-20T17:26:51.023496+0000","last_clean":"2024-04-20T17:26:51.023496+0000","last_became_active":"2024-04-20T17:24:37.009687+0000","last_became_peered":"2024-04-20T17:24:37.009687+0000","last_unstale":"2024-04-20T17:26:51.023496+0000","last_undegraded":"2024-04-20T17:26:51.023496+0000","last_fullsized":"2024-04-20T17:26:51.023496+0000","mapping_epoch":445,"log_start":"464'5901","ondisk_log_start":"464'5901","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"487'14083","last_scrub_stamp":"2024-04-20T17:26:26.494158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:26.494158+0000","objects_scrubbed":10068,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:40.599471+0000","scrub_duration":1814,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":533725184,"num_objects":8144,"num_object_clones":0,"num_object_copies":16288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13128,"num_write_kb":680704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"493'12920","reported_seq":13119,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-04-20T17:26:48.179495+0000","last_change":"2024-04-20T17:25:50.237975+0000","last_active":"2024-04-20T17:26:48.179495+0000","last_peered":"2024-04-20T17:26:48.179495+0000","last_clean":"2024-04-20T17:26:48.179495+0000","last_became_active":"2024-04-20T17:24:37.009803+0000","last_became_peered":"2024-04-20T17:24:37.009803+0000","last_unstale":"2024-04-20T17:26:48.179495+0000","last_undegraded":"2024-04-20T17:26:48.179495+0000","last_fullsized":"2024-04-20T17:26:48.179495+0000","mapping_epoch":445,"log_start":"454'2900","ondisk_log_start":"454'2900","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'8646","last_scrub_stamp":"2024-04-20T17:25:50.237772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:25:50.237772+0000","objects_scrubbed":8579,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:16.056979+0000","scrub_duration":1753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":543162368,"num_objects":8288,"num_object_clones":0,"num_object_copies":16576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12920,"num_write_kb":678656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"493'13292","reported_seq":13496,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-04-20T17:26:51.282893+0000","last_change":"2024-04-20T17:25:54.810708+0000","last_active":"2024-04-20T17:26:51.282893+0000","last_peered":"2024-04-20T17:26:51.282893+0000","last_clean":"2024-04-20T17:26:51.282893+0000","last_became_active":"2024-04-20T17:24:36.895371+0000","last_became_peered":"2024-04-20T17:24:36.895371+0000","last_unstale":"2024-04-20T17:26:51.282893+0000","last_undegraded":"2024-04-20T17:26:51.282893+0000","last_fullsized":"2024-04-20T17:26:51.282893+0000","mapping_epoch":445,"log_start":"455'3200","ondisk_log_start":"455'3200","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'9084","last_scrub_stamp":"2024-04-20T17:25:54.810541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:25:54.810541+0000","objects_scrubbed":9023,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:26:57.947227+0000","scrub_duration":1728,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537264128,"num_objects":8198,"num_object_clones":0,"num_object_copies":16396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13292,"num_write_kb":687680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"493'12930","reported_seq":13141,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-04-20T17:26:47.476992+0000","last_change":"2024-04-20T17:26:30.993672+0000","last_active":"2024-04-20T17:26:47.476992+0000","last_peered":"2024-04-20T17:26:47.476992+0000","last_clean":"2024-04-20T17:26:47.476992+0000","last_became_active":"2024-04-20T17:24:36.574038+0000","last_became_peered":"2024-04-20T17:24:36.574038+0000","last_unstale":"2024-04-20T17:26:47.476992+0000","last_undegraded":"2024-04-20T17:26:47.476992+0000","last_fullsized":"2024-04-20T17:26:47.476992+0000","mapping_epoch":445,"log_start":"454'2900","ondisk_log_start":"454'2900","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"489'11622","last_scrub_stamp":"2024-04-20T17:26:30.993531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:30.993531+0000","objects_scrubbed":9793,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:56.928446+0000","scrub_duration":2078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":550764544,"num_objects":8404,"num_object_clones":0,"num_object_copies":16808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12930,"num_write_kb":682688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":504,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-04-20T17:26:44.248072+0000","last_change":"2024-04-20T17:25:35.094175+0000","last_active":"2024-04-20T17:26:44.248072+0000","last_peered":"2024-04-20T17:26:44.248072+0000","last_clean":"2024-04-20T17:26:44.248072+0000","last_became_active":"2024-04-20T17:23:24.109202+0000","last_became_peered":"2024-04-20T17:23:24.109202+0000","last_unstale":"2024-04-20T17:26:44.248072+0000","last_undegraded":"2024-04-20T17:26:44.248072+0000","last_fullsized":"2024-04-20T17:26:44.248072+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:25:35.094040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:25:35.094040+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:26:49.811036+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":8638234648,"num_objects":131810,"num_object_clones":0,"num_object_copies":263620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131810,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":208570,"num_write_kb":10892097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17375698944,"data_stored":17375690800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160679,"ondisk_log_size":160679,"up":32,"acting":32,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991485,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2517272,"kb_used_data":2143184,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":91227880,"statfs":{"total":95995035648,"available":93417349120,"internally_reserved":0,"allocated":2194620416,"data_stored":2194250911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":383056324},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253124,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1852880,"kb_used_data":1596808,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":91892272,"statfs":{"total":95995035648,"available":94097686528,"internally_reserved":0,"allocated":1635131392,"data_stored":1634765439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":262207936},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612874,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1886676,"kb_used_data":1576460,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":91858476,"statfs":{"total":95995035648,"available":94063079424,"internally_reserved":0,"allocated":1614295040,"data_stored":1613925035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":317651392},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809104,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":154068,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93591084,"statfs":{"total":95995035648,"available":95837270016,"internally_reserved":0,"allocated":729088,"data_stored":363155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":398,"seq":1709396983860,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1724760,"kb_used_data":1618384,"kb_used_omap":6,"kb_used_meta":106297,"kb_avail":92020392,"statfs":{"total":95995035648,"available":94228881408,"internally_reserved":0,"allocated":1657225216,"data_stored":1656855731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6767,"internal_metadata":108848529},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430488,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1236944,"kb_used_data":1079432,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":92508208,"statfs":{"total":95995035648,"available":94728404992,"internally_reserved":0,"allocated":1105338368,"data_stored":1104972435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":161282502},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575153,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4118288,"kb_used_data":3784200,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":89626864,"statfs":{"total":95995035648,"available":91777908736,"internally_reserved":0,"allocated":3875020800,"data_stored":3874654867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738678,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":861904,"kb_used_data":537800,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92883248,"statfs":{"total":95995035648,"available":95112445952,"internally_reserved":0,"allocated":550707200,"data_stored":550341267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393969,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":582864,"kb_used_data":512264,"kb_used_omap":202,"kb_used_meta":70389,"kb_avail":93162288,"statfs":{"total":95995035648,"available":95398182912,"internally_reserved":0,"allocated":524558336,"data_stored":524192403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":72078807},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182695,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1851408,"kb_used_data":1571720,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":91893744,"statfs":{"total":95995035648,"available":94099193856,"internally_reserved":0,"allocated":1609441280,"data_stored":1609075347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":286390727},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986466,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1733076,"kb_used_data":1575564,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":92012076,"statfs":{"total":95995035648,"available":94220365824,"internally_reserved":0,"allocated":1613377536,"data_stored":1613007531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790237,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1256848,"kb_used_data":1049352,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":92488304,"statfs":{"total":95995035648,"available":94708023296,"internally_reserved":0,"allocated":1074536448,"data_stored":1074170515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":212466117},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":546766848,"data_stored":546766848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3867541504,"data_stored":3867541504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":523501568,"data_stored":523501568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1603731456,"data_stored":1603731456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1605242880,"data_stored":1605238808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1065811968,"data_stored":1065811968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1097924608,"data_stored":1097924608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1652293632,"data_stored":1652293632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1605636096,"data_stored":1605632024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1631715328,"data_stored":1631715328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2175533056,"data_stored":2175533056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:26:54.791 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:26:54.847 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:26:54.881 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:55.094 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:26:55.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:55.099 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:26:55.102 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-20T17:26:55.200 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:26:55.227 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:55.250 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:26:55.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:55.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:26:55.321 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:55 smithi005 ceph-mon[39562]: pgmap v1307: 17 pgs: 17 active+clean; 7.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-04-20T17:26:55.321 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:55 smithi005 ceph-mon[39562]: osdmap e494: 12 total, 12 up, 12 in 2024-04-20T17:26:55.356 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:26:55.381 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:26:55.393 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:55.415 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:26:55.419 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:55 smithi171 ceph-mon[44189]: pgmap v1307: 17 pgs: 17 active+clean; 7.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-04-20T17:26:55.419 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:55 smithi171 ceph-mon[44189]: osdmap e494: 12 total, 12 up, 12 in 2024-04-20T17:26:55.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:55.420 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:26:55.456 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:26:55.485 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:26:55.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:55 smithi106 ceph-mon[44423]: pgmap v1307: 17 pgs: 17 active+clean; 7.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-04-20T17:26:55.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:55 smithi106 ceph-mon[44423]: osdmap e494: 12 total, 12 up, 12 in 2024-04-20T17:26:55.514 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:26:55.516 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:26:55.521 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:26:55.541 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:26:55.563 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:55.574 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:26:55.615 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:26:55.643 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:26:55.676 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:26:55.678 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:26:55.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:55.685 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:26:55.716 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:26:55.744 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:26:55.786 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:26:55.795 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:26:55.814 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:55.837 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:26:55.841 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:26:55.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:55.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:26:55.857 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:26:55.857 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:26:55.857 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:26:55.857 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:26:55.857 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:26:55.858 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:26:55.858 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:26:55.858 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:26:55.858 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:26:55.858 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:26:55.858 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:26:55.858 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:26:55.943 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:26:55.990 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:56.130 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:26:56.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:56.134 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:26:56.235 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:26:56.290 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:56.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:56 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2393159811' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:26:56.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:56.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:56.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:56.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:56.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:56.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:56.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:56 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2393159811' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:26:56.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:56.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:56.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:56.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:56.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:56.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:56.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:56 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2393159811' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:26:56.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:56.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:56.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:56.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:56.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:56.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:57.062 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:26:57.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:57.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:57.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:57.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:57.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:57.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:57.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:57 smithi005 ceph-mon[39562]: pgmap v1309: 17 pgs: 17 active+clean; 7.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-04-20T17:26:57.510 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:26:57.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:57.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:26:57.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:57.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:57.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:57.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:57.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:57.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:57.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:57 smithi171 ceph-mon[44189]: pgmap v1309: 17 pgs: 17 active+clean; 7.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-04-20T17:26:57.614 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:26:57.657 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:57.689 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:26:57.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:57.693 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:26:57.735 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:26:57.735 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:26:57.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:57.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:57.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:57.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:57.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:57.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:26:57.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:57 smithi106 ceph-mon[44423]: pgmap v1309: 17 pgs: 17 active+clean; 7.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-04-20T17:26:57.794 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:26:57.841 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:58.580 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-20T17:26:58.621 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:26:58.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:58.631 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:26:58.732 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:26:58.779 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:58.817 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:26:58.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:58.825 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:26:58.925 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:26:58.955 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:58.959 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:26:58.982 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:26:58.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:58.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:26:59.090 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:26:59.149 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:59.182 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:26:59.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:26:59.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:26:59.291 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:26:59.330 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:26:59.365 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:26:59.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:59 smithi005 ceph-mon[39562]: pgmap v1310: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s 2024-04-20T17:26:59.395 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:59 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/783838459' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:26:59.395 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:26:59 smithi005 ceph-mon[39562]: from='client.26233 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:26:59.562 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:26:59.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:59 smithi171 ceph-mon[44189]: pgmap v1310: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s 2024-04-20T17:26:59.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:59 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/783838459' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:26:59.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:26:59 smithi171 ceph-mon[44189]: from='client.26233 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:26:59.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:59 smithi106 ceph-mon[44423]: pgmap v1310: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s 2024-04-20T17:26:59.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:59 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/783838459' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:26:59.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:26:59 smithi106 ceph-mon[44423]: from='client.26233 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:27:00.349 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:00 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:27:00.349 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:00 smithi005 ceph-mon[39562]: from='client.26233 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:27:00.349 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:00 smithi005 ceph-mon[39562]: osdmap e495: 12 total, 12 up, 12 in 2024-04-20T17:27:00.422 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:27:00.427 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:27:00.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:00 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:27:00.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:00 smithi106 ceph-mon[44423]: from='client.26233 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:27:00.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:00 smithi106 ceph-mon[44423]: osdmap e495: 12 total, 12 up, 12 in 2024-04-20T17:27:00.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:00 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:27:00.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:00 smithi171 ceph-mon[44189]: from='client.26233 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:27:00.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:00 smithi171 ceph-mon[44189]: osdmap e495: 12 total, 12 up, 12 in 2024-04-20T17:27:01.175 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":1312,"stamp":"2024-04-20T17:27:00.231728+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8033403448,"num_objects":122574,"num_object_clones":0,"num_object_copies":245148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":122574,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":217989,"num_write_kb":10895035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160954,"ondisk_log_size":160954,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":11,"kb":1124941824,"kb_used":18509112,"kb_used_data":15765832,"kb_used_omap":224,"kb_used_meta":2742943,"kb_avail":1106432712,"statfs":{"total":1151940427776,"available":1132987097088,"internally_reserved":0,"allocated":16144211968,"data_stored":16139733436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":230126,"internal_metadata":2808773906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,4,8],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-605421568,"num_objects":-9238,"num_object_clones":0,"num_object_copies":-18476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9238,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001610"},"pg_stats":[{"pgid":"4.f","version":"494'13594","reported_seq":13807,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-04-20T17:26:56.310580+0000","last_change":"2024-04-20T17:26:34.139365+0000","last_active":"2024-04-20T17:26:56.310580+0000","last_peered":"2024-04-20T17:26:56.310580+0000","last_clean":"2024-04-20T17:26:56.310580+0000","last_became_active":"2024-04-20T17:24:36.727388+0000","last_became_peered":"2024-04-20T17:24:36.727388+0000","last_unstale":"2024-04-20T17:26:56.310580+0000","last_undegraded":"2024-04-20T17:26:56.310580+0000","last_fullsized":"2024-04-20T17:26:56.310580+0000","mapping_epoch":445,"log_start":"456'3500","ondisk_log_start":"456'3500","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'11858","last_scrub_stamp":"2024-04-20T17:26:34.139195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:34.139195+0000","objects_scrubbed":9615,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:51.741322+0000","scrub_duration":2286,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":510132224,"num_objects":7784,"num_object_clones":0,"num_object_copies":15568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13594,"num_write_kb":684096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"494'13382","reported_seq":13686,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-04-20T17:26:55.764042+0000","last_change":"2024-04-20T17:26:29.940846+0000","last_active":"2024-04-20T17:26:55.764042+0000","last_peered":"2024-04-20T17:26:55.764042+0000","last_clean":"2024-04-20T17:26:55.764042+0000","last_became_active":"2024-04-20T17:24:36.573304+0000","last_became_peered":"2024-04-20T17:24:36.573304+0000","last_unstale":"2024-04-20T17:26:55.764042+0000","last_undegraded":"2024-04-20T17:26:55.764042+0000","last_fullsized":"2024-04-20T17:26:55.764042+0000","mapping_epoch":445,"log_start":"455'3300","ondisk_log_start":"455'3300","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'11392","last_scrub_stamp":"2024-04-20T17:26:29.940585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:29.940585+0000","objects_scrubbed":9772,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:57.030101+0000","scrub_duration":1744,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":506462208,"num_objects":7728,"num_object_clones":0,"num_object_copies":15456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13382,"num_write_kb":675520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"494'13382","reported_seq":13588,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-04-20T17:26:55.743295+0000","last_change":"2024-04-20T17:25:55.471139+0000","last_active":"2024-04-20T17:26:55.743295+0000","last_peered":"2024-04-20T17:26:55.743295+0000","last_clean":"2024-04-20T17:26:55.743295+0000","last_became_active":"2024-04-20T17:24:36.573474+0000","last_became_peered":"2024-04-20T17:24:36.573474+0000","last_unstale":"2024-04-20T17:26:55.743295+0000","last_undegraded":"2024-04-20T17:26:55.743295+0000","last_fullsized":"2024-04-20T17:26:55.743295+0000","mapping_epoch":445,"log_start":"455'3300","ondisk_log_start":"455'3300","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'8994","last_scrub_stamp":"2024-04-20T17:25:55.470885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:25:55.470885+0000","objects_scrubbed":8940,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:01.834108+0000","scrub_duration":1381,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":511508504,"num_objects":7806,"num_object_clones":0,"num_object_copies":15612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7806,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13382,"num_write_kb":677953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"494'13520","reported_seq":13729,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-04-20T17:26:55.756266+0000","last_change":"2024-04-20T17:26:02.737108+0000","last_active":"2024-04-20T17:26:55.756266+0000","last_peered":"2024-04-20T17:26:55.756266+0000","last_clean":"2024-04-20T17:26:55.756266+0000","last_became_active":"2024-04-20T17:24:36.572684+0000","last_became_peered":"2024-04-20T17:24:36.572684+0000","last_unstale":"2024-04-20T17:26:55.756266+0000","last_undegraded":"2024-04-20T17:26:55.756266+0000","last_fullsized":"2024-04-20T17:26:55.756266+0000","mapping_epoch":445,"log_start":"456'3500","ondisk_log_start":"456'3500","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'9546","last_scrub_stamp":"2024-04-20T17:26:02.736996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:02.736996+0000","objects_scrubbed":9482,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:03.229190+0000","scrub_duration":1696,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":511049728,"num_objects":7798,"num_object_clones":0,"num_object_copies":15596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13520,"num_write_kb":682176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"494'13947","reported_seq":14157,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-04-20T17:26:59.345257+0000","last_change":"2024-04-20T17:26:03.163791+0000","last_active":"2024-04-20T17:26:59.345257+0000","last_peered":"2024-04-20T17:26:59.345257+0000","last_clean":"2024-04-20T17:26:59.345257+0000","last_became_active":"2024-04-20T17:24:36.726966+0000","last_became_peered":"2024-04-20T17:24:36.726966+0000","last_unstale":"2024-04-20T17:26:59.345257+0000","last_undegraded":"2024-04-20T17:26:59.345257+0000","last_fullsized":"2024-04-20T17:26:59.345257+0000","mapping_epoch":445,"log_start":"457'3900","ondisk_log_start":"457'3900","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'9675","last_scrub_stamp":"2024-04-20T17:26:03.163686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:03.163686+0000","objects_scrubbed":9632,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:06.504359+0000","scrub_duration":1737,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":495517696,"num_objects":7561,"num_object_clones":0,"num_object_copies":15122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13947,"num_write_kb":688256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"494'13483","reported_seq":13680,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-04-20T17:26:57.509678+0000","last_change":"2024-04-20T17:25:47.229572+0000","last_active":"2024-04-20T17:26:57.509678+0000","last_peered":"2024-04-20T17:26:57.509678+0000","last_clean":"2024-04-20T17:26:57.509678+0000","last_became_active":"2024-04-20T17:24:36.727448+0000","last_became_peered":"2024-04-20T17:24:36.727448+0000","last_unstale":"2024-04-20T17:26:57.509678+0000","last_undegraded":"2024-04-20T17:26:57.509678+0000","last_fullsized":"2024-04-20T17:26:57.509678+0000","mapping_epoch":445,"log_start":"456'3400","ondisk_log_start":"456'3400","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'8313","last_scrub_stamp":"2024-04-20T17:25:47.229419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:25:47.229419+0000","objects_scrubbed":8261,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:15.544542+0000","scrub_duration":1387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":500498432,"num_objects":7637,"num_object_clones":0,"num_object_copies":15274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13483,"num_write_kb":675840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"494'13693","reported_seq":13887,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-04-20T17:26:58.203492+0000","last_change":"2024-04-20T17:25:42.475029+0000","last_active":"2024-04-20T17:26:58.203492+0000","last_peered":"2024-04-20T17:26:58.203492+0000","last_clean":"2024-04-20T17:26:58.203492+0000","last_became_active":"2024-04-20T17:24:36.727451+0000","last_became_peered":"2024-04-20T17:24:36.727451+0000","last_unstale":"2024-04-20T17:26:58.203492+0000","last_undegraded":"2024-04-20T17:26:58.203492+0000","last_fullsized":"2024-04-20T17:26:58.203492+0000","mapping_epoch":445,"log_start":"457'3600","ondisk_log_start":"457'3600","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"471'7991","last_scrub_stamp":"2024-04-20T17:25:42.474753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:25:42.474753+0000","objects_scrubbed":7878,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:01.483744+0000","scrub_duration":2083,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":496566272,"num_objects":7577,"num_object_clones":0,"num_object_copies":15154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13693,"num_write_kb":680640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"494'13573","reported_seq":13768,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-04-20T17:26:58.197393+0000","last_change":"2024-04-20T17:25:43.026635+0000","last_active":"2024-04-20T17:26:58.197393+0000","last_peered":"2024-04-20T17:26:58.197393+0000","last_clean":"2024-04-20T17:26:58.197393+0000","last_became_active":"2024-04-20T17:24:36.895302+0000","last_became_peered":"2024-04-20T17:24:36.895302+0000","last_unstale":"2024-04-20T17:26:58.197393+0000","last_undegraded":"2024-04-20T17:26:58.197393+0000","last_fullsized":"2024-04-20T17:26:58.197393+0000","mapping_epoch":445,"log_start":"457'3500","ondisk_log_start":"457'3500","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"471'7970","last_scrub_stamp":"2024-04-20T17:25:43.026438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:25:43.026438+0000","objects_scrubbed":7867,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:26:59.247649+0000","scrub_duration":1624,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499580928,"num_objects":7623,"num_object_clones":0,"num_object_copies":15246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13573,"num_write_kb":678272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"494'13825","reported_seq":14072,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-04-20T17:26:59.084275+0000","last_change":"2024-04-20T17:26:28.991603+0000","last_active":"2024-04-20T17:26:59.084275+0000","last_peered":"2024-04-20T17:26:59.084275+0000","last_clean":"2024-04-20T17:26:59.084275+0000","last_became_active":"2024-04-20T17:24:36.727119+0000","last_became_peered":"2024-04-20T17:24:36.727119+0000","last_unstale":"2024-04-20T17:26:59.084275+0000","last_undegraded":"2024-04-20T17:26:59.084275+0000","last_fullsized":"2024-04-20T17:26:59.084275+0000","mapping_epoch":445,"log_start":"457'3800","ondisk_log_start":"457'3800","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"487'11515","last_scrub_stamp":"2024-04-20T17:26:28.991420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:28.991420+0000","objects_scrubbed":9918,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:47.134221+0000","scrub_duration":2074,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":493682688,"num_objects":7533,"num_object_clones":0,"num_object_copies":15066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13825,"num_write_kb":683456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"494'13817","reported_seq":14028,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-04-20T17:26:56.023102+0000","last_change":"2024-04-20T17:26:00.309772+0000","last_active":"2024-04-20T17:26:56.023102+0000","last_peered":"2024-04-20T17:26:56.023102+0000","last_clean":"2024-04-20T17:26:56.023102+0000","last_became_active":"2024-04-20T17:24:36.727225+0000","last_became_peered":"2024-04-20T17:24:36.727225+0000","last_unstale":"2024-04-20T17:26:56.023102+0000","last_undegraded":"2024-04-20T17:26:56.023102+0000","last_fullsized":"2024-04-20T17:26:56.023102+0000","mapping_epoch":445,"log_start":"457'3800","ondisk_log_start":"457'3800","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'9588","last_scrub_stamp":"2024-04-20T17:26:00.309610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:00.309610+0000","objects_scrubbed":9527,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:02.777067+0000","scrub_duration":1737,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":511901696,"num_objects":7811,"num_object_clones":0,"num_object_copies":15622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13817,"num_write_kb":692096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"494'13313","reported_seq":13520,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-04-20T17:26:55.990254+0000","last_change":"2024-04-20T17:26:00.587646+0000","last_active":"2024-04-20T17:26:55.990254+0000","last_peered":"2024-04-20T17:26:55.990254+0000","last_clean":"2024-04-20T17:26:55.990254+0000","last_became_active":"2024-04-20T17:24:36.727028+0000","last_became_peered":"2024-04-20T17:24:36.727028+0000","last_unstale":"2024-04-20T17:26:55.990254+0000","last_undegraded":"2024-04-20T17:26:55.990254+0000","last_fullsized":"2024-04-20T17:26:55.990254+0000","mapping_epoch":445,"log_start":"455'3300","ondisk_log_start":"455'3300","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'9204","last_scrub_stamp":"2024-04-20T17:26:00.587445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:00.587445+0000","objects_scrubbed":9139,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:00.762089+0000","scrub_duration":1490,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499056640,"num_objects":7615,"num_object_clones":0,"num_object_copies":15230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13313,"num_write_kb":669696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"494'13649","reported_seq":13859,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-04-20T17:26:58.179674+0000","last_change":"2024-04-20T17:26:32.955835+0000","last_active":"2024-04-20T17:26:58.179674+0000","last_peered":"2024-04-20T17:26:58.179674+0000","last_clean":"2024-04-20T17:26:58.179674+0000","last_became_active":"2024-04-20T17:24:36.895348+0000","last_became_peered":"2024-04-20T17:24:36.895348+0000","last_unstale":"2024-04-20T17:26:58.179674+0000","last_undegraded":"2024-04-20T17:26:58.179674+0000","last_fullsized":"2024-04-20T17:26:58.179674+0000","mapping_epoch":445,"log_start":"457'3600","ondisk_log_start":"457'3600","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'11582","last_scrub_stamp":"2024-04-20T17:26:32.955650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:32.955650+0000","objects_scrubbed":9541,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:49.509455+0000","scrub_duration":1635,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":486604800,"num_objects":7425,"num_object_clones":0,"num_object_copies":14850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13649,"num_write_kb":674368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"494'16327","reported_seq":16701,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-04-20T17:26:56.040369+0000","last_change":"2024-04-20T17:26:26.494294+0000","last_active":"2024-04-20T17:26:56.040369+0000","last_peered":"2024-04-20T17:26:56.040369+0000","last_clean":"2024-04-20T17:26:56.040369+0000","last_became_active":"2024-04-20T17:24:37.009687+0000","last_became_peered":"2024-04-20T17:24:37.009687+0000","last_unstale":"2024-04-20T17:26:56.040369+0000","last_undegraded":"2024-04-20T17:26:56.040369+0000","last_fullsized":"2024-04-20T17:26:56.040369+0000","mapping_epoch":445,"log_start":"465'6301","ondisk_log_start":"465'6301","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"487'14083","last_scrub_stamp":"2024-04-20T17:26:26.494158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:26.494158+0000","objects_scrubbed":10068,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:40.599471+0000","scrub_duration":1814,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":508231680,"num_objects":7755,"num_object_clones":0,"num_object_copies":15510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13517,"num_write_kb":680704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"494'13737","reported_seq":13938,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-04-20T17:26:58.163884+0000","last_change":"2024-04-20T17:25:50.237975+0000","last_active":"2024-04-20T17:26:58.163884+0000","last_peered":"2024-04-20T17:26:58.163884+0000","last_clean":"2024-04-20T17:26:58.163884+0000","last_became_active":"2024-04-20T17:24:37.009803+0000","last_became_peered":"2024-04-20T17:24:37.009803+0000","last_unstale":"2024-04-20T17:26:58.163884+0000","last_undegraded":"2024-04-20T17:26:58.163884+0000","last_fullsized":"2024-04-20T17:26:58.163884+0000","mapping_epoch":445,"log_start":"457'3700","ondisk_log_start":"457'3700","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'8646","last_scrub_stamp":"2024-04-20T17:25:50.237772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:25:50.237772+0000","objects_scrubbed":8579,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:16.056979+0000","scrub_duration":1753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489619456,"num_objects":7471,"num_object_clones":0,"num_object_copies":14942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13737,"num_write_kb":678656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"494'13662","reported_seq":13867,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-04-20T17:26:56.315128+0000","last_change":"2024-04-20T17:25:54.810708+0000","last_active":"2024-04-20T17:26:56.315128+0000","last_peered":"2024-04-20T17:26:56.315128+0000","last_clean":"2024-04-20T17:26:56.315128+0000","last_became_active":"2024-04-20T17:24:36.895371+0000","last_became_peered":"2024-04-20T17:24:36.895371+0000","last_unstale":"2024-04-20T17:26:56.315128+0000","last_undegraded":"2024-04-20T17:26:56.315128+0000","last_fullsized":"2024-04-20T17:26:56.315128+0000","mapping_epoch":445,"log_start":"457'3600","ondisk_log_start":"457'3600","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'9084","last_scrub_stamp":"2024-04-20T17:25:54.810541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:25:54.810541+0000","objects_scrubbed":9023,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:26:57.947227+0000","scrub_duration":1728,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":513015808,"num_objects":7828,"num_object_clones":0,"num_object_copies":15656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13662,"num_write_kb":687680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"494'13714","reported_seq":13927,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-04-20T17:26:57.516883+0000","last_change":"2024-04-20T17:26:30.993672+0000","last_active":"2024-04-20T17:26:57.516883+0000","last_peered":"2024-04-20T17:26:57.516883+0000","last_clean":"2024-04-20T17:26:57.516883+0000","last_became_active":"2024-04-20T17:24:36.574038+0000","last_became_peered":"2024-04-20T17:24:36.574038+0000","last_unstale":"2024-04-20T17:26:57.516883+0000","last_undegraded":"2024-04-20T17:26:57.516883+0000","last_fullsized":"2024-04-20T17:26:57.516883+0000","mapping_epoch":445,"log_start":"457'3700","ondisk_log_start":"457'3700","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"489'11622","last_scrub_stamp":"2024-04-20T17:26:30.993531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:30.993531+0000","objects_scrubbed":9793,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:27:56.928446+0000","scrub_duration":2078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499384320,"num_objects":7620,"num_object_clones":0,"num_object_copies":15240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13714,"num_write_kb":682688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":506,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-04-20T17:26:54.319830+0000","last_change":"2024-04-20T17:25:35.094175+0000","last_active":"2024-04-20T17:26:54.319830+0000","last_peered":"2024-04-20T17:26:54.319830+0000","last_clean":"2024-04-20T17:26:54.319830+0000","last_became_active":"2024-04-20T17:23:24.109202+0000","last_became_peered":"2024-04-20T17:23:24.109202+0000","last_unstale":"2024-04-20T17:26:54.319830+0000","last_undegraded":"2024-04-20T17:26:54.319830+0000","last_fullsized":"2024-04-20T17:26:54.319830+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:25:35.094040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:25:35.094040+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:26:55.142989+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":8032813080,"num_objects":122572,"num_object_clones":0,"num_object_copies":245144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":122572,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":217808,"num_write_kb":10892097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16058490880,"data_stored":16058482736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160817,"ondisk_log_size":160817,"up":32,"acting":32,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991487,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2314660,"kb_used_data":1944732,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":91430492,"statfs":{"total":95995035648,"available":93624823808,"internally_reserved":0,"allocated":1991405568,"data_stored":1991030059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":378796484},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253126,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1690844,"kb_used_data":1443092,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":92054308,"statfs":{"total":95995035648,"available":94263611392,"internally_reserved":0,"allocated":1477726208,"data_stored":1477354251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":253688256},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612875,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1814884,"kb_used_data":1500504,"kb_used_omap":1,"kb_used_meta":314366,"kb_avail":91930268,"statfs":{"total":95995035648,"available":94136594432,"internally_reserved":0,"allocated":1536516096,"data_stored":1536140087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":321911232},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809106,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":154080,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93591072,"statfs":{"total":95995035648,"available":95837257728,"internally_reserved":0,"allocated":741376,"data_stored":369439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":398,"seq":1709396983862,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1587556,"kb_used_data":1472924,"kb_used_omap":6,"kb_used_meta":114617,"kb_avail":92157596,"statfs":{"total":95995035648,"available":94369378304,"internally_reserved":0,"allocated":1508274176,"data_stored":1507898687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6767,"internal_metadata":117368209},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430490,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1148444,"kb_used_data":982612,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":92596708,"statfs":{"total":95995035648,"available":94819028992,"internally_reserved":0,"allocated":1006194688,"data_stored":1005822751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":169802182},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575155,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3783260,"kb_used_data":3445268,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":89961892,"statfs":{"total":95995035648,"available":92120977408,"internally_reserved":0,"allocated":3527954432,"data_stored":3527582495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738680,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":817436,"kb_used_data":489172,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92927716,"statfs":{"total":95995035648,"available":95157981184,"internally_reserved":0,"allocated":500912128,"data_stored":500540191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393970,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":542304,"kb_used_data":488340,"kb_used_omap":202,"kb_used_meta":53749,"kb_avail":93202848,"statfs":{"total":95995035648,"available":95439716352,"internally_reserved":0,"allocated":500060160,"data_stored":499688223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":55039447},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182696,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1784540,"kb_used_data":1500692,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":91960612,"statfs":{"total":95995035648,"available":94167666688,"internally_reserved":0,"allocated":1536708608,"data_stored":1536336671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":290650567},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986467,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1661728,"kb_used_data":1500056,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":92083424,"statfs":{"total":95995035648,"available":94293426176,"internally_reserved":0,"allocated":1536057344,"data_stored":1535681335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790238,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1209376,"kb_used_data":997716,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":92535776,"statfs":{"total":95995035648,"available":94756634624,"internally_reserved":0,"allocated":1021661184,"data_stored":1021289247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":495517696,"data_stored":495517696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3511746560,"data_stored":3511746560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":499056640,"data_stored":499056640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1531510784,"data_stored":1531510784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1529024512,"data_stored":1529020440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1013186560,"data_stored":1013186560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":999882752,"data_stored":999882752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1502216192,"data_stored":1502216192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1530859520,"data_stored":1530855448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1473118208,"data_stored":1473118208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1972371456,"data_stored":1972371456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:27:01.176 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-20T17:27:01.261 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:27:01.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:01 smithi005 ceph-mon[39562]: pgmap v1312: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:27:01.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:01 smithi005 ceph-mon[39562]: from='client.26321 v1:172.21.15.5:0/2485909834' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:27:01.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:01 smithi106 ceph-mon[44423]: pgmap v1312: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:27:01.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:01 smithi106 ceph-mon[44423]: from='client.26321 v1:172.21.15.5:0/2485909834' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:27:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:01 smithi171 ceph-mon[44189]: pgmap v1312: 17 pgs: 17 active+clean; 7.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:27:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:01 smithi171 ceph-mon[44189]: from='client.26321 v1:172.21.15.5:0/2485909834' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:27:02.411 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:27:02.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:02.413 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:27:02.514 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:27:02.569 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:27:02.573 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:02 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:27:02.574 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:02 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:27:02.580 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:02.695 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:02 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:27:02.696 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:02 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:27:02.701 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:27:02.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:02.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:27:02.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:02 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:27:02.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:02 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:27:02.818 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:27:02.887 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:03.101 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:27:03.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:03.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:27:03.216 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:27:03.295 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:03.382 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:27:03.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:03.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:27:03.450 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:27:03.491 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:27:03.509 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:27:03.519 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:03 smithi171 ceph-mon[44189]: 4.4 scrub starts 2024-04-20T17:27:03.519 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:03 smithi171 ceph-mon[44189]: pgmap v1313: 17 pgs: 17 active+clean; 7.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s 2024-04-20T17:27:03.519 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:03 smithi171 ceph-mon[44189]: 4.8 scrub starts 2024-04-20T17:27:03.519 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:03 smithi171 ceph-mon[44189]: osdmap e496: 12 total, 12 up, 12 in 2024-04-20T17:27:03.522 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:03.549 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:27:03.554 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:27:03.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:03.558 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:27:03.586 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:27:03.628 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:27:03.656 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:27:03.659 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:27:03.688 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:03 smithi106 ceph-mon[44423]: 4.4 scrub starts 2024-04-20T17:27:03.688 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:03 smithi106 ceph-mon[44423]: pgmap v1313: 17 pgs: 17 active+clean; 7.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s 2024-04-20T17:27:03.688 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:03 smithi106 ceph-mon[44423]: 4.8 scrub starts 2024-04-20T17:27:03.689 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:03 smithi106 ceph-mon[44423]: osdmap e496: 12 total, 12 up, 12 in 2024-04-20T17:27:03.690 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:27:03.691 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:03.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:03 smithi005 ceph-mon[39562]: 4.4 scrub starts 2024-04-20T17:27:03.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:03 smithi005 ceph-mon[39562]: pgmap v1313: 17 pgs: 17 active+clean; 7.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s 2024-04-20T17:27:03.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:03 smithi005 ceph-mon[39562]: 4.8 scrub starts 2024-04-20T17:27:03.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:03 smithi005 ceph-mon[39562]: osdmap e496: 12 total, 12 up, 12 in 2024-04-20T17:27:03.718 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:27:03.726 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:27:03.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:03.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:27:03.758 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:27:03.798 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:27:03.830 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:27:03.833 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:27:03.877 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:27:03.898 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:03.916 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:27:03.931 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:27:03.932 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:27:03.932 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:27:03.932 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:27:03.932 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:27:03.932 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:27:03.932 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:27:03.932 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:27:03.932 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:27:03.932 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:27:03.932 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:27:03.932 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:27:04.268 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:27:04.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:04.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:27:04.291 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:27:04.292 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:27:04.292 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.292 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:226482 4.7 4:efa4b76d:::benchmark_data_smithi171_122772_object56290:head [delete] snapc 0=[] ondisk+write+known_if_redirected e496)", 2024-04-20T17:27:04.292 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:27:04.273553+0000", 2024-04-20T17:27:04.292 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.017135174, 2024-04-20T17:27:04.292 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.017170707, 2024-04-20T17:27:04.292 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:27:04.292 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:27:04.292 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:27:04.292 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:27:04.292 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:27:04.292 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:04.292 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 226482 2024-04-20T17:27:04.292 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.292 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:27:04.293 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.293 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:27:04.293 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.273553+0000", 2024-04-20T17:27:04.293 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:04.293 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.293 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.293 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:27:04.293 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.273553+0000", 2024-04-20T17:27:04.293 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:04.293 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.293 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.293 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:27:04.293 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.273555+0000", 2024-04-20T17:27:04.293 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1909999999999999e-06 2024-04-20T17:27:04.293 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.294 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.294 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:27:04.294 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.273562+0000", 2024-04-20T17:27:04.294 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5649999999999998e-06 2024-04-20T17:27:04.294 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.294 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.294 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:27:04.294 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.273576+0000", 2024-04-20T17:27:04.294 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3585e-05 2024-04-20T17:27:04.294 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.294 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.294 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:27:04.294 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.273584+0000", 2024-04-20T17:27:04.294 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.3399999999999998e-06 2024-04-20T17:27:04.294 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.295 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.295 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:04.295 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.273645+0000", 2024-04-20T17:27:04.295 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0998000000000001e-05 2024-04-20T17:27:04.295 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.295 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.295 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:27:04.295 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.273763+0000", 2024-04-20T17:27:04.295 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011789400000000001 2024-04-20T17:27:04.295 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.295 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.295 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:27:04.295 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.273874+0000", 2024-04-20T17:27:04.296 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011121299999999999 2024-04-20T17:27:04.296 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.296 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.296 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:27:04.296 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.281906+0000", 2024-04-20T17:27:04.296 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0080317210000000003 2024-04-20T17:27:04.296 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:04.296 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:27:04.296 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:04.296 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.296 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.296 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:226491 4.7 4:ed102a7d:::benchmark_data_smithi171_122772_object56299:head [delete] snapc 0=[] ondisk+write+known_if_redirected e496)", 2024-04-20T17:27:04.296 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:27:04.277237+0000", 2024-04-20T17:27:04.296 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.013451182000000001, 2024-04-20T17:27:04.296 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013577057999999999, 2024-04-20T17:27:04.296 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:27:04.297 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:27:04.297 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:27:04.297 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:27:04.297 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:27:04.297 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:04.297 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 226491 2024-04-20T17:27:04.297 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.297 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:27:04.297 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.297 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:27:04.297 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.277237+0000", 2024-04-20T17:27:04.297 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:04.297 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.297 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.298 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:27:04.298 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.277237+0000", 2024-04-20T17:27:04.298 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:04.298 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.298 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.298 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:27:04.298 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.277239+0000", 2024-04-20T17:27:04.298 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.764e-06 2024-04-20T17:27:04.298 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.298 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.298 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:27:04.298 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.277244+0000", 2024-04-20T17:27:04.298 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2249999999999999e-06 2024-04-20T17:27:04.298 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.298 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.299 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:27:04.299 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.277255+0000", 2024-04-20T17:27:04.299 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0837e-05 2024-04-20T17:27:04.299 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.299 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.299 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:27:04.299 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.277273+0000", 2024-04-20T17:27:04.299 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7700999999999999e-05 2024-04-20T17:27:04.299 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.299 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.299 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:04.299 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.278299+0000", 2024-04-20T17:27:04.299 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0010261980000000001 2024-04-20T17:27:04.299 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.299 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.299 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:27:04.300 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.278401+0000", 2024-04-20T17:27:04.300 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000102118 2024-04-20T17:27:04.300 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.300 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.300 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:27:04.300 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.278477+0000", 2024-04-20T17:27:04.300 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5593999999999995e-05 2024-04-20T17:27:04.300 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.300 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.300 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:27:04.300 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.282131+0000", 2024-04-20T17:27:04.300 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0036543650000000001 2024-04-20T17:27:04.300 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:04.301 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:27:04.301 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:04.301 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.301 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.301 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:226495 4.7 4:e1d9a6f5:::benchmark_data_smithi171_122772_object56303:head [delete] snapc 0=[] ondisk+write+known_if_redirected e496)", 2024-04-20T17:27:04.301 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:27:04.282315+0000", 2024-04-20T17:27:04.301 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.0083730049999999993, 2024-04-20T17:27:04.301 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0085789869999999997, 2024-04-20T17:27:04.301 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:27:04.301 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:27:04.301 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:27:04.301 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:27:04.301 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:27:04.301 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:04.301 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 226495 2024-04-20T17:27:04.302 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.302 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:27:04.302 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.302 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:27:04.302 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.282315+0000", 2024-04-20T17:27:04.302 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:04.302 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.302 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.302 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:27:04.302 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.282315+0000", 2024-04-20T17:27:04.302 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:04.302 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.302 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.302 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:27:04.303 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.282317+0000", 2024-04-20T17:27:04.303 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8610000000000001e-06 2024-04-20T17:27:04.303 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.303 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.303 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:27:04.303 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.282323+0000", 2024-04-20T17:27:04.303 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3149999999999999e-06 2024-04-20T17:27:04.303 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.303 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.303 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:27:04.303 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.282336+0000", 2024-04-20T17:27:04.303 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3125000000000001e-05 2024-04-20T17:27:04.303 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.303 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.303 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:27:04.303 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.282344+0000", 2024-04-20T17:27:04.303 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6899999999999998e-06 2024-04-20T17:27:04.303 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.304 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.304 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:04.304 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.282403+0000", 2024-04-20T17:27:04.304 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8919000000000003e-05 2024-04-20T17:27:04.304 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.304 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.304 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:27:04.304 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.282533+0000", 2024-04-20T17:27:04.304 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012952800000000001 2024-04-20T17:27:04.304 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.304 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.304 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:27:04.304 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.282639+0000", 2024-04-20T17:27:04.304 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000106198 2024-04-20T17:27:04.305 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.305 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.305 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:27:04.305 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.287167+0000", 2024-04-20T17:27:04.305 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0045282789999999996 2024-04-20T17:27:04.305 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:04.305 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:27:04.305 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:04.305 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.305 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.305 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.26074.0:226505 4.e e496/445 4:7352c0d7:::benchmark_data_smithi171_122772_object56313:head v 496'13992, mlcod=496'13990)", 2024-04-20T17:27:04.305 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:27:04.287951+0000", 2024-04-20T17:27:04.305 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.0027374249999999999, 2024-04-20T17:27:04.305 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0030250390000000002, 2024-04-20T17:27:04.305 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:27:04.305 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:27:04.306 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:27:04.306 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:27:04.306 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.306 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:27:04.306 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.287951+0000", 2024-04-20T17:27:04.306 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:04.306 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.306 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.306 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:27:04.306 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.287951+0000", 2024-04-20T17:27:04.306 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:04.306 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.306 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.306 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:27:04.306 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.287953+0000", 2024-04-20T17:27:04.306 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7269999999999999e-06 2024-04-20T17:27:04.306 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.307 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.307 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:27:04.307 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.287961+0000", 2024-04-20T17:27:04.307 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6910000000000003e-06 2024-04-20T17:27:04.307 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.307 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.307 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:27:04.307 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.287975+0000", 2024-04-20T17:27:04.307 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3465000000000001e-05 2024-04-20T17:27:04.307 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.307 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.307 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:27:04.307 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.287983+0000", 2024-04-20T17:27:04.307 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1580000000000005e-06 2024-04-20T17:27:04.307 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.307 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.308 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:04.308 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.288059+0000", 2024-04-20T17:27:04.308 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6011000000000001e-05 2024-04-20T17:27:04.308 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:04.308 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:04.308 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:27:04.308 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:04.288083+0000", 2024-04-20T17:27:04.308 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4389e-05 2024-04-20T17:27:04.308 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:04.308 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:27:04.308 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:04.308 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:04.308 INFO:teuthology.orchestra.run.smithi171.stdout: ], 2024-04-20T17:27:04.308 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 4 2024-04-20T17:27:04.308 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:27:04.375 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:27:04.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:04 smithi005 ceph-mon[39562]: 4.4 scrub ok 2024-04-20T17:27:04.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:04 smithi005 ceph-mon[39562]: 4.8 scrub ok 2024-04-20T17:27:04.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:04 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3692455712' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:27:04.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:04 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:04.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:04 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:04.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:04 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:04.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:04 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:04.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:04 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:04.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:04 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:04.406 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:04.428 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:27:04.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:04.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:27:04.533 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:27:04.573 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:04 smithi171 ceph-mon[44189]: 4.4 scrub ok 2024-04-20T17:27:04.573 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:04 smithi171 ceph-mon[44189]: 4.8 scrub ok 2024-04-20T17:27:04.574 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:04 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3692455712' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:27:04.574 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:04 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:04.574 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:04 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:04.574 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:04 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:04.574 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:04 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:04.574 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:04 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:04.574 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:04 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:04.576 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:04.613 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:27:04.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:04.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:27:04.716 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:27:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:04 smithi106 ceph-mon[44423]: 4.4 scrub ok 2024-04-20T17:27:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:04 smithi106 ceph-mon[44423]: 4.8 scrub ok 2024-04-20T17:27:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:04 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3692455712' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:27:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:04 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:04 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:04 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:04 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:04 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:04 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:04.780 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:05.125 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:27:05.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:05.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:27:05.145 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-20T17:27:05.232 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:27:05.262 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:05.293 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:27:05.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:05.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:27:05.397 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:05.397 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:05.397 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:05.397 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:05.397 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:05.397 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:05.397 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:05 smithi005 ceph-mon[39562]: pgmap v1315: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-04-20T17:27:05.398 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:05 smithi005 ceph-mon[39562]: 4.9 scrub starts 2024-04-20T17:27:05.399 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:05.399 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:05.399 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:05.399 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:05.399 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:05.399 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:05.399 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:05 smithi171 ceph-mon[44189]: pgmap v1315: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-04-20T17:27:05.399 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:05 smithi171 ceph-mon[44189]: 4.9 scrub starts 2024-04-20T17:27:05.403 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:27:05.437 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:05 smithi106 ceph-mon[44423]: pgmap v1315: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-04-20T17:27:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:05 smithi106 ceph-mon[44423]: 4.9 scrub starts 2024-04-20T17:27:05.517 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:27:05.796 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:27:05.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:05.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:27:05.901 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:27:05.930 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:05.953 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:27:05.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:05.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:27:06.059 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:27:06.089 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:06.118 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:27:06.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:06.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:27:06.224 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:27:06.273 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:06.429 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:27:06.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:06.435 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:27:06.536 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:27:06.568 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:06.592 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:27:06.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:06.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:27:06.697 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:27:06.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:06 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:27:06.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:06 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:27:06.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:06 smithi005 ceph-mon[39562]: 4.6 scrub starts 2024-04-20T17:27:06.730 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:06 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:27:06.730 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:06 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:27:06.730 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:06 smithi106 ceph-mon[44423]: 4.6 scrub starts 2024-04-20T17:27:06.732 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:06.762 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:27:06.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:06.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:27:06.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:06 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:27:06.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:06 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:27:06.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:06 smithi171 ceph-mon[44189]: 4.6 scrub starts 2024-04-20T17:27:06.869 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:27:06.899 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:06.923 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:27:06.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:06.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:27:07.030 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:27:07.087 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:07.117 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:27:07.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:07.129 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:27:07.161 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:27:07.229 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:27:07.268 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:07.423 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:27:07.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:07.428 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:27:07.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:07 smithi005 ceph-mon[39562]: pgmap v1316: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-04-20T17:27:07.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:07 smithi005 ceph-mon[39562]: 4.9 scrub ok 2024-04-20T17:27:07.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:07 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3495507066' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:27:07.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:07 smithi005 ceph-mon[39562]: from='client.26330 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:27:07.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:07 smithi005 ceph-mon[39562]: 4.6 scrub ok 2024-04-20T17:27:07.469 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:27:07.529 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:27:07.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:07 smithi171 ceph-mon[44189]: pgmap v1316: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-04-20T17:27:07.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:07 smithi171 ceph-mon[44189]: 4.9 scrub ok 2024-04-20T17:27:07.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:07 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3495507066' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:27:07.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:07 smithi171 ceph-mon[44189]: from='client.26330 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:27:07.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:07 smithi171 ceph-mon[44189]: 4.6 scrub ok 2024-04-20T17:27:07.594 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:07.734 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:27:07.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:07.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:27:07.743 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:27:07.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:07 smithi106 ceph-mon[44423]: pgmap v1316: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-04-20T17:27:07.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:07 smithi106 ceph-mon[44423]: 4.9 scrub ok 2024-04-20T17:27:07.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:07 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3495507066' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:27:07.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:07 smithi106 ceph-mon[44423]: from='client.26330 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:27:07.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:07 smithi106 ceph-mon[44423]: 4.6 scrub ok 2024-04-20T17:27:07.815 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:27:07.816 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:27:07.816 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-04-20T17:27:07.816 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-04-20T17:27:07.816 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:27:07.838 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:27:07.869 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:07.890 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:27:07.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:07.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:27:07.994 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:27:08.039 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:08.120 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:27:08.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:08.124 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:27:08.225 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:27:08.266 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:08.307 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:27:08.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:08.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:27:08.409 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:27:08.423 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:08 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:27:08.424 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:08 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:27:08.424 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:08 smithi171 ceph-mon[44189]: from='client.26330 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:27:08.424 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:08 smithi171 ceph-mon[44189]: osdmap e497: 12 total, 12 up, 12 in 2024-04-20T17:27:08.442 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:08.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:08 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:27:08.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:08 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:27:08.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:08 smithi005 ceph-mon[39562]: from='client.26330 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:27:08.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:08 smithi005 ceph-mon[39562]: osdmap e497: 12 total, 12 up, 12 in 2024-04-20T17:27:08.655 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:27:08.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:08.660 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:27:08.700 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-20T17:27:08.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:08 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:27:08.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:08 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:27:08.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:08 smithi106 ceph-mon[44423]: from='client.26330 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:27:08.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:08 smithi106 ceph-mon[44423]: osdmap e497: 12 total, 12 up, 12 in 2024-04-20T17:27:08.761 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:27:08.796 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:08.820 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:27:08.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:08.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:27:08.925 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:27:08.980 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:09.008 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:27:09.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:09.011 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:27:09.112 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:27:09.133 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:27:09.153 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:09.514 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:27:09.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:09.518 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:27:09.618 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:27:09.652 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:09.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:09 smithi005 ceph-mon[39562]: pgmap v1318: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:27:09.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:09 smithi005 ceph-mon[39562]: 4.0 scrub ok 2024-04-20T17:27:09.699 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:27:09.702 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:09 smithi171 ceph-mon[44189]: pgmap v1318: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:27:09.702 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:09 smithi171 ceph-mon[44189]: 4.0 scrub ok 2024-04-20T17:27:09.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:09.703 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:27:09.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:09 smithi106 ceph-mon[44423]: pgmap v1318: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:27:09.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:09 smithi106 ceph-mon[44423]: 4.0 scrub ok 2024-04-20T17:27:09.805 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:27:09.836 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:10.050 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:27:10.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:10.055 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:27:10.156 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:27:10.187 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:10.279 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:27:10.482 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:27:10.536 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:27:10.581 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:27:10.620 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:27:10.649 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:27:10.682 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:27:10.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:10 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:27:10.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:10 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:27:10.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:10 smithi005 ceph-mon[39562]: osdmap e498: 12 total, 12 up, 12 in 2024-04-20T17:27:10.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:10 smithi005 ceph-mon[39562]: 4.b scrub starts 2024-04-20T17:27:10.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:10 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1082283597' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:27:10.734 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:27:10.738 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:27:10.741 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:10 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:27:10.741 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:10 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:27:10.741 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:10 smithi171 ceph-mon[44189]: osdmap e498: 12 total, 12 up, 12 in 2024-04-20T17:27:10.741 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:10 smithi171 ceph-mon[44189]: 4.b scrub starts 2024-04-20T17:27:10.741 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:10 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1082283597' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:27:10.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:10.742 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:27:10.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:10 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:27:10.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:10 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:27:10.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:10 smithi106 ceph-mon[44423]: osdmap e498: 12 total, 12 up, 12 in 2024-04-20T17:27:10.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:10 smithi106 ceph-mon[44423]: 4.b scrub starts 2024-04-20T17:27:10.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:10 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1082283597' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:27:10.768 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:27:10.788 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:27:10.838 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:27:10.842 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:27:10.875 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:10.880 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:27:10.914 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:27:10.938 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:27:10.942 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:27:10.943 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:27:10.943 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:27:10.943 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:27:10.943 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:27:10.943 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:27:10.943 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:27:10.943 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:27:10.943 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:27:10.943 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:27:10.943 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:27:10.943 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:27:11.029 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:27:11.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:11.032 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:27:11.133 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:27:11.165 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:11.186 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:27:11.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:11.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:27:11.291 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:27:11.350 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:11.418 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:27:11.418 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:27:11.471 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:11 smithi106 ceph-mon[44423]: pgmap v1320: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s 2024-04-20T17:27:11.472 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:11.472 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:11.472 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:11.472 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:11.472 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:11.472 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:11.472 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:11 smithi106 ceph-mon[44423]: 4.b scrub ok 2024-04-20T17:27:11.472 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:11.472 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:11.493 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:27:11.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:11.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:27:11.599 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:27:11.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:11 smithi005 ceph-mon[39562]: pgmap v1320: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s 2024-04-20T17:27:11.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:11.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:11.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:11.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:11.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:11.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:11.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:11 smithi005 ceph-mon[39562]: 4.b scrub ok 2024-04-20T17:27:11.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:11.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:11.635 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:11.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:11 smithi171 ceph-mon[44189]: pgmap v1320: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s 2024-04-20T17:27:11.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:11.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:11.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:11.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:11.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:11.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:11.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:11 smithi171 ceph-mon[44189]: 4.b scrub ok 2024-04-20T17:27:11.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:11.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:11.841 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:27:11.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:11.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:27:11.954 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:27:12.006 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:12.328 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-20T17:27:12.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:12.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:12.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:12.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:12.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:12 smithi171 ceph-mon[44189]: 4.d scrub starts 2024-04-20T17:27:12.713 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:27:12.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:12.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:12.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:12.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:12.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:12 smithi106 ceph-mon[44423]: 4.d scrub starts 2024-04-20T17:27:12.840 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:12.841 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:12.841 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:12.841 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:12.841 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:12 smithi005 ceph-mon[39562]: 4.d scrub starts 2024-04-20T17:27:12.880 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:27:12.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:12.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:27:12.987 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:27:13.019 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:13.041 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:27:13.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:13.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:27:13.145 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:27:13.173 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:13.196 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:27:13.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:13.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:27:13.299 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:27:13.333 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:13.356 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:27:13.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:13.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:27:13.460 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:27:13.538 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:13.589 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:27:13.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:13.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:27:13.695 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:27:13.752 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:13 smithi005 ceph-mon[39562]: pgmap v1321: 17 pgs: 17 active+clean; 6.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:27:13.752 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:13 smithi005 ceph-mon[39562]: 4.d scrub ok 2024-04-20T17:27:13.758 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:13.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:13 smithi106 ceph-mon[44423]: pgmap v1321: 17 pgs: 17 active+clean; 6.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:27:13.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:13 smithi106 ceph-mon[44423]: 4.d scrub ok 2024-04-20T17:27:13.800 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:27:13.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:13.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:27:13.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:13 smithi171 ceph-mon[44189]: pgmap v1321: 17 pgs: 17 active+clean; 6.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:27:13.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:13 smithi171 ceph-mon[44189]: 4.d scrub ok 2024-04-20T17:27:13.884 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:27:13.904 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:27:13.990 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:14.017 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:27:14.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:14.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:27:14.130 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:27:14.177 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:14.282 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:27:14.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:14.286 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:27:14.302 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:27:14.387 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:27:14.416 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:14.448 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:27:14.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:14.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:27:14.520 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:27:14.553 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:27:14.585 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:14.615 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:27:14.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:14.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:27:14.724 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:27:14.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:14 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:27:14.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:14 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:27:14.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:14 smithi106 ceph-mon[44423]: osdmap e499: 12 total, 12 up, 12 in 2024-04-20T17:27:14.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:14 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4230677841' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:27:14.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:14 smithi106 ceph-mon[44423]: from='client.26342 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:27:14.761 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:14 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:27:14.762 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:14 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:27:14.762 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:14 smithi171 ceph-mon[44189]: osdmap e499: 12 total, 12 up, 12 in 2024-04-20T17:27:14.762 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:14 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4230677841' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:27:14.762 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:14 smithi171 ceph-mon[44189]: from='client.26342 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:27:14.765 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:14.891 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:14 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:27:14.891 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:14 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:27:14.891 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:14 smithi005 ceph-mon[39562]: osdmap e499: 12 total, 12 up, 12 in 2024-04-20T17:27:14.891 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:14 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4230677841' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:27:14.891 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:14 smithi005 ceph-mon[39562]: from='client.26342 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:27:15.123 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:27:15.128 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:27:15.128 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:27:15.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:15.133 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:27:15.234 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:27:15.267 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:15.301 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:27:15.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:15.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:27:15.408 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:27:15.456 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:15.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:15 smithi005 ceph-mon[39562]: pgmap v1323: 17 pgs: 17 active+clean; 6.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:27:15.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:15 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:27:15.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:15 smithi005 ceph-mon[39562]: from='client.26342 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:27:15.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:15 smithi005 ceph-mon[39562]: osdmap e500: 12 total, 12 up, 12 in 2024-04-20T17:27:15.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:15 smithi106 ceph-mon[44423]: pgmap v1323: 17 pgs: 17 active+clean; 6.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:27:15.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:15 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:27:15.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:15 smithi106 ceph-mon[44423]: from='client.26342 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:27:15.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:15 smithi106 ceph-mon[44423]: osdmap e500: 12 total, 12 up, 12 in 2024-04-20T17:27:15.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:15 smithi171 ceph-mon[44189]: pgmap v1323: 17 pgs: 17 active+clean; 6.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:27:15.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:15 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:27:15.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:15 smithi171 ceph-mon[44189]: from='client.26342 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:27:15.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:15 smithi171 ceph-mon[44189]: osdmap e500: 12 total, 12 up, 12 in 2024-04-20T17:27:16.017 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:27:16.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:16.021 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:27:16.026 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-20T17:27:16.122 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:27:16.165 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:16.191 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:27:16.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:16.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:27:16.302 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:27:16.363 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:16.444 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:27:16.451 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:27:16.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:16.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:27:16.557 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:27:16.586 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:16.612 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:27:16.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:16.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:27:16.715 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:27:16.748 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:16.772 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:27:16.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:16.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:27:16.876 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:27:16.944 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:16.973 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:27:16.977 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:27:16.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:16.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:27:16.997 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:27:17.029 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:27:17.052 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:27:17.087 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:27:17.090 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:27:17.120 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:27:17.126 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:17.150 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:27:17.174 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:27:17.207 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:27:17.241 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:27:17.281 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:27:17.313 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:27:17.315 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:27:17.315 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:27:17.315 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:27:17.315 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:27:17.315 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:27:17.316 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:27:17.316 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:27:17.316 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:27:17.316 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:27:17.316 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:27:17.316 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:27:17.316 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:27:17.333 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:27:17.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:17 smithi171 ceph-mon[44189]: pgmap v1325: 17 pgs: 17 active+clean; 6.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s 2024-04-20T17:27:17.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:17 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:27:17.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:17 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:27:17.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:17 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2635630590' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:27:17.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:17.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:17.338 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:27:17.439 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:27:17.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:17 smithi005 ceph-mon[39562]: pgmap v1325: 17 pgs: 17 active+clean; 6.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s 2024-04-20T17:27:17.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:17 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:27:17.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:17 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:27:17.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:17 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2635630590' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:27:17.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:17.466 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:17 smithi106 ceph-mon[44423]: pgmap v1325: 17 pgs: 17 active+clean; 6.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s 2024-04-20T17:27:17.466 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:17 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:27:17.466 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:17 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:27:17.466 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:17 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2635630590' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:27:17.466 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:17.468 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:17.492 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:27:17.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:17.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:27:17.599 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:27:17.642 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:17.765 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:27:18.405 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:27:18.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:18.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:27:18.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:18 smithi005 ceph-mon[39562]: osdmap e501: 12 total, 12 up, 12 in 2024-04-20T17:27:18.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:18 smithi106 ceph-mon[44423]: osdmap e501: 12 total, 12 up, 12 in 2024-04-20T17:27:18.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.512 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:27:18.533 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:27:18.541 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:18.565 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:27:18.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:18.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:27:18.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:18 smithi171 ceph-mon[44189]: osdmap e501: 12 total, 12 up, 12 in 2024-04-20T17:27:18.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:18.669 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:27:18.705 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:18.737 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:27:18.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:18.739 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:27:18.840 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:27:18.884 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:18.917 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:27:18.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:18.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:27:19.027 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:27:19.055 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:19.084 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:27:19.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:19.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:27:19.102 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:27:19.102 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:27:19.103 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-04-20T17:27:19.103 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-04-20T17:27:19.103 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:27:19.188 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:27:19.239 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:19.398 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:27:19.399 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:19 smithi171 ceph-mon[44189]: pgmap v1327: 17 pgs: 17 active+clean; 6.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s 2024-04-20T17:27:19.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:19.401 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:27:19.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:19 smithi005 ceph-mon[39562]: pgmap v1327: 17 pgs: 17 active+clean; 6.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s 2024-04-20T17:27:19.501 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:27:19.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:19 smithi106 ceph-mon[44423]: pgmap v1327: 17 pgs: 17 active+clean; 6.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s 2024-04-20T17:27:19.538 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:19.897 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:27:19.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:19.902 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:27:19.931 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-20T17:27:20.003 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:27:20.064 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:20.113 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:27:20.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:20.121 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:27:20.222 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:27:20.271 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:20.312 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:27:20.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:20.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:27:20.353 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:27:20.418 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:27:20.435 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:27:20.452 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:20.480 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:27:20.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:20.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:27:20.588 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:27:20.666 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:20.709 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:27:20.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:20.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:27:20.817 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:27:20.868 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:20.911 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:27:20.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:20.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:27:20.937 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:27:21.018 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:27:21.055 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:21.177 INFO:tasks.thrashosds.thrasher:Recovered, killing an osd 2024-04-20T17:27:21.177 INFO:tasks.thrashosds.thrasher:Killing osd 6, live_osds are [0, 1, 3, 4, 5, 6, 8, 9, 10, 11, 2, 7] 2024-04-20T17:27:21.177 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-20T17:27:21.178 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.6 2024-04-20T17:27:21.201 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:27:21.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:21.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:27:21.253 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:27:21.305 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:27:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:21 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:27:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:21 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:27:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:21 smithi171 ceph-mon[44189]: osdmap e502: 12 total, 12 up, 12 in 2024-04-20T17:27:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:21 smithi171 ceph-mon[44189]: pgmap v1329: 17 pgs: 17 active+clean; 6.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-04-20T17:27:21.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:21 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2934680317' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:27:21.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:21 smithi171 ceph-mon[44189]: from='client.26269 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:27:21.364 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:21.394 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:27:21.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:21.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:27:21.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:21 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:27:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:21 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:27:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:21 smithi005 ceph-mon[39562]: osdmap e502: 12 total, 12 up, 12 in 2024-04-20T17:27:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:21 smithi005 ceph-mon[39562]: pgmap v1329: 17 pgs: 17 active+clean; 6.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-04-20T17:27:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:21 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2934680317' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:27:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:21 smithi005 ceph-mon[39562]: from='client.26269 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:27:21.502 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:27:21.511 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:27:21 smithi106 systemd[1]: Stopping Ceph osd.6 for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:27:21.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:21 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:27:21.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:21 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:27:21.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:21 smithi106 ceph-mon[44423]: osdmap e502: 12 total, 12 up, 12 in 2024-04-20T17:27:21.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:21 smithi106 ceph-mon[44423]: pgmap v1329: 17 pgs: 17 active+clean; 6.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-04-20T17:27:21.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:21 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2934680317' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:27:21.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:21 smithi106 ceph-mon[44423]: from='client.26269 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:27:21.543 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:21.753 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:27:21.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:21.759 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:27:21.860 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:27:21.915 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:21.946 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:27:21.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:21.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:27:22.012 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:27:21 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[65549]: 2024-04-20T17:27:21.552+0000 7f88275e9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T17:27:22.012 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:27:21 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[65549]: 2024-04-20T17:27:21.552+0000 7f88275e9700 -1 osd.6 503 *** Got signal Terminated *** 2024-04-20T17:27:22.012 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:27:21 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[65549]: 2024-04-20T17:27:21.552+0000 7f88275e9700 -1 osd.6 503 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T17:27:22.050 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:27:22.090 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:22.134 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:27:22.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:22.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:27:22.243 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:27:22.273 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:22.292 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:27:22.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:22.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:27:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:22 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:27:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:22 smithi171 ceph-mon[44189]: from='client.26269 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:27:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:22 smithi171 ceph-mon[44189]: osdmap e503: 12 total, 12 up, 12 in 2024-04-20T17:27:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:22 smithi171 ceph-mon[44189]: osd.6 marked itself down and dead 2024-04-20T17:27:22.352 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:27:22.353 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:27:22.353 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:27:22.353 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:27:22.353 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.353 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:122903 3.f 3:f1e79158:::benchmark_data_smithi171_110765_object122902:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e384)", 2024-04-20T17:27:22.353 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:45.056333+0000", 2024-04-20T17:27:22.353 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 337.294354403, 2024-04-20T17:27:22.353 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.122128339, 2024-04-20T17:27:22.353 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:27:22.353 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:27:22.353 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:22.353 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:27:22.354 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:27:22.354 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:22.354 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 122903 2024-04-20T17:27:22.354 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.354 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:27:22.354 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.354 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:27:22.354 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.056333+0000", 2024-04-20T17:27:22.354 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.354 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.354 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.354 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:27:22.354 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.056333+0000", 2024-04-20T17:27:22.354 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.355 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.355 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.355 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:27:22.355 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.056335+0000", 2024-04-20T17:27:22.355 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9489999999999999e-06 2024-04-20T17:27:22.355 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.355 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.355 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:27:22.355 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.056364+0000", 2024-04-20T17:27:22.355 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8530999999999998e-05 2024-04-20T17:27:22.355 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.355 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.355 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:27:22.356 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.056373+0000", 2024-04-20T17:27:22.356 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2189999999999993e-06 2024-04-20T17:27:22.356 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.356 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.356 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:27:22.356 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.056380+0000", 2024-04-20T17:27:22.356 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4389999999999997e-06 2024-04-20T17:27:22.356 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.356 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.356 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:22.356 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.088954+0000", 2024-04-20T17:27:22.356 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032574918000000001 2024-04-20T17:27:22.356 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.356 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.357 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:27:22.357 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.089066+0000", 2024-04-20T17:27:22.357 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000111227 2024-04-20T17:27:22.357 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.357 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.357 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:27:22.357 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.089201+0000", 2024-04-20T17:27:22.357 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000135728 2024-04-20T17:27:22.357 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.357 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.357 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:27:22.357 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.178342+0000", 2024-04-20T17:27:22.357 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.089140237999999997 2024-04-20T17:27:22.357 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.357 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.358 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:22.358 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.178368+0000", 2024-04-20T17:27:22.358 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6364e-05 2024-04-20T17:27:22.358 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.358 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.358 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:27:22.358 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.178405+0000", 2024-04-20T17:27:22.358 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7054999999999997e-05 2024-04-20T17:27:22.358 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.358 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.358 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:27:22.358 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:45.178462+0000", 2024-04-20T17:27:22.358 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6671e-05 2024-04-20T17:27:22.359 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.359 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:27:22.359 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.359 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.359 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.359 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:125888 3.f 3:fdac0d8e:::benchmark_data_smithi171_110765_object125887:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-04-20T17:27:22.359 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:47.621625+0000", 2024-04-20T17:27:22.359 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 334.72906282700001, 2024-04-20T17:27:22.359 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.120494553, 2024-04-20T17:27:22.359 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:27:22.359 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:27:22.359 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:22.359 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:27:22.359 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:27:22.360 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:22.360 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 125888 2024-04-20T17:27:22.360 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.360 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:27:22.360 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.360 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:27:22.360 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.621625+0000", 2024-04-20T17:27:22.360 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.360 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.360 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.360 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:27:22.360 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.621625+0000", 2024-04-20T17:27:22.360 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.360 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.360 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.361 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:27:22.361 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.621627+0000", 2024-04-20T17:27:22.361 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.663e-06 2024-04-20T17:27:22.361 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.361 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.361 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:27:22.361 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.621654+0000", 2024-04-20T17:27:22.361 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7107e-05 2024-04-20T17:27:22.361 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.361 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.361 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:27:22.362 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.621664+0000", 2024-04-20T17:27:22.362 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0658e-05 2024-04-20T17:27:22.362 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.362 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.362 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:27:22.362 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.621675+0000", 2024-04-20T17:27:22.363 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0447e-05 2024-04-20T17:27:22.363 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.363 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.363 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:22.363 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.721943+0000", 2024-04-20T17:27:22.363 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.100268565 2024-04-20T17:27:22.363 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.363 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.363 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:27:22.363 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.722065+0000", 2024-04-20T17:27:22.363 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000121941 2024-04-20T17:27:22.363 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.363 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.363 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:27:22.363 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.722151+0000", 2024-04-20T17:27:22.364 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5230999999999998e-05 2024-04-20T17:27:22.364 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.364 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.364 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:22.364 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.729406+0000", 2024-04-20T17:27:22.364 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0072549449999999996 2024-04-20T17:27:22.364 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.364 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.364 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:27:22.364 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.742037+0000", 2024-04-20T17:27:22.364 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.012631008000000001 2024-04-20T17:27:22.364 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.364 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.364 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:27:22.364 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.742072+0000", 2024-04-20T17:27:22.365 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5394e-05 2024-04-20T17:27:22.365 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.365 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.365 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:27:22.365 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.742120+0000", 2024-04-20T17:27:22.365 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7593999999999999e-05 2024-04-20T17:27:22.365 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.365 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:27:22.365 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.365 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.365 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.365 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:125905 3.f 3:fffcbe7e:::benchmark_data_smithi171_110765_object125904:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-04-20T17:27:22.366 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:47.635413+0000", 2024-04-20T17:27:22.366 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 334.71527460700003, 2024-04-20T17:27:22.366 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.120174664, 2024-04-20T17:27:22.366 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:27:22.366 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:27:22.366 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:22.366 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:27:22.366 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:27:22.366 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:22.366 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 125905 2024-04-20T17:27:22.366 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.366 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:27:22.366 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.366 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:27:22.367 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.635413+0000", 2024-04-20T17:27:22.367 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.367 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.367 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.367 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:27:22.367 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.635413+0000", 2024-04-20T17:27:22.367 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.367 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.367 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.367 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:27:22.367 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.635414+0000", 2024-04-20T17:27:22.367 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5300000000000004e-07 2024-04-20T17:27:22.367 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.367 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.368 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:27:22.368 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.635436+0000", 2024-04-20T17:27:22.368 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2535999999999999e-05 2024-04-20T17:27:22.368 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.368 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.368 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:27:22.368 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.635441+0000", 2024-04-20T17:27:22.368 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5619999999999997e-06 2024-04-20T17:27:22.368 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.368 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.368 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:27:22.369 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.635445+0000", 2024-04-20T17:27:22.369 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7129999999999999e-06 2024-04-20T17:27:22.369 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.369 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.369 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:22.369 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.742154+0000", 2024-04-20T17:27:22.369 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.10670937 2024-04-20T17:27:22.369 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.369 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.369 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:27:22.369 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.742286+0000", 2024-04-20T17:27:22.369 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013218099999999999 2024-04-20T17:27:22.369 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.369 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.369 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:27:22.370 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.742436+0000", 2024-04-20T17:27:22.370 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000150118 2024-04-20T17:27:22.370 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.370 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.370 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:27:22.370 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.755452+0000", 2024-04-20T17:27:22.370 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013015633 2024-04-20T17:27:22.370 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.370 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.370 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:22.370 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.755494+0000", 2024-04-20T17:27:22.370 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1928999999999998e-05 2024-04-20T17:27:22.370 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.370 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.371 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:27:22.371 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.755534+0000", 2024-04-20T17:27:22.371 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9906e-05 2024-04-20T17:27:22.371 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.371 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.371 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:27:22.371 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:47.755588+0000", 2024-04-20T17:27:22.371 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4063000000000001e-05 2024-04-20T17:27:22.371 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.371 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:27:22.371 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.371 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.371 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.371 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:128849 3.f 3:f595832e:::benchmark_data_smithi171_110765_object128848:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e386)", 2024-04-20T17:27:22.371 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:21:50.229225+0000", 2024-04-20T17:27:22.372 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 332.12146293799998, 2024-04-20T17:27:22.372 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.122216725, 2024-04-20T17:27:22.372 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:27:22.372 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:27:22.372 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:22.372 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:27:22.372 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:27:22.372 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:22.372 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 128849 2024-04-20T17:27:22.372 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.372 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:27:22.372 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.372 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:27:22.372 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.229225+0000", 2024-04-20T17:27:22.372 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.373 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.373 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.373 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:27:22.373 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.229225+0000", 2024-04-20T17:27:22.373 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.373 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.373 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.373 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:27:22.373 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.229226+0000", 2024-04-20T17:27:22.373 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3880000000000001e-06 2024-04-20T17:27:22.373 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.373 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.373 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:27:22.373 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.229246+0000", 2024-04-20T17:27:22.373 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9684e-05 2024-04-20T17:27:22.374 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.374 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.374 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:27:22.374 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.229254+0000", 2024-04-20T17:27:22.374 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9370000000000004e-06 2024-04-20T17:27:22.374 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.374 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.374 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:27:22.374 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.229259+0000", 2024-04-20T17:27:22.374 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9400000000000001e-06 2024-04-20T17:27:22.374 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.374 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.374 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:22.374 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.342986+0000", 2024-04-20T17:27:22.374 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.11372665 2024-04-20T17:27:22.375 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.375 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.375 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:27:22.375 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.343141+0000", 2024-04-20T17:27:22.375 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015515900000000001 2024-04-20T17:27:22.375 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.375 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.375 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:27:22.375 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.343265+0000", 2024-04-20T17:27:22.375 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012449299999999999 2024-04-20T17:27:22.375 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.375 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.375 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:22.375 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.351255+0000", 2024-04-20T17:27:22.375 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0079898290000000004 2024-04-20T17:27:22.376 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.376 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.376 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:27:22.376 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.351316+0000", 2024-04-20T17:27:22.376 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1113999999999998e-05 2024-04-20T17:27:22.376 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.376 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.376 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:27:22.376 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.351370+0000", 2024-04-20T17:27:22.376 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3705000000000002e-05 2024-04-20T17:27:22.376 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.376 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.376 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:27:22.376 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:21:50.351442+0000", 2024-04-20T17:27:22.376 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1826e-05 2024-04-20T17:27:22.377 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.377 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:27:22.377 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.377 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.377 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.377 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:285694 3.4 3:2277b9d8:::benchmark_data_smithi171_110765_object142479:head [delete] snapc 0=[] ondisk+write+known_if_redirected e435)", 2024-04-20T17:27:22.377 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:24:11.480895+0000", 2024-04-20T17:27:22.377 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 190.869792881, 2024-04-20T17:27:22.377 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.122091148, 2024-04-20T17:27:22.377 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:27:22.377 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:27:22.377 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:22.377 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:27:22.377 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:27:22.377 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:22.378 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 285694 2024-04-20T17:27:22.378 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.378 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:27:22.378 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.378 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:27:22.378 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.480895+0000", 2024-04-20T17:27:22.378 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.378 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.378 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.378 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:27:22.378 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.480895+0000", 2024-04-20T17:27:22.378 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.378 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.378 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.378 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:27:22.379 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.480896+0000", 2024-04-20T17:27:22.379 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0800000000000004e-07 2024-04-20T17:27:22.379 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.379 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.379 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:27:22.379 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.480899+0000", 2024-04-20T17:27:22.379 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0929999999999999e-06 2024-04-20T17:27:22.379 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.379 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.379 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:27:22.379 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.480905+0000", 2024-04-20T17:27:22.379 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8810000000000001e-06 2024-04-20T17:27:22.379 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.379 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.379 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:27:22.380 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.480910+0000", 2024-04-20T17:27:22.380 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6189999999999999e-06 2024-04-20T17:27:22.380 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.380 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.380 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:22.380 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.480984+0000", 2024-04-20T17:27:22.380 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4229999999999999e-05 2024-04-20T17:27:22.380 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.380 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.380 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-04-20T17:27:22.380 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.480988+0000", 2024-04-20T17:27:22.380 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4599999999999999e-06 2024-04-20T17:27:22.380 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.380 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.380 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:22.381 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.571259+0000", 2024-04-20T17:27:22.381 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.090271069999999995 2024-04-20T17:27:22.381 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.381 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.381 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:27:22.381 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.571348+0000", 2024-04-20T17:27:22.381 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9424000000000003e-05 2024-04-20T17:27:22.381 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.381 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.381 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:27:22.381 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.571471+0000", 2024-04-20T17:27:22.381 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000122745 2024-04-20T17:27:22.381 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.381 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.381 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:27:22.382 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.585026+0000", 2024-04-20T17:27:22.382 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013554754 2024-04-20T17:27:22.382 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.382 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.382 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:22.382 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.602885+0000", 2024-04-20T17:27:22.382 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.017859333000000002 2024-04-20T17:27:22.382 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.382 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.382 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:27:22.382 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.602939+0000", 2024-04-20T17:27:22.382 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3983000000000001e-05 2024-04-20T17:27:22.382 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.382 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.382 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:27:22.383 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.602986+0000", 2024-04-20T17:27:22.383 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6947999999999998e-05 2024-04-20T17:27:22.383 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.383 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:27:22.383 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.383 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.383 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.383 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:285710 3.4 3:217e946d:::benchmark_data_smithi171_110765_object142495:head [delete] snapc 0=[] ondisk+write+known_if_redirected e435)", 2024-04-20T17:27:22.383 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:24:11.480918+0000", 2024-04-20T17:27:22.383 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 190.86976998700001, 2024-04-20T17:27:22.383 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.122163023, 2024-04-20T17:27:22.383 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:27:22.383 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:27:22.383 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:22.383 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:27:22.384 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:27:22.384 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:22.384 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 285710 2024-04-20T17:27:22.384 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.384 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:27:22.384 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.384 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:27:22.384 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.480918+0000", 2024-04-20T17:27:22.384 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.384 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.384 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.384 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:27:22.384 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.480918+0000", 2024-04-20T17:27:22.384 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.384 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.385 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.385 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:27:22.385 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.480919+0000", 2024-04-20T17:27:22.385 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3399999999999998e-07 2024-04-20T17:27:22.385 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.385 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.385 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:27:22.385 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.480922+0000", 2024-04-20T17:27:22.385 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1549999999999999e-06 2024-04-20T17:27:22.385 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.385 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.385 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:27:22.385 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.480927+0000", 2024-04-20T17:27:22.385 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1680000000000003e-06 2024-04-20T17:27:22.385 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.386 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.386 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:27:22.386 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.480932+0000", 2024-04-20T17:27:22.386 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9309999999999999e-06 2024-04-20T17:27:22.386 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.386 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.386 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:22.386 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.481007+0000", 2024-04-20T17:27:22.386 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5519999999999995e-05 2024-04-20T17:27:22.386 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.386 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.386 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-04-20T17:27:22.386 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.481011+0000", 2024-04-20T17:27:22.386 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3960000000000002e-06 2024-04-20T17:27:22.386 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.387 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.387 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:22.387 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.575868+0000", 2024-04-20T17:27:22.387 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.094857069000000002 2024-04-20T17:27:22.387 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.387 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.387 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:27:22.387 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.575986+0000", 2024-04-20T17:27:22.387 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000117772 2024-04-20T17:27:22.387 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.387 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.387 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:27:22.387 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.576103+0000", 2024-04-20T17:27:22.387 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000117655 2024-04-20T17:27:22.387 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.388 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.388 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:27:22.388 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.585032+0000", 2024-04-20T17:27:22.388 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.008928933 2024-04-20T17:27:22.388 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.388 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.388 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:22.388 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.603027+0000", 2024-04-20T17:27:22.388 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.017994448999999999 2024-04-20T17:27:22.388 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.388 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.388 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:27:22.388 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.603055+0000", 2024-04-20T17:27:22.388 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8691e-05 2024-04-20T17:27:22.388 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.389 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.389 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:27:22.389 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.603081+0000", 2024-04-20T17:27:22.389 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5550000000000001e-05 2024-04-20T17:27:22.389 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.389 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:27:22.389 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.389 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.389 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.389 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:285716 3.4 3:283f9660:::benchmark_data_smithi171_110765_object142501:head [delete] snapc 0=[] ondisk+write+known_if_redirected e435)", 2024-04-20T17:27:22.389 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:24:11.480938+0000", 2024-04-20T17:27:22.389 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 190.86974935000001, 2024-04-20T17:27:22.389 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.122261565, 2024-04-20T17:27:22.389 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:27:22.389 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:27:22.390 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:22.390 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:27:22.390 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:27:22.390 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:22.390 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 285716 2024-04-20T17:27:22.390 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.390 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:27:22.390 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.390 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:27:22.390 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.480938+0000", 2024-04-20T17:27:22.390 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.390 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.390 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.390 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:27:22.390 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.480938+0000", 2024-04-20T17:27:22.391 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.391 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.391 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.391 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:27:22.391 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.480939+0000", 2024-04-20T17:27:22.391 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3799999999999997e-07 2024-04-20T17:27:22.391 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.391 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.391 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:27:22.391 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.480942+0000", 2024-04-20T17:27:22.391 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9450000000000002e-06 2024-04-20T17:27:22.391 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.391 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.391 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:27:22.391 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.480949+0000", 2024-04-20T17:27:22.392 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7880000000000001e-06 2024-04-20T17:27:22.392 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.392 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.392 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:27:22.392 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.480955+0000", 2024-04-20T17:27:22.392 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7420000000000003e-06 2024-04-20T17:27:22.392 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.392 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.392 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:22.392 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.481030+0000", 2024-04-20T17:27:22.392 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5136999999999996e-05 2024-04-20T17:27:22.392 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.392 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.392 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-04-20T17:27:22.392 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.481033+0000", 2024-04-20T17:27:22.392 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3009999999999998e-06 2024-04-20T17:27:22.393 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.393 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.393 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:22.393 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.579708+0000", 2024-04-20T17:27:22.393 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.098674569000000004 2024-04-20T17:27:22.393 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.393 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.393 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:27:22.393 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.579822+0000", 2024-04-20T17:27:22.393 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000114714 2024-04-20T17:27:22.394 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.394 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.394 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:27:22.394 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.579924+0000", 2024-04-20T17:27:22.394 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000102104 2024-04-20T17:27:22.395 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.395 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.395 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:27:22.395 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.602829+0000", 2024-04-20T17:27:22.395 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022904592000000001 2024-04-20T17:27:22.395 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.395 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.395 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:22.395 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.603124+0000", 2024-04-20T17:27:22.395 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00029547200000000002 2024-04-20T17:27:22.395 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.395 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.395 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:27:22.395 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.603165+0000", 2024-04-20T17:27:22.395 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0679000000000001e-05 2024-04-20T17:27:22.396 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.396 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.396 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:27:22.396 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.603200+0000", 2024-04-20T17:27:22.396 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4884e-05 2024-04-20T17:27:22.396 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.396 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:27:22.396 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.396 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.396 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.397 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25802.0:285763 3.4 3:2c7720fd:::benchmark_data_smithi171_110765_object142548:head [delete] snapc 0=[] ondisk+write+known_if_redirected e435)", 2024-04-20T17:27:22.397 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:24:11.480961+0000", 2024-04-20T17:27:22.397 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 190.86972666200001, 2024-04-20T17:27:22.397 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.12232810299999999, 2024-04-20T17:27:22.397 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:27:22.397 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:27:22.397 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:22.397 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:27:22.397 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25802", 2024-04-20T17:27:22.397 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:22.397 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 285763 2024-04-20T17:27:22.397 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.397 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:27:22.397 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.397 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:27:22.398 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.480961+0000", 2024-04-20T17:27:22.398 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.398 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.398 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.398 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:27:22.398 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.480961+0000", 2024-04-20T17:27:22.398 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.398 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.398 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.398 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:27:22.398 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.480962+0000", 2024-04-20T17:27:22.398 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2799999999999996e-07 2024-04-20T17:27:22.398 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.398 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.398 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:27:22.399 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.480965+0000", 2024-04-20T17:27:22.399 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8949999999999998e-06 2024-04-20T17:27:22.399 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.399 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.399 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:27:22.399 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.480970+0000", 2024-04-20T17:27:22.399 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6160000000000001e-06 2024-04-20T17:27:22.399 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.399 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.399 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:27:22.399 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.480976+0000", 2024-04-20T17:27:22.399 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6910000000000002e-06 2024-04-20T17:27:22.399 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.399 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.400 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:22.400 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.481054+0000", 2024-04-20T17:27:22.400 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7855999999999995e-05 2024-04-20T17:27:22.400 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.400 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.400 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-04-20T17:27:22.400 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.481058+0000", 2024-04-20T17:27:22.400 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7249999999999999e-06 2024-04-20T17:27:22.400 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.400 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.400 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:22.400 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.585066+0000", 2024-04-20T17:27:22.400 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.104008156 2024-04-20T17:27:22.400 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.400 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.400 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:27:22.401 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.585220+0000", 2024-04-20T17:27:22.401 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015381600000000001 2024-04-20T17:27:22.401 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.401 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.401 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:27:22.401 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.585317+0000", 2024-04-20T17:27:22.401 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7325999999999996e-05 2024-04-20T17:27:22.401 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.401 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.401 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:27:22.401 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.602837+0000", 2024-04-20T17:27:22.401 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.017520518999999998 2024-04-20T17:27:22.401 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.401 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.401 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:22.402 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.603242+0000", 2024-04-20T17:27:22.402 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00040505700000000001 2024-04-20T17:27:22.402 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.402 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.402 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:27:22.402 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.603268+0000", 2024-04-20T17:27:22.402 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5817e-05 2024-04-20T17:27:22.402 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.402 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.402 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:27:22.402 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:24:11.603289+0000", 2024-04-20T17:27:22.402 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1001000000000001e-05 2024-04-20T17:27:22.402 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.402 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:27:22.403 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.403 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.403 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.403 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162338 4.3 4:c74ec0af:::benchmark_data_smithi171_122772_object162337:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:27:22.403 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:11.924493+0000", 2024-04-20T17:27:22.403 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 70.426194831999993, 2024-04-20T17:27:22.403 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.209723929, 2024-04-20T17:27:22.403 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:27:22.403 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:27:22.403 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:22.403 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:27:22.403 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:27:22.403 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:22.403 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162338 2024-04-20T17:27:22.403 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.404 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:27:22.404 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.404 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:27:22.404 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.924493+0000", 2024-04-20T17:27:22.404 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.404 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.404 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.404 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:27:22.404 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.924493+0000", 2024-04-20T17:27:22.404 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.404 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.404 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.404 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:27:22.404 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.924495+0000", 2024-04-20T17:27:22.404 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.575e-06 2024-04-20T17:27:22.404 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.405 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.405 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:27:22.405 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.924515+0000", 2024-04-20T17:27:22.405 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0489e-05 2024-04-20T17:27:22.405 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.405 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.405 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:27:22.405 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.924526+0000", 2024-04-20T17:27:22.405 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1309e-05 2024-04-20T17:27:22.405 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.405 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.405 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:27:22.405 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.924534+0000", 2024-04-20T17:27:22.405 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.25e-06 2024-04-20T17:27:22.405 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.406 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.406 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:22.406 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.926564+0000", 2024-04-20T17:27:22.406 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0020306550000000001 2024-04-20T17:27:22.406 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.406 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.406 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:27:22.406 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.926712+0000", 2024-04-20T17:27:22.406 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014766700000000001 2024-04-20T17:27:22.406 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.406 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.407 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:27:22.407 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.926862+0000", 2024-04-20T17:27:22.407 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014963500000000001 2024-04-20T17:27:22.407 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.407 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.407 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:27:22.407 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.932381+0000", 2024-04-20T17:27:22.407 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0055196949999999998 2024-04-20T17:27:22.407 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.407 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.407 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:22.407 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.134102+0000", 2024-04-20T17:27:22.407 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20172087 2024-04-20T17:27:22.407 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.407 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.408 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:27:22.408 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.134166+0000", 2024-04-20T17:27:22.408 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4243000000000001e-05 2024-04-20T17:27:22.408 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.408 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.408 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:27:22.408 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.134217+0000", 2024-04-20T17:27:22.408 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0541000000000003e-05 2024-04-20T17:27:22.408 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.408 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:27:22.408 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.408 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.408 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.409 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162350 4.8 4:193a143c:::benchmark_data_smithi171_122772_object162349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:27:22.409 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:11.931709+0000", 2024-04-20T17:27:22.409 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 70.418978769000006, 2024-04-20T17:27:22.409 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.22858086999999999, 2024-04-20T17:27:22.409 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:27:22.409 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:27:22.409 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:22.409 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:27:22.409 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:27:22.409 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:22.409 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162350 2024-04-20T17:27:22.409 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.409 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:27:22.409 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.409 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:27:22.410 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.931709+0000", 2024-04-20T17:27:22.410 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.410 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.410 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.410 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:27:22.410 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.931709+0000", 2024-04-20T17:27:22.410 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.410 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.410 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.410 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:27:22.410 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.931712+0000", 2024-04-20T17:27:22.410 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.92e-06 2024-04-20T17:27:22.410 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.410 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.411 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:27:22.411 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.931737+0000", 2024-04-20T17:27:22.411 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5106999999999999e-05 2024-04-20T17:27:22.411 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.411 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.411 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:27:22.411 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.931752+0000", 2024-04-20T17:27:22.411 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4891e-05 2024-04-20T17:27:22.411 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.411 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.411 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:27:22.411 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.931761+0000", 2024-04-20T17:27:22.411 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6340000000000007e-06 2024-04-20T17:27:22.412 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.412 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.412 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:22.412 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.931824+0000", 2024-04-20T17:27:22.412 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3019000000000001e-05 2024-04-20T17:27:22.412 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.412 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.412 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:27:22.412 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.931959+0000", 2024-04-20T17:27:22.412 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013509700000000001 2024-04-20T17:27:22.412 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.412 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.412 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:27:22.412 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.932081+0000", 2024-04-20T17:27:22.412 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000121919 2024-04-20T17:27:22.413 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.413 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.413 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:27:22.413 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937096+0000", 2024-04-20T17:27:22.413 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0050154980000000002 2024-04-20T17:27:22.413 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.413 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.413 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:22.413 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160167+0000", 2024-04-20T17:27:22.413 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.223070565 2024-04-20T17:27:22.413 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.413 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.413 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:27:22.413 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160239+0000", 2024-04-20T17:27:22.413 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2729000000000001e-05 2024-04-20T17:27:22.414 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.414 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.414 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:27:22.414 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160290+0000", 2024-04-20T17:27:22.414 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0491000000000002e-05 2024-04-20T17:27:22.414 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.414 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:27:22.414 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.414 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.414 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.414 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162353 4.8 4:1973aaac:::benchmark_data_smithi171_122772_object162352:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:27:22.414 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:11.937307+0000", 2024-04-20T17:27:22.414 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 70.413380630000006, 2024-04-20T17:27:22.414 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.22316846200000001, 2024-04-20T17:27:22.415 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:27:22.415 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:27:22.415 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:22.415 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:27:22.415 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:27:22.415 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:22.415 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162353 2024-04-20T17:27:22.415 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.415 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:27:22.415 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.415 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:27:22.415 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937307+0000", 2024-04-20T17:27:22.415 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.415 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.416 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.416 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:27:22.416 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937307+0000", 2024-04-20T17:27:22.416 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.416 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.416 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.416 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:27:22.416 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937308+0000", 2024-04-20T17:27:22.416 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0210000000000001e-06 2024-04-20T17:27:22.416 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.416 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.416 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:27:22.416 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937330+0000", 2024-04-20T17:27:22.416 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1678000000000001e-05 2024-04-20T17:27:22.416 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.417 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.417 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:27:22.417 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937337+0000", 2024-04-20T17:27:22.417 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4340000000000003e-06 2024-04-20T17:27:22.417 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.417 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.417 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:27:22.417 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937343+0000", 2024-04-20T17:27:22.417 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8599999999999998e-06 2024-04-20T17:27:22.417 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.417 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.417 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:22.417 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937396+0000", 2024-04-20T17:27:22.417 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3124999999999997e-05 2024-04-20T17:27:22.417 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.418 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.418 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:27:22.418 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937544+0000", 2024-04-20T17:27:22.418 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000147398 2024-04-20T17:27:22.418 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.418 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.418 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:27:22.418 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937698+0000", 2024-04-20T17:27:22.418 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015431200000000001 2024-04-20T17:27:22.418 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.418 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.418 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:27:22.418 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.956740+0000", 2024-04-20T17:27:22.418 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019042416999999999 2024-04-20T17:27:22.419 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.419 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.419 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:22.419 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160359+0000", 2024-04-20T17:27:22.419 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20361854900000001 2024-04-20T17:27:22.419 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.419 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.419 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:27:22.419 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160419+0000", 2024-04-20T17:27:22.419 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9735999999999999e-05 2024-04-20T17:27:22.419 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.419 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.419 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:27:22.419 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160476+0000", 2024-04-20T17:27:22.419 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6932000000000003e-05 2024-04-20T17:27:22.419 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.420 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:27:22.420 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.420 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.420 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.420 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162360 4.8 4:12f91459:::benchmark_data_smithi171_122772_object162359:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:27:22.420 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:11.943537+0000", 2024-04-20T17:27:22.420 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 70.407150780999999, 2024-04-20T17:27:22.420 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.21790537500000001, 2024-04-20T17:27:22.420 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:27:22.420 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:27:22.420 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:22.420 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:27:22.420 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:27:22.420 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:22.421 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162360 2024-04-20T17:27:22.421 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.421 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:27:22.421 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.421 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:27:22.421 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.943537+0000", 2024-04-20T17:27:22.421 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.421 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.421 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.421 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:27:22.421 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.943537+0000", 2024-04-20T17:27:22.421 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.421 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.421 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.421 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:27:22.422 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.943539+0000", 2024-04-20T17:27:22.422 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.826e-06 2024-04-20T17:27:22.422 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.422 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.422 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:27:22.422 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.943561+0000", 2024-04-20T17:27:22.422 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2354999999999999e-05 2024-04-20T17:27:22.422 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.422 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.422 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:27:22.422 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.943573+0000", 2024-04-20T17:27:22.422 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1841999999999999e-05 2024-04-20T17:27:22.422 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.422 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.422 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:27:22.423 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.943581+0000", 2024-04-20T17:27:22.423 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1769999999999998e-06 2024-04-20T17:27:22.423 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.423 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.423 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:22.423 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.943637+0000", 2024-04-20T17:27:22.423 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5429000000000001e-05 2024-04-20T17:27:22.423 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.423 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.423 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:27:22.423 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.943771+0000", 2024-04-20T17:27:22.423 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000134027 2024-04-20T17:27:22.423 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.423 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.423 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:27:22.424 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.943930+0000", 2024-04-20T17:27:22.424 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015902900000000001 2024-04-20T17:27:22.424 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.424 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.424 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:27:22.424 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.956748+0000", 2024-04-20T17:27:22.424 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.012818016 2024-04-20T17:27:22.424 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.424 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.424 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:22.424 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.161353+0000", 2024-04-20T17:27:22.424 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20460482199999999 2024-04-20T17:27:22.424 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.424 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.424 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:27:22.424 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.161404+0000", 2024-04-20T17:27:22.425 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1439e-05 2024-04-20T17:27:22.425 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.425 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.425 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:27:22.425 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.161442+0000", 2024-04-20T17:27:22.425 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8413e-05 2024-04-20T17:27:22.425 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.425 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:27:22.425 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.425 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.425 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.425 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162361 4.8 4:19c34efa:::benchmark_data_smithi171_122772_object162360:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:27:22.425 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:11.946327+0000", 2024-04-20T17:27:22.425 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 70.404360807000003, 2024-04-20T17:27:22.425 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.21524243400000001, 2024-04-20T17:27:22.426 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:27:22.426 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:27:22.426 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:22.426 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:27:22.426 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:27:22.426 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:22.426 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162361 2024-04-20T17:27:22.426 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.426 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:27:22.426 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.426 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:27:22.426 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.946327+0000", 2024-04-20T17:27:22.426 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.426 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.426 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.427 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:27:22.427 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.946327+0000", 2024-04-20T17:27:22.427 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.427 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.427 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.427 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:27:22.427 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.946329+0000", 2024-04-20T17:27:22.427 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8470000000000001e-06 2024-04-20T17:27:22.427 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.427 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.427 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:27:22.427 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.946353+0000", 2024-04-20T17:27:22.427 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4491999999999999e-05 2024-04-20T17:27:22.427 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.427 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.428 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:27:22.428 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.946365+0000", 2024-04-20T17:27:22.428 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1688000000000001e-05 2024-04-20T17:27:22.428 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.428 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.428 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:27:22.428 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.946373+0000", 2024-04-20T17:27:22.428 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9049999999999997e-06 2024-04-20T17:27:22.428 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.428 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.428 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:22.428 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.947724+0000", 2024-04-20T17:27:22.428 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0013511409999999999 2024-04-20T17:27:22.428 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.428 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.428 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:27:22.429 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.947841+0000", 2024-04-20T17:27:22.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000116839 2024-04-20T17:27:22.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.429 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.429 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:27:22.429 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.947979+0000", 2024-04-20T17:27:22.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013786500000000001 2024-04-20T17:27:22.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.429 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.429 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:27:22.429 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.956751+0000", 2024-04-20T17:27:22.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0087720309999999996 2024-04-20T17:27:22.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.429 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.429 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:22.430 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.161495+0000", 2024-04-20T17:27:22.430 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20474413799999999 2024-04-20T17:27:22.430 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.430 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.430 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:27:22.430 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.161533+0000", 2024-04-20T17:27:22.430 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.82e-05 2024-04-20T17:27:22.430 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.430 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.430 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:27:22.430 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.161569+0000", 2024-04-20T17:27:22.430 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6288000000000002e-05 2024-04-20T17:27:22.430 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.430 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:27:22.430 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.431 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.431 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.431 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162362 4.8 4:15c10f34:::benchmark_data_smithi171_122772_object162361:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:27:22.431 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:11.947126+0000", 2024-04-20T17:27:22.431 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 70.403561878999994, 2024-04-20T17:27:22.431 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.224067406, 2024-04-20T17:27:22.431 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:27:22.431 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:27:22.431 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:22.431 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:27:22.431 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:27:22.431 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:22.431 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162362 2024-04-20T17:27:22.431 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.431 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:27:22.431 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.432 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:27:22.432 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.947126+0000", 2024-04-20T17:27:22.432 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.432 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.432 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.432 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:27:22.432 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.947126+0000", 2024-04-20T17:27:22.432 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.432 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.432 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.432 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:27:22.432 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.947127+0000", 2024-04-20T17:27:22.432 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5009999999999999e-06 2024-04-20T17:27:22.432 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.432 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.432 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:27:22.433 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.947149+0000", 2024-04-20T17:27:22.433 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1708e-05 2024-04-20T17:27:22.433 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.433 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.433 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:27:22.433 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.947159+0000", 2024-04-20T17:27:22.433 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0244999999999999e-05 2024-04-20T17:27:22.433 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.433 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.433 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:27:22.433 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.947166+0000", 2024-04-20T17:27:22.433 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4110000000000001e-06 2024-04-20T17:27:22.433 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.433 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.434 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:22.434 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.952667+0000", 2024-04-20T17:27:22.434 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0055010679999999996 2024-04-20T17:27:22.434 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.434 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.434 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:27:22.434 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.952814+0000", 2024-04-20T17:27:22.434 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014753499999999999 2024-04-20T17:27:22.434 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.434 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.434 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:27:22.434 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.952948+0000", 2024-04-20T17:27:22.434 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013382499999999999 2024-04-20T17:27:22.434 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.434 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.435 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:27:22.435 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.956887+0000", 2024-04-20T17:27:22.435 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0039385430000000001 2024-04-20T17:27:22.435 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.435 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.435 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:22.435 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.171085+0000", 2024-04-20T17:27:22.435 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.214198641 2024-04-20T17:27:22.435 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.435 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.435 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:27:22.435 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.171143+0000", 2024-04-20T17:27:22.435 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7720000000000003e-05 2024-04-20T17:27:22.435 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.435 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.436 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:27:22.436 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.171193+0000", 2024-04-20T17:27:22.436 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0209e-05 2024-04-20T17:27:22.436 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.436 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:27:22.437 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.437 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.437 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.437 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162374 4.3 4:c9021f98:::benchmark_data_smithi171_122772_object162373:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:27:22.437 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:11.952990+0000", 2024-04-20T17:27:22.437 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 70.397697434999998, 2024-04-20T17:27:22.437 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19610433199999999, 2024-04-20T17:27:22.437 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:27:22.437 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:27:22.437 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:22.437 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:27:22.437 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:27:22.437 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:22.437 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162374 2024-04-20T17:27:22.437 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.438 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:27:22.438 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.438 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:27:22.438 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.952990+0000", 2024-04-20T17:27:22.438 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.438 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.438 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.438 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:27:22.438 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.952990+0000", 2024-04-20T17:27:22.438 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.439 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.439 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:27:22.439 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.952992+0000", 2024-04-20T17:27:22.439 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7060000000000001e-06 2024-04-20T17:27:22.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.439 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.439 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:27:22.439 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.953016+0000", 2024-04-20T17:27:22.439 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4002000000000002e-05 2024-04-20T17:27:22.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.439 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.439 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:27:22.439 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.953026+0000", 2024-04-20T17:27:22.439 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0259e-05 2024-04-20T17:27:22.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.440 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.440 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:27:22.440 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.953034+0000", 2024-04-20T17:27:22.440 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7219999999999999e-06 2024-04-20T17:27:22.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.440 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.440 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:22.440 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.953097+0000", 2024-04-20T17:27:22.440 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2479999999999998e-05 2024-04-20T17:27:22.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.440 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.440 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:27:22.440 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.953175+0000", 2024-04-20T17:27:22.440 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8559000000000005e-05 2024-04-20T17:27:22.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.441 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:27:22.441 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.953274+0000", 2024-04-20T17:27:22.441 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9059000000000001e-05 2024-04-20T17:27:22.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.441 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:27:22.441 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.957017+0000", 2024-04-20T17:27:22.441 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.003742483 2024-04-20T17:27:22.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.441 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:22.441 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.149017+0000", 2024-04-20T17:27:22.441 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19200002899999999 2024-04-20T17:27:22.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.442 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.442 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:27:22.442 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.149053+0000", 2024-04-20T17:27:22.442 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5856999999999999e-05 2024-04-20T17:27:22.442 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.442 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.442 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:27:22.442 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.149095+0000", 2024-04-20T17:27:22.442 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2175999999999997e-05 2024-04-20T17:27:22.442 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.442 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:27:22.442 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.442 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.442 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.442 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162397 4.3 4:c9a84e1e:::benchmark_data_smithi171_122772_object162396:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:27:22.443 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:11.983263+0000", 2024-04-20T17:27:22.443 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 70.367424811999996, 2024-04-20T17:27:22.443 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.177048224, 2024-04-20T17:27:22.443 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:27:22.443 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:27:22.443 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:22.443 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:27:22.443 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:27:22.443 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:22.443 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162397 2024-04-20T17:27:22.443 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.443 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:27:22.443 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.443 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:27:22.443 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.983263+0000", 2024-04-20T17:27:22.444 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.444 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.444 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.444 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:27:22.444 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.983263+0000", 2024-04-20T17:27:22.444 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.444 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.444 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.444 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:27:22.444 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.983266+0000", 2024-04-20T17:27:22.444 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.644e-06 2024-04-20T17:27:22.444 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.444 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.444 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:27:22.444 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.983288+0000", 2024-04-20T17:27:22.444 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2266999999999999e-05 2024-04-20T17:27:22.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.445 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:27:22.445 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.983302+0000", 2024-04-20T17:27:22.445 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4412e-05 2024-04-20T17:27:22.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.445 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:27:22.445 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.983310+0000", 2024-04-20T17:27:22.445 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7780000000000007e-06 2024-04-20T17:27:22.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.445 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:22.445 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.983374+0000", 2024-04-20T17:27:22.445 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3399999999999996e-05 2024-04-20T17:27:22.446 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.446 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.446 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:27:22.446 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.983521+0000", 2024-04-20T17:27:22.446 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014708399999999999 2024-04-20T17:27:22.446 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.446 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.446 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:27:22.446 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.983686+0000", 2024-04-20T17:27:22.446 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00016503000000000001 2024-04-20T17:27:22.446 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.446 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.446 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:27:22.446 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.988826+0000", 2024-04-20T17:27:22.446 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0051401069999999997 2024-04-20T17:27:22.447 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.447 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.447 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:22.447 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160218+0000", 2024-04-20T17:27:22.447 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.17139236599999999 2024-04-20T17:27:22.447 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.447 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.447 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:27:22.447 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160274+0000", 2024-04-20T17:27:22.447 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5739999999999998e-05 2024-04-20T17:27:22.447 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.447 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.447 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:27:22.447 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160311+0000", 2024-04-20T17:27:22.447 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7395999999999999e-05 2024-04-20T17:27:22.448 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.448 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:27:22.448 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.448 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.448 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.448 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162414 4.3 4:cecc3530:::benchmark_data_smithi171_122772_object162413:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:27:22.448 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:12.011130+0000", 2024-04-20T17:27:22.448 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 70.339558279000002, 2024-04-20T17:27:22.448 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.149342215, 2024-04-20T17:27:22.448 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:27:22.448 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:27:22.448 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:22.448 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:27:22.448 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:27:22.448 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:22.449 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162414 2024-04-20T17:27:22.449 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.449 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:27:22.449 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.449 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:27:22.449 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.011130+0000", 2024-04-20T17:27:22.449 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.449 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.449 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.449 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:27:22.449 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.011130+0000", 2024-04-20T17:27:22.449 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.449 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.449 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.449 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:27:22.450 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.011132+0000", 2024-04-20T17:27:22.450 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.402e-06 2024-04-20T17:27:22.450 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.450 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.450 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:27:22.450 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.011160+0000", 2024-04-20T17:27:22.450 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8013000000000001e-05 2024-04-20T17:27:22.450 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.450 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.450 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:27:22.450 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.011172+0000", 2024-04-20T17:27:22.450 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.184e-05 2024-04-20T17:27:22.450 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.450 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.450 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:27:22.451 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.011180+0000", 2024-04-20T17:27:22.451 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.3259999999999996e-06 2024-04-20T17:27:22.451 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.451 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.451 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:22.451 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.011246+0000", 2024-04-20T17:27:22.451 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6074000000000005e-05 2024-04-20T17:27:22.451 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.451 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.451 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:27:22.451 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.011384+0000", 2024-04-20T17:27:22.451 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013756200000000001 2024-04-20T17:27:22.451 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.451 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.451 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:27:22.452 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.011546+0000", 2024-04-20T17:27:22.452 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00016172800000000001 2024-04-20T17:27:22.452 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.452 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.452 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:27:22.452 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.016633+0000", 2024-04-20T17:27:22.452 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0050873589999999996 2024-04-20T17:27:22.452 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.452 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.452 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:22.452 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160374+0000", 2024-04-20T17:27:22.452 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.143740962 2024-04-20T17:27:22.452 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.452 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.452 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:27:22.452 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160428+0000", 2024-04-20T17:27:22.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4406000000000001e-05 2024-04-20T17:27:22.453 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.453 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.453 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:27:22.453 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160472+0000", 2024-04-20T17:27:22.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3543e-05 2024-04-20T17:27:22.453 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.453 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:27:22.453 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.453 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.453 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.453 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162421 4.8 4:1476fb97:::benchmark_data_smithi171_122772_object162420:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:27:22.453 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:12.022530+0000", 2024-04-20T17:27:22.453 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 70.328157548999997, 2024-04-20T17:27:22.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15207664000000001, 2024-04-20T17:27:22.454 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:27:22.454 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:27:22.454 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:22.454 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:27:22.454 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:27:22.454 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:22.454 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162421 2024-04-20T17:27:22.454 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.454 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:27:22.454 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.454 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:27:22.454 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.022530+0000", 2024-04-20T17:27:22.454 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.454 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.454 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.455 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:27:22.455 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.022530+0000", 2024-04-20T17:27:22.455 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.455 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.455 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.455 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:27:22.455 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.022533+0000", 2024-04-20T17:27:22.455 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5909999999999998e-06 2024-04-20T17:27:22.455 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.455 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.455 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:27:22.455 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.022554+0000", 2024-04-20T17:27:22.455 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1398000000000001e-05 2024-04-20T17:27:22.455 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.455 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.456 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:27:22.456 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.022567+0000", 2024-04-20T17:27:22.456 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2343e-05 2024-04-20T17:27:22.456 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.456 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.456 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:27:22.456 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.022574+0000", 2024-04-20T17:27:22.456 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0609999999999998e-06 2024-04-20T17:27:22.456 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.456 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.456 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:22.456 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.022626+0000", 2024-04-20T17:27:22.456 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2064000000000002e-05 2024-04-20T17:27:22.456 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.456 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.456 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:27:22.457 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.022735+0000", 2024-04-20T17:27:22.457 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000108868 2024-04-20T17:27:22.457 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.457 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:27:22.457 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.022833+0000", 2024-04-20T17:27:22.457 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.8540000000000002e-05 2024-04-20T17:27:22.457 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.457 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:27:22.457 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.026748+0000", 2024-04-20T17:27:22.457 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0039153039999999997 2024-04-20T17:27:22.457 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.457 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:22.457 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.174477+0000", 2024-04-20T17:27:22.458 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.14772859399999999 2024-04-20T17:27:22.458 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.458 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.458 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:27:22.458 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.174550+0000", 2024-04-20T17:27:22.458 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3263999999999995e-05 2024-04-20T17:27:22.458 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.458 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.458 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:27:22.458 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.174607+0000", 2024-04-20T17:27:22.458 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6613000000000002e-05 2024-04-20T17:27:22.458 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.458 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:27:22.458 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.458 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.459 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.459 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162423 4.8 4:13f12511:::benchmark_data_smithi171_122772_object162422:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:27:22.459 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:12.026739+0000", 2024-04-20T17:27:22.459 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 70.323948939000005, 2024-04-20T17:27:22.459 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15216422499999999, 2024-04-20T17:27:22.459 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:27:22.459 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:27:22.459 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:22.459 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:27:22.459 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:27:22.459 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:22.459 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162423 2024-04-20T17:27:22.459 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.459 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:27:22.460 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.460 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:27:22.460 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.026739+0000", 2024-04-20T17:27:22.460 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.460 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.460 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.460 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:27:22.460 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.026739+0000", 2024-04-20T17:27:22.460 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.460 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.460 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.460 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:27:22.460 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.026741+0000", 2024-04-20T17:27:22.460 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.764e-06 2024-04-20T17:27:22.460 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.461 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.461 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:27:22.461 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.026765+0000", 2024-04-20T17:27:22.461 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4094999999999999e-05 2024-04-20T17:27:22.461 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.461 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.461 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:27:22.461 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.026774+0000", 2024-04-20T17:27:22.461 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.4700000000000008e-06 2024-04-20T17:27:22.461 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.461 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.461 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:27:22.461 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.026780+0000", 2024-04-20T17:27:22.461 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.976e-06 2024-04-20T17:27:22.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.462 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:22.462 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.026818+0000", 2024-04-20T17:27:22.462 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7407999999999998e-05 2024-04-20T17:27:22.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.462 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:27:22.462 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.026910+0000", 2024-04-20T17:27:22.462 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2390000000000001e-05 2024-04-20T17:27:22.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.462 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:27:22.462 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.027002+0000", 2024-04-20T17:27:22.463 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1989000000000003e-05 2024-04-20T17:27:22.463 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.463 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.463 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:27:22.463 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.033898+0000", 2024-04-20T17:27:22.463 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0068955700000000002 2024-04-20T17:27:22.463 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.463 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.463 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:22.463 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.178824+0000", 2024-04-20T17:27:22.463 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.144926312 2024-04-20T17:27:22.463 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.463 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.463 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:27:22.463 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.178858+0000", 2024-04-20T17:27:22.464 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4459000000000003e-05 2024-04-20T17:27:22.464 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.464 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.464 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:27:22.464 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.178903+0000", 2024-04-20T17:27:22.464 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4792000000000001e-05 2024-04-20T17:27:22.464 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.464 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:27:22.464 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.464 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.464 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.464 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162424 4.8 4:15c23b0b:::benchmark_data_smithi171_122772_object162423:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:27:22.464 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:12.029044+0000", 2024-04-20T17:27:22.464 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 70.321644026000001, 2024-04-20T17:27:22.464 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.14999915, 2024-04-20T17:27:22.465 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:27:22.465 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:27:22.465 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:22.465 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:27:22.465 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:27:22.465 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:22.465 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162424 2024-04-20T17:27:22.465 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.465 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:27:22.465 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.465 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:27:22.465 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.029044+0000", 2024-04-20T17:27:22.465 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.465 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.465 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.466 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:27:22.466 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.029044+0000", 2024-04-20T17:27:22.466 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:27:22.466 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.466 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.466 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:27:22.466 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.029045+0000", 2024-04-20T17:27:22.466 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5960000000000001e-06 2024-04-20T17:27:22.466 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.466 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.466 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:27:22.467 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.029069+0000", 2024-04-20T17:27:22.467 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4023e-05 2024-04-20T17:27:22.467 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.467 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.467 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:27:22.467 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.029082+0000", 2024-04-20T17:27:22.467 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2373e-05 2024-04-20T17:27:22.467 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.467 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.467 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:27:22.467 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.029090+0000", 2024-04-20T17:27:22.467 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5290000000000008e-06 2024-04-20T17:27:22.467 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.468 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.468 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:27:22.468 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.030098+0000", 2024-04-20T17:27:22.468 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0010079150000000001 2024-04-20T17:27:22.468 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.468 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.468 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:27:22.468 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.030236+0000", 2024-04-20T17:27:22.468 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013768999999999999 2024-04-20T17:27:22.468 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.468 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.468 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:27:22.468 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.030381+0000", 2024-04-20T17:27:22.468 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014539300000000001 2024-04-20T17:27:22.469 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.469 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.469 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:27:22.469 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.034013+0000", 2024-04-20T17:27:22.469 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.003631839 2024-04-20T17:27:22.469 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.469 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.469 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:22.469 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.178952+0000", 2024-04-20T17:27:22.469 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.14493904699999999 2024-04-20T17:27:22.469 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.469 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.469 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:27:22.469 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.179004+0000", 2024-04-20T17:27:22.469 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1851999999999998e-05 2024-04-20T17:27:22.470 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:27:22.470 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:27:22.470 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:27:22.470 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.179043+0000", 2024-04-20T17:27:22.470 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8893000000000002e-05 2024-04-20T17:27:22.470 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.470 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:27:22.470 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.470 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:27:22.470 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:27:22.470 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:27:22.470 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:27:22.500 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:22.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:22 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:27:22.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:22 smithi106 ceph-mon[44423]: from='client.26269 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:27:22.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:22 smithi106 ceph-mon[44423]: osdmap e503: 12 total, 12 up, 12 in 2024-04-20T17:27:22.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:22 smithi106 ceph-mon[44423]: osd.6 marked itself down and dead 2024-04-20T17:27:22.563 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:22 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:27:22.563 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:22 smithi005 ceph-mon[39562]: from='client.26269 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:27:22.563 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:22 smithi005 ceph-mon[39562]: osdmap e503: 12 total, 12 up, 12 in 2024-04-20T17:27:22.563 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:22 smithi005 ceph-mon[39562]: osd.6 marked itself down and dead 2024-04-20T17:27:22.598 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:27:22.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:22.601 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:27:22.703 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:27:22.733 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:22.936 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:27:22.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:22.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:27:23.043 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:27:23.058 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-20T17:27:23.072 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:23.090 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:27:23.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:23.093 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:27:23.194 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:27:23.227 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:23.248 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:27:23.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:23.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:27:23.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:23 smithi171 ceph-mon[44189]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T17:27:23.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:23 smithi171 ceph-mon[44189]: osdmap e504: 12 total, 11 up, 12 in 2024-04-20T17:27:23.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:23 smithi171 ceph-mon[44189]: pgmap v1332: 17 pgs: 2 stale+active+clean, 15 active+clean; 5.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s 2024-04-20T17:27:23.351 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:27:23.380 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:23.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:23 smithi005 ceph-mon[39562]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T17:27:23.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:23 smithi005 ceph-mon[39562]: osdmap e504: 12 total, 11 up, 12 in 2024-04-20T17:27:23.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:23 smithi005 ceph-mon[39562]: pgmap v1332: 17 pgs: 2 stale+active+clean, 15 active+clean; 5.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s 2024-04-20T17:27:23.511 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:27:23 smithi106 bash[133438]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6 2024-04-20T17:27:23.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:23 smithi106 ceph-mon[44423]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T17:27:23.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:23 smithi106 ceph-mon[44423]: osdmap e504: 12 total, 11 up, 12 in 2024-04-20T17:27:23.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:23 smithi106 ceph-mon[44423]: pgmap v1332: 17 pgs: 2 stale+active+clean, 15 active+clean; 5.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s 2024-04-20T17:27:23.564 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:27:23.578 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:27:23.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:23.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:27:23.596 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:27:23.615 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:27:23.635 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:27:23.654 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:27:23.674 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:27:23.683 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:27:23.694 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:27:23.713 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:23.714 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:27:23.734 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:27:23.734 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:27:23.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:23.737 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:27:23.755 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:27:23.775 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:27:23.795 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:27:23.795 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:27:23.795 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:27:23.795 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:27:23.795 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:27:23.795 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:27:23.795 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:27:23.795 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-20T17:27:23.795 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:27:23.795 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:27:23.795 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:27:23.795 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:27:23.796 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:27:23.838 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:27:23.867 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:23.995 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:27:24.015 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:27:24.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:24.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:27:24.119 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:27:24.150 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:24.172 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:27:24.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:24.175 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:27:24.276 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:27:24.303 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:24.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:24 smithi005 ceph-mon[39562]: osdmap e505: 12 total, 11 up, 12 in 2024-04-20T17:27:24.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:24 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2936054811' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:27:24.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:24 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:24.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:24 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:24.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:24 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:24.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:24 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:24.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:24 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:24.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:24 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:24.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:24 smithi106 ceph-mon[44423]: osdmap e505: 12 total, 11 up, 12 in 2024-04-20T17:27:24.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:24 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2936054811' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:27:24.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:24 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:24.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:24 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:24.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:24 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:24.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:24 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:24.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:24 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:24.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:24 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:24 smithi171 ceph-mon[44189]: osdmap e505: 12 total, 11 up, 12 in 2024-04-20T17:27:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:24 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2936054811' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:27:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:24 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:24 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:24 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:24 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:24 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:24 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:24.586 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:27:24.587 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:27:24.651 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:27:24.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:24.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:27:24.700 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:27:24.755 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:27:24.799 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:24.820 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:27:24.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:24.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:27:24.924 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:27:24.956 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:24.977 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:27:24.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:24.980 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:27:25.081 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:27:25.110 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:25.184 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:27:25.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:25.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:27:25.288 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:27:25.318 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:25.347 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:27:25.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:25.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:27:25.453 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:27:25.481 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:25 smithi106 ceph-mon[44423]: pgmap v1334: 17 pgs: 1 undersized+degraded+peered, 1 peering, 15 active+clean; 5.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.29k op/s; 5733/186466 objects degraded (3.075%) 2024-04-20T17:27:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:25 smithi106 ceph-mon[44423]: Health check failed: Degraded data redundancy: 5733/186466 objects degraded (3.075%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:27:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:25 smithi171 ceph-mon[44189]: pgmap v1334: 17 pgs: 1 undersized+degraded+peered, 1 peering, 15 active+clean; 5.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.29k op/s; 5733/186466 objects degraded (3.075%) 2024-04-20T17:27:25.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:25 smithi171 ceph-mon[44189]: Health check failed: Degraded data redundancy: 5733/186466 objects degraded (3.075%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:27:25.612 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:25.612 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:25.612 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:25.612 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:25.612 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:25.612 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:25 smithi005 ceph-mon[39562]: pgmap v1334: 17 pgs: 1 undersized+degraded+peered, 1 peering, 15 active+clean; 5.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.29k op/s; 5733/186466 objects degraded (3.075%) 2024-04-20T17:27:25.612 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:25 smithi005 ceph-mon[39562]: Health check failed: Degraded data redundancy: 5733/186466 objects degraded (3.075%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:27:26.186 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:27:26.186 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:27:26.490 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.6.service' 2024-04-20T17:27:26.572 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T17:27:26.572 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-20T17:27:26.572 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd down 6 2024-04-20T17:27:26.588 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:27:26.730 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:27:26.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:26.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:27:26.835 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:27:26.863 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:26.866 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:27:26.961 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:27:26.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:26.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:27:27.069 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:27:27.097 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:27.117 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:27:27.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:27.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:27:27.145 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:27:27.201 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:27:27.221 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:27:27.249 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:27.268 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:27:27.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:27.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:27:27.286 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-20T17:27:27.373 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:27:27.398 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:27.417 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:27:27.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:27.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:27:27.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:27 smithi005 ceph-mon[39562]: pgmap v1335: 17 pgs: 1 undersized+degraded+peered, 1 peering, 15 active+clean; 5.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 5733/184692 objects degraded (3.104%) 2024-04-20T17:27:27.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:27 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3900209325' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:27:27.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:27 smithi005 ceph-mon[39562]: from='client.26366 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:27:27.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:27 smithi106 ceph-mon[44423]: pgmap v1335: 17 pgs: 1 undersized+degraded+peered, 1 peering, 15 active+clean; 5.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 5733/184692 objects degraded (3.104%) 2024-04-20T17:27:27.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:27 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3900209325' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:27:27.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:27 smithi106 ceph-mon[44423]: from='client.26366 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:27:27.521 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:27:27.546 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:27 smithi171 ceph-mon[44189]: pgmap v1335: 17 pgs: 1 undersized+degraded+peered, 1 peering, 15 active+clean; 5.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 5733/184692 objects degraded (3.104%) 2024-04-20T17:27:27.546 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:27 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3900209325' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:27:27.547 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:27 smithi171 ceph-mon[44189]: from='client.26366 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:27:27.548 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:27.569 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:27:27.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:27.572 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:27:27.598 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:27:27.673 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:27:27.700 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:27.719 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:27:27.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:27.722 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:27:27.823 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:27:27.853 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:27.877 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:27:27.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:27.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:27:27.981 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:27:28.007 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:28.025 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:27:28.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:28.028 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:27:28.129 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:27:28.156 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:28.174 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:27:28.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:28.177 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:27:28.278 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:27:28.302 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:28 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:27:28.302 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:28 smithi171 ceph-mon[44189]: from='client.26366 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:27:28.302 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:28 smithi171 ceph-mon[44189]: osdmap e506: 12 total, 11 up, 12 in 2024-04-20T17:27:28.302 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:28 smithi171 ceph-mon[44189]: 4.1 scrub starts 2024-04-20T17:27:28.303 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:28 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:27:28.303 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:28 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:27:28.304 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:28.323 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:27:28.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:28.325 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:27:28.426 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:27:28.453 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:28.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:28 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:27:28.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:28 smithi005 ceph-mon[39562]: from='client.26366 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:27:28.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:28 smithi005 ceph-mon[39562]: osdmap e506: 12 total, 11 up, 12 in 2024-04-20T17:27:28.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:28 smithi005 ceph-mon[39562]: 4.1 scrub starts 2024-04-20T17:27:28.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:28 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:27:28.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:28 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:27:28.471 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:27:28.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:28.473 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:27:28.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:28 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:27:28.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:28 smithi106 ceph-mon[44423]: from='client.26366 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:27:28.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:28 smithi106 ceph-mon[44423]: osdmap e506: 12 total, 11 up, 12 in 2024-04-20T17:27:28.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:28 smithi106 ceph-mon[44423]: 4.1 scrub starts 2024-04-20T17:27:28.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:28 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:27:28.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:28 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:27:28.574 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:27:28.600 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:28.619 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:27:28.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:28.621 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:27:28.722 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:27:28.751 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:29.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:29 smithi005 ceph-mon[39562]: 4.1 scrub ok 2024-04-20T17:27:29.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:29 smithi005 ceph-mon[39562]: osdmap e507: 12 total, 11 up, 12 in 2024-04-20T17:27:29.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:29 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:27:29.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:29 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:27:29.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:29 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:27:29.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:29 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:27:29.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:29 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:27:29.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:29 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:27:29.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:29 smithi005 ceph-mon[39562]: pgmap v1338: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 5.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 536 op/s; 11546/184372 objects degraded (6.262%) 2024-04-20T17:27:29.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:29 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:27:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:29 smithi106 ceph-mon[44423]: 4.1 scrub ok 2024-04-20T17:27:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:29 smithi106 ceph-mon[44423]: osdmap e507: 12 total, 11 up, 12 in 2024-04-20T17:27:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:29 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:27:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:29 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:27:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:29 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:27:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:29 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:27:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:29 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:27:29.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:29 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:27:29.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:29 smithi106 ceph-mon[44423]: pgmap v1338: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 5.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 536 op/s; 11546/184372 objects degraded (6.262%) 2024-04-20T17:27:29.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:29 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:27:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:29 smithi171 ceph-mon[44189]: 4.1 scrub ok 2024-04-20T17:27:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:29 smithi171 ceph-mon[44189]: osdmap e507: 12 total, 11 up, 12 in 2024-04-20T17:27:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:29 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:27:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:29 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:27:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:29 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:27:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:29 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:27:29.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:29 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:27:29.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:29 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:27:29.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:29 smithi171 ceph-mon[44189]: pgmap v1338: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 5.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 536 op/s; 11546/184372 objects degraded (6.262%) 2024-04-20T17:27:29.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:29 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:27:30.280 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:30 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:27:30.280 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:30 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:27:30.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:30 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:27:30.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:30 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:27:30.576 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:27:30.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:30 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:27:30.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:30 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:27:30.595 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:27:30.614 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:27:30.636 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:27:30.656 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:27:30.677 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:27:30.697 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:27:30.717 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:27:30.724 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6 is already down. 2024-04-20T17:27:30.738 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:27:30.759 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:27:30.787 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:27:30.788 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:27:30.788 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:27:30.788 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:27:30.788 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:27:30.788 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:27:30.788 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:27:30.788 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-20T17:27:30.788 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:27:30.788 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:27:30.788 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:27:30.788 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:27:30.788 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:27:30.902 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:27:30.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:30.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:27:31.007 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:27:31.039 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:31.060 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:27:31.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:31.063 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:27:31.164 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:27:31.197 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:30 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mon-a[39539]: 2024-04-20T17:27:30.720+0000 7f73b92a1700 -1 mon.a@0(leader).osd e508 definitely_dead 0 2024-04-20T17:27:31.200 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:31.275 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:27:31.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:31.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:27:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:31 smithi171 ceph-mon[44189]: pgmap v1339: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 5.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 408 op/s; 11546/184372 objects degraded (6.262%) 2024-04-20T17:27:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:31 smithi171 ceph-mon[44189]: osdmap e508: 12 total, 11 up, 12 in 2024-04-20T17:27:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:31 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1181949372' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:27:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:31.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:31.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:31.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:31.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:31 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/418012326' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["6"]}]: dispatch 2024-04-20T17:27:31.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:31.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:31 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 11546/184372 objects degraded (6.262%), 2 pgs degraded (PG_DEGRADED) 2024-04-20T17:27:31.383 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:27:31.413 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:31.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:31 smithi005 ceph-mon[39562]: pgmap v1339: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 5.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 408 op/s; 11546/184372 objects degraded (6.262%) 2024-04-20T17:27:31.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:31 smithi005 ceph-mon[39562]: osdmap e508: 12 total, 11 up, 12 in 2024-04-20T17:27:31.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:31 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1181949372' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:27:31.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:31.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:31.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:31.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:31.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:31.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:31 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/418012326' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["6"]}]: dispatch 2024-04-20T17:27:31.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:31.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:31 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 11546/184372 objects degraded (6.262%), 2 pgs degraded (PG_DEGRADED) 2024-04-20T17:27:31.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:31 smithi106 ceph-mon[44423]: pgmap v1339: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 5.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 408 op/s; 11546/184372 objects degraded (6.262%) 2024-04-20T17:27:31.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:31 smithi106 ceph-mon[44423]: osdmap e508: 12 total, 11 up, 12 in 2024-04-20T17:27:31.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:31 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1181949372' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:27:31.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:31.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:31.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:31.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:31.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:31.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:31 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/418012326' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["6"]}]: dispatch 2024-04-20T17:27:31.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:31.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:31 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 11546/184372 objects degraded (6.262%), 2 pgs degraded (PG_DEGRADED) 2024-04-20T17:27:31.527 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:27:31.715 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-04-20T17:27:31.715 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd out 6 2024-04-20T17:27:32.085 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:27:32.085 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:27:32.085 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-04-20T17:27:32.085 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-04-20T17:27:32.085 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:27:32.137 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:27:32.138 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:27:32.162 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:27:32.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:32.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:27:32.171 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:27:32.266 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:27:32.296 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:32.334 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:27:32.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:32.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:27:32.427 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:32 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:32.427 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:32 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:32.427 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:32 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:32.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:32 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:32.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:32 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:32.438 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:27:32.469 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:32 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:32.469 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:32 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:32.469 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:32 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:32.469 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:32 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:32.469 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:32 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:32.470 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:32.489 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:27:32.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:32.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:27:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:32 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:32 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:32 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:32 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:32 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:32.594 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:27:32.621 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:32.648 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:27:32.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:32.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:27:32.753 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:27:32.785 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:32.794 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-20T17:27:32.805 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:27:32.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:32.808 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:27:32.909 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:27:32.936 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:32.942 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:27:32.962 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:27:32.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:32.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:27:33.065 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:27:33.066 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:27:33.093 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:33.341 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:27:33.343 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:33 smithi106 ceph-mon[44423]: pgmap v1341: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 5.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 26 op/s; 11546/184372 objects degraded (6.262%) 2024-04-20T17:27:33.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:33.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:27:33.354 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:33 smithi005 ceph-mon[39562]: pgmap v1341: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 5.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 26 op/s; 11546/184372 objects degraded (6.262%) 2024-04-20T17:27:33.445 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:27:33.474 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:33.526 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:27:33.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:33.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:27:33.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:33 smithi171 ceph-mon[44189]: pgmap v1341: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 5.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 26 op/s; 11546/184372 objects degraded (6.262%) 2024-04-20T17:27:33.632 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:27:33.661 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:33.913 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:27:33.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:33.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:27:34.017 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:27:34.047 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:34.139 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:27:34.296 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:27:34.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:34.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:27:34.401 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:27:34.434 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:34.455 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:27:34.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:34.458 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:27:34.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:34 smithi005 ceph-mon[39562]: 4.c scrub starts 2024-04-20T17:27:34.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:34 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:27:34.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:34 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:27:34.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:34 smithi005 ceph-mon[39562]: 4.c scrub ok 2024-04-20T17:27:34.533 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:34 smithi106 ceph-mon[44423]: 4.c scrub starts 2024-04-20T17:27:34.534 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:34 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:27:34.534 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:34 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:27:34.534 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:34 smithi106 ceph-mon[44423]: 4.c scrub ok 2024-04-20T17:27:34.559 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:27:34.581 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:27:34.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:34 smithi171 ceph-mon[44189]: 4.c scrub starts 2024-04-20T17:27:34.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:34 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:27:34.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:34 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:27:34.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:34 smithi171 ceph-mon[44189]: 4.c scrub ok 2024-04-20T17:27:34.591 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:34.611 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:27:34.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:34.614 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:27:34.715 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:27:34.743 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:34.865 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:27:34.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:34.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:27:34.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:34 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mon-a[39539]: 2024-04-20T17:27:34.818+0000 7f73b92a1700 -1 mon.a@0(leader).osd e509 definitely_dead 0 2024-04-20T17:27:34.969 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:27:34.998 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:35.020 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:27:35.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:35.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:27:35.123 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:27:35.142 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:27:35.142 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-04-20T17:27:35.142 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-04-20T17:27:35.142 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:27:35.142 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.142 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245460 3.d 3:bba062b0:::benchmark_data_smithi171_110765_object102245:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:27:35.143 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103564+0000", 2024-04-20T17:27:35.143 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 242.03611702200001, 2024-04-20T17:27:35.143 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.086211963000000003, 2024-04-20T17:27:35.143 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:27:35.143 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:27:35.143 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:35.143 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:27:35.143 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:27:35.144 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:35.144 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245460 2024-04-20T17:27:35.144 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.144 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:27:35.144 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.144 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:27:35.144 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103564+0000", 2024-04-20T17:27:35.144 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.144 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.144 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.145 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:27:35.145 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103564+0000", 2024-04-20T17:27:35.145 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.145 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.145 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.145 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:27:35.145 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103565+0000", 2024-04-20T17:27:35.145 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2999999999999999e-07 2024-04-20T17:27:35.145 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.145 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.145 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:27:35.145 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103568+0000", 2024-04-20T17:27:35.145 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9699999999999999e-06 2024-04-20T17:27:35.145 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.146 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.146 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:27:35.146 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103573+0000", 2024-04-20T17:27:35.146 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4220000000000002e-06 2024-04-20T17:27:35.146 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.146 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.146 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:27:35.146 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103577+0000", 2024-04-20T17:27:35.146 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.34e-06 2024-04-20T17:27:35.146 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.146 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.146 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.146 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103883+0000", 2024-04-20T17:27:35.146 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00030601700000000001 2024-04-20T17:27:35.146 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.147 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.147 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:27:35.147 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103890+0000", 2024-04-20T17:27:35.147 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4189999999999996e-06 2024-04-20T17:27:35.147 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.147 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.147 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.147 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.157260+0000", 2024-04-20T17:27:35.147 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.053369403000000003 2024-04-20T17:27:35.147 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.147 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.147 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:27:35.148 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.157381+0000", 2024-04-20T17:27:35.148 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000120776 2024-04-20T17:27:35.148 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.148 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.148 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:27:35.148 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.157507+0000", 2024-04-20T17:27:35.148 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012656000000000001 2024-04-20T17:27:35.148 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.148 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.148 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:27:35.148 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.165767+0000", 2024-04-20T17:27:35.149 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0082595310000000005 2024-04-20T17:27:35.149 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.149 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.149 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:35.149 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189715+0000", 2024-04-20T17:27:35.149 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.023948542999999999 2024-04-20T17:27:35.149 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.149 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.149 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:27:35.149 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189746+0000", 2024-04-20T17:27:35.149 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0939000000000003e-05 2024-04-20T17:27:35.149 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.149 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.149 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:27:35.150 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189776+0000", 2024-04-20T17:27:35.150 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0113000000000001e-05 2024-04-20T17:27:35.150 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.150 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:27:35.150 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.150 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.150 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.150 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245510 3.d 3:b21de460:::benchmark_data_smithi171_110765_object102295:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:27:35.150 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103589+0000", 2024-04-20T17:27:35.150 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 242.03609239799999, 2024-04-20T17:27:35.150 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.086330094999999996, 2024-04-20T17:27:35.150 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:27:35.150 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:27:35.150 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:35.151 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:27:35.151 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:27:35.151 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:35.151 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245510 2024-04-20T17:27:35.151 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.151 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:27:35.151 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.151 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:27:35.151 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103589+0000", 2024-04-20T17:27:35.151 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.151 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.151 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.151 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:27:35.151 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103589+0000", 2024-04-20T17:27:35.151 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.152 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.152 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.152 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:27:35.152 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103590+0000", 2024-04-20T17:27:35.152 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5400000000000001e-07 2024-04-20T17:27:35.152 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.152 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.152 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:27:35.152 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103592+0000", 2024-04-20T17:27:35.153 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.83e-06 2024-04-20T17:27:35.153 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.153 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.153 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:27:35.153 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103600+0000", 2024-04-20T17:27:35.153 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3139999999999998e-06 2024-04-20T17:27:35.153 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.153 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.153 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:27:35.153 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103605+0000", 2024-04-20T17:27:35.153 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.429e-06 2024-04-20T17:27:35.153 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.153 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.153 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.154 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103931+0000", 2024-04-20T17:27:35.154 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00032633599999999999 2024-04-20T17:27:35.154 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.154 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.154 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:27:35.154 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103944+0000", 2024-04-20T17:27:35.154 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.307e-05 2024-04-20T17:27:35.154 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.154 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.154 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.154 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.159475+0000", 2024-04-20T17:27:35.154 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.055531006000000001 2024-04-20T17:27:35.154 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.154 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.155 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:27:35.155 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.159584+0000", 2024-04-20T17:27:35.155 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010849399999999999 2024-04-20T17:27:35.155 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.155 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.155 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:27:35.155 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.159707+0000", 2024-04-20T17:27:35.155 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012344499999999999 2024-04-20T17:27:35.155 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.155 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.155 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:27:35.155 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.171026+0000", 2024-04-20T17:27:35.155 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.01131892 2024-04-20T17:27:35.155 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.155 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.155 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:35.156 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189843+0000", 2024-04-20T17:27:35.156 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.018816330999999999 2024-04-20T17:27:35.156 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.156 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.156 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:27:35.156 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189889+0000", 2024-04-20T17:27:35.156 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6647999999999998e-05 2024-04-20T17:27:35.156 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.156 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.156 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:27:35.156 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189919+0000", 2024-04-20T17:27:35.156 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9717999999999999e-05 2024-04-20T17:27:35.156 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.156 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:27:35.156 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.157 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.157 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.157 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245523 3.d 3:bc29c29a:::benchmark_data_smithi171_110765_object102308:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:27:35.157 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103692+0000", 2024-04-20T17:27:35.157 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 242.035989124, 2024-04-20T17:27:35.157 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.086309881000000005, 2024-04-20T17:27:35.157 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:27:35.157 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:27:35.157 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:35.157 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:27:35.157 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:27:35.157 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:35.157 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245523 2024-04-20T17:27:35.157 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.157 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:27:35.158 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.158 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:27:35.158 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103692+0000", 2024-04-20T17:27:35.158 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.158 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.158 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.158 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:27:35.158 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103692+0000", 2024-04-20T17:27:35.158 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.158 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.158 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.158 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:27:35.158 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103693+0000", 2024-04-20T17:27:35.158 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1710000000000001e-06 2024-04-20T17:27:35.158 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.159 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.159 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:27:35.159 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103699+0000", 2024-04-20T17:27:35.159 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3900000000000001e-06 2024-04-20T17:27:35.159 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.159 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.159 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:27:35.159 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103709+0000", 2024-04-20T17:27:35.159 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9119999999999993e-06 2024-04-20T17:27:35.159 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.159 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.159 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:27:35.159 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103716+0000", 2024-04-20T17:27:35.159 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0260000000000002e-06 2024-04-20T17:27:35.160 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.160 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.160 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.160 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104008+0000", 2024-04-20T17:27:35.160 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00029273899999999998 2024-04-20T17:27:35.160 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.160 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.160 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:27:35.160 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104018+0000", 2024-04-20T17:27:35.160 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5540000000000003e-06 2024-04-20T17:27:35.160 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.160 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.160 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.160 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.161784+0000", 2024-04-20T17:27:35.160 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.057766296000000002 2024-04-20T17:27:35.161 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.161 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.161 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:27:35.161 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.161875+0000", 2024-04-20T17:27:35.161 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0490999999999998e-05 2024-04-20T17:27:35.161 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.161 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.161 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:27:35.161 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.161953+0000", 2024-04-20T17:27:35.161 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8370999999999999e-05 2024-04-20T17:27:35.161 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.161 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.161 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:27:35.161 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.171034+0000", 2024-04-20T17:27:35.162 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0090805629999999998 2024-04-20T17:27:35.162 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.162 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.162 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:35.162 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189952+0000", 2024-04-20T17:27:35.162 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.018918249000000002 2024-04-20T17:27:35.162 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.162 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.162 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:27:35.162 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189980+0000", 2024-04-20T17:27:35.162 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8469000000000001e-05 2024-04-20T17:27:35.162 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.162 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.162 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:27:35.162 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.190002+0000", 2024-04-20T17:27:35.163 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1650000000000001e-05 2024-04-20T17:27:35.163 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.163 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:27:35.163 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.163 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.163 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.163 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245527 3.d 3:bef30612:::benchmark_data_smithi171_110765_object102312:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:27:35.163 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103727+0000", 2024-04-20T17:27:35.163 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 242.035954756, 2024-04-20T17:27:35.163 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.090154580999999998, 2024-04-20T17:27:35.163 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:27:35.163 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:27:35.163 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:35.163 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:27:35.163 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:27:35.164 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:35.164 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245527 2024-04-20T17:27:35.164 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.164 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:27:35.164 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.164 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:27:35.164 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103727+0000", 2024-04-20T17:27:35.164 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.164 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.164 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.164 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:27:35.164 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103727+0000", 2024-04-20T17:27:35.164 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.164 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.164 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.165 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:27:35.165 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103727+0000", 2024-04-20T17:27:35.165 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6400000000000001e-07 2024-04-20T17:27:35.165 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.165 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.165 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:27:35.165 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103731+0000", 2024-04-20T17:27:35.165 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7589999999999999e-06 2024-04-20T17:27:35.165 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.165 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.165 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:27:35.165 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103737+0000", 2024-04-20T17:27:35.165 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.207e-06 2024-04-20T17:27:35.165 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.165 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.166 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:27:35.166 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103743+0000", 2024-04-20T17:27:35.166 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9209999999999997e-06 2024-04-20T17:27:35.166 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.166 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.166 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.166 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104112+0000", 2024-04-20T17:27:35.166 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00036842100000000002 2024-04-20T17:27:35.166 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.166 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.166 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:27:35.166 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104116+0000", 2024-04-20T17:27:35.166 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2930000000000002e-06 2024-04-20T17:27:35.166 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.167 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.167 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.167 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.163799+0000", 2024-04-20T17:27:35.167 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.059683369999999999 2024-04-20T17:27:35.167 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.167 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.167 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:27:35.167 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.163872+0000", 2024-04-20T17:27:35.167 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2502999999999993e-05 2024-04-20T17:27:35.167 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.167 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.167 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:27:35.167 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.163953+0000", 2024-04-20T17:27:35.167 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0758000000000002e-05 2024-04-20T17:27:35.167 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.168 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.168 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:27:35.168 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.176271+0000", 2024-04-20T17:27:35.168 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012318610000000001 2024-04-20T17:27:35.168 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.168 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.168 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:35.168 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.193822+0000", 2024-04-20T17:27:35.168 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.017550603000000001 2024-04-20T17:27:35.168 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.168 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.168 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:27:35.168 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.193856+0000", 2024-04-20T17:27:35.168 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4239000000000002e-05 2024-04-20T17:27:35.168 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.168 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.169 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:27:35.169 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.193881+0000", 2024-04-20T17:27:35.169 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5133000000000001e-05 2024-04-20T17:27:35.169 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.169 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:27:35.169 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.169 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.169 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.169 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245531 3.d 3:ba69417f:::benchmark_data_smithi171_110765_object102316:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:27:35.169 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103783+0000", 2024-04-20T17:27:35.169 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 242.03589857599999, 2024-04-20T17:27:35.170 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.090206175, 2024-04-20T17:27:35.170 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:27:35.170 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:27:35.170 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:35.170 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:27:35.170 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:27:35.170 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:35.170 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245531 2024-04-20T17:27:35.170 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.170 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:27:35.170 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.170 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:27:35.170 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103783+0000", 2024-04-20T17:27:35.170 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.171 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.171 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.171 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:27:35.171 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103783+0000", 2024-04-20T17:27:35.171 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.171 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.171 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.171 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:27:35.171 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103785+0000", 2024-04-20T17:27:35.171 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.809e-06 2024-04-20T17:27:35.171 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.171 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.171 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:27:35.171 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103802+0000", 2024-04-20T17:27:35.171 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7075000000000002e-05 2024-04-20T17:27:35.172 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.172 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.172 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:27:35.172 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103815+0000", 2024-04-20T17:27:35.172 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3019e-05 2024-04-20T17:27:35.172 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.172 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.172 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:27:35.172 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103825+0000", 2024-04-20T17:27:35.172 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8309999999999996e-06 2024-04-20T17:27:35.172 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.172 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.172 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.172 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104133+0000", 2024-04-20T17:27:35.173 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00030879900000000001 2024-04-20T17:27:35.173 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.173 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.173 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:27:35.173 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104136+0000", 2024-04-20T17:27:35.173 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0910000000000001e-06 2024-04-20T17:27:35.173 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.173 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.173 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.173 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.165809+0000", 2024-04-20T17:27:35.173 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.061672957 2024-04-20T17:27:35.173 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.173 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.173 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:27:35.173 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.165889+0000", 2024-04-20T17:27:35.174 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9381000000000005e-05 2024-04-20T17:27:35.174 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.174 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.174 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:27:35.174 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.165967+0000", 2024-04-20T17:27:35.174 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7849999999999995e-05 2024-04-20T17:27:35.174 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.174 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.174 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:27:35.174 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.176279+0000", 2024-04-20T17:27:35.174 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010312702999999999 2024-04-20T17:27:35.174 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.174 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.174 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:35.174 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.193918+0000", 2024-04-20T17:27:35.175 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.017638628999999999 2024-04-20T17:27:35.175 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.175 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.175 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:27:35.175 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.193960+0000", 2024-04-20T17:27:35.175 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1746000000000003e-05 2024-04-20T17:27:35.175 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.175 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.175 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:27:35.175 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.193989+0000", 2024-04-20T17:27:35.175 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9285000000000002e-05 2024-04-20T17:27:35.175 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.175 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:27:35.175 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.175 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.176 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.176 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245546 3.d 3:b08c00b6:::benchmark_data_smithi171_110765_object102331:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:27:35.176 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103839+0000", 2024-04-20T17:27:35.176 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 242.03584186500001, 2024-04-20T17:27:35.176 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.097118706999999999, 2024-04-20T17:27:35.176 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:27:35.176 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:27:35.176 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:35.176 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:27:35.176 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:27:35.176 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:35.176 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245546 2024-04-20T17:27:35.176 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.176 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:27:35.176 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.177 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:27:35.177 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103839+0000", 2024-04-20T17:27:35.177 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.177 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.177 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.177 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:27:35.177 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103839+0000", 2024-04-20T17:27:35.177 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.177 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.177 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.177 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:27:35.177 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103840+0000", 2024-04-20T17:27:35.177 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1800000000000004e-07 2024-04-20T17:27:35.177 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.177 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.178 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:27:35.178 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103845+0000", 2024-04-20T17:27:35.178 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3000000000000003e-06 2024-04-20T17:27:35.178 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.178 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.178 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:27:35.178 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103853+0000", 2024-04-20T17:27:35.178 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9799999999999998e-06 2024-04-20T17:27:35.178 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.178 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.178 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:27:35.178 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103859+0000", 2024-04-20T17:27:35.178 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7020000000000004e-06 2024-04-20T17:27:35.178 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.178 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.179 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.179 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104182+0000", 2024-04-20T17:27:35.179 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00032271300000000003 2024-04-20T17:27:35.179 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.179 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.179 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:27:35.179 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104201+0000", 2024-04-20T17:27:35.179 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8547999999999999e-05 2024-04-20T17:27:35.179 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.179 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.179 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.179 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.167891+0000", 2024-04-20T17:27:35.179 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.063690467000000001 2024-04-20T17:27:35.179 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.179 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.180 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:27:35.180 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.168022+0000", 2024-04-20T17:27:35.180 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013114499999999999 2024-04-20T17:27:35.180 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.180 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.180 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:27:35.180 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.168147+0000", 2024-04-20T17:27:35.180 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012454299999999999 2024-04-20T17:27:35.180 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.180 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.180 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:27:35.180 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.181127+0000", 2024-04-20T17:27:35.180 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012980302000000001 2024-04-20T17:27:35.180 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.180 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.181 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:35.181 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.200861+0000", 2024-04-20T17:27:35.181 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.019734266 2024-04-20T17:27:35.181 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.181 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.181 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:27:35.181 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.200910+0000", 2024-04-20T17:27:35.181 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8294999999999998e-05 2024-04-20T17:27:35.181 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.181 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.181 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:27:35.181 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.200958+0000", 2024-04-20T17:27:35.181 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8528e-05 2024-04-20T17:27:35.181 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.181 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:27:35.182 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.182 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.182 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.182 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245552 3.d 3:b5dda02d:::benchmark_data_smithi171_110765_object102337:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:27:35.182 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103886+0000", 2024-04-20T17:27:35.182 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 242.03579501600001, 2024-04-20T17:27:35.182 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.097162344999999997, 2024-04-20T17:27:35.182 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:27:35.182 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:27:35.182 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:35.182 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:27:35.182 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:27:35.182 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:35.182 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245552 2024-04-20T17:27:35.182 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.183 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:27:35.183 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.183 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:27:35.183 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103886+0000", 2024-04-20T17:27:35.183 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.183 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.183 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.183 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:27:35.183 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103886+0000", 2024-04-20T17:27:35.183 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.183 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.183 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.183 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:27:35.183 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103887+0000", 2024-04-20T17:27:35.183 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2799999999999995e-07 2024-04-20T17:27:35.184 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.184 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.184 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:27:35.184 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103890+0000", 2024-04-20T17:27:35.184 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.084e-06 2024-04-20T17:27:35.184 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.184 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.184 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:27:35.184 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103896+0000", 2024-04-20T17:27:35.184 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3750000000000002e-06 2024-04-20T17:27:35.184 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.184 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.184 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:27:35.184 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103900+0000", 2024-04-20T17:27:35.184 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1679999999999997e-06 2024-04-20T17:27:35.185 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.185 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.185 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.185 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104230+0000", 2024-04-20T17:27:35.185 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00033005300000000002 2024-04-20T17:27:35.185 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.185 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.185 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:27:35.185 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104233+0000", 2024-04-20T17:27:35.185 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2260000000000001e-06 2024-04-20T17:27:35.185 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.185 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.185 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.185 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.171080+0000", 2024-04-20T17:27:35.185 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.066846953000000001 2024-04-20T17:27:35.186 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.186 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.186 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:27:35.186 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.171210+0000", 2024-04-20T17:27:35.186 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000130376 2024-04-20T17:27:35.186 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.186 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.186 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:27:35.186 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.171343+0000", 2024-04-20T17:27:35.186 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013265799999999999 2024-04-20T17:27:35.186 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.186 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.186 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:27:35.186 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.181134+0000", 2024-04-20T17:27:35.187 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0097907150000000002 2024-04-20T17:27:35.187 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.187 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.187 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:35.187 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.200992+0000", 2024-04-20T17:27:35.187 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.019858058000000001 2024-04-20T17:27:35.187 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.187 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.187 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:27:35.187 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.201024+0000", 2024-04-20T17:27:35.187 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1914e-05 2024-04-20T17:27:35.187 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.187 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.187 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:27:35.187 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.201049+0000", 2024-04-20T17:27:35.188 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5037000000000001e-05 2024-04-20T17:27:35.188 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.188 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:27:35.188 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.188 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.188 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.188 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245601 3.d 3:be8206df:::benchmark_data_smithi171_110765_object102386:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:27:35.188 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103906+0000", 2024-04-20T17:27:35.188 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 242.03577525, 2024-04-20T17:27:35.188 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.10597208700000001, 2024-04-20T17:27:35.188 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:27:35.188 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:27:35.188 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:35.188 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:27:35.188 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:27:35.189 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:35.189 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245601 2024-04-20T17:27:35.189 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.189 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:27:35.189 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.189 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:27:35.189 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103906+0000", 2024-04-20T17:27:35.189 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.189 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.189 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.189 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:27:35.189 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103906+0000", 2024-04-20T17:27:35.189 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.189 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.189 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.190 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:27:35.190 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103906+0000", 2024-04-20T17:27:35.190 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6100000000000002e-07 2024-04-20T17:27:35.190 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.190 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.190 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:27:35.190 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103908+0000", 2024-04-20T17:27:35.190 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7439999999999999e-06 2024-04-20T17:27:35.190 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.190 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.190 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:27:35.190 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103911+0000", 2024-04-20T17:27:35.190 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.954e-06 2024-04-20T17:27:35.190 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.191 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.191 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:27:35.191 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103915+0000", 2024-04-20T17:27:35.191 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9450000000000003e-06 2024-04-20T17:27:35.191 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.191 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.191 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.191 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.174025+0000", 2024-04-20T17:27:35.191 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.070109677999999995 2024-04-20T17:27:35.191 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.191 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.191 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:27:35.191 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.174130+0000", 2024-04-20T17:27:35.191 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010515599999999999 2024-04-20T17:27:35.191 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.192 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.192 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:27:35.192 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.174204+0000", 2024-04-20T17:27:35.192 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4097000000000006e-05 2024-04-20T17:27:35.192 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.192 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.192 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:27:35.192 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.181429+0000", 2024-04-20T17:27:35.192 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0072249250000000001 2024-04-20T17:27:35.192 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.192 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.192 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:35.192 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.209781+0000", 2024-04-20T17:27:35.192 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.028351694 2024-04-20T17:27:35.192 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.193 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.193 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:27:35.193 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.209834+0000", 2024-04-20T17:27:35.193 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3309000000000001e-05 2024-04-20T17:27:35.193 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.193 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.193 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:27:35.193 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.209878+0000", 2024-04-20T17:27:35.193 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4224000000000002e-05 2024-04-20T17:27:35.193 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.193 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:27:35.193 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.194 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.194 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.194 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245611 3.d 3:b89d9251:::benchmark_data_smithi171_110765_object102396:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:27:35.194 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103921+0000", 2024-04-20T17:27:35.194 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 242.03576005900001, 2024-04-20T17:27:35.194 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.106082866, 2024-04-20T17:27:35.194 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:27:35.194 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:27:35.194 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:35.194 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:27:35.194 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:27:35.194 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:35.194 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245611 2024-04-20T17:27:35.194 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.195 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:27:35.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.195 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:27:35.195 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103921+0000", 2024-04-20T17:27:35.195 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.195 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:27:35.195 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103921+0000", 2024-04-20T17:27:35.195 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.195 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:27:35.195 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103922+0000", 2024-04-20T17:27:35.195 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1999999999999999e-07 2024-04-20T17:27:35.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.196 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.196 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:27:35.196 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103926+0000", 2024-04-20T17:27:35.196 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3630000000000004e-06 2024-04-20T17:27:35.196 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.196 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.196 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:27:35.196 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103932+0000", 2024-04-20T17:27:35.196 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8259999999999998e-06 2024-04-20T17:27:35.196 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.196 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.196 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:27:35.196 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103948+0000", 2024-04-20T17:27:35.196 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6240999999999999e-05 2024-04-20T17:27:35.196 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.197 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.197 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.176327+0000", 2024-04-20T17:27:35.197 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.072379053999999998 2024-04-20T17:27:35.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.197 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:27:35.197 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.176443+0000", 2024-04-20T17:27:35.197 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011536600000000001 2024-04-20T17:27:35.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.197 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:27:35.197 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.176568+0000", 2024-04-20T17:27:35.197 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000125184 2024-04-20T17:27:35.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.198 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:27:35.198 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.181432+0000", 2024-04-20T17:27:35.198 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0048639720000000003 2024-04-20T17:27:35.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.198 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:35.198 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.209927+0000", 2024-04-20T17:27:35.198 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.028494789999999999 2024-04-20T17:27:35.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.198 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:27:35.198 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.209969+0000", 2024-04-20T17:27:35.198 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1773000000000001e-05 2024-04-20T17:27:35.199 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.199 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.199 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:27:35.199 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.210004+0000", 2024-04-20T17:27:35.199 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5577000000000002e-05 2024-04-20T17:27:35.199 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.199 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:27:35.199 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.199 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.199 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.199 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245620 3.d 3:be474e21:::benchmark_data_smithi171_110765_object102405:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:27:35.199 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103957+0000", 2024-04-20T17:27:35.199 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 242.035724061, 2024-04-20T17:27:35.199 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.112691712, 2024-04-20T17:27:35.199 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:27:35.200 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:27:35.200 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:35.200 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:27:35.200 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:27:35.200 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:35.200 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245620 2024-04-20T17:27:35.200 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.200 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:27:35.200 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.200 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:27:35.200 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103957+0000", 2024-04-20T17:27:35.200 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.200 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.200 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.200 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:27:35.201 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103957+0000", 2024-04-20T17:27:35.201 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.201 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.201 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.201 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:27:35.201 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103958+0000", 2024-04-20T17:27:35.201 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1299999999999999e-07 2024-04-20T17:27:35.201 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.201 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.201 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:27:35.201 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103962+0000", 2024-04-20T17:27:35.201 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6119999999999999e-06 2024-04-20T17:27:35.201 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.201 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.201 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:27:35.202 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103967+0000", 2024-04-20T17:27:35.202 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7679999999999997e-06 2024-04-20T17:27:35.202 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.202 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.202 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:27:35.202 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103971+0000", 2024-04-20T17:27:35.202 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6100000000000002e-06 2024-04-20T17:27:35.202 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.202 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.202 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.202 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.178796+0000", 2024-04-20T17:27:35.202 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.074824511999999996 2024-04-20T17:27:35.202 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.203 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:27:35.203 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.178903+0000", 2024-04-20T17:27:35.203 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000107731 2024-04-20T17:27:35.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.203 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:27:35.203 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.179004+0000", 2024-04-20T17:27:35.203 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000101139 2024-04-20T17:27:35.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.203 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:27:35.203 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.187634+0000", 2024-04-20T17:27:35.203 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0086293210000000006 2024-04-20T17:27:35.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.204 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:35.204 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.216547+0000", 2024-04-20T17:27:35.204 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.028912817 2024-04-20T17:27:35.204 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.204 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:27:35.204 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.216604+0000", 2024-04-20T17:27:35.204 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7009999999999998e-05 2024-04-20T17:27:35.204 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.205 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.205 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:27:35.205 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.216649+0000", 2024-04-20T17:27:35.205 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5379e-05 2024-04-20T17:27:35.205 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.205 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:27:35.205 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.205 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.205 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.205 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245622 3.d 3:b177e667:::benchmark_data_smithi171_110765_object102407:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:27:35.205 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103975+0000", 2024-04-20T17:27:35.205 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 242.035705952, 2024-04-20T17:27:35.205 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.112781889, 2024-04-20T17:27:35.205 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:27:35.205 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:27:35.206 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:35.206 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:27:35.206 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:27:35.206 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:35.206 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245622 2024-04-20T17:27:35.206 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.206 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:27:35.206 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.206 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:27:35.206 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103975+0000", 2024-04-20T17:27:35.206 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.206 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.206 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.206 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:27:35.206 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103975+0000", 2024-04-20T17:27:35.207 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.207 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.207 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.207 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:27:35.207 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103976+0000", 2024-04-20T17:27:35.207 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9000000000000002e-07 2024-04-20T17:27:35.207 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.207 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.207 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:27:35.207 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103978+0000", 2024-04-20T17:27:35.207 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.821e-06 2024-04-20T17:27:35.207 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.207 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.207 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:27:35.208 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103981+0000", 2024-04-20T17:27:35.208 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0350000000000002e-06 2024-04-20T17:27:35.208 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.208 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.208 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:27:35.208 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103984+0000", 2024-04-20T17:27:35.208 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0089999999999999e-06 2024-04-20T17:27:35.208 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.208 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.208 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.208 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.181477+0000", 2024-04-20T17:27:35.208 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.077492970999999994 2024-04-20T17:27:35.208 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.208 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.208 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:27:35.209 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.181593+0000", 2024-04-20T17:27:35.209 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000115979 2024-04-20T17:27:35.209 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.209 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.209 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:27:35.209 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.181711+0000", 2024-04-20T17:27:35.209 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000118245 2024-04-20T17:27:35.209 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.209 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.209 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:27:35.209 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189686+0000", 2024-04-20T17:27:35.209 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0079754270000000002 2024-04-20T17:27:35.209 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.209 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.209 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:35.210 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.216686+0000", 2024-04-20T17:27:35.210 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.026999433 2024-04-20T17:27:35.210 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.210 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:27:35.210 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.216718+0000", 2024-04-20T17:27:35.210 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2570000000000002e-05 2024-04-20T17:27:35.210 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.210 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:27:35.210 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.216757+0000", 2024-04-20T17:27:35.210 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9008999999999999e-05 2024-04-20T17:27:35.210 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.211 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:27:35.211 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.211 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.211 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.211 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245623 3.d 3:becddf3a:::benchmark_data_smithi171_110765_object102408:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:27:35.211 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103997+0000", 2024-04-20T17:27:35.211 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 242.03568481299999, 2024-04-20T17:27:35.211 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.11723939999999999, 2024-04-20T17:27:35.211 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:27:35.211 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:27:35.211 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:35.211 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:27:35.211 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:27:35.211 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:35.211 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245623 2024-04-20T17:27:35.212 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.212 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:27:35.212 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.212 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:27:35.212 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103997+0000", 2024-04-20T17:27:35.212 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.212 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.212 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.212 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:27:35.212 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103997+0000", 2024-04-20T17:27:35.212 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.212 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.212 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.212 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:27:35.212 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103997+0000", 2024-04-20T17:27:35.213 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8099999999999998e-07 2024-04-20T17:27:35.213 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.213 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.213 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:27:35.213 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103999+0000", 2024-04-20T17:27:35.213 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6929999999999999e-06 2024-04-20T17:27:35.213 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.213 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.213 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:27:35.213 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104008+0000", 2024-04-20T17:27:35.213 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0310000000000004e-06 2024-04-20T17:27:35.213 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.213 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.213 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:27:35.213 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104011+0000", 2024-04-20T17:27:35.214 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8060000000000001e-06 2024-04-20T17:27:35.214 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.214 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.214 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.214 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.185112+0000", 2024-04-20T17:27:35.214 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.081100835999999996 2024-04-20T17:27:35.214 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.214 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.214 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:27:35.214 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.185198+0000", 2024-04-20T17:27:35.214 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6045999999999996e-05 2024-04-20T17:27:35.214 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.214 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.214 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:27:35.215 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.185274+0000", 2024-04-20T17:27:35.215 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6091e-05 2024-04-20T17:27:35.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.215 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.215 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:27:35.215 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189787+0000", 2024-04-20T17:27:35.215 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045125410000000001 2024-04-20T17:27:35.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.215 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.215 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:35.215 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.221137+0000", 2024-04-20T17:27:35.215 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.031350204 2024-04-20T17:27:35.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.215 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.215 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:27:35.216 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.221185+0000", 2024-04-20T17:27:35.216 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7410000000000002e-05 2024-04-20T17:27:35.216 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.216 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.216 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:27:35.216 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.221236+0000", 2024-04-20T17:27:35.216 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1360999999999998e-05 2024-04-20T17:27:35.216 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.216 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:27:35.216 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.216 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.216 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.216 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245630 3.d 3:b005673c:::benchmark_data_smithi171_110765_object102415:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:27:35.216 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.104016+0000", 2024-04-20T17:27:35.216 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 242.03566562200001, 2024-04-20T17:27:35.217 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.117355654, 2024-04-20T17:27:35.217 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:27:35.217 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:27:35.217 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:35.217 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:27:35.217 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:27:35.217 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:35.217 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245630 2024-04-20T17:27:35.217 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.217 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:27:35.217 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.217 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:27:35.217 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104016+0000", 2024-04-20T17:27:35.217 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.218 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.218 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:27:35.218 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104016+0000", 2024-04-20T17:27:35.218 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.218 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.218 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:27:35.218 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104016+0000", 2024-04-20T17:27:35.218 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5100000000000001e-07 2024-04-20T17:27:35.218 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.218 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:27:35.218 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104018+0000", 2024-04-20T17:27:35.218 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2409999999999998e-06 2024-04-20T17:27:35.219 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.219 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.219 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:27:35.219 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104022+0000", 2024-04-20T17:27:35.219 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8340000000000001e-06 2024-04-20T17:27:35.219 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.219 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.219 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:27:35.219 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104027+0000", 2024-04-20T17:27:35.219 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.369e-06 2024-04-20T17:27:35.219 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.219 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.219 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.219 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.187654+0000", 2024-04-20T17:27:35.219 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.083627286999999995 2024-04-20T17:27:35.219 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.220 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.220 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:27:35.220 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.187717+0000", 2024-04-20T17:27:35.220 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3417999999999995e-05 2024-04-20T17:27:35.220 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.220 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.220 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:27:35.220 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.187783+0000", 2024-04-20T17:27:35.220 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5574000000000006e-05 2024-04-20T17:27:35.220 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.220 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.220 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:27:35.220 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.190010+0000", 2024-04-20T17:27:35.220 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022267150000000002 2024-04-20T17:27:35.220 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.221 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.221 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:35.221 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.221276+0000", 2024-04-20T17:27:35.221 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.031266250000000002 2024-04-20T17:27:35.221 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.221 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.221 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:27:35.221 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.221332+0000", 2024-04-20T17:27:35.221 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6298999999999997e-05 2024-04-20T17:27:35.221 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.221 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.221 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:27:35.221 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.221371+0000", 2024-04-20T17:27:35.221 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9316000000000001e-05 2024-04-20T17:27:35.221 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.221 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:27:35.222 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.222 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.222 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.222 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245803 3.f 3:fc19cf94:::benchmark_data_smithi171_110765_object102588:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:27:35.222 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:34.105692+0000", 2024-04-20T17:27:35.222 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 241.03398964799999, 2024-04-20T17:27:35.222 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.081553284000000004, 2024-04-20T17:27:35.222 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:27:35.222 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:27:35.222 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:35.222 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:27:35.222 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:27:35.222 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:35.222 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245803 2024-04-20T17:27:35.223 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.223 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:27:35.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.223 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:27:35.223 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105692+0000", 2024-04-20T17:27:35.223 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.223 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.223 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:27:35.223 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105692+0000", 2024-04-20T17:27:35.223 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.223 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.223 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:27:35.223 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105692+0000", 2024-04-20T17:27:35.224 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2199999999999999e-07 2024-04-20T17:27:35.224 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.224 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.224 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:27:35.224 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105694+0000", 2024-04-20T17:27:35.224 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.593e-06 2024-04-20T17:27:35.224 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.224 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.224 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:27:35.224 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105697+0000", 2024-04-20T17:27:35.224 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3060000000000002e-06 2024-04-20T17:27:35.224 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.224 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.224 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:27:35.224 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105700+0000", 2024-04-20T17:27:35.225 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8430000000000001e-06 2024-04-20T17:27:35.225 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.225 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.225 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.225 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105794+0000", 2024-04-20T17:27:35.225 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3948000000000002e-05 2024-04-20T17:27:35.225 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.225 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.225 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:27:35.225 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105797+0000", 2024-04-20T17:27:35.225 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6370000000000001e-06 2024-04-20T17:27:35.225 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.225 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.225 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.225 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.163315+0000", 2024-04-20T17:27:35.226 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.057517243000000003 2024-04-20T17:27:35.226 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.226 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.226 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:27:35.226 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.163373+0000", 2024-04-20T17:27:35.226 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8399000000000001e-05 2024-04-20T17:27:35.226 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.226 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.226 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 11", 2024-04-20T17:27:35.226 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.163436+0000", 2024-04-20T17:27:35.226 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2589999999999995e-05 2024-04-20T17:27:35.226 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.226 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.226 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:27:35.226 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.172663+0000", 2024-04-20T17:27:35.227 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0092275810000000003 2024-04-20T17:27:35.227 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.227 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.227 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:35.227 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.187151+0000", 2024-04-20T17:27:35.227 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.014487635 2024-04-20T17:27:35.227 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.227 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.227 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:27:35.227 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.187204+0000", 2024-04-20T17:27:35.227 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2793000000000001e-05 2024-04-20T17:27:35.227 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.227 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.227 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:27:35.227 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.187245+0000", 2024-04-20T17:27:35.228 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1294000000000001e-05 2024-04-20T17:27:35.228 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.228 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:27:35.228 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.228 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.228 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.228 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245820 3.f 3:f94b0a75:::benchmark_data_smithi171_110765_object102605:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:27:35.228 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:34.105704+0000", 2024-04-20T17:27:35.228 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 241.03397779700001, 2024-04-20T17:27:35.228 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.084848861999999997, 2024-04-20T17:27:35.228 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:27:35.228 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:27:35.228 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:35.228 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:27:35.228 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:27:35.229 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:35.229 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245820 2024-04-20T17:27:35.229 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.229 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:27:35.229 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.229 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:27:35.229 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105704+0000", 2024-04-20T17:27:35.229 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.229 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.229 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.229 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:27:35.229 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105704+0000", 2024-04-20T17:27:35.229 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.229 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.229 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.230 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:27:35.230 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105704+0000", 2024-04-20T17:27:35.230 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1e-07 2024-04-20T17:27:35.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.230 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:27:35.230 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105710+0000", 2024-04-20T17:27:35.230 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6890000000000001e-06 2024-04-20T17:27:35.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.230 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:27:35.230 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105713+0000", 2024-04-20T17:27:35.230 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.061e-06 2024-04-20T17:27:35.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.231 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:27:35.231 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105716+0000", 2024-04-20T17:27:35.231 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7780000000000001e-06 2024-04-20T17:27:35.231 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.231 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.231 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.231 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105816+0000", 2024-04-20T17:27:35.231 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9381000000000003e-05 2024-04-20T17:27:35.231 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.231 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.231 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:27:35.231 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105819+0000", 2024-04-20T17:27:35.231 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9689999999999999e-06 2024-04-20T17:27:35.231 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.231 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.232 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.232 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.165004+0000", 2024-04-20T17:27:35.232 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.059185298999999997 2024-04-20T17:27:35.232 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.232 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.232 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:27:35.232 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.165065+0000", 2024-04-20T17:27:35.232 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1136000000000006e-05 2024-04-20T17:27:35.232 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.232 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.232 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 11", 2024-04-20T17:27:35.232 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.165126+0000", 2024-04-20T17:27:35.232 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0863999999999999e-05 2024-04-20T17:27:35.232 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.232 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.232 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:27:35.233 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.172669+0000", 2024-04-20T17:27:35.233 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.007543396 2024-04-20T17:27:35.233 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.233 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.233 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:35.233 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.190456+0000", 2024-04-20T17:27:35.233 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.017786309 2024-04-20T17:27:35.233 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.233 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.233 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:27:35.233 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.190507+0000", 2024-04-20T17:27:35.233 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1102e-05 2024-04-20T17:27:35.233 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.233 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.233 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:27:35.234 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.190552+0000", 2024-04-20T17:27:35.234 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5568000000000001e-05 2024-04-20T17:27:35.234 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.234 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:27:35.234 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.234 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.234 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.234 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245821 3.f 3:f7671ae3:::benchmark_data_smithi171_110765_object102606:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:27:35.234 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:34.105727+0000", 2024-04-20T17:27:35.234 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 241.03395407100001, 2024-04-20T17:27:35.234 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.084930124999999995, 2024-04-20T17:27:35.234 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:27:35.234 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:27:35.234 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:35.234 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:27:35.235 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:27:35.235 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:35.235 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245821 2024-04-20T17:27:35.235 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.235 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:27:35.235 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.235 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:27:35.235 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105727+0000", 2024-04-20T17:27:35.235 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.235 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.235 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.235 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:27:35.235 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105727+0000", 2024-04-20T17:27:35.235 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.236 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.236 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.236 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:27:35.236 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105728+0000", 2024-04-20T17:27:35.236 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1699999999999999e-07 2024-04-20T17:27:35.236 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.236 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.236 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:27:35.236 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105729+0000", 2024-04-20T17:27:35.236 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4640000000000001e-06 2024-04-20T17:27:35.236 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.236 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.236 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:27:35.236 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105733+0000", 2024-04-20T17:27:35.236 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5860000000000001e-06 2024-04-20T17:27:35.237 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.237 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.237 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:27:35.237 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105735+0000", 2024-04-20T17:27:35.237 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7829999999999999e-06 2024-04-20T17:27:35.237 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.237 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.237 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.237 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105836+0000", 2024-04-20T17:27:35.237 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000100574 2024-04-20T17:27:35.237 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.237 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.237 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:27:35.237 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105841+0000", 2024-04-20T17:27:35.237 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.729e-06 2024-04-20T17:27:35.238 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.238 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.238 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.238 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.166821+0000", 2024-04-20T17:27:35.238 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.060979836000000003 2024-04-20T17:27:35.238 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.239 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.239 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:27:35.239 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.166913+0000", 2024-04-20T17:27:35.239 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2724000000000002e-05 2024-04-20T17:27:35.239 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.239 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.239 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 11", 2024-04-20T17:27:35.239 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.167000+0000", 2024-04-20T17:27:35.239 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6802e-05 2024-04-20T17:27:35.240 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.240 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.240 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:27:35.240 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.174876+0000", 2024-04-20T17:27:35.240 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0078754649999999999 2024-04-20T17:27:35.240 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.240 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.240 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:35.240 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.190587+0000", 2024-04-20T17:27:35.240 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015711573 2024-04-20T17:27:35.240 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.240 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.240 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:27:35.240 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.190626+0000", 2024-04-20T17:27:35.240 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8909999999999998e-05 2024-04-20T17:27:35.241 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.241 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.241 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:27:35.241 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.190657+0000", 2024-04-20T17:27:35.241 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1362000000000002e-05 2024-04-20T17:27:35.241 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.241 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:27:35.241 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.241 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.241 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.241 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245828 3.f 3:f231c1d2:::benchmark_data_smithi171_110765_object102613:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:27:35.241 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:34.105739+0000", 2024-04-20T17:27:35.241 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 241.033942147, 2024-04-20T17:27:35.241 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.085013469999999994, 2024-04-20T17:27:35.241 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:27:35.242 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:27:35.242 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:35.242 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:27:35.242 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:27:35.242 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:35.242 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245828 2024-04-20T17:27:35.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.242 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:27:35.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.242 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:27:35.242 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105739+0000", 2024-04-20T17:27:35.242 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.243 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:27:35.243 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105739+0000", 2024-04-20T17:27:35.243 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.243 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.243 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.243 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:27:35.243 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105740+0000", 2024-04-20T17:27:35.243 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1e-07 2024-04-20T17:27:35.243 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.243 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.243 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:27:35.243 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105741+0000", 2024-04-20T17:27:35.243 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.221e-06 2024-04-20T17:27:35.243 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.244 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.244 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:27:35.244 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105744+0000", 2024-04-20T17:27:35.244 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.892e-06 2024-04-20T17:27:35.244 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.244 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.244 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:27:35.244 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105746+0000", 2024-04-20T17:27:35.244 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6840000000000001e-06 2024-04-20T17:27:35.244 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.244 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.244 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.244 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105858+0000", 2024-04-20T17:27:35.244 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011199199999999999 2024-04-20T17:27:35.244 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.245 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.245 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:27:35.245 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105861+0000", 2024-04-20T17:27:35.245 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9440000000000001e-06 2024-04-20T17:27:35.245 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.245 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.245 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.245 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.168822+0000", 2024-04-20T17:27:35.245 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.062960753999999994 2024-04-20T17:27:35.245 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.245 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.245 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:27:35.245 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.168895+0000", 2024-04-20T17:27:35.245 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2817000000000004e-05 2024-04-20T17:27:35.245 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.246 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.246 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 11", 2024-04-20T17:27:35.246 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.168969+0000", 2024-04-20T17:27:35.246 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4091000000000006e-05 2024-04-20T17:27:35.246 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.246 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.246 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:27:35.246 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.174881+0000", 2024-04-20T17:27:35.246 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0059125870000000004 2024-04-20T17:27:35.246 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.246 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.246 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:35.246 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.190687+0000", 2024-04-20T17:27:35.246 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015805475999999999 2024-04-20T17:27:35.246 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.247 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:27:35.247 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.190724+0000", 2024-04-20T17:27:35.247 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6738e-05 2024-04-20T17:27:35.247 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.247 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:27:35.247 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.190753+0000", 2024-04-20T17:27:35.247 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8963999999999999e-05 2024-04-20T17:27:35.247 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.247 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:27:35.247 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.247 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.247 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245831 3.f 3:fd5d268b:::benchmark_data_smithi171_110765_object102616:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:27:35.248 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:34.105750+0000", 2024-04-20T17:27:35.248 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 241.03393151899999, 2024-04-20T17:27:35.248 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.090378368000000001, 2024-04-20T17:27:35.248 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:27:35.248 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:27:35.248 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:35.248 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:27:35.248 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:27:35.248 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:35.248 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245831 2024-04-20T17:27:35.248 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.248 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:27:35.248 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.248 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:27:35.248 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105750+0000", 2024-04-20T17:27:35.248 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.249 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.249 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:27:35.249 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105750+0000", 2024-04-20T17:27:35.249 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.249 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.249 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:27:35.249 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105750+0000", 2024-04-20T17:27:35.249 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7000000000000001e-07 2024-04-20T17:27:35.249 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.249 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:27:35.249 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105751+0000", 2024-04-20T17:27:35.250 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.091e-06 2024-04-20T17:27:35.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:27:35.250 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105754+0000", 2024-04-20T17:27:35.250 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.559e-06 2024-04-20T17:27:35.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:27:35.250 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105757+0000", 2024-04-20T17:27:35.250 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.819e-06 2024-04-20T17:27:35.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.250 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105878+0000", 2024-04-20T17:27:35.251 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000121839 2024-04-20T17:27:35.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.251 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:27:35.251 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105882+0000", 2024-04-20T17:27:35.251 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.224e-06 2024-04-20T17:27:35.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.251 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.251 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.170692+0000", 2024-04-20T17:27:35.251 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.064810693000000003 2024-04-20T17:27:35.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.251 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:27:35.251 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.170783+0000", 2024-04-20T17:27:35.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0681999999999997e-05 2024-04-20T17:27:35.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.252 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 11", 2024-04-20T17:27:35.252 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.170858+0000", 2024-04-20T17:27:35.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5333999999999994e-05 2024-04-20T17:27:35.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.252 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:27:35.252 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.175098+0000", 2024-04-20T17:27:35.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.004239278 2024-04-20T17:27:35.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.252 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:35.253 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.196021+0000", 2024-04-20T17:27:35.253 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.020923529 2024-04-20T17:27:35.253 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.253 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.253 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:27:35.253 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.196081+0000", 2024-04-20T17:27:35.253 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0188e-05 2024-04-20T17:27:35.253 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.253 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.253 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:27:35.253 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.196128+0000", 2024-04-20T17:27:35.253 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6862e-05 2024-04-20T17:27:35.253 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.253 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:27:35.253 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.254 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.254 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.254 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245857 3.f 3:f1991b6e:::benchmark_data_smithi171_110765_object102642:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:27:35.254 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:34.105770+0000", 2024-04-20T17:27:35.254 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 241.03391099000001, 2024-04-20T17:27:35.254 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.091072343, 2024-04-20T17:27:35.254 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:27:35.254 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:27:35.254 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:35.254 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:27:35.254 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:27:35.254 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:35.254 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245857 2024-04-20T17:27:35.254 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.254 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:27:35.255 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.255 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:27:35.255 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105770+0000", 2024-04-20T17:27:35.255 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.255 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.255 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.255 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:27:35.255 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105770+0000", 2024-04-20T17:27:35.255 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.255 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.255 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.255 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:27:35.255 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105771+0000", 2024-04-20T17:27:35.255 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.84e-07 2024-04-20T17:27:35.255 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:27:35.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105772+0000", 2024-04-20T17:27:35.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3939999999999999e-06 2024-04-20T17:27:35.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:27:35.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105776+0000", 2024-04-20T17:27:35.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8550000000000004e-06 2024-04-20T17:27:35.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:27:35.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105779+0000", 2024-04-20T17:27:35.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.768e-06 2024-04-20T17:27:35.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.257 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.257 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.257 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105899+0000", 2024-04-20T17:27:35.257 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012002600000000001 2024-04-20T17:27:35.257 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.257 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.257 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:27:35.257 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105902+0000", 2024-04-20T17:27:35.257 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9079999999999999e-06 2024-04-20T17:27:35.257 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.257 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.257 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.257 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.172714+0000", 2024-04-20T17:27:35.257 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.066812508000000007 2024-04-20T17:27:35.257 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.258 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.258 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:27:35.258 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.172829+0000", 2024-04-20T17:27:35.258 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000114436 2024-04-20T17:27:35.258 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.258 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.258 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 11", 2024-04-20T17:27:35.258 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.172951+0000", 2024-04-20T17:27:35.258 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012268899999999999 2024-04-20T17:27:35.258 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.258 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.258 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:27:35.258 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.177495+0000", 2024-04-20T17:27:35.258 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045431789999999996 2024-04-20T17:27:35.259 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.259 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.259 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:35.259 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.196749+0000", 2024-04-20T17:27:35.259 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.019254060999999999 2024-04-20T17:27:35.259 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.259 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.259 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:27:35.259 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.196800+0000", 2024-04-20T17:27:35.259 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1209999999999999e-05 2024-04-20T17:27:35.259 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.259 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.259 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:27:35.259 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.196843+0000", 2024-04-20T17:27:35.259 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2925e-05 2024-04-20T17:27:35.260 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.260 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:27:35.260 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.260 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.260 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.260 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245891 3.f 3:fe0f9ce5:::benchmark_data_smithi171_110765_object102676:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:27:35.260 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:34.105792+0000", 2024-04-20T17:27:35.260 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 241.03388969, 2024-04-20T17:27:35.260 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.091162981000000004, 2024-04-20T17:27:35.260 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:27:35.260 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:27:35.260 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:35.260 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:27:35.260 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:27:35.260 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:27:35.260 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245891 2024-04-20T17:27:35.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.261 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:27:35.261 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.261 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:27:35.261 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105792+0000", 2024-04-20T17:27:35.261 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.261 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.261 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:27:35.261 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105792+0000", 2024-04-20T17:27:35.261 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:27:35.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.261 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.261 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:27:35.261 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105792+0000", 2024-04-20T17:27:35.262 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9400000000000001e-07 2024-04-20T17:27:35.262 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.262 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:27:35.262 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105793+0000", 2024-04-20T17:27:35.262 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.398e-06 2024-04-20T17:27:35.262 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.262 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:27:35.262 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105796+0000", 2024-04-20T17:27:35.262 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.959e-06 2024-04-20T17:27:35.262 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.262 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:27:35.262 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105799+0000", 2024-04-20T17:27:35.263 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7719999999999999e-06 2024-04-20T17:27:35.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.263 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.263 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105913+0000", 2024-04-20T17:27:35.263 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011384599999999999 2024-04-20T17:27:35.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.263 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:27:35.263 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105916+0000", 2024-04-20T17:27:35.263 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0110000000000001e-06 2024-04-20T17:27:35.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.263 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:27:35.263 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.175112+0000", 2024-04-20T17:27:35.264 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.069195997999999995 2024-04-20T17:27:35.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.264 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:27:35.264 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.175190+0000", 2024-04-20T17:27:35.264 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7616999999999999e-05 2024-04-20T17:27:35.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.264 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 11", 2024-04-20T17:27:35.264 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.175261+0000", 2024-04-20T17:27:35.264 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1484999999999997e-05 2024-04-20T17:27:35.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.264 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:27:35.264 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.177849+0000", 2024-04-20T17:27:35.264 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0025879589999999999 2024-04-20T17:27:35.265 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.265 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.265 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:35.265 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.196883+0000", 2024-04-20T17:27:35.265 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.019034253000000001 2024-04-20T17:27:35.265 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.265 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.265 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:27:35.265 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.196916+0000", 2024-04-20T17:27:35.265 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2586999999999998e-05 2024-04-20T17:27:35.265 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:27:35.265 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:27:35.265 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:27:35.265 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.196955+0000", 2024-04-20T17:27:35.265 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8801999999999999e-05 2024-04-20T17:27:35.265 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.266 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:27:35.266 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.266 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:27:35.266 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:27:35.266 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:27:35.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:35.267 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:35.267 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:27:35.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:27:35.276 INFO:teuthology.orchestra.run.smithi005.stderr:marked out osd.6. 2024-04-20T17:27:35.369 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:27:35.401 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:35.422 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:27:35.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:35.424 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:27:35.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:35 smithi106 ceph-mon[44423]: pgmap v1342: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 5.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 11546/184372 objects degraded (6.262%) 2024-04-20T17:27:35.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:35 smithi106 ceph-mon[44423]: osdmap e509: 12 total, 11 up, 12 in 2024-04-20T17:27:35.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:35 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1651192012' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-04-20T17:27:35.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:35 smithi106 ceph-mon[44423]: from='client.26299 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-04-20T17:27:35.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:35 smithi106 ceph-mon[44423]: Client client.admin marked osd.6 out, after it was down for 12 seconds 2024-04-20T17:27:35.525 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:27:35.537 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:35 smithi005 ceph-mon[39562]: pgmap v1342: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 5.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 11546/184372 objects degraded (6.262%) 2024-04-20T17:27:35.537 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:35 smithi005 ceph-mon[39562]: osdmap e509: 12 total, 11 up, 12 in 2024-04-20T17:27:35.537 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:35 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1651192012' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-04-20T17:27:35.537 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:35 smithi005 ceph-mon[39562]: from='client.26299 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-04-20T17:27:35.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:35 smithi005 ceph-mon[39562]: Client client.admin marked osd.6 out, after it was down for 12 seconds 2024-04-20T17:27:35.555 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:35 smithi171 ceph-mon[44189]: pgmap v1342: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 5.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 11546/184372 objects degraded (6.262%) 2024-04-20T17:27:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:35 smithi171 ceph-mon[44189]: osdmap e509: 12 total, 11 up, 12 in 2024-04-20T17:27:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:35 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1651192012' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-04-20T17:27:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:35 smithi171 ceph-mon[44189]: from='client.26299 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-04-20T17:27:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:35 smithi171 ceph-mon[44189]: Client client.admin marked osd.6 out, after it was down for 12 seconds 2024-04-20T17:27:35.593 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:27:35.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:35.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:27:35.697 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:27:35.725 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:35.822 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:27:35.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:35.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:27:35.894 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-20T17:27:35.926 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:27:35.958 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:35.980 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:27:35.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:35.983 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:27:36.084 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:27:36.114 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:36.134 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:27:36.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:36.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:27:36.197 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:27:36.237 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:27:36.269 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:36.290 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:27:36.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:36.293 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:27:36.394 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:27:36.425 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:36.446 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:27:36.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:36.448 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:27:36.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:36 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:27:36.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:36 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:27:36.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:36 smithi005 ceph-mon[39562]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T17:27:36.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:36 smithi005 ceph-mon[39562]: from='client.26299 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-04-20T17:27:36.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:36 smithi005 ceph-mon[39562]: osdmap e510: 12 total, 11 up, 11 in 2024-04-20T17:27:36.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:36 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1087886037' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:27:36.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:36 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:27:36.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:36 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1087886037' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:27:36.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:36 smithi005 ceph-mon[39562]: osdmap e511: 12 total, 11 up, 11 in 2024-04-20T17:27:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:36 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:27:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:36 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:27:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:36 smithi106 ceph-mon[44423]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T17:27:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:36 smithi106 ceph-mon[44423]: from='client.26299 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-04-20T17:27:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:36 smithi106 ceph-mon[44423]: osdmap e510: 12 total, 11 up, 11 in 2024-04-20T17:27:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:36 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1087886037' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:27:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:36 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:27:36.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:36 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1087886037' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:27:36.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:36 smithi106 ceph-mon[44423]: osdmap e511: 12 total, 11 up, 11 in 2024-04-20T17:27:36.549 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:27:36.575 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:36 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:27:36.576 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:36 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:27:36.576 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:36 smithi171 ceph-mon[44189]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T17:27:36.576 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:36 smithi171 ceph-mon[44189]: from='client.26299 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-04-20T17:27:36.576 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:36 smithi171 ceph-mon[44189]: osdmap e510: 12 total, 11 up, 11 in 2024-04-20T17:27:36.576 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:36 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1087886037' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:27:36.576 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:36 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:27:36.576 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:36 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1087886037' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:27:36.576 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:36 smithi171 ceph-mon[44189]: osdmap e511: 12 total, 11 up, 11 in 2024-04-20T17:27:36.577 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:36.597 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:27:36.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:36.600 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:27:36.701 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:27:36.730 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:36.749 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:27:36.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:36.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:27:36.853 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:27:36.881 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:37.191 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.6 2024-04-20T17:27:37.191 INFO:tasks.thrashosds.thrasher:remote for osd 6 is ubuntu@smithi106.front.sepia.ceph.com 2024-04-20T17:27:37.191 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.6 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op list-pgs 2024-04-20T17:27:37.402 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:27:37.413 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.6/config 2024-04-20T17:27:37.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:37 smithi005 ceph-mon[39562]: pgmap v1346: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 5.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 11546/184372 objects degraded (6.262%) 2024-04-20T17:27:37.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:37 smithi005 ceph-mon[39562]: osdmap e512: 12 total, 11 up, 11 in 2024-04-20T17:27:37.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:37 smithi106 ceph-mon[44423]: pgmap v1346: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 5.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 11546/184372 objects degraded (6.262%) 2024-04-20T17:27:37.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:37 smithi106 ceph-mon[44423]: osdmap e512: 12 total, 11 up, 11 in 2024-04-20T17:27:37.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:37 smithi171 ceph-mon[44189]: pgmap v1346: 17 pgs: 2 undersized+degraded+peered, 15 active+clean; 5.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 11546/184372 objects degraded (6.262%) 2024-04-20T17:27:37.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:37 smithi171 ceph-mon[44189]: osdmap e512: 12 total, 11 up, 11 in 2024-04-20T17:27:37.930 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:27:37.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:37.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:27:37.942 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:27:37.965 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:27:37.985 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:27:38.005 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:27:38.026 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:27:38.034 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:27:38.048 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:27:38.067 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:38.070 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:27:38.101 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:27:38.125 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:27:38.147 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:27:38.172 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:27:38.172 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:27:38.172 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:27:38.172 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:27:38.172 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:27:38.172 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:27:38.172 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:27:38.172 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-20T17:27:38.172 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:27:38.172 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:27:38.173 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:27:38.173 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:27:38.173 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:27:38.421 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:27:38.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:38.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:27:38.450 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:38 smithi005 ceph-mon[39562]: 4.a starting backfill to osd.10 from (0'0,0'0] MAX to 503'15307 2024-04-20T17:27:38.451 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:38 smithi005 ceph-mon[39562]: 4.5 starting backfill to osd.11 from (0'0,0'0] MAX to 503'15601 2024-04-20T17:27:38.451 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:38 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:27:38.451 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:38 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:27:38.451 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:38 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3850978679' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:27:38.451 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:38.528 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:27:38.558 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:38.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:38 smithi171 ceph-mon[44189]: 4.a starting backfill to osd.10 from (0'0,0'0] MAX to 503'15307 2024-04-20T17:27:38.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:38 smithi171 ceph-mon[44189]: 4.5 starting backfill to osd.11 from (0'0,0'0] MAX to 503'15601 2024-04-20T17:27:38.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:38 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:27:38.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:38 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:27:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:38 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3850978679' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:27:38.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:38.758 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:27:38.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:38 smithi106 ceph-mon[44423]: 4.a starting backfill to osd.10 from (0'0,0'0] MAX to 503'15307 2024-04-20T17:27:38.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:38 smithi106 ceph-mon[44423]: 4.5 starting backfill to osd.11 from (0'0,0'0] MAX to 503'15601 2024-04-20T17:27:38.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:38 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:27:38.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:38 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:27:38.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:38 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3850978679' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:27:38.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:38.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:38.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:27:38.833 INFO:teuthology.orchestra.run.smithi106.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:27:38.865 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:27:38.895 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:39.019 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:27:39.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:39.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:27:39.051 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:27:39.051 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:27:39.124 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:27:39.153 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:39.174 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:27:39.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:39.177 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:27:39.277 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:27:39.285 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.285 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.285 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.285 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.285 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.285 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.285 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.285 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.285 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.285 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.285 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:39 smithi005 ceph-mon[39562]: pgmap v1348: 17 pgs: 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 5.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 11520/184372 objects degraded (6.248%); 432 KiB/s, 6 objects/s recovering 2024-04-20T17:27:39.285 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:39 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 11520/184372 objects degraded (6.248%), 2 pgs degraded (PG_DEGRADED) 2024-04-20T17:27:39.285 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:39 smithi005 ceph-mon[39562]: osdmap e513: 12 total, 11 up, 11 in 2024-04-20T17:27:39.307 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:39.329 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:27:39.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:39.332 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:27:39.433 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:27:39.460 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.460 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.460 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.460 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.460 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.460 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.460 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.460 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.460 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.460 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.461 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:39 smithi106 ceph-mon[44423]: pgmap v1348: 17 pgs: 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 5.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 11520/184372 objects degraded (6.248%); 432 KiB/s, 6 objects/s recovering 2024-04-20T17:27:39.461 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:39 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 11520/184372 objects degraded (6.248%), 2 pgs degraded (PG_DEGRADED) 2024-04-20T17:27:39.461 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:39 smithi106 ceph-mon[44423]: osdmap e513: 12 total, 11 up, 11 in 2024-04-20T17:27:39.463 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:39.482 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:27:39.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:39.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:27:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:39.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:39 smithi171 ceph-mon[44189]: pgmap v1348: 17 pgs: 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 5.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 11520/184372 objects degraded (6.248%); 432 KiB/s, 6 objects/s recovering 2024-04-20T17:27:39.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:39 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 11520/184372 objects degraded (6.248%), 2 pgs degraded (PG_DEGRADED) 2024-04-20T17:27:39.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:39 smithi171 ceph-mon[44189]: osdmap e513: 12 total, 11 up, 11 in 2024-04-20T17:27:39.586 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:27:39.616 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:39.637 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:27:39.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:39.640 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:27:39.741 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:27:39.765 INFO:teuthology.orchestra.run.smithi106.stdout:4.5 2024-04-20T17:27:39.765 INFO:teuthology.orchestra.run.smithi106.stdout:4.a 2024-04-20T17:27:39.770 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:39.792 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:27:39.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:39.795 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:27:39.896 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:27:39.926 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:39.948 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:27:39.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:39.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:27:40.052 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:27:40.082 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:40.281 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:27:40.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:40.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:27:40.384 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:27:40.413 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:40.452 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:27:40.898 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.6 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op export --pgid 4.5 --file /var/log/ceph/exp.4.5.6 2024-04-20T17:27:40.961 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:27:40.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:40.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:27:41.052 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:27:41.065 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:27:41.118 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:27:41.124 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:27:41.124 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:27:41.132 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:41.142 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.6/config 2024-04-20T17:27:41.170 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:27:41.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:41.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:27:41.274 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:27:41.304 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:41.325 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:27:41.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:41.328 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:27:41.413 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:41 smithi005 ceph-mon[39562]: pgmap v1350: 17 pgs: 2 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 5.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 10506/184372 objects degraded (5.698%); 13 MiB/s, 209 objects/s recovering 2024-04-20T17:27:41.414 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:41 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:27:41.414 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:41 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:27:41.429 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:27:41.457 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:41.479 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:27:41.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:41.482 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:27:41.583 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:27:41.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:41 smithi171 ceph-mon[44189]: pgmap v1350: 17 pgs: 2 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 5.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 10506/184372 objects degraded (5.698%); 13 MiB/s, 209 objects/s recovering 2024-04-20T17:27:41.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:41 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:27:41.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:41 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:27:41.612 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:41.635 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:27:41.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:41.638 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:27:41.684 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:41 smithi106 ceph-mon[44423]: pgmap v1350: 17 pgs: 2 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 5.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 10506/184372 objects degraded (5.698%); 13 MiB/s, 209 objects/s recovering 2024-04-20T17:27:41.685 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:41 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:27:41.685 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:41 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:27:41.739 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:27:41.768 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:42.306 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-20T17:27:42.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:42 smithi106 ceph-mon[44423]: osdmap e514: 12 total, 11 up, 11 in 2024-04-20T17:27:42.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:42 smithi171 ceph-mon[44189]: osdmap e514: 12 total, 11 up, 11 in 2024-04-20T17:27:42.607 INFO:teuthology.orchestra.run.smithi106.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:27:42.617 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:27:42.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:42.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:27:42.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:42 smithi005 ceph-mon[39562]: osdmap e514: 12 total, 11 up, 11 in 2024-04-20T17:27:42.721 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:27:42.749 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:42.787 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:27:42.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:42.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:27:42.895 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:27:42.925 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:42.944 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:27:42.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:42.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:27:43.048 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:27:43.078 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:43.100 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:27:43.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:43.104 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:27:43.131 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:27:43.166 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:27:43.205 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:27:43.232 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:43.251 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:27:43.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:43.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:27:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:43 smithi171 ceph-mon[44189]: pgmap v1352: 17 pgs: 2 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 5.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 10506/184372 objects degraded (5.698%); 11 MiB/s, 173 objects/s recovering 2024-04-20T17:27:43.354 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:27:43.382 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:43 smithi005 ceph-mon[39562]: pgmap v1352: 17 pgs: 2 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 5.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 10506/184372 objects degraded (5.698%); 11 MiB/s, 173 objects/s recovering 2024-04-20T17:27:43.383 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:43.404 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:27:43.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:43.407 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:27:43.495 INFO:teuthology.orchestra.run.smithi106.stderr:Exporting 4.5 info 4.5( v 503'15601 (462'5600,503'15601] local-lis/les=445/446 n=5733 ec=445/445 lis/c=445/445 les/c/f=446/446/0 sis=445) 2024-04-20T17:27:43.508 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:27:43.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:43 smithi106 ceph-mon[44423]: pgmap v1352: 17 pgs: 2 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 5.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 10506/184372 objects degraded (5.698%); 11 MiB/s, 173 objects/s recovering 2024-04-20T17:27:43.537 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:43.541 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a000538d:::benchmark_data_smithi171_122772_object107954:head# 2024-04-20T17:27:43.541 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0027801:::benchmark_data_smithi171_122772_object127544:head# 2024-04-20T17:27:43.541 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a002d6fc:::benchmark_data_smithi171_122772_object168070:head# 2024-04-20T17:27:43.542 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0035878:::benchmark_data_smithi171_122772_object126831:head# 2024-04-20T17:27:43.542 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0046a9d:::benchmark_data_smithi171_122772_object120482:head# 2024-04-20T17:27:43.543 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0046e3b:::benchmark_data_smithi171_122772_object150652:head# 2024-04-20T17:27:43.543 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a004b228:::benchmark_data_smithi171_122772_object122765:head# 2024-04-20T17:27:43.543 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a00592c1:::benchmark_data_smithi171_122772_object150774:head# 2024-04-20T17:27:43.544 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0075b00:::benchmark_data_smithi171_122772_object131539:head# 2024-04-20T17:27:43.545 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a007c405:::benchmark_data_smithi171_122772_object98194:head# 2024-04-20T17:27:43.545 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a008e900:::benchmark_data_smithi171_122772_object120791:head# 2024-04-20T17:27:43.546 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0097346:::benchmark_data_smithi171_122772_object84061:head# 2024-04-20T17:27:43.546 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a00a03fc:::benchmark_data_smithi171_122772_object105620:head# 2024-04-20T17:27:43.546 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a00b37a7:::benchmark_data_smithi171_122772_object126331:head# 2024-04-20T17:27:43.547 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a00bb3cf:::benchmark_data_smithi171_122772_object112003:head# 2024-04-20T17:27:43.547 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a00c9ef8:::benchmark_data_smithi171_122772_object148592:head# 2024-04-20T17:27:43.547 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a00e17b4:::benchmark_data_smithi171_122772_object134388:head# 2024-04-20T17:27:43.548 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a00e17f4:::benchmark_data_smithi171_122772_object138284:head# 2024-04-20T17:27:43.548 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a00f8578:::benchmark_data_smithi171_122772_object92270:head# 2024-04-20T17:27:43.549 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a00fd006:::benchmark_data_smithi171_122772_object146275:head# 2024-04-20T17:27:43.549 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a00fe5e3:::benchmark_data_smithi171_122772_object148508:head# 2024-04-20T17:27:43.550 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a011ef33:::benchmark_data_smithi171_122772_object135649:head# 2024-04-20T17:27:43.551 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a01202f5:::benchmark_data_smithi171_122772_object137599:head# 2024-04-20T17:27:43.551 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0131585:::benchmark_data_smithi171_122772_object105119:head# 2024-04-20T17:27:43.551 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a014d493:::benchmark_data_smithi171_122772_object126718:head# 2024-04-20T17:27:43.552 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a01522ff:::benchmark_data_smithi171_122772_object109167:head# 2024-04-20T17:27:43.552 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a015bde0:::benchmark_data_smithi171_122772_object86545:head# 2024-04-20T17:27:43.553 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a017321b:::benchmark_data_smithi171_122772_object104394:head# 2024-04-20T17:27:43.553 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0183041:::benchmark_data_smithi171_122772_object155021:head# 2024-04-20T17:27:43.554 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a018f509:::benchmark_data_smithi171_122772_object104542:head# 2024-04-20T17:27:43.554 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a019432c:::benchmark_data_smithi171_122772_object102066:head# 2024-04-20T17:27:43.554 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a01a94c3:::benchmark_data_smithi171_122772_object166565:head# 2024-04-20T17:27:43.555 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a01b9f09:::benchmark_data_smithi171_122772_object130679:head# 2024-04-20T17:27:43.555 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a01c5c4f:::benchmark_data_smithi171_122772_object162286:head# 2024-04-20T17:27:43.556 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0201eec:::benchmark_data_smithi171_122772_object111745:head# 2024-04-20T17:27:43.556 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0201f89:::benchmark_data_smithi171_122772_object169020:head# 2024-04-20T17:27:43.557 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0205a32:::benchmark_data_smithi171_122772_object140691:head# 2024-04-20T17:27:43.557 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a02069a8:::benchmark_data_smithi171_122772_object158408:head# 2024-04-20T17:27:43.558 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a020c239:::benchmark_data_smithi171_122772_object131170:head# 2024-04-20T17:27:43.558 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0213451:::benchmark_data_smithi171_122772_object83669:head# 2024-04-20T17:27:43.558 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a021aebd:::benchmark_data_smithi171_122772_object110055:head# 2024-04-20T17:27:43.559 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:27:43.560 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0227bbc:::benchmark_data_smithi171_122772_object123533:head# 2024-04-20T17:27:43.560 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0256cbe:::benchmark_data_smithi171_122772_object103878:head# 2024-04-20T17:27:43.560 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0258c61:::benchmark_data_smithi171_122772_object84914:head# 2024-04-20T17:27:43.560 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a02759d8:::benchmark_data_smithi171_122772_object156278:head# 2024-04-20T17:27:43.560 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a02898db:::benchmark_data_smithi171_122772_object153016:head# 2024-04-20T17:27:43.561 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a028bf9d:::benchmark_data_smithi171_122772_object121166:head# 2024-04-20T17:27:43.561 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a02a764b:::benchmark_data_smithi171_122772_object104155:head# 2024-04-20T17:27:43.562 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a02b45b4:::benchmark_data_smithi171_122772_object80948:head# 2024-04-20T17:27:43.562 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a02b9b7c:::benchmark_data_smithi171_122772_object128443:head# 2024-04-20T17:27:43.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:43.563 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:27:43.563 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a02c5cee:::benchmark_data_smithi171_122772_object146669:head# 2024-04-20T17:27:43.563 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a02c7ef5:::benchmark_data_smithi171_122772_object159443:head# 2024-04-20T17:27:43.564 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a02c81e1:::benchmark_data_smithi171_122772_object157547:head# 2024-04-20T17:27:43.564 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a02cb5be:::benchmark_data_smithi171_122772_object164341:head# 2024-04-20T17:27:43.564 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a02e7a4c:::benchmark_data_smithi171_122772_object95590:head# 2024-04-20T17:27:43.565 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a02ea583:::benchmark_data_smithi171_122772_object88041:head# 2024-04-20T17:27:43.565 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a02eb62b:::benchmark_data_smithi171_122772_object86829:head# 2024-04-20T17:27:43.566 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a02fc404:::benchmark_data_smithi171_122772_object167535:head# 2024-04-20T17:27:43.566 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a030237b:::benchmark_data_smithi171_122772_object83925:head# 2024-04-20T17:27:43.566 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a030f91d:::benchmark_data_smithi171_122772_object165575:head# 2024-04-20T17:27:43.567 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0312b83:::benchmark_data_smithi171_122772_object86639:head# 2024-04-20T17:27:43.567 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a03158c8:::benchmark_data_smithi171_122772_object157088:head# 2024-04-20T17:27:43.568 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a031f6e6:::benchmark_data_smithi171_122772_object137876:head# 2024-04-20T17:27:43.568 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0329d3e:::benchmark_data_smithi171_122772_object120884:head# 2024-04-20T17:27:43.568 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a032da83:::benchmark_data_smithi171_122772_object150855:head# 2024-04-20T17:27:43.569 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0340a13:::benchmark_data_smithi171_122772_object110989:head# 2024-04-20T17:27:43.569 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a034f0d3:::benchmark_data_smithi171_122772_object84100:head# 2024-04-20T17:27:43.570 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0353cdd:::benchmark_data_smithi171_122772_object129989:head# 2024-04-20T17:27:43.571 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0368f4a:::benchmark_data_smithi171_122772_object161489:head# 2024-04-20T17:27:43.571 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a036b2fb:::benchmark_data_smithi171_122772_object159566:head# 2024-04-20T17:27:43.572 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a037a940:::benchmark_data_smithi171_122772_object125154:head# 2024-04-20T17:27:43.572 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a037e61a:::benchmark_data_smithi171_122772_object94998:head# 2024-04-20T17:27:43.572 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0385c85:::benchmark_data_smithi171_122772_object94382:head# 2024-04-20T17:27:43.573 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a038d700:::benchmark_data_smithi171_122772_object169540:head# 2024-04-20T17:27:43.573 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a039542f:::benchmark_data_smithi171_122772_object104673:head# 2024-04-20T17:27:43.573 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a03ab17e:::benchmark_data_smithi171_122772_object137729:head# 2024-04-20T17:27:43.574 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a03b2c89:::benchmark_data_smithi171_122772_object110923:head# 2024-04-20T17:27:43.574 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a03b85f6:::benchmark_data_smithi171_122772_object112668:head# 2024-04-20T17:27:43.575 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a03bd528:::benchmark_data_smithi171_122772_object91712:head# 2024-04-20T17:27:43.575 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a03c4401:::benchmark_data_smithi171_122772_object153159:head# 2024-04-20T17:27:43.575 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a03d39f3:::benchmark_data_smithi171_122772_object111897:head# 2024-04-20T17:27:43.576 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a03dcb59:::benchmark_data_smithi171_122772_object78378:head# 2024-04-20T17:27:43.576 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a03e6a7f:::benchmark_data_smithi171_122772_object152887:head# 2024-04-20T17:27:43.577 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a03ee0b7:::benchmark_data_smithi171_122772_object147040:head# 2024-04-20T17:27:43.577 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a03f3b90:::benchmark_data_smithi171_122772_object121173:head# 2024-04-20T17:27:43.578 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a03fa48b:::benchmark_data_smithi171_122772_object111559:head# 2024-04-20T17:27:43.578 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a03fad21:::benchmark_data_smithi171_122772_object101261:head# 2024-04-20T17:27:43.578 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0406e49:::benchmark_data_smithi171_122772_object163440:head# 2024-04-20T17:27:43.579 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a040f3c8:::benchmark_data_smithi171_122772_object106981:head# 2024-04-20T17:27:43.579 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a041600b:::benchmark_data_smithi171_122772_object161603:head# 2024-04-20T17:27:43.580 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a041e501:::benchmark_data_smithi171_122772_object153785:head# 2024-04-20T17:27:43.580 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0421f5c:::benchmark_data_smithi171_122772_object125517:head# 2024-04-20T17:27:43.581 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a044f5d5:::benchmark_data_smithi171_122772_object121132:head# 2024-04-20T17:27:43.581 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0458c8c:::benchmark_data_smithi171_122772_object155444:head# 2024-04-20T17:27:43.582 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a04659b4:::benchmark_data_smithi171_122772_object90726:head# 2024-04-20T17:27:43.582 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a046d372:::benchmark_data_smithi171_122772_object99519:head# 2024-04-20T17:27:43.583 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a047c394:::benchmark_data_smithi171_122772_object110175:head# 2024-04-20T17:27:43.583 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a04e192c:::benchmark_data_smithi171_122772_object112549:head# 2024-04-20T17:27:43.584 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a04f5fbe:::benchmark_data_smithi171_122772_object121780:head# 2024-04-20T17:27:43.584 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a04f8857:::benchmark_data_smithi171_122772_object148970:head# 2024-04-20T17:27:43.585 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0517f07:::benchmark_data_smithi171_122772_object100226:head# 2024-04-20T17:27:43.585 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0519240:::benchmark_data_smithi171_122772_object154757:head# 2024-04-20T17:27:43.586 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a051e05b:::benchmark_data_smithi171_122772_object164401:head# 2024-04-20T17:27:43.586 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0525716:::benchmark_data_smithi171_122772_object78800:head# 2024-04-20T17:27:43.587 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0542278:::benchmark_data_smithi171_122772_object99751:head# 2024-04-20T17:27:43.587 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0546dec:::benchmark_data_smithi171_122772_object100801:head# 2024-04-20T17:27:43.588 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a055fd11:::benchmark_data_smithi171_122772_object158413:head# 2024-04-20T17:27:43.588 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a056102f:::benchmark_data_smithi171_122772_object135487:head# 2024-04-20T17:27:43.589 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a056b1b6:::benchmark_data_smithi171_122772_object117591:head# 2024-04-20T17:27:43.589 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a056c578:::benchmark_data_smithi171_122772_object113439:head# 2024-04-20T17:27:43.590 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a058127d:::benchmark_data_smithi171_122772_object134612:head# 2024-04-20T17:27:43.590 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a059c372:::benchmark_data_smithi171_122772_object98716:head# 2024-04-20T17:27:43.590 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a059d6ae:::benchmark_data_smithi171_122772_object108487:head# 2024-04-20T17:27:43.591 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a05a05f2:::benchmark_data_smithi171_122772_object169794:head# 2024-04-20T17:27:43.591 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a05a9ad7:::benchmark_data_smithi171_122772_object113511:head# 2024-04-20T17:27:43.592 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a05aefac:::benchmark_data_smithi171_122772_object116250:head# 2024-04-20T17:27:43.592 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a05f02d1:::benchmark_data_smithi171_122772_object104851:head# 2024-04-20T17:27:43.593 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a05f0a49:::benchmark_data_smithi171_122772_object106211:head# 2024-04-20T17:27:43.593 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a05f6a62:::benchmark_data_smithi171_122772_object156136:head# 2024-04-20T17:27:43.594 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a060345a:::benchmark_data_smithi171_122772_object161821:head# 2024-04-20T17:27:43.594 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a060d70e:::benchmark_data_smithi171_122772_object168669:head# 2024-04-20T17:27:43.595 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a06304ad:::benchmark_data_smithi171_122772_object140396:head# 2024-04-20T17:27:43.595 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a063ea95:::benchmark_data_smithi171_122772_object91790:head# 2024-04-20T17:27:43.596 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a063f524:::benchmark_data_smithi171_122772_object167014:head# 2024-04-20T17:27:43.597 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a064ea80:::benchmark_data_smithi171_122772_object166896:head# 2024-04-20T17:27:43.597 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a065af90:::benchmark_data_smithi171_122772_object113318:head# 2024-04-20T17:27:43.598 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a066fd10:::benchmark_data_smithi171_122772_object143060:head# 2024-04-20T17:27:43.598 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a06a7f92:::benchmark_data_smithi171_122772_object136146:head# 2024-04-20T17:27:43.599 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a06b275b:::benchmark_data_smithi171_122772_object105099:head# 2024-04-20T17:27:43.599 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a06ca089:::benchmark_data_smithi171_122772_object169551:head# 2024-04-20T17:27:43.600 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a06ce330:::benchmark_data_smithi171_122772_object147827:head# 2024-04-20T17:27:43.600 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a06d18b4:::benchmark_data_smithi171_122772_object119654:head# 2024-04-20T17:27:43.600 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a06d82aa:::benchmark_data_smithi171_122772_object167093:head# 2024-04-20T17:27:43.601 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a06d9cd8:::benchmark_data_smithi171_122772_object139711:head# 2024-04-20T17:27:43.601 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a06ddd3a:::benchmark_data_smithi171_122772_object101204:head# 2024-04-20T17:27:43.602 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0708d06:::benchmark_data_smithi171_122772_object116584:head# 2024-04-20T17:27:43.603 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a071179d:::benchmark_data_smithi171_122772_object139235:head# 2024-04-20T17:27:43.603 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a07147a9:::benchmark_data_smithi171_122772_object85173:head# 2024-04-20T17:27:43.604 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0741e3c:::benchmark_data_smithi171_122772_object105716:head# 2024-04-20T17:27:43.604 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a074ccbc:::benchmark_data_smithi171_122772_object82151:head# 2024-04-20T17:27:43.605 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0752fa4:::benchmark_data_smithi171_122772_object150217:head# 2024-04-20T17:27:43.605 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a076538d:::benchmark_data_smithi171_122772_object105308:head# 2024-04-20T17:27:43.606 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0768bcf:::benchmark_data_smithi171_122772_object113047:head# 2024-04-20T17:27:43.606 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a076f600:::benchmark_data_smithi171_122772_object130269:head# 2024-04-20T17:27:43.607 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a077a3b2:::benchmark_data_smithi171_122772_object111259:head# 2024-04-20T17:27:43.607 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a077fe48:::benchmark_data_smithi171_122772_object132882:head# 2024-04-20T17:27:43.608 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0780753:::benchmark_data_smithi171_122772_object122760:head# 2024-04-20T17:27:43.608 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a078e99d:::benchmark_data_smithi171_122772_object154976:head# 2024-04-20T17:27:43.609 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0790bc3:::benchmark_data_smithi171_122772_object109292:head# 2024-04-20T17:27:43.609 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a07a14ae:::benchmark_data_smithi171_122772_object79491:head# 2024-04-20T17:27:43.610 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a07a2b82:::benchmark_data_smithi171_122772_object98762:head# 2024-04-20T17:27:43.610 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a07b1545:::benchmark_data_smithi171_122772_object96015:head# 2024-04-20T17:27:43.610 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a07c023a:::benchmark_data_smithi171_122772_object82343:head# 2024-04-20T17:27:43.611 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a07c81c6:::benchmark_data_smithi171_122772_object158436:head# 2024-04-20T17:27:43.612 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a07d1f53:::benchmark_data_smithi171_122772_object134980:head# 2024-04-20T17:27:43.612 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a07d2b5b:::benchmark_data_smithi171_122772_object111738:head# 2024-04-20T17:27:43.613 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a07d4505:::benchmark_data_smithi171_122772_object106430:head# 2024-04-20T17:27:43.613 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a07dbb51:::benchmark_data_smithi171_122772_object92765:head# 2024-04-20T17:27:43.614 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a07e2738:::benchmark_data_smithi171_122772_object169164:head# 2024-04-20T17:27:43.615 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a07e91c5:::benchmark_data_smithi171_122772_object88494:head# 2024-04-20T17:27:43.615 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a07f2008:::benchmark_data_smithi171_122772_object159522:head# 2024-04-20T17:27:43.615 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a07f9cf2:::benchmark_data_smithi171_122772_object140211:head# 2024-04-20T17:27:43.616 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a07ff679:::benchmark_data_smithi171_122772_object144519:head# 2024-04-20T17:27:43.616 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a080362a:::benchmark_data_smithi171_122772_object105945:head# 2024-04-20T17:27:43.617 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0803bd1:::benchmark_data_smithi171_122772_object106992:head# 2024-04-20T17:27:43.617 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0815d78:::benchmark_data_smithi171_122772_object112318:head# 2024-04-20T17:27:43.618 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a08182d5:::benchmark_data_smithi171_122772_object147052:head# 2024-04-20T17:27:43.618 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a082632b:::benchmark_data_smithi171_122772_object100957:head# 2024-04-20T17:27:43.619 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a084c1c4:::benchmark_data_smithi171_122772_object143305:head# 2024-04-20T17:27:43.619 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a084fa1d:::benchmark_data_smithi171_122772_object90177:head# 2024-04-20T17:27:43.620 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a08508bf:::benchmark_data_smithi171_122772_object125536:head# 2024-04-20T17:27:43.620 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a086122f:::benchmark_data_smithi171_122772_object122140:head# 2024-04-20T17:27:43.621 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a087fb64:::benchmark_data_smithi171_122772_object137453:head# 2024-04-20T17:27:43.621 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a08845a0:::benchmark_data_smithi171_122772_object110879:head# 2024-04-20T17:27:43.622 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a088ac06:::benchmark_data_smithi171_122772_object125834:head# 2024-04-20T17:27:43.622 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0898ad7:::benchmark_data_smithi171_122772_object136540:head# 2024-04-20T17:27:43.623 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a08a5aa4:::benchmark_data_smithi171_122772_object113784:head# 2024-04-20T17:27:43.623 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a08a89b5:::benchmark_data_smithi171_122772_object78548:head# 2024-04-20T17:27:43.624 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a08aa56b:::benchmark_data_smithi171_122772_object85563:head# 2024-04-20T17:27:43.624 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a08b7846:::benchmark_data_smithi171_122772_object128112:head# 2024-04-20T17:27:43.625 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a08ba3fe:::benchmark_data_smithi171_122772_object103136:head# 2024-04-20T17:27:43.625 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a08be249:::benchmark_data_smithi171_122772_object133803:head# 2024-04-20T17:27:43.626 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a08c786f:::benchmark_data_smithi171_122772_object99934:head# 2024-04-20T17:27:43.626 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a08cad7a:::benchmark_data_smithi171_122772_object99549:head# 2024-04-20T17:27:43.627 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a08d279d:::benchmark_data_smithi171_122772_object104084:head# 2024-04-20T17:27:43.627 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a08d29b0:::benchmark_data_smithi171_122772_object127248:head# 2024-04-20T17:27:43.628 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a08dd588:::benchmark_data_smithi171_122772_object145850:head# 2024-04-20T17:27:43.628 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a08e168d:::benchmark_data_smithi171_122772_object144597:head# 2024-04-20T17:27:43.629 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a08ed38b:::benchmark_data_smithi171_122772_object107063:head# 2024-04-20T17:27:43.629 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0906c31:::benchmark_data_smithi171_122772_object156289:head# 2024-04-20T17:27:43.630 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a090bb93:::benchmark_data_smithi171_122772_object90792:head# 2024-04-20T17:27:43.630 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a09183cc:::benchmark_data_smithi171_122772_object96638:head# 2024-04-20T17:27:43.631 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0918995:::benchmark_data_smithi171_122772_object101399:head# 2024-04-20T17:27:43.631 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a091b14f:::benchmark_data_smithi171_122772_object117164:head# 2024-04-20T17:27:43.632 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a091cb3a:::benchmark_data_smithi171_122772_object168673:head# 2024-04-20T17:27:43.632 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0923b1c:::benchmark_data_smithi171_122772_object131571:head# 2024-04-20T17:27:43.633 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a092abec:::benchmark_data_smithi171_122772_object137701:head# 2024-04-20T17:27:43.633 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0934836:::benchmark_data_smithi171_122772_object147486:head# 2024-04-20T17:27:43.633 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a093bd7b:::benchmark_data_smithi171_122772_object100910:head# 2024-04-20T17:27:43.634 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0942374:::benchmark_data_smithi171_122772_object137233:head# 2024-04-20T17:27:43.634 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a094cc20:::benchmark_data_smithi171_122772_object93216:head# 2024-04-20T17:27:43.635 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0965a2d:::benchmark_data_smithi171_122772_object158677:head# 2024-04-20T17:27:43.636 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a096ddfd:::benchmark_data_smithi171_122772_object94721:head# 2024-04-20T17:27:43.637 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a096eb6a:::benchmark_data_smithi171_122772_object155418:head# 2024-04-20T17:27:43.637 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a096f734:::benchmark_data_smithi171_122772_object140075:head# 2024-04-20T17:27:43.638 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0971344:::benchmark_data_smithi171_122772_object142855:head# 2024-04-20T17:27:43.638 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0974b5d:::benchmark_data_smithi171_122772_object108332:head# 2024-04-20T17:27:43.638 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0990904:::benchmark_data_smithi171_122772_object165982:head# 2024-04-20T17:27:43.639 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a09962aa:::benchmark_data_smithi171_122772_object87905:head# 2024-04-20T17:27:43.639 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a09a7fad:::benchmark_data_smithi171_122772_object119656:head# 2024-04-20T17:27:43.640 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a09a9201:::benchmark_data_smithi171_122772_object123802:head# 2024-04-20T17:27:43.640 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a09ae8cf:::benchmark_data_smithi171_122772_object96818:head# 2024-04-20T17:27:43.641 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a09b2da9:::benchmark_data_smithi171_122772_object111906:head# 2024-04-20T17:27:43.641 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a09b3056:::benchmark_data_smithi171_122772_object133322:head# 2024-04-20T17:27:43.642 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a09d0cca:::benchmark_data_smithi171_122772_object167579:head# 2024-04-20T17:27:43.642 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a09d1488:::benchmark_data_smithi171_122772_object154945:head# 2024-04-20T17:27:43.643 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a09d45f7:::benchmark_data_smithi171_122772_object111949:head# 2024-04-20T17:27:43.643 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a09daa4a:::benchmark_data_smithi171_122772_object87172:head# 2024-04-20T17:27:43.644 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a09ee5dd:::benchmark_data_smithi171_122772_object107405:head# 2024-04-20T17:27:43.644 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a09ee880:::benchmark_data_smithi171_122772_object88771:head# 2024-04-20T17:27:43.645 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a09f0d67:::benchmark_data_smithi171_122772_object140582:head# 2024-04-20T17:27:43.645 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a09f518b:::benchmark_data_smithi171_122772_object108803:head# 2024-04-20T17:27:43.646 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0a1e84e:::benchmark_data_smithi171_122772_object123895:head# 2024-04-20T17:27:43.646 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0a2e9e7:::benchmark_data_smithi171_122772_object135719:head# 2024-04-20T17:27:43.647 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0a2f3f0:::benchmark_data_smithi171_122772_object136776:head# 2024-04-20T17:27:43.647 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0a3f7d0:::benchmark_data_smithi171_122772_object82047:head# 2024-04-20T17:27:43.648 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0a4734b:::benchmark_data_smithi171_122772_object129601:head# 2024-04-20T17:27:43.648 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0a4a389:::benchmark_data_smithi171_122772_object167366:head# 2024-04-20T17:27:43.649 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0a538b6:::benchmark_data_smithi171_122772_object121545:head# 2024-04-20T17:27:43.649 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0a5eb3e:::benchmark_data_smithi171_122772_object148177:head# 2024-04-20T17:27:43.650 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0a64161:::benchmark_data_smithi171_122772_object94952:head# 2024-04-20T17:27:43.650 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0a6529c:::benchmark_data_smithi171_122772_object78595:head# 2024-04-20T17:27:43.651 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0a7a68b:::benchmark_data_smithi171_122772_object92246:head# 2024-04-20T17:27:43.652 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0a7c757:::benchmark_data_smithi171_122772_object160821:head# 2024-04-20T17:27:43.652 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0a84463:::benchmark_data_smithi171_122772_object149513:head# 2024-04-20T17:27:43.653 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0a88177:::benchmark_data_smithi171_122772_object104695:head# 2024-04-20T17:27:43.654 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0a9e36b:::benchmark_data_smithi171_122772_object86233:head# 2024-04-20T17:27:43.655 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0ac0375:::benchmark_data_smithi171_122772_object114209:head# 2024-04-20T17:27:43.655 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0ac1766:::benchmark_data_smithi171_122772_object80237:head# 2024-04-20T17:27:43.655 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0ac3143:::benchmark_data_smithi171_122772_object100354:head# 2024-04-20T17:27:43.655 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0acbf9d:::benchmark_data_smithi171_122772_object81178:head# 2024-04-20T17:27:43.656 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0ad40f3:::benchmark_data_smithi171_122772_object137515:head# 2024-04-20T17:27:43.656 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0ae6e16:::benchmark_data_smithi171_122772_object134844:head# 2024-04-20T17:27:43.657 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0af4066:::benchmark_data_smithi171_122772_object162516:head# 2024-04-20T17:27:43.657 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0af485f:::benchmark_data_smithi171_122772_object165155:head# 2024-04-20T17:27:43.658 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0b042a2:::benchmark_data_smithi171_122772_object137892:head# 2024-04-20T17:27:43.658 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0b0b508:::benchmark_data_smithi171_122772_object131548:head# 2024-04-20T17:27:43.659 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0b235ff:::benchmark_data_smithi171_122772_object120618:head# 2024-04-20T17:27:43.659 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0b24d60:::benchmark_data_smithi171_122772_object161088:head# 2024-04-20T17:27:43.660 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0b34a07:::benchmark_data_smithi171_122772_object142429:head# 2024-04-20T17:27:43.660 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0b41dc9:::benchmark_data_smithi171_122772_object139825:head# 2024-04-20T17:27:43.661 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0b4fc00:::benchmark_data_smithi171_122772_object126948:head# 2024-04-20T17:27:43.661 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0b7e62b:::benchmark_data_smithi171_122772_object106404:head# 2024-04-20T17:27:43.662 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0b89c03:::benchmark_data_smithi171_122772_object91684:head# 2024-04-20T17:27:43.662 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0baed10:::benchmark_data_smithi171_122772_object166292:head# 2024-04-20T17:27:43.663 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0bb6298:::benchmark_data_smithi171_122772_object93799:head# 2024-04-20T17:27:43.663 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:27:43.664 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0bb7379:::benchmark_data_smithi171_122772_object149311:head# 2024-04-20T17:27:43.665 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0bc86b3:::benchmark_data_smithi171_122772_object126072:head# 2024-04-20T17:27:43.665 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0bd092e:::benchmark_data_smithi171_122772_object155800:head# 2024-04-20T17:27:43.665 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0bd48aa:::benchmark_data_smithi171_122772_object88031:head# 2024-04-20T17:27:43.666 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0be77eb:::benchmark_data_smithi171_122772_object129373:head# 2024-04-20T17:27:43.667 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0be7e2a:::benchmark_data_smithi171_122772_object144858:head# 2024-04-20T17:27:43.667 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0bed093:::benchmark_data_smithi171_122772_object111064:head# 2024-04-20T17:27:43.667 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0befab1:::benchmark_data_smithi171_122772_object145886:head# 2024-04-20T17:27:43.668 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0bfca58:::benchmark_data_smithi171_122772_object135638:head# 2024-04-20T17:27:43.668 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0bfd736:::benchmark_data_smithi171_122772_object163702:head# 2024-04-20T17:27:43.668 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0c19d4a:::benchmark_data_smithi171_122772_object141898:head# 2024-04-20T17:27:43.669 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0c1afb9:::benchmark_data_smithi171_122772_object119499:head# 2024-04-20T17:27:43.669 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0c33021:::benchmark_data_smithi171_122772_object152093:head# 2024-04-20T17:27:43.670 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0c42557:::benchmark_data_smithi171_122772_object88480:head# 2024-04-20T17:27:43.670 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0c4ef1b:::benchmark_data_smithi171_122772_object102867:head# 2024-04-20T17:27:43.671 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0c514b0:::benchmark_data_smithi171_122772_object79059:head# 2024-04-20T17:27:43.671 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0c5395e:::benchmark_data_smithi171_122772_object163576:head# 2024-04-20T17:27:43.672 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0c55e97:::benchmark_data_smithi171_122772_object102839:head# 2024-04-20T17:27:43.672 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0c57e1f:::benchmark_data_smithi171_122772_object80850:head# 2024-04-20T17:27:43.673 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0c5b0b8:::benchmark_data_smithi171_122772_object160321:head# 2024-04-20T17:27:43.673 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0c6894e:::benchmark_data_smithi171_122772_object87989:head# 2024-04-20T17:27:43.674 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0c690e4:::benchmark_data_smithi171_122772_object113553:head# 2024-04-20T17:27:43.674 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0c7fe58:::benchmark_data_smithi171_122772_object132124:head# 2024-04-20T17:27:43.675 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0c9b787:::benchmark_data_smithi171_122772_object95068:head# 2024-04-20T17:27:43.675 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0cb0ff8:::benchmark_data_smithi171_122772_object94399:head# 2024-04-20T17:27:43.676 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0cb61fa:::benchmark_data_smithi171_122772_object110693:head# 2024-04-20T17:27:43.676 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0cbf7de:::benchmark_data_smithi171_122772_object155402:head# 2024-04-20T17:27:43.677 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0cccd6c:::benchmark_data_smithi171_122772_object154239:head# 2024-04-20T17:27:43.677 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0cd231c:::benchmark_data_smithi171_122772_object95344:head# 2024-04-20T17:27:43.678 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0cd9bfb:::benchmark_data_smithi171_122772_object160710:head# 2024-04-20T17:27:43.678 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0cdcd6a:::benchmark_data_smithi171_122772_object138623:head# 2024-04-20T17:27:43.681 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0cdfe3f:::benchmark_data_smithi171_122772_object118873:head# 2024-04-20T17:27:43.682 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0cf8d26:::benchmark_data_smithi171_122772_object144887:head# 2024-04-20T17:27:43.683 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0d0d405:::benchmark_data_smithi171_122772_object98030:head# 2024-04-20T17:27:43.683 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0d2ab5d:::benchmark_data_smithi171_122772_object113071:head# 2024-04-20T17:27:43.684 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0d2afec:::benchmark_data_smithi171_122772_object106663:head# 2024-04-20T17:27:43.684 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0d37323:::benchmark_data_smithi171_122772_object151791:head# 2024-04-20T17:27:43.684 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0d47f2c:::benchmark_data_smithi171_122772_object93438:head# 2024-04-20T17:27:43.685 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0d486a4:::benchmark_data_smithi171_122772_object113584:head# 2024-04-20T17:27:43.685 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0d54b09:::benchmark_data_smithi171_122772_object119987:head# 2024-04-20T17:27:43.686 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0d5bde2:::benchmark_data_smithi171_122772_object163202:head# 2024-04-20T17:27:43.686 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0d75b7f:::benchmark_data_smithi171_122772_object79302:head# 2024-04-20T17:27:43.687 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0d8bf22:::benchmark_data_smithi171_122772_object97053:head# 2024-04-20T17:27:43.688 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0d8c24a:::benchmark_data_smithi171_122772_object106689:head# 2024-04-20T17:27:43.688 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0d8ca03:::benchmark_data_smithi171_122772_object114987:head# 2024-04-20T17:27:43.689 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0d958fa:::benchmark_data_smithi171_122772_object91190:head# 2024-04-20T17:27:43.689 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0da3afa:::benchmark_data_smithi171_122772_object86919:head# 2024-04-20T17:27:43.689 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0daea6a:::benchmark_data_smithi171_122772_object85643:head# 2024-04-20T17:27:43.690 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0dc219c:::benchmark_data_smithi171_122772_object152660:head# 2024-04-20T17:27:43.690 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0dca567:::benchmark_data_smithi171_122772_object128370:head# 2024-04-20T17:27:43.691 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0dcf576:::benchmark_data_smithi171_122772_object121746:head# 2024-04-20T17:27:43.691 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0ddb2a5:::benchmark_data_smithi171_122772_object145116:head# 2024-04-20T17:27:43.692 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0dfcd3f:::benchmark_data_smithi171_122772_object123952:head# 2024-04-20T17:27:43.692 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0e0089f:::benchmark_data_smithi171_122772_object114885:head# 2024-04-20T17:27:43.693 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:43.693 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0e0b76f:::benchmark_data_smithi171_122772_object91680:head# 2024-04-20T17:27:43.694 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0e1c346:::benchmark_data_smithi171_122772_object80121:head# 2024-04-20T17:27:43.694 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0e25884:::benchmark_data_smithi171_122772_object120619:head# 2024-04-20T17:27:43.695 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0e263a7:::benchmark_data_smithi171_122772_object131683:head# 2024-04-20T17:27:43.695 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0e28073:::benchmark_data_smithi171_122772_object129590:head# 2024-04-20T17:27:43.695 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0e2e43e:::benchmark_data_smithi171_122772_object147523:head# 2024-04-20T17:27:43.695 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0e3ae0e:::benchmark_data_smithi171_122772_object149681:head# 2024-04-20T17:27:43.696 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0e42c3d:::benchmark_data_smithi171_122772_object125321:head# 2024-04-20T17:27:43.696 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0e6565b:::benchmark_data_smithi171_122772_object161430:head# 2024-04-20T17:27:43.697 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0e7612c:::benchmark_data_smithi171_122772_object156787:head# 2024-04-20T17:27:43.697 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0e87180:::benchmark_data_smithi171_122772_object133603:head# 2024-04-20T17:27:43.698 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0e9b9df:::benchmark_data_smithi171_122772_object125962:head# 2024-04-20T17:27:43.703 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0ea33e6:::benchmark_data_smithi171_122772_object128851:head# 2024-04-20T17:27:43.703 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0eeb31b:::benchmark_data_smithi171_122772_object127626:head# 2024-04-20T17:27:43.703 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0ef432c:::benchmark_data_smithi171_122772_object134728:head# 2024-04-20T17:27:43.703 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0ef5a5c:::benchmark_data_smithi171_122772_object109447:head# 2024-04-20T17:27:43.703 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0ef6e50:::benchmark_data_smithi171_122772_object115312:head# 2024-04-20T17:27:43.703 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0f00dc7:::benchmark_data_smithi171_122772_object97502:head# 2024-04-20T17:27:43.703 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0f22dcf:::benchmark_data_smithi171_122772_object111670:head# 2024-04-20T17:27:43.704 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0f2b4af:::benchmark_data_smithi171_122772_object162323:head# 2024-04-20T17:27:43.704 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0f34a06:::benchmark_data_smithi171_122772_object159511:head# 2024-04-20T17:27:43.704 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0f389e0:::benchmark_data_smithi171_122772_object141967:head# 2024-04-20T17:27:43.704 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0f45235:::benchmark_data_smithi171_122772_object155439:head# 2024-04-20T17:27:43.704 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0f45ae0:::benchmark_data_smithi171_122772_object78924:head# 2024-04-20T17:27:43.704 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0f59b5d:::benchmark_data_smithi171_122772_object81637:head# 2024-04-20T17:27:43.705 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0f63112:::benchmark_data_smithi171_122772_object152494:head# 2024-04-20T17:27:43.705 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0f65c9e:::benchmark_data_smithi171_122772_object156912:head# 2024-04-20T17:27:43.705 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0f685e6:::benchmark_data_smithi171_122772_object106536:head# 2024-04-20T17:27:43.706 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0f707ed:::benchmark_data_smithi171_122772_object105005:head# 2024-04-20T17:27:43.706 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0f786c5:::benchmark_data_smithi171_122772_object101046:head# 2024-04-20T17:27:43.707 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0f8fc3c:::benchmark_data_smithi171_122772_object169441:head# 2024-04-20T17:27:43.707 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0f94a7d:::benchmark_data_smithi171_122772_object126394:head# 2024-04-20T17:27:43.708 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0fa255b:::benchmark_data_smithi171_122772_object115217:head# 2024-04-20T17:27:43.708 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0fa74bb:::benchmark_data_smithi171_122772_object140225:head# 2024-04-20T17:27:43.708 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0fb828b:::benchmark_data_smithi171_122772_object94305:head# 2024-04-20T17:27:43.709 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0fbcee3:::benchmark_data_smithi171_122772_object152102:head# 2024-04-20T17:27:43.709 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0fc3254:::benchmark_data_smithi171_122772_object118846:head# 2024-04-20T17:27:43.710 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0fc5a76:::benchmark_data_smithi171_122772_object147501:head# 2024-04-20T17:27:43.710 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0fcf3cd:::benchmark_data_smithi171_122772_object116998:head# 2024-04-20T17:27:43.710 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0fd0de7:::benchmark_data_smithi171_122772_object127552:head# 2024-04-20T17:27:43.711 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0fd3092:::benchmark_data_smithi171_122772_object120011:head# 2024-04-20T17:27:43.711 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0fe17be:::benchmark_data_smithi171_122772_object167939:head# 2024-04-20T17:27:43.712 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0fe2294:::benchmark_data_smithi171_122772_object120289:head# 2024-04-20T17:27:43.712 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0fe2fd8:::benchmark_data_smithi171_122772_object116004:head# 2024-04-20T17:27:43.713 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0fec231:::benchmark_data_smithi171_122772_object118895:head# 2024-04-20T17:27:43.713 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0fee486:::benchmark_data_smithi171_122772_object80830:head# 2024-04-20T17:27:43.714 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a0ffe363:::benchmark_data_smithi171_122772_object123643:head# 2024-04-20T17:27:43.714 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1008162:::benchmark_data_smithi171_122772_object161690:head# 2024-04-20T17:27:43.715 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:27:43.715 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a101cb71:::benchmark_data_smithi171_122772_object107278:head# 2024-04-20T17:27:43.715 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a102e7ec:::benchmark_data_smithi171_122772_object136264:head# 2024-04-20T17:27:43.716 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1040601:::benchmark_data_smithi171_122772_object153991:head# 2024-04-20T17:27:43.716 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a104c804:::benchmark_data_smithi171_122772_object94371:head# 2024-04-20T17:27:43.717 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1050422:::benchmark_data_smithi171_122772_object109403:head# 2024-04-20T17:27:43.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:43.719 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:27:43.719 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a10569ff:::benchmark_data_smithi171_122772_object168132:head# 2024-04-20T17:27:43.720 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1059769:::benchmark_data_smithi171_122772_object133602:head# 2024-04-20T17:27:43.720 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a105c845:::benchmark_data_smithi171_122772_object94072:head# 2024-04-20T17:27:43.720 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1094899:::benchmark_data_smithi171_122772_object165294:head# 2024-04-20T17:27:43.720 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a10a8ed1:::benchmark_data_smithi171_122772_object125498:head# 2024-04-20T17:27:43.720 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a10ab7cb:::benchmark_data_smithi171_122772_object143824:head# 2024-04-20T17:27:43.721 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a10b2928:::benchmark_data_smithi171_122772_object135129:head# 2024-04-20T17:27:43.721 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a10b7fc1:::benchmark_data_smithi171_122772_object96982:head# 2024-04-20T17:27:43.722 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a10c1d33:::benchmark_data_smithi171_122772_object138910:head# 2024-04-20T17:27:43.722 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a10cef6e:::benchmark_data_smithi171_122772_object130622:head# 2024-04-20T17:27:43.723 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a10d0af7:::benchmark_data_smithi171_122772_object143389:head# 2024-04-20T17:27:43.724 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a10e26f6:::benchmark_data_smithi171_122772_object83156:head# 2024-04-20T17:27:43.724 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a10e70f7:::benchmark_data_smithi171_122772_object153518:head# 2024-04-20T17:27:43.725 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a10eaac0:::benchmark_data_smithi171_122772_object148951:head# 2024-04-20T17:27:43.725 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1107390:::benchmark_data_smithi171_122772_object151557:head# 2024-04-20T17:27:43.726 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1114ca0:::benchmark_data_smithi171_122772_object135612:head# 2024-04-20T17:27:43.726 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1118bcd:::benchmark_data_smithi171_122772_object143221:head# 2024-04-20T17:27:43.727 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a113b488:::benchmark_data_smithi171_122772_object132616:head# 2024-04-20T17:27:43.727 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a114a03d:::benchmark_data_smithi171_122772_object162183:head# 2024-04-20T17:27:43.727 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a116beaf:::benchmark_data_smithi171_122772_object103201:head# 2024-04-20T17:27:43.728 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a118e08d:::benchmark_data_smithi171_122772_object85705:head# 2024-04-20T17:27:43.728 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a11a3975:::benchmark_data_smithi171_122772_object155073:head# 2024-04-20T17:27:43.729 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a11b2507:::benchmark_data_smithi171_122772_object82379:head# 2024-04-20T17:27:43.729 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a11c5ab6:::benchmark_data_smithi171_122772_object141555:head# 2024-04-20T17:27:43.730 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a11f89f1:::benchmark_data_smithi171_122772_object136895:head# 2024-04-20T17:27:43.731 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1221928:::benchmark_data_smithi171_122772_object165317:head# 2024-04-20T17:27:43.731 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a122dcc1:::benchmark_data_smithi171_122772_object123925:head# 2024-04-20T17:27:43.732 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1238e2b:::benchmark_data_smithi171_122772_object143177:head# 2024-04-20T17:27:43.732 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1253456:::benchmark_data_smithi171_122772_object103802:head# 2024-04-20T17:27:43.733 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a125a3b6:::benchmark_data_smithi171_122772_object114497:head# 2024-04-20T17:27:43.733 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a126b9f9:::benchmark_data_smithi171_122772_object145843:head# 2024-04-20T17:27:43.734 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a126e87c:::benchmark_data_smithi171_122772_object116806:head# 2024-04-20T17:27:43.734 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1273844:::benchmark_data_smithi171_122772_object120527:head# 2024-04-20T17:27:43.735 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1282159:::benchmark_data_smithi171_122772_object153137:head# 2024-04-20T17:27:43.735 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a129b005:::benchmark_data_smithi171_122772_object130809:head# 2024-04-20T17:27:43.736 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a129ecd3:::benchmark_data_smithi171_122772_object130348:head# 2024-04-20T17:27:43.736 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a12a50a7:::benchmark_data_smithi171_122772_object109924:head# 2024-04-20T17:27:43.737 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a12af57f:::benchmark_data_smithi171_122772_object115216:head# 2024-04-20T17:27:43.737 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a12b5fdd:::benchmark_data_smithi171_122772_object168656:head# 2024-04-20T17:27:43.738 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a12b63aa:::benchmark_data_smithi171_122772_object84128:head# 2024-04-20T17:27:43.738 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a12da919:::benchmark_data_smithi171_122772_object87379:head# 2024-04-20T17:27:43.739 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a12ee19e:::benchmark_data_smithi171_122772_object102107:head# 2024-04-20T17:27:43.739 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a12f3fbf:::benchmark_data_smithi171_122772_object92967:head# 2024-04-20T17:27:43.740 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a130178f:::benchmark_data_smithi171_122772_object148729:head# 2024-04-20T17:27:43.741 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a130b0cf:::benchmark_data_smithi171_122772_object166351:head# 2024-04-20T17:27:43.741 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a131a723:::benchmark_data_smithi171_122772_object129444:head# 2024-04-20T17:27:43.741 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1321d1a:::benchmark_data_smithi171_122772_object152448:head# 2024-04-20T17:27:43.742 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1334fca:::benchmark_data_smithi171_122772_object118054:head# 2024-04-20T17:27:43.743 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1351589:::benchmark_data_smithi171_122772_object158323:head# 2024-04-20T17:27:43.743 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a135a30f:::benchmark_data_smithi171_122772_object157837:head# 2024-04-20T17:27:43.744 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1361584:::benchmark_data_smithi171_122772_object148801:head# 2024-04-20T17:27:43.744 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1361dfc:::benchmark_data_smithi171_122772_object88301:head# 2024-04-20T17:27:43.745 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1367475:::benchmark_data_smithi171_122772_object165676:head# 2024-04-20T17:27:43.745 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a136d915:::benchmark_data_smithi171_122772_object144403:head# 2024-04-20T17:27:43.746 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a137b4a2:::benchmark_data_smithi171_122772_object148446:head# 2024-04-20T17:27:43.746 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1382961:::benchmark_data_smithi171_122772_object81663:head# 2024-04-20T17:27:43.747 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a13937ca:::benchmark_data_smithi171_122772_object143723:head# 2024-04-20T17:27:43.747 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a139c9c5:::benchmark_data_smithi171_122772_object128047:head# 2024-04-20T17:27:43.748 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a13ad1fc:::benchmark_data_smithi171_122772_object78987:head# 2024-04-20T17:27:43.748 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a13c498d:::benchmark_data_smithi171_122772_object91510:head# 2024-04-20T17:27:43.749 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a13c8fc4:::benchmark_data_smithi171_122772_object101937:head# 2024-04-20T17:27:43.749 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a13cb293:::benchmark_data_smithi171_122772_object134757:head# 2024-04-20T17:27:43.750 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a13d1d57:::benchmark_data_smithi171_122772_object142432:head# 2024-04-20T17:27:43.750 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a13e214c:::benchmark_data_smithi171_122772_object87892:head# 2024-04-20T17:27:43.750 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a13fb824:::benchmark_data_smithi171_122772_object103442:head# 2024-04-20T17:27:43.751 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a13fd776:::benchmark_data_smithi171_122772_object92290:head# 2024-04-20T17:27:43.752 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1403d8d:::benchmark_data_smithi171_122772_object138491:head# 2024-04-20T17:27:43.752 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a140ef6a:::benchmark_data_smithi171_122772_object163377:head# 2024-04-20T17:27:43.753 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1411c7b:::benchmark_data_smithi171_122772_object107223:head# 2024-04-20T17:27:43.753 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a141e5bb:::benchmark_data_smithi171_122772_object138730:head# 2024-04-20T17:27:43.754 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1429e2a:::benchmark_data_smithi171_122772_object117551:head# 2024-04-20T17:27:43.754 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a142ef42:::benchmark_data_smithi171_122772_object109931:head# 2024-04-20T17:27:43.755 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1434c11:::benchmark_data_smithi171_122772_object134338:head# 2024-04-20T17:27:43.755 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a143c08d:::benchmark_data_smithi171_122772_object87823:head# 2024-04-20T17:27:43.756 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a145fdcd:::benchmark_data_smithi171_122772_object142395:head# 2024-04-20T17:27:43.756 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1462f48:::benchmark_data_smithi171_122772_object124008:head# 2024-04-20T17:27:43.757 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a146391c:::benchmark_data_smithi171_122772_object123641:head# 2024-04-20T17:27:43.758 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1465fc6:::benchmark_data_smithi171_122772_object156975:head# 2024-04-20T17:27:43.758 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a147bcfa:::benchmark_data_smithi171_122772_object90303:head# 2024-04-20T17:27:43.759 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a14856e4:::benchmark_data_smithi171_122772_object131181:head# 2024-04-20T17:27:43.759 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a14991b1:::benchmark_data_smithi171_122772_object117761:head# 2024-04-20T17:27:43.760 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a149cd34:::benchmark_data_smithi171_122772_object142391:head# 2024-04-20T17:27:43.760 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a14a9079:::benchmark_data_smithi171_122772_object146587:head# 2024-04-20T17:27:43.761 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a14bb275:::benchmark_data_smithi171_122772_object125918:head# 2024-04-20T17:27:43.761 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a14bfd68:::benchmark_data_smithi171_122772_object125892:head# 2024-04-20T17:27:43.761 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a14d01f3:::benchmark_data_smithi171_122772_object78214:head# 2024-04-20T17:27:43.762 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a14f939e:::benchmark_data_smithi171_122772_object141219:head# 2024-04-20T17:27:43.763 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a151373d:::benchmark_data_smithi171_122772_object141271:head# 2024-04-20T17:27:43.763 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1521972:::benchmark_data_smithi171_122772_object117474:head# 2024-04-20T17:27:43.764 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a15328e7:::benchmark_data_smithi171_122772_object157796:head# 2024-04-20T17:27:43.764 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a153e033:::benchmark_data_smithi171_122772_object132280:head# 2024-04-20T17:27:43.765 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a153f7cd:::benchmark_data_smithi171_122772_object137856:head# 2024-04-20T17:27:43.765 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a154b185:::benchmark_data_smithi171_122772_object168978:head# 2024-04-20T17:27:43.766 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a154c2e6:::benchmark_data_smithi171_122772_object120043:head# 2024-04-20T17:27:43.766 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1551e11:::benchmark_data_smithi171_122772_object125245:head# 2024-04-20T17:27:43.766 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1553b86:::benchmark_data_smithi171_122772_object86906:head# 2024-04-20T17:27:43.767 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a156d5a5:::benchmark_data_smithi171_122772_object121999:head# 2024-04-20T17:27:43.768 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1580990:::benchmark_data_smithi171_122772_object92221:head# 2024-04-20T17:27:43.768 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a159ec24:::benchmark_data_smithi171_122772_object99428:head# 2024-04-20T17:27:43.769 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a15b1022:::benchmark_data_smithi171_122772_object82188:head# 2024-04-20T17:27:43.769 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a15c5688:::benchmark_data_smithi171_122772_object84243:head# 2024-04-20T17:27:43.770 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a15cd326:::benchmark_data_smithi171_122772_object104417:head# 2024-04-20T17:27:43.770 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a15cd39d:::benchmark_data_smithi171_122772_object138988:head# 2024-04-20T17:27:43.771 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a15d481f:::benchmark_data_smithi171_122772_object116771:head# 2024-04-20T17:27:43.771 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a15d4fa3:::benchmark_data_smithi171_122772_object80514:head# 2024-04-20T17:27:43.772 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a15d5e39:::benchmark_data_smithi171_122772_object107870:head# 2024-04-20T17:27:43.772 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a15e137d:::benchmark_data_smithi171_122772_object158486:head# 2024-04-20T17:27:43.773 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a15ea9c5:::benchmark_data_smithi171_122772_object113319:head# 2024-04-20T17:27:43.773 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a15f5aa9:::benchmark_data_smithi171_122772_object162912:head# 2024-04-20T17:27:43.774 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a16091e7:::benchmark_data_smithi171_122772_object107968:head# 2024-04-20T17:27:43.774 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a160a06d:::benchmark_data_smithi171_122772_object115833:head# 2024-04-20T17:27:43.775 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1614b16:::benchmark_data_smithi171_122772_object145289:head# 2024-04-20T17:27:43.775 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a161cba0:::benchmark_data_smithi171_122772_object152028:head# 2024-04-20T17:27:43.776 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1625674:::benchmark_data_smithi171_122772_object152603:head# 2024-04-20T17:27:43.776 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1637534:::benchmark_data_smithi171_122772_object82496:head# 2024-04-20T17:27:43.777 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1645785:::benchmark_data_smithi171_122772_object101395:head# 2024-04-20T17:27:43.777 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a164bc05:::benchmark_data_smithi171_122772_object104820:head# 2024-04-20T17:27:43.777 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1695da9:::benchmark_data_smithi171_122772_object147670:head# 2024-04-20T17:27:43.778 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1696626:::benchmark_data_smithi171_122772_object137800:head# 2024-04-20T17:27:43.778 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1698833:::benchmark_data_smithi171_122772_object124256:head# 2024-04-20T17:27:43.779 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a16bb09c:::benchmark_data_smithi171_122772_object109898:head# 2024-04-20T17:27:43.780 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a16bfbe9:::benchmark_data_smithi171_122772_object97746:head# 2024-04-20T17:27:43.780 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a16f1876:::benchmark_data_smithi171_122772_object152882:head# 2024-04-20T17:27:43.781 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a16f1fb9:::benchmark_data_smithi171_122772_object167051:head# 2024-04-20T17:27:43.781 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a16f431b:::benchmark_data_smithi171_122772_object128942:head# 2024-04-20T17:27:43.782 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a16fd83e:::benchmark_data_smithi171_122772_object78636:head# 2024-04-20T17:27:43.782 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a170cf8d:::benchmark_data_smithi171_122772_object87334:head# 2024-04-20T17:27:43.783 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1717456:::benchmark_data_smithi171_122772_object160309:head# 2024-04-20T17:27:43.784 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a17291ea:::benchmark_data_smithi171_122772_object86023:head# 2024-04-20T17:27:43.784 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1736a3b:::benchmark_data_smithi171_122772_object153861:head# 2024-04-20T17:27:43.785 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a173d526:::benchmark_data_smithi171_122772_object145560:head# 2024-04-20T17:27:43.785 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1740218:::benchmark_data_smithi171_122772_object90579:head# 2024-04-20T17:27:43.786 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a174431e:::benchmark_data_smithi171_122772_object136530:head# 2024-04-20T17:27:43.786 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a174b001:::benchmark_data_smithi171_122772_object97362:head# 2024-04-20T17:27:43.787 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a17698c9:::benchmark_data_smithi171_122772_object115889:head# 2024-04-20T17:27:43.787 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a176ce45:::benchmark_data_smithi171_122772_object97092:head# 2024-04-20T17:27:43.788 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a176f590:::benchmark_data_smithi171_122772_object132385:head# 2024-04-20T17:27:43.788 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a17702f6:::benchmark_data_smithi171_122772_object154041:head# 2024-04-20T17:27:43.788 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a17873f7:::benchmark_data_smithi171_122772_object130946:head# 2024-04-20T17:27:43.789 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a178772c:::benchmark_data_smithi171_122772_object168180:head# 2024-04-20T17:27:43.789 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a17a26eb:::benchmark_data_smithi171_122772_object158925:head# 2024-04-20T17:27:43.790 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a17aa0cf:::benchmark_data_smithi171_122772_object116662:head# 2024-04-20T17:27:43.791 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a17b76ab:::benchmark_data_smithi171_122772_object88208:head# 2024-04-20T17:27:43.791 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a17bae6f:::benchmark_data_smithi171_122772_object142004:head# 2024-04-20T17:27:43.791 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a17c9068:::benchmark_data_smithi171_122772_object90390:head# 2024-04-20T17:27:43.792 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a17cda0b:::benchmark_data_smithi171_122772_object110809:head# 2024-04-20T17:27:43.792 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a17d758c:::benchmark_data_smithi171_122772_object80876:head# 2024-04-20T17:27:43.793 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a17d889a:::benchmark_data_smithi171_122772_object150323:head# 2024-04-20T17:27:43.793 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a17de5f3:::benchmark_data_smithi171_122772_object88521:head# 2024-04-20T17:27:43.794 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a17e0cd3:::benchmark_data_smithi171_122772_object103669:head# 2024-04-20T17:27:43.794 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a17e6252:::benchmark_data_smithi171_122772_object98155:head# 2024-04-20T17:27:43.795 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a17e9797:::benchmark_data_smithi171_122772_object97036:head# 2024-04-20T17:27:43.795 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a17f4fde:::benchmark_data_smithi171_122772_object93542:head# 2024-04-20T17:27:43.796 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a17fc4b6:::benchmark_data_smithi171_122772_object92759:head# 2024-04-20T17:27:43.797 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a17fe657:::benchmark_data_smithi171_122772_object167628:head# 2024-04-20T17:27:43.797 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1806277:::benchmark_data_smithi171_122772_object162740:head# 2024-04-20T17:27:43.798 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1807666:::benchmark_data_smithi171_122772_object91080:head# 2024-04-20T17:27:43.800 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1808fc6:::benchmark_data_smithi171_122772_object167122:head# 2024-04-20T17:27:43.800 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a181b276:::benchmark_data_smithi171_122772_object113097:head# 2024-04-20T17:27:43.800 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1839640:::benchmark_data_smithi171_122772_object127731:head# 2024-04-20T17:27:43.800 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a183ac1b:::benchmark_data_smithi171_122772_object143961:head# 2024-04-20T17:27:43.801 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a183c946:::benchmark_data_smithi171_122772_object140486:head# 2024-04-20T17:27:43.801 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a184b9d0:::benchmark_data_smithi171_122772_object117738:head# 2024-04-20T17:27:43.801 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a18556b0:::benchmark_data_smithi171_122772_object125217:head# 2024-04-20T17:27:43.802 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1888282:::benchmark_data_smithi171_122772_object102158:head# 2024-04-20T17:27:43.802 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a18889e4:::benchmark_data_smithi171_122772_object159447:head# 2024-04-20T17:27:43.803 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a188f140:::benchmark_data_smithi171_122772_object153124:head# 2024-04-20T17:27:43.803 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a18998e9:::benchmark_data_smithi171_122772_object156768:head# 2024-04-20T17:27:43.804 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a189c080:::benchmark_data_smithi171_122772_object138017:head# 2024-04-20T17:27:43.804 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a18a2f48:::benchmark_data_smithi171_122772_object153522:head# 2024-04-20T17:27:43.804 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a18b46cf:::benchmark_data_smithi171_122772_object129512:head# 2024-04-20T17:27:43.805 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a18b9e55:::benchmark_data_smithi171_122772_object155277:head# 2024-04-20T17:27:43.805 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a18c4bee:::benchmark_data_smithi171_122772_object78310:head# 2024-04-20T17:27:43.806 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a18c9563:::benchmark_data_smithi171_122772_object110149:head# 2024-04-20T17:27:43.806 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a18ca7cf:::benchmark_data_smithi171_122772_object155980:head# 2024-04-20T17:27:43.807 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a18cccf3:::benchmark_data_smithi171_122772_object137251:head# 2024-04-20T17:27:43.807 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a18d4e77:::benchmark_data_smithi171_122772_object101157:head# 2024-04-20T17:27:43.808 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a18de7de:::benchmark_data_smithi171_122772_object98400:head# 2024-04-20T17:27:43.809 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a18e60f2:::benchmark_data_smithi171_122772_object155771:head# 2024-04-20T17:27:43.809 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a18fc426:::benchmark_data_smithi171_122772_object124667:head# 2024-04-20T17:27:43.810 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a191a5b4:::benchmark_data_smithi171_122772_object97733:head# 2024-04-20T17:27:43.810 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a191b4a9:::benchmark_data_smithi171_122772_object98475:head# 2024-04-20T17:27:43.811 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a191c52c:::benchmark_data_smithi171_122772_object94193:head# 2024-04-20T17:27:43.811 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1940ebd:::benchmark_data_smithi171_122772_object138579:head# 2024-04-20T17:27:43.812 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a194f3e4:::benchmark_data_smithi171_122772_object163091:head# 2024-04-20T17:27:43.812 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a195f97e:::benchmark_data_smithi171_122772_object157066:head# 2024-04-20T17:27:43.813 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a196e596:::benchmark_data_smithi171_122772_object151976:head# 2024-04-20T17:27:43.813 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a197e52e:::benchmark_data_smithi171_122772_object78713:head# 2024-04-20T17:27:43.814 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a197fc6e:::benchmark_data_smithi171_122772_object157820:head# 2024-04-20T17:27:43.814 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a19885f3:::benchmark_data_smithi171_122772_object156941:head# 2024-04-20T17:27:43.815 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a19902d9:::benchmark_data_smithi171_122772_object150214:head# 2024-04-20T17:27:43.815 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a19b1c15:::benchmark_data_smithi171_122772_object163301:head# 2024-04-20T17:27:43.815 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a19bd27d:::benchmark_data_smithi171_122772_object165681:head# 2024-04-20T17:27:43.816 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a19c398a:::benchmark_data_smithi171_122772_object87569:head# 2024-04-20T17:27:43.816 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a19d9f6d:::benchmark_data_smithi171_122772_object93596:head# 2024-04-20T17:27:43.817 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a19e6d17:::benchmark_data_smithi171_122772_object103315:head# 2024-04-20T17:27:43.817 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a19e8687:::benchmark_data_smithi171_122772_object144115:head# 2024-04-20T17:27:43.818 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a19e87f5:::benchmark_data_smithi171_122772_object142313:head# 2024-04-20T17:27:43.818 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1a13f18:::benchmark_data_smithi171_122772_object125890:head# 2024-04-20T17:27:43.819 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1a379a1:::benchmark_data_smithi171_122772_object138749:head# 2024-04-20T17:27:43.819 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:27:43.820 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1a3d71b:::benchmark_data_smithi171_122772_object152678:head# 2024-04-20T17:27:43.820 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1a50240:::benchmark_data_smithi171_122772_object124170:head# 2024-04-20T17:27:43.820 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1a50963:::benchmark_data_smithi171_122772_object143833:head# 2024-04-20T17:27:43.821 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1a55d46:::benchmark_data_smithi171_122772_object161318:head# 2024-04-20T17:27:43.821 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1a60f30:::benchmark_data_smithi171_122772_object118975:head# 2024-04-20T17:27:43.822 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1a6199b:::benchmark_data_smithi171_122772_object82275:head# 2024-04-20T17:27:43.822 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1a64ba4:::benchmark_data_smithi171_122772_object80651:head# 2024-04-20T17:27:43.823 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1a80715:::benchmark_data_smithi171_122772_object143106:head# 2024-04-20T17:27:43.823 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1a94daf:::benchmark_data_smithi171_122772_object97950:head# 2024-04-20T17:27:43.824 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1a9753a:::benchmark_data_smithi171_122772_object100419:head# 2024-04-20T17:27:43.824 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1a98d8a:::benchmark_data_smithi171_122772_object143325:head# 2024-04-20T17:27:43.824 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1aa6f5a:::benchmark_data_smithi171_122772_object140190:head# 2024-04-20T17:27:43.825 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1aa95ee:::benchmark_data_smithi171_122772_object156281:head# 2024-04-20T17:27:43.826 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1abbdd4:::benchmark_data_smithi171_122772_object120895:head# 2024-04-20T17:27:43.826 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1ac9334:::benchmark_data_smithi171_122772_object146550:head# 2024-04-20T17:27:43.827 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1ad943e:::benchmark_data_smithi171_122772_object83779:head# 2024-04-20T17:27:43.827 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1aee803:::benchmark_data_smithi171_122772_object81890:head# 2024-04-20T17:27:43.828 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1af2d93:::benchmark_data_smithi171_122772_object137360:head# 2024-04-20T17:27:43.828 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1af6231:::benchmark_data_smithi171_122772_object134610:head# 2024-04-20T17:27:43.828 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1b02b5a:::benchmark_data_smithi171_122772_object151975:head# 2024-04-20T17:27:43.829 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1b2bbeb:::benchmark_data_smithi171_122772_object135502:head# 2024-04-20T17:27:43.830 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1b3d582:::benchmark_data_smithi171_122772_object159327:head# 2024-04-20T17:27:43.830 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1b433d5:::benchmark_data_smithi171_122772_object92403:head# 2024-04-20T17:27:43.831 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1b44b61:::benchmark_data_smithi171_122772_object127225:head# 2024-04-20T17:27:43.831 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1b4e809:::benchmark_data_smithi171_122772_object123874:head# 2024-04-20T17:27:43.831 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1b59093:::benchmark_data_smithi171_122772_object137859:head# 2024-04-20T17:27:43.832 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1b59f0c:::benchmark_data_smithi171_122772_object97143:head# 2024-04-20T17:27:43.832 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1b72a1c:::benchmark_data_smithi171_122772_object155828:head# 2024-04-20T17:27:43.833 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1b73a08:::benchmark_data_smithi171_122772_object163916:head# 2024-04-20T17:27:43.833 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1b8ab3a:::benchmark_data_smithi171_122772_object165880:head# 2024-04-20T17:27:43.834 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1ba17da:::benchmark_data_smithi171_122772_object91740:head# 2024-04-20T17:27:43.834 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1bb4693:::benchmark_data_smithi171_122772_object125253:head# 2024-04-20T17:27:43.835 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1bba48e:::benchmark_data_smithi171_122772_object139609:head# 2024-04-20T17:27:43.836 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1bc8ad0:::benchmark_data_smithi171_122772_object100174:head# 2024-04-20T17:27:43.836 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1bcb956:::benchmark_data_smithi171_122772_object90702:head# 2024-04-20T17:27:43.837 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1bdf3b0:::benchmark_data_smithi171_122772_object169702:head# 2024-04-20T17:27:43.838 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1bf7cfa:::benchmark_data_smithi171_122772_object103816:head# 2024-04-20T17:27:43.838 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1bf98e0:::benchmark_data_smithi171_122772_object144608:head# 2024-04-20T17:27:43.839 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1bff17a:::benchmark_data_smithi171_122772_object86198:head# 2024-04-20T17:27:43.839 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1c086fb:::benchmark_data_smithi171_122772_object147011:head# 2024-04-20T17:27:43.840 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1c0ab2e:::benchmark_data_smithi171_122772_object101223:head# 2024-04-20T17:27:43.840 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1c0cc57:::benchmark_data_smithi171_122772_object161304:head# 2024-04-20T17:27:43.841 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1c124fc:::benchmark_data_smithi171_122772_object116784:head# 2024-04-20T17:27:43.841 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1c1373a:::benchmark_data_smithi171_122772_object126391:head# 2024-04-20T17:27:43.842 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1c17b42:::benchmark_data_smithi171_122772_object152866:head# 2024-04-20T17:27:43.842 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1c27d83:::benchmark_data_smithi171_122772_object83902:head# 2024-04-20T17:27:43.843 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1c2b591:::benchmark_data_smithi171_122772_object162889:head# 2024-04-20T17:27:43.843 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1c2f569:::benchmark_data_smithi171_122772_object109047:head# 2024-04-20T17:27:43.844 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1c31b07:::benchmark_data_smithi171_122772_object168398:head# 2024-04-20T17:27:43.844 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1c3a710:::benchmark_data_smithi171_122772_object152100:head# 2024-04-20T17:27:43.845 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1c3c00c:::benchmark_data_smithi171_122772_object97729:head# 2024-04-20T17:27:43.845 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1c43924:::benchmark_data_smithi171_122772_object162177:head# 2024-04-20T17:27:43.846 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1c4629f:::benchmark_data_smithi171_122772_object158713:head# 2024-04-20T17:27:43.846 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1c5172e:::benchmark_data_smithi171_122772_object143010:head# 2024-04-20T17:27:43.847 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1c5876c:::benchmark_data_smithi171_122772_object78403:head# 2024-04-20T17:27:43.847 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1c5d431:::benchmark_data_smithi171_122772_object159170:head# 2024-04-20T17:27:43.848 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1c5fa18:::benchmark_data_smithi171_122772_object91145:head# 2024-04-20T17:27:43.848 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1c6c3a2:::benchmark_data_smithi171_122772_object156425:head# 2024-04-20T17:27:43.849 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1c704d9:::benchmark_data_smithi171_122772_object164086:head# 2024-04-20T17:27:43.849 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1c7f44b:::benchmark_data_smithi171_122772_object165334:head# 2024-04-20T17:27:43.850 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1c8b278:::benchmark_data_smithi171_122772_object85986:head# 2024-04-20T17:27:43.850 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1c98af9:::benchmark_data_smithi171_122772_object132511:head# 2024-04-20T17:27:43.851 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1ca4295:::benchmark_data_smithi171_122772_object117159:head# 2024-04-20T17:27:43.851 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1ca633a:::benchmark_data_smithi171_122772_object131753:head# 2024-04-20T17:27:43.852 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1ca8ffc:::benchmark_data_smithi171_122772_object90102:head# 2024-04-20T17:27:43.852 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:43.853 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1cae4de:::benchmark_data_smithi171_122772_object138060:head# 2024-04-20T17:27:43.853 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1cc82a1:::benchmark_data_smithi171_122772_object79344:head# 2024-04-20T17:27:43.854 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1cced76:::benchmark_data_smithi171_122772_object97959:head# 2024-04-20T17:27:43.854 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1cd17ef:::benchmark_data_smithi171_122772_object165643:head# 2024-04-20T17:27:43.855 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1cd1986:::benchmark_data_smithi171_122772_object123698:head# 2024-04-20T17:27:43.855 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1cdc94a:::benchmark_data_smithi171_122772_object140301:head# 2024-04-20T17:27:43.855 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1d1f6e4:::benchmark_data_smithi171_122772_object161289:head# 2024-04-20T17:27:43.856 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1d35ec8:::benchmark_data_smithi171_122772_object154366:head# 2024-04-20T17:27:43.857 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1d50bd3:::benchmark_data_smithi171_122772_object81848:head# 2024-04-20T17:27:43.857 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1d611c8:::benchmark_data_smithi171_122772_object79970:head# 2024-04-20T17:27:43.858 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1d6c921:::benchmark_data_smithi171_122772_object148294:head# 2024-04-20T17:27:43.858 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1d77ad5:::benchmark_data_smithi171_122772_object148525:head# 2024-04-20T17:27:43.859 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1d8224c:::benchmark_data_smithi171_122772_object84720:head# 2024-04-20T17:27:43.859 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1d8571e:::benchmark_data_smithi171_122772_object123519:head# 2024-04-20T17:27:43.860 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1d95758:::benchmark_data_smithi171_122772_object101674:head# 2024-04-20T17:27:43.860 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1d98f6a:::benchmark_data_smithi171_122772_object91802:head# 2024-04-20T17:27:43.861 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1db06b1:::benchmark_data_smithi171_122772_object167883:head# 2024-04-20T17:27:43.861 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1dbd776:::benchmark_data_smithi171_122772_object95370:head# 2024-04-20T17:27:43.862 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1dc4fc4:::benchmark_data_smithi171_122772_object91066:head# 2024-04-20T17:27:43.862 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1dca3b1:::benchmark_data_smithi171_122772_object137288:head# 2024-04-20T17:27:43.863 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1dd2c0d:::benchmark_data_smithi171_122772_object100739:head# 2024-04-20T17:27:43.863 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1dd5eb9:::benchmark_data_smithi171_122772_object158439:head# 2024-04-20T17:27:43.864 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1ddd612:::benchmark_data_smithi171_122772_object142917:head# 2024-04-20T17:27:43.865 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1df3d34:::benchmark_data_smithi171_122772_object154111:head# 2024-04-20T17:27:43.865 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1df69a9:::benchmark_data_smithi171_122772_object97279:head# 2024-04-20T17:27:43.866 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1e0561d:::benchmark_data_smithi171_122772_object158827:head# 2024-04-20T17:27:43.866 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1e0ef7d:::benchmark_data_smithi171_122772_object119912:head# 2024-04-20T17:27:43.867 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1e385dc:::benchmark_data_smithi171_122772_object169093:head# 2024-04-20T17:27:43.867 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1e398b7:::benchmark_data_smithi171_122772_object158086:head# 2024-04-20T17:27:43.868 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1e3bbf4:::benchmark_data_smithi171_122772_object153711:head# 2024-04-20T17:27:43.868 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1e3cf8d:::benchmark_data_smithi171_122772_object127884:head# 2024-04-20T17:27:43.869 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1e3eff0:::benchmark_data_smithi171_122772_object135354:head# 2024-04-20T17:27:43.869 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1e411fa:::benchmark_data_smithi171_122772_object134995:head# 2024-04-20T17:27:43.870 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1e57254:::benchmark_data_smithi171_122772_object127853:head# 2024-04-20T17:27:43.870 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1e64314:::benchmark_data_smithi171_122772_object97164:head# 2024-04-20T17:27:43.871 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1e66d59:::benchmark_data_smithi171_122772_object154125:head# 2024-04-20T17:27:43.871 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1ea532f:::benchmark_data_smithi171_122772_object163090:head# 2024-04-20T17:27:43.872 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1eb04c1:::benchmark_data_smithi171_122772_object106011:head# 2024-04-20T17:27:43.872 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1eb4fcb:::benchmark_data_smithi171_122772_object79126:head# 2024-04-20T17:27:43.873 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1eb9957:::benchmark_data_smithi171_122772_object100933:head# 2024-04-20T17:27:43.873 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1ebf086:::benchmark_data_smithi171_122772_object156024:head# 2024-04-20T17:27:43.874 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1ebf985:::benchmark_data_smithi171_122772_object153246:head# 2024-04-20T17:27:43.874 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1ec7809:::benchmark_data_smithi171_122772_object156712:head# 2024-04-20T17:27:43.875 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1ecb351:::benchmark_data_smithi171_122772_object152359:head# 2024-04-20T17:27:43.875 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1ecb7f8:::benchmark_data_smithi171_122772_object124921:head# 2024-04-20T17:27:43.876 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:27:43.876 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1eeeeb0:::benchmark_data_smithi171_122772_object153629:head# 2024-04-20T17:27:43.876 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1f014b6:::benchmark_data_smithi171_122772_object133351:head# 2024-04-20T17:27:43.877 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1f1006e:::benchmark_data_smithi171_122772_object126063:head# 2024-04-20T17:27:43.877 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1f241af:::benchmark_data_smithi171_122772_object146398:head# 2024-04-20T17:27:43.878 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1f2d943:::benchmark_data_smithi171_122772_object160645:head# 2024-04-20T17:27:43.878 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1f34c47:::benchmark_data_smithi171_122772_object148983:head# 2024-04-20T17:27:43.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:43.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:27:43.880 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1f3602b:::benchmark_data_smithi171_122772_object120772:head# 2024-04-20T17:27:43.880 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1f36988:::benchmark_data_smithi171_122772_object114623:head# 2024-04-20T17:27:43.880 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1f5e7c7:::benchmark_data_smithi171_122772_object121749:head# 2024-04-20T17:27:43.880 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1f6164f:::benchmark_data_smithi171_122772_object129331:head# 2024-04-20T17:27:43.880 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1f69b98:::benchmark_data_smithi171_122772_object151655:head# 2024-04-20T17:27:43.881 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1f6b6e6:::benchmark_data_smithi171_122772_object135361:head# 2024-04-20T17:27:43.881 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1f6d498:::benchmark_data_smithi171_122772_object164583:head# 2024-04-20T17:27:43.882 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1f893cc:::benchmark_data_smithi171_122772_object138954:head# 2024-04-20T17:27:43.882 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1f8e160:::benchmark_data_smithi171_122772_object136239:head# 2024-04-20T17:27:43.883 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1fa0dc5:::benchmark_data_smithi171_122772_object94761:head# 2024-04-20T17:27:43.883 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1fa6923:::benchmark_data_smithi171_122772_object83828:head# 2024-04-20T17:27:43.884 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1fad3d4:::benchmark_data_smithi171_122772_object86219:head# 2024-04-20T17:27:43.884 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1fbf3ed:::benchmark_data_smithi171_122772_object147695:head# 2024-04-20T17:27:43.885 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1fcbbe1:::benchmark_data_smithi171_122772_object144825:head# 2024-04-20T17:27:43.885 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1fd8585:::benchmark_data_smithi171_122772_object140905:head# 2024-04-20T17:27:43.886 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1fdbb05:::benchmark_data_smithi171_122772_object123466:head# 2024-04-20T17:27:43.886 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1fde8e4:::benchmark_data_smithi171_122772_object157402:head# 2024-04-20T17:27:43.887 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1fe7d4a:::benchmark_data_smithi171_122772_object120066:head# 2024-04-20T17:27:43.887 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1fef6ef:::benchmark_data_smithi171_122772_object101484:head# 2024-04-20T17:27:43.888 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a1ffb206:::benchmark_data_smithi171_122772_object116301:head# 2024-04-20T17:27:43.888 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a200790f:::benchmark_data_smithi171_122772_object149742:head# 2024-04-20T17:27:43.889 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a200dfdb:::benchmark_data_smithi171_122772_object78974:head# 2024-04-20T17:27:43.889 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2011d4a:::benchmark_data_smithi171_122772_object117431:head# 2024-04-20T17:27:43.890 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2044708:::benchmark_data_smithi171_122772_object162911:head# 2024-04-20T17:27:43.891 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a205b227:::benchmark_data_smithi171_122772_object139994:head# 2024-04-20T17:27:43.891 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2060b28:::benchmark_data_smithi171_122772_object105317:head# 2024-04-20T17:27:43.892 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2071a78:::benchmark_data_smithi171_122772_object132957:head# 2024-04-20T17:27:43.892 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2079194:::benchmark_data_smithi171_122772_object138224:head# 2024-04-20T17:27:43.893 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a207e502:::benchmark_data_smithi171_122772_object147570:head# 2024-04-20T17:27:43.893 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2081ea5:::benchmark_data_smithi171_122772_object104023:head# 2024-04-20T17:27:43.894 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2082a2e:::benchmark_data_smithi171_122772_object129930:head# 2024-04-20T17:27:43.894 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2088f30:::benchmark_data_smithi171_122772_object161173:head# 2024-04-20T17:27:43.895 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a208b2b8:::benchmark_data_smithi171_122772_object157344:head# 2024-04-20T17:27:43.895 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a209f7d8:::benchmark_data_smithi171_122772_object107113:head# 2024-04-20T17:27:43.896 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a20a6670:::benchmark_data_smithi171_122772_object130698:head# 2024-04-20T17:27:43.897 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a20aa73b:::benchmark_data_smithi171_122772_object119036:head# 2024-04-20T17:27:43.897 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a20f07d3:::benchmark_data_smithi171_122772_object169198:head# 2024-04-20T17:27:43.898 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a20f6f06:::benchmark_data_smithi171_122772_object138758:head# 2024-04-20T17:27:43.898 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a210066b:::benchmark_data_smithi171_122772_object166951:head# 2024-04-20T17:27:43.903 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a212e211:::benchmark_data_smithi171_122772_object114292:head# 2024-04-20T17:27:43.903 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2148d8b:::benchmark_data_smithi171_122772_object139657:head# 2024-04-20T17:27:43.903 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a214bc6a:::benchmark_data_smithi171_122772_object87252:head# 2024-04-20T17:27:43.903 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a214ee67:::benchmark_data_smithi171_122772_object118121:head# 2024-04-20T17:27:43.903 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a215223a:::benchmark_data_smithi171_122772_object114175:head# 2024-04-20T17:27:43.903 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2161b9b:::benchmark_data_smithi171_122772_object128748:head# 2024-04-20T17:27:43.903 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a21698c9:::benchmark_data_smithi171_122772_object166375:head# 2024-04-20T17:27:43.903 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2189a31:::benchmark_data_smithi171_122772_object169829:head# 2024-04-20T17:27:43.903 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2193eca:::benchmark_data_smithi171_122772_object156740:head# 2024-04-20T17:27:43.904 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a21a282d:::benchmark_data_smithi171_122772_object126465:head# 2024-04-20T17:27:43.904 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a21b42dc:::benchmark_data_smithi171_122772_object130073:head# 2024-04-20T17:27:43.904 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a21bbae7:::benchmark_data_smithi171_122772_object165648:head# 2024-04-20T17:27:43.905 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a21ce2fc:::benchmark_data_smithi171_122772_object121837:head# 2024-04-20T17:27:43.905 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a21e05a2:::benchmark_data_smithi171_122772_object85556:head# 2024-04-20T17:27:43.906 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a21e12bb:::benchmark_data_smithi171_122772_object82219:head# 2024-04-20T17:27:43.906 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a22016ae:::benchmark_data_smithi171_122772_object127063:head# 2024-04-20T17:27:43.907 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2204204:::benchmark_data_smithi171_122772_object96239:head# 2024-04-20T17:27:43.907 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a221d9a5:::benchmark_data_smithi171_122772_object142020:head# 2024-04-20T17:27:43.908 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a223194b:::benchmark_data_smithi171_122772_object119867:head# 2024-04-20T17:27:43.908 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a22365ee:::benchmark_data_smithi171_122772_object150345:head# 2024-04-20T17:27:43.909 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2252380:::benchmark_data_smithi171_122772_object143689:head# 2024-04-20T17:27:43.909 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a226474e:::benchmark_data_smithi171_122772_object160724:head# 2024-04-20T17:27:43.910 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a226e2d3:::benchmark_data_smithi171_122772_object117055:head# 2024-04-20T17:27:43.910 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a226ffff:::benchmark_data_smithi171_122772_object155160:head# 2024-04-20T17:27:43.911 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a227221f:::benchmark_data_smithi171_122772_object99803:head# 2024-04-20T17:27:43.911 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2277b61:::benchmark_data_smithi171_122772_object119958:head# 2024-04-20T17:27:43.912 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a227bc4f:::benchmark_data_smithi171_122772_object107419:head# 2024-04-20T17:27:43.912 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a22820cd:::benchmark_data_smithi171_122772_object135694:head# 2024-04-20T17:27:43.913 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a228b6a6:::benchmark_data_smithi171_122772_object120366:head# 2024-04-20T17:27:43.913 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a228f7f4:::benchmark_data_smithi171_122772_object137400:head# 2024-04-20T17:27:43.914 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a22990f8:::benchmark_data_smithi171_122772_object101994:head# 2024-04-20T17:27:43.914 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a229d2c9:::benchmark_data_smithi171_122772_object111245:head# 2024-04-20T17:27:43.915 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a229e8e0:::benchmark_data_smithi171_122772_object90517:head# 2024-04-20T17:27:43.915 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a22b95f4:::benchmark_data_smithi171_122772_object145757:head# 2024-04-20T17:27:43.916 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a22d8faa:::benchmark_data_smithi171_122772_object135687:head# 2024-04-20T17:27:43.916 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a22d98d7:::benchmark_data_smithi171_122772_object93532:head# 2024-04-20T17:27:43.917 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a22da5ee:::benchmark_data_smithi171_122772_object148535:head# 2024-04-20T17:27:43.917 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a22e2040:::benchmark_data_smithi171_122772_object89757:head# 2024-04-20T17:27:43.918 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a22f050a:::benchmark_data_smithi171_122772_object126576:head# 2024-04-20T17:27:43.918 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a22f4c22:::benchmark_data_smithi171_122772_object90395:head# 2024-04-20T17:27:43.919 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2310a00:::benchmark_data_smithi171_122772_object94742:head# 2024-04-20T17:27:43.920 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2310dee:::benchmark_data_smithi171_122772_object133689:head# 2024-04-20T17:27:43.920 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2312a3c:::benchmark_data_smithi171_122772_object138354:head# 2024-04-20T17:27:43.921 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a232732b:::benchmark_data_smithi171_122772_object132675:head# 2024-04-20T17:27:43.921 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2329d53:::benchmark_data_smithi171_122772_object139831:head# 2024-04-20T17:27:43.922 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2336e38:::benchmark_data_smithi171_122772_object130805:head# 2024-04-20T17:27:43.922 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a233e6bd:::benchmark_data_smithi171_122772_object93553:head# 2024-04-20T17:27:43.923 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a237a885:::benchmark_data_smithi171_122772_object83477:head# 2024-04-20T17:27:43.924 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a237d9b6:::benchmark_data_smithi171_122772_object154553:head# 2024-04-20T17:27:43.924 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a23844a1:::benchmark_data_smithi171_122772_object147468:head# 2024-04-20T17:27:43.925 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2386753:::benchmark_data_smithi171_122772_object98507:head# 2024-04-20T17:27:43.925 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2389809:::benchmark_data_smithi171_122772_object149178:head# 2024-04-20T17:27:43.926 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a238acb5:::benchmark_data_smithi171_122772_object86814:head# 2024-04-20T17:27:43.926 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2393898:::benchmark_data_smithi171_122772_object91233:head# 2024-04-20T17:27:43.927 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a23a386f:::benchmark_data_smithi171_122772_object151969:head# 2024-04-20T17:27:43.927 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a23c0e3e:::benchmark_data_smithi171_122772_object146325:head# 2024-04-20T17:27:43.928 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a23c950d:::benchmark_data_smithi171_122772_object111925:head# 2024-04-20T17:27:43.928 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a23d52e8:::benchmark_data_smithi171_122772_object134311:head# 2024-04-20T17:27:43.929 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a23ddd62:::benchmark_data_smithi171_122772_object115460:head# 2024-04-20T17:27:43.929 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a23df218:::benchmark_data_smithi171_122772_object166634:head# 2024-04-20T17:27:43.930 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a23fc673:::benchmark_data_smithi171_122772_object83501:head# 2024-04-20T17:27:43.930 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2409e1a:::benchmark_data_smithi171_122772_object95393:head# 2024-04-20T17:27:43.932 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2416cc1:::benchmark_data_smithi171_122772_object93322:head# 2024-04-20T17:27:43.932 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a241aada:::benchmark_data_smithi171_122772_object105963:head# 2024-04-20T17:27:43.932 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a241b75d:::benchmark_data_smithi171_122772_object150270:head# 2024-04-20T17:27:43.933 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a24250be:::benchmark_data_smithi171_122772_object121577:head# 2024-04-20T17:27:43.933 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2453f4e:::benchmark_data_smithi171_122772_object136638:head# 2024-04-20T17:27:43.934 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2485f3a:::benchmark_data_smithi171_122772_object147741:head# 2024-04-20T17:27:43.934 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a248f36a:::benchmark_data_smithi171_122772_object151916:head# 2024-04-20T17:27:43.935 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a24911c4:::benchmark_data_smithi171_122772_object140807:head# 2024-04-20T17:27:43.936 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a24958f9:::benchmark_data_smithi171_122772_object140231:head# 2024-04-20T17:27:43.936 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a24ab2fe:::benchmark_data_smithi171_122772_object79564:head# 2024-04-20T17:27:43.938 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a24b5b8c:::benchmark_data_smithi171_122772_object97581:head# 2024-04-20T17:27:43.939 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a24dbe33:::benchmark_data_smithi171_122772_object163460:head# 2024-04-20T17:27:43.939 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a24f10ed:::benchmark_data_smithi171_122772_object140865:head# 2024-04-20T17:27:43.939 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a24fda16:::benchmark_data_smithi171_122772_object82125:head# 2024-04-20T17:27:43.939 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a25111bf:::benchmark_data_smithi171_122772_object109474:head# 2024-04-20T17:27:43.940 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a25265cc:::benchmark_data_smithi171_122772_object117671:head# 2024-04-20T17:27:43.941 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a252d672:::benchmark_data_smithi171_122772_object160297:head# 2024-04-20T17:27:43.941 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a252f2d9:::benchmark_data_smithi171_122772_object157542:head# 2024-04-20T17:27:43.941 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2530a7e:::benchmark_data_smithi171_122772_object114434:head# 2024-04-20T17:27:43.942 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a254daef:::benchmark_data_smithi171_122772_object120053:head# 2024-04-20T17:27:43.942 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a255abd2:::benchmark_data_smithi171_122772_object113507:head# 2024-04-20T17:27:43.942 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2562d86:::benchmark_data_smithi171_122772_object122266:head# 2024-04-20T17:27:43.943 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2597bc1:::benchmark_data_smithi171_122772_object103017:head# 2024-04-20T17:27:43.943 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a25a66c4:::benchmark_data_smithi171_122772_object150597:head# 2024-04-20T17:27:43.944 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a25aa6c0:::benchmark_data_smithi171_122772_object155623:head# 2024-04-20T17:27:43.944 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a25b3a22:::benchmark_data_smithi171_122772_object148760:head# 2024-04-20T17:27:43.945 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a25bef9c:::benchmark_data_smithi171_122772_object155922:head# 2024-04-20T17:27:43.946 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a25cf0b5:::benchmark_data_smithi171_122772_object160428:head# 2024-04-20T17:27:43.946 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a25da818:::benchmark_data_smithi171_122772_object157487:head# 2024-04-20T17:27:43.946 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a25df0f1:::benchmark_data_smithi171_122772_object138706:head# 2024-04-20T17:27:43.946 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a25e1235:::benchmark_data_smithi171_122772_object101480:head# 2024-04-20T17:27:43.947 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a25ee5f2:::benchmark_data_smithi171_122772_object110966:head# 2024-04-20T17:27:43.948 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a25f5c2e:::benchmark_data_smithi171_122772_object85248:head# 2024-04-20T17:27:43.948 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a25fde20:::benchmark_data_smithi171_122772_object168726:head# 2024-04-20T17:27:43.949 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2617a95:::benchmark_data_smithi171_122772_object124561:head# 2024-04-20T17:27:43.949 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2618e51:::benchmark_data_smithi171_122772_object107967:head# 2024-04-20T17:27:43.950 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a261a50e:::benchmark_data_smithi171_122772_object107338:head# 2024-04-20T17:27:43.950 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2627098:::benchmark_data_smithi171_122772_object116290:head# 2024-04-20T17:27:43.951 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a26454ae:::benchmark_data_smithi171_122772_object117712:head# 2024-04-20T17:27:43.951 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a26463b7:::benchmark_data_smithi171_122772_object100482:head# 2024-04-20T17:27:43.952 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a26538e8:::benchmark_data_smithi171_122772_object103708:head# 2024-04-20T17:27:43.952 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a265be4b:::benchmark_data_smithi171_122772_object91447:head# 2024-04-20T17:27:43.953 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a265c0fa:::benchmark_data_smithi171_122772_object157810:head# 2024-04-20T17:27:43.953 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a265f131:::benchmark_data_smithi171_122772_object127668:head# 2024-04-20T17:27:43.954 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2668f5c:::benchmark_data_smithi171_122772_object88409:head# 2024-04-20T17:27:43.954 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2680544:::benchmark_data_smithi171_122772_object90146:head# 2024-04-20T17:27:43.954 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a269b310:::benchmark_data_smithi171_122772_object128877:head# 2024-04-20T17:27:43.955 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a26a7984:::benchmark_data_smithi171_122772_object96579:head# 2024-04-20T17:27:43.956 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a26ac87b:::benchmark_data_smithi171_122772_object157180:head# 2024-04-20T17:27:43.956 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a26c0416:::benchmark_data_smithi171_122772_object156729:head# 2024-04-20T17:27:43.957 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a26d30bd:::benchmark_data_smithi171_122772_object98510:head# 2024-04-20T17:27:43.957 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a26d8369:::benchmark_data_smithi171_122772_object163409:head# 2024-04-20T17:27:43.957 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a26ea1d2:::benchmark_data_smithi171_122772_object166972:head# 2024-04-20T17:27:43.958 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a26f6cfb:::benchmark_data_smithi171_122772_object78202:head# 2024-04-20T17:27:43.958 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a26f821e:::benchmark_data_smithi171_122772_object147203:head# 2024-04-20T17:27:43.959 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a26f8d35:::benchmark_data_smithi171_122772_object163141:head# 2024-04-20T17:27:43.959 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a26fba7d:::benchmark_data_smithi171_122772_object95874:head# 2024-04-20T17:27:43.960 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2710a2c:::benchmark_data_smithi171_122772_object102162:head# 2024-04-20T17:27:43.960 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2725415:::benchmark_data_smithi171_122772_object159881:head# 2024-04-20T17:27:43.961 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2729e61:::benchmark_data_smithi171_122772_object125873:head# 2024-04-20T17:27:43.961 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a272aa1a:::benchmark_data_smithi171_122772_object103715:head# 2024-04-20T17:27:43.962 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a272cfb6:::benchmark_data_smithi171_122772_object162339:head# 2024-04-20T17:27:43.962 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a273710a:::benchmark_data_smithi171_122772_object123109:head# 2024-04-20T17:27:43.963 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2737b52:::benchmark_data_smithi171_122772_object150955:head# 2024-04-20T17:27:43.964 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2739b94:::benchmark_data_smithi171_122772_object122886:head# 2024-04-20T17:27:43.964 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a274872a:::benchmark_data_smithi171_122772_object105937:head# 2024-04-20T17:27:43.964 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a275931b:::benchmark_data_smithi171_122772_object127680:head# 2024-04-20T17:27:43.965 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2760ccc:::benchmark_data_smithi171_122772_object140282:head# 2024-04-20T17:27:43.965 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a27626d5:::benchmark_data_smithi171_122772_object114740:head# 2024-04-20T17:27:43.966 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a27640de:::benchmark_data_smithi171_122772_object144729:head# 2024-04-20T17:27:43.967 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2777237:::benchmark_data_smithi171_122772_object119403:head# 2024-04-20T17:27:43.967 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a279e383:::benchmark_data_smithi171_122772_object107174:head# 2024-04-20T17:27:43.968 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a27a751c:::benchmark_data_smithi171_122772_object134747:head# 2024-04-20T17:27:43.968 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a27aa8ba:::benchmark_data_smithi171_122772_object79383:head# 2024-04-20T17:27:43.969 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a27b020c:::benchmark_data_smithi171_122772_object162815:head# 2024-04-20T17:27:43.969 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a27b66f5:::benchmark_data_smithi171_122772_object96347:head# 2024-04-20T17:27:43.970 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a27bc5b3:::benchmark_data_smithi171_122772_object144241:head# 2024-04-20T17:27:43.970 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a27c25c3:::benchmark_data_smithi171_122772_object120333:head# 2024-04-20T17:27:43.971 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a27e6350:::benchmark_data_smithi171_122772_object123869:head# 2024-04-20T17:27:43.972 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a27e6930:::benchmark_data_smithi171_122772_object91772:head# 2024-04-20T17:27:43.972 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a27e79c9:::benchmark_data_smithi171_122772_object129824:head# 2024-04-20T17:27:43.972 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a280e32b:::benchmark_data_smithi171_122772_object144251:head# 2024-04-20T17:27:43.973 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a280fe35:::benchmark_data_smithi171_122772_object117229:head# 2024-04-20T17:27:43.973 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2816ff7:::benchmark_data_smithi171_122772_object125429:head# 2024-04-20T17:27:43.974 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a282236d:::benchmark_data_smithi171_122772_object106249:head# 2024-04-20T17:27:43.974 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2838209:::benchmark_data_smithi171_122772_object118750:head# 2024-04-20T17:27:43.975 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2843f7b:::benchmark_data_smithi171_122772_object161116:head# 2024-04-20T17:27:43.975 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a284640e:::benchmark_data_smithi171_122772_object167007:head# 2024-04-20T17:27:43.976 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2852f1f:::benchmark_data_smithi171_122772_object95527:head# 2024-04-20T17:27:43.976 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2859ae0:::benchmark_data_smithi171_122772_object164884:head# 2024-04-20T17:27:43.977 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a28789f2:::benchmark_data_smithi171_122772_object120354:head# 2024-04-20T17:27:43.977 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a287b7a6:::benchmark_data_smithi171_122772_object90193:head# 2024-04-20T17:27:43.978 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a287c002:::benchmark_data_smithi171_122772_object108404:head# 2024-04-20T17:27:43.978 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2886a12:::benchmark_data_smithi171_122772_object118454:head# 2024-04-20T17:27:43.979 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a288c170:::benchmark_data_smithi171_122772_object162348:head# 2024-04-20T17:27:43.979 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2895173:::benchmark_data_smithi171_122772_object146122:head# 2024-04-20T17:27:43.979 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:27:43.981 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a28a98c6:::benchmark_data_smithi171_122772_object152094:head# 2024-04-20T17:27:43.981 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a28b1829:::benchmark_data_smithi171_122772_object108993:head# 2024-04-20T17:27:43.982 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a28cb707:::benchmark_data_smithi171_122772_object110438:head# 2024-04-20T17:27:43.982 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a28dbd47:::benchmark_data_smithi171_122772_object149048:head# 2024-04-20T17:27:43.984 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a28ded95:::benchmark_data_smithi171_122772_object80517:head# 2024-04-20T17:27:43.984 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a28e592c:::benchmark_data_smithi171_122772_object124452:head# 2024-04-20T17:27:43.984 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a28ef858:::benchmark_data_smithi171_122772_object147942:head# 2024-04-20T17:27:43.984 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a290d3aa:::benchmark_data_smithi171_122772_object159249:head# 2024-04-20T17:27:43.985 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a29153ad:::benchmark_data_smithi171_122772_object165082:head# 2024-04-20T17:27:43.985 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2930b51:::benchmark_data_smithi171_122772_object148151:head# 2024-04-20T17:27:43.985 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2930eef:::benchmark_data_smithi171_122772_object124383:head# 2024-04-20T17:27:43.986 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a293b5c0:::benchmark_data_smithi171_122772_object100664:head# 2024-04-20T17:27:43.986 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2943315:::benchmark_data_smithi171_122772_object118924:head# 2024-04-20T17:27:43.986 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2962c94:::benchmark_data_smithi171_122772_object144051:head# 2024-04-20T17:27:43.987 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2966471:::benchmark_data_smithi171_122772_object100273:head# 2024-04-20T17:27:43.988 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a296fa8f:::benchmark_data_smithi171_122772_object149979:head# 2024-04-20T17:27:43.988 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2981c92:::benchmark_data_smithi171_122772_object148232:head# 2024-04-20T17:27:43.989 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a299649b:::benchmark_data_smithi171_122772_object84403:head# 2024-04-20T17:27:43.990 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a299b4ab:::benchmark_data_smithi171_122772_object135960:head# 2024-04-20T17:27:43.990 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a29a3360:::benchmark_data_smithi171_122772_object82456:head# 2024-04-20T17:27:43.990 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a29da1ff:::benchmark_data_smithi171_122772_object134369:head# 2024-04-20T17:27:43.991 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a29e62a5:::benchmark_data_smithi171_122772_object151335:head# 2024-04-20T17:27:43.992 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a29ea6d2:::benchmark_data_smithi171_122772_object124230:head# 2024-04-20T17:27:43.992 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a29ed4a6:::benchmark_data_smithi171_122772_object126693:head# 2024-04-20T17:27:43.993 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a29f2dea:::benchmark_data_smithi171_122772_object113387:head# 2024-04-20T17:27:43.993 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a29f6906:::benchmark_data_smithi171_122772_object90063:head# 2024-04-20T17:27:43.994 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2a20c2c:::benchmark_data_smithi171_122772_object161986:head# 2024-04-20T17:27:43.994 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2a250e6:::benchmark_data_smithi171_122772_object152904:head# 2024-04-20T17:27:43.995 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2a39416:::benchmark_data_smithi171_122772_object135234:head# 2024-04-20T17:27:43.996 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2a4568d:::benchmark_data_smithi171_122772_object100656:head# 2024-04-20T17:27:43.996 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2a4e4e9:::benchmark_data_smithi171_122772_object93906:head# 2024-04-20T17:27:43.997 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2a6f13f:::benchmark_data_smithi171_122772_object139188:head# 2024-04-20T17:27:43.997 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2a7e5cb:::benchmark_data_smithi171_122772_object168502:head# 2024-04-20T17:27:43.997 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2a95723:::benchmark_data_smithi171_122772_object125255:head# 2024-04-20T17:27:43.998 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2a9e28e:::benchmark_data_smithi171_122772_object104228:head# 2024-04-20T17:27:43.998 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2ab2cf8:::benchmark_data_smithi171_122772_object128758:head# 2024-04-20T17:27:43.999 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2ab30e5:::benchmark_data_smithi171_122772_object158463:head# 2024-04-20T17:27:43.999 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2ab37b5:::benchmark_data_smithi171_122772_object117053:head# 2024-04-20T17:27:44.000 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2ab3f33:::benchmark_data_smithi171_122772_object112388:head# 2024-04-20T17:27:44.001 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2ab60ac:::benchmark_data_smithi171_122772_object114138:head# 2024-04-20T17:27:44.001 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2ac9d1c:::benchmark_data_smithi171_122772_object118601:head# 2024-04-20T17:27:44.001 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2acb0f6:::benchmark_data_smithi171_122772_object96975:head# 2024-04-20T17:27:44.002 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2ad8935:::benchmark_data_smithi171_122772_object78021:head# 2024-04-20T17:27:44.003 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2ae75f8:::benchmark_data_smithi171_122772_object95728:head# 2024-04-20T17:27:44.003 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2aff558:::benchmark_data_smithi171_122772_object156216:head# 2024-04-20T17:27:44.003 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2affd63:::benchmark_data_smithi171_122772_object109459:head# 2024-04-20T17:27:44.004 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2b0132e:::benchmark_data_smithi171_122772_object114595:head# 2024-04-20T17:27:44.005 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2b167e7:::benchmark_data_smithi171_122772_object165990:head# 2024-04-20T17:27:44.005 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2b226b0:::benchmark_data_smithi171_122772_object127272:head# 2024-04-20T17:27:44.005 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2b2b4ae:::benchmark_data_smithi171_122772_object105312:head# 2024-04-20T17:27:44.006 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2b3326e:::benchmark_data_smithi171_122772_object80937:head# 2024-04-20T17:27:44.007 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2b370e8:::benchmark_data_smithi171_122772_object140459:head# 2024-04-20T17:27:44.007 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2b373de:::benchmark_data_smithi171_122772_object154273:head# 2024-04-20T17:27:44.007 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2b4e5dd:::benchmark_data_smithi171_122772_object100741:head# 2024-04-20T17:27:44.008 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2b5a1c3:::benchmark_data_smithi171_122772_object120108:head# 2024-04-20T17:27:44.008 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2b5fd0c:::benchmark_data_smithi171_122772_object86081:head# 2024-04-20T17:27:44.009 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2b62334:::benchmark_data_smithi171_122772_object126579:head# 2024-04-20T17:27:44.010 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2b62d01:::benchmark_data_smithi171_122772_object144497:head# 2024-04-20T17:27:44.011 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2b65cda:::benchmark_data_smithi171_122772_object80386:head# 2024-04-20T17:27:44.011 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2b685fe:::benchmark_data_smithi171_122772_object86192:head# 2024-04-20T17:27:44.011 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2b6d74c:::benchmark_data_smithi171_122772_object150629:head# 2024-04-20T17:27:44.011 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:44.012 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2b73588:::benchmark_data_smithi171_122772_object123462:head# 2024-04-20T17:27:44.012 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2b73dd2:::benchmark_data_smithi171_122772_object165015:head# 2024-04-20T17:27:44.012 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2b81d9c:::benchmark_data_smithi171_122772_object125400:head# 2024-04-20T17:27:44.013 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2b86508:::benchmark_data_smithi171_122772_object78848:head# 2024-04-20T17:27:44.014 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2b8bace:::benchmark_data_smithi171_122772_object80206:head# 2024-04-20T17:27:44.014 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2b9faa3:::benchmark_data_smithi171_122772_object104853:head# 2024-04-20T17:27:44.015 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2ba7f7e:::benchmark_data_smithi171_122772_object129255:head# 2024-04-20T17:27:44.015 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2ba8080:::benchmark_data_smithi171_122772_object106965:head# 2024-04-20T17:27:44.016 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2badd09:::benchmark_data_smithi171_122772_object160362:head# 2024-04-20T17:27:44.016 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2bdd271:::benchmark_data_smithi171_122772_object158187:head# 2024-04-20T17:27:44.017 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2beb2b2:::benchmark_data_smithi171_122772_object85785:head# 2024-04-20T17:27:44.017 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2bee8de:::benchmark_data_smithi171_122772_object142553:head# 2024-04-20T17:27:44.018 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2bf30fc:::benchmark_data_smithi171_122772_object88366:head# 2024-04-20T17:27:44.018 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2bf76b4:::benchmark_data_smithi171_122772_object135925:head# 2024-04-20T17:27:44.019 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2c09f0c:::benchmark_data_smithi171_122772_object117031:head# 2024-04-20T17:27:44.019 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2c0d964:::benchmark_data_smithi171_122772_object125943:head# 2024-04-20T17:27:44.020 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2c15e98:::benchmark_data_smithi171_122772_object169091:head# 2024-04-20T17:27:44.020 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2c1b3bc:::benchmark_data_smithi171_122772_object148567:head# 2024-04-20T17:27:44.021 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2c249d2:::benchmark_data_smithi171_122772_object146437:head# 2024-04-20T17:27:44.021 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2c29140:::benchmark_data_smithi171_122772_object94574:head# 2024-04-20T17:27:44.022 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2c2dae3:::benchmark_data_smithi171_122772_object156419:head# 2024-04-20T17:27:44.022 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2c34151:::benchmark_data_smithi171_122772_object158975:head# 2024-04-20T17:27:44.023 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2c58f52:::benchmark_data_smithi171_122772_object92920:head# 2024-04-20T17:27:44.023 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2c60794:::benchmark_data_smithi171_122772_object149769:head# 2024-04-20T17:27:44.024 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2c66ebb:::benchmark_data_smithi171_122772_object163807:head# 2024-04-20T17:27:44.024 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2c6c5ce:::benchmark_data_smithi171_122772_object157713:head# 2024-04-20T17:27:44.025 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2c7ed12:::benchmark_data_smithi171_122772_object111185:head# 2024-04-20T17:27:44.025 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2c97032:::benchmark_data_smithi171_122772_object163573:head# 2024-04-20T17:27:44.026 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2ca126a:::benchmark_data_smithi171_122772_object111137:head# 2024-04-20T17:27:44.026 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2ca5271:::benchmark_data_smithi171_122772_object117853:head# 2024-04-20T17:27:44.027 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2caca2b:::benchmark_data_smithi171_122772_object132781:head# 2024-04-20T17:27:44.027 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2cae3f9:::benchmark_data_smithi171_122772_object165910:head# 2024-04-20T17:27:44.027 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2cb3280:::benchmark_data_smithi171_122772_object155504:head# 2024-04-20T17:27:44.028 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2cb3911:::benchmark_data_smithi171_122772_object81853:head# 2024-04-20T17:27:44.029 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2cbc4e8:::benchmark_data_smithi171_122772_object110381:head# 2024-04-20T17:27:44.029 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2cbde9f:::benchmark_data_smithi171_122772_object164661:head# 2024-04-20T17:27:44.030 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2cbe117:::benchmark_data_smithi171_122772_object157178:head# 2024-04-20T17:27:44.030 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2cdc138:::benchmark_data_smithi171_122772_object95005:head# 2024-04-20T17:27:44.031 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2cde073:::benchmark_data_smithi171_122772_object152101:head# 2024-04-20T17:27:44.031 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2cdf74b:::benchmark_data_smithi171_122772_object96407:head# 2024-04-20T17:27:44.032 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2ce8d80:::benchmark_data_smithi171_122772_object102776:head# 2024-04-20T17:27:44.032 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2cf19c3:::benchmark_data_smithi171_122772_object111631:head# 2024-04-20T17:27:44.033 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2cf88ff:::benchmark_data_smithi171_122772_object119805:head# 2024-04-20T17:27:44.033 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2d02124:::benchmark_data_smithi171_122772_object123107:head# 2024-04-20T17:27:44.034 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2d04577:::benchmark_data_smithi171_122772_object84833:head# 2024-04-20T17:27:44.034 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2d106cd:::benchmark_data_smithi171_122772_object152559:head# 2024-04-20T17:27:44.035 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2d174cb:::benchmark_data_smithi171_122772_object106794:head# 2024-04-20T17:27:44.036 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2d23d78:::benchmark_data_smithi171_122772_object155045:head# 2024-04-20T17:27:44.036 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2d28119:::benchmark_data_smithi171_122772_object160307:head# 2024-04-20T17:27:44.036 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2d4603b:::benchmark_data_smithi171_122772_object138650:head# 2024-04-20T17:27:44.037 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2d46c77:::benchmark_data_smithi171_122772_object169304:head# 2024-04-20T17:27:44.038 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2d55740:::benchmark_data_smithi171_122772_object140406:head# 2024-04-20T17:27:44.038 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2d56273:::benchmark_data_smithi171_122772_object114486:head# 2024-04-20T17:27:44.039 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2d65891:::benchmark_data_smithi171_122772_object90238:head# 2024-04-20T17:27:44.039 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2d6ceac:::benchmark_data_smithi171_122772_object154711:head# 2024-04-20T17:27:44.040 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2d7d988:::benchmark_data_smithi171_122772_object164145:head# 2024-04-20T17:27:44.040 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2d8b208:::benchmark_data_smithi171_122772_object119135:head# 2024-04-20T17:27:44.041 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2d936ad:::benchmark_data_smithi171_122772_object133959:head# 2024-04-20T17:27:44.041 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2da1749:::benchmark_data_smithi171_122772_object94681:head# 2024-04-20T17:27:44.042 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2da94cd:::benchmark_data_smithi171_122772_object106675:head# 2024-04-20T17:27:44.042 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2db4e43:::benchmark_data_smithi171_122772_object146837:head# 2024-04-20T17:27:44.043 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2db5024:::benchmark_data_smithi171_122772_object96939:head# 2024-04-20T17:27:44.043 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2db72b8:::benchmark_data_smithi171_122772_object165731:head# 2024-04-20T17:27:44.044 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2dc12da:::benchmark_data_smithi171_122772_object137369:head# 2024-04-20T17:27:44.044 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2dc88d5:::benchmark_data_smithi171_122772_object83168:head# 2024-04-20T17:27:44.045 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2dca4de:::benchmark_data_smithi171_122772_object140662:head# 2024-04-20T17:27:44.045 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2decbe9:::benchmark_data_smithi171_122772_object153544:head# 2024-04-20T17:27:44.046 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2def62c:::benchmark_data_smithi171_122772_object88113:head# 2024-04-20T17:27:44.046 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2dfcb2f:::benchmark_data_smithi171_122772_object97834:head# 2024-04-20T17:27:44.047 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2e01d50:::benchmark_data_smithi171_122772_object141146:head# 2024-04-20T17:27:44.047 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2e0e2a1:::benchmark_data_smithi171_122772_object152653:head# 2024-04-20T17:27:44.048 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2e10fda:::benchmark_data_smithi171_122772_object148813:head# 2024-04-20T17:27:44.048 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2e1b595:::benchmark_data_smithi171_122772_object152175:head# 2024-04-20T17:27:44.049 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2e31377:::benchmark_data_smithi171_122772_object144730:head# 2024-04-20T17:27:44.049 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2e33b58:::benchmark_data_smithi171_122772_object155392:head# 2024-04-20T17:27:44.050 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2e4439a:::benchmark_data_smithi171_122772_object130648:head# 2024-04-20T17:27:44.050 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2e602a1:::benchmark_data_smithi171_122772_object88373:head# 2024-04-20T17:27:44.051 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2e68abe:::benchmark_data_smithi171_122772_object115408:head# 2024-04-20T17:27:44.051 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2e718c1:::benchmark_data_smithi171_122772_object117081:head# 2024-04-20T17:27:44.052 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2e80cba:::benchmark_data_smithi171_122772_object95011:head# 2024-04-20T17:27:44.052 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2e81e7a:::benchmark_data_smithi171_122772_object166087:head# 2024-04-20T17:27:44.053 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2e83916:::benchmark_data_smithi171_122772_object90132:head# 2024-04-20T17:27:44.053 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2e94235:::benchmark_data_smithi171_122772_object122568:head# 2024-04-20T17:27:44.054 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2e9d504:::benchmark_data_smithi171_122772_object137002:head# 2024-04-20T17:27:44.054 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2ea7609:::benchmark_data_smithi171_122772_object153076:head# 2024-04-20T17:27:44.055 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2ebd1e3:::benchmark_data_smithi171_122772_object147984:head# 2024-04-20T17:27:44.056 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2ec1a21:::benchmark_data_smithi171_122772_object84515:head# 2024-04-20T17:27:44.056 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2eca9df:::benchmark_data_smithi171_122772_object140830:head# 2024-04-20T17:27:44.057 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2ee740f:::benchmark_data_smithi171_122772_object132045:head# 2024-04-20T17:27:44.057 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2ee8515:::benchmark_data_smithi171_122772_object81939:head# 2024-04-20T17:27:44.058 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2ee9dca:::benchmark_data_smithi171_122772_object115188:head# 2024-04-20T17:27:44.058 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2eee8e0:::benchmark_data_smithi171_122772_object151891:head# 2024-04-20T17:27:44.059 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2efa3e5:::benchmark_data_smithi171_122772_object106386:head# 2024-04-20T17:27:44.059 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2efb3b5:::benchmark_data_smithi171_122772_object130861:head# 2024-04-20T17:27:44.060 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2f0dcdf:::benchmark_data_smithi171_122772_object133308:head# 2024-04-20T17:27:44.060 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2f121e9:::benchmark_data_smithi171_122772_object122069:head# 2024-04-20T17:27:44.061 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2f27b59:::benchmark_data_smithi171_122772_object96483:head# 2024-04-20T17:27:44.061 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2f3cb49:::benchmark_data_smithi171_122772_object169425:head# 2024-04-20T17:27:44.061 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2f3ce75:::benchmark_data_smithi171_122772_object120768:head# 2024-04-20T17:27:44.062 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2f470a2:::benchmark_data_smithi171_122772_object159072:head# 2024-04-20T17:27:44.062 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2f58c98:::benchmark_data_smithi171_122772_object132989:head# 2024-04-20T17:27:44.063 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2f6f3a2:::benchmark_data_smithi171_122772_object131173:head# 2024-04-20T17:27:44.063 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2f95c29:::benchmark_data_smithi171_122772_object88149:head# 2024-04-20T17:27:44.064 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2f9b1d6:::benchmark_data_smithi171_122772_object132719:head# 2024-04-20T17:27:44.064 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2fa2ff9:::benchmark_data_smithi171_122772_object146897:head# 2024-04-20T17:27:44.065 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2fb06d4:::benchmark_data_smithi171_122772_object133264:head# 2024-04-20T17:27:44.065 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a2fb259f:::benchmark_data_smithi171_122772_object153450:head# 2024-04-20T17:27:44.066 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3010b7d:::benchmark_data_smithi171_122772_object89337:head# 2024-04-20T17:27:44.066 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3016617:::benchmark_data_smithi171_122772_object160980:head# 2024-04-20T17:27:44.067 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3027d9f:::benchmark_data_smithi171_122772_object78144:head# 2024-04-20T17:27:44.067 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a30393d1:::benchmark_data_smithi171_122772_object101636:head# 2024-04-20T17:27:44.068 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3042484:::benchmark_data_smithi171_122772_object104253:head# 2024-04-20T17:27:44.068 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a304581e:::benchmark_data_smithi171_122772_object127445:head# 2024-04-20T17:27:44.069 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3059c38:::benchmark_data_smithi171_122772_object118850:head# 2024-04-20T17:27:44.069 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3059e0b:::benchmark_data_smithi171_122772_object86364:head# 2024-04-20T17:27:44.070 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a305b595:::benchmark_data_smithi171_122772_object142964:head# 2024-04-20T17:27:44.070 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3061661:::benchmark_data_smithi171_122772_object166066:head# 2024-04-20T17:27:44.071 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3062cb5:::benchmark_data_smithi171_122772_object90022:head# 2024-04-20T17:27:44.072 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3084bf5:::benchmark_data_smithi171_122772_object129914:head# 2024-04-20T17:27:44.072 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3085768:::benchmark_data_smithi171_122772_object100777:head# 2024-04-20T17:27:44.073 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a309c9ec:::benchmark_data_smithi171_122772_object139444:head# 2024-04-20T17:27:44.073 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a30a2bc1:::benchmark_data_smithi171_122772_object118948:head# 2024-04-20T17:27:44.074 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a30b7f25:::benchmark_data_smithi171_122772_object103195:head# 2024-04-20T17:27:44.074 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a30ca410:::benchmark_data_smithi171_122772_object84836:head# 2024-04-20T17:27:44.075 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a30cae30:::benchmark_data_smithi171_122772_object169047:head# 2024-04-20T17:27:44.075 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a30cb4b0:::benchmark_data_smithi171_122772_object129494:head# 2024-04-20T17:27:44.076 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a30cceb1:::benchmark_data_smithi171_122772_object114707:head# 2024-04-20T17:27:44.076 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a30e3189:::benchmark_data_smithi171_122772_object127015:head# 2024-04-20T17:27:44.077 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a30e3c4d:::benchmark_data_smithi171_122772_object132888:head# 2024-04-20T17:27:44.077 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a30e4963:::benchmark_data_smithi171_122772_object115060:head# 2024-04-20T17:27:44.077 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a30f71cf:::benchmark_data_smithi171_122772_object129667:head# 2024-04-20T17:27:44.078 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a30f9dd5:::benchmark_data_smithi171_122772_object130087:head# 2024-04-20T17:27:44.078 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3121e37:::benchmark_data_smithi171_122772_object143864:head# 2024-04-20T17:27:44.079 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a31269fa:::benchmark_data_smithi171_122772_object147436:head# 2024-04-20T17:27:44.079 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3126b16:::benchmark_data_smithi171_122772_object104363:head# 2024-04-20T17:27:44.080 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3156b7b:::benchmark_data_smithi171_122772_object100317:head# 2024-04-20T17:27:44.080 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a31690a8:::benchmark_data_smithi171_122772_object133707:head# 2024-04-20T17:27:44.080 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3183d4c:::benchmark_data_smithi171_122772_object84904:head# 2024-04-20T17:27:44.081 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a319e0cc:::benchmark_data_smithi171_122772_object142216:head# 2024-04-20T17:27:44.081 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a31a221e:::benchmark_data_smithi171_122772_object88954:head# 2024-04-20T17:27:44.082 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a31b93fe:::benchmark_data_smithi171_122772_object122675:head# 2024-04-20T17:27:44.083 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a31bc482:::benchmark_data_smithi171_122772_object110032:head# 2024-04-20T17:27:44.083 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a31bea5c:::benchmark_data_smithi171_122772_object83634:head# 2024-04-20T17:27:44.084 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a31c9de7:::benchmark_data_smithi171_122772_object135703:head# 2024-04-20T17:27:44.084 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a31d6f4c:::benchmark_data_smithi171_122772_object106401:head# 2024-04-20T17:27:44.085 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a31ebb27:::benchmark_data_smithi171_122772_object95317:head# 2024-04-20T17:27:44.085 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a31f11a7:::benchmark_data_smithi171_122772_object97402:head# 2024-04-20T17:27:44.086 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3210dce:::benchmark_data_smithi171_122772_object140956:head# 2024-04-20T17:27:44.087 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3215609:::benchmark_data_smithi171_122772_object82689:head# 2024-04-20T17:27:44.087 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a322a2ee:::benchmark_data_smithi171_122772_object96641:head# 2024-04-20T17:27:44.088 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a322c5a1:::benchmark_data_smithi171_122772_object137179:head# 2024-04-20T17:27:44.088 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3231cdd:::benchmark_data_smithi171_122772_object114891:head# 2024-04-20T17:27:44.089 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a32422f8:::benchmark_data_smithi171_122772_object131621:head# 2024-04-20T17:27:44.089 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3242bfc:::benchmark_data_smithi171_122772_object89129:head# 2024-04-20T17:27:44.089 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a325abb4:::benchmark_data_smithi171_122772_object142834:head# 2024-04-20T17:27:44.090 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3262f85:::benchmark_data_smithi171_122772_object163583:head# 2024-04-20T17:27:44.090 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3263477:::benchmark_data_smithi171_122772_object150682:head# 2024-04-20T17:27:44.091 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a328e0ca:::benchmark_data_smithi171_122772_object159041:head# 2024-04-20T17:27:44.091 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3292881:::benchmark_data_smithi171_122772_object123314:head# 2024-04-20T17:27:44.092 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a32a7a9c:::benchmark_data_smithi171_122772_object156301:head# 2024-04-20T17:27:44.092 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a32c023e:::benchmark_data_smithi171_122772_object121185:head# 2024-04-20T17:27:44.093 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a32d40c5:::benchmark_data_smithi171_122772_object121511:head# 2024-04-20T17:27:44.093 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a32d520c:::benchmark_data_smithi171_122772_object91918:head# 2024-04-20T17:27:44.094 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a32e6491:::benchmark_data_smithi171_122772_object136668:head# 2024-04-20T17:27:44.094 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a32f0002:::benchmark_data_smithi171_122772_object122509:head# 2024-04-20T17:27:44.095 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a32fe7c6:::benchmark_data_smithi171_122772_object169511:head# 2024-04-20T17:27:44.095 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3300682:::benchmark_data_smithi171_122772_object169831:head# 2024-04-20T17:27:44.095 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a330d119:::benchmark_data_smithi171_122772_object140214:head# 2024-04-20T17:27:44.096 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a330ecd2:::benchmark_data_smithi171_122772_object131380:head# 2024-04-20T17:27:44.096 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a331a230:::benchmark_data_smithi171_122772_object138363:head# 2024-04-20T17:27:44.097 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a331a566:::benchmark_data_smithi171_122772_object150549:head# 2024-04-20T17:27:44.097 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a331fe1e:::benchmark_data_smithi171_122772_object143795:head# 2024-04-20T17:27:44.098 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3328418:::benchmark_data_smithi171_122772_object107183:head# 2024-04-20T17:27:44.098 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a332e769:::benchmark_data_smithi171_122772_object78140:head# 2024-04-20T17:27:44.098 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3343bf7:::benchmark_data_smithi171_122772_object87573:head# 2024-04-20T17:27:44.099 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3347e86:::benchmark_data_smithi171_122772_object150121:head# 2024-04-20T17:27:44.099 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a33526dc:::benchmark_data_smithi171_122772_object162107:head# 2024-04-20T17:27:44.100 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a335299b:::benchmark_data_smithi171_122772_object141789:head# 2024-04-20T17:27:44.100 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a335a389:::benchmark_data_smithi171_122772_object157408:head# 2024-04-20T17:27:44.101 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a336c239:::benchmark_data_smithi171_122772_object132676:head# 2024-04-20T17:27:44.101 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a336e2e8:::benchmark_data_smithi171_122772_object156685:head# 2024-04-20T17:27:44.102 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3374df8:::benchmark_data_smithi171_122772_object165777:head# 2024-04-20T17:27:44.102 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3375fa7:::benchmark_data_smithi171_122772_object89594:head# 2024-04-20T17:27:44.103 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a337bc3a:::benchmark_data_smithi171_122772_object133695:head# 2024-04-20T17:27:44.103 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a338caaf:::benchmark_data_smithi171_122772_object153367:head# 2024-04-20T17:27:44.104 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3395adb:::benchmark_data_smithi171_122772_object155614:head# 2024-04-20T17:27:44.104 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a33a4239:::benchmark_data_smithi171_122772_object163256:head# 2024-04-20T17:27:44.105 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a33aec41:::benchmark_data_smithi171_122772_object124201:head# 2024-04-20T17:27:44.105 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a33c3d55:::benchmark_data_smithi171_122772_object93384:head# 2024-04-20T17:27:44.106 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a33d5f6e:::benchmark_data_smithi171_122772_object154218:head# 2024-04-20T17:27:44.106 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a33e1453:::benchmark_data_smithi171_122772_object117423:head# 2024-04-20T17:27:44.106 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a33f929f:::benchmark_data_smithi171_122772_object125104:head# 2024-04-20T17:27:44.107 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a34014b9:::benchmark_data_smithi171_122772_object94385:head# 2024-04-20T17:27:44.107 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3410e6d:::benchmark_data_smithi171_122772_object114444:head# 2024-04-20T17:27:44.107 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a341ca6e:::benchmark_data_smithi171_122772_object80362:head# 2024-04-20T17:27:44.108 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3427b2b:::benchmark_data_smithi171_122772_object109725:head# 2024-04-20T17:27:44.109 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3429f8e:::benchmark_data_smithi171_122772_object166593:head# 2024-04-20T17:27:44.109 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a342f3d1:::benchmark_data_smithi171_122772_object163591:head# 2024-04-20T17:27:44.110 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a342ff1d:::benchmark_data_smithi171_122772_object131961:head# 2024-04-20T17:27:44.110 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3430e8f:::benchmark_data_smithi171_122772_object83177:head# 2024-04-20T17:27:44.111 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a343c3a6:::benchmark_data_smithi171_122772_object82036:head# 2024-04-20T17:27:44.111 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a34433b2:::benchmark_data_smithi171_122772_object139113:head# 2024-04-20T17:27:44.112 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a344d497:::benchmark_data_smithi171_122772_object127791:head# 2024-04-20T17:27:44.112 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a344db57:::benchmark_data_smithi171_122772_object123023:head# 2024-04-20T17:27:44.113 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3462a73:::benchmark_data_smithi171_122772_object99465:head# 2024-04-20T17:27:44.113 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3466704:::benchmark_data_smithi171_122772_object112336:head# 2024-04-20T17:27:44.114 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a34786e7:::benchmark_data_smithi171_122772_object152457:head# 2024-04-20T17:27:44.114 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a347d889:::benchmark_data_smithi171_122772_object78551:head# 2024-04-20T17:27:44.114 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3482cc3:::benchmark_data_smithi171_122772_object104174:head# 2024-04-20T17:27:44.115 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3488f3d:::benchmark_data_smithi171_122772_object122306:head# 2024-04-20T17:27:44.115 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3490fc5:::benchmark_data_smithi171_122772_object92450:head# 2024-04-20T17:27:44.116 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3493ade:::benchmark_data_smithi171_122772_object167875:head# 2024-04-20T17:27:44.116 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a34993bd:::benchmark_data_smithi171_122772_object119098:head# 2024-04-20T17:27:44.117 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a34b3c1e:::benchmark_data_smithi171_122772_object154840:head# 2024-04-20T17:27:44.117 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a34b5303:::benchmark_data_smithi171_122772_object169985:head# 2024-04-20T17:27:44.118 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a34c4750:::benchmark_data_smithi171_122772_object116421:head# 2024-04-20T17:27:44.119 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a34c52b0:::benchmark_data_smithi171_122772_object109370:head# 2024-04-20T17:27:44.119 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a34c8912:::benchmark_data_smithi171_122772_object115432:head# 2024-04-20T17:27:44.120 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a34cf842:::benchmark_data_smithi171_122772_object107536:head# 2024-04-20T17:27:44.120 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a34e635c:::benchmark_data_smithi171_122772_object144353:head# 2024-04-20T17:27:44.120 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a34e9946:::benchmark_data_smithi171_122772_object157152:head# 2024-04-20T17:27:44.121 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a34edceb:::benchmark_data_smithi171_122772_object155723:head# 2024-04-20T17:27:44.121 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a34f5f00:::benchmark_data_smithi171_122772_object111427:head# 2024-04-20T17:27:44.122 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a350d0d0:::benchmark_data_smithi171_122772_object153426:head# 2024-04-20T17:27:44.122 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3519ffa:::benchmark_data_smithi171_122772_object165573:head# 2024-04-20T17:27:44.123 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a351bd9b:::benchmark_data_smithi171_122772_object146886:head# 2024-04-20T17:27:44.123 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3529f26:::benchmark_data_smithi171_122772_object168159:head# 2024-04-20T17:27:44.124 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a35381e6:::benchmark_data_smithi171_122772_object158028:head# 2024-04-20T17:27:44.124 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3539b16:::benchmark_data_smithi171_122772_object78149:head# 2024-04-20T17:27:44.124 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a353c746:::benchmark_data_smithi171_122772_object166160:head# 2024-04-20T17:27:44.125 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a353da2e:::benchmark_data_smithi171_122772_object128564:head# 2024-04-20T17:27:44.125 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a354591f:::benchmark_data_smithi171_122772_object134179:head# 2024-04-20T17:27:44.126 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a35596be:::benchmark_data_smithi171_122772_object119683:head# 2024-04-20T17:27:44.126 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a355eaa9:::benchmark_data_smithi171_122772_object118074:head# 2024-04-20T17:27:44.127 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a356118f:::benchmark_data_smithi171_122772_object168609:head# 2024-04-20T17:27:44.127 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3579e24:::benchmark_data_smithi171_122772_object128610:head# 2024-04-20T17:27:44.128 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a358ac05:::benchmark_data_smithi171_122772_object98017:head# 2024-04-20T17:27:44.128 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a358c8b2:::benchmark_data_smithi171_122772_object97022:head# 2024-04-20T17:27:44.129 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3591536:::benchmark_data_smithi171_122772_object143543:head# 2024-04-20T17:27:44.129 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3595034:::benchmark_data_smithi171_122772_object96468:head# 2024-04-20T17:27:44.130 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a359ff61:::benchmark_data_smithi171_122772_object114650:head# 2024-04-20T17:27:44.130 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a35a66a9:::benchmark_data_smithi171_122772_object159582:head# 2024-04-20T17:27:44.130 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a35c76ef:::benchmark_data_smithi171_122772_object131519:head# 2024-04-20T17:27:44.131 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a35cbafd:::benchmark_data_smithi171_122772_object85058:head# 2024-04-20T17:27:44.131 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a35cd0f7:::benchmark_data_smithi171_122772_object102101:head# 2024-04-20T17:27:44.132 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a35e02c4:::benchmark_data_smithi171_122772_object82800:head# 2024-04-20T17:27:44.132 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a35e5bb0:::benchmark_data_smithi171_122772_object118605:head# 2024-04-20T17:27:44.133 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a35e64d7:::benchmark_data_smithi171_122772_object155697:head# 2024-04-20T17:27:44.133 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a35ec8ab:::benchmark_data_smithi171_122772_object160073:head# 2024-04-20T17:27:44.134 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a35f809e:::benchmark_data_smithi171_122772_object104137:head# 2024-04-20T17:27:44.134 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a35ff39e:::benchmark_data_smithi171_122772_object163213:head# 2024-04-20T17:27:44.135 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a360308f:::benchmark_data_smithi171_122772_object111343:head# 2024-04-20T17:27:44.135 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3608b6b:::benchmark_data_smithi171_122772_object83747:head# 2024-04-20T17:27:44.136 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a360a29f:::benchmark_data_smithi171_122772_object123093:head# 2024-04-20T17:27:44.136 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a361d362:::benchmark_data_smithi171_122772_object163246:head# 2024-04-20T17:27:44.136 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3627143:::benchmark_data_smithi171_122772_object106236:head# 2024-04-20T17:27:44.137 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a362f24f:::benchmark_data_smithi171_122772_object115725:head# 2024-04-20T17:27:44.137 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a36328cb:::benchmark_data_smithi171_122772_object91224:head# 2024-04-20T17:27:44.137 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3633385:::benchmark_data_smithi171_122772_object164834:head# 2024-04-20T17:27:44.138 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3640bc3:::benchmark_data_smithi171_122772_object130299:head# 2024-04-20T17:27:44.138 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a36417e2:::benchmark_data_smithi171_122772_object80060:head# 2024-04-20T17:27:44.139 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a364391e:::benchmark_data_smithi171_122772_object169344:head# 2024-04-20T17:27:44.139 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a365a78b:::benchmark_data_smithi171_122772_object78496:head# 2024-04-20T17:27:44.140 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a365fbee:::benchmark_data_smithi171_122772_object123917:head# 2024-04-20T17:27:44.140 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a366b6ed:::benchmark_data_smithi171_122772_object78059:head# 2024-04-20T17:27:44.141 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a367fb74:::benchmark_data_smithi171_122772_object83552:head# 2024-04-20T17:27:44.141 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a36977aa:::benchmark_data_smithi171_122772_object97643:head# 2024-04-20T17:27:44.142 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a369a1b9:::benchmark_data_smithi171_122772_object94287:head# 2024-04-20T17:27:44.142 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a369b465:::benchmark_data_smithi171_122772_object168141:head# 2024-04-20T17:27:44.143 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a36a58fd:::benchmark_data_smithi171_122772_object155954:head# 2024-04-20T17:27:44.143 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a36baaf5:::benchmark_data_smithi171_122772_object138837:head# 2024-04-20T17:27:44.144 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a36bb465:::benchmark_data_smithi171_122772_object91385:head# 2024-04-20T17:27:44.144 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a36c6f1e:::benchmark_data_smithi171_122772_object114311:head# 2024-04-20T17:27:44.145 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a36e2b9d:::benchmark_data_smithi171_122772_object117488:head# 2024-04-20T17:27:44.145 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a36e6ab3:::benchmark_data_smithi171_122772_object164947:head# 2024-04-20T17:27:44.146 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a370ae7a:::benchmark_data_smithi171_122772_object160430:head# 2024-04-20T17:27:44.146 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3723ae4:::benchmark_data_smithi171_122772_object104641:head# 2024-04-20T17:27:44.146 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a37338e2:::benchmark_data_smithi171_122772_object105960:head# 2024-04-20T17:27:44.147 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3758150:::benchmark_data_smithi171_122772_object152241:head# 2024-04-20T17:27:44.147 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3780a49:::benchmark_data_smithi171_122772_object86155:head# 2024-04-20T17:27:44.148 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a379bf98:::benchmark_data_smithi171_122772_object113161:head# 2024-04-20T17:27:44.148 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a37bac6c:::benchmark_data_smithi171_122772_object129597:head# 2024-04-20T17:27:44.149 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a37e408a:::benchmark_data_smithi171_122772_object129561:head# 2024-04-20T17:27:44.149 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a380763f:::benchmark_data_smithi171_122772_object106958:head# 2024-04-20T17:27:44.150 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a380a492:::benchmark_data_smithi171_122772_object131875:head# 2024-04-20T17:27:44.150 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a380bca5:::benchmark_data_smithi171_122772_object81677:head# 2024-04-20T17:27:44.150 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a380e3fa:::benchmark_data_smithi171_122772_object137706:head# 2024-04-20T17:27:44.151 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a38113eb:::benchmark_data_smithi171_122772_object133480:head# 2024-04-20T17:27:44.152 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3813e4b:::benchmark_data_smithi171_122772_object159634:head# 2024-04-20T17:27:44.152 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3815b34:::benchmark_data_smithi171_122772_object104453:head# 2024-04-20T17:27:44.152 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3815cb4:::benchmark_data_smithi171_122772_object151436:head# 2024-04-20T17:27:44.153 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a38254a6:::benchmark_data_smithi171_122772_object89723:head# 2024-04-20T17:27:44.154 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3836f61:::benchmark_data_smithi171_122772_object118505:head# 2024-04-20T17:27:44.154 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3837d7d:::benchmark_data_smithi171_122772_object166669:head# 2024-04-20T17:27:44.155 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3838524:::benchmark_data_smithi171_122772_object108468:head# 2024-04-20T17:27:44.155 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a38396d3:::benchmark_data_smithi171_122772_object129409:head# 2024-04-20T17:27:44.155 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3840737:::benchmark_data_smithi171_122772_object94585:head# 2024-04-20T17:27:44.156 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3849804:::benchmark_data_smithi171_122772_object111402:head# 2024-04-20T17:27:44.156 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a384da85:::benchmark_data_smithi171_122772_object153457:head# 2024-04-20T17:27:44.157 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3852c3b:::benchmark_data_smithi171_122772_object158682:head# 2024-04-20T17:27:44.157 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3853f65:::benchmark_data_smithi171_122772_object79984:head# 2024-04-20T17:27:44.158 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a386c9a6:::benchmark_data_smithi171_122772_object103598:head# 2024-04-20T17:27:44.158 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3871025:::benchmark_data_smithi171_122772_object161151:head# 2024-04-20T17:27:44.159 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3881911:::benchmark_data_smithi171_122772_object129338:head# 2024-04-20T17:27:44.159 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a388405f:::benchmark_data_smithi171_122772_object85400:head# 2024-04-20T17:27:44.160 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a38874ee:::benchmark_data_smithi171_122772_object106319:head# 2024-04-20T17:27:44.160 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a388d38f:::benchmark_data_smithi171_122772_object115404:head# 2024-04-20T17:27:44.161 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a389426c:::benchmark_data_smithi171_122772_object169202:head# 2024-04-20T17:27:44.162 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a389c76f:::benchmark_data_smithi171_122772_object130513:head# 2024-04-20T17:27:44.162 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a38a50ad:::benchmark_data_smithi171_122772_object78072:head# 2024-04-20T17:27:44.163 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a38b733c:::benchmark_data_smithi171_122772_object107532:head# 2024-04-20T17:27:44.163 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a38c4da3:::benchmark_data_smithi171_122772_object106687:head# 2024-04-20T17:27:44.164 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a38c8b4d:::benchmark_data_smithi171_122772_object123926:head# 2024-04-20T17:27:44.164 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a38ea368:::benchmark_data_smithi171_122772_object86054:head# 2024-04-20T17:27:44.165 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3909b64:::benchmark_data_smithi171_122772_object125181:head# 2024-04-20T17:27:44.165 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a39194d9:::benchmark_data_smithi171_122772_object88238:head# 2024-04-20T17:27:44.166 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a392b4e2:::benchmark_data_smithi171_122772_object103455:head# 2024-04-20T17:27:44.166 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3931822:::benchmark_data_smithi171_122772_object163007:head# 2024-04-20T17:27:44.167 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3934ca8:::benchmark_data_smithi171_122772_object169986:head# 2024-04-20T17:27:44.168 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3956867:::benchmark_data_smithi171_122772_object147650:head# 2024-04-20T17:27:44.169 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3958c9d:::benchmark_data_smithi171_122772_object84330:head# 2024-04-20T17:27:44.169 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3963a21:::benchmark_data_smithi171_122772_object140100:head# 2024-04-20T17:27:44.169 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a398ea2f:::benchmark_data_smithi171_122772_object127465:head# 2024-04-20T17:27:44.170 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3996977:::benchmark_data_smithi171_122772_object108521:head# 2024-04-20T17:27:44.170 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a399728f:::benchmark_data_smithi171_122772_object167276:head# 2024-04-20T17:27:44.171 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a399cd0b:::benchmark_data_smithi171_122772_object95363:head# 2024-04-20T17:27:44.171 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a399e968:::benchmark_data_smithi171_122772_object152936:head# 2024-04-20T17:27:44.172 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a39d5181:::benchmark_data_smithi171_122772_object140013:head# 2024-04-20T17:27:44.172 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a39eefe6:::benchmark_data_smithi171_122772_object95230:head# 2024-04-20T17:27:44.173 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a39f4bf5:::benchmark_data_smithi171_122772_object102976:head# 2024-04-20T17:27:44.173 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3a0bac7:::benchmark_data_smithi171_122772_object109196:head# 2024-04-20T17:27:44.174 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3a0cf25:::benchmark_data_smithi171_122772_object92496:head# 2024-04-20T17:27:44.174 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3a11a0f:::benchmark_data_smithi171_122772_object136095:head# 2024-04-20T17:27:44.174 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3a19bb4:::benchmark_data_smithi171_122772_object146838:head# 2024-04-20T17:27:44.175 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3a1debd:::benchmark_data_smithi171_122772_object159315:head# 2024-04-20T17:27:44.176 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3a30050:::benchmark_data_smithi171_122772_object94922:head# 2024-04-20T17:27:44.176 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3a32b3c:::benchmark_data_smithi171_122772_object89969:head# 2024-04-20T17:27:44.177 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3a44b43:::benchmark_data_smithi171_122772_object145354:head# 2024-04-20T17:27:44.178 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3a64707:::benchmark_data_smithi171_122772_object160679:head# 2024-04-20T17:27:44.179 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3a7d2d6:::benchmark_data_smithi171_122772_object153910:head# 2024-04-20T17:27:44.179 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3a8cf00:::benchmark_data_smithi171_122772_object90478:head# 2024-04-20T17:27:44.180 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3a97b5b:::benchmark_data_smithi171_122772_object158792:head# 2024-04-20T17:27:44.180 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3a9aced:::benchmark_data_smithi171_122772_object78101:head# 2024-04-20T17:27:44.180 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3aad8e8:::benchmark_data_smithi171_122772_object132291:head# 2024-04-20T17:27:44.181 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3aecbed:::benchmark_data_smithi171_122772_object162176:head# 2024-04-20T17:27:44.181 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3aede8b:::benchmark_data_smithi171_122772_object97251:head# 2024-04-20T17:27:44.182 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3af0579:::benchmark_data_smithi171_122772_object96245:head# 2024-04-20T17:27:44.183 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3b0388e:::benchmark_data_smithi171_122772_object134585:head# 2024-04-20T17:27:44.183 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3b07edb:::benchmark_data_smithi171_122772_object86774:head# 2024-04-20T17:27:44.183 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3b087a6:::benchmark_data_smithi171_122772_object119006:head# 2024-04-20T17:27:44.184 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3b0bac9:::benchmark_data_smithi171_122772_object90695:head# 2024-04-20T17:27:44.184 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3b131ea:::benchmark_data_smithi171_122772_object80004:head# 2024-04-20T17:27:44.185 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3b358a6:::benchmark_data_smithi171_122772_object167697:head# 2024-04-20T17:27:44.185 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3b3bf9d:::benchmark_data_smithi171_122772_object131399:head# 2024-04-20T17:27:44.186 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3b407cb:::benchmark_data_smithi171_122772_object93041:head# 2024-04-20T17:27:44.186 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3b4f14a:::benchmark_data_smithi171_122772_object120762:head# 2024-04-20T17:27:44.187 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3b51ef0:::benchmark_data_smithi171_122772_object122149:head# 2024-04-20T17:27:44.188 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3b55f38:::benchmark_data_smithi171_122772_object161771:head# 2024-04-20T17:27:44.188 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3b6a494:::benchmark_data_smithi171_122772_object116019:head# 2024-04-20T17:27:44.188 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3b79553:::benchmark_data_smithi171_122772_object137452:head# 2024-04-20T17:27:44.189 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3b7bbb2:::benchmark_data_smithi171_122772_object115250:head# 2024-04-20T17:27:44.190 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3b8f3a3:::benchmark_data_smithi171_122772_object153635:head# 2024-04-20T17:27:44.190 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3b941bb:::benchmark_data_smithi171_122772_object164135:head# 2024-04-20T17:27:44.191 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3ba08c6:::benchmark_data_smithi171_122772_object99439:head# 2024-04-20T17:27:44.191 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3ba9d21:::benchmark_data_smithi171_122772_object165690:head# 2024-04-20T17:27:44.192 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3bc6869:::benchmark_data_smithi171_122772_object100849:head# 2024-04-20T17:27:44.192 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3bcc61b:::benchmark_data_smithi171_122772_object89597:head# 2024-04-20T17:27:44.193 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3bd091a:::benchmark_data_smithi171_122772_object124624:head# 2024-04-20T17:27:44.193 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3bd5396:::benchmark_data_smithi171_122772_object147513:head# 2024-04-20T17:27:44.194 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3bd6df2:::benchmark_data_smithi171_122772_object154567:head# 2024-04-20T17:27:44.194 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3bda17d:::benchmark_data_smithi171_122772_object99435:head# 2024-04-20T17:27:44.194 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3bdc934:::benchmark_data_smithi171_122772_object134492:head# 2024-04-20T17:27:44.195 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3be3f05:::benchmark_data_smithi171_122772_object123338:head# 2024-04-20T17:27:44.195 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3bfd70c:::benchmark_data_smithi171_122772_object107240:head# 2024-04-20T17:27:44.196 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3c1ffbc:::benchmark_data_smithi171_122772_object150054:head# 2024-04-20T17:27:44.196 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3c2ee89:::benchmark_data_smithi171_122772_object147369:head# 2024-04-20T17:27:44.197 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3c3279d:::benchmark_data_smithi171_122772_object122017:head# 2024-04-20T17:27:44.197 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3c4b170:::benchmark_data_smithi171_122772_object168629:head# 2024-04-20T17:27:44.198 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3c4b2b9:::benchmark_data_smithi171_122772_object128843:head# 2024-04-20T17:27:44.199 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3c657a1:::benchmark_data_smithi171_122772_object100271:head# 2024-04-20T17:27:44.199 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3c767d6:::benchmark_data_smithi171_122772_object152623:head# 2024-04-20T17:27:44.200 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3c7b4b6:::benchmark_data_smithi171_122772_object97544:head# 2024-04-20T17:27:44.200 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3c83f92:::benchmark_data_smithi171_122772_object129068:head# 2024-04-20T17:27:44.201 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3c856e2:::benchmark_data_smithi171_122772_object149461:head# 2024-04-20T17:27:44.201 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3c858af:::benchmark_data_smithi171_122772_object92877:head# 2024-04-20T17:27:44.202 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3c8a3b2:::benchmark_data_smithi171_122772_object121207:head# 2024-04-20T17:27:44.202 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3c93831:::benchmark_data_smithi171_122772_object133539:head# 2024-04-20T17:27:44.203 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3c99f79:::benchmark_data_smithi171_122772_object145091:head# 2024-04-20T17:27:44.203 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3c9fc12:::benchmark_data_smithi171_122772_object162999:head# 2024-04-20T17:27:44.204 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3ca92af:::benchmark_data_smithi171_122772_object92940:head# 2024-04-20T17:27:44.205 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3cab72f:::benchmark_data_smithi171_122772_object87298:head# 2024-04-20T17:27:44.205 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3cd0a5e:::benchmark_data_smithi171_122772_object168062:head# 2024-04-20T17:27:44.206 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3cd33d3:::benchmark_data_smithi171_122772_object156439:head# 2024-04-20T17:27:44.207 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3cdd664:::benchmark_data_smithi171_122772_object138999:head# 2024-04-20T17:27:44.207 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3ceaa76:::benchmark_data_smithi171_122772_object89157:head# 2024-04-20T17:27:44.208 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3cee311:::benchmark_data_smithi171_122772_object99414:head# 2024-04-20T17:27:44.208 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3cf0deb:::benchmark_data_smithi171_122772_object118272:head# 2024-04-20T17:27:44.209 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3cfd141:::benchmark_data_smithi171_122772_object114382:head# 2024-04-20T17:27:44.209 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3d062f0:::benchmark_data_smithi171_122772_object102614:head# 2024-04-20T17:27:44.210 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3d0f043:::benchmark_data_smithi171_122772_object84129:head# 2024-04-20T17:27:44.210 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3d1c9ed:::benchmark_data_smithi171_122772_object114704:head# 2024-04-20T17:27:44.211 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3d2a427:::benchmark_data_smithi171_122772_object152208:head# 2024-04-20T17:27:44.211 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3d2b9e4:::benchmark_data_smithi171_122772_object165055:head# 2024-04-20T17:27:44.212 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3d332f2:::benchmark_data_smithi171_122772_object154898:head# 2024-04-20T17:27:44.212 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3d3aba8:::benchmark_data_smithi171_122772_object140057:head# 2024-04-20T17:27:44.213 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3d3c9c7:::benchmark_data_smithi171_122772_object153173:head# 2024-04-20T17:27:44.213 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3d3fb48:::benchmark_data_smithi171_122772_object124603:head# 2024-04-20T17:27:44.214 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3d4296d:::benchmark_data_smithi171_122772_object107570:head# 2024-04-20T17:27:44.214 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3d52b6c:::benchmark_data_smithi171_122772_object131993:head# 2024-04-20T17:27:44.215 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3d61208:::benchmark_data_smithi171_122772_object146952:head# 2024-04-20T17:27:44.215 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3d64248:::benchmark_data_smithi171_122772_object107836:head# 2024-04-20T17:27:44.216 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3d665e7:::benchmark_data_smithi171_122772_object141671:head# 2024-04-20T17:27:44.217 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3d6f0b4:::benchmark_data_smithi171_122772_object139949:head# 2024-04-20T17:27:44.217 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3d7a0d3:::benchmark_data_smithi171_122772_object143440:head# 2024-04-20T17:27:44.217 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3d7ab48:::benchmark_data_smithi171_122772_object155608:head# 2024-04-20T17:27:44.218 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3d7b60c:::benchmark_data_smithi171_122772_object169504:head# 2024-04-20T17:27:44.218 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3d81e1f:::benchmark_data_smithi171_122772_object148263:head# 2024-04-20T17:27:44.219 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3d8dd35:::benchmark_data_smithi171_122772_object97308:head# 2024-04-20T17:27:44.219 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3d999a7:::benchmark_data_smithi171_122772_object150851:head# 2024-04-20T17:27:44.220 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3da18b7:::benchmark_data_smithi171_122772_object127456:head# 2024-04-20T17:27:44.220 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3db49af:::benchmark_data_smithi171_122772_object144719:head# 2024-04-20T17:27:44.221 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3db5a9f:::benchmark_data_smithi171_122772_object130507:head# 2024-04-20T17:27:44.221 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3dbae06:::benchmark_data_smithi171_122772_object123101:head# 2024-04-20T17:27:44.222 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3dc2743:::benchmark_data_smithi171_122772_object78602:head# 2024-04-20T17:27:44.223 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3dc38f9:::benchmark_data_smithi171_122772_object147091:head# 2024-04-20T17:27:44.223 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3dcd581:::benchmark_data_smithi171_122772_object143936:head# 2024-04-20T17:27:44.224 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3dd6415:::benchmark_data_smithi171_122772_object99032:head# 2024-04-20T17:27:44.224 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3ddd4ca:::benchmark_data_smithi171_122772_object134209:head# 2024-04-20T17:27:44.225 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3dee3f0:::benchmark_data_smithi171_122772_object141679:head# 2024-04-20T17:27:44.225 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3df6cfe:::benchmark_data_smithi171_122772_object118512:head# 2024-04-20T17:27:44.226 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3e05468:::benchmark_data_smithi171_122772_object105293:head# 2024-04-20T17:27:44.226 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3e0a040:::benchmark_data_smithi171_122772_object152938:head# 2024-04-20T17:27:44.230 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3e12699:::benchmark_data_smithi171_122772_object139346:head# 2024-04-20T17:27:44.231 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3e1aa2b:::benchmark_data_smithi171_122772_object159510:head# 2024-04-20T17:27:44.231 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3e1f1b1:::benchmark_data_smithi171_122772_object141647:head# 2024-04-20T17:27:44.231 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3e36ffc:::benchmark_data_smithi171_122772_object135598:head# 2024-04-20T17:27:44.231 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3e432b3:::benchmark_data_smithi171_122772_object84103:head# 2024-04-20T17:27:44.231 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3e5aa6c:::benchmark_data_smithi171_122772_object138820:head# 2024-04-20T17:27:44.231 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3e5d947:::benchmark_data_smithi171_122772_object169712:head# 2024-04-20T17:27:44.231 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3e5e177:::benchmark_data_smithi171_122772_object89352:head# 2024-04-20T17:27:44.231 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3e5ea12:::benchmark_data_smithi171_122772_object133867:head# 2024-04-20T17:27:44.232 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3e6529a:::benchmark_data_smithi171_122772_object140499:head# 2024-04-20T17:27:44.232 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3e6867e:::benchmark_data_smithi171_122772_object81537:head# 2024-04-20T17:27:44.232 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3e80c2f:::benchmark_data_smithi171_122772_object144943:head# 2024-04-20T17:27:44.232 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3e906d3:::benchmark_data_smithi171_122772_object160363:head# 2024-04-20T17:27:44.233 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3eb2080:::benchmark_data_smithi171_122772_object78833:head# 2024-04-20T17:27:44.233 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3eb99cd:::benchmark_data_smithi171_122772_object109699:head# 2024-04-20T17:27:44.233 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3ebea71:::benchmark_data_smithi171_122772_object131143:head# 2024-04-20T17:27:44.234 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3ec5d10:::benchmark_data_smithi171_122772_object166624:head# 2024-04-20T17:27:44.234 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3eca2b6:::benchmark_data_smithi171_122772_object167894:head# 2024-04-20T17:27:44.235 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3edeb59:::benchmark_data_smithi171_122772_object99854:head# 2024-04-20T17:27:44.235 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3ee6971:::benchmark_data_smithi171_122772_object133916:head# 2024-04-20T17:27:44.236 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3eeee04:::benchmark_data_smithi171_122772_object130638:head# 2024-04-20T17:27:44.236 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3ef0a64:::benchmark_data_smithi171_122772_object86674:head# 2024-04-20T17:27:44.237 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3ef84cd:::benchmark_data_smithi171_122772_object163292:head# 2024-04-20T17:27:44.237 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3f113f2:::benchmark_data_smithi171_122772_object133604:head# 2024-04-20T17:27:44.238 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3f11ddc:::benchmark_data_smithi171_122772_object144521:head# 2024-04-20T17:27:44.238 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3f1bb22:::benchmark_data_smithi171_122772_object120462:head# 2024-04-20T17:27:44.239 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3f21e27:::benchmark_data_smithi171_122772_object128562:head# 2024-04-20T17:27:44.239 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3f21fef:::benchmark_data_smithi171_122772_object95294:head# 2024-04-20T17:27:44.239 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3f2f652:::benchmark_data_smithi171_122772_object99927:head# 2024-04-20T17:27:44.240 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3f30657:::benchmark_data_smithi171_122772_object89155:head# 2024-04-20T17:27:44.240 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3f39996:::benchmark_data_smithi171_122772_object146135:head# 2024-04-20T17:27:44.241 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3f3cfeb:::benchmark_data_smithi171_122772_object105339:head# 2024-04-20T17:27:44.241 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3f4f3d4:::benchmark_data_smithi171_122772_object130680:head# 2024-04-20T17:27:44.242 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3f62998:::benchmark_data_smithi171_122772_object162075:head# 2024-04-20T17:27:44.242 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3f712ee:::benchmark_data_smithi171_122772_object154409:head# 2024-04-20T17:27:44.242 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3f74595:::benchmark_data_smithi171_122772_object139190:head# 2024-04-20T17:27:44.243 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3f74db1:::benchmark_data_smithi171_122772_object86240:head# 2024-04-20T17:27:44.243 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3f7efd8:::benchmark_data_smithi171_122772_object118003:head# 2024-04-20T17:27:44.244 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3f8afc3:::benchmark_data_smithi171_122772_object161917:head# 2024-04-20T17:27:44.244 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3f8c64f:::benchmark_data_smithi171_122772_object99313:head# 2024-04-20T17:27:44.245 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3fa306e:::benchmark_data_smithi171_122772_object142136:head# 2024-04-20T17:27:44.245 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3fb106f:::benchmark_data_smithi171_122772_object129201:head# 2024-04-20T17:27:44.246 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3fb2590:::benchmark_data_smithi171_122772_object119037:head# 2024-04-20T17:27:44.246 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3fb782b:::benchmark_data_smithi171_122772_object135093:head# 2024-04-20T17:27:44.247 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3fbc888:::benchmark_data_smithi171_122772_object111312:head# 2024-04-20T17:27:44.247 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3fcdd08:::benchmark_data_smithi171_122772_object131882:head# 2024-04-20T17:27:44.248 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3fd0640:::benchmark_data_smithi171_122772_object148971:head# 2024-04-20T17:27:44.248 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3fe5827:::benchmark_data_smithi171_122772_object168691:head# 2024-04-20T17:27:44.249 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3fe5fea:::benchmark_data_smithi171_122772_object98798:head# 2024-04-20T17:27:44.249 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3fe6203:::benchmark_data_smithi171_122772_object148205:head# 2024-04-20T17:27:44.250 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3fead3e:::benchmark_data_smithi171_122772_object97283:head# 2024-04-20T17:27:44.250 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a3feea24:::benchmark_data_smithi171_122772_object133270:head# 2024-04-20T17:27:44.251 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4000638:::benchmark_data_smithi171_122772_object123230:head# 2024-04-20T17:27:44.251 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a40007f0:::benchmark_data_smithi171_122772_object86958:head# 2024-04-20T17:27:44.251 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a40064d8:::benchmark_data_smithi171_122772_object159783:head# 2024-04-20T17:27:44.252 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4006fb0:::benchmark_data_smithi171_122772_object107197:head# 2024-04-20T17:27:44.253 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4009708:::benchmark_data_smithi171_122772_object154781:head# 2024-04-20T17:27:44.253 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4014733:::benchmark_data_smithi171_122772_object159633:head# 2024-04-20T17:27:44.254 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a401a92e:::benchmark_data_smithi171_122772_object116947:head# 2024-04-20T17:27:44.254 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a402d46d:::benchmark_data_smithi171_122772_object123853:head# 2024-04-20T17:27:44.254 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4035bb7:::benchmark_data_smithi171_122772_object127827:head# 2024-04-20T17:27:44.255 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a403f4fc:::benchmark_data_smithi171_122772_object134967:head# 2024-04-20T17:27:44.255 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a404337d:::benchmark_data_smithi171_122772_object155992:head# 2024-04-20T17:27:44.256 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a40449c3:::benchmark_data_smithi171_122772_object134660:head# 2024-04-20T17:27:44.256 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4045368:::benchmark_data_smithi171_122772_object82768:head# 2024-04-20T17:27:44.257 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a404e332:::benchmark_data_smithi171_122772_object81235:head# 2024-04-20T17:27:44.257 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4063883:::benchmark_data_smithi171_122772_object114338:head# 2024-04-20T17:27:44.258 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a406571b:::benchmark_data_smithi171_122772_object78542:head# 2024-04-20T17:27:44.258 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a406ad14:::benchmark_data_smithi171_122772_object122978:head# 2024-04-20T17:27:44.258 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a406c49a:::benchmark_data_smithi171_122772_object136692:head# 2024-04-20T17:27:44.259 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a407257a:::benchmark_data_smithi171_122772_object151081:head# 2024-04-20T17:27:44.259 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a407d5a5:::benchmark_data_smithi171_122772_object128024:head# 2024-04-20T17:27:44.260 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a40939a4:::benchmark_data_smithi171_122772_object106991:head# 2024-04-20T17:27:44.260 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4095cb1:::benchmark_data_smithi171_122772_object148824:head# 2024-04-20T17:27:44.261 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4096263:::benchmark_data_smithi171_122772_object97191:head# 2024-04-20T17:27:44.261 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a40a308b:::benchmark_data_smithi171_122772_object164804:head# 2024-04-20T17:27:44.262 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a40abb35:::benchmark_data_smithi171_122772_object145082:head# 2024-04-20T17:27:44.262 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a40b19bd:::benchmark_data_smithi171_122772_object113103:head# 2024-04-20T17:27:44.263 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a40b267a:::benchmark_data_smithi171_122772_object134525:head# 2024-04-20T17:27:44.263 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a40b9b1c:::benchmark_data_smithi171_122772_object104136:head# 2024-04-20T17:27:44.264 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a40c74b6:::benchmark_data_smithi171_122772_object93248:head# 2024-04-20T17:27:44.264 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a40cd321:::benchmark_data_smithi171_122772_object95558:head# 2024-04-20T17:27:44.265 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a40d06ad:::benchmark_data_smithi171_122772_object103098:head# 2024-04-20T17:27:44.265 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a40e33e8:::benchmark_data_smithi171_122772_object140359:head# 2024-04-20T17:27:44.266 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a40fc8d6:::benchmark_data_smithi171_122772_object162411:head# 2024-04-20T17:27:44.266 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4106fcb:::benchmark_data_smithi171_122772_object143383:head# 2024-04-20T17:27:44.266 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a41070e7:::benchmark_data_smithi171_122772_object114713:head# 2024-04-20T17:27:44.267 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a411d095:::benchmark_data_smithi171_122772_object113211:head# 2024-04-20T17:27:44.267 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a412201c:::benchmark_data_smithi171_122772_object99697:head# 2024-04-20T17:27:44.268 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a412574b:::benchmark_data_smithi171_122772_object86983:head# 2024-04-20T17:27:44.268 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a412cce0:::benchmark_data_smithi171_122772_object121972:head# 2024-04-20T17:27:44.269 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a412cef9:::benchmark_data_smithi171_122772_object120868:head# 2024-04-20T17:27:44.269 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a413dadb:::benchmark_data_smithi171_122772_object111588:head# 2024-04-20T17:27:44.270 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a414d888:::benchmark_data_smithi171_122772_object89090:head# 2024-04-20T17:27:44.270 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4150d14:::benchmark_data_smithi171_122772_object167055:head# 2024-04-20T17:27:44.271 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4161378:::benchmark_data_smithi171_122772_object143925:head# 2024-04-20T17:27:44.271 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4168e86:::benchmark_data_smithi171_122772_object151229:head# 2024-04-20T17:27:44.272 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a416ea72:::benchmark_data_smithi171_122772_object84152:head# 2024-04-20T17:27:44.272 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4170314:::benchmark_data_smithi171_122772_object151840:head# 2024-04-20T17:27:44.273 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4174576:::benchmark_data_smithi171_122772_object98636:head# 2024-04-20T17:27:44.273 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4175ae2:::benchmark_data_smithi171_122772_object124224:head# 2024-04-20T17:27:44.273 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a417ee0e:::benchmark_data_smithi171_122772_object84571:head# 2024-04-20T17:27:44.274 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4185d8a:::benchmark_data_smithi171_122772_object144836:head# 2024-04-20T17:27:44.275 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a418b66c:::benchmark_data_smithi171_122772_object139703:head# 2024-04-20T17:27:44.275 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a419105f:::benchmark_data_smithi171_122772_object138213:head# 2024-04-20T17:27:44.275 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4192bf6:::benchmark_data_smithi171_122772_object150209:head# 2024-04-20T17:27:44.276 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a41a3756:::benchmark_data_smithi171_122772_object116305:head# 2024-04-20T17:27:44.276 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a41a5810:::benchmark_data_smithi171_122772_object136408:head# 2024-04-20T17:27:44.277 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a41b7a74:::benchmark_data_smithi171_122772_object128658:head# 2024-04-20T17:27:44.277 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a41c1e9b:::benchmark_data_smithi171_122772_object134591:head# 2024-04-20T17:27:44.278 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a41c4608:::benchmark_data_smithi171_122772_object140223:head# 2024-04-20T17:27:44.278 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a41cc82c:::benchmark_data_smithi171_122772_object163885:head# 2024-04-20T17:27:44.279 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a41d6b18:::benchmark_data_smithi171_122772_object159026:head# 2024-04-20T17:27:44.279 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a41d6df5:::benchmark_data_smithi171_122772_object114061:head# 2024-04-20T17:27:44.280 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a41dfbf1:::benchmark_data_smithi171_122772_object118560:head# 2024-04-20T17:27:44.280 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a41e5d44:::benchmark_data_smithi171_122772_object82190:head# 2024-04-20T17:27:44.281 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a41edf69:::benchmark_data_smithi171_122772_object80866:head# 2024-04-20T17:27:44.281 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a41f115d:::benchmark_data_smithi171_122772_object144934:head# 2024-04-20T17:27:44.281 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4229047:::benchmark_data_smithi171_122772_object87346:head# 2024-04-20T17:27:44.282 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a422f15f:::benchmark_data_smithi171_122772_object102025:head# 2024-04-20T17:27:44.282 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a42323a9:::benchmark_data_smithi171_122772_object112967:head# 2024-04-20T17:27:44.283 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4234adc:::benchmark_data_smithi171_122772_object92675:head# 2024-04-20T17:27:44.283 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4246795:::benchmark_data_smithi171_122772_object95436:head# 2024-04-20T17:27:44.284 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a426338e:::benchmark_data_smithi171_122772_object112616:head# 2024-04-20T17:27:44.284 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a426db7b:::benchmark_data_smithi171_122772_object125319:head# 2024-04-20T17:27:44.284 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a426fcd1:::benchmark_data_smithi171_122772_object107655:head# 2024-04-20T17:27:44.285 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a42700db:::benchmark_data_smithi171_122772_object121868:head# 2024-04-20T17:27:44.285 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4272610:::benchmark_data_smithi171_122772_object149594:head# 2024-04-20T17:27:44.286 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a427440d:::benchmark_data_smithi171_122772_object142596:head# 2024-04-20T17:27:44.287 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4278f2d:::benchmark_data_smithi171_122772_object156315:head# 2024-04-20T17:27:44.287 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4279249:::benchmark_data_smithi171_122772_object156568:head# 2024-04-20T17:27:44.288 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4287fbd:::benchmark_data_smithi171_122772_object90383:head# 2024-04-20T17:27:44.288 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4294b8e:::benchmark_data_smithi171_122772_object131242:head# 2024-04-20T17:27:44.289 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4296a7f:::benchmark_data_smithi171_122772_object129117:head# 2024-04-20T17:27:44.289 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a429fe80:::benchmark_data_smithi171_122772_object103830:head# 2024-04-20T17:27:44.289 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a42b7eb4:::benchmark_data_smithi171_122772_object110106:head# 2024-04-20T17:27:44.290 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a42c0fcb:::benchmark_data_smithi171_122772_object109593:head# 2024-04-20T17:27:44.290 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a42d78bb:::benchmark_data_smithi171_122772_object145961:head# 2024-04-20T17:27:44.291 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a42f1601:::benchmark_data_smithi171_122772_object164603:head# 2024-04-20T17:27:44.291 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4304cd4:::benchmark_data_smithi171_122772_object152507:head# 2024-04-20T17:27:44.292 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4315687:::benchmark_data_smithi171_122772_object122830:head# 2024-04-20T17:27:44.292 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a432415c:::benchmark_data_smithi171_122772_object133906:head# 2024-04-20T17:27:44.293 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4330921:::benchmark_data_smithi171_122772_object97647:head# 2024-04-20T17:27:44.293 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a43596d9:::benchmark_data_smithi171_122772_object156068:head# 2024-04-20T17:27:44.294 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a435d07f:::benchmark_data_smithi171_122772_object106829:head# 2024-04-20T17:27:44.294 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a435e955:::benchmark_data_smithi171_122772_object128348:head# 2024-04-20T17:27:44.294 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4361d1a:::benchmark_data_smithi171_122772_object96909:head# 2024-04-20T17:27:44.295 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4366297:::benchmark_data_smithi171_122772_object158175:head# 2024-04-20T17:27:44.295 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a43763cc:::benchmark_data_smithi171_122772_object92305:head# 2024-04-20T17:27:44.296 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4388bd0:::benchmark_data_smithi171_122772_object140293:head# 2024-04-20T17:27:44.296 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a438b23b:::benchmark_data_smithi171_122772_object136969:head# 2024-04-20T17:27:44.297 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a439cfa2:::benchmark_data_smithi171_122772_object122544:head# 2024-04-20T17:27:44.297 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a43aade4:::benchmark_data_smithi171_122772_object112371:head# 2024-04-20T17:27:44.298 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a43eafb9:::benchmark_data_smithi171_122772_object99714:head# 2024-04-20T17:27:44.298 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a440753d:::benchmark_data_smithi171_122772_object90005:head# 2024-04-20T17:27:44.299 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4407a88:::benchmark_data_smithi171_122772_object111307:head# 2024-04-20T17:27:44.300 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a440bf4d:::benchmark_data_smithi171_122772_object142635:head# 2024-04-20T17:27:44.300 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a441783c:::benchmark_data_smithi171_122772_object141846:head# 2024-04-20T17:27:44.301 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a441d3de:::benchmark_data_smithi171_122772_object146818:head# 2024-04-20T17:27:44.301 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a442fc04:::benchmark_data_smithi171_122772_object85686:head# 2024-04-20T17:27:44.302 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a444175f:::benchmark_data_smithi171_122772_object136514:head# 2024-04-20T17:27:44.302 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a44456cf:::benchmark_data_smithi171_122772_object109671:head# 2024-04-20T17:27:44.303 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a444715c:::benchmark_data_smithi171_122772_object138780:head# 2024-04-20T17:27:44.303 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a444a8f8:::benchmark_data_smithi171_122772_object110736:head# 2024-04-20T17:27:44.304 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a446920d:::benchmark_data_smithi171_122772_object162569:head# 2024-04-20T17:27:44.305 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a447b4d6:::benchmark_data_smithi171_122772_object78686:head# 2024-04-20T17:27:44.306 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a447bca8:::benchmark_data_smithi171_122772_object126489:head# 2024-04-20T17:27:44.306 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a447cd59:::benchmark_data_smithi171_122772_object121721:head# 2024-04-20T17:27:44.306 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a44e12af:::benchmark_data_smithi171_122772_object135504:head# 2024-04-20T17:27:44.307 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a44eb070:::benchmark_data_smithi171_122772_object128767:head# 2024-04-20T17:27:44.307 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4500e9c:::benchmark_data_smithi171_122772_object83906:head# 2024-04-20T17:27:44.308 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4505460:::benchmark_data_smithi171_122772_object159015:head# 2024-04-20T17:27:44.308 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4505474:::benchmark_data_smithi171_122772_object85811:head# 2024-04-20T17:27:44.309 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a45097fe:::benchmark_data_smithi171_122772_object164000:head# 2024-04-20T17:27:44.309 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a450c60c:::benchmark_data_smithi171_122772_object123409:head# 2024-04-20T17:27:44.310 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a450c69f:::benchmark_data_smithi171_122772_object80439:head# 2024-04-20T17:27:44.311 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a451f7b0:::benchmark_data_smithi171_122772_object92554:head# 2024-04-20T17:27:44.311 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4529729:::benchmark_data_smithi171_122772_object92555:head# 2024-04-20T17:27:44.312 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a452d0a0:::benchmark_data_smithi171_122772_object111499:head# 2024-04-20T17:27:44.312 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4554fc9:::benchmark_data_smithi171_122772_object134159:head# 2024-04-20T17:27:44.312 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4556bca:::benchmark_data_smithi171_122772_object113284:head# 2024-04-20T17:27:44.313 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4563599:::benchmark_data_smithi171_122772_object92188:head# 2024-04-20T17:27:44.313 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4565b0d:::benchmark_data_smithi171_122772_object116178:head# 2024-04-20T17:27:44.314 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4589dbb:::benchmark_data_smithi171_122772_object161365:head# 2024-04-20T17:27:44.314 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a45a17c8:::benchmark_data_smithi171_122772_object123427:head# 2024-04-20T17:27:44.315 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a45ae382:::benchmark_data_smithi171_122772_object146879:head# 2024-04-20T17:27:44.315 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a45bf0a2:::benchmark_data_smithi171_122772_object78643:head# 2024-04-20T17:27:44.316 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a45c0291:::benchmark_data_smithi171_122772_object101029:head# 2024-04-20T17:27:44.316 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a45c75df:::benchmark_data_smithi171_122772_object145853:head# 2024-04-20T17:27:44.317 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a45c7739:::benchmark_data_smithi171_122772_object90628:head# 2024-04-20T17:27:44.317 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a45cc7d3:::benchmark_data_smithi171_122772_object90670:head# 2024-04-20T17:27:44.317 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a45de603:::benchmark_data_smithi171_122772_object146539:head# 2024-04-20T17:27:44.318 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a45ec03b:::benchmark_data_smithi171_122772_object110915:head# 2024-04-20T17:27:44.318 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a45fc3ff:::benchmark_data_smithi171_122772_object86449:head# 2024-04-20T17:27:44.319 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4603690:::benchmark_data_smithi171_122772_object169825:head# 2024-04-20T17:27:44.319 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a46052b5:::benchmark_data_smithi171_122772_object99863:head# 2024-04-20T17:27:44.320 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4609142:::benchmark_data_smithi171_122772_object110547:head# 2024-04-20T17:27:44.320 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a460f6b4:::benchmark_data_smithi171_122772_object80918:head# 2024-04-20T17:27:44.321 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a461587a:::benchmark_data_smithi171_122772_object100936:head# 2024-04-20T17:27:44.321 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a461e589:::benchmark_data_smithi171_122772_object140279:head# 2024-04-20T17:27:44.322 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a461e799:::benchmark_data_smithi171_122772_object164440:head# 2024-04-20T17:27:44.322 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a462d6d0:::benchmark_data_smithi171_122772_object130754:head# 2024-04-20T17:27:44.323 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a462ff7c:::benchmark_data_smithi171_122772_object101247:head# 2024-04-20T17:27:44.323 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a46735b1:::benchmark_data_smithi171_122772_object140200:head# 2024-04-20T17:27:44.324 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4677646:::benchmark_data_smithi171_122772_object103927:head# 2024-04-20T17:27:44.324 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a467a6db:::benchmark_data_smithi171_122772_object139632:head# 2024-04-20T17:27:44.325 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a46880e8:::benchmark_data_smithi171_122772_object78839:head# 2024-04-20T17:27:44.325 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a469c1a2:::benchmark_data_smithi171_122772_object124161:head# 2024-04-20T17:27:44.325 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a46a183e:::benchmark_data_smithi171_122772_object129891:head# 2024-04-20T17:27:44.326 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a46a918d:::benchmark_data_smithi171_122772_object152970:head# 2024-04-20T17:27:44.326 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a46a9455:::benchmark_data_smithi171_122772_object83244:head# 2024-04-20T17:27:44.327 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a46c8851:::benchmark_data_smithi171_122772_object152755:head# 2024-04-20T17:27:44.327 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a46c8f21:::benchmark_data_smithi171_122772_object123692:head# 2024-04-20T17:27:44.328 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a46d31af:::benchmark_data_smithi171_122772_object112172:head# 2024-04-20T17:27:44.328 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a46da395:::benchmark_data_smithi171_122772_object115705:head# 2024-04-20T17:27:44.329 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a46e805f:::benchmark_data_smithi171_122772_object148357:head# 2024-04-20T17:27:44.329 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a46f7013:::benchmark_data_smithi171_122772_object164478:head# 2024-04-20T17:27:44.330 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a46f8f92:::benchmark_data_smithi171_122772_object151827:head# 2024-04-20T17:27:44.330 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a47009fe:::benchmark_data_smithi171_122772_object114493:head# 2024-04-20T17:27:44.331 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a470f657:::benchmark_data_smithi171_122772_object93953:head# 2024-04-20T17:27:44.331 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a47104b5:::benchmark_data_smithi171_122772_object82412:head# 2024-04-20T17:27:44.332 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4714f48:::benchmark_data_smithi171_122772_object109190:head# 2024-04-20T17:27:44.332 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4718963:::benchmark_data_smithi171_122772_object100141:head# 2024-04-20T17:27:44.333 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a471fdce:::benchmark_data_smithi171_122772_object149759:head# 2024-04-20T17:27:44.333 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4724397:::benchmark_data_smithi171_122772_object100330:head# 2024-04-20T17:27:44.334 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4725f3a:::benchmark_data_smithi171_122772_object91924:head# 2024-04-20T17:27:44.334 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a472829c:::benchmark_data_smithi171_122772_object94118:head# 2024-04-20T17:27:44.335 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4728c8a:::benchmark_data_smithi171_122772_object164922:head# 2024-04-20T17:27:44.335 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a472d00f:::benchmark_data_smithi171_122772_object84199:head# 2024-04-20T17:27:44.335 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:44 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 3774/184372 objects degraded (2.047%), 2 pgs degraded (PG_DEGRADED) 2024-04-20T17:27:44.336 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a47307fa:::benchmark_data_smithi171_122772_object158996:head# 2024-04-20T17:27:44.336 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4744d0f:::benchmark_data_smithi171_122772_object118656:head# 2024-04-20T17:27:44.337 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a47561d1:::benchmark_data_smithi171_122772_object85319:head# 2024-04-20T17:27:44.337 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a476b9a0:::benchmark_data_smithi171_122772_object110552:head# 2024-04-20T17:27:44.337 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a477a009:::benchmark_data_smithi171_122772_object96880:head# 2024-04-20T17:27:44.338 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4787748:::benchmark_data_smithi171_122772_object82293:head# 2024-04-20T17:27:44.338 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4789daf:::benchmark_data_smithi171_122772_object98450:head# 2024-04-20T17:27:44.341 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a47926de:::benchmark_data_smithi171_122772_object88310:head# 2024-04-20T17:27:44.342 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a479abf5:::benchmark_data_smithi171_122772_object83791:head# 2024-04-20T17:27:44.342 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a47a8ec2:::benchmark_data_smithi171_122772_object115748:head# 2024-04-20T17:27:44.343 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a47b7612:::benchmark_data_smithi171_122772_object120191:head# 2024-04-20T17:27:44.343 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a47b8cae:::benchmark_data_smithi171_122772_object147036:head# 2024-04-20T17:27:44.343 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a47be47e:::benchmark_data_smithi171_122772_object140892:head# 2024-04-20T17:27:44.344 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a47c6e17:::benchmark_data_smithi171_122772_object94122:head# 2024-04-20T17:27:44.344 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a47f4215:::benchmark_data_smithi171_122772_object109078:head# 2024-04-20T17:27:44.345 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a47ffcfb:::benchmark_data_smithi171_122772_object90052:head# 2024-04-20T17:27:44.346 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4808028:::benchmark_data_smithi171_122772_object111555:head# 2024-04-20T17:27:44.346 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a480db82:::benchmark_data_smithi171_122772_object126809:head# 2024-04-20T17:27:44.346 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4818a62:::benchmark_data_smithi171_122772_object116095:head# 2024-04-20T17:27:44.347 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a481bd6e:::benchmark_data_smithi171_122772_object162036:head# 2024-04-20T17:27:44.347 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a482359c:::benchmark_data_smithi171_122772_object119791:head# 2024-04-20T17:27:44.348 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a482427f:::benchmark_data_smithi171_122772_object117230:head# 2024-04-20T17:27:44.348 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4836466:::benchmark_data_smithi171_122772_object119774:head# 2024-04-20T17:27:44.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4841e63:::benchmark_data_smithi171_122772_object115229:head# 2024-04-20T17:27:44.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4846cc2:::benchmark_data_smithi171_122772_object93492:head# 2024-04-20T17:27:44.350 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a485315b:::benchmark_data_smithi171_122772_object133739:head# 2024-04-20T17:27:44.350 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a485dde3:::benchmark_data_smithi171_122772_object84336:head# 2024-04-20T17:27:44.350 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a486bc51:::benchmark_data_smithi171_122772_object120820:head# 2024-04-20T17:27:44.351 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a48749e3:::benchmark_data_smithi171_122772_object144940:head# 2024-04-20T17:27:44.351 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4891806:::benchmark_data_smithi171_122772_object110308:head# 2024-04-20T17:27:44.352 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4893ef6:::benchmark_data_smithi171_122772_object79198:head# 2024-04-20T17:27:44.352 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a48a1dfe:::benchmark_data_smithi171_122772_object154428:head# 2024-04-20T17:27:44.353 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a48b6d6e:::benchmark_data_smithi171_122772_object154286:head# 2024-04-20T17:27:44.353 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a48bdfb5:::benchmark_data_smithi171_122772_object136487:head# 2024-04-20T17:27:44.354 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a48c2d46:::benchmark_data_smithi171_122772_object152543:head# 2024-04-20T17:27:44.354 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a48c415f:::benchmark_data_smithi171_122772_object135910:head# 2024-04-20T17:27:44.355 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a48c7fb7:::benchmark_data_smithi171_122772_object108563:head# 2024-04-20T17:27:44.355 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a48d5e04:::benchmark_data_smithi171_122772_object165796:head# 2024-04-20T17:27:44.356 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a48dd5ae:::benchmark_data_smithi171_122772_object147532:head# 2024-04-20T17:27:44.356 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a48ed54a:::benchmark_data_smithi171_122772_object107510:head# 2024-04-20T17:27:44.357 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a48f30a6:::benchmark_data_smithi171_122772_object108562:head# 2024-04-20T17:27:44.357 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a490bfee:::benchmark_data_smithi171_122772_object96775:head# 2024-04-20T17:27:44.358 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4924086:::benchmark_data_smithi171_122772_object99500:head# 2024-04-20T17:27:44.358 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a49271c9:::benchmark_data_smithi171_122772_object121882:head# 2024-04-20T17:27:44.359 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a492c7b8:::benchmark_data_smithi171_122772_object103369:head# 2024-04-20T17:27:44.359 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a492d66b:::benchmark_data_smithi171_122772_object100000:head# 2024-04-20T17:27:44.360 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4937506:::benchmark_data_smithi171_122772_object150721:head# 2024-04-20T17:27:44.360 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4937933:::benchmark_data_smithi171_122772_object122196:head# 2024-04-20T17:27:44.361 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4946df8:::benchmark_data_smithi171_122772_object120576:head# 2024-04-20T17:27:44.361 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a494a5d0:::benchmark_data_smithi171_122772_object155142:head# 2024-04-20T17:27:44.362 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4953e65:::benchmark_data_smithi171_122772_object136659:head# 2024-04-20T17:27:44.362 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a495563a:::benchmark_data_smithi171_122772_object107031:head# 2024-04-20T17:27:44.363 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a495801a:::benchmark_data_smithi171_122772_object87576:head# 2024-04-20T17:27:44.363 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a49619a1:::benchmark_data_smithi171_122772_object85500:head# 2024-04-20T17:27:44.363 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a496cc1a:::benchmark_data_smithi171_122772_object146531:head# 2024-04-20T17:27:44.364 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a498320d:::benchmark_data_smithi171_122772_object88450:head# 2024-04-20T17:27:44.364 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4988691:::benchmark_data_smithi171_122772_object145282:head# 2024-04-20T17:27:44.365 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4990e87:::benchmark_data_smithi171_122772_object97344:head# 2024-04-20T17:27:44.365 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4995477:::benchmark_data_smithi171_122772_object156422:head# 2024-04-20T17:27:44.366 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a49aca69:::benchmark_data_smithi171_122772_object98308:head# 2024-04-20T17:27:44.366 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a49b7c69:::benchmark_data_smithi171_122772_object134576:head# 2024-04-20T17:27:44.367 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a49c610c:::benchmark_data_smithi171_122772_object87282:head# 2024-04-20T17:27:44.367 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a49d7916:::benchmark_data_smithi171_122772_object146263:head# 2024-04-20T17:27:44.368 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a49df3f4:::benchmark_data_smithi171_122772_object120271:head# 2024-04-20T17:27:44.368 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4a01796:::benchmark_data_smithi171_122772_object95098:head# 2024-04-20T17:27:44.369 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4a05d6a:::benchmark_data_smithi171_122772_object160468:head# 2024-04-20T17:27:44.369 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4a0d806:::benchmark_data_smithi171_122772_object150071:head# 2024-04-20T17:27:44.370 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4a1d6f0:::benchmark_data_smithi171_122772_object129660:head# 2024-04-20T17:27:44.370 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4a33367:::benchmark_data_smithi171_122772_object124776:head# 2024-04-20T17:27:44.371 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4a3a0fa:::benchmark_data_smithi171_122772_object157748:head# 2024-04-20T17:27:44.371 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4a3b614:::benchmark_data_smithi171_122772_object156195:head# 2024-04-20T17:27:44.372 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4a471e2:::benchmark_data_smithi171_122772_object158071:head# 2024-04-20T17:27:44.373 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4a5236d:::benchmark_data_smithi171_122772_object95222:head# 2024-04-20T17:27:44.374 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4a53eb8:::benchmark_data_smithi171_122772_object115900:head# 2024-04-20T17:27:44.374 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4a6c901:::benchmark_data_smithi171_122772_object167089:head# 2024-04-20T17:27:44.374 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4a6e0d5:::benchmark_data_smithi171_122772_object124576:head# 2024-04-20T17:27:44.375 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4a74388:::benchmark_data_smithi171_122772_object135737:head# 2024-04-20T17:27:44.375 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4a7cd2f:::benchmark_data_smithi171_122772_object126456:head# 2024-04-20T17:27:44.376 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4a8f6f2:::benchmark_data_smithi171_122772_object147182:head# 2024-04-20T17:27:44.376 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4a9b8ec:::benchmark_data_smithi171_122772_object134482:head# 2024-04-20T17:27:44.377 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4aa9e7a:::benchmark_data_smithi171_122772_object162775:head# 2024-04-20T17:27:44.377 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4abc7a4:::benchmark_data_smithi171_122772_object85157:head# 2024-04-20T17:27:44.378 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4ac37b9:::benchmark_data_smithi171_122772_object85974:head# 2024-04-20T17:27:44.378 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4ac603a:::benchmark_data_smithi171_122772_object117912:head# 2024-04-20T17:27:44.379 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4ad8c44:::benchmark_data_smithi171_122772_object124663:head# 2024-04-20T17:27:44.379 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4ad8e64:::benchmark_data_smithi171_122772_object161575:head# 2024-04-20T17:27:44.380 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4ad8fbe:::benchmark_data_smithi171_122772_object80899:head# 2024-04-20T17:27:44.380 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4af6082:::benchmark_data_smithi171_122772_object92569:head# 2024-04-20T17:27:44.380 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4b08e1b:::benchmark_data_smithi171_122772_object110586:head# 2024-04-20T17:27:44.381 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4b0d3a0:::benchmark_data_smithi171_122772_object85721:head# 2024-04-20T17:27:44.382 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4b201c5:::benchmark_data_smithi171_122772_object165123:head# 2024-04-20T17:27:44.382 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4b23580:::benchmark_data_smithi171_122772_object146058:head# 2024-04-20T17:27:44.383 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4b395ab:::benchmark_data_smithi171_122772_object137565:head# 2024-04-20T17:27:44.383 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4b4794c:::benchmark_data_smithi171_122772_object99573:head# 2024-04-20T17:27:44.383 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4b610c9:::benchmark_data_smithi171_122772_object138829:head# 2024-04-20T17:27:44.384 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4b6fd79:::benchmark_data_smithi171_122772_object163152:head# 2024-04-20T17:27:44.385 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4b7f246:::benchmark_data_smithi171_122772_object118428:head# 2024-04-20T17:27:44.385 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4bade22:::benchmark_data_smithi171_122772_object156350:head# 2024-04-20T17:27:44.385 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4bb0c7e:::benchmark_data_smithi171_122772_object156156:head# 2024-04-20T17:27:44.386 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4bb100f:::benchmark_data_smithi171_122772_object160556:head# 2024-04-20T17:27:44.386 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4bbe488:::benchmark_data_smithi171_122772_object107246:head# 2024-04-20T17:27:44.387 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4bdf282:::benchmark_data_smithi171_122772_object147339:head# 2024-04-20T17:27:44.387 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4bf8091:::benchmark_data_smithi171_122772_object149475:head# 2024-04-20T17:27:44.388 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4c02dac:::benchmark_data_smithi171_122772_object125757:head# 2024-04-20T17:27:44.388 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4c035f2:::benchmark_data_smithi171_122772_object104036:head# 2024-04-20T17:27:44.389 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4c199ac:::benchmark_data_smithi171_122772_object114118:head# 2024-04-20T17:27:44.389 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4c1e49d:::benchmark_data_smithi171_122772_object158568:head# 2024-04-20T17:27:44.390 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4c1fc1b:::benchmark_data_smithi171_122772_object125876:head# 2024-04-20T17:27:44.390 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4c1fd43:::benchmark_data_smithi171_122772_object113135:head# 2024-04-20T17:27:44.391 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4c22e88:::benchmark_data_smithi171_122772_object151675:head# 2024-04-20T17:27:44.391 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4c3b993:::benchmark_data_smithi171_122772_object132575:head# 2024-04-20T17:27:44.392 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4c45a0b:::benchmark_data_smithi171_122772_object134782:head# 2024-04-20T17:27:44.393 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4c46df4:::benchmark_data_smithi171_122772_object119808:head# 2024-04-20T17:27:44.393 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4c4c3f0:::benchmark_data_smithi171_122772_object85142:head# 2024-04-20T17:27:44.394 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4c58abf:::benchmark_data_smithi171_122772_object125178:head# 2024-04-20T17:27:44.394 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4c622f9:::benchmark_data_smithi171_122772_object169804:head# 2024-04-20T17:27:44.395 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4c63a1a:::benchmark_data_smithi171_122772_object161642:head# 2024-04-20T17:27:44.395 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4c8369d:::benchmark_data_smithi171_122772_object150657:head# 2024-04-20T17:27:44.396 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4c897fe:::benchmark_data_smithi171_122772_object122377:head# 2024-04-20T17:27:44.396 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4c8c788:::benchmark_data_smithi171_122772_object150533:head# 2024-04-20T17:27:44.397 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4c93204:::benchmark_data_smithi171_122772_object86980:head# 2024-04-20T17:27:44.397 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4c9ae72:::benchmark_data_smithi171_122772_object121146:head# 2024-04-20T17:27:44.398 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4ca7496:::benchmark_data_smithi171_122772_object84635:head# 2024-04-20T17:27:44.398 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4ca9e59:::benchmark_data_smithi171_122772_object79211:head# 2024-04-20T17:27:44.399 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4cb7dd4:::benchmark_data_smithi171_122772_object118654:head# 2024-04-20T17:27:44.400 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4cc1794:::benchmark_data_smithi171_122772_object98515:head# 2024-04-20T17:27:44.400 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4cca0b6:::benchmark_data_smithi171_122772_object136374:head# 2024-04-20T17:27:44.401 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4ccdab5:::benchmark_data_smithi171_122772_object129355:head# 2024-04-20T17:27:44.401 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4cd4625:::benchmark_data_smithi171_122772_object106709:head# 2024-04-20T17:27:44.402 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4cd9185:::benchmark_data_smithi171_122772_object132681:head# 2024-04-20T17:27:44.402 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4cf2846:::benchmark_data_smithi171_122772_object91559:head# 2024-04-20T17:27:44.403 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4cf3879:::benchmark_data_smithi171_122772_object161659:head# 2024-04-20T17:27:44.403 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4cf7e87:::benchmark_data_smithi171_122772_object112902:head# 2024-04-20T17:27:44.404 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4cf9382:::benchmark_data_smithi171_122772_object152952:head# 2024-04-20T17:27:44.404 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4cfdad4:::benchmark_data_smithi171_122772_object118969:head# 2024-04-20T17:27:44.405 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4d0421f:::benchmark_data_smithi171_122772_object169298:head# 2024-04-20T17:27:44.405 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4d554e1:::benchmark_data_smithi171_122772_object112479:head# 2024-04-20T17:27:44.406 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4d55e92:::benchmark_data_smithi171_122772_object117429:head# 2024-04-20T17:27:44.406 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4d58c34:::benchmark_data_smithi171_122772_object119932:head# 2024-04-20T17:27:44.407 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4d5bd65:::benchmark_data_smithi171_122772_object85227:head# 2024-04-20T17:27:44.407 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4d641d7:::benchmark_data_smithi171_122772_object93952:head# 2024-04-20T17:27:44.408 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4d6a049:::benchmark_data_smithi171_122772_object111334:head# 2024-04-20T17:27:44.408 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4d7094c:::benchmark_data_smithi171_122772_object109739:head# 2024-04-20T17:27:44.409 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4d7df9c:::benchmark_data_smithi171_122772_object148276:head# 2024-04-20T17:27:44.409 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4d8168f:::benchmark_data_smithi171_122772_object169090:head# 2024-04-20T17:27:44.410 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4d818fa:::benchmark_data_smithi171_122772_object135331:head# 2024-04-20T17:27:44.410 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4d82b6f:::benchmark_data_smithi171_122772_object116717:head# 2024-04-20T17:27:44.411 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4d8875d:::benchmark_data_smithi171_122772_object132126:head# 2024-04-20T17:27:44.411 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4d89480:::benchmark_data_smithi171_122772_object85414:head# 2024-04-20T17:27:44.412 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4d89f5a:::benchmark_data_smithi171_122772_object116100:head# 2024-04-20T17:27:44.412 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4db75d9:::benchmark_data_smithi171_122772_object130688:head# 2024-04-20T17:27:44.413 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4dba2a0:::benchmark_data_smithi171_122772_object90176:head# 2024-04-20T17:27:44.413 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4dbcb7a:::benchmark_data_smithi171_122772_object169867:head# 2024-04-20T17:27:44.414 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4dd4c8f:::benchmark_data_smithi171_122772_object79964:head# 2024-04-20T17:27:44.415 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4ddcb05:::benchmark_data_smithi171_122772_object148899:head# 2024-04-20T17:27:44.416 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4e01e3d:::benchmark_data_smithi171_122772_object80713:head# 2024-04-20T17:27:44.416 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4e144dc:::benchmark_data_smithi171_122772_object124134:head# 2024-04-20T17:27:44.416 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4e230df:::benchmark_data_smithi171_122772_object98234:head# 2024-04-20T17:27:44.417 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4e330b5:::benchmark_data_smithi171_122772_object154031:head# 2024-04-20T17:27:44.418 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4e3d554:::benchmark_data_smithi171_122772_object109845:head# 2024-04-20T17:27:44.418 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4e664e9:::benchmark_data_smithi171_122772_object125736:head# 2024-04-20T17:27:44.419 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4e6e090:::benchmark_data_smithi171_122772_object95414:head# 2024-04-20T17:27:44.419 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4e6f7b7:::benchmark_data_smithi171_122772_object95369:head# 2024-04-20T17:27:44.420 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4e7082c:::benchmark_data_smithi171_122772_object145152:head# 2024-04-20T17:27:44.420 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4e88a4f:::benchmark_data_smithi171_122772_object148462:head# 2024-04-20T17:27:44.421 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4eac385:::benchmark_data_smithi171_122772_object137425:head# 2024-04-20T17:27:44.422 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4ead24c:::benchmark_data_smithi171_122772_object137526:head# 2024-04-20T17:27:44.422 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4eb5a82:::benchmark_data_smithi171_122772_object133316:head# 2024-04-20T17:27:44.422 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4ecf9ea:::benchmark_data_smithi171_122772_object123357:head# 2024-04-20T17:27:44.423 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4ed3e48:::benchmark_data_smithi171_122772_object81028:head# 2024-04-20T17:27:44.423 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4f06cf7:::benchmark_data_smithi171_122772_object127984:head# 2024-04-20T17:27:44.424 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4f0af64:::benchmark_data_smithi171_122772_object85714:head# 2024-04-20T17:27:44.425 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4f1544e:::benchmark_data_smithi171_122772_object136754:head# 2024-04-20T17:27:44.425 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4f21d70:::benchmark_data_smithi171_122772_object167896:head# 2024-04-20T17:27:44.426 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4f3044c:::benchmark_data_smithi171_122772_object167797:head# 2024-04-20T17:27:44.426 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4f3866e:::benchmark_data_smithi171_122772_object108997:head# 2024-04-20T17:27:44.427 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4f38f8f:::benchmark_data_smithi171_122772_object121422:head# 2024-04-20T17:27:44.427 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4f3a4f7:::benchmark_data_smithi171_122772_object81439:head# 2024-04-20T17:27:44.428 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4f4021f:::benchmark_data_smithi171_122772_object95520:head# 2024-04-20T17:27:44.428 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4f42f00:::benchmark_data_smithi171_122772_object96563:head# 2024-04-20T17:27:44.429 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4f4816e:::benchmark_data_smithi171_122772_object141298:head# 2024-04-20T17:27:44.429 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4f5c922:::benchmark_data_smithi171_122772_object107299:head# 2024-04-20T17:27:44.430 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4f5dfb7:::benchmark_data_smithi171_122772_object148553:head# 2024-04-20T17:27:44.430 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4f656da:::benchmark_data_smithi171_122772_object129109:head# 2024-04-20T17:27:44.431 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4f6fa78:::benchmark_data_smithi171_122772_object110312:head# 2024-04-20T17:27:44.431 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4f7bedd:::benchmark_data_smithi171_122772_object79811:head# 2024-04-20T17:27:44.432 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4f7dfe0:::benchmark_data_smithi171_122772_object83978:head# 2024-04-20T17:27:44.432 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4f83922:::benchmark_data_smithi171_122772_object128050:head# 2024-04-20T17:27:44.433 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4f87857:::benchmark_data_smithi171_122772_object165886:head# 2024-04-20T17:27:44.433 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4f967cd:::benchmark_data_smithi171_122772_object86090:head# 2024-04-20T17:27:44.434 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4fb1161:::benchmark_data_smithi171_122772_object134275:head# 2024-04-20T17:27:44.434 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4fcb7b4:::benchmark_data_smithi171_122772_object112591:head# 2024-04-20T17:27:44.435 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4fdd238:::benchmark_data_smithi171_122772_object153799:head# 2024-04-20T17:27:44.435 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4fe43fd:::benchmark_data_smithi171_122772_object134112:head# 2024-04-20T17:27:44.436 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4febf3c:::benchmark_data_smithi171_122772_object118010:head# 2024-04-20T17:27:44.436 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4ff0969:::benchmark_data_smithi171_122772_object83565:head# 2024-04-20T17:27:44.437 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4ffb77a:::benchmark_data_smithi171_122772_object154331:head# 2024-04-20T17:27:44.437 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a4ffbcc4:::benchmark_data_smithi171_122772_object134316:head# 2024-04-20T17:27:44.438 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a500f577:::benchmark_data_smithi171_122772_object111776:head# 2024-04-20T17:27:44.438 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5017ab7:::benchmark_data_smithi171_122772_object140059:head# 2024-04-20T17:27:44.439 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a50307dc:::benchmark_data_smithi171_122772_object138566:head# 2024-04-20T17:27:44.440 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5033336:::benchmark_data_smithi171_122772_object111872:head# 2024-04-20T17:27:44.440 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5033900:::benchmark_data_smithi171_122772_object93154:head# 2024-04-20T17:27:44.441 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a50354df:::benchmark_data_smithi171_122772_object96088:head# 2024-04-20T17:27:44.441 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5039f21:::benchmark_data_smithi171_122772_object168627:head# 2024-04-20T17:27:44.442 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a503d075:::benchmark_data_smithi171_122772_object155278:head# 2024-04-20T17:27:44.442 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a503d547:::benchmark_data_smithi171_122772_object115203:head# 2024-04-20T17:27:44.443 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a504076e:::benchmark_data_smithi171_122772_object153632:head# 2024-04-20T17:27:44.443 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a505bd58:::benchmark_data_smithi171_122772_object102004:head# 2024-04-20T17:27:44.444 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5063c72:::benchmark_data_smithi171_122772_object96221:head# 2024-04-20T17:27:44.445 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5066d44:::benchmark_data_smithi171_122772_object129815:head# 2024-04-20T17:27:44.445 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5089a0b:::benchmark_data_smithi171_122772_object82826:head# 2024-04-20T17:27:44.446 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5095847:::benchmark_data_smithi171_122772_object167380:head# 2024-04-20T17:27:44.446 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a50a9b74:::benchmark_data_smithi171_122772_object147261:head# 2024-04-20T17:27:44.447 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a50aedb0:::benchmark_data_smithi171_122772_object98539:head# 2024-04-20T17:27:44.447 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a50b0610:::benchmark_data_smithi171_122772_object144424:head# 2024-04-20T17:27:44.448 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a50ca327:::benchmark_data_smithi171_122772_object161469:head# 2024-04-20T17:27:44.448 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a50d8744:::benchmark_data_smithi171_122772_object123057:head# 2024-04-20T17:27:44.449 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a50f0b0e:::benchmark_data_smithi171_122772_object92745:head# 2024-04-20T17:27:44.449 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a50fed09:::benchmark_data_smithi171_122772_object98066:head# 2024-04-20T17:27:44.449 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5110b68:::benchmark_data_smithi171_122772_object116112:head# 2024-04-20T17:27:44.450 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a51121a1:::benchmark_data_smithi171_122772_object88848:head# 2024-04-20T17:27:44.451 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a51244b0:::benchmark_data_smithi171_122772_object108534:head# 2024-04-20T17:27:44.451 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a51340af:::benchmark_data_smithi171_122772_object144806:head# 2024-04-20T17:27:44.452 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a514780c:::benchmark_data_smithi171_122772_object133735:head# 2024-04-20T17:27:44.452 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a51479f6:::benchmark_data_smithi171_122772_object79775:head# 2024-04-20T17:27:44.453 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5148319:::benchmark_data_smithi171_122772_object135675:head# 2024-04-20T17:27:44.453 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a514e4dc:::benchmark_data_smithi171_122772_object112331:head# 2024-04-20T17:27:44.454 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a515b6a5:::benchmark_data_smithi171_122772_object119284:head# 2024-04-20T17:27:44.454 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a51601de:::benchmark_data_smithi171_122772_object150911:head# 2024-04-20T17:27:44.455 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a516046d:::benchmark_data_smithi171_122772_object119103:head# 2024-04-20T17:27:44.455 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a51899f4:::benchmark_data_smithi171_122772_object144373:head# 2024-04-20T17:27:44.456 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5192a5d:::benchmark_data_smithi171_122772_object132494:head# 2024-04-20T17:27:44.457 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5197b6a:::benchmark_data_smithi171_122772_object154181:head# 2024-04-20T17:27:44.457 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a519c9cd:::benchmark_data_smithi171_122772_object160643:head# 2024-04-20T17:27:44.458 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a51b07da:::benchmark_data_smithi171_122772_object84968:head# 2024-04-20T17:27:44.458 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a51cc990:::benchmark_data_smithi171_122772_object117563:head# 2024-04-20T17:27:44.459 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a51cd350:::benchmark_data_smithi171_122772_object108351:head# 2024-04-20T17:27:44.459 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a51ceb9b:::benchmark_data_smithi171_122772_object137690:head# 2024-04-20T17:27:44.460 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a51d6a99:::benchmark_data_smithi171_122772_object125721:head# 2024-04-20T17:27:44.460 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a51da7e5:::benchmark_data_smithi171_122772_object154921:head# 2024-04-20T17:27:44.461 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a51e1d5f:::benchmark_data_smithi171_122772_object78811:head# 2024-04-20T17:27:44.461 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a51e4fd0:::benchmark_data_smithi171_122772_object95739:head# 2024-04-20T17:27:44.462 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a51e8806:::benchmark_data_smithi171_122772_object164719:head# 2024-04-20T17:27:44.463 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a51f1139:::benchmark_data_smithi171_122772_object144790:head# 2024-04-20T17:27:44.463 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a51f3888:::benchmark_data_smithi171_122772_object86793:head# 2024-04-20T17:27:44.464 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5212f26:::benchmark_data_smithi171_122772_object163330:head# 2024-04-20T17:27:44.464 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5215dc6:::benchmark_data_smithi171_122772_object150937:head# 2024-04-20T17:27:44.465 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a521d102:::benchmark_data_smithi171_122772_object105094:head# 2024-04-20T17:27:44.465 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5223f84:::benchmark_data_smithi171_122772_object162943:head# 2024-04-20T17:27:44.466 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a522ee0e:::benchmark_data_smithi171_122772_object157116:head# 2024-04-20T17:27:44.466 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a524dc3f:::benchmark_data_smithi171_122772_object108039:head# 2024-04-20T17:27:44.467 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5254a6f:::benchmark_data_smithi171_122772_object115154:head# 2024-04-20T17:27:44.467 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a525cddf:::benchmark_data_smithi171_122772_object152585:head# 2024-04-20T17:27:44.468 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a526402c:::benchmark_data_smithi171_122772_object151222:head# 2024-04-20T17:27:44.468 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a526d6dd:::benchmark_data_smithi171_122772_object130573:head# 2024-04-20T17:27:44.469 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a527413f:::benchmark_data_smithi171_122772_object142344:head# 2024-04-20T17:27:44.469 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a527418d:::benchmark_data_smithi171_122772_object140261:head# 2024-04-20T17:27:44.470 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a528127d:::benchmark_data_smithi171_122772_object117527:head# 2024-04-20T17:27:44.471 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a529ad13:::benchmark_data_smithi171_122772_object134981:head# 2024-04-20T17:27:44.471 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a529d8b7:::benchmark_data_smithi171_122772_object125163:head# 2024-04-20T17:27:44.471 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a52a43c2:::benchmark_data_smithi171_122772_object101441:head# 2024-04-20T17:27:44.472 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a52a9061:::benchmark_data_smithi171_122772_object113167:head# 2024-04-20T17:27:44.472 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a52af482:::benchmark_data_smithi171_122772_object119401:head# 2024-04-20T17:27:44.473 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a52b0e47:::benchmark_data_smithi171_122772_object145423:head# 2024-04-20T17:27:44.473 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a52bb233:::benchmark_data_smithi171_122772_object143611:head# 2024-04-20T17:27:44.474 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a52bcbd0:::benchmark_data_smithi171_122772_object117856:head# 2024-04-20T17:27:44.474 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a52be18f:::benchmark_data_smithi171_122772_object81438:head# 2024-04-20T17:27:44.475 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a52c9908:::benchmark_data_smithi171_122772_object99736:head# 2024-04-20T17:27:44.475 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a52cbadd:::benchmark_data_smithi171_122772_object102601:head# 2024-04-20T17:27:44.476 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a52d378b:::benchmark_data_smithi171_122772_object88486:head# 2024-04-20T17:27:44.476 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a52e7431:::benchmark_data_smithi171_122772_object94129:head# 2024-04-20T17:27:44.477 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a52ef067:::benchmark_data_smithi171_122772_object145670:head# 2024-04-20T17:27:44.477 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a52f7bec:::benchmark_data_smithi171_122772_object142199:head# 2024-04-20T17:27:44.478 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a52fc78f:::benchmark_data_smithi171_122772_object147153:head# 2024-04-20T17:27:44.478 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5309ba5:::benchmark_data_smithi171_122772_object124340:head# 2024-04-20T17:27:44.478 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5312bf7:::benchmark_data_smithi171_122772_object163743:head# 2024-04-20T17:27:44.479 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a531be4c:::benchmark_data_smithi171_122772_object130704:head# 2024-04-20T17:27:44.479 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a532e4ff:::benchmark_data_smithi171_122772_object122234:head# 2024-04-20T17:27:44.480 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a533732f:::benchmark_data_smithi171_122772_object118011:head# 2024-04-20T17:27:44.481 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a533e667:::benchmark_data_smithi171_122772_object154357:head# 2024-04-20T17:27:44.481 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a536669b:::benchmark_data_smithi171_122772_object92353:head# 2024-04-20T17:27:44.482 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5371959:::benchmark_data_smithi171_122772_object147710:head# 2024-04-20T17:27:44.482 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a53823d3:::benchmark_data_smithi171_122772_object121982:head# 2024-04-20T17:27:44.483 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5388870:::benchmark_data_smithi171_122772_object170033:head# 2024-04-20T17:27:44.484 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a53987ac:::benchmark_data_smithi171_122772_object169871:head# 2024-04-20T17:27:44.484 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a53a7bd9:::benchmark_data_smithi171_122772_object113325:head# 2024-04-20T17:27:44.484 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a53a7fe3:::benchmark_data_smithi171_122772_object84469:head# 2024-04-20T17:27:44.485 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a53c0738:::benchmark_data_smithi171_122772_object134715:head# 2024-04-20T17:27:44.485 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a53cca88:::benchmark_data_smithi171_122772_object162615:head# 2024-04-20T17:27:44.486 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a53f2122:::benchmark_data_smithi171_122772_object162627:head# 2024-04-20T17:27:44.486 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a540876e:::benchmark_data_smithi171_122772_object119114:head# 2024-04-20T17:27:44.487 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5419f5b:::benchmark_data_smithi171_122772_object97388:head# 2024-04-20T17:27:44.487 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a54217d8:::benchmark_data_smithi171_122772_object168517:head# 2024-04-20T17:27:44.488 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a542c987:::benchmark_data_smithi171_122772_object161738:head# 2024-04-20T17:27:44.488 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a54429c0:::benchmark_data_smithi171_122772_object147707:head# 2024-04-20T17:27:44.489 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a545cce9:::benchmark_data_smithi171_122772_object142597:head# 2024-04-20T17:27:44.489 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a54690dd:::benchmark_data_smithi171_122772_object117446:head# 2024-04-20T17:27:44.490 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a546c3c9:::benchmark_data_smithi171_122772_object118549:head# 2024-04-20T17:27:44.491 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5483aa0:::benchmark_data_smithi171_122772_object79922:head# 2024-04-20T17:27:44.492 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5485ceb:::benchmark_data_smithi171_122772_object101744:head# 2024-04-20T17:27:44.492 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5486225:::benchmark_data_smithi171_122772_object111890:head# 2024-04-20T17:27:44.493 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a548cb1f:::benchmark_data_smithi171_122772_object82499:head# 2024-04-20T17:27:44.493 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5498706:::benchmark_data_smithi171_122772_object84349:head# 2024-04-20T17:27:44.494 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a549a011:::benchmark_data_smithi171_122772_object89405:head# 2024-04-20T17:27:44.495 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a549b6a3:::benchmark_data_smithi171_122772_object114097:head# 2024-04-20T17:27:44.495 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a54a4a73:::benchmark_data_smithi171_122772_object112447:head# 2024-04-20T17:27:44.495 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a54a5fdf:::benchmark_data_smithi171_122772_object119421:head# 2024-04-20T17:27:44.496 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a54a686b:::benchmark_data_smithi171_122772_object85968:head# 2024-04-20T17:27:44.496 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a54a6a4f:::benchmark_data_smithi171_122772_object159061:head# 2024-04-20T17:27:44.497 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a54adbb1:::benchmark_data_smithi171_122772_object169848:head# 2024-04-20T17:27:44.497 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a54b726e:::benchmark_data_smithi171_122772_object90240:head# 2024-04-20T17:27:44.498 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a54b7a7e:::benchmark_data_smithi171_122772_object143399:head# 2024-04-20T17:27:44.498 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a54bf63a:::benchmark_data_smithi171_122772_object105723:head# 2024-04-20T17:27:44.499 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a54c3741:::benchmark_data_smithi171_122772_object88207:head# 2024-04-20T17:27:44.499 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a54c4f42:::benchmark_data_smithi171_122772_object125659:head# 2024-04-20T17:27:44.500 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a54ca07b:::benchmark_data_smithi171_122772_object109291:head# 2024-04-20T17:27:44.500 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a54d1bde:::benchmark_data_smithi171_122772_object88136:head# 2024-04-20T17:27:44.501 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a54d364d:::benchmark_data_smithi171_122772_object111713:head# 2024-04-20T17:27:44.501 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a54dbf9e:::benchmark_data_smithi171_122772_object111031:head# 2024-04-20T17:27:44.502 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a54eda52:::benchmark_data_smithi171_122772_object102953:head# 2024-04-20T17:27:44.502 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a54f183e:::benchmark_data_smithi171_122772_object162998:head# 2024-04-20T17:27:44.503 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a54ff0fb:::benchmark_data_smithi171_122772_object152544:head# 2024-04-20T17:27:44.503 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a550636b:::benchmark_data_smithi171_122772_object137651:head# 2024-04-20T17:27:44.504 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a550b3f9:::benchmark_data_smithi171_122772_object139146:head# 2024-04-20T17:27:44.505 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5510191:::benchmark_data_smithi171_122772_object80840:head# 2024-04-20T17:27:44.505 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5514cd3:::benchmark_data_smithi171_122772_object160625:head# 2024-04-20T17:27:44.506 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a55264a1:::benchmark_data_smithi171_122772_object80599:head# 2024-04-20T17:27:44.506 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a552f6b7:::benchmark_data_smithi171_122772_object160628:head# 2024-04-20T17:27:44.507 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5531254:::benchmark_data_smithi171_122772_object165763:head# 2024-04-20T17:27:44.507 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5535791:::benchmark_data_smithi171_122772_object167930:head# 2024-04-20T17:27:44.508 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a553b517:::benchmark_data_smithi171_122772_object152562:head# 2024-04-20T17:27:44.508 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5545e8a:::benchmark_data_smithi171_122772_object149288:head# 2024-04-20T17:27:44.509 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a555c7c2:::benchmark_data_smithi171_122772_object149102:head# 2024-04-20T17:27:44.509 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a555dc47:::benchmark_data_smithi171_122772_object137405:head# 2024-04-20T17:27:44.510 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5563037:::benchmark_data_smithi171_122772_object92721:head# 2024-04-20T17:27:44.510 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5569d4b:::benchmark_data_smithi171_122772_object95069:head# 2024-04-20T17:27:44.511 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a55955cc:::benchmark_data_smithi171_122772_object139890:head# 2024-04-20T17:27:44.511 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a559c6df:::benchmark_data_smithi171_122772_object148845:head# 2024-04-20T17:27:44.512 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a559e32c:::benchmark_data_smithi171_122772_object140672:head# 2024-04-20T17:27:44.512 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a55a654d:::benchmark_data_smithi171_122772_object164599:head# 2024-04-20T17:27:44.513 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a55c211a:::benchmark_data_smithi171_122772_object123906:head# 2024-04-20T17:27:44.513 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a55cfcf5:::benchmark_data_smithi171_122772_object98483:head# 2024-04-20T17:27:44.514 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a55d3304:::benchmark_data_smithi171_122772_object146785:head# 2024-04-20T17:27:44.519 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a55e668b:::benchmark_data_smithi171_122772_object100675:head# 2024-04-20T17:27:44.519 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a55fa5fe:::benchmark_data_smithi171_122772_object88099:head# 2024-04-20T17:27:44.519 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a55fcc80:::benchmark_data_smithi171_122772_object144048:head# 2024-04-20T17:27:44.519 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a55fe3b1:::benchmark_data_smithi171_122772_object95474:head# 2024-04-20T17:27:44.519 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5600f78:::benchmark_data_smithi171_122772_object128838:head# 2024-04-20T17:27:44.519 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5606493:::benchmark_data_smithi171_122772_object140688:head# 2024-04-20T17:27:44.519 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5641d95:::benchmark_data_smithi171_122772_object165319:head# 2024-04-20T17:27:44.519 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a565ce64:::benchmark_data_smithi171_122772_object102394:head# 2024-04-20T17:27:44.519 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a567d049:::benchmark_data_smithi171_122772_object167060:head# 2024-04-20T17:27:44.520 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5682629:::benchmark_data_smithi171_122772_object144294:head# 2024-04-20T17:27:44.520 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a568a86a:::benchmark_data_smithi171_122772_object123469:head# 2024-04-20T17:27:44.520 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5697adf:::benchmark_data_smithi171_122772_object140571:head# 2024-04-20T17:27:44.521 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a56ac76b:::benchmark_data_smithi171_122772_object83662:head# 2024-04-20T17:27:44.521 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a56bc3c6:::benchmark_data_smithi171_122772_object147929:head# 2024-04-20T17:27:44.521 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a56c73b1:::benchmark_data_smithi171_122772_object128974:head# 2024-04-20T17:27:44.522 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a56d4126:::benchmark_data_smithi171_122772_object85671:head# 2024-04-20T17:27:44.523 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a56d9e99:::benchmark_data_smithi171_122772_object102184:head# 2024-04-20T17:27:44.523 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a56ed56c:::benchmark_data_smithi171_122772_object93844:head# 2024-04-20T17:27:44.524 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a56fb52d:::benchmark_data_smithi171_122772_object109333:head# 2024-04-20T17:27:44.524 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a57063e4:::benchmark_data_smithi171_122772_object138831:head# 2024-04-20T17:27:44.525 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5708641:::benchmark_data_smithi171_122772_object169476:head# 2024-04-20T17:27:44.525 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a570c222:::benchmark_data_smithi171_122772_object150133:head# 2024-04-20T17:27:44.526 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a57179ba:::benchmark_data_smithi171_122772_object105023:head# 2024-04-20T17:27:44.526 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a57213bd:::benchmark_data_smithi171_122772_object114786:head# 2024-04-20T17:27:44.527 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a572c447:::benchmark_data_smithi171_122772_object163049:head# 2024-04-20T17:27:44.527 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a573aff0:::benchmark_data_smithi171_122772_object151913:head# 2024-04-20T17:27:44.528 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a57537bd:::benchmark_data_smithi171_122772_object132323:head# 2024-04-20T17:27:44.528 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5767a86:::benchmark_data_smithi171_122772_object111434:head# 2024-04-20T17:27:44.529 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5771a8a:::benchmark_data_smithi171_122772_object113968:head# 2024-04-20T17:27:44.529 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a577d92d:::benchmark_data_smithi171_122772_object130624:head# 2024-04-20T17:27:44.530 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a579afef:::benchmark_data_smithi171_122772_object164615:head# 2024-04-20T17:27:44.530 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a57a351a:::benchmark_data_smithi171_122772_object95067:head# 2024-04-20T17:27:44.531 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a57b8d0b:::benchmark_data_smithi171_122772_object110612:head# 2024-04-20T17:27:44.531 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a57c7d48:::benchmark_data_smithi171_122772_object122576:head# 2024-04-20T17:27:44.532 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a57cc70f:::benchmark_data_smithi171_122772_object110366:head# 2024-04-20T17:27:44.532 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a57d4e96:::benchmark_data_smithi171_122772_object90573:head# 2024-04-20T17:27:44.533 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a57db99c:::benchmark_data_smithi171_122772_object102985:head# 2024-04-20T17:27:44.533 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a57f01f4:::benchmark_data_smithi171_122772_object158850:head# 2024-04-20T17:27:44.534 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a57fdaa9:::benchmark_data_smithi171_122772_object80823:head# 2024-04-20T17:27:44.534 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a57feeef:::benchmark_data_smithi171_122772_object97415:head# 2024-04-20T17:27:44.535 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5817421:::benchmark_data_smithi171_122772_object148939:head# 2024-04-20T17:27:44.535 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a58181ae:::benchmark_data_smithi171_122772_object113602:head# 2024-04-20T17:27:44.536 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a581cf27:::benchmark_data_smithi171_122772_object157591:head# 2024-04-20T17:27:44.536 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5832821:::benchmark_data_smithi171_122772_object134873:head# 2024-04-20T17:27:44.537 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5833ac3:::benchmark_data_smithi171_122772_object96743:head# 2024-04-20T17:27:44.537 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5836350:::benchmark_data_smithi171_122772_object120277:head# 2024-04-20T17:27:44.538 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a585eb1f:::benchmark_data_smithi171_122772_object151566:head# 2024-04-20T17:27:44.538 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a58635f6:::benchmark_data_smithi171_122772_object139126:head# 2024-04-20T17:27:44.539 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5868f83:::benchmark_data_smithi171_122772_object93258:head# 2024-04-20T17:27:44.539 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a587142d:::benchmark_data_smithi171_122772_object158607:head# 2024-04-20T17:27:44.540 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a588983a:::benchmark_data_smithi171_122772_object161905:head# 2024-04-20T17:27:44.540 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5892c0b:::benchmark_data_smithi171_122772_object113194:head# 2024-04-20T17:27:44.541 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a58947bd:::benchmark_data_smithi171_122772_object163764:head# 2024-04-20T17:27:44.541 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a589dcd7:::benchmark_data_smithi171_122772_object110456:head# 2024-04-20T17:27:44.542 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a58a5407:::benchmark_data_smithi171_122772_object169168:head# 2024-04-20T17:27:44.542 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a58a7d84:::benchmark_data_smithi171_122772_object118277:head# 2024-04-20T17:27:44.543 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a58a8b78:::benchmark_data_smithi171_122772_object144904:head# 2024-04-20T17:27:44.543 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a58b20db:::benchmark_data_smithi171_122772_object86497:head# 2024-04-20T17:27:44.544 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a58b895a:::benchmark_data_smithi171_122772_object86293:head# 2024-04-20T17:27:44.544 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a58c9cda:::benchmark_data_smithi171_122772_object86708:head# 2024-04-20T17:27:44.545 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a58d20a7:::benchmark_data_smithi171_122772_object148488:head# 2024-04-20T17:27:44.545 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a594090d:::benchmark_data_smithi171_122772_object97075:head# 2024-04-20T17:27:44.546 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5942447:::benchmark_data_smithi171_122772_object138848:head# 2024-04-20T17:27:44.546 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5949589:::benchmark_data_smithi171_122772_object146923:head# 2024-04-20T17:27:44.547 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a594d2ce:::benchmark_data_smithi171_122772_object137918:head# 2024-04-20T17:27:44.547 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a598e16c:::benchmark_data_smithi171_122772_object118073:head# 2024-04-20T17:27:44.548 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a59a06c0:::benchmark_data_smithi171_122772_object119846:head# 2024-04-20T17:27:44.548 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a59a15c7:::benchmark_data_smithi171_122772_object128784:head# 2024-04-20T17:27:44.549 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a59a7bfa:::benchmark_data_smithi171_122772_object139466:head# 2024-04-20T17:27:44.549 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a59ae31f:::benchmark_data_smithi171_122772_object163466:head# 2024-04-20T17:27:44.550 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a59af4cf:::benchmark_data_smithi171_122772_object132905:head# 2024-04-20T17:27:44.550 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a59c4e9a:::benchmark_data_smithi171_122772_object98788:head# 2024-04-20T17:27:44.551 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a59e01b2:::benchmark_data_smithi171_122772_object102350:head# 2024-04-20T17:27:44.551 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a59e35df:::benchmark_data_smithi171_122772_object124168:head# 2024-04-20T17:27:44.552 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a59e4e7f:::benchmark_data_smithi171_122772_object88081:head# 2024-04-20T17:27:44.552 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a59f0302:::benchmark_data_smithi171_122772_object136311:head# 2024-04-20T17:27:44.553 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a59fb6c4:::benchmark_data_smithi171_122772_object108148:head# 2024-04-20T17:27:44.553 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5a00629:::benchmark_data_smithi171_122772_object168816:head# 2024-04-20T17:27:44.554 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5a02a67:::benchmark_data_smithi171_122772_object125766:head# 2024-04-20T17:27:44.554 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5a087e7:::benchmark_data_smithi171_122772_object166080:head# 2024-04-20T17:27:44.555 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5a30d13:::benchmark_data_smithi171_122772_object83099:head# 2024-04-20T17:27:44.555 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5a3c064:::benchmark_data_smithi171_122772_object114766:head# 2024-04-20T17:27:44.555 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5a4c914:::benchmark_data_smithi171_122772_object88684:head# 2024-04-20T17:27:44.556 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5a58517:::benchmark_data_smithi171_122772_object135471:head# 2024-04-20T17:27:44.557 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5a597fc:::benchmark_data_smithi171_122772_object111651:head# 2024-04-20T17:27:44.557 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5a5fc4a:::benchmark_data_smithi171_122772_object81469:head# 2024-04-20T17:27:44.558 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5a74780:::benchmark_data_smithi171_122772_object127481:head# 2024-04-20T17:27:44.558 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5a76ab1:::benchmark_data_smithi171_122772_object158082:head# 2024-04-20T17:27:44.558 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5a78e38:::benchmark_data_smithi171_122772_object142935:head# 2024-04-20T17:27:44.559 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5a7a760:::benchmark_data_smithi171_122772_object160155:head# 2024-04-20T17:27:44.560 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5a7ffa8:::benchmark_data_smithi171_122772_object93624:head# 2024-04-20T17:27:44.560 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5a804be:::benchmark_data_smithi171_122772_object136564:head# 2024-04-20T17:27:44.561 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5a855d7:::benchmark_data_smithi171_122772_object138118:head# 2024-04-20T17:27:44.561 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5a8a151:::benchmark_data_smithi171_122772_object113605:head# 2024-04-20T17:27:44.562 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5a9296e:::benchmark_data_smithi171_122772_object123639:head# 2024-04-20T17:27:44.562 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5a9c3bc:::benchmark_data_smithi171_122772_object81191:head# 2024-04-20T17:27:44.563 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5aa8a5d:::benchmark_data_smithi171_122772_object110658:head# 2024-04-20T17:27:44.563 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5ab996d:::benchmark_data_smithi171_122772_object103237:head# 2024-04-20T17:27:44.563 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5ad4316:::benchmark_data_smithi171_122772_object163276:head# 2024-04-20T17:27:44.564 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5ae5e19:::benchmark_data_smithi171_122772_object119675:head# 2024-04-20T17:27:44.564 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5af7309:::benchmark_data_smithi171_122772_object130740:head# 2024-04-20T17:27:44.565 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5b0775d:::benchmark_data_smithi171_122772_object96535:head# 2024-04-20T17:27:44.565 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5b07d78:::benchmark_data_smithi171_122772_object100752:head# 2024-04-20T17:27:44.566 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5b13038:::benchmark_data_smithi171_122772_object165236:head# 2024-04-20T17:27:44.566 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5b19810:::benchmark_data_smithi171_122772_object90428:head# 2024-04-20T17:27:44.567 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5b2bb0c:::benchmark_data_smithi171_122772_object152138:head# 2024-04-20T17:27:44.567 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5b42f9d:::benchmark_data_smithi171_122772_object150800:head# 2024-04-20T17:27:44.568 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5b5e4ea:::benchmark_data_smithi171_122772_object101181:head# 2024-04-20T17:27:44.568 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5b609bc:::benchmark_data_smithi171_122772_object120998:head# 2024-04-20T17:27:44.569 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5b634fc:::benchmark_data_smithi171_122772_object134012:head# 2024-04-20T17:27:44.570 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5b763b2:::benchmark_data_smithi171_122772_object82302:head# 2024-04-20T17:27:44.570 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5b76dda:::benchmark_data_smithi171_122772_object81809:head# 2024-04-20T17:27:44.571 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5b833b1:::benchmark_data_smithi171_122772_object150751:head# 2024-04-20T17:27:44.571 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5b97ca2:::benchmark_data_smithi171_122772_object87643:head# 2024-04-20T17:27:44.572 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5bb6bce:::benchmark_data_smithi171_122772_object168451:head# 2024-04-20T17:27:44.572 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5bc6162:::benchmark_data_smithi171_122772_object148099:head# 2024-04-20T17:27:44.573 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5bc617d:::benchmark_data_smithi171_122772_object130096:head# 2024-04-20T17:27:44.574 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5bc8388:::benchmark_data_smithi171_122772_object132026:head# 2024-04-20T17:27:44.574 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5bc97b5:::benchmark_data_smithi171_122772_object96604:head# 2024-04-20T17:27:44.575 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5bd99ad:::benchmark_data_smithi171_122772_object152221:head# 2024-04-20T17:27:44.575 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5bdda12:::benchmark_data_smithi171_122772_object92360:head# 2024-04-20T17:27:44.576 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5be103c:::benchmark_data_smithi171_122772_object87855:head# 2024-04-20T17:27:44.577 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5be99bc:::benchmark_data_smithi171_122772_object107358:head# 2024-04-20T17:27:44.577 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5c05953:::benchmark_data_smithi171_122772_object157474:head# 2024-04-20T17:27:44.578 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5c16e1b:::benchmark_data_smithi171_122772_object113006:head# 2024-04-20T17:27:44.578 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5c26842:::benchmark_data_smithi171_122772_object87732:head# 2024-04-20T17:27:44.579 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5c2c43c:::benchmark_data_smithi171_122772_object125067:head# 2024-04-20T17:27:44.579 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5c2d588:::benchmark_data_smithi171_122772_object137754:head# 2024-04-20T17:27:44.580 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5c30249:::benchmark_data_smithi171_122772_object94826:head# 2024-04-20T17:27:44.581 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5c33eac:::benchmark_data_smithi171_122772_object84422:head# 2024-04-20T17:27:44.581 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5c344a0:::benchmark_data_smithi171_122772_object146565:head# 2024-04-20T17:27:44.582 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5c34d19:::benchmark_data_smithi171_122772_object128962:head# 2024-04-20T17:27:44.582 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5c426f4:::benchmark_data_smithi171_122772_object141566:head# 2024-04-20T17:27:44.583 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5c46eab:::benchmark_data_smithi171_122772_object169467:head# 2024-04-20T17:27:44.583 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5c48cea:::benchmark_data_smithi171_122772_object78237:head# 2024-04-20T17:27:44.584 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5c553f5:::benchmark_data_smithi171_122772_object130133:head# 2024-04-20T17:27:44.584 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5c5e0c7:::benchmark_data_smithi171_122772_object158396:head# 2024-04-20T17:27:44.585 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5c61da6:::benchmark_data_smithi171_122772_object107930:head# 2024-04-20T17:27:44.585 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5c6acbd:::benchmark_data_smithi171_122772_object135614:head# 2024-04-20T17:27:44.586 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5c6d52f:::benchmark_data_smithi171_122772_object96643:head# 2024-04-20T17:27:44.586 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5c76275:::benchmark_data_smithi171_122772_object97216:head# 2024-04-20T17:27:44.587 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5c7b2be:::benchmark_data_smithi171_122772_object159743:head# 2024-04-20T17:27:44.587 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5c87d85:::benchmark_data_smithi171_122772_object128809:head# 2024-04-20T17:27:44.588 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5c8be18:::benchmark_data_smithi171_122772_object116257:head# 2024-04-20T17:27:44.588 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5c8fe72:::benchmark_data_smithi171_122772_object142191:head# 2024-04-20T17:27:44.589 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5c94fb6:::benchmark_data_smithi171_122772_object159728:head# 2024-04-20T17:27:44.589 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5cce885:::benchmark_data_smithi171_122772_object116544:head# 2024-04-20T17:27:44.590 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5cd3820:::benchmark_data_smithi171_122772_object113162:head# 2024-04-20T17:27:44.590 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5cdacf4:::benchmark_data_smithi171_122772_object89603:head# 2024-04-20T17:27:44.591 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5cf1ed2:::benchmark_data_smithi171_122772_object136053:head# 2024-04-20T17:27:44.591 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5cf4642:::benchmark_data_smithi171_122772_object89263:head# 2024-04-20T17:27:44.592 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5cf4895:::benchmark_data_smithi171_122772_object110299:head# 2024-04-20T17:27:44.592 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5cfa8d6:::benchmark_data_smithi171_122772_object80271:head# 2024-04-20T17:27:44.593 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5d03d95:::benchmark_data_smithi171_122772_object147568:head# 2024-04-20T17:27:44.593 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5d064a8:::benchmark_data_smithi171_122772_object98866:head# 2024-04-20T17:27:44.594 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5d19740:::benchmark_data_smithi171_122772_object153760:head# 2024-04-20T17:27:44.595 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5d1b35c:::benchmark_data_smithi171_122772_object100985:head# 2024-04-20T17:27:44.595 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5d28db3:::benchmark_data_smithi171_122772_object124700:head# 2024-04-20T17:27:44.596 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5d3478f:::benchmark_data_smithi171_122772_object102488:head# 2024-04-20T17:27:44.596 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5d5b33e:::benchmark_data_smithi171_122772_object146489:head# 2024-04-20T17:27:44.597 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5d5f7b1:::benchmark_data_smithi171_122772_object103997:head# 2024-04-20T17:27:44.598 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5d64ac0:::benchmark_data_smithi171_122772_object138545:head# 2024-04-20T17:27:44.598 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5d68b7e:::benchmark_data_smithi171_122772_object99317:head# 2024-04-20T17:27:44.598 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5d6bd31:::benchmark_data_smithi171_122772_object99999:head# 2024-04-20T17:27:44.599 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5d76865:::benchmark_data_smithi171_122772_object93559:head# 2024-04-20T17:27:44.600 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5d7e3e8:::benchmark_data_smithi171_122772_object125991:head# 2024-04-20T17:27:44.600 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5d91efc:::benchmark_data_smithi171_122772_object95016:head# 2024-04-20T17:27:44.601 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5d95be2:::benchmark_data_smithi171_122772_object101628:head# 2024-04-20T17:27:44.601 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5da2570:::benchmark_data_smithi171_122772_object163502:head# 2024-04-20T17:27:44.601 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5dab9f2:::benchmark_data_smithi171_122772_object95063:head# 2024-04-20T17:27:44.602 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5db3a48:::benchmark_data_smithi171_122772_object106736:head# 2024-04-20T17:27:44.602 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5dbb8a2:::benchmark_data_smithi171_122772_object115615:head# 2024-04-20T17:27:44.603 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5dc5d9e:::benchmark_data_smithi171_122772_object155115:head# 2024-04-20T17:27:44.603 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5dc6dde:::benchmark_data_smithi171_122772_object145510:head# 2024-04-20T17:27:44.604 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5dc83bf:::benchmark_data_smithi171_122772_object95633:head# 2024-04-20T17:27:44.604 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5dcf10f:::benchmark_data_smithi171_122772_object111510:head# 2024-04-20T17:27:44.605 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5dd1c8a:::benchmark_data_smithi171_122772_object122049:head# 2024-04-20T17:27:44.605 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5dd3c78:::benchmark_data_smithi171_122772_object132053:head# 2024-04-20T17:27:44.606 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5dd5ba0:::benchmark_data_smithi171_122772_object116146:head# 2024-04-20T17:27:44.607 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5de497d:::benchmark_data_smithi171_122772_object146383:head# 2024-04-20T17:27:44.608 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5de5acc:::benchmark_data_smithi171_122772_object114749:head# 2024-04-20T17:27:44.608 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5dfbf66:::benchmark_data_smithi171_122772_object142468:head# 2024-04-20T17:27:44.609 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5e15c9e:::benchmark_data_smithi171_122772_object122399:head# 2024-04-20T17:27:44.609 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5e2e23c:::benchmark_data_smithi171_122772_object116034:head# 2024-04-20T17:27:44.610 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5e34d86:::benchmark_data_smithi171_122772_object109799:head# 2024-04-20T17:27:44.610 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5e45edc:::benchmark_data_smithi171_122772_object97324:head# 2024-04-20T17:27:44.611 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5e48d5e:::benchmark_data_smithi171_122772_object152718:head# 2024-04-20T17:27:44.611 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5e49ea7:::benchmark_data_smithi171_122772_object164326:head# 2024-04-20T17:27:44.612 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5e4b340:::benchmark_data_smithi171_122772_object127154:head# 2024-04-20T17:27:44.612 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5e52f1d:::benchmark_data_smithi171_122772_object120341:head# 2024-04-20T17:27:44.612 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5e60de4:::benchmark_data_smithi171_122772_object145491:head# 2024-04-20T17:27:44.613 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5e625f8:::benchmark_data_smithi171_122772_object90768:head# 2024-04-20T17:27:44.614 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5e80877:::benchmark_data_smithi171_122772_object159150:head# 2024-04-20T17:27:44.614 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5e80f23:::benchmark_data_smithi171_122772_object89957:head# 2024-04-20T17:27:44.615 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5e87744:::benchmark_data_smithi171_122772_object169240:head# 2024-04-20T17:27:44.615 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5e9706d:::benchmark_data_smithi171_122772_object87018:head# 2024-04-20T17:27:44.616 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5eb032c:::benchmark_data_smithi171_122772_object89744:head# 2024-04-20T17:27:44.616 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5ebad05:::benchmark_data_smithi171_122772_object108265:head# 2024-04-20T17:27:44.617 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5ec284d:::benchmark_data_smithi171_122772_object98367:head# 2024-04-20T17:27:44.617 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5ec88cb:::benchmark_data_smithi171_122772_object103447:head# 2024-04-20T17:27:44.618 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5ed1523:::benchmark_data_smithi171_122772_object108104:head# 2024-04-20T17:27:44.619 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5ed31a5:::benchmark_data_smithi171_122772_object108152:head# 2024-04-20T17:27:44.619 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5ee3011:::benchmark_data_smithi171_122772_object119369:head# 2024-04-20T17:27:44.620 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5ef2a28:::benchmark_data_smithi171_122772_object156654:head# 2024-04-20T17:27:44.620 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5efc32e:::benchmark_data_smithi171_122772_object79018:head# 2024-04-20T17:27:44.621 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5f05709:::benchmark_data_smithi171_122772_object137671:head# 2024-04-20T17:27:44.621 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5f0704b:::benchmark_data_smithi171_122772_object106553:head# 2024-04-20T17:27:44.622 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5f36946:::benchmark_data_smithi171_122772_object155870:head# 2024-04-20T17:27:44.622 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5f3792c:::benchmark_data_smithi171_122772_object93229:head# 2024-04-20T17:27:44.623 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5f3ed55:::benchmark_data_smithi171_122772_object152812:head# 2024-04-20T17:27:44.623 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5f610e4:::benchmark_data_smithi171_122772_object146823:head# 2024-04-20T17:27:44.624 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5f68df1:::benchmark_data_smithi171_122772_object145810:head# 2024-04-20T17:27:44.624 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5f6eef9:::benchmark_data_smithi171_122772_object134779:head# 2024-04-20T17:27:44.625 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5f726fa:::benchmark_data_smithi171_122772_object146092:head# 2024-04-20T17:27:44.625 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5f759b2:::benchmark_data_smithi171_122772_object152164:head# 2024-04-20T17:27:44.626 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5f76a3a:::benchmark_data_smithi171_122772_object167336:head# 2024-04-20T17:27:44.627 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5f7aa67:::benchmark_data_smithi171_122772_object136299:head# 2024-04-20T17:27:44.628 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5f8a759:::benchmark_data_smithi171_122772_object148494:head# 2024-04-20T17:27:44.628 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5f9a88f:::benchmark_data_smithi171_122772_object94491:head# 2024-04-20T17:27:44.628 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5fa68b2:::benchmark_data_smithi171_122772_object120320:head# 2024-04-20T17:27:44.629 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5fb1f53:::benchmark_data_smithi171_122772_object135943:head# 2024-04-20T17:27:44.629 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5fb784e:::benchmark_data_smithi171_122772_object138552:head# 2024-04-20T17:27:44.630 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5fbfa1e:::benchmark_data_smithi171_122772_object117333:head# 2024-04-20T17:27:44.630 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5fc66b7:::benchmark_data_smithi171_122772_object137505:head# 2024-04-20T17:27:44.631 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5ff342a:::benchmark_data_smithi171_122772_object155725:head# 2024-04-20T17:27:44.631 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a5ff5a95:::benchmark_data_smithi171_122772_object146504:head# 2024-04-20T17:27:44.632 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a60140d2:::benchmark_data_smithi171_122772_object119940:head# 2024-04-20T17:27:44.632 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6030d2e:::benchmark_data_smithi171_122772_object169360:head# 2024-04-20T17:27:44.633 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6042569:::benchmark_data_smithi171_122772_object88491:head# 2024-04-20T17:27:44.633 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a604bddc:::benchmark_data_smithi171_122772_object109546:head# 2024-04-20T17:27:44.634 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a606a3a9:::benchmark_data_smithi171_122772_object80916:head# 2024-04-20T17:27:44.634 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a606e11e:::benchmark_data_smithi171_122772_object93144:head# 2024-04-20T17:27:44.635 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a60760c6:::benchmark_data_smithi171_122772_object94456:head# 2024-04-20T17:27:44.635 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a608b901:::benchmark_data_smithi171_122772_object140105:head# 2024-04-20T17:27:44.636 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a608e2f4:::benchmark_data_smithi171_122772_object106213:head# 2024-04-20T17:27:44.636 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6095de0:::benchmark_data_smithi171_122772_object92799:head# 2024-04-20T17:27:44.637 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a609c4c1:::benchmark_data_smithi171_122772_object125793:head# 2024-04-20T17:27:44.637 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a60a4698:::benchmark_data_smithi171_122772_object137997:head# 2024-04-20T17:27:44.637 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a60a626a:::benchmark_data_smithi171_122772_object134697:head# 2024-04-20T17:27:44.638 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a60a939c:::benchmark_data_smithi171_122772_object148627:head# 2024-04-20T17:27:44.639 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a60b2acc:::benchmark_data_smithi171_122772_object146898:head# 2024-04-20T17:27:44.639 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a60bcfcf:::benchmark_data_smithi171_122772_object89364:head# 2024-04-20T17:27:44.640 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a60d392d:::benchmark_data_smithi171_122772_object142186:head# 2024-04-20T17:27:44.640 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a60d6113:::benchmark_data_smithi171_122772_object78639:head# 2024-04-20T17:27:44.641 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a60d9375:::benchmark_data_smithi171_122772_object91982:head# 2024-04-20T17:27:44.641 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a60e1950:::benchmark_data_smithi171_122772_object94264:head# 2024-04-20T17:27:44.642 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a60eeaf3:::benchmark_data_smithi171_122772_object152627:head# 2024-04-20T17:27:44.642 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a60f0903:::benchmark_data_smithi171_122772_object122264:head# 2024-04-20T17:27:44.643 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a60f0d30:::benchmark_data_smithi171_122772_object85629:head# 2024-04-20T17:27:44.643 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6105ad8:::benchmark_data_smithi171_122772_object150762:head# 2024-04-20T17:27:44.644 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a610c839:::benchmark_data_smithi171_122772_object134805:head# 2024-04-20T17:27:44.644 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6115645:::benchmark_data_smithi171_122772_object83041:head# 2024-04-20T17:27:44.645 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6116853:::benchmark_data_smithi171_122772_object141432:head# 2024-04-20T17:27:44.645 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a611dd70:::benchmark_data_smithi171_122772_object108610:head# 2024-04-20T17:27:44.646 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6125077:::benchmark_data_smithi171_122772_object154810:head# 2024-04-20T17:27:44.646 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a612916f:::benchmark_data_smithi171_122772_object108006:head# 2024-04-20T17:27:44.647 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a61351ea:::benchmark_data_smithi171_122772_object166901:head# 2024-04-20T17:27:44.647 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a61387da:::benchmark_data_smithi171_122772_object133572:head# 2024-04-20T17:27:44.648 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a61393a9:::benchmark_data_smithi171_122772_object124068:head# 2024-04-20T17:27:44.648 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a614bbfb:::benchmark_data_smithi171_122772_object163566:head# 2024-04-20T17:27:44.649 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6153311:::benchmark_data_smithi171_122772_object140109:head# 2024-04-20T17:27:44.649 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a617c986:::benchmark_data_smithi171_122772_object85726:head# 2024-04-20T17:27:44.650 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a61928dc:::benchmark_data_smithi171_122772_object103829:head# 2024-04-20T17:27:44.650 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6192b27:::benchmark_data_smithi171_122772_object144423:head# 2024-04-20T17:27:44.650 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a619be09:::benchmark_data_smithi171_122772_object105607:head# 2024-04-20T17:27:44.651 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a61aabb7:::benchmark_data_smithi171_122772_object169399:head# 2024-04-20T17:27:44.651 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a61b1e14:::benchmark_data_smithi171_122772_object129566:head# 2024-04-20T17:27:44.652 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a61b368f:::benchmark_data_smithi171_122772_object163789:head# 2024-04-20T17:27:44.652 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a61c3616:::benchmark_data_smithi171_122772_object126265:head# 2024-04-20T17:27:44.653 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a61df7da:::benchmark_data_smithi171_122772_object116323:head# 2024-04-20T17:27:44.653 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a61f2387:::benchmark_data_smithi171_122772_object80216:head# 2024-04-20T17:27:44.654 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a61fbe0c:::benchmark_data_smithi171_122772_object152551:head# 2024-04-20T17:27:44.654 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6218f85:::benchmark_data_smithi171_122772_object126613:head# 2024-04-20T17:27:44.655 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a62254f0:::benchmark_data_smithi171_122772_object122536:head# 2024-04-20T17:27:44.655 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a622ae5d:::benchmark_data_smithi171_122772_object101378:head# 2024-04-20T17:27:44.656 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a622b04f:::benchmark_data_smithi171_122772_object119810:head# 2024-04-20T17:27:44.656 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a622eda4:::benchmark_data_smithi171_122772_object105402:head# 2024-04-20T17:27:44.657 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6251607:::benchmark_data_smithi171_122772_object114193:head# 2024-04-20T17:27:44.657 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a625608f:::benchmark_data_smithi171_122772_object120324:head# 2024-04-20T17:27:44.658 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a625679f:::benchmark_data_smithi171_122772_object118608:head# 2024-04-20T17:27:44.658 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a626540b:::benchmark_data_smithi171_122772_object169023:head# 2024-04-20T17:27:44.659 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a62695d9:::benchmark_data_smithi171_122772_object79822:head# 2024-04-20T17:27:44.659 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6275883:::benchmark_data_smithi171_122772_object154609:head# 2024-04-20T17:27:44.659 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a627769f:::benchmark_data_smithi171_122772_object142371:head# 2024-04-20T17:27:44.660 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6285e0b:::benchmark_data_smithi171_122772_object80104:head# 2024-04-20T17:27:44.660 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6289098:::benchmark_data_smithi171_122772_object144660:head# 2024-04-20T17:27:44.661 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a628ba73:::benchmark_data_smithi171_122772_object159191:head# 2024-04-20T17:27:44.662 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a62a123c:::benchmark_data_smithi171_122772_object165448:head# 2024-04-20T17:27:44.662 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a62ae012:::benchmark_data_smithi171_122772_object153161:head# 2024-04-20T17:27:44.663 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a62b848c:::benchmark_data_smithi171_122772_object147582:head# 2024-04-20T17:27:44.664 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a62c3fe1:::benchmark_data_smithi171_122772_object151339:head# 2024-04-20T17:27:44.664 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a62c958a:::benchmark_data_smithi171_122772_object86045:head# 2024-04-20T17:27:44.665 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a62d94e5:::benchmark_data_smithi171_122772_object165001:head# 2024-04-20T17:27:44.665 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a62ede17:::benchmark_data_smithi171_122772_object135296:head# 2024-04-20T17:27:44.666 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6307ee4:::benchmark_data_smithi171_122772_object144419:head# 2024-04-20T17:27:44.666 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a630ab84:::benchmark_data_smithi171_122772_object149236:head# 2024-04-20T17:27:44.667 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a630f86d:::benchmark_data_smithi171_122772_object147253:head# 2024-04-20T17:27:44.667 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a630f9f3:::benchmark_data_smithi171_122772_object162432:head# 2024-04-20T17:27:44.668 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a631df93:::benchmark_data_smithi171_122772_object119388:head# 2024-04-20T17:27:44.668 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6321ec2:::benchmark_data_smithi171_122772_object154863:head# 2024-04-20T17:27:44.668 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6326738:::benchmark_data_smithi171_122772_object167887:head# 2024-04-20T17:27:44.669 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a634a530:::benchmark_data_smithi171_122772_object117955:head# 2024-04-20T17:27:44.670 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a634a9df:::benchmark_data_smithi171_122772_object152181:head# 2024-04-20T17:27:44.670 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a63590f5:::benchmark_data_smithi171_122772_object169199:head# 2024-04-20T17:27:44.670 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6366529:::benchmark_data_smithi171_122772_object159818:head# 2024-04-20T17:27:44.671 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6368e31:::benchmark_data_smithi171_122772_object168578:head# 2024-04-20T17:27:44.672 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6373cda:::benchmark_data_smithi171_122772_object98931:head# 2024-04-20T17:27:44.672 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6376e12:::benchmark_data_smithi171_122772_object138845:head# 2024-04-20T17:27:44.673 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6385365:::benchmark_data_smithi171_122772_object112861:head# 2024-04-20T17:27:44.673 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6389c26:::benchmark_data_smithi171_122772_object81266:head# 2024-04-20T17:27:44.674 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a638d6e2:::benchmark_data_smithi171_122772_object110846:head# 2024-04-20T17:27:44.674 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a63996b0:::benchmark_data_smithi171_122772_object138066:head# 2024-04-20T17:27:44.675 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a63b50a1:::benchmark_data_smithi171_122772_object112457:head# 2024-04-20T17:27:44.675 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a63b923d:::benchmark_data_smithi171_122772_object168355:head# 2024-04-20T17:27:44.676 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a63bab28:::benchmark_data_smithi171_122772_object117420:head# 2024-04-20T17:27:44.676 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a63cd20b:::benchmark_data_smithi171_122772_object166827:head# 2024-04-20T17:27:44.677 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a63cda1b:::benchmark_data_smithi171_122772_object78693:head# 2024-04-20T17:27:44.677 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6403b85:::benchmark_data_smithi171_122772_object129214:head# 2024-04-20T17:27:44.678 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a640d356:::benchmark_data_smithi171_122772_object111116:head# 2024-04-20T17:27:44.678 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a640ebe4:::benchmark_data_smithi171_122772_object119621:head# 2024-04-20T17:27:44.679 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a642393a:::benchmark_data_smithi171_122772_object168383:head# 2024-04-20T17:27:44.679 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a643048b:::benchmark_data_smithi171_122772_object147093:head# 2024-04-20T17:27:44.679 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a643e2b3:::benchmark_data_smithi171_122772_object134343:head# 2024-04-20T17:27:44.680 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a646dc6c:::benchmark_data_smithi171_122772_object110019:head# 2024-04-20T17:27:44.680 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6475f6e:::benchmark_data_smithi171_122772_object86455:head# 2024-04-20T17:27:44.680 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a64765aa:::benchmark_data_smithi171_122772_object154122:head# 2024-04-20T17:27:44.681 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a64784a6:::benchmark_data_smithi171_122772_object109526:head# 2024-04-20T17:27:44.681 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a647a2ac:::benchmark_data_smithi171_122772_object147541:head# 2024-04-20T17:27:44.682 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a648438e:::benchmark_data_smithi171_122772_object164381:head# 2024-04-20T17:27:44.682 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6488585:::benchmark_data_smithi171_122772_object97895:head# 2024-04-20T17:27:44.683 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a649096f:::benchmark_data_smithi171_122772_object132900:head# 2024-04-20T17:27:44.683 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a649322c:::benchmark_data_smithi171_122772_object126468:head# 2024-04-20T17:27:44.684 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a649b1ff:::benchmark_data_smithi171_122772_object114577:head# 2024-04-20T17:27:44.684 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a649f9f4:::benchmark_data_smithi171_122772_object141005:head# 2024-04-20T17:27:44.684 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a64aec3f:::benchmark_data_smithi171_122772_object150724:head# 2024-04-20T17:27:44.685 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a64b7c46:::benchmark_data_smithi171_122772_object110833:head# 2024-04-20T17:27:44.685 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a64b9cea:::benchmark_data_smithi171_122772_object155282:head# 2024-04-20T17:27:44.686 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a64cfbe4:::benchmark_data_smithi171_122772_object165649:head# 2024-04-20T17:27:44.686 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a64ddc3a:::benchmark_data_smithi171_122772_object85120:head# 2024-04-20T17:27:44.687 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a64f9627:::benchmark_data_smithi171_122772_object89498:head# 2024-04-20T17:27:44.687 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6502de1:::benchmark_data_smithi171_122772_object110284:head# 2024-04-20T17:27:44.687 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6513677:::benchmark_data_smithi171_122772_object107975:head# 2024-04-20T17:27:44.688 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6521b15:::benchmark_data_smithi171_122772_object168134:head# 2024-04-20T17:27:44.688 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a65276b6:::benchmark_data_smithi171_122772_object155161:head# 2024-04-20T17:27:44.689 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a65311c8:::benchmark_data_smithi171_122772_object84076:head# 2024-04-20T17:27:44.689 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a655ff2f:::benchmark_data_smithi171_122772_object165076:head# 2024-04-20T17:27:44.690 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a657179a:::benchmark_data_smithi171_122772_object100480:head# 2024-04-20T17:27:44.690 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6576237:::benchmark_data_smithi171_122772_object154558:head# 2024-04-20T17:27:44.691 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a657d606:::benchmark_data_smithi171_122772_object154926:head# 2024-04-20T17:27:44.691 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a658f62e:::benchmark_data_smithi171_122772_object117522:head# 2024-04-20T17:27:44.691 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a659a52b:::benchmark_data_smithi171_122772_object150956:head# 2024-04-20T17:27:44.692 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a659fbb3:::benchmark_data_smithi171_122772_object141406:head# 2024-04-20T17:27:44.693 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a65b0a74:::benchmark_data_smithi171_122772_object126857:head# 2024-04-20T17:27:44.693 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a65bad8a:::benchmark_data_smithi171_122772_object100811:head# 2024-04-20T17:27:44.694 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a65c5690:::benchmark_data_smithi171_122772_object117391:head# 2024-04-20T17:27:44.694 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a65d1dd6:::benchmark_data_smithi171_122772_object154134:head# 2024-04-20T17:27:44.695 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a65d9b96:::benchmark_data_smithi171_122772_object77999:head# 2024-04-20T17:27:44.695 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a65dac27:::benchmark_data_smithi171_122772_object158166:head# 2024-04-20T17:27:44.695 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a65dd353:::benchmark_data_smithi171_122772_object101272:head# 2024-04-20T17:27:44.696 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a65e777a:::benchmark_data_smithi171_122772_object81120:head# 2024-04-20T17:27:44.696 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a65f3c3e:::benchmark_data_smithi171_122772_object155441:head# 2024-04-20T17:27:44.697 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a65fd432:::benchmark_data_smithi171_122772_object130780:head# 2024-04-20T17:27:44.697 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a65ff32a:::benchmark_data_smithi171_122772_object125247:head# 2024-04-20T17:27:44.698 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6614bd7:::benchmark_data_smithi171_122772_object124710:head# 2024-04-20T17:27:44.698 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a662df12:::benchmark_data_smithi171_122772_object114518:head# 2024-04-20T17:27:44.699 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a663ec62:::benchmark_data_smithi171_122772_object102213:head# 2024-04-20T17:27:44.699 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a665fd03:::benchmark_data_smithi171_122772_object147511:head# 2024-04-20T17:27:44.700 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6661345:::benchmark_data_smithi171_122772_object101792:head# 2024-04-20T17:27:44.700 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a667aebb:::benchmark_data_smithi171_122772_object114676:head# 2024-04-20T17:27:44.700 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a668abc7:::benchmark_data_smithi171_122772_object147271:head# 2024-04-20T17:27:44.701 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a66a1837:::benchmark_data_smithi171_122772_object158693:head# 2024-04-20T17:27:44.701 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a66a2f75:::benchmark_data_smithi171_122772_object89027:head# 2024-04-20T17:27:44.702 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a66ae30a:::benchmark_data_smithi171_122772_object86332:head# 2024-04-20T17:27:44.702 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a66b3449:::benchmark_data_smithi171_122772_object142944:head# 2024-04-20T17:27:44.703 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a66c4d57:::benchmark_data_smithi171_122772_object122329:head# 2024-04-20T17:27:44.703 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a66cd839:::benchmark_data_smithi171_122772_object149266:head# 2024-04-20T17:27:44.704 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a66dd8dc:::benchmark_data_smithi171_122772_object97024:head# 2024-04-20T17:27:44.704 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a66dff14:::benchmark_data_smithi171_122772_object78700:head# 2024-04-20T17:27:44.704 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a66e4f45:::benchmark_data_smithi171_122772_object98982:head# 2024-04-20T17:27:44.705 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a66edba3:::benchmark_data_smithi171_122772_object89097:head# 2024-04-20T17:27:44.705 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a66f166b:::benchmark_data_smithi171_122772_object91777:head# 2024-04-20T17:27:44.706 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a670bf33:::benchmark_data_smithi171_122772_object161968:head# 2024-04-20T17:27:44.706 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a670d8b1:::benchmark_data_smithi171_122772_object102196:head# 2024-04-20T17:27:44.707 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6718fbd:::benchmark_data_smithi171_122772_object159401:head# 2024-04-20T17:27:44.708 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a672c49f:::benchmark_data_smithi171_122772_object106713:head# 2024-04-20T17:27:44.708 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6747729:::benchmark_data_smithi171_122772_object84816:head# 2024-04-20T17:27:44.709 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a674e1dd:::benchmark_data_smithi171_122772_object121150:head# 2024-04-20T17:27:44.709 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6753f99:::benchmark_data_smithi171_122772_object168652:head# 2024-04-20T17:27:44.710 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6759ca5:::benchmark_data_smithi171_122772_object120158:head# 2024-04-20T17:27:44.710 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6762395:::benchmark_data_smithi171_122772_object114313:head# 2024-04-20T17:27:44.711 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a67782da:::benchmark_data_smithi171_122772_object134405:head# 2024-04-20T17:27:44.711 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6785bbe:::benchmark_data_smithi171_122772_object139424:head# 2024-04-20T17:27:44.712 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a678e5a6:::benchmark_data_smithi171_122772_object121266:head# 2024-04-20T17:27:44.712 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6790e03:::benchmark_data_smithi171_122772_object152362:head# 2024-04-20T17:27:44.713 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6796137:::benchmark_data_smithi171_122772_object166965:head# 2024-04-20T17:27:44.713 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a67b4ad2:::benchmark_data_smithi171_122772_object82254:head# 2024-04-20T17:27:44.714 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a67b9b5f:::benchmark_data_smithi171_122772_object88124:head# 2024-04-20T17:27:44.714 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a67bdf09:::benchmark_data_smithi171_122772_object112776:head# 2024-04-20T17:27:44.715 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a67c3640:::benchmark_data_smithi171_122772_object120129:head# 2024-04-20T17:27:44.715 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a67c3cab:::benchmark_data_smithi171_122772_object160152:head# 2024-04-20T17:27:44.716 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a67cc4e9:::benchmark_data_smithi171_122772_object93537:head# 2024-04-20T17:27:44.716 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a67d88fc:::benchmark_data_smithi171_122772_object115704:head# 2024-04-20T17:27:44.717 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a67f51d8:::benchmark_data_smithi171_122772_object134420:head# 2024-04-20T17:27:44.717 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a67f8263:::benchmark_data_smithi171_122772_object134440:head# 2024-04-20T17:27:44.718 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6801c03:::benchmark_data_smithi171_122772_object89180:head# 2024-04-20T17:27:44.718 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a68025da:::benchmark_data_smithi171_122772_object154820:head# 2024-04-20T17:27:44.723 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6811178:::benchmark_data_smithi171_122772_object130472:head# 2024-04-20T17:27:44.723 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6812c37:::benchmark_data_smithi171_122772_object164529:head# 2024-04-20T17:27:44.723 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a682ed6f:::benchmark_data_smithi171_122772_object135171:head# 2024-04-20T17:27:44.723 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a683300d:::benchmark_data_smithi171_122772_object102085:head# 2024-04-20T17:27:44.723 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a68335c2:::benchmark_data_smithi171_122772_object131133:head# 2024-04-20T17:27:44.723 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6843e1c:::benchmark_data_smithi171_122772_object82793:head# 2024-04-20T17:27:44.723 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6846e41:::benchmark_data_smithi171_122772_object96052:head# 2024-04-20T17:27:44.723 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a68479f0:::benchmark_data_smithi171_122772_object143892:head# 2024-04-20T17:27:44.723 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a685b88b:::benchmark_data_smithi171_122772_object97233:head# 2024-04-20T17:27:44.723 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a68630b5:::benchmark_data_smithi171_122772_object154196:head# 2024-04-20T17:27:44.724 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a687b1b9:::benchmark_data_smithi171_122772_object80814:head# 2024-04-20T17:27:44.724 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6884b54:::benchmark_data_smithi171_122772_object119652:head# 2024-04-20T17:27:44.724 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a68b7e71:::benchmark_data_smithi171_122772_object123700:head# 2024-04-20T17:27:44.724 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a68e657e:::benchmark_data_smithi171_122772_object126440:head# 2024-04-20T17:27:44.725 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a68e8890:::benchmark_data_smithi171_122772_object112168:head# 2024-04-20T17:27:44.725 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a68eda38:::benchmark_data_smithi171_122772_object129229:head# 2024-04-20T17:27:44.726 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a68f6931:::benchmark_data_smithi171_122772_object91431:head# 2024-04-20T17:27:44.726 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a68ffe5c:::benchmark_data_smithi171_122772_object138518:head# 2024-04-20T17:27:44.727 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6906197:::benchmark_data_smithi171_122772_object108905:head# 2024-04-20T17:27:44.728 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6907729:::benchmark_data_smithi171_122772_object91328:head# 2024-04-20T17:27:44.728 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a692861e:::benchmark_data_smithi171_122772_object156395:head# 2024-04-20T17:27:44.729 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6954c5d:::benchmark_data_smithi171_122772_object91788:head# 2024-04-20T17:27:44.729 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a698506e:::benchmark_data_smithi171_122772_object114146:head# 2024-04-20T17:27:44.730 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a698b745:::benchmark_data_smithi171_122772_object169491:head# 2024-04-20T17:27:44.730 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6993dea:::benchmark_data_smithi171_122772_object136435:head# 2024-04-20T17:27:44.731 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a699d813:::benchmark_data_smithi171_122772_object130158:head# 2024-04-20T17:27:44.731 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a69afe77:::benchmark_data_smithi171_122772_object148395:head# 2024-04-20T17:27:44.731 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a69b38e0:::benchmark_data_smithi171_122772_object153358:head# 2024-04-20T17:27:44.732 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a69ba5b4:::benchmark_data_smithi171_122772_object141719:head# 2024-04-20T17:27:44.733 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a69cae54:::benchmark_data_smithi171_122772_object144188:head# 2024-04-20T17:27:44.733 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a69d0cae:::benchmark_data_smithi171_122772_object78782:head# 2024-04-20T17:27:44.734 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a69dcdce:::benchmark_data_smithi171_122772_object123486:head# 2024-04-20T17:27:44.734 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a69eabe9:::benchmark_data_smithi171_122772_object95270:head# 2024-04-20T17:27:44.735 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a69fbeed:::benchmark_data_smithi171_122772_object140879:head# 2024-04-20T17:27:44.735 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6a0fbfe:::benchmark_data_smithi171_122772_object160255:head# 2024-04-20T17:27:44.736 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6a11f66:::benchmark_data_smithi171_122772_object145976:head# 2024-04-20T17:27:44.736 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6a21563:::benchmark_data_smithi171_122772_object154481:head# 2024-04-20T17:27:44.737 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6a2a9a7:::benchmark_data_smithi171_122772_object114104:head# 2024-04-20T17:27:44.737 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6a2fe0e:::benchmark_data_smithi171_122772_object118431:head# 2024-04-20T17:27:44.737 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6a3995d:::benchmark_data_smithi171_122772_object137284:head# 2024-04-20T17:27:44.738 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6a3d8cc:::benchmark_data_smithi171_122772_object84291:head# 2024-04-20T17:27:44.738 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6a3dc75:::benchmark_data_smithi171_122772_object165856:head# 2024-04-20T17:27:44.739 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6a3fe54:::benchmark_data_smithi171_122772_object148095:head# 2024-04-20T17:27:44.739 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6a44070:::benchmark_data_smithi171_122772_object82453:head# 2024-04-20T17:27:44.740 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6a5b10b:::benchmark_data_smithi171_122772_object83732:head# 2024-04-20T17:27:44.740 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6a5b3b9:::benchmark_data_smithi171_122772_object102635:head# 2024-04-20T17:27:44.741 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6a6f0c4:::benchmark_data_smithi171_122772_object155098:head# 2024-04-20T17:27:44.741 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6a8f6c8:::benchmark_data_smithi171_122772_object85572:head# 2024-04-20T17:27:44.742 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6a902a0:::benchmark_data_smithi171_122772_object107399:head# 2024-04-20T17:27:44.742 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6a934a3:::benchmark_data_smithi171_122772_object97498:head# 2024-04-20T17:27:44.743 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6a934d5:::benchmark_data_smithi171_122772_object128626:head# 2024-04-20T17:27:44.744 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6a93bad:::benchmark_data_smithi171_122772_object122247:head# 2024-04-20T17:27:44.744 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6aa22ed:::benchmark_data_smithi171_122772_object92347:head# 2024-04-20T17:27:44.745 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6aa8313:::benchmark_data_smithi171_122772_object110282:head# 2024-04-20T17:27:44.745 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6aa9c14:::benchmark_data_smithi171_122772_object87166:head# 2024-04-20T17:27:44.746 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6ab0d60:::benchmark_data_smithi171_122772_object117508:head# 2024-04-20T17:27:44.746 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6ab9138:::benchmark_data_smithi171_122772_object96984:head# 2024-04-20T17:27:44.746 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6abefc6:::benchmark_data_smithi171_122772_object90639:head# 2024-04-20T17:27:44.747 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6abf1a4:::benchmark_data_smithi171_122772_object164841:head# 2024-04-20T17:27:44.747 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6ac0bcd:::benchmark_data_smithi171_122772_object94139:head# 2024-04-20T17:27:44.748 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6aebf47:::benchmark_data_smithi171_122772_object96567:head# 2024-04-20T17:27:44.749 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6af245f:::benchmark_data_smithi171_122772_object152204:head# 2024-04-20T17:27:44.749 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6af9be9:::benchmark_data_smithi171_122772_object152020:head# 2024-04-20T17:27:44.749 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6b006e6:::benchmark_data_smithi171_122772_object124631:head# 2024-04-20T17:27:44.750 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6b0b82a:::benchmark_data_smithi171_122772_object162410:head# 2024-04-20T17:27:44.751 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6b0cf49:::benchmark_data_smithi171_122772_object135123:head# 2024-04-20T17:27:44.751 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6b0eebd:::benchmark_data_smithi171_122772_object108715:head# 2024-04-20T17:27:44.751 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6b24869:::benchmark_data_smithi171_122772_object105646:head# 2024-04-20T17:27:44.752 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6b3bb7c:::benchmark_data_smithi171_122772_object160174:head# 2024-04-20T17:27:44.752 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6b3c0a9:::benchmark_data_smithi171_122772_object143029:head# 2024-04-20T17:27:44.753 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6b407b0:::benchmark_data_smithi171_122772_object124088:head# 2024-04-20T17:27:44.753 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6b441cc:::benchmark_data_smithi171_122772_object126904:head# 2024-04-20T17:27:44.754 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6b539de:::benchmark_data_smithi171_122772_object155867:head# 2024-04-20T17:27:44.754 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6b67e2b:::benchmark_data_smithi171_122772_object110628:head# 2024-04-20T17:27:44.755 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6b6cee1:::benchmark_data_smithi171_122772_object141127:head# 2024-04-20T17:27:44.756 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6b86e2f:::benchmark_data_smithi171_122772_object97333:head# 2024-04-20T17:27:44.756 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6ba0f15:::benchmark_data_smithi171_122772_object123532:head# 2024-04-20T17:27:44.756 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6bb29e0:::benchmark_data_smithi171_122772_object145280:head# 2024-04-20T17:27:44.757 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6bb7e94:::benchmark_data_smithi171_122772_object87142:head# 2024-04-20T17:27:44.757 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6bc3b3e:::benchmark_data_smithi171_122772_object141938:head# 2024-04-20T17:27:44.758 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6bcf274:::benchmark_data_smithi171_122772_object91110:head# 2024-04-20T17:27:44.758 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6bd60f0:::benchmark_data_smithi171_122772_object142135:head# 2024-04-20T17:27:44.759 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6be2588:::benchmark_data_smithi171_122772_object105509:head# 2024-04-20T17:27:44.759 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6be62f9:::benchmark_data_smithi171_122772_object133788:head# 2024-04-20T17:27:44.760 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6bf20f1:::benchmark_data_smithi171_122772_object112036:head# 2024-04-20T17:27:44.760 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6c0a794:::benchmark_data_smithi171_122772_object104032:head# 2024-04-20T17:27:44.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:44 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 3774/184372 objects degraded (2.047%), 2 pgs degraded (PG_DEGRADED) 2024-04-20T17:27:44.761 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6c13f8f:::benchmark_data_smithi171_122772_object131889:head# 2024-04-20T17:27:44.761 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6c151f2:::benchmark_data_smithi171_122772_object157794:head# 2024-04-20T17:27:44.761 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6c356d2:::benchmark_data_smithi171_122772_object161079:head# 2024-04-20T17:27:44.762 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6c36159:::benchmark_data_smithi171_122772_object145915:head# 2024-04-20T17:27:44.762 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6c3751c:::benchmark_data_smithi171_122772_object94516:head# 2024-04-20T17:27:44.763 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6c3cb37:::benchmark_data_smithi171_122772_object144949:head# 2024-04-20T17:27:44.763 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6c415ad:::benchmark_data_smithi171_122772_object126705:head# 2024-04-20T17:27:44.764 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6c45573:::benchmark_data_smithi171_122772_object103409:head# 2024-04-20T17:27:44.764 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6c4b29d:::benchmark_data_smithi171_122772_object107849:head# 2024-04-20T17:27:44.765 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6c4ccdc:::benchmark_data_smithi171_122772_object122347:head# 2024-04-20T17:27:44.766 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6c50145:::benchmark_data_smithi171_122772_object78895:head# 2024-04-20T17:27:44.766 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6c5380d:::benchmark_data_smithi171_122772_object161130:head# 2024-04-20T17:27:44.767 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6c610c8:::benchmark_data_smithi171_122772_object104738:head# 2024-04-20T17:27:44.767 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6c6b0d5:::benchmark_data_smithi171_122772_object166140:head# 2024-04-20T17:27:44.768 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6c6cd9b:::benchmark_data_smithi171_122772_object93895:head# 2024-04-20T17:27:44.769 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6c70fd6:::benchmark_data_smithi171_122772_object138856:head# 2024-04-20T17:27:44.769 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6c74256:::benchmark_data_smithi171_122772_object115163:head# 2024-04-20T17:27:44.770 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6c7b3ff:::benchmark_data_smithi171_122772_object162550:head# 2024-04-20T17:27:44.770 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6c86d9d:::benchmark_data_smithi171_122772_object120593:head# 2024-04-20T17:27:44.771 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6ca0176:::benchmark_data_smithi171_122772_object154938:head# 2024-04-20T17:27:44.771 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6ca3587:::benchmark_data_smithi171_122772_object162289:head# 2024-04-20T17:27:44.772 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6cae3d7:::benchmark_data_smithi171_122772_object148377:head# 2024-04-20T17:27:44.772 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6cba535:::benchmark_data_smithi171_122772_object155183:head# 2024-04-20T17:27:44.773 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6cbf451:::benchmark_data_smithi171_122772_object157328:head# 2024-04-20T17:27:44.773 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6cc3d45:::benchmark_data_smithi171_122772_object115445:head# 2024-04-20T17:27:44.774 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6ccf194:::benchmark_data_smithi171_122772_object85430:head# 2024-04-20T17:27:44.774 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6ce9bc5:::benchmark_data_smithi171_122772_object147962:head# 2024-04-20T17:27:44.775 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6ceb662:::benchmark_data_smithi171_122772_object89733:head# 2024-04-20T17:27:44.775 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6cfd0be:::benchmark_data_smithi171_122772_object168466:head# 2024-04-20T17:27:44.776 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6cff26d:::benchmark_data_smithi171_122772_object102656:head# 2024-04-20T17:27:44.776 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6d0686f:::benchmark_data_smithi171_122772_object112005:head# 2024-04-20T17:27:44.777 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6d14435:::benchmark_data_smithi171_122772_object121335:head# 2024-04-20T17:27:44.777 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6d1932f:::benchmark_data_smithi171_122772_object165013:head# 2024-04-20T17:27:44.778 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6d21878:::benchmark_data_smithi171_122772_object134834:head# 2024-04-20T17:27:44.778 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6d32d7f:::benchmark_data_smithi171_122772_object152597:head# 2024-04-20T17:27:44.779 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6d3d962:::benchmark_data_smithi171_122772_object167764:head# 2024-04-20T17:27:44.779 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6d52a05:::benchmark_data_smithi171_122772_object80197:head# 2024-04-20T17:27:44.780 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6d5d800:::benchmark_data_smithi171_122772_object91283:head# 2024-04-20T17:27:44.780 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6d67a6b:::benchmark_data_smithi171_122772_object162549:head# 2024-04-20T17:27:44.781 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6d7392d:::benchmark_data_smithi171_122772_object88713:head# 2024-04-20T17:27:44.781 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6da18f8:::benchmark_data_smithi171_122772_object105697:head# 2024-04-20T17:27:44.782 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6daf88a:::benchmark_data_smithi171_122772_object96222:head# 2024-04-20T17:27:44.782 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6db5471:::benchmark_data_smithi171_122772_object130565:head# 2024-04-20T17:27:44.783 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6dbdad2:::benchmark_data_smithi171_122772_object92649:head# 2024-04-20T17:27:44.783 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6dc20aa:::benchmark_data_smithi171_122772_object130443:head# 2024-04-20T17:27:44.784 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6dc27c7:::benchmark_data_smithi171_122772_object118171:head# 2024-04-20T17:27:44.784 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6dcfe53:::benchmark_data_smithi171_122772_object141227:head# 2024-04-20T17:27:44.785 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6defdab:::benchmark_data_smithi171_122772_object111698:head# 2024-04-20T17:27:44.785 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6df6d2a:::benchmark_data_smithi171_122772_object96320:head# 2024-04-20T17:27:44.786 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6df9ccc:::benchmark_data_smithi171_122772_object81470:head# 2024-04-20T17:27:44.786 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6df9fe8:::benchmark_data_smithi171_122772_object126856:head# 2024-04-20T17:27:44.787 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6dfb968:::benchmark_data_smithi171_122772_object105900:head# 2024-04-20T17:27:44.787 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6dfd7e5:::benchmark_data_smithi171_122772_object113039:head# 2024-04-20T17:27:44.788 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6e06f7f:::benchmark_data_smithi171_122772_object124374:head# 2024-04-20T17:27:44.788 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6e0a72a:::benchmark_data_smithi171_122772_object146307:head# 2024-04-20T17:27:44.789 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6e0d575:::benchmark_data_smithi171_122772_object148323:head# 2024-04-20T17:27:44.789 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6e20997:::benchmark_data_smithi171_122772_object104995:head# 2024-04-20T17:27:44.790 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6e23425:::benchmark_data_smithi171_122772_object121466:head# 2024-04-20T17:27:44.790 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6e282b3:::benchmark_data_smithi171_122772_object132679:head# 2024-04-20T17:27:44.791 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6e40fbd:::benchmark_data_smithi171_122772_object93766:head# 2024-04-20T17:27:44.791 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6e5b938:::benchmark_data_smithi171_122772_object99003:head# 2024-04-20T17:27:44.792 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6e7d15b:::benchmark_data_smithi171_122772_object167983:head# 2024-04-20T17:27:44.792 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6e85078:::benchmark_data_smithi171_122772_object126318:head# 2024-04-20T17:27:44.793 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6e912ab:::benchmark_data_smithi171_122772_object87900:head# 2024-04-20T17:27:44.793 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6ea11c4:::benchmark_data_smithi171_122772_object120931:head# 2024-04-20T17:27:44.794 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6eae928:::benchmark_data_smithi171_122772_object83586:head# 2024-04-20T17:27:44.794 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6eb162d:::benchmark_data_smithi171_122772_object133559:head# 2024-04-20T17:27:44.795 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6ec23b9:::benchmark_data_smithi171_122772_object102716:head# 2024-04-20T17:27:44.795 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6ed4a37:::benchmark_data_smithi171_122772_object110070:head# 2024-04-20T17:27:44.796 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6ed9fbe:::benchmark_data_smithi171_122772_object106218:head# 2024-04-20T17:27:44.796 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6edb182:::benchmark_data_smithi171_122772_object102240:head# 2024-04-20T17:27:44.797 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6eddbf7:::benchmark_data_smithi171_122772_object116247:head# 2024-04-20T17:27:44.797 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6ee1fbd:::benchmark_data_smithi171_122772_object118830:head# 2024-04-20T17:27:44.798 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6ef92f1:::benchmark_data_smithi171_122772_object124375:head# 2024-04-20T17:27:44.798 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6f03d61:::benchmark_data_smithi171_122772_object149018:head# 2024-04-20T17:27:44.799 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6f04a68:::benchmark_data_smithi171_122772_object155763:head# 2024-04-20T17:27:44.799 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6f0ce0b:::benchmark_data_smithi171_122772_object166708:head# 2024-04-20T17:27:44.799 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6f1be5d:::benchmark_data_smithi171_122772_object139417:head# 2024-04-20T17:27:44.800 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6f38b0e:::benchmark_data_smithi171_122772_object79538:head# 2024-04-20T17:27:44.801 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6f43530:::benchmark_data_smithi171_122772_object169267:head# 2024-04-20T17:27:44.801 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6f4b031:::benchmark_data_smithi171_122772_object119525:head# 2024-04-20T17:27:44.802 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6f52240:::benchmark_data_smithi171_122772_object125689:head# 2024-04-20T17:27:44.802 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6f53d2b:::benchmark_data_smithi171_122772_object125752:head# 2024-04-20T17:27:44.803 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6f5530b:::benchmark_data_smithi171_122772_object107241:head# 2024-04-20T17:27:44.803 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6f5d2b1:::benchmark_data_smithi171_122772_object81481:head# 2024-04-20T17:27:44.803 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6f5f8ad:::benchmark_data_smithi171_122772_object140423:head# 2024-04-20T17:27:44.804 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6f6d344:::benchmark_data_smithi171_122772_object88214:head# 2024-04-20T17:27:44.804 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6f6e4ed:::benchmark_data_smithi171_122772_object167352:head# 2024-04-20T17:27:44.805 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6f76b09:::benchmark_data_smithi171_122772_object168501:head# 2024-04-20T17:27:44.805 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6f8c00a:::benchmark_data_smithi171_122772_object90491:head# 2024-04-20T17:27:44.806 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6fa2f39:::benchmark_data_smithi171_122772_object161941:head# 2024-04-20T17:27:44.806 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6fab74f:::benchmark_data_smithi171_122772_object163454:head# 2024-04-20T17:27:44.807 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6fad2b3:::benchmark_data_smithi171_122772_object89168:head# 2024-04-20T17:27:44.807 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6fbf3ac:::benchmark_data_smithi171_122772_object111125:head# 2024-04-20T17:27:44.808 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6fc16e8:::benchmark_data_smithi171_122772_object92959:head# 2024-04-20T17:27:44.808 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6fc5e88:::benchmark_data_smithi171_122772_object167452:head# 2024-04-20T17:27:44.809 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6fd16d1:::benchmark_data_smithi171_122772_object112063:head# 2024-04-20T17:27:44.809 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6fd2804:::benchmark_data_smithi171_122772_object98071:head# 2024-04-20T17:27:44.810 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6fd2a9a:::benchmark_data_smithi171_122772_object111733:head# 2024-04-20T17:27:44.811 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6fd3c30:::benchmark_data_smithi171_122772_object117044:head# 2024-04-20T17:27:44.811 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6fe4cd8:::benchmark_data_smithi171_122772_object131989:head# 2024-04-20T17:27:44.812 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6fed86c:::benchmark_data_smithi171_122772_object89471:head# 2024-04-20T17:27:44.812 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a6ff2f2f:::benchmark_data_smithi171_122772_object100152:head# 2024-04-20T17:27:44.813 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a700999d:::benchmark_data_smithi171_122772_object111280:head# 2024-04-20T17:27:44.813 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7013c91:::benchmark_data_smithi171_122772_object88911:head# 2024-04-20T17:27:44.814 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7017bcb:::benchmark_data_smithi171_122772_object83727:head# 2024-04-20T17:27:44.814 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a702540c:::benchmark_data_smithi171_122772_object167108:head# 2024-04-20T17:27:44.815 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a703d212:::benchmark_data_smithi171_122772_object169030:head# 2024-04-20T17:27:44.815 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a704912b:::benchmark_data_smithi171_122772_object135766:head# 2024-04-20T17:27:44.816 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a70515ee:::benchmark_data_smithi171_122772_object82373:head# 2024-04-20T17:27:44.816 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7051675:::benchmark_data_smithi171_122772_object118352:head# 2024-04-20T17:27:44.817 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7055481:::benchmark_data_smithi171_122772_object109816:head# 2024-04-20T17:27:44.817 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7055647:::benchmark_data_smithi171_122772_object98664:head# 2024-04-20T17:27:44.818 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a70761c5:::benchmark_data_smithi171_122772_object152282:head# 2024-04-20T17:27:44.819 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7080b2a:::benchmark_data_smithi171_122772_object138443:head# 2024-04-20T17:27:44.819 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a70a2f84:::benchmark_data_smithi171_122772_object149898:head# 2024-04-20T17:27:44.820 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a70b0f3c:::benchmark_data_smithi171_122772_object125821:head# 2024-04-20T17:27:44.821 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a70b6069:::benchmark_data_smithi171_122772_object159281:head# 2024-04-20T17:27:44.821 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a70cc585:::benchmark_data_smithi171_122772_object125515:head# 2024-04-20T17:27:44.822 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a70d240b:::benchmark_data_smithi171_122772_object92240:head# 2024-04-20T17:27:44.822 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a70d3e03:::benchmark_data_smithi171_122772_object130457:head# 2024-04-20T17:27:44.823 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a70d7ad0:::benchmark_data_smithi171_122772_object131383:head# 2024-04-20T17:27:44.823 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a70dc902:::benchmark_data_smithi171_122772_object84410:head# 2024-04-20T17:27:44.824 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a70dd961:::benchmark_data_smithi171_122772_object120723:head# 2024-04-20T17:27:44.824 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a70e3e36:::benchmark_data_smithi171_122772_object109462:head# 2024-04-20T17:27:44.825 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a70e5c88:::benchmark_data_smithi171_122772_object133461:head# 2024-04-20T17:27:44.825 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a70e78da:::benchmark_data_smithi171_122772_object118334:head# 2024-04-20T17:27:44.825 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a70fab50:::benchmark_data_smithi171_122772_object80844:head# 2024-04-20T17:27:44.826 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a70fc853:::benchmark_data_smithi171_122772_object96338:head# 2024-04-20T17:27:44.826 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a71096ba:::benchmark_data_smithi171_122772_object117895:head# 2024-04-20T17:27:44.827 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a711127c:::benchmark_data_smithi171_122772_object116166:head# 2024-04-20T17:27:44.828 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a711ae11:::benchmark_data_smithi171_122772_object105202:head# 2024-04-20T17:27:44.828 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a711d402:::benchmark_data_smithi171_122772_object84377:head# 2024-04-20T17:27:44.828 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7121bf5:::benchmark_data_smithi171_122772_object112089:head# 2024-04-20T17:27:44.829 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7126c92:::benchmark_data_smithi171_122772_object146589:head# 2024-04-20T17:27:44.830 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a712b787:::benchmark_data_smithi171_122772_object159735:head# 2024-04-20T17:27:44.830 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7133369:::benchmark_data_smithi171_122772_object161931:head# 2024-04-20T17:27:44.831 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a714a6cb:::benchmark_data_smithi171_122772_object155315:head# 2024-04-20T17:27:44.831 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a715ba6b:::benchmark_data_smithi171_122772_object79568:head# 2024-04-20T17:27:44.832 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a715f8b2:::benchmark_data_smithi171_122772_object105156:head# 2024-04-20T17:27:44.832 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a717984c:::benchmark_data_smithi171_122772_object129333:head# 2024-04-20T17:27:44.833 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a71a811c:::benchmark_data_smithi171_122772_object102289:head# 2024-04-20T17:27:44.833 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a71af9c2:::benchmark_data_smithi171_122772_object141908:head# 2024-04-20T17:27:44.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:44 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 3774/184372 objects degraded (2.047%), 2 pgs degraded (PG_DEGRADED) 2024-04-20T17:27:44.834 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a71c789d:::benchmark_data_smithi171_122772_object91497:head# 2024-04-20T17:27:44.834 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a71d7d10:::benchmark_data_smithi171_122772_object136851:head# 2024-04-20T17:27:44.834 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a71dbc10:::benchmark_data_smithi171_122772_object155372:head# 2024-04-20T17:27:44.835 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a71eb790:::benchmark_data_smithi171_122772_object137820:head# 2024-04-20T17:27:44.835 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7202654:::benchmark_data_smithi171_122772_object123687:head# 2024-04-20T17:27:44.836 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7209fa1:::benchmark_data_smithi171_122772_object112745:head# 2024-04-20T17:27:44.837 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a720cdc4:::benchmark_data_smithi171_122772_object112151:head# 2024-04-20T17:27:44.837 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a720f870:::benchmark_data_smithi171_122772_object157291:head# 2024-04-20T17:27:44.838 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a722d5cf:::benchmark_data_smithi171_122772_object134632:head# 2024-04-20T17:27:44.838 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a722e208:::benchmark_data_smithi171_122772_object92703:head# 2024-04-20T17:27:44.839 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a723bc4c:::benchmark_data_smithi171_122772_object88846:head# 2024-04-20T17:27:44.839 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a723c0fc:::benchmark_data_smithi171_122772_object146782:head# 2024-04-20T17:27:44.839 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a72422f9:::benchmark_data_smithi171_122772_object163694:head# 2024-04-20T17:27:44.840 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a725413b:::benchmark_data_smithi171_122772_object147098:head# 2024-04-20T17:27:44.840 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7255665:::benchmark_data_smithi171_122772_object137411:head# 2024-04-20T17:27:44.841 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7260f57:::benchmark_data_smithi171_122772_object150000:head# 2024-04-20T17:27:44.842 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7261544:::benchmark_data_smithi171_122772_object132854:head# 2024-04-20T17:27:44.842 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7263c00:::benchmark_data_smithi171_122772_object145263:head# 2024-04-20T17:27:44.843 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a726991a:::benchmark_data_smithi171_122772_object117808:head# 2024-04-20T17:27:44.843 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a726de43:::benchmark_data_smithi171_122772_object168471:head# 2024-04-20T17:27:44.843 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a727113a:::benchmark_data_smithi171_122772_object119084:head# 2024-04-20T17:27:44.844 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a72713d5:::benchmark_data_smithi171_122772_object161773:head# 2024-04-20T17:27:44.845 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a72741c9:::benchmark_data_smithi171_122772_object137751:head# 2024-04-20T17:27:44.845 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a727f7a3:::benchmark_data_smithi171_122772_object90142:head# 2024-04-20T17:27:44.846 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a728bdda:::benchmark_data_smithi171_122772_object150140:head# 2024-04-20T17:27:44.846 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7299672:::benchmark_data_smithi171_122772_object143806:head# 2024-04-20T17:27:44.847 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a729d72b:::benchmark_data_smithi171_122772_object147656:head# 2024-04-20T17:27:44.847 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a72a01df:::benchmark_data_smithi171_122772_object113541:head# 2024-04-20T17:27:44.848 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a72b2e6f:::benchmark_data_smithi171_122772_object132442:head# 2024-04-20T17:27:44.848 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a72b7428:::benchmark_data_smithi171_122772_object93992:head# 2024-04-20T17:27:44.848 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a72b803d:::benchmark_data_smithi171_122772_object138818:head# 2024-04-20T17:27:44.849 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a72d6ec5:::benchmark_data_smithi171_122772_object120481:head# 2024-04-20T17:27:44.849 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a72e1b23:::benchmark_data_smithi171_122772_object102326:head# 2024-04-20T17:27:44.850 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a72e51fb:::benchmark_data_smithi171_122772_object106504:head# 2024-04-20T17:27:44.850 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7308c73:::benchmark_data_smithi171_122772_object101045:head# 2024-04-20T17:27:44.851 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a730db76:::benchmark_data_smithi171_122772_object84803:head# 2024-04-20T17:27:44.851 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a731075e:::benchmark_data_smithi171_122772_object138135:head# 2024-04-20T17:27:44.852 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a73128b3:::benchmark_data_smithi171_122772_object85139:head# 2024-04-20T17:27:44.852 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7321cab:::benchmark_data_smithi171_122772_object84747:head# 2024-04-20T17:27:44.853 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a732ea28:::benchmark_data_smithi171_122772_object118869:head# 2024-04-20T17:27:44.853 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a734127e:::benchmark_data_smithi171_122772_object116851:head# 2024-04-20T17:27:44.854 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a735bb1f:::benchmark_data_smithi171_122772_object144090:head# 2024-04-20T17:27:44.854 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7363a33:::benchmark_data_smithi171_122772_object155618:head# 2024-04-20T17:27:44.855 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a736e188:::benchmark_data_smithi171_122772_object81738:head# 2024-04-20T17:27:44.855 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7372913:::benchmark_data_smithi171_122772_object114133:head# 2024-04-20T17:27:44.855 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a738601a:::benchmark_data_smithi171_122772_object132005:head# 2024-04-20T17:27:44.856 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a738779d:::benchmark_data_smithi171_122772_object144668:head# 2024-04-20T17:27:44.856 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7395bef:::benchmark_data_smithi171_122772_object129093:head# 2024-04-20T17:27:44.857 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a73a45ed:::benchmark_data_smithi171_122772_object91165:head# 2024-04-20T17:27:44.857 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a73aba83:::benchmark_data_smithi171_122772_object106472:head# 2024-04-20T17:27:44.858 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a73b6b15:::benchmark_data_smithi171_122772_object84254:head# 2024-04-20T17:27:44.858 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a73bbf34:::benchmark_data_smithi171_122772_object133912:head# 2024-04-20T17:27:44.859 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a73ceede:::benchmark_data_smithi171_122772_object98770:head# 2024-04-20T17:27:44.859 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a73cf79d:::benchmark_data_smithi171_122772_object106558:head# 2024-04-20T17:27:44.860 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7413fbb:::benchmark_data_smithi171_122772_object141751:head# 2024-04-20T17:27:44.860 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a742c794:::benchmark_data_smithi171_122772_object84546:head# 2024-04-20T17:27:44.861 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a745a855:::benchmark_data_smithi171_122772_object130509:head# 2024-04-20T17:27:44.861 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a747e3d2:::benchmark_data_smithi171_122772_object148620:head# 2024-04-20T17:27:44.862 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7496790:::benchmark_data_smithi171_122772_object131497:head# 2024-04-20T17:27:44.862 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a749cd16:::benchmark_data_smithi171_122772_object121170:head# 2024-04-20T17:27:44.863 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a74ba378:::benchmark_data_smithi171_122772_object110596:head# 2024-04-20T17:27:44.863 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a74c9c4b:::benchmark_data_smithi171_122772_object159839:head# 2024-04-20T17:27:44.864 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a74f23bc:::benchmark_data_smithi171_122772_object145686:head# 2024-04-20T17:27:44.864 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a74f2879:::benchmark_data_smithi171_122772_object169205:head# 2024-04-20T17:27:44.865 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a74ff05d:::benchmark_data_smithi171_122772_object137886:head# 2024-04-20T17:27:44.865 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7502eac:::benchmark_data_smithi171_122772_object142498:head# 2024-04-20T17:27:44.866 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7508e54:::benchmark_data_smithi171_122772_object103351:head# 2024-04-20T17:27:44.866 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7510b77:::benchmark_data_smithi171_122772_object169404:head# 2024-04-20T17:27:44.867 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a751d9f5:::benchmark_data_smithi171_122772_object152570:head# 2024-04-20T17:27:44.867 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7527862:::benchmark_data_smithi171_122772_object90107:head# 2024-04-20T17:27:44.867 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7531f40:::benchmark_data_smithi171_122772_object143465:head# 2024-04-20T17:27:44.868 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7535374:::benchmark_data_smithi171_122772_object158358:head# 2024-04-20T17:27:44.868 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a754129e:::benchmark_data_smithi171_122772_object107454:head# 2024-04-20T17:27:44.869 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7554c11:::benchmark_data_smithi171_122772_object125270:head# 2024-04-20T17:27:44.869 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a756afd5:::benchmark_data_smithi171_122772_object149640:head# 2024-04-20T17:27:44.870 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a757b64d:::benchmark_data_smithi171_122772_object115450:head# 2024-04-20T17:27:44.870 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7583f65:::benchmark_data_smithi171_122772_object129928:head# 2024-04-20T17:27:44.871 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a75890e4:::benchmark_data_smithi171_122772_object78020:head# 2024-04-20T17:27:44.871 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a758c317:::benchmark_data_smithi171_122772_object141936:head# 2024-04-20T17:27:44.872 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a759e887:::benchmark_data_smithi171_122772_object153110:head# 2024-04-20T17:27:44.872 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a75bee81:::benchmark_data_smithi171_122772_object150621:head# 2024-04-20T17:27:44.873 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a75cad7c:::benchmark_data_smithi171_122772_object149922:head# 2024-04-20T17:27:44.873 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a75d466e:::benchmark_data_smithi171_122772_object117131:head# 2024-04-20T17:27:44.874 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a75df1d2:::benchmark_data_smithi171_122772_object126804:head# 2024-04-20T17:27:44.874 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a75e266b:::benchmark_data_smithi171_122772_object153387:head# 2024-04-20T17:27:44.874 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a75e47d1:::benchmark_data_smithi171_122772_object80474:head# 2024-04-20T17:27:44.875 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a75e5a78:::benchmark_data_smithi171_122772_object142518:head# 2024-04-20T17:27:44.875 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a75e91ee:::benchmark_data_smithi171_122772_object102140:head# 2024-04-20T17:27:44.876 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a75ed982:::benchmark_data_smithi171_122772_object80540:head# 2024-04-20T17:27:44.876 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7632a5c:::benchmark_data_smithi171_122772_object147226:head# 2024-04-20T17:27:44.877 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a764155b:::benchmark_data_smithi171_122772_object115087:head# 2024-04-20T17:27:44.877 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7645eae:::benchmark_data_smithi171_122772_object90467:head# 2024-04-20T17:27:44.878 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a764d057:::benchmark_data_smithi171_122772_object140454:head# 2024-04-20T17:27:44.878 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a76590c7:::benchmark_data_smithi171_122772_object148708:head# 2024-04-20T17:27:44.879 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a767857d:::benchmark_data_smithi171_122772_object111447:head# 2024-04-20T17:27:44.879 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a767b8ab:::benchmark_data_smithi171_122772_object167275:head# 2024-04-20T17:27:44.880 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7686e93:::benchmark_data_smithi171_122772_object83639:head# 2024-04-20T17:27:44.880 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a769cb8a:::benchmark_data_smithi171_122772_object110788:head# 2024-04-20T17:27:44.881 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a76a3114:::benchmark_data_smithi171_122772_object149702:head# 2024-04-20T17:27:44.881 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a76a5798:::benchmark_data_smithi171_122772_object163650:head# 2024-04-20T17:27:44.882 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a76bb844:::benchmark_data_smithi171_122772_object143022:head# 2024-04-20T17:27:44.882 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a76c039a:::benchmark_data_smithi171_122772_object87979:head# 2024-04-20T17:27:44.883 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a76cee39:::benchmark_data_smithi171_122772_object132920:head# 2024-04-20T17:27:44.883 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a76d4521:::benchmark_data_smithi171_122772_object91633:head# 2024-04-20T17:27:44.884 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a76d8d09:::benchmark_data_smithi171_122772_object101511:head# 2024-04-20T17:27:44.884 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a76deff1:::benchmark_data_smithi171_122772_object145439:head# 2024-04-20T17:27:44.885 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a76f1853:::benchmark_data_smithi171_122772_object120977:head# 2024-04-20T17:27:44.885 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a76f208b:::benchmark_data_smithi171_122772_object155123:head# 2024-04-20T17:27:44.886 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a76f33ae:::benchmark_data_smithi171_122772_object98799:head# 2024-04-20T17:27:44.886 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a770daa1:::benchmark_data_smithi171_122772_object105959:head# 2024-04-20T17:27:44.887 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7718d9f:::benchmark_data_smithi171_122772_object90068:head# 2024-04-20T17:27:44.887 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a77246c8:::benchmark_data_smithi171_122772_object102322:head# 2024-04-20T17:27:44.888 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7727eb4:::benchmark_data_smithi171_122772_object145895:head# 2024-04-20T17:27:44.888 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7735424:::benchmark_data_smithi171_122772_object151987:head# 2024-04-20T17:27:44.889 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a773f190:::benchmark_data_smithi171_122772_object164973:head# 2024-04-20T17:27:44.889 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a77560f9:::benchmark_data_smithi171_122772_object93653:head# 2024-04-20T17:27:44.890 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a775da7c:::benchmark_data_smithi171_122772_object107008:head# 2024-04-20T17:27:44.890 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7772f78:::benchmark_data_smithi171_122772_object102949:head# 2024-04-20T17:27:44.890 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7775aa6:::benchmark_data_smithi171_122772_object169282:head# 2024-04-20T17:27:44.891 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a777cc64:::benchmark_data_smithi171_122772_object113953:head# 2024-04-20T17:27:44.891 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a77888de:::benchmark_data_smithi171_122772_object166481:head# 2024-04-20T17:27:44.892 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a778ec95:::benchmark_data_smithi171_122772_object111548:head# 2024-04-20T17:27:44.893 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7796c14:::benchmark_data_smithi171_122772_object134596:head# 2024-04-20T17:27:44.893 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a77a21c2:::benchmark_data_smithi171_122772_object140570:head# 2024-04-20T17:27:44.894 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a77b3e05:::benchmark_data_smithi171_122772_object94213:head# 2024-04-20T17:27:44.894 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a77b6829:::benchmark_data_smithi171_122772_object88859:head# 2024-04-20T17:27:44.895 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a77c9c3b:::benchmark_data_smithi171_122772_object152722:head# 2024-04-20T17:27:44.895 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a77cfaa7:::benchmark_data_smithi171_122772_object98906:head# 2024-04-20T17:27:44.896 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a77e7ef2:::benchmark_data_smithi171_122772_object155796:head# 2024-04-20T17:27:44.896 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a77ed79b:::benchmark_data_smithi171_122772_object146765:head# 2024-04-20T17:27:44.897 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a77fcf97:::benchmark_data_smithi171_122772_object125558:head# 2024-04-20T17:27:44.897 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7808924:::benchmark_data_smithi171_122772_object116972:head# 2024-04-20T17:27:44.897 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a780b4b9:::benchmark_data_smithi171_122772_object86132:head# 2024-04-20T17:27:44.898 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7812be8:::benchmark_data_smithi171_122772_object156453:head# 2024-04-20T17:27:44.899 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a782082a:::benchmark_data_smithi171_122772_object87529:head# 2024-04-20T17:27:44.899 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7836eba:::benchmark_data_smithi171_122772_object160399:head# 2024-04-20T17:27:44.900 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a783819a:::benchmark_data_smithi171_122772_object167287:head# 2024-04-20T17:27:44.900 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a783f8d4:::benchmark_data_smithi171_122772_object155498:head# 2024-04-20T17:27:44.901 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a785eeff:::benchmark_data_smithi171_122772_object82592:head# 2024-04-20T17:27:44.901 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7867a08:::benchmark_data_smithi171_122772_object141489:head# 2024-04-20T17:27:44.902 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7868f63:::benchmark_data_smithi171_122772_object167125:head# 2024-04-20T17:27:44.902 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a786d63f:::benchmark_data_smithi171_122772_object96287:head# 2024-04-20T17:27:44.903 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a78842ae:::benchmark_data_smithi171_122772_object81124:head# 2024-04-20T17:27:44.903 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a788ecf8:::benchmark_data_smithi171_122772_object137954:head# 2024-04-20T17:27:44.903 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a789e003:::benchmark_data_smithi171_122772_object117489:head# 2024-04-20T17:27:44.904 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a78b8e5c:::benchmark_data_smithi171_122772_object106461:head# 2024-04-20T17:27:44.904 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a78d6be8:::benchmark_data_smithi171_122772_object150352:head# 2024-04-20T17:27:44.905 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a78dc97c:::benchmark_data_smithi171_122772_object78047:head# 2024-04-20T17:27:44.906 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a78eaef6:::benchmark_data_smithi171_122772_object123438:head# 2024-04-20T17:27:44.906 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a78f62ca:::benchmark_data_smithi171_122772_object93065:head# 2024-04-20T17:27:44.907 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a78fda7a:::benchmark_data_smithi171_122772_object153517:head# 2024-04-20T17:27:44.907 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a78fe255:::benchmark_data_smithi171_122772_object127137:head# 2024-04-20T17:27:44.908 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7916af1:::benchmark_data_smithi171_122772_object108347:head# 2024-04-20T17:27:44.908 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a792950d:::benchmark_data_smithi171_122772_object140130:head# 2024-04-20T17:27:44.909 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a792a978:::benchmark_data_smithi171_122772_object88388:head# 2024-04-20T17:27:44.910 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a792c241:::benchmark_data_smithi171_122772_object163010:head# 2024-04-20T17:27:44.910 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a793192b:::benchmark_data_smithi171_122772_object79535:head# 2024-04-20T17:27:44.911 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7936255:::benchmark_data_smithi171_122772_object167796:head# 2024-04-20T17:27:44.911 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a793ca22:::benchmark_data_smithi171_122772_object112878:head# 2024-04-20T17:27:44.911 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a793d32a:::benchmark_data_smithi171_122772_object124959:head# 2024-04-20T17:27:44.912 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a793ff1c:::benchmark_data_smithi171_122772_object83325:head# 2024-04-20T17:27:44.912 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a794c663:::benchmark_data_smithi171_122772_object155296:head# 2024-04-20T17:27:44.913 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a794d404:::benchmark_data_smithi171_122772_object94374:head# 2024-04-20T17:27:44.914 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a794d5aa:::benchmark_data_smithi171_122772_object80010:head# 2024-04-20T17:27:44.914 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7952dc8:::benchmark_data_smithi171_122772_object128006:head# 2024-04-20T17:27:44.915 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a79680a1:::benchmark_data_smithi171_122772_object125333:head# 2024-04-20T17:27:44.915 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7972b0f:::benchmark_data_smithi171_122772_object143946:head# 2024-04-20T17:27:44.916 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7974a76:::benchmark_data_smithi171_122772_object112502:head# 2024-04-20T17:27:44.916 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a797756b:::benchmark_data_smithi171_122772_object139769:head# 2024-04-20T17:27:44.917 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a797e23b:::benchmark_data_smithi171_122772_object102678:head# 2024-04-20T17:27:44.917 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a797f196:::benchmark_data_smithi171_122772_object114795:head# 2024-04-20T17:27:44.918 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a79837f4:::benchmark_data_smithi171_122772_object119672:head# 2024-04-20T17:27:44.918 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a798dcce:::benchmark_data_smithi171_122772_object110591:head# 2024-04-20T17:27:44.919 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a798fe3b:::benchmark_data_smithi171_122772_object108688:head# 2024-04-20T17:27:44.919 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a799be26:::benchmark_data_smithi171_122772_object134223:head# 2024-04-20T17:27:44.920 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a79a9978:::benchmark_data_smithi171_122772_object152596:head# 2024-04-20T17:27:44.921 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a79c173d:::benchmark_data_smithi171_122772_object133189:head# 2024-04-20T17:27:44.921 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a79cba46:::benchmark_data_smithi171_122772_object155344:head# 2024-04-20T17:27:44.921 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a79d9228:::benchmark_data_smithi171_122772_object120299:head# 2024-04-20T17:27:44.922 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a79ed524:::benchmark_data_smithi171_122772_object122257:head# 2024-04-20T17:27:44.927 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7a042cf:::benchmark_data_smithi171_122772_object99340:head# 2024-04-20T17:27:44.928 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7a05383:::benchmark_data_smithi171_122772_object132339:head# 2024-04-20T17:27:44.928 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7a06e75:::benchmark_data_smithi171_122772_object112795:head# 2024-04-20T17:27:44.928 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7a1165c:::benchmark_data_smithi171_122772_object101164:head# 2024-04-20T17:27:44.928 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7a17522:::benchmark_data_smithi171_122772_object83176:head# 2024-04-20T17:27:44.928 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7a182fe:::benchmark_data_smithi171_122772_object101761:head# 2024-04-20T17:27:44.928 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7a339e4:::benchmark_data_smithi171_122772_object83273:head# 2024-04-20T17:27:44.928 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7a3dec0:::benchmark_data_smithi171_122772_object136099:head# 2024-04-20T17:27:44.928 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7a44180:::benchmark_data_smithi171_122772_object78745:head# 2024-04-20T17:27:44.928 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7a45d19:::benchmark_data_smithi171_122772_object140171:head# 2024-04-20T17:27:44.928 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7a4ee36:::benchmark_data_smithi171_122772_object82385:head# 2024-04-20T17:27:44.929 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7a6710b:::benchmark_data_smithi171_122772_object131149:head# 2024-04-20T17:27:44.929 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7a6e5f3:::benchmark_data_smithi171_122772_object142100:head# 2024-04-20T17:27:44.929 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7a76435:::benchmark_data_smithi171_122772_object162883:head# 2024-04-20T17:27:44.930 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7a97274:::benchmark_data_smithi171_122772_object126381:head# 2024-04-20T17:27:44.930 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7a9d2b0:::benchmark_data_smithi171_122772_object115387:head# 2024-04-20T17:27:44.931 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7a9f9d6:::benchmark_data_smithi171_122772_object126165:head# 2024-04-20T17:27:44.931 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7aa7537:::benchmark_data_smithi171_122772_object101495:head# 2024-04-20T17:27:44.932 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7ab9f54:::benchmark_data_smithi171_122772_object84093:head# 2024-04-20T17:27:44.932 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7abecb5:::benchmark_data_smithi171_122772_object148347:head# 2024-04-20T17:27:44.933 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7ac17d4:::benchmark_data_smithi171_122772_object95642:head# 2024-04-20T17:27:44.933 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7ac3158:::benchmark_data_smithi171_122772_object81511:head# 2024-04-20T17:27:44.934 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7ace06a:::benchmark_data_smithi171_122772_object116877:head# 2024-04-20T17:27:44.935 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7ae7890:::benchmark_data_smithi171_122772_object136832:head# 2024-04-20T17:27:44.935 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7aec3db:::benchmark_data_smithi171_122772_object121947:head# 2024-04-20T17:27:44.935 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7b127e2:::benchmark_data_smithi171_122772_object165651:head# 2024-04-20T17:27:44.936 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7b18eb0:::benchmark_data_smithi171_122772_object104487:head# 2024-04-20T17:27:44.936 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7b205b8:::benchmark_data_smithi171_122772_object161271:head# 2024-04-20T17:27:44.937 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7b320c2:::benchmark_data_smithi171_122772_object150801:head# 2024-04-20T17:27:44.937 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7b3df26:::benchmark_data_smithi171_122772_object142612:head# 2024-04-20T17:27:44.938 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7b45ad7:::benchmark_data_smithi171_122772_object93113:head# 2024-04-20T17:27:44.938 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7b46e6d:::benchmark_data_smithi171_122772_object85916:head# 2024-04-20T17:27:44.939 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7b63c6e:::benchmark_data_smithi171_122772_object104908:head# 2024-04-20T17:27:44.939 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7b74f66:::benchmark_data_smithi171_122772_object145982:head# 2024-04-20T17:27:44.940 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7b838b4:::benchmark_data_smithi171_122772_object165464:head# 2024-04-20T17:27:44.941 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7b86f4d:::benchmark_data_smithi171_122772_object149955:head# 2024-04-20T17:27:44.941 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7b8960f:::benchmark_data_smithi171_122772_object165326:head# 2024-04-20T17:27:44.942 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7b8f4d8:::benchmark_data_smithi171_122772_object163741:head# 2024-04-20T17:27:44.942 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7b95d19:::benchmark_data_smithi171_122772_object107364:head# 2024-04-20T17:27:44.943 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7bae9dc:::benchmark_data_smithi171_122772_object167323:head# 2024-04-20T17:27:44.943 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7bbae11:::benchmark_data_smithi171_122772_object168278:head# 2024-04-20T17:27:44.944 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7bbc68e:::benchmark_data_smithi171_122772_object114585:head# 2024-04-20T17:27:44.944 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7bbc717:::benchmark_data_smithi171_122772_object168550:head# 2024-04-20T17:27:44.945 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7bd5fb1:::benchmark_data_smithi171_122772_object134190:head# 2024-04-20T17:27:44.945 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7bf1a76:::benchmark_data_smithi171_122772_object139980:head# 2024-04-20T17:27:44.946 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7bf52b5:::benchmark_data_smithi171_122772_object147360:head# 2024-04-20T17:27:44.946 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7c11641:::benchmark_data_smithi171_122772_object129689:head# 2024-04-20T17:27:44.947 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7c1dad3:::benchmark_data_smithi171_122772_object93941:head# 2024-04-20T17:27:44.947 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7c73089:::benchmark_data_smithi171_122772_object108014:head# 2024-04-20T17:27:44.948 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7c7a08a:::benchmark_data_smithi171_122772_object97249:head# 2024-04-20T17:27:44.948 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7c7a926:::benchmark_data_smithi171_122772_object163841:head# 2024-04-20T17:27:44.949 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7c88725:::benchmark_data_smithi171_122772_object133557:head# 2024-04-20T17:27:44.949 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7c8a38e:::benchmark_data_smithi171_122772_object113526:head# 2024-04-20T17:27:44.950 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7c90d9a:::benchmark_data_smithi171_122772_object143661:head# 2024-04-20T17:27:44.950 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7c93fd6:::benchmark_data_smithi171_122772_object99132:head# 2024-04-20T17:27:44.951 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7cd7d00:::benchmark_data_smithi171_122772_object115516:head# 2024-04-20T17:27:44.951 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7cde7b9:::benchmark_data_smithi171_122772_object101913:head# 2024-04-20T17:27:44.952 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7cdeef5:::benchmark_data_smithi171_122772_object156681:head# 2024-04-20T17:27:44.952 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7cec1d5:::benchmark_data_smithi171_122772_object157166:head# 2024-04-20T17:27:44.953 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7cefed3:::benchmark_data_smithi171_122772_object95300:head# 2024-04-20T17:27:44.953 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7d07f19:::benchmark_data_smithi171_122772_object103037:head# 2024-04-20T17:27:44.954 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7d1ea3d:::benchmark_data_smithi171_122772_object79597:head# 2024-04-20T17:27:44.954 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7d25153:::benchmark_data_smithi171_122772_object131158:head# 2024-04-20T17:27:44.955 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7d2d833:::benchmark_data_smithi171_122772_object102606:head# 2024-04-20T17:27:44.955 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7d6546d:::benchmark_data_smithi171_122772_object118113:head# 2024-04-20T17:27:44.956 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7d6910d:::benchmark_data_smithi171_122772_object113930:head# 2024-04-20T17:27:44.956 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7d79659:::benchmark_data_smithi171_122772_object120789:head# 2024-04-20T17:27:44.957 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7d79f66:::benchmark_data_smithi171_122772_object112742:head# 2024-04-20T17:27:44.957 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7d7a1da:::benchmark_data_smithi171_122772_object103818:head# 2024-04-20T17:27:44.958 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7d929c1:::benchmark_data_smithi171_122772_object124076:head# 2024-04-20T17:27:44.958 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7d97f7c:::benchmark_data_smithi171_122772_object135148:head# 2024-04-20T17:27:44.959 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7d9ec25:::benchmark_data_smithi171_122772_object125412:head# 2024-04-20T17:27:44.959 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7d9ef0b:::benchmark_data_smithi171_122772_object153527:head# 2024-04-20T17:27:44.960 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7db6547:::benchmark_data_smithi171_122772_object148035:head# 2024-04-20T17:27:44.960 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7dcb094:::benchmark_data_smithi171_122772_object163719:head# 2024-04-20T17:27:44.961 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7de4db5:::benchmark_data_smithi171_122772_object138896:head# 2024-04-20T17:27:44.961 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7deebd7:::benchmark_data_smithi171_122772_object88561:head# 2024-04-20T17:27:44.962 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7def82b:::benchmark_data_smithi171_122772_object158338:head# 2024-04-20T17:27:44.962 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7df24e3:::benchmark_data_smithi171_122772_object157995:head# 2024-04-20T17:27:44.962 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7dfa466:::benchmark_data_smithi171_122772_object92209:head# 2024-04-20T17:27:44.963 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7dfe928:::benchmark_data_smithi171_122772_object124994:head# 2024-04-20T17:27:44.963 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7e00801:::benchmark_data_smithi171_122772_object101827:head# 2024-04-20T17:27:44.963 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7e37b3c:::benchmark_data_smithi171_122772_object81833:head# 2024-04-20T17:27:44.964 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7e424ca:::benchmark_data_smithi171_122772_object141739:head# 2024-04-20T17:27:44.965 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7e446c9:::benchmark_data_smithi171_122772_object85744:head# 2024-04-20T17:27:44.965 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7e4832a:::benchmark_data_smithi171_122772_object89474:head# 2024-04-20T17:27:44.966 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7e543a2:::benchmark_data_smithi171_122772_object145461:head# 2024-04-20T17:27:44.966 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7e60d58:::benchmark_data_smithi171_122772_object111736:head# 2024-04-20T17:27:44.967 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7e74cbc:::benchmark_data_smithi171_122772_object125627:head# 2024-04-20T17:27:44.967 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7e92954:::benchmark_data_smithi171_122772_object122715:head# 2024-04-20T17:27:44.968 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7e9c8d6:::benchmark_data_smithi171_122772_object163627:head# 2024-04-20T17:27:44.968 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7eadef4:::benchmark_data_smithi171_122772_object112160:head# 2024-04-20T17:27:44.968 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7eaed49:::benchmark_data_smithi171_122772_object96724:head# 2024-04-20T17:27:44.969 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7eba603:::benchmark_data_smithi171_122772_object129033:head# 2024-04-20T17:27:44.969 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7ec4c13:::benchmark_data_smithi171_122772_object164422:head# 2024-04-20T17:27:44.970 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7ec78d4:::benchmark_data_smithi171_122772_object79530:head# 2024-04-20T17:27:44.970 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7ecf898:::benchmark_data_smithi171_122772_object83458:head# 2024-04-20T17:27:44.971 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7edd6b4:::benchmark_data_smithi171_122772_object169866:head# 2024-04-20T17:27:44.971 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7ee57cf:::benchmark_data_smithi171_122772_object140969:head# 2024-04-20T17:27:44.971 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7ef08a6:::benchmark_data_smithi171_122772_object113964:head# 2024-04-20T17:27:44.972 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7ef82b7:::benchmark_data_smithi171_122772_object156495:head# 2024-04-20T17:27:44.972 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7f020e5:::benchmark_data_smithi171_122772_object96672:head# 2024-04-20T17:27:44.973 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7f0256d:::benchmark_data_smithi171_122772_object139868:head# 2024-04-20T17:27:44.974 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7f163ca:::benchmark_data_smithi171_122772_object137332:head# 2024-04-20T17:27:44.974 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7f3653c:::benchmark_data_smithi171_122772_object158222:head# 2024-04-20T17:27:44.974 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7f3740f:::benchmark_data_smithi171_122772_object159762:head# 2024-04-20T17:27:44.975 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7f40443:::benchmark_data_smithi171_122772_object140878:head# 2024-04-20T17:27:44.976 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7f48afc:::benchmark_data_smithi171_122772_object104074:head# 2024-04-20T17:27:44.976 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7f4bf3b:::benchmark_data_smithi171_122772_object136308:head# 2024-04-20T17:27:44.976 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7f4e07b:::benchmark_data_smithi171_122772_object147388:head# 2024-04-20T17:27:44.977 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7f6709c:::benchmark_data_smithi171_122772_object155375:head# 2024-04-20T17:27:44.978 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7f7deae:::benchmark_data_smithi171_122772_object134224:head# 2024-04-20T17:27:44.978 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7f8d24c:::benchmark_data_smithi171_122772_object165307:head# 2024-04-20T17:27:44.979 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7f9e32a:::benchmark_data_smithi171_122772_object129400:head# 2024-04-20T17:27:44.979 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7fa2fa1:::benchmark_data_smithi171_122772_object79503:head# 2024-04-20T17:27:44.980 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7fb2320:::benchmark_data_smithi171_122772_object102561:head# 2024-04-20T17:27:44.980 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7fb99a9:::benchmark_data_smithi171_122772_object114053:head# 2024-04-20T17:27:44.981 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7fd5d1a:::benchmark_data_smithi171_122772_object163679:head# 2024-04-20T17:27:44.981 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7fd93ee:::benchmark_data_smithi171_122772_object122420:head# 2024-04-20T17:27:44.982 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a7fdbe71:::benchmark_data_smithi171_122772_object79415:head# 2024-04-20T17:27:44.982 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a800d57e:::benchmark_data_smithi171_122772_object78608:head# 2024-04-20T17:27:44.983 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a800f6b3:::benchmark_data_smithi171_122772_object130365:head# 2024-04-20T17:27:44.983 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8029a95:::benchmark_data_smithi171_122772_object114951:head# 2024-04-20T17:27:44.984 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a803a84f:::benchmark_data_smithi171_122772_object122710:head# 2024-04-20T17:27:44.984 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8055ab6:::benchmark_data_smithi171_122772_object162172:head# 2024-04-20T17:27:44.985 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a805f6fd:::benchmark_data_smithi171_122772_object146001:head# 2024-04-20T17:27:44.985 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8061634:::benchmark_data_smithi171_122772_object133865:head# 2024-04-20T17:27:44.986 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a806831f:::benchmark_data_smithi171_122772_object161231:head# 2024-04-20T17:27:44.986 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a807156f:::benchmark_data_smithi171_122772_object170152:head# 2024-04-20T17:27:44.987 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a80727ee:::benchmark_data_smithi171_122772_object105837:head# 2024-04-20T17:27:44.987 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a808c866:::benchmark_data_smithi171_122772_object145968:head# 2024-04-20T17:27:44.988 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a808e410:::benchmark_data_smithi171_122772_object111056:head# 2024-04-20T17:27:44.988 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8095d0b:::benchmark_data_smithi171_122772_object108325:head# 2024-04-20T17:27:44.989 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a80a0229:::benchmark_data_smithi171_122772_object119240:head# 2024-04-20T17:27:44.989 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a80bfbb9:::benchmark_data_smithi171_122772_object99890:head# 2024-04-20T17:27:44.990 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a80d07ef:::benchmark_data_smithi171_122772_object153197:head# 2024-04-20T17:27:44.990 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a80e2452:::benchmark_data_smithi171_122772_object131428:head# 2024-04-20T17:27:44.991 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a80ed4fd:::benchmark_data_smithi171_122772_object91393:head# 2024-04-20T17:27:44.991 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a80efd00:::benchmark_data_smithi171_122772_object83058:head# 2024-04-20T17:27:44.992 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a80f3fc4:::benchmark_data_smithi171_122772_object102907:head# 2024-04-20T17:27:44.992 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a81009dc:::benchmark_data_smithi171_122772_object167673:head# 2024-04-20T17:27:44.992 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a810c5d0:::benchmark_data_smithi171_122772_object160631:head# 2024-04-20T17:27:44.993 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a810f214:::benchmark_data_smithi171_122772_object159460:head# 2024-04-20T17:27:44.993 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a81168cb:::benchmark_data_smithi171_122772_object163998:head# 2024-04-20T17:27:44.994 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a811ea4f:::benchmark_data_smithi171_122772_object140947:head# 2024-04-20T17:27:44.994 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a811f91e:::benchmark_data_smithi171_122772_object155135:head# 2024-04-20T17:27:44.995 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8168f12:::benchmark_data_smithi171_122772_object121901:head# 2024-04-20T17:27:44.995 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8173332:::benchmark_data_smithi171_122772_object162364:head# 2024-04-20T17:27:44.996 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a817cc49:::benchmark_data_smithi171_122772_object110277:head# 2024-04-20T17:27:44.997 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a81968b5:::benchmark_data_smithi171_122772_object141861:head# 2024-04-20T17:27:44.997 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a81bc1ee:::benchmark_data_smithi171_122772_object131979:head# 2024-04-20T17:27:44.997 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a81c235e:::benchmark_data_smithi171_122772_object110897:head# 2024-04-20T17:27:44.998 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a81c82cd:::benchmark_data_smithi171_122772_object166950:head# 2024-04-20T17:27:44.998 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a81cdc44:::benchmark_data_smithi171_122772_object92961:head# 2024-04-20T17:27:44.999 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a81cef6d:::benchmark_data_smithi171_122772_object111451:head# 2024-04-20T17:27:44.999 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a81dcd5c:::benchmark_data_smithi171_122772_object155070:head# 2024-04-20T17:27:44.999 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a81f1011:::benchmark_data_smithi171_122772_object120316:head# 2024-04-20T17:27:45.000 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a81f8c8a:::benchmark_data_smithi171_122772_object97783:head# 2024-04-20T17:27:45.001 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a81fca61:::benchmark_data_smithi171_122772_object166137:head# 2024-04-20T17:27:45.001 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a81ff882:::benchmark_data_smithi171_122772_object101423:head# 2024-04-20T17:27:45.001 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a82170d5:::benchmark_data_smithi171_122772_object99531:head# 2024-04-20T17:27:45.002 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a821d697:::benchmark_data_smithi171_122772_object125038:head# 2024-04-20T17:27:45.002 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a821f77b:::benchmark_data_smithi171_122772_object155583:head# 2024-04-20T17:27:45.003 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a82213d6:::benchmark_data_smithi171_122772_object134543:head# 2024-04-20T17:27:45.003 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8221e4b:::benchmark_data_smithi171_122772_object86305:head# 2024-04-20T17:27:45.004 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a82295a6:::benchmark_data_smithi171_122772_object140608:head# 2024-04-20T17:27:45.005 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8234788:::benchmark_data_smithi171_122772_object169499:head# 2024-04-20T17:27:45.005 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a823b394:::benchmark_data_smithi171_122772_object160355:head# 2024-04-20T17:27:45.006 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a823b4dc:::benchmark_data_smithi171_122772_object142451:head# 2024-04-20T17:27:45.006 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a824539e:::benchmark_data_smithi171_122772_object139490:head# 2024-04-20T17:27:45.007 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8249eb7:::benchmark_data_smithi171_122772_object138388:head# 2024-04-20T17:27:45.007 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a824c6ba:::benchmark_data_smithi171_122772_object164587:head# 2024-04-20T17:27:45.008 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a824ce2c:::benchmark_data_smithi171_122772_object132178:head# 2024-04-20T17:27:45.008 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8256c4a:::benchmark_data_smithi171_122772_object143209:head# 2024-04-20T17:27:45.008 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a825dc20:::benchmark_data_smithi171_122772_object140136:head# 2024-04-20T17:27:45.009 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8263cd2:::benchmark_data_smithi171_122772_object161932:head# 2024-04-20T17:27:45.009 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a826747e:::benchmark_data_smithi171_122772_object111463:head# 2024-04-20T17:27:45.010 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a826b0c7:::benchmark_data_smithi171_122772_object122123:head# 2024-04-20T17:27:45.010 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a827963f:::benchmark_data_smithi171_122772_object154072:head# 2024-04-20T17:27:45.011 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a827d56f:::benchmark_data_smithi171_122772_object113740:head# 2024-04-20T17:27:45.011 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8281a3c:::benchmark_data_smithi171_122772_object93263:head# 2024-04-20T17:27:45.012 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a828ad4f:::benchmark_data_smithi171_122772_object135547:head# 2024-04-20T17:27:45.012 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a82997d6:::benchmark_data_smithi171_122772_object148440:head# 2024-04-20T17:27:45.012 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a82a3b93:::benchmark_data_smithi171_122772_object81885:head# 2024-04-20T17:27:45.013 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a82a4159:::benchmark_data_smithi171_122772_object165513:head# 2024-04-20T17:27:45.013 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a82a6221:::benchmark_data_smithi171_122772_object159068:head# 2024-04-20T17:27:45.014 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a82b099a:::benchmark_data_smithi171_122772_object145084:head# 2024-04-20T17:27:45.015 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a82cfcd6:::benchmark_data_smithi171_122772_object132607:head# 2024-04-20T17:27:45.015 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a82d1e59:::benchmark_data_smithi171_122772_object164590:head# 2024-04-20T17:27:45.016 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a82da1e2:::benchmark_data_smithi171_122772_object79615:head# 2024-04-20T17:27:45.016 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a830389d:::benchmark_data_smithi171_122772_object134041:head# 2024-04-20T17:27:45.017 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a831cd71:::benchmark_data_smithi171_122772_object112536:head# 2024-04-20T17:27:45.017 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8333f6d:::benchmark_data_smithi171_122772_object103845:head# 2024-04-20T17:27:45.018 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8335312:::benchmark_data_smithi171_122772_object89472:head# 2024-04-20T17:27:45.018 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8340fdd:::benchmark_data_smithi171_122772_object93483:head# 2024-04-20T17:27:45.019 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8344e1d:::benchmark_data_smithi171_122772_object84300:head# 2024-04-20T17:27:45.019 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a834e06f:::benchmark_data_smithi171_122772_object122620:head# 2024-04-20T17:27:45.020 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a836112c:::benchmark_data_smithi171_122772_object109450:head# 2024-04-20T17:27:45.020 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a836f1eb:::benchmark_data_smithi171_122772_object81247:head# 2024-04-20T17:27:45.021 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8374e6c:::benchmark_data_smithi171_122772_object127949:head# 2024-04-20T17:27:45.021 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8376fe8:::benchmark_data_smithi171_122772_object111094:head# 2024-04-20T17:27:45.022 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a839287b:::benchmark_data_smithi171_122772_object156367:head# 2024-04-20T17:27:45.022 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a83b68b5:::benchmark_data_smithi171_122772_object123800:head# 2024-04-20T17:27:45.023 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a83bd942:::benchmark_data_smithi171_122772_object153642:head# 2024-04-20T17:27:45.023 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a83bebe0:::benchmark_data_smithi171_122772_object112344:head# 2024-04-20T17:27:45.024 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a83c4837:::benchmark_data_smithi171_122772_object127375:head# 2024-04-20T17:27:45.024 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a83cbe00:::benchmark_data_smithi171_122772_object163350:head# 2024-04-20T17:27:45.025 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a83f11c2:::benchmark_data_smithi171_122772_object117755:head# 2024-04-20T17:27:45.025 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a83f1603:::benchmark_data_smithi171_122772_object154954:head# 2024-04-20T17:27:45.026 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8425eb4:::benchmark_data_smithi171_122772_object164373:head# 2024-04-20T17:27:45.027 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a842a7a6:::benchmark_data_smithi171_122772_object156504:head# 2024-04-20T17:27:45.027 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a842f367:::benchmark_data_smithi171_122772_object94873:head# 2024-04-20T17:27:45.028 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a843da3d:::benchmark_data_smithi171_122772_object164634:head# 2024-04-20T17:27:45.028 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a84442e1:::benchmark_data_smithi171_122772_object102577:head# 2024-04-20T17:27:45.028 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a844e9f2:::benchmark_data_smithi171_122772_object113469:head# 2024-04-20T17:27:45.029 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a844f893:::benchmark_data_smithi171_122772_object104722:head# 2024-04-20T17:27:45.029 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a84549bc:::benchmark_data_smithi171_122772_object114021:head# 2024-04-20T17:27:45.030 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8474234:::benchmark_data_smithi171_122772_object78242:head# 2024-04-20T17:27:45.030 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8475429:::benchmark_data_smithi171_122772_object161614:head# 2024-04-20T17:27:45.031 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a847a721:::benchmark_data_smithi171_122772_object163579:head# 2024-04-20T17:27:45.031 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a847cb4c:::benchmark_data_smithi171_122772_object81281:head# 2024-04-20T17:27:45.032 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a847edf2:::benchmark_data_smithi171_122772_object91195:head# 2024-04-20T17:27:45.032 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a848dff5:::benchmark_data_smithi171_122772_object100782:head# 2024-04-20T17:27:45.033 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a84963ea:::benchmark_data_smithi171_122772_object106927:head# 2024-04-20T17:27:45.033 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a849e09a:::benchmark_data_smithi171_122772_object117576:head# 2024-04-20T17:27:45.034 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a84a1f62:::benchmark_data_smithi171_122772_object84185:head# 2024-04-20T17:27:45.034 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a84a2d80:::benchmark_data_smithi171_122772_object140365:head# 2024-04-20T17:27:45.035 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a84b21f4:::benchmark_data_smithi171_122772_object144027:head# 2024-04-20T17:27:45.035 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a84ba5c0:::benchmark_data_smithi171_122772_object154513:head# 2024-04-20T17:27:45.036 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a84baa60:::benchmark_data_smithi171_122772_object100108:head# 2024-04-20T17:27:45.036 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a84bd4ac:::benchmark_data_smithi171_122772_object153067:head# 2024-04-20T17:27:45.037 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a84d8aac:::benchmark_data_smithi171_122772_object102164:head# 2024-04-20T17:27:45.037 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a84e13c0:::benchmark_data_smithi171_122772_object85888:head# 2024-04-20T17:27:45.038 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a84fc6c6:::benchmark_data_smithi171_122772_object93035:head# 2024-04-20T17:27:45.038 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a84fc86c:::benchmark_data_smithi171_122772_object93945:head# 2024-04-20T17:27:45.039 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8503999:::benchmark_data_smithi171_122772_object134331:head# 2024-04-20T17:27:45.039 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8508c24:::benchmark_data_smithi171_122772_object104970:head# 2024-04-20T17:27:45.040 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a850b37f:::benchmark_data_smithi171_122772_object98801:head# 2024-04-20T17:27:45.040 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a851e080:::benchmark_data_smithi171_122772_object140519:head# 2024-04-20T17:27:45.040 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a854a761:::benchmark_data_smithi171_122772_object155114:head# 2024-04-20T17:27:45.041 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8554515:::benchmark_data_smithi171_122772_object122217:head# 2024-04-20T17:27:45.042 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8565a0b:::benchmark_data_smithi171_122772_object136309:head# 2024-04-20T17:27:45.042 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a856b4d5:::benchmark_data_smithi171_122772_object161760:head# 2024-04-20T17:27:45.043 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8574faa:::benchmark_data_smithi171_122772_object143117:head# 2024-04-20T17:27:45.044 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a857582d:::benchmark_data_smithi171_122772_object160708:head# 2024-04-20T17:27:45.044 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a85764b5:::benchmark_data_smithi171_122772_object82814:head# 2024-04-20T17:27:45.045 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a857b0ff:::benchmark_data_smithi171_122772_object142893:head# 2024-04-20T17:27:45.045 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a857f83d:::benchmark_data_smithi171_122772_object141551:head# 2024-04-20T17:27:45.046 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a857fd4e:::benchmark_data_smithi171_122772_object164904:head# 2024-04-20T17:27:45.046 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a85838c0:::benchmark_data_smithi171_122772_object140514:head# 2024-04-20T17:27:45.047 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8583a67:::benchmark_data_smithi171_122772_object105512:head# 2024-04-20T17:27:45.047 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8588364:::benchmark_data_smithi171_122772_object87266:head# 2024-04-20T17:27:45.048 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a858ff2d:::benchmark_data_smithi171_122772_object134781:head# 2024-04-20T17:27:45.048 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a85917d7:::benchmark_data_smithi171_122772_object157998:head# 2024-04-20T17:27:45.049 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8596873:::benchmark_data_smithi171_122772_object96266:head# 2024-04-20T17:27:45.049 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a85b025d:::benchmark_data_smithi171_122772_object132747:head# 2024-04-20T17:27:45.050 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a85ce9f9:::benchmark_data_smithi171_122772_object113661:head# 2024-04-20T17:27:45.050 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a85d518f:::benchmark_data_smithi171_122772_object119490:head# 2024-04-20T17:27:45.051 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a85df846:::benchmark_data_smithi171_122772_object157563:head# 2024-04-20T17:27:45.051 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a85e1aa4:::benchmark_data_smithi171_122772_object152179:head# 2024-04-20T17:27:45.051 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a85ee58a:::benchmark_data_smithi171_122772_object91039:head# 2024-04-20T17:27:45.052 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a86040fd:::benchmark_data_smithi171_122772_object142049:head# 2024-04-20T17:27:45.052 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8605afb:::benchmark_data_smithi171_122772_object98760:head# 2024-04-20T17:27:45.053 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8609723:::benchmark_data_smithi171_122772_object88875:head# 2024-04-20T17:27:45.053 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8610d59:::benchmark_data_smithi171_122772_object115388:head# 2024-04-20T17:27:45.054 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a861be01:::benchmark_data_smithi171_122772_object125359:head# 2024-04-20T17:27:45.054 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a861e9ed:::benchmark_data_smithi171_122772_object90681:head# 2024-04-20T17:27:45.055 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8622fab:::benchmark_data_smithi171_122772_object101290:head# 2024-04-20T17:27:45.055 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8632959:::benchmark_data_smithi171_122772_object138482:head# 2024-04-20T17:27:45.056 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a863c577:::benchmark_data_smithi171_122772_object132482:head# 2024-04-20T17:27:45.056 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a863d61b:::benchmark_data_smithi171_122772_object127709:head# 2024-04-20T17:27:45.057 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8646de4:::benchmark_data_smithi171_122772_object142720:head# 2024-04-20T17:27:45.057 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8647026:::benchmark_data_smithi171_122772_object101309:head# 2024-04-20T17:27:45.058 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a864d070:::benchmark_data_smithi171_122772_object151777:head# 2024-04-20T17:27:45.058 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a865fa48:::benchmark_data_smithi171_122772_object126032:head# 2024-04-20T17:27:45.058 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a86a6e86:::benchmark_data_smithi171_122772_object121604:head# 2024-04-20T17:27:45.059 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a86a7b6e:::benchmark_data_smithi171_122772_object81334:head# 2024-04-20T17:27:45.059 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a86b195b:::benchmark_data_smithi171_122772_object128320:head# 2024-04-20T17:27:45.060 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a86c95c3:::benchmark_data_smithi171_122772_object116642:head# 2024-04-20T17:27:45.060 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a86cd020:::benchmark_data_smithi171_122772_object154033:head# 2024-04-20T17:27:45.061 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a86ea6e4:::benchmark_data_smithi171_122772_object101837:head# 2024-04-20T17:27:45.061 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a86f04b5:::benchmark_data_smithi171_122772_object165727:head# 2024-04-20T17:27:45.062 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a86f4373:::benchmark_data_smithi171_122772_object106300:head# 2024-04-20T17:27:45.062 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a86f594a:::benchmark_data_smithi171_122772_object164302:head# 2024-04-20T17:27:45.062 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a870fb1b:::benchmark_data_smithi171_122772_object160351:head# 2024-04-20T17:27:45.063 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a87160ca:::benchmark_data_smithi171_122772_object148574:head# 2024-04-20T17:27:45.063 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a87179bb:::benchmark_data_smithi171_122772_object119812:head# 2024-04-20T17:27:45.064 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a871eefe:::benchmark_data_smithi171_122772_object113165:head# 2024-04-20T17:27:45.065 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a873f5f6:::benchmark_data_smithi171_122772_object152549:head# 2024-04-20T17:27:45.065 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a875496b:::benchmark_data_smithi171_122772_object102459:head# 2024-04-20T17:27:45.066 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a875cb7d:::benchmark_data_smithi171_122772_object163421:head# 2024-04-20T17:27:45.066 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a876666c:::benchmark_data_smithi171_122772_object82882:head# 2024-04-20T17:27:45.067 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a876c8c1:::benchmark_data_smithi171_122772_object86450:head# 2024-04-20T17:27:45.067 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a877e772:::benchmark_data_smithi171_122772_object79780:head# 2024-04-20T17:27:45.068 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8786999:::benchmark_data_smithi171_122772_object112528:head# 2024-04-20T17:27:45.069 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a879e50d:::benchmark_data_smithi171_122772_object82390:head# 2024-04-20T17:27:45.069 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a87c3333:::benchmark_data_smithi171_122772_object90666:head# 2024-04-20T17:27:45.069 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a87d6914:::benchmark_data_smithi171_122772_object86625:head# 2024-04-20T17:27:45.070 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a87e444c:::benchmark_data_smithi171_122772_object98478:head# 2024-04-20T17:27:45.070 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a87f3d5f:::benchmark_data_smithi171_122772_object115267:head# 2024-04-20T17:27:45.071 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a88008a3:::benchmark_data_smithi171_122772_object86815:head# 2024-04-20T17:27:45.071 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a881287d:::benchmark_data_smithi171_122772_object157874:head# 2024-04-20T17:27:45.072 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a881a1bb:::benchmark_data_smithi171_122772_object127617:head# 2024-04-20T17:27:45.072 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a88302f2:::benchmark_data_smithi171_122772_object154017:head# 2024-04-20T17:27:45.073 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a883a8ff:::benchmark_data_smithi171_122772_object162779:head# 2024-04-20T17:27:45.073 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a88413ce:::benchmark_data_smithi171_122772_object84034:head# 2024-04-20T17:27:45.074 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8841f48:::benchmark_data_smithi171_122772_object161135:head# 2024-04-20T17:27:45.074 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8844718:::benchmark_data_smithi171_122772_object118111:head# 2024-04-20T17:27:45.075 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a884e474:::benchmark_data_smithi171_122772_object94337:head# 2024-04-20T17:27:45.075 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8853661:::benchmark_data_smithi171_122772_object125284:head# 2024-04-20T17:27:45.076 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8876c42:::benchmark_data_smithi171_122772_object107149:head# 2024-04-20T17:27:45.076 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8895e90:::benchmark_data_smithi171_122772_object134864:head# 2024-04-20T17:27:45.077 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8899503:::benchmark_data_smithi171_122772_object161139:head# 2024-04-20T17:27:45.077 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a88a2f5d:::benchmark_data_smithi171_122772_object111619:head# 2024-04-20T17:27:45.078 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a88a36cd:::benchmark_data_smithi171_122772_object137988:head# 2024-04-20T17:27:45.078 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a88a5ca5:::benchmark_data_smithi171_122772_object121929:head# 2024-04-20T17:27:45.079 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a88a9ef0:::benchmark_data_smithi171_122772_object128545:head# 2024-04-20T17:27:45.079 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a88aef7b:::benchmark_data_smithi171_122772_object135652:head# 2024-04-20T17:27:45.080 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a88cae0b:::benchmark_data_smithi171_122772_object169413:head# 2024-04-20T17:27:45.080 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a88d4f28:::benchmark_data_smithi171_122772_object85383:head# 2024-04-20T17:27:45.081 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a88ea8c0:::benchmark_data_smithi171_122772_object80943:head# 2024-04-20T17:27:45.081 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a88f27ad:::benchmark_data_smithi171_122772_object138164:head# 2024-04-20T17:27:45.081 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8913e78:::benchmark_data_smithi171_122772_object150772:head# 2024-04-20T17:27:45.082 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a891f0d2:::benchmark_data_smithi171_122772_object94915:head# 2024-04-20T17:27:45.082 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a891f386:::benchmark_data_smithi171_122772_object84390:head# 2024-04-20T17:27:45.083 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a892a437:::benchmark_data_smithi171_122772_object162856:head# 2024-04-20T17:27:45.083 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a892b79a:::benchmark_data_smithi171_122772_object107984:head# 2024-04-20T17:27:45.084 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a893a789:::benchmark_data_smithi171_122772_object125812:head# 2024-04-20T17:27:45.084 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8950623:::benchmark_data_smithi171_122772_object111359:head# 2024-04-20T17:27:45.085 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8953a8e:::benchmark_data_smithi171_122772_object142910:head# 2024-04-20T17:27:45.086 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a895a0db:::benchmark_data_smithi171_122772_object103741:head# 2024-04-20T17:27:45.086 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8979297:::benchmark_data_smithi171_122772_object137712:head# 2024-04-20T17:27:45.087 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a897ae88:::benchmark_data_smithi171_122772_object85315:head# 2024-04-20T17:27:45.087 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a897c0be:::benchmark_data_smithi171_122772_object163009:head# 2024-04-20T17:27:45.088 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a89a1410:::benchmark_data_smithi171_122772_object110824:head# 2024-04-20T17:27:45.088 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a89beb46:::benchmark_data_smithi171_122772_object137144:head# 2024-04-20T17:27:45.089 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a89c15bf:::benchmark_data_smithi171_122772_object153691:head# 2024-04-20T17:27:45.089 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a89d08fc:::benchmark_data_smithi171_122772_object164728:head# 2024-04-20T17:27:45.090 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a89d88a8:::benchmark_data_smithi171_122772_object148100:head# 2024-04-20T17:27:45.090 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a89e3162:::benchmark_data_smithi171_122772_object91884:head# 2024-04-20T17:27:45.090 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a89e61b4:::benchmark_data_smithi171_122772_object117091:head# 2024-04-20T17:27:45.091 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a89ecefa:::benchmark_data_smithi171_122772_object144596:head# 2024-04-20T17:27:45.092 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a89f6ca3:::benchmark_data_smithi171_122772_object159118:head# 2024-04-20T17:27:45.092 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8a1aae8:::benchmark_data_smithi171_122772_object149387:head# 2024-04-20T17:27:45.093 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8a33115:::benchmark_data_smithi171_122772_object81521:head# 2024-04-20T17:27:45.093 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8a39e85:::benchmark_data_smithi171_122772_object101774:head# 2024-04-20T17:27:45.093 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8a3dda5:::benchmark_data_smithi171_122772_object88786:head# 2024-04-20T17:27:45.094 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8a41e2d:::benchmark_data_smithi171_122772_object91634:head# 2024-04-20T17:27:45.094 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8a4b75b:::benchmark_data_smithi171_122772_object152445:head# 2024-04-20T17:27:45.095 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8a4cc01:::benchmark_data_smithi171_122772_object94339:head# 2024-04-20T17:27:45.095 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8a4f7ac:::benchmark_data_smithi171_122772_object101186:head# 2024-04-20T17:27:45.096 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8a4ffeb:::benchmark_data_smithi171_122772_object85473:head# 2024-04-20T17:27:45.096 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8a6544a:::benchmark_data_smithi171_122772_object117833:head# 2024-04-20T17:27:45.097 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8a6f733:::benchmark_data_smithi171_122772_object168036:head# 2024-04-20T17:27:45.097 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8a7abf6:::benchmark_data_smithi171_122772_object87174:head# 2024-04-20T17:27:45.098 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8a7d91a:::benchmark_data_smithi171_122772_object87080:head# 2024-04-20T17:27:45.098 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8a7ef80:::benchmark_data_smithi171_122772_object93233:head# 2024-04-20T17:27:45.099 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8a86592:::benchmark_data_smithi171_122772_object145676:head# 2024-04-20T17:27:45.100 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8a8ab77:::benchmark_data_smithi171_122772_object126849:head# 2024-04-20T17:27:45.100 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8a8d3f8:::benchmark_data_smithi171_122772_object117221:head# 2024-04-20T17:27:45.100 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8a90111:::benchmark_data_smithi171_122772_object146697:head# 2024-04-20T17:27:45.101 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8a95d94:::benchmark_data_smithi171_122772_object128197:head# 2024-04-20T17:27:45.101 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8a99ee7:::benchmark_data_smithi171_122772_object127346:head# 2024-04-20T17:27:45.102 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8a9dade:::benchmark_data_smithi171_122772_object140553:head# 2024-04-20T17:27:45.102 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8aa22be:::benchmark_data_smithi171_122772_object134899:head# 2024-04-20T17:27:45.103 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8aa49cc:::benchmark_data_smithi171_122772_object106308:head# 2024-04-20T17:27:45.103 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8aa667a:::benchmark_data_smithi171_122772_object151881:head# 2024-04-20T17:27:45.104 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8aab019:::benchmark_data_smithi171_122772_object121091:head# 2024-04-20T17:27:45.104 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8aab0d9:::benchmark_data_smithi171_122772_object158039:head# 2024-04-20T17:27:45.105 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8ab2367:::benchmark_data_smithi171_122772_object107512:head# 2024-04-20T17:27:45.105 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8ab8748:::benchmark_data_smithi171_122772_object85672:head# 2024-04-20T17:27:45.106 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8abbc77:::benchmark_data_smithi171_122772_object160056:head# 2024-04-20T17:27:45.106 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8ace5d1:::benchmark_data_smithi171_122772_object165858:head# 2024-04-20T17:27:45.107 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8ad6820:::benchmark_data_smithi171_122772_object100559:head# 2024-04-20T17:27:45.107 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8aec6a6:::benchmark_data_smithi171_122772_object112486:head# 2024-04-20T17:27:45.108 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8af8cdf:::benchmark_data_smithi171_122772_object110581:head# 2024-04-20T17:27:45.108 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8af9aec:::benchmark_data_smithi171_122772_object122781:head# 2024-04-20T17:27:45.109 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8b0851c:::benchmark_data_smithi171_122772_object90594:head# 2024-04-20T17:27:45.109 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8b19b56:::benchmark_data_smithi171_122772_object104957:head# 2024-04-20T17:27:45.109 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8b224d5:::benchmark_data_smithi171_122772_object102205:head# 2024-04-20T17:27:45.110 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8b367b5:::benchmark_data_smithi171_122772_object118006:head# 2024-04-20T17:27:45.111 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8b52631:::benchmark_data_smithi171_122772_object152183:head# 2024-04-20T17:27:45.111 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8b786eb:::benchmark_data_smithi171_122772_object85882:head# 2024-04-20T17:27:45.112 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8b82ba7:::benchmark_data_smithi171_122772_object163048:head# 2024-04-20T17:27:45.112 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8b852bd:::benchmark_data_smithi171_122772_object104979:head# 2024-04-20T17:27:45.113 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8b9888e:::benchmark_data_smithi171_122772_object134084:head# 2024-04-20T17:27:45.113 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8ba05e5:::benchmark_data_smithi171_122772_object157986:head# 2024-04-20T17:27:45.114 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8bbcb4a:::benchmark_data_smithi171_122772_object135709:head# 2024-04-20T17:27:45.114 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8bc9d3d:::benchmark_data_smithi171_122772_object158348:head# 2024-04-20T17:27:45.115 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8bcef54:::benchmark_data_smithi171_122772_object123546:head# 2024-04-20T17:27:45.115 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8be1847:::benchmark_data_smithi171_122772_object95578:head# 2024-04-20T17:27:45.116 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8be2834:::benchmark_data_smithi171_122772_object162860:head# 2024-04-20T17:27:45.116 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8c01f48:::benchmark_data_smithi171_122772_object99204:head# 2024-04-20T17:27:45.117 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8c03d51:::benchmark_data_smithi171_122772_object136284:head# 2024-04-20T17:27:45.117 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8c153c9:::benchmark_data_smithi171_122772_object152602:head# 2024-04-20T17:27:45.118 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8c5480b:::benchmark_data_smithi171_122772_object84155:head# 2024-04-20T17:27:45.118 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8c5ae85:::benchmark_data_smithi171_122772_object116491:head# 2024-04-20T17:27:45.119 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8c64a70:::benchmark_data_smithi171_122772_object169828:head# 2024-04-20T17:27:45.119 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8c800f2:::benchmark_data_smithi171_122772_object85415:head# 2024-04-20T17:27:45.120 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8c881f7:::benchmark_data_smithi171_122772_object151355:head# 2024-04-20T17:27:45.120 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8cb164b:::benchmark_data_smithi171_122772_object140612:head# 2024-04-20T17:27:45.121 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8cbbe0f:::benchmark_data_smithi171_122772_object120686:head# 2024-04-20T17:27:45.121 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8cc3ca1:::benchmark_data_smithi171_122772_object151255:head# 2024-04-20T17:27:45.122 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8cc7524:::benchmark_data_smithi171_122772_object89902:head# 2024-04-20T17:27:45.122 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8cdd450:::benchmark_data_smithi171_122772_object119827:head# 2024-04-20T17:27:45.122 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8cf1df4:::benchmark_data_smithi171_122772_object142838:head# 2024-04-20T17:27:45.123 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8d08bfd:::benchmark_data_smithi171_122772_object123293:head# 2024-04-20T17:27:45.123 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8d0fc49:::benchmark_data_smithi171_122772_object89881:head# 2024-04-20T17:27:45.124 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8d13250:::benchmark_data_smithi171_122772_object83518:head# 2024-04-20T17:27:45.124 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8d15e0e:::benchmark_data_smithi171_122772_object129767:head# 2024-04-20T17:27:45.125 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8d2bf50:::benchmark_data_smithi171_122772_object110374:head# 2024-04-20T17:27:45.125 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8d2d99b:::benchmark_data_smithi171_122772_object150886:head# 2024-04-20T17:27:45.126 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8d4cdf0:::benchmark_data_smithi171_122772_object92804:head# 2024-04-20T17:27:45.126 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8d4e2f0:::benchmark_data_smithi171_122772_object89429:head# 2024-04-20T17:27:45.127 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8d5f19e:::benchmark_data_smithi171_122772_object168263:head# 2024-04-20T17:27:45.128 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8d6e628:::benchmark_data_smithi171_122772_object95808:head# 2024-04-20T17:27:45.128 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8d6ee62:::benchmark_data_smithi171_122772_object140317:head# 2024-04-20T17:27:45.129 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8d85f56:::benchmark_data_smithi171_122772_object107465:head# 2024-04-20T17:27:45.129 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8d8a65f:::benchmark_data_smithi171_122772_object166392:head# 2024-04-20T17:27:45.130 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8d8bff8:::benchmark_data_smithi171_122772_object151972:head# 2024-04-20T17:27:45.130 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8d9101e:::benchmark_data_smithi171_122772_object102881:head# 2024-04-20T17:27:45.131 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8d91052:::benchmark_data_smithi171_122772_object166294:head# 2024-04-20T17:27:45.131 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8d91b99:::benchmark_data_smithi171_122772_object110644:head# 2024-04-20T17:27:45.132 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8d9a159:::benchmark_data_smithi171_122772_object86261:head# 2024-04-20T17:27:45.132 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8d9dccd:::benchmark_data_smithi171_122772_object133842:head# 2024-04-20T17:27:45.133 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8da21a1:::benchmark_data_smithi171_122772_object98526:head# 2024-04-20T17:27:45.133 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8dab1a8:::benchmark_data_smithi171_122772_object135436:head# 2024-04-20T17:27:45.134 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8dbe8be:::benchmark_data_smithi171_122772_object98672:head# 2024-04-20T17:27:45.134 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8dc676d:::benchmark_data_smithi171_122772_object113371:head# 2024-04-20T17:27:45.135 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8dca0b8:::benchmark_data_smithi171_122772_object145922:head# 2024-04-20T17:27:45.135 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8dcdf41:::benchmark_data_smithi171_122772_object108047:head# 2024-04-20T17:27:45.136 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8dde754:::benchmark_data_smithi171_122772_object107033:head# 2024-04-20T17:27:45.136 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8de5c9b:::benchmark_data_smithi171_122772_object98833:head# 2024-04-20T17:27:45.136 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8df4c52:::benchmark_data_smithi171_122772_object97929:head# 2024-04-20T17:27:45.137 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8df8404:::benchmark_data_smithi171_122772_object130406:head# 2024-04-20T17:27:45.138 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8df8e0d:::benchmark_data_smithi171_122772_object156880:head# 2024-04-20T17:27:45.138 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8e32917:::benchmark_data_smithi171_122772_object109954:head# 2024-04-20T17:27:45.139 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8e4650d:::benchmark_data_smithi171_122772_object112378:head# 2024-04-20T17:27:45.139 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8e4d238:::benchmark_data_smithi171_122772_object96954:head# 2024-04-20T17:27:45.139 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8e6c7d7:::benchmark_data_smithi171_122772_object96231:head# 2024-04-20T17:27:45.140 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8e6f1be:::benchmark_data_smithi171_122772_object153657:head# 2024-04-20T17:27:45.140 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8e8941d:::benchmark_data_smithi171_122772_object153872:head# 2024-04-20T17:27:45.141 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8e9460a:::benchmark_data_smithi171_122772_object114771:head# 2024-04-20T17:27:45.141 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8eadd9c:::benchmark_data_smithi171_122772_object79441:head# 2024-04-20T17:27:45.142 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8eb5321:::benchmark_data_smithi171_122772_object103473:head# 2024-04-20T17:27:45.142 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8ec2c4a:::benchmark_data_smithi171_122772_object151925:head# 2024-04-20T17:27:45.143 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8eda2b3:::benchmark_data_smithi171_122772_object146246:head# 2024-04-20T17:27:45.143 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8eda328:::benchmark_data_smithi171_122772_object162608:head# 2024-04-20T17:27:45.144 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8edf7d5:::benchmark_data_smithi171_122772_object134952:head# 2024-04-20T17:27:45.144 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8ef8766:::benchmark_data_smithi171_122772_object138992:head# 2024-04-20T17:27:45.145 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8f14fdf:::benchmark_data_smithi171_122772_object90440:head# 2024-04-20T17:27:45.146 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8f18b0e:::benchmark_data_smithi171_122772_object124009:head# 2024-04-20T17:27:45.146 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8f2765a:::benchmark_data_smithi171_122772_object88230:head# 2024-04-20T17:27:45.147 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8f2eb5c:::benchmark_data_smithi171_122772_object124114:head# 2024-04-20T17:27:45.147 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8f3650a:::benchmark_data_smithi171_122772_object132867:head# 2024-04-20T17:27:45.147 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8f50753:::benchmark_data_smithi171_122772_object100253:head# 2024-04-20T17:27:45.148 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8f53072:::benchmark_data_smithi171_122772_object111445:head# 2024-04-20T17:27:45.148 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8f61dc2:::benchmark_data_smithi171_122772_object94319:head# 2024-04-20T17:27:45.149 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8f6d66f:::benchmark_data_smithi171_122772_object101194:head# 2024-04-20T17:27:45.149 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8f6dcc3:::benchmark_data_smithi171_122772_object83866:head# 2024-04-20T17:27:45.149 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8f71e9a:::benchmark_data_smithi171_122772_object155883:head# 2024-04-20T17:27:45.150 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8f7aa84:::benchmark_data_smithi171_122772_object133051:head# 2024-04-20T17:27:45.150 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8f81597:::benchmark_data_smithi171_122772_object164611:head# 2024-04-20T17:27:45.151 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8f87c87:::benchmark_data_smithi171_122772_object109695:head# 2024-04-20T17:27:45.151 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8f97bd7:::benchmark_data_smithi171_122772_object145245:head# 2024-04-20T17:27:45.152 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8fb2ab8:::benchmark_data_smithi171_122772_object98398:head# 2024-04-20T17:27:45.152 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8fb444d:::benchmark_data_smithi171_122772_object87375:head# 2024-04-20T17:27:45.153 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8fda48c:::benchmark_data_smithi171_122772_object158578:head# 2024-04-20T17:27:45.153 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8fddf8f:::benchmark_data_smithi171_122772_object149201:head# 2024-04-20T17:27:45.154 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8fdfc95:::benchmark_data_smithi171_122772_object132172:head# 2024-04-20T17:27:45.154 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8feb67d:::benchmark_data_smithi171_122772_object92030:head# 2024-04-20T17:27:45.155 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8feee3f:::benchmark_data_smithi171_122772_object127919:head# 2024-04-20T17:27:45.156 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8fefdd0:::benchmark_data_smithi171_122772_object86436:head# 2024-04-20T17:27:45.156 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a8ff828f:::benchmark_data_smithi171_122772_object94290:head# 2024-04-20T17:27:45.156 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9000097:::benchmark_data_smithi171_122772_object115713:head# 2024-04-20T17:27:45.157 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9001b69:::benchmark_data_smithi171_122772_object82650:head# 2024-04-20T17:27:45.157 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9011cf0:::benchmark_data_smithi171_122772_object139114:head# 2024-04-20T17:27:45.158 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9013fb3:::benchmark_data_smithi171_122772_object120676:head# 2024-04-20T17:27:45.158 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9020e08:::benchmark_data_smithi171_122772_object165130:head# 2024-04-20T17:27:45.159 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a902e042:::benchmark_data_smithi171_122772_object92404:head# 2024-04-20T17:27:45.159 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a903575a:::benchmark_data_smithi171_122772_object110352:head# 2024-04-20T17:27:45.160 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a90371c0:::benchmark_data_smithi171_122772_object78933:head# 2024-04-20T17:27:45.160 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9047094:::benchmark_data_smithi171_122772_object158270:head# 2024-04-20T17:27:45.161 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a90479e0:::benchmark_data_smithi171_122772_object169681:head# 2024-04-20T17:27:45.161 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9048821:::benchmark_data_smithi171_122772_object154297:head# 2024-04-20T17:27:45.162 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9050d26:::benchmark_data_smithi171_122772_object94273:head# 2024-04-20T17:27:45.162 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a905cdd0:::benchmark_data_smithi171_122772_object94564:head# 2024-04-20T17:27:45.163 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9079ba4:::benchmark_data_smithi171_122772_object123843:head# 2024-04-20T17:27:45.163 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a907a14b:::benchmark_data_smithi171_122772_object161998:head# 2024-04-20T17:27:45.164 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a908038b:::benchmark_data_smithi171_122772_object92271:head# 2024-04-20T17:27:45.164 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a908283f:::benchmark_data_smithi171_122772_object105395:head# 2024-04-20T17:27:45.164 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9086608:::benchmark_data_smithi171_122772_object113117:head# 2024-04-20T17:27:45.165 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a908a89a:::benchmark_data_smithi171_122772_object126600:head# 2024-04-20T17:27:45.166 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9091445:::benchmark_data_smithi171_122772_object93743:head# 2024-04-20T17:27:45.166 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a90b2da0:::benchmark_data_smithi171_122772_object80068:head# 2024-04-20T17:27:45.167 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a90bb955:::benchmark_data_smithi171_122772_object149896:head# 2024-04-20T17:27:45.167 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a90d3175:::benchmark_data_smithi171_122772_object135004:head# 2024-04-20T17:27:45.168 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a90e9a3a:::benchmark_data_smithi171_122772_object156806:head# 2024-04-20T17:27:45.168 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a910a3bf:::benchmark_data_smithi171_122772_object148714:head# 2024-04-20T17:27:45.169 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a911878d:::benchmark_data_smithi171_122772_object168837:head# 2024-04-20T17:27:45.169 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a913aad3:::benchmark_data_smithi171_122772_object108522:head# 2024-04-20T17:27:45.170 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a91401e0:::benchmark_data_smithi171_122772_object108829:head# 2024-04-20T17:27:45.170 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9141a1e:::benchmark_data_smithi171_122772_object116721:head# 2024-04-20T17:27:45.171 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a91465c1:::benchmark_data_smithi171_122772_object82764:head# 2024-04-20T17:27:45.171 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9147b1c:::benchmark_data_smithi171_122772_object154845:head# 2024-04-20T17:27:45.172 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9149b06:::benchmark_data_smithi171_122772_object167944:head# 2024-04-20T17:27:45.172 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a915363f:::benchmark_data_smithi171_122772_object152620:head# 2024-04-20T17:27:45.173 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a91543d4:::benchmark_data_smithi171_122772_object155751:head# 2024-04-20T17:27:45.173 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a915fe57:::benchmark_data_smithi171_122772_object164750:head# 2024-04-20T17:27:45.174 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9167872:::benchmark_data_smithi171_122772_object141790:head# 2024-04-20T17:27:45.174 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a917c1be:::benchmark_data_smithi171_122772_object84658:head# 2024-04-20T17:27:45.174 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a917d0be:::benchmark_data_smithi171_122772_object122380:head# 2024-04-20T17:27:45.175 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9186c87:::benchmark_data_smithi171_122772_object160358:head# 2024-04-20T17:27:45.175 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a918dbc1:::benchmark_data_smithi171_122772_object143471:head# 2024-04-20T17:27:45.176 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9198779:::benchmark_data_smithi171_122772_object139255:head# 2024-04-20T17:27:45.176 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a919ff35:::benchmark_data_smithi171_122772_object160651:head# 2024-04-20T17:27:45.177 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a91bb8fe:::benchmark_data_smithi171_122772_object126047:head# 2024-04-20T17:27:45.177 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a91c4efd:::benchmark_data_smithi171_122772_object100083:head# 2024-04-20T17:27:45.178 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a91d4a50:::benchmark_data_smithi171_122772_object99351:head# 2024-04-20T17:27:45.178 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a91dbeb7:::benchmark_data_smithi171_122772_object131447:head# 2024-04-20T17:27:45.179 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a91e20e7:::benchmark_data_smithi171_122772_object146639:head# 2024-04-20T17:27:45.179 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a91ee5b9:::benchmark_data_smithi171_122772_object134946:head# 2024-04-20T17:27:45.180 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a91f0ae6:::benchmark_data_smithi171_122772_object111662:head# 2024-04-20T17:27:45.180 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a91f65bc:::benchmark_data_smithi171_122772_object86886:head# 2024-04-20T17:27:45.181 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a92120cf:::benchmark_data_smithi171_122772_object142916:head# 2024-04-20T17:27:45.181 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a92174a2:::benchmark_data_smithi171_122772_object141825:head# 2024-04-20T17:27:45.182 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9221816:::benchmark_data_smithi171_122772_object151486:head# 2024-04-20T17:27:45.182 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9224a93:::benchmark_data_smithi171_122772_object141128:head# 2024-04-20T17:27:45.183 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a922e45f:::benchmark_data_smithi171_122772_object79821:head# 2024-04-20T17:27:45.184 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a926c008:::benchmark_data_smithi171_122772_object155036:head# 2024-04-20T17:27:45.184 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9275ae7:::benchmark_data_smithi171_122772_object164438:head# 2024-04-20T17:27:45.184 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a92799b7:::benchmark_data_smithi171_122772_object159853:head# 2024-04-20T17:27:45.185 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a927b67e:::benchmark_data_smithi171_122772_object144058:head# 2024-04-20T17:27:45.186 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9285cec:::benchmark_data_smithi171_122772_object141761:head# 2024-04-20T17:27:45.186 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9286b8b:::benchmark_data_smithi171_122772_object135010:head# 2024-04-20T17:27:45.187 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a928a667:::benchmark_data_smithi171_122772_object106601:head# 2024-04-20T17:27:45.187 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a92af8c6:::benchmark_data_smithi171_122772_object94763:head# 2024-04-20T17:27:45.187 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a92b9761:::benchmark_data_smithi171_122772_object136706:head# 2024-04-20T17:27:45.188 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a92c3f6b:::benchmark_data_smithi171_122772_object157269:head# 2024-04-20T17:27:45.188 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a92cd5d2:::benchmark_data_smithi171_122772_object144095:head# 2024-04-20T17:27:45.189 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a92e2810:::benchmark_data_smithi171_122772_object166516:head# 2024-04-20T17:27:45.190 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a92e3fcc:::benchmark_data_smithi171_122772_object110968:head# 2024-04-20T17:27:45.191 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a930a692:::benchmark_data_smithi171_122772_object120488:head# 2024-04-20T17:27:45.191 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9321934:::benchmark_data_smithi171_122772_object110118:head# 2024-04-20T17:27:45.192 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9332220:::benchmark_data_smithi171_122772_object138044:head# 2024-04-20T17:27:45.193 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a93377a4:::benchmark_data_smithi171_122772_object150273:head# 2024-04-20T17:27:45.193 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9342ba8:::benchmark_data_smithi171_122772_object159622:head# 2024-04-20T17:27:45.194 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a934627d:::benchmark_data_smithi171_122772_object121533:head# 2024-04-20T17:27:45.194 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a93606bb:::benchmark_data_smithi171_122772_object86734:head# 2024-04-20T17:27:45.195 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a936ec53:::benchmark_data_smithi171_122772_object132636:head# 2024-04-20T17:27:45.195 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a937325a:::benchmark_data_smithi171_122772_object114274:head# 2024-04-20T17:27:45.195 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9381355:::benchmark_data_smithi171_122772_object109093:head# 2024-04-20T17:27:45.196 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9386da2:::benchmark_data_smithi171_122772_object133027:head# 2024-04-20T17:27:45.196 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a938e3d1:::benchmark_data_smithi171_122772_object143018:head# 2024-04-20T17:27:45.197 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a939cb97:::benchmark_data_smithi171_122772_object121397:head# 2024-04-20T17:27:45.197 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a93a026c:::benchmark_data_smithi171_122772_object132536:head# 2024-04-20T17:27:45.198 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a93ae99a:::benchmark_data_smithi171_122772_object159645:head# 2024-04-20T17:27:45.198 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a93be36a:::benchmark_data_smithi171_122772_object122354:head# 2024-04-20T17:27:45.199 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a93cd34f:::benchmark_data_smithi171_122772_object137272:head# 2024-04-20T17:27:45.199 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a93d5190:::benchmark_data_smithi171_122772_object131010:head# 2024-04-20T17:27:45.200 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a93d58a3:::benchmark_data_smithi171_122772_object150201:head# 2024-04-20T17:27:45.200 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a93ec20e:::benchmark_data_smithi171_122772_object138139:head# 2024-04-20T17:27:45.201 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a93ec6b5:::benchmark_data_smithi171_122772_object119189:head# 2024-04-20T17:27:45.201 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a93ee059:::benchmark_data_smithi171_122772_object154832:head# 2024-04-20T17:27:45.202 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a93ff242:::benchmark_data_smithi171_122772_object137974:head# 2024-04-20T17:27:45.202 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a940362a:::benchmark_data_smithi171_122772_object146729:head# 2024-04-20T17:27:45.203 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a940e22f:::benchmark_data_smithi171_122772_object150949:head# 2024-04-20T17:27:45.203 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a940ea54:::benchmark_data_smithi171_122772_object132060:head# 2024-04-20T17:27:45.204 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a940f211:::benchmark_data_smithi171_122772_object115025:head# 2024-04-20T17:27:45.204 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9427a2e:::benchmark_data_smithi171_122772_object132438:head# 2024-04-20T17:27:45.205 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9435143:::benchmark_data_smithi171_122772_object108971:head# 2024-04-20T17:27:45.205 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a943e251:::benchmark_data_smithi171_122772_object104658:head# 2024-04-20T17:27:45.206 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a94514ef:::benchmark_data_smithi171_122772_object79195:head# 2024-04-20T17:27:45.206 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a945823c:::benchmark_data_smithi171_122772_object164105:head# 2024-04-20T17:27:45.207 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a947074f:::benchmark_data_smithi171_122772_object111513:head# 2024-04-20T17:27:45.207 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9487f09:::benchmark_data_smithi171_122772_object161326:head# 2024-04-20T17:27:45.208 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a948a696:::benchmark_data_smithi171_122772_object97284:head# 2024-04-20T17:27:45.208 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a948c143:::benchmark_data_smithi171_122772_object158224:head# 2024-04-20T17:27:45.209 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a949591f:::benchmark_data_smithi171_122772_object127713:head# 2024-04-20T17:27:45.210 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a949ccce:::benchmark_data_smithi171_122772_object157737:head# 2024-04-20T17:27:45.211 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a94b0790:::benchmark_data_smithi171_122772_object150408:head# 2024-04-20T17:27:45.211 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a94b74f4:::benchmark_data_smithi171_122772_object106256:head# 2024-04-20T17:27:45.211 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a94e3769:::benchmark_data_smithi171_122772_object149270:head# 2024-04-20T17:27:45.211 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a94e9b62:::benchmark_data_smithi171_122772_object141599:head# 2024-04-20T17:27:45.211 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a94ea0a2:::benchmark_data_smithi171_122772_object80452:head# 2024-04-20T17:27:45.212 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a94ea261:::benchmark_data_smithi171_122772_object95621:head# 2024-04-20T17:27:45.212 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a94eedb4:::benchmark_data_smithi171_122772_object122776:head# 2024-04-20T17:27:45.213 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a94f7044:::benchmark_data_smithi171_122772_object158184:head# 2024-04-20T17:27:45.213 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a951254a:::benchmark_data_smithi171_122772_object132420:head# 2024-04-20T17:27:45.214 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9513dbf:::benchmark_data_smithi171_122772_object125609:head# 2024-04-20T17:27:45.214 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9524e1b:::benchmark_data_smithi171_122772_object107945:head# 2024-04-20T17:27:45.214 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9528ef6:::benchmark_data_smithi171_122772_object119671:head# 2024-04-20T17:27:45.215 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9529e65:::benchmark_data_smithi171_122772_object157603:head# 2024-04-20T17:27:45.215 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a952cdf8:::benchmark_data_smithi171_122772_object170044:head# 2024-04-20T17:27:45.216 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a95329c8:::benchmark_data_smithi171_122772_object135994:head# 2024-04-20T17:27:45.217 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9537ef6:::benchmark_data_smithi171_122772_object139362:head# 2024-04-20T17:27:45.217 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a953a3ff:::benchmark_data_smithi171_122772_object84189:head# 2024-04-20T17:27:45.217 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a953de8b:::benchmark_data_smithi171_122772_object169323:head# 2024-04-20T17:27:45.218 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9540d80:::benchmark_data_smithi171_122772_object127325:head# 2024-04-20T17:27:45.218 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a95681f3:::benchmark_data_smithi171_122772_object132815:head# 2024-04-20T17:27:45.218 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9578156:::benchmark_data_smithi171_122772_object99752:head# 2024-04-20T17:27:45.219 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9578f95:::benchmark_data_smithi171_122772_object143831:head# 2024-04-20T17:27:45.219 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9596032:::benchmark_data_smithi171_122772_object145967:head# 2024-04-20T17:27:45.220 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9597ab8:::benchmark_data_smithi171_122772_object84107:head# 2024-04-20T17:27:45.220 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a959aead:::benchmark_data_smithi171_122772_object156777:head# 2024-04-20T17:27:45.221 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a95a10b2:::benchmark_data_smithi171_122772_object115936:head# 2024-04-20T17:27:45.222 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a95bba40:::benchmark_data_smithi171_122772_object122708:head# 2024-04-20T17:27:45.222 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a95c2b12:::benchmark_data_smithi171_122772_object118551:head# 2024-04-20T17:27:45.222 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a95cb667:::benchmark_data_smithi171_122772_object164993:head# 2024-04-20T17:27:45.223 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a95da63e:::benchmark_data_smithi171_122772_object97988:head# 2024-04-20T17:27:45.223 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a95db821:::benchmark_data_smithi171_122772_object86100:head# 2024-04-20T17:27:45.224 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a95e0dcd:::benchmark_data_smithi171_122772_object131175:head# 2024-04-20T17:27:45.224 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a95e7607:::benchmark_data_smithi171_122772_object109311:head# 2024-04-20T17:27:45.225 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a960ad2e:::benchmark_data_smithi171_122772_object136872:head# 2024-04-20T17:27:45.225 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9611e70:::benchmark_data_smithi171_122772_object115684:head# 2024-04-20T17:27:45.226 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9614085:::benchmark_data_smithi171_122772_object148749:head# 2024-04-20T17:27:45.226 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9617232:::benchmark_data_smithi171_122772_object166759:head# 2024-04-20T17:27:45.227 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a961d1e4:::benchmark_data_smithi171_122772_object147491:head# 2024-04-20T17:27:45.227 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9649cd8:::benchmark_data_smithi171_122772_object134503:head# 2024-04-20T17:27:45.227 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9650de7:::benchmark_data_smithi171_122772_object160199:head# 2024-04-20T17:27:45.228 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9658e2e:::benchmark_data_smithi171_122772_object92242:head# 2024-04-20T17:27:45.229 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a965c020:::benchmark_data_smithi171_122772_object123213:head# 2024-04-20T17:27:45.229 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a966483c:::benchmark_data_smithi171_122772_object127499:head# 2024-04-20T17:27:45.230 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a966aabf:::benchmark_data_smithi171_122772_object139530:head# 2024-04-20T17:27:45.230 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9686733:::benchmark_data_smithi171_122772_object92854:head# 2024-04-20T17:27:45.231 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9688fe8:::benchmark_data_smithi171_122772_object79276:head# 2024-04-20T17:27:45.232 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a968fe99:::benchmark_data_smithi171_122772_object102681:head# 2024-04-20T17:27:45.232 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9698a80:::benchmark_data_smithi171_122772_object121452:head# 2024-04-20T17:27:45.232 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a96a3e9d:::benchmark_data_smithi171_122772_object129545:head# 2024-04-20T17:27:45.233 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a96a7c3f:::benchmark_data_smithi171_122772_object87668:head# 2024-04-20T17:27:45.233 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a96ac024:::benchmark_data_smithi171_122772_object98925:head# 2024-04-20T17:27:45.234 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a96aea4e:::benchmark_data_smithi171_122772_object138042:head# 2024-04-20T17:27:45.234 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a96c02a7:::benchmark_data_smithi171_122772_object132087:head# 2024-04-20T17:27:45.235 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a96c4ce4:::benchmark_data_smithi171_122772_object85663:head# 2024-04-20T17:27:45.235 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a96c5a5c:::benchmark_data_smithi171_122772_object159536:head# 2024-04-20T17:27:45.236 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a96c7773:::benchmark_data_smithi171_122772_object87423:head# 2024-04-20T17:27:45.236 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a96ea941:::benchmark_data_smithi171_122772_object111159:head# 2024-04-20T17:27:45.237 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a96eff83:::benchmark_data_smithi171_122772_object139074:head# 2024-04-20T17:27:45.238 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a96f2fae:::benchmark_data_smithi171_122772_object105020:head# 2024-04-20T17:27:45.238 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a96f3b36:::benchmark_data_smithi171_122772_object79497:head# 2024-04-20T17:27:45.238 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9709a1d:::benchmark_data_smithi171_122772_object112237:head# 2024-04-20T17:27:45.239 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a970ec0b:::benchmark_data_smithi171_122772_object107662:head# 2024-04-20T17:27:45.239 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9717b47:::benchmark_data_smithi171_122772_object169077:head# 2024-04-20T17:27:45.240 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9724fb5:::benchmark_data_smithi171_122772_object143336:head# 2024-04-20T17:27:45.241 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a972adc2:::benchmark_data_smithi171_122772_object117575:head# 2024-04-20T17:27:45.241 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a97327c8:::benchmark_data_smithi171_122772_object79080:head# 2024-04-20T17:27:45.241 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a975313e:::benchmark_data_smithi171_122772_object127091:head# 2024-04-20T17:27:45.242 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a976da03:::benchmark_data_smithi171_122772_object137398:head# 2024-04-20T17:27:45.242 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9777b4d:::benchmark_data_smithi171_122772_object78530:head# 2024-04-20T17:27:45.243 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a977ce0d:::benchmark_data_smithi171_122772_object112974:head# 2024-04-20T17:27:45.244 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a97898eb:::benchmark_data_smithi171_122772_object78575:head# 2024-04-20T17:27:45.244 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a979815f:::benchmark_data_smithi171_122772_object157620:head# 2024-04-20T17:27:45.244 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a97b1574:::benchmark_data_smithi171_122772_object92028:head# 2024-04-20T17:27:45.245 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a97bbfee:::benchmark_data_smithi171_122772_object153887:head# 2024-04-20T17:27:45.245 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a97cd7f8:::benchmark_data_smithi171_122772_object114972:head# 2024-04-20T17:27:45.245 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a97da6ea:::benchmark_data_smithi171_122772_object99285:head# 2024-04-20T17:27:45.246 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a97fa459:::benchmark_data_smithi171_122772_object141161:head# 2024-04-20T17:27:45.247 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a97fe340:::benchmark_data_smithi171_122772_object108966:head# 2024-04-20T17:27:45.247 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a980ae11:::benchmark_data_smithi171_122772_object168545:head# 2024-04-20T17:27:45.248 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a980c803:::benchmark_data_smithi171_122772_object160038:head# 2024-04-20T17:27:45.249 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9817b62:::benchmark_data_smithi171_122772_object79571:head# 2024-04-20T17:27:45.249 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9841bd4:::benchmark_data_smithi171_122772_object137834:head# 2024-04-20T17:27:45.250 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9841cd7:::benchmark_data_smithi171_122772_object154098:head# 2024-04-20T17:27:45.250 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a98462f3:::benchmark_data_smithi171_122772_object148848:head# 2024-04-20T17:27:45.250 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9871362:::benchmark_data_smithi171_122772_object86521:head# 2024-04-20T17:27:45.250 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9888a5d:::benchmark_data_smithi171_122772_object125323:head# 2024-04-20T17:27:45.251 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9889c3b:::benchmark_data_smithi171_122772_object136261:head# 2024-04-20T17:27:45.251 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a989c1a5:::benchmark_data_smithi171_122772_object94265:head# 2024-04-20T17:27:45.252 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a98abe68:::benchmark_data_smithi171_122772_object115650:head# 2024-04-20T17:27:45.252 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a98bb79b:::benchmark_data_smithi171_122772_object110887:head# 2024-04-20T17:27:45.253 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a98bba28:::benchmark_data_smithi171_122772_object121522:head# 2024-04-20T17:27:45.253 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a98c21cc:::benchmark_data_smithi171_122772_object105443:head# 2024-04-20T17:27:45.254 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a98e4d92:::benchmark_data_smithi171_122772_object130316:head# 2024-04-20T17:27:45.255 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a98f5d74:::benchmark_data_smithi171_122772_object92046:head# 2024-04-20T17:27:45.255 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a98f9583:::benchmark_data_smithi171_122772_object149300:head# 2024-04-20T17:27:45.256 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9907bcc:::benchmark_data_smithi171_122772_object112212:head# 2024-04-20T17:27:45.256 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9909a9d:::benchmark_data_smithi171_122772_object165975:head# 2024-04-20T17:27:45.256 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a991d33b:::benchmark_data_smithi171_122772_object91562:head# 2024-04-20T17:27:45.257 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a99374ef:::benchmark_data_smithi171_122772_object97064:head# 2024-04-20T17:27:45.257 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9938a10:::benchmark_data_smithi171_122772_object160738:head# 2024-04-20T17:27:45.258 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a993b038:::benchmark_data_smithi171_122772_object146070:head# 2024-04-20T17:27:45.258 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a993e2d7:::benchmark_data_smithi171_122772_object108202:head# 2024-04-20T17:27:45.259 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a995cfa0:::benchmark_data_smithi171_122772_object159093:head# 2024-04-20T17:27:45.259 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a995f248:::benchmark_data_smithi171_122772_object164768:head# 2024-04-20T17:27:45.260 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9963bb8:::benchmark_data_smithi171_122772_object86272:head# 2024-04-20T17:27:45.260 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a996d501:::benchmark_data_smithi171_122772_object155201:head# 2024-04-20T17:27:45.261 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a99aff1d:::benchmark_data_smithi171_122772_object131573:head# 2024-04-20T17:27:45.261 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a99b38bf:::benchmark_data_smithi171_122772_object146412:head# 2024-04-20T17:27:45.262 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a99bcb01:::benchmark_data_smithi171_122772_object78157:head# 2024-04-20T17:27:45.262 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a99d91f9:::benchmark_data_smithi171_122772_object155827:head# 2024-04-20T17:27:45.263 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a99f82b5:::benchmark_data_smithi171_122772_object144193:head# 2024-04-20T17:27:45.263 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a99ff5cf:::benchmark_data_smithi171_122772_object144402:head# 2024-04-20T17:27:45.264 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9a04e6f:::benchmark_data_smithi171_122772_object110196:head# 2024-04-20T17:27:45.264 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9a21119:::benchmark_data_smithi171_122772_object134620:head# 2024-04-20T17:27:45.264 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9a442f3:::benchmark_data_smithi171_122772_object152152:head# 2024-04-20T17:27:45.265 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9a65a1a:::benchmark_data_smithi171_122772_object90482:head# 2024-04-20T17:27:45.265 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9a8aff2:::benchmark_data_smithi171_122772_object126943:head# 2024-04-20T17:27:45.266 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9aa1022:::benchmark_data_smithi171_122772_object161119:head# 2024-04-20T17:27:45.266 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9abb11f:::benchmark_data_smithi171_122772_object130911:head# 2024-04-20T17:27:45.267 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9ada06d:::benchmark_data_smithi171_122772_object109098:head# 2024-04-20T17:27:45.267 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9adec7b:::benchmark_data_smithi171_122772_object119243:head# 2024-04-20T17:27:45.268 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9afcfda:::benchmark_data_smithi171_122772_object113134:head# 2024-04-20T17:27:45.268 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9b0fd6c:::benchmark_data_smithi171_122772_object161716:head# 2024-04-20T17:27:45.269 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9b16756:::benchmark_data_smithi171_122772_object116591:head# 2024-04-20T17:27:45.269 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9b249fc:::benchmark_data_smithi171_122772_object96443:head# 2024-04-20T17:27:45.270 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9b2bdb4:::benchmark_data_smithi171_122772_object135780:head# 2024-04-20T17:27:45.270 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9b40a64:::benchmark_data_smithi171_122772_object96035:head# 2024-04-20T17:27:45.271 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9b41c06:::benchmark_data_smithi171_122772_object146621:head# 2024-04-20T17:27:45.272 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9b4223d:::benchmark_data_smithi171_122772_object90263:head# 2024-04-20T17:27:45.273 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9b5166c:::benchmark_data_smithi171_122772_object120472:head# 2024-04-20T17:27:45.273 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9b58ec9:::benchmark_data_smithi171_122772_object117034:head# 2024-04-20T17:27:45.273 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9b590cc:::benchmark_data_smithi171_122772_object124914:head# 2024-04-20T17:27:45.273 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9b5c92f:::benchmark_data_smithi171_122772_object104958:head# 2024-04-20T17:27:45.275 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9b638d7:::benchmark_data_smithi171_122772_object169895:head# 2024-04-20T17:27:45.275 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9b7e707:::benchmark_data_smithi171_122772_object130579:head# 2024-04-20T17:27:45.275 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9b8616e:::benchmark_data_smithi171_122772_object163126:head# 2024-04-20T17:27:45.275 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9b87df9:::benchmark_data_smithi171_122772_object151446:head# 2024-04-20T17:27:45.275 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9b8da28:::benchmark_data_smithi171_122772_object135518:head# 2024-04-20T17:27:45.276 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9b9163b:::benchmark_data_smithi171_122772_object89638:head# 2024-04-20T17:27:45.276 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9ba33c4:::benchmark_data_smithi171_122772_object149686:head# 2024-04-20T17:27:45.277 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9ba91b8:::benchmark_data_smithi171_122772_object125390:head# 2024-04-20T17:27:45.278 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9bb3004:::benchmark_data_smithi171_122772_object168083:head# 2024-04-20T17:27:45.278 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9bc5477:::benchmark_data_smithi171_122772_object107373:head# 2024-04-20T17:27:45.279 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9bc8268:::benchmark_data_smithi171_122772_object113708:head# 2024-04-20T17:27:45.279 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9bc8b0a:::benchmark_data_smithi171_122772_object118486:head# 2024-04-20T17:27:45.279 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9bcdb29:::benchmark_data_smithi171_122772_object147220:head# 2024-04-20T17:27:45.280 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9bd14c5:::benchmark_data_smithi171_122772_object117176:head# 2024-04-20T17:27:45.280 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9be0989:::benchmark_data_smithi171_122772_object167844:head# 2024-04-20T17:27:45.282 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9be9c24:::benchmark_data_smithi171_122772_object101109:head# 2024-04-20T17:27:45.282 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9bec3e2:::benchmark_data_smithi171_122772_object126568:head# 2024-04-20T17:27:45.282 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9c0e7c2:::benchmark_data_smithi171_122772_object163155:head# 2024-04-20T17:27:45.283 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9c10212:::benchmark_data_smithi171_122772_object165258:head# 2024-04-20T17:27:45.283 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9c20e01:::benchmark_data_smithi171_122772_object115726:head# 2024-04-20T17:27:45.283 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9c33389:::benchmark_data_smithi171_122772_object140440:head# 2024-04-20T17:27:45.284 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9c353ee:::benchmark_data_smithi171_122772_object155898:head# 2024-04-20T17:27:45.284 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9c393c8:::benchmark_data_smithi171_122772_object122709:head# 2024-04-20T17:27:45.285 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9c53772:::benchmark_data_smithi171_122772_object88988:head# 2024-04-20T17:27:45.285 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9c568e3:::benchmark_data_smithi171_122772_object102402:head# 2024-04-20T17:27:45.285 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9c59a28:::benchmark_data_smithi171_122772_object110327:head# 2024-04-20T17:27:45.287 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9c6573c:::benchmark_data_smithi171_122772_object115608:head# 2024-04-20T17:27:45.287 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9c658c8:::benchmark_data_smithi171_122772_object165290:head# 2024-04-20T17:27:45.287 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9c84877:::benchmark_data_smithi171_122772_object160135:head# 2024-04-20T17:27:45.288 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9c9bdce:::benchmark_data_smithi171_122772_object96838:head# 2024-04-20T17:27:45.288 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9ca1946:::benchmark_data_smithi171_122772_object118251:head# 2024-04-20T17:27:45.288 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9ca8530:::benchmark_data_smithi171_122772_object106351:head# 2024-04-20T17:27:45.290 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9caaf77:::benchmark_data_smithi171_122772_object130162:head# 2024-04-20T17:27:45.290 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9cadf30:::benchmark_data_smithi171_122772_object126189:head# 2024-04-20T17:27:45.290 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9caf3e1:::benchmark_data_smithi171_122772_object159655:head# 2024-04-20T17:27:45.290 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9caf8be:::benchmark_data_smithi171_122772_object111285:head# 2024-04-20T17:27:45.291 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9cb6623:::benchmark_data_smithi171_122772_object87636:head# 2024-04-20T17:27:45.291 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9ccd0d6:::benchmark_data_smithi171_122772_object117380:head# 2024-04-20T17:27:45.291 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9cd30fd:::benchmark_data_smithi171_122772_object125844:head# 2024-04-20T17:27:45.292 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9cd8e82:::benchmark_data_smithi171_122772_object136152:head# 2024-04-20T17:27:45.292 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9cdff6b:::benchmark_data_smithi171_122772_object137128:head# 2024-04-20T17:27:45.293 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9ce4def:::benchmark_data_smithi171_122772_object96855:head# 2024-04-20T17:27:45.293 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9cf210f:::benchmark_data_smithi171_122772_object142330:head# 2024-04-20T17:27:45.294 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9d08402:::benchmark_data_smithi171_122772_object137598:head# 2024-04-20T17:27:45.295 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9d15a92:::benchmark_data_smithi171_122772_object129765:head# 2024-04-20T17:27:45.295 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9d179d4:::benchmark_data_smithi171_122772_object157163:head# 2024-04-20T17:27:45.296 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9d1a5b9:::benchmark_data_smithi171_122772_object154547:head# 2024-04-20T17:27:45.296 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9d1d11e:::benchmark_data_smithi171_122772_object105965:head# 2024-04-20T17:27:45.297 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9d26cff:::benchmark_data_smithi171_122772_object157202:head# 2024-04-20T17:27:45.297 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9d2f1a5:::benchmark_data_smithi171_122772_object141363:head# 2024-04-20T17:27:45.297 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9d3239d:::benchmark_data_smithi171_122772_object97288:head# 2024-04-20T17:27:45.298 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9d436b5:::benchmark_data_smithi171_122772_object95800:head# 2024-04-20T17:27:45.298 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9d50bac:::benchmark_data_smithi171_122772_object141282:head# 2024-04-20T17:27:45.299 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9d5d47c:::benchmark_data_smithi171_122772_object124826:head# 2024-04-20T17:27:45.299 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9d6ccbb:::benchmark_data_smithi171_122772_object133431:head# 2024-04-20T17:27:45.300 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9d81e46:::benchmark_data_smithi171_122772_object131442:head# 2024-04-20T17:27:45.301 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9d8ff68:::benchmark_data_smithi171_122772_object98351:head# 2024-04-20T17:27:45.301 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9d9268c:::benchmark_data_smithi171_122772_object95572:head# 2024-04-20T17:27:45.301 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9d9ae9f:::benchmark_data_smithi171_122772_object139349:head# 2024-04-20T17:27:45.301 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9d9e09e:::benchmark_data_smithi171_122772_object140696:head# 2024-04-20T17:27:45.302 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9dac594:::benchmark_data_smithi171_122772_object162210:head# 2024-04-20T17:27:45.303 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9db6e83:::benchmark_data_smithi171_122772_object80549:head# 2024-04-20T17:27:45.303 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9dbe7da:::benchmark_data_smithi171_122772_object90879:head# 2024-04-20T17:27:45.303 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9dd3409:::benchmark_data_smithi171_122772_object90485:head# 2024-04-20T17:27:45.304 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9dd521a:::benchmark_data_smithi171_122772_object101073:head# 2024-04-20T17:27:45.304 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9dd630c:::benchmark_data_smithi171_122772_object132553:head# 2024-04-20T17:27:45.305 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9dde5a1:::benchmark_data_smithi171_122772_object109568:head# 2024-04-20T17:27:45.305 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9de0643:::benchmark_data_smithi171_122772_object98489:head# 2024-04-20T17:27:45.306 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9de9090:::benchmark_data_smithi171_122772_object141987:head# 2024-04-20T17:27:45.307 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9df70ce:::benchmark_data_smithi171_122772_object112253:head# 2024-04-20T17:27:45.308 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9dfa9a6:::benchmark_data_smithi171_122772_object98006:head# 2024-04-20T17:27:45.308 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9e036b8:::benchmark_data_smithi171_122772_object133349:head# 2024-04-20T17:27:45.308 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9e04e6b:::benchmark_data_smithi171_122772_object169190:head# 2024-04-20T17:27:45.309 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9e09006:::benchmark_data_smithi171_122772_object85626:head# 2024-04-20T17:27:45.309 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9e0bf8b:::benchmark_data_smithi171_122772_object85101:head# 2024-04-20T17:27:45.310 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9e0fa5e:::benchmark_data_smithi171_122772_object97784:head# 2024-04-20T17:27:45.310 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9e1d3db:::benchmark_data_smithi171_122772_object110904:head# 2024-04-20T17:27:45.310 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9e28f0e:::benchmark_data_smithi171_122772_object118717:head# 2024-04-20T17:27:45.311 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9e2d148:::benchmark_data_smithi171_122772_object158629:head# 2024-04-20T17:27:45.311 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9e406c1:::benchmark_data_smithi171_122772_object112521:head# 2024-04-20T17:27:45.312 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9e42791:::benchmark_data_smithi171_122772_object136916:head# 2024-04-20T17:27:45.312 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9e443fe:::benchmark_data_smithi171_122772_object132954:head# 2024-04-20T17:27:45.313 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9e6676e:::benchmark_data_smithi171_122772_object156850:head# 2024-04-20T17:27:45.313 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9e68875:::benchmark_data_smithi171_122772_object117747:head# 2024-04-20T17:27:45.314 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9e835b6:::benchmark_data_smithi171_122772_object105167:head# 2024-04-20T17:27:45.316 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9e88b88:::benchmark_data_smithi171_122772_object157431:head# 2024-04-20T17:27:45.316 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9ea0d99:::benchmark_data_smithi171_122772_object96695:head# 2024-04-20T17:27:45.316 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9ea3b2a:::benchmark_data_smithi171_122772_object108774:head# 2024-04-20T17:27:45.316 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9ed7b41:::benchmark_data_smithi171_122772_object159988:head# 2024-04-20T17:27:45.317 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9eda8a9:::benchmark_data_smithi171_122772_object89680:head# 2024-04-20T17:27:45.317 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9ee18ce:::benchmark_data_smithi171_122772_object147714:head# 2024-04-20T17:27:45.317 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9eef6e3:::benchmark_data_smithi171_122772_object90541:head# 2024-04-20T17:27:45.318 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9efc0ee:::benchmark_data_smithi171_122772_object133122:head# 2024-04-20T17:27:45.319 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9f13cfc:::benchmark_data_smithi171_122772_object141796:head# 2024-04-20T17:27:45.319 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9f16db8:::benchmark_data_smithi171_122772_object155018:head# 2024-04-20T17:27:45.319 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9f2d220:::benchmark_data_smithi171_122772_object99381:head# 2024-04-20T17:27:45.320 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9f37401:::benchmark_data_smithi171_122772_object159272:head# 2024-04-20T17:27:45.320 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9f43302:::benchmark_data_smithi171_122772_object138398:head# 2024-04-20T17:27:45.320 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9f61cef:::benchmark_data_smithi171_122772_object157374:head# 2024-04-20T17:27:45.321 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9f691ea:::benchmark_data_smithi171_122772_object118956:head# 2024-04-20T17:27:45.321 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9f7dad9:::benchmark_data_smithi171_122772_object163979:head# 2024-04-20T17:27:45.322 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9f88f95:::benchmark_data_smithi171_122772_object103093:head# 2024-04-20T17:27:45.323 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9f90f5c:::benchmark_data_smithi171_122772_object131412:head# 2024-04-20T17:27:45.323 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9f94ca1:::benchmark_data_smithi171_122772_object90326:head# 2024-04-20T17:27:45.324 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9fa1869:::benchmark_data_smithi171_122772_object158138:head# 2024-04-20T17:27:45.325 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9fb866f:::benchmark_data_smithi171_122772_object80775:head# 2024-04-20T17:27:45.325 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9fc1d5c:::benchmark_data_smithi171_122772_object124739:head# 2024-04-20T17:27:45.325 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9fccdf7:::benchmark_data_smithi171_122772_object150841:head# 2024-04-20T17:27:45.326 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9fd31d6:::benchmark_data_smithi171_122772_object129706:head# 2024-04-20T17:27:45.327 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9fd9cdf:::benchmark_data_smithi171_122772_object157561:head# 2024-04-20T17:27:45.327 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:a9fe9f4f:::benchmark_data_smithi171_122772_object153788:head# 2024-04-20T17:27:45.328 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa000f50:::benchmark_data_smithi171_122772_object163307:head# 2024-04-20T17:27:45.329 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa00a32b:::benchmark_data_smithi171_122772_object79900:head# 2024-04-20T17:27:45.329 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa00bb63:::benchmark_data_smithi171_122772_object98167:head# 2024-04-20T17:27:45.329 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa02e0ad:::benchmark_data_smithi171_122772_object99488:head# 2024-04-20T17:27:45.330 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa03496d:::benchmark_data_smithi171_122772_object169938:head# 2024-04-20T17:27:45.330 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa03d7a4:::benchmark_data_smithi171_122772_object103612:head# 2024-04-20T17:27:45.331 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa04a5ab:::benchmark_data_smithi171_122772_object144821:head# 2024-04-20T17:27:45.331 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa057b49:::benchmark_data_smithi171_122772_object159588:head# 2024-04-20T17:27:45.332 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa0590c1:::benchmark_data_smithi171_122772_object137449:head# 2024-04-20T17:27:45.333 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa05f2de:::benchmark_data_smithi171_122772_object103405:head# 2024-04-20T17:27:45.333 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa0607d2:::benchmark_data_smithi171_122772_object115535:head# 2024-04-20T17:27:45.333 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa06b7ad:::benchmark_data_smithi171_122772_object144099:head# 2024-04-20T17:27:45.334 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa0701f8:::benchmark_data_smithi171_122772_object94760:head# 2024-04-20T17:27:45.334 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa077474:::benchmark_data_smithi171_122772_object162269:head# 2024-04-20T17:27:45.335 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa07818e:::benchmark_data_smithi171_122772_object159697:head# 2024-04-20T17:27:45.338 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa07c358:::benchmark_data_smithi171_122772_object111613:head# 2024-04-20T17:27:45.339 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa085760:::benchmark_data_smithi171_122772_object116937:head# 2024-04-20T17:27:45.339 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa0924a9:::benchmark_data_smithi171_122772_object159105:head# 2024-04-20T17:27:45.340 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa097752:::benchmark_data_smithi171_122772_object158488:head# 2024-04-20T17:27:45.340 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa099e0f:::benchmark_data_smithi171_122772_object159721:head# 2024-04-20T17:27:45.340 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa0a8138:::benchmark_data_smithi171_122772_object94985:head# 2024-04-20T17:27:45.340 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa0ab447:::benchmark_data_smithi171_122772_object169550:head# 2024-04-20T17:27:45.340 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa0adfe7:::benchmark_data_smithi171_122772_object93242:head# 2024-04-20T17:27:45.340 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa0b407b:::benchmark_data_smithi171_122772_object101848:head# 2024-04-20T17:27:45.343 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa0b69e8:::benchmark_data_smithi171_122772_object118446:head# 2024-04-20T17:27:45.343 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa0bcfae:::benchmark_data_smithi171_122772_object161900:head# 2024-04-20T17:27:45.343 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa0ccdc9:::benchmark_data_smithi171_122772_object123233:head# 2024-04-20T17:27:45.343 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa0cd052:::benchmark_data_smithi171_122772_object109015:head# 2024-04-20T17:27:45.344 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa0cd293:::benchmark_data_smithi171_122772_object141053:head# 2024-04-20T17:27:45.344 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa0cf47d:::benchmark_data_smithi171_122772_object167272:head# 2024-04-20T17:27:45.344 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa0dcd00:::benchmark_data_smithi171_122772_object160149:head# 2024-04-20T17:27:45.345 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:27:45.346 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa0e9cdb:::benchmark_data_smithi171_122772_object142082:head# 2024-04-20T17:27:45.347 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa0fc070:::benchmark_data_smithi171_122772_object112761:head# 2024-04-20T17:27:45.347 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa0fd94f:::benchmark_data_smithi171_122772_object142454:head# 2024-04-20T17:27:45.347 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa115205:::benchmark_data_smithi171_122772_object140977:head# 2024-04-20T17:27:45.347 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa117b61:::benchmark_data_smithi171_122772_object108017:head# 2024-04-20T17:27:45.347 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa120909:::benchmark_data_smithi171_122772_object114350:head# 2024-04-20T17:27:45.347 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa12c916:::benchmark_data_smithi171_122772_object157684:head# 2024-04-20T17:27:45.347 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa12ffa8:::benchmark_data_smithi171_122772_object122642:head# 2024-04-20T17:27:45.348 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa144d49:::benchmark_data_smithi171_122772_object163700:head# 2024-04-20T17:27:45.348 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa181965:::benchmark_data_smithi171_122772_object111363:head# 2024-04-20T17:27:45.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa189854:::benchmark_data_smithi171_122772_object128178:head# 2024-04-20T17:27:45.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa18da96:::benchmark_data_smithi171_122772_object101370:head# 2024-04-20T17:27:45.350 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa192ac5:::benchmark_data_smithi171_122772_object122734:head# 2024-04-20T17:27:45.351 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa1a032f:::benchmark_data_smithi171_122772_object87691:head# 2024-04-20T17:27:45.351 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa1a6b7a:::benchmark_data_smithi171_122772_object125534:head# 2024-04-20T17:27:45.351 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa1b784e:::benchmark_data_smithi171_122772_object106564:head# 2024-04-20T17:27:45.352 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa1bc5c5:::benchmark_data_smithi171_122772_object160150:head# 2024-04-20T17:27:45.353 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa1cbf9a:::benchmark_data_smithi171_122772_object163610:head# 2024-04-20T17:27:45.353 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa1ccee9:::benchmark_data_smithi171_122772_object148270:head# 2024-04-20T17:27:45.354 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa1ee408:::benchmark_data_smithi171_122772_object163811:head# 2024-04-20T17:27:45.354 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa1ef955:::benchmark_data_smithi171_122772_object87236:head# 2024-04-20T17:27:45.355 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa1f8ed2:::benchmark_data_smithi171_122772_object127542:head# 2024-04-20T17:27:45.355 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa20af63:::benchmark_data_smithi171_122772_object159994:head# 2024-04-20T17:27:45.355 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa222112:::benchmark_data_smithi171_122772_object97946:head# 2024-04-20T17:27:45.357 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa224c1a:::benchmark_data_smithi171_122772_object82843:head# 2024-04-20T17:27:45.357 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa22a910:::benchmark_data_smithi171_122772_object114459:head# 2024-04-20T17:27:45.357 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa22c954:::benchmark_data_smithi171_122772_object133162:head# 2024-04-20T17:27:45.357 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa234195:::benchmark_data_smithi171_122772_object138940:head# 2024-04-20T17:27:45.357 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa23b972:::benchmark_data_smithi171_122772_object131234:head# 2024-04-20T17:27:45.359 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa247052:::benchmark_data_smithi171_122772_object167892:head# 2024-04-20T17:27:45.359 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa2710b3:::benchmark_data_smithi171_122772_object169471:head# 2024-04-20T17:27:45.359 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa2765f2:::benchmark_data_smithi171_122772_object90164:head# 2024-04-20T17:27:45.359 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa2afe09:::benchmark_data_smithi171_122772_object131068:head# 2024-04-20T17:27:45.361 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa2b63ea:::benchmark_data_smithi171_122772_object149906:head# 2024-04-20T17:27:45.361 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa2bfd7c:::benchmark_data_smithi171_122772_object157265:head# 2024-04-20T17:27:45.361 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa2c062f:::benchmark_data_smithi171_122772_object156080:head# 2024-04-20T17:27:45.361 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa2c38f1:::benchmark_data_smithi171_122772_object108928:head# 2024-04-20T17:27:45.362 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa2c5801:::benchmark_data_smithi171_122772_object148906:head# 2024-04-20T17:27:45.362 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa2c7c60:::benchmark_data_smithi171_122772_object131953:head# 2024-04-20T17:27:45.363 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa2d3435:::benchmark_data_smithi171_122772_object133757:head# 2024-04-20T17:27:45.363 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa2dd3fa:::benchmark_data_smithi171_122772_object160136:head# 2024-04-20T17:27:45.363 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa2e072d:::benchmark_data_smithi171_122772_object117687:head# 2024-04-20T17:27:45.364 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa2e1c7c:::benchmark_data_smithi171_122772_object134322:head# 2024-04-20T17:27:45.364 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa2e2d75:::benchmark_data_smithi171_122772_object82681:head# 2024-04-20T17:27:45.365 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa2e7870:::benchmark_data_smithi171_122772_object118970:head# 2024-04-20T17:27:45.365 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa2f0519:::benchmark_data_smithi171_122772_object113199:head# 2024-04-20T17:27:45.365 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa2fd1e2:::benchmark_data_smithi171_122772_object120315:head# 2024-04-20T17:27:45.366 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa304edc:::benchmark_data_smithi171_122772_object168933:head# 2024-04-20T17:27:45.367 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa30a344:::benchmark_data_smithi171_122772_object148399:head# 2024-04-20T17:27:45.367 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa326b6b:::benchmark_data_smithi171_122772_object139942:head# 2024-04-20T17:27:45.368 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa32d26e:::benchmark_data_smithi171_122772_object155809:head# 2024-04-20T17:27:45.368 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa3349a3:::benchmark_data_smithi171_122772_object163704:head# 2024-04-20T17:27:45.368 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa33fddd:::benchmark_data_smithi171_122772_object107046:head# 2024-04-20T17:27:45.369 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa353fde:::benchmark_data_smithi171_122772_object151845:head# 2024-04-20T17:27:45.370 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa365f89:::benchmark_data_smithi171_122772_object78450:head# 2024-04-20T17:27:45.371 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa368e62:::benchmark_data_smithi171_122772_object78618:head# 2024-04-20T17:27:45.371 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa36dd3b:::benchmark_data_smithi171_122772_object96032:head# 2024-04-20T17:27:45.372 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa375c6d:::benchmark_data_smithi171_122772_object134860:head# 2024-04-20T17:27:45.372 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa37fefe:::benchmark_data_smithi171_122772_object79797:head# 2024-04-20T17:27:45.373 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa387ea7:::benchmark_data_smithi171_122772_object115102:head# 2024-04-20T17:27:45.374 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa38d33f:::benchmark_data_smithi171_122772_object130781:head# 2024-04-20T17:27:45.374 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa3b27c9:::benchmark_data_smithi171_122772_object130149:head# 2024-04-20T17:27:45.374 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa3b38f4:::benchmark_data_smithi171_122772_object98159:head# 2024-04-20T17:27:45.374 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa3b9827:::benchmark_data_smithi171_122772_object93863:head# 2024-04-20T17:27:45.376 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa3cdef5:::benchmark_data_smithi171_122772_object89803:head# 2024-04-20T17:27:45.376 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa3d3b35:::benchmark_data_smithi171_122772_object87918:head# 2024-04-20T17:27:45.376 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa3e2854:::benchmark_data_smithi171_122772_object156178:head# 2024-04-20T17:27:45.377 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa3eecf8:::benchmark_data_smithi171_122772_object158670:head# 2024-04-20T17:27:45.377 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa402b4e:::benchmark_data_smithi171_122772_object125670:head# 2024-04-20T17:27:45.378 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa404f5a:::benchmark_data_smithi171_122772_object152905:head# 2024-04-20T17:27:45.378 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa41447a:::benchmark_data_smithi171_122772_object114800:head# 2024-04-20T17:27:45.378 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa421b78:::benchmark_data_smithi171_122772_object151136:head# 2024-04-20T17:27:45.379 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa43d878:::benchmark_data_smithi171_122772_object108557:head# 2024-04-20T17:27:45.379 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa450a87:::benchmark_data_smithi171_122772_object104431:head# 2024-04-20T17:27:45.380 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa45235a:::benchmark_data_smithi171_122772_object145450:head# 2024-04-20T17:27:45.380 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa45c748:::benchmark_data_smithi171_122772_object136632:head# 2024-04-20T17:27:45.381 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa4609f6:::benchmark_data_smithi171_122772_object120793:head# 2024-04-20T17:27:45.381 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa461824:::benchmark_data_smithi171_122772_object144932:head# 2024-04-20T17:27:45.382 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa463c79:::benchmark_data_smithi171_122772_object79805:head# 2024-04-20T17:27:45.382 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa499061:::benchmark_data_smithi171_122772_object118581:head# 2024-04-20T17:27:45.383 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa4a284f:::benchmark_data_smithi171_122772_object145719:head# 2024-04-20T17:27:45.383 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa4a3c4b:::benchmark_data_smithi171_122772_object132302:head# 2024-04-20T17:27:45.383 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa4ac7bb:::benchmark_data_smithi171_122772_object162600:head# 2024-04-20T17:27:45.384 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa4b75db:::benchmark_data_smithi171_122772_object147867:head# 2024-04-20T17:27:45.385 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa4bcac9:::benchmark_data_smithi171_122772_object80275:head# 2024-04-20T17:27:45.386 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa4c069b:::benchmark_data_smithi171_122772_object145316:head# 2024-04-20T17:27:45.386 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa4c3142:::benchmark_data_smithi171_122772_object126976:head# 2024-04-20T17:27:45.387 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa4cb8ae:::benchmark_data_smithi171_122772_object101306:head# 2024-04-20T17:27:45.387 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa4f0393:::benchmark_data_smithi171_122772_object86627:head# 2024-04-20T17:27:45.387 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa4f9760:::benchmark_data_smithi171_122772_object153891:head# 2024-04-20T17:27:45.388 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa4fd23a:::benchmark_data_smithi171_122772_object97449:head# 2024-04-20T17:27:45.389 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa5167ff:::benchmark_data_smithi171_122772_object84440:head# 2024-04-20T17:27:45.389 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa51d541:::benchmark_data_smithi171_122772_object87397:head# 2024-04-20T17:27:45.389 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa525dee:::benchmark_data_smithi171_122772_object155373:head# 2024-04-20T17:27:45.390 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa52c024:::benchmark_data_smithi171_122772_object127705:head# 2024-04-20T17:27:45.391 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa52fde5:::benchmark_data_smithi171_122772_object97457:head# 2024-04-20T17:27:45.391 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa530123:::benchmark_data_smithi171_122772_object96082:head# 2024-04-20T17:27:45.392 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa5343d8:::benchmark_data_smithi171_122772_object166802:head# 2024-04-20T17:27:45.392 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa53ddb4:::benchmark_data_smithi171_122772_object142614:head# 2024-04-20T17:27:45.393 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa53f358:::benchmark_data_smithi171_122772_object141507:head# 2024-04-20T17:27:45.394 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa540159:::benchmark_data_smithi171_122772_object151390:head# 2024-04-20T17:27:45.394 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa54795f:::benchmark_data_smithi171_122772_object161805:head# 2024-04-20T17:27:45.394 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa54e559:::benchmark_data_smithi171_122772_object146967:head# 2024-04-20T17:27:45.395 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa54eef1:::benchmark_data_smithi171_122772_object145856:head# 2024-04-20T17:27:45.395 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa55302b:::benchmark_data_smithi171_122772_object103295:head# 2024-04-20T17:27:45.395 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa5605c6:::benchmark_data_smithi171_122772_object147863:head# 2024-04-20T17:27:45.396 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa58cee5:::benchmark_data_smithi171_122772_object98690:head# 2024-04-20T17:27:45.397 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa5a8853:::benchmark_data_smithi171_122772_object156838:head# 2024-04-20T17:27:45.397 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa5b355e:::benchmark_data_smithi171_122772_object163228:head# 2024-04-20T17:27:45.397 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa5c0f91:::benchmark_data_smithi171_122772_object88442:head# 2024-04-20T17:27:45.398 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa5c11b2:::benchmark_data_smithi171_122772_object106902:head# 2024-04-20T17:27:45.399 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa5d465c:::benchmark_data_smithi171_122772_object164556:head# 2024-04-20T17:27:45.399 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa5d475e:::benchmark_data_smithi171_122772_object143642:head# 2024-04-20T17:27:45.399 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa5e7666:::benchmark_data_smithi171_122772_object102308:head# 2024-04-20T17:27:45.401 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa5eb695:::benchmark_data_smithi171_122772_object107448:head# 2024-04-20T17:27:45.401 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa5f0227:::benchmark_data_smithi171_122772_object150105:head# 2024-04-20T17:27:45.401 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa5f2143:::benchmark_data_smithi171_122772_object140090:head# 2024-04-20T17:27:45.402 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa5f80d2:::benchmark_data_smithi171_122772_object150254:head# 2024-04-20T17:27:45.402 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa60edea:::benchmark_data_smithi171_122772_object84896:head# 2024-04-20T17:27:45.403 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa611ef4:::benchmark_data_smithi171_122772_object83003:head# 2024-04-20T17:27:45.403 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa623170:::benchmark_data_smithi171_122772_object105382:head# 2024-04-20T17:27:45.404 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa634763:::benchmark_data_smithi171_122772_object156210:head# 2024-04-20T17:27:45.404 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa638beb:::benchmark_data_smithi171_122772_object169400:head# 2024-04-20T17:27:45.404 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa63f45b:::benchmark_data_smithi171_122772_object99107:head# 2024-04-20T17:27:45.405 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa642129:::benchmark_data_smithi171_122772_object164100:head# 2024-04-20T17:27:45.405 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa6562c8:::benchmark_data_smithi171_122772_object78772:head# 2024-04-20T17:27:45.406 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa65ae74:::benchmark_data_smithi171_122772_object144145:head# 2024-04-20T17:27:45.407 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa65eb30:::benchmark_data_smithi171_122772_object119281:head# 2024-04-20T17:27:45.407 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa66d796:::benchmark_data_smithi171_122772_object118763:head# 2024-04-20T17:27:45.408 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa67465d:::benchmark_data_smithi171_122772_object166775:head# 2024-04-20T17:27:45.408 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa67b413:::benchmark_data_smithi171_122772_object145357:head# 2024-04-20T17:27:45.409 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa67b42b:::benchmark_data_smithi171_122772_object142685:head# 2024-04-20T17:27:45.409 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa67bc41:::benchmark_data_smithi171_122772_object118151:head# 2024-04-20T17:27:45.411 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa685e8d:::benchmark_data_smithi171_122772_object128500:head# 2024-04-20T17:27:45.411 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa687c15:::benchmark_data_smithi171_122772_object158654:head# 2024-04-20T17:27:45.411 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa68c76b:::benchmark_data_smithi171_122772_object130336:head# 2024-04-20T17:27:45.411 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa690698:::benchmark_data_smithi171_122772_object153508:head# 2024-04-20T17:27:45.412 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa6a289a:::benchmark_data_smithi171_122772_object121854:head# 2024-04-20T17:27:45.412 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa6ae3b7:::benchmark_data_smithi171_122772_object146314:head# 2024-04-20T17:27:45.413 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa6bd85f:::benchmark_data_smithi171_122772_object154710:head# 2024-04-20T17:27:45.413 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa6c3550:::benchmark_data_smithi171_122772_object147215:head# 2024-04-20T17:27:45.414 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa6ccba6:::benchmark_data_smithi171_122772_object168273:head# 2024-04-20T17:27:45.414 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa6d138c:::benchmark_data_smithi171_122772_object147284:head# 2024-04-20T17:27:45.414 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa6de083:::benchmark_data_smithi171_122772_object104578:head# 2024-04-20T17:27:45.415 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa6e899f:::benchmark_data_smithi171_122772_object155846:head# 2024-04-20T17:27:45.415 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa6fe5eb:::benchmark_data_smithi171_122772_object131574:head# 2024-04-20T17:27:45.416 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa705016:::benchmark_data_smithi171_122772_object132235:head# 2024-04-20T17:27:45.417 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa71758e:::benchmark_data_smithi171_122772_object109958:head# 2024-04-20T17:27:45.418 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa7216ad:::benchmark_data_smithi171_122772_object155752:head# 2024-04-20T17:27:45.418 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa725dff:::benchmark_data_smithi171_122772_object109793:head# 2024-04-20T17:27:45.418 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa72b4ea:::benchmark_data_smithi171_122772_object164671:head# 2024-04-20T17:27:45.420 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa73daab:::benchmark_data_smithi171_122772_object148844:head# 2024-04-20T17:27:45.420 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa746522:::benchmark_data_smithi171_122772_object134644:head# 2024-04-20T17:27:45.420 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa7471d0:::benchmark_data_smithi171_122772_object82432:head# 2024-04-20T17:27:45.420 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa74a00b:::benchmark_data_smithi171_122772_object148097:head# 2024-04-20T17:27:45.421 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa7508ee:::benchmark_data_smithi171_122772_object117585:head# 2024-04-20T17:27:45.422 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa752767:::benchmark_data_smithi171_122772_object143636:head# 2024-04-20T17:27:45.422 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa763625:::benchmark_data_smithi171_122772_object119559:head# 2024-04-20T17:27:45.423 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa76acba:::benchmark_data_smithi171_122772_object147001:head# 2024-04-20T17:27:45.423 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa76d479:::benchmark_data_smithi171_122772_object166021:head# 2024-04-20T17:27:45.423 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa7725bb:::benchmark_data_smithi171_122772_object146140:head# 2024-04-20T17:27:45.424 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa775c60:::benchmark_data_smithi171_122772_object166688:head# 2024-04-20T17:27:45.424 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa775d50:::benchmark_data_smithi171_122772_object137386:head# 2024-04-20T17:27:45.425 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa780cf8:::benchmark_data_smithi171_122772_object129938:head# 2024-04-20T17:27:45.425 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa7a4605:::benchmark_data_smithi171_122772_object91130:head# 2024-04-20T17:27:45.426 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa7a6eaa:::benchmark_data_smithi171_122772_object134118:head# 2024-04-20T17:27:45.426 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa7a74c8:::benchmark_data_smithi171_122772_object168918:head# 2024-04-20T17:27:45.426 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa7af730:::benchmark_data_smithi171_122772_object79057:head# 2024-04-20T17:27:45.427 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa7b13ae:::benchmark_data_smithi171_122772_object84796:head# 2024-04-20T17:27:45.427 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa7b3838:::benchmark_data_smithi171_122772_object144436:head# 2024-04-20T17:27:45.427 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa7c4d3e:::benchmark_data_smithi171_122772_object108126:head# 2024-04-20T17:27:45.428 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa7e4da9:::benchmark_data_smithi171_122772_object87969:head# 2024-04-20T17:27:45.428 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa7f59c8:::benchmark_data_smithi171_122772_object142019:head# 2024-04-20T17:27:45.429 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa7fec98:::benchmark_data_smithi171_122772_object100129:head# 2024-04-20T17:27:45.429 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa80617a:::benchmark_data_smithi171_122772_object152673:head# 2024-04-20T17:27:45.430 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa81add1:::benchmark_data_smithi171_122772_object133821:head# 2024-04-20T17:27:45.431 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa8657ab:::benchmark_data_smithi171_122772_object101915:head# 2024-04-20T17:27:45.431 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa86c0e6:::benchmark_data_smithi171_122772_object147303:head# 2024-04-20T17:27:45.432 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa88fc50:::benchmark_data_smithi171_122772_object122845:head# 2024-04-20T17:27:45.432 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa89819a:::benchmark_data_smithi171_122772_object160310:head# 2024-04-20T17:27:45.433 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa8abe5a:::benchmark_data_smithi171_122772_object154514:head# 2024-04-20T17:27:45.433 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa8acb14:::benchmark_data_smithi171_122772_object87464:head# 2024-04-20T17:27:45.434 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa8e9b97:::benchmark_data_smithi171_122772_object102054:head# 2024-04-20T17:27:45.434 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa8ebc18:::benchmark_data_smithi171_122772_object152307:head# 2024-04-20T17:27:45.435 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa8fb87e:::benchmark_data_smithi171_122772_object147508:head# 2024-04-20T17:27:45.436 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa904685:::benchmark_data_smithi171_122772_object102134:head# 2024-04-20T17:27:45.436 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa904956:::benchmark_data_smithi171_122772_object164762:head# 2024-04-20T17:27:45.436 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa904fc0:::benchmark_data_smithi171_122772_object139140:head# 2024-04-20T17:27:45.437 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa916661:::benchmark_data_smithi171_122772_object165232:head# 2024-04-20T17:27:45.437 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa92b71e:::benchmark_data_smithi171_122772_object109866:head# 2024-04-20T17:27:45.437 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa93641a:::benchmark_data_smithi171_122772_object81065:head# 2024-04-20T17:27:45.439 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa95428a:::benchmark_data_smithi171_122772_object162489:head# 2024-04-20T17:27:45.439 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa95a55d:::benchmark_data_smithi171_122772_object103137:head# 2024-04-20T17:27:45.439 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa961f17:::benchmark_data_smithi171_122772_object165812:head# 2024-04-20T17:27:45.440 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa9649ae:::benchmark_data_smithi171_122772_object132545:head# 2024-04-20T17:27:45.440 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa972047:::benchmark_data_smithi171_122772_object164757:head# 2024-04-20T17:27:45.441 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa97c64a:::benchmark_data_smithi171_122772_object166647:head# 2024-04-20T17:27:45.441 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa97e228:::benchmark_data_smithi171_122772_object95684:head# 2024-04-20T17:27:45.441 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa97e3f8:::benchmark_data_smithi171_122772_object125094:head# 2024-04-20T17:27:45.441 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa988695:::benchmark_data_smithi171_122772_object170147:head# 2024-04-20T17:27:45.442 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa9a5f60:::benchmark_data_smithi171_122772_object119939:head# 2024-04-20T17:27:45.443 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa9b84ba:::benchmark_data_smithi171_122772_object151094:head# 2024-04-20T17:27:45.443 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa9bce43:::benchmark_data_smithi171_122772_object168759:head# 2024-04-20T17:27:45.443 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa9cb6b7:::benchmark_data_smithi171_122772_object149115:head# 2024-04-20T17:27:45.444 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa9d7139:::benchmark_data_smithi171_122772_object133857:head# 2024-04-20T17:27:45.445 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa9dcfac:::benchmark_data_smithi171_122772_object116208:head# 2024-04-20T17:27:45.445 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aa9ec56f:::benchmark_data_smithi171_122772_object102144:head# 2024-04-20T17:27:45.445 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaa02996:::benchmark_data_smithi171_122772_object123739:head# 2024-04-20T17:27:45.446 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaa26dec:::benchmark_data_smithi171_122772_object98183:head# 2024-04-20T17:27:45.446 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaa31d2d:::benchmark_data_smithi171_122772_object136121:head# 2024-04-20T17:27:45.446 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaa3ce7d:::benchmark_data_smithi171_122772_object163183:head# 2024-04-20T17:27:45.448 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaa45304:::benchmark_data_smithi171_122772_object106901:head# 2024-04-20T17:27:45.448 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaa48f82:::benchmark_data_smithi171_122772_object160680:head# 2024-04-20T17:27:45.448 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaa4cab3:::benchmark_data_smithi171_122772_object99221:head# 2024-04-20T17:27:45.449 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaa50d58:::benchmark_data_smithi171_122772_object131263:head# 2024-04-20T17:27:45.449 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaa53087:::benchmark_data_smithi171_122772_object159333:head# 2024-04-20T17:27:45.450 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaa57dbb:::benchmark_data_smithi171_122772_object102755:head# 2024-04-20T17:27:45.451 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaa5b83d:::benchmark_data_smithi171_122772_object135601:head# 2024-04-20T17:27:45.451 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaa71be2:::benchmark_data_smithi171_122772_object157245:head# 2024-04-20T17:27:45.452 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaa7f338:::benchmark_data_smithi171_122772_object169100:head# 2024-04-20T17:27:45.452 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaa8bed5:::benchmark_data_smithi171_122772_object152250:head# 2024-04-20T17:27:45.454 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaa91788:::benchmark_data_smithi171_122772_object121871:head# 2024-04-20T17:27:45.454 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaa99071:::benchmark_data_smithi171_122772_object89261:head# 2024-04-20T17:27:45.454 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaabf934:::benchmark_data_smithi171_122772_object107941:head# 2024-04-20T17:27:45.454 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaac8415:::benchmark_data_smithi171_122772_object122829:head# 2024-04-20T17:27:45.455 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaad054f:::benchmark_data_smithi171_122772_object131057:head# 2024-04-20T17:27:45.455 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaad6b19:::benchmark_data_smithi171_122772_object116124:head# 2024-04-20T17:27:45.456 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaad856e:::benchmark_data_smithi171_122772_object79958:head# 2024-04-20T17:27:45.456 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaadd71c:::benchmark_data_smithi171_122772_object133571:head# 2024-04-20T17:27:45.457 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aab3996a:::benchmark_data_smithi171_122772_object102305:head# 2024-04-20T17:27:45.457 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aab435da:::benchmark_data_smithi171_122772_object107839:head# 2024-04-20T17:27:45.457 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aab5762d:::benchmark_data_smithi171_122772_object113668:head# 2024-04-20T17:27:45.458 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aab5ad9c:::benchmark_data_smithi171_122772_object78360:head# 2024-04-20T17:27:45.458 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aab7f488:::benchmark_data_smithi171_122772_object154989:head# 2024-04-20T17:27:45.458 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aab8d552:::benchmark_data_smithi171_122772_object115393:head# 2024-04-20T17:27:45.459 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aab8dd47:::benchmark_data_smithi171_122772_object90030:head# 2024-04-20T17:27:45.459 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aab8e1bf:::benchmark_data_smithi171_122772_object126040:head# 2024-04-20T17:27:45.460 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aab9ab91:::benchmark_data_smithi171_122772_object144272:head# 2024-04-20T17:27:45.460 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaba3182:::benchmark_data_smithi171_122772_object88070:head# 2024-04-20T17:27:45.460 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aabdbff8:::benchmark_data_smithi171_122772_object129638:head# 2024-04-20T17:27:45.461 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aabed876:::benchmark_data_smithi171_122772_object165919:head# 2024-04-20T17:27:45.461 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aac001c4:::benchmark_data_smithi171_122772_object119520:head# 2024-04-20T17:27:45.462 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aac0f81b:::benchmark_data_smithi171_122772_object140606:head# 2024-04-20T17:27:45.462 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aac1939d:::benchmark_data_smithi171_122772_object141818:head# 2024-04-20T17:27:45.463 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aac23016:::benchmark_data_smithi171_122772_object155015:head# 2024-04-20T17:27:45.463 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aac32d4f:::benchmark_data_smithi171_122772_object80043:head# 2024-04-20T17:27:45.464 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aac4d67a:::benchmark_data_smithi171_122772_object80993:head# 2024-04-20T17:27:45.464 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aac4e4a6:::benchmark_data_smithi171_122772_object83234:head# 2024-04-20T17:27:45.464 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aac5dccb:::benchmark_data_smithi171_122772_object97562:head# 2024-04-20T17:27:45.465 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aac68a7d:::benchmark_data_smithi171_122772_object162355:head# 2024-04-20T17:27:45.465 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aac6ff10:::benchmark_data_smithi171_122772_object137682:head# 2024-04-20T17:27:45.466 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aac867eb:::benchmark_data_smithi171_122772_object159605:head# 2024-04-20T17:27:45.466 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aac912fc:::benchmark_data_smithi171_122772_object101825:head# 2024-04-20T17:27:45.467 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aacbaea5:::benchmark_data_smithi171_122772_object103318:head# 2024-04-20T17:27:45.467 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aacc7ee6:::benchmark_data_smithi171_122772_object117898:head# 2024-04-20T17:27:45.468 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aacc8d6a:::benchmark_data_smithi171_122772_object86428:head# 2024-04-20T17:27:45.469 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aacd2c0d:::benchmark_data_smithi171_122772_object153532:head# 2024-04-20T17:27:45.469 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aacea66f:::benchmark_data_smithi171_122772_object145346:head# 2024-04-20T17:27:45.469 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aad05fe8:::benchmark_data_smithi171_122772_object102689:head# 2024-04-20T17:27:45.470 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aad1e6e0:::benchmark_data_smithi171_122772_object150831:head# 2024-04-20T17:27:45.470 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aad274c1:::benchmark_data_smithi171_122772_object94601:head# 2024-04-20T17:27:45.471 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aad4ccbd:::benchmark_data_smithi171_122772_object105887:head# 2024-04-20T17:27:45.471 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aad63683:::benchmark_data_smithi171_122772_object156002:head# 2024-04-20T17:27:45.472 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aad65aa5:::benchmark_data_smithi171_122772_object143477:head# 2024-04-20T17:27:45.472 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aad72561:::benchmark_data_smithi171_122772_object106291:head# 2024-04-20T17:27:45.474 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aad83163:::benchmark_data_smithi171_122772_object126476:head# 2024-04-20T17:27:45.474 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aad93df9:::benchmark_data_smithi171_122772_object114726:head# 2024-04-20T17:27:45.474 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aada887c:::benchmark_data_smithi171_122772_object82588:head# 2024-04-20T17:27:45.474 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aadbb37a:::benchmark_data_smithi171_122772_object154994:head# 2024-04-20T17:27:45.475 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aadbdb15:::benchmark_data_smithi171_122772_object81047:head# 2024-04-20T17:27:45.476 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aadbfee0:::benchmark_data_smithi171_122772_object136495:head# 2024-04-20T17:27:45.476 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aadf7452:::benchmark_data_smithi171_122772_object109035:head# 2024-04-20T17:27:45.477 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aadfa849:::benchmark_data_smithi171_122772_object119892:head# 2024-04-20T17:27:45.477 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aadfbca6:::benchmark_data_smithi171_122772_object137869:head# 2024-04-20T17:27:45.477 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aadfc777:::benchmark_data_smithi171_122772_object84086:head# 2024-04-20T17:27:45.478 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aae23072:::benchmark_data_smithi171_122772_object133021:head# 2024-04-20T17:27:45.478 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aae47d88:::benchmark_data_smithi171_122772_object147638:head# 2024-04-20T17:27:45.479 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aae4e26c:::benchmark_data_smithi171_122772_object88928:head# 2024-04-20T17:27:45.479 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aae58ba7:::benchmark_data_smithi171_122772_object92618:head# 2024-04-20T17:27:45.480 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aae68330:::benchmark_data_smithi171_122772_object96022:head# 2024-04-20T17:27:45.480 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aae6cc04:::benchmark_data_smithi171_122772_object114217:head# 2024-04-20T17:27:45.481 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aae6dca9:::benchmark_data_smithi171_122772_object100163:head# 2024-04-20T17:27:45.481 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aae75ea9:::benchmark_data_smithi171_122772_object107575:head# 2024-04-20T17:27:45.482 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aae7e103:::benchmark_data_smithi171_122772_object165152:head# 2024-04-20T17:27:45.482 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aae80fc2:::benchmark_data_smithi171_122772_object127772:head# 2024-04-20T17:27:45.483 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaeaf6fd:::benchmark_data_smithi171_122772_object78520:head# 2024-04-20T17:27:45.483 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaeb06f4:::benchmark_data_smithi171_122772_object87705:head# 2024-04-20T17:27:45.485 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaec3bff:::benchmark_data_smithi171_122772_object91041:head# 2024-04-20T17:27:45.485 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaec407c:::benchmark_data_smithi171_122772_object147802:head# 2024-04-20T17:27:45.486 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaed906a:::benchmark_data_smithi171_122772_object127803:head# 2024-04-20T17:27:45.486 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaee093b:::benchmark_data_smithi171_122772_object141006:head# 2024-04-20T17:27:45.486 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaeea48b:::benchmark_data_smithi171_122772_object128745:head# 2024-04-20T17:27:45.487 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaeedaaf:::benchmark_data_smithi171_122772_object79212:head# 2024-04-20T17:27:45.487 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaeeeacd:::benchmark_data_smithi171_122772_object139485:head# 2024-04-20T17:27:45.488 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaef0e06:::benchmark_data_smithi171_122772_object169418:head# 2024-04-20T17:27:45.488 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaef293a:::benchmark_data_smithi171_122772_object143877:head# 2024-04-20T17:27:45.489 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaf057bc:::benchmark_data_smithi171_122772_object104874:head# 2024-04-20T17:27:45.490 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaf08bbb:::benchmark_data_smithi171_122772_object115004:head# 2024-04-20T17:27:45.490 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaf2dd1a:::benchmark_data_smithi171_122772_object80950:head# 2024-04-20T17:27:45.490 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaf32931:::benchmark_data_smithi171_122772_object97460:head# 2024-04-20T17:27:45.491 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaf4b87c:::benchmark_data_smithi171_122772_object114625:head# 2024-04-20T17:27:45.491 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaf653e9:::benchmark_data_smithi171_122772_object115887:head# 2024-04-20T17:27:45.492 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaf6b91e:::benchmark_data_smithi171_122772_object78900:head# 2024-04-20T17:27:45.492 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaf713fa:::benchmark_data_smithi171_122772_object112921:head# 2024-04-20T17:27:45.493 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaf739ec:::benchmark_data_smithi171_122772_object165636:head# 2024-04-20T17:27:45.493 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaf7664e:::benchmark_data_smithi171_122772_object130127:head# 2024-04-20T17:27:45.494 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aafa9da8:::benchmark_data_smithi171_122772_object168639:head# 2024-04-20T17:27:45.494 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aafacec7:::benchmark_data_smithi171_122772_object145529:head# 2024-04-20T17:27:45.495 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aafb2dad:::benchmark_data_smithi171_122772_object110900:head# 2024-04-20T17:27:45.495 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aafb7147:::benchmark_data_smithi171_122772_object80459:head# 2024-04-20T17:27:45.496 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aafc57d7:::benchmark_data_smithi171_122772_object97547:head# 2024-04-20T17:27:45.496 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aafce4d3:::benchmark_data_smithi171_122772_object168116:head# 2024-04-20T17:27:45.497 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aafdee72:::benchmark_data_smithi171_122772_object159983:head# 2024-04-20T17:27:45.497 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aafe089c:::benchmark_data_smithi171_122772_object113535:head# 2024-04-20T17:27:45.498 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aafe562a:::benchmark_data_smithi171_122772_object156920:head# 2024-04-20T17:27:45.499 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aaffa8a9:::benchmark_data_smithi171_122772_object87006:head# 2024-04-20T17:27:45.499 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab02176e:::benchmark_data_smithi171_122772_object162453:head# 2024-04-20T17:27:45.500 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab02280d:::benchmark_data_smithi171_122772_object145586:head# 2024-04-20T17:27:45.501 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab022998:::benchmark_data_smithi171_122772_object152941:head# 2024-04-20T17:27:45.501 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab025765:::benchmark_data_smithi171_122772_object157239:head# 2024-04-20T17:27:45.501 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab03f332:::benchmark_data_smithi171_122772_object88088:head# 2024-04-20T17:27:45.502 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab04c63c:::benchmark_data_smithi171_122772_object162884:head# 2024-04-20T17:27:45.502 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab0551d7:::benchmark_data_smithi171_122772_object91532:head# 2024-04-20T17:27:45.503 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab06021a:::benchmark_data_smithi171_122772_object82042:head# 2024-04-20T17:27:45.503 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab06eaad:::benchmark_data_smithi171_122772_object138043:head# 2024-04-20T17:27:45.504 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab0798ba:::benchmark_data_smithi171_122772_object80137:head# 2024-04-20T17:27:45.505 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab07ed4c:::benchmark_data_smithi171_122772_object111748:head# 2024-04-20T17:27:45.505 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab08912e:::benchmark_data_smithi171_122772_object153035:head# 2024-04-20T17:27:45.505 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab092b5c:::benchmark_data_smithi171_122772_object164783:head# 2024-04-20T17:27:45.506 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab0957c4:::benchmark_data_smithi171_122772_object144928:head# 2024-04-20T17:27:45.506 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab09c0e9:::benchmark_data_smithi171_122772_object115401:head# 2024-04-20T17:27:45.506 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab09c3da:::benchmark_data_smithi171_122772_object84221:head# 2024-04-20T17:27:45.507 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab0bcf6a:::benchmark_data_smithi171_122772_object150216:head# 2024-04-20T17:27:45.507 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab0cb644:::benchmark_data_smithi171_122772_object157691:head# 2024-04-20T17:27:45.508 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab0cccaf:::benchmark_data_smithi171_122772_object78545:head# 2024-04-20T17:27:45.509 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab0ce798:::benchmark_data_smithi171_122772_object162526:head# 2024-04-20T17:27:45.509 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab0e0097:::benchmark_data_smithi171_122772_object104498:head# 2024-04-20T17:27:45.509 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab0e300f:::benchmark_data_smithi171_122772_object164180:head# 2024-04-20T17:27:45.510 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab0ff561:::benchmark_data_smithi171_122772_object128680:head# 2024-04-20T17:27:45.510 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab103b75:::benchmark_data_smithi171_122772_object114989:head# 2024-04-20T17:27:45.511 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab118c11:::benchmark_data_smithi171_122772_object119732:head# 2024-04-20T17:27:45.511 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab122729:::benchmark_data_smithi171_122772_object121199:head# 2024-04-20T17:27:45.511 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab129153:::benchmark_data_smithi171_122772_object137289:head# 2024-04-20T17:27:45.512 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab1316ad:::benchmark_data_smithi171_122772_object96734:head# 2024-04-20T17:27:45.512 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab131df9:::benchmark_data_smithi171_122772_object109086:head# 2024-04-20T17:27:45.512 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab141ed0:::benchmark_data_smithi171_122772_object83555:head# 2024-04-20T17:27:45.513 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab1521a6:::benchmark_data_smithi171_122772_object127806:head# 2024-04-20T17:27:45.514 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab1589ff:::benchmark_data_smithi171_122772_object84607:head# 2024-04-20T17:27:45.514 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab1675d7:::benchmark_data_smithi171_122772_object146125:head# 2024-04-20T17:27:45.515 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab1677b6:::benchmark_data_smithi171_122772_object115456:head# 2024-04-20T17:27:45.515 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab168667:::benchmark_data_smithi171_122772_object78453:head# 2024-04-20T17:27:45.515 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab1754d6:::benchmark_data_smithi171_122772_object157392:head# 2024-04-20T17:27:45.516 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab177e90:::benchmark_data_smithi171_122772_object82138:head# 2024-04-20T17:27:45.516 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab1982d3:::benchmark_data_smithi171_122772_object86274:head# 2024-04-20T17:27:45.517 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab1a6067:::benchmark_data_smithi171_122772_object89114:head# 2024-04-20T17:27:45.517 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab1b90b6:::benchmark_data_smithi171_122772_object156815:head# 2024-04-20T17:27:45.518 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab1bbf13:::benchmark_data_smithi171_122772_object137648:head# 2024-04-20T17:27:45.518 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab1c727c:::benchmark_data_smithi171_122772_object90055:head# 2024-04-20T17:27:45.519 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab1cae04:::benchmark_data_smithi171_122772_object124747:head# 2024-04-20T17:27:45.519 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab1cdaf0:::benchmark_data_smithi171_122772_object82936:head# 2024-04-20T17:27:45.519 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab1d8c98:::benchmark_data_smithi171_122772_object150684:head# 2024-04-20T17:27:45.520 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab1e40b5:::benchmark_data_smithi171_122772_object84930:head# 2024-04-20T17:27:45.521 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab1e6ca8:::benchmark_data_smithi171_122772_object151919:head# 2024-04-20T17:27:45.522 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab1f386c:::benchmark_data_smithi171_122772_object110936:head# 2024-04-20T17:27:45.522 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab1f3ee2:::benchmark_data_smithi171_122772_object154038:head# 2024-04-20T17:27:45.522 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab1feb25:::benchmark_data_smithi171_122772_object147286:head# 2024-04-20T17:27:45.523 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab200495:::benchmark_data_smithi171_122772_object103547:head# 2024-04-20T17:27:45.523 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab200ad0:::benchmark_data_smithi171_122772_object144674:head# 2024-04-20T17:27:45.523 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab201909:::benchmark_data_smithi171_122772_object154289:head# 2024-04-20T17:27:45.524 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab2048af:::benchmark_data_smithi171_122772_object143917:head# 2024-04-20T17:27:45.525 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab211959:::benchmark_data_smithi171_122772_object85326:head# 2024-04-20T17:27:45.525 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab219c44:::benchmark_data_smithi171_122772_object133274:head# 2024-04-20T17:27:45.526 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab23eb49:::benchmark_data_smithi171_122772_object150215:head# 2024-04-20T17:27:45.526 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab247747:::benchmark_data_smithi171_122772_object89618:head# 2024-04-20T17:27:45.526 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab25945b:::benchmark_data_smithi171_122772_object165443:head# 2024-04-20T17:27:45.527 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab26a234:::benchmark_data_smithi171_122772_object169221:head# 2024-04-20T17:27:45.527 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab2796c3:::benchmark_data_smithi171_122772_object107942:head# 2024-04-20T17:27:45.527 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab27e33c:::benchmark_data_smithi171_122772_object108449:head# 2024-04-20T17:27:45.529 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab283dca:::benchmark_data_smithi171_122772_object162968:head# 2024-04-20T17:27:45.529 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab28be66:::benchmark_data_smithi171_122772_object115162:head# 2024-04-20T17:27:45.529 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab291ce5:::benchmark_data_smithi171_122772_object137806:head# 2024-04-20T17:27:45.529 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab2944fd:::benchmark_data_smithi171_122772_object93514:head# 2024-04-20T17:27:45.530 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab299209:::benchmark_data_smithi171_122772_object120853:head# 2024-04-20T17:27:45.530 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab2a107b:::benchmark_data_smithi171_122772_object154755:head# 2024-04-20T17:27:45.531 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab2b25ae:::benchmark_data_smithi171_122772_object111614:head# 2024-04-20T17:27:45.531 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab2c92f0:::benchmark_data_smithi171_122772_object142372:head# 2024-04-20T17:27:45.532 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab2cb07d:::benchmark_data_smithi171_122772_object145015:head# 2024-04-20T17:27:45.532 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab2d0750:::benchmark_data_smithi171_122772_object134850:head# 2024-04-20T17:27:45.533 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab2d2d89:::benchmark_data_smithi171_122772_object86743:head# 2024-04-20T17:27:45.534 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab2d3d3a:::benchmark_data_smithi171_122772_object161514:head# 2024-04-20T17:27:45.534 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab2ee2cf:::benchmark_data_smithi171_122772_object92620:head# 2024-04-20T17:27:45.534 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab2f6919:::benchmark_data_smithi171_122772_object119020:head# 2024-04-20T17:27:45.534 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab2fa544:::benchmark_data_smithi171_122772_object123866:head# 2024-04-20T17:27:45.535 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab3003cb:::benchmark_data_smithi171_122772_object141824:head# 2024-04-20T17:27:45.535 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab301a9f:::benchmark_data_smithi171_122772_object167895:head# 2024-04-20T17:27:45.536 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab30805e:::benchmark_data_smithi171_122772_object157958:head# 2024-04-20T17:27:45.536 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab319a6a:::benchmark_data_smithi171_122772_object144104:head# 2024-04-20T17:27:45.536 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab339955:::benchmark_data_smithi171_122772_object156543:head# 2024-04-20T17:27:45.537 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab33bfd8:::benchmark_data_smithi171_122772_object169189:head# 2024-04-20T17:27:45.537 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab33e17e:::benchmark_data_smithi171_122772_object153247:head# 2024-04-20T17:27:45.538 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab34d45c:::benchmark_data_smithi171_122772_object129840:head# 2024-04-20T17:27:45.538 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab34eee2:::benchmark_data_smithi171_122772_object150113:head# 2024-04-20T17:27:45.539 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab35e8c4:::benchmark_data_smithi171_122772_object91973:head# 2024-04-20T17:27:45.539 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab388e1b:::benchmark_data_smithi171_122772_object148678:head# 2024-04-20T17:27:45.540 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab38d6ea:::benchmark_data_smithi171_122772_object88490:head# 2024-04-20T17:27:45.540 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab399dda:::benchmark_data_smithi171_122772_object150635:head# 2024-04-20T17:27:45.541 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab39d313:::benchmark_data_smithi171_122772_object165131:head# 2024-04-20T17:27:45.542 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab39d7fd:::benchmark_data_smithi171_122772_object78569:head# 2024-04-20T17:27:45.542 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab3a3639:::benchmark_data_smithi171_122772_object167058:head# 2024-04-20T17:27:45.542 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab3ad0a2:::benchmark_data_smithi171_122772_object85504:head# 2024-04-20T17:27:45.542 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab3b297a:::benchmark_data_smithi171_122772_object131160:head# 2024-04-20T17:27:45.543 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab3b5ad0:::benchmark_data_smithi171_122772_object80694:head# 2024-04-20T17:27:45.543 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab3c6b92:::benchmark_data_smithi171_122772_object165882:head# 2024-04-20T17:27:45.544 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab3d1fc8:::benchmark_data_smithi171_122772_object139660:head# 2024-04-20T17:27:45.544 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab3d21a8:::benchmark_data_smithi171_122772_object104841:head# 2024-04-20T17:27:45.545 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab3d8077:::benchmark_data_smithi171_122772_object128262:head# 2024-04-20T17:27:45.545 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab3d8b2f:::benchmark_data_smithi171_122772_object114505:head# 2024-04-20T17:27:45.547 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab3dd402:::benchmark_data_smithi171_122772_object131995:head# 2024-04-20T17:27:45.547 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab3e8d8f:::benchmark_data_smithi171_122772_object146553:head# 2024-04-20T17:27:45.547 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab3e9987:::benchmark_data_smithi171_122772_object113838:head# 2024-04-20T17:27:45.548 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab3f1ca9:::benchmark_data_smithi171_122772_object111699:head# 2024-04-20T17:27:45.548 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab3ffa71:::benchmark_data_smithi171_122772_object92217:head# 2024-04-20T17:27:45.548 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab40c9b7:::benchmark_data_smithi171_122772_object161267:head# 2024-04-20T17:27:45.549 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab42c2ff:::benchmark_data_smithi171_122772_object85913:head# 2024-04-20T17:27:45.549 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab4323ef:::benchmark_data_smithi171_122772_object122402:head# 2024-04-20T17:27:45.550 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab43c226:::benchmark_data_smithi171_122772_object138590:head# 2024-04-20T17:27:45.551 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab444ca5:::benchmark_data_smithi171_122772_object94103:head# 2024-04-20T17:27:45.551 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab44bc0d:::benchmark_data_smithi171_122772_object78267:head# 2024-04-20T17:27:45.551 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab456b7d:::benchmark_data_smithi171_122772_object152719:head# 2024-04-20T17:27:45.552 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab45d6c0:::benchmark_data_smithi171_122772_object118604:head# 2024-04-20T17:27:45.552 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab46191b:::benchmark_data_smithi171_122772_object83761:head# 2024-04-20T17:27:45.553 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab4626b3:::benchmark_data_smithi171_122772_object140789:head# 2024-04-20T17:27:45.553 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab46c83b:::benchmark_data_smithi171_122772_object143916:head# 2024-04-20T17:27:45.554 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab475e2b:::benchmark_data_smithi171_122772_object89146:head# 2024-04-20T17:27:45.554 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab48a4b2:::benchmark_data_smithi171_122772_object150908:head# 2024-04-20T17:27:45.555 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab49a53e:::benchmark_data_smithi171_122772_object121441:head# 2024-04-20T17:27:45.555 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab49e7e8:::benchmark_data_smithi171_122772_object134253:head# 2024-04-20T17:27:45.555 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab4ad733:::benchmark_data_smithi171_122772_object131828:head# 2024-04-20T17:27:45.556 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab4c27bd:::benchmark_data_smithi171_122772_object139579:head# 2024-04-20T17:27:45.556 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab4c8593:::benchmark_data_smithi171_122772_object144648:head# 2024-04-20T17:27:45.556 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab4d306c:::benchmark_data_smithi171_122772_object164224:head# 2024-04-20T17:27:45.558 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab4eb77f:::benchmark_data_smithi171_122772_object81599:head# 2024-04-20T17:27:45.558 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab4ebe61:::benchmark_data_smithi171_122772_object149278:head# 2024-04-20T17:27:45.558 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab4fb1a7:::benchmark_data_smithi171_122772_object107555:head# 2024-04-20T17:27:45.558 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab4ff5c1:::benchmark_data_smithi171_122772_object151599:head# 2024-04-20T17:27:45.560 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab50a30b:::benchmark_data_smithi171_122772_object165748:head# 2024-04-20T17:27:45.560 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab51990d:::benchmark_data_smithi171_122772_object114872:head# 2024-04-20T17:27:45.560 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab532b68:::benchmark_data_smithi171_122772_object154890:head# 2024-04-20T17:27:45.560 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab5391bb:::benchmark_data_smithi171_122772_object157836:head# 2024-04-20T17:27:45.561 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab542eb8:::benchmark_data_smithi171_122772_object101547:head# 2024-04-20T17:27:45.561 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab551bd8:::benchmark_data_smithi171_122772_object157604:head# 2024-04-20T17:27:45.561 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab552fc1:::benchmark_data_smithi171_122772_object83106:head# 2024-04-20T17:27:45.563 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab56bb50:::benchmark_data_smithi171_122772_object115132:head# 2024-04-20T17:27:45.563 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab578c86:::benchmark_data_smithi171_122772_object86363:head# 2024-04-20T17:27:45.563 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab58087a:::benchmark_data_smithi171_122772_object159951:head# 2024-04-20T17:27:45.563 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab597dea:::benchmark_data_smithi171_122772_object80360:head# 2024-04-20T17:27:45.564 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab5b79f4:::benchmark_data_smithi171_122772_object93957:head# 2024-04-20T17:27:45.565 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab5bbaf2:::benchmark_data_smithi171_122772_object141216:head# 2024-04-20T17:27:45.565 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab5c7da6:::benchmark_data_smithi171_122772_object117859:head# 2024-04-20T17:27:45.566 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab5d5eb4:::benchmark_data_smithi171_122772_object156547:head# 2024-04-20T17:27:45.566 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab5ece38:::benchmark_data_smithi171_122772_object169798:head# 2024-04-20T17:27:45.566 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab5f973d:::benchmark_data_smithi171_122772_object123989:head# 2024-04-20T17:27:45.567 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab5fd8a2:::benchmark_data_smithi171_122772_object84685:head# 2024-04-20T17:27:45.567 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab604977:::benchmark_data_smithi171_122772_object104629:head# 2024-04-20T17:27:45.568 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab60ab0f:::benchmark_data_smithi171_122772_object155634:head# 2024-04-20T17:27:45.569 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab60eb99:::benchmark_data_smithi171_122772_object90973:head# 2024-04-20T17:27:45.569 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab60f172:::benchmark_data_smithi171_122772_object121327:head# 2024-04-20T17:27:45.569 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab64114e:::benchmark_data_smithi171_122772_object132863:head# 2024-04-20T17:27:45.570 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab658ff3:::benchmark_data_smithi171_122772_object115365:head# 2024-04-20T17:27:45.570 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab65b128:::benchmark_data_smithi171_122772_object160450:head# 2024-04-20T17:27:45.571 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab65f948:::benchmark_data_smithi171_122772_object164348:head# 2024-04-20T17:27:45.571 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab662160:::benchmark_data_smithi171_122772_object147626:head# 2024-04-20T17:27:45.572 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab67c016:::benchmark_data_smithi171_122772_object104139:head# 2024-04-20T17:27:45.573 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab680a27:::benchmark_data_smithi171_122772_object126529:head# 2024-04-20T17:27:45.573 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab69c625:::benchmark_data_smithi171_122772_object95813:head# 2024-04-20T17:27:45.573 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab69ef77:::benchmark_data_smithi171_122772_object160058:head# 2024-04-20T17:27:45.574 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab6a511b:::benchmark_data_smithi171_122772_object102856:head# 2024-04-20T17:27:45.574 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab6da968:::benchmark_data_smithi171_122772_object105747:head# 2024-04-20T17:27:45.575 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab6ea764:::benchmark_data_smithi171_122772_object83561:head# 2024-04-20T17:27:45.576 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab6ec394:::benchmark_data_smithi171_122772_object81353:head# 2024-04-20T17:27:45.576 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab707d89:::benchmark_data_smithi171_122772_object94254:head# 2024-04-20T17:27:45.576 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab70c90e:::benchmark_data_smithi171_122772_object151736:head# 2024-04-20T17:27:45.577 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab7135e1:::benchmark_data_smithi171_122772_object141508:head# 2024-04-20T17:27:45.577 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab716656:::benchmark_data_smithi171_122772_object159948:head# 2024-04-20T17:27:45.578 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab72f472:::benchmark_data_smithi171_122772_object165352:head# 2024-04-20T17:27:45.578 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab72f94f:::benchmark_data_smithi171_122772_object85650:head# 2024-04-20T17:27:45.579 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab742930:::benchmark_data_smithi171_122772_object111792:head# 2024-04-20T17:27:45.580 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab7468eb:::benchmark_data_smithi171_122772_object154726:head# 2024-04-20T17:27:45.580 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab74ce04:::benchmark_data_smithi171_122772_object82964:head# 2024-04-20T17:27:45.580 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab74e91f:::benchmark_data_smithi171_122772_object126622:head# 2024-04-20T17:27:45.580 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab757c0d:::benchmark_data_smithi171_122772_object168034:head# 2024-04-20T17:27:45.580 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab75a77c:::benchmark_data_smithi171_122772_object126174:head# 2024-04-20T17:27:45.581 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab7720d1:::benchmark_data_smithi171_122772_object145314:head# 2024-04-20T17:27:45.582 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab7794a7:::benchmark_data_smithi171_122772_object100632:head# 2024-04-20T17:27:45.582 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab78ba52:::benchmark_data_smithi171_122772_object128516:head# 2024-04-20T17:27:45.582 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab7902c3:::benchmark_data_smithi171_122772_object123651:head# 2024-04-20T17:27:45.583 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab796078:::benchmark_data_smithi171_122772_object100485:head# 2024-04-20T17:27:45.583 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab79bebd:::benchmark_data_smithi171_122772_object104305:head# 2024-04-20T17:27:45.583 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab7bd121:::benchmark_data_smithi171_122772_object85969:head# 2024-04-20T17:27:45.584 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab7c718f:::benchmark_data_smithi171_122772_object150415:head# 2024-04-20T17:27:45.584 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab7c949a:::benchmark_data_smithi171_122772_object162171:head# 2024-04-20T17:27:45.585 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab7d65bd:::benchmark_data_smithi171_122772_object88008:head# 2024-04-20T17:27:45.585 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab7e204d:::benchmark_data_smithi171_122772_object124895:head# 2024-04-20T17:27:45.586 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab7e4938:::benchmark_data_smithi171_122772_object80521:head# 2024-04-20T17:27:45.586 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab7ec6ad:::benchmark_data_smithi171_122772_object162952:head# 2024-04-20T17:27:45.586 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab7ffd20:::benchmark_data_smithi171_122772_object159337:head# 2024-04-20T17:27:45.587 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab804652:::benchmark_data_smithi171_122772_object152607:head# 2024-04-20T17:27:45.587 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab81932d:::benchmark_data_smithi171_122772_object88946:head# 2024-04-20T17:27:45.588 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab81c85e:::benchmark_data_smithi171_122772_object141214:head# 2024-04-20T17:27:45.588 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab82c13f:::benchmark_data_smithi171_122772_object161198:head# 2024-04-20T17:27:45.589 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab82c190:::benchmark_data_smithi171_122772_object102096:head# 2024-04-20T17:27:45.589 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab82cbe0:::benchmark_data_smithi171_122772_object131340:head# 2024-04-20T17:27:45.590 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab82f1f3:::benchmark_data_smithi171_122772_object82360:head# 2024-04-20T17:27:45.590 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab831435:::benchmark_data_smithi171_122772_object86493:head# 2024-04-20T17:27:45.590 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab850d50:::benchmark_data_smithi171_122772_object111287:head# 2024-04-20T17:27:45.591 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab850e0e:::benchmark_data_smithi171_122772_object121073:head# 2024-04-20T17:27:45.591 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab853973:::benchmark_data_smithi171_122772_object86113:head# 2024-04-20T17:27:45.593 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab856928:::benchmark_data_smithi171_122772_object115439:head# 2024-04-20T17:27:45.593 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab863f48:::benchmark_data_smithi171_122772_object85010:head# 2024-04-20T17:27:45.593 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab8672c9:::benchmark_data_smithi171_122772_object110497:head# 2024-04-20T17:27:45.593 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab88c33a:::benchmark_data_smithi171_122772_object159081:head# 2024-04-20T17:27:45.594 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab891b77:::benchmark_data_smithi171_122772_object138162:head# 2024-04-20T17:27:45.594 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab8af5fc:::benchmark_data_smithi171_122772_object142488:head# 2024-04-20T17:27:45.594 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab8c0a53:::benchmark_data_smithi171_122772_object120592:head# 2024-04-20T17:27:45.595 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab8c2991:::benchmark_data_smithi171_122772_object91657:head# 2024-04-20T17:27:45.595 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab8c4e25:::benchmark_data_smithi171_122772_object146350:head# 2024-04-20T17:27:45.596 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab8c55ba:::benchmark_data_smithi171_122772_object123953:head# 2024-04-20T17:27:45.596 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab8c6c5b:::benchmark_data_smithi171_122772_object168809:head# 2024-04-20T17:27:45.597 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab8c7151:::benchmark_data_smithi171_122772_object146211:head# 2024-04-20T17:27:45.597 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab8da14c:::benchmark_data_smithi171_122772_object85096:head# 2024-04-20T17:27:45.598 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab8f10c9:::benchmark_data_smithi171_122772_object94415:head# 2024-04-20T17:27:45.599 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab8fc5c0:::benchmark_data_smithi171_122772_object135428:head# 2024-04-20T17:27:45.599 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab9260d4:::benchmark_data_smithi171_122772_object141156:head# 2024-04-20T17:27:45.599 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab92c091:::benchmark_data_smithi171_122772_object88192:head# 2024-04-20T17:27:45.600 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab934f2c:::benchmark_data_smithi171_122772_object124459:head# 2024-04-20T17:27:45.600 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab9385f2:::benchmark_data_smithi171_122772_object110413:head# 2024-04-20T17:27:45.600 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab9485bd:::benchmark_data_smithi171_122772_object162544:head# 2024-04-20T17:27:45.601 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab9686c4:::benchmark_data_smithi171_122772_object162633:head# 2024-04-20T17:27:45.601 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab96f775:::benchmark_data_smithi171_122772_object151866:head# 2024-04-20T17:27:45.601 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab96ff5a:::benchmark_data_smithi171_122772_object138369:head# 2024-04-20T17:27:45.602 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab972af3:::benchmark_data_smithi171_122772_object82331:head# 2024-04-20T17:27:45.602 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab9876ba:::benchmark_data_smithi171_122772_object84879:head# 2024-04-20T17:27:45.603 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab99df3b:::benchmark_data_smithi171_122772_object98796:head# 2024-04-20T17:27:45.603 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab9ae2d2:::benchmark_data_smithi171_122772_object88615:head# 2024-04-20T17:27:45.604 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab9af0ac:::benchmark_data_smithi171_122772_object93211:head# 2024-04-20T17:27:45.605 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab9b0c80:::benchmark_data_smithi171_122772_object135070:head# 2024-04-20T17:27:45.605 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab9b2718:::benchmark_data_smithi171_122772_object102301:head# 2024-04-20T17:27:45.606 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab9b6d47:::benchmark_data_smithi171_122772_object140857:head# 2024-04-20T17:27:45.606 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab9c6f28:::benchmark_data_smithi171_122772_object89380:head# 2024-04-20T17:27:45.607 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab9c72e7:::benchmark_data_smithi171_122772_object94155:head# 2024-04-20T17:27:45.607 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab9cc13d:::benchmark_data_smithi171_122772_object104196:head# 2024-04-20T17:27:45.608 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab9d2806:::benchmark_data_smithi171_122772_object143008:head# 2024-04-20T17:27:45.608 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab9d53f1:::benchmark_data_smithi171_122772_object134175:head# 2024-04-20T17:27:45.609 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab9e0a3c:::benchmark_data_smithi171_122772_object106716:head# 2024-04-20T17:27:45.609 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab9ee9b0:::benchmark_data_smithi171_122772_object136981:head# 2024-04-20T17:27:45.610 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab9f031f:::benchmark_data_smithi171_122772_object158986:head# 2024-04-20T17:27:45.610 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab9f53d6:::benchmark_data_smithi171_122772_object131142:head# 2024-04-20T17:27:45.610 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ab9f70eb:::benchmark_data_smithi171_122772_object89926:head# 2024-04-20T17:27:45.611 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aba17c2c:::benchmark_data_smithi171_122772_object154088:head# 2024-04-20T17:27:45.611 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aba1c4d4:::benchmark_data_smithi171_122772_object104223:head# 2024-04-20T17:27:45.611 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aba1e928:::benchmark_data_smithi171_122772_object113538:head# 2024-04-20T17:27:45.612 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aba43695:::benchmark_data_smithi171_122772_object144229:head# 2024-04-20T17:27:45.612 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aba479f4:::benchmark_data_smithi171_122772_object138993:head# 2024-04-20T17:27:45.613 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aba4f597:::benchmark_data_smithi171_122772_object139541:head# 2024-04-20T17:27:45.613 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aba5cbed:::benchmark_data_smithi171_122772_object113828:head# 2024-04-20T17:27:45.614 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aba6d127:::benchmark_data_smithi171_122772_object156684:head# 2024-04-20T17:27:45.614 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aba7622d:::benchmark_data_smithi171_122772_object79178:head# 2024-04-20T17:27:45.615 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aba801af:::benchmark_data_smithi171_122772_object116268:head# 2024-04-20T17:27:45.615 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aba8151a:::benchmark_data_smithi171_122772_object99913:head# 2024-04-20T17:27:45.616 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abaa3b2a:::benchmark_data_smithi171_122772_object154872:head# 2024-04-20T17:27:45.616 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abaab8ff:::benchmark_data_smithi171_122772_object107953:head# 2024-04-20T17:27:45.617 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abab9221:::benchmark_data_smithi171_122772_object97357:head# 2024-04-20T17:27:45.617 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abac5f69:::benchmark_data_smithi171_122772_object110307:head# 2024-04-20T17:27:45.618 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abacb953:::benchmark_data_smithi171_122772_object122573:head# 2024-04-20T17:27:45.619 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abacf803:::benchmark_data_smithi171_122772_object99674:head# 2024-04-20T17:27:45.619 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abadb475:::benchmark_data_smithi171_122772_object169569:head# 2024-04-20T17:27:45.619 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abaed156:::benchmark_data_smithi171_122772_object115784:head# 2024-04-20T17:27:45.620 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abb15847:::benchmark_data_smithi171_122772_object122632:head# 2024-04-20T17:27:45.620 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abb27db5:::benchmark_data_smithi171_122772_object116930:head# 2024-04-20T17:27:45.620 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abb4b13f:::benchmark_data_smithi171_122772_object106207:head# 2024-04-20T17:27:45.621 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abb4fc06:::benchmark_data_smithi171_122772_object105140:head# 2024-04-20T17:27:45.621 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abb535da:::benchmark_data_smithi171_122772_object78687:head# 2024-04-20T17:27:45.622 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abb79a41:::benchmark_data_smithi171_122772_object169627:head# 2024-04-20T17:27:45.623 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abb833c3:::benchmark_data_smithi171_122772_object112038:head# 2024-04-20T17:27:45.623 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abb84882:::benchmark_data_smithi171_122772_object127419:head# 2024-04-20T17:27:45.623 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abb9ec05:::benchmark_data_smithi171_122772_object142856:head# 2024-04-20T17:27:45.624 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abba1011:::benchmark_data_smithi171_122772_object144630:head# 2024-04-20T17:27:45.624 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abbb0b6f:::benchmark_data_smithi171_122772_object134283:head# 2024-04-20T17:27:45.625 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abbba119:::benchmark_data_smithi171_122772_object145712:head# 2024-04-20T17:27:45.625 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abbc6488:::benchmark_data_smithi171_122772_object106336:head# 2024-04-20T17:27:45.626 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abbebf90:::benchmark_data_smithi171_122772_object168046:head# 2024-04-20T17:27:45.626 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abbee1d1:::benchmark_data_smithi171_122772_object152848:head# 2024-04-20T17:27:45.627 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abbf6149:::benchmark_data_smithi171_122772_object136908:head# 2024-04-20T17:27:45.627 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abbfe252:::benchmark_data_smithi171_122772_object106833:head# 2024-04-20T17:27:45.628 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abc028f9:::benchmark_data_smithi171_122772_object170138:head# 2024-04-20T17:27:45.628 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abc030c1:::benchmark_data_smithi171_122772_object98738:head# 2024-04-20T17:27:45.628 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abc0a9fd:::benchmark_data_smithi171_122772_object124582:head# 2024-04-20T17:27:45.629 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abc0aa29:::benchmark_data_smithi171_122772_object99950:head# 2024-04-20T17:27:45.629 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abc0d490:::benchmark_data_smithi171_122772_object107441:head# 2024-04-20T17:27:45.630 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abc1ec32:::benchmark_data_smithi171_122772_object142522:head# 2024-04-20T17:27:45.630 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abc1f50b:::benchmark_data_smithi171_122772_object99090:head# 2024-04-20T17:27:45.631 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abc23368:::benchmark_data_smithi171_122772_object157942:head# 2024-04-20T17:27:45.631 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abc531d1:::benchmark_data_smithi171_122772_object115189:head# 2024-04-20T17:27:45.632 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abc57b93:::benchmark_data_smithi171_122772_object152823:head# 2024-04-20T17:27:45.632 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abc5a041:::benchmark_data_smithi171_122772_object158706:head# 2024-04-20T17:27:45.633 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abc743ce:::benchmark_data_smithi171_122772_object91782:head# 2024-04-20T17:27:45.633 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abc7961c:::benchmark_data_smithi171_122772_object94370:head# 2024-04-20T17:27:45.634 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abc7ef81:::benchmark_data_smithi171_122772_object145975:head# 2024-04-20T17:27:45.634 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abc88493:::benchmark_data_smithi171_122772_object79763:head# 2024-04-20T17:27:45.635 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abca336d:::benchmark_data_smithi171_122772_object79732:head# 2024-04-20T17:27:45.635 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abcb9b37:::benchmark_data_smithi171_122772_object152163:head# 2024-04-20T17:27:45.636 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abcbb150:::benchmark_data_smithi171_122772_object140933:head# 2024-04-20T17:27:45.636 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abcc815a:::benchmark_data_smithi171_122772_object97182:head# 2024-04-20T17:27:45.637 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abccf3b7:::benchmark_data_smithi171_122772_object165699:head# 2024-04-20T17:27:45.637 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abcd5738:::benchmark_data_smithi171_122772_object105215:head# 2024-04-20T17:27:45.638 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abce7800:::benchmark_data_smithi171_122772_object167018:head# 2024-04-20T17:27:45.639 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abce7ef9:::benchmark_data_smithi171_122772_object77961:head# 2024-04-20T17:27:45.639 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abcfe117:::benchmark_data_smithi171_122772_object146703:head# 2024-04-20T17:27:45.639 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abd1974a:::benchmark_data_smithi171_122772_object163869:head# 2024-04-20T17:27:45.640 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abd1e7a1:::benchmark_data_smithi171_122772_object158465:head# 2024-04-20T17:27:45.640 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abd28685:::benchmark_data_smithi171_122772_object156438:head# 2024-04-20T17:27:45.641 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abd2adfb:::benchmark_data_smithi171_122772_object121495:head# 2024-04-20T17:27:45.641 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abd3c5d7:::benchmark_data_smithi171_122772_object100889:head# 2024-04-20T17:27:45.641 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abd43b71:::benchmark_data_smithi171_122772_object160170:head# 2024-04-20T17:27:45.642 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abd64ffa:::benchmark_data_smithi171_122772_object78877:head# 2024-04-20T17:27:45.642 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abd66211:::benchmark_data_smithi171_122772_object79926:head# 2024-04-20T17:27:45.643 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abd66758:::benchmark_data_smithi171_122772_object141872:head# 2024-04-20T17:27:45.643 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abd6dd79:::benchmark_data_smithi171_122772_object95228:head# 2024-04-20T17:27:45.644 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abd6ee52:::benchmark_data_smithi171_122772_object117300:head# 2024-04-20T17:27:45.644 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abd76480:::benchmark_data_smithi171_122772_object111278:head# 2024-04-20T17:27:45.645 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abd89813:::benchmark_data_smithi171_122772_object136658:head# 2024-04-20T17:27:45.645 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abd91102:::benchmark_data_smithi171_122772_object145389:head# 2024-04-20T17:27:45.646 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abd9ca66:::benchmark_data_smithi171_122772_object83858:head# 2024-04-20T17:27:45.647 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abd9e4b6:::benchmark_data_smithi171_122772_object133654:head# 2024-04-20T17:27:45.647 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abda15ed:::benchmark_data_smithi171_122772_object153731:head# 2024-04-20T17:27:45.647 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abda61f5:::benchmark_data_smithi171_122772_object100793:head# 2024-04-20T17:27:45.648 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abda8df3:::benchmark_data_smithi171_122772_object82738:head# 2024-04-20T17:27:45.648 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abdbab89:::benchmark_data_smithi171_122772_object137027:head# 2024-04-20T17:27:45.649 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abdd9c5b:::benchmark_data_smithi171_122772_object126191:head# 2024-04-20T17:27:45.649 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abdda621:::benchmark_data_smithi171_122772_object133340:head# 2024-04-20T17:27:45.649 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abe0a0bc:::benchmark_data_smithi171_122772_object136800:head# 2024-04-20T17:27:45.650 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abe12c89:::benchmark_data_smithi171_122772_object125660:head# 2024-04-20T17:27:45.650 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abe2496f:::benchmark_data_smithi171_122772_object101129:head# 2024-04-20T17:27:45.651 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abe2e866:::benchmark_data_smithi171_122772_object166205:head# 2024-04-20T17:27:45.651 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abe3fa1c:::benchmark_data_smithi171_122772_object115194:head# 2024-04-20T17:27:45.652 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abe3faaa:::benchmark_data_smithi171_122772_object94197:head# 2024-04-20T17:27:45.652 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abe44c69:::benchmark_data_smithi171_122772_object114902:head# 2024-04-20T17:27:45.652 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abe5e38c:::benchmark_data_smithi171_122772_object111971:head# 2024-04-20T17:27:45.653 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abe71d67:::benchmark_data_smithi171_122772_object150639:head# 2024-04-20T17:27:45.653 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abe76ff1:::benchmark_data_smithi171_122772_object114340:head# 2024-04-20T17:27:45.654 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abe7c504:::benchmark_data_smithi171_122772_object110161:head# 2024-04-20T17:27:45.654 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abe7eb78:::benchmark_data_smithi171_122772_object126911:head# 2024-04-20T17:27:45.654 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abeb44f1:::benchmark_data_smithi171_122772_object168542:head# 2024-04-20T17:27:45.655 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abeb4869:::benchmark_data_smithi171_122772_object101954:head# 2024-04-20T17:27:45.656 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abec8c94:::benchmark_data_smithi171_122772_object91581:head# 2024-04-20T17:27:45.656 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abece7b3:::benchmark_data_smithi171_122772_object96825:head# 2024-04-20T17:27:45.656 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abece85f:::benchmark_data_smithi171_122772_object112584:head# 2024-04-20T17:27:45.657 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abed7d0b:::benchmark_data_smithi171_122772_object102312:head# 2024-04-20T17:27:45.657 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abedc8cb:::benchmark_data_smithi171_122772_object134887:head# 2024-04-20T17:27:45.658 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abef1011:::benchmark_data_smithi171_122772_object115758:head# 2024-04-20T17:27:45.658 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abf01953:::benchmark_data_smithi171_122772_object151993:head# 2024-04-20T17:27:45.659 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abf139a1:::benchmark_data_smithi171_122772_object157761:head# 2024-04-20T17:27:45.659 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abf18dae:::benchmark_data_smithi171_122772_object140192:head# 2024-04-20T17:27:45.659 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abf1cf7e:::benchmark_data_smithi171_122772_object113715:head# 2024-04-20T17:27:45.660 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abf30fae:::benchmark_data_smithi171_122772_object120188:head# 2024-04-20T17:27:45.660 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abf403aa:::benchmark_data_smithi171_122772_object119484:head# 2024-04-20T17:27:45.660 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abf46632:::benchmark_data_smithi171_122772_object154191:head# 2024-04-20T17:27:45.661 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abf5f0ff:::benchmark_data_smithi171_122772_object82049:head# 2024-04-20T17:27:45.661 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abf64d21:::benchmark_data_smithi171_122772_object161995:head# 2024-04-20T17:27:45.662 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abf68ce2:::benchmark_data_smithi171_122772_object111228:head# 2024-04-20T17:27:45.662 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abf6adbc:::benchmark_data_smithi171_122772_object127587:head# 2024-04-20T17:27:45.663 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abf89656:::benchmark_data_smithi171_122772_object128940:head# 2024-04-20T17:27:45.663 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abf89b7c:::benchmark_data_smithi171_122772_object99456:head# 2024-04-20T17:27:45.663 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abfa5995:::benchmark_data_smithi171_122772_object106068:head# 2024-04-20T17:27:45.664 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abfbd1cf:::benchmark_data_smithi171_122772_object100400:head# 2024-04-20T17:27:45.664 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abfcdef0:::benchmark_data_smithi171_122772_object103763:head# 2024-04-20T17:27:45.665 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abfd1782:::benchmark_data_smithi171_122772_object147828:head# 2024-04-20T17:27:45.665 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abfd6a5a:::benchmark_data_smithi171_122772_object160514:head# 2024-04-20T17:27:45.666 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abfd8071:::benchmark_data_smithi171_122772_object150991:head# 2024-04-20T17:27:45.666 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abfd998b:::benchmark_data_smithi171_122772_object94608:head# 2024-04-20T17:27:45.666 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abfdeebb:::benchmark_data_smithi171_122772_object121902:head# 2024-04-20T17:27:45.667 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:abfeb73e:::benchmark_data_smithi171_122772_object103647:head# 2024-04-20T17:27:45.668 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac0061dc:::benchmark_data_smithi171_122772_object98081:head# 2024-04-20T17:27:45.668 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac00931c:::benchmark_data_smithi171_122772_object114712:head# 2024-04-20T17:27:45.668 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac02116c:::benchmark_data_smithi171_122772_object79562:head# 2024-04-20T17:27:45.669 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac02667a:::benchmark_data_smithi171_122772_object156004:head# 2024-04-20T17:27:45.669 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac028c7c:::benchmark_data_smithi171_122772_object111706:head# 2024-04-20T17:27:45.670 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac03e49c:::benchmark_data_smithi171_122772_object163174:head# 2024-04-20T17:27:45.670 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac03f798:::benchmark_data_smithi171_122772_object146017:head# 2024-04-20T17:27:45.671 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac045fdd:::benchmark_data_smithi171_122772_object160656:head# 2024-04-20T17:27:45.671 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac06e514:::benchmark_data_smithi171_122772_object89050:head# 2024-04-20T17:27:45.672 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac07570e:::benchmark_data_smithi171_122772_object114017:head# 2024-04-20T17:27:45.672 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac076b88:::benchmark_data_smithi171_122772_object152577:head# 2024-04-20T17:27:45.672 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac078dd5:::benchmark_data_smithi171_122772_object99682:head# 2024-04-20T17:27:45.672 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac09f811:::benchmark_data_smithi171_122772_object112704:head# 2024-04-20T17:27:45.673 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac0a253a:::benchmark_data_smithi171_122772_object113252:head# 2024-04-20T17:27:45.673 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac0a7593:::benchmark_data_smithi171_122772_object155663:head# 2024-04-20T17:27:45.674 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac0aa761:::benchmark_data_smithi171_122772_object92422:head# 2024-04-20T17:27:45.674 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac0aefde:::benchmark_data_smithi171_122772_object154391:head# 2024-04-20T17:27:45.675 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac0d867d:::benchmark_data_smithi171_122772_object78180:head# 2024-04-20T17:27:45.675 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac0feff6:::benchmark_data_smithi171_122772_object93100:head# 2024-04-20T17:27:45.676 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac1061fa:::benchmark_data_smithi171_122772_object122629:head# 2024-04-20T17:27:45.676 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac10daed:::benchmark_data_smithi171_122772_object150887:head# 2024-04-20T17:27:45.676 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac113583:::benchmark_data_smithi171_122772_object115205:head# 2024-04-20T17:27:45.678 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac12ae96:::benchmark_data_smithi171_122772_object124141:head# 2024-04-20T17:27:45.678 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac13d459:::benchmark_data_smithi171_122772_object88474:head# 2024-04-20T17:27:45.678 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac13dfa2:::benchmark_data_smithi171_122772_object169693:head# 2024-04-20T17:27:45.679 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac14a74a:::benchmark_data_smithi171_122772_object148004:head# 2024-04-20T17:27:45.679 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac150690:::benchmark_data_smithi171_122772_object120125:head# 2024-04-20T17:27:45.679 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac15b05b:::benchmark_data_smithi171_122772_object102141:head# 2024-04-20T17:27:45.680 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac1810bb:::benchmark_data_smithi171_122772_object91525:head# 2024-04-20T17:27:45.680 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac189579:::benchmark_data_smithi171_122772_object149575:head# 2024-04-20T17:27:45.680 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac18cf3a:::benchmark_data_smithi171_122772_object121978:head# 2024-04-20T17:27:45.681 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac1935ef:::benchmark_data_smithi171_122772_object111492:head# 2024-04-20T17:27:45.682 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac1963e4:::benchmark_data_smithi171_122772_object95275:head# 2024-04-20T17:27:45.682 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac19c2c3:::benchmark_data_smithi171_122772_object144187:head# 2024-04-20T17:27:45.682 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac1cb89d:::benchmark_data_smithi171_122772_object111466:head# 2024-04-20T17:27:45.683 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac1cd6cd:::benchmark_data_smithi171_122772_object104361:head# 2024-04-20T17:27:45.683 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac1d200d:::benchmark_data_smithi171_122772_object137462:head# 2024-04-20T17:27:45.684 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac1d2a92:::benchmark_data_smithi171_122772_object82270:head# 2024-04-20T17:27:45.684 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac1d2dd2:::benchmark_data_smithi171_122772_object83928:head# 2024-04-20T17:27:45.684 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac1e047c:::benchmark_data_smithi171_122772_object150147:head# 2024-04-20T17:27:45.685 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac1e6f29:::benchmark_data_smithi171_122772_object159020:head# 2024-04-20T17:27:45.685 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac1ea0ed:::benchmark_data_smithi171_122772_object144288:head# 2024-04-20T17:27:45.685 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac1f68fe:::benchmark_data_smithi171_122772_object113069:head# 2024-04-20T17:27:45.686 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac1f6901:::benchmark_data_smithi171_122772_object94328:head# 2024-04-20T17:27:45.686 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac1f9256:::benchmark_data_smithi171_122772_object131338:head# 2024-04-20T17:27:45.687 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac20718e:::benchmark_data_smithi171_122772_object167644:head# 2024-04-20T17:27:45.688 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac2086d9:::benchmark_data_smithi171_122772_object131843:head# 2024-04-20T17:27:45.688 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac2183e6:::benchmark_data_smithi171_122772_object81293:head# 2024-04-20T17:27:45.688 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac21c144:::benchmark_data_smithi171_122772_object117288:head# 2024-04-20T17:27:45.689 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac2353f9:::benchmark_data_smithi171_122772_object87922:head# 2024-04-20T17:27:45.689 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac24166f:::benchmark_data_smithi171_122772_object112308:head# 2024-04-20T17:27:45.690 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac244437:::benchmark_data_smithi171_122772_object95115:head# 2024-04-20T17:27:45.690 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac24b47d:::benchmark_data_smithi171_122772_object99685:head# 2024-04-20T17:27:45.691 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac24f001:::benchmark_data_smithi171_122772_object131276:head# 2024-04-20T17:27:45.691 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac2505f4:::benchmark_data_smithi171_122772_object99331:head# 2024-04-20T17:27:45.691 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac26921e:::benchmark_data_smithi171_122772_object152159:head# 2024-04-20T17:27:45.692 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:45 smithi005 ceph-mon[39562]: pgmap v1353: 17 pgs: 2 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 5.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 3774/184372 objects degraded (2.047%); 69 MiB/s, 1.10k objects/s recovering 2024-04-20T17:27:45.692 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:45 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3938960208' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:27:45.693 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac280429:::benchmark_data_smithi171_122772_object154488:head# 2024-04-20T17:27:45.693 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac28d4ff:::benchmark_data_smithi171_122772_object139973:head# 2024-04-20T17:27:45.693 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac2945a6:::benchmark_data_smithi171_122772_object137419:head# 2024-04-20T17:27:45.694 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac296ce5:::benchmark_data_smithi171_122772_object99909:head# 2024-04-20T17:27:45.694 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac2997d7:::benchmark_data_smithi171_122772_object94925:head# 2024-04-20T17:27:45.694 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac2bb09f:::benchmark_data_smithi171_122772_object145284:head# 2024-04-20T17:27:45.694 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac2d0d57:::benchmark_data_smithi171_122772_object143819:head# 2024-04-20T17:27:45.695 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac2e6525:::benchmark_data_smithi171_122772_object139371:head# 2024-04-20T17:27:45.695 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac2ec574:::benchmark_data_smithi171_122772_object154260:head# 2024-04-20T17:27:45.696 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac319634:::benchmark_data_smithi171_122772_object109591:head# 2024-04-20T17:27:45.696 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac31b53a:::benchmark_data_smithi171_122772_object104039:head# 2024-04-20T17:27:45.696 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac321885:::benchmark_data_smithi171_122772_object107430:head# 2024-04-20T17:27:45.697 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac33088c:::benchmark_data_smithi171_122772_object143585:head# 2024-04-20T17:27:45.697 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac33a064:::benchmark_data_smithi171_122772_object168012:head# 2024-04-20T17:27:45.698 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac33b137:::benchmark_data_smithi171_122772_object93394:head# 2024-04-20T17:27:45.698 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac354c68:::benchmark_data_smithi171_122772_object164211:head# 2024-04-20T17:27:45.699 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac36cd2e:::benchmark_data_smithi171_122772_object150188:head# 2024-04-20T17:27:45.699 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac3724c8:::benchmark_data_smithi171_122772_object136207:head# 2024-04-20T17:27:45.699 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac37cc5b:::benchmark_data_smithi171_122772_object167911:head# 2024-04-20T17:27:45.700 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac37e3d4:::benchmark_data_smithi171_122772_object102401:head# 2024-04-20T17:27:45.700 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac3837d7:::benchmark_data_smithi171_122772_object110600:head# 2024-04-20T17:27:45.701 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac3a1397:::benchmark_data_smithi171_122772_object96054:head# 2024-04-20T17:27:45.701 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac3af310:::benchmark_data_smithi171_122772_object86191:head# 2024-04-20T17:27:45.701 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac3ced0f:::benchmark_data_smithi171_122772_object92229:head# 2024-04-20T17:27:45.702 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac3d8605:::benchmark_data_smithi171_122772_object110826:head# 2024-04-20T17:27:45.702 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac402cd3:::benchmark_data_smithi171_122772_object107345:head# 2024-04-20T17:27:45.703 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac4080b9:::benchmark_data_smithi171_122772_object137319:head# 2024-04-20T17:27:45.703 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac413e5c:::benchmark_data_smithi171_122772_object109423:head# 2024-04-20T17:27:45.704 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac427622:::benchmark_data_smithi171_122772_object80345:head# 2024-04-20T17:27:45.704 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac43117f:::benchmark_data_smithi171_122772_object107172:head# 2024-04-20T17:27:45.705 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac432ebd:::benchmark_data_smithi171_122772_object132059:head# 2024-04-20T17:27:45.705 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac438947:::benchmark_data_smithi171_122772_object125039:head# 2024-04-20T17:27:45.706 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac4439d8:::benchmark_data_smithi171_122772_object113119:head# 2024-04-20T17:27:45.706 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac44a663:::benchmark_data_smithi171_122772_object111130:head# 2024-04-20T17:27:45.706 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac45cb55:::benchmark_data_smithi171_122772_object124323:head# 2024-04-20T17:27:45.707 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac475f88:::benchmark_data_smithi171_122772_object156896:head# 2024-04-20T17:27:45.707 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac4850f3:::benchmark_data_smithi171_122772_object150856:head# 2024-04-20T17:27:45.708 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac485da1:::benchmark_data_smithi171_122772_object117910:head# 2024-04-20T17:27:45.708 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac48b6b9:::benchmark_data_smithi171_122772_object162510:head# 2024-04-20T17:27:45.708 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac4906fc:::benchmark_data_smithi171_122772_object105978:head# 2024-04-20T17:27:45.709 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac497840:::benchmark_data_smithi171_122772_object131168:head# 2024-04-20T17:27:45.709 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac49edc6:::benchmark_data_smithi171_122772_object169238:head# 2024-04-20T17:27:45.710 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac49f234:::benchmark_data_smithi171_122772_object145816:head# 2024-04-20T17:27:45.710 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac4ab17f:::benchmark_data_smithi171_122772_object115421:head# 2024-04-20T17:27:45.711 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac4bab2f:::benchmark_data_smithi171_122772_object154324:head# 2024-04-20T17:27:45.711 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac4e206d:::benchmark_data_smithi171_122772_object169936:head# 2024-04-20T17:27:45.712 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac4e9e9b:::benchmark_data_smithi171_122772_object92256:head# 2024-04-20T17:27:45.712 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac4f211b:::benchmark_data_smithi171_122772_object90774:head# 2024-04-20T17:27:45.712 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac4f8056:::benchmark_data_smithi171_122772_object94650:head# 2024-04-20T17:27:45.713 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac501636:::benchmark_data_smithi171_122772_object154963:head# 2024-04-20T17:27:45.713 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac50f589:::benchmark_data_smithi171_122772_object145361:head# 2024-04-20T17:27:45.714 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac517780:::benchmark_data_smithi171_122772_object137157:head# 2024-04-20T17:27:45.714 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac527384:::benchmark_data_smithi171_122772_object139253:head# 2024-04-20T17:27:45.714 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac52dea4:::benchmark_data_smithi171_122772_object138571:head# 2024-04-20T17:27:45.715 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac530afe:::benchmark_data_smithi171_122772_object116398:head# 2024-04-20T17:27:45.716 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac5341f5:::benchmark_data_smithi171_122772_object145702:head# 2024-04-20T17:27:45.716 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac5352c1:::benchmark_data_smithi171_122772_object106012:head# 2024-04-20T17:27:45.717 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac549d75:::benchmark_data_smithi171_122772_object108630:head# 2024-04-20T17:27:45.717 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac54a9e3:::benchmark_data_smithi171_122772_object142952:head# 2024-04-20T17:27:45.717 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac5543eb:::benchmark_data_smithi171_122772_object142717:head# 2024-04-20T17:27:45.718 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac556681:::benchmark_data_smithi171_122772_object129573:head# 2024-04-20T17:27:45.718 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac556f12:::benchmark_data_smithi171_122772_object112959:head# 2024-04-20T17:27:45.719 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac557f37:::benchmark_data_smithi171_122772_object102900:head# 2024-04-20T17:27:45.720 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac55e18b:::benchmark_data_smithi171_122772_object80202:head# 2024-04-20T17:27:45.720 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac55f1e4:::benchmark_data_smithi171_122772_object110258:head# 2024-04-20T17:27:45.720 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac5774f2:::benchmark_data_smithi171_122772_object144432:head# 2024-04-20T17:27:45.720 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac580cd7:::benchmark_data_smithi171_122772_object114600:head# 2024-04-20T17:27:45.721 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac589ead:::benchmark_data_smithi171_122772_object95873:head# 2024-04-20T17:27:45.721 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac59dce8:::benchmark_data_smithi171_122772_object166536:head# 2024-04-20T17:27:45.721 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac5abac8:::benchmark_data_smithi171_122772_object120537:head# 2024-04-20T17:27:45.722 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac5ac502:::benchmark_data_smithi171_122772_object120900:head# 2024-04-20T17:27:45.722 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac5c86de:::benchmark_data_smithi171_122772_object87097:head# 2024-04-20T17:27:45.723 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac5caf46:::benchmark_data_smithi171_122772_object148528:head# 2024-04-20T17:27:45.723 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac5cf957:::benchmark_data_smithi171_122772_object166830:head# 2024-04-20T17:27:45.724 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac5d2b32:::benchmark_data_smithi171_122772_object112407:head# 2024-04-20T17:27:45.724 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac5e83d4:::benchmark_data_smithi171_122772_object79555:head# 2024-04-20T17:27:45.725 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac5fa20d:::benchmark_data_smithi171_122772_object148426:head# 2024-04-20T17:27:45.725 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac6048e6:::benchmark_data_smithi171_122772_object111655:head# 2024-04-20T17:27:45.726 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac60f477:::benchmark_data_smithi171_122772_object161969:head# 2024-04-20T17:27:45.726 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac60f976:::benchmark_data_smithi171_122772_object144804:head# 2024-04-20T17:27:45.727 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac611884:::benchmark_data_smithi171_122772_object105672:head# 2024-04-20T17:27:45.727 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac6198a8:::benchmark_data_smithi171_122772_object111523:head# 2024-04-20T17:27:45.727 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac61f663:::benchmark_data_smithi171_122772_object157773:head# 2024-04-20T17:27:45.728 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac6293f3:::benchmark_data_smithi171_122772_object141815:head# 2024-04-20T17:27:45.728 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac642e24:::benchmark_data_smithi171_122772_object103265:head# 2024-04-20T17:27:45.728 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac644244:::benchmark_data_smithi171_122772_object94631:head# 2024-04-20T17:27:45.729 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac66b745:::benchmark_data_smithi171_122772_object108536:head# 2024-04-20T17:27:45.729 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac6792ab:::benchmark_data_smithi171_122772_object84954:head# 2024-04-20T17:27:45.730 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac67c113:::benchmark_data_smithi171_122772_object129285:head# 2024-04-20T17:27:45.730 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac68d819:::benchmark_data_smithi171_122772_object119246:head# 2024-04-20T17:27:45.731 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac692e67:::benchmark_data_smithi171_122772_object98472:head# 2024-04-20T17:27:45.731 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac69e260:::benchmark_data_smithi171_122772_object98905:head# 2024-04-20T17:27:45.732 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac6a2fb1:::benchmark_data_smithi171_122772_object137175:head# 2024-04-20T17:27:45.732 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac6a6731:::benchmark_data_smithi171_122772_object137778:head# 2024-04-20T17:27:45.733 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac6a9077:::benchmark_data_smithi171_122772_object110226:head# 2024-04-20T17:27:45.734 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac6b04b5:::benchmark_data_smithi171_122772_object131423:head# 2024-04-20T17:27:45.734 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac6b981b:::benchmark_data_smithi171_122772_object97802:head# 2024-04-20T17:27:45.734 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac6c5767:::benchmark_data_smithi171_122772_object144902:head# 2024-04-20T17:27:45.734 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac6cc587:::benchmark_data_smithi171_122772_object106834:head# 2024-04-20T17:27:45.735 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac6edbe4:::benchmark_data_smithi171_122772_object160254:head# 2024-04-20T17:27:45.736 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac6f9e66:::benchmark_data_smithi171_122772_object139145:head# 2024-04-20T17:27:45.736 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac6fbd97:::benchmark_data_smithi171_122772_object165665:head# 2024-04-20T17:27:45.736 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac6fe56c:::benchmark_data_smithi171_122772_object155577:head# 2024-04-20T17:27:45.736 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac701ad3:::benchmark_data_smithi171_122772_object150934:head# 2024-04-20T17:27:45.737 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac7167c8:::benchmark_data_smithi171_122772_object156736:head# 2024-04-20T17:27:45.737 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac72fc82:::benchmark_data_smithi171_122772_object144498:head# 2024-04-20T17:27:45.738 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac73254d:::benchmark_data_smithi171_122772_object93149:head# 2024-04-20T17:27:45.739 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac737fe7:::benchmark_data_smithi171_122772_object94418:head# 2024-04-20T17:27:45.739 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac74339a:::benchmark_data_smithi171_122772_object105218:head# 2024-04-20T17:27:45.739 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac74d505:::benchmark_data_smithi171_122772_object121482:head# 2024-04-20T17:27:45.740 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac756fad:::benchmark_data_smithi171_122772_object117622:head# 2024-04-20T17:27:45.740 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac765296:::benchmark_data_smithi171_122772_object152777:head# 2024-04-20T17:27:45.740 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac76ade9:::benchmark_data_smithi171_122772_object90401:head# 2024-04-20T17:27:45.741 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac7766b0:::benchmark_data_smithi171_122772_object95508:head# 2024-04-20T17:27:45.742 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac781eeb:::benchmark_data_smithi171_122772_object77958:head# 2024-04-20T17:27:45.742 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac789d9b:::benchmark_data_smithi171_122772_object94458:head# 2024-04-20T17:27:45.743 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac79bada:::benchmark_data_smithi171_122772_object87343:head# 2024-04-20T17:27:45.743 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac7c9508:::benchmark_data_smithi171_122772_object129397:head# 2024-04-20T17:27:45.747 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac7d534b:::benchmark_data_smithi171_122772_object104537:head# 2024-04-20T17:27:45.747 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac7d9a64:::benchmark_data_smithi171_122772_object134517:head# 2024-04-20T17:27:45.748 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac7dad6e:::benchmark_data_smithi171_122772_object123120:head# 2024-04-20T17:27:45.748 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac7de6ad:::benchmark_data_smithi171_122772_object100185:head# 2024-04-20T17:27:45.748 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac7f8be9:::benchmark_data_smithi171_122772_object152301:head# 2024-04-20T17:27:45.748 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac8028e6:::benchmark_data_smithi171_122772_object93165:head# 2024-04-20T17:27:45.748 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac806cc3:::benchmark_data_smithi171_122772_object98209:head# 2024-04-20T17:27:45.748 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac80e89e:::benchmark_data_smithi171_122772_object84383:head# 2024-04-20T17:27:45.748 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac80ec05:::benchmark_data_smithi171_122772_object123846:head# 2024-04-20T17:27:45.748 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac8171ce:::benchmark_data_smithi171_122772_object165260:head# 2024-04-20T17:27:45.749 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac81e239:::benchmark_data_smithi171_122772_object122219:head# 2024-04-20T17:27:45.749 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac8206ea:::benchmark_data_smithi171_122772_object123857:head# 2024-04-20T17:27:45.749 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac82ea61:::benchmark_data_smithi171_122772_object80302:head# 2024-04-20T17:27:45.749 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac839033:::benchmark_data_smithi171_122772_object120600:head# 2024-04-20T17:27:45.750 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac843c1a:::benchmark_data_smithi171_122772_object109972:head# 2024-04-20T17:27:45.750 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac854c75:::benchmark_data_smithi171_122772_object161018:head# 2024-04-20T17:27:45.751 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac85e6a9:::benchmark_data_smithi171_122772_object165753:head# 2024-04-20T17:27:45.751 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac8631f6:::benchmark_data_smithi171_122772_object89693:head# 2024-04-20T17:27:45.752 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac86e191:::benchmark_data_smithi171_122772_object106722:head# 2024-04-20T17:27:45.752 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac87a659:::benchmark_data_smithi171_122772_object104413:head# 2024-04-20T17:27:45.752 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac8840ae:::benchmark_data_smithi171_122772_object95784:head# 2024-04-20T17:27:45.753 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac88d2b2:::benchmark_data_smithi171_122772_object143829:head# 2024-04-20T17:27:45.753 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac892c61:::benchmark_data_smithi171_122772_object133344:head# 2024-04-20T17:27:45.754 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac89f58e:::benchmark_data_smithi171_122772_object90166:head# 2024-04-20T17:27:45.754 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac8a4c34:::benchmark_data_smithi171_122772_object169259:head# 2024-04-20T17:27:45.755 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac8aa412:::benchmark_data_smithi171_122772_object155179:head# 2024-04-20T17:27:45.755 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac8b0ae4:::benchmark_data_smithi171_122772_object122223:head# 2024-04-20T17:27:45.756 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac8c7592:::benchmark_data_smithi171_122772_object165744:head# 2024-04-20T17:27:45.756 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac8c994e:::benchmark_data_smithi171_122772_object167144:head# 2024-04-20T17:27:45.757 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac8d3ae0:::benchmark_data_smithi171_122772_object157862:head# 2024-04-20T17:27:45.758 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac8d526d:::benchmark_data_smithi171_122772_object106664:head# 2024-04-20T17:27:45.758 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac8d7c34:::benchmark_data_smithi171_122772_object136833:head# 2024-04-20T17:27:45.759 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac8dba9a:::benchmark_data_smithi171_122772_object161452:head# 2024-04-20T17:27:45.759 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac8f4c26:::benchmark_data_smithi171_122772_object111325:head# 2024-04-20T17:27:45.760 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac8f6f00:::benchmark_data_smithi171_122772_object137555:head# 2024-04-20T17:27:45.760 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac903ed1:::benchmark_data_smithi171_122772_object146610:head# 2024-04-20T17:27:45.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:45 smithi106 ceph-mon[44423]: pgmap v1353: 17 pgs: 2 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 5.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 3774/184372 objects degraded (2.047%); 69 MiB/s, 1.10k objects/s recovering 2024-04-20T17:27:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:45 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3938960208' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:27:45.761 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac9098c0:::benchmark_data_smithi171_122772_object123957:head# 2024-04-20T17:27:45.761 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac9122a9:::benchmark_data_smithi171_122772_object110331:head# 2024-04-20T17:27:45.762 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac912561:::benchmark_data_smithi171_122772_object144128:head# 2024-04-20T17:27:45.762 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac913728:::benchmark_data_smithi171_122772_object156083:head# 2024-04-20T17:27:45.763 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac92a2ea:::benchmark_data_smithi171_122772_object89006:head# 2024-04-20T17:27:45.763 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac92dfed:::benchmark_data_smithi171_122772_object118576:head# 2024-04-20T17:27:45.763 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac93a566:::benchmark_data_smithi171_122772_object127017:head# 2024-04-20T17:27:45.764 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac96e3e4:::benchmark_data_smithi171_122772_object144916:head# 2024-04-20T17:27:45.764 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac96f488:::benchmark_data_smithi171_122772_object96848:head# 2024-04-20T17:27:45.765 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac97e2cf:::benchmark_data_smithi171_122772_object127748:head# 2024-04-20T17:27:45.766 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac9aeaa0:::benchmark_data_smithi171_122772_object101351:head# 2024-04-20T17:27:45.766 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac9bd810:::benchmark_data_smithi171_122772_object91776:head# 2024-04-20T17:27:45.767 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac9be2a5:::benchmark_data_smithi171_122772_object92640:head# 2024-04-20T17:27:45.767 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac9c155a:::benchmark_data_smithi171_122772_object164655:head# 2024-04-20T17:27:45.768 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac9c416e:::benchmark_data_smithi171_122772_object83583:head# 2024-04-20T17:27:45.768 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ac9d9b0b:::benchmark_data_smithi171_122772_object94112:head# 2024-04-20T17:27:45.769 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aca0a8e2:::benchmark_data_smithi171_122772_object105940:head# 2024-04-20T17:27:45.770 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aca12f78:::benchmark_data_smithi171_122772_object96417:head# 2024-04-20T17:27:45.770 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aca17065:::benchmark_data_smithi171_122772_object164968:head# 2024-04-20T17:27:45.771 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aca2a28d:::benchmark_data_smithi171_122772_object155493:head# 2024-04-20T17:27:45.771 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aca32aec:::benchmark_data_smithi171_122772_object161590:head# 2024-04-20T17:27:45.771 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aca5d00c:::benchmark_data_smithi171_122772_object108738:head# 2024-04-20T17:27:45.772 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aca5e196:::benchmark_data_smithi171_122772_object98306:head# 2024-04-20T17:27:45.772 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aca66769:::benchmark_data_smithi171_122772_object92225:head# 2024-04-20T17:27:45.773 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aca7911b:::benchmark_data_smithi171_122772_object153057:head# 2024-04-20T17:27:45.773 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aca7d9b3:::benchmark_data_smithi171_122772_object96025:head# 2024-04-20T17:27:45.774 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aca874bf:::benchmark_data_smithi171_122772_object138474:head# 2024-04-20T17:27:45.774 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acab3c8e:::benchmark_data_smithi171_122772_object147897:head# 2024-04-20T17:27:45.774 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acab88db:::benchmark_data_smithi171_122772_object140568:head# 2024-04-20T17:27:45.775 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acaba81c:::benchmark_data_smithi171_122772_object158698:head# 2024-04-20T17:27:45.775 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acac137d:::benchmark_data_smithi171_122772_object149256:head# 2024-04-20T17:27:45.776 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acaccddc:::benchmark_data_smithi171_122772_object94990:head# 2024-04-20T17:27:45.777 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acad90a9:::benchmark_data_smithi171_122772_object167071:head# 2024-04-20T17:27:45.777 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acadc151:::benchmark_data_smithi171_122772_object169661:head# 2024-04-20T17:27:45.778 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acae98ea:::benchmark_data_smithi171_122772_object158003:head# 2024-04-20T17:27:45.778 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acaece2b:::benchmark_data_smithi171_122772_object81305:head# 2024-04-20T17:27:45.779 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acaf0b60:::benchmark_data_smithi171_122772_object130206:head# 2024-04-20T17:27:45.779 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acaf36e9:::benchmark_data_smithi171_122772_object122813:head# 2024-04-20T17:27:45.780 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acaf6770:::benchmark_data_smithi171_122772_object80371:head# 2024-04-20T17:27:45.780 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb16eef:::benchmark_data_smithi171_122772_object116950:head# 2024-04-20T17:27:45.781 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb1d044:::benchmark_data_smithi171_122772_object116249:head# 2024-04-20T17:27:45.781 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb1f10b:::benchmark_data_smithi171_122772_object121595:head# 2024-04-20T17:27:45.782 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb2414f:::benchmark_data_smithi171_122772_object89272:head# 2024-04-20T17:27:45.782 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb29ed5:::benchmark_data_smithi171_122772_object130791:head# 2024-04-20T17:27:45.783 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb2ef61:::benchmark_data_smithi171_122772_object143344:head# 2024-04-20T17:27:45.784 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb2fe8a:::benchmark_data_smithi171_122772_object153780:head# 2024-04-20T17:27:45.784 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb310f6:::benchmark_data_smithi171_122772_object131402:head# 2024-04-20T17:27:45.784 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb361fc:::benchmark_data_smithi171_122772_object139322:head# 2024-04-20T17:27:45.785 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb41368:::benchmark_data_smithi171_122772_object106476:head# 2024-04-20T17:27:45.786 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb42697:::benchmark_data_smithi171_122772_object145647:head# 2024-04-20T17:27:45.786 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb4624b:::benchmark_data_smithi171_122772_object126594:head# 2024-04-20T17:27:45.787 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb49421:::benchmark_data_smithi171_122772_object156771:head# 2024-04-20T17:27:45.787 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb50c13:::benchmark_data_smithi171_122772_object116462:head# 2024-04-20T17:27:45.788 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb52e9e:::benchmark_data_smithi171_122772_object108995:head# 2024-04-20T17:27:45.788 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb597bd:::benchmark_data_smithi171_122772_object142671:head# 2024-04-20T17:27:45.789 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb60abe:::benchmark_data_smithi171_122772_object100036:head# 2024-04-20T17:27:45.789 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb643d8:::benchmark_data_smithi171_122772_object135365:head# 2024-04-20T17:27:45.790 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb692ad:::benchmark_data_smithi171_122772_object87540:head# 2024-04-20T17:27:45.790 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb6c2cb:::benchmark_data_smithi171_122772_object106596:head# 2024-04-20T17:27:45.791 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb6c995:::benchmark_data_smithi171_122772_object108151:head# 2024-04-20T17:27:45.791 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb73e48:::benchmark_data_smithi171_122772_object163179:head# 2024-04-20T17:27:45.792 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb74519:::benchmark_data_smithi171_122772_object163257:head# 2024-04-20T17:27:45.792 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb78e56:::benchmark_data_smithi171_122772_object149486:head# 2024-04-20T17:27:45.793 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb813dd:::benchmark_data_smithi171_122772_object103917:head# 2024-04-20T17:27:45.793 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb89dc5:::benchmark_data_smithi171_122772_object167381:head# 2024-04-20T17:27:45.793 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb8fa45:::benchmark_data_smithi171_122772_object105007:head# 2024-04-20T17:27:45.794 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acb9f5a1:::benchmark_data_smithi171_122772_object77998:head# 2024-04-20T17:27:45.794 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acbbad89:::benchmark_data_smithi171_122772_object122593:head# 2024-04-20T17:27:45.795 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acbbbc82:::benchmark_data_smithi171_122772_object102215:head# 2024-04-20T17:27:45.795 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acbcb502:::benchmark_data_smithi171_122772_object156704:head# 2024-04-20T17:27:45.796 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acbdd088:::benchmark_data_smithi171_122772_object104863:head# 2024-04-20T17:27:45.796 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acbde92e:::benchmark_data_smithi171_122772_object168187:head# 2024-04-20T17:27:45.797 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acbf8ed1:::benchmark_data_smithi171_122772_object155105:head# 2024-04-20T17:27:45.797 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acbff07a:::benchmark_data_smithi171_122772_object149592:head# 2024-04-20T17:27:45.798 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acc1cc6d:::benchmark_data_smithi171_122772_object143372:head# 2024-04-20T17:27:45.798 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acc41756:::benchmark_data_smithi171_122772_object107484:head# 2024-04-20T17:27:45.799 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acc486bd:::benchmark_data_smithi171_122772_object127005:head# 2024-04-20T17:27:45.799 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acc77b1b:::benchmark_data_smithi171_122772_object137722:head# 2024-04-20T17:27:45.800 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acc7a7af:::benchmark_data_smithi171_122772_object157548:head# 2024-04-20T17:27:45.800 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acc7d793:::benchmark_data_smithi171_122772_object78539:head# 2024-04-20T17:27:45.800 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acc7fce4:::benchmark_data_smithi171_122772_object126743:head# 2024-04-20T17:27:45.801 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acc86ed4:::benchmark_data_smithi171_122772_object132042:head# 2024-04-20T17:27:45.801 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acc8bb3d:::benchmark_data_smithi171_122772_object124419:head# 2024-04-20T17:27:45.802 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acc8e112:::benchmark_data_smithi171_122772_object86163:head# 2024-04-20T17:27:45.802 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acc9c908:::benchmark_data_smithi171_122772_object98318:head# 2024-04-20T17:27:45.803 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acc9de92:::benchmark_data_smithi171_122772_object115098:head# 2024-04-20T17:27:45.803 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acca7837:::benchmark_data_smithi171_122772_object136774:head# 2024-04-20T17:27:45.804 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:accbd127:::benchmark_data_smithi171_122772_object148514:head# 2024-04-20T17:27:45.804 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:accf2763:::benchmark_data_smithi171_122772_object137485:head# 2024-04-20T17:27:45.805 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:accf82df:::benchmark_data_smithi171_122772_object109043:head# 2024-04-20T17:27:45.805 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acd01bcd:::benchmark_data_smithi171_122772_object102406:head# 2024-04-20T17:27:45.806 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acd08180:::benchmark_data_smithi171_122772_object135289:head# 2024-04-20T17:27:45.806 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acd1883a:::benchmark_data_smithi171_122772_object84984:head# 2024-04-20T17:27:45.807 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acd1b078:::benchmark_data_smithi171_122772_object124445:head# 2024-04-20T17:27:45.807 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acd253c0:::benchmark_data_smithi171_122772_object79517:head# 2024-04-20T17:27:45.808 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acd2a1d6:::benchmark_data_smithi171_122772_object112104:head# 2024-04-20T17:27:45.808 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acd2b522:::benchmark_data_smithi171_122772_object148260:head# 2024-04-20T17:27:45.809 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acd360a6:::benchmark_data_smithi171_122772_object139511:head# 2024-04-20T17:27:45.809 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acd393d8:::benchmark_data_smithi171_122772_object119259:head# 2024-04-20T17:27:45.810 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acd5167c:::benchmark_data_smithi171_122772_object118735:head# 2024-04-20T17:27:45.811 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acd703c4:::benchmark_data_smithi171_122772_object125426:head# 2024-04-20T17:27:45.811 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acd8285d:::benchmark_data_smithi171_122772_object133132:head# 2024-04-20T17:27:45.812 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acd880a4:::benchmark_data_smithi171_122772_object139912:head# 2024-04-20T17:27:45.812 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acd88136:::benchmark_data_smithi171_122772_object126342:head# 2024-04-20T17:27:45.813 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acd89464:::benchmark_data_smithi171_122772_object127759:head# 2024-04-20T17:27:45.813 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acd8bc87:::benchmark_data_smithi171_122772_object79674:head# 2024-04-20T17:27:45.814 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acda4685:::benchmark_data_smithi171_122772_object131625:head# 2024-04-20T17:27:45.814 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acddd7ab:::benchmark_data_smithi171_122772_object96422:head# 2024-04-20T17:27:45.815 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ace056a1:::benchmark_data_smithi171_122772_object132647:head# 2024-04-20T17:27:45.815 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ace0587a:::benchmark_data_smithi171_122772_object103485:head# 2024-04-20T17:27:45.816 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ace1db5f:::benchmark_data_smithi171_122772_object107892:head# 2024-04-20T17:27:45.817 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ace21f63:::benchmark_data_smithi171_122772_object157722:head# 2024-04-20T17:27:45.817 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ace24dfd:::benchmark_data_smithi171_122772_object115527:head# 2024-04-20T17:27:45.817 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ace26036:::benchmark_data_smithi171_122772_object98200:head# 2024-04-20T17:27:45.817 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ace3d62f:::benchmark_data_smithi171_122772_object82063:head# 2024-04-20T17:27:45.818 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ace55360:::benchmark_data_smithi171_122772_object147020:head# 2024-04-20T17:27:45.818 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ace5efaf:::benchmark_data_smithi171_122772_object162060:head# 2024-04-20T17:27:45.819 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ace602d5:::benchmark_data_smithi171_122772_object158390:head# 2024-04-20T17:27:45.819 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ace6a8bb:::benchmark_data_smithi171_122772_object119241:head# 2024-04-20T17:27:45.820 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ace7b166:::benchmark_data_smithi171_122772_object98366:head# 2024-04-20T17:27:45.820 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ace9de52:::benchmark_data_smithi171_122772_object94603:head# 2024-04-20T17:27:45.821 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acea1a49:::benchmark_data_smithi171_122772_object135159:head# 2024-04-20T17:27:45.821 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acea3559:::benchmark_data_smithi171_122772_object139105:head# 2024-04-20T17:27:45.822 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acebaa54:::benchmark_data_smithi171_122772_object111562:head# 2024-04-20T17:27:45.822 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acebb140:::benchmark_data_smithi171_122772_object125361:head# 2024-04-20T17:27:45.823 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acebeb4c:::benchmark_data_smithi171_122772_object152851:head# 2024-04-20T17:27:45.823 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aceca955:::benchmark_data_smithi171_122772_object92740:head# 2024-04-20T17:27:45.824 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aced48fd:::benchmark_data_smithi171_122772_object86265:head# 2024-04-20T17:27:45.824 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aced6010:::benchmark_data_smithi171_122772_object169894:head# 2024-04-20T17:27:45.825 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acee2f35:::benchmark_data_smithi171_122772_object104184:head# 2024-04-20T17:27:45.825 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acf07314:::benchmark_data_smithi171_122772_object123702:head# 2024-04-20T17:27:45.825 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acf19c0b:::benchmark_data_smithi171_122772_object107389:head# 2024-04-20T17:27:45.826 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acf2c30e:::benchmark_data_smithi171_122772_object85702:head# 2024-04-20T17:27:45.826 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acf3326f:::benchmark_data_smithi171_122772_object119795:head# 2024-04-20T17:27:45.827 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acf4f5a1:::benchmark_data_smithi171_122772_object158298:head# 2024-04-20T17:27:45.827 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acf56d37:::benchmark_data_smithi171_122772_object134298:head# 2024-04-20T17:27:45.828 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acf5e68e:::benchmark_data_smithi171_122772_object93339:head# 2024-04-20T17:27:45.828 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acf6e4ff:::benchmark_data_smithi171_122772_object138866:head# 2024-04-20T17:27:45.828 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acf78341:::benchmark_data_smithi171_122772_object108042:head# 2024-04-20T17:27:45.829 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acf8c70b:::benchmark_data_smithi171_122772_object159565:head# 2024-04-20T17:27:45.830 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acf8d26b:::benchmark_data_smithi171_122772_object145951:head# 2024-04-20T17:27:45.830 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acf928df:::benchmark_data_smithi171_122772_object108492:head# 2024-04-20T17:27:45.830 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acf9db94:::benchmark_data_smithi171_122772_object159765:head# 2024-04-20T17:27:45.831 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acfa1f62:::benchmark_data_smithi171_122772_object138130:head# 2024-04-20T17:27:45.831 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acfa6560:::benchmark_data_smithi171_122772_object78323:head# 2024-04-20T17:27:45.831 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acfb6368:::benchmark_data_smithi171_122772_object127356:head# 2024-04-20T17:27:45.832 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acfbb790:::benchmark_data_smithi171_122772_object128124:head# 2024-04-20T17:27:45.833 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acfc3436:::benchmark_data_smithi171_122772_object157546:head# 2024-04-20T17:27:45.833 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acfdb411:::benchmark_data_smithi171_122772_object95191:head# 2024-04-20T17:27:45.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:45 smithi171 ceph-mon[44189]: pgmap v1353: 17 pgs: 2 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 5.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 3774/184372 objects degraded (2.047%); 69 MiB/s, 1.10k objects/s recovering 2024-04-20T17:27:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:45 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3938960208' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:27:45.834 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acfdbf1d:::benchmark_data_smithi171_122772_object147204:head# 2024-04-20T17:27:45.834 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acfdfe1a:::benchmark_data_smithi171_122772_object154081:head# 2024-04-20T17:27:45.835 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acfe5ffe:::benchmark_data_smithi171_122772_object125964:head# 2024-04-20T17:27:45.835 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acfea00a:::benchmark_data_smithi171_122772_object144840:head# 2024-04-20T17:27:45.836 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:acff2a96:::benchmark_data_smithi171_122772_object98041:head# 2024-04-20T17:27:45.836 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad00220e:::benchmark_data_smithi171_122772_object139509:head# 2024-04-20T17:27:45.837 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad012bd9:::benchmark_data_smithi171_122772_object127471:head# 2024-04-20T17:27:45.837 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad01409a:::benchmark_data_smithi171_122772_object169784:head# 2024-04-20T17:27:45.838 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad015e88:::benchmark_data_smithi171_122772_object161647:head# 2024-04-20T17:27:45.838 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad01608d:::benchmark_data_smithi171_122772_object79658:head# 2024-04-20T17:27:45.839 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad0209a8:::benchmark_data_smithi171_122772_object108410:head# 2024-04-20T17:27:45.839 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad02917b:::benchmark_data_smithi171_122772_object159371:head# 2024-04-20T17:27:45.840 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad02f6ed:::benchmark_data_smithi171_122772_object145572:head# 2024-04-20T17:27:45.840 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad030ea4:::benchmark_data_smithi171_122772_object137230:head# 2024-04-20T17:27:45.841 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad03d6ec:::benchmark_data_smithi171_122772_object101346:head# 2024-04-20T17:27:45.841 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad03fbf9:::benchmark_data_smithi171_122772_object108232:head# 2024-04-20T17:27:45.842 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad05d24e:::benchmark_data_smithi171_122772_object98175:head# 2024-04-20T17:27:45.842 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad072a54:::benchmark_data_smithi171_122772_object169159:head# 2024-04-20T17:27:45.842 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad09f6ed:::benchmark_data_smithi171_122772_object112760:head# 2024-04-20T17:27:45.843 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad0a52b6:::benchmark_data_smithi171_122772_object168599:head# 2024-04-20T17:27:45.844 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad0acf61:::benchmark_data_smithi171_122772_object106995:head# 2024-04-20T17:27:45.844 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad0ae0df:::benchmark_data_smithi171_122772_object159713:head# 2024-04-20T17:27:45.844 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad0aefa1:::benchmark_data_smithi171_122772_object166974:head# 2024-04-20T17:27:45.845 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad0be21d:::benchmark_data_smithi171_122772_object134131:head# 2024-04-20T17:27:45.845 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad0bfba0:::benchmark_data_smithi171_122772_object157243:head# 2024-04-20T17:27:45.846 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad0ccf40:::benchmark_data_smithi171_122772_object88014:head# 2024-04-20T17:27:45.846 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad0d403d:::benchmark_data_smithi171_122772_object168125:head# 2024-04-20T17:27:45.847 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad0e39d4:::benchmark_data_smithi171_122772_object90637:head# 2024-04-20T17:27:45.847 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad0f5d3a:::benchmark_data_smithi171_122772_object110124:head# 2024-04-20T17:27:45.848 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad0f790a:::benchmark_data_smithi171_122772_object115921:head# 2024-04-20T17:27:45.848 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad0fc7e7:::benchmark_data_smithi171_122772_object133055:head# 2024-04-20T17:27:45.848 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad100433:::benchmark_data_smithi171_122772_object153041:head# 2024-04-20T17:27:45.849 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad103138:::benchmark_data_smithi171_122772_object90429:head# 2024-04-20T17:27:45.849 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad104b37:::benchmark_data_smithi171_122772_object155409:head# 2024-04-20T17:27:45.850 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad116e27:::benchmark_data_smithi171_122772_object125155:head# 2024-04-20T17:27:45.850 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad11e3f0:::benchmark_data_smithi171_122772_object160877:head# 2024-04-20T17:27:45.851 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad128861:::benchmark_data_smithi171_122772_object110964:head# 2024-04-20T17:27:45.851 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad12bbf1:::benchmark_data_smithi171_122772_object127996:head# 2024-04-20T17:27:45.852 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad12c084:::benchmark_data_smithi171_122772_object110424:head# 2024-04-20T17:27:45.852 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad12e94c:::benchmark_data_smithi171_122772_object121219:head# 2024-04-20T17:27:45.853 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad13d5e6:::benchmark_data_smithi171_122772_object169789:head# 2024-04-20T17:27:45.853 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad140fe6:::benchmark_data_smithi171_122772_object169968:head# 2024-04-20T17:27:45.854 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad169a77:::benchmark_data_smithi171_122772_object98276:head# 2024-04-20T17:27:45.854 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad16de5b:::benchmark_data_smithi171_122772_object111735:head# 2024-04-20T17:27:45.854 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad173b49:::benchmark_data_smithi171_122772_object91613:head# 2024-04-20T17:27:45.855 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad17da71:::benchmark_data_smithi171_122772_object119289:head# 2024-04-20T17:27:45.856 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad190573:::benchmark_data_smithi171_122772_object120938:head# 2024-04-20T17:27:45.856 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad19f43a:::benchmark_data_smithi171_122772_object151142:head# 2024-04-20T17:27:45.857 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad1a5c17:::benchmark_data_smithi171_122772_object138618:head# 2024-04-20T17:27:45.857 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad1c9d44:::benchmark_data_smithi171_122772_object169610:head# 2024-04-20T17:27:45.858 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad1d7f88:::benchmark_data_smithi171_122772_object113787:head# 2024-04-20T17:27:45.858 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad1ee138:::benchmark_data_smithi171_122772_object84407:head# 2024-04-20T17:27:45.859 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad1fb785:::benchmark_data_smithi171_122772_object107980:head# 2024-04-20T17:27:45.859 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad2018eb:::benchmark_data_smithi171_122772_object137718:head# 2024-04-20T17:27:45.860 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad208c72:::benchmark_data_smithi171_122772_object148992:head# 2024-04-20T17:27:45.860 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad20ddba:::benchmark_data_smithi171_122772_object144061:head# 2024-04-20T17:27:45.861 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad21308f:::benchmark_data_smithi171_122772_object124352:head# 2024-04-20T17:27:45.862 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad231b2c:::benchmark_data_smithi171_122772_object85420:head# 2024-04-20T17:27:45.862 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad2491a6:::benchmark_data_smithi171_122772_object135676:head# 2024-04-20T17:27:45.862 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad24f36f:::benchmark_data_smithi171_122772_object102823:head# 2024-04-20T17:27:45.863 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad26db30:::benchmark_data_smithi171_122772_object121606:head# 2024-04-20T17:27:45.863 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad274bd1:::benchmark_data_smithi171_122772_object162102:head# 2024-04-20T17:27:45.864 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad27da0f:::benchmark_data_smithi171_122772_object119990:head# 2024-04-20T17:27:45.864 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad2814dc:::benchmark_data_smithi171_122772_object115639:head# 2024-04-20T17:27:45.865 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad2852b9:::benchmark_data_smithi171_122772_object96987:head# 2024-04-20T17:27:45.865 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad288405:::benchmark_data_smithi171_122772_object101212:head# 2024-04-20T17:27:45.866 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad2944ec:::benchmark_data_smithi171_122772_object154236:head# 2024-04-20T17:27:45.866 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad29b6ef:::benchmark_data_smithi171_122772_object100453:head# 2024-04-20T17:27:45.867 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad29c7e7:::benchmark_data_smithi171_122772_object98365:head# 2024-04-20T17:27:45.867 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad29ddb5:::benchmark_data_smithi171_122772_object114553:head# 2024-04-20T17:27:45.868 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad2a50ab:::benchmark_data_smithi171_122772_object111718:head# 2024-04-20T17:27:45.868 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad2bcaf6:::benchmark_data_smithi171_122772_object113204:head# 2024-04-20T17:27:45.869 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad2c6ab0:::benchmark_data_smithi171_122772_object88910:head# 2024-04-20T17:27:45.869 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad2db39c:::benchmark_data_smithi171_122772_object138333:head# 2024-04-20T17:27:45.869 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad2ee2a5:::benchmark_data_smithi171_122772_object115880:head# 2024-04-20T17:27:45.870 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad2ef549:::benchmark_data_smithi171_122772_object159963:head# 2024-04-20T17:27:45.870 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad2f684e:::benchmark_data_smithi171_122772_object125289:head# 2024-04-20T17:27:45.871 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad2fadd0:::benchmark_data_smithi171_122772_object86188:head# 2024-04-20T17:27:45.871 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad30a33f:::benchmark_data_smithi171_122772_object140037:head# 2024-04-20T17:27:45.871 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad31b320:::benchmark_data_smithi171_122772_object139035:head# 2024-04-20T17:27:45.872 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad32e8fd:::benchmark_data_smithi171_122772_object138405:head# 2024-04-20T17:27:45.872 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad33af90:::benchmark_data_smithi171_122772_object88849:head# 2024-04-20T17:27:45.873 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad354b49:::benchmark_data_smithi171_122772_object134446:head# 2024-04-20T17:27:45.873 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad35bc05:::benchmark_data_smithi171_122772_object79657:head# 2024-04-20T17:27:45.874 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad35d9af:::benchmark_data_smithi171_122772_object130433:head# 2024-04-20T17:27:45.874 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad3a2d17:::benchmark_data_smithi171_122772_object120020:head# 2024-04-20T17:27:45.874 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad3a6c43:::benchmark_data_smithi171_122772_object113501:head# 2024-04-20T17:27:45.875 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad3ae3c7:::benchmark_data_smithi171_122772_object169214:head# 2024-04-20T17:27:45.875 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad3afde1:::benchmark_data_smithi171_122772_object82989:head# 2024-04-20T17:27:45.875 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad3baa5a:::benchmark_data_smithi171_122772_object147450:head# 2024-04-20T17:27:45.876 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad3c6a33:::benchmark_data_smithi171_122772_object83636:head# 2024-04-20T17:27:45.876 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad3d15a9:::benchmark_data_smithi171_122772_object146076:head# 2024-04-20T17:27:45.877 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad3d925e:::benchmark_data_smithi171_122772_object112984:head# 2024-04-20T17:27:45.877 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad3e187a:::benchmark_data_smithi171_122772_object104963:head# 2024-04-20T17:27:45.878 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad3e439b:::benchmark_data_smithi171_122772_object160266:head# 2024-04-20T17:27:45.878 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad3f3704:::benchmark_data_smithi171_122772_object120010:head# 2024-04-20T17:27:45.879 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad40fa88:::benchmark_data_smithi171_122772_object165834:head# 2024-04-20T17:27:45.879 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad41188a:::benchmark_data_smithi171_122772_object138716:head# 2024-04-20T17:27:45.880 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad424ced:::benchmark_data_smithi171_122772_object163343:head# 2024-04-20T17:27:45.880 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad42797a:::benchmark_data_smithi171_122772_object114909:head# 2024-04-20T17:27:45.881 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad4345d0:::benchmark_data_smithi171_122772_object86857:head# 2024-04-20T17:27:45.881 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad450a84:::benchmark_data_smithi171_122772_object115972:head# 2024-04-20T17:27:45.882 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad46de7e:::benchmark_data_smithi171_122772_object131705:head# 2024-04-20T17:27:45.882 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad4727b8:::benchmark_data_smithi171_122772_object83846:head# 2024-04-20T17:27:45.883 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad47e8c2:::benchmark_data_smithi171_122772_object83513:head# 2024-04-20T17:27:45.883 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad4806e4:::benchmark_data_smithi171_122772_object89463:head# 2024-04-20T17:27:45.883 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad48c4f0:::benchmark_data_smithi171_122772_object142780:head# 2024-04-20T17:27:45.884 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad4ae65b:::benchmark_data_smithi171_122772_object147762:head# 2024-04-20T17:27:45.884 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad4bf642:::benchmark_data_smithi171_122772_object90231:head# 2024-04-20T17:27:45.885 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad4c2728:::benchmark_data_smithi171_122772_object79452:head# 2024-04-20T17:27:45.885 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad4c2e50:::benchmark_data_smithi171_122772_object98876:head# 2024-04-20T17:27:45.886 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad4f5189:::benchmark_data_smithi171_122772_object164295:head# 2024-04-20T17:27:45.886 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad4f87cc:::benchmark_data_smithi171_122772_object105730:head# 2024-04-20T17:27:45.886 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad5022ae:::benchmark_data_smithi171_122772_object138863:head# 2024-04-20T17:27:45.887 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad504f60:::benchmark_data_smithi171_122772_object99696:head# 2024-04-20T17:27:45.888 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad517a24:::benchmark_data_smithi171_122772_object83198:head# 2024-04-20T17:27:45.888 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad52f1ed:::benchmark_data_smithi171_122772_object134802:head# 2024-04-20T17:27:45.888 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad536687:::benchmark_data_smithi171_122772_object127162:head# 2024-04-20T17:27:45.889 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad542d1d:::benchmark_data_smithi171_122772_object144997:head# 2024-04-20T17:27:45.890 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad54353a:::benchmark_data_smithi171_122772_object167880:head# 2024-04-20T17:27:45.890 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad5533b9:::benchmark_data_smithi171_122772_object115224:head# 2024-04-20T17:27:45.891 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad57bfa1:::benchmark_data_smithi171_122772_object118785:head# 2024-04-20T17:27:45.891 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad58aa84:::benchmark_data_smithi171_122772_object153680:head# 2024-04-20T17:27:45.891 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad5b0a4e:::benchmark_data_smithi171_122772_object108554:head# 2024-04-20T17:27:45.892 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad5b725c:::benchmark_data_smithi171_122772_object159027:head# 2024-04-20T17:27:45.892 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad5c8dae:::benchmark_data_smithi171_122772_object159751:head# 2024-04-20T17:27:45.893 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad5d2dbb:::benchmark_data_smithi171_122772_object87042:head# 2024-04-20T17:27:45.893 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad5dcf58:::benchmark_data_smithi171_122772_object167847:head# 2024-04-20T17:27:45.894 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad5e8adb:::benchmark_data_smithi171_122772_object168559:head# 2024-04-20T17:27:45.894 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad603787:::benchmark_data_smithi171_122772_object127348:head# 2024-04-20T17:27:45.895 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad608712:::benchmark_data_smithi171_122772_object138604:head# 2024-04-20T17:27:45.895 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad60f79a:::benchmark_data_smithi171_122772_object125696:head# 2024-04-20T17:27:45.895 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad61fd36:::benchmark_data_smithi171_122772_object133255:head# 2024-04-20T17:27:45.896 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad642bd3:::benchmark_data_smithi171_122772_object115129:head# 2024-04-20T17:27:45.897 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad65c825:::benchmark_data_smithi171_122772_object147546:head# 2024-04-20T17:27:45.897 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad66474d:::benchmark_data_smithi171_122772_object125095:head# 2024-04-20T17:27:45.897 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad673bb7:::benchmark_data_smithi171_122772_object81339:head# 2024-04-20T17:27:45.898 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad688ee1:::benchmark_data_smithi171_122772_object126140:head# 2024-04-20T17:27:45.898 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad6b6ac6:::benchmark_data_smithi171_122772_object98595:head# 2024-04-20T17:27:45.899 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad6bad0a:::benchmark_data_smithi171_122772_object139958:head# 2024-04-20T17:27:45.899 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad6fcfb6:::benchmark_data_smithi171_122772_object83726:head# 2024-04-20T17:27:45.900 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad723187:::benchmark_data_smithi171_122772_object152797:head# 2024-04-20T17:27:45.900 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad7362f3:::benchmark_data_smithi171_122772_object141311:head# 2024-04-20T17:27:45.901 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad737e28:::benchmark_data_smithi171_122772_object140498:head# 2024-04-20T17:27:45.901 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad750161:::benchmark_data_smithi171_122772_object92892:head# 2024-04-20T17:27:45.902 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad750473:::benchmark_data_smithi171_122772_object161248:head# 2024-04-20T17:27:45.902 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad76fa02:::benchmark_data_smithi171_122772_object100422:head# 2024-04-20T17:27:45.903 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad779e6a:::benchmark_data_smithi171_122772_object138779:head# 2024-04-20T17:27:45.903 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad77daf2:::benchmark_data_smithi171_122772_object161393:head# 2024-04-20T17:27:45.904 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad78c4c5:::benchmark_data_smithi171_122772_object101439:head# 2024-04-20T17:27:45.905 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad78df47:::benchmark_data_smithi171_122772_object152399:head# 2024-04-20T17:27:45.905 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad7ac00e:::benchmark_data_smithi171_122772_object112962:head# 2024-04-20T17:27:45.905 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad7be04a:::benchmark_data_smithi171_122772_object84413:head# 2024-04-20T17:27:45.906 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad7c68fb:::benchmark_data_smithi171_122772_object141012:head# 2024-04-20T17:27:45.906 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad7c8426:::benchmark_data_smithi171_122772_object104947:head# 2024-04-20T17:27:45.907 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad7d40d5:::benchmark_data_smithi171_122772_object140945:head# 2024-04-20T17:27:45.907 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad801900:::benchmark_data_smithi171_122772_object140238:head# 2024-04-20T17:27:45.908 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad809a8c:::benchmark_data_smithi171_122772_object100222:head# 2024-04-20T17:27:45.908 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad815251:::benchmark_data_smithi171_122772_object127376:head# 2024-04-20T17:27:45.909 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad815bc9:::benchmark_data_smithi171_122772_object97575:head# 2024-04-20T17:27:45.909 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad8167df:::benchmark_data_smithi171_122772_object167960:head# 2024-04-20T17:27:45.910 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad81d050:::benchmark_data_smithi171_122772_object161340:head# 2024-04-20T17:27:45.911 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad81f5dc:::benchmark_data_smithi171_122772_object105189:head# 2024-04-20T17:27:45.911 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad824108:::benchmark_data_smithi171_122772_object92396:head# 2024-04-20T17:27:45.912 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad838bd6:::benchmark_data_smithi171_122772_object124277:head# 2024-04-20T17:27:45.912 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad83d162:::benchmark_data_smithi171_122772_object158801:head# 2024-04-20T17:27:45.912 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad84307e:::benchmark_data_smithi171_122772_object101433:head# 2024-04-20T17:27:45.913 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad843c51:::benchmark_data_smithi171_122772_object165048:head# 2024-04-20T17:27:45.913 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad84b848:::benchmark_data_smithi171_122772_object101743:head# 2024-04-20T17:27:45.914 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad859c6b:::benchmark_data_smithi171_122772_object85575:head# 2024-04-20T17:27:45.915 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad85c79b:::benchmark_data_smithi171_122772_object146141:head# 2024-04-20T17:27:45.915 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad85d2d5:::benchmark_data_smithi171_122772_object109257:head# 2024-04-20T17:27:45.916 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad86711a:::benchmark_data_smithi171_122772_object140104:head# 2024-04-20T17:27:45.916 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad87e6e4:::benchmark_data_smithi171_122772_object136821:head# 2024-04-20T17:27:45.916 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad884f90:::benchmark_data_smithi171_122772_object87302:head# 2024-04-20T17:27:45.917 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad8908f0:::benchmark_data_smithi171_122772_object83448:head# 2024-04-20T17:27:45.917 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad8aa52d:::benchmark_data_smithi171_122772_object90780:head# 2024-04-20T17:27:45.918 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad8dc09e:::benchmark_data_smithi171_122772_object117789:head# 2024-04-20T17:27:45.918 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad8dd4a0:::benchmark_data_smithi171_122772_object142239:head# 2024-04-20T17:27:45.919 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad8deb82:::benchmark_data_smithi171_122772_object118871:head# 2024-04-20T17:27:45.919 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad8e0196:::benchmark_data_smithi171_122772_object81361:head# 2024-04-20T17:27:45.919 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad8eac2f:::benchmark_data_smithi171_122772_object137194:head# 2024-04-20T17:27:45.919 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad8f3f6f:::benchmark_data_smithi171_122772_object101999:head# 2024-04-20T17:27:45.920 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad8f43aa:::benchmark_data_smithi171_122772_object132015:head# 2024-04-20T17:27:45.920 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad8fe4fb:::benchmark_data_smithi171_122772_object107393:head# 2024-04-20T17:27:45.921 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad903484:::benchmark_data_smithi171_122772_object146161:head# 2024-04-20T17:27:45.921 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad906b4f:::benchmark_data_smithi171_122772_object145628:head# 2024-04-20T17:27:45.921 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad91d503:::benchmark_data_smithi171_122772_object91188:head# 2024-04-20T17:27:45.922 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad9206cc:::benchmark_data_smithi171_122772_object124345:head# 2024-04-20T17:27:45.922 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad920765:::benchmark_data_smithi171_122772_object145870:head# 2024-04-20T17:27:45.923 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad932bf7:::benchmark_data_smithi171_122772_object139014:head# 2024-04-20T17:27:45.923 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad94ea4d:::benchmark_data_smithi171_122772_object150559:head# 2024-04-20T17:27:45.924 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad951f82:::benchmark_data_smithi171_122772_object93049:head# 2024-04-20T17:27:45.924 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad95c476:::benchmark_data_smithi171_122772_object148946:head# 2024-04-20T17:27:45.925 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad96a6c9:::benchmark_data_smithi171_122772_object160549:head# 2024-04-20T17:27:45.925 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad96aaec:::benchmark_data_smithi171_122772_object79633:head# 2024-04-20T17:27:45.926 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad97e21b:::benchmark_data_smithi171_122772_object165660:head# 2024-04-20T17:27:45.926 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad987093:::benchmark_data_smithi171_122772_object131334:head# 2024-04-20T17:27:45.926 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad98c010:::benchmark_data_smithi171_122772_object156194:head# 2024-04-20T17:27:45.927 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad98f54a:::benchmark_data_smithi171_122772_object151448:head# 2024-04-20T17:27:45.927 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad990338:::benchmark_data_smithi171_122772_object82322:head# 2024-04-20T17:27:45.928 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad9a3253:::benchmark_data_smithi171_122772_object118602:head# 2024-04-20T17:27:45.928 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad9bd7d6:::benchmark_data_smithi171_122772_object145759:head# 2024-04-20T17:27:45.929 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad9bed69:::benchmark_data_smithi171_122772_object123411:head# 2024-04-20T17:27:45.929 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad9da353:::benchmark_data_smithi171_122772_object97487:head# 2024-04-20T17:27:45.930 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad9eb316:::benchmark_data_smithi171_122772_object81358:head# 2024-04-20T17:27:45.930 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ad9fb696:::benchmark_data_smithi171_122772_object124931:head# 2024-04-20T17:27:45.931 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ada0451f:::benchmark_data_smithi171_122772_object153371:head# 2024-04-20T17:27:45.931 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ada08441:::benchmark_data_smithi171_122772_object127511:head# 2024-04-20T17:27:45.932 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ada09a4a:::benchmark_data_smithi171_122772_object125257:head# 2024-04-20T17:27:45.932 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ada111ea:::benchmark_data_smithi171_122772_object129283:head# 2024-04-20T17:27:45.933 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ada1bf7e:::benchmark_data_smithi171_122772_object109157:head# 2024-04-20T17:27:45.933 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ada2a2ba:::benchmark_data_smithi171_122772_object126607:head# 2024-04-20T17:27:45.934 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ada2c9e7:::benchmark_data_smithi171_122772_object90027:head# 2024-04-20T17:27:45.934 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ada2f6d0:::benchmark_data_smithi171_122772_object114823:head# 2024-04-20T17:27:45.935 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ada328d9:::benchmark_data_smithi171_122772_object80575:head# 2024-04-20T17:27:45.935 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ada38632:::benchmark_data_smithi171_122772_object130040:head# 2024-04-20T17:27:45.936 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ada55edc:::benchmark_data_smithi171_122772_object162826:head# 2024-04-20T17:27:45.936 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ada5b884:::benchmark_data_smithi171_122772_object80904:head# 2024-04-20T17:27:45.937 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ada6278b:::benchmark_data_smithi171_122772_object133092:head# 2024-04-20T17:27:45.937 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ada849d4:::benchmark_data_smithi171_122772_object132853:head# 2024-04-20T17:27:45.938 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ada884e1:::benchmark_data_smithi171_122772_object162260:head# 2024-04-20T17:27:45.938 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ada8bf4c:::benchmark_data_smithi171_122772_object145935:head# 2024-04-20T17:27:45.938 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ada9ae98:::benchmark_data_smithi171_122772_object85015:head# 2024-04-20T17:27:45.939 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adaaf968:::benchmark_data_smithi171_122772_object141281:head# 2024-04-20T17:27:45.940 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adaca7c0:::benchmark_data_smithi171_122772_object158763:head# 2024-04-20T17:27:45.940 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adacf396:::benchmark_data_smithi171_122772_object136117:head# 2024-04-20T17:27:45.941 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adaee6a8:::benchmark_data_smithi171_122772_object88452:head# 2024-04-20T17:27:45.941 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adaef04b:::benchmark_data_smithi171_122772_object169976:head# 2024-04-20T17:27:45.941 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adaf87ba:::benchmark_data_smithi171_122772_object158144:head# 2024-04-20T17:27:45.942 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adb08471:::benchmark_data_smithi171_122772_object106343:head# 2024-04-20T17:27:45.942 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adb12599:::benchmark_data_smithi171_122772_object123598:head# 2024-04-20T17:27:45.943 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adb1eac9:::benchmark_data_smithi171_122772_object146179:head# 2024-04-20T17:27:45.943 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adb21f8a:::benchmark_data_smithi171_122772_object80779:head# 2024-04-20T17:27:45.943 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adb24f77:::benchmark_data_smithi171_122772_object102294:head# 2024-04-20T17:27:45.944 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adb31b72:::benchmark_data_smithi171_122772_object133428:head# 2024-04-20T17:27:45.944 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adb34aff:::benchmark_data_smithi171_122772_object105036:head# 2024-04-20T17:27:45.945 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adb59721:::benchmark_data_smithi171_122772_object116889:head# 2024-04-20T17:27:45.945 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adb5af23:::benchmark_data_smithi171_122772_object120761:head# 2024-04-20T17:27:45.945 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adb6d8e9:::benchmark_data_smithi171_122772_object156700:head# 2024-04-20T17:27:45.946 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adb7e60a:::benchmark_data_smithi171_122772_object159190:head# 2024-04-20T17:27:45.946 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adb8318d:::benchmark_data_smithi171_122772_object82550:head# 2024-04-20T17:27:45.947 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adb84554:::benchmark_data_smithi171_122772_object140709:head# 2024-04-20T17:27:45.950 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adb8bc52:::benchmark_data_smithi171_122772_object122650:head# 2024-04-20T17:27:45.950 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adb8c6bf:::benchmark_data_smithi171_122772_object79081:head# 2024-04-20T17:27:45.950 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adbb357b:::benchmark_data_smithi171_122772_object141300:head# 2024-04-20T17:27:45.950 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adbb62a2:::benchmark_data_smithi171_122772_object142562:head# 2024-04-20T17:27:45.950 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adbcc08d:::benchmark_data_smithi171_122772_object106624:head# 2024-04-20T17:27:45.950 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adbd1226:::benchmark_data_smithi171_122772_object131076:head# 2024-04-20T17:27:45.950 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adbd19ae:::benchmark_data_smithi171_122772_object79787:head# 2024-04-20T17:27:45.951 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adbd58dd:::benchmark_data_smithi171_122772_object78087:head# 2024-04-20T17:27:45.951 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adbe7d62:::benchmark_data_smithi171_122772_object138586:head# 2024-04-20T17:27:45.951 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adc036cb:::benchmark_data_smithi171_122772_object153489:head# 2024-04-20T17:27:45.952 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adc11682:::benchmark_data_smithi171_122772_object88796:head# 2024-04-20T17:27:45.952 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adc31313:::benchmark_data_smithi171_122772_object84072:head# 2024-04-20T17:27:45.953 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adc3393a:::benchmark_data_smithi171_122772_object138362:head# 2024-04-20T17:27:45.953 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adc3501f:::benchmark_data_smithi171_122772_object160343:head# 2024-04-20T17:27:45.954 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adc3df13:::benchmark_data_smithi171_122772_object122379:head# 2024-04-20T17:27:45.954 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adc43a98:::benchmark_data_smithi171_122772_object108717:head# 2024-04-20T17:27:45.955 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adc44073:::benchmark_data_smithi171_122772_object104103:head# 2024-04-20T17:27:45.955 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adc4ac0b:::benchmark_data_smithi171_122772_object159895:head# 2024-04-20T17:27:45.955 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adc5b0f3:::benchmark_data_smithi171_122772_object154946:head# 2024-04-20T17:27:45.956 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adc61fd6:::benchmark_data_smithi171_122772_object164177:head# 2024-04-20T17:27:45.956 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adc65f05:::benchmark_data_smithi171_122772_object134067:head# 2024-04-20T17:27:45.957 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adc70fd8:::benchmark_data_smithi171_122772_object113127:head# 2024-04-20T17:27:45.957 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adc7de35:::benchmark_data_smithi171_122772_object86032:head# 2024-04-20T17:27:45.957 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adc86c61:::benchmark_data_smithi171_122772_object115555:head# 2024-04-20T17:27:45.958 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adc87417:::benchmark_data_smithi171_122772_object151074:head# 2024-04-20T17:27:45.959 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adc8accf:::benchmark_data_smithi171_122772_object92171:head# 2024-04-20T17:27:45.959 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adcb6c33:::benchmark_data_smithi171_122772_object125055:head# 2024-04-20T17:27:45.959 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adcb9745:::benchmark_data_smithi171_122772_object164732:head# 2024-04-20T17:27:45.960 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adcc6856:::benchmark_data_smithi171_122772_object123537:head# 2024-04-20T17:27:45.960 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adcd175b:::benchmark_data_smithi171_122772_object157669:head# 2024-04-20T17:27:45.961 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adcd9675:::benchmark_data_smithi171_122772_object99005:head# 2024-04-20T17:27:45.961 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adcdcea9:::benchmark_data_smithi171_122772_object105575:head# 2024-04-20T17:27:45.962 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adce41d0:::benchmark_data_smithi171_122772_object160775:head# 2024-04-20T17:27:45.962 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adced7c8:::benchmark_data_smithi171_122772_object162109:head# 2024-04-20T17:27:45.963 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adcf0bf6:::benchmark_data_smithi171_122772_object155787:head# 2024-04-20T17:27:45.963 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adcf14fb:::benchmark_data_smithi171_122772_object117809:head# 2024-04-20T17:27:45.963 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adcfa2c5:::benchmark_data_smithi171_122772_object146805:head# 2024-04-20T17:27:45.964 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:add0a0fc:::benchmark_data_smithi171_122772_object145556:head# 2024-04-20T17:27:45.964 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:add0cee0:::benchmark_data_smithi171_122772_object156387:head# 2024-04-20T17:27:45.964 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:add4602e:::benchmark_data_smithi171_122772_object137543:head# 2024-04-20T17:27:45.965 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:add54929:::benchmark_data_smithi171_122772_object152105:head# 2024-04-20T17:27:45.966 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:add55e39:::benchmark_data_smithi171_122772_object158007:head# 2024-04-20T17:27:45.966 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:add6f739:::benchmark_data_smithi171_122772_object167252:head# 2024-04-20T17:27:45.967 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:add7b9a2:::benchmark_data_smithi171_122772_object95349:head# 2024-04-20T17:27:45.967 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:add8b65a:::benchmark_data_smithi171_122772_object160965:head# 2024-04-20T17:27:45.967 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adda6173:::benchmark_data_smithi171_122772_object118448:head# 2024-04-20T17:27:45.968 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adda9331:::benchmark_data_smithi171_122772_object80066:head# 2024-04-20T17:27:45.968 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:addab75e:::benchmark_data_smithi171_122772_object138212:head# 2024-04-20T17:27:45.969 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:addafc57:::benchmark_data_smithi171_122772_object119509:head# 2024-04-20T17:27:45.969 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:addb4690:::benchmark_data_smithi171_122772_object137629:head# 2024-04-20T17:27:45.970 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:addbe0c4:::benchmark_data_smithi171_122772_object118783:head# 2024-04-20T17:27:45.970 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:addc5953:::benchmark_data_smithi171_122772_object150423:head# 2024-04-20T17:27:45.970 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adde866e:::benchmark_data_smithi171_122772_object159237:head# 2024-04-20T17:27:45.971 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:addeff4b:::benchmark_data_smithi171_122772_object130260:head# 2024-04-20T17:27:45.971 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:addf1de4:::benchmark_data_smithi171_122772_object168254:head# 2024-04-20T17:27:45.971 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:addf63b1:::benchmark_data_smithi171_122772_object102743:head# 2024-04-20T17:27:45.972 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:addf7d1a:::benchmark_data_smithi171_122772_object106332:head# 2024-04-20T17:27:45.972 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:addfcb5f:::benchmark_data_smithi171_122772_object154823:head# 2024-04-20T17:27:45.973 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ade12d4c:::benchmark_data_smithi171_122772_object87326:head# 2024-04-20T17:27:45.973 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ade3269b:::benchmark_data_smithi171_122772_object84101:head# 2024-04-20T17:27:45.974 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ade3dcdf:::benchmark_data_smithi171_122772_object153877:head# 2024-04-20T17:27:45.974 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ade47089:::benchmark_data_smithi171_122772_object87792:head# 2024-04-20T17:27:45.975 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ade4ed80:::benchmark_data_smithi171_122772_object96767:head# 2024-04-20T17:27:45.975 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ade6cb79:::benchmark_data_smithi171_122772_object123899:head# 2024-04-20T17:27:45.976 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ade782b9:::benchmark_data_smithi171_122772_object142588:head# 2024-04-20T17:27:45.976 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ade8f34b:::benchmark_data_smithi171_122772_object161676:head# 2024-04-20T17:27:45.977 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adea2a24:::benchmark_data_smithi171_122772_object88147:head# 2024-04-20T17:27:45.977 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adea84b2:::benchmark_data_smithi171_122772_object120387:head# 2024-04-20T17:27:45.977 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adeaab36:::benchmark_data_smithi171_122772_object160203:head# 2024-04-20T17:27:45.978 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adeb31ab:::benchmark_data_smithi171_122772_object143574:head# 2024-04-20T17:27:45.978 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adeb4d85:::benchmark_data_smithi171_122772_object126109:head# 2024-04-20T17:27:45.979 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adec14de:::benchmark_data_smithi171_122772_object138819:head# 2024-04-20T17:27:45.979 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aded7ce2:::benchmark_data_smithi171_122772_object89752:head# 2024-04-20T17:27:45.980 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adee01ff:::benchmark_data_smithi171_122772_object136179:head# 2024-04-20T17:27:45.980 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adee3d6d:::benchmark_data_smithi171_122772_object165711:head# 2024-04-20T17:27:45.980 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adeea1b6:::benchmark_data_smithi171_122772_object85712:head# 2024-04-20T17:27:45.981 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adeef6ce:::benchmark_data_smithi171_122772_object148380:head# 2024-04-20T17:27:45.981 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adf01d7e:::benchmark_data_smithi171_122772_object150409:head# 2024-04-20T17:27:45.982 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adf0fe63:::benchmark_data_smithi171_122772_object144465:head# 2024-04-20T17:27:45.982 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adf24341:::benchmark_data_smithi171_122772_object92467:head# 2024-04-20T17:27:45.983 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adf27e7b:::benchmark_data_smithi171_122772_object130812:head# 2024-04-20T17:27:45.983 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adf3e1f0:::benchmark_data_smithi171_122772_object121861:head# 2024-04-20T17:27:45.984 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adf4639a:::benchmark_data_smithi171_122772_object120624:head# 2024-04-20T17:27:45.984 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adf5d18e:::benchmark_data_smithi171_122772_object116787:head# 2024-04-20T17:27:45.984 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adf63b34:::benchmark_data_smithi171_122772_object98512:head# 2024-04-20T17:27:45.985 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adf6a9eb:::benchmark_data_smithi171_122772_object118351:head# 2024-04-20T17:27:45.985 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adf6b8a3:::benchmark_data_smithi171_122772_object153176:head# 2024-04-20T17:27:45.986 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adf6cdf8:::benchmark_data_smithi171_122772_object135343:head# 2024-04-20T17:27:45.986 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adf853fb:::benchmark_data_smithi171_122772_object101991:head# 2024-04-20T17:27:45.986 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adf8897d:::benchmark_data_smithi171_122772_object123834:head# 2024-04-20T17:27:45.987 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adf954c2:::benchmark_data_smithi171_122772_object169419:head# 2024-04-20T17:27:45.987 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adfa156f:::benchmark_data_smithi171_122772_object137101:head# 2024-04-20T17:27:45.988 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adfbd3d2:::benchmark_data_smithi171_122772_object158586:head# 2024-04-20T17:27:45.988 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adfc0ed4:::benchmark_data_smithi171_122772_object80778:head# 2024-04-20T17:27:45.989 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adfc7cd4:::benchmark_data_smithi171_122772_object120894:head# 2024-04-20T17:27:45.989 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adfc99b1:::benchmark_data_smithi171_122772_object81086:head# 2024-04-20T17:27:45.989 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adfe16c2:::benchmark_data_smithi171_122772_object100919:head# 2024-04-20T17:27:45.990 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adfede68:::benchmark_data_smithi171_122772_object164734:head# 2024-04-20T17:27:45.990 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:adffa530:::benchmark_data_smithi171_122772_object91908:head# 2024-04-20T17:27:45.991 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae00ee2b:::benchmark_data_smithi171_122772_object149089:head# 2024-04-20T17:27:45.991 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae02ac55:::benchmark_data_smithi171_122772_object119326:head# 2024-04-20T17:27:45.992 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae02cf06:::benchmark_data_smithi171_122772_object146470:head# 2024-04-20T17:27:45.992 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae035598:::benchmark_data_smithi171_122772_object107095:head# 2024-04-20T17:27:45.993 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae0359c8:::benchmark_data_smithi171_122772_object120883:head# 2024-04-20T17:27:45.993 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae04781d:::benchmark_data_smithi171_122772_object120812:head# 2024-04-20T17:27:45.994 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae04bd94:::benchmark_data_smithi171_122772_object140584:head# 2024-04-20T17:27:45.995 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae04c247:::benchmark_data_smithi171_122772_object132034:head# 2024-04-20T17:27:45.995 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae052ec5:::benchmark_data_smithi171_122772_object166005:head# 2024-04-20T17:27:45.995 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae087e0c:::benchmark_data_smithi171_122772_object146588:head# 2024-04-20T17:27:45.996 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae0895c2:::benchmark_data_smithi171_122772_object155475:head# 2024-04-20T17:27:45.996 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae08adef:::benchmark_data_smithi171_122772_object122198:head# 2024-04-20T17:27:45.996 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae09a84c:::benchmark_data_smithi171_122772_object107926:head# 2024-04-20T17:27:45.997 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae0acf3b:::benchmark_data_smithi171_122772_object143045:head# 2024-04-20T17:27:45.997 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae0b2253:::benchmark_data_smithi171_122772_object113557:head# 2024-04-20T17:27:45.998 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae0b5ad7:::benchmark_data_smithi171_122772_object125130:head# 2024-04-20T17:27:45.998 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae0ba967:::benchmark_data_smithi171_122772_object135355:head# 2024-04-20T17:27:45.999 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae0c215e:::benchmark_data_smithi171_122772_object87393:head# 2024-04-20T17:27:45.999 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae0cb396:::benchmark_data_smithi171_122772_object135557:head# 2024-04-20T17:27:46.000 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae0d29a5:::benchmark_data_smithi171_122772_object119044:head# 2024-04-20T17:27:46.000 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae0e0606:::benchmark_data_smithi171_122772_object88410:head# 2024-04-20T17:27:46.001 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae0eeb73:::benchmark_data_smithi171_122772_object161226:head# 2024-04-20T17:27:46.001 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae0facb1:::benchmark_data_smithi171_122772_object81836:head# 2024-04-20T17:27:46.002 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae10821b:::benchmark_data_smithi171_122772_object152389:head# 2024-04-20T17:27:46.002 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae10c12e:::benchmark_data_smithi171_122772_object114479:head# 2024-04-20T17:27:46.003 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae11d3be:::benchmark_data_smithi171_122772_object164322:head# 2024-04-20T17:27:46.003 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae11e8e2:::benchmark_data_smithi171_122772_object140464:head# 2024-04-20T17:27:46.003 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae12a2bf:::benchmark_data_smithi171_122772_object123887:head# 2024-04-20T17:27:46.004 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae12a996:::benchmark_data_smithi171_122772_object105919:head# 2024-04-20T17:27:46.004 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae132234:::benchmark_data_smithi171_122772_object127073:head# 2024-04-20T17:27:46.005 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae1359ab:::benchmark_data_smithi171_122772_object89390:head# 2024-04-20T17:27:46.005 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae1431f1:::benchmark_data_smithi171_122772_object138204:head# 2024-04-20T17:27:46.005 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae1558d7:::benchmark_data_smithi171_122772_object78670:head# 2024-04-20T17:27:46.006 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae173bb9:::benchmark_data_smithi171_122772_object91742:head# 2024-04-20T17:27:46.006 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae17502d:::benchmark_data_smithi171_122772_object99627:head# 2024-04-20T17:27:46.007 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae17711a:::benchmark_data_smithi171_122772_object152091:head# 2024-04-20T17:27:46.007 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae191dbb:::benchmark_data_smithi171_122772_object158579:head# 2024-04-20T17:27:46.007 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae19aa1e:::benchmark_data_smithi171_122772_object167030:head# 2024-04-20T17:27:46.008 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae1b4a90:::benchmark_data_smithi171_122772_object125620:head# 2024-04-20T17:27:46.008 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae1bd489:::benchmark_data_smithi171_122772_object150928:head# 2024-04-20T17:27:46.009 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae1c0b4a:::benchmark_data_smithi171_122772_object78074:head# 2024-04-20T17:27:46.009 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae1c34b0:::benchmark_data_smithi171_122772_object88347:head# 2024-04-20T17:27:46.010 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae1c66c3:::benchmark_data_smithi171_122772_object125374:head# 2024-04-20T17:27:46.010 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae1c878f:::benchmark_data_smithi171_122772_object132599:head# 2024-04-20T17:27:46.010 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae1d9aa5:::benchmark_data_smithi171_122772_object162610:head# 2024-04-20T17:27:46.011 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae1e9b2e:::benchmark_data_smithi171_122772_object144001:head# 2024-04-20T17:27:46.011 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae211ab3:::benchmark_data_smithi171_122772_object92885:head# 2024-04-20T17:27:46.012 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae218af3:::benchmark_data_smithi171_122772_object110176:head# 2024-04-20T17:27:46.013 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae22bcd0:::benchmark_data_smithi171_122772_object135044:head# 2024-04-20T17:27:46.013 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae23d891:::benchmark_data_smithi171_122772_object97942:head# 2024-04-20T17:27:46.013 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae241d09:::benchmark_data_smithi171_122772_object144573:head# 2024-04-20T17:27:46.014 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae24891e:::benchmark_data_smithi171_122772_object117820:head# 2024-04-20T17:27:46.014 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae27a390:::benchmark_data_smithi171_122772_object82597:head# 2024-04-20T17:27:46.015 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae27cdd3:::benchmark_data_smithi171_122772_object123845:head# 2024-04-20T17:27:46.015 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae27f483:::benchmark_data_smithi171_122772_object131499:head# 2024-04-20T17:27:46.015 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae29412b:::benchmark_data_smithi171_122772_object105155:head# 2024-04-20T17:27:46.016 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae29d8ac:::benchmark_data_smithi171_122772_object153934:head# 2024-04-20T17:27:46.016 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae2a646b:::benchmark_data_smithi171_122772_object137581:head# 2024-04-20T17:27:46.017 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae2b90f7:::benchmark_data_smithi171_122772_object139473:head# 2024-04-20T17:27:46.017 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae2c7476:::benchmark_data_smithi171_122772_object120703:head# 2024-04-20T17:27:46.018 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae2d1912:::benchmark_data_smithi171_122772_object166446:head# 2024-04-20T17:27:46.019 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae2e2221:::benchmark_data_smithi171_122772_object95555:head# 2024-04-20T17:27:46.019 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae2eba69:::benchmark_data_smithi171_122772_object119891:head# 2024-04-20T17:27:46.019 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae308a67:::benchmark_data_smithi171_122772_object84651:head# 2024-04-20T17:27:46.020 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae31c387:::benchmark_data_smithi171_122772_object163617:head# 2024-04-20T17:27:46.020 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae326300:::benchmark_data_smithi171_122772_object111873:head# 2024-04-20T17:27:46.021 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae327bcc:::benchmark_data_smithi171_122772_object126183:head# 2024-04-20T17:27:46.022 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae328d98:::benchmark_data_smithi171_122772_object119083:head# 2024-04-20T17:27:46.022 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae33561e:::benchmark_data_smithi171_122772_object78355:head# 2024-04-20T17:27:46.022 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae3481c1:::benchmark_data_smithi171_122772_object128739:head# 2024-04-20T17:27:46.023 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae349dc9:::benchmark_data_smithi171_122772_object149355:head# 2024-04-20T17:27:46.023 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae34d4ea:::benchmark_data_smithi171_122772_object79914:head# 2024-04-20T17:27:46.024 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae35b7ab:::benchmark_data_smithi171_122772_object129106:head# 2024-04-20T17:27:46.024 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae35f0ac:::benchmark_data_smithi171_122772_object82467:head# 2024-04-20T17:27:46.025 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae3661a9:::benchmark_data_smithi171_122772_object82801:head# 2024-04-20T17:27:46.025 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae38fa0d:::benchmark_data_smithi171_122772_object90677:head# 2024-04-20T17:27:46.026 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae395466:::benchmark_data_smithi171_122772_object131482:head# 2024-04-20T17:27:46.026 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae39afb4:::benchmark_data_smithi171_122772_object112905:head# 2024-04-20T17:27:46.026 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae3adfc2:::benchmark_data_smithi171_122772_object161429:head# 2024-04-20T17:27:46.027 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae3b2eb9:::benchmark_data_smithi171_122772_object134266:head# 2024-04-20T17:27:46.027 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae3b70ea:::benchmark_data_smithi171_122772_object123312:head# 2024-04-20T17:27:46.028 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae3ccf45:::benchmark_data_smithi171_122772_object82685:head# 2024-04-20T17:27:46.028 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae3d056d:::benchmark_data_smithi171_122772_object91894:head# 2024-04-20T17:27:46.029 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae3d27a2:::benchmark_data_smithi171_122772_object139633:head# 2024-04-20T17:27:46.029 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae3dd9ac:::benchmark_data_smithi171_122772_object116825:head# 2024-04-20T17:27:46.029 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae3fac02:::benchmark_data_smithi171_122772_object111750:head# 2024-04-20T17:27:46.030 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae44c6d6:::benchmark_data_smithi171_122772_object135661:head# 2024-04-20T17:27:46.030 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae476210:::benchmark_data_smithi171_122772_object132949:head# 2024-04-20T17:27:46.031 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae4779a2:::benchmark_data_smithi171_122772_object143011:head# 2024-04-20T17:27:46.031 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae479fc3:::benchmark_data_smithi171_122772_object109019:head# 2024-04-20T17:27:46.031 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae47dfe6:::benchmark_data_smithi171_122772_object87648:head# 2024-04-20T17:27:46.032 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae47e041:::benchmark_data_smithi171_122772_object108458:head# 2024-04-20T17:27:46.032 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae47e7f4:::benchmark_data_smithi171_122772_object144703:head# 2024-04-20T17:27:46.033 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae48fbf0:::benchmark_data_smithi171_122772_object126618:head# 2024-04-20T17:27:46.033 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae4b3d08:::benchmark_data_smithi171_122772_object115466:head# 2024-04-20T17:27:46.034 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae4c85a0:::benchmark_data_smithi171_122772_object150611:head# 2024-04-20T17:27:46.034 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae4d43bf:::benchmark_data_smithi171_122772_object154712:head# 2024-04-20T17:27:46.035 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae4d66d4:::benchmark_data_smithi171_122772_object129468:head# 2024-04-20T17:27:46.035 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae4dbbcc:::benchmark_data_smithi171_122772_object90890:head# 2024-04-20T17:27:46.035 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae4e1995:::benchmark_data_smithi171_122772_object121021:head# 2024-04-20T17:27:46.036 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae4eb065:::benchmark_data_smithi171_122772_object118835:head# 2024-04-20T17:27:46.036 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae4f01c6:::benchmark_data_smithi171_122772_object97940:head# 2024-04-20T17:27:46.037 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae4f9f8c:::benchmark_data_smithi171_122772_object132772:head# 2024-04-20T17:27:46.038 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae50373c:::benchmark_data_smithi171_122772_object164851:head# 2024-04-20T17:27:46.038 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae506765:::benchmark_data_smithi171_122772_object161283:head# 2024-04-20T17:27:46.038 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae51220b:::benchmark_data_smithi171_122772_object162338:head# 2024-04-20T17:27:46.039 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae521f02:::benchmark_data_smithi171_122772_object148063:head# 2024-04-20T17:27:46.039 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae52268d:::benchmark_data_smithi171_122772_object83994:head# 2024-04-20T17:27:46.040 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae5270cc:::benchmark_data_smithi171_122772_object165814:head# 2024-04-20T17:27:46.040 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae52a3e3:::benchmark_data_smithi171_122772_object83605:head# 2024-04-20T17:27:46.041 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae52a891:::benchmark_data_smithi171_122772_object165489:head# 2024-04-20T17:27:46.041 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae531d73:::benchmark_data_smithi171_122772_object113811:head# 2024-04-20T17:27:46.042 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae53845e:::benchmark_data_smithi171_122772_object160719:head# 2024-04-20T17:27:46.042 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae54307c:::benchmark_data_smithi171_122772_object158534:head# 2024-04-20T17:27:46.043 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae546772:::benchmark_data_smithi171_122772_object124533:head# 2024-04-20T17:27:46.043 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae54dd17:::benchmark_data_smithi171_122772_object130186:head# 2024-04-20T17:27:46.044 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae54f9fd:::benchmark_data_smithi171_122772_object163987:head# 2024-04-20T17:27:46.044 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae54ffa9:::benchmark_data_smithi171_122772_object101578:head# 2024-04-20T17:27:46.044 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae55dcd3:::benchmark_data_smithi171_122772_object116495:head# 2024-04-20T17:27:46.045 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae563f55:::benchmark_data_smithi171_122772_object96012:head# 2024-04-20T17:27:46.045 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae56be15:::benchmark_data_smithi171_122772_object94274:head# 2024-04-20T17:27:46.046 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae56c8e8:::benchmark_data_smithi171_122772_object139431:head# 2024-04-20T17:27:46.047 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae574281:::benchmark_data_smithi171_122772_object170070:head# 2024-04-20T17:27:46.047 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae57cbfc:::benchmark_data_smithi171_122772_object83361:head# 2024-04-20T17:27:46.048 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae57dc38:::benchmark_data_smithi171_122772_object78720:head# 2024-04-20T17:27:46.048 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae58be36:::benchmark_data_smithi171_122772_object104639:head# 2024-04-20T17:27:46.048 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae58d959:::benchmark_data_smithi171_122772_object109576:head# 2024-04-20T17:27:46.049 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae59177e:::benchmark_data_smithi171_122772_object77984:head# 2024-04-20T17:27:46.049 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae5917cb:::benchmark_data_smithi171_122772_object133915:head# 2024-04-20T17:27:46.050 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae592df1:::benchmark_data_smithi171_122772_object112290:head# 2024-04-20T17:27:46.050 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae59dde0:::benchmark_data_smithi171_122772_object98548:head# 2024-04-20T17:27:46.051 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae5a9222:::benchmark_data_smithi171_122772_object118578:head# 2024-04-20T17:27:46.051 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae5c0900:::benchmark_data_smithi171_122772_object145952:head# 2024-04-20T17:27:46.051 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae5cb951:::benchmark_data_smithi171_122772_object138097:head# 2024-04-20T17:27:46.052 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae5cbe1b:::benchmark_data_smithi171_122772_object163314:head# 2024-04-20T17:27:46.052 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae5d0ab6:::benchmark_data_smithi171_122772_object162046:head# 2024-04-20T17:27:46.053 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae5f429f:::benchmark_data_smithi171_122772_object106776:head# 2024-04-20T17:27:46.053 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae608d47:::benchmark_data_smithi171_122772_object129296:head# 2024-04-20T17:27:46.054 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae60cb36:::benchmark_data_smithi171_122772_object166636:head# 2024-04-20T17:27:46.054 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae61b41c:::benchmark_data_smithi171_122772_object164748:head# 2024-04-20T17:27:46.055 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae625675:::benchmark_data_smithi171_122772_object112174:head# 2024-04-20T17:27:46.055 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae62a4a0:::benchmark_data_smithi171_122772_object146486:head# 2024-04-20T17:27:46.055 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae631662:::benchmark_data_smithi171_122772_object97146:head# 2024-04-20T17:27:46.056 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae63fb81:::benchmark_data_smithi171_122772_object80461:head# 2024-04-20T17:27:46.056 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae64465b:::benchmark_data_smithi171_122772_object115849:head# 2024-04-20T17:27:46.056 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae64702f:::benchmark_data_smithi171_122772_object85907:head# 2024-04-20T17:27:46.057 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae647b0d:::benchmark_data_smithi171_122772_object148264:head# 2024-04-20T17:27:46.057 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae64921c:::benchmark_data_smithi171_122772_object82044:head# 2024-04-20T17:27:46.058 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae65b6bd:::benchmark_data_smithi171_122772_object133363:head# 2024-04-20T17:27:46.058 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae65fd84:::benchmark_data_smithi171_122772_object140157:head# 2024-04-20T17:27:46.059 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae661287:::benchmark_data_smithi171_122772_object146271:head# 2024-04-20T17:27:46.059 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae661911:::benchmark_data_smithi171_122772_object103716:head# 2024-04-20T17:27:46.060 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae66283d:::benchmark_data_smithi171_122772_object98848:head# 2024-04-20T17:27:46.060 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae670e04:::benchmark_data_smithi171_122772_object89875:head# 2024-04-20T17:27:46.060 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae67a399:::benchmark_data_smithi171_122772_object151292:head# 2024-04-20T17:27:46.061 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae67e2a4:::benchmark_data_smithi171_122772_object94527:head# 2024-04-20T17:27:46.061 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae6948eb:::benchmark_data_smithi171_122772_object162132:head# 2024-04-20T17:27:46.062 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae69b04f:::benchmark_data_smithi171_122772_object128624:head# 2024-04-20T17:27:46.062 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae6c96f9:::benchmark_data_smithi171_122772_object95478:head# 2024-04-20T17:27:46.063 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae6dda13:::benchmark_data_smithi171_122772_object88797:head# 2024-04-20T17:27:46.063 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae6ed4db:::benchmark_data_smithi171_122772_object105884:head# 2024-04-20T17:27:46.063 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae6f26ac:::benchmark_data_smithi171_122772_object99977:head# 2024-04-20T17:27:46.064 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae6f3f1a:::benchmark_data_smithi171_122772_object168493:head# 2024-04-20T17:27:46.064 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae707080:::benchmark_data_smithi171_122772_object99694:head# 2024-04-20T17:27:46.065 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae70c378:::benchmark_data_smithi171_122772_object114348:head# 2024-04-20T17:27:46.065 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae70d770:::benchmark_data_smithi171_122772_object144775:head# 2024-04-20T17:27:46.066 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae71ac70:::benchmark_data_smithi171_122772_object149659:head# 2024-04-20T17:27:46.066 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae7255c0:::benchmark_data_smithi171_122772_object163386:head# 2024-04-20T17:27:46.066 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae732b29:::benchmark_data_smithi171_122772_object165071:head# 2024-04-20T17:27:46.067 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae750434:::benchmark_data_smithi171_122772_object87623:head# 2024-04-20T17:27:46.067 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae75be6c:::benchmark_data_smithi171_122772_object139223:head# 2024-04-20T17:27:46.068 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae780848:::benchmark_data_smithi171_122772_object122485:head# 2024-04-20T17:27:46.068 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae781a40:::benchmark_data_smithi171_122772_object122124:head# 2024-04-20T17:27:46.069 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae78d71d:::benchmark_data_smithi171_122772_object131569:head# 2024-04-20T17:27:46.069 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae79ef1a:::benchmark_data_smithi171_122772_object168589:head# 2024-04-20T17:27:46.070 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae7ccf5e:::benchmark_data_smithi171_122772_object119380:head# 2024-04-20T17:27:46.070 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae7d3b7a:::benchmark_data_smithi171_122772_object113166:head# 2024-04-20T17:27:46.071 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae7daf63:::benchmark_data_smithi171_122772_object118409:head# 2024-04-20T17:27:46.071 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae7dd5b1:::benchmark_data_smithi171_122772_object154470:head# 2024-04-20T17:27:46.072 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae7f13da:::benchmark_data_smithi171_122772_object101818:head# 2024-04-20T17:27:46.072 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae7f63b2:::benchmark_data_smithi171_122772_object133986:head# 2024-04-20T17:27:46.073 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae80d9bc:::benchmark_data_smithi171_122772_object141353:head# 2024-04-20T17:27:46.073 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae80f909:::benchmark_data_smithi171_122772_object133632:head# 2024-04-20T17:27:46.073 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae8348fb:::benchmark_data_smithi171_122772_object109545:head# 2024-04-20T17:27:46.074 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae83b3ed:::benchmark_data_smithi171_122772_object89817:head# 2024-04-20T17:27:46.074 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae846493:::benchmark_data_smithi171_122772_object104199:head# 2024-04-20T17:27:46.075 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae8640d4:::benchmark_data_smithi171_122772_object107415:head# 2024-04-20T17:27:46.075 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae867a3f:::benchmark_data_smithi171_122772_object152739:head# 2024-04-20T17:27:46.075 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae86e380:::benchmark_data_smithi171_122772_object80252:head# 2024-04-20T17:27:46.076 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae87906e:::benchmark_data_smithi171_122772_object168594:head# 2024-04-20T17:27:46.077 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae87a5da:::benchmark_data_smithi171_122772_object123383:head# 2024-04-20T17:27:46.077 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae88fbac:::benchmark_data_smithi171_122772_object168991:head# 2024-04-20T17:27:46.078 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae896ab7:::benchmark_data_smithi171_122772_object148645:head# 2024-04-20T17:27:46.078 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae8a1174:::benchmark_data_smithi171_122772_object123380:head# 2024-04-20T17:27:46.078 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae8a7670:::benchmark_data_smithi171_122772_object108176:head# 2024-04-20T17:27:46.079 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae8b07bc:::benchmark_data_smithi171_122772_object158052:head# 2024-04-20T17:27:46.079 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae8c4c93:::benchmark_data_smithi171_122772_object136336:head# 2024-04-20T17:27:46.080 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae8ca826:::benchmark_data_smithi171_122772_object159906:head# 2024-04-20T17:27:46.080 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae8dfd54:::benchmark_data_smithi171_122772_object97229:head# 2024-04-20T17:27:46.081 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae8f5c34:::benchmark_data_smithi171_122772_object154676:head# 2024-04-20T17:27:46.081 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae8fa36c:::benchmark_data_smithi171_122772_object149738:head# 2024-04-20T17:27:46.082 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae8fac54:::benchmark_data_smithi171_122772_object95273:head# 2024-04-20T17:27:46.082 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae8fc797:::benchmark_data_smithi171_122772_object133205:head# 2024-04-20T17:27:46.083 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae90092a:::benchmark_data_smithi171_122772_object161051:head# 2024-04-20T17:27:46.083 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae901142:::benchmark_data_smithi171_122772_object157035:head# 2024-04-20T17:27:46.084 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae906cfd:::benchmark_data_smithi171_122772_object140867:head# 2024-04-20T17:27:46.084 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae91112b:::benchmark_data_smithi171_122772_object162332:head# 2024-04-20T17:27:46.085 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae916ef2:::benchmark_data_smithi171_122772_object145255:head# 2024-04-20T17:27:46.085 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae91a480:::benchmark_data_smithi171_122772_object141266:head# 2024-04-20T17:27:46.086 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae93e39c:::benchmark_data_smithi171_122772_object78627:head# 2024-04-20T17:27:46.086 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae953d5b:::benchmark_data_smithi171_122772_object164005:head# 2024-04-20T17:27:46.087 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae9593f6:::benchmark_data_smithi171_122772_object100624:head# 2024-04-20T17:27:46.087 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae95a861:::benchmark_data_smithi171_122772_object144999:head# 2024-04-20T17:27:46.088 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae96aa37:::benchmark_data_smithi171_122772_object116302:head# 2024-04-20T17:27:46.088 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae97d67d:::benchmark_data_smithi171_122772_object161023:head# 2024-04-20T17:27:46.089 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae99e783:::benchmark_data_smithi171_122772_object84951:head# 2024-04-20T17:27:46.089 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae9ab1d9:::benchmark_data_smithi171_122772_object146955:head# 2024-04-20T17:27:46.090 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae9af7f2:::benchmark_data_smithi171_122772_object101260:head# 2024-04-20T17:27:46.090 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae9afbd8:::benchmark_data_smithi171_122772_object107507:head# 2024-04-20T17:27:46.091 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae9bbd98:::benchmark_data_smithi171_122772_object158194:head# 2024-04-20T17:27:46.091 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae9c0033:::benchmark_data_smithi171_122772_object151239:head# 2024-04-20T17:27:46.092 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae9c0def:::benchmark_data_smithi171_122772_object96269:head# 2024-04-20T17:27:46.092 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae9ced93:::benchmark_data_smithi171_122772_object140070:head# 2024-04-20T17:27:46.093 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae9d282e:::benchmark_data_smithi171_122772_object128181:head# 2024-04-20T17:27:46.093 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae9d3eda:::benchmark_data_smithi171_122772_object129439:head# 2024-04-20T17:27:46.094 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae9ea756:::benchmark_data_smithi171_122772_object159758:head# 2024-04-20T17:27:46.094 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae9faeec:::benchmark_data_smithi171_122772_object97120:head# 2024-04-20T17:27:46.095 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ae9fe3d4:::benchmark_data_smithi171_122772_object110701:head# 2024-04-20T17:27:46.095 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aea25b23:::benchmark_data_smithi171_122772_object91116:head# 2024-04-20T17:27:46.096 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aea2df9c:::benchmark_data_smithi171_122772_object92651:head# 2024-04-20T17:27:46.096 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aea375a3:::benchmark_data_smithi171_122772_object129177:head# 2024-04-20T17:27:46.097 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aea3ebc6:::benchmark_data_smithi171_122772_object115295:head# 2024-04-20T17:27:46.097 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aea43016:::benchmark_data_smithi171_122772_object114162:head# 2024-04-20T17:27:46.098 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aea52c71:::benchmark_data_smithi171_122772_object105101:head# 2024-04-20T17:27:46.098 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aea5ded9:::benchmark_data_smithi171_122772_object128602:head# 2024-04-20T17:27:46.098 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aea6f5b5:::benchmark_data_smithi171_122772_object89220:head# 2024-04-20T17:27:46.099 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aea7bbf2:::benchmark_data_smithi171_122772_object151855:head# 2024-04-20T17:27:46.099 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aea832ff:::benchmark_data_smithi171_122772_object115346:head# 2024-04-20T17:27:46.100 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aea91af4:::benchmark_data_smithi171_122772_object90003:head# 2024-04-20T17:27:46.101 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aea971d7:::benchmark_data_smithi171_122772_object155002:head# 2024-04-20T17:27:46.101 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeaabefc:::benchmark_data_smithi171_122772_object109260:head# 2024-04-20T17:27:46.101 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeaaebdd:::benchmark_data_smithi171_122772_object123225:head# 2024-04-20T17:27:46.102 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeab1c4e:::benchmark_data_smithi171_122772_object98089:head# 2024-04-20T17:27:46.102 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeada183:::benchmark_data_smithi171_122772_object119058:head# 2024-04-20T17:27:46.102 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeadd269:::benchmark_data_smithi171_122772_object136584:head# 2024-04-20T17:27:46.103 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeae8dfa:::benchmark_data_smithi171_122772_object104900:head# 2024-04-20T17:27:46.103 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeafb5b9:::benchmark_data_smithi171_122772_object166426:head# 2024-04-20T17:27:46.104 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeb11dd8:::benchmark_data_smithi171_122772_object80822:head# 2024-04-20T17:27:46.104 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeb14435:::benchmark_data_smithi171_122772_object165467:head# 2024-04-20T17:27:46.104 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeb1e7d8:::benchmark_data_smithi171_122772_object168426:head# 2024-04-20T17:27:46.105 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeb291a6:::benchmark_data_smithi171_122772_object133088:head# 2024-04-20T17:27:46.105 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeb359a3:::benchmark_data_smithi171_122772_object155253:head# 2024-04-20T17:27:46.105 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeb39555:::benchmark_data_smithi171_122772_object93776:head# 2024-04-20T17:27:46.106 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeb4d6a0:::benchmark_data_smithi171_122772_object166017:head# 2024-04-20T17:27:46.107 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeb6cbba:::benchmark_data_smithi171_122772_object125331:head# 2024-04-20T17:27:46.107 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeb7172d:::benchmark_data_smithi171_122772_object97938:head# 2024-04-20T17:27:46.107 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeb9138a:::benchmark_data_smithi171_122772_object97253:head# 2024-04-20T17:27:46.108 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeb91419:::benchmark_data_smithi171_122772_object94392:head# 2024-04-20T17:27:46.108 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aebc407f:::benchmark_data_smithi171_122772_object78070:head# 2024-04-20T17:27:46.108 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aebc53ab:::benchmark_data_smithi171_122772_object142889:head# 2024-04-20T17:27:46.109 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aebc6713:::benchmark_data_smithi171_122772_object144989:head# 2024-04-20T17:27:46.109 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aebcd1ed:::benchmark_data_smithi171_122772_object162169:head# 2024-04-20T17:27:46.110 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aebd8ad2:::benchmark_data_smithi171_122772_object124493:head# 2024-04-20T17:27:46.110 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aebe6244:::benchmark_data_smithi171_122772_object122503:head# 2024-04-20T17:27:46.111 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aebe8fb6:::benchmark_data_smithi171_122772_object145321:head# 2024-04-20T17:27:46.111 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aebecbaf:::benchmark_data_smithi171_122772_object107177:head# 2024-04-20T17:27:46.112 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aebf5f14:::benchmark_data_smithi171_122772_object135596:head# 2024-04-20T17:27:46.112 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aebf813d:::benchmark_data_smithi171_122772_object101670:head# 2024-04-20T17:27:46.113 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aec06376:::benchmark_data_smithi171_122772_object101227:head# 2024-04-20T17:27:46.113 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aec0660c:::benchmark_data_smithi171_122772_object117977:head# 2024-04-20T17:27:46.114 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aec17987:::benchmark_data_smithi171_122772_object120368:head# 2024-04-20T17:27:46.114 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aec245e9:::benchmark_data_smithi171_122772_object104778:head# 2024-04-20T17:27:46.114 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aec28e78:::benchmark_data_smithi171_122772_object108631:head# 2024-04-20T17:27:46.115 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aec313de:::benchmark_data_smithi171_122772_object151661:head# 2024-04-20T17:27:46.115 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aec36963:::benchmark_data_smithi171_122772_object122943:head# 2024-04-20T17:27:46.116 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aec408a2:::benchmark_data_smithi171_122772_object153425:head# 2024-04-20T17:27:46.116 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aec57ed8:::benchmark_data_smithi171_122772_object109584:head# 2024-04-20T17:27:46.117 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aec5b309:::benchmark_data_smithi171_122772_object166081:head# 2024-04-20T17:27:46.117 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aec6244f:::benchmark_data_smithi171_122772_object145677:head# 2024-04-20T17:27:46.117 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aec848ae:::benchmark_data_smithi171_122772_object118695:head# 2024-04-20T17:27:46.118 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aec8a87a:::benchmark_data_smithi171_122772_object141043:head# 2024-04-20T17:27:46.119 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aec9133c:::benchmark_data_smithi171_122772_object125958:head# 2024-04-20T17:27:46.119 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aecac376:::benchmark_data_smithi171_122772_object139519:head# 2024-04-20T17:27:46.119 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aecb215f:::benchmark_data_smithi171_122772_object140996:head# 2024-04-20T17:27:46.120 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aecb5ab7:::benchmark_data_smithi171_122772_object145442:head# 2024-04-20T17:27:46.120 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aecce966:::benchmark_data_smithi171_122772_object138979:head# 2024-04-20T17:27:46.121 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aecd7564:::benchmark_data_smithi171_122772_object87344:head# 2024-04-20T17:27:46.121 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aecd8db1:::benchmark_data_smithi171_122772_object84828:head# 2024-04-20T17:27:46.122 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aecdf76a:::benchmark_data_smithi171_122772_object108077:head# 2024-04-20T17:27:46.122 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aecdfcb7:::benchmark_data_smithi171_122772_object92836:head# 2024-04-20T17:27:46.123 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeceb341:::benchmark_data_smithi171_122772_object146641:head# 2024-04-20T17:27:46.123 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aecf0d22:::benchmark_data_smithi171_122772_object135548:head# 2024-04-20T17:27:46.123 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aecf7468:::benchmark_data_smithi171_122772_object87430:head# 2024-04-20T17:27:46.124 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aecf9d35:::benchmark_data_smithi171_122772_object84919:head# 2024-04-20T17:27:46.124 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aecfeebd:::benchmark_data_smithi171_122772_object140551:head# 2024-04-20T17:27:46.125 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aecff2bc:::benchmark_data_smithi171_122772_object124393:head# 2024-04-20T17:27:46.125 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aed01af9:::benchmark_data_smithi171_122772_object157213:head# 2024-04-20T17:27:46.125 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aed05aa7:::benchmark_data_smithi171_122772_object117275:head# 2024-04-20T17:27:46.126 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aed19578:::benchmark_data_smithi171_122772_object95453:head# 2024-04-20T17:27:46.126 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aed265d1:::benchmark_data_smithi171_122772_object105685:head# 2024-04-20T17:27:46.127 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aed513ba:::benchmark_data_smithi171_122772_object100761:head# 2024-04-20T17:27:46.127 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aed6c1a6:::benchmark_data_smithi171_122772_object126350:head# 2024-04-20T17:27:46.127 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aed7efcb:::benchmark_data_smithi171_122772_object124362:head# 2024-04-20T17:27:46.128 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aed89cd5:::benchmark_data_smithi171_122772_object160138:head# 2024-04-20T17:27:46.128 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aed8b52b:::benchmark_data_smithi171_122772_object161600:head# 2024-04-20T17:27:46.129 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aed8c7fb:::benchmark_data_smithi171_122772_object108947:head# 2024-04-20T17:27:46.129 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aed8f658:::benchmark_data_smithi171_122772_object151491:head# 2024-04-20T17:27:46.130 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aed94509:::benchmark_data_smithi171_122772_object151076:head# 2024-04-20T17:27:46.130 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeda2ef8:::benchmark_data_smithi171_122772_object85427:head# 2024-04-20T17:27:46.131 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeda7b65:::benchmark_data_smithi171_122772_object104140:head# 2024-04-20T17:27:46.131 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aedb1ecb:::benchmark_data_smithi171_122772_object125483:head# 2024-04-20T17:27:46.131 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aedd85be:::benchmark_data_smithi171_122772_object88775:head# 2024-04-20T17:27:46.132 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeddb420:::benchmark_data_smithi171_122772_object156890:head# 2024-04-20T17:27:46.132 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeddcf6d:::benchmark_data_smithi171_122772_object149857:head# 2024-04-20T17:27:46.133 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aede7ac6:::benchmark_data_smithi171_122772_object145898:head# 2024-04-20T17:27:46.133 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aedf5bc9:::benchmark_data_smithi171_122772_object166884:head# 2024-04-20T17:27:46.133 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aee00701:::benchmark_data_smithi171_122772_object169038:head# 2024-04-20T17:27:46.134 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aee05389:::benchmark_data_smithi171_122772_object159540:head# 2024-04-20T17:27:46.134 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aee11f83:::benchmark_data_smithi171_122772_object169719:head# 2024-04-20T17:27:46.135 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aee160e7:::benchmark_data_smithi171_122772_object113327:head# 2024-04-20T17:27:46.135 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aee2264e:::benchmark_data_smithi171_122772_object93331:head# 2024-04-20T17:27:46.136 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aee3b069:::benchmark_data_smithi171_122772_object103858:head# 2024-04-20T17:27:46.136 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aee41420:::benchmark_data_smithi171_122772_object92479:head# 2024-04-20T17:27:46.137 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aee5314f:::benchmark_data_smithi171_122772_object144937:head# 2024-04-20T17:27:46.137 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aee5e264:::benchmark_data_smithi171_122772_object140491:head# 2024-04-20T17:27:46.138 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aee68573:::benchmark_data_smithi171_122772_object107859:head# 2024-04-20T17:27:46.138 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aee7b905:::benchmark_data_smithi171_122772_object89184:head# 2024-04-20T17:27:46.139 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aee7f675:::benchmark_data_smithi171_122772_object79596:head# 2024-04-20T17:27:46.139 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aee84150:::benchmark_data_smithi171_122772_object86777:head# 2024-04-20T17:27:46.139 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aee96b2c:::benchmark_data_smithi171_122772_object84120:head# 2024-04-20T17:27:46.140 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aee9968a:::benchmark_data_smithi171_122772_object94414:head# 2024-04-20T17:27:46.140 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aee9fa0f:::benchmark_data_smithi171_122772_object162811:head# 2024-04-20T17:27:46.141 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeebd7f5:::benchmark_data_smithi171_122772_object127750:head# 2024-04-20T17:27:46.141 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeec95d5:::benchmark_data_smithi171_122772_object143309:head# 2024-04-20T17:27:46.142 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeed53ae:::benchmark_data_smithi171_122772_object165040:head# 2024-04-20T17:27:46.142 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeed9a7a:::benchmark_data_smithi171_122772_object106239:head# 2024-04-20T17:27:46.142 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeef5f1a:::benchmark_data_smithi171_122772_object126714:head# 2024-04-20T17:27:46.143 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeefca1f:::benchmark_data_smithi171_122772_object99062:head# 2024-04-20T17:27:46.144 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aef02110:::benchmark_data_smithi171_122772_object107961:head# 2024-04-20T17:27:46.144 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aef0a46f:::benchmark_data_smithi171_122772_object160555:head# 2024-04-20T17:27:46.144 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aef100f6:::benchmark_data_smithi171_122772_object163696:head# 2024-04-20T17:27:46.145 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aef12d3a:::benchmark_data_smithi171_122772_object91685:head# 2024-04-20T17:27:46.145 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aef14993:::benchmark_data_smithi171_122772_object118242:head# 2024-04-20T17:27:46.146 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aef14ad2:::benchmark_data_smithi171_122772_object150112:head# 2024-04-20T17:27:46.146 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aef200f3:::benchmark_data_smithi171_122772_object141853:head# 2024-04-20T17:27:46.147 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aef203d2:::benchmark_data_smithi171_122772_object126104:head# 2024-04-20T17:27:46.147 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aef24b32:::benchmark_data_smithi171_122772_object88895:head# 2024-04-20T17:27:46.148 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aef2505b:::benchmark_data_smithi171_122772_object125263:head# 2024-04-20T17:27:46.148 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aef27639:::benchmark_data_smithi171_122772_object86121:head# 2024-04-20T17:27:46.149 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aef29fb6:::benchmark_data_smithi171_122772_object116675:head# 2024-04-20T17:27:46.149 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aef3a8e2:::benchmark_data_smithi171_122772_object81461:head# 2024-04-20T17:27:46.149 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aef44cf0:::benchmark_data_smithi171_122772_object155376:head# 2024-04-20T17:27:46.150 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aef529ce:::benchmark_data_smithi171_122772_object104800:head# 2024-04-20T17:27:46.150 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aef54c93:::benchmark_data_smithi171_122772_object152008:head# 2024-04-20T17:27:46.151 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aef54ef8:::benchmark_data_smithi171_122772_object95978:head# 2024-04-20T17:27:46.151 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aef7191f:::benchmark_data_smithi171_122772_object106384:head# 2024-04-20T17:27:46.152 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aef783ce:::benchmark_data_smithi171_122772_object167469:head# 2024-04-20T17:27:46.152 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aef7caf5:::benchmark_data_smithi171_122772_object122282:head# 2024-04-20T17:27:46.153 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aef7d990:::benchmark_data_smithi171_122772_object131461:head# 2024-04-20T17:27:46.154 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aef82975:::benchmark_data_smithi171_122772_object116903:head# 2024-04-20T17:27:46.154 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aef8612d:::benchmark_data_smithi171_122772_object95996:head# 2024-04-20T17:27:46.154 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aef87a6a:::benchmark_data_smithi171_122772_object167210:head# 2024-04-20T17:27:46.155 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aef96823:::benchmark_data_smithi171_122772_object164092:head# 2024-04-20T17:27:46.155 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aefadfe6:::benchmark_data_smithi171_122772_object104597:head# 2024-04-20T17:27:46.156 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aefb7e2b:::benchmark_data_smithi171_122772_object112729:head# 2024-04-20T17:27:46.156 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aefeb281:::benchmark_data_smithi171_122772_object106182:head# 2024-04-20T17:27:46.157 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aeff8d06:::benchmark_data_smithi171_122772_object159429:head# 2024-04-20T17:27:46.157 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af001f77:::benchmark_data_smithi171_122772_object106905:head# 2024-04-20T17:27:46.157 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af00388b:::benchmark_data_smithi171_122772_object160413:head# 2024-04-20T17:27:46.158 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af02f028:::benchmark_data_smithi171_122772_object107983:head# 2024-04-20T17:27:46.159 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af0324da:::benchmark_data_smithi171_122772_object142101:head# 2024-04-20T17:27:46.159 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af049a8c:::benchmark_data_smithi171_122772_object116099:head# 2024-04-20T17:27:46.160 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af04a2e6:::benchmark_data_smithi171_122772_object147140:head# 2024-04-20T17:27:46.160 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af04ff70:::benchmark_data_smithi171_122772_object102001:head# 2024-04-20T17:27:46.160 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af055523:::benchmark_data_smithi171_122772_object87642:head# 2024-04-20T17:27:46.161 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af0594a8:::benchmark_data_smithi171_122772_object104086:head# 2024-04-20T17:27:46.161 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af0640a0:::benchmark_data_smithi171_122772_object93855:head# 2024-04-20T17:27:46.162 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af077c1d:::benchmark_data_smithi171_122772_object126736:head# 2024-04-20T17:27:46.162 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af078236:::benchmark_data_smithi171_122772_object136930:head# 2024-04-20T17:27:46.162 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af07c4ee:::benchmark_data_smithi171_122772_object99048:head# 2024-04-20T17:27:46.163 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af085405:::benchmark_data_smithi171_122772_object165121:head# 2024-04-20T17:27:46.163 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af0aa6d8:::benchmark_data_smithi171_122772_object166456:head# 2024-04-20T17:27:46.164 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af0af0c4:::benchmark_data_smithi171_122772_object115787:head# 2024-04-20T17:27:46.164 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af0b0a00:::benchmark_data_smithi171_122772_object91603:head# 2024-04-20T17:27:46.165 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af0b8d18:::benchmark_data_smithi171_122772_object118137:head# 2024-04-20T17:27:46.165 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af0c3b42:::benchmark_data_smithi171_122772_object136816:head# 2024-04-20T17:27:46.165 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af0c42c4:::benchmark_data_smithi171_122772_object115402:head# 2024-04-20T17:27:46.166 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af0c6a9e:::benchmark_data_smithi171_122772_object148924:head# 2024-04-20T17:27:46.166 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af0c9490:::benchmark_data_smithi171_122772_object139724:head# 2024-04-20T17:27:46.167 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af0cf3af:::benchmark_data_smithi171_122772_object114171:head# 2024-04-20T17:27:46.167 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af0dccae:::benchmark_data_smithi171_122772_object156991:head# 2024-04-20T17:27:46.168 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af0ef4e9:::benchmark_data_smithi171_122772_object152365:head# 2024-04-20T17:27:46.168 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af0f4d2d:::benchmark_data_smithi171_122772_object79982:head# 2024-04-20T17:27:46.169 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af0f92da:::benchmark_data_smithi171_122772_object152286:head# 2024-04-20T17:27:46.169 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af0fcaa1:::benchmark_data_smithi171_122772_object126635:head# 2024-04-20T17:27:46.170 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af0feb55:::benchmark_data_smithi171_122772_object92639:head# 2024-04-20T17:27:46.170 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af10bcbd:::benchmark_data_smithi171_122772_object126291:head# 2024-04-20T17:27:46.170 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af10bd42:::benchmark_data_smithi171_122772_object168894:head# 2024-04-20T17:27:46.171 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af111d1f:::benchmark_data_smithi171_122772_object113177:head# 2024-04-20T17:27:46.171 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af112e8e:::benchmark_data_smithi171_122772_object151019:head# 2024-04-20T17:27:46.172 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af124336:::benchmark_data_smithi171_122772_object93878:head# 2024-04-20T17:27:46.172 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af1349a5:::benchmark_data_smithi171_122772_object102280:head# 2024-04-20T17:27:46.173 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af1421f8:::benchmark_data_smithi171_122772_object105746:head# 2024-04-20T17:27:46.173 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af15f97a:::benchmark_data_smithi171_122772_object153081:head# 2024-04-20T17:27:46.174 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af17fe2c:::benchmark_data_smithi171_122772_object147773:head# 2024-04-20T17:27:46.174 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af18f2f3:::benchmark_data_smithi171_122772_object147433:head# 2024-04-20T17:27:46.175 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af197023:::benchmark_data_smithi171_122772_object106085:head# 2024-04-20T17:27:46.175 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af197738:::benchmark_data_smithi171_122772_object89986:head# 2024-04-20T17:27:46.175 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af1a980a:::benchmark_data_smithi171_122772_object146061:head# 2024-04-20T17:27:46.176 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af1b6896:::benchmark_data_smithi171_122772_object96276:head# 2024-04-20T17:27:46.176 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af1b6f05:::benchmark_data_smithi171_122772_object78932:head# 2024-04-20T17:27:46.177 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af1f9afc:::benchmark_data_smithi171_122772_object168792:head# 2024-04-20T17:27:46.177 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af1fe5ed:::benchmark_data_smithi171_122772_object154282:head# 2024-04-20T17:27:46.178 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af215da3:::benchmark_data_smithi171_122772_object160370:head# 2024-04-20T17:27:46.178 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af21b54b:::benchmark_data_smithi171_122772_object78005:head# 2024-04-20T17:27:46.179 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af22277d:::benchmark_data_smithi171_122772_object128687:head# 2024-04-20T17:27:46.179 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af231f30:::benchmark_data_smithi171_122772_object116861:head# 2024-04-20T17:27:46.180 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af236d38:::benchmark_data_smithi171_122772_object144966:head# 2024-04-20T17:27:46.180 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af23eea6:::benchmark_data_smithi171_122772_object154457:head# 2024-04-20T17:27:46.180 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af256505:::benchmark_data_smithi171_122772_object148534:head# 2024-04-20T17:27:46.181 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af265bab:::benchmark_data_smithi171_122772_object150750:head# 2024-04-20T17:27:46.182 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af287d28:::benchmark_data_smithi171_122772_object146563:head# 2024-04-20T17:27:46.182 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af28f808:::benchmark_data_smithi171_122772_object105791:head# 2024-04-20T17:27:46.182 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af294e95:::benchmark_data_smithi171_122772_object146207:head# 2024-04-20T17:27:46.183 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af2990df:::benchmark_data_smithi171_122772_object128905:head# 2024-04-20T17:27:46.183 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af29c2b1:::benchmark_data_smithi171_122772_object99469:head# 2024-04-20T17:27:46.183 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af2c41ed:::benchmark_data_smithi171_122772_object99199:head# 2024-04-20T17:27:46.184 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af2c4962:::benchmark_data_smithi171_122772_object112415:head# 2024-04-20T17:27:46.184 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af2cf566:::benchmark_data_smithi171_122772_object92390:head# 2024-04-20T17:27:46.185 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af2d22dd:::benchmark_data_smithi171_122772_object154966:head# 2024-04-20T17:27:46.185 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af2d6c07:::benchmark_data_smithi171_122772_object122964:head# 2024-04-20T17:27:46.185 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af2de6b6:::benchmark_data_smithi171_122772_object128646:head# 2024-04-20T17:27:46.186 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af2ebc36:::benchmark_data_smithi171_122772_object96458:head# 2024-04-20T17:27:46.187 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af2ed2bb:::benchmark_data_smithi171_122772_object152769:head# 2024-04-20T17:27:46.187 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af2f6ad8:::benchmark_data_smithi171_122772_object142319:head# 2024-04-20T17:27:46.187 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af2fcc61:::benchmark_data_smithi171_122772_object115687:head# 2024-04-20T17:27:46.188 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af31600a:::benchmark_data_smithi171_122772_object89323:head# 2024-04-20T17:27:46.188 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af3219ba:::benchmark_data_smithi171_122772_object83885:head# 2024-04-20T17:27:46.189 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af327d49:::benchmark_data_smithi171_122772_object130713:head# 2024-04-20T17:27:46.189 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af32a280:::benchmark_data_smithi171_122772_object157549:head# 2024-04-20T17:27:46.189 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af334568:::benchmark_data_smithi171_122772_object82461:head# 2024-04-20T17:27:46.190 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af3350ec:::benchmark_data_smithi171_122772_object163481:head# 2024-04-20T17:27:46.190 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af337520:::benchmark_data_smithi171_122772_object142240:head# 2024-04-20T17:27:46.191 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af34c3d7:::benchmark_data_smithi171_122772_object165566:head# 2024-04-20T17:27:46.191 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af35c7f2:::benchmark_data_smithi171_122772_object136782:head# 2024-04-20T17:27:46.192 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af35d7fe:::benchmark_data_smithi171_122772_object159163:head# 2024-04-20T17:27:46.192 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af35e785:::benchmark_data_smithi171_122772_object96265:head# 2024-04-20T17:27:46.193 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af36feb7:::benchmark_data_smithi171_122772_object160566:head# 2024-04-20T17:27:46.193 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af376d3b:::benchmark_data_smithi171_122772_object147826:head# 2024-04-20T17:27:46.193 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af37a597:::benchmark_data_smithi171_122772_object94714:head# 2024-04-20T17:27:46.194 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af3829eb:::benchmark_data_smithi171_122772_object142644:head# 2024-04-20T17:27:46.194 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af388f48:::benchmark_data_smithi171_122772_object136449:head# 2024-04-20T17:27:46.195 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af3945d0:::benchmark_data_smithi171_122772_object100702:head# 2024-04-20T17:27:46.195 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af3a03a1:::benchmark_data_smithi171_122772_object167633:head# 2024-04-20T17:27:46.196 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af3c34c6:::benchmark_data_smithi171_122772_object127389:head# 2024-04-20T17:27:46.196 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af3c63cb:::benchmark_data_smithi171_122772_object136459:head# 2024-04-20T17:27:46.197 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af3c7c15:::benchmark_data_smithi171_122772_object104903:head# 2024-04-20T17:27:46.197 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af3e9d13:::benchmark_data_smithi171_122772_object128233:head# 2024-04-20T17:27:46.197 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af3f5b83:::benchmark_data_smithi171_122772_object136954:head# 2024-04-20T17:27:46.201 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af3fdddb:::benchmark_data_smithi171_122772_object168486:head# 2024-04-20T17:27:46.202 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af3fed90:::benchmark_data_smithi171_122772_object138895:head# 2024-04-20T17:27:46.202 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af408659:::benchmark_data_smithi171_122772_object140825:head# 2024-04-20T17:27:46.202 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af40decd:::benchmark_data_smithi171_122772_object95911:head# 2024-04-20T17:27:46.204 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af4521f3:::benchmark_data_smithi171_122772_object104242:head# 2024-04-20T17:27:46.204 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af456ea7:::benchmark_data_smithi171_122772_object120244:head# 2024-04-20T17:27:46.205 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af45a946:::benchmark_data_smithi171_122772_object112213:head# 2024-04-20T17:27:46.205 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af468800:::benchmark_data_smithi171_122772_object133904:head# 2024-04-20T17:27:46.205 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af469cde:::benchmark_data_smithi171_122772_object168896:head# 2024-04-20T17:27:46.206 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af469f87:::benchmark_data_smithi171_122772_object80114:head# 2024-04-20T17:27:46.206 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af46d195:::benchmark_data_smithi171_122772_object81719:head# 2024-04-20T17:27:46.206 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af477fbb:::benchmark_data_smithi171_122772_object130518:head# 2024-04-20T17:27:46.207 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af4a4068:::benchmark_data_smithi171_122772_object117500:head# 2024-04-20T17:27:46.207 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af4ab5f1:::benchmark_data_smithi171_122772_object168291:head# 2024-04-20T17:27:46.208 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af4d5c74:::benchmark_data_smithi171_122772_object91232:head# 2024-04-20T17:27:46.208 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af4e1a5c:::benchmark_data_smithi171_122772_object96594:head# 2024-04-20T17:27:46.208 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af4ead33:::benchmark_data_smithi171_122772_object163709:head# 2024-04-20T17:27:46.209 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af4f3ab9:::benchmark_data_smithi171_122772_object152048:head# 2024-04-20T17:27:46.209 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af501448:::benchmark_data_smithi171_122772_object133253:head# 2024-04-20T17:27:46.209 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af54975e:::benchmark_data_smithi171_122772_object140364:head# 2024-04-20T17:27:46.210 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af54b043:::benchmark_data_smithi171_122772_object103757:head# 2024-04-20T17:27:46.210 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af54e979:::benchmark_data_smithi171_122772_object86585:head# 2024-04-20T17:27:46.211 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af556bd3:::benchmark_data_smithi171_122772_object135021:head# 2024-04-20T17:27:46.214 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af574a83:::benchmark_data_smithi171_122772_object126480:head# 2024-04-20T17:27:46.215 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af587f82:::benchmark_data_smithi171_122772_object83304:head# 2024-04-20T17:27:46.215 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af5950a7:::benchmark_data_smithi171_122772_object125883:head# 2024-04-20T17:27:46.216 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af5a6ebb:::benchmark_data_smithi171_122772_object96577:head# 2024-04-20T17:27:46.216 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af5ac45f:::benchmark_data_smithi171_122772_object94095:head# 2024-04-20T17:27:46.217 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af5b5c58:::benchmark_data_smithi171_122772_object164179:head# 2024-04-20T17:27:46.218 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af5b624c:::benchmark_data_smithi171_122772_object85259:head# 2024-04-20T17:27:46.218 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af5c7149:::benchmark_data_smithi171_122772_object133694:head# 2024-04-20T17:27:46.219 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af6060bd:::benchmark_data_smithi171_122772_object123753:head# 2024-04-20T17:27:46.219 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af61578e:::benchmark_data_smithi171_122772_object83486:head# 2024-04-20T17:27:46.219 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af61bbd4:::benchmark_data_smithi171_122772_object152095:head# 2024-04-20T17:27:46.220 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af62ac95:::benchmark_data_smithi171_122772_object119267:head# 2024-04-20T17:27:46.220 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af63955b:::benchmark_data_smithi171_122772_object117676:head# 2024-04-20T17:27:46.220 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af63b598:::benchmark_data_smithi171_122772_object152310:head# 2024-04-20T17:27:46.221 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af6413f9:::benchmark_data_smithi171_122772_object156509:head# 2024-04-20T17:27:46.221 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af64aef1:::benchmark_data_smithi171_122772_object128590:head# 2024-04-20T17:27:46.222 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af6576a4:::benchmark_data_smithi171_122772_object80695:head# 2024-04-20T17:27:46.222 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af65ee98:::benchmark_data_smithi171_122772_object149405:head# 2024-04-20T17:27:46.223 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af65f1cd:::benchmark_data_smithi171_122772_object160208:head# 2024-04-20T17:27:46.223 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af65fbf9:::benchmark_data_smithi171_122772_object103375:head# 2024-04-20T17:27:46.223 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af6692cf:::benchmark_data_smithi171_122772_object114195:head# 2024-04-20T17:27:46.224 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af68707f:::benchmark_data_smithi171_122772_object131269:head# 2024-04-20T17:27:46.224 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af68e003:::benchmark_data_smithi171_122772_object135072:head# 2024-04-20T17:27:46.224 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af69d2e5:::benchmark_data_smithi171_122772_object153777:head# 2024-04-20T17:27:46.225 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af6a2df2:::benchmark_data_smithi171_122772_object170180:head# 2024-04-20T17:27:46.225 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af6a3283:::benchmark_data_smithi171_122772_object89977:head# 2024-04-20T17:27:46.226 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af6b3582:::benchmark_data_smithi171_122772_object167449:head# 2024-04-20T17:27:46.229 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af6bb9b5:::benchmark_data_smithi171_122772_object162548:head# 2024-04-20T17:27:46.231 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af6beefc:::benchmark_data_smithi171_122772_object158635:head# 2024-04-20T17:27:46.232 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af6c6f29:::benchmark_data_smithi171_122772_object113168:head# 2024-04-20T17:27:46.232 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af6cba4f:::benchmark_data_smithi171_122772_object129009:head# 2024-04-20T17:27:46.233 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af6d3aae:::benchmark_data_smithi171_122772_object141324:head# 2024-04-20T17:27:46.233 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af6d5bb2:::benchmark_data_smithi171_122772_object88906:head# 2024-04-20T17:27:46.233 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af6d6f69:::benchmark_data_smithi171_122772_object121040:head# 2024-04-20T17:27:46.234 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af6e1dae:::benchmark_data_smithi171_122772_object104211:head# 2024-04-20T17:27:46.235 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af6ef414:::benchmark_data_smithi171_122772_object144496:head# 2024-04-20T17:27:46.235 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af6ef4d3:::benchmark_data_smithi171_122772_object152555:head# 2024-04-20T17:27:46.236 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af6f3260:::benchmark_data_smithi171_122772_object98438:head# 2024-04-20T17:27:46.236 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af6f3482:::benchmark_data_smithi171_122772_object159837:head# 2024-04-20T17:27:46.236 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af6f34b8:::benchmark_data_smithi171_122772_object79508:head# 2024-04-20T17:27:46.237 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af6f3a7b:::benchmark_data_smithi171_122772_object125442:head# 2024-04-20T17:27:46.237 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af6f884f:::benchmark_data_smithi171_122772_object107371:head# 2024-04-20T17:27:46.238 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af709bf2:::benchmark_data_smithi171_122772_object149418:head# 2024-04-20T17:27:46.239 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af712c1c:::benchmark_data_smithi171_122772_object169772:head# 2024-04-20T17:27:46.239 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af71b7c2:::benchmark_data_smithi171_122772_object134643:head# 2024-04-20T17:27:46.239 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af724dd3:::benchmark_data_smithi171_122772_object82648:head# 2024-04-20T17:27:46.240 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af72c3e1:::benchmark_data_smithi171_122772_object129130:head# 2024-04-20T17:27:46.242 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af73a8a7:::benchmark_data_smithi171_122772_object163750:head# 2024-04-20T17:27:46.242 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af73a9ed:::benchmark_data_smithi171_122772_object81256:head# 2024-04-20T17:27:46.243 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af74c506:::benchmark_data_smithi171_122772_object82031:head# 2024-04-20T17:27:46.243 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af7514db:::benchmark_data_smithi171_122772_object81337:head# 2024-04-20T17:27:46.243 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af75d50d:::benchmark_data_smithi171_122772_object151269:head# 2024-04-20T17:27:46.243 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af78c233:::benchmark_data_smithi171_122772_object148009:head# 2024-04-20T17:27:46.243 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af7917c2:::benchmark_data_smithi171_122772_object143319:head# 2024-04-20T17:27:46.243 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af7ac07d:::benchmark_data_smithi171_122772_object86039:head# 2024-04-20T17:27:46.244 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af7c155b:::benchmark_data_smithi171_122772_object79319:head# 2024-04-20T17:27:46.244 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af7c42cf:::benchmark_data_smithi171_122772_object159181:head# 2024-04-20T17:27:46.244 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af7c4d0c:::benchmark_data_smithi171_122772_object162837:head# 2024-04-20T17:27:46.245 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af7d0507:::benchmark_data_smithi171_122772_object94932:head# 2024-04-20T17:27:46.245 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af7dcf57:::benchmark_data_smithi171_122772_object135874:head# 2024-04-20T17:27:46.246 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af7ea145:::benchmark_data_smithi171_122772_object151745:head# 2024-04-20T17:27:46.246 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af7eb14e:::benchmark_data_smithi171_122772_object127974:head# 2024-04-20T17:27:46.247 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af7fbde5:::benchmark_data_smithi171_122772_object81301:head# 2024-04-20T17:27:46.248 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af814483:::benchmark_data_smithi171_122772_object123568:head# 2024-04-20T17:27:46.248 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af81dda7:::benchmark_data_smithi171_122772_object145766:head# 2024-04-20T17:27:46.248 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af8207a8:::benchmark_data_smithi171_122772_object148923:head# 2024-04-20T17:27:46.249 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af82295f:::benchmark_data_smithi171_122772_object104872:head# 2024-04-20T17:27:46.249 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af823b1e:::benchmark_data_smithi171_122772_object144584:head# 2024-04-20T17:27:46.249 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af834364:::benchmark_data_smithi171_122772_object162091:head# 2024-04-20T17:27:46.250 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af837c8f:::benchmark_data_smithi171_122772_object161060:head# 2024-04-20T17:27:46.250 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af83faca:::benchmark_data_smithi171_122772_object126700:head# 2024-04-20T17:27:46.251 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af85447a:::benchmark_data_smithi171_122772_object100935:head# 2024-04-20T17:27:46.251 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af865f9f:::benchmark_data_smithi171_122772_object121463:head# 2024-04-20T17:27:46.252 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af86dfb2:::benchmark_data_smithi171_122772_object104627:head# 2024-04-20T17:27:46.252 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af86fe03:::benchmark_data_smithi171_122772_object130603:head# 2024-04-20T17:27:46.253 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af878265:::benchmark_data_smithi171_122772_object116199:head# 2024-04-20T17:27:46.253 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af87ff55:::benchmark_data_smithi171_122772_object101257:head# 2024-04-20T17:27:46.253 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af88d72d:::benchmark_data_smithi171_122772_object89973:head# 2024-04-20T17:27:46.254 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af89ba57:::benchmark_data_smithi171_122772_object78414:head# 2024-04-20T17:27:46.254 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af89d3af:::benchmark_data_smithi171_122772_object90095:head# 2024-04-20T17:27:46.255 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af8a7f5f:::benchmark_data_smithi171_122772_object165444:head# 2024-04-20T17:27:46.255 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af8b5301:::benchmark_data_smithi171_122772_object130161:head# 2024-04-20T17:27:46.256 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af8c710f:::benchmark_data_smithi171_122772_object84642:head# 2024-04-20T17:27:46.256 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af8d07c6:::benchmark_data_smithi171_122772_object141067:head# 2024-04-20T17:27:46.257 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af8d0b0e:::benchmark_data_smithi171_122772_object153284:head# 2024-04-20T17:27:46.257 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af8d115f:::benchmark_data_smithi171_122772_object87988:head# 2024-04-20T17:27:46.257 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af8d8255:::benchmark_data_smithi171_122772_object136690:head# 2024-04-20T17:27:46.258 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af8df1e2:::benchmark_data_smithi171_122772_object133394:head# 2024-04-20T17:27:46.258 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af8fb3bc:::benchmark_data_smithi171_122772_object78419:head# 2024-04-20T17:27:46.259 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af902acf:::benchmark_data_smithi171_122772_object84743:head# 2024-04-20T17:27:46.259 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af9184e7:::benchmark_data_smithi171_122772_object158294:head# 2024-04-20T17:27:46.260 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af918e74:::benchmark_data_smithi171_122772_object113965:head# 2024-04-20T17:27:46.260 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af9263da:::benchmark_data_smithi171_122772_object152295:head# 2024-04-20T17:27:46.260 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af92e2ca:::benchmark_data_smithi171_122772_object91419:head# 2024-04-20T17:27:46.261 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af93c012:::benchmark_data_smithi171_122772_object142237:head# 2024-04-20T17:27:46.261 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af952983:::benchmark_data_smithi171_122772_object99408:head# 2024-04-20T17:27:46.262 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af956d5e:::benchmark_data_smithi171_122772_object156720:head# 2024-04-20T17:27:46.262 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af956fc3:::benchmark_data_smithi171_122772_object147256:head# 2024-04-20T17:27:46.262 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af958f7c:::benchmark_data_smithi171_122772_object130354:head# 2024-04-20T17:27:46.263 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af96ad14:::benchmark_data_smithi171_122772_object138509:head# 2024-04-20T17:27:46.263 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:27:46.264 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af96f7cd:::benchmark_data_smithi171_122772_object114343:head# 2024-04-20T17:27:46.264 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af971535:::benchmark_data_smithi171_122772_object79680:head# 2024-04-20T17:27:46.264 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af97e8bf:::benchmark_data_smithi171_122772_object92430:head# 2024-04-20T17:27:46.265 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af98170b:::benchmark_data_smithi171_122772_object93091:head# 2024-04-20T17:27:46.265 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af989ad5:::benchmark_data_smithi171_122772_object102337:head# 2024-04-20T17:27:46.265 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af99902b:::benchmark_data_smithi171_122772_object104241:head# 2024-04-20T17:27:46.266 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af9ad628:::benchmark_data_smithi171_122772_object135478:head# 2024-04-20T17:27:46.266 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af9d539b:::benchmark_data_smithi171_122772_object136378:head# 2024-04-20T17:27:46.267 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af9d92ab:::benchmark_data_smithi171_122772_object169170:head# 2024-04-20T17:27:46.267 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:af9ef014:::benchmark_data_smithi171_122772_object128913:head# 2024-04-20T17:27:46.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:46.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:27:46.269 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afa019da:::benchmark_data_smithi171_122772_object94773:head# 2024-04-20T17:27:46.269 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afa1063a:::benchmark_data_smithi171_122772_object130715:head# 2024-04-20T17:27:46.269 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afa17377:::benchmark_data_smithi171_122772_object97656:head# 2024-04-20T17:27:46.269 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afa1afc0:::benchmark_data_smithi171_122772_object145971:head# 2024-04-20T17:27:46.270 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afa2d15d:::benchmark_data_smithi171_122772_object145609:head# 2024-04-20T17:27:46.270 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afa3809d:::benchmark_data_smithi171_122772_object109758:head# 2024-04-20T17:27:46.271 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afa63c6e:::benchmark_data_smithi171_122772_object86822:head# 2024-04-20T17:27:46.271 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afa6627e:::benchmark_data_smithi171_122772_object85145:head# 2024-04-20T17:27:46.272 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afa73344:::benchmark_data_smithi171_122772_object84514:head# 2024-04-20T17:27:46.272 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afa76db1:::benchmark_data_smithi171_122772_object123847:head# 2024-04-20T17:27:46.273 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afa7fb2c:::benchmark_data_smithi171_122772_object93112:head# 2024-04-20T17:27:46.273 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afa91dd7:::benchmark_data_smithi171_122772_object86936:head# 2024-04-20T17:27:46.273 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afa951a8:::benchmark_data_smithi171_122772_object139128:head# 2024-04-20T17:27:46.274 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afa9984c:::benchmark_data_smithi171_122772_object141385:head# 2024-04-20T17:27:46.275 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afab0829:::benchmark_data_smithi171_122772_object84988:head# 2024-04-20T17:27:46.275 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afab1198:::benchmark_data_smithi171_122772_object120388:head# 2024-04-20T17:27:46.275 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afab40c7:::benchmark_data_smithi171_122772_object124162:head# 2024-04-20T17:27:46.276 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afab7599:::benchmark_data_smithi171_122772_object80323:head# 2024-04-20T17:27:46.276 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afac6f2e:::benchmark_data_smithi171_122772_object128046:head# 2024-04-20T17:27:46.276 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afac9a61:::benchmark_data_smithi171_122772_object163659:head# 2024-04-20T17:27:46.277 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afad8679:::benchmark_data_smithi171_122772_object152768:head# 2024-04-20T17:27:46.277 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afadc7ec:::benchmark_data_smithi171_122772_object159694:head# 2024-04-20T17:27:46.278 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afadda0f:::benchmark_data_smithi171_122772_object142148:head# 2024-04-20T17:27:46.278 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afaea273:::benchmark_data_smithi171_122772_object126360:head# 2024-04-20T17:27:46.279 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afaee23b:::benchmark_data_smithi171_122772_object156583:head# 2024-04-20T17:27:46.279 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afafc6b7:::benchmark_data_smithi171_122772_object98685:head# 2024-04-20T17:27:46.280 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afb04d06:::benchmark_data_smithi171_122772_object90493:head# 2024-04-20T17:27:46.280 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afb1aea6:::benchmark_data_smithi171_122772_object95668:head# 2024-04-20T17:27:46.281 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afb3023b:::benchmark_data_smithi171_122772_object98051:head# 2024-04-20T17:27:46.281 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afb3ab84:::benchmark_data_smithi171_122772_object88535:head# 2024-04-20T17:27:46.281 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afb405d1:::benchmark_data_smithi171_122772_object142057:head# 2024-04-20T17:27:46.282 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afb5c130:::benchmark_data_smithi171_122772_object142760:head# 2024-04-20T17:27:46.283 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afb67209:::benchmark_data_smithi171_122772_object158029:head# 2024-04-20T17:27:46.283 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afb6cf4b:::benchmark_data_smithi171_122772_object123552:head# 2024-04-20T17:27:46.283 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afb81901:::benchmark_data_smithi171_122772_object165089:head# 2024-04-20T17:27:46.284 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afb94677:::benchmark_data_smithi171_122772_object167245:head# 2024-04-20T17:27:46.284 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afb956c3:::benchmark_data_smithi171_122772_object87708:head# 2024-04-20T17:27:46.284 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afb9889f:::benchmark_data_smithi171_122772_object87846:head# 2024-04-20T17:27:46.285 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afba4727:::benchmark_data_smithi171_122772_object77957:head# 2024-04-20T17:27:46.285 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afbb8a57:::benchmark_data_smithi171_122772_object160693:head# 2024-04-20T17:27:46.286 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afbc4736:::benchmark_data_smithi171_122772_object139647:head# 2024-04-20T17:27:46.286 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afbc60a2:::benchmark_data_smithi171_122772_object164428:head# 2024-04-20T17:27:46.287 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afbc9785:::benchmark_data_smithi171_122772_object112181:head# 2024-04-20T17:27:46.287 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afbcf183:::benchmark_data_smithi171_122772_object87316:head# 2024-04-20T17:27:46.287 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afbd1feb:::benchmark_data_smithi171_122772_object150933:head# 2024-04-20T17:27:46.288 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afbd22d6:::benchmark_data_smithi171_122772_object117625:head# 2024-04-20T17:27:46.289 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afbe19d8:::benchmark_data_smithi171_122772_object92993:head# 2024-04-20T17:27:46.289 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afbe638f:::benchmark_data_smithi171_122772_object124015:head# 2024-04-20T17:27:46.289 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afbef0c4:::benchmark_data_smithi171_122772_object78994:head# 2024-04-20T17:27:46.289 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afbfc998:::benchmark_data_smithi171_122772_object90196:head# 2024-04-20T17:27:46.290 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afc12ceb:::benchmark_data_smithi171_122772_object79340:head# 2024-04-20T17:27:46.290 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afc2dc19:::benchmark_data_smithi171_122772_object153092:head# 2024-04-20T17:27:46.291 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afc31bc1:::benchmark_data_smithi171_122772_object141677:head# 2024-04-20T17:27:46.291 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afc350c3:::benchmark_data_smithi171_122772_object166658:head# 2024-04-20T17:27:46.292 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afc41eae:::benchmark_data_smithi171_122772_object115737:head# 2024-04-20T17:27:46.292 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afc49bb4:::benchmark_data_smithi171_122772_object120085:head# 2024-04-20T17:27:46.292 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afc5c018:::benchmark_data_smithi171_122772_object113789:head# 2024-04-20T17:27:46.293 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afc60c96:::benchmark_data_smithi171_122772_object161497:head# 2024-04-20T17:27:46.293 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afc61ca6:::benchmark_data_smithi171_122772_object163703:head# 2024-04-20T17:27:46.294 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afc6e572:::benchmark_data_smithi171_122772_object134166:head# 2024-04-20T17:27:46.294 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afc7e223:::benchmark_data_smithi171_122772_object106867:head# 2024-04-20T17:27:46.295 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afc8b47c:::benchmark_data_smithi171_122772_object128011:head# 2024-04-20T17:27:46.295 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afc8ea64:::benchmark_data_smithi171_122772_object126469:head# 2024-04-20T17:27:46.295 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afc8f7a8:::benchmark_data_smithi171_122772_object118844:head# 2024-04-20T17:27:46.296 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afc9a313:::benchmark_data_smithi171_122772_object155626:head# 2024-04-20T17:27:46.296 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afcaab50:::benchmark_data_smithi171_122772_object114228:head# 2024-04-20T17:27:46.297 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afcb1edd:::benchmark_data_smithi171_122772_object117463:head# 2024-04-20T17:27:46.297 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afcb255d:::benchmark_data_smithi171_122772_object114893:head# 2024-04-20T17:27:46.298 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afcb7636:::benchmark_data_smithi171_122772_object158780:head# 2024-04-20T17:27:46.298 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afcd1fcf:::benchmark_data_smithi171_122772_object100089:head# 2024-04-20T17:27:46.299 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afce80b7:::benchmark_data_smithi171_122772_object155350:head# 2024-04-20T17:27:46.299 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afcf311a:::benchmark_data_smithi171_122772_object169140:head# 2024-04-20T17:27:46.300 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afd093a8:::benchmark_data_smithi171_122772_object149009:head# 2024-04-20T17:27:46.300 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afd16e4a:::benchmark_data_smithi171_122772_object102826:head# 2024-04-20T17:27:46.300 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afd1985d:::benchmark_data_smithi171_122772_object111315:head# 2024-04-20T17:27:46.301 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afd1bfa3:::benchmark_data_smithi171_122772_object161444:head# 2024-04-20T17:27:46.301 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afd2f2b5:::benchmark_data_smithi171_122772_object81777:head# 2024-04-20T17:27:46.302 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afd33222:::benchmark_data_smithi171_122772_object108599:head# 2024-04-20T17:27:46.302 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afd3693e:::benchmark_data_smithi171_122772_object141863:head# 2024-04-20T17:27:46.303 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afd387c8:::benchmark_data_smithi171_122772_object135131:head# 2024-04-20T17:27:46.303 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afd3bd88:::benchmark_data_smithi171_122772_object112255:head# 2024-04-20T17:27:46.304 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afd49ce1:::benchmark_data_smithi171_122772_object160192:head# 2024-04-20T17:27:46.304 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afd5597e:::benchmark_data_smithi171_122772_object142076:head# 2024-04-20T17:27:46.304 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afd5a1d9:::benchmark_data_smithi171_122772_object144561:head# 2024-04-20T17:27:46.305 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afd9169f:::benchmark_data_smithi171_122772_object139080:head# 2024-04-20T17:27:46.305 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afdb0c56:::benchmark_data_smithi171_122772_object93965:head# 2024-04-20T17:27:46.306 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afdb2116:::benchmark_data_smithi171_122772_object86183:head# 2024-04-20T17:27:46.306 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afdb2bcc:::benchmark_data_smithi171_122772_object165347:head# 2024-04-20T17:27:46.307 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afdbcc2e:::benchmark_data_smithi171_122772_object78656:head# 2024-04-20T17:27:46.307 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afdc2953:::benchmark_data_smithi171_122772_object107903:head# 2024-04-20T17:27:46.308 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afdc6395:::benchmark_data_smithi171_122772_object87136:head# 2024-04-20T17:27:46.308 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afdd9777:::benchmark_data_smithi171_122772_object133779:head# 2024-04-20T17:27:46.309 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afde41b5:::benchmark_data_smithi171_122772_object104338:head# 2024-04-20T17:27:46.309 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afdef5fe:::benchmark_data_smithi171_122772_object133161:head# 2024-04-20T17:27:46.309 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afdef7e0:::benchmark_data_smithi171_122772_object90656:head# 2024-04-20T17:27:46.310 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afdf3244:::benchmark_data_smithi171_122772_object88361:head# 2024-04-20T17:27:46.310 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afe0e434:::benchmark_data_smithi171_122772_object169487:head# 2024-04-20T17:27:46.311 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afe3ea85:::benchmark_data_smithi171_122772_object151050:head# 2024-04-20T17:27:46.311 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afe44635:::benchmark_data_smithi171_122772_object146590:head# 2024-04-20T17:27:46.312 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afe46f12:::benchmark_data_smithi171_122772_object150017:head# 2024-04-20T17:27:46.313 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afe4b2e1:::benchmark_data_smithi171_122772_object84773:head# 2024-04-20T17:27:46.313 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afe53c18:::benchmark_data_smithi171_122772_object88665:head# 2024-04-20T17:27:46.313 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afe5e91c:::benchmark_data_smithi171_122772_object147684:head# 2024-04-20T17:27:46.313 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afe6c6f8:::benchmark_data_smithi171_122772_object148496:head# 2024-04-20T17:27:46.314 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afe7986f:::benchmark_data_smithi171_122772_object134878:head# 2024-04-20T17:27:46.314 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afe80c37:::benchmark_data_smithi171_122772_object101081:head# 2024-04-20T17:27:46.315 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afe8ab35:::benchmark_data_smithi171_122772_object144586:head# 2024-04-20T17:27:46.315 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afe91888:::benchmark_data_smithi171_122772_object106347:head# 2024-04-20T17:27:46.315 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afe9fd76:::benchmark_data_smithi171_122772_object117042:head# 2024-04-20T17:27:46.316 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afea4725:::benchmark_data_smithi171_122772_object118884:head# 2024-04-20T17:27:46.316 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afeaf556:::benchmark_data_smithi171_122772_object134863:head# 2024-04-20T17:27:46.317 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afec7c70:::benchmark_data_smithi171_122772_object102948:head# 2024-04-20T17:27:46.317 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afecd2be:::benchmark_data_smithi171_122772_object95010:head# 2024-04-20T17:27:46.318 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afedd9fa:::benchmark_data_smithi171_122772_object118612:head# 2024-04-20T17:27:46.318 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afede448:::benchmark_data_smithi171_122772_object124996:head# 2024-04-20T17:27:46.319 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afedeb28:::benchmark_data_smithi171_122772_object111824:head# 2024-04-20T17:27:46.319 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:afedf794:::benchmark_data_smithi171_122772_object90365:head# 2024-04-20T17:27:46.319 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aff164c4:::benchmark_data_smithi171_122772_object168910:head# 2024-04-20T17:27:46.320 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aff1a21b:::benchmark_data_smithi171_122772_object157502:head# 2024-04-20T17:27:46.320 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aff240c4:::benchmark_data_smithi171_122772_object167647:head# 2024-04-20T17:27:46.320 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aff25d05:::benchmark_data_smithi171_122772_object114242:head# 2024-04-20T17:27:46.321 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aff388fa:::benchmark_data_smithi171_122772_object122904:head# 2024-04-20T17:27:46.321 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aff38ef4:::benchmark_data_smithi171_122772_object156027:head# 2024-04-20T17:27:46.322 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aff47bfd:::benchmark_data_smithi171_122772_object125056:head# 2024-04-20T17:27:46.322 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aff4808d:::benchmark_data_smithi171_122772_object157207:head# 2024-04-20T17:27:46.323 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aff517db:::benchmark_data_smithi171_122772_object124827:head# 2024-04-20T17:27:46.323 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aff662fb:::benchmark_data_smithi171_122772_object126043:head# 2024-04-20T17:27:46.324 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aff67b55:::benchmark_data_smithi171_122772_object126516:head# 2024-04-20T17:27:46.324 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aff67c91:::benchmark_data_smithi171_122772_object92829:head# 2024-04-20T17:27:46.325 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aff789f1:::benchmark_data_smithi171_122772_object127724:head# 2024-04-20T17:27:46.325 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aff7a8d3:::benchmark_data_smithi171_122772_object163442:head# 2024-04-20T17:27:46.325 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aff844e1:::benchmark_data_smithi171_122772_object163092:head# 2024-04-20T17:27:46.326 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aff85d85:::benchmark_data_smithi171_122772_object162659:head# 2024-04-20T17:27:46.326 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aff8cdaa:::benchmark_data_smithi171_122772_object120111:head# 2024-04-20T17:27:46.326 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aff91294:::benchmark_data_smithi171_122772_object134847:head# 2024-04-20T17:27:46.327 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aff93c9d:::benchmark_data_smithi171_122772_object100378:head# 2024-04-20T17:27:46.327 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:aff99b83:::benchmark_data_smithi171_122772_object144627:head# 2024-04-20T17:27:46.328 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:affab38c:::benchmark_data_smithi171_122772_object82721:head# 2024-04-20T17:27:46.328 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:affae371:::benchmark_data_smithi171_122772_object84667:head# 2024-04-20T17:27:46.329 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:affba10c:::benchmark_data_smithi171_122772_object90149:head# 2024-04-20T17:27:46.329 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:affc5fe1:::benchmark_data_smithi171_122772_object112543:head# 2024-04-20T17:27:46.330 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:affd4112:::benchmark_data_smithi171_122772_object87390:head# 2024-04-20T17:27:46.330 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:affdc9e1:::benchmark_data_smithi171_122772_object119785:head# 2024-04-20T17:27:46.331 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:affe63f3:::benchmark_data_smithi171_122772_object101392:head# 2024-04-20T17:27:46.331 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:affe8207:::benchmark_data_smithi171_122772_object117658:head# 2024-04-20T17:27:46.332 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:affebe5b:::benchmark_data_smithi171_122772_object135561:head# 2024-04-20T17:27:46.336 INFO:teuthology.orchestra.run.smithi106.stderr:Export successful 2024-04-20T17:27:46.369 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:27:46.404 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:46.620 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:46 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:27:46.620 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:46 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3938960208' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:27:46.620 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:46 smithi005 ceph-mon[39562]: osdmap e515: 12 total, 11 up, 11 in 2024-04-20T17:27:46.631 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:46 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:27:46.631 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:46 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3938960208' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:27:46.631 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:46 smithi106 ceph-mon[44423]: osdmap e515: 12 total, 11 up, 11 in 2024-04-20T17:27:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:46 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:27:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:46 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3938960208' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:27:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:46 smithi171 ceph-mon[44189]: osdmap e515: 12 total, 11 up, 11 in 2024-04-20T17:27:46.865 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:27:46.865 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:27:47.376 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.6 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --force --op remove --pgid 4.5 2024-04-20T17:27:47.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:47 smithi005 ceph-mon[39562]: pgmap v1355: 17 pgs: 2 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 5.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 3774/184372 objects degraded (2.047%); 61 MiB/s, 971 objects/s recovering 2024-04-20T17:27:47.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:47 smithi005 ceph-mon[39562]: osdmap e516: 12 total, 11 up, 11 in 2024-04-20T17:27:47.548 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-20T17:27:47.558 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:27:47.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:47.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:27:47.621 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.6/config 2024-04-20T17:27:47.662 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:27:47.686 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:47 smithi106 ceph-mon[44423]: pgmap v1355: 17 pgs: 2 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 5.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 3774/184372 objects degraded (2.047%); 61 MiB/s, 971 objects/s recovering 2024-04-20T17:27:47.686 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:47 smithi106 ceph-mon[44423]: osdmap e516: 12 total, 11 up, 11 in 2024-04-20T17:27:47.688 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:47 smithi171 ceph-mon[44189]: pgmap v1355: 17 pgs: 2 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 5.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 3774/184372 objects degraded (2.047%); 61 MiB/s, 971 objects/s recovering 2024-04-20T17:27:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:47 smithi171 ceph-mon[44189]: osdmap e516: 12 total, 11 up, 11 in 2024-04-20T17:27:47.886 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:27:47.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:47.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:27:47.991 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:27:48.001 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:27:48.023 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:48.044 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:27:48.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:48.047 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:27:48.054 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:27:48.065 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:27:48.075 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:27:48.096 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:27:48.116 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:27:48.136 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:27:48.149 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:27:48.156 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:27:48.176 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:48.179 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:27:48.201 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:27:48.221 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:27:48.245 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:27:48.265 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:27:48.265 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:27:48.266 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:27:48.266 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:27:48.266 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:27:48.266 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:27:48.266 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:27:48.266 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-20T17:27:48.266 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:27:48.266 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:27:48.266 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:27:48.266 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:27:48.266 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:27:48.330 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:27:48.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:48.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:27:48.435 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:27:48.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:48 smithi005 ceph-mon[39562]: osdmap e517: 12 total, 11 up, 11 in 2024-04-20T17:27:48.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:48 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2813961276' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:27:48.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:48.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:48.497 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:48.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:48 smithi106 ceph-mon[44423]: osdmap e517: 12 total, 11 up, 11 in 2024-04-20T17:27:48.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:48 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2813961276' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:27:48.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:48.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:48.519 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:27:48.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:48.525 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:27:48.625 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:27:48.679 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:48 smithi171 ceph-mon[44189]: osdmap e517: 12 total, 11 up, 11 in 2024-04-20T17:27:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:48 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2813961276' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:27:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.083 INFO:teuthology.orchestra.run.smithi106.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:27:49.143 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:27:49.144 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:27:49.144 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:27:49.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:49.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:27:49.252 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:27:49.292 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:49.324 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:27:49.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:49.331 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:27:49.356 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.357 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.357 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.357 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.357 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.357 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.357 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.357 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:49 smithi106 ceph-mon[44423]: pgmap v1358: 17 pgs: 2 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 5.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 3774/184372 objects degraded (2.047%); 70 MiB/s, 1.12k objects/s recovering 2024-04-20T17:27:49.357 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.357 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.357 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:49 smithi106 ceph-mon[44423]: osdmap e518: 12 total, 11 up, 11 in 2024-04-20T17:27:49.432 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:27:49.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:49 smithi005 ceph-mon[39562]: pgmap v1358: 17 pgs: 2 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 5.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 3774/184372 objects degraded (2.047%); 70 MiB/s, 1.12k objects/s recovering 2024-04-20T17:27:49.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:49 smithi005 ceph-mon[39562]: osdmap e518: 12 total, 11 up, 11 in 2024-04-20T17:27:49.469 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:49 smithi171 ceph-mon[44189]: pgmap v1358: 17 pgs: 2 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 5.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 3774/184372 objects degraded (2.047%); 70 MiB/s, 1.12k objects/s recovering 2024-04-20T17:27:49.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:49.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:49 smithi171 ceph-mon[44189]: osdmap e518: 12 total, 11 up, 11 in 2024-04-20T17:27:49.624 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:27:49.671 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:27:49.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:49.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:27:49.776 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:27:49.819 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:49.849 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:27:49.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:49.853 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:27:49.954 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:27:49.992 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:50.259 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:27:50.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:50.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:27:50.366 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:27:50.399 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:50.427 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:27:50.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:50.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:27:50.514 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:50 smithi005 ceph-mon[39562]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3774/184372 objects degraded (2.047%), 2 pgs degraded) 2024-04-20T17:27:50.530 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:27:50.588 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:50.755 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:27:50.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:50.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:27:50.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:50 smithi106 ceph-mon[44423]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3774/184372 objects degraded (2.047%), 2 pgs degraded) 2024-04-20T17:27:50.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:50 smithi171 ceph-mon[44189]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3774/184372 objects degraded (2.047%), 2 pgs degraded) 2024-04-20T17:27:50.859 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:27:50.908 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:50.969 INFO:teuthology.orchestra.run.smithi106.stdout: marking collection for removal 2024-04-20T17:27:50.970 INFO:teuthology.orchestra.run.smithi106.stdout:setting '_remove' omap key 2024-04-20T17:27:50.974 INFO:teuthology.orchestra.run.smithi106.stdout:finish_remove_pgs 4.5_head removing 4.5 2024-04-20T17:27:51.079 INFO:teuthology.orchestra.run.smithi106.stdout:Remove successful 2024-04-20T17:27:51.146 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:27:51.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:51 smithi005 ceph-mon[39562]: pgmap v1360: 17 pgs: 17 active+clean; 5.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 54 op/s; 48 MiB/s, 769 objects/s recovering 2024-04-20T17:27:51.513 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:27:51.519 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:27:51.519 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:27:51.519 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:27:51.519 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:27:51.519 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.519 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:268280 2.3 2:c0f96bdd:::benchmark_data_smithi171_78918_object107286:head [delete] snapc 0=[] ondisk+write+known_if_redirected e309)", 2024-04-20T17:27:51.519 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:17:55.131150+0000", 2024-04-20T17:27:51.519 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 596.38119658899996, 2024-04-20T17:27:51.519 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.634024288, 2024-04-20T17:27:51.519 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:27:51.520 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:27:51.520 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:51.520 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:27:51.520 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:27:51.520 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:27:51.520 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 268280 2024-04-20T17:27:51.520 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.520 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:27:51.520 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.520 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:27:51.520 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.131150+0000", 2024-04-20T17:27:51.520 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.520 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.520 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.521 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:27:51.521 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.131150+0000", 2024-04-20T17:27:51.521 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.521 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.521 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.521 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:27:51.521 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.131152+0000", 2024-04-20T17:27:51.521 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1179999999999999e-06 2024-04-20T17:27:51.521 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.521 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.521 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:27:51.521 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.131158+0000", 2024-04-20T17:27:51.521 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7150000000000001e-06 2024-04-20T17:27:51.521 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.522 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.522 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:27:51.522 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.131170+0000", 2024-04-20T17:27:51.522 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1433999999999999e-05 2024-04-20T17:27:51.522 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.522 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.522 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:27:51.522 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.131179+0000", 2024-04-20T17:27:51.522 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0420000000000008e-06 2024-04-20T17:27:51.522 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.522 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.522 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:27:51.522 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.185255+0000", 2024-04-20T17:27:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.054076235 2024-04-20T17:27:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:27:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.187152+0000", 2024-04-20T17:27:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001896539 2024-04-20T17:27:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:27:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.925874+0000", 2024-04-20T17:27:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.73872241400000005 2024-04-20T17:27:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:27:51.524 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:55.928102+0000", 2024-04-20T17:27:51.524 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0022278860000000001 2024-04-20T17:27:51.524 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.524 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.524 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:27:51.524 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:57.681606+0000", 2024-04-20T17:27:51.524 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.753504239 2024-04-20T17:27:51.524 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.524 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.524 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:27:51.524 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:57.685368+0000", 2024-04-20T17:27:51.524 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0037615859999999999 2024-04-20T17:27:51.524 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.525 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.525 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:27:51.525 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:57.693441+0000", 2024-04-20T17:27:51.525 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0080737340000000008 2024-04-20T17:27:51.525 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.525 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.525 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:27:51.525 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:57.734945+0000", 2024-04-20T17:27:51.525 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.041503962999999998 2024-04-20T17:27:51.525 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.525 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.525 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:51.525 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:57.755109+0000", 2024-04-20T17:27:51.525 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.020163945999999999 2024-04-20T17:27:51.526 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.526 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.526 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:27:51.526 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:57.757740+0000", 2024-04-20T17:27:51.526 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002630453 2024-04-20T17:27:51.526 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.526 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.526 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:27:51.526 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:17:57.765174+0000", 2024-04-20T17:27:51.526 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0074339840000000003 2024-04-20T17:27:51.526 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.526 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:27:51.526 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:272485 2.3 2:cf8b6f46:::benchmark_data_smithi171_78918_object111491:head [delete] snapc 0=[] ondisk+write+known_if_redirected e316)", 2024-04-20T17:27:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:18:13.931178+0000", 2024-04-20T17:27:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 577.58116840299999, 2024-04-20T17:27:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5463639260000002, 2024-04-20T17:27:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:27:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:27:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:27:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:27:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:27:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 272485 2024-04-20T17:27:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:27:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:27:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:13.931178+0000", 2024-04-20T17:27:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:27:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:13.931178+0000", 2024-04-20T17:27:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:27:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:13.931180+0000", 2024-04-20T17:27:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0200000000000001e-06 2024-04-20T17:27:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:27:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:13.931186+0000", 2024-04-20T17:27:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5209999999999999e-06 2024-04-20T17:27:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:27:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:13.931198+0000", 2024-04-20T17:27:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2126999999999999e-05 2024-04-20T17:27:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:27:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:13.931205+0000", 2024-04-20T17:27:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0010000000000004e-06 2024-04-20T17:27:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.530 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.530 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:27:51.530 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:14.024529+0000", 2024-04-20T17:27:51.530 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.093323992999999994 2024-04-20T17:27:51.530 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.530 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.530 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:27:51.530 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:14.026231+0000", 2024-04-20T17:27:51.530 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0017012640000000001 2024-04-20T17:27:51.530 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.530 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.530 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:27:51.530 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:14.688291+0000", 2024-04-20T17:27:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.66206054700000005 2024-04-20T17:27:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:27:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:14.690674+0000", 2024-04-20T17:27:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002383282 2024-04-20T17:27:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:27:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:16.448551+0000", 2024-04-20T17:27:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7578762220000002 2024-04-20T17:27:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:27:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:16.451664+0000", 2024-04-20T17:27:51.532 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003113476 2024-04-20T17:27:51.532 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.532 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.532 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:27:51.532 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:16.457746+0000", 2024-04-20T17:27:51.532 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0060814440000000001 2024-04-20T17:27:51.532 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.532 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.532 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:51.532 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:16.468555+0000", 2024-04-20T17:27:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010809440999999999 2024-04-20T17:27:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:27:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:16.469204+0000", 2024-04-20T17:27:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00064895199999999995 2024-04-20T17:27:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:27:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:16.470865+0000", 2024-04-20T17:27:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00166113 2024-04-20T17:27:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:27:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:16.477542+0000", 2024-04-20T17:27:51.534 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0066765059999999996 2024-04-20T17:27:51.534 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.534 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:27:51.534 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.534 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.534 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.534 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.24736.0:273306 2.3 2:c7ad72a8:::benchmark_data_smithi171_78918_object112312:head [delete] snapc 0=[] ondisk+write+known_if_redirected e317)", 2024-04-20T17:27:51.534 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:18:17.740700+0000", 2024-04-20T17:27:51.534 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 573.77164626900003, 2024-04-20T17:27:51.534 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7316630169999998, 2024-04-20T17:27:51.534 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:27:51.534 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:27:51.534 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:51.534 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:27:51.535 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.24736", 2024-04-20T17:27:51.535 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2761877519", 2024-04-20T17:27:51.535 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 273306 2024-04-20T17:27:51.535 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.535 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:27:51.535 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.535 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:27:51.535 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:17.740700+0000", 2024-04-20T17:27:51.535 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.535 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.535 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.535 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:27:51.535 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:17.740700+0000", 2024-04-20T17:27:51.535 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.535 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.535 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.536 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:27:51.536 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:17.740703+0000", 2024-04-20T17:27:51.536 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3699999999999999e-06 2024-04-20T17:27:51.536 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.536 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.536 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:27:51.536 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:17.740713+0000", 2024-04-20T17:27:51.536 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7049999999999994e-06 2024-04-20T17:27:51.536 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.536 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.536 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:27:51.536 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:17.740727+0000", 2024-04-20T17:27:51.536 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3828999999999999e-05 2024-04-20T17:27:51.536 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.536 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.536 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:27:51.537 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:17.740736+0000", 2024-04-20T17:27:51.537 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.3209999999999994e-06 2024-04-20T17:27:51.537 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.537 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.537 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:27:51.537 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:17.819955+0000", 2024-04-20T17:27:51.537 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.079219081999999996 2024-04-20T17:27:51.537 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.537 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.537 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for missing object", 2024-04-20T17:27:51.537 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:17.822169+0000", 2024-04-20T17:27:51.537 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0022142680000000001 2024-04-20T17:27:51.537 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.537 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.538 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:27:51.538 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:18.772171+0000", 2024-04-20T17:27:51.538 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.95000131600000004 2024-04-20T17:27:51.538 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.538 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.538 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:27:51.538 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:18.774199+0000", 2024-04-20T17:27:51.538 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0020284610000000001 2024-04-20T17:27:51.538 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.538 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.538 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:27:51.538 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:20.414977+0000", 2024-04-20T17:27:51.538 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6407782499999999 2024-04-20T17:27:51.539 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.539 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.539 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:27:51.539 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:20.417976+0000", 2024-04-20T17:27:51.539 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002998213 2024-04-20T17:27:51.539 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.539 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.539 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:27:51.539 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:20.423980+0000", 2024-04-20T17:27:51.539 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.006004634 2024-04-20T17:27:51.539 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.539 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.539 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:27:51.539 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:20.460896+0000", 2024-04-20T17:27:51.539 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.036915765000000003 2024-04-20T17:27:51.540 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.540 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.540 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:51.540 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:20.462179+0000", 2024-04-20T17:27:51.540 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0012827940000000001 2024-04-20T17:27:51.540 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.540 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.540 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:27:51.540 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:20.464524+0000", 2024-04-20T17:27:51.540 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0023448779999999999 2024-04-20T17:27:51.540 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.540 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.540 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:27:51.540 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:18:20.472363+0000", 2024-04-20T17:27:51.541 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0078391309999999992 2024-04-20T17:27:51.541 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.541 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:27:51.541 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.541 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.541 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.541 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:207708 4.b 4:d8bef688:::benchmark_data_smithi171_122772_object37516:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:27:51.541 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:48.969421+0000", 2024-04-20T17:27:51.541 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 62.542924792000001, 2024-04-20T17:27:51.541 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.21427681800000001, 2024-04-20T17:27:51.541 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:27:51.541 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:27:51.541 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:51.541 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:27:51.542 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:27:51.542 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:51.542 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 207708 2024-04-20T17:27:51.542 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.542 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:27:51.542 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.542 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:27:51.542 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.969421+0000", 2024-04-20T17:27:51.542 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.542 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.542 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.542 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:27:51.542 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.969421+0000", 2024-04-20T17:27:51.543 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.543 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.543 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.543 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:27:51.543 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.969424+0000", 2024-04-20T17:27:51.543 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5280000000000002e-06 2024-04-20T17:27:51.543 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.543 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.543 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:27:51.543 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.969432+0000", 2024-04-20T17:27:51.543 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0439999999999996e-06 2024-04-20T17:27:51.543 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.543 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.543 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:27:51.544 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.969448+0000", 2024-04-20T17:27:51.544 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6099999999999998e-05 2024-04-20T17:27:51.544 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.544 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.544 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:27:51.544 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.969458+0000", 2024-04-20T17:27:51.544 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7040000000000006e-06 2024-04-20T17:27:51.544 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.544 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.544 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:27:51.544 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.969514+0000", 2024-04-20T17:27:51.544 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5930000000000002e-05 2024-04-20T17:27:51.544 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.544 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.545 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:27:51.545 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.969613+0000", 2024-04-20T17:27:51.545 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9448000000000001e-05 2024-04-20T17:27:51.545 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.545 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.545 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:27:51.545 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.969690+0000", 2024-04-20T17:27:51.545 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7033999999999995e-05 2024-04-20T17:27:51.545 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.545 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.545 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:27:51.545 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.974635+0000", 2024-04-20T17:27:51.545 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0049453270000000002 2024-04-20T17:27:51.545 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.545 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.546 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:51.546 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.183628+0000", 2024-04-20T17:27:51.546 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.20899236600000001 2024-04-20T17:27:51.546 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.546 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.546 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:27:51.546 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.183663+0000", 2024-04-20T17:27:51.546 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5371999999999999e-05 2024-04-20T17:27:51.546 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.546 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.546 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:27:51.546 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.183698+0000", 2024-04-20T17:27:51.546 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4965000000000002e-05 2024-04-20T17:27:51.546 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.547 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:27:51.547 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.547 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.547 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.547 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:207720 4.b 4:da516ea6:::benchmark_data_smithi171_122772_object37528:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:27:51.547 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:48.975163+0000", 2024-04-20T17:27:51.547 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 62.537183401999997, 2024-04-20T17:27:51.547 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.213299244, 2024-04-20T17:27:51.547 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:27:51.547 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:27:51.547 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:51.547 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:27:51.547 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:27:51.547 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:51.548 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 207720 2024-04-20T17:27:51.548 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.548 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:27:51.548 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.548 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:27:51.548 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.975163+0000", 2024-04-20T17:27:51.548 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.548 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.548 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.548 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:27:51.548 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.975163+0000", 2024-04-20T17:27:51.548 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.548 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.549 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.549 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:27:51.549 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.975165+0000", 2024-04-20T17:27:51.549 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5330000000000001e-06 2024-04-20T17:27:51.549 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.549 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.549 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:27:51.549 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.975171+0000", 2024-04-20T17:27:51.549 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7790000000000001e-06 2024-04-20T17:27:51.549 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.549 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.549 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:27:51.549 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.975182+0000", 2024-04-20T17:27:51.550 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0543000000000001e-05 2024-04-20T17:27:51.550 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.550 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.550 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:27:51.550 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.975188+0000", 2024-04-20T17:27:51.550 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5400000000000001e-06 2024-04-20T17:27:51.550 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.550 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.550 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:27:51.551 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.975239+0000", 2024-04-20T17:27:51.551 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1264999999999998e-05 2024-04-20T17:27:51.551 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.551 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.551 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:27:51.551 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.975353+0000", 2024-04-20T17:27:51.551 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011350599999999999 2024-04-20T17:27:51.551 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.551 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.551 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:27:51.551 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.975436+0000", 2024-04-20T17:27:51.551 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3159999999999997e-05 2024-04-20T17:27:51.551 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.551 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.551 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:27:51.552 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.989992+0000", 2024-04-20T17:27:51.552 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014555653999999999 2024-04-20T17:27:51.552 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.552 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.552 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:51.552 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.188400+0000", 2024-04-20T17:27:51.552 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.19840873000000001 2024-04-20T17:27:51.552 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.552 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.552 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:27:51.552 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.188435+0000", 2024-04-20T17:27:51.552 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4418999999999997e-05 2024-04-20T17:27:51.552 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.552 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.553 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:27:51.553 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.188462+0000", 2024-04-20T17:27:51.553 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7115e-05 2024-04-20T17:27:51.553 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.553 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:27:51.553 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.553 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.553 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.553 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:207727 4.b 4:dc09fbba:::benchmark_data_smithi171_122772_object37535:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:27:51.553 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:48.979624+0000", 2024-04-20T17:27:51.553 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 62.532722100999997, 2024-04-20T17:27:51.553 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.20892259699999999, 2024-04-20T17:27:51.553 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:27:51.553 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:27:51.554 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:51.554 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:27:51.554 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:27:51.554 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:51.554 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 207727 2024-04-20T17:27:51.554 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.554 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:27:51.554 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.554 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:27:51.554 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.979624+0000", 2024-04-20T17:27:51.554 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.554 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.554 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.554 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:27:51.555 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.979624+0000", 2024-04-20T17:27:51.555 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.555 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.555 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.555 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:27:51.555 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.979626+0000", 2024-04-20T17:27:51.555 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1720000000000001e-06 2024-04-20T17:27:51.555 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.555 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.555 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:27:51.555 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.979632+0000", 2024-04-20T17:27:51.555 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0769999999999996e-06 2024-04-20T17:27:51.555 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.555 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.555 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:27:51.556 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.979647+0000", 2024-04-20T17:27:51.556 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5158e-05 2024-04-20T17:27:51.556 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.556 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.556 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:27:51.556 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.979657+0000", 2024-04-20T17:27:51.556 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9189999999999994e-06 2024-04-20T17:27:51.556 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.556 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.556 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:27:51.556 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.979956+0000", 2024-04-20T17:27:51.556 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00029903699999999999 2024-04-20T17:27:51.556 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.556 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.556 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:27:51.557 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.980090+0000", 2024-04-20T17:27:51.557 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013391100000000001 2024-04-20T17:27:51.557 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.557 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.557 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:27:51.557 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.980196+0000", 2024-04-20T17:27:51.557 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000105494 2024-04-20T17:27:51.557 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.557 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.557 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:27:51.557 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.990002+0000", 2024-04-20T17:27:51.557 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0098060109999999999 2024-04-20T17:27:51.557 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.557 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.557 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:51.558 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.188494+0000", 2024-04-20T17:27:51.558 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.19849267000000001 2024-04-20T17:27:51.558 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.558 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.558 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:27:51.558 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.188522+0000", 2024-04-20T17:27:51.558 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7817000000000002e-05 2024-04-20T17:27:51.558 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.558 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.558 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:27:51.558 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.188547+0000", 2024-04-20T17:27:51.558 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4331000000000001e-05 2024-04-20T17:27:51.558 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.558 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:27:51.559 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.559 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.559 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.559 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:207733 4.b 4:df9f7fcf:::benchmark_data_smithi171_122772_object37541:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:27:51.559 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:48.983759+0000", 2024-04-20T17:27:51.559 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 62.528587356000003, 2024-04-20T17:27:51.559 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.214777986, 2024-04-20T17:27:51.559 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:27:51.559 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:27:51.559 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:51.559 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:27:51.559 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:27:51.559 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:51.559 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 207733 2024-04-20T17:27:51.559 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.560 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:27:51.560 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.560 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:27:51.560 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.983759+0000", 2024-04-20T17:27:51.560 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.560 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.560 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.560 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:27:51.560 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.983759+0000", 2024-04-20T17:27:51.560 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.560 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.560 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.560 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:27:51.560 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.983762+0000", 2024-04-20T17:27:51.561 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.467e-06 2024-04-20T17:27:51.561 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.561 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.561 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:27:51.561 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.983772+0000", 2024-04-20T17:27:51.561 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4229999999999994e-06 2024-04-20T17:27:51.561 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.561 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.561 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:27:51.561 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.983784+0000", 2024-04-20T17:27:51.561 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2437e-05 2024-04-20T17:27:51.561 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.561 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.561 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:27:51.562 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.983792+0000", 2024-04-20T17:27:51.562 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4209999999999998e-06 2024-04-20T17:27:51.562 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.562 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.562 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:27:51.562 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.985019+0000", 2024-04-20T17:27:51.562 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001227302 2024-04-20T17:27:51.562 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.562 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.562 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:27:51.562 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.985157+0000", 2024-04-20T17:27:51.562 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013808900000000001 2024-04-20T17:27:51.562 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.562 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.563 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:27:51.563 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.985294+0000", 2024-04-20T17:27:51.563 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013690800000000001 2024-04-20T17:27:51.563 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.563 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.563 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:27:51.563 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.990105+0000", 2024-04-20T17:27:51.563 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0048111919999999997 2024-04-20T17:27:51.563 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.563 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.563 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:51.563 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.198445+0000", 2024-04-20T17:27:51.563 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.20834012199999999 2024-04-20T17:27:51.563 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.564 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.564 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:27:51.564 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.198494+0000", 2024-04-20T17:27:51.564 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8949000000000002e-05 2024-04-20T17:27:51.564 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.564 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.564 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:27:51.564 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.198537+0000", 2024-04-20T17:27:51.564 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2676000000000003e-05 2024-04-20T17:27:51.564 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.564 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:27:51.564 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.564 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.564 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.565 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:207757 4.b 4:de0a109c:::benchmark_data_smithi171_122772_object37565:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:27:51.565 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:49.001251+0000", 2024-04-20T17:27:51.565 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 62.511094833000001, 2024-04-20T17:27:51.565 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.19744542300000001, 2024-04-20T17:27:51.565 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:27:51.565 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:27:51.565 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:51.565 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:27:51.565 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:27:51.565 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:51.565 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 207757 2024-04-20T17:27:51.565 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.565 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:27:51.565 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.566 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:27:51.566 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.001251+0000", 2024-04-20T17:27:51.566 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.566 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.566 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.566 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:27:51.566 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.001251+0000", 2024-04-20T17:27:51.566 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.566 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.566 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.566 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:27:51.566 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.001254+0000", 2024-04-20T17:27:51.566 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.19e-06 2024-04-20T17:27:51.566 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.567 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.567 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:27:51.567 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.001264+0000", 2024-04-20T17:27:51.567 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8449999999999998e-06 2024-04-20T17:27:51.567 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.567 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.567 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:27:51.567 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.001295+0000", 2024-04-20T17:27:51.567 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1170999999999997e-05 2024-04-20T17:27:51.567 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.567 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.567 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:27:51.567 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.001307+0000", 2024-04-20T17:27:51.567 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1559000000000001e-05 2024-04-20T17:27:51.568 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.568 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.568 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:27:51.568 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.001370+0000", 2024-04-20T17:27:51.568 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3133999999999996e-05 2024-04-20T17:27:51.568 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.568 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.568 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:27:51.568 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.001471+0000", 2024-04-20T17:27:51.568 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010098399999999999 2024-04-20T17:27:51.568 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.568 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.568 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:27:51.568 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.001561+0000", 2024-04-20T17:27:51.568 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0149999999999996e-05 2024-04-20T17:27:51.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.569 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:27:51.569 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.005406+0000", 2024-04-20T17:27:51.569 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0038443280000000002 2024-04-20T17:27:51.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.569 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:51.569 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.198589+0000", 2024-04-20T17:27:51.569 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.19318358299999999 2024-04-20T17:27:51.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.569 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:27:51.569 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.198647+0000", 2024-04-20T17:27:51.570 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8081999999999997e-05 2024-04-20T17:27:51.570 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.570 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.570 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:27:51.570 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.198697+0000", 2024-04-20T17:27:51.570 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9397000000000002e-05 2024-04-20T17:27:51.570 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.570 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:27:51.570 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.570 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.570 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.570 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:207773 4.b 4:dbe3c1a3:::benchmark_data_smithi171_122772_object37581:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:27:51.570 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:49.013042+0000", 2024-04-20T17:27:51.570 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 62.499303842000003, 2024-04-20T17:27:51.571 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.192015144, 2024-04-20T17:27:51.571 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:27:51.571 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:27:51.571 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:51.571 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:27:51.571 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:27:51.571 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:51.571 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 207773 2024-04-20T17:27:51.571 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.571 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:27:51.571 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.571 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:27:51.571 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.013042+0000", 2024-04-20T17:27:51.571 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.572 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.572 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:27:51.572 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.013042+0000", 2024-04-20T17:27:51.572 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.572 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.572 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:27:51.572 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.013045+0000", 2024-04-20T17:27:51.572 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0070000000000002e-06 2024-04-20T17:27:51.572 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.572 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:27:51.572 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.013055+0000", 2024-04-20T17:27:51.572 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9599999999999995e-06 2024-04-20T17:27:51.573 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.573 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.573 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:27:51.573 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.013071+0000", 2024-04-20T17:27:51.573 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5273000000000001e-05 2024-04-20T17:27:51.573 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.573 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.573 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:27:51.573 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.013078+0000", 2024-04-20T17:27:51.573 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7090000000000002e-06 2024-04-20T17:27:51.573 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.573 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.573 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:27:51.573 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.013145+0000", 2024-04-20T17:27:51.574 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6511000000000001e-05 2024-04-20T17:27:51.574 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.574 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.574 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:27:51.574 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.013290+0000", 2024-04-20T17:27:51.574 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014574600000000001 2024-04-20T17:27:51.574 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.574 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.574 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:27:51.574 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.013431+0000", 2024-04-20T17:27:51.574 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014091700000000001 2024-04-20T17:27:51.574 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.574 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.574 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:27:51.575 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.017169+0000", 2024-04-20T17:27:51.575 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0037376710000000001 2024-04-20T17:27:51.575 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.575 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.575 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:51.575 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.204968+0000", 2024-04-20T17:27:51.575 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.187798989 2024-04-20T17:27:51.575 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.575 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.575 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:27:51.575 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.205025+0000", 2024-04-20T17:27:51.575 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6965000000000001e-05 2024-04-20T17:27:51.575 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.575 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.575 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:27:51.576 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.205057+0000", 2024-04-20T17:27:51.576 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2396e-05 2024-04-20T17:27:51.576 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.576 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:27:51.576 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.576 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.576 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.576 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:207785 4.b 4:db29997d:::benchmark_data_smithi171_122772_object37593:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:27:51.576 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:49.024845+0000", 2024-04-20T17:27:51.576 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 62.487501057999999, 2024-04-20T17:27:51.576 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.180316225, 2024-04-20T17:27:51.576 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:27:51.576 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:27:51.576 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:51.577 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:27:51.577 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:27:51.577 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:51.577 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 207785 2024-04-20T17:27:51.577 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.577 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:27:51.577 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.577 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:27:51.577 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.024845+0000", 2024-04-20T17:27:51.577 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.577 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.577 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.577 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:27:51.577 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.024845+0000", 2024-04-20T17:27:51.578 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.578 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.578 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.578 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:27:51.578 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.024848+0000", 2024-04-20T17:27:51.578 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4930000000000001e-06 2024-04-20T17:27:51.578 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.578 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.578 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:27:51.578 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.024855+0000", 2024-04-20T17:27:51.578 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2819999999999999e-06 2024-04-20T17:27:51.578 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.578 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.578 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:27:51.579 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.024871+0000", 2024-04-20T17:27:51.579 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6087e-05 2024-04-20T17:27:51.579 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.579 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.579 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:27:51.579 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.024878+0000", 2024-04-20T17:27:51.579 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1239999999999999e-06 2024-04-20T17:27:51.579 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.579 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.579 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:27:51.579 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.024945+0000", 2024-04-20T17:27:51.579 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6458999999999995e-05 2024-04-20T17:27:51.579 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.579 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.580 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:27:51.580 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.025092+0000", 2024-04-20T17:27:51.580 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014733899999999999 2024-04-20T17:27:51.580 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.580 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.580 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:27:51.580 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.025229+0000", 2024-04-20T17:27:51.580 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013684600000000001 2024-04-20T17:27:51.580 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.580 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.580 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:27:51.580 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.030149+0000", 2024-04-20T17:27:51.580 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0049204469999999997 2024-04-20T17:27:51.580 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.580 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.581 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:51.581 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.205100+0000", 2024-04-20T17:27:51.581 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17495043399999999 2024-04-20T17:27:51.581 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.581 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.581 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:27:51.581 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.205133+0000", 2024-04-20T17:27:51.581 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.307e-05 2024-04-20T17:27:51.581 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.581 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.581 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:27:51.581 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.205161+0000", 2024-04-20T17:27:51.581 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8643999999999999e-05 2024-04-20T17:27:51.581 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.581 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:27:51.582 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.582 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.582 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.582 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:207792 4.b 4:da25c4e5:::benchmark_data_smithi171_122772_object37600:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:27:51.582 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:49.030811+0000", 2024-04-20T17:27:51.582 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 62.481534629999999, 2024-04-20T17:27:51.582 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17443872099999999, 2024-04-20T17:27:51.582 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:27:51.582 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:27:51.582 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:51.582 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:27:51.582 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:27:51.582 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:51.582 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 207792 2024-04-20T17:27:51.583 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.583 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:27:51.583 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.583 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:27:51.583 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.030811+0000", 2024-04-20T17:27:51.583 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.583 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.583 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.583 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:27:51.583 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.030811+0000", 2024-04-20T17:27:51.583 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.583 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.583 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.583 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:27:51.583 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.030814+0000", 2024-04-20T17:27:51.584 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.311e-06 2024-04-20T17:27:51.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.584 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:27:51.584 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.030823+0000", 2024-04-20T17:27:51.584 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.5589999999999994e-06 2024-04-20T17:27:51.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.584 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:27:51.584 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.030835+0000", 2024-04-20T17:27:51.584 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2007999999999999e-05 2024-04-20T17:27:51.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.584 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:27:51.585 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.030845+0000", 2024-04-20T17:27:51.585 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7389999999999994e-06 2024-04-20T17:27:51.585 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.585 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.585 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:27:51.585 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.030908+0000", 2024-04-20T17:27:51.585 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2891000000000005e-05 2024-04-20T17:27:51.585 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.585 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.585 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:27:51.585 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.031046+0000", 2024-04-20T17:27:51.585 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013822300000000001 2024-04-20T17:27:51.585 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.585 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.586 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:27:51.586 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.031177+0000", 2024-04-20T17:27:51.586 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013072100000000001 2024-04-20T17:27:51.586 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.586 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.586 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:27:51.586 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.035767+0000", 2024-04-20T17:27:51.586 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.004590055 2024-04-20T17:27:51.586 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.586 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.586 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:51.586 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.205188+0000", 2024-04-20T17:27:51.586 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.169421345 2024-04-20T17:27:51.586 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.587 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.587 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:27:51.587 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.205226+0000", 2024-04-20T17:27:51.587 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7698999999999999e-05 2024-04-20T17:27:51.587 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.587 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.587 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:27:51.587 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.205250+0000", 2024-04-20T17:27:51.587 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4170000000000001e-05 2024-04-20T17:27:51.587 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.587 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:27:51.587 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.587 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.587 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.588 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:207802 4.b 4:daaa6950:::benchmark_data_smithi171_122772_object37610:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:27:51.588 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:49.038445+0000", 2024-04-20T17:27:51.588 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 62.473901071, 2024-04-20T17:27:51.588 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17428120799999999, 2024-04-20T17:27:51.588 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:27:51.588 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:27:51.588 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:51.588 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:27:51.588 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:27:51.588 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:51.588 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 207802 2024-04-20T17:27:51.588 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.588 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:27:51.588 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.588 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:27:51.589 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.038445+0000", 2024-04-20T17:27:51.589 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.589 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.589 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.589 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:27:51.589 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.038445+0000", 2024-04-20T17:27:51.589 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.589 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.589 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.589 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:27:51.589 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.038447+0000", 2024-04-20T17:27:51.589 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9750000000000001e-06 2024-04-20T17:27:51.589 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.590 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.590 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:27:51.590 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.038453+0000", 2024-04-20T17:27:51.590 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1840000000000004e-06 2024-04-20T17:27:51.590 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.590 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.590 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:27:51.590 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.038465+0000", 2024-04-20T17:27:51.590 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2254e-05 2024-04-20T17:27:51.590 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.590 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.590 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:27:51.590 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.038472+0000", 2024-04-20T17:27:51.590 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6000000000000003e-06 2024-04-20T17:27:51.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.591 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.591 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:27:51.591 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.038533+0000", 2024-04-20T17:27:51.591 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0813000000000002e-05 2024-04-20T17:27:51.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.591 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.591 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:27:51.591 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.038628+0000", 2024-04-20T17:27:51.591 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4847999999999997e-05 2024-04-20T17:27:51.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.591 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.591 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:27:51.591 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.038708+0000", 2024-04-20T17:27:51.591 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0465999999999998e-05 2024-04-20T17:27:51.592 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.592 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.592 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:27:51.592 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.042385+0000", 2024-04-20T17:27:51.592 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003677067 2024-04-20T17:27:51.592 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.592 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.592 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:51.592 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.212627+0000", 2024-04-20T17:27:51.592 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17024218399999999 2024-04-20T17:27:51.592 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.592 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.592 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:27:51.592 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.212673+0000", 2024-04-20T17:27:51.593 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5843000000000001e-05 2024-04-20T17:27:51.593 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.593 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.593 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:27:51.593 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.212726+0000", 2024-04-20T17:27:51.593 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2973999999999998e-05 2024-04-20T17:27:51.593 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.593 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:27:51.593 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.593 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.593 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.593 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:207806 4.b 4:de8d426d:::benchmark_data_smithi171_122772_object37614:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:27:51.593 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:49.043798+0000", 2024-04-20T17:27:51.593 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 62.468548362999996, 2024-04-20T17:27:51.594 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.169070463, 2024-04-20T17:27:51.594 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:27:51.594 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:27:51.594 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:51.594 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:27:51.594 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:27:51.594 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:51.594 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 207806 2024-04-20T17:27:51.594 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.594 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:27:51.594 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.594 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:27:51.594 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.043798+0000", 2024-04-20T17:27:51.594 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.595 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.595 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:27:51.595 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.043798+0000", 2024-04-20T17:27:51.595 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.595 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.595 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:27:51.595 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.043800+0000", 2024-04-20T17:27:51.595 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3369999999999998e-06 2024-04-20T17:27:51.595 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.595 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:27:51.595 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.043806+0000", 2024-04-20T17:27:51.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1290000000000001e-06 2024-04-20T17:27:51.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.596 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:27:51.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.043818+0000", 2024-04-20T17:27:51.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1552999999999999e-05 2024-04-20T17:27:51.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.596 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:27:51.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.043824+0000", 2024-04-20T17:27:51.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5830000000000003e-06 2024-04-20T17:27:51.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.596 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:27:51.597 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.043888+0000", 2024-04-20T17:27:51.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3929000000000004e-05 2024-04-20T17:27:51.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.597 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:27:51.597 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.043972+0000", 2024-04-20T17:27:51.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3868000000000004e-05 2024-04-20T17:27:51.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.597 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:27:51.597 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.044045+0000", 2024-04-20T17:27:51.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3164000000000006e-05 2024-04-20T17:27:51.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.597 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:27:51.598 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.047745+0000", 2024-04-20T17:27:51.598 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0037001109999999999 2024-04-20T17:27:51.598 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.598 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.598 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:51.598 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.212776+0000", 2024-04-20T17:27:51.598 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.16503074400000001 2024-04-20T17:27:51.598 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.598 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.598 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:27:51.598 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.212830+0000", 2024-04-20T17:27:51.598 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3383e-05 2024-04-20T17:27:51.598 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.599 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.599 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:27:51.599 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.212868+0000", 2024-04-20T17:27:51.599 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8661999999999997e-05 2024-04-20T17:27:51.599 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.599 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:27:51.599 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.599 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.599 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.599 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:207829 4.b 4:dcebd7b4:::benchmark_data_smithi171_122772_object37637:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:27:51.599 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:49.065769+0000", 2024-04-20T17:27:51.599 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 62.446577183999999, 2024-04-20T17:27:51.599 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.150629235, 2024-04-20T17:27:51.599 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:27:51.600 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:27:51.600 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:51.600 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:27:51.600 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:27:51.600 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:51.600 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 207829 2024-04-20T17:27:51.600 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.600 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:27:51.600 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.600 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:27:51.600 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.065769+0000", 2024-04-20T17:27:51.600 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.600 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.600 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:27:51.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.065769+0000", 2024-04-20T17:27:51.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:27:51.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.065771+0000", 2024-04-20T17:27:51.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.345e-06 2024-04-20T17:27:51.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:27:51.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.065779+0000", 2024-04-20T17:27:51.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8089999999999993e-06 2024-04-20T17:27:51.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:27:51.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.065796+0000", 2024-04-20T17:27:51.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7231000000000002e-05 2024-04-20T17:27:51.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:27:51.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.065804+0000", 2024-04-20T17:27:51.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.525e-06 2024-04-20T17:27:51.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:27:51.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.065866+0000", 2024-04-20T17:27:51.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2204000000000003e-05 2024-04-20T17:27:51.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.603 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:27:51.603 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.065964+0000", 2024-04-20T17:27:51.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7664999999999994e-05 2024-04-20T17:27:51.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.603 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:27:51.603 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.066035+0000", 2024-04-20T17:27:51.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1517e-05 2024-04-20T17:27:51.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.603 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:27:51.603 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.069743+0000", 2024-04-20T17:27:51.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0037077439999999998 2024-04-20T17:27:51.604 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.604 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:51.604 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.216290+0000", 2024-04-20T17:27:51.604 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.146547179 2024-04-20T17:27:51.604 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.604 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:27:51.604 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.216350+0000", 2024-04-20T17:27:51.604 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0006000000000001e-05 2024-04-20T17:27:51.604 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.604 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:27:51.604 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.216398+0000", 2024-04-20T17:27:51.604 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8010000000000003e-05 2024-04-20T17:27:51.605 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.605 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:27:51.605 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.605 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.605 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.605 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:207852 4.b 4:df68e84b:::benchmark_data_smithi171_122772_object37660:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:27:51.605 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:49.093847+0000", 2024-04-20T17:27:51.605 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 62.418499396000001, 2024-04-20T17:27:51.605 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.12622141000000001, 2024-04-20T17:27:51.605 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:27:51.605 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:27:51.605 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:51.605 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:27:51.605 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:27:51.606 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:51.606 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 207852 2024-04-20T17:27:51.606 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.606 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:27:51.606 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.606 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:27:51.606 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.093847+0000", 2024-04-20T17:27:51.606 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.606 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.606 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.606 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:27:51.606 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.093847+0000", 2024-04-20T17:27:51.606 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.606 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.606 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.607 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:27:51.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.093849+0000", 2024-04-20T17:27:51.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3290000000000001e-06 2024-04-20T17:27:51.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.607 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:27:51.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.093857+0000", 2024-04-20T17:27:51.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2730000000000002e-06 2024-04-20T17:27:51.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.607 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:27:51.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.093873+0000", 2024-04-20T17:27:51.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5560999999999999e-05 2024-04-20T17:27:51.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.608 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.608 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:27:51.608 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.093882+0000", 2024-04-20T17:27:51.608 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9160000000000007e-06 2024-04-20T17:27:51.608 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.608 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.608 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:27:51.608 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.093967+0000", 2024-04-20T17:27:51.608 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5352000000000005e-05 2024-04-20T17:27:51.608 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.608 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.608 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:27:51.608 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.094105+0000", 2024-04-20T17:27:51.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013794499999999999 2024-04-20T17:27:51.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.609 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:27:51.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.094238+0000", 2024-04-20T17:27:51.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013271999999999999 2024-04-20T17:27:51.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.609 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:27:51.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.101892+0000", 2024-04-20T17:27:51.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0076538300000000004 2024-04-20T17:27:51.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.609 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:51.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.219960+0000", 2024-04-20T17:27:51.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.118068546 2024-04-20T17:27:51.610 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.610 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.610 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:27:51.610 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.220019+0000", 2024-04-20T17:27:51.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8569000000000001e-05 2024-04-20T17:27:51.610 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.610 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.610 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:27:51.610 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.220068+0000", 2024-04-20T17:27:51.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9369000000000001e-05 2024-04-20T17:27:51.610 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.610 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:27:51.611 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.611 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.611 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:207855 4.b 4:dbfba1ea:::benchmark_data_smithi171_122772_object37663:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:27:51.611 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:49.098673+0000", 2024-04-20T17:27:51.611 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 62.413673265, 2024-04-20T17:27:51.611 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.12152695500000001, 2024-04-20T17:27:51.611 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:27:51.611 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:27:51.611 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:51.611 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:27:51.611 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:27:51.611 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:51.611 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 207855 2024-04-20T17:27:51.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.611 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:27:51.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.612 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:27:51.612 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.098673+0000", 2024-04-20T17:27:51.612 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.612 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:27:51.612 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.098673+0000", 2024-04-20T17:27:51.612 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.612 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:27:51.612 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.098676+0000", 2024-04-20T17:27:51.612 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6429999999999999e-06 2024-04-20T17:27:51.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.613 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.613 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:27:51.613 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.098694+0000", 2024-04-20T17:27:51.613 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8252000000000001e-05 2024-04-20T17:27:51.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.613 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.613 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:27:51.613 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.098709+0000", 2024-04-20T17:27:51.613 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5187e-05 2024-04-20T17:27:51.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.613 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.613 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:27:51.613 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.098716+0000", 2024-04-20T17:27:51.613 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8159999999999996e-06 2024-04-20T17:27:51.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.614 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:27:51.614 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.098831+0000", 2024-04-20T17:27:51.614 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011483400000000001 2024-04-20T17:27:51.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.614 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:27:51.614 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.098920+0000", 2024-04-20T17:27:51.614 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9468999999999993e-05 2024-04-20T17:27:51.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.614 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:27:51.614 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.099009+0000", 2024-04-20T17:27:51.615 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8702000000000006e-05 2024-04-20T17:27:51.615 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.615 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:27:51.615 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.102024+0000", 2024-04-20T17:27:51.615 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0030156789999999998 2024-04-20T17:27:51.615 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.615 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:51.615 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.220109+0000", 2024-04-20T17:27:51.615 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.118084894 2024-04-20T17:27:51.615 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.615 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:27:51.616 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.220159+0000", 2024-04-20T17:27:51.616 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9611999999999999e-05 2024-04-20T17:27:51.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.616 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:27:51.616 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.220200+0000", 2024-04-20T17:27:51.616 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0867e-05 2024-04-20T17:27:51.616 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.616 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:27:51.616 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.616 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:246323 4.b 4:d6e9374e:::benchmark_data_smithi171_122772_object76131:head [delete] snapc 0=[] ondisk+write+known_if_redirected e502)", 2024-04-20T17:27:51.616 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:27:20.721113+0000", 2024-04-20T17:27:51.616 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 30.791233369, 2024-04-20T17:27:51.617 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.044076762999999998, 2024-04-20T17:27:51.617 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:27:51.617 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:27:51.617 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:51.617 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:27:51.617 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:27:51.617 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:51.617 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 246323 2024-04-20T17:27:51.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.617 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:27:51.617 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.617 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:27:51.617 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.721113+0000", 2024-04-20T17:27:51.617 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.618 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.618 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:27:51.618 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.721113+0000", 2024-04-20T17:27:51.618 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.618 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.618 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:27:51.618 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.721113+0000", 2024-04-20T17:27:51.618 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3799999999999997e-07 2024-04-20T17:27:51.618 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.618 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:27:51.618 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.721116+0000", 2024-04-20T17:27:51.619 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7140000000000002e-06 2024-04-20T17:27:51.619 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.619 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:27:51.619 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.721123+0000", 2024-04-20T17:27:51.619 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.477e-06 2024-04-20T17:27:51.619 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.619 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:27:51.619 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.721128+0000", 2024-04-20T17:27:51.619 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8540000000000002e-06 2024-04-20T17:27:51.619 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.619 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:27:51.620 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.735665+0000", 2024-04-20T17:27:51.620 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014536201 2024-04-20T17:27:51.620 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.620 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.620 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:27:51.620 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.735807+0000", 2024-04-20T17:27:51.620 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000142087 2024-04-20T17:27:51.620 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.620 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.620 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:27:51.620 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.735952+0000", 2024-04-20T17:27:51.620 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014521299999999999 2024-04-20T17:27:51.620 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.620 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.621 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:27:51.621 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.743867+0000", 2024-04-20T17:27:51.621 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0079148139999999992 2024-04-20T17:27:51.621 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.621 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.621 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:51.621 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.765085+0000", 2024-04-20T17:27:51.621 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.02121779 2024-04-20T17:27:51.621 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.621 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.621 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:27:51.621 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.765141+0000", 2024-04-20T17:27:51.621 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5980000000000003e-05 2024-04-20T17:27:51.621 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.622 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:27:51.622 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.765190+0000", 2024-04-20T17:27:51.622 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8995000000000001e-05 2024-04-20T17:27:51.622 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.622 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:27:51.622 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.622 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:246424 4.b 4:db8b6109:::benchmark_data_smithi171_122772_object76232:head [delete] snapc 0=[] ondisk+write+known_if_redirected e502)", 2024-04-20T17:27:51.622 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:27:20.821182+0000", 2024-04-20T17:27:51.622 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 30.691163794000001, 2024-04-20T17:27:51.622 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.044028312999999999, 2024-04-20T17:27:51.622 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:27:51.623 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:27:51.623 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:51.623 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:27:51.623 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:27:51.623 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:51.623 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 246424 2024-04-20T17:27:51.623 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.623 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:27:51.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.623 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:27:51.623 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.821182+0000", 2024-04-20T17:27:51.623 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.623 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.623 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:27:51.624 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.821182+0000", 2024-04-20T17:27:51.624 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.624 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.624 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.624 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:27:51.624 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.821185+0000", 2024-04-20T17:27:51.624 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.762e-06 2024-04-20T17:27:51.624 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.624 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.624 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:27:51.624 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.821193+0000", 2024-04-20T17:27:51.624 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3149999999999992e-06 2024-04-20T17:27:51.624 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.625 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.625 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:27:51.625 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.821216+0000", 2024-04-20T17:27:51.625 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2314e-05 2024-04-20T17:27:51.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.625 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.625 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:27:51.625 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.821228+0000", 2024-04-20T17:27:51.625 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2007999999999999e-05 2024-04-20T17:27:51.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.625 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.625 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:27:51.625 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.821307+0000", 2024-04-20T17:27:51.625 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8887999999999994e-05 2024-04-20T17:27:51.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.626 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.626 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:27:51.626 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.821473+0000", 2024-04-20T17:27:51.626 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00016594299999999999 2024-04-20T17:27:51.626 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.626 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.626 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:27:51.626 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.821605+0000", 2024-04-20T17:27:51.626 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013271999999999999 2024-04-20T17:27:51.626 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.626 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.626 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:27:51.626 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.832396+0000", 2024-04-20T17:27:51.626 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010790933000000001 2024-04-20T17:27:51.627 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.627 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.627 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:51.627 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.865103+0000", 2024-04-20T17:27:51.627 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.032706582999999997 2024-04-20T17:27:51.627 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.627 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.627 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:27:51.627 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.865160+0000", 2024-04-20T17:27:51.627 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7478000000000001e-05 2024-04-20T17:27:51.627 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.627 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.627 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:27:51.628 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.865211+0000", 2024-04-20T17:27:51.628 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0368999999999998e-05 2024-04-20T17:27:51.628 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.628 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:27:51.628 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.628 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.628 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.628 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:249431 4.b 4:d3998a0f:::benchmark_data_smithi171_122772_object79239:head [delete] snapc 0=[] ondisk+write+known_if_redirected e518)", 2024-04-20T17:27:51.628 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:27:49.358745+0000", 2024-04-20T17:27:51.628 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 2.153600994, 2024-04-20T17:27:51.628 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.042527699000000002, 2024-04-20T17:27:51.628 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:27:51.628 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:27:51.628 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:51.629 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:27:51.629 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:27:51.629 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:51.629 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 249431 2024-04-20T17:27:51.629 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.629 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:27:51.629 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.629 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:27:51.629 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:49.358745+0000", 2024-04-20T17:27:51.629 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.629 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.629 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.629 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:27:51.629 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:49.358745+0000", 2024-04-20T17:27:51.629 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.629 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.630 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.630 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:27:51.630 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:49.358746+0000", 2024-04-20T17:27:51.630 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.062e-06 2024-04-20T17:27:51.630 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.630 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.630 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:27:51.630 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:49.358750+0000", 2024-04-20T17:27:51.630 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2390000000000004e-06 2024-04-20T17:27:51.630 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.630 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.630 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:27:51.630 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:49.358757+0000", 2024-04-20T17:27:51.631 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7599999999999997e-06 2024-04-20T17:27:51.631 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.631 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.631 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:27:51.631 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:49.358764+0000", 2024-04-20T17:27:51.631 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5069999999999997e-06 2024-04-20T17:27:51.631 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.631 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.631 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:27:51.631 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:49.367320+0000", 2024-04-20T17:27:51.631 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.008556391 2024-04-20T17:27:51.631 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.631 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.631 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:27:51.632 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:49.367461+0000", 2024-04-20T17:27:51.632 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000140906 2024-04-20T17:27:51.632 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.632 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.632 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:27:51.632 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:49.367587+0000", 2024-04-20T17:27:51.632 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012575100000000001 2024-04-20T17:27:51.632 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.632 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.632 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:27:51.632 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:49.377683+0000", 2024-04-20T17:27:51.632 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010096647 2024-04-20T17:27:51.632 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.632 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.633 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:51.633 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:49.401168+0000", 2024-04-20T17:27:51.633 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023484445999999999 2024-04-20T17:27:51.633 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.633 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.633 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:27:51.633 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:49.401221+0000", 2024-04-20T17:27:51.633 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3097999999999999e-05 2024-04-20T17:27:51.633 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.633 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.633 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:27:51.633 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:49.401273+0000", 2024-04-20T17:27:51.633 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1891999999999997e-05 2024-04-20T17:27:51.633 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.634 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:27:51.634 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.634 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.634 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.634 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:251668 4.b 4:def8e2b4:::benchmark_data_smithi171_122772_object81476:head [delete] snapc 0=[] ondisk+write+known_if_redirected e518)", 2024-04-20T17:27:51.634 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:27:51.241488+0000", 2024-04-20T17:27:51.634 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.27085810799999999, 2024-04-20T17:27:51.634 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.022543377999999999, 2024-04-20T17:27:51.634 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:27:51.634 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:27:51.634 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:27:51.634 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:27:51.634 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:27:51.635 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:27:51.635 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 251668 2024-04-20T17:27:51.635 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.635 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:27:51.635 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.635 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:27:51.635 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:51.241488+0000", 2024-04-20T17:27:51.635 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.635 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.635 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.635 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:27:51.635 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:51.241488+0000", 2024-04-20T17:27:51.635 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:27:51.635 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.636 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.636 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:27:51.636 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:51.241489+0000", 2024-04-20T17:27:51.636 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.06e-06 2024-04-20T17:27:51.636 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.636 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.636 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:27:51.636 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:51.241493+0000", 2024-04-20T17:27:51.636 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2749999999999997e-06 2024-04-20T17:27:51.636 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.636 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.636 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:27:51.636 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:51.241502+0000", 2024-04-20T17:27:51.636 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6600000000000001e-06 2024-04-20T17:27:51.636 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.637 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.637 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:27:51.637 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:51.241509+0000", 2024-04-20T17:27:51.637 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3819999999999998e-06 2024-04-20T17:27:51.637 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.637 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.637 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:27:51.637 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:51.245686+0000", 2024-04-20T17:27:51.637 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041770389999999996 2024-04-20T17:27:51.637 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.637 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.637 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:27:51.637 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:51.245800+0000", 2024-04-20T17:27:51.637 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011355 2024-04-20T17:27:51.637 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.637 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.638 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:27:51.638 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:51.245873+0000", 2024-04-20T17:27:51.638 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3282999999999996e-05 2024-04-20T17:27:51.638 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.638 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.638 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:27:51.638 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:51.255594+0000", 2024-04-20T17:27:51.638 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0097202940000000009 2024-04-20T17:27:51.638 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.638 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.638 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:27:51.638 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:51.263937+0000", 2024-04-20T17:27:51.638 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00834353 2024-04-20T17:27:51.638 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.639 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.639 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:27:51.639 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:51.263995+0000", 2024-04-20T17:27:51.639 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8214999999999997e-05 2024-04-20T17:27:51.639 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:27:51.639 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:27:51.639 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:27:51.639 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:51.264031+0000", 2024-04-20T17:27:51.639 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6090000000000002e-05 2024-04-20T17:27:51.639 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.639 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:27:51.639 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.639 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:27:51.639 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:27:51.639 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:27:51.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:51.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:27:51.741 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:27:51.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:51 smithi106 ceph-mon[44423]: pgmap v1360: 17 pgs: 17 active+clean; 5.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 54 op/s; 48 MiB/s, 769 objects/s recovering 2024-04-20T17:27:51.793 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:51.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:51 smithi171 ceph-mon[44189]: pgmap v1360: 17 pgs: 17 active+clean; 5.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 54 op/s; 48 MiB/s, 769 objects/s recovering 2024-04-20T17:27:52.054 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:27:52.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:52.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:27:52.160 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:27:52.208 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:52.239 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:27:52.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:52.244 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:27:52.345 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:27:52.386 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:52.571 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.6 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.5.6 2024-04-20T17:27:52.585 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:27:52.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:52.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:27:52.689 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:27:52.738 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:52.826 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.6/config 2024-04-20T17:27:53.365 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:27:53.405 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-20T17:27:53.438 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:27:53.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:53.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:27:53.444 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:27:53.545 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:27:53.578 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:53.605 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:27:53.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:53.608 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:27:53.709 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:27:53.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:53 smithi005 ceph-mon[39562]: pgmap v1361: 17 pgs: 17 active+clean; 5.5 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 310 op/s; 39 MiB/s, 628 objects/s recovering 2024-04-20T17:27:53.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:53 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2911734113' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:27:53.737 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:53 smithi106 ceph-mon[44423]: pgmap v1361: 17 pgs: 17 active+clean; 5.5 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 310 op/s; 39 MiB/s, 628 objects/s recovering 2024-04-20T17:27:53.737 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:53 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2911734113' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:27:53.762 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:53.806 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:27:53.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:53.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:27:53.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:53 smithi171 ceph-mon[44189]: pgmap v1361: 17 pgs: 17 active+clean; 5.5 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 310 op/s; 39 MiB/s, 628 objects/s recovering 2024-04-20T17:27:53.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:53 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2911734113' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:27:53.909 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:27:53.947 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:53.968 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:27:53.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:53.972 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:27:54.073 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:27:54.123 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:54.151 INFO:teuthology.orchestra.run.smithi106.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:27:54.690 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:27:54.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:54.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:27:54.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:54 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:27:54.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:54 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2911734113' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:27:54.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:54 smithi005 ceph-mon[39562]: osdmap e519: 12 total, 11 up, 11 in 2024-04-20T17:27:54.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:54 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:27:54.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:54 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2911734113' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:27:54.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:54 smithi106 ceph-mon[44423]: osdmap e519: 12 total, 11 up, 11 in 2024-04-20T17:27:54.796 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:27:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:54 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:27:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:54 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2911734113' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:27:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:54 smithi171 ceph-mon[44189]: osdmap e519: 12 total, 11 up, 11 in 2024-04-20T17:27:54.847 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:54.870 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:27:54.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:54.873 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:27:54.904 INFO:teuthology.orchestra.run.smithi106.stdout:get_pg_num_history pg_num_history pg_num_history(e503 pg_nums {1={14=1},4={445=16}} deleted_pools ) 2024-04-20T17:27:54.904 INFO:teuthology.orchestra.run.smithi106.stdout:Importing pgid 4.5 2024-04-20T17:27:54.915 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a000538d:::benchmark_data_smithi171_122772_object107954:head# 2024-04-20T17:27:54.915 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.915 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0027801:::benchmark_data_smithi171_122772_object127544:head# 2024-04-20T17:27:54.915 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.916 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a002d6fc:::benchmark_data_smithi171_122772_object168070:head# 2024-04-20T17:27:54.916 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.916 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0035878:::benchmark_data_smithi171_122772_object126831:head# 2024-04-20T17:27:54.916 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.916 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0046a9d:::benchmark_data_smithi171_122772_object120482:head# 2024-04-20T17:27:54.916 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.916 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0046e3b:::benchmark_data_smithi171_122772_object150652:head# 2024-04-20T17:27:54.916 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.916 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a004b228:::benchmark_data_smithi171_122772_object122765:head# 2024-04-20T17:27:54.917 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.918 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a00592c1:::benchmark_data_smithi171_122772_object150774:head# 2024-04-20T17:27:54.918 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.918 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0075b00:::benchmark_data_smithi171_122772_object131539:head# 2024-04-20T17:27:54.918 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.918 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a007c405:::benchmark_data_smithi171_122772_object98194:head# 2024-04-20T17:27:54.918 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.918 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a008e900:::benchmark_data_smithi171_122772_object120791:head# 2024-04-20T17:27:54.918 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.918 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0097346:::benchmark_data_smithi171_122772_object84061:head# 2024-04-20T17:27:54.918 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.918 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a00a03fc:::benchmark_data_smithi171_122772_object105620:head# 2024-04-20T17:27:54.919 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.920 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a00b37a7:::benchmark_data_smithi171_122772_object126331:head# 2024-04-20T17:27:54.920 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.920 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a00bb3cf:::benchmark_data_smithi171_122772_object112003:head# 2024-04-20T17:27:54.920 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.920 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a00c9ef8:::benchmark_data_smithi171_122772_object148592:head# 2024-04-20T17:27:54.920 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.920 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a00e17b4:::benchmark_data_smithi171_122772_object134388:head# 2024-04-20T17:27:54.920 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.920 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a00e17f4:::benchmark_data_smithi171_122772_object138284:head# 2024-04-20T17:27:54.920 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.920 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a00f8578:::benchmark_data_smithi171_122772_object92270:head# 2024-04-20T17:27:54.921 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.925 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a00fd006:::benchmark_data_smithi171_122772_object146275:head# 2024-04-20T17:27:54.925 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.925 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a00fe5e3:::benchmark_data_smithi171_122772_object148508:head# 2024-04-20T17:27:54.925 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.925 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a011ef33:::benchmark_data_smithi171_122772_object135649:head# 2024-04-20T17:27:54.925 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.925 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a01202f5:::benchmark_data_smithi171_122772_object137599:head# 2024-04-20T17:27:54.925 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.925 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0131585:::benchmark_data_smithi171_122772_object105119:head# 2024-04-20T17:27:54.925 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.925 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a014d493:::benchmark_data_smithi171_122772_object126718:head# 2024-04-20T17:27:54.925 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.925 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a01522ff:::benchmark_data_smithi171_122772_object109167:head# 2024-04-20T17:27:54.925 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.926 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a015bde0:::benchmark_data_smithi171_122772_object86545:head# 2024-04-20T17:27:54.926 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.926 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a017321b:::benchmark_data_smithi171_122772_object104394:head# 2024-04-20T17:27:54.926 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.926 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0183041:::benchmark_data_smithi171_122772_object155021:head# 2024-04-20T17:27:54.926 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.926 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a018f509:::benchmark_data_smithi171_122772_object104542:head# 2024-04-20T17:27:54.926 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.926 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a019432c:::benchmark_data_smithi171_122772_object102066:head# 2024-04-20T17:27:54.926 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.926 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a01a94c3:::benchmark_data_smithi171_122772_object166565:head# 2024-04-20T17:27:54.926 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.926 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a01b9f09:::benchmark_data_smithi171_122772_object130679:head# 2024-04-20T17:27:54.926 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.927 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a01c5c4f:::benchmark_data_smithi171_122772_object162286:head# 2024-04-20T17:27:54.927 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.927 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0201eec:::benchmark_data_smithi171_122772_object111745:head# 2024-04-20T17:27:54.927 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.927 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0201f89:::benchmark_data_smithi171_122772_object169020:head# 2024-04-20T17:27:54.927 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.927 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0205a32:::benchmark_data_smithi171_122772_object140691:head# 2024-04-20T17:27:54.927 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.927 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a02069a8:::benchmark_data_smithi171_122772_object158408:head# 2024-04-20T17:27:54.927 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.927 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a020c239:::benchmark_data_smithi171_122772_object131170:head# 2024-04-20T17:27:54.927 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.927 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0213451:::benchmark_data_smithi171_122772_object83669:head# 2024-04-20T17:27:54.927 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.927 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a021aebd:::benchmark_data_smithi171_122772_object110055:head# 2024-04-20T17:27:54.928 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.928 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0227bbc:::benchmark_data_smithi171_122772_object123533:head# 2024-04-20T17:27:54.928 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.928 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0256cbe:::benchmark_data_smithi171_122772_object103878:head# 2024-04-20T17:27:54.928 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.932 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0258c61:::benchmark_data_smithi171_122772_object84914:head# 2024-04-20T17:27:54.932 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.932 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a02759d8:::benchmark_data_smithi171_122772_object156278:head# 2024-04-20T17:27:54.932 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.932 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a02898db:::benchmark_data_smithi171_122772_object153016:head# 2024-04-20T17:27:54.932 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.932 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a028bf9d:::benchmark_data_smithi171_122772_object121166:head# 2024-04-20T17:27:54.932 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.932 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a02a764b:::benchmark_data_smithi171_122772_object104155:head# 2024-04-20T17:27:54.932 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.933 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a02b45b4:::benchmark_data_smithi171_122772_object80948:head# 2024-04-20T17:27:54.933 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.933 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a02b9b7c:::benchmark_data_smithi171_122772_object128443:head# 2024-04-20T17:27:54.933 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.933 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a02c5cee:::benchmark_data_smithi171_122772_object146669:head# 2024-04-20T17:27:54.933 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.933 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a02c7ef5:::benchmark_data_smithi171_122772_object159443:head# 2024-04-20T17:27:54.933 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.933 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a02c81e1:::benchmark_data_smithi171_122772_object157547:head# 2024-04-20T17:27:54.933 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.933 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a02cb5be:::benchmark_data_smithi171_122772_object164341:head# 2024-04-20T17:27:54.933 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.934 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a02e7a4c:::benchmark_data_smithi171_122772_object95590:head# 2024-04-20T17:27:54.934 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.934 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a02ea583:::benchmark_data_smithi171_122772_object88041:head# 2024-04-20T17:27:54.934 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.934 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a02eb62b:::benchmark_data_smithi171_122772_object86829:head# 2024-04-20T17:27:54.934 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.934 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a02fc404:::benchmark_data_smithi171_122772_object167535:head# 2024-04-20T17:27:54.934 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.934 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a030237b:::benchmark_data_smithi171_122772_object83925:head# 2024-04-20T17:27:54.934 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.934 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a030f91d:::benchmark_data_smithi171_122772_object165575:head# 2024-04-20T17:27:54.934 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.934 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0312b83:::benchmark_data_smithi171_122772_object86639:head# 2024-04-20T17:27:54.935 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.935 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a03158c8:::benchmark_data_smithi171_122772_object157088:head# 2024-04-20T17:27:54.935 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.935 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a031f6e6:::benchmark_data_smithi171_122772_object137876:head# 2024-04-20T17:27:54.935 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.935 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0329d3e:::benchmark_data_smithi171_122772_object120884:head# 2024-04-20T17:27:54.935 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.935 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a032da83:::benchmark_data_smithi171_122772_object150855:head# 2024-04-20T17:27:54.935 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.935 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0340a13:::benchmark_data_smithi171_122772_object110989:head# 2024-04-20T17:27:54.935 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.935 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a034f0d3:::benchmark_data_smithi171_122772_object84100:head# 2024-04-20T17:27:54.935 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.935 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0353cdd:::benchmark_data_smithi171_122772_object129989:head# 2024-04-20T17:27:54.935 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.936 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0368f4a:::benchmark_data_smithi171_122772_object161489:head# 2024-04-20T17:27:54.936 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.936 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a036b2fb:::benchmark_data_smithi171_122772_object159566:head# 2024-04-20T17:27:54.936 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.943 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a037a940:::benchmark_data_smithi171_122772_object125154:head# 2024-04-20T17:27:54.943 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.943 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a037e61a:::benchmark_data_smithi171_122772_object94998:head# 2024-04-20T17:27:54.944 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.944 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0385c85:::benchmark_data_smithi171_122772_object94382:head# 2024-04-20T17:27:54.944 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.944 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a038d700:::benchmark_data_smithi171_122772_object169540:head# 2024-04-20T17:27:54.944 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.944 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a039542f:::benchmark_data_smithi171_122772_object104673:head# 2024-04-20T17:27:54.944 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.944 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a03ab17e:::benchmark_data_smithi171_122772_object137729:head# 2024-04-20T17:27:54.944 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.944 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a03b2c89:::benchmark_data_smithi171_122772_object110923:head# 2024-04-20T17:27:54.944 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.944 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a03b85f6:::benchmark_data_smithi171_122772_object112668:head# 2024-04-20T17:27:54.944 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.944 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a03bd528:::benchmark_data_smithi171_122772_object91712:head# 2024-04-20T17:27:54.945 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.945 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a03c4401:::benchmark_data_smithi171_122772_object153159:head# 2024-04-20T17:27:54.945 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.945 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a03d39f3:::benchmark_data_smithi171_122772_object111897:head# 2024-04-20T17:27:54.945 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.945 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a03dcb59:::benchmark_data_smithi171_122772_object78378:head# 2024-04-20T17:27:54.945 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.945 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a03e6a7f:::benchmark_data_smithi171_122772_object152887:head# 2024-04-20T17:27:54.945 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.945 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a03ee0b7:::benchmark_data_smithi171_122772_object147040:head# 2024-04-20T17:27:54.945 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.945 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a03f3b90:::benchmark_data_smithi171_122772_object121173:head# 2024-04-20T17:27:54.945 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.945 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a03fa48b:::benchmark_data_smithi171_122772_object111559:head# 2024-04-20T17:27:54.945 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.946 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a03fad21:::benchmark_data_smithi171_122772_object101261:head# 2024-04-20T17:27:54.946 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.946 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0406e49:::benchmark_data_smithi171_122772_object163440:head# 2024-04-20T17:27:54.946 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.946 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a040f3c8:::benchmark_data_smithi171_122772_object106981:head# 2024-04-20T17:27:54.946 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.946 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a041600b:::benchmark_data_smithi171_122772_object161603:head# 2024-04-20T17:27:54.946 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.946 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a041e501:::benchmark_data_smithi171_122772_object153785:head# 2024-04-20T17:27:54.946 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.946 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0421f5c:::benchmark_data_smithi171_122772_object125517:head# 2024-04-20T17:27:54.946 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.946 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a044f5d5:::benchmark_data_smithi171_122772_object121132:head# 2024-04-20T17:27:54.946 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.947 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0458c8c:::benchmark_data_smithi171_122772_object155444:head# 2024-04-20T17:27:54.947 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.947 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a04659b4:::benchmark_data_smithi171_122772_object90726:head# 2024-04-20T17:27:54.947 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.947 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a046d372:::benchmark_data_smithi171_122772_object99519:head# 2024-04-20T17:27:54.947 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.947 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a047c394:::benchmark_data_smithi171_122772_object110175:head# 2024-04-20T17:27:54.947 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.947 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a04e192c:::benchmark_data_smithi171_122772_object112549:head# 2024-04-20T17:27:54.947 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.947 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a04f5fbe:::benchmark_data_smithi171_122772_object121780:head# 2024-04-20T17:27:54.947 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.947 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a04f8857:::benchmark_data_smithi171_122772_object148970:head# 2024-04-20T17:27:54.947 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.948 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0517f07:::benchmark_data_smithi171_122772_object100226:head# 2024-04-20T17:27:54.948 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.948 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0519240:::benchmark_data_smithi171_122772_object154757:head# 2024-04-20T17:27:54.948 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.948 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a051e05b:::benchmark_data_smithi171_122772_object164401:head# 2024-04-20T17:27:54.948 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.948 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0525716:::benchmark_data_smithi171_122772_object78800:head# 2024-04-20T17:27:54.948 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.948 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0542278:::benchmark_data_smithi171_122772_object99751:head# 2024-04-20T17:27:54.948 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.948 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0546dec:::benchmark_data_smithi171_122772_object100801:head# 2024-04-20T17:27:54.948 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.948 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a055fd11:::benchmark_data_smithi171_122772_object158413:head# 2024-04-20T17:27:54.948 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.948 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a056102f:::benchmark_data_smithi171_122772_object135487:head# 2024-04-20T17:27:54.948 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.949 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a056b1b6:::benchmark_data_smithi171_122772_object117591:head# 2024-04-20T17:27:54.949 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.949 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a056c578:::benchmark_data_smithi171_122772_object113439:head# 2024-04-20T17:27:54.949 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.949 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a058127d:::benchmark_data_smithi171_122772_object134612:head# 2024-04-20T17:27:54.949 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.949 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a059c372:::benchmark_data_smithi171_122772_object98716:head# 2024-04-20T17:27:54.949 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.949 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a059d6ae:::benchmark_data_smithi171_122772_object108487:head# 2024-04-20T17:27:54.949 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.949 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a05a05f2:::benchmark_data_smithi171_122772_object169794:head# 2024-04-20T17:27:54.949 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.949 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a05a9ad7:::benchmark_data_smithi171_122772_object113511:head# 2024-04-20T17:27:54.950 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.950 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a05aefac:::benchmark_data_smithi171_122772_object116250:head# 2024-04-20T17:27:54.950 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.963 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a05f02d1:::benchmark_data_smithi171_122772_object104851:head# 2024-04-20T17:27:54.963 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.963 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a05f0a49:::benchmark_data_smithi171_122772_object106211:head# 2024-04-20T17:27:54.963 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.963 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a05f6a62:::benchmark_data_smithi171_122772_object156136:head# 2024-04-20T17:27:54.963 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.963 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a060345a:::benchmark_data_smithi171_122772_object161821:head# 2024-04-20T17:27:54.963 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.963 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a060d70e:::benchmark_data_smithi171_122772_object168669:head# 2024-04-20T17:27:54.963 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.963 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a06304ad:::benchmark_data_smithi171_122772_object140396:head# 2024-04-20T17:27:54.964 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.964 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a063ea95:::benchmark_data_smithi171_122772_object91790:head# 2024-04-20T17:27:54.964 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.964 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a063f524:::benchmark_data_smithi171_122772_object167014:head# 2024-04-20T17:27:54.964 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.964 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a064ea80:::benchmark_data_smithi171_122772_object166896:head# 2024-04-20T17:27:54.964 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.964 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a065af90:::benchmark_data_smithi171_122772_object113318:head# 2024-04-20T17:27:54.964 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.964 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a066fd10:::benchmark_data_smithi171_122772_object143060:head# 2024-04-20T17:27:54.964 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.964 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a06a7f92:::benchmark_data_smithi171_122772_object136146:head# 2024-04-20T17:27:54.964 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.964 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a06b275b:::benchmark_data_smithi171_122772_object105099:head# 2024-04-20T17:27:54.965 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.965 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a06ca089:::benchmark_data_smithi171_122772_object169551:head# 2024-04-20T17:27:54.965 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.965 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a06ce330:::benchmark_data_smithi171_122772_object147827:head# 2024-04-20T17:27:54.965 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.965 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a06d18b4:::benchmark_data_smithi171_122772_object119654:head# 2024-04-20T17:27:54.965 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.965 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a06d82aa:::benchmark_data_smithi171_122772_object167093:head# 2024-04-20T17:27:54.965 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.965 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a06d9cd8:::benchmark_data_smithi171_122772_object139711:head# 2024-04-20T17:27:54.965 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.965 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a06ddd3a:::benchmark_data_smithi171_122772_object101204:head# 2024-04-20T17:27:54.965 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.966 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0708d06:::benchmark_data_smithi171_122772_object116584:head# 2024-04-20T17:27:54.966 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.966 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a071179d:::benchmark_data_smithi171_122772_object139235:head# 2024-04-20T17:27:54.966 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.966 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a07147a9:::benchmark_data_smithi171_122772_object85173:head# 2024-04-20T17:27:54.966 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.966 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0741e3c:::benchmark_data_smithi171_122772_object105716:head# 2024-04-20T17:27:54.966 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.966 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a074ccbc:::benchmark_data_smithi171_122772_object82151:head# 2024-04-20T17:27:54.966 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.966 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0752fa4:::benchmark_data_smithi171_122772_object150217:head# 2024-04-20T17:27:54.966 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.966 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a076538d:::benchmark_data_smithi171_122772_object105308:head# 2024-04-20T17:27:54.966 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.967 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0768bcf:::benchmark_data_smithi171_122772_object113047:head# 2024-04-20T17:27:54.967 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.967 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a076f600:::benchmark_data_smithi171_122772_object130269:head# 2024-04-20T17:27:54.967 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.967 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a077a3b2:::benchmark_data_smithi171_122772_object111259:head# 2024-04-20T17:27:54.967 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.967 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a077fe48:::benchmark_data_smithi171_122772_object132882:head# 2024-04-20T17:27:54.967 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.967 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0780753:::benchmark_data_smithi171_122772_object122760:head# 2024-04-20T17:27:54.967 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.967 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a078e99d:::benchmark_data_smithi171_122772_object154976:head# 2024-04-20T17:27:54.967 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.967 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0790bc3:::benchmark_data_smithi171_122772_object109292:head# 2024-04-20T17:27:54.968 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.968 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a07a14ae:::benchmark_data_smithi171_122772_object79491:head# 2024-04-20T17:27:54.968 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.968 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a07a2b82:::benchmark_data_smithi171_122772_object98762:head# 2024-04-20T17:27:54.968 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.968 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a07b1545:::benchmark_data_smithi171_122772_object96015:head# 2024-04-20T17:27:54.968 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.968 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a07c023a:::benchmark_data_smithi171_122772_object82343:head# 2024-04-20T17:27:54.968 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.968 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a07c81c6:::benchmark_data_smithi171_122772_object158436:head# 2024-04-20T17:27:54.968 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.968 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a07d1f53:::benchmark_data_smithi171_122772_object134980:head# 2024-04-20T17:27:54.968 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.969 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a07d2b5b:::benchmark_data_smithi171_122772_object111738:head# 2024-04-20T17:27:54.969 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.969 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a07d4505:::benchmark_data_smithi171_122772_object106430:head# 2024-04-20T17:27:54.969 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.969 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a07dbb51:::benchmark_data_smithi171_122772_object92765:head# 2024-04-20T17:27:54.969 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.969 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a07e2738:::benchmark_data_smithi171_122772_object169164:head# 2024-04-20T17:27:54.969 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.969 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a07e91c5:::benchmark_data_smithi171_122772_object88494:head# 2024-04-20T17:27:54.969 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.969 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a07f2008:::benchmark_data_smithi171_122772_object159522:head# 2024-04-20T17:27:54.969 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.969 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a07f9cf2:::benchmark_data_smithi171_122772_object140211:head# 2024-04-20T17:27:54.969 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.969 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a07ff679:::benchmark_data_smithi171_122772_object144519:head# 2024-04-20T17:27:54.970 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.970 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a080362a:::benchmark_data_smithi171_122772_object105945:head# 2024-04-20T17:27:54.970 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.970 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0803bd1:::benchmark_data_smithi171_122772_object106992:head# 2024-04-20T17:27:54.970 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.970 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0815d78:::benchmark_data_smithi171_122772_object112318:head# 2024-04-20T17:27:54.970 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.970 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a08182d5:::benchmark_data_smithi171_122772_object147052:head# 2024-04-20T17:27:54.970 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.970 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a082632b:::benchmark_data_smithi171_122772_object100957:head# 2024-04-20T17:27:54.970 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.970 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a084c1c4:::benchmark_data_smithi171_122772_object143305:head# 2024-04-20T17:27:54.970 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.970 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a084fa1d:::benchmark_data_smithi171_122772_object90177:head# 2024-04-20T17:27:54.971 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.971 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a08508bf:::benchmark_data_smithi171_122772_object125536:head# 2024-04-20T17:27:54.971 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.971 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a086122f:::benchmark_data_smithi171_122772_object122140:head# 2024-04-20T17:27:54.971 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.971 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a087fb64:::benchmark_data_smithi171_122772_object137453:head# 2024-04-20T17:27:54.971 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.971 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a08845a0:::benchmark_data_smithi171_122772_object110879:head# 2024-04-20T17:27:54.971 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.971 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a088ac06:::benchmark_data_smithi171_122772_object125834:head# 2024-04-20T17:27:54.971 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.971 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0898ad7:::benchmark_data_smithi171_122772_object136540:head# 2024-04-20T17:27:54.971 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.971 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a08a5aa4:::benchmark_data_smithi171_122772_object113784:head# 2024-04-20T17:27:54.972 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.972 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a08a89b5:::benchmark_data_smithi171_122772_object78548:head# 2024-04-20T17:27:54.972 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.972 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a08aa56b:::benchmark_data_smithi171_122772_object85563:head# 2024-04-20T17:27:54.972 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.972 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a08b7846:::benchmark_data_smithi171_122772_object128112:head# 2024-04-20T17:27:54.972 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.972 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a08ba3fe:::benchmark_data_smithi171_122772_object103136:head# 2024-04-20T17:27:54.972 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.972 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a08be249:::benchmark_data_smithi171_122772_object133803:head# 2024-04-20T17:27:54.972 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.973 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a08c786f:::benchmark_data_smithi171_122772_object99934:head# 2024-04-20T17:27:54.973 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.973 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a08cad7a:::benchmark_data_smithi171_122772_object99549:head# 2024-04-20T17:27:54.973 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.973 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a08d279d:::benchmark_data_smithi171_122772_object104084:head# 2024-04-20T17:27:54.973 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.973 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a08d29b0:::benchmark_data_smithi171_122772_object127248:head# 2024-04-20T17:27:54.973 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.973 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a08dd588:::benchmark_data_smithi171_122772_object145850:head# 2024-04-20T17:27:54.973 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.973 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a08e168d:::benchmark_data_smithi171_122772_object144597:head# 2024-04-20T17:27:54.973 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.973 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a08ed38b:::benchmark_data_smithi171_122772_object107063:head# 2024-04-20T17:27:54.974 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.974 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0906c31:::benchmark_data_smithi171_122772_object156289:head# 2024-04-20T17:27:54.974 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.974 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a090bb93:::benchmark_data_smithi171_122772_object90792:head# 2024-04-20T17:27:54.974 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.974 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a09183cc:::benchmark_data_smithi171_122772_object96638:head# 2024-04-20T17:27:54.974 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.974 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0918995:::benchmark_data_smithi171_122772_object101399:head# 2024-04-20T17:27:54.974 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.974 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a091b14f:::benchmark_data_smithi171_122772_object117164:head# 2024-04-20T17:27:54.974 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.974 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a091cb3a:::benchmark_data_smithi171_122772_object168673:head# 2024-04-20T17:27:54.974 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.975 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:27:54.985 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0923b1c:::benchmark_data_smithi171_122772_object131571:head# 2024-04-20T17:27:54.985 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.985 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a092abec:::benchmark_data_smithi171_122772_object137701:head# 2024-04-20T17:27:54.986 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.986 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0934836:::benchmark_data_smithi171_122772_object147486:head# 2024-04-20T17:27:54.986 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.986 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a093bd7b:::benchmark_data_smithi171_122772_object100910:head# 2024-04-20T17:27:54.986 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.986 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0942374:::benchmark_data_smithi171_122772_object137233:head# 2024-04-20T17:27:54.986 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.986 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a094cc20:::benchmark_data_smithi171_122772_object93216:head# 2024-04-20T17:27:54.986 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.986 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0965a2d:::benchmark_data_smithi171_122772_object158677:head# 2024-04-20T17:27:54.986 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.986 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a096ddfd:::benchmark_data_smithi171_122772_object94721:head# 2024-04-20T17:27:54.986 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.987 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a096eb6a:::benchmark_data_smithi171_122772_object155418:head# 2024-04-20T17:27:54.987 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.987 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a096f734:::benchmark_data_smithi171_122772_object140075:head# 2024-04-20T17:27:54.987 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.987 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0971344:::benchmark_data_smithi171_122772_object142855:head# 2024-04-20T17:27:54.987 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.987 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0974b5d:::benchmark_data_smithi171_122772_object108332:head# 2024-04-20T17:27:54.987 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.987 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0990904:::benchmark_data_smithi171_122772_object165982:head# 2024-04-20T17:27:54.987 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.987 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a09962aa:::benchmark_data_smithi171_122772_object87905:head# 2024-04-20T17:27:54.988 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.988 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a09a7fad:::benchmark_data_smithi171_122772_object119656:head# 2024-04-20T17:27:54.988 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.988 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a09a9201:::benchmark_data_smithi171_122772_object123802:head# 2024-04-20T17:27:54.988 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.988 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a09ae8cf:::benchmark_data_smithi171_122772_object96818:head# 2024-04-20T17:27:54.988 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.988 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a09b2da9:::benchmark_data_smithi171_122772_object111906:head# 2024-04-20T17:27:54.988 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.988 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a09b3056:::benchmark_data_smithi171_122772_object133322:head# 2024-04-20T17:27:54.988 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.988 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a09d0cca:::benchmark_data_smithi171_122772_object167579:head# 2024-04-20T17:27:54.988 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.988 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a09d1488:::benchmark_data_smithi171_122772_object154945:head# 2024-04-20T17:27:54.988 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.989 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a09d45f7:::benchmark_data_smithi171_122772_object111949:head# 2024-04-20T17:27:54.989 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.989 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a09daa4a:::benchmark_data_smithi171_122772_object87172:head# 2024-04-20T17:27:54.989 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.989 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a09ee5dd:::benchmark_data_smithi171_122772_object107405:head# 2024-04-20T17:27:54.989 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.989 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a09ee880:::benchmark_data_smithi171_122772_object88771:head# 2024-04-20T17:27:54.989 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.989 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a09f0d67:::benchmark_data_smithi171_122772_object140582:head# 2024-04-20T17:27:54.989 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.989 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a09f518b:::benchmark_data_smithi171_122772_object108803:head# 2024-04-20T17:27:54.989 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.989 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0a1e84e:::benchmark_data_smithi171_122772_object123895:head# 2024-04-20T17:27:54.990 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.990 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0a2e9e7:::benchmark_data_smithi171_122772_object135719:head# 2024-04-20T17:27:54.990 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.990 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0a2f3f0:::benchmark_data_smithi171_122772_object136776:head# 2024-04-20T17:27:54.990 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.990 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0a3f7d0:::benchmark_data_smithi171_122772_object82047:head# 2024-04-20T17:27:54.990 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.990 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0a4734b:::benchmark_data_smithi171_122772_object129601:head# 2024-04-20T17:27:54.990 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.990 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0a4a389:::benchmark_data_smithi171_122772_object167366:head# 2024-04-20T17:27:54.990 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.990 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0a538b6:::benchmark_data_smithi171_122772_object121545:head# 2024-04-20T17:27:54.990 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.990 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0a5eb3e:::benchmark_data_smithi171_122772_object148177:head# 2024-04-20T17:27:54.991 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.991 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0a64161:::benchmark_data_smithi171_122772_object94952:head# 2024-04-20T17:27:54.991 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.991 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0a6529c:::benchmark_data_smithi171_122772_object78595:head# 2024-04-20T17:27:54.991 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.991 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0a7a68b:::benchmark_data_smithi171_122772_object92246:head# 2024-04-20T17:27:54.991 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.991 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0a7c757:::benchmark_data_smithi171_122772_object160821:head# 2024-04-20T17:27:54.991 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.991 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0a84463:::benchmark_data_smithi171_122772_object149513:head# 2024-04-20T17:27:54.991 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.991 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0a88177:::benchmark_data_smithi171_122772_object104695:head# 2024-04-20T17:27:54.991 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.992 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0a9e36b:::benchmark_data_smithi171_122772_object86233:head# 2024-04-20T17:27:54.992 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.992 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0ac0375:::benchmark_data_smithi171_122772_object114209:head# 2024-04-20T17:27:54.992 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.992 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0ac1766:::benchmark_data_smithi171_122772_object80237:head# 2024-04-20T17:27:54.992 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.992 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0ac3143:::benchmark_data_smithi171_122772_object100354:head# 2024-04-20T17:27:54.992 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.992 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0acbf9d:::benchmark_data_smithi171_122772_object81178:head# 2024-04-20T17:27:54.992 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.992 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0ad40f3:::benchmark_data_smithi171_122772_object137515:head# 2024-04-20T17:27:54.992 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.992 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0ae6e16:::benchmark_data_smithi171_122772_object134844:head# 2024-04-20T17:27:54.992 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.993 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0af4066:::benchmark_data_smithi171_122772_object162516:head# 2024-04-20T17:27:54.993 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.993 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0af485f:::benchmark_data_smithi171_122772_object165155:head# 2024-04-20T17:27:54.993 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.993 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0b042a2:::benchmark_data_smithi171_122772_object137892:head# 2024-04-20T17:27:54.993 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.993 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0b0b508:::benchmark_data_smithi171_122772_object131548:head# 2024-04-20T17:27:54.993 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.993 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0b235ff:::benchmark_data_smithi171_122772_object120618:head# 2024-04-20T17:27:54.993 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.993 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0b24d60:::benchmark_data_smithi171_122772_object161088:head# 2024-04-20T17:27:54.993 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.993 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0b34a07:::benchmark_data_smithi171_122772_object142429:head# 2024-04-20T17:27:54.993 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.994 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0b41dc9:::benchmark_data_smithi171_122772_object139825:head# 2024-04-20T17:27:54.994 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.994 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0b4fc00:::benchmark_data_smithi171_122772_object126948:head# 2024-04-20T17:27:54.994 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.994 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0b7e62b:::benchmark_data_smithi171_122772_object106404:head# 2024-04-20T17:27:54.994 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.994 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0b89c03:::benchmark_data_smithi171_122772_object91684:head# 2024-04-20T17:27:54.994 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.994 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0baed10:::benchmark_data_smithi171_122772_object166292:head# 2024-04-20T17:27:54.994 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.994 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0bb6298:::benchmark_data_smithi171_122772_object93799:head# 2024-04-20T17:27:54.994 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.994 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0bb7379:::benchmark_data_smithi171_122772_object149311:head# 2024-04-20T17:27:54.994 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.994 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0bc86b3:::benchmark_data_smithi171_122772_object126072:head# 2024-04-20T17:27:54.995 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:54.995 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0bd092e:::benchmark_data_smithi171_122772_object155800:head# 2024-04-20T17:27:54.995 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.000 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0bd48aa:::benchmark_data_smithi171_122772_object88031:head# 2024-04-20T17:27:55.000 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.000 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0be77eb:::benchmark_data_smithi171_122772_object129373:head# 2024-04-20T17:27:55.000 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.000 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0be7e2a:::benchmark_data_smithi171_122772_object144858:head# 2024-04-20T17:27:55.000 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.000 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0bed093:::benchmark_data_smithi171_122772_object111064:head# 2024-04-20T17:27:55.000 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.001 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0befab1:::benchmark_data_smithi171_122772_object145886:head# 2024-04-20T17:27:55.001 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.001 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0bfca58:::benchmark_data_smithi171_122772_object135638:head# 2024-04-20T17:27:55.001 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.001 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0bfd736:::benchmark_data_smithi171_122772_object163702:head# 2024-04-20T17:27:55.001 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.001 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0c19d4a:::benchmark_data_smithi171_122772_object141898:head# 2024-04-20T17:27:55.001 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.001 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0c1afb9:::benchmark_data_smithi171_122772_object119499:head# 2024-04-20T17:27:55.001 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.001 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0c33021:::benchmark_data_smithi171_122772_object152093:head# 2024-04-20T17:27:55.001 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.001 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0c42557:::benchmark_data_smithi171_122772_object88480:head# 2024-04-20T17:27:55.001 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.002 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0c4ef1b:::benchmark_data_smithi171_122772_object102867:head# 2024-04-20T17:27:55.002 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.002 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0c514b0:::benchmark_data_smithi171_122772_object79059:head# 2024-04-20T17:27:55.002 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.002 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0c5395e:::benchmark_data_smithi171_122772_object163576:head# 2024-04-20T17:27:55.002 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.002 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0c55e97:::benchmark_data_smithi171_122772_object102839:head# 2024-04-20T17:27:55.002 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.002 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0c57e1f:::benchmark_data_smithi171_122772_object80850:head# 2024-04-20T17:27:55.002 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.002 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0c5b0b8:::benchmark_data_smithi171_122772_object160321:head# 2024-04-20T17:27:55.002 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.002 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0c6894e:::benchmark_data_smithi171_122772_object87989:head# 2024-04-20T17:27:55.002 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.003 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0c690e4:::benchmark_data_smithi171_122772_object113553:head# 2024-04-20T17:27:55.003 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.003 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0c7fe58:::benchmark_data_smithi171_122772_object132124:head# 2024-04-20T17:27:55.003 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.003 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0c9b787:::benchmark_data_smithi171_122772_object95068:head# 2024-04-20T17:27:55.003 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.003 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0cb0ff8:::benchmark_data_smithi171_122772_object94399:head# 2024-04-20T17:27:55.003 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.003 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0cb61fa:::benchmark_data_smithi171_122772_object110693:head# 2024-04-20T17:27:55.003 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.003 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0cbf7de:::benchmark_data_smithi171_122772_object155402:head# 2024-04-20T17:27:55.003 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.004 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0cccd6c:::benchmark_data_smithi171_122772_object154239:head# 2024-04-20T17:27:55.004 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.004 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0cd231c:::benchmark_data_smithi171_122772_object95344:head# 2024-04-20T17:27:55.004 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.004 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0cd9bfb:::benchmark_data_smithi171_122772_object160710:head# 2024-04-20T17:27:55.004 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.004 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0cdcd6a:::benchmark_data_smithi171_122772_object138623:head# 2024-04-20T17:27:55.004 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.004 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0cdfe3f:::benchmark_data_smithi171_122772_object118873:head# 2024-04-20T17:27:55.004 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.004 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0cf8d26:::benchmark_data_smithi171_122772_object144887:head# 2024-04-20T17:27:55.004 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.004 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0d0d405:::benchmark_data_smithi171_122772_object98030:head# 2024-04-20T17:27:55.004 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.005 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0d2ab5d:::benchmark_data_smithi171_122772_object113071:head# 2024-04-20T17:27:55.005 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.005 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0d2afec:::benchmark_data_smithi171_122772_object106663:head# 2024-04-20T17:27:55.005 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.005 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0d37323:::benchmark_data_smithi171_122772_object151791:head# 2024-04-20T17:27:55.005 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.005 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0d47f2c:::benchmark_data_smithi171_122772_object93438:head# 2024-04-20T17:27:55.005 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.005 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0d486a4:::benchmark_data_smithi171_122772_object113584:head# 2024-04-20T17:27:55.005 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.005 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0d54b09:::benchmark_data_smithi171_122772_object119987:head# 2024-04-20T17:27:55.005 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.005 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0d5bde2:::benchmark_data_smithi171_122772_object163202:head# 2024-04-20T17:27:55.005 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.005 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0d75b7f:::benchmark_data_smithi171_122772_object79302:head# 2024-04-20T17:27:55.006 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.006 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0d8bf22:::benchmark_data_smithi171_122772_object97053:head# 2024-04-20T17:27:55.006 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.006 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0d8c24a:::benchmark_data_smithi171_122772_object106689:head# 2024-04-20T17:27:55.006 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.006 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0d8ca03:::benchmark_data_smithi171_122772_object114987:head# 2024-04-20T17:27:55.006 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.006 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0d958fa:::benchmark_data_smithi171_122772_object91190:head# 2024-04-20T17:27:55.006 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.006 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0da3afa:::benchmark_data_smithi171_122772_object86919:head# 2024-04-20T17:27:55.006 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.006 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0daea6a:::benchmark_data_smithi171_122772_object85643:head# 2024-04-20T17:27:55.007 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.007 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0dc219c:::benchmark_data_smithi171_122772_object152660:head# 2024-04-20T17:27:55.007 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.007 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0dca567:::benchmark_data_smithi171_122772_object128370:head# 2024-04-20T17:27:55.007 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.007 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0dcf576:::benchmark_data_smithi171_122772_object121746:head# 2024-04-20T17:27:55.007 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.007 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0ddb2a5:::benchmark_data_smithi171_122772_object145116:head# 2024-04-20T17:27:55.007 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.007 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0dfcd3f:::benchmark_data_smithi171_122772_object123952:head# 2024-04-20T17:27:55.007 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.007 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0e0089f:::benchmark_data_smithi171_122772_object114885:head# 2024-04-20T17:27:55.007 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.008 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0e0b76f:::benchmark_data_smithi171_122772_object91680:head# 2024-04-20T17:27:55.008 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.008 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0e1c346:::benchmark_data_smithi171_122772_object80121:head# 2024-04-20T17:27:55.008 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.008 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0e25884:::benchmark_data_smithi171_122772_object120619:head# 2024-04-20T17:27:55.008 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.017 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0e263a7:::benchmark_data_smithi171_122772_object131683:head# 2024-04-20T17:27:55.017 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.017 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0e28073:::benchmark_data_smithi171_122772_object129590:head# 2024-04-20T17:27:55.017 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.017 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0e2e43e:::benchmark_data_smithi171_122772_object147523:head# 2024-04-20T17:27:55.017 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.017 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0e3ae0e:::benchmark_data_smithi171_122772_object149681:head# 2024-04-20T17:27:55.017 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.017 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0e42c3d:::benchmark_data_smithi171_122772_object125321:head# 2024-04-20T17:27:55.017 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.017 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0e6565b:::benchmark_data_smithi171_122772_object161430:head# 2024-04-20T17:27:55.018 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.018 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0e7612c:::benchmark_data_smithi171_122772_object156787:head# 2024-04-20T17:27:55.018 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.018 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0e87180:::benchmark_data_smithi171_122772_object133603:head# 2024-04-20T17:27:55.018 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.018 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0e9b9df:::benchmark_data_smithi171_122772_object125962:head# 2024-04-20T17:27:55.018 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.018 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0ea33e6:::benchmark_data_smithi171_122772_object128851:head# 2024-04-20T17:27:55.018 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.018 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0eeb31b:::benchmark_data_smithi171_122772_object127626:head# 2024-04-20T17:27:55.018 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.018 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0ef432c:::benchmark_data_smithi171_122772_object134728:head# 2024-04-20T17:27:55.018 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.019 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0ef5a5c:::benchmark_data_smithi171_122772_object109447:head# 2024-04-20T17:27:55.019 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.019 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0ef6e50:::benchmark_data_smithi171_122772_object115312:head# 2024-04-20T17:27:55.019 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.019 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0f00dc7:::benchmark_data_smithi171_122772_object97502:head# 2024-04-20T17:27:55.019 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.019 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0f22dcf:::benchmark_data_smithi171_122772_object111670:head# 2024-04-20T17:27:55.019 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.019 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0f2b4af:::benchmark_data_smithi171_122772_object162323:head# 2024-04-20T17:27:55.019 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.019 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0f34a06:::benchmark_data_smithi171_122772_object159511:head# 2024-04-20T17:27:55.019 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.019 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0f389e0:::benchmark_data_smithi171_122772_object141967:head# 2024-04-20T17:27:55.019 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.020 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0f45235:::benchmark_data_smithi171_122772_object155439:head# 2024-04-20T17:27:55.020 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.020 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0f45ae0:::benchmark_data_smithi171_122772_object78924:head# 2024-04-20T17:27:55.020 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.020 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0f59b5d:::benchmark_data_smithi171_122772_object81637:head# 2024-04-20T17:27:55.020 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.020 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0f63112:::benchmark_data_smithi171_122772_object152494:head# 2024-04-20T17:27:55.020 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.020 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0f65c9e:::benchmark_data_smithi171_122772_object156912:head# 2024-04-20T17:27:55.020 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.020 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0f685e6:::benchmark_data_smithi171_122772_object106536:head# 2024-04-20T17:27:55.020 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.020 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0f707ed:::benchmark_data_smithi171_122772_object105005:head# 2024-04-20T17:27:55.021 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.021 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0f786c5:::benchmark_data_smithi171_122772_object101046:head# 2024-04-20T17:27:55.021 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.021 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0f8fc3c:::benchmark_data_smithi171_122772_object169441:head# 2024-04-20T17:27:55.021 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.021 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0f94a7d:::benchmark_data_smithi171_122772_object126394:head# 2024-04-20T17:27:55.021 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.021 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0fa255b:::benchmark_data_smithi171_122772_object115217:head# 2024-04-20T17:27:55.021 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.021 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0fa74bb:::benchmark_data_smithi171_122772_object140225:head# 2024-04-20T17:27:55.021 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.021 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0fb828b:::benchmark_data_smithi171_122772_object94305:head# 2024-04-20T17:27:55.021 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.021 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0fbcee3:::benchmark_data_smithi171_122772_object152102:head# 2024-04-20T17:27:55.022 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.022 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0fc3254:::benchmark_data_smithi171_122772_object118846:head# 2024-04-20T17:27:55.022 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.022 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0fc5a76:::benchmark_data_smithi171_122772_object147501:head# 2024-04-20T17:27:55.022 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.022 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0fcf3cd:::benchmark_data_smithi171_122772_object116998:head# 2024-04-20T17:27:55.022 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.022 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0fd0de7:::benchmark_data_smithi171_122772_object127552:head# 2024-04-20T17:27:55.022 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.022 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0fd3092:::benchmark_data_smithi171_122772_object120011:head# 2024-04-20T17:27:55.022 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.022 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0fe17be:::benchmark_data_smithi171_122772_object167939:head# 2024-04-20T17:27:55.022 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.023 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0fe2294:::benchmark_data_smithi171_122772_object120289:head# 2024-04-20T17:27:55.023 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.023 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0fe2fd8:::benchmark_data_smithi171_122772_object116004:head# 2024-04-20T17:27:55.023 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.023 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0fec231:::benchmark_data_smithi171_122772_object118895:head# 2024-04-20T17:27:55.023 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.023 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0fee486:::benchmark_data_smithi171_122772_object80830:head# 2024-04-20T17:27:55.023 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.023 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a0ffe363:::benchmark_data_smithi171_122772_object123643:head# 2024-04-20T17:27:55.023 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.023 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1008162:::benchmark_data_smithi171_122772_object161690:head# 2024-04-20T17:27:55.023 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.023 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a101cb71:::benchmark_data_smithi171_122772_object107278:head# 2024-04-20T17:27:55.023 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.024 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a102e7ec:::benchmark_data_smithi171_122772_object136264:head# 2024-04-20T17:27:55.024 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.024 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1040601:::benchmark_data_smithi171_122772_object153991:head# 2024-04-20T17:27:55.024 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.024 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a104c804:::benchmark_data_smithi171_122772_object94371:head# 2024-04-20T17:27:55.024 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.024 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1050422:::benchmark_data_smithi171_122772_object109403:head# 2024-04-20T17:27:55.024 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.024 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a10569ff:::benchmark_data_smithi171_122772_object168132:head# 2024-04-20T17:27:55.024 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.024 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1059769:::benchmark_data_smithi171_122772_object133602:head# 2024-04-20T17:27:55.024 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.024 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a105c845:::benchmark_data_smithi171_122772_object94072:head# 2024-04-20T17:27:55.024 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.025 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1094899:::benchmark_data_smithi171_122772_object165294:head# 2024-04-20T17:27:55.025 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.025 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a10a8ed1:::benchmark_data_smithi171_122772_object125498:head# 2024-04-20T17:27:55.025 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.025 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a10ab7cb:::benchmark_data_smithi171_122772_object143824:head# 2024-04-20T17:27:55.025 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.025 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a10b2928:::benchmark_data_smithi171_122772_object135129:head# 2024-04-20T17:27:55.025 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.025 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a10b7fc1:::benchmark_data_smithi171_122772_object96982:head# 2024-04-20T17:27:55.025 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.025 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a10c1d33:::benchmark_data_smithi171_122772_object138910:head# 2024-04-20T17:27:55.025 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.025 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a10cef6e:::benchmark_data_smithi171_122772_object130622:head# 2024-04-20T17:27:55.025 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.026 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a10d0af7:::benchmark_data_smithi171_122772_object143389:head# 2024-04-20T17:27:55.026 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.026 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a10e26f6:::benchmark_data_smithi171_122772_object83156:head# 2024-04-20T17:27:55.026 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.026 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a10e70f7:::benchmark_data_smithi171_122772_object153518:head# 2024-04-20T17:27:55.026 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.027 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:55.038 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a10eaac0:::benchmark_data_smithi171_122772_object148951:head# 2024-04-20T17:27:55.038 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.038 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1107390:::benchmark_data_smithi171_122772_object151557:head# 2024-04-20T17:27:55.038 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.038 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1114ca0:::benchmark_data_smithi171_122772_object135612:head# 2024-04-20T17:27:55.038 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.038 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1118bcd:::benchmark_data_smithi171_122772_object143221:head# 2024-04-20T17:27:55.039 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.039 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a113b488:::benchmark_data_smithi171_122772_object132616:head# 2024-04-20T17:27:55.039 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.039 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a114a03d:::benchmark_data_smithi171_122772_object162183:head# 2024-04-20T17:27:55.039 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.039 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a116beaf:::benchmark_data_smithi171_122772_object103201:head# 2024-04-20T17:27:55.039 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.039 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a118e08d:::benchmark_data_smithi171_122772_object85705:head# 2024-04-20T17:27:55.039 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.039 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a11a3975:::benchmark_data_smithi171_122772_object155073:head# 2024-04-20T17:27:55.039 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.039 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a11b2507:::benchmark_data_smithi171_122772_object82379:head# 2024-04-20T17:27:55.039 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.040 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a11c5ab6:::benchmark_data_smithi171_122772_object141555:head# 2024-04-20T17:27:55.040 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.040 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a11f89f1:::benchmark_data_smithi171_122772_object136895:head# 2024-04-20T17:27:55.040 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.040 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1221928:::benchmark_data_smithi171_122772_object165317:head# 2024-04-20T17:27:55.040 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.040 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a122dcc1:::benchmark_data_smithi171_122772_object123925:head# 2024-04-20T17:27:55.040 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.040 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1238e2b:::benchmark_data_smithi171_122772_object143177:head# 2024-04-20T17:27:55.040 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.040 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1253456:::benchmark_data_smithi171_122772_object103802:head# 2024-04-20T17:27:55.040 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.040 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a125a3b6:::benchmark_data_smithi171_122772_object114497:head# 2024-04-20T17:27:55.040 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.040 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a126b9f9:::benchmark_data_smithi171_122772_object145843:head# 2024-04-20T17:27:55.041 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.041 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a126e87c:::benchmark_data_smithi171_122772_object116806:head# 2024-04-20T17:27:55.041 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.041 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1273844:::benchmark_data_smithi171_122772_object120527:head# 2024-04-20T17:27:55.041 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.041 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1282159:::benchmark_data_smithi171_122772_object153137:head# 2024-04-20T17:27:55.041 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.041 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a129b005:::benchmark_data_smithi171_122772_object130809:head# 2024-04-20T17:27:55.041 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.041 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a129ecd3:::benchmark_data_smithi171_122772_object130348:head# 2024-04-20T17:27:55.041 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.041 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a12a50a7:::benchmark_data_smithi171_122772_object109924:head# 2024-04-20T17:27:55.041 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.041 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a12af57f:::benchmark_data_smithi171_122772_object115216:head# 2024-04-20T17:27:55.042 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.042 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a12b5fdd:::benchmark_data_smithi171_122772_object168656:head# 2024-04-20T17:27:55.042 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.042 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a12b63aa:::benchmark_data_smithi171_122772_object84128:head# 2024-04-20T17:27:55.042 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.042 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a12da919:::benchmark_data_smithi171_122772_object87379:head# 2024-04-20T17:27:55.042 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.042 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a12ee19e:::benchmark_data_smithi171_122772_object102107:head# 2024-04-20T17:27:55.042 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.042 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a12f3fbf:::benchmark_data_smithi171_122772_object92967:head# 2024-04-20T17:27:55.042 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.042 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a130178f:::benchmark_data_smithi171_122772_object148729:head# 2024-04-20T17:27:55.042 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.042 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a130b0cf:::benchmark_data_smithi171_122772_object166351:head# 2024-04-20T17:27:55.043 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.043 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a131a723:::benchmark_data_smithi171_122772_object129444:head# 2024-04-20T17:27:55.043 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.043 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1321d1a:::benchmark_data_smithi171_122772_object152448:head# 2024-04-20T17:27:55.043 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.043 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1334fca:::benchmark_data_smithi171_122772_object118054:head# 2024-04-20T17:27:55.043 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.043 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1351589:::benchmark_data_smithi171_122772_object158323:head# 2024-04-20T17:27:55.043 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.043 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a135a30f:::benchmark_data_smithi171_122772_object157837:head# 2024-04-20T17:27:55.043 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.043 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1361584:::benchmark_data_smithi171_122772_object148801:head# 2024-04-20T17:27:55.043 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.043 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1361dfc:::benchmark_data_smithi171_122772_object88301:head# 2024-04-20T17:27:55.044 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.044 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1367475:::benchmark_data_smithi171_122772_object165676:head# 2024-04-20T17:27:55.044 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.044 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a136d915:::benchmark_data_smithi171_122772_object144403:head# 2024-04-20T17:27:55.044 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.044 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a137b4a2:::benchmark_data_smithi171_122772_object148446:head# 2024-04-20T17:27:55.044 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.044 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1382961:::benchmark_data_smithi171_122772_object81663:head# 2024-04-20T17:27:55.044 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.044 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a13937ca:::benchmark_data_smithi171_122772_object143723:head# 2024-04-20T17:27:55.044 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.044 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a139c9c5:::benchmark_data_smithi171_122772_object128047:head# 2024-04-20T17:27:55.044 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.044 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a13ad1fc:::benchmark_data_smithi171_122772_object78987:head# 2024-04-20T17:27:55.044 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.045 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a13c498d:::benchmark_data_smithi171_122772_object91510:head# 2024-04-20T17:27:55.045 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.045 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a13c8fc4:::benchmark_data_smithi171_122772_object101937:head# 2024-04-20T17:27:55.045 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.045 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a13cb293:::benchmark_data_smithi171_122772_object134757:head# 2024-04-20T17:27:55.045 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.045 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a13d1d57:::benchmark_data_smithi171_122772_object142432:head# 2024-04-20T17:27:55.045 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.045 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a13e214c:::benchmark_data_smithi171_122772_object87892:head# 2024-04-20T17:27:55.045 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.045 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a13fb824:::benchmark_data_smithi171_122772_object103442:head# 2024-04-20T17:27:55.045 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.045 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a13fd776:::benchmark_data_smithi171_122772_object92290:head# 2024-04-20T17:27:55.045 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.045 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1403d8d:::benchmark_data_smithi171_122772_object138491:head# 2024-04-20T17:27:55.046 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.046 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a140ef6a:::benchmark_data_smithi171_122772_object163377:head# 2024-04-20T17:27:55.046 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.046 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1411c7b:::benchmark_data_smithi171_122772_object107223:head# 2024-04-20T17:27:55.046 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.046 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a141e5bb:::benchmark_data_smithi171_122772_object138730:head# 2024-04-20T17:27:55.046 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.046 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1429e2a:::benchmark_data_smithi171_122772_object117551:head# 2024-04-20T17:27:55.046 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.046 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a142ef42:::benchmark_data_smithi171_122772_object109931:head# 2024-04-20T17:27:55.046 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.046 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1434c11:::benchmark_data_smithi171_122772_object134338:head# 2024-04-20T17:27:55.046 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.046 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a143c08d:::benchmark_data_smithi171_122772_object87823:head# 2024-04-20T17:27:55.047 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.047 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a145fdcd:::benchmark_data_smithi171_122772_object142395:head# 2024-04-20T17:27:55.047 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.047 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1462f48:::benchmark_data_smithi171_122772_object124008:head# 2024-04-20T17:27:55.047 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.047 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a146391c:::benchmark_data_smithi171_122772_object123641:head# 2024-04-20T17:27:55.047 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.047 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1465fc6:::benchmark_data_smithi171_122772_object156975:head# 2024-04-20T17:27:55.047 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.047 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a147bcfa:::benchmark_data_smithi171_122772_object90303:head# 2024-04-20T17:27:55.047 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.047 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a14856e4:::benchmark_data_smithi171_122772_object131181:head# 2024-04-20T17:27:55.047 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.047 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a14991b1:::benchmark_data_smithi171_122772_object117761:head# 2024-04-20T17:27:55.048 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.048 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a149cd34:::benchmark_data_smithi171_122772_object142391:head# 2024-04-20T17:27:55.048 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.048 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a14a9079:::benchmark_data_smithi171_122772_object146587:head# 2024-04-20T17:27:55.048 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.048 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a14bb275:::benchmark_data_smithi171_122772_object125918:head# 2024-04-20T17:27:55.048 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.048 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a14bfd68:::benchmark_data_smithi171_122772_object125892:head# 2024-04-20T17:27:55.048 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.048 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a14d01f3:::benchmark_data_smithi171_122772_object78214:head# 2024-04-20T17:27:55.048 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.048 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a14f939e:::benchmark_data_smithi171_122772_object141219:head# 2024-04-20T17:27:55.048 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.048 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a151373d:::benchmark_data_smithi171_122772_object141271:head# 2024-04-20T17:27:55.048 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.049 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1521972:::benchmark_data_smithi171_122772_object117474:head# 2024-04-20T17:27:55.049 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.049 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a15328e7:::benchmark_data_smithi171_122772_object157796:head# 2024-04-20T17:27:55.049 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.049 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a153e033:::benchmark_data_smithi171_122772_object132280:head# 2024-04-20T17:27:55.049 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.049 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a153f7cd:::benchmark_data_smithi171_122772_object137856:head# 2024-04-20T17:27:55.049 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.049 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a154b185:::benchmark_data_smithi171_122772_object168978:head# 2024-04-20T17:27:55.049 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.049 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a154c2e6:::benchmark_data_smithi171_122772_object120043:head# 2024-04-20T17:27:55.049 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.049 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1551e11:::benchmark_data_smithi171_122772_object125245:head# 2024-04-20T17:27:55.049 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.049 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1553b86:::benchmark_data_smithi171_122772_object86906:head# 2024-04-20T17:27:55.050 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.050 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a156d5a5:::benchmark_data_smithi171_122772_object121999:head# 2024-04-20T17:27:55.050 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:55.059 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:27:55.060 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:27:55.066 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1580990:::benchmark_data_smithi171_122772_object92221:head# 2024-04-20T17:27:55.066 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.066 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a159ec24:::benchmark_data_smithi171_122772_object99428:head# 2024-04-20T17:27:55.066 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.067 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a15b1022:::benchmark_data_smithi171_122772_object82188:head# 2024-04-20T17:27:55.067 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.067 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a15c5688:::benchmark_data_smithi171_122772_object84243:head# 2024-04-20T17:27:55.067 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.067 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a15cd326:::benchmark_data_smithi171_122772_object104417:head# 2024-04-20T17:27:55.067 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.067 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a15cd39d:::benchmark_data_smithi171_122772_object138988:head# 2024-04-20T17:27:55.067 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.067 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a15d481f:::benchmark_data_smithi171_122772_object116771:head# 2024-04-20T17:27:55.067 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.067 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a15d4fa3:::benchmark_data_smithi171_122772_object80514:head# 2024-04-20T17:27:55.067 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.067 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a15d5e39:::benchmark_data_smithi171_122772_object107870:head# 2024-04-20T17:27:55.067 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.068 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a15e137d:::benchmark_data_smithi171_122772_object158486:head# 2024-04-20T17:27:55.068 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.068 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a15ea9c5:::benchmark_data_smithi171_122772_object113319:head# 2024-04-20T17:27:55.068 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.068 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a15f5aa9:::benchmark_data_smithi171_122772_object162912:head# 2024-04-20T17:27:55.068 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.068 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a16091e7:::benchmark_data_smithi171_122772_object107968:head# 2024-04-20T17:27:55.068 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.068 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a160a06d:::benchmark_data_smithi171_122772_object115833:head# 2024-04-20T17:27:55.068 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.068 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1614b16:::benchmark_data_smithi171_122772_object145289:head# 2024-04-20T17:27:55.068 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.068 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a161cba0:::benchmark_data_smithi171_122772_object152028:head# 2024-04-20T17:27:55.068 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.068 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1625674:::benchmark_data_smithi171_122772_object152603:head# 2024-04-20T17:27:55.069 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.069 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1637534:::benchmark_data_smithi171_122772_object82496:head# 2024-04-20T17:27:55.069 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.069 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1645785:::benchmark_data_smithi171_122772_object101395:head# 2024-04-20T17:27:55.069 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.069 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a164bc05:::benchmark_data_smithi171_122772_object104820:head# 2024-04-20T17:27:55.069 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.069 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1695da9:::benchmark_data_smithi171_122772_object147670:head# 2024-04-20T17:27:55.069 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.069 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1696626:::benchmark_data_smithi171_122772_object137800:head# 2024-04-20T17:27:55.069 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.069 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1698833:::benchmark_data_smithi171_122772_object124256:head# 2024-04-20T17:27:55.069 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.069 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a16bb09c:::benchmark_data_smithi171_122772_object109898:head# 2024-04-20T17:27:55.069 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.070 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a16bfbe9:::benchmark_data_smithi171_122772_object97746:head# 2024-04-20T17:27:55.070 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.070 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a16f1876:::benchmark_data_smithi171_122772_object152882:head# 2024-04-20T17:27:55.070 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.070 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a16f1fb9:::benchmark_data_smithi171_122772_object167051:head# 2024-04-20T17:27:55.070 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.070 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a16f431b:::benchmark_data_smithi171_122772_object128942:head# 2024-04-20T17:27:55.070 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.070 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a16fd83e:::benchmark_data_smithi171_122772_object78636:head# 2024-04-20T17:27:55.070 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.070 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a170cf8d:::benchmark_data_smithi171_122772_object87334:head# 2024-04-20T17:27:55.070 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.070 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1717456:::benchmark_data_smithi171_122772_object160309:head# 2024-04-20T17:27:55.070 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a17291ea:::benchmark_data_smithi171_122772_object86023:head# 2024-04-20T17:27:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1736a3b:::benchmark_data_smithi171_122772_object153861:head# 2024-04-20T17:27:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a173d526:::benchmark_data_smithi171_122772_object145560:head# 2024-04-20T17:27:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1740218:::benchmark_data_smithi171_122772_object90579:head# 2024-04-20T17:27:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a174431e:::benchmark_data_smithi171_122772_object136530:head# 2024-04-20T17:27:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a174b001:::benchmark_data_smithi171_122772_object97362:head# 2024-04-20T17:27:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a17698c9:::benchmark_data_smithi171_122772_object115889:head# 2024-04-20T17:27:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a176ce45:::benchmark_data_smithi171_122772_object97092:head# 2024-04-20T17:27:55.071 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a176f590:::benchmark_data_smithi171_122772_object132385:head# 2024-04-20T17:27:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a17702f6:::benchmark_data_smithi171_122772_object154041:head# 2024-04-20T17:27:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a17873f7:::benchmark_data_smithi171_122772_object130946:head# 2024-04-20T17:27:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a178772c:::benchmark_data_smithi171_122772_object168180:head# 2024-04-20T17:27:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a17a26eb:::benchmark_data_smithi171_122772_object158925:head# 2024-04-20T17:27:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a17aa0cf:::benchmark_data_smithi171_122772_object116662:head# 2024-04-20T17:27:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a17b76ab:::benchmark_data_smithi171_122772_object88208:head# 2024-04-20T17:27:55.072 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a17bae6f:::benchmark_data_smithi171_122772_object142004:head# 2024-04-20T17:27:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a17c9068:::benchmark_data_smithi171_122772_object90390:head# 2024-04-20T17:27:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a17cda0b:::benchmark_data_smithi171_122772_object110809:head# 2024-04-20T17:27:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a17d758c:::benchmark_data_smithi171_122772_object80876:head# 2024-04-20T17:27:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a17d889a:::benchmark_data_smithi171_122772_object150323:head# 2024-04-20T17:27:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a17de5f3:::benchmark_data_smithi171_122772_object88521:head# 2024-04-20T17:27:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a17e0cd3:::benchmark_data_smithi171_122772_object103669:head# 2024-04-20T17:27:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.074 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a17e6252:::benchmark_data_smithi171_122772_object98155:head# 2024-04-20T17:27:55.074 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.074 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a17e9797:::benchmark_data_smithi171_122772_object97036:head# 2024-04-20T17:27:55.074 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.074 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a17f4fde:::benchmark_data_smithi171_122772_object93542:head# 2024-04-20T17:27:55.074 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.074 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a17fc4b6:::benchmark_data_smithi171_122772_object92759:head# 2024-04-20T17:27:55.074 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.074 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a17fe657:::benchmark_data_smithi171_122772_object167628:head# 2024-04-20T17:27:55.074 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.074 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1806277:::benchmark_data_smithi171_122772_object162740:head# 2024-04-20T17:27:55.074 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.074 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1807666:::benchmark_data_smithi171_122772_object91080:head# 2024-04-20T17:27:55.075 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.075 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1808fc6:::benchmark_data_smithi171_122772_object167122:head# 2024-04-20T17:27:55.075 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.075 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a181b276:::benchmark_data_smithi171_122772_object113097:head# 2024-04-20T17:27:55.075 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.075 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1839640:::benchmark_data_smithi171_122772_object127731:head# 2024-04-20T17:27:55.075 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.075 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a183ac1b:::benchmark_data_smithi171_122772_object143961:head# 2024-04-20T17:27:55.075 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.075 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a183c946:::benchmark_data_smithi171_122772_object140486:head# 2024-04-20T17:27:55.075 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.075 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a184b9d0:::benchmark_data_smithi171_122772_object117738:head# 2024-04-20T17:27:55.075 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.076 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a18556b0:::benchmark_data_smithi171_122772_object125217:head# 2024-04-20T17:27:55.076 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.076 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1888282:::benchmark_data_smithi171_122772_object102158:head# 2024-04-20T17:27:55.076 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.076 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a18889e4:::benchmark_data_smithi171_122772_object159447:head# 2024-04-20T17:27:55.076 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.076 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a188f140:::benchmark_data_smithi171_122772_object153124:head# 2024-04-20T17:27:55.076 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.076 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a18998e9:::benchmark_data_smithi171_122772_object156768:head# 2024-04-20T17:27:55.076 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.076 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a189c080:::benchmark_data_smithi171_122772_object138017:head# 2024-04-20T17:27:55.076 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.076 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a18a2f48:::benchmark_data_smithi171_122772_object153522:head# 2024-04-20T17:27:55.076 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.077 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a18b46cf:::benchmark_data_smithi171_122772_object129512:head# 2024-04-20T17:27:55.077 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.077 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a18b9e55:::benchmark_data_smithi171_122772_object155277:head# 2024-04-20T17:27:55.077 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.077 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a18c4bee:::benchmark_data_smithi171_122772_object78310:head# 2024-04-20T17:27:55.077 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.077 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a18c9563:::benchmark_data_smithi171_122772_object110149:head# 2024-04-20T17:27:55.077 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.077 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a18ca7cf:::benchmark_data_smithi171_122772_object155980:head# 2024-04-20T17:27:55.077 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.077 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a18cccf3:::benchmark_data_smithi171_122772_object137251:head# 2024-04-20T17:27:55.077 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.077 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a18d4e77:::benchmark_data_smithi171_122772_object101157:head# 2024-04-20T17:27:55.077 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.078 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a18de7de:::benchmark_data_smithi171_122772_object98400:head# 2024-04-20T17:27:55.078 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.078 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a18e60f2:::benchmark_data_smithi171_122772_object155771:head# 2024-04-20T17:27:55.078 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.078 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a18fc426:::benchmark_data_smithi171_122772_object124667:head# 2024-04-20T17:27:55.078 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.078 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a191a5b4:::benchmark_data_smithi171_122772_object97733:head# 2024-04-20T17:27:55.078 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.078 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a191b4a9:::benchmark_data_smithi171_122772_object98475:head# 2024-04-20T17:27:55.078 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.078 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a191c52c:::benchmark_data_smithi171_122772_object94193:head# 2024-04-20T17:27:55.078 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.078 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1940ebd:::benchmark_data_smithi171_122772_object138579:head# 2024-04-20T17:27:55.079 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.079 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a194f3e4:::benchmark_data_smithi171_122772_object163091:head# 2024-04-20T17:27:55.079 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.079 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a195f97e:::benchmark_data_smithi171_122772_object157066:head# 2024-04-20T17:27:55.079 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.079 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a196e596:::benchmark_data_smithi171_122772_object151976:head# 2024-04-20T17:27:55.079 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.079 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a197e52e:::benchmark_data_smithi171_122772_object78713:head# 2024-04-20T17:27:55.079 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.079 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a197fc6e:::benchmark_data_smithi171_122772_object157820:head# 2024-04-20T17:27:55.079 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.079 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a19885f3:::benchmark_data_smithi171_122772_object156941:head# 2024-04-20T17:27:55.079 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.079 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a19902d9:::benchmark_data_smithi171_122772_object150214:head# 2024-04-20T17:27:55.079 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.080 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a19b1c15:::benchmark_data_smithi171_122772_object163301:head# 2024-04-20T17:27:55.080 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.080 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a19bd27d:::benchmark_data_smithi171_122772_object165681:head# 2024-04-20T17:27:55.080 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.080 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a19c398a:::benchmark_data_smithi171_122772_object87569:head# 2024-04-20T17:27:55.080 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.080 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a19d9f6d:::benchmark_data_smithi171_122772_object93596:head# 2024-04-20T17:27:55.080 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.080 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a19e6d17:::benchmark_data_smithi171_122772_object103315:head# 2024-04-20T17:27:55.080 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.080 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a19e8687:::benchmark_data_smithi171_122772_object144115:head# 2024-04-20T17:27:55.080 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.080 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a19e87f5:::benchmark_data_smithi171_122772_object142313:head# 2024-04-20T17:27:55.080 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.080 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1a13f18:::benchmark_data_smithi171_122772_object125890:head# 2024-04-20T17:27:55.081 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.081 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1a379a1:::benchmark_data_smithi171_122772_object138749:head# 2024-04-20T17:27:55.081 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.081 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1a3d71b:::benchmark_data_smithi171_122772_object152678:head# 2024-04-20T17:27:55.081 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.093 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1a50240:::benchmark_data_smithi171_122772_object124170:head# 2024-04-20T17:27:55.093 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.093 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1a50963:::benchmark_data_smithi171_122772_object143833:head# 2024-04-20T17:27:55.093 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.093 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1a55d46:::benchmark_data_smithi171_122772_object161318:head# 2024-04-20T17:27:55.093 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.093 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1a60f30:::benchmark_data_smithi171_122772_object118975:head# 2024-04-20T17:27:55.093 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.093 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1a6199b:::benchmark_data_smithi171_122772_object82275:head# 2024-04-20T17:27:55.093 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.093 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1a64ba4:::benchmark_data_smithi171_122772_object80651:head# 2024-04-20T17:27:55.093 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.093 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1a80715:::benchmark_data_smithi171_122772_object143106:head# 2024-04-20T17:27:55.094 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.094 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1a94daf:::benchmark_data_smithi171_122772_object97950:head# 2024-04-20T17:27:55.094 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.094 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1a9753a:::benchmark_data_smithi171_122772_object100419:head# 2024-04-20T17:27:55.094 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.094 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1a98d8a:::benchmark_data_smithi171_122772_object143325:head# 2024-04-20T17:27:55.094 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.094 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1aa6f5a:::benchmark_data_smithi171_122772_object140190:head# 2024-04-20T17:27:55.094 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.094 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1aa95ee:::benchmark_data_smithi171_122772_object156281:head# 2024-04-20T17:27:55.094 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.094 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1abbdd4:::benchmark_data_smithi171_122772_object120895:head# 2024-04-20T17:27:55.094 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.095 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1ac9334:::benchmark_data_smithi171_122772_object146550:head# 2024-04-20T17:27:55.095 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.095 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1ad943e:::benchmark_data_smithi171_122772_object83779:head# 2024-04-20T17:27:55.095 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.095 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1aee803:::benchmark_data_smithi171_122772_object81890:head# 2024-04-20T17:27:55.095 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.095 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1af2d93:::benchmark_data_smithi171_122772_object137360:head# 2024-04-20T17:27:55.095 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.095 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1af6231:::benchmark_data_smithi171_122772_object134610:head# 2024-04-20T17:27:55.095 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.095 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1b02b5a:::benchmark_data_smithi171_122772_object151975:head# 2024-04-20T17:27:55.095 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.095 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1b2bbeb:::benchmark_data_smithi171_122772_object135502:head# 2024-04-20T17:27:55.095 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.095 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1b3d582:::benchmark_data_smithi171_122772_object159327:head# 2024-04-20T17:27:55.096 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.096 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1b433d5:::benchmark_data_smithi171_122772_object92403:head# 2024-04-20T17:27:55.096 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.096 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1b44b61:::benchmark_data_smithi171_122772_object127225:head# 2024-04-20T17:27:55.096 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.096 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1b4e809:::benchmark_data_smithi171_122772_object123874:head# 2024-04-20T17:27:55.096 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.096 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1b59093:::benchmark_data_smithi171_122772_object137859:head# 2024-04-20T17:27:55.096 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.096 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1b59f0c:::benchmark_data_smithi171_122772_object97143:head# 2024-04-20T17:27:55.096 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.096 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1b72a1c:::benchmark_data_smithi171_122772_object155828:head# 2024-04-20T17:27:55.096 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.096 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1b73a08:::benchmark_data_smithi171_122772_object163916:head# 2024-04-20T17:27:55.096 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.097 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1b8ab3a:::benchmark_data_smithi171_122772_object165880:head# 2024-04-20T17:27:55.097 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.097 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1ba17da:::benchmark_data_smithi171_122772_object91740:head# 2024-04-20T17:27:55.097 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.097 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1bb4693:::benchmark_data_smithi171_122772_object125253:head# 2024-04-20T17:27:55.097 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.097 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1bba48e:::benchmark_data_smithi171_122772_object139609:head# 2024-04-20T17:27:55.097 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.097 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1bc8ad0:::benchmark_data_smithi171_122772_object100174:head# 2024-04-20T17:27:55.097 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.097 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1bcb956:::benchmark_data_smithi171_122772_object90702:head# 2024-04-20T17:27:55.097 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.097 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1bdf3b0:::benchmark_data_smithi171_122772_object169702:head# 2024-04-20T17:27:55.097 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.098 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1bf7cfa:::benchmark_data_smithi171_122772_object103816:head# 2024-04-20T17:27:55.098 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.098 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1bf98e0:::benchmark_data_smithi171_122772_object144608:head# 2024-04-20T17:27:55.098 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.098 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1bff17a:::benchmark_data_smithi171_122772_object86198:head# 2024-04-20T17:27:55.098 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.098 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1c086fb:::benchmark_data_smithi171_122772_object147011:head# 2024-04-20T17:27:55.098 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.098 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1c0ab2e:::benchmark_data_smithi171_122772_object101223:head# 2024-04-20T17:27:55.098 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.098 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1c0cc57:::benchmark_data_smithi171_122772_object161304:head# 2024-04-20T17:27:55.098 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.098 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1c124fc:::benchmark_data_smithi171_122772_object116784:head# 2024-04-20T17:27:55.098 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.098 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1c1373a:::benchmark_data_smithi171_122772_object126391:head# 2024-04-20T17:27:55.099 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.099 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1c17b42:::benchmark_data_smithi171_122772_object152866:head# 2024-04-20T17:27:55.099 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.099 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1c27d83:::benchmark_data_smithi171_122772_object83902:head# 2024-04-20T17:27:55.099 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.099 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1c2b591:::benchmark_data_smithi171_122772_object162889:head# 2024-04-20T17:27:55.099 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.099 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1c2f569:::benchmark_data_smithi171_122772_object109047:head# 2024-04-20T17:27:55.099 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.099 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1c31b07:::benchmark_data_smithi171_122772_object168398:head# 2024-04-20T17:27:55.099 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.099 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1c3a710:::benchmark_data_smithi171_122772_object152100:head# 2024-04-20T17:27:55.099 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.100 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1c3c00c:::benchmark_data_smithi171_122772_object97729:head# 2024-04-20T17:27:55.100 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.100 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1c43924:::benchmark_data_smithi171_122772_object162177:head# 2024-04-20T17:27:55.100 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.100 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1c4629f:::benchmark_data_smithi171_122772_object158713:head# 2024-04-20T17:27:55.100 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.100 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1c5172e:::benchmark_data_smithi171_122772_object143010:head# 2024-04-20T17:27:55.100 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.100 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1c5876c:::benchmark_data_smithi171_122772_object78403:head# 2024-04-20T17:27:55.100 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.100 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1c5d431:::benchmark_data_smithi171_122772_object159170:head# 2024-04-20T17:27:55.100 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.100 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1c5fa18:::benchmark_data_smithi171_122772_object91145:head# 2024-04-20T17:27:55.100 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.101 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1c6c3a2:::benchmark_data_smithi171_122772_object156425:head# 2024-04-20T17:27:55.101 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.101 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1c704d9:::benchmark_data_smithi171_122772_object164086:head# 2024-04-20T17:27:55.101 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.101 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1c7f44b:::benchmark_data_smithi171_122772_object165334:head# 2024-04-20T17:27:55.101 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.101 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1c8b278:::benchmark_data_smithi171_122772_object85986:head# 2024-04-20T17:27:55.101 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.101 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1c98af9:::benchmark_data_smithi171_122772_object132511:head# 2024-04-20T17:27:55.101 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.101 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1ca4295:::benchmark_data_smithi171_122772_object117159:head# 2024-04-20T17:27:55.101 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.101 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1ca633a:::benchmark_data_smithi171_122772_object131753:head# 2024-04-20T17:27:55.101 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.101 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1ca8ffc:::benchmark_data_smithi171_122772_object90102:head# 2024-04-20T17:27:55.102 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.102 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1cae4de:::benchmark_data_smithi171_122772_object138060:head# 2024-04-20T17:27:55.102 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.102 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1cc82a1:::benchmark_data_smithi171_122772_object79344:head# 2024-04-20T17:27:55.102 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.102 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1cced76:::benchmark_data_smithi171_122772_object97959:head# 2024-04-20T17:27:55.102 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.102 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1cd17ef:::benchmark_data_smithi171_122772_object165643:head# 2024-04-20T17:27:55.102 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.102 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1cd1986:::benchmark_data_smithi171_122772_object123698:head# 2024-04-20T17:27:55.102 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.102 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1cdc94a:::benchmark_data_smithi171_122772_object140301:head# 2024-04-20T17:27:55.102 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.102 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1d1f6e4:::benchmark_data_smithi171_122772_object161289:head# 2024-04-20T17:27:55.103 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.103 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1d35ec8:::benchmark_data_smithi171_122772_object154366:head# 2024-04-20T17:27:55.103 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.103 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1d50bd3:::benchmark_data_smithi171_122772_object81848:head# 2024-04-20T17:27:55.103 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.103 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1d611c8:::benchmark_data_smithi171_122772_object79970:head# 2024-04-20T17:27:55.103 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.103 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1d6c921:::benchmark_data_smithi171_122772_object148294:head# 2024-04-20T17:27:55.103 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.103 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1d77ad5:::benchmark_data_smithi171_122772_object148525:head# 2024-04-20T17:27:55.103 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.103 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1d8224c:::benchmark_data_smithi171_122772_object84720:head# 2024-04-20T17:27:55.103 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.103 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1d8571e:::benchmark_data_smithi171_122772_object123519:head# 2024-04-20T17:27:55.103 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.104 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1d95758:::benchmark_data_smithi171_122772_object101674:head# 2024-04-20T17:27:55.104 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.104 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1d98f6a:::benchmark_data_smithi171_122772_object91802:head# 2024-04-20T17:27:55.104 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.104 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1db06b1:::benchmark_data_smithi171_122772_object167883:head# 2024-04-20T17:27:55.104 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.104 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1dbd776:::benchmark_data_smithi171_122772_object95370:head# 2024-04-20T17:27:55.104 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.104 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1dc4fc4:::benchmark_data_smithi171_122772_object91066:head# 2024-04-20T17:27:55.104 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.104 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1dca3b1:::benchmark_data_smithi171_122772_object137288:head# 2024-04-20T17:27:55.104 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.104 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1dd2c0d:::benchmark_data_smithi171_122772_object100739:head# 2024-04-20T17:27:55.104 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.105 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1dd5eb9:::benchmark_data_smithi171_122772_object158439:head# 2024-04-20T17:27:55.105 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.105 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1ddd612:::benchmark_data_smithi171_122772_object142917:head# 2024-04-20T17:27:55.105 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.105 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1df3d34:::benchmark_data_smithi171_122772_object154111:head# 2024-04-20T17:27:55.105 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.105 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1df69a9:::benchmark_data_smithi171_122772_object97279:head# 2024-04-20T17:27:55.105 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.105 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1e0561d:::benchmark_data_smithi171_122772_object158827:head# 2024-04-20T17:27:55.105 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.105 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1e0ef7d:::benchmark_data_smithi171_122772_object119912:head# 2024-04-20T17:27:55.105 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.105 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1e385dc:::benchmark_data_smithi171_122772_object169093:head# 2024-04-20T17:27:55.105 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.119 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1e398b7:::benchmark_data_smithi171_122772_object158086:head# 2024-04-20T17:27:55.119 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.119 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1e3bbf4:::benchmark_data_smithi171_122772_object153711:head# 2024-04-20T17:27:55.119 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.119 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1e3cf8d:::benchmark_data_smithi171_122772_object127884:head# 2024-04-20T17:27:55.119 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.119 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1e3eff0:::benchmark_data_smithi171_122772_object135354:head# 2024-04-20T17:27:55.119 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.120 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1e411fa:::benchmark_data_smithi171_122772_object134995:head# 2024-04-20T17:27:55.120 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.120 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1e57254:::benchmark_data_smithi171_122772_object127853:head# 2024-04-20T17:27:55.120 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.120 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1e64314:::benchmark_data_smithi171_122772_object97164:head# 2024-04-20T17:27:55.120 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.120 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1e66d59:::benchmark_data_smithi171_122772_object154125:head# 2024-04-20T17:27:55.120 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.120 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1ea532f:::benchmark_data_smithi171_122772_object163090:head# 2024-04-20T17:27:55.120 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.120 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1eb04c1:::benchmark_data_smithi171_122772_object106011:head# 2024-04-20T17:27:55.120 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.121 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1eb4fcb:::benchmark_data_smithi171_122772_object79126:head# 2024-04-20T17:27:55.121 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.121 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1eb9957:::benchmark_data_smithi171_122772_object100933:head# 2024-04-20T17:27:55.121 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.121 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1ebf086:::benchmark_data_smithi171_122772_object156024:head# 2024-04-20T17:27:55.121 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.121 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1ebf985:::benchmark_data_smithi171_122772_object153246:head# 2024-04-20T17:27:55.121 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.121 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1ec7809:::benchmark_data_smithi171_122772_object156712:head# 2024-04-20T17:27:55.121 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.121 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1ecb351:::benchmark_data_smithi171_122772_object152359:head# 2024-04-20T17:27:55.121 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.121 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1ecb7f8:::benchmark_data_smithi171_122772_object124921:head# 2024-04-20T17:27:55.121 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.122 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1eeeeb0:::benchmark_data_smithi171_122772_object153629:head# 2024-04-20T17:27:55.122 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.122 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1f014b6:::benchmark_data_smithi171_122772_object133351:head# 2024-04-20T17:27:55.122 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.122 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1f1006e:::benchmark_data_smithi171_122772_object126063:head# 2024-04-20T17:27:55.122 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.122 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1f241af:::benchmark_data_smithi171_122772_object146398:head# 2024-04-20T17:27:55.122 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.122 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1f2d943:::benchmark_data_smithi171_122772_object160645:head# 2024-04-20T17:27:55.122 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.122 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1f34c47:::benchmark_data_smithi171_122772_object148983:head# 2024-04-20T17:27:55.122 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.122 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1f3602b:::benchmark_data_smithi171_122772_object120772:head# 2024-04-20T17:27:55.122 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.123 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1f36988:::benchmark_data_smithi171_122772_object114623:head# 2024-04-20T17:27:55.123 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.123 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1f5e7c7:::benchmark_data_smithi171_122772_object121749:head# 2024-04-20T17:27:55.123 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.123 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1f6164f:::benchmark_data_smithi171_122772_object129331:head# 2024-04-20T17:27:55.123 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.123 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1f69b98:::benchmark_data_smithi171_122772_object151655:head# 2024-04-20T17:27:55.123 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.123 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1f6b6e6:::benchmark_data_smithi171_122772_object135361:head# 2024-04-20T17:27:55.123 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.123 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1f6d498:::benchmark_data_smithi171_122772_object164583:head# 2024-04-20T17:27:55.123 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.123 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1f893cc:::benchmark_data_smithi171_122772_object138954:head# 2024-04-20T17:27:55.123 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.124 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1f8e160:::benchmark_data_smithi171_122772_object136239:head# 2024-04-20T17:27:55.124 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.124 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1fa0dc5:::benchmark_data_smithi171_122772_object94761:head# 2024-04-20T17:27:55.124 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.124 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1fa6923:::benchmark_data_smithi171_122772_object83828:head# 2024-04-20T17:27:55.124 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.124 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1fad3d4:::benchmark_data_smithi171_122772_object86219:head# 2024-04-20T17:27:55.124 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.124 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1fbf3ed:::benchmark_data_smithi171_122772_object147695:head# 2024-04-20T17:27:55.124 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.124 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1fcbbe1:::benchmark_data_smithi171_122772_object144825:head# 2024-04-20T17:27:55.124 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.124 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1fd8585:::benchmark_data_smithi171_122772_object140905:head# 2024-04-20T17:27:55.124 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.124 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1fdbb05:::benchmark_data_smithi171_122772_object123466:head# 2024-04-20T17:27:55.125 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.125 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1fde8e4:::benchmark_data_smithi171_122772_object157402:head# 2024-04-20T17:27:55.125 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.125 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1fe7d4a:::benchmark_data_smithi171_122772_object120066:head# 2024-04-20T17:27:55.125 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.125 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1fef6ef:::benchmark_data_smithi171_122772_object101484:head# 2024-04-20T17:27:55.125 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.125 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a1ffb206:::benchmark_data_smithi171_122772_object116301:head# 2024-04-20T17:27:55.125 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.125 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a200790f:::benchmark_data_smithi171_122772_object149742:head# 2024-04-20T17:27:55.125 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.125 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a200dfdb:::benchmark_data_smithi171_122772_object78974:head# 2024-04-20T17:27:55.125 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.125 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2011d4a:::benchmark_data_smithi171_122772_object117431:head# 2024-04-20T17:27:55.125 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.126 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2044708:::benchmark_data_smithi171_122772_object162911:head# 2024-04-20T17:27:55.126 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.126 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a205b227:::benchmark_data_smithi171_122772_object139994:head# 2024-04-20T17:27:55.126 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.126 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2060b28:::benchmark_data_smithi171_122772_object105317:head# 2024-04-20T17:27:55.126 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.126 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2071a78:::benchmark_data_smithi171_122772_object132957:head# 2024-04-20T17:27:55.126 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.126 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2079194:::benchmark_data_smithi171_122772_object138224:head# 2024-04-20T17:27:55.126 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.126 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a207e502:::benchmark_data_smithi171_122772_object147570:head# 2024-04-20T17:27:55.126 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.126 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2081ea5:::benchmark_data_smithi171_122772_object104023:head# 2024-04-20T17:27:55.126 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.127 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2082a2e:::benchmark_data_smithi171_122772_object129930:head# 2024-04-20T17:27:55.127 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.127 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2088f30:::benchmark_data_smithi171_122772_object161173:head# 2024-04-20T17:27:55.127 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.127 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a208b2b8:::benchmark_data_smithi171_122772_object157344:head# 2024-04-20T17:27:55.127 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.127 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a209f7d8:::benchmark_data_smithi171_122772_object107113:head# 2024-04-20T17:27:55.127 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.127 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a20a6670:::benchmark_data_smithi171_122772_object130698:head# 2024-04-20T17:27:55.127 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.127 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a20aa73b:::benchmark_data_smithi171_122772_object119036:head# 2024-04-20T17:27:55.127 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.127 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a20f07d3:::benchmark_data_smithi171_122772_object169198:head# 2024-04-20T17:27:55.127 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.127 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a20f6f06:::benchmark_data_smithi171_122772_object138758:head# 2024-04-20T17:27:55.128 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.128 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a210066b:::benchmark_data_smithi171_122772_object166951:head# 2024-04-20T17:27:55.128 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.128 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a212e211:::benchmark_data_smithi171_122772_object114292:head# 2024-04-20T17:27:55.128 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.128 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2148d8b:::benchmark_data_smithi171_122772_object139657:head# 2024-04-20T17:27:55.128 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.128 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a214bc6a:::benchmark_data_smithi171_122772_object87252:head# 2024-04-20T17:27:55.128 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.128 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a214ee67:::benchmark_data_smithi171_122772_object118121:head# 2024-04-20T17:27:55.128 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.128 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a215223a:::benchmark_data_smithi171_122772_object114175:head# 2024-04-20T17:27:55.128 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.128 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2161b9b:::benchmark_data_smithi171_122772_object128748:head# 2024-04-20T17:27:55.129 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.129 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a21698c9:::benchmark_data_smithi171_122772_object166375:head# 2024-04-20T17:27:55.129 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.129 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2189a31:::benchmark_data_smithi171_122772_object169829:head# 2024-04-20T17:27:55.129 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.129 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2193eca:::benchmark_data_smithi171_122772_object156740:head# 2024-04-20T17:27:55.129 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.129 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a21a282d:::benchmark_data_smithi171_122772_object126465:head# 2024-04-20T17:27:55.129 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.129 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a21b42dc:::benchmark_data_smithi171_122772_object130073:head# 2024-04-20T17:27:55.129 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.129 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a21bbae7:::benchmark_data_smithi171_122772_object165648:head# 2024-04-20T17:27:55.129 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.129 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a21ce2fc:::benchmark_data_smithi171_122772_object121837:head# 2024-04-20T17:27:55.129 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.130 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a21e05a2:::benchmark_data_smithi171_122772_object85556:head# 2024-04-20T17:27:55.130 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.130 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a21e12bb:::benchmark_data_smithi171_122772_object82219:head# 2024-04-20T17:27:55.130 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.130 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a22016ae:::benchmark_data_smithi171_122772_object127063:head# 2024-04-20T17:27:55.130 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.130 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2204204:::benchmark_data_smithi171_122772_object96239:head# 2024-04-20T17:27:55.130 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.130 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a221d9a5:::benchmark_data_smithi171_122772_object142020:head# 2024-04-20T17:27:55.130 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.130 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a223194b:::benchmark_data_smithi171_122772_object119867:head# 2024-04-20T17:27:55.130 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.130 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a22365ee:::benchmark_data_smithi171_122772_object150345:head# 2024-04-20T17:27:55.130 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.131 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2252380:::benchmark_data_smithi171_122772_object143689:head# 2024-04-20T17:27:55.131 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.131 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a226474e:::benchmark_data_smithi171_122772_object160724:head# 2024-04-20T17:27:55.131 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.131 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a226e2d3:::benchmark_data_smithi171_122772_object117055:head# 2024-04-20T17:27:55.131 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.131 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a226ffff:::benchmark_data_smithi171_122772_object155160:head# 2024-04-20T17:27:55.131 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.131 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a227221f:::benchmark_data_smithi171_122772_object99803:head# 2024-04-20T17:27:55.131 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.131 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2277b61:::benchmark_data_smithi171_122772_object119958:head# 2024-04-20T17:27:55.131 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.131 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a227bc4f:::benchmark_data_smithi171_122772_object107419:head# 2024-04-20T17:27:55.131 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.131 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a22820cd:::benchmark_data_smithi171_122772_object135694:head# 2024-04-20T17:27:55.132 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.132 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a228b6a6:::benchmark_data_smithi171_122772_object120366:head# 2024-04-20T17:27:55.132 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.132 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a228f7f4:::benchmark_data_smithi171_122772_object137400:head# 2024-04-20T17:27:55.132 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.132 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a22990f8:::benchmark_data_smithi171_122772_object101994:head# 2024-04-20T17:27:55.132 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.132 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a229d2c9:::benchmark_data_smithi171_122772_object111245:head# 2024-04-20T17:27:55.132 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.132 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a229e8e0:::benchmark_data_smithi171_122772_object90517:head# 2024-04-20T17:27:55.132 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.132 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a22b95f4:::benchmark_data_smithi171_122772_object145757:head# 2024-04-20T17:27:55.132 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.132 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a22d8faa:::benchmark_data_smithi171_122772_object135687:head# 2024-04-20T17:27:55.132 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.133 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a22d98d7:::benchmark_data_smithi171_122772_object93532:head# 2024-04-20T17:27:55.133 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.133 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a22da5ee:::benchmark_data_smithi171_122772_object148535:head# 2024-04-20T17:27:55.133 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.133 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a22e2040:::benchmark_data_smithi171_122772_object89757:head# 2024-04-20T17:27:55.133 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.133 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a22f050a:::benchmark_data_smithi171_122772_object126576:head# 2024-04-20T17:27:55.133 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.136 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a22f4c22:::benchmark_data_smithi171_122772_object90395:head# 2024-04-20T17:27:55.137 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.137 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2310a00:::benchmark_data_smithi171_122772_object94742:head# 2024-04-20T17:27:55.137 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.137 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2310dee:::benchmark_data_smithi171_122772_object133689:head# 2024-04-20T17:27:55.137 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.137 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2312a3c:::benchmark_data_smithi171_122772_object138354:head# 2024-04-20T17:27:55.137 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.137 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a232732b:::benchmark_data_smithi171_122772_object132675:head# 2024-04-20T17:27:55.137 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.137 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2329d53:::benchmark_data_smithi171_122772_object139831:head# 2024-04-20T17:27:55.137 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.137 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2336e38:::benchmark_data_smithi171_122772_object130805:head# 2024-04-20T17:27:55.138 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.138 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a233e6bd:::benchmark_data_smithi171_122772_object93553:head# 2024-04-20T17:27:55.138 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.138 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a237a885:::benchmark_data_smithi171_122772_object83477:head# 2024-04-20T17:27:55.138 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.138 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a237d9b6:::benchmark_data_smithi171_122772_object154553:head# 2024-04-20T17:27:55.138 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.138 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a23844a1:::benchmark_data_smithi171_122772_object147468:head# 2024-04-20T17:27:55.138 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.138 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2386753:::benchmark_data_smithi171_122772_object98507:head# 2024-04-20T17:27:55.138 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.138 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2389809:::benchmark_data_smithi171_122772_object149178:head# 2024-04-20T17:27:55.138 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.139 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a238acb5:::benchmark_data_smithi171_122772_object86814:head# 2024-04-20T17:27:55.139 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.139 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2393898:::benchmark_data_smithi171_122772_object91233:head# 2024-04-20T17:27:55.139 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.139 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a23a386f:::benchmark_data_smithi171_122772_object151969:head# 2024-04-20T17:27:55.139 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.139 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a23c0e3e:::benchmark_data_smithi171_122772_object146325:head# 2024-04-20T17:27:55.139 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.139 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a23c950d:::benchmark_data_smithi171_122772_object111925:head# 2024-04-20T17:27:55.139 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.139 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a23d52e8:::benchmark_data_smithi171_122772_object134311:head# 2024-04-20T17:27:55.139 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.139 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a23ddd62:::benchmark_data_smithi171_122772_object115460:head# 2024-04-20T17:27:55.139 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.140 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a23df218:::benchmark_data_smithi171_122772_object166634:head# 2024-04-20T17:27:55.140 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.140 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a23fc673:::benchmark_data_smithi171_122772_object83501:head# 2024-04-20T17:27:55.140 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.140 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2409e1a:::benchmark_data_smithi171_122772_object95393:head# 2024-04-20T17:27:55.140 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.140 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2416cc1:::benchmark_data_smithi171_122772_object93322:head# 2024-04-20T17:27:55.140 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.140 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a241aada:::benchmark_data_smithi171_122772_object105963:head# 2024-04-20T17:27:55.140 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.140 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a241b75d:::benchmark_data_smithi171_122772_object150270:head# 2024-04-20T17:27:55.140 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.140 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a24250be:::benchmark_data_smithi171_122772_object121577:head# 2024-04-20T17:27:55.141 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.141 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2453f4e:::benchmark_data_smithi171_122772_object136638:head# 2024-04-20T17:27:55.141 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.141 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2485f3a:::benchmark_data_smithi171_122772_object147741:head# 2024-04-20T17:27:55.141 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.141 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a248f36a:::benchmark_data_smithi171_122772_object151916:head# 2024-04-20T17:27:55.141 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.141 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a24911c4:::benchmark_data_smithi171_122772_object140807:head# 2024-04-20T17:27:55.141 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.141 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a24958f9:::benchmark_data_smithi171_122772_object140231:head# 2024-04-20T17:27:55.141 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.141 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a24ab2fe:::benchmark_data_smithi171_122772_object79564:head# 2024-04-20T17:27:55.141 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.141 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a24b5b8c:::benchmark_data_smithi171_122772_object97581:head# 2024-04-20T17:27:55.141 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.142 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a24dbe33:::benchmark_data_smithi171_122772_object163460:head# 2024-04-20T17:27:55.142 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.142 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a24f10ed:::benchmark_data_smithi171_122772_object140865:head# 2024-04-20T17:27:55.142 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.142 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a24fda16:::benchmark_data_smithi171_122772_object82125:head# 2024-04-20T17:27:55.142 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.142 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a25111bf:::benchmark_data_smithi171_122772_object109474:head# 2024-04-20T17:27:55.142 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.142 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a25265cc:::benchmark_data_smithi171_122772_object117671:head# 2024-04-20T17:27:55.142 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.142 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a252d672:::benchmark_data_smithi171_122772_object160297:head# 2024-04-20T17:27:55.142 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.142 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a252f2d9:::benchmark_data_smithi171_122772_object157542:head# 2024-04-20T17:27:55.143 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.143 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2530a7e:::benchmark_data_smithi171_122772_object114434:head# 2024-04-20T17:27:55.143 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.143 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a254daef:::benchmark_data_smithi171_122772_object120053:head# 2024-04-20T17:27:55.143 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.143 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a255abd2:::benchmark_data_smithi171_122772_object113507:head# 2024-04-20T17:27:55.143 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.143 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2562d86:::benchmark_data_smithi171_122772_object122266:head# 2024-04-20T17:27:55.143 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.143 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2597bc1:::benchmark_data_smithi171_122772_object103017:head# 2024-04-20T17:27:55.143 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.143 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a25a66c4:::benchmark_data_smithi171_122772_object150597:head# 2024-04-20T17:27:55.143 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.143 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a25aa6c0:::benchmark_data_smithi171_122772_object155623:head# 2024-04-20T17:27:55.143 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.143 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a25b3a22:::benchmark_data_smithi171_122772_object148760:head# 2024-04-20T17:27:55.144 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.144 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a25bef9c:::benchmark_data_smithi171_122772_object155922:head# 2024-04-20T17:27:55.144 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.144 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a25cf0b5:::benchmark_data_smithi171_122772_object160428:head# 2024-04-20T17:27:55.144 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.144 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a25da818:::benchmark_data_smithi171_122772_object157487:head# 2024-04-20T17:27:55.144 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.144 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a25df0f1:::benchmark_data_smithi171_122772_object138706:head# 2024-04-20T17:27:55.144 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.144 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a25e1235:::benchmark_data_smithi171_122772_object101480:head# 2024-04-20T17:27:55.144 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.144 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a25ee5f2:::benchmark_data_smithi171_122772_object110966:head# 2024-04-20T17:27:55.144 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.144 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a25f5c2e:::benchmark_data_smithi171_122772_object85248:head# 2024-04-20T17:27:55.145 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.145 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a25fde20:::benchmark_data_smithi171_122772_object168726:head# 2024-04-20T17:27:55.145 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.164 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2617a95:::benchmark_data_smithi171_122772_object124561:head# 2024-04-20T17:27:55.164 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.164 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2618e51:::benchmark_data_smithi171_122772_object107967:head# 2024-04-20T17:27:55.164 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.164 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a261a50e:::benchmark_data_smithi171_122772_object107338:head# 2024-04-20T17:27:55.165 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.165 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2627098:::benchmark_data_smithi171_122772_object116290:head# 2024-04-20T17:27:55.165 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.165 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a26454ae:::benchmark_data_smithi171_122772_object117712:head# 2024-04-20T17:27:55.165 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.165 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a26463b7:::benchmark_data_smithi171_122772_object100482:head# 2024-04-20T17:27:55.165 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.165 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a26538e8:::benchmark_data_smithi171_122772_object103708:head# 2024-04-20T17:27:55.165 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.165 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a265be4b:::benchmark_data_smithi171_122772_object91447:head# 2024-04-20T17:27:55.165 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.166 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a265c0fa:::benchmark_data_smithi171_122772_object157810:head# 2024-04-20T17:27:55.166 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.166 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a265f131:::benchmark_data_smithi171_122772_object127668:head# 2024-04-20T17:27:55.166 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.166 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2668f5c:::benchmark_data_smithi171_122772_object88409:head# 2024-04-20T17:27:55.166 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.166 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2680544:::benchmark_data_smithi171_122772_object90146:head# 2024-04-20T17:27:55.166 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.166 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a269b310:::benchmark_data_smithi171_122772_object128877:head# 2024-04-20T17:27:55.166 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.166 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a26a7984:::benchmark_data_smithi171_122772_object96579:head# 2024-04-20T17:27:55.167 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.167 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a26ac87b:::benchmark_data_smithi171_122772_object157180:head# 2024-04-20T17:27:55.167 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.167 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a26c0416:::benchmark_data_smithi171_122772_object156729:head# 2024-04-20T17:27:55.167 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.167 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a26d30bd:::benchmark_data_smithi171_122772_object98510:head# 2024-04-20T17:27:55.167 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.167 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a26d8369:::benchmark_data_smithi171_122772_object163409:head# 2024-04-20T17:27:55.167 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.167 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a26ea1d2:::benchmark_data_smithi171_122772_object166972:head# 2024-04-20T17:27:55.167 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.167 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a26f6cfb:::benchmark_data_smithi171_122772_object78202:head# 2024-04-20T17:27:55.168 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.168 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a26f821e:::benchmark_data_smithi171_122772_object147203:head# 2024-04-20T17:27:55.168 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.168 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a26f8d35:::benchmark_data_smithi171_122772_object163141:head# 2024-04-20T17:27:55.168 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.168 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a26fba7d:::benchmark_data_smithi171_122772_object95874:head# 2024-04-20T17:27:55.168 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.168 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2710a2c:::benchmark_data_smithi171_122772_object102162:head# 2024-04-20T17:27:55.168 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.168 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2725415:::benchmark_data_smithi171_122772_object159881:head# 2024-04-20T17:27:55.168 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.168 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2729e61:::benchmark_data_smithi171_122772_object125873:head# 2024-04-20T17:27:55.169 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.169 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a272aa1a:::benchmark_data_smithi171_122772_object103715:head# 2024-04-20T17:27:55.169 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.169 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a272cfb6:::benchmark_data_smithi171_122772_object162339:head# 2024-04-20T17:27:55.169 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.169 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a273710a:::benchmark_data_smithi171_122772_object123109:head# 2024-04-20T17:27:55.169 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.169 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2737b52:::benchmark_data_smithi171_122772_object150955:head# 2024-04-20T17:27:55.169 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.169 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2739b94:::benchmark_data_smithi171_122772_object122886:head# 2024-04-20T17:27:55.169 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.169 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a274872a:::benchmark_data_smithi171_122772_object105937:head# 2024-04-20T17:27:55.170 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.170 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a275931b:::benchmark_data_smithi171_122772_object127680:head# 2024-04-20T17:27:55.170 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.170 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2760ccc:::benchmark_data_smithi171_122772_object140282:head# 2024-04-20T17:27:55.170 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.170 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a27626d5:::benchmark_data_smithi171_122772_object114740:head# 2024-04-20T17:27:55.170 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.170 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a27640de:::benchmark_data_smithi171_122772_object144729:head# 2024-04-20T17:27:55.170 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.171 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2777237:::benchmark_data_smithi171_122772_object119403:head# 2024-04-20T17:27:55.171 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.171 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a279e383:::benchmark_data_smithi171_122772_object107174:head# 2024-04-20T17:27:55.171 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.171 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a27a751c:::benchmark_data_smithi171_122772_object134747:head# 2024-04-20T17:27:55.171 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.171 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a27aa8ba:::benchmark_data_smithi171_122772_object79383:head# 2024-04-20T17:27:55.171 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.171 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a27b020c:::benchmark_data_smithi171_122772_object162815:head# 2024-04-20T17:27:55.171 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.171 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a27b66f5:::benchmark_data_smithi171_122772_object96347:head# 2024-04-20T17:27:55.171 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.171 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a27bc5b3:::benchmark_data_smithi171_122772_object144241:head# 2024-04-20T17:27:55.172 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.172 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a27c25c3:::benchmark_data_smithi171_122772_object120333:head# 2024-04-20T17:27:55.172 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.172 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a27e6350:::benchmark_data_smithi171_122772_object123869:head# 2024-04-20T17:27:55.172 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.172 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a27e6930:::benchmark_data_smithi171_122772_object91772:head# 2024-04-20T17:27:55.172 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.172 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a27e79c9:::benchmark_data_smithi171_122772_object129824:head# 2024-04-20T17:27:55.172 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.172 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a280e32b:::benchmark_data_smithi171_122772_object144251:head# 2024-04-20T17:27:55.172 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.172 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a280fe35:::benchmark_data_smithi171_122772_object117229:head# 2024-04-20T17:27:55.173 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.173 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2816ff7:::benchmark_data_smithi171_122772_object125429:head# 2024-04-20T17:27:55.173 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.173 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a282236d:::benchmark_data_smithi171_122772_object106249:head# 2024-04-20T17:27:55.173 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.173 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2838209:::benchmark_data_smithi171_122772_object118750:head# 2024-04-20T17:27:55.173 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.173 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2843f7b:::benchmark_data_smithi171_122772_object161116:head# 2024-04-20T17:27:55.173 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.173 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a284640e:::benchmark_data_smithi171_122772_object167007:head# 2024-04-20T17:27:55.173 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.173 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2852f1f:::benchmark_data_smithi171_122772_object95527:head# 2024-04-20T17:27:55.173 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.174 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2859ae0:::benchmark_data_smithi171_122772_object164884:head# 2024-04-20T17:27:55.174 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.174 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a28789f2:::benchmark_data_smithi171_122772_object120354:head# 2024-04-20T17:27:55.174 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.174 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a287b7a6:::benchmark_data_smithi171_122772_object90193:head# 2024-04-20T17:27:55.174 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.174 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a287c002:::benchmark_data_smithi171_122772_object108404:head# 2024-04-20T17:27:55.174 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.174 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2886a12:::benchmark_data_smithi171_122772_object118454:head# 2024-04-20T17:27:55.174 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.174 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a288c170:::benchmark_data_smithi171_122772_object162348:head# 2024-04-20T17:27:55.174 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.175 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2895173:::benchmark_data_smithi171_122772_object146122:head# 2024-04-20T17:27:55.175 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.175 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a28a98c6:::benchmark_data_smithi171_122772_object152094:head# 2024-04-20T17:27:55.175 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.175 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a28b1829:::benchmark_data_smithi171_122772_object108993:head# 2024-04-20T17:27:55.175 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.175 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a28cb707:::benchmark_data_smithi171_122772_object110438:head# 2024-04-20T17:27:55.175 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.175 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a28dbd47:::benchmark_data_smithi171_122772_object149048:head# 2024-04-20T17:27:55.175 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.175 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a28ded95:::benchmark_data_smithi171_122772_object80517:head# 2024-04-20T17:27:55.176 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.176 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a28e592c:::benchmark_data_smithi171_122772_object124452:head# 2024-04-20T17:27:55.176 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.176 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a28ef858:::benchmark_data_smithi171_122772_object147942:head# 2024-04-20T17:27:55.176 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.176 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a290d3aa:::benchmark_data_smithi171_122772_object159249:head# 2024-04-20T17:27:55.176 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.176 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a29153ad:::benchmark_data_smithi171_122772_object165082:head# 2024-04-20T17:27:55.176 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.176 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2930b51:::benchmark_data_smithi171_122772_object148151:head# 2024-04-20T17:27:55.176 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.176 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2930eef:::benchmark_data_smithi171_122772_object124383:head# 2024-04-20T17:27:55.177 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.177 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a293b5c0:::benchmark_data_smithi171_122772_object100664:head# 2024-04-20T17:27:55.177 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.177 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2943315:::benchmark_data_smithi171_122772_object118924:head# 2024-04-20T17:27:55.177 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.177 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2962c94:::benchmark_data_smithi171_122772_object144051:head# 2024-04-20T17:27:55.177 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.177 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2966471:::benchmark_data_smithi171_122772_object100273:head# 2024-04-20T17:27:55.177 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.177 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a296fa8f:::benchmark_data_smithi171_122772_object149979:head# 2024-04-20T17:27:55.177 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.177 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2981c92:::benchmark_data_smithi171_122772_object148232:head# 2024-04-20T17:27:55.177 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.178 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a299649b:::benchmark_data_smithi171_122772_object84403:head# 2024-04-20T17:27:55.178 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.178 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a299b4ab:::benchmark_data_smithi171_122772_object135960:head# 2024-04-20T17:27:55.178 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.178 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a29a3360:::benchmark_data_smithi171_122772_object82456:head# 2024-04-20T17:27:55.178 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.178 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a29da1ff:::benchmark_data_smithi171_122772_object134369:head# 2024-04-20T17:27:55.178 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.178 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a29e62a5:::benchmark_data_smithi171_122772_object151335:head# 2024-04-20T17:27:55.178 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.178 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a29ea6d2:::benchmark_data_smithi171_122772_object124230:head# 2024-04-20T17:27:55.178 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.179 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a29ed4a6:::benchmark_data_smithi171_122772_object126693:head# 2024-04-20T17:27:55.179 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.179 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a29f2dea:::benchmark_data_smithi171_122772_object113387:head# 2024-04-20T17:27:55.179 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.179 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a29f6906:::benchmark_data_smithi171_122772_object90063:head# 2024-04-20T17:27:55.179 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.179 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2a20c2c:::benchmark_data_smithi171_122772_object161986:head# 2024-04-20T17:27:55.179 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.179 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2a250e6:::benchmark_data_smithi171_122772_object152904:head# 2024-04-20T17:27:55.179 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.179 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2a39416:::benchmark_data_smithi171_122772_object135234:head# 2024-04-20T17:27:55.179 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.180 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2a4568d:::benchmark_data_smithi171_122772_object100656:head# 2024-04-20T17:27:55.180 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.180 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2a4e4e9:::benchmark_data_smithi171_122772_object93906:head# 2024-04-20T17:27:55.180 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.180 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2a6f13f:::benchmark_data_smithi171_122772_object139188:head# 2024-04-20T17:27:55.180 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.180 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2a7e5cb:::benchmark_data_smithi171_122772_object168502:head# 2024-04-20T17:27:55.180 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.180 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2a95723:::benchmark_data_smithi171_122772_object125255:head# 2024-04-20T17:27:55.180 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.180 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2a9e28e:::benchmark_data_smithi171_122772_object104228:head# 2024-04-20T17:27:55.181 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.181 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2ab2cf8:::benchmark_data_smithi171_122772_object128758:head# 2024-04-20T17:27:55.181 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.181 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2ab30e5:::benchmark_data_smithi171_122772_object158463:head# 2024-04-20T17:27:55.181 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.181 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2ab37b5:::benchmark_data_smithi171_122772_object117053:head# 2024-04-20T17:27:55.181 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.181 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2ab3f33:::benchmark_data_smithi171_122772_object112388:head# 2024-04-20T17:27:55.181 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.181 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2ab60ac:::benchmark_data_smithi171_122772_object114138:head# 2024-04-20T17:27:55.181 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.181 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2ac9d1c:::benchmark_data_smithi171_122772_object118601:head# 2024-04-20T17:27:55.181 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.182 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2acb0f6:::benchmark_data_smithi171_122772_object96975:head# 2024-04-20T17:27:55.182 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.182 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2ad8935:::benchmark_data_smithi171_122772_object78021:head# 2024-04-20T17:27:55.182 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.182 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2ae75f8:::benchmark_data_smithi171_122772_object95728:head# 2024-04-20T17:27:55.182 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.182 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2aff558:::benchmark_data_smithi171_122772_object156216:head# 2024-04-20T17:27:55.182 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.182 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2affd63:::benchmark_data_smithi171_122772_object109459:head# 2024-04-20T17:27:55.182 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.182 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2b0132e:::benchmark_data_smithi171_122772_object114595:head# 2024-04-20T17:27:55.182 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.183 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2b167e7:::benchmark_data_smithi171_122772_object165990:head# 2024-04-20T17:27:55.183 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.183 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2b226b0:::benchmark_data_smithi171_122772_object127272:head# 2024-04-20T17:27:55.183 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.183 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2b2b4ae:::benchmark_data_smithi171_122772_object105312:head# 2024-04-20T17:27:55.183 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.183 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2b3326e:::benchmark_data_smithi171_122772_object80937:head# 2024-04-20T17:27:55.183 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.183 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:27:55.219 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2b370e8:::benchmark_data_smithi171_122772_object140459:head# 2024-04-20T17:27:55.219 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.219 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2b373de:::benchmark_data_smithi171_122772_object154273:head# 2024-04-20T17:27:55.219 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.219 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2b4e5dd:::benchmark_data_smithi171_122772_object100741:head# 2024-04-20T17:27:55.219 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.219 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2b5a1c3:::benchmark_data_smithi171_122772_object120108:head# 2024-04-20T17:27:55.219 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.220 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2b5fd0c:::benchmark_data_smithi171_122772_object86081:head# 2024-04-20T17:27:55.220 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.220 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2b62334:::benchmark_data_smithi171_122772_object126579:head# 2024-04-20T17:27:55.220 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.220 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2b62d01:::benchmark_data_smithi171_122772_object144497:head# 2024-04-20T17:27:55.220 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.220 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2b65cda:::benchmark_data_smithi171_122772_object80386:head# 2024-04-20T17:27:55.220 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.220 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2b685fe:::benchmark_data_smithi171_122772_object86192:head# 2024-04-20T17:27:55.220 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.220 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2b6d74c:::benchmark_data_smithi171_122772_object150629:head# 2024-04-20T17:27:55.220 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.221 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2b73588:::benchmark_data_smithi171_122772_object123462:head# 2024-04-20T17:27:55.221 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.221 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2b73dd2:::benchmark_data_smithi171_122772_object165015:head# 2024-04-20T17:27:55.221 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.221 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2b81d9c:::benchmark_data_smithi171_122772_object125400:head# 2024-04-20T17:27:55.221 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.221 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2b86508:::benchmark_data_smithi171_122772_object78848:head# 2024-04-20T17:27:55.221 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.221 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2b8bace:::benchmark_data_smithi171_122772_object80206:head# 2024-04-20T17:27:55.221 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.221 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2b9faa3:::benchmark_data_smithi171_122772_object104853:head# 2024-04-20T17:27:55.221 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.222 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2ba7f7e:::benchmark_data_smithi171_122772_object129255:head# 2024-04-20T17:27:55.222 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.222 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2ba8080:::benchmark_data_smithi171_122772_object106965:head# 2024-04-20T17:27:55.222 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.222 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2badd09:::benchmark_data_smithi171_122772_object160362:head# 2024-04-20T17:27:55.222 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.222 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2bdd271:::benchmark_data_smithi171_122772_object158187:head# 2024-04-20T17:27:55.222 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.222 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2beb2b2:::benchmark_data_smithi171_122772_object85785:head# 2024-04-20T17:27:55.222 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.223 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2bee8de:::benchmark_data_smithi171_122772_object142553:head# 2024-04-20T17:27:55.223 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.223 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2bf30fc:::benchmark_data_smithi171_122772_object88366:head# 2024-04-20T17:27:55.223 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.223 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2bf76b4:::benchmark_data_smithi171_122772_object135925:head# 2024-04-20T17:27:55.223 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.223 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2c09f0c:::benchmark_data_smithi171_122772_object117031:head# 2024-04-20T17:27:55.223 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.223 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2c0d964:::benchmark_data_smithi171_122772_object125943:head# 2024-04-20T17:27:55.223 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.223 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2c15e98:::benchmark_data_smithi171_122772_object169091:head# 2024-04-20T17:27:55.223 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.223 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2c1b3bc:::benchmark_data_smithi171_122772_object148567:head# 2024-04-20T17:27:55.224 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.224 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2c249d2:::benchmark_data_smithi171_122772_object146437:head# 2024-04-20T17:27:55.224 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.224 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2c29140:::benchmark_data_smithi171_122772_object94574:head# 2024-04-20T17:27:55.224 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.224 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2c2dae3:::benchmark_data_smithi171_122772_object156419:head# 2024-04-20T17:27:55.224 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.224 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2c34151:::benchmark_data_smithi171_122772_object158975:head# 2024-04-20T17:27:55.224 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.224 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2c58f52:::benchmark_data_smithi171_122772_object92920:head# 2024-04-20T17:27:55.224 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.224 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2c60794:::benchmark_data_smithi171_122772_object149769:head# 2024-04-20T17:27:55.225 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.225 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2c66ebb:::benchmark_data_smithi171_122772_object163807:head# 2024-04-20T17:27:55.225 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.225 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2c6c5ce:::benchmark_data_smithi171_122772_object157713:head# 2024-04-20T17:27:55.225 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.225 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2c7ed12:::benchmark_data_smithi171_122772_object111185:head# 2024-04-20T17:27:55.225 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.225 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2c97032:::benchmark_data_smithi171_122772_object163573:head# 2024-04-20T17:27:55.225 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.225 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2ca126a:::benchmark_data_smithi171_122772_object111137:head# 2024-04-20T17:27:55.225 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.225 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2ca5271:::benchmark_data_smithi171_122772_object117853:head# 2024-04-20T17:27:55.226 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.226 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2caca2b:::benchmark_data_smithi171_122772_object132781:head# 2024-04-20T17:27:55.226 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.226 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2cae3f9:::benchmark_data_smithi171_122772_object165910:head# 2024-04-20T17:27:55.226 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.226 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2cb3280:::benchmark_data_smithi171_122772_object155504:head# 2024-04-20T17:27:55.226 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.226 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2cb3911:::benchmark_data_smithi171_122772_object81853:head# 2024-04-20T17:27:55.226 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.226 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2cbc4e8:::benchmark_data_smithi171_122772_object110381:head# 2024-04-20T17:27:55.226 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.227 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2cbde9f:::benchmark_data_smithi171_122772_object164661:head# 2024-04-20T17:27:55.227 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.227 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2cbe117:::benchmark_data_smithi171_122772_object157178:head# 2024-04-20T17:27:55.227 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.227 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2cdc138:::benchmark_data_smithi171_122772_object95005:head# 2024-04-20T17:27:55.227 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.227 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2cde073:::benchmark_data_smithi171_122772_object152101:head# 2024-04-20T17:27:55.227 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.227 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2cdf74b:::benchmark_data_smithi171_122772_object96407:head# 2024-04-20T17:27:55.227 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.227 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2ce8d80:::benchmark_data_smithi171_122772_object102776:head# 2024-04-20T17:27:55.227 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.228 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2cf19c3:::benchmark_data_smithi171_122772_object111631:head# 2024-04-20T17:27:55.228 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.228 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2cf88ff:::benchmark_data_smithi171_122772_object119805:head# 2024-04-20T17:27:55.228 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.228 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2d02124:::benchmark_data_smithi171_122772_object123107:head# 2024-04-20T17:27:55.228 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.228 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2d04577:::benchmark_data_smithi171_122772_object84833:head# 2024-04-20T17:27:55.228 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.228 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2d106cd:::benchmark_data_smithi171_122772_object152559:head# 2024-04-20T17:27:55.228 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.228 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2d174cb:::benchmark_data_smithi171_122772_object106794:head# 2024-04-20T17:27:55.228 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.228 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2d23d78:::benchmark_data_smithi171_122772_object155045:head# 2024-04-20T17:27:55.229 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.229 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2d28119:::benchmark_data_smithi171_122772_object160307:head# 2024-04-20T17:27:55.229 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.229 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2d4603b:::benchmark_data_smithi171_122772_object138650:head# 2024-04-20T17:27:55.229 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.229 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2d46c77:::benchmark_data_smithi171_122772_object169304:head# 2024-04-20T17:27:55.229 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.229 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2d55740:::benchmark_data_smithi171_122772_object140406:head# 2024-04-20T17:27:55.229 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.229 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2d56273:::benchmark_data_smithi171_122772_object114486:head# 2024-04-20T17:27:55.229 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.230 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2d65891:::benchmark_data_smithi171_122772_object90238:head# 2024-04-20T17:27:55.230 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.230 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2d6ceac:::benchmark_data_smithi171_122772_object154711:head# 2024-04-20T17:27:55.230 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.230 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2d7d988:::benchmark_data_smithi171_122772_object164145:head# 2024-04-20T17:27:55.230 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.230 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2d8b208:::benchmark_data_smithi171_122772_object119135:head# 2024-04-20T17:27:55.230 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.230 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2d936ad:::benchmark_data_smithi171_122772_object133959:head# 2024-04-20T17:27:55.230 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.230 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2da1749:::benchmark_data_smithi171_122772_object94681:head# 2024-04-20T17:27:55.231 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.231 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2da94cd:::benchmark_data_smithi171_122772_object106675:head# 2024-04-20T17:27:55.231 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.231 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2db4e43:::benchmark_data_smithi171_122772_object146837:head# 2024-04-20T17:27:55.231 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.231 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2db5024:::benchmark_data_smithi171_122772_object96939:head# 2024-04-20T17:27:55.231 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.231 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2db72b8:::benchmark_data_smithi171_122772_object165731:head# 2024-04-20T17:27:55.231 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.231 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2dc12da:::benchmark_data_smithi171_122772_object137369:head# 2024-04-20T17:27:55.231 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.232 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2dc88d5:::benchmark_data_smithi171_122772_object83168:head# 2024-04-20T17:27:55.232 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.232 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2dca4de:::benchmark_data_smithi171_122772_object140662:head# 2024-04-20T17:27:55.232 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.232 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2decbe9:::benchmark_data_smithi171_122772_object153544:head# 2024-04-20T17:27:55.232 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.232 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2def62c:::benchmark_data_smithi171_122772_object88113:head# 2024-04-20T17:27:55.232 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.232 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2dfcb2f:::benchmark_data_smithi171_122772_object97834:head# 2024-04-20T17:27:55.232 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.232 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2e01d50:::benchmark_data_smithi171_122772_object141146:head# 2024-04-20T17:27:55.232 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.233 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2e0e2a1:::benchmark_data_smithi171_122772_object152653:head# 2024-04-20T17:27:55.233 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.233 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2e10fda:::benchmark_data_smithi171_122772_object148813:head# 2024-04-20T17:27:55.233 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.233 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2e1b595:::benchmark_data_smithi171_122772_object152175:head# 2024-04-20T17:27:55.233 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.233 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2e31377:::benchmark_data_smithi171_122772_object144730:head# 2024-04-20T17:27:55.233 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.233 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2e33b58:::benchmark_data_smithi171_122772_object155392:head# 2024-04-20T17:27:55.233 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.233 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2e4439a:::benchmark_data_smithi171_122772_object130648:head# 2024-04-20T17:27:55.233 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.233 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2e602a1:::benchmark_data_smithi171_122772_object88373:head# 2024-04-20T17:27:55.234 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.234 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2e68abe:::benchmark_data_smithi171_122772_object115408:head# 2024-04-20T17:27:55.234 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.234 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2e718c1:::benchmark_data_smithi171_122772_object117081:head# 2024-04-20T17:27:55.234 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.234 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2e80cba:::benchmark_data_smithi171_122772_object95011:head# 2024-04-20T17:27:55.234 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.234 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2e81e7a:::benchmark_data_smithi171_122772_object166087:head# 2024-04-20T17:27:55.234 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.234 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2e83916:::benchmark_data_smithi171_122772_object90132:head# 2024-04-20T17:27:55.234 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.234 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2e94235:::benchmark_data_smithi171_122772_object122568:head# 2024-04-20T17:27:55.235 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.235 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2e9d504:::benchmark_data_smithi171_122772_object137002:head# 2024-04-20T17:27:55.235 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.235 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2ea7609:::benchmark_data_smithi171_122772_object153076:head# 2024-04-20T17:27:55.235 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.235 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2ebd1e3:::benchmark_data_smithi171_122772_object147984:head# 2024-04-20T17:27:55.235 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.235 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2ec1a21:::benchmark_data_smithi171_122772_object84515:head# 2024-04-20T17:27:55.235 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.235 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2eca9df:::benchmark_data_smithi171_122772_object140830:head# 2024-04-20T17:27:55.235 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.236 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2ee740f:::benchmark_data_smithi171_122772_object132045:head# 2024-04-20T17:27:55.236 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.236 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2ee8515:::benchmark_data_smithi171_122772_object81939:head# 2024-04-20T17:27:55.236 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.236 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2ee9dca:::benchmark_data_smithi171_122772_object115188:head# 2024-04-20T17:27:55.236 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.236 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2eee8e0:::benchmark_data_smithi171_122772_object151891:head# 2024-04-20T17:27:55.236 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.236 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2efa3e5:::benchmark_data_smithi171_122772_object106386:head# 2024-04-20T17:27:55.236 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.237 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2efb3b5:::benchmark_data_smithi171_122772_object130861:head# 2024-04-20T17:27:55.237 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.237 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2f0dcdf:::benchmark_data_smithi171_122772_object133308:head# 2024-04-20T17:27:55.237 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.237 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2f121e9:::benchmark_data_smithi171_122772_object122069:head# 2024-04-20T17:27:55.237 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.237 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2f27b59:::benchmark_data_smithi171_122772_object96483:head# 2024-04-20T17:27:55.237 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.237 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2f3cb49:::benchmark_data_smithi171_122772_object169425:head# 2024-04-20T17:27:55.237 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.237 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2f3ce75:::benchmark_data_smithi171_122772_object120768:head# 2024-04-20T17:27:55.238 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.238 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2f470a2:::benchmark_data_smithi171_122772_object159072:head# 2024-04-20T17:27:55.238 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.238 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2f58c98:::benchmark_data_smithi171_122772_object132989:head# 2024-04-20T17:27:55.238 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.238 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2f6f3a2:::benchmark_data_smithi171_122772_object131173:head# 2024-04-20T17:27:55.238 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.238 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2f95c29:::benchmark_data_smithi171_122772_object88149:head# 2024-04-20T17:27:55.238 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.238 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2f9b1d6:::benchmark_data_smithi171_122772_object132719:head# 2024-04-20T17:27:55.238 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.239 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2fa2ff9:::benchmark_data_smithi171_122772_object146897:head# 2024-04-20T17:27:55.239 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.239 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2fb06d4:::benchmark_data_smithi171_122772_object133264:head# 2024-04-20T17:27:55.239 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.239 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a2fb259f:::benchmark_data_smithi171_122772_object153450:head# 2024-04-20T17:27:55.239 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.239 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3010b7d:::benchmark_data_smithi171_122772_object89337:head# 2024-04-20T17:27:55.239 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.239 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3016617:::benchmark_data_smithi171_122772_object160980:head# 2024-04-20T17:27:55.239 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.239 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3027d9f:::benchmark_data_smithi171_122772_object78144:head# 2024-04-20T17:27:55.239 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.239 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a30393d1:::benchmark_data_smithi171_122772_object101636:head# 2024-04-20T17:27:55.240 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.240 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3042484:::benchmark_data_smithi171_122772_object104253:head# 2024-04-20T17:27:55.240 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.240 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a304581e:::benchmark_data_smithi171_122772_object127445:head# 2024-04-20T17:27:55.240 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.240 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3059c38:::benchmark_data_smithi171_122772_object118850:head# 2024-04-20T17:27:55.240 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.240 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3059e0b:::benchmark_data_smithi171_122772_object86364:head# 2024-04-20T17:27:55.240 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.240 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a305b595:::benchmark_data_smithi171_122772_object142964:head# 2024-04-20T17:27:55.240 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.240 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3061661:::benchmark_data_smithi171_122772_object166066:head# 2024-04-20T17:27:55.241 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.241 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3062cb5:::benchmark_data_smithi171_122772_object90022:head# 2024-04-20T17:27:55.241 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.241 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3084bf5:::benchmark_data_smithi171_122772_object129914:head# 2024-04-20T17:27:55.241 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.241 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3085768:::benchmark_data_smithi171_122772_object100777:head# 2024-04-20T17:27:55.241 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.241 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a309c9ec:::benchmark_data_smithi171_122772_object139444:head# 2024-04-20T17:27:55.241 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.241 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a30a2bc1:::benchmark_data_smithi171_122772_object118948:head# 2024-04-20T17:27:55.241 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.242 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a30b7f25:::benchmark_data_smithi171_122772_object103195:head# 2024-04-20T17:27:55.242 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.242 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a30ca410:::benchmark_data_smithi171_122772_object84836:head# 2024-04-20T17:27:55.242 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.242 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a30cae30:::benchmark_data_smithi171_122772_object169047:head# 2024-04-20T17:27:55.242 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.242 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a30cb4b0:::benchmark_data_smithi171_122772_object129494:head# 2024-04-20T17:27:55.242 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.242 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a30cceb1:::benchmark_data_smithi171_122772_object114707:head# 2024-04-20T17:27:55.242 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.242 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a30e3189:::benchmark_data_smithi171_122772_object127015:head# 2024-04-20T17:27:55.242 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.243 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a30e3c4d:::benchmark_data_smithi171_122772_object132888:head# 2024-04-20T17:27:55.243 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.243 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a30e4963:::benchmark_data_smithi171_122772_object115060:head# 2024-04-20T17:27:55.243 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.243 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a30f71cf:::benchmark_data_smithi171_122772_object129667:head# 2024-04-20T17:27:55.243 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.243 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a30f9dd5:::benchmark_data_smithi171_122772_object130087:head# 2024-04-20T17:27:55.243 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.243 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3121e37:::benchmark_data_smithi171_122772_object143864:head# 2024-04-20T17:27:55.243 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.243 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a31269fa:::benchmark_data_smithi171_122772_object147436:head# 2024-04-20T17:27:55.243 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.244 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3126b16:::benchmark_data_smithi171_122772_object104363:head# 2024-04-20T17:27:55.244 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.244 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3156b7b:::benchmark_data_smithi171_122772_object100317:head# 2024-04-20T17:27:55.244 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.244 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a31690a8:::benchmark_data_smithi171_122772_object133707:head# 2024-04-20T17:27:55.244 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.244 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3183d4c:::benchmark_data_smithi171_122772_object84904:head# 2024-04-20T17:27:55.244 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.244 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a319e0cc:::benchmark_data_smithi171_122772_object142216:head# 2024-04-20T17:27:55.244 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.244 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a31a221e:::benchmark_data_smithi171_122772_object88954:head# 2024-04-20T17:27:55.244 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.245 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a31b93fe:::benchmark_data_smithi171_122772_object122675:head# 2024-04-20T17:27:55.245 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.245 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a31bc482:::benchmark_data_smithi171_122772_object110032:head# 2024-04-20T17:27:55.245 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.245 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a31bea5c:::benchmark_data_smithi171_122772_object83634:head# 2024-04-20T17:27:55.245 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.245 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a31c9de7:::benchmark_data_smithi171_122772_object135703:head# 2024-04-20T17:27:55.245 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.245 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a31d6f4c:::benchmark_data_smithi171_122772_object106401:head# 2024-04-20T17:27:55.245 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.245 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a31ebb27:::benchmark_data_smithi171_122772_object95317:head# 2024-04-20T17:27:55.245 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.246 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a31f11a7:::benchmark_data_smithi171_122772_object97402:head# 2024-04-20T17:27:55.246 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.246 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3210dce:::benchmark_data_smithi171_122772_object140956:head# 2024-04-20T17:27:55.246 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.246 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3215609:::benchmark_data_smithi171_122772_object82689:head# 2024-04-20T17:27:55.246 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.246 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a322a2ee:::benchmark_data_smithi171_122772_object96641:head# 2024-04-20T17:27:55.246 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.246 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a322c5a1:::benchmark_data_smithi171_122772_object137179:head# 2024-04-20T17:27:55.246 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.246 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3231cdd:::benchmark_data_smithi171_122772_object114891:head# 2024-04-20T17:27:55.246 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.247 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a32422f8:::benchmark_data_smithi171_122772_object131621:head# 2024-04-20T17:27:55.247 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.247 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3242bfc:::benchmark_data_smithi171_122772_object89129:head# 2024-04-20T17:27:55.247 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.247 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a325abb4:::benchmark_data_smithi171_122772_object142834:head# 2024-04-20T17:27:55.247 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.247 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3262f85:::benchmark_data_smithi171_122772_object163583:head# 2024-04-20T17:27:55.247 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.247 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3263477:::benchmark_data_smithi171_122772_object150682:head# 2024-04-20T17:27:55.247 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.247 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a328e0ca:::benchmark_data_smithi171_122772_object159041:head# 2024-04-20T17:27:55.247 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.248 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3292881:::benchmark_data_smithi171_122772_object123314:head# 2024-04-20T17:27:55.248 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.248 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a32a7a9c:::benchmark_data_smithi171_122772_object156301:head# 2024-04-20T17:27:55.248 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.248 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a32c023e:::benchmark_data_smithi171_122772_object121185:head# 2024-04-20T17:27:55.248 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.248 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a32d40c5:::benchmark_data_smithi171_122772_object121511:head# 2024-04-20T17:27:55.248 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.248 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a32d520c:::benchmark_data_smithi171_122772_object91918:head# 2024-04-20T17:27:55.248 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.248 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a32e6491:::benchmark_data_smithi171_122772_object136668:head# 2024-04-20T17:27:55.248 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.248 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a32f0002:::benchmark_data_smithi171_122772_object122509:head# 2024-04-20T17:27:55.249 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.249 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a32fe7c6:::benchmark_data_smithi171_122772_object169511:head# 2024-04-20T17:27:55.249 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.249 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3300682:::benchmark_data_smithi171_122772_object169831:head# 2024-04-20T17:27:55.249 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.249 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a330d119:::benchmark_data_smithi171_122772_object140214:head# 2024-04-20T17:27:55.249 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.249 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a330ecd2:::benchmark_data_smithi171_122772_object131380:head# 2024-04-20T17:27:55.249 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.249 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a331a230:::benchmark_data_smithi171_122772_object138363:head# 2024-04-20T17:27:55.249 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.249 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a331a566:::benchmark_data_smithi171_122772_object150549:head# 2024-04-20T17:27:55.249 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.250 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a331fe1e:::benchmark_data_smithi171_122772_object143795:head# 2024-04-20T17:27:55.250 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.250 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3328418:::benchmark_data_smithi171_122772_object107183:head# 2024-04-20T17:27:55.250 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.250 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a332e769:::benchmark_data_smithi171_122772_object78140:head# 2024-04-20T17:27:55.250 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.250 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3343bf7:::benchmark_data_smithi171_122772_object87573:head# 2024-04-20T17:27:55.250 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.250 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3347e86:::benchmark_data_smithi171_122772_object150121:head# 2024-04-20T17:27:55.250 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.250 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a33526dc:::benchmark_data_smithi171_122772_object162107:head# 2024-04-20T17:27:55.250 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.250 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a335299b:::benchmark_data_smithi171_122772_object141789:head# 2024-04-20T17:27:55.251 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.251 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a335a389:::benchmark_data_smithi171_122772_object157408:head# 2024-04-20T17:27:55.251 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.251 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a336c239:::benchmark_data_smithi171_122772_object132676:head# 2024-04-20T17:27:55.251 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.251 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a336e2e8:::benchmark_data_smithi171_122772_object156685:head# 2024-04-20T17:27:55.251 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.251 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3374df8:::benchmark_data_smithi171_122772_object165777:head# 2024-04-20T17:27:55.251 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.251 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3375fa7:::benchmark_data_smithi171_122772_object89594:head# 2024-04-20T17:27:55.251 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.251 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a337bc3a:::benchmark_data_smithi171_122772_object133695:head# 2024-04-20T17:27:55.252 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.252 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a338caaf:::benchmark_data_smithi171_122772_object153367:head# 2024-04-20T17:27:55.252 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.252 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3395adb:::benchmark_data_smithi171_122772_object155614:head# 2024-04-20T17:27:55.252 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.252 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a33a4239:::benchmark_data_smithi171_122772_object163256:head# 2024-04-20T17:27:55.252 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.252 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a33aec41:::benchmark_data_smithi171_122772_object124201:head# 2024-04-20T17:27:55.252 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.252 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a33c3d55:::benchmark_data_smithi171_122772_object93384:head# 2024-04-20T17:27:55.252 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.252 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a33d5f6e:::benchmark_data_smithi171_122772_object154218:head# 2024-04-20T17:27:55.253 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.253 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a33e1453:::benchmark_data_smithi171_122772_object117423:head# 2024-04-20T17:27:55.253 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.253 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a33f929f:::benchmark_data_smithi171_122772_object125104:head# 2024-04-20T17:27:55.253 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.253 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a34014b9:::benchmark_data_smithi171_122772_object94385:head# 2024-04-20T17:27:55.253 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.253 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3410e6d:::benchmark_data_smithi171_122772_object114444:head# 2024-04-20T17:27:55.253 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.253 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a341ca6e:::benchmark_data_smithi171_122772_object80362:head# 2024-04-20T17:27:55.253 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.253 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3427b2b:::benchmark_data_smithi171_122772_object109725:head# 2024-04-20T17:27:55.254 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.254 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3429f8e:::benchmark_data_smithi171_122772_object166593:head# 2024-04-20T17:27:55.254 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.254 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a342f3d1:::benchmark_data_smithi171_122772_object163591:head# 2024-04-20T17:27:55.254 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.254 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a342ff1d:::benchmark_data_smithi171_122772_object131961:head# 2024-04-20T17:27:55.254 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.254 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3430e8f:::benchmark_data_smithi171_122772_object83177:head# 2024-04-20T17:27:55.254 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.254 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a343c3a6:::benchmark_data_smithi171_122772_object82036:head# 2024-04-20T17:27:55.254 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.254 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a34433b2:::benchmark_data_smithi171_122772_object139113:head# 2024-04-20T17:27:55.255 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.255 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a344d497:::benchmark_data_smithi171_122772_object127791:head# 2024-04-20T17:27:55.255 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.255 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a344db57:::benchmark_data_smithi171_122772_object123023:head# 2024-04-20T17:27:55.255 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.255 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3462a73:::benchmark_data_smithi171_122772_object99465:head# 2024-04-20T17:27:55.255 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.255 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3466704:::benchmark_data_smithi171_122772_object112336:head# 2024-04-20T17:27:55.255 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.255 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a34786e7:::benchmark_data_smithi171_122772_object152457:head# 2024-04-20T17:27:55.255 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.256 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:55.277 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:27:55.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:55.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:27:55.301 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a347d889:::benchmark_data_smithi171_122772_object78551:head# 2024-04-20T17:27:55.301 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.301 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3482cc3:::benchmark_data_smithi171_122772_object104174:head# 2024-04-20T17:27:55.302 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.302 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3488f3d:::benchmark_data_smithi171_122772_object122306:head# 2024-04-20T17:27:55.302 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.302 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3490fc5:::benchmark_data_smithi171_122772_object92450:head# 2024-04-20T17:27:55.302 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.302 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3493ade:::benchmark_data_smithi171_122772_object167875:head# 2024-04-20T17:27:55.302 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.302 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a34993bd:::benchmark_data_smithi171_122772_object119098:head# 2024-04-20T17:27:55.302 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.302 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a34b3c1e:::benchmark_data_smithi171_122772_object154840:head# 2024-04-20T17:27:55.302 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.303 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a34b5303:::benchmark_data_smithi171_122772_object169985:head# 2024-04-20T17:27:55.303 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.303 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a34c4750:::benchmark_data_smithi171_122772_object116421:head# 2024-04-20T17:27:55.303 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.303 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a34c52b0:::benchmark_data_smithi171_122772_object109370:head# 2024-04-20T17:27:55.303 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.303 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a34c8912:::benchmark_data_smithi171_122772_object115432:head# 2024-04-20T17:27:55.303 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.303 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a34cf842:::benchmark_data_smithi171_122772_object107536:head# 2024-04-20T17:27:55.303 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.303 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a34e635c:::benchmark_data_smithi171_122772_object144353:head# 2024-04-20T17:27:55.303 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.304 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a34e9946:::benchmark_data_smithi171_122772_object157152:head# 2024-04-20T17:27:55.304 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.304 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a34edceb:::benchmark_data_smithi171_122772_object155723:head# 2024-04-20T17:27:55.304 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.304 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a34f5f00:::benchmark_data_smithi171_122772_object111427:head# 2024-04-20T17:27:55.304 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.304 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a350d0d0:::benchmark_data_smithi171_122772_object153426:head# 2024-04-20T17:27:55.304 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.304 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3519ffa:::benchmark_data_smithi171_122772_object165573:head# 2024-04-20T17:27:55.304 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.304 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a351bd9b:::benchmark_data_smithi171_122772_object146886:head# 2024-04-20T17:27:55.304 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.305 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3529f26:::benchmark_data_smithi171_122772_object168159:head# 2024-04-20T17:27:55.305 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.305 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a35381e6:::benchmark_data_smithi171_122772_object158028:head# 2024-04-20T17:27:55.305 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.305 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3539b16:::benchmark_data_smithi171_122772_object78149:head# 2024-04-20T17:27:55.305 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.305 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a353c746:::benchmark_data_smithi171_122772_object166160:head# 2024-04-20T17:27:55.305 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.305 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a353da2e:::benchmark_data_smithi171_122772_object128564:head# 2024-04-20T17:27:55.305 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.305 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a354591f:::benchmark_data_smithi171_122772_object134179:head# 2024-04-20T17:27:55.306 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.306 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a35596be:::benchmark_data_smithi171_122772_object119683:head# 2024-04-20T17:27:55.306 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.306 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a355eaa9:::benchmark_data_smithi171_122772_object118074:head# 2024-04-20T17:27:55.306 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.306 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a356118f:::benchmark_data_smithi171_122772_object168609:head# 2024-04-20T17:27:55.306 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.306 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3579e24:::benchmark_data_smithi171_122772_object128610:head# 2024-04-20T17:27:55.306 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.306 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a358ac05:::benchmark_data_smithi171_122772_object98017:head# 2024-04-20T17:27:55.306 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.307 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a358c8b2:::benchmark_data_smithi171_122772_object97022:head# 2024-04-20T17:27:55.307 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.307 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3591536:::benchmark_data_smithi171_122772_object143543:head# 2024-04-20T17:27:55.307 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.307 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3595034:::benchmark_data_smithi171_122772_object96468:head# 2024-04-20T17:27:55.307 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.307 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a359ff61:::benchmark_data_smithi171_122772_object114650:head# 2024-04-20T17:27:55.307 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.307 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a35a66a9:::benchmark_data_smithi171_122772_object159582:head# 2024-04-20T17:27:55.307 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.307 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a35c76ef:::benchmark_data_smithi171_122772_object131519:head# 2024-04-20T17:27:55.308 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.308 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a35cbafd:::benchmark_data_smithi171_122772_object85058:head# 2024-04-20T17:27:55.308 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.308 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a35cd0f7:::benchmark_data_smithi171_122772_object102101:head# 2024-04-20T17:27:55.308 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.308 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a35e02c4:::benchmark_data_smithi171_122772_object82800:head# 2024-04-20T17:27:55.308 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.308 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a35e5bb0:::benchmark_data_smithi171_122772_object118605:head# 2024-04-20T17:27:55.308 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.308 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a35e64d7:::benchmark_data_smithi171_122772_object155697:head# 2024-04-20T17:27:55.308 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.308 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a35ec8ab:::benchmark_data_smithi171_122772_object160073:head# 2024-04-20T17:27:55.309 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.309 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a35f809e:::benchmark_data_smithi171_122772_object104137:head# 2024-04-20T17:27:55.309 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.309 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a35ff39e:::benchmark_data_smithi171_122772_object163213:head# 2024-04-20T17:27:55.309 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.309 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a360308f:::benchmark_data_smithi171_122772_object111343:head# 2024-04-20T17:27:55.309 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.309 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3608b6b:::benchmark_data_smithi171_122772_object83747:head# 2024-04-20T17:27:55.309 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.309 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a360a29f:::benchmark_data_smithi171_122772_object123093:head# 2024-04-20T17:27:55.309 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.309 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a361d362:::benchmark_data_smithi171_122772_object163246:head# 2024-04-20T17:27:55.310 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.310 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3627143:::benchmark_data_smithi171_122772_object106236:head# 2024-04-20T17:27:55.310 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.310 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a362f24f:::benchmark_data_smithi171_122772_object115725:head# 2024-04-20T17:27:55.310 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.310 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a36328cb:::benchmark_data_smithi171_122772_object91224:head# 2024-04-20T17:27:55.310 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.310 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3633385:::benchmark_data_smithi171_122772_object164834:head# 2024-04-20T17:27:55.310 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.310 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3640bc3:::benchmark_data_smithi171_122772_object130299:head# 2024-04-20T17:27:55.310 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.311 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a36417e2:::benchmark_data_smithi171_122772_object80060:head# 2024-04-20T17:27:55.311 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.311 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a364391e:::benchmark_data_smithi171_122772_object169344:head# 2024-04-20T17:27:55.311 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.311 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a365a78b:::benchmark_data_smithi171_122772_object78496:head# 2024-04-20T17:27:55.311 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.311 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a365fbee:::benchmark_data_smithi171_122772_object123917:head# 2024-04-20T17:27:55.311 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.311 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a366b6ed:::benchmark_data_smithi171_122772_object78059:head# 2024-04-20T17:27:55.311 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.311 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a367fb74:::benchmark_data_smithi171_122772_object83552:head# 2024-04-20T17:27:55.311 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.311 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a36977aa:::benchmark_data_smithi171_122772_object97643:head# 2024-04-20T17:27:55.312 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.312 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a369a1b9:::benchmark_data_smithi171_122772_object94287:head# 2024-04-20T17:27:55.312 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.312 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a369b465:::benchmark_data_smithi171_122772_object168141:head# 2024-04-20T17:27:55.312 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.312 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a36a58fd:::benchmark_data_smithi171_122772_object155954:head# 2024-04-20T17:27:55.312 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.312 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a36baaf5:::benchmark_data_smithi171_122772_object138837:head# 2024-04-20T17:27:55.312 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.312 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a36bb465:::benchmark_data_smithi171_122772_object91385:head# 2024-04-20T17:27:55.312 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.312 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a36c6f1e:::benchmark_data_smithi171_122772_object114311:head# 2024-04-20T17:27:55.312 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.313 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a36e2b9d:::benchmark_data_smithi171_122772_object117488:head# 2024-04-20T17:27:55.313 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.313 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a36e6ab3:::benchmark_data_smithi171_122772_object164947:head# 2024-04-20T17:27:55.313 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.313 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a370ae7a:::benchmark_data_smithi171_122772_object160430:head# 2024-04-20T17:27:55.313 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.313 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3723ae4:::benchmark_data_smithi171_122772_object104641:head# 2024-04-20T17:27:55.313 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.313 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a37338e2:::benchmark_data_smithi171_122772_object105960:head# 2024-04-20T17:27:55.313 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.313 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3758150:::benchmark_data_smithi171_122772_object152241:head# 2024-04-20T17:27:55.313 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.313 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3780a49:::benchmark_data_smithi171_122772_object86155:head# 2024-04-20T17:27:55.314 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.314 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a379bf98:::benchmark_data_smithi171_122772_object113161:head# 2024-04-20T17:27:55.314 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.314 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a37bac6c:::benchmark_data_smithi171_122772_object129597:head# 2024-04-20T17:27:55.314 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.314 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a37e408a:::benchmark_data_smithi171_122772_object129561:head# 2024-04-20T17:27:55.314 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.314 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a380763f:::benchmark_data_smithi171_122772_object106958:head# 2024-04-20T17:27:55.314 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.314 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a380a492:::benchmark_data_smithi171_122772_object131875:head# 2024-04-20T17:27:55.314 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.314 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a380bca5:::benchmark_data_smithi171_122772_object81677:head# 2024-04-20T17:27:55.315 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.315 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a380e3fa:::benchmark_data_smithi171_122772_object137706:head# 2024-04-20T17:27:55.315 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.315 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a38113eb:::benchmark_data_smithi171_122772_object133480:head# 2024-04-20T17:27:55.315 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.315 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3813e4b:::benchmark_data_smithi171_122772_object159634:head# 2024-04-20T17:27:55.315 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.315 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3815b34:::benchmark_data_smithi171_122772_object104453:head# 2024-04-20T17:27:55.315 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.315 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3815cb4:::benchmark_data_smithi171_122772_object151436:head# 2024-04-20T17:27:55.315 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.315 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a38254a6:::benchmark_data_smithi171_122772_object89723:head# 2024-04-20T17:27:55.315 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.316 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3836f61:::benchmark_data_smithi171_122772_object118505:head# 2024-04-20T17:27:55.316 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.316 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3837d7d:::benchmark_data_smithi171_122772_object166669:head# 2024-04-20T17:27:55.316 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.316 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3838524:::benchmark_data_smithi171_122772_object108468:head# 2024-04-20T17:27:55.316 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.316 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a38396d3:::benchmark_data_smithi171_122772_object129409:head# 2024-04-20T17:27:55.316 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.316 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3840737:::benchmark_data_smithi171_122772_object94585:head# 2024-04-20T17:27:55.316 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.316 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3849804:::benchmark_data_smithi171_122772_object111402:head# 2024-04-20T17:27:55.316 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.316 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a384da85:::benchmark_data_smithi171_122772_object153457:head# 2024-04-20T17:27:55.317 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.317 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3852c3b:::benchmark_data_smithi171_122772_object158682:head# 2024-04-20T17:27:55.317 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.317 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3853f65:::benchmark_data_smithi171_122772_object79984:head# 2024-04-20T17:27:55.317 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.317 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a386c9a6:::benchmark_data_smithi171_122772_object103598:head# 2024-04-20T17:27:55.317 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.317 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3871025:::benchmark_data_smithi171_122772_object161151:head# 2024-04-20T17:27:55.317 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.317 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3881911:::benchmark_data_smithi171_122772_object129338:head# 2024-04-20T17:27:55.317 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.317 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a388405f:::benchmark_data_smithi171_122772_object85400:head# 2024-04-20T17:27:55.317 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.318 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a38874ee:::benchmark_data_smithi171_122772_object106319:head# 2024-04-20T17:27:55.318 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.318 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a388d38f:::benchmark_data_smithi171_122772_object115404:head# 2024-04-20T17:27:55.318 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.318 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a389426c:::benchmark_data_smithi171_122772_object169202:head# 2024-04-20T17:27:55.318 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.318 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a389c76f:::benchmark_data_smithi171_122772_object130513:head# 2024-04-20T17:27:55.318 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.318 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a38a50ad:::benchmark_data_smithi171_122772_object78072:head# 2024-04-20T17:27:55.318 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.318 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a38b733c:::benchmark_data_smithi171_122772_object107532:head# 2024-04-20T17:27:55.319 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.319 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a38c4da3:::benchmark_data_smithi171_122772_object106687:head# 2024-04-20T17:27:55.319 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.319 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a38c8b4d:::benchmark_data_smithi171_122772_object123926:head# 2024-04-20T17:27:55.319 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.319 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a38ea368:::benchmark_data_smithi171_122772_object86054:head# 2024-04-20T17:27:55.319 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.319 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3909b64:::benchmark_data_smithi171_122772_object125181:head# 2024-04-20T17:27:55.319 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.319 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a39194d9:::benchmark_data_smithi171_122772_object88238:head# 2024-04-20T17:27:55.320 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.320 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a392b4e2:::benchmark_data_smithi171_122772_object103455:head# 2024-04-20T17:27:55.320 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.320 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3931822:::benchmark_data_smithi171_122772_object163007:head# 2024-04-20T17:27:55.320 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.320 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3934ca8:::benchmark_data_smithi171_122772_object169986:head# 2024-04-20T17:27:55.320 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.320 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3956867:::benchmark_data_smithi171_122772_object147650:head# 2024-04-20T17:27:55.320 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.320 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3958c9d:::benchmark_data_smithi171_122772_object84330:head# 2024-04-20T17:27:55.320 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.320 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3963a21:::benchmark_data_smithi171_122772_object140100:head# 2024-04-20T17:27:55.321 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.321 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a398ea2f:::benchmark_data_smithi171_122772_object127465:head# 2024-04-20T17:27:55.321 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.321 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3996977:::benchmark_data_smithi171_122772_object108521:head# 2024-04-20T17:27:55.321 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.321 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a399728f:::benchmark_data_smithi171_122772_object167276:head# 2024-04-20T17:27:55.321 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.321 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a399cd0b:::benchmark_data_smithi171_122772_object95363:head# 2024-04-20T17:27:55.321 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.321 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a399e968:::benchmark_data_smithi171_122772_object152936:head# 2024-04-20T17:27:55.322 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.322 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a39d5181:::benchmark_data_smithi171_122772_object140013:head# 2024-04-20T17:27:55.322 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.322 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a39eefe6:::benchmark_data_smithi171_122772_object95230:head# 2024-04-20T17:27:55.322 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.322 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a39f4bf5:::benchmark_data_smithi171_122772_object102976:head# 2024-04-20T17:27:55.322 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.322 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3a0bac7:::benchmark_data_smithi171_122772_object109196:head# 2024-04-20T17:27:55.322 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.323 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3a0cf25:::benchmark_data_smithi171_122772_object92496:head# 2024-04-20T17:27:55.323 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.323 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3a11a0f:::benchmark_data_smithi171_122772_object136095:head# 2024-04-20T17:27:55.323 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.323 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3a19bb4:::benchmark_data_smithi171_122772_object146838:head# 2024-04-20T17:27:55.323 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.323 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3a1debd:::benchmark_data_smithi171_122772_object159315:head# 2024-04-20T17:27:55.323 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.323 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3a30050:::benchmark_data_smithi171_122772_object94922:head# 2024-04-20T17:27:55.323 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.323 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3a32b3c:::benchmark_data_smithi171_122772_object89969:head# 2024-04-20T17:27:55.324 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.324 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3a44b43:::benchmark_data_smithi171_122772_object145354:head# 2024-04-20T17:27:55.324 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.324 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3a64707:::benchmark_data_smithi171_122772_object160679:head# 2024-04-20T17:27:55.324 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.324 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3a7d2d6:::benchmark_data_smithi171_122772_object153910:head# 2024-04-20T17:27:55.324 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.324 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3a8cf00:::benchmark_data_smithi171_122772_object90478:head# 2024-04-20T17:27:55.324 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.324 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3a97b5b:::benchmark_data_smithi171_122772_object158792:head# 2024-04-20T17:27:55.324 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.324 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3a9aced:::benchmark_data_smithi171_122772_object78101:head# 2024-04-20T17:27:55.325 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.325 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3aad8e8:::benchmark_data_smithi171_122772_object132291:head# 2024-04-20T17:27:55.325 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.325 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3aecbed:::benchmark_data_smithi171_122772_object162176:head# 2024-04-20T17:27:55.325 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.325 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3aede8b:::benchmark_data_smithi171_122772_object97251:head# 2024-04-20T17:27:55.325 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.325 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3af0579:::benchmark_data_smithi171_122772_object96245:head# 2024-04-20T17:27:55.325 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.325 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3b0388e:::benchmark_data_smithi171_122772_object134585:head# 2024-04-20T17:27:55.325 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.325 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3b07edb:::benchmark_data_smithi171_122772_object86774:head# 2024-04-20T17:27:55.326 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.326 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3b087a6:::benchmark_data_smithi171_122772_object119006:head# 2024-04-20T17:27:55.326 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.326 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3b0bac9:::benchmark_data_smithi171_122772_object90695:head# 2024-04-20T17:27:55.326 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.326 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3b131ea:::benchmark_data_smithi171_122772_object80004:head# 2024-04-20T17:27:55.326 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.326 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3b358a6:::benchmark_data_smithi171_122772_object167697:head# 2024-04-20T17:27:55.326 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.326 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3b3bf9d:::benchmark_data_smithi171_122772_object131399:head# 2024-04-20T17:27:55.326 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.326 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3b407cb:::benchmark_data_smithi171_122772_object93041:head# 2024-04-20T17:27:55.327 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.327 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3b4f14a:::benchmark_data_smithi171_122772_object120762:head# 2024-04-20T17:27:55.327 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.327 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3b51ef0:::benchmark_data_smithi171_122772_object122149:head# 2024-04-20T17:27:55.327 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.327 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3b55f38:::benchmark_data_smithi171_122772_object161771:head# 2024-04-20T17:27:55.327 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.327 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3b6a494:::benchmark_data_smithi171_122772_object116019:head# 2024-04-20T17:27:55.327 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.327 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3b79553:::benchmark_data_smithi171_122772_object137452:head# 2024-04-20T17:27:55.328 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.328 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3b7bbb2:::benchmark_data_smithi171_122772_object115250:head# 2024-04-20T17:27:55.328 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.328 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3b8f3a3:::benchmark_data_smithi171_122772_object153635:head# 2024-04-20T17:27:55.328 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.328 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3b941bb:::benchmark_data_smithi171_122772_object164135:head# 2024-04-20T17:27:55.328 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.328 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3ba08c6:::benchmark_data_smithi171_122772_object99439:head# 2024-04-20T17:27:55.328 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.328 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3ba9d21:::benchmark_data_smithi171_122772_object165690:head# 2024-04-20T17:27:55.328 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.328 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3bc6869:::benchmark_data_smithi171_122772_object100849:head# 2024-04-20T17:27:55.328 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.329 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3bcc61b:::benchmark_data_smithi171_122772_object89597:head# 2024-04-20T17:27:55.329 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.329 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3bd091a:::benchmark_data_smithi171_122772_object124624:head# 2024-04-20T17:27:55.329 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.329 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3bd5396:::benchmark_data_smithi171_122772_object147513:head# 2024-04-20T17:27:55.329 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.329 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3bd6df2:::benchmark_data_smithi171_122772_object154567:head# 2024-04-20T17:27:55.329 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.329 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3bda17d:::benchmark_data_smithi171_122772_object99435:head# 2024-04-20T17:27:55.329 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.329 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3bdc934:::benchmark_data_smithi171_122772_object134492:head# 2024-04-20T17:27:55.329 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.329 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3be3f05:::benchmark_data_smithi171_122772_object123338:head# 2024-04-20T17:27:55.330 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.330 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3bfd70c:::benchmark_data_smithi171_122772_object107240:head# 2024-04-20T17:27:55.330 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.330 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3c1ffbc:::benchmark_data_smithi171_122772_object150054:head# 2024-04-20T17:27:55.330 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.330 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3c2ee89:::benchmark_data_smithi171_122772_object147369:head# 2024-04-20T17:27:55.330 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.330 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3c3279d:::benchmark_data_smithi171_122772_object122017:head# 2024-04-20T17:27:55.330 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.330 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3c4b170:::benchmark_data_smithi171_122772_object168629:head# 2024-04-20T17:27:55.331 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.331 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3c4b2b9:::benchmark_data_smithi171_122772_object128843:head# 2024-04-20T17:27:55.331 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.331 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3c657a1:::benchmark_data_smithi171_122772_object100271:head# 2024-04-20T17:27:55.331 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.331 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3c767d6:::benchmark_data_smithi171_122772_object152623:head# 2024-04-20T17:27:55.331 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.331 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3c7b4b6:::benchmark_data_smithi171_122772_object97544:head# 2024-04-20T17:27:55.331 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.332 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3c83f92:::benchmark_data_smithi171_122772_object129068:head# 2024-04-20T17:27:55.332 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.332 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3c856e2:::benchmark_data_smithi171_122772_object149461:head# 2024-04-20T17:27:55.332 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.332 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3c858af:::benchmark_data_smithi171_122772_object92877:head# 2024-04-20T17:27:55.332 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.332 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3c8a3b2:::benchmark_data_smithi171_122772_object121207:head# 2024-04-20T17:27:55.332 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.332 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3c93831:::benchmark_data_smithi171_122772_object133539:head# 2024-04-20T17:27:55.332 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.332 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3c99f79:::benchmark_data_smithi171_122772_object145091:head# 2024-04-20T17:27:55.333 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.333 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3c9fc12:::benchmark_data_smithi171_122772_object162999:head# 2024-04-20T17:27:55.333 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.333 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3ca92af:::benchmark_data_smithi171_122772_object92940:head# 2024-04-20T17:27:55.333 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.333 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3cab72f:::benchmark_data_smithi171_122772_object87298:head# 2024-04-20T17:27:55.333 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.333 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3cd0a5e:::benchmark_data_smithi171_122772_object168062:head# 2024-04-20T17:27:55.333 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.333 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3cd33d3:::benchmark_data_smithi171_122772_object156439:head# 2024-04-20T17:27:55.333 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.333 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3cdd664:::benchmark_data_smithi171_122772_object138999:head# 2024-04-20T17:27:55.334 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.334 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3ceaa76:::benchmark_data_smithi171_122772_object89157:head# 2024-04-20T17:27:55.334 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.334 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3cee311:::benchmark_data_smithi171_122772_object99414:head# 2024-04-20T17:27:55.334 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.334 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3cf0deb:::benchmark_data_smithi171_122772_object118272:head# 2024-04-20T17:27:55.334 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.334 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3cfd141:::benchmark_data_smithi171_122772_object114382:head# 2024-04-20T17:27:55.334 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.335 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3d062f0:::benchmark_data_smithi171_122772_object102614:head# 2024-04-20T17:27:55.335 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.335 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3d0f043:::benchmark_data_smithi171_122772_object84129:head# 2024-04-20T17:27:55.335 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.335 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3d1c9ed:::benchmark_data_smithi171_122772_object114704:head# 2024-04-20T17:27:55.335 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.335 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3d2a427:::benchmark_data_smithi171_122772_object152208:head# 2024-04-20T17:27:55.335 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.335 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3d2b9e4:::benchmark_data_smithi171_122772_object165055:head# 2024-04-20T17:27:55.335 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.336 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3d332f2:::benchmark_data_smithi171_122772_object154898:head# 2024-04-20T17:27:55.336 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.336 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3d3aba8:::benchmark_data_smithi171_122772_object140057:head# 2024-04-20T17:27:55.336 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.336 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3d3c9c7:::benchmark_data_smithi171_122772_object153173:head# 2024-04-20T17:27:55.336 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.336 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3d3fb48:::benchmark_data_smithi171_122772_object124603:head# 2024-04-20T17:27:55.336 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.336 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3d4296d:::benchmark_data_smithi171_122772_object107570:head# 2024-04-20T17:27:55.337 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.337 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3d52b6c:::benchmark_data_smithi171_122772_object131993:head# 2024-04-20T17:27:55.337 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.337 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3d61208:::benchmark_data_smithi171_122772_object146952:head# 2024-04-20T17:27:55.337 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.337 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3d64248:::benchmark_data_smithi171_122772_object107836:head# 2024-04-20T17:27:55.337 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.337 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3d665e7:::benchmark_data_smithi171_122772_object141671:head# 2024-04-20T17:27:55.337 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.337 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3d6f0b4:::benchmark_data_smithi171_122772_object139949:head# 2024-04-20T17:27:55.338 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.338 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3d7a0d3:::benchmark_data_smithi171_122772_object143440:head# 2024-04-20T17:27:55.338 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.338 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3d7ab48:::benchmark_data_smithi171_122772_object155608:head# 2024-04-20T17:27:55.338 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.338 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3d7b60c:::benchmark_data_smithi171_122772_object169504:head# 2024-04-20T17:27:55.338 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.338 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3d81e1f:::benchmark_data_smithi171_122772_object148263:head# 2024-04-20T17:27:55.338 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.339 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3d8dd35:::benchmark_data_smithi171_122772_object97308:head# 2024-04-20T17:27:55.339 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.339 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3d999a7:::benchmark_data_smithi171_122772_object150851:head# 2024-04-20T17:27:55.339 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.339 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3da18b7:::benchmark_data_smithi171_122772_object127456:head# 2024-04-20T17:27:55.339 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.339 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3db49af:::benchmark_data_smithi171_122772_object144719:head# 2024-04-20T17:27:55.339 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.339 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3db5a9f:::benchmark_data_smithi171_122772_object130507:head# 2024-04-20T17:27:55.339 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.339 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3dbae06:::benchmark_data_smithi171_122772_object123101:head# 2024-04-20T17:27:55.340 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.340 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3dc2743:::benchmark_data_smithi171_122772_object78602:head# 2024-04-20T17:27:55.340 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.340 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3dc38f9:::benchmark_data_smithi171_122772_object147091:head# 2024-04-20T17:27:55.340 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.340 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3dcd581:::benchmark_data_smithi171_122772_object143936:head# 2024-04-20T17:27:55.340 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.340 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3dd6415:::benchmark_data_smithi171_122772_object99032:head# 2024-04-20T17:27:55.340 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.341 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3ddd4ca:::benchmark_data_smithi171_122772_object134209:head# 2024-04-20T17:27:55.341 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.341 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3dee3f0:::benchmark_data_smithi171_122772_object141679:head# 2024-04-20T17:27:55.341 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.341 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3df6cfe:::benchmark_data_smithi171_122772_object118512:head# 2024-04-20T17:27:55.341 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.341 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3e05468:::benchmark_data_smithi171_122772_object105293:head# 2024-04-20T17:27:55.341 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.341 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3e0a040:::benchmark_data_smithi171_122772_object152938:head# 2024-04-20T17:27:55.341 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.342 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3e12699:::benchmark_data_smithi171_122772_object139346:head# 2024-04-20T17:27:55.342 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.342 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3e1aa2b:::benchmark_data_smithi171_122772_object159510:head# 2024-04-20T17:27:55.342 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.342 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3e1f1b1:::benchmark_data_smithi171_122772_object141647:head# 2024-04-20T17:27:55.342 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.342 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3e36ffc:::benchmark_data_smithi171_122772_object135598:head# 2024-04-20T17:27:55.342 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.342 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3e432b3:::benchmark_data_smithi171_122772_object84103:head# 2024-04-20T17:27:55.343 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.343 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3e5aa6c:::benchmark_data_smithi171_122772_object138820:head# 2024-04-20T17:27:55.343 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.343 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3e5d947:::benchmark_data_smithi171_122772_object169712:head# 2024-04-20T17:27:55.343 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.343 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3e5e177:::benchmark_data_smithi171_122772_object89352:head# 2024-04-20T17:27:55.343 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.343 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3e5ea12:::benchmark_data_smithi171_122772_object133867:head# 2024-04-20T17:27:55.343 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.344 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3e6529a:::benchmark_data_smithi171_122772_object140499:head# 2024-04-20T17:27:55.344 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.344 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3e6867e:::benchmark_data_smithi171_122772_object81537:head# 2024-04-20T17:27:55.344 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.344 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3e80c2f:::benchmark_data_smithi171_122772_object144943:head# 2024-04-20T17:27:55.344 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.344 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3e906d3:::benchmark_data_smithi171_122772_object160363:head# 2024-04-20T17:27:55.344 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.344 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3eb2080:::benchmark_data_smithi171_122772_object78833:head# 2024-04-20T17:27:55.345 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.345 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3eb99cd:::benchmark_data_smithi171_122772_object109699:head# 2024-04-20T17:27:55.345 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.345 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3ebea71:::benchmark_data_smithi171_122772_object131143:head# 2024-04-20T17:27:55.345 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.345 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3ec5d10:::benchmark_data_smithi171_122772_object166624:head# 2024-04-20T17:27:55.345 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.345 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3eca2b6:::benchmark_data_smithi171_122772_object167894:head# 2024-04-20T17:27:55.345 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.346 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3edeb59:::benchmark_data_smithi171_122772_object99854:head# 2024-04-20T17:27:55.346 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.346 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3ee6971:::benchmark_data_smithi171_122772_object133916:head# 2024-04-20T17:27:55.346 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.346 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3eeee04:::benchmark_data_smithi171_122772_object130638:head# 2024-04-20T17:27:55.346 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.346 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3ef0a64:::benchmark_data_smithi171_122772_object86674:head# 2024-04-20T17:27:55.346 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.346 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3ef84cd:::benchmark_data_smithi171_122772_object163292:head# 2024-04-20T17:27:55.347 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.347 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3f113f2:::benchmark_data_smithi171_122772_object133604:head# 2024-04-20T17:27:55.347 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.347 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3f11ddc:::benchmark_data_smithi171_122772_object144521:head# 2024-04-20T17:27:55.347 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.347 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3f1bb22:::benchmark_data_smithi171_122772_object120462:head# 2024-04-20T17:27:55.347 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.347 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3f21e27:::benchmark_data_smithi171_122772_object128562:head# 2024-04-20T17:27:55.347 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.347 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3f21fef:::benchmark_data_smithi171_122772_object95294:head# 2024-04-20T17:27:55.347 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.348 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3f2f652:::benchmark_data_smithi171_122772_object99927:head# 2024-04-20T17:27:55.348 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.348 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3f30657:::benchmark_data_smithi171_122772_object89155:head# 2024-04-20T17:27:55.348 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.348 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3f39996:::benchmark_data_smithi171_122772_object146135:head# 2024-04-20T17:27:55.348 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.348 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3f3cfeb:::benchmark_data_smithi171_122772_object105339:head# 2024-04-20T17:27:55.348 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.348 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3f4f3d4:::benchmark_data_smithi171_122772_object130680:head# 2024-04-20T17:27:55.348 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.349 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3f62998:::benchmark_data_smithi171_122772_object162075:head# 2024-04-20T17:27:55.349 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.349 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3f712ee:::benchmark_data_smithi171_122772_object154409:head# 2024-04-20T17:27:55.349 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.349 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3f74595:::benchmark_data_smithi171_122772_object139190:head# 2024-04-20T17:27:55.349 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.349 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3f74db1:::benchmark_data_smithi171_122772_object86240:head# 2024-04-20T17:27:55.349 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.349 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3f7efd8:::benchmark_data_smithi171_122772_object118003:head# 2024-04-20T17:27:55.349 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.350 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3f8afc3:::benchmark_data_smithi171_122772_object161917:head# 2024-04-20T17:27:55.350 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.350 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3f8c64f:::benchmark_data_smithi171_122772_object99313:head# 2024-04-20T17:27:55.350 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.350 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3fa306e:::benchmark_data_smithi171_122772_object142136:head# 2024-04-20T17:27:55.350 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.350 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3fb106f:::benchmark_data_smithi171_122772_object129201:head# 2024-04-20T17:27:55.350 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.350 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3fb2590:::benchmark_data_smithi171_122772_object119037:head# 2024-04-20T17:27:55.351 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.351 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3fb782b:::benchmark_data_smithi171_122772_object135093:head# 2024-04-20T17:27:55.351 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.351 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3fbc888:::benchmark_data_smithi171_122772_object111312:head# 2024-04-20T17:27:55.351 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.351 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3fcdd08:::benchmark_data_smithi171_122772_object131882:head# 2024-04-20T17:27:55.351 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.351 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3fd0640:::benchmark_data_smithi171_122772_object148971:head# 2024-04-20T17:27:55.351 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.351 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3fe5827:::benchmark_data_smithi171_122772_object168691:head# 2024-04-20T17:27:55.352 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.352 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3fe5fea:::benchmark_data_smithi171_122772_object98798:head# 2024-04-20T17:27:55.352 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.352 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3fe6203:::benchmark_data_smithi171_122772_object148205:head# 2024-04-20T17:27:55.352 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.352 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3fead3e:::benchmark_data_smithi171_122772_object97283:head# 2024-04-20T17:27:55.352 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.352 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a3feea24:::benchmark_data_smithi171_122772_object133270:head# 2024-04-20T17:27:55.352 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.352 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4000638:::benchmark_data_smithi171_122772_object123230:head# 2024-04-20T17:27:55.353 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.353 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a40007f0:::benchmark_data_smithi171_122772_object86958:head# 2024-04-20T17:27:55.353 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.353 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a40064d8:::benchmark_data_smithi171_122772_object159783:head# 2024-04-20T17:27:55.353 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.353 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4006fb0:::benchmark_data_smithi171_122772_object107197:head# 2024-04-20T17:27:55.353 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.353 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4009708:::benchmark_data_smithi171_122772_object154781:head# 2024-04-20T17:27:55.353 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.353 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4014733:::benchmark_data_smithi171_122772_object159633:head# 2024-04-20T17:27:55.353 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.353 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a401a92e:::benchmark_data_smithi171_122772_object116947:head# 2024-04-20T17:27:55.354 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.354 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a402d46d:::benchmark_data_smithi171_122772_object123853:head# 2024-04-20T17:27:55.354 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.354 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4035bb7:::benchmark_data_smithi171_122772_object127827:head# 2024-04-20T17:27:55.354 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.354 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a403f4fc:::benchmark_data_smithi171_122772_object134967:head# 2024-04-20T17:27:55.354 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.354 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a404337d:::benchmark_data_smithi171_122772_object155992:head# 2024-04-20T17:27:55.354 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.354 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a40449c3:::benchmark_data_smithi171_122772_object134660:head# 2024-04-20T17:27:55.354 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.355 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4045368:::benchmark_data_smithi171_122772_object82768:head# 2024-04-20T17:27:55.355 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.355 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a404e332:::benchmark_data_smithi171_122772_object81235:head# 2024-04-20T17:27:55.355 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.355 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4063883:::benchmark_data_smithi171_122772_object114338:head# 2024-04-20T17:27:55.355 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.355 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a406571b:::benchmark_data_smithi171_122772_object78542:head# 2024-04-20T17:27:55.355 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.355 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a406ad14:::benchmark_data_smithi171_122772_object122978:head# 2024-04-20T17:27:55.355 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.355 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a406c49a:::benchmark_data_smithi171_122772_object136692:head# 2024-04-20T17:27:55.356 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.356 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a407257a:::benchmark_data_smithi171_122772_object151081:head# 2024-04-20T17:27:55.356 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.356 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a407d5a5:::benchmark_data_smithi171_122772_object128024:head# 2024-04-20T17:27:55.356 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.356 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a40939a4:::benchmark_data_smithi171_122772_object106991:head# 2024-04-20T17:27:55.356 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.356 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4095cb1:::benchmark_data_smithi171_122772_object148824:head# 2024-04-20T17:27:55.356 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.356 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4096263:::benchmark_data_smithi171_122772_object97191:head# 2024-04-20T17:27:55.356 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.356 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a40a308b:::benchmark_data_smithi171_122772_object164804:head# 2024-04-20T17:27:55.357 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.357 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a40abb35:::benchmark_data_smithi171_122772_object145082:head# 2024-04-20T17:27:55.357 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.357 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a40b19bd:::benchmark_data_smithi171_122772_object113103:head# 2024-04-20T17:27:55.357 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.357 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a40b267a:::benchmark_data_smithi171_122772_object134525:head# 2024-04-20T17:27:55.357 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.357 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a40b9b1c:::benchmark_data_smithi171_122772_object104136:head# 2024-04-20T17:27:55.357 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.357 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a40c74b6:::benchmark_data_smithi171_122772_object93248:head# 2024-04-20T17:27:55.357 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.357 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a40cd321:::benchmark_data_smithi171_122772_object95558:head# 2024-04-20T17:27:55.357 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.358 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a40d06ad:::benchmark_data_smithi171_122772_object103098:head# 2024-04-20T17:27:55.358 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.358 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a40e33e8:::benchmark_data_smithi171_122772_object140359:head# 2024-04-20T17:27:55.358 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.358 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a40fc8d6:::benchmark_data_smithi171_122772_object162411:head# 2024-04-20T17:27:55.358 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.358 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4106fcb:::benchmark_data_smithi171_122772_object143383:head# 2024-04-20T17:27:55.358 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.399 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:27:55.441 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a41070e7:::benchmark_data_smithi171_122772_object114713:head# 2024-04-20T17:27:55.441 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.441 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a411d095:::benchmark_data_smithi171_122772_object113211:head# 2024-04-20T17:27:55.441 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.441 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a412201c:::benchmark_data_smithi171_122772_object99697:head# 2024-04-20T17:27:55.441 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.441 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a412574b:::benchmark_data_smithi171_122772_object86983:head# 2024-04-20T17:27:55.441 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.441 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a412cce0:::benchmark_data_smithi171_122772_object121972:head# 2024-04-20T17:27:55.442 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.442 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a412cef9:::benchmark_data_smithi171_122772_object120868:head# 2024-04-20T17:27:55.442 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.442 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a413dadb:::benchmark_data_smithi171_122772_object111588:head# 2024-04-20T17:27:55.442 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.442 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a414d888:::benchmark_data_smithi171_122772_object89090:head# 2024-04-20T17:27:55.442 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.442 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4150d14:::benchmark_data_smithi171_122772_object167055:head# 2024-04-20T17:27:55.442 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.442 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4161378:::benchmark_data_smithi171_122772_object143925:head# 2024-04-20T17:27:55.443 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.443 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4168e86:::benchmark_data_smithi171_122772_object151229:head# 2024-04-20T17:27:55.443 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.443 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a416ea72:::benchmark_data_smithi171_122772_object84152:head# 2024-04-20T17:27:55.443 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.443 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4170314:::benchmark_data_smithi171_122772_object151840:head# 2024-04-20T17:27:55.443 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.443 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4174576:::benchmark_data_smithi171_122772_object98636:head# 2024-04-20T17:27:55.443 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.443 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4175ae2:::benchmark_data_smithi171_122772_object124224:head# 2024-04-20T17:27:55.443 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.444 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a417ee0e:::benchmark_data_smithi171_122772_object84571:head# 2024-04-20T17:27:55.444 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.444 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4185d8a:::benchmark_data_smithi171_122772_object144836:head# 2024-04-20T17:27:55.444 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.444 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a418b66c:::benchmark_data_smithi171_122772_object139703:head# 2024-04-20T17:27:55.444 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.444 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a419105f:::benchmark_data_smithi171_122772_object138213:head# 2024-04-20T17:27:55.444 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.444 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4192bf6:::benchmark_data_smithi171_122772_object150209:head# 2024-04-20T17:27:55.444 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.444 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a41a3756:::benchmark_data_smithi171_122772_object116305:head# 2024-04-20T17:27:55.444 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.445 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a41a5810:::benchmark_data_smithi171_122772_object136408:head# 2024-04-20T17:27:55.445 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.445 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a41b7a74:::benchmark_data_smithi171_122772_object128658:head# 2024-04-20T17:27:55.445 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.445 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a41c1e9b:::benchmark_data_smithi171_122772_object134591:head# 2024-04-20T17:27:55.445 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.445 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a41c4608:::benchmark_data_smithi171_122772_object140223:head# 2024-04-20T17:27:55.445 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.445 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a41cc82c:::benchmark_data_smithi171_122772_object163885:head# 2024-04-20T17:27:55.445 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.445 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a41d6b18:::benchmark_data_smithi171_122772_object159026:head# 2024-04-20T17:27:55.445 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.446 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a41d6df5:::benchmark_data_smithi171_122772_object114061:head# 2024-04-20T17:27:55.446 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.446 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a41dfbf1:::benchmark_data_smithi171_122772_object118560:head# 2024-04-20T17:27:55.446 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.446 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a41e5d44:::benchmark_data_smithi171_122772_object82190:head# 2024-04-20T17:27:55.446 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.446 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a41edf69:::benchmark_data_smithi171_122772_object80866:head# 2024-04-20T17:27:55.446 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.446 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a41f115d:::benchmark_data_smithi171_122772_object144934:head# 2024-04-20T17:27:55.446 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.446 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4229047:::benchmark_data_smithi171_122772_object87346:head# 2024-04-20T17:27:55.447 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.447 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a422f15f:::benchmark_data_smithi171_122772_object102025:head# 2024-04-20T17:27:55.447 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.447 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a42323a9:::benchmark_data_smithi171_122772_object112967:head# 2024-04-20T17:27:55.447 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.447 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4234adc:::benchmark_data_smithi171_122772_object92675:head# 2024-04-20T17:27:55.447 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.447 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4246795:::benchmark_data_smithi171_122772_object95436:head# 2024-04-20T17:27:55.447 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.447 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a426338e:::benchmark_data_smithi171_122772_object112616:head# 2024-04-20T17:27:55.447 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.447 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a426db7b:::benchmark_data_smithi171_122772_object125319:head# 2024-04-20T17:27:55.448 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.448 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a426fcd1:::benchmark_data_smithi171_122772_object107655:head# 2024-04-20T17:27:55.448 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.448 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a42700db:::benchmark_data_smithi171_122772_object121868:head# 2024-04-20T17:27:55.448 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.448 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4272610:::benchmark_data_smithi171_122772_object149594:head# 2024-04-20T17:27:55.448 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.448 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a427440d:::benchmark_data_smithi171_122772_object142596:head# 2024-04-20T17:27:55.448 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.448 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4278f2d:::benchmark_data_smithi171_122772_object156315:head# 2024-04-20T17:27:55.448 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.448 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4279249:::benchmark_data_smithi171_122772_object156568:head# 2024-04-20T17:27:55.448 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.449 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4287fbd:::benchmark_data_smithi171_122772_object90383:head# 2024-04-20T17:27:55.449 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.449 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4294b8e:::benchmark_data_smithi171_122772_object131242:head# 2024-04-20T17:27:55.449 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.449 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4296a7f:::benchmark_data_smithi171_122772_object129117:head# 2024-04-20T17:27:55.449 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.449 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a429fe80:::benchmark_data_smithi171_122772_object103830:head# 2024-04-20T17:27:55.449 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.449 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a42b7eb4:::benchmark_data_smithi171_122772_object110106:head# 2024-04-20T17:27:55.449 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.449 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a42c0fcb:::benchmark_data_smithi171_122772_object109593:head# 2024-04-20T17:27:55.450 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.450 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a42d78bb:::benchmark_data_smithi171_122772_object145961:head# 2024-04-20T17:27:55.450 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.450 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a42f1601:::benchmark_data_smithi171_122772_object164603:head# 2024-04-20T17:27:55.450 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.450 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4304cd4:::benchmark_data_smithi171_122772_object152507:head# 2024-04-20T17:27:55.450 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.450 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4315687:::benchmark_data_smithi171_122772_object122830:head# 2024-04-20T17:27:55.450 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.450 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a432415c:::benchmark_data_smithi171_122772_object133906:head# 2024-04-20T17:27:55.450 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.450 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4330921:::benchmark_data_smithi171_122772_object97647:head# 2024-04-20T17:27:55.451 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.451 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a43596d9:::benchmark_data_smithi171_122772_object156068:head# 2024-04-20T17:27:55.451 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.451 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a435d07f:::benchmark_data_smithi171_122772_object106829:head# 2024-04-20T17:27:55.451 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.451 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a435e955:::benchmark_data_smithi171_122772_object128348:head# 2024-04-20T17:27:55.451 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.451 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4361d1a:::benchmark_data_smithi171_122772_object96909:head# 2024-04-20T17:27:55.451 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.451 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4366297:::benchmark_data_smithi171_122772_object158175:head# 2024-04-20T17:27:55.451 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.451 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a43763cc:::benchmark_data_smithi171_122772_object92305:head# 2024-04-20T17:27:55.452 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.452 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4388bd0:::benchmark_data_smithi171_122772_object140293:head# 2024-04-20T17:27:55.452 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.452 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a438b23b:::benchmark_data_smithi171_122772_object136969:head# 2024-04-20T17:27:55.452 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.452 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a439cfa2:::benchmark_data_smithi171_122772_object122544:head# 2024-04-20T17:27:55.452 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.452 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a43aade4:::benchmark_data_smithi171_122772_object112371:head# 2024-04-20T17:27:55.452 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.452 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a43eafb9:::benchmark_data_smithi171_122772_object99714:head# 2024-04-20T17:27:55.452 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.452 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a440753d:::benchmark_data_smithi171_122772_object90005:head# 2024-04-20T17:27:55.453 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.453 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4407a88:::benchmark_data_smithi171_122772_object111307:head# 2024-04-20T17:27:55.453 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.453 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a440bf4d:::benchmark_data_smithi171_122772_object142635:head# 2024-04-20T17:27:55.453 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.453 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a441783c:::benchmark_data_smithi171_122772_object141846:head# 2024-04-20T17:27:55.453 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.453 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a441d3de:::benchmark_data_smithi171_122772_object146818:head# 2024-04-20T17:27:55.453 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.453 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a442fc04:::benchmark_data_smithi171_122772_object85686:head# 2024-04-20T17:27:55.453 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.453 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a444175f:::benchmark_data_smithi171_122772_object136514:head# 2024-04-20T17:27:55.454 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.454 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a44456cf:::benchmark_data_smithi171_122772_object109671:head# 2024-04-20T17:27:55.454 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.454 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a444715c:::benchmark_data_smithi171_122772_object138780:head# 2024-04-20T17:27:55.454 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.454 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a444a8f8:::benchmark_data_smithi171_122772_object110736:head# 2024-04-20T17:27:55.454 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.454 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a446920d:::benchmark_data_smithi171_122772_object162569:head# 2024-04-20T17:27:55.454 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.454 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a447b4d6:::benchmark_data_smithi171_122772_object78686:head# 2024-04-20T17:27:55.454 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.454 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a447bca8:::benchmark_data_smithi171_122772_object126489:head# 2024-04-20T17:27:55.455 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.455 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a447cd59:::benchmark_data_smithi171_122772_object121721:head# 2024-04-20T17:27:55.455 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.455 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a44e12af:::benchmark_data_smithi171_122772_object135504:head# 2024-04-20T17:27:55.455 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.455 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a44eb070:::benchmark_data_smithi171_122772_object128767:head# 2024-04-20T17:27:55.455 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.455 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4500e9c:::benchmark_data_smithi171_122772_object83906:head# 2024-04-20T17:27:55.455 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.455 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4505460:::benchmark_data_smithi171_122772_object159015:head# 2024-04-20T17:27:55.455 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.455 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4505474:::benchmark_data_smithi171_122772_object85811:head# 2024-04-20T17:27:55.456 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.456 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a45097fe:::benchmark_data_smithi171_122772_object164000:head# 2024-04-20T17:27:55.456 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.456 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a450c60c:::benchmark_data_smithi171_122772_object123409:head# 2024-04-20T17:27:55.456 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.456 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a450c69f:::benchmark_data_smithi171_122772_object80439:head# 2024-04-20T17:27:55.456 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.456 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a451f7b0:::benchmark_data_smithi171_122772_object92554:head# 2024-04-20T17:27:55.456 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.456 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4529729:::benchmark_data_smithi171_122772_object92555:head# 2024-04-20T17:27:55.456 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.456 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a452d0a0:::benchmark_data_smithi171_122772_object111499:head# 2024-04-20T17:27:55.456 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.457 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4554fc9:::benchmark_data_smithi171_122772_object134159:head# 2024-04-20T17:27:55.457 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.457 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4556bca:::benchmark_data_smithi171_122772_object113284:head# 2024-04-20T17:27:55.457 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.457 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4563599:::benchmark_data_smithi171_122772_object92188:head# 2024-04-20T17:27:55.457 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.457 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4565b0d:::benchmark_data_smithi171_122772_object116178:head# 2024-04-20T17:27:55.457 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.457 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4589dbb:::benchmark_data_smithi171_122772_object161365:head# 2024-04-20T17:27:55.457 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.457 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a45a17c8:::benchmark_data_smithi171_122772_object123427:head# 2024-04-20T17:27:55.457 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.458 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a45ae382:::benchmark_data_smithi171_122772_object146879:head# 2024-04-20T17:27:55.458 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.458 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a45bf0a2:::benchmark_data_smithi171_122772_object78643:head# 2024-04-20T17:27:55.458 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.458 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a45c0291:::benchmark_data_smithi171_122772_object101029:head# 2024-04-20T17:27:55.458 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.458 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a45c75df:::benchmark_data_smithi171_122772_object145853:head# 2024-04-20T17:27:55.458 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.458 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a45c7739:::benchmark_data_smithi171_122772_object90628:head# 2024-04-20T17:27:55.458 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.458 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a45cc7d3:::benchmark_data_smithi171_122772_object90670:head# 2024-04-20T17:27:55.458 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.459 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a45de603:::benchmark_data_smithi171_122772_object146539:head# 2024-04-20T17:27:55.459 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.459 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a45ec03b:::benchmark_data_smithi171_122772_object110915:head# 2024-04-20T17:27:55.459 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.459 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a45fc3ff:::benchmark_data_smithi171_122772_object86449:head# 2024-04-20T17:27:55.459 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.459 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4603690:::benchmark_data_smithi171_122772_object169825:head# 2024-04-20T17:27:55.459 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.459 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a46052b5:::benchmark_data_smithi171_122772_object99863:head# 2024-04-20T17:27:55.459 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.459 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4609142:::benchmark_data_smithi171_122772_object110547:head# 2024-04-20T17:27:55.459 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.460 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a460f6b4:::benchmark_data_smithi171_122772_object80918:head# 2024-04-20T17:27:55.460 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.460 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a461587a:::benchmark_data_smithi171_122772_object100936:head# 2024-04-20T17:27:55.460 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.460 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a461e589:::benchmark_data_smithi171_122772_object140279:head# 2024-04-20T17:27:55.460 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.460 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a461e799:::benchmark_data_smithi171_122772_object164440:head# 2024-04-20T17:27:55.460 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.460 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a462d6d0:::benchmark_data_smithi171_122772_object130754:head# 2024-04-20T17:27:55.460 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.460 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a462ff7c:::benchmark_data_smithi171_122772_object101247:head# 2024-04-20T17:27:55.460 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.460 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a46735b1:::benchmark_data_smithi171_122772_object140200:head# 2024-04-20T17:27:55.461 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.461 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4677646:::benchmark_data_smithi171_122772_object103927:head# 2024-04-20T17:27:55.461 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.461 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a467a6db:::benchmark_data_smithi171_122772_object139632:head# 2024-04-20T17:27:55.461 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.461 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a46880e8:::benchmark_data_smithi171_122772_object78839:head# 2024-04-20T17:27:55.461 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.461 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a469c1a2:::benchmark_data_smithi171_122772_object124161:head# 2024-04-20T17:27:55.461 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.461 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a46a183e:::benchmark_data_smithi171_122772_object129891:head# 2024-04-20T17:27:55.461 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.461 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a46a918d:::benchmark_data_smithi171_122772_object152970:head# 2024-04-20T17:27:55.462 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.462 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a46a9455:::benchmark_data_smithi171_122772_object83244:head# 2024-04-20T17:27:55.462 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.462 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a46c8851:::benchmark_data_smithi171_122772_object152755:head# 2024-04-20T17:27:55.462 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.462 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a46c8f21:::benchmark_data_smithi171_122772_object123692:head# 2024-04-20T17:27:55.462 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.462 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a46d31af:::benchmark_data_smithi171_122772_object112172:head# 2024-04-20T17:27:55.462 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.462 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a46da395:::benchmark_data_smithi171_122772_object115705:head# 2024-04-20T17:27:55.462 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.462 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a46e805f:::benchmark_data_smithi171_122772_object148357:head# 2024-04-20T17:27:55.463 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.463 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a46f7013:::benchmark_data_smithi171_122772_object164478:head# 2024-04-20T17:27:55.463 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.463 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a46f8f92:::benchmark_data_smithi171_122772_object151827:head# 2024-04-20T17:27:55.463 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.463 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a47009fe:::benchmark_data_smithi171_122772_object114493:head# 2024-04-20T17:27:55.463 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.463 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a470f657:::benchmark_data_smithi171_122772_object93953:head# 2024-04-20T17:27:55.463 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.463 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a47104b5:::benchmark_data_smithi171_122772_object82412:head# 2024-04-20T17:27:55.463 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.463 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4714f48:::benchmark_data_smithi171_122772_object109190:head# 2024-04-20T17:27:55.464 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.464 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4718963:::benchmark_data_smithi171_122772_object100141:head# 2024-04-20T17:27:55.464 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.464 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a471fdce:::benchmark_data_smithi171_122772_object149759:head# 2024-04-20T17:27:55.464 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.464 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4724397:::benchmark_data_smithi171_122772_object100330:head# 2024-04-20T17:27:55.464 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.464 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4725f3a:::benchmark_data_smithi171_122772_object91924:head# 2024-04-20T17:27:55.464 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.464 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a472829c:::benchmark_data_smithi171_122772_object94118:head# 2024-04-20T17:27:55.464 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.465 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4728c8a:::benchmark_data_smithi171_122772_object164922:head# 2024-04-20T17:27:55.465 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.465 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a472d00f:::benchmark_data_smithi171_122772_object84199:head# 2024-04-20T17:27:55.465 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.465 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a47307fa:::benchmark_data_smithi171_122772_object158996:head# 2024-04-20T17:27:55.465 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.465 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4744d0f:::benchmark_data_smithi171_122772_object118656:head# 2024-04-20T17:27:55.465 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.465 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a47561d1:::benchmark_data_smithi171_122772_object85319:head# 2024-04-20T17:27:55.465 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.465 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a476b9a0:::benchmark_data_smithi171_122772_object110552:head# 2024-04-20T17:27:55.465 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.465 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a477a009:::benchmark_data_smithi171_122772_object96880:head# 2024-04-20T17:27:55.466 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.466 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4787748:::benchmark_data_smithi171_122772_object82293:head# 2024-04-20T17:27:55.466 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.466 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4789daf:::benchmark_data_smithi171_122772_object98450:head# 2024-04-20T17:27:55.466 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.466 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a47926de:::benchmark_data_smithi171_122772_object88310:head# 2024-04-20T17:27:55.466 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.466 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a479abf5:::benchmark_data_smithi171_122772_object83791:head# 2024-04-20T17:27:55.466 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.466 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a47a8ec2:::benchmark_data_smithi171_122772_object115748:head# 2024-04-20T17:27:55.466 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.466 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a47b7612:::benchmark_data_smithi171_122772_object120191:head# 2024-04-20T17:27:55.467 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.467 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a47b8cae:::benchmark_data_smithi171_122772_object147036:head# 2024-04-20T17:27:55.467 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.467 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a47be47e:::benchmark_data_smithi171_122772_object140892:head# 2024-04-20T17:27:55.467 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.467 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a47c6e17:::benchmark_data_smithi171_122772_object94122:head# 2024-04-20T17:27:55.467 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.467 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a47f4215:::benchmark_data_smithi171_122772_object109078:head# 2024-04-20T17:27:55.467 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.467 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a47ffcfb:::benchmark_data_smithi171_122772_object90052:head# 2024-04-20T17:27:55.467 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.467 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4808028:::benchmark_data_smithi171_122772_object111555:head# 2024-04-20T17:27:55.467 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.468 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a480db82:::benchmark_data_smithi171_122772_object126809:head# 2024-04-20T17:27:55.468 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.468 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4818a62:::benchmark_data_smithi171_122772_object116095:head# 2024-04-20T17:27:55.468 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.468 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a481bd6e:::benchmark_data_smithi171_122772_object162036:head# 2024-04-20T17:27:55.468 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.468 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a482359c:::benchmark_data_smithi171_122772_object119791:head# 2024-04-20T17:27:55.468 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.468 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a482427f:::benchmark_data_smithi171_122772_object117230:head# 2024-04-20T17:27:55.468 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.468 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4836466:::benchmark_data_smithi171_122772_object119774:head# 2024-04-20T17:27:55.468 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.468 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4841e63:::benchmark_data_smithi171_122772_object115229:head# 2024-04-20T17:27:55.469 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.469 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4846cc2:::benchmark_data_smithi171_122772_object93492:head# 2024-04-20T17:27:55.469 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.469 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a485315b:::benchmark_data_smithi171_122772_object133739:head# 2024-04-20T17:27:55.469 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.469 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a485dde3:::benchmark_data_smithi171_122772_object84336:head# 2024-04-20T17:27:55.469 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.469 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a486bc51:::benchmark_data_smithi171_122772_object120820:head# 2024-04-20T17:27:55.469 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.469 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a48749e3:::benchmark_data_smithi171_122772_object144940:head# 2024-04-20T17:27:55.469 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.469 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4891806:::benchmark_data_smithi171_122772_object110308:head# 2024-04-20T17:27:55.470 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.470 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4893ef6:::benchmark_data_smithi171_122772_object79198:head# 2024-04-20T17:27:55.470 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.470 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a48a1dfe:::benchmark_data_smithi171_122772_object154428:head# 2024-04-20T17:27:55.470 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.470 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a48b6d6e:::benchmark_data_smithi171_122772_object154286:head# 2024-04-20T17:27:55.470 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.470 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a48bdfb5:::benchmark_data_smithi171_122772_object136487:head# 2024-04-20T17:27:55.470 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.470 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a48c2d46:::benchmark_data_smithi171_122772_object152543:head# 2024-04-20T17:27:55.470 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.471 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a48c415f:::benchmark_data_smithi171_122772_object135910:head# 2024-04-20T17:27:55.471 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.471 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a48c7fb7:::benchmark_data_smithi171_122772_object108563:head# 2024-04-20T17:27:55.471 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.471 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a48d5e04:::benchmark_data_smithi171_122772_object165796:head# 2024-04-20T17:27:55.471 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.471 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a48dd5ae:::benchmark_data_smithi171_122772_object147532:head# 2024-04-20T17:27:55.471 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.471 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a48ed54a:::benchmark_data_smithi171_122772_object107510:head# 2024-04-20T17:27:55.471 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.471 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a48f30a6:::benchmark_data_smithi171_122772_object108562:head# 2024-04-20T17:27:55.472 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.472 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a490bfee:::benchmark_data_smithi171_122772_object96775:head# 2024-04-20T17:27:55.472 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.472 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4924086:::benchmark_data_smithi171_122772_object99500:head# 2024-04-20T17:27:55.472 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.472 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a49271c9:::benchmark_data_smithi171_122772_object121882:head# 2024-04-20T17:27:55.472 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.472 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a492c7b8:::benchmark_data_smithi171_122772_object103369:head# 2024-04-20T17:27:55.472 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.472 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a492d66b:::benchmark_data_smithi171_122772_object100000:head# 2024-04-20T17:27:55.472 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.472 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4937506:::benchmark_data_smithi171_122772_object150721:head# 2024-04-20T17:27:55.473 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.473 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4937933:::benchmark_data_smithi171_122772_object122196:head# 2024-04-20T17:27:55.473 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.473 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4946df8:::benchmark_data_smithi171_122772_object120576:head# 2024-04-20T17:27:55.473 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.473 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a494a5d0:::benchmark_data_smithi171_122772_object155142:head# 2024-04-20T17:27:55.473 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.473 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4953e65:::benchmark_data_smithi171_122772_object136659:head# 2024-04-20T17:27:55.473 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.473 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a495563a:::benchmark_data_smithi171_122772_object107031:head# 2024-04-20T17:27:55.473 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.474 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a495801a:::benchmark_data_smithi171_122772_object87576:head# 2024-04-20T17:27:55.474 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.474 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a49619a1:::benchmark_data_smithi171_122772_object85500:head# 2024-04-20T17:27:55.474 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.474 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a496cc1a:::benchmark_data_smithi171_122772_object146531:head# 2024-04-20T17:27:55.474 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.474 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a498320d:::benchmark_data_smithi171_122772_object88450:head# 2024-04-20T17:27:55.474 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.474 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4988691:::benchmark_data_smithi171_122772_object145282:head# 2024-04-20T17:27:55.474 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.475 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4990e87:::benchmark_data_smithi171_122772_object97344:head# 2024-04-20T17:27:55.475 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.475 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4995477:::benchmark_data_smithi171_122772_object156422:head# 2024-04-20T17:27:55.475 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.475 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a49aca69:::benchmark_data_smithi171_122772_object98308:head# 2024-04-20T17:27:55.475 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.475 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a49b7c69:::benchmark_data_smithi171_122772_object134576:head# 2024-04-20T17:27:55.475 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.475 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a49c610c:::benchmark_data_smithi171_122772_object87282:head# 2024-04-20T17:27:55.475 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.476 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a49d7916:::benchmark_data_smithi171_122772_object146263:head# 2024-04-20T17:27:55.476 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.476 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a49df3f4:::benchmark_data_smithi171_122772_object120271:head# 2024-04-20T17:27:55.476 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.476 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4a01796:::benchmark_data_smithi171_122772_object95098:head# 2024-04-20T17:27:55.476 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.476 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4a05d6a:::benchmark_data_smithi171_122772_object160468:head# 2024-04-20T17:27:55.476 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.476 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4a0d806:::benchmark_data_smithi171_122772_object150071:head# 2024-04-20T17:27:55.477 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.477 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4a1d6f0:::benchmark_data_smithi171_122772_object129660:head# 2024-04-20T17:27:55.477 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.477 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4a33367:::benchmark_data_smithi171_122772_object124776:head# 2024-04-20T17:27:55.477 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.477 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4a3a0fa:::benchmark_data_smithi171_122772_object157748:head# 2024-04-20T17:27:55.477 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.477 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4a3b614:::benchmark_data_smithi171_122772_object156195:head# 2024-04-20T17:27:55.477 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.478 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4a471e2:::benchmark_data_smithi171_122772_object158071:head# 2024-04-20T17:27:55.478 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.478 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4a5236d:::benchmark_data_smithi171_122772_object95222:head# 2024-04-20T17:27:55.478 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.478 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4a53eb8:::benchmark_data_smithi171_122772_object115900:head# 2024-04-20T17:27:55.478 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.478 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4a6c901:::benchmark_data_smithi171_122772_object167089:head# 2024-04-20T17:27:55.478 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.478 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4a6e0d5:::benchmark_data_smithi171_122772_object124576:head# 2024-04-20T17:27:55.478 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.479 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4a74388:::benchmark_data_smithi171_122772_object135737:head# 2024-04-20T17:27:55.479 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.479 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4a7cd2f:::benchmark_data_smithi171_122772_object126456:head# 2024-04-20T17:27:55.479 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.479 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4a8f6f2:::benchmark_data_smithi171_122772_object147182:head# 2024-04-20T17:27:55.479 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.479 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4a9b8ec:::benchmark_data_smithi171_122772_object134482:head# 2024-04-20T17:27:55.479 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.479 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4aa9e7a:::benchmark_data_smithi171_122772_object162775:head# 2024-04-20T17:27:55.479 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.480 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4abc7a4:::benchmark_data_smithi171_122772_object85157:head# 2024-04-20T17:27:55.480 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.480 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4ac37b9:::benchmark_data_smithi171_122772_object85974:head# 2024-04-20T17:27:55.480 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.480 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4ac603a:::benchmark_data_smithi171_122772_object117912:head# 2024-04-20T17:27:55.480 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.480 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4ad8c44:::benchmark_data_smithi171_122772_object124663:head# 2024-04-20T17:27:55.480 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.480 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4ad8e64:::benchmark_data_smithi171_122772_object161575:head# 2024-04-20T17:27:55.481 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.481 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4ad8fbe:::benchmark_data_smithi171_122772_object80899:head# 2024-04-20T17:27:55.481 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.481 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4af6082:::benchmark_data_smithi171_122772_object92569:head# 2024-04-20T17:27:55.481 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.481 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4b08e1b:::benchmark_data_smithi171_122772_object110586:head# 2024-04-20T17:27:55.481 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.481 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4b0d3a0:::benchmark_data_smithi171_122772_object85721:head# 2024-04-20T17:27:55.481 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.481 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4b201c5:::benchmark_data_smithi171_122772_object165123:head# 2024-04-20T17:27:55.481 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.482 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4b23580:::benchmark_data_smithi171_122772_object146058:head# 2024-04-20T17:27:55.482 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.482 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4b395ab:::benchmark_data_smithi171_122772_object137565:head# 2024-04-20T17:27:55.482 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.482 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4b4794c:::benchmark_data_smithi171_122772_object99573:head# 2024-04-20T17:27:55.482 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.482 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4b610c9:::benchmark_data_smithi171_122772_object138829:head# 2024-04-20T17:27:55.482 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.482 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4b6fd79:::benchmark_data_smithi171_122772_object163152:head# 2024-04-20T17:27:55.482 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.483 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4b7f246:::benchmark_data_smithi171_122772_object118428:head# 2024-04-20T17:27:55.483 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.483 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4bade22:::benchmark_data_smithi171_122772_object156350:head# 2024-04-20T17:27:55.483 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.483 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4bb0c7e:::benchmark_data_smithi171_122772_object156156:head# 2024-04-20T17:27:55.483 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.483 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4bb100f:::benchmark_data_smithi171_122772_object160556:head# 2024-04-20T17:27:55.483 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.483 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4bbe488:::benchmark_data_smithi171_122772_object107246:head# 2024-04-20T17:27:55.483 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.484 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4bdf282:::benchmark_data_smithi171_122772_object147339:head# 2024-04-20T17:27:55.484 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.484 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4bf8091:::benchmark_data_smithi171_122772_object149475:head# 2024-04-20T17:27:55.484 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.484 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4c02dac:::benchmark_data_smithi171_122772_object125757:head# 2024-04-20T17:27:55.484 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.484 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4c035f2:::benchmark_data_smithi171_122772_object104036:head# 2024-04-20T17:27:55.484 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.484 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4c199ac:::benchmark_data_smithi171_122772_object114118:head# 2024-04-20T17:27:55.484 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.484 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4c1e49d:::benchmark_data_smithi171_122772_object158568:head# 2024-04-20T17:27:55.485 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.485 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4c1fc1b:::benchmark_data_smithi171_122772_object125876:head# 2024-04-20T17:27:55.485 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.485 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4c1fd43:::benchmark_data_smithi171_122772_object113135:head# 2024-04-20T17:27:55.485 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.485 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4c22e88:::benchmark_data_smithi171_122772_object151675:head# 2024-04-20T17:27:55.485 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.485 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4c3b993:::benchmark_data_smithi171_122772_object132575:head# 2024-04-20T17:27:55.485 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.485 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4c45a0b:::benchmark_data_smithi171_122772_object134782:head# 2024-04-20T17:27:55.486 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.486 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4c46df4:::benchmark_data_smithi171_122772_object119808:head# 2024-04-20T17:27:55.486 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.486 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4c4c3f0:::benchmark_data_smithi171_122772_object85142:head# 2024-04-20T17:27:55.486 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.486 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4c58abf:::benchmark_data_smithi171_122772_object125178:head# 2024-04-20T17:27:55.486 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.486 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4c622f9:::benchmark_data_smithi171_122772_object169804:head# 2024-04-20T17:27:55.486 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.486 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4c63a1a:::benchmark_data_smithi171_122772_object161642:head# 2024-04-20T17:27:55.486 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.487 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4c8369d:::benchmark_data_smithi171_122772_object150657:head# 2024-04-20T17:27:55.487 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.487 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4c897fe:::benchmark_data_smithi171_122772_object122377:head# 2024-04-20T17:27:55.487 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.487 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4c8c788:::benchmark_data_smithi171_122772_object150533:head# 2024-04-20T17:27:55.487 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.487 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4c93204:::benchmark_data_smithi171_122772_object86980:head# 2024-04-20T17:27:55.487 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.487 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4c9ae72:::benchmark_data_smithi171_122772_object121146:head# 2024-04-20T17:27:55.487 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.487 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4ca7496:::benchmark_data_smithi171_122772_object84635:head# 2024-04-20T17:27:55.488 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.488 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4ca9e59:::benchmark_data_smithi171_122772_object79211:head# 2024-04-20T17:27:55.488 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.488 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4cb7dd4:::benchmark_data_smithi171_122772_object118654:head# 2024-04-20T17:27:55.488 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.488 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4cc1794:::benchmark_data_smithi171_122772_object98515:head# 2024-04-20T17:27:55.488 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.488 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4cca0b6:::benchmark_data_smithi171_122772_object136374:head# 2024-04-20T17:27:55.488 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.488 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4ccdab5:::benchmark_data_smithi171_122772_object129355:head# 2024-04-20T17:27:55.488 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.489 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4cd4625:::benchmark_data_smithi171_122772_object106709:head# 2024-04-20T17:27:55.489 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.489 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4cd9185:::benchmark_data_smithi171_122772_object132681:head# 2024-04-20T17:27:55.489 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.489 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4cf2846:::benchmark_data_smithi171_122772_object91559:head# 2024-04-20T17:27:55.489 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.489 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4cf3879:::benchmark_data_smithi171_122772_object161659:head# 2024-04-20T17:27:55.489 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.489 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4cf7e87:::benchmark_data_smithi171_122772_object112902:head# 2024-04-20T17:27:55.489 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.489 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4cf9382:::benchmark_data_smithi171_122772_object152952:head# 2024-04-20T17:27:55.490 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.490 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4cfdad4:::benchmark_data_smithi171_122772_object118969:head# 2024-04-20T17:27:55.490 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.490 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4d0421f:::benchmark_data_smithi171_122772_object169298:head# 2024-04-20T17:27:55.490 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.490 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4d554e1:::benchmark_data_smithi171_122772_object112479:head# 2024-04-20T17:27:55.490 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.490 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4d55e92:::benchmark_data_smithi171_122772_object117429:head# 2024-04-20T17:27:55.490 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.490 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4d58c34:::benchmark_data_smithi171_122772_object119932:head# 2024-04-20T17:27:55.490 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.491 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4d5bd65:::benchmark_data_smithi171_122772_object85227:head# 2024-04-20T17:27:55.491 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.491 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4d641d7:::benchmark_data_smithi171_122772_object93952:head# 2024-04-20T17:27:55.491 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.491 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4d6a049:::benchmark_data_smithi171_122772_object111334:head# 2024-04-20T17:27:55.491 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.491 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4d7094c:::benchmark_data_smithi171_122772_object109739:head# 2024-04-20T17:27:55.491 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.491 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4d7df9c:::benchmark_data_smithi171_122772_object148276:head# 2024-04-20T17:27:55.491 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.491 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4d8168f:::benchmark_data_smithi171_122772_object169090:head# 2024-04-20T17:27:55.492 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.492 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4d818fa:::benchmark_data_smithi171_122772_object135331:head# 2024-04-20T17:27:55.492 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.492 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4d82b6f:::benchmark_data_smithi171_122772_object116717:head# 2024-04-20T17:27:55.492 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.492 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4d8875d:::benchmark_data_smithi171_122772_object132126:head# 2024-04-20T17:27:55.492 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.492 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4d89480:::benchmark_data_smithi171_122772_object85414:head# 2024-04-20T17:27:55.492 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.492 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4d89f5a:::benchmark_data_smithi171_122772_object116100:head# 2024-04-20T17:27:55.493 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.493 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4db75d9:::benchmark_data_smithi171_122772_object130688:head# 2024-04-20T17:27:55.493 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.493 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4dba2a0:::benchmark_data_smithi171_122772_object90176:head# 2024-04-20T17:27:55.493 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.493 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4dbcb7a:::benchmark_data_smithi171_122772_object169867:head# 2024-04-20T17:27:55.493 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.493 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4dd4c8f:::benchmark_data_smithi171_122772_object79964:head# 2024-04-20T17:27:55.493 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.493 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4ddcb05:::benchmark_data_smithi171_122772_object148899:head# 2024-04-20T17:27:55.493 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.493 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4e01e3d:::benchmark_data_smithi171_122772_object80713:head# 2024-04-20T17:27:55.494 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.494 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4e144dc:::benchmark_data_smithi171_122772_object124134:head# 2024-04-20T17:27:55.494 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.494 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4e230df:::benchmark_data_smithi171_122772_object98234:head# 2024-04-20T17:27:55.494 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.494 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4e330b5:::benchmark_data_smithi171_122772_object154031:head# 2024-04-20T17:27:55.494 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.494 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4e3d554:::benchmark_data_smithi171_122772_object109845:head# 2024-04-20T17:27:55.494 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.494 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4e664e9:::benchmark_data_smithi171_122772_object125736:head# 2024-04-20T17:27:55.494 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.495 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4e6e090:::benchmark_data_smithi171_122772_object95414:head# 2024-04-20T17:27:55.495 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.495 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4e6f7b7:::benchmark_data_smithi171_122772_object95369:head# 2024-04-20T17:27:55.495 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.495 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4e7082c:::benchmark_data_smithi171_122772_object145152:head# 2024-04-20T17:27:55.495 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.495 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4e88a4f:::benchmark_data_smithi171_122772_object148462:head# 2024-04-20T17:27:55.495 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.495 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4eac385:::benchmark_data_smithi171_122772_object137425:head# 2024-04-20T17:27:55.495 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.495 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4ead24c:::benchmark_data_smithi171_122772_object137526:head# 2024-04-20T17:27:55.495 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.495 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4eb5a82:::benchmark_data_smithi171_122772_object133316:head# 2024-04-20T17:27:55.496 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.496 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4ecf9ea:::benchmark_data_smithi171_122772_object123357:head# 2024-04-20T17:27:55.496 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.496 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4ed3e48:::benchmark_data_smithi171_122772_object81028:head# 2024-04-20T17:27:55.496 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.496 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4f06cf7:::benchmark_data_smithi171_122772_object127984:head# 2024-04-20T17:27:55.496 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.496 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4f0af64:::benchmark_data_smithi171_122772_object85714:head# 2024-04-20T17:27:55.496 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.496 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4f1544e:::benchmark_data_smithi171_122772_object136754:head# 2024-04-20T17:27:55.496 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.497 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4f21d70:::benchmark_data_smithi171_122772_object167896:head# 2024-04-20T17:27:55.497 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.497 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4f3044c:::benchmark_data_smithi171_122772_object167797:head# 2024-04-20T17:27:55.497 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.497 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4f3866e:::benchmark_data_smithi171_122772_object108997:head# 2024-04-20T17:27:55.497 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.497 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4f38f8f:::benchmark_data_smithi171_122772_object121422:head# 2024-04-20T17:27:55.497 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.497 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4f3a4f7:::benchmark_data_smithi171_122772_object81439:head# 2024-04-20T17:27:55.497 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.497 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4f4021f:::benchmark_data_smithi171_122772_object95520:head# 2024-04-20T17:27:55.497 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.498 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4f42f00:::benchmark_data_smithi171_122772_object96563:head# 2024-04-20T17:27:55.498 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.498 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4f4816e:::benchmark_data_smithi171_122772_object141298:head# 2024-04-20T17:27:55.498 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.498 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4f5c922:::benchmark_data_smithi171_122772_object107299:head# 2024-04-20T17:27:55.498 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.498 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4f5dfb7:::benchmark_data_smithi171_122772_object148553:head# 2024-04-20T17:27:55.498 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.498 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4f656da:::benchmark_data_smithi171_122772_object129109:head# 2024-04-20T17:27:55.498 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.498 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4f6fa78:::benchmark_data_smithi171_122772_object110312:head# 2024-04-20T17:27:55.498 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.499 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4f7bedd:::benchmark_data_smithi171_122772_object79811:head# 2024-04-20T17:27:55.499 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.499 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4f7dfe0:::benchmark_data_smithi171_122772_object83978:head# 2024-04-20T17:27:55.499 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.499 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4f83922:::benchmark_data_smithi171_122772_object128050:head# 2024-04-20T17:27:55.499 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.499 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4f87857:::benchmark_data_smithi171_122772_object165886:head# 2024-04-20T17:27:55.499 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.499 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4f967cd:::benchmark_data_smithi171_122772_object86090:head# 2024-04-20T17:27:55.499 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.499 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4fb1161:::benchmark_data_smithi171_122772_object134275:head# 2024-04-20T17:27:55.500 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.500 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4fcb7b4:::benchmark_data_smithi171_122772_object112591:head# 2024-04-20T17:27:55.500 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.500 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4fdd238:::benchmark_data_smithi171_122772_object153799:head# 2024-04-20T17:27:55.500 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.500 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4fe43fd:::benchmark_data_smithi171_122772_object134112:head# 2024-04-20T17:27:55.500 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.500 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4febf3c:::benchmark_data_smithi171_122772_object118010:head# 2024-04-20T17:27:55.500 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.500 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4ff0969:::benchmark_data_smithi171_122772_object83565:head# 2024-04-20T17:27:55.501 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.501 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4ffb77a:::benchmark_data_smithi171_122772_object154331:head# 2024-04-20T17:27:55.501 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.501 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a4ffbcc4:::benchmark_data_smithi171_122772_object134316:head# 2024-04-20T17:27:55.501 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.501 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a500f577:::benchmark_data_smithi171_122772_object111776:head# 2024-04-20T17:27:55.501 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.501 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5017ab7:::benchmark_data_smithi171_122772_object140059:head# 2024-04-20T17:27:55.501 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.501 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a50307dc:::benchmark_data_smithi171_122772_object138566:head# 2024-04-20T17:27:55.501 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.501 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5033336:::benchmark_data_smithi171_122772_object111872:head# 2024-04-20T17:27:55.502 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.502 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5033900:::benchmark_data_smithi171_122772_object93154:head# 2024-04-20T17:27:55.502 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.502 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a50354df:::benchmark_data_smithi171_122772_object96088:head# 2024-04-20T17:27:55.502 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.502 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5039f21:::benchmark_data_smithi171_122772_object168627:head# 2024-04-20T17:27:55.502 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.502 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a503d075:::benchmark_data_smithi171_122772_object155278:head# 2024-04-20T17:27:55.502 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.502 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a503d547:::benchmark_data_smithi171_122772_object115203:head# 2024-04-20T17:27:55.503 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.503 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a504076e:::benchmark_data_smithi171_122772_object153632:head# 2024-04-20T17:27:55.503 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.503 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a505bd58:::benchmark_data_smithi171_122772_object102004:head# 2024-04-20T17:27:55.503 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.503 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5063c72:::benchmark_data_smithi171_122772_object96221:head# 2024-04-20T17:27:55.503 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.503 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5066d44:::benchmark_data_smithi171_122772_object129815:head# 2024-04-20T17:27:55.503 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.503 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5089a0b:::benchmark_data_smithi171_122772_object82826:head# 2024-04-20T17:27:55.503 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.503 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5095847:::benchmark_data_smithi171_122772_object167380:head# 2024-04-20T17:27:55.504 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.504 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a50a9b74:::benchmark_data_smithi171_122772_object147261:head# 2024-04-20T17:27:55.504 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.504 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a50aedb0:::benchmark_data_smithi171_122772_object98539:head# 2024-04-20T17:27:55.504 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.504 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a50b0610:::benchmark_data_smithi171_122772_object144424:head# 2024-04-20T17:27:55.504 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.504 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a50ca327:::benchmark_data_smithi171_122772_object161469:head# 2024-04-20T17:27:55.504 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.504 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a50d8744:::benchmark_data_smithi171_122772_object123057:head# 2024-04-20T17:27:55.504 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.504 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a50f0b0e:::benchmark_data_smithi171_122772_object92745:head# 2024-04-20T17:27:55.504 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.505 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a50fed09:::benchmark_data_smithi171_122772_object98066:head# 2024-04-20T17:27:55.505 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.505 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5110b68:::benchmark_data_smithi171_122772_object116112:head# 2024-04-20T17:27:55.505 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.505 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a51121a1:::benchmark_data_smithi171_122772_object88848:head# 2024-04-20T17:27:55.505 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.505 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a51244b0:::benchmark_data_smithi171_122772_object108534:head# 2024-04-20T17:27:55.505 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.505 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a51340af:::benchmark_data_smithi171_122772_object144806:head# 2024-04-20T17:27:55.505 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.505 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a514780c:::benchmark_data_smithi171_122772_object133735:head# 2024-04-20T17:27:55.505 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.505 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a51479f6:::benchmark_data_smithi171_122772_object79775:head# 2024-04-20T17:27:55.506 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.506 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5148319:::benchmark_data_smithi171_122772_object135675:head# 2024-04-20T17:27:55.506 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.506 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a514e4dc:::benchmark_data_smithi171_122772_object112331:head# 2024-04-20T17:27:55.506 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.506 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a515b6a5:::benchmark_data_smithi171_122772_object119284:head# 2024-04-20T17:27:55.506 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.506 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a51601de:::benchmark_data_smithi171_122772_object150911:head# 2024-04-20T17:27:55.506 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.506 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a516046d:::benchmark_data_smithi171_122772_object119103:head# 2024-04-20T17:27:55.506 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.506 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a51899f4:::benchmark_data_smithi171_122772_object144373:head# 2024-04-20T17:27:55.507 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.507 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5192a5d:::benchmark_data_smithi171_122772_object132494:head# 2024-04-20T17:27:55.507 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.507 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5197b6a:::benchmark_data_smithi171_122772_object154181:head# 2024-04-20T17:27:55.507 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.507 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a519c9cd:::benchmark_data_smithi171_122772_object160643:head# 2024-04-20T17:27:55.507 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.507 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a51b07da:::benchmark_data_smithi171_122772_object84968:head# 2024-04-20T17:27:55.507 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.507 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a51cc990:::benchmark_data_smithi171_122772_object117563:head# 2024-04-20T17:27:55.507 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.507 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a51cd350:::benchmark_data_smithi171_122772_object108351:head# 2024-04-20T17:27:55.508 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.508 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a51ceb9b:::benchmark_data_smithi171_122772_object137690:head# 2024-04-20T17:27:55.508 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.508 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a51d6a99:::benchmark_data_smithi171_122772_object125721:head# 2024-04-20T17:27:55.508 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.508 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a51da7e5:::benchmark_data_smithi171_122772_object154921:head# 2024-04-20T17:27:55.508 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.508 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a51e1d5f:::benchmark_data_smithi171_122772_object78811:head# 2024-04-20T17:27:55.508 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.508 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a51e4fd0:::benchmark_data_smithi171_122772_object95739:head# 2024-04-20T17:27:55.508 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.508 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a51e8806:::benchmark_data_smithi171_122772_object164719:head# 2024-04-20T17:27:55.509 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.509 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a51f1139:::benchmark_data_smithi171_122772_object144790:head# 2024-04-20T17:27:55.509 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.509 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a51f3888:::benchmark_data_smithi171_122772_object86793:head# 2024-04-20T17:27:55.509 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.509 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5212f26:::benchmark_data_smithi171_122772_object163330:head# 2024-04-20T17:27:55.509 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.509 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5215dc6:::benchmark_data_smithi171_122772_object150937:head# 2024-04-20T17:27:55.509 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.509 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a521d102:::benchmark_data_smithi171_122772_object105094:head# 2024-04-20T17:27:55.509 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.509 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5223f84:::benchmark_data_smithi171_122772_object162943:head# 2024-04-20T17:27:55.510 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.510 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a522ee0e:::benchmark_data_smithi171_122772_object157116:head# 2024-04-20T17:27:55.510 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.510 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a524dc3f:::benchmark_data_smithi171_122772_object108039:head# 2024-04-20T17:27:55.510 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.510 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5254a6f:::benchmark_data_smithi171_122772_object115154:head# 2024-04-20T17:27:55.510 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.510 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a525cddf:::benchmark_data_smithi171_122772_object152585:head# 2024-04-20T17:27:55.510 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.510 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a526402c:::benchmark_data_smithi171_122772_object151222:head# 2024-04-20T17:27:55.510 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.510 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a526d6dd:::benchmark_data_smithi171_122772_object130573:head# 2024-04-20T17:27:55.511 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.511 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a527413f:::benchmark_data_smithi171_122772_object142344:head# 2024-04-20T17:27:55.511 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.511 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a527418d:::benchmark_data_smithi171_122772_object140261:head# 2024-04-20T17:27:55.511 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.511 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a528127d:::benchmark_data_smithi171_122772_object117527:head# 2024-04-20T17:27:55.511 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.511 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a529ad13:::benchmark_data_smithi171_122772_object134981:head# 2024-04-20T17:27:55.511 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.511 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a529d8b7:::benchmark_data_smithi171_122772_object125163:head# 2024-04-20T17:27:55.511 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.511 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a52a43c2:::benchmark_data_smithi171_122772_object101441:head# 2024-04-20T17:27:55.511 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.512 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a52a9061:::benchmark_data_smithi171_122772_object113167:head# 2024-04-20T17:27:55.512 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.512 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a52af482:::benchmark_data_smithi171_122772_object119401:head# 2024-04-20T17:27:55.512 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.512 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a52b0e47:::benchmark_data_smithi171_122772_object145423:head# 2024-04-20T17:27:55.512 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.512 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a52bb233:::benchmark_data_smithi171_122772_object143611:head# 2024-04-20T17:27:55.512 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.512 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a52bcbd0:::benchmark_data_smithi171_122772_object117856:head# 2024-04-20T17:27:55.512 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.512 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a52be18f:::benchmark_data_smithi171_122772_object81438:head# 2024-04-20T17:27:55.512 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.513 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a52c9908:::benchmark_data_smithi171_122772_object99736:head# 2024-04-20T17:27:55.513 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.513 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a52cbadd:::benchmark_data_smithi171_122772_object102601:head# 2024-04-20T17:27:55.513 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.513 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a52d378b:::benchmark_data_smithi171_122772_object88486:head# 2024-04-20T17:27:55.513 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.513 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a52e7431:::benchmark_data_smithi171_122772_object94129:head# 2024-04-20T17:27:55.513 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.513 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a52ef067:::benchmark_data_smithi171_122772_object145670:head# 2024-04-20T17:27:55.513 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.513 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a52f7bec:::benchmark_data_smithi171_122772_object142199:head# 2024-04-20T17:27:55.513 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.514 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a52fc78f:::benchmark_data_smithi171_122772_object147153:head# 2024-04-20T17:27:55.514 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.514 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5309ba5:::benchmark_data_smithi171_122772_object124340:head# 2024-04-20T17:27:55.514 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.514 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5312bf7:::benchmark_data_smithi171_122772_object163743:head# 2024-04-20T17:27:55.514 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.514 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a531be4c:::benchmark_data_smithi171_122772_object130704:head# 2024-04-20T17:27:55.514 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.514 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a532e4ff:::benchmark_data_smithi171_122772_object122234:head# 2024-04-20T17:27:55.514 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.514 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a533732f:::benchmark_data_smithi171_122772_object118011:head# 2024-04-20T17:27:55.514 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.515 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a533e667:::benchmark_data_smithi171_122772_object154357:head# 2024-04-20T17:27:55.515 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.515 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a536669b:::benchmark_data_smithi171_122772_object92353:head# 2024-04-20T17:27:55.515 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.515 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5371959:::benchmark_data_smithi171_122772_object147710:head# 2024-04-20T17:27:55.515 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.515 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a53823d3:::benchmark_data_smithi171_122772_object121982:head# 2024-04-20T17:27:55.515 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.515 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5388870:::benchmark_data_smithi171_122772_object170033:head# 2024-04-20T17:27:55.515 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.515 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a53987ac:::benchmark_data_smithi171_122772_object169871:head# 2024-04-20T17:27:55.515 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.516 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a53a7bd9:::benchmark_data_smithi171_122772_object113325:head# 2024-04-20T17:27:55.516 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.516 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a53a7fe3:::benchmark_data_smithi171_122772_object84469:head# 2024-04-20T17:27:55.516 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.516 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a53c0738:::benchmark_data_smithi171_122772_object134715:head# 2024-04-20T17:27:55.516 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.516 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a53cca88:::benchmark_data_smithi171_122772_object162615:head# 2024-04-20T17:27:55.516 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.516 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a53f2122:::benchmark_data_smithi171_122772_object162627:head# 2024-04-20T17:27:55.516 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.516 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a540876e:::benchmark_data_smithi171_122772_object119114:head# 2024-04-20T17:27:55.516 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.517 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5419f5b:::benchmark_data_smithi171_122772_object97388:head# 2024-04-20T17:27:55.517 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.517 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a54217d8:::benchmark_data_smithi171_122772_object168517:head# 2024-04-20T17:27:55.517 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.517 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a542c987:::benchmark_data_smithi171_122772_object161738:head# 2024-04-20T17:27:55.517 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.517 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a54429c0:::benchmark_data_smithi171_122772_object147707:head# 2024-04-20T17:27:55.517 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.517 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a545cce9:::benchmark_data_smithi171_122772_object142597:head# 2024-04-20T17:27:55.517 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.517 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a54690dd:::benchmark_data_smithi171_122772_object117446:head# 2024-04-20T17:27:55.517 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.518 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a546c3c9:::benchmark_data_smithi171_122772_object118549:head# 2024-04-20T17:27:55.518 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.518 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5483aa0:::benchmark_data_smithi171_122772_object79922:head# 2024-04-20T17:27:55.518 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.518 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5485ceb:::benchmark_data_smithi171_122772_object101744:head# 2024-04-20T17:27:55.518 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.518 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5486225:::benchmark_data_smithi171_122772_object111890:head# 2024-04-20T17:27:55.518 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.518 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a548cb1f:::benchmark_data_smithi171_122772_object82499:head# 2024-04-20T17:27:55.518 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.518 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5498706:::benchmark_data_smithi171_122772_object84349:head# 2024-04-20T17:27:55.519 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.519 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a549a011:::benchmark_data_smithi171_122772_object89405:head# 2024-04-20T17:27:55.519 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.519 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a549b6a3:::benchmark_data_smithi171_122772_object114097:head# 2024-04-20T17:27:55.519 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.519 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a54a4a73:::benchmark_data_smithi171_122772_object112447:head# 2024-04-20T17:27:55.519 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.519 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a54a5fdf:::benchmark_data_smithi171_122772_object119421:head# 2024-04-20T17:27:55.519 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.519 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a54a686b:::benchmark_data_smithi171_122772_object85968:head# 2024-04-20T17:27:55.519 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.519 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a54a6a4f:::benchmark_data_smithi171_122772_object159061:head# 2024-04-20T17:27:55.520 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.520 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a54adbb1:::benchmark_data_smithi171_122772_object169848:head# 2024-04-20T17:27:55.520 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.520 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a54b726e:::benchmark_data_smithi171_122772_object90240:head# 2024-04-20T17:27:55.520 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.520 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a54b7a7e:::benchmark_data_smithi171_122772_object143399:head# 2024-04-20T17:27:55.520 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.520 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a54bf63a:::benchmark_data_smithi171_122772_object105723:head# 2024-04-20T17:27:55.520 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.520 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a54c3741:::benchmark_data_smithi171_122772_object88207:head# 2024-04-20T17:27:55.520 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.520 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a54c4f42:::benchmark_data_smithi171_122772_object125659:head# 2024-04-20T17:27:55.521 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.521 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a54ca07b:::benchmark_data_smithi171_122772_object109291:head# 2024-04-20T17:27:55.521 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.521 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a54d1bde:::benchmark_data_smithi171_122772_object88136:head# 2024-04-20T17:27:55.521 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.521 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a54d364d:::benchmark_data_smithi171_122772_object111713:head# 2024-04-20T17:27:55.521 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.521 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a54dbf9e:::benchmark_data_smithi171_122772_object111031:head# 2024-04-20T17:27:55.521 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.521 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a54eda52:::benchmark_data_smithi171_122772_object102953:head# 2024-04-20T17:27:55.522 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.522 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a54f183e:::benchmark_data_smithi171_122772_object162998:head# 2024-04-20T17:27:55.522 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.522 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a54ff0fb:::benchmark_data_smithi171_122772_object152544:head# 2024-04-20T17:27:55.522 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.522 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a550636b:::benchmark_data_smithi171_122772_object137651:head# 2024-04-20T17:27:55.522 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.522 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a550b3f9:::benchmark_data_smithi171_122772_object139146:head# 2024-04-20T17:27:55.522 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.522 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5510191:::benchmark_data_smithi171_122772_object80840:head# 2024-04-20T17:27:55.522 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.522 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5514cd3:::benchmark_data_smithi171_122772_object160625:head# 2024-04-20T17:27:55.523 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.523 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a55264a1:::benchmark_data_smithi171_122772_object80599:head# 2024-04-20T17:27:55.523 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.523 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a552f6b7:::benchmark_data_smithi171_122772_object160628:head# 2024-04-20T17:27:55.523 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.523 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5531254:::benchmark_data_smithi171_122772_object165763:head# 2024-04-20T17:27:55.523 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.523 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5535791:::benchmark_data_smithi171_122772_object167930:head# 2024-04-20T17:27:55.523 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.523 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a553b517:::benchmark_data_smithi171_122772_object152562:head# 2024-04-20T17:27:55.523 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.524 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5545e8a:::benchmark_data_smithi171_122772_object149288:head# 2024-04-20T17:27:55.524 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.524 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a555c7c2:::benchmark_data_smithi171_122772_object149102:head# 2024-04-20T17:27:55.524 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.524 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a555dc47:::benchmark_data_smithi171_122772_object137405:head# 2024-04-20T17:27:55.524 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.524 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5563037:::benchmark_data_smithi171_122772_object92721:head# 2024-04-20T17:27:55.524 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.524 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5569d4b:::benchmark_data_smithi171_122772_object95069:head# 2024-04-20T17:27:55.524 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.525 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a55955cc:::benchmark_data_smithi171_122772_object139890:head# 2024-04-20T17:27:55.525 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.525 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a559c6df:::benchmark_data_smithi171_122772_object148845:head# 2024-04-20T17:27:55.525 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.525 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a559e32c:::benchmark_data_smithi171_122772_object140672:head# 2024-04-20T17:27:55.525 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.525 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a55a654d:::benchmark_data_smithi171_122772_object164599:head# 2024-04-20T17:27:55.525 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.525 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a55c211a:::benchmark_data_smithi171_122772_object123906:head# 2024-04-20T17:27:55.525 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.525 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a55cfcf5:::benchmark_data_smithi171_122772_object98483:head# 2024-04-20T17:27:55.526 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.526 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a55d3304:::benchmark_data_smithi171_122772_object146785:head# 2024-04-20T17:27:55.526 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.526 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a55e668b:::benchmark_data_smithi171_122772_object100675:head# 2024-04-20T17:27:55.526 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.526 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a55fa5fe:::benchmark_data_smithi171_122772_object88099:head# 2024-04-20T17:27:55.526 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.526 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a55fcc80:::benchmark_data_smithi171_122772_object144048:head# 2024-04-20T17:27:55.526 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.526 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a55fe3b1:::benchmark_data_smithi171_122772_object95474:head# 2024-04-20T17:27:55.526 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.526 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5600f78:::benchmark_data_smithi171_122772_object128838:head# 2024-04-20T17:27:55.527 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.527 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5606493:::benchmark_data_smithi171_122772_object140688:head# 2024-04-20T17:27:55.527 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.527 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5641d95:::benchmark_data_smithi171_122772_object165319:head# 2024-04-20T17:27:55.527 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.527 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a565ce64:::benchmark_data_smithi171_122772_object102394:head# 2024-04-20T17:27:55.527 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.527 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a567d049:::benchmark_data_smithi171_122772_object167060:head# 2024-04-20T17:27:55.527 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.527 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5682629:::benchmark_data_smithi171_122772_object144294:head# 2024-04-20T17:27:55.527 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.528 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a568a86a:::benchmark_data_smithi171_122772_object123469:head# 2024-04-20T17:27:55.528 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.528 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5697adf:::benchmark_data_smithi171_122772_object140571:head# 2024-04-20T17:27:55.528 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.528 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a56ac76b:::benchmark_data_smithi171_122772_object83662:head# 2024-04-20T17:27:55.528 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.528 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a56bc3c6:::benchmark_data_smithi171_122772_object147929:head# 2024-04-20T17:27:55.528 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.528 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a56c73b1:::benchmark_data_smithi171_122772_object128974:head# 2024-04-20T17:27:55.528 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.528 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a56d4126:::benchmark_data_smithi171_122772_object85671:head# 2024-04-20T17:27:55.528 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.529 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a56d9e99:::benchmark_data_smithi171_122772_object102184:head# 2024-04-20T17:27:55.529 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.529 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a56ed56c:::benchmark_data_smithi171_122772_object93844:head# 2024-04-20T17:27:55.529 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.529 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a56fb52d:::benchmark_data_smithi171_122772_object109333:head# 2024-04-20T17:27:55.529 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.529 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a57063e4:::benchmark_data_smithi171_122772_object138831:head# 2024-04-20T17:27:55.529 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.529 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5708641:::benchmark_data_smithi171_122772_object169476:head# 2024-04-20T17:27:55.529 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.529 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a570c222:::benchmark_data_smithi171_122772_object150133:head# 2024-04-20T17:27:55.529 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.529 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a57179ba:::benchmark_data_smithi171_122772_object105023:head# 2024-04-20T17:27:55.530 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.530 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a57213bd:::benchmark_data_smithi171_122772_object114786:head# 2024-04-20T17:27:55.530 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.530 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a572c447:::benchmark_data_smithi171_122772_object163049:head# 2024-04-20T17:27:55.530 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.530 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a573aff0:::benchmark_data_smithi171_122772_object151913:head# 2024-04-20T17:27:55.530 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.530 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a57537bd:::benchmark_data_smithi171_122772_object132323:head# 2024-04-20T17:27:55.530 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.530 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5767a86:::benchmark_data_smithi171_122772_object111434:head# 2024-04-20T17:27:55.530 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.531 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5771a8a:::benchmark_data_smithi171_122772_object113968:head# 2024-04-20T17:27:55.531 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.531 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a577d92d:::benchmark_data_smithi171_122772_object130624:head# 2024-04-20T17:27:55.531 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.531 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a579afef:::benchmark_data_smithi171_122772_object164615:head# 2024-04-20T17:27:55.531 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.531 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a57a351a:::benchmark_data_smithi171_122772_object95067:head# 2024-04-20T17:27:55.531 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.531 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a57b8d0b:::benchmark_data_smithi171_122772_object110612:head# 2024-04-20T17:27:55.531 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.531 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a57c7d48:::benchmark_data_smithi171_122772_object122576:head# 2024-04-20T17:27:55.531 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.532 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a57cc70f:::benchmark_data_smithi171_122772_object110366:head# 2024-04-20T17:27:55.532 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.532 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a57d4e96:::benchmark_data_smithi171_122772_object90573:head# 2024-04-20T17:27:55.532 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.532 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a57db99c:::benchmark_data_smithi171_122772_object102985:head# 2024-04-20T17:27:55.532 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.532 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a57f01f4:::benchmark_data_smithi171_122772_object158850:head# 2024-04-20T17:27:55.532 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.532 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a57fdaa9:::benchmark_data_smithi171_122772_object80823:head# 2024-04-20T17:27:55.532 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.532 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a57feeef:::benchmark_data_smithi171_122772_object97415:head# 2024-04-20T17:27:55.532 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.533 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5817421:::benchmark_data_smithi171_122772_object148939:head# 2024-04-20T17:27:55.533 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.533 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a58181ae:::benchmark_data_smithi171_122772_object113602:head# 2024-04-20T17:27:55.533 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.533 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a581cf27:::benchmark_data_smithi171_122772_object157591:head# 2024-04-20T17:27:55.533 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:55.536 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:55.536 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:27:55.536 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:27:55.612 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5832821:::benchmark_data_smithi171_122772_object134873:head# 2024-04-20T17:27:55.612 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.612 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5833ac3:::benchmark_data_smithi171_122772_object96743:head# 2024-04-20T17:27:55.613 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.613 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5836350:::benchmark_data_smithi171_122772_object120277:head# 2024-04-20T17:27:55.613 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.613 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a585eb1f:::benchmark_data_smithi171_122772_object151566:head# 2024-04-20T17:27:55.613 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.613 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a58635f6:::benchmark_data_smithi171_122772_object139126:head# 2024-04-20T17:27:55.613 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.613 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5868f83:::benchmark_data_smithi171_122772_object93258:head# 2024-04-20T17:27:55.613 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.613 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a587142d:::benchmark_data_smithi171_122772_object158607:head# 2024-04-20T17:27:55.614 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.614 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a588983a:::benchmark_data_smithi171_122772_object161905:head# 2024-04-20T17:27:55.614 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.614 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5892c0b:::benchmark_data_smithi171_122772_object113194:head# 2024-04-20T17:27:55.614 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.614 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a58947bd:::benchmark_data_smithi171_122772_object163764:head# 2024-04-20T17:27:55.614 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.614 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a589dcd7:::benchmark_data_smithi171_122772_object110456:head# 2024-04-20T17:27:55.614 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.614 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a58a5407:::benchmark_data_smithi171_122772_object169168:head# 2024-04-20T17:27:55.614 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.615 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a58a7d84:::benchmark_data_smithi171_122772_object118277:head# 2024-04-20T17:27:55.615 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.615 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a58a8b78:::benchmark_data_smithi171_122772_object144904:head# 2024-04-20T17:27:55.615 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.615 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a58b20db:::benchmark_data_smithi171_122772_object86497:head# 2024-04-20T17:27:55.615 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.615 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a58b895a:::benchmark_data_smithi171_122772_object86293:head# 2024-04-20T17:27:55.615 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.615 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a58c9cda:::benchmark_data_smithi171_122772_object86708:head# 2024-04-20T17:27:55.615 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.615 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a58d20a7:::benchmark_data_smithi171_122772_object148488:head# 2024-04-20T17:27:55.615 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.616 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a594090d:::benchmark_data_smithi171_122772_object97075:head# 2024-04-20T17:27:55.616 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.616 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5942447:::benchmark_data_smithi171_122772_object138848:head# 2024-04-20T17:27:55.616 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.616 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5949589:::benchmark_data_smithi171_122772_object146923:head# 2024-04-20T17:27:55.616 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.616 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a594d2ce:::benchmark_data_smithi171_122772_object137918:head# 2024-04-20T17:27:55.616 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.616 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a598e16c:::benchmark_data_smithi171_122772_object118073:head# 2024-04-20T17:27:55.616 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.616 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a59a06c0:::benchmark_data_smithi171_122772_object119846:head# 2024-04-20T17:27:55.616 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.617 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a59a15c7:::benchmark_data_smithi171_122772_object128784:head# 2024-04-20T17:27:55.617 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.617 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a59a7bfa:::benchmark_data_smithi171_122772_object139466:head# 2024-04-20T17:27:55.617 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.617 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a59ae31f:::benchmark_data_smithi171_122772_object163466:head# 2024-04-20T17:27:55.617 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.617 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a59af4cf:::benchmark_data_smithi171_122772_object132905:head# 2024-04-20T17:27:55.617 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.617 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a59c4e9a:::benchmark_data_smithi171_122772_object98788:head# 2024-04-20T17:27:55.617 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.617 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a59e01b2:::benchmark_data_smithi171_122772_object102350:head# 2024-04-20T17:27:55.618 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.618 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a59e35df:::benchmark_data_smithi171_122772_object124168:head# 2024-04-20T17:27:55.618 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.618 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a59e4e7f:::benchmark_data_smithi171_122772_object88081:head# 2024-04-20T17:27:55.618 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.618 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a59f0302:::benchmark_data_smithi171_122772_object136311:head# 2024-04-20T17:27:55.618 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.618 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a59fb6c4:::benchmark_data_smithi171_122772_object108148:head# 2024-04-20T17:27:55.618 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.618 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5a00629:::benchmark_data_smithi171_122772_object168816:head# 2024-04-20T17:27:55.618 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.619 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5a02a67:::benchmark_data_smithi171_122772_object125766:head# 2024-04-20T17:27:55.619 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.619 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5a087e7:::benchmark_data_smithi171_122772_object166080:head# 2024-04-20T17:27:55.619 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.619 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5a30d13:::benchmark_data_smithi171_122772_object83099:head# 2024-04-20T17:27:55.619 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.619 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5a3c064:::benchmark_data_smithi171_122772_object114766:head# 2024-04-20T17:27:55.619 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.619 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5a4c914:::benchmark_data_smithi171_122772_object88684:head# 2024-04-20T17:27:55.619 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.619 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5a58517:::benchmark_data_smithi171_122772_object135471:head# 2024-04-20T17:27:55.620 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.620 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5a597fc:::benchmark_data_smithi171_122772_object111651:head# 2024-04-20T17:27:55.620 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.620 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5a5fc4a:::benchmark_data_smithi171_122772_object81469:head# 2024-04-20T17:27:55.620 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.620 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5a74780:::benchmark_data_smithi171_122772_object127481:head# 2024-04-20T17:27:55.620 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.620 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5a76ab1:::benchmark_data_smithi171_122772_object158082:head# 2024-04-20T17:27:55.620 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.620 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5a78e38:::benchmark_data_smithi171_122772_object142935:head# 2024-04-20T17:27:55.620 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.621 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5a7a760:::benchmark_data_smithi171_122772_object160155:head# 2024-04-20T17:27:55.621 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.621 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5a7ffa8:::benchmark_data_smithi171_122772_object93624:head# 2024-04-20T17:27:55.621 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.621 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5a804be:::benchmark_data_smithi171_122772_object136564:head# 2024-04-20T17:27:55.621 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.621 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5a855d7:::benchmark_data_smithi171_122772_object138118:head# 2024-04-20T17:27:55.621 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.621 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5a8a151:::benchmark_data_smithi171_122772_object113605:head# 2024-04-20T17:27:55.621 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.621 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5a9296e:::benchmark_data_smithi171_122772_object123639:head# 2024-04-20T17:27:55.622 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.622 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5a9c3bc:::benchmark_data_smithi171_122772_object81191:head# 2024-04-20T17:27:55.622 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.622 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5aa8a5d:::benchmark_data_smithi171_122772_object110658:head# 2024-04-20T17:27:55.622 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.622 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5ab996d:::benchmark_data_smithi171_122772_object103237:head# 2024-04-20T17:27:55.622 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.622 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5ad4316:::benchmark_data_smithi171_122772_object163276:head# 2024-04-20T17:27:55.622 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.622 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5ae5e19:::benchmark_data_smithi171_122772_object119675:head# 2024-04-20T17:27:55.622 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.623 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5af7309:::benchmark_data_smithi171_122772_object130740:head# 2024-04-20T17:27:55.623 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.623 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5b0775d:::benchmark_data_smithi171_122772_object96535:head# 2024-04-20T17:27:55.623 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.623 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5b07d78:::benchmark_data_smithi171_122772_object100752:head# 2024-04-20T17:27:55.623 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.623 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5b13038:::benchmark_data_smithi171_122772_object165236:head# 2024-04-20T17:27:55.623 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.623 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5b19810:::benchmark_data_smithi171_122772_object90428:head# 2024-04-20T17:27:55.623 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.623 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5b2bb0c:::benchmark_data_smithi171_122772_object152138:head# 2024-04-20T17:27:55.623 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.624 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5b42f9d:::benchmark_data_smithi171_122772_object150800:head# 2024-04-20T17:27:55.624 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.624 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5b5e4ea:::benchmark_data_smithi171_122772_object101181:head# 2024-04-20T17:27:55.624 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.624 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5b609bc:::benchmark_data_smithi171_122772_object120998:head# 2024-04-20T17:27:55.624 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.624 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5b634fc:::benchmark_data_smithi171_122772_object134012:head# 2024-04-20T17:27:55.624 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.624 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5b763b2:::benchmark_data_smithi171_122772_object82302:head# 2024-04-20T17:27:55.624 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.624 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5b76dda:::benchmark_data_smithi171_122772_object81809:head# 2024-04-20T17:27:55.625 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.625 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5b833b1:::benchmark_data_smithi171_122772_object150751:head# 2024-04-20T17:27:55.625 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.625 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5b97ca2:::benchmark_data_smithi171_122772_object87643:head# 2024-04-20T17:27:55.625 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.625 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5bb6bce:::benchmark_data_smithi171_122772_object168451:head# 2024-04-20T17:27:55.625 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.625 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5bc6162:::benchmark_data_smithi171_122772_object148099:head# 2024-04-20T17:27:55.625 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.625 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5bc617d:::benchmark_data_smithi171_122772_object130096:head# 2024-04-20T17:27:55.625 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.625 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5bc8388:::benchmark_data_smithi171_122772_object132026:head# 2024-04-20T17:27:55.626 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.626 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5bc97b5:::benchmark_data_smithi171_122772_object96604:head# 2024-04-20T17:27:55.626 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.626 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5bd99ad:::benchmark_data_smithi171_122772_object152221:head# 2024-04-20T17:27:55.626 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.626 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5bdda12:::benchmark_data_smithi171_122772_object92360:head# 2024-04-20T17:27:55.626 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.626 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5be103c:::benchmark_data_smithi171_122772_object87855:head# 2024-04-20T17:27:55.626 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.626 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5be99bc:::benchmark_data_smithi171_122772_object107358:head# 2024-04-20T17:27:55.626 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.626 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5c05953:::benchmark_data_smithi171_122772_object157474:head# 2024-04-20T17:27:55.626 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.627 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5c16e1b:::benchmark_data_smithi171_122772_object113006:head# 2024-04-20T17:27:55.627 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.627 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5c26842:::benchmark_data_smithi171_122772_object87732:head# 2024-04-20T17:27:55.627 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.627 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5c2c43c:::benchmark_data_smithi171_122772_object125067:head# 2024-04-20T17:27:55.627 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.627 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5c2d588:::benchmark_data_smithi171_122772_object137754:head# 2024-04-20T17:27:55.627 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.627 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5c30249:::benchmark_data_smithi171_122772_object94826:head# 2024-04-20T17:27:55.627 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.627 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5c33eac:::benchmark_data_smithi171_122772_object84422:head# 2024-04-20T17:27:55.627 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.627 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5c344a0:::benchmark_data_smithi171_122772_object146565:head# 2024-04-20T17:27:55.628 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.628 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5c34d19:::benchmark_data_smithi171_122772_object128962:head# 2024-04-20T17:27:55.628 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.628 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5c426f4:::benchmark_data_smithi171_122772_object141566:head# 2024-04-20T17:27:55.628 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.628 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5c46eab:::benchmark_data_smithi171_122772_object169467:head# 2024-04-20T17:27:55.628 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.628 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5c48cea:::benchmark_data_smithi171_122772_object78237:head# 2024-04-20T17:27:55.628 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.628 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5c553f5:::benchmark_data_smithi171_122772_object130133:head# 2024-04-20T17:27:55.628 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.628 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5c5e0c7:::benchmark_data_smithi171_122772_object158396:head# 2024-04-20T17:27:55.629 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.629 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5c61da6:::benchmark_data_smithi171_122772_object107930:head# 2024-04-20T17:27:55.629 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.629 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5c6acbd:::benchmark_data_smithi171_122772_object135614:head# 2024-04-20T17:27:55.629 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.629 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5c6d52f:::benchmark_data_smithi171_122772_object96643:head# 2024-04-20T17:27:55.629 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.629 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5c76275:::benchmark_data_smithi171_122772_object97216:head# 2024-04-20T17:27:55.629 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.629 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5c7b2be:::benchmark_data_smithi171_122772_object159743:head# 2024-04-20T17:27:55.629 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.629 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5c87d85:::benchmark_data_smithi171_122772_object128809:head# 2024-04-20T17:27:55.629 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.630 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5c8be18:::benchmark_data_smithi171_122772_object116257:head# 2024-04-20T17:27:55.630 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.630 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5c8fe72:::benchmark_data_smithi171_122772_object142191:head# 2024-04-20T17:27:55.630 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.630 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5c94fb6:::benchmark_data_smithi171_122772_object159728:head# 2024-04-20T17:27:55.630 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.630 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5cce885:::benchmark_data_smithi171_122772_object116544:head# 2024-04-20T17:27:55.630 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.630 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5cd3820:::benchmark_data_smithi171_122772_object113162:head# 2024-04-20T17:27:55.630 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.630 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5cdacf4:::benchmark_data_smithi171_122772_object89603:head# 2024-04-20T17:27:55.630 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.631 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5cf1ed2:::benchmark_data_smithi171_122772_object136053:head# 2024-04-20T17:27:55.631 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.631 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5cf4642:::benchmark_data_smithi171_122772_object89263:head# 2024-04-20T17:27:55.631 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.631 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5cf4895:::benchmark_data_smithi171_122772_object110299:head# 2024-04-20T17:27:55.631 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.631 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5cfa8d6:::benchmark_data_smithi171_122772_object80271:head# 2024-04-20T17:27:55.631 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.631 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5d03d95:::benchmark_data_smithi171_122772_object147568:head# 2024-04-20T17:27:55.631 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.631 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5d064a8:::benchmark_data_smithi171_122772_object98866:head# 2024-04-20T17:27:55.632 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.632 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5d19740:::benchmark_data_smithi171_122772_object153760:head# 2024-04-20T17:27:55.632 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.632 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5d1b35c:::benchmark_data_smithi171_122772_object100985:head# 2024-04-20T17:27:55.632 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.632 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5d28db3:::benchmark_data_smithi171_122772_object124700:head# 2024-04-20T17:27:55.632 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.632 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5d3478f:::benchmark_data_smithi171_122772_object102488:head# 2024-04-20T17:27:55.632 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.632 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5d5b33e:::benchmark_data_smithi171_122772_object146489:head# 2024-04-20T17:27:55.632 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.632 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5d5f7b1:::benchmark_data_smithi171_122772_object103997:head# 2024-04-20T17:27:55.632 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.633 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5d64ac0:::benchmark_data_smithi171_122772_object138545:head# 2024-04-20T17:27:55.633 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.633 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5d68b7e:::benchmark_data_smithi171_122772_object99317:head# 2024-04-20T17:27:55.633 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.633 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5d6bd31:::benchmark_data_smithi171_122772_object99999:head# 2024-04-20T17:27:55.633 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.633 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5d76865:::benchmark_data_smithi171_122772_object93559:head# 2024-04-20T17:27:55.633 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.633 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5d7e3e8:::benchmark_data_smithi171_122772_object125991:head# 2024-04-20T17:27:55.633 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.633 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5d91efc:::benchmark_data_smithi171_122772_object95016:head# 2024-04-20T17:27:55.633 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.634 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5d95be2:::benchmark_data_smithi171_122772_object101628:head# 2024-04-20T17:27:55.634 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.634 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5da2570:::benchmark_data_smithi171_122772_object163502:head# 2024-04-20T17:27:55.634 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.634 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5dab9f2:::benchmark_data_smithi171_122772_object95063:head# 2024-04-20T17:27:55.634 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.634 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5db3a48:::benchmark_data_smithi171_122772_object106736:head# 2024-04-20T17:27:55.634 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.634 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5dbb8a2:::benchmark_data_smithi171_122772_object115615:head# 2024-04-20T17:27:55.634 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.634 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5dc5d9e:::benchmark_data_smithi171_122772_object155115:head# 2024-04-20T17:27:55.634 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.635 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5dc6dde:::benchmark_data_smithi171_122772_object145510:head# 2024-04-20T17:27:55.635 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.635 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5dc83bf:::benchmark_data_smithi171_122772_object95633:head# 2024-04-20T17:27:55.635 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.635 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5dcf10f:::benchmark_data_smithi171_122772_object111510:head# 2024-04-20T17:27:55.635 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.635 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5dd1c8a:::benchmark_data_smithi171_122772_object122049:head# 2024-04-20T17:27:55.635 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.635 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5dd3c78:::benchmark_data_smithi171_122772_object132053:head# 2024-04-20T17:27:55.635 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.635 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5dd5ba0:::benchmark_data_smithi171_122772_object116146:head# 2024-04-20T17:27:55.636 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.636 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5de497d:::benchmark_data_smithi171_122772_object146383:head# 2024-04-20T17:27:55.636 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.636 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5de5acc:::benchmark_data_smithi171_122772_object114749:head# 2024-04-20T17:27:55.636 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.636 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5dfbf66:::benchmark_data_smithi171_122772_object142468:head# 2024-04-20T17:27:55.636 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.636 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5e15c9e:::benchmark_data_smithi171_122772_object122399:head# 2024-04-20T17:27:55.636 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.636 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5e2e23c:::benchmark_data_smithi171_122772_object116034:head# 2024-04-20T17:27:55.636 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.636 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5e34d86:::benchmark_data_smithi171_122772_object109799:head# 2024-04-20T17:27:55.636 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.637 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5e45edc:::benchmark_data_smithi171_122772_object97324:head# 2024-04-20T17:27:55.637 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.637 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5e48d5e:::benchmark_data_smithi171_122772_object152718:head# 2024-04-20T17:27:55.637 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.637 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5e49ea7:::benchmark_data_smithi171_122772_object164326:head# 2024-04-20T17:27:55.637 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.637 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5e4b340:::benchmark_data_smithi171_122772_object127154:head# 2024-04-20T17:27:55.637 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.637 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5e52f1d:::benchmark_data_smithi171_122772_object120341:head# 2024-04-20T17:27:55.637 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.637 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5e60de4:::benchmark_data_smithi171_122772_object145491:head# 2024-04-20T17:27:55.637 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.638 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5e625f8:::benchmark_data_smithi171_122772_object90768:head# 2024-04-20T17:27:55.638 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.638 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5e80877:::benchmark_data_smithi171_122772_object159150:head# 2024-04-20T17:27:55.638 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.638 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5e80f23:::benchmark_data_smithi171_122772_object89957:head# 2024-04-20T17:27:55.638 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.638 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5e87744:::benchmark_data_smithi171_122772_object169240:head# 2024-04-20T17:27:55.638 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.638 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5e9706d:::benchmark_data_smithi171_122772_object87018:head# 2024-04-20T17:27:55.638 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.638 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5eb032c:::benchmark_data_smithi171_122772_object89744:head# 2024-04-20T17:27:55.638 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.639 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5ebad05:::benchmark_data_smithi171_122772_object108265:head# 2024-04-20T17:27:55.639 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.639 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5ec284d:::benchmark_data_smithi171_122772_object98367:head# 2024-04-20T17:27:55.639 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.639 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5ec88cb:::benchmark_data_smithi171_122772_object103447:head# 2024-04-20T17:27:55.639 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.639 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5ed1523:::benchmark_data_smithi171_122772_object108104:head# 2024-04-20T17:27:55.639 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.639 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5ed31a5:::benchmark_data_smithi171_122772_object108152:head# 2024-04-20T17:27:55.639 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.639 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5ee3011:::benchmark_data_smithi171_122772_object119369:head# 2024-04-20T17:27:55.639 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.639 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5ef2a28:::benchmark_data_smithi171_122772_object156654:head# 2024-04-20T17:27:55.640 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.640 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5efc32e:::benchmark_data_smithi171_122772_object79018:head# 2024-04-20T17:27:55.640 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.640 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5f05709:::benchmark_data_smithi171_122772_object137671:head# 2024-04-20T17:27:55.640 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.640 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5f0704b:::benchmark_data_smithi171_122772_object106553:head# 2024-04-20T17:27:55.640 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.640 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5f36946:::benchmark_data_smithi171_122772_object155870:head# 2024-04-20T17:27:55.640 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.640 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5f3792c:::benchmark_data_smithi171_122772_object93229:head# 2024-04-20T17:27:55.640 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.640 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5f3ed55:::benchmark_data_smithi171_122772_object152812:head# 2024-04-20T17:27:55.641 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.641 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5f610e4:::benchmark_data_smithi171_122772_object146823:head# 2024-04-20T17:27:55.641 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.641 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5f68df1:::benchmark_data_smithi171_122772_object145810:head# 2024-04-20T17:27:55.641 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.641 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5f6eef9:::benchmark_data_smithi171_122772_object134779:head# 2024-04-20T17:27:55.641 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.641 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5f726fa:::benchmark_data_smithi171_122772_object146092:head# 2024-04-20T17:27:55.641 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.641 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5f759b2:::benchmark_data_smithi171_122772_object152164:head# 2024-04-20T17:27:55.641 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.641 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5f76a3a:::benchmark_data_smithi171_122772_object167336:head# 2024-04-20T17:27:55.641 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.642 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5f7aa67:::benchmark_data_smithi171_122772_object136299:head# 2024-04-20T17:27:55.642 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.642 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5f8a759:::benchmark_data_smithi171_122772_object148494:head# 2024-04-20T17:27:55.642 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.642 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5f9a88f:::benchmark_data_smithi171_122772_object94491:head# 2024-04-20T17:27:55.642 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.642 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5fa68b2:::benchmark_data_smithi171_122772_object120320:head# 2024-04-20T17:27:55.642 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.642 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5fb1f53:::benchmark_data_smithi171_122772_object135943:head# 2024-04-20T17:27:55.642 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.642 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5fb784e:::benchmark_data_smithi171_122772_object138552:head# 2024-04-20T17:27:55.643 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.643 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5fbfa1e:::benchmark_data_smithi171_122772_object117333:head# 2024-04-20T17:27:55.643 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.643 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5fc66b7:::benchmark_data_smithi171_122772_object137505:head# 2024-04-20T17:27:55.643 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.643 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5ff342a:::benchmark_data_smithi171_122772_object155725:head# 2024-04-20T17:27:55.643 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.643 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a5ff5a95:::benchmark_data_smithi171_122772_object146504:head# 2024-04-20T17:27:55.643 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.643 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a60140d2:::benchmark_data_smithi171_122772_object119940:head# 2024-04-20T17:27:55.643 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.644 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6030d2e:::benchmark_data_smithi171_122772_object169360:head# 2024-04-20T17:27:55.644 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.644 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6042569:::benchmark_data_smithi171_122772_object88491:head# 2024-04-20T17:27:55.644 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.644 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a604bddc:::benchmark_data_smithi171_122772_object109546:head# 2024-04-20T17:27:55.644 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.644 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a606a3a9:::benchmark_data_smithi171_122772_object80916:head# 2024-04-20T17:27:55.644 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.644 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a606e11e:::benchmark_data_smithi171_122772_object93144:head# 2024-04-20T17:27:55.645 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.645 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a60760c6:::benchmark_data_smithi171_122772_object94456:head# 2024-04-20T17:27:55.645 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.645 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a608b901:::benchmark_data_smithi171_122772_object140105:head# 2024-04-20T17:27:55.645 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.645 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a608e2f4:::benchmark_data_smithi171_122772_object106213:head# 2024-04-20T17:27:55.645 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.645 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6095de0:::benchmark_data_smithi171_122772_object92799:head# 2024-04-20T17:27:55.645 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.645 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a609c4c1:::benchmark_data_smithi171_122772_object125793:head# 2024-04-20T17:27:55.646 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.646 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a60a4698:::benchmark_data_smithi171_122772_object137997:head# 2024-04-20T17:27:55.646 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.646 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a60a626a:::benchmark_data_smithi171_122772_object134697:head# 2024-04-20T17:27:55.646 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.646 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a60a939c:::benchmark_data_smithi171_122772_object148627:head# 2024-04-20T17:27:55.646 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.646 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a60b2acc:::benchmark_data_smithi171_122772_object146898:head# 2024-04-20T17:27:55.646 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.646 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a60bcfcf:::benchmark_data_smithi171_122772_object89364:head# 2024-04-20T17:27:55.647 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.647 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a60d392d:::benchmark_data_smithi171_122772_object142186:head# 2024-04-20T17:27:55.647 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.647 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a60d6113:::benchmark_data_smithi171_122772_object78639:head# 2024-04-20T17:27:55.647 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.647 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a60d9375:::benchmark_data_smithi171_122772_object91982:head# 2024-04-20T17:27:55.647 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.647 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a60e1950:::benchmark_data_smithi171_122772_object94264:head# 2024-04-20T17:27:55.647 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.647 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a60eeaf3:::benchmark_data_smithi171_122772_object152627:head# 2024-04-20T17:27:55.647 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.648 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a60f0903:::benchmark_data_smithi171_122772_object122264:head# 2024-04-20T17:27:55.648 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.648 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a60f0d30:::benchmark_data_smithi171_122772_object85629:head# 2024-04-20T17:27:55.648 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.648 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6105ad8:::benchmark_data_smithi171_122772_object150762:head# 2024-04-20T17:27:55.648 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.648 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a610c839:::benchmark_data_smithi171_122772_object134805:head# 2024-04-20T17:27:55.648 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.648 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6115645:::benchmark_data_smithi171_122772_object83041:head# 2024-04-20T17:27:55.648 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.648 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6116853:::benchmark_data_smithi171_122772_object141432:head# 2024-04-20T17:27:55.648 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.649 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a611dd70:::benchmark_data_smithi171_122772_object108610:head# 2024-04-20T17:27:55.649 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.649 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6125077:::benchmark_data_smithi171_122772_object154810:head# 2024-04-20T17:27:55.649 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.649 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a612916f:::benchmark_data_smithi171_122772_object108006:head# 2024-04-20T17:27:55.649 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.649 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a61351ea:::benchmark_data_smithi171_122772_object166901:head# 2024-04-20T17:27:55.649 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.649 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a61387da:::benchmark_data_smithi171_122772_object133572:head# 2024-04-20T17:27:55.649 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.649 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a61393a9:::benchmark_data_smithi171_122772_object124068:head# 2024-04-20T17:27:55.650 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.650 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a614bbfb:::benchmark_data_smithi171_122772_object163566:head# 2024-04-20T17:27:55.650 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.650 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6153311:::benchmark_data_smithi171_122772_object140109:head# 2024-04-20T17:27:55.650 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.650 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a617c986:::benchmark_data_smithi171_122772_object85726:head# 2024-04-20T17:27:55.650 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.650 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a61928dc:::benchmark_data_smithi171_122772_object103829:head# 2024-04-20T17:27:55.650 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.650 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6192b27:::benchmark_data_smithi171_122772_object144423:head# 2024-04-20T17:27:55.651 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.651 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a619be09:::benchmark_data_smithi171_122772_object105607:head# 2024-04-20T17:27:55.651 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.651 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a61aabb7:::benchmark_data_smithi171_122772_object169399:head# 2024-04-20T17:27:55.651 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.651 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a61b1e14:::benchmark_data_smithi171_122772_object129566:head# 2024-04-20T17:27:55.651 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.651 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a61b368f:::benchmark_data_smithi171_122772_object163789:head# 2024-04-20T17:27:55.651 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.651 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a61c3616:::benchmark_data_smithi171_122772_object126265:head# 2024-04-20T17:27:55.651 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.652 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a61df7da:::benchmark_data_smithi171_122772_object116323:head# 2024-04-20T17:27:55.652 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.652 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a61f2387:::benchmark_data_smithi171_122772_object80216:head# 2024-04-20T17:27:55.652 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.652 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a61fbe0c:::benchmark_data_smithi171_122772_object152551:head# 2024-04-20T17:27:55.652 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.652 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6218f85:::benchmark_data_smithi171_122772_object126613:head# 2024-04-20T17:27:55.652 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.652 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a62254f0:::benchmark_data_smithi171_122772_object122536:head# 2024-04-20T17:27:55.652 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.652 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a622ae5d:::benchmark_data_smithi171_122772_object101378:head# 2024-04-20T17:27:55.652 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.652 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a622b04f:::benchmark_data_smithi171_122772_object119810:head# 2024-04-20T17:27:55.653 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.653 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a622eda4:::benchmark_data_smithi171_122772_object105402:head# 2024-04-20T17:27:55.653 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.653 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6251607:::benchmark_data_smithi171_122772_object114193:head# 2024-04-20T17:27:55.653 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.653 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a625608f:::benchmark_data_smithi171_122772_object120324:head# 2024-04-20T17:27:55.653 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.653 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a625679f:::benchmark_data_smithi171_122772_object118608:head# 2024-04-20T17:27:55.653 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.653 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a626540b:::benchmark_data_smithi171_122772_object169023:head# 2024-04-20T17:27:55.653 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.653 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a62695d9:::benchmark_data_smithi171_122772_object79822:head# 2024-04-20T17:27:55.653 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.653 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6275883:::benchmark_data_smithi171_122772_object154609:head# 2024-04-20T17:27:55.654 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.654 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a627769f:::benchmark_data_smithi171_122772_object142371:head# 2024-04-20T17:27:55.654 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.654 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6285e0b:::benchmark_data_smithi171_122772_object80104:head# 2024-04-20T17:27:55.654 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.654 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6289098:::benchmark_data_smithi171_122772_object144660:head# 2024-04-20T17:27:55.654 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.654 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a628ba73:::benchmark_data_smithi171_122772_object159191:head# 2024-04-20T17:27:55.654 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.654 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a62a123c:::benchmark_data_smithi171_122772_object165448:head# 2024-04-20T17:27:55.654 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.654 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a62ae012:::benchmark_data_smithi171_122772_object153161:head# 2024-04-20T17:27:55.655 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.655 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a62b848c:::benchmark_data_smithi171_122772_object147582:head# 2024-04-20T17:27:55.655 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.655 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a62c3fe1:::benchmark_data_smithi171_122772_object151339:head# 2024-04-20T17:27:55.655 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.655 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a62c958a:::benchmark_data_smithi171_122772_object86045:head# 2024-04-20T17:27:55.655 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.655 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a62d94e5:::benchmark_data_smithi171_122772_object165001:head# 2024-04-20T17:27:55.655 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.655 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a62ede17:::benchmark_data_smithi171_122772_object135296:head# 2024-04-20T17:27:55.655 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.655 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6307ee4:::benchmark_data_smithi171_122772_object144419:head# 2024-04-20T17:27:55.656 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.656 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a630ab84:::benchmark_data_smithi171_122772_object149236:head# 2024-04-20T17:27:55.656 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.656 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a630f86d:::benchmark_data_smithi171_122772_object147253:head# 2024-04-20T17:27:55.656 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.656 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a630f9f3:::benchmark_data_smithi171_122772_object162432:head# 2024-04-20T17:27:55.656 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.656 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a631df93:::benchmark_data_smithi171_122772_object119388:head# 2024-04-20T17:27:55.656 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.656 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6321ec2:::benchmark_data_smithi171_122772_object154863:head# 2024-04-20T17:27:55.656 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.657 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6326738:::benchmark_data_smithi171_122772_object167887:head# 2024-04-20T17:27:55.657 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.657 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a634a530:::benchmark_data_smithi171_122772_object117955:head# 2024-04-20T17:27:55.657 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.657 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a634a9df:::benchmark_data_smithi171_122772_object152181:head# 2024-04-20T17:27:55.657 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.657 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a63590f5:::benchmark_data_smithi171_122772_object169199:head# 2024-04-20T17:27:55.657 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.657 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6366529:::benchmark_data_smithi171_122772_object159818:head# 2024-04-20T17:27:55.657 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.658 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6368e31:::benchmark_data_smithi171_122772_object168578:head# 2024-04-20T17:27:55.658 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.658 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6373cda:::benchmark_data_smithi171_122772_object98931:head# 2024-04-20T17:27:55.658 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.658 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6376e12:::benchmark_data_smithi171_122772_object138845:head# 2024-04-20T17:27:55.658 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.658 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6385365:::benchmark_data_smithi171_122772_object112861:head# 2024-04-20T17:27:55.658 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.658 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6389c26:::benchmark_data_smithi171_122772_object81266:head# 2024-04-20T17:27:55.659 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.659 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a638d6e2:::benchmark_data_smithi171_122772_object110846:head# 2024-04-20T17:27:55.659 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.659 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a63996b0:::benchmark_data_smithi171_122772_object138066:head# 2024-04-20T17:27:55.659 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.659 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a63b50a1:::benchmark_data_smithi171_122772_object112457:head# 2024-04-20T17:27:55.659 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.659 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a63b923d:::benchmark_data_smithi171_122772_object168355:head# 2024-04-20T17:27:55.659 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.659 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a63bab28:::benchmark_data_smithi171_122772_object117420:head# 2024-04-20T17:27:55.660 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.660 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a63cd20b:::benchmark_data_smithi171_122772_object166827:head# 2024-04-20T17:27:55.660 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.660 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a63cda1b:::benchmark_data_smithi171_122772_object78693:head# 2024-04-20T17:27:55.660 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.660 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6403b85:::benchmark_data_smithi171_122772_object129214:head# 2024-04-20T17:27:55.660 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.660 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a640d356:::benchmark_data_smithi171_122772_object111116:head# 2024-04-20T17:27:55.660 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.660 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a640ebe4:::benchmark_data_smithi171_122772_object119621:head# 2024-04-20T17:27:55.660 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.661 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a642393a:::benchmark_data_smithi171_122772_object168383:head# 2024-04-20T17:27:55.661 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.661 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a643048b:::benchmark_data_smithi171_122772_object147093:head# 2024-04-20T17:27:55.661 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.661 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a643e2b3:::benchmark_data_smithi171_122772_object134343:head# 2024-04-20T17:27:55.661 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.661 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a646dc6c:::benchmark_data_smithi171_122772_object110019:head# 2024-04-20T17:27:55.661 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.661 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6475f6e:::benchmark_data_smithi171_122772_object86455:head# 2024-04-20T17:27:55.661 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.662 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a64765aa:::benchmark_data_smithi171_122772_object154122:head# 2024-04-20T17:27:55.662 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.662 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a64784a6:::benchmark_data_smithi171_122772_object109526:head# 2024-04-20T17:27:55.662 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.662 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a647a2ac:::benchmark_data_smithi171_122772_object147541:head# 2024-04-20T17:27:55.662 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.662 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a648438e:::benchmark_data_smithi171_122772_object164381:head# 2024-04-20T17:27:55.662 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.663 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6488585:::benchmark_data_smithi171_122772_object97895:head# 2024-04-20T17:27:55.663 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.663 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a649096f:::benchmark_data_smithi171_122772_object132900:head# 2024-04-20T17:27:55.663 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.663 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a649322c:::benchmark_data_smithi171_122772_object126468:head# 2024-04-20T17:27:55.663 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.663 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a649b1ff:::benchmark_data_smithi171_122772_object114577:head# 2024-04-20T17:27:55.663 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.663 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a649f9f4:::benchmark_data_smithi171_122772_object141005:head# 2024-04-20T17:27:55.663 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.664 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a64aec3f:::benchmark_data_smithi171_122772_object150724:head# 2024-04-20T17:27:55.664 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.664 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a64b7c46:::benchmark_data_smithi171_122772_object110833:head# 2024-04-20T17:27:55.664 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.664 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a64b9cea:::benchmark_data_smithi171_122772_object155282:head# 2024-04-20T17:27:55.664 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.664 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a64cfbe4:::benchmark_data_smithi171_122772_object165649:head# 2024-04-20T17:27:55.664 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.664 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a64ddc3a:::benchmark_data_smithi171_122772_object85120:head# 2024-04-20T17:27:55.664 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.664 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a64f9627:::benchmark_data_smithi171_122772_object89498:head# 2024-04-20T17:27:55.664 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.665 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6502de1:::benchmark_data_smithi171_122772_object110284:head# 2024-04-20T17:27:55.665 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.665 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6513677:::benchmark_data_smithi171_122772_object107975:head# 2024-04-20T17:27:55.665 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.665 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6521b15:::benchmark_data_smithi171_122772_object168134:head# 2024-04-20T17:27:55.665 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.665 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a65276b6:::benchmark_data_smithi171_122772_object155161:head# 2024-04-20T17:27:55.665 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.665 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a65311c8:::benchmark_data_smithi171_122772_object84076:head# 2024-04-20T17:27:55.665 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.666 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a655ff2f:::benchmark_data_smithi171_122772_object165076:head# 2024-04-20T17:27:55.666 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.666 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a657179a:::benchmark_data_smithi171_122772_object100480:head# 2024-04-20T17:27:55.666 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.666 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6576237:::benchmark_data_smithi171_122772_object154558:head# 2024-04-20T17:27:55.666 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.666 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a657d606:::benchmark_data_smithi171_122772_object154926:head# 2024-04-20T17:27:55.666 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.666 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a658f62e:::benchmark_data_smithi171_122772_object117522:head# 2024-04-20T17:27:55.666 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.666 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a659a52b:::benchmark_data_smithi171_122772_object150956:head# 2024-04-20T17:27:55.667 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.667 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a659fbb3:::benchmark_data_smithi171_122772_object141406:head# 2024-04-20T17:27:55.667 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.667 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a65b0a74:::benchmark_data_smithi171_122772_object126857:head# 2024-04-20T17:27:55.667 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.667 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a65bad8a:::benchmark_data_smithi171_122772_object100811:head# 2024-04-20T17:27:55.667 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.667 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a65c5690:::benchmark_data_smithi171_122772_object117391:head# 2024-04-20T17:27:55.667 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.667 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a65d1dd6:::benchmark_data_smithi171_122772_object154134:head# 2024-04-20T17:27:55.667 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.667 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a65d9b96:::benchmark_data_smithi171_122772_object77999:head# 2024-04-20T17:27:55.668 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.668 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a65dac27:::benchmark_data_smithi171_122772_object158166:head# 2024-04-20T17:27:55.668 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.668 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a65dd353:::benchmark_data_smithi171_122772_object101272:head# 2024-04-20T17:27:55.668 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.668 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a65e777a:::benchmark_data_smithi171_122772_object81120:head# 2024-04-20T17:27:55.668 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.668 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a65f3c3e:::benchmark_data_smithi171_122772_object155441:head# 2024-04-20T17:27:55.668 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.668 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a65fd432:::benchmark_data_smithi171_122772_object130780:head# 2024-04-20T17:27:55.668 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.668 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a65ff32a:::benchmark_data_smithi171_122772_object125247:head# 2024-04-20T17:27:55.668 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.669 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6614bd7:::benchmark_data_smithi171_122772_object124710:head# 2024-04-20T17:27:55.669 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.669 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a662df12:::benchmark_data_smithi171_122772_object114518:head# 2024-04-20T17:27:55.669 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.669 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a663ec62:::benchmark_data_smithi171_122772_object102213:head# 2024-04-20T17:27:55.669 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.669 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a665fd03:::benchmark_data_smithi171_122772_object147511:head# 2024-04-20T17:27:55.669 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.669 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6661345:::benchmark_data_smithi171_122772_object101792:head# 2024-04-20T17:27:55.669 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.669 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a667aebb:::benchmark_data_smithi171_122772_object114676:head# 2024-04-20T17:27:55.669 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.669 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a668abc7:::benchmark_data_smithi171_122772_object147271:head# 2024-04-20T17:27:55.670 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.670 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a66a1837:::benchmark_data_smithi171_122772_object158693:head# 2024-04-20T17:27:55.670 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.670 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a66a2f75:::benchmark_data_smithi171_122772_object89027:head# 2024-04-20T17:27:55.670 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.670 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a66ae30a:::benchmark_data_smithi171_122772_object86332:head# 2024-04-20T17:27:55.670 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.670 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a66b3449:::benchmark_data_smithi171_122772_object142944:head# 2024-04-20T17:27:55.670 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.670 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a66c4d57:::benchmark_data_smithi171_122772_object122329:head# 2024-04-20T17:27:55.671 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.671 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a66cd839:::benchmark_data_smithi171_122772_object149266:head# 2024-04-20T17:27:55.671 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.671 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a66dd8dc:::benchmark_data_smithi171_122772_object97024:head# 2024-04-20T17:27:55.671 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.671 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a66dff14:::benchmark_data_smithi171_122772_object78700:head# 2024-04-20T17:27:55.671 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.671 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a66e4f45:::benchmark_data_smithi171_122772_object98982:head# 2024-04-20T17:27:55.671 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.671 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a66edba3:::benchmark_data_smithi171_122772_object89097:head# 2024-04-20T17:27:55.671 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.672 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a66f166b:::benchmark_data_smithi171_122772_object91777:head# 2024-04-20T17:27:55.672 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.672 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a670bf33:::benchmark_data_smithi171_122772_object161968:head# 2024-04-20T17:27:55.672 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.672 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a670d8b1:::benchmark_data_smithi171_122772_object102196:head# 2024-04-20T17:27:55.672 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.672 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6718fbd:::benchmark_data_smithi171_122772_object159401:head# 2024-04-20T17:27:55.672 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.672 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a672c49f:::benchmark_data_smithi171_122772_object106713:head# 2024-04-20T17:27:55.672 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.673 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6747729:::benchmark_data_smithi171_122772_object84816:head# 2024-04-20T17:27:55.673 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.673 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a674e1dd:::benchmark_data_smithi171_122772_object121150:head# 2024-04-20T17:27:55.673 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.673 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6753f99:::benchmark_data_smithi171_122772_object168652:head# 2024-04-20T17:27:55.673 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.673 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6759ca5:::benchmark_data_smithi171_122772_object120158:head# 2024-04-20T17:27:55.673 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.673 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6762395:::benchmark_data_smithi171_122772_object114313:head# 2024-04-20T17:27:55.673 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.673 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a67782da:::benchmark_data_smithi171_122772_object134405:head# 2024-04-20T17:27:55.674 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.674 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6785bbe:::benchmark_data_smithi171_122772_object139424:head# 2024-04-20T17:27:55.674 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.674 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a678e5a6:::benchmark_data_smithi171_122772_object121266:head# 2024-04-20T17:27:55.674 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.674 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6790e03:::benchmark_data_smithi171_122772_object152362:head# 2024-04-20T17:27:55.674 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.674 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6796137:::benchmark_data_smithi171_122772_object166965:head# 2024-04-20T17:27:55.674 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.674 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a67b4ad2:::benchmark_data_smithi171_122772_object82254:head# 2024-04-20T17:27:55.674 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.675 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a67b9b5f:::benchmark_data_smithi171_122772_object88124:head# 2024-04-20T17:27:55.675 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.675 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a67bdf09:::benchmark_data_smithi171_122772_object112776:head# 2024-04-20T17:27:55.675 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.675 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a67c3640:::benchmark_data_smithi171_122772_object120129:head# 2024-04-20T17:27:55.675 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.675 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a67c3cab:::benchmark_data_smithi171_122772_object160152:head# 2024-04-20T17:27:55.675 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.675 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a67cc4e9:::benchmark_data_smithi171_122772_object93537:head# 2024-04-20T17:27:55.675 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.675 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a67d88fc:::benchmark_data_smithi171_122772_object115704:head# 2024-04-20T17:27:55.675 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.676 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a67f51d8:::benchmark_data_smithi171_122772_object134420:head# 2024-04-20T17:27:55.676 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.676 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a67f8263:::benchmark_data_smithi171_122772_object134440:head# 2024-04-20T17:27:55.676 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.676 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6801c03:::benchmark_data_smithi171_122772_object89180:head# 2024-04-20T17:27:55.676 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.676 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a68025da:::benchmark_data_smithi171_122772_object154820:head# 2024-04-20T17:27:55.676 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.676 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6811178:::benchmark_data_smithi171_122772_object130472:head# 2024-04-20T17:27:55.676 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.676 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6812c37:::benchmark_data_smithi171_122772_object164529:head# 2024-04-20T17:27:55.676 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.677 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a682ed6f:::benchmark_data_smithi171_122772_object135171:head# 2024-04-20T17:27:55.677 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.677 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a683300d:::benchmark_data_smithi171_122772_object102085:head# 2024-04-20T17:27:55.677 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.677 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a68335c2:::benchmark_data_smithi171_122772_object131133:head# 2024-04-20T17:27:55.677 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.677 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6843e1c:::benchmark_data_smithi171_122772_object82793:head# 2024-04-20T17:27:55.677 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.677 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6846e41:::benchmark_data_smithi171_122772_object96052:head# 2024-04-20T17:27:55.677 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.677 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a68479f0:::benchmark_data_smithi171_122772_object143892:head# 2024-04-20T17:27:55.677 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.678 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a685b88b:::benchmark_data_smithi171_122772_object97233:head# 2024-04-20T17:27:55.678 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.678 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a68630b5:::benchmark_data_smithi171_122772_object154196:head# 2024-04-20T17:27:55.678 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.678 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a687b1b9:::benchmark_data_smithi171_122772_object80814:head# 2024-04-20T17:27:55.678 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.678 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6884b54:::benchmark_data_smithi171_122772_object119652:head# 2024-04-20T17:27:55.678 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.678 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a68b7e71:::benchmark_data_smithi171_122772_object123700:head# 2024-04-20T17:27:55.678 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.678 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a68e657e:::benchmark_data_smithi171_122772_object126440:head# 2024-04-20T17:27:55.678 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.679 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a68e8890:::benchmark_data_smithi171_122772_object112168:head# 2024-04-20T17:27:55.679 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.679 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a68eda38:::benchmark_data_smithi171_122772_object129229:head# 2024-04-20T17:27:55.679 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.679 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a68f6931:::benchmark_data_smithi171_122772_object91431:head# 2024-04-20T17:27:55.679 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.679 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a68ffe5c:::benchmark_data_smithi171_122772_object138518:head# 2024-04-20T17:27:55.679 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.679 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6906197:::benchmark_data_smithi171_122772_object108905:head# 2024-04-20T17:27:55.679 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.680 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6907729:::benchmark_data_smithi171_122772_object91328:head# 2024-04-20T17:27:55.680 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.680 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a692861e:::benchmark_data_smithi171_122772_object156395:head# 2024-04-20T17:27:55.680 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.680 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6954c5d:::benchmark_data_smithi171_122772_object91788:head# 2024-04-20T17:27:55.680 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.680 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a698506e:::benchmark_data_smithi171_122772_object114146:head# 2024-04-20T17:27:55.680 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.680 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a698b745:::benchmark_data_smithi171_122772_object169491:head# 2024-04-20T17:27:55.680 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.680 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6993dea:::benchmark_data_smithi171_122772_object136435:head# 2024-04-20T17:27:55.680 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.680 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a699d813:::benchmark_data_smithi171_122772_object130158:head# 2024-04-20T17:27:55.681 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.681 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a69afe77:::benchmark_data_smithi171_122772_object148395:head# 2024-04-20T17:27:55.681 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.681 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a69b38e0:::benchmark_data_smithi171_122772_object153358:head# 2024-04-20T17:27:55.681 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.681 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a69ba5b4:::benchmark_data_smithi171_122772_object141719:head# 2024-04-20T17:27:55.681 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.681 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a69cae54:::benchmark_data_smithi171_122772_object144188:head# 2024-04-20T17:27:55.681 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.681 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a69d0cae:::benchmark_data_smithi171_122772_object78782:head# 2024-04-20T17:27:55.681 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.681 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a69dcdce:::benchmark_data_smithi171_122772_object123486:head# 2024-04-20T17:27:55.682 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.682 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a69eabe9:::benchmark_data_smithi171_122772_object95270:head# 2024-04-20T17:27:55.682 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.682 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a69fbeed:::benchmark_data_smithi171_122772_object140879:head# 2024-04-20T17:27:55.682 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.682 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6a0fbfe:::benchmark_data_smithi171_122772_object160255:head# 2024-04-20T17:27:55.682 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.682 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6a11f66:::benchmark_data_smithi171_122772_object145976:head# 2024-04-20T17:27:55.682 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.682 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6a21563:::benchmark_data_smithi171_122772_object154481:head# 2024-04-20T17:27:55.682 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.683 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6a2a9a7:::benchmark_data_smithi171_122772_object114104:head# 2024-04-20T17:27:55.683 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.683 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6a2fe0e:::benchmark_data_smithi171_122772_object118431:head# 2024-04-20T17:27:55.683 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.683 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6a3995d:::benchmark_data_smithi171_122772_object137284:head# 2024-04-20T17:27:55.683 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.683 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6a3d8cc:::benchmark_data_smithi171_122772_object84291:head# 2024-04-20T17:27:55.683 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.683 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6a3dc75:::benchmark_data_smithi171_122772_object165856:head# 2024-04-20T17:27:55.683 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.683 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6a3fe54:::benchmark_data_smithi171_122772_object148095:head# 2024-04-20T17:27:55.684 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.684 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6a44070:::benchmark_data_smithi171_122772_object82453:head# 2024-04-20T17:27:55.684 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.684 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6a5b10b:::benchmark_data_smithi171_122772_object83732:head# 2024-04-20T17:27:55.684 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.684 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6a5b3b9:::benchmark_data_smithi171_122772_object102635:head# 2024-04-20T17:27:55.684 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.684 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6a6f0c4:::benchmark_data_smithi171_122772_object155098:head# 2024-04-20T17:27:55.684 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.684 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6a8f6c8:::benchmark_data_smithi171_122772_object85572:head# 2024-04-20T17:27:55.684 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.685 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6a902a0:::benchmark_data_smithi171_122772_object107399:head# 2024-04-20T17:27:55.685 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.685 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6a934a3:::benchmark_data_smithi171_122772_object97498:head# 2024-04-20T17:27:55.685 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.685 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6a934d5:::benchmark_data_smithi171_122772_object128626:head# 2024-04-20T17:27:55.685 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.685 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6a93bad:::benchmark_data_smithi171_122772_object122247:head# 2024-04-20T17:27:55.685 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.685 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6aa22ed:::benchmark_data_smithi171_122772_object92347:head# 2024-04-20T17:27:55.685 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.686 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6aa8313:::benchmark_data_smithi171_122772_object110282:head# 2024-04-20T17:27:55.686 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.686 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6aa9c14:::benchmark_data_smithi171_122772_object87166:head# 2024-04-20T17:27:55.759 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.759 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6ab0d60:::benchmark_data_smithi171_122772_object117508:head# 2024-04-20T17:27:55.759 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.759 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6ab9138:::benchmark_data_smithi171_122772_object96984:head# 2024-04-20T17:27:55.759 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.760 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6abefc6:::benchmark_data_smithi171_122772_object90639:head# 2024-04-20T17:27:55.760 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.760 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6abf1a4:::benchmark_data_smithi171_122772_object164841:head# 2024-04-20T17:27:55.760 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.760 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6ac0bcd:::benchmark_data_smithi171_122772_object94139:head# 2024-04-20T17:27:55.760 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.760 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6aebf47:::benchmark_data_smithi171_122772_object96567:head# 2024-04-20T17:27:55.760 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.760 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6af245f:::benchmark_data_smithi171_122772_object152204:head# 2024-04-20T17:27:55.760 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.760 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6af9be9:::benchmark_data_smithi171_122772_object152020:head# 2024-04-20T17:27:55.761 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.761 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6b006e6:::benchmark_data_smithi171_122772_object124631:head# 2024-04-20T17:27:55.761 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.761 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6b0b82a:::benchmark_data_smithi171_122772_object162410:head# 2024-04-20T17:27:55.761 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.761 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6b0cf49:::benchmark_data_smithi171_122772_object135123:head# 2024-04-20T17:27:55.761 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.761 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6b0eebd:::benchmark_data_smithi171_122772_object108715:head# 2024-04-20T17:27:55.761 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.761 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6b24869:::benchmark_data_smithi171_122772_object105646:head# 2024-04-20T17:27:55.761 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.761 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6b3bb7c:::benchmark_data_smithi171_122772_object160174:head# 2024-04-20T17:27:55.762 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.762 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6b3c0a9:::benchmark_data_smithi171_122772_object143029:head# 2024-04-20T17:27:55.762 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.762 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6b407b0:::benchmark_data_smithi171_122772_object124088:head# 2024-04-20T17:27:55.762 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.762 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6b441cc:::benchmark_data_smithi171_122772_object126904:head# 2024-04-20T17:27:55.762 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.762 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6b539de:::benchmark_data_smithi171_122772_object155867:head# 2024-04-20T17:27:55.762 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.762 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6b67e2b:::benchmark_data_smithi171_122772_object110628:head# 2024-04-20T17:27:55.763 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.763 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6b6cee1:::benchmark_data_smithi171_122772_object141127:head# 2024-04-20T17:27:55.763 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.763 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6b86e2f:::benchmark_data_smithi171_122772_object97333:head# 2024-04-20T17:27:55.763 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.763 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6ba0f15:::benchmark_data_smithi171_122772_object123532:head# 2024-04-20T17:27:55.763 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.763 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6bb29e0:::benchmark_data_smithi171_122772_object145280:head# 2024-04-20T17:27:55.763 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.763 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6bb7e94:::benchmark_data_smithi171_122772_object87142:head# 2024-04-20T17:27:55.763 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.763 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6bc3b3e:::benchmark_data_smithi171_122772_object141938:head# 2024-04-20T17:27:55.763 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.764 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6bcf274:::benchmark_data_smithi171_122772_object91110:head# 2024-04-20T17:27:55.764 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.764 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6bd60f0:::benchmark_data_smithi171_122772_object142135:head# 2024-04-20T17:27:55.764 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.764 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6be2588:::benchmark_data_smithi171_122772_object105509:head# 2024-04-20T17:27:55.764 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.764 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6be62f9:::benchmark_data_smithi171_122772_object133788:head# 2024-04-20T17:27:55.764 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.764 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6bf20f1:::benchmark_data_smithi171_122772_object112036:head# 2024-04-20T17:27:55.764 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.764 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6c0a794:::benchmark_data_smithi171_122772_object104032:head# 2024-04-20T17:27:55.764 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.765 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6c13f8f:::benchmark_data_smithi171_122772_object131889:head# 2024-04-20T17:27:55.765 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.765 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6c151f2:::benchmark_data_smithi171_122772_object157794:head# 2024-04-20T17:27:55.765 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.765 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6c356d2:::benchmark_data_smithi171_122772_object161079:head# 2024-04-20T17:27:55.765 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.765 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6c36159:::benchmark_data_smithi171_122772_object145915:head# 2024-04-20T17:27:55.765 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.765 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6c3751c:::benchmark_data_smithi171_122772_object94516:head# 2024-04-20T17:27:55.765 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.765 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6c3cb37:::benchmark_data_smithi171_122772_object144949:head# 2024-04-20T17:27:55.765 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.766 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6c415ad:::benchmark_data_smithi171_122772_object126705:head# 2024-04-20T17:27:55.766 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.766 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6c45573:::benchmark_data_smithi171_122772_object103409:head# 2024-04-20T17:27:55.766 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.766 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6c4b29d:::benchmark_data_smithi171_122772_object107849:head# 2024-04-20T17:27:55.766 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.766 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6c4ccdc:::benchmark_data_smithi171_122772_object122347:head# 2024-04-20T17:27:55.766 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.766 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6c50145:::benchmark_data_smithi171_122772_object78895:head# 2024-04-20T17:27:55.766 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.766 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6c5380d:::benchmark_data_smithi171_122772_object161130:head# 2024-04-20T17:27:55.767 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.767 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6c610c8:::benchmark_data_smithi171_122772_object104738:head# 2024-04-20T17:27:55.767 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.767 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6c6b0d5:::benchmark_data_smithi171_122772_object166140:head# 2024-04-20T17:27:55.767 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.767 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6c6cd9b:::benchmark_data_smithi171_122772_object93895:head# 2024-04-20T17:27:55.767 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.767 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6c70fd6:::benchmark_data_smithi171_122772_object138856:head# 2024-04-20T17:27:55.767 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.767 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6c74256:::benchmark_data_smithi171_122772_object115163:head# 2024-04-20T17:27:55.767 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.768 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6c7b3ff:::benchmark_data_smithi171_122772_object162550:head# 2024-04-20T17:27:55.768 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.768 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6c86d9d:::benchmark_data_smithi171_122772_object120593:head# 2024-04-20T17:27:55.768 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.768 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6ca0176:::benchmark_data_smithi171_122772_object154938:head# 2024-04-20T17:27:55.768 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.768 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6ca3587:::benchmark_data_smithi171_122772_object162289:head# 2024-04-20T17:27:55.768 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.768 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6cae3d7:::benchmark_data_smithi171_122772_object148377:head# 2024-04-20T17:27:55.768 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.768 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6cba535:::benchmark_data_smithi171_122772_object155183:head# 2024-04-20T17:27:55.769 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.769 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6cbf451:::benchmark_data_smithi171_122772_object157328:head# 2024-04-20T17:27:55.769 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.769 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6cc3d45:::benchmark_data_smithi171_122772_object115445:head# 2024-04-20T17:27:55.769 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.769 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6ccf194:::benchmark_data_smithi171_122772_object85430:head# 2024-04-20T17:27:55.769 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.769 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6ce9bc5:::benchmark_data_smithi171_122772_object147962:head# 2024-04-20T17:27:55.770 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.770 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6ceb662:::benchmark_data_smithi171_122772_object89733:head# 2024-04-20T17:27:55.770 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.770 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6cfd0be:::benchmark_data_smithi171_122772_object168466:head# 2024-04-20T17:27:55.770 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.770 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6cff26d:::benchmark_data_smithi171_122772_object102656:head# 2024-04-20T17:27:55.770 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.770 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6d0686f:::benchmark_data_smithi171_122772_object112005:head# 2024-04-20T17:27:55.770 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.770 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6d14435:::benchmark_data_smithi171_122772_object121335:head# 2024-04-20T17:27:55.770 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.771 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6d1932f:::benchmark_data_smithi171_122772_object165013:head# 2024-04-20T17:27:55.771 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.771 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6d21878:::benchmark_data_smithi171_122772_object134834:head# 2024-04-20T17:27:55.771 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.771 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6d32d7f:::benchmark_data_smithi171_122772_object152597:head# 2024-04-20T17:27:55.771 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.771 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6d3d962:::benchmark_data_smithi171_122772_object167764:head# 2024-04-20T17:27:55.771 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.771 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6d52a05:::benchmark_data_smithi171_122772_object80197:head# 2024-04-20T17:27:55.771 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.771 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6d5d800:::benchmark_data_smithi171_122772_object91283:head# 2024-04-20T17:27:55.772 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.772 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6d67a6b:::benchmark_data_smithi171_122772_object162549:head# 2024-04-20T17:27:55.772 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.772 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6d7392d:::benchmark_data_smithi171_122772_object88713:head# 2024-04-20T17:27:55.772 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.772 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6da18f8:::benchmark_data_smithi171_122772_object105697:head# 2024-04-20T17:27:55.772 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.772 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6daf88a:::benchmark_data_smithi171_122772_object96222:head# 2024-04-20T17:27:55.772 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.772 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6db5471:::benchmark_data_smithi171_122772_object130565:head# 2024-04-20T17:27:55.772 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.772 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6dbdad2:::benchmark_data_smithi171_122772_object92649:head# 2024-04-20T17:27:55.773 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.773 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6dc20aa:::benchmark_data_smithi171_122772_object130443:head# 2024-04-20T17:27:55.773 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.773 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6dc27c7:::benchmark_data_smithi171_122772_object118171:head# 2024-04-20T17:27:55.773 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.773 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6dcfe53:::benchmark_data_smithi171_122772_object141227:head# 2024-04-20T17:27:55.773 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.773 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6defdab:::benchmark_data_smithi171_122772_object111698:head# 2024-04-20T17:27:55.773 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.773 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6df6d2a:::benchmark_data_smithi171_122772_object96320:head# 2024-04-20T17:27:55.773 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.773 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6df9ccc:::benchmark_data_smithi171_122772_object81470:head# 2024-04-20T17:27:55.774 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.774 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6df9fe8:::benchmark_data_smithi171_122772_object126856:head# 2024-04-20T17:27:55.774 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.774 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6dfb968:::benchmark_data_smithi171_122772_object105900:head# 2024-04-20T17:27:55.774 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.774 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6dfd7e5:::benchmark_data_smithi171_122772_object113039:head# 2024-04-20T17:27:55.774 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.774 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6e06f7f:::benchmark_data_smithi171_122772_object124374:head# 2024-04-20T17:27:55.774 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.774 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6e0a72a:::benchmark_data_smithi171_122772_object146307:head# 2024-04-20T17:27:55.774 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.775 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6e0d575:::benchmark_data_smithi171_122772_object148323:head# 2024-04-20T17:27:55.775 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.775 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6e20997:::benchmark_data_smithi171_122772_object104995:head# 2024-04-20T17:27:55.775 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.775 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6e23425:::benchmark_data_smithi171_122772_object121466:head# 2024-04-20T17:27:55.775 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.775 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6e282b3:::benchmark_data_smithi171_122772_object132679:head# 2024-04-20T17:27:55.775 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.775 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6e40fbd:::benchmark_data_smithi171_122772_object93766:head# 2024-04-20T17:27:55.775 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.775 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6e5b938:::benchmark_data_smithi171_122772_object99003:head# 2024-04-20T17:27:55.775 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.776 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6e7d15b:::benchmark_data_smithi171_122772_object167983:head# 2024-04-20T17:27:55.776 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.776 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6e85078:::benchmark_data_smithi171_122772_object126318:head# 2024-04-20T17:27:55.776 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.776 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6e912ab:::benchmark_data_smithi171_122772_object87900:head# 2024-04-20T17:27:55.776 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.776 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6ea11c4:::benchmark_data_smithi171_122772_object120931:head# 2024-04-20T17:27:55.776 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.776 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6eae928:::benchmark_data_smithi171_122772_object83586:head# 2024-04-20T17:27:55.776 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.776 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6eb162d:::benchmark_data_smithi171_122772_object133559:head# 2024-04-20T17:27:55.776 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.777 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6ec23b9:::benchmark_data_smithi171_122772_object102716:head# 2024-04-20T17:27:55.777 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.777 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6ed4a37:::benchmark_data_smithi171_122772_object110070:head# 2024-04-20T17:27:55.777 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.777 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6ed9fbe:::benchmark_data_smithi171_122772_object106218:head# 2024-04-20T17:27:55.777 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.777 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6edb182:::benchmark_data_smithi171_122772_object102240:head# 2024-04-20T17:27:55.777 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.777 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6eddbf7:::benchmark_data_smithi171_122772_object116247:head# 2024-04-20T17:27:55.777 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.777 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6ee1fbd:::benchmark_data_smithi171_122772_object118830:head# 2024-04-20T17:27:55.777 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.778 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6ef92f1:::benchmark_data_smithi171_122772_object124375:head# 2024-04-20T17:27:55.778 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.778 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6f03d61:::benchmark_data_smithi171_122772_object149018:head# 2024-04-20T17:27:55.778 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.778 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6f04a68:::benchmark_data_smithi171_122772_object155763:head# 2024-04-20T17:27:55.778 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.778 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6f0ce0b:::benchmark_data_smithi171_122772_object166708:head# 2024-04-20T17:27:55.778 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.778 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6f1be5d:::benchmark_data_smithi171_122772_object139417:head# 2024-04-20T17:27:55.778 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.778 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6f38b0e:::benchmark_data_smithi171_122772_object79538:head# 2024-04-20T17:27:55.779 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.779 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6f43530:::benchmark_data_smithi171_122772_object169267:head# 2024-04-20T17:27:55.779 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.779 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6f4b031:::benchmark_data_smithi171_122772_object119525:head# 2024-04-20T17:27:55.779 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.779 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6f52240:::benchmark_data_smithi171_122772_object125689:head# 2024-04-20T17:27:55.779 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.779 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6f53d2b:::benchmark_data_smithi171_122772_object125752:head# 2024-04-20T17:27:55.779 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.779 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6f5530b:::benchmark_data_smithi171_122772_object107241:head# 2024-04-20T17:27:55.779 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.780 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6f5d2b1:::benchmark_data_smithi171_122772_object81481:head# 2024-04-20T17:27:55.780 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.780 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6f5f8ad:::benchmark_data_smithi171_122772_object140423:head# 2024-04-20T17:27:55.780 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.780 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6f6d344:::benchmark_data_smithi171_122772_object88214:head# 2024-04-20T17:27:55.780 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.780 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6f6e4ed:::benchmark_data_smithi171_122772_object167352:head# 2024-04-20T17:27:55.780 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.780 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6f76b09:::benchmark_data_smithi171_122772_object168501:head# 2024-04-20T17:27:55.780 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.780 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6f8c00a:::benchmark_data_smithi171_122772_object90491:head# 2024-04-20T17:27:55.780 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.781 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6fa2f39:::benchmark_data_smithi171_122772_object161941:head# 2024-04-20T17:27:55.781 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.781 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6fab74f:::benchmark_data_smithi171_122772_object163454:head# 2024-04-20T17:27:55.781 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.781 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6fad2b3:::benchmark_data_smithi171_122772_object89168:head# 2024-04-20T17:27:55.781 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.781 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6fbf3ac:::benchmark_data_smithi171_122772_object111125:head# 2024-04-20T17:27:55.781 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.781 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6fc16e8:::benchmark_data_smithi171_122772_object92959:head# 2024-04-20T17:27:55.781 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.781 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6fc5e88:::benchmark_data_smithi171_122772_object167452:head# 2024-04-20T17:27:55.781 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.782 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6fd16d1:::benchmark_data_smithi171_122772_object112063:head# 2024-04-20T17:27:55.782 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.782 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6fd2804:::benchmark_data_smithi171_122772_object98071:head# 2024-04-20T17:27:55.782 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.782 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6fd2a9a:::benchmark_data_smithi171_122772_object111733:head# 2024-04-20T17:27:55.782 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.782 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6fd3c30:::benchmark_data_smithi171_122772_object117044:head# 2024-04-20T17:27:55.782 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.782 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6fe4cd8:::benchmark_data_smithi171_122772_object131989:head# 2024-04-20T17:27:55.782 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.782 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6fed86c:::benchmark_data_smithi171_122772_object89471:head# 2024-04-20T17:27:55.782 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.783 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a6ff2f2f:::benchmark_data_smithi171_122772_object100152:head# 2024-04-20T17:27:55.783 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.783 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a700999d:::benchmark_data_smithi171_122772_object111280:head# 2024-04-20T17:27:55.783 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.783 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7013c91:::benchmark_data_smithi171_122772_object88911:head# 2024-04-20T17:27:55.783 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.783 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7017bcb:::benchmark_data_smithi171_122772_object83727:head# 2024-04-20T17:27:55.783 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.783 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a702540c:::benchmark_data_smithi171_122772_object167108:head# 2024-04-20T17:27:55.783 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.783 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a703d212:::benchmark_data_smithi171_122772_object169030:head# 2024-04-20T17:27:55.783 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.783 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a704912b:::benchmark_data_smithi171_122772_object135766:head# 2024-04-20T17:27:55.784 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.784 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a70515ee:::benchmark_data_smithi171_122772_object82373:head# 2024-04-20T17:27:55.784 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.784 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7051675:::benchmark_data_smithi171_122772_object118352:head# 2024-04-20T17:27:55.784 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.784 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7055481:::benchmark_data_smithi171_122772_object109816:head# 2024-04-20T17:27:55.784 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.784 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7055647:::benchmark_data_smithi171_122772_object98664:head# 2024-04-20T17:27:55.784 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.784 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a70761c5:::benchmark_data_smithi171_122772_object152282:head# 2024-04-20T17:27:55.784 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.784 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7080b2a:::benchmark_data_smithi171_122772_object138443:head# 2024-04-20T17:27:55.785 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.785 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a70a2f84:::benchmark_data_smithi171_122772_object149898:head# 2024-04-20T17:27:55.785 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.785 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a70b0f3c:::benchmark_data_smithi171_122772_object125821:head# 2024-04-20T17:27:55.785 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.785 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a70b6069:::benchmark_data_smithi171_122772_object159281:head# 2024-04-20T17:27:55.785 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.785 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a70cc585:::benchmark_data_smithi171_122772_object125515:head# 2024-04-20T17:27:55.785 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.785 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a70d240b:::benchmark_data_smithi171_122772_object92240:head# 2024-04-20T17:27:55.785 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.785 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a70d3e03:::benchmark_data_smithi171_122772_object130457:head# 2024-04-20T17:27:55.785 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.786 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a70d7ad0:::benchmark_data_smithi171_122772_object131383:head# 2024-04-20T17:27:55.786 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.786 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a70dc902:::benchmark_data_smithi171_122772_object84410:head# 2024-04-20T17:27:55.786 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.786 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a70dd961:::benchmark_data_smithi171_122772_object120723:head# 2024-04-20T17:27:55.786 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.786 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a70e3e36:::benchmark_data_smithi171_122772_object109462:head# 2024-04-20T17:27:55.786 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.786 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a70e5c88:::benchmark_data_smithi171_122772_object133461:head# 2024-04-20T17:27:55.786 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.786 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a70e78da:::benchmark_data_smithi171_122772_object118334:head# 2024-04-20T17:27:55.786 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.787 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a70fab50:::benchmark_data_smithi171_122772_object80844:head# 2024-04-20T17:27:55.787 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.787 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a70fc853:::benchmark_data_smithi171_122772_object96338:head# 2024-04-20T17:27:55.787 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.787 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a71096ba:::benchmark_data_smithi171_122772_object117895:head# 2024-04-20T17:27:55.787 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.787 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a711127c:::benchmark_data_smithi171_122772_object116166:head# 2024-04-20T17:27:55.787 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.787 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a711ae11:::benchmark_data_smithi171_122772_object105202:head# 2024-04-20T17:27:55.787 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.787 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a711d402:::benchmark_data_smithi171_122772_object84377:head# 2024-04-20T17:27:55.787 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.788 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7121bf5:::benchmark_data_smithi171_122772_object112089:head# 2024-04-20T17:27:55.788 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.788 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7126c92:::benchmark_data_smithi171_122772_object146589:head# 2024-04-20T17:27:55.788 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.788 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a712b787:::benchmark_data_smithi171_122772_object159735:head# 2024-04-20T17:27:55.788 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.788 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7133369:::benchmark_data_smithi171_122772_object161931:head# 2024-04-20T17:27:55.788 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.788 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a714a6cb:::benchmark_data_smithi171_122772_object155315:head# 2024-04-20T17:27:55.788 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.788 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a715ba6b:::benchmark_data_smithi171_122772_object79568:head# 2024-04-20T17:27:55.788 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.789 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a715f8b2:::benchmark_data_smithi171_122772_object105156:head# 2024-04-20T17:27:55.789 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.789 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a717984c:::benchmark_data_smithi171_122772_object129333:head# 2024-04-20T17:27:55.789 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.789 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a71a811c:::benchmark_data_smithi171_122772_object102289:head# 2024-04-20T17:27:55.789 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.789 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a71af9c2:::benchmark_data_smithi171_122772_object141908:head# 2024-04-20T17:27:55.789 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.789 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a71c789d:::benchmark_data_smithi171_122772_object91497:head# 2024-04-20T17:27:55.789 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.789 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a71d7d10:::benchmark_data_smithi171_122772_object136851:head# 2024-04-20T17:27:55.789 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.790 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a71dbc10:::benchmark_data_smithi171_122772_object155372:head# 2024-04-20T17:27:55.790 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.790 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a71eb790:::benchmark_data_smithi171_122772_object137820:head# 2024-04-20T17:27:55.790 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.790 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7202654:::benchmark_data_smithi171_122772_object123687:head# 2024-04-20T17:27:55.790 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.790 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7209fa1:::benchmark_data_smithi171_122772_object112745:head# 2024-04-20T17:27:55.790 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.790 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a720cdc4:::benchmark_data_smithi171_122772_object112151:head# 2024-04-20T17:27:55.790 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.790 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a720f870:::benchmark_data_smithi171_122772_object157291:head# 2024-04-20T17:27:55.790 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.791 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a722d5cf:::benchmark_data_smithi171_122772_object134632:head# 2024-04-20T17:27:55.791 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.791 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a722e208:::benchmark_data_smithi171_122772_object92703:head# 2024-04-20T17:27:55.791 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.791 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:27:55.792 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:55 smithi005 ceph-mon[39562]: pgmap v1363: 17 pgs: 17 active+clean; 5.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 642 op/s; 34 MiB/s, 546 objects/s recovering 2024-04-20T17:27:55.793 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:27:55.831 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:55.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:55 smithi171 ceph-mon[44189]: pgmap v1363: 17 pgs: 17 active+clean; 5.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 642 op/s; 34 MiB/s, 546 objects/s recovering 2024-04-20T17:27:55.866 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a723bc4c:::benchmark_data_smithi171_122772_object88846:head# 2024-04-20T17:27:55.866 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.866 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a723c0fc:::benchmark_data_smithi171_122772_object146782:head# 2024-04-20T17:27:55.866 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.866 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a72422f9:::benchmark_data_smithi171_122772_object163694:head# 2024-04-20T17:27:55.866 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.866 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a725413b:::benchmark_data_smithi171_122772_object147098:head# 2024-04-20T17:27:55.867 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.867 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7255665:::benchmark_data_smithi171_122772_object137411:head# 2024-04-20T17:27:55.867 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.867 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7260f57:::benchmark_data_smithi171_122772_object150000:head# 2024-04-20T17:27:55.867 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.867 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7261544:::benchmark_data_smithi171_122772_object132854:head# 2024-04-20T17:27:55.867 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.867 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7263c00:::benchmark_data_smithi171_122772_object145263:head# 2024-04-20T17:27:55.867 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.867 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a726991a:::benchmark_data_smithi171_122772_object117808:head# 2024-04-20T17:27:55.867 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.867 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a726de43:::benchmark_data_smithi171_122772_object168471:head# 2024-04-20T17:27:55.868 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.868 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a727113a:::benchmark_data_smithi171_122772_object119084:head# 2024-04-20T17:27:55.868 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.868 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a72713d5:::benchmark_data_smithi171_122772_object161773:head# 2024-04-20T17:27:55.868 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.868 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a72741c9:::benchmark_data_smithi171_122772_object137751:head# 2024-04-20T17:27:55.868 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.868 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a727f7a3:::benchmark_data_smithi171_122772_object90142:head# 2024-04-20T17:27:55.868 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.868 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a728bdda:::benchmark_data_smithi171_122772_object150140:head# 2024-04-20T17:27:55.868 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.868 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7299672:::benchmark_data_smithi171_122772_object143806:head# 2024-04-20T17:27:55.869 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.869 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a729d72b:::benchmark_data_smithi171_122772_object147656:head# 2024-04-20T17:27:55.869 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.869 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a72a01df:::benchmark_data_smithi171_122772_object113541:head# 2024-04-20T17:27:55.869 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.869 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a72b2e6f:::benchmark_data_smithi171_122772_object132442:head# 2024-04-20T17:27:55.869 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.869 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a72b7428:::benchmark_data_smithi171_122772_object93992:head# 2024-04-20T17:27:55.869 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.869 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a72b803d:::benchmark_data_smithi171_122772_object138818:head# 2024-04-20T17:27:55.869 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.869 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a72d6ec5:::benchmark_data_smithi171_122772_object120481:head# 2024-04-20T17:27:55.870 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.870 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a72e1b23:::benchmark_data_smithi171_122772_object102326:head# 2024-04-20T17:27:55.870 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.870 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a72e51fb:::benchmark_data_smithi171_122772_object106504:head# 2024-04-20T17:27:55.870 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.870 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7308c73:::benchmark_data_smithi171_122772_object101045:head# 2024-04-20T17:27:55.870 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.870 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a730db76:::benchmark_data_smithi171_122772_object84803:head# 2024-04-20T17:27:55.870 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.870 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a731075e:::benchmark_data_smithi171_122772_object138135:head# 2024-04-20T17:27:55.870 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.870 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a73128b3:::benchmark_data_smithi171_122772_object85139:head# 2024-04-20T17:27:55.871 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.871 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7321cab:::benchmark_data_smithi171_122772_object84747:head# 2024-04-20T17:27:55.871 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.871 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a732ea28:::benchmark_data_smithi171_122772_object118869:head# 2024-04-20T17:27:55.871 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.871 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a734127e:::benchmark_data_smithi171_122772_object116851:head# 2024-04-20T17:27:55.871 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.871 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a735bb1f:::benchmark_data_smithi171_122772_object144090:head# 2024-04-20T17:27:55.871 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.871 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7363a33:::benchmark_data_smithi171_122772_object155618:head# 2024-04-20T17:27:55.871 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.871 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a736e188:::benchmark_data_smithi171_122772_object81738:head# 2024-04-20T17:27:55.872 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.872 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7372913:::benchmark_data_smithi171_122772_object114133:head# 2024-04-20T17:27:55.872 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.872 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a738601a:::benchmark_data_smithi171_122772_object132005:head# 2024-04-20T17:27:55.872 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.872 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a738779d:::benchmark_data_smithi171_122772_object144668:head# 2024-04-20T17:27:55.872 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.872 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7395bef:::benchmark_data_smithi171_122772_object129093:head# 2024-04-20T17:27:55.872 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.872 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a73a45ed:::benchmark_data_smithi171_122772_object91165:head# 2024-04-20T17:27:55.872 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.872 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a73aba83:::benchmark_data_smithi171_122772_object106472:head# 2024-04-20T17:27:55.873 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.873 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a73b6b15:::benchmark_data_smithi171_122772_object84254:head# 2024-04-20T17:27:55.873 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.873 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a73bbf34:::benchmark_data_smithi171_122772_object133912:head# 2024-04-20T17:27:55.873 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.873 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a73ceede:::benchmark_data_smithi171_122772_object98770:head# 2024-04-20T17:27:55.873 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.873 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a73cf79d:::benchmark_data_smithi171_122772_object106558:head# 2024-04-20T17:27:55.873 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.873 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7413fbb:::benchmark_data_smithi171_122772_object141751:head# 2024-04-20T17:27:55.873 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.873 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a742c794:::benchmark_data_smithi171_122772_object84546:head# 2024-04-20T17:27:55.873 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.874 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a745a855:::benchmark_data_smithi171_122772_object130509:head# 2024-04-20T17:27:55.874 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.874 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a747e3d2:::benchmark_data_smithi171_122772_object148620:head# 2024-04-20T17:27:55.874 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.874 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7496790:::benchmark_data_smithi171_122772_object131497:head# 2024-04-20T17:27:55.874 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.874 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a749cd16:::benchmark_data_smithi171_122772_object121170:head# 2024-04-20T17:27:55.874 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.874 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a74ba378:::benchmark_data_smithi171_122772_object110596:head# 2024-04-20T17:27:55.874 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.875 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a74c9c4b:::benchmark_data_smithi171_122772_object159839:head# 2024-04-20T17:27:55.875 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.875 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a74f23bc:::benchmark_data_smithi171_122772_object145686:head# 2024-04-20T17:27:55.875 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.875 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a74f2879:::benchmark_data_smithi171_122772_object169205:head# 2024-04-20T17:27:55.875 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.875 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a74ff05d:::benchmark_data_smithi171_122772_object137886:head# 2024-04-20T17:27:55.875 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.875 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7502eac:::benchmark_data_smithi171_122772_object142498:head# 2024-04-20T17:27:55.875 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.875 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7508e54:::benchmark_data_smithi171_122772_object103351:head# 2024-04-20T17:27:55.875 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.876 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7510b77:::benchmark_data_smithi171_122772_object169404:head# 2024-04-20T17:27:55.876 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.876 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a751d9f5:::benchmark_data_smithi171_122772_object152570:head# 2024-04-20T17:27:55.876 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.876 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7527862:::benchmark_data_smithi171_122772_object90107:head# 2024-04-20T17:27:55.876 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.876 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7531f40:::benchmark_data_smithi171_122772_object143465:head# 2024-04-20T17:27:55.876 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.876 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7535374:::benchmark_data_smithi171_122772_object158358:head# 2024-04-20T17:27:55.876 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.876 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a754129e:::benchmark_data_smithi171_122772_object107454:head# 2024-04-20T17:27:55.876 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.876 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7554c11:::benchmark_data_smithi171_122772_object125270:head# 2024-04-20T17:27:55.877 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.877 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a756afd5:::benchmark_data_smithi171_122772_object149640:head# 2024-04-20T17:27:55.877 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.877 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a757b64d:::benchmark_data_smithi171_122772_object115450:head# 2024-04-20T17:27:55.877 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.877 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7583f65:::benchmark_data_smithi171_122772_object129928:head# 2024-04-20T17:27:55.877 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.877 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a75890e4:::benchmark_data_smithi171_122772_object78020:head# 2024-04-20T17:27:55.877 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.877 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a758c317:::benchmark_data_smithi171_122772_object141936:head# 2024-04-20T17:27:55.877 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.877 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a759e887:::benchmark_data_smithi171_122772_object153110:head# 2024-04-20T17:27:55.877 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.878 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a75bee81:::benchmark_data_smithi171_122772_object150621:head# 2024-04-20T17:27:55.878 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.878 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a75cad7c:::benchmark_data_smithi171_122772_object149922:head# 2024-04-20T17:27:55.878 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.878 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a75d466e:::benchmark_data_smithi171_122772_object117131:head# 2024-04-20T17:27:55.878 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.878 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a75df1d2:::benchmark_data_smithi171_122772_object126804:head# 2024-04-20T17:27:55.878 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.878 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a75e266b:::benchmark_data_smithi171_122772_object153387:head# 2024-04-20T17:27:55.878 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.878 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a75e47d1:::benchmark_data_smithi171_122772_object80474:head# 2024-04-20T17:27:55.878 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.879 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a75e5a78:::benchmark_data_smithi171_122772_object142518:head# 2024-04-20T17:27:55.879 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.879 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a75e91ee:::benchmark_data_smithi171_122772_object102140:head# 2024-04-20T17:27:55.879 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.879 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a75ed982:::benchmark_data_smithi171_122772_object80540:head# 2024-04-20T17:27:55.879 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.879 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7632a5c:::benchmark_data_smithi171_122772_object147226:head# 2024-04-20T17:27:55.879 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.879 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a764155b:::benchmark_data_smithi171_122772_object115087:head# 2024-04-20T17:27:55.879 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.879 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7645eae:::benchmark_data_smithi171_122772_object90467:head# 2024-04-20T17:27:55.879 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.880 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a764d057:::benchmark_data_smithi171_122772_object140454:head# 2024-04-20T17:27:55.880 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.880 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a76590c7:::benchmark_data_smithi171_122772_object148708:head# 2024-04-20T17:27:55.880 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.880 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a767857d:::benchmark_data_smithi171_122772_object111447:head# 2024-04-20T17:27:55.880 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.880 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a767b8ab:::benchmark_data_smithi171_122772_object167275:head# 2024-04-20T17:27:55.880 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.880 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7686e93:::benchmark_data_smithi171_122772_object83639:head# 2024-04-20T17:27:55.880 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.880 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a769cb8a:::benchmark_data_smithi171_122772_object110788:head# 2024-04-20T17:27:55.880 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.881 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a76a3114:::benchmark_data_smithi171_122772_object149702:head# 2024-04-20T17:27:55.881 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.881 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a76a5798:::benchmark_data_smithi171_122772_object163650:head# 2024-04-20T17:27:55.881 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.881 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a76bb844:::benchmark_data_smithi171_122772_object143022:head# 2024-04-20T17:27:55.881 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.881 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a76c039a:::benchmark_data_smithi171_122772_object87979:head# 2024-04-20T17:27:55.881 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.881 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a76cee39:::benchmark_data_smithi171_122772_object132920:head# 2024-04-20T17:27:55.881 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.881 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a76d4521:::benchmark_data_smithi171_122772_object91633:head# 2024-04-20T17:27:55.881 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.881 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a76d8d09:::benchmark_data_smithi171_122772_object101511:head# 2024-04-20T17:27:55.882 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.882 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a76deff1:::benchmark_data_smithi171_122772_object145439:head# 2024-04-20T17:27:55.882 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.882 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a76f1853:::benchmark_data_smithi171_122772_object120977:head# 2024-04-20T17:27:55.882 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.882 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a76f208b:::benchmark_data_smithi171_122772_object155123:head# 2024-04-20T17:27:55.882 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.882 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a76f33ae:::benchmark_data_smithi171_122772_object98799:head# 2024-04-20T17:27:55.882 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.882 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a770daa1:::benchmark_data_smithi171_122772_object105959:head# 2024-04-20T17:27:55.882 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.883 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7718d9f:::benchmark_data_smithi171_122772_object90068:head# 2024-04-20T17:27:55.883 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.883 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a77246c8:::benchmark_data_smithi171_122772_object102322:head# 2024-04-20T17:27:55.883 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.883 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7727eb4:::benchmark_data_smithi171_122772_object145895:head# 2024-04-20T17:27:55.883 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.883 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7735424:::benchmark_data_smithi171_122772_object151987:head# 2024-04-20T17:27:55.883 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.883 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a773f190:::benchmark_data_smithi171_122772_object164973:head# 2024-04-20T17:27:55.883 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.883 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a77560f9:::benchmark_data_smithi171_122772_object93653:head# 2024-04-20T17:27:55.883 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.884 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a775da7c:::benchmark_data_smithi171_122772_object107008:head# 2024-04-20T17:27:55.884 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.884 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7772f78:::benchmark_data_smithi171_122772_object102949:head# 2024-04-20T17:27:55.884 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.884 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7775aa6:::benchmark_data_smithi171_122772_object169282:head# 2024-04-20T17:27:55.884 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.884 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a777cc64:::benchmark_data_smithi171_122772_object113953:head# 2024-04-20T17:27:55.884 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.884 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a77888de:::benchmark_data_smithi171_122772_object166481:head# 2024-04-20T17:27:55.884 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.884 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a778ec95:::benchmark_data_smithi171_122772_object111548:head# 2024-04-20T17:27:55.884 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.885 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7796c14:::benchmark_data_smithi171_122772_object134596:head# 2024-04-20T17:27:55.885 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.885 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a77a21c2:::benchmark_data_smithi171_122772_object140570:head# 2024-04-20T17:27:55.885 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.885 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a77b3e05:::benchmark_data_smithi171_122772_object94213:head# 2024-04-20T17:27:55.885 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.885 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a77b6829:::benchmark_data_smithi171_122772_object88859:head# 2024-04-20T17:27:55.885 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.885 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a77c9c3b:::benchmark_data_smithi171_122772_object152722:head# 2024-04-20T17:27:55.885 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.885 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a77cfaa7:::benchmark_data_smithi171_122772_object98906:head# 2024-04-20T17:27:55.886 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.886 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a77e7ef2:::benchmark_data_smithi171_122772_object155796:head# 2024-04-20T17:27:55.886 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.886 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a77ed79b:::benchmark_data_smithi171_122772_object146765:head# 2024-04-20T17:27:55.886 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.886 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a77fcf97:::benchmark_data_smithi171_122772_object125558:head# 2024-04-20T17:27:55.886 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.886 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7808924:::benchmark_data_smithi171_122772_object116972:head# 2024-04-20T17:27:55.886 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.886 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a780b4b9:::benchmark_data_smithi171_122772_object86132:head# 2024-04-20T17:27:55.886 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.886 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7812be8:::benchmark_data_smithi171_122772_object156453:head# 2024-04-20T17:27:55.887 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.887 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a782082a:::benchmark_data_smithi171_122772_object87529:head# 2024-04-20T17:27:55.887 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.887 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7836eba:::benchmark_data_smithi171_122772_object160399:head# 2024-04-20T17:27:55.887 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.887 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a783819a:::benchmark_data_smithi171_122772_object167287:head# 2024-04-20T17:27:55.887 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.887 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a783f8d4:::benchmark_data_smithi171_122772_object155498:head# 2024-04-20T17:27:55.887 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.887 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a785eeff:::benchmark_data_smithi171_122772_object82592:head# 2024-04-20T17:27:55.887 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.887 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7867a08:::benchmark_data_smithi171_122772_object141489:head# 2024-04-20T17:27:55.888 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.888 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7868f63:::benchmark_data_smithi171_122772_object167125:head# 2024-04-20T17:27:55.888 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.888 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a786d63f:::benchmark_data_smithi171_122772_object96287:head# 2024-04-20T17:27:55.888 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.888 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a78842ae:::benchmark_data_smithi171_122772_object81124:head# 2024-04-20T17:27:55.888 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.888 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a788ecf8:::benchmark_data_smithi171_122772_object137954:head# 2024-04-20T17:27:55.888 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.888 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a789e003:::benchmark_data_smithi171_122772_object117489:head# 2024-04-20T17:27:55.888 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.888 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a78b8e5c:::benchmark_data_smithi171_122772_object106461:head# 2024-04-20T17:27:55.888 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a78d6be8:::benchmark_data_smithi171_122772_object150352:head# 2024-04-20T17:27:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a78dc97c:::benchmark_data_smithi171_122772_object78047:head# 2024-04-20T17:27:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a78eaef6:::benchmark_data_smithi171_122772_object123438:head# 2024-04-20T17:27:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a78f62ca:::benchmark_data_smithi171_122772_object93065:head# 2024-04-20T17:27:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a78fda7a:::benchmark_data_smithi171_122772_object153517:head# 2024-04-20T17:27:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a78fe255:::benchmark_data_smithi171_122772_object127137:head# 2024-04-20T17:27:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7916af1:::benchmark_data_smithi171_122772_object108347:head# 2024-04-20T17:27:55.890 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.890 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a792950d:::benchmark_data_smithi171_122772_object140130:head# 2024-04-20T17:27:55.890 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.890 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a792a978:::benchmark_data_smithi171_122772_object88388:head# 2024-04-20T17:27:55.890 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.890 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a792c241:::benchmark_data_smithi171_122772_object163010:head# 2024-04-20T17:27:55.890 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.890 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a793192b:::benchmark_data_smithi171_122772_object79535:head# 2024-04-20T17:27:55.890 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.890 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7936255:::benchmark_data_smithi171_122772_object167796:head# 2024-04-20T17:27:55.890 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.890 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a793ca22:::benchmark_data_smithi171_122772_object112878:head# 2024-04-20T17:27:55.890 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.891 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a793d32a:::benchmark_data_smithi171_122772_object124959:head# 2024-04-20T17:27:55.891 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.891 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a793ff1c:::benchmark_data_smithi171_122772_object83325:head# 2024-04-20T17:27:55.891 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.891 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a794c663:::benchmark_data_smithi171_122772_object155296:head# 2024-04-20T17:27:55.891 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.891 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a794d404:::benchmark_data_smithi171_122772_object94374:head# 2024-04-20T17:27:55.891 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.891 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a794d5aa:::benchmark_data_smithi171_122772_object80010:head# 2024-04-20T17:27:55.891 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.891 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7952dc8:::benchmark_data_smithi171_122772_object128006:head# 2024-04-20T17:27:55.891 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.891 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a79680a1:::benchmark_data_smithi171_122772_object125333:head# 2024-04-20T17:27:55.892 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.892 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7972b0f:::benchmark_data_smithi171_122772_object143946:head# 2024-04-20T17:27:55.892 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.892 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7974a76:::benchmark_data_smithi171_122772_object112502:head# 2024-04-20T17:27:55.892 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.892 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a797756b:::benchmark_data_smithi171_122772_object139769:head# 2024-04-20T17:27:55.892 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.892 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a797e23b:::benchmark_data_smithi171_122772_object102678:head# 2024-04-20T17:27:55.892 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.892 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a797f196:::benchmark_data_smithi171_122772_object114795:head# 2024-04-20T17:27:55.892 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.892 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a79837f4:::benchmark_data_smithi171_122772_object119672:head# 2024-04-20T17:27:55.892 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.893 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a798dcce:::benchmark_data_smithi171_122772_object110591:head# 2024-04-20T17:27:55.893 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.893 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a798fe3b:::benchmark_data_smithi171_122772_object108688:head# 2024-04-20T17:27:55.893 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.893 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a799be26:::benchmark_data_smithi171_122772_object134223:head# 2024-04-20T17:27:55.893 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.893 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a79a9978:::benchmark_data_smithi171_122772_object152596:head# 2024-04-20T17:27:55.893 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.893 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a79c173d:::benchmark_data_smithi171_122772_object133189:head# 2024-04-20T17:27:55.893 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.893 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a79cba46:::benchmark_data_smithi171_122772_object155344:head# 2024-04-20T17:27:55.894 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.894 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a79d9228:::benchmark_data_smithi171_122772_object120299:head# 2024-04-20T17:27:55.894 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.894 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a79ed524:::benchmark_data_smithi171_122772_object122257:head# 2024-04-20T17:27:55.894 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.894 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7a042cf:::benchmark_data_smithi171_122772_object99340:head# 2024-04-20T17:27:55.894 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.894 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7a05383:::benchmark_data_smithi171_122772_object132339:head# 2024-04-20T17:27:55.894 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.894 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7a06e75:::benchmark_data_smithi171_122772_object112795:head# 2024-04-20T17:27:55.894 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.895 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7a1165c:::benchmark_data_smithi171_122772_object101164:head# 2024-04-20T17:27:55.895 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.895 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7a17522:::benchmark_data_smithi171_122772_object83176:head# 2024-04-20T17:27:55.895 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.895 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7a182fe:::benchmark_data_smithi171_122772_object101761:head# 2024-04-20T17:27:55.895 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.895 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7a339e4:::benchmark_data_smithi171_122772_object83273:head# 2024-04-20T17:27:55.895 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.895 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7a3dec0:::benchmark_data_smithi171_122772_object136099:head# 2024-04-20T17:27:55.895 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.895 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7a44180:::benchmark_data_smithi171_122772_object78745:head# 2024-04-20T17:27:55.895 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.895 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7a45d19:::benchmark_data_smithi171_122772_object140171:head# 2024-04-20T17:27:55.896 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.896 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7a4ee36:::benchmark_data_smithi171_122772_object82385:head# 2024-04-20T17:27:55.896 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.896 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7a6710b:::benchmark_data_smithi171_122772_object131149:head# 2024-04-20T17:27:55.896 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.896 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7a6e5f3:::benchmark_data_smithi171_122772_object142100:head# 2024-04-20T17:27:55.896 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.896 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7a76435:::benchmark_data_smithi171_122772_object162883:head# 2024-04-20T17:27:55.896 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.896 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7a97274:::benchmark_data_smithi171_122772_object126381:head# 2024-04-20T17:27:55.896 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.896 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7a9d2b0:::benchmark_data_smithi171_122772_object115387:head# 2024-04-20T17:27:55.897 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.897 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7a9f9d6:::benchmark_data_smithi171_122772_object126165:head# 2024-04-20T17:27:55.897 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.897 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7aa7537:::benchmark_data_smithi171_122772_object101495:head# 2024-04-20T17:27:55.897 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.897 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7ab9f54:::benchmark_data_smithi171_122772_object84093:head# 2024-04-20T17:27:55.897 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.897 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7abecb5:::benchmark_data_smithi171_122772_object148347:head# 2024-04-20T17:27:55.897 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.897 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7ac17d4:::benchmark_data_smithi171_122772_object95642:head# 2024-04-20T17:27:55.897 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.898 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7ac3158:::benchmark_data_smithi171_122772_object81511:head# 2024-04-20T17:27:55.898 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.898 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7ace06a:::benchmark_data_smithi171_122772_object116877:head# 2024-04-20T17:27:55.898 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.898 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7ae7890:::benchmark_data_smithi171_122772_object136832:head# 2024-04-20T17:27:55.898 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.898 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7aec3db:::benchmark_data_smithi171_122772_object121947:head# 2024-04-20T17:27:55.898 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.898 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7b127e2:::benchmark_data_smithi171_122772_object165651:head# 2024-04-20T17:27:55.898 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.898 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7b18eb0:::benchmark_data_smithi171_122772_object104487:head# 2024-04-20T17:27:55.898 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.899 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7b205b8:::benchmark_data_smithi171_122772_object161271:head# 2024-04-20T17:27:55.899 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.899 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7b320c2:::benchmark_data_smithi171_122772_object150801:head# 2024-04-20T17:27:55.899 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.899 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7b3df26:::benchmark_data_smithi171_122772_object142612:head# 2024-04-20T17:27:55.899 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.899 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7b45ad7:::benchmark_data_smithi171_122772_object93113:head# 2024-04-20T17:27:55.899 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.899 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7b46e6d:::benchmark_data_smithi171_122772_object85916:head# 2024-04-20T17:27:55.899 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.899 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7b63c6e:::benchmark_data_smithi171_122772_object104908:head# 2024-04-20T17:27:55.899 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.900 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7b74f66:::benchmark_data_smithi171_122772_object145982:head# 2024-04-20T17:27:55.900 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.900 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7b838b4:::benchmark_data_smithi171_122772_object165464:head# 2024-04-20T17:27:55.900 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.900 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7b86f4d:::benchmark_data_smithi171_122772_object149955:head# 2024-04-20T17:27:55.900 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.900 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7b8960f:::benchmark_data_smithi171_122772_object165326:head# 2024-04-20T17:27:55.900 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.900 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7b8f4d8:::benchmark_data_smithi171_122772_object163741:head# 2024-04-20T17:27:55.900 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.900 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7b95d19:::benchmark_data_smithi171_122772_object107364:head# 2024-04-20T17:27:55.900 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.901 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7bae9dc:::benchmark_data_smithi171_122772_object167323:head# 2024-04-20T17:27:55.901 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.901 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7bbae11:::benchmark_data_smithi171_122772_object168278:head# 2024-04-20T17:27:55.901 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.901 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7bbc68e:::benchmark_data_smithi171_122772_object114585:head# 2024-04-20T17:27:55.901 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.901 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7bbc717:::benchmark_data_smithi171_122772_object168550:head# 2024-04-20T17:27:55.901 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.901 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7bd5fb1:::benchmark_data_smithi171_122772_object134190:head# 2024-04-20T17:27:55.901 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.901 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7bf1a76:::benchmark_data_smithi171_122772_object139980:head# 2024-04-20T17:27:55.901 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.902 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7bf52b5:::benchmark_data_smithi171_122772_object147360:head# 2024-04-20T17:27:55.902 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.902 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7c11641:::benchmark_data_smithi171_122772_object129689:head# 2024-04-20T17:27:55.902 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.902 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7c1dad3:::benchmark_data_smithi171_122772_object93941:head# 2024-04-20T17:27:55.902 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.902 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7c73089:::benchmark_data_smithi171_122772_object108014:head# 2024-04-20T17:27:55.902 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.902 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7c7a08a:::benchmark_data_smithi171_122772_object97249:head# 2024-04-20T17:27:55.902 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.902 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7c7a926:::benchmark_data_smithi171_122772_object163841:head# 2024-04-20T17:27:55.902 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.903 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7c88725:::benchmark_data_smithi171_122772_object133557:head# 2024-04-20T17:27:55.903 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.903 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7c8a38e:::benchmark_data_smithi171_122772_object113526:head# 2024-04-20T17:27:55.903 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.903 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7c90d9a:::benchmark_data_smithi171_122772_object143661:head# 2024-04-20T17:27:55.903 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.903 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7c93fd6:::benchmark_data_smithi171_122772_object99132:head# 2024-04-20T17:27:55.903 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.903 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7cd7d00:::benchmark_data_smithi171_122772_object115516:head# 2024-04-20T17:27:55.903 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.903 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7cde7b9:::benchmark_data_smithi171_122772_object101913:head# 2024-04-20T17:27:55.903 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.904 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7cdeef5:::benchmark_data_smithi171_122772_object156681:head# 2024-04-20T17:27:55.904 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.904 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7cec1d5:::benchmark_data_smithi171_122772_object157166:head# 2024-04-20T17:27:55.904 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.904 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7cefed3:::benchmark_data_smithi171_122772_object95300:head# 2024-04-20T17:27:55.904 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.904 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7d07f19:::benchmark_data_smithi171_122772_object103037:head# 2024-04-20T17:27:55.904 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.904 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7d1ea3d:::benchmark_data_smithi171_122772_object79597:head# 2024-04-20T17:27:55.904 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.904 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7d25153:::benchmark_data_smithi171_122772_object131158:head# 2024-04-20T17:27:55.904 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.905 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7d2d833:::benchmark_data_smithi171_122772_object102606:head# 2024-04-20T17:27:55.905 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.905 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7d6546d:::benchmark_data_smithi171_122772_object118113:head# 2024-04-20T17:27:55.905 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.905 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7d6910d:::benchmark_data_smithi171_122772_object113930:head# 2024-04-20T17:27:55.905 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.905 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7d79659:::benchmark_data_smithi171_122772_object120789:head# 2024-04-20T17:27:55.905 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.905 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7d79f66:::benchmark_data_smithi171_122772_object112742:head# 2024-04-20T17:27:55.905 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.905 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7d7a1da:::benchmark_data_smithi171_122772_object103818:head# 2024-04-20T17:27:55.905 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.906 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7d929c1:::benchmark_data_smithi171_122772_object124076:head# 2024-04-20T17:27:55.906 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.906 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7d97f7c:::benchmark_data_smithi171_122772_object135148:head# 2024-04-20T17:27:55.906 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.906 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7d9ec25:::benchmark_data_smithi171_122772_object125412:head# 2024-04-20T17:27:55.906 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.906 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7d9ef0b:::benchmark_data_smithi171_122772_object153527:head# 2024-04-20T17:27:55.906 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.906 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7db6547:::benchmark_data_smithi171_122772_object148035:head# 2024-04-20T17:27:55.906 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.906 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7dcb094:::benchmark_data_smithi171_122772_object163719:head# 2024-04-20T17:27:55.906 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.907 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7de4db5:::benchmark_data_smithi171_122772_object138896:head# 2024-04-20T17:27:55.907 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.907 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7deebd7:::benchmark_data_smithi171_122772_object88561:head# 2024-04-20T17:27:55.907 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.907 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7def82b:::benchmark_data_smithi171_122772_object158338:head# 2024-04-20T17:27:55.907 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.907 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7df24e3:::benchmark_data_smithi171_122772_object157995:head# 2024-04-20T17:27:55.907 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.907 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7dfa466:::benchmark_data_smithi171_122772_object92209:head# 2024-04-20T17:27:55.907 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.907 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7dfe928:::benchmark_data_smithi171_122772_object124994:head# 2024-04-20T17:27:55.907 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.907 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7e00801:::benchmark_data_smithi171_122772_object101827:head# 2024-04-20T17:27:55.908 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.908 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7e37b3c:::benchmark_data_smithi171_122772_object81833:head# 2024-04-20T17:27:55.908 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.908 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7e424ca:::benchmark_data_smithi171_122772_object141739:head# 2024-04-20T17:27:55.908 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.908 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7e446c9:::benchmark_data_smithi171_122772_object85744:head# 2024-04-20T17:27:55.908 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.908 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7e4832a:::benchmark_data_smithi171_122772_object89474:head# 2024-04-20T17:27:55.908 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.908 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7e543a2:::benchmark_data_smithi171_122772_object145461:head# 2024-04-20T17:27:55.908 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.908 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7e60d58:::benchmark_data_smithi171_122772_object111736:head# 2024-04-20T17:27:55.909 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.909 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7e74cbc:::benchmark_data_smithi171_122772_object125627:head# 2024-04-20T17:27:55.909 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.909 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7e92954:::benchmark_data_smithi171_122772_object122715:head# 2024-04-20T17:27:55.909 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.909 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7e9c8d6:::benchmark_data_smithi171_122772_object163627:head# 2024-04-20T17:27:55.909 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.909 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7eadef4:::benchmark_data_smithi171_122772_object112160:head# 2024-04-20T17:27:55.909 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.909 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7eaed49:::benchmark_data_smithi171_122772_object96724:head# 2024-04-20T17:27:55.909 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.909 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7eba603:::benchmark_data_smithi171_122772_object129033:head# 2024-04-20T17:27:55.910 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.910 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7ec4c13:::benchmark_data_smithi171_122772_object164422:head# 2024-04-20T17:27:55.910 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.910 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7ec78d4:::benchmark_data_smithi171_122772_object79530:head# 2024-04-20T17:27:55.910 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.910 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7ecf898:::benchmark_data_smithi171_122772_object83458:head# 2024-04-20T17:27:55.910 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.910 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7edd6b4:::benchmark_data_smithi171_122772_object169866:head# 2024-04-20T17:27:55.910 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.910 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7ee57cf:::benchmark_data_smithi171_122772_object140969:head# 2024-04-20T17:27:55.910 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.910 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7ef08a6:::benchmark_data_smithi171_122772_object113964:head# 2024-04-20T17:27:55.911 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.911 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7ef82b7:::benchmark_data_smithi171_122772_object156495:head# 2024-04-20T17:27:55.911 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.911 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7f020e5:::benchmark_data_smithi171_122772_object96672:head# 2024-04-20T17:27:55.911 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.911 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7f0256d:::benchmark_data_smithi171_122772_object139868:head# 2024-04-20T17:27:55.911 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.911 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7f163ca:::benchmark_data_smithi171_122772_object137332:head# 2024-04-20T17:27:55.911 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.911 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7f3653c:::benchmark_data_smithi171_122772_object158222:head# 2024-04-20T17:27:55.911 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.911 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7f3740f:::benchmark_data_smithi171_122772_object159762:head# 2024-04-20T17:27:55.911 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.912 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7f40443:::benchmark_data_smithi171_122772_object140878:head# 2024-04-20T17:27:55.912 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.912 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7f48afc:::benchmark_data_smithi171_122772_object104074:head# 2024-04-20T17:27:55.912 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.912 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7f4bf3b:::benchmark_data_smithi171_122772_object136308:head# 2024-04-20T17:27:55.912 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.912 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7f4e07b:::benchmark_data_smithi171_122772_object147388:head# 2024-04-20T17:27:55.912 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.912 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7f6709c:::benchmark_data_smithi171_122772_object155375:head# 2024-04-20T17:27:55.912 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.912 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7f7deae:::benchmark_data_smithi171_122772_object134224:head# 2024-04-20T17:27:55.912 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.912 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7f8d24c:::benchmark_data_smithi171_122772_object165307:head# 2024-04-20T17:27:55.912 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.913 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7f9e32a:::benchmark_data_smithi171_122772_object129400:head# 2024-04-20T17:27:55.913 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.913 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7fa2fa1:::benchmark_data_smithi171_122772_object79503:head# 2024-04-20T17:27:55.913 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.913 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7fb2320:::benchmark_data_smithi171_122772_object102561:head# 2024-04-20T17:27:55.913 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.913 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7fb99a9:::benchmark_data_smithi171_122772_object114053:head# 2024-04-20T17:27:55.913 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.913 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7fd5d1a:::benchmark_data_smithi171_122772_object163679:head# 2024-04-20T17:27:55.913 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.913 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7fd93ee:::benchmark_data_smithi171_122772_object122420:head# 2024-04-20T17:27:55.913 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.914 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a7fdbe71:::benchmark_data_smithi171_122772_object79415:head# 2024-04-20T17:27:55.914 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.914 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a800d57e:::benchmark_data_smithi171_122772_object78608:head# 2024-04-20T17:27:55.914 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.914 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a800f6b3:::benchmark_data_smithi171_122772_object130365:head# 2024-04-20T17:27:55.914 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.914 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8029a95:::benchmark_data_smithi171_122772_object114951:head# 2024-04-20T17:27:55.914 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.914 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a803a84f:::benchmark_data_smithi171_122772_object122710:head# 2024-04-20T17:27:55.914 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.914 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8055ab6:::benchmark_data_smithi171_122772_object162172:head# 2024-04-20T17:27:55.914 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.915 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a805f6fd:::benchmark_data_smithi171_122772_object146001:head# 2024-04-20T17:27:55.915 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.915 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8061634:::benchmark_data_smithi171_122772_object133865:head# 2024-04-20T17:27:55.915 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.915 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a806831f:::benchmark_data_smithi171_122772_object161231:head# 2024-04-20T17:27:55.915 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.915 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a807156f:::benchmark_data_smithi171_122772_object170152:head# 2024-04-20T17:27:55.915 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.915 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a80727ee:::benchmark_data_smithi171_122772_object105837:head# 2024-04-20T17:27:55.915 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.915 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a808c866:::benchmark_data_smithi171_122772_object145968:head# 2024-04-20T17:27:55.915 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.916 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a808e410:::benchmark_data_smithi171_122772_object111056:head# 2024-04-20T17:27:55.916 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.916 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8095d0b:::benchmark_data_smithi171_122772_object108325:head# 2024-04-20T17:27:55.916 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.916 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a80a0229:::benchmark_data_smithi171_122772_object119240:head# 2024-04-20T17:27:55.916 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.916 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a80bfbb9:::benchmark_data_smithi171_122772_object99890:head# 2024-04-20T17:27:55.916 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.916 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a80d07ef:::benchmark_data_smithi171_122772_object153197:head# 2024-04-20T17:27:55.916 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.916 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a80e2452:::benchmark_data_smithi171_122772_object131428:head# 2024-04-20T17:27:55.916 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.917 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a80ed4fd:::benchmark_data_smithi171_122772_object91393:head# 2024-04-20T17:27:55.917 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.917 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a80efd00:::benchmark_data_smithi171_122772_object83058:head# 2024-04-20T17:27:55.917 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.917 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a80f3fc4:::benchmark_data_smithi171_122772_object102907:head# 2024-04-20T17:27:55.917 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.917 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a81009dc:::benchmark_data_smithi171_122772_object167673:head# 2024-04-20T17:27:55.917 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.917 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a810c5d0:::benchmark_data_smithi171_122772_object160631:head# 2024-04-20T17:27:55.917 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.917 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a810f214:::benchmark_data_smithi171_122772_object159460:head# 2024-04-20T17:27:55.917 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.917 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a81168cb:::benchmark_data_smithi171_122772_object163998:head# 2024-04-20T17:27:55.918 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.918 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a811ea4f:::benchmark_data_smithi171_122772_object140947:head# 2024-04-20T17:27:55.918 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.918 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a811f91e:::benchmark_data_smithi171_122772_object155135:head# 2024-04-20T17:27:55.918 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.918 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8168f12:::benchmark_data_smithi171_122772_object121901:head# 2024-04-20T17:27:55.918 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.918 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8173332:::benchmark_data_smithi171_122772_object162364:head# 2024-04-20T17:27:55.918 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.918 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a817cc49:::benchmark_data_smithi171_122772_object110277:head# 2024-04-20T17:27:55.918 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.918 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a81968b5:::benchmark_data_smithi171_122772_object141861:head# 2024-04-20T17:27:55.919 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.919 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a81bc1ee:::benchmark_data_smithi171_122772_object131979:head# 2024-04-20T17:27:55.919 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.919 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a81c235e:::benchmark_data_smithi171_122772_object110897:head# 2024-04-20T17:27:55.919 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.919 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a81c82cd:::benchmark_data_smithi171_122772_object166950:head# 2024-04-20T17:27:55.919 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.919 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a81cdc44:::benchmark_data_smithi171_122772_object92961:head# 2024-04-20T17:27:55.919 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.919 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a81cef6d:::benchmark_data_smithi171_122772_object111451:head# 2024-04-20T17:27:55.919 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.919 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a81dcd5c:::benchmark_data_smithi171_122772_object155070:head# 2024-04-20T17:27:55.920 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.920 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a81f1011:::benchmark_data_smithi171_122772_object120316:head# 2024-04-20T17:27:55.920 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.920 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a81f8c8a:::benchmark_data_smithi171_122772_object97783:head# 2024-04-20T17:27:55.920 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.920 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a81fca61:::benchmark_data_smithi171_122772_object166137:head# 2024-04-20T17:27:55.920 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.920 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a81ff882:::benchmark_data_smithi171_122772_object101423:head# 2024-04-20T17:27:55.920 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.920 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a82170d5:::benchmark_data_smithi171_122772_object99531:head# 2024-04-20T17:27:55.920 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.921 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a821d697:::benchmark_data_smithi171_122772_object125038:head# 2024-04-20T17:27:55.921 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.921 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a821f77b:::benchmark_data_smithi171_122772_object155583:head# 2024-04-20T17:27:55.921 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.921 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a82213d6:::benchmark_data_smithi171_122772_object134543:head# 2024-04-20T17:27:55.921 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.921 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8221e4b:::benchmark_data_smithi171_122772_object86305:head# 2024-04-20T17:27:55.921 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.921 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a82295a6:::benchmark_data_smithi171_122772_object140608:head# 2024-04-20T17:27:55.921 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.921 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8234788:::benchmark_data_smithi171_122772_object169499:head# 2024-04-20T17:27:55.921 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.922 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a823b394:::benchmark_data_smithi171_122772_object160355:head# 2024-04-20T17:27:55.922 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.922 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a823b4dc:::benchmark_data_smithi171_122772_object142451:head# 2024-04-20T17:27:55.922 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.922 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a824539e:::benchmark_data_smithi171_122772_object139490:head# 2024-04-20T17:27:55.922 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.922 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8249eb7:::benchmark_data_smithi171_122772_object138388:head# 2024-04-20T17:27:55.922 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.922 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a824c6ba:::benchmark_data_smithi171_122772_object164587:head# 2024-04-20T17:27:55.922 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.922 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a824ce2c:::benchmark_data_smithi171_122772_object132178:head# 2024-04-20T17:27:55.922 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.923 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8256c4a:::benchmark_data_smithi171_122772_object143209:head# 2024-04-20T17:27:55.923 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.923 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a825dc20:::benchmark_data_smithi171_122772_object140136:head# 2024-04-20T17:27:55.923 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.923 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8263cd2:::benchmark_data_smithi171_122772_object161932:head# 2024-04-20T17:27:55.923 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.923 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a826747e:::benchmark_data_smithi171_122772_object111463:head# 2024-04-20T17:27:55.923 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.923 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a826b0c7:::benchmark_data_smithi171_122772_object122123:head# 2024-04-20T17:27:55.923 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.923 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a827963f:::benchmark_data_smithi171_122772_object154072:head# 2024-04-20T17:27:55.923 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.923 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a827d56f:::benchmark_data_smithi171_122772_object113740:head# 2024-04-20T17:27:55.924 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.924 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8281a3c:::benchmark_data_smithi171_122772_object93263:head# 2024-04-20T17:27:55.924 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.924 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a828ad4f:::benchmark_data_smithi171_122772_object135547:head# 2024-04-20T17:27:55.924 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.924 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a82997d6:::benchmark_data_smithi171_122772_object148440:head# 2024-04-20T17:27:55.924 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.924 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a82a3b93:::benchmark_data_smithi171_122772_object81885:head# 2024-04-20T17:27:55.924 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.924 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a82a4159:::benchmark_data_smithi171_122772_object165513:head# 2024-04-20T17:27:55.924 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.924 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a82a6221:::benchmark_data_smithi171_122772_object159068:head# 2024-04-20T17:27:55.925 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.925 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a82b099a:::benchmark_data_smithi171_122772_object145084:head# 2024-04-20T17:27:55.925 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.925 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a82cfcd6:::benchmark_data_smithi171_122772_object132607:head# 2024-04-20T17:27:55.925 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.925 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a82d1e59:::benchmark_data_smithi171_122772_object164590:head# 2024-04-20T17:27:55.925 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.925 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a82da1e2:::benchmark_data_smithi171_122772_object79615:head# 2024-04-20T17:27:55.925 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.925 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a830389d:::benchmark_data_smithi171_122772_object134041:head# 2024-04-20T17:27:55.925 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.925 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a831cd71:::benchmark_data_smithi171_122772_object112536:head# 2024-04-20T17:27:55.926 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.926 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8333f6d:::benchmark_data_smithi171_122772_object103845:head# 2024-04-20T17:27:55.926 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.926 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8335312:::benchmark_data_smithi171_122772_object89472:head# 2024-04-20T17:27:55.926 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.926 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8340fdd:::benchmark_data_smithi171_122772_object93483:head# 2024-04-20T17:27:55.926 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.926 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8344e1d:::benchmark_data_smithi171_122772_object84300:head# 2024-04-20T17:27:55.926 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.926 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a834e06f:::benchmark_data_smithi171_122772_object122620:head# 2024-04-20T17:27:55.926 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.927 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a836112c:::benchmark_data_smithi171_122772_object109450:head# 2024-04-20T17:27:55.927 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.927 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a836f1eb:::benchmark_data_smithi171_122772_object81247:head# 2024-04-20T17:27:55.927 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.927 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8374e6c:::benchmark_data_smithi171_122772_object127949:head# 2024-04-20T17:27:55.927 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.927 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8376fe8:::benchmark_data_smithi171_122772_object111094:head# 2024-04-20T17:27:55.927 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.927 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a839287b:::benchmark_data_smithi171_122772_object156367:head# 2024-04-20T17:27:55.927 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.927 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a83b68b5:::benchmark_data_smithi171_122772_object123800:head# 2024-04-20T17:27:55.927 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.928 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a83bd942:::benchmark_data_smithi171_122772_object153642:head# 2024-04-20T17:27:55.928 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.928 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a83bebe0:::benchmark_data_smithi171_122772_object112344:head# 2024-04-20T17:27:55.928 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.928 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a83c4837:::benchmark_data_smithi171_122772_object127375:head# 2024-04-20T17:27:55.928 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.928 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a83cbe00:::benchmark_data_smithi171_122772_object163350:head# 2024-04-20T17:27:55.928 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.928 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a83f11c2:::benchmark_data_smithi171_122772_object117755:head# 2024-04-20T17:27:55.928 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.928 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a83f1603:::benchmark_data_smithi171_122772_object154954:head# 2024-04-20T17:27:55.929 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.929 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8425eb4:::benchmark_data_smithi171_122772_object164373:head# 2024-04-20T17:27:55.929 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.929 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a842a7a6:::benchmark_data_smithi171_122772_object156504:head# 2024-04-20T17:27:55.929 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.929 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a842f367:::benchmark_data_smithi171_122772_object94873:head# 2024-04-20T17:27:55.929 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.929 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a843da3d:::benchmark_data_smithi171_122772_object164634:head# 2024-04-20T17:27:55.929 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.929 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a84442e1:::benchmark_data_smithi171_122772_object102577:head# 2024-04-20T17:27:55.929 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.929 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a844e9f2:::benchmark_data_smithi171_122772_object113469:head# 2024-04-20T17:27:55.930 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.930 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a844f893:::benchmark_data_smithi171_122772_object104722:head# 2024-04-20T17:27:55.930 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.930 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a84549bc:::benchmark_data_smithi171_122772_object114021:head# 2024-04-20T17:27:55.930 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.930 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8474234:::benchmark_data_smithi171_122772_object78242:head# 2024-04-20T17:27:55.930 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.930 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8475429:::benchmark_data_smithi171_122772_object161614:head# 2024-04-20T17:27:55.930 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.930 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a847a721:::benchmark_data_smithi171_122772_object163579:head# 2024-04-20T17:27:55.930 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.930 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a847cb4c:::benchmark_data_smithi171_122772_object81281:head# 2024-04-20T17:27:55.931 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.931 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a847edf2:::benchmark_data_smithi171_122772_object91195:head# 2024-04-20T17:27:55.931 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.931 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a848dff5:::benchmark_data_smithi171_122772_object100782:head# 2024-04-20T17:27:55.931 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.931 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a84963ea:::benchmark_data_smithi171_122772_object106927:head# 2024-04-20T17:27:55.931 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.931 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a849e09a:::benchmark_data_smithi171_122772_object117576:head# 2024-04-20T17:27:55.931 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.931 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a84a1f62:::benchmark_data_smithi171_122772_object84185:head# 2024-04-20T17:27:55.931 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.931 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a84a2d80:::benchmark_data_smithi171_122772_object140365:head# 2024-04-20T17:27:55.931 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.932 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a84b21f4:::benchmark_data_smithi171_122772_object144027:head# 2024-04-20T17:27:55.932 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.932 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a84ba5c0:::benchmark_data_smithi171_122772_object154513:head# 2024-04-20T17:27:55.932 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.932 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a84baa60:::benchmark_data_smithi171_122772_object100108:head# 2024-04-20T17:27:55.932 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.932 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a84bd4ac:::benchmark_data_smithi171_122772_object153067:head# 2024-04-20T17:27:55.932 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.932 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a84d8aac:::benchmark_data_smithi171_122772_object102164:head# 2024-04-20T17:27:55.932 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.932 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a84e13c0:::benchmark_data_smithi171_122772_object85888:head# 2024-04-20T17:27:55.933 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.933 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a84fc6c6:::benchmark_data_smithi171_122772_object93035:head# 2024-04-20T17:27:55.933 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.933 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a84fc86c:::benchmark_data_smithi171_122772_object93945:head# 2024-04-20T17:27:55.933 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.933 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8503999:::benchmark_data_smithi171_122772_object134331:head# 2024-04-20T17:27:55.933 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.933 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8508c24:::benchmark_data_smithi171_122772_object104970:head# 2024-04-20T17:27:55.933 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.933 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a850b37f:::benchmark_data_smithi171_122772_object98801:head# 2024-04-20T17:27:55.933 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.933 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a851e080:::benchmark_data_smithi171_122772_object140519:head# 2024-04-20T17:27:55.934 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.934 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a854a761:::benchmark_data_smithi171_122772_object155114:head# 2024-04-20T17:27:55.934 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.934 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8554515:::benchmark_data_smithi171_122772_object122217:head# 2024-04-20T17:27:55.934 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.934 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8565a0b:::benchmark_data_smithi171_122772_object136309:head# 2024-04-20T17:27:55.934 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.934 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a856b4d5:::benchmark_data_smithi171_122772_object161760:head# 2024-04-20T17:27:55.934 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.934 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8574faa:::benchmark_data_smithi171_122772_object143117:head# 2024-04-20T17:27:55.935 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.935 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a857582d:::benchmark_data_smithi171_122772_object160708:head# 2024-04-20T17:27:55.935 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.935 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a85764b5:::benchmark_data_smithi171_122772_object82814:head# 2024-04-20T17:27:55.935 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.935 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a857b0ff:::benchmark_data_smithi171_122772_object142893:head# 2024-04-20T17:27:55.935 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.935 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a857f83d:::benchmark_data_smithi171_122772_object141551:head# 2024-04-20T17:27:55.935 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.935 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a857fd4e:::benchmark_data_smithi171_122772_object164904:head# 2024-04-20T17:27:55.935 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.935 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a85838c0:::benchmark_data_smithi171_122772_object140514:head# 2024-04-20T17:27:55.936 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.936 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8583a67:::benchmark_data_smithi171_122772_object105512:head# 2024-04-20T17:27:55.936 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.936 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8588364:::benchmark_data_smithi171_122772_object87266:head# 2024-04-20T17:27:55.936 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.936 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a858ff2d:::benchmark_data_smithi171_122772_object134781:head# 2024-04-20T17:27:55.936 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.936 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a85917d7:::benchmark_data_smithi171_122772_object157998:head# 2024-04-20T17:27:55.936 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.936 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8596873:::benchmark_data_smithi171_122772_object96266:head# 2024-04-20T17:27:55.936 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.936 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a85b025d:::benchmark_data_smithi171_122772_object132747:head# 2024-04-20T17:27:55.937 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.937 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a85ce9f9:::benchmark_data_smithi171_122772_object113661:head# 2024-04-20T17:27:55.937 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.937 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a85d518f:::benchmark_data_smithi171_122772_object119490:head# 2024-04-20T17:27:55.937 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.937 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a85df846:::benchmark_data_smithi171_122772_object157563:head# 2024-04-20T17:27:55.937 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.937 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a85e1aa4:::benchmark_data_smithi171_122772_object152179:head# 2024-04-20T17:27:55.937 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.937 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a85ee58a:::benchmark_data_smithi171_122772_object91039:head# 2024-04-20T17:27:55.937 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.938 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a86040fd:::benchmark_data_smithi171_122772_object142049:head# 2024-04-20T17:27:55.938 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.938 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8605afb:::benchmark_data_smithi171_122772_object98760:head# 2024-04-20T17:27:55.938 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.938 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8609723:::benchmark_data_smithi171_122772_object88875:head# 2024-04-20T17:27:55.938 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.938 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8610d59:::benchmark_data_smithi171_122772_object115388:head# 2024-04-20T17:27:55.938 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.938 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a861be01:::benchmark_data_smithi171_122772_object125359:head# 2024-04-20T17:27:55.938 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.938 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a861e9ed:::benchmark_data_smithi171_122772_object90681:head# 2024-04-20T17:27:55.939 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.939 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8622fab:::benchmark_data_smithi171_122772_object101290:head# 2024-04-20T17:27:55.939 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.939 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8632959:::benchmark_data_smithi171_122772_object138482:head# 2024-04-20T17:27:55.939 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.939 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a863c577:::benchmark_data_smithi171_122772_object132482:head# 2024-04-20T17:27:55.939 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.939 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a863d61b:::benchmark_data_smithi171_122772_object127709:head# 2024-04-20T17:27:55.939 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.939 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8646de4:::benchmark_data_smithi171_122772_object142720:head# 2024-04-20T17:27:55.939 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.939 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8647026:::benchmark_data_smithi171_122772_object101309:head# 2024-04-20T17:27:55.940 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.940 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a864d070:::benchmark_data_smithi171_122772_object151777:head# 2024-04-20T17:27:55.940 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.940 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a865fa48:::benchmark_data_smithi171_122772_object126032:head# 2024-04-20T17:27:55.940 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.940 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a86a6e86:::benchmark_data_smithi171_122772_object121604:head# 2024-04-20T17:27:55.940 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.940 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a86a7b6e:::benchmark_data_smithi171_122772_object81334:head# 2024-04-20T17:27:55.940 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.940 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a86b195b:::benchmark_data_smithi171_122772_object128320:head# 2024-04-20T17:27:55.940 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.940 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a86c95c3:::benchmark_data_smithi171_122772_object116642:head# 2024-04-20T17:27:55.940 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.941 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a86cd020:::benchmark_data_smithi171_122772_object154033:head# 2024-04-20T17:27:55.941 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.941 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a86ea6e4:::benchmark_data_smithi171_122772_object101837:head# 2024-04-20T17:27:55.941 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.941 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a86f04b5:::benchmark_data_smithi171_122772_object165727:head# 2024-04-20T17:27:55.941 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.941 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a86f4373:::benchmark_data_smithi171_122772_object106300:head# 2024-04-20T17:27:55.941 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.941 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a86f594a:::benchmark_data_smithi171_122772_object164302:head# 2024-04-20T17:27:55.941 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.941 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a870fb1b:::benchmark_data_smithi171_122772_object160351:head# 2024-04-20T17:27:55.941 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.942 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a87160ca:::benchmark_data_smithi171_122772_object148574:head# 2024-04-20T17:27:55.942 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.942 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a87179bb:::benchmark_data_smithi171_122772_object119812:head# 2024-04-20T17:27:55.942 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.942 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a871eefe:::benchmark_data_smithi171_122772_object113165:head# 2024-04-20T17:27:55.942 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.942 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a873f5f6:::benchmark_data_smithi171_122772_object152549:head# 2024-04-20T17:27:55.942 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.942 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a875496b:::benchmark_data_smithi171_122772_object102459:head# 2024-04-20T17:27:55.942 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.942 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a875cb7d:::benchmark_data_smithi171_122772_object163421:head# 2024-04-20T17:27:55.942 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.943 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a876666c:::benchmark_data_smithi171_122772_object82882:head# 2024-04-20T17:27:55.943 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.943 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a876c8c1:::benchmark_data_smithi171_122772_object86450:head# 2024-04-20T17:27:55.943 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.943 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a877e772:::benchmark_data_smithi171_122772_object79780:head# 2024-04-20T17:27:55.943 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.943 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8786999:::benchmark_data_smithi171_122772_object112528:head# 2024-04-20T17:27:55.943 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.943 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a879e50d:::benchmark_data_smithi171_122772_object82390:head# 2024-04-20T17:27:55.943 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.943 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a87c3333:::benchmark_data_smithi171_122772_object90666:head# 2024-04-20T17:27:55.943 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.943 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a87d6914:::benchmark_data_smithi171_122772_object86625:head# 2024-04-20T17:27:55.944 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.944 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a87e444c:::benchmark_data_smithi171_122772_object98478:head# 2024-04-20T17:27:55.944 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.944 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a87f3d5f:::benchmark_data_smithi171_122772_object115267:head# 2024-04-20T17:27:55.944 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.944 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a88008a3:::benchmark_data_smithi171_122772_object86815:head# 2024-04-20T17:27:55.944 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.944 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a881287d:::benchmark_data_smithi171_122772_object157874:head# 2024-04-20T17:27:55.944 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.944 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a881a1bb:::benchmark_data_smithi171_122772_object127617:head# 2024-04-20T17:27:55.944 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.945 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a88302f2:::benchmark_data_smithi171_122772_object154017:head# 2024-04-20T17:27:55.945 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.945 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a883a8ff:::benchmark_data_smithi171_122772_object162779:head# 2024-04-20T17:27:55.945 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.945 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a88413ce:::benchmark_data_smithi171_122772_object84034:head# 2024-04-20T17:27:55.945 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.945 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8841f48:::benchmark_data_smithi171_122772_object161135:head# 2024-04-20T17:27:55.945 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.945 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8844718:::benchmark_data_smithi171_122772_object118111:head# 2024-04-20T17:27:55.945 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.945 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a884e474:::benchmark_data_smithi171_122772_object94337:head# 2024-04-20T17:27:55.945 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.946 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8853661:::benchmark_data_smithi171_122772_object125284:head# 2024-04-20T17:27:55.946 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.946 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8876c42:::benchmark_data_smithi171_122772_object107149:head# 2024-04-20T17:27:55.946 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.946 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8895e90:::benchmark_data_smithi171_122772_object134864:head# 2024-04-20T17:27:55.946 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.946 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8899503:::benchmark_data_smithi171_122772_object161139:head# 2024-04-20T17:27:55.946 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.946 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a88a2f5d:::benchmark_data_smithi171_122772_object111619:head# 2024-04-20T17:27:55.946 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.946 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a88a36cd:::benchmark_data_smithi171_122772_object137988:head# 2024-04-20T17:27:55.947 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.947 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a88a5ca5:::benchmark_data_smithi171_122772_object121929:head# 2024-04-20T17:27:55.947 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.947 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a88a9ef0:::benchmark_data_smithi171_122772_object128545:head# 2024-04-20T17:27:55.947 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.947 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a88aef7b:::benchmark_data_smithi171_122772_object135652:head# 2024-04-20T17:27:55.947 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.947 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a88cae0b:::benchmark_data_smithi171_122772_object169413:head# 2024-04-20T17:27:55.947 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.947 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a88d4f28:::benchmark_data_smithi171_122772_object85383:head# 2024-04-20T17:27:55.947 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.947 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a88ea8c0:::benchmark_data_smithi171_122772_object80943:head# 2024-04-20T17:27:55.947 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.948 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a88f27ad:::benchmark_data_smithi171_122772_object138164:head# 2024-04-20T17:27:55.948 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.948 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8913e78:::benchmark_data_smithi171_122772_object150772:head# 2024-04-20T17:27:55.948 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.948 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a891f0d2:::benchmark_data_smithi171_122772_object94915:head# 2024-04-20T17:27:55.948 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.948 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a891f386:::benchmark_data_smithi171_122772_object84390:head# 2024-04-20T17:27:55.948 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.948 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a892a437:::benchmark_data_smithi171_122772_object162856:head# 2024-04-20T17:27:55.948 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.948 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a892b79a:::benchmark_data_smithi171_122772_object107984:head# 2024-04-20T17:27:55.948 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.949 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a893a789:::benchmark_data_smithi171_122772_object125812:head# 2024-04-20T17:27:55.949 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.949 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8950623:::benchmark_data_smithi171_122772_object111359:head# 2024-04-20T17:27:55.949 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.949 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8953a8e:::benchmark_data_smithi171_122772_object142910:head# 2024-04-20T17:27:55.949 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.949 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a895a0db:::benchmark_data_smithi171_122772_object103741:head# 2024-04-20T17:27:55.949 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.949 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8979297:::benchmark_data_smithi171_122772_object137712:head# 2024-04-20T17:27:55.949 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.949 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a897ae88:::benchmark_data_smithi171_122772_object85315:head# 2024-04-20T17:27:55.949 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.949 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a897c0be:::benchmark_data_smithi171_122772_object163009:head# 2024-04-20T17:27:55.950 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.950 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a89a1410:::benchmark_data_smithi171_122772_object110824:head# 2024-04-20T17:27:55.950 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.950 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a89beb46:::benchmark_data_smithi171_122772_object137144:head# 2024-04-20T17:27:55.950 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.950 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a89c15bf:::benchmark_data_smithi171_122772_object153691:head# 2024-04-20T17:27:55.950 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.950 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a89d08fc:::benchmark_data_smithi171_122772_object164728:head# 2024-04-20T17:27:55.950 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.950 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a89d88a8:::benchmark_data_smithi171_122772_object148100:head# 2024-04-20T17:27:55.950 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.950 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a89e3162:::benchmark_data_smithi171_122772_object91884:head# 2024-04-20T17:27:55.951 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.951 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a89e61b4:::benchmark_data_smithi171_122772_object117091:head# 2024-04-20T17:27:55.951 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.951 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a89ecefa:::benchmark_data_smithi171_122772_object144596:head# 2024-04-20T17:27:55.951 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.951 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a89f6ca3:::benchmark_data_smithi171_122772_object159118:head# 2024-04-20T17:27:55.951 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.951 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8a1aae8:::benchmark_data_smithi171_122772_object149387:head# 2024-04-20T17:27:55.951 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.951 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8a33115:::benchmark_data_smithi171_122772_object81521:head# 2024-04-20T17:27:55.951 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.951 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8a39e85:::benchmark_data_smithi171_122772_object101774:head# 2024-04-20T17:27:55.951 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.952 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8a3dda5:::benchmark_data_smithi171_122772_object88786:head# 2024-04-20T17:27:55.952 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.952 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8a41e2d:::benchmark_data_smithi171_122772_object91634:head# 2024-04-20T17:27:55.952 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.952 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8a4b75b:::benchmark_data_smithi171_122772_object152445:head# 2024-04-20T17:27:55.952 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.952 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8a4cc01:::benchmark_data_smithi171_122772_object94339:head# 2024-04-20T17:27:55.952 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.952 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8a4f7ac:::benchmark_data_smithi171_122772_object101186:head# 2024-04-20T17:27:55.952 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.952 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8a4ffeb:::benchmark_data_smithi171_122772_object85473:head# 2024-04-20T17:27:55.952 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.953 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8a6544a:::benchmark_data_smithi171_122772_object117833:head# 2024-04-20T17:27:55.953 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.953 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8a6f733:::benchmark_data_smithi171_122772_object168036:head# 2024-04-20T17:27:55.953 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.953 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8a7abf6:::benchmark_data_smithi171_122772_object87174:head# 2024-04-20T17:27:55.953 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.953 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8a7d91a:::benchmark_data_smithi171_122772_object87080:head# 2024-04-20T17:27:55.953 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.953 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8a7ef80:::benchmark_data_smithi171_122772_object93233:head# 2024-04-20T17:27:55.953 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.953 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8a86592:::benchmark_data_smithi171_122772_object145676:head# 2024-04-20T17:27:55.953 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.954 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8a8ab77:::benchmark_data_smithi171_122772_object126849:head# 2024-04-20T17:27:55.954 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.954 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8a8d3f8:::benchmark_data_smithi171_122772_object117221:head# 2024-04-20T17:27:55.954 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.954 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8a90111:::benchmark_data_smithi171_122772_object146697:head# 2024-04-20T17:27:55.954 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.954 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8a95d94:::benchmark_data_smithi171_122772_object128197:head# 2024-04-20T17:27:55.954 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.954 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8a99ee7:::benchmark_data_smithi171_122772_object127346:head# 2024-04-20T17:27:55.954 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.954 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8a9dade:::benchmark_data_smithi171_122772_object140553:head# 2024-04-20T17:27:55.955 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.955 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8aa22be:::benchmark_data_smithi171_122772_object134899:head# 2024-04-20T17:27:55.955 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.955 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8aa49cc:::benchmark_data_smithi171_122772_object106308:head# 2024-04-20T17:27:55.955 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.955 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8aa667a:::benchmark_data_smithi171_122772_object151881:head# 2024-04-20T17:27:55.955 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.955 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8aab019:::benchmark_data_smithi171_122772_object121091:head# 2024-04-20T17:27:55.955 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.955 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8aab0d9:::benchmark_data_smithi171_122772_object158039:head# 2024-04-20T17:27:55.955 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.955 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8ab2367:::benchmark_data_smithi171_122772_object107512:head# 2024-04-20T17:27:55.956 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.956 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8ab8748:::benchmark_data_smithi171_122772_object85672:head# 2024-04-20T17:27:55.956 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.956 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8abbc77:::benchmark_data_smithi171_122772_object160056:head# 2024-04-20T17:27:55.956 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.956 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8ace5d1:::benchmark_data_smithi171_122772_object165858:head# 2024-04-20T17:27:55.956 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.956 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8ad6820:::benchmark_data_smithi171_122772_object100559:head# 2024-04-20T17:27:55.956 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.956 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8aec6a6:::benchmark_data_smithi171_122772_object112486:head# 2024-04-20T17:27:55.956 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.956 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8af8cdf:::benchmark_data_smithi171_122772_object110581:head# 2024-04-20T17:27:55.956 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.957 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8af9aec:::benchmark_data_smithi171_122772_object122781:head# 2024-04-20T17:27:55.957 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.957 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8b0851c:::benchmark_data_smithi171_122772_object90594:head# 2024-04-20T17:27:55.957 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.957 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8b19b56:::benchmark_data_smithi171_122772_object104957:head# 2024-04-20T17:27:55.957 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.957 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8b224d5:::benchmark_data_smithi171_122772_object102205:head# 2024-04-20T17:27:55.957 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.957 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8b367b5:::benchmark_data_smithi171_122772_object118006:head# 2024-04-20T17:27:55.957 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.957 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8b52631:::benchmark_data_smithi171_122772_object152183:head# 2024-04-20T17:27:55.957 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.957 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8b786eb:::benchmark_data_smithi171_122772_object85882:head# 2024-04-20T17:27:55.958 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.958 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8b82ba7:::benchmark_data_smithi171_122772_object163048:head# 2024-04-20T17:27:55.958 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.958 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8b852bd:::benchmark_data_smithi171_122772_object104979:head# 2024-04-20T17:27:55.958 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.958 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8b9888e:::benchmark_data_smithi171_122772_object134084:head# 2024-04-20T17:27:55.958 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.958 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8ba05e5:::benchmark_data_smithi171_122772_object157986:head# 2024-04-20T17:27:55.958 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.958 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8bbcb4a:::benchmark_data_smithi171_122772_object135709:head# 2024-04-20T17:27:55.958 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.958 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8bc9d3d:::benchmark_data_smithi171_122772_object158348:head# 2024-04-20T17:27:55.959 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.959 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8bcef54:::benchmark_data_smithi171_122772_object123546:head# 2024-04-20T17:27:55.959 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.959 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8be1847:::benchmark_data_smithi171_122772_object95578:head# 2024-04-20T17:27:55.959 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.959 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8be2834:::benchmark_data_smithi171_122772_object162860:head# 2024-04-20T17:27:55.959 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.959 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8c01f48:::benchmark_data_smithi171_122772_object99204:head# 2024-04-20T17:27:55.959 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.959 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8c03d51:::benchmark_data_smithi171_122772_object136284:head# 2024-04-20T17:27:55.959 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.959 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8c153c9:::benchmark_data_smithi171_122772_object152602:head# 2024-04-20T17:27:55.960 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.960 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8c5480b:::benchmark_data_smithi171_122772_object84155:head# 2024-04-20T17:27:55.960 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.960 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8c5ae85:::benchmark_data_smithi171_122772_object116491:head# 2024-04-20T17:27:55.960 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.960 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8c64a70:::benchmark_data_smithi171_122772_object169828:head# 2024-04-20T17:27:55.960 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.960 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8c800f2:::benchmark_data_smithi171_122772_object85415:head# 2024-04-20T17:27:55.960 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.960 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8c881f7:::benchmark_data_smithi171_122772_object151355:head# 2024-04-20T17:27:55.960 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.960 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8cb164b:::benchmark_data_smithi171_122772_object140612:head# 2024-04-20T17:27:55.960 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.961 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8cbbe0f:::benchmark_data_smithi171_122772_object120686:head# 2024-04-20T17:27:55.961 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.961 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8cc3ca1:::benchmark_data_smithi171_122772_object151255:head# 2024-04-20T17:27:55.961 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.961 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8cc7524:::benchmark_data_smithi171_122772_object89902:head# 2024-04-20T17:27:55.961 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.961 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8cdd450:::benchmark_data_smithi171_122772_object119827:head# 2024-04-20T17:27:55.961 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.961 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8cf1df4:::benchmark_data_smithi171_122772_object142838:head# 2024-04-20T17:27:55.961 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:55.962 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:55 smithi106 ceph-mon[44423]: pgmap v1363: 17 pgs: 17 active+clean; 5.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 642 op/s; 34 MiB/s, 546 objects/s recovering 2024-04-20T17:27:55.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:55.965 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:27:55.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:27:56.067 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:27:56.118 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8d08bfd:::benchmark_data_smithi171_122772_object123293:head# 2024-04-20T17:27:56.118 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.118 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8d0fc49:::benchmark_data_smithi171_122772_object89881:head# 2024-04-20T17:27:56.118 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.118 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8d13250:::benchmark_data_smithi171_122772_object83518:head# 2024-04-20T17:27:56.118 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.118 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8d15e0e:::benchmark_data_smithi171_122772_object129767:head# 2024-04-20T17:27:56.118 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.118 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8d2bf50:::benchmark_data_smithi171_122772_object110374:head# 2024-04-20T17:27:56.119 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.119 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8d2d99b:::benchmark_data_smithi171_122772_object150886:head# 2024-04-20T17:27:56.119 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.119 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8d4cdf0:::benchmark_data_smithi171_122772_object92804:head# 2024-04-20T17:27:56.119 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.119 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8d4e2f0:::benchmark_data_smithi171_122772_object89429:head# 2024-04-20T17:27:56.119 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.119 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8d5f19e:::benchmark_data_smithi171_122772_object168263:head# 2024-04-20T17:27:56.119 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.119 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8d6e628:::benchmark_data_smithi171_122772_object95808:head# 2024-04-20T17:27:56.119 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.119 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8d6ee62:::benchmark_data_smithi171_122772_object140317:head# 2024-04-20T17:27:56.119 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.120 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8d85f56:::benchmark_data_smithi171_122772_object107465:head# 2024-04-20T17:27:56.120 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.120 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8d8a65f:::benchmark_data_smithi171_122772_object166392:head# 2024-04-20T17:27:56.120 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.120 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8d8bff8:::benchmark_data_smithi171_122772_object151972:head# 2024-04-20T17:27:56.120 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.120 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8d9101e:::benchmark_data_smithi171_122772_object102881:head# 2024-04-20T17:27:56.120 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.120 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8d91052:::benchmark_data_smithi171_122772_object166294:head# 2024-04-20T17:27:56.120 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.120 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8d91b99:::benchmark_data_smithi171_122772_object110644:head# 2024-04-20T17:27:56.120 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.120 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8d9a159:::benchmark_data_smithi171_122772_object86261:head# 2024-04-20T17:27:56.121 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.121 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8d9dccd:::benchmark_data_smithi171_122772_object133842:head# 2024-04-20T17:27:56.121 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.121 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8da21a1:::benchmark_data_smithi171_122772_object98526:head# 2024-04-20T17:27:56.121 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.121 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8dab1a8:::benchmark_data_smithi171_122772_object135436:head# 2024-04-20T17:27:56.121 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.121 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8dbe8be:::benchmark_data_smithi171_122772_object98672:head# 2024-04-20T17:27:56.121 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.121 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8dc676d:::benchmark_data_smithi171_122772_object113371:head# 2024-04-20T17:27:56.121 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.121 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8dca0b8:::benchmark_data_smithi171_122772_object145922:head# 2024-04-20T17:27:56.121 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.121 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8dcdf41:::benchmark_data_smithi171_122772_object108047:head# 2024-04-20T17:27:56.122 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.122 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8dde754:::benchmark_data_smithi171_122772_object107033:head# 2024-04-20T17:27:56.122 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.122 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8de5c9b:::benchmark_data_smithi171_122772_object98833:head# 2024-04-20T17:27:56.122 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.122 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8df4c52:::benchmark_data_smithi171_122772_object97929:head# 2024-04-20T17:27:56.122 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.122 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8df8404:::benchmark_data_smithi171_122772_object130406:head# 2024-04-20T17:27:56.122 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.122 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8df8e0d:::benchmark_data_smithi171_122772_object156880:head# 2024-04-20T17:27:56.122 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.122 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8e32917:::benchmark_data_smithi171_122772_object109954:head# 2024-04-20T17:27:56.123 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.123 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8e4650d:::benchmark_data_smithi171_122772_object112378:head# 2024-04-20T17:27:56.123 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.123 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8e4d238:::benchmark_data_smithi171_122772_object96954:head# 2024-04-20T17:27:56.123 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.123 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8e6c7d7:::benchmark_data_smithi171_122772_object96231:head# 2024-04-20T17:27:56.123 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.123 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8e6f1be:::benchmark_data_smithi171_122772_object153657:head# 2024-04-20T17:27:56.123 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.123 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8e8941d:::benchmark_data_smithi171_122772_object153872:head# 2024-04-20T17:27:56.123 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.124 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8e9460a:::benchmark_data_smithi171_122772_object114771:head# 2024-04-20T17:27:56.124 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.124 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8eadd9c:::benchmark_data_smithi171_122772_object79441:head# 2024-04-20T17:27:56.124 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.124 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8eb5321:::benchmark_data_smithi171_122772_object103473:head# 2024-04-20T17:27:56.124 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.124 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8ec2c4a:::benchmark_data_smithi171_122772_object151925:head# 2024-04-20T17:27:56.124 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.124 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8eda2b3:::benchmark_data_smithi171_122772_object146246:head# 2024-04-20T17:27:56.124 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.124 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8eda328:::benchmark_data_smithi171_122772_object162608:head# 2024-04-20T17:27:56.124 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.125 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8edf7d5:::benchmark_data_smithi171_122772_object134952:head# 2024-04-20T17:27:56.125 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.125 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8ef8766:::benchmark_data_smithi171_122772_object138992:head# 2024-04-20T17:27:56.125 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.125 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8f14fdf:::benchmark_data_smithi171_122772_object90440:head# 2024-04-20T17:27:56.125 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.125 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8f18b0e:::benchmark_data_smithi171_122772_object124009:head# 2024-04-20T17:27:56.125 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.125 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8f2765a:::benchmark_data_smithi171_122772_object88230:head# 2024-04-20T17:27:56.125 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.126 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8f2eb5c:::benchmark_data_smithi171_122772_object124114:head# 2024-04-20T17:27:56.126 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.126 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8f3650a:::benchmark_data_smithi171_122772_object132867:head# 2024-04-20T17:27:56.126 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.126 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8f50753:::benchmark_data_smithi171_122772_object100253:head# 2024-04-20T17:27:56.126 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.126 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8f53072:::benchmark_data_smithi171_122772_object111445:head# 2024-04-20T17:27:56.126 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.126 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8f61dc2:::benchmark_data_smithi171_122772_object94319:head# 2024-04-20T17:27:56.126 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.126 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8f6d66f:::benchmark_data_smithi171_122772_object101194:head# 2024-04-20T17:27:56.127 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.127 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8f6dcc3:::benchmark_data_smithi171_122772_object83866:head# 2024-04-20T17:27:56.127 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.127 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8f71e9a:::benchmark_data_smithi171_122772_object155883:head# 2024-04-20T17:27:56.127 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.127 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8f7aa84:::benchmark_data_smithi171_122772_object133051:head# 2024-04-20T17:27:56.127 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.127 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8f81597:::benchmark_data_smithi171_122772_object164611:head# 2024-04-20T17:27:56.127 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.127 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8f87c87:::benchmark_data_smithi171_122772_object109695:head# 2024-04-20T17:27:56.127 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.127 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8f97bd7:::benchmark_data_smithi171_122772_object145245:head# 2024-04-20T17:27:56.128 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.128 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8fb2ab8:::benchmark_data_smithi171_122772_object98398:head# 2024-04-20T17:27:56.128 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.128 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8fb444d:::benchmark_data_smithi171_122772_object87375:head# 2024-04-20T17:27:56.128 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.128 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8fda48c:::benchmark_data_smithi171_122772_object158578:head# 2024-04-20T17:27:56.128 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.128 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8fddf8f:::benchmark_data_smithi171_122772_object149201:head# 2024-04-20T17:27:56.128 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.128 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8fdfc95:::benchmark_data_smithi171_122772_object132172:head# 2024-04-20T17:27:56.128 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.128 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8feb67d:::benchmark_data_smithi171_122772_object92030:head# 2024-04-20T17:27:56.129 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.129 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8feee3f:::benchmark_data_smithi171_122772_object127919:head# 2024-04-20T17:27:56.129 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.129 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8fefdd0:::benchmark_data_smithi171_122772_object86436:head# 2024-04-20T17:27:56.129 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.129 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a8ff828f:::benchmark_data_smithi171_122772_object94290:head# 2024-04-20T17:27:56.129 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.129 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9000097:::benchmark_data_smithi171_122772_object115713:head# 2024-04-20T17:27:56.129 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.129 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9001b69:::benchmark_data_smithi171_122772_object82650:head# 2024-04-20T17:27:56.129 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.130 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9011cf0:::benchmark_data_smithi171_122772_object139114:head# 2024-04-20T17:27:56.130 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.130 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9013fb3:::benchmark_data_smithi171_122772_object120676:head# 2024-04-20T17:27:56.130 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.130 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9020e08:::benchmark_data_smithi171_122772_object165130:head# 2024-04-20T17:27:56.130 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.130 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a902e042:::benchmark_data_smithi171_122772_object92404:head# 2024-04-20T17:27:56.130 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.130 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a903575a:::benchmark_data_smithi171_122772_object110352:head# 2024-04-20T17:27:56.130 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.130 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a90371c0:::benchmark_data_smithi171_122772_object78933:head# 2024-04-20T17:27:56.130 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.131 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9047094:::benchmark_data_smithi171_122772_object158270:head# 2024-04-20T17:27:56.131 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.131 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a90479e0:::benchmark_data_smithi171_122772_object169681:head# 2024-04-20T17:27:56.131 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.131 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9048821:::benchmark_data_smithi171_122772_object154297:head# 2024-04-20T17:27:56.131 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.131 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9050d26:::benchmark_data_smithi171_122772_object94273:head# 2024-04-20T17:27:56.131 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.131 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a905cdd0:::benchmark_data_smithi171_122772_object94564:head# 2024-04-20T17:27:56.131 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.131 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9079ba4:::benchmark_data_smithi171_122772_object123843:head# 2024-04-20T17:27:56.131 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.132 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a907a14b:::benchmark_data_smithi171_122772_object161998:head# 2024-04-20T17:27:56.132 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.132 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a908038b:::benchmark_data_smithi171_122772_object92271:head# 2024-04-20T17:27:56.132 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.132 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a908283f:::benchmark_data_smithi171_122772_object105395:head# 2024-04-20T17:27:56.132 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.132 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9086608:::benchmark_data_smithi171_122772_object113117:head# 2024-04-20T17:27:56.132 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.132 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a908a89a:::benchmark_data_smithi171_122772_object126600:head# 2024-04-20T17:27:56.132 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.132 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9091445:::benchmark_data_smithi171_122772_object93743:head# 2024-04-20T17:27:56.132 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.133 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a90b2da0:::benchmark_data_smithi171_122772_object80068:head# 2024-04-20T17:27:56.133 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.133 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a90bb955:::benchmark_data_smithi171_122772_object149896:head# 2024-04-20T17:27:56.133 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.133 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a90d3175:::benchmark_data_smithi171_122772_object135004:head# 2024-04-20T17:27:56.133 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.133 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a90e9a3a:::benchmark_data_smithi171_122772_object156806:head# 2024-04-20T17:27:56.133 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.133 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a910a3bf:::benchmark_data_smithi171_122772_object148714:head# 2024-04-20T17:27:56.133 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.133 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a911878d:::benchmark_data_smithi171_122772_object168837:head# 2024-04-20T17:27:56.134 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.134 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a913aad3:::benchmark_data_smithi171_122772_object108522:head# 2024-04-20T17:27:56.134 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.134 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a91401e0:::benchmark_data_smithi171_122772_object108829:head# 2024-04-20T17:27:56.134 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.134 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9141a1e:::benchmark_data_smithi171_122772_object116721:head# 2024-04-20T17:27:56.134 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.134 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a91465c1:::benchmark_data_smithi171_122772_object82764:head# 2024-04-20T17:27:56.134 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.134 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9147b1c:::benchmark_data_smithi171_122772_object154845:head# 2024-04-20T17:27:56.135 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.135 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9149b06:::benchmark_data_smithi171_122772_object167944:head# 2024-04-20T17:27:56.135 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.135 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a915363f:::benchmark_data_smithi171_122772_object152620:head# 2024-04-20T17:27:56.135 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.135 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a91543d4:::benchmark_data_smithi171_122772_object155751:head# 2024-04-20T17:27:56.135 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.135 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a915fe57:::benchmark_data_smithi171_122772_object164750:head# 2024-04-20T17:27:56.135 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.135 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9167872:::benchmark_data_smithi171_122772_object141790:head# 2024-04-20T17:27:56.135 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.136 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a917c1be:::benchmark_data_smithi171_122772_object84658:head# 2024-04-20T17:27:56.136 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.136 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a917d0be:::benchmark_data_smithi171_122772_object122380:head# 2024-04-20T17:27:56.136 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.136 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9186c87:::benchmark_data_smithi171_122772_object160358:head# 2024-04-20T17:27:56.136 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.136 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a918dbc1:::benchmark_data_smithi171_122772_object143471:head# 2024-04-20T17:27:56.136 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.136 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9198779:::benchmark_data_smithi171_122772_object139255:head# 2024-04-20T17:27:56.136 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.136 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a919ff35:::benchmark_data_smithi171_122772_object160651:head# 2024-04-20T17:27:56.136 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.136 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a91bb8fe:::benchmark_data_smithi171_122772_object126047:head# 2024-04-20T17:27:56.137 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.137 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a91c4efd:::benchmark_data_smithi171_122772_object100083:head# 2024-04-20T17:27:56.137 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.137 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a91d4a50:::benchmark_data_smithi171_122772_object99351:head# 2024-04-20T17:27:56.137 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.137 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a91dbeb7:::benchmark_data_smithi171_122772_object131447:head# 2024-04-20T17:27:56.137 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.137 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a91e20e7:::benchmark_data_smithi171_122772_object146639:head# 2024-04-20T17:27:56.137 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.137 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a91ee5b9:::benchmark_data_smithi171_122772_object134946:head# 2024-04-20T17:27:56.137 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.137 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a91f0ae6:::benchmark_data_smithi171_122772_object111662:head# 2024-04-20T17:27:56.138 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.138 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a91f65bc:::benchmark_data_smithi171_122772_object86886:head# 2024-04-20T17:27:56.138 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.138 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a92120cf:::benchmark_data_smithi171_122772_object142916:head# 2024-04-20T17:27:56.138 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.138 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a92174a2:::benchmark_data_smithi171_122772_object141825:head# 2024-04-20T17:27:56.138 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.138 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9221816:::benchmark_data_smithi171_122772_object151486:head# 2024-04-20T17:27:56.138 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.138 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9224a93:::benchmark_data_smithi171_122772_object141128:head# 2024-04-20T17:27:56.138 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.139 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a922e45f:::benchmark_data_smithi171_122772_object79821:head# 2024-04-20T17:27:56.139 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.139 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a926c008:::benchmark_data_smithi171_122772_object155036:head# 2024-04-20T17:27:56.139 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.139 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9275ae7:::benchmark_data_smithi171_122772_object164438:head# 2024-04-20T17:27:56.139 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.139 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a92799b7:::benchmark_data_smithi171_122772_object159853:head# 2024-04-20T17:27:56.139 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.139 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a927b67e:::benchmark_data_smithi171_122772_object144058:head# 2024-04-20T17:27:56.139 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.139 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9285cec:::benchmark_data_smithi171_122772_object141761:head# 2024-04-20T17:27:56.139 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.140 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9286b8b:::benchmark_data_smithi171_122772_object135010:head# 2024-04-20T17:27:56.140 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.140 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a928a667:::benchmark_data_smithi171_122772_object106601:head# 2024-04-20T17:27:56.140 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.140 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a92af8c6:::benchmark_data_smithi171_122772_object94763:head# 2024-04-20T17:27:56.140 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.140 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a92b9761:::benchmark_data_smithi171_122772_object136706:head# 2024-04-20T17:27:56.140 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.140 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a92c3f6b:::benchmark_data_smithi171_122772_object157269:head# 2024-04-20T17:27:56.140 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.140 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a92cd5d2:::benchmark_data_smithi171_122772_object144095:head# 2024-04-20T17:27:56.140 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.141 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a92e2810:::benchmark_data_smithi171_122772_object166516:head# 2024-04-20T17:27:56.141 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.141 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a92e3fcc:::benchmark_data_smithi171_122772_object110968:head# 2024-04-20T17:27:56.141 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.141 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a930a692:::benchmark_data_smithi171_122772_object120488:head# 2024-04-20T17:27:56.141 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.141 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9321934:::benchmark_data_smithi171_122772_object110118:head# 2024-04-20T17:27:56.141 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.141 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9332220:::benchmark_data_smithi171_122772_object138044:head# 2024-04-20T17:27:56.141 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.141 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a93377a4:::benchmark_data_smithi171_122772_object150273:head# 2024-04-20T17:27:56.142 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.142 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9342ba8:::benchmark_data_smithi171_122772_object159622:head# 2024-04-20T17:27:56.142 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.142 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a934627d:::benchmark_data_smithi171_122772_object121533:head# 2024-04-20T17:27:56.142 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.142 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a93606bb:::benchmark_data_smithi171_122772_object86734:head# 2024-04-20T17:27:56.142 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.142 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a936ec53:::benchmark_data_smithi171_122772_object132636:head# 2024-04-20T17:27:56.142 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.142 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a937325a:::benchmark_data_smithi171_122772_object114274:head# 2024-04-20T17:27:56.142 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.142 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9381355:::benchmark_data_smithi171_122772_object109093:head# 2024-04-20T17:27:56.143 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.143 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9386da2:::benchmark_data_smithi171_122772_object133027:head# 2024-04-20T17:27:56.143 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.143 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a938e3d1:::benchmark_data_smithi171_122772_object143018:head# 2024-04-20T17:27:56.143 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.143 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a939cb97:::benchmark_data_smithi171_122772_object121397:head# 2024-04-20T17:27:56.143 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.143 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a93a026c:::benchmark_data_smithi171_122772_object132536:head# 2024-04-20T17:27:56.143 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.143 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a93ae99a:::benchmark_data_smithi171_122772_object159645:head# 2024-04-20T17:27:56.143 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.143 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a93be36a:::benchmark_data_smithi171_122772_object122354:head# 2024-04-20T17:27:56.144 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.144 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a93cd34f:::benchmark_data_smithi171_122772_object137272:head# 2024-04-20T17:27:56.144 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.144 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a93d5190:::benchmark_data_smithi171_122772_object131010:head# 2024-04-20T17:27:56.144 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.144 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a93d58a3:::benchmark_data_smithi171_122772_object150201:head# 2024-04-20T17:27:56.144 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.144 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a93ec20e:::benchmark_data_smithi171_122772_object138139:head# 2024-04-20T17:27:56.144 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.144 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a93ec6b5:::benchmark_data_smithi171_122772_object119189:head# 2024-04-20T17:27:56.144 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.144 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a93ee059:::benchmark_data_smithi171_122772_object154832:head# 2024-04-20T17:27:56.145 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.145 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a93ff242:::benchmark_data_smithi171_122772_object137974:head# 2024-04-20T17:27:56.145 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.145 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a940362a:::benchmark_data_smithi171_122772_object146729:head# 2024-04-20T17:27:56.145 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.145 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a940e22f:::benchmark_data_smithi171_122772_object150949:head# 2024-04-20T17:27:56.145 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.145 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a940ea54:::benchmark_data_smithi171_122772_object132060:head# 2024-04-20T17:27:56.145 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.145 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a940f211:::benchmark_data_smithi171_122772_object115025:head# 2024-04-20T17:27:56.145 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.146 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9427a2e:::benchmark_data_smithi171_122772_object132438:head# 2024-04-20T17:27:56.146 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.146 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9435143:::benchmark_data_smithi171_122772_object108971:head# 2024-04-20T17:27:56.146 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.146 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a943e251:::benchmark_data_smithi171_122772_object104658:head# 2024-04-20T17:27:56.146 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.146 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a94514ef:::benchmark_data_smithi171_122772_object79195:head# 2024-04-20T17:27:56.146 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.146 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a945823c:::benchmark_data_smithi171_122772_object164105:head# 2024-04-20T17:27:56.146 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.146 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a947074f:::benchmark_data_smithi171_122772_object111513:head# 2024-04-20T17:27:56.147 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.147 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9487f09:::benchmark_data_smithi171_122772_object161326:head# 2024-04-20T17:27:56.147 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.147 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a948a696:::benchmark_data_smithi171_122772_object97284:head# 2024-04-20T17:27:56.147 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.147 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a948c143:::benchmark_data_smithi171_122772_object158224:head# 2024-04-20T17:27:56.147 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.147 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a949591f:::benchmark_data_smithi171_122772_object127713:head# 2024-04-20T17:27:56.147 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.147 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a949ccce:::benchmark_data_smithi171_122772_object157737:head# 2024-04-20T17:27:56.147 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.147 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a94b0790:::benchmark_data_smithi171_122772_object150408:head# 2024-04-20T17:27:56.148 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.148 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a94b74f4:::benchmark_data_smithi171_122772_object106256:head# 2024-04-20T17:27:56.148 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.148 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a94e3769:::benchmark_data_smithi171_122772_object149270:head# 2024-04-20T17:27:56.148 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.148 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a94e9b62:::benchmark_data_smithi171_122772_object141599:head# 2024-04-20T17:27:56.148 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.148 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a94ea0a2:::benchmark_data_smithi171_122772_object80452:head# 2024-04-20T17:27:56.148 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.148 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a94ea261:::benchmark_data_smithi171_122772_object95621:head# 2024-04-20T17:27:56.148 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.148 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a94eedb4:::benchmark_data_smithi171_122772_object122776:head# 2024-04-20T17:27:56.149 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.149 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a94f7044:::benchmark_data_smithi171_122772_object158184:head# 2024-04-20T17:27:56.149 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.149 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a951254a:::benchmark_data_smithi171_122772_object132420:head# 2024-04-20T17:27:56.149 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.149 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9513dbf:::benchmark_data_smithi171_122772_object125609:head# 2024-04-20T17:27:56.149 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.149 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9524e1b:::benchmark_data_smithi171_122772_object107945:head# 2024-04-20T17:27:56.149 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.149 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9528ef6:::benchmark_data_smithi171_122772_object119671:head# 2024-04-20T17:27:56.149 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.149 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9529e65:::benchmark_data_smithi171_122772_object157603:head# 2024-04-20T17:27:56.150 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.150 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a952cdf8:::benchmark_data_smithi171_122772_object170044:head# 2024-04-20T17:27:56.150 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.150 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a95329c8:::benchmark_data_smithi171_122772_object135994:head# 2024-04-20T17:27:56.150 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.150 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9537ef6:::benchmark_data_smithi171_122772_object139362:head# 2024-04-20T17:27:56.150 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.150 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a953a3ff:::benchmark_data_smithi171_122772_object84189:head# 2024-04-20T17:27:56.150 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.150 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a953de8b:::benchmark_data_smithi171_122772_object169323:head# 2024-04-20T17:27:56.150 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.151 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9540d80:::benchmark_data_smithi171_122772_object127325:head# 2024-04-20T17:27:56.151 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.151 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a95681f3:::benchmark_data_smithi171_122772_object132815:head# 2024-04-20T17:27:56.151 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.151 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9578156:::benchmark_data_smithi171_122772_object99752:head# 2024-04-20T17:27:56.151 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.151 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9578f95:::benchmark_data_smithi171_122772_object143831:head# 2024-04-20T17:27:56.151 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.151 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9596032:::benchmark_data_smithi171_122772_object145967:head# 2024-04-20T17:27:56.151 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.151 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9597ab8:::benchmark_data_smithi171_122772_object84107:head# 2024-04-20T17:27:56.151 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.152 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a959aead:::benchmark_data_smithi171_122772_object156777:head# 2024-04-20T17:27:56.152 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.152 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a95a10b2:::benchmark_data_smithi171_122772_object115936:head# 2024-04-20T17:27:56.152 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.152 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a95bba40:::benchmark_data_smithi171_122772_object122708:head# 2024-04-20T17:27:56.152 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.152 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a95c2b12:::benchmark_data_smithi171_122772_object118551:head# 2024-04-20T17:27:56.152 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.152 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a95cb667:::benchmark_data_smithi171_122772_object164993:head# 2024-04-20T17:27:56.152 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.152 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a95da63e:::benchmark_data_smithi171_122772_object97988:head# 2024-04-20T17:27:56.152 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.152 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a95db821:::benchmark_data_smithi171_122772_object86100:head# 2024-04-20T17:27:56.152 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.153 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a95e0dcd:::benchmark_data_smithi171_122772_object131175:head# 2024-04-20T17:27:56.153 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.153 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a95e7607:::benchmark_data_smithi171_122772_object109311:head# 2024-04-20T17:27:56.153 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.153 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a960ad2e:::benchmark_data_smithi171_122772_object136872:head# 2024-04-20T17:27:56.153 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.153 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9611e70:::benchmark_data_smithi171_122772_object115684:head# 2024-04-20T17:27:56.153 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.153 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9614085:::benchmark_data_smithi171_122772_object148749:head# 2024-04-20T17:27:56.153 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.153 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9617232:::benchmark_data_smithi171_122772_object166759:head# 2024-04-20T17:27:56.154 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.154 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a961d1e4:::benchmark_data_smithi171_122772_object147491:head# 2024-04-20T17:27:56.154 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.154 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9649cd8:::benchmark_data_smithi171_122772_object134503:head# 2024-04-20T17:27:56.154 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.154 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9650de7:::benchmark_data_smithi171_122772_object160199:head# 2024-04-20T17:27:56.154 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.154 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9658e2e:::benchmark_data_smithi171_122772_object92242:head# 2024-04-20T17:27:56.154 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.154 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a965c020:::benchmark_data_smithi171_122772_object123213:head# 2024-04-20T17:27:56.154 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.155 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a966483c:::benchmark_data_smithi171_122772_object127499:head# 2024-04-20T17:27:56.155 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.155 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a966aabf:::benchmark_data_smithi171_122772_object139530:head# 2024-04-20T17:27:56.155 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.155 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9686733:::benchmark_data_smithi171_122772_object92854:head# 2024-04-20T17:27:56.155 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.155 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9688fe8:::benchmark_data_smithi171_122772_object79276:head# 2024-04-20T17:27:56.155 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.155 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a968fe99:::benchmark_data_smithi171_122772_object102681:head# 2024-04-20T17:27:56.155 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.155 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9698a80:::benchmark_data_smithi171_122772_object121452:head# 2024-04-20T17:27:56.155 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.155 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a96a3e9d:::benchmark_data_smithi171_122772_object129545:head# 2024-04-20T17:27:56.156 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.156 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a96a7c3f:::benchmark_data_smithi171_122772_object87668:head# 2024-04-20T17:27:56.156 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.156 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a96ac024:::benchmark_data_smithi171_122772_object98925:head# 2024-04-20T17:27:56.156 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.156 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a96aea4e:::benchmark_data_smithi171_122772_object138042:head# 2024-04-20T17:27:56.156 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.156 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a96c02a7:::benchmark_data_smithi171_122772_object132087:head# 2024-04-20T17:27:56.156 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.156 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a96c4ce4:::benchmark_data_smithi171_122772_object85663:head# 2024-04-20T17:27:56.156 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.156 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a96c5a5c:::benchmark_data_smithi171_122772_object159536:head# 2024-04-20T17:27:56.157 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.157 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a96c7773:::benchmark_data_smithi171_122772_object87423:head# 2024-04-20T17:27:56.157 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.157 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a96ea941:::benchmark_data_smithi171_122772_object111159:head# 2024-04-20T17:27:56.157 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.157 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a96eff83:::benchmark_data_smithi171_122772_object139074:head# 2024-04-20T17:27:56.157 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.157 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a96f2fae:::benchmark_data_smithi171_122772_object105020:head# 2024-04-20T17:27:56.157 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.157 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a96f3b36:::benchmark_data_smithi171_122772_object79497:head# 2024-04-20T17:27:56.157 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.157 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9709a1d:::benchmark_data_smithi171_122772_object112237:head# 2024-04-20T17:27:56.157 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.158 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a970ec0b:::benchmark_data_smithi171_122772_object107662:head# 2024-04-20T17:27:56.158 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.158 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9717b47:::benchmark_data_smithi171_122772_object169077:head# 2024-04-20T17:27:56.158 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.158 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9724fb5:::benchmark_data_smithi171_122772_object143336:head# 2024-04-20T17:27:56.158 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.158 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a972adc2:::benchmark_data_smithi171_122772_object117575:head# 2024-04-20T17:27:56.158 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.158 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a97327c8:::benchmark_data_smithi171_122772_object79080:head# 2024-04-20T17:27:56.158 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.158 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a975313e:::benchmark_data_smithi171_122772_object127091:head# 2024-04-20T17:27:56.159 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.159 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a976da03:::benchmark_data_smithi171_122772_object137398:head# 2024-04-20T17:27:56.159 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.159 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9777b4d:::benchmark_data_smithi171_122772_object78530:head# 2024-04-20T17:27:56.159 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.159 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a977ce0d:::benchmark_data_smithi171_122772_object112974:head# 2024-04-20T17:27:56.159 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.159 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a97898eb:::benchmark_data_smithi171_122772_object78575:head# 2024-04-20T17:27:56.159 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.159 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a979815f:::benchmark_data_smithi171_122772_object157620:head# 2024-04-20T17:27:56.159 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.159 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a97b1574:::benchmark_data_smithi171_122772_object92028:head# 2024-04-20T17:27:56.160 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.160 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a97bbfee:::benchmark_data_smithi171_122772_object153887:head# 2024-04-20T17:27:56.160 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.160 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a97cd7f8:::benchmark_data_smithi171_122772_object114972:head# 2024-04-20T17:27:56.160 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.160 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a97da6ea:::benchmark_data_smithi171_122772_object99285:head# 2024-04-20T17:27:56.160 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.160 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a97fa459:::benchmark_data_smithi171_122772_object141161:head# 2024-04-20T17:27:56.160 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.160 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a97fe340:::benchmark_data_smithi171_122772_object108966:head# 2024-04-20T17:27:56.160 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.160 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a980ae11:::benchmark_data_smithi171_122772_object168545:head# 2024-04-20T17:27:56.160 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.161 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a980c803:::benchmark_data_smithi171_122772_object160038:head# 2024-04-20T17:27:56.161 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.161 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9817b62:::benchmark_data_smithi171_122772_object79571:head# 2024-04-20T17:27:56.161 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.161 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9841bd4:::benchmark_data_smithi171_122772_object137834:head# 2024-04-20T17:27:56.161 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.161 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9841cd7:::benchmark_data_smithi171_122772_object154098:head# 2024-04-20T17:27:56.161 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.161 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a98462f3:::benchmark_data_smithi171_122772_object148848:head# 2024-04-20T17:27:56.161 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.161 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9871362:::benchmark_data_smithi171_122772_object86521:head# 2024-04-20T17:27:56.161 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.162 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9888a5d:::benchmark_data_smithi171_122772_object125323:head# 2024-04-20T17:27:56.162 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.162 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9889c3b:::benchmark_data_smithi171_122772_object136261:head# 2024-04-20T17:27:56.162 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.162 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a989c1a5:::benchmark_data_smithi171_122772_object94265:head# 2024-04-20T17:27:56.162 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.162 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a98abe68:::benchmark_data_smithi171_122772_object115650:head# 2024-04-20T17:27:56.162 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.162 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a98bb79b:::benchmark_data_smithi171_122772_object110887:head# 2024-04-20T17:27:56.162 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.162 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a98bba28:::benchmark_data_smithi171_122772_object121522:head# 2024-04-20T17:27:56.162 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.163 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a98c21cc:::benchmark_data_smithi171_122772_object105443:head# 2024-04-20T17:27:56.163 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.163 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a98e4d92:::benchmark_data_smithi171_122772_object130316:head# 2024-04-20T17:27:56.163 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.163 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a98f5d74:::benchmark_data_smithi171_122772_object92046:head# 2024-04-20T17:27:56.163 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.163 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a98f9583:::benchmark_data_smithi171_122772_object149300:head# 2024-04-20T17:27:56.163 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.163 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9907bcc:::benchmark_data_smithi171_122772_object112212:head# 2024-04-20T17:27:56.163 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.163 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9909a9d:::benchmark_data_smithi171_122772_object165975:head# 2024-04-20T17:27:56.163 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.163 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a991d33b:::benchmark_data_smithi171_122772_object91562:head# 2024-04-20T17:27:56.164 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.164 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a99374ef:::benchmark_data_smithi171_122772_object97064:head# 2024-04-20T17:27:56.164 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.164 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9938a10:::benchmark_data_smithi171_122772_object160738:head# 2024-04-20T17:27:56.164 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.164 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a993b038:::benchmark_data_smithi171_122772_object146070:head# 2024-04-20T17:27:56.164 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.164 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a993e2d7:::benchmark_data_smithi171_122772_object108202:head# 2024-04-20T17:27:56.164 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.164 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a995cfa0:::benchmark_data_smithi171_122772_object159093:head# 2024-04-20T17:27:56.164 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.164 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a995f248:::benchmark_data_smithi171_122772_object164768:head# 2024-04-20T17:27:56.164 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.164 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9963bb8:::benchmark_data_smithi171_122772_object86272:head# 2024-04-20T17:27:56.165 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.165 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a996d501:::benchmark_data_smithi171_122772_object155201:head# 2024-04-20T17:27:56.165 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.165 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a99aff1d:::benchmark_data_smithi171_122772_object131573:head# 2024-04-20T17:27:56.165 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.165 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a99b38bf:::benchmark_data_smithi171_122772_object146412:head# 2024-04-20T17:27:56.165 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.165 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a99bcb01:::benchmark_data_smithi171_122772_object78157:head# 2024-04-20T17:27:56.165 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.165 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a99d91f9:::benchmark_data_smithi171_122772_object155827:head# 2024-04-20T17:27:56.165 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.165 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a99f82b5:::benchmark_data_smithi171_122772_object144193:head# 2024-04-20T17:27:56.166 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.166 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a99ff5cf:::benchmark_data_smithi171_122772_object144402:head# 2024-04-20T17:27:56.166 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.166 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9a04e6f:::benchmark_data_smithi171_122772_object110196:head# 2024-04-20T17:27:56.166 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.166 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9a21119:::benchmark_data_smithi171_122772_object134620:head# 2024-04-20T17:27:56.166 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.166 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9a442f3:::benchmark_data_smithi171_122772_object152152:head# 2024-04-20T17:27:56.166 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.166 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9a65a1a:::benchmark_data_smithi171_122772_object90482:head# 2024-04-20T17:27:56.166 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.166 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9a8aff2:::benchmark_data_smithi171_122772_object126943:head# 2024-04-20T17:27:56.166 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.167 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9aa1022:::benchmark_data_smithi171_122772_object161119:head# 2024-04-20T17:27:56.167 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.167 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9abb11f:::benchmark_data_smithi171_122772_object130911:head# 2024-04-20T17:27:56.167 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.167 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9ada06d:::benchmark_data_smithi171_122772_object109098:head# 2024-04-20T17:27:56.167 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.167 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9adec7b:::benchmark_data_smithi171_122772_object119243:head# 2024-04-20T17:27:56.167 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.167 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9afcfda:::benchmark_data_smithi171_122772_object113134:head# 2024-04-20T17:27:56.167 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.167 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9b0fd6c:::benchmark_data_smithi171_122772_object161716:head# 2024-04-20T17:27:56.167 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.168 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9b16756:::benchmark_data_smithi171_122772_object116591:head# 2024-04-20T17:27:56.168 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.168 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9b249fc:::benchmark_data_smithi171_122772_object96443:head# 2024-04-20T17:27:56.168 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.168 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9b2bdb4:::benchmark_data_smithi171_122772_object135780:head# 2024-04-20T17:27:56.168 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.168 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9b40a64:::benchmark_data_smithi171_122772_object96035:head# 2024-04-20T17:27:56.168 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.168 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9b41c06:::benchmark_data_smithi171_122772_object146621:head# 2024-04-20T17:27:56.168 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.168 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9b4223d:::benchmark_data_smithi171_122772_object90263:head# 2024-04-20T17:27:56.168 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.168 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9b5166c:::benchmark_data_smithi171_122772_object120472:head# 2024-04-20T17:27:56.169 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.169 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9b58ec9:::benchmark_data_smithi171_122772_object117034:head# 2024-04-20T17:27:56.169 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.169 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9b590cc:::benchmark_data_smithi171_122772_object124914:head# 2024-04-20T17:27:56.169 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.169 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9b5c92f:::benchmark_data_smithi171_122772_object104958:head# 2024-04-20T17:27:56.169 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.169 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9b638d7:::benchmark_data_smithi171_122772_object169895:head# 2024-04-20T17:27:56.169 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.169 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9b7e707:::benchmark_data_smithi171_122772_object130579:head# 2024-04-20T17:27:56.169 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.169 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9b8616e:::benchmark_data_smithi171_122772_object163126:head# 2024-04-20T17:27:56.170 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.170 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9b87df9:::benchmark_data_smithi171_122772_object151446:head# 2024-04-20T17:27:56.170 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.170 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9b8da28:::benchmark_data_smithi171_122772_object135518:head# 2024-04-20T17:27:56.170 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.170 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9b9163b:::benchmark_data_smithi171_122772_object89638:head# 2024-04-20T17:27:56.170 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.170 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9ba33c4:::benchmark_data_smithi171_122772_object149686:head# 2024-04-20T17:27:56.170 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.171 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9ba91b8:::benchmark_data_smithi171_122772_object125390:head# 2024-04-20T17:27:56.171 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.171 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9bb3004:::benchmark_data_smithi171_122772_object168083:head# 2024-04-20T17:27:56.171 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.171 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9bc5477:::benchmark_data_smithi171_122772_object107373:head# 2024-04-20T17:27:56.171 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.171 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9bc8268:::benchmark_data_smithi171_122772_object113708:head# 2024-04-20T17:27:56.171 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.171 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9bc8b0a:::benchmark_data_smithi171_122772_object118486:head# 2024-04-20T17:27:56.171 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.171 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9bcdb29:::benchmark_data_smithi171_122772_object147220:head# 2024-04-20T17:27:56.171 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.172 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9bd14c5:::benchmark_data_smithi171_122772_object117176:head# 2024-04-20T17:27:56.172 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.172 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9be0989:::benchmark_data_smithi171_122772_object167844:head# 2024-04-20T17:27:56.172 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.172 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9be9c24:::benchmark_data_smithi171_122772_object101109:head# 2024-04-20T17:27:56.172 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.172 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9bec3e2:::benchmark_data_smithi171_122772_object126568:head# 2024-04-20T17:27:56.172 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.172 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9c0e7c2:::benchmark_data_smithi171_122772_object163155:head# 2024-04-20T17:27:56.172 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.172 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9c10212:::benchmark_data_smithi171_122772_object165258:head# 2024-04-20T17:27:56.172 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.173 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9c20e01:::benchmark_data_smithi171_122772_object115726:head# 2024-04-20T17:27:56.173 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.173 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9c33389:::benchmark_data_smithi171_122772_object140440:head# 2024-04-20T17:27:56.173 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.173 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9c353ee:::benchmark_data_smithi171_122772_object155898:head# 2024-04-20T17:27:56.173 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.173 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9c393c8:::benchmark_data_smithi171_122772_object122709:head# 2024-04-20T17:27:56.173 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.173 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9c53772:::benchmark_data_smithi171_122772_object88988:head# 2024-04-20T17:27:56.173 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.173 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9c568e3:::benchmark_data_smithi171_122772_object102402:head# 2024-04-20T17:27:56.173 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.173 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9c59a28:::benchmark_data_smithi171_122772_object110327:head# 2024-04-20T17:27:56.174 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.174 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9c6573c:::benchmark_data_smithi171_122772_object115608:head# 2024-04-20T17:27:56.174 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.174 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9c658c8:::benchmark_data_smithi171_122772_object165290:head# 2024-04-20T17:27:56.174 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.174 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9c84877:::benchmark_data_smithi171_122772_object160135:head# 2024-04-20T17:27:56.174 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.174 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9c9bdce:::benchmark_data_smithi171_122772_object96838:head# 2024-04-20T17:27:56.174 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.174 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9ca1946:::benchmark_data_smithi171_122772_object118251:head# 2024-04-20T17:27:56.174 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.174 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9ca8530:::benchmark_data_smithi171_122772_object106351:head# 2024-04-20T17:27:56.175 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.175 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9caaf77:::benchmark_data_smithi171_122772_object130162:head# 2024-04-20T17:27:56.175 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.175 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9cadf30:::benchmark_data_smithi171_122772_object126189:head# 2024-04-20T17:27:56.175 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.175 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9caf3e1:::benchmark_data_smithi171_122772_object159655:head# 2024-04-20T17:27:56.175 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.175 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9caf8be:::benchmark_data_smithi171_122772_object111285:head# 2024-04-20T17:27:56.175 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.175 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9cb6623:::benchmark_data_smithi171_122772_object87636:head# 2024-04-20T17:27:56.175 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.175 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9ccd0d6:::benchmark_data_smithi171_122772_object117380:head# 2024-04-20T17:27:56.175 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.176 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9cd30fd:::benchmark_data_smithi171_122772_object125844:head# 2024-04-20T17:27:56.176 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.176 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9cd8e82:::benchmark_data_smithi171_122772_object136152:head# 2024-04-20T17:27:56.176 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.176 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9cdff6b:::benchmark_data_smithi171_122772_object137128:head# 2024-04-20T17:27:56.176 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.176 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9ce4def:::benchmark_data_smithi171_122772_object96855:head# 2024-04-20T17:27:56.176 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.176 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9cf210f:::benchmark_data_smithi171_122772_object142330:head# 2024-04-20T17:27:56.176 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.176 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9d08402:::benchmark_data_smithi171_122772_object137598:head# 2024-04-20T17:27:56.176 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.176 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9d15a92:::benchmark_data_smithi171_122772_object129765:head# 2024-04-20T17:27:56.177 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.177 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9d179d4:::benchmark_data_smithi171_122772_object157163:head# 2024-04-20T17:27:56.177 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.177 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9d1a5b9:::benchmark_data_smithi171_122772_object154547:head# 2024-04-20T17:27:56.177 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.177 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9d1d11e:::benchmark_data_smithi171_122772_object105965:head# 2024-04-20T17:27:56.177 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.177 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9d26cff:::benchmark_data_smithi171_122772_object157202:head# 2024-04-20T17:27:56.177 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.177 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9d2f1a5:::benchmark_data_smithi171_122772_object141363:head# 2024-04-20T17:27:56.177 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.177 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9d3239d:::benchmark_data_smithi171_122772_object97288:head# 2024-04-20T17:27:56.178 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.178 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9d436b5:::benchmark_data_smithi171_122772_object95800:head# 2024-04-20T17:27:56.178 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.178 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9d50bac:::benchmark_data_smithi171_122772_object141282:head# 2024-04-20T17:27:56.178 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.178 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9d5d47c:::benchmark_data_smithi171_122772_object124826:head# 2024-04-20T17:27:56.178 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.178 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9d6ccbb:::benchmark_data_smithi171_122772_object133431:head# 2024-04-20T17:27:56.178 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.178 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9d81e46:::benchmark_data_smithi171_122772_object131442:head# 2024-04-20T17:27:56.178 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.178 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9d8ff68:::benchmark_data_smithi171_122772_object98351:head# 2024-04-20T17:27:56.178 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.179 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9d9268c:::benchmark_data_smithi171_122772_object95572:head# 2024-04-20T17:27:56.179 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.179 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9d9ae9f:::benchmark_data_smithi171_122772_object139349:head# 2024-04-20T17:27:56.179 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.179 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9d9e09e:::benchmark_data_smithi171_122772_object140696:head# 2024-04-20T17:27:56.179 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.179 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9dac594:::benchmark_data_smithi171_122772_object162210:head# 2024-04-20T17:27:56.179 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.179 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9db6e83:::benchmark_data_smithi171_122772_object80549:head# 2024-04-20T17:27:56.181 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.181 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9dbe7da:::benchmark_data_smithi171_122772_object90879:head# 2024-04-20T17:27:56.182 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.182 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9dd3409:::benchmark_data_smithi171_122772_object90485:head# 2024-04-20T17:27:56.182 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.182 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9dd521a:::benchmark_data_smithi171_122772_object101073:head# 2024-04-20T17:27:56.182 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.182 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9dd630c:::benchmark_data_smithi171_122772_object132553:head# 2024-04-20T17:27:56.182 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.182 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9dde5a1:::benchmark_data_smithi171_122772_object109568:head# 2024-04-20T17:27:56.182 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.182 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9de0643:::benchmark_data_smithi171_122772_object98489:head# 2024-04-20T17:27:56.182 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.182 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9de9090:::benchmark_data_smithi171_122772_object141987:head# 2024-04-20T17:27:56.182 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.183 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9df70ce:::benchmark_data_smithi171_122772_object112253:head# 2024-04-20T17:27:56.183 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.183 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9dfa9a6:::benchmark_data_smithi171_122772_object98006:head# 2024-04-20T17:27:56.183 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.183 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9e036b8:::benchmark_data_smithi171_122772_object133349:head# 2024-04-20T17:27:56.183 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.183 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9e04e6b:::benchmark_data_smithi171_122772_object169190:head# 2024-04-20T17:27:56.183 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.183 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9e09006:::benchmark_data_smithi171_122772_object85626:head# 2024-04-20T17:27:56.183 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.183 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9e0bf8b:::benchmark_data_smithi171_122772_object85101:head# 2024-04-20T17:27:56.183 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.184 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9e0fa5e:::benchmark_data_smithi171_122772_object97784:head# 2024-04-20T17:27:56.184 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.184 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9e1d3db:::benchmark_data_smithi171_122772_object110904:head# 2024-04-20T17:27:56.184 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.184 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9e28f0e:::benchmark_data_smithi171_122772_object118717:head# 2024-04-20T17:27:56.184 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.184 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9e2d148:::benchmark_data_smithi171_122772_object158629:head# 2024-04-20T17:27:56.184 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.184 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9e406c1:::benchmark_data_smithi171_122772_object112521:head# 2024-04-20T17:27:56.184 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.184 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9e42791:::benchmark_data_smithi171_122772_object136916:head# 2024-04-20T17:27:56.184 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.184 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9e443fe:::benchmark_data_smithi171_122772_object132954:head# 2024-04-20T17:27:56.185 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.185 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9e6676e:::benchmark_data_smithi171_122772_object156850:head# 2024-04-20T17:27:56.185 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.185 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9e68875:::benchmark_data_smithi171_122772_object117747:head# 2024-04-20T17:27:56.185 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.185 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9e835b6:::benchmark_data_smithi171_122772_object105167:head# 2024-04-20T17:27:56.185 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.185 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9e88b88:::benchmark_data_smithi171_122772_object157431:head# 2024-04-20T17:27:56.185 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.185 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9ea0d99:::benchmark_data_smithi171_122772_object96695:head# 2024-04-20T17:27:56.185 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.185 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9ea3b2a:::benchmark_data_smithi171_122772_object108774:head# 2024-04-20T17:27:56.186 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.186 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9ed7b41:::benchmark_data_smithi171_122772_object159988:head# 2024-04-20T17:27:56.186 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.186 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9eda8a9:::benchmark_data_smithi171_122772_object89680:head# 2024-04-20T17:27:56.186 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.186 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9ee18ce:::benchmark_data_smithi171_122772_object147714:head# 2024-04-20T17:27:56.186 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.186 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9eef6e3:::benchmark_data_smithi171_122772_object90541:head# 2024-04-20T17:27:56.186 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.186 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9efc0ee:::benchmark_data_smithi171_122772_object133122:head# 2024-04-20T17:27:56.186 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.186 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9f13cfc:::benchmark_data_smithi171_122772_object141796:head# 2024-04-20T17:27:56.186 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.187 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9f16db8:::benchmark_data_smithi171_122772_object155018:head# 2024-04-20T17:27:56.187 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.187 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9f2d220:::benchmark_data_smithi171_122772_object99381:head# 2024-04-20T17:27:56.187 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.187 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9f37401:::benchmark_data_smithi171_122772_object159272:head# 2024-04-20T17:27:56.187 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.187 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9f43302:::benchmark_data_smithi171_122772_object138398:head# 2024-04-20T17:27:56.187 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.187 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9f61cef:::benchmark_data_smithi171_122772_object157374:head# 2024-04-20T17:27:56.187 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.187 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9f691ea:::benchmark_data_smithi171_122772_object118956:head# 2024-04-20T17:27:56.187 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.188 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9f7dad9:::benchmark_data_smithi171_122772_object163979:head# 2024-04-20T17:27:56.188 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.188 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9f88f95:::benchmark_data_smithi171_122772_object103093:head# 2024-04-20T17:27:56.188 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.188 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9f90f5c:::benchmark_data_smithi171_122772_object131412:head# 2024-04-20T17:27:56.188 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.188 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9f94ca1:::benchmark_data_smithi171_122772_object90326:head# 2024-04-20T17:27:56.188 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.188 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9fa1869:::benchmark_data_smithi171_122772_object158138:head# 2024-04-20T17:27:56.188 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.188 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9fb866f:::benchmark_data_smithi171_122772_object80775:head# 2024-04-20T17:27:56.188 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.189 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9fc1d5c:::benchmark_data_smithi171_122772_object124739:head# 2024-04-20T17:27:56.189 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.189 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9fccdf7:::benchmark_data_smithi171_122772_object150841:head# 2024-04-20T17:27:56.189 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.189 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9fd31d6:::benchmark_data_smithi171_122772_object129706:head# 2024-04-20T17:27:56.189 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.189 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9fd9cdf:::benchmark_data_smithi171_122772_object157561:head# 2024-04-20T17:27:56.189 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.189 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:a9fe9f4f:::benchmark_data_smithi171_122772_object153788:head# 2024-04-20T17:27:56.189 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.189 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa000f50:::benchmark_data_smithi171_122772_object163307:head# 2024-04-20T17:27:56.189 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.189 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa00a32b:::benchmark_data_smithi171_122772_object79900:head# 2024-04-20T17:27:56.190 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.190 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa00bb63:::benchmark_data_smithi171_122772_object98167:head# 2024-04-20T17:27:56.190 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.190 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa02e0ad:::benchmark_data_smithi171_122772_object99488:head# 2024-04-20T17:27:56.190 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.190 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa03496d:::benchmark_data_smithi171_122772_object169938:head# 2024-04-20T17:27:56.190 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.190 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa03d7a4:::benchmark_data_smithi171_122772_object103612:head# 2024-04-20T17:27:56.190 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.190 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa04a5ab:::benchmark_data_smithi171_122772_object144821:head# 2024-04-20T17:27:56.190 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.190 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa057b49:::benchmark_data_smithi171_122772_object159588:head# 2024-04-20T17:27:56.191 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.191 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa0590c1:::benchmark_data_smithi171_122772_object137449:head# 2024-04-20T17:27:56.191 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.191 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa05f2de:::benchmark_data_smithi171_122772_object103405:head# 2024-04-20T17:27:56.191 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.191 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa0607d2:::benchmark_data_smithi171_122772_object115535:head# 2024-04-20T17:27:56.191 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.191 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa06b7ad:::benchmark_data_smithi171_122772_object144099:head# 2024-04-20T17:27:56.191 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.191 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa0701f8:::benchmark_data_smithi171_122772_object94760:head# 2024-04-20T17:27:56.191 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.191 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa077474:::benchmark_data_smithi171_122772_object162269:head# 2024-04-20T17:27:56.191 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.192 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa07818e:::benchmark_data_smithi171_122772_object159697:head# 2024-04-20T17:27:56.192 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.192 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa07c358:::benchmark_data_smithi171_122772_object111613:head# 2024-04-20T17:27:56.192 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.192 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa085760:::benchmark_data_smithi171_122772_object116937:head# 2024-04-20T17:27:56.192 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.192 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa0924a9:::benchmark_data_smithi171_122772_object159105:head# 2024-04-20T17:27:56.192 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.192 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa097752:::benchmark_data_smithi171_122772_object158488:head# 2024-04-20T17:27:56.192 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.192 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa099e0f:::benchmark_data_smithi171_122772_object159721:head# 2024-04-20T17:27:56.192 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.192 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa0a8138:::benchmark_data_smithi171_122772_object94985:head# 2024-04-20T17:27:56.193 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.193 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa0ab447:::benchmark_data_smithi171_122772_object169550:head# 2024-04-20T17:27:56.193 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.193 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa0adfe7:::benchmark_data_smithi171_122772_object93242:head# 2024-04-20T17:27:56.193 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.193 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa0b407b:::benchmark_data_smithi171_122772_object101848:head# 2024-04-20T17:27:56.193 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.193 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa0b69e8:::benchmark_data_smithi171_122772_object118446:head# 2024-04-20T17:27:56.193 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.193 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa0bcfae:::benchmark_data_smithi171_122772_object161900:head# 2024-04-20T17:27:56.193 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.193 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa0ccdc9:::benchmark_data_smithi171_122772_object123233:head# 2024-04-20T17:27:56.194 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.194 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa0cd052:::benchmark_data_smithi171_122772_object109015:head# 2024-04-20T17:27:56.194 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.194 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa0cd293:::benchmark_data_smithi171_122772_object141053:head# 2024-04-20T17:27:56.194 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.194 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa0cf47d:::benchmark_data_smithi171_122772_object167272:head# 2024-04-20T17:27:56.194 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.194 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa0dcd00:::benchmark_data_smithi171_122772_object160149:head# 2024-04-20T17:27:56.194 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.194 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa0e9cdb:::benchmark_data_smithi171_122772_object142082:head# 2024-04-20T17:27:56.194 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.194 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa0fc070:::benchmark_data_smithi171_122772_object112761:head# 2024-04-20T17:27:56.194 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.195 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa0fd94f:::benchmark_data_smithi171_122772_object142454:head# 2024-04-20T17:27:56.195 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.195 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa115205:::benchmark_data_smithi171_122772_object140977:head# 2024-04-20T17:27:56.195 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.195 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa117b61:::benchmark_data_smithi171_122772_object108017:head# 2024-04-20T17:27:56.195 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.195 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa120909:::benchmark_data_smithi171_122772_object114350:head# 2024-04-20T17:27:56.195 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.195 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa12c916:::benchmark_data_smithi171_122772_object157684:head# 2024-04-20T17:27:56.195 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.195 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa12ffa8:::benchmark_data_smithi171_122772_object122642:head# 2024-04-20T17:27:56.195 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.195 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa144d49:::benchmark_data_smithi171_122772_object163700:head# 2024-04-20T17:27:56.196 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.196 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa181965:::benchmark_data_smithi171_122772_object111363:head# 2024-04-20T17:27:56.196 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.196 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa189854:::benchmark_data_smithi171_122772_object128178:head# 2024-04-20T17:27:56.196 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.196 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa18da96:::benchmark_data_smithi171_122772_object101370:head# 2024-04-20T17:27:56.196 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.196 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa192ac5:::benchmark_data_smithi171_122772_object122734:head# 2024-04-20T17:27:56.196 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.196 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa1a032f:::benchmark_data_smithi171_122772_object87691:head# 2024-04-20T17:27:56.196 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.196 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa1a6b7a:::benchmark_data_smithi171_122772_object125534:head# 2024-04-20T17:27:56.197 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.197 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa1b784e:::benchmark_data_smithi171_122772_object106564:head# 2024-04-20T17:27:56.197 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.197 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa1bc5c5:::benchmark_data_smithi171_122772_object160150:head# 2024-04-20T17:27:56.197 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.197 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa1cbf9a:::benchmark_data_smithi171_122772_object163610:head# 2024-04-20T17:27:56.197 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.197 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa1ccee9:::benchmark_data_smithi171_122772_object148270:head# 2024-04-20T17:27:56.197 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.197 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa1ee408:::benchmark_data_smithi171_122772_object163811:head# 2024-04-20T17:27:56.197 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.197 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa1ef955:::benchmark_data_smithi171_122772_object87236:head# 2024-04-20T17:27:56.197 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.198 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa1f8ed2:::benchmark_data_smithi171_122772_object127542:head# 2024-04-20T17:27:56.198 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.198 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa20af63:::benchmark_data_smithi171_122772_object159994:head# 2024-04-20T17:27:56.198 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.198 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa222112:::benchmark_data_smithi171_122772_object97946:head# 2024-04-20T17:27:56.198 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.198 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa224c1a:::benchmark_data_smithi171_122772_object82843:head# 2024-04-20T17:27:56.198 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.198 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa22a910:::benchmark_data_smithi171_122772_object114459:head# 2024-04-20T17:27:56.198 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.198 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa22c954:::benchmark_data_smithi171_122772_object133162:head# 2024-04-20T17:27:56.198 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.199 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa234195:::benchmark_data_smithi171_122772_object138940:head# 2024-04-20T17:27:56.199 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.199 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa23b972:::benchmark_data_smithi171_122772_object131234:head# 2024-04-20T17:27:56.199 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.199 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa247052:::benchmark_data_smithi171_122772_object167892:head# 2024-04-20T17:27:56.199 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.199 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa2710b3:::benchmark_data_smithi171_122772_object169471:head# 2024-04-20T17:27:56.199 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.199 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa2765f2:::benchmark_data_smithi171_122772_object90164:head# 2024-04-20T17:27:56.199 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.199 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa2afe09:::benchmark_data_smithi171_122772_object131068:head# 2024-04-20T17:27:56.199 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.199 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa2b63ea:::benchmark_data_smithi171_122772_object149906:head# 2024-04-20T17:27:56.200 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.200 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa2bfd7c:::benchmark_data_smithi171_122772_object157265:head# 2024-04-20T17:27:56.200 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.200 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa2c062f:::benchmark_data_smithi171_122772_object156080:head# 2024-04-20T17:27:56.200 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.200 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa2c38f1:::benchmark_data_smithi171_122772_object108928:head# 2024-04-20T17:27:56.200 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.200 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa2c5801:::benchmark_data_smithi171_122772_object148906:head# 2024-04-20T17:27:56.200 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.200 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa2c7c60:::benchmark_data_smithi171_122772_object131953:head# 2024-04-20T17:27:56.200 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.200 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa2d3435:::benchmark_data_smithi171_122772_object133757:head# 2024-04-20T17:27:56.200 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.201 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa2dd3fa:::benchmark_data_smithi171_122772_object160136:head# 2024-04-20T17:27:56.201 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.201 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa2e072d:::benchmark_data_smithi171_122772_object117687:head# 2024-04-20T17:27:56.201 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.201 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa2e1c7c:::benchmark_data_smithi171_122772_object134322:head# 2024-04-20T17:27:56.201 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.201 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa2e2d75:::benchmark_data_smithi171_122772_object82681:head# 2024-04-20T17:27:56.201 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.201 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa2e7870:::benchmark_data_smithi171_122772_object118970:head# 2024-04-20T17:27:56.201 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.201 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa2f0519:::benchmark_data_smithi171_122772_object113199:head# 2024-04-20T17:27:56.201 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.202 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa2fd1e2:::benchmark_data_smithi171_122772_object120315:head# 2024-04-20T17:27:56.202 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.202 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa304edc:::benchmark_data_smithi171_122772_object168933:head# 2024-04-20T17:27:56.202 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.202 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa30a344:::benchmark_data_smithi171_122772_object148399:head# 2024-04-20T17:27:56.202 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.202 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa326b6b:::benchmark_data_smithi171_122772_object139942:head# 2024-04-20T17:27:56.202 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.202 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa32d26e:::benchmark_data_smithi171_122772_object155809:head# 2024-04-20T17:27:56.202 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.202 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa3349a3:::benchmark_data_smithi171_122772_object163704:head# 2024-04-20T17:27:56.202 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.203 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa33fddd:::benchmark_data_smithi171_122772_object107046:head# 2024-04-20T17:27:56.203 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.203 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa353fde:::benchmark_data_smithi171_122772_object151845:head# 2024-04-20T17:27:56.203 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.203 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa365f89:::benchmark_data_smithi171_122772_object78450:head# 2024-04-20T17:27:56.203 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.203 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa368e62:::benchmark_data_smithi171_122772_object78618:head# 2024-04-20T17:27:56.203 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.203 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa36dd3b:::benchmark_data_smithi171_122772_object96032:head# 2024-04-20T17:27:56.203 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.203 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa375c6d:::benchmark_data_smithi171_122772_object134860:head# 2024-04-20T17:27:56.203 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.203 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa37fefe:::benchmark_data_smithi171_122772_object79797:head# 2024-04-20T17:27:56.204 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.204 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa387ea7:::benchmark_data_smithi171_122772_object115102:head# 2024-04-20T17:27:56.204 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.204 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa38d33f:::benchmark_data_smithi171_122772_object130781:head# 2024-04-20T17:27:56.204 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.204 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa3b27c9:::benchmark_data_smithi171_122772_object130149:head# 2024-04-20T17:27:56.204 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.204 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa3b38f4:::benchmark_data_smithi171_122772_object98159:head# 2024-04-20T17:27:56.204 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.204 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa3b9827:::benchmark_data_smithi171_122772_object93863:head# 2024-04-20T17:27:56.204 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.204 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa3cdef5:::benchmark_data_smithi171_122772_object89803:head# 2024-04-20T17:27:56.205 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.205 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa3d3b35:::benchmark_data_smithi171_122772_object87918:head# 2024-04-20T17:27:56.205 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.205 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa3e2854:::benchmark_data_smithi171_122772_object156178:head# 2024-04-20T17:27:56.205 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.205 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa3eecf8:::benchmark_data_smithi171_122772_object158670:head# 2024-04-20T17:27:56.205 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.205 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa402b4e:::benchmark_data_smithi171_122772_object125670:head# 2024-04-20T17:27:56.205 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.205 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa404f5a:::benchmark_data_smithi171_122772_object152905:head# 2024-04-20T17:27:56.205 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.205 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa41447a:::benchmark_data_smithi171_122772_object114800:head# 2024-04-20T17:27:56.205 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.206 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa421b78:::benchmark_data_smithi171_122772_object151136:head# 2024-04-20T17:27:56.206 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.206 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa43d878:::benchmark_data_smithi171_122772_object108557:head# 2024-04-20T17:27:56.206 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.206 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa450a87:::benchmark_data_smithi171_122772_object104431:head# 2024-04-20T17:27:56.206 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.206 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa45235a:::benchmark_data_smithi171_122772_object145450:head# 2024-04-20T17:27:56.206 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.206 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa45c748:::benchmark_data_smithi171_122772_object136632:head# 2024-04-20T17:27:56.206 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.206 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa4609f6:::benchmark_data_smithi171_122772_object120793:head# 2024-04-20T17:27:56.206 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.207 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa461824:::benchmark_data_smithi171_122772_object144932:head# 2024-04-20T17:27:56.207 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.207 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa463c79:::benchmark_data_smithi171_122772_object79805:head# 2024-04-20T17:27:56.207 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.207 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa499061:::benchmark_data_smithi171_122772_object118581:head# 2024-04-20T17:27:56.207 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.207 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa4a284f:::benchmark_data_smithi171_122772_object145719:head# 2024-04-20T17:27:56.207 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.207 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa4a3c4b:::benchmark_data_smithi171_122772_object132302:head# 2024-04-20T17:27:56.207 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.207 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa4ac7bb:::benchmark_data_smithi171_122772_object162600:head# 2024-04-20T17:27:56.207 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.207 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa4b75db:::benchmark_data_smithi171_122772_object147867:head# 2024-04-20T17:27:56.208 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.208 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa4bcac9:::benchmark_data_smithi171_122772_object80275:head# 2024-04-20T17:27:56.208 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.208 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa4c069b:::benchmark_data_smithi171_122772_object145316:head# 2024-04-20T17:27:56.208 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.208 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa4c3142:::benchmark_data_smithi171_122772_object126976:head# 2024-04-20T17:27:56.208 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.208 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa4cb8ae:::benchmark_data_smithi171_122772_object101306:head# 2024-04-20T17:27:56.208 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.208 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa4f0393:::benchmark_data_smithi171_122772_object86627:head# 2024-04-20T17:27:56.208 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.208 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa4f9760:::benchmark_data_smithi171_122772_object153891:head# 2024-04-20T17:27:56.209 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.209 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa4fd23a:::benchmark_data_smithi171_122772_object97449:head# 2024-04-20T17:27:56.209 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.209 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa5167ff:::benchmark_data_smithi171_122772_object84440:head# 2024-04-20T17:27:56.209 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.209 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa51d541:::benchmark_data_smithi171_122772_object87397:head# 2024-04-20T17:27:56.209 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.209 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa525dee:::benchmark_data_smithi171_122772_object155373:head# 2024-04-20T17:27:56.209 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.209 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa52c024:::benchmark_data_smithi171_122772_object127705:head# 2024-04-20T17:27:56.209 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.209 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa52fde5:::benchmark_data_smithi171_122772_object97457:head# 2024-04-20T17:27:56.209 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.210 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa530123:::benchmark_data_smithi171_122772_object96082:head# 2024-04-20T17:27:56.210 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.210 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa5343d8:::benchmark_data_smithi171_122772_object166802:head# 2024-04-20T17:27:56.210 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.210 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa53ddb4:::benchmark_data_smithi171_122772_object142614:head# 2024-04-20T17:27:56.210 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.210 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa53f358:::benchmark_data_smithi171_122772_object141507:head# 2024-04-20T17:27:56.210 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.210 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa540159:::benchmark_data_smithi171_122772_object151390:head# 2024-04-20T17:27:56.210 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.210 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa54795f:::benchmark_data_smithi171_122772_object161805:head# 2024-04-20T17:27:56.210 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.211 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa54e559:::benchmark_data_smithi171_122772_object146967:head# 2024-04-20T17:27:56.211 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.211 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa54eef1:::benchmark_data_smithi171_122772_object145856:head# 2024-04-20T17:27:56.211 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.211 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa55302b:::benchmark_data_smithi171_122772_object103295:head# 2024-04-20T17:27:56.211 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.211 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa5605c6:::benchmark_data_smithi171_122772_object147863:head# 2024-04-20T17:27:56.211 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.211 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa58cee5:::benchmark_data_smithi171_122772_object98690:head# 2024-04-20T17:27:56.211 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.211 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa5a8853:::benchmark_data_smithi171_122772_object156838:head# 2024-04-20T17:27:56.211 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.211 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa5b355e:::benchmark_data_smithi171_122772_object163228:head# 2024-04-20T17:27:56.212 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.212 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa5c0f91:::benchmark_data_smithi171_122772_object88442:head# 2024-04-20T17:27:56.212 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.212 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa5c11b2:::benchmark_data_smithi171_122772_object106902:head# 2024-04-20T17:27:56.212 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.212 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa5d465c:::benchmark_data_smithi171_122772_object164556:head# 2024-04-20T17:27:56.212 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.212 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa5d475e:::benchmark_data_smithi171_122772_object143642:head# 2024-04-20T17:27:56.212 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.212 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa5e7666:::benchmark_data_smithi171_122772_object102308:head# 2024-04-20T17:27:56.212 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.212 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa5eb695:::benchmark_data_smithi171_122772_object107448:head# 2024-04-20T17:27:56.212 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.213 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa5f0227:::benchmark_data_smithi171_122772_object150105:head# 2024-04-20T17:27:56.213 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.213 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa5f2143:::benchmark_data_smithi171_122772_object140090:head# 2024-04-20T17:27:56.213 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.213 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa5f80d2:::benchmark_data_smithi171_122772_object150254:head# 2024-04-20T17:27:56.213 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.213 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa60edea:::benchmark_data_smithi171_122772_object84896:head# 2024-04-20T17:27:56.213 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.213 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa611ef4:::benchmark_data_smithi171_122772_object83003:head# 2024-04-20T17:27:56.213 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.213 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa623170:::benchmark_data_smithi171_122772_object105382:head# 2024-04-20T17:27:56.213 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.214 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa634763:::benchmark_data_smithi171_122772_object156210:head# 2024-04-20T17:27:56.214 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.214 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa638beb:::benchmark_data_smithi171_122772_object169400:head# 2024-04-20T17:27:56.214 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.214 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa63f45b:::benchmark_data_smithi171_122772_object99107:head# 2024-04-20T17:27:56.214 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.214 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa642129:::benchmark_data_smithi171_122772_object164100:head# 2024-04-20T17:27:56.214 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.214 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa6562c8:::benchmark_data_smithi171_122772_object78772:head# 2024-04-20T17:27:56.214 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.214 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa65ae74:::benchmark_data_smithi171_122772_object144145:head# 2024-04-20T17:27:56.214 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.215 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa65eb30:::benchmark_data_smithi171_122772_object119281:head# 2024-04-20T17:27:56.215 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.215 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa66d796:::benchmark_data_smithi171_122772_object118763:head# 2024-04-20T17:27:56.215 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.215 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa67465d:::benchmark_data_smithi171_122772_object166775:head# 2024-04-20T17:27:56.215 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.215 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa67b413:::benchmark_data_smithi171_122772_object145357:head# 2024-04-20T17:27:56.215 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.215 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa67b42b:::benchmark_data_smithi171_122772_object142685:head# 2024-04-20T17:27:56.215 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.215 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa67bc41:::benchmark_data_smithi171_122772_object118151:head# 2024-04-20T17:27:56.215 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.215 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa685e8d:::benchmark_data_smithi171_122772_object128500:head# 2024-04-20T17:27:56.216 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.216 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa687c15:::benchmark_data_smithi171_122772_object158654:head# 2024-04-20T17:27:56.216 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.216 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa68c76b:::benchmark_data_smithi171_122772_object130336:head# 2024-04-20T17:27:56.216 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.216 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa690698:::benchmark_data_smithi171_122772_object153508:head# 2024-04-20T17:27:56.216 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.216 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa6a289a:::benchmark_data_smithi171_122772_object121854:head# 2024-04-20T17:27:56.216 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.216 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa6ae3b7:::benchmark_data_smithi171_122772_object146314:head# 2024-04-20T17:27:56.216 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.216 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa6bd85f:::benchmark_data_smithi171_122772_object154710:head# 2024-04-20T17:27:56.217 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.217 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa6c3550:::benchmark_data_smithi171_122772_object147215:head# 2024-04-20T17:27:56.217 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.217 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa6ccba6:::benchmark_data_smithi171_122772_object168273:head# 2024-04-20T17:27:56.217 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.217 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa6d138c:::benchmark_data_smithi171_122772_object147284:head# 2024-04-20T17:27:56.217 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.217 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa6de083:::benchmark_data_smithi171_122772_object104578:head# 2024-04-20T17:27:56.217 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.217 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa6e899f:::benchmark_data_smithi171_122772_object155846:head# 2024-04-20T17:27:56.217 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.217 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa6fe5eb:::benchmark_data_smithi171_122772_object131574:head# 2024-04-20T17:27:56.217 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.218 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa705016:::benchmark_data_smithi171_122772_object132235:head# 2024-04-20T17:27:56.218 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.218 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa71758e:::benchmark_data_smithi171_122772_object109958:head# 2024-04-20T17:27:56.218 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.218 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa7216ad:::benchmark_data_smithi171_122772_object155752:head# 2024-04-20T17:27:56.218 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.218 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa725dff:::benchmark_data_smithi171_122772_object109793:head# 2024-04-20T17:27:56.218 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.218 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa72b4ea:::benchmark_data_smithi171_122772_object164671:head# 2024-04-20T17:27:56.218 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.218 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa73daab:::benchmark_data_smithi171_122772_object148844:head# 2024-04-20T17:27:56.219 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.219 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa746522:::benchmark_data_smithi171_122772_object134644:head# 2024-04-20T17:27:56.219 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.219 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa7471d0:::benchmark_data_smithi171_122772_object82432:head# 2024-04-20T17:27:56.219 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.219 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa74a00b:::benchmark_data_smithi171_122772_object148097:head# 2024-04-20T17:27:56.219 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.219 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa7508ee:::benchmark_data_smithi171_122772_object117585:head# 2024-04-20T17:27:56.219 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.219 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa752767:::benchmark_data_smithi171_122772_object143636:head# 2024-04-20T17:27:56.219 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.219 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa763625:::benchmark_data_smithi171_122772_object119559:head# 2024-04-20T17:27:56.219 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.220 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa76acba:::benchmark_data_smithi171_122772_object147001:head# 2024-04-20T17:27:56.220 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.220 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa76d479:::benchmark_data_smithi171_122772_object166021:head# 2024-04-20T17:27:56.220 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.220 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa7725bb:::benchmark_data_smithi171_122772_object146140:head# 2024-04-20T17:27:56.220 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.220 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa775c60:::benchmark_data_smithi171_122772_object166688:head# 2024-04-20T17:27:56.220 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.220 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa775d50:::benchmark_data_smithi171_122772_object137386:head# 2024-04-20T17:27:56.220 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.220 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa780cf8:::benchmark_data_smithi171_122772_object129938:head# 2024-04-20T17:27:56.221 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.221 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa7a4605:::benchmark_data_smithi171_122772_object91130:head# 2024-04-20T17:27:56.221 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.221 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa7a6eaa:::benchmark_data_smithi171_122772_object134118:head# 2024-04-20T17:27:56.221 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.221 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa7a74c8:::benchmark_data_smithi171_122772_object168918:head# 2024-04-20T17:27:56.221 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.221 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa7af730:::benchmark_data_smithi171_122772_object79057:head# 2024-04-20T17:27:56.221 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.221 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa7b13ae:::benchmark_data_smithi171_122772_object84796:head# 2024-04-20T17:27:56.221 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.221 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa7b3838:::benchmark_data_smithi171_122772_object144436:head# 2024-04-20T17:27:56.221 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.222 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa7c4d3e:::benchmark_data_smithi171_122772_object108126:head# 2024-04-20T17:27:56.222 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.222 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa7e4da9:::benchmark_data_smithi171_122772_object87969:head# 2024-04-20T17:27:56.222 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.222 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa7f59c8:::benchmark_data_smithi171_122772_object142019:head# 2024-04-20T17:27:56.222 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.222 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa7fec98:::benchmark_data_smithi171_122772_object100129:head# 2024-04-20T17:27:56.222 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.222 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa80617a:::benchmark_data_smithi171_122772_object152673:head# 2024-04-20T17:27:56.222 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.222 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa81add1:::benchmark_data_smithi171_122772_object133821:head# 2024-04-20T17:27:56.223 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.223 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa8657ab:::benchmark_data_smithi171_122772_object101915:head# 2024-04-20T17:27:56.223 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.223 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa86c0e6:::benchmark_data_smithi171_122772_object147303:head# 2024-04-20T17:27:56.223 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.223 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa88fc50:::benchmark_data_smithi171_122772_object122845:head# 2024-04-20T17:27:56.223 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.223 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa89819a:::benchmark_data_smithi171_122772_object160310:head# 2024-04-20T17:27:56.223 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.223 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa8abe5a:::benchmark_data_smithi171_122772_object154514:head# 2024-04-20T17:27:56.223 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.223 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa8acb14:::benchmark_data_smithi171_122772_object87464:head# 2024-04-20T17:27:56.224 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.224 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa8e9b97:::benchmark_data_smithi171_122772_object102054:head# 2024-04-20T17:27:56.224 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.224 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa8ebc18:::benchmark_data_smithi171_122772_object152307:head# 2024-04-20T17:27:56.224 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.224 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa8fb87e:::benchmark_data_smithi171_122772_object147508:head# 2024-04-20T17:27:56.224 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.224 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa904685:::benchmark_data_smithi171_122772_object102134:head# 2024-04-20T17:27:56.224 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.224 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa904956:::benchmark_data_smithi171_122772_object164762:head# 2024-04-20T17:27:56.224 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.224 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa904fc0:::benchmark_data_smithi171_122772_object139140:head# 2024-04-20T17:27:56.225 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.225 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa916661:::benchmark_data_smithi171_122772_object165232:head# 2024-04-20T17:27:56.225 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.225 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa92b71e:::benchmark_data_smithi171_122772_object109866:head# 2024-04-20T17:27:56.225 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.225 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa93641a:::benchmark_data_smithi171_122772_object81065:head# 2024-04-20T17:27:56.225 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.225 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa95428a:::benchmark_data_smithi171_122772_object162489:head# 2024-04-20T17:27:56.225 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.225 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa95a55d:::benchmark_data_smithi171_122772_object103137:head# 2024-04-20T17:27:56.225 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.225 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa961f17:::benchmark_data_smithi171_122772_object165812:head# 2024-04-20T17:27:56.225 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.226 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa9649ae:::benchmark_data_smithi171_122772_object132545:head# 2024-04-20T17:27:56.226 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.226 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa972047:::benchmark_data_smithi171_122772_object164757:head# 2024-04-20T17:27:56.226 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.226 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa97c64a:::benchmark_data_smithi171_122772_object166647:head# 2024-04-20T17:27:56.226 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.226 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa97e228:::benchmark_data_smithi171_122772_object95684:head# 2024-04-20T17:27:56.226 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.226 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa97e3f8:::benchmark_data_smithi171_122772_object125094:head# 2024-04-20T17:27:56.226 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.226 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa988695:::benchmark_data_smithi171_122772_object170147:head# 2024-04-20T17:27:56.226 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.227 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa9a5f60:::benchmark_data_smithi171_122772_object119939:head# 2024-04-20T17:27:56.227 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.227 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa9b84ba:::benchmark_data_smithi171_122772_object151094:head# 2024-04-20T17:27:56.227 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.227 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa9bce43:::benchmark_data_smithi171_122772_object168759:head# 2024-04-20T17:27:56.227 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.227 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa9cb6b7:::benchmark_data_smithi171_122772_object149115:head# 2024-04-20T17:27:56.227 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.227 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa9d7139:::benchmark_data_smithi171_122772_object133857:head# 2024-04-20T17:27:56.227 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.227 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa9dcfac:::benchmark_data_smithi171_122772_object116208:head# 2024-04-20T17:27:56.227 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.227 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aa9ec56f:::benchmark_data_smithi171_122772_object102144:head# 2024-04-20T17:27:56.228 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.228 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaa02996:::benchmark_data_smithi171_122772_object123739:head# 2024-04-20T17:27:56.228 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.228 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaa26dec:::benchmark_data_smithi171_122772_object98183:head# 2024-04-20T17:27:56.228 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.228 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaa31d2d:::benchmark_data_smithi171_122772_object136121:head# 2024-04-20T17:27:56.228 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.228 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaa3ce7d:::benchmark_data_smithi171_122772_object163183:head# 2024-04-20T17:27:56.228 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.228 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaa45304:::benchmark_data_smithi171_122772_object106901:head# 2024-04-20T17:27:56.228 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.228 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaa48f82:::benchmark_data_smithi171_122772_object160680:head# 2024-04-20T17:27:56.229 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.229 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaa4cab3:::benchmark_data_smithi171_122772_object99221:head# 2024-04-20T17:27:56.229 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.229 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaa50d58:::benchmark_data_smithi171_122772_object131263:head# 2024-04-20T17:27:56.229 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.229 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaa53087:::benchmark_data_smithi171_122772_object159333:head# 2024-04-20T17:27:56.229 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.229 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaa57dbb:::benchmark_data_smithi171_122772_object102755:head# 2024-04-20T17:27:56.229 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.229 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaa5b83d:::benchmark_data_smithi171_122772_object135601:head# 2024-04-20T17:27:56.229 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.229 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaa71be2:::benchmark_data_smithi171_122772_object157245:head# 2024-04-20T17:27:56.229 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.230 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaa7f338:::benchmark_data_smithi171_122772_object169100:head# 2024-04-20T17:27:56.230 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.230 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaa8bed5:::benchmark_data_smithi171_122772_object152250:head# 2024-04-20T17:27:56.230 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.230 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaa91788:::benchmark_data_smithi171_122772_object121871:head# 2024-04-20T17:27:56.230 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.230 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaa99071:::benchmark_data_smithi171_122772_object89261:head# 2024-04-20T17:27:56.230 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.230 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaabf934:::benchmark_data_smithi171_122772_object107941:head# 2024-04-20T17:27:56.230 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.230 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaac8415:::benchmark_data_smithi171_122772_object122829:head# 2024-04-20T17:27:56.230 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.231 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaad054f:::benchmark_data_smithi171_122772_object131057:head# 2024-04-20T17:27:56.231 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.231 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaad6b19:::benchmark_data_smithi171_122772_object116124:head# 2024-04-20T17:27:56.231 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.231 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaad856e:::benchmark_data_smithi171_122772_object79958:head# 2024-04-20T17:27:56.231 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.231 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaadd71c:::benchmark_data_smithi171_122772_object133571:head# 2024-04-20T17:27:56.231 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.231 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aab3996a:::benchmark_data_smithi171_122772_object102305:head# 2024-04-20T17:27:56.231 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.231 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aab435da:::benchmark_data_smithi171_122772_object107839:head# 2024-04-20T17:27:56.231 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.231 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aab5762d:::benchmark_data_smithi171_122772_object113668:head# 2024-04-20T17:27:56.232 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.232 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aab5ad9c:::benchmark_data_smithi171_122772_object78360:head# 2024-04-20T17:27:56.232 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.232 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aab7f488:::benchmark_data_smithi171_122772_object154989:head# 2024-04-20T17:27:56.232 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.232 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aab8d552:::benchmark_data_smithi171_122772_object115393:head# 2024-04-20T17:27:56.232 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.232 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aab8dd47:::benchmark_data_smithi171_122772_object90030:head# 2024-04-20T17:27:56.232 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.232 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aab8e1bf:::benchmark_data_smithi171_122772_object126040:head# 2024-04-20T17:27:56.232 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.233 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aab9ab91:::benchmark_data_smithi171_122772_object144272:head# 2024-04-20T17:27:56.233 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.233 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaba3182:::benchmark_data_smithi171_122772_object88070:head# 2024-04-20T17:27:56.233 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.233 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aabdbff8:::benchmark_data_smithi171_122772_object129638:head# 2024-04-20T17:27:56.233 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.233 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aabed876:::benchmark_data_smithi171_122772_object165919:head# 2024-04-20T17:27:56.233 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.233 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aac001c4:::benchmark_data_smithi171_122772_object119520:head# 2024-04-20T17:27:56.233 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.233 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aac0f81b:::benchmark_data_smithi171_122772_object140606:head# 2024-04-20T17:27:56.233 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.234 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aac1939d:::benchmark_data_smithi171_122772_object141818:head# 2024-04-20T17:27:56.234 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.234 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aac23016:::benchmark_data_smithi171_122772_object155015:head# 2024-04-20T17:27:56.234 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.234 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aac32d4f:::benchmark_data_smithi171_122772_object80043:head# 2024-04-20T17:27:56.234 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.234 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aac4d67a:::benchmark_data_smithi171_122772_object80993:head# 2024-04-20T17:27:56.234 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.234 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aac4e4a6:::benchmark_data_smithi171_122772_object83234:head# 2024-04-20T17:27:56.234 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.234 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aac5dccb:::benchmark_data_smithi171_122772_object97562:head# 2024-04-20T17:27:56.234 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.235 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aac68a7d:::benchmark_data_smithi171_122772_object162355:head# 2024-04-20T17:27:56.235 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.235 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aac6ff10:::benchmark_data_smithi171_122772_object137682:head# 2024-04-20T17:27:56.235 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.235 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aac867eb:::benchmark_data_smithi171_122772_object159605:head# 2024-04-20T17:27:56.235 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.235 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aac912fc:::benchmark_data_smithi171_122772_object101825:head# 2024-04-20T17:27:56.235 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.235 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aacbaea5:::benchmark_data_smithi171_122772_object103318:head# 2024-04-20T17:27:56.235 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.235 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aacc7ee6:::benchmark_data_smithi171_122772_object117898:head# 2024-04-20T17:27:56.235 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.236 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aacc8d6a:::benchmark_data_smithi171_122772_object86428:head# 2024-04-20T17:27:56.236 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.236 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aacd2c0d:::benchmark_data_smithi171_122772_object153532:head# 2024-04-20T17:27:56.236 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.236 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aacea66f:::benchmark_data_smithi171_122772_object145346:head# 2024-04-20T17:27:56.236 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.236 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aad05fe8:::benchmark_data_smithi171_122772_object102689:head# 2024-04-20T17:27:56.236 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.236 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aad1e6e0:::benchmark_data_smithi171_122772_object150831:head# 2024-04-20T17:27:56.236 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.236 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aad274c1:::benchmark_data_smithi171_122772_object94601:head# 2024-04-20T17:27:56.236 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.236 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aad4ccbd:::benchmark_data_smithi171_122772_object105887:head# 2024-04-20T17:27:56.237 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.237 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aad63683:::benchmark_data_smithi171_122772_object156002:head# 2024-04-20T17:27:56.237 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.237 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aad65aa5:::benchmark_data_smithi171_122772_object143477:head# 2024-04-20T17:27:56.237 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.237 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aad72561:::benchmark_data_smithi171_122772_object106291:head# 2024-04-20T17:27:56.237 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.237 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aad83163:::benchmark_data_smithi171_122772_object126476:head# 2024-04-20T17:27:56.237 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.237 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aad93df9:::benchmark_data_smithi171_122772_object114726:head# 2024-04-20T17:27:56.237 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.237 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aada887c:::benchmark_data_smithi171_122772_object82588:head# 2024-04-20T17:27:56.237 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.238 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aadbb37a:::benchmark_data_smithi171_122772_object154994:head# 2024-04-20T17:27:56.238 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.238 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aadbdb15:::benchmark_data_smithi171_122772_object81047:head# 2024-04-20T17:27:56.238 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.238 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aadbfee0:::benchmark_data_smithi171_122772_object136495:head# 2024-04-20T17:27:56.238 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.238 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aadf7452:::benchmark_data_smithi171_122772_object109035:head# 2024-04-20T17:27:56.238 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.238 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aadfa849:::benchmark_data_smithi171_122772_object119892:head# 2024-04-20T17:27:56.238 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.238 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aadfbca6:::benchmark_data_smithi171_122772_object137869:head# 2024-04-20T17:27:56.238 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.239 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aadfc777:::benchmark_data_smithi171_122772_object84086:head# 2024-04-20T17:27:56.239 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.239 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aae23072:::benchmark_data_smithi171_122772_object133021:head# 2024-04-20T17:27:56.239 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.239 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aae47d88:::benchmark_data_smithi171_122772_object147638:head# 2024-04-20T17:27:56.239 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.239 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aae4e26c:::benchmark_data_smithi171_122772_object88928:head# 2024-04-20T17:27:56.239 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.239 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aae58ba7:::benchmark_data_smithi171_122772_object92618:head# 2024-04-20T17:27:56.239 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.239 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aae68330:::benchmark_data_smithi171_122772_object96022:head# 2024-04-20T17:27:56.239 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.239 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aae6cc04:::benchmark_data_smithi171_122772_object114217:head# 2024-04-20T17:27:56.240 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.240 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aae6dca9:::benchmark_data_smithi171_122772_object100163:head# 2024-04-20T17:27:56.240 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.240 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aae75ea9:::benchmark_data_smithi171_122772_object107575:head# 2024-04-20T17:27:56.240 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.240 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aae7e103:::benchmark_data_smithi171_122772_object165152:head# 2024-04-20T17:27:56.240 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.240 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aae80fc2:::benchmark_data_smithi171_122772_object127772:head# 2024-04-20T17:27:56.240 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.240 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaeaf6fd:::benchmark_data_smithi171_122772_object78520:head# 2024-04-20T17:27:56.240 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.240 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaeb06f4:::benchmark_data_smithi171_122772_object87705:head# 2024-04-20T17:27:56.240 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.241 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaec3bff:::benchmark_data_smithi171_122772_object91041:head# 2024-04-20T17:27:56.241 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.241 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaec407c:::benchmark_data_smithi171_122772_object147802:head# 2024-04-20T17:27:56.241 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.241 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaed906a:::benchmark_data_smithi171_122772_object127803:head# 2024-04-20T17:27:56.241 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.241 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaee093b:::benchmark_data_smithi171_122772_object141006:head# 2024-04-20T17:27:56.241 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.241 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaeea48b:::benchmark_data_smithi171_122772_object128745:head# 2024-04-20T17:27:56.241 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.241 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaeedaaf:::benchmark_data_smithi171_122772_object79212:head# 2024-04-20T17:27:56.241 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.242 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaeeeacd:::benchmark_data_smithi171_122772_object139485:head# 2024-04-20T17:27:56.242 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.242 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaef0e06:::benchmark_data_smithi171_122772_object169418:head# 2024-04-20T17:27:56.242 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.242 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaef293a:::benchmark_data_smithi171_122772_object143877:head# 2024-04-20T17:27:56.242 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.242 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaf057bc:::benchmark_data_smithi171_122772_object104874:head# 2024-04-20T17:27:56.242 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.242 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaf08bbb:::benchmark_data_smithi171_122772_object115004:head# 2024-04-20T17:27:56.242 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.242 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaf2dd1a:::benchmark_data_smithi171_122772_object80950:head# 2024-04-20T17:27:56.242 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.243 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaf32931:::benchmark_data_smithi171_122772_object97460:head# 2024-04-20T17:27:56.243 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.243 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaf4b87c:::benchmark_data_smithi171_122772_object114625:head# 2024-04-20T17:27:56.243 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.243 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaf653e9:::benchmark_data_smithi171_122772_object115887:head# 2024-04-20T17:27:56.243 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.243 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaf6b91e:::benchmark_data_smithi171_122772_object78900:head# 2024-04-20T17:27:56.243 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.243 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaf713fa:::benchmark_data_smithi171_122772_object112921:head# 2024-04-20T17:27:56.243 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.243 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaf739ec:::benchmark_data_smithi171_122772_object165636:head# 2024-04-20T17:27:56.243 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.243 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaf7664e:::benchmark_data_smithi171_122772_object130127:head# 2024-04-20T17:27:56.244 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.244 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aafa9da8:::benchmark_data_smithi171_122772_object168639:head# 2024-04-20T17:27:56.244 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.244 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aafacec7:::benchmark_data_smithi171_122772_object145529:head# 2024-04-20T17:27:56.244 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.244 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aafb2dad:::benchmark_data_smithi171_122772_object110900:head# 2024-04-20T17:27:56.244 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.244 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aafb7147:::benchmark_data_smithi171_122772_object80459:head# 2024-04-20T17:27:56.244 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.244 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aafc57d7:::benchmark_data_smithi171_122772_object97547:head# 2024-04-20T17:27:56.244 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.244 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aafce4d3:::benchmark_data_smithi171_122772_object168116:head# 2024-04-20T17:27:56.245 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.245 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aafdee72:::benchmark_data_smithi171_122772_object159983:head# 2024-04-20T17:27:56.245 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.245 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aafe089c:::benchmark_data_smithi171_122772_object113535:head# 2024-04-20T17:27:56.245 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.245 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aafe562a:::benchmark_data_smithi171_122772_object156920:head# 2024-04-20T17:27:56.245 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.245 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aaffa8a9:::benchmark_data_smithi171_122772_object87006:head# 2024-04-20T17:27:56.245 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.245 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab02176e:::benchmark_data_smithi171_122772_object162453:head# 2024-04-20T17:27:56.245 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.245 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab02280d:::benchmark_data_smithi171_122772_object145586:head# 2024-04-20T17:27:56.246 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.246 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab022998:::benchmark_data_smithi171_122772_object152941:head# 2024-04-20T17:27:56.246 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.246 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab025765:::benchmark_data_smithi171_122772_object157239:head# 2024-04-20T17:27:56.246 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.246 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab03f332:::benchmark_data_smithi171_122772_object88088:head# 2024-04-20T17:27:56.246 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.246 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab04c63c:::benchmark_data_smithi171_122772_object162884:head# 2024-04-20T17:27:56.246 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.246 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab0551d7:::benchmark_data_smithi171_122772_object91532:head# 2024-04-20T17:27:56.246 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.247 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab06021a:::benchmark_data_smithi171_122772_object82042:head# 2024-04-20T17:27:56.247 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.247 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab06eaad:::benchmark_data_smithi171_122772_object138043:head# 2024-04-20T17:27:56.247 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.247 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab0798ba:::benchmark_data_smithi171_122772_object80137:head# 2024-04-20T17:27:56.247 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.247 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab07ed4c:::benchmark_data_smithi171_122772_object111748:head# 2024-04-20T17:27:56.247 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.247 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab08912e:::benchmark_data_smithi171_122772_object153035:head# 2024-04-20T17:27:56.247 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.247 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab092b5c:::benchmark_data_smithi171_122772_object164783:head# 2024-04-20T17:27:56.247 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.248 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab0957c4:::benchmark_data_smithi171_122772_object144928:head# 2024-04-20T17:27:56.248 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.248 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab09c0e9:::benchmark_data_smithi171_122772_object115401:head# 2024-04-20T17:27:56.248 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.248 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab09c3da:::benchmark_data_smithi171_122772_object84221:head# 2024-04-20T17:27:56.248 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.248 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab0bcf6a:::benchmark_data_smithi171_122772_object150216:head# 2024-04-20T17:27:56.248 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.248 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab0cb644:::benchmark_data_smithi171_122772_object157691:head# 2024-04-20T17:27:56.248 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.248 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab0cccaf:::benchmark_data_smithi171_122772_object78545:head# 2024-04-20T17:27:56.249 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.249 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab0ce798:::benchmark_data_smithi171_122772_object162526:head# 2024-04-20T17:27:56.249 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.249 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab0e0097:::benchmark_data_smithi171_122772_object104498:head# 2024-04-20T17:27:56.249 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.249 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab0e300f:::benchmark_data_smithi171_122772_object164180:head# 2024-04-20T17:27:56.249 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.249 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab0ff561:::benchmark_data_smithi171_122772_object128680:head# 2024-04-20T17:27:56.249 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.249 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab103b75:::benchmark_data_smithi171_122772_object114989:head# 2024-04-20T17:27:56.249 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.249 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab118c11:::benchmark_data_smithi171_122772_object119732:head# 2024-04-20T17:27:56.250 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.250 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab122729:::benchmark_data_smithi171_122772_object121199:head# 2024-04-20T17:27:56.250 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.250 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab129153:::benchmark_data_smithi171_122772_object137289:head# 2024-04-20T17:27:56.250 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.250 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab1316ad:::benchmark_data_smithi171_122772_object96734:head# 2024-04-20T17:27:56.250 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.250 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab131df9:::benchmark_data_smithi171_122772_object109086:head# 2024-04-20T17:27:56.250 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.250 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab141ed0:::benchmark_data_smithi171_122772_object83555:head# 2024-04-20T17:27:56.250 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.251 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab1521a6:::benchmark_data_smithi171_122772_object127806:head# 2024-04-20T17:27:56.251 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.251 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab1589ff:::benchmark_data_smithi171_122772_object84607:head# 2024-04-20T17:27:56.251 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.251 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab1675d7:::benchmark_data_smithi171_122772_object146125:head# 2024-04-20T17:27:56.251 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.251 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab1677b6:::benchmark_data_smithi171_122772_object115456:head# 2024-04-20T17:27:56.251 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.251 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab168667:::benchmark_data_smithi171_122772_object78453:head# 2024-04-20T17:27:56.251 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.251 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab1754d6:::benchmark_data_smithi171_122772_object157392:head# 2024-04-20T17:27:56.251 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.252 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab177e90:::benchmark_data_smithi171_122772_object82138:head# 2024-04-20T17:27:56.252 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.252 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab1982d3:::benchmark_data_smithi171_122772_object86274:head# 2024-04-20T17:27:56.252 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.252 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab1a6067:::benchmark_data_smithi171_122772_object89114:head# 2024-04-20T17:27:56.252 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.252 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab1b90b6:::benchmark_data_smithi171_122772_object156815:head# 2024-04-20T17:27:56.252 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.252 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab1bbf13:::benchmark_data_smithi171_122772_object137648:head# 2024-04-20T17:27:56.252 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.252 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab1c727c:::benchmark_data_smithi171_122772_object90055:head# 2024-04-20T17:27:56.252 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.253 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab1cae04:::benchmark_data_smithi171_122772_object124747:head# 2024-04-20T17:27:56.253 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.253 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab1cdaf0:::benchmark_data_smithi171_122772_object82936:head# 2024-04-20T17:27:56.253 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.253 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab1d8c98:::benchmark_data_smithi171_122772_object150684:head# 2024-04-20T17:27:56.253 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.253 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab1e40b5:::benchmark_data_smithi171_122772_object84930:head# 2024-04-20T17:27:56.253 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.253 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab1e6ca8:::benchmark_data_smithi171_122772_object151919:head# 2024-04-20T17:27:56.253 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.253 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab1f386c:::benchmark_data_smithi171_122772_object110936:head# 2024-04-20T17:27:56.253 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.253 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab1f3ee2:::benchmark_data_smithi171_122772_object154038:head# 2024-04-20T17:27:56.254 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.254 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab1feb25:::benchmark_data_smithi171_122772_object147286:head# 2024-04-20T17:27:56.254 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.254 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab200495:::benchmark_data_smithi171_122772_object103547:head# 2024-04-20T17:27:56.254 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.254 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab200ad0:::benchmark_data_smithi171_122772_object144674:head# 2024-04-20T17:27:56.254 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.254 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab201909:::benchmark_data_smithi171_122772_object154289:head# 2024-04-20T17:27:56.254 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.254 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab2048af:::benchmark_data_smithi171_122772_object143917:head# 2024-04-20T17:27:56.254 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.255 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab211959:::benchmark_data_smithi171_122772_object85326:head# 2024-04-20T17:27:56.255 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.255 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab219c44:::benchmark_data_smithi171_122772_object133274:head# 2024-04-20T17:27:56.255 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.255 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab23eb49:::benchmark_data_smithi171_122772_object150215:head# 2024-04-20T17:27:56.255 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.255 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab247747:::benchmark_data_smithi171_122772_object89618:head# 2024-04-20T17:27:56.255 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.255 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab25945b:::benchmark_data_smithi171_122772_object165443:head# 2024-04-20T17:27:56.255 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.255 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab26a234:::benchmark_data_smithi171_122772_object169221:head# 2024-04-20T17:27:56.255 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.256 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab2796c3:::benchmark_data_smithi171_122772_object107942:head# 2024-04-20T17:27:56.256 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.256 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab27e33c:::benchmark_data_smithi171_122772_object108449:head# 2024-04-20T17:27:56.256 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.256 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab283dca:::benchmark_data_smithi171_122772_object162968:head# 2024-04-20T17:27:56.256 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.256 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab28be66:::benchmark_data_smithi171_122772_object115162:head# 2024-04-20T17:27:56.256 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.256 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab291ce5:::benchmark_data_smithi171_122772_object137806:head# 2024-04-20T17:27:56.256 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.256 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab2944fd:::benchmark_data_smithi171_122772_object93514:head# 2024-04-20T17:27:56.256 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.257 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab299209:::benchmark_data_smithi171_122772_object120853:head# 2024-04-20T17:27:56.257 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.257 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab2a107b:::benchmark_data_smithi171_122772_object154755:head# 2024-04-20T17:27:56.257 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.257 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab2b25ae:::benchmark_data_smithi171_122772_object111614:head# 2024-04-20T17:27:56.257 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.257 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab2c92f0:::benchmark_data_smithi171_122772_object142372:head# 2024-04-20T17:27:56.257 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.257 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab2cb07d:::benchmark_data_smithi171_122772_object145015:head# 2024-04-20T17:27:56.257 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.257 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab2d0750:::benchmark_data_smithi171_122772_object134850:head# 2024-04-20T17:27:56.257 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.257 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab2d2d89:::benchmark_data_smithi171_122772_object86743:head# 2024-04-20T17:27:56.258 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.258 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab2d3d3a:::benchmark_data_smithi171_122772_object161514:head# 2024-04-20T17:27:56.258 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.258 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab2ee2cf:::benchmark_data_smithi171_122772_object92620:head# 2024-04-20T17:27:56.258 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.258 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab2f6919:::benchmark_data_smithi171_122772_object119020:head# 2024-04-20T17:27:56.258 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.258 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab2fa544:::benchmark_data_smithi171_122772_object123866:head# 2024-04-20T17:27:56.258 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.258 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab3003cb:::benchmark_data_smithi171_122772_object141824:head# 2024-04-20T17:27:56.258 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.258 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab301a9f:::benchmark_data_smithi171_122772_object167895:head# 2024-04-20T17:27:56.259 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.259 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab30805e:::benchmark_data_smithi171_122772_object157958:head# 2024-04-20T17:27:56.259 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.259 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab319a6a:::benchmark_data_smithi171_122772_object144104:head# 2024-04-20T17:27:56.259 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.259 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab339955:::benchmark_data_smithi171_122772_object156543:head# 2024-04-20T17:27:56.259 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.259 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab33bfd8:::benchmark_data_smithi171_122772_object169189:head# 2024-04-20T17:27:56.259 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.259 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab33e17e:::benchmark_data_smithi171_122772_object153247:head# 2024-04-20T17:27:56.259 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.259 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab34d45c:::benchmark_data_smithi171_122772_object129840:head# 2024-04-20T17:27:56.259 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.260 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab34eee2:::benchmark_data_smithi171_122772_object150113:head# 2024-04-20T17:27:56.260 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.260 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab35e8c4:::benchmark_data_smithi171_122772_object91973:head# 2024-04-20T17:27:56.260 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.260 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab388e1b:::benchmark_data_smithi171_122772_object148678:head# 2024-04-20T17:27:56.260 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.260 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab38d6ea:::benchmark_data_smithi171_122772_object88490:head# 2024-04-20T17:27:56.260 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.260 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab399dda:::benchmark_data_smithi171_122772_object150635:head# 2024-04-20T17:27:56.260 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.260 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab39d313:::benchmark_data_smithi171_122772_object165131:head# 2024-04-20T17:27:56.260 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.261 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab39d7fd:::benchmark_data_smithi171_122772_object78569:head# 2024-04-20T17:27:56.261 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.261 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab3a3639:::benchmark_data_smithi171_122772_object167058:head# 2024-04-20T17:27:56.261 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.261 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab3ad0a2:::benchmark_data_smithi171_122772_object85504:head# 2024-04-20T17:27:56.261 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.261 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab3b297a:::benchmark_data_smithi171_122772_object131160:head# 2024-04-20T17:27:56.261 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.261 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab3b5ad0:::benchmark_data_smithi171_122772_object80694:head# 2024-04-20T17:27:56.261 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.261 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab3c6b92:::benchmark_data_smithi171_122772_object165882:head# 2024-04-20T17:27:56.261 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.261 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab3d1fc8:::benchmark_data_smithi171_122772_object139660:head# 2024-04-20T17:27:56.262 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.262 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab3d21a8:::benchmark_data_smithi171_122772_object104841:head# 2024-04-20T17:27:56.262 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.262 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab3d8077:::benchmark_data_smithi171_122772_object128262:head# 2024-04-20T17:27:56.262 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.262 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab3d8b2f:::benchmark_data_smithi171_122772_object114505:head# 2024-04-20T17:27:56.262 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.262 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab3dd402:::benchmark_data_smithi171_122772_object131995:head# 2024-04-20T17:27:56.262 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.262 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab3e8d8f:::benchmark_data_smithi171_122772_object146553:head# 2024-04-20T17:27:56.262 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.262 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab3e9987:::benchmark_data_smithi171_122772_object113838:head# 2024-04-20T17:27:56.263 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.263 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab3f1ca9:::benchmark_data_smithi171_122772_object111699:head# 2024-04-20T17:27:56.263 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.263 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab3ffa71:::benchmark_data_smithi171_122772_object92217:head# 2024-04-20T17:27:56.263 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.263 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab40c9b7:::benchmark_data_smithi171_122772_object161267:head# 2024-04-20T17:27:56.263 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.263 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab42c2ff:::benchmark_data_smithi171_122772_object85913:head# 2024-04-20T17:27:56.263 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.263 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab4323ef:::benchmark_data_smithi171_122772_object122402:head# 2024-04-20T17:27:56.263 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.263 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab43c226:::benchmark_data_smithi171_122772_object138590:head# 2024-04-20T17:27:56.264 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.264 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab444ca5:::benchmark_data_smithi171_122772_object94103:head# 2024-04-20T17:27:56.264 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.264 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab44bc0d:::benchmark_data_smithi171_122772_object78267:head# 2024-04-20T17:27:56.264 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.264 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab456b7d:::benchmark_data_smithi171_122772_object152719:head# 2024-04-20T17:27:56.264 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.264 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab45d6c0:::benchmark_data_smithi171_122772_object118604:head# 2024-04-20T17:27:56.264 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.264 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab46191b:::benchmark_data_smithi171_122772_object83761:head# 2024-04-20T17:27:56.264 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.264 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab4626b3:::benchmark_data_smithi171_122772_object140789:head# 2024-04-20T17:27:56.264 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.265 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab46c83b:::benchmark_data_smithi171_122772_object143916:head# 2024-04-20T17:27:56.265 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.265 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab475e2b:::benchmark_data_smithi171_122772_object89146:head# 2024-04-20T17:27:56.265 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.265 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab48a4b2:::benchmark_data_smithi171_122772_object150908:head# 2024-04-20T17:27:56.265 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.265 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab49a53e:::benchmark_data_smithi171_122772_object121441:head# 2024-04-20T17:27:56.265 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.265 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab49e7e8:::benchmark_data_smithi171_122772_object134253:head# 2024-04-20T17:27:56.265 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.265 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab4ad733:::benchmark_data_smithi171_122772_object131828:head# 2024-04-20T17:27:56.265 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.265 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab4c27bd:::benchmark_data_smithi171_122772_object139579:head# 2024-04-20T17:27:56.266 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.266 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab4c8593:::benchmark_data_smithi171_122772_object144648:head# 2024-04-20T17:27:56.266 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.266 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab4d306c:::benchmark_data_smithi171_122772_object164224:head# 2024-04-20T17:27:56.266 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.266 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab4eb77f:::benchmark_data_smithi171_122772_object81599:head# 2024-04-20T17:27:56.266 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.266 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab4ebe61:::benchmark_data_smithi171_122772_object149278:head# 2024-04-20T17:27:56.266 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.266 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab4fb1a7:::benchmark_data_smithi171_122772_object107555:head# 2024-04-20T17:27:56.266 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.266 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab4ff5c1:::benchmark_data_smithi171_122772_object151599:head# 2024-04-20T17:27:56.267 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.267 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab50a30b:::benchmark_data_smithi171_122772_object165748:head# 2024-04-20T17:27:56.267 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.267 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab51990d:::benchmark_data_smithi171_122772_object114872:head# 2024-04-20T17:27:56.267 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.267 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab532b68:::benchmark_data_smithi171_122772_object154890:head# 2024-04-20T17:27:56.267 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.267 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab5391bb:::benchmark_data_smithi171_122772_object157836:head# 2024-04-20T17:27:56.267 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.267 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab542eb8:::benchmark_data_smithi171_122772_object101547:head# 2024-04-20T17:27:56.267 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.267 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab551bd8:::benchmark_data_smithi171_122772_object157604:head# 2024-04-20T17:27:56.267 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.268 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab552fc1:::benchmark_data_smithi171_122772_object83106:head# 2024-04-20T17:27:56.268 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.268 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab56bb50:::benchmark_data_smithi171_122772_object115132:head# 2024-04-20T17:27:56.268 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.268 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab578c86:::benchmark_data_smithi171_122772_object86363:head# 2024-04-20T17:27:56.268 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.268 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab58087a:::benchmark_data_smithi171_122772_object159951:head# 2024-04-20T17:27:56.268 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.268 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab597dea:::benchmark_data_smithi171_122772_object80360:head# 2024-04-20T17:27:56.268 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.268 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab5b79f4:::benchmark_data_smithi171_122772_object93957:head# 2024-04-20T17:27:56.268 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.268 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab5bbaf2:::benchmark_data_smithi171_122772_object141216:head# 2024-04-20T17:27:56.269 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.269 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab5c7da6:::benchmark_data_smithi171_122772_object117859:head# 2024-04-20T17:27:56.269 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.269 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab5d5eb4:::benchmark_data_smithi171_122772_object156547:head# 2024-04-20T17:27:56.269 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.269 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab5ece38:::benchmark_data_smithi171_122772_object169798:head# 2024-04-20T17:27:56.269 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.269 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab5f973d:::benchmark_data_smithi171_122772_object123989:head# 2024-04-20T17:27:56.269 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.269 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab5fd8a2:::benchmark_data_smithi171_122772_object84685:head# 2024-04-20T17:27:56.269 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.269 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab604977:::benchmark_data_smithi171_122772_object104629:head# 2024-04-20T17:27:56.270 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.270 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab60ab0f:::benchmark_data_smithi171_122772_object155634:head# 2024-04-20T17:27:56.270 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.270 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab60eb99:::benchmark_data_smithi171_122772_object90973:head# 2024-04-20T17:27:56.270 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.270 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab60f172:::benchmark_data_smithi171_122772_object121327:head# 2024-04-20T17:27:56.270 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.270 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab64114e:::benchmark_data_smithi171_122772_object132863:head# 2024-04-20T17:27:56.270 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.270 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab658ff3:::benchmark_data_smithi171_122772_object115365:head# 2024-04-20T17:27:56.270 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.270 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab65b128:::benchmark_data_smithi171_122772_object160450:head# 2024-04-20T17:27:56.271 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.271 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab65f948:::benchmark_data_smithi171_122772_object164348:head# 2024-04-20T17:27:56.271 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.271 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab662160:::benchmark_data_smithi171_122772_object147626:head# 2024-04-20T17:27:56.271 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.271 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab67c016:::benchmark_data_smithi171_122772_object104139:head# 2024-04-20T17:27:56.271 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.271 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab680a27:::benchmark_data_smithi171_122772_object126529:head# 2024-04-20T17:27:56.271 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.271 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab69c625:::benchmark_data_smithi171_122772_object95813:head# 2024-04-20T17:27:56.271 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.271 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab69ef77:::benchmark_data_smithi171_122772_object160058:head# 2024-04-20T17:27:56.271 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.272 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab6a511b:::benchmark_data_smithi171_122772_object102856:head# 2024-04-20T17:27:56.272 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.272 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab6da968:::benchmark_data_smithi171_122772_object105747:head# 2024-04-20T17:27:56.272 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.272 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab6ea764:::benchmark_data_smithi171_122772_object83561:head# 2024-04-20T17:27:56.272 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.272 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab6ec394:::benchmark_data_smithi171_122772_object81353:head# 2024-04-20T17:27:56.272 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.272 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab707d89:::benchmark_data_smithi171_122772_object94254:head# 2024-04-20T17:27:56.272 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.272 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab70c90e:::benchmark_data_smithi171_122772_object151736:head# 2024-04-20T17:27:56.272 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.272 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab7135e1:::benchmark_data_smithi171_122772_object141508:head# 2024-04-20T17:27:56.273 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.273 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab716656:::benchmark_data_smithi171_122772_object159948:head# 2024-04-20T17:27:56.273 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.273 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab72f472:::benchmark_data_smithi171_122772_object165352:head# 2024-04-20T17:27:56.273 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.273 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab72f94f:::benchmark_data_smithi171_122772_object85650:head# 2024-04-20T17:27:56.273 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.273 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab742930:::benchmark_data_smithi171_122772_object111792:head# 2024-04-20T17:27:56.273 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.273 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab7468eb:::benchmark_data_smithi171_122772_object154726:head# 2024-04-20T17:27:56.273 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.273 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab74ce04:::benchmark_data_smithi171_122772_object82964:head# 2024-04-20T17:27:56.273 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.274 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab74e91f:::benchmark_data_smithi171_122772_object126622:head# 2024-04-20T17:27:56.274 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.274 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab757c0d:::benchmark_data_smithi171_122772_object168034:head# 2024-04-20T17:27:56.274 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.274 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab75a77c:::benchmark_data_smithi171_122772_object126174:head# 2024-04-20T17:27:56.274 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.274 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab7720d1:::benchmark_data_smithi171_122772_object145314:head# 2024-04-20T17:27:56.274 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.274 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab7794a7:::benchmark_data_smithi171_122772_object100632:head# 2024-04-20T17:27:56.274 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.274 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab78ba52:::benchmark_data_smithi171_122772_object128516:head# 2024-04-20T17:27:56.275 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.275 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab7902c3:::benchmark_data_smithi171_122772_object123651:head# 2024-04-20T17:27:56.275 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.275 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab796078:::benchmark_data_smithi171_122772_object100485:head# 2024-04-20T17:27:56.275 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.275 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab79bebd:::benchmark_data_smithi171_122772_object104305:head# 2024-04-20T17:27:56.275 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.275 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab7bd121:::benchmark_data_smithi171_122772_object85969:head# 2024-04-20T17:27:56.275 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.275 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab7c718f:::benchmark_data_smithi171_122772_object150415:head# 2024-04-20T17:27:56.275 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.275 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab7c949a:::benchmark_data_smithi171_122772_object162171:head# 2024-04-20T17:27:56.276 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.276 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab7d65bd:::benchmark_data_smithi171_122772_object88008:head# 2024-04-20T17:27:56.276 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.276 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab7e204d:::benchmark_data_smithi171_122772_object124895:head# 2024-04-20T17:27:56.276 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.276 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab7e4938:::benchmark_data_smithi171_122772_object80521:head# 2024-04-20T17:27:56.276 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.276 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab7ec6ad:::benchmark_data_smithi171_122772_object162952:head# 2024-04-20T17:27:56.276 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.276 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab7ffd20:::benchmark_data_smithi171_122772_object159337:head# 2024-04-20T17:27:56.276 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.276 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab804652:::benchmark_data_smithi171_122772_object152607:head# 2024-04-20T17:27:56.276 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.277 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab81932d:::benchmark_data_smithi171_122772_object88946:head# 2024-04-20T17:27:56.277 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.277 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab81c85e:::benchmark_data_smithi171_122772_object141214:head# 2024-04-20T17:27:56.277 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.277 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab82c13f:::benchmark_data_smithi171_122772_object161198:head# 2024-04-20T17:27:56.277 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.277 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab82c190:::benchmark_data_smithi171_122772_object102096:head# 2024-04-20T17:27:56.277 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.277 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab82cbe0:::benchmark_data_smithi171_122772_object131340:head# 2024-04-20T17:27:56.277 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.277 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab82f1f3:::benchmark_data_smithi171_122772_object82360:head# 2024-04-20T17:27:56.277 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.278 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab831435:::benchmark_data_smithi171_122772_object86493:head# 2024-04-20T17:27:56.278 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.278 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab850d50:::benchmark_data_smithi171_122772_object111287:head# 2024-04-20T17:27:56.278 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.278 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab850e0e:::benchmark_data_smithi171_122772_object121073:head# 2024-04-20T17:27:56.278 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.278 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab853973:::benchmark_data_smithi171_122772_object86113:head# 2024-04-20T17:27:56.278 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.278 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab856928:::benchmark_data_smithi171_122772_object115439:head# 2024-04-20T17:27:56.278 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.278 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab863f48:::benchmark_data_smithi171_122772_object85010:head# 2024-04-20T17:27:56.278 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.279 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab8672c9:::benchmark_data_smithi171_122772_object110497:head# 2024-04-20T17:27:56.279 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.279 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab88c33a:::benchmark_data_smithi171_122772_object159081:head# 2024-04-20T17:27:56.279 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.279 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab891b77:::benchmark_data_smithi171_122772_object138162:head# 2024-04-20T17:27:56.279 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.279 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab8af5fc:::benchmark_data_smithi171_122772_object142488:head# 2024-04-20T17:27:56.279 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.279 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab8c0a53:::benchmark_data_smithi171_122772_object120592:head# 2024-04-20T17:27:56.279 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.279 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab8c2991:::benchmark_data_smithi171_122772_object91657:head# 2024-04-20T17:27:56.279 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.279 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab8c4e25:::benchmark_data_smithi171_122772_object146350:head# 2024-04-20T17:27:56.280 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.280 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab8c55ba:::benchmark_data_smithi171_122772_object123953:head# 2024-04-20T17:27:56.280 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.280 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab8c6c5b:::benchmark_data_smithi171_122772_object168809:head# 2024-04-20T17:27:56.280 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.280 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab8c7151:::benchmark_data_smithi171_122772_object146211:head# 2024-04-20T17:27:56.280 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.280 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab8da14c:::benchmark_data_smithi171_122772_object85096:head# 2024-04-20T17:27:56.280 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.280 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab8f10c9:::benchmark_data_smithi171_122772_object94415:head# 2024-04-20T17:27:56.280 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.280 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab8fc5c0:::benchmark_data_smithi171_122772_object135428:head# 2024-04-20T17:27:56.281 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.281 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab9260d4:::benchmark_data_smithi171_122772_object141156:head# 2024-04-20T17:27:56.281 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.281 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab92c091:::benchmark_data_smithi171_122772_object88192:head# 2024-04-20T17:27:56.281 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.281 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab934f2c:::benchmark_data_smithi171_122772_object124459:head# 2024-04-20T17:27:56.281 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.281 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab9385f2:::benchmark_data_smithi171_122772_object110413:head# 2024-04-20T17:27:56.281 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.281 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab9485bd:::benchmark_data_smithi171_122772_object162544:head# 2024-04-20T17:27:56.281 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.281 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab9686c4:::benchmark_data_smithi171_122772_object162633:head# 2024-04-20T17:27:56.282 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.282 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab96f775:::benchmark_data_smithi171_122772_object151866:head# 2024-04-20T17:27:56.282 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.282 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab96ff5a:::benchmark_data_smithi171_122772_object138369:head# 2024-04-20T17:27:56.282 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.282 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab972af3:::benchmark_data_smithi171_122772_object82331:head# 2024-04-20T17:27:56.282 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.282 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab9876ba:::benchmark_data_smithi171_122772_object84879:head# 2024-04-20T17:27:56.282 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.282 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab99df3b:::benchmark_data_smithi171_122772_object98796:head# 2024-04-20T17:27:56.282 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.282 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab9ae2d2:::benchmark_data_smithi171_122772_object88615:head# 2024-04-20T17:27:56.283 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.283 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab9af0ac:::benchmark_data_smithi171_122772_object93211:head# 2024-04-20T17:27:56.283 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.283 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab9b0c80:::benchmark_data_smithi171_122772_object135070:head# 2024-04-20T17:27:56.283 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.283 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab9b2718:::benchmark_data_smithi171_122772_object102301:head# 2024-04-20T17:27:56.283 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.283 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab9b6d47:::benchmark_data_smithi171_122772_object140857:head# 2024-04-20T17:27:56.283 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.283 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab9c6f28:::benchmark_data_smithi171_122772_object89380:head# 2024-04-20T17:27:56.283 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.283 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab9c72e7:::benchmark_data_smithi171_122772_object94155:head# 2024-04-20T17:27:56.283 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.284 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab9cc13d:::benchmark_data_smithi171_122772_object104196:head# 2024-04-20T17:27:56.284 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.284 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab9d2806:::benchmark_data_smithi171_122772_object143008:head# 2024-04-20T17:27:56.284 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.284 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab9d53f1:::benchmark_data_smithi171_122772_object134175:head# 2024-04-20T17:27:56.284 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.284 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab9e0a3c:::benchmark_data_smithi171_122772_object106716:head# 2024-04-20T17:27:56.284 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.284 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab9ee9b0:::benchmark_data_smithi171_122772_object136981:head# 2024-04-20T17:27:56.284 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.284 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab9f031f:::benchmark_data_smithi171_122772_object158986:head# 2024-04-20T17:27:56.284 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.284 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab9f53d6:::benchmark_data_smithi171_122772_object131142:head# 2024-04-20T17:27:56.285 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.285 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ab9f70eb:::benchmark_data_smithi171_122772_object89926:head# 2024-04-20T17:27:56.285 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.285 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aba17c2c:::benchmark_data_smithi171_122772_object154088:head# 2024-04-20T17:27:56.285 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.285 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aba1c4d4:::benchmark_data_smithi171_122772_object104223:head# 2024-04-20T17:27:56.285 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.285 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aba1e928:::benchmark_data_smithi171_122772_object113538:head# 2024-04-20T17:27:56.285 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.285 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aba43695:::benchmark_data_smithi171_122772_object144229:head# 2024-04-20T17:27:56.285 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.285 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aba479f4:::benchmark_data_smithi171_122772_object138993:head# 2024-04-20T17:27:56.286 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.286 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aba4f597:::benchmark_data_smithi171_122772_object139541:head# 2024-04-20T17:27:56.286 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.286 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aba5cbed:::benchmark_data_smithi171_122772_object113828:head# 2024-04-20T17:27:56.286 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.286 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aba6d127:::benchmark_data_smithi171_122772_object156684:head# 2024-04-20T17:27:56.286 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.286 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aba7622d:::benchmark_data_smithi171_122772_object79178:head# 2024-04-20T17:27:56.286 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.286 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aba801af:::benchmark_data_smithi171_122772_object116268:head# 2024-04-20T17:27:56.286 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.286 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aba8151a:::benchmark_data_smithi171_122772_object99913:head# 2024-04-20T17:27:56.287 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.287 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abaa3b2a:::benchmark_data_smithi171_122772_object154872:head# 2024-04-20T17:27:56.287 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.287 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abaab8ff:::benchmark_data_smithi171_122772_object107953:head# 2024-04-20T17:27:56.287 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.287 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abab9221:::benchmark_data_smithi171_122772_object97357:head# 2024-04-20T17:27:56.287 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.287 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abac5f69:::benchmark_data_smithi171_122772_object110307:head# 2024-04-20T17:27:56.287 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.287 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abacb953:::benchmark_data_smithi171_122772_object122573:head# 2024-04-20T17:27:56.287 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.287 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abacf803:::benchmark_data_smithi171_122772_object99674:head# 2024-04-20T17:27:56.288 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.288 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abadb475:::benchmark_data_smithi171_122772_object169569:head# 2024-04-20T17:27:56.288 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.288 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abaed156:::benchmark_data_smithi171_122772_object115784:head# 2024-04-20T17:27:56.288 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.288 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abb15847:::benchmark_data_smithi171_122772_object122632:head# 2024-04-20T17:27:56.288 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.288 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abb27db5:::benchmark_data_smithi171_122772_object116930:head# 2024-04-20T17:27:56.288 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.288 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abb4b13f:::benchmark_data_smithi171_122772_object106207:head# 2024-04-20T17:27:56.288 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.288 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abb4fc06:::benchmark_data_smithi171_122772_object105140:head# 2024-04-20T17:27:56.288 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.289 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abb535da:::benchmark_data_smithi171_122772_object78687:head# 2024-04-20T17:27:56.289 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.289 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abb79a41:::benchmark_data_smithi171_122772_object169627:head# 2024-04-20T17:27:56.289 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.289 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abb833c3:::benchmark_data_smithi171_122772_object112038:head# 2024-04-20T17:27:56.289 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.289 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abb84882:::benchmark_data_smithi171_122772_object127419:head# 2024-04-20T17:27:56.289 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.289 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abb9ec05:::benchmark_data_smithi171_122772_object142856:head# 2024-04-20T17:27:56.289 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.289 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abba1011:::benchmark_data_smithi171_122772_object144630:head# 2024-04-20T17:27:56.290 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.290 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abbb0b6f:::benchmark_data_smithi171_122772_object134283:head# 2024-04-20T17:27:56.290 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.290 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abbba119:::benchmark_data_smithi171_122772_object145712:head# 2024-04-20T17:27:56.290 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.290 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abbc6488:::benchmark_data_smithi171_122772_object106336:head# 2024-04-20T17:27:56.290 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.290 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abbebf90:::benchmark_data_smithi171_122772_object168046:head# 2024-04-20T17:27:56.290 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.290 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abbee1d1:::benchmark_data_smithi171_122772_object152848:head# 2024-04-20T17:27:56.290 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.291 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abbf6149:::benchmark_data_smithi171_122772_object136908:head# 2024-04-20T17:27:56.291 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.291 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abbfe252:::benchmark_data_smithi171_122772_object106833:head# 2024-04-20T17:27:56.291 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.291 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abc028f9:::benchmark_data_smithi171_122772_object170138:head# 2024-04-20T17:27:56.291 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.291 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abc030c1:::benchmark_data_smithi171_122772_object98738:head# 2024-04-20T17:27:56.291 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.291 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abc0a9fd:::benchmark_data_smithi171_122772_object124582:head# 2024-04-20T17:27:56.291 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.291 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abc0aa29:::benchmark_data_smithi171_122772_object99950:head# 2024-04-20T17:27:56.291 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.292 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abc0d490:::benchmark_data_smithi171_122772_object107441:head# 2024-04-20T17:27:56.292 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.292 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abc1ec32:::benchmark_data_smithi171_122772_object142522:head# 2024-04-20T17:27:56.292 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.292 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abc1f50b:::benchmark_data_smithi171_122772_object99090:head# 2024-04-20T17:27:56.292 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.292 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abc23368:::benchmark_data_smithi171_122772_object157942:head# 2024-04-20T17:27:56.292 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.292 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abc531d1:::benchmark_data_smithi171_122772_object115189:head# 2024-04-20T17:27:56.292 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.292 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abc57b93:::benchmark_data_smithi171_122772_object152823:head# 2024-04-20T17:27:56.292 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.293 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abc5a041:::benchmark_data_smithi171_122772_object158706:head# 2024-04-20T17:27:56.293 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.293 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abc743ce:::benchmark_data_smithi171_122772_object91782:head# 2024-04-20T17:27:56.293 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.293 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abc7961c:::benchmark_data_smithi171_122772_object94370:head# 2024-04-20T17:27:56.293 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.293 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abc7ef81:::benchmark_data_smithi171_122772_object145975:head# 2024-04-20T17:27:56.293 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.293 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abc88493:::benchmark_data_smithi171_122772_object79763:head# 2024-04-20T17:27:56.293 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.293 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abca336d:::benchmark_data_smithi171_122772_object79732:head# 2024-04-20T17:27:56.293 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.294 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abcb9b37:::benchmark_data_smithi171_122772_object152163:head# 2024-04-20T17:27:56.294 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.294 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abcbb150:::benchmark_data_smithi171_122772_object140933:head# 2024-04-20T17:27:56.294 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.294 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abcc815a:::benchmark_data_smithi171_122772_object97182:head# 2024-04-20T17:27:56.294 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.294 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abccf3b7:::benchmark_data_smithi171_122772_object165699:head# 2024-04-20T17:27:56.294 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.294 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abcd5738:::benchmark_data_smithi171_122772_object105215:head# 2024-04-20T17:27:56.294 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.294 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abce7800:::benchmark_data_smithi171_122772_object167018:head# 2024-04-20T17:27:56.294 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.295 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abce7ef9:::benchmark_data_smithi171_122772_object77961:head# 2024-04-20T17:27:56.295 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.295 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abcfe117:::benchmark_data_smithi171_122772_object146703:head# 2024-04-20T17:27:56.295 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.295 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abd1974a:::benchmark_data_smithi171_122772_object163869:head# 2024-04-20T17:27:56.295 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.295 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abd1e7a1:::benchmark_data_smithi171_122772_object158465:head# 2024-04-20T17:27:56.295 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.295 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abd28685:::benchmark_data_smithi171_122772_object156438:head# 2024-04-20T17:27:56.295 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.295 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abd2adfb:::benchmark_data_smithi171_122772_object121495:head# 2024-04-20T17:27:56.295 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.296 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abd3c5d7:::benchmark_data_smithi171_122772_object100889:head# 2024-04-20T17:27:56.296 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.296 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abd43b71:::benchmark_data_smithi171_122772_object160170:head# 2024-04-20T17:27:56.296 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.296 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abd64ffa:::benchmark_data_smithi171_122772_object78877:head# 2024-04-20T17:27:56.296 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.296 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abd66211:::benchmark_data_smithi171_122772_object79926:head# 2024-04-20T17:27:56.296 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.296 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abd66758:::benchmark_data_smithi171_122772_object141872:head# 2024-04-20T17:27:56.296 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.296 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abd6dd79:::benchmark_data_smithi171_122772_object95228:head# 2024-04-20T17:27:56.296 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.296 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abd6ee52:::benchmark_data_smithi171_122772_object117300:head# 2024-04-20T17:27:56.297 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.297 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abd76480:::benchmark_data_smithi171_122772_object111278:head# 2024-04-20T17:27:56.297 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.297 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abd89813:::benchmark_data_smithi171_122772_object136658:head# 2024-04-20T17:27:56.297 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.297 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abd91102:::benchmark_data_smithi171_122772_object145389:head# 2024-04-20T17:27:56.297 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.297 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abd9ca66:::benchmark_data_smithi171_122772_object83858:head# 2024-04-20T17:27:56.297 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.297 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abd9e4b6:::benchmark_data_smithi171_122772_object133654:head# 2024-04-20T17:27:56.297 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.297 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abda15ed:::benchmark_data_smithi171_122772_object153731:head# 2024-04-20T17:27:56.297 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.298 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abda61f5:::benchmark_data_smithi171_122772_object100793:head# 2024-04-20T17:27:56.298 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.298 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abda8df3:::benchmark_data_smithi171_122772_object82738:head# 2024-04-20T17:27:56.298 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.298 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abdbab89:::benchmark_data_smithi171_122772_object137027:head# 2024-04-20T17:27:56.298 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.298 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abdd9c5b:::benchmark_data_smithi171_122772_object126191:head# 2024-04-20T17:27:56.298 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.298 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abdda621:::benchmark_data_smithi171_122772_object133340:head# 2024-04-20T17:27:56.298 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.298 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abe0a0bc:::benchmark_data_smithi171_122772_object136800:head# 2024-04-20T17:27:56.298 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.298 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abe12c89:::benchmark_data_smithi171_122772_object125660:head# 2024-04-20T17:27:56.299 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.299 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abe2496f:::benchmark_data_smithi171_122772_object101129:head# 2024-04-20T17:27:56.299 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.299 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abe2e866:::benchmark_data_smithi171_122772_object166205:head# 2024-04-20T17:27:56.299 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.299 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abe3fa1c:::benchmark_data_smithi171_122772_object115194:head# 2024-04-20T17:27:56.299 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.299 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abe3faaa:::benchmark_data_smithi171_122772_object94197:head# 2024-04-20T17:27:56.299 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.299 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abe44c69:::benchmark_data_smithi171_122772_object114902:head# 2024-04-20T17:27:56.299 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.299 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abe5e38c:::benchmark_data_smithi171_122772_object111971:head# 2024-04-20T17:27:56.300 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.300 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abe71d67:::benchmark_data_smithi171_122772_object150639:head# 2024-04-20T17:27:56.300 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.300 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abe76ff1:::benchmark_data_smithi171_122772_object114340:head# 2024-04-20T17:27:56.300 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.300 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abe7c504:::benchmark_data_smithi171_122772_object110161:head# 2024-04-20T17:27:56.300 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.300 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abe7eb78:::benchmark_data_smithi171_122772_object126911:head# 2024-04-20T17:27:56.300 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.300 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abeb44f1:::benchmark_data_smithi171_122772_object168542:head# 2024-04-20T17:27:56.300 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.300 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abeb4869:::benchmark_data_smithi171_122772_object101954:head# 2024-04-20T17:27:56.300 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.301 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abec8c94:::benchmark_data_smithi171_122772_object91581:head# 2024-04-20T17:27:56.301 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.301 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abece7b3:::benchmark_data_smithi171_122772_object96825:head# 2024-04-20T17:27:56.301 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.301 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abece85f:::benchmark_data_smithi171_122772_object112584:head# 2024-04-20T17:27:56.301 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.301 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abed7d0b:::benchmark_data_smithi171_122772_object102312:head# 2024-04-20T17:27:56.301 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.301 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abedc8cb:::benchmark_data_smithi171_122772_object134887:head# 2024-04-20T17:27:56.301 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.301 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abef1011:::benchmark_data_smithi171_122772_object115758:head# 2024-04-20T17:27:56.302 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.302 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abf01953:::benchmark_data_smithi171_122772_object151993:head# 2024-04-20T17:27:56.302 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.302 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abf139a1:::benchmark_data_smithi171_122772_object157761:head# 2024-04-20T17:27:56.302 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.302 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abf18dae:::benchmark_data_smithi171_122772_object140192:head# 2024-04-20T17:27:56.302 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.302 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abf1cf7e:::benchmark_data_smithi171_122772_object113715:head# 2024-04-20T17:27:56.302 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.302 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abf30fae:::benchmark_data_smithi171_122772_object120188:head# 2024-04-20T17:27:56.302 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.302 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abf403aa:::benchmark_data_smithi171_122772_object119484:head# 2024-04-20T17:27:56.302 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.303 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abf46632:::benchmark_data_smithi171_122772_object154191:head# 2024-04-20T17:27:56.303 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.303 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abf5f0ff:::benchmark_data_smithi171_122772_object82049:head# 2024-04-20T17:27:56.303 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.303 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abf64d21:::benchmark_data_smithi171_122772_object161995:head# 2024-04-20T17:27:56.303 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.303 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abf68ce2:::benchmark_data_smithi171_122772_object111228:head# 2024-04-20T17:27:56.303 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.303 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abf6adbc:::benchmark_data_smithi171_122772_object127587:head# 2024-04-20T17:27:56.303 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.303 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abf89656:::benchmark_data_smithi171_122772_object128940:head# 2024-04-20T17:27:56.303 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.303 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abf89b7c:::benchmark_data_smithi171_122772_object99456:head# 2024-04-20T17:27:56.304 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.304 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abfa5995:::benchmark_data_smithi171_122772_object106068:head# 2024-04-20T17:27:56.304 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.304 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abfbd1cf:::benchmark_data_smithi171_122772_object100400:head# 2024-04-20T17:27:56.304 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.304 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abfcdef0:::benchmark_data_smithi171_122772_object103763:head# 2024-04-20T17:27:56.304 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.304 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abfd1782:::benchmark_data_smithi171_122772_object147828:head# 2024-04-20T17:27:56.304 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.304 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abfd6a5a:::benchmark_data_smithi171_122772_object160514:head# 2024-04-20T17:27:56.304 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.304 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abfd8071:::benchmark_data_smithi171_122772_object150991:head# 2024-04-20T17:27:56.304 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.305 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abfd998b:::benchmark_data_smithi171_122772_object94608:head# 2024-04-20T17:27:56.305 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.305 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abfdeebb:::benchmark_data_smithi171_122772_object121902:head# 2024-04-20T17:27:56.305 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.305 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:abfeb73e:::benchmark_data_smithi171_122772_object103647:head# 2024-04-20T17:27:56.305 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.305 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac0061dc:::benchmark_data_smithi171_122772_object98081:head# 2024-04-20T17:27:56.305 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.305 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac00931c:::benchmark_data_smithi171_122772_object114712:head# 2024-04-20T17:27:56.305 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.305 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac02116c:::benchmark_data_smithi171_122772_object79562:head# 2024-04-20T17:27:56.305 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.306 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac02667a:::benchmark_data_smithi171_122772_object156004:head# 2024-04-20T17:27:56.306 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.306 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac028c7c:::benchmark_data_smithi171_122772_object111706:head# 2024-04-20T17:27:56.306 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.306 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac03e49c:::benchmark_data_smithi171_122772_object163174:head# 2024-04-20T17:27:56.306 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.306 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac03f798:::benchmark_data_smithi171_122772_object146017:head# 2024-04-20T17:27:56.306 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.306 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac045fdd:::benchmark_data_smithi171_122772_object160656:head# 2024-04-20T17:27:56.306 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.306 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac06e514:::benchmark_data_smithi171_122772_object89050:head# 2024-04-20T17:27:56.306 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.307 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac07570e:::benchmark_data_smithi171_122772_object114017:head# 2024-04-20T17:27:56.307 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.307 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac076b88:::benchmark_data_smithi171_122772_object152577:head# 2024-04-20T17:27:56.307 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.307 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac078dd5:::benchmark_data_smithi171_122772_object99682:head# 2024-04-20T17:27:56.307 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.307 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac09f811:::benchmark_data_smithi171_122772_object112704:head# 2024-04-20T17:27:56.307 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.307 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac0a253a:::benchmark_data_smithi171_122772_object113252:head# 2024-04-20T17:27:56.307 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.307 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac0a7593:::benchmark_data_smithi171_122772_object155663:head# 2024-04-20T17:27:56.307 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.308 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac0aa761:::benchmark_data_smithi171_122772_object92422:head# 2024-04-20T17:27:56.308 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.308 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac0aefde:::benchmark_data_smithi171_122772_object154391:head# 2024-04-20T17:27:56.308 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.308 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac0d867d:::benchmark_data_smithi171_122772_object78180:head# 2024-04-20T17:27:56.308 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.308 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac0feff6:::benchmark_data_smithi171_122772_object93100:head# 2024-04-20T17:27:56.308 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.308 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac1061fa:::benchmark_data_smithi171_122772_object122629:head# 2024-04-20T17:27:56.308 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.308 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac10daed:::benchmark_data_smithi171_122772_object150887:head# 2024-04-20T17:27:56.308 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.308 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac113583:::benchmark_data_smithi171_122772_object115205:head# 2024-04-20T17:27:56.309 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.309 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac12ae96:::benchmark_data_smithi171_122772_object124141:head# 2024-04-20T17:27:56.309 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.309 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac13d459:::benchmark_data_smithi171_122772_object88474:head# 2024-04-20T17:27:56.309 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.309 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac13dfa2:::benchmark_data_smithi171_122772_object169693:head# 2024-04-20T17:27:56.309 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.309 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac14a74a:::benchmark_data_smithi171_122772_object148004:head# 2024-04-20T17:27:56.309 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.309 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac150690:::benchmark_data_smithi171_122772_object120125:head# 2024-04-20T17:27:56.309 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.309 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac15b05b:::benchmark_data_smithi171_122772_object102141:head# 2024-04-20T17:27:56.309 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.310 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac1810bb:::benchmark_data_smithi171_122772_object91525:head# 2024-04-20T17:27:56.310 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.310 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac189579:::benchmark_data_smithi171_122772_object149575:head# 2024-04-20T17:27:56.310 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.310 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac18cf3a:::benchmark_data_smithi171_122772_object121978:head# 2024-04-20T17:27:56.310 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.310 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac1935ef:::benchmark_data_smithi171_122772_object111492:head# 2024-04-20T17:27:56.310 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.310 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac1963e4:::benchmark_data_smithi171_122772_object95275:head# 2024-04-20T17:27:56.310 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.310 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac19c2c3:::benchmark_data_smithi171_122772_object144187:head# 2024-04-20T17:27:56.310 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.311 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac1cb89d:::benchmark_data_smithi171_122772_object111466:head# 2024-04-20T17:27:56.311 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.311 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac1cd6cd:::benchmark_data_smithi171_122772_object104361:head# 2024-04-20T17:27:56.311 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.311 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac1d200d:::benchmark_data_smithi171_122772_object137462:head# 2024-04-20T17:27:56.311 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.311 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac1d2a92:::benchmark_data_smithi171_122772_object82270:head# 2024-04-20T17:27:56.311 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.311 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac1d2dd2:::benchmark_data_smithi171_122772_object83928:head# 2024-04-20T17:27:56.311 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.311 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac1e047c:::benchmark_data_smithi171_122772_object150147:head# 2024-04-20T17:27:56.311 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.312 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac1e6f29:::benchmark_data_smithi171_122772_object159020:head# 2024-04-20T17:27:56.312 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.312 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac1ea0ed:::benchmark_data_smithi171_122772_object144288:head# 2024-04-20T17:27:56.312 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.312 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac1f68fe:::benchmark_data_smithi171_122772_object113069:head# 2024-04-20T17:27:56.312 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.312 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac1f6901:::benchmark_data_smithi171_122772_object94328:head# 2024-04-20T17:27:56.312 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.312 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac1f9256:::benchmark_data_smithi171_122772_object131338:head# 2024-04-20T17:27:56.312 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.312 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac20718e:::benchmark_data_smithi171_122772_object167644:head# 2024-04-20T17:27:56.313 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.313 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac2086d9:::benchmark_data_smithi171_122772_object131843:head# 2024-04-20T17:27:56.313 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.313 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac2183e6:::benchmark_data_smithi171_122772_object81293:head# 2024-04-20T17:27:56.313 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.313 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac21c144:::benchmark_data_smithi171_122772_object117288:head# 2024-04-20T17:27:56.313 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.313 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac2353f9:::benchmark_data_smithi171_122772_object87922:head# 2024-04-20T17:27:56.313 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.313 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac24166f:::benchmark_data_smithi171_122772_object112308:head# 2024-04-20T17:27:56.313 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.313 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac244437:::benchmark_data_smithi171_122772_object95115:head# 2024-04-20T17:27:56.313 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.314 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac24b47d:::benchmark_data_smithi171_122772_object99685:head# 2024-04-20T17:27:56.314 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.314 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac24f001:::benchmark_data_smithi171_122772_object131276:head# 2024-04-20T17:27:56.314 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.314 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac2505f4:::benchmark_data_smithi171_122772_object99331:head# 2024-04-20T17:27:56.314 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.314 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac26921e:::benchmark_data_smithi171_122772_object152159:head# 2024-04-20T17:27:56.314 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.314 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac280429:::benchmark_data_smithi171_122772_object154488:head# 2024-04-20T17:27:56.314 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.314 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac28d4ff:::benchmark_data_smithi171_122772_object139973:head# 2024-04-20T17:27:56.314 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.315 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac2945a6:::benchmark_data_smithi171_122772_object137419:head# 2024-04-20T17:27:56.315 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.315 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac296ce5:::benchmark_data_smithi171_122772_object99909:head# 2024-04-20T17:27:56.315 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.315 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac2997d7:::benchmark_data_smithi171_122772_object94925:head# 2024-04-20T17:27:56.315 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.315 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac2bb09f:::benchmark_data_smithi171_122772_object145284:head# 2024-04-20T17:27:56.315 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.315 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac2d0d57:::benchmark_data_smithi171_122772_object143819:head# 2024-04-20T17:27:56.315 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.315 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac2e6525:::benchmark_data_smithi171_122772_object139371:head# 2024-04-20T17:27:56.315 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.316 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac2ec574:::benchmark_data_smithi171_122772_object154260:head# 2024-04-20T17:27:56.316 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.316 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac319634:::benchmark_data_smithi171_122772_object109591:head# 2024-04-20T17:27:56.316 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.316 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac31b53a:::benchmark_data_smithi171_122772_object104039:head# 2024-04-20T17:27:56.316 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.316 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac321885:::benchmark_data_smithi171_122772_object107430:head# 2024-04-20T17:27:56.316 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.316 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac33088c:::benchmark_data_smithi171_122772_object143585:head# 2024-04-20T17:27:56.316 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.316 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac33a064:::benchmark_data_smithi171_122772_object168012:head# 2024-04-20T17:27:56.316 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.316 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac33b137:::benchmark_data_smithi171_122772_object93394:head# 2024-04-20T17:27:56.317 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.317 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac354c68:::benchmark_data_smithi171_122772_object164211:head# 2024-04-20T17:27:56.317 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.317 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac36cd2e:::benchmark_data_smithi171_122772_object150188:head# 2024-04-20T17:27:56.317 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.317 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac3724c8:::benchmark_data_smithi171_122772_object136207:head# 2024-04-20T17:27:56.317 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.317 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac37cc5b:::benchmark_data_smithi171_122772_object167911:head# 2024-04-20T17:27:56.317 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.317 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac37e3d4:::benchmark_data_smithi171_122772_object102401:head# 2024-04-20T17:27:56.317 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.317 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac3837d7:::benchmark_data_smithi171_122772_object110600:head# 2024-04-20T17:27:56.317 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.318 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac3a1397:::benchmark_data_smithi171_122772_object96054:head# 2024-04-20T17:27:56.318 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.318 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac3af310:::benchmark_data_smithi171_122772_object86191:head# 2024-04-20T17:27:56.318 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.318 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac3ced0f:::benchmark_data_smithi171_122772_object92229:head# 2024-04-20T17:27:56.318 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.318 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac3d8605:::benchmark_data_smithi171_122772_object110826:head# 2024-04-20T17:27:56.318 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.318 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac402cd3:::benchmark_data_smithi171_122772_object107345:head# 2024-04-20T17:27:56.318 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.318 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac4080b9:::benchmark_data_smithi171_122772_object137319:head# 2024-04-20T17:27:56.319 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.319 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac413e5c:::benchmark_data_smithi171_122772_object109423:head# 2024-04-20T17:27:56.319 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.319 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac427622:::benchmark_data_smithi171_122772_object80345:head# 2024-04-20T17:27:56.319 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.319 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac43117f:::benchmark_data_smithi171_122772_object107172:head# 2024-04-20T17:27:56.319 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.330 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac432ebd:::benchmark_data_smithi171_122772_object132059:head# 2024-04-20T17:27:56.330 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.330 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac438947:::benchmark_data_smithi171_122772_object125039:head# 2024-04-20T17:27:56.330 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.330 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac4439d8:::benchmark_data_smithi171_122772_object113119:head# 2024-04-20T17:27:56.330 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.330 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac44a663:::benchmark_data_smithi171_122772_object111130:head# 2024-04-20T17:27:56.331 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.331 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac45cb55:::benchmark_data_smithi171_122772_object124323:head# 2024-04-20T17:27:56.331 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.331 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac475f88:::benchmark_data_smithi171_122772_object156896:head# 2024-04-20T17:27:56.331 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.331 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac4850f3:::benchmark_data_smithi171_122772_object150856:head# 2024-04-20T17:27:56.331 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.331 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac485da1:::benchmark_data_smithi171_122772_object117910:head# 2024-04-20T17:27:56.331 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.331 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac48b6b9:::benchmark_data_smithi171_122772_object162510:head# 2024-04-20T17:27:56.331 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.331 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac4906fc:::benchmark_data_smithi171_122772_object105978:head# 2024-04-20T17:27:56.331 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.332 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac497840:::benchmark_data_smithi171_122772_object131168:head# 2024-04-20T17:27:56.332 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.332 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac49edc6:::benchmark_data_smithi171_122772_object169238:head# 2024-04-20T17:27:56.332 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.332 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac49f234:::benchmark_data_smithi171_122772_object145816:head# 2024-04-20T17:27:56.332 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.332 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac4ab17f:::benchmark_data_smithi171_122772_object115421:head# 2024-04-20T17:27:56.332 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.332 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac4bab2f:::benchmark_data_smithi171_122772_object154324:head# 2024-04-20T17:27:56.332 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.332 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac4e206d:::benchmark_data_smithi171_122772_object169936:head# 2024-04-20T17:27:56.333 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.333 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac4e9e9b:::benchmark_data_smithi171_122772_object92256:head# 2024-04-20T17:27:56.333 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.333 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac4f211b:::benchmark_data_smithi171_122772_object90774:head# 2024-04-20T17:27:56.333 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.333 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac4f8056:::benchmark_data_smithi171_122772_object94650:head# 2024-04-20T17:27:56.333 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.333 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac501636:::benchmark_data_smithi171_122772_object154963:head# 2024-04-20T17:27:56.333 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.333 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac50f589:::benchmark_data_smithi171_122772_object145361:head# 2024-04-20T17:27:56.333 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.333 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac517780:::benchmark_data_smithi171_122772_object137157:head# 2024-04-20T17:27:56.334 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.334 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac527384:::benchmark_data_smithi171_122772_object139253:head# 2024-04-20T17:27:56.334 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.334 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac52dea4:::benchmark_data_smithi171_122772_object138571:head# 2024-04-20T17:27:56.334 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.334 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac530afe:::benchmark_data_smithi171_122772_object116398:head# 2024-04-20T17:27:56.334 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.334 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac5341f5:::benchmark_data_smithi171_122772_object145702:head# 2024-04-20T17:27:56.334 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.334 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac5352c1:::benchmark_data_smithi171_122772_object106012:head# 2024-04-20T17:27:56.334 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.335 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac549d75:::benchmark_data_smithi171_122772_object108630:head# 2024-04-20T17:27:56.335 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.335 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac54a9e3:::benchmark_data_smithi171_122772_object142952:head# 2024-04-20T17:27:56.335 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.335 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac5543eb:::benchmark_data_smithi171_122772_object142717:head# 2024-04-20T17:27:56.335 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.335 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac556681:::benchmark_data_smithi171_122772_object129573:head# 2024-04-20T17:27:56.335 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.335 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac556f12:::benchmark_data_smithi171_122772_object112959:head# 2024-04-20T17:27:56.335 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.335 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac557f37:::benchmark_data_smithi171_122772_object102900:head# 2024-04-20T17:27:56.336 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.336 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac55e18b:::benchmark_data_smithi171_122772_object80202:head# 2024-04-20T17:27:56.336 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.336 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac55f1e4:::benchmark_data_smithi171_122772_object110258:head# 2024-04-20T17:27:56.336 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.336 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac5774f2:::benchmark_data_smithi171_122772_object144432:head# 2024-04-20T17:27:56.336 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.336 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac580cd7:::benchmark_data_smithi171_122772_object114600:head# 2024-04-20T17:27:56.336 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.336 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac589ead:::benchmark_data_smithi171_122772_object95873:head# 2024-04-20T17:27:56.336 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.336 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac59dce8:::benchmark_data_smithi171_122772_object166536:head# 2024-04-20T17:27:56.336 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.337 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac5abac8:::benchmark_data_smithi171_122772_object120537:head# 2024-04-20T17:27:56.337 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.337 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac5ac502:::benchmark_data_smithi171_122772_object120900:head# 2024-04-20T17:27:56.337 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.337 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac5c86de:::benchmark_data_smithi171_122772_object87097:head# 2024-04-20T17:27:56.337 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.337 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac5caf46:::benchmark_data_smithi171_122772_object148528:head# 2024-04-20T17:27:56.337 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.337 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac5cf957:::benchmark_data_smithi171_122772_object166830:head# 2024-04-20T17:27:56.337 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.337 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac5d2b32:::benchmark_data_smithi171_122772_object112407:head# 2024-04-20T17:27:56.337 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.338 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac5e83d4:::benchmark_data_smithi171_122772_object79555:head# 2024-04-20T17:27:56.338 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.338 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac5fa20d:::benchmark_data_smithi171_122772_object148426:head# 2024-04-20T17:27:56.338 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.338 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac6048e6:::benchmark_data_smithi171_122772_object111655:head# 2024-04-20T17:27:56.338 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.338 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac60f477:::benchmark_data_smithi171_122772_object161969:head# 2024-04-20T17:27:56.338 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.338 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac60f976:::benchmark_data_smithi171_122772_object144804:head# 2024-04-20T17:27:56.338 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.338 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac611884:::benchmark_data_smithi171_122772_object105672:head# 2024-04-20T17:27:56.339 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.339 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac6198a8:::benchmark_data_smithi171_122772_object111523:head# 2024-04-20T17:27:56.339 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.339 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac61f663:::benchmark_data_smithi171_122772_object157773:head# 2024-04-20T17:27:56.339 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.339 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac6293f3:::benchmark_data_smithi171_122772_object141815:head# 2024-04-20T17:27:56.339 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.339 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac642e24:::benchmark_data_smithi171_122772_object103265:head# 2024-04-20T17:27:56.339 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.339 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac644244:::benchmark_data_smithi171_122772_object94631:head# 2024-04-20T17:27:56.339 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.339 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac66b745:::benchmark_data_smithi171_122772_object108536:head# 2024-04-20T17:27:56.339 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.340 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac6792ab:::benchmark_data_smithi171_122772_object84954:head# 2024-04-20T17:27:56.340 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.340 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac67c113:::benchmark_data_smithi171_122772_object129285:head# 2024-04-20T17:27:56.340 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.340 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac68d819:::benchmark_data_smithi171_122772_object119246:head# 2024-04-20T17:27:56.340 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.340 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac692e67:::benchmark_data_smithi171_122772_object98472:head# 2024-04-20T17:27:56.340 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.340 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac69e260:::benchmark_data_smithi171_122772_object98905:head# 2024-04-20T17:27:56.340 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.340 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac6a2fb1:::benchmark_data_smithi171_122772_object137175:head# 2024-04-20T17:27:56.340 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.340 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac6a6731:::benchmark_data_smithi171_122772_object137778:head# 2024-04-20T17:27:56.341 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.341 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac6a9077:::benchmark_data_smithi171_122772_object110226:head# 2024-04-20T17:27:56.341 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.341 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac6b04b5:::benchmark_data_smithi171_122772_object131423:head# 2024-04-20T17:27:56.341 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.341 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac6b981b:::benchmark_data_smithi171_122772_object97802:head# 2024-04-20T17:27:56.341 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.341 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac6c5767:::benchmark_data_smithi171_122772_object144902:head# 2024-04-20T17:27:56.341 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.341 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac6cc587:::benchmark_data_smithi171_122772_object106834:head# 2024-04-20T17:27:56.341 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.341 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac6edbe4:::benchmark_data_smithi171_122772_object160254:head# 2024-04-20T17:27:56.341 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.342 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac6f9e66:::benchmark_data_smithi171_122772_object139145:head# 2024-04-20T17:27:56.342 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.342 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac6fbd97:::benchmark_data_smithi171_122772_object165665:head# 2024-04-20T17:27:56.342 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.342 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac6fe56c:::benchmark_data_smithi171_122772_object155577:head# 2024-04-20T17:27:56.342 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.342 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac701ad3:::benchmark_data_smithi171_122772_object150934:head# 2024-04-20T17:27:56.342 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.342 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac7167c8:::benchmark_data_smithi171_122772_object156736:head# 2024-04-20T17:27:56.342 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.342 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac72fc82:::benchmark_data_smithi171_122772_object144498:head# 2024-04-20T17:27:56.342 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.343 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac73254d:::benchmark_data_smithi171_122772_object93149:head# 2024-04-20T17:27:56.343 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.343 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:56.349 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac737fe7:::benchmark_data_smithi171_122772_object94418:head# 2024-04-20T17:27:56.349 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.349 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac74339a:::benchmark_data_smithi171_122772_object105218:head# 2024-04-20T17:27:56.349 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.349 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac74d505:::benchmark_data_smithi171_122772_object121482:head# 2024-04-20T17:27:56.349 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.349 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac756fad:::benchmark_data_smithi171_122772_object117622:head# 2024-04-20T17:27:56.349 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.349 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac765296:::benchmark_data_smithi171_122772_object152777:head# 2024-04-20T17:27:56.349 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.349 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac76ade9:::benchmark_data_smithi171_122772_object90401:head# 2024-04-20T17:27:56.349 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.350 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac7766b0:::benchmark_data_smithi171_122772_object95508:head# 2024-04-20T17:27:56.350 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.350 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac781eeb:::benchmark_data_smithi171_122772_object77958:head# 2024-04-20T17:27:56.350 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.350 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac789d9b:::benchmark_data_smithi171_122772_object94458:head# 2024-04-20T17:27:56.350 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.350 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac79bada:::benchmark_data_smithi171_122772_object87343:head# 2024-04-20T17:27:56.350 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.350 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac7c9508:::benchmark_data_smithi171_122772_object129397:head# 2024-04-20T17:27:56.350 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.350 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac7d534b:::benchmark_data_smithi171_122772_object104537:head# 2024-04-20T17:27:56.350 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.350 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac7d9a64:::benchmark_data_smithi171_122772_object134517:head# 2024-04-20T17:27:56.351 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.351 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac7dad6e:::benchmark_data_smithi171_122772_object123120:head# 2024-04-20T17:27:56.351 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.351 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac7de6ad:::benchmark_data_smithi171_122772_object100185:head# 2024-04-20T17:27:56.351 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.351 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac7f8be9:::benchmark_data_smithi171_122772_object152301:head# 2024-04-20T17:27:56.351 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.351 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac8028e6:::benchmark_data_smithi171_122772_object93165:head# 2024-04-20T17:27:56.351 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.351 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac806cc3:::benchmark_data_smithi171_122772_object98209:head# 2024-04-20T17:27:56.351 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.351 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac80e89e:::benchmark_data_smithi171_122772_object84383:head# 2024-04-20T17:27:56.351 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.352 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac80ec05:::benchmark_data_smithi171_122772_object123846:head# 2024-04-20T17:27:56.352 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.352 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac8171ce:::benchmark_data_smithi171_122772_object165260:head# 2024-04-20T17:27:56.352 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.352 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac81e239:::benchmark_data_smithi171_122772_object122219:head# 2024-04-20T17:27:56.352 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.352 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac8206ea:::benchmark_data_smithi171_122772_object123857:head# 2024-04-20T17:27:56.352 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.352 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac82ea61:::benchmark_data_smithi171_122772_object80302:head# 2024-04-20T17:27:56.352 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.352 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac839033:::benchmark_data_smithi171_122772_object120600:head# 2024-04-20T17:27:56.352 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.352 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac843c1a:::benchmark_data_smithi171_122772_object109972:head# 2024-04-20T17:27:56.353 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.353 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac854c75:::benchmark_data_smithi171_122772_object161018:head# 2024-04-20T17:27:56.353 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.353 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac85e6a9:::benchmark_data_smithi171_122772_object165753:head# 2024-04-20T17:27:56.353 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.353 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac8631f6:::benchmark_data_smithi171_122772_object89693:head# 2024-04-20T17:27:56.353 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.353 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac86e191:::benchmark_data_smithi171_122772_object106722:head# 2024-04-20T17:27:56.353 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.353 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac87a659:::benchmark_data_smithi171_122772_object104413:head# 2024-04-20T17:27:56.353 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.353 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac8840ae:::benchmark_data_smithi171_122772_object95784:head# 2024-04-20T17:27:56.354 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.354 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac88d2b2:::benchmark_data_smithi171_122772_object143829:head# 2024-04-20T17:27:56.354 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.354 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac892c61:::benchmark_data_smithi171_122772_object133344:head# 2024-04-20T17:27:56.354 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.354 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac89f58e:::benchmark_data_smithi171_122772_object90166:head# 2024-04-20T17:27:56.354 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.354 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac8a4c34:::benchmark_data_smithi171_122772_object169259:head# 2024-04-20T17:27:56.354 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.354 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac8aa412:::benchmark_data_smithi171_122772_object155179:head# 2024-04-20T17:27:56.354 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.354 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac8b0ae4:::benchmark_data_smithi171_122772_object122223:head# 2024-04-20T17:27:56.355 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.355 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac8c7592:::benchmark_data_smithi171_122772_object165744:head# 2024-04-20T17:27:56.355 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.355 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac8c994e:::benchmark_data_smithi171_122772_object167144:head# 2024-04-20T17:27:56.355 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.355 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac8d3ae0:::benchmark_data_smithi171_122772_object157862:head# 2024-04-20T17:27:56.355 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.355 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac8d526d:::benchmark_data_smithi171_122772_object106664:head# 2024-04-20T17:27:56.355 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.355 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac8d7c34:::benchmark_data_smithi171_122772_object136833:head# 2024-04-20T17:27:56.355 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.356 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:27:56.356 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac8dba9a:::benchmark_data_smithi171_122772_object161452:head# 2024-04-20T17:27:56.356 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.356 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac8f4c26:::benchmark_data_smithi171_122772_object111325:head# 2024-04-20T17:27:56.357 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.357 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac8f6f00:::benchmark_data_smithi171_122772_object137555:head# 2024-04-20T17:27:56.357 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.357 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac903ed1:::benchmark_data_smithi171_122772_object146610:head# 2024-04-20T17:27:56.357 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.357 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac9098c0:::benchmark_data_smithi171_122772_object123957:head# 2024-04-20T17:27:56.357 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.357 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac9122a9:::benchmark_data_smithi171_122772_object110331:head# 2024-04-20T17:27:56.357 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:56.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:27:56.462 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:27:56.534 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac912561:::benchmark_data_smithi171_122772_object144128:head# 2024-04-20T17:27:56.534 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.534 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac913728:::benchmark_data_smithi171_122772_object156083:head# 2024-04-20T17:27:56.534 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.534 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac92a2ea:::benchmark_data_smithi171_122772_object89006:head# 2024-04-20T17:27:56.534 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.534 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac92dfed:::benchmark_data_smithi171_122772_object118576:head# 2024-04-20T17:27:56.534 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.535 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac93a566:::benchmark_data_smithi171_122772_object127017:head# 2024-04-20T17:27:56.535 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.535 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac96e3e4:::benchmark_data_smithi171_122772_object144916:head# 2024-04-20T17:27:56.535 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.535 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac96f488:::benchmark_data_smithi171_122772_object96848:head# 2024-04-20T17:27:56.535 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.535 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac97e2cf:::benchmark_data_smithi171_122772_object127748:head# 2024-04-20T17:27:56.535 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.535 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac9aeaa0:::benchmark_data_smithi171_122772_object101351:head# 2024-04-20T17:27:56.535 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.535 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac9bd810:::benchmark_data_smithi171_122772_object91776:head# 2024-04-20T17:27:56.536 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.536 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac9be2a5:::benchmark_data_smithi171_122772_object92640:head# 2024-04-20T17:27:56.536 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.536 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac9c155a:::benchmark_data_smithi171_122772_object164655:head# 2024-04-20T17:27:56.536 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.536 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac9c416e:::benchmark_data_smithi171_122772_object83583:head# 2024-04-20T17:27:56.536 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.536 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ac9d9b0b:::benchmark_data_smithi171_122772_object94112:head# 2024-04-20T17:27:56.536 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.536 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aca0a8e2:::benchmark_data_smithi171_122772_object105940:head# 2024-04-20T17:27:56.536 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.536 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aca12f78:::benchmark_data_smithi171_122772_object96417:head# 2024-04-20T17:27:56.536 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.536 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aca17065:::benchmark_data_smithi171_122772_object164968:head# 2024-04-20T17:27:56.537 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.537 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aca2a28d:::benchmark_data_smithi171_122772_object155493:head# 2024-04-20T17:27:56.537 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.537 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aca32aec:::benchmark_data_smithi171_122772_object161590:head# 2024-04-20T17:27:56.537 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.537 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aca5d00c:::benchmark_data_smithi171_122772_object108738:head# 2024-04-20T17:27:56.537 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.537 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aca5e196:::benchmark_data_smithi171_122772_object98306:head# 2024-04-20T17:27:56.537 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.537 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aca66769:::benchmark_data_smithi171_122772_object92225:head# 2024-04-20T17:27:56.537 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.537 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aca7911b:::benchmark_data_smithi171_122772_object153057:head# 2024-04-20T17:27:56.537 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.538 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aca7d9b3:::benchmark_data_smithi171_122772_object96025:head# 2024-04-20T17:27:56.538 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.538 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aca874bf:::benchmark_data_smithi171_122772_object138474:head# 2024-04-20T17:27:56.538 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.538 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acab3c8e:::benchmark_data_smithi171_122772_object147897:head# 2024-04-20T17:27:56.538 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.538 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acab88db:::benchmark_data_smithi171_122772_object140568:head# 2024-04-20T17:27:56.538 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.538 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acaba81c:::benchmark_data_smithi171_122772_object158698:head# 2024-04-20T17:27:56.538 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.538 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acac137d:::benchmark_data_smithi171_122772_object149256:head# 2024-04-20T17:27:56.538 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.539 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acaccddc:::benchmark_data_smithi171_122772_object94990:head# 2024-04-20T17:27:56.539 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.539 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acad90a9:::benchmark_data_smithi171_122772_object167071:head# 2024-04-20T17:27:56.539 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.539 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acadc151:::benchmark_data_smithi171_122772_object169661:head# 2024-04-20T17:27:56.539 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.539 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acae98ea:::benchmark_data_smithi171_122772_object158003:head# 2024-04-20T17:27:56.539 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.539 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acaece2b:::benchmark_data_smithi171_122772_object81305:head# 2024-04-20T17:27:56.539 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.539 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acaf0b60:::benchmark_data_smithi171_122772_object130206:head# 2024-04-20T17:27:56.539 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.539 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acaf36e9:::benchmark_data_smithi171_122772_object122813:head# 2024-04-20T17:27:56.540 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.540 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acaf6770:::benchmark_data_smithi171_122772_object80371:head# 2024-04-20T17:27:56.540 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.540 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb16eef:::benchmark_data_smithi171_122772_object116950:head# 2024-04-20T17:27:56.540 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.540 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb1d044:::benchmark_data_smithi171_122772_object116249:head# 2024-04-20T17:27:56.540 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.540 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb1f10b:::benchmark_data_smithi171_122772_object121595:head# 2024-04-20T17:27:56.540 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.540 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb2414f:::benchmark_data_smithi171_122772_object89272:head# 2024-04-20T17:27:56.540 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.540 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb29ed5:::benchmark_data_smithi171_122772_object130791:head# 2024-04-20T17:27:56.540 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.541 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb2ef61:::benchmark_data_smithi171_122772_object143344:head# 2024-04-20T17:27:56.541 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.541 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb2fe8a:::benchmark_data_smithi171_122772_object153780:head# 2024-04-20T17:27:56.541 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.541 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb310f6:::benchmark_data_smithi171_122772_object131402:head# 2024-04-20T17:27:56.541 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.541 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb361fc:::benchmark_data_smithi171_122772_object139322:head# 2024-04-20T17:27:56.541 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.541 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb41368:::benchmark_data_smithi171_122772_object106476:head# 2024-04-20T17:27:56.541 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.541 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb42697:::benchmark_data_smithi171_122772_object145647:head# 2024-04-20T17:27:56.541 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.541 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb4624b:::benchmark_data_smithi171_122772_object126594:head# 2024-04-20T17:27:56.541 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.542 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb49421:::benchmark_data_smithi171_122772_object156771:head# 2024-04-20T17:27:56.542 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.542 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb50c13:::benchmark_data_smithi171_122772_object116462:head# 2024-04-20T17:27:56.542 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.542 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb52e9e:::benchmark_data_smithi171_122772_object108995:head# 2024-04-20T17:27:56.542 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.542 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb597bd:::benchmark_data_smithi171_122772_object142671:head# 2024-04-20T17:27:56.542 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.542 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb60abe:::benchmark_data_smithi171_122772_object100036:head# 2024-04-20T17:27:56.542 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.542 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb643d8:::benchmark_data_smithi171_122772_object135365:head# 2024-04-20T17:27:56.542 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.543 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb692ad:::benchmark_data_smithi171_122772_object87540:head# 2024-04-20T17:27:56.543 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.543 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb6c2cb:::benchmark_data_smithi171_122772_object106596:head# 2024-04-20T17:27:56.543 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.543 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb6c995:::benchmark_data_smithi171_122772_object108151:head# 2024-04-20T17:27:56.543 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.543 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb73e48:::benchmark_data_smithi171_122772_object163179:head# 2024-04-20T17:27:56.543 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.543 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb74519:::benchmark_data_smithi171_122772_object163257:head# 2024-04-20T17:27:56.543 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.543 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb78e56:::benchmark_data_smithi171_122772_object149486:head# 2024-04-20T17:27:56.543 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.543 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb813dd:::benchmark_data_smithi171_122772_object103917:head# 2024-04-20T17:27:56.543 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.544 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb89dc5:::benchmark_data_smithi171_122772_object167381:head# 2024-04-20T17:27:56.544 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.544 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb8fa45:::benchmark_data_smithi171_122772_object105007:head# 2024-04-20T17:27:56.544 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.544 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acb9f5a1:::benchmark_data_smithi171_122772_object77998:head# 2024-04-20T17:27:56.544 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.544 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acbbad89:::benchmark_data_smithi171_122772_object122593:head# 2024-04-20T17:27:56.544 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.544 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acbbbc82:::benchmark_data_smithi171_122772_object102215:head# 2024-04-20T17:27:56.544 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.544 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acbcb502:::benchmark_data_smithi171_122772_object156704:head# 2024-04-20T17:27:56.544 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.545 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acbdd088:::benchmark_data_smithi171_122772_object104863:head# 2024-04-20T17:27:56.545 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.545 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acbde92e:::benchmark_data_smithi171_122772_object168187:head# 2024-04-20T17:27:56.545 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.545 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acbf8ed1:::benchmark_data_smithi171_122772_object155105:head# 2024-04-20T17:27:56.545 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.545 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acbff07a:::benchmark_data_smithi171_122772_object149592:head# 2024-04-20T17:27:56.545 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.545 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acc1cc6d:::benchmark_data_smithi171_122772_object143372:head# 2024-04-20T17:27:56.545 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.545 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acc41756:::benchmark_data_smithi171_122772_object107484:head# 2024-04-20T17:27:56.546 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.546 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acc486bd:::benchmark_data_smithi171_122772_object127005:head# 2024-04-20T17:27:56.546 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.546 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acc77b1b:::benchmark_data_smithi171_122772_object137722:head# 2024-04-20T17:27:56.546 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.546 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acc7a7af:::benchmark_data_smithi171_122772_object157548:head# 2024-04-20T17:27:56.546 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.546 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acc7d793:::benchmark_data_smithi171_122772_object78539:head# 2024-04-20T17:27:56.546 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.546 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acc7fce4:::benchmark_data_smithi171_122772_object126743:head# 2024-04-20T17:27:56.546 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.547 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acc86ed4:::benchmark_data_smithi171_122772_object132042:head# 2024-04-20T17:27:56.547 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.547 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acc8bb3d:::benchmark_data_smithi171_122772_object124419:head# 2024-04-20T17:27:56.547 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.547 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acc8e112:::benchmark_data_smithi171_122772_object86163:head# 2024-04-20T17:27:56.547 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.547 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acc9c908:::benchmark_data_smithi171_122772_object98318:head# 2024-04-20T17:27:56.547 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.547 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acc9de92:::benchmark_data_smithi171_122772_object115098:head# 2024-04-20T17:27:56.547 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.547 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acca7837:::benchmark_data_smithi171_122772_object136774:head# 2024-04-20T17:27:56.547 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.547 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:accbd127:::benchmark_data_smithi171_122772_object148514:head# 2024-04-20T17:27:56.548 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.548 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:accf2763:::benchmark_data_smithi171_122772_object137485:head# 2024-04-20T17:27:56.548 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.548 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:accf82df:::benchmark_data_smithi171_122772_object109043:head# 2024-04-20T17:27:56.548 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.548 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acd01bcd:::benchmark_data_smithi171_122772_object102406:head# 2024-04-20T17:27:56.548 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.548 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acd08180:::benchmark_data_smithi171_122772_object135289:head# 2024-04-20T17:27:56.548 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.548 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acd1883a:::benchmark_data_smithi171_122772_object84984:head# 2024-04-20T17:27:56.548 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.548 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acd1b078:::benchmark_data_smithi171_122772_object124445:head# 2024-04-20T17:27:56.548 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.549 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acd253c0:::benchmark_data_smithi171_122772_object79517:head# 2024-04-20T17:27:56.549 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.549 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acd2a1d6:::benchmark_data_smithi171_122772_object112104:head# 2024-04-20T17:27:56.549 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.549 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acd2b522:::benchmark_data_smithi171_122772_object148260:head# 2024-04-20T17:27:56.549 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.549 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acd360a6:::benchmark_data_smithi171_122772_object139511:head# 2024-04-20T17:27:56.549 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.549 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acd393d8:::benchmark_data_smithi171_122772_object119259:head# 2024-04-20T17:27:56.549 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.549 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acd5167c:::benchmark_data_smithi171_122772_object118735:head# 2024-04-20T17:27:56.550 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.550 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acd703c4:::benchmark_data_smithi171_122772_object125426:head# 2024-04-20T17:27:56.550 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.550 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acd8285d:::benchmark_data_smithi171_122772_object133132:head# 2024-04-20T17:27:56.550 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.550 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acd880a4:::benchmark_data_smithi171_122772_object139912:head# 2024-04-20T17:27:56.550 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.550 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acd88136:::benchmark_data_smithi171_122772_object126342:head# 2024-04-20T17:27:56.550 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.550 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acd89464:::benchmark_data_smithi171_122772_object127759:head# 2024-04-20T17:27:56.550 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.551 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acd8bc87:::benchmark_data_smithi171_122772_object79674:head# 2024-04-20T17:27:56.551 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.551 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acda4685:::benchmark_data_smithi171_122772_object131625:head# 2024-04-20T17:27:56.551 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.551 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acddd7ab:::benchmark_data_smithi171_122772_object96422:head# 2024-04-20T17:27:56.551 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.551 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ace056a1:::benchmark_data_smithi171_122772_object132647:head# 2024-04-20T17:27:56.551 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.551 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ace0587a:::benchmark_data_smithi171_122772_object103485:head# 2024-04-20T17:27:56.551 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.551 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ace1db5f:::benchmark_data_smithi171_122772_object107892:head# 2024-04-20T17:27:56.551 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.552 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ace21f63:::benchmark_data_smithi171_122772_object157722:head# 2024-04-20T17:27:56.552 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.552 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ace24dfd:::benchmark_data_smithi171_122772_object115527:head# 2024-04-20T17:27:56.552 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.552 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ace26036:::benchmark_data_smithi171_122772_object98200:head# 2024-04-20T17:27:56.552 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.552 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ace3d62f:::benchmark_data_smithi171_122772_object82063:head# 2024-04-20T17:27:56.552 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.552 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ace55360:::benchmark_data_smithi171_122772_object147020:head# 2024-04-20T17:27:56.552 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.552 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ace5efaf:::benchmark_data_smithi171_122772_object162060:head# 2024-04-20T17:27:56.553 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.553 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ace602d5:::benchmark_data_smithi171_122772_object158390:head# 2024-04-20T17:27:56.553 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.553 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ace6a8bb:::benchmark_data_smithi171_122772_object119241:head# 2024-04-20T17:27:56.553 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.553 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ace7b166:::benchmark_data_smithi171_122772_object98366:head# 2024-04-20T17:27:56.553 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.553 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ace9de52:::benchmark_data_smithi171_122772_object94603:head# 2024-04-20T17:27:56.553 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.553 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acea1a49:::benchmark_data_smithi171_122772_object135159:head# 2024-04-20T17:27:56.553 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.553 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acea3559:::benchmark_data_smithi171_122772_object139105:head# 2024-04-20T17:27:56.553 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.553 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acebaa54:::benchmark_data_smithi171_122772_object111562:head# 2024-04-20T17:27:56.554 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.554 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acebb140:::benchmark_data_smithi171_122772_object125361:head# 2024-04-20T17:27:56.554 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.554 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acebeb4c:::benchmark_data_smithi171_122772_object152851:head# 2024-04-20T17:27:56.554 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.554 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aceca955:::benchmark_data_smithi171_122772_object92740:head# 2024-04-20T17:27:56.554 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.554 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aced48fd:::benchmark_data_smithi171_122772_object86265:head# 2024-04-20T17:27:56.554 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.554 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aced6010:::benchmark_data_smithi171_122772_object169894:head# 2024-04-20T17:27:56.555 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.555 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acee2f35:::benchmark_data_smithi171_122772_object104184:head# 2024-04-20T17:27:56.555 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.555 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acf07314:::benchmark_data_smithi171_122772_object123702:head# 2024-04-20T17:27:56.555 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.555 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acf19c0b:::benchmark_data_smithi171_122772_object107389:head# 2024-04-20T17:27:56.555 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.555 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acf2c30e:::benchmark_data_smithi171_122772_object85702:head# 2024-04-20T17:27:56.555 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.555 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acf3326f:::benchmark_data_smithi171_122772_object119795:head# 2024-04-20T17:27:56.555 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.556 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acf4f5a1:::benchmark_data_smithi171_122772_object158298:head# 2024-04-20T17:27:56.556 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.556 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acf56d37:::benchmark_data_smithi171_122772_object134298:head# 2024-04-20T17:27:56.556 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.556 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acf5e68e:::benchmark_data_smithi171_122772_object93339:head# 2024-04-20T17:27:56.556 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.556 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acf6e4ff:::benchmark_data_smithi171_122772_object138866:head# 2024-04-20T17:27:56.556 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.556 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acf78341:::benchmark_data_smithi171_122772_object108042:head# 2024-04-20T17:27:56.556 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.556 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acf8c70b:::benchmark_data_smithi171_122772_object159565:head# 2024-04-20T17:27:56.556 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.556 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acf8d26b:::benchmark_data_smithi171_122772_object145951:head# 2024-04-20T17:27:56.557 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.557 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acf928df:::benchmark_data_smithi171_122772_object108492:head# 2024-04-20T17:27:56.557 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.557 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acf9db94:::benchmark_data_smithi171_122772_object159765:head# 2024-04-20T17:27:56.557 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.557 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acfa1f62:::benchmark_data_smithi171_122772_object138130:head# 2024-04-20T17:27:56.557 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.557 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acfa6560:::benchmark_data_smithi171_122772_object78323:head# 2024-04-20T17:27:56.557 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.557 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acfb6368:::benchmark_data_smithi171_122772_object127356:head# 2024-04-20T17:27:56.557 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.557 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acfbb790:::benchmark_data_smithi171_122772_object128124:head# 2024-04-20T17:27:56.558 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.558 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acfc3436:::benchmark_data_smithi171_122772_object157546:head# 2024-04-20T17:27:56.558 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.558 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acfdb411:::benchmark_data_smithi171_122772_object95191:head# 2024-04-20T17:27:56.558 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.558 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acfdbf1d:::benchmark_data_smithi171_122772_object147204:head# 2024-04-20T17:27:56.558 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.558 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acfdfe1a:::benchmark_data_smithi171_122772_object154081:head# 2024-04-20T17:27:56.558 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.558 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acfe5ffe:::benchmark_data_smithi171_122772_object125964:head# 2024-04-20T17:27:56.558 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.558 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acfea00a:::benchmark_data_smithi171_122772_object144840:head# 2024-04-20T17:27:56.558 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.559 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:acff2a96:::benchmark_data_smithi171_122772_object98041:head# 2024-04-20T17:27:56.559 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.559 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad00220e:::benchmark_data_smithi171_122772_object139509:head# 2024-04-20T17:27:56.559 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.559 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad012bd9:::benchmark_data_smithi171_122772_object127471:head# 2024-04-20T17:27:56.559 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.559 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad01409a:::benchmark_data_smithi171_122772_object169784:head# 2024-04-20T17:27:56.559 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.559 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad015e88:::benchmark_data_smithi171_122772_object161647:head# 2024-04-20T17:27:56.559 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.559 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad01608d:::benchmark_data_smithi171_122772_object79658:head# 2024-04-20T17:27:56.559 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.560 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad0209a8:::benchmark_data_smithi171_122772_object108410:head# 2024-04-20T17:27:56.560 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.560 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad02917b:::benchmark_data_smithi171_122772_object159371:head# 2024-04-20T17:27:56.560 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.560 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad02f6ed:::benchmark_data_smithi171_122772_object145572:head# 2024-04-20T17:27:56.560 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.560 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad030ea4:::benchmark_data_smithi171_122772_object137230:head# 2024-04-20T17:27:56.560 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.560 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad03d6ec:::benchmark_data_smithi171_122772_object101346:head# 2024-04-20T17:27:56.560 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.560 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad03fbf9:::benchmark_data_smithi171_122772_object108232:head# 2024-04-20T17:27:56.560 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.561 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad05d24e:::benchmark_data_smithi171_122772_object98175:head# 2024-04-20T17:27:56.561 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.561 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad072a54:::benchmark_data_smithi171_122772_object169159:head# 2024-04-20T17:27:56.561 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.561 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad09f6ed:::benchmark_data_smithi171_122772_object112760:head# 2024-04-20T17:27:56.561 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.561 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad0a52b6:::benchmark_data_smithi171_122772_object168599:head# 2024-04-20T17:27:56.561 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.561 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad0acf61:::benchmark_data_smithi171_122772_object106995:head# 2024-04-20T17:27:56.561 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.561 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad0ae0df:::benchmark_data_smithi171_122772_object159713:head# 2024-04-20T17:27:56.561 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.561 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad0aefa1:::benchmark_data_smithi171_122772_object166974:head# 2024-04-20T17:27:56.562 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.562 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad0be21d:::benchmark_data_smithi171_122772_object134131:head# 2024-04-20T17:27:56.562 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.562 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad0bfba0:::benchmark_data_smithi171_122772_object157243:head# 2024-04-20T17:27:56.562 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.562 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad0ccf40:::benchmark_data_smithi171_122772_object88014:head# 2024-04-20T17:27:56.562 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.562 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad0d403d:::benchmark_data_smithi171_122772_object168125:head# 2024-04-20T17:27:56.562 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.562 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad0e39d4:::benchmark_data_smithi171_122772_object90637:head# 2024-04-20T17:27:56.562 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.562 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad0f5d3a:::benchmark_data_smithi171_122772_object110124:head# 2024-04-20T17:27:56.563 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.563 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad0f790a:::benchmark_data_smithi171_122772_object115921:head# 2024-04-20T17:27:56.563 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.563 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad0fc7e7:::benchmark_data_smithi171_122772_object133055:head# 2024-04-20T17:27:56.563 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.563 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad100433:::benchmark_data_smithi171_122772_object153041:head# 2024-04-20T17:27:56.563 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.563 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad103138:::benchmark_data_smithi171_122772_object90429:head# 2024-04-20T17:27:56.563 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.563 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad104b37:::benchmark_data_smithi171_122772_object155409:head# 2024-04-20T17:27:56.563 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.563 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad116e27:::benchmark_data_smithi171_122772_object125155:head# 2024-04-20T17:27:56.563 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.564 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad11e3f0:::benchmark_data_smithi171_122772_object160877:head# 2024-04-20T17:27:56.564 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.564 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad128861:::benchmark_data_smithi171_122772_object110964:head# 2024-04-20T17:27:56.564 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.564 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad12bbf1:::benchmark_data_smithi171_122772_object127996:head# 2024-04-20T17:27:56.564 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.564 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad12c084:::benchmark_data_smithi171_122772_object110424:head# 2024-04-20T17:27:56.564 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.564 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad12e94c:::benchmark_data_smithi171_122772_object121219:head# 2024-04-20T17:27:56.564 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.565 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad13d5e6:::benchmark_data_smithi171_122772_object169789:head# 2024-04-20T17:27:56.565 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.565 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad140fe6:::benchmark_data_smithi171_122772_object169968:head# 2024-04-20T17:27:56.565 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.565 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad169a77:::benchmark_data_smithi171_122772_object98276:head# 2024-04-20T17:27:56.565 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.565 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad16de5b:::benchmark_data_smithi171_122772_object111735:head# 2024-04-20T17:27:56.565 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.565 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad173b49:::benchmark_data_smithi171_122772_object91613:head# 2024-04-20T17:27:56.565 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.565 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad17da71:::benchmark_data_smithi171_122772_object119289:head# 2024-04-20T17:27:56.566 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.566 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad190573:::benchmark_data_smithi171_122772_object120938:head# 2024-04-20T17:27:56.566 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.566 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad19f43a:::benchmark_data_smithi171_122772_object151142:head# 2024-04-20T17:27:56.566 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.566 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad1a5c17:::benchmark_data_smithi171_122772_object138618:head# 2024-04-20T17:27:56.566 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.566 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad1c9d44:::benchmark_data_smithi171_122772_object169610:head# 2024-04-20T17:27:56.566 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.566 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad1d7f88:::benchmark_data_smithi171_122772_object113787:head# 2024-04-20T17:27:56.566 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.566 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad1ee138:::benchmark_data_smithi171_122772_object84407:head# 2024-04-20T17:27:56.567 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.567 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad1fb785:::benchmark_data_smithi171_122772_object107980:head# 2024-04-20T17:27:56.567 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.567 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad2018eb:::benchmark_data_smithi171_122772_object137718:head# 2024-04-20T17:27:56.567 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.567 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad208c72:::benchmark_data_smithi171_122772_object148992:head# 2024-04-20T17:27:56.567 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.567 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad20ddba:::benchmark_data_smithi171_122772_object144061:head# 2024-04-20T17:27:56.567 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.567 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad21308f:::benchmark_data_smithi171_122772_object124352:head# 2024-04-20T17:27:56.567 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.567 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad231b2c:::benchmark_data_smithi171_122772_object85420:head# 2024-04-20T17:27:56.568 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.568 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad2491a6:::benchmark_data_smithi171_122772_object135676:head# 2024-04-20T17:27:56.568 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.568 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad24f36f:::benchmark_data_smithi171_122772_object102823:head# 2024-04-20T17:27:56.568 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.568 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad26db30:::benchmark_data_smithi171_122772_object121606:head# 2024-04-20T17:27:56.568 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.568 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad274bd1:::benchmark_data_smithi171_122772_object162102:head# 2024-04-20T17:27:56.568 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.568 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad27da0f:::benchmark_data_smithi171_122772_object119990:head# 2024-04-20T17:27:56.568 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.568 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad2814dc:::benchmark_data_smithi171_122772_object115639:head# 2024-04-20T17:27:56.569 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.569 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad2852b9:::benchmark_data_smithi171_122772_object96987:head# 2024-04-20T17:27:56.569 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.569 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad288405:::benchmark_data_smithi171_122772_object101212:head# 2024-04-20T17:27:56.569 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.569 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad2944ec:::benchmark_data_smithi171_122772_object154236:head# 2024-04-20T17:27:56.569 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.569 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad29b6ef:::benchmark_data_smithi171_122772_object100453:head# 2024-04-20T17:27:56.569 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.569 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad29c7e7:::benchmark_data_smithi171_122772_object98365:head# 2024-04-20T17:27:56.569 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.569 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad29ddb5:::benchmark_data_smithi171_122772_object114553:head# 2024-04-20T17:27:56.570 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.570 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad2a50ab:::benchmark_data_smithi171_122772_object111718:head# 2024-04-20T17:27:56.570 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.570 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad2bcaf6:::benchmark_data_smithi171_122772_object113204:head# 2024-04-20T17:27:56.570 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.570 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad2c6ab0:::benchmark_data_smithi171_122772_object88910:head# 2024-04-20T17:27:56.570 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.570 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad2db39c:::benchmark_data_smithi171_122772_object138333:head# 2024-04-20T17:27:56.570 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.570 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad2ee2a5:::benchmark_data_smithi171_122772_object115880:head# 2024-04-20T17:27:56.570 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.571 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad2ef549:::benchmark_data_smithi171_122772_object159963:head# 2024-04-20T17:27:56.571 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.571 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad2f684e:::benchmark_data_smithi171_122772_object125289:head# 2024-04-20T17:27:56.571 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.571 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad2fadd0:::benchmark_data_smithi171_122772_object86188:head# 2024-04-20T17:27:56.571 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.571 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad30a33f:::benchmark_data_smithi171_122772_object140037:head# 2024-04-20T17:27:56.571 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.571 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad31b320:::benchmark_data_smithi171_122772_object139035:head# 2024-04-20T17:27:56.571 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.571 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad32e8fd:::benchmark_data_smithi171_122772_object138405:head# 2024-04-20T17:27:56.572 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.572 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad33af90:::benchmark_data_smithi171_122772_object88849:head# 2024-04-20T17:27:56.572 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.572 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad354b49:::benchmark_data_smithi171_122772_object134446:head# 2024-04-20T17:27:56.572 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.572 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad35bc05:::benchmark_data_smithi171_122772_object79657:head# 2024-04-20T17:27:56.572 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.572 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad35d9af:::benchmark_data_smithi171_122772_object130433:head# 2024-04-20T17:27:56.572 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.572 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad3a2d17:::benchmark_data_smithi171_122772_object120020:head# 2024-04-20T17:27:56.572 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.572 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad3a6c43:::benchmark_data_smithi171_122772_object113501:head# 2024-04-20T17:27:56.572 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.573 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad3ae3c7:::benchmark_data_smithi171_122772_object169214:head# 2024-04-20T17:27:56.573 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.573 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad3afde1:::benchmark_data_smithi171_122772_object82989:head# 2024-04-20T17:27:56.573 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.573 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad3baa5a:::benchmark_data_smithi171_122772_object147450:head# 2024-04-20T17:27:56.573 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.573 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad3c6a33:::benchmark_data_smithi171_122772_object83636:head# 2024-04-20T17:27:56.573 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.573 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad3d15a9:::benchmark_data_smithi171_122772_object146076:head# 2024-04-20T17:27:56.573 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.573 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad3d925e:::benchmark_data_smithi171_122772_object112984:head# 2024-04-20T17:27:56.573 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.573 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad3e187a:::benchmark_data_smithi171_122772_object104963:head# 2024-04-20T17:27:56.574 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.574 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad3e439b:::benchmark_data_smithi171_122772_object160266:head# 2024-04-20T17:27:56.574 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.574 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad3f3704:::benchmark_data_smithi171_122772_object120010:head# 2024-04-20T17:27:56.574 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.574 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad40fa88:::benchmark_data_smithi171_122772_object165834:head# 2024-04-20T17:27:56.574 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.574 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad41188a:::benchmark_data_smithi171_122772_object138716:head# 2024-04-20T17:27:56.574 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.574 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad424ced:::benchmark_data_smithi171_122772_object163343:head# 2024-04-20T17:27:56.574 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.575 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad42797a:::benchmark_data_smithi171_122772_object114909:head# 2024-04-20T17:27:56.575 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.575 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad4345d0:::benchmark_data_smithi171_122772_object86857:head# 2024-04-20T17:27:56.575 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.575 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad450a84:::benchmark_data_smithi171_122772_object115972:head# 2024-04-20T17:27:56.575 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.575 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad46de7e:::benchmark_data_smithi171_122772_object131705:head# 2024-04-20T17:27:56.575 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.575 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad4727b8:::benchmark_data_smithi171_122772_object83846:head# 2024-04-20T17:27:56.575 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.575 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad47e8c2:::benchmark_data_smithi171_122772_object83513:head# 2024-04-20T17:27:56.575 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.576 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad4806e4:::benchmark_data_smithi171_122772_object89463:head# 2024-04-20T17:27:56.576 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.576 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad48c4f0:::benchmark_data_smithi171_122772_object142780:head# 2024-04-20T17:27:56.576 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.576 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad4ae65b:::benchmark_data_smithi171_122772_object147762:head# 2024-04-20T17:27:56.576 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.576 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad4bf642:::benchmark_data_smithi171_122772_object90231:head# 2024-04-20T17:27:56.576 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.576 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad4c2728:::benchmark_data_smithi171_122772_object79452:head# 2024-04-20T17:27:56.576 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.576 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad4c2e50:::benchmark_data_smithi171_122772_object98876:head# 2024-04-20T17:27:56.576 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.577 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad4f5189:::benchmark_data_smithi171_122772_object164295:head# 2024-04-20T17:27:56.577 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.577 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad4f87cc:::benchmark_data_smithi171_122772_object105730:head# 2024-04-20T17:27:56.577 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.577 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad5022ae:::benchmark_data_smithi171_122772_object138863:head# 2024-04-20T17:27:56.577 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.577 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad504f60:::benchmark_data_smithi171_122772_object99696:head# 2024-04-20T17:27:56.577 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.577 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad517a24:::benchmark_data_smithi171_122772_object83198:head# 2024-04-20T17:27:56.577 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.577 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad52f1ed:::benchmark_data_smithi171_122772_object134802:head# 2024-04-20T17:27:56.577 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.577 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad536687:::benchmark_data_smithi171_122772_object127162:head# 2024-04-20T17:27:56.578 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.578 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad542d1d:::benchmark_data_smithi171_122772_object144997:head# 2024-04-20T17:27:56.578 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.578 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad54353a:::benchmark_data_smithi171_122772_object167880:head# 2024-04-20T17:27:56.578 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.578 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad5533b9:::benchmark_data_smithi171_122772_object115224:head# 2024-04-20T17:27:56.578 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.578 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad57bfa1:::benchmark_data_smithi171_122772_object118785:head# 2024-04-20T17:27:56.578 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.578 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad58aa84:::benchmark_data_smithi171_122772_object153680:head# 2024-04-20T17:27:56.578 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.579 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad5b0a4e:::benchmark_data_smithi171_122772_object108554:head# 2024-04-20T17:27:56.579 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.579 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad5b725c:::benchmark_data_smithi171_122772_object159027:head# 2024-04-20T17:27:56.579 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.579 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad5c8dae:::benchmark_data_smithi171_122772_object159751:head# 2024-04-20T17:27:56.579 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.579 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad5d2dbb:::benchmark_data_smithi171_122772_object87042:head# 2024-04-20T17:27:56.579 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.579 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad5dcf58:::benchmark_data_smithi171_122772_object167847:head# 2024-04-20T17:27:56.579 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.579 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad5e8adb:::benchmark_data_smithi171_122772_object168559:head# 2024-04-20T17:27:56.579 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.579 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad603787:::benchmark_data_smithi171_122772_object127348:head# 2024-04-20T17:27:56.579 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.580 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad608712:::benchmark_data_smithi171_122772_object138604:head# 2024-04-20T17:27:56.580 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.580 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad60f79a:::benchmark_data_smithi171_122772_object125696:head# 2024-04-20T17:27:56.580 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.580 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad61fd36:::benchmark_data_smithi171_122772_object133255:head# 2024-04-20T17:27:56.580 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.580 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad642bd3:::benchmark_data_smithi171_122772_object115129:head# 2024-04-20T17:27:56.580 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.580 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad65c825:::benchmark_data_smithi171_122772_object147546:head# 2024-04-20T17:27:56.580 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.580 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad66474d:::benchmark_data_smithi171_122772_object125095:head# 2024-04-20T17:27:56.580 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.581 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad673bb7:::benchmark_data_smithi171_122772_object81339:head# 2024-04-20T17:27:56.581 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.581 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad688ee1:::benchmark_data_smithi171_122772_object126140:head# 2024-04-20T17:27:56.581 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.581 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad6b6ac6:::benchmark_data_smithi171_122772_object98595:head# 2024-04-20T17:27:56.581 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.581 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad6bad0a:::benchmark_data_smithi171_122772_object139958:head# 2024-04-20T17:27:56.581 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.581 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad6fcfb6:::benchmark_data_smithi171_122772_object83726:head# 2024-04-20T17:27:56.581 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.581 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad723187:::benchmark_data_smithi171_122772_object152797:head# 2024-04-20T17:27:56.581 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.582 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad7362f3:::benchmark_data_smithi171_122772_object141311:head# 2024-04-20T17:27:56.582 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.582 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad737e28:::benchmark_data_smithi171_122772_object140498:head# 2024-04-20T17:27:56.582 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.582 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad750161:::benchmark_data_smithi171_122772_object92892:head# 2024-04-20T17:27:56.582 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.582 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad750473:::benchmark_data_smithi171_122772_object161248:head# 2024-04-20T17:27:56.582 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.582 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad76fa02:::benchmark_data_smithi171_122772_object100422:head# 2024-04-20T17:27:56.582 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.582 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad779e6a:::benchmark_data_smithi171_122772_object138779:head# 2024-04-20T17:27:56.582 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.582 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad77daf2:::benchmark_data_smithi171_122772_object161393:head# 2024-04-20T17:27:56.583 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.583 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad78c4c5:::benchmark_data_smithi171_122772_object101439:head# 2024-04-20T17:27:56.583 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.583 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad78df47:::benchmark_data_smithi171_122772_object152399:head# 2024-04-20T17:27:56.583 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.583 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad7ac00e:::benchmark_data_smithi171_122772_object112962:head# 2024-04-20T17:27:56.583 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.583 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad7be04a:::benchmark_data_smithi171_122772_object84413:head# 2024-04-20T17:27:56.583 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.583 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad7c68fb:::benchmark_data_smithi171_122772_object141012:head# 2024-04-20T17:27:56.583 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.583 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad7c8426:::benchmark_data_smithi171_122772_object104947:head# 2024-04-20T17:27:56.583 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.584 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad7d40d5:::benchmark_data_smithi171_122772_object140945:head# 2024-04-20T17:27:56.584 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.584 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad801900:::benchmark_data_smithi171_122772_object140238:head# 2024-04-20T17:27:56.584 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.584 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad809a8c:::benchmark_data_smithi171_122772_object100222:head# 2024-04-20T17:27:56.584 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.584 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad815251:::benchmark_data_smithi171_122772_object127376:head# 2024-04-20T17:27:56.584 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.584 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad815bc9:::benchmark_data_smithi171_122772_object97575:head# 2024-04-20T17:27:56.584 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.584 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad8167df:::benchmark_data_smithi171_122772_object167960:head# 2024-04-20T17:27:56.584 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.584 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad81d050:::benchmark_data_smithi171_122772_object161340:head# 2024-04-20T17:27:56.585 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.585 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad81f5dc:::benchmark_data_smithi171_122772_object105189:head# 2024-04-20T17:27:56.585 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.585 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad824108:::benchmark_data_smithi171_122772_object92396:head# 2024-04-20T17:27:56.585 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.585 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad838bd6:::benchmark_data_smithi171_122772_object124277:head# 2024-04-20T17:27:56.585 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.585 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad83d162:::benchmark_data_smithi171_122772_object158801:head# 2024-04-20T17:27:56.585 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.585 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad84307e:::benchmark_data_smithi171_122772_object101433:head# 2024-04-20T17:27:56.585 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.585 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad843c51:::benchmark_data_smithi171_122772_object165048:head# 2024-04-20T17:27:56.586 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.586 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad84b848:::benchmark_data_smithi171_122772_object101743:head# 2024-04-20T17:27:56.586 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.586 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad859c6b:::benchmark_data_smithi171_122772_object85575:head# 2024-04-20T17:27:56.586 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.586 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad85c79b:::benchmark_data_smithi171_122772_object146141:head# 2024-04-20T17:27:56.586 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.586 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad85d2d5:::benchmark_data_smithi171_122772_object109257:head# 2024-04-20T17:27:56.590 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.590 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad86711a:::benchmark_data_smithi171_122772_object140104:head# 2024-04-20T17:27:56.591 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.591 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad87e6e4:::benchmark_data_smithi171_122772_object136821:head# 2024-04-20T17:27:56.591 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.591 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad884f90:::benchmark_data_smithi171_122772_object87302:head# 2024-04-20T17:27:56.591 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.591 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad8908f0:::benchmark_data_smithi171_122772_object83448:head# 2024-04-20T17:27:56.591 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.591 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad8aa52d:::benchmark_data_smithi171_122772_object90780:head# 2024-04-20T17:27:56.591 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.591 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad8dc09e:::benchmark_data_smithi171_122772_object117789:head# 2024-04-20T17:27:56.591 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.592 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad8dd4a0:::benchmark_data_smithi171_122772_object142239:head# 2024-04-20T17:27:56.592 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.592 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad8deb82:::benchmark_data_smithi171_122772_object118871:head# 2024-04-20T17:27:56.592 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.592 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad8e0196:::benchmark_data_smithi171_122772_object81361:head# 2024-04-20T17:27:56.592 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.592 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad8eac2f:::benchmark_data_smithi171_122772_object137194:head# 2024-04-20T17:27:56.592 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.592 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad8f3f6f:::benchmark_data_smithi171_122772_object101999:head# 2024-04-20T17:27:56.592 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.592 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad8f43aa:::benchmark_data_smithi171_122772_object132015:head# 2024-04-20T17:27:56.592 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.592 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad8fe4fb:::benchmark_data_smithi171_122772_object107393:head# 2024-04-20T17:27:56.593 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.593 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad903484:::benchmark_data_smithi171_122772_object146161:head# 2024-04-20T17:27:56.593 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.593 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad906b4f:::benchmark_data_smithi171_122772_object145628:head# 2024-04-20T17:27:56.593 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.593 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad91d503:::benchmark_data_smithi171_122772_object91188:head# 2024-04-20T17:27:56.593 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.593 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad9206cc:::benchmark_data_smithi171_122772_object124345:head# 2024-04-20T17:27:56.593 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.593 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad920765:::benchmark_data_smithi171_122772_object145870:head# 2024-04-20T17:27:56.593 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.594 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad932bf7:::benchmark_data_smithi171_122772_object139014:head# 2024-04-20T17:27:56.594 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.594 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad94ea4d:::benchmark_data_smithi171_122772_object150559:head# 2024-04-20T17:27:56.594 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.594 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad951f82:::benchmark_data_smithi171_122772_object93049:head# 2024-04-20T17:27:56.594 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.594 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad95c476:::benchmark_data_smithi171_122772_object148946:head# 2024-04-20T17:27:56.594 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.594 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad96a6c9:::benchmark_data_smithi171_122772_object160549:head# 2024-04-20T17:27:56.594 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.594 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad96aaec:::benchmark_data_smithi171_122772_object79633:head# 2024-04-20T17:27:56.594 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.595 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad97e21b:::benchmark_data_smithi171_122772_object165660:head# 2024-04-20T17:27:56.595 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.595 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad987093:::benchmark_data_smithi171_122772_object131334:head# 2024-04-20T17:27:56.595 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.595 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad98c010:::benchmark_data_smithi171_122772_object156194:head# 2024-04-20T17:27:56.595 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.595 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad98f54a:::benchmark_data_smithi171_122772_object151448:head# 2024-04-20T17:27:56.595 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.595 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad990338:::benchmark_data_smithi171_122772_object82322:head# 2024-04-20T17:27:56.595 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.595 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad9a3253:::benchmark_data_smithi171_122772_object118602:head# 2024-04-20T17:27:56.595 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.596 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad9bd7d6:::benchmark_data_smithi171_122772_object145759:head# 2024-04-20T17:27:56.596 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.596 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad9bed69:::benchmark_data_smithi171_122772_object123411:head# 2024-04-20T17:27:56.596 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.596 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad9da353:::benchmark_data_smithi171_122772_object97487:head# 2024-04-20T17:27:56.596 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.596 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad9eb316:::benchmark_data_smithi171_122772_object81358:head# 2024-04-20T17:27:56.596 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.596 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ad9fb696:::benchmark_data_smithi171_122772_object124931:head# 2024-04-20T17:27:56.596 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.596 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ada0451f:::benchmark_data_smithi171_122772_object153371:head# 2024-04-20T17:27:56.596 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.596 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ada08441:::benchmark_data_smithi171_122772_object127511:head# 2024-04-20T17:27:56.597 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.597 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ada09a4a:::benchmark_data_smithi171_122772_object125257:head# 2024-04-20T17:27:56.597 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.597 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ada111ea:::benchmark_data_smithi171_122772_object129283:head# 2024-04-20T17:27:56.597 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.597 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ada1bf7e:::benchmark_data_smithi171_122772_object109157:head# 2024-04-20T17:27:56.597 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.597 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ada2a2ba:::benchmark_data_smithi171_122772_object126607:head# 2024-04-20T17:27:56.597 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.597 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ada2c9e7:::benchmark_data_smithi171_122772_object90027:head# 2024-04-20T17:27:56.597 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.597 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ada2f6d0:::benchmark_data_smithi171_122772_object114823:head# 2024-04-20T17:27:56.597 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.598 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ada328d9:::benchmark_data_smithi171_122772_object80575:head# 2024-04-20T17:27:56.598 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.598 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ada38632:::benchmark_data_smithi171_122772_object130040:head# 2024-04-20T17:27:56.598 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.598 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ada55edc:::benchmark_data_smithi171_122772_object162826:head# 2024-04-20T17:27:56.598 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.598 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ada5b884:::benchmark_data_smithi171_122772_object80904:head# 2024-04-20T17:27:56.598 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.598 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ada6278b:::benchmark_data_smithi171_122772_object133092:head# 2024-04-20T17:27:56.598 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.598 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ada849d4:::benchmark_data_smithi171_122772_object132853:head# 2024-04-20T17:27:56.599 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.599 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ada884e1:::benchmark_data_smithi171_122772_object162260:head# 2024-04-20T17:27:56.599 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.599 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ada8bf4c:::benchmark_data_smithi171_122772_object145935:head# 2024-04-20T17:27:56.599 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.599 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ada9ae98:::benchmark_data_smithi171_122772_object85015:head# 2024-04-20T17:27:56.599 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.599 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adaaf968:::benchmark_data_smithi171_122772_object141281:head# 2024-04-20T17:27:56.599 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.599 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adaca7c0:::benchmark_data_smithi171_122772_object158763:head# 2024-04-20T17:27:56.599 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.599 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adacf396:::benchmark_data_smithi171_122772_object136117:head# 2024-04-20T17:27:56.600 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.600 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adaee6a8:::benchmark_data_smithi171_122772_object88452:head# 2024-04-20T17:27:56.600 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.600 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adaef04b:::benchmark_data_smithi171_122772_object169976:head# 2024-04-20T17:27:56.600 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.600 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adaf87ba:::benchmark_data_smithi171_122772_object158144:head# 2024-04-20T17:27:56.600 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.600 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adb08471:::benchmark_data_smithi171_122772_object106343:head# 2024-04-20T17:27:56.600 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.600 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adb12599:::benchmark_data_smithi171_122772_object123598:head# 2024-04-20T17:27:56.600 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.600 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adb1eac9:::benchmark_data_smithi171_122772_object146179:head# 2024-04-20T17:27:56.600 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.601 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adb21f8a:::benchmark_data_smithi171_122772_object80779:head# 2024-04-20T17:27:56.601 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.601 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adb24f77:::benchmark_data_smithi171_122772_object102294:head# 2024-04-20T17:27:56.601 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.601 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adb31b72:::benchmark_data_smithi171_122772_object133428:head# 2024-04-20T17:27:56.601 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.601 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adb34aff:::benchmark_data_smithi171_122772_object105036:head# 2024-04-20T17:27:56.601 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.601 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adb59721:::benchmark_data_smithi171_122772_object116889:head# 2024-04-20T17:27:56.601 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.601 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adb5af23:::benchmark_data_smithi171_122772_object120761:head# 2024-04-20T17:27:56.602 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.602 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adb6d8e9:::benchmark_data_smithi171_122772_object156700:head# 2024-04-20T17:27:56.602 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.602 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adb7e60a:::benchmark_data_smithi171_122772_object159190:head# 2024-04-20T17:27:56.602 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.602 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adb8318d:::benchmark_data_smithi171_122772_object82550:head# 2024-04-20T17:27:56.602 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.602 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adb84554:::benchmark_data_smithi171_122772_object140709:head# 2024-04-20T17:27:56.602 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.602 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adb8bc52:::benchmark_data_smithi171_122772_object122650:head# 2024-04-20T17:27:56.603 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.603 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adb8c6bf:::benchmark_data_smithi171_122772_object79081:head# 2024-04-20T17:27:56.603 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.603 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adbb357b:::benchmark_data_smithi171_122772_object141300:head# 2024-04-20T17:27:56.603 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.603 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adbb62a2:::benchmark_data_smithi171_122772_object142562:head# 2024-04-20T17:27:56.603 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.603 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adbcc08d:::benchmark_data_smithi171_122772_object106624:head# 2024-04-20T17:27:56.603 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.603 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adbd1226:::benchmark_data_smithi171_122772_object131076:head# 2024-04-20T17:27:56.604 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.604 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adbd19ae:::benchmark_data_smithi171_122772_object79787:head# 2024-04-20T17:27:56.604 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.604 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adbd58dd:::benchmark_data_smithi171_122772_object78087:head# 2024-04-20T17:27:56.604 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.604 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adbe7d62:::benchmark_data_smithi171_122772_object138586:head# 2024-04-20T17:27:56.604 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.604 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adc036cb:::benchmark_data_smithi171_122772_object153489:head# 2024-04-20T17:27:56.604 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.604 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adc11682:::benchmark_data_smithi171_122772_object88796:head# 2024-04-20T17:27:56.604 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.604 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adc31313:::benchmark_data_smithi171_122772_object84072:head# 2024-04-20T17:27:56.605 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.605 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adc3393a:::benchmark_data_smithi171_122772_object138362:head# 2024-04-20T17:27:56.605 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.605 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adc3501f:::benchmark_data_smithi171_122772_object160343:head# 2024-04-20T17:27:56.605 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.605 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adc3df13:::benchmark_data_smithi171_122772_object122379:head# 2024-04-20T17:27:56.605 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.605 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adc43a98:::benchmark_data_smithi171_122772_object108717:head# 2024-04-20T17:27:56.605 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.605 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adc44073:::benchmark_data_smithi171_122772_object104103:head# 2024-04-20T17:27:56.605 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.605 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adc4ac0b:::benchmark_data_smithi171_122772_object159895:head# 2024-04-20T17:27:56.606 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.606 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adc5b0f3:::benchmark_data_smithi171_122772_object154946:head# 2024-04-20T17:27:56.606 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.606 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adc61fd6:::benchmark_data_smithi171_122772_object164177:head# 2024-04-20T17:27:56.606 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.606 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adc65f05:::benchmark_data_smithi171_122772_object134067:head# 2024-04-20T17:27:56.606 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.606 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adc70fd8:::benchmark_data_smithi171_122772_object113127:head# 2024-04-20T17:27:56.606 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.606 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adc7de35:::benchmark_data_smithi171_122772_object86032:head# 2024-04-20T17:27:56.606 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.606 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adc86c61:::benchmark_data_smithi171_122772_object115555:head# 2024-04-20T17:27:56.607 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.607 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adc87417:::benchmark_data_smithi171_122772_object151074:head# 2024-04-20T17:27:56.607 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.607 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adc8accf:::benchmark_data_smithi171_122772_object92171:head# 2024-04-20T17:27:56.607 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.607 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adcb6c33:::benchmark_data_smithi171_122772_object125055:head# 2024-04-20T17:27:56.607 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.607 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adcb9745:::benchmark_data_smithi171_122772_object164732:head# 2024-04-20T17:27:56.607 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.607 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adcc6856:::benchmark_data_smithi171_122772_object123537:head# 2024-04-20T17:27:56.607 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.607 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adcd175b:::benchmark_data_smithi171_122772_object157669:head# 2024-04-20T17:27:56.607 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.608 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adcd9675:::benchmark_data_smithi171_122772_object99005:head# 2024-04-20T17:27:56.608 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.608 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adcdcea9:::benchmark_data_smithi171_122772_object105575:head# 2024-04-20T17:27:56.608 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.608 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adce41d0:::benchmark_data_smithi171_122772_object160775:head# 2024-04-20T17:27:56.608 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.608 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adced7c8:::benchmark_data_smithi171_122772_object162109:head# 2024-04-20T17:27:56.608 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.608 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adcf0bf6:::benchmark_data_smithi171_122772_object155787:head# 2024-04-20T17:27:56.608 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.608 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adcf14fb:::benchmark_data_smithi171_122772_object117809:head# 2024-04-20T17:27:56.608 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.609 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adcfa2c5:::benchmark_data_smithi171_122772_object146805:head# 2024-04-20T17:27:56.609 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.609 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:add0a0fc:::benchmark_data_smithi171_122772_object145556:head# 2024-04-20T17:27:56.609 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.609 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:add0cee0:::benchmark_data_smithi171_122772_object156387:head# 2024-04-20T17:27:56.609 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.609 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:add4602e:::benchmark_data_smithi171_122772_object137543:head# 2024-04-20T17:27:56.609 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.609 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:add54929:::benchmark_data_smithi171_122772_object152105:head# 2024-04-20T17:27:56.609 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.609 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:add55e39:::benchmark_data_smithi171_122772_object158007:head# 2024-04-20T17:27:56.609 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.609 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:add6f739:::benchmark_data_smithi171_122772_object167252:head# 2024-04-20T17:27:56.609 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.610 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:add7b9a2:::benchmark_data_smithi171_122772_object95349:head# 2024-04-20T17:27:56.610 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.610 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:add8b65a:::benchmark_data_smithi171_122772_object160965:head# 2024-04-20T17:27:56.610 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.610 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adda6173:::benchmark_data_smithi171_122772_object118448:head# 2024-04-20T17:27:56.610 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.610 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adda9331:::benchmark_data_smithi171_122772_object80066:head# 2024-04-20T17:27:56.610 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.610 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:addab75e:::benchmark_data_smithi171_122772_object138212:head# 2024-04-20T17:27:56.610 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.610 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:addafc57:::benchmark_data_smithi171_122772_object119509:head# 2024-04-20T17:27:56.610 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.611 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:addb4690:::benchmark_data_smithi171_122772_object137629:head# 2024-04-20T17:27:56.611 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.611 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:addbe0c4:::benchmark_data_smithi171_122772_object118783:head# 2024-04-20T17:27:56.611 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.611 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:addc5953:::benchmark_data_smithi171_122772_object150423:head# 2024-04-20T17:27:56.611 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.611 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adde866e:::benchmark_data_smithi171_122772_object159237:head# 2024-04-20T17:27:56.611 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.611 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:addeff4b:::benchmark_data_smithi171_122772_object130260:head# 2024-04-20T17:27:56.611 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.611 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:addf1de4:::benchmark_data_smithi171_122772_object168254:head# 2024-04-20T17:27:56.611 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.611 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:addf63b1:::benchmark_data_smithi171_122772_object102743:head# 2024-04-20T17:27:56.611 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.612 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:addf7d1a:::benchmark_data_smithi171_122772_object106332:head# 2024-04-20T17:27:56.612 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.612 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:addfcb5f:::benchmark_data_smithi171_122772_object154823:head# 2024-04-20T17:27:56.612 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.612 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ade12d4c:::benchmark_data_smithi171_122772_object87326:head# 2024-04-20T17:27:56.612 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.612 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ade3269b:::benchmark_data_smithi171_122772_object84101:head# 2024-04-20T17:27:56.612 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.612 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ade3dcdf:::benchmark_data_smithi171_122772_object153877:head# 2024-04-20T17:27:56.612 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.612 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ade47089:::benchmark_data_smithi171_122772_object87792:head# 2024-04-20T17:27:56.612 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.612 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ade4ed80:::benchmark_data_smithi171_122772_object96767:head# 2024-04-20T17:27:56.612 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.613 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ade6cb79:::benchmark_data_smithi171_122772_object123899:head# 2024-04-20T17:27:56.613 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.613 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ade782b9:::benchmark_data_smithi171_122772_object142588:head# 2024-04-20T17:27:56.613 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.613 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ade8f34b:::benchmark_data_smithi171_122772_object161676:head# 2024-04-20T17:27:56.613 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.613 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adea2a24:::benchmark_data_smithi171_122772_object88147:head# 2024-04-20T17:27:56.613 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.613 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adea84b2:::benchmark_data_smithi171_122772_object120387:head# 2024-04-20T17:27:56.613 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.613 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adeaab36:::benchmark_data_smithi171_122772_object160203:head# 2024-04-20T17:27:56.613 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.613 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adeb31ab:::benchmark_data_smithi171_122772_object143574:head# 2024-04-20T17:27:56.614 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.614 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adeb4d85:::benchmark_data_smithi171_122772_object126109:head# 2024-04-20T17:27:56.614 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.614 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adec14de:::benchmark_data_smithi171_122772_object138819:head# 2024-04-20T17:27:56.614 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.614 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aded7ce2:::benchmark_data_smithi171_122772_object89752:head# 2024-04-20T17:27:56.614 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.614 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adee01ff:::benchmark_data_smithi171_122772_object136179:head# 2024-04-20T17:27:56.614 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.614 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adee3d6d:::benchmark_data_smithi171_122772_object165711:head# 2024-04-20T17:27:56.614 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.614 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adeea1b6:::benchmark_data_smithi171_122772_object85712:head# 2024-04-20T17:27:56.614 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.615 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adeef6ce:::benchmark_data_smithi171_122772_object148380:head# 2024-04-20T17:27:56.615 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.615 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adf01d7e:::benchmark_data_smithi171_122772_object150409:head# 2024-04-20T17:27:56.615 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.615 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adf0fe63:::benchmark_data_smithi171_122772_object144465:head# 2024-04-20T17:27:56.615 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.615 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adf24341:::benchmark_data_smithi171_122772_object92467:head# 2024-04-20T17:27:56.615 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.615 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adf27e7b:::benchmark_data_smithi171_122772_object130812:head# 2024-04-20T17:27:56.615 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.615 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adf3e1f0:::benchmark_data_smithi171_122772_object121861:head# 2024-04-20T17:27:56.615 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.615 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adf4639a:::benchmark_data_smithi171_122772_object120624:head# 2024-04-20T17:27:56.615 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.616 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adf5d18e:::benchmark_data_smithi171_122772_object116787:head# 2024-04-20T17:27:56.616 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.616 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adf63b34:::benchmark_data_smithi171_122772_object98512:head# 2024-04-20T17:27:56.616 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.616 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adf6a9eb:::benchmark_data_smithi171_122772_object118351:head# 2024-04-20T17:27:56.616 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.616 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adf6b8a3:::benchmark_data_smithi171_122772_object153176:head# 2024-04-20T17:27:56.616 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.616 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adf6cdf8:::benchmark_data_smithi171_122772_object135343:head# 2024-04-20T17:27:56.616 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.616 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adf853fb:::benchmark_data_smithi171_122772_object101991:head# 2024-04-20T17:27:56.616 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.616 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adf8897d:::benchmark_data_smithi171_122772_object123834:head# 2024-04-20T17:27:56.617 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.617 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adf954c2:::benchmark_data_smithi171_122772_object169419:head# 2024-04-20T17:27:56.617 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.617 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adfa156f:::benchmark_data_smithi171_122772_object137101:head# 2024-04-20T17:27:56.617 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.617 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adfbd3d2:::benchmark_data_smithi171_122772_object158586:head# 2024-04-20T17:27:56.617 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.617 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adfc0ed4:::benchmark_data_smithi171_122772_object80778:head# 2024-04-20T17:27:56.617 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.617 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adfc7cd4:::benchmark_data_smithi171_122772_object120894:head# 2024-04-20T17:27:56.617 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.617 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adfc99b1:::benchmark_data_smithi171_122772_object81086:head# 2024-04-20T17:27:56.617 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.617 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adfe16c2:::benchmark_data_smithi171_122772_object100919:head# 2024-04-20T17:27:56.618 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.618 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adfede68:::benchmark_data_smithi171_122772_object164734:head# 2024-04-20T17:27:56.618 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.618 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:adffa530:::benchmark_data_smithi171_122772_object91908:head# 2024-04-20T17:27:56.618 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.618 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae00ee2b:::benchmark_data_smithi171_122772_object149089:head# 2024-04-20T17:27:56.618 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.618 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae02ac55:::benchmark_data_smithi171_122772_object119326:head# 2024-04-20T17:27:56.618 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.618 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae02cf06:::benchmark_data_smithi171_122772_object146470:head# 2024-04-20T17:27:56.618 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.618 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae035598:::benchmark_data_smithi171_122772_object107095:head# 2024-04-20T17:27:56.619 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.619 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae0359c8:::benchmark_data_smithi171_122772_object120883:head# 2024-04-20T17:27:56.619 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.619 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae04781d:::benchmark_data_smithi171_122772_object120812:head# 2024-04-20T17:27:56.619 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.619 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae04bd94:::benchmark_data_smithi171_122772_object140584:head# 2024-04-20T17:27:56.619 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.619 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae04c247:::benchmark_data_smithi171_122772_object132034:head# 2024-04-20T17:27:56.619 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.619 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae052ec5:::benchmark_data_smithi171_122772_object166005:head# 2024-04-20T17:27:56.619 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.620 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae087e0c:::benchmark_data_smithi171_122772_object146588:head# 2024-04-20T17:27:56.620 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.620 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae0895c2:::benchmark_data_smithi171_122772_object155475:head# 2024-04-20T17:27:56.620 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.620 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae08adef:::benchmark_data_smithi171_122772_object122198:head# 2024-04-20T17:27:56.620 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.620 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae09a84c:::benchmark_data_smithi171_122772_object107926:head# 2024-04-20T17:27:56.620 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.620 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae0acf3b:::benchmark_data_smithi171_122772_object143045:head# 2024-04-20T17:27:56.620 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.620 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae0b2253:::benchmark_data_smithi171_122772_object113557:head# 2024-04-20T17:27:56.620 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.620 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae0b5ad7:::benchmark_data_smithi171_122772_object125130:head# 2024-04-20T17:27:56.620 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.621 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae0ba967:::benchmark_data_smithi171_122772_object135355:head# 2024-04-20T17:27:56.621 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.621 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae0c215e:::benchmark_data_smithi171_122772_object87393:head# 2024-04-20T17:27:56.621 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.621 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae0cb396:::benchmark_data_smithi171_122772_object135557:head# 2024-04-20T17:27:56.621 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.621 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae0d29a5:::benchmark_data_smithi171_122772_object119044:head# 2024-04-20T17:27:56.621 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.621 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae0e0606:::benchmark_data_smithi171_122772_object88410:head# 2024-04-20T17:27:56.621 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.621 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae0eeb73:::benchmark_data_smithi171_122772_object161226:head# 2024-04-20T17:27:56.621 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.621 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae0facb1:::benchmark_data_smithi171_122772_object81836:head# 2024-04-20T17:27:56.622 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.622 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae10821b:::benchmark_data_smithi171_122772_object152389:head# 2024-04-20T17:27:56.622 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.622 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae10c12e:::benchmark_data_smithi171_122772_object114479:head# 2024-04-20T17:27:56.622 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.622 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae11d3be:::benchmark_data_smithi171_122772_object164322:head# 2024-04-20T17:27:56.622 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.622 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae11e8e2:::benchmark_data_smithi171_122772_object140464:head# 2024-04-20T17:27:56.622 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.622 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae12a2bf:::benchmark_data_smithi171_122772_object123887:head# 2024-04-20T17:27:56.622 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.622 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae12a996:::benchmark_data_smithi171_122772_object105919:head# 2024-04-20T17:27:56.623 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.623 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae132234:::benchmark_data_smithi171_122772_object127073:head# 2024-04-20T17:27:56.623 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.623 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae1359ab:::benchmark_data_smithi171_122772_object89390:head# 2024-04-20T17:27:56.623 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.623 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae1431f1:::benchmark_data_smithi171_122772_object138204:head# 2024-04-20T17:27:56.623 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.623 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae1558d7:::benchmark_data_smithi171_122772_object78670:head# 2024-04-20T17:27:56.623 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.623 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae173bb9:::benchmark_data_smithi171_122772_object91742:head# 2024-04-20T17:27:56.623 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.623 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae17502d:::benchmark_data_smithi171_122772_object99627:head# 2024-04-20T17:27:56.624 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.624 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae17711a:::benchmark_data_smithi171_122772_object152091:head# 2024-04-20T17:27:56.624 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.624 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae191dbb:::benchmark_data_smithi171_122772_object158579:head# 2024-04-20T17:27:56.624 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.624 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae19aa1e:::benchmark_data_smithi171_122772_object167030:head# 2024-04-20T17:27:56.624 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.624 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae1b4a90:::benchmark_data_smithi171_122772_object125620:head# 2024-04-20T17:27:56.624 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.624 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae1bd489:::benchmark_data_smithi171_122772_object150928:head# 2024-04-20T17:27:56.624 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.624 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae1c0b4a:::benchmark_data_smithi171_122772_object78074:head# 2024-04-20T17:27:56.624 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.625 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae1c34b0:::benchmark_data_smithi171_122772_object88347:head# 2024-04-20T17:27:56.625 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.625 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae1c66c3:::benchmark_data_smithi171_122772_object125374:head# 2024-04-20T17:27:56.625 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.625 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae1c878f:::benchmark_data_smithi171_122772_object132599:head# 2024-04-20T17:27:56.625 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.625 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae1d9aa5:::benchmark_data_smithi171_122772_object162610:head# 2024-04-20T17:27:56.625 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.625 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae1e9b2e:::benchmark_data_smithi171_122772_object144001:head# 2024-04-20T17:27:56.625 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.625 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae211ab3:::benchmark_data_smithi171_122772_object92885:head# 2024-04-20T17:27:56.625 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.625 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae218af3:::benchmark_data_smithi171_122772_object110176:head# 2024-04-20T17:27:56.625 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.626 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae22bcd0:::benchmark_data_smithi171_122772_object135044:head# 2024-04-20T17:27:56.626 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.626 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae23d891:::benchmark_data_smithi171_122772_object97942:head# 2024-04-20T17:27:56.626 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.626 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae241d09:::benchmark_data_smithi171_122772_object144573:head# 2024-04-20T17:27:56.626 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.626 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae24891e:::benchmark_data_smithi171_122772_object117820:head# 2024-04-20T17:27:56.626 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.626 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae27a390:::benchmark_data_smithi171_122772_object82597:head# 2024-04-20T17:27:56.626 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.626 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae27cdd3:::benchmark_data_smithi171_122772_object123845:head# 2024-04-20T17:27:56.626 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.626 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae27f483:::benchmark_data_smithi171_122772_object131499:head# 2024-04-20T17:27:56.627 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.627 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae29412b:::benchmark_data_smithi171_122772_object105155:head# 2024-04-20T17:27:56.627 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.627 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae29d8ac:::benchmark_data_smithi171_122772_object153934:head# 2024-04-20T17:27:56.627 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.627 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae2a646b:::benchmark_data_smithi171_122772_object137581:head# 2024-04-20T17:27:56.627 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.627 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae2b90f7:::benchmark_data_smithi171_122772_object139473:head# 2024-04-20T17:27:56.627 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.627 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae2c7476:::benchmark_data_smithi171_122772_object120703:head# 2024-04-20T17:27:56.627 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.627 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae2d1912:::benchmark_data_smithi171_122772_object166446:head# 2024-04-20T17:27:56.627 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.627 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae2e2221:::benchmark_data_smithi171_122772_object95555:head# 2024-04-20T17:27:56.628 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.628 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae2eba69:::benchmark_data_smithi171_122772_object119891:head# 2024-04-20T17:27:56.628 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.628 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae308a67:::benchmark_data_smithi171_122772_object84651:head# 2024-04-20T17:27:56.628 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.628 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae31c387:::benchmark_data_smithi171_122772_object163617:head# 2024-04-20T17:27:56.628 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.628 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae326300:::benchmark_data_smithi171_122772_object111873:head# 2024-04-20T17:27:56.628 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.628 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae327bcc:::benchmark_data_smithi171_122772_object126183:head# 2024-04-20T17:27:56.628 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.628 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae328d98:::benchmark_data_smithi171_122772_object119083:head# 2024-04-20T17:27:56.628 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.628 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae33561e:::benchmark_data_smithi171_122772_object78355:head# 2024-04-20T17:27:56.629 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.629 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae3481c1:::benchmark_data_smithi171_122772_object128739:head# 2024-04-20T17:27:56.629 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.629 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae349dc9:::benchmark_data_smithi171_122772_object149355:head# 2024-04-20T17:27:56.629 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.629 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae34d4ea:::benchmark_data_smithi171_122772_object79914:head# 2024-04-20T17:27:56.629 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.629 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae35b7ab:::benchmark_data_smithi171_122772_object129106:head# 2024-04-20T17:27:56.629 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.629 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae35f0ac:::benchmark_data_smithi171_122772_object82467:head# 2024-04-20T17:27:56.629 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.629 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae3661a9:::benchmark_data_smithi171_122772_object82801:head# 2024-04-20T17:27:56.629 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.629 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae38fa0d:::benchmark_data_smithi171_122772_object90677:head# 2024-04-20T17:27:56.630 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.630 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae395466:::benchmark_data_smithi171_122772_object131482:head# 2024-04-20T17:27:56.630 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.630 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae39afb4:::benchmark_data_smithi171_122772_object112905:head# 2024-04-20T17:27:56.630 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.630 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae3adfc2:::benchmark_data_smithi171_122772_object161429:head# 2024-04-20T17:27:56.630 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.630 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae3b2eb9:::benchmark_data_smithi171_122772_object134266:head# 2024-04-20T17:27:56.630 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.630 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae3b70ea:::benchmark_data_smithi171_122772_object123312:head# 2024-04-20T17:27:56.630 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.630 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae3ccf45:::benchmark_data_smithi171_122772_object82685:head# 2024-04-20T17:27:56.630 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.631 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae3d056d:::benchmark_data_smithi171_122772_object91894:head# 2024-04-20T17:27:56.631 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.631 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae3d27a2:::benchmark_data_smithi171_122772_object139633:head# 2024-04-20T17:27:56.631 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.631 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae3dd9ac:::benchmark_data_smithi171_122772_object116825:head# 2024-04-20T17:27:56.631 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.631 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae3fac02:::benchmark_data_smithi171_122772_object111750:head# 2024-04-20T17:27:56.631 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.631 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae44c6d6:::benchmark_data_smithi171_122772_object135661:head# 2024-04-20T17:27:56.631 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.631 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae476210:::benchmark_data_smithi171_122772_object132949:head# 2024-04-20T17:27:56.631 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.631 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae4779a2:::benchmark_data_smithi171_122772_object143011:head# 2024-04-20T17:27:56.631 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.632 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae479fc3:::benchmark_data_smithi171_122772_object109019:head# 2024-04-20T17:27:56.632 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.632 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae47dfe6:::benchmark_data_smithi171_122772_object87648:head# 2024-04-20T17:27:56.632 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.632 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae47e041:::benchmark_data_smithi171_122772_object108458:head# 2024-04-20T17:27:56.632 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.632 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae47e7f4:::benchmark_data_smithi171_122772_object144703:head# 2024-04-20T17:27:56.632 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.632 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae48fbf0:::benchmark_data_smithi171_122772_object126618:head# 2024-04-20T17:27:56.632 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.632 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae4b3d08:::benchmark_data_smithi171_122772_object115466:head# 2024-04-20T17:27:56.633 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.633 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae4c85a0:::benchmark_data_smithi171_122772_object150611:head# 2024-04-20T17:27:56.633 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.633 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae4d43bf:::benchmark_data_smithi171_122772_object154712:head# 2024-04-20T17:27:56.633 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.633 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae4d66d4:::benchmark_data_smithi171_122772_object129468:head# 2024-04-20T17:27:56.633 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.633 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae4dbbcc:::benchmark_data_smithi171_122772_object90890:head# 2024-04-20T17:27:56.633 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.633 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae4e1995:::benchmark_data_smithi171_122772_object121021:head# 2024-04-20T17:27:56.633 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.633 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae4eb065:::benchmark_data_smithi171_122772_object118835:head# 2024-04-20T17:27:56.634 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.634 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae4f01c6:::benchmark_data_smithi171_122772_object97940:head# 2024-04-20T17:27:56.634 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.634 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae4f9f8c:::benchmark_data_smithi171_122772_object132772:head# 2024-04-20T17:27:56.634 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.634 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae50373c:::benchmark_data_smithi171_122772_object164851:head# 2024-04-20T17:27:56.634 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.634 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae506765:::benchmark_data_smithi171_122772_object161283:head# 2024-04-20T17:27:56.634 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.634 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae51220b:::benchmark_data_smithi171_122772_object162338:head# 2024-04-20T17:27:56.635 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.635 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae521f02:::benchmark_data_smithi171_122772_object148063:head# 2024-04-20T17:27:56.635 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.635 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae52268d:::benchmark_data_smithi171_122772_object83994:head# 2024-04-20T17:27:56.635 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.635 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae5270cc:::benchmark_data_smithi171_122772_object165814:head# 2024-04-20T17:27:56.635 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.635 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae52a3e3:::benchmark_data_smithi171_122772_object83605:head# 2024-04-20T17:27:56.635 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.635 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae52a891:::benchmark_data_smithi171_122772_object165489:head# 2024-04-20T17:27:56.635 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.635 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae531d73:::benchmark_data_smithi171_122772_object113811:head# 2024-04-20T17:27:56.636 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.636 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae53845e:::benchmark_data_smithi171_122772_object160719:head# 2024-04-20T17:27:56.636 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.636 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae54307c:::benchmark_data_smithi171_122772_object158534:head# 2024-04-20T17:27:56.636 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.636 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae546772:::benchmark_data_smithi171_122772_object124533:head# 2024-04-20T17:27:56.636 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.636 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae54dd17:::benchmark_data_smithi171_122772_object130186:head# 2024-04-20T17:27:56.636 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.636 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae54f9fd:::benchmark_data_smithi171_122772_object163987:head# 2024-04-20T17:27:56.636 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.636 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae54ffa9:::benchmark_data_smithi171_122772_object101578:head# 2024-04-20T17:27:56.636 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.637 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae55dcd3:::benchmark_data_smithi171_122772_object116495:head# 2024-04-20T17:27:56.637 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.637 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae563f55:::benchmark_data_smithi171_122772_object96012:head# 2024-04-20T17:27:56.637 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.637 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae56be15:::benchmark_data_smithi171_122772_object94274:head# 2024-04-20T17:27:56.637 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.637 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae56c8e8:::benchmark_data_smithi171_122772_object139431:head# 2024-04-20T17:27:56.637 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.637 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae574281:::benchmark_data_smithi171_122772_object170070:head# 2024-04-20T17:27:56.637 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.637 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae57cbfc:::benchmark_data_smithi171_122772_object83361:head# 2024-04-20T17:27:56.637 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.638 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae57dc38:::benchmark_data_smithi171_122772_object78720:head# 2024-04-20T17:27:56.638 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.638 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae58be36:::benchmark_data_smithi171_122772_object104639:head# 2024-04-20T17:27:56.638 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.638 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae58d959:::benchmark_data_smithi171_122772_object109576:head# 2024-04-20T17:27:56.638 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.638 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae59177e:::benchmark_data_smithi171_122772_object77984:head# 2024-04-20T17:27:56.638 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.638 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae5917cb:::benchmark_data_smithi171_122772_object133915:head# 2024-04-20T17:27:56.638 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.638 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae592df1:::benchmark_data_smithi171_122772_object112290:head# 2024-04-20T17:27:56.639 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.639 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae59dde0:::benchmark_data_smithi171_122772_object98548:head# 2024-04-20T17:27:56.639 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.639 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae5a9222:::benchmark_data_smithi171_122772_object118578:head# 2024-04-20T17:27:56.639 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.639 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae5c0900:::benchmark_data_smithi171_122772_object145952:head# 2024-04-20T17:27:56.639 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.639 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae5cb951:::benchmark_data_smithi171_122772_object138097:head# 2024-04-20T17:27:56.639 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.639 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae5cbe1b:::benchmark_data_smithi171_122772_object163314:head# 2024-04-20T17:27:56.639 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.639 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae5d0ab6:::benchmark_data_smithi171_122772_object162046:head# 2024-04-20T17:27:56.639 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.640 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae5f429f:::benchmark_data_smithi171_122772_object106776:head# 2024-04-20T17:27:56.640 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.640 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae608d47:::benchmark_data_smithi171_122772_object129296:head# 2024-04-20T17:27:56.640 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.640 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae60cb36:::benchmark_data_smithi171_122772_object166636:head# 2024-04-20T17:27:56.640 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.640 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae61b41c:::benchmark_data_smithi171_122772_object164748:head# 2024-04-20T17:27:56.640 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.640 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae625675:::benchmark_data_smithi171_122772_object112174:head# 2024-04-20T17:27:56.640 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.640 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae62a4a0:::benchmark_data_smithi171_122772_object146486:head# 2024-04-20T17:27:56.640 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.640 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae631662:::benchmark_data_smithi171_122772_object97146:head# 2024-04-20T17:27:56.640 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.641 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae63fb81:::benchmark_data_smithi171_122772_object80461:head# 2024-04-20T17:27:56.641 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.641 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae64465b:::benchmark_data_smithi171_122772_object115849:head# 2024-04-20T17:27:56.641 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.641 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae64702f:::benchmark_data_smithi171_122772_object85907:head# 2024-04-20T17:27:56.641 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.641 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae647b0d:::benchmark_data_smithi171_122772_object148264:head# 2024-04-20T17:27:56.641 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.641 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae64921c:::benchmark_data_smithi171_122772_object82044:head# 2024-04-20T17:27:56.641 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.641 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae65b6bd:::benchmark_data_smithi171_122772_object133363:head# 2024-04-20T17:27:56.641 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.641 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae65fd84:::benchmark_data_smithi171_122772_object140157:head# 2024-04-20T17:27:56.641 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.642 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae661287:::benchmark_data_smithi171_122772_object146271:head# 2024-04-20T17:27:56.642 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.642 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae661911:::benchmark_data_smithi171_122772_object103716:head# 2024-04-20T17:27:56.642 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.642 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae66283d:::benchmark_data_smithi171_122772_object98848:head# 2024-04-20T17:27:56.642 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.642 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae670e04:::benchmark_data_smithi171_122772_object89875:head# 2024-04-20T17:27:56.642 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.642 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae67a399:::benchmark_data_smithi171_122772_object151292:head# 2024-04-20T17:27:56.642 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.642 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae67e2a4:::benchmark_data_smithi171_122772_object94527:head# 2024-04-20T17:27:56.642 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.642 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae6948eb:::benchmark_data_smithi171_122772_object162132:head# 2024-04-20T17:27:56.643 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.643 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae69b04f:::benchmark_data_smithi171_122772_object128624:head# 2024-04-20T17:27:56.643 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.643 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae6c96f9:::benchmark_data_smithi171_122772_object95478:head# 2024-04-20T17:27:56.643 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.643 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae6dda13:::benchmark_data_smithi171_122772_object88797:head# 2024-04-20T17:27:56.643 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.643 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae6ed4db:::benchmark_data_smithi171_122772_object105884:head# 2024-04-20T17:27:56.643 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.643 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae6f26ac:::benchmark_data_smithi171_122772_object99977:head# 2024-04-20T17:27:56.643 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.643 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae6f3f1a:::benchmark_data_smithi171_122772_object168493:head# 2024-04-20T17:27:56.643 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.643 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae707080:::benchmark_data_smithi171_122772_object99694:head# 2024-04-20T17:27:56.644 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.644 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae70c378:::benchmark_data_smithi171_122772_object114348:head# 2024-04-20T17:27:56.644 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.644 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae70d770:::benchmark_data_smithi171_122772_object144775:head# 2024-04-20T17:27:56.644 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.644 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae71ac70:::benchmark_data_smithi171_122772_object149659:head# 2024-04-20T17:27:56.644 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.644 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae7255c0:::benchmark_data_smithi171_122772_object163386:head# 2024-04-20T17:27:56.644 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.644 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae732b29:::benchmark_data_smithi171_122772_object165071:head# 2024-04-20T17:27:56.644 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.644 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae750434:::benchmark_data_smithi171_122772_object87623:head# 2024-04-20T17:27:56.645 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.645 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae75be6c:::benchmark_data_smithi171_122772_object139223:head# 2024-04-20T17:27:56.645 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.645 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae780848:::benchmark_data_smithi171_122772_object122485:head# 2024-04-20T17:27:56.645 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.645 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae781a40:::benchmark_data_smithi171_122772_object122124:head# 2024-04-20T17:27:56.645 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.645 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae78d71d:::benchmark_data_smithi171_122772_object131569:head# 2024-04-20T17:27:56.645 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.645 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae79ef1a:::benchmark_data_smithi171_122772_object168589:head# 2024-04-20T17:27:56.645 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.646 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae7ccf5e:::benchmark_data_smithi171_122772_object119380:head# 2024-04-20T17:27:56.646 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.646 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae7d3b7a:::benchmark_data_smithi171_122772_object113166:head# 2024-04-20T17:27:56.646 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.646 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae7daf63:::benchmark_data_smithi171_122772_object118409:head# 2024-04-20T17:27:56.646 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.646 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae7dd5b1:::benchmark_data_smithi171_122772_object154470:head# 2024-04-20T17:27:56.646 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.646 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae7f13da:::benchmark_data_smithi171_122772_object101818:head# 2024-04-20T17:27:56.646 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.646 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae7f63b2:::benchmark_data_smithi171_122772_object133986:head# 2024-04-20T17:27:56.646 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.646 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae80d9bc:::benchmark_data_smithi171_122772_object141353:head# 2024-04-20T17:27:56.647 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.647 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae80f909:::benchmark_data_smithi171_122772_object133632:head# 2024-04-20T17:27:56.647 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.647 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae8348fb:::benchmark_data_smithi171_122772_object109545:head# 2024-04-20T17:27:56.647 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.647 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae83b3ed:::benchmark_data_smithi171_122772_object89817:head# 2024-04-20T17:27:56.647 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.647 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae846493:::benchmark_data_smithi171_122772_object104199:head# 2024-04-20T17:27:56.647 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.647 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae8640d4:::benchmark_data_smithi171_122772_object107415:head# 2024-04-20T17:27:56.647 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.647 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae867a3f:::benchmark_data_smithi171_122772_object152739:head# 2024-04-20T17:27:56.647 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.647 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae86e380:::benchmark_data_smithi171_122772_object80252:head# 2024-04-20T17:27:56.648 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.648 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae87906e:::benchmark_data_smithi171_122772_object168594:head# 2024-04-20T17:27:56.648 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.648 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae87a5da:::benchmark_data_smithi171_122772_object123383:head# 2024-04-20T17:27:56.648 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.648 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae88fbac:::benchmark_data_smithi171_122772_object168991:head# 2024-04-20T17:27:56.648 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.648 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae896ab7:::benchmark_data_smithi171_122772_object148645:head# 2024-04-20T17:27:56.648 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.648 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae8a1174:::benchmark_data_smithi171_122772_object123380:head# 2024-04-20T17:27:56.648 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.648 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae8a7670:::benchmark_data_smithi171_122772_object108176:head# 2024-04-20T17:27:56.648 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.649 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae8b07bc:::benchmark_data_smithi171_122772_object158052:head# 2024-04-20T17:27:56.649 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.649 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae8c4c93:::benchmark_data_smithi171_122772_object136336:head# 2024-04-20T17:27:56.649 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.649 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae8ca826:::benchmark_data_smithi171_122772_object159906:head# 2024-04-20T17:27:56.649 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.649 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae8dfd54:::benchmark_data_smithi171_122772_object97229:head# 2024-04-20T17:27:56.649 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.649 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae8f5c34:::benchmark_data_smithi171_122772_object154676:head# 2024-04-20T17:27:56.649 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.649 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae8fa36c:::benchmark_data_smithi171_122772_object149738:head# 2024-04-20T17:27:56.649 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.649 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae8fac54:::benchmark_data_smithi171_122772_object95273:head# 2024-04-20T17:27:56.650 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.650 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae8fc797:::benchmark_data_smithi171_122772_object133205:head# 2024-04-20T17:27:56.650 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.650 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae90092a:::benchmark_data_smithi171_122772_object161051:head# 2024-04-20T17:27:56.650 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.650 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae901142:::benchmark_data_smithi171_122772_object157035:head# 2024-04-20T17:27:56.650 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.650 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae906cfd:::benchmark_data_smithi171_122772_object140867:head# 2024-04-20T17:27:56.650 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.650 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae91112b:::benchmark_data_smithi171_122772_object162332:head# 2024-04-20T17:27:56.650 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.650 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae916ef2:::benchmark_data_smithi171_122772_object145255:head# 2024-04-20T17:27:56.650 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.651 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae91a480:::benchmark_data_smithi171_122772_object141266:head# 2024-04-20T17:27:56.651 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.651 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae93e39c:::benchmark_data_smithi171_122772_object78627:head# 2024-04-20T17:27:56.651 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.651 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae953d5b:::benchmark_data_smithi171_122772_object164005:head# 2024-04-20T17:27:56.651 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.651 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae9593f6:::benchmark_data_smithi171_122772_object100624:head# 2024-04-20T17:27:56.651 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.651 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae95a861:::benchmark_data_smithi171_122772_object144999:head# 2024-04-20T17:27:56.651 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.651 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae96aa37:::benchmark_data_smithi171_122772_object116302:head# 2024-04-20T17:27:56.651 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.651 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae97d67d:::benchmark_data_smithi171_122772_object161023:head# 2024-04-20T17:27:56.652 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.652 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae99e783:::benchmark_data_smithi171_122772_object84951:head# 2024-04-20T17:27:56.652 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.652 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae9ab1d9:::benchmark_data_smithi171_122772_object146955:head# 2024-04-20T17:27:56.652 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.652 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae9af7f2:::benchmark_data_smithi171_122772_object101260:head# 2024-04-20T17:27:56.652 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.652 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae9afbd8:::benchmark_data_smithi171_122772_object107507:head# 2024-04-20T17:27:56.652 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.652 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae9bbd98:::benchmark_data_smithi171_122772_object158194:head# 2024-04-20T17:27:56.652 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.652 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae9c0033:::benchmark_data_smithi171_122772_object151239:head# 2024-04-20T17:27:56.652 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.652 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae9c0def:::benchmark_data_smithi171_122772_object96269:head# 2024-04-20T17:27:56.653 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.653 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae9ced93:::benchmark_data_smithi171_122772_object140070:head# 2024-04-20T17:27:56.653 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.653 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae9d282e:::benchmark_data_smithi171_122772_object128181:head# 2024-04-20T17:27:56.653 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.653 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae9d3eda:::benchmark_data_smithi171_122772_object129439:head# 2024-04-20T17:27:56.653 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.653 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae9ea756:::benchmark_data_smithi171_122772_object159758:head# 2024-04-20T17:27:56.653 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.653 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae9faeec:::benchmark_data_smithi171_122772_object97120:head# 2024-04-20T17:27:56.653 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.653 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ae9fe3d4:::benchmark_data_smithi171_122772_object110701:head# 2024-04-20T17:27:56.654 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.654 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aea25b23:::benchmark_data_smithi171_122772_object91116:head# 2024-04-20T17:27:56.654 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.654 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aea2df9c:::benchmark_data_smithi171_122772_object92651:head# 2024-04-20T17:27:56.654 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.654 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aea375a3:::benchmark_data_smithi171_122772_object129177:head# 2024-04-20T17:27:56.654 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.654 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aea3ebc6:::benchmark_data_smithi171_122772_object115295:head# 2024-04-20T17:27:56.654 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.654 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aea43016:::benchmark_data_smithi171_122772_object114162:head# 2024-04-20T17:27:56.654 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.654 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aea52c71:::benchmark_data_smithi171_122772_object105101:head# 2024-04-20T17:27:56.655 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.655 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aea5ded9:::benchmark_data_smithi171_122772_object128602:head# 2024-04-20T17:27:56.655 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.655 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aea6f5b5:::benchmark_data_smithi171_122772_object89220:head# 2024-04-20T17:27:56.655 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.655 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aea7bbf2:::benchmark_data_smithi171_122772_object151855:head# 2024-04-20T17:27:56.655 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.655 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aea832ff:::benchmark_data_smithi171_122772_object115346:head# 2024-04-20T17:27:56.655 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.655 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aea91af4:::benchmark_data_smithi171_122772_object90003:head# 2024-04-20T17:27:56.655 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.655 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aea971d7:::benchmark_data_smithi171_122772_object155002:head# 2024-04-20T17:27:56.656 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.656 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeaabefc:::benchmark_data_smithi171_122772_object109260:head# 2024-04-20T17:27:56.656 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.656 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeaaebdd:::benchmark_data_smithi171_122772_object123225:head# 2024-04-20T17:27:56.656 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.656 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeab1c4e:::benchmark_data_smithi171_122772_object98089:head# 2024-04-20T17:27:56.656 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.656 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeada183:::benchmark_data_smithi171_122772_object119058:head# 2024-04-20T17:27:56.656 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.656 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeadd269:::benchmark_data_smithi171_122772_object136584:head# 2024-04-20T17:27:56.656 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.656 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeae8dfa:::benchmark_data_smithi171_122772_object104900:head# 2024-04-20T17:27:56.657 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.657 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeafb5b9:::benchmark_data_smithi171_122772_object166426:head# 2024-04-20T17:27:56.657 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.657 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeb11dd8:::benchmark_data_smithi171_122772_object80822:head# 2024-04-20T17:27:56.657 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.657 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeb14435:::benchmark_data_smithi171_122772_object165467:head# 2024-04-20T17:27:56.657 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.657 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeb1e7d8:::benchmark_data_smithi171_122772_object168426:head# 2024-04-20T17:27:56.657 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.657 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeb291a6:::benchmark_data_smithi171_122772_object133088:head# 2024-04-20T17:27:56.657 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.657 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeb359a3:::benchmark_data_smithi171_122772_object155253:head# 2024-04-20T17:27:56.657 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.657 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeb39555:::benchmark_data_smithi171_122772_object93776:head# 2024-04-20T17:27:56.658 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.658 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeb4d6a0:::benchmark_data_smithi171_122772_object166017:head# 2024-04-20T17:27:56.658 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.658 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeb6cbba:::benchmark_data_smithi171_122772_object125331:head# 2024-04-20T17:27:56.658 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.658 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeb7172d:::benchmark_data_smithi171_122772_object97938:head# 2024-04-20T17:27:56.658 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.658 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeb9138a:::benchmark_data_smithi171_122772_object97253:head# 2024-04-20T17:27:56.658 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.658 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeb91419:::benchmark_data_smithi171_122772_object94392:head# 2024-04-20T17:27:56.658 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.658 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aebc407f:::benchmark_data_smithi171_122772_object78070:head# 2024-04-20T17:27:56.658 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.659 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aebc53ab:::benchmark_data_smithi171_122772_object142889:head# 2024-04-20T17:27:56.659 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.659 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aebc6713:::benchmark_data_smithi171_122772_object144989:head# 2024-04-20T17:27:56.659 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.659 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aebcd1ed:::benchmark_data_smithi171_122772_object162169:head# 2024-04-20T17:27:56.659 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.659 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aebd8ad2:::benchmark_data_smithi171_122772_object124493:head# 2024-04-20T17:27:56.659 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.659 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aebe6244:::benchmark_data_smithi171_122772_object122503:head# 2024-04-20T17:27:56.659 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.659 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aebe8fb6:::benchmark_data_smithi171_122772_object145321:head# 2024-04-20T17:27:56.659 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.659 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aebecbaf:::benchmark_data_smithi171_122772_object107177:head# 2024-04-20T17:27:56.659 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.660 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aebf5f14:::benchmark_data_smithi171_122772_object135596:head# 2024-04-20T17:27:56.660 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.660 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aebf813d:::benchmark_data_smithi171_122772_object101670:head# 2024-04-20T17:27:56.660 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.660 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aec06376:::benchmark_data_smithi171_122772_object101227:head# 2024-04-20T17:27:56.660 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.660 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aec0660c:::benchmark_data_smithi171_122772_object117977:head# 2024-04-20T17:27:56.660 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.660 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aec17987:::benchmark_data_smithi171_122772_object120368:head# 2024-04-20T17:27:56.660 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.660 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aec245e9:::benchmark_data_smithi171_122772_object104778:head# 2024-04-20T17:27:56.660 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.661 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aec28e78:::benchmark_data_smithi171_122772_object108631:head# 2024-04-20T17:27:56.661 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.661 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aec313de:::benchmark_data_smithi171_122772_object151661:head# 2024-04-20T17:27:56.661 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.661 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aec36963:::benchmark_data_smithi171_122772_object122943:head# 2024-04-20T17:27:56.661 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.661 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aec408a2:::benchmark_data_smithi171_122772_object153425:head# 2024-04-20T17:27:56.661 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.661 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aec57ed8:::benchmark_data_smithi171_122772_object109584:head# 2024-04-20T17:27:56.661 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.661 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aec5b309:::benchmark_data_smithi171_122772_object166081:head# 2024-04-20T17:27:56.661 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.662 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aec6244f:::benchmark_data_smithi171_122772_object145677:head# 2024-04-20T17:27:56.662 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.662 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aec848ae:::benchmark_data_smithi171_122772_object118695:head# 2024-04-20T17:27:56.662 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.662 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aec8a87a:::benchmark_data_smithi171_122772_object141043:head# 2024-04-20T17:27:56.662 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.662 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aec9133c:::benchmark_data_smithi171_122772_object125958:head# 2024-04-20T17:27:56.662 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.662 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aecac376:::benchmark_data_smithi171_122772_object139519:head# 2024-04-20T17:27:56.662 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.662 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aecb215f:::benchmark_data_smithi171_122772_object140996:head# 2024-04-20T17:27:56.662 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.662 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aecb5ab7:::benchmark_data_smithi171_122772_object145442:head# 2024-04-20T17:27:56.663 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.663 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aecce966:::benchmark_data_smithi171_122772_object138979:head# 2024-04-20T17:27:56.663 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.663 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aecd7564:::benchmark_data_smithi171_122772_object87344:head# 2024-04-20T17:27:56.663 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.663 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aecd8db1:::benchmark_data_smithi171_122772_object84828:head# 2024-04-20T17:27:56.663 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.663 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aecdf76a:::benchmark_data_smithi171_122772_object108077:head# 2024-04-20T17:27:56.663 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.663 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aecdfcb7:::benchmark_data_smithi171_122772_object92836:head# 2024-04-20T17:27:56.663 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.663 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeceb341:::benchmark_data_smithi171_122772_object146641:head# 2024-04-20T17:27:56.663 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.663 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aecf0d22:::benchmark_data_smithi171_122772_object135548:head# 2024-04-20T17:27:56.664 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.664 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aecf7468:::benchmark_data_smithi171_122772_object87430:head# 2024-04-20T17:27:56.664 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.664 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aecf9d35:::benchmark_data_smithi171_122772_object84919:head# 2024-04-20T17:27:56.664 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.664 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aecfeebd:::benchmark_data_smithi171_122772_object140551:head# 2024-04-20T17:27:56.664 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.664 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aecff2bc:::benchmark_data_smithi171_122772_object124393:head# 2024-04-20T17:27:56.664 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.664 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aed01af9:::benchmark_data_smithi171_122772_object157213:head# 2024-04-20T17:27:56.664 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.664 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aed05aa7:::benchmark_data_smithi171_122772_object117275:head# 2024-04-20T17:27:56.665 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.665 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aed19578:::benchmark_data_smithi171_122772_object95453:head# 2024-04-20T17:27:56.665 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.665 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aed265d1:::benchmark_data_smithi171_122772_object105685:head# 2024-04-20T17:27:56.665 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.665 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aed513ba:::benchmark_data_smithi171_122772_object100761:head# 2024-04-20T17:27:56.665 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.665 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aed6c1a6:::benchmark_data_smithi171_122772_object126350:head# 2024-04-20T17:27:56.665 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.665 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aed7efcb:::benchmark_data_smithi171_122772_object124362:head# 2024-04-20T17:27:56.665 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.665 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aed89cd5:::benchmark_data_smithi171_122772_object160138:head# 2024-04-20T17:27:56.666 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.666 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aed8b52b:::benchmark_data_smithi171_122772_object161600:head# 2024-04-20T17:27:56.666 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.666 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aed8c7fb:::benchmark_data_smithi171_122772_object108947:head# 2024-04-20T17:27:56.666 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.666 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aed8f658:::benchmark_data_smithi171_122772_object151491:head# 2024-04-20T17:27:56.666 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.666 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aed94509:::benchmark_data_smithi171_122772_object151076:head# 2024-04-20T17:27:56.666 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.666 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeda2ef8:::benchmark_data_smithi171_122772_object85427:head# 2024-04-20T17:27:56.666 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.667 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeda7b65:::benchmark_data_smithi171_122772_object104140:head# 2024-04-20T17:27:56.667 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.667 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aedb1ecb:::benchmark_data_smithi171_122772_object125483:head# 2024-04-20T17:27:56.667 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.667 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aedd85be:::benchmark_data_smithi171_122772_object88775:head# 2024-04-20T17:27:56.667 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.667 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeddb420:::benchmark_data_smithi171_122772_object156890:head# 2024-04-20T17:27:56.667 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.667 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeddcf6d:::benchmark_data_smithi171_122772_object149857:head# 2024-04-20T17:27:56.667 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.667 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aede7ac6:::benchmark_data_smithi171_122772_object145898:head# 2024-04-20T17:27:56.667 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.668 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aedf5bc9:::benchmark_data_smithi171_122772_object166884:head# 2024-04-20T17:27:56.668 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.668 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aee00701:::benchmark_data_smithi171_122772_object169038:head# 2024-04-20T17:27:56.668 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.668 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aee05389:::benchmark_data_smithi171_122772_object159540:head# 2024-04-20T17:27:56.668 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.668 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aee11f83:::benchmark_data_smithi171_122772_object169719:head# 2024-04-20T17:27:56.668 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.668 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aee160e7:::benchmark_data_smithi171_122772_object113327:head# 2024-04-20T17:27:56.668 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.668 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aee2264e:::benchmark_data_smithi171_122772_object93331:head# 2024-04-20T17:27:56.668 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.668 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aee3b069:::benchmark_data_smithi171_122772_object103858:head# 2024-04-20T17:27:56.669 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.669 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aee41420:::benchmark_data_smithi171_122772_object92479:head# 2024-04-20T17:27:56.669 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.669 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aee5314f:::benchmark_data_smithi171_122772_object144937:head# 2024-04-20T17:27:56.669 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.669 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aee5e264:::benchmark_data_smithi171_122772_object140491:head# 2024-04-20T17:27:56.669 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.669 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aee68573:::benchmark_data_smithi171_122772_object107859:head# 2024-04-20T17:27:56.669 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.669 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aee7b905:::benchmark_data_smithi171_122772_object89184:head# 2024-04-20T17:27:56.669 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.669 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aee7f675:::benchmark_data_smithi171_122772_object79596:head# 2024-04-20T17:27:56.669 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.670 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aee84150:::benchmark_data_smithi171_122772_object86777:head# 2024-04-20T17:27:56.670 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.670 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aee96b2c:::benchmark_data_smithi171_122772_object84120:head# 2024-04-20T17:27:56.670 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.670 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aee9968a:::benchmark_data_smithi171_122772_object94414:head# 2024-04-20T17:27:56.670 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.670 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aee9fa0f:::benchmark_data_smithi171_122772_object162811:head# 2024-04-20T17:27:56.670 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.670 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeebd7f5:::benchmark_data_smithi171_122772_object127750:head# 2024-04-20T17:27:56.670 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.670 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeec95d5:::benchmark_data_smithi171_122772_object143309:head# 2024-04-20T17:27:56.670 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.671 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeed53ae:::benchmark_data_smithi171_122772_object165040:head# 2024-04-20T17:27:56.671 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.671 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeed9a7a:::benchmark_data_smithi171_122772_object106239:head# 2024-04-20T17:27:56.671 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.671 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeef5f1a:::benchmark_data_smithi171_122772_object126714:head# 2024-04-20T17:27:56.671 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.671 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeefca1f:::benchmark_data_smithi171_122772_object99062:head# 2024-04-20T17:27:56.671 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.671 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aef02110:::benchmark_data_smithi171_122772_object107961:head# 2024-04-20T17:27:56.671 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.671 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aef0a46f:::benchmark_data_smithi171_122772_object160555:head# 2024-04-20T17:27:56.671 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.672 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aef100f6:::benchmark_data_smithi171_122772_object163696:head# 2024-04-20T17:27:56.672 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.672 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aef12d3a:::benchmark_data_smithi171_122772_object91685:head# 2024-04-20T17:27:56.672 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.672 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aef14993:::benchmark_data_smithi171_122772_object118242:head# 2024-04-20T17:27:56.672 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.672 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aef14ad2:::benchmark_data_smithi171_122772_object150112:head# 2024-04-20T17:27:56.672 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.672 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aef200f3:::benchmark_data_smithi171_122772_object141853:head# 2024-04-20T17:27:56.672 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.672 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aef203d2:::benchmark_data_smithi171_122772_object126104:head# 2024-04-20T17:27:56.672 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.672 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aef24b32:::benchmark_data_smithi171_122772_object88895:head# 2024-04-20T17:27:56.673 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.673 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aef2505b:::benchmark_data_smithi171_122772_object125263:head# 2024-04-20T17:27:56.673 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.673 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aef27639:::benchmark_data_smithi171_122772_object86121:head# 2024-04-20T17:27:56.673 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.673 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aef29fb6:::benchmark_data_smithi171_122772_object116675:head# 2024-04-20T17:27:56.673 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.673 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aef3a8e2:::benchmark_data_smithi171_122772_object81461:head# 2024-04-20T17:27:56.673 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.673 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aef44cf0:::benchmark_data_smithi171_122772_object155376:head# 2024-04-20T17:27:56.673 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.673 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aef529ce:::benchmark_data_smithi171_122772_object104800:head# 2024-04-20T17:27:56.674 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.674 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aef54c93:::benchmark_data_smithi171_122772_object152008:head# 2024-04-20T17:27:56.674 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.674 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aef54ef8:::benchmark_data_smithi171_122772_object95978:head# 2024-04-20T17:27:56.674 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.674 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aef7191f:::benchmark_data_smithi171_122772_object106384:head# 2024-04-20T17:27:56.674 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.674 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aef783ce:::benchmark_data_smithi171_122772_object167469:head# 2024-04-20T17:27:56.674 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.674 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aef7caf5:::benchmark_data_smithi171_122772_object122282:head# 2024-04-20T17:27:56.674 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.674 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aef7d990:::benchmark_data_smithi171_122772_object131461:head# 2024-04-20T17:27:56.675 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.675 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aef82975:::benchmark_data_smithi171_122772_object116903:head# 2024-04-20T17:27:56.675 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.675 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aef8612d:::benchmark_data_smithi171_122772_object95996:head# 2024-04-20T17:27:56.675 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.675 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aef87a6a:::benchmark_data_smithi171_122772_object167210:head# 2024-04-20T17:27:56.675 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.675 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aef96823:::benchmark_data_smithi171_122772_object164092:head# 2024-04-20T17:27:56.675 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.675 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aefadfe6:::benchmark_data_smithi171_122772_object104597:head# 2024-04-20T17:27:56.675 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.675 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aefb7e2b:::benchmark_data_smithi171_122772_object112729:head# 2024-04-20T17:27:56.676 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.676 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aefeb281:::benchmark_data_smithi171_122772_object106182:head# 2024-04-20T17:27:56.676 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.676 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aeff8d06:::benchmark_data_smithi171_122772_object159429:head# 2024-04-20T17:27:56.676 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.676 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af001f77:::benchmark_data_smithi171_122772_object106905:head# 2024-04-20T17:27:56.676 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.676 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af00388b:::benchmark_data_smithi171_122772_object160413:head# 2024-04-20T17:27:56.676 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.676 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af02f028:::benchmark_data_smithi171_122772_object107983:head# 2024-04-20T17:27:56.676 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.676 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af0324da:::benchmark_data_smithi171_122772_object142101:head# 2024-04-20T17:27:56.676 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.677 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af049a8c:::benchmark_data_smithi171_122772_object116099:head# 2024-04-20T17:27:56.677 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.677 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af04a2e6:::benchmark_data_smithi171_122772_object147140:head# 2024-04-20T17:27:56.677 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.677 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af04ff70:::benchmark_data_smithi171_122772_object102001:head# 2024-04-20T17:27:56.677 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.677 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af055523:::benchmark_data_smithi171_122772_object87642:head# 2024-04-20T17:27:56.677 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.677 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af0594a8:::benchmark_data_smithi171_122772_object104086:head# 2024-04-20T17:27:56.677 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.677 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af0640a0:::benchmark_data_smithi171_122772_object93855:head# 2024-04-20T17:27:56.677 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.677 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af077c1d:::benchmark_data_smithi171_122772_object126736:head# 2024-04-20T17:27:56.678 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.678 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af078236:::benchmark_data_smithi171_122772_object136930:head# 2024-04-20T17:27:56.678 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.678 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af07c4ee:::benchmark_data_smithi171_122772_object99048:head# 2024-04-20T17:27:56.678 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.678 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af085405:::benchmark_data_smithi171_122772_object165121:head# 2024-04-20T17:27:56.678 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.678 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af0aa6d8:::benchmark_data_smithi171_122772_object166456:head# 2024-04-20T17:27:56.678 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.678 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af0af0c4:::benchmark_data_smithi171_122772_object115787:head# 2024-04-20T17:27:56.678 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.679 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af0b0a00:::benchmark_data_smithi171_122772_object91603:head# 2024-04-20T17:27:56.679 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.679 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af0b8d18:::benchmark_data_smithi171_122772_object118137:head# 2024-04-20T17:27:56.679 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.679 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af0c3b42:::benchmark_data_smithi171_122772_object136816:head# 2024-04-20T17:27:56.679 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.679 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af0c42c4:::benchmark_data_smithi171_122772_object115402:head# 2024-04-20T17:27:56.679 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.679 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af0c6a9e:::benchmark_data_smithi171_122772_object148924:head# 2024-04-20T17:27:56.679 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.679 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af0c9490:::benchmark_data_smithi171_122772_object139724:head# 2024-04-20T17:27:56.679 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.679 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af0cf3af:::benchmark_data_smithi171_122772_object114171:head# 2024-04-20T17:27:56.680 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.680 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af0dccae:::benchmark_data_smithi171_122772_object156991:head# 2024-04-20T17:27:56.680 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.680 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af0ef4e9:::benchmark_data_smithi171_122772_object152365:head# 2024-04-20T17:27:56.680 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.680 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af0f4d2d:::benchmark_data_smithi171_122772_object79982:head# 2024-04-20T17:27:56.680 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.680 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af0f92da:::benchmark_data_smithi171_122772_object152286:head# 2024-04-20T17:27:56.680 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.680 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af0fcaa1:::benchmark_data_smithi171_122772_object126635:head# 2024-04-20T17:27:56.680 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.680 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af0feb55:::benchmark_data_smithi171_122772_object92639:head# 2024-04-20T17:27:56.680 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.681 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af10bcbd:::benchmark_data_smithi171_122772_object126291:head# 2024-04-20T17:27:56.681 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.681 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af10bd42:::benchmark_data_smithi171_122772_object168894:head# 2024-04-20T17:27:56.681 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.681 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af111d1f:::benchmark_data_smithi171_122772_object113177:head# 2024-04-20T17:27:56.681 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.681 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af112e8e:::benchmark_data_smithi171_122772_object151019:head# 2024-04-20T17:27:56.681 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.681 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af124336:::benchmark_data_smithi171_122772_object93878:head# 2024-04-20T17:27:56.681 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.681 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af1349a5:::benchmark_data_smithi171_122772_object102280:head# 2024-04-20T17:27:56.681 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.681 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af1421f8:::benchmark_data_smithi171_122772_object105746:head# 2024-04-20T17:27:56.681 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.682 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af15f97a:::benchmark_data_smithi171_122772_object153081:head# 2024-04-20T17:27:56.682 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.682 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af17fe2c:::benchmark_data_smithi171_122772_object147773:head# 2024-04-20T17:27:56.682 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.682 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af18f2f3:::benchmark_data_smithi171_122772_object147433:head# 2024-04-20T17:27:56.682 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.682 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af197023:::benchmark_data_smithi171_122772_object106085:head# 2024-04-20T17:27:56.682 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.682 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af197738:::benchmark_data_smithi171_122772_object89986:head# 2024-04-20T17:27:56.682 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.682 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af1a980a:::benchmark_data_smithi171_122772_object146061:head# 2024-04-20T17:27:56.682 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.683 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af1b6896:::benchmark_data_smithi171_122772_object96276:head# 2024-04-20T17:27:56.683 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.683 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af1b6f05:::benchmark_data_smithi171_122772_object78932:head# 2024-04-20T17:27:56.683 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.683 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af1f9afc:::benchmark_data_smithi171_122772_object168792:head# 2024-04-20T17:27:56.683 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.683 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af1fe5ed:::benchmark_data_smithi171_122772_object154282:head# 2024-04-20T17:27:56.683 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.683 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af215da3:::benchmark_data_smithi171_122772_object160370:head# 2024-04-20T17:27:56.683 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.683 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af21b54b:::benchmark_data_smithi171_122772_object78005:head# 2024-04-20T17:27:56.684 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.684 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af22277d:::benchmark_data_smithi171_122772_object128687:head# 2024-04-20T17:27:56.684 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.684 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af231f30:::benchmark_data_smithi171_122772_object116861:head# 2024-04-20T17:27:56.684 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.684 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af236d38:::benchmark_data_smithi171_122772_object144966:head# 2024-04-20T17:27:56.684 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.684 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af23eea6:::benchmark_data_smithi171_122772_object154457:head# 2024-04-20T17:27:56.684 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.684 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af256505:::benchmark_data_smithi171_122772_object148534:head# 2024-04-20T17:27:56.684 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.685 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af265bab:::benchmark_data_smithi171_122772_object150750:head# 2024-04-20T17:27:56.685 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.685 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af287d28:::benchmark_data_smithi171_122772_object146563:head# 2024-04-20T17:27:56.685 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.685 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af28f808:::benchmark_data_smithi171_122772_object105791:head# 2024-04-20T17:27:56.685 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.685 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af294e95:::benchmark_data_smithi171_122772_object146207:head# 2024-04-20T17:27:56.685 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.685 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af2990df:::benchmark_data_smithi171_122772_object128905:head# 2024-04-20T17:27:56.685 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.685 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af29c2b1:::benchmark_data_smithi171_122772_object99469:head# 2024-04-20T17:27:56.685 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.685 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af2c41ed:::benchmark_data_smithi171_122772_object99199:head# 2024-04-20T17:27:56.686 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.686 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af2c4962:::benchmark_data_smithi171_122772_object112415:head# 2024-04-20T17:27:56.686 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.686 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af2cf566:::benchmark_data_smithi171_122772_object92390:head# 2024-04-20T17:27:56.686 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.686 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af2d22dd:::benchmark_data_smithi171_122772_object154966:head# 2024-04-20T17:27:56.686 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.686 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af2d6c07:::benchmark_data_smithi171_122772_object122964:head# 2024-04-20T17:27:56.686 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.686 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af2de6b6:::benchmark_data_smithi171_122772_object128646:head# 2024-04-20T17:27:56.686 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.686 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af2ebc36:::benchmark_data_smithi171_122772_object96458:head# 2024-04-20T17:27:56.686 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.687 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af2ed2bb:::benchmark_data_smithi171_122772_object152769:head# 2024-04-20T17:27:56.687 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.687 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af2f6ad8:::benchmark_data_smithi171_122772_object142319:head# 2024-04-20T17:27:56.687 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.687 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af2fcc61:::benchmark_data_smithi171_122772_object115687:head# 2024-04-20T17:27:56.687 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.687 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af31600a:::benchmark_data_smithi171_122772_object89323:head# 2024-04-20T17:27:56.687 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.687 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af3219ba:::benchmark_data_smithi171_122772_object83885:head# 2024-04-20T17:27:56.687 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.687 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af327d49:::benchmark_data_smithi171_122772_object130713:head# 2024-04-20T17:27:56.687 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.687 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af32a280:::benchmark_data_smithi171_122772_object157549:head# 2024-04-20T17:27:56.688 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.688 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af334568:::benchmark_data_smithi171_122772_object82461:head# 2024-04-20T17:27:56.688 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.688 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af3350ec:::benchmark_data_smithi171_122772_object163481:head# 2024-04-20T17:27:56.688 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.688 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af337520:::benchmark_data_smithi171_122772_object142240:head# 2024-04-20T17:27:56.688 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.688 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af34c3d7:::benchmark_data_smithi171_122772_object165566:head# 2024-04-20T17:27:56.688 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.688 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af35c7f2:::benchmark_data_smithi171_122772_object136782:head# 2024-04-20T17:27:56.688 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.688 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af35d7fe:::benchmark_data_smithi171_122772_object159163:head# 2024-04-20T17:27:56.688 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.688 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af35e785:::benchmark_data_smithi171_122772_object96265:head# 2024-04-20T17:27:56.689 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.689 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af36feb7:::benchmark_data_smithi171_122772_object160566:head# 2024-04-20T17:27:56.689 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.689 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af376d3b:::benchmark_data_smithi171_122772_object147826:head# 2024-04-20T17:27:56.689 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.689 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af37a597:::benchmark_data_smithi171_122772_object94714:head# 2024-04-20T17:27:56.689 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.689 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af3829eb:::benchmark_data_smithi171_122772_object142644:head# 2024-04-20T17:27:56.689 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.689 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af388f48:::benchmark_data_smithi171_122772_object136449:head# 2024-04-20T17:27:56.689 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.689 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af3945d0:::benchmark_data_smithi171_122772_object100702:head# 2024-04-20T17:27:56.689 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.690 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af3a03a1:::benchmark_data_smithi171_122772_object167633:head# 2024-04-20T17:27:56.690 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.690 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af3c34c6:::benchmark_data_smithi171_122772_object127389:head# 2024-04-20T17:27:56.690 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.690 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af3c63cb:::benchmark_data_smithi171_122772_object136459:head# 2024-04-20T17:27:56.690 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.690 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af3c7c15:::benchmark_data_smithi171_122772_object104903:head# 2024-04-20T17:27:56.690 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.690 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af3e9d13:::benchmark_data_smithi171_122772_object128233:head# 2024-04-20T17:27:56.690 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.690 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af3f5b83:::benchmark_data_smithi171_122772_object136954:head# 2024-04-20T17:27:56.690 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.691 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af3fdddb:::benchmark_data_smithi171_122772_object168486:head# 2024-04-20T17:27:56.691 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.691 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af3fed90:::benchmark_data_smithi171_122772_object138895:head# 2024-04-20T17:27:56.691 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.691 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af408659:::benchmark_data_smithi171_122772_object140825:head# 2024-04-20T17:27:56.691 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.691 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af40decd:::benchmark_data_smithi171_122772_object95911:head# 2024-04-20T17:27:56.691 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.691 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af4521f3:::benchmark_data_smithi171_122772_object104242:head# 2024-04-20T17:27:56.691 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.691 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af456ea7:::benchmark_data_smithi171_122772_object120244:head# 2024-04-20T17:27:56.691 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.691 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af45a946:::benchmark_data_smithi171_122772_object112213:head# 2024-04-20T17:27:56.691 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.692 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af468800:::benchmark_data_smithi171_122772_object133904:head# 2024-04-20T17:27:56.692 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.692 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af469cde:::benchmark_data_smithi171_122772_object168896:head# 2024-04-20T17:27:56.692 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.692 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af469f87:::benchmark_data_smithi171_122772_object80114:head# 2024-04-20T17:27:56.692 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.692 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af46d195:::benchmark_data_smithi171_122772_object81719:head# 2024-04-20T17:27:56.692 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.692 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af477fbb:::benchmark_data_smithi171_122772_object130518:head# 2024-04-20T17:27:56.692 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.692 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af4a4068:::benchmark_data_smithi171_122772_object117500:head# 2024-04-20T17:27:56.693 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.693 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af4ab5f1:::benchmark_data_smithi171_122772_object168291:head# 2024-04-20T17:27:56.693 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.693 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af4d5c74:::benchmark_data_smithi171_122772_object91232:head# 2024-04-20T17:27:56.693 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.693 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af4e1a5c:::benchmark_data_smithi171_122772_object96594:head# 2024-04-20T17:27:56.693 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.693 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af4ead33:::benchmark_data_smithi171_122772_object163709:head# 2024-04-20T17:27:56.693 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.693 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af4f3ab9:::benchmark_data_smithi171_122772_object152048:head# 2024-04-20T17:27:56.693 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.693 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af501448:::benchmark_data_smithi171_122772_object133253:head# 2024-04-20T17:27:56.693 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.694 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af54975e:::benchmark_data_smithi171_122772_object140364:head# 2024-04-20T17:27:56.694 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.694 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af54b043:::benchmark_data_smithi171_122772_object103757:head# 2024-04-20T17:27:56.694 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.694 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af54e979:::benchmark_data_smithi171_122772_object86585:head# 2024-04-20T17:27:56.694 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.694 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af556bd3:::benchmark_data_smithi171_122772_object135021:head# 2024-04-20T17:27:56.694 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.694 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af574a83:::benchmark_data_smithi171_122772_object126480:head# 2024-04-20T17:27:56.694 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.694 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af587f82:::benchmark_data_smithi171_122772_object83304:head# 2024-04-20T17:27:56.695 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.695 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af5950a7:::benchmark_data_smithi171_122772_object125883:head# 2024-04-20T17:27:56.695 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.695 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af5a6ebb:::benchmark_data_smithi171_122772_object96577:head# 2024-04-20T17:27:56.695 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.695 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af5ac45f:::benchmark_data_smithi171_122772_object94095:head# 2024-04-20T17:27:56.695 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.695 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af5b5c58:::benchmark_data_smithi171_122772_object164179:head# 2024-04-20T17:27:56.695 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.695 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af5b624c:::benchmark_data_smithi171_122772_object85259:head# 2024-04-20T17:27:56.695 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.695 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af5c7149:::benchmark_data_smithi171_122772_object133694:head# 2024-04-20T17:27:56.696 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.696 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af6060bd:::benchmark_data_smithi171_122772_object123753:head# 2024-04-20T17:27:56.696 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.696 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af61578e:::benchmark_data_smithi171_122772_object83486:head# 2024-04-20T17:27:56.696 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.696 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af61bbd4:::benchmark_data_smithi171_122772_object152095:head# 2024-04-20T17:27:56.696 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.696 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af62ac95:::benchmark_data_smithi171_122772_object119267:head# 2024-04-20T17:27:56.696 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.696 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af63955b:::benchmark_data_smithi171_122772_object117676:head# 2024-04-20T17:27:56.696 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.696 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af63b598:::benchmark_data_smithi171_122772_object152310:head# 2024-04-20T17:27:56.696 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.696 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af6413f9:::benchmark_data_smithi171_122772_object156509:head# 2024-04-20T17:27:56.697 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.697 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af64aef1:::benchmark_data_smithi171_122772_object128590:head# 2024-04-20T17:27:56.697 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.697 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af6576a4:::benchmark_data_smithi171_122772_object80695:head# 2024-04-20T17:27:56.697 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.697 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af65ee98:::benchmark_data_smithi171_122772_object149405:head# 2024-04-20T17:27:56.697 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.697 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af65f1cd:::benchmark_data_smithi171_122772_object160208:head# 2024-04-20T17:27:56.697 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.697 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af65fbf9:::benchmark_data_smithi171_122772_object103375:head# 2024-04-20T17:27:56.697 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.697 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af6692cf:::benchmark_data_smithi171_122772_object114195:head# 2024-04-20T17:27:56.697 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.698 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af68707f:::benchmark_data_smithi171_122772_object131269:head# 2024-04-20T17:27:56.698 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.698 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af68e003:::benchmark_data_smithi171_122772_object135072:head# 2024-04-20T17:27:56.698 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.698 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af69d2e5:::benchmark_data_smithi171_122772_object153777:head# 2024-04-20T17:27:56.698 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.698 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af6a2df2:::benchmark_data_smithi171_122772_object170180:head# 2024-04-20T17:27:56.698 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.698 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af6a3283:::benchmark_data_smithi171_122772_object89977:head# 2024-04-20T17:27:56.698 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.698 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af6b3582:::benchmark_data_smithi171_122772_object167449:head# 2024-04-20T17:27:56.698 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.699 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af6bb9b5:::benchmark_data_smithi171_122772_object162548:head# 2024-04-20T17:27:56.699 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.699 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af6beefc:::benchmark_data_smithi171_122772_object158635:head# 2024-04-20T17:27:56.699 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.699 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af6c6f29:::benchmark_data_smithi171_122772_object113168:head# 2024-04-20T17:27:56.699 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.699 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af6cba4f:::benchmark_data_smithi171_122772_object129009:head# 2024-04-20T17:27:56.699 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.699 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af6d3aae:::benchmark_data_smithi171_122772_object141324:head# 2024-04-20T17:27:56.699 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.699 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af6d5bb2:::benchmark_data_smithi171_122772_object88906:head# 2024-04-20T17:27:56.699 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.699 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af6d6f69:::benchmark_data_smithi171_122772_object121040:head# 2024-04-20T17:27:56.699 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.700 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af6e1dae:::benchmark_data_smithi171_122772_object104211:head# 2024-04-20T17:27:56.700 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.700 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af6ef414:::benchmark_data_smithi171_122772_object144496:head# 2024-04-20T17:27:56.700 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.700 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af6ef4d3:::benchmark_data_smithi171_122772_object152555:head# 2024-04-20T17:27:56.700 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.700 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af6f3260:::benchmark_data_smithi171_122772_object98438:head# 2024-04-20T17:27:56.700 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.700 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af6f3482:::benchmark_data_smithi171_122772_object159837:head# 2024-04-20T17:27:56.700 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.700 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af6f34b8:::benchmark_data_smithi171_122772_object79508:head# 2024-04-20T17:27:56.700 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.700 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af6f3a7b:::benchmark_data_smithi171_122772_object125442:head# 2024-04-20T17:27:56.701 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.701 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af6f884f:::benchmark_data_smithi171_122772_object107371:head# 2024-04-20T17:27:56.701 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.701 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af709bf2:::benchmark_data_smithi171_122772_object149418:head# 2024-04-20T17:27:56.701 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.701 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af712c1c:::benchmark_data_smithi171_122772_object169772:head# 2024-04-20T17:27:56.701 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.701 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af71b7c2:::benchmark_data_smithi171_122772_object134643:head# 2024-04-20T17:27:56.701 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.701 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af724dd3:::benchmark_data_smithi171_122772_object82648:head# 2024-04-20T17:27:56.701 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.701 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af72c3e1:::benchmark_data_smithi171_122772_object129130:head# 2024-04-20T17:27:56.701 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.701 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af73a8a7:::benchmark_data_smithi171_122772_object163750:head# 2024-04-20T17:27:56.702 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.702 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af73a9ed:::benchmark_data_smithi171_122772_object81256:head# 2024-04-20T17:27:56.702 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.702 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af74c506:::benchmark_data_smithi171_122772_object82031:head# 2024-04-20T17:27:56.702 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.702 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af7514db:::benchmark_data_smithi171_122772_object81337:head# 2024-04-20T17:27:56.702 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.702 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af75d50d:::benchmark_data_smithi171_122772_object151269:head# 2024-04-20T17:27:56.702 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.702 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af78c233:::benchmark_data_smithi171_122772_object148009:head# 2024-04-20T17:27:56.702 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.703 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af7917c2:::benchmark_data_smithi171_122772_object143319:head# 2024-04-20T17:27:56.703 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.703 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af7ac07d:::benchmark_data_smithi171_122772_object86039:head# 2024-04-20T17:27:56.703 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.703 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af7c155b:::benchmark_data_smithi171_122772_object79319:head# 2024-04-20T17:27:56.703 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.703 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af7c42cf:::benchmark_data_smithi171_122772_object159181:head# 2024-04-20T17:27:56.703 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.703 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af7c4d0c:::benchmark_data_smithi171_122772_object162837:head# 2024-04-20T17:27:56.703 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.703 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af7d0507:::benchmark_data_smithi171_122772_object94932:head# 2024-04-20T17:27:56.703 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.703 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af7dcf57:::benchmark_data_smithi171_122772_object135874:head# 2024-04-20T17:27:56.703 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.704 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af7ea145:::benchmark_data_smithi171_122772_object151745:head# 2024-04-20T17:27:56.704 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.704 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af7eb14e:::benchmark_data_smithi171_122772_object127974:head# 2024-04-20T17:27:56.704 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.704 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af7fbde5:::benchmark_data_smithi171_122772_object81301:head# 2024-04-20T17:27:56.704 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.704 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af814483:::benchmark_data_smithi171_122772_object123568:head# 2024-04-20T17:27:56.704 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.704 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af81dda7:::benchmark_data_smithi171_122772_object145766:head# 2024-04-20T17:27:56.704 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.704 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af8207a8:::benchmark_data_smithi171_122772_object148923:head# 2024-04-20T17:27:56.704 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.704 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af82295f:::benchmark_data_smithi171_122772_object104872:head# 2024-04-20T17:27:56.705 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.705 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af823b1e:::benchmark_data_smithi171_122772_object144584:head# 2024-04-20T17:27:56.705 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.705 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af834364:::benchmark_data_smithi171_122772_object162091:head# 2024-04-20T17:27:56.705 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.705 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af837c8f:::benchmark_data_smithi171_122772_object161060:head# 2024-04-20T17:27:56.705 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.705 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af83faca:::benchmark_data_smithi171_122772_object126700:head# 2024-04-20T17:27:56.705 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.705 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af85447a:::benchmark_data_smithi171_122772_object100935:head# 2024-04-20T17:27:56.705 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.705 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af865f9f:::benchmark_data_smithi171_122772_object121463:head# 2024-04-20T17:27:56.706 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.706 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af86dfb2:::benchmark_data_smithi171_122772_object104627:head# 2024-04-20T17:27:56.706 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.706 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af86fe03:::benchmark_data_smithi171_122772_object130603:head# 2024-04-20T17:27:56.706 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.706 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af878265:::benchmark_data_smithi171_122772_object116199:head# 2024-04-20T17:27:56.706 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.706 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af87ff55:::benchmark_data_smithi171_122772_object101257:head# 2024-04-20T17:27:56.706 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.706 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af88d72d:::benchmark_data_smithi171_122772_object89973:head# 2024-04-20T17:27:56.706 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.706 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af89ba57:::benchmark_data_smithi171_122772_object78414:head# 2024-04-20T17:27:56.706 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.707 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af89d3af:::benchmark_data_smithi171_122772_object90095:head# 2024-04-20T17:27:56.707 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.707 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af8a7f5f:::benchmark_data_smithi171_122772_object165444:head# 2024-04-20T17:27:56.707 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.707 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af8b5301:::benchmark_data_smithi171_122772_object130161:head# 2024-04-20T17:27:56.707 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.707 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af8c710f:::benchmark_data_smithi171_122772_object84642:head# 2024-04-20T17:27:56.707 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.707 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af8d07c6:::benchmark_data_smithi171_122772_object141067:head# 2024-04-20T17:27:56.707 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.707 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af8d0b0e:::benchmark_data_smithi171_122772_object153284:head# 2024-04-20T17:27:56.707 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.707 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af8d115f:::benchmark_data_smithi171_122772_object87988:head# 2024-04-20T17:27:56.708 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.708 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af8d8255:::benchmark_data_smithi171_122772_object136690:head# 2024-04-20T17:27:56.708 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.708 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af8df1e2:::benchmark_data_smithi171_122772_object133394:head# 2024-04-20T17:27:56.708 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.708 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af8fb3bc:::benchmark_data_smithi171_122772_object78419:head# 2024-04-20T17:27:56.708 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.708 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af902acf:::benchmark_data_smithi171_122772_object84743:head# 2024-04-20T17:27:56.708 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.708 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af9184e7:::benchmark_data_smithi171_122772_object158294:head# 2024-04-20T17:27:56.708 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.708 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af918e74:::benchmark_data_smithi171_122772_object113965:head# 2024-04-20T17:27:56.708 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.709 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af9263da:::benchmark_data_smithi171_122772_object152295:head# 2024-04-20T17:27:56.709 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.709 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af92e2ca:::benchmark_data_smithi171_122772_object91419:head# 2024-04-20T17:27:56.709 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.709 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af93c012:::benchmark_data_smithi171_122772_object142237:head# 2024-04-20T17:27:56.709 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.709 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af952983:::benchmark_data_smithi171_122772_object99408:head# 2024-04-20T17:27:56.709 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.709 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af956d5e:::benchmark_data_smithi171_122772_object156720:head# 2024-04-20T17:27:56.709 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.709 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af956fc3:::benchmark_data_smithi171_122772_object147256:head# 2024-04-20T17:27:56.710 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.710 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af958f7c:::benchmark_data_smithi171_122772_object130354:head# 2024-04-20T17:27:56.710 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.710 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af96ad14:::benchmark_data_smithi171_122772_object138509:head# 2024-04-20T17:27:56.710 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.710 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af96f7cd:::benchmark_data_smithi171_122772_object114343:head# 2024-04-20T17:27:56.710 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.710 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af971535:::benchmark_data_smithi171_122772_object79680:head# 2024-04-20T17:27:56.710 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.710 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af97e8bf:::benchmark_data_smithi171_122772_object92430:head# 2024-04-20T17:27:56.710 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.710 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af98170b:::benchmark_data_smithi171_122772_object93091:head# 2024-04-20T17:27:56.711 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.711 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af989ad5:::benchmark_data_smithi171_122772_object102337:head# 2024-04-20T17:27:56.711 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.711 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af99902b:::benchmark_data_smithi171_122772_object104241:head# 2024-04-20T17:27:56.711 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.711 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af9ad628:::benchmark_data_smithi171_122772_object135478:head# 2024-04-20T17:27:56.711 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.711 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af9d539b:::benchmark_data_smithi171_122772_object136378:head# 2024-04-20T17:27:56.711 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.711 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af9d92ab:::benchmark_data_smithi171_122772_object169170:head# 2024-04-20T17:27:56.711 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.711 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:af9ef014:::benchmark_data_smithi171_122772_object128913:head# 2024-04-20T17:27:56.712 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.712 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afa019da:::benchmark_data_smithi171_122772_object94773:head# 2024-04-20T17:27:56.712 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.712 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afa1063a:::benchmark_data_smithi171_122772_object130715:head# 2024-04-20T17:27:56.712 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.712 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afa17377:::benchmark_data_smithi171_122772_object97656:head# 2024-04-20T17:27:56.712 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.712 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afa1afc0:::benchmark_data_smithi171_122772_object145971:head# 2024-04-20T17:27:56.712 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.712 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afa2d15d:::benchmark_data_smithi171_122772_object145609:head# 2024-04-20T17:27:56.712 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.712 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afa3809d:::benchmark_data_smithi171_122772_object109758:head# 2024-04-20T17:27:56.712 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.712 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afa63c6e:::benchmark_data_smithi171_122772_object86822:head# 2024-04-20T17:27:56.713 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.713 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afa6627e:::benchmark_data_smithi171_122772_object85145:head# 2024-04-20T17:27:56.713 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.713 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afa73344:::benchmark_data_smithi171_122772_object84514:head# 2024-04-20T17:27:56.713 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.713 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afa76db1:::benchmark_data_smithi171_122772_object123847:head# 2024-04-20T17:27:56.713 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.713 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afa7fb2c:::benchmark_data_smithi171_122772_object93112:head# 2024-04-20T17:27:56.713 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.713 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afa91dd7:::benchmark_data_smithi171_122772_object86936:head# 2024-04-20T17:27:56.713 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.713 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afa951a8:::benchmark_data_smithi171_122772_object139128:head# 2024-04-20T17:27:56.713 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.713 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afa9984c:::benchmark_data_smithi171_122772_object141385:head# 2024-04-20T17:27:56.714 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.714 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afab0829:::benchmark_data_smithi171_122772_object84988:head# 2024-04-20T17:27:56.714 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.714 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afab1198:::benchmark_data_smithi171_122772_object120388:head# 2024-04-20T17:27:56.714 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.714 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afab40c7:::benchmark_data_smithi171_122772_object124162:head# 2024-04-20T17:27:56.714 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.714 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afab7599:::benchmark_data_smithi171_122772_object80323:head# 2024-04-20T17:27:56.714 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.714 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afac6f2e:::benchmark_data_smithi171_122772_object128046:head# 2024-04-20T17:27:56.715 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.715 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afac9a61:::benchmark_data_smithi171_122772_object163659:head# 2024-04-20T17:27:56.715 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.715 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afad8679:::benchmark_data_smithi171_122772_object152768:head# 2024-04-20T17:27:56.715 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.715 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afadc7ec:::benchmark_data_smithi171_122772_object159694:head# 2024-04-20T17:27:56.715 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.715 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afadda0f:::benchmark_data_smithi171_122772_object142148:head# 2024-04-20T17:27:56.715 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.715 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afaea273:::benchmark_data_smithi171_122772_object126360:head# 2024-04-20T17:27:56.715 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.715 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afaee23b:::benchmark_data_smithi171_122772_object156583:head# 2024-04-20T17:27:56.716 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.716 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afafc6b7:::benchmark_data_smithi171_122772_object98685:head# 2024-04-20T17:27:56.716 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.716 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afb04d06:::benchmark_data_smithi171_122772_object90493:head# 2024-04-20T17:27:56.716 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.716 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afb1aea6:::benchmark_data_smithi171_122772_object95668:head# 2024-04-20T17:27:56.716 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.716 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afb3023b:::benchmark_data_smithi171_122772_object98051:head# 2024-04-20T17:27:56.716 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.716 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afb3ab84:::benchmark_data_smithi171_122772_object88535:head# 2024-04-20T17:27:56.716 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.717 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afb405d1:::benchmark_data_smithi171_122772_object142057:head# 2024-04-20T17:27:56.717 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.717 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afb5c130:::benchmark_data_smithi171_122772_object142760:head# 2024-04-20T17:27:56.717 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.717 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afb67209:::benchmark_data_smithi171_122772_object158029:head# 2024-04-20T17:27:56.717 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.717 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afb6cf4b:::benchmark_data_smithi171_122772_object123552:head# 2024-04-20T17:27:56.717 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.717 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afb81901:::benchmark_data_smithi171_122772_object165089:head# 2024-04-20T17:27:56.717 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.717 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afb94677:::benchmark_data_smithi171_122772_object167245:head# 2024-04-20T17:27:56.717 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.717 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afb956c3:::benchmark_data_smithi171_122772_object87708:head# 2024-04-20T17:27:56.718 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.718 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afb9889f:::benchmark_data_smithi171_122772_object87846:head# 2024-04-20T17:27:56.718 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.718 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afba4727:::benchmark_data_smithi171_122772_object77957:head# 2024-04-20T17:27:56.718 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.718 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afbb8a57:::benchmark_data_smithi171_122772_object160693:head# 2024-04-20T17:27:56.718 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.718 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afbc4736:::benchmark_data_smithi171_122772_object139647:head# 2024-04-20T17:27:56.718 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.718 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afbc60a2:::benchmark_data_smithi171_122772_object164428:head# 2024-04-20T17:27:56.718 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.718 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afbc9785:::benchmark_data_smithi171_122772_object112181:head# 2024-04-20T17:27:56.719 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.719 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afbcf183:::benchmark_data_smithi171_122772_object87316:head# 2024-04-20T17:27:56.719 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.719 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afbd1feb:::benchmark_data_smithi171_122772_object150933:head# 2024-04-20T17:27:56.719 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.719 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afbd22d6:::benchmark_data_smithi171_122772_object117625:head# 2024-04-20T17:27:56.719 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.719 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afbe19d8:::benchmark_data_smithi171_122772_object92993:head# 2024-04-20T17:27:56.719 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.719 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afbe638f:::benchmark_data_smithi171_122772_object124015:head# 2024-04-20T17:27:56.719 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.719 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afbef0c4:::benchmark_data_smithi171_122772_object78994:head# 2024-04-20T17:27:56.719 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.720 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afbfc998:::benchmark_data_smithi171_122772_object90196:head# 2024-04-20T17:27:56.720 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.720 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afc12ceb:::benchmark_data_smithi171_122772_object79340:head# 2024-04-20T17:27:56.720 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.720 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afc2dc19:::benchmark_data_smithi171_122772_object153092:head# 2024-04-20T17:27:56.720 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.720 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afc31bc1:::benchmark_data_smithi171_122772_object141677:head# 2024-04-20T17:27:56.720 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.720 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afc350c3:::benchmark_data_smithi171_122772_object166658:head# 2024-04-20T17:27:56.720 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.720 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afc41eae:::benchmark_data_smithi171_122772_object115737:head# 2024-04-20T17:27:56.720 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.720 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afc49bb4:::benchmark_data_smithi171_122772_object120085:head# 2024-04-20T17:27:56.721 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.721 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afc5c018:::benchmark_data_smithi171_122772_object113789:head# 2024-04-20T17:27:56.721 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.721 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afc60c96:::benchmark_data_smithi171_122772_object161497:head# 2024-04-20T17:27:56.721 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.721 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afc61ca6:::benchmark_data_smithi171_122772_object163703:head# 2024-04-20T17:27:56.721 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.721 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afc6e572:::benchmark_data_smithi171_122772_object134166:head# 2024-04-20T17:27:56.721 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.721 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afc7e223:::benchmark_data_smithi171_122772_object106867:head# 2024-04-20T17:27:56.721 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.721 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afc8b47c:::benchmark_data_smithi171_122772_object128011:head# 2024-04-20T17:27:56.722 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.722 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afc8ea64:::benchmark_data_smithi171_122772_object126469:head# 2024-04-20T17:27:56.722 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.722 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afc8f7a8:::benchmark_data_smithi171_122772_object118844:head# 2024-04-20T17:27:56.722 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.722 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afc9a313:::benchmark_data_smithi171_122772_object155626:head# 2024-04-20T17:27:56.722 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.722 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afcaab50:::benchmark_data_smithi171_122772_object114228:head# 2024-04-20T17:27:56.722 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.722 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afcb1edd:::benchmark_data_smithi171_122772_object117463:head# 2024-04-20T17:27:56.722 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.722 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afcb255d:::benchmark_data_smithi171_122772_object114893:head# 2024-04-20T17:27:56.723 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.723 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afcb7636:::benchmark_data_smithi171_122772_object158780:head# 2024-04-20T17:27:56.723 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.723 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afcd1fcf:::benchmark_data_smithi171_122772_object100089:head# 2024-04-20T17:27:56.723 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.723 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afce80b7:::benchmark_data_smithi171_122772_object155350:head# 2024-04-20T17:27:56.723 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.723 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afcf311a:::benchmark_data_smithi171_122772_object169140:head# 2024-04-20T17:27:56.723 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.723 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afd093a8:::benchmark_data_smithi171_122772_object149009:head# 2024-04-20T17:27:56.723 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.723 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afd16e4a:::benchmark_data_smithi171_122772_object102826:head# 2024-04-20T17:27:56.724 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.724 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afd1985d:::benchmark_data_smithi171_122772_object111315:head# 2024-04-20T17:27:56.724 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.724 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afd1bfa3:::benchmark_data_smithi171_122772_object161444:head# 2024-04-20T17:27:56.724 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.724 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afd2f2b5:::benchmark_data_smithi171_122772_object81777:head# 2024-04-20T17:27:56.724 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.724 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afd33222:::benchmark_data_smithi171_122772_object108599:head# 2024-04-20T17:27:56.724 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.724 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afd3693e:::benchmark_data_smithi171_122772_object141863:head# 2024-04-20T17:27:56.724 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.724 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afd387c8:::benchmark_data_smithi171_122772_object135131:head# 2024-04-20T17:27:56.724 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.724 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afd3bd88:::benchmark_data_smithi171_122772_object112255:head# 2024-04-20T17:27:56.725 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.725 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afd49ce1:::benchmark_data_smithi171_122772_object160192:head# 2024-04-20T17:27:56.725 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.725 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afd5597e:::benchmark_data_smithi171_122772_object142076:head# 2024-04-20T17:27:56.725 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.725 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afd5a1d9:::benchmark_data_smithi171_122772_object144561:head# 2024-04-20T17:27:56.725 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.725 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afd9169f:::benchmark_data_smithi171_122772_object139080:head# 2024-04-20T17:27:56.725 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.725 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afdb0c56:::benchmark_data_smithi171_122772_object93965:head# 2024-04-20T17:27:56.725 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.725 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afdb2116:::benchmark_data_smithi171_122772_object86183:head# 2024-04-20T17:27:56.725 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.726 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afdb2bcc:::benchmark_data_smithi171_122772_object165347:head# 2024-04-20T17:27:56.726 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.726 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afdbcc2e:::benchmark_data_smithi171_122772_object78656:head# 2024-04-20T17:27:56.726 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.726 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afdc2953:::benchmark_data_smithi171_122772_object107903:head# 2024-04-20T17:27:56.726 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.726 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afdc6395:::benchmark_data_smithi171_122772_object87136:head# 2024-04-20T17:27:56.726 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.726 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afdd9777:::benchmark_data_smithi171_122772_object133779:head# 2024-04-20T17:27:56.726 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.726 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afde41b5:::benchmark_data_smithi171_122772_object104338:head# 2024-04-20T17:27:56.726 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.727 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afdef5fe:::benchmark_data_smithi171_122772_object133161:head# 2024-04-20T17:27:56.727 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.727 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afdef7e0:::benchmark_data_smithi171_122772_object90656:head# 2024-04-20T17:27:56.727 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.727 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afdf3244:::benchmark_data_smithi171_122772_object88361:head# 2024-04-20T17:27:56.727 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.727 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afe0e434:::benchmark_data_smithi171_122772_object169487:head# 2024-04-20T17:27:56.727 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.727 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afe3ea85:::benchmark_data_smithi171_122772_object151050:head# 2024-04-20T17:27:56.727 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.727 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afe44635:::benchmark_data_smithi171_122772_object146590:head# 2024-04-20T17:27:56.727 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.727 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afe46f12:::benchmark_data_smithi171_122772_object150017:head# 2024-04-20T17:27:56.728 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.728 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afe4b2e1:::benchmark_data_smithi171_122772_object84773:head# 2024-04-20T17:27:56.728 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.728 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afe53c18:::benchmark_data_smithi171_122772_object88665:head# 2024-04-20T17:27:56.728 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.728 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afe5e91c:::benchmark_data_smithi171_122772_object147684:head# 2024-04-20T17:27:56.728 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.728 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afe6c6f8:::benchmark_data_smithi171_122772_object148496:head# 2024-04-20T17:27:56.728 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.728 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afe7986f:::benchmark_data_smithi171_122772_object134878:head# 2024-04-20T17:27:56.728 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.728 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afe80c37:::benchmark_data_smithi171_122772_object101081:head# 2024-04-20T17:27:56.728 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.728 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afe8ab35:::benchmark_data_smithi171_122772_object144586:head# 2024-04-20T17:27:56.729 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.729 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afe91888:::benchmark_data_smithi171_122772_object106347:head# 2024-04-20T17:27:56.729 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.729 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afe9fd76:::benchmark_data_smithi171_122772_object117042:head# 2024-04-20T17:27:56.729 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.729 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afea4725:::benchmark_data_smithi171_122772_object118884:head# 2024-04-20T17:27:56.729 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.729 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afeaf556:::benchmark_data_smithi171_122772_object134863:head# 2024-04-20T17:27:56.729 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.729 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afec7c70:::benchmark_data_smithi171_122772_object102948:head# 2024-04-20T17:27:56.729 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.729 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afecd2be:::benchmark_data_smithi171_122772_object95010:head# 2024-04-20T17:27:56.729 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.730 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afedd9fa:::benchmark_data_smithi171_122772_object118612:head# 2024-04-20T17:27:56.730 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.730 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afede448:::benchmark_data_smithi171_122772_object124996:head# 2024-04-20T17:27:56.730 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.730 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afedeb28:::benchmark_data_smithi171_122772_object111824:head# 2024-04-20T17:27:56.730 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.730 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:afedf794:::benchmark_data_smithi171_122772_object90365:head# 2024-04-20T17:27:56.730 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.730 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aff164c4:::benchmark_data_smithi171_122772_object168910:head# 2024-04-20T17:27:56.730 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.730 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aff1a21b:::benchmark_data_smithi171_122772_object157502:head# 2024-04-20T17:27:56.730 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.730 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aff240c4:::benchmark_data_smithi171_122772_object167647:head# 2024-04-20T17:27:56.731 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.731 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aff25d05:::benchmark_data_smithi171_122772_object114242:head# 2024-04-20T17:27:56.731 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.731 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aff388fa:::benchmark_data_smithi171_122772_object122904:head# 2024-04-20T17:27:56.731 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.731 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aff38ef4:::benchmark_data_smithi171_122772_object156027:head# 2024-04-20T17:27:56.731 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.731 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aff47bfd:::benchmark_data_smithi171_122772_object125056:head# 2024-04-20T17:27:56.731 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.731 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aff4808d:::benchmark_data_smithi171_122772_object157207:head# 2024-04-20T17:27:56.731 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.731 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aff517db:::benchmark_data_smithi171_122772_object124827:head# 2024-04-20T17:27:56.731 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.732 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aff662fb:::benchmark_data_smithi171_122772_object126043:head# 2024-04-20T17:27:56.732 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.732 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aff67b55:::benchmark_data_smithi171_122772_object126516:head# 2024-04-20T17:27:56.732 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.732 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aff67c91:::benchmark_data_smithi171_122772_object92829:head# 2024-04-20T17:27:56.732 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.732 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aff789f1:::benchmark_data_smithi171_122772_object127724:head# 2024-04-20T17:27:56.732 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.732 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aff7a8d3:::benchmark_data_smithi171_122772_object163442:head# 2024-04-20T17:27:56.732 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.732 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aff844e1:::benchmark_data_smithi171_122772_object163092:head# 2024-04-20T17:27:56.732 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.732 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aff85d85:::benchmark_data_smithi171_122772_object162659:head# 2024-04-20T17:27:56.732 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.733 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aff8cdaa:::benchmark_data_smithi171_122772_object120111:head# 2024-04-20T17:27:56.733 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.733 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aff91294:::benchmark_data_smithi171_122772_object134847:head# 2024-04-20T17:27:56.733 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.733 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aff93c9d:::benchmark_data_smithi171_122772_object100378:head# 2024-04-20T17:27:56.733 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.733 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:aff99b83:::benchmark_data_smithi171_122772_object144627:head# 2024-04-20T17:27:56.733 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.733 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:affab38c:::benchmark_data_smithi171_122772_object82721:head# 2024-04-20T17:27:56.733 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.733 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:affae371:::benchmark_data_smithi171_122772_object84667:head# 2024-04-20T17:27:56.733 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.733 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:affba10c:::benchmark_data_smithi171_122772_object90149:head# 2024-04-20T17:27:56.733 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.734 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:affc5fe1:::benchmark_data_smithi171_122772_object112543:head# 2024-04-20T17:27:56.734 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.734 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:affd4112:::benchmark_data_smithi171_122772_object87390:head# 2024-04-20T17:27:56.734 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.734 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:affdc9e1:::benchmark_data_smithi171_122772_object119785:head# 2024-04-20T17:27:56.734 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.734 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:affe63f3:::benchmark_data_smithi171_122772_object101392:head# 2024-04-20T17:27:56.734 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.734 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:affe8207:::benchmark_data_smithi171_122772_object117658:head# 2024-04-20T17:27:56.734 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.734 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:affebe5b:::benchmark_data_smithi171_122772_object135561:head# 2024-04-20T17:27:56.734 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:27:56.734 INFO:teuthology.orchestra.run.smithi106.stdout:write_pg epoch 503 info 4.5( v 503'15601 (462'5600,503'15601] local-lis/les=445/446 n=5733 ec=445/445 lis/c=445/445 les/c/f=446/446/0 sis=445) 2024-04-20T17:27:56.735 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:27:56.736 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:56.737 INFO:teuthology.orchestra.run.smithi106.stdout:Import successful 2024-04-20T17:27:56.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:56.738 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:27:56.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:27:56.838 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:27:56.878 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:56.904 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:27:56.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:56.907 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:27:57.003 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:27:57.003 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:27:57.004 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:27:57.004 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:27:57.004 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:27:57.004 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:27:57.004 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:27:57.004 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:27:57.004 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:27:57.004 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:27:57.004 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:27:57.004 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:27:57.004 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:27:57.004 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:27:57.005 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:27:57.005 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:27:57.005 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:27:57.005 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-20T17:27:57.005 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:27:57.005 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:27:57.005 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:27:57.005 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:27:57.005 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:27:57.007 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:27:57.059 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:57.740 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:57 smithi005 ceph-mon[39562]: pgmap v1364: 17 pgs: 17 active+clean; 5.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 881 op/s; 29 MiB/s, 471 objects/s recovering 2024-04-20T17:27:57.740 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:57 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/49996970' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:27:57.740 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:57.740 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:57.740 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:57.741 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:57.741 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:57.741 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:57.741 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:27:57.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:57.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:27:57.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:57 smithi106 ceph-mon[44423]: pgmap v1364: 17 pgs: 17 active+clean; 5.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 881 op/s; 29 MiB/s, 471 objects/s recovering 2024-04-20T17:27:57.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:57 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/49996970' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:27:57.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:57.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:57.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:57.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:57.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:57.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:57.809 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:27:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:57 smithi171 ceph-mon[44189]: pgmap v1364: 17 pgs: 17 active+clean; 5.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 881 op/s; 29 MiB/s, 471 objects/s recovering 2024-04-20T17:27:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:57 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/49996970' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:27:57.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:57.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:57.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:57.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:57.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:57.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:57.849 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:27:57.884 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:57.931 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/exp.4.5.6 2024-04-20T17:27:57.940 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:27:57.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:57.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:27:58.020 INFO:tasks.thrashosds.thrasher:Waiting for clean again 2024-04-20T17:27:58.020 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-20T17:27:58.020 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:27:58.061 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:27:58.103 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:58.128 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:27:58.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:58.134 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:27:58.235 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:27:58.266 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:58.291 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:27:58.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:58.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:27:58.396 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:27:58.439 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:27:58.440 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:27:58.453 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:58.512 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:27:58.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:58.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:27:58.626 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:27:58.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:58.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:58.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:58.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:58.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:58.664 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:58.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:58.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:58.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:58.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:58.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:58.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:58.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:58.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:58.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:58.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:27:59.036 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:27:59.037 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:27:59.231 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:27:59.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:59.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:27:59.335 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:27:59.372 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:59.400 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:27:59.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:59.404 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:27:59.505 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:27:59.540 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:59.567 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:27:59.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:27:59.573 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:27:59.674 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:27:59.701 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:27:59 smithi005 ceph-mon[39562]: pgmap v1365: 17 pgs: 17 active+clean; 5.1 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 792 op/s; 24 MiB/s, 381 objects/s recovering 2024-04-20T17:27:59.727 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:27:59.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:27:59 smithi106 ceph-mon[44423]: pgmap v1365: 17 pgs: 17 active+clean; 5.1 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 792 op/s; 24 MiB/s, 381 objects/s recovering 2024-04-20T17:27:59.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:27:59 smithi171 ceph-mon[44189]: pgmap v1365: 17 pgs: 17 active+clean; 5.1 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 792 op/s; 24 MiB/s, 381 objects/s recovering 2024-04-20T17:28:00.065 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:28:00.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:00.100 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:28:00.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:28:00.201 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:28:00.240 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:00.265 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:28:00.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:00.270 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:28:00.371 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:28:00.404 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:00.440 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:28:00.441 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:28:00.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:00.445 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:28:00.546 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:28:00.610 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:00.877 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:28:00.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:00.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:28:00.991 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:28:01.035 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:01.074 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:28:01.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:01.078 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:28:01.179 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:28:01.225 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:01.290 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:28:01.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:01.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:28:01.355 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-20T17:28:01.401 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:28:01.435 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:01.462 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:28:01.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:01.466 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:28:01.567 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:28:01.642 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:01.754 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:28:01.757 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:01 smithi005 ceph-mon[39562]: pgmap v1366: 17 pgs: 17 active+clean; 5.0 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-20T17:28:01.759 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:01 smithi106 ceph-mon[44423]: pgmap v1366: 17 pgs: 17 active+clean; 5.0 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-20T17:28:01.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:01.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:28:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:01 smithi171 ceph-mon[44189]: pgmap v1366: 17 pgs: 17 active+clean; 5.0 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-20T17:28:01.862 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:28:01.914 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:01.978 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:28:01.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:01.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:28:02.087 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:28:02.152 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:02.294 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:28:02.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:02.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:28:02.408 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:28:02.445 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:02.478 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:28:02.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:02.488 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:28:02.525 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:28:02.589 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:28:02.651 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:02.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:02 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1004278958' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:28:02.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:02 smithi005 ceph-mon[39562]: from='client.26426 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:28:02.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:02 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1004278958' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:28:02.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:02 smithi106 ceph-mon[44423]: from='client.26426 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:28:02.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:02 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1004278958' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:28:02.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:02 smithi171 ceph-mon[44189]: from='client.26426 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:28:02.866 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:28:02.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:02.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:28:02.976 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:28:03.015 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:03.053 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:28:03.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:03.057 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:28:03.122 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:28:03.158 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:28:03.197 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:03.221 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:28:03.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:03.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:28:03.325 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:28:03.388 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:03 smithi005 ceph-mon[39562]: pgmap v1367: 17 pgs: 17 active+clean; 4.8 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-20T17:28:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:03 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:28:03.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:03 smithi005 ceph-mon[39562]: from='client.26426 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:28:03.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:03 smithi005 ceph-mon[39562]: osdmap e520: 12 total, 11 up, 11 in 2024-04-20T17:28:03.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:03 smithi106 ceph-mon[44423]: pgmap v1367: 17 pgs: 17 active+clean; 4.8 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-20T17:28:03.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:03 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:28:03.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:03 smithi106 ceph-mon[44423]: from='client.26426 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:28:03.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:03 smithi106 ceph-mon[44423]: osdmap e520: 12 total, 11 up, 11 in 2024-04-20T17:28:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:03 smithi171 ceph-mon[44189]: pgmap v1367: 17 pgs: 17 active+clean; 4.8 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-20T17:28:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:03 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:28:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:03 smithi171 ceph-mon[44189]: from='client.26426 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:28:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:03 smithi171 ceph-mon[44189]: osdmap e520: 12 total, 11 up, 11 in 2024-04-20T17:28:04.410 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:28:04.781 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:28:04.782 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:28:04.811 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:04 smithi005 ceph-mon[39562]: 4.3 scrub starts 2024-04-20T17:28:04.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:04 smithi171 ceph-mon[44189]: 4.3 scrub starts 2024-04-20T17:28:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:04 smithi106 ceph-mon[44423]: 4.3 scrub starts 2024-04-20T17:28:05.785 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:05 smithi005 ceph-mon[39562]: pgmap v1369: 17 pgs: 17 active+clean; 4.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-20T17:28:05.786 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:05 smithi005 ceph-mon[39562]: 4.3 scrub ok 2024-04-20T17:28:05.786 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:05 smithi005 ceph-mon[39562]: from='client.26432 v1:172.21.15.5:0/240376780' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:28:05.812 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:28:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:05 smithi171 ceph-mon[44189]: pgmap v1369: 17 pgs: 17 active+clean; 4.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-20T17:28:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:05 smithi171 ceph-mon[44189]: 4.3 scrub ok 2024-04-20T17:28:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:05 smithi171 ceph-mon[44189]: from='client.26432 v1:172.21.15.5:0/240376780' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:28:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:05 smithi106 ceph-mon[44423]: pgmap v1369: 17 pgs: 17 active+clean; 4.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-20T17:28:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:05 smithi106 ceph-mon[44423]: 4.3 scrub ok 2024-04-20T17:28:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:05 smithi106 ceph-mon[44423]: from='client.26432 v1:172.21.15.5:0/240376780' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:28:06.577 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:28:06.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:06.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:28:06.682 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:28:06.714 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:06.737 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:28:06.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:06.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:28:06.843 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:28:06.880 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:06.910 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:28:06.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:06.917 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:28:07.018 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:28:07.050 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:07.072 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:28:07.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:07.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:28:07.176 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:28:07.216 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:07.242 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:28:07.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:07.250 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:28:07.352 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:28:07.401 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:07.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:07 smithi005 ceph-mon[39562]: pgmap v1370: 17 pgs: 17 active+clean; 4.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-20T17:28:07.424 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:07 smithi005 ceph-mon[39562]: 4.f scrub starts 2024-04-20T17:28:07.432 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:28:07.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:07.436 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:28:07.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:07 smithi106 ceph-mon[44423]: pgmap v1370: 17 pgs: 17 active+clean; 4.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-20T17:28:07.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:07 smithi106 ceph-mon[44423]: 4.f scrub starts 2024-04-20T17:28:07.536 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:28:07.583 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:07.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:07 smithi171 ceph-mon[44189]: pgmap v1370: 17 pgs: 17 active+clean; 4.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-20T17:28:07.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:07 smithi171 ceph-mon[44189]: 4.f scrub starts 2024-04-20T17:28:08.014 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":1369,"stamp":"2024-04-20T17:28:04.248498+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4964287032,"num_objects":75743,"num_object_clones":0,"num_object_copies":151486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75743,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":264820,"num_write_kb":10895035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11554,"num_bytes_recovered":758794368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160878,"ondisk_log_size":160878,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":12526044,"kb_used_data":9765372,"kb_used_omap":223,"kb_used_meta":2760352,"kb_avail":1018670628,"statfs":{"total":1055945392128,"available":1043118723072,"internally_reserved":0,"allocated":9999740928,"data_stored":9996099368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":228532,"internal_metadata":2826601292},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-787152896,"num_objects":-12011,"num_object_clones":0,"num_object_copies":-24022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12011,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005155"},"pg_stats":[{"pgid":"4.f","version":"519'16543","reported_seq":16806,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-04-20T17:28:01.369791+0000","last_change":"2024-04-20T17:26:34.139365+0000","last_active":"2024-04-20T17:28:01.369791+0000","last_peered":"2024-04-20T17:28:01.369791+0000","last_clean":"2024-04-20T17:28:01.369791+0000","last_became_active":"2024-04-20T17:24:36.727388+0000","last_became_peered":"2024-04-20T17:24:36.727388+0000","last_unstale":"2024-04-20T17:28:01.369791+0000","last_undegraded":"2024-04-20T17:28:01.369791+0000","last_fullsized":"2024-04-20T17:28:01.369791+0000","mapping_epoch":445,"log_start":"466'6500","ondisk_log_start":"466'6500","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'11858","last_scrub_stamp":"2024-04-20T17:26:34.139195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:34.139195+0000","objects_scrubbed":9615,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:28:03.616460+0000","scrub_duration":2286,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316866560,"num_objects":4835,"num_object_clones":0,"num_object_copies":9670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16543,"num_write_kb":684096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"519'16229","reported_seq":16582,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-04-20T17:28:00.706477+0000","last_change":"2024-04-20T17:26:29.940846+0000","last_active":"2024-04-20T17:28:00.706477+0000","last_peered":"2024-04-20T17:28:00.706477+0000","last_clean":"2024-04-20T17:28:00.706477+0000","last_became_active":"2024-04-20T17:24:36.573304+0000","last_became_peered":"2024-04-20T17:24:36.573304+0000","last_unstale":"2024-04-20T17:28:00.706477+0000","last_undegraded":"2024-04-20T17:28:00.706477+0000","last_fullsized":"2024-04-20T17:28:00.706477+0000","mapping_epoch":445,"log_start":"465'6200","ondisk_log_start":"465'6200","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'11392","last_scrub_stamp":"2024-04-20T17:26:29.940585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:29.940585+0000","objects_scrubbed":9772,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:28:02.232062+0000","scrub_duration":1744,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319881216,"num_objects":4881,"num_object_clones":0,"num_object_copies":9762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16229,"num_write_kb":675520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"519'16343","reported_seq":16672,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-04-20T17:28:00.746998+0000","last_change":"2024-04-20T17:27:12.446072+0000","last_active":"2024-04-20T17:28:00.746998+0000","last_peered":"2024-04-20T17:28:00.746998+0000","last_clean":"2024-04-20T17:28:00.746998+0000","last_became_active":"2024-04-20T17:24:36.573474+0000","last_became_peered":"2024-04-20T17:24:36.573474+0000","last_unstale":"2024-04-20T17:28:00.746998+0000","last_undegraded":"2024-04-20T17:28:00.746998+0000","last_fullsized":"2024-04-20T17:28:00.746998+0000","mapping_epoch":445,"log_start":"465'6300","ondisk_log_start":"465'6300","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'14624","last_scrub_stamp":"2024-04-20T17:27:12.445846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:27:12.445846+0000","objects_scrubbed":6598,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:28:26.100189+0000","scrub_duration":1092,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317456408,"num_objects":4845,"num_object_clones":0,"num_object_copies":9690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4845,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16343,"num_write_kb":677953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"519'16474","reported_seq":16799,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-04-20T17:28:00.714205+0000","last_change":"2024-04-20T17:27:33.703877+0000","last_active":"2024-04-20T17:28:00.714205+0000","last_peered":"2024-04-20T17:28:00.714205+0000","last_clean":"2024-04-20T17:28:00.714205+0000","last_became_active":"2024-04-20T17:24:36.572684+0000","last_became_peered":"2024-04-20T17:24:36.572684+0000","last_unstale":"2024-04-20T17:28:00.714205+0000","last_undegraded":"2024-04-20T17:28:00.714205+0000","last_fullsized":"2024-04-20T17:28:00.714205+0000","mapping_epoch":445,"log_start":"466'6400","ondisk_log_start":"466'6400","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'15537","last_scrub_stamp":"2024-04-20T17:27:33.703752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:27:33.703752+0000","objects_scrubbed":5781,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:28:57.317422+0000","scrub_duration":482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317456384,"num_objects":4844,"num_object_clones":0,"num_object_copies":9688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16474,"num_write_kb":682176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"519'16611","reported_seq":16947,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-04-20T17:27:59.343402+0000","last_change":"2024-04-20T17:27:10.808368+0000","last_active":"2024-04-20T17:27:59.343402+0000","last_peered":"2024-04-20T17:27:59.343402+0000","last_clean":"2024-04-20T17:27:59.343402+0000","last_became_active":"2024-04-20T17:24:36.726966+0000","last_became_peered":"2024-04-20T17:24:36.726966+0000","last_unstale":"2024-04-20T17:27:59.343402+0000","last_undegraded":"2024-04-20T17:27:59.343402+0000","last_fullsized":"2024-04-20T17:27:59.343402+0000","mapping_epoch":445,"log_start":"466'6600","ondisk_log_start":"466'6600","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'14809","last_scrub_stamp":"2024-04-20T17:27:10.808242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:27:10.808242+0000","objects_scrubbed":6743,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:28:28.593585+0000","scrub_duration":1324,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320929792,"num_objects":4897,"num_object_clones":0,"num_object_copies":9794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16611,"num_write_kb":688256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"520'16516","reported_seq":28401,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-20T17:28:04.143820+0000","last_change":"2024-04-20T17:27:46.381849+0000","last_active":"2024-04-20T17:28:04.143820+0000","last_peered":"2024-04-20T17:28:04.143820+0000","last_clean":"2024-04-20T17:28:04.143820+0000","last_became_active":"2024-04-20T17:27:46.346628+0000","last_became_peered":"2024-04-20T17:27:46.346628+0000","last_unstale":"2024-04-20T17:28:04.143820+0000","last_undegraded":"2024-04-20T17:28:04.143820+0000","last_fullsized":"2024-04-20T17:28:04.143820+0000","mapping_epoch":515,"log_start":"466'6507","ondisk_log_start":"466'6507","created":445,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'8313","last_scrub_stamp":"2024-04-20T17:25:47.229419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:25:47.229419+0000","objects_scrubbed":8261,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:28:06.956155+0000","scrub_duration":1387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301727744,"num_objects":4604,"num_object_clones":0,"num_object_copies":9208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16516,"num_write_kb":675840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5813,"num_bytes_recovered":380960768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.9","version":"520'16637","reported_seq":16961,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-20T17:28:03.287078+0000","last_change":"2024-04-20T17:27:06.351709+0000","last_active":"2024-04-20T17:28:03.287078+0000","last_peered":"2024-04-20T17:28:03.287078+0000","last_clean":"2024-04-20T17:28:03.287078+0000","last_became_active":"2024-04-20T17:24:36.727451+0000","last_became_peered":"2024-04-20T17:24:36.727451+0000","last_unstale":"2024-04-20T17:28:03.287078+0000","last_undegraded":"2024-04-20T17:28:03.287078+0000","last_fullsized":"2024-04-20T17:28:03.287078+0000","mapping_epoch":445,"log_start":"466'6600","ondisk_log_start":"466'6600","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"496'14292","last_scrub_stamp":"2024-04-20T17:27:06.351549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:27:06.351549+0000","objects_scrubbed":7062,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:28:14.057017+0000","scrub_duration":2072,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303628288,"num_objects":4633,"num_object_clones":0,"num_object_copies":9266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16637,"num_write_kb":680640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"520'16439","reported_seq":16769,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-20T17:28:03.271394+0000","last_change":"2024-04-20T17:27:03.418089+0000","last_active":"2024-04-20T17:28:03.271394+0000","last_peered":"2024-04-20T17:28:03.271394+0000","last_clean":"2024-04-20T17:28:03.271394+0000","last_became_active":"2024-04-20T17:24:36.895302+0000","last_became_peered":"2024-04-20T17:24:36.895302+0000","last_unstale":"2024-04-20T17:28:03.271394+0000","last_undegraded":"2024-04-20T17:28:03.271394+0000","last_fullsized":"2024-04-20T17:28:03.271394+0000","mapping_epoch":445,"log_start":"466'6400","ondisk_log_start":"466'6400","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"496'13971","last_scrub_stamp":"2024-04-20T17:27:03.418005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:27:03.418005+0000","objects_scrubbed":7260,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:28:08.949972+0000","scrub_duration":1113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311754752,"num_objects":4757,"num_object_clones":0,"num_object_copies":9514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16439,"num_write_kb":678272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"520'16795","reported_seq":17093,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-20T17:28:04.130319+0000","last_change":"2024-04-20T17:26:28.991603+0000","last_active":"2024-04-20T17:28:04.130319+0000","last_peered":"2024-04-20T17:28:04.130319+0000","last_clean":"2024-04-20T17:28:04.130319+0000","last_became_active":"2024-04-20T17:24:36.727119+0000","last_became_peered":"2024-04-20T17:24:36.727119+0000","last_unstale":"2024-04-20T17:28:04.130319+0000","last_undegraded":"2024-04-20T17:28:04.130319+0000","last_fullsized":"2024-04-20T17:28:04.130319+0000","mapping_epoch":445,"log_start":"466'6700","ondisk_log_start":"466'6700","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"487'11515","last_scrub_stamp":"2024-04-20T17:26:28.991420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:28.991420+0000","objects_scrubbed":9918,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:28:06.956453+0000","scrub_duration":2074,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299040768,"num_objects":4563,"num_object_clones":0,"num_object_copies":9126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16795,"num_write_kb":683456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"519'16800","reported_seq":17140,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-04-20T17:28:01.084661+0000","last_change":"2024-04-20T17:27:07.010838+0000","last_active":"2024-04-20T17:28:01.084661+0000","last_peered":"2024-04-20T17:28:01.084661+0000","last_clean":"2024-04-20T17:28:01.084661+0000","last_became_active":"2024-04-20T17:24:36.727225+0000","last_became_peered":"2024-04-20T17:24:36.727225+0000","last_unstale":"2024-04-20T17:28:01.084661+0000","last_undegraded":"2024-04-20T17:28:01.084661+0000","last_fullsized":"2024-04-20T17:28:01.084661+0000","mapping_epoch":445,"log_start":"466'6700","ondisk_log_start":"466'6700","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"496'14636","last_scrub_stamp":"2024-04-20T17:27:07.010636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:27:07.010636+0000","objects_scrubbed":7025,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:28:20.464573+0000","scrub_duration":1326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316407808,"num_objects":4828,"num_object_clones":0,"num_object_copies":9656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16800,"num_write_kb":692096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"519'16213","reported_seq":16549,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-04-20T17:28:00.988181+0000","last_change":"2024-04-20T17:27:03.397197+0000","last_active":"2024-04-20T17:28:00.988181+0000","last_peered":"2024-04-20T17:28:00.988181+0000","last_clean":"2024-04-20T17:28:00.988181+0000","last_became_active":"2024-04-20T17:24:36.727028+0000","last_became_peered":"2024-04-20T17:24:36.727028+0000","last_unstale":"2024-04-20T17:28:00.988181+0000","last_undegraded":"2024-04-20T17:28:00.988181+0000","last_fullsized":"2024-04-20T17:28:00.988181+0000","mapping_epoch":445,"log_start":"465'6200","ondisk_log_start":"465'6200","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"496'13881","last_scrub_stamp":"2024-04-20T17:27:03.396894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:27:03.396894+0000","objects_scrubbed":7099,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:28:23.160149+0000","scrub_duration":1458,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309002240,"num_objects":4715,"num_object_clones":0,"num_object_copies":9430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16213,"num_write_kb":669696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"520'16482","reported_seq":16744,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-20T17:28:03.265941+0000","last_change":"2024-04-20T17:26:32.955835+0000","last_active":"2024-04-20T17:28:03.265941+0000","last_peered":"2024-04-20T17:28:03.265941+0000","last_clean":"2024-04-20T17:28:03.265941+0000","last_became_active":"2024-04-20T17:24:36.895348+0000","last_became_peered":"2024-04-20T17:24:36.895348+0000","last_unstale":"2024-04-20T17:28:03.265941+0000","last_undegraded":"2024-04-20T17:28:03.265941+0000","last_fullsized":"2024-04-20T17:28:03.265941+0000","mapping_epoch":445,"log_start":"466'6400","ondisk_log_start":"466'6400","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'11582","last_scrub_stamp":"2024-04-20T17:26:32.955650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:32.955650+0000","objects_scrubbed":9541,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1635,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300941312,"num_objects":4592,"num_object_clones":0,"num_object_copies":9184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16482,"num_write_kb":674368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"519'19326","reported_seq":19749,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-04-20T17:28:01.077425+0000","last_change":"2024-04-20T17:26:26.494294+0000","last_active":"2024-04-20T17:28:01.077425+0000","last_peered":"2024-04-20T17:28:01.077425+0000","last_clean":"2024-04-20T17:28:01.077425+0000","last_became_active":"2024-04-20T17:24:37.009687+0000","last_became_peered":"2024-04-20T17:24:37.009687+0000","last_unstale":"2024-04-20T17:28:01.077425+0000","last_undegraded":"2024-04-20T17:28:01.077425+0000","last_fullsized":"2024-04-20T17:28:01.077425+0000","mapping_epoch":445,"log_start":"475'9301","ondisk_log_start":"475'9301","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"487'14083","last_scrub_stamp":"2024-04-20T17:26:26.494158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:26.494158+0000","objects_scrubbed":10068,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:28:03.720496+0000","scrub_duration":1814,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311689216,"num_objects":4756,"num_object_clones":0,"num_object_copies":9512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16516,"num_write_kb":680704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"520'16625","reported_seq":16942,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-20T17:28:03.276909+0000","last_change":"2024-04-20T17:27:27.933912+0000","last_active":"2024-04-20T17:28:03.276909+0000","last_peered":"2024-04-20T17:28:03.276909+0000","last_clean":"2024-04-20T17:28:03.276909+0000","last_became_active":"2024-04-20T17:24:37.009803+0000","last_became_peered":"2024-04-20T17:24:37.009803+0000","last_unstale":"2024-04-20T17:28:03.276909+0000","last_undegraded":"2024-04-20T17:28:03.276909+0000","last_fullsized":"2024-04-20T17:28:03.276909+0000","mapping_epoch":445,"log_start":"466'6600","ondisk_log_start":"466'6600","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'15551","last_scrub_stamp":"2024-04-20T17:27:27.933796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:27:27.933796+0000","objects_scrubbed":5657,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:28:50.363996+0000","scrub_duration":489,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300351488,"num_objects":4583,"num_object_clones":0,"num_object_copies":9166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16625,"num_write_kb":678656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"519'16689","reported_seq":17021,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-04-20T17:28:01.369632+0000","last_change":"2024-04-20T17:27:08.825322+0000","last_active":"2024-04-20T17:28:01.369632+0000","last_peered":"2024-04-20T17:28:01.369632+0000","last_clean":"2024-04-20T17:28:01.369632+0000","last_became_active":"2024-04-20T17:24:36.895371+0000","last_became_peered":"2024-04-20T17:24:36.895371+0000","last_unstale":"2024-04-20T17:28:01.369632+0000","last_undegraded":"2024-04-20T17:28:01.369632+0000","last_fullsized":"2024-04-20T17:28:01.369632+0000","mapping_epoch":445,"log_start":"466'6600","ondisk_log_start":"466'6600","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"497'14671","last_scrub_stamp":"2024-04-20T17:27:08.825116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:27:08.825116+0000","objects_scrubbed":6901,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:28:13.899253+0000","scrub_duration":1664,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314638336,"num_objects":4801,"num_object_clones":0,"num_object_copies":9602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16689,"num_write_kb":687680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"520'16727","reported_seq":28469,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-20T17:28:04.129020+0000","last_change":"2024-04-20T17:27:48.390344+0000","last_active":"2024-04-20T17:28:04.129020+0000","last_peered":"2024-04-20T17:28:04.129020+0000","last_clean":"2024-04-20T17:28:04.129020+0000","last_became_active":"2024-04-20T17:27:48.359386+0000","last_became_peered":"2024-04-20T17:27:48.359386+0000","last_unstale":"2024-04-20T17:28:04.129020+0000","last_undegraded":"2024-04-20T17:28:04.129020+0000","last_fullsized":"2024-04-20T17:28:04.129020+0000","mapping_epoch":517,"log_start":"467'6700","ondisk_log_start":"467'6700","created":445,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"489'11622","last_scrub_stamp":"2024-04-20T17:26:30.993531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:30.993531+0000","objects_scrubbed":9793,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:28:06.956191+0000","scrub_duration":2078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301924352,"num_objects":4607,"num_object_clones":0,"num_object_copies":9214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16727,"num_write_kb":682688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5733,"num_bytes_recovered":375717888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":565,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-04-20T17:27:53.440054+0000","last_change":"2024-04-20T17:27:05.254171+0000","last_active":"2024-04-20T17:27:53.440054+0000","last_peered":"2024-04-20T17:27:53.440054+0000","last_clean":"2024-04-20T17:27:53.440054+0000","last_became_active":"2024-04-20T17:23:24.109202+0000","last_became_peered":"2024-04-20T17:23:24.109202+0000","last_unstale":"2024-04-20T17:27:53.440054+0000","last_undegraded":"2024-04-20T17:27:53.440054+0000","last_fullsized":"2024-04-20T17:27:53.440054+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:27:05.254029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:27:05.254029+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:28:33.580617+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":4963696664,"num_objects":75741,"num_object_clones":0,"num_object_copies":151482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75741,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":264639,"num_write_kb":10892097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11546,"num_bytes_recovered":756678656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10718814208,"data_stored":10718806064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160741,"ondisk_log_size":160741,"up":32,"acting":32,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991500,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1906592,"kb_used_data":1506132,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":91838560,"statfs":{"total":95995035648,"available":94042685440,"internally_reserved":0,"allocated":1542279168,"data_stored":1541941532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":410057156},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253139,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1470228,"kb_used_data":1182220,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":92274924,"statfs":{"total":95995035648,"available":94489522176,"internally_reserved":0,"allocated":1210593280,"data_stored":1210259196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":294910400},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612888,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1226060,"kb_used_data":928324,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":92519092,"statfs":{"total":95995035648,"available":94739550208,"internally_reserved":0,"allocated":950603776,"data_stored":950271694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":304871872},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809119,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":154008,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93591144,"statfs":{"total":95995035648,"available":95837331456,"internally_reserved":0,"allocated":667648,"data_stored":333584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":398,"seq":1709396983875,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1050640,"kb_used_data":913800,"kb_used_omap":6,"kb_used_meta":136825,"kb_avail":92694512,"statfs":{"total":95995035648,"available":94919180288,"internally_reserved":0,"allocated":935731200,"data_stored":935399638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6767,"internal_metadata":140109201},"hb_peers":[0,1,2,3,4,5,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575168,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2481096,"kb_used_data":2120896,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":91264056,"statfs":{"total":95995035648,"available":93454393344,"internally_reserved":0,"allocated":2171797504,"data_stored":2171469494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738692,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":634056,"kb_used_data":314112,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":93111096,"statfs":{"total":95995035648,"available":95345762304,"internally_reserved":0,"allocated":321650688,"data_stored":321322678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393983,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":372168,"kb_used_data":305728,"kb_used_omap":202,"kb_used_meta":66229,"kb_avail":93372984,"statfs":{"total":95995035648,"available":95613935616,"internally_reserved":0,"allocated":313065472,"data_stored":312737462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":67818967},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182709,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1239560,"kb_used_data":937664,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92505592,"statfs":{"total":95995035648,"available":94725726208,"internally_reserved":0,"allocated":960167936,"data_stored":959839926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":309131719},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986480,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1137228,"kb_used_data":939460,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":92607924,"statfs":{"total":95995035648,"available":94830514176,"internally_reserved":0,"allocated":962007040,"data_stored":961674958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790251,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":854408,"kb_used_data":616384,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":92890744,"statfs":{"total":95995035648,"available":95120121856,"internally_reserved":0,"allocated":631177216,"data_stored":630849206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":320929792,"data_stored":320929792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2155151360,"data_stored":2155151360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":309002240,"data_stored":309002240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":945881088,"data_stored":945881088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":954667008,"data_stored":954662936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":625016832,"data_stored":625016832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":804257792,"data_stored":804257792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":931659776,"data_stored":931659776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":943722496,"data_stored":943718424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1206321152,"data_stored":1206321152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1522204672,"data_stored":1522204672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:28:08.016 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:28:08.229 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:08 smithi005 ceph-mon[39562]: 4.e scrub starts 2024-04-20T17:28:08.229 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:08 smithi005 ceph-mon[39562]: 4.f scrub ok 2024-04-20T17:28:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:08 smithi106 ceph-mon[44423]: 4.e scrub starts 2024-04-20T17:28:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:08 smithi106 ceph-mon[44423]: 4.f scrub ok 2024-04-20T17:28:08.567 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:28:08.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:08.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:28:08.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:08 smithi171 ceph-mon[44189]: 4.e scrub starts 2024-04-20T17:28:08.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:08 smithi171 ceph-mon[44189]: 4.f scrub ok 2024-04-20T17:28:08.671 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:28:08.714 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:08.737 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:28:08.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:08.744 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:28:08.837 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:28:08.844 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:28:08.934 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:28:08.945 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:08.981 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:28:09.021 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:28:09.064 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:28:09.071 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:28:09.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:09.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:28:09.107 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:28:09.144 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:28:09.178 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:28:09.184 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:28:09.219 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:28:09.222 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:09.253 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:28:09.287 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:28:09.319 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:28:09.320 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:28:09.320 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:28:09.320 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:28:09.320 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:28:09.320 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:28:09.320 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:28:09.320 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-20T17:28:09.320 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:28:09.320 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:28:09.320 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:28:09.321 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:28:09.321 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:28:09.504 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:28:09.505 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:28:09.505 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:28:09.505 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:28:09.505 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:28:09.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:09 smithi106 ceph-mon[44423]: 4.e scrub ok 2024-04-20T17:28:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:09 smithi106 ceph-mon[44423]: 4.a scrub starts 2024-04-20T17:28:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:09 smithi106 ceph-mon[44423]: pgmap v1371: 17 pgs: 17 active+clean; 4.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-20T17:28:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:09 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1192578175' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:28:09.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:09.530 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:09 smithi005 ceph-mon[39562]: 4.e scrub ok 2024-04-20T17:28:09.530 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:09 smithi005 ceph-mon[39562]: 4.a scrub starts 2024-04-20T17:28:09.530 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:09 smithi005 ceph-mon[39562]: pgmap v1371: 17 pgs: 17 active+clean; 4.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-20T17:28:09.531 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:09 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1192578175' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:28:09.531 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:09.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:09 smithi171 ceph-mon[44189]: 4.e scrub ok 2024-04-20T17:28:09.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:09 smithi171 ceph-mon[44189]: 4.a scrub starts 2024-04-20T17:28:09.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:09 smithi171 ceph-mon[44189]: pgmap v1371: 17 pgs: 17 active+clean; 4.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-20T17:28:09.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:09 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1192578175' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:28:09.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.410 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:28:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:10 smithi106 ceph-mon[44423]: 4.a scrub ok 2024-04-20T17:28:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.537 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:10 smithi005 ceph-mon[39562]: 4.a scrub ok 2024-04-20T17:28:10.537 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:10 smithi171 ceph-mon[44189]: 4.a scrub ok 2024-04-20T17:28:10.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:10.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:11.306 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:28:11.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:11.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:28:11.411 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:28:11.451 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:11 smithi106 ceph-mon[44423]: 4.5 scrub starts 2024-04-20T17:28:11.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:11 smithi106 ceph-mon[44423]: pgmap v1372: 17 pgs: 17 active+clean; 4.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-20T17:28:11.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:11 smithi171 ceph-mon[44189]: 4.5 scrub starts 2024-04-20T17:28:11.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:11 smithi171 ceph-mon[44189]: pgmap v1372: 17 pgs: 17 active+clean; 4.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-20T17:28:11.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:11 smithi005 ceph-mon[39562]: 4.5 scrub starts 2024-04-20T17:28:11.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:11 smithi005 ceph-mon[39562]: pgmap v1372: 17 pgs: 17 active+clean; 4.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-20T17:28:11.787 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:28:11.788 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:28:12.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:12 smithi005 ceph-mon[39562]: 4.5 scrub ok 2024-04-20T17:28:12.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:12 smithi106 ceph-mon[44423]: 4.5 scrub ok 2024-04-20T17:28:12.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:12 smithi171 ceph-mon[44189]: 4.5 scrub ok 2024-04-20T17:28:12.596 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-20T17:28:12.842 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:28:12.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:12.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:28:12.947 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:28:12.988 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:13.205 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:28:13.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:13.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:28:13.264 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:28:13.310 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:28:13.336 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:13.361 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:28:13.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:13.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:28:13.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:13 smithi005 ceph-mon[39562]: pgmap v1373: 17 pgs: 17 active+clean; 4.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:28:13.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:13 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3981170782' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:28:13.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:13 smithi005 ceph-mon[39562]: from='client.26441 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:28:13.465 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:28:13.511 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:13.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:13 smithi106 ceph-mon[44423]: pgmap v1373: 17 pgs: 17 active+clean; 4.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:28:13.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:13 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3981170782' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:28:13.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:13 smithi106 ceph-mon[44423]: from='client.26441 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:28:13.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:13 smithi171 ceph-mon[44189]: pgmap v1373: 17 pgs: 17 active+clean; 4.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:28:13.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:13 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3981170782' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:28:13.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:13 smithi171 ceph-mon[44189]: from='client.26441 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:28:13.789 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:28:14.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:14 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:28:14.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:14 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:28:14.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:14 smithi005 ceph-mon[39562]: from='client.26441 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:28:14.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:14 smithi005 ceph-mon[39562]: osdmap e521: 12 total, 11 up, 11 in 2024-04-20T17:28:14.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:14 smithi005 ceph-mon[39562]: 4.8 deep-scrub starts 2024-04-20T17:28:14.508 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:28:14.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:14.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:28:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:14 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:28:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:14 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:28:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:14 smithi171 ceph-mon[44189]: from='client.26441 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:28:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:14 smithi171 ceph-mon[44189]: osdmap e521: 12 total, 11 up, 11 in 2024-04-20T17:28:14.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:14 smithi171 ceph-mon[44189]: 4.8 deep-scrub starts 2024-04-20T17:28:14.616 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:28:14.669 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:14.757 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:28:14.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:14 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:28:14.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:14 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:28:14.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:14 smithi106 ceph-mon[44423]: from='client.26441 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:28:14.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:14 smithi106 ceph-mon[44423]: osdmap e521: 12 total, 11 up, 11 in 2024-04-20T17:28:14.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:14 smithi106 ceph-mon[44423]: 4.8 deep-scrub starts 2024-04-20T17:28:14.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:14.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:28:14.864 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:28:14.919 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:14.950 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:28:14.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:14.954 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:28:15.046 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:28:15.055 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:28:15.105 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:28:15.108 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:15.355 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:15 smithi005 ceph-mon[39562]: 4.7 scrub starts 2024-04-20T17:28:15.355 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:15 smithi005 ceph-mon[39562]: pgmap v1375: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.9 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:28:15.356 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:15 smithi005 ceph-mon[39562]: 4.9 scrub starts 2024-04-20T17:28:15.369 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:28:15.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:15.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:28:15.413 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:28:15.424 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:28:15.474 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:28:15.520 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:15 smithi171 ceph-mon[44189]: 4.7 scrub starts 2024-04-20T17:28:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:15 smithi171 ceph-mon[44189]: pgmap v1375: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.9 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:28:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:15 smithi171 ceph-mon[44189]: 4.9 scrub starts 2024-04-20T17:28:15.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:15 smithi106 ceph-mon[44423]: 4.7 scrub starts 2024-04-20T17:28:15.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:15 smithi106 ceph-mon[44423]: pgmap v1375: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.9 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:28:15.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:15 smithi106 ceph-mon[44423]: 4.9 scrub starts 2024-04-20T17:28:16.316 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:16 smithi005 ceph-mon[39562]: 4.7 scrub ok 2024-04-20T17:28:16.317 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:16 smithi005 ceph-mon[39562]: from='client.16389 v1:172.21.15.5:0/3564240279' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:28:16.317 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:16 smithi005 ceph-mon[39562]: 4.9 scrub ok 2024-04-20T17:28:16.317 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:16 smithi005 ceph-mon[39562]: 4.0 deep-scrub starts 2024-04-20T17:28:16.531 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:28:16.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:16 smithi171 ceph-mon[44189]: 4.7 scrub ok 2024-04-20T17:28:16.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:16 smithi171 ceph-mon[44189]: from='client.16389 v1:172.21.15.5:0/3564240279' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:28:16.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:16 smithi171 ceph-mon[44189]: 4.9 scrub ok 2024-04-20T17:28:16.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:16 smithi171 ceph-mon[44189]: 4.0 deep-scrub starts 2024-04-20T17:28:16.682 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:28:16.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:16.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:28:16.712 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":1375,"stamp":"2024-04-20T17:28:14.257131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4190831160,"num_objects":63941,"num_object_clones":0,"num_object_copies":127882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63941,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":276622,"num_write_kb":10895035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11554,"num_bytes_recovered":758794368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161180,"ondisk_log_size":161180,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":11058376,"kb_used_data":8251816,"kb_used_omap":223,"kb_used_meta":2806176,"kb_avail":1020138296,"statfs":{"total":1055945392128,"available":1044621615104,"internally_reserved":0,"allocated":8449859584,"data_stored":8446163538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":228532,"internal_metadata":2873525068},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,5,6,5,8,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-773455872,"num_objects":-11802,"num_object_clones":0,"num_object_copies":-23604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11802,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.008632"},"pg_stats":[{"pgid":"4.f","version":"520'17288","reported_seq":17606,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-20T17:28:11.354481+0000","last_change":"2024-04-20T17:28:07.573178+0000","last_active":"2024-04-20T17:28:11.354481+0000","last_peered":"2024-04-20T17:28:11.354481+0000","last_clean":"2024-04-20T17:28:11.354481+0000","last_became_active":"2024-04-20T17:24:36.727388+0000","last_became_peered":"2024-04-20T17:24:36.727388+0000","last_unstale":"2024-04-20T17:28:11.354481+0000","last_undegraded":"2024-04-20T17:28:11.354481+0000","last_fullsized":"2024-04-20T17:28:11.354481+0000","mapping_epoch":445,"log_start":"468'7200","ondisk_log_start":"468'7200","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"520'17014","last_scrub_stamp":"2024-04-20T17:28:07.572947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:28:07.572947+0000","objects_scrubbed":4396,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:29:36.608617+0000","scrub_duration":965,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268042240,"num_objects":4090,"num_object_clones":0,"num_object_copies":8180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17288,"num_write_kb":684096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"520'16971","reported_seq":17377,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-20T17:28:10.754441+0000","last_change":"2024-04-20T17:28:07.942489+0000","last_active":"2024-04-20T17:28:10.754441+0000","last_peered":"2024-04-20T17:28:10.754441+0000","last_clean":"2024-04-20T17:28:10.754441+0000","last_became_active":"2024-04-20T17:24:36.573304+0000","last_became_peered":"2024-04-20T17:24:36.573304+0000","last_unstale":"2024-04-20T17:28:10.754441+0000","last_undegraded":"2024-04-20T17:28:10.754441+0000","last_fullsized":"2024-04-20T17:28:10.754441+0000","mapping_epoch":445,"log_start":"468'6900","ondisk_log_start":"468'6900","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"520'16753","last_scrub_stamp":"2024-04-20T17:28:07.942273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:28:07.942273+0000","objects_scrubbed":4374,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:29:18.655187+0000","scrub_duration":627,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271253504,"num_objects":4139,"num_object_clones":0,"num_object_copies":8278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16971,"num_write_kb":675520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"520'17096","reported_seq":17427,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-20T17:28:10.772437+0000","last_change":"2024-04-20T17:27:12.446072+0000","last_active":"2024-04-20T17:28:10.772437+0000","last_peered":"2024-04-20T17:28:10.772437+0000","last_clean":"2024-04-20T17:28:10.772437+0000","last_became_active":"2024-04-20T17:24:36.573474+0000","last_became_peered":"2024-04-20T17:24:36.573474+0000","last_unstale":"2024-04-20T17:28:10.772437+0000","last_undegraded":"2024-04-20T17:28:10.772437+0000","last_fullsized":"2024-04-20T17:28:10.772437+0000","mapping_epoch":445,"log_start":"468'7000","ondisk_log_start":"468'7000","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'14624","last_scrub_stamp":"2024-04-20T17:27:12.445846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:27:12.445846+0000","objects_scrubbed":6598,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:28:26.100189+0000","scrub_duration":1092,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268107800,"num_objects":4092,"num_object_clones":0,"num_object_copies":8184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4092,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":17096,"num_write_kb":677953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"520'17198","reported_seq":17525,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-20T17:28:10.734451+0000","last_change":"2024-04-20T17:27:33.703877+0000","last_active":"2024-04-20T17:28:10.734451+0000","last_peered":"2024-04-20T17:28:10.734451+0000","last_clean":"2024-04-20T17:28:10.734451+0000","last_became_active":"2024-04-20T17:24:36.572684+0000","last_became_peered":"2024-04-20T17:24:36.572684+0000","last_unstale":"2024-04-20T17:28:10.734451+0000","last_undegraded":"2024-04-20T17:28:10.734451+0000","last_fullsized":"2024-04-20T17:28:10.734451+0000","mapping_epoch":445,"log_start":"468'7100","ondisk_log_start":"468'7100","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'15537","last_scrub_stamp":"2024-04-20T17:27:33.703752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:27:33.703752+0000","objects_scrubbed":5781,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:28:57.317422+0000","scrub_duration":482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270008320,"num_objects":4120,"num_object_clones":0,"num_object_copies":8240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17198,"num_write_kb":682176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"520'17351","reported_seq":17689,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-20T17:28:09.352137+0000","last_change":"2024-04-20T17:27:10.808368+0000","last_active":"2024-04-20T17:28:09.352137+0000","last_peered":"2024-04-20T17:28:09.352137+0000","last_clean":"2024-04-20T17:28:09.352137+0000","last_became_active":"2024-04-20T17:24:36.726966+0000","last_became_peered":"2024-04-20T17:24:36.726966+0000","last_unstale":"2024-04-20T17:28:09.352137+0000","last_undegraded":"2024-04-20T17:28:09.352137+0000","last_fullsized":"2024-04-20T17:28:09.352137+0000","mapping_epoch":445,"log_start":"468'7300","ondisk_log_start":"468'7300","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'14809","last_scrub_stamp":"2024-04-20T17:27:10.808242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:27:10.808242+0000","objects_scrubbed":6743,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:28:28.593585+0000","scrub_duration":1324,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272433152,"num_objects":4157,"num_object_clones":0,"num_object_copies":8314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17351,"num_write_kb":688256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"521'17279","reported_seq":29217,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-04-20T17:28:14.159052+0000","last_change":"2024-04-20T17:28:08.952814+0000","last_active":"2024-04-20T17:28:14.159052+0000","last_peered":"2024-04-20T17:28:14.159052+0000","last_clean":"2024-04-20T17:28:14.159052+0000","last_became_active":"2024-04-20T17:27:46.346628+0000","last_became_peered":"2024-04-20T17:27:46.346628+0000","last_unstale":"2024-04-20T17:28:14.159052+0000","last_undegraded":"2024-04-20T17:28:14.159052+0000","last_fullsized":"2024-04-20T17:28:14.159052+0000","mapping_epoch":515,"log_start":"468'7207","ondisk_log_start":"468'7207","created":445,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"520'16893","last_scrub_stamp":"2024-04-20T17:28:08.952689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:28:08.952689+0000","objects_scrubbed":4261,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:29:22.645977+0000","scrub_duration":878,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251723776,"num_objects":3841,"num_object_clones":0,"num_object_copies":7682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17279,"num_write_kb":675840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5813,"num_bytes_recovered":380960768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.9","version":"521'17385","reported_seq":17710,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-04-20T17:28:13.291519+0000","last_change":"2024-04-20T17:27:06.351709+0000","last_active":"2024-04-20T17:28:13.291519+0000","last_peered":"2024-04-20T17:28:13.291519+0000","last_clean":"2024-04-20T17:28:13.291519+0000","last_became_active":"2024-04-20T17:24:36.727451+0000","last_became_peered":"2024-04-20T17:24:36.727451+0000","last_unstale":"2024-04-20T17:28:13.291519+0000","last_undegraded":"2024-04-20T17:28:13.291519+0000","last_fullsized":"2024-04-20T17:28:13.291519+0000","mapping_epoch":445,"log_start":"468'7300","ondisk_log_start":"468'7300","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"496'14292","last_scrub_stamp":"2024-04-20T17:27:06.351549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:27:06.351549+0000","objects_scrubbed":7062,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:28:14.057017+0000","scrub_duration":2072,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254607360,"num_objects":3885,"num_object_clones":0,"num_object_copies":7770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17385,"num_write_kb":680640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"521'17219","reported_seq":17551,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-04-20T17:28:13.278110+0000","last_change":"2024-04-20T17:27:03.418089+0000","last_active":"2024-04-20T17:28:13.278110+0000","last_peered":"2024-04-20T17:28:13.278110+0000","last_clean":"2024-04-20T17:28:13.278110+0000","last_became_active":"2024-04-20T17:24:36.895302+0000","last_became_peered":"2024-04-20T17:24:36.895302+0000","last_unstale":"2024-04-20T17:28:13.278110+0000","last_undegraded":"2024-04-20T17:28:13.278110+0000","last_fullsized":"2024-04-20T17:28:13.278110+0000","mapping_epoch":445,"log_start":"468'7200","ondisk_log_start":"468'7200","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"496'13971","last_scrub_stamp":"2024-04-20T17:27:03.418005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:27:03.418005+0000","objects_scrubbed":7260,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260636672,"num_objects":3977,"num_object_clones":0,"num_object_copies":7954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17219,"num_write_kb":678272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"521'17522","reported_seq":17829,"reported_epoch":521,"state":"active+clean+scrubbing","last_fresh":"2024-04-20T17:28:14.173661+0000","last_change":"2024-04-20T17:28:14.083337+0000","last_active":"2024-04-20T17:28:14.173661+0000","last_peered":"2024-04-20T17:28:14.173661+0000","last_clean":"2024-04-20T17:28:14.173661+0000","last_became_active":"2024-04-20T17:24:36.727119+0000","last_became_peered":"2024-04-20T17:24:36.727119+0000","last_unstale":"2024-04-20T17:28:14.173661+0000","last_undegraded":"2024-04-20T17:28:14.173661+0000","last_fullsized":"2024-04-20T17:28:14.173661+0000","mapping_epoch":445,"log_start":"470'7500","ondisk_log_start":"470'7500","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"487'11515","last_scrub_stamp":"2024-04-20T17:26:28.991420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:28.991420+0000","objects_scrubbed":399,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"scrubbing for 0s","scrub_duration":2074,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251396096,"num_objects":3836,"num_object_clones":0,"num_object_copies":7672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17522,"num_write_kb":683456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"520'17551","reported_seq":17893,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-20T17:28:10.977045+0000","last_change":"2024-04-20T17:27:07.010838+0000","last_active":"2024-04-20T17:28:10.977045+0000","last_peered":"2024-04-20T17:28:10.977045+0000","last_clean":"2024-04-20T17:28:10.977045+0000","last_became_active":"2024-04-20T17:24:36.727225+0000","last_became_peered":"2024-04-20T17:24:36.727225+0000","last_unstale":"2024-04-20T17:28:10.977045+0000","last_undegraded":"2024-04-20T17:28:10.977045+0000","last_fullsized":"2024-04-20T17:28:10.977045+0000","mapping_epoch":445,"log_start":"468'7500","ondisk_log_start":"468'7500","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"496'14636","last_scrub_stamp":"2024-04-20T17:27:07.010636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:27:07.010636+0000","objects_scrubbed":7025,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:28:20.464573+0000","scrub_duration":1326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267190272,"num_objects":4077,"num_object_clones":0,"num_object_copies":8154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17551,"num_write_kb":692096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"520'16942","reported_seq":17280,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-20T17:28:11.003730+0000","last_change":"2024-04-20T17:27:03.397197+0000","last_active":"2024-04-20T17:28:11.003730+0000","last_peered":"2024-04-20T17:28:11.003730+0000","last_clean":"2024-04-20T17:28:11.003730+0000","last_became_active":"2024-04-20T17:24:36.727028+0000","last_became_peered":"2024-04-20T17:24:36.727028+0000","last_unstale":"2024-04-20T17:28:11.003730+0000","last_undegraded":"2024-04-20T17:28:11.003730+0000","last_fullsized":"2024-04-20T17:28:11.003730+0000","mapping_epoch":445,"log_start":"468'6900","ondisk_log_start":"468'6900","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"496'13881","last_scrub_stamp":"2024-04-20T17:27:03.396894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:27:03.396894+0000","objects_scrubbed":7099,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:28:23.160149+0000","scrub_duration":1458,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261226496,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16942,"num_write_kb":669696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"521'17197","reported_seq":17513,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-04-20T17:28:13.265526+0000","last_change":"2024-04-20T17:28:04.384518+0000","last_active":"2024-04-20T17:28:13.265526+0000","last_peered":"2024-04-20T17:28:13.265526+0000","last_clean":"2024-04-20T17:28:13.265526+0000","last_became_active":"2024-04-20T17:24:36.895348+0000","last_became_peered":"2024-04-20T17:24:36.895348+0000","last_unstale":"2024-04-20T17:28:13.265526+0000","last_undegraded":"2024-04-20T17:28:13.265526+0000","last_fullsized":"2024-04-20T17:28:13.265526+0000","mapping_epoch":445,"log_start":"468'7100","ondisk_log_start":"468'7100","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"520'16565","last_scrub_stamp":"2024-04-20T17:28:04.384399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:28:04.384399+0000","objects_scrubbed":4550,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:29:27.877844+0000","scrub_duration":821,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254083072,"num_objects":3877,"num_object_clones":0,"num_object_copies":7754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17197,"num_write_kb":674368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"520'20054","reported_seq":20479,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-20T17:28:11.071511+0000","last_change":"2024-04-20T17:26:26.494294+0000","last_active":"2024-04-20T17:28:11.071511+0000","last_peered":"2024-04-20T17:28:11.071511+0000","last_clean":"2024-04-20T17:28:11.071511+0000","last_became_active":"2024-04-20T17:24:37.009687+0000","last_became_peered":"2024-04-20T17:24:37.009687+0000","last_unstale":"2024-04-20T17:28:11.071511+0000","last_undegraded":"2024-04-20T17:28:11.071511+0000","last_fullsized":"2024-04-20T17:28:11.071511+0000","mapping_epoch":445,"log_start":"477'10001","ondisk_log_start":"477'10001","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"487'14083","last_scrub_stamp":"2024-04-20T17:26:26.494158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:26:26.494158+0000","objects_scrubbed":10068,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1814,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263979008,"num_objects":4028,"num_object_clones":0,"num_object_copies":8056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17244,"num_write_kb":680704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"521'17376","reported_seq":17695,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-04-20T17:28:13.292883+0000","last_change":"2024-04-20T17:27:27.933912+0000","last_active":"2024-04-20T17:28:13.292883+0000","last_peered":"2024-04-20T17:28:13.292883+0000","last_clean":"2024-04-20T17:28:13.292883+0000","last_became_active":"2024-04-20T17:24:37.009803+0000","last_became_peered":"2024-04-20T17:24:37.009803+0000","last_unstale":"2024-04-20T17:28:13.292883+0000","last_undegraded":"2024-04-20T17:28:13.292883+0000","last_fullsized":"2024-04-20T17:28:13.292883+0000","mapping_epoch":445,"log_start":"468'7300","ondisk_log_start":"468'7300","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'15551","last_scrub_stamp":"2024-04-20T17:27:27.933796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:27:27.933796+0000","objects_scrubbed":5657,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:28:50.363996+0000","scrub_duration":489,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251133952,"num_objects":3832,"num_object_clones":0,"num_object_copies":7664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17376,"num_write_kb":678656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"520'17391","reported_seq":17726,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-20T17:28:11.339062+0000","last_change":"2024-04-20T17:27:08.825322+0000","last_active":"2024-04-20T17:28:11.339062+0000","last_peered":"2024-04-20T17:28:11.339062+0000","last_clean":"2024-04-20T17:28:11.339062+0000","last_became_active":"2024-04-20T17:24:36.895371+0000","last_became_peered":"2024-04-20T17:24:36.895371+0000","last_unstale":"2024-04-20T17:28:11.339062+0000","last_undegraded":"2024-04-20T17:28:11.339062+0000","last_fullsized":"2024-04-20T17:28:11.339062+0000","mapping_epoch":445,"log_start":"468'7300","ondisk_log_start":"468'7300","created":445,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"497'14671","last_scrub_stamp":"2024-04-20T17:27:08.825116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:27:08.825116+0000","objects_scrubbed":6901,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:28:13.899253+0000","scrub_duration":1664,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268632064,"num_objects":4099,"num_object_clones":0,"num_object_copies":8198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17391,"num_write_kb":687680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"521'17431","reported_seq":29225,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-04-20T17:28:14.120660+0000","last_change":"2024-04-20T17:28:10.904093+0000","last_active":"2024-04-20T17:28:14.120660+0000","last_peered":"2024-04-20T17:28:14.120660+0000","last_clean":"2024-04-20T17:28:14.120660+0000","last_became_active":"2024-04-20T17:27:48.359386+0000","last_became_peered":"2024-04-20T17:27:48.359386+0000","last_unstale":"2024-04-20T17:28:14.120660+0000","last_undegraded":"2024-04-20T17:28:14.120660+0000","last_fullsized":"2024-04-20T17:28:14.120660+0000","mapping_epoch":517,"log_start":"470'7400","ondisk_log_start":"470'7400","created":445,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"520'17196","last_scrub_stamp":"2024-04-20T17:28:10.903939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:24:35.556630+0000","last_clean_scrub_stamp":"2024-04-20T17:28:10.903939+0000","objects_scrubbed":4168,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:29:12.236937+0000","scrub_duration":833,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255787008,"num_objects":3903,"num_object_clones":0,"num_object_copies":7806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17431,"num_write_kb":682688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5733,"num_bytes_recovered":375717888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":567,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-20T17:28:02.498663+0000","last_change":"2024-04-20T17:27:05.254171+0000","last_active":"2024-04-20T17:28:02.498663+0000","last_peered":"2024-04-20T17:28:02.498663+0000","last_clean":"2024-04-20T17:28:02.498663+0000","last_became_active":"2024-04-20T17:23:24.109202+0000","last_became_peered":"2024-04-20T17:23:24.109202+0000","last_unstale":"2024-04-20T17:28:02.498663+0000","last_undegraded":"2024-04-20T17:28:02.498663+0000","last_fullsized":"2024-04-20T17:28:02.498663+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:27:05.254029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:27:05.254029+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:28:33.580617+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":4190240792,"num_objects":63939,"num_object_clones":0,"num_object_copies":127878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63939,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":276441,"num_write_kb":10892097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11546,"num_bytes_recovered":756678656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9171836928,"data_stored":9171828784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161043,"ondisk_log_size":161043,"up":32,"acting":32,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991502,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1666076,"kb_used_data":1265620,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":92079076,"statfs":{"total":95995035648,"available":94288973824,"internally_reserved":0,"allocated":1295994880,"data_stored":1295657244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":410057156},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253141,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1290148,"kb_used_data":992408,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":92455004,"statfs":{"total":95995035648,"available":94673924096,"internally_reserved":0,"allocated":1016225792,"data_stored":1015885654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":304871872},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612890,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1110104,"kb_used_data":785936,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92635048,"statfs":{"total":95995035648,"available":94858289152,"internally_reserved":0,"allocated":804798464,"data_stored":804460328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":331872704},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809121,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":154008,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93591144,"statfs":{"total":95995035648,"available":95837331456,"internally_reserved":0,"allocated":667648,"data_stored":333584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":398,"seq":1709396983877,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":899996,"kb_used_data":771476,"kb_used_omap":6,"kb_used_meta":128505,"kb_avail":92845156,"statfs":{"total":95995035648,"available":95073439744,"internally_reserved":0,"allocated":789991424,"data_stored":789653808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6767,"internal_metadata":131589521},"hb_peers":[0,1,2,3,4,5,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575170,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2148500,"kb_used_data":1788300,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":91596652,"statfs":{"total":95995035648,"available":93794971648,"internally_reserved":0,"allocated":1831219200,"data_stored":1830885136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738694,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":595092,"kb_used_data":270988,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":93150060,"statfs":{"total":95995035648,"available":95385661440,"internally_reserved":0,"allocated":277491712,"data_stored":277157648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":202,"seq":867583393985,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":330324,"kb_used_data":259724,"kb_used_omap":202,"kb_used_meta":70389,"kb_avail":93414828,"statfs":{"total":95995035648,"available":95656783872,"internally_reserved":0,"allocated":265957376,"data_stored":265623312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":72078807},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182711,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1123156,"kb_used_data":794892,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92621996,"statfs":{"total":95995035648,"available":94844923904,"internally_reserved":0,"allocated":813969408,"data_stored":813635344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":336132551},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986482,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":991448,"kb_used_data":797840,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":92753704,"statfs":{"total":95995035648,"available":94979792896,"internally_reserved":0,"allocated":816988160,"data_stored":816650024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":198244805},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790253,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":749524,"kb_used_data":523980,"kb_used_omap":1,"kb_used_meta":225534,"kb_avail":92995628,"statfs":{"total":95995035648,"available":95227523072,"internally_reserved":0,"allocated":536555520,"data_stored":536221456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":230947269},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":272433152,"data_stored":272433152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1814036480,"data_stored":1814036480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":261226496,"data_stored":261226496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":804126720,"data_stored":804126720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":809373696,"data_stored":809369624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":529334272,"data_stored":529334272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":804257792,"data_stored":804257792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":785252352,"data_stored":785252352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":797642752,"data_stored":797638680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1013972992,"data_stored":1013972992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1280180224,"data_stored":1280180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:28:16.713 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-20T17:28:16.713 INFO:tasks.thrashosds.thrasher:Waiting for trim 2024-04-20T17:28:16.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:16 smithi106 ceph-mon[44423]: 4.7 scrub ok 2024-04-20T17:28:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:16 smithi106 ceph-mon[44423]: from='client.16389 v1:172.21.15.5:0/3564240279' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:28:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:16 smithi106 ceph-mon[44423]: 4.9 scrub ok 2024-04-20T17:28:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:16 smithi106 ceph-mon[44423]: 4.0 deep-scrub starts 2024-04-20T17:28:16.789 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:28:16.824 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:16.850 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:28:16.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:16.853 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:28:16.954 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:28:16.986 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:17.061 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:28:17.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:17.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:28:17.166 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:28:17.198 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:17.552 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:28:17.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:17.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:28:17.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:17 smithi171 ceph-mon[44189]: 4.2 deep-scrub starts 2024-04-20T17:28:17.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:17 smithi171 ceph-mon[44189]: pgmap v1376: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 3.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-20T17:28:17.658 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:28:17.698 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:17 smithi005 ceph-mon[39562]: 4.2 deep-scrub starts 2024-04-20T17:28:17.698 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:17 smithi005 ceph-mon[39562]: pgmap v1376: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 3.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-20T17:28:17.700 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:17.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:17 smithi106 ceph-mon[44423]: 4.2 deep-scrub starts 2024-04-20T17:28:17.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:17 smithi106 ceph-mon[44423]: pgmap v1376: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 3.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-20T17:28:17.907 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:28:17.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:17.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:28:18.011 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:28:18.048 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:18.078 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:28:18.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:18.081 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:28:18.182 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:28:18.209 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:18.232 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:28:18.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:18.238 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:28:18.339 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:28:18.385 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:18.399 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:28:18.634 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:28:18.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:18.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:28:18.742 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:28:18.780 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:19.036 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:28:19.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:19.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:28:19.146 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:28:19.182 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:19.206 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:28:19.206 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:28:19.206 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:28:19.206 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:28:19.206 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.207 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.25802.0:243442 3.4 3:25fd1bc9:::benchmark_data_smithi171_110765_object100227:head [delete] snapc 0=[] ondisk+write+known_if_redirected e416)", 2024-04-20T17:28:19.207 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:23:29.372681+0000", 2024-04-20T17:28:19.207 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 289.82540405899999, 2024-04-20T17:28:19.207 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.096654121999999995, 2024-04-20T17:28:19.207 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:28:19.207 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:28:19.207 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:28:19.207 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:28:19.207 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.25802", 2024-04-20T17:28:19.207 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:28:19.207 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 243442 2024-04-20T17:28:19.207 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.207 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:28:19.207 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.207 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:28:19.208 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.372681+0000", 2024-04-20T17:28:19.208 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.208 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.208 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.208 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:28:19.208 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.372681+0000", 2024-04-20T17:28:19.208 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.208 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.208 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.208 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:28:19.208 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.372681+0000", 2024-04-20T17:28:19.208 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8400000000000004e-07 2024-04-20T17:28:19.208 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.208 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.208 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:28:19.209 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.372685+0000", 2024-04-20T17:28:19.209 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6030000000000001e-06 2024-04-20T17:28:19.209 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.209 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.209 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:28:19.209 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.372690+0000", 2024-04-20T17:28:19.209 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3279999999999996e-06 2024-04-20T17:28:19.209 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.209 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.209 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:28:19.209 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.372695+0000", 2024-04-20T17:28:19.209 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0259999999999998e-06 2024-04-20T17:28:19.209 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.209 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.210 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:28:19.210 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.377789+0000", 2024-04-20T17:28:19.210 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0050939519999999997 2024-04-20T17:28:19.210 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.210 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.210 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:28:19.210 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.377904+0000", 2024-04-20T17:28:19.210 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011461800000000001 2024-04-20T17:28:19.210 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.210 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.210 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 11", 2024-04-20T17:28:19.210 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.378011+0000", 2024-04-20T17:28:19.210 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010702800000000001 2024-04-20T17:28:19.210 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.211 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.211 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:28:19.211 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.386721+0000", 2024-04-20T17:28:19.211 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0087100849999999994 2024-04-20T17:28:19.211 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.211 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.211 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:28:19.212 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.469245+0000", 2024-04-20T17:28:19.212 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.082523795999999996 2024-04-20T17:28:19.212 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.212 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.212 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:28:19.212 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.469296+0000", 2024-04-20T17:28:19.212 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1668000000000001e-05 2024-04-20T17:28:19.212 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.212 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.212 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:28:19.212 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.469335+0000", 2024-04-20T17:28:19.212 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8334000000000003e-05 2024-04-20T17:28:19.212 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.213 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:28:19.213 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.213 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.213 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.213 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.25802.0:243452 3.4 3:2fa8b7ed:::benchmark_data_smithi171_110765_object100237:head [delete] snapc 0=[] ondisk+write+known_if_redirected e416)", 2024-04-20T17:28:19.213 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:23:29.382317+0000", 2024-04-20T17:28:19.213 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 289.81576786599999, 2024-04-20T17:28:19.213 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.087112933000000004, 2024-04-20T17:28:19.213 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:28:19.213 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:28:19.213 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:28:19.213 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:28:19.213 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.25802", 2024-04-20T17:28:19.213 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:28:19.214 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 243452 2024-04-20T17:28:19.214 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.214 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:28:19.214 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.214 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:28:19.214 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.382317+0000", 2024-04-20T17:28:19.214 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.214 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.214 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.214 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:28:19.214 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.382317+0000", 2024-04-20T17:28:19.214 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.214 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.215 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.215 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:28:19.215 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.382320+0000", 2024-04-20T17:28:19.215 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6519999999999999e-06 2024-04-20T17:28:19.215 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.215 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.215 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:28:19.215 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.382330+0000", 2024-04-20T17:28:19.215 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6029999999999994e-06 2024-04-20T17:28:19.215 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.215 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.215 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:28:19.215 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.382347+0000", 2024-04-20T17:28:19.215 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7016000000000001e-05 2024-04-20T17:28:19.216 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.216 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.216 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:28:19.216 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.382358+0000", 2024-04-20T17:28:19.216 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1229999999999999e-05 2024-04-20T17:28:19.216 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.216 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.216 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:28:19.216 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.382646+0000", 2024-04-20T17:28:19.216 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00028770999999999999 2024-04-20T17:28:19.216 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.216 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.217 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:28:19.217 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.382719+0000", 2024-04-20T17:28:19.217 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2669000000000005e-05 2024-04-20T17:28:19.217 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.217 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.217 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 11", 2024-04-20T17:28:19.217 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.382784+0000", 2024-04-20T17:28:19.217 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5502999999999999e-05 2024-04-20T17:28:19.217 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.217 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.217 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:28:19.217 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.386973+0000", 2024-04-20T17:28:19.217 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041887950000000004 2024-04-20T17:28:19.217 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.217 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.218 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:28:19.218 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.469369+0000", 2024-04-20T17:28:19.218 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.082395741999999994 2024-04-20T17:28:19.218 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.218 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.218 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:28:19.218 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.469402+0000", 2024-04-20T17:28:19.218 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3065000000000003e-05 2024-04-20T17:28:19.218 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.218 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.219 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:28:19.219 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:23:29.469430+0000", 2024-04-20T17:28:19.219 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7948000000000001e-05 2024-04-20T17:28:19.219 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.219 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:28:19.219 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.219 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.219 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.219 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:51496 4.6 e455/445 4:65c60a47:::benchmark_data_smithi171_122772_object51495:head v 455'3348, mlcod=455'3345)", 2024-04-20T17:28:19.219 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:02.037814+0000", 2024-04-20T17:28:19.219 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 197.16027036, 2024-04-20T17:28:19.219 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.086449278000000004, 2024-04-20T17:28:19.219 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:28:19.219 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:28:19.220 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:28:19.220 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:28:19.220 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.220 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:28:19.220 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.037814+0000", 2024-04-20T17:28:19.220 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.220 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.220 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.220 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:28:19.220 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.037814+0000", 2024-04-20T17:28:19.220 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.221 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.221 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.221 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:28:19.221 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.037816+0000", 2024-04-20T17:28:19.221 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4360000000000001e-06 2024-04-20T17:28:19.221 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.221 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.221 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:28:19.221 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.037861+0000", 2024-04-20T17:28:19.221 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4929000000000003e-05 2024-04-20T17:28:19.221 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.221 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.221 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:28:19.222 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.037870+0000", 2024-04-20T17:28:19.222 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9430000000000006e-06 2024-04-20T17:28:19.222 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.222 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.222 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:28:19.222 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.037876+0000", 2024-04-20T17:28:19.222 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4409999999999998e-06 2024-04-20T17:28:19.222 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.223 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.223 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:28:19.223 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.038362+0000", 2024-04-20T17:28:19.223 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00048568999999999999 2024-04-20T17:28:19.223 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.223 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.223 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:28:19.223 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.038381+0000", 2024-04-20T17:28:19.223 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9242999999999999e-05 2024-04-20T17:28:19.223 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.223 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.223 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:28:19.223 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.039822+0000", 2024-04-20T17:28:19.223 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001441178 2024-04-20T17:28:19.224 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.224 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.224 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:28:19.224 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.124264+0000", 2024-04-20T17:28:19.224 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.084441418000000004 2024-04-20T17:28:19.224 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.224 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:28:19.224 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.224 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.224 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.224 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:51502 4.b e455/445 4:d57cdef5:::benchmark_data_smithi171_122772_object51501:head v 455'3266, mlcod=455'3264)", 2024-04-20T17:28:19.224 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:02.038477+0000", 2024-04-20T17:28:19.224 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 197.159607833, 2024-04-20T17:28:19.224 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.088332016999999999, 2024-04-20T17:28:19.225 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:28:19.225 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:28:19.225 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:28:19.225 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:28:19.225 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.225 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:28:19.225 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.038477+0000", 2024-04-20T17:28:19.225 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.225 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.225 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.226 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:28:19.226 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.038477+0000", 2024-04-20T17:28:19.226 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.226 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.226 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.226 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:28:19.226 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.038478+0000", 2024-04-20T17:28:19.226 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4580000000000001e-06 2024-04-20T17:28:19.226 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.226 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.226 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:28:19.226 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.038504+0000", 2024-04-20T17:28:19.226 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5624000000000001e-05 2024-04-20T17:28:19.226 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.227 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.227 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:28:19.227 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.038513+0000", 2024-04-20T17:28:19.227 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9490000000000002e-06 2024-04-20T17:28:19.227 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.227 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.227 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:28:19.227 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.038520+0000", 2024-04-20T17:28:19.227 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9959999999999996e-06 2024-04-20T17:28:19.227 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.227 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.228 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:28:19.228 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.038563+0000", 2024-04-20T17:28:19.228 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3368000000000002e-05 2024-04-20T17:28:19.228 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.228 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.228 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:28:19.228 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.038581+0000", 2024-04-20T17:28:19.228 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7524999999999999e-05 2024-04-20T17:28:19.228 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.228 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.228 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:28:19.229 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.126754+0000", 2024-04-20T17:28:19.229 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.088172871 2024-04-20T17:28:19.229 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.229 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.229 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:28:19.229 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.126809+0000", 2024-04-20T17:28:19.229 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5226000000000003e-05 2024-04-20T17:28:19.229 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.229 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:28:19.229 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.229 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.229 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.230 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:51504 4.6 e455/445 4:66a041f5:::benchmark_data_smithi171_122772_object51503:head v 455'3349, mlcod=455'3345)", 2024-04-20T17:28:19.230 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:02.039938+0000", 2024-04-20T17:28:19.230 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 197.15814634399999, 2024-04-20T17:28:19.230 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.089091318000000003, 2024-04-20T17:28:19.230 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:28:19.230 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:28:19.231 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:28:19.231 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:28:19.231 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.231 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:28:19.231 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.039938+0000", 2024-04-20T17:28:19.231 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.232 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.232 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.232 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:28:19.232 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.039938+0000", 2024-04-20T17:28:19.232 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.232 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.232 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.232 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:28:19.233 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.039940+0000", 2024-04-20T17:28:19.233 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.415e-06 2024-04-20T17:28:19.233 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.233 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.233 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:28:19.233 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.040076+0000", 2024-04-20T17:28:19.233 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013595400000000001 2024-04-20T17:28:19.233 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.233 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.233 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:28:19.233 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.124224+0000", 2024-04-20T17:28:19.233 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.084147949999999999 2024-04-20T17:28:19.233 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.233 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.233 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:28:19.233 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.124237+0000", 2024-04-20T17:28:19.234 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3183999999999999e-05 2024-04-20T17:28:19.234 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.234 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.234 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:28:19.234 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.124308+0000", 2024-04-20T17:28:19.234 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1657999999999997e-05 2024-04-20T17:28:19.234 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.234 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.234 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:28:19.234 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.124331+0000", 2024-04-20T17:28:19.234 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2918999999999999e-05 2024-04-20T17:28:19.234 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.234 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.235 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:28:19.235 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.129000+0000", 2024-04-20T17:28:19.235 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0046690960000000002 2024-04-20T17:28:19.235 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.235 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.235 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:28:19.235 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.129030+0000", 2024-04-20T17:28:19.235 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9142e-05 2024-04-20T17:28:19.235 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.235 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:28:19.235 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.235 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.235 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.236 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:51513 4.a e455/445 4:5c308534:::benchmark_data_smithi171_122772_object51512:head v 455'3138, mlcod=455'3136)", 2024-04-20T17:28:19.236 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:02.045064+0000", 2024-04-20T17:28:19.236 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 197.153020669, 2024-04-20T17:28:19.236 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.084044983000000004, 2024-04-20T17:28:19.236 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:28:19.236 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:28:19.236 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:28:19.236 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:28:19.236 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.236 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:28:19.237 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.045064+0000", 2024-04-20T17:28:19.237 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.237 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.237 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.237 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:28:19.237 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.045064+0000", 2024-04-20T17:28:19.237 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.237 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.237 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.237 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:28:19.237 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.045065+0000", 2024-04-20T17:28:19.237 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0869999999999999e-06 2024-04-20T17:28:19.237 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.238 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.238 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:28:19.238 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.045118+0000", 2024-04-20T17:28:19.238 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2933999999999998e-05 2024-04-20T17:28:19.238 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.238 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.238 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:28:19.239 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.124563+0000", 2024-04-20T17:28:19.239 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.079445509999999997 2024-04-20T17:28:19.239 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.239 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.239 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:28:19.239 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.124571+0000", 2024-04-20T17:28:19.239 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2699999999999999e-06 2024-04-20T17:28:19.239 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.239 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.239 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:28:19.239 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.126109+0000", 2024-04-20T17:28:19.239 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0015385139999999999 2024-04-20T17:28:19.240 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.240 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.240 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:28:19.240 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.126131+0000", 2024-04-20T17:28:19.240 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1313000000000001e-05 2024-04-20T17:28:19.240 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.240 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.240 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:28:19.240 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.129062+0000", 2024-04-20T17:28:19.240 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0029310629999999998 2024-04-20T17:28:19.240 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.240 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.240 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:28:19.240 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.129109+0000", 2024-04-20T17:28:19.240 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7292000000000001e-05 2024-04-20T17:28:19.240 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.241 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:28:19.241 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.241 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.241 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.241 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:51519 4.4 e455/445 4:2ff1b437:::benchmark_data_smithi171_122772_object51518:head v 455'3185, mlcod=455'3183)", 2024-04-20T17:28:19.241 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:02.045175+0000", 2024-04-20T17:28:19.241 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 197.15290984399999, 2024-04-20T17:28:19.241 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.083446963999999998, 2024-04-20T17:28:19.241 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:28:19.241 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:28:19.241 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:28:19.241 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:28:19.241 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.241 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:28:19.242 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.045175+0000", 2024-04-20T17:28:19.242 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.242 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.242 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.242 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:28:19.242 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.045175+0000", 2024-04-20T17:28:19.242 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.242 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.242 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.242 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:28:19.242 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.045176+0000", 2024-04-20T17:28:19.242 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2100000000000001e-06 2024-04-20T17:28:19.242 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.243 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.243 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:28:19.243 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.045196+0000", 2024-04-20T17:28:19.243 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9738e-05 2024-04-20T17:28:19.243 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.243 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.243 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:28:19.243 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.124330+0000", 2024-04-20T17:28:19.243 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.079134322000000007 2024-04-20T17:28:19.243 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.243 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.243 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:28:19.243 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.124340+0000", 2024-04-20T17:28:19.243 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0456e-05 2024-04-20T17:28:19.243 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.244 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.244 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:28:19.244 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.124975+0000", 2024-04-20T17:28:19.244 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00063459699999999998 2024-04-20T17:28:19.244 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.244 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.244 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:28:19.244 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.124995+0000", 2024-04-20T17:28:19.244 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0346000000000001e-05 2024-04-20T17:28:19.244 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.244 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.244 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:28:19.244 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.128577+0000", 2024-04-20T17:28:19.245 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0035820359999999998 2024-04-20T17:28:19.245 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.245 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.245 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:28:19.245 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.128622+0000", 2024-04-20T17:28:19.245 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4258999999999998e-05 2024-04-20T17:28:19.245 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.245 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:28:19.245 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.245 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.245 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.245 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:57137 4.f e457/445 4:f70469a5:::benchmark_data_smithi171_122772_object57136:head v 457'3576, mlcod=457'3574)", 2024-04-20T17:28:19.245 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:04.313573+0000", 2024-04-20T17:28:19.245 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 194.88451164700001, 2024-04-20T17:28:19.245 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057404561999999999, 2024-04-20T17:28:19.246 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:28:19.246 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:28:19.246 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:28:19.246 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:28:19.246 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.246 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:28:19.246 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.313573+0000", 2024-04-20T17:28:19.246 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.246 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.246 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.247 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:28:19.247 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.313573+0000", 2024-04-20T17:28:19.247 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.247 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.247 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.247 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:28:19.247 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.313574+0000", 2024-04-20T17:28:19.247 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.319e-06 2024-04-20T17:28:19.247 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.247 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.247 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:28:19.248 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.313596+0000", 2024-04-20T17:28:19.248 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1600999999999999e-05 2024-04-20T17:28:19.248 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.248 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.248 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:28:19.248 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.365609+0000", 2024-04-20T17:28:19.248 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.052013082000000002 2024-04-20T17:28:19.248 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.248 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.248 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:28:19.248 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.365621+0000", 2024-04-20T17:28:19.248 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1661000000000001e-05 2024-04-20T17:28:19.248 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.248 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.249 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:28:19.249 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.365933+0000", 2024-04-20T17:28:19.249 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00031213500000000001 2024-04-20T17:28:19.249 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.249 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.249 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:28:19.249 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.365959+0000", 2024-04-20T17:28:19.249 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6234999999999999e-05 2024-04-20T17:28:19.249 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.249 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.249 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:28:19.249 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.370947+0000", 2024-04-20T17:28:19.249 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0049884020000000003 2024-04-20T17:28:19.249 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.249 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.250 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:28:19.250 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:04.370977+0000", 2024-04-20T17:28:19.250 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0127000000000001e-05 2024-04-20T17:28:19.250 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.250 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:28:19.250 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.250 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.250 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.250 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:205219 4.a e493/445 4:5a4385d3:::benchmark_data_smithi171_122772_object35027:head v 493'12725, mlcod=493'12723)", 2024-04-20T17:28:19.250 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:47.000582+0000", 2024-04-20T17:28:19.250 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 92.197502712000002, 2024-04-20T17:28:19.250 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.065701915, 2024-04-20T17:28:19.250 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:28:19.250 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:28:19.251 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:28:19.251 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:28:19.251 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.251 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:28:19.251 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:47.000582+0000", 2024-04-20T17:28:19.251 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.251 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.251 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.251 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:28:19.251 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:47.000582+0000", 2024-04-20T17:28:19.251 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.251 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.251 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.251 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:28:19.251 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:47.000583+0000", 2024-04-20T17:28:19.252 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.331e-06 2024-04-20T17:28:19.252 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.252 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.252 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:28:19.252 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:47.000589+0000", 2024-04-20T17:28:19.252 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5339999999999999e-06 2024-04-20T17:28:19.252 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.252 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.252 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:28:19.252 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:47.000596+0000", 2024-04-20T17:28:19.253 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4150000000000002e-06 2024-04-20T17:28:19.253 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.253 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.253 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:28:19.253 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:47.000601+0000", 2024-04-20T17:28:19.253 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0950000000000003e-06 2024-04-20T17:28:19.254 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.254 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.254 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:28:19.254 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:47.000649+0000", 2024-04-20T17:28:19.254 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7784999999999997e-05 2024-04-20T17:28:19.254 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.254 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.255 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:28:19.255 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:47.000661+0000", 2024-04-20T17:28:19.255 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.238e-05 2024-04-20T17:28:19.255 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.255 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.255 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:28:19.255 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:47.066238+0000", 2024-04-20T17:28:19.255 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.065576871999999994 2024-04-20T17:28:19.256 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.256 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.256 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:28:19.256 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:47.066284+0000", 2024-04-20T17:28:19.256 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5503000000000001e-05 2024-04-20T17:28:19.256 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.256 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:28:19.257 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.257 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.257 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.257 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:248141 4.a 4.2521eb5a (undecoded) ondisk+retry+write+known_if_redirected e511)", 2024-04-20T17:28:19.257 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:27:37.213906+0000", 2024-04-20T17:28:19.257 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 41.984178315000001, 2024-04-20T17:28:19.257 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.133879943, 2024-04-20T17:28:19.258 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:28:19.258 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:28:19.258 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-04-20T17:28:19.258 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:28:19.258 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:28:19.258 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:28:19.258 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 248141 2024-04-20T17:28:19.259 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.259 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:28:19.259 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.259 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:28:19.259 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.213906+0000", 2024-04-20T17:28:19.259 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.259 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.259 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.259 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:28:19.259 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.213906+0000", 2024-04-20T17:28:19.260 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.260 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.260 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.260 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:28:19.260 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.213909+0000", 2024-04-20T17:28:19.260 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5000000000000002e-06 2024-04-20T17:28:19.260 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.260 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.261 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:28:19.261 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.213915+0000", 2024-04-20T17:28:19.261 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3029999999999996e-06 2024-04-20T17:28:19.261 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.261 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.261 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:28:19.261 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.213926+0000", 2024-04-20T17:28:19.261 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.098e-05 2024-04-20T17:28:19.261 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.262 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.262 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:28:19.262 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.213937+0000", 2024-04-20T17:28:19.262 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.059e-05 2024-04-20T17:28:19.262 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.262 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.262 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:28:19.262 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214024+0000", 2024-04-20T17:28:19.262 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7726000000000007e-05 2024-04-20T17:28:19.263 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.263 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.263 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:28:19.263 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214033+0000", 2024-04-20T17:28:19.263 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.568e-06 2024-04-20T17:28:19.263 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.263 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.263 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:28:19.263 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.347749+0000", 2024-04-20T17:28:19.264 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.13371580899999999 2024-04-20T17:28:19.264 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.264 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.264 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:28:19.264 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.347786+0000", 2024-04-20T17:28:19.264 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7466999999999999e-05 2024-04-20T17:28:19.264 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.264 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:28:19.265 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.265 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.265 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.265 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:248145 4.a 4.f1af26da (undecoded) ondisk+retry+write+known_if_redirected e511)", 2024-04-20T17:28:19.265 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:27:37.213959+0000", 2024-04-20T17:28:19.265 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 41.984125114000001, 2024-04-20T17:28:19.265 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.13396334400000001, 2024-04-20T17:28:19.265 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:28:19.265 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:28:19.265 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-04-20T17:28:19.265 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:28:19.265 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:28:19.265 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:28:19.265 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 248145 2024-04-20T17:28:19.265 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.266 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:28:19.266 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.266 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:28:19.266 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.213959+0000", 2024-04-20T17:28:19.266 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.266 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.266 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.266 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:28:19.266 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.213959+0000", 2024-04-20T17:28:19.266 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.266 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.266 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.266 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:28:19.267 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.213960+0000", 2024-04-20T17:28:19.267 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3799999999999996e-07 2024-04-20T17:28:19.267 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.267 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.267 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:28:19.267 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.213963+0000", 2024-04-20T17:28:19.267 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0369999999999999e-06 2024-04-20T17:28:19.267 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.267 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.268 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:28:19.268 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.213970+0000", 2024-04-20T17:28:19.268 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3289999999999999e-06 2024-04-20T17:28:19.268 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.268 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.268 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:28:19.268 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.213975+0000", 2024-04-20T17:28:19.268 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5339999999999999e-06 2024-04-20T17:28:19.268 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.268 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.268 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:28:19.268 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214063+0000", 2024-04-20T17:28:19.268 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7665999999999997e-05 2024-04-20T17:28:19.269 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.269 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.269 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:28:19.269 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214068+0000", 2024-04-20T17:28:19.269 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6539999999999998e-06 2024-04-20T17:28:19.269 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.269 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.269 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:28:19.269 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.347916+0000", 2024-04-20T17:28:19.269 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.13384848999999999 2024-04-20T17:28:19.269 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.269 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.269 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:28:19.269 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.347923+0000", 2024-04-20T17:28:19.270 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7959999999999998e-06 2024-04-20T17:28:19.270 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.270 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:28:19.270 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.270 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.270 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.270 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:248156 4.a 4.1afff0ca (undecoded) ondisk+retry+write+known_if_redirected e511)", 2024-04-20T17:28:19.270 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:27:37.213983+0000", 2024-04-20T17:28:19.270 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 41.984101183, 2024-04-20T17:28:19.270 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.133983138, 2024-04-20T17:28:19.270 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:28:19.270 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:28:19.270 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-04-20T17:28:19.270 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:28:19.270 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:28:19.271 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:28:19.271 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 248156 2024-04-20T17:28:19.271 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.271 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:28:19.271 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.271 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:28:19.271 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.213983+0000", 2024-04-20T17:28:19.271 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.271 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.271 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.271 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:28:19.271 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.213983+0000", 2024-04-20T17:28:19.271 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.271 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.271 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.271 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:28:19.272 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.213984+0000", 2024-04-20T17:28:19.272 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1399999999999997e-07 2024-04-20T17:28:19.272 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.272 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.272 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:28:19.272 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.213986+0000", 2024-04-20T17:28:19.272 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9259999999999999e-06 2024-04-20T17:28:19.272 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.272 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.272 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:28:19.272 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.213991+0000", 2024-04-20T17:28:19.272 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.848e-06 2024-04-20T17:28:19.272 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.272 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.272 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:28:19.272 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.213995+0000", 2024-04-20T17:28:19.273 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.549e-06 2024-04-20T17:28:19.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.273 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.273 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:28:19.273 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214093+0000", 2024-04-20T17:28:19.273 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7314999999999999e-05 2024-04-20T17:28:19.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.273 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.273 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:28:19.273 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214101+0000", 2024-04-20T17:28:19.273 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1030000000000002e-06 2024-04-20T17:28:19.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.273 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.273 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:28:19.273 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.347958+0000", 2024-04-20T17:28:19.273 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.133857114 2024-04-20T17:28:19.274 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.274 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.274 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:28:19.274 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.347967+0000", 2024-04-20T17:28:19.274 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7690000000000002e-06 2024-04-20T17:28:19.274 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.274 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:28:19.274 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.274 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.274 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.274 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:248181 4.a 4.9677f25a (undecoded) ondisk+retry+write+known_if_redirected e511)", 2024-04-20T17:28:19.274 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:27:37.214001+0000", 2024-04-20T17:28:19.274 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 41.984084060000001, 2024-04-20T17:28:19.274 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.13399296199999999, 2024-04-20T17:28:19.275 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:28:19.275 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:28:19.275 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-04-20T17:28:19.275 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:28:19.275 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:28:19.275 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:28:19.275 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 248181 2024-04-20T17:28:19.275 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.275 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:28:19.275 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.275 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:28:19.275 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214001+0000", 2024-04-20T17:28:19.275 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.275 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.275 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.275 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:28:19.276 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214001+0000", 2024-04-20T17:28:19.276 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.276 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.276 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.276 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:28:19.276 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214001+0000", 2024-04-20T17:28:19.276 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2300000000000002e-07 2024-04-20T17:28:19.276 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.276 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.276 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:28:19.276 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214003+0000", 2024-04-20T17:28:19.276 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3300000000000001e-06 2024-04-20T17:28:19.276 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.276 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.276 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:28:19.276 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214008+0000", 2024-04-20T17:28:19.277 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4399999999999998e-06 2024-04-20T17:28:19.277 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.277 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.277 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:28:19.277 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214011+0000", 2024-04-20T17:28:19.277 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0529999999999998e-06 2024-04-20T17:28:19.277 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.277 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.277 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:28:19.277 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214126+0000", 2024-04-20T17:28:19.277 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000115448 2024-04-20T17:28:19.277 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.277 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.277 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:28:19.277 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214130+0000", 2024-04-20T17:28:19.277 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9940000000000002e-06 2024-04-20T17:28:19.278 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.278 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.278 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:28:19.278 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.347989+0000", 2024-04-20T17:28:19.278 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.133858545 2024-04-20T17:28:19.278 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.278 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.278 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:28:19.278 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.347993+0000", 2024-04-20T17:28:19.278 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.729e-06 2024-04-20T17:28:19.278 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.278 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:28:19.278 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.279 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.279 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.279 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:248192 4.a 4.6627f8ba (undecoded) ondisk+retry+write+known_if_redirected e511)", 2024-04-20T17:28:19.279 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:27:37.214015+0000", 2024-04-20T17:28:19.279 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 41.984069965000003, 2024-04-20T17:28:19.279 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.13409586000000001, 2024-04-20T17:28:19.279 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:28:19.279 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:28:19.279 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-04-20T17:28:19.279 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:28:19.279 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:28:19.279 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:28:19.279 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 248192 2024-04-20T17:28:19.279 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.280 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:28:19.280 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.280 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:28:19.280 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214015+0000", 2024-04-20T17:28:19.280 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.280 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.280 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.280 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:28:19.280 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214015+0000", 2024-04-20T17:28:19.280 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.280 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.280 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.280 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:28:19.280 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214015+0000", 2024-04-20T17:28:19.281 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2899999999999999e-07 2024-04-20T17:28:19.281 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.281 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.281 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:28:19.281 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214016+0000", 2024-04-20T17:28:19.281 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.455e-06 2024-04-20T17:28:19.281 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.281 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.281 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:28:19.281 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214021+0000", 2024-04-20T17:28:19.281 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3549999999999998e-06 2024-04-20T17:28:19.281 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.281 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.281 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:28:19.281 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214024+0000", 2024-04-20T17:28:19.282 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9450000000000002e-06 2024-04-20T17:28:19.282 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.282 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.282 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:28:19.282 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214154+0000", 2024-04-20T17:28:19.282 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013002900000000001 2024-04-20T17:28:19.282 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.282 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.282 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:28:19.282 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214158+0000", 2024-04-20T17:28:19.282 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7960000000000002e-06 2024-04-20T17:28:19.282 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.282 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.283 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:28:19.283 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.348106+0000", 2024-04-20T17:28:19.283 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.13394864100000001 2024-04-20T17:28:19.283 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.283 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:28:19.283 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.348110+0000", 2024-04-20T17:28:19.283 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3100000000000002e-06 2024-04-20T17:28:19.283 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.283 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:28:19.283 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.283 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.283 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:248202 4.a 4.d9a2372a (undecoded) ondisk+retry+write+known_if_redirected e511)", 2024-04-20T17:28:19.283 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:27:37.214027+0000", 2024-04-20T17:28:19.283 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 41.984057855000003, 2024-04-20T17:28:19.284 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.134111487, 2024-04-20T17:28:19.284 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:28:19.284 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:28:19.284 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-04-20T17:28:19.284 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:28:19.284 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:28:19.284 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:28:19.284 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 248202 2024-04-20T17:28:19.284 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.284 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:28:19.284 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.284 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:28:19.284 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214027+0000", 2024-04-20T17:28:19.284 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.284 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.285 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.285 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:28:19.285 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214027+0000", 2024-04-20T17:28:19.285 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.285 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.285 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.285 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:28:19.285 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214027+0000", 2024-04-20T17:28:19.285 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6100000000000002e-07 2024-04-20T17:28:19.285 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.285 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.285 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:28:19.285 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214028+0000", 2024-04-20T17:28:19.285 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.325e-06 2024-04-20T17:28:19.285 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.285 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.286 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:28:19.286 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214031+0000", 2024-04-20T17:28:19.286 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7779999999999999e-06 2024-04-20T17:28:19.286 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.286 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.286 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:28:19.286 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214034+0000", 2024-04-20T17:28:19.286 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7839999999999999e-06 2024-04-20T17:28:19.286 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.286 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.286 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:28:19.286 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214181+0000", 2024-04-20T17:28:19.286 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000146957 2024-04-20T17:28:19.287 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.287 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.287 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:28:19.287 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214185+0000", 2024-04-20T17:28:19.287 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.856e-06 2024-04-20T17:28:19.287 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.287 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.287 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:28:19.287 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.348132+0000", 2024-04-20T17:28:19.287 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.13394740399999999 2024-04-20T17:28:19.287 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.287 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.287 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:28:19.287 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.348138+0000", 2024-04-20T17:28:19.288 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0220000000000001e-06 2024-04-20T17:28:19.288 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.288 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:28:19.288 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.288 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.288 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.288 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:248209 4.a 4.672396fa (undecoded) ondisk+retry+write+known_if_redirected e511)", 2024-04-20T17:28:19.288 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:27:37.214037+0000", 2024-04-20T17:28:19.288 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 41.984047793999999, 2024-04-20T17:28:19.288 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.134119498, 2024-04-20T17:28:19.288 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:28:19.288 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:28:19.288 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-04-20T17:28:19.288 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:28:19.288 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:28:19.289 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:28:19.289 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 248209 2024-04-20T17:28:19.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.289 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:28:19.289 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.289 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:28:19.289 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214037+0000", 2024-04-20T17:28:19.289 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.289 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.289 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:28:19.289 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214037+0000", 2024-04-20T17:28:19.289 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.289 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.290 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:28:19.290 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214037+0000", 2024-04-20T17:28:19.290 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1600000000000002e-07 2024-04-20T17:28:19.290 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.290 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.290 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:28:19.290 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214038+0000", 2024-04-20T17:28:19.290 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.097e-06 2024-04-20T17:28:19.290 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.290 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.290 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:28:19.290 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214041+0000", 2024-04-20T17:28:19.290 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2370000000000001e-06 2024-04-20T17:28:19.291 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.291 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.291 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:28:19.291 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214044+0000", 2024-04-20T17:28:19.291 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.695e-06 2024-04-20T17:28:19.291 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.291 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.291 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:28:19.291 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214204+0000", 2024-04-20T17:28:19.291 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015972 2024-04-20T17:28:19.291 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.291 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.291 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:28:19.291 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.214208+0000", 2024-04-20T17:28:19.291 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7160000000000001e-06 2024-04-20T17:28:19.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.292 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:28:19.292 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.348152+0000", 2024-04-20T17:28:19.292 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.133944598 2024-04-20T17:28:19.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.292 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:28:19.292 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:37.348156+0000", 2024-04-20T17:28:19.292 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1189999999999999e-06 2024-04-20T17:28:19.292 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.292 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:28:19.292 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.293 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:275344 4.6 e520/445 4:6ce75c1e:::benchmark_data_smithi171_122772_object105152:head v 520'17562, mlcod=520'17550)", 2024-04-20T17:28:19.293 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:28:11.201360+0000", 2024-04-20T17:28:19.293 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 7.9967249599999999, 2024-04-20T17:28:19.293 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.059392998000000002, 2024-04-20T17:28:19.293 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:28:19.293 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:28:19.293 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:28:19.293 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:28:19.293 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.293 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:28:19.293 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.201360+0000", 2024-04-20T17:28:19.293 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.293 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.293 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.293 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:28:19.294 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.201360+0000", 2024-04-20T17:28:19.294 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.294 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.294 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.294 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:28:19.294 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.201360+0000", 2024-04-20T17:28:19.294 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2599999999999998e-07 2024-04-20T17:28:19.294 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.294 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.294 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:28:19.294 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.201363+0000", 2024-04-20T17:28:19.295 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.365e-06 2024-04-20T17:28:19.295 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.295 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:28:19.295 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.201367+0000", 2024-04-20T17:28:19.295 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0169999999999998e-06 2024-04-20T17:28:19.295 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.296 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:28:19.296 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.201371+0000", 2024-04-20T17:28:19.296 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6169999999999998e-06 2024-04-20T17:28:19.296 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.296 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:28:19.296 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.235536+0000", 2024-04-20T17:28:19.296 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.034165027000000001 2024-04-20T17:28:19.296 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.297 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:28:19.297 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.235556+0000", 2024-04-20T17:28:19.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9544000000000001e-05 2024-04-20T17:28:19.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.297 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.297 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:28:19.297 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.260719+0000", 2024-04-20T17:28:19.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.025163241999999999 2024-04-20T17:28:19.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.297 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.297 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:28:19.297 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.260753+0000", 2024-04-20T17:28:19.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3859999999999998e-05 2024-04-20T17:28:19.297 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.297 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:28:19.297 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.298 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.298 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.298 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:275365 4.6 e520/445 4:61938403:::benchmark_data_smithi171_122772_object105173:head v 520'17563, mlcod=520'17550)", 2024-04-20T17:28:19.298 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:28:11.201375+0000", 2024-04-20T17:28:19.298 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 7.9967094119999995, 2024-04-20T17:28:19.298 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.059402183999999997, 2024-04-20T17:28:19.298 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:28:19.298 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:28:19.298 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:28:19.299 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:28:19.299 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.299 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:28:19.299 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.201375+0000", 2024-04-20T17:28:19.299 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.299 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.299 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.299 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:28:19.299 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.201375+0000", 2024-04-20T17:28:19.299 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.300 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.300 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:28:19.300 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.201375+0000", 2024-04-20T17:28:19.300 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5699999999999999e-07 2024-04-20T17:28:19.300 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.300 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:28:19.300 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.201379+0000", 2024-04-20T17:28:19.300 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1140000000000002e-06 2024-04-20T17:28:19.300 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.300 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:28:19.300 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.201384+0000", 2024-04-20T17:28:19.300 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.327e-06 2024-04-20T17:28:19.301 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.301 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.301 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:28:19.301 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.201388+0000", 2024-04-20T17:28:19.301 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7280000000000001e-06 2024-04-20T17:28:19.301 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.301 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.301 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:28:19.301 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.239262+0000", 2024-04-20T17:28:19.301 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.037874146999999997 2024-04-20T17:28:19.301 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.301 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.301 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:28:19.301 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.239291+0000", 2024-04-20T17:28:19.302 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8856999999999998e-05 2024-04-20T17:28:19.302 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.302 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.302 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:28:19.302 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.260756+0000", 2024-04-20T17:28:19.302 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.021465563999999999 2024-04-20T17:28:19.302 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.302 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.302 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:28:19.302 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.260777+0000", 2024-04-20T17:28:19.302 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1189999999999999e-05 2024-04-20T17:28:19.302 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.302 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:28:19.302 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.303 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.303 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.303 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:275373 4.6 e520/445 4:6fa28d7d:::benchmark_data_smithi171_122772_object105181:head v 520'17564, mlcod=520'17550)", 2024-04-20T17:28:19.303 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:28:11.201392+0000", 2024-04-20T17:28:19.303 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 7.9966927759999997, 2024-04-20T17:28:19.303 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.059419597999999997, 2024-04-20T17:28:19.303 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:28:19.303 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:28:19.303 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:28:19.303 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:28:19.303 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.303 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:28:19.303 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.201392+0000", 2024-04-20T17:28:19.305 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.305 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.305 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:28:19.305 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.201392+0000", 2024-04-20T17:28:19.305 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.305 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.305 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:28:19.305 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.201392+0000", 2024-04-20T17:28:19.306 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8099999999999999e-07 2024-04-20T17:28:19.306 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.306 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:28:19.306 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.201411+0000", 2024-04-20T17:28:19.306 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8584999999999999e-05 2024-04-20T17:28:19.306 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.306 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:28:19.306 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.201417+0000", 2024-04-20T17:28:19.306 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0549999999999996e-06 2024-04-20T17:28:19.306 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.306 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:28:19.307 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.201421+0000", 2024-04-20T17:28:19.307 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1980000000000003e-06 2024-04-20T17:28:19.307 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.307 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.307 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:28:19.307 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.243901+0000", 2024-04-20T17:28:19.307 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.042480058000000001 2024-04-20T17:28:19.307 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.307 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.307 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:28:19.307 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.243923+0000", 2024-04-20T17:28:19.307 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1539000000000002e-05 2024-04-20T17:28:19.307 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.307 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.308 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:28:19.308 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.260783+0000", 2024-04-20T17:28:19.308 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.016860089000000002 2024-04-20T17:28:19.308 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.308 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.308 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:28:19.308 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.260811+0000", 2024-04-20T17:28:19.308 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8793e-05 2024-04-20T17:28:19.308 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.308 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:28:19.308 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.308 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.309 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.309 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26074.0:275377 4.6 e520/445 4:6ec2869e:::benchmark_data_smithi171_122772_object105185:head v 520'17565, mlcod=520'17550)", 2024-04-20T17:28:19.309 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:28:11.201426+0000", 2024-04-20T17:28:19.309 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 7.9966583739999999, 2024-04-20T17:28:19.309 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.059417465000000003, 2024-04-20T17:28:19.309 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:28:19.309 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:28:19.309 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:28:19.309 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:28:19.309 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.310 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:28:19.310 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.201426+0000", 2024-04-20T17:28:19.310 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.310 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.310 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.310 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:28:19.310 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.201426+0000", 2024-04-20T17:28:19.310 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:28:19.310 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.310 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.310 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:28:19.310 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.201427+0000", 2024-04-20T17:28:19.311 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3200000000000001e-07 2024-04-20T17:28:19.311 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.311 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.311 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:28:19.311 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.201430+0000", 2024-04-20T17:28:19.311 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3979999999999999e-06 2024-04-20T17:28:19.311 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.311 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.312 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:28:19.312 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.201435+0000", 2024-04-20T17:28:19.312 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4620000000000004e-06 2024-04-20T17:28:19.312 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.312 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.312 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:28:19.312 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.201439+0000", 2024-04-20T17:28:19.312 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9790000000000004e-06 2024-04-20T17:28:19.312 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.312 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.312 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:28:19.312 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.248475+0000", 2024-04-20T17:28:19.312 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.047035450999999999 2024-04-20T17:28:19.312 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.312 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.313 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:28:19.313 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.248495+0000", 2024-04-20T17:28:19.313 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0256999999999999e-05 2024-04-20T17:28:19.313 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.313 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.313 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:28:19.313 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.260814+0000", 2024-04-20T17:28:19.313 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012318656000000001 2024-04-20T17:28:19.313 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:28:19.313 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:28:19.313 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:28:19.313 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:11.260844+0000", 2024-04-20T17:28:19.313 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.993e-05 2024-04-20T17:28:19.313 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.313 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:28:19.314 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.314 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:28:19.314 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:28:19.314 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:28:19.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:19 smithi005 ceph-mon[39562]: pgmap v1377: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 14 active+clean; 3.7 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:28:19.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:19 smithi171 ceph-mon[44189]: pgmap v1377: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 14 active+clean; 3.7 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:28:19.755 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:28:19.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:19 smithi106 ceph-mon[44423]: pgmap v1377: 17 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 14 active+clean; 3.7 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:28:19.789 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:28:19.814 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:28:19.839 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:28:19.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:19.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:28:19.852 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:28:19.893 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:28:19.918 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:28:19.940 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:28:19.944 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:28:19.969 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:28:19.980 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:20.003 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:28:20.033 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:28:20.058 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:28:20.059 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:28:20.059 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:28:20.059 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:28:20.059 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:28:20.059 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:28:20.059 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:28:20.059 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-20T17:28:20.059 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:28:20.059 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:28:20.059 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:28:20.059 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:28:20.059 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:28:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:20 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1773216052' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:28:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:20 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:28:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:20 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:28:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:20.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:20 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1773216052' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:28:20.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:20.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:20.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:20 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:28:20.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:20 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:28:20.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:20.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:20 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1773216052' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:28:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:20 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:28:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:20 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:28:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:21.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:21.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:21.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:21.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:21.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:21.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:21.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:21.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:21.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:21 smithi171 ceph-mon[44189]: pgmap v1378: 17 pgs: 3 active+clean+scrubbing+deep, 14 active+clean; 3.6 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-20T17:28:21.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:21 smithi171 ceph-mon[44189]: osdmap e522: 12 total, 11 up, 11 in 2024-04-20T17:28:21.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:21.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:21.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:21.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:21.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:21.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:21.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:21.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:21.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:21 smithi005 ceph-mon[39562]: pgmap v1378: 17 pgs: 3 active+clean+scrubbing+deep, 14 active+clean; 3.6 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-20T17:28:21.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:21 smithi005 ceph-mon[39562]: osdmap e522: 12 total, 11 up, 11 in 2024-04-20T17:28:21.755 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:28:21.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:21.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:28:21.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:21.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:21.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:21.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:21.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:21.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:21.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:21.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:21.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:21 smithi106 ceph-mon[44423]: pgmap v1378: 17 pgs: 3 active+clean+scrubbing+deep, 14 active+clean; 3.6 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-20T17:28:21.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:21 smithi106 ceph-mon[44423]: osdmap e522: 12 total, 11 up, 11 in 2024-04-20T17:28:21.790 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-20T17:28:21.862 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:28:21.906 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:22.031 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:28:22.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:22.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:28:22.140 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:28:22.175 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:22.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:22 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:28:22.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:22 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:28:22.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:22 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:28:22.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:22 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:28:22.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:22 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:28:22.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:22 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:28:22.830 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:28:22.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:22.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:28:22.842 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:28:22.843 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:28:22.881 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:28:22.934 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:28:22.969 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:23.131 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:28:23.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:23.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:28:23.244 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:28:23.274 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:23.296 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:28:23.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:23.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:28:23.371 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:28:23.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:23 smithi005 ceph-mon[39562]: pgmap v1380: 17 pgs: 3 active+clean+scrubbing+deep, 14 active+clean; 3.5 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-20T17:28:23.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:23 smithi005 ceph-mon[39562]: osdmap e523: 12 total, 11 up, 11 in 2024-04-20T17:28:23.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:23 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2862816361' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:28:23.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:23 smithi005 ceph-mon[39562]: from='client.26359 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:28:23.399 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:28:23.430 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:23.454 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:28:23.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:23.458 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:28:23.559 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:28:23.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:23 smithi171 ceph-mon[44189]: pgmap v1380: 17 pgs: 3 active+clean+scrubbing+deep, 14 active+clean; 3.5 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-20T17:28:23.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:23 smithi171 ceph-mon[44189]: osdmap e523: 12 total, 11 up, 11 in 2024-04-20T17:28:23.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:23 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2862816361' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:28:23.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:23 smithi171 ceph-mon[44189]: from='client.26359 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:28:23.589 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:23.615 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:28:23.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:23.619 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:28:23.720 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:28:23.754 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:23.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:23 smithi106 ceph-mon[44423]: pgmap v1380: 17 pgs: 3 active+clean+scrubbing+deep, 14 active+clean; 3.5 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-20T17:28:23.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:23 smithi106 ceph-mon[44423]: osdmap e523: 12 total, 11 up, 11 in 2024-04-20T17:28:23.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:23 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2862816361' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:28:23.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:23 smithi106 ceph-mon[44423]: from='client.26359 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:28:23.778 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:28:23.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:23.782 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:28:23.882 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:28:23.920 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:23.944 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:28:23.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:23.950 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:28:24.050 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:28:24.082 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:24.109 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:28:24.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:24.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:28:24.213 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:28:24.240 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:24.266 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:28:24.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:24.273 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:28:24.374 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:28:24.411 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:24 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:28:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:24 smithi171 ceph-mon[44189]: from='client.26359 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:28:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:24 smithi171 ceph-mon[44189]: osdmap e524: 12 total, 11 up, 11 in 2024-04-20T17:28:24.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:24 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:28:24.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:24 smithi005 ceph-mon[39562]: from='client.26359 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:28:24.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:24 smithi005 ceph-mon[39562]: osdmap e524: 12 total, 11 up, 11 in 2024-04-20T17:28:24.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:24 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:28:24.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:24 smithi106 ceph-mon[44423]: from='client.26359 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:28:24.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:24 smithi106 ceph-mon[44423]: osdmap e524: 12 total, 11 up, 11 in 2024-04-20T17:28:24.797 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:28:24.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:24.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:28:24.844 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:28:24.901 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:28:24.931 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:24.955 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:28:24.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:24.962 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:28:25.063 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:28:25.095 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:25.127 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:28:25.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:25.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:28:25.232 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:28:25.263 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:25.294 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:28:25.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:25.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:28:25.352 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:25 smithi005 ceph-mon[39562]: pgmap v1383: 17 pgs: 3 active+clean+scrubbing+deep, 14 active+clean; 3.4 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 866 op/s 2024-04-20T17:28:25.397 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:28:25.428 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:25.448 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:28:25.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:25.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:28:25.467 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:28:25.467 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:28:25.482 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:28:25.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:25 smithi106 ceph-mon[44423]: pgmap v1383: 17 pgs: 3 active+clean+scrubbing+deep, 14 active+clean; 3.4 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 866 op/s 2024-04-20T17:28:25.556 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:28:25.588 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:25.615 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:28:25.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:25 smithi171 ceph-mon[44189]: pgmap v1383: 17 pgs: 3 active+clean+scrubbing+deep, 14 active+clean; 3.4 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 866 op/s 2024-04-20T17:28:25.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:25.622 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:28:25.722 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:28:25.754 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:25.782 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:28:25.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:25.784 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:28:25.885 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:28:25.929 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:25.956 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:28:25.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:25.964 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:28:26.066 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:28:26.111 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:26.346 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:28:27.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:27 smithi005 ceph-mon[39562]: pgmap v1384: 17 pgs: 3 active+clean+scrubbing+deep, 14 active+clean; 3.3 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 785 op/s 2024-04-20T17:28:27.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:27 smithi106 ceph-mon[44423]: pgmap v1384: 17 pgs: 3 active+clean+scrubbing+deep, 14 active+clean; 3.3 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 785 op/s 2024-04-20T17:28:27.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:27 smithi171 ceph-mon[44189]: pgmap v1384: 17 pgs: 3 active+clean+scrubbing+deep, 14 active+clean; 3.3 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 785 op/s 2024-04-20T17:28:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:29 smithi171 ceph-mon[44189]: pgmap v1385: 17 pgs: 3 active+clean+scrubbing+deep, 14 active+clean; 3.3 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 645 op/s 2024-04-20T17:28:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:29 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:28:29.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:29 smithi005 ceph-mon[39562]: pgmap v1385: 17 pgs: 3 active+clean+scrubbing+deep, 14 active+clean; 3.3 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 645 op/s 2024-04-20T17:28:29.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:29 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:28:29.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:29 smithi106 ceph-mon[44423]: pgmap v1385: 17 pgs: 3 active+clean+scrubbing+deep, 14 active+clean; 3.3 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 645 op/s 2024-04-20T17:28:29.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:29 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:28:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:31 smithi171 ceph-mon[44189]: pgmap v1386: 17 pgs: 3 active+clean+scrubbing+deep, 14 active+clean; 3.2 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 632 op/s 2024-04-20T17:28:31.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:31 smithi005 ceph-mon[39562]: pgmap v1386: 17 pgs: 3 active+clean+scrubbing+deep, 14 active+clean; 3.2 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 632 op/s 2024-04-20T17:28:31.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:31 smithi106 ceph-mon[44423]: pgmap v1386: 17 pgs: 3 active+clean+scrubbing+deep, 14 active+clean; 3.2 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 632 op/s 2024-04-20T17:28:33.100 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:28:33.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:33.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:28:33.204 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:28:33.262 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:33.298 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:28:33.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:33.307 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:28:33.408 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:28:33.437 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:33.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:33 smithi005 ceph-mon[39562]: pgmap v1387: 17 pgs: 3 active+clean+scrubbing+deep, 14 active+clean; 3.1 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 682 op/s 2024-04-20T17:28:33.472 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:28:33.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:33.474 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:28:33.575 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:28:33.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:33 smithi171 ceph-mon[44189]: pgmap v1387: 17 pgs: 3 active+clean+scrubbing+deep, 14 active+clean; 3.1 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 682 op/s 2024-04-20T17:28:33.607 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:33.641 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:28:33.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:33.643 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:28:33.744 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:28:33.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:33 smithi106 ceph-mon[44423]: pgmap v1387: 17 pgs: 3 active+clean+scrubbing+deep, 14 active+clean; 3.1 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 682 op/s 2024-04-20T17:28:33.776 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:34.319 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-20T17:28:34.331 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:28:34.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:34.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:28:34.439 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:28:34.467 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:34.493 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:28:34.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:34.503 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:28:34.604 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:28:34.634 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:34.658 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:28:34.660 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:34 smithi106 ceph-mon[44423]: 4.2 deep-scrub ok 2024-04-20T17:28:34.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:34.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:28:34.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:34 smithi005 ceph-mon[39562]: 4.2 deep-scrub ok 2024-04-20T17:28:34.761 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:28:34.792 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:34 smithi171 ceph-mon[44189]: 4.2 deep-scrub ok 2024-04-20T17:28:34.796 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:34.825 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:28:34.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:34.829 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:28:34.930 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:28:34.969 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:34.997 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:28:35.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:35.001 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:28:35.102 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:28:35.149 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:35.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:35 smithi005 ceph-mon[39562]: pgmap v1388: 17 pgs: 3 active+clean+scrubbing+deep, 14 active+clean; 3.0 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 766 op/s 2024-04-20T17:28:35.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:35 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:28:35.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:35 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:28:35.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:35 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:28:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:35 smithi106 ceph-mon[44423]: pgmap v1388: 17 pgs: 3 active+clean+scrubbing+deep, 14 active+clean; 3.0 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 766 op/s 2024-04-20T17:28:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:35 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:28:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:35 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:28:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:35 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:28:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:35 smithi171 ceph-mon[44189]: pgmap v1388: 17 pgs: 3 active+clean+scrubbing+deep, 14 active+clean; 3.0 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 766 op/s 2024-04-20T17:28:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:35 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:28:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:35 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:28:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:35 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:28:36.996 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:28:37.015 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:28:37.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:37.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:28:37.119 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:28:37.167 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:37.622 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:37 smithi005 ceph-mon[39562]: pgmap v1389: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 2.9 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 699 op/s 2024-04-20T17:28:37.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:37 smithi106 ceph-mon[44423]: pgmap v1389: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 2.9 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 699 op/s 2024-04-20T17:28:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:37 smithi171 ceph-mon[44189]: pgmap v1389: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 2.9 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 699 op/s 2024-04-20T17:28:38.338 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:28:38.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:38.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:28:38.443 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:28:38.475 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:38.607 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:28:38.607 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:28:38.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:38.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:28:38.668 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:28:38.695 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:28:38.717 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:28:38.733 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:28:38.745 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:38.764 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:28:38.773 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:28:38.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:38.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:28:38.796 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:28:38.821 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:28:38.845 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:28:38.878 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:28:38.882 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:28:38.906 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:38.910 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:28:38.926 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:28:38.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:38.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:28:38.936 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:28:38.969 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:28:38.970 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:28:38.970 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:28:38.970 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:28:38.970 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:28:38.970 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:28:38.970 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:28:38.970 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-20T17:28:38.970 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:28:38.970 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:28:38.970 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:28:38.970 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:28:38.970 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:28:39.031 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:28:39.063 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:39.094 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:28:39.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:39.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:28:39.202 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:28:39.230 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:39.252 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:28:39.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:39.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:28:39.356 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:28:39.383 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:39.402 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:28:39.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:39.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:28:39.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:39 smithi005 ceph-mon[39562]: pgmap v1390: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 2.9 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 615 op/s 2024-04-20T17:28:39.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:39 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1075329855' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:28:39.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:39.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:39.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:39.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:39.506 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:28:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:39 smithi106 ceph-mon[44423]: pgmap v1390: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 2.9 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 615 op/s 2024-04-20T17:28:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:39 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1075329855' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:28:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:39.547 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:39.570 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:28:39.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:39.574 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:28:39.675 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:28:39.712 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:39.773 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:28:39.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:39.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:28:39.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:39 smithi171 ceph-mon[44189]: pgmap v1390: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 2.9 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 615 op/s 2024-04-20T17:28:39.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:39 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1075329855' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:28:39.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:39.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:39.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:39.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:39.877 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:28:39.921 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:40.435 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:28:40.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:40.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:40.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:40.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:40.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:40.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:40.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:40.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:40 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1327388659' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:28:40.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:40 smithi005 ceph-mon[39562]: from='client.26465 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:28:40.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:40.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:40.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:40.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:40.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:40.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:40.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:40.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:40 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1327388659' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:28:40.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:40 smithi106 ceph-mon[44423]: from='client.26465 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:28:40.781 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:28:40.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:40.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:28:40.789 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:28:40.789 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:28:40.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:40.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:40.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:40.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:40.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:40.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:40.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:40.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:40 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1327388659' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:28:40.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:40 smithi171 ceph-mon[44189]: from='client.26465 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:28:40.885 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:28:40.932 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:41.288 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:28:41.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:41.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:28:41.396 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:28:41.437 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:41.464 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:28:41.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:41.468 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:28:41.569 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:28:41.612 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:41 smithi005 ceph-mon[39562]: 4.8 deep-scrub ok 2024-04-20T17:28:41.612 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:41 smithi005 ceph-mon[39562]: pgmap v1391: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 2.7 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 747 op/s 2024-04-20T17:28:41.613 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:41 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:28:41.613 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:41 smithi005 ceph-mon[39562]: from='client.26465 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:28:41.613 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:41 smithi005 ceph-mon[39562]: osdmap e525: 12 total, 11 up, 11 in 2024-04-20T17:28:41.613 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:41.657 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:28:41.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:41.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:28:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:41 smithi106 ceph-mon[44423]: 4.8 deep-scrub ok 2024-04-20T17:28:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:41 smithi106 ceph-mon[44423]: pgmap v1391: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 2.7 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 747 op/s 2024-04-20T17:28:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:41 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:28:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:41 smithi106 ceph-mon[44423]: from='client.26465 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:28:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:41 smithi106 ceph-mon[44423]: osdmap e525: 12 total, 11 up, 11 in 2024-04-20T17:28:41.764 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:28:41.795 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:41 smithi171 ceph-mon[44189]: 4.8 deep-scrub ok 2024-04-20T17:28:41.795 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:41 smithi171 ceph-mon[44189]: pgmap v1391: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 2.7 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 747 op/s 2024-04-20T17:28:41.795 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:41 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:28:41.795 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:41 smithi171 ceph-mon[44189]: from='client.26465 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:28:41.795 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:41 smithi171 ceph-mon[44189]: osdmap e525: 12 total, 11 up, 11 in 2024-04-20T17:28:41.797 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:41.838 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:28:41.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:41.841 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:28:41.942 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:28:42.005 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:42.118 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:28:42.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:42.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:28:42.223 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:28:42.257 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:42.286 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:28:42.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:42.293 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:28:42.301 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:28:42.301 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:28:42.301 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:28:42.301 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:28:42.301 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:28:42.394 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:28:42.448 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:42.525 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:28:42.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:42.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:28:42.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:42 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:28:42.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:42 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:28:42.630 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:28:42.678 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:42 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:28:42.678 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:42 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:28:42.684 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:42.710 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:28:42.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:42.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:28:42.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:42 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:28:42.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:42 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:28:42.790 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:28:42.823 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:28:42.851 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:42.874 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:28:42.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:42.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:28:42.980 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:28:43.017 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:43.051 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:28:43.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:43.063 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:28:43.133 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-20T17:28:43.166 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:28:43.216 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:28:43.252 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:43.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:43 smithi005 ceph-mon[39562]: pgmap v1393: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.6 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 779 op/s 2024-04-20T17:28:43.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:43 smithi005 ceph-mon[39562]: osdmap e526: 12 total, 11 up, 11 in 2024-04-20T17:28:43.479 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:28:43.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:43.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:28:43.587 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:28:43.642 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:43.697 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:28:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:43 smithi106 ceph-mon[44423]: pgmap v1393: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.6 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 779 op/s 2024-04-20T17:28:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:43 smithi106 ceph-mon[44423]: osdmap e526: 12 total, 11 up, 11 in 2024-04-20T17:28:43.762 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:28:43.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:43.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:28:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:43 smithi171 ceph-mon[44189]: pgmap v1393: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.6 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 779 op/s 2024-04-20T17:28:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:43 smithi171 ceph-mon[44189]: osdmap e526: 12 total, 11 up, 11 in 2024-04-20T17:28:43.866 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:28:43.934 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:44.101 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:28:44.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:44.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:28:44.206 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:28:44.236 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:44.260 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:28:44.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:44.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:28:44.367 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:28:44.412 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:44.437 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:28:44.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:44.445 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:28:44.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:44 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:28:44.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:44 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:28:44.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:44 smithi005 ceph-mon[39562]: osdmap e527: 12 total, 11 up, 11 in 2024-04-20T17:28:44.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:44 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:28:44.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:44 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:28:44.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:44 smithi106 ceph-mon[44423]: osdmap e527: 12 total, 11 up, 11 in 2024-04-20T17:28:44.546 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:28:44.592 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:44.668 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:28:44.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:44.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:28:44.699 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:28:44.740 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:28:44.776 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:28:44.824 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:44 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:28:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:44 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:28:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:44 smithi171 ceph-mon[44189]: osdmap e527: 12 total, 11 up, 11 in 2024-04-20T17:28:45.221 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:28:45.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:45.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:28:45.328 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:28:45.386 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:45.529 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:28:45.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:45.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:28:45.533 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:45 smithi005 ceph-mon[39562]: pgmap v1396: 17 pgs: 17 active+clean; 2.5 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-20T17:28:45.533 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:45 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:28:45.533 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:45 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:28:45.633 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:28:45.660 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:45.681 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:28:45.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:45.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:28:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:45 smithi106 ceph-mon[44423]: pgmap v1396: 17 pgs: 17 active+clean; 2.5 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-20T17:28:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:45 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:28:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:45 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:28:45.784 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:28:45.829 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:45 smithi171 ceph-mon[44189]: pgmap v1396: 17 pgs: 17 active+clean; 2.5 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-20T17:28:45.829 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:45 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:28:45.830 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:45 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:28:45.845 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:45.866 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:28:45.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:45.869 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:28:45.970 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:28:45.999 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:46.031 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:28:46.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:46.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:28:46.134 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:28:46.162 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:46.185 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:28:46.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:46.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:28:46.293 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:28:46.336 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:46.414 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:28:46.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:46.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:28:46.509 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:28:46.509 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:28:46.509 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:28:46.509 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:28:46.509 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:28:46.509 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:28:46.509 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:28:46.509 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:28:46.509 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:28:46.510 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:28:46.510 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:28:46.510 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:28:46.510 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:28:46.510 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:28:46.510 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:28:46.510 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:28:46.510 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:28:46.510 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-20T17:28:46.510 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:28:46.510 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:28:46.511 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:28:46.511 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:28:46.511 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:28:46.518 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:28:46.547 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:46.566 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:28:46.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:46.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:28:46.675 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:28:46.703 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:46.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:46 smithi005 ceph-mon[39562]: osdmap e528: 12 total, 11 up, 11 in 2024-04-20T17:28:46.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:46 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/750831844' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:28:46.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:46.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:46.725 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:28:46.727 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:46 smithi106 ceph-mon[44423]: osdmap e528: 12 total, 11 up, 11 in 2024-04-20T17:28:46.727 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:46 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/750831844' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:28:46.727 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:46.727 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:46.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:46.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:28:46.829 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:28:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:46 smithi171 ceph-mon[44189]: osdmap e528: 12 total, 11 up, 11 in 2024-04-20T17:28:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:46 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/750831844' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:28:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:46.878 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:47.246 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:28:47.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:47.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:28:47.353 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:28:47.405 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:47.677 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:47.677 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:47.677 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:47.677 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:47.678 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:47.678 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:47.678 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:47.678 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:47.678 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:47 smithi005 ceph-mon[39562]: pgmap v1398: 17 pgs: 17 active+clean; 2.3 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-20T17:28:47.678 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:47.681 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:28:47.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:47.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:28:47.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:47.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:47.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:47.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:47.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:47.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:47.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:47.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:47.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:47 smithi106 ceph-mon[44423]: pgmap v1398: 17 pgs: 17 active+clean; 2.3 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-20T17:28:47.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:47.789 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:28:47.817 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:47 smithi171 ceph-mon[44189]: pgmap v1398: 17 pgs: 17 active+clean; 2.3 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-20T17:28:47.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:47.840 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:28:47.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:47.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:28:47.943 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:28:47.972 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:47.995 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:28:47.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:47.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:28:48.099 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:28:48.126 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:48.151 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:28:48.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:48.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:28:48.255 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:28:48.283 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:48.309 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:28:48.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:48.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:28:48.415 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:28:48.480 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:48.691 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:28:48.691 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:28:48.695 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:28:48.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:48.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:28:48.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:48 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:28:48.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:48 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:28:48.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:48 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:28:48.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:48 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:28:48.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:48 smithi106 ceph-mon[44423]: osdmap e529: 12 total, 11 up, 11 in 2024-04-20T17:28:48.802 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:28:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:48 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:28:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:48 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:28:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:48 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:28:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:48 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:28:48.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:48 smithi171 ceph-mon[44189]: osdmap e529: 12 total, 11 up, 11 in 2024-04-20T17:28:48.841 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:48.842 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:48 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:28:48.842 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:48 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:28:48.843 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:48 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:28:48.843 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:48 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:28:48.843 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:48 smithi005 ceph-mon[39562]: osdmap e529: 12 total, 11 up, 11 in 2024-04-20T17:28:48.955 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:28:48.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:48.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:28:49.069 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:28:49.134 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:49.246 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:28:49.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:49.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:28:49.291 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:28:49.291 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:28:49.357 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:28:49.412 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:49.520 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:28:49.545 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:28:49.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:49.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:28:49.651 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:28:49.691 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:49.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:49 smithi005 ceph-mon[39562]: pgmap v1400: 17 pgs: 17 active+clean; 2.2 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 989 op/s 2024-04-20T17:28:49.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:49 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2436682448' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:28:49.720 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:28:49.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:49.725 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:28:49.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:49 smithi106 ceph-mon[44423]: pgmap v1400: 17 pgs: 17 active+clean; 2.2 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 989 op/s 2024-04-20T17:28:49.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:49 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2436682448' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:28:49.826 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:28:49.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:49 smithi171 ceph-mon[44189]: pgmap v1400: 17 pgs: 17 active+clean; 2.2 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 989 op/s 2024-04-20T17:28:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:49 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2436682448' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:28:49.877 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:49.909 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:28:49.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:49.916 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:28:50.017 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:28:50.064 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:50.092 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:28:50.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:50.096 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:28:50.197 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:28:50.228 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:50.248 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:28:50.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:50.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:28:50.253 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-20T17:28:50.353 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:28:50.383 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:50.412 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:28:50.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:50.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:28:50.505 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:50 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:28:50.505 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:50 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:28:50.505 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:50 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:28:50.505 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:50 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2436682448' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:28:50.505 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:50 smithi106 ceph-mon[44423]: osdmap e530: 12 total, 11 up, 11 in 2024-04-20T17:28:50.505 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:50 smithi106 ceph-mon[44423]: 4.4 scrub starts 2024-04-20T17:28:50.516 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:28:50.553 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:50.576 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:28:50.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:50.580 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:28:50.618 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:28:50.680 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:28:50.692 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:28:50.717 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:50.742 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:28:50.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:50.747 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:28:50.756 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:50 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:28:50.756 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:50 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:28:50.756 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:50 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:28:50.756 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:50 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2436682448' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:28:50.756 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:50 smithi005 ceph-mon[39562]: osdmap e530: 12 total, 11 up, 11 in 2024-04-20T17:28:50.756 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:50 smithi005 ceph-mon[39562]: 4.4 scrub starts 2024-04-20T17:28:50.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:50 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:28:50.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:50 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:28:50.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:50 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:28:50.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:50 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2436682448' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:28:50.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:50 smithi171 ceph-mon[44189]: osdmap e530: 12 total, 11 up, 11 in 2024-04-20T17:28:50.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:50 smithi171 ceph-mon[44189]: 4.4 scrub starts 2024-04-20T17:28:50.847 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:28:50.883 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:50.926 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:28:50.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:50.930 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:28:51.030 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:28:51.070 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:51.098 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:28:51.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:51.101 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:28:51.126 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:28:51.202 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:28:51.234 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:51.262 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:28:51.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:51.266 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:28:51.367 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:28:51.413 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:51.445 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:28:51.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:51.451 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:28:51.552 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:28:51.585 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:51.679 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:51 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:28:51.679 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:51 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:28:51.679 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:51 smithi005 ceph-mon[39562]: pgmap v1402: 17 pgs: 17 active+clean; 2.1 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-20T17:28:51.679 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:51 smithi005 ceph-mon[39562]: 4.4 scrub ok 2024-04-20T17:28:51.679 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:51 smithi005 ceph-mon[39562]: 4.6 scrub starts 2024-04-20T17:28:51.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:51 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:28:51.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:51 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:28:51.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:51 smithi171 ceph-mon[44189]: pgmap v1402: 17 pgs: 17 active+clean; 2.1 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-20T17:28:51.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:51 smithi171 ceph-mon[44189]: 4.4 scrub ok 2024-04-20T17:28:51.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:51 smithi171 ceph-mon[44189]: 4.6 scrub starts 2024-04-20T17:28:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:51 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:28:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:51 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:28:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:51 smithi106 ceph-mon[44423]: pgmap v1402: 17 pgs: 17 active+clean; 2.1 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-20T17:28:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:51 smithi106 ceph-mon[44423]: 4.4 scrub ok 2024-04-20T17:28:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:51 smithi106 ceph-mon[44423]: 4.6 scrub starts 2024-04-20T17:28:52.195 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:28:52.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:52.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:28:52.299 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:28:52.366 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:52.581 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:28:52.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:52.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:28:52.686 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:28:52.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:52 smithi005 ceph-mon[39562]: 4.d scrub starts 2024-04-20T17:28:52.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:52 smithi005 ceph-mon[39562]: 4.d scrub ok 2024-04-20T17:28:52.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:52 smithi005 ceph-mon[39562]: 4.0 scrub starts 2024-04-20T17:28:52.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:52 smithi005 ceph-mon[39562]: 4.6 scrub ok 2024-04-20T17:28:52.714 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:28:52.725 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:52.756 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:28:52.760 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:28:52.760 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:28:52.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:52 smithi106 ceph-mon[44423]: 4.d scrub starts 2024-04-20T17:28:52.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:52 smithi106 ceph-mon[44423]: 4.d scrub ok 2024-04-20T17:28:52.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:52 smithi106 ceph-mon[44423]: 4.0 scrub starts 2024-04-20T17:28:52.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:52 smithi106 ceph-mon[44423]: 4.6 scrub ok 2024-04-20T17:28:52.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:52.763 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:28:52.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:52 smithi171 ceph-mon[44189]: 4.d scrub starts 2024-04-20T17:28:52.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:52 smithi171 ceph-mon[44189]: 4.d scrub ok 2024-04-20T17:28:52.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:52 smithi171 ceph-mon[44189]: 4.0 scrub starts 2024-04-20T17:28:52.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:52 smithi171 ceph-mon[44189]: 4.6 scrub ok 2024-04-20T17:28:52.864 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:28:52.935 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:52.970 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:28:52.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:52.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:28:53.080 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:28:53.133 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:53.220 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:28:53.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:53.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:28:53.287 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:28:53.299 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:28:53.326 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:28:53.335 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:28:53.358 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:53.367 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:28:53.404 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:28:53.436 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:28:53.462 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:28:53.491 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:28:53.491 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:28:53.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:53.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:28:53.510 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-20T17:28:53.526 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:28:53.575 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:28:53.596 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:28:53.607 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:28:53.644 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:53.647 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:28:53.648 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:28:53.648 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:28:53.648 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:28:53.648 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:28:53.648 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:28:53.648 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:28:53.648 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-20T17:28:53.648 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:28:53.648 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:28:53.648 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:28:53.648 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:28:53.649 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:28:53.667 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:28:53.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:53.670 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:28:53.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:53 smithi106 ceph-mon[44423]: 4.0 scrub ok 2024-04-20T17:28:53.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:53 smithi106 ceph-mon[44423]: pgmap v1403: 17 pgs: 17 active+clean; 1.9 GiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.33k op/s 2024-04-20T17:28:53.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:53 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2244239189' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:28:53.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:53 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:53.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:53 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:53.770 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:28:53.802 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:53 smithi171 ceph-mon[44189]: 4.0 scrub ok 2024-04-20T17:28:53.802 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:53 smithi171 ceph-mon[44189]: pgmap v1403: 17 pgs: 17 active+clean; 1.9 GiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.33k op/s 2024-04-20T17:28:53.803 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:53 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2244239189' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:28:53.803 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:53 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:53.803 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:53 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:53.805 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:53.819 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:53 smithi005 ceph-mon[39562]: 4.0 scrub ok 2024-04-20T17:28:53.819 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:53 smithi005 ceph-mon[39562]: pgmap v1403: 17 pgs: 17 active+clean; 1.9 GiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.33k op/s 2024-04-20T17:28:53.819 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:53 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2244239189' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:28:53.819 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:53 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:53.819 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:53 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:53.829 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:28:53.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:53.835 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:28:53.872 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:28:53.936 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:28:53.972 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:54.007 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:28:54.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:54.010 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:28:54.111 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:28:54.174 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:54.346 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:28:54.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:54.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:28:54.449 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:28:54.481 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:54.732 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:28:54.734 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:54.734 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:54.734 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:54.734 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:54.734 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:54.734 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:54.734 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:54.734 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:54.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:54.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:28:54.765 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:54.765 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:54.765 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:54.765 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:54.765 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:54.766 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:54.766 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:54.766 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:54.801 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:28:54.801 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:28:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:54.839 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:28:54.901 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:54.926 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:28:54.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:54.931 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:28:55.032 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:28:55.074 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:55.441 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:28:55.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:55.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:28:55.555 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:28:55.590 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:55.620 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:28:55.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:55.623 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:28:55.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:55.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:55 smithi005 ceph-mon[39562]: pgmap v1404: 17 pgs: 17 active+clean; 1.7 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:28:55.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:55 smithi005 ceph-mon[39562]: 4.b scrub starts 2024-04-20T17:28:55.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:55 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:28:55.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:55 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:28:55.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:55 smithi005 ceph-mon[39562]: 4.b scrub ok 2024-04-20T17:28:55.724 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:28:55.775 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:55.801 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:28:55.809 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:55.809 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:55 smithi171 ceph-mon[44189]: pgmap v1404: 17 pgs: 17 active+clean; 1.7 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:28:55.809 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:55 smithi171 ceph-mon[44189]: 4.b scrub starts 2024-04-20T17:28:55.809 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:55 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:28:55.809 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:55 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:28:55.809 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:55 smithi171 ceph-mon[44189]: 4.b scrub ok 2024-04-20T17:28:55.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:55.810 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:28:55.911 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:28:55.961 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:28:55.961 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:28:55.961 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-04-20T17:28:55.961 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-04-20T17:28:55.961 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:28:55.970 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:55.984 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:28:55.984 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:55 smithi106 ceph-mon[44423]: pgmap v1404: 17 pgs: 17 active+clean; 1.7 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:28:55.985 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:55 smithi106 ceph-mon[44423]: 4.b scrub starts 2024-04-20T17:28:55.985 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:55 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:28:55.985 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:55 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:28:55.985 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:55 smithi106 ceph-mon[44423]: 4.b scrub ok 2024-04-20T17:28:56.062 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:28:56.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:56.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:28:56.166 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:28:56.200 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:56.225 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:28:56.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:56.228 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:28:56.329 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:28:56.387 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:56.428 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:28:56.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:56.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:28:56.536 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:28:56.565 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:56 smithi171 ceph-mon[44189]: osdmap e531: 12 total, 11 up, 11 in 2024-04-20T17:28:56.565 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:56 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2475122765' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:28:56.577 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:56.604 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:28:56.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:56.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:28:56.684 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:56 smithi106 ceph-mon[44423]: osdmap e531: 12 total, 11 up, 11 in 2024-04-20T17:28:56.684 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:56 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2475122765' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:28:56.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:56 smithi005 ceph-mon[39562]: osdmap e531: 12 total, 11 up, 11 in 2024-04-20T17:28:56.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:56 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2475122765' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:28:56.712 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:28:56.713 INFO:tasks.thrashosds.thrasher:Reviving osd 6 2024-04-20T17:28:56.713 INFO:tasks.cephadm.osd.6:Restarting osd.6 (starting--it wasn't running)... 2024-04-20T17:28:56.714 DEBUG:teuthology.orchestra.run.smithi106:osd.6> sudo journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.6.service 2024-04-20T17:28:56.716 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl start ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.6 2024-04-20T17:28:56.717 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-20T17:28:56.755 INFO:journalctl@ceph.osd.6.smithi106.stdout:-- Logs begin at Sat 2024-04-20 16:47:44 UTC. -- 2024-04-20T17:28:56.759 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:56.791 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:28:56.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:56.796 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:28:56.802 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:28:56.896 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:28:56.957 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:57.006 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:28:57.011 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:28:56 smithi106 systemd[1]: Starting Ceph osd.6 for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:28:57.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:57.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:28:57.116 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:28:57.151 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:57.202 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:28:57.232 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:28:57.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:57.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:28:57.255 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:28:57.340 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:28:57.373 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:57.413 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:28:57.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:57.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:28:57.518 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:28:57.561 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:57.592 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:28:57.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:57 smithi106 ceph-mon[44423]: pgmap v1406: 17 pgs: 17 active+clean; 1.6 GiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.38k op/s 2024-04-20T17:28:57.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:57 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:28:57.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:57 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:28:57.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:57 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2475122765' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:28:57.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:57 smithi106 ceph-mon[44423]: osdmap e532: 12 total, 11 up, 11 in 2024-04-20T17:28:57.813 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:28:57.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:57.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:28:57.821 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:57 smithi005 ceph-mon[39562]: pgmap v1406: 17 pgs: 17 active+clean; 1.6 GiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.38k op/s 2024-04-20T17:28:57.821 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:57 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:28:57.821 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:57 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:28:57.821 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:57 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2475122765' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:28:57.821 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:57 smithi005 ceph-mon[39562]: osdmap e532: 12 total, 11 up, 11 in 2024-04-20T17:28:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:57 smithi171 ceph-mon[44189]: pgmap v1406: 17 pgs: 17 active+clean; 1.6 GiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 1.38k op/s 2024-04-20T17:28:57.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:57 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:28:57.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:57 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:28:57.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:57 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2475122765' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:28:57.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:57 smithi171 ceph-mon[44189]: osdmap e532: 12 total, 11 up, 11 in 2024-04-20T17:28:57.921 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:28:57.971 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:58.000 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:28:58.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:58.007 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:28:58.108 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:28:58.148 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:28:58 smithi106 bash[138678]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T17:28:58.148 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:28:58 smithi106 bash[138678]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-20T17:28:58.148 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:28:58 smithi106 bash[138678]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-20T17:28:58.148 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:28:58 smithi106 bash[138678]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-20T17:28:58.148 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:28:58 smithi106 bash[138678]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-20T17:28:58.148 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:28:58 smithi106 bash[138678]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-04-20T17:28:58.148 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:28:58 smithi106 bash[138678]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-20T17:28:58.149 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:28:58 smithi106 bash[138678]: --> ceph-volume raw activate successful for osd ID: 6 2024-04-20T17:28:58.152 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:58.410 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:28:58.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:58.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:28:58.515 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:28:58.545 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:58.577 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:58 smithi005 ceph-mon[39562]: 4.1 scrub starts 2024-04-20T17:28:58.577 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:58 smithi005 ceph-mon[39562]: 4.1 scrub ok 2024-04-20T17:28:58.602 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:28:58.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:58.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:28:58.713 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:28:58.741 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:58 smithi106 ceph-mon[44423]: 4.1 scrub starts 2024-04-20T17:28:58.741 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:58 smithi106 ceph-mon[44423]: 4.1 scrub ok 2024-04-20T17:28:58.743 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:58.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:58 smithi171 ceph-mon[44189]: 4.1 scrub starts 2024-04-20T17:28:58.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:58 smithi171 ceph-mon[44189]: 4.1 scrub ok 2024-04-20T17:28:59.396 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:28:59.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:59.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:28:59.500 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:28:59.554 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:59.604 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:28:59.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:59.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:28:59.710 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:28:59.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:28:59 smithi005 ceph-mon[39562]: pgmap v1408: 17 pgs: 17 active+clean; 1.5 GiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-20T17:28:59.739 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:28:59 smithi106 ceph-mon[44423]: pgmap v1408: 17 pgs: 17 active+clean; 1.5 GiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-20T17:28:59.742 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:59.798 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:28:59.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:59.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:28:59.813 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:28:59.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:28:59 smithi171 ceph-mon[44189]: pgmap v1408: 17 pgs: 17 active+clean; 1.5 GiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-20T17:28:59.840 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:28:59.867 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:28:59.902 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:28:59.908 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:28:59.936 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:28:59.943 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:28:59.966 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:28:59.968 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:28:59.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:28:59.977 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:28:59.998 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:29:00.023 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:29:00.061 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:29:00.077 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:29:00.098 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:29:00.135 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:29:00.135 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:29:00.135 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:29:00.135 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:29:00.135 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:29:00.135 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:29:00.135 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:29:00.135 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-20T17:29:00.136 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:29:00.136 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:29:00.136 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:29:00.136 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:29:00.136 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:29:00.136 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:00.160 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:29:00.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:00.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:29:00.267 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:29:00.303 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:00.348 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:29:00.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:00.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:29:00.452 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:29:00.487 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:00.511 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:29:00.513 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:29:00.513 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:29:00.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:00.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:29:00.618 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:29:00.682 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:00.727 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:29:00.765 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:00 smithi005 ceph-mon[39562]: 4.c scrub starts 2024-04-20T17:29:00.766 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:00 smithi005 ceph-mon[39562]: 4.c scrub ok 2024-04-20T17:29:00.766 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:00 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1197595213' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:29:00.766 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:00.766 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:00.766 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:00.766 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:00.766 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:00.766 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:00.843 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:29:00.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:00.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:29:00.893 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:29:00.893 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:29:00.956 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:29:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:00 smithi106 ceph-mon[44423]: 4.c scrub starts 2024-04-20T17:29:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:00 smithi106 ceph-mon[44423]: 4.c scrub ok 2024-04-20T17:29:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:00 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1197595213' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:29:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:01.017 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:01.055 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:29:01.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:01.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:29:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:00 smithi171 ceph-mon[44189]: 4.c scrub starts 2024-04-20T17:29:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:00 smithi171 ceph-mon[44189]: 4.c scrub ok 2024-04-20T17:29:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:00 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1197595213' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:29:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:01.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:01.140 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:29:01.158 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:29:01.206 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:01.228 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-20T17:29:01.417 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:29:01.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:01.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:29:01.511 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:29:01 smithi106 bash[139277]: 4a3296b54c49fd56196f010a335ef51581eaf966636199b5168cee12bf63e7c4 2024-04-20T17:29:01.511 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:29:01 smithi106 systemd[1]: Started Ceph osd.6 for 13c6e76c-ff37-11ee-bc93-c7b262605968. 2024-04-20T17:29:01.520 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:29:01.569 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:01.651 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:29:01.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:01.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:29:01.654 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:01.654 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:01.654 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:01.654 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:01.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:01.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:01 smithi005 ceph-mon[39562]: pgmap v1409: 17 pgs: 17 active+clean; 1.4 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.08k op/s 2024-04-20T17:29:01.753 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-20T17:29:01.755 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:29:01.800 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:29:01.840 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:02.011 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:29:01 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:29:01.663+0000 7f4b1d05f7c0 -1 Falling back to public interface 2024-04-20T17:29:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:01 smithi106 ceph-mon[44423]: pgmap v1409: 17 pgs: 17 active+clean; 1.4 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.08k op/s 2024-04-20T17:29:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:02.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:02.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:02.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:01 smithi171 ceph-mon[44189]: pgmap v1409: 17 pgs: 17 active+clean; 1.4 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.08k op/s 2024-04-20T17:29:02.296 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:29:02.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:02.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:29:02.401 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:29:02.430 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:02.457 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:29:02.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:02.463 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:29:02.565 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:29:02.608 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:02.654 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:29:02.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:02.661 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:29:02.762 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:29:02.796 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:02.830 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:29:02.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:02.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:29:02.865 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:29:02.894 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:29:02.934 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:29:02.967 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:03.116 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:29:03.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:03.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:29:03.222 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:29:03.251 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:03.500 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:29:03.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:03.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:29:03.604 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:29:03.638 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:03.715 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:29:03.719 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:03 smithi106 ceph-mon[44423]: pgmap v1410: 17 pgs: 17 active+clean; 1.2 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:29:03.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:03.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:29:03.821 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:29:03.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:03 smithi005 ceph-mon[39562]: pgmap v1410: 17 pgs: 17 active+clean; 1.2 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:29:03.873 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:04.041 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:29:04.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:04.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:29:04.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:03 smithi171 ceph-mon[44189]: pgmap v1410: 17 pgs: 17 active+clean; 1.2 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:29:04.150 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:29:04.200 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:04.566 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:29:04.813 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-04-20T17:29:04.813 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-04-20T17:29:04.813 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-04-20T17:29:04.813 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-04-20T17:29:04.813 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-04-20T17:29:04.813 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-04-20T17:29:04.813 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-04-20T17:29:04.813 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-04-20T17:29:04.813 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-04-20T17:29:04.814 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-04-20T17:29:04.814 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator fragmentation block 2024-04-20T17:29:04.814 INFO:teuthology.orchestra.run.smithi106.stderr:admin_socket: invalid command 2024-04-20T17:29:04.906 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:04 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:29:04.907 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:04 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:29:04.907 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:04 smithi005 ceph-mon[39562]: osdmap e533: 12 total, 11 up, 11 in 2024-04-20T17:29:05.011 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:29:04 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:29:04.683+0000 7f4b1d05f7c0 -1 osd.6 503 log_to_monitors true 2024-04-20T17:29:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:04 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:29:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:04 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:29:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:04 smithi106 ceph-mon[44423]: osdmap e533: 12 total, 11 up, 11 in 2024-04-20T17:29:05.073 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:29:05.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:05.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:29:05.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:04 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:29:05.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:04 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:29:05.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:04 smithi171 ceph-mon[44189]: osdmap e533: 12 total, 11 up, 11 in 2024-04-20T17:29:05.177 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:29:05.216 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:05.241 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:29:05.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:05.244 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:29:05.345 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:29:05.379 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:05.430 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:29:05.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:05.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:29:05.537 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:29:05.577 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:05.605 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:29:05.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:05.606 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:29:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:05 smithi106 ceph-mon[44423]: pgmap v1412: 17 pgs: 17 active+clean; 1.0 GiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-20T17:29:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:05 smithi106 ceph-mon[44423]: from='osd.6 v1:172.21.15.106:6808/2439382182' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-20T17:29:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:05 smithi106 ceph-mon[44423]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-20T17:29:05.652 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:05 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3789432288' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:29:05.704 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:29:05.707 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:29:05.770 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:05 smithi171 ceph-mon[44189]: pgmap v1412: 17 pgs: 17 active+clean; 1.0 GiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-20T17:29:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:05 smithi171 ceph-mon[44189]: from='osd.6 v1:172.21.15.106:6808/2439382182' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-20T17:29:05.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:05 smithi171 ceph-mon[44189]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-20T17:29:05.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:05 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3789432288' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:29:05.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:05 smithi005 ceph-mon[39562]: pgmap v1412: 17 pgs: 17 active+clean; 1.0 GiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-20T17:29:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:05 smithi005 ceph-mon[39562]: from='osd.6 v1:172.21.15.106:6808/2439382182' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-20T17:29:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:05 smithi005 ceph-mon[39562]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-20T17:29:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:05 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3789432288' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:29:05.994 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:29:06.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:06.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:29:06.011 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:29:05 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:29:05.676+0000 7f4b0a4af700 -1 osd.6 503 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T17:29:06.101 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:29:06.153 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:06.185 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:29:06.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:06.193 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:29:06.277 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-20T17:29:06.277 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-04-20T17:29:06.294 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:29:06.345 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:06.410 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:29:06.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:06.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:29:06.518 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:29:06.554 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:06.578 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:29:06.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:06.585 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:29:06.588 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:29:06.588 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:29:06.686 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:29:06.722 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:06.754 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:29:06.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:06.757 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:29:06.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:06 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:29:06.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:06 smithi171 ceph-mon[44189]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-20T17:29:06.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:06 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3789432288' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:29:06.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:06 smithi171 ceph-mon[44189]: osdmap e534: 12 total, 11 up, 11 in 2024-04-20T17:29:06.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:06 smithi171 ceph-mon[44189]: from='osd.6 v1:172.21.15.106:6808/2439382182' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:29:06.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:06 smithi171 ceph-mon[44189]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:29:06.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:06 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:29:06.859 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:29:06.903 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:06.929 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:29:06.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:06.933 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:29:06.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:06 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:29:06.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:06 smithi005 ceph-mon[39562]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-20T17:29:06.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:06 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3789432288' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:29:06.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:06 smithi005 ceph-mon[39562]: osdmap e534: 12 total, 11 up, 11 in 2024-04-20T17:29:06.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:06 smithi005 ceph-mon[39562]: from='osd.6 v1:172.21.15.106:6808/2439382182' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:29:06.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:06 smithi005 ceph-mon[39562]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:29:06.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:06 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:29:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:06 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:29:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:06 smithi106 ceph-mon[44423]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-20T17:29:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:06 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3789432288' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:29:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:06 smithi106 ceph-mon[44423]: osdmap e534: 12 total, 11 up, 11 in 2024-04-20T17:29:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:06 smithi106 ceph-mon[44423]: from='osd.6 v1:172.21.15.106:6808/2439382182' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:29:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:06 smithi106 ceph-mon[44423]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:29:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:06 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:29:07.034 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:29:07.076 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:07.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:07.125 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:29:07.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:29:07.227 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:29:07.277 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:07.302 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:29:07.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:07.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:29:07.332 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:29:07.377 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:29:07.407 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:29:07.408 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:29:07.443 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:29:07.447 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:07.477 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:29:07.484 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:29:07.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:07.491 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:29:07.514 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:29:07.549 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:29:07.582 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:29:07.592 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:29:07.609 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:29:07.639 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:07.645 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:29:07.670 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:29:07.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:07.674 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:29:07.676 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:29:07.707 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:29:07.708 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:29:07.708 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:29:07.708 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:29:07.708 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:29:07.708 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:29:07.708 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:29:07.708 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-04-20T17:29:07.708 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:29:07.709 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:29:07.709 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:29:07.709 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:29:07.709 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:29:07.775 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:29:07.823 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:07 smithi171 ceph-mon[44189]: pgmap v1414: 17 pgs: 17 active+clean; 875 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.39k op/s 2024-04-20T17:29:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:07 smithi171 ceph-mon[44189]: osd.6 v1:172.21.15.106:6808/2439382182 boot 2024-04-20T17:29:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:07 smithi171 ceph-mon[44189]: osdmap e535: 12 total, 12 up, 11 in 2024-04-20T17:29:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:07 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:29:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:07 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3113465854' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:29:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:07.945 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:29:07.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:07.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:29:07.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:07 smithi005 ceph-mon[39562]: pgmap v1414: 17 pgs: 17 active+clean; 875 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.39k op/s 2024-04-20T17:29:07.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:07 smithi005 ceph-mon[39562]: osd.6 v1:172.21.15.106:6808/2439382182 boot 2024-04-20T17:29:07.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:07 smithi005 ceph-mon[39562]: osdmap e535: 12 total, 12 up, 11 in 2024-04-20T17:29:07.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:07 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:29:07.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:07 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3113465854' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:29:07.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:07.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:07 smithi106 ceph-mon[44423]: pgmap v1414: 17 pgs: 17 active+clean; 875 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.39k op/s 2024-04-20T17:29:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:07 smithi106 ceph-mon[44423]: osd.6 v1:172.21.15.106:6808/2439382182 boot 2024-04-20T17:29:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:07 smithi106 ceph-mon[44423]: osdmap e535: 12 total, 12 up, 11 in 2024-04-20T17:29:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:07 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-20T17:29:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:07 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3113465854' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:29:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:08.053 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:29:08.092 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:08.582 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:29:08.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:08.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:29:08.687 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:29:08.744 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:08.769 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:29:08.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:08.776 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:08.776 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:08 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:29:08.777 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:08 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:29:08.777 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:08.777 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:08.777 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:08.777 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:08.777 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:08.777 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:08.777 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:08.777 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:08.777 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:08.777 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:08.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:29:08.878 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:29:08.940 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:08.966 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:29:08.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:08.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:29:08.995 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:29:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:08 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:29:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:08 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:29:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:09.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:09.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:09.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:09.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:09.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:09.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:09.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:09.072 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:29:09.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:09.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:08 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:29:09.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:08 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:29:09.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:09.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:09.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:09.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:09.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:09.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:09.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:09.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:09.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:09.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:09.131 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:09.165 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:29:09.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:09.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:29:09.275 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:29:09.306 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:09.433 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:29:09.960 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:29:09.962 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:09 smithi106 ceph-mon[44423]: pgmap v1416: 17 pgs: 17 active+clean; 828 MiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 987 op/s 2024-04-20T17:29:09.962 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:09 smithi106 ceph-mon[44423]: osdmap e536: 12 total, 12 up, 11 in 2024-04-20T17:29:09.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:09.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:29:09.966 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-20T17:29:09.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:09 smithi005 ceph-mon[39562]: pgmap v1416: 17 pgs: 17 active+clean; 828 MiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 987 op/s 2024-04-20T17:29:09.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:09 smithi005 ceph-mon[39562]: osdmap e536: 12 total, 12 up, 11 in 2024-04-20T17:29:10.063 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:29:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:09 smithi171 ceph-mon[44189]: pgmap v1416: 17 pgs: 17 active+clean; 828 MiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 987 op/s 2024-04-20T17:29:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:09 smithi171 ceph-mon[44189]: osdmap e536: 12 total, 12 up, 11 in 2024-04-20T17:29:10.099 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:10.124 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:29:10.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:10.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:29:10.234 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:29:10.283 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:10.327 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:29:10.551 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:29:11.278 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-20T17:29:11.945 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:29:11.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:11.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:29:11.958 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:11 smithi005 ceph-mon[39562]: pgmap v1418: 17 pgs: 17 active+clean; 643 MiB data, 4.3 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-20T17:29:11.958 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:11 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:29:11.958 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:11 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:29:11.958 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:11 smithi005 ceph-mon[39562]: osdmap e537: 12 total, 12 up, 11 in 2024-04-20T17:29:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:11 smithi106 ceph-mon[44423]: pgmap v1418: 17 pgs: 17 active+clean; 643 MiB data, 4.3 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-20T17:29:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:11 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:29:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:11 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:29:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:11 smithi106 ceph-mon[44423]: osdmap e537: 12 total, 12 up, 11 in 2024-04-20T17:29:12.048 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:29:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:11 smithi171 ceph-mon[44189]: pgmap v1418: 17 pgs: 17 active+clean; 643 MiB data, 4.3 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-20T17:29:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:11 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:29:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:11 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:29:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:11 smithi171 ceph-mon[44189]: osdmap e537: 12 total, 12 up, 11 in 2024-04-20T17:29:12.092 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:12.121 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:29:12.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:12.127 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:29:12.228 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:29:12.265 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:12.293 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:29:12.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:12.298 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:29:12.399 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:29:12.413 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:29:12.447 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:12.448 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:29:12.448 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:29:12.448 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-04-20T17:29:12.448 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-04-20T17:29:12.448 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:29:13.076 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:29:13.401 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:29:13.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:13.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:29:13.506 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:29:13.550 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:13.579 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:29:13.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:13.585 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:29:13.686 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:29:13.741 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:13.832 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:29:13.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:13 smithi171 ceph-mon[44189]: pgmap v1420: 17 pgs: 17 active+clean; 457 MiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-20T17:29:13.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:13.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:29:13.944 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:29:13.979 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:13 smithi106 ceph-mon[44423]: pgmap v1420: 17 pgs: 17 active+clean; 457 MiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-20T17:29:13.983 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:14.009 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:29:14.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:14.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:29:14.119 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:29:14.177 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:13 smithi005 ceph-mon[39562]: pgmap v1420: 17 pgs: 17 active+clean; 457 MiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-20T17:29:14.180 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:14.316 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:29:14.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:14.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:29:14.436 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:29:14.440 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-20T17:29:14.468 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:14.902 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:29:14.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:14 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/908157837' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:29:15.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:14 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/908157837' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:29:15.033 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:29:15.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:15.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:29:15.064 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:29:15.066 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:29:15.067 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:29:15.067 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-04-20T17:29:15.067 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:29:15.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:14 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/908157837' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:29:15.138 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:29:15.171 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:15.417 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:29:15.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:15.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:29:15.524 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:29:15.559 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:15.828 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:29:15.864 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:29:15.895 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:29:15.928 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:29:15.960 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:29:15.989 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:29:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:15 smithi106 ceph-mon[44423]: pgmap v1421: 17 pgs: 17 active+clean; 273 MiB data, 3.6 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-20T17:29:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:15 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:29:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:15 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/908157837' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:29:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:15 smithi106 ceph-mon[44423]: osdmap e538: 12 total, 12 up, 11 in 2024-04-20T17:29:16.021 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:29:16.056 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:29:16.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:15 smithi171 ceph-mon[44189]: pgmap v1421: 17 pgs: 17 active+clean; 273 MiB data, 3.6 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-20T17:29:16.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:15 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:29:16.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:15 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/908157837' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:29:16.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:15 smithi171 ceph-mon[44189]: osdmap e538: 12 total, 12 up, 11 in 2024-04-20T17:29:16.084 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:15 smithi005 ceph-mon[39562]: pgmap v1421: 17 pgs: 17 active+clean; 273 MiB data, 3.6 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-20T17:29:16.084 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:15 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:29:16.084 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:15 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/908157837' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:29:16.084 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:15 smithi005 ceph-mon[39562]: osdmap e538: 12 total, 12 up, 11 in 2024-04-20T17:29:16.102 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:29:16.109 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:29:16.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:16.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:29:16.142 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:29:16.175 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:29:16.189 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Removed 170189 objects 2024-04-20T17:29:16.190 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Clean up completed and total clean up time :177.704 2024-04-20T17:29:16.203 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:29:16.204 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:29:16.204 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:29:16.204 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:29:16.204 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:29:16.204 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:29:16.204 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:29:16.204 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:29:16.204 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:29:16.204 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:29:16.205 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:29:16.205 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:29:16.205 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:29:16.213 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:29:16.244 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:16.495 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:29:16.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:16.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:29:16.599 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:29:16.626 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:16.644 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:29:16.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:16.647 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:29:16.748 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:29:16.774 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:16.792 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:29:16.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:16.795 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:29:16.827 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-04-20T17:29:16.827 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-04-20T17:29:16.855 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:16 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3500542530' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:29:16.855 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:16.856 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:16.856 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:16.856 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:16.856 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:16.856 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:16.856 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:16.856 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:16.856 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:16.856 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:16.882 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:16 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3500542530' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:29:16.883 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:16.883 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:16.883 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:16.883 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:16.883 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:16.883 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:16.883 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:16.883 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:16.883 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:16.883 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:16.896 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:29:16.923 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:17.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:16 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3500542530' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:29:17.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:17.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:17.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:17.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:17.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:17.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:17.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:17.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:17.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:17.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:17.107 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:29:17.239 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:29:17.240 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 1, 3, 4, 5, 8, 9, 10, 11, 2, 7, 6] 2024-04-20T17:29:17.240 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-20T17:29:17.240 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.3 2024-04-20T17:29:17.372 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:29:17.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:17.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:29:17.476 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:29:17.507 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:17.581 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:29:17.583 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:29:17 smithi005 systemd[1]: Stopping Ceph osd.3 for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:29:17.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:17.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:29:17.646 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:29:17.646 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:29:17.684 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:29:17.713 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:17.733 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:29:17.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:17.736 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:29:17.837 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:29:17.867 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:17.904 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:29:17.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:17.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:29:17.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:17.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:17.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:17 smithi005 ceph-mon[39562]: pgmap v1423: 17 pgs: 17 active+clean; 134 MiB data, 3.3 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.46k op/s 2024-04-20T17:29:17.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:17 smithi005 ceph-mon[39562]: osd.3 marked itself down and dead 2024-04-20T17:29:17.962 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:29:17 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-3[80507]: 2024-04-20T17:29:17.585+0000 7f2376390700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T17:29:17.962 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:29:17 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-3[80507]: 2024-04-20T17:29:17.585+0000 7f2376390700 -1 osd.3 538 *** Got signal Terminated *** 2024-04-20T17:29:17.962 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:29:17 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-3[80507]: 2024-04-20T17:29:17.585+0000 7f2376390700 -1 osd.3 538 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T17:29:18.008 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:29:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:17 smithi106 ceph-mon[44423]: pgmap v1423: 17 pgs: 17 active+clean; 134 MiB data, 3.3 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.46k op/s 2024-04-20T17:29:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:17 smithi106 ceph-mon[44423]: osd.3 marked itself down and dead 2024-04-20T17:29:18.038 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:18.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:18.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:18.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:17 smithi171 ceph-mon[44189]: pgmap v1423: 17 pgs: 17 active+clean; 134 MiB data, 3.3 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.46k op/s 2024-04-20T17:29:18.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:17 smithi171 ceph-mon[44189]: osd.3 marked itself down and dead 2024-04-20T17:29:18.124 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:29:18.336 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-20T17:29:18.489 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:29:18.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:18.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:29:18.582 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:29:18.593 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:29:18.607 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:29:18.621 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:18.641 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:29:18.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:18.645 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:29:18.738 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:18 smithi005 ceph-mon[39562]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T17:29:18.738 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:18 smithi005 ceph-mon[39562]: osdmap e539: 12 total, 11 up, 11 in 2024-04-20T17:29:18.746 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:29:18.774 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:18.794 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:29:18.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:18.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:29:18.888 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:18 smithi171 ceph-mon[44189]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T17:29:18.888 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:18 smithi171 ceph-mon[44189]: osdmap e539: 12 total, 11 up, 11 in 2024-04-20T17:29:18.897 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:29:18.923 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:18.942 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:29:18.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:18.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:29:19.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:18 smithi106 ceph-mon[44423]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T17:29:19.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:18 smithi106 ceph-mon[44423]: osdmap e539: 12 total, 11 up, 11 in 2024-04-20T17:29:19.043 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:29:19.045 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:29:19.072 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:19.091 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:29:19.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:19.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:29:19.194 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:29:19.221 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:19.239 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:29:19.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:19.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:29:19.344 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:29:19.369 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:19.388 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:29:19.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:19.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:29:19.492 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:29:19.522 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:19 smithi171 ceph-mon[44189]: pgmap v1425: 17 pgs: 2 stale+active+clean, 15 active+clean; 90 MiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-20T17:29:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:19 smithi171 ceph-mon[44189]: osdmap e540: 12 total, 11 up, 11 in 2024-04-20T17:29:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:19 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/20945305' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-20T17:29:19.970 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:29:19.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:19.972 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:29:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:19 smithi106 ceph-mon[44423]: pgmap v1425: 17 pgs: 2 stale+active+clean, 15 active+clean; 90 MiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-20T17:29:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:19 smithi106 ceph-mon[44423]: osdmap e540: 12 total, 11 up, 11 in 2024-04-20T17:29:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:19 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/20945305' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-20T17:29:20.017 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:19 smithi005 ceph-mon[39562]: pgmap v1425: 17 pgs: 2 stale+active+clean, 15 active+clean; 90 MiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-20T17:29:20.017 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:19 smithi005 ceph-mon[39562]: osdmap e540: 12 total, 11 up, 11 in 2024-04-20T17:29:20.017 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:19 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/20945305' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-20T17:29:20.073 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:29:20.104 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:20.123 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:29:20.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:20.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:29:20.227 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:29:20.244 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'unique_pool_2' removed 2024-04-20T17:29:20.264 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:20.270 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:29:20 smithi005 bash[226506]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-3 2024-04-20T17:29:20.287 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:29:20.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:20.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:29:20.393 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:29:20.421 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:20.440 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:29:20.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:20.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:29:20.470 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:29:20.470 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:29:20.545 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:29:20.573 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:20.671 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:29:20.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:20.674 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:29:20.775 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:29:20.805 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:20.929 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:29:20.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:20.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:29:21.033 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:29:21.060 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:21.097 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:29:21.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:21.100 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:29:21.122 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-20T17:29:21.201 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:29:21.230 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:21.251 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:29:21.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:21.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:29:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:21 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/20945305' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-04-20T17:29:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:21 smithi171 ceph-mon[44189]: osdmap e541: 12 total, 11 up, 11 in 2024-04-20T17:29:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:21 smithi171 ceph-mon[44189]: pgmap v1428: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:21.354 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:29:21.385 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:21.404 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:29:21.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:21.408 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:29:21.416 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:29:21.430 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:29:21.451 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-20T17:29:21.452 INFO:tasks.radosbench:Beginning radosbench... 2024-04-20T17:29:21.452 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-04-20T17:29:21.452 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-04-20T17:29:21.508 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:29:21.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:21 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/20945305' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-04-20T17:29:21.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:21 smithi106 ceph-mon[44423]: osdmap e541: 12 total, 11 up, 11 in 2024-04-20T17:29:21.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:21 smithi106 ceph-mon[44423]: pgmap v1428: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:21.533 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:21 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/20945305' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-04-20T17:29:21.533 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:21 smithi005 ceph-mon[39562]: osdmap e541: 12 total, 11 up, 11 in 2024-04-20T17:29:21.533 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:21 smithi005 ceph-mon[39562]: pgmap v1428: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:21.536 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:21.556 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:29:21.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:21.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:29:21.662 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:29:21.692 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:21.711 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:29:21.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:21.714 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:29:21.815 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:29:21.843 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:21.915 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:29:21.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:21.918 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:29:22.019 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:29:22.052 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:22.074 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:29:22.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:22.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:29:22.177 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:29:22.205 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:22.224 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:29:22.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:22.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:29:22.328 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:29:22.358 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:22.378 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:29:22.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:22.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:29:22.482 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:29:22.508 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:22.526 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:29:22.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:22.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:29:22.630 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:29:22.658 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:22.678 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:29:22.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:22.682 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:29:22.783 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:29:22.811 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:22.815 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:29:22.883 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:29:22.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:22.886 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:29:22.987 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:29:23.016 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:23.113 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:29:23.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:23.117 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:29:23.148 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:29:23.148 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:29:23.218 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:29:23.249 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:23.322 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:29:23.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:23.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:29:23.337 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:29:23.427 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:29:23.457 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:23.472 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:23 smithi171 ceph-mon[44189]: pgmap v1429: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 981 GiB / 983 GiB avail 2024-04-20T17:29:23.472 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:23 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1799350329' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:29:23.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:23 smithi005 ceph-mon[39562]: pgmap v1429: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 981 GiB / 983 GiB avail 2024-04-20T17:29:23.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:23 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1799350329' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:29:23.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:23 smithi106 ceph-mon[44423]: pgmap v1429: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 981 GiB / 983 GiB avail 2024-04-20T17:29:23.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:23 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1799350329' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:29:23.778 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-20T17:29:23.906 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:29:23.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:23.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:29:23.923 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:29:23.943 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:29:23.962 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:29:23.983 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:29:24.004 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:29:24.013 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:29:24.024 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:29:24.044 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:29:24.045 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:24.064 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:29:24.066 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:29:24.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:24.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:29:24.081 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:29:24.084 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:29:24.104 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:29:24.123 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:29:24.124 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:29:24.124 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:29:24.124 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:29:24.124 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-20T17:29:24.124 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:29:24.124 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:29:24.124 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:29:24.124 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:29:24.124 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:29:24.124 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:29:24.125 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:29:24.125 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:29:24.170 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:29:24.201 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:24.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:24 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:29:24.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:24 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1799350329' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:29:24.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:24 smithi005 ceph-mon[39562]: osdmap e542: 12 total, 11 up, 11 in 2024-04-20T17:29:24.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:24 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1964427514' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:29:24.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:24 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:24 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:29:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:24 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1799350329' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:29:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:24 smithi171 ceph-mon[44189]: osdmap e542: 12 total, 11 up, 11 in 2024-04-20T17:29:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:24 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1964427514' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:29:24.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:24 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:24.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:24 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:29:24.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:24 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1799350329' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:29:24.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:24 smithi106 ceph-mon[44423]: osdmap e542: 12 total, 11 up, 11 in 2024-04-20T17:29:24.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:24 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1964427514' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:29:24.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:24 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:25 smithi171 ceph-mon[44189]: pgmap v1431: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:25.586 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.586 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.586 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.586 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.586 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.586 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.587 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.587 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.587 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.587 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.587 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:25 smithi005 ceph-mon[39562]: pgmap v1431: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:25.587 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:29:25.588 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:29:25.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:25.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:25 smithi106 ceph-mon[44423]: pgmap v1431: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:25.864 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:29:25.865 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:29:25.865 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:29:25.865 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:29:25.865 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.865 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:158227 4.2 4:49e0ccb7:::benchmark_data_smithi171_122772_object158226:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e482)", 2024-04-20T17:29:25.865 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:08.022669+0000", 2024-04-20T17:29:25.865 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 197.840051149, 2024-04-20T17:29:25.865 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.13561293899999999, 2024-04-20T17:29:25.865 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:29:25.865 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:29:25.865 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:29:25.865 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:29:25.866 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:29:25.866 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:29:25.866 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 158227 2024-04-20T17:29:25.866 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.866 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:29:25.866 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.866 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:29:25.866 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.022669+0000", 2024-04-20T17:29:25.866 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.866 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.866 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.866 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:29:25.866 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.022669+0000", 2024-04-20T17:29:25.867 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.867 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.867 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.867 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:29:25.867 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.022670+0000", 2024-04-20T17:29:25.867 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1299999999999999e-07 2024-04-20T17:29:25.867 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.867 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.867 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:29:25.867 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.022704+0000", 2024-04-20T17:29:25.867 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4038999999999997e-05 2024-04-20T17:29:25.867 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.867 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.868 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:29:25.868 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.022712+0000", 2024-04-20T17:29:25.868 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2600000000000005e-06 2024-04-20T17:29:25.868 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.868 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.868 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:29:25.868 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.022718+0000", 2024-04-20T17:29:25.868 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9190000000000004e-06 2024-04-20T17:29:25.868 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.868 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.868 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:29:25.868 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.058012+0000", 2024-04-20T17:29:25.868 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.035293366999999999 2024-04-20T17:29:25.869 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.869 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.869 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:29:25.869 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.058140+0000", 2024-04-20T17:29:25.869 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012853300000000001 2024-04-20T17:29:25.869 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.869 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.869 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:29:25.869 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.058307+0000", 2024-04-20T17:29:25.869 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00016693200000000001 2024-04-20T17:29:25.869 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.869 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.869 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:29:25.869 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.063270+0000", 2024-04-20T17:29:25.869 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0049627849999999999 2024-04-20T17:29:25.870 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.870 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.870 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:29:25.870 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.158144+0000", 2024-04-20T17:29:25.870 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.094873825999999994 2024-04-20T17:29:25.870 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.870 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.870 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:29:25.870 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.158211+0000", 2024-04-20T17:29:25.870 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7372000000000005e-05 2024-04-20T17:29:25.870 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.870 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.870 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:29:25.871 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.158282+0000", 2024-04-20T17:29:25.871 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1093000000000005e-05 2024-04-20T17:29:25.871 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.871 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:29:25.871 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.871 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.871 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.871 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:158545 4.2 4:4e9d0a93:::benchmark_data_smithi171_122772_object158544:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e482)", 2024-04-20T17:29:25.871 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:08.328262+0000", 2024-04-20T17:29:25.871 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 197.53445808999999, 2024-04-20T17:29:25.871 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.135778385, 2024-04-20T17:29:25.871 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:29:25.871 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:29:25.872 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:29:25.872 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:29:25.872 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:29:25.872 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:29:25.872 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 158545 2024-04-20T17:29:25.872 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.872 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:29:25.872 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.872 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:29:25.872 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.328262+0000", 2024-04-20T17:29:25.872 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.872 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.872 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.872 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:29:25.872 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.328262+0000", 2024-04-20T17:29:25.873 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.873 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.873 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.873 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:29:25.873 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.328264+0000", 2024-04-20T17:29:25.873 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.33e-06 2024-04-20T17:29:25.873 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.873 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.873 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:29:25.873 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.328285+0000", 2024-04-20T17:29:25.873 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1305999999999999e-05 2024-04-20T17:29:25.873 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.873 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.874 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:29:25.874 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.328296+0000", 2024-04-20T17:29:25.874 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1343e-05 2024-04-20T17:29:25.874 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.874 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.874 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:29:25.874 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.328304+0000", 2024-04-20T17:29:25.874 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3250000000000002e-06 2024-04-20T17:29:25.874 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.874 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.874 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:29:25.874 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.437252+0000", 2024-04-20T17:29:25.874 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.108948847 2024-04-20T17:29:25.874 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.875 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.875 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:29:25.875 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.437407+0000", 2024-04-20T17:29:25.875 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015485499999999999 2024-04-20T17:29:25.875 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.875 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.875 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:29:25.875 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.437511+0000", 2024-04-20T17:29:25.875 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010353699999999999 2024-04-20T17:29:25.875 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.875 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.875 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:29:25.875 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.445343+0000", 2024-04-20T17:29:25.875 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0078318910000000005 2024-04-20T17:29:25.876 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.876 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.876 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:29:25.876 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.463963+0000", 2024-04-20T17:29:25.876 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018619858999999999 2024-04-20T17:29:25.876 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.876 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.876 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:29:25.876 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.463996+0000", 2024-04-20T17:29:25.876 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3600999999999999e-05 2024-04-20T17:29:25.876 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.876 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.876 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:29:25.877 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.464041+0000", 2024-04-20T17:29:25.877 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4490999999999998e-05 2024-04-20T17:29:25.877 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.877 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:29:25.877 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.877 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.877 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.877 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:158549 4.2 4:49a4cca7:::benchmark_data_smithi171_122772_object158548:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e482)", 2024-04-20T17:29:25.877 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:08.332525+0000", 2024-04-20T17:29:25.877 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 197.53019574000001, 2024-04-20T17:29:25.877 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.13162052799999999, 2024-04-20T17:29:25.877 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:29:25.877 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:29:25.877 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:29:25.877 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:29:25.878 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:29:25.878 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:29:25.878 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 158549 2024-04-20T17:29:25.878 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.878 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:29:25.878 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.878 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:29:25.878 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.332525+0000", 2024-04-20T17:29:25.878 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.878 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.878 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.878 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:29:25.878 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.332525+0000", 2024-04-20T17:29:25.879 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.879 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.879 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.879 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:29:25.879 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.332527+0000", 2024-04-20T17:29:25.879 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0049999999999999e-06 2024-04-20T17:29:25.879 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.879 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.879 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:29:25.879 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.332549+0000", 2024-04-20T17:29:25.879 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2297999999999999e-05 2024-04-20T17:29:25.879 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.879 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.879 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:29:25.880 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.332560+0000", 2024-04-20T17:29:25.880 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1001e-05 2024-04-20T17:29:25.880 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.880 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.880 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:29:25.880 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.332566+0000", 2024-04-20T17:29:25.880 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7359999999999998e-06 2024-04-20T17:29:25.880 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.880 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.880 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:29:25.880 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.441479+0000", 2024-04-20T17:29:25.880 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.108913145 2024-04-20T17:29:25.880 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.880 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.881 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:29:25.881 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.441607+0000", 2024-04-20T17:29:25.881 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000127942 2024-04-20T17:29:25.881 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.881 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.881 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:29:25.881 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.441702+0000", 2024-04-20T17:29:25.881 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5395000000000004e-05 2024-04-20T17:29:25.881 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.881 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.881 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:29:25.881 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.445424+0000", 2024-04-20T17:29:25.881 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0037214819999999999 2024-04-20T17:29:25.882 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.882 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.882 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:29:25.882 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.464051+0000", 2024-04-20T17:29:25.882 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018627281999999998 2024-04-20T17:29:25.882 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.882 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.882 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:29:25.882 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.464099+0000", 2024-04-20T17:29:25.882 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7806999999999998e-05 2024-04-20T17:29:25.882 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.882 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.882 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:29:25.882 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.464145+0000", 2024-04-20T17:29:25.883 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6434999999999998e-05 2024-04-20T17:29:25.883 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.883 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:29:25.883 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.883 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.883 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.883 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:159061 4.2 4:4cdccfcc:::benchmark_data_smithi171_122772_object159060:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e482)", 2024-04-20T17:29:25.883 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:08.824368+0000", 2024-04-20T17:29:25.883 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 197.03835226199999, 2024-04-20T17:29:25.883 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15598972799999999, 2024-04-20T17:29:25.883 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:29:25.883 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:29:25.883 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:29:25.884 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:29:25.884 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:29:25.884 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:29:25.884 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 159061 2024-04-20T17:29:25.884 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.884 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:29:25.884 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.884 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:29:25.884 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824368+0000", 2024-04-20T17:29:25.884 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.884 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.884 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.884 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:29:25.884 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824368+0000", 2024-04-20T17:29:25.884 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.885 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.885 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.885 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:29:25.885 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824371+0000", 2024-04-20T17:29:25.885 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.424e-06 2024-04-20T17:29:25.885 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.885 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.885 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:29:25.885 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824398+0000", 2024-04-20T17:29:25.885 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7310000000000001e-05 2024-04-20T17:29:25.885 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.885 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.885 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:29:25.885 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824408+0000", 2024-04-20T17:29:25.886 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0295999999999999e-05 2024-04-20T17:29:25.886 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.886 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.886 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:29:25.886 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824414+0000", 2024-04-20T17:29:25.886 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9989999999999997e-06 2024-04-20T17:29:25.886 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.886 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.886 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:29:25.886 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.827811+0000", 2024-04-20T17:29:25.886 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0033968319999999998 2024-04-20T17:29:25.886 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.886 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.887 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:29:25.887 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.827940+0000", 2024-04-20T17:29:25.887 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012889600000000001 2024-04-20T17:29:25.887 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.887 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.887 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:29:25.887 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.828088+0000", 2024-04-20T17:29:25.887 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014852100000000001 2024-04-20T17:29:25.887 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.887 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.887 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:29:25.887 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.836604+0000", 2024-04-20T17:29:25.887 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0085154889999999994 2024-04-20T17:29:25.887 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.888 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.888 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:29:25.888 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980241+0000", 2024-04-20T17:29:25.888 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.14363725299999999 2024-04-20T17:29:25.888 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.888 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.888 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:29:25.888 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980298+0000", 2024-04-20T17:29:25.888 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6462999999999998e-05 2024-04-20T17:29:25.888 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.888 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.888 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:29:25.888 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980358+0000", 2024-04-20T17:29:25.888 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0244999999999997e-05 2024-04-20T17:29:25.889 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.889 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:29:25.889 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.889 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.889 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.889 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:159066 4.2 4:41a9bca0:::benchmark_data_smithi171_122772_object159065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e482)", 2024-04-20T17:29:25.889 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:08.824564+0000", 2024-04-20T17:29:25.889 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 197.038156493, 2024-04-20T17:29:25.889 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.155931294, 2024-04-20T17:29:25.889 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:29:25.889 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:29:25.889 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:29:25.889 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:29:25.889 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:29:25.889 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:29:25.890 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 159066 2024-04-20T17:29:25.890 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.890 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:29:25.890 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.890 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:29:25.890 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824564+0000", 2024-04-20T17:29:25.890 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.890 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.890 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.890 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:29:25.890 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824564+0000", 2024-04-20T17:29:25.890 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.890 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.890 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.891 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:29:25.891 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824565+0000", 2024-04-20T17:29:25.891 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1510000000000001e-06 2024-04-20T17:29:25.891 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.891 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.891 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:29:25.891 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824583+0000", 2024-04-20T17:29:25.891 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7839e-05 2024-04-20T17:29:25.891 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.891 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.891 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:29:25.891 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824592+0000", 2024-04-20T17:29:25.891 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9290000000000004e-06 2024-04-20T17:29:25.891 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.892 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.892 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:29:25.892 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824596+0000", 2024-04-20T17:29:25.892 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3590000000000001e-06 2024-04-20T17:29:25.892 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.892 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.892 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:29:25.892 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.832585+0000", 2024-04-20T17:29:25.892 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0079889399999999999 2024-04-20T17:29:25.892 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.892 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.892 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:29:25.892 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.832687+0000", 2024-04-20T17:29:25.892 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000101487 2024-04-20T17:29:25.893 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.893 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.893 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:29:25.893 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.832792+0000", 2024-04-20T17:29:25.893 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010508100000000001 2024-04-20T17:29:25.893 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.893 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.893 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:29:25.893 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980174+0000", 2024-04-20T17:29:25.893 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.14738206500000001 2024-04-20T17:29:25.893 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.893 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.893 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:29:25.894 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980405+0000", 2024-04-20T17:29:25.894 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00023092499999999999 2024-04-20T17:29:25.894 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.894 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.894 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:29:25.894 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980449+0000", 2024-04-20T17:29:25.894 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4351000000000003e-05 2024-04-20T17:29:25.894 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.894 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.894 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:29:25.894 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980495+0000", 2024-04-20T17:29:25.894 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6167e-05 2024-04-20T17:29:25.894 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.894 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:29:25.895 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.895 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.895 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.895 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop_reply(client.26074.0:159061 4.2 e482/445 ondisk, result = 0)", 2024-04-20T17:29:25.895 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:08.839536+0000", 2024-04-20T17:29:25.895 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 197.023184606, 2024-04-20T17:29:25.895 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.14083410099999999, 2024-04-20T17:29:25.895 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:29:25.895 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:29:25.895 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:29:25.895 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:29:25.895 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.895 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:29:25.895 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.839536+0000", 2024-04-20T17:29:25.896 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.896 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.896 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.896 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:29:25.896 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.839536+0000", 2024-04-20T17:29:25.896 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.896 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.896 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.896 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:29:25.896 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.839538+0000", 2024-04-20T17:29:25.896 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.951e-06 2024-04-20T17:29:25.896 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.896 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.896 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:29:25.896 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.839545+0000", 2024-04-20T17:29:25.897 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.114e-06 2024-04-20T17:29:25.897 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.897 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.897 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:29:25.897 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.887991+0000", 2024-04-20T17:29:25.897 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.048445941999999999 2024-04-20T17:29:25.897 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.897 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.897 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:29:25.897 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.888002+0000", 2024-04-20T17:29:25.897 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1425e-05 2024-04-20T17:29:25.897 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.897 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.897 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:29:25.898 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980224+0000", 2024-04-20T17:29:25.898 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.092222193999999993 2024-04-20T17:29:25.898 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.898 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.898 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:29:25.898 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980234+0000", 2024-04-20T17:29:25.898 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1099999999999992e-06 2024-04-20T17:29:25.898 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.898 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.898 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:29:25.898 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980370+0000", 2024-04-20T17:29:25.898 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000136365 2024-04-20T17:29:25.898 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.898 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:29:25.899 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.899 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.899 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.899 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:159675 4.2 4:45a9239e:::benchmark_data_smithi171_122772_object159674:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e482)", 2024-04-20T17:29:25.899 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:09.405527+0000", 2024-04-20T17:29:25.899 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 196.45719353199999, 2024-04-20T17:29:25.899 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.162906468, 2024-04-20T17:29:25.899 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:29:25.899 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:29:25.899 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:29:25.899 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:29:25.899 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:29:25.899 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:29:25.899 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 159675 2024-04-20T17:29:25.899 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.899 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:29:25.900 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.900 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:29:25.900 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.405527+0000", 2024-04-20T17:29:25.900 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.900 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.900 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.900 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:29:25.900 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.405527+0000", 2024-04-20T17:29:25.900 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.900 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.900 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.900 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:29:25.900 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.405528+0000", 2024-04-20T17:29:25.900 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.4e-07 2024-04-20T17:29:25.901 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.901 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.901 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:29:25.901 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.405546+0000", 2024-04-20T17:29:25.901 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8229000000000001e-05 2024-04-20T17:29:25.901 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.901 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.901 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:29:25.901 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.405553+0000", 2024-04-20T17:29:25.901 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8109999999999997e-06 2024-04-20T17:29:25.901 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.901 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.901 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:29:25.902 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.405559+0000", 2024-04-20T17:29:25.902 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1500000000000004e-06 2024-04-20T17:29:25.902 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.902 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.902 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:29:25.902 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.437169+0000", 2024-04-20T17:29:25.902 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.031610304999999998 2024-04-20T17:29:25.902 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.902 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.902 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:29:25.902 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.437261+0000", 2024-04-20T17:29:25.902 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1257999999999999e-05 2024-04-20T17:29:25.902 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.903 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.903 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:29:25.903 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.437390+0000", 2024-04-20T17:29:25.903 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000129777 2024-04-20T17:29:25.903 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.903 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.903 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:29:25.903 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.451123+0000", 2024-04-20T17:29:25.903 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013732496 2024-04-20T17:29:25.903 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.903 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.903 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:29:25.903 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.568336+0000", 2024-04-20T17:29:25.903 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.11721342899999999 2024-04-20T17:29:25.903 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.904 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.904 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:29:25.904 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.568379+0000", 2024-04-20T17:29:25.904 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2344e-05 2024-04-20T17:29:25.904 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.904 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.904 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:29:25.904 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.568433+0000", 2024-04-20T17:29:25.904 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4728999999999997e-05 2024-04-20T17:29:25.904 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.904 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:29:25.904 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.904 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.904 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.904 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:159676 4.2 4:43aa80d4:::benchmark_data_smithi171_122772_object159675:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e482)", 2024-04-20T17:29:25.905 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:09.407698+0000", 2024-04-20T17:29:25.905 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 196.45502245599999, 2024-04-20T17:29:25.905 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16090689399999999, 2024-04-20T17:29:25.905 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:29:25.905 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:29:25.905 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:29:25.905 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:29:25.905 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:29:25.905 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:29:25.905 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 159676 2024-04-20T17:29:25.905 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.905 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:29:25.905 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.905 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:29:25.905 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.407698+0000", 2024-04-20T17:29:25.906 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.906 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.906 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.906 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:29:25.906 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.407698+0000", 2024-04-20T17:29:25.906 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.906 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.906 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.906 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:29:25.906 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.407699+0000", 2024-04-20T17:29:25.906 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4309999999999999e-06 2024-04-20T17:29:25.906 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.906 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.906 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:29:25.907 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.407720+0000", 2024-04-20T17:29:25.907 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.09e-05 2024-04-20T17:29:25.907 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.907 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.907 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:29:25.907 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.407732+0000", 2024-04-20T17:29:25.907 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1494999999999999e-05 2024-04-20T17:29:25.907 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.907 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.907 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:29:25.907 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.407740+0000", 2024-04-20T17:29:25.907 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9459999999999998e-06 2024-04-20T17:29:25.907 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.907 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.907 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:29:25.908 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.441541+0000", 2024-04-20T17:29:25.908 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.033800864 2024-04-20T17:29:25.908 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.908 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.908 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:29:25.908 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.441652+0000", 2024-04-20T17:29:25.908 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000111749 2024-04-20T17:29:25.908 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.908 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.908 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:29:25.908 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.442325+0000", 2024-04-20T17:29:25.908 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00067265499999999998 2024-04-20T17:29:25.908 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.909 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.909 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:29:25.909 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.451395+0000", 2024-04-20T17:29:25.909 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0090699519999999992 2024-04-20T17:29:25.909 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.909 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.909 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:29:25.909 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.568492+0000", 2024-04-20T17:29:25.909 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.117097259 2024-04-20T17:29:25.909 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.909 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.909 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:29:25.909 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.568546+0000", 2024-04-20T17:29:25.909 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3780999999999999e-05 2024-04-20T17:29:25.909 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.910 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.910 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:29:25.910 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.568605+0000", 2024-04-20T17:29:25.910 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8862e-05 2024-04-20T17:29:25.910 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.910 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:29:25.910 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.910 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.910 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.910 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:159684 4.2 4:43ec9ad4:::benchmark_data_smithi171_122772_object159683:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e482)", 2024-04-20T17:29:25.910 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:09.415274+0000", 2024-04-20T17:29:25.910 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 196.44744681399999, 2024-04-20T17:29:25.910 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15358047, 2024-04-20T17:29:25.910 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:29:25.910 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:29:25.911 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:29:25.911 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:29:25.911 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:29:25.911 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:29:25.911 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 159684 2024-04-20T17:29:25.911 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.911 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:29:25.911 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.911 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:29:25.911 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.415274+0000", 2024-04-20T17:29:25.911 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.911 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.911 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.911 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:29:25.912 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.415274+0000", 2024-04-20T17:29:25.912 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.912 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.912 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.912 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:29:25.912 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.415276+0000", 2024-04-20T17:29:25.912 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.216e-06 2024-04-20T17:29:25.912 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.912 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.912 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:29:25.912 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.415300+0000", 2024-04-20T17:29:25.912 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4321e-05 2024-04-20T17:29:25.912 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.912 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.913 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:29:25.913 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.415312+0000", 2024-04-20T17:29:25.913 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1511e-05 2024-04-20T17:29:25.913 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.913 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.913 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:29:25.913 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.415319+0000", 2024-04-20T17:29:25.913 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8690000000000004e-06 2024-04-20T17:29:25.913 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.913 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.913 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:29:25.913 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.447198+0000", 2024-04-20T17:29:25.913 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.031878451000000002 2024-04-20T17:29:25.914 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.914 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.914 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:29:25.914 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.447351+0000", 2024-04-20T17:29:25.914 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015293300000000001 2024-04-20T17:29:25.914 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.914 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.914 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:29:25.914 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.447511+0000", 2024-04-20T17:29:25.914 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000159872 2024-04-20T17:29:25.914 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.914 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.914 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:29:25.915 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.451449+0000", 2024-04-20T17:29:25.915 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0039380539999999999 2024-04-20T17:29:25.915 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.915 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.915 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:29:25.915 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.568785+0000", 2024-04-20T17:29:25.915 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.11733666099999999 2024-04-20T17:29:25.915 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.915 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.915 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:29:25.915 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.568820+0000", 2024-04-20T17:29:25.915 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4586999999999999e-05 2024-04-20T17:29:25.915 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.915 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.916 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:29:25.916 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.568854+0000", 2024-04-20T17:29:25.916 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3995000000000002e-05 2024-04-20T17:29:25.916 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.916 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:29:25.916 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.916 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.916 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.916 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275200 4.6 4:65b8144d:::benchmark_data_smithi171_122772_object105008:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:29:25.916 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:10.998559+0000", 2024-04-20T17:29:25.916 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 74.864161843999995, 2024-04-20T17:29:25.916 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.21653923, 2024-04-20T17:29:25.916 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:29:25.916 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:29:25.917 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:29:25.917 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:29:25.917 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:29:25.917 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:29:25.917 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275200 2024-04-20T17:29:25.917 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.917 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:29:25.917 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.917 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:29:25.917 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:10.998559+0000", 2024-04-20T17:29:25.917 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.917 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.917 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.917 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:29:25.918 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:10.998559+0000", 2024-04-20T17:29:25.918 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.918 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.918 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.918 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:29:25.918 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:10.998560+0000", 2024-04-20T17:29:25.918 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9120000000000001e-06 2024-04-20T17:29:25.918 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.918 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.918 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:29:25.918 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:10.998566+0000", 2024-04-20T17:29:25.918 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6099999999999997e-06 2024-04-20T17:29:25.918 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.918 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.919 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:29:25.919 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:10.998578+0000", 2024-04-20T17:29:25.919 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1810999999999999e-05 2024-04-20T17:29:25.919 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.919 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.919 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:29:25.919 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:10.998586+0000", 2024-04-20T17:29:25.919 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.174e-06 2024-04-20T17:29:25.919 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.919 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.919 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:29:25.919 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:10.998644+0000", 2024-04-20T17:29:25.919 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7729000000000002e-05 2024-04-20T17:29:25.919 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.919 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.919 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:29:25.920 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:10.998762+0000", 2024-04-20T17:29:25.920 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000118021 2024-04-20T17:29:25.920 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.920 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.920 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:29:25.920 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:10.998873+0000", 2024-04-20T17:29:25.920 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011142 2024-04-20T17:29:25.920 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.920 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.920 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:29:25.920 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.003756+0000", 2024-04-20T17:29:25.920 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0048830059999999996 2024-04-20T17:29:25.921 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.921 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.921 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:29:25.921 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.215029+0000", 2024-04-20T17:29:25.921 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.21127307100000001 2024-04-20T17:29:25.921 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.921 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.921 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:29:25.921 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.215064+0000", 2024-04-20T17:29:25.921 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4406999999999998e-05 2024-04-20T17:29:25.921 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.921 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.921 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:29:25.921 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.215098+0000", 2024-04-20T17:29:25.921 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4069000000000002e-05 2024-04-20T17:29:25.922 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.922 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:29:25.922 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.922 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.922 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.922 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275211 4.6 4:6b91bd7d:::benchmark_data_smithi171_122772_object105019:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:29:25.922 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:11.008900+0000", 2024-04-20T17:29:25.922 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 74.853820475999996, 2024-04-20T17:29:25.922 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20628696999999999, 2024-04-20T17:29:25.922 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:29:25.922 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:29:25.922 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:29:25.922 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:29:25.923 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:29:25.923 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:29:25.923 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275211 2024-04-20T17:29:25.923 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.923 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:29:25.923 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.923 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:29:25.923 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.008900+0000", 2024-04-20T17:29:25.923 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.923 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.923 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.923 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:29:25.923 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.008900+0000", 2024-04-20T17:29:25.923 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.923 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.923 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.924 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:29:25.924 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.008902+0000", 2024-04-20T17:29:25.924 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1569999999999998e-06 2024-04-20T17:29:25.924 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.924 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.924 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:29:25.924 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.008909+0000", 2024-04-20T17:29:25.924 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5810000000000002e-06 2024-04-20T17:29:25.924 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.924 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.924 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:29:25.924 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.008922+0000", 2024-04-20T17:29:25.924 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3468999999999999e-05 2024-04-20T17:29:25.924 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.925 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.925 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:29:25.925 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.008929+0000", 2024-04-20T17:29:25.925 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2919999999999998e-06 2024-04-20T17:29:25.925 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.925 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.925 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:29:25.925 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.008986+0000", 2024-04-20T17:29:25.925 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6913999999999998e-05 2024-04-20T17:29:25.925 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.925 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.925 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:29:25.926 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.009110+0000", 2024-04-20T17:29:25.926 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012400599999999999 2024-04-20T17:29:25.926 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.926 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.926 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:29:25.926 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.009222+0000", 2024-04-20T17:29:25.926 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000111906 2024-04-20T17:29:25.926 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.926 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.926 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:29:25.926 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.014311+0000", 2024-04-20T17:29:25.926 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0050890980000000002 2024-04-20T17:29:25.926 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.926 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.926 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:29:25.927 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.215130+0000", 2024-04-20T17:29:25.927 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20081843799999999 2024-04-20T17:29:25.927 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.927 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.927 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:29:25.927 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.215160+0000", 2024-04-20T17:29:25.927 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0673000000000003e-05 2024-04-20T17:29:25.927 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.927 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.927 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:29:25.927 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.215187+0000", 2024-04-20T17:29:25.927 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6435999999999999e-05 2024-04-20T17:29:25.927 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.927 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:29:25.928 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.928 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.928 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.928 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275217 4.6 4:6f52af41:::benchmark_data_smithi171_122772_object105025:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:29:25.928 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:11.014905+0000", 2024-04-20T17:29:25.928 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 74.847815518000004, 2024-04-20T17:29:25.928 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20678690999999999, 2024-04-20T17:29:25.928 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:29:25.928 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:29:25.928 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:29:25.928 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:29:25.928 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:29:25.928 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:29:25.928 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275217 2024-04-20T17:29:25.929 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.929 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:29:25.929 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.929 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:29:25.929 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.014905+0000", 2024-04-20T17:29:25.929 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.929 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.929 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.929 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:29:25.929 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.014905+0000", 2024-04-20T17:29:25.929 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.929 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.929 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.929 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:29:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.014907+0000", 2024-04-20T17:29:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.967e-06 2024-04-20T17:29:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:29:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.014912+0000", 2024-04-20T17:29:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3759999999999999e-06 2024-04-20T17:29:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:29:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.014925+0000", 2024-04-20T17:29:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3174e-05 2024-04-20T17:29:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:29:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.014932+0000", 2024-04-20T17:29:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8229999999999997e-06 2024-04-20T17:29:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:29:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.014985+0000", 2024-04-20T17:29:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3062000000000001e-05 2024-04-20T17:29:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:29:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.015090+0000", 2024-04-20T17:29:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000104502 2024-04-20T17:29:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:29:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.015196+0000", 2024-04-20T17:29:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000106044 2024-04-20T17:29:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:29:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.024773+0000", 2024-04-20T17:29:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0095773449999999993 2024-04-20T17:29:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:29:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.221616+0000", 2024-04-20T17:29:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.196842454 2024-04-20T17:29:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:29:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.221649+0000", 2024-04-20T17:29:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3352999999999997e-05 2024-04-20T17:29:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:29:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.221692+0000", 2024-04-20T17:29:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2809999999999998e-05 2024-04-20T17:29:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:29:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275240 4.6 4:6dacae24:::benchmark_data_smithi171_122772_object105048:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:29:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:11.031577+0000", 2024-04-20T17:29:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 74.831143144999999, 2024-04-20T17:29:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19679422799999999, 2024-04-20T17:29:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:29:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:29:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:29:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:29:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:29:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:29:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275240 2024-04-20T17:29:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:29:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:29:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.031577+0000", 2024-04-20T17:29:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.935 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.935 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:29:25.935 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.031577+0000", 2024-04-20T17:29:25.935 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.935 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.935 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.935 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:29:25.935 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.031579+0000", 2024-04-20T17:29:25.935 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4610000000000001e-06 2024-04-20T17:29:25.935 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.935 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.935 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:29:25.935 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.031583+0000", 2024-04-20T17:29:25.935 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6870000000000002e-06 2024-04-20T17:29:25.935 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:29:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.031594+0000", 2024-04-20T17:29:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0594000000000001e-05 2024-04-20T17:29:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:29:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.031601+0000", 2024-04-20T17:29:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9240000000000001e-06 2024-04-20T17:29:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:29:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.031653+0000", 2024-04-20T17:29:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2228999999999997e-05 2024-04-20T17:29:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.937 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.937 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:29:25.937 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.031761+0000", 2024-04-20T17:29:25.937 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000108008 2024-04-20T17:29:25.937 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.937 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.937 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:29:25.937 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.031871+0000", 2024-04-20T17:29:25.937 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000110331 2024-04-20T17:29:25.937 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.937 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.937 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:29:25.937 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.036276+0000", 2024-04-20T17:29:25.938 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.004404799 2024-04-20T17:29:25.938 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.938 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.938 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:29:25.938 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.228303+0000", 2024-04-20T17:29:25.938 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19202655299999999 2024-04-20T17:29:25.938 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.938 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.938 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:29:25.938 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.228336+0000", 2024-04-20T17:29:25.938 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3281000000000002e-05 2024-04-20T17:29:25.938 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:29:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.228371+0000", 2024-04-20T17:29:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5361000000000002e-05 2024-04-20T17:29:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:29:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275248 4.6 4:65347144:::benchmark_data_smithi171_122772_object105056:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:29:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:11.041968+0000", 2024-04-20T17:29:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 74.820752830999993, 2024-04-20T17:29:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.18653969300000001, 2024-04-20T17:29:25.940 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:29:25.940 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:29:25.940 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:29:25.940 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:29:25.940 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:29:25.940 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:29:25.940 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275248 2024-04-20T17:29:25.940 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.940 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:29:25.940 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.940 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:29:25.940 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.041968+0000", 2024-04-20T17:29:25.940 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.940 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:29:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.041968+0000", 2024-04-20T17:29:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:29:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.041970+0000", 2024-04-20T17:29:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1229999999999998e-06 2024-04-20T17:29:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:29:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.041976+0000", 2024-04-20T17:29:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.852e-06 2024-04-20T17:29:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.942 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:29:25.942 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.041987+0000", 2024-04-20T17:29:25.942 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1355000000000001e-05 2024-04-20T17:29:25.942 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.942 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.942 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:29:25.942 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.041995+0000", 2024-04-20T17:29:25.942 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4540000000000007e-06 2024-04-20T17:29:25.942 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.942 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.942 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:29:25.943 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.042050+0000", 2024-04-20T17:29:25.943 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4239e-05 2024-04-20T17:29:25.943 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.943 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.943 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:29:25.943 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.042138+0000", 2024-04-20T17:29:25.943 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8004999999999995e-05 2024-04-20T17:29:25.943 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.943 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.943 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:29:25.943 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.042261+0000", 2024-04-20T17:29:25.943 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012317299999999999 2024-04-20T17:29:25.944 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.944 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.944 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:29:25.944 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.046505+0000", 2024-04-20T17:29:25.944 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0042438600000000003 2024-04-20T17:29:25.944 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.944 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.944 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:29:25.944 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.228413+0000", 2024-04-20T17:29:25.944 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.18190864500000001 2024-04-20T17:29:25.944 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.944 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.945 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:29:25.945 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.228465+0000", 2024-04-20T17:29:25.945 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2074000000000003e-05 2024-04-20T17:29:25.945 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.945 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.945 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:29:25.945 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.228507+0000", 2024-04-20T17:29:25.945 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1912999999999997e-05 2024-04-20T17:29:25.945 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.945 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:29:25.945 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.946 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.946 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.946 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275263 4.6 4:6160fa82:::benchmark_data_smithi171_122772_object105071:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:29:25.946 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:11.051143+0000", 2024-04-20T17:29:25.946 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 74.811576912999996, 2024-04-20T17:29:25.946 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.18078492099999999, 2024-04-20T17:29:25.946 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:29:25.946 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:29:25.946 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:29:25.946 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:29:25.946 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:29:25.946 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:29:25.946 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275263 2024-04-20T17:29:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:29:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:29:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.051143+0000", 2024-04-20T17:29:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:29:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.051143+0000", 2024-04-20T17:29:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:29:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.051146+0000", 2024-04-20T17:29:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1730000000000002e-06 2024-04-20T17:29:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:29:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.051152+0000", 2024-04-20T17:29:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8579999999999996e-06 2024-04-20T17:29:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:29:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.051165+0000", 2024-04-20T17:29:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3661e-05 2024-04-20T17:29:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:29:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.051173+0000", 2024-04-20T17:29:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8329999999999994e-06 2024-04-20T17:29:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:29:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.051233+0000", 2024-04-20T17:29:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9837999999999999e-05 2024-04-20T17:29:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:29:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.051354+0000", 2024-04-20T17:29:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012070699999999999 2024-04-20T17:29:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:29:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.051468+0000", 2024-04-20T17:29:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000114658 2024-04-20T17:29:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:29:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.056305+0000", 2024-04-20T17:29:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0048365329999999996 2024-04-20T17:29:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:29:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.231849+0000", 2024-04-20T17:29:25.951 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.175544645 2024-04-20T17:29:25.951 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.951 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.951 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:29:25.951 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.231886+0000", 2024-04-20T17:29:25.951 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6322999999999998e-05 2024-04-20T17:29:25.951 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.951 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.951 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:29:25.951 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.231928+0000", 2024-04-20T17:29:25.951 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2691999999999997e-05 2024-04-20T17:29:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:29:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275278 4.6 4:6f1c4c64:::benchmark_data_smithi171_122772_object105086:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:29:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:11.064797+0000", 2024-04-20T17:29:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 74.797923728000001, 2024-04-20T17:29:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.167260615, 2024-04-20T17:29:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:29:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:29:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:29:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:29:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:29:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:29:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275278 2024-04-20T17:29:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:29:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:29:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.064797+0000", 2024-04-20T17:29:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:29:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.064797+0000", 2024-04-20T17:29:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:29:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.064798+0000", 2024-04-20T17:29:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.567e-06 2024-04-20T17:29:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:29:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.064803+0000", 2024-04-20T17:29:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0429999999999998e-06 2024-04-20T17:29:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:29:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.064815+0000", 2024-04-20T17:29:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.148e-05 2024-04-20T17:29:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:29:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.064822+0000", 2024-04-20T17:29:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2359999999999998e-06 2024-04-20T17:29:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:29:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.064880+0000", 2024-04-20T17:29:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7698999999999998e-05 2024-04-20T17:29:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.956 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.956 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:29:25.956 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.064999+0000", 2024-04-20T17:29:25.956 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011958 2024-04-20T17:29:25.956 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.956 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.956 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:29:25.956 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.065120+0000", 2024-04-20T17:29:25.956 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000120512 2024-04-20T17:29:25.956 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.956 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.956 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:29:25.956 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.069852+0000", 2024-04-20T17:29:25.956 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0047317180000000002 2024-04-20T17:29:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:29:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.231968+0000", 2024-04-20T17:29:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.162116383 2024-04-20T17:29:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:29:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.232017+0000", 2024-04-20T17:29:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9614999999999998e-05 2024-04-20T17:29:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:29:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.232057+0000", 2024-04-20T17:29:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9782e-05 2024-04-20T17:29:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:29:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275285 4.6 4:63a3635d:::benchmark_data_smithi171_122772_object105093:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:29:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:11.074387+0000", 2024-04-20T17:29:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 74.788333417999993, 2024-04-20T17:29:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16127122899999999, 2024-04-20T17:29:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:29:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:29:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:29:25.959 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:29:25.959 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:29:25.959 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:29:25.959 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275285 2024-04-20T17:29:25.959 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.959 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:29:25.959 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.959 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:29:25.959 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.074387+0000", 2024-04-20T17:29:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:29:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.074387+0000", 2024-04-20T17:29:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:29:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.074388+0000", 2024-04-20T17:29:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.339e-06 2024-04-20T17:29:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:29:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.074393+0000", 2024-04-20T17:29:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4259999999999996e-06 2024-04-20T17:29:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:29:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.074402+0000", 2024-04-20T17:29:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6609999999999995e-06 2024-04-20T17:29:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:29:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.074408+0000", 2024-04-20T17:29:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6520000000000003e-06 2024-04-20T17:29:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:29:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.074455+0000", 2024-04-20T17:29:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6717e-05 2024-04-20T17:29:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:29:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.074530+0000", 2024-04-20T17:29:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5661000000000006e-05 2024-04-20T17:29:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:29:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.074597+0000", 2024-04-20T17:29:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6418e-05 2024-04-20T17:29:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:29:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.078692+0000", 2024-04-20T17:29:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0040947350000000004 2024-04-20T17:29:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:29:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.235564+0000", 2024-04-20T17:29:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15687225099999999 2024-04-20T17:29:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:29:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.235616+0000", 2024-04-20T17:29:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2599999999999998e-05 2024-04-20T17:29:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:29:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.235658+0000", 2024-04-20T17:29:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1768999999999999e-05 2024-04-20T17:29:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:29:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275296 4.6 4:6850fa57:::benchmark_data_smithi171_122772_object105104:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:29:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:11.083403+0000", 2024-04-20T17:29:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 74.779317460000001, 2024-04-20T17:29:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15593427100000001, 2024-04-20T17:29:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:29:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:29:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:29:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:29:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:29:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:29:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275296 2024-04-20T17:29:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:29:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:29:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.083403+0000", 2024-04-20T17:29:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.966 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:29:25.966 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.083403+0000", 2024-04-20T17:29:25.966 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.966 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.966 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.966 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:29:25.966 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.083406+0000", 2024-04-20T17:29:25.966 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8370000000000001e-06 2024-04-20T17:29:25.966 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:29:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.083414+0000", 2024-04-20T17:29:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2630000000000003e-06 2024-04-20T17:29:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:29:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.083431+0000", 2024-04-20T17:29:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6560000000000001e-05 2024-04-20T17:29:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:29:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.083441+0000", 2024-04-20T17:29:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0132000000000001e-05 2024-04-20T17:29:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:29:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.083503+0000", 2024-04-20T17:29:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2119000000000006e-05 2024-04-20T17:29:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:29:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.083625+0000", 2024-04-20T17:29:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000122547 2024-04-20T17:29:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:29:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.083735+0000", 2024-04-20T17:29:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000109764 2024-04-20T17:29:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:29:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.088797+0000", 2024-04-20T17:29:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0050613510000000004 2024-04-20T17:29:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:29:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.239279+0000", 2024-04-20T17:29:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15048260599999999 2024-04-20T17:29:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:29:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.239310+0000", 2024-04-20T17:29:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1300999999999997e-05 2024-04-20T17:29:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:29:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.239337+0000", 2024-04-20T17:29:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6791000000000001e-05 2024-04-20T17:29:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:29:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275312 4.6 4:6a829b24:::benchmark_data_smithi171_122772_object105120:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:29:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:11.101309+0000", 2024-04-20T17:29:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 74.761411666000001, 2024-04-20T17:29:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.14268697999999999, 2024-04-20T17:29:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:29:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:29:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:29:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:29:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:29:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:29:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275312 2024-04-20T17:29:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:29:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:29:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.101309+0000", 2024-04-20T17:29:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.972 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.972 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:29:25.972 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.101309+0000", 2024-04-20T17:29:25.972 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.972 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.972 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.972 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:29:25.972 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.101312+0000", 2024-04-20T17:29:25.972 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3110000000000001e-06 2024-04-20T17:29:25.972 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.972 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.972 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:29:25.972 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.101320+0000", 2024-04-20T17:29:25.973 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7659999999999998e-06 2024-04-20T17:29:25.973 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.973 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.973 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:29:25.973 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.101337+0000", 2024-04-20T17:29:25.973 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7345000000000001e-05 2024-04-20T17:29:25.973 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.973 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.973 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:29:25.973 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.101347+0000", 2024-04-20T17:29:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.594e-06 2024-04-20T17:29:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:29:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.101415+0000", 2024-04-20T17:29:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8751000000000006e-05 2024-04-20T17:29:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:29:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.101540+0000", 2024-04-20T17:29:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000124029 2024-04-20T17:29:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:29:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.101658+0000", 2024-04-20T17:29:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000118235 2024-04-20T17:29:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:29:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.106325+0000", 2024-04-20T17:29:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0046667490000000004 2024-04-20T17:29:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:29:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.243904+0000", 2024-04-20T17:29:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.137579902 2024-04-20T17:29:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:29:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.243956+0000", 2024-04-20T17:29:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1690999999999997e-05 2024-04-20T17:29:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:29:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.243996+0000", 2024-04-20T17:29:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9607000000000002e-05 2024-04-20T17:29:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:29:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:281743 4.2 4:46c29c23:::benchmark_data_smithi171_122772_object111551:head [delete] snapc 0=[] ondisk+write+known_if_redirected e521)", 2024-04-20T17:29:25.977 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:17.725385+0000", 2024-04-20T17:29:25.977 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 68.137335340000007, 2024-04-20T17:29:25.977 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.13120236499999999, 2024-04-20T17:29:25.977 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:29:25.977 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:29:25.977 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:29:25.977 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:29:25.977 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:29:25.977 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:29:25.977 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 281743 2024-04-20T17:29:25.977 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.977 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:29:25.977 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:29:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.725385+0000", 2024-04-20T17:29:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:29:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.725385+0000", 2024-04-20T17:29:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:29:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:29:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.725386+0000", 2024-04-20T17:29:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.268e-06 2024-04-20T17:29:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:29:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.725391+0000", 2024-04-20T17:29:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4560000000000002e-06 2024-04-20T17:29:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:29:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.725400+0000", 2024-04-20T17:29:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0459999999999994e-06 2024-04-20T17:29:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:29:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.725406+0000", 2024-04-20T17:29:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8130000000000001e-06 2024-04-20T17:29:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:29:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.731194+0000", 2024-04-20T17:29:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0057884779999999997 2024-04-20T17:29:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:29:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.731302+0000", 2024-04-20T17:29:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000107431 2024-04-20T17:29:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:29:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.731445+0000", 2024-04-20T17:29:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014385 2024-04-20T17:29:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:29:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.736049+0000", 2024-04-20T17:29:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0046034229999999997 2024-04-20T17:29:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:29:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.856468+0000", 2024-04-20T17:29:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.12041884 2024-04-20T17:29:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:29:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.856523+0000", 2024-04-20T17:29:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5148999999999997e-05 2024-04-20T17:29:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:29:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:29:25.982 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:29:25.982 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.856587+0000", 2024-04-20T17:29:25.982 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4610999999999995e-05 2024-04-20T17:29:25.982 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.982 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:29:25.982 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.982 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:29:25.982 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:29:25.982 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:29:26.212 DEBUG:teuthology.orchestra.run.smithi005:> sudo pkill -f 'journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.3.service' 2024-04-20T17:29:26.245 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:29:26 smithi005 systemd[1]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.3.service: Succeeded. 2024-04-20T17:29:26.245 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:29:26 smithi005 systemd[1]: Stopped Ceph osd.3 for 13c6e76c-ff37-11ee-bc93-c7b262605968. 2024-04-20T17:29:26.255 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:29:26.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:26.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:29:26.263 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T17:29:26.264 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-20T17:29:26.264 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.3 2024-04-20T17:29:26.264 INFO:tasks.thrashosds.thrasher:remote for osd 3 is ubuntu@smithi005.front.sepia.ceph.com 2024-04-20T17:29:26.264 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.3 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op list-pgs 2024-04-20T17:29:26.360 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:29:26.393 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:26.415 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:29:26.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:26.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:29:26.519 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:29:26.525 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.3/config 2024-04-20T17:29:26.548 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:26.567 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-20T17:29:26.630 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:29:26.828 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:29:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:27 smithi171 ceph-mon[44189]: pgmap v1432: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:27.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:27 smithi106 ceph-mon[44423]: pgmap v1432: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:27.589 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:29:27.598 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:29:27.600 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:27 smithi005 ceph-mon[39562]: pgmap v1432: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:27.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:27.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:29:27.702 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:29:27.732 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:27.828 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:29:27.931 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:29:27.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:27.941 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:29:28.043 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:29:28.075 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:28.097 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:29:28.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:28.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:29:28.201 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:29:28.231 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:28.249 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'unique_pool_3' created 2024-04-20T17:29:28.252 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:29:28.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:28.255 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:29:28.356 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:29:28.383 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:28.481 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:29:28.483 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:28 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2999453754' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-04-20T17:29:28.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:28.484 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:29:28.493 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:29:28.493 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:29:28.494 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 0 2024-04-20T17:29:28.494 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:29:28.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:28 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2999453754' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-04-20T17:29:28.585 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:29:28.602 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:28 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2999453754' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-04-20T17:29:28.615 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:28.941 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-04-20T17:29:28.964 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:29:28.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:28.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:29:28.981 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:29:29.010 INFO:teuthology.orchestra.run.smithi005.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:29:29.069 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:29:29.102 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:29.123 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:29:29.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:29.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:29:29.167 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-20T17:29:29.227 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:29:29.259 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:29.279 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:29:29.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:29.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:29:29.383 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:29:29.409 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:29 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2999453754' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-04-20T17:29:29.409 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:29 smithi106 ceph-mon[44423]: osdmap e543: 12 total, 11 up, 11 in 2024-04-20T17:29:29.409 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:29 smithi106 ceph-mon[44423]: pgmap v1434: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:29.411 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:29.430 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:29:29.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:29.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:29:29.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:29 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2999453754' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-04-20T17:29:29.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:29 smithi005 ceph-mon[39562]: osdmap e543: 12 total, 11 up, 11 in 2024-04-20T17:29:29.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:29 smithi005 ceph-mon[39562]: pgmap v1434: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:29.475 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:29:29.534 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:29:29.558 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:29 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2999453754' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-04-20T17:29:29.558 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:29 smithi171 ceph-mon[44189]: osdmap e543: 12 total, 11 up, 11 in 2024-04-20T17:29:29.559 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:29 smithi171 ceph-mon[44189]: pgmap v1434: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:29.560 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:29.607 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:29:29.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:29.610 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:29:29.711 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:29:29.737 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:30.285 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:29:30.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:30.289 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:29:30.390 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:29:30.423 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:30.445 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:29:30.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:30.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:29:30.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:30 smithi106 ceph-mon[44423]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-20T17:29:30.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:30 smithi106 ceph-mon[44423]: osdmap e544: 12 total, 11 up, 11 in 2024-04-20T17:29:30.522 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:30 smithi171 ceph-mon[44189]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-20T17:29:30.522 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:30 smithi171 ceph-mon[44189]: osdmap e544: 12 total, 11 up, 11 in 2024-04-20T17:29:30.549 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:29:30.579 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:30.600 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:29:30.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:30.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:29:30.704 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:29:30.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:30 smithi005 ceph-mon[39562]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-20T17:29:30.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:30 smithi005 ceph-mon[39562]: osdmap e544: 12 total, 11 up, 11 in 2024-04-20T17:29:30.733 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:30.752 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:29:30.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:30.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:29:30.856 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:29:30.890 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:30.930 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:29:30.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:30.937 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:29:31.038 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:29:31.070 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:31.145 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:29:31.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:31.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:29:31.250 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:29:31.280 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:31.301 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:29:31.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:31.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:29:31.405 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:29:31.434 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:31.454 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:29:31.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:31.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:29:31.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:31 smithi005 ceph-mon[39562]: osdmap e545: 12 total, 11 up, 11 in 2024-04-20T17:29:31.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:31 smithi005 ceph-mon[39562]: pgmap v1437: 17 pgs: 7 creating+peering, 9 unknown, 1 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:31.478 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:31 smithi171 ceph-mon[44189]: osdmap e545: 12 total, 11 up, 11 in 2024-04-20T17:29:31.479 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:31 smithi171 ceph-mon[44189]: pgmap v1437: 17 pgs: 7 creating+peering, 9 unknown, 1 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:31.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:31 smithi106 ceph-mon[44423]: osdmap e545: 12 total, 11 up, 11 in 2024-04-20T17:29:31.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:31 smithi106 ceph-mon[44423]: pgmap v1437: 17 pgs: 7 creating+peering, 9 unknown, 1 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:31.557 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:29:31.585 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:31.604 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:29:31.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:31.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:29:31.668 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:29:31.709 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:29:31.740 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:31.761 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:29:31.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:31.765 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:29:31.765 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:29:31.766 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:29:31.766 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-04-20T17:29:31.766 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-04-20T17:29:31.766 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:29:31.792 INFO:teuthology.orchestra.run.smithi005.stdout:4.0 2024-04-20T17:29:31.793 INFO:teuthology.orchestra.run.smithi005.stdout:4.f 2024-04-20T17:29:31.793 INFO:teuthology.orchestra.run.smithi005.stdout:4.c 2024-04-20T17:29:31.865 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:29:31.893 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:31.931 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:29:31.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:31.933 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:29:32.034 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:29:32.061 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:32.370 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-20T17:29:32.409 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:29:32.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:32.411 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:29:32.512 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:29:32.538 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:32.557 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:29:32.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:32.560 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:29:32.661 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:29:32.676 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:29:32.690 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:32.888 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:29:32.907 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:29:32.927 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:29:32.939 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:29:32.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:32.943 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:29:32.948 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:29:32.968 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:29:32.989 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:29:33.009 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:29:33.029 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:29:33.044 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:29:33.050 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:29:33.070 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:29:33.077 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:33.090 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:29:33.091 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:29:33.091 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:29:33.091 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:29:33.091 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-20T17:29:33.091 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:29:33.091 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:29:33.091 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:29:33.091 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:29:33.091 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:29:33.091 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:29:33.091 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:29:33.092 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:29:33.099 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:29:33.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:33.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:29:33.203 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:29:33.236 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:33.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:33 smithi005 ceph-mon[39562]: pgmap v1438: 17 pgs: 7 creating+peering, 1 unknown, 1 undersized+peered, 8 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:33.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:33 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1106401451' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:29:33.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:33.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:33 smithi171 ceph-mon[44189]: pgmap v1438: 17 pgs: 7 creating+peering, 1 unknown, 1 undersized+peered, 8 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:33.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:33 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1106401451' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:29:33.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:33.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:33 smithi106 ceph-mon[44423]: pgmap v1438: 17 pgs: 7 creating+peering, 1 unknown, 1 undersized+peered, 8 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:33.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:33 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1106401451' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:29:33.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:33.812 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.3 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op export --pgid 4.f --file /var/log/ceph/exp.4.f.3 2024-04-20T17:29:34.106 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.3/config 2024-04-20T17:29:34.374 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.375 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.375 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.375 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.375 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.375 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.375 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.375 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.375 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.375 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.389 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:29:34.389 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:29:34.412 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.412 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.412 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.412 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.412 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.413 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.413 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.413 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.413 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.413 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:34.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:35.003 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-20T17:29:35.292 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:29:35.510 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:35 smithi005 ceph-mon[39562]: pgmap v1439: 17 pgs: 1 undersized+peered, 16 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:35.510 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:35 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:29:35.551 INFO:teuthology.orchestra.run.smithi005.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:29:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:35 smithi171 ceph-mon[44189]: pgmap v1439: 17 pgs: 1 undersized+peered, 16 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:35.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:35 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:29:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:35 smithi106 ceph-mon[44423]: pgmap v1439: 17 pgs: 1 undersized+peered, 16 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:35 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:29:36.187 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:29:36.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:36.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:29:36.292 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:29:36.325 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:36.350 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:29:36.405 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:29:36.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:36.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:29:36.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:36 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1577070332' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:29:36.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:36 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:29:36.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:36 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:29:36.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:36 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:29:36.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:36 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:29:36.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:36 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:29:36.524 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:29:36.553 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:36.606 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:29:36.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:36.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:29:36.623 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:36 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1577070332' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:29:36.624 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:36 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:29:36.624 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:36 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:29:36.624 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:36 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:29:36.624 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:36 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:29:36.624 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:36 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:29:36.711 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:29:36.739 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:36.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:36 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1577070332' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:29:36.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:36 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:29:36.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:36 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:29:36.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:36 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:29:36.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:36 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:29:36.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:36 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:29:36.937 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:29:36.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:36.941 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:29:37.042 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:29:37.072 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:37.170 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:29:37.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:37.173 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:29:37.190 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:29:37.191 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:29:37.274 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:29:37.303 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:37.347 INFO:teuthology.orchestra.run.smithi005.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-04-20T17:29:37.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:37 smithi005 ceph-mon[39562]: pgmap v1440: 17 pgs: 1 undersized+peered, 16 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:37.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:37 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:29:37.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:37 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1577070332' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:29:37.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:37 smithi005 ceph-mon[39562]: osdmap e546: 12 total, 11 up, 11 in 2024-04-20T17:29:37.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:37 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1486988523' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:29:37.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:37 smithi005 ceph-mon[39562]: from='client.26446 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:29:37.510 INFO:teuthology.orchestra.run.smithi005.stderr:Exporting 4.f info 4.f( v 538'21378 (487'11300,538'21378] local-lis/les=445/446 n=0 ec=445/445 lis/c=445/445 les/c/f=446/446/0 sis=445) 2024-04-20T17:29:37.517 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:37 smithi171 ceph-mon[44189]: pgmap v1440: 17 pgs: 1 undersized+peered, 16 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:37.517 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:37 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:29:37.517 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:37 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1577070332' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:29:37.517 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:37 smithi171 ceph-mon[44189]: osdmap e546: 12 total, 11 up, 11 in 2024-04-20T17:29:37.517 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:37 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1486988523' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:29:37.517 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:37 smithi171 ceph-mon[44189]: from='client.26446 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:29:37.570 INFO:teuthology.orchestra.run.smithi005.stderr:Export successful 2024-04-20T17:29:37.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:37 smithi106 ceph-mon[44423]: pgmap v1440: 17 pgs: 1 undersized+peered, 16 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:37.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:37 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:29:37.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:37 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1577070332' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:29:37.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:37 smithi106 ceph-mon[44423]: osdmap e546: 12 total, 11 up, 11 in 2024-04-20T17:29:37.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:37 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1486988523' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:29:37.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:37 smithi106 ceph-mon[44423]: from='client.26446 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:29:37.841 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-20T17:29:38.144 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:29:38.253 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:29:38.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:38.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:29:38.357 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:29:38.385 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:38.387 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:29:38.387 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:29:38.407 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:29:38.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:38.409 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:29:38.510 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:29:38.537 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:38.609 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:29:38.610 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:38 smithi171 ceph-mon[44189]: from='client.26446 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-20T17:29:38.610 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:38 smithi171 ceph-mon[44189]: osdmap e547: 12 total, 11 up, 11 in 2024-04-20T17:29:38.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:38.612 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:29:38.685 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:38 smithi005 ceph-mon[39562]: from='client.26446 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-20T17:29:38.685 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:38 smithi005 ceph-mon[39562]: osdmap e547: 12 total, 11 up, 11 in 2024-04-20T17:29:38.713 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:29:38.746 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:38.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:38 smithi106 ceph-mon[44423]: from='client.26446 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-20T17:29:38.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:38 smithi106 ceph-mon[44423]: osdmap e547: 12 total, 11 up, 11 in 2024-04-20T17:29:39.097 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:29:39.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:39.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:29:39.168 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:29:39.202 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:29:39.241 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:39.605 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:39 smithi171 ceph-mon[44189]: pgmap v1443: 17 pgs: 1 undersized+peered, 16 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:39.606 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:39 smithi171 ceph-mon[44189]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-20T17:29:39.677 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:39 smithi005 ceph-mon[39562]: pgmap v1443: 17 pgs: 1 undersized+peered, 16 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:39.678 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:39 smithi005 ceph-mon[39562]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-20T17:29:39.696 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:29:39.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:39.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:29:39.705 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.3 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --force --op remove --pgid 4.f 2024-04-20T17:29:39.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:39 smithi106 ceph-mon[44423]: pgmap v1443: 17 pgs: 1 undersized+peered, 16 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:39.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:39 smithi106 ceph-mon[44423]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-20T17:29:39.791 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:29:39.800 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:29:39.833 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:39.842 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:29:39.842 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:29:39.853 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:29:39.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:39.856 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:29:39.956 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:29:39.984 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:40.067 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.3/config 2024-04-20T17:29:40.082 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:29:40.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:40.085 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:29:40.115 DEBUG:teuthology.orchestra.run.smithi171:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-04-20T17:29:40.127 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-20T17:29:40.128 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-20T17:29:40.186 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:29:40.212 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:40.220 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:hints = 1 2024-04-20T17:29:40.220 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-20T17:29:40.220 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Object prefix: benchmark_data_smithi171_137281 2024-04-20T17:29:40.220 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:29:40.220 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 0 0 0 0 0 0 - 0 2024-04-20T17:29:40.388 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:29:40.512 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-20T17:29:40.574 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:29:40.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:40.578 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:29:40.679 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:29:40.712 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:40.734 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:29:40.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:40.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:29:40.811 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:29:40.838 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:29:40.870 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:40.892 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:29:40.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:40.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:29:40.996 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:29:41.022 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:41.044 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:29:41.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:41.047 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:29:41.148 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:29:41.179 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:41.220 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 1 16 203 187 11.6865 11.6875 0.00120179 0.0022051 2024-04-20T17:29:41.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:41 smithi005 ceph-mon[39562]: pgmap v1444: 17 pgs: 1 undersized+peered, 16 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:41.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:41 smithi171 ceph-mon[44189]: pgmap v1444: 17 pgs: 1 undersized+peered, 16 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:41 smithi106 ceph-mon[44423]: pgmap v1444: 17 pgs: 1 undersized+peered, 16 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:41.999 INFO:teuthology.orchestra.run.smithi005.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:29:42.028 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:29:42.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:42.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:29:42.132 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:29:42.162 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:42.221 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 2 16 203 187 5.84318 0 - 0.0022051 2024-04-20T17:29:42.259 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:29:42.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:42.262 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:29:42.363 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:29:42.392 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:42.515 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:29:42.516 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:29:42.517 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:29:42.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:42.518 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:29:42.619 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:29:42.646 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:42.743 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:29:42.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:42.746 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:29:42.847 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:29:42.878 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:42.911 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:29:42.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:42.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:29:42.953 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:29:43.016 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:29:43.045 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:43.203 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-20T17:29:43.221 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 3 16 203 187 3.89538 0 - 0.0022051 2024-04-20T17:29:43.371 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:29:43.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:43 smithi005 ceph-mon[39562]: pgmap v1445: 17 pgs: 1 undersized+peered, 16 active+clean; 6.1 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 704 KiB/s wr, 10 op/s 2024-04-20T17:29:43.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:43 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1030495913' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:29:43.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:43 smithi005 ceph-mon[39562]: from='client.26461 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:29:43.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:43 smithi171 ceph-mon[44189]: pgmap v1445: 17 pgs: 1 undersized+peered, 16 active+clean; 6.1 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 704 KiB/s wr, 10 op/s 2024-04-20T17:29:43.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:43 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1030495913' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:29:43.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:43 smithi171 ceph-mon[44189]: from='client.26461 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:29:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:43 smithi106 ceph-mon[44423]: pgmap v1445: 17 pgs: 1 undersized+peered, 16 active+clean; 6.1 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 704 KiB/s wr, 10 op/s 2024-04-20T17:29:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:43 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1030495913' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:29:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:43 smithi106 ceph-mon[44423]: from='client.26461 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:29:44.221 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 4 16 203 187 2.92151 0 - 0.0022051 2024-04-20T17:29:44.295 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:29:44.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:44.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:29:44.405 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:29:44.440 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:44.461 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:29:44.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:44.464 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:29:44.565 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:29:44.593 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:44 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:29:44.593 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:44 smithi005 ceph-mon[39562]: from='client.26461 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:29:44.593 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:44 smithi005 ceph-mon[39562]: osdmap e548: 12 total, 11 up, 11 in 2024-04-20T17:29:44.595 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:44.694 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:29:44.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:44.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:29:44.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:44 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:29:44.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:44 smithi106 ceph-mon[44423]: from='client.26461 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:29:44.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:44 smithi106 ceph-mon[44423]: osdmap e548: 12 total, 11 up, 11 in 2024-04-20T17:29:44.798 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:29:44.827 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:44 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:29:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:44 smithi171 ceph-mon[44189]: from='client.26461 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:29:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:44 smithi171 ceph-mon[44189]: osdmap e548: 12 total, 11 up, 11 in 2024-04-20T17:29:44.980 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:29:44.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:44.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:29:45.052 INFO:teuthology.orchestra.run.smithi005.stdout: marking collection for removal 2024-04-20T17:29:45.052 INFO:teuthology.orchestra.run.smithi005.stdout:setting '_remove' omap key 2024-04-20T17:29:45.061 INFO:teuthology.orchestra.run.smithi005.stdout:finish_remove_pgs 4.f_head removing 4.f 2024-04-20T17:29:45.077 INFO:teuthology.orchestra.run.smithi005.stdout:Remove successful 2024-04-20T17:29:45.084 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:29:45.114 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:45.221 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 5 16 203 187 2.3372 0 - 0.0022051 2024-04-20T17:29:45.464 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:29:45.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:45.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:29:45.567 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:29:45.600 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:45.622 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:29:45.624 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:45 smithi171 ceph-mon[44189]: pgmap v1447: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 1.5 MiB/s wr, 23 op/s 2024-04-20T17:29:45.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:45.625 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:29:45.666 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:29:45.686 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:29:45.705 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:29:45.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:45 smithi005 ceph-mon[39562]: pgmap v1447: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 1.5 MiB/s wr, 23 op/s 2024-04-20T17:29:45.726 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:29:45.727 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:29:45.746 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:29:45.757 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:45 smithi106 ceph-mon[44423]: pgmap v1447: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 1.5 MiB/s wr, 23 op/s 2024-04-20T17:29:45.766 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:29:45.778 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:29:45.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:45.781 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:29:45.787 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:29:45.807 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:29:45.828 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:29:45.848 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:29:45.868 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:29:45.868 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:29:45.869 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:29:45.869 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:29:45.869 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-20T17:29:45.869 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:29:45.869 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:29:45.869 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:29:45.869 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:29:45.870 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:29:45.870 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:29:45.870 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:29:45.870 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:29:45.882 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:29:45.915 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:45.937 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:29:45.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:45.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:29:46.041 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:29:46.071 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:46.221 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 6 16 203 187 1.94766 0 - 0.0022051 2024-04-20T17:29:46.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:46 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2313393421' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:29:46.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:46.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:46.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:46.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:46.621 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:29:46.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:46.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:29:46.644 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:29:46.645 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:29:46.725 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:29:46.754 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:46.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:46 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2313393421' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:29:46.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:46.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:46.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:46.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:46 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2313393421' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:29:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:46.973 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:29:47.002 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:29:47.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:47.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:29:47.007 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.3 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.f.3 2024-04-20T17:29:47.107 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:29:47.136 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:47.221 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 7 16 203 187 1.66942 0 - 0.0022051 2024-04-20T17:29:47.235 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:29:47.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:47.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:29:47.273 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.3/config 2024-04-20T17:29:47.340 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:29:47.372 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:47.394 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:29:47.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:47.397 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:29:47.425 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:29:47.498 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:29:47.529 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:47.529 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:47.529 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:47.529 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:47.529 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:47.529 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:47.529 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:47.529 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:47 smithi005 ceph-mon[39562]: pgmap v1448: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 1.3 MiB/s wr, 20 op/s 2024-04-20T17:29:47.531 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:47.554 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:29:47.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:47.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:29:47.658 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:29:47.687 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:47.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:47.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:47.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:47.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:47.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:47.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:47.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:47.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:47 smithi106 ceph-mon[44423]: pgmap v1448: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 1.3 MiB/s wr, 20 op/s 2024-04-20T17:29:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:47 smithi171 ceph-mon[44189]: pgmap v1448: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 1.3 MiB/s wr, 20 op/s 2024-04-20T17:29:48.041 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:29:48.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:48.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:29:48.146 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:29:48.179 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:48.200 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:29:48.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:48.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:29:48.221 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 8 16 203 187 1.46075 0 - 0.0022051 2024-04-20T17:29:48.304 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:29:48.335 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:48.534 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:29:48.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:48.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:29:48.639 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:29:48.646 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:29:48.672 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:48.693 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:29:48.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:48.696 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:29:48.797 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:29:48.829 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:48.849 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:29:48.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:48.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:29:48.953 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:29:48.983 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:49.081 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:29:49.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:49.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:29:49.185 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:29:49.217 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:49.222 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 9 16 203 187 1.29844 0 - 0.0022051 2024-04-20T17:29:49.239 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:29:49.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:49.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:29:49.344 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:29:49.374 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:49.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:49 smithi106 ceph-mon[44423]: pgmap v1449: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 1.2 MiB/s wr, 18 op/s 2024-04-20T17:29:49.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:49 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:29:49.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:49 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:29:49.672 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:49 smithi005 ceph-mon[39562]: pgmap v1449: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 1.2 MiB/s wr, 18 op/s 2024-04-20T17:29:49.672 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:49 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:29:49.672 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:49 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:29:49.717 INFO:teuthology.orchestra.run.smithi005.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:29:49.800 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:29:49.801 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:29:49.801 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-04-20T17:29:49.801 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:29:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:49 smithi171 ceph-mon[44189]: pgmap v1449: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 1.2 MiB/s wr, 18 op/s 2024-04-20T17:29:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:49 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:29:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:49 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:29:49.949 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:29:50.222 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 10 16 203 187 1.1686 0 - 0.0022051 2024-04-20T17:29:50.397 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:29:50.400 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-20T17:29:50.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:50 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:29:50.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:50 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:29:50.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:50 smithi005 ceph-mon[39562]: osdmap e549: 12 total, 11 up, 11 in 2024-04-20T17:29:50.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:50 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:29:50.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:50 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:29:50.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:50 smithi106 ceph-mon[44423]: osdmap e549: 12 total, 11 up, 11 in 2024-04-20T17:29:50.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:50 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:29:50.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:50 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:29:50.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:50 smithi171 ceph-mon[44189]: osdmap e549: 12 total, 11 up, 11 in 2024-04-20T17:29:51.055 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:29:51.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:51.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:29:51.160 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:29:51.193 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:51.213 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:29:51.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:51.217 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:29:51.222 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 11 16 203 187 1.06236 0 - 0.0022051 2024-04-20T17:29:51.318 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:29:51.348 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:51.369 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:29:51.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:51.372 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:29:51.473 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:29:51.500 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:51 smithi171 ceph-mon[44189]: pgmap v1451: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 792 KiB/s wr, 12 op/s 2024-04-20T17:29:51.502 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:51.522 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:29:51.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:51.525 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:29:51.626 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:29:51.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:51 smithi005 ceph-mon[39562]: pgmap v1451: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 792 KiB/s wr, 12 op/s 2024-04-20T17:29:51.658 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:51.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:51 smithi106 ceph-mon[44423]: pgmap v1451: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 792 KiB/s wr, 12 op/s 2024-04-20T17:29:52.108 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:29:52.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:52.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:29:52.212 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:29:52.222 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 12 16 203 187 0.973826 0 - 0.0022051 2024-04-20T17:29:52.246 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:52.267 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:29:52.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:52.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:29:52.371 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:29:52.403 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:52.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:52 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:29:52.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:52 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:29:52.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:52 smithi106 ceph-mon[44423]: osdmap e550: 12 total, 11 up, 11 in 2024-04-20T17:29:52.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:52 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:29:52.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:52 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:29:52.547 INFO:teuthology.orchestra.run.smithi005.stdout:get_pg_num_history pg_num_history pg_num_history(e538 pg_nums {1={14=1},4={445=16}} deleted_pools ) 2024-04-20T17:29:52.547 INFO:teuthology.orchestra.run.smithi005.stdout:Importing pgid 4.f 2024-04-20T17:29:52.562 INFO:teuthology.orchestra.run.smithi005.stdout:write_pg epoch 538 info 4.f( v 538'21378 (487'11300,538'21378] local-lis/les=445/446 n=0 ec=445/445 lis/c=445/445 les/c/f=446/446/0 sis=445) 2024-04-20T17:29:52.626 INFO:teuthology.orchestra.run.smithi005.stdout:Import successful 2024-04-20T17:29:52.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:52 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:29:52.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:52 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:29:52.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:52 smithi005 ceph-mon[39562]: osdmap e550: 12 total, 11 up, 11 in 2024-04-20T17:29:52.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:52 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:29:52.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:52 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:29:52.741 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:29:52.760 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:29:52.780 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:29:52.801 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:29:52.821 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:29:52.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:52 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:29:52.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:52 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:29:52.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:52 smithi171 ceph-mon[44189]: osdmap e550: 12 total, 11 up, 11 in 2024-04-20T17:29:52.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:52 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:29:52.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:52 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:29:52.842 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:29:52.863 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:29:52.883 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:29:52.904 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:29:52.925 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:29:52.946 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:29:52.946 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:29:52.946 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:29:52.946 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:29:52.946 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-20T17:29:52.946 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:29:52.946 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:29:52.947 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:29:52.947 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:29:52.947 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:29:52.947 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:29:52.947 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:29:52.947 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:29:53.222 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 13 16 203 187 0.898914 0 - 0.0022051 2024-04-20T17:29:53.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:53 smithi005 ceph-mon[39562]: pgmap v1453: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:53.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:53 smithi005 ceph-mon[39562]: osdmap e551: 12 total, 11 up, 11 in 2024-04-20T17:29:53.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:53 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1916024084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:29:53.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:53 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:53.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:53 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:53.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:53 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:53.755 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:29:53.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:53.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:29:53.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:53 smithi106 ceph-mon[44423]: pgmap v1453: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:53.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:53 smithi106 ceph-mon[44423]: osdmap e551: 12 total, 11 up, 11 in 2024-04-20T17:29:53.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:53 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1916024084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:29:53.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:53 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:53.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:53 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:53.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:53 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:53.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:53 smithi171 ceph-mon[44189]: pgmap v1453: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:53.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:53 smithi171 ceph-mon[44189]: osdmap e551: 12 total, 11 up, 11 in 2024-04-20T17:29:53.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:53 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1916024084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:29:53.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:53 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:53.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:53 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:53.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:53 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:53.860 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:29:53.893 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:53.914 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:29:53.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:53.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:29:54.018 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:29:54.052 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:54.126 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:29:54.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:54.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:29:54.222 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 14 16 203 187 0.834705 0 - 0.0022051 2024-04-20T17:29:54.230 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:29:54.262 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:54.284 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:29:54.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:54.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:29:54.388 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:29:54.419 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:54.440 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:29:54.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:54.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:29:54.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:54.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:54.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:54.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:54.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:54.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:54.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:54.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:54.544 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:29:54.574 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:54.594 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:29:54.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:54.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:29:54.649 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/exp.4.f.3 2024-04-20T17:29:54.675 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:54.675 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:54.675 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:54.676 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:54.676 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:54.676 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:54.676 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:54.676 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:54.699 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:29:54.727 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:54.746 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:29:54.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:54.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:29:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:54.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:54.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:54.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:29:54.850 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:29:54.882 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:54.904 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:29:54.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:54.907 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:29:55.008 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:29:55.039 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:55.059 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:29:55.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:55.063 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:29:55.164 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:29:55.176 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:29:55.177 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:29:55.192 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:55.211 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:29:55.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:55.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:29:55.222 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 15 16 203 187 0.77906 0 - 0.0022051 2024-04-20T17:29:55.249 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:29:55.315 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:29:55.342 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:55.361 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:29:55.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:55.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:29:55.406 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:29:55.426 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:55 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:29:55.426 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:55 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:29:55.426 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:55 smithi005 ceph-mon[39562]: pgmap v1455: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:55.426 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:55 smithi005 ceph-mon[39562]: osdmap e552: 12 total, 11 up, 11 in 2024-04-20T17:29:55.426 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:55 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4123275005' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:29:55.465 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:29:55.497 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:55.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:55 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:29:55.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:55 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:29:55.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:55 smithi106 ceph-mon[44423]: pgmap v1455: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:55.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:55 smithi106 ceph-mon[44423]: osdmap e552: 12 total, 11 up, 11 in 2024-04-20T17:29:55.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:55 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4123275005' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:29:55.517 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:29:55.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:55.521 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:29:55.622 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:29:55.649 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:55.668 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:29:55.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:55.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:29:55.772 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:29:55.802 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:55 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:29:55.802 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:55 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:29:55.802 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:55 smithi171 ceph-mon[44189]: pgmap v1455: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:55.802 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:55 smithi171 ceph-mon[44189]: osdmap e552: 12 total, 11 up, 11 in 2024-04-20T17:29:55.802 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:55 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4123275005' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:29:55.804 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:55.825 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:29:55.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:55.829 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:29:55.930 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:29:55.959 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:56.223 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 16 16 203 187 0.730369 0 - 0.0022051 2024-04-20T17:29:56.608 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:29:56.610 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:56 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:29:56.610 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:56 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4123275005' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:29:56.610 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:56 smithi005 ceph-mon[39562]: osdmap e553: 12 total, 11 up, 11 in 2024-04-20T17:29:56.610 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:56 smithi005 ceph-mon[39562]: osdmap e554: 12 total, 11 up, 11 in 2024-04-20T17:29:56.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:56.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:29:56.712 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:29:56.742 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:56 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:29:56.743 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:56 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4123275005' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:29:56.743 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:56 smithi106 ceph-mon[44423]: osdmap e553: 12 total, 11 up, 11 in 2024-04-20T17:29:56.743 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:56 smithi106 ceph-mon[44423]: osdmap e554: 12 total, 11 up, 11 in 2024-04-20T17:29:56.745 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:56.766 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:29:56.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:56.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:29:56.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:56 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:29:56.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:56 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4123275005' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:29:56.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:56 smithi171 ceph-mon[44189]: osdmap e553: 12 total, 11 up, 11 in 2024-04-20T17:29:56.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:56 smithi171 ceph-mon[44189]: osdmap e554: 12 total, 11 up, 11 in 2024-04-20T17:29:56.871 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:29:56.903 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:56.925 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:29:56.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:56.928 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:29:57.029 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:29:57.064 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:57.086 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:29:57.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:57.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:29:57.178 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:29:57.191 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:29:57.223 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 17 16 203 187 0.687405 0 - 0.0022051 2024-04-20T17:29:57.224 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:57.246 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:29:57.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:57.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:29:57.351 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:29:57.384 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:57.407 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:29:57.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:57.410 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:29:57.511 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:29:57.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:57 smithi106 ceph-mon[44423]: pgmap v1459: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:57.542 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:57.644 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:29:57.647 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:57 smithi005 ceph-mon[39562]: pgmap v1459: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:57.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:57.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:29:57.656 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:29:57.656 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:29:57.656 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:29:57.749 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:29:57.779 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:57 smithi171 ceph-mon[44189]: pgmap v1459: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:57.782 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:57.804 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:29:57.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:57.807 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:29:57.908 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:29:57.940 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:57.962 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:29:57.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:57.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:29:58.066 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:29:58.095 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:58.223 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 18 16 203 187 0.649215 0 - 0.0022051 2024-04-20T17:29:58.243 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:29:59.223 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 19 16 203 187 0.615045 0 - 0.0022051 2024-04-20T17:29:59.543 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-20T17:29:59.543 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:29:59.544 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:29:59.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:59.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:29:59.648 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:29:59.679 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:29:59 smithi106 ceph-mon[44423]: pgmap v1460: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:59.681 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 8, 9, 10, 11, 7] out_osds: [6] dead_osds: [3] live_osds: [0, 1, 4, 5, 8, 9, 10, 11, 2, 7, 6] 2024-04-20T17:29:59.681 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-04-20T17:29:59.681 INFO:tasks.cephadm.osd.3:Restarting osd.3 (starting--it wasn't running)... 2024-04-20T17:29:59.681 DEBUG:teuthology.orchestra.run.smithi005:osd.3> sudo journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.3.service 2024-04-20T17:29:59.682 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:59.684 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl start ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.3 2024-04-20T17:29:59.703 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:29:59.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:59.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:29:59.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:29:59 smithi005 ceph-mon[39562]: pgmap v1460: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:59.714 INFO:journalctl@ceph.osd.3.smithi005.stdout:-- Logs begin at Sat 2024-04-20 16:47:38 UTC. -- 2024-04-20T17:29:59.806 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:29:59.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:29:59 smithi171 ceph-mon[44189]: pgmap v1460: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:29:59.839 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:29:59.860 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:29:59.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:29:59.863 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:29:59.961 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:29:59 smithi005 systemd[1]: Starting Ceph osd.3 for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:29:59.964 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:29:59.997 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:00.020 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:30:00.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:00.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:30:00.124 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:30:00.154 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:00.174 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:30:00.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:00.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:30:00.223 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:30:00.224386 min lat: 0.00096363 max lat: 0.00698829 avg lat: 0.0022051 2024-04-20T17:30:00.223 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:30:00.224 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 20 16 203 187 0.584294 0 - 0.0022051 2024-04-20T17:30:00.278 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:30:00.307 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:00.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:00.331 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:30:00.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:30:00.432 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:30:00.465 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:00.488 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:30:00.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:00.491 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:30:00.591 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:30:00.623 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:00 smithi005 ceph-mon[39562]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 osds down 2024-04-20T17:30:00.623 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:00 smithi005 ceph-mon[39562]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-04-20T17:30:00.623 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:00 smithi005 ceph-mon[39562]: [WRN] OSD_DOWN: 1 osds down 2024-04-20T17:30:00.623 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:00 smithi005 ceph-mon[39562]: osd.3 (root=default,host=smithi005) is down 2024-04-20T17:30:00.626 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:00.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:00 smithi106 ceph-mon[44423]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 osds down 2024-04-20T17:30:00.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:00 smithi106 ceph-mon[44423]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-04-20T17:30:00.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:00 smithi106 ceph-mon[44423]: [WRN] OSD_DOWN: 1 osds down 2024-04-20T17:30:00.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:00 smithi106 ceph-mon[44423]: osd.3 (root=default,host=smithi005) is down 2024-04-20T17:30:00.814 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:30:00.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:00 smithi171 ceph-mon[44189]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 osds down 2024-04-20T17:30:00.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:00 smithi171 ceph-mon[44189]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-04-20T17:30:00.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:00 smithi171 ceph-mon[44189]: [WRN] OSD_DOWN: 1 osds down 2024-04-20T17:30:00.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:00 smithi171 ceph-mon[44189]: osd.3 (root=default,host=smithi005) is down 2024-04-20T17:30:00.834 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:30:00.854 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:30:00.875 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:30:00.897 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:30:00.919 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:30:00.940 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:30:00.961 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:30:00.981 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:30:01.004 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:30:01.025 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:30:01.026 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:30:01.026 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:30:01.026 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:30:01.026 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-20T17:30:01.026 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:30:01.026 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:30:01.026 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:30:01.026 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:30:01.026 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:30:01.026 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:30:01.026 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:30:01.026 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:30:01.211 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:30:00 smithi005 bash[233274]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T17:30:01.211 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:30:00 smithi005 bash[233274]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-20T17:30:01.212 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:30:00 smithi005 bash[233274]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-20T17:30:01.212 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:30:00 smithi005 bash[233274]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-20T17:30:01.212 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:30:00 smithi005 bash[233274]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-20T17:30:01.212 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:30:00 smithi005 bash[233274]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-04-20T17:30:01.212 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:30:00 smithi005 bash[233274]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-20T17:30:01.212 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:30:00 smithi005 bash[233274]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-20T17:30:01.223 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 21 16 203 187 0.55647 0 - 0.0022051 2024-04-20T17:30:01.477 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:30:01.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:01.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:30:01.581 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:30:01.614 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:01.636 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:30:01.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:01.639 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:30:01.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:01 smithi005 ceph-mon[39562]: pgmap v1461: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:30:01.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:01 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2348103566' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:30:01.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:01.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:01.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:01.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:01 smithi005 ceph-mon[39562]: osdmap e555: 12 total, 11 up, 11 in 2024-04-20T17:30:01.740 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:30:01.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:01 smithi106 ceph-mon[44423]: pgmap v1461: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:30:01.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:01 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2348103566' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:30:01.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:01.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:01.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:01.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:01 smithi106 ceph-mon[44423]: osdmap e555: 12 total, 11 up, 11 in 2024-04-20T17:30:01.772 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:01 smithi171 ceph-mon[44189]: pgmap v1461: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:30:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:01 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2348103566' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:30:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:01 smithi171 ceph-mon[44189]: osdmap e555: 12 total, 11 up, 11 in 2024-04-20T17:30:01.970 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:30:01.971 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:30:02.224 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 22 16 203 187 0.531175 0 - 0.0022051 2024-04-20T17:30:02.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:02.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:02.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:02.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:02.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:02.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:02.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:02.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:02.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:02 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:30:02.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:02 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:30:02.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:02.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:02.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:02.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:02.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:02.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:02.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:02.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:02.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:02 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:30:02.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:02 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:30:02.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:02.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:02.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:02.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:02.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:02.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:02.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:02.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:02.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:02 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:30:02.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:02 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:30:03.024 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:30:03.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:03.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:30:03.129 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:30:03.161 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:03.224 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 23 16 203 187 0.50808 0 - 0.0022051 2024-04-20T17:30:03.260 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:30:03.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:03.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:30:03.366 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:30:03.401 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:03.422 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:30:03.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:03.425 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:30:03.526 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:30:03.560 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:03.573 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:03 smithi005 ceph-mon[39562]: pgmap v1463: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:30:03.573 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:03 smithi005 ceph-mon[39562]: osdmap e556: 12 total, 11 up, 11 in 2024-04-20T17:30:03.581 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:30:03.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:03.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:30:03.686 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:30:03.715 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:03 smithi171 ceph-mon[44189]: pgmap v1463: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:30:03.715 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:03 smithi171 ceph-mon[44189]: osdmap e556: 12 total, 11 up, 11 in 2024-04-20T17:30:03.718 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:03.739 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:30:03.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:03.742 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:30:03.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:03 smithi106 ceph-mon[44423]: pgmap v1463: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:30:03.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:03 smithi106 ceph-mon[44423]: osdmap e556: 12 total, 11 up, 11 in 2024-04-20T17:30:03.822 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:30:03.843 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:30:03.876 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:03.898 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-20T17:30:03.898 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:30:03.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:03.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:30:03.972 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:30:04.002 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:30:04.044 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:04.147 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:30:04.148 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:30:03 smithi005 bash[233808]: 34ca38613563be9ee984a81e05872777ff67121882776fbc7e713edb0667ef7f 2024-04-20T17:30:04.148 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:30:03 smithi005 systemd[1]: Started Ceph osd.3 for 13c6e76c-ff37-11ee-bc93-c7b262605968. 2024-04-20T17:30:04.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:04.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:30:04.224 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 24 16 203 187 0.48691 0 - 0.0022051 2024-04-20T17:30:04.250 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:30:04.282 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:04.304 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:30:04.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:04.307 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:30:04.400 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:30:04 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-3[233833]: 2024-04-20T17:30:04.296+0000 7fee1bbe87c0 -1 Falling back to public interface 2024-04-20T17:30:04.408 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:30:04.425 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:30:04.443 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:04.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:04 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1185739753' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:30:04.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:04 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1185739753' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:30:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:04 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1185739753' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:30:05.224 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 25 16 203 187 0.467434 0 - 0.0022051 2024-04-20T17:30:05.286 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:30:05.287 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:30:05.294 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:30:05.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:05.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:30:05.402 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:30:05.432 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:05.700 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:05 smithi005 ceph-mon[39562]: pgmap v1465: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:30:05.700 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:05 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:30:05.700 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:05 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1185739753' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:30:05.700 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:05 smithi005 ceph-mon[39562]: osdmap e557: 12 total, 11 up, 11 in 2024-04-20T17:30:05.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:05 smithi106 ceph-mon[44423]: pgmap v1465: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:30:05.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:05 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:30:05.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:05 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1185739753' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:30:05.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:05 smithi106 ceph-mon[44423]: osdmap e557: 12 total, 11 up, 11 in 2024-04-20T17:30:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:05 smithi171 ceph-mon[44189]: pgmap v1465: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:30:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:05 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:30:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:05 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1185739753' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:30:05.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:05 smithi171 ceph-mon[44189]: osdmap e557: 12 total, 11 up, 11 in 2024-04-20T17:30:06.224 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 26 16 203 187 0.449455 0 - 0.0022051 2024-04-20T17:30:07.086 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:30:07.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:07.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:30:07.195 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:30:07.224 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 27 16 203 187 0.43281 0 - 0.0022051 2024-04-20T17:30:07.227 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:07.249 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:30:07.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:07.252 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:30:07.326 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:30:07.347 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:07 smithi005 ceph-mon[39562]: osdmap e558: 12 total, 11 up, 11 in 2024-04-20T17:30:07.347 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:07 smithi005 ceph-mon[39562]: pgmap v1468: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:30:07.353 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:30:07.388 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:07.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:07 smithi106 ceph-mon[44423]: osdmap e558: 12 total, 11 up, 11 in 2024-04-20T17:30:07.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:07 smithi106 ceph-mon[44423]: pgmap v1468: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:30:07.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:07 smithi171 ceph-mon[44189]: osdmap e558: 12 total, 11 up, 11 in 2024-04-20T17:30:07.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:07 smithi171 ceph-mon[44189]: pgmap v1468: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:30:07.682 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:30:07 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-3[233833]: 2024-04-20T17:30:07.499+0000 7fee1bbe87c0 -1 osd.3 538 log_to_monitors true 2024-04-20T17:30:07.839 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:30:07.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:07.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:30:07.946 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:30:07.978 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:07.999 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:30:08.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:08.002 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:30:08.036 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:30:08.036 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:30:08.036 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:30:08.036 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:30:08.037 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.037 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:51519 4.4 4:2ff1b437:::benchmark_data_smithi171_122772_object51518:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e455)", 2024-04-20T17:30:08.037 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:02.044802+0000", 2024-04-20T17:30:08.037 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 305.613878333, 2024-04-20T17:30:08.037 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.084118595000000004, 2024-04-20T17:30:08.037 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:30:08.037 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:30:08.037 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:08.037 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:30:08.037 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:30:08.037 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:30:08.037 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 51519 2024-04-20T17:30:08.037 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.037 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:30:08.037 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.038 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:30:08.038 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.044802+0000", 2024-04-20T17:30:08.038 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.038 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.038 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.038 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:30:08.038 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.044802+0000", 2024-04-20T17:30:08.038 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.038 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.038 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.038 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:30:08.038 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.044809+0000", 2024-04-20T17:30:08.039 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1289999999999998e-06 2024-04-20T17:30:08.039 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.039 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.039 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:30:08.039 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.044839+0000", 2024-04-20T17:30:08.039 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0119000000000001e-05 2024-04-20T17:30:08.039 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.039 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.039 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:30:08.039 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.044852+0000", 2024-04-20T17:30:08.039 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3057000000000001e-05 2024-04-20T17:30:08.039 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.040 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.040 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:30:08.040 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.044858+0000", 2024-04-20T17:30:08.040 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1650000000000003e-06 2024-04-20T17:30:08.040 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.040 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.040 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:30:08.040 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.044904+0000", 2024-04-20T17:30:08.040 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5238000000000003e-05 2024-04-20T17:30:08.040 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.040 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.040 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:30:08.041 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.044994+0000", 2024-04-20T17:30:08.041 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0774999999999997e-05 2024-04-20T17:30:08.041 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.041 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.041 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:30:08.041 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.045095+0000", 2024-04-20T17:30:08.041 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000100974 2024-04-20T17:30:08.041 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.041 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.041 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:30:08.041 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.046359+0000", 2024-04-20T17:30:08.041 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001263772 2024-04-20T17:30:08.041 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.041 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.042 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:08.042 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.128799+0000", 2024-04-20T17:30:08.042 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.082439839000000001 2024-04-20T17:30:08.042 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.042 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.042 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:30:08.042 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.128869+0000", 2024-04-20T17:30:08.042 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0104999999999994e-05 2024-04-20T17:30:08.042 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.042 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.042 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:30:08.042 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.128920+0000", 2024-04-20T17:30:08.043 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1422000000000003e-05 2024-04-20T17:30:08.043 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.043 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:30:08.043 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.043 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.043 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.043 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:51521 4.4 4:20cdb096:::benchmark_data_smithi171_122772_object51520:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e455)", 2024-04-20T17:30:08.043 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:02.047465+0000", 2024-04-20T17:30:08.043 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 305.61121504300002, 2024-04-20T17:30:08.043 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.081531580000000006, 2024-04-20T17:30:08.043 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:30:08.043 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:30:08.044 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:08.044 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:30:08.044 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:30:08.044 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:30:08.044 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 51521 2024-04-20T17:30:08.044 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.044 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:30:08.044 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.044 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:30:08.044 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.047465+0000", 2024-04-20T17:30:08.044 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.044 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.044 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.044 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:30:08.045 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.047465+0000", 2024-04-20T17:30:08.045 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.045 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.045 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.045 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:30:08.045 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.047469+0000", 2024-04-20T17:30:08.045 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4529999999999998e-06 2024-04-20T17:30:08.045 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.045 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.045 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:30:08.045 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.047526+0000", 2024-04-20T17:30:08.045 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7479000000000003e-05 2024-04-20T17:30:08.045 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.045 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.046 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:30:08.046 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.047548+0000", 2024-04-20T17:30:08.046 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1803000000000001e-05 2024-04-20T17:30:08.046 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.046 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.046 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:30:08.046 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.047562+0000", 2024-04-20T17:30:08.046 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4239999999999999e-05 2024-04-20T17:30:08.046 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.046 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.046 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:30:08.046 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.047633+0000", 2024-04-20T17:30:08.046 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1061000000000003e-05 2024-04-20T17:30:08.047 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.047 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.047 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:30:08.047 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.047708+0000", 2024-04-20T17:30:08.047 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5047000000000002e-05 2024-04-20T17:30:08.047 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.047 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.047 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:30:08.047 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.047786+0000", 2024-04-20T17:30:08.047 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7820999999999999e-05 2024-04-20T17:30:08.047 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.047 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.047 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:30:08.047 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.048483+0000", 2024-04-20T17:30:08.047 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00069709600000000002 2024-04-20T17:30:08.048 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.048 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.048 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:08.048 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.128948+0000", 2024-04-20T17:30:08.048 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.080464534000000004 2024-04-20T17:30:08.048 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.048 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.048 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:30:08.048 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.128977+0000", 2024-04-20T17:30:08.048 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9026e-05 2024-04-20T17:30:08.048 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.048 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.048 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:30:08.048 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:02.128997+0000", 2024-04-20T17:30:08.048 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0020000000000001e-05 2024-04-20T17:30:08.048 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.049 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:30:08.049 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.049 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.049 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.049 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:58977 4.4 4:2dde5292:::benchmark_data_smithi171_122772_object58976:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e457)", 2024-04-20T17:30:08.049 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:05.101763+0000", 2024-04-20T17:30:08.049 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 302.55691695299998, 2024-04-20T17:30:08.049 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.056875791000000002, 2024-04-20T17:30:08.049 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:30:08.049 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:30:08.049 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:08.049 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:30:08.049 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:30:08.049 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:30:08.049 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 58977 2024-04-20T17:30:08.050 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.050 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:30:08.050 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.050 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:30:08.050 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.101763+0000", 2024-04-20T17:30:08.050 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.050 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.050 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.050 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:30:08.050 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.101763+0000", 2024-04-20T17:30:08.050 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.050 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.050 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.050 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:30:08.050 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.101771+0000", 2024-04-20T17:30:08.051 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.072e-06 2024-04-20T17:30:08.051 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.051 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.051 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:30:08.051 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.101808+0000", 2024-04-20T17:30:08.051 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6607999999999999e-05 2024-04-20T17:30:08.051 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.051 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.051 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:30:08.051 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.101819+0000", 2024-04-20T17:30:08.052 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1121000000000001e-05 2024-04-20T17:30:08.052 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.052 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.052 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:30:08.052 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.101827+0000", 2024-04-20T17:30:08.052 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0669999999999992e-06 2024-04-20T17:30:08.052 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.052 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.052 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:30:08.052 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.102029+0000", 2024-04-20T17:30:08.052 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00020151300000000001 2024-04-20T17:30:08.052 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.052 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.052 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:30:08.053 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.102151+0000", 2024-04-20T17:30:08.053 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012266499999999999 2024-04-20T17:30:08.053 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.053 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.053 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:30:08.053 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.149674+0000", 2024-04-20T17:30:08.053 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.047522913999999999 2024-04-20T17:30:08.053 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.053 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.053 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:08.053 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.155538+0000", 2024-04-20T17:30:08.053 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0058639160000000003 2024-04-20T17:30:08.053 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.053 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.054 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:30:08.054 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.158550+0000", 2024-04-20T17:30:08.054 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003011746 2024-04-20T17:30:08.054 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.054 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.054 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:30:08.054 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.158596+0000", 2024-04-20T17:30:08.054 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6388999999999999e-05 2024-04-20T17:30:08.054 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.054 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.054 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:30:08.054 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.158639+0000", 2024-04-20T17:30:08.054 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.278e-05 2024-04-20T17:30:08.054 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.055 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:30:08.055 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.055 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.055 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.055 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:58990 4.4 4:2532e405:::benchmark_data_smithi171_122772_object58989:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e457)", 2024-04-20T17:30:08.055 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:05.105472+0000", 2024-04-20T17:30:08.055 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 302.55320767900002, 2024-04-20T17:30:08.055 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.053244023000000001, 2024-04-20T17:30:08.055 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:30:08.055 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:30:08.055 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:08.055 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:30:08.055 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:30:08.056 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:30:08.056 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 58990 2024-04-20T17:30:08.056 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.056 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:30:08.056 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.056 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:30:08.056 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.105472+0000", 2024-04-20T17:30:08.056 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.056 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.056 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.056 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:30:08.056 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.105472+0000", 2024-04-20T17:30:08.056 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.056 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.057 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.057 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:30:08.057 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.105475+0000", 2024-04-20T17:30:08.057 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8210000000000002e-06 2024-04-20T17:30:08.057 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.057 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.057 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:30:08.057 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.105988+0000", 2024-04-20T17:30:08.057 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00051262200000000001 2024-04-20T17:30:08.057 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.057 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.057 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:30:08.057 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.149649+0000", 2024-04-20T17:30:08.057 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.043660860000000003 2024-04-20T17:30:08.057 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.057 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.058 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:30:08.058 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.149667+0000", 2024-04-20T17:30:08.058 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7881e-05 2024-04-20T17:30:08.058 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.058 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.058 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:30:08.058 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.151166+0000", 2024-04-20T17:30:08.058 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0014996219999999999 2024-04-20T17:30:08.058 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.058 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.058 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:30:08.058 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.151285+0000", 2024-04-20T17:30:08.058 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000118966 2024-04-20T17:30:08.059 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.059 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.059 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:30:08.059 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.151381+0000", 2024-04-20T17:30:08.059 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.5586000000000003e-05 2024-04-20T17:30:08.059 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.059 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.059 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:08.059 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.155562+0000", 2024-04-20T17:30:08.059 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041813900000000001 2024-04-20T17:30:08.059 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.059 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.059 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:30:08.059 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.158650+0000", 2024-04-20T17:30:08.060 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0030878400000000001 2024-04-20T17:30:08.060 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.060 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.060 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:30:08.060 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.158695+0000", 2024-04-20T17:30:08.060 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5164999999999999e-05 2024-04-20T17:30:08.060 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.060 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.060 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:30:08.060 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.158717+0000", 2024-04-20T17:30:08.060 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1270000000000001e-05 2024-04-20T17:30:08.060 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.060 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:30:08.060 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.060 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.060 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.061 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:59059 4.4 4:2e8fb1fa:::benchmark_data_smithi171_122772_object59058:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e457)", 2024-04-20T17:30:08.061 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:05.150043+0000", 2024-04-20T17:30:08.061 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 302.50863706299998, 2024-04-20T17:30:08.061 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.044922940000000001, 2024-04-20T17:30:08.061 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:30:08.061 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:30:08.061 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:08.061 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:30:08.061 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:30:08.061 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:30:08.061 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 59059 2024-04-20T17:30:08.061 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.061 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:30:08.061 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.061 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:30:08.061 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.150043+0000", 2024-04-20T17:30:08.062 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.062 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.062 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.062 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:30:08.062 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.150043+0000", 2024-04-20T17:30:08.062 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.062 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.062 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.062 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:30:08.062 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.150044+0000", 2024-04-20T17:30:08.062 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0800000000000005e-07 2024-04-20T17:30:08.062 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.062 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.062 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:30:08.062 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.150112+0000", 2024-04-20T17:30:08.062 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8039000000000004e-05 2024-04-20T17:30:08.062 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.063 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.063 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:30:08.063 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.150117+0000", 2024-04-20T17:30:08.063 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7389999999999996e-06 2024-04-20T17:30:08.063 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.063 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.063 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:30:08.063 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.150122+0000", 2024-04-20T17:30:08.063 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1139999999999997e-06 2024-04-20T17:30:08.063 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.063 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.063 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:30:08.063 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.161671+0000", 2024-04-20T17:30:08.063 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011548424 2024-04-20T17:30:08.064 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.064 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.064 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:30:08.064 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.161741+0000", 2024-04-20T17:30:08.064 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9944000000000007e-05 2024-04-20T17:30:08.064 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.064 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.064 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:30:08.064 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.161835+0000", 2024-04-20T17:30:08.064 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.3854e-05 2024-04-20T17:30:08.064 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.064 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.064 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:30:08.065 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.164717+0000", 2024-04-20T17:30:08.065 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002881991 2024-04-20T17:30:08.065 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.065 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.065 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:08.065 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.167864+0000", 2024-04-20T17:30:08.065 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003146949 2024-04-20T17:30:08.065 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.065 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.065 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:30:08.065 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.167910+0000", 2024-04-20T17:30:08.065 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6564999999999999e-05 2024-04-20T17:30:08.065 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.065 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.066 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:30:08.066 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.194966+0000", 2024-04-20T17:30:08.066 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.027055813000000001 2024-04-20T17:30:08.066 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.066 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:30:08.066 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.066 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.066 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.066 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:59065 4.4 4:21d9e786:::benchmark_data_smithi171_122772_object59064:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e457)", 2024-04-20T17:30:08.066 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:05.150128+0000", 2024-04-20T17:30:08.066 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 302.50855169099998, 2024-04-20T17:30:08.066 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.047209095, 2024-04-20T17:30:08.066 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:30:08.067 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:30:08.067 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:08.067 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:30:08.067 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:30:08.067 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:30:08.067 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 59065 2024-04-20T17:30:08.067 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.067 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:30:08.067 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.067 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:30:08.067 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.150128+0000", 2024-04-20T17:30:08.067 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.067 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.068 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.068 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:30:08.068 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.150128+0000", 2024-04-20T17:30:08.068 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.068 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.068 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.068 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:30:08.068 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.150129+0000", 2024-04-20T17:30:08.068 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.75e-07 2024-04-20T17:30:08.068 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.068 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.068 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:30:08.068 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.150181+0000", 2024-04-20T17:30:08.069 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1959000000000002e-05 2024-04-20T17:30:08.069 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.069 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.069 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:30:08.069 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.150186+0000", 2024-04-20T17:30:08.069 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.434e-06 2024-04-20T17:30:08.069 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.069 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.069 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:30:08.069 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.150191+0000", 2024-04-20T17:30:08.069 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7870000000000001e-06 2024-04-20T17:30:08.069 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.069 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.070 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:30:08.070 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.163071+0000", 2024-04-20T17:30:08.070 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012879493000000001 2024-04-20T17:30:08.070 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.070 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.070 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:30:08.070 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.163173+0000", 2024-04-20T17:30:08.070 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000101999 2024-04-20T17:30:08.070 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.070 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.070 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:30:08.070 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.163301+0000", 2024-04-20T17:30:08.070 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012849399999999999 2024-04-20T17:30:08.070 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.070 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.071 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:30:08.071 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.166033+0000", 2024-04-20T17:30:08.071 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0027314879999999998 2024-04-20T17:30:08.071 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.071 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.071 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:08.071 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.197201+0000", 2024-04-20T17:30:08.071 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.031168095999999999 2024-04-20T17:30:08.071 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.071 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.071 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:30:08.071 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.197273+0000", 2024-04-20T17:30:08.071 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2248000000000004e-05 2024-04-20T17:30:08.072 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.072 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.072 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:30:08.072 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.197338+0000", 2024-04-20T17:30:08.072 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4522000000000003e-05 2024-04-20T17:30:08.072 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.072 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:30:08.072 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.072 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.072 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.072 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:59083 4.4 4:26fffb8e:::benchmark_data_smithi171_122772_object59082:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e457)", 2024-04-20T17:30:08.072 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:05.150197+0000", 2024-04-20T17:30:08.072 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 302.508483265, 2024-04-20T17:30:08.073 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.047286645000000002, 2024-04-20T17:30:08.073 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:30:08.073 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:30:08.073 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:08.073 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:30:08.073 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:30:08.073 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:30:08.073 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 59083 2024-04-20T17:30:08.073 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.073 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:30:08.073 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.073 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:30:08.073 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.150197+0000", 2024-04-20T17:30:08.073 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.073 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.074 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.074 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:30:08.074 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.150197+0000", 2024-04-20T17:30:08.074 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.074 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.074 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.074 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:30:08.074 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.150197+0000", 2024-04-20T17:30:08.074 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2e-07 2024-04-20T17:30:08.074 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.074 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.074 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:30:08.074 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.150246+0000", 2024-04-20T17:30:08.074 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8167999999999997e-05 2024-04-20T17:30:08.075 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.075 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.075 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:30:08.075 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.150251+0000", 2024-04-20T17:30:08.075 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5130000000000004e-06 2024-04-20T17:30:08.075 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.075 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.075 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:30:08.075 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.150256+0000", 2024-04-20T17:30:08.075 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6349999999999997e-06 2024-04-20T17:30:08.075 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.075 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.075 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:30:08.075 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.164749+0000", 2024-04-20T17:30:08.076 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014493281 2024-04-20T17:30:08.076 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.076 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.076 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:30:08.076 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.164849+0000", 2024-04-20T17:30:08.076 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010030999999999999 2024-04-20T17:30:08.076 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.076 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.076 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:30:08.076 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.164931+0000", 2024-04-20T17:30:08.076 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1453999999999996e-05 2024-04-20T17:30:08.076 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.076 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.076 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:30:08.077 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.197123+0000", 2024-04-20T17:30:08.077 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.032192463999999997 2024-04-20T17:30:08.077 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.077 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.077 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:08.077 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.197393+0000", 2024-04-20T17:30:08.077 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00026953200000000001 2024-04-20T17:30:08.077 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.077 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.077 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:30:08.077 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.197442+0000", 2024-04-20T17:30:08.077 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8851999999999999e-05 2024-04-20T17:30:08.077 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.077 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.078 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:30:08.078 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.197484+0000", 2024-04-20T17:30:08.078 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1916000000000003e-05 2024-04-20T17:30:08.078 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.078 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:30:08.078 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.078 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.078 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.078 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:59091 4.4 4:2650808a:::benchmark_data_smithi171_122772_object59090:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e457)", 2024-04-20T17:30:08.078 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:05.153391+0000", 2024-04-20T17:30:08.078 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 302.50528883200002, 2024-04-20T17:30:08.078 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.044216990999999997, 2024-04-20T17:30:08.079 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:30:08.079 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:30:08.079 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:08.079 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:30:08.079 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:30:08.079 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:30:08.079 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 59091 2024-04-20T17:30:08.079 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.079 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:30:08.079 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.079 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:30:08.079 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.153391+0000", 2024-04-20T17:30:08.079 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.079 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.079 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.080 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:30:08.080 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.153391+0000", 2024-04-20T17:30:08.080 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.080 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.080 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.080 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:30:08.080 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.153393+0000", 2024-04-20T17:30:08.080 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7e-06 2024-04-20T17:30:08.080 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.080 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.080 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:30:08.080 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.153448+0000", 2024-04-20T17:30:08.081 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4605999999999999e-05 2024-04-20T17:30:08.081 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.081 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.081 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:30:08.081 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.153457+0000", 2024-04-20T17:30:08.081 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2080000000000006e-06 2024-04-20T17:30:08.081 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.081 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.081 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:30:08.081 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.153462+0000", 2024-04-20T17:30:08.081 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6210000000000001e-06 2024-04-20T17:30:08.081 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.081 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.081 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:30:08.082 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.166119+0000", 2024-04-20T17:30:08.082 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012656687 2024-04-20T17:30:08.082 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.082 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.082 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:30:08.082 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.166190+0000", 2024-04-20T17:30:08.082 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1027999999999998e-05 2024-04-20T17:30:08.082 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.082 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.082 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:30:08.082 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.166274+0000", 2024-04-20T17:30:08.083 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3609000000000005e-05 2024-04-20T17:30:08.083 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.083 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.083 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:30:08.083 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.197130+0000", 2024-04-20T17:30:08.083 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.030856071999999998 2024-04-20T17:30:08.083 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.083 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.083 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:08.083 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.197527+0000", 2024-04-20T17:30:08.083 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000397499 2024-04-20T17:30:08.083 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.083 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.084 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:30:08.084 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.197576+0000", 2024-04-20T17:30:08.084 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8699000000000002e-05 2024-04-20T17:30:08.084 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.084 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.084 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:30:08.084 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:05.197608+0000", 2024-04-20T17:30:08.084 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2261999999999997e-05 2024-04-20T17:30:08.084 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.084 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:30:08.084 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.084 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.084 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.084 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:63535 4.4 4:2d7f2e3e:::benchmark_data_smithi171_122772_object63534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e458)", 2024-04-20T17:30:08.085 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:06.884854+0000", 2024-04-20T17:30:08.085 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 300.77382663399999, 2024-04-20T17:30:08.085 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.055304684, 2024-04-20T17:30:08.085 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:30:08.085 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:30:08.085 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:08.085 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:30:08.085 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:30:08.085 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:30:08.085 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 63535 2024-04-20T17:30:08.085 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.085 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:30:08.085 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.085 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:30:08.086 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.884854+0000", 2024-04-20T17:30:08.086 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.086 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.086 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.086 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:30:08.086 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.884854+0000", 2024-04-20T17:30:08.086 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.086 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.086 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.086 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:30:08.086 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.884855+0000", 2024-04-20T17:30:08.086 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0100000000000001e-06 2024-04-20T17:30:08.086 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.086 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.087 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:30:08.087 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.884965+0000", 2024-04-20T17:30:08.087 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011016 2024-04-20T17:30:08.087 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.087 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.087 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:30:08.087 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.884972+0000", 2024-04-20T17:30:08.087 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6290000000000001e-06 2024-04-20T17:30:08.087 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.087 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.087 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:30:08.087 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.884979+0000", 2024-04-20T17:30:08.087 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2940000000000002e-06 2024-04-20T17:30:08.087 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.088 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.088 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:30:08.088 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.886817+0000", 2024-04-20T17:30:08.088 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0018385719999999999 2024-04-20T17:30:08.088 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.088 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.088 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:30:08.088 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.886947+0000", 2024-04-20T17:30:08.088 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000130307 2024-04-20T17:30:08.088 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.088 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.088 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:30:08.088 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.887087+0000", 2024-04-20T17:30:08.089 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013925299999999999 2024-04-20T17:30:08.089 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.089 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.089 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:30:08.089 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.889496+0000", 2024-04-20T17:30:08.089 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0024089860000000001 2024-04-20T17:30:08.089 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.089 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.089 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:08.089 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.940063+0000", 2024-04-20T17:30:08.089 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.050567127000000003 2024-04-20T17:30:08.089 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.089 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.090 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:30:08.090 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.940116+0000", 2024-04-20T17:30:08.090 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2958999999999999e-05 2024-04-20T17:30:08.090 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.090 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.090 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:30:08.090 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.940158+0000", 2024-04-20T17:30:08.090 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2386999999999999e-05 2024-04-20T17:30:08.090 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.090 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:30:08.090 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.090 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.090 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.090 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:63556 4.4 4:2c77a19e:::benchmark_data_smithi171_122772_object63555:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e458)", 2024-04-20T17:30:08.091 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:25:06.898058+0000", 2024-04-20T17:30:08.091 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 300.76062206099999, 2024-04-20T17:30:08.091 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.045497470999999998, 2024-04-20T17:30:08.091 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:30:08.091 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:30:08.091 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:08.091 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:30:08.091 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:30:08.091 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:30:08.091 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 63556 2024-04-20T17:30:08.091 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.091 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:30:08.092 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.092 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:30:08.092 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.898058+0000", 2024-04-20T17:30:08.092 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.092 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.092 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.092 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:30:08.092 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.898058+0000", 2024-04-20T17:30:08.092 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.092 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.092 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.092 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:30:08.092 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.898060+0000", 2024-04-20T17:30:08.093 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2790000000000001e-06 2024-04-20T17:30:08.093 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.093 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.093 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:30:08.093 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.898158+0000", 2024-04-20T17:30:08.093 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7209999999999999e-05 2024-04-20T17:30:08.093 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.093 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.093 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:30:08.093 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.898169+0000", 2024-04-20T17:30:08.093 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1382e-05 2024-04-20T17:30:08.093 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.093 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.094 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:30:08.094 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.898176+0000", 2024-04-20T17:30:08.094 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7290000000000003e-06 2024-04-20T17:30:08.094 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.094 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.094 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:30:08.094 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.898222+0000", 2024-04-20T17:30:08.094 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5850000000000003e-05 2024-04-20T17:30:08.094 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.094 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.094 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:30:08.094 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.898325+0000", 2024-04-20T17:30:08.094 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000103924 2024-04-20T17:30:08.095 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.095 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.095 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:30:08.095 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.898407+0000", 2024-04-20T17:30:08.095 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1874999999999998e-05 2024-04-20T17:30:08.095 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.095 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.095 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:30:08.095 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.899932+0000", 2024-04-20T17:30:08.095 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0015242249999999999 2024-04-20T17:30:08.095 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.096 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.096 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:08.096 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.943449+0000", 2024-04-20T17:30:08.096 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.043517137999999997 2024-04-20T17:30:08.096 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.096 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.096 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:30:08.096 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.943508+0000", 2024-04-20T17:30:08.096 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9582e-05 2024-04-20T17:30:08.097 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.097 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.097 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:30:08.097 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:25:06.943556+0000", 2024-04-20T17:30:08.097 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7277000000000002e-05 2024-04-20T17:30:08.097 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.097 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:30:08.097 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.097 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.097 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.097 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:169197 4.4 4:2f880fd5:::benchmark_data_smithi171_122772_object169196:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e486)", 2024-04-20T17:30:08.097 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:17.640381+0000", 2024-04-20T17:30:08.097 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 230.018299364, 2024-04-20T17:30:08.097 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.048248721000000001, 2024-04-20T17:30:08.097 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:30:08.098 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:30:08.098 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:08.098 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:30:08.098 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:30:08.098 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:30:08.098 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 169197 2024-04-20T17:30:08.098 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.098 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:30:08.098 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.098 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:30:08.098 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:17.640381+0000", 2024-04-20T17:30:08.098 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.098 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.098 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.099 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:30:08.099 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:17.640381+0000", 2024-04-20T17:30:08.099 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.099 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.099 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.099 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:30:08.099 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:17.640383+0000", 2024-04-20T17:30:08.099 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0020000000000001e-06 2024-04-20T17:30:08.099 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.099 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.099 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:30:08.099 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:17.640437+0000", 2024-04-20T17:30:08.099 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.431e-05 2024-04-20T17:30:08.099 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.099 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.100 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:30:08.100 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:17.640445+0000", 2024-04-20T17:30:08.100 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1659999999999994e-06 2024-04-20T17:30:08.100 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.100 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.100 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:30:08.100 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:17.640451+0000", 2024-04-20T17:30:08.100 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2850000000000002e-06 2024-04-20T17:30:08.100 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.100 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.100 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:30:08.100 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:17.640489+0000", 2024-04-20T17:30:08.100 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8139000000000002e-05 2024-04-20T17:30:08.100 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.100 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.100 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:30:08.101 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:17.640570+0000", 2024-04-20T17:30:08.101 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0926000000000004e-05 2024-04-20T17:30:08.101 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.101 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.101 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:30:08.101 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:17.640658+0000", 2024-04-20T17:30:08.101 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8187000000000001e-05 2024-04-20T17:30:08.101 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.101 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.101 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:30:08.101 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:17.643832+0000", 2024-04-20T17:30:08.101 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003174563 2024-04-20T17:30:08.102 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.102 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.102 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:08.102 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:17.688522+0000", 2024-04-20T17:30:08.102 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.044689705000000003 2024-04-20T17:30:08.102 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.102 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.102 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:30:08.102 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:17.688582+0000", 2024-04-20T17:30:08.102 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0213000000000001e-05 2024-04-20T17:30:08.102 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.102 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.102 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:30:08.102 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:17.688630+0000", 2024-04-20T17:30:08.103 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7225000000000003e-05 2024-04-20T17:30:08.103 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.103 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:30:08.103 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.103 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.105 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.105 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:205239 4.4 4:2c85adc5:::benchmark_data_smithi171_122772_object35047:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:30:08.106 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:47.015559+0000", 2024-04-20T17:30:08.106 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 200.643120724, 2024-04-20T17:30:08.106 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.050199922000000001, 2024-04-20T17:30:08.106 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:30:08.106 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:30:08.106 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:08.106 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:30:08.106 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:30:08.106 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:30:08.106 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 205239 2024-04-20T17:30:08.106 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.106 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:30:08.106 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.107 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:30:08.107 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:47.015559+0000", 2024-04-20T17:30:08.107 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.107 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.107 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.107 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:30:08.107 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:47.015559+0000", 2024-04-20T17:30:08.107 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.107 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.107 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.107 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:30:08.107 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:47.015560+0000", 2024-04-20T17:30:08.107 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2300000000000001e-07 2024-04-20T17:30:08.107 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.107 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.108 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:30:08.108 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:47.015563+0000", 2024-04-20T17:30:08.108 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7669999999999999e-06 2024-04-20T17:30:08.108 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.108 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.108 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:30:08.108 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:47.015566+0000", 2024-04-20T17:30:08.108 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4209999999999999e-06 2024-04-20T17:30:08.108 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.108 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.108 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:30:08.108 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:47.015569+0000", 2024-04-20T17:30:08.108 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0489999999999999e-06 2024-04-20T17:30:08.108 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.109 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.109 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:30:08.109 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:47.018769+0000", 2024-04-20T17:30:08.109 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0032002060000000001 2024-04-20T17:30:08.109 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.109 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.109 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:30:08.109 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:47.018842+0000", 2024-04-20T17:30:08.109 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2914e-05 2024-04-20T17:30:08.109 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.109 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.109 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:30:08.109 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:47.018909+0000", 2024-04-20T17:30:08.109 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6717000000000005e-05 2024-04-20T17:30:08.109 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.110 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.110 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:30:08.110 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:47.021402+0000", 2024-04-20T17:30:08.110 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002493224 2024-04-20T17:30:08.110 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.110 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.110 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:08.110 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:47.065655+0000", 2024-04-20T17:30:08.110 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.044252863000000003 2024-04-20T17:30:08.110 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.110 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.110 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:30:08.110 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:47.065711+0000", 2024-04-20T17:30:08.110 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5340000000000002e-05 2024-04-20T17:30:08.111 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.111 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.111 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:30:08.111 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:47.065759+0000", 2024-04-20T17:30:08.111 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8798000000000003e-05 2024-04-20T17:30:08.111 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.111 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:30:08.111 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.111 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.111 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.111 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:208169 4.4 4:276edba7:::benchmark_data_smithi171_122772_object37977:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:30:08.111 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:49.395751+0000", 2024-04-20T17:30:08.111 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 198.26292919299999, 2024-04-20T17:30:08.111 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.042371544999999997, 2024-04-20T17:30:08.112 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:30:08.112 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:30:08.112 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:08.112 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:30:08.112 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:30:08.112 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:30:08.112 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 208169 2024-04-20T17:30:08.112 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.112 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:30:08.112 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.112 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:30:08.112 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.395751+0000", 2024-04-20T17:30:08.112 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.112 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.112 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.113 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:30:08.113 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.395751+0000", 2024-04-20T17:30:08.113 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.113 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.113 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.113 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:30:08.113 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.395752+0000", 2024-04-20T17:30:08.113 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.331e-06 2024-04-20T17:30:08.113 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.113 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.113 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:30:08.113 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.395757+0000", 2024-04-20T17:30:08.113 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0919999999999997e-06 2024-04-20T17:30:08.113 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.113 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.113 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:30:08.114 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.395767+0000", 2024-04-20T17:30:08.114 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.995e-06 2024-04-20T17:30:08.114 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.114 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.114 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:30:08.114 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.395774+0000", 2024-04-20T17:30:08.114 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0600000000000002e-06 2024-04-20T17:30:08.114 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.114 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.114 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:30:08.114 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.399325+0000", 2024-04-20T17:30:08.114 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0035500760000000001 2024-04-20T17:30:08.114 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.114 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.115 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:30:08.115 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.399448+0000", 2024-04-20T17:30:08.115 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0001236 2024-04-20T17:30:08.115 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.115 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.115 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:30:08.115 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.399550+0000", 2024-04-20T17:30:08.115 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000101709 2024-04-20T17:30:08.115 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.115 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.115 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:30:08.115 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.408012+0000", 2024-04-20T17:30:08.115 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0084624580000000008 2024-04-20T17:30:08.115 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.116 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.116 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:08.116 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.438023+0000", 2024-04-20T17:30:08.116 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.030010564999999999 2024-04-20T17:30:08.116 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.116 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.116 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:30:08.116 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.438075+0000", 2024-04-20T17:30:08.116 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2357e-05 2024-04-20T17:30:08.116 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.116 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.116 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:30:08.116 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.438123+0000", 2024-04-20T17:30:08.116 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7302000000000002e-05 2024-04-20T17:30:08.117 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.117 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:30:08.117 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.117 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.117 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.117 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:214332 4.4 4:20f0e731:::benchmark_data_smithi171_122772_object44140:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:30:08.117 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:54.274418+0000", 2024-04-20T17:30:08.117 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 193.38426210399999, 2024-04-20T17:30:08.117 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.042338022000000003, 2024-04-20T17:30:08.117 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:30:08.117 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:30:08.117 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:08.117 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:30:08.117 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:30:08.117 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:30:08.118 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 214332 2024-04-20T17:30:08.118 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.118 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:30:08.118 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.118 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:30:08.118 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:54.274418+0000", 2024-04-20T17:30:08.118 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.118 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.118 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.118 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:30:08.118 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:54.274418+0000", 2024-04-20T17:30:08.118 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.118 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.118 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.119 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:30:08.119 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:54.274420+0000", 2024-04-20T17:30:08.119 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1339999999999998e-06 2024-04-20T17:30:08.119 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.119 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.119 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:30:08.119 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:54.274427+0000", 2024-04-20T17:30:08.119 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5919999999999997e-06 2024-04-20T17:30:08.119 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.119 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.119 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:30:08.119 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:54.274439+0000", 2024-04-20T17:30:08.119 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2357999999999999e-05 2024-04-20T17:30:08.120 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.120 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.120 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:30:08.120 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:54.274447+0000", 2024-04-20T17:30:08.120 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3110000000000006e-06 2024-04-20T17:30:08.120 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.120 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.120 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:30:08.120 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:54.289706+0000", 2024-04-20T17:30:08.120 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015258249 2024-04-20T17:30:08.120 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.120 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.120 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:30:08.120 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:54.289849+0000", 2024-04-20T17:30:08.120 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000143163 2024-04-20T17:30:08.120 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.121 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.121 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:30:08.121 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:54.289988+0000", 2024-04-20T17:30:08.121 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013955200000000001 2024-04-20T17:30:08.121 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.121 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.121 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:30:08.121 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:54.301313+0000", 2024-04-20T17:30:08.121 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011324209 2024-04-20T17:30:08.121 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.121 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.121 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:08.121 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:54.316651+0000", 2024-04-20T17:30:08.121 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015338424 2024-04-20T17:30:08.122 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.122 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.122 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:30:08.122 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:54.316706+0000", 2024-04-20T17:30:08.122 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.503e-05 2024-04-20T17:30:08.122 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.122 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.122 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:30:08.122 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:54.316756+0000", 2024-04-20T17:30:08.122 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0000000000000002e-05 2024-04-20T17:30:08.122 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.122 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:30:08.122 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.122 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.122 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.123 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:214337 4.4 4:2d7dd2f6:::benchmark_data_smithi171_122772_object44145:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:30:08.123 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:54.274533+0000", 2024-04-20T17:30:08.123 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 193.384147329, 2024-04-20T17:30:08.123 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.052783238000000003, 2024-04-20T17:30:08.123 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:30:08.123 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:30:08.123 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:08.123 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:30:08.123 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:30:08.123 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:30:08.123 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 214337 2024-04-20T17:30:08.123 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.123 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:30:08.123 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.123 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:30:08.124 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:54.274533+0000", 2024-04-20T17:30:08.124 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.124 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.124 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.124 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:30:08.124 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:54.274533+0000", 2024-04-20T17:30:08.124 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.124 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.124 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.124 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:30:08.124 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:54.274534+0000", 2024-04-20T17:30:08.124 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2330000000000001e-06 2024-04-20T17:30:08.124 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.125 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.125 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:30:08.125 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:54.274539+0000", 2024-04-20T17:30:08.125 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6779999999999999e-06 2024-04-20T17:30:08.125 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.125 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.125 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:30:08.125 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:54.274548+0000", 2024-04-20T17:30:08.125 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7839999999999992e-06 2024-04-20T17:30:08.125 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.125 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.125 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:30:08.125 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:54.274554+0000", 2024-04-20T17:30:08.125 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7150000000000001e-06 2024-04-20T17:30:08.126 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.126 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.126 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:30:08.126 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:54.295747+0000", 2024-04-20T17:30:08.126 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.021192725999999999 2024-04-20T17:30:08.126 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.126 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.126 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:30:08.126 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:54.295904+0000", 2024-04-20T17:30:08.126 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015745399999999999 2024-04-20T17:30:08.126 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.126 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.126 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:30:08.127 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:54.296053+0000", 2024-04-20T17:30:08.127 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000149049 2024-04-20T17:30:08.127 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.127 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.127 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:30:08.127 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:54.307980+0000", 2024-04-20T17:30:08.127 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011926305999999999 2024-04-20T17:30:08.127 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.127 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.127 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:08.127 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:54.327191+0000", 2024-04-20T17:30:08.127 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.019211675000000001 2024-04-20T17:30:08.127 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.127 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.128 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:30:08.128 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:54.327255+0000", 2024-04-20T17:30:08.128 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3108999999999995e-05 2024-04-20T17:30:08.128 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.128 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.128 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:30:08.128 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:54.327316+0000", 2024-04-20T17:30:08.128 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1508999999999997e-05 2024-04-20T17:30:08.128 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.128 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:30:08.128 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.128 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.128 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.129 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:246477 4.4 4:2d6aa2ad:::benchmark_data_smithi171_122772_object76285:head [delete] snapc 0=[] ondisk+write+known_if_redirected e502)", 2024-04-20T17:30:08.129 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:27:20.872999+0000", 2024-04-20T17:30:08.129 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 166.78568142099999, 2024-04-20T17:30:08.129 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057819019999999999, 2024-04-20T17:30:08.129 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:30:08.129 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:30:08.129 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:08.129 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:30:08.129 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:30:08.129 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:30:08.129 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 246477 2024-04-20T17:30:08.129 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.129 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:30:08.129 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.130 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:30:08.130 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.872999+0000", 2024-04-20T17:30:08.130 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.130 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.130 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.130 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:30:08.130 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.872999+0000", 2024-04-20T17:30:08.130 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.130 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.130 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.130 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:30:08.130 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.873000+0000", 2024-04-20T17:30:08.130 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.421e-06 2024-04-20T17:30:08.130 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.131 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.131 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:30:08.131 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.873006+0000", 2024-04-20T17:30:08.131 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8329999999999999e-06 2024-04-20T17:30:08.131 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.131 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.131 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:30:08.131 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.873019+0000", 2024-04-20T17:30:08.131 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3397000000000001e-05 2024-04-20T17:30:08.131 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.131 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.131 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:30:08.131 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.873027+0000", 2024-04-20T17:30:08.131 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1300000000000003e-06 2024-04-20T17:30:08.131 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.132 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.132 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:30:08.132 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.881356+0000", 2024-04-20T17:30:08.132 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0083299829999999991 2024-04-20T17:30:08.132 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.132 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.132 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:30:08.132 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.881490+0000", 2024-04-20T17:30:08.132 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013394000000000001 2024-04-20T17:30:08.132 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.132 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.132 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:30:08.132 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.881622+0000", 2024-04-20T17:30:08.132 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000131798 2024-04-20T17:30:08.132 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.133 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.133 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:30:08.133 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.891327+0000", 2024-04-20T17:30:08.133 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0097050179999999993 2024-04-20T17:30:08.133 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.133 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.133 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:08.133 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.930696+0000", 2024-04-20T17:30:08.133 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.039368893000000002 2024-04-20T17:30:08.133 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.133 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.133 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:30:08.133 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.930759+0000", 2024-04-20T17:30:08.133 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2445000000000003e-05 2024-04-20T17:30:08.134 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.134 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.134 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:30:08.134 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.930818+0000", 2024-04-20T17:30:08.134 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9162e-05 2024-04-20T17:30:08.134 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.134 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:30:08.134 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.134 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.134 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.134 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:246479 4.4 4:2e6f790e:::benchmark_data_smithi171_122772_object76287:head [delete] snapc 0=[] ondisk+write+known_if_redirected e502)", 2024-04-20T17:30:08.134 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:27:20.875534+0000", 2024-04-20T17:30:08.134 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 166.78314617999999, 2024-04-20T17:30:08.134 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.055418654999999997, 2024-04-20T17:30:08.135 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:30:08.135 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:30:08.135 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:08.135 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:30:08.135 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:30:08.135 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:30:08.135 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 246479 2024-04-20T17:30:08.135 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.135 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:30:08.135 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.135 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:30:08.135 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.875534+0000", 2024-04-20T17:30:08.135 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.135 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.135 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.136 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:30:08.136 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.875534+0000", 2024-04-20T17:30:08.136 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.136 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.136 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.136 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:30:08.136 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.875537+0000", 2024-04-20T17:30:08.136 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2109999999999998e-06 2024-04-20T17:30:08.136 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.136 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.136 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:30:08.136 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.875549+0000", 2024-04-20T17:30:08.136 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.223e-05 2024-04-20T17:30:08.137 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.137 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.137 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:30:08.137 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.875573+0000", 2024-04-20T17:30:08.137 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3271000000000001e-05 2024-04-20T17:30:08.137 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.137 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.137 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:30:08.137 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.875585+0000", 2024-04-20T17:30:08.137 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2578000000000001e-05 2024-04-20T17:30:08.137 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.137 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.137 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:30:08.137 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.886250+0000", 2024-04-20T17:30:08.138 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010664666999999999 2024-04-20T17:30:08.138 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.138 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.138 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:30:08.138 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.886372+0000", 2024-04-20T17:30:08.138 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000121914 2024-04-20T17:30:08.138 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.138 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.138 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:30:08.138 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.886484+0000", 2024-04-20T17:30:08.138 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000112528 2024-04-20T17:30:08.138 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.138 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.138 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:30:08.138 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.891486+0000", 2024-04-20T17:30:08.139 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0050019180000000002 2024-04-20T17:30:08.139 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.139 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.139 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:08.139 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.930860+0000", 2024-04-20T17:30:08.139 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.039373416000000001 2024-04-20T17:30:08.139 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.139 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.139 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:30:08.139 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.930910+0000", 2024-04-20T17:30:08.139 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0334999999999998e-05 2024-04-20T17:30:08.139 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.139 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.139 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:30:08.140 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:27:20.930953+0000", 2024-04-20T17:30:08.140 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2586999999999997e-05 2024-04-20T17:30:08.140 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.140 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:30:08.140 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.140 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.140 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.140 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:303968 4.4 4:2bacde25:::benchmark_data_smithi171_122772_object133776:head [delete] snapc 0=[] ondisk+write+known_if_redirected e527)", 2024-04-20T17:30:08.140 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:28:45.345822+0000", 2024-04-20T17:30:08.140 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 82.312858546000001, 2024-04-20T17:30:08.140 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.042765878, 2024-04-20T17:30:08.140 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:30:08.140 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:30:08.141 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:08.141 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:30:08.141 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:30:08.141 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:30:08.141 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 303968 2024-04-20T17:30:08.141 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.141 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:30:08.141 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.141 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:30:08.141 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.345822+0000", 2024-04-20T17:30:08.141 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.141 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.141 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.142 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:30:08.142 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.345822+0000", 2024-04-20T17:30:08.142 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.142 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.142 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.142 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:30:08.142 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.345824+0000", 2024-04-20T17:30:08.142 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.086e-06 2024-04-20T17:30:08.142 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.142 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.142 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:30:08.142 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.345830+0000", 2024-04-20T17:30:08.142 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8819999999999998e-06 2024-04-20T17:30:08.143 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.143 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.143 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:30:08.143 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.345843+0000", 2024-04-20T17:30:08.143 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3609e-05 2024-04-20T17:30:08.143 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.143 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.143 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:30:08.143 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.345850+0000", 2024-04-20T17:30:08.143 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1999999999999997e-06 2024-04-20T17:30:08.143 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.143 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.143 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:30:08.143 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.345893+0000", 2024-04-20T17:30:08.143 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2382999999999997e-05 2024-04-20T17:30:08.144 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.144 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.144 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:30:08.144 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.346041+0000", 2024-04-20T17:30:08.144 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014827500000000001 2024-04-20T17:30:08.144 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.144 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.144 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:30:08.144 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.346173+0000", 2024-04-20T17:30:08.144 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000131495 2024-04-20T17:30:08.144 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.144 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.144 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:30:08.145 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.350226+0000", 2024-04-20T17:30:08.145 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0040536890000000001 2024-04-20T17:30:08.145 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.145 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.145 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:08.145 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.388469+0000", 2024-04-20T17:30:08.145 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.038243265999999998 2024-04-20T17:30:08.145 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.145 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.145 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:30:08.145 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.388530+0000", 2024-04-20T17:30:08.145 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0386000000000001e-05 2024-04-20T17:30:08.145 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.145 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.146 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:30:08.146 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.388587+0000", 2024-04-20T17:30:08.146 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7606999999999999e-05 2024-04-20T17:30:08.146 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.146 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:30:08.146 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.146 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.146 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.146 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:303972 4.4 4:20450e0c:::benchmark_data_smithi171_122772_object133780:head [delete] snapc 0=[] ondisk+write+known_if_redirected e527)", 2024-04-20T17:30:08.146 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:28:45.349766+0000", 2024-04-20T17:30:08.146 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 82.308914621, 2024-04-20T17:30:08.146 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.047533793999999997, 2024-04-20T17:30:08.146 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:30:08.146 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:30:08.147 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:08.147 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:30:08.147 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:30:08.147 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:30:08.147 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 303972 2024-04-20T17:30:08.147 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.147 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:30:08.147 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.147 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:30:08.147 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.349766+0000", 2024-04-20T17:30:08.147 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.147 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.147 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.147 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:30:08.148 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.349766+0000", 2024-04-20T17:30:08.148 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.148 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.148 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.148 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:30:08.148 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.349769+0000", 2024-04-20T17:30:08.148 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4929999999999998e-06 2024-04-20T17:30:08.148 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.148 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.148 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:30:08.148 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.349779+0000", 2024-04-20T17:30:08.148 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.5079999999999994e-06 2024-04-20T17:30:08.148 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.148 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.148 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:30:08.148 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.349802+0000", 2024-04-20T17:30:08.149 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3269999999999999e-05 2024-04-20T17:30:08.149 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.149 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.149 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:30:08.149 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.349813+0000", 2024-04-20T17:30:08.149 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1355999999999999e-05 2024-04-20T17:30:08.149 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.149 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.149 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:30:08.149 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.350246+0000", 2024-04-20T17:30:08.150 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00043303299999999999 2024-04-20T17:30:08.150 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.150 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.150 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:30:08.150 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.350350+0000", 2024-04-20T17:30:08.150 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000103897 2024-04-20T17:30:08.150 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.150 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.150 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:30:08.150 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.350500+0000", 2024-04-20T17:30:08.150 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000149448 2024-04-20T17:30:08.150 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.150 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.150 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:30:08.151 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.354776+0000", 2024-04-20T17:30:08.151 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0042764700000000001 2024-04-20T17:30:08.151 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.151 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.151 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:08.151 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.397160+0000", 2024-04-20T17:30:08.151 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.042383781000000002 2024-04-20T17:30:08.151 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.151 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.151 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:30:08.151 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.397219+0000", 2024-04-20T17:30:08.151 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9598000000000001e-05 2024-04-20T17:30:08.151 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.152 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.152 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:30:08.152 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.397299+0000", 2024-04-20T17:30:08.152 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9939999999999997e-05 2024-04-20T17:30:08.152 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.152 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:30:08.152 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.152 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.153 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.153 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:303988 4.4 4:28c45dfc:::benchmark_data_smithi171_122772_object133796:head [delete] snapc 0=[] ondisk+write+known_if_redirected e527)", 2024-04-20T17:30:08.153 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:28:45.357626+0000", 2024-04-20T17:30:08.153 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 82.301053809999999, 2024-04-20T17:30:08.153 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.043019210000000002, 2024-04-20T17:30:08.153 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:30:08.153 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:30:08.153 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:08.153 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:30:08.153 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:30:08.153 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:30:08.153 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 303988 2024-04-20T17:30:08.153 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.153 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:30:08.154 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.154 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:30:08.154 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.357626+0000", 2024-04-20T17:30:08.154 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.154 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.154 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.154 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:30:08.154 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.357626+0000", 2024-04-20T17:30:08.154 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:30:08.154 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.154 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.154 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:30:08.154 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.357627+0000", 2024-04-20T17:30:08.154 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3e-07 2024-04-20T17:30:08.154 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.155 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.155 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:30:08.155 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.357631+0000", 2024-04-20T17:30:08.155 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2440000000000004e-06 2024-04-20T17:30:08.155 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.155 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.155 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:30:08.155 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.357647+0000", 2024-04-20T17:30:08.155 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5245999999999999e-05 2024-04-20T17:30:08.155 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.155 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.155 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:30:08.155 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.357653+0000", 2024-04-20T17:30:08.155 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4710000000000004e-06 2024-04-20T17:30:08.155 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.156 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.156 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:30:08.156 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.362828+0000", 2024-04-20T17:30:08.156 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0051751949999999996 2024-04-20T17:30:08.156 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.156 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.156 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:30:08.156 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.362928+0000", 2024-04-20T17:30:08.156 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000100107 2024-04-20T17:30:08.156 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.156 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.156 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:30:08.156 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.363003+0000", 2024-04-20T17:30:08.156 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4346999999999998e-05 2024-04-20T17:30:08.156 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.157 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.157 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:30:08.157 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.373664+0000", 2024-04-20T17:30:08.157 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.01066156 2024-04-20T17:30:08.157 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.157 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.157 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:08.157 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.400524+0000", 2024-04-20T17:30:08.157 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.026859554000000001 2024-04-20T17:30:08.157 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.157 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.157 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:30:08.157 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.400589+0000", 2024-04-20T17:30:08.157 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5586999999999994e-05 2024-04-20T17:30:08.157 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:30:08.158 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:30:08.158 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:30:08.158 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:28:45.400646+0000", 2024-04-20T17:30:08.158 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6168999999999997e-05 2024-04-20T17:30:08.158 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.158 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:30:08.158 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.158 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:30:08.158 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:30:08.158 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:30:08.158 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:30:08.191 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:08.212 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:30:08.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:08.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:30:08.225 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 28 16 203 187 0.417353 0 - 0.0022051 2024-04-20T17:30:08.316 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:30:08.348 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:08.370 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:30:08.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:08.373 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:30:08.474 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:30:08.504 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:08 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:30:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:08 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:30:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:08 smithi106 ceph-mon[44423]: osdmap e559: 12 total, 11 up, 11 in 2024-04-20T17:30:08.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:08 smithi106 ceph-mon[44423]: from='osd.3 v1:172.21.15.5:6813/2272824046' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-20T17:30:08.525 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:30:08.527 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:08 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:30:08.527 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:08 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:30:08.527 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:08 smithi171 ceph-mon[44189]: osdmap e559: 12 total, 11 up, 11 in 2024-04-20T17:30:08.527 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:08 smithi171 ceph-mon[44189]: from='osd.3 v1:172.21.15.5:6813/2272824046' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-20T17:30:08.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:08.528 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:30:08.629 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:30:08.657 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:30:08 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-3[233833]: 2024-04-20T17:30:08.265+0000 7fee09038700 -1 osd.3 538 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T17:30:08.658 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:08 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:30:08.658 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:08 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:30:08.658 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:08 smithi005 ceph-mon[39562]: osdmap e559: 12 total, 11 up, 11 in 2024-04-20T17:30:08.658 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:08 smithi005 ceph-mon[39562]: from='osd.3 v1:172.21.15.5:6813/2272824046' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-20T17:30:08.661 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:08.681 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:30:08.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:08.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:30:08.786 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:30:08.814 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:08.833 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:30:08.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:08.836 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:30:08.937 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:30:08.969 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:09.225 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 29 16 203 187 0.402961 0 - 0.0022051 2024-04-20T17:30:09.516 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-20T17:30:09.547 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:09 smithi106 ceph-mon[44423]: from='osd.3 v1:172.21.15.5:6813/2272824046' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-20T17:30:09.547 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:09 smithi106 ceph-mon[44423]: osdmap e560: 12 total, 11 up, 11 in 2024-04-20T17:30:09.547 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:09 smithi106 ceph-mon[44423]: from='osd.3 v1:172.21.15.5:6813/2272824046' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:30:09.547 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:09 smithi106 ceph-mon[44423]: pgmap v1471: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail 2024-04-20T17:30:09.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:09 smithi171 ceph-mon[44189]: from='osd.3 v1:172.21.15.5:6813/2272824046' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-20T17:30:09.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:09 smithi171 ceph-mon[44189]: osdmap e560: 12 total, 11 up, 11 in 2024-04-20T17:30:09.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:09 smithi171 ceph-mon[44189]: from='osd.3 v1:172.21.15.5:6813/2272824046' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:30:09.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:09 smithi171 ceph-mon[44189]: pgmap v1471: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail 2024-04-20T17:30:09.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:09 smithi005 ceph-mon[39562]: from='osd.3 v1:172.21.15.5:6813/2272824046' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-20T17:30:09.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:09 smithi005 ceph-mon[39562]: osdmap e560: 12 total, 11 up, 11 in 2024-04-20T17:30:09.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:09 smithi005 ceph-mon[39562]: from='osd.3 v1:172.21.15.5:6813/2272824046' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:30:09.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:09 smithi005 ceph-mon[39562]: pgmap v1471: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail 2024-04-20T17:30:09.718 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:30:09.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:09.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:30:09.738 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:30:09.738 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:30:09.738 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-04-20T17:30:09.738 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:30:09.798 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:30:09.823 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:30:09.853 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:09.954 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:30:09.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:09.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:30:10.058 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:30:10.087 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:10.106 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:30:10.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:10.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:30:10.210 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:30:10.224 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 30 16 203 187 0.38953 0 - 0.0022051 2024-04-20T17:30:10.242 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:10.263 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:30:10.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:10.266 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:30:10.338 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:10 smithi106 ceph-mon[44423]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T17:30:10.338 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:10 smithi106 ceph-mon[44423]: osd.3 v1:172.21.15.5:6813/2272824046 boot 2024-04-20T17:30:10.338 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:10 smithi106 ceph-mon[44423]: osdmap e561: 12 total, 12 up, 11 in 2024-04-20T17:30:10.338 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:10 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T17:30:10.367 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:30:10.389 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:30:10.396 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:10.402 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:30:10.424 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:30:10.443 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:30:10.463 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:30:10.483 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:30:10.503 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:30:10.523 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:30:10.545 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:30:10.571 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:30:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:10 smithi171 ceph-mon[44189]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T17:30:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:10 smithi171 ceph-mon[44189]: osd.3 v1:172.21.15.5:6813/2272824046 boot 2024-04-20T17:30:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:10 smithi171 ceph-mon[44189]: osdmap e561: 12 total, 12 up, 11 in 2024-04-20T17:30:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:10 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T17:30:10.609 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:30:10.643 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:30:10.644 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:30:10.646 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:10 smithi005 ceph-mon[39562]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T17:30:10.646 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:10 smithi005 ceph-mon[39562]: osd.3 v1:172.21.15.5:6813/2272824046 boot 2024-04-20T17:30:10.646 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:10 smithi005 ceph-mon[39562]: osdmap e561: 12 total, 12 up, 11 in 2024-04-20T17:30:10.646 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:10 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-20T17:30:10.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:10.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:30:10.680 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:30:10.694 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:30:10.694 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:30:10.694 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:30:10.694 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-04-20T17:30:10.694 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:30:10.694 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:30:10.694 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:30:10.695 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:30:10.695 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:30:10.695 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:30:10.695 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:30:10.695 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:30:10.749 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:30:10.795 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:11.225 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 31 15 4054 4039 8.14204 8.025 0.00249246 0.122812 2024-04-20T17:30:11.360 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:30:11.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:11.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:30:11.391 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 0 1 2024-04-20T17:30:11.465 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:30:11.500 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:11.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:11 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:30:11.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:11 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:30:11.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:11 smithi005 ceph-mon[39562]: osdmap e562: 12 total, 12 up, 11 in 2024-04-20T17:30:11.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:11 smithi005 ceph-mon[39562]: pgmap v1474: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail 2024-04-20T17:30:11.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:11 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4103128900' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:30:11.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.538 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.539 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.539 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.539 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.539 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.539 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.539 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.539 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:11 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:30:11.539 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:11 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:30:11.539 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.539 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:11 smithi005 ceph-mon[39562]: osdmap e563: 12 total, 12 up, 11 in 2024-04-20T17:30:11.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:11 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:30:11.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:11 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:30:11.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:11 smithi171 ceph-mon[44189]: osdmap e562: 12 total, 12 up, 11 in 2024-04-20T17:30:11.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:11 smithi171 ceph-mon[44189]: pgmap v1474: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail 2024-04-20T17:30:11.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:11 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4103128900' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:30:11.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:11 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:30:11.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:11 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:30:11.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:11 smithi171 ceph-mon[44189]: osdmap e563: 12 total, 12 up, 11 in 2024-04-20T17:30:11.694 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.694 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:11 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:30:11.694 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:11 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:30:11.694 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:11 smithi106 ceph-mon[44423]: osdmap e562: 12 total, 12 up, 11 in 2024-04-20T17:30:11.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:11 smithi106 ceph-mon[44423]: pgmap v1474: 17 pgs: 1 undersized+peered, 16 active+clean; 12 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail 2024-04-20T17:30:11.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:11 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4103128900' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:30:11.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:11 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:30:11.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:11 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:30:11.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:11.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:11 smithi106 ceph-mon[44423]: osdmap e563: 12 total, 12 up, 11 in 2024-04-20T17:30:11.704 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:30:11.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:11.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:30:11.809 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:30:11.846 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:12.225 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 32 16 9418 9402 18.3608 335.188 0.00336627 0.054459 2024-04-20T17:30:12.448 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:30:12.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:12.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:12.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:12 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:30:12.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:12 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:30:12.567 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:12.567 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:12.567 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:12 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:30:12.567 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:12 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:30:12.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:12.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:12.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:12 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:30:12.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:12 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:30:12.710 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:30:12.710 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:30:13.225 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 33 16 14164 14148 26.7919 296.625 0.00238811 0.0373223 2024-04-20T17:30:13.304 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:30:13.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:13.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:30:13.407 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:30:13.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:13 smithi005 ceph-mon[39562]: pgmap v1476: 17 pgs: 17 active+clean; 84 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 18 MiB/s wr, 283 op/s 2024-04-20T17:30:13.444 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:13.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:13 smithi171 ceph-mon[44189]: pgmap v1476: 17 pgs: 17 active+clean; 84 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 18 MiB/s wr, 283 op/s 2024-04-20T17:30:13.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:13 smithi106 ceph-mon[44423]: pgmap v1476: 17 pgs: 17 active+clean; 84 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 18 MiB/s wr, 283 op/s 2024-04-20T17:30:13.856 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-20T17:30:13.896 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:30:13.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:13.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:30:14.000 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:30:14.038 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:14.062 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:30:14.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:14.065 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:30:14.167 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:30:14.202 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:14.225 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 34 16 18813 18797 34.5488 290.562 0.00224118 0.0289399 2024-04-20T17:30:14.227 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:30:14.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:14.230 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:30:14.260 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:30:14.331 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:30:14.370 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:14.396 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:30:14.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:14.400 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:30:14.420 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:30:14.501 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:30:14.538 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:14.562 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:30:14.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:14.566 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:30:14.667 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:30:14.718 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:15.184 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:30:15.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:15.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:30:15.225 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 35 16 22724 22708 40.5447 244.438 0.00309416 0.0246621 2024-04-20T17:30:15.289 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:30:15.318 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:30:15.352 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:15.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:15 smithi005 ceph-mon[39562]: pgmap v1477: 17 pgs: 17 active+clean; 599 MiB data, 3.7 GiB used, 980 GiB / 983 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-04-20T17:30:15.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:15 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3447516574' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:30:15.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:15 smithi106 ceph-mon[44423]: pgmap v1477: 17 pgs: 17 active+clean; 599 MiB data, 3.7 GiB used, 980 GiB / 983 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-04-20T17:30:15.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:15 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3447516574' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:30:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:15 smithi171 ceph-mon[44189]: pgmap v1477: 17 pgs: 17 active+clean; 599 MiB data, 3.7 GiB used, 980 GiB / 983 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-04-20T17:30:15.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:15 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3447516574' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:30:16.202 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:30:16.212 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:30:16.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:16.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:30:16.225 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 36 16 26568 26552 46.0913 240.25 0.00281408 0.0216938 2024-04-20T17:30:16.260 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.0 to 1 (10000) 2024-04-20T17:30:16.285 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:30:16.285 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:30:16.320 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:30:16.368 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:16.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:16 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:30:16.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:16 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:30:16.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:16 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3447516574' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:30:16.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:16 smithi005 ceph-mon[39562]: osdmap e564: 12 total, 12 up, 11 in 2024-04-20T17:30:16.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:16 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1686450559' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:30:16.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:16 smithi005 ceph-mon[39562]: from='client.26518 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:30:16.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:16 smithi005 ceph-mon[39562]: from='client.26518 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-20T17:30:16.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:16 smithi005 ceph-mon[39562]: osdmap e565: 12 total, 12 up, 11 in 2024-04-20T17:30:16.477 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:30:16.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:16.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:30:16.582 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:30:16.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:16 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:30:16.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:16 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:30:16.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:16 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3447516574' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:30:16.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:16 smithi171 ceph-mon[44189]: osdmap e564: 12 total, 12 up, 11 in 2024-04-20T17:30:16.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:16 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1686450559' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:30:16.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:16 smithi171 ceph-mon[44189]: from='client.26518 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:30:16.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:16 smithi171 ceph-mon[44189]: from='client.26518 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-20T17:30:16.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:16 smithi171 ceph-mon[44189]: osdmap e565: 12 total, 12 up, 11 in 2024-04-20T17:30:16.620 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:16.647 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:30:16.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:16.651 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:30:16.656 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:16 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:30:16.656 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:16 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:30:16.656 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:16 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3447516574' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:30:16.656 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:16 smithi106 ceph-mon[44423]: osdmap e564: 12 total, 12 up, 11 in 2024-04-20T17:30:16.656 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:16 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1686450559' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:30:16.656 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:16 smithi106 ceph-mon[44423]: from='client.26518 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:30:16.656 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:16 smithi106 ceph-mon[44423]: from='client.26518 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-20T17:30:16.656 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:16 smithi106 ceph-mon[44423]: osdmap e565: 12 total, 12 up, 11 in 2024-04-20T17:30:16.751 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:30:16.792 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:17.046 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:30:17.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:17.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:30:17.054 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-20T17:30:17.152 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:30:17.198 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:17.225 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 37 16 30267 30251 51.0932 231.188 0.00275126 0.0195696 2024-04-20T17:30:17.226 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:30:17.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:17.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:30:17.298 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:17 smithi106 ceph-mon[44423]: pgmap v1480: 17 pgs: 17 active+clean; 1.3 GiB data, 4.9 GiB used, 979 GiB / 983 GiB avail; 219 MiB/s wr, 3.51k op/s 2024-04-20T17:30:17.331 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:30:17.372 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:17.400 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:30:17.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:17.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:30:17.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:17 smithi005 ceph-mon[39562]: pgmap v1480: 17 pgs: 17 active+clean; 1.3 GiB data, 4.9 GiB used, 979 GiB / 983 GiB avail; 219 MiB/s wr, 3.51k op/s 2024-04-20T17:30:17.494 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:30:17.504 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:30:17.548 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:17.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:17 smithi171 ceph-mon[44189]: pgmap v1480: 17 pgs: 17 active+clean; 1.3 GiB data, 4.9 GiB used, 979 GiB / 983 GiB avail; 219 MiB/s wr, 3.51k op/s 2024-04-20T17:30:17.598 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:30:17.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:17.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:30:17.651 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 1 1 2024-04-20T17:30:17.707 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:30:17.745 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:17.773 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:30:17.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:17.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:30:17.879 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:30:17.935 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:18.152 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:30:18.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:18.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:30:18.225 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 38 16 33659 33643 55.3269 212 0.00540094 0.0180727 2024-04-20T17:30:18.228 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:30:18.241 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:30:18.257 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:30:18.290 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:18.318 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:30:18.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:18.322 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:30:18.423 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:30:18.457 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:18.485 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:30:18.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:18.488 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:30:18.589 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:30:18.629 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:18.655 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:30:18.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:18.659 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:30:18.760 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:30:18.818 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:19.023 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:30:19.067 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:30:19.109 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:30:19.157 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:30:19.193 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:30:19.225 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 39 16 36685 36669 58.757 189.125 0.00384849 0.0170173 2024-04-20T17:30:19.231 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:30:19.265 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:30:19.297 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:30:19.344 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:30:19.377 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:30:19.414 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:30:19.453 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:30:19.460 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:30:19.460 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:30:19.460 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:30:19.460 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:30:19.460 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:30:19.460 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:30:19.460 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:30:19.460 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:30:19.460 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:30:19.460 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:30:19.460 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:30:19.460 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:30:19.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:19 smithi171 ceph-mon[44189]: pgmap v1481: 17 pgs: 17 active+clean; 1.4 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 188 MiB/s wr, 3.01k op/s 2024-04-20T17:30:19.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:19 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1451236896' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:30:19.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:19.611 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:30:19.611 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-04-20T17:30:19.611 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-04-20T17:30:19.611 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:30:19.611 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.611 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245460 3.d 3:bba062b0:::benchmark_data_smithi171_110765_object102245:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:30:19.611 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103564+0000", 2024-04-20T17:30:19.611 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 406.504603035, 2024-04-20T17:30:19.611 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.086211963000000003, 2024-04-20T17:30:19.612 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:30:19.612 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:30:19.612 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:19.612 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:30:19.612 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:30:19.612 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:30:19.612 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245460 2024-04-20T17:30:19.612 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.612 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:30:19.612 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.612 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:30:19.612 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103564+0000", 2024-04-20T17:30:19.613 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.613 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.613 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.613 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:30:19.613 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103564+0000", 2024-04-20T17:30:19.613 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.613 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.613 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.613 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:30:19.613 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103565+0000", 2024-04-20T17:30:19.613 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2999999999999999e-07 2024-04-20T17:30:19.613 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.613 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.613 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:30:19.614 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103568+0000", 2024-04-20T17:30:19.614 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9699999999999999e-06 2024-04-20T17:30:19.614 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.614 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.614 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:30:19.614 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103573+0000", 2024-04-20T17:30:19.614 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4220000000000002e-06 2024-04-20T17:30:19.614 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.614 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.614 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:30:19.614 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103577+0000", 2024-04-20T17:30:19.614 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.34e-06 2024-04-20T17:30:19.615 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.615 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.615 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.615 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103883+0000", 2024-04-20T17:30:19.615 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00030601700000000001 2024-04-20T17:30:19.615 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.615 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.615 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:30:19.615 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103890+0000", 2024-04-20T17:30:19.615 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4189999999999996e-06 2024-04-20T17:30:19.616 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.616 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.616 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.616 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.157260+0000", 2024-04-20T17:30:19.616 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.053369403000000003 2024-04-20T17:30:19.616 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.616 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.616 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:30:19.616 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.157381+0000", 2024-04-20T17:30:19.616 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000120776 2024-04-20T17:30:19.616 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.616 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.616 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:30:19.617 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.157507+0000", 2024-04-20T17:30:19.617 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012656000000000001 2024-04-20T17:30:19.617 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.617 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.617 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:30:19.617 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.165767+0000", 2024-04-20T17:30:19.617 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0082595310000000005 2024-04-20T17:30:19.617 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.617 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.617 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:19.617 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189715+0000", 2024-04-20T17:30:19.617 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.023948542999999999 2024-04-20T17:30:19.617 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.617 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.618 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:30:19.618 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189746+0000", 2024-04-20T17:30:19.618 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0939000000000003e-05 2024-04-20T17:30:19.618 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.618 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.618 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:30:19.618 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189776+0000", 2024-04-20T17:30:19.618 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0113000000000001e-05 2024-04-20T17:30:19.618 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.618 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:30:19.618 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.618 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.618 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.619 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245510 3.d 3:b21de460:::benchmark_data_smithi171_110765_object102295:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:30:19.619 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103589+0000", 2024-04-20T17:30:19.619 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 406.50457841100001, 2024-04-20T17:30:19.619 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.086330094999999996, 2024-04-20T17:30:19.619 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:30:19.619 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:30:19.619 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:19.619 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:30:19.619 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:30:19.619 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:30:19.620 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245510 2024-04-20T17:30:19.620 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.620 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:30:19.620 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.620 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:30:19.620 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103589+0000", 2024-04-20T17:30:19.620 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.620 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.620 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.620 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:30:19.620 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103589+0000", 2024-04-20T17:30:19.620 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.620 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.621 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.621 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:30:19.621 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103590+0000", 2024-04-20T17:30:19.621 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5400000000000001e-07 2024-04-20T17:30:19.621 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.621 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.621 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:30:19.621 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103592+0000", 2024-04-20T17:30:19.621 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.83e-06 2024-04-20T17:30:19.621 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.621 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.621 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:30:19.621 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103600+0000", 2024-04-20T17:30:19.621 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3139999999999998e-06 2024-04-20T17:30:19.621 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.622 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.622 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:30:19.622 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103605+0000", 2024-04-20T17:30:19.622 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.429e-06 2024-04-20T17:30:19.622 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.622 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.622 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.622 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103931+0000", 2024-04-20T17:30:19.622 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00032633599999999999 2024-04-20T17:30:19.622 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.622 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.622 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:30:19.622 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103944+0000", 2024-04-20T17:30:19.623 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.307e-05 2024-04-20T17:30:19.623 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.623 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.623 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.623 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.159475+0000", 2024-04-20T17:30:19.623 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.055531006000000001 2024-04-20T17:30:19.623 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.623 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.623 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:30:19.623 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.159584+0000", 2024-04-20T17:30:19.623 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010849399999999999 2024-04-20T17:30:19.623 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.623 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.623 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:30:19.624 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.159707+0000", 2024-04-20T17:30:19.624 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012344499999999999 2024-04-20T17:30:19.624 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.624 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.624 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:30:19.624 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.171026+0000", 2024-04-20T17:30:19.624 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.01131892 2024-04-20T17:30:19.624 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.624 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.624 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:19.624 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189843+0000", 2024-04-20T17:30:19.624 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.018816330999999999 2024-04-20T17:30:19.624 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.624 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.625 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:30:19.625 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189889+0000", 2024-04-20T17:30:19.625 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6647999999999998e-05 2024-04-20T17:30:19.625 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.625 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.625 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:30:19.625 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189919+0000", 2024-04-20T17:30:19.625 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9717999999999999e-05 2024-04-20T17:30:19.625 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.625 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:30:19.625 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.625 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.625 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.625 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245523 3.d 3:bc29c29a:::benchmark_data_smithi171_110765_object102308:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:30:19.625 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103692+0000", 2024-04-20T17:30:19.626 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 406.50447513699999, 2024-04-20T17:30:19.626 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.086309881000000005, 2024-04-20T17:30:19.626 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:30:19.626 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:30:19.626 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:19.626 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:30:19.626 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:30:19.626 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:30:19.626 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245523 2024-04-20T17:30:19.626 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.626 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:30:19.626 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.626 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:30:19.627 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103692+0000", 2024-04-20T17:30:19.627 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.627 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.627 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.627 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:30:19.627 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103692+0000", 2024-04-20T17:30:19.627 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.627 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.627 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.627 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:30:19.627 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103693+0000", 2024-04-20T17:30:19.627 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1710000000000001e-06 2024-04-20T17:30:19.627 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.627 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.628 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:30:19.628 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103699+0000", 2024-04-20T17:30:19.628 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3900000000000001e-06 2024-04-20T17:30:19.628 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.628 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.628 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:30:19.628 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103709+0000", 2024-04-20T17:30:19.628 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9119999999999993e-06 2024-04-20T17:30:19.628 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.628 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.628 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:30:19.628 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103716+0000", 2024-04-20T17:30:19.629 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0260000000000002e-06 2024-04-20T17:30:19.629 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.629 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.629 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.629 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104008+0000", 2024-04-20T17:30:19.629 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00029273899999999998 2024-04-20T17:30:19.629 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.629 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.629 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:30:19.629 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104018+0000", 2024-04-20T17:30:19.629 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5540000000000003e-06 2024-04-20T17:30:19.629 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.629 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.630 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.630 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.161784+0000", 2024-04-20T17:30:19.630 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.057766296000000002 2024-04-20T17:30:19.630 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.630 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.630 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:30:19.630 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.161875+0000", 2024-04-20T17:30:19.630 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0490999999999998e-05 2024-04-20T17:30:19.630 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.630 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.630 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:30:19.630 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.161953+0000", 2024-04-20T17:30:19.630 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8370999999999999e-05 2024-04-20T17:30:19.631 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.631 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.631 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:30:19.631 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.171034+0000", 2024-04-20T17:30:19.631 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0090805629999999998 2024-04-20T17:30:19.631 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.631 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.631 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:19.631 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189952+0000", 2024-04-20T17:30:19.631 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.018918249000000002 2024-04-20T17:30:19.631 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.631 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.632 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:30:19.632 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189980+0000", 2024-04-20T17:30:19.632 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8469000000000001e-05 2024-04-20T17:30:19.632 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.632 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.632 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:30:19.632 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.190002+0000", 2024-04-20T17:30:19.632 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1650000000000001e-05 2024-04-20T17:30:19.632 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.632 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:30:19.632 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.632 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.632 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.633 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245527 3.d 3:bef30612:::benchmark_data_smithi171_110765_object102312:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:30:19.633 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103727+0000", 2024-04-20T17:30:19.633 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 406.50444076899998, 2024-04-20T17:30:19.633 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.090154580999999998, 2024-04-20T17:30:19.633 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:30:19.633 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:30:19.633 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:19.633 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:30:19.633 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:30:19.633 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:30:19.633 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245527 2024-04-20T17:30:19.633 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.633 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:30:19.634 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.634 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:30:19.634 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103727+0000", 2024-04-20T17:30:19.634 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.634 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.634 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.634 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:30:19.634 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103727+0000", 2024-04-20T17:30:19.634 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.634 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.634 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.634 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:30:19.635 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103727+0000", 2024-04-20T17:30:19.635 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6400000000000001e-07 2024-04-20T17:30:19.635 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.635 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.635 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:30:19.635 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103731+0000", 2024-04-20T17:30:19.635 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7589999999999999e-06 2024-04-20T17:30:19.635 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.635 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.635 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:30:19.635 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103737+0000", 2024-04-20T17:30:19.635 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.207e-06 2024-04-20T17:30:19.636 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.636 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.636 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:30:19.636 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103743+0000", 2024-04-20T17:30:19.636 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9209999999999997e-06 2024-04-20T17:30:19.636 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.636 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.636 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.636 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104112+0000", 2024-04-20T17:30:19.636 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00036842100000000002 2024-04-20T17:30:19.636 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.636 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.636 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:30:19.636 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104116+0000", 2024-04-20T17:30:19.636 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2930000000000002e-06 2024-04-20T17:30:19.637 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.637 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.637 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.637 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.163799+0000", 2024-04-20T17:30:19.637 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.059683369999999999 2024-04-20T17:30:19.637 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.637 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.637 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:30:19.637 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.163872+0000", 2024-04-20T17:30:19.637 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2502999999999993e-05 2024-04-20T17:30:19.637 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.637 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.637 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:30:19.637 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.163953+0000", 2024-04-20T17:30:19.637 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0758000000000002e-05 2024-04-20T17:30:19.638 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.638 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.638 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:30:19.638 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.176271+0000", 2024-04-20T17:30:19.638 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012318610000000001 2024-04-20T17:30:19.638 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.638 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.638 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:19.638 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.193822+0000", 2024-04-20T17:30:19.638 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.017550603000000001 2024-04-20T17:30:19.638 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.638 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.638 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:30:19.638 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.193856+0000", 2024-04-20T17:30:19.639 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4239000000000002e-05 2024-04-20T17:30:19.639 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.639 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.639 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:30:19.639 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.193881+0000", 2024-04-20T17:30:19.639 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5133000000000001e-05 2024-04-20T17:30:19.639 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.639 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:30:19.639 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.639 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.639 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.639 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245531 3.d 3:ba69417f:::benchmark_data_smithi171_110765_object102316:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:30:19.639 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103783+0000", 2024-04-20T17:30:19.639 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 406.50438458899998, 2024-04-20T17:30:19.640 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.090206175, 2024-04-20T17:30:19.640 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:30:19.640 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:30:19.640 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:19.640 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:30:19.640 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:30:19.640 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:30:19.640 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245531 2024-04-20T17:30:19.640 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.640 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:30:19.640 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.640 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:30:19.640 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103783+0000", 2024-04-20T17:30:19.640 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.641 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.641 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.641 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:30:19.641 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103783+0000", 2024-04-20T17:30:19.641 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.641 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.641 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.641 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:30:19.641 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103785+0000", 2024-04-20T17:30:19.641 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.809e-06 2024-04-20T17:30:19.641 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.641 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.641 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:30:19.641 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103802+0000", 2024-04-20T17:30:19.642 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7075000000000002e-05 2024-04-20T17:30:19.642 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.642 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.642 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:30:19.642 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103815+0000", 2024-04-20T17:30:19.642 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3019e-05 2024-04-20T17:30:19.642 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.642 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.642 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:30:19.642 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103825+0000", 2024-04-20T17:30:19.642 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8309999999999996e-06 2024-04-20T17:30:19.642 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.642 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.642 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.643 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104133+0000", 2024-04-20T17:30:19.643 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00030879900000000001 2024-04-20T17:30:19.643 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.643 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.643 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:30:19.643 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104136+0000", 2024-04-20T17:30:19.643 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0910000000000001e-06 2024-04-20T17:30:19.643 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.643 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.643 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.643 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.165809+0000", 2024-04-20T17:30:19.643 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.061672957 2024-04-20T17:30:19.643 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.644 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.644 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:30:19.644 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.165889+0000", 2024-04-20T17:30:19.644 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9381000000000005e-05 2024-04-20T17:30:19.644 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.644 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.644 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:30:19.644 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.165967+0000", 2024-04-20T17:30:19.644 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7849999999999995e-05 2024-04-20T17:30:19.644 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.644 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.644 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:30:19.644 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.176279+0000", 2024-04-20T17:30:19.644 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010312702999999999 2024-04-20T17:30:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.193918+0000", 2024-04-20T17:30:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.017638628999999999 2024-04-20T17:30:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:30:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.193960+0000", 2024-04-20T17:30:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1746000000000003e-05 2024-04-20T17:30:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:30:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.193989+0000", 2024-04-20T17:30:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9285000000000002e-05 2024-04-20T17:30:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:30:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245546 3.d 3:b08c00b6:::benchmark_data_smithi171_110765_object102331:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:30:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103839+0000", 2024-04-20T17:30:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 406.50432787800003, 2024-04-20T17:30:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.097118706999999999, 2024-04-20T17:30:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:30:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:30:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:30:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:30:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:30:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245546 2024-04-20T17:30:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:30:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:30:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103839+0000", 2024-04-20T17:30:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:30:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103839+0000", 2024-04-20T17:30:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.648 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.648 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.648 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:30:19.648 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103840+0000", 2024-04-20T17:30:19.648 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1800000000000004e-07 2024-04-20T17:30:19.648 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.648 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.648 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:30:19.648 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103845+0000", 2024-04-20T17:30:19.648 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3000000000000003e-06 2024-04-20T17:30:19.648 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.648 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.648 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:30:19.648 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103853+0000", 2024-04-20T17:30:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9799999999999998e-06 2024-04-20T17:30:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:30:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103859+0000", 2024-04-20T17:30:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7020000000000004e-06 2024-04-20T17:30:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104182+0000", 2024-04-20T17:30:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00032271300000000003 2024-04-20T17:30:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:30:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104201+0000", 2024-04-20T17:30:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8547999999999999e-05 2024-04-20T17:30:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.167891+0000", 2024-04-20T17:30:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.063690467000000001 2024-04-20T17:30:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:30:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.168022+0000", 2024-04-20T17:30:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013114499999999999 2024-04-20T17:30:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:30:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.168147+0000", 2024-04-20T17:30:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012454299999999999 2024-04-20T17:30:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:30:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.181127+0000", 2024-04-20T17:30:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012980302000000001 2024-04-20T17:30:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.200861+0000", 2024-04-20T17:30:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.019734266 2024-04-20T17:30:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:30:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.200910+0000", 2024-04-20T17:30:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8294999999999998e-05 2024-04-20T17:30:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:30:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.200958+0000", 2024-04-20T17:30:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8528e-05 2024-04-20T17:30:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:30:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245552 3.d 3:b5dda02d:::benchmark_data_smithi171_110765_object102337:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:30:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103886+0000", 2024-04-20T17:30:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 406.50428102900003, 2024-04-20T17:30:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.097162344999999997, 2024-04-20T17:30:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:30:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:30:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:30:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:30:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:30:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245552 2024-04-20T17:30:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:30:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:30:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103886+0000", 2024-04-20T17:30:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.654 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:30:19.654 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103886+0000", 2024-04-20T17:30:19.654 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.654 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.654 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.654 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:30:19.654 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103887+0000", 2024-04-20T17:30:19.654 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2799999999999995e-07 2024-04-20T17:30:19.654 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.654 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.654 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:30:19.654 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103890+0000", 2024-04-20T17:30:19.654 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.084e-06 2024-04-20T17:30:19.655 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.655 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.655 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:30:19.655 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103896+0000", 2024-04-20T17:30:19.655 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3750000000000002e-06 2024-04-20T17:30:19.655 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.655 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.655 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:30:19.655 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103900+0000", 2024-04-20T17:30:19.655 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1679999999999997e-06 2024-04-20T17:30:19.655 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.655 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.655 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104230+0000", 2024-04-20T17:30:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00033005300000000002 2024-04-20T17:30:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:30:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104233+0000", 2024-04-20T17:30:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2260000000000001e-06 2024-04-20T17:30:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.171080+0000", 2024-04-20T17:30:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.066846953000000001 2024-04-20T17:30:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:30:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.171210+0000", 2024-04-20T17:30:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000130376 2024-04-20T17:30:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:30:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.171343+0000", 2024-04-20T17:30:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013265799999999999 2024-04-20T17:30:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:30:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.181134+0000", 2024-04-20T17:30:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0097907150000000002 2024-04-20T17:30:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.200992+0000", 2024-04-20T17:30:19.658 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.019858058000000001 2024-04-20T17:30:19.658 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.658 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.658 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:30:19.658 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.201024+0000", 2024-04-20T17:30:19.658 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1914e-05 2024-04-20T17:30:19.658 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.658 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.658 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:30:19.658 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.201049+0000", 2024-04-20T17:30:19.658 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5037000000000001e-05 2024-04-20T17:30:19.658 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.658 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:30:19.658 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.659 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.659 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.659 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245601 3.d 3:be8206df:::benchmark_data_smithi171_110765_object102386:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:30:19.659 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103906+0000", 2024-04-20T17:30:19.659 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 406.50426126299999, 2024-04-20T17:30:19.659 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.10597208700000001, 2024-04-20T17:30:19.659 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:30:19.659 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:30:19.659 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:19.659 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:30:19.659 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:30:19.659 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:30:19.659 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245601 2024-04-20T17:30:19.660 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.660 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:30:19.660 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.660 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:30:19.660 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103906+0000", 2024-04-20T17:30:19.660 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.660 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.660 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.660 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:30:19.660 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103906+0000", 2024-04-20T17:30:19.660 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.660 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.660 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.660 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:30:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103906+0000", 2024-04-20T17:30:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6100000000000002e-07 2024-04-20T17:30:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:30:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103908+0000", 2024-04-20T17:30:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7439999999999999e-06 2024-04-20T17:30:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:30:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103911+0000", 2024-04-20T17:30:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.954e-06 2024-04-20T17:30:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:30:19.662 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103915+0000", 2024-04-20T17:30:19.662 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9450000000000003e-06 2024-04-20T17:30:19.662 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.662 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.662 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.662 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.174025+0000", 2024-04-20T17:30:19.662 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.070109677999999995 2024-04-20T17:30:19.662 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.662 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.662 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:30:19.662 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.174130+0000", 2024-04-20T17:30:19.662 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010515599999999999 2024-04-20T17:30:19.662 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.662 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.663 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:30:19.663 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.174204+0000", 2024-04-20T17:30:19.663 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4097000000000006e-05 2024-04-20T17:30:19.663 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.663 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.663 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:30:19.663 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.181429+0000", 2024-04-20T17:30:19.663 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0072249250000000001 2024-04-20T17:30:19.663 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.663 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.663 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:19.663 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.209781+0000", 2024-04-20T17:30:19.663 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.028351694 2024-04-20T17:30:19.663 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:30:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.209834+0000", 2024-04-20T17:30:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3309000000000001e-05 2024-04-20T17:30:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:30:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.209878+0000", 2024-04-20T17:30:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4224000000000002e-05 2024-04-20T17:30:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:30:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245611 3.d 3:b89d9251:::benchmark_data_smithi171_110765_object102396:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:30:19.665 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103921+0000", 2024-04-20T17:30:19.665 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 406.504246072, 2024-04-20T17:30:19.665 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.106082866, 2024-04-20T17:30:19.665 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:30:19.665 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:30:19.665 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:19.665 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:30:19.665 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:30:19.665 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:30:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245611 2024-04-20T17:30:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:30:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:30:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103921+0000", 2024-04-20T17:30:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:30:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103921+0000", 2024-04-20T17:30:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:30:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103922+0000", 2024-04-20T17:30:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1999999999999999e-07 2024-04-20T17:30:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:30:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103926+0000", 2024-04-20T17:30:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3630000000000004e-06 2024-04-20T17:30:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:30:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103932+0000", 2024-04-20T17:30:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8259999999999998e-06 2024-04-20T17:30:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:30:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103948+0000", 2024-04-20T17:30:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6240999999999999e-05 2024-04-20T17:30:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.176327+0000", 2024-04-20T17:30:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.072379053999999998 2024-04-20T17:30:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:30:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.176443+0000", 2024-04-20T17:30:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011536600000000001 2024-04-20T17:30:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:30:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.176568+0000", 2024-04-20T17:30:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000125184 2024-04-20T17:30:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:30:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.181432+0000", 2024-04-20T17:30:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0048639720000000003 2024-04-20T17:30:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.209927+0000", 2024-04-20T17:30:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.028494789999999999 2024-04-20T17:30:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:30:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.209969+0000", 2024-04-20T17:30:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1773000000000001e-05 2024-04-20T17:30:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:30:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.210004+0000", 2024-04-20T17:30:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5577000000000002e-05 2024-04-20T17:30:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:30:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245620 3.d 3:be474e21:::benchmark_data_smithi171_110765_object102405:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:30:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103957+0000", 2024-04-20T17:30:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 406.50421007400001, 2024-04-20T17:30:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.112691712, 2024-04-20T17:30:19.671 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:30:19.671 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:30:19.671 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:19.671 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:30:19.671 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:30:19.671 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:30:19.671 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245620 2024-04-20T17:30:19.671 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.671 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:30:19.671 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.671 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:30:19.671 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103957+0000", 2024-04-20T17:30:19.671 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.671 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.671 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.672 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:30:19.672 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103957+0000", 2024-04-20T17:30:19.672 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.672 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.672 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.672 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:30:19.672 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103958+0000", 2024-04-20T17:30:19.672 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1299999999999999e-07 2024-04-20T17:30:19.672 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.672 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.672 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:30:19.672 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103962+0000", 2024-04-20T17:30:19.672 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6119999999999999e-06 2024-04-20T17:30:19.672 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.672 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.672 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:30:19.673 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103967+0000", 2024-04-20T17:30:19.673 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7679999999999997e-06 2024-04-20T17:30:19.673 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.673 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.673 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:30:19.673 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103971+0000", 2024-04-20T17:30:19.673 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6100000000000002e-06 2024-04-20T17:30:19.673 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.673 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.673 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.673 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.178796+0000", 2024-04-20T17:30:19.673 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.074824511999999996 2024-04-20T17:30:19.673 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.673 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.674 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:30:19.674 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.178903+0000", 2024-04-20T17:30:19.674 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000107731 2024-04-20T17:30:19.674 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.674 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.674 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:30:19.674 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.179004+0000", 2024-04-20T17:30:19.674 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000101139 2024-04-20T17:30:19.674 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.674 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.674 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:30:19.674 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.187634+0000", 2024-04-20T17:30:19.675 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0086293210000000006 2024-04-20T17:30:19.675 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.675 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.675 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:19.675 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.216547+0000", 2024-04-20T17:30:19.675 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.028912817 2024-04-20T17:30:19.675 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.675 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.675 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:30:19.675 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.216604+0000", 2024-04-20T17:30:19.675 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7009999999999998e-05 2024-04-20T17:30:19.675 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.675 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.675 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:30:19.676 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.216649+0000", 2024-04-20T17:30:19.676 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5379e-05 2024-04-20T17:30:19.676 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.676 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:30:19.676 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.676 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.676 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.676 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245622 3.d 3:b177e667:::benchmark_data_smithi171_110765_object102407:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:30:19.676 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103975+0000", 2024-04-20T17:30:19.676 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 406.50419196500002, 2024-04-20T17:30:19.676 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.112781889, 2024-04-20T17:30:19.676 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:30:19.676 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:30:19.676 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:19.676 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:30:19.677 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:30:19.677 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:30:19.677 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245622 2024-04-20T17:30:19.677 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.677 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:30:19.677 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.677 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:30:19.677 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103975+0000", 2024-04-20T17:30:19.677 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.677 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.677 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.677 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:30:19.677 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103975+0000", 2024-04-20T17:30:19.677 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.677 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.677 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.677 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:30:19.678 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103976+0000", 2024-04-20T17:30:19.678 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9000000000000002e-07 2024-04-20T17:30:19.678 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.678 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.678 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:30:19.678 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103978+0000", 2024-04-20T17:30:19.678 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.821e-06 2024-04-20T17:30:19.678 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.678 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.678 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:30:19.678 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103981+0000", 2024-04-20T17:30:19.678 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0350000000000002e-06 2024-04-20T17:30:19.678 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.678 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.679 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:30:19.679 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103984+0000", 2024-04-20T17:30:19.679 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0089999999999999e-06 2024-04-20T17:30:19.679 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.679 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.679 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.679 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.181477+0000", 2024-04-20T17:30:19.679 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.077492970999999994 2024-04-20T17:30:19.679 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.679 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.679 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:30:19.679 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.181593+0000", 2024-04-20T17:30:19.679 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000115979 2024-04-20T17:30:19.679 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.679 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.679 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:30:19.679 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.181711+0000", 2024-04-20T17:30:19.680 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000118245 2024-04-20T17:30:19.680 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.680 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.680 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:30:19.680 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189686+0000", 2024-04-20T17:30:19.680 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0079754270000000002 2024-04-20T17:30:19.680 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.680 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.680 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:19.680 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.216686+0000", 2024-04-20T17:30:19.680 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.026999433 2024-04-20T17:30:19.680 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.680 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.680 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:30:19.680 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.216718+0000", 2024-04-20T17:30:19.680 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2570000000000002e-05 2024-04-20T17:30:19.681 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.681 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.681 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:30:19.681 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.216757+0000", 2024-04-20T17:30:19.681 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9008999999999999e-05 2024-04-20T17:30:19.681 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.681 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:30:19.681 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.681 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.681 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.681 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245623 3.d 3:becddf3a:::benchmark_data_smithi171_110765_object102408:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:30:19.681 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.103997+0000", 2024-04-20T17:30:19.681 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 406.50417082600001, 2024-04-20T17:30:19.681 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.11723939999999999, 2024-04-20T17:30:19.681 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:30:19.682 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:30:19.682 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:19.682 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:30:19.682 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:30:19.682 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:30:19.682 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245623 2024-04-20T17:30:19.682 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.682 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:30:19.682 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.682 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:30:19.682 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103997+0000", 2024-04-20T17:30:19.682 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.683 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.683 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.683 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:30:19.683 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103997+0000", 2024-04-20T17:30:19.683 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.683 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.683 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.683 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:30:19.683 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103997+0000", 2024-04-20T17:30:19.683 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8099999999999998e-07 2024-04-20T17:30:19.683 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.683 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.683 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:30:19.683 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.103999+0000", 2024-04-20T17:30:19.683 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6929999999999999e-06 2024-04-20T17:30:19.684 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.684 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.684 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:30:19.684 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104008+0000", 2024-04-20T17:30:19.684 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0310000000000004e-06 2024-04-20T17:30:19.684 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.684 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.684 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:30:19.684 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104011+0000", 2024-04-20T17:30:19.684 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8060000000000001e-06 2024-04-20T17:30:19.684 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.684 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.684 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.684 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.185112+0000", 2024-04-20T17:30:19.684 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.081100835999999996 2024-04-20T17:30:19.684 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.685 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.685 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:30:19.685 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.185198+0000", 2024-04-20T17:30:19.685 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6045999999999996e-05 2024-04-20T17:30:19.685 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.685 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.685 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:30:19.685 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.185274+0000", 2024-04-20T17:30:19.685 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6091e-05 2024-04-20T17:30:19.685 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.685 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.685 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:30:19.685 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.189787+0000", 2024-04-20T17:30:19.685 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045125410000000001 2024-04-20T17:30:19.685 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.686 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.686 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:19.686 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.221137+0000", 2024-04-20T17:30:19.686 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.031350204 2024-04-20T17:30:19.686 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.686 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.686 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:30:19.686 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.221185+0000", 2024-04-20T17:30:19.686 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7410000000000002e-05 2024-04-20T17:30:19.686 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.686 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.686 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:30:19.687 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.221236+0000", 2024-04-20T17:30:19.687 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1360999999999998e-05 2024-04-20T17:30:19.687 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.687 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:30:19.687 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.687 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.687 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.687 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245630 3.d 3:b005673c:::benchmark_data_smithi171_110765_object102415:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:30:19.687 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:33.104016+0000", 2024-04-20T17:30:19.687 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 406.50415163500003, 2024-04-20T17:30:19.687 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.117355654, 2024-04-20T17:30:19.687 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:30:19.687 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:30:19.687 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:19.688 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:30:19.688 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:30:19.688 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:30:19.688 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245630 2024-04-20T17:30:19.688 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.688 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:30:19.688 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.688 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:30:19.688 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104016+0000", 2024-04-20T17:30:19.688 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.688 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.688 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.688 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:30:19.689 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104016+0000", 2024-04-20T17:30:19.689 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.689 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.689 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.689 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:30:19.689 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104016+0000", 2024-04-20T17:30:19.689 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5100000000000001e-07 2024-04-20T17:30:19.689 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.689 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.689 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:30:19.689 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104018+0000", 2024-04-20T17:30:19.689 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2409999999999998e-06 2024-04-20T17:30:19.689 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.690 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.690 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:30:19.690 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104022+0000", 2024-04-20T17:30:19.690 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8340000000000001e-06 2024-04-20T17:30:19.690 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.690 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.690 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:30:19.690 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.104027+0000", 2024-04-20T17:30:19.690 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.369e-06 2024-04-20T17:30:19.690 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.690 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.690 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.691 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.187654+0000", 2024-04-20T17:30:19.691 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.083627286999999995 2024-04-20T17:30:19.691 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.691 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.691 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:30:19.691 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.187717+0000", 2024-04-20T17:30:19.691 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3417999999999995e-05 2024-04-20T17:30:19.691 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.691 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.691 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:30:19.691 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.187783+0000", 2024-04-20T17:30:19.691 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5574000000000006e-05 2024-04-20T17:30:19.691 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.692 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.692 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:30:19.692 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.190010+0000", 2024-04-20T17:30:19.692 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022267150000000002 2024-04-20T17:30:19.692 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.692 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.692 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:19.692 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.221276+0000", 2024-04-20T17:30:19.692 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.031266250000000002 2024-04-20T17:30:19.692 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.692 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.692 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:30:19.692 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.221332+0000", 2024-04-20T17:30:19.693 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6298999999999997e-05 2024-04-20T17:30:19.693 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.693 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.693 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:30:19.693 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:33.221371+0000", 2024-04-20T17:30:19.693 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9316000000000001e-05 2024-04-20T17:30:19.693 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.693 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:30:19.693 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.693 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.693 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.693 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245820 3.f 3:f94b0a75:::benchmark_data_smithi171_110765_object102605:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:30:19.693 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:34.105704+0000", 2024-04-20T17:30:19.693 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 405.50246380999999, 2024-04-20T17:30:19.694 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.084848861999999997, 2024-04-20T17:30:19.694 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:30:19.694 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:30:19.694 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:19.694 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:30:19.694 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:30:19.694 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:30:19.694 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245820 2024-04-20T17:30:19.694 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.694 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:30:19.694 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.695 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:30:19.695 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105704+0000", 2024-04-20T17:30:19.695 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.695 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.695 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.695 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:30:19.695 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105704+0000", 2024-04-20T17:30:19.695 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.695 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.695 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.695 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:30:19.695 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105704+0000", 2024-04-20T17:30:19.695 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1e-07 2024-04-20T17:30:19.696 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.696 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.696 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:30:19.696 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105710+0000", 2024-04-20T17:30:19.696 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6890000000000001e-06 2024-04-20T17:30:19.696 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.696 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.696 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:30:19.696 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105713+0000", 2024-04-20T17:30:19.696 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.061e-06 2024-04-20T17:30:19.696 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.696 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.696 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:30:19.696 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105716+0000", 2024-04-20T17:30:19.696 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7780000000000001e-06 2024-04-20T17:30:19.697 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.697 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.697 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.697 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105816+0000", 2024-04-20T17:30:19.697 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9381000000000003e-05 2024-04-20T17:30:19.697 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.697 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.697 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:30:19.697 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105819+0000", 2024-04-20T17:30:19.697 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9689999999999999e-06 2024-04-20T17:30:19.697 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.697 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.697 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.697 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.165004+0000", 2024-04-20T17:30:19.697 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.059185298999999997 2024-04-20T17:30:19.698 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.698 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.698 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:30:19.698 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.165065+0000", 2024-04-20T17:30:19.698 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1136000000000006e-05 2024-04-20T17:30:19.698 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.698 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.698 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 11", 2024-04-20T17:30:19.698 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.165126+0000", 2024-04-20T17:30:19.698 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0863999999999999e-05 2024-04-20T17:30:19.698 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.698 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.698 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:30:19.698 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.172669+0000", 2024-04-20T17:30:19.699 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.007543396 2024-04-20T17:30:19.699 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.699 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.699 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:19.699 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.190456+0000", 2024-04-20T17:30:19.699 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.017786309 2024-04-20T17:30:19.699 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.699 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.699 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:30:19.699 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.190507+0000", 2024-04-20T17:30:19.699 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1102e-05 2024-04-20T17:30:19.699 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.699 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.699 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:30:19.699 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.190552+0000", 2024-04-20T17:30:19.700 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5568000000000001e-05 2024-04-20T17:30:19.700 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.700 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:30:19.700 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.700 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.700 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.700 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245821 3.f 3:f7671ae3:::benchmark_data_smithi171_110765_object102606:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:30:19.700 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:34.105727+0000", 2024-04-20T17:30:19.700 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 405.502440084, 2024-04-20T17:30:19.700 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.084930124999999995, 2024-04-20T17:30:19.700 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:30:19.700 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:30:19.700 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:19.701 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:30:19.701 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:30:19.701 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:30:19.701 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245821 2024-04-20T17:30:19.701 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.701 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:30:19.701 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.701 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:30:19.701 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105727+0000", 2024-04-20T17:30:19.701 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.701 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.701 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.701 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:30:19.701 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105727+0000", 2024-04-20T17:30:19.702 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.702 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.702 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.702 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:30:19.702 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105728+0000", 2024-04-20T17:30:19.702 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1699999999999999e-07 2024-04-20T17:30:19.702 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.702 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.702 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:30:19.702 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105729+0000", 2024-04-20T17:30:19.702 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4640000000000001e-06 2024-04-20T17:30:19.702 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.702 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.702 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:30:19.703 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105733+0000", 2024-04-20T17:30:19.703 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5860000000000001e-06 2024-04-20T17:30:19.703 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.703 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.703 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:30:19.703 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105735+0000", 2024-04-20T17:30:19.703 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7829999999999999e-06 2024-04-20T17:30:19.703 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.703 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.703 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.703 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105836+0000", 2024-04-20T17:30:19.703 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000100574 2024-04-20T17:30:19.703 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.703 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.703 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:30:19.704 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105841+0000", 2024-04-20T17:30:19.704 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.729e-06 2024-04-20T17:30:19.704 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.704 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.704 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.704 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.166821+0000", 2024-04-20T17:30:19.704 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.060979836000000003 2024-04-20T17:30:19.704 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.704 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.704 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:30:19.704 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.166913+0000", 2024-04-20T17:30:19.704 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2724000000000002e-05 2024-04-20T17:30:19.704 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.704 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.705 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 11", 2024-04-20T17:30:19.705 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.167000+0000", 2024-04-20T17:30:19.705 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6802e-05 2024-04-20T17:30:19.705 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.705 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.705 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:30:19.705 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.174876+0000", 2024-04-20T17:30:19.705 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0078754649999999999 2024-04-20T17:30:19.705 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.705 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.705 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:19.705 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.190587+0000", 2024-04-20T17:30:19.705 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015711573 2024-04-20T17:30:19.705 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.706 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.706 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:30:19.706 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.190626+0000", 2024-04-20T17:30:19.706 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8909999999999998e-05 2024-04-20T17:30:19.706 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.706 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.706 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:30:19.706 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.190657+0000", 2024-04-20T17:30:19.706 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1362000000000002e-05 2024-04-20T17:30:19.706 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.706 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:30:19.706 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.707 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.707 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.707 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245828 3.f 3:f231c1d2:::benchmark_data_smithi171_110765_object102613:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:30:19.707 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:34.105739+0000", 2024-04-20T17:30:19.707 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 405.50242816000002, 2024-04-20T17:30:19.707 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.085013469999999994, 2024-04-20T17:30:19.707 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:30:19.707 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:30:19.707 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:19.707 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:30:19.707 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:30:19.707 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:30:19.707 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245828 2024-04-20T17:30:19.707 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.708 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:30:19.708 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.708 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:30:19.708 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105739+0000", 2024-04-20T17:30:19.708 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.708 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.708 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.708 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:30:19.708 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105739+0000", 2024-04-20T17:30:19.708 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.708 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.708 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.709 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:30:19.709 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105740+0000", 2024-04-20T17:30:19.709 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1e-07 2024-04-20T17:30:19.709 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.709 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.709 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:30:19.709 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105741+0000", 2024-04-20T17:30:19.709 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.221e-06 2024-04-20T17:30:19.709 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.709 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.709 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:30:19.709 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105744+0000", 2024-04-20T17:30:19.709 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.892e-06 2024-04-20T17:30:19.709 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.710 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.710 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:30:19.710 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105746+0000", 2024-04-20T17:30:19.710 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6840000000000001e-06 2024-04-20T17:30:19.710 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.710 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.710 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.710 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105858+0000", 2024-04-20T17:30:19.710 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011199199999999999 2024-04-20T17:30:19.710 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.710 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.710 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:30:19.710 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105861+0000", 2024-04-20T17:30:19.710 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9440000000000001e-06 2024-04-20T17:30:19.711 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.711 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.711 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.711 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.168822+0000", 2024-04-20T17:30:19.711 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.062960753999999994 2024-04-20T17:30:19.711 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.711 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.711 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:30:19.711 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.168895+0000", 2024-04-20T17:30:19.711 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2817000000000004e-05 2024-04-20T17:30:19.711 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.711 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.711 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 11", 2024-04-20T17:30:19.711 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.168969+0000", 2024-04-20T17:30:19.712 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4091000000000006e-05 2024-04-20T17:30:19.712 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.712 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.712 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:30:19.712 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.174881+0000", 2024-04-20T17:30:19.712 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0059125870000000004 2024-04-20T17:30:19.712 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.712 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.712 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:19.712 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.190687+0000", 2024-04-20T17:30:19.712 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015805475999999999 2024-04-20T17:30:19.712 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.712 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.713 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:30:19.713 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.190724+0000", 2024-04-20T17:30:19.713 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6738e-05 2024-04-20T17:30:19.713 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.713 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.713 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:30:19.713 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.190753+0000", 2024-04-20T17:30:19.713 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8963999999999999e-05 2024-04-20T17:30:19.713 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.713 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:30:19.713 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.713 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.713 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.713 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245831 3.f 3:fd5d268b:::benchmark_data_smithi171_110765_object102616:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:30:19.714 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:34.105750+0000", 2024-04-20T17:30:19.714 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 405.50241753199998, 2024-04-20T17:30:19.714 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.090378368000000001, 2024-04-20T17:30:19.714 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:30:19.714 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:30:19.714 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:19.714 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:30:19.714 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:30:19.714 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:30:19.714 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245831 2024-04-20T17:30:19.714 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.714 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:30:19.714 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.714 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:30:19.715 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105750+0000", 2024-04-20T17:30:19.715 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.715 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.715 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.715 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:30:19.715 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105750+0000", 2024-04-20T17:30:19.715 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.715 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.715 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.715 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:30:19.715 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105750+0000", 2024-04-20T17:30:19.715 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7000000000000001e-07 2024-04-20T17:30:19.715 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.715 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.715 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:30:19.716 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105751+0000", 2024-04-20T17:30:19.716 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.091e-06 2024-04-20T17:30:19.716 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.716 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.716 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:30:19.716 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105754+0000", 2024-04-20T17:30:19.716 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.559e-06 2024-04-20T17:30:19.716 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.716 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.716 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:30:19.716 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105757+0000", 2024-04-20T17:30:19.716 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.819e-06 2024-04-20T17:30:19.716 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.716 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.716 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.717 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105878+0000", 2024-04-20T17:30:19.717 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000121839 2024-04-20T17:30:19.717 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.717 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.717 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:30:19.717 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105882+0000", 2024-04-20T17:30:19.717 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.224e-06 2024-04-20T17:30:19.717 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.717 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.717 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.717 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.170692+0000", 2024-04-20T17:30:19.717 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.064810693000000003 2024-04-20T17:30:19.717 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.717 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.717 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:30:19.718 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.170783+0000", 2024-04-20T17:30:19.718 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0681999999999997e-05 2024-04-20T17:30:19.718 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.718 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.718 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 11", 2024-04-20T17:30:19.718 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.170858+0000", 2024-04-20T17:30:19.718 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5333999999999994e-05 2024-04-20T17:30:19.718 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.718 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.718 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:30:19.718 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.175098+0000", 2024-04-20T17:30:19.718 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.004239278 2024-04-20T17:30:19.718 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.718 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:19.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.196021+0000", 2024-04-20T17:30:19.719 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.020923529 2024-04-20T17:30:19.719 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.719 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:30:19.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.196081+0000", 2024-04-20T17:30:19.719 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0188e-05 2024-04-20T17:30:19.719 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.719 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:30:19.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.196128+0000", 2024-04-20T17:30:19.719 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6862e-05 2024-04-20T17:30:19.719 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.719 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:30:19.720 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.720 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.720 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.720 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245857 3.f 3:f1991b6e:::benchmark_data_smithi171_110765_object102642:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:30:19.720 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:34.105770+0000", 2024-04-20T17:30:19.720 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 405.502397003, 2024-04-20T17:30:19.720 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.091072343, 2024-04-20T17:30:19.720 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:30:19.720 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:30:19.720 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:19.720 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:30:19.720 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:30:19.720 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:30:19.720 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245857 2024-04-20T17:30:19.720 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.721 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:30:19.721 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.721 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:30:19.721 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105770+0000", 2024-04-20T17:30:19.721 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.721 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.721 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.721 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:30:19.721 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105770+0000", 2024-04-20T17:30:19.721 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.721 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.721 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.721 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:30:19.721 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105771+0000", 2024-04-20T17:30:19.722 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.84e-07 2024-04-20T17:30:19.722 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.722 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.722 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:30:19.722 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105772+0000", 2024-04-20T17:30:19.722 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3939999999999999e-06 2024-04-20T17:30:19.722 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.722 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.722 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:30:19.722 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105776+0000", 2024-04-20T17:30:19.722 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8550000000000004e-06 2024-04-20T17:30:19.722 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.722 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.722 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:30:19.723 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105779+0000", 2024-04-20T17:30:19.723 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.768e-06 2024-04-20T17:30:19.723 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.723 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.723 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.723 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105899+0000", 2024-04-20T17:30:19.723 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012002600000000001 2024-04-20T17:30:19.723 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.723 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.723 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:30:19.723 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105902+0000", 2024-04-20T17:30:19.723 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9079999999999999e-06 2024-04-20T17:30:19.723 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.723 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.724 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.724 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.172714+0000", 2024-04-20T17:30:19.724 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.066812508000000007 2024-04-20T17:30:19.724 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.724 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.724 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:30:19.724 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.172829+0000", 2024-04-20T17:30:19.724 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000114436 2024-04-20T17:30:19.724 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.724 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.724 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 11", 2024-04-20T17:30:19.724 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.172951+0000", 2024-04-20T17:30:19.724 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012268899999999999 2024-04-20T17:30:19.724 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.725 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.725 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:30:19.725 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.177495+0000", 2024-04-20T17:30:19.725 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045431789999999996 2024-04-20T17:30:19.725 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.725 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.725 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:19.725 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.196749+0000", 2024-04-20T17:30:19.725 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.019254060999999999 2024-04-20T17:30:19.725 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.725 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.725 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:30:19.725 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.196800+0000", 2024-04-20T17:30:19.725 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1209999999999999e-05 2024-04-20T17:30:19.726 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.726 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.726 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:30:19.726 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.196843+0000", 2024-04-20T17:30:19.726 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2925e-05 2024-04-20T17:30:19.726 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.726 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:30:19.726 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.726 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.726 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.726 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:245891 3.f 3:fe0f9ce5:::benchmark_data_smithi171_110765_object102676:head [delete] snapc 0=[] ondisk+write+known_if_redirected e419)", 2024-04-20T17:30:19.726 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:34.105792+0000", 2024-04-20T17:30:19.726 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 405.50237570299998, 2024-04-20T17:30:19.727 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.091162981000000004, 2024-04-20T17:30:19.727 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:30:19.727 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:30:19.727 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:19.727 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:30:19.727 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:30:19.727 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:30:19.727 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 245891 2024-04-20T17:30:19.727 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.727 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:30:19.727 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.727 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:30:19.727 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105792+0000", 2024-04-20T17:30:19.727 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.727 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.728 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.728 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:30:19.728 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105792+0000", 2024-04-20T17:30:19.728 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.728 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.728 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.728 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:30:19.728 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105792+0000", 2024-04-20T17:30:19.728 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9400000000000001e-07 2024-04-20T17:30:19.728 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.728 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.728 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:30:19.728 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105793+0000", 2024-04-20T17:30:19.728 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.398e-06 2024-04-20T17:30:19.729 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.729 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.729 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:30:19.729 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105796+0000", 2024-04-20T17:30:19.729 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.959e-06 2024-04-20T17:30:19.729 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.729 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.729 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:30:19.729 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105799+0000", 2024-04-20T17:30:19.729 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7719999999999999e-06 2024-04-20T17:30:19.729 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.729 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.729 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.729 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105913+0000", 2024-04-20T17:30:19.729 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011384599999999999 2024-04-20T17:30:19.729 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.730 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.730 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:30:19.730 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.105916+0000", 2024-04-20T17:30:19.730 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0110000000000001e-06 2024-04-20T17:30:19.730 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.730 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.730 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.730 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.175112+0000", 2024-04-20T17:30:19.730 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.069195997999999995 2024-04-20T17:30:19.730 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.730 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.730 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:30:19.730 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.175190+0000", 2024-04-20T17:30:19.730 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7616999999999999e-05 2024-04-20T17:30:19.731 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.731 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.731 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 11", 2024-04-20T17:30:19.731 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.175261+0000", 2024-04-20T17:30:19.731 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1484999999999997e-05 2024-04-20T17:30:19.731 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.731 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.731 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:30:19.731 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.177849+0000", 2024-04-20T17:30:19.731 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0025879589999999999 2024-04-20T17:30:19.731 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.731 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.731 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:30:19.731 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.196883+0000", 2024-04-20T17:30:19.731 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.019034253000000001 2024-04-20T17:30:19.731 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.732 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.732 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:30:19.732 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.196916+0000", 2024-04-20T17:30:19.732 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2586999999999998e-05 2024-04-20T17:30:19.732 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.732 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.732 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:30:19.732 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:34.196955+0000", 2024-04-20T17:30:19.732 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8801999999999999e-05 2024-04-20T17:30:19.732 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.732 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:30:19.732 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.732 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.732 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.733 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26074.0:282382 4.0 e521/445 4:09730dbe:::benchmark_data_smithi171_122772_object112190:head v 521'18245, mlcod=521'18231)", 2024-04-20T17:30:19.733 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:28:18.745809+0000", 2024-04-20T17:30:19.733 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 120.86235811900001, 2024-04-20T17:30:19.733 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.087159433999999994, 2024-04-20T17:30:19.733 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:30:19.733 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:30:19.733 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:30:19.733 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:30:19.733 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.733 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:30:19.733 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:28:18.745809+0000", 2024-04-20T17:30:19.733 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.733 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.733 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.733 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:30:19.734 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:28:18.745809+0000", 2024-04-20T17:30:19.734 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:30:19.734 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.734 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.734 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:30:19.734 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:28:18.745810+0000", 2024-04-20T17:30:19.734 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1030000000000001e-06 2024-04-20T17:30:19.734 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.734 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.734 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:30:19.734 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:28:18.745815+0000", 2024-04-20T17:30:19.734 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8459999999999999e-06 2024-04-20T17:30:19.734 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.734 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.735 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:30:19.735 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:28:18.745823+0000", 2024-04-20T17:30:19.735 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8280000000000003e-06 2024-04-20T17:30:19.735 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.735 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.735 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:30:19.735 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:28:18.745829+0000", 2024-04-20T17:30:19.735 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.032e-06 2024-04-20T17:30:19.735 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.735 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.735 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:30:19.735 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:28:18.785215+0000", 2024-04-20T17:30:19.735 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.039386351 2024-04-20T17:30:19.735 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.735 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.736 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:30:19.736 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:28:18.785228+0000", 2024-04-20T17:30:19.736 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3019e-05 2024-04-20T17:30:19.736 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.736 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.736 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:30:19.736 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:28:18.832922+0000", 2024-04-20T17:30:19.736 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.047693896999999999 2024-04-20T17:30:19.736 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:30:19.736 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:30:19.736 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:30:19.736 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:28:18.832969+0000", 2024-04-20T17:30:19.736 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6357999999999999e-05 2024-04-20T17:30:19.736 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.737 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:30:19.737 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.737 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:30:19.737 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:30:19.737 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:30:19.738 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:19 smithi005 ceph-mon[39562]: pgmap v1481: 17 pgs: 17 active+clean; 1.4 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 188 MiB/s wr, 3.01k op/s 2024-04-20T17:30:19.738 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:19 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1451236896' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:30:19.738 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:19.738 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:19 smithi106 ceph-mon[44423]: pgmap v1481: 17 pgs: 17 active+clean; 1.4 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 188 MiB/s wr, 3.01k op/s 2024-04-20T17:30:19.738 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:19 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1451236896' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:30:19.738 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:19.882 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:30:19.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:19.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:30:19.990 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:30:20.032 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:20.064 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:30:20.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:20.068 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:30:20.169 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:30:20.208 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:20.225 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:30:20.226619 min lat: 0.00096363 max lat: 30.3388 avg lat: 0.0160504 2024-04-20T17:30:20.225 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:30:20.226 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 40 15 39889 39874 62.2953 200.312 0.00378585 0.0160504 2024-04-20T17:30:20.310 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:30:20.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:20.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:30:20.352 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-20T17:30:20.416 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:30:20.461 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.665 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.672 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:30:20.672 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:30:20.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:20.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:30:20.741 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:30:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:20.776 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:30:20.827 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:21.226 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 41 15 42892 42877 65.3531 187.688 0.00288261 0.0153 2024-04-20T17:30:21.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:21.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:21 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:30:21.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:21 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:30:21.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:21 smithi171 ceph-mon[44189]: pgmap v1482: 17 pgs: 17 active+clean; 1.9 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 228 MiB/s wr, 3.65k op/s 2024-04-20T17:30:21.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:21 smithi171 ceph-mon[44189]: osdmap e566: 12 total, 12 up, 11 in 2024-04-20T17:30:21.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:21 smithi171 ceph-mon[44189]: osdmap e567: 12 total, 12 up, 11 in 2024-04-20T17:30:21.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:21.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:21 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:30:21.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:21 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:30:21.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:21 smithi005 ceph-mon[39562]: pgmap v1482: 17 pgs: 17 active+clean; 1.9 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 228 MiB/s wr, 3.65k op/s 2024-04-20T17:30:21.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:21 smithi005 ceph-mon[39562]: osdmap e566: 12 total, 12 up, 11 in 2024-04-20T17:30:21.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:21 smithi005 ceph-mon[39562]: osdmap e567: 12 total, 12 up, 11 in 2024-04-20T17:30:21.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:21.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:21 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:30:21.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:21 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:30:21.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:21 smithi106 ceph-mon[44423]: pgmap v1482: 17 pgs: 17 active+clean; 1.9 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 228 MiB/s wr, 3.65k op/s 2024-04-20T17:30:21.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:21 smithi106 ceph-mon[44423]: osdmap e566: 12 total, 12 up, 11 in 2024-04-20T17:30:21.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:21 smithi106 ceph-mon[44423]: osdmap e567: 12 total, 12 up, 11 in 2024-04-20T17:30:22.226 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 42 16 45845 45829 68.1895 184.5 0.00514505 0.014663 2024-04-20T17:30:22.339 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.1 to 1 (10000) 2024-04-20T17:30:22.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:22 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1478698194' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:30:22.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:22 smithi005 ceph-mon[39562]: from='client.26527 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:30:22.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:22 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1478698194' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:30:22.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:22 smithi106 ceph-mon[44423]: from='client.26527 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:30:22.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:22 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1478698194' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:30:22.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:22 smithi171 ceph-mon[44189]: from='client.26527 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:30:22.715 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:30:22.853 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:30:22.854 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:30:22.854 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-04-20T17:30:22.854 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:30:23.226 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 43 16 48568 48552 70.5611 170.188 0.00252457 0.0141702 2024-04-20T17:30:23.510 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:30:23.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:23.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:30:23.515 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 2 1 2024-04-20T17:30:23.580 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:23 smithi106 ceph-mon[44423]: pgmap v1485: 17 pgs: 17 active+clean; 2.4 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 181 MiB/s wr, 2.89k op/s 2024-04-20T17:30:23.580 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:23 smithi106 ceph-mon[44423]: from='client.26527 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-20T17:30:23.580 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:23 smithi106 ceph-mon[44423]: osdmap e568: 12 total, 12 up, 11 in 2024-04-20T17:30:23.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:23 smithi171 ceph-mon[44189]: pgmap v1485: 17 pgs: 17 active+clean; 2.4 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 181 MiB/s wr, 2.89k op/s 2024-04-20T17:30:23.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:23 smithi171 ceph-mon[44189]: from='client.26527 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-20T17:30:23.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:23 smithi171 ceph-mon[44189]: osdmap e568: 12 total, 12 up, 11 in 2024-04-20T17:30:23.607 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:23 smithi005 ceph-mon[39562]: pgmap v1485: 17 pgs: 17 active+clean; 2.4 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 181 MiB/s wr, 2.89k op/s 2024-04-20T17:30:23.607 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:23 smithi005 ceph-mon[39562]: from='client.26527 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-20T17:30:23.607 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:23 smithi005 ceph-mon[39562]: osdmap e568: 12 total, 12 up, 11 in 2024-04-20T17:30:23.614 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:30:23.615 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-20T17:30:23.626 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:30:23.699 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:23.920 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:30:23.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:23.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:30:23.961 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:30:24.025 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:30:24.063 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:24.088 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:30:24.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:24.091 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:30:24.193 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:30:24.226 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 44 16 51106 51090 72.5622 158.625 0.00323132 0.0137788 2024-04-20T17:30:24.249 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:24.346 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:30:24.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:24 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/351255768' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:30:24.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:24 smithi005 ceph-mon[39562]: from='client.26530 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:30:24.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:24 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/351255768' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:30:24.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:24 smithi106 ceph-mon[44423]: from='client.26530 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:30:24.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:24 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/351255768' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:30:24.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:24 smithi171 ceph-mon[44189]: from='client.26530 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:30:25.213 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:30:25.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:25.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:30:25.226 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 45 15 53836 53821 74.7424 170.688 0.00916749 0.0133777 2024-04-20T17:30:25.257 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:30:25.316 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:30:25.349 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:25 smithi106 ceph-mon[44423]: pgmap v1487: 17 pgs: 17 active+clean; 2.8 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 244 MiB/s wr, 3.91k op/s 2024-04-20T17:30:25.349 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:25 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:30:25.349 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:25 smithi106 ceph-mon[44423]: from='client.26530 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:30:25.349 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:25 smithi106 ceph-mon[44423]: osdmap e569: 12 total, 12 up, 11 in 2024-04-20T17:30:25.352 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:25.411 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:30:25.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:25.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:30:25.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:25 smithi005 ceph-mon[39562]: pgmap v1487: 17 pgs: 17 active+clean; 2.8 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 244 MiB/s wr, 3.91k op/s 2024-04-20T17:30:25.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:25 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:30:25.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:25 smithi005 ceph-mon[39562]: from='client.26530 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:30:25.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:25 smithi005 ceph-mon[39562]: osdmap e569: 12 total, 12 up, 11 in 2024-04-20T17:30:25.518 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:30:25.567 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:25.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:25 smithi171 ceph-mon[44189]: pgmap v1487: 17 pgs: 17 active+clean; 2.8 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 244 MiB/s wr, 3.91k op/s 2024-04-20T17:30:25.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:25 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:30:25.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:25 smithi171 ceph-mon[44189]: from='client.26530 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:30:25.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:25 smithi171 ceph-mon[44189]: osdmap e569: 12 total, 12 up, 11 in 2024-04-20T17:30:26.111 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:30:26.111 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:30:26.111 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-04-20T17:30:26.112 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-04-20T17:30:26.112 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:30:26.226 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 46 16 56470 56454 76.6946 164.562 0.00343361 0.0130367 2024-04-20T17:30:26.542 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:30:26.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:26.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:30:26.647 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:30:26.683 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:26.708 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:30:26.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:26.713 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:30:26.814 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:30:26.845 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-20T17:30:26.864 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:26.865 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:30:26.900 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:30:26.942 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:30:26.976 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:30:27.008 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:30:27.025 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:30:27.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:27.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:30:27.047 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:30:27.090 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:30:27.129 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:30:27.131 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:30:27.158 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:30:27.170 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:27.185 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:30:27.200 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:30:27.201 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:30:27.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:27.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:30:27.221 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:30:27.226 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 47 16 58817 58801 78.1835 146.688 0.00734847 0.0127888 2024-04-20T17:30:27.256 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:30:27.266 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:30:27.267 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:30:27.267 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:30:27.267 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:30:27.267 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:30:27.267 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:30:27.267 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:30:27.267 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:30:27.267 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:30:27.268 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:30:27.268 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:30:27.268 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:30:27.306 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:30:27.341 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:27.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:27 smithi005 ceph-mon[39562]: pgmap v1489: 17 pgs: 17 active+clean; 3.2 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 234 MiB/s wr, 3.74k op/s 2024-04-20T17:30:27.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:27 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2712457442' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:30:27.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:27.462 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:27 smithi106 ceph-mon[44423]: pgmap v1489: 17 pgs: 17 active+clean; 3.2 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 234 MiB/s wr, 3.74k op/s 2024-04-20T17:30:27.462 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:27 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2712457442' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:30:27.462 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:27.472 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:30:27.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:27.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:30:27.582 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:30:27.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:27 smithi171 ceph-mon[44189]: pgmap v1489: 17 pgs: 17 active+clean; 3.2 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 234 MiB/s wr, 3.74k op/s 2024-04-20T17:30:27.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:27 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2712457442' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:30:27.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:27.618 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:27.643 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:30:27.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:27.647 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:30:27.748 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:30:27.786 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:27.810 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:30:27.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:27.812 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:30:27.913 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:30:27.954 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:27.979 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:30:27.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:27.983 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:30:28.052 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:30:28.085 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:30:28.123 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:28.145 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:30:28.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:28.147 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:30:28.226 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 48 16 61234 61218 79.7015 151.062 0.00573468 0.0125451 2024-04-20T17:30:28.249 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:30:28.321 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:28.358 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:30:28.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:28.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.369 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.370 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.370 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:30:28.471 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:30:28.539 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:28.570 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.570 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.570 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.571 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.571 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.571 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.571 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.571 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.571 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.571 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.571 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:28.699 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:30:28.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:28.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:30:28.807 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:30:28.845 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:28.869 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:30:28.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:28.873 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:30:28.934 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:30:28.974 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:30:29.018 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:29.221 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:30:29.226 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 49 16 63413 63397 80.854 136.188 0.00282828 0.0123665 2024-04-20T17:30:29.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:29.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:30:29.318 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.2 to 1 (10000) 2024-04-20T17:30:29.328 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:30:29.361 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:29.398 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:30:29.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:29.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:30:29.486 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:29 smithi005 ceph-mon[39562]: pgmap v1490: 17 pgs: 17 active+clean; 3.3 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-04-20T17:30:29.487 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:29 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3853429118' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-20T17:30:29.487 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:29 smithi005 ceph-mon[39562]: from='client.26642 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-20T17:30:29.504 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:30:29.549 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:29.561 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:30:29.561 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:30:29.583 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:29 smithi106 ceph-mon[44423]: pgmap v1490: 17 pgs: 17 active+clean; 3.3 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-04-20T17:30:29.583 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:29 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3853429118' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-20T17:30:29.583 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:29 smithi106 ceph-mon[44423]: from='client.26642 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-20T17:30:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:29 smithi171 ceph-mon[44189]: pgmap v1490: 17 pgs: 17 active+clean; 3.3 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-04-20T17:30:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:29 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3853429118' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-20T17:30:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:29 smithi171 ceph-mon[44189]: from='client.26642 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-20T17:30:29.764 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:30:29.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:29.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:30:29.870 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:30:29.921 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:30.226 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 50 16 65770 65754 82.1829 147.312 0.0106412 0.0121661 2024-04-20T17:30:30.319 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-20T17:30:30.357 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:30 smithi005 ceph-mon[39562]: from='client.26642 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-20T17:30:30.357 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:30 smithi005 ceph-mon[39562]: osdmap e570: 12 total, 12 up, 11 in 2024-04-20T17:30:30.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:30 smithi171 ceph-mon[44189]: from='client.26642 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-20T17:30:30.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:30 smithi171 ceph-mon[44189]: osdmap e570: 12 total, 12 up, 11 in 2024-04-20T17:30:30.587 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:30:30.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:30.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:30:30.693 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:30:30.724 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:30 smithi106 ceph-mon[44423]: from='client.26642 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-20T17:30:30.724 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:30 smithi106 ceph-mon[44423]: osdmap e570: 12 total, 12 up, 11 in 2024-04-20T17:30:30.729 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:30.732 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:30:30.752 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:30:30.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:30.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:30:30.856 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:30:30.887 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:30.915 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:30:30.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:30.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:30:31.021 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:30:31.070 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 3 1 2024-04-20T17:30:31.082 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:30:31.098 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:31.226 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 51 16 67599 67583 82.8127 114.312 0.00609271 0.012074 2024-04-20T17:30:31.301 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:31 smithi005 ceph-mon[39562]: pgmap v1492: 17 pgs: 17 active+clean; 3.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 160 MiB/s wr, 2.56k op/s 2024-04-20T17:30:31.301 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:31 smithi005 ceph-mon[39562]: osdmap e571: 12 total, 12 up, 11 in 2024-04-20T17:30:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:31 smithi171 ceph-mon[44189]: pgmap v1492: 17 pgs: 17 active+clean; 3.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 160 MiB/s wr, 2.56k op/s 2024-04-20T17:30:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:31 smithi171 ceph-mon[44189]: osdmap e571: 12 total, 12 up, 11 in 2024-04-20T17:30:31.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:31 smithi106 ceph-mon[44423]: pgmap v1492: 17 pgs: 17 active+clean; 3.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 160 MiB/s wr, 2.56k op/s 2024-04-20T17:30:31.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:31 smithi106 ceph-mon[44423]: osdmap e571: 12 total, 12 up, 11 in 2024-04-20T17:30:32.226 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 52 15 69885 69870 83.9687 142.938 0.00808984 0.011908 2024-04-20T17:30:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:30:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:30:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {auth=35,config=2,health=12,kv=51} crc {auth=3501613957,config=1460075751,health=3340005642,kv=469769941}) 2024-04-20T17:30:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {kv=10,logm=90} crc {kv=3352514985,logm=2154142799}) 2024-04-20T17:30:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2846279674}) 2024-04-20T17:30:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2806371401}) 2024-04-20T17:30:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2693939271}) 2024-04-20T17:30:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3707464954}) 2024-04-20T17:30:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=718407477}) 2024-04-20T17:30:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3826819162}) 2024-04-20T17:30:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=318019471}) 2024-04-20T17:30:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=136006853}) 2024-04-20T17:30:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1472021849}) 2024-04-20T17:30:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2429512985}) 2024-04-20T17:30:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=65162728}) 2024-04-20T17:30:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1353057610}) 2024-04-20T17:30:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1315322634}) 2024-04-20T17:30:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2295397768}) 2024-04-20T17:30:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2164977201}) 2024-04-20T17:30:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3080443781}) 2024-04-20T17:30:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3458437396}) 2024-04-20T17:30:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3619534990}) 2024-04-20T17:30:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=691124781}) 2024-04-20T17:30:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=510637057}) 2024-04-20T17:30:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3272023158}) 2024-04-20T17:30:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2404672104}) 2024-04-20T17:30:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2531406145}) 2024-04-20T17:30:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2856946334}) 2024-04-20T17:30:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1695580323}) 2024-04-20T17:30:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3361609094}) 2024-04-20T17:30:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=195331691}) 2024-04-20T17:30:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=10570526}) 2024-04-20T17:30:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2420306687}) 2024-04-20T17:30:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1384300435}) 2024-04-20T17:30:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3629315285}) 2024-04-20T17:30:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1323233169}) 2024-04-20T17:30:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1045111103}) 2024-04-20T17:30:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2804996069}) 2024-04-20T17:30:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=448042281}) 2024-04-20T17:30:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2455753902}) 2024-04-20T17:30:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3879514570}) 2024-04-20T17:30:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=992822108}) 2024-04-20T17:30:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3197367689}) 2024-04-20T17:30:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2560099742}) 2024-04-20T17:30:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1771671763}) 2024-04-20T17:30:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3839940352}) 2024-04-20T17:30:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4204553052}) 2024-04-20T17:30:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2128121683}) 2024-04-20T17:30:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1277551412}) 2024-04-20T17:30:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4137276982}) 2024-04-20T17:30:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2323765031}) 2024-04-20T17:30:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1079570649}) 2024-04-20T17:30:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3960634503}) 2024-04-20T17:30:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2383546190}) 2024-04-20T17:30:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3054844644}) 2024-04-20T17:30:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=520803765}) 2024-04-20T17:30:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=384988739}) 2024-04-20T17:30:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103557962}) 2024-04-20T17:30:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3542518144}) 2024-04-20T17:30:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=109061184}) 2024-04-20T17:30:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1459669592}) 2024-04-20T17:30:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1083707770}) 2024-04-20T17:30:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1127315693}) 2024-04-20T17:30:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1273867427}) 2024-04-20T17:30:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3110145756}) 2024-04-20T17:30:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3169143997}) 2024-04-20T17:30:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3687030896}) 2024-04-20T17:30:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2028764743}) 2024-04-20T17:30:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3078769672}) 2024-04-20T17:30:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=31,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=34} crc {logm=2439704879,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=343319945,mon_config_key=2614144526}) 2024-04-20T17:30:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=635936590}) 2024-04-20T17:30:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1837529206}) 2024-04-20T17:30:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=56,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=26} crc {mon_config_key=1568907755,monmap=717113318,osd_metadata=3196995629,osd_pg_creating=4149350133,osdmap=3958816468}) 2024-04-20T17:30:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1366693140}) 2024-04-20T17:30:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=30} crc {osdmap=3159114650}) 2024-04-20T17:30:32.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:30:32.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:30:32.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {auth=35,config=2,health=12,kv=51} crc {auth=3501613957,config=1460075751,health=3340005642,kv=469769941}) 2024-04-20T17:30:32.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {kv=10,logm=90} crc {kv=3352514985,logm=2154142799}) 2024-04-20T17:30:32.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2846279674}) 2024-04-20T17:30:32.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2806371401}) 2024-04-20T17:30:32.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2693939271}) 2024-04-20T17:30:32.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3707464954}) 2024-04-20T17:30:32.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=718407477}) 2024-04-20T17:30:32.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3826819162}) 2024-04-20T17:30:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=318019471}) 2024-04-20T17:30:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=136006853}) 2024-04-20T17:30:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1472021849}) 2024-04-20T17:30:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2429512985}) 2024-04-20T17:30:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=65162728}) 2024-04-20T17:30:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1353057610}) 2024-04-20T17:30:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1315322634}) 2024-04-20T17:30:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2295397768}) 2024-04-20T17:30:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2164977201}) 2024-04-20T17:30:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3080443781}) 2024-04-20T17:30:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3458437396}) 2024-04-20T17:30:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3619534990}) 2024-04-20T17:30:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=691124781}) 2024-04-20T17:30:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=510637057}) 2024-04-20T17:30:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3272023158}) 2024-04-20T17:30:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2404672104}) 2024-04-20T17:30:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2531406145}) 2024-04-20T17:30:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2856946334}) 2024-04-20T17:30:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1695580323}) 2024-04-20T17:30:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3361609094}) 2024-04-20T17:30:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=195331691}) 2024-04-20T17:30:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=10570526}) 2024-04-20T17:30:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2420306687}) 2024-04-20T17:30:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1384300435}) 2024-04-20T17:30:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3629315285}) 2024-04-20T17:30:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1323233169}) 2024-04-20T17:30:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1045111103}) 2024-04-20T17:30:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2804996069}) 2024-04-20T17:30:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=448042281}) 2024-04-20T17:30:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2455753902}) 2024-04-20T17:30:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3879514570}) 2024-04-20T17:30:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=992822108}) 2024-04-20T17:30:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3197367689}) 2024-04-20T17:30:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2560099742}) 2024-04-20T17:30:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1771671763}) 2024-04-20T17:30:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3839940352}) 2024-04-20T17:30:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4204553052}) 2024-04-20T17:30:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2128121683}) 2024-04-20T17:30:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1277551412}) 2024-04-20T17:30:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4137276982}) 2024-04-20T17:30:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2323765031}) 2024-04-20T17:30:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1079570649}) 2024-04-20T17:30:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3960634503}) 2024-04-20T17:30:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2383546190}) 2024-04-20T17:30:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3054844644}) 2024-04-20T17:30:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=520803765}) 2024-04-20T17:30:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=384988739}) 2024-04-20T17:30:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103557962}) 2024-04-20T17:30:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3542518144}) 2024-04-20T17:30:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=109061184}) 2024-04-20T17:30:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1459669592}) 2024-04-20T17:30:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1083707770}) 2024-04-20T17:30:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1127315693}) 2024-04-20T17:30:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1273867427}) 2024-04-20T17:30:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3110145756}) 2024-04-20T17:30:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3169143997}) 2024-04-20T17:30:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3687030896}) 2024-04-20T17:30:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2028764743}) 2024-04-20T17:30:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3078769672}) 2024-04-20T17:30:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=31,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=34} crc {logm=2439704879,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=343319945,mon_config_key=2614144526}) 2024-04-20T17:30:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=635936590}) 2024-04-20T17:30:32.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1837529206}) 2024-04-20T17:30:32.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=56,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=26} crc {mon_config_key=1568907755,monmap=717113318,osd_metadata=3196995629,osd_pg_creating=4149350133,osdmap=3958816468}) 2024-04-20T17:30:32.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1366693140}) 2024-04-20T17:30:32.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=30} crc {osdmap=3159114650}) 2024-04-20T17:30:32.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:30:32.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:30:32.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {auth=35,config=2,health=12,kv=51} crc {auth=3501613957,config=1460075751,health=3340005642,kv=469769941}) 2024-04-20T17:30:32.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {kv=10,logm=90} crc {kv=3352514985,logm=2154142799}) 2024-04-20T17:30:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2846279674}) 2024-04-20T17:30:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2806371401}) 2024-04-20T17:30:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2693939271}) 2024-04-20T17:30:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3707464954}) 2024-04-20T17:30:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=718407477}) 2024-04-20T17:30:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3826819162}) 2024-04-20T17:30:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=318019471}) 2024-04-20T17:30:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=136006853}) 2024-04-20T17:30:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1472021849}) 2024-04-20T17:30:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2429512985}) 2024-04-20T17:30:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=65162728}) 2024-04-20T17:30:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1353057610}) 2024-04-20T17:30:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1315322634}) 2024-04-20T17:30:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2295397768}) 2024-04-20T17:30:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2164977201}) 2024-04-20T17:30:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3080443781}) 2024-04-20T17:30:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3458437396}) 2024-04-20T17:30:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3619534990}) 2024-04-20T17:30:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=691124781}) 2024-04-20T17:30:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=510637057}) 2024-04-20T17:30:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3272023158}) 2024-04-20T17:30:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2404672104}) 2024-04-20T17:30:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2531406145}) 2024-04-20T17:30:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2856946334}) 2024-04-20T17:30:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1695580323}) 2024-04-20T17:30:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3361609094}) 2024-04-20T17:30:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=195331691}) 2024-04-20T17:30:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=10570526}) 2024-04-20T17:30:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2420306687}) 2024-04-20T17:30:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1384300435}) 2024-04-20T17:30:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3629315285}) 2024-04-20T17:30:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1323233169}) 2024-04-20T17:30:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1045111103}) 2024-04-20T17:30:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2804996069}) 2024-04-20T17:30:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=448042281}) 2024-04-20T17:30:32.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2455753902}) 2024-04-20T17:30:32.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3879514570}) 2024-04-20T17:30:32.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=992822108}) 2024-04-20T17:30:32.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3197367689}) 2024-04-20T17:30:32.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2560099742}) 2024-04-20T17:30:32.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1771671763}) 2024-04-20T17:30:32.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3839940352}) 2024-04-20T17:30:32.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4204553052}) 2024-04-20T17:30:32.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2128121683}) 2024-04-20T17:30:32.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1277551412}) 2024-04-20T17:30:32.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4137276982}) 2024-04-20T17:30:32.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2323765031}) 2024-04-20T17:30:32.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1079570649}) 2024-04-20T17:30:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3960634503}) 2024-04-20T17:30:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2383546190}) 2024-04-20T17:30:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3054844644}) 2024-04-20T17:30:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=520803765}) 2024-04-20T17:30:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=384988739}) 2024-04-20T17:30:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103557962}) 2024-04-20T17:30:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3542518144}) 2024-04-20T17:30:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=109061184}) 2024-04-20T17:30:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1459669592}) 2024-04-20T17:30:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1083707770}) 2024-04-20T17:30:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1127315693}) 2024-04-20T17:30:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1273867427}) 2024-04-20T17:30:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3110145756}) 2024-04-20T17:30:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3169143997}) 2024-04-20T17:30:32.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3687030896}) 2024-04-20T17:30:32.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2028764743}) 2024-04-20T17:30:32.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3078769672}) 2024-04-20T17:30:32.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=31,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=34} crc {logm=2439704879,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=343319945,mon_config_key=2614144526}) 2024-04-20T17:30:32.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=635936590}) 2024-04-20T17:30:32.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1837529206}) 2024-04-20T17:30:32.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=56,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=26} crc {mon_config_key=1568907755,monmap=717113318,osd_metadata=3196995629,osd_pg_creating=4149350133,osdmap=3958816468}) 2024-04-20T17:30:32.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1366693140}) 2024-04-20T17:30:32.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=30} crc {osdmap=3159114650}) 2024-04-20T17:30:32.998 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:30:33.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:33.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:30:33.102 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:30:33.137 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:33.164 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:30:33.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:33.169 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:30:33.227 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 53 16 72186 72170 85.0964 143.75 0.00483587 0.0117498 2024-04-20T17:30:33.269 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:30:33.328 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:33.377 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:30:33.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:33.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:30:33.482 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:30:33.525 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:33.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:33 smithi171 ceph-mon[44189]: pgmap v1494: 17 pgs: 17 active+clean; 4.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 147 MiB/s wr, 2.35k op/s 2024-04-20T17:30:33.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:33 smithi005 ceph-mon[39562]: pgmap v1494: 17 pgs: 17 active+clean; 4.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 147 MiB/s wr, 2.35k op/s 2024-04-20T17:30:33.752 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:30:33.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:33.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:30:33.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:33 smithi106 ceph-mon[44423]: pgmap v1494: 17 pgs: 17 active+clean; 4.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 147 MiB/s wr, 2.35k op/s 2024-04-20T17:30:33.862 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:30:33.892 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:33.915 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:30:33.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:33.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:30:34.019 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:30:34.055 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:34.078 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:30:34.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:34.081 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:30:34.182 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:30:34.221 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:34.227 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 54 16 74202 74186 85.8537 126 0.00970848 0.0116456 2024-04-20T17:30:34.252 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:30:34.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:34.254 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:30:34.331 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:34 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1186822230' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:30:34.331 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:34 smithi171 ceph-mon[44189]: from='client.26648 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:30:34.345 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:30:34.355 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:30:34.392 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:34.418 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:30:34.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:34.424 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:30:34.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:34 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1186822230' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:30:34.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:34 smithi005 ceph-mon[39562]: from='client.26648 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:30:34.525 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:30:34.566 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:34.596 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:30:34.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:34.600 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:30:34.701 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:30:34.731 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:34.756 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:30:34.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:34.759 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:30:34.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:34 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1186822230' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:30:34.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:34 smithi106 ceph-mon[44423]: from='client.26648 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:30:34.859 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:30:34.894 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:34.922 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:30:34.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:34.925 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:30:35.027 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:30:35.066 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:35.093 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:30:35.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:35.099 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:30:35.200 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:30:35.227 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 55 16 76309 76293 86.6868 131.688 0.00736679 0.0115346 2024-04-20T17:30:35.234 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:35.258 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:30:35.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:35.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:30:35.316 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:30:35.363 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:30:35.396 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:35.426 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:30:35.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:35.432 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:30:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:35 smithi106 ceph-mon[44423]: pgmap v1495: 17 pgs: 17 active+clean; 4.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 135 MiB/s wr, 2.16k op/s 2024-04-20T17:30:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:35 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:30:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:35 smithi106 ceph-mon[44423]: from='client.26648 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:30:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:35 smithi106 ceph-mon[44423]: osdmap e572: 12 total, 12 up, 11 in 2024-04-20T17:30:35.533 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:30:35.587 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:35 smithi005 ceph-mon[39562]: pgmap v1495: 17 pgs: 17 active+clean; 4.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 135 MiB/s wr, 2.16k op/s 2024-04-20T17:30:35.587 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:35 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:30:35.587 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:35 smithi005 ceph-mon[39562]: from='client.26648 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:30:35.587 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:35 smithi005 ceph-mon[39562]: osdmap e572: 12 total, 12 up, 11 in 2024-04-20T17:30:35.592 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:35.630 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:30:35.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:35.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:30:35.737 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:30:35.803 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:35 smithi171 ceph-mon[44189]: pgmap v1495: 17 pgs: 17 active+clean; 4.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 135 MiB/s wr, 2.16k op/s 2024-04-20T17:30:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:35 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:30:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:35 smithi171 ceph-mon[44189]: from='client.26648 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:30:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:35 smithi171 ceph-mon[44189]: osdmap e572: 12 total, 12 up, 11 in 2024-04-20T17:30:36.016 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:30:36.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:36.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:30:36.123 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:30:36.158 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:36.186 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:30:36.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:36.190 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:30:36.227 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 56 15 78261 78246 87.3183 122.062 0.0148761 0.011451 2024-04-20T17:30:36.291 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:30:36.343 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:30:36.345 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:36.374 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:30:36.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:36.377 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:30:36.478 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:30:36.513 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:36.540 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:30:36.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:36.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:30:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:36 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:30:36.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:36 smithi171 ceph-mon[44189]: osdmap e573: 12 total, 12 up, 11 in 2024-04-20T17:30:36.644 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:30:36.691 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:36 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:30:36.691 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:36 smithi005 ceph-mon[39562]: osdmap e573: 12 total, 12 up, 11 in 2024-04-20T17:30:36.695 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:36.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:36 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:30:36.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:36 smithi106 ceph-mon[44423]: osdmap e573: 12 total, 12 up, 11 in 2024-04-20T17:30:36.774 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:30:36.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:36.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:30:36.885 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:30:36.922 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:36.954 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:30:36.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:36.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:30:37.060 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:30:37.114 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:37.216 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:30:37.217 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:30:37.227 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 57 16 79911 79895 87.5944 103.062 0.00378561 0.0114139 2024-04-20T17:30:37.379 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:30:37.579 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:30:37.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:37.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:30:37.688 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:30:37.689 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:37 smithi005 ceph-mon[39562]: pgmap v1498: 17 pgs: 17 active+clean; 4.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 171 MiB/s wr, 2.73k op/s 2024-04-20T17:30:37.689 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:37 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:30:37.689 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:37 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:30:37.689 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:37 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:30:37.689 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:37 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:30:37.690 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:37 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:30:37.690 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:37 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:30:37.741 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:38.011 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:30:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:37 smithi106 ceph-mon[44423]: pgmap v1498: 17 pgs: 17 active+clean; 4.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 171 MiB/s wr, 2.73k op/s 2024-04-20T17:30:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:37 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:30:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:37 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:30:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:37 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:30:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:37 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:30:38.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:37 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:30:38.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:37 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:30:38.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:38.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:30:38.038 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-20T17:30:38.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:37 smithi171 ceph-mon[44189]: pgmap v1498: 17 pgs: 17 active+clean; 4.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 171 MiB/s wr, 2.73k op/s 2024-04-20T17:30:38.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:37 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:30:38.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:37 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:30:38.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:37 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:30:38.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:37 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:30:38.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:37 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:30:38.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:37 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:30:38.117 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:30:38.165 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:38.227 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 58 16 81730 81714 88.0441 113.688 0.0136025 0.0113556 2024-04-20T17:30:38.277 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:30:38.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:38.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:30:38.386 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:30:38.418 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:38.441 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:30:38.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:38.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:30:38.548 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:30:38.595 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:38.824 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:30:38.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:38.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:30:38.845 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:30:38.845 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:30:38.845 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:30:38.930 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:30:38.976 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:30:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:30:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:30:39.026 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:30:39.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:39.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:30:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:30:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:30:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:30:39.141 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:30:39.192 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:39.227 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 59 16 83618 83602 88.5517 118 0.00927114 0.0112909 2024-04-20T17:30:39.340 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:30:39.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:39.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:30:39.449 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:30:39.506 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:39.588 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:30:39.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:39.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:30:39.594 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:30:39.695 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:30:39.727 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:39.752 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:30:39.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:39.755 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:30:39.842 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:39 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:30:39.842 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:39 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:30:39.842 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:39 smithi005 ceph-mon[39562]: pgmap v1499: 17 pgs: 17 active+clean; 4.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-04-20T17:30:39.842 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:39 smithi005 ceph-mon[39562]: osdmap e574: 12 total, 12 up, 11 in 2024-04-20T17:30:39.856 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:30:39.889 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:39.918 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:30:39.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:39.924 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:30:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:39 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:30:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:39 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:30:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:39 smithi106 ceph-mon[44423]: pgmap v1499: 17 pgs: 17 active+clean; 4.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-04-20T17:30:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:39 smithi106 ceph-mon[44423]: osdmap e574: 12 total, 12 up, 11 in 2024-04-20T17:30:40.025 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:30:40.056 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:40.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:39 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:30:40.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:39 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:30:40.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:39 smithi171 ceph-mon[44189]: pgmap v1499: 17 pgs: 17 active+clean; 4.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-04-20T17:30:40.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:39 smithi171 ceph-mon[44189]: osdmap e574: 12 total, 12 up, 11 in 2024-04-20T17:30:40.085 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:30:40.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:40.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:30:40.189 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:30:40.227 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:30:40.228215 min lat: 0.00096363 max lat: 30.3388 avg lat: 0.0112414 2024-04-20T17:30:40.227 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:30:40.228 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 60 16 85410 85394 88.9423 112 0.0204984 0.0112414 2024-04-20T17:30:40.232 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:30:40.264 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:40.342 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:30:40.364 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:30:40.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:40.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:30:40.385 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:30:40.419 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:30:40.449 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:30:40.474 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:30:40.486 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:30:40.508 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:40.521 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:30:40.531 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:30:40.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:40.535 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:30:40.555 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:30:40.590 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:30:40.618 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:30:40.636 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:30:40.651 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:30:40.669 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:40.684 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:30:40.697 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:30:40.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:40.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:30:40.726 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:30:40.738 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:30:40.738 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:30:40.738 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:30:40.738 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:30:40.738 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:30:40.738 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:30:40.738 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:30:40.738 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:30:40.738 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:30:40.739 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:30:40.739 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:30:40.739 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:30:40.746 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.3 to 1 (10000) 2024-04-20T17:30:40.800 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:30:40.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:40 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2801699498' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:30:40.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:40 smithi171 ceph-mon[44189]: from='client.26542 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:30:40.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:40 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2442403146' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:30:40.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:40.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:40.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:40.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:40.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:40.838 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:40.864 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:30:40.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:40.870 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:30:40.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:40 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2801699498' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:30:40.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:40 smithi005 ceph-mon[39562]: from='client.26542 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:30:40.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:40 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2442403146' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:30:40.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:40.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:40.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:40.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:40.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:40.972 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:30:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:40 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2801699498' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:30:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:40 smithi106 ceph-mon[44423]: from='client.26542 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:30:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:40 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2442403146' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:30:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:41.012 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:41.040 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:30:41.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:41.043 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:30:41.144 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:30:41.215 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:41.227 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 61 16 87159 87143 89.2761 109.312 0.00367684 0.0111989 2024-04-20T17:30:41.484 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:30:41.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:41.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:30:41.589 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:30:41.622 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:41.649 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:30:41.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:41.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:30:41.754 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:30:41.787 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:41.810 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:30:41.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:41.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:30:41.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:41 smithi005 ceph-mon[39562]: pgmap v1501: 17 pgs: 17 active+clean; 4.9 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 105 MiB/s wr, 1.69k op/s 2024-04-20T17:30:41.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:41.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:41.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:41.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:41.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:41.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:41.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:41.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:41 smithi005 ceph-mon[39562]: from='client.26542 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-20T17:30:41.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:41 smithi005 ceph-mon[39562]: osdmap e575: 12 total, 12 up, 11 in 2024-04-20T17:30:41.916 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:30:41.955 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:41.983 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:30:41.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:41.991 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:41 smithi171 ceph-mon[44189]: pgmap v1501: 17 pgs: 17 active+clean; 4.9 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 105 MiB/s wr, 1.69k op/s 2024-04-20T17:30:41.991 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:41.991 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:41.991 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:41.991 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:41.991 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:41.991 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:41.992 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:41.992 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:41 smithi171 ceph-mon[44189]: from='client.26542 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-20T17:30:41.992 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:41 smithi171 ceph-mon[44189]: osdmap e575: 12 total, 12 up, 11 in 2024-04-20T17:30:41.992 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:30:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:41 smithi106 ceph-mon[44423]: pgmap v1501: 17 pgs: 17 active+clean; 4.9 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 105 MiB/s wr, 1.69k op/s 2024-04-20T17:30:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:42.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:42.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:42.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:42.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:41 smithi106 ceph-mon[44423]: from='client.26542 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-20T17:30:42.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:41 smithi106 ceph-mon[44423]: osdmap e575: 12 total, 12 up, 11 in 2024-04-20T17:30:42.093 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:30:42.132 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:42.163 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:30:42.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:42.168 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:30:42.227 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 62 15 89067 89052 89.7604 119.312 0.00411376 0.0111393 2024-04-20T17:30:42.269 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:30:42.310 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:42.338 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:30:42.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:42.344 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:30:42.445 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:30:42.477 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:42.504 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:30:42.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:42.510 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:30:42.611 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:30:42.654 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:42.683 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:30:42.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:42.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:30:42.788 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:30:42.835 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:43.227 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 63 16 90971 90955 90.2233 118.938 0.0104901 0.0110821 2024-04-20T17:30:43.502 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 4 1 2024-04-20T17:30:43.896 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:30:43.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:43.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:30:44.002 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:30:44.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:43 smithi106 ceph-mon[44423]: pgmap v1503: 17 pgs: 17 active+clean; 5.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-04-20T17:30:44.033 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:44.063 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:30:44.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:44.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:30:44.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:43 smithi171 ceph-mon[44189]: pgmap v1503: 17 pgs: 17 active+clean; 5.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-04-20T17:30:44.167 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:30:44.203 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:44.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:43 smithi005 ceph-mon[39562]: pgmap v1503: 17 pgs: 17 active+clean; 5.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-04-20T17:30:44.223 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:30:44.227 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 64 15 92756 92741 90.5576 111.625 0.00977929 0.0110415 2024-04-20T17:30:44.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:44.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:30:44.275 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:30:44.275 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:30:44.275 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:30:44.275 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:30:44.276 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:30:44.329 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:30:44.370 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:44.399 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:30:44.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:44.403 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:30:44.504 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:30:44.564 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:44.629 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:30:44.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:44.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:30:44.643 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:30:44.736 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:30:44.765 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:44.793 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:30:44.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:44.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:30:44.899 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:30:45.010 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:45.227 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 65 16 94459 94443 90.8008 106.375 0.00493761 0.0110118 2024-04-20T17:30:45.400 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:30:45.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:45.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:30:45.411 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-20T17:30:45.508 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:30:45.535 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:30:45.542 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:45.563 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:30:45.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:45.567 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:30:45.574 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:30:45.668 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:30:45.717 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:45.760 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:30:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:45 smithi171 ceph-mon[44189]: pgmap v1504: 17 pgs: 17 active+clean; 5.5 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-04-20T17:30:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:45 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2938091713' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:30:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:45 smithi171 ceph-mon[44189]: from='client.26663 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:30:45.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:45 smithi005 ceph-mon[39562]: pgmap v1504: 17 pgs: 17 active+clean; 5.5 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-04-20T17:30:45.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:45 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2938091713' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:30:45.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:45 smithi005 ceph-mon[39562]: from='client.26663 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:30:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:45 smithi106 ceph-mon[44423]: pgmap v1504: 17 pgs: 17 active+clean; 5.5 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-04-20T17:30:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:45 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2938091713' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:30:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:45 smithi106 ceph-mon[44423]: from='client.26663 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:30:46.228 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 66 16 96109 96093 90.9874 103.125 0.00436415 0.010989 2024-04-20T17:30:46.715 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:30:46.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:46.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:30:46.824 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:30:46.858 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:46.893 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:30:46.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:46.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:30:46.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:46 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:30:46.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:46 smithi005 ceph-mon[39562]: from='client.26663 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:30:46.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:46 smithi005 ceph-mon[39562]: osdmap e576: 12 total, 12 up, 11 in 2024-04-20T17:30:46.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:46 smithi005 ceph-mon[39562]: 5.9 scrub starts 2024-04-20T17:30:46.998 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:30:47.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:46 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:30:47.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:46 smithi106 ceph-mon[44423]: from='client.26663 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:30:47.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:46 smithi106 ceph-mon[44423]: osdmap e576: 12 total, 12 up, 11 in 2024-04-20T17:30:47.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:46 smithi106 ceph-mon[44423]: 5.9 scrub starts 2024-04-20T17:30:47.034 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:46 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:30:47.035 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:46 smithi171 ceph-mon[44189]: from='client.26663 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:30:47.035 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:46 smithi171 ceph-mon[44189]: osdmap e576: 12 total, 12 up, 11 in 2024-04-20T17:30:47.035 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:46 smithi171 ceph-mon[44189]: 5.9 scrub starts 2024-04-20T17:30:47.037 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:47.067 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:30:47.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:47.073 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:30:47.174 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:30:47.211 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:47.228 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 67 16 97731 97715 91.1423 101.375 0.00634833 0.0109693 2024-04-20T17:30:47.235 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:30:47.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:47.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:30:47.338 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:30:47.394 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:47 smithi106 ceph-mon[44423]: 5.8 scrub starts 2024-04-20T17:30:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:47 smithi106 ceph-mon[44423]: pgmap v1506: 17 pgs: 17 active+clean; 5.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-04-20T17:30:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:47 smithi106 ceph-mon[44423]: 5.e scrub starts 2024-04-20T17:30:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:47 smithi106 ceph-mon[44423]: 5.6 scrub starts 2024-04-20T17:30:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:47 smithi106 ceph-mon[44423]: 5.9 scrub ok 2024-04-20T17:30:48.067 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:30:48.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:48.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:30:48.077 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:47 smithi005 ceph-mon[39562]: 5.8 scrub starts 2024-04-20T17:30:48.077 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:47 smithi005 ceph-mon[39562]: pgmap v1506: 17 pgs: 17 active+clean; 5.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-04-20T17:30:48.077 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:47 smithi005 ceph-mon[39562]: 5.e scrub starts 2024-04-20T17:30:48.077 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:47 smithi005 ceph-mon[39562]: 5.6 scrub starts 2024-04-20T17:30:48.077 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:47 smithi005 ceph-mon[39562]: 5.9 scrub ok 2024-04-20T17:30:48.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:47 smithi171 ceph-mon[44189]: 5.8 scrub starts 2024-04-20T17:30:48.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:47 smithi171 ceph-mon[44189]: pgmap v1506: 17 pgs: 17 active+clean; 5.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-04-20T17:30:48.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:47 smithi171 ceph-mon[44189]: 5.e scrub starts 2024-04-20T17:30:48.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:47 smithi171 ceph-mon[44189]: 5.6 scrub starts 2024-04-20T17:30:48.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:47 smithi171 ceph-mon[44189]: 5.9 scrub ok 2024-04-20T17:30:48.174 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:30:48.228 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 68 16 99152 99136 91.1079 88.8125 0.00352971 0.0109741 2024-04-20T17:30:48.234 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:48.288 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:30:48.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:48.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:30:48.398 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:30:48.454 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:49.038 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:30:49.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:49.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:30:49.042 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:48 smithi005 ceph-mon[39562]: 5.d scrub starts 2024-04-20T17:30:49.043 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:48 smithi005 ceph-mon[39562]: 5.8 scrub ok 2024-04-20T17:30:49.043 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:48 smithi005 ceph-mon[39562]: 5.e scrub ok 2024-04-20T17:30:49.043 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:48 smithi005 ceph-mon[39562]: 5.6 scrub ok 2024-04-20T17:30:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:48 smithi171 ceph-mon[44189]: 5.d scrub starts 2024-04-20T17:30:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:48 smithi171 ceph-mon[44189]: 5.8 scrub ok 2024-04-20T17:30:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:48 smithi171 ceph-mon[44189]: 5.e scrub ok 2024-04-20T17:30:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:48 smithi171 ceph-mon[44189]: 5.6 scrub ok 2024-04-20T17:30:49.143 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:30:49.178 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:48 smithi106 ceph-mon[44423]: 5.d scrub starts 2024-04-20T17:30:49.178 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:48 smithi106 ceph-mon[44423]: 5.8 scrub ok 2024-04-20T17:30:49.179 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:48 smithi106 ceph-mon[44423]: 5.e scrub ok 2024-04-20T17:30:49.179 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:48 smithi106 ceph-mon[44423]: 5.6 scrub ok 2024-04-20T17:30:49.186 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:49.205 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:30:49.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:49.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:30:49.228 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 69 16 100760 100744 91.2439 100.5 0.00828468 0.0109579 2024-04-20T17:30:49.312 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:30:49.350 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:49.380 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:30:49.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:49.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:30:49.484 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:30:49.517 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:49.547 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:30:49.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:49.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:30:49.627 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:30:49.651 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:30:49.698 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:49.737 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:30:49.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:49.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:30:49.788 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.4 to 1 (10000) 2024-04-20T17:30:49.841 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:30:49.898 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:49 smithi171 ceph-mon[44189]: 5.d scrub ok 2024-04-20T17:30:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:49 smithi171 ceph-mon[44189]: pgmap v1507: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.8 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-04-20T17:30:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:49 smithi171 ceph-mon[44189]: 5.f scrub starts 2024-04-20T17:30:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:49 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2922630074' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-20T17:30:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:49 smithi171 ceph-mon[44189]: from='client.26560 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-20T17:30:50.111 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:30:50.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:49 smithi005 ceph-mon[39562]: 5.d scrub ok 2024-04-20T17:30:50.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:49 smithi005 ceph-mon[39562]: pgmap v1507: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.8 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-04-20T17:30:50.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:49 smithi005 ceph-mon[39562]: 5.f scrub starts 2024-04-20T17:30:50.151 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:49 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2922630074' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-20T17:30:50.151 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:49 smithi005 ceph-mon[39562]: from='client.26560 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-20T17:30:50.228 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 70 16 102362 102346 91.3707 100.125 0.00447309 0.0109428 2024-04-20T17:30:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:49 smithi106 ceph-mon[44423]: 5.d scrub ok 2024-04-20T17:30:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:49 smithi106 ceph-mon[44423]: pgmap v1507: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.8 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-04-20T17:30:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:49 smithi106 ceph-mon[44423]: 5.f scrub starts 2024-04-20T17:30:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:49 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2922630074' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-20T17:30:50.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:49 smithi106 ceph-mon[44423]: from='client.26560 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-20T17:30:50.377 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:30:50.416 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:30:50.453 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:30:50.502 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:30:50.532 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:30:50.564 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:30:50.565 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:30:50.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:50.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:30:50.602 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:30:50.638 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:30:50.670 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:30:50.674 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:30:50.697 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:50.717 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:30:50.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:50.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:30:50.722 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:30:50.762 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:30:50.798 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:30:50.808 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:30:50.808 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:30:50.808 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:30:50.808 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:30:50.809 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:30:50.809 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:30:50.809 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:30:50.809 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:30:50.809 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:30:50.809 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:30:50.809 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:30:50.809 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:30:50.821 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:30:50.838 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:50 smithi005 ceph-mon[39562]: from='client.26560 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-20T17:30:50.838 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:50 smithi005 ceph-mon[39562]: osdmap e577: 12 total, 12 up, 11 in 2024-04-20T17:30:50.838 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:50 smithi005 ceph-mon[39562]: 5.f scrub ok 2024-04-20T17:30:50.838 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:50 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/958466246' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:30:50.838 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:50.838 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:50.838 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:50.838 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:50.838 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:50.839 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:50.839 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:50.854 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:50.881 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:30:50.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:50.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:30:50.986 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:30:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:50 smithi106 ceph-mon[44423]: from='client.26560 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-20T17:30:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:50 smithi106 ceph-mon[44423]: osdmap e577: 12 total, 12 up, 11 in 2024-04-20T17:30:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:50 smithi106 ceph-mon[44423]: 5.f scrub ok 2024-04-20T17:30:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:50 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/958466246' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:30:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:51.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:51.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:51.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:51.041 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:51.076 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:30:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:50 smithi171 ceph-mon[44189]: from='client.26560 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-20T17:30:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:50 smithi171 ceph-mon[44189]: osdmap e577: 12 total, 12 up, 11 in 2024-04-20T17:30:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:50 smithi171 ceph-mon[44189]: 5.f scrub ok 2024-04-20T17:30:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:50 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/958466246' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:30:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:51.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:51.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:51.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:51.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:30:51.123 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 5 1 2024-04-20T17:30:51.187 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:30:51.228 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 71 16 103849 103833 91.3926 92.9375 0.011412 0.0109401 2024-04-20T17:30:51.277 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:51.314 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:30:51.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:51.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:30:51.422 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:30:51.490 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:51 smithi171 ceph-mon[44189]: pgmap v1509: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 103 MiB/s wr, 1.64k op/s 2024-04-20T17:30:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:51 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:30:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:51 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:30:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:51 smithi171 ceph-mon[44189]: 5.7 scrub starts 2024-04-20T17:30:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:52.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:51 smithi171 ceph-mon[44189]: osdmap e578: 12 total, 12 up, 11 in 2024-04-20T17:30:52.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:51 smithi171 ceph-mon[44189]: 5.c scrub starts 2024-04-20T17:30:52.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:51 smithi171 ceph-mon[44189]: 5.b scrub starts 2024-04-20T17:30:52.223 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:51 smithi005 ceph-mon[39562]: pgmap v1509: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 103 MiB/s wr, 1.64k op/s 2024-04-20T17:30:52.223 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:52.224 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:51 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:30:52.224 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:51 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:30:52.224 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:52.224 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:52.224 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:52.224 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:51 smithi005 ceph-mon[39562]: 5.7 scrub starts 2024-04-20T17:30:52.224 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:52.224 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:51 smithi005 ceph-mon[39562]: osdmap e578: 12 total, 12 up, 11 in 2024-04-20T17:30:52.224 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:51 smithi005 ceph-mon[39562]: 5.c scrub starts 2024-04-20T17:30:52.225 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:51 smithi005 ceph-mon[39562]: 5.b scrub starts 2024-04-20T17:30:52.228 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 72 16 105260 105244 91.348 88.1875 0.0128657 0.0109461 2024-04-20T17:30:52.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:51 smithi106 ceph-mon[44423]: pgmap v1509: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 103 MiB/s wr, 1.64k op/s 2024-04-20T17:30:52.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:52.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:51 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:30:52.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:51 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:30:52.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:52.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:52.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:52.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:51 smithi106 ceph-mon[44423]: 5.7 scrub starts 2024-04-20T17:30:52.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:30:52.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:51 smithi106 ceph-mon[44423]: osdmap e578: 12 total, 12 up, 11 in 2024-04-20T17:30:52.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:51 smithi106 ceph-mon[44423]: 5.c scrub starts 2024-04-20T17:30:52.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:51 smithi106 ceph-mon[44423]: 5.b scrub starts 2024-04-20T17:30:52.762 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:30:52.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:52.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:30:52.781 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:30:52.858 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:30:52.877 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:30:52.906 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:52.932 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:30:52.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:52.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:30:53.036 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:30:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:52 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:30:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:52 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:30:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:52 smithi171 ceph-mon[44189]: osdmap e579: 12 total, 12 up, 11 in 2024-04-20T17:30:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:52 smithi171 ceph-mon[44189]: 5.7 scrub ok 2024-04-20T17:30:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:52 smithi171 ceph-mon[44189]: 5.c scrub ok 2024-04-20T17:30:53.084 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:53.111 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:30:53.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:53.122 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:52 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:30:53.122 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:52 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:30:53.122 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:52 smithi106 ceph-mon[44423]: osdmap e579: 12 total, 12 up, 11 in 2024-04-20T17:30:53.122 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:52 smithi106 ceph-mon[44423]: 5.7 scrub ok 2024-04-20T17:30:53.122 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:52 smithi106 ceph-mon[44423]: 5.c scrub ok 2024-04-20T17:30:53.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:30:53.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:52 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:30:53.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:52 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:30:53.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:52 smithi005 ceph-mon[39562]: osdmap e579: 12 total, 12 up, 11 in 2024-04-20T17:30:53.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:52 smithi005 ceph-mon[39562]: 5.7 scrub ok 2024-04-20T17:30:53.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:52 smithi005 ceph-mon[39562]: 5.c scrub ok 2024-04-20T17:30:53.223 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:30:53.228 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 73 16 106871 106855 91.4758 100.688 0.00447548 0.01093 2024-04-20T17:30:53.284 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:53.374 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:30:53.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:53.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:30:53.479 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:30:53.523 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:53.546 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:30:53.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:53.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:30:53.652 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:30:53.690 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:53.719 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:30:53.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:53.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:30:53.823 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:30:53.836 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:53 smithi106 ceph-mon[44423]: pgmap v1512: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 6.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-20T17:30:53.836 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:53 smithi106 ceph-mon[44423]: 5.b scrub ok 2024-04-20T17:30:53.836 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:53 smithi106 ceph-mon[44423]: 5.1 scrub starts 2024-04-20T17:30:53.849 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:30:53.887 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:53.938 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:30:53.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:30:53.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:30:54.047 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:30:54.054 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:30:54.054 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:30:54.093 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:30:54.099 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:53 smithi005 ceph-mon[39562]: pgmap v1512: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 6.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-20T17:30:54.099 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:53 smithi005 ceph-mon[39562]: 5.b scrub ok 2024-04-20T17:30:54.100 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:53 smithi005 ceph-mon[39562]: 5.1 scrub starts 2024-04-20T17:30:54.228 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 74 16 108345 108329 91.4845 92.125 0.00825988 0.0109293 2024-04-20T17:30:54.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:53 smithi171 ceph-mon[44189]: pgmap v1512: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 6.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-20T17:30:54.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:53 smithi171 ceph-mon[44189]: 5.b scrub ok 2024-04-20T17:30:54.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:53 smithi171 ceph-mon[44189]: 5.1 scrub starts 2024-04-20T17:30:55.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:54 smithi005 ceph-mon[39562]: 5.1 scrub ok 2024-04-20T17:30:55.228 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 75 16 109998 109982 91.6421 103.312 0.00626362 0.0109104 2024-04-20T17:30:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:54 smithi106 ceph-mon[44423]: 5.1 scrub ok 2024-04-20T17:30:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:54 smithi171 ceph-mon[44189]: 5.1 scrub ok 2024-04-20T17:30:56.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:55 smithi005 ceph-mon[39562]: pgmap v1513: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 117 MiB/s wr, 1.88k op/s 2024-04-20T17:30:56.228 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 76 16 111518 111502 91.6862 95 0.0161153 0.0109052 2024-04-20T17:30:56.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:55 smithi106 ceph-mon[44423]: pgmap v1513: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 117 MiB/s wr, 1.88k op/s 2024-04-20T17:30:56.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:55 smithi171 ceph-mon[44189]: pgmap v1513: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 117 MiB/s wr, 1.88k op/s 2024-04-20T17:30:57.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:56 smithi005 ceph-mon[39562]: 5.2 scrub starts 2024-04-20T17:30:57.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:56 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:30:57.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:56 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:30:57.228 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 77 16 113031 113015 91.7234 94.5625 0.00520758 0.0109012 2024-04-20T17:30:57.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:56 smithi106 ceph-mon[44423]: 5.2 scrub starts 2024-04-20T17:30:57.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:56 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:30:57.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:56 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:30:57.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:56 smithi171 ceph-mon[44189]: 5.2 scrub starts 2024-04-20T17:30:57.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:56 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:30:57.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:56 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:30:58.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:57 smithi005 ceph-mon[39562]: pgmap v1514: 17 pgs: 17 active+clean; 6.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-04-20T17:30:58.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:57 smithi005 ceph-mon[39562]: 5.2 scrub ok 2024-04-20T17:30:58.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:57 smithi005 ceph-mon[39562]: osdmap e580: 12 total, 12 up, 11 in 2024-04-20T17:30:58.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:57 smithi005 ceph-mon[39562]: 5.a scrub starts 2024-04-20T17:30:58.229 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 78 16 114494 114478 91.7197 91.4375 0.00884294 0.0109015 2024-04-20T17:30:58.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:57 smithi106 ceph-mon[44423]: pgmap v1514: 17 pgs: 17 active+clean; 6.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-04-20T17:30:58.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:57 smithi106 ceph-mon[44423]: 5.2 scrub ok 2024-04-20T17:30:58.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:57 smithi106 ceph-mon[44423]: osdmap e580: 12 total, 12 up, 11 in 2024-04-20T17:30:58.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:57 smithi106 ceph-mon[44423]: 5.a scrub starts 2024-04-20T17:30:58.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:57 smithi171 ceph-mon[44189]: pgmap v1514: 17 pgs: 17 active+clean; 6.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-04-20T17:30:58.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:57 smithi171 ceph-mon[44189]: 5.2 scrub ok 2024-04-20T17:30:58.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:57 smithi171 ceph-mon[44189]: osdmap e580: 12 total, 12 up, 11 in 2024-04-20T17:30:58.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:57 smithi171 ceph-mon[44189]: 5.a scrub starts 2024-04-20T17:30:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:58 smithi005 ceph-mon[39562]: 5.3 scrub starts 2024-04-20T17:30:59.228 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 79 16 115630 115614 91.4573 71 0.00422075 0.0109325 2024-04-20T17:30:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:58 smithi106 ceph-mon[44423]: 5.3 scrub starts 2024-04-20T17:30:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:58 smithi171 ceph-mon[44189]: 5.3 scrub starts 2024-04-20T17:31:00.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:59 smithi005 ceph-mon[39562]: pgmap v1516: 17 pgs: 17 active+clean; 6.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-04-20T17:31:00.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:59 smithi005 ceph-mon[39562]: 5.5 scrub starts 2024-04-20T17:31:00.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:30:59 smithi005 ceph-mon[39562]: 5.a scrub ok 2024-04-20T17:31:00.229 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:31:00.229859 min lat: 0.00096363 max lat: 30.3388 avg lat: 0.010933 2024-04-20T17:31:00.229 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:31:00.229 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 80 16 117087 117071 91.4523 91.0625 0.0071219 0.010933 2024-04-20T17:31:00.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:59 smithi106 ceph-mon[44423]: pgmap v1516: 17 pgs: 17 active+clean; 6.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-04-20T17:31:00.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:59 smithi106 ceph-mon[44423]: 5.5 scrub starts 2024-04-20T17:31:00.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:30:59 smithi106 ceph-mon[44423]: 5.a scrub ok 2024-04-20T17:31:00.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:59 smithi171 ceph-mon[44189]: pgmap v1516: 17 pgs: 17 active+clean; 6.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-04-20T17:31:00.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:59 smithi171 ceph-mon[44189]: 5.5 scrub starts 2024-04-20T17:31:00.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:30:59 smithi171 ceph-mon[44189]: 5.a scrub ok 2024-04-20T17:31:00.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:00 smithi005 ceph-mon[39562]: 5.3 scrub ok 2024-04-20T17:31:00.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:00 smithi005 ceph-mon[39562]: 5.4 scrub starts 2024-04-20T17:31:01.230 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 81 16 118537 118521 91.4403 90.625 0.00439911 0.0109347 2024-04-20T17:31:01.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:00 smithi106 ceph-mon[44423]: 5.3 scrub ok 2024-04-20T17:31:01.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:00 smithi106 ceph-mon[44423]: 5.4 scrub starts 2024-04-20T17:31:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:00 smithi171 ceph-mon[44189]: 5.3 scrub ok 2024-04-20T17:31:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:00 smithi171 ceph-mon[44189]: 5.4 scrub starts 2024-04-20T17:31:02.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:01 smithi005 ceph-mon[39562]: pgmap v1517: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-04-20T17:31:02.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:01 smithi005 ceph-mon[39562]: 5.5 scrub ok 2024-04-20T17:31:02.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:01 smithi005 ceph-mon[39562]: 5.0 scrub starts 2024-04-20T17:31:02.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:01 smithi005 ceph-mon[39562]: 5.4 scrub ok 2024-04-20T17:31:02.231 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 82 16 119926 119910 91.3838 86.8125 0.006077 0.0109413 2024-04-20T17:31:02.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:01 smithi106 ceph-mon[44423]: pgmap v1517: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-04-20T17:31:02.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:01 smithi106 ceph-mon[44423]: 5.5 scrub ok 2024-04-20T17:31:02.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:01 smithi106 ceph-mon[44423]: 5.0 scrub starts 2024-04-20T17:31:02.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:01 smithi106 ceph-mon[44423]: 5.4 scrub ok 2024-04-20T17:31:02.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:01 smithi171 ceph-mon[44189]: pgmap v1517: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-04-20T17:31:02.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:01 smithi171 ceph-mon[44189]: 5.5 scrub ok 2024-04-20T17:31:02.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:01 smithi171 ceph-mon[44189]: 5.0 scrub starts 2024-04-20T17:31:02.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:01 smithi171 ceph-mon[44189]: 5.4 scrub ok 2024-04-20T17:31:03.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:02 smithi005 ceph-mon[39562]: 5.0 scrub ok 2024-04-20T17:31:03.230 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 83 15 121500 121485 91.4687 98.4375 0.00625241 0.0109313 2024-04-20T17:31:03.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:02 smithi106 ceph-mon[44423]: 5.0 scrub ok 2024-04-20T17:31:03.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:02 smithi171 ceph-mon[44189]: 5.0 scrub ok 2024-04-20T17:31:04.025 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:31:04.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:04.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:31:04.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:03 smithi005 ceph-mon[39562]: pgmap v1518: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-20T17:31:04.221 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:31:04.230 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 84 16 123070 123054 91.5471 98.0625 0.0121328 0.0109219 2024-04-20T17:31:04.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:03 smithi106 ceph-mon[44423]: pgmap v1518: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-20T17:31:04.271 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:04.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:03 smithi171 ceph-mon[44189]: pgmap v1518: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-20T17:31:05.045 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:31:05.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:05.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:31:05.086 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-20T17:31:05.149 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:31:05.205 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:05.231 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 85 16 124546 124530 91.5552 92.25 0.00989662 0.010921 2024-04-20T17:31:05.231 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:31:05.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:05.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:31:05.335 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:31:05.372 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:05.392 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:31:05.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:05.395 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:31:05.496 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:31:05.526 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:05.549 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:31:05.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:05.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:31:05.653 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:31:05.682 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:05.703 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:31:05.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:05.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:31:05.807 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:31:05.885 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:05.905 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:05 smithi106 ceph-mon[44423]: pgmap v1519: 17 pgs: 17 active+clean; 7.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-20T17:31:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:05 smithi005 ceph-mon[39562]: pgmap v1519: 17 pgs: 17 active+clean; 7.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-20T17:31:06.053 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:31:06.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:06.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:31:06.159 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:31:06.189 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:06.214 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:31:06.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:06.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:31:06.231 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 86 16 125915 125899 91.4855 85.5625 0.00684305 0.0109294 2024-04-20T17:31:06.317 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:31:06.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:05 smithi171 ceph-mon[44189]: pgmap v1519: 17 pgs: 17 active+clean; 7.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-20T17:31:06.349 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:06.373 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:31:06.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:06.378 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:31:06.479 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:31:06.524 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:06.558 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:31:06.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:06.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:31:06.664 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:31:06.710 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:06.737 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:31:06.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:06.739 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:31:06.840 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:31:06.867 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:31:06.922 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:07.231 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 87 16 127303 127287 91.431 86.75 0.00529403 0.010935 2024-04-20T17:31:07.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:07 smithi171 ceph-mon[44189]: pgmap v1520: 17 pgs: 17 active+clean; 7.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-04-20T17:31:07.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:07 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2756359341' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:31:07.625 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:31:07.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:07 smithi005 ceph-mon[39562]: pgmap v1520: 17 pgs: 17 active+clean; 7.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-04-20T17:31:07.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:07 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2756359341' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:31:07.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:07 smithi106 ceph-mon[44423]: pgmap v1520: 17 pgs: 17 active+clean; 7.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-04-20T17:31:07.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:07 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2756359341' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:31:07.804 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:31:08.231 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 88 16 128683 128667 91.372 86.25 0.0042816 0.0109426 2024-04-20T17:31:08.239 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:31:08.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:08.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:31:08.343 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:31:08.373 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:08.399 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:31:08.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:08.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:31:08.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:08 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:31:08.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:08 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:31:08.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:08 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2756359341' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:31:08.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:08 smithi005 ceph-mon[39562]: osdmap e581: 12 total, 12 up, 11 in 2024-04-20T17:31:08.504 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:31:08.568 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:08.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:08 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:31:08.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:08 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:31:08.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:08 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2756359341' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:31:08.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:08 smithi171 ceph-mon[44189]: osdmap e581: 12 total, 12 up, 11 in 2024-04-20T17:31:08.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:08 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:31:08.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:08 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:31:08.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:08 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2756359341' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:31:08.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:08 smithi106 ceph-mon[44423]: osdmap e581: 12 total, 12 up, 11 in 2024-04-20T17:31:08.798 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:31:08.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:08.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:31:08.905 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:31:08.941 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:08.967 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:31:08.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:08.971 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:31:09.072 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:31:09.108 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:09.133 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:31:09.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:09.139 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:31:09.231 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 89 16 130023 130007 91.2863 83.75 0.0106375 0.010953 2024-04-20T17:31:09.239 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:31:09.269 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:09.288 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:09 smithi171 ceph-mon[44189]: pgmap v1522: 17 pgs: 17 active+clean; 7.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-04-20T17:31:09.288 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:09 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1130405932' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:31:09.288 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:09 smithi171 ceph-mon[44189]: from='client.26575 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:31:09.293 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:31:09.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:09.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:31:09.314 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.5 to 1 (10000) 2024-04-20T17:31:09.397 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:31:09.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:09 smithi005 ceph-mon[39562]: pgmap v1522: 17 pgs: 17 active+clean; 7.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-04-20T17:31:09.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:09 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1130405932' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:31:09.449 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:09 smithi005 ceph-mon[39562]: from='client.26575 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:31:09.452 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:09.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:09 smithi106 ceph-mon[44423]: pgmap v1522: 17 pgs: 17 active+clean; 7.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-04-20T17:31:09.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:09 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1130405932' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:31:09.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:09 smithi106 ceph-mon[44423]: from='client.26575 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:31:09.943 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:31:09.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:09.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:31:10.051 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:31:10.094 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:10.118 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:31:10.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:10.122 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:31:10.223 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:31:10.231 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Total time run: 90.0102 2024-04-20T17:31:10.231 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Total writes made: 131431 2024-04-20T17:31:10.231 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Write size: 65536 2024-04-20T17:31:10.231 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Object size: 65536 2024-04-20T17:31:10.231 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Bandwidth (MB/sec): 91.2612 2024-04-20T17:31:10.231 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Stddev Bandwidth: 80.0947 2024-04-20T17:31:10.231 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Max bandwidth (MB/sec): 335.188 2024-04-20T17:31:10.231 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Min bandwidth (MB/sec): 0 2024-04-20T17:31:10.232 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Average IOPS: 1460 2024-04-20T17:31:10.232 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Stddev IOPS: 1281.52 2024-04-20T17:31:10.232 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Max IOPS: 5363 2024-04-20T17:31:10.232 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Min IOPS: 0 2024-04-20T17:31:10.232 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Average Latency(s): 0.0109569 2024-04-20T17:31:10.232 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Stddev Latency(s): 0.33445 2024-04-20T17:31:10.232 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Max latency(s): 30.3388 2024-04-20T17:31:10.232 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Min latency(s): 0.00096363 2024-04-20T17:31:10.238 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Cleaning up (deleting benchmark objects) 2024-04-20T17:31:10.243 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:31:10.259 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:10.532 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:10 smithi005 ceph-mon[39562]: from='client.26575 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-20T17:31:10.532 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:10 smithi005 ceph-mon[39562]: osdmap e582: 12 total, 12 up, 11 in 2024-04-20T17:31:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:10 smithi171 ceph-mon[44189]: from='client.26575 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-20T17:31:10.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:10 smithi171 ceph-mon[44189]: osdmap e582: 12 total, 12 up, 11 in 2024-04-20T17:31:10.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:10 smithi106 ceph-mon[44423]: from='client.26575 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-20T17:31:10.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:10 smithi106 ceph-mon[44423]: osdmap e582: 12 total, 12 up, 11 in 2024-04-20T17:31:11.005 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 8 1 2024-04-20T17:31:11.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:11 smithi171 ceph-mon[44189]: pgmap v1524: 17 pgs: 17 active+clean; 7.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-04-20T17:31:11.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:11 smithi005 ceph-mon[39562]: pgmap v1524: 17 pgs: 17 active+clean; 7.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-04-20T17:31:11.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:11 smithi106 ceph-mon[44423]: pgmap v1524: 17 pgs: 17 active+clean; 7.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-04-20T17:31:11.848 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:31:11.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:11.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:31:11.954 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:31:11.993 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:12.017 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:31:12.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:12.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:31:12.121 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:31:12.161 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:12.192 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:31:12.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:12.199 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:31:12.300 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:31:12.336 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:12.359 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:31:12.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:12.363 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:31:12.414 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:12 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:31:12.414 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:12 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:31:12.414 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:12 smithi005 ceph-mon[39562]: osdmap e583: 12 total, 12 up, 11 in 2024-04-20T17:31:12.464 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:31:12.498 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:12 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:31:12.498 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:12 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:31:12.498 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:12 smithi171 ceph-mon[44189]: osdmap e583: 12 total, 12 up, 11 in 2024-04-20T17:31:12.500 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:12.527 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:31:12.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:12.533 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:31:12.634 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:31:12.658 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:12 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:31:12.659 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:12 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:31:12.659 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:12 smithi106 ceph-mon[44423]: osdmap e583: 12 total, 12 up, 11 in 2024-04-20T17:31:12.661 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:12.683 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:31:12.685 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:31:12.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:12.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:31:12.731 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:31:12.765 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:31:12.772 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:31:12.773 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:31:12.773 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-04-20T17:31:12.773 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:31:12.791 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:31:12.793 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:31:12.797 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:31:12.805 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:31:12.820 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:12.841 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:31:12.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:12.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:31:12.853 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:31:12.889 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:31:12.924 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:31:12.944 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:31:12.961 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:31:12.976 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:12.993 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:31:12.996 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:31:12.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:12.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:31:13.030 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:31:13.071 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:31:13.100 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:31:13.101 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:31:13.111 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:31:13.111 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:31:13.111 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:31:13.111 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:31:13.111 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:31:13.112 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:31:13.112 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:31:13.112 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:31:13.112 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:31:13.112 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:31:13.112 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:31:13.112 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:31:13.130 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:13.159 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:31:13.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:13.163 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:31:13.264 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:31:13.292 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:13.316 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:31:13.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:13.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:31:13.420 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:31:13.453 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:13.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:13 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:31:13.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:13 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:31:13.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:13 smithi005 ceph-mon[39562]: pgmap v1526: 17 pgs: 17 active+clean; 7.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 68 MiB/s wr, 1.17k op/s 2024-04-20T17:31:13.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:13 smithi005 ceph-mon[39562]: osdmap e584: 12 total, 12 up, 11 in 2024-04-20T17:31:13.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:13 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1192747420' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:31:13.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:13 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:13.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:13 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:13.476 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:31:13.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:13.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:31:13.580 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:31:13.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:13 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:31:13.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:13 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:31:13.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:13 smithi171 ceph-mon[44189]: pgmap v1526: 17 pgs: 17 active+clean; 7.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 68 MiB/s wr, 1.17k op/s 2024-04-20T17:31:13.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:13 smithi171 ceph-mon[44189]: osdmap e584: 12 total, 12 up, 11 in 2024-04-20T17:31:13.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:13 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1192747420' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:31:13.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:13 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:13.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:13 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:13.645 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:13.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:13 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:31:13.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:13 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:31:13.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:13 smithi106 ceph-mon[44423]: pgmap v1526: 17 pgs: 17 active+clean; 7.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 68 MiB/s wr, 1.17k op/s 2024-04-20T17:31:13.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:13 smithi106 ceph-mon[44423]: osdmap e584: 12 total, 12 up, 11 in 2024-04-20T17:31:13.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:13 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1192747420' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:31:13.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:13 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:13.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:13 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:13.869 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-20T17:31:14.611 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.611 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.611 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.611 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.611 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.611 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.611 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.611 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.611 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.611 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.614 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:31:14.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:14.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:31:14.724 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:31:14.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.771 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.771 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.772 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.772 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.772 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.772 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.772 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.772 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.772 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.772 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:14.772 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:14.798 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:31:14.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:14.801 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:31:14.902 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:31:14.933 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:14.957 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:31:14.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:14.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:31:15.060 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:31:15.088 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:15.112 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:31:15.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:15.115 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:31:15.216 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:31:15.248 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:15.275 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:31:15.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:15.282 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:31:15.383 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:31:15.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:15 smithi005 ceph-mon[39562]: pgmap v1528: 17 pgs: 17 active+clean; 7.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 170 B/s rd, 73 MiB/s wr, 1.67k op/s 2024-04-20T17:31:15.443 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:15.478 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:31:15.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:15.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:31:15.584 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:31:15.618 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:15.620 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:15 smithi106 ceph-mon[44423]: pgmap v1528: 17 pgs: 17 active+clean; 7.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 170 B/s rd, 73 MiB/s wr, 1.67k op/s 2024-04-20T17:31:15.642 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:31:15.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:15.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:31:15.748 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:31:15.800 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:15.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:15 smithi171 ceph-mon[44189]: pgmap v1528: 17 pgs: 17 active+clean; 7.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 170 B/s rd, 73 MiB/s wr, 1.67k op/s 2024-04-20T17:31:15.849 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:31:15.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:15.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:31:15.967 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:31:15.998 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:16.023 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:31:16.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:16.032 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:31:16.134 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:31:16.179 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:16.206 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:31:16.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:16.210 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:31:16.311 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:31:16.364 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:16.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:16 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:31:16.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:16 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:31:16.609 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:31:16.641 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:31:16.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:16.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:31:16.749 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:31:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:16 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:31:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:16 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:31:16.783 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:16.784 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:16 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:31:16.784 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:16 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:31:16.809 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:31:16.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:16.812 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:31:16.913 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:31:17.001 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:17.225 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:31:17.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:17.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:31:17.330 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:31:17.360 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:17.386 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:31:17.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:17.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:31:17.490 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:31:17.551 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:17 smithi005 ceph-mon[39562]: pgmap v1529: 17 pgs: 17 active+clean; 7.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 145 B/s rd, 35 MiB/s wr, 1.50k op/s 2024-04-20T17:31:17.551 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:17 smithi005 ceph-mon[39562]: osdmap e585: 12 total, 12 up, 11 in 2024-04-20T17:31:17.554 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:17.631 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:31:17.635 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:31:17.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:17.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:31:17.741 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:31:17.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:17 smithi106 ceph-mon[44423]: pgmap v1529: 17 pgs: 17 active+clean; 7.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 145 B/s rd, 35 MiB/s wr, 1.50k op/s 2024-04-20T17:31:17.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:17 smithi106 ceph-mon[44423]: osdmap e585: 12 total, 12 up, 11 in 2024-04-20T17:31:17.768 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:17.790 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:31:17.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:17.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:31:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:17 smithi171 ceph-mon[44189]: pgmap v1529: 17 pgs: 17 active+clean; 7.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 145 B/s rd, 35 MiB/s wr, 1.50k op/s 2024-04-20T17:31:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:17 smithi171 ceph-mon[44189]: osdmap e585: 12 total, 12 up, 11 in 2024-04-20T17:31:17.894 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:31:17.928 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:17.956 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:31:17.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:17.960 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:31:18.061 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:31:18.091 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:18.117 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:31:18.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:18.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:31:18.221 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:31:18.255 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:18.293 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:31:18.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:18.298 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:31:18.377 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:18 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/214665091' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:31:18.377 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:18 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1550435934' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-20T17:31:18.377 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:18 smithi005 ceph-mon[39562]: from='client.26699 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-20T17:31:18.378 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:18 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/214665091' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:31:18.378 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:18 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1550435934' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-20T17:31:18.378 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:18 smithi106 ceph-mon[44423]: from='client.26699 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-20T17:31:18.393 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.8 to 1 (10000) 2024-04-20T17:31:18.399 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:31:18.409 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:31:18.470 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:18.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:18 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/214665091' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:31:18.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:18 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1550435934' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-20T17:31:18.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:18 smithi171 ceph-mon[44189]: from='client.26699 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-20T17:31:19.343 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 9 1 2024-04-20T17:31:19.543 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:31:19.558 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:31:19.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:19.579 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:19 smithi005 ceph-mon[39562]: pgmap v1531: 17 pgs: 17 active+clean; 7.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 146 B/s rd, 8.3 MiB/s wr, 1.06k op/s 2024-04-20T17:31:19.579 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:19 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:31:19.579 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:19 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/214665091' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:31:19.579 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:19 smithi005 ceph-mon[39562]: from='client.26699 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-20T17:31:19.579 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:19 smithi005 ceph-mon[39562]: osdmap e586: 12 total, 12 up, 11 in 2024-04-20T17:31:19.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:31:19.681 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:31:19.745 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:19.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:19 smithi106 ceph-mon[44423]: pgmap v1531: 17 pgs: 17 active+clean; 7.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 146 B/s rd, 8.3 MiB/s wr, 1.06k op/s 2024-04-20T17:31:19.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:19 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:31:19.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:19 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/214665091' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:31:19.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:19 smithi106 ceph-mon[44423]: from='client.26699 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-20T17:31:19.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:19 smithi106 ceph-mon[44423]: osdmap e586: 12 total, 12 up, 11 in 2024-04-20T17:31:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:19 smithi171 ceph-mon[44189]: pgmap v1531: 17 pgs: 17 active+clean; 7.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 146 B/s rd, 8.3 MiB/s wr, 1.06k op/s 2024-04-20T17:31:19.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:19 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:31:19.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:19 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/214665091' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:31:19.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:19 smithi171 ceph-mon[44189]: from='client.26699 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-20T17:31:19.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:19 smithi171 ceph-mon[44189]: osdmap e586: 12 total, 12 up, 11 in 2024-04-20T17:31:20.013 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:31:20.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:20.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:31:20.121 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:31:20.153 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:20.176 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:31:20.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:20.179 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:31:20.280 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:31:20.340 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:20.589 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:31:20.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:20.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:31:20.696 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:31:20.738 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:20.845 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:31:20.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:20.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:31:20.948 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:31:20.981 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:21.002 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:31:21.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:21.005 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:31:21.106 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:31:21.143 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:21.176 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:31:21.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:21.183 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:31:21.284 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:31:21.355 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:21.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:21 smithi005 ceph-mon[39562]: pgmap v1533: 17 pgs: 17 active+clean; 7.4 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 128 B/s rd, 7.3 MiB/s wr, 1.30k op/s 2024-04-20T17:31:21.393 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:31:21.393 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:31:21.393 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:31:21.393 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:31:21.393 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:31:21.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:21 smithi106 ceph-mon[44423]: pgmap v1533: 17 pgs: 17 active+clean; 7.4 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 128 B/s rd, 7.3 MiB/s wr, 1.30k op/s 2024-04-20T17:31:21.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:21 smithi171 ceph-mon[44189]: pgmap v1533: 17 pgs: 17 active+clean; 7.4 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 128 B/s rd, 7.3 MiB/s wr, 1.30k op/s 2024-04-20T17:31:22.436 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:31:22.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:22.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:31:22.543 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:31:22.576 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:22.602 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:31:22.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:22.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:31:22.629 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:31:22.711 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:31:22.741 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:22.769 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:31:22.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:22.771 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:31:22.873 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:31:22.911 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:22.940 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:31:22.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:22.946 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:31:23.047 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:31:23.114 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:23.150 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-20T17:31:23.331 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:31:23.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:23.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:31:23.441 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:31:23.496 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:23.543 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:31:23.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:23.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:31:23.652 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:31:23.664 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:31:23.670 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:31:23.679 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:23 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:31:23.679 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:23 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:31:23.679 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:23 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:31:23.679 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:23 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:31:23.681 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:23.705 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:31:23.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:23.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:31:23.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:23 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:31:23.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:23 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:31:23.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:23 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:31:23.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:23 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:31:23.776 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:31:23.810 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:31:23.826 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:31:23.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:23 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:31:23.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:23 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:31:23.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:23 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:31:23.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:23 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:31:23.841 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:23.861 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:31:23.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:23.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:31:23.866 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:31:23.899 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:31:23.927 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:31:23.964 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:31:23.965 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:31:23.994 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:24.012 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:31:24.016 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:31:24.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:24.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:31:24.042 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:31:24.067 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:31:24.126 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:31:24.133 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:31:24.161 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:31:24.181 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:24.200 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:31:24.209 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:31:24.212 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:31:24.213 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:31:24.213 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:31:24.213 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:31:24.213 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:31:24.213 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:31:24.213 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:31:24.213 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:31:24.213 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:31:24.213 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:31:24.213 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:31:24.213 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:31:24.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:24.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:31:24.315 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:31:24.350 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:24.379 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:31:24.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:24.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:31:24.486 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:31:24.519 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:24.543 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:31:24.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:24.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:31:24.649 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:31:24.679 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:24 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:31:24.679 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:24 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:31:24.679 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:24 smithi106 ceph-mon[44423]: osdmap e587: 12 total, 12 up, 11 in 2024-04-20T17:31:24.679 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:24 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:31:24.679 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:24 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:31:24.679 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:24 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:31:24.679 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:24 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1768170371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:31:24.679 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:24 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:31:24.679 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:31:24.680 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:31:24.680 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:24 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:31:24.680 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:24 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:24.680 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:24 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:24.681 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:24.712 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:31:24.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:24 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:31:24.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:24 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:31:24.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:24 smithi005 ceph-mon[39562]: osdmap e587: 12 total, 12 up, 11 in 2024-04-20T17:31:24.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:24 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:31:24.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:24 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:31:24.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:24 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:31:24.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:24 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1768170371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:31:24.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:24 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:31:24.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:24 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:31:24.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:24 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:31:24.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:24 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:31:24.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:24 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:24.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:24 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:24.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:24.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:31:24.817 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:31:24.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:24 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:31:24.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:24 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:31:24.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:24 smithi171 ceph-mon[44189]: osdmap e587: 12 total, 12 up, 11 in 2024-04-20T17:31:24.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:24 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:31:24.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:24 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:31:24.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:24 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:31:24.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:24 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1768170371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:31:24.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:24 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:31:24.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:24 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:31:24.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:24 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:31:24.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:24 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:31:24.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:24 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:24.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:24 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:24.880 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:25.217 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:31:25.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:25.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:31:25.333 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:31:25.385 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:25.523 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:31:25.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:25.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:31:25.627 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:31:25.670 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:25.673 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.673 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.674 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.674 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.674 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.674 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.674 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.674 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.674 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.674 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:25 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.674 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:25 smithi005 ceph-mon[39562]: pgmap v1536: 17 pgs: 17 active+clean; 7.0 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-20T17:31:25.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:25 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:25 smithi106 ceph-mon[44423]: pgmap v1536: 17 pgs: 17 active+clean; 7.0 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-20T17:31:25.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:25 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:25.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:25 smithi171 ceph-mon[44189]: pgmap v1536: 17 pgs: 17 active+clean; 7.0 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-20T17:31:26.839 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:31:26.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:26.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:31:26.950 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:31:26.986 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:27.012 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:31:27.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:27.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:31:27.117 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:31:27.148 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:27.174 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:31:27.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:27.178 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:31:27.279 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:31:27.331 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:27.459 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:31:27.463 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:31:27.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:27.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:31:27.568 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:31:27.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:27 smithi171 ceph-mon[44189]: pgmap v1537: 17 pgs: 17 active+clean; 6.8 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.59k op/s 2024-04-20T17:31:27.594 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:27 smithi106 ceph-mon[44423]: pgmap v1537: 17 pgs: 17 active+clean; 6.8 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.59k op/s 2024-04-20T17:31:27.597 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:27.620 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:31:27.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:27.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:31:27.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:27 smithi005 ceph-mon[39562]: pgmap v1537: 17 pgs: 17 active+clean; 6.8 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.59k op/s 2024-04-20T17:31:27.724 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:31:27.758 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:27.782 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:31:27.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:27.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:31:27.891 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:31:27.921 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:27.947 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:31:27.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:27.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:31:28.050 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:31:28.093 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:28.140 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:31:28.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:28.149 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:31:28.250 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:31:28.298 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:28.340 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.9 to 1 (10000) 2024-04-20T17:31:28.400 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:31:28.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:28 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:31:28.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:28 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:31:28.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:28 smithi005 ceph-mon[39562]: osdmap e588: 12 total, 12 up, 11 in 2024-04-20T17:31:28.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:28 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3445969327' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:31:28.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:28 smithi005 ceph-mon[39562]: from='client.26708 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:31:28.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:28 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:31:28.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:28 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:31:28.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:28 smithi171 ceph-mon[44189]: osdmap e588: 12 total, 12 up, 11 in 2024-04-20T17:31:28.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:28 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3445969327' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:31:28.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:28 smithi171 ceph-mon[44189]: from='client.26708 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:31:28.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:28 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:31:28.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:28 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:31:28.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:28 smithi106 ceph-mon[44423]: osdmap e588: 12 total, 12 up, 11 in 2024-04-20T17:31:28.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:28 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3445969327' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:31:28.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:28 smithi106 ceph-mon[44423]: from='client.26708 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:31:29.354 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:31:29.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:29 smithi005 ceph-mon[39562]: from='client.26708 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-20T17:31:29.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:29 smithi005 ceph-mon[39562]: pgmap v1540: 17 pgs: 17 active+clean; 6.8 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-20T17:31:29.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:29 smithi005 ceph-mon[39562]: osdmap e589: 12 total, 12 up, 11 in 2024-04-20T17:31:29.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:29 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3930738404' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:31:29.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:29 smithi005 ceph-mon[39562]: from='client.26714 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:31:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:29 smithi171 ceph-mon[44189]: from='client.26708 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-20T17:31:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:29 smithi171 ceph-mon[44189]: pgmap v1540: 17 pgs: 17 active+clean; 6.8 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-20T17:31:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:29 smithi171 ceph-mon[44189]: osdmap e589: 12 total, 12 up, 11 in 2024-04-20T17:31:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:29 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3930738404' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:31:29.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:29 smithi171 ceph-mon[44189]: from='client.26714 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:31:29.696 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:31:29.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:29.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:31:29.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:29 smithi106 ceph-mon[44423]: from='client.26708 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-20T17:31:29.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:29 smithi106 ceph-mon[44423]: pgmap v1540: 17 pgs: 17 active+clean; 6.8 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-20T17:31:29.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:29 smithi106 ceph-mon[44423]: osdmap e589: 12 total, 12 up, 11 in 2024-04-20T17:31:29.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:29 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3930738404' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:31:29.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:29 smithi106 ceph-mon[44423]: from='client.26714 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:31:29.808 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:31:29.837 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:29.863 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:31:29.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:29.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:31:29.968 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:31:30.013 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:30.106 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:31:30.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:30.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:31:30.140 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 10 1 2024-04-20T17:31:30.215 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:31:30.244 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:30.267 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:31:30.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:30.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:31:30.373 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:31:30.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:30 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:31:30.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:30 smithi005 ceph-mon[39562]: from='client.26714 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:31:30.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:30 smithi005 ceph-mon[39562]: osdmap e590: 12 total, 12 up, 11 in 2024-04-20T17:31:30.465 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:30.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:30 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:31:30.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:30 smithi106 ceph-mon[44423]: from='client.26714 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:31:30.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:30 smithi106 ceph-mon[44423]: osdmap e590: 12 total, 12 up, 11 in 2024-04-20T17:31:30.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:30 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:31:30.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:30 smithi171 ceph-mon[44189]: from='client.26714 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:31:30.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:30 smithi171 ceph-mon[44189]: osdmap e590: 12 total, 12 up, 11 in 2024-04-20T17:31:30.932 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:31:30.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:30.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:31:31.038 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:31:31.067 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:31.090 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:31:31.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:31.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:31:31.193 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:31:31.224 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:31.247 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:31:31.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:31.251 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:31:31.264 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:31:31.294 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:31:31.294 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:31:31.297 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:31:31.352 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:31:31.364 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:31 smithi106 ceph-mon[44423]: pgmap v1542: 17 pgs: 17 active+clean; 6.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-20T17:31:31.387 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:31.409 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:31:31.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:31.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:31:31.515 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:31:31.569 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:31.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:31 smithi171 ceph-mon[44189]: pgmap v1542: 17 pgs: 17 active+clean; 6.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-20T17:31:31.594 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:31 smithi005 ceph-mon[39562]: pgmap v1542: 17 pgs: 17 active+clean; 6.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-20T17:31:31.838 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:31:31.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:31.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:31:31.945 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:31:31.992 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:32.272 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:31:32.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:32.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:31:32.380 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:31:32.417 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:32.445 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:31:32.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:32.449 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:31:32.549 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:31:32.578 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:32.604 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:31:32.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:32.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:31:32.706 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:31:32.707 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:31:32.735 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:32.757 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:31:32.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:32.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:31:32.863 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:31:32.914 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:33.140 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:31:33.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:33.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:31:33.245 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:31:33.295 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:33.464 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:31:33.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:33.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:31:33.569 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:31:33.610 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:33.681 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:31:33.683 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:33 smithi005 ceph-mon[39562]: pgmap v1543: 17 pgs: 17 active+clean; 6.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-20T17:31:33.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:33.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:31:33.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:33 smithi106 ceph-mon[44423]: pgmap v1543: 17 pgs: 17 active+clean; 6.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-20T17:31:33.792 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:31:33.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:33 smithi171 ceph-mon[44189]: pgmap v1543: 17 pgs: 17 active+clean; 6.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-04-20T17:31:33.841 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:34.100 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:31:34.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:34.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:31:34.213 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:31:34.241 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:34.270 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:31:34.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:34.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:31:34.375 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:31:34.405 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:34.433 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:31:34.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:34.436 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:31:34.537 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:31:34.595 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:34.722 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:31:34.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:34.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:31:34.827 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:31:34.868 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:34.894 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:31:34.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:34.897 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:31:34.998 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:31:35.061 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:35.223 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-20T17:31:35.273 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:31:35.273 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:31:35.274 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:31:35.274 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:31:35.274 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:31:35.274 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:31:35.274 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:31:35.274 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:31:35.274 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:31:35.274 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:31:35.274 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:31:35.277 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:31:35.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:35.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:31:35.316 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:31:35.340 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:31:35.340 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:31:35.341 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:31:35.341 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:31:35.341 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:31:35.341 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:31:35.341 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:31:35.341 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:31:35.341 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:31:35.341 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:31:35.341 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:31:35.341 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:31:35.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:35 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:31:35.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:35 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:31:35.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:35 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:31:35.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:35 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:31:35.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:35 smithi005 ceph-mon[39562]: pgmap v1544: 17 pgs: 17 active+clean; 6.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.35k op/s 2024-04-20T17:31:35.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:35 smithi005 ceph-mon[39562]: osdmap e591: 12 total, 12 up, 11 in 2024-04-20T17:31:35.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:35 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1132005186' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:31:35.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:35.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:35.386 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:31:35.418 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:35.451 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:31:35.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:35.456 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:31:35.557 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:31:35.590 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:35.614 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:31:35.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:35.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:31:35.719 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:31:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:35 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:31:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:35 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:31:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:35 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:31:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:35 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:31:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:35 smithi106 ceph-mon[44423]: pgmap v1544: 17 pgs: 17 active+clean; 6.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.35k op/s 2024-04-20T17:31:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:35 smithi106 ceph-mon[44423]: osdmap e591: 12 total, 12 up, 11 in 2024-04-20T17:31:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:35 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1132005186' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:31:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:35.769 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:35.807 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:31:35.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:35.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:31:35.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:35 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:31:35.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:35 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:31:35.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:35 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:31:35.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:35 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:31:35.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:35 smithi171 ceph-mon[44189]: pgmap v1544: 17 pgs: 17 active+clean; 6.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.35k op/s 2024-04-20T17:31:35.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:35 smithi171 ceph-mon[44189]: osdmap e591: 12 total, 12 up, 11 in 2024-04-20T17:31:35.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:35 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1132005186' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:31:35.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:35.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:35.919 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:31:35.948 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:35.975 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:31:35.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:35.977 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:31:36.077 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:31:36.113 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:36.243 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:31:36.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:36.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:31:36.251 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:31:36.352 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:31:36.383 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:36.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.395 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.395 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.395 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.395 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.395 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.395 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:36 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/802494244' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:31:36.395 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:36 smithi005 ceph-mon[39562]: from='client.26723 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:31:36.409 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:31:36.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:36.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:31:36.425 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.10 to 1 (10000) 2024-04-20T17:31:36.514 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:31:36.547 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:36.571 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:31:36.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:36.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:31:36.677 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:31:36.718 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.718 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.718 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.718 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.718 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.718 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.718 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.718 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.718 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.718 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.718 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:36 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/802494244' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:31:36.719 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:36 smithi171 ceph-mon[44189]: from='client.26723 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:31:36.719 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:36.742 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:31:36.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:36.745 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:31:36.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:36.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:36 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/802494244' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:31:36.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:36 smithi106 ceph-mon[44423]: from='client.26723 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:31:36.846 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:31:36.899 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:37.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:37 smithi005 ceph-mon[39562]: pgmap v1546: 17 pgs: 17 active+clean; 6.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.51k op/s 2024-04-20T17:31:37.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:37 smithi005 ceph-mon[39562]: from='client.26723 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-20T17:31:37.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:37 smithi005 ceph-mon[39562]: osdmap e592: 12 total, 12 up, 11 in 2024-04-20T17:31:37.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:37 smithi106 ceph-mon[44423]: pgmap v1546: 17 pgs: 17 active+clean; 6.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.51k op/s 2024-04-20T17:31:37.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:37 smithi106 ceph-mon[44423]: from='client.26723 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-20T17:31:37.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:37 smithi106 ceph-mon[44423]: osdmap e592: 12 total, 12 up, 11 in 2024-04-20T17:31:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:37 smithi171 ceph-mon[44189]: pgmap v1546: 17 pgs: 17 active+clean; 6.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.51k op/s 2024-04-20T17:31:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:37 smithi171 ceph-mon[44189]: from='client.26723 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-20T17:31:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:37 smithi171 ceph-mon[44189]: osdmap e592: 12 total, 12 up, 11 in 2024-04-20T17:31:38.301 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:31:39.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:39 smithi005 ceph-mon[39562]: pgmap v1548: 17 pgs: 17 active+clean; 6.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-20T17:31:39.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:39 smithi106 ceph-mon[44423]: pgmap v1548: 17 pgs: 17 active+clean; 6.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-20T17:31:39.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:39 smithi171 ceph-mon[44189]: pgmap v1548: 17 pgs: 17 active+clean; 6.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-20T17:31:40.580 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 11 1 2024-04-20T17:31:40.774 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:31:40.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:40.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:31:40.881 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:31:40.949 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:41.101 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:31:41.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:41.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:31:41.212 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:31:41.249 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:41.287 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:31:41.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:41.296 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:31:41.397 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:31:41.409 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:41 smithi005 ceph-mon[39562]: pgmap v1549: 17 pgs: 17 active+clean; 5.9 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-20T17:31:41.431 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:41.462 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:31:41.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:41.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:31:41.544 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:31:41.566 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:31:41.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:41 smithi171 ceph-mon[44189]: pgmap v1549: 17 pgs: 17 active+clean; 5.9 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-20T17:31:41.594 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:41.617 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:31:41.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:41.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:31:41.721 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:31:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:41 smithi106 ceph-mon[44423]: pgmap v1549: 17 pgs: 17 active+clean; 5.9 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-20T17:31:41.773 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:41.822 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:31:41.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:41.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:31:41.928 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:31:41.981 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:42.453 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:31:42.459 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:31:42.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:42.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:31:42.569 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:31:42.598 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:42.620 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:31:42.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:42.625 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:31:42.725 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:31:42.766 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:42.793 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:31:42.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:42.795 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:31:42.896 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:31:42.932 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:42.969 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:31:42.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:42.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:31:43.071 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:31:43.106 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:43.131 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:31:43.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:43.136 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:31:43.237 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:31:43.268 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:43.291 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:31:43.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:43.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:31:43.397 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:31:43.429 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:43.466 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:31:43.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:43.469 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:31:43.570 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:31:43.606 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:43.632 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:31:43.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:43.636 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:31:43.651 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:31:43.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:43 smithi005 ceph-mon[39562]: pgmap v1550: 17 pgs: 17 active+clean; 5.7 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:31:43.737 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:31:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:43 smithi106 ceph-mon[44423]: pgmap v1550: 17 pgs: 17 active+clean; 5.7 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:31:43.765 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:43.796 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:31:43.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:43.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:31:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:43 smithi171 ceph-mon[44189]: pgmap v1550: 17 pgs: 17 active+clean; 5.7 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:31:43.903 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:31:43.935 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:43.965 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:31:43.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:43.969 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:31:44.069 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:31:44.144 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:44.182 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:31:44.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:44.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:31:44.295 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:31:44.330 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:44.357 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:31:44.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:44.363 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:31:44.444 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:31:44.464 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:31:44.524 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:44.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:44 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/886657952' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:31:44.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:44 smithi171 ceph-mon[44189]: from='client.26726 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:31:44.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:44 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/886657952' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:31:44.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:44 smithi005 ceph-mon[39562]: from='client.26726 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:31:44.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:44 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/886657952' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:31:44.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:44 smithi106 ceph-mon[44423]: from='client.26726 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:31:44.808 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:31:44.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:44.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:31:44.913 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:31:44.941 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:44.969 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:31:44.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:44.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:31:45.073 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:31:45.101 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:45.126 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:31:45.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:45.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:31:45.231 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:31:45.309 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:45.357 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:31:45.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:45.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:31:45.467 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:31:45.499 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:45.521 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:31:45.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:45.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:31:45.628 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:31:45.676 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:45.677 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:45 smithi005 ceph-mon[39562]: pgmap v1551: 17 pgs: 17 active+clean; 5.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:31:45.677 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:45 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:31:45.677 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:45 smithi005 ceph-mon[39562]: from='client.26726 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:31:45.677 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:45 smithi005 ceph-mon[39562]: osdmap e593: 12 total, 12 up, 11 in 2024-04-20T17:31:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:45 smithi106 ceph-mon[44423]: pgmap v1551: 17 pgs: 17 active+clean; 5.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:31:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:45 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:31:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:45 smithi106 ceph-mon[44423]: from='client.26726 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:31:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:45 smithi106 ceph-mon[44423]: osdmap e593: 12 total, 12 up, 11 in 2024-04-20T17:31:45.808 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:31:45.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:45.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:31:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:45 smithi171 ceph-mon[44189]: pgmap v1551: 17 pgs: 17 active+clean; 5.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:31:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:45 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:31:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:45 smithi171 ceph-mon[44189]: from='client.26726 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:31:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:45 smithi171 ceph-mon[44189]: osdmap e593: 12 total, 12 up, 11 in 2024-04-20T17:31:45.917 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:31:45.958 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:46.227 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:31:46.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:46.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:31:46.335 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:31:46.370 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:46.414 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:31:46.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:46.417 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:31:46.518 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:31:46.550 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:46.569 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:46 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:31:46.569 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:46 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:31:46.569 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:46 smithi005 ceph-mon[39562]: osdmap e594: 12 total, 12 up, 11 in 2024-04-20T17:31:46.572 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:31:46.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:46.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:31:46.677 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:31:46.706 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:46 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:31:46.706 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:46 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:31:46.706 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:46 smithi171 ceph-mon[44189]: osdmap e594: 12 total, 12 up, 11 in 2024-04-20T17:31:46.712 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:46.738 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:31:46.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:46.741 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:31:46.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:46 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:31:46.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:46 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:31:46.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:46 smithi106 ceph-mon[44423]: osdmap e594: 12 total, 12 up, 11 in 2024-04-20T17:31:46.842 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:31:46.858 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:31:46.858 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:31:46.858 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:31:46.859 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.16506.0:178448 5.c e594/543 5:3187be0b:::benchmark_data_smithi171_137281_object47014:head v 594'11235, mlcod=594'11233)", 2024-04-20T17:31:46.859 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:31:46.855138+0000", 2024-04-20T17:31:46.859 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.0034479659999999998, 2024-04-20T17:31:46.859 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0034796219999999999, 2024-04-20T17:31:46.859 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:31:46.859 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:31:46.859 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:31:46.859 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:31:46.859 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:31:46.859 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:31:46.859 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:46.855138+0000", 2024-04-20T17:31:46.859 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:31:46.859 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:31:46.859 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:31:46.860 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:31:46.860 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:46.855138+0000", 2024-04-20T17:31:46.860 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:31:46.860 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:31:46.860 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:31:46.860 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:31:46.860 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:46.855140+0000", 2024-04-20T17:31:46.860 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.137e-06 2024-04-20T17:31:46.860 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:31:46.860 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:31:46.860 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:31:46.860 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:46.855149+0000", 2024-04-20T17:31:46.860 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4979999999999995e-06 2024-04-20T17:31:46.860 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:31:46.861 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:31:46.861 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:31:46.861 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:46.855166+0000", 2024-04-20T17:31:46.861 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6713000000000001e-05 2024-04-20T17:31:46.861 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:31:46.861 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:31:46.861 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:31:46.861 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:46.855175+0000", 2024-04-20T17:31:46.861 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7630000000000006e-06 2024-04-20T17:31:46.861 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:31:46.861 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:31:46.861 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:31:46.861 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:46.855247+0000", 2024-04-20T17:31:46.861 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2098e-05 2024-04-20T17:31:46.861 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:31:46.861 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:31:46.862 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:31:46.862 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:46.855269+0000", 2024-04-20T17:31:46.862 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1529e-05 2024-04-20T17:31:46.862 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:31:46.862 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:31:46.862 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:31:46.862 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:31:46.862 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:31:46.862 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 1 2024-04-20T17:31:46.862 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:31:46.872 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:46.897 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:31:46.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:46.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:31:47.003 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:31:47.036 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:47.059 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:31:47.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:47.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:31:47.163 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:31:47.222 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:47.505 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:31:47.537 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:31:47.572 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:31:47.612 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:31:47.651 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:31:47.687 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:31:47.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:47 smithi005 ceph-mon[39562]: pgmap v1554: 17 pgs: 17 active+clean; 5.3 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.49k op/s 2024-04-20T17:31:47.721 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:31:47.753 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:31:47.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:47 smithi106 ceph-mon[44423]: pgmap v1554: 17 pgs: 17 active+clean; 5.3 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.49k op/s 2024-04-20T17:31:47.810 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:31:47.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:47.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:31:47.822 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:31:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:47 smithi171 ceph-mon[44189]: pgmap v1554: 17 pgs: 17 active+clean; 5.3 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.49k op/s 2024-04-20T17:31:47.847 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-20T17:31:47.851 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:31:47.892 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:31:47.914 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:31:47.953 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:31:47.972 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:31:47.972 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:31:47.972 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:31:47.972 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:31:47.972 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:31:47.972 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:31:47.972 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:31:47.972 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:31:47.973 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:31:47.973 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:31:47.973 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:31:47.973 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:31:47.999 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:48.615 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:31:48.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:48 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3778922843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:31:48.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:48.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:48.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:48.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:48.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:48.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:48.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:48 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3778922843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:31:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:48.951 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:48 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3778922843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:31:48.951 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:48.951 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:48.951 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:48.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:48.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:48.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:48.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:48.973 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:31:48.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:48.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:31:48.995 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:31:49.089 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:31:49.135 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:49.289 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:31:49.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:49.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:31:49.394 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:31:49.430 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:49.459 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:31:49.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:49.464 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:31:49.501 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.11 to 1 (10000) 2024-04-20T17:31:49.540 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:49.541 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:49.541 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:49.541 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:49.541 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:49.541 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:49 smithi005 ceph-mon[39562]: pgmap v1555: 17 pgs: 17 active+clean; 5.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-20T17:31:49.541 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:49 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1353880765' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:31:49.541 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:49 smithi005 ceph-mon[39562]: from='client.26620 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:31:49.565 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:31:49.618 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:49.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:49.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:49.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:49.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:49.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:49.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:49 smithi106 ceph-mon[44423]: pgmap v1555: 17 pgs: 17 active+clean; 5.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-20T17:31:49.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:49 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1353880765' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:31:49.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:49 smithi106 ceph-mon[44423]: from='client.26620 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:31:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:31:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:49 smithi171 ceph-mon[44189]: pgmap v1555: 17 pgs: 17 active+clean; 5.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.20k op/s 2024-04-20T17:31:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:49 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1353880765' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:31:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:49 smithi171 ceph-mon[44189]: from='client.26620 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:31:49.841 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:31:49.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:49.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:31:49.952 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:31:49.983 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:50.008 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:31:50.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:50.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:31:50.115 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:31:50.167 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:50.195 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:31:50.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:50.199 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:31:50.300 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:31:50.359 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:50.532 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:31:50.532 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:31:50.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:50.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:31:50.636 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:31:50.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:50 smithi005 ceph-mon[39562]: from='client.26620 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-20T17:31:50.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:50 smithi005 ceph-mon[39562]: osdmap e595: 12 total, 12 up, 11 in 2024-04-20T17:31:50.673 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:50.700 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:31:50.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:50.704 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:31:50.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:50 smithi106 ceph-mon[44423]: from='client.26620 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-20T17:31:50.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:50 smithi106 ceph-mon[44423]: osdmap e595: 12 total, 12 up, 11 in 2024-04-20T17:31:50.805 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:31:50.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:50 smithi171 ceph-mon[44189]: from='client.26620 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-20T17:31:50.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:50 smithi171 ceph-mon[44189]: osdmap e595: 12 total, 12 up, 11 in 2024-04-20T17:31:50.835 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:50.857 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:31:50.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:50.863 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:31:50.964 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:31:50.992 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:51.000 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 7 1 2024-04-20T17:31:51.009 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:31:51.015 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:31:51.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:51.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:31:51.121 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:31:51.156 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:51.183 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:31:51.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:51.186 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:31:51.287 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:31:51.379 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:51.409 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:31:51.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:51.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:31:51.512 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:51 smithi005 ceph-mon[39562]: pgmap v1557: 17 pgs: 17 active+clean; 5.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.00k op/s 2024-04-20T17:31:51.520 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:31:51.575 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:51.740 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:31:51.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:51.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:31:51.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:51 smithi171 ceph-mon[44189]: pgmap v1557: 17 pgs: 17 active+clean; 5.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.00k op/s 2024-04-20T17:31:51.851 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:31:51.900 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:51 smithi106 ceph-mon[44423]: pgmap v1557: 17 pgs: 17 active+clean; 5.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.00k op/s 2024-04-20T17:31:52.067 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:31:52.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:52.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:31:52.173 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:31:52.177 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:31:52.177 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:31:52.221 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:52.370 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:31:52.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:52.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:31:52.474 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:31:52.504 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:52.533 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:31:52.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:52.537 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:31:52.638 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:31:52.676 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:52.718 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:31:52.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:52.726 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:31:52.827 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:31:52.858 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:52.881 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:31:52.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:52.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:31:52.942 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-20T17:31:52.990 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:31:53.019 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:53.041 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:31:53.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:53.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:31:53.145 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:31:53.229 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:53.507 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:31:53.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:53.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:31:53.596 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:31:53.606 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:31:53.611 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:31:53.679 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:53.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:53 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:31:53.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:53 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:31:53.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:53 smithi171 ceph-mon[44189]: pgmap v1558: 17 pgs: 17 active+clean; 4.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:31:53.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:53 smithi171 ceph-mon[44189]: osdmap e596: 12 total, 12 up, 11 in 2024-04-20T17:31:53.918 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:31:53.920 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:53 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:31:53.920 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:53 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:31:53.920 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:53 smithi005 ceph-mon[39562]: pgmap v1558: 17 pgs: 17 active+clean; 4.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:31:53.920 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:53 smithi005 ceph-mon[39562]: osdmap e596: 12 total, 12 up, 11 in 2024-04-20T17:31:54.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:53 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:31:54.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:53 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:31:54.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:53 smithi106 ceph-mon[44423]: pgmap v1558: 17 pgs: 17 active+clean; 4.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:31:54.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:53 smithi106 ceph-mon[44423]: osdmap e596: 12 total, 12 up, 11 in 2024-04-20T17:31:55.573 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:31:55.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:55 smithi171 ceph-mon[44189]: pgmap v1560: 17 pgs: 17 active+clean; 4.7 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-20T17:31:55.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:55 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:31:55.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:55 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3498464586' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:31:55.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:55 smithi171 ceph-mon[44189]: from='client.26626 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:31:55.968 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:55 smithi005 ceph-mon[39562]: pgmap v1560: 17 pgs: 17 active+clean; 4.7 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-20T17:31:55.968 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:55 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:31:55.968 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:55 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3498464586' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:31:55.968 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:55 smithi005 ceph-mon[39562]: from='client.26626 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:31:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:55 smithi106 ceph-mon[44423]: pgmap v1560: 17 pgs: 17 active+clean; 4.7 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-20T17:31:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:55 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:31:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:55 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3498464586' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:31:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:55 smithi106 ceph-mon[44423]: from='client.26626 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:31:56.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:56 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:31:56.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:56 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:31:56.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:56 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:31:56.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:56 smithi005 ceph-mon[39562]: from='client.26626 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:31:56.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:56 smithi005 ceph-mon[39562]: osdmap e597: 12 total, 12 up, 11 in 2024-04-20T17:31:56.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:56 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:31:56.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:56 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:31:56.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:56 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:31:56.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:56 smithi171 ceph-mon[44189]: from='client.26626 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:31:56.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:56 smithi171 ceph-mon[44189]: osdmap e597: 12 total, 12 up, 11 in 2024-04-20T17:31:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:56 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:31:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:56 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:31:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:56 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:31:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:56 smithi106 ceph-mon[44423]: from='client.26626 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:31:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:56 smithi106 ceph-mon[44423]: osdmap e597: 12 total, 12 up, 11 in 2024-04-20T17:31:57.795 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:31:57.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:57.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:31:57.803 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:57 smithi005 ceph-mon[39562]: 5.9 scrub starts 2024-04-20T17:31:57.803 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:57 smithi005 ceph-mon[39562]: 5.9 scrub ok 2024-04-20T17:31:57.803 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:57 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:31:57.803 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:57 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:31:57.803 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:57 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:31:57.803 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:57 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:31:57.803 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:57 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:31:57.803 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:57 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:31:57.803 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:57 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:31:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:57 smithi171 ceph-mon[44189]: 5.9 scrub starts 2024-04-20T17:31:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:57 smithi171 ceph-mon[44189]: 5.9 scrub ok 2024-04-20T17:31:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:57 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:31:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:57 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:31:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:57 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:31:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:57 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:31:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:57 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:31:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:57 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:31:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:57 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:31:57.857 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:31:57.857 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:31:57.904 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:31:57.943 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:57 smithi106 ceph-mon[44423]: 5.9 scrub starts 2024-04-20T17:31:57.943 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:57 smithi106 ceph-mon[44423]: 5.9 scrub ok 2024-04-20T17:31:57.943 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:57 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:31:57.943 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:57 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:31:57.943 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:57 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:31:57.944 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:57 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:31:57.944 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:57 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:31:57.944 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:57 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:31:57.944 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:57 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:31:57.945 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:57.970 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:31:57.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:57.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:31:58.079 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:31:58.137 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:58.182 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:31:58.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:58.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:31:58.288 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:31:58.319 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:58.345 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:31:58.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:58.347 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:31:58.448 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:31:58.500 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:58.738 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:31:58.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:58.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:31:58.844 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:31:58.869 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:58 smithi106 ceph-mon[44423]: 5.6 deep-scrub starts 2024-04-20T17:31:58.871 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:58.898 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:31:58.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:58.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:31:58.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:58 smithi005 ceph-mon[39562]: 5.6 deep-scrub starts 2024-04-20T17:31:59.004 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:31:59.038 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:59.059 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:31:59.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:59.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:31:59.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:58 smithi171 ceph-mon[44189]: 5.6 deep-scrub starts 2024-04-20T17:31:59.165 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:31:59.200 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:59.420 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:31:59.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:31:59.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:31:59.526 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:31:59.566 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:31:59.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:31:59 smithi005 ceph-mon[39562]: pgmap v1563: 17 pgs: 17 active+clean; 4.5 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:32:00.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:31:59 smithi106 ceph-mon[44423]: pgmap v1563: 17 pgs: 17 active+clean; 4.5 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:32:00.065 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-20T17:32:00.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:31:59 smithi171 ceph-mon[44189]: pgmap v1563: 17 pgs: 17 active+clean; 4.5 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:32:00.398 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:32:00.545 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:32:00.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:00.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:32:00.650 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:32:00.713 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:00.947 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:32:00.948 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:32:00.973 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:32:00.978 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:32:00.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:00.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:32:01.019 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:32:01.049 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:32:01.083 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:32:01.085 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:32:01.117 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:01.117 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:32:01.122 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:32:01.140 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:32:01.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:01.144 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:32:01.160 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:32:01.185 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:32:01.210 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:32:01.241 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:32:01.244 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:32:01.281 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:01.286 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:32:01.328 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:32:01.328 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:32:01.328 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:32:01.328 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:32:01.329 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:32:01.329 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:32:01.329 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:32:01.329 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:32:01.329 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:32:01.329 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:32:01.329 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:32:01.329 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:32:01.625 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:01 smithi106 ceph-mon[44423]: pgmap v1564: 17 pgs: 17 active+clean; 4.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-20T17:32:01.625 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:01 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2436950767' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:32:01.627 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:01 smithi005 ceph-mon[39562]: pgmap v1564: 17 pgs: 17 active+clean; 4.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-20T17:32:01.627 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:01 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2436950767' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:32:01.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:01 smithi005 ceph-mon[39562]: from='client.26632 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:32:01.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:01 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/397038417' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:32:01.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:01.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:01.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:01.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:01.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:01.648 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.7 to 1 (10000) 2024-04-20T17:32:01.932 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:32:01.936 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:01 smithi106 ceph-mon[44423]: from='client.26632 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:32:01.936 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:01 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/397038417' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:32:01.936 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:01.936 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:01.936 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:01.936 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:01.936 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:01.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:01.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:32:02.038 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:32:02.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:01 smithi171 ceph-mon[44189]: pgmap v1564: 17 pgs: 17 active+clean; 4.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-20T17:32:02.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:01 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2436950767' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:32:02.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:01 smithi171 ceph-mon[44189]: from='client.26632 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:32:02.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:01 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/397038417' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:32:02.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:02.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:02.078 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:02.078 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:02.078 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:02.079 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:02.104 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:32:02.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:02.108 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:32:02.209 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:32:02.257 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:02.370 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:32:02.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:02.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:32:02.477 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:32:02.522 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:02.530 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:32:02.530 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:32:02.530 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.530 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.16506.0:196416 5.1 5:894f4ee5:::benchmark_data_smithi171_137281_object64982:head [delete] snapc 0=[] ondisk+write+known_if_redirected e598)", 2024-04-20T17:32:02.530 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:32:02.517491+0000", 2024-04-20T17:32:02.530 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.011051296, 2024-04-20T17:32:02.530 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011086708000000001, 2024-04-20T17:32:02.530 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:32:02.530 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:32:02.530 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:32:02.530 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:32:02.530 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.16506", 2024-04-20T17:32:02.530 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:32:02.531 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 196416 2024-04-20T17:32:02.531 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.531 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:32:02.531 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.531 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:32:02.531 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.517491+0000", 2024-04-20T17:32:02.531 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:02.531 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.531 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.531 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:32:02.531 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.517491+0000", 2024-04-20T17:32:02.531 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:02.531 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.531 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.532 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:32:02.532 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.517492+0000", 2024-04-20T17:32:02.532 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7090000000000001e-06 2024-04-20T17:32:02.532 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.532 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.532 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:32:02.532 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.517498+0000", 2024-04-20T17:32:02.532 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5140000000000001e-06 2024-04-20T17:32:02.532 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.532 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.532 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:32:02.532 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.517509+0000", 2024-04-20T17:32:02.532 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1151e-05 2024-04-20T17:32:02.532 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.533 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.533 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:32:02.533 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.517517+0000", 2024-04-20T17:32:02.533 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.388e-06 2024-04-20T17:32:02.533 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.533 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.533 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:32:02.533 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.520953+0000", 2024-04-20T17:32:02.533 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0034358850000000001 2024-04-20T17:32:02.533 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.533 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.533 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:32:02.533 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.521057+0000", 2024-04-20T17:32:02.533 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000103707 2024-04-20T17:32:02.534 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.534 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.534 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:32:02.534 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.521157+0000", 2024-04-20T17:32:02.534 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000100481 2024-04-20T17:32:02.534 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.534 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.534 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:32:02.534 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.526167+0000", 2024-04-20T17:32:02.534 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0050092670000000004 2024-04-20T17:32:02.534 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:02.534 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:02.534 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:02.534 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.535 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.535 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.16506.0:196422 5.f 5.87966c4f (undecoded) ondisk+write+known_if_redirected e598)", 2024-04-20T17:32:02.535 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:32:02.524540+0000", 2024-04-20T17:32:02.535 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.00400191, 2024-04-20T17:32:02.535 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0041271069999999997, 2024-04-20T17:32:02.535 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:32:02.535 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:32:02.535 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "queued for pg", 2024-04-20T17:32:02.535 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:32:02.535 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.16506", 2024-04-20T17:32:02.535 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:32:02.535 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 196422 2024-04-20T17:32:02.535 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.535 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:32:02.536 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.536 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:32:02.536 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.524540+0000", 2024-04-20T17:32:02.536 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:02.536 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.536 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.536 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:32:02.536 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.524540+0000", 2024-04-20T17:32:02.536 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:02.536 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.536 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.536 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:32:02.536 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.524543+0000", 2024-04-20T17:32:02.536 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9450000000000002e-06 2024-04-20T17:32:02.536 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.537 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.537 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:32:02.537 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.524550+0000", 2024-04-20T17:32:02.537 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1709999999999996e-06 2024-04-20T17:32:02.537 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.537 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.537 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:32:02.537 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.524561+0000", 2024-04-20T17:32:02.537 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0477e-05 2024-04-20T17:32:02.537 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.537 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.537 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:32:02.537 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.524570+0000", 2024-04-20T17:32:02.537 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6870000000000006e-06 2024-04-20T17:32:02.537 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:02.537 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:02.538 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:02.538 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.538 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.538 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.16506.0:196396 5.f 5:fb47c3fc:::benchmark_data_smithi171_137281_object64962:head [delete] snapc 0=[] ondisk+write+known_if_redirected e598)", 2024-04-20T17:32:02.538 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:32:02.501444+0000", 2024-04-20T17:32:02.538 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.027098087, 2024-04-20T17:32:02.538 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.027287109, 2024-04-20T17:32:02.538 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:32:02.538 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:32:02.538 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:32:02.539 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:32:02.539 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.16506", 2024-04-20T17:32:02.539 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:32:02.539 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 196396 2024-04-20T17:32:02.539 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.539 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:32:02.539 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.539 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:32:02.539 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.501444+0000", 2024-04-20T17:32:02.539 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:02.539 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.539 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.539 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:32:02.539 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.501444+0000", 2024-04-20T17:32:02.539 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:02.540 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.540 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.540 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:32:02.540 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.501445+0000", 2024-04-20T17:32:02.540 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1009999999999999e-06 2024-04-20T17:32:02.540 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.540 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.540 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:32:02.540 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.501449+0000", 2024-04-20T17:32:02.540 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9519999999999996e-06 2024-04-20T17:32:02.540 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.540 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.540 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:32:02.540 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.501456+0000", 2024-04-20T17:32:02.540 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5989999999999996e-06 2024-04-20T17:32:02.541 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.541 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.541 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:32:02.541 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.501463+0000", 2024-04-20T17:32:02.541 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4620000000000001e-06 2024-04-20T17:32:02.541 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.541 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.541 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:32:02.541 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.506085+0000", 2024-04-20T17:32:02.541 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0046221889999999996 2024-04-20T17:32:02.541 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.541 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.541 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:32:02.541 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.506210+0000", 2024-04-20T17:32:02.541 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012527299999999999 2024-04-20T17:32:02.542 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.542 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.542 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:32:02.542 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.506335+0000", 2024-04-20T17:32:02.542 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012435899999999999 2024-04-20T17:32:02.542 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.542 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.542 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:32:02.542 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.525629+0000", 2024-04-20T17:32:02.542 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.019294221 2024-04-20T17:32:02.542 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:02.542 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:02.542 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:02.542 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.543 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.543 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.16506.0:196402 5.f 5:feeb1ca7:::benchmark_data_smithi171_137281_object64968:head [delete] snapc 0=[] ondisk+write+known_if_redirected e598)", 2024-04-20T17:32:02.543 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:32:02.506389+0000", 2024-04-20T17:32:02.543 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.022152939999999999, 2024-04-20T17:32:02.543 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.022426966999999999, 2024-04-20T17:32:02.543 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:32:02.543 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:32:02.543 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:32:02.543 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:32:02.543 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.16506", 2024-04-20T17:32:02.543 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:32:02.543 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 196402 2024-04-20T17:32:02.543 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.544 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:32:02.544 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.544 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:32:02.544 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.506389+0000", 2024-04-20T17:32:02.544 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:02.544 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.544 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.544 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:32:02.544 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.506389+0000", 2024-04-20T17:32:02.544 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:02.544 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.544 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.544 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:32:02.544 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.506390+0000", 2024-04-20T17:32:02.545 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8500000000000001e-07 2024-04-20T17:32:02.545 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.545 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.545 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:32:02.545 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.506393+0000", 2024-04-20T17:32:02.545 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9119999999999998e-06 2024-04-20T17:32:02.545 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.545 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.545 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:32:02.545 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.506403+0000", 2024-04-20T17:32:02.545 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0047999999999999e-05 2024-04-20T17:32:02.545 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.545 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.545 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:32:02.545 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.506408+0000", 2024-04-20T17:32:02.546 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.451e-06 2024-04-20T17:32:02.546 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.546 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.546 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:32:02.546 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.511164+0000", 2024-04-20T17:32:02.546 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0047563550000000003 2024-04-20T17:32:02.546 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.546 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.546 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:32:02.546 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.511274+0000", 2024-04-20T17:32:02.547 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000109391 2024-04-20T17:32:02.547 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.547 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.547 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:32:02.547 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.511345+0000", 2024-04-20T17:32:02.547 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1440999999999996e-05 2024-04-20T17:32:02.547 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.547 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.547 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:32:02.547 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.525635+0000", 2024-04-20T17:32:02.547 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.014289380000000001 2024-04-20T17:32:02.547 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:02.547 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:02.547 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:02.547 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.548 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.548 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.16506.0:196404 5.f 5:fca35097:::benchmark_data_smithi171_137281_object64970:head [delete] snapc 0=[] ondisk+write+known_if_redirected e598)", 2024-04-20T17:32:02.548 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:32:02.506485+0000", 2024-04-20T17:32:02.548 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.022056561999999998, 2024-04-20T17:32:02.548 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.022419560000000002, 2024-04-20T17:32:02.548 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:32:02.548 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:32:02.548 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:32:02.548 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:32:02.548 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.16506", 2024-04-20T17:32:02.548 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:32:02.548 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 196404 2024-04-20T17:32:02.548 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.548 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:32:02.549 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.549 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:32:02.549 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.506485+0000", 2024-04-20T17:32:02.549 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:02.549 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.549 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.549 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:32:02.549 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.506485+0000", 2024-04-20T17:32:02.549 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:02.549 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.549 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.549 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:32:02.549 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.506486+0000", 2024-04-20T17:32:02.549 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9400000000000004e-07 2024-04-20T17:32:02.550 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.550 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.550 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:32:02.550 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.506489+0000", 2024-04-20T17:32:02.550 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0599999999999999e-06 2024-04-20T17:32:02.550 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.550 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.550 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:32:02.550 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.506496+0000", 2024-04-20T17:32:02.550 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7329999999999997e-06 2024-04-20T17:32:02.550 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.550 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.550 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:32:02.551 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.506500+0000", 2024-04-20T17:32:02.551 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1670000000000001e-06 2024-04-20T17:32:02.551 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.551 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.551 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:32:02.551 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.515595+0000", 2024-04-20T17:32:02.551 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0090951069999999998 2024-04-20T17:32:02.551 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.551 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.551 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:32:02.551 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.515706+0000", 2024-04-20T17:32:02.551 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011090399999999999 2024-04-20T17:32:02.551 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.551 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.552 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:32:02.552 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.515818+0000", 2024-04-20T17:32:02.552 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000111573 2024-04-20T17:32:02.552 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:02.552 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:02.552 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:02.552 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.552 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.552 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.16506.0:196406 5.f 5:fcc452f9:::benchmark_data_smithi171_137281_object64972:head [delete] snapc 0=[] ondisk+write+known_if_redirected e598)", 2024-04-20T17:32:02.552 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:32:02.506888+0000", 2024-04-20T17:32:02.552 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.021654066999999999, 2024-04-20T17:32:02.552 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.022105703000000001, 2024-04-20T17:32:02.552 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:32:02.552 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:32:02.553 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:32:02.553 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:32:02.553 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.16506", 2024-04-20T17:32:02.553 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:32:02.553 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 196406 2024-04-20T17:32:02.553 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.553 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:32:02.553 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.553 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:32:02.553 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.506888+0000", 2024-04-20T17:32:02.553 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:02.553 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.553 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.553 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:32:02.553 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.506888+0000", 2024-04-20T17:32:02.554 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:02.554 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.554 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.554 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:32:02.554 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.506889+0000", 2024-04-20T17:32:02.554 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.46e-06 2024-04-20T17:32:02.554 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.554 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.554 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:32:02.554 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.506895+0000", 2024-04-20T17:32:02.554 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4759999999999997e-06 2024-04-20T17:32:02.554 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.554 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.554 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:32:02.555 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.506909+0000", 2024-04-20T17:32:02.555 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3746e-05 2024-04-20T17:32:02.555 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.555 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.555 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:32:02.555 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.506917+0000", 2024-04-20T17:32:02.555 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1219999999999995e-06 2024-04-20T17:32:02.555 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.555 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.555 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:32:02.555 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.520471+0000", 2024-04-20T17:32:02.555 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.013554093999999999 2024-04-20T17:32:02.555 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.555 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.555 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:32:02.556 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.520594+0000", 2024-04-20T17:32:02.556 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012325699999999999 2024-04-20T17:32:02.556 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.556 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.556 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:32:02.556 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.520730+0000", 2024-04-20T17:32:02.556 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013619700000000001 2024-04-20T17:32:02.556 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:02.556 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:02.556 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:02.556 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.556 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.556 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.16506.0:196409 5.f 5:faab9d4b:::benchmark_data_smithi171_137281_object64975:head [delete] snapc 0=[] ondisk+write+known_if_redirected e598)", 2024-04-20T17:32:02.556 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:32:02.507695+0000", 2024-04-20T17:32:02.556 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.020846588999999999, 2024-04-20T17:32:02.557 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.021378403000000001, 2024-04-20T17:32:02.557 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:32:02.557 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:32:02.557 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:32:02.557 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:32:02.557 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.16506", 2024-04-20T17:32:02.557 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:32:02.557 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 196409 2024-04-20T17:32:02.557 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.557 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:32:02.557 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.557 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:32:02.557 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.507695+0000", 2024-04-20T17:32:02.557 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:02.557 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.558 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.558 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:32:02.558 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.507695+0000", 2024-04-20T17:32:02.558 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:02.558 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.558 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.558 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:32:02.558 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.507697+0000", 2024-04-20T17:32:02.558 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.53e-06 2024-04-20T17:32:02.558 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.558 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.558 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:32:02.558 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.507702+0000", 2024-04-20T17:32:02.558 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4070000000000001e-06 2024-04-20T17:32:02.559 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.559 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.559 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:32:02.559 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.507717+0000", 2024-04-20T17:32:02.559 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4778e-05 2024-04-20T17:32:02.559 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.559 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.559 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:32:02.559 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.507726+0000", 2024-04-20T17:32:02.559 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7120000000000006e-06 2024-04-20T17:32:02.559 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.559 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.559 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:32:02.559 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.525664+0000", 2024-04-20T17:32:02.559 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.017937760000000001 2024-04-20T17:32:02.559 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.560 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.560 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:32:02.560 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.525787+0000", 2024-04-20T17:32:02.560 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012336899999999999 2024-04-20T17:32:02.560 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:02.560 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:02.560 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:32:02.560 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.525900+0000", 2024-04-20T17:32:02.560 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000112868 2024-04-20T17:32:02.560 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:02.560 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:02.560 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:02.560 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:02.560 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-04-20T17:32:02.561 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 7 2024-04-20T17:32:02.561 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:32:02.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:02.562 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:32:02.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:32:02.663 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:32:02.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:02.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:02.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:02.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:02.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:02.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:02.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:02.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:02 smithi005 ceph-mon[39562]: from='client.26632 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-20T17:32:02.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:02 smithi005 ceph-mon[39562]: osdmap e598: 12 total, 12 up, 11 in 2024-04-20T17:32:02.708 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:02.832 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:32:02.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:02.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:32:02.874 INFO:tasks.thrashosds.thrasher:test_map_discontinuity 2024-04-20T17:32:02.874 INFO:tasks.thrashosds.thrasher:Waiting for recovery 2024-04-20T17:32:02.874 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-20T17:32:02.874 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-20T17:32:02.875 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:32:02.941 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:32:02.970 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:02.970 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:02.970 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:02.970 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:02.970 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:02.970 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:02.971 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:02.971 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:02 smithi171 ceph-mon[44189]: from='client.26632 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-20T17:32:02.971 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:02 smithi171 ceph-mon[44189]: osdmap e598: 12 total, 12 up, 11 in 2024-04-20T17:32:02.973 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:02.996 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:32:03.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:03.002 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:32:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:03.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:03.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:03.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:03.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:03.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:03.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:03.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:02 smithi106 ceph-mon[44423]: from='client.26632 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-20T17:32:03.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:02 smithi106 ceph-mon[44423]: osdmap e598: 12 total, 12 up, 11 in 2024-04-20T17:32:03.103 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:32:03.130 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:03.153 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:32:03.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:03.156 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:32:03.257 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:32:03.306 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:03.343 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:32:03.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:03.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:32:03.439 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-20T17:32:03.447 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:32:03.481 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:03.504 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:32:03.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:03.507 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:32:03.608 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:32:03.656 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:03.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:03 smithi106 ceph-mon[44423]: pgmap v1566: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 4.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-20T17:32:03.795 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:32:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:03 smithi171 ceph-mon[44189]: pgmap v1566: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 4.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-20T17:32:03.884 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:03 smithi005 ceph-mon[39562]: pgmap v1566: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 4.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-20T17:32:04.131 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:32:04.317 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:32:04.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:04.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:32:04.369 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:32:04.425 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:32:04.458 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:04.484 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:32:04.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:04.487 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:32:04.588 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:32:04.626 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:04.654 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:32:04.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:04.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:32:04.758 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:32:04.791 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:04.817 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:32:04.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:04.822 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:32:04.923 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:32:04.960 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:05.176 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:32:05.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:05.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:32:05.283 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:32:05.318 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:05.528 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:32:05.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:05.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:32:05.632 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:32:05.661 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:05.685 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:32:05.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:05.691 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:32:05.792 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:32:05.807 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:32:05.807 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:32:05.807 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-04-20T17:32:05.807 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-04-20T17:32:05.807 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:32:05.848 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:05 smithi005 ceph-mon[39562]: pgmap v1567: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 4.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-20T17:32:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:05 smithi106 ceph-mon[44423]: pgmap v1567: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 4.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-20T17:32:06.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:05 smithi171 ceph-mon[44189]: pgmap v1567: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 4.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-20T17:32:06.554 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-20T17:32:06.676 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:32:06.922 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:32:06.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:06 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1556778762' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:32:06.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:06 smithi005 ceph-mon[39562]: from='client.26756 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:32:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:06 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1556778762' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:32:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:06 smithi106 ceph-mon[44423]: from='client.26756 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:32:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:06 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1556778762' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:32:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:06 smithi171 ceph-mon[44189]: from='client.26756 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:32:07.425 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:32:07.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:07.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:32:07.533 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:32:07.579 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:07.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:07 smithi005 ceph-mon[39562]: 5.d scrub starts 2024-04-20T17:32:07.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:07 smithi005 ceph-mon[39562]: pgmap v1568: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-20T17:32:07.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:07 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:32:07.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:07 smithi005 ceph-mon[39562]: from='client.26756 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:32:07.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:07 smithi005 ceph-mon[39562]: osdmap e599: 12 total, 12 up, 11 in 2024-04-20T17:32:07.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:07 smithi106 ceph-mon[44423]: 5.d scrub starts 2024-04-20T17:32:07.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:07 smithi106 ceph-mon[44423]: pgmap v1568: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-20T17:32:07.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:07 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:32:07.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:07 smithi106 ceph-mon[44423]: from='client.26756 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:32:07.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:07 smithi106 ceph-mon[44423]: osdmap e599: 12 total, 12 up, 11 in 2024-04-20T17:32:07.943 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:32:07.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:07.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:32:08.046 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:32:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:07 smithi171 ceph-mon[44189]: 5.d scrub starts 2024-04-20T17:32:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:07 smithi171 ceph-mon[44189]: pgmap v1568: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-20T17:32:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:07 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:32:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:07 smithi171 ceph-mon[44189]: from='client.26756 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:32:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:07 smithi171 ceph-mon[44189]: osdmap e599: 12 total, 12 up, 11 in 2024-04-20T17:32:08.110 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:08.145 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:32:08.146 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:32:08.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:08.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:32:08.268 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:32:08.306 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:08.333 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:32:08.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:08.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:32:08.440 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:32:08.472 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:08.493 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:32:08.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:08.496 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:32:08.597 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:32:08.628 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:08.673 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:32:08.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:08.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:32:08.782 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:32:08.836 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:08.970 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:32:08.970 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-04-20T17:32:08.970 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-04-20T17:32:08.971 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:32:08.971 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.971 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:164759 3.3 3:c1715c23:::benchmark_data_smithi171_110765_object21544:head [delete] snapc 0=[] ondisk+write+known_if_redirected e393)", 2024-04-20T17:32:08.971 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:22:20.361595+0000", 2024-04-20T17:32:08.971 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 588.60559831099999, 2024-04-20T17:32:08.971 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.055921910999999998, 2024-04-20T17:32:08.971 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:32:08.971 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:32:08.971 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:08.971 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:32:08.971 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:32:08.971 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:32:08.971 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 164759 2024-04-20T17:32:08.971 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.971 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:32:08.972 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.972 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:32:08.972 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.361595+0000", 2024-04-20T17:32:08.972 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:08.972 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.972 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.972 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:32:08.972 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.361595+0000", 2024-04-20T17:32:08.972 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:08.972 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.972 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.972 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:32:08.973 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.361597+0000", 2024-04-20T17:32:08.973 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.838e-06 2024-04-20T17:32:08.973 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.973 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.973 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:32:08.973 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.361602+0000", 2024-04-20T17:32:08.973 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7960000000000001e-06 2024-04-20T17:32:08.973 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.973 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.973 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:32:08.973 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.361612+0000", 2024-04-20T17:32:08.973 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.132e-06 2024-04-20T17:32:08.974 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.974 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.974 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:32:08.974 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.361617+0000", 2024-04-20T17:32:08.974 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.874e-06 2024-04-20T17:32:08.974 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.974 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.974 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:32:08.974 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.361674+0000", 2024-04-20T17:32:08.974 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6490000000000003e-05 2024-04-20T17:32:08.974 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.974 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.974 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:32:08.975 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.361785+0000", 2024-04-20T17:32:08.975 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000111494 2024-04-20T17:32:08.975 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.975 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.975 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:32:08.975 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.361900+0000", 2024-04-20T17:32:08.975 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011406899999999999 2024-04-20T17:32:08.975 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.975 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.975 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:32:08.975 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.370403+0000", 2024-04-20T17:32:08.975 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0085035909999999996 2024-04-20T17:32:08.975 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.975 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.976 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:08.976 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.417418+0000", 2024-04-20T17:32:08.976 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.047014528999999999 2024-04-20T17:32:08.976 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.976 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.976 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:32:08.976 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.417472+0000", 2024-04-20T17:32:08.976 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4088999999999997e-05 2024-04-20T17:32:08.976 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.976 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.976 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:32:08.976 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.417517+0000", 2024-04-20T17:32:08.976 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5009000000000002e-05 2024-04-20T17:32:08.976 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:08.977 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:08.977 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:08.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.977 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:164770 3.3 3:c7d5b1bb:::benchmark_data_smithi171_110765_object21555:head [delete] snapc 0=[] ondisk+write+known_if_redirected e393)", 2024-04-20T17:32:08.977 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:22:20.371577+0000", 2024-04-20T17:32:08.977 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 588.595616339, 2024-04-20T17:32:08.977 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.058087044999999997, 2024-04-20T17:32:08.977 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:32:08.977 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:32:08.977 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:08.977 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:32:08.977 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:32:08.977 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:32:08.977 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 164770 2024-04-20T17:32:08.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.978 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:32:08.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.978 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:32:08.978 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.371577+0000", 2024-04-20T17:32:08.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:08.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.978 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:32:08.978 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.371577+0000", 2024-04-20T17:32:08.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:08.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.978 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:32:08.979 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.371579+0000", 2024-04-20T17:32:08.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8789999999999999e-06 2024-04-20T17:32:08.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.979 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:32:08.979 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.371584+0000", 2024-04-20T17:32:08.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3809999999999998e-06 2024-04-20T17:32:08.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.979 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:32:08.979 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.371595+0000", 2024-04-20T17:32:08.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1100999999999999e-05 2024-04-20T17:32:08.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.979 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:32:08.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.371602+0000", 2024-04-20T17:32:08.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0600000000000002e-06 2024-04-20T17:32:08.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:32:08.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.373811+0000", 2024-04-20T17:32:08.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022089789999999998 2024-04-20T17:32:08.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:32:08.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.373942+0000", 2024-04-20T17:32:08.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013118 2024-04-20T17:32:08.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:32:08.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.374068+0000", 2024-04-20T17:32:08.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012536099999999999 2024-04-20T17:32:08.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:32:08.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.379040+0000", 2024-04-20T17:32:08.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.004971998 2024-04-20T17:32:08.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:08.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.429564+0000", 2024-04-20T17:32:08.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.050524544999999997 2024-04-20T17:32:08.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.982 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:32:08.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.429618+0000", 2024-04-20T17:32:08.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3520999999999998e-05 2024-04-20T17:32:08.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.982 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:32:08.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.429664+0000", 2024-04-20T17:32:08.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.604e-05 2024-04-20T17:32:08.982 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:08.982 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:08.982 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:08.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.982 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:164771 3.3 3:c84e3de8:::benchmark_data_smithi171_110765_object21556:head [delete] snapc 0=[] ondisk+write+known_if_redirected e393)", 2024-04-20T17:32:08.983 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:22:20.371771+0000", 2024-04-20T17:32:08.983 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 588.59542217800004, 2024-04-20T17:32:08.983 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.058036722999999998, 2024-04-20T17:32:08.983 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:32:08.983 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:32:08.983 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:08.983 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:32:08.983 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:32:08.983 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:32:08.983 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 164771 2024-04-20T17:32:08.983 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.983 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:32:08.983 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.983 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:32:08.983 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.371771+0000", 2024-04-20T17:32:08.983 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:08.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.984 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:32:08.984 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.371771+0000", 2024-04-20T17:32:08.984 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:08.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.984 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:32:08.984 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.371772+0000", 2024-04-20T17:32:08.984 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.071e-06 2024-04-20T17:32:08.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.984 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:32:08.984 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.371776+0000", 2024-04-20T17:32:08.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4829999999999999e-06 2024-04-20T17:32:08.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:32:08.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.371783+0000", 2024-04-20T17:32:08.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3000000000000004e-06 2024-04-20T17:32:08.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:32:08.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.371788+0000", 2024-04-20T17:32:08.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3430000000000003e-06 2024-04-20T17:32:08.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:32:08.986 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.376884+0000", 2024-04-20T17:32:08.986 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0050953839999999997 2024-04-20T17:32:08.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.986 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.986 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:32:08.986 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.377004+0000", 2024-04-20T17:32:08.986 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000120436 2024-04-20T17:32:08.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.986 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.986 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:32:08.986 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.377125+0000", 2024-04-20T17:32:08.986 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000120794 2024-04-20T17:32:08.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.987 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.987 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:32:08.987 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.379143+0000", 2024-04-20T17:32:08.987 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020177680000000001 2024-04-20T17:32:08.987 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.987 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.987 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:08.987 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.429710+0000", 2024-04-20T17:32:08.987 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.050567237000000001 2024-04-20T17:32:08.987 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.987 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.987 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:32:08.987 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.429763+0000", 2024-04-20T17:32:08.988 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3476000000000001e-05 2024-04-20T17:32:08.988 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.988 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.988 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:32:08.988 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:22:20.429808+0000", 2024-04-20T17:32:08.988 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4431000000000002e-05 2024-04-20T17:32:08.988 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:08.988 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:08.988 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:08.988 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.988 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.988 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:230195 3.3 3:cac27cb4:::benchmark_data_smithi171_110765_object86980:head [delete] snapc 0=[] ondisk+write+known_if_redirected e409)", 2024-04-20T17:32:08.989 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:15.050603+0000", 2024-04-20T17:32:08.989 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 533.91659054199999, 2024-04-20T17:32:08.989 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.070679689000000004, 2024-04-20T17:32:08.989 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:32:08.989 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:32:08.989 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:08.989 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:32:08.989 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:32:08.989 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:32:08.989 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 230195 2024-04-20T17:32:08.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.989 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:32:08.989 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.990 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:32:08.990 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.050603+0000", 2024-04-20T17:32:08.990 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:08.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.990 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.990 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:32:08.990 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.050603+0000", 2024-04-20T17:32:08.990 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:08.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.990 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.990 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:32:08.990 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.050604+0000", 2024-04-20T17:32:08.991 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.48e-06 2024-04-20T17:32:08.991 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.991 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.991 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:32:08.991 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.050609+0000", 2024-04-20T17:32:08.991 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7389999999999999e-06 2024-04-20T17:32:08.991 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.991 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.991 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:32:08.991 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.050619+0000", 2024-04-20T17:32:08.991 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7610000000000002e-06 2024-04-20T17:32:08.991 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.991 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.991 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:32:08.992 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.050627+0000", 2024-04-20T17:32:08.992 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3799999999999994e-06 2024-04-20T17:32:08.992 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.992 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.992 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:32:08.992 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.050676+0000", 2024-04-20T17:32:08.992 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9363000000000002e-05 2024-04-20T17:32:08.992 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.992 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.992 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:32:08.992 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.050787+0000", 2024-04-20T17:32:08.992 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011053 2024-04-20T17:32:08.992 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.992 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.993 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:32:08.993 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.050906+0000", 2024-04-20T17:32:08.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000118736 2024-04-20T17:32:08.993 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.993 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:32:08.993 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.054523+0000", 2024-04-20T17:32:08.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0036171269999999999 2024-04-20T17:32:08.993 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.993 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:08.993 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.121187+0000", 2024-04-20T17:32:08.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.066664243999999998 2024-04-20T17:32:08.993 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.994 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:32:08.994 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.121220+0000", 2024-04-20T17:32:08.994 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3359999999999999e-05 2024-04-20T17:32:08.994 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.994 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.994 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:32:08.994 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.121282+0000", 2024-04-20T17:32:08.994 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1969000000000002e-05 2024-04-20T17:32:08.994 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:08.994 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:08.994 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:08.994 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.994 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.994 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:230197 3.3 3:cb24cc1d:::benchmark_data_smithi171_110765_object86982:head [delete] snapc 0=[] ondisk+write+known_if_redirected e409)", 2024-04-20T17:32:08.995 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:23:15.054810+0000", 2024-04-20T17:32:08.995 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 533.91238305800005, 2024-04-20T17:32:08.995 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.066635147000000006, 2024-04-20T17:32:08.995 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:32:08.995 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:32:08.995 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:08.995 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:32:08.995 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:32:08.995 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:32:08.995 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 230197 2024-04-20T17:32:08.995 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.995 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:32:08.995 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.995 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:32:08.995 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.054810+0000", 2024-04-20T17:32:08.996 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:08.996 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.996 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.996 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:32:08.996 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.054810+0000", 2024-04-20T17:32:08.996 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:08.996 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.996 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.996 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:32:08.996 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.054812+0000", 2024-04-20T17:32:08.996 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8959999999999999e-06 2024-04-20T17:32:08.996 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.996 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.996 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:32:08.997 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.054818+0000", 2024-04-20T17:32:08.997 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9050000000000002e-06 2024-04-20T17:32:08.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.997 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:32:08.997 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.054828+0000", 2024-04-20T17:32:08.997 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0519999999999999e-05 2024-04-20T17:32:08.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.997 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:32:08.997 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.054836+0000", 2024-04-20T17:32:08.997 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6680000000000001e-06 2024-04-20T17:32:08.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.997 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:32:08.997 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.054882+0000", 2024-04-20T17:32:08.998 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6198000000000001e-05 2024-04-20T17:32:08.998 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.998 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:32:08.998 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.055021+0000", 2024-04-20T17:32:08.998 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013839000000000001 2024-04-20T17:32:08.998 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.998 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:32:08.998 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.055147+0000", 2024-04-20T17:32:08.998 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012633300000000001 2024-04-20T17:32:08.998 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.999 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:32:08.999 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.058294+0000", 2024-04-20T17:32:08.999 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00314694 2024-04-20T17:32:08.999 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.999 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.999 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:08.999 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.121349+0000", 2024-04-20T17:32:08.999 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.063054889000000003 2024-04-20T17:32:08.999 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.999 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:08.999 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:32:08.999 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.121402+0000", 2024-04-20T17:32:08.999 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2719999999999997e-05 2024-04-20T17:32:08.999 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:08.999 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.000 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:32:09.000 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:23:15.121445+0000", 2024-04-20T17:32:09.000 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3687999999999999e-05 2024-04-20T17:32:09.000 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.000 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:09.000 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.000 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.000 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.000 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25802.0:279875 3.3 3:ca72fdb9:::benchmark_data_smithi171_110765_object136660:head [delete] snapc 0=[] ondisk+write+known_if_redirected e433)", 2024-04-20T17:32:09.000 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:24:05.819548+0000", 2024-04-20T17:32:09.000 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 483.14764507500001, 2024-04-20T17:32:09.000 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.058138337999999998, 2024-04-20T17:32:09.000 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:32:09.000 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:32:09.001 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:09.001 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:32:09.001 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25802", 2024-04-20T17:32:09.001 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/733485184", 2024-04-20T17:32:09.001 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 279875 2024-04-20T17:32:09.001 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.001 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:32:09.001 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.001 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:32:09.001 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:24:05.819548+0000", 2024-04-20T17:32:09.001 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.001 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.001 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.001 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:32:09.002 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:24:05.819548+0000", 2024-04-20T17:32:09.002 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.002 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:32:09.002 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:24:05.819549+0000", 2024-04-20T17:32:09.002 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7000000000000003e-07 2024-04-20T17:32:09.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.002 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:32:09.002 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:24:05.819552+0000", 2024-04-20T17:32:09.002 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9639999999999999e-06 2024-04-20T17:32:09.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.003 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:32:09.003 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:24:05.819556+0000", 2024-04-20T17:32:09.003 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.065e-06 2024-04-20T17:32:09.003 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.003 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:32:09.003 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:24:05.819560+0000", 2024-04-20T17:32:09.003 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7330000000000001e-06 2024-04-20T17:32:09.003 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.003 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:32:09.003 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:24:05.823294+0000", 2024-04-20T17:32:09.003 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037338359999999999 2024-04-20T17:32:09.003 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.004 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.004 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:32:09.004 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:24:05.823423+0000", 2024-04-20T17:32:09.004 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012909400000000001 2024-04-20T17:32:09.004 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.004 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.004 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0", 2024-04-20T17:32:09.004 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:24:05.823547+0000", 2024-04-20T17:32:09.004 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000124051 2024-04-20T17:32:09.004 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.004 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.004 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:32:09.004 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:24:05.841165+0000", 2024-04-20T17:32:09.005 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.017618156999999999 2024-04-20T17:32:09.005 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.005 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.005 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:09.005 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:24:05.877588+0000", 2024-04-20T17:32:09.005 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.036423171999999997 2024-04-20T17:32:09.005 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.005 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.005 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:32:09.005 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:24:05.877639+0000", 2024-04-20T17:32:09.005 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0992999999999998e-05 2024-04-20T17:32:09.005 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.005 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.005 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:32:09.005 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:24:05.877686+0000", 2024-04-20T17:32:09.006 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7403e-05 2024-04-20T17:32:09.006 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.006 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:09.006 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.006 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.006 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.006 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26074.0:57710 4.c 4:3b7a6ad0:::benchmark_data_smithi171_122772_object57709:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e457)", 2024-04-20T17:32:09.006 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:25:04.576003+0000", 2024-04-20T17:32:09.006 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 424.39119017299998, 2024-04-20T17:32:09.006 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.068512081000000002, 2024-04-20T17:32:09.006 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:32:09.006 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:32:09.006 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:09.006 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:32:09.006 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26074", 2024-04-20T17:32:09.007 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:32:09.007 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 57710 2024-04-20T17:32:09.007 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.007 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:32:09.007 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.007 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:32:09.007 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.576003+0000", 2024-04-20T17:32:09.007 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.007 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.007 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.007 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:32:09.007 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.576003+0000", 2024-04-20T17:32:09.007 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.007 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.008 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.008 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:32:09.008 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.576005+0000", 2024-04-20T17:32:09.008 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.249e-06 2024-04-20T17:32:09.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.008 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.008 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:32:09.008 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.576054+0000", 2024-04-20T17:32:09.008 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8282000000000003e-05 2024-04-20T17:32:09.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.008 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.008 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:32:09.008 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.576066+0000", 2024-04-20T17:32:09.008 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2549e-05 2024-04-20T17:32:09.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.009 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.009 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:32:09.009 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.576073+0000", 2024-04-20T17:32:09.009 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1409999999999999e-06 2024-04-20T17:32:09.009 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.009 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.009 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:32:09.009 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.576131+0000", 2024-04-20T17:32:09.009 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7941e-05 2024-04-20T17:32:09.009 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.009 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.009 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:32:09.009 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.576299+0000", 2024-04-20T17:32:09.009 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016742499999999999 2024-04-20T17:32:09.009 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.009 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.010 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:32:09.010 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.576454+0000", 2024-04-20T17:32:09.010 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000155437 2024-04-20T17:32:09.010 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.010 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.010 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:32:09.010 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.577978+0000", 2024-04-20T17:32:09.010 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001523592 2024-04-20T17:32:09.010 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.010 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.010 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:09.010 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.644404+0000", 2024-04-20T17:32:09.010 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.066426764999999999 2024-04-20T17:32:09.011 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.011 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.011 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:32:09.011 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.644460+0000", 2024-04-20T17:32:09.011 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6072000000000001e-05 2024-04-20T17:32:09.011 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.011 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.011 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:32:09.011 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.644515+0000", 2024-04-20T17:32:09.011 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4627999999999999e-05 2024-04-20T17:32:09.011 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.011 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:09.011 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.011 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.011 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.012 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26074.0:57713 4.c 4:38f8b18d:::benchmark_data_smithi171_122772_object57712:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e457)", 2024-04-20T17:32:09.012 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:25:04.578710+0000", 2024-04-20T17:32:09.012 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 424.38848330500002, 2024-04-20T17:32:09.012 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.065931321000000001, 2024-04-20T17:32:09.012 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:32:09.012 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:32:09.012 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:09.012 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:32:09.012 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26074", 2024-04-20T17:32:09.012 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:32:09.012 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 57713 2024-04-20T17:32:09.012 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.012 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:32:09.012 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.013 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:32:09.013 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578710+0000", 2024-04-20T17:32:09.013 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.013 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.013 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.013 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:32:09.013 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578710+0000", 2024-04-20T17:32:09.013 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.013 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.013 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.013 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:32:09.013 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578712+0000", 2024-04-20T17:32:09.013 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6700000000000001e-06 2024-04-20T17:32:09.013 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.014 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.014 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:32:09.014 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578769+0000", 2024-04-20T17:32:09.014 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7652999999999998e-05 2024-04-20T17:32:09.014 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.014 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.014 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:32:09.014 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578779+0000", 2024-04-20T17:32:09.014 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0049e-05 2024-04-20T17:32:09.014 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.015 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:32:09.015 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578785+0000", 2024-04-20T17:32:09.015 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.852e-06 2024-04-20T17:32:09.015 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.015 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:32:09.015 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578830+0000", 2024-04-20T17:32:09.015 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5249e-05 2024-04-20T17:32:09.015 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.015 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:32:09.015 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578927+0000", 2024-04-20T17:32:09.015 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6475e-05 2024-04-20T17:32:09.016 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.016 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.016 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:32:09.016 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.579034+0000", 2024-04-20T17:32:09.016 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010687399999999999 2024-04-20T17:32:09.016 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.016 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.016 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:32:09.016 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.581249+0000", 2024-04-20T17:32:09.016 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022152859999999999 2024-04-20T17:32:09.016 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.016 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.017 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:09.017 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.644557+0000", 2024-04-20T17:32:09.017 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.063307873000000001 2024-04-20T17:32:09.017 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.017 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.017 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:32:09.017 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.644598+0000", 2024-04-20T17:32:09.017 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1362999999999997e-05 2024-04-20T17:32:09.017 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.017 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.017 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:32:09.017 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.644641+0000", 2024-04-20T17:32:09.017 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2976999999999999e-05 2024-04-20T17:32:09.018 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.018 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:09.018 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.018 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.018 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.018 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26074.0:57714 4.c 4:386280b9:::benchmark_data_smithi171_122772_object57713:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e457)", 2024-04-20T17:32:09.018 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:25:04.578796+0000", 2024-04-20T17:32:09.018 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 424.38839737500001, 2024-04-20T17:32:09.018 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.066014529000000002, 2024-04-20T17:32:09.018 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:32:09.018 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:32:09.018 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:09.018 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:32:09.018 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26074", 2024-04-20T17:32:09.019 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:32:09.019 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 57714 2024-04-20T17:32:09.019 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.019 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:32:09.019 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.019 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:32:09.019 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578796+0000", 2024-04-20T17:32:09.019 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.019 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.019 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.019 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:32:09.019 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578796+0000", 2024-04-20T17:32:09.019 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.019 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.019 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.019 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:32:09.020 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578797+0000", 2024-04-20T17:32:09.020 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4399999999999999e-07 2024-04-20T17:32:09.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.020 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:32:09.020 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578820+0000", 2024-04-20T17:32:09.020 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3425999999999999e-05 2024-04-20T17:32:09.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.020 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:32:09.020 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578824+0000", 2024-04-20T17:32:09.020 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5229999999999997e-06 2024-04-20T17:32:09.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.020 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:32:09.021 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.578830+0000", 2024-04-20T17:32:09.021 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2580000000000003e-06 2024-04-20T17:32:09.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.021 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:32:09.021 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.580253+0000", 2024-04-20T17:32:09.021 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0014233679999999999 2024-04-20T17:32:09.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.021 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:32:09.021 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.580361+0000", 2024-04-20T17:32:09.021 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010793699999999999 2024-04-20T17:32:09.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:32:09.022 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.580465+0000", 2024-04-20T17:32:09.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000104218 2024-04-20T17:32:09.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:32:09.022 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.581337+0000", 2024-04-20T17:32:09.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00087222799999999996 2024-04-20T17:32:09.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:09.022 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.644712+0000", 2024-04-20T17:32:09.023 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.063374377999999995 2024-04-20T17:32:09.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.023 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:32:09.023 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.644766+0000", 2024-04-20T17:32:09.023 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4304e-05 2024-04-20T17:32:09.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.023 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:32:09.023 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:04.644810+0000", 2024-04-20T17:32:09.023 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4144999999999998e-05 2024-04-20T17:32:09.023 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.023 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:09.023 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.024 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.024 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.024 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26074.0:67472 4.e 4:717ad074:::benchmark_data_smithi171_122772_object67471:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e458)", 2024-04-20T17:32:09.024 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:25:08.545538+0000", 2024-04-20T17:32:09.024 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 420.42165557999999, 2024-04-20T17:32:09.024 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.066188420999999997, 2024-04-20T17:32:09.024 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:32:09.024 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:32:09.024 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:09.024 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:32:09.024 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26074", 2024-04-20T17:32:09.024 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:32:09.024 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67472 2024-04-20T17:32:09.024 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.025 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:32:09.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.025 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:32:09.025 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.545538+0000", 2024-04-20T17:32:09.025 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.025 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.025 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:32:09.025 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.545538+0000", 2024-04-20T17:32:09.025 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.025 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.025 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:32:09.025 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.545538+0000", 2024-04-20T17:32:09.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1299999999999998e-07 2024-04-20T17:32:09.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.026 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:32:09.026 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.545721+0000", 2024-04-20T17:32:09.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000182504 2024-04-20T17:32:09.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.026 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:32:09.026 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.545728+0000", 2024-04-20T17:32:09.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0489999999999998e-06 2024-04-20T17:32:09.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.027 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:32:09.027 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.545734+0000", 2024-04-20T17:32:09.027 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6550000000000001e-06 2024-04-20T17:32:09.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.027 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:32:09.027 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.548549+0000", 2024-04-20T17:32:09.027 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0028149730000000001 2024-04-20T17:32:09.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.027 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:32:09.027 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.548628+0000", 2024-04-20T17:32:09.027 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9009000000000002e-05 2024-04-20T17:32:09.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.028 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:32:09.028 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.548705+0000", 2024-04-20T17:32:09.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7286999999999994e-05 2024-04-20T17:32:09.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.028 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:32:09.028 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.551339+0000", 2024-04-20T17:32:09.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0026335859999999998 2024-04-20T17:32:09.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.028 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:09.028 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611636+0000", 2024-04-20T17:32:09.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.060297852999999998 2024-04-20T17:32:09.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.029 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:32:09.029 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611696+0000", 2024-04-20T17:32:09.029 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9827000000000002e-05 2024-04-20T17:32:09.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.029 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:32:09.029 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611726+0000", 2024-04-20T17:32:09.029 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9765e-05 2024-04-20T17:32:09.029 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.029 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:09.029 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.030 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26074.0:67478 4.e 4:74b2a46e:::benchmark_data_smithi171_122772_object67477:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e458)", 2024-04-20T17:32:09.030 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:25:08.548203+0000", 2024-04-20T17:32:09.030 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 420.41899051399997, 2024-04-20T17:32:09.030 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.063669150999999993, 2024-04-20T17:32:09.030 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:32:09.030 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:32:09.030 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:09.030 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:32:09.030 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26074", 2024-04-20T17:32:09.030 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:32:09.030 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67478 2024-04-20T17:32:09.030 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.030 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:32:09.030 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.030 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:32:09.031 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.548203+0000", 2024-04-20T17:32:09.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.031 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:32:09.031 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.548203+0000", 2024-04-20T17:32:09.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.031 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:32:09.031 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.548204+0000", 2024-04-20T17:32:09.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.502e-06 2024-04-20T17:32:09.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.032 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:32:09.032 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.548262+0000", 2024-04-20T17:32:09.032 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8094000000000003e-05 2024-04-20T17:32:09.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.032 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:32:09.032 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.548271+0000", 2024-04-20T17:32:09.032 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.303e-06 2024-04-20T17:32:09.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.032 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:32:09.032 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.548276+0000", 2024-04-20T17:32:09.032 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7309999999999998e-06 2024-04-20T17:32:09.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.033 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:32:09.033 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.549768+0000", 2024-04-20T17:32:09.033 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0014920210000000001 2024-04-20T17:32:09.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.033 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:32:09.033 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.549870+0000", 2024-04-20T17:32:09.033 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000102002 2024-04-20T17:32:09.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.033 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:32:09.034 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.549998+0000", 2024-04-20T17:32:09.034 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000128007 2024-04-20T17:32:09.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.034 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:32:09.034 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.551461+0000", 2024-04-20T17:32:09.034 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001463015 2024-04-20T17:32:09.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.034 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:09.034 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611818+0000", 2024-04-20T17:32:09.034 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.060356685 2024-04-20T17:32:09.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.035 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:32:09.035 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611851+0000", 2024-04-20T17:32:09.035 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3114000000000002e-05 2024-04-20T17:32:09.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.035 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.035 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:32:09.035 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611872+0000", 2024-04-20T17:32:09.035 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0676999999999999e-05 2024-04-20T17:32:09.035 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.035 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:09.035 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.035 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.036 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26074.0:67486 4.e 4:7a975fb4:::benchmark_data_smithi171_122772_object67485:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e458)", 2024-04-20T17:32:09.036 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:25:08.551436+0000", 2024-04-20T17:32:09.036 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 420.41575752099999, 2024-04-20T17:32:09.036 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.060534031000000002, 2024-04-20T17:32:09.036 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:32:09.036 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:32:09.036 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:09.036 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:32:09.036 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26074", 2024-04-20T17:32:09.036 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:32:09.036 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67486 2024-04-20T17:32:09.036 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.036 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:32:09.036 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.036 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:32:09.037 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.551436+0000", 2024-04-20T17:32:09.037 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.037 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.037 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:32:09.037 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.551436+0000", 2024-04-20T17:32:09.037 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.037 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.037 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:32:09.037 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.551437+0000", 2024-04-20T17:32:09.037 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4699999999999999e-06 2024-04-20T17:32:09.037 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.037 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:32:09.038 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.551506+0000", 2024-04-20T17:32:09.038 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9208000000000006e-05 2024-04-20T17:32:09.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.038 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.038 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:32:09.038 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.551516+0000", 2024-04-20T17:32:09.038 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0173999999999999e-05 2024-04-20T17:32:09.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.038 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.038 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:32:09.038 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.551524+0000", 2024-04-20T17:32:09.038 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3159999999999999e-06 2024-04-20T17:32:09.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.038 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.038 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:32:09.038 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.551564+0000", 2024-04-20T17:32:09.039 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0593999999999998e-05 2024-04-20T17:32:09.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.039 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:32:09.040 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.551655+0000", 2024-04-20T17:32:09.040 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0486000000000001e-05 2024-04-20T17:32:09.040 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.040 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.040 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:32:09.040 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.551735+0000", 2024-04-20T17:32:09.040 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0178000000000004e-05 2024-04-20T17:32:09.040 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.040 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.040 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:32:09.041 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553066+0000", 2024-04-20T17:32:09.041 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001330721 2024-04-20T17:32:09.041 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.041 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.041 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:09.041 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611896+0000", 2024-04-20T17:32:09.041 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.058829748000000001 2024-04-20T17:32:09.041 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.041 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.041 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:32:09.041 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611945+0000", 2024-04-20T17:32:09.041 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9048000000000002e-05 2024-04-20T17:32:09.041 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.041 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.042 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:32:09.042 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611970+0000", 2024-04-20T17:32:09.042 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5088000000000001e-05 2024-04-20T17:32:09.042 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.042 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:09.042 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.042 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.042 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.042 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26074.0:67489 4.e 4:7f789828:::benchmark_data_smithi171_122772_object67488:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e458)", 2024-04-20T17:32:09.042 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:25:08.553188+0000", 2024-04-20T17:32:09.042 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 420.414004847, 2024-04-20T17:32:09.042 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.058877112000000002, 2024-04-20T17:32:09.042 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:32:09.043 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:32:09.043 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:09.043 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:32:09.043 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26074", 2024-04-20T17:32:09.043 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:32:09.043 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67489 2024-04-20T17:32:09.043 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.043 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:32:09.043 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.043 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:32:09.043 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553188+0000", 2024-04-20T17:32:09.043 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.043 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.043 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.043 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:32:09.044 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553188+0000", 2024-04-20T17:32:09.044 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.044 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.044 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.044 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:32:09.044 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553190+0000", 2024-04-20T17:32:09.044 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.578e-06 2024-04-20T17:32:09.044 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.044 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.044 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:32:09.044 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553282+0000", 2024-04-20T17:32:09.044 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2596999999999995e-05 2024-04-20T17:32:09.044 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.044 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.045 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:32:09.045 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553293+0000", 2024-04-20T17:32:09.045 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0372e-05 2024-04-20T17:32:09.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.045 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:32:09.045 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553306+0000", 2024-04-20T17:32:09.045 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2894000000000001e-05 2024-04-20T17:32:09.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.045 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:32:09.045 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553349+0000", 2024-04-20T17:32:09.045 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3517999999999999e-05 2024-04-20T17:32:09.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.046 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:32:09.046 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553442+0000", 2024-04-20T17:32:09.046 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2981999999999998e-05 2024-04-20T17:32:09.046 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.046 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.046 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:32:09.046 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553542+0000", 2024-04-20T17:32:09.046 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9889000000000005e-05 2024-04-20T17:32:09.046 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.046 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.046 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:32:09.046 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611740+0000", 2024-04-20T17:32:09.046 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.058197394 2024-04-20T17:32:09.046 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.047 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.047 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:09.047 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.612014+0000", 2024-04-20T17:32:09.047 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00027397700000000001 2024-04-20T17:32:09.047 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.047 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.047 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:32:09.047 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.612040+0000", 2024-04-20T17:32:09.047 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6957e-05 2024-04-20T17:32:09.047 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.047 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.047 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:32:09.047 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.612065+0000", 2024-04-20T17:32:09.047 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4953999999999999e-05 2024-04-20T17:32:09.047 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.048 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:09.048 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.048 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.048 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.048 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26074.0:67492 4.c 4:3e44fca8:::benchmark_data_smithi171_122772_object67491:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e458)", 2024-04-20T17:32:09.048 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:25:08.553598+0000", 2024-04-20T17:32:09.048 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 420.41359518399997, 2024-04-20T17:32:09.048 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.058295435999999999, 2024-04-20T17:32:09.048 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:32:09.048 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:32:09.048 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:09.048 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:32:09.048 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26074", 2024-04-20T17:32:09.048 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:32:09.048 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67492 2024-04-20T17:32:09.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.049 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:32:09.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.049 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:32:09.049 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553598+0000", 2024-04-20T17:32:09.049 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.049 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:32:09.049 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553598+0000", 2024-04-20T17:32:09.049 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.049 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:32:09.049 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553600+0000", 2024-04-20T17:32:09.050 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5349999999999999e-06 2024-04-20T17:32:09.050 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.050 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.050 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:32:09.050 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553655+0000", 2024-04-20T17:32:09.050 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5262e-05 2024-04-20T17:32:09.050 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.050 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.050 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:32:09.050 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553664+0000", 2024-04-20T17:32:09.050 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7630000000000006e-06 2024-04-20T17:32:09.050 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.050 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.050 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:32:09.051 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553670+0000", 2024-04-20T17:32:09.051 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.934e-06 2024-04-20T17:32:09.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.051 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.051 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:32:09.051 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553714+0000", 2024-04-20T17:32:09.051 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3523999999999999e-05 2024-04-20T17:32:09.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.051 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.051 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:32:09.051 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553787+0000", 2024-04-20T17:32:09.051 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2996000000000004e-05 2024-04-20T17:32:09.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.051 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.052 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:32:09.052 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553880+0000", 2024-04-20T17:32:09.052 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3465e-05 2024-04-20T17:32:09.052 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.052 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.052 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:32:09.052 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611756+0000", 2024-04-20T17:32:09.052 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.057875886000000001 2024-04-20T17:32:09.052 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.052 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.052 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:09.052 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611802+0000", 2024-04-20T17:32:09.052 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5352999999999998e-05 2024-04-20T17:32:09.052 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.052 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.053 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:32:09.053 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611853+0000", 2024-04-20T17:32:09.053 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1350000000000001e-05 2024-04-20T17:32:09.053 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.053 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.053 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:32:09.053 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611893+0000", 2024-04-20T17:32:09.053 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0367999999999997e-05 2024-04-20T17:32:09.053 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.053 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:09.053 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.053 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.053 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.053 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26074.0:67493 4.e 4:7c90e581:::benchmark_data_smithi171_122772_object67492:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e458)", 2024-04-20T17:32:09.054 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:25:08.553815+0000", 2024-04-20T17:32:09.054 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 420.41337861400001, 2024-04-20T17:32:09.054 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.058399287000000001, 2024-04-20T17:32:09.054 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:32:09.054 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:32:09.054 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:09.054 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:32:09.054 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26074", 2024-04-20T17:32:09.054 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:32:09.054 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67493 2024-04-20T17:32:09.054 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.054 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:32:09.054 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.055 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:32:09.055 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553815+0000", 2024-04-20T17:32:09.055 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.055 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.055 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.055 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:32:09.055 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553815+0000", 2024-04-20T17:32:09.055 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.055 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.055 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.055 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:32:09.055 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553816+0000", 2024-04-20T17:32:09.055 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1829999999999999e-06 2024-04-20T17:32:09.055 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.055 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.056 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:32:09.056 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553890+0000", 2024-04-20T17:32:09.056 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4521e-05 2024-04-20T17:32:09.056 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.056 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.056 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:32:09.056 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553897+0000", 2024-04-20T17:32:09.056 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0770000000000002e-06 2024-04-20T17:32:09.056 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.056 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.056 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:32:09.056 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553903+0000", 2024-04-20T17:32:09.056 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.784e-06 2024-04-20T17:32:09.056 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.056 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.056 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:32:09.057 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.555067+0000", 2024-04-20T17:32:09.057 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001163772 2024-04-20T17:32:09.057 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.057 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.057 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:32:09.057 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.555152+0000", 2024-04-20T17:32:09.057 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5455000000000001e-05 2024-04-20T17:32:09.057 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.057 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.057 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:32:09.057 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.555229+0000", 2024-04-20T17:32:09.057 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7152999999999998e-05 2024-04-20T17:32:09.057 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.057 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.057 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:32:09.057 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611786+0000", 2024-04-20T17:32:09.058 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.056556228 2024-04-20T17:32:09.058 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.058 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.058 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:09.058 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.612111+0000", 2024-04-20T17:32:09.058 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00032532900000000002 2024-04-20T17:32:09.058 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.058 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.058 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:32:09.058 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.612193+0000", 2024-04-20T17:32:09.058 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2028999999999997e-05 2024-04-20T17:32:09.058 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.058 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.058 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:32:09.059 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.612214+0000", 2024-04-20T17:32:09.059 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0755999999999999e-05 2024-04-20T17:32:09.059 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.059 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:09.059 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.059 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.059 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.059 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26074.0:67494 4.c 4:305ad90d:::benchmark_data_smithi171_122772_object67493:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e458)", 2024-04-20T17:32:09.059 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:25:08.553916+0000", 2024-04-20T17:32:09.059 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 420.41327745400002, 2024-04-20T17:32:09.059 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.058192206000000003, 2024-04-20T17:32:09.059 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:32:09.059 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:32:09.059 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:09.059 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:32:09.060 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26074", 2024-04-20T17:32:09.060 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:32:09.060 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67494 2024-04-20T17:32:09.060 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.060 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:32:09.060 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.060 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:32:09.060 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553916+0000", 2024-04-20T17:32:09.060 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.060 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.060 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.060 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:32:09.060 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553916+0000", 2024-04-20T17:32:09.060 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.060 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.061 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.061 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:32:09.061 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.553916+0000", 2024-04-20T17:32:09.061 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3200000000000004e-07 2024-04-20T17:32:09.061 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.061 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.061 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:32:09.061 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.554003+0000", 2024-04-20T17:32:09.061 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6440999999999995e-05 2024-04-20T17:32:09.061 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.061 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.061 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:32:09.061 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.554016+0000", 2024-04-20T17:32:09.061 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3345e-05 2024-04-20T17:32:09.061 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.062 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.062 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:32:09.062 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.554022+0000", 2024-04-20T17:32:09.062 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1580000000000002e-06 2024-04-20T17:32:09.062 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.062 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.062 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:32:09.062 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.555541+0000", 2024-04-20T17:32:09.062 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0015190970000000001 2024-04-20T17:32:09.062 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.062 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.062 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:32:09.062 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.555658+0000", 2024-04-20T17:32:09.062 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000116316 2024-04-20T17:32:09.063 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.063 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.063 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:32:09.063 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.555776+0000", 2024-04-20T17:32:09.063 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000117861 2024-04-20T17:32:09.063 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.063 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.063 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:32:09.063 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611763+0000", 2024-04-20T17:32:09.063 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.055987396000000002 2024-04-20T17:32:09.063 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.063 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.063 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:09.063 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611967+0000", 2024-04-20T17:32:09.064 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00020356899999999999 2024-04-20T17:32:09.064 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.064 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.064 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:32:09.064 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.612034+0000", 2024-04-20T17:32:09.064 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7879000000000005e-05 2024-04-20T17:32:09.064 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.064 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.064 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:32:09.064 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.612108+0000", 2024-04-20T17:32:09.064 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3411999999999994e-05 2024-04-20T17:32:09.064 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.064 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:09.064 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.064 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.065 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.065 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26074.0:67498 4.d 4:b63a32dc:::benchmark_data_smithi171_122772_object67497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e458)", 2024-04-20T17:32:09.065 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:25:08.557009+0000", 2024-04-20T17:32:09.065 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 420.41018406799998, 2024-04-20T17:32:09.065 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.059470428999999998, 2024-04-20T17:32:09.065 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:32:09.065 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:32:09.065 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:09.065 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:32:09.065 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26074", 2024-04-20T17:32:09.065 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:32:09.065 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67498 2024-04-20T17:32:09.065 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.065 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:32:09.065 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.066 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:32:09.066 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.557009+0000", 2024-04-20T17:32:09.066 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.066 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.066 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.066 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:32:09.066 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.557009+0000", 2024-04-20T17:32:09.066 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.066 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.066 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.066 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:32:09.066 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.557010+0000", 2024-04-20T17:32:09.066 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3459999999999999e-06 2024-04-20T17:32:09.066 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.067 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.067 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:32:09.067 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.557120+0000", 2024-04-20T17:32:09.067 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010925999999999999 2024-04-20T17:32:09.067 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.067 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.067 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:32:09.067 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611595+0000", 2024-04-20T17:32:09.067 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.054474886 2024-04-20T17:32:09.067 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.067 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.067 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:32:09.067 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611608+0000", 2024-04-20T17:32:09.067 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3089e-05 2024-04-20T17:32:09.067 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.068 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.068 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:32:09.068 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611663+0000", 2024-04-20T17:32:09.068 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5305e-05 2024-04-20T17:32:09.068 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.068 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.068 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:32:09.068 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611814+0000", 2024-04-20T17:32:09.068 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015124600000000001 2024-04-20T17:32:09.068 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.068 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.068 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 9", 2024-04-20T17:32:09.068 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.611994+0000", 2024-04-20T17:32:09.068 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00018020699999999999 2024-04-20T17:32:09.068 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.069 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.069 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:32:09.069 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.616331+0000", 2024-04-20T17:32:09.069 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.004336749 2024-04-20T17:32:09.069 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.069 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.069 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:09.069 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.616375+0000", 2024-04-20T17:32:09.069 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3829000000000003e-05 2024-04-20T17:32:09.069 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.069 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.069 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:32:09.069 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.616432+0000", 2024-04-20T17:32:09.069 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7243000000000001e-05 2024-04-20T17:32:09.069 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.070 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.070 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:32:09.070 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:25:08.616480+0000", 2024-04-20T17:32:09.070 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7268999999999998e-05 2024-04-20T17:32:09.070 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.070 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:09.070 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.070 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.070 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.070 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.16506.0:164649 5.c 5:383c785e:::benchmark_data_smithi171_137281_object33215:head [delete] snapc 0=[] ondisk+write+known_if_redirected e591)", 2024-04-20T17:32:09.070 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:31:35.686043+0000", 2024-04-20T17:32:09.070 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 33.281150412000002, 2024-04-20T17:32:09.070 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.042156316999999999, 2024-04-20T17:32:09.070 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:32:09.071 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:32:09.071 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:09.071 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:32:09.071 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.16506", 2024-04-20T17:32:09.071 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:32:09.071 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 164649 2024-04-20T17:32:09.071 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.071 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:32:09.071 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.071 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:32:09.071 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:31:35.686043+0000", 2024-04-20T17:32:09.071 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.071 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.071 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.071 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:32:09.072 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:31:35.686043+0000", 2024-04-20T17:32:09.072 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.072 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.072 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.072 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:32:09.072 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:31:35.686044+0000", 2024-04-20T17:32:09.072 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.542e-06 2024-04-20T17:32:09.072 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.072 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.072 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:32:09.072 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:31:35.686049+0000", 2024-04-20T17:32:09.072 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1499999999999998e-06 2024-04-20T17:32:09.072 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.072 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.072 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:32:09.073 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:31:35.686059+0000", 2024-04-20T17:32:09.073 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.329e-06 2024-04-20T17:32:09.073 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.073 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.073 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:32:09.073 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:31:35.686067+0000", 2024-04-20T17:32:09.073 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7819999999999994e-06 2024-04-20T17:32:09.073 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.073 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.073 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:32:09.073 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:31:35.691176+0000", 2024-04-20T17:32:09.073 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0051089960000000002 2024-04-20T17:32:09.073 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.073 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.074 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:32:09.074 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:31:35.691282+0000", 2024-04-20T17:32:09.074 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010620600000000001 2024-04-20T17:32:09.074 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.074 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.074 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:32:09.074 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:31:35.691356+0000", 2024-04-20T17:32:09.074 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4605999999999997e-05 2024-04-20T17:32:09.074 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.074 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.074 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:32:09.074 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:31:35.694881+0000", 2024-04-20T17:32:09.074 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0035242770000000001 2024-04-20T17:32:09.074 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.074 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.075 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:09.075 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:31:35.728137+0000", 2024-04-20T17:32:09.075 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.033255996000000003 2024-04-20T17:32:09.075 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.075 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.075 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:32:09.075 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:31:35.728171+0000", 2024-04-20T17:32:09.075 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4751e-05 2024-04-20T17:32:09.075 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.075 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.075 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:32:09.075 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:31:35.728199+0000", 2024-04-20T17:32:09.075 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7682e-05 2024-04-20T17:32:09.076 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.076 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:09.076 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.076 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.076 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.076 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.16506.0:196396 5.f 5:fb47c3fc:::benchmark_data_smithi171_137281_object64962:head [delete] snapc 0=[] ondisk+write+known_if_redirected e598)", 2024-04-20T17:32:09.076 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:32:02.501444+0000", 2024-04-20T17:32:09.076 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 6.4657493309999996, 2024-04-20T17:32:09.076 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.039483710999999998, 2024-04-20T17:32:09.076 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:32:09.076 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:32:09.076 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:09.076 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:32:09.076 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.16506", 2024-04-20T17:32:09.076 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:32:09.077 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 196396 2024-04-20T17:32:09.077 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.077 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:32:09.077 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.077 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:32:09.077 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.501444+0000", 2024-04-20T17:32:09.077 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.077 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.077 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.077 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:32:09.077 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.501444+0000", 2024-04-20T17:32:09.077 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.077 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.077 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.077 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:32:09.078 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.501445+0000", 2024-04-20T17:32:09.078 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1009999999999999e-06 2024-04-20T17:32:09.078 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.078 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.078 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:32:09.078 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.501449+0000", 2024-04-20T17:32:09.078 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9519999999999996e-06 2024-04-20T17:32:09.078 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.078 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.078 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:32:09.078 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.501456+0000", 2024-04-20T17:32:09.078 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5989999999999996e-06 2024-04-20T17:32:09.078 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.078 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.079 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:32:09.079 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.501463+0000", 2024-04-20T17:32:09.079 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4620000000000001e-06 2024-04-20T17:32:09.079 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.079 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.079 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:32:09.079 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.506085+0000", 2024-04-20T17:32:09.079 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0046221889999999996 2024-04-20T17:32:09.079 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.079 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.079 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:32:09.079 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.506210+0000", 2024-04-20T17:32:09.079 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012527299999999999 2024-04-20T17:32:09.079 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.080 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.080 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5", 2024-04-20T17:32:09.080 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.506335+0000", 2024-04-20T17:32:09.080 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012435899999999999 2024-04-20T17:32:09.080 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.080 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.080 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:32:09.080 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.525629+0000", 2024-04-20T17:32:09.080 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.019294221 2024-04-20T17:32:09.080 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.080 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.080 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:09.080 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.540830+0000", 2024-04-20T17:32:09.080 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015200842000000001 2024-04-20T17:32:09.081 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.081 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.081 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:32:09.081 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.540881+0000", 2024-04-20T17:32:09.081 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1607000000000002e-05 2024-04-20T17:32:09.081 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.081 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.081 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:32:09.081 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:02.540928+0000", 2024-04-20T17:32:09.081 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6106000000000002e-05 2024-04-20T17:32:09.081 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.081 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:09.081 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.081 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.081 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.082 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.16506.0:197905 5.1 5:86558532:::benchmark_data_smithi171_137281_object66471:head [delete] snapc 0=[] ondisk+write+known_if_redirected e598)", 2024-04-20T17:32:09.082 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:32:04.011011+0000", 2024-04-20T17:32:09.082 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 4.9561822680000001, 2024-04-20T17:32:09.082 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.040555348999999997, 2024-04-20T17:32:09.082 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:32:09.082 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:32:09.082 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:09.082 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:32:09.082 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.16506", 2024-04-20T17:32:09.082 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:32:09.082 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 197905 2024-04-20T17:32:09.082 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.082 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:32:09.082 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.082 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:32:09.083 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:04.011011+0000", 2024-04-20T17:32:09.083 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.083 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.083 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.083 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:32:09.083 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:04.011011+0000", 2024-04-20T17:32:09.083 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:32:09.083 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.083 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.083 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:32:09.083 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:04.011012+0000", 2024-04-20T17:32:09.083 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4169999999999999e-06 2024-04-20T17:32:09.083 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.083 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.083 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:32:09.084 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:04.011016+0000", 2024-04-20T17:32:09.084 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4589999999999998e-06 2024-04-20T17:32:09.084 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.084 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.084 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:32:09.084 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:04.011023+0000", 2024-04-20T17:32:09.084 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0949999999999998e-06 2024-04-20T17:32:09.084 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.084 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.084 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:32:09.084 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:04.011029+0000", 2024-04-20T17:32:09.084 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0889999999999996e-06 2024-04-20T17:32:09.084 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.084 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.085 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:32:09.085 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:04.017113+0000", 2024-04-20T17:32:09.085 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0060841280000000003 2024-04-20T17:32:09.085 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.085 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.085 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:32:09.085 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:04.017194+0000", 2024-04-20T17:32:09.085 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0568999999999994e-05 2024-04-20T17:32:09.085 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.085 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.085 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:32:09.085 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:04.017280+0000", 2024-04-20T17:32:09.085 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6722000000000001e-05 2024-04-20T17:32:09.085 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.085 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.085 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:32:09.086 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:04.020426+0000", 2024-04-20T17:32:09.086 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0031457910000000002 2024-04-20T17:32:09.086 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.086 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.086 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:09.086 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:04.051468+0000", 2024-04-20T17:32:09.086 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.031041870999999999 2024-04-20T17:32:09.086 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.086 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.086 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:32:09.086 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:04.051526+0000", 2024-04-20T17:32:09.086 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8375000000000003e-05 2024-04-20T17:32:09.086 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:32:09.087 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:32:09.087 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:32:09.087 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:32:04.051566+0000", 2024-04-20T17:32:09.087 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9833000000000003e-05 2024-04-20T17:32:09.087 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.087 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:09.087 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.087 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:32:09.087 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:32:09.087 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:32:09.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:09.088 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:32:09.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:32:09.189 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:32:09.221 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:09.246 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:32:09.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:09.249 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:32:09.350 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:32:09.378 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:09.403 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:32:09.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:09.409 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:32:09.510 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:32:09.555 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:09.778 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-20T17:32:09.843 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:32:09.844 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":599,"fsid":"13c6e76c-ff37-11ee-bc93-c7b262605968","created":"2024-04-20T16:58:25.272557+0000","modified":"2024-04-20T17:32:06.663761+0000","last_up_change":"2024-04-20T17:30:09.260532+0000","last_in_change":"2024-04-20T17:27:35.262728+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":31,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T17:01:35.909493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.109999656677246,"score_stable":11.109999656677246,"optimal_score":0.18000000715255737,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.83594763278961182,"average_primary_affinity":0.9100000262260437,"average_primary_affinity_weighted":0.8399999737739563}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2024-04-20T17:29:28.058450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"547","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.7400000095367432,"score_stable":2.059999942779541,"optimal_score":1,"raw_score_acting":2.7400000095367432,"raw_score_stable":2.059999942779541,"primary_affinity_weighted":0.83594763278961182,"average_primary_affinity":0.9100000262260437,"average_primary_affinity_weighted":0.8399999737739563,"error_message":"pool 5 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate"}}],"osds":[{"osd":0,"uuid":"ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":543,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6801","nonce":3901484705}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6802","nonce":3901484705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6804","nonce":3901484705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6803","nonce":3901484705}]},"public_addr":"172.21.15.5:6801/3901484705","cluster_addr":"172.21.15.5:6802/3901484705","heartbeat_back_addr":"172.21.15.5:6804/3901484705","heartbeat_front_addr":"172.21.15.5:6803/3901484705","state":["exists","up"]},{"osd":1,"uuid":"aed0c8eb-0e81-4e85-ab8f-a60420060656","up":1,"in":1,"weight":1,"primary_affinity":0.0313720703125,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":539,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6805","nonce":3391893914}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6806","nonce":3391893914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6808","nonce":3391893914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6807","nonce":3391893914}]},"public_addr":"172.21.15.5:6805/3391893914","cluster_addr":"172.21.15.5:6806/3391893914","heartbeat_back_addr":"172.21.15.5:6808/3391893914","heartbeat_front_addr":"172.21.15.5:6807/3391893914","state":["exists","up"]},{"osd":2,"uuid":"f733a6a2-19c2-4c7a-af9c-04e282062bb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":20,"last_clean_end":169,"up_from":202,"up_thru":543,"down_at":170,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6809","nonce":2120275501}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6810","nonce":2120275501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6812","nonce":2120275501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6811","nonce":2120275501}]},"public_addr":"172.21.15.5:6809/2120275501","cluster_addr":"172.21.15.5:6810/2120275501","heartbeat_back_addr":"172.21.15.5:6812/2120275501","heartbeat_front_addr":"172.21.15.5:6811/2120275501","state":["exists","up"]},{"osd":3,"uuid":"8c92f1df-e298-4606-88ae-b8a8e79628a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":538,"up_from":561,"up_thru":445,"down_at":539,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6813","nonce":2272824046}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6814","nonce":2272824046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6816","nonce":2272824046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6815","nonce":2272824046}]},"public_addr":"172.21.15.5:6813/2272824046","cluster_addr":"172.21.15.5:6814/2272824046","heartbeat_back_addr":"172.21.15.5:6816/2272824046","heartbeat_front_addr":"172.21.15.5:6815/2272824046","state":["exists","up"]},{"osd":4,"uuid":"46df16f1-9607-4d7b-9be8-9d4236e160d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":561,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6800","nonce":1839524469}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6801","nonce":1839524469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6803","nonce":1839524469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6802","nonce":1839524469}]},"public_addr":"172.21.15.106:6800/1839524469","cluster_addr":"172.21.15.106:6801/1839524469","heartbeat_back_addr":"172.21.15.106:6803/1839524469","heartbeat_front_addr":"172.21.15.106:6802/1839524469","state":["exists","up"]},{"osd":5,"uuid":"6603231d-1e5c-4575-bfaa-2eeca2c12215","up":1,"in":1,"weight":1,"primary_affinity":0,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":543,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6804","nonce":2450315908}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6805","nonce":2450315908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6807","nonce":2450315908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6806","nonce":2450315908}]},"public_addr":"172.21.15.106:6804/2450315908","cluster_addr":"172.21.15.106:6805/2450315908","heartbeat_back_addr":"172.21.15.106:6807/2450315908","heartbeat_front_addr":"172.21.15.106:6806/2450315908","state":["exists","up"]},{"osd":6,"uuid":"783a55a6-f176-42be-8c8b-af8481d08dc9","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":48,"last_clean_end":503,"up_from":535,"up_thru":445,"down_at":504,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6808","nonce":2439382182}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6809","nonce":2439382182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6811","nonce":2439382182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6810","nonce":2439382182}]},"public_addr":"172.21.15.106:6808/2439382182","cluster_addr":"172.21.15.106:6809/2439382182","heartbeat_back_addr":"172.21.15.106:6811/2439382182","heartbeat_front_addr":"172.21.15.106:6810/2439382182","state":["exists","up"]},{"osd":7,"uuid":"b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":56,"last_clean_end":364,"up_from":398,"up_thru":543,"down_at":365,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6812","nonce":1997086583}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6813","nonce":1997086583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6815","nonce":1997086583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6814","nonce":1997086583}]},"public_addr":"172.21.15.106:6812/1997086583","cluster_addr":"172.21.15.106:6813/1997086583","heartbeat_back_addr":"172.21.15.106:6815/1997086583","heartbeat_front_addr":"172.21.15.106:6814/1997086583","state":["exists","up"]},{"osd":8,"uuid":"1cb232ad-1f59-4a0a-a215-e67419ad6476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":418,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6800","nonce":4123058942}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6801","nonce":4123058942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6803","nonce":4123058942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6802","nonce":4123058942}]},"public_addr":"172.21.15.171:6800/4123058942","cluster_addr":"172.21.15.171:6801/4123058942","heartbeat_back_addr":"172.21.15.171:6803/4123058942","heartbeat_front_addr":"172.21.15.171:6802/4123058942","state":["exists","up"]},{"osd":9,"uuid":"b6c17147-b3dc-475b-bb20-f9f712fb99e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":543,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6804","nonce":2877589636}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6805","nonce":2877589636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6807","nonce":2877589636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6806","nonce":2877589636}]},"public_addr":"172.21.15.171:6804/2877589636","cluster_addr":"172.21.15.171:6805/2877589636","heartbeat_back_addr":"172.21.15.171:6807/2877589636","heartbeat_front_addr":"172.21.15.171:6806/2877589636","state":["exists","up"]},{"osd":10,"uuid":"1adcdf55-f002-4a39-91e7-378e5fae849c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":543,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6808","nonce":181490621}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6809","nonce":181490621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6811","nonce":181490621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6810","nonce":181490621}]},"public_addr":"172.21.15.171:6808/181490621","cluster_addr":"172.21.15.171:6809/181490621","heartbeat_back_addr":"172.21.15.171:6811/181490621","heartbeat_front_addr":"172.21.15.171:6810/181490621","state":["exists","up"]},{"osd":11,"uuid":"737c5aec-694a-4968-819d-fccf22661f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":92,"up_thru":543,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6812","nonce":3373585701}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6813","nonce":3373585701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6815","nonce":3373585701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6814","nonce":3373585701}]},"public_addr":"172.21.15.171:6812/3373585701","cluster_addr":"172.21.15.171:6813/3373585701","heartbeat_back_addr":"172.21.15.171:6815/3373585701","heartbeat_front_addr":"172.21.15.171:6814/3373585701","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:30:51.364350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:30:50.629837+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:31:22.857465+0000","dead_epoch":169},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:31:33.723139+0000","dead_epoch":538},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:31:52.127683+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:30:56.656083+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-20T17:31:33.784895+0000","dead_epoch":503},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:31:15.540728+0000","dead_epoch":364},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:31:11.199183+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:31:27.205187+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:31:11.760959+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:31:45.416812+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.5:0/3881726425":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/338649532":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/1227843664":"2024-04-21T16:58:55.392162+0000","172.21.15.5:6800/1876064631":"2024-04-21T16:58:55.392162+0000","172.21.15.5:0/160689449":"2024-04-21T16:58:55.392162+0000","172.21.15.5:6800/3521715286":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/3841663462":"2024-04-21T16:58:55.392162+0000","172.21.15.5:0/687690919":"2024-04-21T16:59:25.808337+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T17:32:09.852 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:09 smithi005 ceph-mon[39562]: pgmap v1570: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-20T17:32:09.987 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:32:10.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:09 smithi106 ceph-mon[44423]: pgmap v1570: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-20T17:32:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:09 smithi171 ceph-mon[44189]: pgmap v1570: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-20T17:32:10.132 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:32:10.326 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:32:10.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:10.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:32:10.431 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:32:10.482 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:10.680 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:32:10.711 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:32:10.748 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:32:10.779 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:32:10.814 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:32:10.850 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:32:10.884 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:32:10.923 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:32:10.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:10 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1835345287' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:32:10.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:10 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2725004215' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:32:10.967 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:32:11.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:10 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1835345287' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:32:11.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:10 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2725004215' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:32:11.013 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:32:11.015 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-20T17:32:11.052 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:32:11.053 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:32:11.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:11.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:32:11.082 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:32:11.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:10 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1835345287' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:32:11.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:10 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2725004215' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:32:11.090 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:32:11.101 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:32:11.101 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:32:11.101 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:32:11.101 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:32:11.101 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:32:11.101 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:32:11.101 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:32:11.101 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:32:11.101 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:32:11.101 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:32:11.101 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:32:11.101 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:32:11.160 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:32:11.208 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:11.350 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:32:11.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:11.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:32:11.460 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:32:11.491 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:11.514 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:32:11.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:11.519 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:32:11.620 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:32:11.656 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:11.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:11 smithi106 ceph-mon[44423]: pgmap v1571: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.7 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 873 op/s 2024-04-20T17:32:11.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:11.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:11.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:11.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:11.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:11.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:11.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:11.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:11 smithi171 ceph-mon[44189]: pgmap v1571: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.7 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 873 op/s 2024-04-20T17:32:11.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:11.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:11.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:11.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:11.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:11.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:11.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:11.858 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:32:11.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:11.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:32:11.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:11 smithi005 ceph-mon[39562]: pgmap v1571: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.7 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 873 op/s 2024-04-20T17:32:11.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:11.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:11.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:11.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:11.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:11.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:11.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:11.965 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:32:12.017 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:12.173 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:32:12.174 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:32:12.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:12.232 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:32:12.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:32:12.332 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:32:12.365 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:12.392 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:32:12.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:12.395 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:32:12.496 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:32:12.546 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:12.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:12.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:12.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:12.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:12.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:13.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:13.080 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-20T17:32:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:13.444 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:32:14.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:13 smithi106 ceph-mon[44423]: pgmap v1572: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-20T17:32:14.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:13 smithi171 ceph-mon[44189]: pgmap v1572: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-20T17:32:14.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:13 smithi005 ceph-mon[39562]: pgmap v1572: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-20T17:32:14.244 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:32:14.720 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:32:14.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:14.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:32:14.831 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:32:14.857 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:14.883 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:32:14.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:14.890 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:32:14.991 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:32:15.032 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:15.513 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:32:15.513 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:32:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:15 smithi106 ceph-mon[44423]: pgmap v1573: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.4 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-20T17:32:16.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:15 smithi171 ceph-mon[44189]: pgmap v1573: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.4 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-20T17:32:16.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:15 smithi005 ceph-mon[39562]: pgmap v1573: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.4 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-20T17:32:16.465 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-20T17:32:16.810 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:32:16.914 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:32:16.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:16.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:32:16.982 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:32:17.021 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:32:17.068 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:17.296 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:32:17.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:17.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:32:17.403 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:32:17.431 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:17.455 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:32:17.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:17.461 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:32:17.562 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:32:17.591 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:17.614 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:32:17.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:17.617 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:32:17.719 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:32:17.731 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:17 smithi106 ceph-mon[44423]: pgmap v1574: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.3 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 955 op/s 2024-04-20T17:32:17.766 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:17.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:17 smithi005 ceph-mon[39562]: pgmap v1574: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.3 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 955 op/s 2024-04-20T17:32:17.988 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:32:18.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:18.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:32:18.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:17 smithi171 ceph-mon[44189]: pgmap v1574: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.3 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 955 op/s 2024-04-20T17:32:18.102 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:32:18.135 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:18.242 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:32:18.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:18.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:32:18.348 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:32:18.385 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:18.591 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:32:18.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:18.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:32:18.696 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:32:18.727 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:18.750 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:32:18.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:18.753 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:32:18.853 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:32:18.904 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:18.942 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:32:18.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:18.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:32:18.953 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:32:18.954 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:32:19.047 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:32:19.080 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:19.376 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:32:19.416 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:32:19.452 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:32:19.490 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:32:19.527 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:32:19.572 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:32:19.605 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:32:19.639 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:32:19.668 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:32:19.695 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-20T17:32:19.699 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:32:19.736 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:32:19.746 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:32:19.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:19.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:32:19.761 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:32:19.770 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:32:19.777 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:32:19.777 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:32:19.777 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:32:19.777 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:32:19.777 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:32:19.778 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:32:19.778 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:32:19.778 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:32:19.778 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:32:19.778 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:32:19.778 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:32:19.778 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:32:19.850 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:32:19.886 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:19.924 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:32:19.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:19.930 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:32:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:19 smithi106 ceph-mon[44423]: pgmap v1575: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.3 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 898 op/s 2024-04-20T17:32:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:19 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3118942291' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:32:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:19 smithi106 ceph-mon[44423]: from='client.26768 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:32:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:19 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1331041332' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:32:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:19 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:32:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:19 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:32:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:20.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:20.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:20.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:20.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:20.030 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:32:20.058 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:32:20.071 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:19 smithi171 ceph-mon[44189]: pgmap v1575: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.3 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 898 op/s 2024-04-20T17:32:20.071 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:19 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3118942291' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:32:20.071 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:19 smithi171 ceph-mon[44189]: from='client.26768 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:32:20.071 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:19 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1331041332' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:32:20.071 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:19 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:32:20.071 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:19 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:32:20.071 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:20.071 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:20.071 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:20.071 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:20.071 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:20.074 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:20.110 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:32:20.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:20.115 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:32:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:19 smithi005 ceph-mon[39562]: pgmap v1575: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.3 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 898 op/s 2024-04-20T17:32:20.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:19 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3118942291' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:32:20.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:19 smithi005 ceph-mon[39562]: from='client.26768 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:32:20.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:19 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1331041332' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:32:20.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:19 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:32:20.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:19 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:32:20.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:20.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:20.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:20.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:20.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:20.216 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:32:20.247 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:20.272 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:32:20.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:20.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:32:20.376 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:32:20.425 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:20.786 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:32:20.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:20.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:32:20.895 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:32:20.929 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:20 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:32:20.929 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:20 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:32:20.930 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:20.930 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:20.930 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:20.930 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:20.930 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:20.930 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:20.930 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:20 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:32:20.930 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:20 smithi171 ceph-mon[44189]: from='client.26768 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:32:20.930 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:20 smithi171 ceph-mon[44189]: osdmap e600: 12 total, 12 up, 11 in 2024-04-20T17:32:20.930 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:20.936 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:20 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:32:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:20 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:32:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:20 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:32:21.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:20 smithi106 ceph-mon[44423]: from='client.26768 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:32:21.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:20 smithi106 ceph-mon[44423]: osdmap e600: 12 total, 12 up, 11 in 2024-04-20T17:32:21.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:21.041 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:32:21.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:21.051 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:32:21.152 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:32:21.181 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:21.209 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:32:21.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:21.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:32:21.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:20 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:32:21.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:20 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:32:21.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:21.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:21.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:21.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:21.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:21.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:21.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:20 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:32:21.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:20 smithi005 ceph-mon[39562]: from='client.26768 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:32:21.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:20 smithi005 ceph-mon[39562]: osdmap e600: 12 total, 12 up, 11 in 2024-04-20T17:32:21.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:21.316 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:32:21.345 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:21.371 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:32:21.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:21.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:32:21.476 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:32:21.504 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:21.525 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:32:21.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:21.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:32:21.632 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:32:21.660 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:21.685 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:32:21.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:21.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:32:21.789 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:32:21.824 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:21.854 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:32:21.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:21.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:32:21.958 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:32:22.011 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:21 smithi171 ceph-mon[44189]: pgmap v1577: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.1 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-20T17:32:22.011 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:21 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:32:22.011 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:21 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:32:22.011 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:21 smithi171 ceph-mon[44189]: osdmap e601: 12 total, 12 up, 11 in 2024-04-20T17:32:22.014 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:22.046 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:32:22.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:22.049 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:32:22.092 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:32:22.092 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:32:22.092 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:22.092 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.16506.0:216861 5.e e601/543 5:75f0a3f0:::benchmark_data_smithi171_137281_object85427:head v 601'13363, mlcod=601'13361)", 2024-04-20T17:32:22.092 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:32:22.091088+0000", 2024-04-20T17:32:22.092 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.00051442399999999998, 2024-04-20T17:32:22.092 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00053832900000000004, 2024-04-20T17:32:22.092 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:32:22.092 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:32:22.092 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:32:22.092 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:32:22.092 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:32:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:22.091088+0000", 2024-04-20T17:32:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:32:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:22.091088+0000", 2024-04-20T17:32:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:32:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:22.091091+0000", 2024-04-20T17:32:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9490000000000001e-06 2024-04-20T17:32:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:22.094 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:32:22.094 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:22.091107+0000", 2024-04-20T17:32:22.094 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6704000000000001e-05 2024-04-20T17:32:22.094 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:22.094 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:22.094 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:32:22.094 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:22.091122+0000", 2024-04-20T17:32:22.094 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4619e-05 2024-04-20T17:32:22.094 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:22.094 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:22.094 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:32:22.094 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:22.091132+0000", 2024-04-20T17:32:22.094 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9199999999999999e-06 2024-04-20T17:32:22.094 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:22.095 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:22.095 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:32:22.095 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:22.091201+0000", 2024-04-20T17:32:22.095 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9165999999999995e-05 2024-04-20T17:32:22.095 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:22.095 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:22.095 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:32:22.095 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:22.091237+0000", 2024-04-20T17:32:22.095 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5642999999999997e-05 2024-04-20T17:32:22.095 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:22.095 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:32:22.095 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:22.096 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:22.096 INFO:teuthology.orchestra.run.smithi171.stdout: ], 2024-04-20T17:32:22.096 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 1 2024-04-20T17:32:22.096 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:32:22.150 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:32:22.196 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:22.251 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:21 smithi005 ceph-mon[39562]: pgmap v1577: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.1 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-20T17:32:22.251 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:21 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:32:22.251 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:21 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:32:22.251 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:21 smithi005 ceph-mon[39562]: osdmap e601: 12 total, 12 up, 11 in 2024-04-20T17:32:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:21 smithi106 ceph-mon[44423]: pgmap v1577: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.1 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-20T17:32:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:21 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:32:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:21 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:32:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:21 smithi106 ceph-mon[44423]: osdmap e601: 12 total, 12 up, 11 in 2024-04-20T17:32:22.777 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:32:22.858 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:32:22.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:22.864 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:32:22.898 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-20T17:32:22.965 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:32:23.013 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:23.045 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:32:23.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:23.054 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:32:23.155 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:32:23.185 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:23.202 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:32:23.211 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:32:23.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:23.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:32:23.299 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:32:23.315 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:32:23.352 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:23.401 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:32:23.559 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:32:23.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:23.563 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:32:23.664 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:32:23.699 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:23.728 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:32:23.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:23.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:32:23.833 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:32:23.863 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:23.887 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:32:23.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:23.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:32:23.994 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:32:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:23 smithi106 ceph-mon[44423]: pgmap v1579: 17 pgs: 17 active+clean; 3.0 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 981 op/s 2024-04-20T17:32:24.024 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:24.053 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:32:24.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:24.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:32:24.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:23 smithi171 ceph-mon[44189]: pgmap v1579: 17 pgs: 17 active+clean; 3.0 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 981 op/s 2024-04-20T17:32:24.157 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:32:24.207 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:23 smithi005 ceph-mon[39562]: pgmap v1579: 17 pgs: 17 active+clean; 3.0 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 981 op/s 2024-04-20T17:32:24.725 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:32:25.108 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:32:25.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:25.112 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:32:25.213 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:32:25.245 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:25.275 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:32:25.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:25.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:32:25.380 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:32:25.410 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:25.431 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:32:25.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:25.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:32:25.535 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:32:25.579 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:25.635 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:32:25.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:25.641 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:32:25.692 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:32:25.692 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:32:25.692 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-04-20T17:32:25.692 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-04-20T17:32:25.692 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:32:25.742 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:32:25.772 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:25.794 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:32:25.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:25.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:32:25.899 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:32:25.978 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:26.072 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:25 smithi171 ceph-mon[44189]: pgmap v1580: 17 pgs: 17 active+clean; 2.8 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-20T17:32:26.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:25 smithi005 ceph-mon[39562]: pgmap v1580: 17 pgs: 17 active+clean; 2.8 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-20T17:32:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:25 smithi106 ceph-mon[44423]: pgmap v1580: 17 pgs: 17 active+clean; 2.8 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s 2024-04-20T17:32:26.405 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-20T17:32:26.445 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:32:26.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:26.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:32:26.552 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:32:26.603 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:26.635 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:32:26.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:26.639 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:32:26.741 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:32:26.771 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:26.779 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:32:26.795 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:32:26.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:26.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:32:26.833 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:32:26.901 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:32:26.934 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:26 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/756987402' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:32:26.934 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:26 smithi171 ceph-mon[44189]: from='client.26668 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:32:26.936 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:26.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:26 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/756987402' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:32:26.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:26 smithi005 ceph-mon[39562]: from='client.26668 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:32:27.066 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:32:27.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:27.070 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:32:27.104 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:32:27.104 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:32:27.110 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:32:27.150 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:32:27.170 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:32:27.178 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:32:27.197 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:26 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/756987402' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:32:27.197 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:26 smithi106 ceph-mon[44423]: from='client.26668 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:32:27.199 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:27.216 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:32:27.227 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:32:27.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:27.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:32:27.248 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:32:27.276 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:32:27.317 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:32:27.332 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:32:27.354 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:32:27.360 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:27.382 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:32:27.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:27.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:32:27.388 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:32:27.416 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:32:27.425 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:32:27.425 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:32:27.425 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:32:27.425 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:32:27.425 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:32:27.425 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:32:27.425 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:32:27.426 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:32:27.426 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:32:27.426 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:32:27.426 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:32:27.426 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:32:27.486 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:32:27.547 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:27 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:32:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:27 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:32:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:27 smithi171 ceph-mon[44189]: pgmap v1581: 17 pgs: 17 active+clean; 2.6 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-20T17:32:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:27 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:32:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:27 smithi171 ceph-mon[44189]: from='client.26668 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:32:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:27 smithi171 ceph-mon[44189]: osdmap e602: 12 total, 12 up, 11 in 2024-04-20T17:32:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:27 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4155159072' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:32:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:27 smithi171 ceph-mon[44189]: 5.c scrub starts 2024-04-20T17:32:28.106 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:32:28.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:27 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:32:28.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:27 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:32:28.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:27 smithi005 ceph-mon[39562]: pgmap v1581: 17 pgs: 17 active+clean; 2.6 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-20T17:32:28.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:27 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:32:28.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:27 smithi005 ceph-mon[39562]: from='client.26668 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:32:28.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:27 smithi005 ceph-mon[39562]: osdmap e602: 12 total, 12 up, 11 in 2024-04-20T17:32:28.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:27 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4155159072' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:32:28.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.116 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.116 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:27 smithi005 ceph-mon[39562]: 5.c scrub starts 2024-04-20T17:32:28.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:28.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:32:28.218 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:32:28.258 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:28.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:27 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:32:28.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:27 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:32:28.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:27 smithi106 ceph-mon[44423]: pgmap v1581: 17 pgs: 17 active+clean; 2.6 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-20T17:32:28.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:27 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:32:28.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:27 smithi106 ceph-mon[44423]: from='client.26668 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:32:28.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:27 smithi106 ceph-mon[44423]: osdmap e602: 12 total, 12 up, 11 in 2024-04-20T17:32:28.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:27 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4155159072' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:32:28.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:27 smithi106 ceph-mon[44423]: 5.c scrub starts 2024-04-20T17:32:28.513 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:32:28.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:28.517 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:32:28.618 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:32:28.647 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:28.670 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:32:28.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:28.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:32:28.782 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:32:28.799 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:32:28.800 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:32:28.813 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:28.842 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:32:28.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:28.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:32:28.946 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:32:28.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:28 smithi005 ceph-mon[39562]: 5.2 scrub starts 2024-04-20T17:32:28.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:28.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:28 smithi005 ceph-mon[39562]: 5.c scrub ok 2024-04-20T17:32:28.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:28 smithi005 ceph-mon[39562]: 5.2 scrub ok 2024-04-20T17:32:28.986 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:29.010 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:32:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:28 smithi106 ceph-mon[44423]: 5.2 scrub starts 2024-04-20T17:32:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:28 smithi106 ceph-mon[44423]: 5.c scrub ok 2024-04-20T17:32:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:28 smithi106 ceph-mon[44423]: 5.2 scrub ok 2024-04-20T17:32:29.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:29.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:32:29.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:29.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:28 smithi171 ceph-mon[44189]: 5.2 scrub starts 2024-04-20T17:32:29.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:29.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:28 smithi171 ceph-mon[44189]: 5.c scrub ok 2024-04-20T17:32:29.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:28 smithi171 ceph-mon[44189]: 5.2 scrub ok 2024-04-20T17:32:29.117 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:32:29.169 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:29.208 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:32:29.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:29.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:32:29.316 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:32:29.356 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:29.378 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:32:29.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:29.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:32:29.481 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:32:29.499 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-20T17:32:29.549 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:29.580 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:32:29.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:29.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:32:29.684 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:32:29.718 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:29.762 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:32:29.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:29.765 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:32:29.833 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:29 smithi005 ceph-mon[39562]: pgmap v1583: 17 pgs: 17 active+clean; 2.6 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-20T17:32:29.833 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:29 smithi005 ceph-mon[39562]: 5.f scrub starts 2024-04-20T17:32:29.866 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:32:29.868 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:32:29.891 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:32:29.901 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:29.936 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:32:29.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:29.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:32:30.041 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:32:30.070 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:30.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:29 smithi171 ceph-mon[44189]: pgmap v1583: 17 pgs: 17 active+clean; 2.6 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-20T17:32:30.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:29 smithi171 ceph-mon[44189]: 5.f scrub starts 2024-04-20T17:32:30.098 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:32:30.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:30.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:32:30.105 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:29 smithi106 ceph-mon[44423]: pgmap v1583: 17 pgs: 17 active+clean; 2.6 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-20T17:32:30.105 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:29 smithi106 ceph-mon[44423]: 5.f scrub starts 2024-04-20T17:32:30.204 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:32:30.233 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:30.256 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:32:30.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:30.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:32:30.312 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:32:30.359 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:32:30.403 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:30.481 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:32:30.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:30.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:32:30.586 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:32:30.625 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:30.706 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:32:30.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:30.711 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:32:30.723 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:32:30.812 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:32:30.844 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:30.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:30 smithi005 ceph-mon[39562]: 5.5 scrub starts 2024-04-20T17:32:30.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:30 smithi005 ceph-mon[39562]: 5.f scrub ok 2024-04-20T17:32:30.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:30 smithi005 ceph-mon[39562]: 5.5 scrub ok 2024-04-20T17:32:30.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:30 smithi005 ceph-mon[39562]: 5.1 scrub starts 2024-04-20T17:32:31.015 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-20T17:32:31.016 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:32:31.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:30 smithi171 ceph-mon[44189]: 5.5 scrub starts 2024-04-20T17:32:31.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:30 smithi171 ceph-mon[44189]: 5.f scrub ok 2024-04-20T17:32:31.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:30 smithi171 ceph-mon[44189]: 5.5 scrub ok 2024-04-20T17:32:31.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:30 smithi171 ceph-mon[44189]: 5.1 scrub starts 2024-04-20T17:32:31.199 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:32:31.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:31.202 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:32:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:30 smithi106 ceph-mon[44423]: 5.5 scrub starts 2024-04-20T17:32:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:30 smithi106 ceph-mon[44423]: 5.f scrub ok 2024-04-20T17:32:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:30 smithi106 ceph-mon[44423]: 5.5 scrub ok 2024-04-20T17:32:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:30 smithi106 ceph-mon[44423]: 5.1 scrub starts 2024-04-20T17:32:31.303 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:32:31.367 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:31.408 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:32:31.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:31.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:32:31.513 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:32:31.572 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:31.825 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:32:31.825 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:32:31.837 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:32:31.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:31.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:32:31.886 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:32:31.955 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:32:32.008 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:32.189 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:31 smithi171 ceph-mon[44189]: 5.3 scrub starts 2024-04-20T17:32:32.189 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:31 smithi171 ceph-mon[44189]: pgmap v1584: 17 pgs: 17 active+clean; 2.4 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-20T17:32:32.189 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:31 smithi171 ceph-mon[44189]: 5.6 scrub starts 2024-04-20T17:32:32.190 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:31 smithi171 ceph-mon[44189]: 5.1 scrub ok 2024-04-20T17:32:32.190 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:31 smithi171 ceph-mon[44189]: 5.3 scrub ok 2024-04-20T17:32:32.190 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:31 smithi171 ceph-mon[44189]: 5.e scrub starts 2024-04-20T17:32:32.190 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:31 smithi171 ceph-mon[44189]: 5.6 scrub ok 2024-04-20T17:32:32.190 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:31 smithi171 ceph-mon[44189]: 5.e scrub ok 2024-04-20T17:32:32.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:31 smithi005 ceph-mon[39562]: 5.3 scrub starts 2024-04-20T17:32:32.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:31 smithi005 ceph-mon[39562]: pgmap v1584: 17 pgs: 17 active+clean; 2.4 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-20T17:32:32.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:31 smithi005 ceph-mon[39562]: 5.6 scrub starts 2024-04-20T17:32:32.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:31 smithi005 ceph-mon[39562]: 5.1 scrub ok 2024-04-20T17:32:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:31 smithi005 ceph-mon[39562]: 5.3 scrub ok 2024-04-20T17:32:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:31 smithi005 ceph-mon[39562]: 5.e scrub starts 2024-04-20T17:32:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:31 smithi005 ceph-mon[39562]: 5.6 scrub ok 2024-04-20T17:32:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:31 smithi005 ceph-mon[39562]: 5.e scrub ok 2024-04-20T17:32:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:31 smithi106 ceph-mon[44423]: 5.3 scrub starts 2024-04-20T17:32:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:31 smithi106 ceph-mon[44423]: pgmap v1584: 17 pgs: 17 active+clean; 2.4 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-20T17:32:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:31 smithi106 ceph-mon[44423]: 5.6 scrub starts 2024-04-20T17:32:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:31 smithi106 ceph-mon[44423]: 5.1 scrub ok 2024-04-20T17:32:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:31 smithi106 ceph-mon[44423]: 5.3 scrub ok 2024-04-20T17:32:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:31 smithi106 ceph-mon[44423]: 5.e scrub starts 2024-04-20T17:32:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:31 smithi106 ceph-mon[44423]: 5.6 scrub ok 2024-04-20T17:32:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:31 smithi106 ceph-mon[44423]: 5.e scrub ok 2024-04-20T17:32:32.376 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:32:32.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:32.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:32:32.481 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:32:32.523 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:32.536 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-20T17:32:32.656 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:32:32.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:32.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:32:32.764 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:32:32.797 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:32.819 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:32:32.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:32.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:32:32.879 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:32:32.923 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:32:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:32 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:32:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:32 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:32:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:32 smithi005 ceph-mon[39562]: osdmap e603: 12 total, 12 up, 11 in 2024-04-20T17:32:32.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:32 smithi005 ceph-mon[39562]: 5.7 scrub starts 2024-04-20T17:32:32.981 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:33.161 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:32 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:32:33.161 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:32 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:32:33.161 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:32 smithi171 ceph-mon[44189]: osdmap e603: 12 total, 12 up, 11 in 2024-04-20T17:32:33.161 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:32 smithi171 ceph-mon[44189]: 5.7 scrub starts 2024-04-20T17:32:33.219 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:32:33.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:33.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:32:33.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:32 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:32:33.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:32 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:32:33.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:32 smithi106 ceph-mon[44423]: osdmap e603: 12 total, 12 up, 11 in 2024-04-20T17:32:33.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:32 smithi106 ceph-mon[44423]: 5.7 scrub starts 2024-04-20T17:32:33.323 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:32:33.354 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:33.382 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:32:33.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:33.387 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:32:33.488 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:32:33.522 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:33.740 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:32:33.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:33.744 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:32:33.845 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:32:33.872 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:33.902 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:32:33.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:33.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:32:33.930 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:32:33.952 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:32:34.005 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:32:34.046 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:33 smithi005 ceph-mon[39562]: pgmap v1586: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 2.2 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-20T17:32:34.046 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:33 smithi005 ceph-mon[39562]: 5.7 scrub ok 2024-04-20T17:32:34.047 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:33 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:32:34.047 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:33 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:32:34.047 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:33 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2489956433' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:32:34.051 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:33 smithi171 ceph-mon[44189]: pgmap v1586: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 2.2 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-20T17:32:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:33 smithi171 ceph-mon[44189]: 5.7 scrub ok 2024-04-20T17:32:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:33 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:32:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:33 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:32:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:33 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2489956433' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:32:34.094 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:32:34.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:34.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:32:34.203 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:32:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:33 smithi106 ceph-mon[44423]: pgmap v1586: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 2.2 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-20T17:32:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:33 smithi106 ceph-mon[44423]: 5.7 scrub ok 2024-04-20T17:32:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:33 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:32:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:33 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:32:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:33 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2489956433' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:32:34.265 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:34.832 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:32:34.832 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:32:34.938 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:32:34.938 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:32:34.938 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:32:34.938 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:32:34.951 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:32:34.951 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:32:34.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:34.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:32:34.991 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:32:35.026 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:32:35.056 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:32:35.063 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:32:35.097 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:35.103 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:34 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:32:35.103 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:34 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:32:35.103 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:34 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2489956433' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:32:35.103 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:34 smithi005 ceph-mon[39562]: osdmap e604: 12 total, 12 up, 11 in 2024-04-20T17:32:35.103 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:34 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1834043550' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:32:35.104 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:32:35.144 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:32:35.178 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:32:35.188 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:34 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:32:35.188 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:34 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:32:35.188 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:34 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2489956433' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:32:35.189 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:34 smithi171 ceph-mon[44189]: osdmap e604: 12 total, 12 up, 11 in 2024-04-20T17:32:35.189 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:34 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1834043550' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:32:35.218 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:32:35.221 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:32:35.221 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:32:35.221 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:32:35.221 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:32:35.221 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:32:35.221 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:32:35.221 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:32:35.221 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:32:35.221 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:32:35.221 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:32:35.221 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:32:35.222 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:32:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:34 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:32:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:34 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:32:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:34 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2489956433' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:32:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:34 smithi106 ceph-mon[44423]: osdmap e604: 12 total, 12 up, 11 in 2024-04-20T17:32:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:34 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1834043550' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:32:36.190 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:35 smithi005 ceph-mon[39562]: pgmap v1588: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 2.1 GiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-20T17:32:36.190 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.190 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.190 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.190 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.190 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.190 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.190 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.190 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.190 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:35 smithi005 ceph-mon[39562]: 5.d scrub starts 2024-04-20T17:32:36.190 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.191 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.191 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.191 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:35 smithi005 ceph-mon[39562]: 5.d scrub ok 2024-04-20T17:32:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:35 smithi106 ceph-mon[44423]: pgmap v1588: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 2.1 GiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-20T17:32:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:35 smithi106 ceph-mon[44423]: 5.d scrub starts 2024-04-20T17:32:36.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:35 smithi106 ceph-mon[44423]: 5.d scrub ok 2024-04-20T17:32:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:35 smithi171 ceph-mon[44189]: pgmap v1588: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 2.1 GiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-20T17:32:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:35 smithi171 ceph-mon[44189]: 5.d scrub starts 2024-04-20T17:32:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:36.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:35 smithi171 ceph-mon[44189]: 5.d scrub ok 2024-04-20T17:32:37.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:37.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:36 smithi005 ceph-mon[39562]: 5.4 deep-scrub starts 2024-04-20T17:32:37.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:36 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:32:37.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:36 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:32:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:36 smithi106 ceph-mon[44423]: 5.4 deep-scrub starts 2024-04-20T17:32:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:36 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:32:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:36 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:32:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:36 smithi171 ceph-mon[44189]: 5.4 deep-scrub starts 2024-04-20T17:32:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:36 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:32:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:36 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:32:37.910 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:37 smithi005 ceph-mon[39562]: pgmap v1589: 17 pgs: 17 active+clean; 1.9 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.31k op/s 2024-04-20T17:32:37.910 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:37 smithi005 ceph-mon[39562]: 5.a deep-scrub starts 2024-04-20T17:32:37.911 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:37 smithi005 ceph-mon[39562]: 5.b scrub starts 2024-04-20T17:32:37.911 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:37 smithi005 ceph-mon[39562]: 5.8 scrub starts 2024-04-20T17:32:37.911 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:37 smithi005 ceph-mon[39562]: osdmap e605: 12 total, 12 up, 11 in 2024-04-20T17:32:37.911 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:37 smithi005 ceph-mon[39562]: 5.8 scrub ok 2024-04-20T17:32:38.161 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:32:38.165 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:32:38.217 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:32:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:37 smithi106 ceph-mon[44423]: pgmap v1589: 17 pgs: 17 active+clean; 1.9 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.31k op/s 2024-04-20T17:32:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:37 smithi106 ceph-mon[44423]: 5.a deep-scrub starts 2024-04-20T17:32:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:37 smithi106 ceph-mon[44423]: 5.b scrub starts 2024-04-20T17:32:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:37 smithi106 ceph-mon[44423]: 5.8 scrub starts 2024-04-20T17:32:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:37 smithi106 ceph-mon[44423]: osdmap e605: 12 total, 12 up, 11 in 2024-04-20T17:32:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:37 smithi106 ceph-mon[44423]: 5.8 scrub ok 2024-04-20T17:32:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:37 smithi171 ceph-mon[44189]: pgmap v1589: 17 pgs: 17 active+clean; 1.9 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.31k op/s 2024-04-20T17:32:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:37 smithi171 ceph-mon[44189]: 5.a deep-scrub starts 2024-04-20T17:32:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:37 smithi171 ceph-mon[44189]: 5.b scrub starts 2024-04-20T17:32:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:37 smithi171 ceph-mon[44189]: 5.8 scrub starts 2024-04-20T17:32:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:37 smithi171 ceph-mon[44189]: osdmap e605: 12 total, 12 up, 11 in 2024-04-20T17:32:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:37 smithi171 ceph-mon[44189]: 5.8 scrub ok 2024-04-20T17:32:38.785 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:32:38.918 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:38 smithi005 ceph-mon[39562]: 5.b scrub ok 2024-04-20T17:32:38.918 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:38 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:32:38.918 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:38 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:32:38.918 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:38 smithi005 ceph-mon[39562]: from='client.26792 v1:172.21.15.5:0/697018174' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:32:38.918 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:38 smithi005 ceph-mon[39562]: osdmap e606: 12 total, 12 up, 11 in 2024-04-20T17:32:39.117 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":1590,"stamp":"2024-04-20T17:32:37.323611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2022703672,"num_objects":30858,"num_object_clones":0,"num_object_copies":61716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30858,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":232189,"num_write_kb":8414523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160760,"ondisk_log_size":160760,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":7493652,"kb_used_data":4008352,"kb_used_omap":225,"kb_used_meta":3484894,"kb_avail":1023703020,"statfs":{"total":1055945392128,"available":1048271892480,"internally_reserved":0,"allocated":4104552448,"data_stored":4100538441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":231348,"internal_metadata":3568531532},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-569901056,"num_objects":-8696,"num_object_clones":0,"num_object_copies":-17392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8696,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.967131"},"pg_stats":[{"pgid":"5.f","version":"604'14528","reported_seq":14759,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-04-20T17:32:35.854991+0000","last_change":"2024-04-20T17:32:29.834668+0000","last_active":"2024-04-20T17:32:35.854991+0000","last_peered":"2024-04-20T17:32:35.854991+0000","last_clean":"2024-04-20T17:32:35.854991+0000","last_became_active":"2024-04-20T17:29:29.256875+0000","last_became_peered":"2024-04-20T17:29:29.256875+0000","last_unstale":"2024-04-20T17:32:35.854991+0000","last_undegraded":"2024-04-20T17:32:35.854991+0000","last_fullsized":"2024-04-20T17:32:35.854991+0000","mapping_epoch":543,"log_start":"571'4500","ondisk_log_start":"571'4500","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'14088","last_scrub_stamp":"2024-04-20T17:32:29.834573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:32:29.834573+0000","objects_scrubbed":2321,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:33:31.351043+0000","scrub_duration":354,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122552320,"num_objects":1870,"num_object_clones":0,"num_object_copies":3740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14528,"num_write_kb":524736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"604'14338","reported_seq":14567,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-04-20T17:32:36.150744+0000","last_change":"2024-04-20T17:32:31.358348+0000","last_active":"2024-04-20T17:32:36.150744+0000","last_peered":"2024-04-20T17:32:36.150744+0000","last_clean":"2024-04-20T17:32:36.150744+0000","last_became_active":"2024-04-20T17:29:29.326722+0000","last_became_peered":"2024-04-20T17:29:29.326722+0000","last_unstale":"2024-04-20T17:32:36.150744+0000","last_undegraded":"2024-04-20T17:32:36.150744+0000","last_fullsized":"2024-04-20T17:32:36.150744+0000","mapping_epoch":543,"log_start":"571'4300","ondisk_log_start":"571'4300","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'14003","last_scrub_stamp":"2024-04-20T17:32:31.358187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:32:31.358187+0000","objects_scrubbed":2220,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:33:47.525578+0000","scrub_duration":320,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122945536,"num_objects":1876,"num_object_clones":0,"num_object_copies":3752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14338,"num_write_kb":518848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"603'14463","reported_seq":14700,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-04-20T17:32:33.778381+0000","last_change":"2024-04-20T17:32:06.687923+0000","last_active":"2024-04-20T17:32:33.778381+0000","last_peered":"2024-04-20T17:32:33.778381+0000","last_clean":"2024-04-20T17:32:33.778381+0000","last_became_active":"2024-04-20T17:29:29.257014+0000","last_became_peered":"2024-04-20T17:29:29.257014+0000","last_unstale":"2024-04-20T17:32:33.778381+0000","last_undegraded":"2024-04-20T17:32:33.778381+0000","last_fullsized":"2024-04-20T17:32:33.778381+0000","mapping_epoch":543,"log_start":"571'4400","ondisk_log_start":"571'4400","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6222","last_scrub_stamp":"2024-04-20T17:30:48.196236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:30:48.196236+0000","objects_scrubbed":3399,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":936,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131858456,"num_objects":2013,"num_object_clones":0,"num_object_copies":4026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2013,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14463,"num_write_kb":527169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"604'14622","reported_seq":14858,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-04-20T17:32:35.861142+0000","last_change":"2024-04-20T17:32:27.917596+0000","last_active":"2024-04-20T17:32:35.861142+0000","last_peered":"2024-04-20T17:32:35.861142+0000","last_clean":"2024-04-20T17:32:35.861142+0000","last_became_active":"2024-04-20T17:29:29.597181+0000","last_became_peered":"2024-04-20T17:29:29.597181+0000","last_unstale":"2024-04-20T17:32:35.861142+0000","last_undegraded":"2024-04-20T17:32:35.861142+0000","last_fullsized":"2024-04-20T17:32:35.861142+0000","mapping_epoch":543,"log_start":"571'4600","ondisk_log_start":"571'4600","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'14052","last_scrub_stamp":"2024-04-20T17:32:27.917451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:32:27.917451+0000","objects_scrubbed":2462,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:33:53.258599+0000","scrub_duration":432,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122945536,"num_objects":1876,"num_object_clones":0,"num_object_copies":3752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14622,"num_write_kb":527936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"604'14701","reported_seq":14905,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-04-20T17:32:36.197236+0000","last_change":"2024-04-20T17:30:52.425416+0000","last_active":"2024-04-20T17:32:36.197236+0000","last_peered":"2024-04-20T17:32:36.197236+0000","last_clean":"2024-04-20T17:32:36.197236+0000","last_became_active":"2024-04-20T17:29:29.258913+0000","last_became_peered":"2024-04-20T17:29:29.258913+0000","last_unstale":"2024-04-20T17:32:36.197236+0000","last_undegraded":"2024-04-20T17:32:36.197236+0000","last_fullsized":"2024-04-20T17:32:36.197236+0000","mapping_epoch":543,"log_start":"571'4700","ondisk_log_start":"571'4700","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"579'6639","last_scrub_stamp":"2024-04-20T17:30:52.425214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:30:52.425214+0000","objects_scrubbed":6583,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1314,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123011072,"num_objects":1877,"num_object_clones":0,"num_object_copies":3754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14701,"num_write_kb":530496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"604'14462","reported_seq":14670,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-04-20T17:32:36.370128+0000","last_change":"2024-04-20T17:30:58.969305+0000","last_active":"2024-04-20T17:32:36.370128+0000","last_peered":"2024-04-20T17:32:36.370128+0000","last_clean":"2024-04-20T17:32:36.370128+0000","last_became_active":"2024-04-20T17:29:29.326880+0000","last_became_peered":"2024-04-20T17:29:29.326880+0000","last_unstale":"2024-04-20T17:32:36.370128+0000","last_undegraded":"2024-04-20T17:32:36.370128+0000","last_fullsized":"2024-04-20T17:32:36.370128+0000","mapping_epoch":543,"log_start":"571'4400","ondisk_log_start":"571'4400","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"580'7147","last_scrub_stamp":"2024-04-20T17:30:58.969065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:30:58.969065+0000","objects_scrubbed":7078,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1547,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122421248,"num_objects":1868,"num_object_clones":0,"num_object_copies":3736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14462,"num_write_kb":522560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"603'14116","reported_seq":14365,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-04-20T17:32:32.456801+0000","last_change":"2024-04-20T17:31:56.425171+0000","last_active":"2024-04-20T17:32:32.456801+0000","last_peered":"2024-04-20T17:32:32.456801+0000","last_clean":"2024-04-20T17:32:32.456801+0000","last_became_active":"2024-04-20T17:29:29.261848+0000","last_became_peered":"2024-04-20T17:29:29.261848+0000","last_unstale":"2024-04-20T17:32:32.456801+0000","last_undegraded":"2024-04-20T17:32:32.456801+0000","last_fullsized":"2024-04-20T17:32:32.456801+0000","mapping_epoch":543,"log_start":"570'4100","ondisk_log_start":"570'4100","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"597'11709","last_scrub_stamp":"2024-04-20T17:31:56.425068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:31:56.425068+0000","objects_scrubbed":4425,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:33:05.354947+0000","scrub_duration":716,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130940928,"num_objects":1998,"num_object_clones":0,"num_object_copies":3996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14116,"num_write_kb":515648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"603'14504","reported_seq":14701,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-04-20T17:32:33.759868+0000","last_change":"2024-04-20T17:30:47.337665+0000","last_active":"2024-04-20T17:32:33.759868+0000","last_peered":"2024-04-20T17:32:33.759868+0000","last_clean":"2024-04-20T17:32:33.759868+0000","last_became_active":"2024-04-20T17:29:29.255991+0000","last_became_peered":"2024-04-20T17:29:29.255991+0000","last_unstale":"2024-04-20T17:32:33.759868+0000","last_undegraded":"2024-04-20T17:32:33.759868+0000","last_fullsized":"2024-04-20T17:32:33.759868+0000","mapping_epoch":543,"log_start":"571'4500","ondisk_log_start":"571'4500","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6174","last_scrub_stamp":"2024-04-20T17:30:47.337557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:30:47.337557+0000","objects_scrubbed":6125,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131727360,"num_objects":2010,"num_object_clones":0,"num_object_copies":4020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14504,"num_write_kb":528448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"604'14731","reported_seq":14964,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-04-20T17:32:35.855871+0000","last_change":"2024-04-20T17:32:32.839424+0000","last_active":"2024-04-20T17:32:35.855871+0000","last_peered":"2024-04-20T17:32:35.855871+0000","last_clean":"2024-04-20T17:32:35.855871+0000","last_became_active":"2024-04-20T17:29:29.597100+0000","last_became_peered":"2024-04-20T17:29:29.597100+0000","last_unstale":"2024-04-20T17:32:35.855871+0000","last_undegraded":"2024-04-20T17:32:35.855871+0000","last_fullsized":"2024-04-20T17:32:35.855871+0000","mapping_epoch":543,"log_start":"571'4700","ondisk_log_start":"571'4700","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"603'14522","last_scrub_stamp":"2024-04-20T17:32:32.839241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:32:32.839241+0000","objects_scrubbed":2040,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:33:38.680637+0000","scrub_duration":327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119734272,"num_objects":1827,"num_object_clones":0,"num_object_copies":3654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14731,"num_write_kb":529856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"604'17851","reported_seq":18215,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-04-20T17:32:36.339317+0000","last_change":"2024-04-20T17:32:31.325556+0000","last_active":"2024-04-20T17:32:36.339317+0000","last_peered":"2024-04-20T17:32:36.339317+0000","last_clean":"2024-04-20T17:32:36.339317+0000","last_became_active":"2024-04-20T17:29:29.256893+0000","last_became_peered":"2024-04-20T17:29:29.256893+0000","last_unstale":"2024-04-20T17:32:36.339317+0000","last_undegraded":"2024-04-20T17:32:36.339317+0000","last_fullsized":"2024-04-20T17:32:36.339317+0000","mapping_epoch":543,"log_start":"580'7801","ondisk_log_start":"580'7801","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'17462","last_scrub_stamp":"2024-04-20T17:32:31.325390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:32:31.325390+0000","objects_scrubbed":2263,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:33:34.941208+0000","scrub_duration":566,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121831424,"num_objects":1859,"num_object_clones":0,"num_object_copies":3718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14735,"num_write_kb":531008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"604'14498","reported_seq":14744,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-04-20T17:32:34.295151+0000","last_change":"2024-04-20T17:32:29.992977+0000","last_active":"2024-04-20T17:32:34.295151+0000","last_peered":"2024-04-20T17:32:34.295151+0000","last_clean":"2024-04-20T17:32:34.295151+0000","last_became_active":"2024-04-20T17:29:29.597164+0000","last_became_peered":"2024-04-20T17:29:29.597164+0000","last_unstale":"2024-04-20T17:32:34.295151+0000","last_undegraded":"2024-04-20T17:32:34.295151+0000","last_fullsized":"2024-04-20T17:32:34.295151+0000","mapping_epoch":543,"log_start":"571'4400","ondisk_log_start":"571'4400","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'14168","last_scrub_stamp":"2024-04-20T17:32:29.992695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:32:29.992695+0000","objects_scrubbed":2420,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:33:30.001427+0000","scrub_duration":649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135266304,"num_objects":2064,"num_object_clones":0,"num_object_copies":4128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14498,"num_write_kb":529984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"604'14352","reported_seq":14593,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-04-20T17:32:34.296375+0000","last_change":"2024-04-20T17:32:31.009734+0000","last_active":"2024-04-20T17:32:34.296375+0000","last_peered":"2024-04-20T17:32:34.296375+0000","last_clean":"2024-04-20T17:32:34.296375+0000","last_became_active":"2024-04-20T17:29:29.256975+0000","last_became_peered":"2024-04-20T17:29:29.256975+0000","last_unstale":"2024-04-20T17:32:34.296375+0000","last_undegraded":"2024-04-20T17:32:34.296375+0000","last_fullsized":"2024-04-20T17:32:34.296375+0000","mapping_epoch":543,"log_start":"571'4300","ondisk_log_start":"571'4300","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'14068","last_scrub_stamp":"2024-04-20T17:32:31.009512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:32:31.009512+0000","objects_scrubbed":2297,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:33:37.681947+0000","scrub_duration":662,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130285568,"num_objects":1988,"num_object_clones":0,"num_object_copies":3976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14352,"num_write_kb":522880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"604'14501","reported_seq":14742,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-04-20T17:32:34.332148+0000","last_change":"2024-04-20T17:32:28.059569+0000","last_active":"2024-04-20T17:32:34.332148+0000","last_peered":"2024-04-20T17:32:34.332148+0000","last_clean":"2024-04-20T17:32:34.332148+0000","last_became_active":"2024-04-20T17:29:29.263022+0000","last_became_peered":"2024-04-20T17:29:29.263022+0000","last_unstale":"2024-04-20T17:32:34.332148+0000","last_undegraded":"2024-04-20T17:32:34.332148+0000","last_fullsized":"2024-04-20T17:32:34.332148+0000","mapping_epoch":543,"log_start":"571'4500","ondisk_log_start":"571'4500","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'14040","last_scrub_stamp":"2024-04-20T17:32:28.059229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:32:28.059229+0000","objects_scrubbed":2434,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:33:28.159376+0000","scrub_duration":694,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126943232,"num_objects":1937,"num_object_clones":0,"num_object_copies":3874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14501,"num_write_kb":526016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"604'14531","reported_seq":14767,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-04-20T17:32:35.858284+0000","last_change":"2024-04-20T17:32:30.907332+0000","last_active":"2024-04-20T17:32:35.858284+0000","last_peered":"2024-04-20T17:32:35.858284+0000","last_clean":"2024-04-20T17:32:35.858284+0000","last_became_active":"2024-04-20T17:30:10.553600+0000","last_became_peered":"2024-04-20T17:30:10.553600+0000","last_unstale":"2024-04-20T17:32:35.858284+0000","last_undegraded":"2024-04-20T17:32:35.858284+0000","last_fullsized":"2024-04-20T17:32:35.858284+0000","mapping_epoch":561,"log_start":"571'4500","ondisk_log_start":"571'4500","created":543,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'14176","last_scrub_stamp":"2024-04-20T17:32:30.907250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:32:30.907250+0000","objects_scrubbed":2253,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:34:00.001201+0000","scrub_duration":400,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123666432,"num_objects":1887,"num_object_clones":0,"num_object_copies":3774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14531,"num_write_kb":525376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"604'14472","reported_seq":14685,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-04-20T17:32:34.317630+0000","last_change":"2024-04-20T17:31:02.423156+0000","last_active":"2024-04-20T17:32:34.317630+0000","last_peered":"2024-04-20T17:32:34.317630+0000","last_clean":"2024-04-20T17:32:34.317630+0000","last_became_active":"2024-04-20T17:29:29.597475+0000","last_became_peered":"2024-04-20T17:29:29.597475+0000","last_unstale":"2024-04-20T17:32:34.317630+0000","last_undegraded":"2024-04-20T17:32:34.317630+0000","last_fullsized":"2024-04-20T17:32:34.317630+0000","mapping_epoch":543,"log_start":"571'4400","ondisk_log_start":"571'4400","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"580'7523","last_scrub_stamp":"2024-04-20T17:31:02.423013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:31:02.423013+0000","objects_scrubbed":7462,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126615552,"num_objects":1932,"num_object_clones":0,"num_object_copies":3864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14472,"num_write_kb":524928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"604'14454","reported_seq":14665,"reported_epoch":604,"state":"active+clean","last_fresh":"2024-04-20T17:32:34.267412+0000","last_change":"2024-04-20T17:31:01.490383+0000","last_active":"2024-04-20T17:32:34.267412+0000","last_peered":"2024-04-20T17:32:34.267412+0000","last_clean":"2024-04-20T17:32:34.267412+0000","last_became_active":"2024-04-20T17:29:29.263290+0000","last_became_peered":"2024-04-20T17:29:29.263290+0000","last_unstale":"2024-04-20T17:32:34.267412+0000","last_undegraded":"2024-04-20T17:32:34.267412+0000","last_fullsized":"2024-04-20T17:32:34.267412+0000","mapping_epoch":543,"log_start":"571'4400","ondisk_log_start":"571'4400","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"580'7467","last_scrub_stamp":"2024-04-20T17:31:01.490094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:31:01.490094+0000","objects_scrubbed":7387,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1450,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129368064,"num_objects":1974,"num_object_clones":0,"num_object_copies":3948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14454,"num_write_kb":525696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":758,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-04-20T17:32:31.901250+0000","last_change":"2024-04-20T17:31:54.780064+0000","last_active":"2024-04-20T17:32:31.901250+0000","last_peered":"2024-04-20T17:32:31.901250+0000","last_clean":"2024-04-20T17:32:31.901250+0000","last_became_active":"2024-04-20T17:23:24.109202+0000","last_became_peered":"2024-04-20T17:23:24.109202+0000","last_unstale":"2024-04-20T17:32:31.901250+0000","last_undegraded":"2024-04-20T17:32:31.901250+0000","last_fullsized":"2024-04-20T17:32:31.901250+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:31:54.779921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:31:54.779921+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:33:15.516753+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":2022113304,"num_objects":30856,"num_object_clones":0,"num_object_copies":61712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30856,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":232008,"num_write_kb":8411585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4025425920,"data_stored":4025417776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160623,"ondisk_log_size":160623,"up":32,"acting":32,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991555,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":943764,"kb_used_data":389196,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92801388,"statfs":{"total":95995035648,"available":95028621312,"internally_reserved":0,"allocated":398536704,"data_stored":398168815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":567867844},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253194,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":803164,"kb_used_data":379020,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":92941988,"statfs":{"total":95995035648,"available":95172595712,"internally_reserved":0,"allocated":388116480,"data_stored":387750211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":434239936},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612943,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":615192,"kb_used_data":241100,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":93129960,"statfs":{"total":95995035648,"available":95365079040,"internally_reserved":0,"allocated":246886400,"data_stored":246520151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":383056320},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809173,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":490508,"kb_used_data":270528,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93254644,"statfs":{"total":95995035648,"available":95492755456,"internally_reserved":0,"allocated":277020672,"data_stored":276660427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":398,"seq":1709396983929,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":296772,"kb_used_data":132156,"kb_used_omap":6,"kb_used_meta":164601,"kb_avail":93448380,"statfs":{"total":95995035648,"available":95691141120,"internally_reserved":0,"allocated":135327744,"data_stored":134969951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6767,"internal_metadata":168551825},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":2297807503404,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575223,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":951636,"kb_used_data":478988,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":92793516,"statfs":{"total":95995035648,"available":95020560384,"internally_reserved":0,"allocated":490483712,"data_stored":490117463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738747,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1064536,"kb_used_data":640460,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":92680616,"statfs":{"total":95995035648,"available":94904950784,"internally_reserved":0,"allocated":655831040,"data_stored":655464791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394038,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":473432,"kb_used_data":365324,"kb_used_omap":202,"kb_used_meta":107893,"kb_avail":93271720,"statfs":{"total":95995035648,"available":95510241280,"internally_reserved":0,"allocated":374091776,"data_stored":373725527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":110482903},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653086,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":190540,"kb_used_data":135936,"kb_used_omap":4,"kb_used_meta":54587,"kb_avail":93554612,"statfs":{"total":95995035648,"available":95799922688,"internally_reserved":0,"allocated":139198464,"data_stored":138838219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":55897799},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986535,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":674772,"kb_used_data":368716,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":93070380,"statfs":{"total":95995035648,"available":95304069120,"internally_reserved":0,"allocated":377565184,"data_stored":377198935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790306,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":989336,"kb_used_data":606928,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92755816,"statfs":{"total":95995035648,"available":94981955584,"internally_reserved":0,"allocated":621494272,"data_stored":621123951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":654114816,"data_stored":654114816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":473235456,"data_stored":473235456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":363397120,"data_stored":363397120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":135790592,"data_stored":135790592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":369098752,"data_stored":369098752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":612569088,"data_stored":612565016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":130940928,"data_stored":130940928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":268894208,"data_stored":268894208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":245956608,"data_stored":245956608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":383057920,"data_stored":383057920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":388370432,"data_stored":388366360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:32:39.118 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:32:39.240 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:32:39.252 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:38 smithi171 ceph-mon[44189]: 5.b scrub ok 2024-04-20T17:32:39.253 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:38 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:32:39.253 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:38 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:32:39.253 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:38 smithi171 ceph-mon[44189]: from='client.26792 v1:172.21.15.5:0/697018174' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:32:39.253 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:38 smithi171 ceph-mon[44189]: osdmap e606: 12 total, 12 up, 11 in 2024-04-20T17:32:39.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:38 smithi106 ceph-mon[44423]: 5.b scrub ok 2024-04-20T17:32:39.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:38 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:32:39.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:38 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:32:39.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:38 smithi106 ceph-mon[44423]: from='client.26792 v1:172.21.15.5:0/697018174' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:32:39.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:38 smithi106 ceph-mon[44423]: osdmap e606: 12 total, 12 up, 11 in 2024-04-20T17:32:39.289 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-20T17:32:39.309 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:32:39.329 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:32:39.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:39.334 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:32:39.435 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:32:39.471 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:39.494 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:32:39.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:39.499 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:32:39.600 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:32:39.668 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:39.686 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:32:39.803 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:32:39.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:39.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:32:39.909 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:32:39.952 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:40.036 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:32:40.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:40.041 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:32:40.142 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:32:40.169 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:39 smithi106 ceph-mon[44423]: pgmap v1592: 17 pgs: 17 active+clean; 1.9 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-20T17:32:40.170 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:40.190 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:32:40.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:40.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:32:40.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:39 smithi005 ceph-mon[39562]: pgmap v1592: 17 pgs: 17 active+clean; 1.9 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-20T17:32:40.270 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:39 smithi171 ceph-mon[44189]: pgmap v1592: 17 pgs: 17 active+clean; 1.9 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-20T17:32:40.298 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:32:40.339 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:40.594 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:32:40.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:40.599 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:32:40.700 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:32:40.742 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:40.760 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:32:40.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:40.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:32:40.864 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:32:40.897 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:41.161 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:32:41.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:41.164 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:32:41.265 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:32:41.304 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:41.334 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:32:41.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:41.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:32:41.444 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:32:41.491 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:41 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:32:41.492 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:41 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:32:41.492 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:41 smithi171 ceph-mon[44189]: osdmap e607: 12 total, 12 up, 11 in 2024-04-20T17:32:41.492 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:41 smithi171 ceph-mon[44189]: pgmap v1594: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.7 GiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 884 op/s 2024-04-20T17:32:41.496 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:41.532 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:32:41.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:41.538 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:32:41.639 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:32:41.688 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:41 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:32:41.688 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:41 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:32:41.689 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:41 smithi005 ceph-mon[39562]: osdmap e607: 12 total, 12 up, 11 in 2024-04-20T17:32:41.689 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:41 smithi005 ceph-mon[39562]: pgmap v1594: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.7 GiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 884 op/s 2024-04-20T17:32:41.698 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:41.738 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:32:41.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:41.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:32:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:41 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:32:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:41 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:32:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:41 smithi106 ceph-mon[44423]: osdmap e607: 12 total, 12 up, 11 in 2024-04-20T17:32:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:41 smithi106 ceph-mon[44423]: pgmap v1594: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.7 GiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 884 op/s 2024-04-20T17:32:41.764 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:32:41.831 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:32:41.831 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:32:41.832 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:32:41.832 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:32:41.832 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.832 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:159061 4.2 4:4cdccfcc:::benchmark_data_smithi171_122772_object159060:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e482)", 2024-04-20T17:32:41.832 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:08.824368+0000", 2024-04-20T17:32:41.832 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 393.00395147400002, 2024-04-20T17:32:41.832 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15598972799999999, 2024-04-20T17:32:41.832 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:32:41.832 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:32:41.832 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:41.832 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:32:41.832 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:32:41.832 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:32:41.832 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 159061 2024-04-20T17:32:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:32:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:32:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824368+0000", 2024-04-20T17:32:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:32:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824368+0000", 2024-04-20T17:32:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:32:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824371+0000", 2024-04-20T17:32:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.424e-06 2024-04-20T17:32:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:32:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824398+0000", 2024-04-20T17:32:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7310000000000001e-05 2024-04-20T17:32:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:32:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824408+0000", 2024-04-20T17:32:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0295999999999999e-05 2024-04-20T17:32:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.835 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:32:41.835 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824414+0000", 2024-04-20T17:32:41.835 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9989999999999997e-06 2024-04-20T17:32:41.835 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.835 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.835 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:32:41.835 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.827811+0000", 2024-04-20T17:32:41.835 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0033968319999999998 2024-04-20T17:32:41.835 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.835 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.835 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:32:41.835 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.827940+0000", 2024-04-20T17:32:41.835 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012889600000000001 2024-04-20T17:32:41.836 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.836 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.836 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:32:41.836 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.828088+0000", 2024-04-20T17:32:41.836 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014852100000000001 2024-04-20T17:32:41.836 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.836 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.836 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:32:41.836 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.836604+0000", 2024-04-20T17:32:41.836 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0085154889999999994 2024-04-20T17:32:41.836 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.836 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.836 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:41.837 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980241+0000", 2024-04-20T17:32:41.837 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.14363725299999999 2024-04-20T17:32:41.837 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.837 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.837 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:32:41.837 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980298+0000", 2024-04-20T17:32:41.837 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6462999999999998e-05 2024-04-20T17:32:41.837 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.837 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.837 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:32:41.837 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980358+0000", 2024-04-20T17:32:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0244999999999997e-05 2024-04-20T17:32:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:32:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:159066 4.2 4:41a9bca0:::benchmark_data_smithi171_122772_object159065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e482)", 2024-04-20T17:32:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:08.824564+0000", 2024-04-20T17:32:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 393.003755705, 2024-04-20T17:32:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.155931294, 2024-04-20T17:32:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:32:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:32:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:41.839 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:32:41.839 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:32:41.839 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:32:41.839 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 159066 2024-04-20T17:32:41.839 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.839 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:32:41.839 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.839 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:32:41.839 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824564+0000", 2024-04-20T17:32:41.839 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.839 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.839 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.839 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:32:41.839 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824564+0000", 2024-04-20T17:32:41.839 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.840 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.840 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.840 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:32:41.840 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824565+0000", 2024-04-20T17:32:41.840 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1510000000000001e-06 2024-04-20T17:32:41.840 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.840 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.840 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:32:41.840 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824583+0000", 2024-04-20T17:32:41.840 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7839e-05 2024-04-20T17:32:41.840 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.840 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.840 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:32:41.840 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824592+0000", 2024-04-20T17:32:41.840 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9290000000000004e-06 2024-04-20T17:32:41.841 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.841 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.841 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:32:41.841 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824596+0000", 2024-04-20T17:32:41.841 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3590000000000001e-06 2024-04-20T17:32:41.841 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.841 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.841 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:32:41.842 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.832585+0000", 2024-04-20T17:32:41.842 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0079889399999999999 2024-04-20T17:32:41.842 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.842 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.842 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:32:41.842 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.832687+0000", 2024-04-20T17:32:41.842 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000101487 2024-04-20T17:32:41.842 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.842 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.842 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:32:41.842 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.832792+0000", 2024-04-20T17:32:41.842 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010508100000000001 2024-04-20T17:32:41.842 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.842 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.843 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:32:41.843 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980174+0000", 2024-04-20T17:32:41.843 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.14738206500000001 2024-04-20T17:32:41.843 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.843 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.843 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:41.843 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980405+0000", 2024-04-20T17:32:41.843 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00023092499999999999 2024-04-20T17:32:41.843 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.843 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.843 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:32:41.843 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980449+0000", 2024-04-20T17:32:41.844 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4351000000000003e-05 2024-04-20T17:32:41.844 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.844 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.844 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:32:41.844 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980495+0000", 2024-04-20T17:32:41.844 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6167e-05 2024-04-20T17:32:41.844 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.844 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:32:41.844 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.844 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.844 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.844 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop_reply(client.26074.0:159061 4.2 e482/445 ondisk, result = 0)", 2024-04-20T17:32:41.844 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:08.839536+0000", 2024-04-20T17:32:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 392.988783818, 2024-04-20T17:32:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.14083410099999999, 2024-04-20T17:32:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:32:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:32:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:32:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:32:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:32:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.839536+0000", 2024-04-20T17:32:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:32:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.839536+0000", 2024-04-20T17:32:41.846 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.846 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.846 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.846 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:32:41.846 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.839538+0000", 2024-04-20T17:32:41.846 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.951e-06 2024-04-20T17:32:41.846 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.846 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.846 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:32:41.846 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.839545+0000", 2024-04-20T17:32:41.846 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.114e-06 2024-04-20T17:32:41.846 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.846 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.847 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:32:41.847 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.887991+0000", 2024-04-20T17:32:41.847 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.048445941999999999 2024-04-20T17:32:41.847 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.847 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.847 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:32:41.847 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.888002+0000", 2024-04-20T17:32:41.847 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1425e-05 2024-04-20T17:32:41.847 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.847 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.847 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:32:41.847 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980224+0000", 2024-04-20T17:32:41.847 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.092222193999999993 2024-04-20T17:32:41.848 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.848 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.848 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:32:41.848 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980234+0000", 2024-04-20T17:32:41.848 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1099999999999992e-06 2024-04-20T17:32:41.848 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.848 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.848 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:32:41.848 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980370+0000", 2024-04-20T17:32:41.848 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000136365 2024-04-20T17:32:41.849 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.849 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:32:41.849 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.849 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.849 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.849 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:159675 4.2 4:45a9239e:::benchmark_data_smithi171_122772_object159674:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e482)", 2024-04-20T17:32:41.849 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:09.405527+0000", 2024-04-20T17:32:41.849 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 392.42279274399999, 2024-04-20T17:32:41.849 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.162906468, 2024-04-20T17:32:41.849 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:32:41.849 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:32:41.849 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:41.850 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:32:41.850 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:32:41.850 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:32:41.850 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 159675 2024-04-20T17:32:41.850 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.850 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:32:41.850 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.850 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:32:41.850 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.405527+0000", 2024-04-20T17:32:41.851 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.851 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.851 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.851 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:32:41.851 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.405527+0000", 2024-04-20T17:32:41.851 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.851 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.851 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.851 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:32:41.851 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.405528+0000", 2024-04-20T17:32:41.851 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.4e-07 2024-04-20T17:32:41.851 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.851 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.851 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:32:41.852 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.405546+0000", 2024-04-20T17:32:41.852 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8229000000000001e-05 2024-04-20T17:32:41.852 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.852 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.852 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:32:41.852 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.405553+0000", 2024-04-20T17:32:41.852 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8109999999999997e-06 2024-04-20T17:32:41.852 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.852 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.852 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:32:41.852 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.405559+0000", 2024-04-20T17:32:41.852 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1500000000000004e-06 2024-04-20T17:32:41.853 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.853 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.853 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:32:41.853 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.437169+0000", 2024-04-20T17:32:41.853 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.031610304999999998 2024-04-20T17:32:41.853 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.853 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.853 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:32:41.853 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.437261+0000", 2024-04-20T17:32:41.853 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1257999999999999e-05 2024-04-20T17:32:41.853 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.853 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.853 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:32:41.854 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.437390+0000", 2024-04-20T17:32:41.854 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000129777 2024-04-20T17:32:41.854 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.854 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.854 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:32:41.854 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.451123+0000", 2024-04-20T17:32:41.854 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013732496 2024-04-20T17:32:41.854 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.854 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.854 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:41.854 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.568336+0000", 2024-04-20T17:32:41.854 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.11721342899999999 2024-04-20T17:32:41.854 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.854 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.855 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:32:41.855 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.568379+0000", 2024-04-20T17:32:41.855 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2344e-05 2024-04-20T17:32:41.855 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.855 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.855 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:32:41.855 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.568433+0000", 2024-04-20T17:32:41.855 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4728999999999997e-05 2024-04-20T17:32:41.855 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.855 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:32:41.855 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.855 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.855 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.855 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:159676 4.2 4:43aa80d4:::benchmark_data_smithi171_122772_object159675:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e482)", 2024-04-20T17:32:41.856 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:09.407698+0000", 2024-04-20T17:32:41.856 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 392.42062166800002, 2024-04-20T17:32:41.856 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16090689399999999, 2024-04-20T17:32:41.856 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:32:41.856 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:32:41.856 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:41.856 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:32:41.856 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:32:41.856 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:32:41.856 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 159676 2024-04-20T17:32:41.856 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.856 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:32:41.856 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.856 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:32:41.856 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.407698+0000", 2024-04-20T17:32:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:32:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.407698+0000", 2024-04-20T17:32:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:32:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.407699+0000", 2024-04-20T17:32:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4309999999999999e-06 2024-04-20T17:32:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:32:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.407720+0000", 2024-04-20T17:32:41.858 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.09e-05 2024-04-20T17:32:41.858 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.858 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.858 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:32:41.858 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.407732+0000", 2024-04-20T17:32:41.858 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1494999999999999e-05 2024-04-20T17:32:41.858 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.858 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.858 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:32:41.858 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.407740+0000", 2024-04-20T17:32:41.858 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9459999999999998e-06 2024-04-20T17:32:41.858 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.858 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.858 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:32:41.859 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.441541+0000", 2024-04-20T17:32:41.859 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.033800864 2024-04-20T17:32:41.859 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.859 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.859 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:32:41.859 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.441652+0000", 2024-04-20T17:32:41.859 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000111749 2024-04-20T17:32:41.859 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.859 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.859 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:32:41.859 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.442325+0000", 2024-04-20T17:32:41.859 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00067265499999999998 2024-04-20T17:32:41.859 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.859 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.859 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:32:41.859 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.451395+0000", 2024-04-20T17:32:41.860 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0090699519999999992 2024-04-20T17:32:41.860 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.860 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.860 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:41.860 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.568492+0000", 2024-04-20T17:32:41.860 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.117097259 2024-04-20T17:32:41.860 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.860 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.860 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:32:41.860 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.568546+0000", 2024-04-20T17:32:41.860 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3780999999999999e-05 2024-04-20T17:32:41.860 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.860 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.860 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:32:41.861 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.568605+0000", 2024-04-20T17:32:41.861 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8862e-05 2024-04-20T17:32:41.861 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.861 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:32:41.861 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.861 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.861 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.861 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:159684 4.2 4:43ec9ad4:::benchmark_data_smithi171_122772_object159683:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e482)", 2024-04-20T17:32:41.861 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:09.415274+0000", 2024-04-20T17:32:41.861 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 392.41304602600002, 2024-04-20T17:32:41.861 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15358047, 2024-04-20T17:32:41.861 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:32:41.861 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:32:41.861 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:41.862 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:32:41.862 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:32:41.862 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:32:41.862 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 159684 2024-04-20T17:32:41.862 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.862 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:32:41.862 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.862 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:32:41.862 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.415274+0000", 2024-04-20T17:32:41.862 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.862 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.862 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.862 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:32:41.863 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.415274+0000", 2024-04-20T17:32:41.863 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.863 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.863 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.863 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:32:41.863 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.415276+0000", 2024-04-20T17:32:41.863 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.216e-06 2024-04-20T17:32:41.863 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.863 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.863 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:32:41.863 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.415300+0000", 2024-04-20T17:32:41.863 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4321e-05 2024-04-20T17:32:41.863 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.864 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.864 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:32:41.864 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.415312+0000", 2024-04-20T17:32:41.864 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1511e-05 2024-04-20T17:32:41.864 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.864 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.864 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:32:41.864 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.415319+0000", 2024-04-20T17:32:41.864 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8690000000000004e-06 2024-04-20T17:32:41.864 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.864 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.864 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:32:41.864 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.447198+0000", 2024-04-20T17:32:41.865 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.031878451000000002 2024-04-20T17:32:41.865 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.865 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.865 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:32:41.865 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.447351+0000", 2024-04-20T17:32:41.865 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015293300000000001 2024-04-20T17:32:41.865 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.865 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.865 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:32:41.865 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.447511+0000", 2024-04-20T17:32:41.865 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000159872 2024-04-20T17:32:41.865 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.865 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.865 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:32:41.866 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.451449+0000", 2024-04-20T17:32:41.866 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0039380539999999999 2024-04-20T17:32:41.866 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.866 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.866 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:41.866 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.568785+0000", 2024-04-20T17:32:41.866 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.11733666099999999 2024-04-20T17:32:41.866 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.866 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.866 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:32:41.866 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.568820+0000", 2024-04-20T17:32:41.866 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4586999999999999e-05 2024-04-20T17:32:41.867 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.867 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.867 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:32:41.867 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.568854+0000", 2024-04-20T17:32:41.867 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3995000000000002e-05 2024-04-20T17:32:41.867 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.867 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:32:41.867 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.867 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.867 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.867 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275200 4.6 4:65b8144d:::benchmark_data_smithi171_122772_object105008:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:32:41.867 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:10.998559+0000", 2024-04-20T17:32:41.867 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 270.829761056, 2024-04-20T17:32:41.867 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.21653923, 2024-04-20T17:32:41.868 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:32:41.868 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:32:41.868 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:41.868 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:32:41.868 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:32:41.868 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:32:41.868 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275200 2024-04-20T17:32:41.868 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.868 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:32:41.868 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.868 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:32:41.868 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:10.998559+0000", 2024-04-20T17:32:41.868 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.868 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.869 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.869 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:32:41.869 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:10.998559+0000", 2024-04-20T17:32:41.869 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.869 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.869 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.869 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:32:41.869 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:10.998560+0000", 2024-04-20T17:32:41.869 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9120000000000001e-06 2024-04-20T17:32:41.869 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.869 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.869 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:32:41.869 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:10.998566+0000", 2024-04-20T17:32:41.870 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6099999999999997e-06 2024-04-20T17:32:41.870 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.870 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.870 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:32:41.870 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:10.998578+0000", 2024-04-20T17:32:41.870 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1810999999999999e-05 2024-04-20T17:32:41.870 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.870 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.870 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:32:41.870 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:10.998586+0000", 2024-04-20T17:32:41.870 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.174e-06 2024-04-20T17:32:41.870 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.870 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.870 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:32:41.870 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:10.998644+0000", 2024-04-20T17:32:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7729000000000002e-05 2024-04-20T17:32:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:32:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:10.998762+0000", 2024-04-20T17:32:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000118021 2024-04-20T17:32:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:32:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:10.998873+0000", 2024-04-20T17:32:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011142 2024-04-20T17:32:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:32:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.003756+0000", 2024-04-20T17:32:41.872 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0048830059999999996 2024-04-20T17:32:41.872 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.872 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.872 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:41.872 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.215029+0000", 2024-04-20T17:32:41.872 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.21127307100000001 2024-04-20T17:32:41.872 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.872 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.872 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:32:41.872 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.215064+0000", 2024-04-20T17:32:41.872 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4406999999999998e-05 2024-04-20T17:32:41.872 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.872 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.872 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:32:41.872 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.215098+0000", 2024-04-20T17:32:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4069000000000002e-05 2024-04-20T17:32:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:32:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275211 4.6 4:6b91bd7d:::benchmark_data_smithi171_122772_object105019:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:32:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:11.008900+0000", 2024-04-20T17:32:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 270.81941968799998, 2024-04-20T17:32:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20628696999999999, 2024-04-20T17:32:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:32:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:32:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:32:41.874 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:32:41.874 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:32:41.874 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275211 2024-04-20T17:32:41.874 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.874 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:32:41.874 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.874 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:32:41.874 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.008900+0000", 2024-04-20T17:32:41.874 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.874 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.874 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.874 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:32:41.874 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.008900+0000", 2024-04-20T17:32:41.875 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.875 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.875 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.875 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:32:41.875 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.008902+0000", 2024-04-20T17:32:41.875 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1569999999999998e-06 2024-04-20T17:32:41.875 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.875 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.875 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:32:41.875 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.008909+0000", 2024-04-20T17:32:41.875 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5810000000000002e-06 2024-04-20T17:32:41.875 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.875 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.876 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:32:41.876 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.008922+0000", 2024-04-20T17:32:41.876 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3468999999999999e-05 2024-04-20T17:32:41.876 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.876 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.876 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:32:41.876 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.008929+0000", 2024-04-20T17:32:41.876 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2919999999999998e-06 2024-04-20T17:32:41.876 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.876 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.876 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:32:41.877 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.008986+0000", 2024-04-20T17:32:41.877 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6913999999999998e-05 2024-04-20T17:32:41.877 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.877 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.877 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:32:41.877 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.009110+0000", 2024-04-20T17:32:41.877 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012400599999999999 2024-04-20T17:32:41.877 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.877 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.877 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:32:41.877 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.009222+0000", 2024-04-20T17:32:41.877 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000111906 2024-04-20T17:32:41.877 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.878 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.878 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:32:41.878 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.014311+0000", 2024-04-20T17:32:41.878 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0050890980000000002 2024-04-20T17:32:41.878 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.878 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.878 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:41.878 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.215130+0000", 2024-04-20T17:32:41.878 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20081843799999999 2024-04-20T17:32:41.878 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.878 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.878 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:32:41.878 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.215160+0000", 2024-04-20T17:32:41.878 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0673000000000003e-05 2024-04-20T17:32:41.879 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.879 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.879 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:32:41.879 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.215187+0000", 2024-04-20T17:32:41.879 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6435999999999999e-05 2024-04-20T17:32:41.879 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.879 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:32:41.879 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.879 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.879 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.879 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275217 4.6 4:6f52af41:::benchmark_data_smithi171_122772_object105025:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:32:41.879 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:11.014905+0000", 2024-04-20T17:32:41.879 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 270.81341472999998, 2024-04-20T17:32:41.879 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20678690999999999, 2024-04-20T17:32:41.880 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:32:41.880 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:32:41.880 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:41.880 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:32:41.880 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:32:41.880 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:32:41.880 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275217 2024-04-20T17:32:41.880 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.880 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:32:41.880 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.880 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:32:41.880 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.014905+0000", 2024-04-20T17:32:41.880 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.881 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.881 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.881 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:32:41.881 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.014905+0000", 2024-04-20T17:32:41.881 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.881 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.881 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.881 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:32:41.881 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.014907+0000", 2024-04-20T17:32:41.881 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.967e-06 2024-04-20T17:32:41.881 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.881 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.881 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:32:41.881 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.014912+0000", 2024-04-20T17:32:41.881 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3759999999999999e-06 2024-04-20T17:32:41.882 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.882 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.882 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:32:41.882 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.014925+0000", 2024-04-20T17:32:41.882 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3174e-05 2024-04-20T17:32:41.882 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.882 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.882 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:32:41.882 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.014932+0000", 2024-04-20T17:32:41.882 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8229999999999997e-06 2024-04-20T17:32:41.882 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.882 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.883 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:32:41.883 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.014985+0000", 2024-04-20T17:32:41.883 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3062000000000001e-05 2024-04-20T17:32:41.883 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.883 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.883 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:32:41.883 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.015090+0000", 2024-04-20T17:32:41.883 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000104502 2024-04-20T17:32:41.883 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.883 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.883 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:32:41.883 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.015196+0000", 2024-04-20T17:32:41.883 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000106044 2024-04-20T17:32:41.884 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.884 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.884 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:32:41.884 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.024773+0000", 2024-04-20T17:32:41.884 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0095773449999999993 2024-04-20T17:32:41.884 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.884 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.884 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:41.884 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.221616+0000", 2024-04-20T17:32:41.884 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.196842454 2024-04-20T17:32:41.884 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.884 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.884 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:32:41.884 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.221649+0000", 2024-04-20T17:32:41.885 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3352999999999997e-05 2024-04-20T17:32:41.885 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.885 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.885 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:32:41.885 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.221692+0000", 2024-04-20T17:32:41.885 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2809999999999998e-05 2024-04-20T17:32:41.885 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.885 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:32:41.885 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.885 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.885 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.885 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275240 4.6 4:6dacae24:::benchmark_data_smithi171_122772_object105048:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:32:41.885 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:11.031577+0000", 2024-04-20T17:32:41.885 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 270.79674235700003, 2024-04-20T17:32:41.885 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19679422799999999, 2024-04-20T17:32:41.886 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:32:41.886 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:32:41.886 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:41.886 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:32:41.886 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:32:41.886 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:32:41.886 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275240 2024-04-20T17:32:41.886 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.886 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:32:41.886 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.886 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:32:41.886 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.031577+0000", 2024-04-20T17:32:41.886 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.887 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.887 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.887 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:32:41.887 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.031577+0000", 2024-04-20T17:32:41.887 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.887 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.887 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.887 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:32:41.887 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.031579+0000", 2024-04-20T17:32:41.887 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4610000000000001e-06 2024-04-20T17:32:41.887 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.887 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.888 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:32:41.888 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.031583+0000", 2024-04-20T17:32:41.888 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6870000000000002e-06 2024-04-20T17:32:41.888 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.888 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.888 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:32:41.888 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.031594+0000", 2024-04-20T17:32:41.888 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0594000000000001e-05 2024-04-20T17:32:41.888 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.888 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.888 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:32:41.888 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.031601+0000", 2024-04-20T17:32:41.889 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9240000000000001e-06 2024-04-20T17:32:41.889 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.889 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.889 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:32:41.889 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.031653+0000", 2024-04-20T17:32:41.889 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2228999999999997e-05 2024-04-20T17:32:41.889 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.889 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.889 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:32:41.889 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.031761+0000", 2024-04-20T17:32:41.889 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000108008 2024-04-20T17:32:41.889 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.889 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.890 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:32:41.890 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.031871+0000", 2024-04-20T17:32:41.890 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000110331 2024-04-20T17:32:41.890 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.890 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.890 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:32:41.890 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.036276+0000", 2024-04-20T17:32:41.890 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.004404799 2024-04-20T17:32:41.890 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.890 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.890 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:41.891 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.228303+0000", 2024-04-20T17:32:41.891 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19202655299999999 2024-04-20T17:32:41.891 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.891 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.891 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:32:41.891 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.228336+0000", 2024-04-20T17:32:41.891 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3281000000000002e-05 2024-04-20T17:32:41.891 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.891 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.891 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:32:41.891 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.228371+0000", 2024-04-20T17:32:41.891 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5361000000000002e-05 2024-04-20T17:32:41.892 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.892 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:32:41.892 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.892 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.892 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.892 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275248 4.6 4:65347144:::benchmark_data_smithi171_122772_object105056:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:32:41.892 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:11.041968+0000", 2024-04-20T17:32:41.892 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 270.78635204300002, 2024-04-20T17:32:41.892 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.18653969300000001, 2024-04-20T17:32:41.892 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:32:41.892 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:32:41.892 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:41.892 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:32:41.893 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:32:41.893 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:32:41.893 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275248 2024-04-20T17:32:41.893 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.893 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:32:41.893 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.893 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:32:41.893 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.041968+0000", 2024-04-20T17:32:41.893 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.893 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.893 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.894 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:32:41.894 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.041968+0000", 2024-04-20T17:32:41.894 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.894 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.894 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.894 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:32:41.895 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.041970+0000", 2024-04-20T17:32:41.895 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1229999999999998e-06 2024-04-20T17:32:41.895 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.895 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.895 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:32:41.895 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.041976+0000", 2024-04-20T17:32:41.895 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.852e-06 2024-04-20T17:32:41.895 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.895 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.895 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:32:41.895 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.041987+0000", 2024-04-20T17:32:41.895 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1355000000000001e-05 2024-04-20T17:32:41.895 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.896 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.896 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:32:41.896 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.041995+0000", 2024-04-20T17:32:41.896 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4540000000000007e-06 2024-04-20T17:32:41.896 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.896 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.896 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:32:41.896 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.042050+0000", 2024-04-20T17:32:41.896 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4239e-05 2024-04-20T17:32:41.896 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.896 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.896 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:32:41.896 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.042138+0000", 2024-04-20T17:32:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8004999999999995e-05 2024-04-20T17:32:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:32:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.042261+0000", 2024-04-20T17:32:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012317299999999999 2024-04-20T17:32:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:32:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.046505+0000", 2024-04-20T17:32:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0042438600000000003 2024-04-20T17:32:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.898 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:41.898 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.228413+0000", 2024-04-20T17:32:41.898 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.18190864500000001 2024-04-20T17:32:41.898 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.898 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.898 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:32:41.898 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.228465+0000", 2024-04-20T17:32:41.898 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2074000000000003e-05 2024-04-20T17:32:41.898 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.898 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.898 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:32:41.899 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.228507+0000", 2024-04-20T17:32:41.899 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1912999999999997e-05 2024-04-20T17:32:41.899 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.899 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:32:41.899 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.899 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.899 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.899 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275263 4.6 4:6160fa82:::benchmark_data_smithi171_122772_object105071:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:32:41.899 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:11.051143+0000", 2024-04-20T17:32:41.899 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 270.77717612499998, 2024-04-20T17:32:41.899 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.18078492099999999, 2024-04-20T17:32:41.899 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:32:41.899 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:32:41.900 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:41.900 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:32:41.900 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:32:41.900 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:32:41.900 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275263 2024-04-20T17:32:41.900 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.900 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:32:41.900 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.900 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:32:41.900 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.051143+0000", 2024-04-20T17:32:41.900 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.900 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.900 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:32:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.051143+0000", 2024-04-20T17:32:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:32:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.051146+0000", 2024-04-20T17:32:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1730000000000002e-06 2024-04-20T17:32:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:32:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.051152+0000", 2024-04-20T17:32:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8579999999999996e-06 2024-04-20T17:32:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.902 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:32:41.902 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.051165+0000", 2024-04-20T17:32:41.902 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3661e-05 2024-04-20T17:32:41.902 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.902 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.902 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:32:41.902 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.051173+0000", 2024-04-20T17:32:41.902 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8329999999999994e-06 2024-04-20T17:32:41.902 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.902 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.902 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:32:41.902 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.051233+0000", 2024-04-20T17:32:41.902 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9837999999999999e-05 2024-04-20T17:32:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:32:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.051354+0000", 2024-04-20T17:32:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012070699999999999 2024-04-20T17:32:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:32:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.051468+0000", 2024-04-20T17:32:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000114658 2024-04-20T17:32:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:32:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.056305+0000", 2024-04-20T17:32:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0048365329999999996 2024-04-20T17:32:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.231849+0000", 2024-04-20T17:32:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.175544645 2024-04-20T17:32:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:32:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.231886+0000", 2024-04-20T17:32:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6322999999999998e-05 2024-04-20T17:32:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:32:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.231928+0000", 2024-04-20T17:32:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2691999999999997e-05 2024-04-20T17:32:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:32:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275278 4.6 4:6f1c4c64:::benchmark_data_smithi171_122772_object105086:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:32:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:11.064797+0000", 2024-04-20T17:32:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 270.76352293999997, 2024-04-20T17:32:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.167260615, 2024-04-20T17:32:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:32:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:32:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:32:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:32:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:32:41.906 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275278 2024-04-20T17:32:41.906 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.906 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:32:41.906 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.906 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:32:41.906 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.064797+0000", 2024-04-20T17:32:41.906 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.906 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.906 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.906 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:32:41.906 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.064797+0000", 2024-04-20T17:32:41.906 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:32:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.064798+0000", 2024-04-20T17:32:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.567e-06 2024-04-20T17:32:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:32:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.064803+0000", 2024-04-20T17:32:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0429999999999998e-06 2024-04-20T17:32:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:32:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.064815+0000", 2024-04-20T17:32:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.148e-05 2024-04-20T17:32:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:32:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.064822+0000", 2024-04-20T17:32:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2359999999999998e-06 2024-04-20T17:32:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:32:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.064880+0000", 2024-04-20T17:32:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7698999999999998e-05 2024-04-20T17:32:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:32:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.064999+0000", 2024-04-20T17:32:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011958 2024-04-20T17:32:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:32:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.065120+0000", 2024-04-20T17:32:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000120512 2024-04-20T17:32:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:32:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.069852+0000", 2024-04-20T17:32:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0047317180000000002 2024-04-20T17:32:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.231968+0000", 2024-04-20T17:32:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.162116383 2024-04-20T17:32:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:32:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.232017+0000", 2024-04-20T17:32:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9614999999999998e-05 2024-04-20T17:32:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:32:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.232057+0000", 2024-04-20T17:32:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9782e-05 2024-04-20T17:32:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:32:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.911 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.911 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.911 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275285 4.6 4:63a3635d:::benchmark_data_smithi171_122772_object105093:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:32:41.911 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:11.074387+0000", 2024-04-20T17:32:41.911 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 270.75393263000001, 2024-04-20T17:32:41.911 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16127122899999999, 2024-04-20T17:32:41.911 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:32:41.911 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:32:41.911 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:41.911 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:32:41.911 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:32:41.911 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:32:41.911 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275285 2024-04-20T17:32:41.911 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:32:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:32:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.074387+0000", 2024-04-20T17:32:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:32:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.074387+0000", 2024-04-20T17:32:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:32:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.074388+0000", 2024-04-20T17:32:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.339e-06 2024-04-20T17:32:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:32:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.074393+0000", 2024-04-20T17:32:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4259999999999996e-06 2024-04-20T17:32:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:32:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.074402+0000", 2024-04-20T17:32:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6609999999999995e-06 2024-04-20T17:32:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:32:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.074408+0000", 2024-04-20T17:32:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6520000000000003e-06 2024-04-20T17:32:41.914 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.914 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.914 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:32:41.914 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.074455+0000", 2024-04-20T17:32:41.914 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6717e-05 2024-04-20T17:32:41.914 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.914 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.914 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:32:41.914 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.074530+0000", 2024-04-20T17:32:41.918 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5661000000000006e-05 2024-04-20T17:32:41.918 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.918 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.918 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:32:41.918 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.074597+0000", 2024-04-20T17:32:41.918 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6418e-05 2024-04-20T17:32:41.918 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.918 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.918 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:32:41.918 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.078692+0000", 2024-04-20T17:32:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0040947350000000004 2024-04-20T17:32:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.235564+0000", 2024-04-20T17:32:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15687225099999999 2024-04-20T17:32:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:32:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.235616+0000", 2024-04-20T17:32:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2599999999999998e-05 2024-04-20T17:32:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:32:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.235658+0000", 2024-04-20T17:32:41.920 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1768999999999999e-05 2024-04-20T17:32:41.920 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.920 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:32:41.920 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.920 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.920 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.920 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275296 4.6 4:6850fa57:::benchmark_data_smithi171_122772_object105104:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:32:41.920 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:11.083403+0000", 2024-04-20T17:32:41.920 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 270.74491667199999, 2024-04-20T17:32:41.920 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15593427100000001, 2024-04-20T17:32:41.920 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:32:41.920 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:32:41.920 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:32:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:32:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:32:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275296 2024-04-20T17:32:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:32:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:32:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.083403+0000", 2024-04-20T17:32:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:32:41.922 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.083403+0000", 2024-04-20T17:32:41.922 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.922 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.922 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.922 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:32:41.922 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.083406+0000", 2024-04-20T17:32:41.922 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8370000000000001e-06 2024-04-20T17:32:41.922 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.922 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.922 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:32:41.922 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.083414+0000", 2024-04-20T17:32:41.922 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2630000000000003e-06 2024-04-20T17:32:41.922 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:32:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.083431+0000", 2024-04-20T17:32:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6560000000000001e-05 2024-04-20T17:32:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:32:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.083441+0000", 2024-04-20T17:32:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0132000000000001e-05 2024-04-20T17:32:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:32:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.083503+0000", 2024-04-20T17:32:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2119000000000006e-05 2024-04-20T17:32:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:32:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.083625+0000", 2024-04-20T17:32:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000122547 2024-04-20T17:32:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:32:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.083735+0000", 2024-04-20T17:32:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000109764 2024-04-20T17:32:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:32:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.088797+0000", 2024-04-20T17:32:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0050613510000000004 2024-04-20T17:32:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.239279+0000", 2024-04-20T17:32:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15048260599999999 2024-04-20T17:32:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:32:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.239310+0000", 2024-04-20T17:32:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1300999999999997e-05 2024-04-20T17:32:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:32:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.239337+0000", 2024-04-20T17:32:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6791000000000001e-05 2024-04-20T17:32:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:32:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275312 4.6 4:6a829b24:::benchmark_data_smithi171_122772_object105120:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:32:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:11.101309+0000", 2024-04-20T17:32:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 270.72701087799999, 2024-04-20T17:32:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.14268697999999999, 2024-04-20T17:32:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:32:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:32:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:32:41.927 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:32:41.927 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:32:41.927 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275312 2024-04-20T17:32:41.927 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.927 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:32:41.927 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.927 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:32:41.927 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.101309+0000", 2024-04-20T17:32:41.927 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.927 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.927 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.927 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:32:41.927 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.101309+0000", 2024-04-20T17:32:41.928 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.928 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.928 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.928 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:32:41.928 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.101312+0000", 2024-04-20T17:32:41.928 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3110000000000001e-06 2024-04-20T17:32:41.928 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.928 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.928 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:32:41.928 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.101320+0000", 2024-04-20T17:32:41.928 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7659999999999998e-06 2024-04-20T17:32:41.928 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.928 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:32:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.101337+0000", 2024-04-20T17:32:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7345000000000001e-05 2024-04-20T17:32:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:32:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.101347+0000", 2024-04-20T17:32:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.594e-06 2024-04-20T17:32:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:32:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.101415+0000", 2024-04-20T17:32:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8751000000000006e-05 2024-04-20T17:32:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.930 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:32:41.930 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.101540+0000", 2024-04-20T17:32:41.930 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000124029 2024-04-20T17:32:41.930 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.930 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.930 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:32:41.930 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.101658+0000", 2024-04-20T17:32:41.930 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000118235 2024-04-20T17:32:41.930 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.930 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.930 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:32:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.106325+0000", 2024-04-20T17:32:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0046667490000000004 2024-04-20T17:32:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.243904+0000", 2024-04-20T17:32:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.137579902 2024-04-20T17:32:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:32:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.243956+0000", 2024-04-20T17:32:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1690999999999997e-05 2024-04-20T17:32:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:32:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.243996+0000", 2024-04-20T17:32:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9607000000000002e-05 2024-04-20T17:32:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:32:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.16506.0:202760 5.b 5:d9759bc2:::benchmark_data_smithi171_137281_object71326:head [delete] snapc 0=[] ondisk+write+known_if_redirected e599)", 2024-04-20T17:32:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:32:08.720790+0000", 2024-04-20T17:32:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 33.107529395999997, 2024-04-20T17:32:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.22549492700000001, 2024-04-20T17:32:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:32:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:32:41.933 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:41.933 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:32:41.933 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.16506", 2024-04-20T17:32:41.933 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:32:41.933 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 202760 2024-04-20T17:32:41.933 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.933 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:32:41.933 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.933 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:32:41.933 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.720790+0000", 2024-04-20T17:32:41.933 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.933 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.933 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.934 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:32:41.934 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.720790+0000", 2024-04-20T17:32:41.934 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.934 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.934 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.934 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:32:41.934 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.720793+0000", 2024-04-20T17:32:41.934 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3539999999999998e-06 2024-04-20T17:32:41.934 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.934 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.934 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:32:41.934 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.720800+0000", 2024-04-20T17:32:41.934 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.046e-06 2024-04-20T17:32:41.935 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.935 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.935 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:32:41.935 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.720814+0000", 2024-04-20T17:32:41.935 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4158000000000001e-05 2024-04-20T17:32:41.935 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.935 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.935 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:32:41.935 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.720821+0000", 2024-04-20T17:32:41.935 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7499999999999997e-06 2024-04-20T17:32:41.935 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.935 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.935 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:32:41.936 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.720879+0000", 2024-04-20T17:32:41.936 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8934999999999997e-05 2024-04-20T17:32:41.936 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.936 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.936 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:32:41.936 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.721008+0000", 2024-04-20T17:32:41.936 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012829700000000001 2024-04-20T17:32:41.936 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.936 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.936 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:32:41.936 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.721129+0000", 2024-04-20T17:32:41.936 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000121704 2024-04-20T17:32:41.936 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.937 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.937 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:32:41.937 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.729580+0000", 2024-04-20T17:32:41.937 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0084508080000000006 2024-04-20T17:32:41.937 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.937 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.937 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:41.937 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.946173+0000", 2024-04-20T17:32:41.937 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.21659226000000001 2024-04-20T17:32:41.937 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.937 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.937 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:32:41.938 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.946239+0000", 2024-04-20T17:32:41.938 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6212999999999998e-05 2024-04-20T17:32:41.938 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.938 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.938 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:32:41.938 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.946285+0000", 2024-04-20T17:32:41.938 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6402000000000001e-05 2024-04-20T17:32:41.938 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.938 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:32:41.938 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.938 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.16506.0:202763 5.b 5:dbb9b06e:::benchmark_data_smithi171_137281_object71329:head [delete] snapc 0=[] ondisk+write+known_if_redirected e599)", 2024-04-20T17:32:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:32:08.722780+0000", 2024-04-20T17:32:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 33.105539866999997, 2024-04-20T17:32:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.22980968099999999, 2024-04-20T17:32:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:32:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:32:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:32:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.16506", 2024-04-20T17:32:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:32:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 202763 2024-04-20T17:32:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.940 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:32:41.940 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.940 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:32:41.940 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.722780+0000", 2024-04-20T17:32:41.940 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.940 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.940 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.940 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:32:41.940 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.722780+0000", 2024-04-20T17:32:41.940 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.940 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.940 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.940 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:32:41.940 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.722781+0000", 2024-04-20T17:32:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8700000000000004e-07 2024-04-20T17:32:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:32:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.722784+0000", 2024-04-20T17:32:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0740000000000001e-06 2024-04-20T17:32:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:32:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.722793+0000", 2024-04-20T17:32:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3300000000000005e-06 2024-04-20T17:32:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:32:41.942 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.722800+0000", 2024-04-20T17:32:41.942 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3239999999999997e-06 2024-04-20T17:32:41.942 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.942 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.942 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:32:41.942 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.725063+0000", 2024-04-20T17:32:41.942 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0022626 2024-04-20T17:32:41.942 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.942 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.942 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:32:41.942 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.725184+0000", 2024-04-20T17:32:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000120701 2024-04-20T17:32:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:32:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.725275+0000", 2024-04-20T17:32:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1162000000000006e-05 2024-04-20T17:32:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:32:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.729728+0000", 2024-04-20T17:32:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0044527710000000003 2024-04-20T17:32:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.952529+0000", 2024-04-20T17:32:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.22280121 2024-04-20T17:32:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:32:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.952562+0000", 2024-04-20T17:32:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3253000000000002e-05 2024-04-20T17:32:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:32:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.952589+0000", 2024-04-20T17:32:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7368999999999998e-05 2024-04-20T17:32:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:32:41.945 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.945 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.945 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.945 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.16506.0:202825 5.b 5:d9279200:::benchmark_data_smithi171_137281_object71391:head [delete] snapc 0=[] ondisk+write+known_if_redirected e599)", 2024-04-20T17:32:41.945 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:32:08.801332+0000", 2024-04-20T17:32:41.945 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 33.026987194999997, 2024-04-20T17:32:41.945 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15138621499999999, 2024-04-20T17:32:41.945 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:32:41.945 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:32:41.945 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:41.945 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:32:41.945 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.16506", 2024-04-20T17:32:41.945 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:32:41.946 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 202825 2024-04-20T17:32:41.946 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.946 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:32:41.946 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.946 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:32:41.946 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.801332+0000", 2024-04-20T17:32:41.946 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.946 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.946 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.946 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:32:41.946 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.801332+0000", 2024-04-20T17:32:41.946 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.946 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:32:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.801335+0000", 2024-04-20T17:32:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.852e-06 2024-04-20T17:32:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:32:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.801342+0000", 2024-04-20T17:32:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0480000000000001e-06 2024-04-20T17:32:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:32:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.801358+0000", 2024-04-20T17:32:41.948 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5489e-05 2024-04-20T17:32:41.948 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.948 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.948 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:32:41.948 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.801367+0000", 2024-04-20T17:32:41.948 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2490000000000007e-06 2024-04-20T17:32:41.948 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.948 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.948 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:32:41.948 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.801422+0000", 2024-04-20T17:32:41.948 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5275000000000002e-05 2024-04-20T17:32:41.948 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.948 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.948 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:32:41.949 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.801531+0000", 2024-04-20T17:32:41.949 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000108706 2024-04-20T17:32:41.949 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.949 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.949 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:32:41.949 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.801616+0000", 2024-04-20T17:32:41.949 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5022e-05 2024-04-20T17:32:41.949 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.949 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.949 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:32:41.949 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.805722+0000", 2024-04-20T17:32:41.949 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0041055939999999997 2024-04-20T17:32:41.950 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.950 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.950 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:41.950 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.952646+0000", 2024-04-20T17:32:41.950 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.146924468 2024-04-20T17:32:41.950 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.950 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.950 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:32:41.950 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.952689+0000", 2024-04-20T17:32:41.950 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3019000000000003e-05 2024-04-20T17:32:41.950 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.950 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.950 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:32:41.951 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.952719+0000", 2024-04-20T17:32:41.951 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9493e-05 2024-04-20T17:32:41.951 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.951 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:32:41.951 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.951 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.951 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.951 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:32:41.952 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.16506.0:202840 5.b 5:d515e270:::benchmark_data_smithi171_137281_object71406:head [delete] snapc 0=[] ondisk+write+known_if_redirected e599)", 2024-04-20T17:32:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:32:08.814812+0000", 2024-04-20T17:32:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 33.013507445999998, 2024-04-20T17:32:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.13804498100000001, 2024-04-20T17:32:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:32:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:32:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:32:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:32:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.16506", 2024-04-20T17:32:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:32:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 202840 2024-04-20T17:32:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:32:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:32:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.814812+0000", 2024-04-20T17:32:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:32:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.814812+0000", 2024-04-20T17:32:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:32:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:32:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.814814+0000", 2024-04-20T17:32:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7349999999999999e-06 2024-04-20T17:32:41.955 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.955 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.955 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:32:41.955 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.814819+0000", 2024-04-20T17:32:41.955 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6999999999999999e-06 2024-04-20T17:32:41.955 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.955 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.955 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:32:41.955 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.814828+0000", 2024-04-20T17:32:41.955 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2499999999999995e-06 2024-04-20T17:32:41.955 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.955 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.956 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:32:41.956 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.814835+0000", 2024-04-20T17:32:41.956 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6029999999999999e-06 2024-04-20T17:32:41.956 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.956 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.956 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:32:41.956 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.814896+0000", 2024-04-20T17:32:41.956 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0744e-05 2024-04-20T17:32:41.956 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.956 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.956 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:32:41.956 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.815015+0000", 2024-04-20T17:32:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011930399999999999 2024-04-20T17:32:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:32:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.815108+0000", 2024-04-20T17:32:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2404000000000005e-05 2024-04-20T17:32:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:32:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.818852+0000", 2024-04-20T17:32:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.003743803 2024-04-20T17:32:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:32:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.952772+0000", 2024-04-20T17:32:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.13392010300000001 2024-04-20T17:32:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:32:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.952824+0000", 2024-04-20T17:32:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1805999999999998e-05 2024-04-20T17:32:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:32:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:32:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:32:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.952857+0000", 2024-04-20T17:32:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3528999999999997e-05 2024-04-20T17:32:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:32:41.959 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.959 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:32:41.959 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:32:41.959 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:32:41.990 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:42.557 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-20T17:32:42.946 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:32:43.083 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:32:43.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:43.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:32:43.190 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:32:43.221 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:43.245 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:32:43.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:43.249 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:32:43.350 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:32:43.378 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:43.397 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:32:43.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:43.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:32:43.502 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:32:43.529 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:43.534 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:43 smithi171 ceph-mon[44189]: pgmap v1595: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.6 GiB data, 6.6 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 863 op/s 2024-04-20T17:32:43.549 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:32:43.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:43.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:32:43.653 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:32:43.679 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:43 smithi106 ceph-mon[44423]: pgmap v1595: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.6 GiB data, 6.6 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 863 op/s 2024-04-20T17:32:43.680 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:43.699 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:32:43.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:43.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:32:43.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:43 smithi005 ceph-mon[39562]: pgmap v1595: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.6 GiB data, 6.6 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 863 op/s 2024-04-20T17:32:43.805 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:32:43.850 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:43.954 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:32:43.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:43.960 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:32:44.061 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:32:44.137 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:44.341 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:32:44.341 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:32:44.347 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:32:44.363 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:32:44.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:44.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:32:44.384 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:32:44.420 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:32:44.460 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:32:44.463 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:32:44.468 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:32:44.493 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:32:44.494 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:44.517 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:32:44.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:44.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:32:44.532 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:32:44.565 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:32:44.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:44 smithi171 ceph-mon[44189]: 5.0 scrub starts 2024-04-20T17:32:44.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:44 smithi171 ceph-mon[44189]: 5.0 scrub ok 2024-04-20T17:32:44.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:44 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3667498627' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:32:44.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:44 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3044543552' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:32:44.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:44 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:44.600 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:32:44.620 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:32:44.634 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:32:44.647 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:44.671 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:32:44.674 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:32:44.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:44.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:32:44.688 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:32:44.688 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:32:44.688 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:32:44.688 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:32:44.688 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:32:44.688 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:32:44.688 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:32:44.688 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:32:44.688 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:32:44.688 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:32:44.688 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:32:44.689 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:32:44.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:44 smithi005 ceph-mon[39562]: 5.0 scrub starts 2024-04-20T17:32:44.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:44 smithi005 ceph-mon[39562]: 5.0 scrub ok 2024-04-20T17:32:44.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:44 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3667498627' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:32:44.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:44 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3044543552' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:32:44.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:44 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:44.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:44 smithi106 ceph-mon[44423]: 5.0 scrub starts 2024-04-20T17:32:44.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:44 smithi106 ceph-mon[44423]: 5.0 scrub ok 2024-04-20T17:32:44.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:44 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3667498627' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:32:44.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:44 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3044543552' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:32:44.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:44 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:44.778 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:32:44.845 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:44.898 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:32:44.898 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:32:45.010 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:32:45.022 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:32:45.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:45 smithi005 ceph-mon[39562]: pgmap v1596: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.5 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 795 op/s 2024-04-20T17:32:45.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:45 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:32:45.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:45 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3667498627' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:32:45.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:45 smithi005 ceph-mon[39562]: osdmap e608: 12 total, 12 up, 11 in 2024-04-20T17:32:45.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:45 smithi005 ceph-mon[39562]: from='client.26810 v1:172.21.15.5:0/639651833' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:32:45.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:45 smithi171 ceph-mon[44189]: pgmap v1596: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.5 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 795 op/s 2024-04-20T17:32:45.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:45 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:32:45.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:45 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3667498627' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:32:45.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:45 smithi171 ceph-mon[44189]: osdmap e608: 12 total, 12 up, 11 in 2024-04-20T17:32:45.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:45 smithi171 ceph-mon[44189]: from='client.26810 v1:172.21.15.5:0/639651833' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:32:45.638 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-20T17:32:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:45 smithi106 ceph-mon[44423]: pgmap v1596: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.5 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 795 op/s 2024-04-20T17:32:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:45 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:32:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:45 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3667498627' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:32:45.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:45 smithi106 ceph-mon[44423]: osdmap e608: 12 total, 12 up, 11 in 2024-04-20T17:32:45.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:45.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:45 smithi106 ceph-mon[44423]: from='client.26810 v1:172.21.15.5:0/639651833' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:32:45.933 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":1597,"stamp":"2024-04-20T17:32:44.431940+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1607729720,"num_objects":24526,"num_object_clones":0,"num_object_copies":49052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24526,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":238521,"num_write_kb":8414523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161225,"ondisk_log_size":161225,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":6751068,"kb_used_data":3208744,"kb_used_omap":225,"kb_used_meta":3541790,"kb_avail":1024445604,"statfs":{"total":1055945392128,"available":1049032298496,"internally_reserved":0,"allocated":3285753856,"data_stored":3281511697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":231348,"internal_metadata":3626793036},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,5,8,9,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-393412608,"num_objects":-6003,"num_object_clones":0,"num_object_copies":-12006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6003,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.106500"},"pg_stats":[{"pgid":"5.f","version":"607'14758","reported_seq":14995,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-04-20T17:32:40.776069+0000","last_change":"2024-04-20T17:32:29.834668+0000","last_active":"2024-04-20T17:32:40.776069+0000","last_peered":"2024-04-20T17:32:40.776069+0000","last_clean":"2024-04-20T17:32:40.776069+0000","last_became_active":"2024-04-20T17:29:29.256875+0000","last_became_peered":"2024-04-20T17:29:29.256875+0000","last_unstale":"2024-04-20T17:32:40.776069+0000","last_undegraded":"2024-04-20T17:32:40.776069+0000","last_fullsized":"2024-04-20T17:32:40.776069+0000","mapping_epoch":543,"log_start":"572'4700","ondisk_log_start":"572'4700","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'14088","last_scrub_stamp":"2024-04-20T17:32:29.834573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:32:29.834573+0000","objects_scrubbed":2321,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:33:31.351043+0000","scrub_duration":354,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107479040,"num_objects":1640,"num_object_clones":0,"num_object_copies":3280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14758,"num_write_kb":524736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"607'14590","reported_seq":14825,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-04-20T17:32:41.230099+0000","last_change":"2024-04-20T17:32:31.358348+0000","last_active":"2024-04-20T17:32:41.230099+0000","last_peered":"2024-04-20T17:32:41.230099+0000","last_clean":"2024-04-20T17:32:41.230099+0000","last_became_active":"2024-04-20T17:29:29.326722+0000","last_became_peered":"2024-04-20T17:29:29.326722+0000","last_unstale":"2024-04-20T17:32:41.230099+0000","last_undegraded":"2024-04-20T17:32:41.230099+0000","last_fullsized":"2024-04-20T17:32:41.230099+0000","mapping_epoch":543,"log_start":"571'4500","ondisk_log_start":"571'4500","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'14003","last_scrub_stamp":"2024-04-20T17:32:31.358187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:32:31.358187+0000","objects_scrubbed":2220,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:33:47.525578+0000","scrub_duration":320,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106430464,"num_objects":1624,"num_object_clones":0,"num_object_copies":3248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14590,"num_write_kb":518848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"607'14969","reported_seq":15241,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-04-20T17:32:43.810183+0000","last_change":"2024-04-20T17:32:35.426303+0000","last_active":"2024-04-20T17:32:43.810183+0000","last_peered":"2024-04-20T17:32:43.810183+0000","last_clean":"2024-04-20T17:32:43.810183+0000","last_became_active":"2024-04-20T17:29:29.257014+0000","last_became_peered":"2024-04-20T17:29:29.257014+0000","last_unstale":"2024-04-20T17:32:43.810183+0000","last_undegraded":"2024-04-20T17:32:43.810183+0000","last_fullsized":"2024-04-20T17:32:43.810183+0000","mapping_epoch":543,"log_start":"572'4900","ondisk_log_start":"572'4900","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"604'14565","last_scrub_stamp":"2024-04-20T17:32:35.426139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:32:35.426139+0000","objects_scrubbed":1921,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:33:53.042110+0000","scrub_duration":318,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98697240,"num_objects":1507,"num_object_clones":0,"num_object_copies":3014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1507,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14969,"num_write_kb":527169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"607'14877","reported_seq":15120,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-04-20T17:32:40.876272+0000","last_change":"2024-04-20T17:32:27.917596+0000","last_active":"2024-04-20T17:32:40.876272+0000","last_peered":"2024-04-20T17:32:40.876272+0000","last_clean":"2024-04-20T17:32:40.876272+0000","last_became_active":"2024-04-20T17:29:29.597181+0000","last_became_peered":"2024-04-20T17:29:29.597181+0000","last_unstale":"2024-04-20T17:32:40.876272+0000","last_undegraded":"2024-04-20T17:32:40.876272+0000","last_fullsized":"2024-04-20T17:32:40.876272+0000","mapping_epoch":543,"log_start":"572'4800","ondisk_log_start":"572'4800","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'14052","last_scrub_stamp":"2024-04-20T17:32:27.917451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:32:27.917451+0000","objects_scrubbed":2462,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:33:53.258599+0000","scrub_duration":432,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106233856,"num_objects":1621,"num_object_clones":0,"num_object_copies":3242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14877,"num_write_kb":527936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"607'14945","reported_seq":15182,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-04-20T17:32:41.125823+0000","last_change":"2024-04-20T17:32:37.559106+0000","last_active":"2024-04-20T17:32:41.125823+0000","last_peered":"2024-04-20T17:32:41.125823+0000","last_clean":"2024-04-20T17:32:41.125823+0000","last_became_active":"2024-04-20T17:29:29.258913+0000","last_became_peered":"2024-04-20T17:29:29.258913+0000","last_unstale":"2024-04-20T17:32:41.125823+0000","last_undegraded":"2024-04-20T17:32:41.125823+0000","last_fullsized":"2024-04-20T17:32:41.125823+0000","mapping_epoch":543,"log_start":"572'4900","ondisk_log_start":"572'4900","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"605'14783","last_scrub_stamp":"2024-04-20T17:32:37.558953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:32:37.558953+0000","objects_scrubbed":1803,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:34:03.010635+0000","scrub_duration":410,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107020288,"num_objects":1633,"num_object_clones":0,"num_object_copies":3266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14945,"num_write_kb":530496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"607'15420","reported_seq":15664,"reported_epoch":607,"state":"active+clean+scrubbing+deep","last_fresh":"2024-04-20T17:32:41.430144+0000","last_change":"2024-04-20T17:32:36.751547+0000","last_active":"2024-04-20T17:32:41.430144+0000","last_peered":"2024-04-20T17:32:41.430144+0000","last_clean":"2024-04-20T17:32:41.430144+0000","last_became_active":"2024-04-20T17:29:29.326880+0000","last_became_peered":"2024-04-20T17:29:29.326880+0000","last_unstale":"2024-04-20T17:32:41.430144+0000","last_undegraded":"2024-04-20T17:32:41.430144+0000","last_fullsized":"2024-04-20T17:32:41.430144+0000","mapping_epoch":543,"log_start":"575'5400","ondisk_log_start":"575'5400","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"580'7147","last_scrub_stamp":"2024-04-20T17:30:58.969065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:30:58.969065+0000","objects_scrubbed":710,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"deep scrubbing for 4s","scrub_duration":1547,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106168320,"num_objects":1620,"num_object_clones":0,"num_object_copies":3240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14710,"num_write_kb":522560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"607'14667","reported_seq":14925,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-04-20T17:32:42.443024+0000","last_change":"2024-04-20T17:31:56.425171+0000","last_active":"2024-04-20T17:32:42.443024+0000","last_peered":"2024-04-20T17:32:42.443024+0000","last_clean":"2024-04-20T17:32:42.443024+0000","last_became_active":"2024-04-20T17:29:29.261848+0000","last_became_peered":"2024-04-20T17:29:29.261848+0000","last_unstale":"2024-04-20T17:32:42.443024+0000","last_undegraded":"2024-04-20T17:32:42.443024+0000","last_fullsized":"2024-04-20T17:32:42.443024+0000","mapping_epoch":543,"log_start":"571'4600","ondisk_log_start":"571'4600","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"597'11709","last_scrub_stamp":"2024-04-20T17:31:56.425068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:31:56.425068+0000","objects_scrubbed":4425,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:33:05.354947+0000","scrub_duration":716,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94830592,"num_objects":1447,"num_object_clones":0,"num_object_copies":2894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14667,"num_write_kb":515648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"607'15040","reported_seq":15273,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-04-20T17:32:43.785446+0000","last_change":"2024-04-20T17:32:37.558844+0000","last_active":"2024-04-20T17:32:43.785446+0000","last_peered":"2024-04-20T17:32:43.785446+0000","last_clean":"2024-04-20T17:32:43.785446+0000","last_became_active":"2024-04-20T17:29:29.255991+0000","last_became_peered":"2024-04-20T17:29:29.255991+0000","last_unstale":"2024-04-20T17:32:43.785446+0000","last_undegraded":"2024-04-20T17:32:43.785446+0000","last_fullsized":"2024-04-20T17:32:43.785446+0000","mapping_epoch":543,"log_start":"573'5000","ondisk_log_start":"573'5000","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"605'14742","last_scrub_stamp":"2024-04-20T17:32:37.558591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:32:37.558591+0000","objects_scrubbed":1778,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:34:03.243753+0000","scrub_duration":382,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96600064,"num_objects":1474,"num_object_clones":0,"num_object_copies":2948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15040,"num_write_kb":528448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"607'14998","reported_seq":15237,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-04-20T17:32:40.851303+0000","last_change":"2024-04-20T17:32:32.839424+0000","last_active":"2024-04-20T17:32:40.851303+0000","last_peered":"2024-04-20T17:32:40.851303+0000","last_clean":"2024-04-20T17:32:40.851303+0000","last_became_active":"2024-04-20T17:29:29.597100+0000","last_became_peered":"2024-04-20T17:29:29.597100+0000","last_unstale":"2024-04-20T17:32:40.851303+0000","last_undegraded":"2024-04-20T17:32:40.851303+0000","last_fullsized":"2024-04-20T17:32:40.851303+0000","mapping_epoch":543,"log_start":"572'4900","ondisk_log_start":"572'4900","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"603'14522","last_scrub_stamp":"2024-04-20T17:32:32.839241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:32:32.839241+0000","objects_scrubbed":2040,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:33:38.680637+0000","scrub_duration":327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102236160,"num_objects":1560,"num_object_clones":0,"num_object_copies":3120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14998,"num_write_kb":529856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"607'18096","reported_seq":18466,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-04-20T17:32:41.394277+0000","last_change":"2024-04-20T17:32:31.325556+0000","last_active":"2024-04-20T17:32:41.394277+0000","last_peered":"2024-04-20T17:32:41.394277+0000","last_clean":"2024-04-20T17:32:41.394277+0000","last_became_active":"2024-04-20T17:29:29.256893+0000","last_became_peered":"2024-04-20T17:29:29.256893+0000","last_unstale":"2024-04-20T17:32:41.394277+0000","last_undegraded":"2024-04-20T17:32:41.394277+0000","last_fullsized":"2024-04-20T17:32:41.394277+0000","mapping_epoch":543,"log_start":"580'8001","ondisk_log_start":"580'8001","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'17462","last_scrub_stamp":"2024-04-20T17:32:31.325390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:32:31.325390+0000","objects_scrubbed":2263,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:33:34.941208+0000","scrub_duration":566,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105775104,"num_objects":1614,"num_object_clones":0,"num_object_copies":3228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14980,"num_write_kb":531008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"607'15089","reported_seq":15341,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-04-20T17:32:44.300994+0000","last_change":"2024-04-20T17:32:29.992977+0000","last_active":"2024-04-20T17:32:44.300994+0000","last_peered":"2024-04-20T17:32:44.300994+0000","last_clean":"2024-04-20T17:32:44.300994+0000","last_became_active":"2024-04-20T17:29:29.597164+0000","last_became_peered":"2024-04-20T17:29:29.597164+0000","last_unstale":"2024-04-20T17:32:44.300994+0000","last_undegraded":"2024-04-20T17:32:44.300994+0000","last_fullsized":"2024-04-20T17:32:44.300994+0000","mapping_epoch":543,"log_start":"573'5000","ondisk_log_start":"573'5000","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'14168","last_scrub_stamp":"2024-04-20T17:32:29.992695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:32:29.992695+0000","objects_scrubbed":2420,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:33:30.001427+0000","scrub_duration":649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96534528,"num_objects":1473,"num_object_clones":0,"num_object_copies":2946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15089,"num_write_kb":529984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"607'14889","reported_seq":15137,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-04-20T17:32:44.308665+0000","last_change":"2024-04-20T17:32:31.009734+0000","last_active":"2024-04-20T17:32:44.308665+0000","last_peered":"2024-04-20T17:32:44.308665+0000","last_clean":"2024-04-20T17:32:44.308665+0000","last_became_active":"2024-04-20T17:29:29.256975+0000","last_became_peered":"2024-04-20T17:29:29.256975+0000","last_unstale":"2024-04-20T17:32:44.308665+0000","last_undegraded":"2024-04-20T17:32:44.308665+0000","last_fullsized":"2024-04-20T17:32:44.308665+0000","mapping_epoch":543,"log_start":"573'4800","ondisk_log_start":"573'4800","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'14068","last_scrub_stamp":"2024-04-20T17:32:31.009512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:32:31.009512+0000","objects_scrubbed":2297,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:33:37.681947+0000","scrub_duration":662,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95092736,"num_objects":1451,"num_object_clones":0,"num_object_copies":2902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14889,"num_write_kb":522880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"607'15023","reported_seq":15271,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-04-20T17:32:44.320917+0000","last_change":"2024-04-20T17:32:28.059569+0000","last_active":"2024-04-20T17:32:44.320917+0000","last_peered":"2024-04-20T17:32:44.320917+0000","last_clean":"2024-04-20T17:32:44.320917+0000","last_became_active":"2024-04-20T17:29:29.263022+0000","last_became_peered":"2024-04-20T17:29:29.263022+0000","last_unstale":"2024-04-20T17:32:44.320917+0000","last_undegraded":"2024-04-20T17:32:44.320917+0000","last_fullsized":"2024-04-20T17:32:44.320917+0000","mapping_epoch":543,"log_start":"573'5000","ondisk_log_start":"573'5000","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'14040","last_scrub_stamp":"2024-04-20T17:32:28.059229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:32:28.059229+0000","objects_scrubbed":2434,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:33:28.159376+0000","scrub_duration":694,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92733440,"num_objects":1415,"num_object_clones":0,"num_object_copies":2830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15023,"num_write_kb":526016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10],"acting":[0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"607'14800","reported_seq":15041,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-04-20T17:32:40.889326+0000","last_change":"2024-04-20T17:32:30.907332+0000","last_active":"2024-04-20T17:32:40.889326+0000","last_peered":"2024-04-20T17:32:40.889326+0000","last_clean":"2024-04-20T17:32:40.889326+0000","last_became_active":"2024-04-20T17:30:10.553600+0000","last_became_peered":"2024-04-20T17:30:10.553600+0000","last_unstale":"2024-04-20T17:32:40.889326+0000","last_undegraded":"2024-04-20T17:32:40.889326+0000","last_fullsized":"2024-04-20T17:32:40.889326+0000","mapping_epoch":561,"log_start":"572'4700","ondisk_log_start":"572'4700","created":543,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'14176","last_scrub_stamp":"2024-04-20T17:32:30.907250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:32:30.907250+0000","objects_scrubbed":2253,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:34:00.001201+0000","scrub_duration":400,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106037248,"num_objects":1618,"num_object_clones":0,"num_object_copies":3236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14800,"num_write_kb":525376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"607'14990","reported_seq":15232,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-04-20T17:32:44.300930+0000","last_change":"2024-04-20T17:32:43.455590+0000","last_active":"2024-04-20T17:32:44.300930+0000","last_peered":"2024-04-20T17:32:44.300930+0000","last_clean":"2024-04-20T17:32:44.300930+0000","last_became_active":"2024-04-20T17:29:29.597475+0000","last_became_peered":"2024-04-20T17:29:29.597475+0000","last_unstale":"2024-04-20T17:32:44.300930+0000","last_undegraded":"2024-04-20T17:32:44.300930+0000","last_fullsized":"2024-04-20T17:32:44.300930+0000","mapping_epoch":543,"log_start":"573'4900","ondisk_log_start":"573'4900","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"607'14941","last_scrub_stamp":"2024-04-20T17:32:43.455480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:32:43.455480+0000","objects_scrubbed":1472,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:34:10.125724+0000","scrub_duration":289,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92667904,"num_objects":1414,"num_object_clones":0,"num_object_copies":2828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14990,"num_write_kb":524928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"607'16238","reported_seq":16504,"reported_epoch":607,"state":"active+clean+scrubbing+deep","last_fresh":"2024-04-20T17:32:44.329076+0000","last_change":"2024-04-20T17:32:36.074813+0000","last_active":"2024-04-20T17:32:44.329076+0000","last_peered":"2024-04-20T17:32:44.329076+0000","last_clean":"2024-04-20T17:32:44.329076+0000","last_became_active":"2024-04-20T17:29:29.263290+0000","last_became_peered":"2024-04-20T17:29:29.263290+0000","last_unstale":"2024-04-20T17:32:44.329076+0000","last_undegraded":"2024-04-20T17:32:44.329076+0000","last_fullsized":"2024-04-20T17:32:44.329076+0000","mapping_epoch":543,"log_start":"576'6200","ondisk_log_start":"576'6200","created":543,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"580'7467","last_scrub_stamp":"2024-04-20T17:31:01.490094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:29:28.240412+0000","last_clean_scrub_stamp":"2024-04-20T17:31:01.490094+0000","objects_scrubbed":1223,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"deep scrubbing for 8s","scrub_duration":1450,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92602368,"num_objects":1413,"num_object_clones":0,"num_object_copies":2826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15015,"num_write_kb":525696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":766,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-04-20T17:32:40.381477+0000","last_change":"2024-04-20T17:31:54.780064+0000","last_active":"2024-04-20T17:32:40.381477+0000","last_peered":"2024-04-20T17:32:40.381477+0000","last_clean":"2024-04-20T17:32:40.381477+0000","last_became_active":"2024-04-20T17:23:24.109202+0000","last_became_peered":"2024-04-20T17:23:24.109202+0000","last_unstale":"2024-04-20T17:32:40.381477+0000","last_undegraded":"2024-04-20T17:32:40.381477+0000","last_fullsized":"2024-04-20T17:32:40.381477+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:31:54.779921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:31:54.779921+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:33:15.516753+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":1607139352,"num_objects":24524,"num_object_clones":0,"num_object_copies":49048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24524,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":238340,"num_write_kb":8411585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3217760256,"data_stored":3217752112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161088,"ondisk_log_size":161088,"up":32,"acting":32,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991557,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":850180,"kb_used_data":287292,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":92894972,"statfs":{"total":95995035648,"available":95124451328,"internally_reserved":0,"allocated":294187008,"data_stored":293795103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":576387524},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253196,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":710136,"kb_used_data":273584,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93035016,"statfs":{"total":95995035648,"available":95267856384,"internally_reserved":0,"allocated":280150016,"data_stored":279765735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":447019456},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612944,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":587452,"kb_used_data":209200,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":93157700,"statfs":{"total":95995035648,"available":95393484800,"internally_reserved":0,"allocated":214220800,"data_stored":213836539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":387316160},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809175,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":427708,"kb_used_data":198000,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":93317444,"statfs":{"total":95995035648,"available":95557062656,"internally_reserved":0,"allocated":202752000,"data_stored":202367739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":235207109},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":398,"seq":1709396983931,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":264128,"kb_used_data":95352,"kb_used_omap":6,"kb_used_meta":168761,"kb_avail":93481024,"statfs":{"total":95995035648,"available":95724568576,"internally_reserved":0,"allocated":97640448,"data_stored":97252635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6767,"internal_metadata":172811665},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":2297807503405,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575224,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":871164,"kb_used_data":410992,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":92873988,"statfs":{"total":95995035648,"available":95102963712,"internally_reserved":0,"allocated":420855808,"data_stored":420471547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738749,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":912568,"kb_used_data":480048,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":92832584,"statfs":{"total":95995035648,"available":95060566016,"internally_reserved":0,"allocated":491569152,"data_stored":491184891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394039,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":430396,"kb_used_data":313968,"kb_used_omap":202,"kb_used_meta":116213,"kb_avail":93314756,"statfs":{"total":95995035648,"available":95554310144,"internally_reserved":0,"allocated":321503232,"data_stored":321118971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":119002583},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653088,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":153016,"kb_used_data":98416,"kb_used_omap":4,"kb_used_meta":54587,"kb_avail":93592136,"statfs":{"total":95995035648,"available":95838347264,"internally_reserved":0,"allocated":100777984,"data_stored":100393723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":55897799},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986536,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":627264,"kb_used_data":317040,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":93117888,"statfs":{"total":95995035648,"available":95352717312,"internally_reserved":0,"allocated":324648960,"data_stored":324264699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790307,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":917056,"kb_used_data":524852,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92828096,"statfs":{"total":95995035648,"available":95055970304,"internally_reserved":0,"allocated":537448448,"data_stored":537060115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":477036544,"data_stored":477036544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":409534464,"data_stored":409534464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":314966016,"data_stored":314966016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98304000,"data_stored":98304000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":319553536,"data_stored":319553536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":530190336,"data_stored":530186264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":94830592,"data_stored":94830592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":196345856,"data_stored":196345856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":213450752,"data_stored":213450752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":277938176,"data_stored":277938176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":285609984,"data_stored":285605912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:32:45.934 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-20T17:32:46.019 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:32:46.334 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:32:46.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:46.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:32:46.439 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:32:46.486 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:46.563 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:46.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:46.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:46.841 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:32:46.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:46.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:32:46.948 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:32:46.992 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:47.019 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:32:47.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:47.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:32:47.128 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:32:47.157 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:47.179 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:32:47.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:47.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:32:47.282 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:32:47.326 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:47.356 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:32:47.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:47.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:32:47.460 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:32:47.527 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:47.563 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:32:47.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:47.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:32:47.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:47 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:32:47.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:47 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:32:47.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:47 smithi171 ceph-mon[44189]: pgmap v1598: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.4 GiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 932 op/s 2024-04-20T17:32:47.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:47 smithi171 ceph-mon[44189]: osdmap e609: 12 total, 12 up, 11 in 2024-04-20T17:32:47.667 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:32:47.701 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:47.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:47 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:32:47.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:47 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:32:47.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:47 smithi005 ceph-mon[39562]: pgmap v1598: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.4 GiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 932 op/s 2024-04-20T17:32:47.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:47 smithi005 ceph-mon[39562]: osdmap e609: 12 total, 12 up, 11 in 2024-04-20T17:32:47.724 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:32:47.729 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:47 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:32:47.729 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:47 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:32:47.729 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:47 smithi106 ceph-mon[44423]: pgmap v1598: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.4 GiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 932 op/s 2024-04-20T17:32:47.730 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:47 smithi106 ceph-mon[44423]: osdmap e609: 12 total, 12 up, 11 in 2024-04-20T17:32:47.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:47.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:32:47.832 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:32:47.902 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:47.903 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:32:47.903 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:32:47.935 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:32:47.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:47.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:32:48.049 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:32:48.108 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:48.152 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:32:48.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:48.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:32:48.256 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:32:48.296 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:48.343 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:32:48.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:48 smithi171 ceph-mon[44189]: 5.4 deep-scrub ok 2024-04-20T17:32:48.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:48 smithi171 ceph-mon[44189]: 5.a deep-scrub ok 2024-04-20T17:32:48.628 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-20T17:32:48.662 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:32:48.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:48.665 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:32:48.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:48 smithi106 ceph-mon[44423]: 5.4 deep-scrub ok 2024-04-20T17:32:48.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:48 smithi106 ceph-mon[44423]: 5.a deep-scrub ok 2024-04-20T17:32:48.766 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:32:48.773 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:32:48.795 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:48.820 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:32:48.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:48.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:32:48.872 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:32:48.906 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:48 smithi005 ceph-mon[39562]: 5.4 deep-scrub ok 2024-04-20T17:32:48.906 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:48 smithi005 ceph-mon[39562]: 5.a deep-scrub ok 2024-04-20T17:32:48.924 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:32:48.959 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:48.979 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:32:48.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:48.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:32:49.030 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:32:49.083 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:32:49.122 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:49.149 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:32:49.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:49.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:32:49.253 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:32:49.300 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:49.559 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:49 smithi171 ceph-mon[44189]: pgmap v1600: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.4 GiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 767 op/s 2024-04-20T17:32:49.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:49 smithi005 ceph-mon[39562]: pgmap v1600: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.4 GiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 767 op/s 2024-04-20T17:32:49.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:49 smithi106 ceph-mon[44423]: pgmap v1600: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 1.4 GiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 767 op/s 2024-04-20T17:32:50.974 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:32:50.975 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:32:51.659 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:51 smithi171 ceph-mon[44189]: pgmap v1601: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.2 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 781 op/s 2024-04-20T17:32:51.701 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-20T17:32:51.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:51 smithi106 ceph-mon[44423]: pgmap v1601: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.2 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 781 op/s 2024-04-20T17:32:51.770 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:32:51.771 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:51 smithi005 ceph-mon[39562]: pgmap v1601: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.2 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 781 op/s 2024-04-20T17:32:51.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:51.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:32:51.875 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:32:51.908 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:51.930 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:32:51.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:51.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:32:52.034 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:32:52.051 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:32:52.073 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:52.340 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:32:52.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:52.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:32:52.447 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:32:52.488 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:52.881 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:32:52.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:52.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:32:52.890 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:32:52.985 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:32:53.017 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:53.042 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:32:53.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:53.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:32:53.146 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:32:53.183 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:53.340 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:32:53.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:53.345 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:32:53.446 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:32:53.501 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:53.692 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:53 smithi171 ceph-mon[44189]: pgmap v1602: 17 pgs: 17 active+clean; 1.1 GiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1k op/s 2024-04-20T17:32:53.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:53 smithi106 ceph-mon[44423]: pgmap v1602: 17 pgs: 17 active+clean; 1.1 GiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1k op/s 2024-04-20T17:32:53.868 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:32:53.870 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:53 smithi005 ceph-mon[39562]: pgmap v1602: 17 pgs: 17 active+clean; 1.1 GiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1k op/s 2024-04-20T17:32:53.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:53.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:32:53.972 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:32:54.008 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:54.029 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:32:54.030 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:32:54.030 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:32:54.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:54.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:32:54.134 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:32:54.185 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:54.189 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:32:54.223 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:32:54.260 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:32:54.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:54.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:32:54.274 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:32:54.307 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:32:54.339 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:32:54.368 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:32:54.369 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:32:54.408 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:32:54.415 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:54.456 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:32:54.481 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:32:54.522 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:32:54.556 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:32:54.583 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:32:54.584 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:32:54.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:54.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:32:54.604 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:32:54.604 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:32:54.604 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:32:54.604 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:32:54.604 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:32:54.604 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:32:54.604 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:32:54.604 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:32:54.604 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:32:54.604 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:32:54.604 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:32:54.604 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:32:54.691 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:32:54.707 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:54 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3802656149' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:32:54.707 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:54.741 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-20T17:32:54.750 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:54.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:54 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3802656149' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:32:54.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:55.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:54 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3802656149' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:32:55.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:55.131 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:32:55.152 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:32:55.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:55.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:32:55.265 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:32:55.302 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:55.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:55.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:55.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:55.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:55.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:55 smithi171 ceph-mon[44189]: pgmap v1603: 17 pgs: 17 active+clean; 961 MiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 921 op/s 2024-04-20T17:32:55.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:55.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:55.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:55.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:55.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:55.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:55.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:55.937 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:55.937 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:55.937 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:55.937 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:55.937 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:55 smithi005 ceph-mon[39562]: pgmap v1603: 17 pgs: 17 active+clean; 961 MiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 921 op/s 2024-04-20T17:32:55.937 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:55.937 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:55.937 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:55.937 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:55.937 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:55.937 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:55.937 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:55.957 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:32:55.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:55.964 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:32:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:55 smithi106 ceph-mon[44423]: pgmap v1603: 17 pgs: 17 active+clean; 961 MiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 921 op/s 2024-04-20T17:32:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:56.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:56.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:32:56.065 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:32:56.103 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:56.133 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:32:56.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:56.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:32:56.242 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:32:56.288 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:56.491 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:32:56.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:56.494 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:32:56.550 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:32:56.595 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:32:56.633 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:56.788 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:32:56.798 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:56 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:32:56.799 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:56 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:32:56.799 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:56 smithi005 ceph-mon[39562]: osdmap e610: 12 total, 12 up, 11 in 2024-04-20T17:32:56.799 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:56 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1678772567' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:32:56.799 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:56 smithi005 ceph-mon[39562]: from='client.26704 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:32:56.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:56.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:32:56.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:56 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:32:56.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:56 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:32:56.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:56 smithi171 ceph-mon[44189]: osdmap e610: 12 total, 12 up, 11 in 2024-04-20T17:32:56.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:56 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1678772567' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:32:56.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:56 smithi171 ceph-mon[44189]: from='client.26704 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:32:56.900 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:32:56.932 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:56.960 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:32:56.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:56.964 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:32:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:56 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:32:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:56 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:32:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:56 smithi106 ceph-mon[44423]: osdmap e610: 12 total, 12 up, 11 in 2024-04-20T17:32:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:56 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1678772567' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:32:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:56 smithi106 ceph-mon[44423]: from='client.26704 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:32:57.065 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:32:57.099 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:57.309 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:32:57.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:57.316 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:32:57.417 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:32:57.449 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:57.472 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:32:57.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:57.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:32:57.540 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:57 smithi171 ceph-mon[44189]: pgmap v1605: 17 pgs: 17 active+clean; 755 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-20T17:32:57.540 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:57 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:32:57.579 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:32:57.630 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:57.640 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:32:57.641 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:32:57.641 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-04-20T17:32:57.641 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-04-20T17:32:57.641 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:32:57.659 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:32:57.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:57.664 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:32:57.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:57 smithi005 ceph-mon[39562]: pgmap v1605: 17 pgs: 17 active+clean; 755 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-20T17:32:57.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:57 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:32:57.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:57 smithi005 ceph-mon[39562]: from='client.26704 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:32:57.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:57 smithi005 ceph-mon[39562]: osdmap e611: 12 total, 12 up, 11 in 2024-04-20T17:32:57.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:57 smithi106 ceph-mon[44423]: pgmap v1605: 17 pgs: 17 active+clean; 755 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-20T17:32:57.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:57 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:32:57.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:57 smithi106 ceph-mon[44423]: from='client.26704 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:32:57.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:57 smithi106 ceph-mon[44423]: osdmap e611: 12 total, 12 up, 11 in 2024-04-20T17:32:57.764 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:32:57.794 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:57 smithi171 ceph-mon[44189]: from='client.26704 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:32:57.794 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:57 smithi171 ceph-mon[44189]: osdmap e611: 12 total, 12 up, 11 in 2024-04-20T17:32:57.798 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:57.859 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:32:57.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:57.867 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:32:57.968 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:32:57.972 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:32:58.007 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:58.083 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:32:58.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:58.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:32:58.195 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:32:58.229 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:58.337 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:32:58.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:58.341 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:32:58.376 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-20T17:32:58.441 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:32:58.526 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:32:58.547 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:58.721 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:32:58.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:58.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:32:58.784 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:32:58.834 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:32:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:58 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:32:58.862 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:58 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:32:58.871 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:59.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:58 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:32:59.082 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:32:59.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:59.087 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:32:59.188 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:32:59.219 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:59.238 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:32:59.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:59.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:32:59.342 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:32:59.395 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:59.424 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:32:59.441 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:32:59.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:59.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:32:59.547 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:32:59.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:32:59 smithi171 ceph-mon[44189]: pgmap v1607: 17 pgs: 17 active+clean; 685 MiB data, 4.9 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:32:59.584 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:32:59.610 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:32:59.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:32:59.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:32:59.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:32:59 smithi005 ceph-mon[39562]: pgmap v1607: 17 pgs: 17 active+clean; 685 MiB data, 4.9 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:32:59.714 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:32:59.776 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:00.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:32:59 smithi106 ceph-mon[44423]: pgmap v1607: 17 pgs: 17 active+clean; 685 MiB data, 4.9 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:33:00.020 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:33:00.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:00.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:33:00.094 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:33:00.132 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:33:00.165 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:00.191 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:33:00.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:00.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:33:00.295 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:33:00.332 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:00.366 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:33:00.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:00.370 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:33:00.471 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:33:00.505 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:00.564 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:33:00.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:00.574 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:33:00.675 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:33:00.723 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:00.739 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:33:00.739 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:33:00.739 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:33:00.740 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:33:00.740 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.740 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162338 4.3 4:c74ec0af:::benchmark_data_smithi171_122772_object162337:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:33:00.740 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:11.924493+0000", 2024-04-20T17:33:00.740 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 408.81240621000001, 2024-04-20T17:33:00.740 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.209723929, 2024-04-20T17:33:00.740 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:33:00.740 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:33:00.740 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:00.740 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:33:00.740 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:33:00.740 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:33:00.740 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162338 2024-04-20T17:33:00.740 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.741 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:33:00.741 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.741 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:33:00.741 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.924493+0000", 2024-04-20T17:33:00.741 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.741 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.741 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.741 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:33:00.741 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.924493+0000", 2024-04-20T17:33:00.741 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.741 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.741 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.741 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:33:00.741 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.924495+0000", 2024-04-20T17:33:00.742 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.575e-06 2024-04-20T17:33:00.742 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.742 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.742 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:33:00.742 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.924515+0000", 2024-04-20T17:33:00.742 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0489e-05 2024-04-20T17:33:00.742 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.742 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.742 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:33:00.742 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.924526+0000", 2024-04-20T17:33:00.742 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1309e-05 2024-04-20T17:33:00.742 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.742 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.742 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:33:00.743 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.924534+0000", 2024-04-20T17:33:00.743 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.25e-06 2024-04-20T17:33:00.743 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.743 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.743 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:33:00.743 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.926564+0000", 2024-04-20T17:33:00.743 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0020306550000000001 2024-04-20T17:33:00.743 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.743 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.743 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:33:00.743 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.926712+0000", 2024-04-20T17:33:00.743 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014766700000000001 2024-04-20T17:33:00.743 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.743 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.743 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:33:00.744 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.926862+0000", 2024-04-20T17:33:00.744 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014963500000000001 2024-04-20T17:33:00.744 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.744 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.744 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:33:00.744 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.932381+0000", 2024-04-20T17:33:00.744 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0055196949999999998 2024-04-20T17:33:00.744 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.744 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.744 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:33:00.744 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.134102+0000", 2024-04-20T17:33:00.744 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20172087 2024-04-20T17:33:00.744 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.744 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.744 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:33:00.745 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.134166+0000", 2024-04-20T17:33:00.745 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4243000000000001e-05 2024-04-20T17:33:00.745 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.745 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.745 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:33:00.745 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.134217+0000", 2024-04-20T17:33:00.745 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0541000000000003e-05 2024-04-20T17:33:00.745 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.745 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:33:00.745 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.745 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.745 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.745 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162350 4.8 4:193a143c:::benchmark_data_smithi171_122772_object162349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:33:00.745 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:11.931709+0000", 2024-04-20T17:33:00.746 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 408.80519014700002, 2024-04-20T17:33:00.746 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.22858086999999999, 2024-04-20T17:33:00.746 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:33:00.746 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:33:00.746 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:00.746 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:33:00.746 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:33:00.746 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:33:00.746 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162350 2024-04-20T17:33:00.746 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.746 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:33:00.746 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.747 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:33:00.747 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.931709+0000", 2024-04-20T17:33:00.747 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.747 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.747 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.747 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:33:00.747 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.931709+0000", 2024-04-20T17:33:00.747 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.747 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.747 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.747 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:33:00.747 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.931712+0000", 2024-04-20T17:33:00.748 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.92e-06 2024-04-20T17:33:00.748 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.748 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.748 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:33:00.748 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.931737+0000", 2024-04-20T17:33:00.748 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5106999999999999e-05 2024-04-20T17:33:00.748 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.748 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.748 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:33:00.748 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.931752+0000", 2024-04-20T17:33:00.749 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4891e-05 2024-04-20T17:33:00.749 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.749 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.749 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:33:00.749 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.931761+0000", 2024-04-20T17:33:00.749 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6340000000000007e-06 2024-04-20T17:33:00.749 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.749 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.749 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:33:00.749 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.931824+0000", 2024-04-20T17:33:00.749 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3019000000000001e-05 2024-04-20T17:33:00.749 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.749 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.749 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:33:00.749 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.931959+0000", 2024-04-20T17:33:00.750 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013509700000000001 2024-04-20T17:33:00.750 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.750 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.750 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:33:00.750 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.932081+0000", 2024-04-20T17:33:00.750 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000121919 2024-04-20T17:33:00.750 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.750 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.750 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:33:00.750 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937096+0000", 2024-04-20T17:33:00.750 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0050154980000000002 2024-04-20T17:33:00.750 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.750 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.750 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:33:00.751 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160167+0000", 2024-04-20T17:33:00.751 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.223070565 2024-04-20T17:33:00.751 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.751 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.751 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:33:00.751 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160239+0000", 2024-04-20T17:33:00.751 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2729000000000001e-05 2024-04-20T17:33:00.751 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.751 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.751 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:33:00.751 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160290+0000", 2024-04-20T17:33:00.751 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0491000000000002e-05 2024-04-20T17:33:00.751 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.751 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:33:00.751 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.752 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.752 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.752 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162353 4.8 4:1973aaac:::benchmark_data_smithi171_122772_object162352:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:33:00.752 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:11.937307+0000", 2024-04-20T17:33:00.752 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 408.79959200799999, 2024-04-20T17:33:00.752 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.22316846200000001, 2024-04-20T17:33:00.752 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:33:00.752 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:33:00.752 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:00.752 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:33:00.752 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:33:00.752 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:33:00.752 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162353 2024-04-20T17:33:00.752 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.752 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:33:00.753 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.753 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:33:00.753 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937307+0000", 2024-04-20T17:33:00.753 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.753 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.753 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.753 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:33:00.753 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937307+0000", 2024-04-20T17:33:00.753 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.753 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.753 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.753 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:33:00.753 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937308+0000", 2024-04-20T17:33:00.753 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0210000000000001e-06 2024-04-20T17:33:00.754 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.754 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.754 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:33:00.754 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937330+0000", 2024-04-20T17:33:00.754 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1678000000000001e-05 2024-04-20T17:33:00.754 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.754 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.754 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:33:00.754 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937337+0000", 2024-04-20T17:33:00.754 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4340000000000003e-06 2024-04-20T17:33:00.754 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.754 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.754 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:33:00.754 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937343+0000", 2024-04-20T17:33:00.755 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8599999999999998e-06 2024-04-20T17:33:00.755 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.755 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.755 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:33:00.755 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937396+0000", 2024-04-20T17:33:00.755 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3124999999999997e-05 2024-04-20T17:33:00.755 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.755 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.755 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:33:00.755 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937544+0000", 2024-04-20T17:33:00.755 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000147398 2024-04-20T17:33:00.755 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.755 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.755 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:33:00.755 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937698+0000", 2024-04-20T17:33:00.756 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015431200000000001 2024-04-20T17:33:00.756 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.756 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.756 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:33:00.756 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.956740+0000", 2024-04-20T17:33:00.756 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019042416999999999 2024-04-20T17:33:00.756 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.756 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.756 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:33:00.756 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160359+0000", 2024-04-20T17:33:00.756 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20361854900000001 2024-04-20T17:33:00.756 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.756 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.756 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:33:00.756 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160419+0000", 2024-04-20T17:33:00.757 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9735999999999999e-05 2024-04-20T17:33:00.757 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.757 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.757 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:33:00.757 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160476+0000", 2024-04-20T17:33:00.757 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6932000000000003e-05 2024-04-20T17:33:00.757 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.757 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:33:00.757 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.757 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.757 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.757 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162360 4.8 4:12f91459:::benchmark_data_smithi171_122772_object162359:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:33:00.757 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:11.943537+0000", 2024-04-20T17:33:00.758 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 408.79336215900003, 2024-04-20T17:33:00.758 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.21790537500000001, 2024-04-20T17:33:00.758 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:33:00.758 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:33:00.758 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:00.758 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:33:00.758 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:33:00.758 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:33:00.758 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162360 2024-04-20T17:33:00.759 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.759 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:33:00.759 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.759 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:33:00.759 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.943537+0000", 2024-04-20T17:33:00.759 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.759 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.759 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.759 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:33:00.759 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.943537+0000", 2024-04-20T17:33:00.759 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.759 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.759 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.759 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:33:00.760 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.943539+0000", 2024-04-20T17:33:00.760 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.826e-06 2024-04-20T17:33:00.760 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.760 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.760 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:33:00.760 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.943561+0000", 2024-04-20T17:33:00.760 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2354999999999999e-05 2024-04-20T17:33:00.760 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.760 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.760 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:33:00.760 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.943573+0000", 2024-04-20T17:33:00.760 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1841999999999999e-05 2024-04-20T17:33:00.760 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.760 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.761 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:33:00.761 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.943581+0000", 2024-04-20T17:33:00.761 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1769999999999998e-06 2024-04-20T17:33:00.761 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.761 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.761 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:33:00.761 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.943637+0000", 2024-04-20T17:33:00.761 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5429000000000001e-05 2024-04-20T17:33:00.761 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.761 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.761 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:33:00.761 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.943771+0000", 2024-04-20T17:33:00.761 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000134027 2024-04-20T17:33:00.761 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.762 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.762 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:33:00.762 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.943930+0000", 2024-04-20T17:33:00.762 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015902900000000001 2024-04-20T17:33:00.762 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.762 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.762 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:33:00.762 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.956748+0000", 2024-04-20T17:33:00.762 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.012818016 2024-04-20T17:33:00.762 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.762 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.762 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:33:00.762 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.161353+0000", 2024-04-20T17:33:00.762 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20460482199999999 2024-04-20T17:33:00.763 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.763 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.763 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:33:00.763 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.161404+0000", 2024-04-20T17:33:00.763 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1439e-05 2024-04-20T17:33:00.763 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.763 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.763 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:33:00.763 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.161442+0000", 2024-04-20T17:33:00.763 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8413e-05 2024-04-20T17:33:00.763 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.763 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:33:00.763 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.763 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.763 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.764 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162361 4.8 4:19c34efa:::benchmark_data_smithi171_122772_object162360:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:33:00.764 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:11.946327+0000", 2024-04-20T17:33:00.764 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 408.79057218499997, 2024-04-20T17:33:00.764 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.21524243400000001, 2024-04-20T17:33:00.764 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:33:00.764 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:33:00.764 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:00.764 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:33:00.764 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:33:00.764 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:33:00.764 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162361 2024-04-20T17:33:00.764 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.764 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:33:00.764 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.764 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:33:00.765 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.946327+0000", 2024-04-20T17:33:00.765 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.765 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.765 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.765 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:33:00.765 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.946327+0000", 2024-04-20T17:33:00.765 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.765 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.765 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.765 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:33:00.765 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.946329+0000", 2024-04-20T17:33:00.765 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8470000000000001e-06 2024-04-20T17:33:00.765 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.765 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.765 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:33:00.766 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.946353+0000", 2024-04-20T17:33:00.766 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4491999999999999e-05 2024-04-20T17:33:00.766 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.766 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.766 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:33:00.766 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.946365+0000", 2024-04-20T17:33:00.766 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1688000000000001e-05 2024-04-20T17:33:00.766 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.766 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.766 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:33:00.766 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.946373+0000", 2024-04-20T17:33:00.766 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9049999999999997e-06 2024-04-20T17:33:00.766 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.766 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.766 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:33:00.767 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.947724+0000", 2024-04-20T17:33:00.767 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0013511409999999999 2024-04-20T17:33:00.767 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.767 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.767 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:33:00.767 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.947841+0000", 2024-04-20T17:33:00.767 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000116839 2024-04-20T17:33:00.767 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.767 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.767 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:33:00.767 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.947979+0000", 2024-04-20T17:33:00.767 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013786500000000001 2024-04-20T17:33:00.767 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.767 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.768 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:33:00.768 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.956751+0000", 2024-04-20T17:33:00.768 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0087720309999999996 2024-04-20T17:33:00.768 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.768 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.768 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:33:00.768 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.161495+0000", 2024-04-20T17:33:00.768 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20474413799999999 2024-04-20T17:33:00.768 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.768 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.768 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:33:00.768 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.161533+0000", 2024-04-20T17:33:00.768 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.82e-05 2024-04-20T17:33:00.768 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.769 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.769 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:33:00.769 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.161569+0000", 2024-04-20T17:33:00.769 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6288000000000002e-05 2024-04-20T17:33:00.769 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.769 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:33:00.769 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.769 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.769 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.769 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162362 4.8 4:15c10f34:::benchmark_data_smithi171_122772_object162361:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:33:00.769 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:11.947126+0000", 2024-04-20T17:33:00.769 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 408.78977325699998, 2024-04-20T17:33:00.769 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.224067406, 2024-04-20T17:33:00.769 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:33:00.770 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:33:00.770 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:00.770 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:33:00.770 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:33:00.770 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:33:00.770 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162362 2024-04-20T17:33:00.770 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.770 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:33:00.770 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.770 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:33:00.770 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.947126+0000", 2024-04-20T17:33:00.770 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.770 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.770 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.770 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:33:00.771 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.947126+0000", 2024-04-20T17:33:00.771 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.771 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.771 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.771 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:33:00.771 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.947127+0000", 2024-04-20T17:33:00.771 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5009999999999999e-06 2024-04-20T17:33:00.771 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.771 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.771 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:33:00.771 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.947149+0000", 2024-04-20T17:33:00.771 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1708e-05 2024-04-20T17:33:00.771 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.771 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.772 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:33:00.772 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.947159+0000", 2024-04-20T17:33:00.772 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0244999999999999e-05 2024-04-20T17:33:00.772 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.772 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.772 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:33:00.772 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.947166+0000", 2024-04-20T17:33:00.772 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4110000000000001e-06 2024-04-20T17:33:00.772 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.772 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.772 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:33:00.772 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.952667+0000", 2024-04-20T17:33:00.772 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0055010679999999996 2024-04-20T17:33:00.772 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.772 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.773 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:33:00.773 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.952814+0000", 2024-04-20T17:33:00.773 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014753499999999999 2024-04-20T17:33:00.773 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.773 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.773 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:33:00.773 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.952948+0000", 2024-04-20T17:33:00.773 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013382499999999999 2024-04-20T17:33:00.773 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.773 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.773 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:33:00.773 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.956887+0000", 2024-04-20T17:33:00.773 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0039385430000000001 2024-04-20T17:33:00.773 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.774 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.774 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:33:00.774 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.171085+0000", 2024-04-20T17:33:00.774 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.214198641 2024-04-20T17:33:00.774 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.774 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.774 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:33:00.774 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.171143+0000", 2024-04-20T17:33:00.774 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7720000000000003e-05 2024-04-20T17:33:00.774 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.774 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.774 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:33:00.774 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.171193+0000", 2024-04-20T17:33:00.774 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0209e-05 2024-04-20T17:33:00.775 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.775 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:33:00.775 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.775 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.775 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.775 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162374 4.3 4:c9021f98:::benchmark_data_smithi171_122772_object162373:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:33:00.775 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:11.952990+0000", 2024-04-20T17:33:00.775 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 408.78390881299998, 2024-04-20T17:33:00.775 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19610433199999999, 2024-04-20T17:33:00.775 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:33:00.775 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:33:00.775 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:00.775 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:33:00.775 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:33:00.775 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:33:00.776 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162374 2024-04-20T17:33:00.776 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.776 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:33:00.776 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.776 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:33:00.776 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.952990+0000", 2024-04-20T17:33:00.776 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.776 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.776 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.776 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:33:00.776 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.952990+0000", 2024-04-20T17:33:00.776 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.776 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.776 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.776 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:33:00.777 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.952992+0000", 2024-04-20T17:33:00.777 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7060000000000001e-06 2024-04-20T17:33:00.777 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.777 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.777 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:33:00.777 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.953016+0000", 2024-04-20T17:33:00.777 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4002000000000002e-05 2024-04-20T17:33:00.777 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.777 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.777 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:33:00.777 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.953026+0000", 2024-04-20T17:33:00.777 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0259e-05 2024-04-20T17:33:00.778 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.778 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.778 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:33:00.778 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.953034+0000", 2024-04-20T17:33:00.778 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7219999999999999e-06 2024-04-20T17:33:00.778 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.778 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.778 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:33:00.778 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.953097+0000", 2024-04-20T17:33:00.778 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2479999999999998e-05 2024-04-20T17:33:00.778 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.778 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.778 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:33:00.778 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.953175+0000", 2024-04-20T17:33:00.778 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8559000000000005e-05 2024-04-20T17:33:00.779 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.779 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.779 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:33:00.779 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.953274+0000", 2024-04-20T17:33:00.779 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9059000000000001e-05 2024-04-20T17:33:00.779 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.779 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.779 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:33:00.779 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.957017+0000", 2024-04-20T17:33:00.779 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.003742483 2024-04-20T17:33:00.779 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.779 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.779 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:33:00.779 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.149017+0000", 2024-04-20T17:33:00.779 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19200002899999999 2024-04-20T17:33:00.780 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.780 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.780 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:33:00.780 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.149053+0000", 2024-04-20T17:33:00.780 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5856999999999999e-05 2024-04-20T17:33:00.780 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.780 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.780 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:33:00.780 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.149095+0000", 2024-04-20T17:33:00.780 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2175999999999997e-05 2024-04-20T17:33:00.780 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.780 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:33:00.780 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.780 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.781 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.781 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162397 4.3 4:c9a84e1e:::benchmark_data_smithi171_122772_object162396:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:33:00.781 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:11.983263+0000", 2024-04-20T17:33:00.781 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 408.75363619000001, 2024-04-20T17:33:00.781 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.177048224, 2024-04-20T17:33:00.781 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:33:00.781 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:33:00.781 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:00.781 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:33:00.781 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:33:00.781 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:33:00.781 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162397 2024-04-20T17:33:00.781 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.781 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:33:00.782 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.782 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:33:00.782 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.983263+0000", 2024-04-20T17:33:00.782 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.782 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.782 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.782 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:33:00.782 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.983263+0000", 2024-04-20T17:33:00.782 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.782 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.782 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.782 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:33:00.782 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.983266+0000", 2024-04-20T17:33:00.782 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.644e-06 2024-04-20T17:33:00.782 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.783 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.783 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:33:00.783 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.983288+0000", 2024-04-20T17:33:00.783 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2266999999999999e-05 2024-04-20T17:33:00.783 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.783 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.783 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:33:00.783 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.983302+0000", 2024-04-20T17:33:00.783 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4412e-05 2024-04-20T17:33:00.783 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.783 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.783 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:33:00.783 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.983310+0000", 2024-04-20T17:33:00.783 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7780000000000007e-06 2024-04-20T17:33:00.784 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.784 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.784 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:33:00.784 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.983374+0000", 2024-04-20T17:33:00.784 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3399999999999996e-05 2024-04-20T17:33:00.784 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.784 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.784 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:33:00.784 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.983521+0000", 2024-04-20T17:33:00.784 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014708399999999999 2024-04-20T17:33:00.784 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.784 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.784 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:33:00.784 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.983686+0000", 2024-04-20T17:33:00.784 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00016503000000000001 2024-04-20T17:33:00.785 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.785 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.785 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:33:00.785 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.988826+0000", 2024-04-20T17:33:00.785 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0051401069999999997 2024-04-20T17:33:00.785 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.785 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.785 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:33:00.785 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160218+0000", 2024-04-20T17:33:00.785 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.17139236599999999 2024-04-20T17:33:00.785 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.785 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.785 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:33:00.785 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160274+0000", 2024-04-20T17:33:00.786 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5739999999999998e-05 2024-04-20T17:33:00.786 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.786 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.786 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:33:00.786 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160311+0000", 2024-04-20T17:33:00.786 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7395999999999999e-05 2024-04-20T17:33:00.786 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.786 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:33:00.786 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.786 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.786 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.786 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:250984 4.1 4:8e67478e:::benchmark_data_smithi171_122772_object80792:head [delete] snapc 0=[] ondisk+write+known_if_redirected e518)", 2024-04-20T17:33:00.786 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:27:50.635874+0000", 2024-04-20T17:33:00.786 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 310.10102546899998, 2024-04-20T17:33:00.787 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.23012532499999999, 2024-04-20T17:33:00.787 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:33:00.787 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:33:00.787 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:00.787 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:33:00.787 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:33:00.787 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:33:00.787 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 250984 2024-04-20T17:33:00.787 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.787 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:33:00.787 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.787 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:33:00.787 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.635874+0000", 2024-04-20T17:33:00.787 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.787 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.788 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.788 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:33:00.788 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.635874+0000", 2024-04-20T17:33:00.788 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.788 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.788 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.788 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:33:00.788 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.635875+0000", 2024-04-20T17:33:00.788 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6600000000000005e-07 2024-04-20T17:33:00.788 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.788 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.788 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:33:00.788 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.635878+0000", 2024-04-20T17:33:00.788 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3450000000000002e-06 2024-04-20T17:33:00.788 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.789 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.789 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:33:00.789 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.635884+0000", 2024-04-20T17:33:00.789 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1059999999999996e-06 2024-04-20T17:33:00.789 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.789 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.789 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:33:00.789 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.635890+0000", 2024-04-20T17:33:00.789 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.553e-06 2024-04-20T17:33:00.789 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.789 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.789 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:33:00.789 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.640085+0000", 2024-04-20T17:33:00.789 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.004194985 2024-04-20T17:33:00.789 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.790 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.790 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:33:00.790 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.640223+0000", 2024-04-20T17:33:00.790 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000138348 2024-04-20T17:33:00.790 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.790 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.790 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:33:00.790 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.640304+0000", 2024-04-20T17:33:00.790 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1385999999999996e-05 2024-04-20T17:33:00.790 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.790 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.790 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:33:00.790 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.646374+0000", 2024-04-20T17:33:00.790 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0060697379999999999 2024-04-20T17:33:00.790 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.791 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.791 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:33:00.791 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.865929+0000", 2024-04-20T17:33:00.791 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.219554844 2024-04-20T17:33:00.791 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.791 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.791 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:33:00.791 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.865960+0000", 2024-04-20T17:33:00.791 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1092e-05 2024-04-20T17:33:00.791 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.791 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.791 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:33:00.791 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.865999+0000", 2024-04-20T17:33:00.791 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9062e-05 2024-04-20T17:33:00.791 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.792 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:33:00.792 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.792 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.792 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.792 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:251049 4.1 4:80d2b251:::benchmark_data_smithi171_122772_object80857:head [delete] snapc 0=[] ondisk+write+known_if_redirected e518)", 2024-04-20T17:33:00.792 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:27:50.690715+0000", 2024-04-20T17:33:00.792 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 310.04618412399998, 2024-04-20T17:33:00.792 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.17539388, 2024-04-20T17:33:00.792 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:33:00.792 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:33:00.792 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:00.792 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:33:00.792 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:33:00.792 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:33:00.792 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 251049 2024-04-20T17:33:00.793 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.793 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:33:00.793 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.793 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:33:00.793 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.690715+0000", 2024-04-20T17:33:00.793 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.793 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.793 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.793 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:33:00.793 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.690715+0000", 2024-04-20T17:33:00.793 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.793 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.793 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.793 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:33:00.793 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.690718+0000", 2024-04-20T17:33:00.794 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5500000000000001e-06 2024-04-20T17:33:00.794 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.794 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.794 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:33:00.794 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.690724+0000", 2024-04-20T17:33:00.794 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6640000000000001e-06 2024-04-20T17:33:00.794 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.794 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.794 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:33:00.794 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.690737+0000", 2024-04-20T17:33:00.794 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3164e-05 2024-04-20T17:33:00.794 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.794 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.794 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:33:00.795 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.690746+0000", 2024-04-20T17:33:00.795 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.371e-06 2024-04-20T17:33:00.795 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.795 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.795 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:33:00.795 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.690817+0000", 2024-04-20T17:33:00.795 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0928999999999998e-05 2024-04-20T17:33:00.795 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.795 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.795 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:33:00.795 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.690943+0000", 2024-04-20T17:33:00.795 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012669600000000001 2024-04-20T17:33:00.795 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.795 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.795 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:33:00.796 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.691075+0000", 2024-04-20T17:33:00.796 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013154 2024-04-20T17:33:00.796 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.796 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.796 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:33:00.796 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.695316+0000", 2024-04-20T17:33:00.796 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0042411999999999997 2024-04-20T17:33:00.796 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.796 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.796 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:33:00.796 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.866028+0000", 2024-04-20T17:33:00.796 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.17071159899999999 2024-04-20T17:33:00.796 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.796 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.796 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:33:00.797 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.866060+0000", 2024-04-20T17:33:00.797 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2076999999999998e-05 2024-04-20T17:33:00.797 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.797 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.797 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:33:00.797 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.866109+0000", 2024-04-20T17:33:00.797 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9089999999999999e-05 2024-04-20T17:33:00.797 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.797 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:33:00.797 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.797 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.797 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.797 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:280492 4.8 4:1927ac75:::benchmark_data_smithi171_122772_object110300:head [delete] snapc 0=[] ondisk+write+known_if_redirected e521)", 2024-04-20T17:33:00.797 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:15.885627+0000", 2024-04-20T17:33:00.798 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 284.85127182500003, 2024-04-20T17:33:00.798 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.195642228, 2024-04-20T17:33:00.798 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:33:00.798 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:33:00.798 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:00.798 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:33:00.798 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:33:00.798 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:33:00.798 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 280492 2024-04-20T17:33:00.798 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.798 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:33:00.798 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.798 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:33:00.798 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.885627+0000", 2024-04-20T17:33:00.799 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.799 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.799 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.799 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:33:00.799 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.885627+0000", 2024-04-20T17:33:00.799 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.799 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.799 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.799 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:33:00.799 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.885629+0000", 2024-04-20T17:33:00.799 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7290000000000001e-06 2024-04-20T17:33:00.799 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.799 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.799 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:33:00.799 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.885634+0000", 2024-04-20T17:33:00.800 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5120000000000002e-06 2024-04-20T17:33:00.800 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.800 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.800 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:33:00.800 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.885642+0000", 2024-04-20T17:33:00.800 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8900000000000007e-06 2024-04-20T17:33:00.800 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.800 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.800 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:33:00.800 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.885647+0000", 2024-04-20T17:33:00.800 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2939999999999996e-06 2024-04-20T17:33:00.800 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.800 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.800 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:33:00.800 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.996163+0000", 2024-04-20T17:33:00.801 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.110515739 2024-04-20T17:33:00.801 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.801 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.801 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:33:00.801 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.996255+0000", 2024-04-20T17:33:00.801 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2628999999999997e-05 2024-04-20T17:33:00.801 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.801 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.801 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:33:00.801 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.996334+0000", 2024-04-20T17:33:00.801 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9103999999999993e-05 2024-04-20T17:33:00.801 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.801 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.801 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:33:00.801 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.015536+0000", 2024-04-20T17:33:00.802 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019201419000000001 2024-04-20T17:33:00.802 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.802 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.802 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:33:00.802 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.081190+0000", 2024-04-20T17:33:00.802 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.065653883999999996 2024-04-20T17:33:00.802 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.802 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.802 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:33:00.802 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.081230+0000", 2024-04-20T17:33:00.802 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0348000000000001e-05 2024-04-20T17:33:00.802 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.802 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.802 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:33:00.803 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.081270+0000", 2024-04-20T17:33:00.803 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.968e-05 2024-04-20T17:33:00.803 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.803 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:33:00.803 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.803 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.803 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.803 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:280495 4.8 4:122e2751:::benchmark_data_smithi171_122772_object110303:head [delete] snapc 0=[] ondisk+write+known_if_redirected e521)", 2024-04-20T17:33:00.803 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:15.891104+0000", 2024-04-20T17:33:00.803 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 284.845795173, 2024-04-20T17:33:00.803 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19031331200000001, 2024-04-20T17:33:00.803 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:33:00.803 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:33:00.803 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:00.803 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:33:00.804 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:33:00.804 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:33:00.804 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 280495 2024-04-20T17:33:00.804 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.804 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:33:00.804 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.804 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:33:00.804 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.891104+0000", 2024-04-20T17:33:00.804 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.804 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.804 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.804 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:33:00.804 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.891104+0000", 2024-04-20T17:33:00.804 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.804 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.804 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.805 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:33:00.805 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.891106+0000", 2024-04-20T17:33:00.805 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.429e-06 2024-04-20T17:33:00.805 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.805 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.805 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:33:00.805 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.891112+0000", 2024-04-20T17:33:00.805 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0869999999999997e-06 2024-04-20T17:33:00.805 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.805 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.805 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:33:00.805 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.891122+0000", 2024-04-20T17:33:00.805 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0141e-05 2024-04-20T17:33:00.805 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.805 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.806 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:33:00.806 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.891129+0000", 2024-04-20T17:33:00.806 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2699999999999999e-06 2024-04-20T17:33:00.806 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.806 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.806 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:33:00.806 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.001431+0000", 2024-04-20T17:33:00.806 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.110301732 2024-04-20T17:33:00.806 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.806 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.806 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:33:00.806 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.001510+0000", 2024-04-20T17:33:00.806 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9711999999999998e-05 2024-04-20T17:33:00.806 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.806 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.807 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:33:00.807 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.001610+0000", 2024-04-20T17:33:00.807 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9686999999999996e-05 2024-04-20T17:33:00.807 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.807 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.807 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:33:00.807 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.015543+0000", 2024-04-20T17:33:00.807 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013933332999999999 2024-04-20T17:33:00.807 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.807 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.807 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:33:00.807 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.081324+0000", 2024-04-20T17:33:00.807 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.065780901000000003 2024-04-20T17:33:00.807 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.807 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.808 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:33:00.808 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.081365+0000", 2024-04-20T17:33:00.808 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1131999999999999e-05 2024-04-20T17:33:00.808 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.808 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.808 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:33:00.808 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.081417+0000", 2024-04-20T17:33:00.808 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1888000000000002e-05 2024-04-20T17:33:00.808 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.808 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:33:00.808 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.808 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.808 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.808 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:280571 4.8 4:157d77de:::benchmark_data_smithi171_122772_object110379:head [delete] snapc 0=[] ondisk+write+known_if_redirected e521)", 2024-04-20T17:33:00.809 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:16.010716+0000", 2024-04-20T17:33:00.809 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 284.72618287300003, 2024-04-20T17:33:00.809 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.178831402, 2024-04-20T17:33:00.809 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:33:00.809 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:33:00.809 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:00.809 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:33:00.809 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:33:00.809 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:33:00.809 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 280571 2024-04-20T17:33:00.809 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.809 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:33:00.809 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.809 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:33:00.809 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.010716+0000", 2024-04-20T17:33:00.810 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.810 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.810 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.810 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:33:00.810 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.010716+0000", 2024-04-20T17:33:00.810 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.810 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.810 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.810 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:33:00.810 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.010717+0000", 2024-04-20T17:33:00.810 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.057e-06 2024-04-20T17:33:00.810 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.810 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.810 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:33:00.811 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.010722+0000", 2024-04-20T17:33:00.811 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4499999999999997e-06 2024-04-20T17:33:00.811 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.811 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.811 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:33:00.811 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.010730+0000", 2024-04-20T17:33:00.811 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6620000000000005e-06 2024-04-20T17:33:00.811 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.811 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.811 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:33:00.811 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.010735+0000", 2024-04-20T17:33:00.811 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9229999999999999e-06 2024-04-20T17:33:00.811 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.811 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.811 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:33:00.812 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.015594+0000", 2024-04-20T17:33:00.812 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0048587580000000003 2024-04-20T17:33:00.812 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.812 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.812 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:33:00.812 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.015723+0000", 2024-04-20T17:33:00.812 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012861 2024-04-20T17:33:00.812 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.812 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.812 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:33:00.812 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.015847+0000", 2024-04-20T17:33:00.812 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012468700000000001 2024-04-20T17:33:00.812 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.812 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.812 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:33:00.812 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.023498+0000", 2024-04-20T17:33:00.813 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0076504340000000002 2024-04-20T17:33:00.813 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.813 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.813 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:33:00.813 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.189467+0000", 2024-04-20T17:33:00.813 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16596942100000001 2024-04-20T17:33:00.813 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.813 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.813 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:33:00.813 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.189503+0000", 2024-04-20T17:33:00.813 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5531999999999998e-05 2024-04-20T17:33:00.813 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.813 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.813 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:33:00.814 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.189548+0000", 2024-04-20T17:33:00.814 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4867999999999998e-05 2024-04-20T17:33:00.814 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.814 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:33:00.814 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.814 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.814 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.814 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:280572 4.8 4:10b6d45a:::benchmark_data_smithi171_122772_object110380:head [delete] snapc 0=[] ondisk+write+known_if_redirected e521)", 2024-04-20T17:33:00.814 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:16.010744+0000", 2024-04-20T17:33:00.814 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 284.72615518200001, 2024-04-20T17:33:00.814 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.17898547200000001, 2024-04-20T17:33:00.814 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:33:00.814 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:33:00.814 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:00.814 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:33:00.815 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:33:00.815 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:33:00.815 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 280572 2024-04-20T17:33:00.815 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.815 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:33:00.815 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.815 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:33:00.815 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.010744+0000", 2024-04-20T17:33:00.815 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.815 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.815 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.815 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:33:00.815 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.010744+0000", 2024-04-20T17:33:00.815 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.815 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.816 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.816 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:33:00.816 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.010745+0000", 2024-04-20T17:33:00.816 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4499999999999996e-07 2024-04-20T17:33:00.816 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.816 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.816 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:33:00.816 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.010748+0000", 2024-04-20T17:33:00.816 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0309999999999999e-06 2024-04-20T17:33:00.816 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.816 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.816 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:33:00.816 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.010753+0000", 2024-04-20T17:33:00.816 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5690000000000004e-06 2024-04-20T17:33:00.816 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.817 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.817 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:33:00.817 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.010758+0000", 2024-04-20T17:33:00.817 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1109999999999999e-06 2024-04-20T17:33:00.817 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.817 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.817 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:33:00.817 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.019864+0000", 2024-04-20T17:33:00.817 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0091055289999999994 2024-04-20T17:33:00.817 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.817 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.817 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:33:00.817 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.019943+0000", 2024-04-20T17:33:00.817 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8638000000000002e-05 2024-04-20T17:33:00.817 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.818 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.818 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:33:00.818 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.020025+0000", 2024-04-20T17:33:00.818 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1961999999999999e-05 2024-04-20T17:33:00.818 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.818 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.818 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:33:00.818 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.023639+0000", 2024-04-20T17:33:00.818 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0036145470000000001 2024-04-20T17:33:00.818 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.818 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.818 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:33:00.818 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.189605+0000", 2024-04-20T17:33:00.818 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.165965522 2024-04-20T17:33:00.818 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.819 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.819 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:33:00.819 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.189668+0000", 2024-04-20T17:33:00.819 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2952000000000003e-05 2024-04-20T17:33:00.819 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.819 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.819 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:33:00.819 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.189730+0000", 2024-04-20T17:33:00.819 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1865999999999993e-05 2024-04-20T17:33:00.819 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.819 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:33:00.819 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.819 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.819 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.820 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:280618 4.8 4:1aaeafa9:::benchmark_data_smithi171_122772_object110426:head [delete] snapc 0=[] ondisk+write+known_if_redirected e521)", 2024-04-20T17:33:00.820 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:16.102539+0000", 2024-04-20T17:33:00.820 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 284.63436003700002, 2024-04-20T17:33:00.820 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16693295799999999, 2024-04-20T17:33:00.820 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:33:00.820 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:33:00.820 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:00.820 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:33:00.820 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:33:00.820 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:33:00.820 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 280618 2024-04-20T17:33:00.820 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.820 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:33:00.820 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.820 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:33:00.821 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.102539+0000", 2024-04-20T17:33:00.821 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.821 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.821 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.821 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:33:00.821 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.102539+0000", 2024-04-20T17:33:00.821 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.821 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.821 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.821 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:33:00.821 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.102541+0000", 2024-04-20T17:33:00.821 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.153e-06 2024-04-20T17:33:00.821 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.821 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.821 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:33:00.822 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.102548+0000", 2024-04-20T17:33:00.822 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.37e-06 2024-04-20T17:33:00.822 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.822 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.822 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:33:00.822 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.102560+0000", 2024-04-20T17:33:00.822 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2442e-05 2024-04-20T17:33:00.822 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.822 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.822 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:33:00.822 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.102568+0000", 2024-04-20T17:33:00.822 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5229999999999998e-06 2024-04-20T17:33:00.822 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.822 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.823 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:33:00.823 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.191248+0000", 2024-04-20T17:33:00.823 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.088680160999999993 2024-04-20T17:33:00.823 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.823 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.823 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:33:00.823 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.191364+0000", 2024-04-20T17:33:00.823 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011568799999999999 2024-04-20T17:33:00.823 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.823 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.823 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:33:00.823 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.191480+0000", 2024-04-20T17:33:00.823 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000116234 2024-04-20T17:33:00.823 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.823 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.824 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:33:00.824 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.211071+0000", 2024-04-20T17:33:00.824 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019591377 2024-04-20T17:33:00.824 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.824 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.824 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:33:00.824 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.269349+0000", 2024-04-20T17:33:00.824 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.058277429999999998 2024-04-20T17:33:00.824 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.824 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.824 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:33:00.824 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.269410+0000", 2024-04-20T17:33:00.824 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1648000000000003e-05 2024-04-20T17:33:00.824 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.824 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.825 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:33:00.825 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.269472+0000", 2024-04-20T17:33:00.825 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1932000000000003e-05 2024-04-20T17:33:00.825 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.825 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:33:00.825 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.825 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.825 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.825 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:281568 4.8 4:1fa3e07e:::benchmark_data_smithi171_122772_object111376:head [delete] snapc 0=[] ondisk+write+known_if_redirected e521)", 2024-04-20T17:33:00.825 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:17.475059+0000", 2024-04-20T17:33:00.825 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 283.26183978799997, 2024-04-20T17:33:00.825 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.17411728200000001, 2024-04-20T17:33:00.825 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:33:00.825 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:33:00.825 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:00.826 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:33:00.826 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:33:00.826 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:33:00.826 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 281568 2024-04-20T17:33:00.826 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.826 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:33:00.826 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.826 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:33:00.826 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.475059+0000", 2024-04-20T17:33:00.826 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.826 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.826 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.826 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:33:00.826 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.475059+0000", 2024-04-20T17:33:00.827 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.827 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.827 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.827 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:33:00.827 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.475060+0000", 2024-04-20T17:33:00.827 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.3300000000000001e-07 2024-04-20T17:33:00.827 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.827 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.827 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:33:00.827 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.475064+0000", 2024-04-20T17:33:00.827 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3100000000000001e-06 2024-04-20T17:33:00.827 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.827 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.827 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:33:00.828 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.475070+0000", 2024-04-20T17:33:00.828 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9e-06 2024-04-20T17:33:00.828 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.828 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.828 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:33:00.828 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.475077+0000", 2024-04-20T17:33:00.828 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0859999999999998e-06 2024-04-20T17:33:00.828 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.828 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.828 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:33:00.828 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.563751+0000", 2024-04-20T17:33:00.828 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.088674175999999993 2024-04-20T17:33:00.828 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.828 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.828 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:33:00.829 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.563852+0000", 2024-04-20T17:33:00.829 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010081000000000001 2024-04-20T17:33:00.829 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.829 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.829 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:33:00.829 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.563955+0000", 2024-04-20T17:33:00.829 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000103577 2024-04-20T17:33:00.829 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.829 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.829 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:33:00.829 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.577489+0000", 2024-04-20T17:33:00.829 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013533991 2024-04-20T17:33:00.829 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.829 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.829 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:33:00.830 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.649060+0000", 2024-04-20T17:33:00.830 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.071571368999999996 2024-04-20T17:33:00.830 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.830 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.830 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:33:00.830 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.649115+0000", 2024-04-20T17:33:00.830 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4540999999999998e-05 2024-04-20T17:33:00.830 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.830 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.830 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:33:00.830 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.649177+0000", 2024-04-20T17:33:00.830 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1688999999999998e-05 2024-04-20T17:33:00.830 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.830 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:33:00.831 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.831 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.831 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.831 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:282370 4.8 4:113c9101:::benchmark_data_smithi171_122772_object112178:head [delete] snapc 0=[] ondisk+write+known_if_redirected e521)", 2024-04-20T17:33:00.831 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:18.673315+0000", 2024-04-20T17:33:00.831 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 282.06358435300001, 2024-04-20T17:33:00.831 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16601888300000001, 2024-04-20T17:33:00.831 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:33:00.831 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:33:00.831 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:00.831 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:33:00.831 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:33:00.831 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:33:00.831 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 282370 2024-04-20T17:33:00.831 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.831 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:33:00.832 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.832 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:33:00.832 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:18.673315+0000", 2024-04-20T17:33:00.832 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.832 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.832 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.832 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:33:00.832 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:18.673315+0000", 2024-04-20T17:33:00.832 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.832 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.832 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.832 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:33:00.832 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:18.673317+0000", 2024-04-20T17:33:00.832 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.063e-06 2024-04-20T17:33:00.833 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.833 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.833 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:33:00.833 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:18.673322+0000", 2024-04-20T17:33:00.833 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5319999999999997e-06 2024-04-20T17:33:00.833 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.833 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.833 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:33:00.833 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:18.673332+0000", 2024-04-20T17:33:00.833 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7249999999999992e-06 2024-04-20T17:33:00.833 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.833 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.833 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:33:00.833 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:18.673338+0000", 2024-04-20T17:33:00.834 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6790000000000002e-06 2024-04-20T17:33:00.834 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.834 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.834 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:33:00.834 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:18.750016+0000", 2024-04-20T17:33:00.834 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.076678272000000006 2024-04-20T17:33:00.834 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.834 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.834 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:33:00.834 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:18.750138+0000", 2024-04-20T17:33:00.834 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000121966 2024-04-20T17:33:00.834 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.834 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.834 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:33:00.835 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:18.750279+0000", 2024-04-20T17:33:00.835 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000140956 2024-04-20T17:33:00.835 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.835 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.835 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:33:00.835 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:18.757772+0000", 2024-04-20T17:33:00.835 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0074926599999999999 2024-04-20T17:33:00.835 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.835 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.835 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:33:00.835 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:18.839228+0000", 2024-04-20T17:33:00.835 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.081456732000000004 2024-04-20T17:33:00.835 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.835 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.835 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:33:00.836 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:18.839282+0000", 2024-04-20T17:33:00.836 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3358e-05 2024-04-20T17:33:00.836 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.836 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.836 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:33:00.836 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:18.839334+0000", 2024-04-20T17:33:00.836 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1940000000000001e-05 2024-04-20T17:33:00.836 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.836 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:33:00.836 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.836 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.836 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.836 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:283096 4.8 4:1ed17a90:::benchmark_data_smithi171_122772_object112904:head [delete] snapc 0=[] ondisk+write+known_if_redirected e521)", 2024-04-20T17:33:00.836 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:19.770227+0000", 2024-04-20T17:33:00.837 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 280.96667210800001, 2024-04-20T17:33:00.837 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.175826486, 2024-04-20T17:33:00.837 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:33:00.837 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:33:00.837 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:00.837 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:33:00.837 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:33:00.837 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:33:00.837 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 283096 2024-04-20T17:33:00.837 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.837 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:33:00.837 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.837 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:33:00.837 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.770227+0000", 2024-04-20T17:33:00.837 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.838 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.838 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.838 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:33:00.838 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.770227+0000", 2024-04-20T17:33:00.838 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.838 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.838 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.838 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:33:00.838 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.770229+0000", 2024-04-20T17:33:00.838 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3520000000000001e-06 2024-04-20T17:33:00.838 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.838 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.838 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:33:00.838 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.770235+0000", 2024-04-20T17:33:00.838 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6389999999999997e-06 2024-04-20T17:33:00.839 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.839 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.839 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:33:00.839 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.770249+0000", 2024-04-20T17:33:00.839 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3909e-05 2024-04-20T17:33:00.839 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.839 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.839 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:33:00.839 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.770257+0000", 2024-04-20T17:33:00.839 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2029999999999992e-06 2024-04-20T17:33:00.839 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.839 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.839 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:33:00.839 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.866656+0000", 2024-04-20T17:33:00.840 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.096398567000000004 2024-04-20T17:33:00.840 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.840 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.840 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:33:00.840 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.866782+0000", 2024-04-20T17:33:00.840 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012599599999999999 2024-04-20T17:33:00.840 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.840 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.840 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:33:00.840 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.866908+0000", 2024-04-20T17:33:00.840 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000126372 2024-04-20T17:33:00.840 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.840 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.840 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:33:00.840 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.875989+0000", 2024-04-20T17:33:00.841 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0090805780000000006 2024-04-20T17:33:00.841 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.841 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.841 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:33:00.841 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.945960+0000", 2024-04-20T17:33:00.841 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.069971729999999996 2024-04-20T17:33:00.841 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.841 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.841 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:33:00.841 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.945995+0000", 2024-04-20T17:33:00.841 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4789999999999997e-05 2024-04-20T17:33:00.841 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.841 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.841 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:33:00.841 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.946054+0000", 2024-04-20T17:33:00.842 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8350000000000002e-05 2024-04-20T17:33:00.842 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.842 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:33:00.842 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.842 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.842 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.842 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:283104 4.8 4:15144f8b:::benchmark_data_smithi171_122772_object112912:head [delete] snapc 0=[] ondisk+write+known_if_redirected e521)", 2024-04-20T17:33:00.842 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:19.779189+0000", 2024-04-20T17:33:00.842 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 280.95771039200002, 2024-04-20T17:33:00.842 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.17012118200000001, 2024-04-20T17:33:00.842 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:33:00.842 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:33:00.842 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:00.843 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:33:00.843 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:33:00.843 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:33:00.843 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 283104 2024-04-20T17:33:00.843 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.843 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:33:00.843 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.843 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:33:00.843 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.779189+0000", 2024-04-20T17:33:00.843 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.843 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.843 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.843 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:33:00.843 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.779189+0000", 2024-04-20T17:33:00.843 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.844 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.844 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.844 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:33:00.844 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.779191+0000", 2024-04-20T17:33:00.844 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.26e-06 2024-04-20T17:33:00.844 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.844 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.844 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:33:00.844 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.779197+0000", 2024-04-20T17:33:00.844 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8449999999999999e-06 2024-04-20T17:33:00.844 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.844 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.844 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:33:00.844 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.779217+0000", 2024-04-20T17:33:00.844 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0452999999999999e-05 2024-04-20T17:33:00.845 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.845 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.845 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:33:00.845 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.779224+0000", 2024-04-20T17:33:00.845 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3350000000000003e-06 2024-04-20T17:33:00.845 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.845 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.845 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:33:00.845 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.870907+0000", 2024-04-20T17:33:00.845 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.091683095000000006 2024-04-20T17:33:00.845 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.845 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.845 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:33:00.845 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.871024+0000", 2024-04-20T17:33:00.846 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011728800000000001 2024-04-20T17:33:00.846 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.846 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.846 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:33:00.846 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.871142+0000", 2024-04-20T17:33:00.846 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011751 2024-04-20T17:33:00.846 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.846 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.846 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:33:00.846 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.880380+0000", 2024-04-20T17:33:00.846 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0092385160000000004 2024-04-20T17:33:00.846 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.846 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.846 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:33:00.846 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.949228+0000", 2024-04-20T17:33:00.847 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.068848030000000005 2024-04-20T17:33:00.847 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.847 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.847 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:33:00.847 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.949266+0000", 2024-04-20T17:33:00.847 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7753999999999998e-05 2024-04-20T17:33:00.847 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.847 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.847 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:33:00.847 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.949310+0000", 2024-04-20T17:33:00.847 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4095999999999999e-05 2024-04-20T17:33:00.847 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.847 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:33:00.847 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.848 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.848 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.848 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:283112 4.8 4:1f3b1708:::benchmark_data_smithi171_122772_object112920:head [delete] snapc 0=[] ondisk+write+known_if_redirected e521)", 2024-04-20T17:33:00.848 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:19.789454+0000", 2024-04-20T17:33:00.848 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 280.94744564899997, 2024-04-20T17:33:00.848 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16605162300000001, 2024-04-20T17:33:00.848 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:33:00.848 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:33:00.848 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:00.848 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:33:00.848 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:33:00.848 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:33:00.848 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 283112 2024-04-20T17:33:00.848 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.848 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:33:00.849 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.849 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:33:00.849 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.789454+0000", 2024-04-20T17:33:00.849 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.849 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.849 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.849 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:33:00.849 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.789454+0000", 2024-04-20T17:33:00.849 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:33:00.849 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.849 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.849 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:33:00.849 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.789456+0000", 2024-04-20T17:33:00.849 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.723e-06 2024-04-20T17:33:00.849 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.850 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.850 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:33:00.850 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.789464+0000", 2024-04-20T17:33:00.850 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7519999999999996e-06 2024-04-20T17:33:00.850 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.850 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.850 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:33:00.850 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.789479+0000", 2024-04-20T17:33:00.850 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4479e-05 2024-04-20T17:33:00.850 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.850 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.850 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:33:00.850 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.789488+0000", 2024-04-20T17:33:00.850 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9380000000000004e-06 2024-04-20T17:33:00.851 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.851 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.851 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:33:00.851 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.876116+0000", 2024-04-20T17:33:00.851 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.086627351000000005 2024-04-20T17:33:00.851 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.851 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.851 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:33:00.851 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.876234+0000", 2024-04-20T17:33:00.851 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000118609 2024-04-20T17:33:00.851 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.851 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.851 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:33:00.851 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.876342+0000", 2024-04-20T17:33:00.851 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000107088 2024-04-20T17:33:00.852 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.852 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.852 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:33:00.852 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.897090+0000", 2024-04-20T17:33:00.852 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020748860000000001 2024-04-20T17:33:00.852 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.852 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.852 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:33:00.852 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.955402+0000", 2024-04-20T17:33:00.852 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.058311363999999997 2024-04-20T17:33:00.852 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.852 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.852 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:33:00.852 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.955448+0000", 2024-04-20T17:33:00.852 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6612e-05 2024-04-20T17:33:00.852 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:33:00.853 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:33:00.853 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:33:00.853 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.955505+0000", 2024-04-20T17:33:00.853 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6847e-05 2024-04-20T17:33:00.853 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.853 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:33:00.853 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.853 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:33:00.853 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:33:00.853 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:33:00.924 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:33:00.983 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:33:01.017 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:33:01.044 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:33:01.092 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:33:01.132 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:33:01.166 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:33:01.197 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:33:01.236 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:33:01.283 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:33:01.296 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:33:01.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:01.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:33:01.320 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:33:01.371 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:33:01.388 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:33:01.388 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:33:01.388 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:33:01.389 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:33:01.389 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:33:01.389 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:33:01.389 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:33:01.389 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:33:01.389 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:33:01.389 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:33:01.389 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:33:01.389 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:33:01.401 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:33:01.436 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:01.466 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-20T17:33:01.470 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:33:01.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:01.475 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:33:01.577 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:33:01.612 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:01.630 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:33:01.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:01.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:33:01.737 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:33:01.777 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:01 smithi171 ceph-mon[44189]: pgmap v1608: 17 pgs: 17 active+clean; 529 MiB data, 4.6 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-20T17:33:01.777 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:01 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1436466565' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:33:01.777 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:01.777 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:01.780 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:01.806 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:33:01.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:01.811 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:33:01.912 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:33:01.928 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:01 smithi005 ceph-mon[39562]: pgmap v1608: 17 pgs: 17 active+clean; 529 MiB data, 4.6 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-20T17:33:01.928 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:01 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1436466565' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:33:01.929 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:01.929 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:01.951 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:01.977 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:33:01.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:01.982 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:33:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:01 smithi106 ceph-mon[44423]: pgmap v1608: 17 pgs: 17 active+clean; 529 MiB data, 4.6 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-20T17:33:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:01 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1436466565' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:33:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:02.083 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:33:02.128 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:02.154 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:33:02.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:02.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:33:02.261 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:33:02.305 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:02.333 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:33:02.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:02.337 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:33:02.438 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:33:02.486 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:02.650 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:33:02.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:02.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:33:02.753 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:02.753 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:02.754 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:02.754 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:02.754 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:02.754 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:02.754 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:02.754 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:02.754 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:02.754 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:02.754 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:33:02.795 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:02.816 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:33:02.818 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:02.818 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:02.818 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:02.818 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:02.818 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:02.819 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:02.819 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:02.819 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:02.819 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:02.819 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:02.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:02.820 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:33:02.921 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:33:02.969 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:03.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:03.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:03.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:03.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:03.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:03.085 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:33:03.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:03.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:33:03.192 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:33:03.250 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:03 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:33:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:03 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:33:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:03 smithi171 ceph-mon[44189]: pgmap v1609: 17 pgs: 17 active+clean; 349 MiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-20T17:33:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:03 smithi171 ceph-mon[44189]: osdmap e612: 12 total, 12 up, 11 in 2024-04-20T17:33:03.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:03 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:33:03.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:03 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:33:03.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:03 smithi005 ceph-mon[39562]: pgmap v1609: 17 pgs: 17 active+clean; 349 MiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-20T17:33:03.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:03 smithi005 ceph-mon[39562]: osdmap e612: 12 total, 12 up, 11 in 2024-04-20T17:33:03.982 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:33:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:03 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:33:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:03 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:33:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:03 smithi106 ceph-mon[44423]: pgmap v1609: 17 pgs: 17 active+clean; 349 MiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-20T17:33:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:03 smithi106 ceph-mon[44423]: osdmap e612: 12 total, 12 up, 11 in 2024-04-20T17:33:04.039 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:33:04.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:04.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:33:04.144 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:33:04.204 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:04.291 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:33:04.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:04.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:33:04.395 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:33:04.432 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:04.457 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:33:04.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:04.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:33:04.562 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:33:04.577 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:04 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1231149088' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:33:04.577 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:04 smithi005 ceph-mon[39562]: from='client.26713 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:33:04.597 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:04.597 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:33:04.625 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:33:04.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:04.631 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:33:04.711 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Removed 131431 objects 2024-04-20T17:33:04.712 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Clean up completed and total clean up time :114.473 2024-04-20T17:33:04.733 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:33:04.758 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:04 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1231149088' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:33:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:04 smithi106 ceph-mon[44423]: from='client.26713 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:33:04.777 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:33:04.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:04.779 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:33:04.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:04 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1231149088' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:33:04.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:04 smithi171 ceph-mon[44189]: from='client.26713 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:33:04.880 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:33:04.906 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:04.925 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:33:04.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:04.928 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:33:05.029 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:33:05.055 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:05.073 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:33:05.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:05.076 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:33:05.134 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:33:05.177 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:33:05.205 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:05.224 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:33:05.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:05.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:33:05.328 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:33:05.354 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:05.373 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:33:05.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:05.376 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:33:05.477 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:33:05.507 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:05.605 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:33:05.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:05.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:33:05.709 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:33:05.738 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:05 smithi171 ceph-mon[44189]: pgmap v1611: 17 pgs: 17 active+clean; 238 MiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-20T17:33:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:05 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:33:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:05 smithi171 ceph-mon[44189]: from='client.26713 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:33:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:05 smithi171 ceph-mon[44189]: osdmap e613: 12 total, 12 up, 11 in 2024-04-20T17:33:05.838 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:05 smithi005 ceph-mon[39562]: pgmap v1611: 17 pgs: 17 active+clean; 238 MiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-20T17:33:05.838 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:05 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:33:05.838 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:05 smithi005 ceph-mon[39562]: from='client.26713 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:33:05.839 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:05 smithi005 ceph-mon[39562]: osdmap e613: 12 total, 12 up, 11 in 2024-04-20T17:33:05.935 INFO:tasks.thrashosds.thrasher:Recovered, killing an osd 2024-04-20T17:33:05.935 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 1, 4, 5, 8, 9, 10, 11, 2, 7, 6, 3] 2024-04-20T17:33:05.935 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-20T17:33:05.936 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.1 2024-04-20T17:33:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:05 smithi106 ceph-mon[44423]: pgmap v1611: 17 pgs: 17 active+clean; 238 MiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-20T17:33:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:05 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:33:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:05 smithi106 ceph-mon[44423]: from='client.26713 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:33:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:05 smithi106 ceph-mon[44423]: osdmap e613: 12 total, 12 up, 11 in 2024-04-20T17:33:06.211 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:33:05 smithi005 systemd[1]: Stopping Ceph osd.1 for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:33:06.687 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:33:06.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:06.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:33:06.792 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:33:06.825 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:06.846 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:33:06.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:06.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:33:06.950 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:33:06.982 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:07.202 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:33:07.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:07.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:33:07.306 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:33:07.340 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:07.361 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:33:07.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:07.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:33:07.465 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:33:07.498 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:07.520 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:33:07.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:07.523 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:33:07.576 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-04-20T17:33:07.577 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-04-20T17:33:07.604 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:07 smithi005 ceph-mon[39562]: pgmap v1613: 17 pgs: 17 active+clean; 58 MiB data, 3.6 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-20T17:33:07.624 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:33:07.654 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:07 smithi106 ceph-mon[44423]: pgmap v1613: 17 pgs: 17 active+clean; 58 MiB data, 3.6 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-20T17:33:07.656 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:07.678 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:33:07.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:07.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:33:07.688 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:33:07.782 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:33:07.812 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:07.832 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:33:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:07 smithi171 ceph-mon[44189]: pgmap v1613: 17 pgs: 17 active+clean; 58 MiB data, 3.6 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-20T17:33:07.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:07.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:33:07.936 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:33:07.982 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:08.133 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:33:08.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:08.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:33:08.238 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:33:08.268 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:08.863 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:33:08.864 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:33:08.919 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:33:08.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:08.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:33:09.024 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:33:09.043 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:33:09.055 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:08 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:33:09.055 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:08 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:33:09.055 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:08 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:33:09.055 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:08 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:33:09.056 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:08 smithi171 ceph-mon[44189]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:33:09.058 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:09.070 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:33:08 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-1[64731]: 2024-04-20T17:33:08.915+0000 7f8202eab700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T17:33:09.071 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:33:08 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-1[64731]: 2024-04-20T17:33:08.915+0000 7f8202eab700 -1 osd.1 613 *** Got signal Terminated *** 2024-04-20T17:33:09.071 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:33:08 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-1[64731]: 2024-04-20T17:33:08.915+0000 7f8202eab700 -1 osd.1 613 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T17:33:09.071 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:08 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:33:09.071 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:08 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:33:09.071 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:08 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:33:09.071 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:08 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:33:09.071 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:08 smithi005 ceph-mon[39562]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:33:09.078 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:33:09.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:09.082 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:33:09.183 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:33:09.213 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:09.233 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:33:09.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:09.237 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:33:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:08 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:33:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:08 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:33:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:08 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:33:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:08 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:33:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:08 smithi106 ceph-mon[44423]: from='mgr.14148 ' entity='mgr.y' 2024-04-20T17:33:09.337 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:33:09.365 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:09.547 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-20T17:33:09.564 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:33:09.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:09.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:33:09.668 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:33:09.696 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:09.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:09 smithi005 ceph-mon[39562]: pgmap v1614: 17 pgs: 17 active+clean; 16 MiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-20T17:33:09.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:09 smithi005 ceph-mon[39562]: osd.1 marked itself down and dead 2024-04-20T17:33:10.060 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:33:10.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:10.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:33:10.074 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:33:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:09 smithi171 ceph-mon[44189]: pgmap v1614: 17 pgs: 17 active+clean; 16 MiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-20T17:33:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:09 smithi171 ceph-mon[44189]: osd.1 marked itself down and dead 2024-04-20T17:33:10.126 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:33:10.129 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:33:10.164 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:33:10.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:09 smithi106 ceph-mon[44423]: pgmap v1614: 17 pgs: 17 active+clean; 16 MiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-20T17:33:10.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:09 smithi106 ceph-mon[44423]: osd.1 marked itself down and dead 2024-04-20T17:33:10.197 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:10.219 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:33:10.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:10.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:33:10.323 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:33:10.354 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:10.376 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:33:10.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:10.380 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:33:10.481 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:33:10.513 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:10.534 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:33:10.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:10.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:33:10.639 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:33:10.670 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:10.691 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:33:10.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:10.694 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:33:10.795 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:33:10.828 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:10.853 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:33:10.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:10.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:33:10.957 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:33:10.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:10 smithi005 ceph-mon[39562]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T17:33:10.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:10 smithi005 ceph-mon[39562]: osdmap e614: 12 total, 11 up, 11 in 2024-04-20T17:33:10.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:10 smithi005 ceph-mon[39562]: 5.9 scrub starts 2024-04-20T17:33:10.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:10 smithi005 ceph-mon[39562]: 5.9 scrub ok 2024-04-20T17:33:10.962 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:33:10 smithi005 bash[256757]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-1 2024-04-20T17:33:10.995 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:11.053 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:33:11.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:11.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:33:11.082 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:33:11.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:10 smithi171 ceph-mon[44189]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T17:33:11.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:10 smithi171 ceph-mon[44189]: osdmap e614: 12 total, 11 up, 11 in 2024-04-20T17:33:11.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:10 smithi171 ceph-mon[44189]: 5.9 scrub starts 2024-04-20T17:33:11.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:10 smithi171 ceph-mon[44189]: 5.9 scrub ok 2024-04-20T17:33:11.101 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:33:11.120 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:33:11.141 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:33:11.156 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:33:11.160 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:33:11.184 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:33:11.187 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:11.206 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:33:11.225 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:33:11.226 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:33:11.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:11.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:33:11.249 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:33:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:10 smithi106 ceph-mon[44423]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T17:33:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:10 smithi106 ceph-mon[44423]: osdmap e614: 12 total, 11 up, 11 in 2024-04-20T17:33:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:10 smithi106 ceph-mon[44423]: 5.9 scrub starts 2024-04-20T17:33:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:10 smithi106 ceph-mon[44423]: 5.9 scrub ok 2024-04-20T17:33:11.272 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:33:11.292 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:33:11.293 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:33:11.293 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-20T17:33:11.293 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:33:11.293 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:33:11.293 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:33:11.293 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:33:11.293 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:33:11.293 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:33:11.293 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:33:11.293 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:33:11.294 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:33:11.294 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:33:11.334 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:33:11.365 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:11.514 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:33:11.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:11.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:33:11.619 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:33:11.648 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:11.686 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:33:11.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:11.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:33:11.790 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:33:11.823 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:11.844 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:33:11.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:11.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:33:11.948 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:33:11.978 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:11.998 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:33:12.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:12.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:33:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:11 smithi171 ceph-mon[44189]: pgmap v1616: 17 pgs: 17 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 674 op/s 2024-04-20T17:33:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:11 smithi171 ceph-mon[44189]: osdmap e615: 12 total, 11 up, 11 in 2024-04-20T17:33:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:11 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3020815260' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:33:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.102 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:33:12.134 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:12.156 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:33:12.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:12.159 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:33:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:11 smithi005 ceph-mon[39562]: pgmap v1616: 17 pgs: 17 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 674 op/s 2024-04-20T17:33:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:11 smithi005 ceph-mon[39562]: osdmap e615: 12 total, 11 up, 11 in 2024-04-20T17:33:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:11 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3020815260' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:33:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.260 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:33:12.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:11 smithi106 ceph-mon[44423]: pgmap v1616: 17 pgs: 17 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 674 op/s 2024-04-20T17:33:12.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:11 smithi106 ceph-mon[44423]: osdmap e615: 12 total, 11 up, 11 in 2024-04-20T17:33:12.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:11 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3020815260' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:33:12.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.291 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:12.311 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:33:12.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:12.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:33:12.415 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:33:12.443 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:12.463 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:33:12.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:12.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:33:12.566 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:33:12.599 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:12.620 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:33:12.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:12.624 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:33:12.725 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:33:12.762 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:12.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:12.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:13.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:13.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:14.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:13 smithi171 ceph-mon[44189]: pgmap v1618: 17 pgs: 1 peering, 1 undersized+peered, 15 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 345 op/s 2024-04-20T17:33:14.136 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:13 smithi005 ceph-mon[39562]: pgmap v1618: 17 pgs: 1 peering, 1 undersized+peered, 15 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 345 op/s 2024-04-20T17:33:14.167 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:33:14.168 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:33:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:13 smithi106 ceph-mon[44423]: pgmap v1618: 17 pgs: 1 peering, 1 undersized+peered, 15 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 345 op/s 2024-04-20T17:33:15.115 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:33:15.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:15.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:33:15.220 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:33:15.251 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:15.831 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'unique_pool_3' removed 2024-04-20T17:33:15.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:15 smithi005 ceph-mon[39562]: pgmap v1619: 17 pgs: 1 peering, 2 undersized+peered, 14 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 115 op/s 2024-04-20T17:33:15.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:15 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1894405662' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-20T17:33:16.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:15 smithi171 ceph-mon[44189]: pgmap v1619: 17 pgs: 1 peering, 2 undersized+peered, 14 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 115 op/s 2024-04-20T17:33:16.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:15 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1894405662' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-20T17:33:16.169 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:33:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:15 smithi106 ceph-mon[44423]: pgmap v1619: 17 pgs: 1 peering, 2 undersized+peered, 14 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 115 op/s 2024-04-20T17:33:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:15 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1894405662' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-20T17:33:16.677 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-20T17:33:16.678 INFO:tasks.radosbench:Beginning radosbench... 2024-04-20T17:33:16.678 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-04-20T17:33:16.678 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-04-20T17:33:16.702 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:33:16.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:16.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:33:16.806 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:33:16.877 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:16.897 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:33:16.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:16.900 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:33:16.925 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:16 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1894405662' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-04-20T17:33:16.925 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:16 smithi005 ceph-mon[39562]: osdmap e616: 12 total, 11 up, 11 in 2024-04-20T17:33:17.001 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:33:17.028 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:17.048 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:33:17.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:17.051 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:33:17.152 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:33:17.170 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:33:17.177 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:16 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1894405662' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-04-20T17:33:17.178 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:16 smithi106 ceph-mon[44423]: osdmap e616: 12 total, 11 up, 11 in 2024-04-20T17:33:17.179 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:17.180 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:33:17.200 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:33:17.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:17.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:33:17.302 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:33:17.328 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:16 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1894405662' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-04-20T17:33:17.328 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:16 smithi171 ceph-mon[44189]: osdmap e616: 12 total, 11 up, 11 in 2024-04-20T17:33:17.330 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:17.348 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:33:17.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:17.351 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:33:17.452 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:33:17.479 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:17.498 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:33:17.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:17.500 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:33:17.601 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:33:17.631 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:17.651 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:33:17.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:17.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:33:17.755 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:33:17.792 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:17.880 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:33:17.899 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:33:17.899 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:33:17.899 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:33:17.899 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:33:17.899 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:33:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:17 smithi106 ceph-mon[44423]: pgmap v1621: 1 pgs: 1 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:17 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/646101198' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:33:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:17 smithi106 ceph-mon[44423]: from='client.26728 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:33:18.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:17 smithi005 ceph-mon[39562]: pgmap v1621: 1 pgs: 1 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:18.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:17 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/646101198' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:33:18.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:17 smithi005 ceph-mon[39562]: from='client.26728 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:33:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:17 smithi171 ceph-mon[44189]: pgmap v1621: 1 pgs: 1 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:17 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/646101198' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:33:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:17 smithi171 ceph-mon[44189]: from='client.26728 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:33:18.755 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:33:18.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:18.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:33:18.859 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:33:18.895 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:18.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:18 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:33:18.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:18 smithi005 ceph-mon[39562]: from='client.26728 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:33:18.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:18 smithi005 ceph-mon[39562]: osdmap e617: 12 total, 11 up, 11 in 2024-04-20T17:33:19.145 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:33:19.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:19.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:33:19.251 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:33:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:18 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:33:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:18 smithi106 ceph-mon[44423]: from='client.26728 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:33:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:18 smithi106 ceph-mon[44423]: osdmap e617: 12 total, 11 up, 11 in 2024-04-20T17:33:19.281 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:18 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:33:19.281 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:18 smithi171 ceph-mon[44189]: from='client.26728 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:33:19.282 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:18 smithi171 ceph-mon[44189]: osdmap e617: 12 total, 11 up, 11 in 2024-04-20T17:33:19.284 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:19.306 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:33:19.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:19.309 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:33:19.410 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:33:19.438 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:19.459 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:33:19.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:19.462 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:33:19.563 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:33:19.594 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:19 smithi005 ceph-mon[39562]: pgmap v1623: 1 pgs: 1 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:20.243 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:33:20.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:20.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:33:20.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:19 smithi106 ceph-mon[44423]: pgmap v1623: 1 pgs: 1 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:19 smithi171 ceph-mon[44189]: pgmap v1623: 1 pgs: 1 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:20.348 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:33:20.376 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:20.396 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:33:20.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:20.398 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:33:20.499 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:33:20.527 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:20.546 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:33:20.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:20.549 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:33:20.650 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:33:20.684 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:20.706 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:33:20.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:20.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:33:20.810 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:33:20.838 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:20.857 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:33:20.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:20.859 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:33:20.960 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:33:20.991 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:20 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:33:20.991 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:20 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:33:20.993 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:21.014 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:33:21.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:21.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:33:21.112 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-20T17:33:21.118 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:33:21.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:20 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:33:21.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:20 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:33:21.169 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:21.175 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:33:21.244 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:33:21.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:21.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:33:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:20 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:33:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:20 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:33:21.348 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:33:21.379 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:21.399 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:33:21.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:21.402 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:33:21.503 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:33:21.531 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:21.675 DEBUG:teuthology.orchestra.run.smithi005:> sudo pkill -f 'journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.1.service' 2024-04-20T17:33:21.680 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:33:21.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:21.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:33:21.720 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T17:33:21.720 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-20T17:33:21.720 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd down 1 2024-04-20T17:33:21.785 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:33:21.815 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:21.835 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:33:21.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:21.839 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:33:21.939 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:33:21.973 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:21.994 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:33:21.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:21.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:33:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:21 smithi171 ceph-mon[44189]: pgmap v1624: 1 pgs: 1 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:22.098 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:33:22.125 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:21 smithi005 ceph-mon[39562]: pgmap v1624: 1 pgs: 1 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:22.128 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:21 smithi106 ceph-mon[44423]: pgmap v1624: 1 pgs: 1 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:22.326 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:33:22.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:22.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:33:22.431 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:33:22.463 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:22.485 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:33:22.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:22.488 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:33:22.589 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:33:22.604 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:33:22.608 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:33:22.614 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:33:22.618 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:22.636 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:33:22.638 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:33:22.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:22.642 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:33:22.656 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:33:22.677 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:33:22.697 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:33:22.717 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:33:22.741 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:33:22.742 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:33:22.765 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:33:22.772 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:22.786 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:33:22.807 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:33:22.827 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:33:22.827 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:33:22.828 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-20T17:33:22.828 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:33:22.828 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:33:22.828 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:33:22.828 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:33:22.828 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:33:22.828 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:33:22.828 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:33:22.828 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:33:22.828 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:33:22.828 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:33:22.846 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:33:22.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:22.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:33:22.951 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:33:22.980 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:23.078 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:33:23.079 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:22 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:33:23.080 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:22 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:33:23.080 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:22 smithi005 ceph-mon[39562]: osdmap e618: 12 total, 11 up, 11 in 2024-04-20T17:33:23.080 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:22 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/485933833' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:33:23.080 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:22 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:23.080 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:22 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:23.080 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:22 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:23.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:23.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:33:23.182 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:33:23.209 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:22 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:33:23.210 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:22 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:33:23.210 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:22 smithi171 ceph-mon[44189]: osdmap e618: 12 total, 11 up, 11 in 2024-04-20T17:33:23.210 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:22 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/485933833' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:33:23.210 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:22 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:23.210 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:22 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:23.210 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:22 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:23.211 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:23.232 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:33:23.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:23.235 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:33:23.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:22 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:33:23.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:22 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:33:23.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:22 smithi106 ceph-mon[44423]: osdmap e618: 12 total, 11 up, 11 in 2024-04-20T17:33:23.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:22 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/485933833' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:33:23.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:22 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:23.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:22 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:23.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:22 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:23.335 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:33:23.364 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:23.563 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:33:23.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:23.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:33:23.574 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:33:23.574 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:33:23.667 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:33:23.700 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:23.721 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:33:23.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:23.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:33:23.825 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:33:23.854 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:23.908 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:33:23.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:23.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:33:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:23 smithi106 ceph-mon[44423]: pgmap v1626: 1 pgs: 1 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:24.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:24.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:24.015 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:33:24.048 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:24.068 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:33:24.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:24.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:33:24.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:23 smithi005 ceph-mon[39562]: pgmap v1626: 1 pgs: 1 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:24.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:24.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:24.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:24.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:24.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:24.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:24.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:24.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:24.173 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:33:24.204 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:24.225 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:33:24.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:24.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:33:24.330 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:33:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:23 smithi171 ceph-mon[44189]: pgmap v1626: 1 pgs: 1 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:24.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:24.358 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:25.575 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:33:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:25 smithi005 ceph-mon[39562]: pgmap v1627: 1 pgs: 1 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:25 smithi106 ceph-mon[44423]: pgmap v1627: 1 pgs: 1 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:25 smithi171 ceph-mon[44189]: pgmap v1627: 1 pgs: 1 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:26.360 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:33:26.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:26.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:33:26.465 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:33:26.497 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:26.519 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:33:26.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:26.523 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:33:26.549 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:33:26.624 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:33:26.652 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:27.003 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:33:27.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:27.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:33:27.107 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:33:27.135 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:27.208 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:33:27.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:27.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:33:27.313 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:33:27.346 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:27.348 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'unique_pool_4' created 2024-04-20T17:33:27.367 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:33:27.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:27.370 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:33:27.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:27 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1716796746' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-04-20T17:33:27.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:27 smithi005 ceph-mon[39562]: pgmap v1628: 1 pgs: 1 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:27.471 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:33:27.503 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:27.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:27 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1716796746' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-04-20T17:33:27.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:27 smithi106 ceph-mon[44423]: pgmap v1628: 1 pgs: 1 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:27 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1716796746' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-04-20T17:33:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:27 smithi171 ceph-mon[44189]: pgmap v1628: 1 pgs: 1 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:28.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:28 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1716796746' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-04-20T17:33:28.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:28 smithi005 ceph-mon[39562]: osdmap e619: 12 total, 11 up, 11 in 2024-04-20T17:33:28.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:28 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3455033106' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:33:28.486 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:33:28.486 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:33:28.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:28 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1716796746' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-04-20T17:33:28.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:28 smithi106 ceph-mon[44423]: osdmap e619: 12 total, 11 up, 11 in 2024-04-20T17:33:28.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:28 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3455033106' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:33:28.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:28 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1716796746' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-04-20T17:33:28.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:28 smithi171 ceph-mon[44189]: osdmap e619: 12 total, 11 up, 11 in 2024-04-20T17:33:28.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:28 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3455033106' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:33:28.953 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:33:28.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:28.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:33:29.055 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:33:29.058 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:33:29.090 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:29.112 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:33:29.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:29.115 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:33:29.216 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:33:29.249 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:29.271 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:33:29.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:29.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:33:29.375 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:33:29.404 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:29.423 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:33:29.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:29.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:33:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:29 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:33:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:29 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3455033106' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:33:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:29 smithi106 ceph-mon[44423]: osdmap e620: 12 total, 11 up, 11 in 2024-04-20T17:33:29.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:29 smithi106 ceph-mon[44423]: pgmap v1631: 17 pgs: 7 creating+peering, 1 active+clean, 9 unknown; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:29.527 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:33:29.560 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:29.581 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:33:29.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:29.585 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:33:29.656 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:29 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:33:29.656 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:29 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3455033106' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:33:29.656 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:29 smithi005 ceph-mon[39562]: osdmap e620: 12 total, 11 up, 11 in 2024-04-20T17:33:29.656 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:29 smithi005 ceph-mon[39562]: pgmap v1631: 17 pgs: 7 creating+peering, 1 active+clean, 9 unknown; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:29.686 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:33:29.713 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:29 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:33:29.714 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:29 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3455033106' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:33:29.714 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:29 smithi171 ceph-mon[44189]: osdmap e620: 12 total, 11 up, 11 in 2024-04-20T17:33:29.714 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:29 smithi171 ceph-mon[44189]: pgmap v1631: 17 pgs: 7 creating+peering, 1 active+clean, 9 unknown; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:29.716 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:29.736 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:33:29.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:29.740 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:33:29.841 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:33:29.873 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:29.895 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:33:29.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:29.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:33:29.999 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:33:30.029 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:30.048 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:33:30.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:30.051 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:33:30.077 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-20T17:33:30.152 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:33:30.181 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:30.199 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:33:30.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:30.202 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:33:30.303 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:33:30.331 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:30.349 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:33:30.352 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:33:30.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:30.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:33:30.417 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:33:30.455 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:33:30.482 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:30.579 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:33:30.581 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:30 smithi106 ceph-mon[44423]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-20T17:33:30.581 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:30 smithi106 ceph-mon[44423]: osdmap e621: 12 total, 11 up, 11 in 2024-04-20T17:33:30.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:30.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:33:30.684 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:33:30.696 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:30 smithi005 ceph-mon[39562]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-20T17:33:30.696 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:30 smithi005 ceph-mon[39562]: osdmap e621: 12 total, 11 up, 11 in 2024-04-20T17:33:30.711 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:30 smithi171 ceph-mon[44189]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-20T17:33:30.712 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:30 smithi171 ceph-mon[44189]: osdmap e621: 12 total, 11 up, 11 in 2024-04-20T17:33:30.713 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:30.727 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-04-20T17:33:30.733 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:33:30.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:30.736 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:33:30.837 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:33:30.865 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:30.896 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:33:30.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:30.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:33:31.010 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:33:31.040 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:31.061 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:33:31.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:31.064 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:33:31.166 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:33:31.194 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:31.292 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:33:31.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:31.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:33:31.396 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:33:31.427 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:31.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:31 smithi005 ceph-mon[39562]: pgmap v1633: 17 pgs: 7 creating+peering, 5 active+clean, 5 unknown; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:31.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:31 smithi106 ceph-mon[44423]: pgmap v1633: 17 pgs: 7 creating+peering, 5 active+clean, 5 unknown; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:31.684 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:33:31.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:31 smithi171 ceph-mon[44189]: pgmap v1633: 17 pgs: 7 creating+peering, 5 active+clean, 5 unknown; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:31.977 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:33:31.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:31.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:33:32.081 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:33:32.109 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:32.138 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:33:32.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:32.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:33:32.242 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:33:32.263 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:33:32.263 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:33:32.272 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1 is already down. 2024-04-20T17:33:32.274 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:32.295 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:33:32.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:32.298 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:33:32.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:32 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mon-a[39539]: 2024-04-20T17:33:32.268+0000 7f73b92a1700 -1 mon.a@0(leader).osd e621 definitely_dead 0 2024-04-20T17:33:32.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:32 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3207842253' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["1"]}]: dispatch 2024-04-20T17:33:32.399 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:33:32.428 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:32.449 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:33:32.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:32.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:33:32.553 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:33:32.582 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:32 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3207842253' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["1"]}]: dispatch 2024-04-20T17:33:32.583 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:32 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3207842253' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["1"]}]: dispatch 2024-04-20T17:33:32.981 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-20T17:33:33.011 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:33:33.032 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:33:33.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:33.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:33:33.136 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:33:33.163 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:33.189 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:33:33.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:33.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:33:33.234 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:33:33.293 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:33:33.297 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:33:33.325 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:33.346 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:33:33.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:33.349 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:33:33.451 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:33:33.480 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:33.500 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:33:33.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:33.504 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:33:33.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:33 smithi171 ceph-mon[44189]: pgmap v1634: 17 pgs: 7 creating+peering, 3 undersized+peered, 7 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:33.605 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:33:33.623 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:33 smithi106 ceph-mon[44423]: pgmap v1634: 17 pgs: 7 creating+peering, 3 undersized+peered, 7 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:33.634 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:33.652 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:33:33.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:33.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:33:33.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:33 smithi005 ceph-mon[39562]: pgmap v1634: 17 pgs: 7 creating+peering, 3 undersized+peered, 7 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:33.756 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:33:33.786 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:33.807 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:33:33.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:33.810 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:33:33.911 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:33:33.939 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:33.958 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:33:33.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:33.962 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:33:34.063 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:33:34.098 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:34.198 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:33:34.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:34.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:33:34.302 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:33:34.330 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:34.678 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:33:34.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:34.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:33:34.782 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:33:34.809 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:34.956 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:33:34.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:34.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:33:35.060 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:33:35.093 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:35.114 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:33:35.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:35.117 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:33:35.217 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:33:35.219 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:33:35.219 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:33:35.219 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-04-20T17:33:35.219 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-04-20T17:33:35.219 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:33:35.248 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:35 smithi106 ceph-mon[44423]: pgmap v1635: 17 pgs: 3 undersized+peered, 14 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:35 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3971151414' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:33:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:35.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:35.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:35 smithi005 ceph-mon[39562]: pgmap v1635: 17 pgs: 3 undersized+peered, 14 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:35.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:35 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3971151414' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:33:35.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:35.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:35.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:35 smithi171 ceph-mon[44189]: pgmap v1635: 17 pgs: 3 undersized+peered, 14 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:35 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3971151414' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:33:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:35.895 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-20T17:33:36.159 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:33:36.439 INFO:teuthology.orchestra.run.smithi005.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-04-20T17:33:36.499 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:33:36.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:36.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:33:36.623 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:33:36.656 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:36.677 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:33:36.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:36.680 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:33:36.692 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:36.692 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:36 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:33:36.692 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:36 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:33:36.693 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:36.693 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:36.693 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:36.693 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:36.693 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:36.693 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:36.693 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:36.693 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:36 smithi106 ceph-mon[44423]: osdmap e622: 12 total, 11 up, 11 in 2024-04-20T17:33:36.693 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:36 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1799098363' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:33:36.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:36.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:36 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:33:36.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:36 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:33:36.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:36.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:36.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:36.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:36.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:36.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:36.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:36.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:36 smithi005 ceph-mon[39562]: osdmap e622: 12 total, 11 up, 11 in 2024-04-20T17:33:36.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:36 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1799098363' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:33:36.754 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:33:36.754 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:33:36.754 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:33:36.754 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:33:36.754 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:33:36.754 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:33:36.754 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:33:36.754 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:33:36.754 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:33:36.754 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:33:36.755 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:33:36.755 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:33:36.755 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-20T17:33:36.755 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:33:36.755 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:33:36.756 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:33:36.756 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:33:36.756 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:33:36.756 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:33:36.756 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:33:36.756 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:33:36.756 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:33:36.756 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:33:36.781 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:33:36.810 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:36.810 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:36 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:33:36.810 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:36 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:33:36.810 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:36.811 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:36.811 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:36.811 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:36.811 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:36.811 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:36.811 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:36.811 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:36 smithi171 ceph-mon[44189]: osdmap e622: 12 total, 11 up, 11 in 2024-04-20T17:33:36.811 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:36 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1799098363' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:33:36.812 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:36.833 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:33:36.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:36.836 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:33:36.937 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:33:36.965 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:36.985 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:33:36.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:36.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:33:37.089 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:33:37.121 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:37.322 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:33:37.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:37.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:33:37.331 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 3, 4, 5, 8, 9, 10, 11, 7] 2024-04-20T17:33:37.331 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd out 1 2024-04-20T17:33:37.426 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:33:37.446 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:33:37.453 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:37.475 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:33:37.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:37.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:33:37.580 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:33:37.612 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:37.635 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:33:37.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:37.638 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:33:37.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:37 smithi005 ceph-mon[39562]: pgmap v1637: 17 pgs: 3 undersized+peered, 14 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:37.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:37 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1799098363' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-20T17:33:37.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:37 smithi005 ceph-mon[39562]: osdmap e623: 12 total, 11 up, 11 in 2024-04-20T17:33:37.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:37 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1845282361' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:33:37.723 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:37 smithi106 ceph-mon[44423]: pgmap v1637: 17 pgs: 3 undersized+peered, 14 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:37.723 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:37 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1799098363' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-20T17:33:37.723 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:37 smithi106 ceph-mon[44423]: osdmap e623: 12 total, 11 up, 11 in 2024-04-20T17:33:37.723 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:37 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1845282361' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:33:37.739 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:33:37.767 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:37 smithi171 ceph-mon[44189]: pgmap v1637: 17 pgs: 3 undersized+peered, 14 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:37 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1799098363' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-20T17:33:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:37 smithi171 ceph-mon[44189]: osdmap e623: 12 total, 11 up, 11 in 2024-04-20T17:33:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:37 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1845282361' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:33:38.016 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:33:38.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:38.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:33:38.025 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:33:38.025 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-04-20T17:33:38.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-04-20T17:33:38.026 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [] 2024-04-20T17:33:38.026 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:33:38.120 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:33:38.148 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:38.185 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:33:38.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:38.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:33:38.209 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:33:38.209 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:33:38.288 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:33:38.315 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:38.640 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:38 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:33:38.641 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:38 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1845282361' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:33:38.641 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:38 smithi106 ceph-mon[44423]: osdmap e624: 12 total, 11 up, 11 in 2024-04-20T17:33:38.641 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:33:38.641 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:33:38.641 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:33:38.641 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:33:38.641 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:33:38.641 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:33:38.641 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:33:38.663 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:33:38.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:38.666 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-20T17:33:38.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:33:38.693 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:38 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:33:38.694 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:38 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1845282361' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:33:38.694 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:38 smithi005 ceph-mon[39562]: osdmap e624: 12 total, 11 up, 11 in 2024-04-20T17:33:38.694 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:33:38.694 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:33:38.694 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:33:38.694 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:33:38.694 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:33:38.694 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:33:38.694 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:33:38.767 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:33:38.794 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:38.811 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:33:38.816 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:33:38.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:38.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:33:38.825 DEBUG:teuthology.orchestra.run.smithi171:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-04-20T17:33:38.827 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-20T17:33:38.827 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-20T17:33:38.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:38 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:33:38.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:38 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1845282361' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:33:38.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:38 smithi171 ceph-mon[44189]: osdmap e624: 12 total, 11 up, 11 in 2024-04-20T17:33:38.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:33:38.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:33:38.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:33:38.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:33:38.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:33:38.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:33:38.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:33:38.921 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:33:38.924 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:hints = 1 2024-04-20T17:33:38.924 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-20T17:33:38.924 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Object prefix: benchmark_data_smithi171_147972 2024-04-20T17:33:38.925 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:33:38.925 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 0 0 0 0 0 0 - 0 2024-04-20T17:33:38.951 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:38.970 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:33:38.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:38.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:33:39.073 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:33:39.101 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:39.120 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:33:39.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:39.123 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:33:39.224 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:33:39.253 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:39.601 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:33:39.603 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:39 smithi005 ceph-mon[39562]: pgmap v1640: 17 pgs: 3 undersized+peered, 14 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:39.604 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:39 smithi005 ceph-mon[39562]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-20T17:33:39.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:39.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:33:39.605 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:33:39.706 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:33:39.737 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:39.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:39 smithi106 ceph-mon[44423]: pgmap v1640: 17 pgs: 3 undersized+peered, 14 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:39.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:39 smithi106 ceph-mon[44423]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-20T17:33:39.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:39 smithi171 ceph-mon[44189]: pgmap v1640: 17 pgs: 3 undersized+peered, 14 active+clean; 577 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail 2024-04-20T17:33:39.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:39 smithi171 ceph-mon[44189]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-20T17:33:39.835 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:33:39.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:39.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:33:39.925 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 1 16 58 42 2.62469 2.625 0.00126068 0.00203078 2024-04-20T17:33:39.940 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:33:39.969 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:40.210 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:33:40.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:40 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:33:40.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:40 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:33:40.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:40 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:33:40.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:40 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:33:40.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:40 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:33:40.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:40 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:33:40.846 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:33:40.925 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 2 16 58 42 1.31232 0 - 0.00203078 2024-04-20T17:33:41.318 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:33:41.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:41.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:33:41.422 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:33:41.451 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:41.664 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:33:41.665 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:41 smithi005 ceph-mon[39562]: pgmap v1641: 17 pgs: 3 undersized+peered, 14 active+clean; 769 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 32 KiB/s wr, 0 op/s 2024-04-20T17:33:41.665 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:41 smithi005 ceph-mon[39562]: osdmap e625: 12 total, 11 up, 11 in 2024-04-20T17:33:41.665 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:41 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mon-a[39539]: 2024-04-20T17:33:41.535+0000 7f73b92a1700 -1 mon.a@0(leader).osd e625 definitely_dead 0 2024-04-20T17:33:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:41 smithi106 ceph-mon[44423]: pgmap v1641: 17 pgs: 3 undersized+peered, 14 active+clean; 769 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 32 KiB/s wr, 0 op/s 2024-04-20T17:33:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:41 smithi106 ceph-mon[44423]: osdmap e625: 12 total, 11 up, 11 in 2024-04-20T17:33:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:41 smithi171 ceph-mon[44189]: pgmap v1641: 17 pgs: 3 undersized+peered, 14 active+clean; 769 KiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 32 KiB/s wr, 0 op/s 2024-04-20T17:33:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:41 smithi171 ceph-mon[44189]: osdmap e625: 12 total, 11 up, 11 in 2024-04-20T17:33:41.900 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:33:41.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:41.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:33:41.925 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 3 16 58 42 0.874874 0 - 0.00203078 2024-04-20T17:33:42.005 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:33:42.038 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:42.061 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:33:42.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:42.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:33:42.166 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:33:42.198 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:42.219 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:33:42.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:42.222 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:33:42.323 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:33:42.353 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:42.374 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:33:42.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:42.377 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:33:42.455 INFO:teuthology.orchestra.run.smithi005.stderr:marked out osd.1. 2024-04-20T17:33:42.478 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:33:42.492 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:42 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1061440536' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-04-20T17:33:42.492 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:42 smithi005 ceph-mon[39562]: Client client.admin marked osd.1 out, after it was down for 31 seconds 2024-04-20T17:33:42.492 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:42 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:33:42.492 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:42 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:33:42.511 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:42.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:42 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1061440536' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-04-20T17:33:42.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:42 smithi106 ceph-mon[44423]: Client client.admin marked osd.1 out, after it was down for 31 seconds 2024-04-20T17:33:42.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:42 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:33:42.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:42 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:33:42.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:42 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1061440536' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-04-20T17:33:42.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:42 smithi171 ceph-mon[44189]: Client client.admin marked osd.1 out, after it was down for 31 seconds 2024-04-20T17:33:42.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:42 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:33:42.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:42 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:33:42.926 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 4 16 58 42 0.656158 0 - 0.00203078 2024-04-20T17:33:43.327 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.1 2024-04-20T17:33:43.327 INFO:tasks.thrashosds.thrasher:remote for osd 1 is ubuntu@smithi005.front.sepia.ceph.com 2024-04-20T17:33:43.327 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.1 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op list-pgs 2024-04-20T17:33:43.361 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:33:43.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:43.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:33:43.453 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:33:43.453 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:33:43.453 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-04-20T17:33:43.453 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:33:43.465 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:33:43.498 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:43.520 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:33:43.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:43.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:33:43.624 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:33:43.661 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:43.665 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.1/config 2024-04-20T17:33:43.688 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:33:43.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:43.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:33:43.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:43 smithi005 ceph-mon[39562]: pgmap v1643: 17 pgs: 3 undersized+peered, 14 active+clean; 1.3 MiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 117 KiB/s wr, 1 op/s 2024-04-20T17:33:43.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:43 smithi005 ceph-mon[39562]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T17:33:43.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:43 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1061440536' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-04-20T17:33:43.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:43 smithi005 ceph-mon[39562]: osdmap e626: 12 total, 11 up, 10 in 2024-04-20T17:33:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:43 smithi106 ceph-mon[44423]: pgmap v1643: 17 pgs: 3 undersized+peered, 14 active+clean; 1.3 MiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 117 KiB/s wr, 1 op/s 2024-04-20T17:33:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:43 smithi106 ceph-mon[44423]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T17:33:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:43 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1061440536' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-04-20T17:33:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:43 smithi106 ceph-mon[44423]: osdmap e626: 12 total, 11 up, 10 in 2024-04-20T17:33:43.793 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:33:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:43 smithi171 ceph-mon[44189]: pgmap v1643: 17 pgs: 3 undersized+peered, 14 active+clean; 1.3 MiB data, 3.5 GiB used, 980 GiB / 983 GiB avail; 117 KiB/s wr, 1 op/s 2024-04-20T17:33:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:43 smithi171 ceph-mon[44189]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T17:33:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:43 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1061440536' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-04-20T17:33:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:43 smithi171 ceph-mon[44189]: osdmap e626: 12 total, 11 up, 10 in 2024-04-20T17:33:43.837 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:43.926 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 5 16 2271 2255 28.1822 34.5781 0.00198088 0.0354688 2024-04-20T17:33:44.096 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:33:44.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:44.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:33:44.201 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:33:44.237 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:44.263 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:33:44.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:44.267 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:33:44.369 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:33:44.454 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:44.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:44 smithi005 ceph-mon[39562]: osdmap e627: 12 total, 11 up, 10 in 2024-04-20T17:33:44.481 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:33:44.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:44.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:33:44.552 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-20T17:33:44.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:44 smithi171 ceph-mon[44189]: osdmap e627: 12 total, 11 up, 10 in 2024-04-20T17:33:44.586 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:33:44.620 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:44.648 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:33:44.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:44.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:33:44.665 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:33:44.753 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:33:44.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:44 smithi106 ceph-mon[44423]: osdmap e627: 12 total, 11 up, 10 in 2024-04-20T17:33:44.833 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:44.884 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:33:44.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:44.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:33:44.926 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 6 16 7522 7506 78.1742 328.188 0.00871657 0.0127842 2024-04-20T17:33:44.993 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:33:45.028 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:45.052 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:33:45.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:45.055 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:33:45.156 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:33:45.192 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:45.219 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:33:45.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:45.222 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:33:45.322 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:33:45.323 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:33:45.364 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:33:45.367 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:45.404 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:33:45.449 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:33:45.482 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:33:45.522 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:33:45.564 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:33:45.603 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:33:45.642 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:33:45.679 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:33:45.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:45 smithi005 ceph-mon[39562]: pgmap v1646: 17 pgs: 1 creating+peering, 1 undersized+peered, 15 active+clean; 27 MiB data, 3.0 GiB used, 891 GiB / 894 GiB avail; 4.4 MiB/s wr, 70 op/s 2024-04-20T17:33:45.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:45 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2329457937' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:33:45.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:45.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:45 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:33:45.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:45 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:33:45.713 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:33:45.713 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:33:45.713 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-20T17:33:45.713 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:33:45.713 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:33:45.714 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:33:45.714 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:33:45.714 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:33:45.714 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:33:45.714 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:33:45.714 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:33:45.714 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:33:45.714 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:33:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:45 smithi106 ceph-mon[44423]: pgmap v1646: 17 pgs: 1 creating+peering, 1 undersized+peered, 15 active+clean; 27 MiB data, 3.0 GiB used, 891 GiB / 894 GiB avail; 4.4 MiB/s wr, 70 op/s 2024-04-20T17:33:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:45 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2329457937' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:33:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:45 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:33:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:45 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:33:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:45 smithi171 ceph-mon[44189]: pgmap v1646: 17 pgs: 1 creating+peering, 1 undersized+peered, 15 active+clean; 27 MiB data, 3.0 GiB used, 891 GiB / 894 GiB avail; 4.4 MiB/s wr, 70 op/s 2024-04-20T17:33:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:45 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2329457937' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:33:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:45 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:33:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:45 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:33:45.926 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 7 16 12343 12327 110.045 301.312 0.0047117 0.00908445 2024-04-20T17:33:46.370 INFO:teuthology.orchestra.run.smithi005.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:33:46.540 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:33:46.540 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:33:46.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:46.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:33:46.646 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:33:46.682 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:46.711 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:33:46.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:46.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:46.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:46.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:46 smithi005 ceph-mon[39562]: osdmap e628: 12 total, 11 up, 10 in 2024-04-20T17:33:46.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:46.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:46.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:46.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:46.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:46.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:46.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:46.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:33:46.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:46.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:46.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:46.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:46 smithi106 ceph-mon[44423]: osdmap e628: 12 total, 11 up, 10 in 2024-04-20T17:33:46.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:46.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:46.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:46.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:46.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:46.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:46.816 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:33:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:46 smithi171 ceph-mon[44189]: osdmap e628: 12 total, 11 up, 10 in 2024-04-20T17:33:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:46.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:46.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:46.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:46.854 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:46.926 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 8 16 17145 17129 133.801 300.125 0.00217942 0.00747132 2024-04-20T17:33:47.318 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:33:47.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:47.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:33:47.341 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:33:47.342 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:33:47.427 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:33:47.471 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:47.519 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:33:47.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:47.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:33:47.544 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:47.545 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:47 smithi005 ceph-mon[39562]: pgmap v1648: 17 pgs: 1 creating+peering, 16 active+clean; 165 MiB data, 3.6 GiB used, 890 GiB / 894 GiB avail; 28 MiB/s wr, 441 op/s 2024-04-20T17:33:47.625 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:33:47.651 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:47.669 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:33:47.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:47.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:33:47.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:47.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:47 smithi106 ceph-mon[44423]: pgmap v1648: 17 pgs: 1 creating+peering, 16 active+clean; 165 MiB data, 3.6 GiB used, 890 GiB / 894 GiB avail; 28 MiB/s wr, 441 op/s 2024-04-20T17:33:47.772 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:33:47.806 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:47.806 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:47 smithi171 ceph-mon[44189]: pgmap v1648: 17 pgs: 1 creating+peering, 16 active+clean; 165 MiB data, 3.6 GiB used, 890 GiB / 894 GiB avail; 28 MiB/s wr, 441 op/s 2024-04-20T17:33:47.808 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:47.834 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:33:47.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:47.838 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:33:47.926 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 9 16 21006 20990 145.744 241.312 0.00204625 0.00685943 2024-04-20T17:33:47.939 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:33:47.972 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:47.999 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:33:48.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:48.003 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:33:48.104 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:33:48.142 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:48.164 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:33:48.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:48.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:33:48.268 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:33:48.320 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:48.926 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 10 16 25194 25178 157.342 261.75 0.0041153 0.00635343 2024-04-20T17:33:49.343 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:33:49.730 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:49 smithi005 ceph-mon[39562]: pgmap v1649: 17 pgs: 17 active+clean; 898 MiB data, 4.5 GiB used, 889 GiB / 894 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-04-20T17:33:49.730 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:49 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:33:49.731 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:49 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:33:49.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:49 smithi106 ceph-mon[44423]: pgmap v1649: 17 pgs: 17 active+clean; 898 MiB data, 4.5 GiB used, 889 GiB / 894 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-04-20T17:33:49.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:49 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:33:49.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:49 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:33:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:49 smithi171 ceph-mon[44189]: pgmap v1649: 17 pgs: 17 active+clean; 898 MiB data, 4.5 GiB used, 889 GiB / 894 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-04-20T17:33:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:49 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:33:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:49 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:33:49.926 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 11 16 28800 28784 163.525 225.375 0.00440325 0.0061139 2024-04-20T17:33:50.039 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:33:50.039 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:33:50.272 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:33:50.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:50.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:33:50.294 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:33:50.379 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:33:50.416 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:50.447 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:33:50.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:50.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:33:50.512 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:33:50.553 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:33:50.587 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:50.610 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:33:50.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:50.615 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:33:50.662 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:50 smithi005 ceph-mon[39562]: osdmap e629: 12 total, 11 up, 10 in 2024-04-20T17:33:50.662 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:50 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2239039553' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:33:50.662 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:50 smithi005 ceph-mon[39562]: from='client.26797 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:33:50.715 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:33:50.750 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:50 smithi106 ceph-mon[44423]: osdmap e629: 12 total, 11 up, 10 in 2024-04-20T17:33:50.750 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:50 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2239039553' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:33:50.750 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:50 smithi106 ceph-mon[44423]: from='client.26797 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:33:50.752 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:50.781 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:33:50.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:50.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:33:50.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:50 smithi171 ceph-mon[44189]: osdmap e629: 12 total, 11 up, 10 in 2024-04-20T17:33:50.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:50 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2239039553' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:33:50.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:50 smithi171 ceph-mon[44189]: from='client.26797 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:33:50.886 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:33:50.926 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 12 16 32157 32141 167.38 209.812 0.00476131 0.00597268 2024-04-20T17:33:50.927 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:50.952 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:33:50.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:50.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:33:51.056 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:33:51.093 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:51.122 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:33:51.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:51.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:33:51.228 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:33:51.260 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:51.284 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:33:51.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:51.288 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:33:51.375 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-20T17:33:51.389 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:33:51.425 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:51.452 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:33:51.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:51.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:33:51.557 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:33:51.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:51 smithi171 ceph-mon[44189]: pgmap v1651: 17 pgs: 17 active+clean; 1.3 GiB data, 5.3 GiB used, 889 GiB / 894 GiB avail; 193 MiB/s wr, 3.08k op/s 2024-04-20T17:33:51.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:51 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:33:51.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:51 smithi171 ceph-mon[44189]: from='client.26797 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:33:51.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:51 smithi171 ceph-mon[44189]: osdmap e630: 12 total, 11 up, 10 in 2024-04-20T17:33:51.589 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:51.611 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:33:51.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:51.614 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:33:51.715 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:33:51.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:51 smithi005 ceph-mon[39562]: pgmap v1651: 17 pgs: 17 active+clean; 1.3 GiB data, 5.3 GiB used, 889 GiB / 894 GiB avail; 193 MiB/s wr, 3.08k op/s 2024-04-20T17:33:51.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:51 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:33:51.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:51 smithi005 ceph-mon[39562]: from='client.26797 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:33:51.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:51 smithi005 ceph-mon[39562]: osdmap e630: 12 total, 11 up, 10 in 2024-04-20T17:33:51.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:51 smithi106 ceph-mon[44423]: pgmap v1651: 17 pgs: 17 active+clean; 1.3 GiB data, 5.3 GiB used, 889 GiB / 894 GiB avail; 193 MiB/s wr, 3.08k op/s 2024-04-20T17:33:51.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:51 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:33:51.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:51 smithi106 ceph-mon[44423]: from='client.26797 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:33:51.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:51 smithi106 ceph-mon[44423]: osdmap e630: 12 total, 11 up, 10 in 2024-04-20T17:33:51.789 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:51.926 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 13 15 35216 35201 169.215 191.25 0.00194118 0.00590707 2024-04-20T17:33:52.071 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:33:52.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:52.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:33:52.182 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:33:52.230 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:52.924 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:33:52.926 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 14 16 38271 38255 170.761 190.875 0.00260513 0.00585442 2024-04-20T17:33:52.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:52.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:33:53.033 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:33:53.082 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:53.447 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:33:53.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:53.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:33:53.526 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:33:53.553 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:33:53.582 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:53.608 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:33:53.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:53.612 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:33:53.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:53 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:33:53.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:53 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:33:53.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:53 smithi005 ceph-mon[39562]: pgmap v1653: 17 pgs: 17 active+clean; 1.5 GiB data, 6.3 GiB used, 888 GiB / 894 GiB avail; 204 MiB/s wr, 3.26k op/s 2024-04-20T17:33:53.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:53 smithi005 ceph-mon[39562]: osdmap e631: 12 total, 11 up, 10 in 2024-04-20T17:33:53.713 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:33:53.752 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:53.781 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:33:53.783 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:53 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:33:53.783 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:53 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:33:53.783 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:53 smithi106 ceph-mon[44423]: pgmap v1653: 17 pgs: 17 active+clean; 1.5 GiB data, 6.3 GiB used, 888 GiB / 894 GiB avail; 204 MiB/s wr, 3.26k op/s 2024-04-20T17:33:53.783 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:53 smithi106 ceph-mon[44423]: osdmap e631: 12 total, 11 up, 10 in 2024-04-20T17:33:53.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:53.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:33:53.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:53 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:33:53.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:53 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:33:53.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:53 smithi171 ceph-mon[44189]: pgmap v1653: 17 pgs: 17 active+clean; 1.5 GiB data, 6.3 GiB used, 888 GiB / 894 GiB avail; 204 MiB/s wr, 3.26k op/s 2024-04-20T17:33:53.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:53 smithi171 ceph-mon[44189]: osdmap e631: 12 total, 11 up, 10 in 2024-04-20T17:33:53.886 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:33:53.919 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:53.926 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 15 16 41129 41113 171.284 178.625 0.00418039 0.00583625 2024-04-20T17:33:53.931 INFO:teuthology.orchestra.run.smithi005.stdout:5.7 2024-04-20T17:33:53.932 INFO:teuthology.orchestra.run.smithi005.stdout:5.0 2024-04-20T17:33:53.932 INFO:teuthology.orchestra.run.smithi005.stdout:5.c 2024-04-20T17:33:53.932 INFO:teuthology.orchestra.run.smithi005.stdout:5.5 2024-04-20T17:33:53.945 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:33:53.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:53.948 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:33:54.049 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:33:54.084 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:54.109 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:33:54.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:54.113 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:33:54.215 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:33:54.249 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:54.279 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:33:54.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:54.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:33:54.384 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:33:54.389 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:33:54.430 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:33:54.438 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:54.473 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:33:54.493 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:33:54.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:54.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:33:54.514 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:33:54.545 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:33:54.551 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:33:54.589 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:33:54.608 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:33:54.626 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:33:54.642 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:54.656 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:33:54.665 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:33:54.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:54.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:33:54.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:54 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3392111314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:33:54.695 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:33:54.738 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:33:54.770 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:33:54.786 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:33:54.787 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:33:54.787 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-20T17:33:54.787 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:33:54.787 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:33:54.787 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:33:54.787 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:33:54.787 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:33:54.787 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:33:54.787 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:33:54.787 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:33:54.787 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:33:54.787 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:33:54.799 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:54 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3392111314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:33:54.801 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:54.824 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:33:54.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:54.827 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:33:54.926 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 16 16 43845 43829 171.187 169.75 0.00357864 0.00583989 2024-04-20T17:33:54.928 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:33:54.969 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:54 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3392111314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:33:54.974 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:54.999 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:33:55.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:55.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:33:55.105 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:33:55.149 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:55.411 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:33:55.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:33:55.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:33:55.517 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:33:55.558 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:33:55.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:55 smithi005 ceph-mon[39562]: pgmap v1655: 17 pgs: 17 active+clean; 2.2 GiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 221 MiB/s wr, 3.53k op/s 2024-04-20T17:33:55.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:55.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:55.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:55.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:55.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:55.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:55.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:55.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:55.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:55.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:55.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:55 smithi171 ceph-mon[44189]: pgmap v1655: 17 pgs: 17 active+clean; 2.2 GiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 221 MiB/s wr, 3.53k op/s 2024-04-20T17:33:55.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:55.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:55.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:55.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:55.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:55.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:55.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:55.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:55.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:55.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:55.927 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 17 15 46855 46840 172.186 188.188 0.00345827 0.00580579 2024-04-20T17:33:56.001 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.1 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op export --pgid 5.0 --file /var/log/ceph/exp.5.0.1 2024-04-20T17:33:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:55 smithi106 ceph-mon[44423]: pgmap v1655: 17 pgs: 17 active+clean; 2.2 GiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 221 MiB/s wr, 3.53k op/s 2024-04-20T17:33:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:56.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:56.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:56.927 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 18 16 49599 49583 172.144 171.438 0.00433602 0.00580745 2024-04-20T17:33:56.927 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:33:57.774 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:33:57.774 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:33:57.774 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:33:57.774 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:33:57.774 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.774 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.16506.0:144363 5.1 e586/561 5:82a7e0ba:::benchmark_data_smithi171_137281_object12929:head v 586'9073, mlcod=586'9071)", 2024-04-20T17:33:57.774 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:31:19.613135+0000", 2024-04-20T17:33:57.774 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 157.28901964100001, 2024-04-20T17:33:57.775 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.025346922000000001, 2024-04-20T17:33:57.775 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:33:57.775 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:33:57.775 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:57.775 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:33:57.775 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.775 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:33:57.775 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:19.613135+0000", 2024-04-20T17:33:57.775 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.775 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.775 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.775 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:33:57.775 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:19.613135+0000", 2024-04-20T17:33:57.775 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.776 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.776 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.776 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:33:57.776 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:19.613138+0000", 2024-04-20T17:33:57.776 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3470000000000001e-06 2024-04-20T17:33:57.776 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.776 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.776 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:33:57.776 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:19.613148+0000", 2024-04-20T17:33:57.776 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9890000000000004e-06 2024-04-20T17:33:57.776 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.776 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.777 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:33:57.777 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:19.613165+0000", 2024-04-20T17:33:57.777 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6864999999999999e-05 2024-04-20T17:33:57.777 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.777 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.777 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:33:57.777 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:19.613171+0000", 2024-04-20T17:33:57.777 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5590000000000002e-06 2024-04-20T17:33:57.777 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.777 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.777 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:33:57.777 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:19.618104+0000", 2024-04-20T17:33:57.778 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0049331870000000003 2024-04-20T17:33:57.778 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.778 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.778 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:33:57.778 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:19.618127+0000", 2024-04-20T17:33:57.778 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2515999999999999e-05 2024-04-20T17:33:57.778 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.778 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.778 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:33:57.778 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:19.638429+0000", 2024-04-20T17:33:57.778 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.020301876999999999 2024-04-20T17:33:57.778 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.779 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.779 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:33:57.779 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:19.638482+0000", 2024-04-20T17:33:57.779 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3582000000000003e-05 2024-04-20T17:33:57.779 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.779 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:33:57.779 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.779 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.779 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.779 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.16506.0:197905 5.1 e598/561 5:86558532:::benchmark_data_smithi171_137281_object66471:head v 598'12364, mlcod=598'12359)", 2024-04-20T17:33:57.779 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:04.017484+0000", 2024-04-20T17:33:57.780 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 112.884671349, 2024-04-20T17:33:57.780 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.027949141, 2024-04-20T17:33:57.780 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:33:57.780 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:33:57.780 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:57.780 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:33:57.780 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.780 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:33:57.780 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.017484+0000", 2024-04-20T17:33:57.780 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.780 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.781 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.781 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:33:57.781 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.017484+0000", 2024-04-20T17:33:57.781 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.781 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.781 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.781 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:33:57.782 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.017485+0000", 2024-04-20T17:33:57.782 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.753e-06 2024-04-20T17:33:57.782 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.782 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.782 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:33:57.782 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.017496+0000", 2024-04-20T17:33:57.782 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0497e-05 2024-04-20T17:33:57.782 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.782 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.782 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:33:57.782 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.017509+0000", 2024-04-20T17:33:57.782 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3196e-05 2024-04-20T17:33:57.783 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.783 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.783 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:33:57.783 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.017517+0000", 2024-04-20T17:33:57.783 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0530000000000007e-06 2024-04-20T17:33:57.783 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.783 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.783 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:33:57.783 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.022446+0000", 2024-04-20T17:33:57.783 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0049284639999999996 2024-04-20T17:33:57.783 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.783 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.784 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:33:57.784 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.022466+0000", 2024-04-20T17:33:57.784 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0193000000000001e-05 2024-04-20T17:33:57.784 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.784 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.784 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:33:57.784 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.045373+0000", 2024-04-20T17:33:57.784 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.022907317 2024-04-20T17:33:57.784 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.784 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.784 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:33:57.784 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.045433+0000", 2024-04-20T17:33:57.784 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9667999999999999e-05 2024-04-20T17:33:57.785 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.785 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:33:57.785 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.785 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.785 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.785 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.16506.0:252100 5.1 e609/561 5:82d31a37:::benchmark_data_smithi171_137281_object120666:head v 609'15734, mlcod=609'15730)", 2024-04-20T17:33:57.785 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:55.509742+0000", 2024-04-20T17:33:57.785 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 61.392413126000001, 2024-04-20T17:33:57.785 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.026482856999999999, 2024-04-20T17:33:57.785 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:33:57.785 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:33:57.785 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:57.786 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:33:57.786 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.786 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:33:57.786 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:55.509742+0000", 2024-04-20T17:33:57.786 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.786 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.786 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.786 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:33:57.786 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:55.509742+0000", 2024-04-20T17:33:57.786 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.786 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.786 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.786 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:33:57.787 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:55.509744+0000", 2024-04-20T17:33:57.787 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8819999999999999e-06 2024-04-20T17:33:57.787 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.787 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.787 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:33:57.787 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:55.509752+0000", 2024-04-20T17:33:57.787 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6030000000000005e-06 2024-04-20T17:33:57.787 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.787 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.787 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:33:57.787 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:55.509765+0000", 2024-04-20T17:33:57.787 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2319e-05 2024-04-20T17:33:57.787 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.788 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.788 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:33:57.788 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:55.509772+0000", 2024-04-20T17:33:57.788 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6720000000000004e-06 2024-04-20T17:33:57.788 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.788 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.788 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:33:57.788 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:55.510353+0000", 2024-04-20T17:33:57.788 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00058080299999999996 2024-04-20T17:33:57.788 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.788 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.789 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:33:57.789 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:55.510375+0000", 2024-04-20T17:33:57.789 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1795999999999999e-05 2024-04-20T17:33:57.789 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.789 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.789 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:33:57.789 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:55.536170+0000", 2024-04-20T17:33:57.789 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.025795493999999999 2024-04-20T17:33:57.789 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.789 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.789 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:33:57.789 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:55.536225+0000", 2024-04-20T17:33:57.790 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4287999999999999e-05 2024-04-20T17:33:57.790 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.790 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:33:57.790 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.790 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.790 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.790 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.16506.0:255441 5.1 e611/561 5:825fb458:::benchmark_data_smithi171_137281_object124007:head v 611'15949, mlcod=611'15947)", 2024-04-20T17:33:57.790 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:58.369939+0000", 2024-04-20T17:33:57.790 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 58.53221619, 2024-04-20T17:33:57.790 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.025391361000000001, 2024-04-20T17:33:57.790 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:33:57.791 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:33:57.791 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:57.791 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:33:57.791 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.791 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:33:57.791 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:58.369939+0000", 2024-04-20T17:33:57.791 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.791 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.791 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.791 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:33:57.791 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:58.369939+0000", 2024-04-20T17:33:57.791 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.791 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.791 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.791 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:33:57.792 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:58.369941+0000", 2024-04-20T17:33:57.792 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1459999999999999e-06 2024-04-20T17:33:57.792 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.792 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.792 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:33:57.792 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:58.369951+0000", 2024-04-20T17:33:57.792 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0443e-05 2024-04-20T17:33:57.792 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.792 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.792 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:33:57.792 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:58.369971+0000", 2024-04-20T17:33:57.792 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9766999999999999e-05 2024-04-20T17:33:57.792 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.792 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.793 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:33:57.793 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:58.369980+0000", 2024-04-20T17:33:57.793 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2720000000000003e-06 2024-04-20T17:33:57.793 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.793 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.793 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:33:57.793 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:58.372358+0000", 2024-04-20T17:33:57.793 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002377367 2024-04-20T17:33:57.793 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.793 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.793 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:33:57.793 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:58.372384+0000", 2024-04-20T17:33:57.793 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6154999999999999e-05 2024-04-20T17:33:57.793 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.793 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.794 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:33:57.794 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:58.395269+0000", 2024-04-20T17:33:57.794 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.022885312000000001 2024-04-20T17:33:57.794 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.794 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.794 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:33:57.794 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:58.395330+0000", 2024-04-20T17:33:57.794 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0899000000000001e-05 2024-04-20T17:33:57.794 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.794 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:33:57.794 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.794 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.794 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.794 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.16506.0:256685 5.1 e611/561 5:85de7af1:::benchmark_data_smithi171_137281_object125251:head v 611'16023, mlcod=611'16021)", 2024-04-20T17:33:57.795 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:59.487863+0000", 2024-04-20T17:33:57.795 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 57.414292469000003, 2024-04-20T17:33:57.795 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.025304838, 2024-04-20T17:33:57.795 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:33:57.795 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:33:57.795 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:57.795 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:33:57.795 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.795 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:33:57.795 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:59.487863+0000", 2024-04-20T17:33:57.795 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.795 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.795 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.795 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:33:57.795 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:59.487863+0000", 2024-04-20T17:33:57.796 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.796 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.796 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.796 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:33:57.796 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:59.487865+0000", 2024-04-20T17:33:57.796 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2579999999999998e-06 2024-04-20T17:33:57.796 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.796 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.796 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:33:57.796 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:59.487874+0000", 2024-04-20T17:33:57.796 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0110000000000006e-06 2024-04-20T17:33:57.796 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.796 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.796 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:33:57.796 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:59.487888+0000", 2024-04-20T17:33:57.797 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4588e-05 2024-04-20T17:33:57.797 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.797 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.797 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:33:57.797 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:59.487898+0000", 2024-04-20T17:33:57.797 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2119999999999992e-06 2024-04-20T17:33:57.797 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.797 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.797 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:33:57.797 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:59.488045+0000", 2024-04-20T17:33:57.797 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000147503 2024-04-20T17:33:57.797 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.797 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.797 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:33:57.797 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:59.488067+0000", 2024-04-20T17:33:57.798 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1889999999999999e-05 2024-04-20T17:33:57.798 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.798 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.798 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:33:57.798 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:59.513111+0000", 2024-04-20T17:33:57.798 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.025044132 2024-04-20T17:33:57.798 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.798 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.798 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:33:57.798 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:59.513167+0000", 2024-04-20T17:33:57.798 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6243999999999999e-05 2024-04-20T17:33:57.798 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.798 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:33:57.798 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.799 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.799 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.799 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.16506.0:256708 5.1 e611/561 5:814a016e:::benchmark_data_smithi171_137281_object125274:head v 611'16026, mlcod=611'16021)", 2024-04-20T17:33:57.799 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:59.509959+0000", 2024-04-20T17:33:57.799 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 57.392196327999997, 2024-04-20T17:33:57.799 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.030660063000000001, 2024-04-20T17:33:57.799 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:33:57.799 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:33:57.799 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:57.799 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:33:57.799 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.799 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:33:57.799 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:59.509959+0000", 2024-04-20T17:33:57.799 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.799 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.800 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.800 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:33:57.800 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:59.509959+0000", 2024-04-20T17:33:57.800 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.800 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.800 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.800 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:33:57.800 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:59.509960+0000", 2024-04-20T17:33:57.800 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2799999999999995e-07 2024-04-20T17:33:57.800 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.800 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.800 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:33:57.800 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:59.509966+0000", 2024-04-20T17:33:57.800 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3040000000000001e-06 2024-04-20T17:33:57.800 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.801 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.801 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:33:57.801 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:59.509976+0000", 2024-04-20T17:33:57.801 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0265000000000001e-05 2024-04-20T17:33:57.801 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.801 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.801 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:33:57.801 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:59.509983+0000", 2024-04-20T17:33:57.801 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5209999999999999e-06 2024-04-20T17:33:57.801 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.801 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.801 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:33:57.801 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:59.513241+0000", 2024-04-20T17:33:57.801 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0032583379999999999 2024-04-20T17:33:57.801 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.802 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.802 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:33:57.802 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:59.513267+0000", 2024-04-20T17:33:57.802 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6163999999999999e-05 2024-04-20T17:33:57.802 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.802 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.802 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:33:57.802 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:59.540570+0000", 2024-04-20T17:33:57.802 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.027302521 2024-04-20T17:33:57.802 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.803 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.803 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:33:57.803 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:59.540619+0000", 2024-04-20T17:33:57.803 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9122000000000002e-05 2024-04-20T17:33:57.803 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.803 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:33:57.803 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.803 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.803 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.803 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:4 6.4 6.e10b0ab4 (undecoded) ondisk+retry+write+known_if_redirected e626)", 2024-04-20T17:33:57.803 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:33:43.113943+0000", 2024-04-20T17:33:57.803 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 13.788211849, 2024-04-20T17:33:57.803 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.176808986, 2024-04-20T17:33:57.803 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:33:57.803 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:33:57.804 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-04-20T17:33:57.804 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:33:57.804 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:33:57.804 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:33:57.804 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 4 2024-04-20T17:33:57.804 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.804 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:33:57.804 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.804 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:33:57.804 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.113943+0000", 2024-04-20T17:33:57.804 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.804 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.804 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.804 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:33:57.804 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.113943+0000", 2024-04-20T17:33:57.805 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.805 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.805 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.805 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:33:57.805 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.113946+0000", 2024-04-20T17:33:57.805 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.954e-06 2024-04-20T17:33:57.805 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.805 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.805 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:33:57.805 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114087+0000", 2024-04-20T17:33:57.805 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000140672 2024-04-20T17:33:57.805 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.805 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.805 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:33:57.805 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114109+0000", 2024-04-20T17:33:57.806 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2314e-05 2024-04-20T17:33:57.806 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.806 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.806 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:33:57.806 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114125+0000", 2024-04-20T17:33:57.806 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.579e-05 2024-04-20T17:33:57.806 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.806 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.806 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:33:57.806 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290710+0000", 2024-04-20T17:33:57.806 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17658464300000001 2024-04-20T17:33:57.806 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.806 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.807 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:33:57.807 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290752+0000", 2024-04-20T17:33:57.807 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2613e-05 2024-04-20T17:33:57.807 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.807 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:33:57.807 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.807 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.807 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.807 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:5 6.4 6.d573554 (undecoded) ondisk+retry+write+known_if_redirected e626)", 2024-04-20T17:33:57.807 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:33:43.114145+0000", 2024-04-20T17:33:57.807 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 13.788010074000001, 2024-04-20T17:33:57.808 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17667517699999999, 2024-04-20T17:33:57.808 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:33:57.808 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:33:57.808 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-04-20T17:33:57.808 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:33:57.808 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:33:57.808 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:33:57.808 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 5 2024-04-20T17:33:57.808 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.808 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:33:57.808 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.809 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:33:57.809 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114145+0000", 2024-04-20T17:33:57.809 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.809 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.809 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.809 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:33:57.809 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114145+0000", 2024-04-20T17:33:57.809 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.809 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.809 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.809 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:33:57.810 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114151+0000", 2024-04-20T17:33:57.810 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0519999999999998e-06 2024-04-20T17:33:57.810 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.810 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.810 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:33:57.810 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114178+0000", 2024-04-20T17:33:57.810 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6842000000000002e-05 2024-04-20T17:33:57.810 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.811 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.811 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:33:57.811 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114183+0000", 2024-04-20T17:33:57.811 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4210000000000003e-06 2024-04-20T17:33:57.811 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.811 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.811 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:33:57.811 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114187+0000", 2024-04-20T17:33:57.811 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0940000000000001e-06 2024-04-20T17:33:57.811 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.811 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.811 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:33:57.811 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290813+0000", 2024-04-20T17:33:57.812 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17662583400000001 2024-04-20T17:33:57.812 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.812 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.812 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:33:57.812 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290820+0000", 2024-04-20T17:33:57.812 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.934e-06 2024-04-20T17:33:57.812 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.812 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:33:57.812 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.812 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.812 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.812 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:19 6.4 6.9dea5f14 (undecoded) ondisk+write+known_if_redirected e626)", 2024-04-20T17:33:57.813 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:33:43.114197+0000", 2024-04-20T17:33:57.813 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 13.787958249000001, 2024-04-20T17:33:57.813 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17665645799999999, 2024-04-20T17:33:57.813 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:33:57.813 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:33:57.813 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-04-20T17:33:57.813 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:33:57.813 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:33:57.813 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:33:57.813 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 19 2024-04-20T17:33:57.813 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.813 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:33:57.814 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.814 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:33:57.814 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114197+0000", 2024-04-20T17:33:57.814 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.814 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.814 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.814 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:33:57.814 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114197+0000", 2024-04-20T17:33:57.814 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.814 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.814 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.815 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:33:57.815 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114197+0000", 2024-04-20T17:33:57.815 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2399999999999998e-07 2024-04-20T17:33:57.815 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.815 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.815 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:33:57.815 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114282+0000", 2024-04-20T17:33:57.815 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4831000000000001e-05 2024-04-20T17:33:57.815 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.815 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.815 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:33:57.815 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114289+0000", 2024-04-20T17:33:57.815 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0790000000000003e-06 2024-04-20T17:33:57.815 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.815 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.816 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:33:57.816 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114293+0000", 2024-04-20T17:33:57.816 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9199999999999997e-06 2024-04-20T17:33:57.816 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.816 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.816 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:33:57.816 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290840+0000", 2024-04-20T17:33:57.816 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17654666999999999 2024-04-20T17:33:57.816 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.816 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.817 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:33:57.817 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290853+0000", 2024-04-20T17:33:57.817 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3434e-05 2024-04-20T17:33:57.817 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.817 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:33:57.817 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.817 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.817 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.817 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:30 6.4 6.853ac614 (undecoded) ondisk+write+known_if_redirected e626)", 2024-04-20T17:33:57.817 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:33:43.114301+0000", 2024-04-20T17:33:57.817 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 13.787853723, 2024-04-20T17:33:57.817 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.176585081, 2024-04-20T17:33:57.817 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:33:57.817 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:33:57.817 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-04-20T17:33:57.818 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:33:57.818 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:33:57.818 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:33:57.818 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 30 2024-04-20T17:33:57.818 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.818 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:33:57.818 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.818 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:33:57.818 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114301+0000", 2024-04-20T17:33:57.818 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.818 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.818 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.818 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:33:57.818 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114301+0000", 2024-04-20T17:33:57.818 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.819 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.819 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.819 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:33:57.819 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114302+0000", 2024-04-20T17:33:57.819 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5700000000000002e-07 2024-04-20T17:33:57.819 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.819 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.819 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:33:57.819 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114354+0000", 2024-04-20T17:33:57.819 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2394e-05 2024-04-20T17:33:57.819 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.819 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.819 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:33:57.819 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114361+0000", 2024-04-20T17:33:57.819 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4269999999999996e-06 2024-04-20T17:33:57.820 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.820 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.820 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:33:57.820 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114365+0000", 2024-04-20T17:33:57.820 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3699999999999997e-06 2024-04-20T17:33:57.820 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.820 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.820 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:33:57.820 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290877+0000", 2024-04-20T17:33:57.820 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.176511735 2024-04-20T17:33:57.820 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.820 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.820 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:33:57.820 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290886+0000", 2024-04-20T17:33:57.821 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.5980000000000003e-06 2024-04-20T17:33:57.821 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.821 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:33:57.821 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.821 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.821 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.821 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:35 6.4 6.16044024 (undecoded) ondisk+write+known_if_redirected e626)", 2024-04-20T17:33:57.821 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:33:43.114373+0000", 2024-04-20T17:33:57.821 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 13.787782027, 2024-04-20T17:33:57.821 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17653855700000001, 2024-04-20T17:33:57.821 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:33:57.821 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:33:57.821 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-04-20T17:33:57.821 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:33:57.821 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:33:57.822 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:33:57.822 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 35 2024-04-20T17:33:57.822 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.822 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:33:57.822 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.822 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:33:57.822 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114373+0000", 2024-04-20T17:33:57.822 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.822 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.822 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.822 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:33:57.822 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114373+0000", 2024-04-20T17:33:57.822 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.822 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.822 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.823 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:33:57.823 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114373+0000", 2024-04-20T17:33:57.823 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0999999999999999e-07 2024-04-20T17:33:57.823 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.823 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.823 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:33:57.823 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114419+0000", 2024-04-20T17:33:57.823 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5123999999999997e-05 2024-04-20T17:33:57.823 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.823 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.823 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:33:57.823 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114425+0000", 2024-04-20T17:33:57.823 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0970000000000002e-06 2024-04-20T17:33:57.823 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.823 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.824 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:33:57.824 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114429+0000", 2024-04-20T17:33:57.824 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4089999999999996e-06 2024-04-20T17:33:57.824 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.824 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.824 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:33:57.824 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290905+0000", 2024-04-20T17:33:57.824 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.176475935 2024-04-20T17:33:57.824 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.824 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.824 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:33:57.824 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290912+0000", 2024-04-20T17:33:57.824 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4819999999999999e-06 2024-04-20T17:33:57.824 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.825 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:33:57.825 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.825 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.825 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.825 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:37 6.4 6.1a00bf44 (undecoded) ondisk+write+known_if_redirected e626)", 2024-04-20T17:33:57.825 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:33:43.114438+0000", 2024-04-20T17:33:57.825 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 13.787716779, 2024-04-20T17:33:57.825 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17650761100000001, 2024-04-20T17:33:57.825 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:33:57.825 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:33:57.825 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-04-20T17:33:57.825 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:33:57.825 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:33:57.825 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:33:57.825 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 37 2024-04-20T17:33:57.826 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.826 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:33:57.826 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.826 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:33:57.826 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114438+0000", 2024-04-20T17:33:57.826 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.826 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.826 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.826 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:33:57.826 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114438+0000", 2024-04-20T17:33:57.826 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.826 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.826 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.826 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:33:57.827 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114439+0000", 2024-04-20T17:33:57.827 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5499999999999998e-07 2024-04-20T17:33:57.827 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.827 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.827 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:33:57.827 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114481+0000", 2024-04-20T17:33:57.827 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1878000000000001e-05 2024-04-20T17:33:57.827 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.827 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.827 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:33:57.827 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114487+0000", 2024-04-20T17:33:57.827 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9200000000000001e-06 2024-04-20T17:33:57.827 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.827 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.827 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:33:57.828 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114490+0000", 2024-04-20T17:33:57.828 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8510000000000001e-06 2024-04-20T17:33:57.828 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.828 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.828 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:33:57.828 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290940+0000", 2024-04-20T17:33:57.828 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17644975700000001 2024-04-20T17:33:57.828 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.828 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.828 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:33:57.828 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290946+0000", 2024-04-20T17:33:57.828 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6500000000000001e-06 2024-04-20T17:33:57.828 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.828 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:33:57.828 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.829 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.829 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.829 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:44 6.4 6.58250294 (undecoded) ondisk+write+known_if_redirected e626)", 2024-04-20T17:33:57.829 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:33:43.114498+0000", 2024-04-20T17:33:57.829 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 13.787657454, 2024-04-20T17:33:57.829 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17647228700000001, 2024-04-20T17:33:57.829 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:33:57.829 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:33:57.829 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-04-20T17:33:57.829 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:33:57.829 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:33:57.829 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:33:57.829 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 44 2024-04-20T17:33:57.829 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.829 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:33:57.830 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.830 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:33:57.830 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114498+0000", 2024-04-20T17:33:57.830 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.830 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.830 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.830 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:33:57.830 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114498+0000", 2024-04-20T17:33:57.830 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.830 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.830 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.830 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:33:57.830 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114498+0000", 2024-04-20T17:33:57.830 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5499999999999998e-07 2024-04-20T17:33:57.830 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.831 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.831 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:33:57.831 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114554+0000", 2024-04-20T17:33:57.831 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6028e-05 2024-04-20T17:33:57.831 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.831 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.831 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:33:57.831 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114561+0000", 2024-04-20T17:33:57.831 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5490000000000003e-06 2024-04-20T17:33:57.831 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.831 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.831 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:33:57.831 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114564+0000", 2024-04-20T17:33:57.831 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.72e-06 2024-04-20T17:33:57.831 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.832 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.832 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:33:57.832 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290965+0000", 2024-04-20T17:33:57.832 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17640072300000001 2024-04-20T17:33:57.832 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.832 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.832 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:33:57.832 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290970+0000", 2024-04-20T17:33:57.832 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8119999999999998e-06 2024-04-20T17:33:57.832 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.832 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:33:57.832 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.832 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.832 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.832 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:4 6.4 6:2d50d087:::benchmark_data_smithi171_147972_object3:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e626)", 2024-04-20T17:33:57.832 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:33:43.455948+0000", 2024-04-20T17:33:57.833 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 13.446207464, 2024-04-20T17:33:57.833 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.090798355999999997, 2024-04-20T17:33:57.833 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:33:57.833 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:33:57.833 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:57.833 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:33:57.833 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:33:57.833 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:33:57.833 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 4 2024-04-20T17:33:57.833 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.833 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:33:57.833 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.833 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:33:57.833 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.455948+0000", 2024-04-20T17:33:57.833 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.834 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.834 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.834 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:33:57.834 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.455948+0000", 2024-04-20T17:33:57.834 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.834 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.834 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.834 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:33:57.834 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.455949+0000", 2024-04-20T17:33:57.834 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.609e-06 2024-04-20T17:33:57.834 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.834 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.834 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:33:57.834 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456007+0000", 2024-04-20T17:33:57.835 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8056000000000001e-05 2024-04-20T17:33:57.835 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.835 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.835 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:33:57.835 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456018+0000", 2024-04-20T17:33:57.835 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0644e-05 2024-04-20T17:33:57.835 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.835 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.835 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:33:57.835 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456026+0000", 2024-04-20T17:33:57.835 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4470000000000006e-06 2024-04-20T17:33:57.835 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.835 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.835 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:33:57.836 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456127+0000", 2024-04-20T17:33:57.836 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000100479 2024-04-20T17:33:57.836 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.836 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.836 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:33:57.836 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456141+0000", 2024-04-20T17:33:57.836 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4126e-05 2024-04-20T17:33:57.836 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.836 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.836 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:33:57.836 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.544555+0000", 2024-04-20T17:33:57.836 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.088413759999999994 2024-04-20T17:33:57.836 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.836 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.836 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:33:57.837 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.544733+0000", 2024-04-20T17:33:57.837 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000178636 2024-04-20T17:33:57.837 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.837 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.837 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:33:57.837 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.544907+0000", 2024-04-20T17:33:57.837 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000174146 2024-04-20T17:33:57.837 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.837 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.837 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:33:57.837 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.546641+0000", 2024-04-20T17:33:57.837 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001733248 2024-04-20T17:33:57.837 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.837 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.838 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:33:57.838 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.546688+0000", 2024-04-20T17:33:57.838 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7688999999999997e-05 2024-04-20T17:33:57.838 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.838 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.838 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:33:57.838 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.546719+0000", 2024-04-20T17:33:57.838 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0718e-05 2024-04-20T17:33:57.838 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.838 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.838 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:33:57.838 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.546746+0000", 2024-04-20T17:33:57.838 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6798e-05 2024-04-20T17:33:57.838 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.839 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:33:57.839 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.839 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.839 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.839 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:5 6.4 6:2aaceab0:::benchmark_data_smithi171_147972_object4:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e626)", 2024-04-20T17:33:57.839 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:33:43.456041+0000", 2024-04-20T17:33:57.839 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 13.446113622, 2024-04-20T17:33:57.839 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.090855454000000002, 2024-04-20T17:33:57.839 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:33:57.839 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:33:57.840 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:57.840 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:33:57.840 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:33:57.840 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:33:57.841 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 5 2024-04-20T17:33:57.841 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.841 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:33:57.841 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.841 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:33:57.841 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456041+0000", 2024-04-20T17:33:57.841 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.841 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.841 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.841 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:33:57.842 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456041+0000", 2024-04-20T17:33:57.842 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.842 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.842 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.842 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:33:57.842 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456042+0000", 2024-04-20T17:33:57.842 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9299999999999997e-07 2024-04-20T17:33:57.842 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.842 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.842 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:33:57.842 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456075+0000", 2024-04-20T17:33:57.843 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2805999999999997e-05 2024-04-20T17:33:57.843 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.843 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.843 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:33:57.843 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456080+0000", 2024-04-20T17:33:57.843 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.994e-06 2024-04-20T17:33:57.843 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.843 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.843 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:33:57.843 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456084+0000", 2024-04-20T17:33:57.843 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.856e-06 2024-04-20T17:33:57.844 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.844 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.845 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:33:57.845 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456192+0000", 2024-04-20T17:33:57.845 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000108272 2024-04-20T17:33:57.845 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.845 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.845 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:33:57.845 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456202+0000", 2024-04-20T17:33:57.845 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0451999999999999e-05 2024-04-20T17:33:57.845 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.845 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.845 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:33:57.845 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.545630+0000", 2024-04-20T17:33:57.845 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.089427692000000003 2024-04-20T17:33:57.846 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.846 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.846 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:33:57.846 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.545722+0000", 2024-04-20T17:33:57.846 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2251999999999997e-05 2024-04-20T17:33:57.846 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.847 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.847 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:33:57.847 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.545825+0000", 2024-04-20T17:33:57.847 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000102886 2024-04-20T17:33:57.847 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.847 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.847 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:33:57.847 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.546800+0000", 2024-04-20T17:33:57.847 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00097514100000000001 2024-04-20T17:33:57.847 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.847 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.847 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:33:57.847 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.546833+0000", 2024-04-20T17:33:57.848 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2518999999999998e-05 2024-04-20T17:33:57.848 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.848 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.848 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:33:57.848 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.546876+0000", 2024-04-20T17:33:57.848 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2902999999999999e-05 2024-04-20T17:33:57.848 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.848 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.848 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:33:57.848 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.546897+0000", 2024-04-20T17:33:57.848 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0988e-05 2024-04-20T17:33:57.848 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.848 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:33:57.849 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.849 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.849 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.849 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:19 6.4 6:28fa57b9:::benchmark_data_smithi171_147972_object18:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e626)", 2024-04-20T17:33:57.849 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:33:43.456092+0000", 2024-04-20T17:33:57.849 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 13.446063437999999, 2024-04-20T17:33:57.849 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.091804686999999996, 2024-04-20T17:33:57.849 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:33:57.849 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:33:57.849 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:57.849 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:33:57.850 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:33:57.850 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:33:57.850 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 19 2024-04-20T17:33:57.850 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.850 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:33:57.850 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.850 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:33:57.850 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456092+0000", 2024-04-20T17:33:57.850 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.850 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.850 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.850 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:33:57.850 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456092+0000", 2024-04-20T17:33:57.850 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.851 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.851 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.851 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:33:57.851 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456092+0000", 2024-04-20T17:33:57.851 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7700000000000004e-07 2024-04-20T17:33:57.851 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.851 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.851 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:33:57.851 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456116+0000", 2024-04-20T17:33:57.851 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4014000000000001e-05 2024-04-20T17:33:57.851 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.851 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.851 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:33:57.851 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456120+0000", 2024-04-20T17:33:57.851 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1489999999999996e-06 2024-04-20T17:33:57.852 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.852 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.852 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:33:57.852 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456124+0000", 2024-04-20T17:33:57.852 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.287e-06 2024-04-20T17:33:57.852 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.852 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.852 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:33:57.852 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456250+0000", 2024-04-20T17:33:57.852 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012672700000000001 2024-04-20T17:33:57.852 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.852 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.852 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:33:57.852 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456257+0000", 2024-04-20T17:33:57.852 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0199999999999997e-06 2024-04-20T17:33:57.853 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.853 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.853 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:33:57.853 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.546101+0000", 2024-04-20T17:33:57.853 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.089843671999999999 2024-04-20T17:33:57.853 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.853 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.853 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:33:57.853 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.546227+0000", 2024-04-20T17:33:57.853 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012569600000000001 2024-04-20T17:33:57.853 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.853 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.853 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:33:57.853 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.546390+0000", 2024-04-20T17:33:57.853 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00016291600000000001 2024-04-20T17:33:57.854 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.854 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.854 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:33:57.854 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.547713+0000", 2024-04-20T17:33:57.854 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0013230760000000001 2024-04-20T17:33:57.854 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.854 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.854 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:33:57.854 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.547797+0000", 2024-04-20T17:33:57.854 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4208000000000004e-05 2024-04-20T17:33:57.854 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.854 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.855 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:33:57.855 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.547853+0000", 2024-04-20T17:33:57.855 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6449000000000001e-05 2024-04-20T17:33:57.855 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.855 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.855 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:33:57.855 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.547896+0000", 2024-04-20T17:33:57.855 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2895999999999997e-05 2024-04-20T17:33:57.855 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.855 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:33:57.855 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.855 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.855 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.856 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:30 6.4 6:28635ca1:::benchmark_data_smithi171_147972_object29:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e626)", 2024-04-20T17:33:57.856 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:33:43.456129+0000", 2024-04-20T17:33:57.856 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 13.446026122999999, 2024-04-20T17:33:57.856 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.092482990000000001, 2024-04-20T17:33:57.856 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:33:57.856 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:33:57.856 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:57.856 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:33:57.856 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:33:57.856 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:33:57.856 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 30 2024-04-20T17:33:57.856 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.856 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:33:57.856 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.856 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:33:57.857 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456129+0000", 2024-04-20T17:33:57.857 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.857 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.857 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.857 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:33:57.857 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456129+0000", 2024-04-20T17:33:57.857 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.857 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.857 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.857 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:33:57.857 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456129+0000", 2024-04-20T17:33:57.858 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7300000000000002e-07 2024-04-20T17:33:57.858 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.858 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.858 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:33:57.858 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456179+0000", 2024-04-20T17:33:57.858 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9468000000000001e-05 2024-04-20T17:33:57.858 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.858 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.858 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:33:57.858 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456185+0000", 2024-04-20T17:33:57.858 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6019999999999996e-06 2024-04-20T17:33:57.858 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.858 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.859 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:33:57.859 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456190+0000", 2024-04-20T17:33:57.859 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4789999999999998e-06 2024-04-20T17:33:57.859 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.859 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.859 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:33:57.859 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456304+0000", 2024-04-20T17:33:57.859 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000114276 2024-04-20T17:33:57.860 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.860 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.860 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:33:57.860 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456311+0000", 2024-04-20T17:33:57.860 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9489999999999999e-06 2024-04-20T17:33:57.860 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.860 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.860 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:33:57.861 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.546928+0000", 2024-04-20T17:33:57.861 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.090616885999999994 2024-04-20T17:33:57.861 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.861 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.861 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:33:57.861 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.546992+0000", 2024-04-20T17:33:57.861 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.41e-05 2024-04-20T17:33:57.861 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.861 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.861 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:33:57.861 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.547074+0000", 2024-04-20T17:33:57.862 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1699000000000005e-05 2024-04-20T17:33:57.862 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.862 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.862 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:33:57.862 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.548450+0000", 2024-04-20T17:33:57.862 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0013767460000000001 2024-04-20T17:33:57.862 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.862 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.862 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:33:57.862 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.548514+0000", 2024-04-20T17:33:57.862 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3876999999999998e-05 2024-04-20T17:33:57.862 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.863 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.863 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:33:57.863 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.548571+0000", 2024-04-20T17:33:57.863 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6656000000000001e-05 2024-04-20T17:33:57.863 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.863 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.863 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:33:57.863 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.548612+0000", 2024-04-20T17:33:57.863 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0878999999999999e-05 2024-04-20T17:33:57.863 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.863 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:33:57.863 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.864 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.864 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.864 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:35 6.4 6:24022068:::benchmark_data_smithi171_147972_object34:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e626)", 2024-04-20T17:33:57.864 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:33:43.456196+0000", 2024-04-20T17:33:57.864 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 13.445959061, 2024-04-20T17:33:57.864 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.092576828, 2024-04-20T17:33:57.864 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:33:57.864 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:33:57.864 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:57.864 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:33:57.864 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:33:57.864 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:33:57.864 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 35 2024-04-20T17:33:57.865 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.865 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:33:57.865 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.865 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:33:57.865 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456196+0000", 2024-04-20T17:33:57.865 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.865 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.865 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.865 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:33:57.865 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456196+0000", 2024-04-20T17:33:57.865 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.865 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.866 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.866 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:33:57.866 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456197+0000", 2024-04-20T17:33:57.866 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.82e-07 2024-04-20T17:33:57.866 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.866 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.866 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:33:57.866 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456232+0000", 2024-04-20T17:33:57.866 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5123999999999998e-05 2024-04-20T17:33:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:33:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456237+0000", 2024-04-20T17:33:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1340000000000003e-06 2024-04-20T17:33:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:33:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456240+0000", 2024-04-20T17:33:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3170000000000002e-06 2024-04-20T17:33:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:33:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456352+0000", 2024-04-20T17:33:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000111841 2024-04-20T17:33:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:33:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456359+0000", 2024-04-20T17:33:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7850000000000003e-06 2024-04-20T17:33:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:33:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.547266+0000", 2024-04-20T17:33:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.090906915000000005 2024-04-20T17:33:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:33:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.547359+0000", 2024-04-20T17:33:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2917999999999994e-05 2024-04-20T17:33:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:33:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.547486+0000", 2024-04-20T17:33:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012746099999999999 2024-04-20T17:33:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:33:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.548673+0000", 2024-04-20T17:33:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001186629 2024-04-20T17:33:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:33:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.548684+0000", 2024-04-20T17:33:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1712999999999999e-05 2024-04-20T17:33:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:33:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.548727+0000", 2024-04-20T17:33:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2936999999999999e-05 2024-04-20T17:33:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:33:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.548773+0000", 2024-04-20T17:33:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5472000000000001e-05 2024-04-20T17:33:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:33:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:37 6.4 6:22fd0058:::benchmark_data_smithi171_147972_object36:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e626)", 2024-04-20T17:33:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:33:43.456245+0000", 2024-04-20T17:33:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 13.445909779999999, 2024-04-20T17:33:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.093757171, 2024-04-20T17:33:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:33:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:33:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:33:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:33:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:33:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 37 2024-04-20T17:33:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:33:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:33:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456245+0000", 2024-04-20T17:33:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:33:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456245+0000", 2024-04-20T17:33:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:33:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456246+0000", 2024-04-20T17:33:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5400000000000002e-07 2024-04-20T17:33:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:33:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456297+0000", 2024-04-20T17:33:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1323000000000003e-05 2024-04-20T17:33:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:33:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456302+0000", 2024-04-20T17:33:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9470000000000002e-06 2024-04-20T17:33:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:33:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456306+0000", 2024-04-20T17:33:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6729999999999998e-06 2024-04-20T17:33:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:33:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456392+0000", 2024-04-20T17:33:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5953999999999997e-05 2024-04-20T17:33:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:33:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456398+0000", 2024-04-20T17:33:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5969999999999997e-06 2024-04-20T17:33:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:33:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.547951+0000", 2024-04-20T17:33:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.091553115000000004 2024-04-20T17:33:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:33:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.548056+0000", 2024-04-20T17:33:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000104478 2024-04-20T17:33:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:33:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.548190+0000", 2024-04-20T17:33:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013377199999999999 2024-04-20T17:33:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:33:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.549858+0000", 2024-04-20T17:33:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0016683539999999999 2024-04-20T17:33:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:33:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.549907+0000", 2024-04-20T17:33:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9406000000000001e-05 2024-04-20T17:33:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:33:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.549963+0000", 2024-04-20T17:33:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5566999999999998e-05 2024-04-20T17:33:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:33:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.550002+0000", 2024-04-20T17:33:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9530999999999998e-05 2024-04-20T17:33:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:33:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:44 6.4 6:2940a41a:::benchmark_data_smithi171_147972_object43:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e626)", 2024-04-20T17:33:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:33:43.456311+0000", 2024-04-20T17:33:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 13.445843973000001, 2024-04-20T17:33:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.094448833999999995, 2024-04-20T17:33:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:33:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:33:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:33:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:33:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:33:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:33:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 44 2024-04-20T17:33:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:33:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:33:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456311+0000", 2024-04-20T17:33:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:33:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456311+0000", 2024-04-20T17:33:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:33:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:33:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456312+0000", 2024-04-20T17:33:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8800000000000003e-07 2024-04-20T17:33:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:33:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456362+0000", 2024-04-20T17:33:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0720999999999998e-05 2024-04-20T17:33:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:33:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456367+0000", 2024-04-20T17:33:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7559999999999999e-06 2024-04-20T17:33:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:33:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456371+0000", 2024-04-20T17:33:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5609999999999999e-06 2024-04-20T17:33:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:33:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456422+0000", 2024-04-20T17:33:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1242999999999997e-05 2024-04-20T17:33:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for peered", 2024-04-20T17:33:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.456429+0000", 2024-04-20T17:33:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.776e-06 2024-04-20T17:33:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:33:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.548818+0000", 2024-04-20T17:33:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.092389280000000004 2024-04-20T17:33:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:33:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.548949+0000", 2024-04-20T17:33:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013097200000000001 2024-04-20T17:33:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:33:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.549076+0000", 2024-04-20T17:33:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012732299999999999 2024-04-20T17:33:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:33:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.550651+0000", 2024-04-20T17:33:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0015751300000000001 2024-04-20T17:33:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:33:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.550669+0000", 2024-04-20T17:33:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7694e-05 2024-04-20T17:33:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:33:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.550715+0000", 2024-04-20T17:33:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6329000000000003e-05 2024-04-20T17:33:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:33:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:33:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:33:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.550760+0000", 2024-04-20T17:33:57.885 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4561000000000002e-05 2024-04-20T17:33:57.885 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.885 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:33:57.885 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.885 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:33:57.885 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:33:57.885 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:33:57.886 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:57 smithi171 ceph-mon[44189]: pgmap v1656: 17 pgs: 17 active+clean; 2.5 GiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 171 MiB/s wr, 2.74k op/s 2024-04-20T17:33:57.927 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 19 16 52505 52489 172.642 181.625 0.00609576 0.00579054 2024-04-20T17:33:57.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:57 smithi005 ceph-mon[39562]: pgmap v1656: 17 pgs: 17 active+clean; 2.5 GiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 171 MiB/s wr, 2.74k op/s 2024-04-20T17:33:58.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:57 smithi106 ceph-mon[44423]: pgmap v1656: 17 pgs: 17 active+clean; 2.5 GiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 171 MiB/s wr, 2.74k op/s 2024-04-20T17:33:58.927 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:33:58.928001 min lat: 0.00100551 max lat: 4.62787 avg lat: 0.00579292 2024-04-20T17:33:58.927 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:33:58.927 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 20 16 55250 55234 172.588 171.562 0.00285561 0.00579292 2024-04-20T17:33:59.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:33:59 smithi171 ceph-mon[44189]: pgmap v1657: 17 pgs: 17 active+clean; 3.0 GiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 211 MiB/s wr, 3.37k op/s 2024-04-20T17:33:59.927 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 21 16 57809 57793 171.985 159.938 0.00412068 0.0058131 2024-04-20T17:33:59.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:33:59 smithi005 ceph-mon[39562]: pgmap v1657: 17 pgs: 17 active+clean; 3.0 GiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 211 MiB/s wr, 3.37k op/s 2024-04-20T17:34:00.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:33:59 smithi106 ceph-mon[44423]: pgmap v1657: 17 pgs: 17 active+clean; 3.0 GiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 211 MiB/s wr, 3.37k op/s 2024-04-20T17:34:00.927 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 22 16 60095 60079 170.661 142.875 0.00526052 0.00585823 2024-04-20T17:34:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:01 smithi171 ceph-mon[44189]: pgmap v1658: 17 pgs: 17 active+clean; 3.2 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 181 MiB/s wr, 2.90k op/s 2024-04-20T17:34:01.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:01 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:34:01.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:01 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:34:01.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:01 smithi171 ceph-mon[44189]: osdmap e632: 12 total, 11 up, 10 in 2024-04-20T17:34:01.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:01 smithi171 ceph-mon[44189]: osdmap e633: 12 total, 11 up, 10 in 2024-04-20T17:34:01.927 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 23 16 62376 62360 169.439 142.562 0.00303553 0.00589975 2024-04-20T17:34:01.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:01 smithi005 ceph-mon[39562]: pgmap v1658: 17 pgs: 17 active+clean; 3.2 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 181 MiB/s wr, 2.90k op/s 2024-04-20T17:34:01.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:01 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:34:01.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:01 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:34:01.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:01 smithi005 ceph-mon[39562]: osdmap e632: 12 total, 11 up, 10 in 2024-04-20T17:34:01.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:01 smithi005 ceph-mon[39562]: osdmap e633: 12 total, 11 up, 10 in 2024-04-20T17:34:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:01 smithi106 ceph-mon[44423]: pgmap v1658: 17 pgs: 17 active+clean; 3.2 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 181 MiB/s wr, 2.90k op/s 2024-04-20T17:34:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:01 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:34:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:01 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:34:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:01 smithi106 ceph-mon[44423]: osdmap e632: 12 total, 11 up, 10 in 2024-04-20T17:34:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:01 smithi106 ceph-mon[44423]: osdmap e633: 12 total, 11 up, 10 in 2024-04-20T17:34:02.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:02 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:34:02.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:02 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:34:02.927 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 24 16 64396 64380 167.639 126.25 0.00453892 0.00596369 2024-04-20T17:34:02.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:02 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:34:02.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:02 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:34:03.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:02 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:34:03.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:02 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:34:03.927 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 25 16 66529 66513 166.266 133.312 0.00609331 0.00600912 2024-04-20T17:34:03.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:03 smithi005 ceph-mon[39562]: pgmap v1661: 17 pgs: 17 active+clean; 3.4 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 152 MiB/s wr, 2.44k op/s 2024-04-20T17:34:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:03 smithi106 ceph-mon[44423]: pgmap v1661: 17 pgs: 17 active+clean; 3.4 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 152 MiB/s wr, 2.44k op/s 2024-04-20T17:34:04.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:03 smithi171 ceph-mon[44189]: pgmap v1661: 17 pgs: 17 active+clean; 3.4 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 152 MiB/s wr, 2.44k op/s 2024-04-20T17:34:04.224 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.1/config 2024-04-20T17:34:04.927 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 26 16 68573 68557 164.784 127.75 0.00526469 0.00606575 2024-04-20T17:34:05.930 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 27 16 70636 70620 163.456 128.938 0.00556675 0.00611658 2024-04-20T17:34:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:05 smithi005 ceph-mon[39562]: pgmap v1662: 17 pgs: 17 active+clean; 3.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 173 MiB/s wr, 2.77k op/s 2024-04-20T17:34:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:05 smithi106 ceph-mon[44423]: pgmap v1662: 17 pgs: 17 active+clean; 3.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 173 MiB/s wr, 2.77k op/s 2024-04-20T17:34:06.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:05 smithi171 ceph-mon[44189]: pgmap v1662: 17 pgs: 17 active+clean; 3.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 173 MiB/s wr, 2.77k op/s 2024-04-20T17:34:06.165 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:34:06.166 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:34:06.284 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:34:06.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:06.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:34:06.316 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-20T17:34:06.397 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:34:06.489 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:06.628 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:34:06.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:06.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:34:06.742 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:34:06.777 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:06.807 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:34:06.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:06.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:34:06.913 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:34:06.927 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 28 16 72764 72748 162.368 133 0.00988106 0.00615766 2024-04-20T17:34:06.949 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:06.976 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:34:06.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:06.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:34:07.080 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:34:07.113 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:07.138 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:34:07.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:07.142 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:34:07.242 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:34:07.246 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:34:07.271 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:07.293 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:34:07.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:07.298 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:34:07.399 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:34:07.457 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:07.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:07 smithi171 ceph-mon[44189]: osdmap e634: 12 total, 11 up, 10 in 2024-04-20T17:34:07.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:07 smithi171 ceph-mon[44189]: pgmap v1664: 17 pgs: 17 active+clean; 4.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 137 MiB/s wr, 2.20k op/s 2024-04-20T17:34:07.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:07 smithi005 ceph-mon[39562]: osdmap e634: 12 total, 11 up, 10 in 2024-04-20T17:34:07.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:07 smithi005 ceph-mon[39562]: pgmap v1664: 17 pgs: 17 active+clean; 4.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 137 MiB/s wr, 2.20k op/s 2024-04-20T17:34:07.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:07 smithi106 ceph-mon[44423]: osdmap e634: 12 total, 11 up, 10 in 2024-04-20T17:34:07.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:07 smithi106 ceph-mon[44423]: pgmap v1664: 17 pgs: 17 active+clean; 4.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 137 MiB/s wr, 2.20k op/s 2024-04-20T17:34:07.834 INFO:teuthology.orchestra.run.smithi005.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:34:07.849 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:34:07.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:07.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:34:07.928 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 29 16 74885 74869 161.34 132.562 0.00728881 0.00619688 2024-04-20T17:34:07.958 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:34:08.167 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:34:08.181 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:08.219 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:34:08.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:08.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:34:08.333 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:34:08.365 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:08.390 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:34:08.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:08.395 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:34:08.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:08 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:34:08.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:08 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:34:08.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:08 smithi005 ceph-mon[39562]: osdmap e635: 12 total, 11 up, 10 in 2024-04-20T17:34:08.496 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:34:08.526 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:08.549 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:34:08.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:08.552 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:34:08.653 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:34:08.683 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:08 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:34:08.683 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:08 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:34:08.683 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:08 smithi171 ceph-mon[44189]: osdmap e635: 12 total, 11 up, 10 in 2024-04-20T17:34:08.685 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:08.708 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:34:08.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:08.714 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:34:08.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:08 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:34:08.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:08 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:34:08.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:08 smithi106 ceph-mon[44423]: osdmap e635: 12 total, 11 up, 10 in 2024-04-20T17:34:08.815 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:34:08.844 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:08.872 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:34:08.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:08.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:34:08.928 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 30 16 76846 76830 160.047 122.562 0.00825357 0.00624653 2024-04-20T17:34:08.975 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:34:09.009 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:09.034 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:34:09.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:09.037 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:34:09.138 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:34:09.184 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:09.212 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:34:09.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:09.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:34:09.318 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:34:09.358 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:09.390 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:34:09.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:09.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:34:09.495 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:34:09.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:09 smithi106 ceph-mon[44423]: pgmap v1666: 17 pgs: 17 active+clean; 4.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 126 MiB/s wr, 2.02k op/s 2024-04-20T17:34:09.535 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:09.568 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:34:09.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:09.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:34:09.675 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:34:09.710 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:09.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:09 smithi005 ceph-mon[39562]: pgmap v1666: 17 pgs: 17 active+clean; 4.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 126 MiB/s wr, 2.02k op/s 2024-04-20T17:34:09.738 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:34:09.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:09.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:34:09.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:09 smithi171 ceph-mon[44189]: pgmap v1666: 17 pgs: 17 active+clean; 4.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 126 MiB/s wr, 2.02k op/s 2024-04-20T17:34:09.842 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:34:09.928 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 31 16 78927 78911 159.079 130.062 0.010777 0.00628375 2024-04-20T17:34:10.678 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:10 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:34:10.678 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:10 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:34:10.678 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:10 smithi005 ceph-mon[39562]: osdmap e636: 12 total, 11 up, 10 in 2024-04-20T17:34:10.719 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:10.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:10 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:34:10.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:10 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:34:10.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:10 smithi106 ceph-mon[44423]: osdmap e636: 12 total, 11 up, 10 in 2024-04-20T17:34:10.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:10 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:34:10.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:10 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:34:10.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:10 smithi171 ceph-mon[44189]: osdmap e636: 12 total, 11 up, 10 in 2024-04-20T17:34:10.912 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:34:10.928 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 32 16 81042 81026 158.239 132.188 0.00844222 0.00631794 2024-04-20T17:34:10.945 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:34:10.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:10.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:34:11.064 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:34:11.143 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:11.534 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:11 smithi005 ceph-mon[39562]: pgmap v1668: 17 pgs: 17 active+clean; 4.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-04-20T17:34:11.534 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:11 smithi005 ceph-mon[39562]: osdmap e637: 12 total, 11 up, 10 in 2024-04-20T17:34:11.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:11 smithi106 ceph-mon[44423]: pgmap v1668: 17 pgs: 17 active+clean; 4.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-04-20T17:34:11.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:11 smithi106 ceph-mon[44423]: osdmap e637: 12 total, 11 up, 10 in 2024-04-20T17:34:11.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:11 smithi171 ceph-mon[44189]: pgmap v1668: 17 pgs: 17 active+clean; 4.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-04-20T17:34:11.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:11 smithi171 ceph-mon[44189]: osdmap e637: 12 total, 11 up, 10 in 2024-04-20T17:34:11.920 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:34:11.928 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 33 15 82908 82893 156.979 116.688 0.00556044 0.00636846 2024-04-20T17:34:11.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:11.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:34:12.049 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:34:12.135 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:12.928 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 34 16 84670 84654 155.599 110.062 0.00634527 0.00642534 2024-04-20T17:34:13.315 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:34:13.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:13.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:34:13.420 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:34:13.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:13 smithi005 ceph-mon[39562]: pgmap v1670: 17 pgs: 17 active+clean; 4.7 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-04-20T17:34:13.468 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:13.492 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:34:13.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:13.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:34:13.598 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:34:13.635 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:13.663 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:34:13.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:13.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:34:13.668 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:13 smithi106 ceph-mon[44423]: pgmap v1670: 17 pgs: 17 active+clean; 4.7 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-04-20T17:34:13.768 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:34:13.808 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:13.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:13 smithi171 ceph-mon[44189]: pgmap v1670: 17 pgs: 17 active+clean; 4.7 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-04-20T17:34:13.837 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:34:13.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:13.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:34:13.928 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 35 16 86429 86413 154.294 109.938 0.0117016 0.00647969 2024-04-20T17:34:13.942 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:34:13.973 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:14.001 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:34:14.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:14.005 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:34:14.106 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:34:14.162 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:14.240 INFO:teuthology.orchestra.run.smithi005.stderr:Exporting 5.0 info 5.0( v 613'16404 (577'6400,613'16404] local-lis/les=543/544 n=0 ec=543/543 lis/c=543/543 les/c/f=544/544/0 sis=543) 2024-04-20T17:34:14.331 INFO:teuthology.orchestra.run.smithi005.stderr:Export successful 2024-04-20T17:34:14.913 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:34:14.928 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 36 16 88256 88240 153.18 114.188 0.00403877 0.00652657 2024-04-20T17:34:15.414 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:15 smithi005 ceph-mon[39562]: pgmap v1671: 17 pgs: 17 active+clean; 5.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 147 MiB/s wr, 2.36k op/s 2024-04-20T17:34:15.651 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:34:15.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:15.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:34:15.755 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:34:15.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:15 smithi106 ceph-mon[44423]: pgmap v1671: 17 pgs: 17 active+clean; 5.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 147 MiB/s wr, 2.36k op/s 2024-04-20T17:34:15.782 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:15 smithi171 ceph-mon[44189]: pgmap v1671: 17 pgs: 17 active+clean; 5.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 147 MiB/s wr, 2.36k op/s 2024-04-20T17:34:15.784 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:15.806 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:34:15.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:15.810 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:34:15.840 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:34:15.840 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:34:15.841 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.841 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:89890 6.9 6:9039c2b5:::benchmark_data_smithi171_147972_object89889:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e637)", 2024-04-20T17:34:15.841 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:34:15.833873+0000", 2024-04-20T17:34:15.841 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.0044310499999999997, 2024-04-20T17:34:15.841 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0044676120000000001, 2024-04-20T17:34:15.841 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:34:15.841 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:34:15.841 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:34:15.841 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:34:15.841 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:34:15.841 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:34:15.841 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 89890 2024-04-20T17:34:15.841 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.841 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:34:15.842 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.842 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:34:15.842 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:15.833873+0000", 2024-04-20T17:34:15.842 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:15.842 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.842 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.842 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:34:15.842 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:15.833873+0000", 2024-04-20T17:34:15.842 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:15.842 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.842 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.842 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:34:15.842 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:15.833875+0000", 2024-04-20T17:34:15.843 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7519999999999999e-06 2024-04-20T17:34:15.843 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.843 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.843 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:34:15.843 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:15.833979+0000", 2024-04-20T17:34:15.843 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010401800000000001 2024-04-20T17:34:15.843 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.843 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.843 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:34:15.843 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:15.833995+0000", 2024-04-20T17:34:15.843 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5698000000000001e-05 2024-04-20T17:34:15.843 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.843 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.843 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:34:15.844 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:15.834003+0000", 2024-04-20T17:34:15.844 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9290000000000004e-06 2024-04-20T17:34:15.844 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.844 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.844 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:34:15.844 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:15.834062+0000", 2024-04-20T17:34:15.844 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.838e-05 2024-04-20T17:34:15.844 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.844 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.844 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:34:15.845 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:15.834185+0000", 2024-04-20T17:34:15.845 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000122872 2024-04-20T17:34:15.845 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.845 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.845 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 9", 2024-04-20T17:34:15.845 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:15.834324+0000", 2024-04-20T17:34:15.845 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000138897 2024-04-20T17:34:15.845 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.845 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.845 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:34:15.845 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:15.837372+0000", 2024-04-20T17:34:15.845 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00304842 2024-04-20T17:34:15.845 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:15.846 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:34:15.846 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:15.846 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.846 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.846 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:89898 6.a 6:53484331:::benchmark_data_smithi171_147972_object89897:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e637)", 2024-04-20T17:34:15.846 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:34:15.835297+0000", 2024-04-20T17:34:15.846 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.003006931, 2024-04-20T17:34:15.846 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0031387580000000002, 2024-04-20T17:34:15.846 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:34:15.847 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:34:15.847 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:34:15.847 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:34:15.847 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:34:15.847 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:34:15.847 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 89898 2024-04-20T17:34:15.847 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.848 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:34:15.848 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.848 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:34:15.848 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:15.835297+0000", 2024-04-20T17:34:15.848 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:15.848 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.848 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.848 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:34:15.848 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:15.835297+0000", 2024-04-20T17:34:15.848 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:15.848 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.848 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.848 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:34:15.848 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:15.835298+0000", 2024-04-20T17:34:15.849 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.217e-06 2024-04-20T17:34:15.849 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.849 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.849 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:34:15.849 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:15.835374+0000", 2024-04-20T17:34:15.849 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5241999999999996e-05 2024-04-20T17:34:15.849 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.849 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.849 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:34:15.849 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:15.835382+0000", 2024-04-20T17:34:15.849 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.276e-06 2024-04-20T17:34:15.849 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.849 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.849 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:34:15.850 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:15.835390+0000", 2024-04-20T17:34:15.850 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0690000000000002e-06 2024-04-20T17:34:15.850 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.850 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.850 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:34:15.850 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:15.835440+0000", 2024-04-20T17:34:15.850 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9971e-05 2024-04-20T17:34:15.850 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.850 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.850 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:34:15.850 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:15.835519+0000", 2024-04-20T17:34:15.850 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8718000000000001e-05 2024-04-20T17:34:15.850 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.850 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.851 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:34:15.851 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:15.835609+0000", 2024-04-20T17:34:15.851 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0286999999999998e-05 2024-04-20T17:34:15.851 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:15.851 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:34:15.851 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:15.851 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.851 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.851 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:89902 6.0 6.6fde16b0 (undecoded) ondisk+write+known_if_redirected e637)", 2024-04-20T17:34:15.851 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:34:15.838372+0000", 2024-04-20T17:34:15.851 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 4294967295.9999318, 2024-04-20T17:34:15.851 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013944399999999999, 2024-04-20T17:34:15.851 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:34:15.851 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:34:15.852 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:34:15.852 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:34:15.852 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:34:15.852 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:34:15.852 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 89902 2024-04-20T17:34:15.852 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.852 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:34:15.852 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.852 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:34:15.852 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:15.838372+0000", 2024-04-20T17:34:15.852 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:15.852 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.852 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.852 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:34:15.852 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:15.838372+0000", 2024-04-20T17:34:15.853 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:15.853 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.853 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.853 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:34:15.853 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:15.838375+0000", 2024-04-20T17:34:15.853 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2689999999999998e-06 2024-04-20T17:34:15.853 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.853 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.853 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:34:15.853 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:15.838446+0000", 2024-04-20T17:34:15.853 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1816999999999994e-05 2024-04-20T17:34:15.853 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.853 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.853 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:34:15.853 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:15.838458+0000", 2024-04-20T17:34:15.854 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1080999999999999e-05 2024-04-20T17:34:15.854 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.854 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.854 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:34:15.854 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:15.838466+0000", 2024-04-20T17:34:15.854 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2910000000000007e-06 2024-04-20T17:34:15.854 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:15.854 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:15.854 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:34:15.855 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:15.838520+0000", 2024-04-20T17:34:15.855 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4147000000000002e-05 2024-04-20T17:34:15.855 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:15.855 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:34:15.855 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:15.855 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:15.855 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:34:15.855 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 3 2024-04-20T17:34:15.855 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:34:15.911 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:34:15.928 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 37 16 90068 90052 152.101 113.25 0.00338506 0.0065734 2024-04-20T17:34:15.967 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:16.396 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:34:16.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:16 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4180101156' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:34:16.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:16 smithi005 ceph-mon[39562]: from='client.26806 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:34:16.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:16 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:34:16.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:16 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:34:16.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:16 smithi005 ceph-mon[39562]: from='client.26806 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:34:16.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:16 smithi005 ceph-mon[39562]: osdmap e638: 12 total, 11 up, 10 in 2024-04-20T17:34:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:16 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4180101156' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:34:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:16 smithi106 ceph-mon[44423]: from='client.26806 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:34:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:16 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:34:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:16 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:34:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:16 smithi106 ceph-mon[44423]: from='client.26806 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:34:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:16 smithi106 ceph-mon[44423]: osdmap e638: 12 total, 11 up, 10 in 2024-04-20T17:34:16.831 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-20T17:34:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:16 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4180101156' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:34:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:16 smithi171 ceph-mon[44189]: from='client.26806 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:34:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:16 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:34:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:16 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:34:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:16 smithi171 ceph-mon[44189]: from='client.26806 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:34:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:16 smithi171 ceph-mon[44189]: osdmap e638: 12 total, 11 up, 10 in 2024-04-20T17:34:16.928 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 38 16 91890 91874 151.095 113.875 0.00724367 0.00661691 2024-04-20T17:34:17.150 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:34:17.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:17.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:34:17.257 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:34:17.302 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:17.328 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:34:17.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:17.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:34:17.436 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:34:17.471 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:17.493 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:34:17.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:17.498 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:34:17.601 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:34:17.640 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:17 smithi106 ceph-mon[44423]: pgmap v1673: 17 pgs: 17 active+clean; 5.2 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-04-20T17:34:17.643 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:17.670 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:34:17.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:17.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:34:17.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:17 smithi005 ceph-mon[39562]: pgmap v1673: 17 pgs: 17 active+clean; 5.2 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-04-20T17:34:17.775 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:34:17.825 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:17 smithi171 ceph-mon[44189]: pgmap v1673: 17 pgs: 17 active+clean; 5.2 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-04-20T17:34:17.928 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 39 16 93691 93675 150.106 112.562 0.00422886 0.00666043 2024-04-20T17:34:17.969 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:34:17.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:17.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:34:18.076 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:34:18.106 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:18.127 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:34:18.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:18.134 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:34:18.235 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:34:18.293 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:18.507 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:34:18.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:18.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:34:18.531 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.1 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --force --op remove --pgid 5.0 2024-04-20T17:34:18.614 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:34:18.648 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:18.675 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:34:18.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:18.678 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:34:18.779 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:34:18.810 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:18.833 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:34:18.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:18.836 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:34:18.925 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.1/config 2024-04-20T17:34:18.928 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:34:18.929515 min lat: 0.00100551 max lat: 4.62787 avg lat: 0.00670453 2024-04-20T17:34:18.928 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:34:18.928 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 40 16 95450 95434 149.102 109.938 0.0034777 0.00670453 2024-04-20T17:34:18.932 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:34:18.937 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:34:18.964 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:18.991 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:34:18.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:18.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:34:19.096 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:34:19.130 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:19.158 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:34:19.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:19.163 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:34:19.264 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:34:19.313 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:19.424 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:19 smithi171 ceph-mon[44189]: pgmap v1674: 17 pgs: 17 active+clean; 5.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-04-20T17:34:19.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:19 smithi005 ceph-mon[39562]: pgmap v1674: 17 pgs: 17 active+clean; 5.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-04-20T17:34:19.667 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:34:19.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:19 smithi106 ceph-mon[44423]: pgmap v1674: 17 pgs: 17 active+clean; 5.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-04-20T17:34:19.929 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 41 16 97191 97175 148.119 108.812 0.0121694 0.00674965 2024-04-20T17:34:20.000 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:34:20.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:20.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:34:20.105 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:34:20.133 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:20.158 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:34:20.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:20.163 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:34:20.264 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:34:20.303 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:20.337 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:34:20.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:20.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:34:20.386 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:34:20.421 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:34:20.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:20 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/948568353' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:34:20.440 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:34:20.461 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:34:20.470 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:20.490 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:34:20.492 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:34:20.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:20.495 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:34:20.524 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:34:20.561 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:34:20.595 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:34:20.597 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:34:20.634 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:34:20.657 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:20.674 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:34:20.709 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:34:20.744 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:34:20.744 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:34:20.744 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-20T17:34:20.744 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:34:20.744 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:34:20.744 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:34:20.744 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:34:20.744 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:34:20.744 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:34:20.744 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:34:20.745 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:34:20.745 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:34:20.745 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:34:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:20 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/948568353' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:34:20.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:20 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/948568353' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:34:20.929 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 42 16 98817 98801 147.012 101.625 0.00677716 0.00680086 2024-04-20T17:34:21.029 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:34:21.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:21.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:34:21.137 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:34:21.186 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:21.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:21 smithi005 ceph-mon[39562]: pgmap v1675: 17 pgs: 17 active+clean; 5.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-04-20T17:34:21.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.560 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:34:21.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:21.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:34:21.576 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:34:21.577 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:34:21.666 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:34:21.697 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:21 smithi171 ceph-mon[44189]: pgmap v1675: 17 pgs: 17 active+clean; 5.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-04-20T17:34:21.697 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.697 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.697 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.697 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.697 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.697 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.698 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.698 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.698 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.698 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.698 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:21.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.719 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:34:21.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:21.728 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:34:21.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:21 smithi106 ceph-mon[44423]: pgmap v1675: 17 pgs: 17 active+clean; 5.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-04-20T17:34:21.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:21.829 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:34:21.866 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:21.899 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:34:21.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:21.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:34:21.929 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 43 16 100597 100581 146.18 111.25 0.00434822 0.00683917 2024-04-20T17:34:22.004 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:34:22.041 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:22.064 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:34:22.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:22.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:34:22.170 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:34:22.203 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:22.227 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:34:22.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:22.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:34:22.336 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:34:22.367 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:22.389 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:34:22.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:22.393 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:34:22.494 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:34:22.513 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:22 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:22.531 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:22.556 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:34:22.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:22.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:34:22.604 INFO:teuthology.orchestra.run.smithi005.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:34:22.664 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:34:22.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:22 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:22.715 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:22.800 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:34:22.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:22.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:34:22.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:22 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:22.905 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:34:22.929 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 44 16 102278 102262 145.245 105.062 0.00597275 0.00688369 2024-04-20T17:34:22.939 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:22.963 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:34:22.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:22.968 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:34:23.069 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:34:23.110 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:23.144 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:34:23.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:23.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:34:23.248 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:34:23.281 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:23.313 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:34:23.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:23.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:34:23.345 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:34:23.345 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:34:23.418 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:34:23.444 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:23 smithi106 ceph-mon[44423]: pgmap v1676: 17 pgs: 17 active+clean; 5.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-04-20T17:34:23.449 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:23.474 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:34:23.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:23.479 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:34:23.578 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:34:23.580 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:34:23.609 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:23.630 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:34:23.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:23.633 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:34:23.634 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:23 smithi005 ceph-mon[39562]: pgmap v1676: 17 pgs: 17 active+clean; 5.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-04-20T17:34:23.648 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:34:23.733 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:34:23.797 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:23.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:23 smithi171 ceph-mon[44189]: pgmap v1676: 17 pgs: 17 active+clean; 5.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-04-20T17:34:23.929 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 45 16 103735 103719 144.041 91.0625 0.0135704 0.0069408 2024-04-20T17:34:24.929 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 46 16 105361 105345 143.119 101.625 0.00603408 0.00698556 2024-04-20T17:34:25.290 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-20T17:34:25.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:25 smithi005 ceph-mon[39562]: pgmap v1677: 17 pgs: 17 active+clean; 6.2 GiB data, 16 GiB used, 879 GiB / 894 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-04-20T17:34:25.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:25 smithi106 ceph-mon[44423]: pgmap v1677: 17 pgs: 17 active+clean; 6.2 GiB data, 16 GiB used, 879 GiB / 894 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-04-20T17:34:25.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:25 smithi171 ceph-mon[44189]: pgmap v1677: 17 pgs: 17 active+clean; 6.2 GiB data, 16 GiB used, 879 GiB / 894 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-04-20T17:34:25.929 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 47 16 106927 106911 142.156 97.875 0.012466 0.00703308 2024-04-20T17:34:26.929 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 48 16 108459 108443 141.189 95.75 0.0340315 0.00708167 2024-04-20T17:34:27.099 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:34:27.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:27.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:34:27.205 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:34:27.269 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:27.308 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:34:27.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:27.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:34:27.421 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:34:27.451 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:27.483 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:34:27.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:27.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:34:27.591 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:34:27.639 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:27.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:27 smithi005 ceph-mon[39562]: pgmap v1678: 17 pgs: 17 active+clean; 6.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-04-20T17:34:27.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:27 smithi106 ceph-mon[44423]: pgmap v1678: 17 pgs: 17 active+clean; 6.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-04-20T17:34:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:27 smithi171 ceph-mon[44189]: pgmap v1678: 17 pgs: 17 active+clean; 6.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-04-20T17:34:27.929 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 49 16 109836 109820 140.064 86.0625 0.00657227 0.00713829 2024-04-20T17:34:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:28 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:34:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:28 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:34:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:28 smithi005 ceph-mon[39562]: 6.a scrub starts 2024-04-20T17:34:28.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:28 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:34:28.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:28 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:34:28.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:28 smithi106 ceph-mon[44423]: 6.a scrub starts 2024-04-20T17:34:28.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:28 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:34:28.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:28 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:34:28.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:28 smithi171 ceph-mon[44189]: 6.a scrub starts 2024-04-20T17:34:28.929 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 50 16 111107 111091 138.851 79.4375 0.00703695 0.0072001 2024-04-20T17:34:29.006 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:34:29.013 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:34:29.013 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:34:29.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:29.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:34:29.097 INFO:teuthology.orchestra.run.smithi005.stdout: marking collection for removal 2024-04-20T17:34:29.097 INFO:teuthology.orchestra.run.smithi005.stdout:setting '_remove' omap key 2024-04-20T17:34:29.123 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:34:29.124 INFO:teuthology.orchestra.run.smithi005.stdout:finish_remove_pgs 5.0_head removing 5.0 2024-04-20T17:34:29.178 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:29.192 INFO:teuthology.orchestra.run.smithi005.stdout:Remove successful 2024-04-20T17:34:29.680 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:29 smithi005 ceph-mon[39562]: pgmap v1679: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-04-20T17:34:29.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:29 smithi106 ceph-mon[44423]: pgmap v1679: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-04-20T17:34:29.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:29 smithi171 ceph-mon[44189]: pgmap v1679: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-04-20T17:34:29.929 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 51 16 112502 112486 137.838 87.1875 0.00674184 0.00725355 2024-04-20T17:34:30.251 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:34:30.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:30.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:34:30.364 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:34:30.413 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:30.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:30 smithi005 ceph-mon[39562]: 6.a scrub ok 2024-04-20T17:34:30.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:30 smithi106 ceph-mon[44423]: 6.a scrub ok 2024-04-20T17:34:30.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:30 smithi171 ceph-mon[44189]: 6.a scrub ok 2024-04-20T17:34:30.931 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 52 16 114092 114076 137.098 99.375 0.00437821 0.00729222 2024-04-20T17:34:31.015 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.1 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.5.0.1 2024-04-20T17:34:31.609 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.1/config 2024-04-20T17:34:31.699 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:31 smithi005 ceph-mon[39562]: pgmap v1680: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-04-20T17:34:31.700 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:31 smithi005 ceph-mon[39562]: 6.3 scrub starts 2024-04-20T17:34:31.700 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:31 smithi005 ceph-mon[39562]: 6.0 scrub starts 2024-04-20T17:34:31.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:31 smithi106 ceph-mon[44423]: pgmap v1680: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-04-20T17:34:31.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:31 smithi106 ceph-mon[44423]: 6.3 scrub starts 2024-04-20T17:34:31.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:31 smithi106 ceph-mon[44423]: 6.0 scrub starts 2024-04-20T17:34:31.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:31 smithi171 ceph-mon[44189]: pgmap v1680: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-04-20T17:34:31.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:31 smithi171 ceph-mon[44189]: 6.3 scrub starts 2024-04-20T17:34:31.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:31 smithi171 ceph-mon[44189]: 6.0 scrub starts 2024-04-20T17:34:31.929 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 53 16 115450 115434 136.113 84.875 0.00398108 0.00734521 2024-04-20T17:34:32.700 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:32 smithi005 ceph-mon[39562]: 6.3 scrub ok 2024-04-20T17:34:32.700 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:32 smithi005 ceph-mon[39562]: 6.9 scrub starts 2024-04-20T17:34:32.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:32 smithi106 ceph-mon[44423]: 6.3 scrub ok 2024-04-20T17:34:32.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:32 smithi106 ceph-mon[44423]: 6.9 scrub starts 2024-04-20T17:34:32.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:32 smithi171 ceph-mon[44189]: 6.3 scrub ok 2024-04-20T17:34:32.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:32 smithi171 ceph-mon[44189]: 6.9 scrub starts 2024-04-20T17:34:32.929 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 54 16 116828 116812 135.187 86.125 0.00392194 0.00739465 2024-04-20T17:34:32.989 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:34:32.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:32.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:34:33.096 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:34:33.129 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:33.156 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:34:33.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:33.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:34:33.266 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:34:33.355 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:33.431 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:33 smithi106 ceph-mon[44423]: pgmap v1681: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-04-20T17:34:33.431 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:33 smithi106 ceph-mon[44423]: 6.0 scrub ok 2024-04-20T17:34:33.624 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:34:33.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:33.638 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:33 smithi005 ceph-mon[39562]: pgmap v1681: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-04-20T17:34:33.638 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:33 smithi005 ceph-mon[39562]: 6.0 scrub ok 2024-04-20T17:34:33.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:34:33.739 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:34:33.771 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:33 smithi171 ceph-mon[44189]: pgmap v1681: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-04-20T17:34:33.771 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:33 smithi171 ceph-mon[44189]: 6.0 scrub ok 2024-04-20T17:34:33.776 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:33.805 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:34:33.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:33.809 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:34:33.910 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:34:33.929 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 55 16 118124 118108 134.202 81 0.0117034 0.0074499 2024-04-20T17:34:33.940 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:33.966 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:34:33.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:33.974 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:34:34.075 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:34:34.134 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:34.185 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:34:34.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:34.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:34:34.276 INFO:teuthology.orchestra.run.smithi005.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:34:34.289 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:34:34.339 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:34.364 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:34:34.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:34.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:34:34.468 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:34:34.475 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:34:34.518 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:34.587 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:34:34.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:34.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:34:34.695 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:34:34.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:34 smithi005 ceph-mon[39562]: 6.6 scrub starts 2024-04-20T17:34:34.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:34 smithi005 ceph-mon[39562]: 6.5 scrub starts 2024-04-20T17:34:34.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:34 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/619968064' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:34:34.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:34 smithi005 ceph-mon[39562]: from='client.26921 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:34:34.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:34 smithi005 ceph-mon[39562]: 6.9 scrub ok 2024-04-20T17:34:34.743 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:34.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:34 smithi106 ceph-mon[44423]: 6.6 scrub starts 2024-04-20T17:34:34.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:34 smithi106 ceph-mon[44423]: 6.5 scrub starts 2024-04-20T17:34:34.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:34 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/619968064' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:34:34.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:34 smithi106 ceph-mon[44423]: from='client.26921 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:34:34.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:34 smithi106 ceph-mon[44423]: 6.9 scrub ok 2024-04-20T17:34:34.792 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:34:34.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:34.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:34:34.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:34 smithi171 ceph-mon[44189]: 6.6 scrub starts 2024-04-20T17:34:34.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:34 smithi171 ceph-mon[44189]: 6.5 scrub starts 2024-04-20T17:34:34.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:34 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/619968064' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:34:34.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:34 smithi171 ceph-mon[44189]: from='client.26921 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:34:34.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:34 smithi171 ceph-mon[44189]: 6.9 scrub ok 2024-04-20T17:34:34.902 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:34:34.930 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 56 16 119526 119510 133.37 87.625 0.00637864 0.00749665 2024-04-20T17:34:34.933 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:34.955 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:34:34.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:34.958 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:34:35.059 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:34:35.094 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:35.123 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:34:35.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:35.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:34:35.233 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:34:35.296 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:35.443 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:35 smithi106 ceph-mon[44423]: pgmap v1682: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.1 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-04-20T17:34:35.443 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:35 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:34:35.443 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:35 smithi106 ceph-mon[44423]: from='client.26921 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:34:35.443 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:35 smithi106 ceph-mon[44423]: osdmap e639: 12 total, 11 up, 10 in 2024-04-20T17:34:35.508 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:34:35.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:35.511 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:35 smithi005 ceph-mon[39562]: pgmap v1682: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.1 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-04-20T17:34:35.511 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:35 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:34:35.511 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:35 smithi005 ceph-mon[39562]: from='client.26921 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:34:35.511 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:35 smithi005 ceph-mon[39562]: osdmap e639: 12 total, 11 up, 10 in 2024-04-20T17:34:35.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:34:35.612 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:34:35.663 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:35.732 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:34:35.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:35.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:34:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:35 smithi171 ceph-mon[44189]: pgmap v1682: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 7.1 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-04-20T17:34:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:35 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:34:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:35 smithi171 ceph-mon[44189]: from='client.26921 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:34:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:35 smithi171 ceph-mon[44189]: osdmap e639: 12 total, 11 up, 10 in 2024-04-20T17:34:35.844 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:34:35.880 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:35.909 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:34:35.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:35.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:34:35.930 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 57 16 120970 120954 132.613 90.25 0.00599314 0.00753947 2024-04-20T17:34:36.014 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:34:36.050 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:36.074 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:34:36.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:36.083 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:34:36.175 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:34:36.184 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:34:36.208 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:34:36.212 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:36.235 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:34:36.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:36.242 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:34:36.245 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:34:36.270 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:34:36.307 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:34:36.343 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:34:36.360 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:34:36.376 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:36.402 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:34:36.406 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:34:36.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:36.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:34:36.442 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:34:36.470 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:34:36.509 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:34:36.512 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:34:36.542 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:34:36.542 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:34:36.542 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-20T17:34:36.542 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:34:36.542 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:34:36.542 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:34:36.542 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:34:36.542 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:34:36.542 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:34:36.542 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:34:36.543 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:34:36.543 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:34:36.543 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:34:36.545 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:36.570 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:34:36.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:36.574 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:34:36.675 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:34:36.707 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:36 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2359532398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:34:36.707 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:36.707 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:36 smithi106 ceph-mon[44423]: osdmap e640: 12 total, 11 up, 10 in 2024-04-20T17:34:36.710 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:36.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:36 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2359532398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:34:36.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:36.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:36 smithi005 ceph-mon[39562]: osdmap e640: 12 total, 11 up, 10 in 2024-04-20T17:34:36.739 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:34:36.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:36.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:34:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:36 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2359532398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:34:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:36 smithi171 ceph-mon[44189]: osdmap e640: 12 total, 11 up, 10 in 2024-04-20T17:34:36.844 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:34:36.889 INFO:teuthology.orchestra.run.smithi005.stdout:get_pg_num_history pg_num_history pg_num_history(e613 pg_nums {1={14=1},5={543=16}} deleted_pools ) 2024-04-20T17:34:36.890 INFO:teuthology.orchestra.run.smithi005.stdout:Importing pgid 5.0 2024-04-20T17:34:36.890 INFO:teuthology.orchestra.run.smithi005.stdout:write_pg epoch 613 info 5.0( v 613'16404 (577'6400,613'16404] local-lis/les=543/544 n=0 ec=543/543 lis/c=543/543 les/c/f=544/544/0 sis=543) 2024-04-20T17:34:36.890 INFO:teuthology.orchestra.run.smithi005.stdout:Import successful 2024-04-20T17:34:36.900 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:36.930 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 58 16 122368 122352 131.833 87.375 0.0194202 0.00758393 2024-04-20T17:34:37.113 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:34:37.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:37.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:34:37.221 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:34:37.268 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:37.290 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:34:37.436 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:34:37.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:37.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:34:37.557 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:34:37.615 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:37.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:37 smithi005 ceph-mon[39562]: pgmap v1684: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-04-20T17:34:37.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:37 smithi106 ceph-mon[44423]: pgmap v1684: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-04-20T17:34:37.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:37 smithi171 ceph-mon[44189]: pgmap v1684: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-04-20T17:34:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:37.920 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:34:37.920 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:34:37.930 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 59 16 123752 123736 131.065 86.5 0.0124452 0.00762848 2024-04-20T17:34:37.976 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:34:37.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:37.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:34:38.083 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:34:38.123 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:38.152 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:34:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:38.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:34:38.195 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:34:38.195 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:34:38.195 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:34:38.195 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:34:38.196 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.196 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:207708 4.b 4:d8bef688:::benchmark_data_smithi171_122772_object37516:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:34:38.196 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:48.969421+0000", 2024-04-20T17:34:38.196 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 469.22207419300003, 2024-04-20T17:34:38.196 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.21427681800000001, 2024-04-20T17:34:38.196 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:34:38.196 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:34:38.196 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:34:38.196 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:34:38.196 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:34:38.196 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:34:38.196 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 207708 2024-04-20T17:34:38.196 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.196 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:34:38.197 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.197 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:34:38.197 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.969421+0000", 2024-04-20T17:34:38.197 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.197 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.197 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.197 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:34:38.197 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.969421+0000", 2024-04-20T17:34:38.197 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.197 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.197 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.197 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:34:38.197 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.969424+0000", 2024-04-20T17:34:38.197 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5280000000000002e-06 2024-04-20T17:34:38.198 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.198 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.198 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:34:38.198 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.969432+0000", 2024-04-20T17:34:38.198 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0439999999999996e-06 2024-04-20T17:34:38.198 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.198 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.198 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:34:38.198 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.969448+0000", 2024-04-20T17:34:38.198 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6099999999999998e-05 2024-04-20T17:34:38.198 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.198 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.198 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:34:38.199 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.969458+0000", 2024-04-20T17:34:38.199 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7040000000000006e-06 2024-04-20T17:34:38.199 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.199 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.199 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:34:38.199 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.969514+0000", 2024-04-20T17:34:38.199 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5930000000000002e-05 2024-04-20T17:34:38.199 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.199 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.199 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:34:38.199 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.969613+0000", 2024-04-20T17:34:38.199 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9448000000000001e-05 2024-04-20T17:34:38.199 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.200 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.200 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:34:38.200 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.969690+0000", 2024-04-20T17:34:38.200 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7033999999999995e-05 2024-04-20T17:34:38.200 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.200 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.200 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:34:38.200 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.974635+0000", 2024-04-20T17:34:38.200 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0049453270000000002 2024-04-20T17:34:38.201 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.201 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.201 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:34:38.201 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.183628+0000", 2024-04-20T17:34:38.201 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.20899236600000001 2024-04-20T17:34:38.201 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.201 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.201 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:34:38.201 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.183663+0000", 2024-04-20T17:34:38.201 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5371999999999999e-05 2024-04-20T17:34:38.201 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.201 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.201 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:34:38.201 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.183698+0000", 2024-04-20T17:34:38.202 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4965000000000002e-05 2024-04-20T17:34:38.202 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.202 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:34:38.202 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.202 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.202 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.202 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:207720 4.b 4:da516ea6:::benchmark_data_smithi171_122772_object37528:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:34:38.202 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:48.975163+0000", 2024-04-20T17:34:38.202 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 469.216332803, 2024-04-20T17:34:38.202 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.213299244, 2024-04-20T17:34:38.202 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:34:38.202 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:34:38.202 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:34:38.203 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:34:38.203 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:34:38.203 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:34:38.203 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 207720 2024-04-20T17:34:38.203 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.203 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:34:38.203 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.203 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:34:38.203 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.975163+0000", 2024-04-20T17:34:38.203 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.203 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.203 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.203 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:34:38.203 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.975163+0000", 2024-04-20T17:34:38.204 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.204 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.204 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.204 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:34:38.204 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.975165+0000", 2024-04-20T17:34:38.204 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5330000000000001e-06 2024-04-20T17:34:38.204 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.204 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.204 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:34:38.204 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.975171+0000", 2024-04-20T17:34:38.204 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7790000000000001e-06 2024-04-20T17:34:38.204 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.204 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.204 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:34:38.205 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.975182+0000", 2024-04-20T17:34:38.205 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0543000000000001e-05 2024-04-20T17:34:38.205 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.205 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.205 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:34:38.205 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.975188+0000", 2024-04-20T17:34:38.205 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5400000000000001e-06 2024-04-20T17:34:38.205 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.205 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.205 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:34:38.205 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.975239+0000", 2024-04-20T17:34:38.205 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1264999999999998e-05 2024-04-20T17:34:38.205 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.205 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.205 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:34:38.206 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.975353+0000", 2024-04-20T17:34:38.206 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011350599999999999 2024-04-20T17:34:38.206 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.206 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.206 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:34:38.206 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.975436+0000", 2024-04-20T17:34:38.206 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3159999999999997e-05 2024-04-20T17:34:38.206 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.206 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.206 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:34:38.206 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.989992+0000", 2024-04-20T17:34:38.206 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014555653999999999 2024-04-20T17:34:38.206 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.206 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.207 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:34:38.207 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.188400+0000", 2024-04-20T17:34:38.207 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.19840873000000001 2024-04-20T17:34:38.207 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.207 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.207 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:34:38.207 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.188435+0000", 2024-04-20T17:34:38.207 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4418999999999997e-05 2024-04-20T17:34:38.207 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.207 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.207 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:34:38.207 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.188462+0000", 2024-04-20T17:34:38.207 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7115e-05 2024-04-20T17:34:38.207 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.207 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:34:38.208 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.208 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.208 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.208 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:207727 4.b 4:dc09fbba:::benchmark_data_smithi171_122772_object37535:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:34:38.208 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:48.979624+0000", 2024-04-20T17:34:38.208 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 469.21187150200001, 2024-04-20T17:34:38.208 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.20892259699999999, 2024-04-20T17:34:38.208 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:34:38.208 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:34:38.208 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:34:38.208 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:34:38.208 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:34:38.208 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:34:38.208 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 207727 2024-04-20T17:34:38.208 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.209 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:34:38.209 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.209 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:34:38.209 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.979624+0000", 2024-04-20T17:34:38.209 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.209 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.209 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.209 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:34:38.209 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.979624+0000", 2024-04-20T17:34:38.209 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.209 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.209 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.209 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:34:38.209 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.979626+0000", 2024-04-20T17:34:38.210 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1720000000000001e-06 2024-04-20T17:34:38.210 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.210 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.210 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:34:38.210 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.979632+0000", 2024-04-20T17:34:38.210 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0769999999999996e-06 2024-04-20T17:34:38.211 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.211 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.211 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:34:38.211 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.979647+0000", 2024-04-20T17:34:38.211 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5158e-05 2024-04-20T17:34:38.211 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.211 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.211 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:34:38.211 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.979657+0000", 2024-04-20T17:34:38.211 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9189999999999994e-06 2024-04-20T17:34:38.211 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.211 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.211 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:34:38.212 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.979956+0000", 2024-04-20T17:34:38.212 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00029903699999999999 2024-04-20T17:34:38.212 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.212 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.212 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:34:38.212 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.980090+0000", 2024-04-20T17:34:38.212 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013391100000000001 2024-04-20T17:34:38.212 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.212 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.212 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:34:38.212 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.980196+0000", 2024-04-20T17:34:38.212 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000105494 2024-04-20T17:34:38.212 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.212 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.212 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:34:38.213 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.990002+0000", 2024-04-20T17:34:38.213 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0098060109999999999 2024-04-20T17:34:38.213 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.213 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.213 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:34:38.213 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.188494+0000", 2024-04-20T17:34:38.213 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.19849267000000001 2024-04-20T17:34:38.213 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.213 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.213 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:34:38.213 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.188522+0000", 2024-04-20T17:34:38.213 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7817000000000002e-05 2024-04-20T17:34:38.213 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.213 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.214 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:34:38.214 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.188547+0000", 2024-04-20T17:34:38.214 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4331000000000001e-05 2024-04-20T17:34:38.214 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.214 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:34:38.214 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.214 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.214 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.214 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:207733 4.b 4:df9f7fcf:::benchmark_data_smithi171_122772_object37541:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:34:38.214 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:48.983759+0000", 2024-04-20T17:34:38.214 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 469.20773675700002, 2024-04-20T17:34:38.214 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.214777986, 2024-04-20T17:34:38.214 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:34:38.214 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:34:38.215 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:34:38.215 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:34:38.215 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:34:38.215 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:34:38.215 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 207733 2024-04-20T17:34:38.215 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.215 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:34:38.215 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.215 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:34:38.215 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.983759+0000", 2024-04-20T17:34:38.215 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.215 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.215 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.216 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:34:38.216 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.983759+0000", 2024-04-20T17:34:38.216 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.216 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.216 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.216 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:34:38.216 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.983762+0000", 2024-04-20T17:34:38.216 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.467e-06 2024-04-20T17:34:38.216 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.216 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.216 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:34:38.216 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.983772+0000", 2024-04-20T17:34:38.216 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4229999999999994e-06 2024-04-20T17:34:38.216 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.216 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.216 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:34:38.217 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.983784+0000", 2024-04-20T17:34:38.217 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2437e-05 2024-04-20T17:34:38.217 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.217 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.217 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:34:38.217 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.983792+0000", 2024-04-20T17:34:38.217 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4209999999999998e-06 2024-04-20T17:34:38.217 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.217 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.217 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:34:38.217 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.985019+0000", 2024-04-20T17:34:38.217 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001227302 2024-04-20T17:34:38.217 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.217 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.217 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:34:38.217 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.985157+0000", 2024-04-20T17:34:38.218 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013808900000000001 2024-04-20T17:34:38.218 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.218 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.218 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:34:38.218 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.985294+0000", 2024-04-20T17:34:38.218 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013690800000000001 2024-04-20T17:34:38.218 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.218 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.218 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:34:38.218 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:48.990105+0000", 2024-04-20T17:34:38.218 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0048111919999999997 2024-04-20T17:34:38.218 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.218 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.218 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:34:38.219 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.198445+0000", 2024-04-20T17:34:38.219 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.20834012199999999 2024-04-20T17:34:38.219 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.219 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.219 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:34:38.219 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.198494+0000", 2024-04-20T17:34:38.219 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8949000000000002e-05 2024-04-20T17:34:38.219 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.219 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.219 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:34:38.219 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.198537+0000", 2024-04-20T17:34:38.219 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2676000000000003e-05 2024-04-20T17:34:38.219 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.219 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:34:38.219 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.220 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.220 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.220 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:207757 4.b 4:de0a109c:::benchmark_data_smithi171_122772_object37565:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:34:38.220 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:49.001251+0000", 2024-04-20T17:34:38.220 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 469.19024423399998, 2024-04-20T17:34:38.220 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.19744542300000001, 2024-04-20T17:34:38.220 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:34:38.220 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:34:38.220 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:34:38.220 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:34:38.220 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:34:38.220 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:34:38.220 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 207757 2024-04-20T17:34:38.221 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.221 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:34:38.221 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.221 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:34:38.221 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.001251+0000", 2024-04-20T17:34:38.221 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.221 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.221 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.222 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:34:38.222 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.001251+0000", 2024-04-20T17:34:38.222 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.222 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.222 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.222 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:34:38.222 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.001254+0000", 2024-04-20T17:34:38.222 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.19e-06 2024-04-20T17:34:38.222 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.222 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.222 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:34:38.222 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.001264+0000", 2024-04-20T17:34:38.222 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8449999999999998e-06 2024-04-20T17:34:38.223 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.223 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.223 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:34:38.223 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.001295+0000", 2024-04-20T17:34:38.223 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1170999999999997e-05 2024-04-20T17:34:38.223 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.223 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.223 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:34:38.223 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.001307+0000", 2024-04-20T17:34:38.223 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1559000000000001e-05 2024-04-20T17:34:38.223 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.223 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.223 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:34:38.223 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.001370+0000", 2024-04-20T17:34:38.223 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3133999999999996e-05 2024-04-20T17:34:38.224 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.224 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.224 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:34:38.224 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.001471+0000", 2024-04-20T17:34:38.224 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010098399999999999 2024-04-20T17:34:38.224 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.224 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.224 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:34:38.224 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.001561+0000", 2024-04-20T17:34:38.224 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0149999999999996e-05 2024-04-20T17:34:38.224 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.224 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.224 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:34:38.225 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.005406+0000", 2024-04-20T17:34:38.225 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0038443280000000002 2024-04-20T17:34:38.225 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.225 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.225 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:34:38.225 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.198589+0000", 2024-04-20T17:34:38.225 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.19318358299999999 2024-04-20T17:34:38.225 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.225 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.225 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:34:38.225 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.198647+0000", 2024-04-20T17:34:38.225 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8081999999999997e-05 2024-04-20T17:34:38.225 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.225 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.225 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:34:38.226 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.198697+0000", 2024-04-20T17:34:38.226 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9397000000000002e-05 2024-04-20T17:34:38.226 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.226 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:34:38.226 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.226 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.226 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.226 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:207773 4.b 4:dbe3c1a3:::benchmark_data_smithi171_122772_object37581:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:34:38.226 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:49.013042+0000", 2024-04-20T17:34:38.226 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 469.17845324299998, 2024-04-20T17:34:38.226 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.192015144, 2024-04-20T17:34:38.226 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:34:38.227 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:34:38.227 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:34:38.227 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:34:38.227 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:34:38.227 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:34:38.227 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 207773 2024-04-20T17:34:38.227 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.227 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:34:38.227 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.227 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:34:38.227 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.013042+0000", 2024-04-20T17:34:38.227 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.227 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.227 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.228 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:34:38.228 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.013042+0000", 2024-04-20T17:34:38.228 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.228 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.228 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.228 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:34:38.228 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.013045+0000", 2024-04-20T17:34:38.228 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0070000000000002e-06 2024-04-20T17:34:38.228 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.228 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.228 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:34:38.228 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.013055+0000", 2024-04-20T17:34:38.228 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9599999999999995e-06 2024-04-20T17:34:38.228 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.228 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.229 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:34:38.229 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.013071+0000", 2024-04-20T17:34:38.229 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5273000000000001e-05 2024-04-20T17:34:38.229 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.229 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.229 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:34:38.229 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.013078+0000", 2024-04-20T17:34:38.229 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7090000000000002e-06 2024-04-20T17:34:38.229 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.229 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.229 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:34:38.229 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.013145+0000", 2024-04-20T17:34:38.229 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6511000000000001e-05 2024-04-20T17:34:38.230 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.230 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.230 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:34:38.230 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.013290+0000", 2024-04-20T17:34:38.230 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014574600000000001 2024-04-20T17:34:38.230 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.230 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.230 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:34:38.230 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.013431+0000", 2024-04-20T17:34:38.230 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014091700000000001 2024-04-20T17:34:38.230 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.230 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.230 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:34:38.230 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.017169+0000", 2024-04-20T17:34:38.230 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0037376710000000001 2024-04-20T17:34:38.231 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.231 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.231 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:34:38.231 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.204968+0000", 2024-04-20T17:34:38.231 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.187798989 2024-04-20T17:34:38.231 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.231 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.231 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:34:38.232 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.205025+0000", 2024-04-20T17:34:38.232 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6965000000000001e-05 2024-04-20T17:34:38.232 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.232 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.232 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:34:38.232 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.205057+0000", 2024-04-20T17:34:38.232 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2396e-05 2024-04-20T17:34:38.232 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.232 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:34:38.232 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.232 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.232 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.232 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:207785 4.b 4:db29997d:::benchmark_data_smithi171_122772_object37593:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:34:38.233 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:49.024845+0000", 2024-04-20T17:34:38.233 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 469.16665045899998, 2024-04-20T17:34:38.233 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.180316225, 2024-04-20T17:34:38.233 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:34:38.233 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:34:38.233 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:34:38.233 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:34:38.233 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:34:38.233 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:34:38.233 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 207785 2024-04-20T17:34:38.233 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.233 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:34:38.233 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.233 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:34:38.234 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.024845+0000", 2024-04-20T17:34:38.234 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.234 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.234 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.234 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:34:38.234 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.024845+0000", 2024-04-20T17:34:38.234 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.234 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.234 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.234 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:34:38.234 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.024848+0000", 2024-04-20T17:34:38.234 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4930000000000001e-06 2024-04-20T17:34:38.234 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.235 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.235 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:34:38.235 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.024855+0000", 2024-04-20T17:34:38.235 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2819999999999999e-06 2024-04-20T17:34:38.235 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.235 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.235 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:34:38.235 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.024871+0000", 2024-04-20T17:34:38.235 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6087e-05 2024-04-20T17:34:38.235 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.235 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.235 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:34:38.235 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.024878+0000", 2024-04-20T17:34:38.236 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1239999999999999e-06 2024-04-20T17:34:38.236 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.236 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.236 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:34:38.236 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.024945+0000", 2024-04-20T17:34:38.236 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6458999999999995e-05 2024-04-20T17:34:38.236 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.236 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.236 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:34:38.236 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.025092+0000", 2024-04-20T17:34:38.236 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014733899999999999 2024-04-20T17:34:38.236 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.236 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.236 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:34:38.237 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.025229+0000", 2024-04-20T17:34:38.237 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013684600000000001 2024-04-20T17:34:38.237 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.237 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.237 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:34:38.237 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.030149+0000", 2024-04-20T17:34:38.237 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0049204469999999997 2024-04-20T17:34:38.237 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.237 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.237 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:34:38.237 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.205100+0000", 2024-04-20T17:34:38.237 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17495043399999999 2024-04-20T17:34:38.237 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.237 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.238 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:34:38.238 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.205133+0000", 2024-04-20T17:34:38.238 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.307e-05 2024-04-20T17:34:38.238 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.238 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.238 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:34:38.238 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.205161+0000", 2024-04-20T17:34:38.238 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8643999999999999e-05 2024-04-20T17:34:38.238 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.238 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:34:38.238 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.238 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.239 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.239 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:207792 4.b 4:da25c4e5:::benchmark_data_smithi171_122772_object37600:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:34:38.239 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:49.030811+0000", 2024-04-20T17:34:38.239 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 469.16068403100002, 2024-04-20T17:34:38.239 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17443872099999999, 2024-04-20T17:34:38.239 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:34:38.239 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:34:38.239 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:34:38.239 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:34:38.239 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:34:38.239 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:34:38.239 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 207792 2024-04-20T17:34:38.240 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.240 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:34:38.240 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.240 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:34:38.240 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.030811+0000", 2024-04-20T17:34:38.240 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.240 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.240 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.240 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:34:38.240 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.030811+0000", 2024-04-20T17:34:38.240 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.240 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.240 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.240 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:34:38.240 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.030814+0000", 2024-04-20T17:34:38.241 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.311e-06 2024-04-20T17:34:38.241 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.241 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.241 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:34:38.241 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.030823+0000", 2024-04-20T17:34:38.241 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.5589999999999994e-06 2024-04-20T17:34:38.241 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.241 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.241 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:34:38.241 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.030835+0000", 2024-04-20T17:34:38.241 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2007999999999999e-05 2024-04-20T17:34:38.241 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.242 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.242 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:34:38.242 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.030845+0000", 2024-04-20T17:34:38.242 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7389999999999994e-06 2024-04-20T17:34:38.242 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.242 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.242 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:34:38.242 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.030908+0000", 2024-04-20T17:34:38.242 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2891000000000005e-05 2024-04-20T17:34:38.242 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.243 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.243 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:34:38.243 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.031046+0000", 2024-04-20T17:34:38.243 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013822300000000001 2024-04-20T17:34:38.243 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.243 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.243 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:34:38.243 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.031177+0000", 2024-04-20T17:34:38.243 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013072100000000001 2024-04-20T17:34:38.243 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.243 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.243 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:34:38.243 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.035767+0000", 2024-04-20T17:34:38.244 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.004590055 2024-04-20T17:34:38.244 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.244 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.244 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:34:38.244 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.205188+0000", 2024-04-20T17:34:38.244 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.169421345 2024-04-20T17:34:38.244 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.244 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.244 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:34:38.244 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.205226+0000", 2024-04-20T17:34:38.244 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7698999999999999e-05 2024-04-20T17:34:38.244 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.244 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.244 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:34:38.245 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.205250+0000", 2024-04-20T17:34:38.245 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4170000000000001e-05 2024-04-20T17:34:38.245 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.245 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:34:38.245 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.245 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.245 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.245 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:207802 4.b 4:daaa6950:::benchmark_data_smithi171_122772_object37610:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:34:38.245 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:49.038445+0000", 2024-04-20T17:34:38.245 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 469.15305047200002, 2024-04-20T17:34:38.245 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17428120799999999, 2024-04-20T17:34:38.245 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:34:38.245 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:34:38.245 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:34:38.246 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:34:38.246 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:34:38.246 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:34:38.246 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 207802 2024-04-20T17:34:38.246 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.246 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:34:38.246 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.246 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:34:38.246 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.038445+0000", 2024-04-20T17:34:38.246 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.246 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.246 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.246 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:34:38.246 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.038445+0000", 2024-04-20T17:34:38.247 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.247 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.247 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.247 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:34:38.247 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.038447+0000", 2024-04-20T17:34:38.247 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9750000000000001e-06 2024-04-20T17:34:38.247 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.247 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.247 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:34:38.247 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.038453+0000", 2024-04-20T17:34:38.247 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1840000000000004e-06 2024-04-20T17:34:38.247 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.247 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.247 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:34:38.247 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.038465+0000", 2024-04-20T17:34:38.247 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2254e-05 2024-04-20T17:34:38.248 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.248 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.248 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:34:38.248 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.038472+0000", 2024-04-20T17:34:38.248 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6000000000000003e-06 2024-04-20T17:34:38.248 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.248 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.248 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:34:38.248 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.038533+0000", 2024-04-20T17:34:38.248 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0813000000000002e-05 2024-04-20T17:34:38.248 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.248 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.248 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:34:38.248 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.038628+0000", 2024-04-20T17:34:38.248 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4847999999999997e-05 2024-04-20T17:34:38.248 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.249 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.249 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:34:38.249 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.038708+0000", 2024-04-20T17:34:38.249 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0465999999999998e-05 2024-04-20T17:34:38.249 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.249 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.249 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:34:38.249 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.042385+0000", 2024-04-20T17:34:38.249 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.003677067 2024-04-20T17:34:38.249 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.249 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.249 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:34:38.250 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.212627+0000", 2024-04-20T17:34:38.250 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17024218399999999 2024-04-20T17:34:38.250 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.250 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.250 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:34:38.250 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.212673+0000", 2024-04-20T17:34:38.250 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5843000000000001e-05 2024-04-20T17:34:38.250 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.250 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.250 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:34:38.250 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.212726+0000", 2024-04-20T17:34:38.250 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2973999999999998e-05 2024-04-20T17:34:38.251 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.251 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:34:38.251 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.251 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.251 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.251 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:207806 4.b 4:de8d426d:::benchmark_data_smithi171_122772_object37614:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:34:38.251 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:49.043798+0000", 2024-04-20T17:34:38.251 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 469.14769776399999, 2024-04-20T17:34:38.251 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.169070463, 2024-04-20T17:34:38.251 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:34:38.251 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:34:38.251 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:34:38.251 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:34:38.251 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:34:38.251 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:34:38.251 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 207806 2024-04-20T17:34:38.252 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.252 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:34:38.252 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.252 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:34:38.252 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.043798+0000", 2024-04-20T17:34:38.252 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.252 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.252 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.252 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:34:38.252 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.043798+0000", 2024-04-20T17:34:38.252 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.252 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.252 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.252 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:34:38.252 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.043800+0000", 2024-04-20T17:34:38.253 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3369999999999998e-06 2024-04-20T17:34:38.253 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.253 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.253 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:34:38.253 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.043806+0000", 2024-04-20T17:34:38.253 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1290000000000001e-06 2024-04-20T17:34:38.253 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.253 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.253 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:34:38.253 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.043818+0000", 2024-04-20T17:34:38.253 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1552999999999999e-05 2024-04-20T17:34:38.253 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.253 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.253 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:34:38.253 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.043824+0000", 2024-04-20T17:34:38.254 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5830000000000003e-06 2024-04-20T17:34:38.254 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.254 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.254 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:34:38.254 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.043888+0000", 2024-04-20T17:34:38.254 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3929000000000004e-05 2024-04-20T17:34:38.254 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.254 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.254 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:34:38.254 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.043972+0000", 2024-04-20T17:34:38.254 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3868000000000004e-05 2024-04-20T17:34:38.254 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.254 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.254 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:34:38.255 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.044045+0000", 2024-04-20T17:34:38.255 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3164000000000006e-05 2024-04-20T17:34:38.255 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.255 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.255 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:34:38.255 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.047745+0000", 2024-04-20T17:34:38.255 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0037001109999999999 2024-04-20T17:34:38.255 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.255 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.255 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:34:38.255 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.212776+0000", 2024-04-20T17:34:38.255 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.16503074400000001 2024-04-20T17:34:38.255 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.256 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.256 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:34:38.256 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.212830+0000", 2024-04-20T17:34:38.256 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3383e-05 2024-04-20T17:34:38.256 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.256 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.256 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:34:38.256 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.212868+0000", 2024-04-20T17:34:38.256 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8661999999999997e-05 2024-04-20T17:34:38.256 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.256 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:34:38.256 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.256 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.256 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.256 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:207829 4.b 4:dcebd7b4:::benchmark_data_smithi171_122772_object37637:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:34:38.257 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:49.065769+0000", 2024-04-20T17:34:38.257 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 469.125726585, 2024-04-20T17:34:38.257 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.150629235, 2024-04-20T17:34:38.257 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:34:38.257 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:34:38.257 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:34:38.257 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:34:38.257 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:34:38.257 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:34:38.257 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 207829 2024-04-20T17:34:38.257 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.257 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:34:38.257 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.257 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:34:38.257 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.065769+0000", 2024-04-20T17:34:38.257 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.258 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.258 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.258 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:34:38.258 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.065769+0000", 2024-04-20T17:34:38.258 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.258 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.258 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.258 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:34:38.258 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.065771+0000", 2024-04-20T17:34:38.258 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.345e-06 2024-04-20T17:34:38.258 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.258 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.258 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:34:38.258 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.065779+0000", 2024-04-20T17:34:38.258 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8089999999999993e-06 2024-04-20T17:34:38.259 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.259 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.259 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:34:38.259 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.065796+0000", 2024-04-20T17:34:38.259 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7231000000000002e-05 2024-04-20T17:34:38.259 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.259 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.259 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:34:38.259 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.065804+0000", 2024-04-20T17:34:38.259 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.525e-06 2024-04-20T17:34:38.259 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.259 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.259 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:34:38.259 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.065866+0000", 2024-04-20T17:34:38.260 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2204000000000003e-05 2024-04-20T17:34:38.260 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.260 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.260 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:34:38.260 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.065964+0000", 2024-04-20T17:34:38.260 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7664999999999994e-05 2024-04-20T17:34:38.260 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.260 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.260 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:34:38.260 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.066035+0000", 2024-04-20T17:34:38.260 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1517e-05 2024-04-20T17:34:38.260 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.260 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.260 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:34:38.261 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.069743+0000", 2024-04-20T17:34:38.261 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0037077439999999998 2024-04-20T17:34:38.261 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.261 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.261 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:34:38.261 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.216290+0000", 2024-04-20T17:34:38.261 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.146547179 2024-04-20T17:34:38.261 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.261 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.261 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:34:38.261 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.216350+0000", 2024-04-20T17:34:38.261 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0006000000000001e-05 2024-04-20T17:34:38.261 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.261 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.261 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:34:38.262 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.216398+0000", 2024-04-20T17:34:38.262 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8010000000000003e-05 2024-04-20T17:34:38.262 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.262 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:34:38.262 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.262 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.262 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.262 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:207852 4.b 4:df68e84b:::benchmark_data_smithi171_122772_object37660:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:34:38.262 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:49.093847+0000", 2024-04-20T17:34:38.262 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 469.09764879699998, 2024-04-20T17:34:38.262 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.12622141000000001, 2024-04-20T17:34:38.262 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:34:38.262 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:34:38.262 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:34:38.262 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:34:38.263 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:34:38.263 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:34:38.263 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 207852 2024-04-20T17:34:38.263 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.263 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:34:38.263 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.263 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:34:38.263 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.093847+0000", 2024-04-20T17:34:38.263 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.263 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.263 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.263 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:34:38.263 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.093847+0000", 2024-04-20T17:34:38.263 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.264 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.264 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.264 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:34:38.264 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.093849+0000", 2024-04-20T17:34:38.264 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3290000000000001e-06 2024-04-20T17:34:38.264 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.264 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.264 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:34:38.264 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.093857+0000", 2024-04-20T17:34:38.264 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2730000000000002e-06 2024-04-20T17:34:38.264 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.264 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.264 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:34:38.264 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.093873+0000", 2024-04-20T17:34:38.265 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5560999999999999e-05 2024-04-20T17:34:38.265 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.265 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.265 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:34:38.265 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.093882+0000", 2024-04-20T17:34:38.265 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9160000000000007e-06 2024-04-20T17:34:38.265 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.265 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.265 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:34:38.265 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.093967+0000", 2024-04-20T17:34:38.265 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5352000000000005e-05 2024-04-20T17:34:38.265 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.265 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.265 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:34:38.265 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.094105+0000", 2024-04-20T17:34:38.266 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013794499999999999 2024-04-20T17:34:38.266 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.266 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.266 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:34:38.266 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.094238+0000", 2024-04-20T17:34:38.266 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013271999999999999 2024-04-20T17:34:38.266 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.266 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.266 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:34:38.266 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.101892+0000", 2024-04-20T17:34:38.266 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0076538300000000004 2024-04-20T17:34:38.266 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.266 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.267 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:34:38.267 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.219960+0000", 2024-04-20T17:34:38.267 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.118068546 2024-04-20T17:34:38.267 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.267 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.267 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:34:38.267 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.220019+0000", 2024-04-20T17:34:38.267 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8569000000000001e-05 2024-04-20T17:34:38.267 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.267 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.267 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:34:38.267 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.220068+0000", 2024-04-20T17:34:38.267 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9369000000000001e-05 2024-04-20T17:34:38.267 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.268 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:34:38.268 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.268 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.268 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.268 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26074.0:207855 4.b 4:dbfba1ea:::benchmark_data_smithi171_122772_object37663:head [delete] snapc 0=[] ondisk+write+known_if_redirected e493)", 2024-04-20T17:34:38.268 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:26:49.098673+0000", 2024-04-20T17:34:38.268 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 469.09282266600002, 2024-04-20T17:34:38.268 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.12152695500000001, 2024-04-20T17:34:38.268 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:34:38.268 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:34:38.268 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:34:38.268 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:34:38.268 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26074", 2024-04-20T17:34:38.268 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:34:38.269 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 207855 2024-04-20T17:34:38.269 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.269 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:34:38.269 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.269 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:34:38.269 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.098673+0000", 2024-04-20T17:34:38.269 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.269 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.269 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.269 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:34:38.269 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.098673+0000", 2024-04-20T17:34:38.269 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.269 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.269 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.269 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:34:38.270 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.098676+0000", 2024-04-20T17:34:38.270 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6429999999999999e-06 2024-04-20T17:34:38.270 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.270 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.270 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:34:38.270 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.098694+0000", 2024-04-20T17:34:38.270 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8252000000000001e-05 2024-04-20T17:34:38.270 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.270 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.270 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:34:38.270 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.098709+0000", 2024-04-20T17:34:38.270 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5187e-05 2024-04-20T17:34:38.270 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.270 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.271 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:34:38.271 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.098716+0000", 2024-04-20T17:34:38.271 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8159999999999996e-06 2024-04-20T17:34:38.271 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.271 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.271 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:34:38.271 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.098831+0000", 2024-04-20T17:34:38.271 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011483400000000001 2024-04-20T17:34:38.271 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.271 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.271 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:34:38.271 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.098920+0000", 2024-04-20T17:34:38.271 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9468999999999993e-05 2024-04-20T17:34:38.271 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.271 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.272 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:34:38.272 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.099009+0000", 2024-04-20T17:34:38.272 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8702000000000006e-05 2024-04-20T17:34:38.272 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.272 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.272 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:34:38.272 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.102024+0000", 2024-04-20T17:34:38.272 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0030156789999999998 2024-04-20T17:34:38.272 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.272 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.272 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:34:38.272 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.220109+0000", 2024-04-20T17:34:38.272 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.118084894 2024-04-20T17:34:38.272 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.273 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.273 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:34:38.273 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.220159+0000", 2024-04-20T17:34:38.273 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9611999999999999e-05 2024-04-20T17:34:38.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.273 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.273 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:34:38.273 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:26:49.220200+0000", 2024-04-20T17:34:38.273 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0867e-05 2024-04-20T17:34:38.273 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.273 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:34:38.273 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.273 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.274 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.16506.0:169509 5.b e592/543 5:db99076c:::benchmark_data_smithi171_137281_object38075:head v 592'10695, mlcod=592'10691)", 2024-04-20T17:34:38.274 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:31:39.580065+0000", 2024-04-20T17:34:38.274 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 178.611430052, 2024-04-20T17:34:38.274 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.086079182000000004, 2024-04-20T17:34:38.274 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:34:38.274 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:34:38.274 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:34:38.274 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:34:38.274 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.274 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:34:38.274 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:39.580065+0000", 2024-04-20T17:34:38.274 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.274 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.274 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.274 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:34:38.275 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:39.580065+0000", 2024-04-20T17:34:38.275 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.275 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.275 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.275 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:34:38.275 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:39.580067+0000", 2024-04-20T17:34:38.275 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.153e-06 2024-04-20T17:34:38.275 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.275 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.275 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:34:38.275 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:39.580073+0000", 2024-04-20T17:34:38.275 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5080000000000002e-06 2024-04-20T17:34:38.275 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.275 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.275 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:34:38.276 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:39.580082+0000", 2024-04-20T17:34:38.276 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0000000000000002e-06 2024-04-20T17:34:38.276 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.276 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.276 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:34:38.276 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:39.580088+0000", 2024-04-20T17:34:38.276 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2210000000000002e-06 2024-04-20T17:34:38.276 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.276 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.276 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:34:38.276 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:39.584584+0000", 2024-04-20T17:34:38.276 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0044956279999999998 2024-04-20T17:34:38.276 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.277 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.277 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:34:38.277 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:39.584602+0000", 2024-04-20T17:34:38.277 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7791999999999998e-05 2024-04-20T17:34:38.277 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.277 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.277 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:34:38.277 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:39.666099+0000", 2024-04-20T17:34:38.277 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.081497055999999998 2024-04-20T17:34:38.277 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.277 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.277 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:34:38.277 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:31:39.666145+0000", 2024-04-20T17:34:38.277 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5824e-05 2024-04-20T17:34:38.278 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.278 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:34:38.278 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.278 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.278 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.278 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:58587 6.0 6:004b91e4:::benchmark_data_smithi171_147972_object58586:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e631)", 2024-04-20T17:34:38.278 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:34:00.242469+0000", 2024-04-20T17:34:38.278 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 37.949026646, 2024-04-20T17:34:38.278 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.087995651999999994, 2024-04-20T17:34:38.278 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:34:38.278 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:34:38.278 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:34:38.278 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:34:38.278 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:34:38.278 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:34:38.279 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 58587 2024-04-20T17:34:38.279 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.279 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:34:38.279 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.279 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:34:38.281 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:00.242469+0000", 2024-04-20T17:34:38.281 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.281 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.281 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.281 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:34:38.281 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:00.242469+0000", 2024-04-20T17:34:38.281 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.282 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.282 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.282 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:34:38.282 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:00.242470+0000", 2024-04-20T17:34:38.282 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.429e-06 2024-04-20T17:34:38.282 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.282 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.282 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:34:38.282 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:00.242547+0000", 2024-04-20T17:34:38.282 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7171999999999999e-05 2024-04-20T17:34:38.282 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.282 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.282 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:34:38.282 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:00.242556+0000", 2024-04-20T17:34:38.283 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6060000000000003e-06 2024-04-20T17:34:38.283 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.283 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:34:38.283 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:00.242564+0000", 2024-04-20T17:34:38.283 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7819999999999994e-06 2024-04-20T17:34:38.283 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.283 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:34:38.283 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:00.242624+0000", 2024-04-20T17:34:38.283 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0151000000000001e-05 2024-04-20T17:34:38.283 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.283 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:34:38.283 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:00.242752+0000", 2024-04-20T17:34:38.284 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012796300000000001 2024-04-20T17:34:38.284 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.284 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.284 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:34:38.284 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:00.242877+0000", 2024-04-20T17:34:38.284 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012534199999999999 2024-04-20T17:34:38.284 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.284 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.284 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:34:38.284 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:00.244864+0000", 2024-04-20T17:34:38.284 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0019869290000000001 2024-04-20T17:34:38.284 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.284 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.284 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:34:38.284 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:00.330358+0000", 2024-04-20T17:34:38.285 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.085493342 2024-04-20T17:34:38.285 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.285 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.285 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:34:38.285 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:00.330416+0000", 2024-04-20T17:34:38.285 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8301000000000003e-05 2024-04-20T17:34:38.285 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.285 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.285 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:34:38.285 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:00.330465+0000", 2024-04-20T17:34:38.285 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8634999999999998e-05 2024-04-20T17:34:38.285 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.285 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:34:38.285 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.286 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.286 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.286 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:62050 6.0 6:0f125b26:::benchmark_data_smithi171_147972_object62049:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e633)", 2024-04-20T17:34:38.286 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:34:01.738292+0000", 2024-04-20T17:34:38.286 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 36.453203668999997, 2024-04-20T17:34:38.286 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.085611439999999997, 2024-04-20T17:34:38.286 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:34:38.286 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:34:38.286 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:34:38.286 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:34:38.286 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:34:38.286 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:34:38.286 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 62050 2024-04-20T17:34:38.287 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.287 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:34:38.287 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.288 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:34:38.288 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:01.738292+0000", 2024-04-20T17:34:38.288 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.288 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.288 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.288 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:34:38.288 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:01.738292+0000", 2024-04-20T17:34:38.288 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.288 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.288 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.288 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:34:38.288 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:01.738293+0000", 2024-04-20T17:34:38.288 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.35e-06 2024-04-20T17:34:38.288 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.289 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.289 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:34:38.289 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:01.738350+0000", 2024-04-20T17:34:38.289 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6557000000000001e-05 2024-04-20T17:34:38.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.289 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.289 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:34:38.289 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:01.738358+0000", 2024-04-20T17:34:38.289 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9019999999999999e-06 2024-04-20T17:34:38.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.289 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.289 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:34:38.289 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:01.738363+0000", 2024-04-20T17:34:38.289 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9969999999999998e-06 2024-04-20T17:34:38.290 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.290 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.290 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:34:38.290 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:01.738405+0000", 2024-04-20T17:34:38.290 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.206e-05 2024-04-20T17:34:38.290 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.290 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.290 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:34:38.290 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:01.738493+0000", 2024-04-20T17:34:38.290 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8690000000000006e-05 2024-04-20T17:34:38.290 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.290 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.290 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:34:38.291 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:01.738574+0000", 2024-04-20T17:34:38.291 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0592000000000004e-05 2024-04-20T17:34:38.291 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.291 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.291 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:34:38.291 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:01.740198+0000", 2024-04-20T17:34:38.291 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0016236379999999999 2024-04-20T17:34:38.291 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.291 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.291 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:34:38.291 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:01.823806+0000", 2024-04-20T17:34:38.291 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.083608683000000003 2024-04-20T17:34:38.291 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.292 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:34:38.292 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:01.823874+0000", 2024-04-20T17:34:38.292 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7669999999999994e-05 2024-04-20T17:34:38.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.292 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:34:38.292 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:01.823903+0000", 2024-04-20T17:34:38.292 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9300999999999999e-05 2024-04-20T17:34:38.292 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.292 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:34:38.292 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.293 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:63117 6.0 6:0af12735:::benchmark_data_smithi171_147972_object63116:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e633)", 2024-04-20T17:34:38.293 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:34:02.225104+0000", 2024-04-20T17:34:38.293 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 35.966391936000001, 2024-04-20T17:34:38.293 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10525851999999999, 2024-04-20T17:34:38.293 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:34:38.293 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:34:38.293 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:34:38.293 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:34:38.293 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:34:38.293 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:34:38.293 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 63117 2024-04-20T17:34:38.293 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.293 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:34:38.293 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.293 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:34:38.294 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.225104+0000", 2024-04-20T17:34:38.294 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.294 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.294 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.294 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:34:38.294 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.225104+0000", 2024-04-20T17:34:38.294 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.295 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.295 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:34:38.295 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.225105+0000", 2024-04-20T17:34:38.295 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.375e-06 2024-04-20T17:34:38.295 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.295 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:34:38.295 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.225204+0000", 2024-04-20T17:34:38.295 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9164000000000001e-05 2024-04-20T17:34:38.295 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.295 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:34:38.295 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.225213+0000", 2024-04-20T17:34:38.295 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6090000000000001e-06 2024-04-20T17:34:38.295 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.296 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:34:38.296 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.225218+0000", 2024-04-20T17:34:38.296 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7030000000000003e-06 2024-04-20T17:34:38.296 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.296 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:34:38.296 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.225267+0000", 2024-04-20T17:34:38.296 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8235999999999997e-05 2024-04-20T17:34:38.296 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.296 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:34:38.296 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.225395+0000", 2024-04-20T17:34:38.296 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000128718 2024-04-20T17:34:38.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.297 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.297 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:34:38.297 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.225503+0000", 2024-04-20T17:34:38.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010762100000000001 2024-04-20T17:34:38.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.297 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.297 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:34:38.297 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.227673+0000", 2024-04-20T17:34:38.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0021700460000000001 2024-04-20T17:34:38.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.297 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.297 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:34:38.297 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.330242+0000", 2024-04-20T17:34:38.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10256852700000001 2024-04-20T17:34:38.298 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.298 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.298 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:34:38.298 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.330313+0000", 2024-04-20T17:34:38.298 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1575000000000005e-05 2024-04-20T17:34:38.298 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.298 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.298 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:34:38.298 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.330362+0000", 2024-04-20T17:34:38.298 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8946000000000002e-05 2024-04-20T17:34:38.298 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.298 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:34:38.298 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.298 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.299 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.299 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:63125 6.0 6:0d660bca:::benchmark_data_smithi171_147972_object63124:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e633)", 2024-04-20T17:34:38.299 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:34:02.227894+0000", 2024-04-20T17:34:38.299 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 35.963601840999999, 2024-04-20T17:34:38.299 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.102656784, 2024-04-20T17:34:38.299 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:34:38.299 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:34:38.299 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:34:38.299 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:34:38.299 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:34:38.299 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:34:38.299 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 63125 2024-04-20T17:34:38.299 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.299 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:34:38.299 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.300 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:34:38.300 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.227894+0000", 2024-04-20T17:34:38.300 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.300 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.300 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:34:38.300 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.227894+0000", 2024-04-20T17:34:38.300 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.300 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.300 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:34:38.300 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.227896+0000", 2024-04-20T17:34:38.300 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9649999999999998e-06 2024-04-20T17:34:38.300 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.301 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:34:38.301 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.227929+0000", 2024-04-20T17:34:38.301 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3160000000000001e-05 2024-04-20T17:34:38.301 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.301 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.301 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:34:38.301 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.227937+0000", 2024-04-20T17:34:38.301 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9950000000000005e-06 2024-04-20T17:34:38.301 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.301 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.301 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:34:38.301 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.227942+0000", 2024-04-20T17:34:38.301 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1399999999999999e-06 2024-04-20T17:34:38.301 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.301 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.302 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:34:38.302 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.227975+0000", 2024-04-20T17:34:38.302 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3074000000000002e-05 2024-04-20T17:34:38.302 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.302 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.302 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:34:38.302 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.228058+0000", 2024-04-20T17:34:38.302 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2904000000000004e-05 2024-04-20T17:34:38.302 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.302 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.302 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:34:38.302 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.228139+0000", 2024-04-20T17:34:38.302 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1002999999999997e-05 2024-04-20T17:34:38.302 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.303 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.303 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:34:38.303 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.230748+0000", 2024-04-20T17:34:38.303 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0026092160000000001 2024-04-20T17:34:38.303 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.303 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.303 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:34:38.303 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.330420+0000", 2024-04-20T17:34:38.303 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.099671859000000002 2024-04-20T17:34:38.303 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.303 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.303 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:34:38.303 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.330477+0000", 2024-04-20T17:34:38.303 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6815999999999999e-05 2024-04-20T17:34:38.303 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.304 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.304 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:34:38.304 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.330550+0000", 2024-04-20T17:34:38.304 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3652000000000006e-05 2024-04-20T17:34:38.304 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.304 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:34:38.304 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.304 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.304 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.304 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:63131 6.0 6:0ea5ff68:::benchmark_data_smithi171_147972_object63130:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e633)", 2024-04-20T17:34:38.304 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:34:02.228428+0000", 2024-04-20T17:34:38.304 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 35.963067336000002, 2024-04-20T17:34:38.304 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10229681, 2024-04-20T17:34:38.304 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:34:38.305 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:34:38.305 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:34:38.305 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:34:38.305 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:34:38.305 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:34:38.305 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 63131 2024-04-20T17:34:38.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.305 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:34:38.305 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.305 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:34:38.305 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.228428+0000", 2024-04-20T17:34:38.305 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.305 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.305 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:34:38.305 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.228428+0000", 2024-04-20T17:34:38.306 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.306 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.306 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:34:38.306 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.228429+0000", 2024-04-20T17:34:38.306 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.282e-06 2024-04-20T17:34:38.306 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.306 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:34:38.306 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.228475+0000", 2024-04-20T17:34:38.306 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5296999999999997e-05 2024-04-20T17:34:38.306 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.306 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:34:38.307 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.228482+0000", 2024-04-20T17:34:38.307 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2899999999999997e-06 2024-04-20T17:34:38.307 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.307 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.307 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:34:38.307 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.228487+0000", 2024-04-20T17:34:38.307 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9350000000000002e-06 2024-04-20T17:34:38.307 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.307 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.307 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:34:38.307 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.229730+0000", 2024-04-20T17:34:38.307 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001242618 2024-04-20T17:34:38.307 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.307 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.307 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:34:38.307 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.229808+0000", 2024-04-20T17:34:38.308 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8430999999999995e-05 2024-04-20T17:34:38.308 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.308 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.308 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:34:38.308 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.229885+0000", 2024-04-20T17:34:38.308 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7105000000000001e-05 2024-04-20T17:34:38.308 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.308 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.308 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:34:38.308 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.231846+0000", 2024-04-20T17:34:38.308 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001960495 2024-04-20T17:34:38.308 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.308 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.308 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:34:38.308 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.330623+0000", 2024-04-20T17:34:38.309 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.098777553000000004 2024-04-20T17:34:38.309 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.309 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.309 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:34:38.309 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.330680+0000", 2024-04-20T17:34:38.309 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6915e-05 2024-04-20T17:34:38.309 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.309 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.309 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:34:38.309 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.330725+0000", 2024-04-20T17:34:38.309 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4888999999999997e-05 2024-04-20T17:34:38.309 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.309 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:34:38.309 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.309 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.310 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.310 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:63132 6.0 6:09c327e6:::benchmark_data_smithi171_147972_object63131:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e633)", 2024-04-20T17:34:38.310 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:34:02.228657+0000", 2024-04-20T17:34:38.310 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 35.962838050999999, 2024-04-20T17:34:38.310 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.103457774, 2024-04-20T17:34:38.310 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:34:38.310 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:34:38.310 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:34:38.310 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:34:38.310 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:34:38.310 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:34:38.310 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 63132 2024-04-20T17:34:38.310 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.310 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:34:38.311 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.311 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:34:38.311 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.228657+0000", 2024-04-20T17:34:38.311 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.311 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.311 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.311 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:34:38.311 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.228657+0000", 2024-04-20T17:34:38.311 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:34:38.311 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.311 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.311 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:34:38.311 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.228659+0000", 2024-04-20T17:34:38.311 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.248e-06 2024-04-20T17:34:38.311 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.312 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.312 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:34:38.312 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.228737+0000", 2024-04-20T17:34:38.312 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8089000000000004e-05 2024-04-20T17:34:38.312 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.312 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.312 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:34:38.312 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.228745+0000", 2024-04-20T17:34:38.312 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6479999999999992e-06 2024-04-20T17:34:38.312 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.312 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.312 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:34:38.312 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.228750+0000", 2024-04-20T17:34:38.312 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7559999999999999e-06 2024-04-20T17:34:38.312 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.313 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.313 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:34:38.313 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.230773+0000", 2024-04-20T17:34:38.313 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002022359 2024-04-20T17:34:38.313 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.313 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.313 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:34:38.313 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.230849+0000", 2024-04-20T17:34:38.313 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6057999999999996e-05 2024-04-20T17:34:38.313 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.313 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.313 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:34:38.313 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.230934+0000", 2024-04-20T17:34:38.313 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5749999999999997e-05 2024-04-20T17:34:38.313 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.314 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.314 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:34:38.314 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.231947+0000", 2024-04-20T17:34:38.314 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001012277 2024-04-20T17:34:38.314 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.314 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.314 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:34:38.314 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.330795+0000", 2024-04-20T17:34:38.314 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.098848578000000006 2024-04-20T17:34:38.314 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.314 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.314 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:34:38.314 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.330855+0000", 2024-04-20T17:34:38.314 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9926000000000002e-05 2024-04-20T17:34:38.315 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:34:38.315 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:34:38.315 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:34:38.315 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.332115+0000", 2024-04-20T17:34:38.315 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001260085 2024-04-20T17:34:38.315 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.315 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:34:38.315 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.315 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:34:38.315 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:34:38.315 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:34:38.315 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:34:38.348 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:38.369 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:34:38.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:38.372 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:34:38.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:38 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:34:38.473 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:34:38.523 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:38.569 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:34:38.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:38.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:34:38.664 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/exp.5.0.1 2024-04-20T17:34:38.672 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:34:38.720 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:38 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:34:38.725 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:38.727 INFO:tasks.thrashosds.thrasher:Waiting for clean again 2024-04-20T17:34:38.727 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-20T17:34:38.727 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:34:38.752 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:34:38.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:38.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:34:38.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:38 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:34:38.861 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:34:38.896 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:38.930 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:34:38.931095 min lat: 0.00100551 max lat: 4.62787 avg lat: 0.00767843 2024-04-20T17:34:38.930 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:34:38.930 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 60 16 125032 125016 130.214 80 0.0152484 0.00767843 2024-04-20T17:34:38.955 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:34:38.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:38.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:34:39.060 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:34:39.112 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:39.154 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:34:39.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:39.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:34:39.261 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:34:39.295 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:39.318 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:34:39.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:39.321 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:34:39.362 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-20T17:34:39.392 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:34:39.413 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:34:39.422 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:34:39.458 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:39.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:39 smithi171 ceph-mon[44189]: pgmap v1686: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-20T17:34:39.697 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:39 smithi005 ceph-mon[39562]: pgmap v1686: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-20T17:34:39.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:39 smithi106 ceph-mon[44423]: pgmap v1686: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-20T17:34:39.922 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:34:39.930 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 61 16 126348 126332 129.427 82.25 0.0150335 0.00772502 2024-04-20T17:34:40.238 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:34:40.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:40.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:34:40.289 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:34:40.347 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:34:40.378 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:40.635 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:34:40.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:40.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:34:40.741 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:34:40.777 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:40.910 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:34:40.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:40.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:34:40.930 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 62 16 127693 127677 128.695 84.0625 0.0109622 0.00776859 2024-04-20T17:34:41.013 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:34:41.049 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:41.069 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:34:41.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:41.074 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:34:41.175 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:34:41.206 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:41.226 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:34:41.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:41.229 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:34:41.330 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:34:41.385 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:41.480 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:34:41.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:41.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:34:41.587 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:34:41.627 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:41.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:41 smithi005 ceph-mon[39562]: pgmap v1687: 17 pgs: 17 active+clean; 7.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-04-20T17:34:41.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:41 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:34:41.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:41 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:34:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:41 smithi106 ceph-mon[44423]: pgmap v1687: 17 pgs: 17 active+clean; 7.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-04-20T17:34:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:41 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:34:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:41 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:34:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:41 smithi171 ceph-mon[44189]: pgmap v1687: 17 pgs: 17 active+clean; 7.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-04-20T17:34:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:41 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:34:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:41 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:34:41.930 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 63 16 129049 129033 127.998 84.75 0.005285 0.0078114 2024-04-20T17:34:42.013 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:34:42.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:42.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:34:42.121 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:34:42.149 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:42.171 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:34:42.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:42.174 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:34:42.275 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:34:42.306 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:42.331 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:34:42.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:42.335 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:34:42.436 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:34:42.489 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:42.734 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:42 smithi005 ceph-mon[39562]: osdmap e641: 12 total, 11 up, 10 in 2024-04-20T17:34:42.737 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:42 smithi106 ceph-mon[44423]: osdmap e641: 12 total, 11 up, 10 in 2024-04-20T17:34:42.757 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:34:42.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:42.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:34:42.823 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:34:42.823 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:34:42.823 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.823 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26882.0:130205 6.6 e641/619 6:68d97ba2:::benchmark_data_smithi171_147972_object130204:head v 641'8172, mlcod=641'8169)", 2024-04-20T17:34:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:34:42.817882+0000", 2024-04-20T17:34:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0048741119999999999, 2024-04-20T17:34:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0049041570000000001, 2024-04-20T17:34:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:34:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:34:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-04-20T17:34:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:34:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:34:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.817882+0000", 2024-04-20T17:34:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:34:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:34:42.824 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.817882+0000", 2024-04-20T17:34:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:34:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:34:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.817883+0000", 2024-04-20T17:34:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8999999999999997e-07 2024-04-20T17:34:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:34:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.817917+0000", 2024-04-20T17:34:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3661000000000002e-05 2024-04-20T17:34:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.825 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:34:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.817922+0000", 2024-04-20T17:34:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5079999999999996e-06 2024-04-20T17:34:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:34:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.817927+0000", 2024-04-20T17:34:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1590000000000001e-06 2024-04-20T17:34:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:34:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.817982+0000", 2024-04-20T17:34:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4919000000000001e-05 2024-04-20T17:34:42.826 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:34:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.818003+0000", 2024-04-20T17:34:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1083000000000001e-05 2024-04-20T17:34:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:34:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:34:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:34:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:130210 6.e 6:7ad6c3a4:::benchmark_data_smithi171_147972_object130209:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e641)", 2024-04-20T17:34:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:34:42.818370+0000", 2024-04-20T17:34:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0043867489999999997, 2024-04-20T17:34:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0044854730000000002, 2024-04-20T17:34:42.827 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:34:42.828 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:34:42.828 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:34:42.828 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:34:42.828 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:34:42.828 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:34:42.828 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 130210 2024-04-20T17:34:42.828 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.828 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:34:42.828 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.828 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:34:42.828 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.818370+0000", 2024-04-20T17:34:42.828 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:34:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:34:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.818370+0000", 2024-04-20T17:34:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:34:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:34:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.818372+0000", 2024-04-20T17:34:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9769999999999999e-06 2024-04-20T17:34:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:34:42.829 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.818428+0000", 2024-04-20T17:34:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6332000000000002e-05 2024-04-20T17:34:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:34:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.818438+0000", 2024-04-20T17:34:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8630000000000003e-06 2024-04-20T17:34:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:34:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.818445+0000", 2024-04-20T17:34:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3000000000000004e-06 2024-04-20T17:34:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.830 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:34:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.822163+0000", 2024-04-20T17:34:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037172059999999998 2024-04-20T17:34:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:34:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.822323+0000", 2024-04-20T17:34:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016077600000000001 2024-04-20T17:34:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2", 2024-04-20T17:34:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.822494+0000", 2024-04-20T17:34:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00017101400000000001 2024-04-20T17:34:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:34:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:34:42.831 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:34:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26882.0:130206 6.6 e641/619 6:6c85a1c7:::benchmark_data_smithi171_147972_object130205:head v 641'8173, mlcod=641'8170)", 2024-04-20T17:34:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:34:42.820614+0000", 2024-04-20T17:34:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0021424730000000002, 2024-04-20T17:34:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0023145560000000002, 2024-04-20T17:34:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:34:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:34:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-04-20T17:34:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:34:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:34:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.820614+0000", 2024-04-20T17:34:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:34:42.832 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:34:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.820614+0000", 2024-04-20T17:34:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:34:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:34:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.820615+0000", 2024-04-20T17:34:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4099999999999999e-07 2024-04-20T17:34:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:34:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.820707+0000", 2024-04-20T17:34:42.833 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2485999999999995e-05 2024-04-20T17:34:42.834 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.834 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.834 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:34:42.834 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.820715+0000", 2024-04-20T17:34:42.834 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.61e-06 2024-04-20T17:34:42.834 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.834 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.834 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:34:42.834 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.820721+0000", 2024-04-20T17:34:42.834 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0650000000000004e-06 2024-04-20T17:34:42.834 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.834 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.834 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:34:42.834 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.822175+0000", 2024-04-20T17:34:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001454554 2024-04-20T17:34:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:34:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.822196+0000", 2024-04-20T17:34:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0565e-05 2024-04-20T17:34:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:34:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:34:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:34:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:130211 6.e 6.5c79712e (undecoded) ondisk+write+known_if_redirected e641)", 2024-04-20T17:34:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:34:42.820598+0000", 2024-04-20T17:34:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0021588330000000002, 2024-04-20T17:34:42.835 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0023950690000000001, 2024-04-20T17:34:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:34:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:34:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "queued for pg", 2024-04-20T17:34:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:34:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:34:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:34:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 130211 2024-04-20T17:34:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:34:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:34:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.820598+0000", 2024-04-20T17:34:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:34:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.836 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:34:42.837 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.820598+0000", 2024-04-20T17:34:42.837 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:34:42.837 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.837 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.837 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:34:42.837 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.820599+0000", 2024-04-20T17:34:42.837 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4300000000000001e-06 2024-04-20T17:34:42.837 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.837 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.837 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:34:42.837 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.820753+0000", 2024-04-20T17:34:42.837 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015406600000000001 2024-04-20T17:34:42.838 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.838 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.838 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:34:42.838 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.820760+0000", 2024-04-20T17:34:42.838 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3040000000000001e-06 2024-04-20T17:34:42.838 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:34:42.838 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:34:42.838 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:34:42.838 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:42.820766+0000", 2024-04-20T17:34:42.838 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.037e-06 2024-04-20T17:34:42.838 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:34:42.839 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:34:42.839 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:34:42.839 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:34:42.839 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-04-20T17:34:42.839 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 4 2024-04-20T17:34:42.839 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:34:42.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:42 smithi171 ceph-mon[44189]: osdmap e641: 12 total, 11 up, 10 in 2024-04-20T17:34:42.863 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:34:42.904 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:42.930 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 64 16 130353 130337 127.271 81.5 0.00836251 0.00785601 2024-04-20T17:34:43.270 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:34:43.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:43.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:34:43.384 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:34:43.425 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:43.543 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:34:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:43 smithi106 ceph-mon[44423]: pgmap v1689: 17 pgs: 17 active+clean; 7.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-20T17:34:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:43 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:34:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:43 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:34:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:43 smithi171 ceph-mon[44189]: pgmap v1689: 17 pgs: 17 active+clean; 7.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-20T17:34:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:43 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:34:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:43 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:34:43.846 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:43 smithi005 ceph-mon[39562]: pgmap v1689: 17 pgs: 17 active+clean; 7.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-20T17:34:43.846 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:43 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:34:43.847 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:43 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:34:43.882 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:34:43.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:43.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:34:43.930 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 65 16 131532 131516 126.447 73.6875 0.0196029 0.00790623 2024-04-20T17:34:43.986 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:34:44.041 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:44.082 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:34:44.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:44.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:34:44.188 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:34:44.246 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:44.286 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-20T17:34:44.369 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:34:44.385 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:34:44.387 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:34:44.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:44.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:34:44.490 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:34:44.547 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:34:44.555 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:44.671 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:34:44.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:44 smithi005 ceph-mon[39562]: osdmap e642: 12 total, 11 up, 10 in 2024-04-20T17:34:44.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:44 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/808888248' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:34:44.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:34:44.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:34:44.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:34:44.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:34:44.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:44 smithi005 ceph-mon[39562]: from='client.16833 v1:172.21.15.5:0/836625357' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:34:44.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:44 smithi106 ceph-mon[44423]: osdmap e642: 12 total, 11 up, 10 in 2024-04-20T17:34:44.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:44 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/808888248' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:34:44.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:34:44.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:34:44.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:34:44.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:34:44.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:44 smithi106 ceph-mon[44423]: from='client.16833 v1:172.21.15.5:0/836625357' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:34:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:44 smithi171 ceph-mon[44189]: osdmap e642: 12 total, 11 up, 10 in 2024-04-20T17:34:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:44 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/808888248' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:34:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:34:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:34:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:34:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:34:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:44 smithi171 ceph-mon[44189]: from='client.16833 v1:172.21.15.5:0/836625357' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:34:44.930 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 66 16 132764 132748 125.697 77 0.00511128 0.00795413 2024-04-20T17:34:45.709 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:45 smithi106 ceph-mon[44423]: pgmap v1691: 17 pgs: 17 active+clean; 7.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-04-20T17:34:45.709 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:45 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:34:45.709 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:45 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/808888248' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:34:45.709 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:45 smithi106 ceph-mon[44423]: osdmap e643: 12 total, 11 up, 10 in 2024-04-20T17:34:45.710 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:34:45.710 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:34:45.710 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:34:45.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:45 smithi005 ceph-mon[39562]: pgmap v1691: 17 pgs: 17 active+clean; 7.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-04-20T17:34:45.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:45 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:34:45.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:45 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/808888248' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:34:45.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:45 smithi005 ceph-mon[39562]: osdmap e643: 12 total, 11 up, 10 in 2024-04-20T17:34:45.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:34:45.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:34:45.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:34:45.821 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:34:45.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:45.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:34:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:45 smithi171 ceph-mon[44189]: pgmap v1691: 17 pgs: 17 active+clean; 7.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-04-20T17:34:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:45 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:34:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:45 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/808888248' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:34:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:45 smithi171 ceph-mon[44189]: osdmap e643: 12 total, 11 up, 10 in 2024-04-20T17:34:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:34:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:34:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:34:45.930 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:34:45.931 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 67 16 134163 134147 125.126 87.4375 0.0111469 0.00799063 2024-04-20T17:34:45.967 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:46.029 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:34:46.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:46.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:34:46.133 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:34:46.175 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:46.279 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":1690,"stamp":"2024-04-20T17:34:43.510236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8418034208,"num_objects":128442,"num_object_clones":0,"num_object_copies":256884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128442,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":128621,"num_write_kb":8223098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128577,"ondisk_log_size":128577,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":19876200,"kb_used_data":16334496,"kb_used_omap":224,"kb_used_meta":3541215,"kb_avail":917575320,"statfs":{"total":959950356480,"available":939597127680,"internally_reserved":0,"allocated":16726523904,"data_stored":16722002388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":229753,"internal_metadata":3626204807},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,5,6,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":695992320,"num_objects":10620,"num_object_clones":0,"num_object_copies":21240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10620,"num_write_kb":679680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.119892"},"pg_stats":[{"pgid":"6.f","version":"641'8098","reported_seq":8151,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-04-20T17:34:41.567457+0000","last_change":"2024-04-20T17:33:28.460243+0000","last_active":"2024-04-20T17:34:41.567457+0000","last_peered":"2024-04-20T17:34:41.567457+0000","last_clean":"2024-04-20T17:34:41.567457+0000","last_became_active":"2024-04-20T17:33:28.459992+0000","last_became_peered":"2024-04-20T17:33:28.459992+0000","last_unstale":"2024-04-20T17:34:41.567457+0000","last_undegraded":"2024-04-20T17:34:41.567457+0000","last_fullsized":"2024-04-20T17:34:41.567457+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:33:27.339736+0000","objects_scrubbed":0,"log_size":8098,"log_dups_size":0,"ondisk_log_size":8098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:34:54.687897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":530710528,"num_objects":8098,"num_object_clones":0,"num_object_copies":16196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8098,"num_write_kb":518272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"640'7879","reported_seq":7930,"reported_epoch":640,"state":"active+clean","last_fresh":"2024-04-20T17:34:40.460028+0000","last_change":"2024-04-20T17:33:43.550534+0000","last_active":"2024-04-20T17:34:40.460028+0000","last_peered":"2024-04-20T17:34:40.460028+0000","last_clean":"2024-04-20T17:34:40.460028+0000","last_became_active":"2024-04-20T17:33:43.550299+0000","last_became_peered":"2024-04-20T17:33:43.550299+0000","last_unstale":"2024-04-20T17:34:40.460028+0000","last_undegraded":"2024-04-20T17:34:40.460028+0000","last_fullsized":"2024-04-20T17:34:40.460028+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:33:27.339736+0000","objects_scrubbed":0,"log_size":7879,"log_dups_size":0,"ondisk_log_size":7879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:34:41.123188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":516358144,"num_objects":7879,"num_object_clones":0,"num_object_copies":15758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7879,"num_write_kb":504256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"640'7772","reported_seq":7807,"reported_epoch":640,"state":"active+clean","last_fresh":"2024-04-20T17:34:39.321344+0000","last_change":"2024-04-20T17:33:43.545607+0000","last_active":"2024-04-20T17:34:39.321344+0000","last_peered":"2024-04-20T17:34:39.321344+0000","last_clean":"2024-04-20T17:34:39.321344+0000","last_became_active":"2024-04-20T17:33:43.544300+0000","last_became_peered":"2024-04-20T17:33:43.544300+0000","last_unstale":"2024-04-20T17:34:39.321344+0000","last_undegraded":"2024-04-20T17:34:39.321344+0000","last_fullsized":"2024-04-20T17:34:39.321344+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:33:27.339736+0000","objects_scrubbed":0,"log_size":7772,"log_dups_size":0,"ondisk_log_size":7772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:34:56.537660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":509345792,"num_objects":7772,"num_object_clones":0,"num_object_copies":15544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7772,"num_write_kb":497408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.c","version":"641'8092","reported_seq":8145,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-04-20T17:34:43.429377+0000","last_change":"2024-04-20T17:33:28.356199+0000","last_active":"2024-04-20T17:34:43.429377+0000","last_peered":"2024-04-20T17:34:43.429377+0000","last_clean":"2024-04-20T17:34:43.429377+0000","last_became_active":"2024-04-20T17:33:28.355878+0000","last_became_peered":"2024-04-20T17:33:28.355878+0000","last_unstale":"2024-04-20T17:34:43.429377+0000","last_undegraded":"2024-04-20T17:34:43.429377+0000","last_fullsized":"2024-04-20T17:34:43.429377+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:33:27.339736+0000","objects_scrubbed":0,"log_size":8092,"log_dups_size":0,"ondisk_log_size":8092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:34:50.576624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":530317312,"num_objects":8092,"num_object_clones":0,"num_object_copies":16184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8092,"num_write_kb":517888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"641'8160","reported_seq":8212,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-04-20T17:34:43.470047+0000","last_change":"2024-04-20T17:33:28.354261+0000","last_active":"2024-04-20T17:34:43.470047+0000","last_peered":"2024-04-20T17:34:43.470047+0000","last_clean":"2024-04-20T17:34:43.470047+0000","last_became_active":"2024-04-20T17:33:28.353985+0000","last_became_peered":"2024-04-20T17:33:28.353985+0000","last_unstale":"2024-04-20T17:34:43.470047+0000","last_undegraded":"2024-04-20T17:34:43.470047+0000","last_fullsized":"2024-04-20T17:34:43.470047+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:33:27.339736+0000","objects_scrubbed":0,"log_size":8160,"log_dups_size":0,"ondisk_log_size":8160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":534773760,"num_objects":8160,"num_object_clones":0,"num_object_copies":16320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8160,"num_write_kb":522240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"640'7976","reported_seq":8106,"reported_epoch":640,"state":"active+clean","last_fresh":"2024-04-20T17:34:39.449038+0000","last_change":"2024-04-20T17:34:29.610892+0000","last_active":"2024-04-20T17:34:39.449038+0000","last_peered":"2024-04-20T17:34:39.449038+0000","last_clean":"2024-04-20T17:34:39.449038+0000","last_became_active":"2024-04-20T17:33:28.354030+0000","last_became_peered":"2024-04-20T17:33:28.354030+0000","last_unstale":"2024-04-20T17:34:39.449038+0000","last_undegraded":"2024-04-20T17:34:39.449038+0000","last_fullsized":"2024-04-20T17:34:39.449038+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"638'7110","last_scrub_stamp":"2024-04-20T17:34:29.610691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:34:29.610691+0000","objects_scrubbed":7037,"log_size":7976,"log_dups_size":0,"ondisk_log_size":7976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:35:52.155523+0000","scrub_duration":1493,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":522715136,"num_objects":7976,"num_object_clones":0,"num_object_copies":15952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7976,"num_write_kb":510464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"640'7796","reported_seq":7928,"reported_epoch":640,"state":"active+clean","last_fresh":"2024-04-20T17:34:39.454703+0000","last_change":"2024-04-20T17:34:33.614867+0000","last_active":"2024-04-20T17:34:39.454703+0000","last_peered":"2024-04-20T17:34:39.454703+0000","last_clean":"2024-04-20T17:34:39.454703+0000","last_became_active":"2024-04-20T17:33:28.460508+0000","last_became_peered":"2024-04-20T17:33:28.460508+0000","last_unstale":"2024-04-20T17:34:39.454703+0000","last_undegraded":"2024-04-20T17:34:39.454703+0000","last_fullsized":"2024-04-20T17:34:39.454703+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"638'7310","last_scrub_stamp":"2024-04-20T17:34:33.614684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:34:33.614684+0000","objects_scrubbed":7259,"log_size":7796,"log_dups_size":0,"ondisk_log_size":7796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:35:45.625087+0000","scrub_duration":1518,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":510918656,"num_objects":7796,"num_object_clones":0,"num_object_copies":15592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7796,"num_write_kb":498944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"641'8022","reported_seq":8075,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-04-20T17:34:42.508850+0000","last_change":"2024-04-20T17:33:28.354174+0000","last_active":"2024-04-20T17:34:42.508850+0000","last_peered":"2024-04-20T17:34:42.508850+0000","last_clean":"2024-04-20T17:34:42.508850+0000","last_became_active":"2024-04-20T17:33:28.353875+0000","last_became_peered":"2024-04-20T17:33:28.353875+0000","last_unstale":"2024-04-20T17:34:42.508850+0000","last_undegraded":"2024-04-20T17:34:42.508850+0000","last_fullsized":"2024-04-20T17:34:42.508850+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:33:27.339736+0000","objects_scrubbed":0,"log_size":8022,"log_dups_size":0,"ondisk_log_size":8022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:34:42.869603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":525729792,"num_objects":8022,"num_object_clones":0,"num_object_copies":16044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8022,"num_write_kb":513408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"641'8229","reported_seq":8356,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-04-20T17:34:43.442160+0000","last_change":"2024-04-20T17:34:34.489315+0000","last_active":"2024-04-20T17:34:43.442160+0000","last_peered":"2024-04-20T17:34:43.442160+0000","last_clean":"2024-04-20T17:34:43.442160+0000","last_became_active":"2024-04-20T17:33:28.356179+0000","last_became_peered":"2024-04-20T17:33:28.356179+0000","last_unstale":"2024-04-20T17:34:43.442160+0000","last_undegraded":"2024-04-20T17:34:43.442160+0000","last_fullsized":"2024-04-20T17:34:43.442160+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:33:27.339736+0000","objects_scrubbed":6849,"log_size":8229,"log_dups_size":0,"ondisk_log_size":8229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:34:44.828716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":539295744,"num_objects":8229,"num_object_clones":0,"num_object_copies":16458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8229,"num_write_kb":526656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"641'8243","reported_seq":8364,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-04-20T17:34:42.512312+0000","last_change":"2024-04-20T17:34:34.484244+0000","last_active":"2024-04-20T17:34:42.512312+0000","last_peered":"2024-04-20T17:34:42.512312+0000","last_clean":"2024-04-20T17:34:42.512312+0000","last_became_active":"2024-04-20T17:33:28.460226+0000","last_became_peered":"2024-04-20T17:33:28.460226+0000","last_unstale":"2024-04-20T17:34:42.512312+0000","last_undegraded":"2024-04-20T17:34:42.512312+0000","last_fullsized":"2024-04-20T17:34:42.512312+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:33:27.339736+0000","objects_scrubbed":6499,"log_size":8243,"log_dups_size":0,"ondisk_log_size":8243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540213248,"num_objects":8243,"num_object_clones":0,"num_object_copies":16486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8243,"num_write_kb":527552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"641'8224","reported_seq":8261,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-04-20T17:34:43.422260+0000","last_change":"2024-04-20T17:33:43.545455+0000","last_active":"2024-04-20T17:34:43.422260+0000","last_peered":"2024-04-20T17:34:43.422260+0000","last_clean":"2024-04-20T17:34:43.422260+0000","last_became_active":"2024-04-20T17:33:43.544269+0000","last_became_peered":"2024-04-20T17:33:43.544269+0000","last_unstale":"2024-04-20T17:34:43.422260+0000","last_undegraded":"2024-04-20T17:34:43.422260+0000","last_fullsized":"2024-04-20T17:34:43.422260+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:33:27.339736+0000","objects_scrubbed":0,"log_size":8224,"log_dups_size":0,"ondisk_log_size":8224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:34:54.795716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538968064,"num_objects":8224,"num_object_clones":0,"num_object_copies":16448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8224,"num_write_kb":526336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"641'8032","reported_seq":8164,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-04-20T17:34:42.529882+0000","last_change":"2024-04-20T17:34:31.723664+0000","last_active":"2024-04-20T17:34:42.529882+0000","last_peered":"2024-04-20T17:34:42.529882+0000","last_clean":"2024-04-20T17:34:42.529882+0000","last_became_active":"2024-04-20T17:33:28.353915+0000","last_became_peered":"2024-04-20T17:33:28.353915+0000","last_unstale":"2024-04-20T17:34:42.529882+0000","last_undegraded":"2024-04-20T17:34:42.529882+0000","last_fullsized":"2024-04-20T17:34:42.529882+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"638'7153","last_scrub_stamp":"2024-04-20T17:34:31.723512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:34:31.723512+0000","objects_scrubbed":7103,"log_size":8032,"log_dups_size":0,"ondisk_log_size":8032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:35:43.399646+0000","scrub_duration":1282,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526385152,"num_objects":8032,"num_object_clones":0,"num_object_copies":16064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8032,"num_write_kb":514048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"640'7994","reported_seq":8045,"reported_epoch":640,"state":"active+clean","last_fresh":"2024-04-20T17:34:40.471929+0000","last_change":"2024-04-20T17:33:28.355427+0000","last_active":"2024-04-20T17:34:40.471929+0000","last_peered":"2024-04-20T17:34:40.471929+0000","last_clean":"2024-04-20T17:34:40.471929+0000","last_became_active":"2024-04-20T17:33:28.355126+0000","last_became_peered":"2024-04-20T17:33:28.355126+0000","last_unstale":"2024-04-20T17:34:40.471929+0000","last_undegraded":"2024-04-20T17:34:40.471929+0000","last_fullsized":"2024-04-20T17:34:40.471929+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:33:27.339736+0000","objects_scrubbed":0,"log_size":7994,"log_dups_size":0,"ondisk_log_size":7994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:34:50.404337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":523894784,"num_objects":7994,"num_object_clones":0,"num_object_copies":15988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7994,"num_write_kb":511616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"640'7861","reported_seq":7912,"reported_epoch":640,"state":"active+clean","last_fresh":"2024-04-20T17:34:38.798497+0000","last_change":"2024-04-20T17:33:28.460798+0000","last_active":"2024-04-20T17:34:38.798497+0000","last_peered":"2024-04-20T17:34:38.798497+0000","last_clean":"2024-04-20T17:34:38.798497+0000","last_became_active":"2024-04-20T17:33:28.460449+0000","last_became_peered":"2024-04-20T17:33:28.460449+0000","last_unstale":"2024-04-20T17:34:38.798497+0000","last_undegraded":"2024-04-20T17:34:38.798497+0000","last_fullsized":"2024-04-20T17:34:38.798497+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:33:27.339736+0000","objects_scrubbed":0,"log_size":7861,"log_dups_size":0,"ondisk_log_size":7861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:34:41.390379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":515178496,"num_objects":7861,"num_object_clones":0,"num_object_copies":15722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7861,"num_write_kb":503104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.0","version":"640'7852","reported_seq":7983,"reported_epoch":640,"state":"active+clean","last_fresh":"2024-04-20T17:34:39.447672+0000","last_change":"2024-04-20T17:34:32.958315+0000","last_active":"2024-04-20T17:34:39.447672+0000","last_peered":"2024-04-20T17:34:39.447672+0000","last_clean":"2024-04-20T17:34:39.447672+0000","last_became_active":"2024-04-20T17:33:28.352816+0000","last_became_peered":"2024-04-20T17:33:28.352816+0000","last_unstale":"2024-04-20T17:34:39.447672+0000","last_undegraded":"2024-04-20T17:34:39.447672+0000","last_fullsized":"2024-04-20T17:34:39.447672+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"638'7310","last_scrub_stamp":"2024-04-20T17:34:32.958145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:34:32.958145+0000","objects_scrubbed":7224,"log_size":7852,"log_dups_size":0,"ondisk_log_size":7852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:35:36.957806+0000","scrub_duration":1812,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":514588672,"num_objects":7852,"num_object_clones":0,"num_object_copies":15704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7852,"num_write_kb":502528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"641'8210","reported_seq":8262,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-04-20T17:34:43.460617+0000","last_change":"2024-04-20T17:33:28.356015+0000","last_active":"2024-04-20T17:34:43.460617+0000","last_peered":"2024-04-20T17:34:43.460617+0000","last_clean":"2024-04-20T17:34:43.460617+0000","last_became_active":"2024-04-20T17:33:28.355779+0000","last_became_peered":"2024-04-20T17:33:28.355779+0000","last_unstale":"2024-04-20T17:34:43.460617+0000","last_undegraded":"2024-04-20T17:34:43.460617+0000","last_fullsized":"2024-04-20T17:34:43.460617+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:33:27.339736+0000","objects_scrubbed":0,"log_size":8210,"log_dups_size":0,"ondisk_log_size":8210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538050560,"num_objects":8210,"num_object_clones":0,"num_object_copies":16420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8210,"num_write_kb":525440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":852,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-04-20T17:34:41.486170+0000","last_change":"2024-04-20T17:34:27.569542+0000","last_active":"2024-04-20T17:34:41.486170+0000","last_peered":"2024-04-20T17:34:41.486170+0000","last_clean":"2024-04-20T17:34:41.486170+0000","last_became_active":"2024-04-20T17:23:24.109202+0000","last_became_peered":"2024-04-20T17:23:24.109202+0000","last_unstale":"2024-04-20T17:34:41.486170+0000","last_undegraded":"2024-04-20T17:34:41.486170+0000","last_fullsized":"2024-04-20T17:34:41.486170+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:34:27.569433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:34:27.569433+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:35:53.198377+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":8417443840,"num_objects":128440,"num_object_clones":0,"num_object_copies":256880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":128440,"num_write_kb":8220160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16789209088,"data_stored":16789209088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128440,"ondisk_log_size":128440,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991580,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1579424,"kb_used_data":984600,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":92165728,"statfs":{"total":95995035648,"available":94377705472,"internally_reserved":0,"allocated":1008230400,"data_stored":1007758486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":609089988},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253219,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1475288,"kb_used_data":1002640,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":92269864,"statfs":{"total":95995035648,"available":94484340736,"internally_reserved":0,"allocated":1026703360,"data_stored":1026234998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":483981760},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612968,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1957720,"kb_used_data":1533648,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":91787432,"statfs":{"total":95995035648,"available":93990330368,"internally_reserved":0,"allocated":1570455552,"data_stored":1569987210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":434239936},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809198,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":768984,"kb_used_data":503184,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":92976168,"statfs":{"total":95995035648,"available":95207596032,"internally_reserved":0,"allocated":515260416,"data_stored":514792074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":398,"seq":1709396983955,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2284024,"kb_used_data":2069424,"kb_used_omap":6,"kb_used_meta":214585,"kb_avail":91461128,"statfs":{"total":95995035648,"available":93656195072,"internally_reserved":0,"allocated":2119090176,"data_stored":2118702806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6767,"internal_metadata":219735441},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":0,"seq":2297807503429,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738772,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1986840,"kb_used_data":1508624,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":91758312,"statfs":{"total":95995035648,"available":93960511488,"internally_reserved":0,"allocated":1544830976,"data_stored":1544362634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":489683397},"hb_peers":[2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394063,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1193688,"kb_used_data":1023120,"kb_used_omap":202,"kb_used_meta":170357,"kb_avail":92551464,"statfs":{"total":95995035648,"available":94772699136,"internally_reserved":0,"allocated":1047674880,"data_stored":1047206538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":174446039},"hb_peers":[0,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653113,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3762728,"kb_used_data":3644072,"kb_used_omap":4,"kb_used_meta":118459,"kb_avail":89982424,"statfs":{"total":95995035648,"available":92142002176,"internally_reserved":0,"allocated":3731529728,"data_stored":3731145910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":121302727},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986560,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1862616,"kb_used_data":1520464,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":91882536,"statfs":{"total":95995035648,"available":94087716864,"internally_reserved":0,"allocated":1556955136,"data_stored":1556486794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790331,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3004888,"kb_used_data":2544720,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":90740264,"statfs":{"total":95995035648,"available":92918030336,"internally_reserved":0,"allocated":2605793280,"data_stored":2605324938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1548353536,"data_stored":1548353536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1052835840,"data_stored":1052835840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3756785664,"data_stored":3756785664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1562902528,"data_stored":1562902528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2604924928,"data_stored":2604924928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2121924608,"data_stored":2121924608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":515178496,"data_stored":515178496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1584988160,"data_stored":1584988160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1028653056,"data_stored":1028653056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1012662272,"data_stored":1012662272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:34:46.280 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:34:46.434 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:34:47.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:47.122 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:34:47.122 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 68 16 135436 135420 124.456 79.5625 0.0055215 0.0080335 2024-04-20T17:34:47.122 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:34:47.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:34:47.124 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:34:47.124 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:34:47.124 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-04-20T17:34:47.124 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-04-20T17:34:47.124 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:34:47.124 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:34:47.124 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:34:47.124 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:34:47.124 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:34:47.148 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:34:47.184 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:34:47.221 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:34:47.224 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:34:47.261 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:34:47.268 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:47.294 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:34:47.331 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:34:47.332 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:34:47.332 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-20T17:34:47.332 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:34:47.332 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:34:47.332 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:34:47.332 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:34:47.332 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:34:47.332 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:34:47.332 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:34:47.332 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:34:47.332 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:34:47.332 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:34:47.427 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:34:47.621 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-20T17:34:47.634 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:34:47.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:47.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:34:47.739 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:34:47.774 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:47 smithi106 ceph-mon[44423]: pgmap v1693: 17 pgs: 17 active+clean; 8.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-20T17:34:47.774 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:47 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1353083381' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:34:47.774 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:47.774 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:47.788 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:47.822 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:34:47.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:47.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:34:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:47 smithi171 ceph-mon[44189]: pgmap v1693: 17 pgs: 17 active+clean; 8.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-20T17:34:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:47 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1353083381' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:34:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:47.924 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:34:47.931 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 69 16 136709 136693 123.805 79.5625 0.0163887 0.00807565 2024-04-20T17:34:47.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:47 smithi005 ceph-mon[39562]: pgmap v1693: 17 pgs: 17 active+clean; 8.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-20T17:34:47.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:47 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1353083381' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:34:47.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:47.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:47.975 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:48.046 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:34:48.625 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:48.625 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:48.625 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:48.625 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:48.625 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:48.625 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:48.625 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:48.625 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:48.625 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:48.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:48.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:48.931 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 70 16 138093 138077 123.272 86.5 0.0104499 0.00811067 2024-04-20T17:34:48.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:48.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:48.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:48.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:48.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:48.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:48.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:48.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:48.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:34:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:49 smithi171 ceph-mon[44189]: pgmap v1694: 17 pgs: 17 active+clean; 8.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-04-20T17:34:49.856 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:49 smithi106 ceph-mon[44423]: pgmap v1694: 17 pgs: 17 active+clean; 8.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-04-20T17:34:49.931 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 71 16 139437 139421 122.719 84 0.011059 0.00814716 2024-04-20T17:34:49.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:49 smithi005 ceph-mon[39562]: pgmap v1694: 17 pgs: 17 active+clean; 8.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-04-20T17:34:50.042 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:34:50.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:50.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:34:50.152 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:34:50.182 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:34:50.182 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:34:50.200 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:50.228 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:34:50.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:50.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:34:50.336 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:34:50.375 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:50.419 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:34:50.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:50.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:34:50.527 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:34:50.572 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:50.918 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:34:50.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:50.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:34:50.931 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 72 16 140738 140722 122.144 81.3125 0.00667953 0.00818609 2024-04-20T17:34:50.953 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-20T17:34:51.025 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:34:51.116 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:51.159 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:34:51.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:51.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:34:51.265 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:34:51.325 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:51.368 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:34:51.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:51.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:34:51.392 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:34:51.472 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:34:51.508 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:51.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:51 smithi005 ceph-mon[39562]: pgmap v1695: 17 pgs: 17 active+clean; 8.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-20T17:34:51.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:51 smithi171 ceph-mon[44189]: pgmap v1695: 17 pgs: 17 active+clean; 8.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-20T17:34:51.931 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 73 16 141942 141926 121.501 75.25 0.00995517 0.00822887 2024-04-20T17:34:51.959 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:51 smithi106 ceph-mon[44423]: pgmap v1695: 17 pgs: 17 active+clean; 8.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-20T17:34:52.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:52 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:34:52.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:52 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:34:52.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:52 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:34:52.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:52 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:34:52.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:52 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:34:52.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:52 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:34:52.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:52 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:34:52.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:52 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:34:52.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:52 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:34:52.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:52 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:34:52.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:52 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:34:52.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:52 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:34:52.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:52 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:34:52.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:52 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:34:52.931 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 74 16 143131 143115 120.864 74.3125 0.0122226 0.00827278 2024-04-20T17:34:52.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:52 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:34:52.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:52 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:34:52.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:52 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:34:52.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:52 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:34:52.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:52 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:34:52.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:52 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:34:52.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:52 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:34:53.509 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:34:53.509 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:34:53.931 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 75 16 144396 144380 120.306 79.0625 0.00899196 0.00831069 2024-04-20T17:34:54.165 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:34:54.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:54.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:34:54.270 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:34:54.287 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-20T17:34:54.314 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:54.341 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:34:54.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:54.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:34:54.363 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:34:54.364 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:34:54.447 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:34:54.488 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:54.533 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:34:54.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:54.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:34:54.640 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:34:54.696 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:54.699 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:34:54.743 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:34:54.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:54.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:34:54.851 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:34:54.874 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:34:54.931 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 76 16 145615 145599 119.726 76.1875 0.0166228 0.00835093 2024-04-20T17:34:54.937 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:54 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:34:54.937 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:54 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:34:54.937 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:54 smithi005 ceph-mon[39562]: osdmap e644: 12 total, 11 up, 10 in 2024-04-20T17:34:54.940 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:54.994 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:54 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:34:54.994 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:54 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:34:54.994 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:54 smithi106 ceph-mon[44423]: osdmap e644: 12 total, 11 up, 10 in 2024-04-20T17:34:55.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:54 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:34:55.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:54 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:34:55.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:54 smithi171 ceph-mon[44189]: osdmap e644: 12 total, 11 up, 10 in 2024-04-20T17:34:55.094 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:34:55.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:55.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:34:55.205 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:34:55.274 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:55.278 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:34:55.283 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:34:55.312 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:34:55.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:55.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:34:55.418 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:34:55.459 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:55.931 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 77 16 146804 146788 119.136 74.3125 0.0168701 0.00839252 2024-04-20T17:34:55.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:55 smithi005 ceph-mon[39562]: pgmap v1698: 17 pgs: 17 active+clean; 8.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-04-20T17:34:55.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:55 smithi005 ceph-mon[39562]: from='client.26942 v1:172.21.15.5:0/3789043361' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:34:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:55 smithi106 ceph-mon[44423]: pgmap v1698: 17 pgs: 17 active+clean; 8.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-04-20T17:34:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:55 smithi106 ceph-mon[44423]: from='client.26942 v1:172.21.15.5:0/3789043361' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:34:56.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:55 smithi171 ceph-mon[44189]: pgmap v1698: 17 pgs: 17 active+clean; 8.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-04-20T17:34:56.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:55 smithi171 ceph-mon[44189]: from='client.26942 v1:172.21.15.5:0/3789043361' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:34:56.365 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:34:56.738 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:34:56.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:56.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:34:56.846 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:34:56.879 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:56.900 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:34:56.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:56.904 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:34:56.934 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 78 16 147957 147941 118.532 72.0625 0.0103063 0.00843476 2024-04-20T17:34:56.943 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:34:56.943 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:34:57.005 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:34:57.048 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:57.264 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:34:57.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:57.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:34:57.373 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:34:57.418 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:57.754 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-20T17:34:57.776 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":1698,"stamp":"2024-04-20T17:34:54.397895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":9320333856,"num_objects":142210,"num_object_clones":0,"num_object_copies":284420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142210,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":142389,"num_write_kb":9104250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142345,"ondisk_log_size":142345,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":21801536,"kb_used_data":18121192,"kb_used_omap":224,"kb_used_meta":3680031,"kb_avail":915649984,"statfs":{"total":959950356480,"available":937625583616,"internally_reserved":0,"allocated":18556100608,"data_stored":18552604276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":229753,"internal_metadata":3768352391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":771489792,"num_objects":11772,"num_object_clones":0,"num_object_copies":23544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11772,"num_write_kb":753408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.878072"},"pg_stats":[{"pgid":"6.f","version":"643'8935","reported_seq":8992,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-04-20T17:34:51.586536+0000","last_change":"2024-04-20T17:33:28.460243+0000","last_active":"2024-04-20T17:34:51.586536+0000","last_peered":"2024-04-20T17:34:51.586536+0000","last_clean":"2024-04-20T17:34:51.586536+0000","last_became_active":"2024-04-20T17:33:28.459992+0000","last_became_peered":"2024-04-20T17:33:28.459992+0000","last_unstale":"2024-04-20T17:34:51.586536+0000","last_undegraded":"2024-04-20T17:34:51.586536+0000","last_fullsized":"2024-04-20T17:34:51.586536+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:33:27.339736+0000","objects_scrubbed":0,"log_size":8935,"log_dups_size":0,"ondisk_log_size":8935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:34:54.687897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585564160,"num_objects":8935,"num_object_clones":0,"num_object_copies":17870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8935,"num_write_kb":571840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"643'8720","reported_seq":8777,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-04-20T17:34:50.481846+0000","last_change":"2024-04-20T17:33:43.550534+0000","last_active":"2024-04-20T17:34:50.481846+0000","last_peered":"2024-04-20T17:34:50.481846+0000","last_clean":"2024-04-20T17:34:50.481846+0000","last_became_active":"2024-04-20T17:33:43.550299+0000","last_became_peered":"2024-04-20T17:33:43.550299+0000","last_unstale":"2024-04-20T17:34:50.481846+0000","last_undegraded":"2024-04-20T17:34:50.481846+0000","last_fullsized":"2024-04-20T17:34:50.481846+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:33:27.339736+0000","objects_scrubbed":0,"log_size":8720,"log_dups_size":0,"ondisk_log_size":8720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:34:54.964652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":571473920,"num_objects":8720,"num_object_clones":0,"num_object_copies":17440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8720,"num_write_kb":558080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"644'8982","reported_seq":9025,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-04-20T17:34:54.352579+0000","last_change":"2024-04-20T17:33:43.545607+0000","last_active":"2024-04-20T17:34:54.352579+0000","last_peered":"2024-04-20T17:34:54.352579+0000","last_clean":"2024-04-20T17:34:54.352579+0000","last_became_active":"2024-04-20T17:33:43.544300+0000","last_became_peered":"2024-04-20T17:33:43.544300+0000","last_unstale":"2024-04-20T17:34:54.352579+0000","last_undegraded":"2024-04-20T17:34:54.352579+0000","last_fullsized":"2024-04-20T17:34:54.352579+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:33:27.339736+0000","objects_scrubbed":0,"log_size":8982,"log_dups_size":0,"ondisk_log_size":8982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:34:56.537660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588644352,"num_objects":8982,"num_object_clones":0,"num_object_copies":17964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8982,"num_write_kb":574848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.c","version":"643'8851","reported_seq":8908,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-04-20T17:34:53.429645+0000","last_change":"2024-04-20T17:33:28.356199+0000","last_active":"2024-04-20T17:34:53.429645+0000","last_peered":"2024-04-20T17:34:53.429645+0000","last_clean":"2024-04-20T17:34:53.429645+0000","last_became_active":"2024-04-20T17:33:28.355878+0000","last_became_peered":"2024-04-20T17:33:28.355878+0000","last_unstale":"2024-04-20T17:34:53.429645+0000","last_undegraded":"2024-04-20T17:34:53.429645+0000","last_fullsized":"2024-04-20T17:34:53.429645+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:33:27.339736+0000","objects_scrubbed":0,"log_size":8851,"log_dups_size":0,"ondisk_log_size":8851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:34:56.768590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":580059136,"num_objects":8851,"num_object_clones":0,"num_object_copies":17702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8851,"num_write_kb":566464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"643'8949","reported_seq":9003,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-04-20T17:34:53.457702+0000","last_change":"2024-04-20T17:33:28.354261+0000","last_active":"2024-04-20T17:34:53.457702+0000","last_peered":"2024-04-20T17:34:53.457702+0000","last_clean":"2024-04-20T17:34:53.457702+0000","last_became_active":"2024-04-20T17:33:28.353985+0000","last_became_peered":"2024-04-20T17:33:28.353985+0000","last_unstale":"2024-04-20T17:34:53.457702+0000","last_undegraded":"2024-04-20T17:34:53.457702+0000","last_fullsized":"2024-04-20T17:34:53.457702+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:33:27.339736+0000","objects_scrubbed":0,"log_size":8949,"log_dups_size":0,"ondisk_log_size":8949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:34:54.761500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":586481664,"num_objects":8949,"num_object_clones":0,"num_object_copies":17898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8949,"num_write_kb":572736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"643'8796","reported_seq":8932,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-04-20T17:34:49.472514+0000","last_change":"2024-04-20T17:34:29.610892+0000","last_active":"2024-04-20T17:34:49.472514+0000","last_peered":"2024-04-20T17:34:49.472514+0000","last_clean":"2024-04-20T17:34:49.472514+0000","last_became_active":"2024-04-20T17:33:28.354030+0000","last_became_peered":"2024-04-20T17:33:28.354030+0000","last_unstale":"2024-04-20T17:34:49.472514+0000","last_undegraded":"2024-04-20T17:34:49.472514+0000","last_fullsized":"2024-04-20T17:34:49.472514+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"638'7110","last_scrub_stamp":"2024-04-20T17:34:29.610691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:34:29.610691+0000","objects_scrubbed":7037,"log_size":8796,"log_dups_size":0,"ondisk_log_size":8796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:35:52.155523+0000","scrub_duration":1493,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":576454656,"num_objects":8796,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8796,"num_write_kb":562944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"643'8625","reported_seq":8763,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-04-20T17:34:49.482181+0000","last_change":"2024-04-20T17:34:33.614867+0000","last_active":"2024-04-20T17:34:49.482181+0000","last_peered":"2024-04-20T17:34:49.482181+0000","last_clean":"2024-04-20T17:34:49.482181+0000","last_became_active":"2024-04-20T17:33:28.460508+0000","last_became_peered":"2024-04-20T17:33:28.460508+0000","last_unstale":"2024-04-20T17:34:49.482181+0000","last_undegraded":"2024-04-20T17:34:49.482181+0000","last_fullsized":"2024-04-20T17:34:49.482181+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"638'7310","last_scrub_stamp":"2024-04-20T17:34:33.614684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:34:33.614684+0000","objects_scrubbed":7259,"log_size":8625,"log_dups_size":0,"ondisk_log_size":8625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:35:45.625087+0000","scrub_duration":1518,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":565248000,"num_objects":8625,"num_object_clones":0,"num_object_copies":17250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8625,"num_write_kb":552000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9],"acting":[0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"643'8856","reported_seq":8913,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-04-20T17:34:52.539976+0000","last_change":"2024-04-20T17:33:28.354174+0000","last_active":"2024-04-20T17:34:52.539976+0000","last_peered":"2024-04-20T17:34:52.539976+0000","last_clean":"2024-04-20T17:34:52.539976+0000","last_became_active":"2024-04-20T17:33:28.353875+0000","last_became_peered":"2024-04-20T17:33:28.353875+0000","last_unstale":"2024-04-20T17:34:52.539976+0000","last_undegraded":"2024-04-20T17:34:52.539976+0000","last_fullsized":"2024-04-20T17:34:52.539976+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:33:27.339736+0000","objects_scrubbed":0,"log_size":8856,"log_dups_size":0,"ondisk_log_size":8856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:34:53.411300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":580386816,"num_objects":8856,"num_object_clones":0,"num_object_copies":17712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8856,"num_write_kb":566784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"643'9011","reported_seq":9142,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-04-20T17:34:53.446335+0000","last_change":"2024-04-20T17:34:34.489315+0000","last_active":"2024-04-20T17:34:53.446335+0000","last_peered":"2024-04-20T17:34:53.446335+0000","last_clean":"2024-04-20T17:34:53.446335+0000","last_became_active":"2024-04-20T17:33:28.356179+0000","last_became_peered":"2024-04-20T17:33:28.356179+0000","last_unstale":"2024-04-20T17:34:53.446335+0000","last_undegraded":"2024-04-20T17:34:53.446335+0000","last_fullsized":"2024-04-20T17:34:53.446335+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:33:27.339736+0000","objects_scrubbed":6849,"log_size":9011,"log_dups_size":0,"ondisk_log_size":9011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:34:54.757773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590544896,"num_objects":9011,"num_object_clones":0,"num_object_copies":18022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9011,"num_write_kb":576704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"643'9056","reported_seq":9181,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-04-20T17:34:52.535187+0000","last_change":"2024-04-20T17:34:34.484244+0000","last_active":"2024-04-20T17:34:52.535187+0000","last_peered":"2024-04-20T17:34:52.535187+0000","last_clean":"2024-04-20T17:34:52.535187+0000","last_became_active":"2024-04-20T17:33:28.460226+0000","last_became_peered":"2024-04-20T17:33:28.460226+0000","last_unstale":"2024-04-20T17:34:52.535187+0000","last_undegraded":"2024-04-20T17:34:52.535187+0000","last_fullsized":"2024-04-20T17:34:52.535187+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:33:27.339736+0000","objects_scrubbed":6499,"log_size":9056,"log_dups_size":0,"ondisk_log_size":9056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:34:53.407580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":593494016,"num_objects":9056,"num_object_clones":0,"num_object_copies":18112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9056,"num_write_kb":579584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"643'9009","reported_seq":9050,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-04-20T17:34:53.463120+0000","last_change":"2024-04-20T17:33:43.545455+0000","last_active":"2024-04-20T17:34:53.463120+0000","last_peered":"2024-04-20T17:34:53.463120+0000","last_clean":"2024-04-20T17:34:53.463120+0000","last_became_active":"2024-04-20T17:33:43.544269+0000","last_became_peered":"2024-04-20T17:33:43.544269+0000","last_unstale":"2024-04-20T17:34:53.463120+0000","last_undegraded":"2024-04-20T17:34:53.463120+0000","last_fullsized":"2024-04-20T17:34:53.463120+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:33:27.339736+0000","objects_scrubbed":0,"log_size":9009,"log_dups_size":0,"ondisk_log_size":9009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:34:54.795716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590413824,"num_objects":9009,"num_object_clones":0,"num_object_copies":18018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9009,"num_write_kb":576576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"643'8877","reported_seq":9013,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-04-20T17:34:52.503568+0000","last_change":"2024-04-20T17:34:31.723664+0000","last_active":"2024-04-20T17:34:52.503568+0000","last_peered":"2024-04-20T17:34:52.503568+0000","last_clean":"2024-04-20T17:34:52.503568+0000","last_became_active":"2024-04-20T17:33:28.353915+0000","last_became_peered":"2024-04-20T17:33:28.353915+0000","last_unstale":"2024-04-20T17:34:52.503568+0000","last_undegraded":"2024-04-20T17:34:52.503568+0000","last_fullsized":"2024-04-20T17:34:52.503568+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"638'7153","last_scrub_stamp":"2024-04-20T17:34:31.723512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:34:31.723512+0000","objects_scrubbed":7103,"log_size":8877,"log_dups_size":0,"ondisk_log_size":8877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:35:43.399646+0000","scrub_duration":1282,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":581763072,"num_objects":8877,"num_object_clones":0,"num_object_copies":17754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8877,"num_write_kb":568128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"643'8836","reported_seq":8893,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-04-20T17:34:50.480650+0000","last_change":"2024-04-20T17:33:28.355427+0000","last_active":"2024-04-20T17:34:50.480650+0000","last_peered":"2024-04-20T17:34:50.480650+0000","last_clean":"2024-04-20T17:34:50.480650+0000","last_became_active":"2024-04-20T17:33:28.355126+0000","last_became_peered":"2024-04-20T17:33:28.355126+0000","last_unstale":"2024-04-20T17:34:50.480650+0000","last_undegraded":"2024-04-20T17:34:50.480650+0000","last_fullsized":"2024-04-20T17:34:50.480650+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:33:27.339736+0000","objects_scrubbed":0,"log_size":8836,"log_dups_size":0,"ondisk_log_size":8836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":579076096,"num_objects":8836,"num_object_clones":0,"num_object_copies":17672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8836,"num_write_kb":565504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"644'9089","reported_seq":9148,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-04-20T17:34:53.775921+0000","last_change":"2024-04-20T17:33:28.460798+0000","last_active":"2024-04-20T17:34:53.775921+0000","last_peered":"2024-04-20T17:34:53.775921+0000","last_clean":"2024-04-20T17:34:53.775921+0000","last_became_active":"2024-04-20T17:33:28.460449+0000","last_became_peered":"2024-04-20T17:33:28.460449+0000","last_unstale":"2024-04-20T17:34:53.775921+0000","last_undegraded":"2024-04-20T17:34:53.775921+0000","last_fullsized":"2024-04-20T17:34:53.775921+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:33:27.339736+0000","objects_scrubbed":0,"log_size":9089,"log_dups_size":0,"ondisk_log_size":9089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:34:55.756034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":595656704,"num_objects":9089,"num_object_clones":0,"num_object_copies":18178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9089,"num_write_kb":581696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.0","version":"643'8607","reported_seq":8744,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-04-20T17:34:49.482259+0000","last_change":"2024-04-20T17:34:32.958315+0000","last_active":"2024-04-20T17:34:49.482259+0000","last_peered":"2024-04-20T17:34:49.482259+0000","last_clean":"2024-04-20T17:34:49.482259+0000","last_became_active":"2024-04-20T17:33:28.352816+0000","last_became_peered":"2024-04-20T17:33:28.352816+0000","last_unstale":"2024-04-20T17:34:49.482259+0000","last_undegraded":"2024-04-20T17:34:49.482259+0000","last_fullsized":"2024-04-20T17:34:49.482259+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"638'7310","last_scrub_stamp":"2024-04-20T17:34:32.958145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:34:32.958145+0000","objects_scrubbed":7224,"log_size":8607,"log_dups_size":0,"ondisk_log_size":8607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:35:36.957806+0000","scrub_duration":1812,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":564068352,"num_objects":8607,"num_object_clones":0,"num_object_copies":17214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8607,"num_write_kb":550848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"643'9009","reported_seq":9066,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-04-20T17:34:53.464609+0000","last_change":"2024-04-20T17:33:28.356015+0000","last_active":"2024-04-20T17:34:53.464609+0000","last_peered":"2024-04-20T17:34:53.464609+0000","last_clean":"2024-04-20T17:34:53.464609+0000","last_became_active":"2024-04-20T17:33:28.355779+0000","last_became_peered":"2024-04-20T17:33:28.355779+0000","last_unstale":"2024-04-20T17:34:53.464609+0000","last_undegraded":"2024-04-20T17:34:53.464609+0000","last_fullsized":"2024-04-20T17:34:53.464609+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:33:27.339736+0000","last_clean_scrub_stamp":"2024-04-20T17:33:27.339736+0000","objects_scrubbed":0,"log_size":9009,"log_dups_size":0,"ondisk_log_size":9009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:34:54.757814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590413824,"num_objects":9009,"num_object_clones":0,"num_object_copies":18018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9009,"num_write_kb":576576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":856,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-04-20T17:34:44.526959+0000","last_change":"2024-04-20T17:34:27.569542+0000","last_active":"2024-04-20T17:34:44.526959+0000","last_peered":"2024-04-20T17:34:44.526959+0000","last_clean":"2024-04-20T17:34:44.526959+0000","last_became_active":"2024-04-20T17:23:24.109202+0000","last_became_peered":"2024-04-20T17:23:24.109202+0000","last_unstale":"2024-04-20T17:34:44.526959+0000","last_undegraded":"2024-04-20T17:34:44.526959+0000","last_fullsized":"2024-04-20T17:34:44.526959+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:34:27.569433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:34:27.569433+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:35:53.198377+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":9319743488,"num_objects":142208,"num_object_clones":0,"num_object_copies":284416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":142208,"num_write_kb":9101312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18618843136,"data_stored":18618843136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142208,"ondisk_log_size":142208,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991583,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1729456,"kb_used_data":1142952,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":92015696,"statfs":{"total":95995035648,"available":94224072704,"internally_reserved":0,"allocated":1170382848,"data_stored":1170031558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":600570308},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253222,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1621688,"kb_used_data":1157356,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":92123464,"statfs":{"total":95995035648,"available":94334427136,"internally_reserved":0,"allocated":1185132544,"data_stored":1184778802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":475462080},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532612970,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2126376,"kb_used_data":1693984,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":91618776,"statfs":{"total":95995035648,"available":93817626624,"internally_reserved":0,"allocated":1734639616,"data_stored":1734291898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":442759616},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809201,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":852152,"kb_used_data":582188,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92893000,"statfs":{"total":95995035648,"available":95122432000,"internally_reserved":0,"allocated":596160512,"data_stored":595806790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":398,"seq":1709396983957,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2560688,"kb_used_data":2279464,"kb_used_omap":6,"kb_used_meta":281209,"kb_avail":91184464,"statfs":{"total":95995035648,"available":93372891136,"internally_reserved":0,"allocated":2334171136,"data_stored":2333819866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6767,"internal_metadata":287958417},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":0,"seq":2297807503431,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738774,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2144168,"kb_used_data":1657632,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":91600984,"statfs":{"total":95995035648,"available":93799407616,"internally_reserved":0,"allocated":1697415168,"data_stored":1697067450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394065,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1293224,"kb_used_data":1130976,"kb_used_omap":202,"kb_used_meta":162037,"kb_avail":92451928,"statfs":{"total":95995035648,"available":94670774272,"internally_reserved":0,"allocated":1158119424,"data_stored":1157771706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":165926359},"hb_peers":[0,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653115,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4132072,"kb_used_data":3995552,"kb_used_omap":4,"kb_used_meta":136507,"kb_avail":89613080,"statfs":{"total":95995035648,"available":91763793920,"internally_reserved":0,"allocated":4091445248,"data_stored":4091097530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":139783879},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986562,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2013608,"kb_used_data":1679776,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":91731544,"statfs":{"total":95995035648,"available":93933101056,"internally_reserved":0,"allocated":1720090624,"data_stored":1719742906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790333,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3328104,"kb_used_data":2801312,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":90417048,"statfs":{"total":95995035648,"available":92587057152,"internally_reserved":0,"allocated":2868543488,"data_stored":2868195770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1705771008,"data_stored":1705771008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1162805248,"data_stored":1162805248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4115333120,"data_stored":4115333120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1727463424,"data_stored":1727463424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2868117504,"data_stored":2868117504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2336817152,"data_stored":2336817152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":595656704,"data_stored":595656704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1746075648,"data_stored":1746075648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1186791424,"data_stored":1186791424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1174011904,"data_stored":1174011904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:34:57.778 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-20T17:34:57.778 INFO:tasks.thrashosds.thrasher:Waiting for trim 2024-04-20T17:34:57.786 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:34:57.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:57.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:34:57.893 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:34:57.922 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:57.931 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 79 15 149226 149211 118.037 79.375 0.00995419 0.00847041 2024-04-20T17:34:57.948 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:34:57.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:57.958 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:34:58.012 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:57 smithi005 ceph-mon[39562]: pgmap v1699: 17 pgs: 17 active+clean; 8.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-20T17:34:58.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:57 smithi106 ceph-mon[44423]: pgmap v1699: 17 pgs: 17 active+clean; 8.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-20T17:34:58.059 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:34:58.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:57 smithi171 ceph-mon[44189]: pgmap v1699: 17 pgs: 17 active+clean; 8.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-20T17:34:58.088 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:58.112 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:34:58.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:58.114 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:34:58.203 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:34:58.215 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:34:58.268 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:58.310 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:34:58.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:58.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:34:58.320 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:34:58.415 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:34:58.449 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:58.472 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:34:58.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:58.476 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:34:58.578 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:34:58.623 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:58.658 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:34:58.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:58.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:34:58.762 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:34:58.769 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:34:58.824 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:58 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/688620886' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:34:58.824 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:58 smithi106 ceph-mon[44423]: from='client.26839 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:34:58.834 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:58.932 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:34:58.932708 min lat: 0.00100551 max lat: 4.62787 avg lat: 0.00851242 2024-04-20T17:34:58.932 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:34:58.932 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 80 16 150368 150352 117.452 71.3125 0.00864132 0.00851242 2024-04-20T17:34:58.939 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:58 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/688620886' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:34:58.939 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:58 smithi005 ceph-mon[39562]: from='client.26839 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:34:59.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:58 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/688620886' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:34:59.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:58 smithi171 ceph-mon[44189]: from='client.26839 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:34:59.594 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:34:59.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:59.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:34:59.708 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:34:59.742 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:59.775 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:34:59.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:59.778 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:34:59.879 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:34:59.922 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:59 smithi106 ceph-mon[44423]: pgmap v1700: 17 pgs: 17 active+clean; 8.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-04-20T17:34:59.922 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:59 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:34:59.922 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:59 smithi106 ceph-mon[44423]: from='client.26839 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:34:59.922 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:59 smithi106 ceph-mon[44423]: osdmap e645: 12 total, 11 up, 10 in 2024-04-20T17:34:59.922 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:34:59 smithi106 ceph-mon[44423]: 6.1 scrub starts 2024-04-20T17:34:59.927 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:34:59.932 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 81 15 151615 151600 116.965 78 0.00985123 0.00854855 2024-04-20T17:34:59.962 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:34:59.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:34:59.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:35:00.069 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:35:00.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:59 smithi171 ceph-mon[44189]: pgmap v1700: 17 pgs: 17 active+clean; 8.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-04-20T17:35:00.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:59 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:35:00.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:59 smithi171 ceph-mon[44189]: from='client.26839 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:35:00.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:59 smithi171 ceph-mon[44189]: osdmap e645: 12 total, 11 up, 10 in 2024-04-20T17:35:00.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:34:59 smithi171 ceph-mon[44189]: 6.1 scrub starts 2024-04-20T17:35:00.106 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:00.132 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:35:00.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:00.136 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:35:00.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:59 smithi005 ceph-mon[39562]: pgmap v1700: 17 pgs: 17 active+clean; 8.9 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-04-20T17:35:00.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:59 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:35:00.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:59 smithi005 ceph-mon[39562]: from='client.26839 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:35:00.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:59 smithi005 ceph-mon[39562]: osdmap e645: 12 total, 11 up, 10 in 2024-04-20T17:35:00.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:34:59 smithi005 ceph-mon[39562]: 6.1 scrub starts 2024-04-20T17:35:00.237 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:35:00.275 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:00.309 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:35:00.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:00.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:35:00.340 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:35:00.341 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:35:00.422 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:35:00.451 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:00.472 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:35:00.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:00.478 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:35:00.580 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:35:00.608 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:35:00.642 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:00.645 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:35:00.678 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:35:00.721 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:35:00.766 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:35:00.801 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:35:00.836 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:35:00.908 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:35:00.908 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:35:00.932 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 82 16 152757 152741 116.408 71.3125 0.0123514 0.00858841 2024-04-20T17:35:00.938 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:35:00.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:00 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2115937882' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:35:00.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:00.988 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:35:00.989 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:35:00.989 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-20T17:35:00.989 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:35:00.989 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:35:00.989 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:35:00.989 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:35:00.989 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:35:00.989 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:35:00.989 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:35:00.989 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:35:00.989 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:35:00.989 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:35:01.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:00 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2115937882' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:35:01.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:01.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:00 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2115937882' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:35:01.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:01.111 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-20T17:35:01.418 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:35:01.933 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 83 16 153713 153697 115.726 59.75 0.00533831 0.00863843 2024-04-20T17:35:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:01 smithi171 ceph-mon[44189]: 6.1 scrub ok 2024-04-20T17:35:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:01 smithi171 ceph-mon[44189]: pgmap v1702: 17 pgs: 17 active+clean; 9.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-04-20T17:35:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:01 smithi171 ceph-mon[44189]: 6.6 scrub starts 2024-04-20T17:35:02.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:01 smithi171 ceph-mon[44189]: 6.f scrub starts 2024-04-20T17:35:02.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:01 smithi005 ceph-mon[39562]: 6.1 scrub ok 2024-04-20T17:35:02.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:01 smithi005 ceph-mon[39562]: pgmap v1702: 17 pgs: 17 active+clean; 9.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-04-20T17:35:02.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:01 smithi005 ceph-mon[39562]: 6.6 scrub starts 2024-04-20T17:35:02.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:01 smithi005 ceph-mon[39562]: 6.f scrub starts 2024-04-20T17:35:02.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:01 smithi106 ceph-mon[44423]: 6.1 scrub ok 2024-04-20T17:35:02.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:01 smithi106 ceph-mon[44423]: pgmap v1702: 17 pgs: 17 active+clean; 9.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-04-20T17:35:02.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:01 smithi106 ceph-mon[44423]: 6.6 scrub starts 2024-04-20T17:35:02.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:01 smithi106 ceph-mon[44423]: 6.f scrub starts 2024-04-20T17:35:02.932 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 84 16 154694 154678 115.078 61.3125 0.00664907 0.00868832 2024-04-20T17:35:02.960 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:02.960 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:02 smithi005 ceph-mon[39562]: 6.7 scrub starts 2024-04-20T17:35:02.969 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:35:02.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:02.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:35:02.995 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:35:02.996 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:35:03.072 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:03.073 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:02 smithi171 ceph-mon[44189]: 6.7 scrub starts 2024-04-20T17:35:03.078 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:35:03.117 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:03.147 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:35:03.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:03.153 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:35:03.253 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:35:03.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:03.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:02 smithi106 ceph-mon[44423]: 6.7 scrub starts 2024-04-20T17:35:03.283 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:03.308 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:35:03.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:03.314 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:35:03.415 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:35:03.458 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:03.483 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:35:03.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:03.485 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:35:03.586 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:35:03.645 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:03.680 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:35:03.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:03.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:35:03.733 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:35:03.733 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:35:03.733 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 0 2024-04-20T17:35:03.733 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:35:03.785 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:35:03.840 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:03.883 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:35:03.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:03.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:35:03.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:03 smithi005 ceph-mon[39562]: 6.e scrub starts 2024-04-20T17:35:03.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:03 smithi005 ceph-mon[39562]: pgmap v1703: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.2 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-04-20T17:35:03.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:03 smithi005 ceph-mon[39562]: 6.f scrub ok 2024-04-20T17:35:03.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:03 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:35:03.895 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:03 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:35:03.895 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:03 smithi005 ceph-mon[39562]: 6.6 scrub ok 2024-04-20T17:35:03.932 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 85 15 155753 155738 114.503 66.25 0.0153165 0.00873204 2024-04-20T17:35:03.995 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:35:04.040 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:03 smithi106 ceph-mon[44423]: 6.e scrub starts 2024-04-20T17:35:04.040 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:03 smithi106 ceph-mon[44423]: pgmap v1703: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.2 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-04-20T17:35:04.040 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:03 smithi106 ceph-mon[44423]: 6.f scrub ok 2024-04-20T17:35:04.040 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:03 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:35:04.040 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:03 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:35:04.040 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:03 smithi106 ceph-mon[44423]: 6.6 scrub ok 2024-04-20T17:35:04.046 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:04.080 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:35:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:03 smithi171 ceph-mon[44189]: 6.e scrub starts 2024-04-20T17:35:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:03 smithi171 ceph-mon[44189]: pgmap v1703: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.2 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-04-20T17:35:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:03 smithi171 ceph-mon[44189]: 6.f scrub ok 2024-04-20T17:35:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:03 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:35:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:03 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:35:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:03 smithi171 ceph-mon[44189]: 6.6 scrub ok 2024-04-20T17:35:04.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:04.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:35:04.188 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:35:04.216 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:04.243 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:35:04.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:04.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:35:04.290 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:35:04.352 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:35:04.394 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:04.423 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:35:04.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:04.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:35:04.456 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-20T17:35:04.531 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:35:04.589 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:04.620 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:35:04.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:04.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:35:04.725 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:35:04.766 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:35:04.791 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:04.796 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:04 smithi171 ceph-mon[44189]: osdmap e646: 12 total, 11 up, 10 in 2024-04-20T17:35:04.796 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:04 smithi171 ceph-mon[44189]: 6.7 scrub ok 2024-04-20T17:35:04.797 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:04 smithi171 ceph-mon[44189]: 6.2 scrub starts 2024-04-20T17:35:04.797 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:04 smithi171 ceph-mon[44189]: 6.d scrub starts 2024-04-20T17:35:04.797 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:04 smithi171 ceph-mon[44189]: pgmap v1705: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-20T17:35:04.797 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:04 smithi171 ceph-mon[44189]: 6.e scrub ok 2024-04-20T17:35:04.799 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:35:04.820 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:35:04.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:04.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:35:04.930 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:35:04.933 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 86 15 156739 156724 113.888 61.625 0.0150036 0.00877956 2024-04-20T17:35:04.967 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:04.996 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:35:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:04 smithi106 ceph-mon[44423]: osdmap e646: 12 total, 11 up, 10 in 2024-04-20T17:35:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:04 smithi106 ceph-mon[44423]: 6.7 scrub ok 2024-04-20T17:35:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:04 smithi106 ceph-mon[44423]: 6.2 scrub starts 2024-04-20T17:35:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:04 smithi106 ceph-mon[44423]: 6.d scrub starts 2024-04-20T17:35:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:04 smithi106 ceph-mon[44423]: pgmap v1705: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-20T17:35:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:04 smithi106 ceph-mon[44423]: 6.e scrub ok 2024-04-20T17:35:05.040 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:04 smithi005 ceph-mon[39562]: osdmap e646: 12 total, 11 up, 10 in 2024-04-20T17:35:05.040 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:04 smithi005 ceph-mon[39562]: 6.7 scrub ok 2024-04-20T17:35:05.040 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:04 smithi005 ceph-mon[39562]: 6.2 scrub starts 2024-04-20T17:35:05.040 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:04 smithi005 ceph-mon[39562]: 6.d scrub starts 2024-04-20T17:35:05.040 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:04 smithi005 ceph-mon[39562]: pgmap v1705: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 9.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-20T17:35:05.040 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:04 smithi005 ceph-mon[39562]: 6.e scrub ok 2024-04-20T17:35:05.045 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:35:05.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:05.051 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:35:05.151 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:35:05.192 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:05.219 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:35:05.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:05.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:35:05.324 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:35:05.364 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:05.404 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:35:05.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:05.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:35:05.508 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:35:05.536 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:05.555 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:35:05.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:05.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:35:05.658 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:35:05.713 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:05.762 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:35:05.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:05.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:35:05.816 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:05 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:35:05.816 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:05 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:35:05.873 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:35:05.931 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:05.932 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 87 16 157691 157675 113.263 59.4375 0.0310616 0.00882686 2024-04-20T17:35:05.969 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:05 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:35:05.969 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:05 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:35:05.974 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:35:05.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:05.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:35:06.079 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:35:06.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:05 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:35:06.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:05 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:35:06.111 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:06.139 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:35:06.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:06.143 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:35:06.244 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:35:06.276 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:06.483 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:35:06.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:06.489 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:35:06.590 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:35:06.623 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:06.742 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:35:06.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:06.745 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:35:06.787 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:35:06.787 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:35:06.787 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-04-20T17:35:06.787 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-04-20T17:35:06.787 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:35:06.846 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:35:06.898 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:06.930 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:35:06.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:06.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:35:06.933 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 88 16 158888 158872 112.824 74.8125 0.0116686 0.00886192 2024-04-20T17:35:06.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:06 smithi005 ceph-mon[39562]: 6.d scrub ok 2024-04-20T17:35:06.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:06 smithi005 ceph-mon[39562]: osdmap e647: 12 total, 11 up, 10 in 2024-04-20T17:35:06.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:06 smithi005 ceph-mon[39562]: 6.2 scrub ok 2024-04-20T17:35:06.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:06 smithi005 ceph-mon[39562]: pgmap v1707: 17 pgs: 4 active+clean+scrubbing, 13 active+clean; 9.5 GiB data, 23 GiB used, 872 GiB / 894 GiB avail; 65 MiB/s wr, 1.05k op/s 2024-04-20T17:35:07.033 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:35:07.040 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:35:07.063 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:06 smithi171 ceph-mon[44189]: 6.d scrub ok 2024-04-20T17:35:07.063 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:06 smithi171 ceph-mon[44189]: osdmap e647: 12 total, 11 up, 10 in 2024-04-20T17:35:07.063 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:06 smithi171 ceph-mon[44189]: 6.2 scrub ok 2024-04-20T17:35:07.064 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:06 smithi171 ceph-mon[44189]: pgmap v1707: 17 pgs: 4 active+clean+scrubbing, 13 active+clean; 9.5 GiB data, 23 GiB used, 872 GiB / 894 GiB avail; 65 MiB/s wr, 1.05k op/s 2024-04-20T17:35:07.068 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:06 smithi106 ceph-mon[44423]: 6.d scrub ok 2024-04-20T17:35:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:06 smithi106 ceph-mon[44423]: osdmap e647: 12 total, 11 up, 10 in 2024-04-20T17:35:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:06 smithi106 ceph-mon[44423]: 6.2 scrub ok 2024-04-20T17:35:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:06 smithi106 ceph-mon[44423]: pgmap v1707: 17 pgs: 4 active+clean+scrubbing, 13 active+clean; 9.5 GiB data, 23 GiB used, 872 GiB / 894 GiB avail; 65 MiB/s wr, 1.05k op/s 2024-04-20T17:35:07.827 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:07 smithi005 ceph-mon[39562]: 6.5 scrub starts 2024-04-20T17:35:07.827 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:07 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:35:07.828 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:07 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:35:07.933 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 89 16 160041 160025 112.366 72.0625 0.0253647 0.00889807 2024-04-20T17:35:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:07 smithi171 ceph-mon[44189]: 6.5 scrub starts 2024-04-20T17:35:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:07 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:35:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:07 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:35:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:07 smithi106 ceph-mon[44423]: 6.5 scrub starts 2024-04-20T17:35:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:07 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:35:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:07 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:35:08.858 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:35:08.933 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 90 14 161095 161081 111.851 66 0.022447 0.00893826 2024-04-20T17:35:09.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:08 smithi005 ceph-mon[39562]: osdmap e648: 12 total, 11 up, 10 in 2024-04-20T17:35:09.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:08 smithi005 ceph-mon[39562]: 6.b scrub starts 2024-04-20T17:35:09.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:08 smithi005 ceph-mon[39562]: pgmap v1709: 17 pgs: 4 active+clean+scrubbing, 13 active+clean; 9.6 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-04-20T17:35:09.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:08 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/375274125' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:35:09.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:08 smithi005 ceph-mon[39562]: from='client.26954 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:35:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:08 smithi106 ceph-mon[44423]: osdmap e648: 12 total, 11 up, 10 in 2024-04-20T17:35:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:08 smithi106 ceph-mon[44423]: 6.b scrub starts 2024-04-20T17:35:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:08 smithi106 ceph-mon[44423]: pgmap v1709: 17 pgs: 4 active+clean+scrubbing, 13 active+clean; 9.6 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-04-20T17:35:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:08 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/375274125' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:35:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:08 smithi106 ceph-mon[44423]: from='client.26954 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:35:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:08 smithi171 ceph-mon[44189]: osdmap e648: 12 total, 11 up, 10 in 2024-04-20T17:35:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:08 smithi171 ceph-mon[44189]: 6.b scrub starts 2024-04-20T17:35:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:08 smithi171 ceph-mon[44189]: pgmap v1709: 17 pgs: 4 active+clean+scrubbing, 13 active+clean; 9.6 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-04-20T17:35:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:08 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/375274125' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:35:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:08 smithi171 ceph-mon[44189]: from='client.26954 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:35:09.586 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:35:09.609 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:35:09.630 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:35:09.652 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:35:09.678 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:35:09.699 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:35:09.723 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:35:09.743 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:35:09.765 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:35:09.793 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:35:09.807 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:35:09.807 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:35:09.807 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-20T17:35:09.807 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:35:09.808 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:35:09.808 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:35:09.808 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:35:09.808 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:35:09.808 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:35:09.808 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:35:09.808 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:35:09.808 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:35:09.808 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:35:09.826 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-20T17:35:09.882 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:35:09.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:09.885 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:35:09.934 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Total time run: 90.0123 2024-04-20T17:35:09.934 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Total writes made: 161095 2024-04-20T17:35:09.934 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Write size: 65536 2024-04-20T17:35:09.934 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Object size: 65536 2024-04-20T17:35:09.934 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Bandwidth (MB/sec): 111.856 2024-04-20T17:35:09.934 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Stddev Bandwidth: 61.0856 2024-04-20T17:35:09.934 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Max bandwidth (MB/sec): 328.188 2024-04-20T17:35:09.934 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Min bandwidth (MB/sec): 0 2024-04-20T17:35:09.934 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Average IOPS: 1789 2024-04-20T17:35:09.934 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Stddev IOPS: 977.373 2024-04-20T17:35:09.934 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Max IOPS: 5251 2024-04-20T17:35:09.934 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Min IOPS: 0 2024-04-20T17:35:09.935 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Average Latency(s): 0.00893937 2024-04-20T17:35:09.935 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Stddev Latency(s): 0.0464867 2024-04-20T17:35:09.935 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Max latency(s): 4.62787 2024-04-20T17:35:09.935 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Min latency(s): 0.00100551 2024-04-20T17:35:09.940 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Cleaning up (deleting benchmark objects) 2024-04-20T17:35:09.986 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:35:10.018 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:10.048 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:35:10.053 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:09 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:35:10.053 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:09 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:35:10.053 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:09 smithi171 ceph-mon[44189]: from='client.26954 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:35:10.053 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:09 smithi171 ceph-mon[44189]: osdmap e649: 12 total, 11 up, 10 in 2024-04-20T17:35:10.054 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:09 smithi171 ceph-mon[44189]: 6.5 scrub ok 2024-04-20T17:35:10.054 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:09 smithi171 ceph-mon[44189]: 6.c scrub starts 2024-04-20T17:35:10.054 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:09 smithi171 ceph-mon[44189]: 6.b scrub ok 2024-04-20T17:35:10.054 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:09 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/479270240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:35:10.054 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:10.054 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:10.054 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:10.054 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:10.054 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:10.054 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:10.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:10.055 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:35:10.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:09 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:35:10.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:09 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:35:10.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:09 smithi005 ceph-mon[39562]: from='client.26954 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:35:10.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:09 smithi005 ceph-mon[39562]: osdmap e649: 12 total, 11 up, 10 in 2024-04-20T17:35:10.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:09 smithi005 ceph-mon[39562]: 6.5 scrub ok 2024-04-20T17:35:10.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:09 smithi005 ceph-mon[39562]: 6.c scrub starts 2024-04-20T17:35:10.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:09 smithi005 ceph-mon[39562]: 6.b scrub ok 2024-04-20T17:35:10.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:09 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/479270240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:35:10.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:10.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:10.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:10.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:10.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:10.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:10.156 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:35:10.203 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:09 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:35:10.203 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:09 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:35:10.203 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:09 smithi106 ceph-mon[44423]: from='client.26954 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:35:10.203 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:09 smithi106 ceph-mon[44423]: osdmap e649: 12 total, 11 up, 10 in 2024-04-20T17:35:10.203 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:09 smithi106 ceph-mon[44423]: 6.5 scrub ok 2024-04-20T17:35:10.203 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:09 smithi106 ceph-mon[44423]: 6.c scrub starts 2024-04-20T17:35:10.203 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:09 smithi106 ceph-mon[44423]: 6.b scrub ok 2024-04-20T17:35:10.204 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:09 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/479270240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:35:10.204 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:10.204 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:10.204 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:10.204 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:10.204 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:10.204 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:10.206 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:10.224 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:35:10.236 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:35:10.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:10.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:35:10.342 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:35:10.381 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:10.408 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:35:10.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:10.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:35:10.516 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:35:10.586 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:10.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:10.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:10 smithi005 ceph-mon[39562]: 6.c scrub ok 2024-04-20T17:35:10.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:10.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:10.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:10 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:35:10.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:10 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:35:10.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:10.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:10.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:10 smithi005 ceph-mon[39562]: osdmap e650: 12 total, 11 up, 10 in 2024-04-20T17:35:10.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:10 smithi005 ceph-mon[39562]: pgmap v1712: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 9.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 61 MiB/s wr, 982 op/s 2024-04-20T17:35:11.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:11.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:10 smithi171 ceph-mon[44189]: 6.c scrub ok 2024-04-20T17:35:11.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:11.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:11.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:10 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:35:11.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:10 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:35:11.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:11.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:11.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:10 smithi171 ceph-mon[44189]: osdmap e650: 12 total, 11 up, 10 in 2024-04-20T17:35:11.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:10 smithi171 ceph-mon[44189]: pgmap v1712: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 9.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 61 MiB/s wr, 982 op/s 2024-04-20T17:35:11.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:11.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:10 smithi106 ceph-mon[44423]: 6.c scrub ok 2024-04-20T17:35:11.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:11.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:11.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:10 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:35:11.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:10 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:35:11.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:11.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:11.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:10 smithi106 ceph-mon[44423]: osdmap e650: 12 total, 11 up, 10 in 2024-04-20T17:35:11.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:10 smithi106 ceph-mon[44423]: pgmap v1712: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 9.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 61 MiB/s wr, 982 op/s 2024-04-20T17:35:12.055 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:35:12.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:12.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:35:12.163 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:35:12.206 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:12.436 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:12 smithi171 ceph-mon[44189]: 6.8 scrub starts 2024-04-20T17:35:12.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:12 smithi005 ceph-mon[39562]: 6.8 scrub starts 2024-04-20T17:35:12.491 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:35:12.491 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:35:12.589 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:35:12.595 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:35:12.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:12.596 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:35:12.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:35:12.697 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:35:12.732 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:12 smithi106 ceph-mon[44423]: 6.8 scrub starts 2024-04-20T17:35:12.734 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:12.761 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:35:12.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:12.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:35:12.864 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:35:12.902 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:13.225 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-20T17:35:13.256 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:35:13.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:13.258 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:35:13.359 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:35:13.405 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:13.442 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:35:13.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:13.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:35:13.548 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:35:13.580 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:35:13.607 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:13.634 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:35:13.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:13.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:35:13.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:13 smithi005 ceph-mon[39562]: 6.4 scrub starts 2024-04-20T17:35:13.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:13 smithi005 ceph-mon[39562]: pgmap v1713: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 9.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 54 MiB/s wr, 893 op/s 2024-04-20T17:35:13.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:13 smithi005 ceph-mon[39562]: 6.8 scrub ok 2024-04-20T17:35:13.739 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:35:13.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:13 smithi106 ceph-mon[44423]: 6.4 scrub starts 2024-04-20T17:35:13.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:13 smithi106 ceph-mon[44423]: pgmap v1713: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 9.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 54 MiB/s wr, 893 op/s 2024-04-20T17:35:13.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:13 smithi106 ceph-mon[44423]: 6.8 scrub ok 2024-04-20T17:35:13.783 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:13.826 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:35:13.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:13.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:35:13.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:13 smithi171 ceph-mon[44189]: 6.4 scrub starts 2024-04-20T17:35:13.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:13 smithi171 ceph-mon[44189]: pgmap v1713: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 9.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 54 MiB/s wr, 893 op/s 2024-04-20T17:35:13.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:13 smithi171 ceph-mon[44189]: 6.8 scrub ok 2024-04-20T17:35:13.930 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:35:13.961 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:14.067 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:35:14.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:14.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:35:14.083 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:35:14.172 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:35:14.218 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:14.259 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:35:14.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:14.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:35:14.364 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:35:14.413 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:14.444 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:35:14.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:14.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:35:14.549 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:35:14.569 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:35:14.597 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:35:14.602 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:14.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:14 smithi005 ceph-mon[39562]: pgmap v1714: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.7 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 155 B/s rd, 41 MiB/s wr, 1.06k op/s 2024-04-20T17:35:14.721 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:35:14.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:14 smithi106 ceph-mon[44423]: pgmap v1714: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.7 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 155 B/s rd, 41 MiB/s wr, 1.06k op/s 2024-04-20T17:35:14.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:14.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:35:14.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:14 smithi171 ceph-mon[44189]: pgmap v1714: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.7 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 155 B/s rd, 41 MiB/s wr, 1.06k op/s 2024-04-20T17:35:14.897 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:35:14.958 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:15.456 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:15 smithi171 ceph-mon[44189]: 6.4 scrub ok 2024-04-20T17:35:15.483 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:35:15.483 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:35:15.627 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:35:15.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:15.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:35:15.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:15 smithi005 ceph-mon[39562]: 6.4 scrub ok 2024-04-20T17:35:15.734 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:35:15.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:15 smithi106 ceph-mon[44423]: 6.4 scrub ok 2024-04-20T17:35:15.775 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:15.858 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:35:15.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:15.863 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:35:15.964 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:35:15.996 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:16.039 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:35:16.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:16.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:35:16.144 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:35:16.197 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:16.208 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-20T17:35:16.221 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:35:16.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:16.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:35:16.325 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:35:16.361 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:16.388 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:35:16.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:16.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:35:16.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:16 smithi005 ceph-mon[39562]: pgmap v1715: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.6 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 127 B/s rd, 34 MiB/s wr, 1.09k op/s 2024-04-20T17:35:16.492 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:35:16.527 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:16.549 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:35:16.552 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:35:16.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:16.559 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:35:16.660 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:35:16.710 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:16 smithi106 ceph-mon[44423]: pgmap v1715: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.6 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 127 B/s rd, 34 MiB/s wr, 1.09k op/s 2024-04-20T17:35:16.717 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:16.749 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:35:16.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:16.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:35:16.793 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:35:16.816 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:16 smithi171 ceph-mon[44189]: pgmap v1715: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.6 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 127 B/s rd, 34 MiB/s wr, 1.09k op/s 2024-04-20T17:35:16.852 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:35:16.883 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:16.911 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:35:16.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:16.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:35:17.017 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:35:17.049 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:17.073 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:35:17.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:17.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:35:17.178 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:35:17.222 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:17.248 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:35:17.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:17.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:35:17.355 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:35:17.401 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:17.433 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:35:17.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:17.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:35:17.536 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:35:17.593 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:17.662 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:35:17.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:17.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:35:17.775 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:35:17.812 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:17.837 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:35:17.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:17.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:35:17.911 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:17 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/532621486' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:35:17.911 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:17 smithi005 ceph-mon[39562]: from='client.26963 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:35:17.925 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:35:17.941 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:35:17.974 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:18.005 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:35:18.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:18.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:35:18.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:17 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/532621486' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:35:18.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:17 smithi171 ceph-mon[44189]: from='client.26963 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:35:18.113 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:35:18.153 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:17 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/532621486' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:35:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:17 smithi106 ceph-mon[44423]: from='client.26963 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:35:18.261 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:35:18.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:18.269 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:35:18.369 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:35:18.403 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:18.427 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:35:18.427 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:35:18.434 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:35:18.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:18.438 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:35:18.539 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:35:18.576 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:18.606 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:35:18.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:18.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:35:18.710 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:35:18.761 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:18.912 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:18 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:35:18.913 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:18 smithi106 ceph-mon[44423]: from='client.26963 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:35:18.913 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:18 smithi106 ceph-mon[44423]: osdmap e651: 12 total, 11 up, 10 in 2024-04-20T17:35:18.913 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:18 smithi106 ceph-mon[44423]: pgmap v1717: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 119 B/s rd, 7.8 MiB/s wr, 825 op/s 2024-04-20T17:35:18.913 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:18 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:35:18.913 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:18 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:35:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:18 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:35:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:18 smithi171 ceph-mon[44189]: from='client.26963 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:35:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:18 smithi171 ceph-mon[44189]: osdmap e651: 12 total, 11 up, 10 in 2024-04-20T17:35:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:18 smithi171 ceph-mon[44189]: pgmap v1717: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 119 B/s rd, 7.8 MiB/s wr, 825 op/s 2024-04-20T17:35:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:18 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:35:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:18 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:35:19.139 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-20T17:35:19.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:18 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:35:19.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:18 smithi005 ceph-mon[39562]: from='client.26963 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:35:19.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:18 smithi005 ceph-mon[39562]: osdmap e651: 12 total, 11 up, 10 in 2024-04-20T17:35:19.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:18 smithi005 ceph-mon[39562]: pgmap v1717: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 9.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 119 B/s rd, 7.8 MiB/s wr, 825 op/s 2024-04-20T17:35:19.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:18 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:35:19.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:18 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:35:19.476 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:35:19.613 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:35:19.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:19.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:35:19.729 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:35:19.763 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:19.789 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:35:19.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:19.794 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:35:19.895 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:35:19.925 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:19 smithi005 ceph-mon[39562]: osdmap e652: 12 total, 11 up, 10 in 2024-04-20T17:35:19.932 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:20.025 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:19 smithi171 ceph-mon[44189]: osdmap e652: 12 total, 11 up, 10 in 2024-04-20T17:35:20.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:19 smithi106 ceph-mon[44423]: osdmap e652: 12 total, 11 up, 10 in 2024-04-20T17:35:20.293 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:35:20.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:20.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:35:20.397 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:35:20.450 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:20.481 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:35:20.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:20.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:35:20.587 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:35:20.618 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:20.758 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:35:20.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:20.764 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:35:20.864 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:35:20.898 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:20.934 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:35:20.967 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:35:21.002 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:35:21.036 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:35:21.064 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:35:21.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:20 smithi171 ceph-mon[44189]: pgmap v1719: 17 pgs: 17 active+clean; 9.2 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 127 B/s rd, 3.3 MiB/s wr, 1.23k op/s 2024-04-20T17:35:21.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:20 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4096596092' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:35:21.097 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:35:21.103 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:35:21.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:21.110 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:35:21.131 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:35:21.164 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:35:21.186 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:35:21.211 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:35:21.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:20 smithi005 ceph-mon[39562]: pgmap v1719: 17 pgs: 17 active+clean; 9.2 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 127 B/s rd, 3.3 MiB/s wr, 1.23k op/s 2024-04-20T17:35:21.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:20 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4096596092' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:35:21.222 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:35:21.242 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:21.260 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:35:21.260 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:35:21.260 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-20T17:35:21.260 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:35:21.260 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:35:21.260 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:35:21.260 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:35:21.261 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:35:21.261 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:35:21.261 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:35:21.261 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:35:21.261 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:35:21.261 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:35:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:20 smithi106 ceph-mon[44423]: pgmap v1719: 17 pgs: 17 active+clean; 9.2 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 127 B/s rd, 3.3 MiB/s wr, 1.23k op/s 2024-04-20T17:35:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:20 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4096596092' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:35:21.358 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:35:21.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:21.364 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:35:21.409 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:35:21.409 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:35:21.409 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:35:21.409 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:35:21.410 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.410 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:159061 4.2 4:4cdccfcc:::benchmark_data_smithi171_122772_object159060:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e482)", 2024-04-20T17:35:21.410 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:08.824368+0000", 2024-04-20T17:35:21.410 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 552.582509867, 2024-04-20T17:35:21.410 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15598972799999999, 2024-04-20T17:35:21.410 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:21.410 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:21.410 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:21.410 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:21.410 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:21.410 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:21.410 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 159061 2024-04-20T17:35:21.410 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.411 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:21.411 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.411 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:21.411 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824368+0000", 2024-04-20T17:35:21.411 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.411 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.411 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.411 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:21.411 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824368+0000", 2024-04-20T17:35:21.411 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.411 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.411 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.411 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:21.411 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824371+0000", 2024-04-20T17:35:21.411 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.424e-06 2024-04-20T17:35:21.412 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.412 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.412 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:21.412 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824398+0000", 2024-04-20T17:35:21.412 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7310000000000001e-05 2024-04-20T17:35:21.412 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.412 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.412 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:21.412 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824408+0000", 2024-04-20T17:35:21.412 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0295999999999999e-05 2024-04-20T17:35:21.412 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.412 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.412 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:21.412 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824414+0000", 2024-04-20T17:35:21.412 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9989999999999997e-06 2024-04-20T17:35:21.413 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.413 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.413 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:21.413 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.827811+0000", 2024-04-20T17:35:21.413 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0033968319999999998 2024-04-20T17:35:21.413 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.413 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.413 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:21.413 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.827940+0000", 2024-04-20T17:35:21.413 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012889600000000001 2024-04-20T17:35:21.413 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.413 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.413 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:35:21.413 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.828088+0000", 2024-04-20T17:35:21.414 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014852100000000001 2024-04-20T17:35:21.414 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.414 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.414 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:21.414 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.836604+0000", 2024-04-20T17:35:21.414 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0085154889999999994 2024-04-20T17:35:21.414 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.414 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.414 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:21.414 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980241+0000", 2024-04-20T17:35:21.414 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.14363725299999999 2024-04-20T17:35:21.414 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.414 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.414 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:21.415 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980298+0000", 2024-04-20T17:35:21.415 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6462999999999998e-05 2024-04-20T17:35:21.415 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.415 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.415 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:21.415 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980358+0000", 2024-04-20T17:35:21.415 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0244999999999997e-05 2024-04-20T17:35:21.415 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.415 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:21.415 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.415 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.415 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.415 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:159066 4.2 4:41a9bca0:::benchmark_data_smithi171_122772_object159065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e482)", 2024-04-20T17:35:21.415 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:08.824564+0000", 2024-04-20T17:35:21.416 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 552.58231409799998, 2024-04-20T17:35:21.416 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.155931294, 2024-04-20T17:35:21.416 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:21.416 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:21.416 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:21.416 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:21.416 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:21.416 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:21.416 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 159066 2024-04-20T17:35:21.416 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.416 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:21.416 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.416 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:21.416 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824564+0000", 2024-04-20T17:35:21.417 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.417 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.417 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.417 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:21.417 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824564+0000", 2024-04-20T17:35:21.417 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.417 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.417 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.417 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:21.417 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824565+0000", 2024-04-20T17:35:21.417 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1510000000000001e-06 2024-04-20T17:35:21.417 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.417 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.417 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:21.417 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824583+0000", 2024-04-20T17:35:21.418 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7839e-05 2024-04-20T17:35:21.418 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.418 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.418 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:21.418 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824592+0000", 2024-04-20T17:35:21.418 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9290000000000004e-06 2024-04-20T17:35:21.418 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.418 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.418 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:21.418 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.824596+0000", 2024-04-20T17:35:21.418 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3590000000000001e-06 2024-04-20T17:35:21.418 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.418 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.418 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:21.418 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.832585+0000", 2024-04-20T17:35:21.419 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0079889399999999999 2024-04-20T17:35:21.419 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.419 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.419 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:21.419 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.832687+0000", 2024-04-20T17:35:21.419 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000101487 2024-04-20T17:35:21.419 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.419 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.419 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:35:21.419 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.832792+0000", 2024-04-20T17:35:21.419 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010508100000000001 2024-04-20T17:35:21.419 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.419 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.419 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:21.419 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980174+0000", 2024-04-20T17:35:21.420 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.14738206500000001 2024-04-20T17:35:21.420 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.420 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.420 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:21.420 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980405+0000", 2024-04-20T17:35:21.420 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00023092499999999999 2024-04-20T17:35:21.420 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.420 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.420 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:21.420 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980449+0000", 2024-04-20T17:35:21.420 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4351000000000003e-05 2024-04-20T17:35:21.420 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.420 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.420 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:21.421 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980495+0000", 2024-04-20T17:35:21.421 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6167e-05 2024-04-20T17:35:21.421 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.421 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:21.421 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.421 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.421 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.421 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop_reply(client.26074.0:159061 4.2 e482/445 ondisk, result = 0)", 2024-04-20T17:35:21.421 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:08.839536+0000", 2024-04-20T17:35:21.421 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 552.56734221099998, 2024-04-20T17:35:21.421 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.14083410099999999, 2024-04-20T17:35:21.421 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:21.421 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:21.421 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:35:21.421 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:21.422 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.422 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:21.422 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.839536+0000", 2024-04-20T17:35:21.422 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.422 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.422 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.422 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:21.422 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.839536+0000", 2024-04-20T17:35:21.422 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.422 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.422 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.422 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:21.422 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.839538+0000", 2024-04-20T17:35:21.422 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.951e-06 2024-04-20T17:35:21.423 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.423 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.423 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:21.423 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.839545+0000", 2024-04-20T17:35:21.423 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.114e-06 2024-04-20T17:35:21.423 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.423 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.423 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:21.423 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.887991+0000", 2024-04-20T17:35:21.423 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.048445941999999999 2024-04-20T17:35:21.423 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.423 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.423 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:21.424 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.888002+0000", 2024-04-20T17:35:21.424 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1425e-05 2024-04-20T17:35:21.424 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.424 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.424 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:21.424 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980224+0000", 2024-04-20T17:35:21.424 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.092222193999999993 2024-04-20T17:35:21.424 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.424 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.424 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:21.424 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980234+0000", 2024-04-20T17:35:21.424 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1099999999999992e-06 2024-04-20T17:35:21.424 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.424 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.425 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:21.425 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:08.980370+0000", 2024-04-20T17:35:21.425 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000136365 2024-04-20T17:35:21.425 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.425 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:21.425 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.425 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.425 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.425 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:159675 4.2 4:45a9239e:::benchmark_data_smithi171_122772_object159674:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e482)", 2024-04-20T17:35:21.425 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:09.405527+0000", 2024-04-20T17:35:21.425 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 552.00135113700003, 2024-04-20T17:35:21.425 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.162906468, 2024-04-20T17:35:21.425 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:21.425 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:21.425 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:21.426 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:21.426 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:21.426 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:21.426 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 159675 2024-04-20T17:35:21.426 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.426 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:21.426 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.426 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:21.426 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.405527+0000", 2024-04-20T17:35:21.426 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.426 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.426 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.426 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:21.427 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.405527+0000", 2024-04-20T17:35:21.427 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.427 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.427 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.427 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:21.427 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.405528+0000", 2024-04-20T17:35:21.427 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.4e-07 2024-04-20T17:35:21.427 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.427 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.427 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:21.427 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.405546+0000", 2024-04-20T17:35:21.427 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8229000000000001e-05 2024-04-20T17:35:21.427 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.427 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.427 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:21.428 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.405553+0000", 2024-04-20T17:35:21.428 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8109999999999997e-06 2024-04-20T17:35:21.428 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.428 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.428 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:21.428 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.405559+0000", 2024-04-20T17:35:21.428 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1500000000000004e-06 2024-04-20T17:35:21.428 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.428 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.428 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:21.428 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.437169+0000", 2024-04-20T17:35:21.428 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.031610304999999998 2024-04-20T17:35:21.428 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.428 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.429 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:21.429 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.437261+0000", 2024-04-20T17:35:21.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1257999999999999e-05 2024-04-20T17:35:21.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.429 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.429 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:35:21.429 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.437390+0000", 2024-04-20T17:35:21.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000129777 2024-04-20T17:35:21.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.429 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.429 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:21.429 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.451123+0000", 2024-04-20T17:35:21.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013732496 2024-04-20T17:35:21.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.429 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.430 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:21.430 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.568336+0000", 2024-04-20T17:35:21.430 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.11721342899999999 2024-04-20T17:35:21.430 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.430 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.430 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:21.430 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.568379+0000", 2024-04-20T17:35:21.430 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2344e-05 2024-04-20T17:35:21.430 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.430 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.430 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:21.430 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.568433+0000", 2024-04-20T17:35:21.430 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4728999999999997e-05 2024-04-20T17:35:21.430 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.431 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:21.431 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.431 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.431 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.431 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:159676 4.2 4:43aa80d4:::benchmark_data_smithi171_122772_object159675:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e482)", 2024-04-20T17:35:21.431 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:09.407698+0000", 2024-04-20T17:35:21.431 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 551.99918006099995, 2024-04-20T17:35:21.431 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16090689399999999, 2024-04-20T17:35:21.431 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:21.431 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:21.431 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:21.431 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:21.431 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:21.431 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:21.431 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 159676 2024-04-20T17:35:21.432 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.432 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:21.432 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.432 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:21.432 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.407698+0000", 2024-04-20T17:35:21.432 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.432 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.432 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.432 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:21.432 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.407698+0000", 2024-04-20T17:35:21.432 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.432 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.432 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.432 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:21.432 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.407699+0000", 2024-04-20T17:35:21.433 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4309999999999999e-06 2024-04-20T17:35:21.433 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.433 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.433 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:21.433 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.407720+0000", 2024-04-20T17:35:21.433 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.09e-05 2024-04-20T17:35:21.433 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.433 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.433 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:21.433 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.407732+0000", 2024-04-20T17:35:21.433 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1494999999999999e-05 2024-04-20T17:35:21.433 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.433 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.433 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:21.433 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.407740+0000", 2024-04-20T17:35:21.434 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9459999999999998e-06 2024-04-20T17:35:21.434 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.434 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.434 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:21.434 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.441541+0000", 2024-04-20T17:35:21.434 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.033800864 2024-04-20T17:35:21.434 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.434 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.434 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:21.434 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.441652+0000", 2024-04-20T17:35:21.434 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000111749 2024-04-20T17:35:21.434 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.434 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.434 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:35:21.434 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.442325+0000", 2024-04-20T17:35:21.435 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00067265499999999998 2024-04-20T17:35:21.435 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.435 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.435 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:21.435 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.451395+0000", 2024-04-20T17:35:21.435 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0090699519999999992 2024-04-20T17:35:21.435 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.435 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.435 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:21.435 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.568492+0000", 2024-04-20T17:35:21.435 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.117097259 2024-04-20T17:35:21.435 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.435 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.435 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:21.435 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.568546+0000", 2024-04-20T17:35:21.436 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3780999999999999e-05 2024-04-20T17:35:21.436 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.436 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.436 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:21.436 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.568605+0000", 2024-04-20T17:35:21.436 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8862e-05 2024-04-20T17:35:21.436 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.436 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:21.436 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.436 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.436 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.436 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:159684 4.2 4:43ec9ad4:::benchmark_data_smithi171_122772_object159683:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e482)", 2024-04-20T17:35:21.437 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:09.415274+0000", 2024-04-20T17:35:21.437 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 551.99160441900005, 2024-04-20T17:35:21.437 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15358047, 2024-04-20T17:35:21.437 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:21.437 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:21.437 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:21.437 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:21.437 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:21.437 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:21.437 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 159684 2024-04-20T17:35:21.437 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.437 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:21.437 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.437 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:21.437 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.415274+0000", 2024-04-20T17:35:21.438 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.438 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.438 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.438 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:21.438 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.415274+0000", 2024-04-20T17:35:21.438 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.438 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.438 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.438 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:21.438 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.415276+0000", 2024-04-20T17:35:21.438 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.216e-06 2024-04-20T17:35:21.438 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.438 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.438 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:21.439 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.415300+0000", 2024-04-20T17:35:21.439 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4321e-05 2024-04-20T17:35:21.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.439 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.439 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:21.439 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.415312+0000", 2024-04-20T17:35:21.439 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1511e-05 2024-04-20T17:35:21.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.439 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.439 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:21.439 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.415319+0000", 2024-04-20T17:35:21.439 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8690000000000004e-06 2024-04-20T17:35:21.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.439 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.439 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:21.440 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.447198+0000", 2024-04-20T17:35:21.440 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.031878451000000002 2024-04-20T17:35:21.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.440 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.440 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:21.440 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.447351+0000", 2024-04-20T17:35:21.440 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015293300000000001 2024-04-20T17:35:21.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.440 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.440 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:35:21.440 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.447511+0000", 2024-04-20T17:35:21.440 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000159872 2024-04-20T17:35:21.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.440 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.440 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:21.441 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.451449+0000", 2024-04-20T17:35:21.441 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0039380539999999999 2024-04-20T17:35:21.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.441 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:21.441 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.568785+0000", 2024-04-20T17:35:21.441 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.11733666099999999 2024-04-20T17:35:21.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.441 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:21.441 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.568820+0000", 2024-04-20T17:35:21.441 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4586999999999999e-05 2024-04-20T17:35:21.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.441 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:21.442 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:09.568854+0000", 2024-04-20T17:35:21.442 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3995000000000002e-05 2024-04-20T17:35:21.442 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.442 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:21.442 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.442 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.442 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.442 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275200 4.6 4:65b8144d:::benchmark_data_smithi171_122772_object105008:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:35:21.442 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:10.998559+0000", 2024-04-20T17:35:21.442 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 430.40831944899998, 2024-04-20T17:35:21.442 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.21653923, 2024-04-20T17:35:21.443 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:21.443 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:21.443 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:21.443 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:21.443 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:21.443 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:21.443 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275200 2024-04-20T17:35:21.443 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.443 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:21.443 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.443 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:21.443 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:10.998559+0000", 2024-04-20T17:35:21.444 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.444 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.444 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.444 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:21.444 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:10.998559+0000", 2024-04-20T17:35:21.444 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.444 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.444 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.444 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:21.444 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:10.998560+0000", 2024-04-20T17:35:21.444 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9120000000000001e-06 2024-04-20T17:35:21.444 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.444 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.444 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:21.444 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:10.998566+0000", 2024-04-20T17:35:21.445 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6099999999999997e-06 2024-04-20T17:35:21.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.445 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:21.445 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:10.998578+0000", 2024-04-20T17:35:21.445 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1810999999999999e-05 2024-04-20T17:35:21.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.445 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:21.445 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:10.998586+0000", 2024-04-20T17:35:21.445 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.174e-06 2024-04-20T17:35:21.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.445 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:21.446 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:10.998644+0000", 2024-04-20T17:35:21.446 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7729000000000002e-05 2024-04-20T17:35:21.446 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.446 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.446 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:21.446 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:10.998762+0000", 2024-04-20T17:35:21.446 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000118021 2024-04-20T17:35:21.446 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.446 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.446 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:35:21.446 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:10.998873+0000", 2024-04-20T17:35:21.446 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011142 2024-04-20T17:35:21.446 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.446 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.447 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:21.447 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.003756+0000", 2024-04-20T17:35:21.447 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0048830059999999996 2024-04-20T17:35:21.447 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.447 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.447 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:21.447 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.215029+0000", 2024-04-20T17:35:21.447 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.21127307100000001 2024-04-20T17:35:21.447 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.447 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.447 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:21.447 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.215064+0000", 2024-04-20T17:35:21.447 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4406999999999998e-05 2024-04-20T17:35:21.447 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.448 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.448 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:21.448 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.215098+0000", 2024-04-20T17:35:21.448 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4069000000000002e-05 2024-04-20T17:35:21.448 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.448 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:21.448 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.448 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.448 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.448 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275211 4.6 4:6b91bd7d:::benchmark_data_smithi171_122772_object105019:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:35:21.448 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:11.008900+0000", 2024-04-20T17:35:21.448 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 430.39797808100002, 2024-04-20T17:35:21.448 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20628696999999999, 2024-04-20T17:35:21.448 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:21.448 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:21.449 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:21.449 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:21.449 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:21.449 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:21.449 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275211 2024-04-20T17:35:21.449 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.449 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:21.449 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.449 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:21.449 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.008900+0000", 2024-04-20T17:35:21.449 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.449 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.449 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.449 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:21.449 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.008900+0000", 2024-04-20T17:35:21.450 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.450 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.450 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.450 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:21.450 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.008902+0000", 2024-04-20T17:35:21.450 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1569999999999998e-06 2024-04-20T17:35:21.450 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.450 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.450 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:21.450 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.008909+0000", 2024-04-20T17:35:21.450 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5810000000000002e-06 2024-04-20T17:35:21.450 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.450 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.450 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:21.451 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.008922+0000", 2024-04-20T17:35:21.451 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3468999999999999e-05 2024-04-20T17:35:21.451 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.451 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.451 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:21.451 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.008929+0000", 2024-04-20T17:35:21.451 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2919999999999998e-06 2024-04-20T17:35:21.451 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.451 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.451 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:21.451 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.008986+0000", 2024-04-20T17:35:21.451 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6913999999999998e-05 2024-04-20T17:35:21.451 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.451 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.451 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:21.452 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.009110+0000", 2024-04-20T17:35:21.452 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012400599999999999 2024-04-20T17:35:21.452 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.452 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.452 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:35:21.452 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.009222+0000", 2024-04-20T17:35:21.452 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000111906 2024-04-20T17:35:21.452 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.452 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.452 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:21.452 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.014311+0000", 2024-04-20T17:35:21.452 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0050890980000000002 2024-04-20T17:35:21.452 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.452 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.452 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:21.453 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.215130+0000", 2024-04-20T17:35:21.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20081843799999999 2024-04-20T17:35:21.453 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.453 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.453 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:21.453 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.215160+0000", 2024-04-20T17:35:21.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0673000000000003e-05 2024-04-20T17:35:21.453 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.453 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.453 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:21.453 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.215187+0000", 2024-04-20T17:35:21.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6435999999999999e-05 2024-04-20T17:35:21.453 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.453 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:21.453 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.454 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.454 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.454 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275217 4.6 4:6f52af41:::benchmark_data_smithi171_122772_object105025:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:35:21.454 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:11.014905+0000", 2024-04-20T17:35:21.454 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 430.39197312300001, 2024-04-20T17:35:21.454 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20678690999999999, 2024-04-20T17:35:21.454 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:21.454 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:21.454 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:21.454 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:21.454 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:21.454 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:21.454 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275217 2024-04-20T17:35:21.454 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.455 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:21.455 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.455 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:21.455 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.014905+0000", 2024-04-20T17:35:21.455 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.455 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.455 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.455 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:21.455 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.014905+0000", 2024-04-20T17:35:21.455 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.455 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.455 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.455 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:21.455 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.014907+0000", 2024-04-20T17:35:21.455 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.967e-06 2024-04-20T17:35:21.455 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.456 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.456 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:21.456 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.014912+0000", 2024-04-20T17:35:21.456 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3759999999999999e-06 2024-04-20T17:35:21.456 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.456 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.456 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:21.456 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.014925+0000", 2024-04-20T17:35:21.456 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3174e-05 2024-04-20T17:35:21.456 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.456 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.456 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:21.456 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.014932+0000", 2024-04-20T17:35:21.456 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8229999999999997e-06 2024-04-20T17:35:21.456 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.457 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:21.457 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.014985+0000", 2024-04-20T17:35:21.457 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3062000000000001e-05 2024-04-20T17:35:21.457 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.457 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:21.457 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.015090+0000", 2024-04-20T17:35:21.457 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000104502 2024-04-20T17:35:21.457 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.457 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:35:21.457 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.015196+0000", 2024-04-20T17:35:21.457 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000106044 2024-04-20T17:35:21.458 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.458 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.458 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:21.458 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.024773+0000", 2024-04-20T17:35:21.458 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0095773449999999993 2024-04-20T17:35:21.458 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.458 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.458 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:21.458 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.221616+0000", 2024-04-20T17:35:21.458 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.196842454 2024-04-20T17:35:21.458 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.458 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.458 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:21.458 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.221649+0000", 2024-04-20T17:35:21.459 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3352999999999997e-05 2024-04-20T17:35:21.459 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.459 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.459 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:21.459 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.221692+0000", 2024-04-20T17:35:21.459 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2809999999999998e-05 2024-04-20T17:35:21.459 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.459 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:21.459 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.459 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.459 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.459 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275240 4.6 4:6dacae24:::benchmark_data_smithi171_122772_object105048:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:35:21.459 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:11.031577+0000", 2024-04-20T17:35:21.459 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 430.37530075000001, 2024-04-20T17:35:21.460 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19679422799999999, 2024-04-20T17:35:21.460 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:21.460 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:21.460 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:21.460 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:21.460 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:21.460 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:21.460 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275240 2024-04-20T17:35:21.460 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.460 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:21.460 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.460 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:21.460 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.031577+0000", 2024-04-20T17:35:21.460 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.460 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.461 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.461 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:21.461 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.031577+0000", 2024-04-20T17:35:21.461 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.461 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.461 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.461 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:21.461 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.031579+0000", 2024-04-20T17:35:21.461 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4610000000000001e-06 2024-04-20T17:35:21.461 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.461 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.461 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:21.461 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.031583+0000", 2024-04-20T17:35:21.461 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6870000000000002e-06 2024-04-20T17:35:21.461 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.461 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.462 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:21.462 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.031594+0000", 2024-04-20T17:35:21.462 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0594000000000001e-05 2024-04-20T17:35:21.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.462 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:21.462 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.031601+0000", 2024-04-20T17:35:21.462 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9240000000000001e-06 2024-04-20T17:35:21.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.462 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:21.462 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.031653+0000", 2024-04-20T17:35:21.462 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2228999999999997e-05 2024-04-20T17:35:21.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.463 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:21.463 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.031761+0000", 2024-04-20T17:35:21.463 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000108008 2024-04-20T17:35:21.463 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.463 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.463 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:35:21.463 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.031871+0000", 2024-04-20T17:35:21.463 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000110331 2024-04-20T17:35:21.463 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.463 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.463 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:21.463 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.036276+0000", 2024-04-20T17:35:21.463 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.004404799 2024-04-20T17:35:21.463 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.463 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.464 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:21.464 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.228303+0000", 2024-04-20T17:35:21.464 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19202655299999999 2024-04-20T17:35:21.464 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.464 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.464 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:21.464 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.228336+0000", 2024-04-20T17:35:21.464 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3281000000000002e-05 2024-04-20T17:35:21.464 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.464 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.464 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:21.464 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.228371+0000", 2024-04-20T17:35:21.464 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5361000000000002e-05 2024-04-20T17:35:21.464 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.464 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:21.465 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.465 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.465 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.465 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275248 4.6 4:65347144:::benchmark_data_smithi171_122772_object105056:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:35:21.465 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:11.041968+0000", 2024-04-20T17:35:21.465 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 430.364910436, 2024-04-20T17:35:21.465 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.18653969300000001, 2024-04-20T17:35:21.465 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:21.465 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:21.465 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:21.465 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:21.465 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:21.465 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:21.465 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275248 2024-04-20T17:35:21.466 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.466 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:21.466 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.466 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:21.466 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.041968+0000", 2024-04-20T17:35:21.466 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.466 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.466 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.466 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:21.466 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.041968+0000", 2024-04-20T17:35:21.466 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.466 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.466 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.466 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:21.466 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.041970+0000", 2024-04-20T17:35:21.467 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1229999999999998e-06 2024-04-20T17:35:21.467 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.467 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.467 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:21.467 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.041976+0000", 2024-04-20T17:35:21.467 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.852e-06 2024-04-20T17:35:21.467 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.467 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.467 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:21.467 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.041987+0000", 2024-04-20T17:35:21.467 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1355000000000001e-05 2024-04-20T17:35:21.468 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.468 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.468 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:21.468 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.041995+0000", 2024-04-20T17:35:21.468 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4540000000000007e-06 2024-04-20T17:35:21.468 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.468 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.468 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:21.468 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.042050+0000", 2024-04-20T17:35:21.468 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4239e-05 2024-04-20T17:35:21.468 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.468 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.468 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:21.469 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.042138+0000", 2024-04-20T17:35:21.469 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8004999999999995e-05 2024-04-20T17:35:21.469 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.469 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.469 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:35:21.469 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.042261+0000", 2024-04-20T17:35:21.469 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012317299999999999 2024-04-20T17:35:21.469 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.469 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.469 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:21.469 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.046505+0000", 2024-04-20T17:35:21.469 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0042438600000000003 2024-04-20T17:35:21.469 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.469 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.470 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:21.470 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.228413+0000", 2024-04-20T17:35:21.470 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.18190864500000001 2024-04-20T17:35:21.470 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.470 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.470 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:21.470 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.228465+0000", 2024-04-20T17:35:21.470 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2074000000000003e-05 2024-04-20T17:35:21.470 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.470 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.470 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:21.470 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.228507+0000", 2024-04-20T17:35:21.470 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1912999999999997e-05 2024-04-20T17:35:21.470 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.471 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:21.471 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.471 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.471 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.471 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275263 4.6 4:6160fa82:::benchmark_data_smithi171_122772_object105071:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:35:21.471 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:11.051143+0000", 2024-04-20T17:35:21.471 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 430.35573451800002, 2024-04-20T17:35:21.471 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.18078492099999999, 2024-04-20T17:35:21.471 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:21.471 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:21.471 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:21.471 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:21.471 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:21.471 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:21.471 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275263 2024-04-20T17:35:21.472 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.472 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:21.472 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.472 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:21.472 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.051143+0000", 2024-04-20T17:35:21.472 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.472 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.472 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.472 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:21.472 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.051143+0000", 2024-04-20T17:35:21.472 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.472 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.472 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.472 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:21.472 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.051146+0000", 2024-04-20T17:35:21.473 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1730000000000002e-06 2024-04-20T17:35:21.473 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.473 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.473 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:21.473 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.051152+0000", 2024-04-20T17:35:21.473 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8579999999999996e-06 2024-04-20T17:35:21.473 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.473 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.473 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:21.473 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.051165+0000", 2024-04-20T17:35:21.473 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3661e-05 2024-04-20T17:35:21.473 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.473 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.473 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:21.473 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.051173+0000", 2024-04-20T17:35:21.474 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8329999999999994e-06 2024-04-20T17:35:21.474 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.474 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.474 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:21.474 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.051233+0000", 2024-04-20T17:35:21.474 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9837999999999999e-05 2024-04-20T17:35:21.474 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.474 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.474 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:21.474 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.051354+0000", 2024-04-20T17:35:21.474 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012070699999999999 2024-04-20T17:35:21.474 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.474 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.474 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:35:21.474 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.051468+0000", 2024-04-20T17:35:21.475 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000114658 2024-04-20T17:35:21.475 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.475 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.475 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:21.475 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.056305+0000", 2024-04-20T17:35:21.475 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0048365329999999996 2024-04-20T17:35:21.475 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.475 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.475 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:21.475 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.231849+0000", 2024-04-20T17:35:21.475 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.175544645 2024-04-20T17:35:21.475 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.475 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.475 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:21.475 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.231886+0000", 2024-04-20T17:35:21.476 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6322999999999998e-05 2024-04-20T17:35:21.476 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.476 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.476 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:21.476 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.231928+0000", 2024-04-20T17:35:21.476 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2691999999999997e-05 2024-04-20T17:35:21.476 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.476 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:21.476 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.476 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.476 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.476 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275278 4.6 4:6f1c4c64:::benchmark_data_smithi171_122772_object105086:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:35:21.476 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:11.064797+0000", 2024-04-20T17:35:21.476 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 430.34208133300001, 2024-04-20T17:35:21.477 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.167260615, 2024-04-20T17:35:21.477 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:21.477 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:21.477 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:21.477 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:21.477 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:21.477 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:21.477 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275278 2024-04-20T17:35:21.477 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.477 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:21.477 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.477 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:21.477 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.064797+0000", 2024-04-20T17:35:21.477 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.477 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.478 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.478 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:21.478 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.064797+0000", 2024-04-20T17:35:21.478 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.478 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.478 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.478 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:21.478 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.064798+0000", 2024-04-20T17:35:21.478 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.567e-06 2024-04-20T17:35:21.478 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.478 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.478 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:21.478 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.064803+0000", 2024-04-20T17:35:21.478 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0429999999999998e-06 2024-04-20T17:35:21.479 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.479 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.479 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:21.479 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.064815+0000", 2024-04-20T17:35:21.479 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.148e-05 2024-04-20T17:35:21.479 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.479 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.479 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:21.479 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.064822+0000", 2024-04-20T17:35:21.479 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2359999999999998e-06 2024-04-20T17:35:21.479 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.479 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.479 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:21.479 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.064880+0000", 2024-04-20T17:35:21.479 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7698999999999998e-05 2024-04-20T17:35:21.480 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.480 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.480 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:21.480 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.064999+0000", 2024-04-20T17:35:21.480 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011958 2024-04-20T17:35:21.480 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.480 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.480 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:35:21.480 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.065120+0000", 2024-04-20T17:35:21.480 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000120512 2024-04-20T17:35:21.480 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.480 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.480 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:21.480 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.069852+0000", 2024-04-20T17:35:21.480 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0047317180000000002 2024-04-20T17:35:21.481 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.481 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.481 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:21.481 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.231968+0000", 2024-04-20T17:35:21.481 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.162116383 2024-04-20T17:35:21.481 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.481 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.481 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:21.481 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.232017+0000", 2024-04-20T17:35:21.481 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9614999999999998e-05 2024-04-20T17:35:21.481 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.481 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.481 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:21.481 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.232057+0000", 2024-04-20T17:35:21.481 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9782e-05 2024-04-20T17:35:21.482 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.482 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:21.482 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.482 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.482 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.482 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275285 4.6 4:63a3635d:::benchmark_data_smithi171_122772_object105093:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:35:21.482 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:11.074387+0000", 2024-04-20T17:35:21.482 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 430.33249102299999, 2024-04-20T17:35:21.482 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16127122899999999, 2024-04-20T17:35:21.482 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:21.482 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:21.482 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:21.482 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:21.482 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:21.483 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:21.483 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275285 2024-04-20T17:35:21.483 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.483 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:21.483 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.483 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:21.483 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.074387+0000", 2024-04-20T17:35:21.483 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.483 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.483 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.483 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:21.483 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.074387+0000", 2024-04-20T17:35:21.483 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.483 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.483 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.484 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:21.484 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.074388+0000", 2024-04-20T17:35:21.484 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.339e-06 2024-04-20T17:35:21.484 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.484 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.484 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:21.484 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.074393+0000", 2024-04-20T17:35:21.484 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4259999999999996e-06 2024-04-20T17:35:21.484 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.484 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.484 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:21.484 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.074402+0000", 2024-04-20T17:35:21.484 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6609999999999995e-06 2024-04-20T17:35:21.484 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.484 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.485 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:21.485 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.074408+0000", 2024-04-20T17:35:21.485 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6520000000000003e-06 2024-04-20T17:35:21.485 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.485 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.485 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:21.485 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.074455+0000", 2024-04-20T17:35:21.485 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6717e-05 2024-04-20T17:35:21.485 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.485 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.485 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:21.485 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.074530+0000", 2024-04-20T17:35:21.485 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5661000000000006e-05 2024-04-20T17:35:21.485 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.485 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.486 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:35:21.486 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.074597+0000", 2024-04-20T17:35:21.486 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6418e-05 2024-04-20T17:35:21.486 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.486 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.486 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:21.486 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.078692+0000", 2024-04-20T17:35:21.486 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0040947350000000004 2024-04-20T17:35:21.486 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.486 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.486 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:21.486 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.235564+0000", 2024-04-20T17:35:21.486 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15687225099999999 2024-04-20T17:35:21.486 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.486 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.487 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:21.487 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.235616+0000", 2024-04-20T17:35:21.487 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2599999999999998e-05 2024-04-20T17:35:21.487 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.487 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.487 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:21.487 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.235658+0000", 2024-04-20T17:35:21.487 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1768999999999999e-05 2024-04-20T17:35:21.487 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.487 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:21.487 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.487 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.487 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.487 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275296 4.6 4:6850fa57:::benchmark_data_smithi171_122772_object105104:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:35:21.487 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:11.083403+0000", 2024-04-20T17:35:21.487 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 430.32347506500003, 2024-04-20T17:35:21.488 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15593427100000001, 2024-04-20T17:35:21.488 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:21.488 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:21.488 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:21.488 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:21.488 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:21.488 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:21.488 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275296 2024-04-20T17:35:21.488 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.488 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:21.488 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.488 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:21.488 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.083403+0000", 2024-04-20T17:35:21.488 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.488 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.489 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.489 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:21.489 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.083403+0000", 2024-04-20T17:35:21.489 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.489 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.489 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.489 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:21.489 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.083406+0000", 2024-04-20T17:35:21.489 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8370000000000001e-06 2024-04-20T17:35:21.489 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.489 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.489 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:21.489 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.083414+0000", 2024-04-20T17:35:21.489 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2630000000000003e-06 2024-04-20T17:35:21.489 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.490 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.490 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:21.490 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.083431+0000", 2024-04-20T17:35:21.490 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6560000000000001e-05 2024-04-20T17:35:21.490 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.490 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.490 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:21.490 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.083441+0000", 2024-04-20T17:35:21.490 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0132000000000001e-05 2024-04-20T17:35:21.490 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.490 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.490 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:21.490 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.083503+0000", 2024-04-20T17:35:21.490 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2119000000000006e-05 2024-04-20T17:35:21.491 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.491 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.491 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:21.491 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.083625+0000", 2024-04-20T17:35:21.491 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000122547 2024-04-20T17:35:21.491 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.491 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.491 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:35:21.491 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.083735+0000", 2024-04-20T17:35:21.491 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000109764 2024-04-20T17:35:21.491 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.491 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.491 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:21.491 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.088797+0000", 2024-04-20T17:35:21.491 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0050613510000000004 2024-04-20T17:35:21.492 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.492 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.492 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:21.492 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.239279+0000", 2024-04-20T17:35:21.492 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15048260599999999 2024-04-20T17:35:21.492 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.492 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.492 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:21.492 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.239310+0000", 2024-04-20T17:35:21.492 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1300999999999997e-05 2024-04-20T17:35:21.492 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.492 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.492 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:21.492 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.239337+0000", 2024-04-20T17:35:21.492 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6791000000000001e-05 2024-04-20T17:35:21.493 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.493 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:21.493 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.493 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.493 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.493 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:275312 4.6 4:6a829b24:::benchmark_data_smithi171_122772_object105120:head [delete] snapc 0=[] ondisk+write+known_if_redirected e520)", 2024-04-20T17:35:21.493 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:11.101309+0000", 2024-04-20T17:35:21.493 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 430.30556927100002, 2024-04-20T17:35:21.493 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.14268697999999999, 2024-04-20T17:35:21.493 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:21.493 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:21.493 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:21.493 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:21.493 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:21.493 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:21.494 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 275312 2024-04-20T17:35:21.494 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.494 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:21.494 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.494 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:21.494 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.101309+0000", 2024-04-20T17:35:21.494 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.494 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.494 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.494 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:21.494 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.101309+0000", 2024-04-20T17:35:21.494 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.494 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.494 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.495 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:21.495 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.101312+0000", 2024-04-20T17:35:21.495 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3110000000000001e-06 2024-04-20T17:35:21.495 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.495 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.495 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:21.495 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.101320+0000", 2024-04-20T17:35:21.495 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7659999999999998e-06 2024-04-20T17:35:21.495 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.495 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.495 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:21.495 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.101337+0000", 2024-04-20T17:35:21.495 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7345000000000001e-05 2024-04-20T17:35:21.495 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.495 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.496 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:21.496 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.101347+0000", 2024-04-20T17:35:21.496 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.594e-06 2024-04-20T17:35:21.496 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.496 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.496 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:21.496 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.101415+0000", 2024-04-20T17:35:21.496 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8751000000000006e-05 2024-04-20T17:35:21.496 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.496 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.496 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:21.496 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.101540+0000", 2024-04-20T17:35:21.496 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000124029 2024-04-20T17:35:21.496 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.496 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.497 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:35:21.497 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.101658+0000", 2024-04-20T17:35:21.497 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000118235 2024-04-20T17:35:21.497 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.497 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.497 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:21.497 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.106325+0000", 2024-04-20T17:35:21.497 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0046667490000000004 2024-04-20T17:35:21.497 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.497 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.497 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:21.497 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.243904+0000", 2024-04-20T17:35:21.497 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.137579902 2024-04-20T17:35:21.497 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.497 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.498 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:21.498 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.243956+0000", 2024-04-20T17:35:21.498 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1690999999999997e-05 2024-04-20T17:35:21.498 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.498 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.498 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:21.498 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:11.243996+0000", 2024-04-20T17:35:21.498 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9607000000000002e-05 2024-04-20T17:35:21.498 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.498 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:21.498 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.498 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.498 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.498 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.16506.0:202760 5.b 5:d9759bc2:::benchmark_data_smithi171_137281_object71326:head [delete] snapc 0=[] ondisk+write+known_if_redirected e599)", 2024-04-20T17:35:21.498 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:32:08.720790+0000", 2024-04-20T17:35:21.499 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 192.686087789, 2024-04-20T17:35:21.499 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.22549492700000001, 2024-04-20T17:35:21.499 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:21.499 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:21.499 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:21.499 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:21.499 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.16506", 2024-04-20T17:35:21.499 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:35:21.499 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 202760 2024-04-20T17:35:21.499 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.499 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:21.499 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.499 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:21.499 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.720790+0000", 2024-04-20T17:35:21.499 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.500 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.500 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.500 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:21.500 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.720790+0000", 2024-04-20T17:35:21.500 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.500 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.500 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.500 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:21.500 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.720793+0000", 2024-04-20T17:35:21.500 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3539999999999998e-06 2024-04-20T17:35:21.500 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.500 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.500 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:21.500 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.720800+0000", 2024-04-20T17:35:21.500 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.046e-06 2024-04-20T17:35:21.500 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.501 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.501 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:21.501 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.720814+0000", 2024-04-20T17:35:21.501 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4158000000000001e-05 2024-04-20T17:35:21.501 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.501 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.501 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:21.501 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.720821+0000", 2024-04-20T17:35:21.501 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7499999999999997e-06 2024-04-20T17:35:21.501 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.501 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.501 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:21.501 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.720879+0000", 2024-04-20T17:35:21.501 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8934999999999997e-05 2024-04-20T17:35:21.501 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.502 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.502 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:21.502 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.721008+0000", 2024-04-20T17:35:21.502 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012829700000000001 2024-04-20T17:35:21.502 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.502 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.502 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:35:21.502 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.721129+0000", 2024-04-20T17:35:21.502 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000121704 2024-04-20T17:35:21.502 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.502 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.502 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:21.502 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.729580+0000", 2024-04-20T17:35:21.502 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0084508080000000006 2024-04-20T17:35:21.503 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.503 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.503 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:21.503 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.946173+0000", 2024-04-20T17:35:21.503 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.21659226000000001 2024-04-20T17:35:21.503 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.503 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.503 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:21.503 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.946239+0000", 2024-04-20T17:35:21.503 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6212999999999998e-05 2024-04-20T17:35:21.503 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.503 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.503 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:21.503 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.946285+0000", 2024-04-20T17:35:21.503 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6402000000000001e-05 2024-04-20T17:35:21.504 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.504 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:21.504 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.504 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.504 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.504 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.16506.0:202763 5.b 5:dbb9b06e:::benchmark_data_smithi171_137281_object71329:head [delete] snapc 0=[] ondisk+write+known_if_redirected e599)", 2024-04-20T17:35:21.504 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:32:08.722780+0000", 2024-04-20T17:35:21.504 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 192.68409826000001, 2024-04-20T17:35:21.504 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.22980968099999999, 2024-04-20T17:35:21.504 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:21.504 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:21.504 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:21.504 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:21.504 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.16506", 2024-04-20T17:35:21.504 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:35:21.505 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 202763 2024-04-20T17:35:21.505 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.505 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:21.505 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.505 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:21.505 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.722780+0000", 2024-04-20T17:35:21.505 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.505 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.505 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.505 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:21.505 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.722780+0000", 2024-04-20T17:35:21.505 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.505 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.505 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.505 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:21.506 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.722781+0000", 2024-04-20T17:35:21.506 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8700000000000004e-07 2024-04-20T17:35:21.506 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.506 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.506 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:21.506 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.722784+0000", 2024-04-20T17:35:21.506 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0740000000000001e-06 2024-04-20T17:35:21.506 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.506 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.506 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:21.506 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.722793+0000", 2024-04-20T17:35:21.506 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3300000000000005e-06 2024-04-20T17:35:21.506 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.506 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.507 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:21.507 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.722800+0000", 2024-04-20T17:35:21.507 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3239999999999997e-06 2024-04-20T17:35:21.507 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.507 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.507 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:21.507 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.725063+0000", 2024-04-20T17:35:21.507 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0022626 2024-04-20T17:35:21.507 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.507 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.507 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:21.507 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.725184+0000", 2024-04-20T17:35:21.507 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000120701 2024-04-20T17:35:21.507 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.507 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.508 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:35:21.508 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.725275+0000", 2024-04-20T17:35:21.508 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1162000000000006e-05 2024-04-20T17:35:21.508 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.508 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.508 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:21.508 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.729728+0000", 2024-04-20T17:35:21.508 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0044527710000000003 2024-04-20T17:35:21.508 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.508 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.508 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:21.508 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.952529+0000", 2024-04-20T17:35:21.508 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.22280121 2024-04-20T17:35:21.508 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.508 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.509 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:21.509 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.952562+0000", 2024-04-20T17:35:21.509 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3253000000000002e-05 2024-04-20T17:35:21.509 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.509 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.509 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:21.509 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.952589+0000", 2024-04-20T17:35:21.509 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7368999999999998e-05 2024-04-20T17:35:21.509 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.509 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:21.509 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.509 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.509 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.509 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.16506.0:202825 5.b 5:d9279200:::benchmark_data_smithi171_137281_object71391:head [delete] snapc 0=[] ondisk+write+known_if_redirected e599)", 2024-04-20T17:35:21.509 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:32:08.801332+0000", 2024-04-20T17:35:21.510 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 192.60554558800001, 2024-04-20T17:35:21.510 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15138621499999999, 2024-04-20T17:35:21.510 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:21.510 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:21.510 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:21.510 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:21.510 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.16506", 2024-04-20T17:35:21.510 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:35:21.510 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 202825 2024-04-20T17:35:21.510 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.510 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:21.510 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.510 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:21.510 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.801332+0000", 2024-04-20T17:35:21.511 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.511 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.511 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.511 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:21.511 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.801332+0000", 2024-04-20T17:35:21.511 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.511 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.511 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.511 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:21.511 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.801335+0000", 2024-04-20T17:35:21.511 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.852e-06 2024-04-20T17:35:21.511 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.511 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.511 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:21.511 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.801342+0000", 2024-04-20T17:35:21.512 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0480000000000001e-06 2024-04-20T17:35:21.512 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.512 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.512 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:21.512 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.801358+0000", 2024-04-20T17:35:21.512 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5489e-05 2024-04-20T17:35:21.512 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.512 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.512 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:21.512 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.801367+0000", 2024-04-20T17:35:21.512 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2490000000000007e-06 2024-04-20T17:35:21.512 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.512 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.512 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:21.512 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.801422+0000", 2024-04-20T17:35:21.513 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5275000000000002e-05 2024-04-20T17:35:21.513 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.513 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.513 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:21.513 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.801531+0000", 2024-04-20T17:35:21.513 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000108706 2024-04-20T17:35:21.513 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.513 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.513 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:35:21.513 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.801616+0000", 2024-04-20T17:35:21.513 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5022e-05 2024-04-20T17:35:21.513 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.513 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.513 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:21.513 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.805722+0000", 2024-04-20T17:35:21.514 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0041055939999999997 2024-04-20T17:35:21.514 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.514 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.514 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:21.514 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.952646+0000", 2024-04-20T17:35:21.514 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.146924468 2024-04-20T17:35:21.514 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.514 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.514 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:21.514 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.952689+0000", 2024-04-20T17:35:21.514 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3019000000000003e-05 2024-04-20T17:35:21.514 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.514 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.514 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:21.514 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.952719+0000", 2024-04-20T17:35:21.515 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9493e-05 2024-04-20T17:35:21.515 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.515 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:21.515 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.515 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.515 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.515 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.16506.0:202840 5.b 5:d515e270:::benchmark_data_smithi171_137281_object71406:head [delete] snapc 0=[] ondisk+write+known_if_redirected e599)", 2024-04-20T17:35:21.515 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:32:08.814812+0000", 2024-04-20T17:35:21.515 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 192.59206583899999, 2024-04-20T17:35:21.515 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.13804498100000001, 2024-04-20T17:35:21.515 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:21.515 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:21.515 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:21.515 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:21.515 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.16506", 2024-04-20T17:35:21.516 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:35:21.516 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 202840 2024-04-20T17:35:21.516 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.516 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:21.516 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.516 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:21.516 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.814812+0000", 2024-04-20T17:35:21.516 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.516 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.516 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.516 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:21.516 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.814812+0000", 2024-04-20T17:35:21.516 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:21.516 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.516 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.517 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:21.517 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.814814+0000", 2024-04-20T17:35:21.517 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7349999999999999e-06 2024-04-20T17:35:21.517 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.517 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.517 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:21.517 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.814819+0000", 2024-04-20T17:35:21.517 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6999999999999999e-06 2024-04-20T17:35:21.517 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.517 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.517 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:21.517 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.814828+0000", 2024-04-20T17:35:21.517 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2499999999999995e-06 2024-04-20T17:35:21.517 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.517 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.518 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:21.518 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.814835+0000", 2024-04-20T17:35:21.518 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6029999999999999e-06 2024-04-20T17:35:21.518 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.518 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.518 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:21.518 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.814896+0000", 2024-04-20T17:35:21.518 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0744e-05 2024-04-20T17:35:21.518 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.518 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.518 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:21.518 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.815015+0000", 2024-04-20T17:35:21.518 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011930399999999999 2024-04-20T17:35:21.518 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.518 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.519 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:35:21.519 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.815108+0000", 2024-04-20T17:35:21.519 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2404000000000005e-05 2024-04-20T17:35:21.519 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.519 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.519 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:21.519 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.818852+0000", 2024-04-20T17:35:21.519 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.003743803 2024-04-20T17:35:21.519 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.519 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.519 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:21.519 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.952772+0000", 2024-04-20T17:35:21.519 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.13392010300000001 2024-04-20T17:35:21.519 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.519 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.520 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:21.520 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.952824+0000", 2024-04-20T17:35:21.520 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1805999999999998e-05 2024-04-20T17:35:21.520 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:21.520 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:21.520 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:21.520 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:32:08.952857+0000", 2024-04-20T17:35:21.520 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3528999999999997e-05 2024-04-20T17:35:21.520 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.520 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:21.520 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.520 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:21.520 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:21.520 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:35:21.521 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:35:21.577 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:21.655 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:35:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.125 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-20T17:35:22.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.454 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:35:22.772 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:35:22.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:22.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:35:22.813 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:35:22.813 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:35:22.877 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:35:22.905 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:35:22.920 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:22.965 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:35:22.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:22.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:35:22.979 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:22 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:22.979 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:22 smithi171 ceph-mon[44189]: pgmap v1720: 17 pgs: 17 active+clean; 9.2 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-20T17:35:23.068 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:35:23.098 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:23.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:22 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:23.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:22 smithi005 ceph-mon[39562]: pgmap v1720: 17 pgs: 17 active+clean; 9.2 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-20T17:35:23.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:22 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:23.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:22 smithi106 ceph-mon[44423]: pgmap v1720: 17 pgs: 17 active+clean; 9.2 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-20T17:35:23.751 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:35:23.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:23.754 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:35:23.855 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:35:23.902 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:23.936 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:35:23.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:23.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:35:24.043 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:35:24.079 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:24.178 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:35:24.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:24.182 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:35:24.283 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:35:24.328 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:35:24.328 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:35:24.328 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:24.328 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.26882.0:178459 6.f e652/619 6:f9ec71f1:::benchmark_data_smithi171_147972_object17361:head v 652'11274, mlcod=652'11271)", 2024-04-20T17:35:24.328 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:35:24.323843+0000", 2024-04-20T17:35:24.328 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.0040672420000000004, 2024-04-20T17:35:24.328 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.004094215, 2024-04-20T17:35:24.328 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:24.328 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:24.328 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:35:24.329 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:24.329 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:24.329 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:24.329 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:35:24.323843+0000", 2024-04-20T17:35:24.329 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:24.329 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:24.329 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:24.329 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:24.329 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:35:24.323843+0000", 2024-04-20T17:35:24.329 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:24.329 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:24.329 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:24.329 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:24.329 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:35:24.323844+0000", 2024-04-20T17:35:24.330 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.133e-06 2024-04-20T17:35:24.330 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:24.330 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:24.330 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:24.330 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:35:24.323851+0000", 2024-04-20T17:35:24.330 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.759e-06 2024-04-20T17:35:24.330 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:24.330 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:24.330 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:24.330 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:35:24.323861+0000", 2024-04-20T17:35:24.330 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0788000000000001e-05 2024-04-20T17:35:24.330 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:24.330 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:24.330 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:24.330 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:35:24.323868+0000", 2024-04-20T17:35:24.331 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.029e-06 2024-04-20T17:35:24.331 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:24.331 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:24.331 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:24.331 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:35:24.323916+0000", 2024-04-20T17:35:24.331 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7933000000000003e-05 2024-04-20T17:35:24.331 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:24.331 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:24.331 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:24.331 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:35:24.323935+0000", 2024-04-20T17:35:24.331 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8264999999999999e-05 2024-04-20T17:35:24.331 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:24.331 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:24.331 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:24.332 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:24.332 INFO:teuthology.orchestra.run.smithi171.stdout: ], 2024-04-20T17:35:24.332 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 1 2024-04-20T17:35:24.332 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:35:24.335 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:24.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:24.379 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:35:24.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:35:24.480 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:35:24.516 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:24.564 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:35:24.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:24.567 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:35:24.668 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:35:24.680 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:24 smithi171 ceph-mon[44189]: pgmap v1721: 17 pgs: 17 active+clean; 8.9 GiB data, 22 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-20T17:35:24.716 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:24.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:24 smithi005 ceph-mon[39562]: pgmap v1721: 17 pgs: 17 active+clean; 8.9 GiB data, 22 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-20T17:35:24.741 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:35:24.745 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:24 smithi106 ceph-mon[44423]: pgmap v1721: 17 pgs: 17 active+clean; 8.9 GiB data, 22 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 1.28k op/s 2024-04-20T17:35:24.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:24.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:35:24.815 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:35:24.847 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:35:24.880 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:24.901 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:35:24.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:24.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:35:25.008 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:35:25.046 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-20T17:35:25.082 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:25.132 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:35:25.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:25.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:35:25.243 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:35:25.274 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:25.303 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:35:25.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:25.309 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:35:25.328 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:35:25.362 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:35:25.410 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:35:25.444 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:25.653 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:35:25.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:25.658 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:35:25.760 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:35:25.789 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:26.239 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:35:26.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:26.246 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:35:26.347 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:35:26.379 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:26.495 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:26 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:35:26.495 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:26 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:35:26.495 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:26 smithi005 ceph-mon[39562]: osdmap e653: 12 total, 11 up, 10 in 2024-04-20T17:35:26.495 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:26 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/155060568' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:35:26.495 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:26 smithi005 ceph-mon[39562]: from='client.26872 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:35:26.495 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:26 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:35:26.495 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:26 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:35:26.495 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:26 smithi005 ceph-mon[39562]: pgmap v1723: 17 pgs: 17 active+clean; 8.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 1.33k op/s 2024-04-20T17:35:26.531 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:35:26.735 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:35:26.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:26.739 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:35:26.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:26 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:35:26.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:26 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:35:26.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:26 smithi106 ceph-mon[44423]: osdmap e653: 12 total, 11 up, 10 in 2024-04-20T17:35:26.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:26 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/155060568' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:35:26.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:26 smithi106 ceph-mon[44423]: from='client.26872 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:35:26.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:26 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:35:26.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:26 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:35:26.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:26 smithi106 ceph-mon[44423]: pgmap v1723: 17 pgs: 17 active+clean; 8.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 1.33k op/s 2024-04-20T17:35:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:26 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:35:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:26 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:35:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:26 smithi171 ceph-mon[44189]: osdmap e653: 12 total, 11 up, 10 in 2024-04-20T17:35:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:26 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/155060568' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:35:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:26 smithi171 ceph-mon[44189]: from='client.26872 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:35:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:26 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:35:26.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:26 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:35:26.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:26 smithi171 ceph-mon[44189]: pgmap v1723: 17 pgs: 17 active+clean; 8.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 1.33k op/s 2024-04-20T17:35:26.840 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:35:26.870 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:26.952 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:35:26.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:26.956 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:35:27.057 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:35:27.095 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:27.228 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:35:27.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:27.233 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:35:27.334 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:35:27.384 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:35:27.384 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:35:27.384 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-04-20T17:35:27.384 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-04-20T17:35:27.384 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:35:27.388 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:27.537 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:35:27.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:27.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:35:27.646 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:35:27.699 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:27 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:35:27.699 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:27 smithi005 ceph-mon[39562]: from='client.26872 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:35:27.699 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:27 smithi005 ceph-mon[39562]: osdmap e654: 12 total, 11 up, 10 in 2024-04-20T17:35:27.702 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:27.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:27 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:35:27.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:27 smithi106 ceph-mon[44423]: from='client.26872 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:35:27.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:27 smithi106 ceph-mon[44423]: osdmap e654: 12 total, 11 up, 10 in 2024-04-20T17:35:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:27 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:35:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:27 smithi171 ceph-mon[44189]: from='client.26872 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:35:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:27 smithi171 ceph-mon[44189]: osdmap e654: 12 total, 11 up, 10 in 2024-04-20T17:35:28.021 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:35:28.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:28.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:35:28.074 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-20T17:35:28.138 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:35:28.167 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:28.196 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:35:28.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:28.200 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:35:28.301 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:35:28.332 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:28.361 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:35:28.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:28.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:35:28.425 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:35:28.468 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:35:28.509 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:28.531 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:35:28.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:28.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:35:28.636 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:35:28.709 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:28.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:28 smithi106 ceph-mon[44423]: pgmap v1725: 17 pgs: 17 active+clean; 8.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-20T17:35:28.772 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:28 smithi171 ceph-mon[44189]: pgmap v1725: 17 pgs: 17 active+clean; 8.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-20T17:35:28.816 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:35:28.816 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:28 smithi005 ceph-mon[39562]: pgmap v1725: 17 pgs: 17 active+clean; 8.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-20T17:35:28.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:28.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:35:28.921 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:35:28.978 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:29.116 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:35:29.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:29.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:35:29.208 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:35:29.227 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:35:29.244 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:35:29.264 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:29.283 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:35:29.288 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:35:29.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:29.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:35:29.314 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:35:29.348 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:35:29.381 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:35:29.392 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:35:29.428 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:35:29.433 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:29.457 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:35:29.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:29.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:35:29.478 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:35:29.498 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:35:29.531 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:35:29.562 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:35:29.563 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:35:29.563 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-20T17:35:29.563 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:35:29.563 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:35:29.563 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:35:29.563 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:35:29.563 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:35:29.563 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:35:29.563 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:35:29.563 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:35:29.563 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:35:29.563 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:35:29.565 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:35:29.595 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:29.662 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:29 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3822848500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:35:29.662 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:29.674 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:35:29.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:29.679 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:35:29.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:29 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3822848500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:35:29.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:29.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:29 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3822848500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:35:29.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:29.779 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:35:29.819 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:30.024 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:35:30.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:30.032 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:35:30.068 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:35:30.133 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:35:30.173 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:30.274 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:35:30.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:30.280 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:35:30.321 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:35:30.322 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:35:30.374 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:35:30.375 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:35:30.381 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:35:30.414 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:30.502 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:35:30.565 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:35:30.570 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:30.570 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:30.570 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:30.570 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:30.570 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:30.570 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:30.570 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:30.570 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:30.570 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:30.570 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:30.570 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:30 smithi171 ceph-mon[44189]: pgmap v1726: 17 pgs: 17 active+clean; 8.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 1.38k op/s 2024-04-20T17:35:30.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:30.571 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:35:30.673 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:35:30.703 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:30.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:30.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:30.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:30.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:30.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:30.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:30.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:30.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:30.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:30.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:30.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:30 smithi005 ceph-mon[39562]: pgmap v1726: 17 pgs: 17 active+clean; 8.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 1.38k op/s 2024-04-20T17:35:31.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:31.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:31.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:31.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:31.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:31.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:31.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:31.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:31.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:31.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:31.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:30 smithi106 ceph-mon[44423]: pgmap v1726: 17 pgs: 17 active+clean; 8.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 1.38k op/s 2024-04-20T17:35:31.047 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-20T17:35:31.053 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:35:31.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:31.061 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:35:31.162 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:35:31.204 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:31.232 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:35:31.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:31.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:35:31.339 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:35:31.378 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:31.378 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:35:31.399 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:35:31.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:31.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:35:31.503 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:35:31.547 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:31.587 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:35:31.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:31.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:35:31.693 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:35:31.727 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:31.751 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:35:31.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:31.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:35:31.855 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:35:31.892 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:32.244 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:35:32.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:32.250 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:35:32.351 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:35:32.377 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:35:32.417 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=59,logm=1} crc {auth=155901282,config=1460075751,health=179246142,kv=1447570618,logm=610821111}) 2024-04-20T17:35:32.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1720977174}) 2024-04-20T17:35:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3594143753}) 2024-04-20T17:35:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1502191774}) 2024-04-20T17:35:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=479696460}) 2024-04-20T17:35:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2999020662}) 2024-04-20T17:35:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4168962406}) 2024-04-20T17:35:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1010796833}) 2024-04-20T17:35:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2623010325}) 2024-04-20T17:35:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1390100058}) 2024-04-20T17:35:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3476045037}) 2024-04-20T17:35:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=903846406}) 2024-04-20T17:35:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1299556456}) 2024-04-20T17:35:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1377531572}) 2024-04-20T17:35:32.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2811759264}) 2024-04-20T17:35:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1338009965}) 2024-04-20T17:35:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1390316277}) 2024-04-20T17:35:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3382104207}) 2024-04-20T17:35:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=991013629}) 2024-04-20T17:35:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3771618271}) 2024-04-20T17:35:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3008462934}) 2024-04-20T17:35:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2523039836}) 2024-04-20T17:35:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1140141808}) 2024-04-20T17:35:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512649288}) 2024-04-20T17:35:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1967430481}) 2024-04-20T17:35:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2692641017}) 2024-04-20T17:35:32.586 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2572291463}) 2024-04-20T17:35:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2158081595}) 2024-04-20T17:35:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=996984882}) 2024-04-20T17:35:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3884814478}) 2024-04-20T17:35:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4007929067}) 2024-04-20T17:35:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3366238757}) 2024-04-20T17:35:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2535365673}) 2024-04-20T17:35:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3745611497}) 2024-04-20T17:35:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3715548958}) 2024-04-20T17:35:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=229461521}) 2024-04-20T17:35:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2715346916}) 2024-04-20T17:35:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2462647314}) 2024-04-20T17:35:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=603020674}) 2024-04-20T17:35:32.587 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2410540540}) 2024-04-20T17:35:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3179857386}) 2024-04-20T17:35:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=893770348}) 2024-04-20T17:35:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2076142414}) 2024-04-20T17:35:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2172440440}) 2024-04-20T17:35:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3609961106}) 2024-04-20T17:35:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1254403229}) 2024-04-20T17:35:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=673837400}) 2024-04-20T17:35:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1790337665}) 2024-04-20T17:35:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3575964297}) 2024-04-20T17:35:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3568488074}) 2024-04-20T17:35:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2225366857}) 2024-04-20T17:35:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=97870039}) 2024-04-20T17:35:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2656369641}) 2024-04-20T17:35:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3253436453}) 2024-04-20T17:35:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3268143116}) 2024-04-20T17:35:32.588 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3664642787}) 2024-04-20T17:35:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=373445224}) 2024-04-20T17:35:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2791680415}) 2024-04-20T17:35:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2539659063}) 2024-04-20T17:35:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=267190635}) 2024-04-20T17:35:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3430951115}) 2024-04-20T17:35:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=854195147}) 2024-04-20T17:35:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1163303384}) 2024-04-20T17:35:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4144597519}) 2024-04-20T17:35:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4247996299}) 2024-04-20T17:35:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3827011024}) 2024-04-20T17:35:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3607022389}) 2024-04-20T17:35:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3708441367}) 2024-04-20T17:35:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2885683501}) 2024-04-20T17:35:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=941457875}) 2024-04-20T17:35:32.589 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3596698742}) 2024-04-20T17:35:32.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1888086718}) 2024-04-20T17:35:32.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2352067988}) 2024-04-20T17:35:32.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=352630719}) 2024-04-20T17:35:32.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3802855961}) 2024-04-20T17:35:32.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1321155002}) 2024-04-20T17:35:32.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=91250645}) 2024-04-20T17:35:32.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=49,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=8,mon_config_key=19} crc {logm=1391422185,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=3342376838,mon_config_key=1377445984}) 2024-04-20T17:35:32.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3686027754}) 2024-04-20T17:35:32.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=504889292}) 2024-04-20T17:35:32.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=71,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=11} crc {mon_config_key=2414480156,monmap=717113318,osd_metadata=3196995629,osd_pg_creating=3541239294,osdmap=3111100669}) 2024-04-20T17:35:32.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2487762730}) 2024-04-20T17:35:32.590 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=2} crc {osdmap=3311460918}) 2024-04-20T17:35:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=59,logm=1} crc {auth=155901282,config=1460075751,health=179246142,kv=1447570618,logm=610821111}) 2024-04-20T17:35:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1720977174}) 2024-04-20T17:35:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3594143753}) 2024-04-20T17:35:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1502191774}) 2024-04-20T17:35:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=479696460}) 2024-04-20T17:35:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2999020662}) 2024-04-20T17:35:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4168962406}) 2024-04-20T17:35:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1010796833}) 2024-04-20T17:35:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2623010325}) 2024-04-20T17:35:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1390100058}) 2024-04-20T17:35:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3476045037}) 2024-04-20T17:35:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=903846406}) 2024-04-20T17:35:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1299556456}) 2024-04-20T17:35:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1377531572}) 2024-04-20T17:35:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2811759264}) 2024-04-20T17:35:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1338009965}) 2024-04-20T17:35:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1390316277}) 2024-04-20T17:35:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3382104207}) 2024-04-20T17:35:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=991013629}) 2024-04-20T17:35:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3771618271}) 2024-04-20T17:35:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3008462934}) 2024-04-20T17:35:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2523039836}) 2024-04-20T17:35:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1140141808}) 2024-04-20T17:35:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512649288}) 2024-04-20T17:35:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1967430481}) 2024-04-20T17:35:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2692641017}) 2024-04-20T17:35:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2572291463}) 2024-04-20T17:35:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2158081595}) 2024-04-20T17:35:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=996984882}) 2024-04-20T17:35:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3884814478}) 2024-04-20T17:35:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4007929067}) 2024-04-20T17:35:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3366238757}) 2024-04-20T17:35:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2535365673}) 2024-04-20T17:35:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3745611497}) 2024-04-20T17:35:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3715548958}) 2024-04-20T17:35:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=229461521}) 2024-04-20T17:35:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2715346916}) 2024-04-20T17:35:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2462647314}) 2024-04-20T17:35:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=603020674}) 2024-04-20T17:35:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2410540540}) 2024-04-20T17:35:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3179857386}) 2024-04-20T17:35:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=893770348}) 2024-04-20T17:35:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2076142414}) 2024-04-20T17:35:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2172440440}) 2024-04-20T17:35:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3609961106}) 2024-04-20T17:35:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1254403229}) 2024-04-20T17:35:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=673837400}) 2024-04-20T17:35:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1790337665}) 2024-04-20T17:35:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3575964297}) 2024-04-20T17:35:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3568488074}) 2024-04-20T17:35:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2225366857}) 2024-04-20T17:35:32.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=97870039}) 2024-04-20T17:35:32.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2656369641}) 2024-04-20T17:35:32.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3253436453}) 2024-04-20T17:35:32.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3268143116}) 2024-04-20T17:35:32.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3664642787}) 2024-04-20T17:35:32.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=373445224}) 2024-04-20T17:35:32.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2791680415}) 2024-04-20T17:35:32.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2539659063}) 2024-04-20T17:35:32.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=267190635}) 2024-04-20T17:35:32.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3430951115}) 2024-04-20T17:35:32.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=854195147}) 2024-04-20T17:35:32.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1163303384}) 2024-04-20T17:35:32.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4144597519}) 2024-04-20T17:35:32.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4247996299}) 2024-04-20T17:35:32.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3827011024}) 2024-04-20T17:35:32.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3607022389}) 2024-04-20T17:35:32.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3708441367}) 2024-04-20T17:35:32.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2885683501}) 2024-04-20T17:35:32.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=941457875}) 2024-04-20T17:35:32.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3596698742}) 2024-04-20T17:35:32.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1888086718}) 2024-04-20T17:35:32.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2352067988}) 2024-04-20T17:35:32.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=352630719}) 2024-04-20T17:35:32.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3802855961}) 2024-04-20T17:35:32.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1321155002}) 2024-04-20T17:35:32.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=91250645}) 2024-04-20T17:35:32.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=49,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=8,mon_config_key=19} crc {logm=1391422185,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=3342376838,mon_config_key=1377445984}) 2024-04-20T17:35:32.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3686027754}) 2024-04-20T17:35:32.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=504889292}) 2024-04-20T17:35:32.719 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=71,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=11} crc {mon_config_key=2414480156,monmap=717113318,osd_metadata=3196995629,osd_pg_creating=3541239294,osdmap=3111100669}) 2024-04-20T17:35:32.719 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2487762730}) 2024-04-20T17:35:32.719 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=2} crc {osdmap=3311460918}) 2024-04-20T17:35:32.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=59,logm=1} crc {auth=155901282,config=1460075751,health=179246142,kv=1447570618,logm=610821111}) 2024-04-20T17:35:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1720977174}) 2024-04-20T17:35:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3594143753}) 2024-04-20T17:35:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1502191774}) 2024-04-20T17:35:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=479696460}) 2024-04-20T17:35:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2999020662}) 2024-04-20T17:35:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4168962406}) 2024-04-20T17:35:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1010796833}) 2024-04-20T17:35:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2623010325}) 2024-04-20T17:35:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1390100058}) 2024-04-20T17:35:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3476045037}) 2024-04-20T17:35:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=903846406}) 2024-04-20T17:35:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1299556456}) 2024-04-20T17:35:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1377531572}) 2024-04-20T17:35:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2811759264}) 2024-04-20T17:35:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1338009965}) 2024-04-20T17:35:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1390316277}) 2024-04-20T17:35:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3382104207}) 2024-04-20T17:35:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=991013629}) 2024-04-20T17:35:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3771618271}) 2024-04-20T17:35:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3008462934}) 2024-04-20T17:35:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2523039836}) 2024-04-20T17:35:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1140141808}) 2024-04-20T17:35:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512649288}) 2024-04-20T17:35:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1967430481}) 2024-04-20T17:35:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2692641017}) 2024-04-20T17:35:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2572291463}) 2024-04-20T17:35:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2158081595}) 2024-04-20T17:35:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=996984882}) 2024-04-20T17:35:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3884814478}) 2024-04-20T17:35:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4007929067}) 2024-04-20T17:35:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3366238757}) 2024-04-20T17:35:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2535365673}) 2024-04-20T17:35:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3745611497}) 2024-04-20T17:35:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3715548958}) 2024-04-20T17:35:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=229461521}) 2024-04-20T17:35:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2715346916}) 2024-04-20T17:35:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2462647314}) 2024-04-20T17:35:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=603020674}) 2024-04-20T17:35:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2410540540}) 2024-04-20T17:35:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3179857386}) 2024-04-20T17:35:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=893770348}) 2024-04-20T17:35:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2076142414}) 2024-04-20T17:35:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2172440440}) 2024-04-20T17:35:32.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3609961106}) 2024-04-20T17:35:32.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1254403229}) 2024-04-20T17:35:32.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=673837400}) 2024-04-20T17:35:32.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1790337665}) 2024-04-20T17:35:32.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3575964297}) 2024-04-20T17:35:32.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3568488074}) 2024-04-20T17:35:32.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2225366857}) 2024-04-20T17:35:32.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=97870039}) 2024-04-20T17:35:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2656369641}) 2024-04-20T17:35:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3253436453}) 2024-04-20T17:35:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3268143116}) 2024-04-20T17:35:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3664642787}) 2024-04-20T17:35:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=373445224}) 2024-04-20T17:35:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2791680415}) 2024-04-20T17:35:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2539659063}) 2024-04-20T17:35:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=267190635}) 2024-04-20T17:35:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3430951115}) 2024-04-20T17:35:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=854195147}) 2024-04-20T17:35:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1163303384}) 2024-04-20T17:35:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4144597519}) 2024-04-20T17:35:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4247996299}) 2024-04-20T17:35:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3827011024}) 2024-04-20T17:35:32.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3607022389}) 2024-04-20T17:35:32.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3708441367}) 2024-04-20T17:35:32.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2885683501}) 2024-04-20T17:35:32.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=941457875}) 2024-04-20T17:35:32.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3596698742}) 2024-04-20T17:35:32.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1888086718}) 2024-04-20T17:35:32.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2352067988}) 2024-04-20T17:35:32.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=352630719}) 2024-04-20T17:35:32.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3802855961}) 2024-04-20T17:35:32.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1321155002}) 2024-04-20T17:35:32.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=91250645}) 2024-04-20T17:35:32.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=49,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=8,mon_config_key=19} crc {logm=1391422185,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=3342376838,mon_config_key=1377445984}) 2024-04-20T17:35:32.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3686027754}) 2024-04-20T17:35:32.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=504889292}) 2024-04-20T17:35:32.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=71,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=11} crc {mon_config_key=2414480156,monmap=717113318,osd_metadata=3196995629,osd_pg_creating=3541239294,osdmap=3111100669}) 2024-04-20T17:35:32.768 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2487762730}) 2024-04-20T17:35:32.768 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=2} crc {osdmap=3311460918}) 2024-04-20T17:35:32.807 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:35:32.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:32.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:35:32.910 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:35:32.917 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:35:32.946 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:33.102 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:35:33.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:33.106 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:35:33.207 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:35:33.241 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:33.360 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:35:33.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:33.365 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:35:33.408 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:35:33.408 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:35:33.443 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:35:33.465 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:35:33.500 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:33.521 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:35:33.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:33.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:35:33.625 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:35:33.676 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:33.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:33 smithi005 ceph-mon[39562]: pgmap v1727: 17 pgs: 17 active+clean; 8.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.04k op/s 2024-04-20T17:35:33.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:33 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4182587450' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:35:33.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:33 smithi005 ceph-mon[39562]: from='client.26884 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:35:33.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:33 smithi106 ceph-mon[44423]: pgmap v1727: 17 pgs: 17 active+clean; 8.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.04k op/s 2024-04-20T17:35:33.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:33 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4182587450' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:35:33.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:33 smithi106 ceph-mon[44423]: from='client.26884 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:35:33.769 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:33 smithi171 ceph-mon[44189]: pgmap v1727: 17 pgs: 17 active+clean; 8.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.04k op/s 2024-04-20T17:35:33.769 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:33 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4182587450' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:35:33.769 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:33 smithi171 ceph-mon[44189]: from='client.26884 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:35:34.108 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-20T17:35:34.449 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:35:34.650 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:35:34.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:34.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:35:34.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:34 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:35:34.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:34 smithi005 ceph-mon[39562]: from='client.26884 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:35:34.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:34 smithi005 ceph-mon[39562]: osdmap e655: 12 total, 11 up, 10 in 2024-04-20T17:35:34.706 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:34 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:35:34.706 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:34 smithi171 ceph-mon[44189]: from='client.26884 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:35:34.706 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:34 smithi171 ceph-mon[44189]: osdmap e655: 12 total, 11 up, 10 in 2024-04-20T17:35:34.756 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:35:34.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:34 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:35:34.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:34 smithi106 ceph-mon[44423]: from='client.26884 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:35:34.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:34 smithi106 ceph-mon[44423]: osdmap e655: 12 total, 11 up, 10 in 2024-04-20T17:35:34.841 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:35.298 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:35:35.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:35.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:35:35.404 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:35:35.436 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:35.483 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:35:35.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:35.494 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:35:35.595 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:35:35.658 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:35 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:35:35.659 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:35 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:35:35.659 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:35 smithi005 ceph-mon[39562]: pgmap v1729: 17 pgs: 17 active+clean; 8.2 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-20T17:35:35.659 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:35 smithi005 ceph-mon[39562]: osdmap e656: 12 total, 11 up, 10 in 2024-04-20T17:35:35.683 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:35 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:35:35.684 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:35 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:35:35.684 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:35 smithi171 ceph-mon[44189]: pgmap v1729: 17 pgs: 17 active+clean; 8.2 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-20T17:35:35.684 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:35 smithi171 ceph-mon[44189]: osdmap e656: 12 total, 11 up, 10 in 2024-04-20T17:35:35.686 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:35 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:35:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:35 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:35:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:35 smithi106 ceph-mon[44423]: pgmap v1729: 17 pgs: 17 active+clean; 8.2 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-20T17:35:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:35 smithi106 ceph-mon[44423]: osdmap e656: 12 total, 11 up, 10 in 2024-04-20T17:35:35.887 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:35:35.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:35.891 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:35:35.991 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:35:36.022 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:36.177 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:35:36.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:36.180 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:35:36.281 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:35:36.335 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:36.362 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:35:36.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:36.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:35:36.371 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:35:36.407 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:35:36.441 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:35:36.470 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:35:36.472 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:35:36.503 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:35:36.503 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:35:36.503 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:35:36.511 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:36.531 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:35:36.540 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:35:36.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:36.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:35:36.565 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:35:36.599 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:35:36.629 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:35:36.648 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:35:36.662 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:35:36.692 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:35:36.693 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:35:36.693 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-20T17:35:36.693 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:35:36.693 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:35:36.693 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:35:36.693 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:35:36.693 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:35:36.693 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:35:36.693 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:35:36.693 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:35:36.694 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:35:36.694 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:35:36.699 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:36 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2781065349' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:35:36.699 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:36.699 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:36 smithi005 ceph-mon[39562]: pgmap v1731: 17 pgs: 17 active+clean; 8.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.31k op/s 2024-04-20T17:35:36.700 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:36.739 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:35:36.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:36.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:35:36.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:36 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2781065349' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:35:36.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:36.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:36 smithi106 ceph-mon[44423]: pgmap v1731: 17 pgs: 17 active+clean; 8.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.31k op/s 2024-04-20T17:35:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:36 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2781065349' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:35:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:36 smithi171 ceph-mon[44189]: pgmap v1731: 17 pgs: 17 active+clean; 8.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.31k op/s 2024-04-20T17:35:36.846 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:35:36.911 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:36.943 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:35:36.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:36.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:35:37.048 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:35:37.091 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:37.119 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:35:37.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:37.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:35:37.226 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:35:37.263 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:37.286 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:35:37.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:37.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:35:37.323 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:35:37.390 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:35:37.396 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-20T17:35:37.433 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:37.457 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:35:37.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:37.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:35:37.554 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:35:37.555 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:35:37.561 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:35:37.592 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:37.615 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:35:37.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:37.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:35:37.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:37.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:37.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:37.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:37.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:37.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:37.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:37.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:37.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:37.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:37 smithi005 ceph-mon[39562]: 6.0 scrub starts 2024-04-20T17:35:37.719 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:35:37.736 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:35:37.746 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:37.746 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:37.746 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:37.746 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:37.746 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:37.746 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:37.746 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:37.746 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:37.746 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:37.746 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:37 smithi106 ceph-mon[44423]: 6.0 scrub starts 2024-04-20T17:35:37.748 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:37.775 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:35:37.778 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-04-20T17:35:37.778 INFO:tasks.cephadm.osd.1:Restarting osd.1 (starting--it wasn't running)... 2024-04-20T17:35:37.778 DEBUG:teuthology.orchestra.run.smithi005:osd.1> sudo journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.1.service 2024-04-20T17:35:37.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:37.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:35:37.784 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl start ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.1 2024-04-20T17:35:37.806 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:35:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:37.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:37.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:37.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:37.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:37 smithi171 ceph-mon[44189]: 6.0 scrub starts 2024-04-20T17:35:37.875 INFO:journalctl@ceph.osd.1.smithi005.stdout:-- Logs begin at Sat 2024-04-20 16:47:38 UTC. -- 2024-04-20T17:35:37.881 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:35:37.918 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:37.951 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:35:37.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:37.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:35:38.060 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:35:38.102 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:38.141 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:35:37 smithi005 systemd[1]: Starting Ceph osd.1 for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:35:38.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:38.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:38 smithi171 ceph-mon[44189]: pgmap v1732: 17 pgs: 17 active+clean; 8.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-20T17:35:38.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:38.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:38 smithi106 ceph-mon[44423]: pgmap v1732: 17 pgs: 17 active+clean; 8.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-20T17:35:38.961 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:35:38.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:38.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:38 smithi005 ceph-mon[39562]: pgmap v1732: 17 pgs: 17 active+clean; 8.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-20T17:35:38.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:38.967 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:35:39.068 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:35:39.128 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:39.555 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:35:39.643 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:35:39.643 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:35:39.643 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:35:39.643 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:35:39.643 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.644 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162338 4.3 4:c74ec0af:::benchmark_data_smithi171_122772_object162337:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:35:39.644 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:11.924493+0000", 2024-04-20T17:35:39.644 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 567.71622670099998, 2024-04-20T17:35:39.644 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.209723929, 2024-04-20T17:35:39.644 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:39.644 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:39.644 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:39.644 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:39.644 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:39.644 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:39.644 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162338 2024-04-20T17:35:39.644 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.644 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:39.644 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.645 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:39.645 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.924493+0000", 2024-04-20T17:35:39.645 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.645 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.645 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.645 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:39.645 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.924493+0000", 2024-04-20T17:35:39.645 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.645 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.645 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.645 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:39.645 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.924495+0000", 2024-04-20T17:35:39.645 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.575e-06 2024-04-20T17:35:39.646 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.646 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.646 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:39.646 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.924515+0000", 2024-04-20T17:35:39.646 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0489e-05 2024-04-20T17:35:39.646 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.646 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.646 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:39.646 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.924526+0000", 2024-04-20T17:35:39.646 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1309e-05 2024-04-20T17:35:39.646 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.646 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.646 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:39.646 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.924534+0000", 2024-04-20T17:35:39.647 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.25e-06 2024-04-20T17:35:39.647 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.647 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.647 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:39.647 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.926564+0000", 2024-04-20T17:35:39.647 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0020306550000000001 2024-04-20T17:35:39.647 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.647 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.647 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:39.647 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.926712+0000", 2024-04-20T17:35:39.647 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014766700000000001 2024-04-20T17:35:39.647 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.647 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.648 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:35:39.648 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.926862+0000", 2024-04-20T17:35:39.648 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014963500000000001 2024-04-20T17:35:39.648 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.648 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.648 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:39.648 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.932381+0000", 2024-04-20T17:35:39.648 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0055196949999999998 2024-04-20T17:35:39.648 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.648 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.648 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:39.648 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.134102+0000", 2024-04-20T17:35:39.648 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20172087 2024-04-20T17:35:39.649 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.649 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.649 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:39.649 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.134166+0000", 2024-04-20T17:35:39.649 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4243000000000001e-05 2024-04-20T17:35:39.649 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.649 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.649 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:39.649 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.134217+0000", 2024-04-20T17:35:39.649 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0541000000000003e-05 2024-04-20T17:35:39.649 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.649 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:39.649 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.649 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.649 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.650 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162350 4.8 4:193a143c:::benchmark_data_smithi171_122772_object162349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:35:39.650 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:11.931709+0000", 2024-04-20T17:35:39.650 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 567.709010638, 2024-04-20T17:35:39.650 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.22858086999999999, 2024-04-20T17:35:39.650 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:39.650 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:39.650 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:39.650 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:39.650 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:39.650 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:39.650 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162350 2024-04-20T17:35:39.650 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.651 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:39.651 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.651 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:39.651 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.931709+0000", 2024-04-20T17:35:39.651 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.651 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.651 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.651 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:39.651 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.931709+0000", 2024-04-20T17:35:39.651 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.651 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.652 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.652 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:39.652 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.931712+0000", 2024-04-20T17:35:39.652 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.92e-06 2024-04-20T17:35:39.652 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.652 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.652 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:39.652 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.931737+0000", 2024-04-20T17:35:39.652 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5106999999999999e-05 2024-04-20T17:35:39.652 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.652 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.652 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:39.652 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.931752+0000", 2024-04-20T17:35:39.652 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4891e-05 2024-04-20T17:35:39.652 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.653 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.653 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:39.653 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.931761+0000", 2024-04-20T17:35:39.653 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6340000000000007e-06 2024-04-20T17:35:39.653 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.653 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.653 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:39.653 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.931824+0000", 2024-04-20T17:35:39.653 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3019000000000001e-05 2024-04-20T17:35:39.653 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.653 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.653 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:39.653 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.931959+0000", 2024-04-20T17:35:39.653 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013509700000000001 2024-04-20T17:35:39.654 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.654 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.654 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:35:39.654 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.932081+0000", 2024-04-20T17:35:39.654 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000121919 2024-04-20T17:35:39.654 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.654 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.654 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:39.654 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937096+0000", 2024-04-20T17:35:39.654 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0050154980000000002 2024-04-20T17:35:39.654 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.654 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.654 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:39.655 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160167+0000", 2024-04-20T17:35:39.655 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.223070565 2024-04-20T17:35:39.655 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.655 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.655 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:39.655 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160239+0000", 2024-04-20T17:35:39.655 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2729000000000001e-05 2024-04-20T17:35:39.655 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.655 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.656 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:39.656 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160290+0000", 2024-04-20T17:35:39.656 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0491000000000002e-05 2024-04-20T17:35:39.656 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.656 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:39.656 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.656 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.656 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.656 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162353 4.8 4:1973aaac:::benchmark_data_smithi171_122772_object162352:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:35:39.656 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:11.937307+0000", 2024-04-20T17:35:39.656 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 567.70341249900002, 2024-04-20T17:35:39.656 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.22316846200000001, 2024-04-20T17:35:39.656 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:39.656 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:39.657 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:39.657 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:39.657 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:39.657 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:39.657 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162353 2024-04-20T17:35:39.657 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.657 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:39.657 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.657 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:39.657 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937307+0000", 2024-04-20T17:35:39.657 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.657 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.657 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.657 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:39.657 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937307+0000", 2024-04-20T17:35:39.658 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.658 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.658 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.658 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:39.658 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937308+0000", 2024-04-20T17:35:39.658 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0210000000000001e-06 2024-04-20T17:35:39.658 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.658 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.658 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:39.658 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937330+0000", 2024-04-20T17:35:39.658 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1678000000000001e-05 2024-04-20T17:35:39.658 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.658 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.658 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:39.659 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937337+0000", 2024-04-20T17:35:39.659 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4340000000000003e-06 2024-04-20T17:35:39.659 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.659 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.659 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:39.659 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937343+0000", 2024-04-20T17:35:39.659 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8599999999999998e-06 2024-04-20T17:35:39.659 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.659 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.659 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:39.659 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937396+0000", 2024-04-20T17:35:39.659 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3124999999999997e-05 2024-04-20T17:35:39.659 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.659 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.660 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:39.660 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937544+0000", 2024-04-20T17:35:39.660 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000147398 2024-04-20T17:35:39.660 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.660 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.660 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:35:39.660 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.937698+0000", 2024-04-20T17:35:39.660 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015431200000000001 2024-04-20T17:35:39.660 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.660 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.660 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:39.660 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.956740+0000", 2024-04-20T17:35:39.660 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019042416999999999 2024-04-20T17:35:39.660 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.661 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.661 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:39.661 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160359+0000", 2024-04-20T17:35:39.661 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20361854900000001 2024-04-20T17:35:39.661 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.661 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.661 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:39.661 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160419+0000", 2024-04-20T17:35:39.661 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9735999999999999e-05 2024-04-20T17:35:39.661 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.661 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.661 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:39.661 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160476+0000", 2024-04-20T17:35:39.661 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6932000000000003e-05 2024-04-20T17:35:39.662 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.662 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:39.662 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.662 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.662 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.662 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162360 4.8 4:12f91459:::benchmark_data_smithi171_122772_object162359:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:35:39.662 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:11.943537+0000", 2024-04-20T17:35:39.662 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 567.69718264999995, 2024-04-20T17:35:39.662 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.21790537500000001, 2024-04-20T17:35:39.662 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:39.662 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:39.662 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:39.663 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:39.663 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:39.663 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:39.663 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162360 2024-04-20T17:35:39.663 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.663 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:39.663 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.663 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:39.663 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.943537+0000", 2024-04-20T17:35:39.663 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.663 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.663 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.663 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:39.663 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.943537+0000", 2024-04-20T17:35:39.664 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.664 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.664 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.664 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:39.664 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.943539+0000", 2024-04-20T17:35:39.664 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.826e-06 2024-04-20T17:35:39.664 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.664 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.664 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:39.664 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.943561+0000", 2024-04-20T17:35:39.664 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2354999999999999e-05 2024-04-20T17:35:39.664 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.664 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.664 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:39.664 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.943573+0000", 2024-04-20T17:35:39.665 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1841999999999999e-05 2024-04-20T17:35:39.665 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.665 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.665 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:39.665 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.943581+0000", 2024-04-20T17:35:39.665 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1769999999999998e-06 2024-04-20T17:35:39.665 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.665 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.665 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:39.665 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.943637+0000", 2024-04-20T17:35:39.665 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5429000000000001e-05 2024-04-20T17:35:39.665 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.665 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.665 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:39.665 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.943771+0000", 2024-04-20T17:35:39.666 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000134027 2024-04-20T17:35:39.666 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.666 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.666 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:35:39.666 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.943930+0000", 2024-04-20T17:35:39.666 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015902900000000001 2024-04-20T17:35:39.666 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.666 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.666 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:39.666 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.956748+0000", 2024-04-20T17:35:39.666 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.012818016 2024-04-20T17:35:39.667 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.667 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.667 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:39.667 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.161353+0000", 2024-04-20T17:35:39.667 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20460482199999999 2024-04-20T17:35:39.667 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.667 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.667 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:39.667 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.161404+0000", 2024-04-20T17:35:39.667 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1439e-05 2024-04-20T17:35:39.667 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.667 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.667 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:39.667 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.161442+0000", 2024-04-20T17:35:39.668 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8413e-05 2024-04-20T17:35:39.668 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.668 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:39.668 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.668 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.668 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.668 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162361 4.8 4:19c34efa:::benchmark_data_smithi171_122772_object162360:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:35:39.668 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:11.946327+0000", 2024-04-20T17:35:39.668 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 567.69439267600001, 2024-04-20T17:35:39.668 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.21524243400000001, 2024-04-20T17:35:39.668 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:39.668 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:39.668 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:39.668 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:39.669 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:39.669 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:39.669 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162361 2024-04-20T17:35:39.669 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.669 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:39.669 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.669 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:39.669 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.946327+0000", 2024-04-20T17:35:39.669 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.669 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.669 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.669 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:39.669 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.946327+0000", 2024-04-20T17:35:39.669 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.670 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.670 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.670 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:39.670 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.946329+0000", 2024-04-20T17:35:39.670 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8470000000000001e-06 2024-04-20T17:35:39.670 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.670 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.670 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:39.670 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.946353+0000", 2024-04-20T17:35:39.670 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4491999999999999e-05 2024-04-20T17:35:39.670 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.670 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.670 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:39.670 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.946365+0000", 2024-04-20T17:35:39.671 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1688000000000001e-05 2024-04-20T17:35:39.671 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.671 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.671 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:39.671 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.946373+0000", 2024-04-20T17:35:39.671 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9049999999999997e-06 2024-04-20T17:35:39.671 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.671 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.671 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:39.672 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.947724+0000", 2024-04-20T17:35:39.672 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0013511409999999999 2024-04-20T17:35:39.672 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.672 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.672 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:39.672 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.947841+0000", 2024-04-20T17:35:39.672 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000116839 2024-04-20T17:35:39.672 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.672 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.672 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:35:39.672 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.947979+0000", 2024-04-20T17:35:39.672 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013786500000000001 2024-04-20T17:35:39.672 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.672 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.673 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:39.673 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.956751+0000", 2024-04-20T17:35:39.673 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0087720309999999996 2024-04-20T17:35:39.673 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.673 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.673 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:39.673 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.161495+0000", 2024-04-20T17:35:39.673 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20474413799999999 2024-04-20T17:35:39.673 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.673 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.673 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:39.673 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.161533+0000", 2024-04-20T17:35:39.673 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.82e-05 2024-04-20T17:35:39.674 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.674 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.674 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:39.674 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.161569+0000", 2024-04-20T17:35:39.674 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6288000000000002e-05 2024-04-20T17:35:39.674 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.674 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:39.674 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.674 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.674 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.675 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162362 4.8 4:15c10f34:::benchmark_data_smithi171_122772_object162361:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:35:39.675 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:11.947126+0000", 2024-04-20T17:35:39.675 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 567.69359374800001, 2024-04-20T17:35:39.675 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.224067406, 2024-04-20T17:35:39.675 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:39.675 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:39.675 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:39.675 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:39.675 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:39.675 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:39.675 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162362 2024-04-20T17:35:39.675 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.675 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:39.675 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.676 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:39.676 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.947126+0000", 2024-04-20T17:35:39.676 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.676 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.676 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.676 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:39.676 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.947126+0000", 2024-04-20T17:35:39.676 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.676 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.676 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.676 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:39.676 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.947127+0000", 2024-04-20T17:35:39.676 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5009999999999999e-06 2024-04-20T17:35:39.676 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.676 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.677 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:39.677 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.947149+0000", 2024-04-20T17:35:39.677 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1708e-05 2024-04-20T17:35:39.677 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.677 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.677 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:39.677 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.947159+0000", 2024-04-20T17:35:39.677 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0244999999999999e-05 2024-04-20T17:35:39.677 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.677 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.677 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:39.677 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.947166+0000", 2024-04-20T17:35:39.677 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4110000000000001e-06 2024-04-20T17:35:39.677 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.678 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.678 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:39.678 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.952667+0000", 2024-04-20T17:35:39.678 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0055010679999999996 2024-04-20T17:35:39.678 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.678 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.678 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:39.678 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.952814+0000", 2024-04-20T17:35:39.678 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014753499999999999 2024-04-20T17:35:39.678 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.678 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.678 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:35:39.678 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.952948+0000", 2024-04-20T17:35:39.679 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013382499999999999 2024-04-20T17:35:39.679 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.679 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.679 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:39.679 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.956887+0000", 2024-04-20T17:35:39.679 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0039385430000000001 2024-04-20T17:35:39.679 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.679 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.679 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:39.679 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.171085+0000", 2024-04-20T17:35:39.679 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.214198641 2024-04-20T17:35:39.679 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.679 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.680 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:39.680 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.171143+0000", 2024-04-20T17:35:39.680 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7720000000000003e-05 2024-04-20T17:35:39.680 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.680 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.680 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:39.680 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.171193+0000", 2024-04-20T17:35:39.680 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0209e-05 2024-04-20T17:35:39.680 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.680 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:39.680 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.680 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.680 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.680 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162374 4.3 4:c9021f98:::benchmark_data_smithi171_122772_object162373:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:35:39.680 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:11.952990+0000", 2024-04-20T17:35:39.681 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 567.68772930399996, 2024-04-20T17:35:39.681 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19610433199999999, 2024-04-20T17:35:39.681 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:39.681 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:39.681 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:39.681 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:39.681 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:39.681 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:39.681 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162374 2024-04-20T17:35:39.681 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.681 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:39.681 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.681 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:39.681 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.952990+0000", 2024-04-20T17:35:39.681 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.682 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.682 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.682 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:39.682 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.952990+0000", 2024-04-20T17:35:39.682 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.682 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.682 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.682 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:39.682 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.952992+0000", 2024-04-20T17:35:39.682 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7060000000000001e-06 2024-04-20T17:35:39.682 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.682 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.682 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:39.682 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.953016+0000", 2024-04-20T17:35:39.683 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4002000000000002e-05 2024-04-20T17:35:39.683 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.683 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.683 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:39.683 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.953026+0000", 2024-04-20T17:35:39.683 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0259e-05 2024-04-20T17:35:39.683 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.683 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.683 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:39.683 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.953034+0000", 2024-04-20T17:35:39.683 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7219999999999999e-06 2024-04-20T17:35:39.683 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.683 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.683 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:39.683 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.953097+0000", 2024-04-20T17:35:39.684 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2479999999999998e-05 2024-04-20T17:35:39.684 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.684 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.684 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:39.684 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.953175+0000", 2024-04-20T17:35:39.684 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8559000000000005e-05 2024-04-20T17:35:39.684 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.684 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.684 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:35:39.684 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.953274+0000", 2024-04-20T17:35:39.684 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9059000000000001e-05 2024-04-20T17:35:39.684 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.684 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.684 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:39.684 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.957017+0000", 2024-04-20T17:35:39.684 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.003742483 2024-04-20T17:35:39.685 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.685 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.685 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:39.685 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.149017+0000", 2024-04-20T17:35:39.685 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19200002899999999 2024-04-20T17:35:39.685 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.685 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.685 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:39.685 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.149053+0000", 2024-04-20T17:35:39.685 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5856999999999999e-05 2024-04-20T17:35:39.685 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.685 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.685 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:39.685 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.149095+0000", 2024-04-20T17:35:39.686 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2175999999999997e-05 2024-04-20T17:35:39.686 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.686 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:39.686 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.686 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.686 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.686 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:162397 4.3 4:c9a84e1e:::benchmark_data_smithi171_122772_object162396:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e484)", 2024-04-20T17:35:39.686 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:26:11.983263+0000", 2024-04-20T17:35:39.686 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 567.65745668099999, 2024-04-20T17:35:39.686 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.177048224, 2024-04-20T17:35:39.686 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:39.686 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:39.687 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:39.687 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:39.687 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:39.687 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:39.687 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 162397 2024-04-20T17:35:39.687 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.687 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:39.687 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.687 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:39.688 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.983263+0000", 2024-04-20T17:35:39.688 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.688 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.688 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.688 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:39.688 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.983263+0000", 2024-04-20T17:35:39.688 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.688 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.688 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.688 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:39.688 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.983266+0000", 2024-04-20T17:35:39.689 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.644e-06 2024-04-20T17:35:39.689 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.689 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.689 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:39.689 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.983288+0000", 2024-04-20T17:35:39.689 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2266999999999999e-05 2024-04-20T17:35:39.689 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.689 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.689 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:39.689 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.983302+0000", 2024-04-20T17:35:39.689 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4412e-05 2024-04-20T17:35:39.690 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.690 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.690 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:39.690 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.983310+0000", 2024-04-20T17:35:39.690 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7780000000000007e-06 2024-04-20T17:35:39.690 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.690 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.690 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:39.690 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.983374+0000", 2024-04-20T17:35:39.690 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3399999999999996e-05 2024-04-20T17:35:39.691 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.691 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.691 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:39.691 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.983521+0000", 2024-04-20T17:35:39.691 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014708399999999999 2024-04-20T17:35:39.691 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.691 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.691 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:35:39.691 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.983686+0000", 2024-04-20T17:35:39.691 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00016503000000000001 2024-04-20T17:35:39.691 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.691 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.691 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:39.692 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:11.988826+0000", 2024-04-20T17:35:39.692 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0051401069999999997 2024-04-20T17:35:39.692 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.692 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.692 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:39.692 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160218+0000", 2024-04-20T17:35:39.692 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.17139236599999999 2024-04-20T17:35:39.692 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.692 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.692 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:39.692 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160274+0000", 2024-04-20T17:35:39.692 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5739999999999998e-05 2024-04-20T17:35:39.692 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.692 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.692 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:39.693 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:26:12.160311+0000", 2024-04-20T17:35:39.693 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7395999999999999e-05 2024-04-20T17:35:39.693 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.693 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:39.693 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.693 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.693 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.693 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:250984 4.1 4:8e67478e:::benchmark_data_smithi171_122772_object80792:head [delete] snapc 0=[] ondisk+write+known_if_redirected e518)", 2024-04-20T17:35:39.693 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:27:50.635874+0000", 2024-04-20T17:35:39.693 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 469.00484596000001, 2024-04-20T17:35:39.693 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.23012532499999999, 2024-04-20T17:35:39.693 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:39.693 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:39.693 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:39.694 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:39.694 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:39.694 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:39.694 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 250984 2024-04-20T17:35:39.694 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.694 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:39.694 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.694 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:39.694 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.635874+0000", 2024-04-20T17:35:39.694 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.694 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.694 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.695 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:39.695 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.635874+0000", 2024-04-20T17:35:39.695 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.695 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.695 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.695 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:39.695 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.635875+0000", 2024-04-20T17:35:39.695 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6600000000000005e-07 2024-04-20T17:35:39.695 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.695 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.695 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:39.695 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.635878+0000", 2024-04-20T17:35:39.695 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3450000000000002e-06 2024-04-20T17:35:39.695 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.696 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.696 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:39.696 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.635884+0000", 2024-04-20T17:35:39.696 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1059999999999996e-06 2024-04-20T17:35:39.696 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.696 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.696 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:39.696 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.635890+0000", 2024-04-20T17:35:39.696 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.553e-06 2024-04-20T17:35:39.696 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.696 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.696 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:39.696 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.640085+0000", 2024-04-20T17:35:39.696 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.004194985 2024-04-20T17:35:39.696 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.697 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.697 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:39.697 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.640223+0000", 2024-04-20T17:35:39.697 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000138348 2024-04-20T17:35:39.697 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.697 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.697 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:35:39.697 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.640304+0000", 2024-04-20T17:35:39.697 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1385999999999996e-05 2024-04-20T17:35:39.697 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.697 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.697 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:39.697 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.646374+0000", 2024-04-20T17:35:39.697 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0060697379999999999 2024-04-20T17:35:39.698 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.698 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.698 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:39.698 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.865929+0000", 2024-04-20T17:35:39.698 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.219554844 2024-04-20T17:35:39.698 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.698 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.698 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:39.698 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.865960+0000", 2024-04-20T17:35:39.698 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1092e-05 2024-04-20T17:35:39.698 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.698 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.698 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:39.698 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.865999+0000", 2024-04-20T17:35:39.698 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9062e-05 2024-04-20T17:35:39.699 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.699 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:39.699 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.699 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.699 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.699 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:251049 4.1 4:80d2b251:::benchmark_data_smithi171_122772_object80857:head [delete] snapc 0=[] ondisk+write+known_if_redirected e518)", 2024-04-20T17:35:39.699 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:27:50.690715+0000", 2024-04-20T17:35:39.699 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 468.95000461500001, 2024-04-20T17:35:39.699 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.17539388, 2024-04-20T17:35:39.699 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:39.699 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:39.699 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:39.699 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:39.699 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:39.699 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:39.700 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 251049 2024-04-20T17:35:39.700 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.700 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:39.700 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.700 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:39.700 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.690715+0000", 2024-04-20T17:35:39.700 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.700 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.700 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.700 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:39.700 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.690715+0000", 2024-04-20T17:35:39.700 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.700 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.700 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.701 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:39.701 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.690718+0000", 2024-04-20T17:35:39.701 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5500000000000001e-06 2024-04-20T17:35:39.701 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.701 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.701 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:39.701 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.690724+0000", 2024-04-20T17:35:39.701 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6640000000000001e-06 2024-04-20T17:35:39.701 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.701 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.701 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:39.701 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.690737+0000", 2024-04-20T17:35:39.701 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3164e-05 2024-04-20T17:35:39.701 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.701 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.702 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:39.702 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.690746+0000", 2024-04-20T17:35:39.702 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.371e-06 2024-04-20T17:35:39.702 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.702 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.702 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:39.702 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.690817+0000", 2024-04-20T17:35:39.702 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0928999999999998e-05 2024-04-20T17:35:39.702 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.702 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.702 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:39.702 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.690943+0000", 2024-04-20T17:35:39.702 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012669600000000001 2024-04-20T17:35:39.702 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.703 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.703 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:35:39.703 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.691075+0000", 2024-04-20T17:35:39.703 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013154 2024-04-20T17:35:39.703 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.703 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.703 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:39.703 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.695316+0000", 2024-04-20T17:35:39.703 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0042411999999999997 2024-04-20T17:35:39.703 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.703 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.703 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:39.703 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.866028+0000", 2024-04-20T17:35:39.703 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.17071159899999999 2024-04-20T17:35:39.703 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.704 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.704 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:39.704 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.866060+0000", 2024-04-20T17:35:39.704 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2076999999999998e-05 2024-04-20T17:35:39.704 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.704 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.704 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:39.704 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:27:50.866109+0000", 2024-04-20T17:35:39.704 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9089999999999999e-05 2024-04-20T17:35:39.704 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.704 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:39.704 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.704 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.704 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.704 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:280492 4.8 4:1927ac75:::benchmark_data_smithi171_122772_object110300:head [delete] snapc 0=[] ondisk+write+known_if_redirected e521)", 2024-04-20T17:35:39.705 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:15.885627+0000", 2024-04-20T17:35:39.705 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 443.755092316, 2024-04-20T17:35:39.705 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.195642228, 2024-04-20T17:35:39.705 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:39.705 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:39.705 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:39.705 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:39.705 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:39.705 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:39.705 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 280492 2024-04-20T17:35:39.705 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.705 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:39.705 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.705 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:39.706 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.885627+0000", 2024-04-20T17:35:39.706 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.706 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.706 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.706 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:39.706 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.885627+0000", 2024-04-20T17:35:39.706 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.706 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.706 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.706 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:39.706 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.885629+0000", 2024-04-20T17:35:39.706 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7290000000000001e-06 2024-04-20T17:35:39.706 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.707 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.707 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:39.707 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.885634+0000", 2024-04-20T17:35:39.707 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5120000000000002e-06 2024-04-20T17:35:39.707 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.707 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.707 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:39.707 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.885642+0000", 2024-04-20T17:35:39.707 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8900000000000007e-06 2024-04-20T17:35:39.707 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.707 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.707 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:39.707 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.885647+0000", 2024-04-20T17:35:39.707 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2939999999999996e-06 2024-04-20T17:35:39.708 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.708 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.708 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:39.708 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.996163+0000", 2024-04-20T17:35:39.708 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.110515739 2024-04-20T17:35:39.708 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.708 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.708 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:39.708 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.996255+0000", 2024-04-20T17:35:39.708 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2628999999999997e-05 2024-04-20T17:35:39.708 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.708 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.708 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:35:39.708 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.996334+0000", 2024-04-20T17:35:39.708 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9103999999999993e-05 2024-04-20T17:35:39.709 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.709 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.709 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:39.709 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.015536+0000", 2024-04-20T17:35:39.709 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019201419000000001 2024-04-20T17:35:39.709 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.709 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.709 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:39.709 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.081190+0000", 2024-04-20T17:35:39.709 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.065653883999999996 2024-04-20T17:35:39.709 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.709 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.709 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:39.709 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.081230+0000", 2024-04-20T17:35:39.709 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0348000000000001e-05 2024-04-20T17:35:39.710 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.710 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.710 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:39.710 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.081270+0000", 2024-04-20T17:35:39.710 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.968e-05 2024-04-20T17:35:39.710 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.710 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:39.710 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.710 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.710 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.710 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:280495 4.8 4:122e2751:::benchmark_data_smithi171_122772_object110303:head [delete] snapc 0=[] ondisk+write+known_if_redirected e521)", 2024-04-20T17:35:39.710 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:15.891104+0000", 2024-04-20T17:35:39.710 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 443.74961566399998, 2024-04-20T17:35:39.710 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19031331200000001, 2024-04-20T17:35:39.711 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:39.711 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:39.711 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:39.711 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:39.711 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:39.711 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:39.711 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 280495 2024-04-20T17:35:39.711 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.711 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:39.711 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.711 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:39.711 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.891104+0000", 2024-04-20T17:35:39.711 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.711 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.712 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.712 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:39.712 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.891104+0000", 2024-04-20T17:35:39.712 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.712 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.712 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.712 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:39.712 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.891106+0000", 2024-04-20T17:35:39.712 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.429e-06 2024-04-20T17:35:39.712 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.712 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.712 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:39.712 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.891112+0000", 2024-04-20T17:35:39.712 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0869999999999997e-06 2024-04-20T17:35:39.712 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.713 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.713 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:39.713 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.891122+0000", 2024-04-20T17:35:39.713 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0141e-05 2024-04-20T17:35:39.713 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.713 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.713 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:39.713 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:15.891129+0000", 2024-04-20T17:35:39.713 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2699999999999999e-06 2024-04-20T17:35:39.713 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.713 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.713 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:39.713 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.001431+0000", 2024-04-20T17:35:39.713 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.110301732 2024-04-20T17:35:39.713 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.714 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.714 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:39.714 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.001510+0000", 2024-04-20T17:35:39.714 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9711999999999998e-05 2024-04-20T17:35:39.714 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.714 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.714 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:35:39.714 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.001610+0000", 2024-04-20T17:35:39.714 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9686999999999996e-05 2024-04-20T17:35:39.714 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.714 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.714 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:39.714 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.015543+0000", 2024-04-20T17:35:39.714 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013933332999999999 2024-04-20T17:35:39.715 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.715 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.715 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:39.715 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.081324+0000", 2024-04-20T17:35:39.715 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.065780901000000003 2024-04-20T17:35:39.715 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.715 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.715 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:39.715 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.081365+0000", 2024-04-20T17:35:39.715 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1131999999999999e-05 2024-04-20T17:35:39.715 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.715 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.715 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:39.715 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.081417+0000", 2024-04-20T17:35:39.716 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1888000000000002e-05 2024-04-20T17:35:39.716 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.716 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:39.716 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.716 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.716 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.716 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:280571 4.8 4:157d77de:::benchmark_data_smithi171_122772_object110379:head [delete] snapc 0=[] ondisk+write+known_if_redirected e521)", 2024-04-20T17:35:39.716 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:16.010716+0000", 2024-04-20T17:35:39.716 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 443.630003364, 2024-04-20T17:35:39.716 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.178831402, 2024-04-20T17:35:39.716 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:39.716 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:39.716 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:39.716 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:39.716 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:39.717 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:39.717 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 280571 2024-04-20T17:35:39.717 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.717 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:39.717 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.717 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:39.717 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.010716+0000", 2024-04-20T17:35:39.717 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.717 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.717 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.717 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:39.717 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.010716+0000", 2024-04-20T17:35:39.717 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.717 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.718 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.718 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:39.718 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.010717+0000", 2024-04-20T17:35:39.718 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.057e-06 2024-04-20T17:35:39.718 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.718 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.718 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:39.718 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.010722+0000", 2024-04-20T17:35:39.718 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4499999999999997e-06 2024-04-20T17:35:39.718 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.718 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.718 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:39.718 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.010730+0000", 2024-04-20T17:35:39.718 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6620000000000005e-06 2024-04-20T17:35:39.719 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.719 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.719 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:39.719 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.010735+0000", 2024-04-20T17:35:39.719 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9229999999999999e-06 2024-04-20T17:35:39.719 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.719 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.719 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:39.719 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.015594+0000", 2024-04-20T17:35:39.719 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0048587580000000003 2024-04-20T17:35:39.719 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.719 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.719 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:39.719 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.015723+0000", 2024-04-20T17:35:39.719 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012861 2024-04-20T17:35:39.720 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.720 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.720 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:35:39.720 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.015847+0000", 2024-04-20T17:35:39.720 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012468700000000001 2024-04-20T17:35:39.720 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.720 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.720 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:39.720 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.023498+0000", 2024-04-20T17:35:39.720 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0076504340000000002 2024-04-20T17:35:39.720 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.720 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.720 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:39.720 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.189467+0000", 2024-04-20T17:35:39.720 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16596942100000001 2024-04-20T17:35:39.721 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.721 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.721 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:39.721 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.189503+0000", 2024-04-20T17:35:39.721 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5531999999999998e-05 2024-04-20T17:35:39.721 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.721 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.721 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:39.721 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.189548+0000", 2024-04-20T17:35:39.721 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4867999999999998e-05 2024-04-20T17:35:39.721 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.721 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:39.721 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.721 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.722 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.722 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:280572 4.8 4:10b6d45a:::benchmark_data_smithi171_122772_object110380:head [delete] snapc 0=[] ondisk+write+known_if_redirected e521)", 2024-04-20T17:35:39.722 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:16.010744+0000", 2024-04-20T17:35:39.722 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 443.62997567299999, 2024-04-20T17:35:39.722 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.17898547200000001, 2024-04-20T17:35:39.722 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:39.722 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:39.722 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:39.722 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:39.722 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:39.722 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:39.722 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 280572 2024-04-20T17:35:39.722 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.722 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:39.722 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.723 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:39.723 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.010744+0000", 2024-04-20T17:35:39.723 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.723 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.723 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.723 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:39.723 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.010744+0000", 2024-04-20T17:35:39.723 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.723 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.723 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.723 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:39.723 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.010745+0000", 2024-04-20T17:35:39.723 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4499999999999996e-07 2024-04-20T17:35:39.723 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.723 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.724 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:39.724 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.010748+0000", 2024-04-20T17:35:39.724 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0309999999999999e-06 2024-04-20T17:35:39.724 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.724 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.724 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:39.724 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.010753+0000", 2024-04-20T17:35:39.724 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5690000000000004e-06 2024-04-20T17:35:39.724 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.724 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.724 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:39.724 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.010758+0000", 2024-04-20T17:35:39.724 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1109999999999999e-06 2024-04-20T17:35:39.724 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.724 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.725 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:39.725 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.019864+0000", 2024-04-20T17:35:39.725 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0091055289999999994 2024-04-20T17:35:39.725 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.725 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.725 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:39.725 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.019943+0000", 2024-04-20T17:35:39.725 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8638000000000002e-05 2024-04-20T17:35:39.725 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.725 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.725 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:35:39.725 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.020025+0000", 2024-04-20T17:35:39.725 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1961999999999999e-05 2024-04-20T17:35:39.725 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.725 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.726 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:39.726 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.023639+0000", 2024-04-20T17:35:39.726 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0036145470000000001 2024-04-20T17:35:39.726 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.726 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.726 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:39.726 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.189605+0000", 2024-04-20T17:35:39.726 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.165965522 2024-04-20T17:35:39.726 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.726 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.726 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:39.726 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.189668+0000", 2024-04-20T17:35:39.726 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2952000000000003e-05 2024-04-20T17:35:39.726 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.727 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.727 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:39.727 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.189730+0000", 2024-04-20T17:35:39.727 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1865999999999993e-05 2024-04-20T17:35:39.727 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.727 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:39.727 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.727 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.727 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.727 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:280618 4.8 4:1aaeafa9:::benchmark_data_smithi171_122772_object110426:head [delete] snapc 0=[] ondisk+write+known_if_redirected e521)", 2024-04-20T17:35:39.727 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:16.102539+0000", 2024-04-20T17:35:39.727 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 443.538180528, 2024-04-20T17:35:39.727 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16693295799999999, 2024-04-20T17:35:39.727 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:39.727 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:39.728 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:39.728 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:39.728 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:39.728 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:39.728 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 280618 2024-04-20T17:35:39.728 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.728 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:39.728 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.728 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:39.728 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.102539+0000", 2024-04-20T17:35:39.728 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.728 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.728 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.728 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:39.728 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.102539+0000", 2024-04-20T17:35:39.729 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.729 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.729 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.729 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:39.729 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.102541+0000", 2024-04-20T17:35:39.729 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.153e-06 2024-04-20T17:35:39.729 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.729 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.729 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:39.729 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.102548+0000", 2024-04-20T17:35:39.729 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.37e-06 2024-04-20T17:35:39.729 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.729 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.729 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:39.730 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.102560+0000", 2024-04-20T17:35:39.730 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2442e-05 2024-04-20T17:35:39.730 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.730 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.730 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:39.730 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.102568+0000", 2024-04-20T17:35:39.730 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5229999999999998e-06 2024-04-20T17:35:39.730 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.730 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.730 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:39.730 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.191248+0000", 2024-04-20T17:35:39.730 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.088680160999999993 2024-04-20T17:35:39.730 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.730 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.730 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:39.731 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.191364+0000", 2024-04-20T17:35:39.731 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011568799999999999 2024-04-20T17:35:39.731 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.731 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.731 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:35:39.731 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.191480+0000", 2024-04-20T17:35:39.731 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000116234 2024-04-20T17:35:39.731 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.731 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.731 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:39.731 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.211071+0000", 2024-04-20T17:35:39.731 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019591377 2024-04-20T17:35:39.731 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.731 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.732 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:39.732 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.269349+0000", 2024-04-20T17:35:39.732 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.058277429999999998 2024-04-20T17:35:39.732 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.732 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.732 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:39.732 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.269410+0000", 2024-04-20T17:35:39.732 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1648000000000003e-05 2024-04-20T17:35:39.732 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.732 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.732 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:39.732 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:16.269472+0000", 2024-04-20T17:35:39.732 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1932000000000003e-05 2024-04-20T17:35:39.732 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.732 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:39.733 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.733 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.733 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.733 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:281568 4.8 4:1fa3e07e:::benchmark_data_smithi171_122772_object111376:head [delete] snapc 0=[] ondisk+write+known_if_redirected e521)", 2024-04-20T17:35:39.733 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:17.475059+0000", 2024-04-20T17:35:39.733 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 442.16566027900001, 2024-04-20T17:35:39.733 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.17411728200000001, 2024-04-20T17:35:39.733 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:39.733 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:39.733 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:39.733 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:39.733 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:39.733 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:39.733 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 281568 2024-04-20T17:35:39.733 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.734 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:39.734 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.734 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:39.734 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.475059+0000", 2024-04-20T17:35:39.734 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.734 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.734 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.734 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:39.734 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.475059+0000", 2024-04-20T17:35:39.734 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.734 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.734 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.734 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:39.734 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.475060+0000", 2024-04-20T17:35:39.734 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.3300000000000001e-07 2024-04-20T17:35:39.735 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.735 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.735 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:39.735 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.475064+0000", 2024-04-20T17:35:39.735 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3100000000000001e-06 2024-04-20T17:35:39.735 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.735 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.735 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:39.735 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.475070+0000", 2024-04-20T17:35:39.735 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9e-06 2024-04-20T17:35:39.735 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.735 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.735 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:39.735 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.475077+0000", 2024-04-20T17:35:39.735 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0859999999999998e-06 2024-04-20T17:35:39.736 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.736 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.736 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:39.736 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.563751+0000", 2024-04-20T17:35:39.736 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.088674175999999993 2024-04-20T17:35:39.736 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.736 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.736 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:39.736 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.563852+0000", 2024-04-20T17:35:39.736 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010081000000000001 2024-04-20T17:35:39.736 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.736 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.736 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:35:39.736 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.563955+0000", 2024-04-20T17:35:39.736 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000103577 2024-04-20T17:35:39.737 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.737 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.737 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:39.737 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.577489+0000", 2024-04-20T17:35:39.737 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013533991 2024-04-20T17:35:39.737 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.737 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.737 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:39.737 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.649060+0000", 2024-04-20T17:35:39.737 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.071571368999999996 2024-04-20T17:35:39.737 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.737 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.737 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:39.737 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.649115+0000", 2024-04-20T17:35:39.737 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4540999999999998e-05 2024-04-20T17:35:39.738 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.738 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.738 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:39.738 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:17.649177+0000", 2024-04-20T17:35:39.738 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1688999999999998e-05 2024-04-20T17:35:39.738 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.738 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:39.738 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.738 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.738 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.738 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:282370 4.8 4:113c9101:::benchmark_data_smithi171_122772_object112178:head [delete] snapc 0=[] ondisk+write+known_if_redirected e521)", 2024-04-20T17:35:39.738 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:18.673315+0000", 2024-04-20T17:35:39.738 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 440.96740484399999, 2024-04-20T17:35:39.738 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16601888300000001, 2024-04-20T17:35:39.738 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:39.739 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:39.739 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:39.739 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:39.739 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:39.739 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:39.739 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 282370 2024-04-20T17:35:39.739 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.739 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:39.739 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.739 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:39.739 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:18.673315+0000", 2024-04-20T17:35:39.739 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.739 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.739 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.739 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:39.740 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:18.673315+0000", 2024-04-20T17:35:39.740 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.740 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.740 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.740 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:39.740 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:18.673317+0000", 2024-04-20T17:35:39.740 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.063e-06 2024-04-20T17:35:39.740 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.740 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.740 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:39.740 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:18.673322+0000", 2024-04-20T17:35:39.740 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5319999999999997e-06 2024-04-20T17:35:39.740 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.740 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.741 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:39.741 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:18.673332+0000", 2024-04-20T17:35:39.741 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7249999999999992e-06 2024-04-20T17:35:39.741 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.741 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.741 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:39.741 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:18.673338+0000", 2024-04-20T17:35:39.741 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6790000000000002e-06 2024-04-20T17:35:39.741 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.741 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.741 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:39.741 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:18.750016+0000", 2024-04-20T17:35:39.741 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.076678272000000006 2024-04-20T17:35:39.741 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.742 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.742 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:39.742 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:18.750138+0000", 2024-04-20T17:35:39.742 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000121966 2024-04-20T17:35:39.742 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.742 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.742 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:35:39.742 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:18.750279+0000", 2024-04-20T17:35:39.742 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000140956 2024-04-20T17:35:39.742 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.742 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.742 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:39.742 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:18.757772+0000", 2024-04-20T17:35:39.742 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0074926599999999999 2024-04-20T17:35:39.743 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.743 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.743 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:39.743 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:18.839228+0000", 2024-04-20T17:35:39.743 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.081456732000000004 2024-04-20T17:35:39.743 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.743 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.743 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:39.743 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:18.839282+0000", 2024-04-20T17:35:39.743 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3358e-05 2024-04-20T17:35:39.743 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.743 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.743 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:39.743 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:18.839334+0000", 2024-04-20T17:35:39.743 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1940000000000001e-05 2024-04-20T17:35:39.744 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.744 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:39.744 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.744 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.744 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.744 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:283096 4.8 4:1ed17a90:::benchmark_data_smithi171_122772_object112904:head [delete] snapc 0=[] ondisk+write+known_if_redirected e521)", 2024-04-20T17:35:39.744 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:19.770227+0000", 2024-04-20T17:35:39.744 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 439.87049259899999, 2024-04-20T17:35:39.744 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.175826486, 2024-04-20T17:35:39.744 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:39.744 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:39.744 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:39.744 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:39.744 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:39.745 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:39.745 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 283096 2024-04-20T17:35:39.745 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.745 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:39.745 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.745 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:39.745 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.770227+0000", 2024-04-20T17:35:39.745 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.745 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.745 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.745 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:39.745 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.770227+0000", 2024-04-20T17:35:39.745 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.745 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.745 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.746 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:39.746 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.770229+0000", 2024-04-20T17:35:39.746 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3520000000000001e-06 2024-04-20T17:35:39.746 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.746 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.746 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:39.746 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.770235+0000", 2024-04-20T17:35:39.746 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6389999999999997e-06 2024-04-20T17:35:39.746 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.746 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.746 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:39.746 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.770249+0000", 2024-04-20T17:35:39.746 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3909e-05 2024-04-20T17:35:39.746 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.747 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.747 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:39.747 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.770257+0000", 2024-04-20T17:35:39.747 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2029999999999992e-06 2024-04-20T17:35:39.747 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.747 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.747 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:39.747 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.866656+0000", 2024-04-20T17:35:39.747 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.096398567000000004 2024-04-20T17:35:39.747 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.747 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.747 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:39.747 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.866782+0000", 2024-04-20T17:35:39.747 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012599599999999999 2024-04-20T17:35:39.747 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.748 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.748 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:35:39.748 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.866908+0000", 2024-04-20T17:35:39.748 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000126372 2024-04-20T17:35:39.748 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.748 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.748 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:39.748 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.875989+0000", 2024-04-20T17:35:39.748 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0090805780000000006 2024-04-20T17:35:39.748 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.748 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.748 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:39.748 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.945960+0000", 2024-04-20T17:35:39.748 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.069971729999999996 2024-04-20T17:35:39.748 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.749 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.749 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:39.749 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.945995+0000", 2024-04-20T17:35:39.749 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4789999999999997e-05 2024-04-20T17:35:39.749 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.749 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.749 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:39.749 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.946054+0000", 2024-04-20T17:35:39.749 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8350000000000002e-05 2024-04-20T17:35:39.749 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.749 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:39.749 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.749 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.749 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.749 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:283104 4.8 4:15144f8b:::benchmark_data_smithi171_122772_object112912:head [delete] snapc 0=[] ondisk+write+known_if_redirected e521)", 2024-04-20T17:35:39.750 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:19.779189+0000", 2024-04-20T17:35:39.750 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 439.861530883, 2024-04-20T17:35:39.750 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.17012118200000001, 2024-04-20T17:35:39.750 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:39.750 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:39.750 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:39.750 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:39.750 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:39.750 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:39.750 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 283104 2024-04-20T17:35:39.750 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.750 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:39.750 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.750 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:39.750 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.779189+0000", 2024-04-20T17:35:39.751 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.751 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.751 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.751 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:39.751 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.779189+0000", 2024-04-20T17:35:39.751 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.751 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.751 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.751 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:39.751 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.779191+0000", 2024-04-20T17:35:39.751 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.26e-06 2024-04-20T17:35:39.751 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.751 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.751 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:39.751 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.779197+0000", 2024-04-20T17:35:39.752 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8449999999999999e-06 2024-04-20T17:35:39.752 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.752 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.752 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:39.752 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.779217+0000", 2024-04-20T17:35:39.752 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0452999999999999e-05 2024-04-20T17:35:39.752 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.752 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.752 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:39.752 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.779224+0000", 2024-04-20T17:35:39.752 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3350000000000003e-06 2024-04-20T17:35:39.752 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.752 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.752 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:39.752 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.870907+0000", 2024-04-20T17:35:39.753 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.091683095000000006 2024-04-20T17:35:39.753 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.753 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.753 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:39.753 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.871024+0000", 2024-04-20T17:35:39.753 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011728800000000001 2024-04-20T17:35:39.753 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.753 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.753 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:35:39.753 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.871142+0000", 2024-04-20T17:35:39.753 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011751 2024-04-20T17:35:39.753 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.753 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.753 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:39.753 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.880380+0000", 2024-04-20T17:35:39.754 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0092385160000000004 2024-04-20T17:35:39.754 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.754 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.754 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:39.754 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.949228+0000", 2024-04-20T17:35:39.754 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.068848030000000005 2024-04-20T17:35:39.754 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.754 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.754 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:39.754 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.949266+0000", 2024-04-20T17:35:39.754 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7753999999999998e-05 2024-04-20T17:35:39.754 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.754 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.754 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:39.754 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.949310+0000", 2024-04-20T17:35:39.755 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4095999999999999e-05 2024-04-20T17:35:39.755 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.755 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:39.755 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.755 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.755 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.755 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26074.0:283112 4.8 4:1f3b1708:::benchmark_data_smithi171_122772_object112920:head [delete] snapc 0=[] ondisk+write+known_if_redirected e521)", 2024-04-20T17:35:39.755 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:28:19.789454+0000", 2024-04-20T17:35:39.755 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 439.85126614000001, 2024-04-20T17:35:39.755 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16605162300000001, 2024-04-20T17:35:39.755 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:35:39.755 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:35:39.755 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:35:39.755 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:35:39.755 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26074", 2024-04-20T17:35:39.756 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2086562157", 2024-04-20T17:35:39.756 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 283112 2024-04-20T17:35:39.756 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.756 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:35:39.756 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.756 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:35:39.756 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.789454+0000", 2024-04-20T17:35:39.756 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.756 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.756 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.756 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:35:39.756 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.789454+0000", 2024-04-20T17:35:39.756 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:35:39.756 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.756 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.757 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:35:39.757 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.789456+0000", 2024-04-20T17:35:39.757 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.723e-06 2024-04-20T17:35:39.757 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.757 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.757 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:35:39.757 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.789464+0000", 2024-04-20T17:35:39.757 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7519999999999996e-06 2024-04-20T17:35:39.757 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.757 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.757 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:35:39.757 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.789479+0000", 2024-04-20T17:35:39.757 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4479e-05 2024-04-20T17:35:39.757 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.758 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.758 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:35:39.758 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.789488+0000", 2024-04-20T17:35:39.758 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9380000000000004e-06 2024-04-20T17:35:39.758 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.758 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.758 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:35:39.758 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.876116+0000", 2024-04-20T17:35:39.758 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.086627351000000005 2024-04-20T17:35:39.758 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.758 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.758 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:35:39.758 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.876234+0000", 2024-04-20T17:35:39.758 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000118609 2024-04-20T17:35:39.758 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.759 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.759 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 7", 2024-04-20T17:35:39.759 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.876342+0000", 2024-04-20T17:35:39.759 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000107088 2024-04-20T17:35:39.759 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.759 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.759 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:35:39.759 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.897090+0000", 2024-04-20T17:35:39.759 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020748860000000001 2024-04-20T17:35:39.759 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.759 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.759 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:35:39.759 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.955402+0000", 2024-04-20T17:35:39.759 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.058311363999999997 2024-04-20T17:35:39.759 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.760 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.760 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:35:39.760 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.955448+0000", 2024-04-20T17:35:39.760 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6612e-05 2024-04-20T17:35:39.760 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:35:39.760 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:35:39.760 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:35:39.760 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:28:19.955505+0000", 2024-04-20T17:35:39.760 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6847e-05 2024-04-20T17:35:39.760 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.760 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:39.760 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.760 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:35:39.760 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:35:39.760 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:35:39.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:39 smithi106 ceph-mon[44423]: 6.0 scrub ok 2024-04-20T17:35:39.761 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:39 smithi171 ceph-mon[44189]: 6.0 scrub ok 2024-04-20T17:35:39.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:39.762 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:35:39.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:35:39.791 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:39 smithi005 ceph-mon[39562]: 6.0 scrub ok 2024-04-20T17:35:39.863 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:35:39.895 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:39.920 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:35:39.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:39.925 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:35:40.027 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:35:40.077 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:40.190 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:35:40.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:40.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:35:40.298 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:35:40.328 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:40.403 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:35:40.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:40.407 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:35:40.433 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-20T17:35:40.508 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:35:40.582 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:40.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:40 smithi171 ceph-mon[44189]: pgmap v1733: 17 pgs: 17 active+clean; 7.8 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 1.35k op/s 2024-04-20T17:35:40.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:40 smithi005 ceph-mon[39562]: pgmap v1733: 17 pgs: 17 active+clean; 7.8 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 1.35k op/s 2024-04-20T17:35:40.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:40 smithi106 ceph-mon[44423]: pgmap v1733: 17 pgs: 17 active+clean; 7.8 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 1.35k op/s 2024-04-20T17:35:41.462 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:35:41 smithi005 bash[271602]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T17:35:41.462 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:35:41 smithi005 bash[271602]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-20T17:35:41.462 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:35:41 smithi005 bash[271602]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-20T17:35:41.462 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:35:41 smithi005 bash[271602]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-20T17:35:41.462 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:35:41 smithi005 bash[271602]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-20T17:35:41.462 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:35:41 smithi005 bash[271602]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-20T17:35:41.462 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:35:41 smithi005 bash[271602]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-20T17:35:41.462 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:35:41 smithi005 bash[271602]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-20T17:35:41.874 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:35:41.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:41.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:35:41.980 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:35:42.018 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:42.038 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:35:42.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:42.043 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:35:42.144 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:35:42.176 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:42.200 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:35:42.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:42.206 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:35:42.306 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:35:42.361 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:42.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:42 smithi005 ceph-mon[39562]: pgmap v1734: 17 pgs: 17 active+clean; 7.7 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-20T17:35:42.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:42 smithi106 ceph-mon[44423]: pgmap v1734: 17 pgs: 17 active+clean; 7.7 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-20T17:35:42.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:42 smithi171 ceph-mon[44189]: pgmap v1734: 17 pgs: 17 active+clean; 7.7 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-20T17:35:42.924 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:35:42.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:42.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:35:43.031 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:35:43.070 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:43.100 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:35:43.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:43.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:35:43.208 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:35:43.237 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:43.262 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:35:43.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:43.266 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:35:43.285 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:35:43.290 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:35:43.367 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:35:43.412 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:43.434 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:35:43.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:43.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:35:43.539 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:35:43.603 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:43.676 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:35:43.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:43.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:35:43.780 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:35:43.781 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:35:43.863 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:43 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3939807179' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:35:43.872 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:44.059 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-20T17:35:44.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:43 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3939807179' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:35:44.203 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:35:44 smithi005 bash[272163]: cdf9fe1990bb0bf4e7bef5d8dc49761c4afde5f8dd52e38d6b49b79440b28490 2024-04-20T17:35:44.203 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:35:44 smithi005 systemd[1]: Started Ceph osd.1 for 13c6e76c-ff37-11ee-bc93-c7b262605968. 2024-04-20T17:35:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:43 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3939807179' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:35:44.788 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:35:44 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-1[272232]: 2024-04-20T17:35:44.521+0000 7fdfc75b87c0 -1 Falling back to public interface 2024-04-20T17:35:45.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:44 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:35:45.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:44 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:35:45.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:44 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3939807179' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:35:45.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:44 smithi171 ceph-mon[44189]: osdmap e657: 12 total, 11 up, 10 in 2024-04-20T17:35:45.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:44 smithi171 ceph-mon[44189]: pgmap v1736: 17 pgs: 17 active+clean; 7.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-20T17:35:45.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:44 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:35:45.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:44 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:35:45.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:44 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3939807179' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:35:45.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:44 smithi005 ceph-mon[39562]: osdmap e657: 12 total, 11 up, 10 in 2024-04-20T17:35:45.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:44 smithi005 ceph-mon[39562]: pgmap v1736: 17 pgs: 17 active+clean; 7.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-20T17:35:45.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:44 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:35:45.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:44 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:35:45.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:44 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3939807179' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:35:45.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:44 smithi106 ceph-mon[44423]: osdmap e657: 12 total, 11 up, 10 in 2024-04-20T17:35:45.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:44 smithi106 ceph-mon[44423]: pgmap v1736: 17 pgs: 17 active+clean; 7.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 1.21k op/s 2024-04-20T17:35:45.448 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:35:45.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:45.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:35:45.561 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:35:45.592 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:45.634 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:35:45.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:45.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:35:45.741 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:35:45.772 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:45.798 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:35:45.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:45.804 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:35:45.905 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:35:45.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:45 smithi005 ceph-mon[39562]: 6.3 scrub starts 2024-04-20T17:35:45.958 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:46.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:45 smithi106 ceph-mon[44423]: 6.3 scrub starts 2024-04-20T17:35:46.051 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:35:46.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:46.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:35:46.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:45 smithi171 ceph-mon[44189]: 6.3 scrub starts 2024-04-20T17:35:46.157 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:35:46.186 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:46.212 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:35:46.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:46.216 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:35:46.317 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:35:46.368 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:46.409 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:35:46.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:46.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:35:46.515 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:35:46.555 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:46.587 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:35:46.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:46.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:35:46.696 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:35:46.762 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:46.797 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:46 smithi005 ceph-mon[39562]: pgmap v1737: 17 pgs: 17 active+clean; 7.4 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-20T17:35:46.797 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:46 smithi005 ceph-mon[39562]: 6.3 scrub ok 2024-04-20T17:35:46.952 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:35:46.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:46.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:35:46.985 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:35:47.062 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:35:47.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:46 smithi171 ceph-mon[44189]: pgmap v1737: 17 pgs: 17 active+clean; 7.4 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-20T17:35:47.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:46 smithi171 ceph-mon[44189]: 6.3 scrub ok 2024-04-20T17:35:47.094 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:47.115 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:35:47.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:47.118 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:35:47.219 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:35:47.251 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:47.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:46 smithi106 ceph-mon[44423]: pgmap v1737: 17 pgs: 17 active+clean; 7.4 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-20T17:35:47.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:46 smithi106 ceph-mon[44423]: 6.3 scrub ok 2024-04-20T17:35:47.272 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:35:47.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:47.275 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:35:47.377 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:35:47.449 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:47.508 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:35:47.877 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:47 smithi005 ceph-mon[39562]: from='osd.1 v1:172.21.15.5:6805/2415652792' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-20T17:35:47.877 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:47 smithi005 ceph-mon[39562]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-20T17:35:47.877 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:35:47 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-1[272232]: 2024-04-20T17:35:47.603+0000 7fdfc75b87c0 -1 osd.1 613 log_to_monitors true 2024-04-20T17:35:47.877 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:35:47 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-1[272232]: 2024-04-20T17:35:47.874+0000 7fdfb8a10700 -1 osd.1 658 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T17:35:48.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:47 smithi171 ceph-mon[44189]: from='osd.1 v1:172.21.15.5:6805/2415652792' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-20T17:35:48.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:47 smithi171 ceph-mon[44189]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-20T17:35:48.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:47 smithi106 ceph-mon[44423]: from='osd.1 v1:172.21.15.5:6805/2415652792' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-20T17:35:48.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:47 smithi106 ceph-mon[44423]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-20T17:35:49.080 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:48 smithi005 ceph-mon[39562]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-20T17:35:49.081 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:48 smithi005 ceph-mon[39562]: osdmap e658: 12 total, 11 up, 10 in 2024-04-20T17:35:49.081 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:48 smithi005 ceph-mon[39562]: from='osd.1 v1:172.21.15.5:6805/2415652792' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:35:49.081 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:48 smithi005 ceph-mon[39562]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:35:49.081 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:48 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T17:35:49.081 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:48 smithi005 ceph-mon[39562]: pgmap v1739: 17 pgs: 17 active+clean; 7.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-20T17:35:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:48 smithi171 ceph-mon[44189]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-20T17:35:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:48 smithi171 ceph-mon[44189]: osdmap e658: 12 total, 11 up, 10 in 2024-04-20T17:35:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:48 smithi171 ceph-mon[44189]: from='osd.1 v1:172.21.15.5:6805/2415652792' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:35:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:48 smithi171 ceph-mon[44189]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:35:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:48 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T17:35:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:48 smithi171 ceph-mon[44189]: pgmap v1739: 17 pgs: 17 active+clean; 7.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-20T17:35:49.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:48 smithi106 ceph-mon[44423]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-20T17:35:49.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:48 smithi106 ceph-mon[44423]: osdmap e658: 12 total, 11 up, 10 in 2024-04-20T17:35:49.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:48 smithi106 ceph-mon[44423]: from='osd.1 v1:172.21.15.5:6805/2415652792' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:35:49.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:48 smithi106 ceph-mon[44423]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:35:49.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:48 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T17:35:49.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:48 smithi106 ceph-mon[44423]: pgmap v1739: 17 pgs: 17 active+clean; 7.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-20T17:35:49.902 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:35:49.902 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:35:49.902 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:35:49.902 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:35:49.902 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:35:49.902 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:35:49.902 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:35:49.902 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:35:49.902 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:35:49.903 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:35:49.991 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:35:49.991 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:35:49.992 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:35:49.992 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:35:49.992 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:35:49.992 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:35:49.992 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:35:49.992 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:35:49.992 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:35:49.992 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:35:49.992 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:35:49.993 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:35:49.993 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:35:49.993 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:35:50.094 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:49 smithi171 ceph-mon[44189]: osd.1 v1:172.21.15.5:6805/2415652792 boot 2024-04-20T17:35:50.094 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:49 smithi171 ceph-mon[44189]: osdmap e659: 12 total, 12 up, 10 in 2024-04-20T17:35:50.094 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:49 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T17:35:50.094 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:49 smithi171 ceph-mon[44189]: 6.9 scrub starts 2024-04-20T17:35:50.094 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:49 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2497391534' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:35:50.094 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:50.094 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:50.094 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:50.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:49 smithi005 ceph-mon[39562]: osd.1 v1:172.21.15.5:6805/2415652792 boot 2024-04-20T17:35:50.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:49 smithi005 ceph-mon[39562]: osdmap e659: 12 total, 12 up, 10 in 2024-04-20T17:35:50.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:49 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T17:35:50.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:49 smithi005 ceph-mon[39562]: 6.9 scrub starts 2024-04-20T17:35:50.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:49 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2497391534' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:35:50.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:50.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:50.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:50.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:49 smithi106 ceph-mon[44423]: osd.1 v1:172.21.15.5:6805/2415652792 boot 2024-04-20T17:35:50.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:49 smithi106 ceph-mon[44423]: osdmap e659: 12 total, 12 up, 10 in 2024-04-20T17:35:50.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:49 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-20T17:35:50.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:49 smithi106 ceph-mon[44423]: 6.9 scrub starts 2024-04-20T17:35:50.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:49 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2497391534' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:35:50.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:50.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:50.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:50.682 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:35:50.683 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:35:50.683 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:35:50.683 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:206716 6.f 6:f5cf8540:::benchmark_data_smithi171_147972_object45618:head [delete] snapc 0=[] ondisk+write+known_if_redirected e658)", 2024-04-20T17:35:50.683 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:35:48.188819+0000", 2024-04-20T17:35:50.683 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.001005157, 2024-04-20T17:35:50.683 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0010275099999999999, 2024-04-20T17:35:50.683 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:35:50.683 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:35:50.683 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:35:50.683 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:35:50.683 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:35:50.683 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:35:50.683 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 206716 2024-04-20T17:35:50.683 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:35:50.684 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:35:50.684 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:35:50.684 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:35:50.684 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:48.188819+0000", 2024-04-20T17:35:50.684 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:35:50.684 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:35:50.684 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:35:50.684 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:35:50.684 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:48.188819+0000", 2024-04-20T17:35:50.684 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:35:50.684 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:35:50.684 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:35:50.684 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:35:50.684 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:48.188822+0000", 2024-04-20T17:35:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0309999999999999e-06 2024-04-20T17:35:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:35:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:35:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:35:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:48.188830+0000", 2024-04-20T17:35:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9710000000000004e-06 2024-04-20T17:35:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:35:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:35:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:35:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:48.188850+0000", 2024-04-20T17:35:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0225e-05 2024-04-20T17:35:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:35:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:35:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:35:50.685 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:48.188860+0000", 2024-04-20T17:35:50.686 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9690000000000006e-06 2024-04-20T17:35:50.686 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:35:50.686 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:35:50.686 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:35:50.686 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:48.188926+0000", 2024-04-20T17:35:50.686 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6223999999999995e-05 2024-04-20T17:35:50.686 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:35:50.686 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:35:50.686 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:35:50.686 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:48.189065+0000", 2024-04-20T17:35:50.686 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013813599999999999 2024-04-20T17:35:50.686 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:35:50.686 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:35:50.686 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:35:50.687 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:48.189199+0000", 2024-04-20T17:35:50.687 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013447599999999999 2024-04-20T17:35:50.687 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:35:50.687 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:35:50.687 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:35:50.687 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:35:50.687 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:35:50.687 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 1 2024-04-20T17:35:50.687 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:35:50.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:50.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:50 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:35:50.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:50 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:35:50.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:50.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:50.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:50.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:50.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:50.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:50 smithi005 ceph-mon[39562]: osdmap e660: 12 total, 12 up, 10 in 2024-04-20T17:35:50.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:50.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:50.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:50.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:50 smithi005 ceph-mon[39562]: pgmap v1742: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:35:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:50 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:35:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:50 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:35:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:51.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:51.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:51.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:50 smithi106 ceph-mon[44423]: osdmap e660: 12 total, 12 up, 10 in 2024-04-20T17:35:51.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:51.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:51.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:51.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:50 smithi106 ceph-mon[44423]: pgmap v1742: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:35:51.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:51.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:50 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:35:51.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:50 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:35:51.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:51.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:51.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:51.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:50 smithi171 ceph-mon[44189]: osdmap e660: 12 total, 12 up, 10 in 2024-04-20T17:35:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:50 smithi171 ceph-mon[44189]: pgmap v1742: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:35:52.166 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:52.166 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:51 smithi005 ceph-mon[39562]: 6.9 scrub ok 2024-04-20T17:35:52.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:52.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:51 smithi106 ceph-mon[44423]: 6.9 scrub ok 2024-04-20T17:35:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:35:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:51 smithi171 ceph-mon[44189]: 6.9 scrub ok 2024-04-20T17:35:52.865 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:52 smithi005 ceph-mon[39562]: pgmap v1743: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-20T17:35:52.866 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:52 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:35:53.079 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:35:53.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:53.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:35:53.185 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:35:53.232 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:52 smithi106 ceph-mon[44423]: pgmap v1743: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-20T17:35:53.232 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:52 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:35:53.234 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:53.269 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:35:53.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:53.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:35:53.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:52 smithi171 ceph-mon[44189]: pgmap v1743: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 1.03k op/s 2024-04-20T17:35:53.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:52 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:35:53.375 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:35:53.458 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:53.595 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:35:53.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:53.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:35:53.637 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-20T17:35:53.711 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:35:53.758 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:54.094 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:35:54.115 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:35:54.115 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:35:54.116 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:35:54.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:54.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:35:54.220 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:35:54.259 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:54.494 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:54 smithi106 ceph-mon[44423]: pgmap v1744: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-20T17:35:54.494 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:54 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:35:54.494 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:54 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:35:54.640 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:35:54.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:54.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:35:54.744 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:35:54.769 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:54 smithi005 ceph-mon[39562]: pgmap v1744: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-20T17:35:54.770 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:54 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:35:54.770 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:54 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:35:54.782 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:54.821 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:35:54.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:54.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:35:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:54 smithi171 ceph-mon[44189]: pgmap v1744: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-20T17:35:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:54 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:35:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:54 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:35:54.924 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:35:54.978 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:55.000 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:35:55.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:55.003 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:35:55.104 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:35:55.128 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:35:55.144 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:55.169 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:35:55.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:55.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:35:55.276 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:35:55.311 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:55.336 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:35:55.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:55.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:35:55.442 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:35:55.481 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:55.504 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:55 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:35:55.504 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:55 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:35:55.504 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:55 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:35:55.736 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:55 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:35:55.736 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:55 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:35:55.736 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:55 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:35:55.743 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:35:55.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:55.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:35:55.774 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:35:55.775 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-04-20T17:35:55.775 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-20T17:35:55.775 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-20T17:35:55.775 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-04-20T17:35:55.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:55 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:35:55.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:55 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:35:55.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:55 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:35:55.850 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:35:55.880 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:55.902 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:35:55.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:55.905 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:35:56.006 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:35:56.056 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:56.093 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:35:56.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:56.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:35:56.103 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:35:56.167 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:35:56.198 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:35:56.226 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:56.248 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:35:56.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:56.252 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:35:56.353 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:35:56.382 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:56.401 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:35:56.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:56.410 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:35:56.511 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:35:56.539 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:56.568 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:35:56.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:56.571 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:35:56.672 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:35:56.700 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:56.723 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:35:56.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:56.727 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:35:56.828 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:35:56.885 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:56.917 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:35:56.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:56.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:35:57.020 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:35:57.090 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:57.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:57 smithi106 ceph-mon[44423]: pgmap v1745: 17 pgs: 17 active+clean; 6.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-20T17:35:57.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:57 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2976613521' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:35:57.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:57 smithi106 ceph-mon[44423]: from='client.26908 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:35:57.533 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:35:57.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:57 smithi005 ceph-mon[39562]: pgmap v1745: 17 pgs: 17 active+clean; 6.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-20T17:35:57.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:57 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2976613521' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:35:57.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:57 smithi005 ceph-mon[39562]: from='client.26908 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:35:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:57 smithi171 ceph-mon[44189]: pgmap v1745: 17 pgs: 17 active+clean; 6.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.25k op/s 2024-04-20T17:35:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:57 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2976613521' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:35:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:57 smithi171 ceph-mon[44189]: from='client.26908 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:35:58.341 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:35:58.341 INFO:teuthology.orchestra.run.smithi106.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-20T17:35:58.342 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:35:58.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:58 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:35:58.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:58 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:35:58.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:58 smithi005 ceph-mon[39562]: 6.a scrub starts 2024-04-20T17:35:58.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:58 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:35:58.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:58 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:35:58.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:58 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:35:58.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:58 smithi005 ceph-mon[39562]: from='client.26908 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:35:58.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:58 smithi005 ceph-mon[39562]: osdmap e661: 12 total, 12 up, 10 in 2024-04-20T17:35:58.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:58 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:35:58.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:58 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:35:58.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:58 smithi106 ceph-mon[44423]: 6.a scrub starts 2024-04-20T17:35:58.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:58 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:35:58.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:58 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:35:58.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:58 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:35:58.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:58 smithi106 ceph-mon[44423]: from='client.26908 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:35:58.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:58 smithi106 ceph-mon[44423]: osdmap e661: 12 total, 12 up, 10 in 2024-04-20T17:35:58.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:58 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:35:58.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:58 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:35:58.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:58 smithi171 ceph-mon[44189]: 6.a scrub starts 2024-04-20T17:35:58.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:58 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:35:58.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:58 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:35:58.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:58 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:35:58.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:58 smithi171 ceph-mon[44189]: from='client.26908 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:35:58.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:58 smithi171 ceph-mon[44189]: osdmap e661: 12 total, 12 up, 10 in 2024-04-20T17:35:59.085 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:35:59.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:59.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:35:59.200 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:35:59.253 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:59.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:35:59 smithi005 ceph-mon[39562]: pgmap v1747: 17 pgs: 17 active+clean; 6.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 911 op/s 2024-04-20T17:35:59.538 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:35:59.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:59.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:35:59.646 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:35:59.677 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:59.705 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:35:59.706 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:35:59 smithi171 ceph-mon[44189]: pgmap v1747: 17 pgs: 17 active+clean; 6.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 911 op/s 2024-04-20T17:35:59.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:59.707 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:35:59.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:35:59 smithi106 ceph-mon[44423]: pgmap v1747: 17 pgs: 17 active+clean; 6.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 911 op/s 2024-04-20T17:35:59.808 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:35:59.844 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:35:59.874 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:35:59.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:35:59.881 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:35:59.982 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:36:00.003 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:36:00.014 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:00.041 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:36:00.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:00.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:36:00.148 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:36:00.202 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:00.225 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:36:00.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:00.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:36:00.334 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:36:00.364 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:00.390 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:36:00.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:00.395 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:36:00.496 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:36:00.524 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:00.550 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:36:00.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:00.553 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:36:00.654 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:36:00.682 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:00.704 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:36:00.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:00.709 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:36:00.810 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:36:00.866 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:00.995 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:36:00.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:00.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:36:01.099 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:36:01.118 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:36:01.118 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:36:01.136 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:01.156 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:36:01.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:01.159 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:36:01.260 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:36:01.290 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:01.311 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:36:01.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:01.314 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:36:01.415 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:36:01.437 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:01 smithi171 ceph-mon[44189]: pgmap v1748: 17 pgs: 17 active+clean; 6.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:36:01.457 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:01.577 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:36:01.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:01 smithi005 ceph-mon[39562]: pgmap v1748: 17 pgs: 17 active+clean; 6.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:36:01.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:01 smithi106 ceph-mon[44423]: pgmap v1748: 17 pgs: 17 active+clean; 6.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:36:01.811 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-20T17:36:02.223 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:36:02.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:02.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:36:02.332 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:36:02.375 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:02.404 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:36:02.410 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:36:02.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:02.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:36:02.514 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:36:02.558 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:02.574 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:36:02.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:02.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:36:02.679 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:36:02.706 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:02.732 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:36:02.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:02.735 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:36:02.836 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:36:02.866 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:02.893 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:36:02.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:02.896 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:36:02.997 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:36:03.037 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:03.248 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:36:03.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:03.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:36:03.268 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:36:03.357 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:36:03.389 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:03.409 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:36:03.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:03.412 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:36:03.513 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:36:03.552 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:03.575 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:03 smithi005 ceph-mon[39562]: pgmap v1749: 17 pgs: 17 active+clean; 6.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:36:03.581 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:36:03.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:03.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:36:03.686 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:36:03.722 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:03.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:03 smithi106 ceph-mon[44423]: pgmap v1749: 17 pgs: 17 active+clean; 6.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:36:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:03 smithi171 ceph-mon[44189]: pgmap v1749: 17 pgs: 17 active+clean; 6.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:36:04.187 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:36:04.190 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:36:04.201 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 8, 9, 10, 11, 7] out_osds: [6, 1] dead_osds: [] live_osds: [0, 4, 5, 8, 9, 10, 11, 2, 7, 6, 3, 1] 2024-04-20T17:36:04.201 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:36:04.202 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-20T17:36:04.202 INFO:tasks.thrashosds.thrasher:Setting osd 9 primary_affinity to 1.000000 2024-04-20T17:36:04.202 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd primary-affinity 9 1 2024-04-20T17:36:04.228 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:36:04.265 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:36:04.311 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:36:04.360 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:36:04.390 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:36:04.425 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:36:04.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:04 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2976287403' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:36:04.463 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:36:04.499 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:36:04.545 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:36:04.580 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:36:04.609 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:36:04.609 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:36:04.609 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:36:04.609 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:36:04.609 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:36:04.609 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:36:04.609 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:36:04.609 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:36:04.609 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:36:04.609 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:36:04.609 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:36:04.609 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:36:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:04 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2976287403' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:36:04.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:04 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2976287403' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:36:04.995 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:36:05.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:05.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:36:05.105 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:36:05.146 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:05.171 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:36:05.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:05.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:36:05.279 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:36:05.309 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:05.331 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:36:05.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:05.336 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:36:05.437 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:36:05.464 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.464 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.464 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.464 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.464 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.464 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.465 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.465 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:05 smithi106 ceph-mon[44423]: pgmap v1750: 17 pgs: 17 active+clean; 6.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-20T17:36:05.465 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.465 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.465 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.465 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.465 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.471 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:05.500 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:36:05.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:05.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:36:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:05 smithi171 ceph-mon[44189]: pgmap v1750: 17 pgs: 17 active+clean; 6.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-20T17:36:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.603 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:36:05.636 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:05.664 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:36:05.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:05.669 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:36:05.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:05 smithi005 ceph-mon[39562]: pgmap v1750: 17 pgs: 17 active+clean; 6.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-20T17:36:05.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:05.770 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:36:05.815 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:06.499 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:36:06.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:06.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:36:06.610 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:36:06.656 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:06.738 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:06 smithi005 ceph-mon[39562]: pgmap v1751: 17 pgs: 17 active+clean; 5.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-20T17:36:06.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:06 smithi106 ceph-mon[44423]: pgmap v1751: 17 pgs: 17 active+clean; 5.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-20T17:36:06.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:06 smithi171 ceph-mon[44189]: pgmap v1751: 17 pgs: 17 active+clean; 5.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-20T17:36:06.920 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:36:06.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:06.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:36:07.027 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:36:07.027 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:36:07.027 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:36:07.028 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:36:07.028 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.028 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:196584 5.6 5:69ff5259:::benchmark_data_smithi171_137281_object65150:head [delete] snapc 0=[] ondisk+write+known_if_redirected e598)", 2024-04-20T17:36:07.028 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:02.691746+0000", 2024-04-20T17:36:07.028 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 244.33110921100001, 2024-04-20T17:36:07.028 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.16427309100000001, 2024-04-20T17:36:07.028 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:07.029 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:07.029 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:07.029 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:07.029 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:36:07.029 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:36:07.029 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 196584 2024-04-20T17:36:07.029 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.030 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:07.030 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.030 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:07.030 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:02.691746+0000", 2024-04-20T17:36:07.030 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.030 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.031 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.031 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:07.031 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:02.691746+0000", 2024-04-20T17:36:07.031 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.031 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.031 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.031 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:07.031 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:02.691747+0000", 2024-04-20T17:36:07.031 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8000000000000005e-07 2024-04-20T17:36:07.031 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.031 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.031 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:07.032 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:02.691751+0000", 2024-04-20T17:36:07.032 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.924e-06 2024-04-20T17:36:07.032 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.032 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.032 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:07.032 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:02.691757+0000", 2024-04-20T17:36:07.032 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7009999999999999e-06 2024-04-20T17:36:07.032 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.032 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.032 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:07.032 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:02.691765+0000", 2024-04-20T17:36:07.032 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2799999999999998e-06 2024-04-20T17:36:07.032 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.032 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.033 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:07.033 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:02.701628+0000", 2024-04-20T17:36:07.033 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0098631819999999999 2024-04-20T17:36:07.033 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.033 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.033 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:07.033 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:02.701700+0000", 2024-04-20T17:36:07.033 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2372000000000004e-05 2024-04-20T17:36:07.033 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.033 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.033 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:36:07.033 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:02.701779+0000", 2024-04-20T17:36:07.033 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9191999999999997e-05 2024-04-20T17:36:07.033 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.034 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.034 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:07.034 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:02.815484+0000", 2024-04-20T17:36:07.034 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.113704508 2024-04-20T17:36:07.034 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.034 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.034 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:07.034 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:02.855925+0000", 2024-04-20T17:36:07.034 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.040441062999999999 2024-04-20T17:36:07.034 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.034 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.035 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:07.035 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:02.855968+0000", 2024-04-20T17:36:07.035 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2648999999999998e-05 2024-04-20T17:36:07.035 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.035 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.035 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:07.035 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:02.856019+0000", 2024-04-20T17:36:07.035 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1440000000000002e-05 2024-04-20T17:36:07.035 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.035 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:07.035 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.035 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.035 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.036 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:198141 5.6 5:6c7b0382:::benchmark_data_smithi171_137281_object66707:head [delete] snapc 0=[] ondisk+write+known_if_redirected e598)", 2024-04-20T17:36:07.036 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:04.223408+0000", 2024-04-20T17:36:07.036 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 242.799447413, 2024-04-20T17:36:07.036 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.15999193, 2024-04-20T17:36:07.036 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:07.036 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:07.036 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:07.036 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:07.036 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:36:07.036 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:36:07.036 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 198141 2024-04-20T17:36:07.036 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.036 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:07.036 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.037 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:07.037 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.223408+0000", 2024-04-20T17:36:07.037 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.037 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.037 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.037 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:07.037 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.223408+0000", 2024-04-20T17:36:07.037 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.037 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.037 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.037 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:07.037 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.223410+0000", 2024-04-20T17:36:07.037 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9089999999999998e-06 2024-04-20T17:36:07.037 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.038 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.038 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:07.038 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.223414+0000", 2024-04-20T17:36:07.038 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7219999999999999e-06 2024-04-20T17:36:07.038 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.038 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.038 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:07.038 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.223426+0000", 2024-04-20T17:36:07.038 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1256e-05 2024-04-20T17:36:07.038 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.038 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.038 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:07.038 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.223434+0000", 2024-04-20T17:36:07.038 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0930000000000003e-06 2024-04-20T17:36:07.039 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.039 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.039 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:07.039 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.230634+0000", 2024-04-20T17:36:07.039 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0072005439999999997 2024-04-20T17:36:07.039 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.039 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.039 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:07.039 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.230729+0000", 2024-04-20T17:36:07.039 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4617000000000006e-05 2024-04-20T17:36:07.039 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.039 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.039 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:36:07.039 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.230811+0000", 2024-04-20T17:36:07.040 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1767000000000005e-05 2024-04-20T17:36:07.040 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.040 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.040 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:07.040 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.241681+0000", 2024-04-20T17:36:07.040 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010870591000000001 2024-04-20T17:36:07.040 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.040 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.040 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:07.040 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.383298+0000", 2024-04-20T17:36:07.040 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.14161648399999999 2024-04-20T17:36:07.040 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.040 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.040 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:07.041 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.383339+0000", 2024-04-20T17:36:07.041 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0923000000000001e-05 2024-04-20T17:36:07.041 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.041 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.041 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:07.041 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.383400+0000", 2024-04-20T17:36:07.041 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1023999999999997e-05 2024-04-20T17:36:07.041 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.041 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:07.041 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.041 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.041 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.041 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:198148 5.6 5:63270389:::benchmark_data_smithi171_137281_object66714:head [delete] snapc 0=[] ondisk+write+known_if_redirected e598)", 2024-04-20T17:36:07.042 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:04.232148+0000", 2024-04-20T17:36:07.042 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 242.79070720799999, 2024-04-20T17:36:07.042 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.15136398700000001, 2024-04-20T17:36:07.042 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:07.042 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:07.042 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:07.042 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:07.042 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:36:07.042 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:36:07.042 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 198148 2024-04-20T17:36:07.042 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.042 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:07.042 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.042 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:07.043 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.232148+0000", 2024-04-20T17:36:07.043 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.043 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.043 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.043 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:07.043 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.232148+0000", 2024-04-20T17:36:07.043 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.043 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.043 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.043 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:07.043 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.232150+0000", 2024-04-20T17:36:07.043 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1900000000000002e-06 2024-04-20T17:36:07.043 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.043 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.044 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:07.044 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.232156+0000", 2024-04-20T17:36:07.044 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9719999999999997e-06 2024-04-20T17:36:07.044 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.044 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.044 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:07.044 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.232170+0000", 2024-04-20T17:36:07.044 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3326000000000001e-05 2024-04-20T17:36:07.044 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.044 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.044 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:07.044 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.232178+0000", 2024-04-20T17:36:07.044 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5390000000000007e-06 2024-04-20T17:36:07.044 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.045 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.045 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:07.045 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.236305+0000", 2024-04-20T17:36:07.045 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041266840000000003 2024-04-20T17:36:07.045 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.045 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.045 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:07.045 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.236457+0000", 2024-04-20T17:36:07.045 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015202 2024-04-20T17:36:07.045 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.045 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.045 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:36:07.045 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.236614+0000", 2024-04-20T17:36:07.045 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000156935 2024-04-20T17:36:07.046 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.046 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.046 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:07.046 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.383413+0000", 2024-04-20T17:36:07.046 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.14679882999999999 2024-04-20T17:36:07.046 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.046 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.046 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:07.046 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.383449+0000", 2024-04-20T17:36:07.046 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6427999999999997e-05 2024-04-20T17:36:07.046 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.046 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.046 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:07.047 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.383480+0000", 2024-04-20T17:36:07.047 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1452000000000003e-05 2024-04-20T17:36:07.047 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.047 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.047 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:07.047 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.383512+0000", 2024-04-20T17:36:07.047 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1610999999999999e-05 2024-04-20T17:36:07.047 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.047 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:07.047 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.047 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.047 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.047 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:198157 5.6 5:6e8b4717:::benchmark_data_smithi171_137281_object66723:head [delete] snapc 0=[] ondisk+write+known_if_redirected e598)", 2024-04-20T17:36:07.047 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:04.241922+0000", 2024-04-20T17:36:07.047 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 242.780933433, 2024-04-20T17:36:07.048 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17007925700000001, 2024-04-20T17:36:07.048 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:07.048 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:07.048 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:07.048 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:07.048 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:36:07.048 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:36:07.048 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 198157 2024-04-20T17:36:07.048 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.048 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:07.048 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.048 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:07.049 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.241922+0000", 2024-04-20T17:36:07.049 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.049 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.049 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.049 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:07.049 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.241922+0000", 2024-04-20T17:36:07.049 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.049 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.049 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.049 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:07.049 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.241925+0000", 2024-04-20T17:36:07.049 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7939999999999998e-06 2024-04-20T17:36:07.049 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.049 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.050 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:07.050 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.241931+0000", 2024-04-20T17:36:07.050 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7490000000000001e-06 2024-04-20T17:36:07.050 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.050 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.050 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:07.050 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.241944+0000", 2024-04-20T17:36:07.050 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2159e-05 2024-04-20T17:36:07.050 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.051 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.051 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:07.051 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.241952+0000", 2024-04-20T17:36:07.051 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2349999999999999e-06 2024-04-20T17:36:07.051 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.051 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.051 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:07.051 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.386210+0000", 2024-04-20T17:36:07.051 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.14425862 2024-04-20T17:36:07.051 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.051 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.051 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:07.051 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.386323+0000", 2024-04-20T17:36:07.051 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000112991 2024-04-20T17:36:07.052 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.052 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.052 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:36:07.052 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.386425+0000", 2024-04-20T17:36:07.052 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010176899999999999 2024-04-20T17:36:07.052 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.052 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.052 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:07.052 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.399037+0000", 2024-04-20T17:36:07.052 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012612254 2024-04-20T17:36:07.052 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.052 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.052 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:07.052 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.407304+0000", 2024-04-20T17:36:07.053 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0082665110000000007 2024-04-20T17:36:07.053 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.053 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.053 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:07.053 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.407369+0000", 2024-04-20T17:36:07.053 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5469999999999995e-05 2024-04-20T17:36:07.053 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.053 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.053 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:07.053 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.412001+0000", 2024-04-20T17:36:07.053 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0046317049999999998 2024-04-20T17:36:07.053 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.053 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:07.053 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.053 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.054 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.054 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:219957 5.6 5:6e2fd4d9:::benchmark_data_smithi171_137281_object88523:head [delete] snapc 0=[] ondisk+write+known_if_redirected e601)", 2024-04-20T17:36:07.054 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:24.617680+0000", 2024-04-20T17:36:07.054 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 222.40517499200001, 2024-04-20T17:36:07.054 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.226933211, 2024-04-20T17:36:07.054 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:07.054 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:07.054 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:07.054 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:07.054 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:36:07.054 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:36:07.055 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 219957 2024-04-20T17:36:07.055 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.055 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:07.055 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.055 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:07.055 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.617680+0000", 2024-04-20T17:36:07.055 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.055 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.055 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.055 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:07.055 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.617680+0000", 2024-04-20T17:36:07.055 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.055 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.056 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.056 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:07.056 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.617682+0000", 2024-04-20T17:36:07.056 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1550000000000001e-06 2024-04-20T17:36:07.056 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.056 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.056 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:07.056 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.617688+0000", 2024-04-20T17:36:07.056 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.733e-06 2024-04-20T17:36:07.056 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.057 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.057 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:07.057 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.617702+0000", 2024-04-20T17:36:07.057 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4236999999999999e-05 2024-04-20T17:36:07.057 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.057 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.057 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:07.057 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.617710+0000", 2024-04-20T17:36:07.057 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0539999999999995e-06 2024-04-20T17:36:07.057 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.057 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.057 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:07.057 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.617766+0000", 2024-04-20T17:36:07.057 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5640000000000003e-05 2024-04-20T17:36:07.058 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.058 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.058 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:07.058 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.617861+0000", 2024-04-20T17:36:07.058 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.5098000000000003e-05 2024-04-20T17:36:07.058 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.058 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.058 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:36:07.059 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.617990+0000", 2024-04-20T17:36:07.059 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012899400000000001 2024-04-20T17:36:07.059 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.059 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.059 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:07.059 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.622858+0000", 2024-04-20T17:36:07.059 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0048682309999999998 2024-04-20T17:36:07.060 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.060 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.060 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:07.060 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.844495+0000", 2024-04-20T17:36:07.060 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.22163646200000001 2024-04-20T17:36:07.060 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.060 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.060 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:07.060 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.844555+0000", 2024-04-20T17:36:07.060 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0582999999999999e-05 2024-04-20T17:36:07.060 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.060 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.060 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:07.061 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.844613+0000", 2024-04-20T17:36:07.061 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8023999999999999e-05 2024-04-20T17:36:07.061 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.061 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:07.061 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.061 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.061 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.061 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:219963 5.6 5:65be2a8e:::benchmark_data_smithi171_137281_object88529:head [delete] snapc 0=[] ondisk+write+known_if_redirected e601)", 2024-04-20T17:36:07.061 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:24.627025+0000", 2024-04-20T17:36:07.061 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 222.39583032100001, 2024-04-20T17:36:07.061 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.22374935600000001, 2024-04-20T17:36:07.061 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:07.061 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:07.061 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:07.062 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:07.062 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:36:07.062 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:36:07.062 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 219963 2024-04-20T17:36:07.062 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.062 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:07.062 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.062 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:07.062 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627025+0000", 2024-04-20T17:36:07.062 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.062 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.062 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.062 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:07.063 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627025+0000", 2024-04-20T17:36:07.063 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.063 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.063 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.063 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:07.063 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627027+0000", 2024-04-20T17:36:07.063 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.34e-06 2024-04-20T17:36:07.063 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.063 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.063 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:07.063 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627034+0000", 2024-04-20T17:36:07.063 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9299999999999997e-06 2024-04-20T17:36:07.063 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.063 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.063 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:07.064 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627050+0000", 2024-04-20T17:36:07.064 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6090999999999999e-05 2024-04-20T17:36:07.064 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.064 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.064 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:07.064 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627060+0000", 2024-04-20T17:36:07.064 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.312e-06 2024-04-20T17:36:07.064 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.064 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.064 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:07.064 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627134+0000", 2024-04-20T17:36:07.064 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4357999999999995e-05 2024-04-20T17:36:07.064 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.064 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.065 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:07.065 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627247+0000", 2024-04-20T17:36:07.065 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000113101 2024-04-20T17:36:07.065 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.065 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.065 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:36:07.065 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627398+0000", 2024-04-20T17:36:07.065 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015086799999999999 2024-04-20T17:36:07.065 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.065 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.065 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:07.065 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.636898+0000", 2024-04-20T17:36:07.065 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0094998559999999992 2024-04-20T17:36:07.065 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.066 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.066 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:07.066 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.850666+0000", 2024-04-20T17:36:07.066 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.213767862 2024-04-20T17:36:07.066 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.066 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.066 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:07.066 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.850723+0000", 2024-04-20T17:36:07.066 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6935999999999998e-05 2024-04-20T17:36:07.066 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.066 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.066 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:07.066 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.850774+0000", 2024-04-20T17:36:07.067 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1702000000000001e-05 2024-04-20T17:36:07.067 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.067 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:07.067 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.067 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.067 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.067 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:219967 5.6 5:6a45f1ea:::benchmark_data_smithi171_137281_object88533:head [delete] snapc 0=[] ondisk+write+known_if_redirected e601)", 2024-04-20T17:36:07.067 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:24.627084+0000", 2024-04-20T17:36:07.067 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 222.39577132599999, 2024-04-20T17:36:07.067 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.22382991299999999, 2024-04-20T17:36:07.067 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:07.067 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:07.067 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:07.067 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:07.068 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:36:07.068 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:36:07.068 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 219967 2024-04-20T17:36:07.068 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.068 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:07.068 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.068 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:07.068 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627084+0000", 2024-04-20T17:36:07.068 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.068 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.068 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.068 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:07.068 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627084+0000", 2024-04-20T17:36:07.068 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.068 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.069 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.069 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:07.069 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627085+0000", 2024-04-20T17:36:07.069 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.85e-07 2024-04-20T17:36:07.069 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.069 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.069 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:07.069 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627088+0000", 2024-04-20T17:36:07.069 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2590000000000001e-06 2024-04-20T17:36:07.069 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.069 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.069 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:07.069 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627095+0000", 2024-04-20T17:36:07.069 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0960000000000003e-06 2024-04-20T17:36:07.070 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.070 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.070 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:07.070 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627099+0000", 2024-04-20T17:36:07.070 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8539999999999999e-06 2024-04-20T17:36:07.070 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.071 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.071 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:07.071 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.632025+0000", 2024-04-20T17:36:07.071 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0049258870000000003 2024-04-20T17:36:07.071 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.071 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.071 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:07.071 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.632149+0000", 2024-04-20T17:36:07.071 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012428 2024-04-20T17:36:07.071 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.071 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.071 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:36:07.071 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.632268+0000", 2024-04-20T17:36:07.071 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000119244 2024-04-20T17:36:07.072 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.072 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.072 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:07.072 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.637004+0000", 2024-04-20T17:36:07.072 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0047353550000000001 2024-04-20T17:36:07.072 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.072 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.072 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:07.072 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.850820+0000", 2024-04-20T17:36:07.072 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.21381650099999999 2024-04-20T17:36:07.072 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.072 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.072 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:07.072 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.850871+0000", 2024-04-20T17:36:07.073 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1261999999999998e-05 2024-04-20T17:36:07.073 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.073 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.073 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:07.073 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.850914+0000", 2024-04-20T17:36:07.073 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2389999999999999e-05 2024-04-20T17:36:07.073 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.073 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:07.073 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.073 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.073 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.073 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:220024 5.6 5:642af313:::benchmark_data_smithi171_137281_object88590:head [delete] snapc 0=[] ondisk+write+known_if_redirected e601)", 2024-04-20T17:36:07.073 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:24.674168+0000", 2024-04-20T17:36:07.073 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 222.348687509, 2024-04-20T17:36:07.074 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.183402497, 2024-04-20T17:36:07.074 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:07.074 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:07.074 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:07.074 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:07.074 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:36:07.074 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:36:07.074 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 220024 2024-04-20T17:36:07.074 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.074 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:07.075 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.075 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:07.075 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.674168+0000", 2024-04-20T17:36:07.075 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.075 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.075 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.075 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:07.075 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.674168+0000", 2024-04-20T17:36:07.075 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.075 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.075 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.075 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:07.075 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.674169+0000", 2024-04-20T17:36:07.075 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.353e-06 2024-04-20T17:36:07.076 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.076 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.076 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:07.076 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.674174+0000", 2024-04-20T17:36:07.076 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4800000000000003e-06 2024-04-20T17:36:07.076 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.076 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.076 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:07.076 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.674182+0000", 2024-04-20T17:36:07.076 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8459999999999997e-06 2024-04-20T17:36:07.076 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.076 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.076 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:07.076 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.674189+0000", 2024-04-20T17:36:07.077 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8530000000000003e-06 2024-04-20T17:36:07.077 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.077 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.077 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:07.077 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.674243+0000", 2024-04-20T17:36:07.077 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3439000000000001e-05 2024-04-20T17:36:07.077 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.077 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.077 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:07.077 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.674423+0000", 2024-04-20T17:36:07.077 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00017983300000000001 2024-04-20T17:36:07.077 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.077 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.077 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:36:07.077 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.674535+0000", 2024-04-20T17:36:07.078 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011236899999999999 2024-04-20T17:36:07.078 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.078 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.078 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:07.078 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.679774+0000", 2024-04-20T17:36:07.078 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0052394939999999999 2024-04-20T17:36:07.078 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.078 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.078 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:07.078 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.857487+0000", 2024-04-20T17:36:07.078 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17771221500000001 2024-04-20T17:36:07.078 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.079 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.079 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:07.079 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.857522+0000", 2024-04-20T17:36:07.079 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4938000000000003e-05 2024-04-20T17:36:07.079 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.079 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.079 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:07.079 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.857570+0000", 2024-04-20T17:36:07.079 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8677000000000002e-05 2024-04-20T17:36:07.079 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.079 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:07.079 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.079 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.079 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.079 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:220029 5.6 5:610901c7:::benchmark_data_smithi171_137281_object88595:head [delete] snapc 0=[] ondisk+write+known_if_redirected e601)", 2024-04-20T17:36:07.080 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:24.684367+0000", 2024-04-20T17:36:07.080 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 222.33848777200001, 2024-04-20T17:36:07.080 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.173362769, 2024-04-20T17:36:07.080 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:07.080 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:07.080 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:07.080 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:07.080 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:36:07.080 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:36:07.080 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 220029 2024-04-20T17:36:07.080 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.080 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:07.080 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.081 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:07.081 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.684367+0000", 2024-04-20T17:36:07.081 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.081 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.081 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.081 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:07.081 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.684367+0000", 2024-04-20T17:36:07.081 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.081 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.081 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.082 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:07.082 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.684369+0000", 2024-04-20T17:36:07.082 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.875e-06 2024-04-20T17:36:07.082 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.082 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.082 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:07.082 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.684374+0000", 2024-04-20T17:36:07.082 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5970000000000002e-06 2024-04-20T17:36:07.082 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.082 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.082 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:07.082 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.684383+0000", 2024-04-20T17:36:07.083 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2289999999999992e-06 2024-04-20T17:36:07.083 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.083 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.083 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:07.083 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.684390+0000", 2024-04-20T17:36:07.083 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4019999999999999e-06 2024-04-20T17:36:07.083 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.083 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.083 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:07.083 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.684446+0000", 2024-04-20T17:36:07.084 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6387000000000001e-05 2024-04-20T17:36:07.084 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.084 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.084 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:07.084 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.684579+0000", 2024-04-20T17:36:07.084 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013314800000000001 2024-04-20T17:36:07.084 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.084 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.084 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:36:07.084 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.684706+0000", 2024-04-20T17:36:07.084 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012701999999999999 2024-04-20T17:36:07.084 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.085 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.085 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:07.085 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.689259+0000", 2024-04-20T17:36:07.085 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0045530309999999999 2024-04-20T17:36:07.085 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.085 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.085 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:07.085 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.857634+0000", 2024-04-20T17:36:07.085 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.16837489999999999 2024-04-20T17:36:07.085 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.085 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.085 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:07.086 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.857689+0000", 2024-04-20T17:36:07.086 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5019999999999998e-05 2024-04-20T17:36:07.086 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.086 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.086 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:07.086 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.857730+0000", 2024-04-20T17:36:07.086 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1159999999999999e-05 2024-04-20T17:36:07.086 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.086 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:07.086 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.086 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.087 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.087 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:234153 5.a 5:58b994fd:::benchmark_data_smithi171_137281_object102719:head [delete] snapc 0=[] ondisk+write+known_if_redirected e604)", 2024-04-20T17:36:07.087 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:37.046372+0000", 2024-04-20T17:36:07.087 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 209.976483323, 2024-04-20T17:36:07.087 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.15147975799999999, 2024-04-20T17:36:07.087 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:07.087 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:07.087 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:07.087 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:07.087 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:36:07.087 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:36:07.087 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 234153 2024-04-20T17:36:07.087 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.087 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:07.088 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.088 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:07.088 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:37.046372+0000", 2024-04-20T17:36:07.088 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.088 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.088 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.088 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:07.088 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:37.046372+0000", 2024-04-20T17:36:07.088 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.088 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.088 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.088 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:07.089 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:37.046374+0000", 2024-04-20T17:36:07.089 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6610000000000001e-06 2024-04-20T17:36:07.089 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.089 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.089 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:07.089 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:37.046380+0000", 2024-04-20T17:36:07.089 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5599999999999999e-06 2024-04-20T17:36:07.089 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.089 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.089 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:07.089 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:37.046387+0000", 2024-04-20T17:36:07.089 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.002e-06 2024-04-20T17:36:07.089 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.090 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.090 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:07.090 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:37.046393+0000", 2024-04-20T17:36:07.090 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6629999999999998e-06 2024-04-20T17:36:07.090 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.090 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.090 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:07.090 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:37.073213+0000", 2024-04-20T17:36:07.090 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.026819886000000001 2024-04-20T17:36:07.090 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.090 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.090 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:07.091 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:37.073346+0000", 2024-04-20T17:36:07.091 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000133032 2024-04-20T17:36:07.091 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.091 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.091 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:36:07.091 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:37.073462+0000", 2024-04-20T17:36:07.091 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000116709 2024-04-20T17:36:07.091 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.091 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.091 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:07.091 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:37.083070+0000", 2024-04-20T17:36:07.091 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0096075359999999999 2024-04-20T17:36:07.092 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.092 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.092 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:07.092 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:37.197726+0000", 2024-04-20T17:36:07.092 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.114656064 2024-04-20T17:36:07.092 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.092 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.092 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:07.092 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:37.197785+0000", 2024-04-20T17:36:07.092 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9305999999999998e-05 2024-04-20T17:36:07.092 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.092 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.092 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:07.093 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:37.197852+0000", 2024-04-20T17:36:07.093 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6339000000000003e-05 2024-04-20T17:36:07.093 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.093 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:07.093 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.093 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.093 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.093 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:236589 5.a 5:55c87012:::benchmark_data_smithi171_137281_object105155:head [delete] snapc 0=[] ondisk+write+known_if_redirected e606)", 2024-04-20T17:36:07.093 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:40.279469+0000", 2024-04-20T17:36:07.093 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 206.74338600900001, 2024-04-20T17:36:07.093 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.16665637899999999, 2024-04-20T17:36:07.093 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:07.094 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:07.094 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:07.094 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:07.094 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:36:07.094 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:36:07.094 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 236589 2024-04-20T17:36:07.094 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.094 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:07.094 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.094 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:07.094 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.279469+0000", 2024-04-20T17:36:07.095 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.095 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.095 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.095 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:07.095 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.279469+0000", 2024-04-20T17:36:07.095 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.095 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.095 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.095 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:07.095 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.279471+0000", 2024-04-20T17:36:07.095 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3230000000000001e-06 2024-04-20T17:36:07.096 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.096 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.096 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:07.096 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.279475+0000", 2024-04-20T17:36:07.096 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0640000000000004e-06 2024-04-20T17:36:07.096 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.096 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.096 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:07.096 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.279487+0000", 2024-04-20T17:36:07.096 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2568999999999999e-05 2024-04-20T17:36:07.096 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.096 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.096 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:07.096 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.279494+0000", 2024-04-20T17:36:07.097 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7209999999999997e-06 2024-04-20T17:36:07.097 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.097 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.097 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:07.097 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.308785+0000", 2024-04-20T17:36:07.097 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.029291159000000001 2024-04-20T17:36:07.097 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.097 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.097 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:07.097 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.308895+0000", 2024-04-20T17:36:07.097 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000109713 2024-04-20T17:36:07.098 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.098 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.098 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:36:07.098 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.309020+0000", 2024-04-20T17:36:07.098 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012495799999999999 2024-04-20T17:36:07.098 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.098 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.098 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:07.098 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.319829+0000", 2024-04-20T17:36:07.098 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010809526 2024-04-20T17:36:07.098 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.099 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.099 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:07.099 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.446006+0000", 2024-04-20T17:36:07.099 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.12617643100000001 2024-04-20T17:36:07.099 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.099 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.099 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:07.099 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.446062+0000", 2024-04-20T17:36:07.099 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6418000000000001e-05 2024-04-20T17:36:07.099 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.099 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.099 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:07.100 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.446126+0000", 2024-04-20T17:36:07.100 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3497000000000005e-05 2024-04-20T17:36:07.100 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.100 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:07.100 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.100 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.100 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.100 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:236595 5.a 5:546f56ab:::benchmark_data_smithi171_137281_object105161:head [delete] snapc 0=[] ondisk+write+known_if_redirected e606)", 2024-04-20T17:36:07.100 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:40.292587+0000", 2024-04-20T17:36:07.100 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 206.730267785, 2024-04-20T17:36:07.100 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.15402600699999999, 2024-04-20T17:36:07.101 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:07.101 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:07.101 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:07.101 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:07.101 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:36:07.101 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:36:07.101 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 236595 2024-04-20T17:36:07.101 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.101 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:07.101 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.101 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:07.101 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.292587+0000", 2024-04-20T17:36:07.102 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.102 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.102 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.102 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:07.102 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.292587+0000", 2024-04-20T17:36:07.102 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.102 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.102 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.102 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:07.102 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.292589+0000", 2024-04-20T17:36:07.102 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6649999999999999e-06 2024-04-20T17:36:07.103 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.103 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.103 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:07.103 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.292594+0000", 2024-04-20T17:36:07.103 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.091e-06 2024-04-20T17:36:07.103 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.103 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.103 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:07.103 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.292603+0000", 2024-04-20T17:36:07.103 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5439999999999999e-06 2024-04-20T17:36:07.103 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.104 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.104 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:07.104 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.292610+0000", 2024-04-20T17:36:07.104 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9789999999999996e-06 2024-04-20T17:36:07.104 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.104 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.104 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:07.104 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.313922+0000", 2024-04-20T17:36:07.104 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.021312607000000001 2024-04-20T17:36:07.104 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.104 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.104 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:07.104 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.314051+0000", 2024-04-20T17:36:07.105 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012910100000000001 2024-04-20T17:36:07.105 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.105 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.105 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:36:07.105 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.314197+0000", 2024-04-20T17:36:07.105 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000145143 2024-04-20T17:36:07.105 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.105 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.105 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:07.105 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.320588+0000", 2024-04-20T17:36:07.105 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0063915200000000004 2024-04-20T17:36:07.106 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.106 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.106 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:07.106 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.446510+0000", 2024-04-20T17:36:07.106 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.12592145800000001 2024-04-20T17:36:07.106 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.106 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.106 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:07.106 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.446562+0000", 2024-04-20T17:36:07.106 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2089000000000002e-05 2024-04-20T17:36:07.106 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.106 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.107 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:07.107 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.446613+0000", 2024-04-20T17:36:07.107 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.181e-05 2024-04-20T17:36:07.107 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.107 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:07.107 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.107 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.107 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.107 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:253233 5.6 5:6a18544b:::benchmark_data_smithi171_137281_object121799:head [delete] snapc 0=[] ondisk+write+known_if_redirected e610)", 2024-04-20T17:36:07.107 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:56.422462+0000", 2024-04-20T17:36:07.107 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 190.60039317600001, 2024-04-20T17:36:07.107 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.22659600899999999, 2024-04-20T17:36:07.108 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:07.108 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:07.108 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:07.108 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:07.108 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:36:07.108 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:36:07.108 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 253233 2024-04-20T17:36:07.108 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.108 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:07.108 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.108 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:07.108 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.422462+0000", 2024-04-20T17:36:07.109 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.109 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.109 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.109 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:07.109 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.422462+0000", 2024-04-20T17:36:07.109 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.109 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.109 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.109 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:07.109 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.422465+0000", 2024-04-20T17:36:07.109 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6060000000000001e-06 2024-04-20T17:36:07.109 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.110 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.110 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:07.110 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.422472+0000", 2024-04-20T17:36:07.110 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8680000000000001e-06 2024-04-20T17:36:07.110 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.110 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.110 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:07.110 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.422483+0000", 2024-04-20T17:36:07.110 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1797000000000001e-05 2024-04-20T17:36:07.110 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.110 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.111 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:07.111 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.422490+0000", 2024-04-20T17:36:07.111 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2230000000000004e-06 2024-04-20T17:36:07.111 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.111 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.111 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:07.111 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.422540+0000", 2024-04-20T17:36:07.111 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0278999999999998e-05 2024-04-20T17:36:07.111 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.111 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.111 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:07.112 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.422638+0000", 2024-04-20T17:36:07.112 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8335e-05 2024-04-20T17:36:07.112 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.112 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.112 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:36:07.112 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.422716+0000", 2024-04-20T17:36:07.112 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7702999999999998e-05 2024-04-20T17:36:07.112 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.112 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.112 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:07.112 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.432595+0000", 2024-04-20T17:36:07.112 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0098791590000000002 2024-04-20T17:36:07.112 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.113 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.113 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:07.113 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.648976+0000", 2024-04-20T17:36:07.113 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.21638126999999999 2024-04-20T17:36:07.113 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.113 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.113 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:07.113 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.649010+0000", 2024-04-20T17:36:07.113 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3269999999999998e-05 2024-04-20T17:36:07.113 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.113 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.113 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:07.114 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.649058+0000", 2024-04-20T17:36:07.114 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8498999999999998e-05 2024-04-20T17:36:07.114 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.114 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:07.114 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.114 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.114 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.114 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:253242 5.6 5:66b7bb4b:::benchmark_data_smithi171_137281_object121808:head [delete] snapc 0=[] ondisk+write+known_if_redirected e610)", 2024-04-20T17:36:07.114 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:56.427676+0000", 2024-04-20T17:36:07.114 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 190.59517879000001, 2024-04-20T17:36:07.114 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.22493042599999999, 2024-04-20T17:36:07.114 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:07.115 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:07.115 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:07.115 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:07.115 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:36:07.115 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:36:07.115 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 253242 2024-04-20T17:36:07.115 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.115 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:07.115 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.115 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:07.115 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.427676+0000", 2024-04-20T17:36:07.115 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.116 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.116 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.116 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:07.116 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.427676+0000", 2024-04-20T17:36:07.116 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.116 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.116 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.116 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:07.116 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.427679+0000", 2024-04-20T17:36:07.116 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1880000000000001e-06 2024-04-20T17:36:07.116 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.116 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.116 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:07.117 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.427685+0000", 2024-04-20T17:36:07.117 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1190000000000002e-06 2024-04-20T17:36:07.117 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.117 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.117 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:07.117 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.427698+0000", 2024-04-20T17:36:07.117 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2909e-05 2024-04-20T17:36:07.117 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.117 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.117 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:07.117 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.427706+0000", 2024-04-20T17:36:07.117 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9889999999999992e-06 2024-04-20T17:36:07.118 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.118 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.118 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:07.118 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.427762+0000", 2024-04-20T17:36:07.118 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6514999999999997e-05 2024-04-20T17:36:07.118 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.118 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.118 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:07.118 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.427906+0000", 2024-04-20T17:36:07.118 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000143988 2024-04-20T17:36:07.118 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.118 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.118 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:36:07.119 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.428043+0000", 2024-04-20T17:36:07.119 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013689400000000001 2024-04-20T17:36:07.119 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.119 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.119 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:07.119 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.432679+0000", 2024-04-20T17:36:07.119 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0046360430000000003 2024-04-20T17:36:07.119 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.119 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.119 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:07.119 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.652522+0000", 2024-04-20T17:36:07.119 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.21984308799999999 2024-04-20T17:36:07.119 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.120 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.120 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:07.120 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.652562+0000", 2024-04-20T17:36:07.120 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9517999999999997e-05 2024-04-20T17:36:07.120 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.120 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.120 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:07.120 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.652607+0000", 2024-04-20T17:36:07.120 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5175e-05 2024-04-20T17:36:07.120 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.120 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:07.120 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.120 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.121 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.121 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:253270 5.6 5:631ba04b:::benchmark_data_smithi171_137281_object121836:head [delete] snapc 0=[] ondisk+write+known_if_redirected e610)", 2024-04-20T17:36:07.121 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:56.450099+0000", 2024-04-20T17:36:07.121 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 190.57275633, 2024-04-20T17:36:07.121 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.20781598400000001, 2024-04-20T17:36:07.121 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:07.121 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:07.121 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:07.121 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:07.121 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:36:07.121 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:36:07.121 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 253270 2024-04-20T17:36:07.121 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.122 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:07.122 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.122 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:07.122 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.450099+0000", 2024-04-20T17:36:07.122 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.122 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.122 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.122 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:07.122 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.450099+0000", 2024-04-20T17:36:07.122 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.122 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.122 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.123 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:07.123 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.450101+0000", 2024-04-20T17:36:07.123 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5179999999999999e-06 2024-04-20T17:36:07.123 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.123 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.123 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:07.123 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.450109+0000", 2024-04-20T17:36:07.123 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2350000000000002e-06 2024-04-20T17:36:07.123 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.123 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.123 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:07.123 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.450124+0000", 2024-04-20T17:36:07.123 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5304000000000001e-05 2024-04-20T17:36:07.123 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.124 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.124 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:07.124 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.450133+0000", 2024-04-20T17:36:07.124 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7390000000000005e-06 2024-04-20T17:36:07.124 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.124 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.124 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:07.124 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.450200+0000", 2024-04-20T17:36:07.124 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7656000000000004e-05 2024-04-20T17:36:07.124 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.124 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.124 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:07.125 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.450338+0000", 2024-04-20T17:36:07.125 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000137451 2024-04-20T17:36:07.125 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.125 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.125 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:36:07.125 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.450474+0000", 2024-04-20T17:36:07.125 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013609599999999999 2024-04-20T17:36:07.125 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.125 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.125 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:07.125 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.455275+0000", 2024-04-20T17:36:07.126 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.004800627 2024-04-20T17:36:07.126 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.126 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.126 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:07.126 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.657827+0000", 2024-04-20T17:36:07.126 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.20255267599999999 2024-04-20T17:36:07.126 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.126 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.126 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:07.126 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.657865+0000", 2024-04-20T17:36:07.126 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8145000000000002e-05 2024-04-20T17:36:07.126 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.126 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.127 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:07.127 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.657915+0000", 2024-04-20T17:36:07.127 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9536999999999997e-05 2024-04-20T17:36:07.127 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.127 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:07.127 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.127 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.127 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.127 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:253281 5.6 5:622f9e9c:::benchmark_data_smithi171_137281_object121847:head [delete] snapc 0=[] ondisk+write+known_if_redirected e610)", 2024-04-20T17:36:07.127 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:56.460513+0000", 2024-04-20T17:36:07.127 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 190.56234213600001, 2024-04-20T17:36:07.127 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.19750880000000001, 2024-04-20T17:36:07.128 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:07.128 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:07.128 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:07.128 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:07.128 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:36:07.128 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:36:07.128 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 253281 2024-04-20T17:36:07.128 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.128 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:07.128 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.128 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:07.128 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.460513+0000", 2024-04-20T17:36:07.128 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.129 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.129 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.129 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:07.129 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.460513+0000", 2024-04-20T17:36:07.129 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.129 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.129 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.129 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:07.129 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.460515+0000", 2024-04-20T17:36:07.129 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3429999999999998e-06 2024-04-20T17:36:07.129 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.130 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.130 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:07.130 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.460523+0000", 2024-04-20T17:36:07.130 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1860000000000003e-06 2024-04-20T17:36:07.130 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.130 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.130 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:07.130 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.460536+0000", 2024-04-20T17:36:07.130 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3023999999999999e-05 2024-04-20T17:36:07.130 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.130 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.130 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:07.130 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.460544+0000", 2024-04-20T17:36:07.131 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1149999999999994e-06 2024-04-20T17:36:07.131 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.131 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.131 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:07.131 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.460605+0000", 2024-04-20T17:36:07.131 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1480000000000001e-05 2024-04-20T17:36:07.131 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.131 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.131 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:07.131 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.460742+0000", 2024-04-20T17:36:07.131 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013681500000000001 2024-04-20T17:36:07.131 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.131 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.132 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:36:07.132 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.460881+0000", 2024-04-20T17:36:07.132 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000139004 2024-04-20T17:36:07.132 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.132 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.132 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:07.132 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.471685+0000", 2024-04-20T17:36:07.132 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010803696999999999 2024-04-20T17:36:07.132 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.132 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.132 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:07.132 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.657962+0000", 2024-04-20T17:36:07.133 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.18627682200000001 2024-04-20T17:36:07.133 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.133 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.133 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:07.133 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.657995+0000", 2024-04-20T17:36:07.133 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3003999999999998e-05 2024-04-20T17:36:07.133 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.133 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.133 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:07.133 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.658022+0000", 2024-04-20T17:36:07.133 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7310000000000001e-05 2024-04-20T17:36:07.133 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.134 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:07.134 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.134 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.134 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.134 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:253283 5.6 5:668e3393:::benchmark_data_smithi171_137281_object121849:head [delete] snapc 0=[] ondisk+write+known_if_redirected e610)", 2024-04-20T17:36:07.134 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:56.461378+0000", 2024-04-20T17:36:07.134 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 190.56147771799999, 2024-04-20T17:36:07.134 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.19674983099999999, 2024-04-20T17:36:07.134 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:07.134 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:07.134 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:07.135 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:07.135 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:36:07.135 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:36:07.135 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 253283 2024-04-20T17:36:07.135 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.135 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:07.135 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.135 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:07.135 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.461378+0000", 2024-04-20T17:36:07.135 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.135 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.135 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.135 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:07.136 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.461378+0000", 2024-04-20T17:36:07.136 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.136 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.136 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.136 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:07.136 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.461379+0000", 2024-04-20T17:36:07.136 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.403e-06 2024-04-20T17:36:07.136 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.136 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.136 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:07.136 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.461384+0000", 2024-04-20T17:36:07.136 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.135e-06 2024-04-20T17:36:07.136 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.137 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.137 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:07.137 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.461396+0000", 2024-04-20T17:36:07.137 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2194999999999999e-05 2024-04-20T17:36:07.137 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.137 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.137 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:07.137 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.461404+0000", 2024-04-20T17:36:07.137 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5109999999999997e-06 2024-04-20T17:36:07.137 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.137 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.137 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:07.138 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.467170+0000", 2024-04-20T17:36:07.138 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0057657259999999997 2024-04-20T17:36:07.138 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.138 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.138 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:07.138 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.467308+0000", 2024-04-20T17:36:07.138 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013891000000000001 2024-04-20T17:36:07.138 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.138 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.138 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:36:07.138 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.467433+0000", 2024-04-20T17:36:07.138 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012483500000000001 2024-04-20T17:36:07.138 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.139 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.139 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:07.139 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.472308+0000", 2024-04-20T17:36:07.139 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0048747540000000002 2024-04-20T17:36:07.139 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.139 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.139 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:07.139 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.658061+0000", 2024-04-20T17:36:07.139 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.18575340700000001 2024-04-20T17:36:07.139 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.139 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.139 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:07.139 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.658093+0000", 2024-04-20T17:36:07.140 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1270999999999999e-05 2024-04-20T17:36:07.140 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.140 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.140 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:07.140 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.658127+0000", 2024-04-20T17:36:07.140 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4684000000000002e-05 2024-04-20T17:36:07.140 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.140 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:07.140 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.140 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.140 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.141 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:253301 5.6 5:6a3a8ecf:::benchmark_data_smithi171_137281_object121867:head [delete] snapc 0=[] ondisk+write+known_if_redirected e610)", 2024-04-20T17:36:07.141 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:56.474259+0000", 2024-04-20T17:36:07.141 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 190.548596546, 2024-04-20T17:36:07.141 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.19229986700000001, 2024-04-20T17:36:07.141 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:07.141 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:07.141 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:07.141 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:07.141 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:36:07.141 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:36:07.141 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 253301 2024-04-20T17:36:07.141 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.142 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:07.142 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.142 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:07.142 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.474259+0000", 2024-04-20T17:36:07.142 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.142 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.142 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.142 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:07.142 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.474259+0000", 2024-04-20T17:36:07.142 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.142 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.142 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.143 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:07.143 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.474261+0000", 2024-04-20T17:36:07.143 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6759999999999999e-06 2024-04-20T17:36:07.143 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.143 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.143 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:07.143 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.474269+0000", 2024-04-20T17:36:07.143 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6040000000000004e-06 2024-04-20T17:36:07.143 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.143 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.143 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:07.143 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.474293+0000", 2024-04-20T17:36:07.143 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3589000000000001e-05 2024-04-20T17:36:07.144 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.144 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.144 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:07.144 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.474302+0000", 2024-04-20T17:36:07.144 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.747e-06 2024-04-20T17:36:07.144 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.144 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.144 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:07.144 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.474359+0000", 2024-04-20T17:36:07.144 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6913000000000002e-05 2024-04-20T17:36:07.144 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.144 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.144 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:07.145 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.474483+0000", 2024-04-20T17:36:07.145 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012419199999999999 2024-04-20T17:36:07.145 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.145 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.145 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:36:07.145 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.474605+0000", 2024-04-20T17:36:07.145 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000121712 2024-04-20T17:36:07.145 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.145 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.145 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:07.145 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.479728+0000", 2024-04-20T17:36:07.146 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0051231050000000002 2024-04-20T17:36:07.146 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.146 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.146 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:07.146 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.666462+0000", 2024-04-20T17:36:07.146 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.186733538 2024-04-20T17:36:07.146 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.146 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.146 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:07.146 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.666522+0000", 2024-04-20T17:36:07.146 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0130000000000002e-05 2024-04-20T17:36:07.146 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.147 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.147 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:07.147 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.666559+0000", 2024-04-20T17:36:07.147 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6661e-05 2024-04-20T17:36:07.147 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.147 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:07.147 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.147 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.147 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.147 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:253316 5.6 5:672cd2ba:::benchmark_data_smithi171_137281_object121882:head [delete] snapc 0=[] ondisk+write+known_if_redirected e610)", 2024-04-20T17:36:07.147 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:56.488372+0000", 2024-04-20T17:36:07.147 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 190.53448331800001, 2024-04-20T17:36:07.147 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17831455500000001, 2024-04-20T17:36:07.147 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:07.148 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:07.148 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:07.148 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:07.148 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:36:07.148 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:36:07.148 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 253316 2024-04-20T17:36:07.148 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.148 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:07.148 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.148 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:07.148 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.488372+0000", 2024-04-20T17:36:07.148 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.148 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.149 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.149 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:07.149 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.488372+0000", 2024-04-20T17:36:07.149 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.149 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.149 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.149 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:07.149 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.488374+0000", 2024-04-20T17:36:07.149 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.385e-06 2024-04-20T17:36:07.149 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.149 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.149 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:07.149 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.488380+0000", 2024-04-20T17:36:07.149 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4979999999999997e-06 2024-04-20T17:36:07.149 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.150 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.150 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:07.150 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.488393+0000", 2024-04-20T17:36:07.150 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3067999999999999e-05 2024-04-20T17:36:07.150 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.150 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.150 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:07.150 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.488400+0000", 2024-04-20T17:36:07.150 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2989999999999999e-06 2024-04-20T17:36:07.150 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.150 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.150 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:07.150 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.488458+0000", 2024-04-20T17:36:07.151 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7785000000000003e-05 2024-04-20T17:36:07.151 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.151 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.151 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:07.151 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.488594+0000", 2024-04-20T17:36:07.151 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013595799999999999 2024-04-20T17:36:07.151 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.151 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.151 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:36:07.151 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.488715+0000", 2024-04-20T17:36:07.151 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012079299999999999 2024-04-20T17:36:07.151 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.152 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.152 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:07.152 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.493470+0000", 2024-04-20T17:36:07.152 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0047552330000000002 2024-04-20T17:36:07.152 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.152 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.152 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:07.152 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.666619+0000", 2024-04-20T17:36:07.152 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.173149519 2024-04-20T17:36:07.152 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.152 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.152 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:07.152 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.666648+0000", 2024-04-20T17:36:07.152 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8915999999999999e-05 2024-04-20T17:36:07.153 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.153 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.153 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:07.153 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.666686+0000", 2024-04-20T17:36:07.153 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8101e-05 2024-04-20T17:36:07.153 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.153 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:07.153 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.153 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.153 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.153 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:253330 5.6 5:6b7ab629:::benchmark_data_smithi171_137281_object121896:head [delete] snapc 0=[] ondisk+write+known_if_redirected e610)", 2024-04-20T17:36:07.154 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:56.499739+0000", 2024-04-20T17:36:07.154 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 190.52311616899999, 2024-04-20T17:36:07.154 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.167089714, 2024-04-20T17:36:07.154 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:07.154 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:07.154 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:07.154 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:07.154 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:36:07.154 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:36:07.154 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 253330 2024-04-20T17:36:07.155 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.155 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:07.155 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.155 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:07.155 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.499739+0000", 2024-04-20T17:36:07.155 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.155 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.155 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.155 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:07.155 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.499739+0000", 2024-04-20T17:36:07.155 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:07.155 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.156 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.156 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:07.156 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.499742+0000", 2024-04-20T17:36:07.156 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5780000000000001e-06 2024-04-20T17:36:07.156 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.156 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.156 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:07.156 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.499750+0000", 2024-04-20T17:36:07.156 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6819999999999992e-06 2024-04-20T17:36:07.156 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.156 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.156 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:07.157 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.499766+0000", 2024-04-20T17:36:07.157 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5987000000000001e-05 2024-04-20T17:36:07.157 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.157 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.157 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:07.157 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.499774+0000", 2024-04-20T17:36:07.157 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5059999999999998e-06 2024-04-20T17:36:07.157 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.157 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.157 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:07.157 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.499832+0000", 2024-04-20T17:36:07.158 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8065000000000001e-05 2024-04-20T17:36:07.158 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.158 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.158 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:07.158 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.499921+0000", 2024-04-20T17:36:07.158 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9501999999999998e-05 2024-04-20T17:36:07.158 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.158 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.158 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:36:07.158 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.499994+0000", 2024-04-20T17:36:07.158 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2384000000000004e-05 2024-04-20T17:36:07.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.159 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.159 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:07.159 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.504608+0000", 2024-04-20T17:36:07.159 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0046146870000000001 2024-04-20T17:36:07.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.159 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.159 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:07.159 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.666734+0000", 2024-04-20T17:36:07.159 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.162125306 2024-04-20T17:36:07.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.159 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.159 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:07.160 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.666787+0000", 2024-04-20T17:36:07.160 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3316000000000002e-05 2024-04-20T17:36:07.160 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:07.160 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:07.160 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:07.160 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.666829+0000", 2024-04-20T17:36:07.160 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1700999999999999e-05 2024-04-20T17:36:07.160 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.160 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:07.160 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.160 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:07.161 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:07.161 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:36:07.161 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:36:07.195 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:07.228 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:36:07.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:07.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:36:07.334 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:36:07.362 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:07.386 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:36:07.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:07.390 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:36:07.491 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:36:07.533 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:07.900 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-20T17:36:08.196 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:36:08.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:08.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:36:08.232 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:36:08.278 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:36:08.301 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:36:08.336 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:08.509 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:36:08.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:08.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:36:08.614 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:36:08.643 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:08.666 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:36:08.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:08.670 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:36:08.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:08 smithi106 ceph-mon[44423]: pgmap v1752: 17 pgs: 17 active+clean; 5.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:36:08.769 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:36:08.771 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:36:08.774 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:08 smithi005 ceph-mon[39562]: pgmap v1752: 17 pgs: 17 active+clean; 5.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:36:08.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:08 smithi171 ceph-mon[44189]: pgmap v1752: 17 pgs: 17 active+clean; 5.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:36:08.898 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:08.917 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:36:08.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:08.921 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:36:09.022 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:36:09.073 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:09.279 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:36:09.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:09.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:36:09.383 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:36:09.442 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:09.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:09 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1998414005' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:36:09.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:09 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:36:09.977 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:36:10.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:09 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1998414005' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:36:10.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:09 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:36:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:09 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1998414005' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:36:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:09 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:36:10.502 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:36:10.502 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:36:10.502 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.502 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:232925 6.5 6:af827050:::benchmark_data_smithi171_147972_object71827:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:36:10.502 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:36:10.493640+0000", 2024-04-20T17:36:10.503 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0072417180000000003, 2024-04-20T17:36:10.503 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0072741680000000001, 2024-04-20T17:36:10.503 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:36:10.503 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:36:10.503 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:36:10.503 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:36:10.503 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:36:10.503 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:36:10.503 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 232925 2024-04-20T17:36:10.503 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.503 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:36:10.503 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.503 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:36:10.503 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.493640+0000", 2024-04-20T17:36:10.504 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:36:10.504 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.504 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.504 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:36:10.504 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.493640+0000", 2024-04-20T17:36:10.504 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:36:10.504 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.504 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.504 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:36:10.504 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.493642+0000", 2024-04-20T17:36:10.504 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9549999999999999e-06 2024-04-20T17:36:10.504 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.504 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.504 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:36:10.505 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.493647+0000", 2024-04-20T17:36:10.505 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4650000000000002e-06 2024-04-20T17:36:10.505 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.505 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.505 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:36:10.505 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.493660+0000", 2024-04-20T17:36:10.505 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2653999999999999e-05 2024-04-20T17:36:10.505 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.505 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.505 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:36:10.505 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.493668+0000", 2024-04-20T17:36:10.505 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3790000000000006e-06 2024-04-20T17:36:10.505 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.506 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.506 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:36:10.506 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.493729+0000", 2024-04-20T17:36:10.506 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0769e-05 2024-04-20T17:36:10.506 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.506 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.506 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:36:10.506 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.493851+0000", 2024-04-20T17:36:10.506 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012209800000000001 2024-04-20T17:36:10.506 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.506 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.506 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 9", 2024-04-20T17:36:10.506 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.493983+0000", 2024-04-20T17:36:10.506 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013189300000000001 2024-04-20T17:36:10.507 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.507 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.507 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:36:10.507 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.497715+0000", 2024-04-20T17:36:10.507 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037318339999999998 2024-04-20T17:36:10.507 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:36:10.507 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:36:10.507 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:36:10.507 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.507 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.507 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:232927 6.5 6:aef4ab6a:::benchmark_data_smithi171_147972_object71829:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:36:10.507 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:36:10.498449+0000", 2024-04-20T17:36:10.507 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0024330419999999998, 2024-04-20T17:36:10.507 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0025581499999999999, 2024-04-20T17:36:10.508 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:36:10.508 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:36:10.508 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:36:10.508 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:36:10.508 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:36:10.508 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:36:10.508 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 232927 2024-04-20T17:36:10.508 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.508 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:36:10.508 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.508 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:36:10.508 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.498449+0000", 2024-04-20T17:36:10.508 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:36:10.508 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.509 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.509 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:36:10.509 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.498449+0000", 2024-04-20T17:36:10.509 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:36:10.509 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.509 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.509 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:36:10.509 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.498451+0000", 2024-04-20T17:36:10.509 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9309999999999998e-06 2024-04-20T17:36:10.509 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.509 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.509 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:36:10.509 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.498458+0000", 2024-04-20T17:36:10.509 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4379999999999998e-06 2024-04-20T17:36:10.509 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.510 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.510 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:36:10.510 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.498471+0000", 2024-04-20T17:36:10.510 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3108999999999999e-05 2024-04-20T17:36:10.510 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.510 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.510 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:36:10.510 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.498480+0000", 2024-04-20T17:36:10.510 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.636e-06 2024-04-20T17:36:10.510 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.510 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.510 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:36:10.510 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.498533+0000", 2024-04-20T17:36:10.511 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2834999999999998e-05 2024-04-20T17:36:10.511 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.511 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.511 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:36:10.511 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.498631+0000", 2024-04-20T17:36:10.511 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8152999999999994e-05 2024-04-20T17:36:10.511 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.511 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.511 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 9", 2024-04-20T17:36:10.511 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.498708+0000", 2024-04-20T17:36:10.511 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7298000000000004e-05 2024-04-20T17:36:10.512 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:36:10.512 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:36:10.512 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:36:10.512 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.512 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.512 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:232931 6.8 6:1db000c0:::benchmark_data_smithi171_147972_object71833:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:36:10.512 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:36:10.499401+0000", 2024-04-20T17:36:10.512 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.00148105, 2024-04-20T17:36:10.512 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0016792440000000001, 2024-04-20T17:36:10.512 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:36:10.512 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:36:10.512 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:36:10.513 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:36:10.513 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:36:10.513 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:36:10.513 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 232931 2024-04-20T17:36:10.513 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.513 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:36:10.513 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.513 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:36:10.513 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.499401+0000", 2024-04-20T17:36:10.513 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:36:10.513 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.513 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.513 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:36:10.513 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.499401+0000", 2024-04-20T17:36:10.513 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:36:10.514 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.514 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.514 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:36:10.514 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.499402+0000", 2024-04-20T17:36:10.514 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.17e-06 2024-04-20T17:36:10.514 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.514 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.514 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:36:10.514 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.499406+0000", 2024-04-20T17:36:10.514 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.048e-06 2024-04-20T17:36:10.514 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.514 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.514 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:36:10.514 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.499418+0000", 2024-04-20T17:36:10.515 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2118e-05 2024-04-20T17:36:10.515 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.515 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.515 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:36:10.515 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.499426+0000", 2024-04-20T17:36:10.515 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9219999999999997e-06 2024-04-20T17:36:10.515 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.515 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.515 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:36:10.515 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.499485+0000", 2024-04-20T17:36:10.515 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9171e-05 2024-04-20T17:36:10.515 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.515 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.515 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:36:10.515 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.499595+0000", 2024-04-20T17:36:10.516 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000109623 2024-04-20T17:36:10.516 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:36:10.516 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:36:10.516 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 11", 2024-04-20T17:36:10.516 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:10.499694+0000", 2024-04-20T17:36:10.516 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9767999999999998e-05 2024-04-20T17:36:10.516 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:36:10.516 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:36:10.516 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:36:10.516 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:36:10.516 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-04-20T17:36:10.516 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 3 2024-04-20T17:36:10.516 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:36:10.738 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:36:10.741 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:36:10.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:10.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:36:10.854 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:36:10.901 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:10.903 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:10 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1998414005' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:36:10.903 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:10 smithi005 ceph-mon[39562]: osdmap e662: 12 total, 12 up, 10 in 2024-04-20T17:36:10.903 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:10 smithi005 ceph-mon[39562]: pgmap v1754: 17 pgs: 17 active+clean; 5.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-20T17:36:10.934 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:10 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1998414005' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:36:10.934 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:10 smithi106 ceph-mon[44423]: osdmap e662: 12 total, 12 up, 10 in 2024-04-20T17:36:10.934 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:10 smithi106 ceph-mon[44423]: pgmap v1754: 17 pgs: 17 active+clean; 5.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-20T17:36:10.975 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:36:10.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:10.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:36:11.084 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:36:11.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:10 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1998414005' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:36:11.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:10 smithi171 ceph-mon[44189]: osdmap e662: 12 total, 12 up, 10 in 2024-04-20T17:36:11.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:10 smithi171 ceph-mon[44189]: pgmap v1754: 17 pgs: 17 active+clean; 5.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-20T17:36:11.128 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:11.286 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:36:11.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:11.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:36:11.325 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-20T17:36:11.392 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:36:11.438 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:11.575 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:36:11.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:11.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:36:11.680 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:36:11.730 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:11.734 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:36:11.998 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:36:12.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:12.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:36:12.103 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:36:12.131 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:12.162 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:36:12.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:12.220 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:36:12.321 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:36:12.368 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:12.455 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-04-20T17:36:12.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:12 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/206436132' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:36:12.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:12 smithi005 ceph-mon[39562]: from='client.27023 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:36:12.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:12 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:36:12.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:12 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:36:12.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:12 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/206436132' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:36:12.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:12 smithi106 ceph-mon[44423]: from='client.27023 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:36:12.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:12 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:36:12.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:12 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:36:12.768 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:36:12.853 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:12 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/206436132' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:36:12.853 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:12 smithi171 ceph-mon[44189]: from='client.27023 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:36:12.853 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:12 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:36:12.853 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:12 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:36:13.239 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:36:13.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:13.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:36:13.343 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:36:13.399 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:13.451 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:13 smithi106 ceph-mon[44423]: pgmap v1755: 17 pgs: 17 active+clean; 5.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-20T17:36:13.452 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:13 smithi106 ceph-mon[44423]: from='client.27023 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-04-20T17:36:13.452 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:13 smithi106 ceph-mon[44423]: osdmap e663: 12 total, 12 up, 10 in 2024-04-20T17:36:13.519 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:36:13.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:13.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:36:13.627 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:36:13.713 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:13 smithi171 ceph-mon[44189]: pgmap v1755: 17 pgs: 17 active+clean; 5.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-20T17:36:13.713 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:13 smithi171 ceph-mon[44189]: from='client.27023 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-04-20T17:36:13.713 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:13 smithi171 ceph-mon[44189]: osdmap e663: 12 total, 12 up, 10 in 2024-04-20T17:36:13.715 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:13.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:13 smithi005 ceph-mon[39562]: pgmap v1755: 17 pgs: 17 active+clean; 5.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.19k op/s 2024-04-20T17:36:13.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:13 smithi005 ceph-mon[39562]: from='client.27023 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-04-20T17:36:13.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:13 smithi005 ceph-mon[39562]: osdmap e663: 12 total, 12 up, 10 in 2024-04-20T17:36:13.735 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:36:13.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:13.738 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:36:13.839 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:36:13.879 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:36:13.879 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:36:13.879 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-04-20T17:36:13.879 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-04-20T17:36:13.880 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:36:13.887 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:14.161 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:36:14.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:14.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:36:14.272 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:36:14.326 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:14.376 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:36:14.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:14.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:36:14.484 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:36:14.512 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:14.535 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:36:14.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:14.538 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:36:14.639 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:36:14.666 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:14.688 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:36:14.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:14.693 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:36:14.729 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:14 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:36:14.729 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:14 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:36:14.730 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:14 smithi005 ceph-mon[39562]: osdmap e664: 12 total, 12 up, 10 in 2024-04-20T17:36:14.730 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:14 smithi005 ceph-mon[39562]: pgmap v1758: 17 pgs: 17 active+clean; 5.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.42k op/s 2024-04-20T17:36:14.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:14 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:36:14.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:14 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:36:14.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:14 smithi106 ceph-mon[44423]: osdmap e664: 12 total, 12 up, 10 in 2024-04-20T17:36:14.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:14 smithi106 ceph-mon[44423]: pgmap v1758: 17 pgs: 17 active+clean; 5.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.42k op/s 2024-04-20T17:36:14.795 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:36:14.833 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:14.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:14 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:36:14.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:14 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:36:14.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:14 smithi171 ceph-mon[44189]: osdmap e664: 12 total, 12 up, 10 in 2024-04-20T17:36:14.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:14 smithi171 ceph-mon[44189]: pgmap v1758: 17 pgs: 17 active+clean; 5.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.42k op/s 2024-04-20T17:36:15.215 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:36:15.216 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:36:15.216 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:36:15.216 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:36:15.216 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:36:15.248 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:36:15.275 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:36:15.313 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:36:15.341 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:36:15.402 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:36:15.424 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:36:15.439 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:36:15.480 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:36:15.482 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:36:15.482 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:36:15.482 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:36:15.482 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:36:15.482 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:36:15.482 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:36:15.482 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:36:15.482 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:36:15.482 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:36:15.482 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:36:15.483 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:36:15.483 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:36:15.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:15 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/314857917' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:36:15.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:15.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:15 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/314857917' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:36:15.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:15.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:15 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/314857917' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:36:15.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:16 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:36:16.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:16 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:36:16.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:16 smithi106 ceph-mon[44423]: pgmap v1759: 17 pgs: 17 active+clean; 5.2 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 976 op/s 2024-04-20T17:36:16.798 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:36:16.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:16.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:36:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:16 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:36:16.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:16 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:36:16.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:16 smithi171 ceph-mon[44189]: pgmap v1759: 17 pgs: 17 active+clean; 5.2 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 976 op/s 2024-04-20T17:36:16.835 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.835 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.835 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.835 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:16.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:16 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:36:16.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:16 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:36:16.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:16 smithi005 ceph-mon[39562]: pgmap v1759: 17 pgs: 17 active+clean; 5.2 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 976 op/s 2024-04-20T17:36:16.903 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:36:16.938 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:17.642 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:17 smithi005 ceph-mon[39562]: osdmap e665: 12 total, 12 up, 10 in 2024-04-20T17:36:17.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:17 smithi106 ceph-mon[44423]: osdmap e665: 12 total, 12 up, 10 in 2024-04-20T17:36:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:17 smithi171 ceph-mon[44189]: osdmap e665: 12 total, 12 up, 10 in 2024-04-20T17:36:18.504 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:36:18.504 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:18 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1279638350' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:36:18.504 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:18 smithi106 ceph-mon[44423]: pgmap v1761: 17 pgs: 17 active+clean; 5.1 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-20T17:36:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:18 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1279638350' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:36:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:18 smithi171 ceph-mon[44189]: pgmap v1761: 17 pgs: 17 active+clean; 5.1 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-20T17:36:18.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:18 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1279638350' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:36:18.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:18 smithi005 ceph-mon[39562]: pgmap v1761: 17 pgs: 17 active+clean; 5.1 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-20T17:36:19.000 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:36:19.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:19.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:36:19.027 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-20T17:36:19.114 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:36:19.168 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:19.201 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:36:19.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:19.207 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:36:19.308 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:36:19.345 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:36:19.383 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:19.424 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:36:19.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:19.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:36:19.434 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:36:19.529 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:36:19.564 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:19.586 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:36:19.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:19.591 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:36:19.692 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:36:19.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:19 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:36:19.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:19 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:36:19.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:19 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:36:19.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:19 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:36:19.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:19 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:36:19.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:19 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1279638350' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:36:19.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:19 smithi005 ceph-mon[39562]: osdmap e666: 12 total, 12 up, 10 in 2024-04-20T17:36:19.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:19 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:36:19.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:19 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:36:19.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:19 smithi005 ceph-mon[39562]: 6.a scrub starts 2024-04-20T17:36:19.756 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:19.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:19 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:36:19.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:19 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:36:19.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:19 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:36:19.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:19 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:36:19.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:19 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:36:19.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:19 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1279638350' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:36:19.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:19 smithi106 ceph-mon[44423]: osdmap e666: 12 total, 12 up, 10 in 2024-04-20T17:36:19.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:19 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:36:19.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:19 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:36:19.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:19 smithi106 ceph-mon[44423]: 6.a scrub starts 2024-04-20T17:36:19.787 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:36:19.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:19.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:36:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:19 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:36:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:19 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:36:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:19 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:36:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:19 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:36:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:19 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:36:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:19 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1279638350' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:36:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:19 smithi171 ceph-mon[44189]: osdmap e666: 12 total, 12 up, 10 in 2024-04-20T17:36:19.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:19 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:36:19.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:19 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:36:19.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:19 smithi171 ceph-mon[44189]: 6.a scrub starts 2024-04-20T17:36:19.848 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:36:19.899 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:36:19.944 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:20.158 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:36:20.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:20.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:36:20.267 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:36:20.295 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:20.320 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:36:20.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:20.322 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:36:20.395 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 8, 9, 10, 11, 7] out_osds: [6, 1] dead_osds: [] live_osds: [0, 4, 5, 8, 9, 10, 11, 2, 7, 6, 3, 1] 2024-04-20T17:36:20.396 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 0 1 2024-04-20T17:36:20.423 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:36:20.496 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:20 smithi171 ceph-mon[44189]: osdmap e667: 12 total, 12 up, 10 in 2024-04-20T17:36:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:20 smithi171 ceph-mon[44189]: 6.a scrub ok 2024-04-20T17:36:20.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:20 smithi171 ceph-mon[44189]: pgmap v1764: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-20T17:36:20.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:20 smithi005 ceph-mon[39562]: osdmap e667: 12 total, 12 up, 10 in 2024-04-20T17:36:20.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:20 smithi005 ceph-mon[39562]: 6.a scrub ok 2024-04-20T17:36:20.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:20 smithi005 ceph-mon[39562]: pgmap v1764: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-20T17:36:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:20 smithi106 ceph-mon[44423]: osdmap e667: 12 total, 12 up, 10 in 2024-04-20T17:36:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:20 smithi106 ceph-mon[44423]: 6.a scrub ok 2024-04-20T17:36:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:20 smithi106 ceph-mon[44423]: pgmap v1764: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-20T17:36:21.407 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:36:21.477 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:36:21.477 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:36:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:21 smithi171 ceph-mon[44189]: 6.f scrub starts 2024-04-20T17:36:22.182 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:21 smithi106 ceph-mon[44423]: 6.f scrub starts 2024-04-20T17:36:22.207 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-20T17:36:22.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:21 smithi005 ceph-mon[39562]: 6.f scrub starts 2024-04-20T17:36:22.630 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:36:22.780 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:36:22.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:22.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:36:22.885 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:36:22.914 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:22.940 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:36:22.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:22.945 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:36:23.046 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:36:23.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:22 smithi171 ceph-mon[44189]: 6.f scrub ok 2024-04-20T17:36:23.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:22 smithi171 ceph-mon[44189]: pgmap v1765: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-20T17:36:23.087 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:22 smithi005 ceph-mon[39562]: 6.f scrub ok 2024-04-20T17:36:23.087 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:22 smithi005 ceph-mon[39562]: pgmap v1765: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-20T17:36:23.091 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:23.213 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:22 smithi106 ceph-mon[44423]: 6.f scrub ok 2024-04-20T17:36:23.213 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:22 smithi106 ceph-mon[44423]: pgmap v1765: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-20T17:36:23.307 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:36:23.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:23.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:36:23.384 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:36:23.400 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:36:23.424 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:36:23.455 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:23.559 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:36:23.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:23.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:36:23.665 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:36:23.706 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:23.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:23 smithi005 ceph-mon[39562]: 6.e scrub starts 2024-04-20T17:36:23.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:23 smithi005 ceph-mon[39562]: 6.1 scrub starts 2024-04-20T17:36:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:23 smithi106 ceph-mon[44423]: 6.e scrub starts 2024-04-20T17:36:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:23 smithi106 ceph-mon[44423]: 6.1 scrub starts 2024-04-20T17:36:24.037 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:36:24.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:24.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:36:24.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:23 smithi171 ceph-mon[44189]: 6.e scrub starts 2024-04-20T17:36:24.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:23 smithi171 ceph-mon[44189]: 6.1 scrub starts 2024-04-20T17:36:24.141 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:36:24.198 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:24.422 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:36:24.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:24.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:36:24.527 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:36:24.566 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:24.707 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:36:24.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:24.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:36:24.746 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:36:24.746 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:36:24.810 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:36:24.866 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:25.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:24 smithi106 ceph-mon[44423]: 6.e scrub ok 2024-04-20T17:36:25.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:24 smithi106 ceph-mon[44423]: 6.1 scrub ok 2024-04-20T17:36:25.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:24 smithi106 ceph-mon[44423]: pgmap v1766: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-20T17:36:25.020 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:36:25.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:25.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:36:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:24 smithi171 ceph-mon[44189]: 6.e scrub ok 2024-04-20T17:36:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:24 smithi171 ceph-mon[44189]: 6.1 scrub ok 2024-04-20T17:36:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:24 smithi171 ceph-mon[44189]: pgmap v1766: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-20T17:36:25.109 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:24 smithi005 ceph-mon[39562]: 6.e scrub ok 2024-04-20T17:36:25.109 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:24 smithi005 ceph-mon[39562]: 6.1 scrub ok 2024-04-20T17:36:25.109 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:24 smithi005 ceph-mon[39562]: pgmap v1766: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.24k op/s 2024-04-20T17:36:25.125 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:36:25.172 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:25.530 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-20T17:36:25.532 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:36:25.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:25.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:36:25.637 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:36:25.685 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:25.723 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:36:25.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:25.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:36:25.828 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:36:25.900 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:25.912 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:36:26.194 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:25 smithi106 ceph-mon[44423]: 6.7 scrub starts 2024-04-20T17:36:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:25 smithi005 ceph-mon[39562]: 6.7 scrub starts 2024-04-20T17:36:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:25 smithi171 ceph-mon[44189]: 6.7 scrub starts 2024-04-20T17:36:26.549 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:36:26.579 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:36:26.618 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:36:26.654 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:36:26.687 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:36:26.716 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:36:26.774 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:36:26.784 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:36:26.815 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:36:26.841 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:36:26.878 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:36:26.913 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:36:26.916 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:36:26.916 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:36:26.916 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:36:26.916 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:36:26.917 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:36:26.917 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:36:26.917 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:36:26.917 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:36:26.917 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:36:26.917 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:36:26.917 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:36:26.917 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:36:27.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:27 smithi106 ceph-mon[44423]: 6.7 scrub ok 2024-04-20T17:36:27.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:27 smithi106 ceph-mon[44423]: pgmap v1767: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-20T17:36:27.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:27 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3548212958' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:36:27.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:27.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:27.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:27.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:27.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:27.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:27.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:27.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:27 smithi005 ceph-mon[39562]: 6.7 scrub ok 2024-04-20T17:36:27.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:27 smithi005 ceph-mon[39562]: pgmap v1767: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-20T17:36:27.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:27 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3548212958' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:36:27.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:27.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:27.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:27.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:27.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:27.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:27.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:27 smithi171 ceph-mon[44189]: 6.7 scrub ok 2024-04-20T17:36:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:27 smithi171 ceph-mon[44189]: pgmap v1767: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.26k op/s 2024-04-20T17:36:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:27 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3548212958' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:36:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:28.014 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:36:28.014 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:36:28.014 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-04-20T17:36:28.014 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-04-20T17:36:28.014 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:36:28.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:28 smithi005 ceph-mon[39562]: 6.c scrub starts 2024-04-20T17:36:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:28 smithi005 ceph-mon[39562]: 6.8 scrub starts 2024-04-20T17:36:28.738 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:28.738 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:28.738 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:28.738 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:28.738 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:28.739 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:28 smithi106 ceph-mon[44423]: 6.c scrub starts 2024-04-20T17:36:28.739 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:28 smithi106 ceph-mon[44423]: 6.8 scrub starts 2024-04-20T17:36:28.773 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-20T17:36:28.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:28.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:28.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:28.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:28.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:28.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:28 smithi171 ceph-mon[44189]: 6.c scrub starts 2024-04-20T17:36:28.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:28 smithi171 ceph-mon[44189]: 6.8 scrub starts 2024-04-20T17:36:29.183 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:36:29.399 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:36:29.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:29.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:36:29.485 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:36:29.508 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:36:29.568 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:29.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:29 smithi005 ceph-mon[39562]: 6.8 scrub ok 2024-04-20T17:36:29.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:29 smithi005 ceph-mon[39562]: pgmap v1768: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.4 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:36:29.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:29 smithi005 ceph-mon[39562]: 6.c scrub ok 2024-04-20T17:36:29.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:29 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:36:29.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:29 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:36:29.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:29 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2215038691' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:36:29.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:29 smithi005 ceph-mon[39562]: from='client.27044 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:36:29.746 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:29 smithi106 ceph-mon[44423]: 6.8 scrub ok 2024-04-20T17:36:29.746 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:29 smithi106 ceph-mon[44423]: pgmap v1768: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.4 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:36:29.746 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:29 smithi106 ceph-mon[44423]: 6.c scrub ok 2024-04-20T17:36:29.746 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:29 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:36:29.746 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:29 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:36:29.746 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:29 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2215038691' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:36:29.746 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:29 smithi106 ceph-mon[44423]: from='client.27044 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:36:29.798 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:36:29.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:29.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:36:29.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:29 smithi171 ceph-mon[44189]: 6.8 scrub ok 2024-04-20T17:36:29.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:29 smithi171 ceph-mon[44189]: pgmap v1768: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.4 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:36:29.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:29 smithi171 ceph-mon[44189]: 6.c scrub ok 2024-04-20T17:36:29.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:29 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:36:29.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:29 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:36:29.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:29 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2215038691' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:36:29.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:29 smithi171 ceph-mon[44189]: from='client.27044 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:36:29.902 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:36:29.935 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:29.964 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:36:29.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:29.967 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:36:30.068 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:36:30.097 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:30.124 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:36:30.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:30.131 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:36:30.232 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:36:30.298 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:30.338 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:36:30.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:30.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:36:30.447 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:36:30.465 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:30 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:36:30.466 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:30 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:36:30.466 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:30 smithi106 ceph-mon[44423]: from='client.27044 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:36:30.466 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:30 smithi106 ceph-mon[44423]: osdmap e668: 12 total, 12 up, 10 in 2024-04-20T17:36:30.466 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:30 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2899969415' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:36:30.466 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:30 smithi106 ceph-mon[44423]: from='client.27047 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:36:30.481 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.0 to 1 (10000) 2024-04-20T17:36:30.492 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:30.515 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:36:30.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:30.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:36:30.620 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:36:30.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:30 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:36:30.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:30 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:36:30.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:30 smithi005 ceph-mon[39562]: from='client.27044 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:36:30.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:30 smithi005 ceph-mon[39562]: osdmap e668: 12 total, 12 up, 10 in 2024-04-20T17:36:30.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:30 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2899969415' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:36:30.652 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:30 smithi005 ceph-mon[39562]: from='client.27047 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:36:30.654 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:30.732 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:36:30.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:30.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:36:30.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:30 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:36:30.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:30 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:36:30.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:30 smithi171 ceph-mon[44189]: from='client.27044 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:36:30.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:30 smithi171 ceph-mon[44189]: osdmap e668: 12 total, 12 up, 10 in 2024-04-20T17:36:30.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:30 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2899969415' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:36:30.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:30 smithi171 ceph-mon[44189]: from='client.27047 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:36:30.840 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:36:30.891 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:30.927 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:36:30.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:30.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:36:31.040 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:36:31.076 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:36:31.082 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:31.226 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:36:31.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:31.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:36:31.235 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 2 1 2024-04-20T17:36:31.286 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:36:31.286 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-04-20T17:36:31.286 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-04-20T17:36:31.286 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [] 2024-04-20T17:36:31.286 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:36:31.333 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:36:31.381 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:31.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:31 smithi106 ceph-mon[44423]: 6.5 deep-scrub starts 2024-04-20T17:36:31.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:31 smithi106 ceph-mon[44423]: pgmap v1770: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-20T17:36:31.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:31 smithi106 ceph-mon[44423]: from='client.27047 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-20T17:36:31.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:31 smithi106 ceph-mon[44423]: osdmap e669: 12 total, 12 up, 10 in 2024-04-20T17:36:31.539 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:36:31.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:31.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:36:31.547 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:36:31.645 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:36:31.685 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:31.720 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:36:31.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:31.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:36:31.731 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:31 smithi005 ceph-mon[39562]: 6.5 deep-scrub starts 2024-04-20T17:36:31.731 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:31 smithi005 ceph-mon[39562]: pgmap v1770: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-20T17:36:31.731 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:31 smithi005 ceph-mon[39562]: from='client.27047 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-20T17:36:31.731 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:31 smithi005 ceph-mon[39562]: osdmap e669: 12 total, 12 up, 10 in 2024-04-20T17:36:31.831 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:36:31.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:31 smithi171 ceph-mon[44189]: 6.5 deep-scrub starts 2024-04-20T17:36:31.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:31 smithi171 ceph-mon[44189]: pgmap v1770: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-20T17:36:31.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:31 smithi171 ceph-mon[44189]: from='client.27047 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-20T17:36:31.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:31 smithi171 ceph-mon[44189]: osdmap e669: 12 total, 12 up, 10 in 2024-04-20T17:36:31.886 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:31.923 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:36:31.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:31.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:36:32.033 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:36:32.067 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:32.080 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-20T17:36:32.095 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:36:32.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:32.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:36:32.212 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:36:32.259 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:32.283 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:36:32.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:32.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:36:32.333 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:36:32.390 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:36:32.422 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:32.443 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:36:32.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:32.450 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:36:32.493 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:32 smithi106 ceph-mon[44423]: 6.b scrub starts 2024-04-20T17:36:32.493 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:32 smithi106 ceph-mon[44423]: pgmap v1772: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 965 op/s 2024-04-20T17:36:32.551 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:36:32.594 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:32.626 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:36:32.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:32.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:36:32.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:32 smithi005 ceph-mon[39562]: 6.b scrub starts 2024-04-20T17:36:32.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:32 smithi005 ceph-mon[39562]: pgmap v1772: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 965 op/s 2024-04-20T17:36:32.728 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:36:32.757 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:32 smithi171 ceph-mon[44189]: 6.b scrub starts 2024-04-20T17:36:32.758 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:32 smithi171 ceph-mon[44189]: pgmap v1772: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 965 op/s 2024-04-20T17:36:32.760 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:32.781 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:36:32.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:32.784 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:36:32.885 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:36:32.929 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:33.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:33 smithi106 ceph-mon[44423]: 6.b scrub ok 2024-04-20T17:36:33.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:33 smithi171 ceph-mon[44189]: 6.b scrub ok 2024-04-20T17:36:33.891 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:36:33.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:33.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:36:33.903 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:33 smithi005 ceph-mon[39562]: 6.b scrub ok 2024-04-20T17:36:34.000 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:36:34.037 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:34.061 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:36:34.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:34.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:36:34.167 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:36:34.201 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:34.224 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:36:34.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:34.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:36:34.328 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:36:34.362 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:34.384 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:36:34.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:34.390 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:36:34.470 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:36:34.491 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:36:34.522 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:36:34.537 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:36:34.600 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:34.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:34 smithi005 ceph-mon[39562]: pgmap v1773: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.9 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-20T17:36:34.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:34 smithi171 ceph-mon[44189]: pgmap v1773: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.9 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-20T17:36:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:34 smithi106 ceph-mon[44423]: pgmap v1773: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.9 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 1.14k op/s 2024-04-20T17:36:35.277 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:36:35.309 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:36:35.345 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:36:35.372 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:36:35.409 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:36:35.448 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:36:35.467 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:36:35.502 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:36:35.532 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:36:35.560 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:36:35.584 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:36:35.618 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:36:35.621 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:36:35.621 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:36:35.621 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:36:35.621 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:36:35.621 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:36:35.621 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:36:35.621 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:36:35.622 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:36:35.622 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:36:35.622 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:36:35.622 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:36:35.622 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:36:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:35 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1023328875' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:36:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:35.946 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:35 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1023328875' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:36:35.946 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:35 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1023328875' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:36:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:36.542 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:36:36.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:36.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:36:36.658 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:36:36.694 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:36.736 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:36:36.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:36.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:36:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:36.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:36.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:36.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:36.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:36 smithi171 ceph-mon[44189]: pgmap v1774: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.9 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-20T17:36:36.843 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:36:36.887 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:36.887 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:36.887 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:36.887 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:36.887 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:36.887 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:36.887 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:36.887 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:36.887 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:36.887 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:36.888 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:36.888 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:36 smithi005 ceph-mon[39562]: pgmap v1774: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.9 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-20T17:36:36.889 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:37.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:37.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:37.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:37.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:37.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:37.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:37.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:37.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:37.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:36 smithi106 ceph-mon[44423]: pgmap v1774: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.9 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s 2024-04-20T17:36:37.472 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:36:37.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:37.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:36:37.576 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:36:37.616 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:37.655 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:36:37.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:37.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:36:37.759 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:36:37.792 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:37 smithi171 ceph-mon[44189]: 6.6 scrub starts 2024-04-20T17:36:37.792 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:37 smithi171 ceph-mon[44189]: 6.d deep-scrub starts 2024-04-20T17:36:37.794 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:37.823 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:36:37.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:37.826 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:36:37.927 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:36:37.958 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:37 smithi005 ceph-mon[39562]: 6.6 scrub starts 2024-04-20T17:36:37.958 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:37 smithi005 ceph-mon[39562]: 6.d deep-scrub starts 2024-04-20T17:36:37.962 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:37 smithi106 ceph-mon[44423]: 6.6 scrub starts 2024-04-20T17:36:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:37 smithi106 ceph-mon[44423]: 6.d deep-scrub starts 2024-04-20T17:36:38.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:38 smithi005 ceph-mon[39562]: 6.6 scrub ok 2024-04-20T17:36:38.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:38 smithi005 ceph-mon[39562]: pgmap v1775: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 905 op/s 2024-04-20T17:36:38.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:38 smithi171 ceph-mon[44189]: 6.6 scrub ok 2024-04-20T17:36:38.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:38 smithi171 ceph-mon[44189]: pgmap v1775: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 905 op/s 2024-04-20T17:36:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:38 smithi106 ceph-mon[44423]: 6.6 scrub ok 2024-04-20T17:36:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:38 smithi106 ceph-mon[44423]: pgmap v1775: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 905 op/s 2024-04-20T17:36:39.226 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:36:39.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:39.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:36:39.330 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:36:39.375 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:39.454 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:36:39.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:39.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:36:39.560 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:36:39.590 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:39.620 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:36:39.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:39.625 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:36:39.720 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:39 smithi005 ceph-mon[39562]: 6.2 scrub starts 2024-04-20T17:36:39.720 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:39 smithi005 ceph-mon[39562]: 6.4 scrub starts 2024-04-20T17:36:39.720 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:39 smithi005 ceph-mon[39562]: 6.2 scrub ok 2024-04-20T17:36:39.720 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:39 smithi005 ceph-mon[39562]: 6.4 scrub ok 2024-04-20T17:36:39.726 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:36:39.758 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:39.780 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:36:39.782 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:39 smithi171 ceph-mon[44189]: 6.2 scrub starts 2024-04-20T17:36:39.782 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:39 smithi171 ceph-mon[44189]: 6.4 scrub starts 2024-04-20T17:36:39.782 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:39 smithi171 ceph-mon[44189]: 6.2 scrub ok 2024-04-20T17:36:39.782 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:39 smithi171 ceph-mon[44189]: 6.4 scrub ok 2024-04-20T17:36:39.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:39.784 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:36:39.885 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:36:39.936 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:40.006 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:36:40.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:40.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:36:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:39 smithi106 ceph-mon[44423]: 6.2 scrub starts 2024-04-20T17:36:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:39 smithi106 ceph-mon[44423]: 6.4 scrub starts 2024-04-20T17:36:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:39 smithi106 ceph-mon[44423]: 6.2 scrub ok 2024-04-20T17:36:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:39 smithi106 ceph-mon[44423]: 6.4 scrub ok 2024-04-20T17:36:40.110 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:36:40.145 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:40.440 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:36:40.547 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:36:40.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:40 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/313576002' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:36:40.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:40 smithi005 ceph-mon[39562]: pgmap v1776: 17 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 14 active+clean; 3.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-20T17:36:40.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:40 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/313576002' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:36:40.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:40 smithi171 ceph-mon[44189]: pgmap v1776: 17 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 14 active+clean; 3.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-20T17:36:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:40 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/313576002' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:36:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:40 smithi106 ceph-mon[44423]: pgmap v1776: 17 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 14 active+clean; 3.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-20T17:36:41.800 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:36:41.802 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:41 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:36:41.803 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:41 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/313576002' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:36:41.803 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:41 smithi005 ceph-mon[39562]: osdmap e670: 12 total, 12 up, 10 in 2024-04-20T17:36:41.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:41.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:36:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:41 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:36:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:41 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/313576002' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:36:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:41 smithi171 ceph-mon[44189]: osdmap e670: 12 total, 12 up, 10 in 2024-04-20T17:36:41.905 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:36:41.944 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:41.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:41 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:36:41.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:41 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/313576002' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:36:41.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:41 smithi106 ceph-mon[44423]: osdmap e670: 12 total, 12 up, 10 in 2024-04-20T17:36:41.972 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:36:41.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:41.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:36:42.078 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:36:42.109 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:42.131 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:36:42.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:42.134 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:36:42.235 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:36:42.264 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:42.298 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:36:42.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:42.301 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:36:42.402 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:36:42.430 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:42.454 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:36:42.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:42.458 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:36:42.558 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.2 to 1 (10000) 2024-04-20T17:36:42.559 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:36:42.594 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:42.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:42 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:36:42.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:42 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:36:42.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:42 smithi171 ceph-mon[44189]: osdmap e671: 12 total, 12 up, 10 in 2024-04-20T17:36:42.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:42 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/680620415' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-20T17:36:42.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:42 smithi171 ceph-mon[44189]: pgmap v1779: 17 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 14 active+clean; 3.5 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 826 op/s 2024-04-20T17:36:42.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:42 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:36:42.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:42 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:36:42.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:42 smithi005 ceph-mon[39562]: osdmap e671: 12 total, 12 up, 10 in 2024-04-20T17:36:42.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:42 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/680620415' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-20T17:36:42.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:42 smithi005 ceph-mon[39562]: pgmap v1779: 17 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 14 active+clean; 3.5 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 826 op/s 2024-04-20T17:36:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:42 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:36:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:42 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:36:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:42 smithi106 ceph-mon[44423]: osdmap e671: 12 total, 12 up, 10 in 2024-04-20T17:36:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:42 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/680620415' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-20T17:36:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:42 smithi106 ceph-mon[44423]: pgmap v1779: 17 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 14 active+clean; 3.5 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 826 op/s 2024-04-20T17:36:43.654 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:36:43.655 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:36:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:43 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/680620415' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-20T17:36:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:43 smithi171 ceph-mon[44189]: osdmap e672: 12 total, 12 up, 10 in 2024-04-20T17:36:43.846 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:36:43.848 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:36:43.851 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:43 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/680620415' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-20T17:36:43.851 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:43 smithi005 ceph-mon[39562]: osdmap e672: 12 total, 12 up, 10 in 2024-04-20T17:36:43.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:43.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:36:43.953 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:36:43.987 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:44.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:43 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/680620415' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-20T17:36:44.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:43 smithi106 ceph-mon[44423]: osdmap e672: 12 total, 12 up, 10 in 2024-04-20T17:36:44.016 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:36:44.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:44.020 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:36:44.121 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:36:44.165 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:44.199 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:36:44.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:44.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:36:44.302 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:36:44.331 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:44.352 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:36:44.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:44.355 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:36:44.456 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:36:44.507 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:44.533 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-20T17:36:44.534 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:36:44.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:44.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:36:44.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:44 smithi171 ceph-mon[44189]: pgmap v1781: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 3.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-20T17:36:44.637 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:36:44.670 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:44.698 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:36:44.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:44.701 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:36:44.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:44 smithi106 ceph-mon[44423]: pgmap v1781: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 3.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-20T17:36:44.801 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:36:44.833 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:44 smithi005 ceph-mon[39562]: pgmap v1781: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 3.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-20T17:36:44.836 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:44.920 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:36:44.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:44.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:36:45.025 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:36:45.079 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:45.102 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 3 1 2024-04-20T17:36:45.225 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:36:45.561 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:36:45.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:45.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:36:45.672 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:36:45.711 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:45.739 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:36:45.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:45.743 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:36:45.844 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:36:45.875 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:46 smithi171 ceph-mon[44189]: pgmap v1782: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 3.3 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 799 op/s 2024-04-20T17:36:46.835 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:36:46.840 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:46 smithi005 ceph-mon[39562]: pgmap v1782: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 3.3 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 799 op/s 2024-04-20T17:36:46.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:46.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:36:46.852 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:36:46.922 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:36:46.942 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:36:46.976 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:46 smithi106 ceph-mon[44423]: pgmap v1782: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 3.3 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 799 op/s 2024-04-20T17:36:46.979 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:47.009 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:36:47.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:47.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:36:47.119 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:36:47.119 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:36:47.171 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:47.389 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:36:47.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:47.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:36:47.494 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:36:47.540 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:47.574 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:36:47.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:47.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:36:47.658 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:47 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:36:47.658 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:47 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:36:47.680 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:36:47.717 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:47.741 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:36:47.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:47.745 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:36:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:47 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:36:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:47 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:36:47.846 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:36:47.878 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:47.908 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:36:47.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:47.912 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:36:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:47 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:36:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:47 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:36:48.013 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:36:48.046 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:48.075 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:36:48.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:48.083 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:36:48.184 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:36:48.235 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:48.446 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:36:48.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:48.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:36:48.549 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:36:48.604 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:48.742 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:36:48.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:48.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:36:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:48 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:36:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:48 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:36:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:48 smithi171 ceph-mon[44189]: osdmap e673: 12 total, 12 up, 10 in 2024-04-20T17:36:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:48 smithi171 ceph-mon[44189]: pgmap v1784: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 3.2 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 770 op/s 2024-04-20T17:36:48.849 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:36:48.905 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:48 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:36:48.905 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:48 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:36:48.905 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:48 smithi005 ceph-mon[39562]: osdmap e673: 12 total, 12 up, 10 in 2024-04-20T17:36:48.905 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:48 smithi005 ceph-mon[39562]: pgmap v1784: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 3.2 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 770 op/s 2024-04-20T17:36:48.911 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:48 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:36:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:48 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:36:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:48 smithi106 ceph-mon[44423]: osdmap e673: 12 total, 12 up, 10 in 2024-04-20T17:36:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:48 smithi106 ceph-mon[44423]: pgmap v1784: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 3.2 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 770 op/s 2024-04-20T17:36:49.067 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:36:49.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:49.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:36:49.174 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:36:49.206 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:50.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:50 smithi171 ceph-mon[44189]: pgmap v1785: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 3.1 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 969 op/s 2024-04-20T17:36:50.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:50 smithi005 ceph-mon[39562]: pgmap v1785: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 3.1 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 969 op/s 2024-04-20T17:36:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:50 smithi106 ceph-mon[44423]: pgmap v1785: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 3.1 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 969 op/s 2024-04-20T17:36:51.600 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:36:51.600 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:36:51.600 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:36:51.600 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:36:51.600 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:36:51.600 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:36:51.600 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:36:51.600 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:36:51.600 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:36:51.600 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:36:51.600 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:36:51.600 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:36:51.601 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:36:51.601 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:36:51.601 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:36:51.601 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:36:51.601 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:36:51.601 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:36:51.601 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:36:51.601 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:36:51.601 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:36:51.601 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:36:51.601 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:36:51.601 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:36:51.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:51 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4216311248' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:36:51.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:51.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:51.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:51 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4216311248' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:36:51.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:51.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:51 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4216311248' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:36:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.476 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:36:52.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:52.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:36:52.577 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:36:52.578 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:36:52.578 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:36:52.578 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:36:52.578 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:36:52.587 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:36:52.616 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:52.645 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:36:52.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:52.653 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:36:52.754 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:36:52.806 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.806 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.806 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.807 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.807 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.807 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.807 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.807 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.807 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.807 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.807 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:52 smithi005 ceph-mon[39562]: pgmap v1786: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 3.0 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 860 op/s 2024-04-20T17:36:52.809 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.809 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.809 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.809 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.809 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.809 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.809 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.809 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.809 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.809 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.809 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:52 smithi106 ceph-mon[44423]: pgmap v1786: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 3.0 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 860 op/s 2024-04-20T17:36:52.812 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:52.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:36:52.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:52 smithi171 ceph-mon[44189]: pgmap v1786: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 3.0 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 860 op/s 2024-04-20T17:36:52.843 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:36:52.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:52.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:36:52.948 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:36:52.980 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:53.555 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.3 to 1 (10000) 2024-04-20T17:36:53.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:53 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/682595733' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:36:53.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:53 smithi005 ceph-mon[39562]: from='client.27077 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:36:53.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:53 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/682595733' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:36:53.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:53 smithi171 ceph-mon[44189]: from='client.27077 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:36:53.860 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:36:53.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:53.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:36:53.899 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-20T17:36:53.969 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:36:54.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:53 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/682595733' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:36:54.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:53 smithi106 ceph-mon[44423]: from='client.27077 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:36:54.033 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:54.607 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:36:54.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:54.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:36:54.712 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:36:54.755 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:54.779 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:36:54.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:54.782 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:36:54.825 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:54 smithi005 ceph-mon[39562]: 6.5 deep-scrub ok 2024-04-20T17:36:54.825 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:54 smithi005 ceph-mon[39562]: from='client.27077 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-20T17:36:54.825 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:54 smithi005 ceph-mon[39562]: osdmap e674: 12 total, 12 up, 10 in 2024-04-20T17:36:54.825 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:54 smithi005 ceph-mon[39562]: pgmap v1788: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 2.9 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 876 op/s 2024-04-20T17:36:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:54 smithi171 ceph-mon[44189]: 6.5 deep-scrub ok 2024-04-20T17:36:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:54 smithi171 ceph-mon[44189]: from='client.27077 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-20T17:36:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:54 smithi171 ceph-mon[44189]: osdmap e674: 12 total, 12 up, 10 in 2024-04-20T17:36:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:54 smithi171 ceph-mon[44189]: pgmap v1788: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 2.9 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 876 op/s 2024-04-20T17:36:54.883 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:36:54.937 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:55.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:54 smithi106 ceph-mon[44423]: 6.5 deep-scrub ok 2024-04-20T17:36:55.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:54 smithi106 ceph-mon[44423]: from='client.27077 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-20T17:36:55.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:54 smithi106 ceph-mon[44423]: osdmap e674: 12 total, 12 up, 10 in 2024-04-20T17:36:55.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:54 smithi106 ceph-mon[44423]: pgmap v1788: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 2.9 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 876 op/s 2024-04-20T17:36:55.298 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:36:55.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:55.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:36:55.342 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 4 1 2024-04-20T17:36:55.402 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:36:55.429 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:36:55.504 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:55.557 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:55 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3678147068' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:36:55.558 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:55 smithi005 ceph-mon[39562]: from='client.26974 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:36:55.558 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:55 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:36:55.572 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:36:55.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:55 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3678147068' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:36:55.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:55 smithi171 ceph-mon[44189]: from='client.26974 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:36:55.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:55 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:36:55.965 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:36:55.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:55.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:36:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:55 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3678147068' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:36:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:55 smithi106 ceph-mon[44423]: from='client.26974 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:36:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:55 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:36:56.073 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:36:56.125 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:56.402 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:36:56.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:56.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:36:56.506 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:36:56.608 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:36:56.628 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:56.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:56 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:36:56.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:56 smithi005 ceph-mon[39562]: from='client.26974 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:36:56.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:56 smithi005 ceph-mon[39562]: osdmap e675: 12 total, 12 up, 10 in 2024-04-20T17:36:56.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:56 smithi005 ceph-mon[39562]: pgmap v1790: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 2.8 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 964 op/s 2024-04-20T17:36:56.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:56 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:36:56.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:56 smithi171 ceph-mon[44189]: from='client.26974 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:36:56.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:56 smithi171 ceph-mon[44189]: osdmap e675: 12 total, 12 up, 10 in 2024-04-20T17:36:56.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:56 smithi171 ceph-mon[44189]: pgmap v1790: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 2.8 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 964 op/s 2024-04-20T17:36:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:56 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:36:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:56 smithi106 ceph-mon[44423]: from='client.26974 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:36:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:56 smithi106 ceph-mon[44423]: osdmap e675: 12 total, 12 up, 10 in 2024-04-20T17:36:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:56 smithi106 ceph-mon[44423]: pgmap v1790: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 2.8 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 964 op/s 2024-04-20T17:36:57.679 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:36:57.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:57.683 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:36:57.784 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:36:57.839 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:57.858 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:36:57.898 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:36:57.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:57.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:36:57.915 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:36:57.932 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:36:57.932 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:36:57.932 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:36:57.932 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:36:57.932 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.932 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:4 6.4 6.e10b0ab4 (undecoded) ondisk+retry+write+known_if_redirected e626)", 2024-04-20T17:36:57.932 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:33:43.113943+0000", 2024-04-20T17:36:57.932 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 194.81530574999999, 2024-04-20T17:36:57.933 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.176808986, 2024-04-20T17:36:57.933 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:57.933 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:57.933 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-04-20T17:36:57.933 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:57.933 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:36:57.933 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:36:57.933 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 4 2024-04-20T17:36:57.933 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.933 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:57.933 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.933 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:57.933 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.113943+0000", 2024-04-20T17:36:57.933 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:57.934 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.934 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.934 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:57.934 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.113943+0000", 2024-04-20T17:36:57.934 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:57.934 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.934 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.934 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:57.934 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.113946+0000", 2024-04-20T17:36:57.934 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.954e-06 2024-04-20T17:36:57.934 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.934 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.935 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:57.935 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114087+0000", 2024-04-20T17:36:57.935 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000140672 2024-04-20T17:36:57.935 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.935 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.935 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:57.935 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114109+0000", 2024-04-20T17:36:57.935 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2314e-05 2024-04-20T17:36:57.935 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.935 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.935 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:57.935 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114125+0000", 2024-04-20T17:36:57.935 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.579e-05 2024-04-20T17:36:57.935 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.936 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.936 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:57.936 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290710+0000", 2024-04-20T17:36:57.936 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17658464300000001 2024-04-20T17:36:57.936 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.936 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.936 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:57.936 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290752+0000", 2024-04-20T17:36:57.936 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2613e-05 2024-04-20T17:36:57.937 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:57.937 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:57.937 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:57.937 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.937 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.937 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:5 6.4 6.d573554 (undecoded) ondisk+retry+write+known_if_redirected e626)", 2024-04-20T17:36:57.937 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:33:43.114145+0000", 2024-04-20T17:36:57.937 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 194.815103975, 2024-04-20T17:36:57.937 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17667517699999999, 2024-04-20T17:36:57.937 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:57.937 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:57.937 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-04-20T17:36:57.937 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:57.937 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:36:57.938 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:36:57.938 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 5 2024-04-20T17:36:57.938 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.938 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:57.938 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.938 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:57.939 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114145+0000", 2024-04-20T17:36:57.939 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:57.939 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.939 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.939 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:57.939 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114145+0000", 2024-04-20T17:36:57.939 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:57.939 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.940 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.940 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:57.940 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114151+0000", 2024-04-20T17:36:57.940 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0519999999999998e-06 2024-04-20T17:36:57.940 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.940 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.940 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:57.940 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114178+0000", 2024-04-20T17:36:57.941 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6842000000000002e-05 2024-04-20T17:36:57.941 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.941 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.941 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:57.941 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114183+0000", 2024-04-20T17:36:57.941 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4210000000000003e-06 2024-04-20T17:36:57.941 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.941 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.942 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:57.942 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114187+0000", 2024-04-20T17:36:57.942 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0940000000000001e-06 2024-04-20T17:36:57.942 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.942 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.942 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:57.943 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290813+0000", 2024-04-20T17:36:57.943 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17662583400000001 2024-04-20T17:36:57.943 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.943 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.943 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:57.943 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290820+0000", 2024-04-20T17:36:57.943 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.934e-06 2024-04-20T17:36:57.944 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:57.944 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:57.944 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:57.944 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.944 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.944 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:19 6.4 6.9dea5f14 (undecoded) ondisk+write+known_if_redirected e626)", 2024-04-20T17:36:57.944 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:33:43.114197+0000", 2024-04-20T17:36:57.944 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 194.81505215000001, 2024-04-20T17:36:57.945 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17665645799999999, 2024-04-20T17:36:57.945 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:57.945 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:57.945 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-04-20T17:36:57.945 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:57.945 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:36:57.945 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:36:57.945 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 19 2024-04-20T17:36:57.945 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.945 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:57.945 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.945 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:57.945 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114197+0000", 2024-04-20T17:36:57.945 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:57.946 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.946 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.946 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:57.946 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114197+0000", 2024-04-20T17:36:57.946 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:57.946 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.946 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.946 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:57.946 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114197+0000", 2024-04-20T17:36:57.946 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2399999999999998e-07 2024-04-20T17:36:57.947 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.947 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.947 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:57.947 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114282+0000", 2024-04-20T17:36:57.947 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4831000000000001e-05 2024-04-20T17:36:57.947 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.947 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.947 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:57.947 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114289+0000", 2024-04-20T17:36:57.947 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0790000000000003e-06 2024-04-20T17:36:57.947 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.947 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.947 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:57.947 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114293+0000", 2024-04-20T17:36:57.948 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9199999999999997e-06 2024-04-20T17:36:57.948 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.948 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.948 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:57.948 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290840+0000", 2024-04-20T17:36:57.948 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17654666999999999 2024-04-20T17:36:57.948 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.948 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.948 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:57.948 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290853+0000", 2024-04-20T17:36:57.948 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3434e-05 2024-04-20T17:36:57.948 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:57.948 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:57.948 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:57.949 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.949 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.949 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:30 6.4 6.853ac614 (undecoded) ondisk+write+known_if_redirected e626)", 2024-04-20T17:36:57.949 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:33:43.114301+0000", 2024-04-20T17:36:57.949 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 194.81494762400001, 2024-04-20T17:36:57.949 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.176585081, 2024-04-20T17:36:57.949 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:57.949 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:57.949 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-04-20T17:36:57.949 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:57.949 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:36:57.949 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:36:57.949 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 30 2024-04-20T17:36:57.949 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.949 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:57.949 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.950 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:57.950 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114301+0000", 2024-04-20T17:36:57.950 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:57.950 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.950 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.950 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:57.950 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114301+0000", 2024-04-20T17:36:57.950 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:57.950 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.950 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.950 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:57.951 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114302+0000", 2024-04-20T17:36:57.951 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5700000000000002e-07 2024-04-20T17:36:57.951 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.951 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.951 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:57.951 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114354+0000", 2024-04-20T17:36:57.952 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2394e-05 2024-04-20T17:36:57.952 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.952 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.952 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:57.952 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114361+0000", 2024-04-20T17:36:57.952 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4269999999999996e-06 2024-04-20T17:36:57.953 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.953 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.953 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:57.953 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114365+0000", 2024-04-20T17:36:57.953 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3699999999999997e-06 2024-04-20T17:36:57.953 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.954 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.954 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:57.954 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290877+0000", 2024-04-20T17:36:57.954 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.176511735 2024-04-20T17:36:57.954 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.954 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.954 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:57.954 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290886+0000", 2024-04-20T17:36:57.954 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.5980000000000003e-06 2024-04-20T17:36:57.954 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:57.954 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:57.954 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:57.954 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.955 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.955 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:35 6.4 6.16044024 (undecoded) ondisk+write+known_if_redirected e626)", 2024-04-20T17:36:57.955 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:33:43.114373+0000", 2024-04-20T17:36:57.955 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 194.81487592799999, 2024-04-20T17:36:57.955 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17653855700000001, 2024-04-20T17:36:57.955 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:57.955 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:57.955 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-04-20T17:36:57.955 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:57.955 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:36:57.955 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:36:57.955 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 35 2024-04-20T17:36:57.955 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.955 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:57.956 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.956 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:57.956 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114373+0000", 2024-04-20T17:36:57.956 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:57.956 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.956 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.956 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:57.956 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114373+0000", 2024-04-20T17:36:57.956 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:57.956 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.956 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.956 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:57.956 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114373+0000", 2024-04-20T17:36:57.957 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0999999999999999e-07 2024-04-20T17:36:57.957 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.957 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.957 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:57.957 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114419+0000", 2024-04-20T17:36:57.957 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5123999999999997e-05 2024-04-20T17:36:57.957 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.957 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.957 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:57.957 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114425+0000", 2024-04-20T17:36:57.957 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0970000000000002e-06 2024-04-20T17:36:57.957 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.957 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.957 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:57.958 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114429+0000", 2024-04-20T17:36:57.958 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4089999999999996e-06 2024-04-20T17:36:57.958 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.958 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.958 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:57.958 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290905+0000", 2024-04-20T17:36:57.958 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.176475935 2024-04-20T17:36:57.958 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.958 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.958 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:57.958 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290912+0000", 2024-04-20T17:36:57.958 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4819999999999999e-06 2024-04-20T17:36:57.958 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:57.959 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:57.959 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:57.959 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.959 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.959 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:37 6.4 6.1a00bf44 (undecoded) ondisk+write+known_if_redirected e626)", 2024-04-20T17:36:57.959 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:33:43.114438+0000", 2024-04-20T17:36:57.959 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 194.81481067999999, 2024-04-20T17:36:57.959 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17650761100000001, 2024-04-20T17:36:57.959 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:57.959 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:57.959 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-04-20T17:36:57.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:57.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:36:57.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:36:57.959 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 37 2024-04-20T17:36:57.960 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.960 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:57.960 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.960 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:57.960 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114438+0000", 2024-04-20T17:36:57.960 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:57.960 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.960 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.960 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:57.960 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114438+0000", 2024-04-20T17:36:57.960 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:57.960 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.960 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.960 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:57.960 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114439+0000", 2024-04-20T17:36:57.961 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5499999999999998e-07 2024-04-20T17:36:57.961 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.961 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.961 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:57.961 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114481+0000", 2024-04-20T17:36:57.961 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1878000000000001e-05 2024-04-20T17:36:57.961 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.961 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.961 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:57.961 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114487+0000", 2024-04-20T17:36:57.961 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9200000000000001e-06 2024-04-20T17:36:57.961 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.961 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.962 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:57.962 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114490+0000", 2024-04-20T17:36:57.962 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8510000000000001e-06 2024-04-20T17:36:57.962 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.962 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.962 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:57.962 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290940+0000", 2024-04-20T17:36:57.962 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17644975700000001 2024-04-20T17:36:57.962 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.962 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.962 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:57.962 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290946+0000", 2024-04-20T17:36:57.963 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6500000000000001e-06 2024-04-20T17:36:57.963 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:57.963 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:57.963 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:57.963 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.963 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.963 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:44 6.4 6.58250294 (undecoded) ondisk+write+known_if_redirected e626)", 2024-04-20T17:36:57.963 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:33:43.114498+0000", 2024-04-20T17:36:57.963 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 194.814751355, 2024-04-20T17:36:57.963 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17647228700000001, 2024-04-20T17:36:57.963 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:57.963 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:57.964 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-04-20T17:36:57.964 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:57.964 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:36:57.964 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:36:57.964 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 44 2024-04-20T17:36:57.964 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.964 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:57.964 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.964 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:57.964 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114498+0000", 2024-04-20T17:36:57.964 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:57.964 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.964 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.964 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:57.965 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114498+0000", 2024-04-20T17:36:57.965 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:57.965 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.965 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.965 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:57.965 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114498+0000", 2024-04-20T17:36:57.965 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5499999999999998e-07 2024-04-20T17:36:57.965 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.965 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.965 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:57.965 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114554+0000", 2024-04-20T17:36:57.965 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6028e-05 2024-04-20T17:36:57.965 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.965 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.965 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:57.966 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114561+0000", 2024-04-20T17:36:57.966 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5490000000000003e-06 2024-04-20T17:36:57.966 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.966 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.966 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:57.966 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.114564+0000", 2024-04-20T17:36:57.966 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.72e-06 2024-04-20T17:36:57.966 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.966 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.966 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:57.966 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290965+0000", 2024-04-20T17:36:57.966 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17640072300000001 2024-04-20T17:36:57.966 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.966 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.967 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:57.967 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:33:43.290970+0000", 2024-04-20T17:36:57.967 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8119999999999998e-06 2024-04-20T17:36:57.967 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:57.967 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:57.967 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:57.967 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.967 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.967 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26882.0:63117 6.0 e633/619 6:0af12735:::benchmark_data_smithi171_147972_object63116:head v 633'3936, mlcod=633'3933)", 2024-04-20T17:36:57.967 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:34:02.225578+0000", 2024-04-20T17:36:57.967 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 175.70367098099999, 2024-04-20T17:36:57.967 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10437700799999999, 2024-04-20T17:36:57.968 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:57.968 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:57.968 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:57.968 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:57.968 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.968 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:57.968 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.225578+0000", 2024-04-20T17:36:57.968 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:57.968 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.968 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.968 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:57.968 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.225578+0000", 2024-04-20T17:36:57.968 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:57.968 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.969 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.969 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:57.969 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.225579+0000", 2024-04-20T17:36:57.969 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.217e-06 2024-04-20T17:36:57.969 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.969 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.969 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:57.969 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.225603+0000", 2024-04-20T17:36:57.969 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3592e-05 2024-04-20T17:36:57.969 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.969 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.969 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:57.969 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.225611+0000", 2024-04-20T17:36:57.969 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6829999999999997e-06 2024-04-20T17:36:57.969 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.970 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.970 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:57.970 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.225618+0000", 2024-04-20T17:36:57.970 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.387e-06 2024-04-20T17:36:57.970 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.970 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.970 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:57.970 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.226225+0000", 2024-04-20T17:36:57.970 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00060705799999999995 2024-04-20T17:36:57.970 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.970 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.970 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:57.970 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.226244+0000", 2024-04-20T17:36:57.971 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9032e-05 2024-04-20T17:36:57.971 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.971 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.971 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:57.971 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.329906+0000", 2024-04-20T17:36:57.971 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.103662165 2024-04-20T17:36:57.971 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.971 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.971 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:57.971 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.329955+0000", 2024-04-20T17:36:57.971 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8874e-05 2024-04-20T17:36:57.971 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:57.971 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:57.972 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:57.972 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.972 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.972 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:63122 6.4 6:201b6f74:::benchmark_data_smithi171_147972_object63121:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e633)", 2024-04-20T17:36:57.972 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:34:02.225613+0000", 2024-04-20T17:36:57.972 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 175.70363603199999, 2024-04-20T17:36:57.972 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.105414228, 2024-04-20T17:36:57.972 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:57.972 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:57.972 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:57.972 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:57.972 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:36:57.972 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:36:57.972 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 63122 2024-04-20T17:36:57.972 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.973 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:57.973 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.973 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:57.973 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.225613+0000", 2024-04-20T17:36:57.973 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:57.973 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.973 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.973 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:57.973 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.225613+0000", 2024-04-20T17:36:57.973 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:57.973 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.973 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.973 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:57.973 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.225614+0000", 2024-04-20T17:36:57.973 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1799999999999999e-06 2024-04-20T17:36:57.973 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.974 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.974 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:57.974 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.225674+0000", 2024-04-20T17:36:57.974 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9898999999999997e-05 2024-04-20T17:36:57.974 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.974 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.974 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:57.974 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.225684+0000", 2024-04-20T17:36:57.974 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6420000000000002e-06 2024-04-20T17:36:57.974 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.974 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.974 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:57.974 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.225691+0000", 2024-04-20T17:36:57.974 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5610000000000001e-06 2024-04-20T17:36:57.974 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.975 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.975 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:57.975 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.225743+0000", 2024-04-20T17:36:57.975 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2104999999999997e-05 2024-04-20T17:36:57.975 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.975 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.975 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:57.975 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.225856+0000", 2024-04-20T17:36:57.975 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011268999999999999 2024-04-20T17:36:57.975 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.975 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.975 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:36:57.975 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.225976+0000", 2024-04-20T17:36:57.975 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000119608 2024-04-20T17:36:57.975 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.976 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.976 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:57.976 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.326463+0000", 2024-04-20T17:36:57.976 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.100487496 2024-04-20T17:36:57.976 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.976 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.976 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:57.976 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.330935+0000", 2024-04-20T17:36:57.976 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0044723970000000004 2024-04-20T17:36:57.976 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.976 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.976 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:57.976 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.330985+0000", 2024-04-20T17:36:57.976 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9583999999999998e-05 2024-04-20T17:36:57.977 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.977 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.977 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:57.977 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.331027+0000", 2024-04-20T17:36:57.977 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2066e-05 2024-04-20T17:36:57.977 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:57.977 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:57.977 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:57.977 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.977 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.977 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:63126 6.b 6:d5c3020f:::benchmark_data_smithi171_147972_object63125:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e633)", 2024-04-20T17:36:57.977 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:34:02.228094+0000", 2024-04-20T17:36:57.977 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 175.70115528599999, 2024-04-20T17:36:57.977 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.103377888, 2024-04-20T17:36:57.977 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:57.977 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:57.977 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:57.978 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:57.978 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:36:57.978 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:36:57.978 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 63126 2024-04-20T17:36:57.978 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.978 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:57.978 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.978 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:57.978 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.228094+0000", 2024-04-20T17:36:57.978 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:57.978 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.978 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.978 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:57.979 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.228094+0000", 2024-04-20T17:36:57.979 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:57.979 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.979 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.979 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:57.979 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.228095+0000", 2024-04-20T17:36:57.979 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8670000000000001e-06 2024-04-20T17:36:57.979 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.979 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.979 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:57.979 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.228159+0000", 2024-04-20T17:36:57.979 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3225999999999994e-05 2024-04-20T17:36:57.979 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.979 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.979 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:57.980 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.228168+0000", 2024-04-20T17:36:57.980 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0000000000000002e-06 2024-04-20T17:36:57.980 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.980 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.980 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:57.980 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.228174+0000", 2024-04-20T17:36:57.980 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7549999999999997e-06 2024-04-20T17:36:57.980 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.980 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.980 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:57.980 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.228218+0000", 2024-04-20T17:36:57.980 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3597999999999998e-05 2024-04-20T17:36:57.980 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.980 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.980 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:57.981 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.228346+0000", 2024-04-20T17:36:57.981 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012748800000000001 2024-04-20T17:36:57.981 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.981 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.981 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 7", 2024-04-20T17:36:57.981 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.324999+0000", 2024-04-20T17:36:57.981 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.096653506 2024-04-20T17:36:57.981 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.981 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.981 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:57.981 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.331180+0000", 2024-04-20T17:36:57.981 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0061809489999999998 2024-04-20T17:36:57.982 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.982 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.982 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:57.982 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.331374+0000", 2024-04-20T17:36:57.982 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00019414200000000001 2024-04-20T17:36:57.982 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.982 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.982 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:57.982 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.331430+0000", 2024-04-20T17:36:57.982 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5473999999999998e-05 2024-04-20T17:36:57.982 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.982 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.982 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:57.982 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:34:02.331472+0000", 2024-04-20T17:36:57.982 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1882999999999999e-05 2024-04-20T17:36:57.983 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:57.983 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:57.983 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:57.983 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.983 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.983 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:155961 6.6 6:62ebcb10:::benchmark_data_smithi171_147972_object155960:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e646)", 2024-04-20T17:36:57.983 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:35:04.114739+0000", 2024-04-20T17:36:57.983 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 113.81450968599999, 2024-04-20T17:36:57.983 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.124179953, 2024-04-20T17:36:57.983 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:57.983 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:57.983 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:57.983 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:57.983 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:36:57.983 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:36:57.984 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 155961 2024-04-20T17:36:57.984 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.984 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:57.984 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.984 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:57.984 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:04.114739+0000", 2024-04-20T17:36:57.984 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:57.984 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.984 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.984 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:57.984 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:04.114739+0000", 2024-04-20T17:36:57.984 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:57.984 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.984 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.985 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:57.985 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:04.114740+0000", 2024-04-20T17:36:57.985 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.046e-06 2024-04-20T17:36:57.985 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.985 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.985 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:57.985 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:04.114785+0000", 2024-04-20T17:36:57.985 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5170999999999998e-05 2024-04-20T17:36:57.985 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.985 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.985 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:57.985 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:04.114792+0000", 2024-04-20T17:36:57.985 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5540000000000003e-06 2024-04-20T17:36:57.985 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.986 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.986 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:57.986 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:04.114800+0000", 2024-04-20T17:36:57.986 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4549999999999995e-06 2024-04-20T17:36:57.986 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.986 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.986 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:57.986 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:04.114848+0000", 2024-04-20T17:36:57.986 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7352999999999999e-05 2024-04-20T17:36:57.986 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.986 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.987 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:57.987 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:04.114962+0000", 2024-04-20T17:36:57.987 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011466700000000001 2024-04-20T17:36:57.987 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.987 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.987 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 4", 2024-04-20T17:36:57.987 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:04.115090+0000", 2024-04-20T17:36:57.987 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000127835 2024-04-20T17:36:57.987 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.987 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.987 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:57.987 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:04.120513+0000", 2024-04-20T17:36:57.987 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0054230579999999997 2024-04-20T17:36:57.987 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.987 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.987 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:57.987 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:04.238796+0000", 2024-04-20T17:36:57.988 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.118282963 2024-04-20T17:36:57.988 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.988 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.988 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:57.988 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:04.238852+0000", 2024-04-20T17:36:57.988 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5179000000000001e-05 2024-04-20T17:36:57.988 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.988 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.989 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:57.989 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:04.238919+0000", 2024-04-20T17:36:57.989 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7671999999999999e-05 2024-04-20T17:36:57.989 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:57.989 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:57.989 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:57.989 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.989 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.989 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:180898 6.4 6:252deb3b:::benchmark_data_smithi171_147972_object19800:head [delete] snapc 0=[] ondisk+write+known_if_redirected e653)", 2024-04-20T17:36:57.989 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:35:26.369439+0000", 2024-04-20T17:36:57.989 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 91.559810174000006, 2024-04-20T17:36:57.989 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.122666505, 2024-04-20T17:36:57.989 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:57.989 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:57.990 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:57.990 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:57.990 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:36:57.990 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:36:57.990 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 180898 2024-04-20T17:36:57.990 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.990 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:57.990 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.990 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:57.990 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.369439+0000", 2024-04-20T17:36:57.990 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:57.990 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.990 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.990 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:57.990 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.369439+0000", 2024-04-20T17:36:57.991 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:57.991 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.991 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.991 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:57.991 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.369441+0000", 2024-04-20T17:36:57.991 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.84e-06 2024-04-20T17:36:57.991 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.991 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.991 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:57.991 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.369446+0000", 2024-04-20T17:36:57.991 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9740000000000001e-06 2024-04-20T17:36:57.991 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.991 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.991 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:57.992 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.369455+0000", 2024-04-20T17:36:57.992 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7259999999999997e-06 2024-04-20T17:36:57.992 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.992 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.992 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:57.992 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.369463+0000", 2024-04-20T17:36:57.992 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3339999999999996e-06 2024-04-20T17:36:57.992 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.992 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.992 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:57.992 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.369979+0000", 2024-04-20T17:36:57.992 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00051681600000000002 2024-04-20T17:36:57.992 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.993 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.993 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:57.993 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.370063+0000", 2024-04-20T17:36:57.993 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3233999999999996e-05 2024-04-20T17:36:57.993 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.993 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.993 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:36:57.993 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.370129+0000", 2024-04-20T17:36:57.993 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6097000000000001e-05 2024-04-20T17:36:57.993 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.993 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.993 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:57.993 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.377632+0000", 2024-04-20T17:36:57.993 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0075031760000000003 2024-04-20T17:36:57.993 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.993 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.993 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:57.994 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.491996+0000", 2024-04-20T17:36:57.994 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.114364389 2024-04-20T17:36:57.994 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.994 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.994 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:57.994 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492055+0000", 2024-04-20T17:36:57.994 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8598999999999999e-05 2024-04-20T17:36:57.994 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.994 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.994 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:57.994 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492105+0000", 2024-04-20T17:36:57.994 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0319999999999999e-05 2024-04-20T17:36:57.994 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:57.994 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:57.995 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:57.995 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.995 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.995 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:180901 6.c 6:32888237:::benchmark_data_smithi171_147972_object19803:head [delete] snapc 0=[] ondisk+write+known_if_redirected e653)", 2024-04-20T17:36:57.995 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:35:26.369556+0000", 2024-04-20T17:36:57.995 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 91.559692928999993, 2024-04-20T17:36:57.995 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.122677468, 2024-04-20T17:36:57.995 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:57.996 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:57.996 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:57.996 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:57.996 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:36:57.996 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:36:57.996 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 180901 2024-04-20T17:36:57.996 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.997 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:57.997 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.997 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:57.997 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.369556+0000", 2024-04-20T17:36:57.997 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:57.997 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.997 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.998 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:57.998 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.369556+0000", 2024-04-20T17:36:57.998 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:57.998 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.998 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.998 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:57.999 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.369557+0000", 2024-04-20T17:36:57.999 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.474e-06 2024-04-20T17:36:57.999 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:57.999 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:57.999 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:57.999 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.369562+0000", 2024-04-20T17:36:57.999 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9779999999999996e-06 2024-04-20T17:36:57.999 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.000 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.000 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:58.000 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.369573+0000", 2024-04-20T17:36:58.000 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0849000000000001e-05 2024-04-20T17:36:58.000 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.000 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.000 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:58.000 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.369581+0000", 2024-04-20T17:36:58.000 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0590000000000003e-06 2024-04-20T17:36:58.000 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.000 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.000 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:58.000 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.370067+0000", 2024-04-20T17:36:58.000 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000485198 2024-04-20T17:36:58.001 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.001 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.001 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:58.001 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.370167+0000", 2024-04-20T17:36:58.001 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000100283 2024-04-20T17:36:58.001 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.001 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.001 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:36:58.001 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.370269+0000", 2024-04-20T17:36:58.001 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000102538 2024-04-20T17:36:58.001 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.001 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.001 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:58.002 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.491974+0000", 2024-04-20T17:36:58.002 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.12170478899999999 2024-04-20T17:36:58.002 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.002 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.002 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:58.002 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492152+0000", 2024-04-20T17:36:58.002 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00017778499999999999 2024-04-20T17:36:58.002 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.002 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.003 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:58.003 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492203+0000", 2024-04-20T17:36:58.003 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1306e-05 2024-04-20T17:36:58.003 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.003 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.003 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:58.003 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492233+0000", 2024-04-20T17:36:58.003 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0209000000000002e-05 2024-04-20T17:36:58.003 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:58.003 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:58.003 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:58.003 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.003 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.004 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:180906 6.4 6:251cc9ba:::benchmark_data_smithi171_147972_object19808:head [delete] snapc 0=[] ondisk+write+known_if_redirected e653)", 2024-04-20T17:36:58.004 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:35:26.370225+0000", 2024-04-20T17:36:58.004 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 91.559023671000006, 2024-04-20T17:36:58.004 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.122022784, 2024-04-20T17:36:58.004 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:58.004 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:58.004 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:58.004 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:58.004 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:36:58.004 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:36:58.004 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 180906 2024-04-20T17:36:58.004 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.004 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:58.004 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.005 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:58.005 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.370225+0000", 2024-04-20T17:36:58.005 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:58.005 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.005 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.005 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:58.005 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.370225+0000", 2024-04-20T17:36:58.005 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:58.005 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.005 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.005 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:58.005 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.370226+0000", 2024-04-20T17:36:58.005 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1930000000000001e-06 2024-04-20T17:36:58.005 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.005 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.006 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:58.006 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.370231+0000", 2024-04-20T17:36:58.006 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4290000000000003e-06 2024-04-20T17:36:58.006 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.006 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.006 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:58.006 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.370240+0000", 2024-04-20T17:36:58.006 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7159999999999992e-06 2024-04-20T17:36:58.006 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.006 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.006 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:58.006 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.370247+0000", 2024-04-20T17:36:58.006 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0790000000000003e-06 2024-04-20T17:36:58.007 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.007 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.007 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:58.007 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.373809+0000", 2024-04-20T17:36:58.007 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0035624239999999998 2024-04-20T17:36:58.007 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.007 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.007 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:58.007 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.373896+0000", 2024-04-20T17:36:58.007 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6737999999999995e-05 2024-04-20T17:36:58.007 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.007 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.007 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:36:58.007 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.373972+0000", 2024-04-20T17:36:58.007 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6013999999999994e-05 2024-04-20T17:36:58.008 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.008 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.008 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:58.008 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492127+0000", 2024-04-20T17:36:58.008 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.118154907 2024-04-20T17:36:58.008 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.008 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.008 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:58.008 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492165+0000", 2024-04-20T17:36:58.008 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8161999999999998e-05 2024-04-20T17:36:58.008 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.008 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.008 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:58.008 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492207+0000", 2024-04-20T17:36:58.009 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2592999999999997e-05 2024-04-20T17:36:58.009 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.009 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.009 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:58.009 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492248+0000", 2024-04-20T17:36:58.009 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0528999999999998e-05 2024-04-20T17:36:58.009 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:58.009 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:58.009 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:58.009 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.009 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.009 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:180909 6.c 6:3a31a9e3:::benchmark_data_smithi171_147972_object19811:head [delete] snapc 0=[] ondisk+write+known_if_redirected e653)", 2024-04-20T17:36:58.009 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:35:26.376350+0000", 2024-04-20T17:36:58.009 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 91.552898584999994, 2024-04-20T17:36:58.010 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.116013197, 2024-04-20T17:36:58.010 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:58.010 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:58.010 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:58.010 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:58.010 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:36:58.010 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:36:58.010 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 180909 2024-04-20T17:36:58.010 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.010 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:58.010 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.010 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:58.010 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.376350+0000", 2024-04-20T17:36:58.010 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:58.011 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.011 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.011 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:58.011 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.376350+0000", 2024-04-20T17:36:58.011 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:58.011 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.011 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.011 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:58.011 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.376352+0000", 2024-04-20T17:36:58.011 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6199999999999999e-06 2024-04-20T17:36:58.011 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.011 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.011 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:58.011 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.376357+0000", 2024-04-20T17:36:58.012 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7879999999999997e-06 2024-04-20T17:36:58.012 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.012 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.012 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:58.012 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.376366+0000", 2024-04-20T17:36:58.012 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1309999999999995e-06 2024-04-20T17:36:58.012 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.012 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.012 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:58.012 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.376373+0000", 2024-04-20T17:36:58.012 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9609999999999999e-06 2024-04-20T17:36:58.013 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.013 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.013 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:58.013 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.377578+0000", 2024-04-20T17:36:58.013 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0012052790000000001 2024-04-20T17:36:58.013 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.013 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.013 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:58.013 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.377679+0000", 2024-04-20T17:36:58.013 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000101099 2024-04-20T17:36:58.013 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.013 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.013 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:36:58.013 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.487071+0000", 2024-04-20T17:36:58.014 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10939191299999999 2024-04-20T17:36:58.014 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.014 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.014 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:58.014 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492270+0000", 2024-04-20T17:36:58.014 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0051986300000000001 2024-04-20T17:36:58.014 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.014 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.014 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:58.014 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492290+0000", 2024-04-20T17:36:58.014 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0381999999999999e-05 2024-04-20T17:36:58.014 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.014 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.015 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:58.015 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492329+0000", 2024-04-20T17:36:58.015 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9292999999999998e-05 2024-04-20T17:36:58.015 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.015 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.015 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:58.015 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492364+0000", 2024-04-20T17:36:58.015 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4100999999999998e-05 2024-04-20T17:36:58.015 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:58.015 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:58.015 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:58.015 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.016 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.016 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26882.0:180907 6.0 e653/619 6:039ac935:::benchmark_data_smithi171_147972_object19809:head v 653'11227, mlcod=653'11225)", 2024-04-20T17:36:58.016 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:35:26.376542+0000", 2024-04-20T17:36:58.016 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 91.552706925999999, 2024-04-20T17:36:58.016 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.11055441100000001, 2024-04-20T17:36:58.016 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:58.016 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:58.016 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:58.016 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:58.016 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.016 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:58.016 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.376542+0000", 2024-04-20T17:36:58.017 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:58.017 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.017 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.017 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:58.017 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.376542+0000", 2024-04-20T17:36:58.017 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:58.017 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.017 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.017 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:58.017 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.376543+0000", 2024-04-20T17:36:58.017 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0690000000000001e-06 2024-04-20T17:36:58.017 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.017 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.017 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:58.017 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.376550+0000", 2024-04-20T17:36:58.018 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4640000000000003e-06 2024-04-20T17:36:58.018 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.018 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.018 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:58.018 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.376557+0000", 2024-04-20T17:36:58.018 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1509999999999998e-06 2024-04-20T17:36:58.018 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.018 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.018 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:58.018 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.376562+0000", 2024-04-20T17:36:58.018 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3820000000000003e-06 2024-04-20T17:36:58.018 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.018 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.019 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:58.019 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.376596+0000", 2024-04-20T17:36:58.019 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4029000000000002e-05 2024-04-20T17:36:58.019 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.019 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.019 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:58.019 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.376609+0000", 2024-04-20T17:36:58.019 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3241000000000001e-05 2024-04-20T17:36:58.019 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.019 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.019 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:58.019 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.487026+0000", 2024-04-20T17:36:58.019 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.11041664700000001 2024-04-20T17:36:58.019 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.020 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.020 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:58.020 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.487096+0000", 2024-04-20T17:36:58.020 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0427999999999997e-05 2024-04-20T17:36:58.020 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:58.020 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:58.020 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:58.020 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.020 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.020 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26882.0:180908 6.2 e653/619 6:4681e151:::benchmark_data_smithi171_147972_object19810:head v 653'11432, mlcod=653'11430)", 2024-04-20T17:36:58.020 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:35:26.376668+0000", 2024-04-20T17:36:58.020 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 91.552580723999995, 2024-04-20T17:36:58.020 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.110416717, 2024-04-20T17:36:58.020 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:58.020 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:58.021 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:58.021 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:58.021 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.021 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:58.021 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.376668+0000", 2024-04-20T17:36:58.021 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:58.021 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.021 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.021 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:58.021 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.376668+0000", 2024-04-20T17:36:58.021 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:58.021 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.021 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.022 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:58.022 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.376669+0000", 2024-04-20T17:36:58.022 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.28e-06 2024-04-20T17:36:58.022 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.022 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.022 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:58.022 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.376676+0000", 2024-04-20T17:36:58.022 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.472e-06 2024-04-20T17:36:58.022 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.022 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.022 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:58.022 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.376683+0000", 2024-04-20T17:36:58.022 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8469999999999998e-06 2024-04-20T17:36:58.023 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.023 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.023 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:58.023 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.376687+0000", 2024-04-20T17:36:58.023 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.605e-06 2024-04-20T17:36:58.023 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.023 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.023 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:58.023 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.376734+0000", 2024-04-20T17:36:58.023 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6813000000000001e-05 2024-04-20T17:36:58.023 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.023 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.023 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:58.023 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.376752+0000", 2024-04-20T17:36:58.024 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8034999999999999e-05 2024-04-20T17:36:58.024 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.024 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.024 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:58.024 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.487021+0000", 2024-04-20T17:36:58.024 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.11026868400000001 2024-04-20T17:36:58.024 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.024 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.024 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:58.024 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.487085+0000", 2024-04-20T17:36:58.024 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3980999999999996e-05 2024-04-20T17:36:58.024 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:58.024 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:58.024 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:58.024 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.025 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.025 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(client.26882.0:180898 6.4 e653/626 ondisk, result = 0)", 2024-04-20T17:36:58.025 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:35:26.380181+0000", 2024-04-20T17:36:58.025 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 91.549067819000001, 2024-04-20T17:36:58.025 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.111933744, 2024-04-20T17:36:58.025 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:58.025 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:58.025 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:36:58.025 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:58.025 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.025 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:58.025 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.380181+0000", 2024-04-20T17:36:58.025 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:58.025 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.025 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.025 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:58.026 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.380181+0000", 2024-04-20T17:36:58.026 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:58.026 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.026 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.026 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:58.026 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.380182+0000", 2024-04-20T17:36:58.026 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9099999999999991e-07 2024-04-20T17:36:58.026 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.026 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.026 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:58.026 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.380187+0000", 2024-04-20T17:36:58.026 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.561e-06 2024-04-20T17:36:58.026 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.026 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.027 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:58.027 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.487001+0000", 2024-04-20T17:36:58.027 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.106814415 2024-04-20T17:36:58.027 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.027 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.027 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:58.027 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.487021+0000", 2024-04-20T17:36:58.027 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0038e-05 2024-04-20T17:36:58.027 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.028 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.028 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:58.028 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.491982+0000", 2024-04-20T17:36:58.028 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0049612340000000001 2024-04-20T17:36:58.028 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.028 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.028 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:58.028 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.491990+0000", 2024-04-20T17:36:58.028 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9810000000000003e-06 2024-04-20T17:36:58.028 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.028 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.028 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:58.028 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492115+0000", 2024-04-20T17:36:58.028 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012452399999999999 2024-04-20T17:36:58.029 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:58.029 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:58.029 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:58.029 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.029 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.029 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:180914 6.c 6:383ce87c:::benchmark_data_smithi171_147972_object19816:head [delete] snapc 0=[] ondisk+write+known_if_redirected e653)", 2024-04-20T17:36:58.029 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:35:26.382615+0000", 2024-04-20T17:36:58.029 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 91.546633772000007, 2024-04-20T17:36:58.029 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.125150133, 2024-04-20T17:36:58.029 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:58.030 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:58.030 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:58.030 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:36:58.030 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:36:58.030 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:36:58.030 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 180914 2024-04-20T17:36:58.030 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.030 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:58.030 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.030 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:58.030 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.382615+0000", 2024-04-20T17:36:58.030 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:58.031 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.031 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.031 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:58.031 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.382615+0000", 2024-04-20T17:36:58.031 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:58.031 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.031 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.031 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:58.031 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.382617+0000", 2024-04-20T17:36:58.031 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9470000000000002e-06 2024-04-20T17:36:58.031 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.031 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.031 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:58.032 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.382624+0000", 2024-04-20T17:36:58.032 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6189999999999996e-06 2024-04-20T17:36:58.032 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.032 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.032 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:58.032 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.487002+0000", 2024-04-20T17:36:58.032 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.104378027 2024-04-20T17:36:58.032 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.032 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.032 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:58.032 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.487022+0000", 2024-04-20T17:36:58.032 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0659999999999999e-05 2024-04-20T17:36:58.033 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.033 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.033 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:58.033 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492412+0000", 2024-04-20T17:36:58.033 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0053898410000000002 2024-04-20T17:36:58.033 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.033 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.033 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:58.033 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492527+0000", 2024-04-20T17:36:58.033 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011448 2024-04-20T17:36:58.033 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.033 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.034 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:36:58.034 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492618+0000", 2024-04-20T17:36:58.034 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1150000000000007e-05 2024-04-20T17:36:58.034 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.034 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.035 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:36:58.035 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.507586+0000", 2024-04-20T17:36:58.035 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014967694 2024-04-20T17:36:58.035 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.035 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.035 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:36:58.036 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.507667+0000", 2024-04-20T17:36:58.036 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1562000000000003e-05 2024-04-20T17:36:58.036 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.036 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.036 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:58.036 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.507721+0000", 2024-04-20T17:36:58.037 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3615000000000001e-05 2024-04-20T17:36:58.037 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.037 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.037 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:58.037 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.507765+0000", 2024-04-20T17:36:58.037 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4537999999999999e-05 2024-04-20T17:36:58.038 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:58.038 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:58.038 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:58.038 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.038 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.038 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.26882.0:180915 6.2 e653/619 6:4449404c:::benchmark_data_smithi171_147972_object19817:head v 653'11433, mlcod=653'11430)", 2024-04-20T17:36:58.039 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:35:26.383044+0000", 2024-04-20T17:36:58.039 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 91.546204563000003, 2024-04-20T17:36:58.039 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.11263754300000001, 2024-04-20T17:36:58.039 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:36:58.039 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:36:58.040 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:36:58.040 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:36:58.040 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.040 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:36:58.040 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.383044+0000", 2024-04-20T17:36:58.040 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:58.040 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.040 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.040 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:36:58.040 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.383044+0000", 2024-04-20T17:36:58.040 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:36:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:36:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.383045+0000", 2024-04-20T17:36:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.062e-06 2024-04-20T17:36:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:36:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.383052+0000", 2024-04-20T17:36:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5509999999999996e-06 2024-04-20T17:36:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:36:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.487012+0000", 2024-04-20T17:36:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10395961200000001 2024-04-20T17:36:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:36:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.487023+0000", 2024-04-20T17:36:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1394e-05 2024-04-20T17:36:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:36:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.487114+0000", 2024-04-20T17:36:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1285000000000004e-05 2024-04-20T17:36:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:36:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.487139+0000", 2024-04-20T17:36:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4399000000000001e-05 2024-04-20T17:36:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:36:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.495633+0000", 2024-04-20T17:36:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0084938989999999992 2024-04-20T17:36:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:36:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:36:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:36:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.495682+0000", 2024-04-20T17:36:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9341000000000001e-05 2024-04-20T17:36:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:36:58.044 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:36:58.044 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:36:58.044 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:36:58.082 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:58.116 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:36:58.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:58.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:36:58.223 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:36:58.255 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:58.281 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:36:58.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:58.285 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:36:58.386 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:36:58.424 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:58.452 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:36:58.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:58.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:36:58.556 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:36:58.590 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:58.621 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:36:58.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:58.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:36:58.724 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:36:58.764 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:58.787 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:36:58.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:58.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:36:58.895 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:36:58.909 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:58 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:36:58.909 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:58 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:36:58.909 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:36:58 smithi005 ceph-mon[39562]: pgmap v1791: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.7 GiB data, 9.2 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 791 op/s 2024-04-20T17:36:58.931 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:59.139 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:36:59.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:36:59.142 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:36:59.185 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:58 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:36:59.185 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:58 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:36:59.185 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:36:58 smithi171 ceph-mon[44189]: pgmap v1791: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.7 GiB data, 9.2 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 791 op/s 2024-04-20T17:36:59.244 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:36:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:58 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:36:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:58 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:36:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:36:58 smithi106 ceph-mon[44423]: pgmap v1791: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 2.7 GiB data, 9.2 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 791 op/s 2024-04-20T17:36:59.289 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:36:59.388 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:37:00.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:00 smithi171 ceph-mon[44189]: pgmap v1792: 17 pgs: 17 active+clean; 2.5 GiB data, 9.0 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-20T17:37:00.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:00 smithi005 ceph-mon[39562]: pgmap v1792: 17 pgs: 17 active+clean; 2.5 GiB data, 9.0 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-20T17:37:01.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:00 smithi106 ceph-mon[44423]: pgmap v1792: 17 pgs: 17 active+clean; 2.5 GiB data, 9.0 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.13k op/s 2024-04-20T17:37:02.160 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:37:02.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:02.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:37:02.276 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:37:02.327 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:02.599 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:37:02.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:02.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:37:02.647 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-20T17:37:02.708 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:37:02.806 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:03.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:03 smithi005 ceph-mon[39562]: pgmap v1793: 17 pgs: 17 active+clean; 2.4 GiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-20T17:37:03.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:03 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:37:03.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:03 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:37:03.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:03 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:37:03.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:03 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:37:03.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:03 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:37:03.582 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:37:03.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:03.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:37:03.697 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:37:03.736 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:03.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:03 smithi106 ceph-mon[44423]: pgmap v1793: 17 pgs: 17 active+clean; 2.4 GiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-20T17:37:03.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:03 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:37:03.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:03 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:37:03.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:03 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:37:03.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:03 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:37:03.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:03 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:37:03.813 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:37:03.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:03.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:37:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:03 smithi171 ceph-mon[44189]: pgmap v1793: 17 pgs: 17 active+clean; 2.4 GiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-20T17:37:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:03 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:37:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:03 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:37:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:03 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:37:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:03 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:37:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:03 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:37:03.922 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:37:03.968 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:04.614 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:37:04.614 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:37:04.614 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:37:04.614 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:37:04.614 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:37:04.614 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:37:04.615 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:37:04.615 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:37:04.615 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:37:04.615 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:37:04.615 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:37:04.615 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:37:04.615 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:37:04.615 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:37:04.615 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:37:04.615 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:37:04.616 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:37:04.616 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:37:04.616 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:37:04.616 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:37:04.616 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:37:04.616 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:37:04.616 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:37:04.616 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:37:04.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:04 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1152813296' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:37:04.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:04 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/372013000' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-20T17:37:04.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:04 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:04 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1152813296' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:37:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:04 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/372013000' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-20T17:37:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:04 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:04.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:04 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1152813296' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:37:04.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:04 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/372013000' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-20T17:37:04.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:04 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:04.946 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.4 to 1 (10000) 2024-04-20T17:37:05.142 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:37:05.148 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:37:05.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:05.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:37:05.253 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:37:05.296 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:05.332 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:37:05.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:05.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:37:05.437 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:37:05.499 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:05.560 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:37:05.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:05.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:37:05.674 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:37:05.706 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.706 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.706 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.707 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.707 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.707 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.707 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.707 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.707 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.707 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.707 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.707 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:05 smithi106 ceph-mon[44423]: pgmap v1794: 17 pgs: 17 active+clean; 2.2 GiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-20T17:37:05.707 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:05 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/372013000' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-20T17:37:05.707 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:05 smithi106 ceph-mon[44423]: osdmap e676: 12 total, 12 up, 10 in 2024-04-20T17:37:05.714 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:05.747 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:37:05.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:05.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:37:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:05 smithi171 ceph-mon[44189]: pgmap v1794: 17 pgs: 17 active+clean; 2.2 GiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-20T17:37:05.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:05 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/372013000' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-20T17:37:05.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:05 smithi171 ceph-mon[44189]: osdmap e676: 12 total, 12 up, 10 in 2024-04-20T17:37:05.853 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:37:05.905 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.905 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.905 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.905 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.905 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.905 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.906 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.906 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.906 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.906 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.906 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:05.906 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:05 smithi005 ceph-mon[39562]: pgmap v1794: 17 pgs: 17 active+clean; 2.2 GiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.11k op/s 2024-04-20T17:37:05.906 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:05 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/372013000' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-20T17:37:05.906 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:05 smithi005 ceph-mon[39562]: osdmap e676: 12 total, 12 up, 10 in 2024-04-20T17:37:05.919 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:06.381 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:37:06.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:06.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:37:06.485 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:37:06.521 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:06.892 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:37:06.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:06.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:37:06.927 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 5 1 2024-04-20T17:37:06.997 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:37:07.031 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:07.033 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:37:07.057 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:37:07.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:07.061 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:37:07.162 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:37:07.240 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:07.310 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:37:07.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:07.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:37:07.419 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:37:07.479 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:07.507 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:37:07.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:07.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:37:07.610 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:37:07.672 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:07 smithi005 ceph-mon[39562]: pgmap v1796: 17 pgs: 17 active+clean; 2.1 GiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:37:07.680 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:07.744 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:37:07.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:07.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:37:07.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:07 smithi106 ceph-mon[44423]: pgmap v1796: 17 pgs: 17 active+clean; 2.1 GiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:37:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:07 smithi171 ceph-mon[44189]: pgmap v1796: 17 pgs: 17 active+clean; 2.1 GiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.15k op/s 2024-04-20T17:37:07.855 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:37:07.892 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:07.933 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:37:07.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:07.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:37:08.046 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:37:08.079 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:08.104 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:37:08.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:08.110 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:37:08.211 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:37:08.242 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:08.267 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:37:08.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:08.270 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:37:08.372 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:37:08.400 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:08.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:08.427 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:37:08.427 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:37:08.528 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:37:08.561 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:08.595 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:37:08.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:08.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:37:08.700 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:37:08.767 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:08 smithi005 ceph-mon[39562]: pgmap v1797: 17 pgs: 17 active+clean; 2.0 GiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-20T17:37:08.768 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:37:08.769 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:08.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:08 smithi171 ceph-mon[44189]: pgmap v1797: 17 pgs: 17 active+clean; 2.0 GiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-20T17:37:08.888 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:37:08.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:08.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:37:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:08 smithi106 ceph-mon[44423]: pgmap v1797: 17 pgs: 17 active+clean; 2.0 GiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-20T17:37:09.013 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:37:09.065 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:09.070 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:37:09.575 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:37:09.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:09 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3909786030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:37:09.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:09 smithi171 ceph-mon[44189]: from='client.26986 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:37:09.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:09 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3909786030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:37:09.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:09 smithi005 ceph-mon[39562]: from='client.26986 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:37:10.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:09 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3909786030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:37:10.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:09 smithi106 ceph-mon[44423]: from='client.26986 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:37:10.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:10 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:37:10.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:10 smithi005 ceph-mon[39562]: from='client.26986 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:37:10.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:10 smithi005 ceph-mon[39562]: osdmap e677: 12 total, 12 up, 10 in 2024-04-20T17:37:10.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:10 smithi005 ceph-mon[39562]: 6.9 scrub starts 2024-04-20T17:37:10.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:10 smithi005 ceph-mon[39562]: pgmap v1799: 17 pgs: 17 active+clean; 1.7 GiB data, 7.5 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.34k op/s 2024-04-20T17:37:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:10 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:37:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:10 smithi171 ceph-mon[44189]: from='client.26986 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:37:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:10 smithi171 ceph-mon[44189]: osdmap e677: 12 total, 12 up, 10 in 2024-04-20T17:37:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:10 smithi171 ceph-mon[44189]: 6.9 scrub starts 2024-04-20T17:37:10.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:10 smithi171 ceph-mon[44189]: pgmap v1799: 17 pgs: 17 active+clean; 1.7 GiB data, 7.5 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.34k op/s 2024-04-20T17:37:11.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:10 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:37:11.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:10 smithi106 ceph-mon[44423]: from='client.26986 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:37:11.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:10 smithi106 ceph-mon[44423]: osdmap e677: 12 total, 12 up, 10 in 2024-04-20T17:37:11.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:10 smithi106 ceph-mon[44423]: 6.9 scrub starts 2024-04-20T17:37:11.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:10 smithi106 ceph-mon[44423]: pgmap v1799: 17 pgs: 17 active+clean; 1.7 GiB data, 7.5 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.34k op/s 2024-04-20T17:37:11.447 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:37:11.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:11.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:37:11.557 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:37:11.586 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:11.612 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:37:11.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:11.622 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:37:11.723 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:37:11.759 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:11.786 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:37:11.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:11.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:37:11.822 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:37:11.822 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:37:11.895 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:37:11.929 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:11.953 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:37:11.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:11.957 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:37:12.058 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:37:12.104 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:12.487 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:37:12.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:12.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:37:12.599 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:37:12.636 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:12.665 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:37:12.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:12.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:37:12.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:12 smithi005 ceph-mon[39562]: 6.9 scrub ok 2024-04-20T17:37:12.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:12 smithi106 ceph-mon[44423]: 6.9 scrub ok 2024-04-20T17:37:12.771 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:37:12.828 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:12.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:12 smithi171 ceph-mon[44189]: 6.9 scrub ok 2024-04-20T17:37:12.863 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:37:12.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:12.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:37:12.966 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-20T17:37:12.969 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:37:13.004 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:13.031 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:37:13.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:13.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:37:13.136 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:37:13.163 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:13.196 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:37:13.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:13.200 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:37:13.301 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:37:13.351 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:13.410 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:37:13.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:13.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:37:13.516 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:37:13.553 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:13 smithi005 ceph-mon[39562]: 6.d scrub starts 2024-04-20T17:37:13.554 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:13 smithi005 ceph-mon[39562]: 6.d scrub ok 2024-04-20T17:37:13.554 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:13 smithi005 ceph-mon[39562]: pgmap v1800: 17 pgs: 17 active+clean; 1.7 GiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-20T17:37:13.595 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:13.705 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:37:13.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:13.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:37:13.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:13 smithi106 ceph-mon[44423]: 6.d scrub starts 2024-04-20T17:37:13.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:13 smithi106 ceph-mon[44423]: 6.d scrub ok 2024-04-20T17:37:13.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:13 smithi106 ceph-mon[44423]: pgmap v1800: 17 pgs: 17 active+clean; 1.7 GiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-20T17:37:13.810 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:37:13.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:13 smithi171 ceph-mon[44189]: 6.d scrub starts 2024-04-20T17:37:13.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:13 smithi171 ceph-mon[44189]: 6.d scrub ok 2024-04-20T17:37:13.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:13 smithi171 ceph-mon[44189]: pgmap v1800: 17 pgs: 17 active+clean; 1.7 GiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.05k op/s 2024-04-20T17:37:13.838 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:13.867 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:37:13.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:13.874 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:37:13.975 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:37:14.004 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:14.025 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:37:14.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:14.032 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:37:14.133 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:37:14.176 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:14.186 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:37:14.203 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:37:14.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:14.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:37:14.254 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:37:14.308 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:37:14.338 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:14.359 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:37:14.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:14.365 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:37:14.466 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:37:14.477 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:14 smithi106 ceph-mon[44423]: 6.3 scrub starts 2024-04-20T17:37:14.478 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:14 smithi106 ceph-mon[44423]: 6.3 scrub ok 2024-04-20T17:37:14.503 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:14.541 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:37:14.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:14.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:37:14.649 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:37:14.686 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:14.713 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:37:14.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:14.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:37:14.736 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:14 smithi005 ceph-mon[39562]: 6.3 scrub starts 2024-04-20T17:37:14.736 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:14 smithi005 ceph-mon[39562]: 6.3 scrub ok 2024-04-20T17:37:14.818 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:37:14.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:14 smithi171 ceph-mon[44189]: 6.3 scrub starts 2024-04-20T17:37:14.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:14 smithi171 ceph-mon[44189]: 6.3 scrub ok 2024-04-20T17:37:14.854 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:14.898 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:37:14.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:14.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:37:15.007 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:37:15.065 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:15.509 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.5 to 1 (10000) 2024-04-20T17:37:15.604 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:37:15.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:15 smithi106 ceph-mon[44423]: pgmap v1801: 17 pgs: 17 active+clean; 1.5 GiB data, 7.0 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:37:15.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:15 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2351659073' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:37:15.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:15 smithi171 ceph-mon[44189]: pgmap v1801: 17 pgs: 17 active+clean; 1.5 GiB data, 7.0 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:37:15.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:15 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2351659073' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:37:15.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:15 smithi005 ceph-mon[39562]: pgmap v1801: 17 pgs: 17 active+clean; 1.5 GiB data, 7.0 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:37:15.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:15 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2351659073' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:37:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:16 smithi106 ceph-mon[44423]: 6.0 scrub starts 2024-04-20T17:37:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:16 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2351659073' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-20T17:37:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:16 smithi106 ceph-mon[44423]: osdmap e678: 12 total, 12 up, 10 in 2024-04-20T17:37:16.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:16 smithi106 ceph-mon[44423]: pgmap v1803: 17 pgs: 17 active+clean; 1.3 GiB data, 6.6 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-20T17:37:16.776 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:16 smithi005 ceph-mon[39562]: 6.0 scrub starts 2024-04-20T17:37:16.776 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:16 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2351659073' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-20T17:37:16.777 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:16 smithi005 ceph-mon[39562]: osdmap e678: 12 total, 12 up, 10 in 2024-04-20T17:37:16.777 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:16 smithi005 ceph-mon[39562]: pgmap v1803: 17 pgs: 17 active+clean; 1.3 GiB data, 6.6 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-20T17:37:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:16 smithi171 ceph-mon[44189]: 6.0 scrub starts 2024-04-20T17:37:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:16 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2351659073' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-20T17:37:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:16 smithi171 ceph-mon[44189]: osdmap e678: 12 total, 12 up, 10 in 2024-04-20T17:37:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:16 smithi171 ceph-mon[44189]: pgmap v1803: 17 pgs: 17 active+clean; 1.3 GiB data, 6.6 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-20T17:37:17.741 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:37:17.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:17.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:37:17.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:17 smithi106 ceph-mon[44423]: 6.0 scrub ok 2024-04-20T17:37:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:17 smithi171 ceph-mon[44189]: 6.0 scrub ok 2024-04-20T17:37:17.852 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:37:17.902 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:17.907 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:17 smithi005 ceph-mon[39562]: 6.0 scrub ok 2024-04-20T17:37:17.932 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:37:17.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:17.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:37:18.040 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:37:18.072 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:18.099 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:37:18.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:18.106 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:37:18.207 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:37:18.243 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:18.263 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 8 1 2024-04-20T17:37:18.275 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:37:18.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:18.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:37:18.379 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:37:18.417 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:18.446 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:37:18.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:18.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:37:18.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:18 smithi106 ceph-mon[44423]: pgmap v1804: 17 pgs: 17 active+clean; 1.2 GiB data, 6.4 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-20T17:37:18.513 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:18 smithi005 ceph-mon[39562]: pgmap v1804: 17 pgs: 17 active+clean; 1.2 GiB data, 6.4 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-20T17:37:18.554 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:37:18.581 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:18.602 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:37:18.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:18.610 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:37:18.711 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:37:18.797 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:18.831 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:37:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:18 smithi171 ceph-mon[44189]: pgmap v1804: 17 pgs: 17 active+clean; 1.2 GiB data, 6.4 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.09k op/s 2024-04-20T17:37:18.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:18.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:37:18.944 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:37:18.976 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:19.016 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:37:19.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:19.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:37:19.128 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:37:19.169 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:19.227 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:37:19.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:19.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:37:19.333 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:37:19.385 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:19.468 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:37:19.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:19.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:37:19.541 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:37:19.572 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:37:19.577 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:37:19.610 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:37:19.633 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:19.645 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:37:19.704 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:37:19.735 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:37:19.755 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:37:19.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:19.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:37:19.769 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:19 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:37:19.769 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:19 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:37:19.769 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:19 smithi005 ceph-mon[39562]: osdmap e679: 12 total, 12 up, 10 in 2024-04-20T17:37:19.769 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:19 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/175822389' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:37:19.777 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:37:19.817 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:37:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:19 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:37:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:19 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:37:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:19 smithi171 ceph-mon[44189]: osdmap e679: 12 total, 12 up, 10 in 2024-04-20T17:37:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:19 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/175822389' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:37:19.836 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:37:19.854 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:37:19.854 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:37:19.854 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:37:19.854 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:37:19.854 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:37:19.855 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:37:19.868 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:37:19.896 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:37:19.937 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:37:19.946 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:19.970 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:37:19.975 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:37:19.976 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:37:19.976 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:37:19.976 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:37:19.976 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:37:19.976 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:37:19.976 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:37:19.976 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:37:19.976 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:37:19.977 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:37:19.977 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:37:19.977 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:37:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:19 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:37:20.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:19 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:37:20.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:19 smithi106 ceph-mon[44423]: osdmap e679: 12 total, 12 up, 10 in 2024-04-20T17:37:20.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:19 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/175822389' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:37:20.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:20.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:20.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:20.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:20.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:20.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:20.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:20.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:20.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:20 smithi171 ceph-mon[44189]: pgmap v1806: 17 pgs: 17 active+clean; 1.0 GiB data, 6.1 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.35k op/s 2024-04-20T17:37:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:20 smithi005 ceph-mon[39562]: pgmap v1806: 17 pgs: 17 active+clean; 1.0 GiB data, 6.1 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.35k op/s 2024-04-20T17:37:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:21.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:21.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:21.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:21.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:20 smithi106 ceph-mon[44423]: pgmap v1806: 17 pgs: 17 active+clean; 1.0 GiB data, 6.1 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.35k op/s 2024-04-20T17:37:21.547 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:21.548 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:21.548 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:21.548 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:21.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:21.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:21.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:21.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:21.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:21.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:21.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:21.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:22.315 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:37:22.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:22.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:37:22.419 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:37:22.448 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:22.473 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:37:22.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:22.477 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:37:22.578 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:37:22.614 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:22.637 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:37:22.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:22.640 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:37:22.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:22 smithi005 ceph-mon[39562]: pgmap v1807: 17 pgs: 17 active+clean; 977 MiB data, 5.8 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-20T17:37:22.741 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:37:22.780 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:22.805 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:37:22.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:22.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:37:22.812 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:22 smithi106 ceph-mon[44423]: pgmap v1807: 17 pgs: 17 active+clean; 977 MiB data, 5.8 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-20T17:37:22.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:22 smithi171 ceph-mon[44189]: pgmap v1807: 17 pgs: 17 active+clean; 977 MiB data, 5.8 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s 2024-04-20T17:37:22.913 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:37:22.948 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:22.979 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:37:22.984 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:37:22.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:22.990 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:37:23.091 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:37:23.118 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:23.143 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:37:23.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:23.146 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:37:23.248 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:37:23.301 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:23.576 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-20T17:37:23.769 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:37:23.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:23.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:37:23.875 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:37:23.920 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:23.947 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:37:23.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:23.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:37:24.047 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:37:24.055 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:37:24.085 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:24.118 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:37:24.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:24.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:37:24.222 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:37:24.318 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:24.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:24 smithi005 ceph-mon[39562]: pgmap v1808: 17 pgs: 17 active+clean; 768 MiB data, 5.5 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-20T17:37:24.809 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:37:24.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:24.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:37:24.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:24 smithi171 ceph-mon[44189]: pgmap v1808: 17 pgs: 17 active+clean; 768 MiB data, 5.5 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-20T17:37:24.911 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:37:24.951 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:24 smithi106 ceph-mon[44423]: pgmap v1808: 17 pgs: 17 active+clean; 768 MiB data, 5.5 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.16k op/s 2024-04-20T17:37:24.951 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:24.989 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:37:24.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:24.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:37:25.092 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:37:25.119 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:25.566 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.8 to 1 (10000) 2024-04-20T17:37:25.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:25 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1697286671' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-20T17:37:25.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:25 smithi171 ceph-mon[44189]: from='client.27001 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-20T17:37:25.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:25 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1697286671' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-20T17:37:25.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:25 smithi005 ceph-mon[39562]: from='client.27001 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-20T17:37:25.978 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:37:25.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:25.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:37:26.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:25 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1697286671' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-20T17:37:26.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:25 smithi106 ceph-mon[44423]: from='client.27001 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-20T17:37:26.083 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:37:26.124 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:26.160 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:37:26.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:26.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:37:26.268 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:37:26.299 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:26.331 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:37:26.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:26.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:37:26.438 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:37:26.489 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:26.514 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:37:26.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:26.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:37:26.623 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:37:26.626 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:37:26.632 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:37:26.653 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:26.680 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:37:26.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:26.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:37:26.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:26 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:37:26.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:26 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:37:26.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:26 smithi106 ceph-mon[44423]: from='client.27001 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-20T17:37:26.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:26 smithi106 ceph-mon[44423]: osdmap e680: 12 total, 12 up, 10 in 2024-04-20T17:37:26.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:26 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:37:26.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:26 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:37:26.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:26 smithi106 ceph-mon[44423]: pgmap v1810: 17 pgs: 17 active+clean; 626 MiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-20T17:37:26.785 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:37:26.822 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:26 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:37:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:26 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:37:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:26 smithi171 ceph-mon[44189]: from='client.27001 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-20T17:37:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:26 smithi171 ceph-mon[44189]: osdmap e680: 12 total, 12 up, 10 in 2024-04-20T17:37:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:26 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:37:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:26 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:37:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:26 smithi171 ceph-mon[44189]: pgmap v1810: 17 pgs: 17 active+clean; 626 MiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-20T17:37:26.844 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:37:26.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:26.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:37:26.948 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:37:26.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:26 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:37:26.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:26 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:37:26.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:26 smithi005 ceph-mon[39562]: from='client.27001 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-20T17:37:26.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:26 smithi005 ceph-mon[39562]: osdmap e680: 12 total, 12 up, 10 in 2024-04-20T17:37:26.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:26 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:37:26.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:26 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:37:26.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:26 smithi005 ceph-mon[39562]: pgmap v1810: 17 pgs: 17 active+clean; 626 MiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.30k op/s 2024-04-20T17:37:26.983 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:27.008 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:37:27.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:27.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:37:27.049 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 9 1 2024-04-20T17:37:27.115 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:37:27.148 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:27.171 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:37:27.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:27.183 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:37:27.284 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:37:27.313 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:27.333 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:37:27.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:27.336 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:37:27.437 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:37:27.475 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:27 smithi171 ceph-mon[44189]: osdmap e681: 12 total, 12 up, 10 in 2024-04-20T17:37:27.877 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:27 smithi005 ceph-mon[39562]: osdmap e681: 12 total, 12 up, 10 in 2024-04-20T17:37:28.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:27 smithi106 ceph-mon[44423]: osdmap e681: 12 total, 12 up, 10 in 2024-04-20T17:37:28.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:28 smithi005 ceph-mon[39562]: 6.7 scrub starts 2024-04-20T17:37:28.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:28 smithi005 ceph-mon[39562]: 6.7 scrub ok 2024-04-20T17:37:28.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:28 smithi005 ceph-mon[39562]: pgmap v1812: 17 pgs: 17 active+clean; 535 MiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-20T17:37:28.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:28 smithi005 ceph-mon[39562]: 6.a scrub starts 2024-04-20T17:37:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:28 smithi106 ceph-mon[44423]: 6.7 scrub starts 2024-04-20T17:37:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:28 smithi106 ceph-mon[44423]: 6.7 scrub ok 2024-04-20T17:37:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:28 smithi106 ceph-mon[44423]: pgmap v1812: 17 pgs: 17 active+clean; 535 MiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-20T17:37:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:28 smithi106 ceph-mon[44423]: 6.a scrub starts 2024-04-20T17:37:29.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:28 smithi171 ceph-mon[44189]: 6.7 scrub starts 2024-04-20T17:37:29.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:28 smithi171 ceph-mon[44189]: 6.7 scrub ok 2024-04-20T17:37:29.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:28 smithi171 ceph-mon[44189]: pgmap v1812: 17 pgs: 17 active+clean; 535 MiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s 2024-04-20T17:37:29.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:28 smithi171 ceph-mon[44189]: 6.a scrub starts 2024-04-20T17:37:29.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:29 smithi005 ceph-mon[39562]: 6.a scrub ok 2024-04-20T17:37:29.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:29 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1973404304' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:37:29.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:29 smithi005 ceph-mon[39562]: from='client.27113 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:37:30.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:29 smithi106 ceph-mon[44423]: 6.a scrub ok 2024-04-20T17:37:30.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:29 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1973404304' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:37:30.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:29 smithi106 ceph-mon[44423]: from='client.27113 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:37:30.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:29 smithi171 ceph-mon[44189]: 6.a scrub ok 2024-04-20T17:37:30.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:29 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1973404304' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:37:30.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:29 smithi171 ceph-mon[44189]: from='client.27113 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:37:30.354 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:37:30.896 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:30 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:37:30.896 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:30 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:37:30.896 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:30 smithi005 ceph-mon[39562]: from='client.27113 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:37:30.897 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:30 smithi005 ceph-mon[39562]: osdmap e682: 12 total, 12 up, 10 in 2024-04-20T17:37:30.897 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:30 smithi005 ceph-mon[39562]: pgmap v1814: 17 pgs: 17 active+clean; 305 MiB data, 4.7 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:37:31.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:30 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:37:31.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:30 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:37:31.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:30 smithi106 ceph-mon[44423]: from='client.27113 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:37:31.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:30 smithi106 ceph-mon[44423]: osdmap e682: 12 total, 12 up, 10 in 2024-04-20T17:37:31.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:30 smithi106 ceph-mon[44423]: pgmap v1814: 17 pgs: 17 active+clean; 305 MiB data, 4.7 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:37:31.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:30 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:37:31.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:30 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:37:31.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:30 smithi171 ceph-mon[44189]: from='client.27113 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:37:31.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:30 smithi171 ceph-mon[44189]: osdmap e682: 12 total, 12 up, 10 in 2024-04-20T17:37:31.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:30 smithi171 ceph-mon[44189]: pgmap v1814: 17 pgs: 17 active+clean; 305 MiB data, 4.7 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:37:31.775 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:37:31.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:31.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:37:31.886 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:37:31.913 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:31.936 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:37:31.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:31.939 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:37:32.040 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:37:32.043 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:37:32.066 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:32.087 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:37:32.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:32.095 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:37:32.137 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:37:32.186 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:37:32.196 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:37:32.219 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:37:32.240 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:32.254 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:37:32.273 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:37:32.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:32.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:37:32.306 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:37:32.349 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:37:32.377 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:37:32.382 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:37:32.408 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:32.409 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:37:32.413 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Removed 161095 objects 2024-04-20T17:37:32.413 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Clean up completed and total clean up time :142.473 2024-04-20T17:37:32.427 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:37:32.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:32.429 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:37:32.430 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:37:32.450 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:37:32.471 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:37:32.490 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:37:32.492 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:37:32.492 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:37:32.492 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:37:32.492 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:37:32.492 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:37:32.492 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:37:32.492 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:37:32.492 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:37:32.492 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:37:32.492 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:37:32.492 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:37:32.492 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:37:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:32 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:37:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:32 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:37:32.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:32 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/893167617' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:37:32.515 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:32 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:37:32.515 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:32 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:37:32.515 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:32 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/893167617' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:37:32.530 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:37:32.556 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:32.574 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:37:32.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:32.577 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:37:32.678 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:37:32.702 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:32 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:37:32.703 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:32 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:37:32.703 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:32 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/893167617' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:37:32.705 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:32.723 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:37:32.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:32.726 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:37:32.827 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:37:32.859 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:32.866 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:37:32.866 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:37:33.063 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:37:33.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:33.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:37:33.167 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:37:33.199 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:33.220 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:37:33.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:33.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:37:33.324 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:37:33.352 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:33.371 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:37:33.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:33.374 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:37:33.458 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:37:33.474 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:37:33.533 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:33.561 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:37:33.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:33.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:37:33.588 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-20T17:37:33.665 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:37:33.700 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:33.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:33 smithi005 ceph-mon[39562]: pgmap v1815: 17 pgs: 17 active+clean; 235 MiB data, 4.4 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-20T17:37:33.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:33 smithi005 ceph-mon[39562]: osdmap e683: 12 total, 12 up, 10 in 2024-04-20T17:37:33.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:33 smithi005 ceph-mon[39562]: 6.1 scrub starts 2024-04-20T17:37:33.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:33 smithi005 ceph-mon[39562]: 6.1 scrub ok 2024-04-20T17:37:33.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:33 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:37:33.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:33 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:37:33.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:33 smithi106 ceph-mon[44423]: pgmap v1815: 17 pgs: 17 active+clean; 235 MiB data, 4.4 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-20T17:37:33.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:33 smithi106 ceph-mon[44423]: osdmap e683: 12 total, 12 up, 10 in 2024-04-20T17:37:33.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:33 smithi106 ceph-mon[44423]: 6.1 scrub starts 2024-04-20T17:37:33.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:33 smithi106 ceph-mon[44423]: 6.1 scrub ok 2024-04-20T17:37:33.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:33 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:37:33.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:33 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:37:33.800 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:37:33.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:33.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:37:33.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:33 smithi171 ceph-mon[44189]: pgmap v1815: 17 pgs: 17 active+clean; 235 MiB data, 4.4 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s 2024-04-20T17:37:33.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:33 smithi171 ceph-mon[44189]: osdmap e683: 12 total, 12 up, 10 in 2024-04-20T17:37:33.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:33.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:33 smithi171 ceph-mon[44189]: 6.1 scrub starts 2024-04-20T17:37:33.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:33 smithi171 ceph-mon[44189]: 6.1 scrub ok 2024-04-20T17:37:33.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:33 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:37:33.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:33 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:37:33.903 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:37:33.931 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:34.390 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:37:34.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:34.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:37:34.494 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:37:34.527 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:34.549 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:37:34.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:34.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:37:34.653 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:37:34.685 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:34.706 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:37:34.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:34.710 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:37:34.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:34 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:37:34.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:34 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:37:34.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:34 smithi005 ceph-mon[39562]: osdmap e684: 12 total, 12 up, 10 in 2024-04-20T17:37:34.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:34 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:37:34.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:34 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:37:34.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:34 smithi106 ceph-mon[44423]: osdmap e684: 12 total, 12 up, 10 in 2024-04-20T17:37:34.810 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:37:34.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:34 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:37:34.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:34 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:37:34.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:34 smithi171 ceph-mon[44189]: osdmap e684: 12 total, 12 up, 10 in 2024-04-20T17:37:34.843 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:34.864 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:37:34.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:34.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:37:34.968 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:37:34.999 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:35.019 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:37:35.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:35.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:37:35.123 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:37:35.159 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:35.173 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:37:35.173 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:37:35.187 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:37:35.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:35.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:37:35.296 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:37:35.327 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:35.427 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:37:35.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:35.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:37:35.532 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:37:35.563 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:35.697 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:35 smithi005 ceph-mon[39562]: pgmap v1818: 17 pgs: 17 active+clean; 74 MiB data, 4.2 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:37:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:35 smithi106 ceph-mon[44423]: pgmap v1818: 17 pgs: 17 active+clean; 74 MiB data, 4.2 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:37:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:35 smithi171 ceph-mon[44189]: pgmap v1818: 17 pgs: 17 active+clean; 74 MiB data, 4.2 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:37:36.348 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:37:36.479 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.9 to 1 (10000) 2024-04-20T17:37:36.512 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:37:36.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:36.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:37:36.617 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:37:36.650 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:36.671 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:37:36.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:36.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:37:36.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:36 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:37:36.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:36 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:37:36.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:36 smithi005 ceph-mon[39562]: osdmap e685: 12 total, 12 up, 10 in 2024-04-20T17:37:36.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:36 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/819996782' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:37:36.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:36 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:37:36.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:36 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:37:36.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:36 smithi106 ceph-mon[44423]: osdmap e685: 12 total, 12 up, 10 in 2024-04-20T17:37:36.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:36 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/819996782' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:37:36.775 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:37:36.780 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-04-20T17:37:36.781 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-04-20T17:37:36.803 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:36 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:37:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:36 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:37:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:36 smithi171 ceph-mon[44189]: osdmap e685: 12 total, 12 up, 10 in 2024-04-20T17:37:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:36 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/819996782' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:37:36.930 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:37:36.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:36.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:37:37.034 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:37:37.062 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:37.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:37 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:37:37.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:37 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:37:37.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:37 smithi106 ceph-mon[44423]: pgmap v1820: 17 pgs: 17 active+clean; 3.2 MiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 805 op/s 2024-04-20T17:37:37.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:37 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/819996782' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-20T17:37:37.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:37 smithi106 ceph-mon[44423]: osdmap e686: 12 total, 12 up, 10 in 2024-04-20T17:37:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:37 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:37:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:37 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:37:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:37 smithi171 ceph-mon[44189]: pgmap v1820: 17 pgs: 17 active+clean; 3.2 MiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 805 op/s 2024-04-20T17:37:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:37 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/819996782' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-20T17:37:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:37 smithi171 ceph-mon[44189]: osdmap e686: 12 total, 12 up, 10 in 2024-04-20T17:37:37.945 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:37 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:37:37.946 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:37 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:37:37.946 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:37 smithi005 ceph-mon[39562]: pgmap v1820: 17 pgs: 17 active+clean; 3.2 MiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 805 op/s 2024-04-20T17:37:37.946 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:37 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/819996782' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-20T17:37:37.946 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:37 smithi005 ceph-mon[39562]: osdmap e686: 12 total, 12 up, 10 in 2024-04-20T17:37:38.321 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:37:38.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:38.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:37:38.426 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:37:38.456 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:38.582 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:37:38.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:38.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:37:38.688 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:37:38.720 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:38.742 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:37:38.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:38.745 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:37:38.846 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:37:38.880 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:38.900 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:37:38.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:38.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:37:39.004 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:37:39.034 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:39.054 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:37:39.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:39.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:37:39.158 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:37:39.187 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:39.205 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:37:39.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:39.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:37:39.309 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:37:39.312 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:37:39.312 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:37:39.339 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:39.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:39 smithi005 ceph-mon[39562]: pgmap v1822: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 625 op/s 2024-04-20T17:37:39.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:39 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:37:39.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:39 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:37:39.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:39 smithi106 ceph-mon[44423]: pgmap v1822: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 625 op/s 2024-04-20T17:37:39.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:39 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:37:39.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:39 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:37:39.788 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:37:39.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:39.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:37:39.797 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 10 1 2024-04-20T17:37:39.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:39 smithi171 ceph-mon[44189]: pgmap v1822: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 625 op/s 2024-04-20T17:37:39.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:39 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:37:39.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:39 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:37:39.892 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:37:39.925 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:39.947 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:37:39.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:39.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:37:40.051 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:37:40.080 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:40.205 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:37:40.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:40.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:37:40.212 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:37:40.309 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:37:40.342 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:40.363 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:37:40.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:40.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:37:40.467 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:37:40.500 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:40.521 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:37:40.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:40.524 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:37:40.625 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:37:40.656 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:40 smithi106 ceph-mon[44423]: osdmap e687: 12 total, 12 up, 10 in 2024-04-20T17:37:40.658 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:40.679 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:37:40.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:40.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:37:40.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:40 smithi005 ceph-mon[39562]: osdmap e687: 12 total, 12 up, 10 in 2024-04-20T17:37:40.783 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:37:40.811 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:40.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:40 smithi171 ceph-mon[44189]: osdmap e687: 12 total, 12 up, 10 in 2024-04-20T17:37:40.885 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:37:40.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:40.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:37:40.892 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:37:40.921 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-20T17:37:40.989 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:37:41.021 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:41.042 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:37:41.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:41.045 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:37:41.146 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:37:41.176 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:41.195 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:37:41.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:41.198 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:37:41.299 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:37:41.327 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:41.452 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:37:41.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:41.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:37:41.500 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:37:41.558 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:37:41.592 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:41.614 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:37:41.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:41.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:37:41.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:41 smithi005 ceph-mon[39562]: pgmap v1824: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 197 op/s 2024-04-20T17:37:41.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:41 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1914781516' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:37:41.718 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:37:41.748 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:41 smithi171 ceph-mon[44189]: pgmap v1824: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 197 op/s 2024-04-20T17:37:41.748 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:41 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1914781516' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:37:41.750 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:41 smithi106 ceph-mon[44423]: pgmap v1824: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 197 op/s 2024-04-20T17:37:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:41 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1914781516' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:37:41.772 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:37:41.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:41.775 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:37:41.876 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:37:41.905 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:42.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:42 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:37:42.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:42 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1914781516' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:37:42.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:42 smithi106 ceph-mon[44423]: osdmap e688: 12 total, 12 up, 10 in 2024-04-20T17:37:42.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:42 smithi106 ceph-mon[44423]: 6.c scrub starts 2024-04-20T17:37:42.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:42 smithi106 ceph-mon[44423]: 6.c scrub ok 2024-04-20T17:37:42.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:42 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:37:42.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:42 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1914781516' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:37:42.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:42 smithi171 ceph-mon[44189]: osdmap e688: 12 total, 12 up, 10 in 2024-04-20T17:37:42.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:42 smithi171 ceph-mon[44189]: 6.c scrub starts 2024-04-20T17:37:42.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:42 smithi171 ceph-mon[44189]: 6.c scrub ok 2024-04-20T17:37:42.950 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:42 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:37:42.950 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:42 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1914781516' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:37:42.950 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:42 smithi005 ceph-mon[39562]: osdmap e688: 12 total, 12 up, 10 in 2024-04-20T17:37:42.951 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:42 smithi005 ceph-mon[39562]: 6.c scrub starts 2024-04-20T17:37:42.951 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:42 smithi005 ceph-mon[39562]: 6.c scrub ok 2024-04-20T17:37:43.653 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:43 smithi005 ceph-mon[39562]: pgmap v1826: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 6 op/s 2024-04-20T17:37:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:43 smithi106 ceph-mon[44423]: pgmap v1826: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 6 op/s 2024-04-20T17:37:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:43 smithi171 ceph-mon[44189]: pgmap v1826: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 6 op/s 2024-04-20T17:37:44.057 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:37:44.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:44.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:37:44.162 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:37:44.183 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:37:44.195 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:44.202 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:37:44.217 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:37:44.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:44.220 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:37:44.222 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:37:44.242 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:37:44.262 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:37:44.281 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:37:44.302 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:37:44.320 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:37:44.322 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:37:44.342 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:37:44.352 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:44.362 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:37:44.374 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:37:44.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:44.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:37:44.383 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:37:44.403 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:37:44.405 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:37:44.405 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:37:44.405 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:37:44.405 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:37:44.405 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:37:44.405 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:37:44.405 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:37:44.405 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:37:44.405 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:37:44.405 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:37:44.405 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:37:44.405 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:37:44.477 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:37:44.503 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'unique_pool_4' removed 2024-04-20T17:37:44.508 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:44.531 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:37:44.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:44.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:37:44.635 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:37:44.673 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:44.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:44 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/635859337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:37:44.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:44 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3652673971' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-20T17:37:44.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:44 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:44.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:44 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/635859337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:37:44.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:44 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3652673971' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-20T17:37:44.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:44 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:44 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/635859337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:37:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:44 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3652673971' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-20T17:37:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:44 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:44.930 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:37:44.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:44.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:37:45.034 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:37:45.061 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:45.080 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:37:45.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:45.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:37:45.184 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:37:45.210 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:45.229 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:37:45.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:45.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:37:45.332 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:37:45.362 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:45.382 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:37:45.383 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:37:45.384 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:37:45.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:45.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:37:45.489 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:37:45.502 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.502 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.502 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.502 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.502 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.502 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.502 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.502 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.502 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.502 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.502 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:45 smithi171 ceph-mon[44189]: pgmap v1827: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 5 op/s 2024-04-20T17:37:45.502 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:45 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3652673971' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-04-20T17:37:45.503 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:45 smithi171 ceph-mon[44189]: osdmap e689: 12 total, 12 up, 10 in 2024-04-20T17:37:45.517 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:45.536 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:37:45.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:45.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:37:45.640 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:37:45.672 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:45.692 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:37:45.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:45.695 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:37:45.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:45 smithi005 ceph-mon[39562]: pgmap v1827: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 5 op/s 2024-04-20T17:37:45.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:45 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3652673971' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-04-20T17:37:45.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:45 smithi005 ceph-mon[39562]: osdmap e689: 12 total, 12 up, 10 in 2024-04-20T17:37:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:45.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:45 smithi106 ceph-mon[44423]: pgmap v1827: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 5 op/s 2024-04-20T17:37:45.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:45 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3652673971' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-04-20T17:37:45.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:45 smithi106 ceph-mon[44423]: osdmap e689: 12 total, 12 up, 10 in 2024-04-20T17:37:45.796 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:37:45.823 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:45.842 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:37:45.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:45.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:37:45.946 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:37:45.972 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:45.990 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:37:45.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:45.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:37:46.094 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:37:46.120 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:46.138 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:37:46.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:46.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:37:46.242 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:37:46.269 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:46.288 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:37:46.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:46.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:37:46.391 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:37:46.423 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:46.444 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:37:46.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:46.447 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:37:46.548 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:37:46.577 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:46.596 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:37:46.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:46.599 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:37:46.700 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:37:46.734 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:46.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:46 smithi106 ceph-mon[44423]: pgmap v1829: 1 pgs: 1 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:37:46.791 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:37:46.792 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:46 smithi005 ceph-mon[39562]: pgmap v1829: 1 pgs: 1 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:37:46.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:46.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:37:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:46 smithi171 ceph-mon[44189]: pgmap v1829: 1 pgs: 1 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:37:46.895 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:37:46.924 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:46.944 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:37:46.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:46.947 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:37:46.947 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:37:47.047 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:37:47.079 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:47.099 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:37:47.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:47.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:37:47.204 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:37:47.231 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:47.252 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:37:47.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:47.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:37:47.279 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-20T17:37:47.280 INFO:tasks.radosbench:Beginning radosbench... 2024-04-20T17:37:47.280 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_5 2024-04-20T17:37:47.280 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pool create unique_pool_5 16 2024-04-20T17:37:47.357 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:37:47.390 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:47.410 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:37:47.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:47.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:37:47.515 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:37:47.544 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:47.565 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:37:47.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:47.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:37:47.669 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:37:47.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:47 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:37:47.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:47 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:37:47.697 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:47 smithi106 ceph-mon[44423]: osdmap e690: 12 total, 12 up, 10 in 2024-04-20T17:37:47.698 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:47.718 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:37:47.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:47.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:37:47.822 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:37:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:47 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:37:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:47 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:37:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:47 smithi171 ceph-mon[44189]: osdmap e690: 12 total, 12 up, 10 in 2024-04-20T17:37:47.850 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:47.870 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:37:47.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:47.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:37:47.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:47 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:37:47.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:47 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:37:47.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:47 smithi005 ceph-mon[39562]: osdmap e690: 12 total, 12 up, 10 in 2024-04-20T17:37:47.974 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:37:48.012 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:48.366 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:37:48.370 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:37:48.371 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:37:48.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:48.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:37:48.473 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:37:48.505 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:48.527 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:37:48.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:48.530 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:37:48.625 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:48 smithi005 ceph-mon[39562]: pgmap v1831: 1 pgs: 1 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:37:48.631 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:37:48.663 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:48.684 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:37:48.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:48.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:37:48.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:48 smithi106 ceph-mon[44423]: pgmap v1831: 1 pgs: 1 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:37:48.788 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:37:48.818 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:48 smithi171 ceph-mon[44189]: pgmap v1831: 1 pgs: 1 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:37:48.821 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:37:48.821 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:48.842 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:37:48.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:48.845 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:37:48.946 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:37:48.976 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:49.015 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:37:49.015 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:37:49.015 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:37:49.015 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:37:49.015 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:37:50.067 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:49 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4098617277' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:37:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:49 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4098617277' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:37:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:49 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4098617277' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:37:50.348 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-20T17:37:50.937 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.10 to 1 (10000) 2024-04-20T17:37:50.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:50 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4098617277' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-20T17:37:50.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:50 smithi005 ceph-mon[39562]: osdmap e691: 12 total, 12 up, 10 in 2024-04-20T17:37:50.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:50 smithi005 ceph-mon[39562]: pgmap v1833: 1 pgs: 1 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:37:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:50 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4098617277' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-20T17:37:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:50 smithi171 ceph-mon[44189]: osdmap e691: 12 total, 12 up, 10 in 2024-04-20T17:37:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:50 smithi171 ceph-mon[44189]: pgmap v1833: 1 pgs: 1 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:37:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:50 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4098617277' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-20T17:37:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:50 smithi106 ceph-mon[44423]: osdmap e691: 12 total, 12 up, 10 in 2024-04-20T17:37:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:50 smithi106 ceph-mon[44423]: pgmap v1833: 1 pgs: 1 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:37:51.580 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 11 1 2024-04-20T17:37:51.663 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:37:51.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:51.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:37:51.767 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:37:51.799 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:51.822 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:37:51.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:51.825 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:37:51.926 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:37:51.957 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:51.977 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:37:51.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:51.980 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:37:52.081 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:37:52.112 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:52.950 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:37:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:52 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:37:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:52 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:37:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:52 smithi171 ceph-mon[44189]: osdmap e692: 12 total, 12 up, 10 in 2024-04-20T17:37:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:52 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:37:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:52 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:37:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:52 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2459914146' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:37:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:52 smithi171 ceph-mon[44189]: pgmap v1835: 1 pgs: 1 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:37:53.101 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:37:53.103 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:37:53.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:52 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:37:53.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:52 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:37:53.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:52 smithi005 ceph-mon[39562]: osdmap e692: 12 total, 12 up, 10 in 2024-04-20T17:37:53.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:52 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:37:53.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:52 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:37:53.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:52 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2459914146' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:37:53.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:52 smithi005 ceph-mon[39562]: pgmap v1835: 1 pgs: 1 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:37:53.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:52 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:37:53.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:52 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:37:53.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:52 smithi106 ceph-mon[44423]: osdmap e692: 12 total, 12 up, 10 in 2024-04-20T17:37:53.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:52 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:37:53.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:52 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:37:53.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:52 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2459914146' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:37:53.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:52 smithi106 ceph-mon[44423]: pgmap v1835: 1 pgs: 1 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:37:54.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:53 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:37:54.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:53 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2459914146' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:37:54.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:53 smithi171 ceph-mon[44189]: osdmap e693: 12 total, 12 up, 10 in 2024-04-20T17:37:54.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:53 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:37:54.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:53 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2459914146' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:37:54.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:53 smithi005 ceph-mon[39562]: osdmap e693: 12 total, 12 up, 10 in 2024-04-20T17:37:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:53 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:37:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:53 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2459914146' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:37:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:53 smithi106 ceph-mon[44423]: osdmap e693: 12 total, 12 up, 10 in 2024-04-20T17:37:54.266 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:37:54.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:54.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:37:54.371 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:37:54.404 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:54.427 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:37:54.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:54.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:37:54.531 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:37:54.561 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:54.582 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:37:54.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:54.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:37:54.686 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:37:54.718 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:54.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:54 smithi005 ceph-mon[39562]: pgmap v1837: 1 pgs: 1 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:37:55.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:54 smithi171 ceph-mon[44189]: pgmap v1837: 1 pgs: 1 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:37:55.167 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:37:55.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:55.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:37:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:54 smithi106 ceph-mon[44423]: pgmap v1837: 1 pgs: 1 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:37:55.271 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:37:55.304 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:55.326 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:37:55.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:55.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:37:55.430 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:37:55.460 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:55.826 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'unique_pool_5' created 2024-04-20T17:37:56.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:55 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3041287379' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-04-20T17:37:56.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:55 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3041287379' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-04-20T17:37:56.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:55 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3041287379' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-04-20T17:37:56.437 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:37:56.456 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:37:56.475 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:37:56.495 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:37:56.515 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:37:56.535 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:37:56.555 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:37:56.575 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:37:56.595 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:37:56.616 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:37:56.636 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:37:56.643 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:37:56.658 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:37:56.659 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:37:56.659 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:37:56.659 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:37:56.659 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:37:56.659 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:37:56.659 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:37:56.659 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:37:56.659 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:37:56.659 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:37:56.659 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:37:56.659 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:37:56.659 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:37:56.694 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:37:56.935 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:56 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3041287379' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-04-20T17:37:56.935 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:56 smithi005 ceph-mon[39562]: osdmap e694: 12 total, 12 up, 10 in 2024-04-20T17:37:56.935 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:56 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4198191275' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:37:56.935 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:56.935 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:56 smithi005 ceph-mon[39562]: pgmap v1839: 17 pgs: 2 creating+peering, 14 unknown, 1 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:37:56.935 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:56.935 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:56.936 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:56.936 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:56.936 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:56.938 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:37:56.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:56.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:37:57.043 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:37:57.075 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:57.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:56 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3041287379' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-04-20T17:37:57.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:56 smithi171 ceph-mon[44189]: osdmap e694: 12 total, 12 up, 10 in 2024-04-20T17:37:57.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:56 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4198191275' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:37:57.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:57.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:56 smithi171 ceph-mon[44189]: pgmap v1839: 17 pgs: 2 creating+peering, 14 unknown, 1 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:37:57.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:57.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:57.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:57.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:57.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:57.097 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:37:57.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:57.100 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:37:57.201 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:37:57.230 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:57.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:56 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3041287379' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-04-20T17:37:57.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:56 smithi106 ceph-mon[44423]: osdmap e694: 12 total, 12 up, 10 in 2024-04-20T17:37:57.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:56 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4198191275' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:37:57.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:57.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:56 smithi106 ceph-mon[44423]: pgmap v1839: 17 pgs: 2 creating+peering, 14 unknown, 1 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:37:57.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:57.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:57.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:57.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:57.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:57.880 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:37:57.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:57.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:37:57.984 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:37:58.011 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:58.011 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:58.011 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:58.011 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:58.011 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:58.012 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:58.012 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:57 smithi005 ceph-mon[39562]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-20T17:37:58.012 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:57 smithi005 ceph-mon[39562]: osdmap e695: 12 total, 12 up, 10 in 2024-04-20T17:37:58.014 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:58.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:58.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:58.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:58.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:58.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:58.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:58.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:57 smithi171 ceph-mon[44189]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-20T17:37:58.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:57 smithi171 ceph-mon[44189]: osdmap e695: 12 total, 12 up, 10 in 2024-04-20T17:37:58.163 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:37:58.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:58.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:37:58.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:58.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:58.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:58.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:58.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:58.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:37:58.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:57 smithi106 ceph-mon[44423]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-20T17:37:58.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:57 smithi106 ceph-mon[44423]: osdmap e695: 12 total, 12 up, 10 in 2024-04-20T17:37:58.267 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:37:58.297 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:58.421 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:37:58.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:58.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:37:58.526 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:37:58.553 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:58.854 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-04-20T17:37:58.902 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:37:58.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:58.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:37:59.006 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:37:59.034 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:58 smithi005 ceph-mon[39562]: osdmap e696: 12 total, 12 up, 10 in 2024-04-20T17:37:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:37:58 smithi005 ceph-mon[39562]: pgmap v1842: 17 pgs: 1 active+clean, 4 creating+peering, 12 unknown; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:37:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:58 smithi106 ceph-mon[44423]: osdmap e696: 12 total, 12 up, 10 in 2024-04-20T17:37:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:37:58 smithi106 ceph-mon[44423]: pgmap v1842: 17 pgs: 1 active+clean, 4 creating+peering, 12 unknown; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:37:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:58 smithi171 ceph-mon[44189]: osdmap e696: 12 total, 12 up, 10 in 2024-04-20T17:37:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:37:58 smithi171 ceph-mon[44189]: pgmap v1842: 17 pgs: 1 active+clean, 4 creating+peering, 12 unknown; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:37:59.389 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:37:59.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:59.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:37:59.411 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:37:59.493 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:37:59.527 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:59.548 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:37:59.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:59.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:37:59.653 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:37:59.683 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:59.702 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:37:59.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:59.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:37:59.756 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:37:59.806 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:37:59.834 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:37:59.873 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:37:59.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:37:59.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:37:59.978 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:38:00.010 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:00.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:00 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1678837072' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:38:00.825 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.11 to 1 (10000) 2024-04-20T17:38:00.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:00 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1678837072' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:38:00.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:00 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1678837072' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:38:01.714 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:38:01.752 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 7 1 2024-04-20T17:38:01.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:01 smithi106 ceph-mon[44423]: pgmap v1843: 17 pgs: 12 active+clean, 4 creating+peering, 1 unknown; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:38:01.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:01 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1678837072' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-20T17:38:01.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:01 smithi106 ceph-mon[44423]: osdmap e697: 12 total, 12 up, 10 in 2024-04-20T17:38:01.824 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:01 smithi005 ceph-mon[39562]: pgmap v1843: 17 pgs: 12 active+clean, 4 creating+peering, 1 unknown; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:38:01.825 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:01 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1678837072' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-20T17:38:01.825 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:01 smithi005 ceph-mon[39562]: osdmap e697: 12 total, 12 up, 10 in 2024-04-20T17:38:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:01 smithi171 ceph-mon[44189]: pgmap v1843: 17 pgs: 12 active+clean, 4 creating+peering, 1 unknown; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:38:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:01 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1678837072' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-20T17:38:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:01 smithi171 ceph-mon[44189]: osdmap e697: 12 total, 12 up, 10 in 2024-04-20T17:38:02.162 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:38:02.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:02.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:38:02.267 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:38:02.300 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:02.323 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:38:02.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:02.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:38:02.427 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:38:02.456 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:02.509 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:38:02.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:02.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:38:02.515 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:38:02.613 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:38:02.648 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:02.669 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:38:02.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:02.672 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:38:02.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:02 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2480302127' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:38:02.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:02 smithi005 ceph-mon[39562]: pgmap v1845: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:38:02.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:02 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2480302127' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:38:02.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:02 smithi106 ceph-mon[44423]: pgmap v1845: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:38:02.774 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:38:02.801 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:02 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2480302127' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:38:02.802 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:02 smithi171 ceph-mon[44189]: pgmap v1845: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:38:02.803 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:02.825 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:38:02.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:02.828 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:38:02.929 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:38:02.960 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:03.489 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:38:03.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:03.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:38:03.585 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:38:03.585 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:38:03.594 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:38:03.623 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:03 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:38:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:03 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2480302127' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:38:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:03 smithi005 ceph-mon[39562]: osdmap e698: 12 total, 12 up, 10 in 2024-04-20T17:38:03.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:03 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:38:03.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:03 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:38:03.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:03 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2480302127' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:38:03.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:03 smithi106 ceph-mon[44423]: osdmap e698: 12 total, 12 up, 10 in 2024-04-20T17:38:03.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:03 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:38:03.779 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:38:03.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:03.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:38:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:03 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:38:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:03 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2480302127' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:38:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:03 smithi171 ceph-mon[44189]: osdmap e698: 12 total, 12 up, 10 in 2024-04-20T17:38:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:03 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:38:03.883 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:38:03.910 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:03.928 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:38:03.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:03.932 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:38:04.033 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:38:04.061 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:04.081 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:38:04.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:04.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:38:04.182 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-20T17:38:04.184 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:38:04.212 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:04.360 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:38:04.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:04.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:38:04.441 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:38:04.465 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:38:04.494 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:04.515 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:38:04.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:04.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:38:04.619 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:38:04.648 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:04.668 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:38:04.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:04.671 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:38:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:04 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:38:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:04 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:38:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:04 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:38:04.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:04 smithi106 ceph-mon[44423]: pgmap v1847: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:38:04.772 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:38:04.796 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:04 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:38:04.796 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:04 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:38:04.796 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:04 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:38:04.796 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:04 smithi005 ceph-mon[39562]: pgmap v1847: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:38:04.797 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:04 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:38:04.797 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:04 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:38:04.797 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:04 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:38:04.797 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:04 smithi171 ceph-mon[44189]: pgmap v1847: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:38:04.799 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:04.818 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:38:04.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:04.822 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:38:04.923 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:38:04.951 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:05.199 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:38:05.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:05.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:38:05.303 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:38:05.333 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:05.354 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:38:05.354 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:38:05.406 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:38:05.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:05.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:38:05.510 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:38:05.539 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:05.593 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:38:05.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:05.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:38:05.697 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:38:05.705 INFO:teuthology.orchestra.run.smithi005.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-04-20T17:38:05.725 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:05.825 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:05 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/656143714' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:38:05.875 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:38:05.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:05.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:38:05.978 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:38:06.008 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:05 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/656143714' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:38:06.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:05 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/656143714' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:38:06.258 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:38:06.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:06.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:38:06.362 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:38:06.395 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:06.417 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:38:06.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:06.420 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:38:06.460 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:38:06.460 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:38:06.521 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:38:06.550 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:06.750 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:38:06.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:06.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:38:06.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:06 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/656143714' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-20T17:38:06.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:06 smithi106 ceph-mon[44423]: osdmap e699: 12 total, 12 up, 10 in 2024-04-20T17:38:06.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:06 smithi106 ceph-mon[44423]: pgmap v1849: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:38:06.854 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:38:06.880 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:06 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/656143714' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-20T17:38:06.880 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:06 smithi005 ceph-mon[39562]: osdmap e699: 12 total, 12 up, 10 in 2024-04-20T17:38:06.880 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:06 smithi005 ceph-mon[39562]: pgmap v1849: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:38:06.882 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:06.956 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:38:06.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:06.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:38:07.059 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:38:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:06 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/656143714' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-20T17:38:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:06 smithi171 ceph-mon[44189]: osdmap e699: 12 total, 12 up, 10 in 2024-04-20T17:38:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:06 smithi171 ceph-mon[44189]: pgmap v1849: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:38:07.088 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:07.193 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-20T17:38:07.212 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:38:07.436 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:38:07.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:07.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:38:07.460 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:38:07.541 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:38:07.567 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:07.765 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:38:07.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:07.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:38:07.870 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:38:07.900 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:07 smithi005 ceph-mon[39562]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-20T17:38:07.901 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:07.988 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:07 smithi106 ceph-mon[44423]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-20T17:38:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:07 smithi171 ceph-mon[44189]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-20T17:38:08.338 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:38:08.352 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:38:08.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:08.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:38:08.453 DEBUG:teuthology.orchestra.run.smithi171:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-04-20T17:38:08.457 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:38:08.458 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-20T17:38:08.458 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-20T17:38:08.487 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:08.558 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:hints = 1 2024-04-20T17:38:08.558 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-20T17:38:08.558 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Object prefix: benchmark_data_smithi171_161871 2024-04-20T17:38:08.559 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:38:08.559 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 0 0 0 0 0 0 - 0 2024-04-20T17:38:08.686 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:38:08.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:08.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:38:08.790 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:38:08.827 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:08.854 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:38:08.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:08.858 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:38:08.959 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:38:09.002 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:08 smithi005 ceph-mon[39562]: pgmap v1850: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:38:09.006 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:09.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:08 smithi106 ceph-mon[44423]: pgmap v1850: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:38:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:08 smithi171 ceph-mon[44189]: pgmap v1850: 17 pgs: 17 active+clean; 577 KiB data, 4.0 GiB used, 890 GiB / 894 GiB avail 2024-04-20T17:38:09.124 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:38:09.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:09.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:38:09.230 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:38:09.263 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:09.288 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:38:09.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:09.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:38:09.340 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:38:09.340 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:38:09.353 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:38:09.388 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:38:09.391 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:38:09.425 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:38:09.448 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:09.470 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:38:09.502 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:38:09.533 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:38:09.559 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 1 16 4846 4830 301.851 301.875 0.00239511 0.00330294 2024-04-20T17:38:09.561 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:38:09.591 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:38:09.633 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:38:09.667 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:38:09.700 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:38:09.734 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:38:09.752 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:38:09.753 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:38:09.753 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:38:09.753 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:38:09.753 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:38:09.753 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:38:09.753 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:38:09.754 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:38:09.754 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:38:09.754 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:38:09.754 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:38:09.754 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:38:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:09 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4180767944' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:38:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:10.073 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-20T17:38:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:09 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4180767944' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:38:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:10.126 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:38:10.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:10.129 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:09 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4180767944' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:38:10.130 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:10.130 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:10.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:38:10.231 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:38:10.277 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:10.385 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:38:10.559 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 2 15 9409 9394 293.539 285.25 0.00363131 0.00339899 2024-04-20T17:38:10.742 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:38:10.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:10.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:38:10.848 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:38:10.885 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:10.928 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:38:10.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:10.939 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:38:10.941 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:10.942 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:10.942 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:10.942 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:10.942 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:10.942 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:10.942 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:10.942 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:10.942 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:10.942 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:10.942 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:10 smithi106 ceph-mon[44423]: pgmap v1851: 17 pgs: 17 active+clean; 160 MiB data, 4.2 GiB used, 890 GiB / 894 GiB avail; 20 MiB/s wr, 318 op/s 2024-04-20T17:38:11.040 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:38:11.076 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:11.076 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:11.076 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:11.076 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:11.076 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:11.076 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:11.076 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:11.076 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:11.076 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:11.076 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:11.076 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:10 smithi171 ceph-mon[44189]: pgmap v1851: 17 pgs: 17 active+clean; 160 MiB data, 4.2 GiB used, 890 GiB / 894 GiB avail; 20 MiB/s wr, 318 op/s 2024-04-20T17:38:11.079 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:11.106 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:38:11.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:11.110 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:38:11.211 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:38:11.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:11.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:11.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:11.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:11.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:11.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:11.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:11.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:11.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:11.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:11.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:10 smithi005 ceph-mon[39562]: pgmap v1851: 17 pgs: 17 active+clean; 160 MiB data, 4.2 GiB used, 890 GiB / 894 GiB avail; 20 MiB/s wr, 318 op/s 2024-04-20T17:38:11.254 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:11.281 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:38:11.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:11.284 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:38:11.385 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:38:11.505 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:11.559 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 3 16 13655 13639 284.123 265.312 0.00238963 0.00351742 2024-04-20T17:38:11.739 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.7 to 1 (10000) 2024-04-20T17:38:11.768 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:11 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1924109070' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:38:11.768 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:11 smithi005 ceph-mon[39562]: from='client.27100 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:38:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:11 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1924109070' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:38:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:11 smithi106 ceph-mon[44423]: from='client.27100 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:38:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:11 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1924109070' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:38:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:11 smithi171 ceph-mon[44189]: from='client.27100 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:38:12.344 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:38:12.344 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-04-20T17:38:12.344 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-04-20T17:38:12.344 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:38:12.344 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.344 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:8 6.e 6:720218ef:::benchmark_data_smithi171_147972_object7:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e627)", 2024-04-20T17:38:12.345 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:33:43.456834+0000", 2024-04-20T17:38:12.345 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 268.88466827600001, 2024-04-20T17:38:12.345 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.097128813999999994, 2024-04-20T17:38:12.345 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:12.345 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:12.345 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:12.345 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:38:12.345 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:38:12.345 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:12.345 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 8 2024-04-20T17:38:12.345 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.345 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:12.345 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.345 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:12.346 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.456834+0000", 2024-04-20T17:38:12.346 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.346 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.346 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.346 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:12.346 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.456834+0000", 2024-04-20T17:38:12.346 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.346 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.346 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.346 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:12.346 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.456836+0000", 2024-04-20T17:38:12.346 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2239999999999998e-06 2024-04-20T17:38:12.346 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.346 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.347 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:12.347 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.456881+0000", 2024-04-20T17:38:12.347 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4685000000000003e-05 2024-04-20T17:38:12.347 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.347 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.347 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:12.347 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.456893+0000", 2024-04-20T17:38:12.347 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2143e-05 2024-04-20T17:38:12.347 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.349 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.349 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:12.350 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.456902+0000", 2024-04-20T17:38:12.350 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2960000000000004e-06 2024-04-20T17:38:12.350 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.350 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.350 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:12.350 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.456954+0000", 2024-04-20T17:38:12.350 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1343e-05 2024-04-20T17:38:12.350 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.350 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.350 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:38:12.350 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.456962+0000", 2024-04-20T17:38:12.350 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9130000000000003e-06 2024-04-20T17:38:12.350 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.351 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.351 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:12.351 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.550705+0000", 2024-04-20T17:38:12.351 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.093743873000000005 2024-04-20T17:38:12.351 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.351 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.351 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:12.351 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.550803+0000", 2024-04-20T17:38:12.351 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7194999999999994e-05 2024-04-20T17:38:12.351 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.351 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.351 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2", 2024-04-20T17:38:12.351 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.550917+0000", 2024-04-20T17:38:12.351 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000114635 2024-04-20T17:38:12.352 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.352 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.352 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:38:12.352 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.552020+0000", 2024-04-20T17:38:12.352 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0011025119999999999 2024-04-20T17:38:12.352 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.352 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.352 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:12.352 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.553878+0000", 2024-04-20T17:38:12.352 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0018585019999999999 2024-04-20T17:38:12.352 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.352 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.352 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:12.352 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.553917+0000", 2024-04-20T17:38:12.353 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8367e-05 2024-04-20T17:38:12.353 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.353 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.353 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:12.353 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.553963+0000", 2024-04-20T17:38:12.353 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6125999999999998e-05 2024-04-20T17:38:12.353 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.353 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:12.353 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.353 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.353 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.353 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:20 6.e 6:724c168c:::benchmark_data_smithi171_147972_object19:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e627)", 2024-04-20T17:38:12.353 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:33:43.456917+0000", 2024-04-20T17:38:12.353 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 268.88458512, 2024-04-20T17:38:12.354 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.097566832000000006, 2024-04-20T17:38:12.354 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:12.354 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:12.354 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:12.354 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:38:12.354 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:38:12.354 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:12.354 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 20 2024-04-20T17:38:12.354 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.354 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:12.354 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.354 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:12.354 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.456917+0000", 2024-04-20T17:38:12.354 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.355 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.355 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.355 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:12.355 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.456917+0000", 2024-04-20T17:38:12.355 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.355 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.355 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.355 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:12.355 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.456918+0000", 2024-04-20T17:38:12.355 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.035e-06 2024-04-20T17:38:12.355 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.355 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.355 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:12.355 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.456964+0000", 2024-04-20T17:38:12.356 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6252999999999999e-05 2024-04-20T17:38:12.356 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.356 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.356 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:12.356 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.456971+0000", 2024-04-20T17:38:12.356 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2960000000000004e-06 2024-04-20T17:38:12.356 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.356 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.356 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:12.356 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.456976+0000", 2024-04-20T17:38:12.356 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1909999999999999e-06 2024-04-20T17:38:12.356 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.356 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.356 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:12.356 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457009+0000", 2024-04-20T17:38:12.357 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2744999999999999e-05 2024-04-20T17:38:12.357 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.357 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.357 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:38:12.357 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457014+0000", 2024-04-20T17:38:12.357 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4439999999999999e-06 2024-04-20T17:38:12.357 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.357 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.357 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:12.357 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.551182+0000", 2024-04-20T17:38:12.357 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.094167718999999997 2024-04-20T17:38:12.357 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.357 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.357 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:12.358 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.551261+0000", 2024-04-20T17:38:12.358 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9594999999999999e-05 2024-04-20T17:38:12.358 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.358 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.358 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2", 2024-04-20T17:38:12.358 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.551356+0000", 2024-04-20T17:38:12.358 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4631999999999998e-05 2024-04-20T17:38:12.358 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.358 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.358 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:38:12.358 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.552628+0000", 2024-04-20T17:38:12.358 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001271887 2024-04-20T17:38:12.358 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.358 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.358 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:12.359 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.554389+0000", 2024-04-20T17:38:12.359 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017610550000000001 2024-04-20T17:38:12.359 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.359 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.359 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:12.359 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.554443+0000", 2024-04-20T17:38:12.359 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3715999999999999e-05 2024-04-20T17:38:12.359 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.359 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.359 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:12.359 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.554484+0000", 2024-04-20T17:38:12.359 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1264000000000003e-05 2024-04-20T17:38:12.359 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.359 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:12.359 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.360 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.360 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.360 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:29 6.e 6:717b5a9e:::benchmark_data_smithi171_147972_object28:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e627)", 2024-04-20T17:38:12.360 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:33:43.456985+0000", 2024-04-20T17:38:12.360 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 268.88451677799998, 2024-04-20T17:38:12.360 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.097665752999999994, 2024-04-20T17:38:12.360 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:12.360 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:12.360 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:12.360 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:38:12.360 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:38:12.360 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:12.360 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 29 2024-04-20T17:38:12.361 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.361 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:12.361 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.361 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:12.361 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.456985+0000", 2024-04-20T17:38:12.361 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.361 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.361 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.361 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:12.361 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.456985+0000", 2024-04-20T17:38:12.361 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.361 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.361 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.361 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:12.362 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.456986+0000", 2024-04-20T17:38:12.362 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8299999999999995e-07 2024-04-20T17:38:12.362 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.362 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.362 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:12.362 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457050+0000", 2024-04-20T17:38:12.362 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4165999999999995e-05 2024-04-20T17:38:12.362 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.362 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.362 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:12.362 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457058+0000", 2024-04-20T17:38:12.362 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5549999999999997e-06 2024-04-20T17:38:12.362 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.362 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.363 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:12.363 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457063+0000", 2024-04-20T17:38:12.363 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.702e-06 2024-04-20T17:38:12.363 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.363 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.363 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:12.363 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457092+0000", 2024-04-20T17:38:12.363 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8883000000000001e-05 2024-04-20T17:38:12.363 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.363 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.363 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:38:12.363 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457095+0000", 2024-04-20T17:38:12.363 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.755e-06 2024-04-20T17:38:12.363 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.364 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.364 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:12.364 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.551566+0000", 2024-04-20T17:38:12.364 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.094470815 2024-04-20T17:38:12.364 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.364 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.364 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:12.364 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.551689+0000", 2024-04-20T17:38:12.364 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012283699999999999 2024-04-20T17:38:12.364 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.364 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.364 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2", 2024-04-20T17:38:12.364 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.551783+0000", 2024-04-20T17:38:12.364 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3807999999999994e-05 2024-04-20T17:38:12.364 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.365 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.365 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:38:12.365 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.553184+0000", 2024-04-20T17:38:12.365 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001400985 2024-04-20T17:38:12.365 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.365 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.365 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:12.365 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.554548+0000", 2024-04-20T17:38:12.365 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013636919999999999 2024-04-20T17:38:12.365 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.365 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.365 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:12.365 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.554609+0000", 2024-04-20T17:38:12.365 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1482000000000005e-05 2024-04-20T17:38:12.366 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.366 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.366 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:12.366 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.554651+0000", 2024-04-20T17:38:12.366 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2190000000000001e-05 2024-04-20T17:38:12.366 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.366 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:12.366 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.366 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.366 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.366 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:58 6.e 6:7b14fdfd:::benchmark_data_smithi171_147972_object57:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e627)", 2024-04-20T17:38:12.366 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:33:43.457071+0000", 2024-04-20T17:38:12.366 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 268.884431103, 2024-04-20T17:38:12.367 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.097978955000000006, 2024-04-20T17:38:12.367 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:12.367 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:12.367 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:12.367 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:38:12.367 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:38:12.367 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:12.367 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 58 2024-04-20T17:38:12.367 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.367 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:12.367 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.367 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:12.367 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457071+0000", 2024-04-20T17:38:12.367 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.367 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.368 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.368 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:12.368 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457071+0000", 2024-04-20T17:38:12.368 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.368 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.368 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.368 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:12.368 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457072+0000", 2024-04-20T17:38:12.368 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8500000000000001e-07 2024-04-20T17:38:12.368 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.368 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.368 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:12.368 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457125+0000", 2024-04-20T17:38:12.368 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3664e-05 2024-04-20T17:38:12.368 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.369 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.369 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:12.369 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457132+0000", 2024-04-20T17:38:12.369 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6900000000000003e-06 2024-04-20T17:38:12.369 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.369 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.369 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:12.369 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457137+0000", 2024-04-20T17:38:12.369 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3460000000000004e-06 2024-04-20T17:38:12.369 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.369 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.369 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:12.369 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457169+0000", 2024-04-20T17:38:12.369 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2632000000000002e-05 2024-04-20T17:38:12.369 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.370 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.370 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:38:12.370 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457173+0000", 2024-04-20T17:38:12.370 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6950000000000002e-06 2024-04-20T17:38:12.370 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.370 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.370 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:12.370 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.552054+0000", 2024-04-20T17:38:12.370 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.094880860999999997 2024-04-20T17:38:12.370 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.370 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.370 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:12.370 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.552175+0000", 2024-04-20T17:38:12.370 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000120877 2024-04-20T17:38:12.371 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.371 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.371 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2", 2024-04-20T17:38:12.371 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.552331+0000", 2024-04-20T17:38:12.371 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015667799999999999 2024-04-20T17:38:12.371 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.371 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.371 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:38:12.371 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.553192+0000", 2024-04-20T17:38:12.371 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00086076100000000003 2024-04-20T17:38:12.371 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.371 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.371 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:12.371 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.554942+0000", 2024-04-20T17:38:12.371 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017503130000000001 2024-04-20T17:38:12.372 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.372 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.372 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:12.372 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.555008+0000", 2024-04-20T17:38:12.372 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5458999999999998e-05 2024-04-20T17:38:12.372 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.372 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.372 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:12.372 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.555050+0000", 2024-04-20T17:38:12.372 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2293999999999999e-05 2024-04-20T17:38:12.372 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.372 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:12.373 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.373 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.373 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.373 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:58580 6.2 6:41b576a0:::benchmark_data_smithi171_147972_object58579:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e631)", 2024-04-20T17:38:12.373 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:34:00.238471+0000", 2024-04-20T17:38:12.373 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 252.103031496, 2024-04-20T17:38:12.373 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.089870243000000002, 2024-04-20T17:38:12.373 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:12.373 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:12.373 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:12.373 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:38:12.373 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:38:12.373 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:12.373 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 58580 2024-04-20T17:38:12.374 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.374 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:12.374 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.374 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:12.374 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:00.238471+0000", 2024-04-20T17:38:12.374 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.374 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.374 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.374 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:12.374 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:00.238471+0000", 2024-04-20T17:38:12.374 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.374 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.374 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.374 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:12.375 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:00.238472+0000", 2024-04-20T17:38:12.375 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.336e-06 2024-04-20T17:38:12.375 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.375 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.375 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:12.375 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:00.238666+0000", 2024-04-20T17:38:12.375 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00019368099999999999 2024-04-20T17:38:12.375 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.375 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.375 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:12.375 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:00.238675+0000", 2024-04-20T17:38:12.375 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4269999999999997e-06 2024-04-20T17:38:12.375 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.375 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.375 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:12.376 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:00.238682+0000", 2024-04-20T17:38:12.376 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1910000000000003e-06 2024-04-20T17:38:12.376 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.376 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.376 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:12.376 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:00.240075+0000", 2024-04-20T17:38:12.376 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013927220000000001 2024-04-20T17:38:12.376 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.376 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.376 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:12.376 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:00.240203+0000", 2024-04-20T17:38:12.376 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000127775 2024-04-20T17:38:12.376 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.376 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.377 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:38:12.377 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:00.240317+0000", 2024-04-20T17:38:12.377 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011428499999999999 2024-04-20T17:38:12.377 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.377 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.377 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:38:12.377 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:00.242384+0000", 2024-04-20T17:38:12.377 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002066656 2024-04-20T17:38:12.377 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.377 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.377 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:12.377 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:00.328280+0000", 2024-04-20T17:38:12.377 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.085896383000000007 2024-04-20T17:38:12.377 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.377 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.378 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:12.378 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:00.328313+0000", 2024-04-20T17:38:12.378 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2975999999999997e-05 2024-04-20T17:38:12.378 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.378 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.378 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:12.378 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:00.328341+0000", 2024-04-20T17:38:12.378 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7810999999999999e-05 2024-04-20T17:38:12.378 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.378 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:12.378 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.378 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.378 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.378 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:58588 6.2 6:4810fb3e:::benchmark_data_smithi171_147972_object58587:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e631)", 2024-04-20T17:38:12.379 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:34:00.242544+0000", 2024-04-20T17:38:12.379 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 252.098958544, 2024-04-20T17:38:12.379 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.089095842999999994, 2024-04-20T17:38:12.379 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:12.379 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:12.379 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:12.379 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:38:12.379 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:38:12.379 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:12.379 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 58588 2024-04-20T17:38:12.379 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.379 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:12.379 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.379 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:12.380 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:00.242544+0000", 2024-04-20T17:38:12.380 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.380 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.380 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.380 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:12.380 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:00.242544+0000", 2024-04-20T17:38:12.380 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.380 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.380 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.380 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:12.380 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:00.242545+0000", 2024-04-20T17:38:12.380 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.621e-06 2024-04-20T17:38:12.380 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.380 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.380 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:12.381 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:00.242632+0000", 2024-04-20T17:38:12.381 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7059999999999997e-05 2024-04-20T17:38:12.381 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.381 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.381 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:12.381 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:00.242648+0000", 2024-04-20T17:38:12.381 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5716999999999999e-05 2024-04-20T17:38:12.381 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.381 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.381 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:12.381 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:00.242654+0000", 2024-04-20T17:38:12.381 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3230000000000003e-06 2024-04-20T17:38:12.381 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.381 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.382 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:12.382 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:00.242704+0000", 2024-04-20T17:38:12.382 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9489e-05 2024-04-20T17:38:12.382 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.382 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.382 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:12.382 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:00.242844+0000", 2024-04-20T17:38:12.382 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013972 2024-04-20T17:38:12.382 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.382 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.382 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:38:12.382 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:00.243007+0000", 2024-04-20T17:38:12.382 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016369099999999999 2024-04-20T17:38:12.382 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.382 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.383 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:38:12.383 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:00.244843+0000", 2024-04-20T17:38:12.383 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0018359819999999999 2024-04-20T17:38:12.383 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.383 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.383 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:12.383 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:00.331542+0000", 2024-04-20T17:38:12.383 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.086698876999999994 2024-04-20T17:38:12.383 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.383 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.383 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:12.383 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:00.331595+0000", 2024-04-20T17:38:12.383 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.274e-05 2024-04-20T17:38:12.383 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.383 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.384 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:12.384 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:00.331640+0000", 2024-04-20T17:38:12.384 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4623000000000003e-05 2024-04-20T17:38:12.384 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.384 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:12.384 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.384 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.384 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.384 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:63147 6.2 6:4c1cb04a:::benchmark_data_smithi171_147972_object63146:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e633)", 2024-04-20T17:38:12.384 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:34:02.239004+0000", 2024-04-20T17:38:12.384 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 250.10249828900001, 2024-04-20T17:38:12.384 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.090954210999999993, 2024-04-20T17:38:12.384 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:12.384 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:12.385 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:12.385 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:38:12.385 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:38:12.385 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:12.385 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 63147 2024-04-20T17:38:12.385 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.385 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:12.385 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.385 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:12.385 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:02.239004+0000", 2024-04-20T17:38:12.385 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.385 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.385 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.385 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:12.385 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:02.239004+0000", 2024-04-20T17:38:12.386 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.386 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.386 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.386 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:12.386 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:02.239006+0000", 2024-04-20T17:38:12.386 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5390000000000002e-06 2024-04-20T17:38:12.386 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.386 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.386 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:12.386 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:02.239055+0000", 2024-04-20T17:38:12.386 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8089e-05 2024-04-20T17:38:12.386 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.386 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.386 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:12.387 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:02.239069+0000", 2024-04-20T17:38:12.387 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4392e-05 2024-04-20T17:38:12.387 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.387 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.387 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:12.387 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:02.239079+0000", 2024-04-20T17:38:12.387 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9580000000000002e-06 2024-04-20T17:38:12.387 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.387 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.387 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:12.387 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:02.239125+0000", 2024-04-20T17:38:12.387 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6474999999999998e-05 2024-04-20T17:38:12.387 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.387 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.388 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:12.388 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:02.239234+0000", 2024-04-20T17:38:12.388 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010870900000000001 2024-04-20T17:38:12.388 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.388 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.388 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:38:12.388 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:02.239323+0000", 2024-04-20T17:38:12.388 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8881999999999994e-05 2024-04-20T17:38:12.388 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.388 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.388 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:38:12.388 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:02.241029+0000", 2024-04-20T17:38:12.388 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001705748 2024-04-20T17:38:12.388 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.389 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.389 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:12.389 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:02.329854+0000", 2024-04-20T17:38:12.389 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.088825374999999998 2024-04-20T17:38:12.389 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.389 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.389 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:12.389 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:02.329901+0000", 2024-04-20T17:38:12.389 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7162e-05 2024-04-20T17:38:12.389 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.389 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.389 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:12.389 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:02.329958+0000", 2024-04-20T17:38:12.389 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6882000000000002e-05 2024-04-20T17:38:12.390 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.390 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:12.390 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.390 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.390 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.390 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:63150 6.2 6:472adba3:::benchmark_data_smithi171_147972_object63149:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e633)", 2024-04-20T17:38:12.390 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:34:02.245884+0000", 2024-04-20T17:38:12.390 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 250.09561867299999, 2024-04-20T17:38:12.390 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.088685341000000001, 2024-04-20T17:38:12.390 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:12.390 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:12.390 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:12.390 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:38:12.390 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:38:12.391 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:12.391 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 63150 2024-04-20T17:38:12.391 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.391 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:12.391 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.391 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:12.391 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:02.245884+0000", 2024-04-20T17:38:12.391 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.391 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.391 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.391 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:12.391 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:02.245884+0000", 2024-04-20T17:38:12.391 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.391 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.392 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.392 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:12.392 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:02.245885+0000", 2024-04-20T17:38:12.392 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.297e-06 2024-04-20T17:38:12.392 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.392 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.392 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:12.392 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:02.245906+0000", 2024-04-20T17:38:12.392 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1297e-05 2024-04-20T17:38:12.392 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.392 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.392 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:12.392 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:02.245916+0000", 2024-04-20T17:38:12.392 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6050000000000004e-06 2024-04-20T17:38:12.392 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.393 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.393 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:12.393 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:02.245950+0000", 2024-04-20T17:38:12.393 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4183000000000001e-05 2024-04-20T17:38:12.393 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.393 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.393 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:12.393 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:02.245997+0000", 2024-04-20T17:38:12.393 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7012999999999999e-05 2024-04-20T17:38:12.393 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.393 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.393 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:12.393 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:02.246085+0000", 2024-04-20T17:38:12.393 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8096999999999994e-05 2024-04-20T17:38:12.394 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.394 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.394 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:38:12.394 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:02.246172+0000", 2024-04-20T17:38:12.394 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7202999999999998e-05 2024-04-20T17:38:12.394 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.394 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.394 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:38:12.394 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:02.247181+0000", 2024-04-20T17:38:12.394 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010092079999999999 2024-04-20T17:38:12.394 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.394 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.394 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:12.394 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:02.334471+0000", 2024-04-20T17:38:12.395 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.087289680999999994 2024-04-20T17:38:12.395 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.395 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.395 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:12.395 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:02.334528+0000", 2024-04-20T17:38:12.395 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6422000000000003e-05 2024-04-20T17:38:12.395 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.395 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.395 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:12.395 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:02.334569+0000", 2024-04-20T17:38:12.395 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1335000000000003e-05 2024-04-20T17:38:12.395 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.395 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:12.395 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.395 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.396 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.396 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:65214 6.2 6:469e7838:::benchmark_data_smithi171_147972_object65213:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e633)", 2024-04-20T17:38:12.396 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:34:03.251069+0000", 2024-04-20T17:38:12.396 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 249.090433601, 2024-04-20T17:38:12.396 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.087828934999999997, 2024-04-20T17:38:12.396 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:12.396 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:12.396 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:12.396 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:38:12.396 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:38:12.396 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:12.396 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 65214 2024-04-20T17:38:12.396 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.396 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:12.396 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.397 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:12.397 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:03.251069+0000", 2024-04-20T17:38:12.397 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.397 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.397 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.397 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:12.397 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:03.251069+0000", 2024-04-20T17:38:12.397 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.397 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.397 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.397 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:12.397 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:03.251070+0000", 2024-04-20T17:38:12.397 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5659999999999999e-06 2024-04-20T17:38:12.397 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.398 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.398 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:12.398 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:03.253320+0000", 2024-04-20T17:38:12.398 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022497289999999998 2024-04-20T17:38:12.398 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.398 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.398 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:12.398 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:03.253335+0000", 2024-04-20T17:38:12.398 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5252999999999999e-05 2024-04-20T17:38:12.398 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.398 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.398 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:12.398 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:03.253345+0000", 2024-04-20T17:38:12.398 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4399999999999994e-06 2024-04-20T17:38:12.398 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.399 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.399 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:12.399 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:03.257165+0000", 2024-04-20T17:38:12.399 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0038202739999999998 2024-04-20T17:38:12.399 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.399 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.399 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:12.399 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:03.257299+0000", 2024-04-20T17:38:12.399 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013364299999999999 2024-04-20T17:38:12.399 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.399 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.399 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:38:12.399 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:03.257442+0000", 2024-04-20T17:38:12.399 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014364 2024-04-20T17:38:12.400 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.400 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.400 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:38:12.400 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:03.259312+0000", 2024-04-20T17:38:12.400 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0018698549999999999 2024-04-20T17:38:12.400 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.400 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.400 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:12.400 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:03.338825+0000", 2024-04-20T17:38:12.400 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.079513340000000002 2024-04-20T17:38:12.400 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.400 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.400 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:12.400 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:03.338861+0000", 2024-04-20T17:38:12.400 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5728000000000001e-05 2024-04-20T17:38:12.401 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.401 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.401 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:12.401 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:34:03.338898+0000", 2024-04-20T17:38:12.401 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6467000000000002e-05 2024-04-20T17:38:12.401 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.401 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:12.401 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.401 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.401 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.401 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:155954 6.2 6:44e5c6b4:::benchmark_data_smithi171_147972_object155953:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e646)", 2024-04-20T17:38:12.401 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:04.109965+0000", 2024-04-20T17:38:12.401 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 188.23153691799999, 2024-04-20T17:38:12.401 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.134973547, 2024-04-20T17:38:12.402 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:12.402 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:12.402 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:12.402 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:38:12.402 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:38:12.402 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:12.402 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 155954 2024-04-20T17:38:12.402 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.402 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:12.402 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.402 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:12.402 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.109965+0000", 2024-04-20T17:38:12.402 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.402 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.403 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.403 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:12.403 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.109965+0000", 2024-04-20T17:38:12.403 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.403 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.403 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.403 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:12.403 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.109967+0000", 2024-04-20T17:38:12.403 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.17e-06 2024-04-20T17:38:12.403 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.403 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.403 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:12.403 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.110007+0000", 2024-04-20T17:38:12.403 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9799999999999998e-05 2024-04-20T17:38:12.404 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.404 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.404 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:12.404 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.110019+0000", 2024-04-20T17:38:12.404 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1402e-05 2024-04-20T17:38:12.404 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.404 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.404 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:12.404 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.110026+0000", 2024-04-20T17:38:12.404 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3490000000000003e-06 2024-04-20T17:38:12.404 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.404 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.404 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:12.404 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.110072+0000", 2024-04-20T17:38:12.404 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6300000000000001e-05 2024-04-20T17:38:12.405 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.405 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.405 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:12.405 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.110206+0000", 2024-04-20T17:38:12.405 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013326500000000001 2024-04-20T17:38:12.405 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.405 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.405 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:38:12.405 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.110350+0000", 2024-04-20T17:38:12.405 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014443400000000001 2024-04-20T17:38:12.405 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.405 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.405 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:38:12.405 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.120041+0000", 2024-04-20T17:38:12.406 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0096908340000000006 2024-04-20T17:38:12.406 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.406 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.406 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:12.406 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.244862+0000", 2024-04-20T17:38:12.406 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12482156899999999 2024-04-20T17:38:12.406 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.406 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.406 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:12.406 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.244895+0000", 2024-04-20T17:38:12.406 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2886000000000003e-05 2024-04-20T17:38:12.406 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.406 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.406 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:12.407 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.244939+0000", 2024-04-20T17:38:12.407 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3538000000000002e-05 2024-04-20T17:38:12.407 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.407 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:12.407 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.407 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.407 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.407 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:155956 6.2 6:45a14e0f:::benchmark_data_smithi171_147972_object155955:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e646)", 2024-04-20T17:38:12.407 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:04.113314+0000", 2024-04-20T17:38:12.407 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 188.228188488, 2024-04-20T17:38:12.407 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13176359100000001, 2024-04-20T17:38:12.407 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:12.407 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:12.407 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:12.407 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:38:12.408 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:38:12.408 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:12.408 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 155956 2024-04-20T17:38:12.408 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.408 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:12.408 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.408 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:12.408 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.113314+0000", 2024-04-20T17:38:12.408 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.408 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.408 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.408 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:12.408 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.113314+0000", 2024-04-20T17:38:12.408 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.408 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.409 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.409 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:12.409 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.113315+0000", 2024-04-20T17:38:12.409 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4559999999999999e-06 2024-04-20T17:38:12.409 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.409 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.409 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:12.409 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.113379+0000", 2024-04-20T17:38:12.409 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3938999999999999e-05 2024-04-20T17:38:12.409 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.409 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.409 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:12.409 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.113389+0000", 2024-04-20T17:38:12.410 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6479999999999998e-06 2024-04-20T17:38:12.410 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.410 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.410 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:12.410 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.113395+0000", 2024-04-20T17:38:12.410 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7559999999999996e-06 2024-04-20T17:38:12.410 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.410 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.410 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:12.410 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114959+0000", 2024-04-20T17:38:12.410 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001564766 2024-04-20T17:38:12.410 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.410 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.410 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:12.411 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.115095+0000", 2024-04-20T17:38:12.411 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000135566 2024-04-20T17:38:12.411 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.411 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.411 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:38:12.411 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.115239+0000", 2024-04-20T17:38:12.411 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014454600000000001 2024-04-20T17:38:12.411 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.411 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.411 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:12.411 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.244987+0000", 2024-04-20T17:38:12.411 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12974762200000001 2024-04-20T17:38:12.411 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.411 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.412 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:38:12.412 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.244997+0000", 2024-04-20T17:38:12.412 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6099999999999995e-06 2024-04-20T17:38:12.412 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.412 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.412 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:12.412 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.245027+0000", 2024-04-20T17:38:12.412 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0741000000000003e-05 2024-04-20T17:38:12.412 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.412 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.412 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:12.412 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.245077+0000", 2024-04-20T17:38:12.412 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9941000000000002e-05 2024-04-20T17:38:12.412 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.412 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:12.412 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.413 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.413 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.413 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:155958 6.2 6:47045901:::benchmark_data_smithi171_147972_object155957:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e646)", 2024-04-20T17:38:12.413 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:04.114372+0000", 2024-04-20T17:38:12.413 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 188.22712987200001, 2024-04-20T17:38:12.413 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13086056099999999, 2024-04-20T17:38:12.413 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:12.413 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:12.413 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:12.413 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:38:12.413 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:38:12.413 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:12.413 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 155958 2024-04-20T17:38:12.414 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.414 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:12.414 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.414 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:12.414 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114372+0000", 2024-04-20T17:38:12.414 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.414 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.414 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.414 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:12.414 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114372+0000", 2024-04-20T17:38:12.414 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.414 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.414 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.414 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:12.415 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114374+0000", 2024-04-20T17:38:12.415 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5370000000000001e-06 2024-04-20T17:38:12.415 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.415 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.415 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:12.415 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114433+0000", 2024-04-20T17:38:12.415 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9064000000000002e-05 2024-04-20T17:38:12.415 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.415 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.415 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:12.415 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114443+0000", 2024-04-20T17:38:12.415 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0236e-05 2024-04-20T17:38:12.415 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.415 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.415 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:12.416 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114450+0000", 2024-04-20T17:38:12.416 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1940000000000001e-06 2024-04-20T17:38:12.416 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.416 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.416 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:12.416 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.120077+0000", 2024-04-20T17:38:12.416 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.005626458 2024-04-20T17:38:12.416 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.416 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.416 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:12.416 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.232545+0000", 2024-04-20T17:38:12.416 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.112467761 2024-04-20T17:38:12.416 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.416 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.417 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:38:12.417 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.232710+0000", 2024-04-20T17:38:12.417 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016564600000000001 2024-04-20T17:38:12.417 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.417 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.417 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:38:12.417 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.245088+0000", 2024-04-20T17:38:12.417 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012377384999999999 2024-04-20T17:38:12.417 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.417 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.417 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:12.417 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.245134+0000", 2024-04-20T17:38:12.417 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5926e-05 2024-04-20T17:38:12.417 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.417 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.418 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:12.418 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.245183+0000", 2024-04-20T17:38:12.418 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9815999999999999e-05 2024-04-20T17:38:12.418 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.418 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.418 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:12.418 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.245233+0000", 2024-04-20T17:38:12.418 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9537999999999999e-05 2024-04-20T17:38:12.418 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.418 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:12.418 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.418 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.418 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.418 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:155959 6.e 6:7825f00e:::benchmark_data_smithi171_147972_object155958:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e646)", 2024-04-20T17:38:12.419 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:04.114461+0000", 2024-04-20T17:38:12.419 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 188.227040964, 2024-04-20T17:38:12.419 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12842199500000001, 2024-04-20T17:38:12.419 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:12.419 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:12.419 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:12.419 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:38:12.419 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:38:12.419 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:12.419 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 155959 2024-04-20T17:38:12.419 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.419 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:12.419 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.419 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:12.420 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114461+0000", 2024-04-20T17:38:12.420 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.420 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.420 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.420 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:12.420 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114461+0000", 2024-04-20T17:38:12.420 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.420 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.420 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.420 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:12.420 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114462+0000", 2024-04-20T17:38:12.420 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7199999999999997e-07 2024-04-20T17:38:12.420 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.420 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.421 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:12.421 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114522+0000", 2024-04-20T17:38:12.421 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9587999999999999e-05 2024-04-20T17:38:12.421 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.421 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.421 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:12.421 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114528+0000", 2024-04-20T17:38:12.421 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7039999999999997e-06 2024-04-20T17:38:12.421 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.421 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.421 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:12.421 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114534+0000", 2024-04-20T17:38:12.421 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.981e-06 2024-04-20T17:38:12.421 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.421 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.422 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:12.422 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.116285+0000", 2024-04-20T17:38:12.422 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001750754 2024-04-20T17:38:12.422 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.422 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.422 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:12.422 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.116400+0000", 2024-04-20T17:38:12.422 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011446999999999999 2024-04-20T17:38:12.422 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.422 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.422 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2", 2024-04-20T17:38:12.422 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.116529+0000", 2024-04-20T17:38:12.422 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012922300000000001 2024-04-20T17:38:12.423 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.423 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.423 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:12.423 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.233387+0000", 2024-04-20T17:38:12.423 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.11685820099999999 2024-04-20T17:38:12.423 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.423 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.423 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:38:12.423 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.242762+0000", 2024-04-20T17:38:12.423 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0093748029999999993 2024-04-20T17:38:12.423 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.423 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.423 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:12.423 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.242820+0000", 2024-04-20T17:38:12.423 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8207e-05 2024-04-20T17:38:12.424 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.424 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.424 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:12.424 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.242883+0000", 2024-04-20T17:38:12.424 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3192000000000001e-05 2024-04-20T17:38:12.424 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.424 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:12.424 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.424 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.424 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.424 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26882.0:155961 6.6 e646/619 6:62ebcb10:::benchmark_data_smithi171_147972_object155960:head v 646'9787, mlcod=646'9785)", 2024-04-20T17:38:12.424 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:04.115125+0000", 2024-04-20T17:38:12.424 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 188.22637724800001, 2024-04-20T17:38:12.424 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.123201243, 2024-04-20T17:38:12.424 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:12.425 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:12.425 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:12.425 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:12.425 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.425 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:12.425 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.115125+0000", 2024-04-20T17:38:12.425 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.425 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.425 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.425 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:12.425 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.115125+0000", 2024-04-20T17:38:12.425 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.425 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.425 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.425 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:12.426 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.115127+0000", 2024-04-20T17:38:12.426 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7489999999999999e-06 2024-04-20T17:38:12.426 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.426 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.426 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:12.426 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.115169+0000", 2024-04-20T17:38:12.426 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2305999999999998e-05 2024-04-20T17:38:12.426 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.426 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.426 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:12.426 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.115178+0000", 2024-04-20T17:38:12.426 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6419999999999996e-06 2024-04-20T17:38:12.426 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.426 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.427 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:12.427 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.115185+0000", 2024-04-20T17:38:12.427 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3780000000000003e-06 2024-04-20T17:38:12.427 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.427 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.427 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:12.427 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.116276+0000", 2024-04-20T17:38:12.427 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010913470000000001 2024-04-20T17:38:12.427 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.427 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.427 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:12.427 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.116294+0000", 2024-04-20T17:38:12.427 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7342000000000001e-05 2024-04-20T17:38:12.427 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.428 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.428 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:12.428 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.238286+0000", 2024-04-20T17:38:12.428 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12199251899999999 2024-04-20T17:38:12.428 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.428 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.428 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:12.428 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.238326+0000", 2024-04-20T17:38:12.428 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9959999999999997e-05 2024-04-20T17:38:12.428 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.428 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:12.428 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.428 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.428 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.428 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.26882.0:155954 6.2 e646/619 ondisk, result = 0)", 2024-04-20T17:38:12.429 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:04.120868+0000", 2024-04-20T17:38:12.429 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 188.22063463200001, 2024-04-20T17:38:12.429 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12407834299999999, 2024-04-20T17:38:12.429 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:12.429 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:12.429 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-04-20T17:38:12.429 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:12.429 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.429 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:12.429 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.120868+0000", 2024-04-20T17:38:12.429 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.429 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.429 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.429 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:12.430 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.120868+0000", 2024-04-20T17:38:12.430 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.430 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.430 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.430 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:12.430 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.120869+0000", 2024-04-20T17:38:12.430 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4619999999999999e-06 2024-04-20T17:38:12.430 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.430 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.430 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:12.430 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.120876+0000", 2024-04-20T17:38:12.430 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6869999999999997e-06 2024-04-20T17:38:12.430 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.430 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.430 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:12.431 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.232196+0000", 2024-04-20T17:38:12.431 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.11132056999999999 2024-04-20T17:38:12.431 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.431 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.431 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:12.431 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.232205+0000", 2024-04-20T17:38:12.431 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1400000000000006e-06 2024-04-20T17:38:12.431 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.431 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.431 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:12.431 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.244854+0000", 2024-04-20T17:38:12.431 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012648995 2024-04-20T17:38:12.431 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.431 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.432 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:12.432 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.244859+0000", 2024-04-20T17:38:12.432 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5510000000000001e-06 2024-04-20T17:38:12.432 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.432 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.432 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:12.432 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.244946+0000", 2024-04-20T17:38:12.432 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6938e-05 2024-04-20T17:38:12.432 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.432 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:12.432 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.432 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.432 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.432 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:155974 6.e 6:7d3bdd8a:::benchmark_data_smithi171_147972_object155973:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e646)", 2024-04-20T17:38:12.432 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:04.129245+0000", 2024-04-20T17:38:12.433 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 188.21225745199999, 2024-04-20T17:38:12.433 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.113882792, 2024-04-20T17:38:12.433 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:12.433 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:12.433 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:12.433 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:38:12.433 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:38:12.433 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:12.433 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 155974 2024-04-20T17:38:12.433 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.433 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:12.433 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.433 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:12.433 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.129245+0000", 2024-04-20T17:38:12.433 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.434 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.434 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.434 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:12.434 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.129245+0000", 2024-04-20T17:38:12.434 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.434 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.434 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.434 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:12.434 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.129247+0000", 2024-04-20T17:38:12.434 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2790000000000001e-06 2024-04-20T17:38:12.434 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.434 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.434 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:12.434 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.129324+0000", 2024-04-20T17:38:12.435 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6453999999999997e-05 2024-04-20T17:38:12.435 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.435 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.435 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:12.435 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.232085+0000", 2024-04-20T17:38:12.435 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.102761221 2024-04-20T17:38:12.435 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.435 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.435 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:12.435 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.232103+0000", 2024-04-20T17:38:12.435 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8057999999999999e-05 2024-04-20T17:38:12.435 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.435 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.435 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:12.435 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.233436+0000", 2024-04-20T17:38:12.436 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013337119999999999 2024-04-20T17:38:12.436 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.436 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.436 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:12.436 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.233544+0000", 2024-04-20T17:38:12.436 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000107865 2024-04-20T17:38:12.436 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.436 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.436 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2", 2024-04-20T17:38:12.436 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.233686+0000", 2024-04-20T17:38:12.436 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000141435 2024-04-20T17:38:12.436 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.436 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.436 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:12.437 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.242948+0000", 2024-04-20T17:38:12.437 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0092626500000000007 2024-04-20T17:38:12.437 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.437 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.437 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:38:12.437 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.243023+0000", 2024-04-20T17:38:12.437 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4752000000000005e-05 2024-04-20T17:38:12.437 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.437 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.437 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:12.437 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.243069+0000", 2024-04-20T17:38:12.437 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5723000000000003e-05 2024-04-20T17:38:12.437 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.437 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.437 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:12.438 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.243128+0000", 2024-04-20T17:38:12.438 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8643000000000001e-05 2024-04-20T17:38:12.438 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.438 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:12.438 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.438 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.438 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.438 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:180908 6.2 6:4681e151:::benchmark_data_smithi171_147972_object19810:head [delete] snapc 0=[] ondisk+write+known_if_redirected e653)", 2024-04-20T17:38:12.438 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:26.376220+0000", 2024-04-20T17:38:12.438 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 165.96528187300001, 2024-04-20T17:38:12.438 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.11115101199999999, 2024-04-20T17:38:12.438 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:12.438 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:12.438 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:12.439 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:38:12.439 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:38:12.439 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:12.439 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 180908 2024-04-20T17:38:12.439 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.439 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:12.439 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.439 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:12.439 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.376220+0000", 2024-04-20T17:38:12.439 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.439 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.439 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.439 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:12.439 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.376220+0000", 2024-04-20T17:38:12.440 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.440 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.440 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.440 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:12.440 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.376224+0000", 2024-04-20T17:38:12.440 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.162e-06 2024-04-20T17:38:12.440 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.440 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.440 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:12.440 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.376232+0000", 2024-04-20T17:38:12.440 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0690000000000002e-06 2024-04-20T17:38:12.440 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.440 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.440 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:12.441 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.376244+0000", 2024-04-20T17:38:12.441 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2551000000000001e-05 2024-04-20T17:38:12.441 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.441 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.441 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:12.441 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.376255+0000", 2024-04-20T17:38:12.441 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0549e-05 2024-04-20T17:38:12.441 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.441 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.441 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:12.441 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.376312+0000", 2024-04-20T17:38:12.441 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7708999999999999e-05 2024-04-20T17:38:12.441 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.441 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.441 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:12.442 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.376417+0000", 2024-04-20T17:38:12.442 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000105049 2024-04-20T17:38:12.442 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.442 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.442 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:38:12.442 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.376493+0000", 2024-04-20T17:38:12.442 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5427999999999997e-05 2024-04-20T17:38:12.442 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.442 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.442 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:38:12.442 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.380255+0000", 2024-04-20T17:38:12.442 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003761838 2024-04-20T17:38:12.442 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.442 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.443 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:12.443 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.487254+0000", 2024-04-20T17:38:12.443 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.10699963899999999 2024-04-20T17:38:12.443 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.443 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.443 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:12.443 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.487316+0000", 2024-04-20T17:38:12.443 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1161000000000006e-05 2024-04-20T17:38:12.443 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.443 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.443 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:12.443 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.487371+0000", 2024-04-20T17:38:12.443 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5856999999999997e-05 2024-04-20T17:38:12.443 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.443 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:12.444 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.444 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.444 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.444 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:180915 6.2 6:4449404c:::benchmark_data_smithi171_147972_object19817:head [delete] snapc 0=[] ondisk+write+known_if_redirected e653)", 2024-04-20T17:38:12.444 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:26.382605+0000", 2024-04-20T17:38:12.444 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 165.95889694499999, 2024-04-20T17:38:12.444 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.115379974, 2024-04-20T17:38:12.444 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:12.444 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:12.444 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:12.444 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:38:12.444 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:38:12.444 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:12.444 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 180915 2024-04-20T17:38:12.445 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.445 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:12.445 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.445 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:12.445 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382605+0000", 2024-04-20T17:38:12.445 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.445 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.445 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.445 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:12.445 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382605+0000", 2024-04-20T17:38:12.445 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.445 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.445 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.445 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:12.446 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382607+0000", 2024-04-20T17:38:12.446 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0159999999999998e-06 2024-04-20T17:38:12.446 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.446 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.446 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:12.446 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382613+0000", 2024-04-20T17:38:12.446 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1659999999999999e-06 2024-04-20T17:38:12.446 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.446 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.446 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:12.446 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382639+0000", 2024-04-20T17:38:12.446 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5096000000000002e-05 2024-04-20T17:38:12.446 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.446 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.447 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:12.447 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382645+0000", 2024-04-20T17:38:12.447 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6470000000000001e-06 2024-04-20T17:38:12.447 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.447 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.447 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:12.447 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382709+0000", 2024-04-20T17:38:12.447 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3996000000000002e-05 2024-04-20T17:38:12.447 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.447 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.447 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:12.447 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382804+0000", 2024-04-20T17:38:12.447 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4871999999999996e-05 2024-04-20T17:38:12.447 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.447 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.448 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:38:12.448 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382873+0000", 2024-04-20T17:38:12.448 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8585999999999997e-05 2024-04-20T17:38:12.448 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.448 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.448 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:38:12.448 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.390622+0000", 2024-04-20T17:38:12.448 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0077494310000000002 2024-04-20T17:38:12.448 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.448 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.448 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:12.448 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.497875+0000", 2024-04-20T17:38:12.448 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.107252942 2024-04-20T17:38:12.448 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.449 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.449 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:12.449 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.497928+0000", 2024-04-20T17:38:12.449 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2814e-05 2024-04-20T17:38:12.449 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.449 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.449 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:12.449 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.497985+0000", 2024-04-20T17:38:12.449 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7408000000000003e-05 2024-04-20T17:38:12.449 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.449 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:12.449 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.449 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.449 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.449 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:180917 6.2 6:4941fea8:::benchmark_data_smithi171_147972_object19819:head [delete] snapc 0=[] ondisk+write+known_if_redirected e653)", 2024-04-20T17:38:12.450 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:26.382662+0000", 2024-04-20T17:38:12.450 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 165.958840184, 2024-04-20T17:38:12.450 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.117894331, 2024-04-20T17:38:12.450 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:12.450 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:12.450 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:12.450 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:38:12.450 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:38:12.450 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:12.450 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 180917 2024-04-20T17:38:12.450 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.450 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:12.450 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.450 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:12.451 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382662+0000", 2024-04-20T17:38:12.451 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.451 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.451 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.451 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:12.451 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382662+0000", 2024-04-20T17:38:12.451 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.451 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.451 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.451 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:12.451 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382663+0000", 2024-04-20T17:38:12.451 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9599999999999998e-07 2024-04-20T17:38:12.451 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.451 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.451 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:12.452 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382666+0000", 2024-04-20T17:38:12.452 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.275e-06 2024-04-20T17:38:12.452 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.452 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.452 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:12.452 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382677+0000", 2024-04-20T17:38:12.452 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0759e-05 2024-04-20T17:38:12.452 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.452 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.452 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:12.452 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382681+0000", 2024-04-20T17:38:12.452 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.25e-06 2024-04-20T17:38:12.452 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.452 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.452 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:12.453 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.386364+0000", 2024-04-20T17:38:12.453 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0036826049999999998 2024-04-20T17:38:12.453 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.453 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.453 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:12.453 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.386490+0000", 2024-04-20T17:38:12.453 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000126716 2024-04-20T17:38:12.453 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.453 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.453 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:38:12.453 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.386614+0000", 2024-04-20T17:38:12.453 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012380799999999999 2024-04-20T17:38:12.453 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.453 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.453 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:38:12.454 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.394543+0000", 2024-04-20T17:38:12.454 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0079287790000000004 2024-04-20T17:38:12.454 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.454 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.454 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:12.454 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.500468+0000", 2024-04-20T17:38:12.454 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.10592457600000001 2024-04-20T17:38:12.454 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.454 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.454 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:12.454 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.500514+0000", 2024-04-20T17:38:12.454 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5905000000000002e-05 2024-04-20T17:38:12.454 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.454 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.455 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:12.455 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.500556+0000", 2024-04-20T17:38:12.455 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2861999999999997e-05 2024-04-20T17:38:12.455 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.455 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:12.455 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.455 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.455 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.455 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:180920 6.2 6:438d9d13:::benchmark_data_smithi171_147972_object19822:head [delete] snapc 0=[] ondisk+write+known_if_redirected e653)", 2024-04-20T17:38:12.455 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:26.383540+0000", 2024-04-20T17:38:12.455 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 165.95796210699999, 2024-04-20T17:38:12.455 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.11715096899999999, 2024-04-20T17:38:12.455 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:12.455 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:12.456 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:12.456 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:38:12.456 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:38:12.456 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:12.456 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 180920 2024-04-20T17:38:12.456 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.456 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:12.456 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.456 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:12.456 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.383540+0000", 2024-04-20T17:38:12.456 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.456 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.456 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.456 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:12.456 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.383540+0000", 2024-04-20T17:38:12.457 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:12.457 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.457 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.457 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:12.457 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.383542+0000", 2024-04-20T17:38:12.457 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4899999999999999e-06 2024-04-20T17:38:12.457 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.457 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.457 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:12.457 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.383549+0000", 2024-04-20T17:38:12.457 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4930000000000001e-06 2024-04-20T17:38:12.457 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.457 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.457 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:12.457 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.383560+0000", 2024-04-20T17:38:12.458 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1024e-05 2024-04-20T17:38:12.458 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.458 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.458 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:12.458 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.383567+0000", 2024-04-20T17:38:12.458 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3359999999999997e-06 2024-04-20T17:38:12.458 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.458 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.458 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:12.458 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.390655+0000", 2024-04-20T17:38:12.458 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0070878219999999997 2024-04-20T17:38:12.458 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.458 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.458 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:12.458 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.390773+0000", 2024-04-20T17:38:12.459 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000117464 2024-04-20T17:38:12.459 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.459 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.459 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:38:12.459 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.390888+0000", 2024-04-20T17:38:12.459 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011526 2024-04-20T17:38:12.459 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.459 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.459 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:38:12.459 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.394670+0000", 2024-04-20T17:38:12.459 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037814910000000001 2024-04-20T17:38:12.459 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.459 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.459 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:12.459 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.500601+0000", 2024-04-20T17:38:12.460 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.10593155 2024-04-20T17:38:12.460 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.460 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.460 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:12.460 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.500653+0000", 2024-04-20T17:38:12.460 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1504e-05 2024-04-20T17:38:12.460 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:12.460 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:12.460 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:12.460 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.500691+0000", 2024-04-20T17:38:12.460 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8535000000000003e-05 2024-04-20T17:38:12.460 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.460 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:12.460 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.460 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:12.461 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:12.461 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:38:12.559 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 4 16 17677 17661 275.931 251.375 0.00243324 0.00362111 2024-04-20T17:38:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:12 smithi106 ceph-mon[44423]: from='client.27100 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-20T17:38:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:12 smithi106 ceph-mon[44423]: osdmap e700: 12 total, 12 up, 10 in 2024-04-20T17:38:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:12 smithi106 ceph-mon[44423]: pgmap v1853: 17 pgs: 17 active+clean; 392 MiB data, 4.6 GiB used, 889 GiB / 894 GiB avail; 49 MiB/s wr, 781 op/s 2024-04-20T17:38:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:12 smithi171 ceph-mon[44189]: from='client.27100 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-20T17:38:13.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:12 smithi171 ceph-mon[44189]: osdmap e700: 12 total, 12 up, 10 in 2024-04-20T17:38:13.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:12 smithi171 ceph-mon[44189]: pgmap v1853: 17 pgs: 17 active+clean; 392 MiB data, 4.6 GiB used, 889 GiB / 894 GiB avail; 49 MiB/s wr, 781 op/s 2024-04-20T17:38:13.106 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-20T17:38:13.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:12 smithi005 ceph-mon[39562]: from='client.27100 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-20T17:38:13.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:12 smithi005 ceph-mon[39562]: osdmap e700: 12 total, 12 up, 10 in 2024-04-20T17:38:13.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:12 smithi005 ceph-mon[39562]: pgmap v1853: 17 pgs: 17 active+clean; 392 MiB data, 4.6 GiB used, 889 GiB / 894 GiB avail; 49 MiB/s wr, 781 op/s 2024-04-20T17:38:13.414 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:38:13.559 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 5 15 21342 21327 266.566 229.125 0.00625133 0.00374926 2024-04-20T17:38:14.559 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 6 16 24843 24827 258.593 218.75 0.00564041 0.00386495 2024-04-20T17:38:15.005 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:14 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:38:15.005 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:14 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:38:15.005 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:14 smithi106 ceph-mon[44423]: osdmap e701: 12 total, 12 up, 10 in 2024-04-20T17:38:15.005 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:14 smithi106 ceph-mon[44423]: pgmap v1855: 17 pgs: 17 active+clean; 876 MiB data, 5.7 GiB used, 888 GiB / 894 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-04-20T17:38:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:14 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:38:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:14 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:38:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:14 smithi171 ceph-mon[44189]: osdmap e701: 12 total, 12 up, 10 in 2024-04-20T17:38:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:14 smithi171 ceph-mon[44189]: pgmap v1855: 17 pgs: 17 active+clean; 876 MiB data, 5.7 GiB used, 888 GiB / 894 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-04-20T17:38:15.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:14 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:38:15.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:14 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:38:15.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:14 smithi005 ceph-mon[39562]: osdmap e701: 12 total, 12 up, 10 in 2024-04-20T17:38:15.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:14 smithi005 ceph-mon[39562]: pgmap v1855: 17 pgs: 17 active+clean; 876 MiB data, 5.7 GiB used, 888 GiB / 894 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-04-20T17:38:15.296 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:38:15.296 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:38:15.296 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-04-20T17:38:15.296 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:38:15.559 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 7 16 28156 28140 251.229 207.062 0.00511863 0.00397836 2024-04-20T17:38:15.875 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:38:15.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:15.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:38:15.986 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:38:16.021 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:16.046 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:38:16.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:16.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:38:16.068 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-20T17:38:16.152 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:38:16.191 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:16.220 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:38:16.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:16.223 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:38:16.324 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:38:16.373 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:16.411 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:38:16.434 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:38:16.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:16.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:38:16.541 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:38:16.560 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 8 16 31419 31403 245.317 203.938 0.00732048 0.00407429 2024-04-20T17:38:16.582 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:16.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:16 smithi005 ceph-mon[39562]: pgmap v1856: 17 pgs: 17 active+clean; 1.4 GiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 184 MiB/s wr, 2.95k op/s 2024-04-20T17:38:16.732 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:38:16.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:16.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:38:16.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:16 smithi171 ceph-mon[44189]: pgmap v1856: 17 pgs: 17 active+clean; 1.4 GiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 184 MiB/s wr, 2.95k op/s 2024-04-20T17:38:16.841 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:38:16.872 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:16 smithi106 ceph-mon[44423]: pgmap v1856: 17 pgs: 17 active+clean; 1.4 GiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 184 MiB/s wr, 2.95k op/s 2024-04-20T17:38:16.875 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:17.325 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:38:17.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:17.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:38:17.428 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:38:17.484 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:17.559 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 9 16 34026 34010 236.162 162.938 0.00871234 0.00423261 2024-04-20T17:38:17.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:17 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2445149109' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:38:17.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:17 smithi106 ceph-mon[44423]: from='client.27212 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:38:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:17 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2445149109' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:38:17.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:17 smithi171 ceph-mon[44189]: from='client.27212 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:38:17.894 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:38:17.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:17 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2445149109' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:38:17.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:17 smithi005 ceph-mon[39562]: from='client.27212 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:38:18.427 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:38:18.520 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:38:18.520 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:38:18.520 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-04-20T17:38:18.520 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-04-20T17:38:18.520 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:38:18.559 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 10 15 36751 36736 229.582 170.375 0.0053136 0.00435399 2024-04-20T17:38:18.759 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:38:18.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:18.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:38:18.786 INFO:tasks.thrashosds.thrasher:test_map_discontinuity 2024-04-20T17:38:18.786 INFO:tasks.thrashosds.thrasher:Waiting for recovery 2024-04-20T17:38:18.786 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-20T17:38:18.786 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-20T17:38:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:18 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:38:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:18 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:38:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:18 smithi171 ceph-mon[44189]: from='client.27212 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:38:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:18 smithi171 ceph-mon[44189]: osdmap e702: 12 total, 12 up, 10 in 2024-04-20T17:38:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:18 smithi171 ceph-mon[44189]: pgmap v1858: 17 pgs: 17 active+clean; 1.6 GiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 182 MiB/s wr, 2.91k op/s 2024-04-20T17:38:18.858 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:18 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:38:18.858 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:18 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:38:18.858 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:18 smithi106 ceph-mon[44423]: from='client.27212 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:38:18.858 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:18 smithi106 ceph-mon[44423]: osdmap e702: 12 total, 12 up, 10 in 2024-04-20T17:38:18.858 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:18 smithi106 ceph-mon[44423]: pgmap v1858: 17 pgs: 17 active+clean; 1.6 GiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 182 MiB/s wr, 2.91k op/s 2024-04-20T17:38:18.865 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:38:18.887 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:18 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:38:18.887 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:18 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:38:18.887 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:18 smithi005 ceph-mon[39562]: from='client.27212 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:38:18.887 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:18 smithi005 ceph-mon[39562]: osdmap e702: 12 total, 12 up, 10 in 2024-04-20T17:38:18.887 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:18 smithi005 ceph-mon[39562]: pgmap v1858: 17 pgs: 17 active+clean; 1.6 GiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 182 MiB/s wr, 2.91k op/s 2024-04-20T17:38:18.897 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:19.245 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:38:19.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:19.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:38:19.267 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-20T17:38:19.349 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:38:19.381 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:19.402 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:38:19.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:19.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:38:19.465 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:38:19.507 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:38:19.551 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:38:19.559 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 11 16 39427 39411 223.909 167.188 0.00343961 0.00446346 2024-04-20T17:38:19.575 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:19.610 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:38:19.664 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:38:19.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:19.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:38:19.770 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:38:19.807 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:19.833 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:38:19.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:19.837 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:38:19.938 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:38:19.980 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:20.007 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:38:20.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:20.011 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:38:20.112 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:38:20.175 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:20.560 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 12 16 42107 42091 219.207 167.5 0.00925642 0.00455964 2024-04-20T17:38:20.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:20 smithi171 ceph-mon[44189]: pgmap v1859: 17 pgs: 17 active+clean; 2.2 GiB data, 8.3 GiB used, 886 GiB / 894 GiB avail; 233 MiB/s wr, 3.72k op/s 2024-04-20T17:38:20.858 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:20 smithi106 ceph-mon[44423]: pgmap v1859: 17 pgs: 17 active+clean; 2.2 GiB data, 8.3 GiB used, 886 GiB / 894 GiB avail; 233 MiB/s wr, 3.72k op/s 2024-04-20T17:38:20.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:20 smithi005 ceph-mon[39562]: pgmap v1859: 17 pgs: 17 active+clean; 2.2 GiB data, 8.3 GiB used, 886 GiB / 894 GiB avail; 233 MiB/s wr, 3.72k op/s 2024-04-20T17:38:21.474 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:38:21.560 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 13 16 44746 44730 215.031 164.938 0.00242424 0.0046289 2024-04-20T17:38:21.872 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:38:21.872 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-04-20T17:38:21.872 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-04-20T17:38:21.873 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:38:21.873 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.873 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26882.0:155965 6.a e646/619 6:5b55946e:::benchmark_data_smithi171_147972_object155964:head v 646'9852, mlcod=646'9848)", 2024-04-20T17:38:21.873 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:04.122078+0000", 2024-04-20T17:38:21.873 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 197.748597225, 2024-04-20T17:38:21.873 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.082524620000000007, 2024-04-20T17:38:21.873 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:21.873 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:21.873 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:21.873 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:21.873 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.873 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:21.873 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.122078+0000", 2024-04-20T17:38:21.873 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.873 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.874 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.874 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:21.874 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.122078+0000", 2024-04-20T17:38:21.874 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.874 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.874 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.874 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:21.874 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.122078+0000", 2024-04-20T17:38:21.874 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.02e-07 2024-04-20T17:38:21.874 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.874 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.874 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:21.874 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.122132+0000", 2024-04-20T17:38:21.875 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3091999999999999e-05 2024-04-20T17:38:21.875 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.875 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.875 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:21.875 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.122138+0000", 2024-04-20T17:38:21.875 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8059999999999997e-06 2024-04-20T17:38:21.875 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.875 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.875 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:21.875 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.122144+0000", 2024-04-20T17:38:21.875 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2599999999999996e-06 2024-04-20T17:38:21.875 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.875 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.875 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:21.875 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.126679+0000", 2024-04-20T17:38:21.876 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045354519999999997 2024-04-20T17:38:21.876 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.876 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.876 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:21.876 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.126693+0000", 2024-04-20T17:38:21.876 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3771000000000001e-05 2024-04-20T17:38:21.876 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.876 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.876 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:21.876 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.204552+0000", 2024-04-20T17:38:21.876 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.077859178000000001 2024-04-20T17:38:21.876 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.876 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.877 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:21.877 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.204602+0000", 2024-04-20T17:38:21.877 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0158999999999999e-05 2024-04-20T17:38:21.877 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.877 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:21.877 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.878 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.878 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.878 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26882.0:155969 6.a e646/619 6:54d746ad:::benchmark_data_smithi171_147972_object155968:head v 646'9853, mlcod=646'9848)", 2024-04-20T17:38:21.878 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:04.127137+0000", 2024-04-20T17:38:21.878 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 197.74353819000001, 2024-04-20T17:38:21.878 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.077496545, 2024-04-20T17:38:21.878 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:21.878 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:21.878 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:21.878 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:21.879 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.879 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:21.879 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.127137+0000", 2024-04-20T17:38:21.879 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.879 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.879 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.879 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:21.879 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.127137+0000", 2024-04-20T17:38:21.879 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.879 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.879 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.879 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:21.879 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.127138+0000", 2024-04-20T17:38:21.880 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3039999999999999e-06 2024-04-20T17:38:21.880 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.880 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.880 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:21.880 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.127186+0000", 2024-04-20T17:38:21.880 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7948999999999998e-05 2024-04-20T17:38:21.880 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.880 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.880 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:21.880 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.127194+0000", 2024-04-20T17:38:21.881 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5159999999999994e-06 2024-04-20T17:38:21.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.881 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:21.881 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.127202+0000", 2024-04-20T17:38:21.881 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6539999999999999e-06 2024-04-20T17:38:21.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.881 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:21.881 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.130187+0000", 2024-04-20T17:38:21.881 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0029851510000000001 2024-04-20T17:38:21.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.881 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:21.882 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.130204+0000", 2024-04-20T17:38:21.882 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6387000000000001e-05 2024-04-20T17:38:21.882 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.882 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.882 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:21.882 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.204609+0000", 2024-04-20T17:38:21.882 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.074404998 2024-04-20T17:38:21.882 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.882 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.882 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:21.882 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.204633+0000", 2024-04-20T17:38:21.882 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4586000000000001e-05 2024-04-20T17:38:21.882 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.882 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:21.883 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.883 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.883 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.883 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26882.0:155976 6.c e646/619 6:398886a2:::benchmark_data_smithi171_147972_object155975:head v 646'9653, mlcod=646'9651)", 2024-04-20T17:38:21.883 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:04.130092+0000", 2024-04-20T17:38:21.883 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 197.74058238500001, 2024-04-20T17:38:21.883 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.086271459999999994, 2024-04-20T17:38:21.883 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:21.883 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:21.883 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:21.883 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:21.883 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.883 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:21.883 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.130092+0000", 2024-04-20T17:38:21.884 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.884 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.884 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.884 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:21.884 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.130092+0000", 2024-04-20T17:38:21.884 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.884 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.884 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.884 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:21.884 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.130094+0000", 2024-04-20T17:38:21.884 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1659999999999999e-06 2024-04-20T17:38:21.884 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.884 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.884 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:21.884 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.130172+0000", 2024-04-20T17:38:21.885 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8158000000000006e-05 2024-04-20T17:38:21.885 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.885 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.885 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:21.885 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.130179+0000", 2024-04-20T17:38:21.885 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1110000000000002e-06 2024-04-20T17:38:21.885 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.885 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.885 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:21.885 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.130185+0000", 2024-04-20T17:38:21.885 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6560000000000003e-06 2024-04-20T17:38:21.885 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.885 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.885 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:21.885 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.130236+0000", 2024-04-20T17:38:21.886 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0482000000000002e-05 2024-04-20T17:38:21.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.886 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:21.886 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.130253+0000", 2024-04-20T17:38:21.886 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7566000000000001e-05 2024-04-20T17:38:21.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.886 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:21.886 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.216320+0000", 2024-04-20T17:38:21.886 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.086066411999999995 2024-04-20T17:38:21.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.886 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:21.887 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.216364+0000", 2024-04-20T17:38:21.887 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3909000000000003e-05 2024-04-20T17:38:21.887 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.887 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:21.887 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.887 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.887 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26882.0:155978 6.c e646/619 6:38a15b0e:::benchmark_data_smithi171_147972_object155977:head v 646'9654, mlcod=646'9651)", 2024-04-20T17:38:21.887 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:04.135091+0000", 2024-04-20T17:38:21.887 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 197.735583862, 2024-04-20T17:38:21.887 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.087895742999999998, 2024-04-20T17:38:21.888 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:21.888 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:21.888 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:21.888 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:21.888 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.888 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:21.888 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.135091+0000", 2024-04-20T17:38:21.888 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.888 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.888 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.888 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:21.889 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.135091+0000", 2024-04-20T17:38:21.889 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.889 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.889 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:21.889 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.135099+0000", 2024-04-20T17:38:21.889 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1850000000000004e-06 2024-04-20T17:38:21.889 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.889 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:21.889 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.135137+0000", 2024-04-20T17:38:21.889 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8234e-05 2024-04-20T17:38:21.889 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.889 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:21.890 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.204537+0000", 2024-04-20T17:38:21.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.069399251999999995 2024-04-20T17:38:21.890 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.890 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:21.890 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.204551+0000", 2024-04-20T17:38:21.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4369000000000001e-05 2024-04-20T17:38:21.890 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.890 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:21.890 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.208977+0000", 2024-04-20T17:38:21.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0044259219999999997 2024-04-20T17:38:21.890 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.891 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:21.891 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.208998+0000", 2024-04-20T17:38:21.891 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0922999999999999e-05 2024-04-20T17:38:21.891 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.891 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.891 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:21.891 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.222951+0000", 2024-04-20T17:38:21.891 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.013953589000000001 2024-04-20T17:38:21.891 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.891 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.891 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:21.891 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.222987+0000", 2024-04-20T17:38:21.891 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5268999999999997e-05 2024-04-20T17:38:21.891 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.892 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:21.892 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.892 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.10.0:86647 6.d e673/626 6:bb5f8ab5:::benchmark_data_smithi171_147972_object127716:head v 673'19352, mlcod=673'19341)", 2024-04-20T17:38:21.892 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:36:51.363685+0000", 2024-04-20T17:38:21.892 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 90.506989716000007, 2024-04-20T17:38:21.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.055841646000000002, 2024-04-20T17:38:21.892 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:21.892 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:21.892 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:21.892 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:21.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:21.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.363685+0000", 2024-04-20T17:38:21.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:21.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.363685+0000", 2024-04-20T17:38:21.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:21.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.363687+0000", 2024-04-20T17:38:21.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5260000000000001e-06 2024-04-20T17:38:21.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.894 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:21.894 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.363695+0000", 2024-04-20T17:38:21.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0909999999999993e-06 2024-04-20T17:38:21.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.894 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:21.894 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.363706+0000", 2024-04-20T17:38:21.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1394e-05 2024-04-20T17:38:21.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.894 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:21.894 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.363712+0000", 2024-04-20T17:38:21.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7189999999999998e-06 2024-04-20T17:38:21.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.895 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:21.895 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.368257+0000", 2024-04-20T17:38:21.895 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045456209999999997 2024-04-20T17:38:21.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.895 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:21.895 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.368271+0000", 2024-04-20T17:38:21.895 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3611000000000001e-05 2024-04-20T17:38:21.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.895 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:21.895 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.419488+0000", 2024-04-20T17:38:21.895 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.051216847000000003 2024-04-20T17:38:21.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.896 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:21.896 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.419527+0000", 2024-04-20T17:38:21.896 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8837000000000001e-05 2024-04-20T17:38:21.896 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.896 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:21.896 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.896 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.896 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26882.0:274876 6.d e673/626 6:b390e088:::benchmark_data_smithi171_147972_object113778:head v 673'19442, mlcod=673'19430)", 2024-04-20T17:38:21.896 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:36:51.758743+0000", 2024-04-20T17:38:21.896 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 90.111931987000006, 2024-04-20T17:38:21.896 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.061122711000000003, 2024-04-20T17:38:21.896 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:21.897 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:21.897 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:21.897 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:21.897 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.897 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:21.897 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.758743+0000", 2024-04-20T17:38:21.897 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.897 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.897 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:21.897 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.758743+0000", 2024-04-20T17:38:21.897 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.897 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:21.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.758744+0000", 2024-04-20T17:38:21.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.375e-06 2024-04-20T17:38:21.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:21.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.758752+0000", 2024-04-20T17:38:21.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4399999999999999e-06 2024-04-20T17:38:21.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:21.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.758760+0000", 2024-04-20T17:38:21.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7430000000000008e-06 2024-04-20T17:38:21.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:21.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.758767+0000", 2024-04-20T17:38:21.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5200000000000003e-06 2024-04-20T17:38:21.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:21.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.782838+0000", 2024-04-20T17:38:21.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.024071018 2024-04-20T17:38:21.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:21.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.782861+0000", 2024-04-20T17:38:21.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3387000000000002e-05 2024-04-20T17:38:21.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:21.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.819834+0000", 2024-04-20T17:38:21.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.036972821000000003 2024-04-20T17:38:21.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:21.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.819865+0000", 2024-04-20T17:38:21.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1406999999999999e-05 2024-04-20T17:38:21.901 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.901 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:21.901 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.901 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26882.0:274879 6.d e673/626 6:bca7387c:::benchmark_data_smithi171_147972_object113781:head v 673'19443, mlcod=673'19430)", 2024-04-20T17:38:21.901 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:36:51.762235+0000", 2024-04-20T17:38:21.901 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 90.108439617000002, 2024-04-20T17:38:21.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.057656376000000002, 2024-04-20T17:38:21.901 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:21.901 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:21.901 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:21.901 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:21.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:21.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.762235+0000", 2024-04-20T17:38:21.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:21.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.762235+0000", 2024-04-20T17:38:21.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:21.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.762237+0000", 2024-04-20T17:38:21.903 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.733e-06 2024-04-20T17:38:21.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.903 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.903 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:21.903 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.762244+0000", 2024-04-20T17:38:21.903 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.72e-06 2024-04-20T17:38:21.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.903 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.903 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:21.903 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.762254+0000", 2024-04-20T17:38:21.903 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9809999999999998e-06 2024-04-20T17:38:21.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.903 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.903 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:21.903 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.762261+0000", 2024-04-20T17:38:21.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1629999999999999e-06 2024-04-20T17:38:21.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.904 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:21.904 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.787494+0000", 2024-04-20T17:38:21.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.025233333 2024-04-20T17:38:21.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.904 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:21.904 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.787518+0000", 2024-04-20T17:38:21.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4216e-05 2024-04-20T17:38:21.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:21.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.819872+0000", 2024-04-20T17:38:21.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.032353621999999999 2024-04-20T17:38:21.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:21.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:51.819891+0000", 2024-04-20T17:38:21.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9607999999999999e-05 2024-04-20T17:38:21.905 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.905 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:21.905 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.906 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26882.0:275354 6.d e673/626 6:b80fdcd9:::benchmark_data_smithi171_147972_object114256:head v 673'19541, mlcod=673'19530)", 2024-04-20T17:38:21.906 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:36:52.230576+0000", 2024-04-20T17:38:21.906 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 89.640099007999993, 2024-04-20T17:38:21.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.056458537000000003, 2024-04-20T17:38:21.906 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:21.906 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:21.906 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:21.906 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:21.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:21.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:52.230576+0000", 2024-04-20T17:38:21.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:21.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:52.230576+0000", 2024-04-20T17:38:21.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:21.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:52.230577+0000", 2024-04-20T17:38:21.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2389999999999999e-06 2024-04-20T17:38:21.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:21.908 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:52.230583+0000", 2024-04-20T17:38:21.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1840000000000004e-06 2024-04-20T17:38:21.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.908 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:21.908 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:52.230592+0000", 2024-04-20T17:38:21.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9050000000000003e-06 2024-04-20T17:38:21.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.909 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:21.909 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:52.230598+0000", 2024-04-20T17:38:21.909 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7250000000000002e-06 2024-04-20T17:38:21.909 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.909 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:21.909 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:52.250931+0000", 2024-04-20T17:38:21.909 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.020333105000000001 2024-04-20T17:38:21.909 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.909 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:21.910 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:52.250956+0000", 2024-04-20T17:38:21.910 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4640000000000001e-05 2024-04-20T17:38:21.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.910 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:21.910 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:52.286990+0000", 2024-04-20T17:38:21.910 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.036034394999999997 2024-04-20T17:38:21.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.910 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:21.910 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:52.287034+0000", 2024-04-20T17:38:21.910 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4344000000000001e-05 2024-04-20T17:38:21.910 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.910 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:21.911 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.911 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.10.0:86882 6.d e673/626 6:bc5979fd:::benchmark_data_smithi171_147972_object129911:head v 673'19587, mlcod=673'19572)", 2024-04-20T17:38:21.911 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:36:52.458708+0000", 2024-04-20T17:38:21.911 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 89.411967192999995, 2024-04-20T17:38:21.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.055511967000000002, 2024-04-20T17:38:21.911 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:21.911 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:21.911 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:21.911 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:21.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.911 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:21.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:52.458708+0000", 2024-04-20T17:38:21.912 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.912 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:21.912 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:52.458708+0000", 2024-04-20T17:38:21.912 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.912 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:21.912 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:52.458709+0000", 2024-04-20T17:38:21.912 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1960000000000001e-06 2024-04-20T17:38:21.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.912 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:21.912 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:52.458715+0000", 2024-04-20T17:38:21.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2049999999999999e-06 2024-04-20T17:38:21.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:21.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:52.458723+0000", 2024-04-20T17:38:21.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3399999999999998e-06 2024-04-20T17:38:21.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:21.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:52.458728+0000", 2024-04-20T17:38:21.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1819999999999997e-06 2024-04-20T17:38:21.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:21.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:52.469854+0000", 2024-04-20T17:38:21.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011125526 2024-04-20T17:38:21.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:21.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:52.469883+0000", 2024-04-20T17:38:21.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9142999999999999e-05 2024-04-20T17:38:21.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:21.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:52.514177+0000", 2024-04-20T17:38:21.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.044294145 2024-04-20T17:38:21.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:21.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:52.514220+0000", 2024-04-20T17:38:21.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2230000000000001e-05 2024-04-20T17:38:21.915 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.915 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:21.915 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.915 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.10.0:87015 6.d e673/626 6:bcec807e:::benchmark_data_smithi171_147972_object126585:head v 673'19720, mlcod=673'19699)", 2024-04-20T17:38:21.915 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:36:53.075327+0000", 2024-04-20T17:38:21.915 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 88.795347718000002, 2024-04-20T17:38:21.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.061497548999999999, 2024-04-20T17:38:21.915 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:21.915 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:21.915 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:21.915 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:21.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.916 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:21.916 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:53.075327+0000", 2024-04-20T17:38:21.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.916 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.916 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:21.916 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:53.075327+0000", 2024-04-20T17:38:21.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.916 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.916 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:21.916 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:53.075328+0000", 2024-04-20T17:38:21.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3179999999999999e-06 2024-04-20T17:38:21.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.917 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:21.917 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:53.075337+0000", 2024-04-20T17:38:21.917 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5029999999999997e-06 2024-04-20T17:38:21.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.917 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:21.917 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:53.075345+0000", 2024-04-20T17:38:21.917 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4640000000000006e-06 2024-04-20T17:38:21.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.917 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:21.917 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:53.075353+0000", 2024-04-20T17:38:21.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8089999999999993e-06 2024-04-20T17:38:21.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.918 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.918 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:21.918 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:53.096144+0000", 2024-04-20T17:38:21.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.020791351999999999 2024-04-20T17:38:21.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.918 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.918 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:21.918 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:53.096164+0000", 2024-04-20T17:38:21.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9266000000000002e-05 2024-04-20T17:38:21.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.918 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.918 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:21.919 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:53.136777+0000", 2024-04-20T17:38:21.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.040613759999999999 2024-04-20T17:38:21.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:21.919 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:53.136825+0000", 2024-04-20T17:38:21.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7076999999999997e-05 2024-04-20T17:38:21.919 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.919 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:21.919 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.919 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.10.0:87016 6.d e673/626 6:bced694b:::benchmark_data_smithi171_147972_object160408:head v 673'19721, mlcod=673'19699)", 2024-04-20T17:38:21.919 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:36:53.079006+0000", 2024-04-20T17:38:21.920 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 88.791668838000007, 2024-04-20T17:38:21.920 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.057851933000000001, 2024-04-20T17:38:21.920 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:21.920 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:21.920 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:21.920 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:21.920 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.920 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:21.920 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:53.079006+0000", 2024-04-20T17:38:21.920 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.920 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.920 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.920 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:21.920 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:53.079006+0000", 2024-04-20T17:38:21.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:21.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:53.079008+0000", 2024-04-20T17:38:21.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8390000000000001e-06 2024-04-20T17:38:21.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:21.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:53.079018+0000", 2024-04-20T17:38:21.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0692e-05 2024-04-20T17:38:21.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:21.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:53.079028+0000", 2024-04-20T17:38:21.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9899999999999992e-06 2024-04-20T17:38:21.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.922 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.922 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:21.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:53.079037+0000", 2024-04-20T17:38:21.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4980000000000006e-06 2024-04-20T17:38:21.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.922 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.922 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:21.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:53.100049+0000", 2024-04-20T17:38:21.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.021012348 2024-04-20T17:38:21.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.922 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.922 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:21.923 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:53.100072+0000", 2024-04-20T17:38:21.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2855000000000001e-05 2024-04-20T17:38:21.923 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.923 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.923 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:21.923 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:53.136827+0000", 2024-04-20T17:38:21.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.036755272999999998 2024-04-20T17:38:21.923 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.923 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.923 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:21.923 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:53.136858+0000", 2024-04-20T17:38:21.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0437999999999999e-05 2024-04-20T17:38:21.923 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.923 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:21.924 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.924 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.924 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.10.0:87287 6.d e674/626 6:be17897b:::benchmark_data_smithi171_147972_object116479:head v 674'19992, mlcod=674'19964)", 2024-04-20T17:38:21.924 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:36:54.277798+0000", 2024-04-20T17:38:21.924 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 87.592876767999996, 2024-04-20T17:38:21.924 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.055216393000000003, 2024-04-20T17:38:21.924 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:21.924 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:21.924 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:21.924 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:21.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.924 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:21.924 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:54.277798+0000", 2024-04-20T17:38:21.924 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.925 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.925 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:21.925 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:54.277798+0000", 2024-04-20T17:38:21.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.925 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.925 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:21.925 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:54.277800+0000", 2024-04-20T17:38:21.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9649999999999998e-06 2024-04-20T17:38:21.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.925 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.925 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:21.925 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:54.277806+0000", 2024-04-20T17:38:21.926 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5509999999999996e-06 2024-04-20T17:38:21.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.926 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:21.926 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:54.277816+0000", 2024-04-20T17:38:21.926 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0429999999999996e-06 2024-04-20T17:38:21.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.926 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:21.926 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:54.277822+0000", 2024-04-20T17:38:21.926 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2750000000000001e-06 2024-04-20T17:38:21.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.926 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:21.927 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:54.310437+0000", 2024-04-20T17:38:21.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.032615301999999999 2024-04-20T17:38:21.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.927 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.927 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:21.927 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:54.310451+0000", 2024-04-20T17:38:21.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4278e-05 2024-04-20T17:38:21.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.927 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.927 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:21.927 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:54.332975+0000", 2024-04-20T17:38:21.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.022523237000000002 2024-04-20T17:38:21.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.927 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.928 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:21.928 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:54.333014+0000", 2024-04-20T17:38:21.928 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9742e-05 2024-04-20T17:38:21.928 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.928 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:21.928 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.928 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.928 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.10.0:87443 6.d e674/626 6:bebf26b8:::benchmark_data_smithi171_147972_object122726:head v 674'20148, mlcod=674'20144)", 2024-04-20T17:38:21.928 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:36:55.036708+0000", 2024-04-20T17:38:21.928 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 86.833966603999997, 2024-04-20T17:38:21.928 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.074454747000000002, 2024-04-20T17:38:21.928 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:21.928 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:21.928 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:21.929 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:21.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:21.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.036708+0000", 2024-04-20T17:38:21.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:21.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.036708+0000", 2024-04-20T17:38:21.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:21.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.036709+0000", 2024-04-20T17:38:21.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3170000000000001e-06 2024-04-20T17:38:21.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:21.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.036716+0000", 2024-04-20T17:38:21.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7819999999999996e-06 2024-04-20T17:38:21.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:21.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.036726+0000", 2024-04-20T17:38:21.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7270000000000002e-06 2024-04-20T17:38:21.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:21.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.036732+0000", 2024-04-20T17:38:21.931 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7620000000000001e-06 2024-04-20T17:38:21.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.931 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.931 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:21.931 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.039984+0000", 2024-04-20T17:38:21.931 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003252581 2024-04-20T17:38:21.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.931 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.931 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:21.931 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.040006+0000", 2024-04-20T17:38:21.931 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1860000000000001e-05 2024-04-20T17:38:21.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.931 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.931 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:21.932 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.111117+0000", 2024-04-20T17:38:21.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.071110518999999997 2024-04-20T17:38:21.932 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.932 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.932 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:21.932 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.111163+0000", 2024-04-20T17:38:21.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6199000000000003e-05 2024-04-20T17:38:21.932 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.932 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:21.932 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.932 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.932 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.932 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.10.0:87444 6.d e674/626 6:bebfb575:::benchmark_data_smithi171_147972_object121308:head v 674'20149, mlcod=674'20144)", 2024-04-20T17:38:21.932 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:36:55.041716+0000", 2024-04-20T17:38:21.933 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 86.828958470000003, 2024-04-20T17:38:21.933 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.069479787000000001, 2024-04-20T17:38:21.933 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:21.933 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:21.933 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:21.933 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:21.933 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.933 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:21.933 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.041716+0000", 2024-04-20T17:38:21.933 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.933 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.933 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:21.933 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.041716+0000", 2024-04-20T17:38:21.933 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.934 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:21.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.041718+0000", 2024-04-20T17:38:21.934 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5740000000000001e-06 2024-04-20T17:38:21.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.934 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:21.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.041728+0000", 2024-04-20T17:38:21.934 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0316000000000001e-05 2024-04-20T17:38:21.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.934 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:21.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.041742+0000", 2024-04-20T17:38:21.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3368e-05 2024-04-20T17:38:21.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:21.935 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.041750+0000", 2024-04-20T17:38:21.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8389999999999996e-06 2024-04-20T17:38:21.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:21.935 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.044930+0000", 2024-04-20T17:38:21.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0031797180000000002 2024-04-20T17:38:21.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:21.936 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.044954+0000", 2024-04-20T17:38:21.936 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3811999999999998e-05 2024-04-20T17:38:21.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.936 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.936 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:21.936 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.111171+0000", 2024-04-20T17:38:21.936 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.066216919999999999 2024-04-20T17:38:21.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.936 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.936 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:21.936 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.111196+0000", 2024-04-20T17:38:21.936 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5239999999999999e-05 2024-04-20T17:38:21.936 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.936 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:21.937 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.937 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.937 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.10.0:87445 6.d e674/626 6:bebfe5a2:::benchmark_data_smithi171_147972_object140148:head v 674'20150, mlcod=674'20144)", 2024-04-20T17:38:21.937 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:36:55.043993+0000", 2024-04-20T17:38:21.937 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 86.826681308999994, 2024-04-20T17:38:21.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.067235078000000004, 2024-04-20T17:38:21.937 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:21.937 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:21.937 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:21.937 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:21.937 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.937 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:21.937 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.043993+0000", 2024-04-20T17:38:21.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:21.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.043993+0000", 2024-04-20T17:38:21.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:21.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.043995+0000", 2024-04-20T17:38:21.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.471e-06 2024-04-20T17:38:21.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:21.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.044001+0000", 2024-04-20T17:38:21.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3790000000000003e-06 2024-04-20T17:38:21.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.939 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.939 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:21.939 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.044011+0000", 2024-04-20T17:38:21.939 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0054000000000001e-05 2024-04-20T17:38:21.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.939 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.939 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:21.939 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.044016+0000", 2024-04-20T17:38:21.939 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1499999999999998e-06 2024-04-20T17:38:21.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.939 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.939 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:21.939 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.049818+0000", 2024-04-20T17:38:21.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0058016580000000003 2024-04-20T17:38:21.940 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.940 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.940 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:21.940 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.049840+0000", 2024-04-20T17:38:21.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1882000000000002e-05 2024-04-20T17:38:21.940 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.940 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.940 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:21.940 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.111203+0000", 2024-04-20T17:38:21.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.061363217999999997 2024-04-20T17:38:21.940 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.940 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.940 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:21.940 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.111229+0000", 2024-04-20T17:38:21.941 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5265999999999998e-05 2024-04-20T17:38:21.941 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.941 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:21.941 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.941 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.941 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.941 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.10.0:87446 6.d e674/626 6:bec01868:::benchmark_data_smithi171_147972_object120107:head v 674'20151, mlcod=674'20144)", 2024-04-20T17:38:21.941 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:36:55.048337+0000", 2024-04-20T17:38:21.941 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 86.822337485000006, 2024-04-20T17:38:21.941 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.062915634999999998, 2024-04-20T17:38:21.941 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:21.941 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:21.941 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:21.941 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:21.941 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.942 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:21.942 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.048337+0000", 2024-04-20T17:38:21.942 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.942 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.942 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:21.942 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.048337+0000", 2024-04-20T17:38:21.942 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.942 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.942 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:21.942 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.048339+0000", 2024-04-20T17:38:21.942 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5829999999999999e-06 2024-04-20T17:38:21.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.942 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.943 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:21.943 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.048347+0000", 2024-04-20T17:38:21.943 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2390000000000002e-06 2024-04-20T17:38:21.943 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.943 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.943 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:21.943 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.048358+0000", 2024-04-20T17:38:21.943 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0856000000000001e-05 2024-04-20T17:38:21.943 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.943 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.943 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:21.943 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.048366+0000", 2024-04-20T17:38:21.943 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9889999999999992e-06 2024-04-20T17:38:21.943 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.943 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.944 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:21.944 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.054832+0000", 2024-04-20T17:38:21.944 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0064657710000000004 2024-04-20T17:38:21.944 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.944 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.944 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:21.944 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.054852+0000", 2024-04-20T17:38:21.944 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0284000000000001e-05 2024-04-20T17:38:21.944 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.944 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.944 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:21.944 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.111232+0000", 2024-04-20T17:38:21.944 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.056379891000000001 2024-04-20T17:38:21.944 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.945 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.945 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:21.945 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.111253+0000", 2024-04-20T17:38:21.945 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1022e-05 2024-04-20T17:38:21.945 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.945 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:21.945 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.945 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.945 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.945 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.10.0:87447 6.d e674/626 6:bec16af4:::benchmark_data_smithi171_147972_object128041:head v 674'20152, mlcod=674'20144)", 2024-04-20T17:38:21.945 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:36:55.052008+0000", 2024-04-20T17:38:21.945 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 86.818667040999998, 2024-04-20T17:38:21.945 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.059272708, 2024-04-20T17:38:21.945 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:21.945 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:21.946 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:21.946 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:21.946 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.946 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:21.946 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.052008+0000", 2024-04-20T17:38:21.946 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.946 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.946 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.946 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:21.946 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.052008+0000", 2024-04-20T17:38:21.946 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.946 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.946 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.946 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:21.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.052010+0000", 2024-04-20T17:38:21.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9350000000000001e-06 2024-04-20T17:38:21.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.947 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.947 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:21.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.052019+0000", 2024-04-20T17:38:21.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5589999999999994e-06 2024-04-20T17:38:21.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.947 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.947 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:21.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.052032+0000", 2024-04-20T17:38:21.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2814e-05 2024-04-20T17:38:21.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.947 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.947 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:21.948 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.052041+0000", 2024-04-20T17:38:21.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7569999999999993e-06 2024-04-20T17:38:21.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.948 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:21.948 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.059556+0000", 2024-04-20T17:38:21.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0075156620000000002 2024-04-20T17:38:21.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.948 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:21.948 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.059573+0000", 2024-04-20T17:38:21.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6932999999999999e-05 2024-04-20T17:38:21.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.949 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:21.949 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.111256+0000", 2024-04-20T17:38:21.949 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.051682830999999999 2024-04-20T17:38:21.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.949 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.949 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:21.949 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.111280+0000", 2024-04-20T17:38:21.949 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4216999999999999e-05 2024-04-20T17:38:21.949 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.949 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:21.949 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.949 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.949 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.10.0:87448 6.d e674/626 6:beca3ed5:::benchmark_data_smithi171_147972_object135325:head v 674'20153, mlcod=674'20144)", 2024-04-20T17:38:21.949 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:36:55.055588+0000", 2024-04-20T17:38:21.950 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 86.815087109999993, 2024-04-20T17:38:21.950 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.065355712999999996, 2024-04-20T17:38:21.950 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:21.950 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:21.950 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:21.950 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:21.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.950 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:21.950 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.055588+0000", 2024-04-20T17:38:21.950 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.950 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:21.950 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.055588+0000", 2024-04-20T17:38:21.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:21.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.055589+0000", 2024-04-20T17:38:21.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.638e-06 2024-04-20T17:38:21.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:21.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.055598+0000", 2024-04-20T17:38:21.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1230000000000006e-06 2024-04-20T17:38:21.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:21.952 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.055623+0000", 2024-04-20T17:38:21.952 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4196999999999999e-05 2024-04-20T17:38:21.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.952 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.952 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:21.952 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.055631+0000", 2024-04-20T17:38:21.952 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0569999999999993e-06 2024-04-20T17:38:21.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.952 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.952 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:21.952 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.064022+0000", 2024-04-20T17:38:21.952 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0083917009999999997 2024-04-20T17:38:21.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.952 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.952 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:21.953 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.064042+0000", 2024-04-20T17:38:21.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0074000000000001e-05 2024-04-20T17:38:21.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.953 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:21.953 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.120900+0000", 2024-04-20T17:38:21.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.056857968000000002 2024-04-20T17:38:21.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.953 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:21.953 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.120943+0000", 2024-04-20T17:38:21.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2954999999999998e-05 2024-04-20T17:38:21.953 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.953 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:21.953 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.954 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.10.0:87449 6.d e674/626 6:beca4548:::benchmark_data_smithi171_147972_object139434:head v 674'20154, mlcod=674'20144)", 2024-04-20T17:38:21.954 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:36:55.060193+0000", 2024-04-20T17:38:21.954 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 86.810481393000003, 2024-04-20T17:38:21.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.060777522, 2024-04-20T17:38:21.954 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:21.954 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:21.954 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:21.954 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:21.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:21.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.060193+0000", 2024-04-20T17:38:21.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:21.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.060193+0000", 2024-04-20T17:38:21.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:21.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.060195+0000", 2024-04-20T17:38:21.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3400000000000001e-06 2024-04-20T17:38:21.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:21.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.060202+0000", 2024-04-20T17:38:21.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9099999999999999e-06 2024-04-20T17:38:21.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.956 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.956 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:21.956 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.060211+0000", 2024-04-20T17:38:21.956 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1730000000000001e-06 2024-04-20T17:38:21.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.956 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.956 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:21.956 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.060218+0000", 2024-04-20T17:38:21.956 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8950000000000001e-06 2024-04-20T17:38:21.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.956 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.956 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:21.956 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.068097+0000", 2024-04-20T17:38:21.957 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0078791629999999998 2024-04-20T17:38:21.957 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.957 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:21.957 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.068118+0000", 2024-04-20T17:38:21.957 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1348999999999998e-05 2024-04-20T17:38:21.957 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.957 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:21.957 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.120947+0000", 2024-04-20T17:38:21.957 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.052829154000000003 2024-04-20T17:38:21.957 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.957 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:21.958 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.120971+0000", 2024-04-20T17:38:21.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3538000000000001e-05 2024-04-20T17:38:21.958 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.958 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:21.958 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.958 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.10.0:87450 6.d e674/626 6:bece47c3:::benchmark_data_smithi171_147972_object135910:head v 674'20155, mlcod=674'20144)", 2024-04-20T17:38:21.958 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:36:55.064082+0000", 2024-04-20T17:38:21.958 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 86.806592948000002, 2024-04-20T17:38:21.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.056914364000000002, 2024-04-20T17:38:21.958 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:38:21.958 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:38:21.958 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:21.959 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:38:21.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:38:21.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.064082+0000", 2024-04-20T17:38:21.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:38:21.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.064082+0000", 2024-04-20T17:38:21.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:38:21.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:38:21.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.064083+0000", 2024-04-20T17:38:21.960 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3060000000000001e-06 2024-04-20T17:38:21.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.960 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.960 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:38:21.960 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.064090+0000", 2024-04-20T17:38:21.960 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8449999999999997e-06 2024-04-20T17:38:21.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.960 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.960 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:38:21.960 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.064097+0000", 2024-04-20T17:38:21.960 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2690000000000002e-06 2024-04-20T17:38:21.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.960 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.960 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:38:21.960 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.064102+0000", 2024-04-20T17:38:21.961 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7049999999999998e-06 2024-04-20T17:38:21.961 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.961 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.961 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:38:21.961 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.072043+0000", 2024-04-20T17:38:21.961 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0079409760000000006 2024-04-20T17:38:21.961 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.961 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.961 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:38:21.961 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.072060+0000", 2024-04-20T17:38:21.961 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.685e-05 2024-04-20T17:38:21.961 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.961 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.961 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:38:21.962 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.120974+0000", 2024-04-20T17:38:21.962 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.048913904000000001 2024-04-20T17:38:21.962 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:38:21.962 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:38:21.962 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:38:21.962 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:36:55.120996+0000", 2024-04-20T17:38:21.962 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2509000000000001e-05 2024-04-20T17:38:21.962 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.962 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:21.962 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.962 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:38:21.962 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:38:21.962 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:38:22.251 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:38:22.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:22.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:38:22.357 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:38:22.399 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:22.428 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:38:22.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:22.431 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:38:22.533 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:38:22.560 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 14 16 46751 46735 208.622 125.312 0.00740489 0.00479048 2024-04-20T17:38:22.599 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:22.690 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-20T17:38:23.051 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:38:23.470 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:23 smithi005 ceph-mon[39562]: pgmap v1860: 17 pgs: 17 active+clean; 2.5 GiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 244 MiB/s wr, 3.91k op/s 2024-04-20T17:38:23.543 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:38:23.560 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 15 16 48926 48910 203.776 135.938 0.00239223 0.00490498 2024-04-20T17:38:23.566 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:38:23.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:23.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:38:23.674 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:38:23.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:23 smithi106 ceph-mon[44423]: pgmap v1860: 17 pgs: 17 active+clean; 2.5 GiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 244 MiB/s wr, 3.91k op/s 2024-04-20T17:38:23.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:23 smithi171 ceph-mon[44189]: pgmap v1860: 17 pgs: 17 active+clean; 2.5 GiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 244 MiB/s wr, 3.91k op/s 2024-04-20T17:38:23.994 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:24.053 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:38:24.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:24.061 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:38:24.162 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:38:24.215 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:24.560 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 16 16 51239 51223 200.074 144.562 0.0112758 0.0049933 2024-04-20T17:38:25.560 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 17 16 53488 53472 196.573 140.562 0.0100239 0.00508555 2024-04-20T17:38:25.704 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:38:25.704 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:38:25.704 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:38:25.704 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:38:25.704 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:38:25.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:25 smithi005 ceph-mon[39562]: pgmap v1861: 17 pgs: 17 active+clean; 2.7 GiB data, 9.5 GiB used, 885 GiB / 894 GiB avail; 193 MiB/s wr, 3.08k op/s 2024-04-20T17:38:25.712 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:38:25.749 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:38:25.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:25 smithi106 ceph-mon[44423]: pgmap v1861: 17 pgs: 17 active+clean; 2.7 GiB data, 9.5 GiB used, 885 GiB / 894 GiB avail; 193 MiB/s wr, 3.08k op/s 2024-04-20T17:38:25.778 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:38:25.808 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:38:25.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:25 smithi171 ceph-mon[44189]: pgmap v1861: 17 pgs: 17 active+clean; 2.7 GiB data, 9.5 GiB used, 885 GiB / 894 GiB avail; 193 MiB/s wr, 3.08k op/s 2024-04-20T17:38:25.885 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:38:25.925 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:38:25.958 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:38:25.960 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:38:25.960 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:38:25.960 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:38:25.960 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:38:25.960 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:38:25.960 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:38:25.961 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:38:25.961 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:38:25.961 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:38:25.961 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:38:25.961 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:38:25.961 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:38:26.560 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 18 16 55606 55590 193.006 132.375 0.00661297 0.00517951 2024-04-20T17:38:26.674 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:38:26.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:26.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:38:26.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:26 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1450006748' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:38:26.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:26 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:26 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:26 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:26 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:26 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:26 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:26 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:26 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:26 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:26 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.769 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:26 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1450006748' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:38:26.769 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:26 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.769 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:26 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.769 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:26 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.769 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:26 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.769 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:26 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.769 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:26 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.769 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:26 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.769 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:26 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.770 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:26 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.770 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:26 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.779 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:38:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:26 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1450006748' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:38:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:26 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:26 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:26 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:26 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:26 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:26 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:26 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:26 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:26 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:26 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:26.840 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:26.877 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:38:26.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:26.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:38:26.988 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:38:27.037 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:27.560 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 19 16 57796 57780 190.051 136.875 0.00789993 0.00525972 2024-04-20T17:38:27.744 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:27.744 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:27.744 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:27 smithi005 ceph-mon[39562]: pgmap v1862: 17 pgs: 17 active+clean; 3.1 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 169 MiB/s wr, 2.70k op/s 2024-04-20T17:38:27.758 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:38:27.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:27.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:38:27.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:27.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:27.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:27 smithi106 ceph-mon[44423]: pgmap v1862: 17 pgs: 17 active+clean; 3.1 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 169 MiB/s wr, 2.70k op/s 2024-04-20T17:38:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:27 smithi171 ceph-mon[44189]: pgmap v1862: 17 pgs: 17 active+clean; 3.1 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 169 MiB/s wr, 2.70k op/s 2024-04-20T17:38:27.861 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:38:27.900 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:28.055 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:38:28.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:28.064 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:38:28.165 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:38:28.215 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:28.367 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:38:28.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:28.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:38:28.471 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:38:28.531 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:28.560 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:38:28.561165 min lat: 0.00100541 max lat: 0.208488 avg lat: 0.005345 2024-04-20T17:38:28.560 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:38:28.560 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 20 16 59860 59844 186.998 129 0.00508809 0.005345 2024-04-20T17:38:28.628 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:38:28.628 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":703,"fsid":"13c6e76c-ff37-11ee-bc93-c7b262605968","created":"2024-04-20T16:58:25.272557+0000","modified":"2024-04-20T17:38:27.481031+0000","last_up_change":"2024-04-20T17:35:48.811164+0000","last_in_change":"2024-04-20T17:33:42.446560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":32,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T17:01:35.909493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.109999656677246,"score_stable":11.109999656677246,"optimal_score":0.18000000715255737,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.83594763278961182,"average_primary_affinity":0.9100000262260437,"average_primary_affinity_weighted":0.8399999737739563}},{"pool":7,"pool_name":"unique_pool_5","create_time":"2024-04-20T17:37:55.619580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"699","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.440000057220459,"score_stable":3.440000057220459,"optimal_score":0.9100000262260437,"raw_score_acting":3.130000114440918,"raw_score_stable":3.130000114440918,"primary_affinity_weighted":0.83594763278961182,"average_primary_affinity":0.9100000262260437,"average_primary_affinity_weighted":0.8399999737739563}}],"osds":[{"osd":0,"uuid":"ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":694,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6801","nonce":3901484705}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6802","nonce":3901484705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6804","nonce":3901484705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6803","nonce":3901484705}]},"public_addr":"172.21.15.5:6801/3901484705","cluster_addr":"172.21.15.5:6802/3901484705","heartbeat_back_addr":"172.21.15.5:6804/3901484705","heartbeat_front_addr":"172.21.15.5:6803/3901484705","state":["exists","up"]},{"osd":1,"uuid":"aed0c8eb-0e81-4e85-ab8f-a60420060656","up":1,"in":0,"weight":0,"primary_affinity":0.0313720703125,"last_clean_begin":13,"last_clean_end":613,"up_from":659,"up_thru":539,"down_at":614,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6805","nonce":2415652792}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6806","nonce":2415652792}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6808","nonce":2415652792}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6807","nonce":2415652792}]},"public_addr":"172.21.15.5:6805/2415652792","cluster_addr":"172.21.15.5:6806/2415652792","heartbeat_back_addr":"172.21.15.5:6808/2415652792","heartbeat_front_addr":"172.21.15.5:6807/2415652792","state":["exists","up"]},{"osd":2,"uuid":"f733a6a2-19c2-4c7a-af9c-04e282062bb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":20,"last_clean_end":169,"up_from":202,"up_thru":694,"down_at":170,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6809","nonce":2120275501}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6810","nonce":2120275501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6812","nonce":2120275501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6811","nonce":2120275501}]},"public_addr":"172.21.15.5:6809/2120275501","cluster_addr":"172.21.15.5:6810/2120275501","heartbeat_back_addr":"172.21.15.5:6812/2120275501","heartbeat_front_addr":"172.21.15.5:6811/2120275501","state":["exists","up"]},{"osd":3,"uuid":"8c92f1df-e298-4606-88ae-b8a8e79628a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":538,"up_from":561,"up_thru":694,"down_at":539,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6813","nonce":2272824046}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6814","nonce":2272824046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6816","nonce":2272824046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6815","nonce":2272824046}]},"public_addr":"172.21.15.5:6813/2272824046","cluster_addr":"172.21.15.5:6814/2272824046","heartbeat_back_addr":"172.21.15.5:6816/2272824046","heartbeat_front_addr":"172.21.15.5:6815/2272824046","state":["exists","up"]},{"osd":4,"uuid":"46df16f1-9607-4d7b-9be8-9d4236e160d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":694,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6800","nonce":1839524469}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6801","nonce":1839524469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6803","nonce":1839524469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6802","nonce":1839524469}]},"public_addr":"172.21.15.106:6800/1839524469","cluster_addr":"172.21.15.106:6801/1839524469","heartbeat_back_addr":"172.21.15.106:6803/1839524469","heartbeat_front_addr":"172.21.15.106:6802/1839524469","state":["exists","up"]},{"osd":5,"uuid":"6603231d-1e5c-4575-bfaa-2eeca2c12215","up":1,"in":1,"weight":1,"primary_affinity":0,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":619,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6804","nonce":2450315908}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6805","nonce":2450315908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6807","nonce":2450315908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6806","nonce":2450315908}]},"public_addr":"172.21.15.106:6804/2450315908","cluster_addr":"172.21.15.106:6805/2450315908","heartbeat_back_addr":"172.21.15.106:6807/2450315908","heartbeat_front_addr":"172.21.15.106:6806/2450315908","state":["exists","up"]},{"osd":6,"uuid":"783a55a6-f176-42be-8c8b-af8481d08dc9","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":48,"last_clean_end":503,"up_from":535,"up_thru":445,"down_at":504,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6808","nonce":2439382182}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6809","nonce":2439382182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6811","nonce":2439382182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6810","nonce":2439382182}]},"public_addr":"172.21.15.106:6808/2439382182","cluster_addr":"172.21.15.106:6809/2439382182","heartbeat_back_addr":"172.21.15.106:6811/2439382182","heartbeat_front_addr":"172.21.15.106:6810/2439382182","state":["exists","up"]},{"osd":7,"uuid":"b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":56,"last_clean_end":364,"up_from":398,"up_thru":619,"down_at":365,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6812","nonce":1997086583}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6813","nonce":1997086583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6815","nonce":1997086583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6814","nonce":1997086583}]},"public_addr":"172.21.15.106:6812/1997086583","cluster_addr":"172.21.15.106:6813/1997086583","heartbeat_back_addr":"172.21.15.106:6815/1997086583","heartbeat_front_addr":"172.21.15.106:6814/1997086583","state":["exists","up"]},{"osd":8,"uuid":"1cb232ad-1f59-4a0a-a215-e67419ad6476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":694,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6800","nonce":4123058942}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6801","nonce":4123058942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6803","nonce":4123058942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6802","nonce":4123058942}]},"public_addr":"172.21.15.171:6800/4123058942","cluster_addr":"172.21.15.171:6801/4123058942","heartbeat_back_addr":"172.21.15.171:6803/4123058942","heartbeat_front_addr":"172.21.15.171:6802/4123058942","state":["exists","up"]},{"osd":9,"uuid":"b6c17147-b3dc-475b-bb20-f9f712fb99e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":694,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6804","nonce":2877589636}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6805","nonce":2877589636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6807","nonce":2877589636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6806","nonce":2877589636}]},"public_addr":"172.21.15.171:6804/2877589636","cluster_addr":"172.21.15.171:6805/2877589636","heartbeat_back_addr":"172.21.15.171:6807/2877589636","heartbeat_front_addr":"172.21.15.171:6806/2877589636","state":["exists","up"]},{"osd":10,"uuid":"1adcdf55-f002-4a39-91e7-378e5fae849c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":626,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6808","nonce":181490621}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6809","nonce":181490621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6811","nonce":181490621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6810","nonce":181490621}]},"public_addr":"172.21.15.171:6808/181490621","cluster_addr":"172.21.15.171:6809/181490621","heartbeat_back_addr":"172.21.15.171:6811/181490621","heartbeat_front_addr":"172.21.15.171:6810/181490621","state":["exists","up"]},{"osd":11,"uuid":"737c5aec-694a-4968-819d-fccf22661f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":92,"up_thru":694,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6812","nonce":3373585701}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6813","nonce":3373585701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6815","nonce":3373585701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6814","nonce":3373585701}]},"public_addr":"172.21.15.171:6812/3373585701","cluster_addr":"172.21.15.171:6813/3373585701","heartbeat_back_addr":"172.21.15.171:6815/3373585701","heartbeat_front_addr":"172.21.15.171:6814/3373585701","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:37:25.721384+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-20T17:38:27.031854+0000","dead_epoch":613},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:37:51.921369+0000","dead_epoch":169},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:37:38.849994+0000","dead_epoch":538},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:38:13.231138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:37:34.959467+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-20T17:37:36.015900+0000","dead_epoch":503},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:37:33.226142+0000","dead_epoch":364},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:37:33.032569+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:37:31.965230+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:37:45.687335+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:37:50.960042+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.5:0/3881726425":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/338649532":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/1227843664":"2024-04-21T16:58:55.392162+0000","172.21.15.5:6800/1876064631":"2024-04-21T16:58:55.392162+0000","172.21.15.5:0/160689449":"2024-04-21T16:58:55.392162+0000","172.21.15.5:6800/3521715286":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/3841663462":"2024-04-21T16:58:55.392162+0000","172.21.15.5:0/687690919":"2024-04-21T16:59:25.808337+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T17:38:28.667 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:38:28.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:28.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:38:28.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:28 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:38:28.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:28 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:38:28.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:28 smithi106 ceph-mon[44423]: osdmap e703: 12 total, 12 up, 10 in 2024-04-20T17:38:28.776 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:38:28.818 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:28 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:38:28.818 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:28 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:38:28.818 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:28 smithi171 ceph-mon[44189]: osdmap e703: 12 total, 12 up, 10 in 2024-04-20T17:38:28.821 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:28.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:28 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:38:28.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:28 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:38:28.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:28 smithi005 ceph-mon[39562]: osdmap e703: 12 total, 12 up, 10 in 2024-04-20T17:38:29.528 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:38:29.561 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 21 16 61869 61853 184.063 125.562 0.00814605 0.00543112 2024-04-20T17:38:29.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:29 smithi106 ceph-mon[44423]: pgmap v1864: 17 pgs: 17 active+clean; 3.2 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 163 MiB/s wr, 2.61k op/s 2024-04-20T17:38:29.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:29 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/74213113' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:38:29.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:29 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3787043964' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:38:29.783 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:29 smithi171 ceph-mon[44189]: pgmap v1864: 17 pgs: 17 active+clean; 3.2 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 163 MiB/s wr, 2.61k op/s 2024-04-20T17:38:29.783 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:29 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/74213113' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:38:29.783 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:29 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3787043964' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:38:29.853 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:38:29.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:29.857 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:38:29.958 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:38:29.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:29 smithi005 ceph-mon[39562]: pgmap v1864: 17 pgs: 17 active+clean; 3.2 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 163 MiB/s wr, 2.61k op/s 2024-04-20T17:38:29.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:29 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/74213113' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:38:29.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:29 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3787043964' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:38:29.982 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:38:29.990 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:30.011 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:38:30.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:30.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:38:30.115 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:38:30.144 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:30.165 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:38:30.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:30.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:38:30.268 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:38:30.295 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:30.315 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:38:30.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:30.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:38:30.419 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:38:30.474 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:30.511 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:38:30.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:30.515 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:38:30.536 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:38:30.536 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:38:30.564 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 22 16 63775 63759 181.087 119.125 0.0120585 0.00552065 2024-04-20T17:38:30.616 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:38:30.654 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:30.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:30 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:38:30.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:30 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3787043964' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:38:30.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:30 smithi106 ceph-mon[44423]: osdmap e704: 12 total, 12 up, 10 in 2024-04-20T17:38:30.781 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:38:30.783 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:30 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:38:30.783 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:30 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3787043964' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:38:30.783 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:30 smithi171 ceph-mon[44189]: osdmap e704: 12 total, 12 up, 10 in 2024-04-20T17:38:30.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:30.784 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:38:30.885 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:38:30.927 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:30 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:38:30.927 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:30 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3787043964' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:38:30.927 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:30 smithi005 ceph-mon[39562]: osdmap e704: 12 total, 12 up, 10 in 2024-04-20T17:38:30.928 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:31.135 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:38:31.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:31.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:38:31.169 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-20T17:38:31.241 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:38:31.267 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-20T17:38:31.294 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:31.516 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:31 smithi171 ceph-mon[44189]: pgmap v1866: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 148 MiB/s wr, 2.37k op/s 2024-04-20T17:38:31.564 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 23 16 65791 65775 178.692 126 0.00717652 0.00559431 2024-04-20T17:38:31.662 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:38:31.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:31 smithi005 ceph-mon[39562]: pgmap v1866: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 148 MiB/s wr, 2.37k op/s 2024-04-20T17:38:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:31 smithi106 ceph-mon[44423]: pgmap v1866: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 148 MiB/s wr, 2.37k op/s 2024-04-20T17:38:32.155 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:38:32.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:32.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:38:32.197 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:38:32.268 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:38:32.307 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:32.463 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:38:32.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:32.467 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:38:32.564 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 24 15 67735 67720 176.312 121.562 0.0044199 0.00566983 2024-04-20T17:38:32.568 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:38:32.569 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:32 smithi171 ceph-mon[44189]: pgmap v1867: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 138 MiB/s wr, 2.20k op/s 2024-04-20T17:38:32.600 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:32.763 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:32 smithi005 ceph-mon[39562]: pgmap v1867: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 138 MiB/s wr, 2.20k op/s 2024-04-20T17:38:32.959 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:38:32.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:32.964 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:38:33.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:32 smithi106 ceph-mon[44423]: pgmap v1867: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 138 MiB/s wr, 2.20k op/s 2024-04-20T17:38:33.064 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:38:33.112 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:33.146 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:38:33.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:33.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:38:33.255 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:38:33.307 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:33.565 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 25 16 69710 69694 174.193 123.375 0.0095571 0.00573867 2024-04-20T17:38:33.569 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:38:33.572 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:33 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:38:33.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:33.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:38:33.644 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:38:33.644 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:38:33.677 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:38:33.721 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:33.915 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:33 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:38:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:33 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:38:34.172 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:38:34.308 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:38:34.357 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:38:34.395 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:38:34.451 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:38:34.502 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:38:34.541 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:38:34.565 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 26 16 71488 71472 171.767 111.125 0.0105283 0.00581974 2024-04-20T17:38:34.580 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:38:34.584 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:34 smithi005 ceph-mon[39562]: osdmap e705: 12 total, 12 up, 10 in 2024-04-20T17:38:34.585 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:34 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/24333812' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:38:34.585 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:34.585 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:34 smithi005 ceph-mon[39562]: pgmap v1869: 17 pgs: 17 active+clean; 4.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 128 MiB/s wr, 2.04k op/s 2024-04-20T17:38:34.618 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:38:34.653 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:38:34.696 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:38:34.698 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:38:34.743 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:38:34.773 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:38:34.790 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:38:34.790 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:38:34.791 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:38:34.791 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:38:34.791 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:38:34.791 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:38:34.791 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:38:34.791 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:38:34.791 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:38:34.791 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:38:34.791 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:38:34.791 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:38:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:34 smithi106 ceph-mon[44423]: osdmap e705: 12 total, 12 up, 10 in 2024-04-20T17:38:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:34 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/24333812' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:38:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:34 smithi106 ceph-mon[44423]: pgmap v1869: 17 pgs: 17 active+clean; 4.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 128 MiB/s wr, 2.04k op/s 2024-04-20T17:38:35.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:34 smithi171 ceph-mon[44189]: osdmap e705: 12 total, 12 up, 10 in 2024-04-20T17:38:35.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:34 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/24333812' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:38:35.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:35.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:34 smithi171 ceph-mon[44189]: pgmap v1869: 17 pgs: 17 active+clean; 4.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 128 MiB/s wr, 2.04k op/s 2024-04-20T17:38:35.391 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:38:35.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:35.394 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:38:35.495 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:38:35.539 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:35.565 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 27 16 73223 73207 169.422 108.438 0.00563142 0.00590065 2024-04-20T17:38:35.692 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:35.693 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:35.693 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:35.693 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:35.693 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:35.693 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:35.693 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:35.693 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:35.693 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:35.693 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:35.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:35.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:35.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:35.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:35.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:35.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:35.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:35.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:35.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:35.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:36.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:36.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:36.097 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:38:36.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:36.100 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:38:36.105 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-20T17:38:36.201 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:38:36.239 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:36.258 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:38:36.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:36.261 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:38:36.362 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:38:36.417 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:36.510 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:38:36.565 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 28 15 74807 74792 166.909 99.0625 0.0172756 0.00598954 2024-04-20T17:38:36.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:36.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:36 smithi005 ceph-mon[39562]: pgmap v1870: 17 pgs: 17 active+clean; 4.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 148 MiB/s wr, 2.37k op/s 2024-04-20T17:38:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:36 smithi106 ceph-mon[44423]: pgmap v1870: 17 pgs: 17 active+clean; 4.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 148 MiB/s wr, 2.37k op/s 2024-04-20T17:38:37.065 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:37.065 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:36 smithi171 ceph-mon[44189]: pgmap v1870: 17 pgs: 17 active+clean; 4.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 148 MiB/s wr, 2.37k op/s 2024-04-20T17:38:37.565 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 29 16 76669 76653 165.164 116.312 0.011298 0.00605301 2024-04-20T17:38:38.215 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:38:38.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:38.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:38:38.325 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:38:38.354 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:38.375 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:38:38.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:38.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:38:38.479 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:38:38.543 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:38.566 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 30 16 78584 78568 163.642 119.688 0.0194891 0.00610963 2024-04-20T17:38:38.658 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:38:38.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:38.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:38:38.779 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:38:38.830 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:38 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:38:38.830 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:38 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:38:38.830 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:38 smithi005 ceph-mon[39562]: osdmap e706: 12 total, 12 up, 10 in 2024-04-20T17:38:38.830 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:38 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:38:38.831 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:38 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:38:38.831 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:38 smithi005 ceph-mon[39562]: pgmap v1872: 17 pgs: 17 active+clean; 4.4 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-04-20T17:38:38.838 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:39.010 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:38 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:38:39.010 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:38 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:38:39.011 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:38 smithi171 ceph-mon[44189]: osdmap e706: 12 total, 12 up, 10 in 2024-04-20T17:38:39.011 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:38 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:38:39.011 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:38 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:38:39.011 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:38 smithi171 ceph-mon[44189]: pgmap v1872: 17 pgs: 17 active+clean; 4.4 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-04-20T17:38:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:38 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:38:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:38 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:38:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:38 smithi106 ceph-mon[44423]: osdmap e706: 12 total, 12 up, 10 in 2024-04-20T17:38:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:38 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:38:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:38 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:38:39.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:38 smithi106 ceph-mon[44423]: pgmap v1872: 17 pgs: 17 active+clean; 4.4 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-04-20T17:38:39.106 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:38:39.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:39.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:38:39.212 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:38:39.279 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:39.317 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:38:39.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:39.321 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:38:39.391 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:38:39.391 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:38:39.391 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:38:39.391 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:38:39.391 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.391 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233747 6.1 6:8163dd93:::benchmark_data_smithi171_147972_object72649:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:38:39.391 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.174286+0000", 2024-04-20T17:38:39.391 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 148.21371624099999, 2024-04-20T17:38:39.391 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.206612618, 2024-04-20T17:38:39.391 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:38:39.392 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:38:39.392 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:39.392 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:38:39.392 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:38:39.392 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:39.392 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233747 2024-04-20T17:38:39.392 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.392 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:38:39.392 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.392 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:38:39.392 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.174286+0000", 2024-04-20T17:38:39.392 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.392 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.392 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.393 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:38:39.393 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.174286+0000", 2024-04-20T17:38:39.393 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.393 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.393 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.393 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:38:39.393 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.174287+0000", 2024-04-20T17:38:39.393 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.33e-06 2024-04-20T17:38:39.393 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.393 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.393 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:38:39.393 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.174291+0000", 2024-04-20T17:38:39.393 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8480000000000003e-06 2024-04-20T17:38:39.393 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.393 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.394 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:38:39.394 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.174300+0000", 2024-04-20T17:38:39.394 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.217e-06 2024-04-20T17:38:39.394 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.394 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.394 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:38:39.394 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.174306+0000", 2024-04-20T17:38:39.394 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.682e-06 2024-04-20T17:38:39.394 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.394 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.394 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:38:39.394 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.174348+0000", 2024-04-20T17:38:39.394 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2347e-05 2024-04-20T17:38:39.394 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.394 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.395 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:38:39.395 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.174461+0000", 2024-04-20T17:38:39.395 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000112728 2024-04-20T17:38:39.395 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.395 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.395 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:38:39.395 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.174571+0000", 2024-04-20T17:38:39.395 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010971699999999999 2024-04-20T17:38:39.395 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.395 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.395 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:38:39.395 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.178595+0000", 2024-04-20T17:38:39.395 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0040239050000000004 2024-04-20T17:38:39.396 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.396 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.396 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:39.396 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.380796+0000", 2024-04-20T17:38:39.396 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20220144600000001 2024-04-20T17:38:39.396 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.396 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.396 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:38:39.396 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.380851+0000", 2024-04-20T17:38:39.396 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4333000000000003e-05 2024-04-20T17:38:39.396 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.396 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.396 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:38:39.396 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.380899+0000", 2024-04-20T17:38:39.397 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8065000000000001e-05 2024-04-20T17:38:39.397 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.397 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:38:39.397 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.397 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.397 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.397 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233760 6.1 6:87254b5c:::benchmark_data_smithi171_147972_object72662:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:38:39.397 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.184502+0000", 2024-04-20T17:38:39.397 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 148.20350053199999, 2024-04-20T17:38:39.397 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.210398054, 2024-04-20T17:38:39.397 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:38:39.397 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:38:39.397 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:39.398 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:38:39.398 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:38:39.398 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:39.398 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233760 2024-04-20T17:38:39.398 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.398 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:38:39.398 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.398 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:38:39.398 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.184502+0000", 2024-04-20T17:38:39.398 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.398 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.398 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.398 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:38:39.399 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.184502+0000", 2024-04-20T17:38:39.399 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.399 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.399 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.399 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:38:39.399 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.184504+0000", 2024-04-20T17:38:39.399 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2869999999999999e-06 2024-04-20T17:38:39.399 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.399 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.399 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:38:39.399 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.184511+0000", 2024-04-20T17:38:39.399 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.55e-06 2024-04-20T17:38:39.399 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.400 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.400 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:38:39.400 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.184522+0000", 2024-04-20T17:38:39.400 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1722000000000001e-05 2024-04-20T17:38:39.400 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.400 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.400 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:38:39.400 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.184529+0000", 2024-04-20T17:38:39.400 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8889999999999996e-06 2024-04-20T17:38:39.400 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.400 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.400 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:38:39.400 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.184586+0000", 2024-04-20T17:38:39.400 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6449000000000001e-05 2024-04-20T17:38:39.400 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.401 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.401 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:38:39.401 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.184719+0000", 2024-04-20T17:38:39.401 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000133149 2024-04-20T17:38:39.401 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.401 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.401 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:38:39.401 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.184849+0000", 2024-04-20T17:38:39.401 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000129927 2024-04-20T17:38:39.401 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.401 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.401 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:38:39.401 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.188857+0000", 2024-04-20T17:38:39.401 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.004008541 2024-04-20T17:38:39.402 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.402 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.402 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:39.402 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.394830+0000", 2024-04-20T17:38:39.402 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20597264900000001 2024-04-20T17:38:39.402 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.402 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.402 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:38:39.402 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.394867+0000", 2024-04-20T17:38:39.402 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6736999999999997e-05 2024-04-20T17:38:39.402 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.402 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.402 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:38:39.403 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.394900+0000", 2024-04-20T17:38:39.403 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3154000000000001e-05 2024-04-20T17:38:39.403 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.403 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:38:39.403 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.403 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.403 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.403 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233773 6.1 6:8680cb34:::benchmark_data_smithi171_147972_object72675:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:38:39.403 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.195965+0000", 2024-04-20T17:38:39.403 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 148.19203680300001, 2024-04-20T17:38:39.403 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.21500868300000001, 2024-04-20T17:38:39.403 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:38:39.403 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:38:39.403 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:39.404 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:38:39.404 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:38:39.404 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:39.404 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233773 2024-04-20T17:38:39.404 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.404 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:38:39.404 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.404 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:38:39.404 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.195965+0000", 2024-04-20T17:38:39.404 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.404 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.404 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.404 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:38:39.404 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.195965+0000", 2024-04-20T17:38:39.404 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.405 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.405 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.405 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:38:39.405 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.195968+0000", 2024-04-20T17:38:39.405 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9359999999999999e-06 2024-04-20T17:38:39.405 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.405 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.405 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:38:39.405 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.195976+0000", 2024-04-20T17:38:39.405 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9459999999999998e-06 2024-04-20T17:38:39.405 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.405 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.405 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:38:39.406 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.195991+0000", 2024-04-20T17:38:39.406 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4728e-05 2024-04-20T17:38:39.406 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.406 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.406 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:38:39.406 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.195999+0000", 2024-04-20T17:38:39.406 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0150000000000003e-06 2024-04-20T17:38:39.406 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.406 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.406 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:38:39.406 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.196043+0000", 2024-04-20T17:38:39.406 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3757000000000001e-05 2024-04-20T17:38:39.407 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.407 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.407 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:38:39.407 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.196159+0000", 2024-04-20T17:38:39.407 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000115686 2024-04-20T17:38:39.407 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.407 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.407 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:38:39.407 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.196299+0000", 2024-04-20T17:38:39.407 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014027000000000001 2024-04-20T17:38:39.407 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.407 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.407 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:38:39.408 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.201108+0000", 2024-04-20T17:38:39.408 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0048087420000000004 2024-04-20T17:38:39.408 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.408 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.408 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:39.408 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.410861+0000", 2024-04-20T17:38:39.408 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.209753467 2024-04-20T17:38:39.408 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.408 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.408 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:38:39.408 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.410922+0000", 2024-04-20T17:38:39.408 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0498000000000003e-05 2024-04-20T17:38:39.408 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.409 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.409 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:38:39.409 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.410974+0000", 2024-04-20T17:38:39.409 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2638e-05 2024-04-20T17:38:39.409 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.409 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:38:39.409 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.409 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.409 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.409 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233783 6.1 6:82c033a0:::benchmark_data_smithi171_147972_object72685:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:38:39.409 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.204583+0000", 2024-04-20T17:38:39.409 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 148.18341901299999, 2024-04-20T17:38:39.409 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20653519100000001, 2024-04-20T17:38:39.409 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:38:39.410 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:38:39.410 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:39.410 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:38:39.410 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:38:39.410 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:39.410 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233783 2024-04-20T17:38:39.410 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.410 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:38:39.410 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.410 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:38:39.410 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.204583+0000", 2024-04-20T17:38:39.410 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.410 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.411 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.411 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:38:39.411 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.204583+0000", 2024-04-20T17:38:39.411 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.411 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.411 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.411 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:38:39.411 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.204586+0000", 2024-04-20T17:38:39.411 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7240000000000001e-06 2024-04-20T17:38:39.411 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.411 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.411 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:38:39.412 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.204592+0000", 2024-04-20T17:38:39.412 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5570000000000003e-06 2024-04-20T17:38:39.412 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.412 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.412 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:38:39.412 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.204603+0000", 2024-04-20T17:38:39.412 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1154e-05 2024-04-20T17:38:39.412 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.412 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.412 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:38:39.412 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.204609+0000", 2024-04-20T17:38:39.412 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7190000000000004e-06 2024-04-20T17:38:39.412 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.413 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.413 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:38:39.413 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.204652+0000", 2024-04-20T17:38:39.413 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2138000000000002e-05 2024-04-20T17:38:39.413 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.413 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.413 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:38:39.413 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.204752+0000", 2024-04-20T17:38:39.413 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000100317 2024-04-20T17:38:39.413 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.413 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.413 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:38:39.413 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.204823+0000", 2024-04-20T17:38:39.414 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1043000000000004e-05 2024-04-20T17:38:39.414 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.414 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.414 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:38:39.414 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.209034+0000", 2024-04-20T17:38:39.414 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0042115750000000004 2024-04-20T17:38:39.414 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.414 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.414 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:39.414 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.411023+0000", 2024-04-20T17:38:39.414 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20198877700000001 2024-04-20T17:38:39.414 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.414 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.414 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:38:39.415 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.411074+0000", 2024-04-20T17:38:39.415 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0321999999999997e-05 2024-04-20T17:38:39.415 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.415 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.415 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:38:39.415 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.411118+0000", 2024-04-20T17:38:39.415 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4864999999999998e-05 2024-04-20T17:38:39.415 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.415 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:38:39.415 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.415 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.415 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.415 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233795 6.1 6:824ee5e7:::benchmark_data_smithi171_147972_object72697:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:38:39.415 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.218816+0000", 2024-04-20T17:38:39.416 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 148.16918652999999, 2024-04-20T17:38:39.416 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19916731200000001, 2024-04-20T17:38:39.416 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:38:39.416 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:38:39.416 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:39.416 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:38:39.416 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:38:39.416 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:39.416 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233795 2024-04-20T17:38:39.416 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.416 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:38:39.416 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.417 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:38:39.417 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.218816+0000", 2024-04-20T17:38:39.417 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.417 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.417 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.417 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:38:39.417 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.218816+0000", 2024-04-20T17:38:39.417 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.417 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.417 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.417 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:38:39.417 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.218821+0000", 2024-04-20T17:38:39.417 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0200000000000002e-06 2024-04-20T17:38:39.417 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.417 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.418 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:38:39.418 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.218836+0000", 2024-04-20T17:38:39.418 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5142e-05 2024-04-20T17:38:39.418 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.418 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.418 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:38:39.418 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.218870+0000", 2024-04-20T17:38:39.418 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4483000000000002e-05 2024-04-20T17:38:39.418 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.418 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.418 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:38:39.418 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.218894+0000", 2024-04-20T17:38:39.418 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.336e-05 2024-04-20T17:38:39.418 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.419 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.419 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:38:39.419 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.218983+0000", 2024-04-20T17:38:39.419 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9497e-05 2024-04-20T17:38:39.419 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.419 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.419 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:38:39.419 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.219086+0000", 2024-04-20T17:38:39.419 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010312800000000001 2024-04-20T17:38:39.419 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.419 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.419 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:38:39.419 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.219191+0000", 2024-04-20T17:38:39.419 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000105009 2024-04-20T17:38:39.419 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.420 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.420 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:38:39.420 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.222869+0000", 2024-04-20T17:38:39.420 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0036778990000000001 2024-04-20T17:38:39.420 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.420 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.420 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:39.420 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.417920+0000", 2024-04-20T17:38:39.420 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.195051113 2024-04-20T17:38:39.420 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.420 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.420 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:38:39.420 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.417953+0000", 2024-04-20T17:38:39.420 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2157999999999999e-05 2024-04-20T17:38:39.420 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.421 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.421 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:38:39.421 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.417983+0000", 2024-04-20T17:38:39.421 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0502999999999999e-05 2024-04-20T17:38:39.421 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.421 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:38:39.421 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.421 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.421 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.421 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233803 6.1 6:8893042e:::benchmark_data_smithi171_147972_object72705:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:38:39.421 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.224771+0000", 2024-04-20T17:38:39.421 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 148.163231419, 2024-04-20T17:38:39.421 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19336389500000001, 2024-04-20T17:38:39.421 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:38:39.422 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:38:39.422 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:39.422 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:38:39.422 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:38:39.422 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:39.422 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233803 2024-04-20T17:38:39.422 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.422 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:38:39.422 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.422 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:38:39.422 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.224771+0000", 2024-04-20T17:38:39.422 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.422 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.422 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.422 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:38:39.423 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.224771+0000", 2024-04-20T17:38:39.423 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.423 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.423 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.423 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:38:39.423 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.224773+0000", 2024-04-20T17:38:39.423 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7999999999999999e-06 2024-04-20T17:38:39.423 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.423 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.423 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:38:39.423 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.224778+0000", 2024-04-20T17:38:39.423 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8279999999999999e-06 2024-04-20T17:38:39.423 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.423 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.423 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:38:39.424 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.224789+0000", 2024-04-20T17:38:39.424 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0720000000000001e-05 2024-04-20T17:38:39.424 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.424 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.424 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:38:39.424 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.224796+0000", 2024-04-20T17:38:39.424 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0809999999999996e-06 2024-04-20T17:38:39.424 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.424 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.424 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:38:39.424 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.224839+0000", 2024-04-20T17:38:39.424 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3154000000000001e-05 2024-04-20T17:38:39.424 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.424 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.424 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:38:39.425 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.224929+0000", 2024-04-20T17:38:39.425 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9203999999999995e-05 2024-04-20T17:38:39.425 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.425 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.425 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:38:39.425 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.225034+0000", 2024-04-20T17:38:39.425 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000105017 2024-04-20T17:38:39.425 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.425 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.425 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:38:39.425 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.228470+0000", 2024-04-20T17:38:39.425 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0034366959999999999 2024-04-20T17:38:39.425 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.425 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.426 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:39.426 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.418044+0000", 2024-04-20T17:38:39.426 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.18957336799999999 2024-04-20T17:38:39.426 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.426 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.426 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:38:39.426 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.418092+0000", 2024-04-20T17:38:39.426 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8359999999999998e-05 2024-04-20T17:38:39.426 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.426 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.426 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:38:39.426 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.418135+0000", 2024-04-20T17:38:39.426 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2667000000000003e-05 2024-04-20T17:38:39.426 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.427 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:38:39.427 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.427 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.427 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.427 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233813 6.1 6:87bd3cce:::benchmark_data_smithi171_147972_object72715:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:38:39.427 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.232901+0000", 2024-04-20T17:38:39.427 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 148.15510162300001, 2024-04-20T17:38:39.427 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.18540720599999999, 2024-04-20T17:38:39.427 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:38:39.427 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:38:39.427 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:39.427 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:38:39.427 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:38:39.427 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:39.427 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233813 2024-04-20T17:38:39.427 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.428 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:38:39.428 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.428 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:38:39.428 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.232901+0000", 2024-04-20T17:38:39.428 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.428 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.428 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.428 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:38:39.428 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.232901+0000", 2024-04-20T17:38:39.428 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.428 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.428 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.428 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:38:39.428 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.232903+0000", 2024-04-20T17:38:39.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1320000000000001e-06 2024-04-20T17:38:39.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.429 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.429 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:38:39.429 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.232910+0000", 2024-04-20T17:38:39.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9269999999999999e-06 2024-04-20T17:38:39.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.429 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.429 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:38:39.429 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.232923+0000", 2024-04-20T17:38:39.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3392e-05 2024-04-20T17:38:39.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.429 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.429 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:38:39.429 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.232931+0000", 2024-04-20T17:38:39.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0499999999999992e-06 2024-04-20T17:38:39.430 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.430 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.430 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:38:39.430 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.232985+0000", 2024-04-20T17:38:39.430 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3690000000000003e-05 2024-04-20T17:38:39.430 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.430 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.430 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:38:39.430 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.233074+0000", 2024-04-20T17:38:39.430 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8876999999999996e-05 2024-04-20T17:38:39.430 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.430 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.430 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:38:39.430 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.233150+0000", 2024-04-20T17:38:39.430 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6335999999999996e-05 2024-04-20T17:38:39.431 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.431 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.431 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:38:39.431 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.236856+0000", 2024-04-20T17:38:39.431 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0037059549999999999 2024-04-20T17:38:39.431 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.431 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.431 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:39.431 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.418212+0000", 2024-04-20T17:38:39.431 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.18135632600000001 2024-04-20T17:38:39.431 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.431 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.431 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:38:39.431 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.418265+0000", 2024-04-20T17:38:39.431 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3086e-05 2024-04-20T17:38:39.432 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.432 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.432 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:38:39.432 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.418308+0000", 2024-04-20T17:38:39.432 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2435000000000003e-05 2024-04-20T17:38:39.432 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.432 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:38:39.432 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.432 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.432 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.432 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233855 6.1 6:8faa32b1:::benchmark_data_smithi171_147972_object72757:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:38:39.432 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.278676+0000", 2024-04-20T17:38:39.432 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 148.10932618000001, 2024-04-20T17:38:39.432 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.13975268499999999, 2024-04-20T17:38:39.432 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:38:39.433 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:38:39.433 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:39.433 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:38:39.433 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:38:39.433 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:39.433 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233855 2024-04-20T17:38:39.433 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.433 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:38:39.433 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.433 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:38:39.433 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.278676+0000", 2024-04-20T17:38:39.433 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.433 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.433 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.433 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:38:39.434 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.278676+0000", 2024-04-20T17:38:39.434 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.434 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.434 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.434 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:38:39.434 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.278679+0000", 2024-04-20T17:38:39.434 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0630000000000002e-06 2024-04-20T17:38:39.434 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.434 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.434 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:38:39.434 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.278688+0000", 2024-04-20T17:38:39.434 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.3780000000000001e-06 2024-04-20T17:38:39.434 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.434 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.435 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:38:39.435 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.278703+0000", 2024-04-20T17:38:39.435 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5302999999999999e-05 2024-04-20T17:38:39.435 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.435 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.435 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:38:39.435 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.278713+0000", 2024-04-20T17:38:39.435 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9040000000000004e-06 2024-04-20T17:38:39.435 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.435 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.435 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:38:39.436 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.278767+0000", 2024-04-20T17:38:39.436 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4245e-05 2024-04-20T17:38:39.436 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.436 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.436 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:38:39.436 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.278869+0000", 2024-04-20T17:38:39.436 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000102377 2024-04-20T17:38:39.436 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.436 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.436 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:38:39.436 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.278951+0000", 2024-04-20T17:38:39.436 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1237999999999997e-05 2024-04-20T17:38:39.437 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.437 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.437 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:38:39.437 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.282731+0000", 2024-04-20T17:38:39.437 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0037807610000000001 2024-04-20T17:38:39.437 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.437 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.437 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:39.437 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.418360+0000", 2024-04-20T17:38:39.437 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.135629006 2024-04-20T17:38:39.437 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.437 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.437 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:38:39.438 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.418406+0000", 2024-04-20T17:38:39.438 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5343999999999998e-05 2024-04-20T17:38:39.438 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.438 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.438 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:38:39.438 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.418429+0000", 2024-04-20T17:38:39.438 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3065999999999999e-05 2024-04-20T17:38:39.438 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.438 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:38:39.438 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.438 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.438 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.438 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233864 6.1 6:866d1e67:::benchmark_data_smithi171_147972_object72766:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:38:39.439 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.286641+0000", 2024-04-20T17:38:39.439 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 148.101361, 2024-04-20T17:38:39.439 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.13675062399999999, 2024-04-20T17:38:39.439 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:38:39.439 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:38:39.439 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:39.439 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:38:39.439 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:38:39.439 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:39.439 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233864 2024-04-20T17:38:39.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.439 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:38:39.439 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.440 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:38:39.440 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.286641+0000", 2024-04-20T17:38:39.440 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.440 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.440 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:38:39.440 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.286641+0000", 2024-04-20T17:38:39.440 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.440 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.440 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:38:39.440 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.286643+0000", 2024-04-20T17:38:39.440 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7969999999999999e-06 2024-04-20T17:38:39.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.440 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.441 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:38:39.441 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.286649+0000", 2024-04-20T17:38:39.441 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1090000000000003e-06 2024-04-20T17:38:39.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.441 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:38:39.441 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.286660+0000", 2024-04-20T17:38:39.441 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0536999999999999e-05 2024-04-20T17:38:39.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.441 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:38:39.441 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.286666+0000", 2024-04-20T17:38:39.441 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5130000000000002e-06 2024-04-20T17:38:39.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.442 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:38:39.442 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.286710+0000", 2024-04-20T17:38:39.442 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3336e-05 2024-04-20T17:38:39.442 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.442 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.442 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:38:39.442 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.286795+0000", 2024-04-20T17:38:39.442 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5648000000000004e-05 2024-04-20T17:38:39.442 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.442 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.442 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:38:39.442 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.286870+0000", 2024-04-20T17:38:39.442 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5222999999999995e-05 2024-04-20T17:38:39.442 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.442 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.443 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:38:39.443 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.290258+0000", 2024-04-20T17:38:39.443 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0033879909999999999 2024-04-20T17:38:39.443 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.443 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.443 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:39.443 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.423320+0000", 2024-04-20T17:38:39.443 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.133061771 2024-04-20T17:38:39.443 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.443 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.443 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:38:39.443 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.423352+0000", 2024-04-20T17:38:39.443 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2239000000000001e-05 2024-04-20T17:38:39.443 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.444 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.444 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:38:39.444 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.423392+0000", 2024-04-20T17:38:39.444 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9459999999999998e-05 2024-04-20T17:38:39.444 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.444 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:38:39.444 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.444 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.444 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.444 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233892 6.1 6:80f20832:::benchmark_data_smithi171_147972_object72794:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:38:39.444 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.327959+0000", 2024-04-20T17:38:39.444 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 148.060043253, 2024-04-20T17:38:39.444 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.095585288000000004, 2024-04-20T17:38:39.444 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:38:39.444 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:38:39.445 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:39.445 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:38:39.445 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:38:39.445 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:38:39.445 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233892 2024-04-20T17:38:39.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.445 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:38:39.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.445 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:38:39.445 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.327959+0000", 2024-04-20T17:38:39.445 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.446 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:38:39.446 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.327959+0000", 2024-04-20T17:38:39.446 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.446 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.446 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.446 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:38:39.446 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.327962+0000", 2024-04-20T17:38:39.446 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9780000000000001e-06 2024-04-20T17:38:39.446 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.446 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.446 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:38:39.446 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.327970+0000", 2024-04-20T17:38:39.446 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2530000000000004e-06 2024-04-20T17:38:39.446 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.447 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.447 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:38:39.447 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.327987+0000", 2024-04-20T17:38:39.447 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6728e-05 2024-04-20T17:38:39.447 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.447 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.447 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:38:39.447 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.327996+0000", 2024-04-20T17:38:39.447 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8009999999999993e-06 2024-04-20T17:38:39.447 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.447 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.447 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:38:39.447 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.328053+0000", 2024-04-20T17:38:39.447 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7158999999999999e-05 2024-04-20T17:38:39.447 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.448 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.448 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:38:39.448 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.328141+0000", 2024-04-20T17:38:39.448 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8145000000000004e-05 2024-04-20T17:38:39.448 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.448 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.448 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:38:39.448 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.328219+0000", 2024-04-20T17:38:39.448 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7718000000000004e-05 2024-04-20T17:38:39.448 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.448 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.448 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:38:39.448 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.331988+0000", 2024-04-20T17:38:39.448 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0037692619999999998 2024-04-20T17:38:39.449 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.449 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.449 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:39.449 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.423457+0000", 2024-04-20T17:38:39.449 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.091468485000000002 2024-04-20T17:38:39.449 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.449 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.449 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:38:39.449 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.423502+0000", 2024-04-20T17:38:39.449 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5269000000000003e-05 2024-04-20T17:38:39.449 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.449 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.449 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:38:39.449 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.423544+0000", 2024-04-20T17:38:39.450 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2490000000000001e-05 2024-04-20T17:38:39.450 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.450 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:38:39.450 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.450 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.450 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.450 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:33450 7.7 7:ec376a9b:::benchmark_data_smithi171_161871_object33449:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e701)", 2024-04-20T17:38:39.450 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:17.289215+0000", 2024-04-20T17:38:39.450 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 22.098787095999999, 2024-04-20T17:38:39.450 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.086241257000000002, 2024-04-20T17:38:39.450 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:38:39.450 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:38:39.450 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:39.450 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:38:39.450 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:38:39.451 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:38:39.451 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 33450 2024-04-20T17:38:39.451 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.451 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:38:39.451 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.451 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:38:39.451 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.289215+0000", 2024-04-20T17:38:39.451 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.451 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.451 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.451 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:38:39.451 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.289215+0000", 2024-04-20T17:38:39.451 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.451 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.452 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.452 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:38:39.452 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.289218+0000", 2024-04-20T17:38:39.452 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.689e-06 2024-04-20T17:38:39.452 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.452 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.452 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:38:39.452 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.289245+0000", 2024-04-20T17:38:39.452 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7450999999999999e-05 2024-04-20T17:38:39.452 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.452 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.452 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:38:39.452 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.289263+0000", 2024-04-20T17:38:39.452 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7394999999999999e-05 2024-04-20T17:38:39.452 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.453 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.453 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:38:39.453 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.289273+0000", 2024-04-20T17:38:39.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0597999999999999e-05 2024-04-20T17:38:39.453 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.453 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.453 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:38:39.453 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.289355+0000", 2024-04-20T17:38:39.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1357e-05 2024-04-20T17:38:39.453 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.453 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.453 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:38:39.453 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.289529+0000", 2024-04-20T17:38:39.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000174816 2024-04-20T17:38:39.453 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.454 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.454 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:38:39.454 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.289667+0000", 2024-04-20T17:38:39.454 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000137677 2024-04-20T17:38:39.454 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.454 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.454 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:38:39.454 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.292033+0000", 2024-04-20T17:38:39.454 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0023653799999999998 2024-04-20T17:38:39.454 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.454 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.454 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:39.454 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.375342+0000", 2024-04-20T17:38:39.454 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.083309026999999994 2024-04-20T17:38:39.454 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.455 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.455 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:38:39.455 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.375401+0000", 2024-04-20T17:38:39.455 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.94e-05 2024-04-20T17:38:39.455 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.455 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.455 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:38:39.455 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.375456+0000", 2024-04-20T17:38:39.455 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5467000000000003e-05 2024-04-20T17:38:39.455 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.455 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:38:39.455 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.455 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.455 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.455 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:33455 7.7 7:eca42211:::benchmark_data_smithi171_161871_object33454:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e701)", 2024-04-20T17:38:39.456 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:17.289858+0000", 2024-04-20T17:38:39.456 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 22.098144034000001, 2024-04-20T17:38:39.456 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.088512918999999995, 2024-04-20T17:38:39.456 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:38:39.456 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:38:39.456 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:39.456 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:38:39.456 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:38:39.456 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:38:39.456 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 33455 2024-04-20T17:38:39.456 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.456 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:38:39.456 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.456 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:38:39.456 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.289858+0000", 2024-04-20T17:38:39.457 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.457 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.457 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:38:39.457 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.289858+0000", 2024-04-20T17:38:39.457 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.457 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.457 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:38:39.457 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.289860+0000", 2024-04-20T17:38:39.457 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.592e-06 2024-04-20T17:38:39.457 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.457 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:38:39.457 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.289884+0000", 2024-04-20T17:38:39.458 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3838000000000001e-05 2024-04-20T17:38:39.458 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.458 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.458 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:38:39.458 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.289896+0000", 2024-04-20T17:38:39.458 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2465e-05 2024-04-20T17:38:39.458 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.458 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.458 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:38:39.458 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.289904+0000", 2024-04-20T17:38:39.458 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2910000000000007e-06 2024-04-20T17:38:39.458 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.458 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.458 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:38:39.458 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.290828+0000", 2024-04-20T17:38:39.458 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00092347800000000004 2024-04-20T17:38:39.459 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.459 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.459 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:38:39.459 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.290951+0000", 2024-04-20T17:38:39.459 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012354799999999999 2024-04-20T17:38:39.459 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.459 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.459 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:38:39.459 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.291079+0000", 2024-04-20T17:38:39.459 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012786699999999999 2024-04-20T17:38:39.459 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.459 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.459 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:38:39.459 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.292930+0000", 2024-04-20T17:38:39.459 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001850747 2024-04-20T17:38:39.460 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.460 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.460 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:39.460 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.378293+0000", 2024-04-20T17:38:39.460 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.085363056000000007 2024-04-20T17:38:39.460 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.460 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.460 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:38:39.460 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.378343+0000", 2024-04-20T17:38:39.460 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9802999999999997e-05 2024-04-20T17:38:39.460 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.460 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.460 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:38:39.460 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.378371+0000", 2024-04-20T17:38:39.460 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8234000000000001e-05 2024-04-20T17:38:39.461 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.461 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:38:39.461 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.461 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.461 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.461 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:33456 7.7 7:ea6467cb:::benchmark_data_smithi171_161871_object33455:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e701)", 2024-04-20T17:38:39.461 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:17.289921+0000", 2024-04-20T17:38:39.461 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 22.098081434000001, 2024-04-20T17:38:39.461 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.088591034999999999, 2024-04-20T17:38:39.461 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:38:39.461 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:38:39.461 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:39.461 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:38:39.461 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:38:39.462 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:38:39.462 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 33456 2024-04-20T17:38:39.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.462 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:38:39.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.462 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:38:39.462 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.289921+0000", 2024-04-20T17:38:39.462 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.462 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:38:39.462 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.289921+0000", 2024-04-20T17:38:39.462 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.463 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:38:39.463 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.289922+0000", 2024-04-20T17:38:39.463 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.85e-07 2024-04-20T17:38:39.463 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.463 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.463 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:38:39.463 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.289943+0000", 2024-04-20T17:38:39.463 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1005999999999998e-05 2024-04-20T17:38:39.463 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.463 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.463 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:38:39.463 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.289947+0000", 2024-04-20T17:38:39.463 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4970000000000003e-06 2024-04-20T17:38:39.463 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.463 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.464 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:38:39.464 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.289952+0000", 2024-04-20T17:38:39.464 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3499999999999999e-06 2024-04-20T17:38:39.464 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.464 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.464 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:38:39.464 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.292057+0000", 2024-04-20T17:38:39.464 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0021055449999999999 2024-04-20T17:38:39.464 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.464 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.464 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:38:39.464 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.292184+0000", 2024-04-20T17:38:39.464 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000126933 2024-04-20T17:38:39.464 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.464 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.465 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:38:39.465 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.292307+0000", 2024-04-20T17:38:39.465 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000123328 2024-04-20T17:38:39.465 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.465 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.465 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:38:39.465 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.293086+0000", 2024-04-20T17:38:39.465 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00077897700000000003 2024-04-20T17:38:39.465 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.465 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.465 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:39.465 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.378422+0000", 2024-04-20T17:38:39.465 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.085335224000000001 2024-04-20T17:38:39.465 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.465 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.465 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:38:39.466 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.378471+0000", 2024-04-20T17:38:39.466 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9796999999999998e-05 2024-04-20T17:38:39.466 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.466 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.466 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:38:39.466 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.378512+0000", 2024-04-20T17:38:39.466 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0493e-05 2024-04-20T17:38:39.466 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.466 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:38:39.466 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.466 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.466 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.466 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:34697 7.7 7:e961be3e:::benchmark_data_smithi171_161871_object34696:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e702)", 2024-04-20T17:38:39.466 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:17.789192+0000", 2024-04-20T17:38:39.467 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 21.598810044, 2024-04-20T17:38:39.467 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.079620435000000003, 2024-04-20T17:38:39.467 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:38:39.467 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:38:39.467 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:39.467 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:38:39.467 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:38:39.467 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:38:39.467 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 34697 2024-04-20T17:38:39.467 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.467 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:38:39.467 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.467 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:38:39.467 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.789192+0000", 2024-04-20T17:38:39.467 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.468 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.468 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.468 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:38:39.468 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.789192+0000", 2024-04-20T17:38:39.468 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.468 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.468 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.468 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:38:39.468 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.789195+0000", 2024-04-20T17:38:39.468 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4159999999999998e-06 2024-04-20T17:38:39.468 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.468 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.468 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:38:39.468 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.789224+0000", 2024-04-20T17:38:39.468 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9692999999999998e-05 2024-04-20T17:38:39.469 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.469 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.469 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:38:39.469 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.789236+0000", 2024-04-20T17:38:39.469 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1494999999999999e-05 2024-04-20T17:38:39.469 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.469 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.469 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:38:39.469 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.789243+0000", 2024-04-20T17:38:39.469 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4800000000000004e-06 2024-04-20T17:38:39.469 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.469 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.469 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:38:39.469 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.789294+0000", 2024-04-20T17:38:39.469 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0481e-05 2024-04-20T17:38:39.469 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.470 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.470 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:38:39.470 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.789421+0000", 2024-04-20T17:38:39.470 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000127553 2024-04-20T17:38:39.470 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.470 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.470 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:38:39.470 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.789507+0000", 2024-04-20T17:38:39.470 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6086000000000002e-05 2024-04-20T17:38:39.470 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.470 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.470 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:38:39.470 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.790667+0000", 2024-04-20T17:38:39.470 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0011596270000000001 2024-04-20T17:38:39.470 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.471 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.471 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:39.471 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.868740+0000", 2024-04-20T17:38:39.471 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.078072638 2024-04-20T17:38:39.471 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.471 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.471 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:38:39.471 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.868775+0000", 2024-04-20T17:38:39.471 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5330999999999998e-05 2024-04-20T17:38:39.471 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.471 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.471 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:38:39.471 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:17.868813+0000", 2024-04-20T17:38:39.471 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7635000000000001e-05 2024-04-20T17:38:39.471 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.472 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:38:39.472 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.472 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.472 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.472 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:44705 7.7 7:eb5ea621:::benchmark_data_smithi171_161871_object44704:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e702)", 2024-04-20T17:38:39.472 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:21.496348+0000", 2024-04-20T17:38:39.472 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 17.891654531, 2024-04-20T17:38:39.472 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.092532040999999995, 2024-04-20T17:38:39.472 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:38:39.472 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:38:39.472 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:39.472 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:38:39.472 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:38:39.472 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:38:39.472 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 44705 2024-04-20T17:38:39.473 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.473 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:38:39.473 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.473 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:38:39.473 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.496348+0000", 2024-04-20T17:38:39.473 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.473 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.473 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.473 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:38:39.473 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.496348+0000", 2024-04-20T17:38:39.473 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.473 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.473 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.473 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:38:39.474 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.496350+0000", 2024-04-20T17:38:39.474 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8199999999999999e-06 2024-04-20T17:38:39.474 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.474 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.474 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:38:39.474 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.496373+0000", 2024-04-20T17:38:39.474 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3728999999999999e-05 2024-04-20T17:38:39.474 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.474 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.474 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:38:39.474 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.496383+0000", 2024-04-20T17:38:39.474 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.8830000000000001e-06 2024-04-20T17:38:39.474 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.475 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.475 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:38:39.475 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.496390+0000", 2024-04-20T17:38:39.475 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8889999999999996e-06 2024-04-20T17:38:39.475 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.475 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.475 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:38:39.475 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.496429+0000", 2024-04-20T17:38:39.475 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8495999999999999e-05 2024-04-20T17:38:39.475 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.475 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.475 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:38:39.475 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.496546+0000", 2024-04-20T17:38:39.475 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000117718 2024-04-20T17:38:39.475 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.476 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.476 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:38:39.476 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.496670+0000", 2024-04-20T17:38:39.476 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012407400000000001 2024-04-20T17:38:39.476 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.476 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.476 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:38:39.476 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.498155+0000", 2024-04-20T17:38:39.476 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0014849729999999999 2024-04-20T17:38:39.476 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.476 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.476 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:39.476 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.588767+0000", 2024-04-20T17:38:39.476 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.090611467000000001 2024-04-20T17:38:39.477 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.477 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.477 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:38:39.477 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.588824+0000", 2024-04-20T17:38:39.477 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6879999999999998e-05 2024-04-20T17:38:39.477 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.477 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.477 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:38:39.477 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.588880+0000", 2024-04-20T17:38:39.477 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6112000000000001e-05 2024-04-20T17:38:39.477 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.477 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:38:39.477 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.477 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.477 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.477 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:44708 7.7 7:e3fbbaaa:::benchmark_data_smithi171_161871_object44707:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e702)", 2024-04-20T17:38:39.478 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:21.498302+0000", 2024-04-20T17:38:39.478 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 17.889700733000002, 2024-04-20T17:38:39.478 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.090786144999999999, 2024-04-20T17:38:39.478 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:38:39.478 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:38:39.478 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:39.478 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:38:39.478 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:38:39.478 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:38:39.478 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 44708 2024-04-20T17:38:39.478 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.478 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:38:39.478 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.478 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:38:39.479 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.498302+0000", 2024-04-20T17:38:39.479 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.479 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.479 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.479 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:38:39.479 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.498302+0000", 2024-04-20T17:38:39.479 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.479 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.479 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.479 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:38:39.479 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.498303+0000", 2024-04-20T17:38:39.479 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8059999999999999e-06 2024-04-20T17:38:39.479 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.479 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.480 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:38:39.480 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.498328+0000", 2024-04-20T17:38:39.480 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5120999999999999e-05 2024-04-20T17:38:39.480 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.480 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.480 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:38:39.480 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.498341+0000", 2024-04-20T17:38:39.480 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2685e-05 2024-04-20T17:38:39.480 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.480 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.480 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:38:39.480 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.498349+0000", 2024-04-20T17:38:39.480 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0229999999999992e-06 2024-04-20T17:38:39.480 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.480 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.481 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:38:39.481 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.498406+0000", 2024-04-20T17:38:39.481 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6743000000000002e-05 2024-04-20T17:38:39.481 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.481 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.481 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:38:39.481 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.498534+0000", 2024-04-20T17:38:39.481 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000127949 2024-04-20T17:38:39.481 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.481 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.481 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:38:39.481 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.498672+0000", 2024-04-20T17:38:39.481 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013822500000000001 2024-04-20T17:38:39.481 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.481 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.482 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:38:39.482 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.499999+0000", 2024-04-20T17:38:39.482 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001326898 2024-04-20T17:38:39.482 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.482 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.482 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:39.482 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.588965+0000", 2024-04-20T17:38:39.482 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.088965603000000004 2024-04-20T17:38:39.482 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.482 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.482 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:38:39.482 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.589039+0000", 2024-04-20T17:38:39.482 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4115000000000004e-05 2024-04-20T17:38:39.482 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.482 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.483 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:38:39.483 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.589088+0000", 2024-04-20T17:38:39.483 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8977000000000002e-05 2024-04-20T17:38:39.483 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.483 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:38:39.483 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.483 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.483 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.483 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:44733 7.7 7:e12c4758:::benchmark_data_smithi171_161871_object44732:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e702)", 2024-04-20T17:38:39.483 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:21.509305+0000", 2024-04-20T17:38:39.483 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 17.878697351, 2024-04-20T17:38:39.483 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.084771441000000003, 2024-04-20T17:38:39.483 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:38:39.483 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:38:39.483 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:39.484 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:38:39.484 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:38:39.484 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:38:39.484 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 44733 2024-04-20T17:38:39.484 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.484 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:38:39.484 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.484 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:38:39.484 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.509305+0000", 2024-04-20T17:38:39.484 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.484 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.484 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.484 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:38:39.484 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.509305+0000", 2024-04-20T17:38:39.484 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.485 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.485 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.485 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:38:39.485 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.509307+0000", 2024-04-20T17:38:39.485 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9939999999999999e-06 2024-04-20T17:38:39.485 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.485 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.485 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:38:39.485 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.509334+0000", 2024-04-20T17:38:39.485 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7463000000000001e-05 2024-04-20T17:38:39.485 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.485 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.485 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:38:39.485 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.509348+0000", 2024-04-20T17:38:39.486 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4109e-05 2024-04-20T17:38:39.486 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.486 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.486 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:38:39.486 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.509356+0000", 2024-04-20T17:38:39.486 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7409999999999992e-06 2024-04-20T17:38:39.486 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.486 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.486 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:38:39.486 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.509414+0000", 2024-04-20T17:38:39.486 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7666e-05 2024-04-20T17:38:39.486 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.486 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.486 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:38:39.486 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.509513+0000", 2024-04-20T17:38:39.487 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9186999999999997e-05 2024-04-20T17:38:39.487 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.487 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.487 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:38:39.487 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.509630+0000", 2024-04-20T17:38:39.487 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000117279 2024-04-20T17:38:39.487 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.487 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.487 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:38:39.487 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.511026+0000", 2024-04-20T17:38:39.487 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001395716 2024-04-20T17:38:39.487 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.487 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.487 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:39.487 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.594012+0000", 2024-04-20T17:38:39.488 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.082986246 2024-04-20T17:38:39.488 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.488 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.488 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:38:39.488 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.594048+0000", 2024-04-20T17:38:39.488 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5852999999999997e-05 2024-04-20T17:38:39.488 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.488 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.488 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:38:39.488 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.594076+0000", 2024-04-20T17:38:39.488 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8187e-05 2024-04-20T17:38:39.488 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.488 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:38:39.488 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.488 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.489 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.489 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:73630 7.7 7:e20d59ab:::benchmark_data_smithi171_161871_object73629:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e705)", 2024-04-20T17:38:39.489 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:35.769376+0000", 2024-04-20T17:38:39.489 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 3.6186266040000001, 2024-04-20T17:38:39.489 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.070118562999999995, 2024-04-20T17:38:39.489 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:38:39.489 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:38:39.489 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:39.489 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:38:39.489 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:38:39.489 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:38:39.489 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 73630 2024-04-20T17:38:39.489 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.489 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:38:39.490 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.490 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:38:39.490 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.769376+0000", 2024-04-20T17:38:39.490 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.490 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.490 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.490 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:38:39.490 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.769376+0000", 2024-04-20T17:38:39.490 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.490 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.490 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.490 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:38:39.490 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.769378+0000", 2024-04-20T17:38:39.490 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8870000000000001e-06 2024-04-20T17:38:39.490 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.491 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.491 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:38:39.491 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.769402+0000", 2024-04-20T17:38:39.491 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4341999999999998e-05 2024-04-20T17:38:39.491 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.491 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.491 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:38:39.491 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.769416+0000", 2024-04-20T17:38:39.491 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4219999999999999e-05 2024-04-20T17:38:39.491 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.491 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.491 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:38:39.491 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.769424+0000", 2024-04-20T17:38:39.491 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9410000000000007e-06 2024-04-20T17:38:39.491 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.491 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.492 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:38:39.492 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.770258+0000", 2024-04-20T17:38:39.492 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00083352899999999995 2024-04-20T17:38:39.492 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.492 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.492 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:38:39.492 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.770348+0000", 2024-04-20T17:38:39.492 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0783000000000001e-05 2024-04-20T17:38:39.492 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.492 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.492 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:38:39.492 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.770429+0000", 2024-04-20T17:38:39.492 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0378999999999997e-05 2024-04-20T17:38:39.492 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.492 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.493 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:38:39.493 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.774268+0000", 2024-04-20T17:38:39.493 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0038396530000000002 2024-04-20T17:38:39.493 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.493 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.493 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:39.493 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.839372+0000", 2024-04-20T17:38:39.493 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.065103243000000005 2024-04-20T17:38:39.493 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.493 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.493 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:38:39.493 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.839446+0000", 2024-04-20T17:38:39.493 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4258000000000006e-05 2024-04-20T17:38:39.493 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.493 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.494 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:38:39.494 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.839494+0000", 2024-04-20T17:38:39.494 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8328000000000002e-05 2024-04-20T17:38:39.494 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.494 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:38:39.494 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.494 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.494 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.494 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:73634 7.7 7:e2a6c0c2:::benchmark_data_smithi171_161871_object73633:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e705)", 2024-04-20T17:38:39.494 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:35.769523+0000", 2024-04-20T17:38:39.494 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 3.6184788879999998, 2024-04-20T17:38:39.494 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.070121006999999999, 2024-04-20T17:38:39.494 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:38:39.494 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:38:39.494 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:39.495 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:38:39.495 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:38:39.495 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:38:39.495 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 73634 2024-04-20T17:38:39.495 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.495 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:38:39.495 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.495 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:38:39.495 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.769523+0000", 2024-04-20T17:38:39.495 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.495 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.495 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.495 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:38:39.495 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.769523+0000", 2024-04-20T17:38:39.495 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.496 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.496 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.496 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:38:39.496 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.769525+0000", 2024-04-20T17:38:39.496 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.403e-06 2024-04-20T17:38:39.496 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.496 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.496 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:38:39.496 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.769545+0000", 2024-04-20T17:38:39.496 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9797999999999999e-05 2024-04-20T17:38:39.496 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.496 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.496 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:38:39.496 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.769558+0000", 2024-04-20T17:38:39.496 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2925000000000001e-05 2024-04-20T17:38:39.496 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.497 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.497 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:38:39.497 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.769566+0000", 2024-04-20T17:38:39.497 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0029999999999994e-06 2024-04-20T17:38:39.497 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.497 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.497 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:38:39.497 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.772298+0000", 2024-04-20T17:38:39.497 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.002732562 2024-04-20T17:38:39.497 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.497 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.497 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:38:39.497 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.772416+0000", 2024-04-20T17:38:39.497 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011772100000000001 2024-04-20T17:38:39.497 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.497 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.498 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:38:39.498 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.772545+0000", 2024-04-20T17:38:39.498 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012919799999999999 2024-04-20T17:38:39.498 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.498 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.498 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:38:39.498 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.774375+0000", 2024-04-20T17:38:39.498 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001830309 2024-04-20T17:38:39.498 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.498 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.498 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:39.498 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.839548+0000", 2024-04-20T17:38:39.498 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.065172437 2024-04-20T17:38:39.498 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.498 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.499 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:38:39.499 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.839605+0000", 2024-04-20T17:38:39.499 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7120000000000002e-05 2024-04-20T17:38:39.499 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.499 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.499 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:38:39.499 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.839644+0000", 2024-04-20T17:38:39.499 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9530999999999998e-05 2024-04-20T17:38:39.499 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.499 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:38:39.499 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.499 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.499 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.499 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:73643 7.7 7:e192d91a:::benchmark_data_smithi171_161871_object73642:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e705)", 2024-04-20T17:38:39.499 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:35.775554+0000", 2024-04-20T17:38:39.500 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 3.6124483760000001, 2024-04-20T17:38:39.500 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.064231960000000005, 2024-04-20T17:38:39.500 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:38:39.500 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:38:39.500 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:38:39.500 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:38:39.500 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:38:39.500 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:38:39.500 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 73643 2024-04-20T17:38:39.500 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.500 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:38:39.500 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.500 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:38:39.500 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.775554+0000", 2024-04-20T17:38:39.500 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.500 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.501 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.501 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:38:39.501 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.775554+0000", 2024-04-20T17:38:39.501 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:39.501 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.501 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.501 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:38:39.501 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.775556+0000", 2024-04-20T17:38:39.501 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7740000000000001e-06 2024-04-20T17:38:39.501 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.501 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.501 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:38:39.501 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.775576+0000", 2024-04-20T17:38:39.501 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0545999999999999e-05 2024-04-20T17:38:39.501 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.501 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.502 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:38:39.502 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.775587+0000", 2024-04-20T17:38:39.502 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0553999999999999e-05 2024-04-20T17:38:39.502 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.502 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.502 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:38:39.502 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.775593+0000", 2024-04-20T17:38:39.502 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1199999999999999e-06 2024-04-20T17:38:39.502 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.502 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.502 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:38:39.502 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.775634+0000", 2024-04-20T17:38:39.502 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0695000000000003e-05 2024-04-20T17:38:39.502 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.502 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.503 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:38:39.503 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.775727+0000", 2024-04-20T17:38:39.503 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3488999999999999e-05 2024-04-20T17:38:39.503 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.503 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.503 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:38:39.503 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.775816+0000", 2024-04-20T17:38:39.503 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8610999999999996e-05 2024-04-20T17:38:39.503 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.503 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.503 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:38:39.503 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.777330+0000", 2024-04-20T17:38:39.503 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001514381 2024-04-20T17:38:39.503 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.503 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.503 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:38:39.504 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.839692+0000", 2024-04-20T17:38:39.504 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.062361530999999998 2024-04-20T17:38:39.504 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.504 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.504 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:38:39.504 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.839745+0000", 2024-04-20T17:38:39.504 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3733999999999997e-05 2024-04-20T17:38:39.504 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:39.504 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:39.504 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:38:39.504 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:35.839786+0000", 2024-04-20T17:38:39.504 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0525000000000002e-05 2024-04-20T17:38:39.504 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.504 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:38:39.504 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.505 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:39.505 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:38:39.505 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:38:39.505 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:38:39.556 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:39.567 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 31 16 80319 80303 161.861 108.438 0.0140869 0.00617602 2024-04-20T17:38:39.608 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:38:39.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:39.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:38:39.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:39 smithi005 ceph-mon[39562]: osdmap e707: 12 total, 12 up, 10 in 2024-04-20T17:38:39.716 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:38:39.765 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:39.809 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:38:39.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:39.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:38:39.920 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:38:39.961 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:39 smithi171 ceph-mon[44189]: osdmap e707: 12 total, 12 up, 10 in 2024-04-20T17:38:39.963 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:39.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:39.994 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:38:39.994 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:38:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:39 smithi106 ceph-mon[44423]: osdmap e707: 12 total, 12 up, 10 in 2024-04-20T17:38:40.095 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:38:40.144 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:40.178 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:38:40.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:40.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:38:40.227 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:38:40.284 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:38:40.341 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:40.371 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:38:40.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:40.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:38:40.442 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-20T17:38:40.481 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:38:40.508 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:40.528 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:38:40.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:40.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:38:40.566 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 32 16 81939 81923 159.967 101.25 0.00435245 0.00624974 2024-04-20T17:38:40.632 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:38:40.654 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:38:40.674 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:38:40.710 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:40.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:40 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3194395304' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:38:40.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:40 smithi106 ceph-mon[44423]: from='client.27127 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:38:40.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:40 smithi106 ceph-mon[44423]: pgmap v1874: 17 pgs: 17 active+clean; 4.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-04-20T17:38:40.832 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:38:40.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:40 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3194395304' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:38:40.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:40 smithi171 ceph-mon[44189]: from='client.27127 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:38:40.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:40 smithi171 ceph-mon[44189]: pgmap v1874: 17 pgs: 17 active+clean; 4.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-04-20T17:38:40.870 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:38:40.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:40.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:38:40.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:40 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3194395304' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:38:40.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:40 smithi005 ceph-mon[39562]: from='client.27127 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:38:40.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:40 smithi005 ceph-mon[39562]: pgmap v1874: 17 pgs: 17 active+clean; 4.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-04-20T17:38:40.979 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:38:41.007 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:41.029 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:38:41.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:41.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:38:41.132 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:38:41.159 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:41.177 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:38:41.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:41.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:38:41.281 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:38:41.312 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:41.566 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 33 16 83337 83321 157.767 87.375 0.00242322 0.00633458 2024-04-20T17:38:41.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:41 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:38:41.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:41 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:38:41.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:41 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:38:41.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:41 smithi005 ceph-mon[39562]: from='client.27127 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:38:41.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:41 smithi005 ceph-mon[39562]: osdmap e708: 12 total, 12 up, 10 in 2024-04-20T17:38:41.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:41 smithi005 ceph-mon[39562]: osdmap e709: 12 total, 12 up, 10 in 2024-04-20T17:38:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:41 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:38:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:41 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:38:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:41 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:38:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:41 smithi106 ceph-mon[44423]: from='client.27127 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:38:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:41 smithi106 ceph-mon[44423]: osdmap e708: 12 total, 12 up, 10 in 2024-04-20T17:38:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:41 smithi106 ceph-mon[44423]: osdmap e709: 12 total, 12 up, 10 in 2024-04-20T17:38:42.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:41 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:38:42.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:41 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:38:42.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:41 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:38:42.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:41 smithi171 ceph-mon[44189]: from='client.27127 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:38:42.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:41 smithi171 ceph-mon[44189]: osdmap e708: 12 total, 12 up, 10 in 2024-04-20T17:38:42.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:41 smithi171 ceph-mon[44189]: osdmap e709: 12 total, 12 up, 10 in 2024-04-20T17:38:42.181 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:38:42.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:42.184 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:38:42.285 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:38:42.314 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:42.337 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:38:42.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:42.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:38:42.487 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:38:42.513 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:42.534 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:38:42.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:42.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:38:42.567 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 34 16 85000 84984 156.184 103.938 0.00647643 0.00640093 2024-04-20T17:38:42.641 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:38:42.718 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:42.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:42 smithi005 ceph-mon[39562]: pgmap v1877: 17 pgs: 17 active+clean; 4.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-04-20T17:38:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:42 smithi106 ceph-mon[44423]: pgmap v1877: 17 pgs: 17 active+clean; 4.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-04-20T17:38:43.014 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:42 smithi171 ceph-mon[44189]: pgmap v1877: 17 pgs: 17 active+clean; 4.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-04-20T17:38:43.348 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:38:43.348 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:38:43.348 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.348 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:86324 7.0 7:04ae58f3:::benchmark_data_smithi171_161871_object86323:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e709)", 2024-04-20T17:38:43.348 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:43.343342+0000", 2024-04-20T17:38:43.349 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.0047789149999999999, 2024-04-20T17:38:43.349 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0048064370000000002, 2024-04-20T17:38:43.349 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:38:43.349 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:38:43.349 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:38:43.349 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:38:43.349 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:38:43.349 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:38:43.349 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 86324 2024-04-20T17:38:43.349 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.349 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:38:43.349 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.349 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:38:43.349 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.343342+0000", 2024-04-20T17:38:43.350 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:43.350 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.350 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.350 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:38:43.350 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.343342+0000", 2024-04-20T17:38:43.350 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:43.350 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.350 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.350 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:38:43.350 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.343345+0000", 2024-04-20T17:38:43.350 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3809999999999999e-06 2024-04-20T17:38:43.350 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.350 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.351 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:38:43.351 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.343373+0000", 2024-04-20T17:38:43.351 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7966e-05 2024-04-20T17:38:43.351 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.351 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.351 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:38:43.351 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.343392+0000", 2024-04-20T17:38:43.351 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8878000000000001e-05 2024-04-20T17:38:43.351 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.351 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.351 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:38:43.351 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.343403+0000", 2024-04-20T17:38:43.351 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0874e-05 2024-04-20T17:38:43.351 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.352 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.352 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:38:43.352 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.343504+0000", 2024-04-20T17:38:43.352 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000101215 2024-04-20T17:38:43.352 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.352 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.352 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:38:43.352 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.343634+0000", 2024-04-20T17:38:43.352 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013022199999999999 2024-04-20T17:38:43.352 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.352 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.352 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:38:43.352 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.343756+0000", 2024-04-20T17:38:43.352 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012228200000000001 2024-04-20T17:38:43.353 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.353 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.353 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:38:43.353 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.347094+0000", 2024-04-20T17:38:43.353 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0033375919999999999 2024-04-20T17:38:43.353 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:43.353 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:38:43.353 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:43.353 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.353 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.353 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.27088.0:86323 7.5 e709/694 7:a9a3bc21:::benchmark_data_smithi171_161871_object86322:head v 709'5399, mlcod=709'5397)", 2024-04-20T17:38:43.353 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:43.347004+0000", 2024-04-20T17:38:43.354 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.001116316, 2024-04-20T17:38:43.354 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00121492, 2024-04-20T17:38:43.354 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:38:43.354 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:38:43.354 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:38:43.354 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:38:43.355 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.355 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:38:43.355 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.347004+0000", 2024-04-20T17:38:43.355 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:43.355 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.355 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.355 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:38:43.355 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.347004+0000", 2024-04-20T17:38:43.355 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:43.355 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.355 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.355 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:38:43.355 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.347006+0000", 2024-04-20T17:38:43.355 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.508e-06 2024-04-20T17:38:43.356 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.356 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.356 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:38:43.356 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.347033+0000", 2024-04-20T17:38:43.356 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7543999999999999e-05 2024-04-20T17:38:43.356 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.356 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.356 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:38:43.356 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.347044+0000", 2024-04-20T17:38:43.356 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1029999999999999e-05 2024-04-20T17:38:43.356 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.356 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.356 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:38:43.356 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.347052+0000", 2024-04-20T17:38:43.356 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9610000000000005e-06 2024-04-20T17:38:43.357 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.357 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.357 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:38:43.357 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.347112+0000", 2024-04-20T17:38:43.357 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9577999999999998e-05 2024-04-20T17:38:43.357 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.357 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.357 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:38:43.357 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.347127+0000", 2024-04-20T17:38:43.357 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5401999999999999e-05 2024-04-20T17:38:43.357 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:43.357 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:38:43.357 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:43.357 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.357 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.358 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:86328 7.0 7:01b39fbc:::benchmark_data_smithi171_161871_object86327:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e709)", 2024-04-20T17:38:43.358 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:43.347269+0000", 2024-04-20T17:38:43.358 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.00085139199999999997, 2024-04-20T17:38:43.358 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001023274, 2024-04-20T17:38:43.358 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:38:43.358 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:38:43.358 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:38:43.358 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:38:43.358 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:38:43.358 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:38:43.358 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 86328 2024-04-20T17:38:43.358 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.358 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:38:43.358 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.359 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:38:43.359 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.347269+0000", 2024-04-20T17:38:43.359 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:43.359 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.359 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.359 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:38:43.359 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.347269+0000", 2024-04-20T17:38:43.359 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:38:43.359 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.359 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.359 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:38:43.359 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.347271+0000", 2024-04-20T17:38:43.359 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2350000000000002e-06 2024-04-20T17:38:43.359 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.359 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.360 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:38:43.360 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.347297+0000", 2024-04-20T17:38:43.360 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5788000000000001e-05 2024-04-20T17:38:43.360 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.360 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.360 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:38:43.360 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.347309+0000", 2024-04-20T17:38:43.360 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1718e-05 2024-04-20T17:38:43.360 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.360 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.360 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:38:43.360 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.347316+0000", 2024-04-20T17:38:43.360 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5399999999999998e-06 2024-04-20T17:38:43.360 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.360 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.360 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:38:43.361 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.347399+0000", 2024-04-20T17:38:43.361 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2640999999999997e-05 2024-04-20T17:38:43.361 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.361 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.361 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:38:43.361 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.347566+0000", 2024-04-20T17:38:43.361 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000166902 2024-04-20T17:38:43.361 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:38:43.361 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:38:43.361 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:38:43.361 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.347757+0000", 2024-04-20T17:38:43.361 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00019120300000000001 2024-04-20T17:38:43.361 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:43.361 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:38:43.361 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:43.362 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:38:43.362 INFO:teuthology.orchestra.run.smithi171.stdout: ], 2024-04-20T17:38:43.362 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 3 2024-04-20T17:38:43.362 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:38:43.485 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:38:43.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:43.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:38:43.566 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 35 16 86702 86686 154.761 106.375 0.00693961 0.0064603 2024-04-20T17:38:43.592 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:38:43.645 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:44.009 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:38:44.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:44.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:38:44.116 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:38:44.160 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:44.373 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:38:44.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:44.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:38:44.482 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:38:44.526 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:44.567 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 36 16 88094 88078 152.879 87 0.012947 0.00653906 2024-04-20T17:38:44.635 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:38:44.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:44.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:38:44.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:44 smithi005 ceph-mon[39562]: pgmap v1878: 17 pgs: 17 active+clean; 5.1 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 123 MiB/s wr, 1.96k op/s 2024-04-20T17:38:44.736 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:38:44.750 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:38:44.768 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:38:44.778 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:44 smithi171 ceph-mon[44189]: pgmap v1878: 17 pgs: 17 active+clean; 5.1 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 123 MiB/s wr, 1.96k op/s 2024-04-20T17:38:44.796 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:38:44.820 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:44.833 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-20T17:38:44.833 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:38:44.845 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:38:44.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:44.849 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:38:44.866 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:38:44.907 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:38:44.937 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:38:44.950 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:38:44.971 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:38:44.975 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:44 smithi106 ceph-mon[44423]: pgmap v1878: 17 pgs: 17 active+clean; 5.1 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 123 MiB/s wr, 1.96k op/s 2024-04-20T17:38:44.977 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:44.997 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:38:44.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:45.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:38:45.002 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:38:45.037 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:38:45.069 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:38:45.101 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:38:45.102 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:38:45.106 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:38:45.106 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:38:45.106 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:38:45.106 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:38:45.106 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:38:45.106 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:38:45.106 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:38:45.106 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:38:45.107 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:38:45.107 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:38:45.107 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:38:45.107 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:38:45.164 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:45.255 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:38:45.387 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:38:45.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:45.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:38:45.494 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:38:45.517 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:38:45.522 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:45.544 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:38:45.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:45.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:38:45.567 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 37 16 89669 89653 151.407 98.4375 0.00567374 0.00660314 2024-04-20T17:38:45.647 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:38:45.675 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:45.698 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:38:45.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:45.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:38:45.802 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:38:45.827 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:45 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1383742589' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:38:45.827 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:45 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:38:45.827 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:45 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:38:45.827 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:45.827 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:45.827 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:45.827 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:45.829 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:45 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1383742589' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:38:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:45 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:38:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:45 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:38:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:45.848 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:38:45.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:45.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:38:45.952 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:38:45.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:45 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1383742589' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:38:45.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:45 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:38:45.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:45 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:38:45.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:45.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:45.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:45.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:45.998 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:46.567 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 38 16 90903 90887 149.452 77.125 0.00343259 0.00667658 2024-04-20T17:38:46.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:46.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:46.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:46.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:46.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:46.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:46.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:46.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:46.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:46 smithi005 ceph-mon[39562]: osdmap e710: 12 total, 12 up, 10 in 2024-04-20T17:38:46.895 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:46 smithi005 ceph-mon[39562]: pgmap v1880: 17 pgs: 17 active+clean; 5.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-04-20T17:38:46.895 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:46 smithi005 ceph-mon[39562]: osdmap e711: 12 total, 12 up, 10 in 2024-04-20T17:38:46.906 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:38:46.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:46.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:38:46.972 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:46.973 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:46.973 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:46.973 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:46.973 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:46.973 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:46.973 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:46.974 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:46.974 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:46 smithi171 ceph-mon[44189]: osdmap e710: 12 total, 12 up, 10 in 2024-04-20T17:38:46.974 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:46 smithi171 ceph-mon[44189]: pgmap v1880: 17 pgs: 17 active+clean; 5.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-04-20T17:38:46.974 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:46 smithi171 ceph-mon[44189]: osdmap e711: 12 total, 12 up, 10 in 2024-04-20T17:38:47.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:47.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:47.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:47.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:47.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:47.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:47.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:47.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:47.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:46 smithi106 ceph-mon[44423]: osdmap e710: 12 total, 12 up, 10 in 2024-04-20T17:38:47.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:46 smithi106 ceph-mon[44423]: pgmap v1880: 17 pgs: 17 active+clean; 5.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-04-20T17:38:47.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:46 smithi106 ceph-mon[44423]: osdmap e711: 12 total, 12 up, 10 in 2024-04-20T17:38:47.012 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:38:47.019 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:38:47.073 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:47.302 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:38:47.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:47.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:38:47.414 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:38:47.445 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:47.464 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:38:47.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:47.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:38:47.568 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:38:47.570 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 39 16 92521 92505 148.201 101.125 0.0153109 0.0067458 2024-04-20T17:38:47.617 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:47.659 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:38:47.659 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:38:48.570 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:38:48.571305 min lat: 0.00100541 max lat: 0.210579 avg lat: 0.00681442 2024-04-20T17:38:48.570 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:38:48.570 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 40 16 93940 93924 146.713 88.6875 0.0123095 0.00681442 2024-04-20T17:38:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:48 smithi171 ceph-mon[44189]: pgmap v1882: 17 pgs: 17 active+clean; 5.4 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-04-20T17:38:48.944 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:38:48.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:48 smithi005 ceph-mon[39562]: pgmap v1882: 17 pgs: 17 active+clean; 5.4 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-04-20T17:38:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:48 smithi106 ceph-mon[44423]: pgmap v1882: 17 pgs: 17 active+clean; 5.4 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-04-20T17:38:49.089 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:38:49.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:49.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:38:49.105 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-20T17:38:49.196 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:38:49.270 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:49.520 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:38:49.570 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 41 16 95406 95390 145.37 91.625 0.0159161 0.0068776 2024-04-20T17:38:49.836 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:38:49.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:49.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:38:49.940 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:38:49.993 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:50.457 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:38:50.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:50.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:38:50.500 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:38:50.562 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:38:50.570 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 42 16 96936 96920 144.185 95.625 0.0131171 0.00693393 2024-04-20T17:38:50.590 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:50.610 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:38:50.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:50.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:38:50.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:50 smithi005 ceph-mon[39562]: pgmap v1883: 17 pgs: 17 active+clean; 5.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 100 MiB/s wr, 1.61k op/s 2024-04-20T17:38:50.713 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:38:50.780 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:50.824 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:38:50.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:50 smithi171 ceph-mon[44189]: pgmap v1883: 17 pgs: 17 active+clean; 5.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 100 MiB/s wr, 1.61k op/s 2024-04-20T17:38:50.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:50.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:38:50.936 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:38:50.982 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:50 smithi106 ceph-mon[44423]: pgmap v1883: 17 pgs: 17 active+clean; 5.7 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 100 MiB/s wr, 1.61k op/s 2024-04-20T17:38:51.017 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:38:51.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:51.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:38:51.122 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:38:51.170 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-20T17:38:51.170 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:38:51.176 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:51.209 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:38:51.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:51.213 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:38:51.314 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:38:51.352 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:51.505 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:38:51.553 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:38:51.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:51.558 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:38:51.570 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 43 16 98366 98350 142.911 89.375 0.0053441 0.00699575 2024-04-20T17:38:51.659 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:38:51.696 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:51.776 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:38:51.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:51.783 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:38:51.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:51 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/983252771' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:38:51.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:51 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:38:51.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:51 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/983252771' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:38:51.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:51 smithi171 ceph-mon[44189]: osdmap e712: 12 total, 12 up, 10 in 2024-04-20T17:38:51.884 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:38:51.935 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:51.948 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:51 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/983252771' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:38:51.948 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:51 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:38:51.948 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:51 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/983252771' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:38:51.948 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:51 smithi005 ceph-mon[39562]: osdmap e712: 12 total, 12 up, 10 in 2024-04-20T17:38:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:51 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/983252771' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:38:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:51 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:38:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:51 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/983252771' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:38:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:51 smithi106 ceph-mon[44423]: osdmap e712: 12 total, 12 up, 10 in 2024-04-20T17:38:52.281 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:38:52.281 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:38:52.570 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 44 16 99891 99875 141.829 95.3125 0.00639857 0.00704899 2024-04-20T17:38:52.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:52 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:38:52.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:52 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:38:52.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:52 smithi005 ceph-mon[39562]: pgmap v1885: 17 pgs: 17 active+clean; 5.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-04-20T17:38:52.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:52 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:38:52.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:52 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:38:52.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:52 smithi171 ceph-mon[44189]: pgmap v1885: 17 pgs: 17 active+clean; 5.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-04-20T17:38:52.908 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:38:52.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:52.916 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:38:52.941 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-20T17:38:53.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:52 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:38:53.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:52 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:38:53.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:52 smithi106 ceph-mon[44423]: pgmap v1885: 17 pgs: 17 active+clean; 5.9 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-04-20T17:38:53.016 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:38:53.069 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:53.092 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:38:53.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:53.096 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:38:53.197 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:38:53.226 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:53.245 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:38:53.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:53.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:38:53.349 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:38:53.376 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:53.383 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:38:53.396 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:38:53.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:53.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:38:53.399 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:38:53.464 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:38:53.496 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:38:53.499 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:38:53.532 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:38:53.539 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:53.568 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:38:53.570 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 45 16 101355 101339 140.711 91.5 0.0163383 0.00710472 2024-04-20T17:38:53.604 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:38:53.640 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:38:53.673 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:38:53.679 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:38:53.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:53.681 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:38:53.717 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:38:53.756 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:38:53.782 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:38:53.791 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:38:53.804 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:53 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/534440863' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:38:53.824 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:38:53.854 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:53 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/534440863' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:38:53.865 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:38:53.869 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:53.877 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:38:53.877 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:38:53.877 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:38:53.877 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:38:53.877 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:38:53.877 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:38:53.878 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:38:53.878 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:38:53.878 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:38:53.878 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:38:53.878 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:38:53.878 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:38:53.908 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:38:53.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:53.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:38:54.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:53 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/534440863' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:38:54.012 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:38:54.053 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:54.265 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:38:54.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:54.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:38:54.373 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:38:54.400 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:54.426 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:38:54.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:54.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:38:54.531 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:38:54.571 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 46 16 102815 102799 139.635 91.25 0.00796492 0.0071595 2024-04-20T17:38:54.573 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:54.599 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.600 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.600 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.600 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.600 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.600 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.610 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:38:54.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:54.614 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:38:54.715 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:38:54.743 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.743 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.744 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.744 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.744 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.744 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.744 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.744 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.746 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.746 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.746 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.746 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:54 smithi106 ceph-mon[44423]: pgmap v1886: 17 pgs: 17 active+clean; 6.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-04-20T17:38:54.746 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:54.773 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:38:54.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:54.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:38:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:54 smithi171 ceph-mon[44189]: pgmap v1886: 17 pgs: 17 active+clean; 6.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-04-20T17:38:54.874 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:38:54.877 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:38:54.917 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:54.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:54.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:54 smithi005 ceph-mon[39562]: pgmap v1886: 17 pgs: 17 active+clean; 6.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-04-20T17:38:55.047 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:38:55.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:55.053 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:38:55.153 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:38:55.221 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:55.574 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 47 16 104285 104269 138.611 91.875 0.013737 0.00721307 2024-04-20T17:38:55.686 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:38:55.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:55.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:38:55.711 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:38:55.711 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:38:55.711 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 0 2024-04-20T17:38:55.711 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:38:55.740 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:55.792 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:38:55.819 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:55.840 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:38:55.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:55.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:38:55.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:55.944 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:38:55.969 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:38:55.970 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:55.992 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:38:55.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:55.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:38:56.096 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:38:56.142 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:56.268 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:38:56.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:56.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:38:56.326 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:38:56.372 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:38:56.399 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:56.419 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:38:56.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:56.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:38:56.456 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-20T17:38:56.523 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:38:56.574 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 48 16 105624 105608 137.467 83.6875 0.019038 0.00727303 2024-04-20T17:38:56.596 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:56.620 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:38:56.623 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:56 smithi005 ceph-mon[39562]: pgmap v1887: 17 pgs: 17 active+clean; 6.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-04-20T17:38:56.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:56.626 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:38:56.653 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:38:56.653 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:38:56.727 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:38:56.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:56 smithi106 ceph-mon[44423]: pgmap v1887: 17 pgs: 17 active+clean; 6.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-04-20T17:38:56.769 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:56 smithi171 ceph-mon[44189]: pgmap v1887: 17 pgs: 17 active+clean; 6.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-04-20T17:38:56.773 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:56.804 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:38:56.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:56.808 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:38:56.859 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:38:56.910 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:38:56.936 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:56.960 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:38:56.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:56.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:38:57.064 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:38:57.092 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:38:57.097 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:57.147 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:38:57.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:57.150 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:38:57.251 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:38:57.328 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:57.361 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:38:57.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:57.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:38:57.465 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:38:57.516 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:57.574 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 49 16 106963 106947 136.369 83.6875 0.00634286 0.00733147 2024-04-20T17:38:57.634 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:57 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:38:57.634 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:57 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:38:57.634 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:57 smithi005 ceph-mon[39562]: from='client.17124 v1:172.21.15.5:0/2793052864' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:38:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:57 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:38:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:57 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:38:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:57 smithi171 ceph-mon[44189]: from='client.17124 v1:172.21.15.5:0/2793052864' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:38:58.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:57 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:38:58.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:57 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:38:58.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:57 smithi106 ceph-mon[44423]: from='client.17124 v1:172.21.15.5:0/2793052864' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:38:58.457 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:38:58.481 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":1887,"stamp":"2024-04-20T17:38:56.466387+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6712066592,"num_objects":102411,"num_object_clones":0,"num_object_copies":204822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102411,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":102590,"num_write_kb":6557114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102546,"ondisk_log_size":102546,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":17555440,"kb_used_data":12994844,"kb_used_omap":224,"kb_used_meta":4560223,"kb_avail":919896080,"statfs":{"total":959950356480,"available":941973585920,"internally_reserved":0,"allocated":13306720256,"data_stored":13302128976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":229753,"internal_metadata":4669668999},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":968097792,"num_objects":14772,"num_object_clones":0,"num_object_copies":29544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14772,"num_write_kb":945408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.990597"},"pg_stats":[{"pgid":"7.f","version":"712'6516","reported_seq":6557,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-20T17:38:54.793388+0000","last_change":"2024-04-20T17:37:56.836004+0000","last_active":"2024-04-20T17:38:54.793388+0000","last_peered":"2024-04-20T17:38:54.793388+0000","last_clean":"2024-04-20T17:38:54.793388+0000","last_became_active":"2024-04-20T17:37:56.835750+0000","last_became_peered":"2024-04-20T17:37:56.835750+0000","last_unstale":"2024-04-20T17:38:54.793388+0000","last_undegraded":"2024-04-20T17:38:54.793388+0000","last_fullsized":"2024-04-20T17:38:54.793388+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":6516,"log_dups_size":0,"ondisk_log_size":6516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:39:14.458226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427032576,"num_objects":6516,"num_object_clones":0,"num_object_copies":13032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6516,"num_write_kb":417024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"712'6067","reported_seq":6112,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-20T17:38:51.712759+0000","last_change":"2024-04-20T17:37:56.833945+0000","last_active":"2024-04-20T17:38:51.712759+0000","last_peered":"2024-04-20T17:38:51.712759+0000","last_clean":"2024-04-20T17:38:51.712759+0000","last_became_active":"2024-04-20T17:37:56.833703+0000","last_became_peered":"2024-04-20T17:37:56.833703+0000","last_unstale":"2024-04-20T17:38:51.712759+0000","last_undegraded":"2024-04-20T17:38:51.712759+0000","last_fullsized":"2024-04-20T17:38:51.712759+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":6067,"log_dups_size":0,"ondisk_log_size":6067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:39:04.033531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397606912,"num_objects":6067,"num_object_clones":0,"num_object_copies":12134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6067,"num_write_kb":388288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"712'6617","reported_seq":6662,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-20T17:38:54.792356+0000","last_change":"2024-04-20T17:37:56.836989+0000","last_active":"2024-04-20T17:38:54.792356+0000","last_peered":"2024-04-20T17:38:54.792356+0000","last_clean":"2024-04-20T17:38:54.792356+0000","last_became_active":"2024-04-20T17:37:56.836496+0000","last_became_peered":"2024-04-20T17:37:56.836496+0000","last_unstale":"2024-04-20T17:38:54.792356+0000","last_undegraded":"2024-04-20T17:38:54.792356+0000","last_fullsized":"2024-04-20T17:38:54.792356+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":6617,"log_dups_size":0,"ondisk_log_size":6617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:39:11.524384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433651712,"num_objects":6617,"num_object_clones":0,"num_object_copies":13234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6617,"num_write_kb":423488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.c","version":"712'6398","reported_seq":6443,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-20T17:38:53.987641+0000","last_change":"2024-04-20T17:37:56.834884+0000","last_active":"2024-04-20T17:38:53.987641+0000","last_peered":"2024-04-20T17:38:53.987641+0000","last_clean":"2024-04-20T17:38:53.987641+0000","last_became_active":"2024-04-20T17:37:56.834181+0000","last_became_peered":"2024-04-20T17:37:56.834181+0000","last_unstale":"2024-04-20T17:38:53.987641+0000","last_undegraded":"2024-04-20T17:38:53.987641+0000","last_fullsized":"2024-04-20T17:38:53.987641+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":6398,"log_dups_size":0,"ondisk_log_size":6398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:39:09.740495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419299328,"num_objects":6398,"num_object_clones":0,"num_object_copies":12796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6398,"num_write_kb":409472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"712'6335","reported_seq":6380,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-20T17:38:54.010273+0000","last_change":"2024-04-20T17:37:56.830736+0000","last_active":"2024-04-20T17:38:54.010273+0000","last_peered":"2024-04-20T17:38:54.010273+0000","last_clean":"2024-04-20T17:38:54.010273+0000","last_became_active":"2024-04-20T17:37:56.830487+0000","last_became_peered":"2024-04-20T17:37:56.830487+0000","last_unstale":"2024-04-20T17:38:54.010273+0000","last_undegraded":"2024-04-20T17:38:54.010273+0000","last_fullsized":"2024-04-20T17:38:54.010273+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":6335,"log_dups_size":0,"ondisk_log_size":6335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:39:21.390993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415170560,"num_objects":6335,"num_object_clones":0,"num_object_copies":12670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6335,"num_write_kb":405440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.a","version":"712'6502","reported_seq":6547,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-20T17:38:55.847561+0000","last_change":"2024-04-20T17:37:56.836237+0000","last_active":"2024-04-20T17:38:55.847561+0000","last_peered":"2024-04-20T17:38:55.847561+0000","last_clean":"2024-04-20T17:38:55.847561+0000","last_became_active":"2024-04-20T17:37:56.835944+0000","last_became_peered":"2024-04-20T17:37:56.835944+0000","last_unstale":"2024-04-20T17:38:55.847561+0000","last_undegraded":"2024-04-20T17:38:55.847561+0000","last_fullsized":"2024-04-20T17:38:55.847561+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":6502,"log_dups_size":0,"ondisk_log_size":6502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:39:22.130326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426115072,"num_objects":6502,"num_object_clones":0,"num_object_copies":13004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6502,"num_write_kb":416128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"712'6455","reported_seq":6500,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-20T17:38:54.789549+0000","last_change":"2024-04-20T17:37:56.835919+0000","last_active":"2024-04-20T17:38:54.789549+0000","last_peered":"2024-04-20T17:38:54.789549+0000","last_clean":"2024-04-20T17:38:54.789549+0000","last_became_active":"2024-04-20T17:37:56.835700+0000","last_became_peered":"2024-04-20T17:37:56.835700+0000","last_unstale":"2024-04-20T17:38:54.789549+0000","last_undegraded":"2024-04-20T17:38:54.789549+0000","last_fullsized":"2024-04-20T17:38:54.789549+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":6455,"log_dups_size":0,"ondisk_log_size":6455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:38:59.554456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423034880,"num_objects":6455,"num_object_clones":0,"num_object_copies":12910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6455,"num_write_kb":413120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11],"acting":[0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"712'6101","reported_seq":6146,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-20T17:38:51.708741+0000","last_change":"2024-04-20T17:37:56.914447+0000","last_active":"2024-04-20T17:38:51.708741+0000","last_peered":"2024-04-20T17:38:51.708741+0000","last_clean":"2024-04-20T17:38:51.708741+0000","last_became_active":"2024-04-20T17:37:56.914124+0000","last_became_peered":"2024-04-20T17:37:56.914124+0000","last_unstale":"2024-04-20T17:38:51.708741+0000","last_undegraded":"2024-04-20T17:38:51.708741+0000","last_fullsized":"2024-04-20T17:38:51.708741+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":6101,"log_dups_size":0,"ondisk_log_size":6101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:39:00.568943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399835136,"num_objects":6101,"num_object_clones":0,"num_object_copies":12202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6101,"num_write_kb":390464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"712'6424","reported_seq":6469,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-20T17:38:53.662046+0000","last_change":"2024-04-20T17:37:56.836954+0000","last_active":"2024-04-20T17:38:53.662046+0000","last_peered":"2024-04-20T17:38:53.662046+0000","last_clean":"2024-04-20T17:38:53.662046+0000","last_became_active":"2024-04-20T17:37:56.836665+0000","last_became_peered":"2024-04-20T17:37:56.836665+0000","last_unstale":"2024-04-20T17:38:53.662046+0000","last_undegraded":"2024-04-20T17:38:53.662046+0000","last_fullsized":"2024-04-20T17:38:53.662046+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":6424,"log_dups_size":0,"ondisk_log_size":6424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:39:03.851782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":421003264,"num_objects":6424,"num_object_clones":0,"num_object_copies":12848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6424,"num_write_kb":411136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.5","version":"712'6386","reported_seq":6431,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-20T17:38:53.995419+0000","last_change":"2024-04-20T17:37:56.834318+0000","last_active":"2024-04-20T17:38:53.995419+0000","last_peered":"2024-04-20T17:38:53.995419+0000","last_clean":"2024-04-20T17:38:53.995419+0000","last_became_active":"2024-04-20T17:37:56.833600+0000","last_became_peered":"2024-04-20T17:37:56.833600+0000","last_unstale":"2024-04-20T17:38:53.995419+0000","last_undegraded":"2024-04-20T17:38:53.995419+0000","last_fullsized":"2024-04-20T17:38:53.995419+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":6386,"log_dups_size":0,"ondisk_log_size":6386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:39:02.278819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418512896,"num_objects":6386,"num_object_clones":0,"num_object_copies":12772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6386,"num_write_kb":408704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.4","version":"712'6353","reported_seq":6398,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-20T17:38:54.765579+0000","last_change":"2024-04-20T17:37:56.834982+0000","last_active":"2024-04-20T17:38:54.765579+0000","last_peered":"2024-04-20T17:38:54.765579+0000","last_clean":"2024-04-20T17:38:54.765579+0000","last_became_active":"2024-04-20T17:37:56.834727+0000","last_became_peered":"2024-04-20T17:37:56.834727+0000","last_unstale":"2024-04-20T17:38:54.765579+0000","last_undegraded":"2024-04-20T17:38:54.765579+0000","last_fullsized":"2024-04-20T17:38:54.765579+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":6353,"log_dups_size":0,"ondisk_log_size":6353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:39:20.179880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416350208,"num_objects":6353,"num_object_clones":0,"num_object_copies":12706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6353,"num_write_kb":406592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"712'6271","reported_seq":6314,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-20T17:38:53.962975+0000","last_change":"2024-04-20T17:37:56.833156+0000","last_active":"2024-04-20T17:38:53.962975+0000","last_peered":"2024-04-20T17:38:53.962975+0000","last_clean":"2024-04-20T17:38:53.962975+0000","last_became_active":"2024-04-20T17:37:56.832911+0000","last_became_peered":"2024-04-20T17:37:56.832911+0000","last_unstale":"2024-04-20T17:38:53.962975+0000","last_undegraded":"2024-04-20T17:38:53.962975+0000","last_fullsized":"2024-04-20T17:38:53.962975+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":6271,"log_dups_size":0,"ondisk_log_size":6271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:38:57.128825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410976256,"num_objects":6271,"num_object_clones":0,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6271,"num_write_kb":401344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"712'6511","reported_seq":6555,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-20T17:38:53.987372+0000","last_change":"2024-04-20T17:37:56.834572+0000","last_active":"2024-04-20T17:38:53.987372+0000","last_peered":"2024-04-20T17:38:53.987372+0000","last_clean":"2024-04-20T17:38:53.987372+0000","last_became_active":"2024-04-20T17:37:56.834292+0000","last_became_peered":"2024-04-20T17:37:56.834292+0000","last_unstale":"2024-04-20T17:38:53.987372+0000","last_undegraded":"2024-04-20T17:38:53.987372+0000","last_fullsized":"2024-04-20T17:38:53.987372+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":6511,"log_dups_size":0,"ondisk_log_size":6511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:39:06.608157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426704896,"num_objects":6511,"num_object_clones":0,"num_object_copies":13022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6511,"num_write_kb":416704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"712'6516","reported_seq":6561,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-20T17:38:55.658990+0000","last_change":"2024-04-20T17:37:56.836188+0000","last_active":"2024-04-20T17:38:55.658990+0000","last_peered":"2024-04-20T17:38:55.658990+0000","last_clean":"2024-04-20T17:38:55.658990+0000","last_became_active":"2024-04-20T17:37:56.835512+0000","last_became_peered":"2024-04-20T17:37:56.835512+0000","last_unstale":"2024-04-20T17:38:55.658990+0000","last_undegraded":"2024-04-20T17:38:55.658990+0000","last_fullsized":"2024-04-20T17:38:55.658990+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":6516,"log_dups_size":0,"ondisk_log_size":6516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:39:16.406056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427032576,"num_objects":6516,"num_object_clones":0,"num_object_copies":13032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6516,"num_write_kb":417024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"712'6553","reported_seq":6598,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-20T17:38:55.821349+0000","last_change":"2024-04-20T17:37:56.834274+0000","last_active":"2024-04-20T17:38:55.821349+0000","last_peered":"2024-04-20T17:38:55.821349+0000","last_clean":"2024-04-20T17:38:55.821349+0000","last_became_active":"2024-04-20T17:37:56.833972+0000","last_became_peered":"2024-04-20T17:37:56.833972+0000","last_unstale":"2024-04-20T17:38:55.821349+0000","last_undegraded":"2024-04-20T17:38:55.821349+0000","last_fullsized":"2024-04-20T17:38:55.821349+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":6553,"log_dups_size":0,"ondisk_log_size":6553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:39:13.669143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429457408,"num_objects":6553,"num_object_clones":0,"num_object_copies":13106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6553,"num_write_kb":419392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.6","version":"712'6404","reported_seq":6449,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-20T17:38:54.787994+0000","last_change":"2024-04-20T17:37:56.914636+0000","last_active":"2024-04-20T17:38:54.787994+0000","last_peered":"2024-04-20T17:38:54.787994+0000","last_clean":"2024-04-20T17:38:54.787994+0000","last_became_active":"2024-04-20T17:37:56.914210+0000","last_became_peered":"2024-04-20T17:37:56.914210+0000","last_unstale":"2024-04-20T17:38:54.787994+0000","last_undegraded":"2024-04-20T17:38:54.787994+0000","last_fullsized":"2024-04-20T17:38:54.787994+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":6404,"log_dups_size":0,"ondisk_log_size":6404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:39:02.094846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419692544,"num_objects":6404,"num_object_clones":0,"num_object_copies":12808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6404,"num_write_kb":409856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1011,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-04-20T17:38:51.504363+0000","last_change":"2024-04-20T17:37:25.403408+0000","last_active":"2024-04-20T17:38:51.504363+0000","last_peered":"2024-04-20T17:38:51.504363+0000","last_clean":"2024-04-20T17:38:51.504363+0000","last_became_active":"2024-04-20T17:23:24.109202+0000","last_became_peered":"2024-04-20T17:23:24.109202+0000","last_unstale":"2024-04-20T17:38:51.504363+0000","last_undegraded":"2024-04-20T17:38:51.504363+0000","last_fullsized":"2024-04-20T17:38:51.504363+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:37:25.403282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:37:25.403282+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6711476224,"num_objects":102409,"num_object_clones":0,"num_object_copies":204818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102409,"num_write_kb":6554176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13404274688,"data_stored":13404274688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102409,"ondisk_log_size":102409,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991631,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2254668,"kb_used_data":1614020,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":91490484,"statfs":{"total":95995035648,"available":93686255616,"internally_reserved":0,"allocated":1652756480,"data_stored":1652293894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":656013764},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253270,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1757956,"kb_used_data":1227004,"kb_used_omap":1,"kb_used_meta":530942,"kb_avail":91987196,"statfs":{"total":95995035648,"available":94194888704,"internally_reserved":0,"allocated":1256452096,"data_stored":1255993062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":543685056},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613019,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1789252,"kb_used_data":1252732,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":91955900,"statfs":{"total":95995035648,"available":94162841600,"internally_reserved":0,"allocated":1282797568,"data_stored":1282338554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":549386688},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809249,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1116168,"kb_used_data":814204,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92628984,"statfs":{"total":95995035648,"available":94852079616,"internally_reserved":0,"allocated":833744896,"data_stored":833285882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":398,"seq":1709396984005,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1145548,"kb_used_data":786564,"kb_used_omap":6,"kb_used_meta":358969,"kb_avail":92599604,"statfs":{"total":95995035648,"available":94821994496,"internally_reserved":0,"allocated":805441536,"data_stored":804978970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6767,"internal_metadata":367584657},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":2297807503479,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":2830383448102,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738823,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2658948,"kb_used_data":2054396,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":91086204,"statfs":{"total":95995035648,"available":93272272896,"internally_reserved":0,"allocated":2103701504,"data_stored":2103242490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394113,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1810360,"kb_used_data":1535664,"kb_used_omap":202,"kb_used_meta":274485,"kb_avail":91934792,"statfs":{"total":95995035648,"available":94141227008,"internally_reserved":0,"allocated":1572519936,"data_stored":1572066926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":281073111},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653163,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2342660,"kb_used_data":2057596,"kb_used_omap":4,"kb_used_meta":285051,"kb_avail":91402492,"statfs":{"total":95995035648,"available":93596151808,"internally_reserved":0,"allocated":2106978304,"data_stored":2106519290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":291892935},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986612,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1233924,"kb_used_data":823740,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92511228,"statfs":{"total":95995035648,"available":94731497472,"internally_reserved":0,"allocated":843509760,"data_stored":843050746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790382,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1445956,"kb_used_data":828924,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":92299196,"statfs":{"total":95995035648,"available":94514376704,"internally_reserved":0,"allocated":848818176,"data_stored":848359162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2119761920,"data_stored":2119761920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1593966592,"data_stored":1593966592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2108620800,"data_stored":2108620800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":848232448,"data_stored":848232448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":854327296,"data_stored":854327296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":814415872,"data_stored":814415872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":841613312,"data_stored":841613312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1285226496,"data_stored":1285226496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1266155520,"data_stored":1266155520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1671954432,"data_stored":1671954432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:38:58.483 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:38:58.574 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 50 16 108429 108413 135.475 91.625 0.0195516 0.0073799 2024-04-20T17:38:58.575 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:38:58.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:58.579 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:38:58.680 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:38:58.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:58 smithi005 ceph-mon[39562]: pgmap v1888: 17 pgs: 17 active+clean; 6.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-04-20T17:38:58.736 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:58.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:58 smithi171 ceph-mon[44189]: pgmap v1888: 17 pgs: 17 active+clean; 6.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-04-20T17:38:59.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:58 smithi106 ceph-mon[44423]: pgmap v1888: 17 pgs: 17 active+clean; 6.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-04-20T17:38:59.221 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:38:59.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:59.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:38:59.330 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:38:59.365 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:59.491 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:38:59.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:38:59.496 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:38:59.574 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 51 16 109659 109643 134.326 76.875 0.00652411 0.00744313 2024-04-20T17:38:59.597 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:38:59.666 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:38:59.691 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:38:59.807 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:38:59.807 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:38:59.807 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-04-20T17:38:59.807 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-04-20T17:38:59.807 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:38:59.828 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:59 smithi171 ceph-mon[44189]: 7.3 scrub starts 2024-04-20T17:38:59.828 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:38:59 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2280199386' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:38:59.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:59 smithi005 ceph-mon[39562]: 7.3 scrub starts 2024-04-20T17:38:59.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:38:59 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2280199386' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:39:00.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:59 smithi106 ceph-mon[44423]: 7.3 scrub starts 2024-04-20T17:39:00.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:38:59 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2280199386' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:39:00.061 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:39:00.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:00.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:39:00.166 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:39:00.218 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:00.574 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 52 16 110936 110920 133.277 79.8125 0.0126086 0.00750158 2024-04-20T17:39:00.688 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:39:00.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:00.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:39:00.803 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:39:00.857 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:00.857 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:00 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:39:00.857 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:00 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:39:00.857 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:00 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2280199386' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:39:00.858 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:00 smithi005 ceph-mon[39562]: osdmap e713: 12 total, 12 up, 10 in 2024-04-20T17:39:00.858 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:00 smithi005 ceph-mon[39562]: pgmap v1890: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-20T17:39:00.999 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:39:01.009 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:39:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:00 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:39:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:00 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:39:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:00 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2280199386' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:39:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:00 smithi106 ceph-mon[44423]: osdmap e713: 12 total, 12 up, 10 in 2024-04-20T17:39:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:00 smithi106 ceph-mon[44423]: pgmap v1890: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-20T17:39:01.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:01.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:39:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:00 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:39:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:00 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:39:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:00 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2280199386' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:39:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:00 smithi171 ceph-mon[44189]: osdmap e713: 12 total, 12 up, 10 in 2024-04-20T17:39:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:00 smithi171 ceph-mon[44189]: pgmap v1890: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-20T17:39:01.114 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:39:01.168 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:01.181 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-20T17:39:01.200 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:39:01.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:01.206 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:39:01.308 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:39:01.373 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:01.408 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:39:01.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:01.414 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:39:01.515 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:39:01.576 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 53 16 112285 112269 132.348 84.3125 0.0197306 0.0075539 2024-04-20T17:39:01.580 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:39:01.583 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:01.648 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:39:01.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:01.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:39:01.670 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:01 smithi171 ceph-mon[44189]: 7.3 scrub ok 2024-04-20T17:39:01.670 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:01 smithi171 ceph-mon[44189]: osdmap e714: 12 total, 12 up, 10 in 2024-04-20T17:39:01.759 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:39:01.789 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:01.808 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:39:01.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:01.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:39:01.914 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:39:01.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:01 smithi005 ceph-mon[39562]: 7.3 scrub ok 2024-04-20T17:39:01.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:01 smithi005 ceph-mon[39562]: osdmap e714: 12 total, 12 up, 10 in 2024-04-20T17:39:01.954 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:01 smithi106 ceph-mon[44423]: 7.3 scrub ok 2024-04-20T17:39:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:01 smithi106 ceph-mon[44423]: osdmap e714: 12 total, 12 up, 10 in 2024-04-20T17:39:02.181 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:39:02.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:02.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:39:02.300 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:39:02.327 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:02.347 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:39:02.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:02.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:39:02.452 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:39:02.480 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:02.501 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:39:02.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:02.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:39:02.576 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 54 16 113441 113425 131.236 72.25 0.00882606 0.00761786 2024-04-20T17:39:02.600 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:39:02.605 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:39:02.633 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:02.657 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:39:02.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:02.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:39:02.708 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:39:02.709 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:39:02.709 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:39:02.709 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:39:02.709 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:39:02.709 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:39:02.709 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:39:02.709 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:39:02.709 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:39:02.710 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:39:02.710 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:39:02.710 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:39:02.710 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:39:02.710 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:39:02.710 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:39:02.710 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:39:02.710 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:39:02.710 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:39:02.710 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:39:02.710 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:39:02.711 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:39:02.711 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:39:02.711 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:39:02.711 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:39:02.761 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:39:02.815 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:02 smithi005 ceph-mon[39562]: 7.8 deep-scrub starts 2024-04-20T17:39:02.815 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:02 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1897526864' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:39:02.815 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:02.815 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:02 smithi005 ceph-mon[39562]: 7.9 scrub starts 2024-04-20T17:39:02.815 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:02.815 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:02.815 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:02 smithi005 ceph-mon[39562]: pgmap v1892: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.7 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-04-20T17:39:02.818 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:02 smithi106 ceph-mon[44423]: 7.8 deep-scrub starts 2024-04-20T17:39:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:02 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1897526864' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:39:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:02 smithi106 ceph-mon[44423]: 7.9 scrub starts 2024-04-20T17:39:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:02 smithi106 ceph-mon[44423]: pgmap v1892: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.7 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-04-20T17:39:03.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:02 smithi171 ceph-mon[44189]: 7.8 deep-scrub starts 2024-04-20T17:39:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:02 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1897526864' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:39:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:02 smithi171 ceph-mon[44189]: 7.9 scrub starts 2024-04-20T17:39:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:03.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:02 smithi171 ceph-mon[44189]: pgmap v1892: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.7 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-04-20T17:39:03.400 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:39:03.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:03.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:39:03.512 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:39:03.539 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:03.560 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:39:03.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:03.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:39:03.576 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 55 16 114429 114413 129.972 61.75 0.0202531 0.00769118 2024-04-20T17:39:03.663 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:39:03.753 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:03.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:03.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:03.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:03.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:03.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:03.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:03.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:03.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:03.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:03.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:03 smithi005 ceph-mon[39562]: 7.5 scrub starts 2024-04-20T17:39:03.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:03 smithi005 ceph-mon[39562]: 7.6 scrub starts 2024-04-20T17:39:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:04.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:04.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:04.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:03 smithi106 ceph-mon[44423]: 7.5 scrub starts 2024-04-20T17:39:04.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:03 smithi106 ceph-mon[44423]: 7.6 scrub starts 2024-04-20T17:39:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:04.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:03 smithi171 ceph-mon[44189]: 7.5 scrub starts 2024-04-20T17:39:04.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:03 smithi171 ceph-mon[44189]: 7.6 scrub starts 2024-04-20T17:39:04.102 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:39:04.297 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:39:04.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:04.298 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:39:04.399 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:39:04.436 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:04.497 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:39:04.516 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:39:04.541 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:39:04.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:04.547 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:39:04.578 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 56 16 115302 115286 128.623 54.5625 0.00850805 0.00777302 2024-04-20T17:39:04.648 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:39:04.675 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:04.695 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:39:04.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:04.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:39:04.801 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:39:04.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:04 smithi171 ceph-mon[44189]: 7.9 scrub ok 2024-04-20T17:39:04.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:04 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:39:04.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:04 smithi171 ceph-mon[44189]: 7.7 scrub starts 2024-04-20T17:39:04.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:04 smithi171 ceph-mon[44189]: 7.6 scrub ok 2024-04-20T17:39:04.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:04 smithi171 ceph-mon[44189]: pgmap v1893: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-04-20T17:39:04.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:04 smithi171 ceph-mon[44189]: from='client.27263 v1:172.21.15.5:0/2935129497' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:39:04.848 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:04 smithi005 ceph-mon[39562]: 7.9 scrub ok 2024-04-20T17:39:04.848 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:04 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:39:04.848 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:04 smithi005 ceph-mon[39562]: 7.7 scrub starts 2024-04-20T17:39:04.848 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:04 smithi005 ceph-mon[39562]: 7.6 scrub ok 2024-04-20T17:39:04.849 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:04 smithi005 ceph-mon[39562]: pgmap v1893: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-04-20T17:39:04.849 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:04 smithi005 ceph-mon[39562]: from='client.27263 v1:172.21.15.5:0/2935129497' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:39:04.856 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:04.918 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:39:04.918 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:39:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:04 smithi106 ceph-mon[44423]: 7.9 scrub ok 2024-04-20T17:39:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:04 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:39:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:04 smithi106 ceph-mon[44423]: 7.7 scrub starts 2024-04-20T17:39:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:04 smithi106 ceph-mon[44423]: 7.6 scrub ok 2024-04-20T17:39:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:04 smithi106 ceph-mon[44423]: pgmap v1893: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-04-20T17:39:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:04 smithi106 ceph-mon[44423]: from='client.27263 v1:172.21.15.5:0/2935129497' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:39:05.449 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:39:05.578 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 57 16 116393 116377 127.563 68.1875 0.0268852 0.00783585 2024-04-20T17:39:05.922 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:05 smithi005 ceph-mon[39562]: 7.5 scrub ok 2024-04-20T17:39:05.922 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:05 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:39:05.922 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:05 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:39:05.922 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:05 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:39:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:05 smithi106 ceph-mon[44423]: 7.5 scrub ok 2024-04-20T17:39:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:05 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:39:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:05 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:39:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:05 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:39:06.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:05 smithi171 ceph-mon[44189]: 7.5 scrub ok 2024-04-20T17:39:06.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:05 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:39:06.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:05 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:39:06.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:05 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:39:06.363 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:39:06.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:06.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:39:06.435 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-20T17:39:06.469 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:39:06.524 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:06.578 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 58 16 117393 117377 126.442 62.5 0.0268705 0.00790667 2024-04-20T17:39:06.800 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:39:06.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:06.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:39:06.829 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:39:06.909 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:39:06.938 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:06.962 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:39:06.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:06.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:39:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:06 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:39:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:06 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:39:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:06 smithi106 ceph-mon[44423]: 7.7 scrub ok 2024-04-20T17:39:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:06 smithi106 ceph-mon[44423]: osdmap e715: 12 total, 12 up, 10 in 2024-04-20T17:39:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:06 smithi106 ceph-mon[44423]: 7.e scrub starts 2024-04-20T17:39:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:06 smithi106 ceph-mon[44423]: pgmap v1895: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-04-20T17:39:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:06 smithi106 ceph-mon[44423]: osdmap e716: 12 total, 12 up, 10 in 2024-04-20T17:39:07.065 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:39:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:06 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:39:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:06 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:39:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:06 smithi171 ceph-mon[44189]: 7.7 scrub ok 2024-04-20T17:39:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:06 smithi171 ceph-mon[44189]: osdmap e715: 12 total, 12 up, 10 in 2024-04-20T17:39:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:06 smithi171 ceph-mon[44189]: 7.e scrub starts 2024-04-20T17:39:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:06 smithi171 ceph-mon[44189]: pgmap v1895: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-04-20T17:39:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:06 smithi171 ceph-mon[44189]: osdmap e716: 12 total, 12 up, 10 in 2024-04-20T17:39:07.092 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:07.113 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:39:07.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:07.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:39:07.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:06 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:39:07.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:06 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:39:07.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:06 smithi005 ceph-mon[39562]: 7.7 scrub ok 2024-04-20T17:39:07.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:06 smithi005 ceph-mon[39562]: osdmap e715: 12 total, 12 up, 10 in 2024-04-20T17:39:07.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:06 smithi005 ceph-mon[39562]: 7.e scrub starts 2024-04-20T17:39:07.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:06 smithi005 ceph-mon[39562]: pgmap v1895: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-04-20T17:39:07.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:06 smithi005 ceph-mon[39562]: osdmap e716: 12 total, 12 up, 10 in 2024-04-20T17:39:07.217 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:39:07.243 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:07.262 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:39:07.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:07.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:39:07.367 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:39:07.394 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:07.414 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:39:07.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:07.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:39:07.518 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:39:07.545 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:07.566 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:39:07.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:07.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:39:07.578 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 59 16 118476 118460 125.446 67.6875 0.00534681 0.00796973 2024-04-20T17:39:07.671 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:39:07.698 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:07.717 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:39:07.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:07.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:39:07.820 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:39:07.848 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:07.868 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:39:07.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:07.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:39:07.972 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:39:08.006 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:07 smithi171 ceph-mon[44189]: 7.2 scrub starts 2024-04-20T17:39:08.010 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:07 smithi106 ceph-mon[44423]: 7.2 scrub starts 2024-04-20T17:39:08.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:07 smithi005 ceph-mon[39562]: 7.2 scrub starts 2024-04-20T17:39:08.578 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:39:08.579142 min lat: 0.00100541 max lat: 0.210579 avg lat: 0.00803231 2024-04-20T17:39:08.578 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:39:08.578 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 60 16 119515 119499 124.438 64.9375 0.0956486 0.00803231 2024-04-20T17:39:08.670 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:39:08.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:08.674 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:39:08.775 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:39:08.804 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:08.824 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:39:08.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:08.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:39:08.928 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:39:08.966 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:08 smithi171 ceph-mon[44189]: 7.e scrub ok 2024-04-20T17:39:08.966 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:08 smithi171 ceph-mon[44189]: pgmap v1897: 17 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 14 active+clean; 7.1 GiB data, 19 GiB used, 876 GiB / 894 GiB avail; 57 MiB/s wr, 919 op/s 2024-04-20T17:39:08.971 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:09.004 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:39:09.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:09.007 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:39:09.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:08 smithi106 ceph-mon[44423]: 7.e scrub ok 2024-04-20T17:39:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:08 smithi106 ceph-mon[44423]: pgmap v1897: 17 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 14 active+clean; 7.1 GiB data, 19 GiB used, 876 GiB / 894 GiB avail; 57 MiB/s wr, 919 op/s 2024-04-20T17:39:09.107 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:39:09.136 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:09.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:08 smithi005 ceph-mon[39562]: 7.e scrub ok 2024-04-20T17:39:09.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:08 smithi005 ceph-mon[39562]: pgmap v1897: 17 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 14 active+clean; 7.1 GiB data, 19 GiB used, 876 GiB / 894 GiB avail; 57 MiB/s wr, 919 op/s 2024-04-20T17:39:09.234 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:39:09.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:09.242 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:39:09.289 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:39:09.289 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:39:09.343 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:39:09.383 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:09.578 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 61 16 120632 120616 123.542 69.8125 0.00365095 0.00808808 2024-04-20T17:39:09.936 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:39:09.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:09.942 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:39:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:09 smithi106 ceph-mon[44423]: 7.2 scrub ok 2024-04-20T17:39:10.043 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:39:10.062 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:09 smithi171 ceph-mon[44189]: 7.2 scrub ok 2024-04-20T17:39:10.075 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:10.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:09 smithi005 ceph-mon[39562]: 7.2 scrub ok 2024-04-20T17:39:10.379 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:39:10.400 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-20T17:39:10.431 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:39:10.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:10.436 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:39:10.537 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:39:10.579 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:10.582 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 62 16 121735 121719 122.655 68.9375 0.0157868 0.00815082 2024-04-20T17:39:10.610 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:39:10.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:10.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:39:10.716 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:39:10.759 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:10.777 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:39:11.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:10 smithi106 ceph-mon[44423]: pgmap v1898: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 7.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-20T17:39:11.067 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:10 smithi171 ceph-mon[44189]: pgmap v1898: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 7.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-20T17:39:11.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:10 smithi005 ceph-mon[39562]: pgmap v1898: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 7.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-20T17:39:11.582 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 63 16 122922 122906 121.885 74.1875 0.0163892 0.00820299 2024-04-20T17:39:12.582 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 64 16 124160 124144 121.19 77.375 0.0101839 0.00825033 2024-04-20T17:39:13.135 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:39:13.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:13.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:39:13.243 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:39:13.270 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:13.289 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:39:13.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:13.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:39:13.333 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:39:13.334 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:39:13.393 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:39:13.420 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:13.440 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:39:13.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:13.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:39:13.544 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:39:13.582 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 65 16 125114 125098 120.243 59.625 0.0175126 0.00831017 2024-04-20T17:39:13.626 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:13.697 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:13 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:39:13.697 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:13 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:39:13.697 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:13 smithi171 ceph-mon[44189]: pgmap v1899: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-04-20T17:39:13.697 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:13 smithi171 ceph-mon[44189]: osdmap e717: 12 total, 12 up, 10 in 2024-04-20T17:39:13.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:13 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:39:13.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:13 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:39:13.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:13 smithi106 ceph-mon[44423]: pgmap v1899: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-04-20T17:39:13.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:13 smithi106 ceph-mon[44423]: osdmap e717: 12 total, 12 up, 10 in 2024-04-20T17:39:13.872 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:13 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:39:13.872 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:13 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:39:13.872 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:13 smithi005 ceph-mon[39562]: pgmap v1899: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-04-20T17:39:13.873 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:13 smithi005 ceph-mon[39562]: osdmap e717: 12 total, 12 up, 10 in 2024-04-20T17:39:13.901 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:39:13.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:13.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:39:13.915 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":1893,"stamp":"2024-04-20T17:39:04.468652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7359889952,"num_objects":112296,"num_object_clones":0,"num_object_copies":224592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":112296,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":112475,"num_write_kb":7189754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112431,"ondisk_log_size":112431,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":18957408,"kb_used_data":14323320,"kb_used_omap":224,"kb_used_meta":4633759,"kb_avail":918494112,"statfs":{"total":959950356480,"available":940537970688,"internally_reserved":0,"allocated":14667079680,"data_stored":14662409748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":229753,"internal_metadata":4744969863},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,6,5,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":647823360,"num_objects":9885,"num_object_clones":0,"num_object_copies":19770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9885,"num_write_kb":632640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002265"},"pg_stats":[{"pgid":"7.f","version":"713'6977","reported_seq":7024,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-04-20T17:38:59.789376+0000","last_change":"2024-04-20T17:37:56.836004+0000","last_active":"2024-04-20T17:38:59.789376+0000","last_peered":"2024-04-20T17:38:59.789376+0000","last_clean":"2024-04-20T17:38:59.789376+0000","last_became_active":"2024-04-20T17:37:56.835750+0000","last_became_peered":"2024-04-20T17:37:56.835750+0000","last_unstale":"2024-04-20T17:38:59.789376+0000","last_undegraded":"2024-04-20T17:38:59.789376+0000","last_fullsized":"2024-04-20T17:38:59.789376+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":6977,"log_dups_size":0,"ondisk_log_size":6977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:39:14.458226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457244672,"num_objects":6977,"num_object_clones":0,"num_object_copies":13954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6977,"num_write_kb":446528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"714'6946","reported_seq":6995,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-04-20T17:39:01.683397+0000","last_change":"2024-04-20T17:37:56.833945+0000","last_active":"2024-04-20T17:39:01.683397+0000","last_peered":"2024-04-20T17:39:01.683397+0000","last_clean":"2024-04-20T17:39:01.683397+0000","last_became_active":"2024-04-20T17:37:56.833703+0000","last_became_peered":"2024-04-20T17:37:56.833703+0000","last_unstale":"2024-04-20T17:39:01.683397+0000","last_undegraded":"2024-04-20T17:39:01.683397+0000","last_fullsized":"2024-04-20T17:39:01.683397+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":6946,"log_dups_size":0,"ondisk_log_size":6946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:39:04.033531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455213056,"num_objects":6946,"num_object_clones":0,"num_object_copies":13892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6946,"num_write_kb":444544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"713'7051","reported_seq":7098,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-04-20T17:38:59.789442+0000","last_change":"2024-04-20T17:37:56.836989+0000","last_active":"2024-04-20T17:38:59.789442+0000","last_peered":"2024-04-20T17:38:59.789442+0000","last_clean":"2024-04-20T17:38:59.789442+0000","last_became_active":"2024-04-20T17:37:56.836496+0000","last_became_peered":"2024-04-20T17:37:56.836496+0000","last_unstale":"2024-04-20T17:38:59.789442+0000","last_undegraded":"2024-04-20T17:38:59.789442+0000","last_fullsized":"2024-04-20T17:38:59.789442+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":7051,"log_dups_size":0,"ondisk_log_size":7051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:39:11.524384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462094336,"num_objects":7051,"num_object_clones":0,"num_object_copies":14102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7051,"num_write_kb":451264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.c","version":"714'7166","reported_seq":7215,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-04-20T17:39:04.001784+0000","last_change":"2024-04-20T17:37:56.834884+0000","last_active":"2024-04-20T17:39:04.001784+0000","last_peered":"2024-04-20T17:39:04.001784+0000","last_clean":"2024-04-20T17:39:04.001784+0000","last_became_active":"2024-04-20T17:37:56.834181+0000","last_became_peered":"2024-04-20T17:37:56.834181+0000","last_unstale":"2024-04-20T17:39:04.001784+0000","last_undegraded":"2024-04-20T17:39:04.001784+0000","last_fullsized":"2024-04-20T17:39:04.001784+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":7166,"log_dups_size":0,"ondisk_log_size":7166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:39:09.740495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":469630976,"num_objects":7166,"num_object_clones":0,"num_object_copies":14332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7166,"num_write_kb":458624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"714'7130","reported_seq":7179,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-04-20T17:39:03.960976+0000","last_change":"2024-04-20T17:37:56.830736+0000","last_active":"2024-04-20T17:39:03.960976+0000","last_peered":"2024-04-20T17:39:03.960976+0000","last_clean":"2024-04-20T17:39:03.960976+0000","last_became_active":"2024-04-20T17:37:56.830487+0000","last_became_peered":"2024-04-20T17:37:56.830487+0000","last_unstale":"2024-04-20T17:39:03.960976+0000","last_undegraded":"2024-04-20T17:39:03.960976+0000","last_fullsized":"2024-04-20T17:39:03.960976+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":7130,"log_dups_size":0,"ondisk_log_size":7130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:39:21.390993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":467271680,"num_objects":7130,"num_object_clones":0,"num_object_copies":14260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7130,"num_write_kb":456320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.a","version":"713'6909","reported_seq":6956,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-04-20T17:39:00.854650+0000","last_change":"2024-04-20T17:37:56.836237+0000","last_active":"2024-04-20T17:39:00.854650+0000","last_peered":"2024-04-20T17:39:00.854650+0000","last_clean":"2024-04-20T17:39:00.854650+0000","last_became_active":"2024-04-20T17:37:56.835944+0000","last_became_peered":"2024-04-20T17:37:56.835944+0000","last_unstale":"2024-04-20T17:39:00.854650+0000","last_undegraded":"2024-04-20T17:39:00.854650+0000","last_fullsized":"2024-04-20T17:39:00.854650+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":6909,"log_dups_size":0,"ondisk_log_size":6909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:39:22.130326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452788224,"num_objects":6909,"num_object_clones":0,"num_object_copies":13818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6909,"num_write_kb":442176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"713'6897","reported_seq":6944,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-04-20T17:38:59.801151+0000","last_change":"2024-04-20T17:37:56.835919+0000","last_active":"2024-04-20T17:38:59.801151+0000","last_peered":"2024-04-20T17:38:59.801151+0000","last_clean":"2024-04-20T17:38:59.801151+0000","last_became_active":"2024-04-20T17:37:56.835700+0000","last_became_peered":"2024-04-20T17:37:56.835700+0000","last_unstale":"2024-04-20T17:38:59.801151+0000","last_undegraded":"2024-04-20T17:38:59.801151+0000","last_fullsized":"2024-04-20T17:38:59.801151+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":6897,"log_dups_size":0,"ondisk_log_size":6897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452001792,"num_objects":6897,"num_object_clones":0,"num_object_copies":13794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6897,"num_write_kb":441408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11],"acting":[0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"714'6959","reported_seq":7008,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-04-20T17:39:01.716759+0000","last_change":"2024-04-20T17:37:56.914447+0000","last_active":"2024-04-20T17:39:01.716759+0000","last_peered":"2024-04-20T17:39:01.716759+0000","last_clean":"2024-04-20T17:39:01.716759+0000","last_became_active":"2024-04-20T17:37:56.914124+0000","last_became_peered":"2024-04-20T17:37:56.914124+0000","last_unstale":"2024-04-20T17:39:01.716759+0000","last_undegraded":"2024-04-20T17:39:01.716759+0000","last_fullsized":"2024-04-20T17:39:01.716759+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":6959,"log_dups_size":0,"ondisk_log_size":6959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456065024,"num_objects":6959,"num_object_clones":0,"num_object_copies":13918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6959,"num_write_kb":445376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"714'7228","reported_seq":7277,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-04-20T17:39:03.656305+0000","last_change":"2024-04-20T17:37:56.836954+0000","last_active":"2024-04-20T17:39:03.656305+0000","last_peered":"2024-04-20T17:39:03.656305+0000","last_clean":"2024-04-20T17:39:03.656305+0000","last_became_active":"2024-04-20T17:37:56.836665+0000","last_became_peered":"2024-04-20T17:37:56.836665+0000","last_unstale":"2024-04-20T17:39:03.656305+0000","last_undegraded":"2024-04-20T17:39:03.656305+0000","last_fullsized":"2024-04-20T17:39:03.656305+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":7228,"log_dups_size":0,"ondisk_log_size":7228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:39:03.851782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473694208,"num_objects":7228,"num_object_clones":0,"num_object_copies":14456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7228,"num_write_kb":462592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.5","version":"714'7203","reported_seq":7320,"reported_epoch":714,"state":"active+clean+scrubbing","last_fresh":"2024-04-20T17:39:04.010752+0000","last_change":"2024-04-20T17:39:02.957900+0000","last_active":"2024-04-20T17:39:04.010752+0000","last_peered":"2024-04-20T17:39:04.010752+0000","last_clean":"2024-04-20T17:39:04.010752+0000","last_became_active":"2024-04-20T17:37:56.833600+0000","last_became_peered":"2024-04-20T17:37:56.833600+0000","last_unstale":"2024-04-20T17:39:04.010752+0000","last_undegraded":"2024-04-20T17:39:04.010752+0000","last_fullsized":"2024-04-20T17:39:04.010752+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":6349,"log_size":7203,"log_dups_size":0,"ondisk_log_size":7203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472055808,"num_objects":7203,"num_object_clones":0,"num_object_copies":14406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7203,"num_write_kb":460992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.4","version":"713'6776","reported_seq":6820,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-04-20T17:38:59.724918+0000","last_change":"2024-04-20T17:37:56.834982+0000","last_active":"2024-04-20T17:38:59.724918+0000","last_peered":"2024-04-20T17:38:59.724918+0000","last_clean":"2024-04-20T17:38:59.724918+0000","last_became_active":"2024-04-20T17:37:56.834727+0000","last_became_peered":"2024-04-20T17:37:56.834727+0000","last_unstale":"2024-04-20T17:38:59.724918+0000","last_undegraded":"2024-04-20T17:38:59.724918+0000","last_fullsized":"2024-04-20T17:38:59.724918+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":6776,"log_dups_size":0,"ondisk_log_size":6776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:39:20.179880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444071936,"num_objects":6776,"num_object_clones":0,"num_object_copies":13552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6776,"num_write_kb":433664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"714'7094","reported_seq":7220,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-04-20T17:39:04.008641+0000","last_change":"2024-04-20T17:39:00.502710+0000","last_active":"2024-04-20T17:39:04.008641+0000","last_peered":"2024-04-20T17:39:04.008641+0000","last_clean":"2024-04-20T17:39:04.008641+0000","last_became_active":"2024-04-20T17:37:56.832911+0000","last_became_peered":"2024-04-20T17:37:56.832911+0000","last_unstale":"2024-04-20T17:39:04.008641+0000","last_undegraded":"2024-04-20T17:39:04.008641+0000","last_fullsized":"2024-04-20T17:39:04.008641+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'6831","last_scrub_stamp":"2024-04-20T17:39:00.502556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:00.502556+0000","objects_scrubbed":6782,"log_size":7094,"log_dups_size":0,"ondisk_log_size":7094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:40:05.804863+0000","scrub_duration":1584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464912384,"num_objects":7094,"num_object_clones":0,"num_object_copies":14188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7094,"num_write_kb":454016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"714'7284","reported_seq":7333,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-04-20T17:39:03.960689+0000","last_change":"2024-04-20T17:37:56.834572+0000","last_active":"2024-04-20T17:39:03.960689+0000","last_peered":"2024-04-20T17:39:03.960689+0000","last_clean":"2024-04-20T17:39:03.960689+0000","last_became_active":"2024-04-20T17:37:56.834292+0000","last_became_peered":"2024-04-20T17:37:56.834292+0000","last_unstale":"2024-04-20T17:39:03.960689+0000","last_undegraded":"2024-04-20T17:39:03.960689+0000","last_fullsized":"2024-04-20T17:39:03.960689+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":7284,"log_dups_size":0,"ondisk_log_size":7284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:39:06.608157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477364224,"num_objects":7284,"num_object_clones":0,"num_object_copies":14568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7284,"num_write_kb":466176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"713'6933","reported_seq":6980,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-04-20T17:39:00.661877+0000","last_change":"2024-04-20T17:37:56.836188+0000","last_active":"2024-04-20T17:39:00.661877+0000","last_peered":"2024-04-20T17:39:00.661877+0000","last_clean":"2024-04-20T17:39:00.661877+0000","last_became_active":"2024-04-20T17:37:56.835512+0000","last_became_peered":"2024-04-20T17:37:56.835512+0000","last_unstale":"2024-04-20T17:39:00.661877+0000","last_undegraded":"2024-04-20T17:39:00.661877+0000","last_fullsized":"2024-04-20T17:39:00.661877+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":6933,"log_dups_size":0,"ondisk_log_size":6933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:39:16.406056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454361088,"num_objects":6933,"num_object_clones":0,"num_object_copies":13866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6933,"num_write_kb":443712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"713'6931","reported_seq":6978,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-04-20T17:39:00.858496+0000","last_change":"2024-04-20T17:37:56.834274+0000","last_active":"2024-04-20T17:39:00.858496+0000","last_peered":"2024-04-20T17:39:00.858496+0000","last_clean":"2024-04-20T17:39:00.858496+0000","last_became_active":"2024-04-20T17:37:56.833972+0000","last_became_peered":"2024-04-20T17:37:56.833972+0000","last_unstale":"2024-04-20T17:39:00.858496+0000","last_undegraded":"2024-04-20T17:39:00.858496+0000","last_fullsized":"2024-04-20T17:39:00.858496+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":6931,"log_dups_size":0,"ondisk_log_size":6931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:39:13.669143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454230016,"num_objects":6931,"num_object_clones":0,"num_object_copies":13862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6931,"num_write_kb":443584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.6","version":"713'6810","reported_seq":6856,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-04-20T17:38:59.797735+0000","last_change":"2024-04-20T17:37:56.914636+0000","last_active":"2024-04-20T17:38:59.797735+0000","last_peered":"2024-04-20T17:38:59.797735+0000","last_clean":"2024-04-20T17:38:59.797735+0000","last_became_active":"2024-04-20T17:37:56.914210+0000","last_became_peered":"2024-04-20T17:37:56.914210+0000","last_unstale":"2024-04-20T17:38:59.797735+0000","last_undegraded":"2024-04-20T17:38:59.797735+0000","last_fullsized":"2024-04-20T17:38:59.797735+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":0,"log_size":6810,"log_dups_size":0,"ondisk_log_size":6810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:39:02.094846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446300160,"num_objects":6810,"num_object_clones":0,"num_object_copies":13620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6810,"num_write_kb":435840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1023,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-04-20T17:39:01.493414+0000","last_change":"2024-04-20T17:38:56.114498+0000","last_active":"2024-04-20T17:39:01.493414+0000","last_peered":"2024-04-20T17:39:01.493414+0000","last_clean":"2024-04-20T17:39:01.493414+0000","last_became_active":"2024-04-20T17:23:24.109202+0000","last_became_peered":"2024-04-20T17:23:24.109202+0000","last_unstale":"2024-04-20T17:39:01.493414+0000","last_undegraded":"2024-04-20T17:39:01.493414+0000","last_fullsized":"2024-04-20T17:39:01.493414+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:38:56.114399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:38:56.114399+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:39:58.350474+0000","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":7359299584,"num_objects":112294,"num_object_clones":0,"num_object_copies":224588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":112294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112294,"num_write_kb":7186816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14736228352,"data_stored":14736228352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112294,"ondisk_log_size":112294,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2115712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991633,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2508456,"kb_used_data":1837276,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":91236696,"statfs":{"total":95995035648,"available":93426376704,"internally_reserved":0,"allocated":1881370624,"data_stored":1880895930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":687274436},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253271,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1832260,"kb_used_data":1313788,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":91912892,"statfs":{"total":95995035648,"available":94118801408,"internally_reserved":0,"allocated":1345318912,"data_stored":1344859878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":530905536},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613020,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1871124,"kb_used_data":1330440,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":91874028,"statfs":{"total":95995035648,"available":94079004672,"internally_reserved":0,"allocated":1362370560,"data_stored":1361905492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":553646528},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809251,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1233436,"kb_used_data":923220,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":92511716,"statfs":{"total":95995035648,"available":94731997184,"internally_reserved":0,"allocated":945377280,"data_stored":944906158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":398,"seq":1709396984007,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1257256,"kb_used_data":894108,"kb_used_omap":6,"kb_used_meta":363129,"kb_avail":92487896,"statfs":{"total":95995035648,"available":94707605504,"internally_reserved":0,"allocated":915566592,"data_stored":915091918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6767,"internal_metadata":371844497},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":2297807503481,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":2830383448104,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738824,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2804292,"kb_used_data":2191420,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":90940860,"statfs":{"total":95995035648,"available":93123440640,"internally_reserved":0,"allocated":2244014080,"data_stored":2243555066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394115,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2044692,"kb_used_data":1761672,"kb_used_omap":202,"kb_used_meta":282805,"kb_avail":91700460,"statfs":{"total":95995035648,"available":93901271040,"internally_reserved":0,"allocated":1803952128,"data_stored":1803487060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":289592791},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653165,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2611616,"kb_used_data":2312660,"kb_used_omap":4,"kb_used_meta":298939,"kb_avail":91133536,"statfs":{"total":95995035648,"available":93320740864,"internally_reserved":0,"allocated":2368163840,"data_stored":2367692718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":306114247},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986613,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1291796,"kb_used_data":877448,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92453356,"statfs":{"total":95995035648,"available":94672236544,"internally_reserved":0,"allocated":898506752,"data_stored":898041684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790383,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1502480,"kb_used_data":881288,"kb_used_omap":1,"kb_used_meta":621182,"kb_avail":92242672,"statfs":{"total":95995035648,"available":94456496128,"internally_reserved":0,"allocated":902438912,"data_stored":901973844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":636090821},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2261843968,"data_stored":2261843968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1823473664,"data_stored":1823473664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2371158016,"data_stored":2371158016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":903348224,"data_stored":903348224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":905576448,"data_stored":905576448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":923926528,"data_stored":923926528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":950665216,"data_stored":950665216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1364852736,"data_stored":1364852736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1350959104,"data_stored":1350959104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1880424448,"data_stored":1880424448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:39:13.916 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-20T17:39:14.006 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:39:14.054 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:14.081 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:39:14.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:14.084 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:39:14.085 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-20T17:39:14.186 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:39:14.226 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:14.262 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:39:14.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:14.266 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:39:14.367 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:39:14.433 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:14.475 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:39:14.582 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 66 16 125947 125931 119.21 52.0625 0.00686127 0.00838646 2024-04-20T17:39:14.618 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:39:14.721 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:14 smithi005 ceph-mon[39562]: 7.d deep-scrub starts 2024-04-20T17:39:14.721 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:14 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/935907532' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:39:14.721 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:14 smithi005 ceph-mon[39562]: from='client.27269 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:39:14.782 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:14 smithi171 ceph-mon[44189]: 7.d deep-scrub starts 2024-04-20T17:39:14.783 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:14 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/935907532' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:39:14.783 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:14 smithi171 ceph-mon[44189]: from='client.27269 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:39:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:14 smithi106 ceph-mon[44423]: 7.d deep-scrub starts 2024-04-20T17:39:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:14 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/935907532' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:39:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:14 smithi106 ceph-mon[44423]: from='client.27269 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:39:15.298 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:39:15.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:15.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:39:15.403 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:39:15.438 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:15.520 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:15 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:39:15.520 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:15 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:39:15.520 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:15 smithi005 ceph-mon[39562]: pgmap v1901: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-04-20T17:39:15.520 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:15 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:39:15.520 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:15 smithi005 ceph-mon[39562]: from='client.27269 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:39:15.520 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:15 smithi005 ceph-mon[39562]: osdmap e718: 12 total, 12 up, 10 in 2024-04-20T17:39:15.582 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 67 16 126893 126877 118.314 59.125 0.0040008 0.00844994 2024-04-20T17:39:15.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:15 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:39:15.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:15 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:39:15.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:15 smithi171 ceph-mon[44189]: pgmap v1901: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-04-20T17:39:15.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:15 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:39:15.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:15 smithi171 ceph-mon[44189]: from='client.27269 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:39:15.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:15 smithi171 ceph-mon[44189]: osdmap e718: 12 total, 12 up, 10 in 2024-04-20T17:39:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:15 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:39:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:15 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:39:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:15 smithi106 ceph-mon[44423]: pgmap v1901: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.5 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-04-20T17:39:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:15 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:39:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:15 smithi106 ceph-mon[44423]: from='client.27269 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:39:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:15 smithi106 ceph-mon[44423]: osdmap e718: 12 total, 12 up, 10 in 2024-04-20T17:39:16.490 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:39:16.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:16.494 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:39:16.582 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 68 16 127789 127773 117.397 56 0.00441563 0.00851466 2024-04-20T17:39:16.595 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:39:16.607 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:39:16.627 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:16.647 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:39:16.693 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:39:16.743 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:39:16.756 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:39:16.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:16.759 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:39:16.766 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:39:16.804 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:39:16.845 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:39:16.860 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:39:16.884 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:39:16.886 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:16.910 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:39:16.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:16.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:39:16.917 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:39:16.950 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:39:16.986 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:39:17.013 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:39:17.019 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:39:17.054 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:39:17.054 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:39:17.054 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:39:17.055 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:39:17.055 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:39:17.055 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:39:17.055 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:39:17.055 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:39:17.055 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:39:17.055 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:39:17.055 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:39:17.055 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:39:17.081 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:17.105 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:39:17.105 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:39:17.105 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:39:17.105 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:39:17.105 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.105 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.26882.0:313715 6.8 e679/619 6:15247d69:::benchmark_data_smithi171_147972_object152617:head v 679'19417, mlcod=679'19411)", 2024-04-20T17:39:17.105 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:37:25.319461+0000", 2024-04-20T17:39:17.106 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 111.78343142600001, 2024-04-20T17:39:17.106 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.086674556999999999, 2024-04-20T17:39:17.106 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:39:17.106 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:39:17.106 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:17.106 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:39:17.106 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.106 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:39:17.106 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.319461+0000", 2024-04-20T17:39:17.106 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.106 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.106 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.106 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:39:17.106 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.319461+0000", 2024-04-20T17:39:17.107 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.107 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.107 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.107 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:39:17.107 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.319462+0000", 2024-04-20T17:39:17.107 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9999999999999997e-07 2024-04-20T17:39:17.107 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.107 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.107 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:39:17.107 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.319467+0000", 2024-04-20T17:39:17.107 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7520000000000004e-06 2024-04-20T17:39:17.107 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.107 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.107 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:39:17.107 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.319472+0000", 2024-04-20T17:39:17.108 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5210000000000002e-06 2024-04-20T17:39:17.108 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.108 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.108 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:39:17.108 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.319477+0000", 2024-04-20T17:39:17.108 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8250000000000004e-06 2024-04-20T17:39:17.108 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.108 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.108 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:39:17.108 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.319539+0000", 2024-04-20T17:39:17.108 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1933000000000005e-05 2024-04-20T17:39:17.108 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.108 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.108 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:39:17.109 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.319556+0000", 2024-04-20T17:39:17.109 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7609e-05 2024-04-20T17:39:17.109 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.109 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.109 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:39:17.109 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.406086+0000", 2024-04-20T17:39:17.109 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.086529668000000004 2024-04-20T17:39:17.109 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.109 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.109 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:39:17.109 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.406136+0000", 2024-04-20T17:39:17.109 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9648999999999998e-05 2024-04-20T17:39:17.109 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.109 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:39:17.110 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.110 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.110 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.110 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.26882.0:313740 6.8 e679/619 6:109323e2:::benchmark_data_smithi171_147972_object152642:head v 679'19418, mlcod=679'19411)", 2024-04-20T17:39:17.110 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:37:25.338547+0000", 2024-04-20T17:39:17.110 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 111.76434544999999, 2024-04-20T17:39:17.110 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.067624529000000003, 2024-04-20T17:39:17.110 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:39:17.110 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:39:17.110 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:17.110 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:39:17.110 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.110 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:39:17.110 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.338547+0000", 2024-04-20T17:39:17.111 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.111 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.111 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.111 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:39:17.111 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.338547+0000", 2024-04-20T17:39:17.111 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.111 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.111 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.111 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:39:17.111 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.338549+0000", 2024-04-20T17:39:17.111 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7129999999999999e-06 2024-04-20T17:39:17.111 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.111 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.111 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:39:17.111 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.338560+0000", 2024-04-20T17:39:17.112 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0908e-05 2024-04-20T17:39:17.112 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.112 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.112 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:39:17.112 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.338570+0000", 2024-04-20T17:39:17.112 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0659000000000001e-05 2024-04-20T17:39:17.112 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.112 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.112 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:39:17.112 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.338579+0000", 2024-04-20T17:39:17.112 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1019999999999997e-06 2024-04-20T17:39:17.112 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.112 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.112 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:39:17.112 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.338638+0000", 2024-04-20T17:39:17.113 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9231000000000003e-05 2024-04-20T17:39:17.113 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.113 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.113 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:39:17.113 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.338659+0000", 2024-04-20T17:39:17.113 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1506999999999999e-05 2024-04-20T17:39:17.113 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.113 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.113 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:39:17.113 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.406141+0000", 2024-04-20T17:39:17.113 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.067482217999999997 2024-04-20T17:39:17.113 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.113 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.113 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:39:17.113 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.406172+0000", 2024-04-20T17:39:17.114 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0190999999999999e-05 2024-04-20T17:39:17.114 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.114 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:39:17.114 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.114 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.114 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.114 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.26882.0:313762 6.8 e679/619 6:15fed4ad:::benchmark_data_smithi171_147972_object152664:head v 679'19419, mlcod=679'19415)", 2024-04-20T17:39:17.114 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:37:25.356126+0000", 2024-04-20T17:39:17.114 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 111.746766261, 2024-04-20T17:39:17.114 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.050072311000000001, 2024-04-20T17:39:17.114 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:39:17.114 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:39:17.114 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:17.114 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:39:17.115 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.115 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:39:17.115 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.356126+0000", 2024-04-20T17:39:17.115 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.115 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.115 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.115 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:39:17.115 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.356126+0000", 2024-04-20T17:39:17.115 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.115 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.115 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.115 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:39:17.115 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.356128+0000", 2024-04-20T17:39:17.115 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1409999999999999e-06 2024-04-20T17:39:17.115 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.116 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.116 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:39:17.116 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.356140+0000", 2024-04-20T17:39:17.116 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1635e-05 2024-04-20T17:39:17.116 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.116 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.116 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:39:17.116 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.356151+0000", 2024-04-20T17:39:17.116 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0995000000000001e-05 2024-04-20T17:39:17.116 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.116 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.116 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:39:17.116 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.356160+0000", 2024-04-20T17:39:17.116 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4409999999999993e-06 2024-04-20T17:39:17.116 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.116 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.117 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:39:17.117 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.356224+0000", 2024-04-20T17:39:17.117 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4292e-05 2024-04-20T17:39:17.117 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.117 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.117 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:39:17.117 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.356245+0000", 2024-04-20T17:39:17.117 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1515e-05 2024-04-20T17:39:17.117 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.117 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.117 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:39:17.117 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.406175+0000", 2024-04-20T17:39:17.117 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.049929603000000003 2024-04-20T17:39:17.117 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.117 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.118 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:39:17.118 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.406199+0000", 2024-04-20T17:39:17.118 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3689e-05 2024-04-20T17:39:17.118 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.118 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:39:17.118 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.118 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.118 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.118 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:53991 7.5 7:a91bcedb:::benchmark_data_smithi171_161871_object53990:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e702)", 2024-04-20T17:39:17.118 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:25.779923+0000", 2024-04-20T17:39:17.118 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 51.322970046999998, 2024-04-20T17:39:17.119 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.069622137000000001, 2024-04-20T17:39:17.119 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:39:17.119 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:39:17.119 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:17.119 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:39:17.119 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:39:17.119 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:39:17.119 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 53991 2024-04-20T17:39:17.119 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.119 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:39:17.119 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.119 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:39:17.119 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.779923+0000", 2024-04-20T17:39:17.119 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.119 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.120 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.120 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:39:17.120 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.779923+0000", 2024-04-20T17:39:17.120 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.120 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.120 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.120 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:39:17.120 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.779924+0000", 2024-04-20T17:39:17.120 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.457e-06 2024-04-20T17:39:17.120 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.120 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.120 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:39:17.120 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.779947+0000", 2024-04-20T17:39:17.120 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3433000000000001e-05 2024-04-20T17:39:17.120 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.121 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.121 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:39:17.121 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.779958+0000", 2024-04-20T17:39:17.121 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0385e-05 2024-04-20T17:39:17.121 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.121 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.121 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:39:17.121 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.779965+0000", 2024-04-20T17:39:17.121 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6029999999999999e-06 2024-04-20T17:39:17.121 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.121 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.121 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:39:17.121 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.781542+0000", 2024-04-20T17:39:17.121 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0015767000000000001 2024-04-20T17:39:17.121 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.122 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.122 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:39:17.122 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.781682+0000", 2024-04-20T17:39:17.122 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014012700000000001 2024-04-20T17:39:17.122 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.122 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.122 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:39:17.122 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.781815+0000", 2024-04-20T17:39:17.122 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013292299999999999 2024-04-20T17:39:17.122 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.122 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.122 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:17.122 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.785249+0000", 2024-04-20T17:39:17.122 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.003433827 2024-04-20T17:39:17.122 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.123 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.123 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:39:17.123 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.849466+0000", 2024-04-20T17:39:17.123 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.06421694 2024-04-20T17:39:17.123 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.123 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.123 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:39:17.123 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.849508+0000", 2024-04-20T17:39:17.123 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1569999999999997e-05 2024-04-20T17:39:17.123 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.123 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.123 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:39:17.123 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.849545+0000", 2024-04-20T17:39:17.123 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7172000000000003e-05 2024-04-20T17:39:17.123 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.124 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:39:17.124 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.124 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.124 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.124 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:53992 7.5 7:ac4011ec:::benchmark_data_smithi171_161871_object53991:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e702)", 2024-04-20T17:39:17.124 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:25.781268+0000", 2024-04-20T17:39:17.124 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 51.321624604999997, 2024-04-20T17:39:17.124 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.068393861, 2024-04-20T17:39:17.124 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:39:17.124 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:39:17.124 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:17.124 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:39:17.124 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:39:17.124 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:39:17.125 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 53992 2024-04-20T17:39:17.125 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.125 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:39:17.125 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.125 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:39:17.125 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.781268+0000", 2024-04-20T17:39:17.125 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.125 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.125 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.125 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:39:17.125 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.781268+0000", 2024-04-20T17:39:17.126 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.126 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.126 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.126 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:39:17.126 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.781270+0000", 2024-04-20T17:39:17.126 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0700000000000001e-06 2024-04-20T17:39:17.126 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.126 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.126 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:39:17.126 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.781296+0000", 2024-04-20T17:39:17.126 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6089999999999999e-05 2024-04-20T17:39:17.126 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.126 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.126 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:39:17.126 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.781307+0000", 2024-04-20T17:39:17.127 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1034e-05 2024-04-20T17:39:17.127 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.127 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.127 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:39:17.127 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.781315+0000", 2024-04-20T17:39:17.127 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2849999999999995e-06 2024-04-20T17:39:17.127 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.127 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.127 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:39:17.127 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.783356+0000", 2024-04-20T17:39:17.127 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0020403069999999999 2024-04-20T17:39:17.127 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.127 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.127 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:39:17.128 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.783427+0000", 2024-04-20T17:39:17.128 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1161000000000005e-05 2024-04-20T17:39:17.128 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.128 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.128 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:39:17.128 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.783549+0000", 2024-04-20T17:39:17.128 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000122187 2024-04-20T17:39:17.128 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.128 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.128 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:17.128 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.849585+0000", 2024-04-20T17:39:17.128 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.066036279000000003 2024-04-20T17:39:17.128 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.128 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.128 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:39:17.129 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.849596+0000", 2024-04-20T17:39:17.129 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0464e-05 2024-04-20T17:39:17.129 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.129 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.129 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:39:17.129 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.849632+0000", 2024-04-20T17:39:17.129 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5864000000000001e-05 2024-04-20T17:39:17.129 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.129 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.129 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:39:17.129 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.849662+0000", 2024-04-20T17:39:17.129 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.012e-05 2024-04-20T17:39:17.129 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.129 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:39:17.129 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.130 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.130 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.130 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:53994 7.b 7:dab5472f:::benchmark_data_smithi171_161871_object53993:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e702)", 2024-04-20T17:39:17.130 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:25.782427+0000", 2024-04-20T17:39:17.130 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 51.320465945000002, 2024-04-20T17:39:17.130 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.065597904999999998, 2024-04-20T17:39:17.130 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:39:17.130 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:39:17.130 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:17.130 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:39:17.130 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:39:17.130 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:39:17.130 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 53994 2024-04-20T17:39:17.130 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.130 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:39:17.131 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.131 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:39:17.131 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782427+0000", 2024-04-20T17:39:17.131 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.131 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.131 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.131 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:39:17.131 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782427+0000", 2024-04-20T17:39:17.131 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.131 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.131 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.131 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:39:17.131 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782428+0000", 2024-04-20T17:39:17.131 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6640000000000001e-06 2024-04-20T17:39:17.131 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.132 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.132 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:39:17.132 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782450+0000", 2024-04-20T17:39:17.132 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1460000000000001e-05 2024-04-20T17:39:17.132 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.132 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.132 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:39:17.132 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782459+0000", 2024-04-20T17:39:17.132 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6169999999999996e-06 2024-04-20T17:39:17.132 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.132 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.132 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:39:17.132 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782466+0000", 2024-04-20T17:39:17.132 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9539999999999998e-06 2024-04-20T17:39:17.133 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.133 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.133 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:39:17.133 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782569+0000", 2024-04-20T17:39:17.133 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010242300000000001 2024-04-20T17:39:17.133 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.133 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.133 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:39:17.133 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782682+0000", 2024-04-20T17:39:17.133 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000113454 2024-04-20T17:39:17.133 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.133 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.134 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:39:17.134 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782802+0000", 2024-04-20T17:39:17.134 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000119993 2024-04-20T17:39:17.134 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.134 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.134 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:39:17.134 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.847413+0000", 2024-04-20T17:39:17.134 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.064610483999999996 2024-04-20T17:39:17.134 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.134 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.134 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:17.134 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.847932+0000", 2024-04-20T17:39:17.134 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00051915299999999995 2024-04-20T17:39:17.134 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.135 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.135 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:39:17.135 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.847982+0000", 2024-04-20T17:39:17.135 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9793000000000003e-05 2024-04-20T17:39:17.135 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.135 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.135 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:39:17.135 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.848025+0000", 2024-04-20T17:39:17.135 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2910000000000001e-05 2024-04-20T17:39:17.135 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.135 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:39:17.135 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.135 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.135 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.135 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:53997 7.b 7:d526e560:::benchmark_data_smithi171_161871_object53996:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e702)", 2024-04-20T17:39:17.136 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:25.782484+0000", 2024-04-20T17:39:17.136 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 51.320408839000002, 2024-04-20T17:39:17.136 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.065644577999999995, 2024-04-20T17:39:17.136 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:39:17.136 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:39:17.136 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:17.136 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:39:17.136 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:39:17.136 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:39:17.136 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 53997 2024-04-20T17:39:17.136 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.136 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:39:17.136 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.136 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:39:17.137 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782484+0000", 2024-04-20T17:39:17.137 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.137 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.137 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.137 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:39:17.137 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782484+0000", 2024-04-20T17:39:17.137 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.137 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.137 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.137 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:39:17.137 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782485+0000", 2024-04-20T17:39:17.137 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6300000000000004e-07 2024-04-20T17:39:17.137 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.137 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.137 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:39:17.138 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782508+0000", 2024-04-20T17:39:17.138 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3625000000000002e-05 2024-04-20T17:39:17.138 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.138 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.138 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:39:17.138 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782514+0000", 2024-04-20T17:39:17.138 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1739999999999997e-06 2024-04-20T17:39:17.138 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.138 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.138 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:39:17.138 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782520+0000", 2024-04-20T17:39:17.138 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2000000000000002e-06 2024-04-20T17:39:17.138 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.138 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.138 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:39:17.139 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.784440+0000", 2024-04-20T17:39:17.139 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001920539 2024-04-20T17:39:17.139 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.139 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.139 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:39:17.139 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.784611+0000", 2024-04-20T17:39:17.139 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00017075200000000001 2024-04-20T17:39:17.139 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.139 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.139 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:39:17.139 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.784752+0000", 2024-04-20T17:39:17.139 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014117200000000001 2024-04-20T17:39:17.139 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.139 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.139 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:39:17.140 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.847421+0000", 2024-04-20T17:39:17.140 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.062669425000000001 2024-04-20T17:39:17.140 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.140 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.140 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:17.140 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.848059+0000", 2024-04-20T17:39:17.140 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00063743100000000004 2024-04-20T17:39:17.140 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.140 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.140 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:39:17.140 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.848098+0000", 2024-04-20T17:39:17.140 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9391000000000003e-05 2024-04-20T17:39:17.140 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.140 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.140 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:39:17.141 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.848128+0000", 2024-04-20T17:39:17.141 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0105999999999999e-05 2024-04-20T17:39:17.141 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.141 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:39:17.141 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.141 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.141 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.141 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:53999 7.5 7:ae11ed78:::benchmark_data_smithi171_161871_object53998:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e702)", 2024-04-20T17:39:17.141 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:25.784882+0000", 2024-04-20T17:39:17.141 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 51.318010544000003, 2024-04-20T17:39:17.141 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.070871580000000003, 2024-04-20T17:39:17.141 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:39:17.141 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:39:17.142 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:17.142 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:39:17.142 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:39:17.142 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:39:17.142 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 53999 2024-04-20T17:39:17.142 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.142 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:39:17.142 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.142 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:39:17.142 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.784882+0000", 2024-04-20T17:39:17.142 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.142 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.142 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.142 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:39:17.142 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.784882+0000", 2024-04-20T17:39:17.143 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.143 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.143 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.143 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:39:17.143 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.784884+0000", 2024-04-20T17:39:17.143 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7090000000000001e-06 2024-04-20T17:39:17.143 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.143 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.143 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:39:17.143 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.784906+0000", 2024-04-20T17:39:17.143 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2039e-05 2024-04-20T17:39:17.143 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.143 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.143 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:39:17.144 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.784915+0000", 2024-04-20T17:39:17.144 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9840000000000007e-06 2024-04-20T17:39:17.144 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.144 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.144 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:39:17.144 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.784922+0000", 2024-04-20T17:39:17.144 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4030000000000001e-06 2024-04-20T17:39:17.144 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.144 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.144 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:39:17.144 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.785285+0000", 2024-04-20T17:39:17.144 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00036267800000000001 2024-04-20T17:39:17.144 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.144 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.144 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:39:17.144 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.847340+0000", 2024-04-20T17:39:17.145 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.062055297000000002 2024-04-20T17:39:17.145 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.145 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.145 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:39:17.145 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.847512+0000", 2024-04-20T17:39:17.145 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00017150499999999999 2024-04-20T17:39:17.145 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.145 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.145 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:39:17.145 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.853827+0000", 2024-04-20T17:39:17.145 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0063151709999999996 2024-04-20T17:39:17.145 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.145 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.145 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:17.145 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.855653+0000", 2024-04-20T17:39:17.146 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0018264519999999999 2024-04-20T17:39:17.146 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.146 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.146 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:39:17.146 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.855709+0000", 2024-04-20T17:39:17.146 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5624000000000001e-05 2024-04-20T17:39:17.146 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.146 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.146 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:39:17.146 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.855754+0000", 2024-04-20T17:39:17.146 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4718000000000001e-05 2024-04-20T17:39:17.146 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.146 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:39:17.146 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.146 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.147 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.147 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:54007 7.b 7:df96cf41:::benchmark_data_smithi171_161871_object54006:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e702)", 2024-04-20T17:39:17.147 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:25.785411+0000", 2024-04-20T17:39:17.147 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 51.317481403999999, 2024-04-20T17:39:17.147 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.067145148000000002, 2024-04-20T17:39:17.147 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:39:17.147 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:39:17.147 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:17.147 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:39:17.147 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:39:17.147 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:39:17.147 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 54007 2024-04-20T17:39:17.147 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.148 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:39:17.148 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.148 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:39:17.148 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.785411+0000", 2024-04-20T17:39:17.148 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.148 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.148 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.148 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:39:17.148 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.785411+0000", 2024-04-20T17:39:17.148 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.148 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.148 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.148 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:39:17.148 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.785413+0000", 2024-04-20T17:39:17.148 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3599999999999999e-06 2024-04-20T17:39:17.149 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.149 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.149 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:39:17.149 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.785435+0000", 2024-04-20T17:39:17.149 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2212e-05 2024-04-20T17:39:17.149 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.149 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.149 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:39:17.149 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.847536+0000", 2024-04-20T17:39:17.149 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.062101330000000003 2024-04-20T17:39:17.149 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.149 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.149 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:39:17.149 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.847545+0000", 2024-04-20T17:39:17.149 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8419999999999994e-06 2024-04-20T17:39:17.150 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.150 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.150 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:39:17.150 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.848154+0000", 2024-04-20T17:39:17.150 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00060890200000000003 2024-04-20T17:39:17.150 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.150 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.150 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:39:17.150 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.848282+0000", 2024-04-20T17:39:17.150 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012847299999999999 2024-04-20T17:39:17.150 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.150 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.150 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:39:17.150 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.848405+0000", 2024-04-20T17:39:17.151 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012244299999999999 2024-04-20T17:39:17.151 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.151 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.151 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:17.151 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.852027+0000", 2024-04-20T17:39:17.151 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.003622063 2024-04-20T17:39:17.151 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.151 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.151 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:39:17.151 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.852428+0000", 2024-04-20T17:39:17.151 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000401297 2024-04-20T17:39:17.151 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.151 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.151 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:39:17.151 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.852497+0000", 2024-04-20T17:39:17.152 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9077000000000003e-05 2024-04-20T17:39:17.152 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.152 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.152 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:39:17.152 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.852556+0000", 2024-04-20T17:39:17.152 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9148999999999999e-05 2024-04-20T17:39:17.152 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.152 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:39:17.152 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.152 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.152 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.152 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.27088.0:54025 7.c e702/694 7:32242b18:::benchmark_data_smithi171_161871_object54024:head v 702'3348, mlcod=702'3346)", 2024-04-20T17:39:17.152 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:25.796222+0000", 2024-04-20T17:39:17.152 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 51.306670548, 2024-04-20T17:39:17.153 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.056523057000000002, 2024-04-20T17:39:17.153 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:39:17.153 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:39:17.153 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:17.153 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:39:17.153 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.153 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:39:17.153 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.796222+0000", 2024-04-20T17:39:17.153 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.153 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.153 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.153 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:39:17.153 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.796222+0000", 2024-04-20T17:39:17.153 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.153 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.153 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.154 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:39:17.154 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.796226+0000", 2024-04-20T17:39:17.154 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8e-06 2024-04-20T17:39:17.154 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.154 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.154 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:39:17.154 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.796301+0000", 2024-04-20T17:39:17.154 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4814000000000006e-05 2024-04-20T17:39:17.154 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.154 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.154 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:39:17.154 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.847347+0000", 2024-04-20T17:39:17.154 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.051046455999999997 2024-04-20T17:39:17.154 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.155 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.155 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:39:17.155 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.847358+0000", 2024-04-20T17:39:17.155 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0852999999999999e-05 2024-04-20T17:39:17.155 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.155 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.155 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:39:17.155 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.847463+0000", 2024-04-20T17:39:17.155 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010525400000000001 2024-04-20T17:39:17.155 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.155 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.155 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:39:17.155 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.847487+0000", 2024-04-20T17:39:17.155 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4015e-05 2024-04-20T17:39:17.155 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.155 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.156 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:39:17.156 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.852700+0000", 2024-04-20T17:39:17.156 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0052130459999999998 2024-04-20T17:39:17.156 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.156 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.156 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:39:17.156 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.852745+0000", 2024-04-20T17:39:17.156 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4818999999999999e-05 2024-04-20T17:39:17.156 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.156 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:39:17.156 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.156 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.156 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.156 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:72010 7.5 7:ae102a2d:::benchmark_data_smithi171_161871_object72009:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e705)", 2024-04-20T17:39:17.156 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:34.873983+0000", 2024-04-20T17:39:17.157 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 42.228909539, 2024-04-20T17:39:17.157 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.059086393000000001, 2024-04-20T17:39:17.157 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:39:17.157 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:39:17.157 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:17.157 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:39:17.157 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:39:17.157 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:39:17.157 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 72010 2024-04-20T17:39:17.157 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.157 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:39:17.157 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.157 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:39:17.157 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.873983+0000", 2024-04-20T17:39:17.157 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.157 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.158 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.158 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:39:17.158 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.873983+0000", 2024-04-20T17:39:17.158 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.158 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.158 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.158 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:39:17.158 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.873985+0000", 2024-04-20T17:39:17.158 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5409999999999999e-06 2024-04-20T17:39:17.158 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.158 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.158 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:39:17.158 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874040+0000", 2024-04-20T17:39:17.158 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5649000000000002e-05 2024-04-20T17:39:17.159 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.159 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.159 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:39:17.159 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874050+0000", 2024-04-20T17:39:17.159 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3149999999999998e-06 2024-04-20T17:39:17.159 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.159 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.159 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:39:17.159 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874057+0000", 2024-04-20T17:39:17.159 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1019999999999999e-06 2024-04-20T17:39:17.159 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.159 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.159 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:39:17.159 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874114+0000", 2024-04-20T17:39:17.159 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6916999999999997e-05 2024-04-20T17:39:17.160 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.160 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.160 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:39:17.160 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874241+0000", 2024-04-20T17:39:17.160 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000127322 2024-04-20T17:39:17.160 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.160 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.160 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:39:17.160 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874375+0000", 2024-04-20T17:39:17.160 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000134055 2024-04-20T17:39:17.160 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.160 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.160 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:17.160 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.927913+0000", 2024-04-20T17:39:17.160 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.053538397000000001 2024-04-20T17:39:17.161 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.161 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.161 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:39:17.161 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.932975+0000", 2024-04-20T17:39:17.161 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0050614520000000001 2024-04-20T17:39:17.161 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.161 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.161 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:39:17.161 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.933024+0000", 2024-04-20T17:39:17.161 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9325e-05 2024-04-20T17:39:17.161 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.161 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.161 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:39:17.161 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.933069+0000", 2024-04-20T17:39:17.162 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5318000000000002e-05 2024-04-20T17:39:17.162 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.162 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:39:17.162 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.162 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.162 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.162 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:72011 7.b 7:dfd3059a:::benchmark_data_smithi171_161871_object72010:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e705)", 2024-04-20T17:39:17.162 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:34.874071+0000", 2024-04-20T17:39:17.162 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 42.228821492999998, 2024-04-20T17:39:17.162 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.058409599999999999, 2024-04-20T17:39:17.162 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:39:17.162 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:39:17.162 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:17.162 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:39:17.163 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:39:17.163 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:39:17.163 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 72011 2024-04-20T17:39:17.163 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.163 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:39:17.163 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.163 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:39:17.163 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874071+0000", 2024-04-20T17:39:17.163 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.163 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.163 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.163 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:39:17.163 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874071+0000", 2024-04-20T17:39:17.163 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.163 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.164 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.164 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:39:17.164 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874072+0000", 2024-04-20T17:39:17.164 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2800000000000005e-07 2024-04-20T17:39:17.164 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.164 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.164 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:39:17.164 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874117+0000", 2024-04-20T17:39:17.164 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5213000000000002e-05 2024-04-20T17:39:17.164 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.164 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.164 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:39:17.164 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874124+0000", 2024-04-20T17:39:17.164 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0720000000000002e-06 2024-04-20T17:39:17.165 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.165 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.165 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:39:17.165 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874129+0000", 2024-04-20T17:39:17.165 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7489999999999998e-06 2024-04-20T17:39:17.165 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.165 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.165 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:39:17.165 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874175+0000", 2024-04-20T17:39:17.165 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6023000000000003e-05 2024-04-20T17:39:17.165 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.165 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.165 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:39:17.165 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874290+0000", 2024-04-20T17:39:17.165 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000115072 2024-04-20T17:39:17.166 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.166 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.166 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:39:17.166 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874403+0000", 2024-04-20T17:39:17.166 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000113006 2024-04-20T17:39:17.166 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.166 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.166 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:17.166 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.929997+0000", 2024-04-20T17:39:17.166 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.055593548999999999 2024-04-20T17:39:17.166 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.166 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.166 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:39:17.167 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.932374+0000", 2024-04-20T17:39:17.167 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0023774590000000002 2024-04-20T17:39:17.167 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.167 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.167 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:39:17.167 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.932426+0000", 2024-04-20T17:39:17.167 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2280000000000001e-05 2024-04-20T17:39:17.167 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.167 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.167 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:39:17.167 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.932481+0000", 2024-04-20T17:39:17.167 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4249000000000002e-05 2024-04-20T17:39:17.167 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.167 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:39:17.167 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.168 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.168 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.168 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:72014 7.b 7:d5e7fe15:::benchmark_data_smithi171_161871_object72013:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e705)", 2024-04-20T17:39:17.168 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:34.874149+0000", 2024-04-20T17:39:17.168 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 42.228743754999996, 2024-04-20T17:39:17.168 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.058553926999999999, 2024-04-20T17:39:17.168 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:39:17.168 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:39:17.168 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:17.168 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:39:17.168 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:39:17.168 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:39:17.168 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 72014 2024-04-20T17:39:17.168 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.168 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:39:17.169 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.169 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:39:17.169 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874149+0000", 2024-04-20T17:39:17.169 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.169 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.169 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.169 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:39:17.169 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874149+0000", 2024-04-20T17:39:17.169 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.169 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.169 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.169 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:39:17.169 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874150+0000", 2024-04-20T17:39:17.169 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5199999999999995e-07 2024-04-20T17:39:17.170 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.170 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.170 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:39:17.170 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874188+0000", 2024-04-20T17:39:17.170 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8006999999999997e-05 2024-04-20T17:39:17.170 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.170 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.170 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:39:17.170 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874193+0000", 2024-04-20T17:39:17.170 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0119999999999996e-06 2024-04-20T17:39:17.170 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.170 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.170 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:39:17.170 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874200+0000", 2024-04-20T17:39:17.170 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9369999999999998e-06 2024-04-20T17:39:17.171 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.171 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.171 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:39:17.171 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.925710+0000", 2024-04-20T17:39:17.171 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.051510565000000001 2024-04-20T17:39:17.171 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.171 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.171 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:39:17.171 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.927891+0000", 2024-04-20T17:39:17.171 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0021803920000000002 2024-04-20T17:39:17.171 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.171 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.171 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:39:17.171 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.927988+0000", 2024-04-20T17:39:17.172 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7677999999999996e-05 2024-04-20T17:39:17.172 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.172 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.172 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:17.172 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.932539+0000", 2024-04-20T17:39:17.172 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0045502750000000003 2024-04-20T17:39:17.172 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.172 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.172 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:39:17.172 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.932616+0000", 2024-04-20T17:39:17.172 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7342999999999995e-05 2024-04-20T17:39:17.172 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.172 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.172 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:39:17.172 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.932663+0000", 2024-04-20T17:39:17.173 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6989e-05 2024-04-20T17:39:17.173 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.173 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.173 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:39:17.173 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.932703+0000", 2024-04-20T17:39:17.173 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9777000000000002e-05 2024-04-20T17:39:17.173 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.173 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:39:17.173 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.173 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.173 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.173 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop_reply(client.27088.0:72011 7.b e705/694 ondisk, result = 0)", 2024-04-20T17:39:17.173 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:34.876315+0000", 2024-04-20T17:39:17.173 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 42.226577931999998, 2024-04-20T17:39:17.173 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.053690789000000003, 2024-04-20T17:39:17.174 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:39:17.174 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:39:17.174 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:39:17.174 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:39:17.174 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.174 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:39:17.174 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.876315+0000", 2024-04-20T17:39:17.174 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.174 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.174 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.174 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:39:17.174 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.876315+0000", 2024-04-20T17:39:17.174 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.174 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.174 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.175 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:39:17.175 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.876316+0000", 2024-04-20T17:39:17.175 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4419999999999999e-06 2024-04-20T17:39:17.175 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.175 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.175 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:39:17.175 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.876321+0000", 2024-04-20T17:39:17.175 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6460000000000001e-06 2024-04-20T17:39:17.175 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.175 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.175 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:39:17.175 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.876332+0000", 2024-04-20T17:39:17.175 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0766e-05 2024-04-20T17:39:17.175 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.176 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.176 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:39:17.176 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.876339+0000", 2024-04-20T17:39:17.176 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6539999999999999e-06 2024-04-20T17:39:17.176 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.176 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.176 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:39:17.176 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.929984+0000", 2024-04-20T17:39:17.176 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.053644758000000001 2024-04-20T17:39:17.176 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.176 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.176 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:39:17.176 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.929991+0000", 2024-04-20T17:39:17.176 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2400000000000001e-06 2024-04-20T17:39:17.176 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.177 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.177 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:39:17.177 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.930005+0000", 2024-04-20T17:39:17.177 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4283e-05 2024-04-20T17:39:17.177 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.177 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:39:17.177 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.177 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.177 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.177 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop_reply(client.27088.0:72010 7.5 e705/694 ondisk, result = 0)", 2024-04-20T17:39:17.177 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:34.876339+0000", 2024-04-20T17:39:17.177 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 42.226553287999998, 2024-04-20T17:39:17.177 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.051582654999999998, 2024-04-20T17:39:17.177 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:39:17.177 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:39:17.178 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:39:17.178 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:39:17.178 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.178 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:39:17.178 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.876339+0000", 2024-04-20T17:39:17.178 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.178 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.178 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.178 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:39:17.178 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.876339+0000", 2024-04-20T17:39:17.178 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.178 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.178 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.178 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:39:17.178 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.876340+0000", 2024-04-20T17:39:17.179 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.155e-06 2024-04-20T17:39:17.179 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.179 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.179 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:39:17.179 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.876345+0000", 2024-04-20T17:39:17.179 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5410000000000002e-06 2024-04-20T17:39:17.179 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.179 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.179 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:39:17.179 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.876352+0000", 2024-04-20T17:39:17.179 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1269999999999997e-06 2024-04-20T17:39:17.179 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.179 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.179 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:39:17.179 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.876358+0000", 2024-04-20T17:39:17.180 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3709999999999999e-06 2024-04-20T17:39:17.180 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.180 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.180 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:39:17.180 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.927900+0000", 2024-04-20T17:39:17.180 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.051542566999999997 2024-04-20T17:39:17.180 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.180 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.180 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:39:17.180 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.927908+0000", 2024-04-20T17:39:17.180 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6029999999999999e-06 2024-04-20T17:39:17.180 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.180 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.180 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:39:17.180 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.927922+0000", 2024-04-20T17:39:17.180 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4290999999999999e-05 2024-04-20T17:39:17.181 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.181 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:39:17.181 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.181 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.181 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.181 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:72021 7.5 7:a792af76:::benchmark_data_smithi171_161871_object72020:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e705)", 2024-04-20T17:39:17.181 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:34.877239+0000", 2024-04-20T17:39:17.181 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 42.225653737000002, 2024-04-20T17:39:17.181 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.060395327999999998, 2024-04-20T17:39:17.181 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:39:17.181 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:39:17.181 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:17.181 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:39:17.181 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:39:17.181 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:39:17.182 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 72021 2024-04-20T17:39:17.182 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.182 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:39:17.182 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.182 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:39:17.182 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.877239+0000", 2024-04-20T17:39:17.182 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.182 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.182 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.182 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:39:17.182 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.877239+0000", 2024-04-20T17:39:17.182 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.182 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.182 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.182 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:39:17.183 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.877241+0000", 2024-04-20T17:39:17.183 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2280000000000001e-06 2024-04-20T17:39:17.183 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.183 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.183 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:39:17.183 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.877281+0000", 2024-04-20T17:39:17.183 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9765000000000003e-05 2024-04-20T17:39:17.183 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.183 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.183 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:39:17.183 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.926595+0000", 2024-04-20T17:39:17.183 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.049313726000000002 2024-04-20T17:39:17.183 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.183 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.183 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:39:17.184 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.926611+0000", 2024-04-20T17:39:17.184 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6154000000000002e-05 2024-04-20T17:39:17.184 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.184 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.184 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:39:17.184 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.927947+0000", 2024-04-20T17:39:17.184 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001335936 2024-04-20T17:39:17.184 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.184 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.184 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:39:17.184 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.928080+0000", 2024-04-20T17:39:17.184 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000133268 2024-04-20T17:39:17.184 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.184 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.184 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:39:17.185 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.928221+0000", 2024-04-20T17:39:17.185 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000140798 2024-04-20T17:39:17.185 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.185 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.185 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:17.185 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.933143+0000", 2024-04-20T17:39:17.185 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0049227079999999996 2024-04-20T17:39:17.185 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.185 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.185 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:39:17.185 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.937539+0000", 2024-04-20T17:39:17.185 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0043956869999999997 2024-04-20T17:39:17.185 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.185 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.185 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:39:17.186 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.937586+0000", 2024-04-20T17:39:17.186 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6527999999999999e-05 2024-04-20T17:39:17.186 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.186 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.186 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:39:17.186 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.937634+0000", 2024-04-20T17:39:17.186 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8529999999999998e-05 2024-04-20T17:39:17.186 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.186 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:39:17.186 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.186 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.186 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.186 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.27088.0:87138 7.c e709/694 7:3ac78cb4:::benchmark_data_smithi171_161871_object87137:head v 709'5492, mlcod=709'5489)", 2024-04-20T17:39:17.186 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:43.842984+0000", 2024-04-20T17:39:17.186 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 33.259908164000002, 2024-04-20T17:39:17.187 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.156380886, 2024-04-20T17:39:17.187 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:39:17.187 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:39:17.187 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:17.187 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:39:17.187 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.187 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:39:17.187 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.842984+0000", 2024-04-20T17:39:17.187 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.187 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.187 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.187 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:39:17.187 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.842984+0000", 2024-04-20T17:39:17.187 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.187 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.188 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.188 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:39:17.188 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.842986+0000", 2024-04-20T17:39:17.189 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7129999999999999e-06 2024-04-20T17:39:17.190 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.190 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.190 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:39:17.190 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.843020+0000", 2024-04-20T17:39:17.190 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4038999999999997e-05 2024-04-20T17:39:17.190 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.190 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.190 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:39:17.190 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.843029+0000", 2024-04-20T17:39:17.190 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0950000000000002e-06 2024-04-20T17:39:17.190 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.190 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.190 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:39:17.190 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.843036+0000", 2024-04-20T17:39:17.191 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0539999999999997e-06 2024-04-20T17:39:17.191 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.191 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.191 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:39:17.191 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.843087+0000", 2024-04-20T17:39:17.191 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0528999999999997e-05 2024-04-20T17:39:17.191 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.191 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.191 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:39:17.191 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.843105+0000", 2024-04-20T17:39:17.191 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8394999999999999e-05 2024-04-20T17:39:17.191 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.191 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.191 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:39:17.192 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.845666+0000", 2024-04-20T17:39:17.192 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0025611219999999999 2024-04-20T17:39:17.192 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.192 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.192 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:39:17.192 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.999365+0000", 2024-04-20T17:39:17.192 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15369893900000001 2024-04-20T17:39:17.192 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.192 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:39:17.192 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.192 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.192 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.192 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.27088.0:87132 7.9 e709/694 7:9eb80540:::benchmark_data_smithi171_161871_object87131:head v 709'5502, mlcod=709'5498)", 2024-04-20T17:39:17.192 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:43.843723+0000", 2024-04-20T17:39:17.192 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 33.259169843000002, 2024-04-20T17:39:17.193 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15567329499999999, 2024-04-20T17:39:17.193 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:39:17.193 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:39:17.193 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:17.193 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:39:17.193 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.193 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:39:17.193 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.843723+0000", 2024-04-20T17:39:17.193 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.193 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.193 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.193 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:39:17.193 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.843723+0000", 2024-04-20T17:39:17.193 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.193 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.194 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.194 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:39:17.194 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.843724+0000", 2024-04-20T17:39:17.194 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4330000000000001e-06 2024-04-20T17:39:17.194 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.194 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.194 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:39:17.194 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.843778+0000", 2024-04-20T17:39:17.194 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3871e-05 2024-04-20T17:39:17.194 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.194 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.194 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:39:17.194 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.843788+0000", 2024-04-20T17:39:17.194 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.4809999999999995e-06 2024-04-20T17:39:17.194 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.195 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.195 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:39:17.195 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.843795+0000", 2024-04-20T17:39:17.195 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7360000000000001e-06 2024-04-20T17:39:17.195 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.195 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.195 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:39:17.195 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.843837+0000", 2024-04-20T17:39:17.195 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1511000000000003e-05 2024-04-20T17:39:17.195 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.195 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.195 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:39:17.195 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.843855+0000", 2024-04-20T17:39:17.195 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8011000000000001e-05 2024-04-20T17:39:17.195 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.195 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.196 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:39:17.196 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.999345+0000", 2024-04-20T17:39:17.196 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.155490355 2024-04-20T17:39:17.196 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.196 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.196 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:39:17.196 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.999396+0000", 2024-04-20T17:39:17.196 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0896999999999997e-05 2024-04-20T17:39:17.196 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.196 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:39:17.196 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.196 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.196 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.196 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:87149 7.5 7:a7f5d85a:::benchmark_data_smithi171_161871_object87148:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e709)", 2024-04-20T17:39:17.197 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:43.846493+0000", 2024-04-20T17:39:17.197 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 33.256399408, 2024-04-20T17:39:17.197 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16122668900000001, 2024-04-20T17:39:17.197 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:39:17.197 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:39:17.197 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:17.197 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:39:17.197 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:39:17.197 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:39:17.197 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 87149 2024-04-20T17:39:17.197 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.197 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:39:17.197 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.197 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:39:17.197 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.846493+0000", 2024-04-20T17:39:17.198 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.198 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.198 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.198 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:39:17.198 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.846493+0000", 2024-04-20T17:39:17.198 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.198 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.198 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.198 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:39:17.198 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.846496+0000", 2024-04-20T17:39:17.198 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3499999999999999e-06 2024-04-20T17:39:17.198 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.198 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.198 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:39:17.198 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.846518+0000", 2024-04-20T17:39:17.199 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2320999999999999e-05 2024-04-20T17:39:17.199 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.199 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.199 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:39:17.199 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.999332+0000", 2024-04-20T17:39:17.199 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15281418299999999 2024-04-20T17:39:17.199 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.199 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.199 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:39:17.199 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.999348+0000", 2024-04-20T17:39:17.199 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6354e-05 2024-04-20T17:39:17.199 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.199 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.199 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:39:17.199 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.999404+0000", 2024-04-20T17:39:17.200 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5164000000000002e-05 2024-04-20T17:39:17.200 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.200 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.200 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:39:17.200 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.999551+0000", 2024-04-20T17:39:17.200 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014728500000000001 2024-04-20T17:39:17.200 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.200 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.200 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:39:17.200 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.999688+0000", 2024-04-20T17:39:17.200 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013765000000000001 2024-04-20T17:39:17.200 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.200 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.200 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:17.200 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:44.007610+0000", 2024-04-20T17:39:17.201 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0079218029999999998 2024-04-20T17:39:17.201 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.201 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.201 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:39:17.201 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:44.007667+0000", 2024-04-20T17:39:17.201 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6849999999999999e-05 2024-04-20T17:39:17.201 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.201 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.201 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:39:17.201 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:44.007694+0000", 2024-04-20T17:39:17.201 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7243e-05 2024-04-20T17:39:17.201 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.201 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.201 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:39:17.201 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:44.007720+0000", 2024-04-20T17:39:17.202 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5486e-05 2024-04-20T17:39:17.202 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.202 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:39:17.202 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.202 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.202 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.202 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.27088.0:87141 7.c e709/694 7:38ce718a:::benchmark_data_smithi171_161871_object87140:head v 709'5493, mlcod=709'5490)", 2024-04-20T17:39:17.202 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:43.847640+0000", 2024-04-20T17:39:17.202 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 33.255252828000003, 2024-04-20T17:39:17.202 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16894878399999999, 2024-04-20T17:39:17.202 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:39:17.202 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:39:17.202 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:17.202 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:39:17.203 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.203 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:39:17.203 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.847640+0000", 2024-04-20T17:39:17.203 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.203 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.203 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.203 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:39:17.203 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.847640+0000", 2024-04-20T17:39:17.203 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:39:17.203 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.203 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.203 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:39:17.203 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.847642+0000", 2024-04-20T17:39:17.203 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8899999999999999e-06 2024-04-20T17:39:17.203 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.204 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.204 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:39:17.204 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.847697+0000", 2024-04-20T17:39:17.204 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5318999999999997e-05 2024-04-20T17:39:17.204 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.204 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.204 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:39:17.204 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.999339+0000", 2024-04-20T17:39:17.204 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15164244499999999 2024-04-20T17:39:17.204 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.204 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.204 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:39:17.204 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.999350+0000", 2024-04-20T17:39:17.204 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0628999999999999e-05 2024-04-20T17:39:17.204 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.204 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.205 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:39:17.205 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.999404+0000", 2024-04-20T17:39:17.205 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3894000000000003e-05 2024-04-20T17:39:17.205 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.205 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.205 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:39:17.205 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.999425+0000", 2024-04-20T17:39:17.205 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0894999999999999e-05 2024-04-20T17:39:17.205 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.205 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.205 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:39:17.205 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:44.016535+0000", 2024-04-20T17:39:17.205 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.017110112 2024-04-20T17:39:17.205 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:39:17.205 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:39:17.205 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:39:17.206 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:44.016589+0000", 2024-04-20T17:39:17.206 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3600000000000002e-05 2024-04-20T17:39:17.206 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.206 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:39:17.206 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.206 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:39:17.206 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:39:17.206 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:39:17.363 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:39:17.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:17.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:39:17.474 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:39:17.502 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:17.525 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:39:17.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:17.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:39:17.582 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 69 16 128648 128632 116.474 53.6875 0.0102331 0.00858129 2024-04-20T17:39:17.629 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:39:17.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:17 smithi005 ceph-mon[39562]: pgmap v1903: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 7.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-04-20T17:39:17.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:17 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1161988889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:39:17.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:17.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:17.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:17.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:17.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:17.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:17.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:17.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:17.686 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:17.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:17 smithi106 ceph-mon[44423]: pgmap v1903: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 7.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-04-20T17:39:17.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:17 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1161988889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:39:17.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:17.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:17.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:17.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:17.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:17.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:17.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:17.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:17.791 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:17 smithi171 ceph-mon[44189]: pgmap v1903: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 7.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-04-20T17:39:17.791 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:17 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1161988889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:39:17.791 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:17.792 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:17.792 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:17.792 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:17.792 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:17.792 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:17.792 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:17.792 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:17.832 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-20T17:39:17.872 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:39:17.965 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:39:17.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:17.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:39:18.075 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:39:18.103 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:18.122 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:39:18.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:18.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:39:18.227 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:39:18.336 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:39:18.343 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:18.391 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:39:18.391 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:39:18.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:18.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:39:18.500 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:39:18.531 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:18.554 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:39:18.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:18.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:39:18.582 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 70 16 129443 129427 115.52 49.6875 0.0187022 0.00865365 2024-04-20T17:39:18.659 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:39:18.674 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:18.674 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:18.674 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:18.674 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:18 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:18.674 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:18 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:39:18.675 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:18 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:39:18.686 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:18.707 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:39:18.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:18.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:39:18.811 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:39:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:18 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:18 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:39:18.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:18 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:39:18.837 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:18.837 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:18.837 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:18.837 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:18 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:18.837 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:18 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:39:18.837 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:18 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:39:18.839 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:18.860 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:39:18.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:18.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:39:18.964 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:39:18.992 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:19.012 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:39:19.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:19.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:39:19.116 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:39:19.151 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:19.177 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:39:19.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:19.182 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:39:19.283 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:39:19.319 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:19.345 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:39:19.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:19.349 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:39:19.450 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:39:19.477 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:19.499 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:39:19.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:19.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:39:19.538 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:19 smithi171 ceph-mon[44189]: pgmap v1904: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 7.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 55 MiB/s wr, 875 op/s 2024-04-20T17:39:19.539 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:19 smithi171 ceph-mon[44189]: osdmap e719: 12 total, 12 up, 10 in 2024-04-20T17:39:19.582 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 71 16 130305 130289 114.652 53.875 0.00514146 0.00871814 2024-04-20T17:39:19.604 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:39:19.637 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:19.664 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:39:19.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:19.668 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:39:19.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:19 smithi106 ceph-mon[44423]: pgmap v1904: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 7.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 55 MiB/s wr, 875 op/s 2024-04-20T17:39:19.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:19 smithi106 ceph-mon[44423]: osdmap e719: 12 total, 12 up, 10 in 2024-04-20T17:39:19.768 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:39:19.796 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:19.817 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:39:19.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:19.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:39:19.921 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:39:19.939 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:19 smithi005 ceph-mon[39562]: pgmap v1904: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 7.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 55 MiB/s wr, 875 op/s 2024-04-20T17:39:19.939 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:19 smithi005 ceph-mon[39562]: osdmap e719: 12 total, 12 up, 10 in 2024-04-20T17:39:19.979 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:39:19.990 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:20.123 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:39:20.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:20.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:39:20.231 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:39:20.260 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:20.285 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:39:20.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:20.288 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:39:20.389 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:39:20.418 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:20.442 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:39:20.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:20.447 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:39:20.548 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:39:20.575 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:20.582 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 72 16 131122 131106 113.769 51.0625 0.00694179 0.00878689 2024-04-20T17:39:20.599 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:39:20.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:20.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:39:20.703 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:39:20.730 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:20.750 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:39:20.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:20.753 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:39:20.854 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:39:20.881 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:20 smithi171 ceph-mon[44189]: pgmap v1906: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 7.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-04-20T17:39:20.883 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:20.909 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:39:20.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:20.917 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:39:20.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:20 smithi005 ceph-mon[39562]: pgmap v1906: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 7.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-04-20T17:39:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:20 smithi106 ceph-mon[44423]: pgmap v1906: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 7.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-04-20T17:39:21.018 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:39:21.109 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:21.583 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 73 16 131947 131931 112.917 51.5625 0.00478834 0.00885356 2024-04-20T17:39:22.583 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 74 16 132821 132805 112.13 54.625 0.00715566 0.00891289 2024-04-20T17:39:22.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:22 smithi106 ceph-mon[44423]: osdmap e720: 12 total, 12 up, 10 in 2024-04-20T17:39:22.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:22 smithi171 ceph-mon[44189]: osdmap e720: 12 total, 12 up, 10 in 2024-04-20T17:39:22.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:22 smithi005 ceph-mon[39562]: osdmap e720: 12 total, 12 up, 10 in 2024-04-20T17:39:23.565 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:39:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:39:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:131494 7.8 7.7f489b38 (undecoded) ondisk+write+known_if_redirected e719)", 2024-04-20T17:39:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:39:21.061433+0000", 2024-04-20T17:39:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.068033671000000004, 2024-04-20T17:39:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.068062152000000001, 2024-04-20T17:39:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:23.565 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:39:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:39:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:39:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 131494 2024-04-20T17:39:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.061433+0000", 2024-04-20T17:39:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.566 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.061433+0000", 2024-04-20T17:39:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.061434+0000", 2024-04-20T17:39:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.001e-06 2024-04-20T17:39:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.061471+0000", 2024-04-20T17:39:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7617999999999998e-05 2024-04-20T17:39:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.567 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.061479+0000", 2024-04-20T17:39:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3810000000000001e-06 2024-04-20T17:39:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.061485+0000", 2024-04-20T17:39:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1759999999999998e-06 2024-04-20T17:39:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(osd.2.0:104822 7.8 e719/694)", 2024-04-20T17:39:23.568 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:39:21.062019+0000", 2024-04-20T17:39:23.569 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.067447763999999993, 2024-04-20T17:39:23.569 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.06754491, 2024-04-20T17:39:23.569 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:23.569 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:23.569 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:39:23.569 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:23.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.569 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:23.569 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.062019+0000", 2024-04-20T17:39:23.569 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.569 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:23.569 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.062019+0000", 2024-04-20T17:39:23.569 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.570 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.570 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.570 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:23.570 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.062021+0000", 2024-04-20T17:39:23.570 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1629999999999999e-06 2024-04-20T17:39:23.570 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.570 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.570 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:23.570 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.062027+0000", 2024-04-20T17:39:23.570 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.331e-06 2024-04-20T17:39:23.570 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.570 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.570 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:23.570 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.062043+0000", 2024-04-20T17:39:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5287000000000001e-05 2024-04-20T17:39:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.062051+0000", 2024-04-20T17:39:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4719999999999995e-06 2024-04-20T17:39:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(osd.2.0:104839 7.8 e719/694)", 2024-04-20T17:39:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:39:21.129337+0000", 2024-04-20T17:39:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.00012930799999999999, 2024-04-20T17:39:23.571 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00031413799999999999, 2024-04-20T17:39:23.572 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:23.572 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:23.572 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:39:23.572 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:23.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.572 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:23.572 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.129337+0000", 2024-04-20T17:39:23.572 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.572 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.572 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:23.572 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.129337+0000", 2024-04-20T17:39:23.572 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.572 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.129339+0000", 2024-04-20T17:39:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.643e-06 2024-04-20T17:39:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.129346+0000", 2024-04-20T17:39:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.956e-06 2024-04-20T17:39:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.129357+0000", 2024-04-20T17:39:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1109e-05 2024-04-20T17:39:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.573 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.129365+0000", 2024-04-20T17:39:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5210000000000002e-06 2024-04-20T17:39:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(osd.2.0:104823 7.8 e719/694)", 2024-04-20T17:39:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:39:21.067832+0000", 2024-04-20T17:39:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.061634373999999999, 2024-04-20T17:39:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.061871450000000001, 2024-04-20T17:39:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:23.574 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:39:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.067832+0000", 2024-04-20T17:39:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.067832+0000", 2024-04-20T17:39:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.067834+0000", 2024-04-20T17:39:23.575 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5340000000000001e-06 2024-04-20T17:39:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.067839+0000", 2024-04-20T17:39:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7230000000000001e-06 2024-04-20T17:39:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.067850+0000", 2024-04-20T17:39:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1079e-05 2024-04-20T17:39:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.067858+0000", 2024-04-20T17:39:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2269999999999996e-06 2024-04-20T17:39:23.576 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:131512 7.8 7.66c15358 (undecoded) ondisk+write+known_if_redirected e719)", 2024-04-20T17:39:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:39:21.075975+0000", 2024-04-20T17:39:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.053491093000000003, 2024-04-20T17:39:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.053784391000000001, 2024-04-20T17:39:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:39:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:39:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:39:23.577 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 131512 2024-04-20T17:39:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.075975+0000", 2024-04-20T17:39:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.075975+0000", 2024-04-20T17:39:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:23.578 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.075977+0000", 2024-04-20T17:39:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.807e-06 2024-04-20T17:39:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.076011+0000", 2024-04-20T17:39:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3606000000000003e-05 2024-04-20T17:39:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.076028+0000", 2024-04-20T17:39:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6949999999999999e-05 2024-04-20T17:39:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.579 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.076036+0000", 2024-04-20T17:39:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3990000000000004e-06 2024-04-20T17:39:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(osd.2.0:104825 7.8 e719/694)", 2024-04-20T17:39:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:39:21.084127+0000", 2024-04-20T17:39:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.045339531000000002, 2024-04-20T17:39:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.045698537999999997, 2024-04-20T17:39:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:39:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:23.580 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.084127+0000", 2024-04-20T17:39:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.084127+0000", 2024-04-20T17:39:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.084129+0000", 2024-04-20T17:39:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.739e-06 2024-04-20T17:39:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.581 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.582 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:23.582 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.084135+0000", 2024-04-20T17:39:23.582 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0100000000000001e-06 2024-04-20T17:39:23.582 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.582 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.582 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:23.582 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.084148+0000", 2024-04-20T17:39:23.582 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2981e-05 2024-04-20T17:39:23.582 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.582 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.582 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:23.582 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.084162+0000", 2024-04-20T17:39:23.582 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4089e-05 2024-04-20T17:39:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(osd.2.0:104826 7.8 e719/694)", 2024-04-20T17:39:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:39:21.084175+0000", 2024-04-20T17:39:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.045291125000000002, 2024-04-20T17:39:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.045701546000000003, 2024-04-20T17:39:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:39:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.583 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:23.584 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.084175+0000", 2024-04-20T17:39:23.584 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.584 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:23.584 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.084175+0000", 2024-04-20T17:39:23.584 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.584 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:23.584 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.084176+0000", 2024-04-20T17:39:23.584 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2900000000000003e-07 2024-04-20T17:39:23.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.584 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.084179+0000", 2024-04-20T17:39:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3330000000000001e-06 2024-04-20T17:39:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.084187+0000", 2024-04-20T17:39:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9759999999999995e-06 2024-04-20T17:39:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.084194+0000", 2024-04-20T17:39:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5130000000000002e-06 2024-04-20T17:39:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:23.585 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.586 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.586 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.586 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(osd.2.0:104827 7.8 e719/694)", 2024-04-20T17:39:23.586 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:39:21.084209+0000", 2024-04-20T17:39:23.586 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.045257618999999999, 2024-04-20T17:39:23.586 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.045716974, 2024-04-20T17:39:23.586 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:23.586 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:23.586 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:39:23.586 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:23.586 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.586 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:23.586 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.084209+0000", 2024-04-20T17:39:23.586 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.084209+0000", 2024-04-20T17:39:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.084209+0000", 2024-04-20T17:39:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0500000000000003e-07 2024-04-20T17:39:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.084213+0000", 2024-04-20T17:39:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2320000000000001e-06 2024-04-20T17:39:23.587 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.084220+0000", 2024-04-20T17:39:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.968e-06 2024-04-20T17:39:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.084225+0000", 2024-04-20T17:39:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9869999999999999e-06 2024-04-20T17:39:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.588 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(osd.2.0:104828 7.8 e719/694)", 2024-04-20T17:39:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:39:21.084231+0000", 2024-04-20T17:39:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.045235272, 2024-04-20T17:39:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.045746925000000001, 2024-04-20T17:39:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:39:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.084231+0000", 2024-04-20T17:39:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:23.589 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.084231+0000", 2024-04-20T17:39:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.084232+0000", 2024-04-20T17:39:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3599999999999999e-07 2024-04-20T17:39:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.084234+0000", 2024-04-20T17:39:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3870000000000002e-06 2024-04-20T17:39:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:23.590 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.084240+0000", 2024-04-20T17:39:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1e-06 2024-04-20T17:39:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.084245+0000", 2024-04-20T17:39:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4240000000000003e-06 2024-04-20T17:39:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:131522 7.8 7.62ccac78 (undecoded) ondisk+write+known_if_redirected e719)", 2024-04-20T17:39:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:39:21.089356+0000", 2024-04-20T17:39:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.040110761000000002, 2024-04-20T17:39:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.040678590000000001, 2024-04-20T17:39:23.591 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:39:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:39:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:39:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 131522 2024-04-20T17:39:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.089356+0000", 2024-04-20T17:39:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.592 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.089356+0000", 2024-04-20T17:39:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.089358+0000", 2024-04-20T17:39:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.311e-06 2024-04-20T17:39:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.089396+0000", 2024-04-20T17:39:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7700000000000002e-05 2024-04-20T17:39:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.593 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.089408+0000", 2024-04-20T17:39:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2320999999999999e-05 2024-04-20T17:39:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.089418+0000", 2024-04-20T17:39:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7869999999999996e-06 2024-04-20T17:39:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.594 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:131473 7.8 7:15c2a5e4:::benchmark_data_smithi171_161871_object131472:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e719)", 2024-04-20T17:39:23.595 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:39:21.025479+0000", 2024-04-20T17:39:23.595 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.10398700800000001, 2024-04-20T17:39:23.595 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10462709100000001, 2024-04-20T17:39:23.595 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:23.595 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:23.595 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:39:23.595 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:23.595 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:39:23.595 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:39:23.595 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 131473 2024-04-20T17:39:23.595 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.595 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:23.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.595 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:23.595 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.025479+0000", 2024-04-20T17:39:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.025479+0000", 2024-04-20T17:39:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.025480+0000", 2024-04-20T17:39:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5600000000000004e-07 2024-04-20T17:39:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:23.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.025520+0000", 2024-04-20T17:39:23.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9305999999999999e-05 2024-04-20T17:39:23.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.597 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:23.597 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.025526+0000", 2024-04-20T17:39:23.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7229999999999998e-06 2024-04-20T17:39:23.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.597 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:23.597 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.025532+0000", 2024-04-20T17:39:23.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.604e-06 2024-04-20T17:39:23.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.597 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:39:23.597 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.038216+0000", 2024-04-20T17:39:23.598 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012683933 2024-04-20T17:39:23.598 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.598 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.598 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:39:23.598 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.038355+0000", 2024-04-20T17:39:23.598 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013914299999999999 2024-04-20T17:39:23.598 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.598 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.598 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 7", 2024-04-20T17:39:23.598 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.038483+0000", 2024-04-20T17:39:23.598 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012793200000000001 2024-04-20T17:39:23.598 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.598 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.598 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:39:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.048475+0000", 2024-04-20T17:39:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0099916850000000001 2024-04-20T17:39:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:131474 7.8 7:1b9a75c3:::benchmark_data_smithi171_161871_object131473:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e719)", 2024-04-20T17:39:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:39:21.025539+0000", 2024-04-20T17:39:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.103927243, 2024-04-20T17:39:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10466499999999999, 2024-04-20T17:39:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:39:23.599 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:23.600 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:39:23.600 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:39:23.600 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 131474 2024-04-20T17:39:23.600 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.600 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:23.600 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.600 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:23.600 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.025539+0000", 2024-04-20T17:39:23.600 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.600 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.600 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.600 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:23.600 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.025539+0000", 2024-04-20T17:39:23.600 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.025540+0000", 2024-04-20T17:39:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3099999999999997e-07 2024-04-20T17:39:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.025567+0000", 2024-04-20T17:39:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7152e-05 2024-04-20T17:39:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.025572+0000", 2024-04-20T17:39:23.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3650000000000003e-06 2024-04-20T17:39:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.025577+0000", 2024-04-20T17:39:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9640000000000002e-06 2024-04-20T17:39:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:39:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.043212+0000", 2024-04-20T17:39:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.017634951999999999 2024-04-20T17:39:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:39:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.043393+0000", 2024-04-20T17:39:23.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000180781 2024-04-20T17:39:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 7", 2024-04-20T17:39:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.043594+0000", 2024-04-20T17:39:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00020129600000000001 2024-04-20T17:39:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(osd.2.0:104829 7.8 e719/694)", 2024-04-20T17:39:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:39:21.092612+0000", 2024-04-20T17:39:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.036854965000000003, 2024-04-20T17:39:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.037665905, 2024-04-20T17:39:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:23.603 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:39:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.092612+0000", 2024-04-20T17:39:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.092612+0000", 2024-04-20T17:39:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:23.604 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.092613+0000", 2024-04-20T17:39:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.483e-06 2024-04-20T17:39:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.092619+0000", 2024-04-20T17:39:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7259999999999999e-06 2024-04-20T17:39:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.092628+0000", 2024-04-20T17:39:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7200000000000001e-06 2024-04-20T17:39:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.092635+0000", 2024-04-20T17:39:23.605 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8750000000000002e-06 2024-04-20T17:39:23.606 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.606 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:23.606 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.606 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.606 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.606 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(osd.2.0:104831 7.8 e719/694)", 2024-04-20T17:39:23.606 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:39:21.092689+0000", 2024-04-20T17:39:23.606 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.036777476000000003, 2024-04-20T17:39:23.606 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.037651072000000001, 2024-04-20T17:39:23.606 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:23.606 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:23.606 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:39:23.606 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:23.606 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.607 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:23.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.092689+0000", 2024-04-20T17:39:23.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.607 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:23.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.092689+0000", 2024-04-20T17:39:23.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.607 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:23.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.092690+0000", 2024-04-20T17:39:23.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2900000000000003e-07 2024-04-20T17:39:23.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.608 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:23.608 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.092693+0000", 2024-04-20T17:39:23.608 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9919999999999999e-06 2024-04-20T17:39:23.608 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.608 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.608 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:23.608 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.092699+0000", 2024-04-20T17:39:23.608 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1750000000000002e-06 2024-04-20T17:39:23.608 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.608 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.608 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:23.608 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.092704+0000", 2024-04-20T17:39:23.608 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9250000000000003e-06 2024-04-20T17:39:23.608 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.608 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:131528 7.8 7.5c966318 (undecoded) ondisk+write+known_if_redirected e719)", 2024-04-20T17:39:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:39:21.094522+0000", 2024-04-20T17:39:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.034944167999999998, 2024-04-20T17:39:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.035866749000000003, 2024-04-20T17:39:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:39:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:39:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:39:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 131528 2024-04-20T17:39:23.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.610 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:23.610 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.610 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:23.610 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.094522+0000", 2024-04-20T17:39:23.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.610 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.610 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.610 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:23.610 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.094522+0000", 2024-04-20T17:39:23.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.610 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.610 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.610 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:23.610 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.094525+0000", 2024-04-20T17:39:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7310000000000002e-06 2024-04-20T17:39:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.094566+0000", 2024-04-20T17:39:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1001999999999998e-05 2024-04-20T17:39:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.094579+0000", 2024-04-20T17:39:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3365e-05 2024-04-20T17:39:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:23.611 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.094588+0000", 2024-04-20T17:39:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4179999999999997e-06 2024-04-20T17:39:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(osd.2.0:104833 7.8 e719/694)", 2024-04-20T17:39:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:39:21.100008+0000", 2024-04-20T17:39:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.029458492999999999, 2024-04-20T17:39:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.030479143, 2024-04-20T17:39:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:39:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:23.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.100008+0000", 2024-04-20T17:39:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.100008+0000", 2024-04-20T17:39:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.100010+0000", 2024-04-20T17:39:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.227e-06 2024-04-20T17:39:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.613 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.614 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:23.614 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.100018+0000", 2024-04-20T17:39:23.614 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1850000000000004e-06 2024-04-20T17:39:23.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.614 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:23.614 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.100033+0000", 2024-04-20T17:39:23.614 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4431e-05 2024-04-20T17:39:23.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.614 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:23.614 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.100043+0000", 2024-04-20T17:39:23.614 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0348e-05 2024-04-20T17:39:23.614 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.614 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:23.615 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.615 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.615 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(osd.2.0:104834 7.8 e719/694)", 2024-04-20T17:39:23.615 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:39:21.102151+0000", 2024-04-20T17:39:23.615 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.027315490000000001, 2024-04-20T17:39:23.615 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.028386591999999999, 2024-04-20T17:39:23.615 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:23.615 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:23.615 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:39:23.615 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:23.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.615 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:23.615 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.102151+0000", 2024-04-20T17:39:23.615 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.102151+0000", 2024-04-20T17:39:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.102153+0000", 2024-04-20T17:39:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8899999999999999e-06 2024-04-20T17:39:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.102160+0000", 2024-04-20T17:39:23.616 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9999999999999999e-06 2024-04-20T17:39:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.102173+0000", 2024-04-20T17:39:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2733e-05 2024-04-20T17:39:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.102181+0000", 2024-04-20T17:39:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6640000000000004e-06 2024-04-20T17:39:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.617 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.618 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(client.27088.0:131473 7.8 e719/694)", 2024-04-20T17:39:23.618 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:39:21.049332+0000", 2024-04-20T17:39:23.618 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.080134416999999999, 2024-04-20T17:39:23.618 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.081256027999999994, 2024-04-20T17:39:23.618 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:23.618 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:23.618 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:39:23.618 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:23.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.618 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:23.618 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.049332+0000", 2024-04-20T17:39:23.618 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.618 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.618 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.049332+0000", 2024-04-20T17:39:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.049334+0000", 2024-04-20T17:39:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8050000000000001e-06 2024-04-20T17:39:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.049340+0000", 2024-04-20T17:39:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0390000000000001e-06 2024-04-20T17:39:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.619 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.049349+0000", 2024-04-20T17:39:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0429999999999996e-06 2024-04-20T17:39:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.049356+0000", 2024-04-20T17:39:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4699999999999996e-06 2024-04-20T17:39:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(client.27088.0:131474 7.8 e719/694)", 2024-04-20T17:39:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:39:21.049372+0000", 2024-04-20T17:39:23.620 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.080094426999999996, 2024-04-20T17:39:23.621 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.081269569, 2024-04-20T17:39:23.621 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:23.621 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:23.621 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:39:23.621 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:23.621 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.621 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:23.621 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.049372+0000", 2024-04-20T17:39:23.621 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.621 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.621 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.621 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:23.621 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.049372+0000", 2024-04-20T17:39:23.621 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.621 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.049373+0000", 2024-04-20T17:39:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9100000000000003e-07 2024-04-20T17:39:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.049376+0000", 2024-04-20T17:39:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6660000000000001e-06 2024-04-20T17:39:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.049382+0000", 2024-04-20T17:39:23.622 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6509999999999998e-06 2024-04-20T17:39:23.623 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.623 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:23.623 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.049388+0000", 2024-04-20T17:39:23.623 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8239999999999996e-06 2024-04-20T17:39:23.623 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.623 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:23.623 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.623 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.623 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(osd.2.0:104835 7.8 e719/694)", 2024-04-20T17:39:23.623 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:39:21.108956+0000", 2024-04-20T17:39:23.623 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.020510509999999999, 2024-04-20T17:39:23.623 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.021733379000000001, 2024-04-20T17:39:23.623 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:23.624 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:23.624 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:39:23.624 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:23.624 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.624 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:23.624 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.108956+0000", 2024-04-20T17:39:23.624 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.624 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.624 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.624 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:23.624 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.108956+0000", 2024-04-20T17:39:23.624 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.624 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.624 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.624 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:23.625 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.108957+0000", 2024-04-20T17:39:23.625 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.443e-06 2024-04-20T17:39:23.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.625 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.625 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:23.625 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.108963+0000", 2024-04-20T17:39:23.625 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7939999999999999e-06 2024-04-20T17:39:23.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.625 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.625 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:23.625 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.108974+0000", 2024-04-20T17:39:23.625 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1178999999999999e-05 2024-04-20T17:39:23.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.625 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.625 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.108983+0000", 2024-04-20T17:39:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8170000000000005e-06 2024-04-20T17:39:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:131489 7.8 7.c02c6538 (undecoded) ondisk+write+known_if_redirected e719)", 2024-04-20T17:39:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:39:21.053126+0000", 2024-04-20T17:39:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.076340416999999994, 2024-04-20T17:39:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.077630594999999997, 2024-04-20T17:39:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:39:23.626 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:39:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:39:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 131489 2024-04-20T17:39:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.053126+0000", 2024-04-20T17:39:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.053126+0000", 2024-04-20T17:39:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.627 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.628 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.628 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:23.628 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.053129+0000", 2024-04-20T17:39:23.628 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5610000000000001e-06 2024-04-20T17:39:23.628 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.628 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.628 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:23.628 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.053163+0000", 2024-04-20T17:39:23.628 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4585999999999997e-05 2024-04-20T17:39:23.628 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.628 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.628 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:23.628 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.053180+0000", 2024-04-20T17:39:23.628 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7278999999999998e-05 2024-04-20T17:39:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.053190+0000", 2024-04-20T17:39:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7920000000000004e-06 2024-04-20T17:39:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(osd.2.0:104821 7.8 e719/694)", 2024-04-20T17:39:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:39:21.057260+0000", 2024-04-20T17:39:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.072206685000000007, 2024-04-20T17:39:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.073580239000000006, 2024-04-20T17:39:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:23.629 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:39:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.057260+0000", 2024-04-20T17:39:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.057260+0000", 2024-04-20T17:39:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:23.630 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.057263+0000", 2024-04-20T17:39:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7080000000000002e-06 2024-04-20T17:39:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.057270+0000", 2024-04-20T17:39:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8159999999999994e-06 2024-04-20T17:39:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.057293+0000", 2024-04-20T17:39:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2365e-05 2024-04-20T17:39:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:23.631 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.057303+0000", 2024-04-20T17:39:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0577999999999999e-05 2024-04-20T17:39:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(osd.2.0:104836 7.8 e719/694)", 2024-04-20T17:39:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:39:21.111841+0000", 2024-04-20T17:39:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.017625790999999998, 2024-04-20T17:39:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.019076952000000001, 2024-04-20T17:39:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:39:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.632 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.111841+0000", 2024-04-20T17:39:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.111841+0000", 2024-04-20T17:39:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.111843+0000", 2024-04-20T17:39:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.187e-06 2024-04-20T17:39:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:23.634 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.111855+0000", 2024-04-20T17:39:23.634 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1772e-05 2024-04-20T17:39:23.634 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.634 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.634 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:23.634 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.111868+0000", 2024-04-20T17:39:23.634 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.325e-05 2024-04-20T17:39:23.634 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.634 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.634 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:23.634 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.111878+0000", 2024-04-20T17:39:23.634 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0063e-05 2024-04-20T17:39:23.634 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.634 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:131493 7.8 7.67bccff8 (undecoded) ondisk+write+known_if_redirected e719)", 2024-04-20T17:39:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:39:21.061339+0000", 2024-04-20T17:39:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.068127071999999997, 2024-04-20T17:39:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.069657966000000002, 2024-04-20T17:39:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:39:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:39:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:39:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 131493 2024-04-20T17:39:23.635 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.061339+0000", 2024-04-20T17:39:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.061339+0000", 2024-04-20T17:39:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.061343+0000", 2024-04-20T17:39:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3390000000000001e-06 2024-04-20T17:39:23.636 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.061383+0000", 2024-04-20T17:39:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0064999999999997e-05 2024-04-20T17:39:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.061405+0000", 2024-04-20T17:39:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1888000000000001e-05 2024-04-20T17:39:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.061415+0000", 2024-04-20T17:39:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0368e-05 2024-04-20T17:39:23.637 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(osd.2.0:104838 7.8 e719/694)", 2024-04-20T17:39:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:39:21.116653+0000", 2024-04-20T17:39:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.012813872, 2024-04-20T17:39:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014405321, 2024-04-20T17:39:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:39:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:23.638 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.116653+0000", 2024-04-20T17:39:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.116653+0000", 2024-04-20T17:39:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.116655+0000", 2024-04-20T17:39:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2110000000000001e-06 2024-04-20T17:39:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.116665+0000", 2024-04-20T17:39:23.639 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0083e-05 2024-04-20T17:39:23.640 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.640 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.640 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:23.640 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.116681+0000", 2024-04-20T17:39:23.640 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5614999999999999e-05 2024-04-20T17:39:23.640 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:23.640 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:23.640 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:23.640 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:39:21.116690+0000", 2024-04-20T17:39:23.640 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1810000000000007e-06 2024-04-20T17:39:23.640 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.640 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:23.640 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.640 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:23.640 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:39:23.641 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 25 2024-04-20T17:39:23.641 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:39:23.641 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 75 16 133697 133681 111.365 54.75 0.0112276 0.00897517 2024-04-20T17:39:23.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:23 smithi106 ceph-mon[44423]: pgmap v1908: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 59 MiB/s wr, 941 op/s 2024-04-20T17:39:23.809 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:39:23.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:23.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:39:23.820 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:23 smithi005 ceph-mon[39562]: pgmap v1908: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 59 MiB/s wr, 941 op/s 2024-04-20T17:39:23.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:23 smithi171 ceph-mon[44189]: pgmap v1908: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.0 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 59 MiB/s wr, 941 op/s 2024-04-20T17:39:23.920 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:39:23.949 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:23.972 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:39:23.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:23.975 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:39:24.076 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:39:24.102 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:24.122 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:39:24.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:24.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:39:24.226 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:39:24.264 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:24.288 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:39:24.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:24.292 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:39:24.393 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:39:24.424 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:24.448 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:39:24.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:24.451 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:39:24.521 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:39:24.523 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-20T17:39:24.551 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:39:24.578 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:24.583 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 76 16 134597 134581 110.64 56.25 0.00434191 0.00903418 2024-04-20T17:39:24.597 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:39:24.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:24.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:39:24.701 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:39:24.728 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:24.748 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:39:24.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:24.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:39:24.852 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:39:24.879 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:24.898 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:39:24.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:24.902 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:39:25.003 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:39:25.033 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:25.061 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:39:25.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:25.065 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:39:25.166 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:39:25.197 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:25.220 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:39:25.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:25.225 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:39:25.326 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:39:25.429 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:25.583 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 77 16 135295 135279 109.769 43.625 0.0068503 0.00910477 2024-04-20T17:39:25.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:25 smithi005 ceph-mon[39562]: pgmap v1909: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.1 GiB data, 21 GiB used, 874 GiB / 894 GiB avail; 51 MiB/s wr, 812 op/s 2024-04-20T17:39:25.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:25 smithi171 ceph-mon[44189]: pgmap v1909: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.1 GiB data, 21 GiB used, 874 GiB / 894 GiB avail; 51 MiB/s wr, 812 op/s 2024-04-20T17:39:26.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:25 smithi106 ceph-mon[44423]: pgmap v1909: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.1 GiB data, 21 GiB used, 874 GiB / 894 GiB avail; 51 MiB/s wr, 812 op/s 2024-04-20T17:39:26.522 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:39:26.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:26.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:39:26.574 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:39:26.583 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 78 16 136157 136141 109.053 53.875 0.0164865 0.00916435 2024-04-20T17:39:26.627 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:39:26.655 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:26.677 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:39:26.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:26.682 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:39:26.783 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:39:26.808 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:26 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3046624374' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:39:26.808 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:26 smithi106 ceph-mon[44423]: from='client.27281 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:39:26.811 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:26.830 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:39:26.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:26.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:39:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:26 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3046624374' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:39:26.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:26 smithi171 ceph-mon[44189]: from='client.27281 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:39:26.933 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:39:26.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:26 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3046624374' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:39:26.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:26 smithi005 ceph-mon[39562]: from='client.27281 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:39:26.983 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:27.067 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:39:27.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:27.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:39:27.176 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:39:27.206 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:27.225 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:39:27.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:27.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:39:27.329 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:39:27.379 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:27.418 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:39:27.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:27.422 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:39:27.523 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:39:27.550 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:27.569 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:39:27.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:27.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:39:27.583 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 79 15 137389 137374 108.648 77.0625 0.00459966 0.00920244 2024-04-20T17:39:27.676 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:39:27.705 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:27.724 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:39:27.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:27.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:39:27.828 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:39:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:27 smithi171 ceph-mon[44189]: pgmap v1910: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.2 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-04-20T17:39:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:27 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:39:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:27 smithi171 ceph-mon[44189]: from='client.27281 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:39:27.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:27 smithi171 ceph-mon[44189]: osdmap e721: 12 total, 12 up, 10 in 2024-04-20T17:39:27.864 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:27.888 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:39:27.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:27.892 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:39:27.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:27 smithi005 ceph-mon[39562]: pgmap v1910: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.2 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-04-20T17:39:27.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:27 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:39:27.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:27 smithi005 ceph-mon[39562]: from='client.27281 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:39:27.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:27 smithi005 ceph-mon[39562]: osdmap e721: 12 total, 12 up, 10 in 2024-04-20T17:39:27.993 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:39:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:27 smithi106 ceph-mon[44423]: pgmap v1910: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 8.2 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-04-20T17:39:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:27 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:39:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:27 smithi106 ceph-mon[44423]: from='client.27281 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:39:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:27 smithi106 ceph-mon[44423]: osdmap e721: 12 total, 12 up, 10 in 2024-04-20T17:39:28.022 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:28.043 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:39:28.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:28.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:39:28.149 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:39:28.181 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:28.212 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:39:28.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:28.217 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:39:28.317 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:39:28.350 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:28.382 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:39:28.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:28.385 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:39:28.409 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:39:28.409 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:39:28.409 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:39:28.409 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:39:28.409 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:39:28.409 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:39:28.409 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:39:28.409 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:39:28.410 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:39:28.426 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:39:28.446 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:39:28.481 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:39:28.486 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:39:28.513 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:28.516 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:39:28.533 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:39:28.533 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:39:28.534 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:39:28.534 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:39:28.534 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:39:28.534 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:39:28.534 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:39:28.534 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:39:28.534 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:39:28.534 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:39:28.534 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:39:28.534 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:39:28.535 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:39:28.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:28.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:39:28.583 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:39:28.584289 min lat: 0.00100541 max lat: 0.210579 avg lat: 0.00924119 2024-04-20T17:39:28.583 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:39:28.583 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 80 16 138551 138535 108.197 72.5625 0.0060526 0.00924119 2024-04-20T17:39:28.638 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:39:28.678 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:28.704 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:39:28.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:28.707 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:39:28.808 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:39:28.833 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:28 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2065313176' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:39:28.833 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:28.833 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:28 smithi106 ceph-mon[44423]: pgmap v1912: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.2 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 43 MiB/s wr, 690 op/s 2024-04-20T17:39:28.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:28 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2065313176' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:39:28.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:28.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:28 smithi171 ceph-mon[44189]: pgmap v1912: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.2 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 43 MiB/s wr, 690 op/s 2024-04-20T17:39:28.835 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:28.855 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:39:28.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:28.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:39:28.945 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:28 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2065313176' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:39:28.945 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:28.945 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:28 smithi005 ceph-mon[39562]: pgmap v1912: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.2 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 43 MiB/s wr, 690 op/s 2024-04-20T17:39:28.958 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:39:28.987 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:29.006 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:39:29.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:29.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:39:29.110 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:39:29.143 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:29.175 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:39:29.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:29.180 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:39:29.281 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:39:29.329 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:29.479 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:39:29.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:29.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:39:29.583 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 81 16 139764 139748 107.797 75.8125 0.00529017 0.00927515 2024-04-20T17:39:29.584 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:39:29.610 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:29.631 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:39:29.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:29.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:39:29.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.734 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:39:29.769 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:29.805 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:39:29.807 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.808 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:29.809 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:39:29.910 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:39:29.935 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.935 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.935 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.936 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.936 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.936 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.936 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.936 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.936 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.936 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.936 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:29.937 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:29.958 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:39:29.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:29.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:39:30.064 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:39:30.117 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:30.139 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:39:30.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:30.142 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:39:30.243 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:39:30.270 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:30.290 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:39:30.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:30.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:39:30.321 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:39:30.394 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:39:30.428 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:30.456 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:39:30.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:30.460 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:39:30.561 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:39:30.583 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 82 16 140944 140928 107.382 73.75 0.0145703 0.00931083 2024-04-20T17:39:30.589 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:30.609 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:39:30.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:30.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:39:30.713 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:39:30.741 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:30 smithi171 ceph-mon[44189]: pgmap v1913: 17 pgs: 17 active+clean; 8.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 58 MiB/s wr, 930 op/s 2024-04-20T17:39:30.744 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:30.772 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:39:30.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:30.776 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:39:30.877 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:39:30.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:30 smithi106 ceph-mon[44423]: pgmap v1913: 17 pgs: 17 active+clean; 8.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 58 MiB/s wr, 930 op/s 2024-04-20T17:39:30.904 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:30.923 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:39:30.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:30.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:39:30.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:30 smithi005 ceph-mon[39562]: pgmap v1913: 17 pgs: 17 active+clean; 8.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 58 MiB/s wr, 930 op/s 2024-04-20T17:39:31.026 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:39:31.060 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:31.085 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:39:31.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:31.089 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:39:31.190 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:39:31.216 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:31.235 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:39:31.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:31.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:39:31.336 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:39:31.339 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:39:31.436 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:39:31.436 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:39:31.436 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:31.511 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:39:31.583 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 83 16 142077 142061 106.942 70.8125 0.0140634 0.00934939 2024-04-20T17:39:32.584 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 84 16 143280 143264 106.563 75.1875 0.0055282 0.00938282 2024-04-20T17:39:33.306 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:39:33.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:33.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:39:33.413 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:39:33.469 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:33.584 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 85 16 144477 144461 106.19 74.8125 0.00906837 0.00941544 2024-04-20T17:39:33.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:33 smithi005 ceph-mon[39562]: pgmap v1914: 17 pgs: 17 active+clean; 8.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 60 MiB/s wr, 956 op/s 2024-04-20T17:39:33.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:33 smithi171 ceph-mon[44189]: pgmap v1914: 17 pgs: 17 active+clean; 8.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 60 MiB/s wr, 956 op/s 2024-04-20T17:39:33.917 INFO:tasks.thrashosds.thrasher:Recovered, killing an osd 2024-04-20T17:39:33.917 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 4, 5, 8, 9, 10, 11, 2, 7, 6, 3, 1] 2024-04-20T17:39:33.917 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-20T17:39:33.917 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.7 2024-04-20T17:39:33.942 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:33 smithi106 ceph-mon[44423]: pgmap v1914: 17 pgs: 17 active+clean; 8.6 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 60 MiB/s wr, 956 op/s 2024-04-20T17:39:34.227 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:39:33 smithi106 systemd[1]: Stopping Ceph osd.7 for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:39:34.424 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-20T17:39:34.511 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:39:34 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[119199]: 2024-04-20T17:39:34.225+0000 7faedc4db700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T17:39:34.511 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:39:34 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[119199]: 2024-04-20T17:39:34.225+0000 7faedc4db700 -1 osd.7 721 *** Got signal Terminated *** 2024-04-20T17:39:34.511 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:39:34 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[119199]: 2024-04-20T17:39:34.225+0000 7faedc4db700 -1 osd.7 721 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T17:39:34.584 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 86 16 145659 145643 105.814 73.875 0.00566296 0.00944789 2024-04-20T17:39:34.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:34 smithi005 ceph-mon[39562]: osd.7 marked itself down and dead 2024-04-20T17:39:34.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:34 smithi171 ceph-mon[44189]: osd.7 marked itself down and dead 2024-04-20T17:39:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:34 smithi106 ceph-mon[44423]: osd.7 marked itself down and dead 2024-04-20T17:39:35.156 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:39:35.156 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:39:35.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:35.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:39:35.262 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:39:35.291 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:35.416 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:39:35.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:35.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:39:35.520 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:39:35.556 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:35.584 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 87 16 145722 145706 104.643 3.9375 0.00412446 0.00944733 2024-04-20T17:39:35.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:35 smithi106 ceph-mon[44423]: pgmap v1915: 17 pgs: 17 active+clean; 8.7 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-04-20T17:39:35.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:35 smithi106 ceph-mon[44423]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T17:39:35.653 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:35 smithi106 ceph-mon[44423]: osdmap e722: 12 total, 11 up, 10 in 2024-04-20T17:39:35.658 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:35 smithi005 ceph-mon[39562]: pgmap v1915: 17 pgs: 17 active+clean; 8.7 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-04-20T17:39:35.659 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:35 smithi005 ceph-mon[39562]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T17:39:35.659 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:35 smithi005 ceph-mon[39562]: osdmap e722: 12 total, 11 up, 10 in 2024-04-20T17:39:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:35 smithi171 ceph-mon[44189]: pgmap v1915: 17 pgs: 17 active+clean; 8.7 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-04-20T17:39:35.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:35 smithi171 ceph-mon[44189]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T17:39:35.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:35 smithi171 ceph-mon[44189]: osdmap e722: 12 total, 11 up, 10 in 2024-04-20T17:39:35.906 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:39:35.908 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:39:35 smithi106 bash[170953]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7 2024-04-20T17:39:35.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:35.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:39:36.011 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:39:36.042 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:36.096 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:39:36.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:36.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:39:36.200 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:39:36.228 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:36.577 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:39:36.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:36.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:39:36.584 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 88 16 145722 145706 103.454 0 - 0.00944733 2024-04-20T17:39:36.682 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:39:36.714 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:36.736 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:39:36.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:36.739 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:39:36.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:36 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:39:36.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:36 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:39:36.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:36 smithi106 ceph-mon[44423]: osdmap e723: 12 total, 11 up, 10 in 2024-04-20T17:39:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:36 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:39:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:36 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:39:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:36 smithi171 ceph-mon[44189]: osdmap e723: 12 total, 11 up, 10 in 2024-04-20T17:39:36.840 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:39:36.851 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:39:36.873 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:36.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:36 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:39:36.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:36 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:39:36.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:36 smithi005 ceph-mon[39562]: osdmap e723: 12 total, 11 up, 10 in 2024-04-20T17:39:37.501 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:39:37.502 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.7.service' 2024-04-20T17:39:37.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:37.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:39:37.563 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:39:37.584 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 89 16 145722 145706 102.292 0 - 0.00944733 2024-04-20T17:39:37.586 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T17:39:37.586 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-20T17:39:37.587 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd down 7 2024-04-20T17:39:37.605 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:39:37.637 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:37.658 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:39:37.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:37.661 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:39:37.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:37 smithi106 ceph-mon[44423]: pgmap v1918: 17 pgs: 1 peering, 1 stale+active+clean, 15 active+clean; 8.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-04-20T17:39:37.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:37 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/632641220' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:39:37.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:37 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3016920265' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:39:37.763 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:39:37.791 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:37.810 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:37 smithi005 ceph-mon[39562]: pgmap v1918: 17 pgs: 1 peering, 1 stale+active+clean, 15 active+clean; 8.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-04-20T17:39:37.810 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:37 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/632641220' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:39:37.810 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:37 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3016920265' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:39:37.811 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:39:37.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:37.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:39:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:37 smithi171 ceph-mon[44189]: pgmap v1918: 17 pgs: 1 peering, 1 stale+active+clean, 15 active+clean; 8.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-04-20T17:39:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:37 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/632641220' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:39:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:37 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3016920265' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:39:37.915 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:39:37.946 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:37.966 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:39:37.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:37.970 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:39:38.071 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:39:38.099 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:38.117 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:39:38.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:38.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:39:38.221 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:39:38.252 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:38.452 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:39:38.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:38.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:39:38.557 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:39:38.576 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:39:38.577 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:39:38.577 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:39:38.577 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:39:38.577 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:39:38.577 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:39:38.577 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:39:38.577 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:39:38.577 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:39:38.577 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:39:38.577 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:39:38.577 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:39:38.577 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:39:38.577 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:39:38.578 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:39:38.578 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:39:38.578 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:39:38.578 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:39:38.578 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:39:38.578 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:39:38.578 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:39:38.578 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:39:38.578 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:39:38.584 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 90 16 145722 145706 101.156 0 - 0.00944733 2024-04-20T17:39:38.584 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:38.603 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:39:38.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:38.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:39:38.707 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:39:38.741 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:38.797 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:39:38.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:38.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:39:38.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:38 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:39:38.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:38 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/632641220' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:39:38.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:38 smithi171 ceph-mon[44189]: osdmap e724: 12 total, 11 up, 10 in 2024-04-20T17:39:38.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:38.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:38.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:38.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:38.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:38.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:38.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:38.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:38.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:38.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:38.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:38 smithi171 ceph-mon[44189]: pgmap v1920: 17 pgs: 1 undersized+degraded+peered, 1 peering, 1 stale+active+clean, 14 active+clean; 8.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 55 MiB/s wr, 873 op/s; 9062/290850 objects degraded (3.116%) 2024-04-20T17:39:38.892 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:38 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:39:38.892 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:38 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/632641220' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:39:38.892 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:38 smithi005 ceph-mon[39562]: osdmap e724: 12 total, 11 up, 10 in 2024-04-20T17:39:38.892 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:38.892 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:38.892 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:38.892 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:38.892 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:38.892 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:38.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:38.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:38.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:38.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:38.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:38 smithi005 ceph-mon[39562]: pgmap v1920: 17 pgs: 1 undersized+degraded+peered, 1 peering, 1 stale+active+clean, 14 active+clean; 8.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 55 MiB/s wr, 873 op/s; 9062/290850 objects degraded (3.116%) 2024-04-20T17:39:38.902 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:39:38.934 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:38.955 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:39:38.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:38.959 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:39:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:38 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:39:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:38 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/632641220' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:39:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:38 smithi106 ceph-mon[44423]: osdmap e724: 12 total, 11 up, 10 in 2024-04-20T17:39:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:39.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:39.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:39.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:39.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:39.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:39.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:39.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:38 smithi106 ceph-mon[44423]: pgmap v1920: 17 pgs: 1 undersized+degraded+peered, 1 peering, 1 stale+active+clean, 14 active+clean; 8.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 55 MiB/s wr, 873 op/s; 9062/290850 objects degraded (3.116%) 2024-04-20T17:39:39.059 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:39:39.089 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:39.109 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:39:39.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:39.111 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:39:39.212 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:39:39.246 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:39.272 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:39:39.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:39.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:39:39.300 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:39:39.301 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:39:39.377 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:39:39.406 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:39.427 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:39:39.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:39.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:39:39.530 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:39:39.557 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:39.576 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:39:39.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:39.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:39:39.585 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 91 16 145722 145706 100.044 0 - 0.00944733 2024-04-20T17:39:39.680 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:39:39.711 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:39.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:39.747 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:39:39.747 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:39:39.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:39 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:39.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:39 smithi106 ceph-mon[44423]: Health check failed: Degraded data redundancy: 9062/290850 objects degraded (3.116%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:39:39.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:39 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:39.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:39 smithi171 ceph-mon[44189]: Health check failed: Degraded data redundancy: 9062/290850 objects degraded (3.116%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:39:39.847 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:39:39.848 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:39 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:39.848 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:39 smithi005 ceph-mon[39562]: Health check failed: Degraded data redundancy: 9062/290850 objects degraded (3.116%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:39:39.874 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:39.894 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:39:39.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:39.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:39:39.998 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:39:40.028 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:40.179 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:39:40.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:40.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:39:40.205 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:39:40.286 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:39:40.314 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:40.334 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:39:40.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:40.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:39:40.438 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:39:40.466 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:40.484 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:39:40.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:40.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:39:40.585 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 92 16 145722 145706 98.9571 0 - 0.00944733 2024-04-20T17:39:40.588 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:39:40.613 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:40 smithi106 ceph-mon[44423]: 7.1 scrub starts 2024-04-20T17:39:40.613 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:40 smithi106 ceph-mon[44423]: 7.f scrub starts 2024-04-20T17:39:40.613 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:40 smithi106 ceph-mon[44423]: pgmap v1921: 17 pgs: 1 active+clean+scrubbing, 3 undersized+degraded+peered, 13 active+clean; 8.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 33 MiB/s wr, 535 op/s; 18059/291420 objects degraded (6.197%) 2024-04-20T17:39:40.615 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:40.633 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:39:40.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:40.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:39:40.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:40 smithi005 ceph-mon[39562]: 7.1 scrub starts 2024-04-20T17:39:40.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:40 smithi005 ceph-mon[39562]: 7.f scrub starts 2024-04-20T17:39:40.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:40 smithi005 ceph-mon[39562]: pgmap v1921: 17 pgs: 1 active+clean+scrubbing, 3 undersized+degraded+peered, 13 active+clean; 8.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 33 MiB/s wr, 535 op/s; 18059/291420 objects degraded (6.197%) 2024-04-20T17:39:40.737 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:39:40.765 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:40.783 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:39:40.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:40.786 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:39:40.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:40 smithi171 ceph-mon[44189]: 7.1 scrub starts 2024-04-20T17:39:40.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:40 smithi171 ceph-mon[44189]: 7.f scrub starts 2024-04-20T17:39:40.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:40 smithi171 ceph-mon[44189]: pgmap v1921: 17 pgs: 1 active+clean+scrubbing, 3 undersized+degraded+peered, 13 active+clean; 8.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 33 MiB/s wr, 535 op/s; 18059/291420 objects degraded (6.197%) 2024-04-20T17:39:40.887 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:39:40.913 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:40.931 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:39:40.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:40.934 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:39:41.035 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:39:41.061 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:41.079 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:39:41.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:41.082 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:39:41.183 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:39:41.212 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:41.231 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:39:41.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:41.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:39:41.302 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:39:41.320 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:39:41.320 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:39:41.320 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:39:41.320 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:39:41.320 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.320 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:196584 5.6 5:69ff5259:::benchmark_data_smithi171_137281_object65150:head [delete] snapc 0=[] ondisk+write+known_if_redirected e598)", 2024-04-20T17:39:41.320 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:02.691746+0000", 2024-04-20T17:39:41.320 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 458.62576497800001, 2024-04-20T17:39:41.320 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.16427309100000001, 2024-04-20T17:39:41.320 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:41.320 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:41.320 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:41.321 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:41.321 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:39:41.321 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:39:41.321 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 196584 2024-04-20T17:39:41.321 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.321 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:41.321 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.321 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:41.321 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:02.691746+0000", 2024-04-20T17:39:41.321 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.321 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.321 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.321 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:41.321 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:02.691746+0000", 2024-04-20T17:39:41.322 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.322 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.322 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.322 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:41.322 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:02.691747+0000", 2024-04-20T17:39:41.322 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8000000000000005e-07 2024-04-20T17:39:41.322 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.322 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.322 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:41.322 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:02.691751+0000", 2024-04-20T17:39:41.322 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.924e-06 2024-04-20T17:39:41.322 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.322 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.322 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:41.322 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:02.691757+0000", 2024-04-20T17:39:41.323 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7009999999999999e-06 2024-04-20T17:39:41.323 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.323 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.323 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:41.323 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:02.691765+0000", 2024-04-20T17:39:41.323 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2799999999999998e-06 2024-04-20T17:39:41.323 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.323 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.323 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:39:41.323 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:02.701628+0000", 2024-04-20T17:39:41.323 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0098631819999999999 2024-04-20T17:39:41.323 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.323 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.323 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:39:41.324 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:02.701700+0000", 2024-04-20T17:39:41.324 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2372000000000004e-05 2024-04-20T17:39:41.324 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.324 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.324 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:39:41.324 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:02.701779+0000", 2024-04-20T17:39:41.324 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9191999999999997e-05 2024-04-20T17:39:41.324 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.324 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.324 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:41.324 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:02.815484+0000", 2024-04-20T17:39:41.324 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.113704508 2024-04-20T17:39:41.324 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.324 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.324 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:39:41.325 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:02.855925+0000", 2024-04-20T17:39:41.325 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.040441062999999999 2024-04-20T17:39:41.325 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.325 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.325 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:39:41.325 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:02.855968+0000", 2024-04-20T17:39:41.325 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2648999999999998e-05 2024-04-20T17:39:41.325 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.325 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.325 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:39:41.325 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:02.856019+0000", 2024-04-20T17:39:41.325 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1440000000000002e-05 2024-04-20T17:39:41.325 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.325 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:41.326 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.326 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.326 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.326 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:198141 5.6 5:6c7b0382:::benchmark_data_smithi171_137281_object66707:head [delete] snapc 0=[] ondisk+write+known_if_redirected e598)", 2024-04-20T17:39:41.326 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:04.223408+0000", 2024-04-20T17:39:41.326 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 457.09410317999999, 2024-04-20T17:39:41.326 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.15999193, 2024-04-20T17:39:41.326 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:41.326 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:41.326 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:41.326 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:41.326 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:39:41.326 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:39:41.326 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 198141 2024-04-20T17:39:41.326 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.327 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:41.327 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.327 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:41.327 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.223408+0000", 2024-04-20T17:39:41.327 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.327 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.327 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.327 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:41.327 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.223408+0000", 2024-04-20T17:39:41.327 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.327 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.327 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.327 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:41.327 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.223410+0000", 2024-04-20T17:39:41.328 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9089999999999998e-06 2024-04-20T17:39:41.328 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.328 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.328 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:41.328 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.223414+0000", 2024-04-20T17:39:41.328 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7219999999999999e-06 2024-04-20T17:39:41.328 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.328 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.328 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:41.328 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.223426+0000", 2024-04-20T17:39:41.328 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1256e-05 2024-04-20T17:39:41.328 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.328 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.328 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:41.328 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.223434+0000", 2024-04-20T17:39:41.329 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0930000000000003e-06 2024-04-20T17:39:41.329 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.329 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.329 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:39:41.329 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.230634+0000", 2024-04-20T17:39:41.329 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0072005439999999997 2024-04-20T17:39:41.329 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.329 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.329 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:39:41.329 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.230729+0000", 2024-04-20T17:39:41.329 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4617000000000006e-05 2024-04-20T17:39:41.329 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.329 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.329 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:39:41.329 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.230811+0000", 2024-04-20T17:39:41.330 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1767000000000005e-05 2024-04-20T17:39:41.330 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.330 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.330 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:39:41.330 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.241681+0000", 2024-04-20T17:39:41.330 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010870591000000001 2024-04-20T17:39:41.330 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.330 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.330 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:41.330 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.383298+0000", 2024-04-20T17:39:41.330 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.14161648399999999 2024-04-20T17:39:41.330 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.330 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.330 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:39:41.331 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.383339+0000", 2024-04-20T17:39:41.331 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0923000000000001e-05 2024-04-20T17:39:41.331 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.331 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.331 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:39:41.331 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.383400+0000", 2024-04-20T17:39:41.331 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1023999999999997e-05 2024-04-20T17:39:41.331 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.331 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:41.331 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.331 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.331 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.331 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:198148 5.6 5:63270389:::benchmark_data_smithi171_137281_object66714:head [delete] snapc 0=[] ondisk+write+known_if_redirected e598)", 2024-04-20T17:39:41.331 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:04.232148+0000", 2024-04-20T17:39:41.332 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 457.08536297500001, 2024-04-20T17:39:41.332 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.15136398700000001, 2024-04-20T17:39:41.332 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:41.332 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:41.332 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:41.332 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:41.332 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:39:41.332 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:39:41.332 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 198148 2024-04-20T17:39:41.332 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.332 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:41.333 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.333 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:41.333 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.232148+0000", 2024-04-20T17:39:41.333 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.333 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.333 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.333 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:41.333 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.232148+0000", 2024-04-20T17:39:41.333 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.333 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.333 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.333 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:41.333 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.232150+0000", 2024-04-20T17:39:41.333 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1900000000000002e-06 2024-04-20T17:39:41.334 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.334 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.334 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:41.334 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.232156+0000", 2024-04-20T17:39:41.334 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9719999999999997e-06 2024-04-20T17:39:41.334 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.334 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.334 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:41.334 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.232170+0000", 2024-04-20T17:39:41.334 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3326000000000001e-05 2024-04-20T17:39:41.334 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.334 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.334 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:41.334 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.232178+0000", 2024-04-20T17:39:41.335 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5390000000000007e-06 2024-04-20T17:39:41.335 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.335 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.335 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:39:41.335 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.236305+0000", 2024-04-20T17:39:41.335 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041266840000000003 2024-04-20T17:39:41.335 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.335 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.335 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:39:41.335 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.236457+0000", 2024-04-20T17:39:41.335 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015202 2024-04-20T17:39:41.335 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.335 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.335 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:39:41.335 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.236614+0000", 2024-04-20T17:39:41.336 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000156935 2024-04-20T17:39:41.336 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.336 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.336 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:39:41.336 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.383413+0000", 2024-04-20T17:39:41.336 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.14679882999999999 2024-04-20T17:39:41.336 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.336 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.336 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:41.336 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.383449+0000", 2024-04-20T17:39:41.336 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6427999999999997e-05 2024-04-20T17:39:41.336 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.336 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.336 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:39:41.337 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.383480+0000", 2024-04-20T17:39:41.337 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1452000000000003e-05 2024-04-20T17:39:41.337 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.337 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.337 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:39:41.337 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.383512+0000", 2024-04-20T17:39:41.337 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1610999999999999e-05 2024-04-20T17:39:41.337 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.337 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:41.337 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.337 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.337 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.337 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:198157 5.6 5:6e8b4717:::benchmark_data_smithi171_137281_object66723:head [delete] snapc 0=[] ondisk+write+known_if_redirected e598)", 2024-04-20T17:39:41.337 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:04.241922+0000", 2024-04-20T17:39:41.338 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 457.07558920000002, 2024-04-20T17:39:41.338 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17007925700000001, 2024-04-20T17:39:41.338 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:41.338 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:41.338 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:41.338 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:41.338 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:39:41.338 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:39:41.338 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 198157 2024-04-20T17:39:41.338 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.338 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:41.338 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.338 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:41.338 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.241922+0000", 2024-04-20T17:39:41.339 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.339 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.339 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.339 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:41.339 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.241922+0000", 2024-04-20T17:39:41.339 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.339 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.339 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.339 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:41.339 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.241925+0000", 2024-04-20T17:39:41.339 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7939999999999998e-06 2024-04-20T17:39:41.339 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.339 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.339 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:41.340 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.241931+0000", 2024-04-20T17:39:41.340 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7490000000000001e-06 2024-04-20T17:39:41.340 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.340 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.340 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:41.340 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.241944+0000", 2024-04-20T17:39:41.340 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2159e-05 2024-04-20T17:39:41.340 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.340 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.340 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:41.340 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.241952+0000", 2024-04-20T17:39:41.340 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2349999999999999e-06 2024-04-20T17:39:41.340 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.340 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.340 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:39:41.341 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.386210+0000", 2024-04-20T17:39:41.341 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.14425862 2024-04-20T17:39:41.341 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.341 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.341 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:39:41.341 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.386323+0000", 2024-04-20T17:39:41.341 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000112991 2024-04-20T17:39:41.341 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.341 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.341 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:39:41.341 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.386425+0000", 2024-04-20T17:39:41.341 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010176899999999999 2024-04-20T17:39:41.341 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.341 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.341 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:41.342 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.399037+0000", 2024-04-20T17:39:41.342 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012612254 2024-04-20T17:39:41.342 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.342 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.342 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:39:41.342 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.407304+0000", 2024-04-20T17:39:41.342 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0082665110000000007 2024-04-20T17:39:41.342 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.342 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.342 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:39:41.342 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.407369+0000", 2024-04-20T17:39:41.342 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5469999999999995e-05 2024-04-20T17:39:41.342 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.342 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.343 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:39:41.343 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:04.412001+0000", 2024-04-20T17:39:41.343 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0046317049999999998 2024-04-20T17:39:41.343 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.343 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:41.343 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.343 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.343 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.343 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:219957 5.6 5:6e2fd4d9:::benchmark_data_smithi171_137281_object88523:head [delete] snapc 0=[] ondisk+write+known_if_redirected e601)", 2024-04-20T17:39:41.343 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:24.617680+0000", 2024-04-20T17:39:41.343 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 436.69983075900001, 2024-04-20T17:39:41.343 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.226933211, 2024-04-20T17:39:41.343 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:41.343 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:41.343 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:41.344 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:41.344 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:39:41.344 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:39:41.344 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 219957 2024-04-20T17:39:41.344 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.344 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:41.344 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.344 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:41.344 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.617680+0000", 2024-04-20T17:39:41.344 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.344 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.344 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.344 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:41.344 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.617680+0000", 2024-04-20T17:39:41.345 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.345 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.345 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.345 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:41.345 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.617682+0000", 2024-04-20T17:39:41.345 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1550000000000001e-06 2024-04-20T17:39:41.345 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.345 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.345 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:41.345 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.617688+0000", 2024-04-20T17:39:41.345 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.733e-06 2024-04-20T17:39:41.345 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.345 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.345 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:41.345 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.617702+0000", 2024-04-20T17:39:41.345 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4236999999999999e-05 2024-04-20T17:39:41.346 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.346 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.346 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:41.346 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.617710+0000", 2024-04-20T17:39:41.346 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0539999999999995e-06 2024-04-20T17:39:41.346 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.346 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.346 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:39:41.346 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.617766+0000", 2024-04-20T17:39:41.346 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5640000000000003e-05 2024-04-20T17:39:41.346 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.346 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.346 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:39:41.346 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.617861+0000", 2024-04-20T17:39:41.347 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.5098000000000003e-05 2024-04-20T17:39:41.347 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.347 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.347 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:39:41.347 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.617990+0000", 2024-04-20T17:39:41.347 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012899400000000001 2024-04-20T17:39:41.347 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.347 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.347 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:39:41.347 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.622858+0000", 2024-04-20T17:39:41.347 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0048682309999999998 2024-04-20T17:39:41.347 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.347 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.347 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:41.347 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.844495+0000", 2024-04-20T17:39:41.348 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.22163646200000001 2024-04-20T17:39:41.348 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.348 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.348 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:39:41.348 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.844555+0000", 2024-04-20T17:39:41.348 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0582999999999999e-05 2024-04-20T17:39:41.348 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.348 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.348 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:39:41.348 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.844613+0000", 2024-04-20T17:39:41.349 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8023999999999999e-05 2024-04-20T17:39:41.349 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.349 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:41.349 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.349 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.349 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.349 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:219963 5.6 5:65be2a8e:::benchmark_data_smithi171_137281_object88529:head [delete] snapc 0=[] ondisk+write+known_if_redirected e601)", 2024-04-20T17:39:41.349 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:24.627025+0000", 2024-04-20T17:39:41.349 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 436.690486088, 2024-04-20T17:39:41.349 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.22374935600000001, 2024-04-20T17:39:41.349 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:41.349 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:41.349 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:41.349 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:41.350 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:39:41.350 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:39:41.350 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 219963 2024-04-20T17:39:41.350 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.350 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:41.350 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.350 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:41.350 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627025+0000", 2024-04-20T17:39:41.350 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.350 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.350 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.350 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:41.350 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627025+0000", 2024-04-20T17:39:41.351 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.351 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.351 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.351 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:41.351 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627027+0000", 2024-04-20T17:39:41.351 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.34e-06 2024-04-20T17:39:41.351 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.351 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.351 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:41.351 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627034+0000", 2024-04-20T17:39:41.351 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9299999999999997e-06 2024-04-20T17:39:41.351 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.351 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.351 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:41.352 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627050+0000", 2024-04-20T17:39:41.352 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6090999999999999e-05 2024-04-20T17:39:41.352 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.352 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.352 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:41.352 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627060+0000", 2024-04-20T17:39:41.352 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.312e-06 2024-04-20T17:39:41.352 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.352 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.352 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:39:41.352 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627134+0000", 2024-04-20T17:39:41.352 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4357999999999995e-05 2024-04-20T17:39:41.352 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.352 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.353 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:39:41.353 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627247+0000", 2024-04-20T17:39:41.353 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000113101 2024-04-20T17:39:41.353 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.353 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.353 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:39:41.353 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627398+0000", 2024-04-20T17:39:41.353 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015086799999999999 2024-04-20T17:39:41.353 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.353 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.353 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:39:41.353 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.636898+0000", 2024-04-20T17:39:41.353 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0094998559999999992 2024-04-20T17:39:41.353 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.353 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.354 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:41.354 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.850666+0000", 2024-04-20T17:39:41.354 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.213767862 2024-04-20T17:39:41.354 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.354 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.354 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:39:41.354 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.850723+0000", 2024-04-20T17:39:41.354 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6935999999999998e-05 2024-04-20T17:39:41.354 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.354 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.355 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:39:41.355 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.850774+0000", 2024-04-20T17:39:41.355 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1702000000000001e-05 2024-04-20T17:39:41.355 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.355 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:41.355 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.355 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.355 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.355 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:219967 5.6 5:6a45f1ea:::benchmark_data_smithi171_137281_object88533:head [delete] snapc 0=[] ondisk+write+known_if_redirected e601)", 2024-04-20T17:39:41.355 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:24.627084+0000", 2024-04-20T17:39:41.355 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 436.69042709299998, 2024-04-20T17:39:41.355 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.22382991299999999, 2024-04-20T17:39:41.355 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:41.355 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:41.355 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:41.356 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:41.356 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:39:41.356 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:39:41.356 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 219967 2024-04-20T17:39:41.356 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.356 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:41.356 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.356 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:41.356 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627084+0000", 2024-04-20T17:39:41.356 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.356 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.356 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.356 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:41.356 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627084+0000", 2024-04-20T17:39:41.356 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.357 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.357 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.357 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:41.357 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627085+0000", 2024-04-20T17:39:41.357 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.85e-07 2024-04-20T17:39:41.357 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.357 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.357 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:41.357 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627088+0000", 2024-04-20T17:39:41.357 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2590000000000001e-06 2024-04-20T17:39:41.357 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.357 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.357 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:41.357 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627095+0000", 2024-04-20T17:39:41.358 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0960000000000003e-06 2024-04-20T17:39:41.358 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.358 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.358 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:41.358 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.627099+0000", 2024-04-20T17:39:41.358 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8539999999999999e-06 2024-04-20T17:39:41.358 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.358 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.358 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:39:41.358 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.632025+0000", 2024-04-20T17:39:41.358 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0049258870000000003 2024-04-20T17:39:41.358 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.358 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.358 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:39:41.358 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.632149+0000", 2024-04-20T17:39:41.359 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012428 2024-04-20T17:39:41.359 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.359 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.359 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:39:41.359 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.632268+0000", 2024-04-20T17:39:41.359 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000119244 2024-04-20T17:39:41.359 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.359 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.359 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:39:41.359 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.637004+0000", 2024-04-20T17:39:41.359 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0047353550000000001 2024-04-20T17:39:41.359 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.359 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.359 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:41.359 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.850820+0000", 2024-04-20T17:39:41.359 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.21381650099999999 2024-04-20T17:39:41.360 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.360 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.360 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:39:41.360 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.850871+0000", 2024-04-20T17:39:41.360 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1261999999999998e-05 2024-04-20T17:39:41.360 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.360 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.360 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:39:41.360 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.850914+0000", 2024-04-20T17:39:41.360 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2389999999999999e-05 2024-04-20T17:39:41.360 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.360 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:41.360 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.360 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.360 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.361 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:220024 5.6 5:642af313:::benchmark_data_smithi171_137281_object88590:head [delete] snapc 0=[] ondisk+write+known_if_redirected e601)", 2024-04-20T17:39:41.361 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:24.674168+0000", 2024-04-20T17:39:41.361 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 436.643343276, 2024-04-20T17:39:41.361 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.183402497, 2024-04-20T17:39:41.361 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:41.361 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:41.361 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:41.361 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:41.361 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:39:41.361 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:39:41.361 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 220024 2024-04-20T17:39:41.361 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.361 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:41.361 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.362 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:41.362 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.674168+0000", 2024-04-20T17:39:41.362 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.362 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.362 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.362 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:41.362 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.674168+0000", 2024-04-20T17:39:41.362 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.362 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.362 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.362 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:41.362 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.674169+0000", 2024-04-20T17:39:41.362 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.353e-06 2024-04-20T17:39:41.363 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.363 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.363 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:41.363 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.674174+0000", 2024-04-20T17:39:41.363 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4800000000000003e-06 2024-04-20T17:39:41.363 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.363 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.363 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:41.363 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.674182+0000", 2024-04-20T17:39:41.363 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8459999999999997e-06 2024-04-20T17:39:41.363 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.363 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.363 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:41.364 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.674189+0000", 2024-04-20T17:39:41.364 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8530000000000003e-06 2024-04-20T17:39:41.364 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.364 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.364 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:39:41.364 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.674243+0000", 2024-04-20T17:39:41.364 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3439000000000001e-05 2024-04-20T17:39:41.364 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.364 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.364 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:39:41.364 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.674423+0000", 2024-04-20T17:39:41.364 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00017983300000000001 2024-04-20T17:39:41.364 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.364 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.364 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:39:41.365 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.674535+0000", 2024-04-20T17:39:41.365 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011236899999999999 2024-04-20T17:39:41.365 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.365 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.365 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:39:41.365 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.679774+0000", 2024-04-20T17:39:41.365 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0052394939999999999 2024-04-20T17:39:41.365 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.365 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.365 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:41.365 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.857487+0000", 2024-04-20T17:39:41.365 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17771221500000001 2024-04-20T17:39:41.365 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.365 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.365 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:39:41.366 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.857522+0000", 2024-04-20T17:39:41.366 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4938000000000003e-05 2024-04-20T17:39:41.366 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.366 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.366 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:39:41.366 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.857570+0000", 2024-04-20T17:39:41.366 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8677000000000002e-05 2024-04-20T17:39:41.366 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.366 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:41.366 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.366 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.366 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.366 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:220029 5.6 5:610901c7:::benchmark_data_smithi171_137281_object88595:head [delete] snapc 0=[] ondisk+write+known_if_redirected e601)", 2024-04-20T17:39:41.366 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:24.684367+0000", 2024-04-20T17:39:41.367 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 436.633143539, 2024-04-20T17:39:41.367 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.173362769, 2024-04-20T17:39:41.367 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:41.367 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:41.367 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:41.367 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:41.367 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:39:41.367 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:39:41.367 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 220029 2024-04-20T17:39:41.367 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.367 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:41.367 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.367 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:41.367 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.684367+0000", 2024-04-20T17:39:41.367 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.368 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.368 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.368 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:41.368 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.684367+0000", 2024-04-20T17:39:41.368 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.368 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.368 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.368 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:41.368 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.684369+0000", 2024-04-20T17:39:41.368 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.875e-06 2024-04-20T17:39:41.368 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.368 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.368 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:41.368 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.684374+0000", 2024-04-20T17:39:41.368 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5970000000000002e-06 2024-04-20T17:39:41.369 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.369 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.369 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:41.369 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.684383+0000", 2024-04-20T17:39:41.369 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2289999999999992e-06 2024-04-20T17:39:41.369 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.369 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.369 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:41.369 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.684390+0000", 2024-04-20T17:39:41.369 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4019999999999999e-06 2024-04-20T17:39:41.369 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.369 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.369 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:39:41.369 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.684446+0000", 2024-04-20T17:39:41.369 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6387000000000001e-05 2024-04-20T17:39:41.370 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.370 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.370 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:39:41.370 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.684579+0000", 2024-04-20T17:39:41.370 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013314800000000001 2024-04-20T17:39:41.370 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.370 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.370 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:39:41.370 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.684706+0000", 2024-04-20T17:39:41.370 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012701999999999999 2024-04-20T17:39:41.370 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.370 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.370 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:39:41.370 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.689259+0000", 2024-04-20T17:39:41.370 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0045530309999999999 2024-04-20T17:39:41.371 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.371 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.371 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:41.371 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.857634+0000", 2024-04-20T17:39:41.371 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.16837489999999999 2024-04-20T17:39:41.371 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.371 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.371 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:39:41.371 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.857689+0000", 2024-04-20T17:39:41.371 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5019999999999998e-05 2024-04-20T17:39:41.371 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.371 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.371 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:39:41.371 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:24.857730+0000", 2024-04-20T17:39:41.372 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1159999999999999e-05 2024-04-20T17:39:41.372 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.372 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:41.372 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.372 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.372 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.372 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:234153 5.a 5:58b994fd:::benchmark_data_smithi171_137281_object102719:head [delete] snapc 0=[] ondisk+write+known_if_redirected e604)", 2024-04-20T17:39:41.372 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:37.046372+0000", 2024-04-20T17:39:41.372 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 424.27113909000002, 2024-04-20T17:39:41.372 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.15147975799999999, 2024-04-20T17:39:41.372 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:41.372 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:41.372 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:41.372 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:41.372 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:39:41.373 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:39:41.373 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 234153 2024-04-20T17:39:41.373 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.373 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:41.373 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.373 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:41.373 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:37.046372+0000", 2024-04-20T17:39:41.373 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.373 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.373 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.373 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:41.373 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:37.046372+0000", 2024-04-20T17:39:41.373 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.373 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.373 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.373 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:41.374 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:37.046374+0000", 2024-04-20T17:39:41.374 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6610000000000001e-06 2024-04-20T17:39:41.374 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.374 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.374 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:41.374 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:37.046380+0000", 2024-04-20T17:39:41.374 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5599999999999999e-06 2024-04-20T17:39:41.374 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.374 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.374 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:41.374 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:37.046387+0000", 2024-04-20T17:39:41.374 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.002e-06 2024-04-20T17:39:41.374 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.374 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.375 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:41.375 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:37.046393+0000", 2024-04-20T17:39:41.375 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6629999999999998e-06 2024-04-20T17:39:41.375 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.375 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.375 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:39:41.375 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:37.073213+0000", 2024-04-20T17:39:41.375 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.026819886000000001 2024-04-20T17:39:41.375 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.375 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.375 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:39:41.375 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:37.073346+0000", 2024-04-20T17:39:41.375 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000133032 2024-04-20T17:39:41.375 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.375 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.376 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:39:41.376 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:37.073462+0000", 2024-04-20T17:39:41.376 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000116709 2024-04-20T17:39:41.376 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.376 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.376 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:39:41.376 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:37.083070+0000", 2024-04-20T17:39:41.376 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0096075359999999999 2024-04-20T17:39:41.376 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.376 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.376 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:41.376 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:37.197726+0000", 2024-04-20T17:39:41.376 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.114656064 2024-04-20T17:39:41.376 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.376 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.377 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:39:41.377 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:37.197785+0000", 2024-04-20T17:39:41.377 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9305999999999998e-05 2024-04-20T17:39:41.377 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.377 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.377 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:39:41.377 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:37.197852+0000", 2024-04-20T17:39:41.377 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6339000000000003e-05 2024-04-20T17:39:41.377 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.377 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:41.377 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.377 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.377 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.377 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:236589 5.a 5:55c87012:::benchmark_data_smithi171_137281_object105155:head [delete] snapc 0=[] ondisk+write+known_if_redirected e606)", 2024-04-20T17:39:41.377 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:40.279469+0000", 2024-04-20T17:39:41.378 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 421.038041776, 2024-04-20T17:39:41.378 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.16665637899999999, 2024-04-20T17:39:41.378 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:41.378 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:41.378 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:41.378 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:41.378 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:39:41.378 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:39:41.378 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 236589 2024-04-20T17:39:41.378 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.378 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:41.378 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.378 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:41.378 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.279469+0000", 2024-04-20T17:39:41.379 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.379 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.379 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.379 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:41.379 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.279469+0000", 2024-04-20T17:39:41.379 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.379 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.379 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.379 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:41.379 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.279471+0000", 2024-04-20T17:39:41.379 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3230000000000001e-06 2024-04-20T17:39:41.379 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.379 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.379 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:41.379 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.279475+0000", 2024-04-20T17:39:41.379 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0640000000000004e-06 2024-04-20T17:39:41.380 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.380 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.380 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:41.380 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.279487+0000", 2024-04-20T17:39:41.380 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2568999999999999e-05 2024-04-20T17:39:41.380 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.380 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.380 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:41.380 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.279494+0000", 2024-04-20T17:39:41.380 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7209999999999997e-06 2024-04-20T17:39:41.380 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.380 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.380 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:39:41.380 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.308785+0000", 2024-04-20T17:39:41.380 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.029291159000000001 2024-04-20T17:39:41.381 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.381 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.381 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:39:41.381 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.308895+0000", 2024-04-20T17:39:41.381 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000109713 2024-04-20T17:39:41.381 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.381 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.381 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:39:41.381 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.309020+0000", 2024-04-20T17:39:41.381 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012495799999999999 2024-04-20T17:39:41.381 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.381 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.381 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:39:41.381 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.319829+0000", 2024-04-20T17:39:41.381 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010809526 2024-04-20T17:39:41.382 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.382 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.382 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:41.382 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.446006+0000", 2024-04-20T17:39:41.382 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.12617643100000001 2024-04-20T17:39:41.382 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.382 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.382 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:39:41.382 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.446062+0000", 2024-04-20T17:39:41.382 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6418000000000001e-05 2024-04-20T17:39:41.382 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.382 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.382 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:39:41.383 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.446126+0000", 2024-04-20T17:39:41.383 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3497000000000005e-05 2024-04-20T17:39:41.383 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.383 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:41.383 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.383 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.383 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.383 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:236595 5.a 5:546f56ab:::benchmark_data_smithi171_137281_object105161:head [delete] snapc 0=[] ondisk+write+known_if_redirected e606)", 2024-04-20T17:39:41.383 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:40.292587+0000", 2024-04-20T17:39:41.383 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 421.02492355200002, 2024-04-20T17:39:41.383 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.15402600699999999, 2024-04-20T17:39:41.383 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:41.383 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:41.384 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:41.384 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:41.384 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:39:41.384 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:39:41.384 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 236595 2024-04-20T17:39:41.384 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.384 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:41.384 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.384 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:41.384 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.292587+0000", 2024-04-20T17:39:41.384 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.384 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.384 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.384 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:41.384 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.292587+0000", 2024-04-20T17:39:41.384 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.385 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.385 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.385 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:41.385 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.292589+0000", 2024-04-20T17:39:41.385 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6649999999999999e-06 2024-04-20T17:39:41.385 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.385 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.385 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:41.385 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.292594+0000", 2024-04-20T17:39:41.385 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.091e-06 2024-04-20T17:39:41.385 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.385 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.385 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:41.385 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.292603+0000", 2024-04-20T17:39:41.386 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5439999999999999e-06 2024-04-20T17:39:41.386 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.386 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.386 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:41.386 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.292610+0000", 2024-04-20T17:39:41.386 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.9789999999999996e-06 2024-04-20T17:39:41.386 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.386 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.386 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:39:41.386 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.313922+0000", 2024-04-20T17:39:41.386 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.021312607000000001 2024-04-20T17:39:41.386 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.386 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.386 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:39:41.386 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.314051+0000", 2024-04-20T17:39:41.387 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012910100000000001 2024-04-20T17:39:41.387 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.387 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.387 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 8", 2024-04-20T17:39:41.387 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.314197+0000", 2024-04-20T17:39:41.387 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000145143 2024-04-20T17:39:41.387 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.387 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.387 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:39:41.387 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.320588+0000", 2024-04-20T17:39:41.387 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0063915200000000004 2024-04-20T17:39:41.387 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.387 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.387 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:41.387 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.446510+0000", 2024-04-20T17:39:41.388 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.12592145800000001 2024-04-20T17:39:41.388 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.388 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.388 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:39:41.388 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.446562+0000", 2024-04-20T17:39:41.388 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2089000000000002e-05 2024-04-20T17:39:41.388 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.388 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.388 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:39:41.388 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:40.446613+0000", 2024-04-20T17:39:41.388 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.181e-05 2024-04-20T17:39:41.388 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.388 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:41.388 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.389 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.389 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.389 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:253233 5.6 5:6a18544b:::benchmark_data_smithi171_137281_object121799:head [delete] snapc 0=[] ondisk+write+known_if_redirected e610)", 2024-04-20T17:39:41.389 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:56.422462+0000", 2024-04-20T17:39:41.389 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 404.89504894300001, 2024-04-20T17:39:41.389 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.22659600899999999, 2024-04-20T17:39:41.389 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:41.389 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:41.389 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:41.389 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:41.389 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:39:41.389 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:39:41.389 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 253233 2024-04-20T17:39:41.389 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.390 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:41.390 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.390 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:41.390 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.422462+0000", 2024-04-20T17:39:41.390 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.390 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.390 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.390 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:41.390 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.422462+0000", 2024-04-20T17:39:41.390 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.390 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.390 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.390 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:41.390 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.422465+0000", 2024-04-20T17:39:41.391 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6060000000000001e-06 2024-04-20T17:39:41.391 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.391 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.391 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:41.391 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.422472+0000", 2024-04-20T17:39:41.391 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8680000000000001e-06 2024-04-20T17:39:41.391 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.391 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.391 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:41.391 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.422483+0000", 2024-04-20T17:39:41.391 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1797000000000001e-05 2024-04-20T17:39:41.391 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.391 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.391 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:41.392 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.422490+0000", 2024-04-20T17:39:41.392 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2230000000000004e-06 2024-04-20T17:39:41.392 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.392 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.392 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:39:41.392 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.422540+0000", 2024-04-20T17:39:41.392 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0278999999999998e-05 2024-04-20T17:39:41.392 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.392 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.392 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:39:41.392 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.422638+0000", 2024-04-20T17:39:41.392 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8335e-05 2024-04-20T17:39:41.392 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.392 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.393 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:39:41.393 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.422716+0000", 2024-04-20T17:39:41.393 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7702999999999998e-05 2024-04-20T17:39:41.393 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.393 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.393 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:39:41.393 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.432595+0000", 2024-04-20T17:39:41.393 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0098791590000000002 2024-04-20T17:39:41.393 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.393 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.393 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:41.393 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.648976+0000", 2024-04-20T17:39:41.393 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.21638126999999999 2024-04-20T17:39:41.393 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.394 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.394 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:39:41.394 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.649010+0000", 2024-04-20T17:39:41.394 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3269999999999998e-05 2024-04-20T17:39:41.394 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.394 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.394 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:39:41.394 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.649058+0000", 2024-04-20T17:39:41.394 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8498999999999998e-05 2024-04-20T17:39:41.394 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.394 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:41.394 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.394 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.395 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.395 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:253242 5.6 5:66b7bb4b:::benchmark_data_smithi171_137281_object121808:head [delete] snapc 0=[] ondisk+write+known_if_redirected e610)", 2024-04-20T17:39:41.395 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:56.427676+0000", 2024-04-20T17:39:41.395 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 404.88983455699997, 2024-04-20T17:39:41.395 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.22493042599999999, 2024-04-20T17:39:41.395 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:41.395 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:41.395 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:41.395 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:41.395 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:39:41.395 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:39:41.395 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 253242 2024-04-20T17:39:41.395 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.395 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:41.396 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.396 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:41.396 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.427676+0000", 2024-04-20T17:39:41.396 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.396 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.396 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.396 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:41.396 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.427676+0000", 2024-04-20T17:39:41.396 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.396 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.396 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.396 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:41.396 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.427679+0000", 2024-04-20T17:39:41.396 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1880000000000001e-06 2024-04-20T17:39:41.397 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.397 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.397 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:41.397 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.427685+0000", 2024-04-20T17:39:41.397 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1190000000000002e-06 2024-04-20T17:39:41.397 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.397 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.397 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:41.397 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.427698+0000", 2024-04-20T17:39:41.397 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2909e-05 2024-04-20T17:39:41.397 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.397 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.397 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:41.397 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.427706+0000", 2024-04-20T17:39:41.397 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9889999999999992e-06 2024-04-20T17:39:41.397 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.398 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.398 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:39:41.398 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.427762+0000", 2024-04-20T17:39:41.398 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6514999999999997e-05 2024-04-20T17:39:41.398 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.398 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.398 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:39:41.398 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.427906+0000", 2024-04-20T17:39:41.398 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000143988 2024-04-20T17:39:41.398 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.398 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.398 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:39:41.398 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.428043+0000", 2024-04-20T17:39:41.398 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013689400000000001 2024-04-20T17:39:41.399 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.399 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.399 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:39:41.399 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.432679+0000", 2024-04-20T17:39:41.399 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0046360430000000003 2024-04-20T17:39:41.399 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.399 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.399 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:41.399 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.652522+0000", 2024-04-20T17:39:41.399 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.21984308799999999 2024-04-20T17:39:41.399 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.399 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.399 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:39:41.399 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.652562+0000", 2024-04-20T17:39:41.399 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9517999999999997e-05 2024-04-20T17:39:41.399 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.400 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.400 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:39:41.400 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.652607+0000", 2024-04-20T17:39:41.400 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5175e-05 2024-04-20T17:39:41.400 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.400 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:41.400 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.400 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.400 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.400 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:253270 5.6 5:631ba04b:::benchmark_data_smithi171_137281_object121836:head [delete] snapc 0=[] ondisk+write+known_if_redirected e610)", 2024-04-20T17:39:41.400 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:56.450099+0000", 2024-04-20T17:39:41.400 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 404.867412097, 2024-04-20T17:39:41.400 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.20781598400000001, 2024-04-20T17:39:41.400 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:41.400 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:41.401 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:41.401 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:41.401 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:39:41.401 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:39:41.401 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 253270 2024-04-20T17:39:41.401 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.401 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:41.401 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.401 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:41.401 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.450099+0000", 2024-04-20T17:39:41.401 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.401 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.401 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.401 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:41.401 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.450099+0000", 2024-04-20T17:39:41.402 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.402 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.402 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.402 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:41.402 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.450101+0000", 2024-04-20T17:39:41.402 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5179999999999999e-06 2024-04-20T17:39:41.402 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.402 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.402 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:41.402 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.450109+0000", 2024-04-20T17:39:41.402 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2350000000000002e-06 2024-04-20T17:39:41.402 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.402 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.402 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:41.403 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.450124+0000", 2024-04-20T17:39:41.403 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5304000000000001e-05 2024-04-20T17:39:41.403 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.403 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.403 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:41.403 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.450133+0000", 2024-04-20T17:39:41.403 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7390000000000005e-06 2024-04-20T17:39:41.403 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.403 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.403 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:39:41.403 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.450200+0000", 2024-04-20T17:39:41.403 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7656000000000004e-05 2024-04-20T17:39:41.403 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.403 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.403 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:39:41.404 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.450338+0000", 2024-04-20T17:39:41.404 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000137451 2024-04-20T17:39:41.404 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.404 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.404 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:39:41.404 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.450474+0000", 2024-04-20T17:39:41.404 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013609599999999999 2024-04-20T17:39:41.404 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.404 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.404 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:39:41.404 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.455275+0000", 2024-04-20T17:39:41.404 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.004800627 2024-04-20T17:39:41.404 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.404 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.405 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:41.405 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.657827+0000", 2024-04-20T17:39:41.405 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.20255267599999999 2024-04-20T17:39:41.405 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.405 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.405 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:39:41.405 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.657865+0000", 2024-04-20T17:39:41.405 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8145000000000002e-05 2024-04-20T17:39:41.405 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.405 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.405 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:39:41.405 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.657915+0000", 2024-04-20T17:39:41.405 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9536999999999997e-05 2024-04-20T17:39:41.405 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.405 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:41.406 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.406 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.406 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.406 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:253281 5.6 5:622f9e9c:::benchmark_data_smithi171_137281_object121847:head [delete] snapc 0=[] ondisk+write+known_if_redirected e610)", 2024-04-20T17:39:41.406 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:56.460513+0000", 2024-04-20T17:39:41.406 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 404.85699790299998, 2024-04-20T17:39:41.406 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.19750880000000001, 2024-04-20T17:39:41.406 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:41.406 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:41.406 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:41.406 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:41.406 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:39:41.406 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:39:41.406 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 253281 2024-04-20T17:39:41.407 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.407 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:41.407 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.407 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:41.407 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.460513+0000", 2024-04-20T17:39:41.407 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.407 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.407 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.407 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:41.407 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.460513+0000", 2024-04-20T17:39:41.407 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.407 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.407 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.407 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:41.408 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.460515+0000", 2024-04-20T17:39:41.408 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3429999999999998e-06 2024-04-20T17:39:41.408 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.408 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.408 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:41.408 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.460523+0000", 2024-04-20T17:39:41.408 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1860000000000003e-06 2024-04-20T17:39:41.408 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.408 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.408 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:41.408 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.460536+0000", 2024-04-20T17:39:41.408 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3023999999999999e-05 2024-04-20T17:39:41.408 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.409 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.409 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:41.409 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.460544+0000", 2024-04-20T17:39:41.409 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1149999999999994e-06 2024-04-20T17:39:41.409 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.409 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.409 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:39:41.409 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.460605+0000", 2024-04-20T17:39:41.409 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1480000000000001e-05 2024-04-20T17:39:41.409 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.409 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.409 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:39:41.409 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.460742+0000", 2024-04-20T17:39:41.409 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013681500000000001 2024-04-20T17:39:41.409 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.410 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.410 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:39:41.410 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.460881+0000", 2024-04-20T17:39:41.410 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000139004 2024-04-20T17:39:41.410 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.410 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.410 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:39:41.410 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.471685+0000", 2024-04-20T17:39:41.410 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010803696999999999 2024-04-20T17:39:41.410 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.410 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.410 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:41.410 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.657962+0000", 2024-04-20T17:39:41.410 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.18627682200000001 2024-04-20T17:39:41.411 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.411 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.411 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:39:41.411 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.657995+0000", 2024-04-20T17:39:41.411 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3003999999999998e-05 2024-04-20T17:39:41.411 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.411 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.411 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:39:41.411 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.658022+0000", 2024-04-20T17:39:41.411 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7310000000000001e-05 2024-04-20T17:39:41.411 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.411 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:41.411 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.411 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.412 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.412 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:253283 5.6 5:668e3393:::benchmark_data_smithi171_137281_object121849:head [delete] snapc 0=[] ondisk+write+known_if_redirected e610)", 2024-04-20T17:39:41.412 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:56.461378+0000", 2024-04-20T17:39:41.412 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 404.85613348499999, 2024-04-20T17:39:41.412 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.19674983099999999, 2024-04-20T17:39:41.412 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:41.412 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:41.412 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:41.412 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:41.412 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:39:41.412 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:39:41.412 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 253283 2024-04-20T17:39:41.412 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.412 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:41.413 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.413 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:41.413 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.461378+0000", 2024-04-20T17:39:41.413 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.413 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.413 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.413 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:41.413 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.461378+0000", 2024-04-20T17:39:41.413 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.413 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.413 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.413 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:41.413 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.461379+0000", 2024-04-20T17:39:41.413 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.403e-06 2024-04-20T17:39:41.413 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.414 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.414 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:41.414 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.461384+0000", 2024-04-20T17:39:41.414 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.135e-06 2024-04-20T17:39:41.414 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.414 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.414 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:41.414 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.461396+0000", 2024-04-20T17:39:41.414 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2194999999999999e-05 2024-04-20T17:39:41.414 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.414 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.414 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:41.414 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.461404+0000", 2024-04-20T17:39:41.414 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5109999999999997e-06 2024-04-20T17:39:41.415 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.415 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.415 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:39:41.415 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.467170+0000", 2024-04-20T17:39:41.415 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0057657259999999997 2024-04-20T17:39:41.415 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.415 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.415 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:39:41.415 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.467308+0000", 2024-04-20T17:39:41.415 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013891000000000001 2024-04-20T17:39:41.415 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.415 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.415 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:39:41.415 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.467433+0000", 2024-04-20T17:39:41.415 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012483500000000001 2024-04-20T17:39:41.415 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.416 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.416 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:39:41.416 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.472308+0000", 2024-04-20T17:39:41.416 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0048747540000000002 2024-04-20T17:39:41.416 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.416 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.416 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:41.416 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.658061+0000", 2024-04-20T17:39:41.416 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.18575340700000001 2024-04-20T17:39:41.416 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.416 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.416 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:39:41.416 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.658093+0000", 2024-04-20T17:39:41.417 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1270999999999999e-05 2024-04-20T17:39:41.417 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.417 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.417 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:39:41.417 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.658127+0000", 2024-04-20T17:39:41.417 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4684000000000002e-05 2024-04-20T17:39:41.417 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.417 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:41.417 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.417 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.417 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.417 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:253301 5.6 5:6a3a8ecf:::benchmark_data_smithi171_137281_object121867:head [delete] snapc 0=[] ondisk+write+known_if_redirected e610)", 2024-04-20T17:39:41.417 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:56.474259+0000", 2024-04-20T17:39:41.417 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 404.84325231299999, 2024-04-20T17:39:41.417 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.19229986700000001, 2024-04-20T17:39:41.418 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:41.418 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:41.418 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:41.418 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:41.418 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:39:41.418 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:39:41.418 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 253301 2024-04-20T17:39:41.418 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.418 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:41.418 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.418 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:41.418 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.474259+0000", 2024-04-20T17:39:41.418 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.418 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.419 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.419 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:41.419 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.474259+0000", 2024-04-20T17:39:41.419 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.419 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.419 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.419 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:41.419 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.474261+0000", 2024-04-20T17:39:41.419 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6759999999999999e-06 2024-04-20T17:39:41.419 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.419 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.419 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:41.419 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.474269+0000", 2024-04-20T17:39:41.419 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6040000000000004e-06 2024-04-20T17:39:41.419 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.420 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.420 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:41.420 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.474293+0000", 2024-04-20T17:39:41.420 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3589000000000001e-05 2024-04-20T17:39:41.420 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.420 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.420 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:41.420 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.474302+0000", 2024-04-20T17:39:41.420 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.747e-06 2024-04-20T17:39:41.420 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.420 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.420 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:39:41.420 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.474359+0000", 2024-04-20T17:39:41.420 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6913000000000002e-05 2024-04-20T17:39:41.420 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.420 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.420 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:39:41.421 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.474483+0000", 2024-04-20T17:39:41.421 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012419199999999999 2024-04-20T17:39:41.421 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.421 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.421 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:39:41.421 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.474605+0000", 2024-04-20T17:39:41.421 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000121712 2024-04-20T17:39:41.421 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.421 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.421 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:39:41.421 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.479728+0000", 2024-04-20T17:39:41.421 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0051231050000000002 2024-04-20T17:39:41.421 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.421 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.421 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:41.421 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.666462+0000", 2024-04-20T17:39:41.422 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.186733538 2024-04-20T17:39:41.422 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.422 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.422 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:39:41.422 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.666522+0000", 2024-04-20T17:39:41.422 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0130000000000002e-05 2024-04-20T17:39:41.422 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.422 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.422 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:39:41.422 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.666559+0000", 2024-04-20T17:39:41.422 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6661e-05 2024-04-20T17:39:41.422 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.422 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:41.422 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.422 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.423 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.423 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:253316 5.6 5:672cd2ba:::benchmark_data_smithi171_137281_object121882:head [delete] snapc 0=[] ondisk+write+known_if_redirected e610)", 2024-04-20T17:39:41.423 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:56.488372+0000", 2024-04-20T17:39:41.423 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 404.82913908500001, 2024-04-20T17:39:41.423 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17831455500000001, 2024-04-20T17:39:41.423 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:41.423 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:41.423 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:41.423 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:41.423 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:39:41.423 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:39:41.423 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 253316 2024-04-20T17:39:41.423 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.423 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:41.423 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.424 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:41.424 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.488372+0000", 2024-04-20T17:39:41.424 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.424 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.424 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.424 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:41.424 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.488372+0000", 2024-04-20T17:39:41.424 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.424 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.424 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.424 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:41.424 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.488374+0000", 2024-04-20T17:39:41.424 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.385e-06 2024-04-20T17:39:41.424 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.424 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.425 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:41.425 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.488380+0000", 2024-04-20T17:39:41.425 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4979999999999997e-06 2024-04-20T17:39:41.425 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.425 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.425 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:41.425 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.488393+0000", 2024-04-20T17:39:41.425 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3067999999999999e-05 2024-04-20T17:39:41.425 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.425 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.425 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:41.425 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.488400+0000", 2024-04-20T17:39:41.425 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2989999999999999e-06 2024-04-20T17:39:41.425 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.425 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.426 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:39:41.426 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.488458+0000", 2024-04-20T17:39:41.426 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7785000000000003e-05 2024-04-20T17:39:41.426 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.426 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.426 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:39:41.426 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.488594+0000", 2024-04-20T17:39:41.426 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013595799999999999 2024-04-20T17:39:41.426 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.426 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.426 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:39:41.426 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.488715+0000", 2024-04-20T17:39:41.426 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012079299999999999 2024-04-20T17:39:41.426 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.427 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.427 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:39:41.427 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.493470+0000", 2024-04-20T17:39:41.427 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0047552330000000002 2024-04-20T17:39:41.427 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.427 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.427 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:41.427 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.666619+0000", 2024-04-20T17:39:41.427 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.173149519 2024-04-20T17:39:41.427 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.427 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.427 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:39:41.427 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.666648+0000", 2024-04-20T17:39:41.427 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8915999999999999e-05 2024-04-20T17:39:41.427 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.428 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.428 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:39:41.428 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.666686+0000", 2024-04-20T17:39:41.428 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8101e-05 2024-04-20T17:39:41.428 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.428 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:41.428 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.428 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.428 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.428 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.16506.0:253330 5.6 5:6b7ab629:::benchmark_data_smithi171_137281_object121896:head [delete] snapc 0=[] ondisk+write+known_if_redirected e610)", 2024-04-20T17:39:41.428 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:32:56.499739+0000", 2024-04-20T17:39:41.428 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 404.81777193599999, 2024-04-20T17:39:41.428 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.167089714, 2024-04-20T17:39:41.428 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:39:41.428 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:39:41.429 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:39:41.429 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:39:41.429 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.16506", 2024-04-20T17:39:41.429 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2740601563", 2024-04-20T17:39:41.429 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 253330 2024-04-20T17:39:41.429 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.429 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:39:41.429 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.429 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:39:41.429 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.499739+0000", 2024-04-20T17:39:41.429 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.429 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.429 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.429 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:39:41.429 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.499739+0000", 2024-04-20T17:39:41.429 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:39:41.430 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.430 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.430 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:39:41.430 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.499742+0000", 2024-04-20T17:39:41.430 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5780000000000001e-06 2024-04-20T17:39:41.430 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.430 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.430 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:39:41.430 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.499750+0000", 2024-04-20T17:39:41.430 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6819999999999992e-06 2024-04-20T17:39:41.430 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.430 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.430 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:39:41.430 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.499766+0000", 2024-04-20T17:39:41.431 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5987000000000001e-05 2024-04-20T17:39:41.431 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.431 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.431 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:39:41.431 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.499774+0000", 2024-04-20T17:39:41.431 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5059999999999998e-06 2024-04-20T17:39:41.431 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.431 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.431 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:39:41.431 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.499832+0000", 2024-04-20T17:39:41.431 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8065000000000001e-05 2024-04-20T17:39:41.431 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.431 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.431 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:39:41.431 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.499921+0000", 2024-04-20T17:39:41.432 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9501999999999998e-05 2024-04-20T17:39:41.432 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.432 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.432 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:39:41.432 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.499994+0000", 2024-04-20T17:39:41.432 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2384000000000004e-05 2024-04-20T17:39:41.432 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.432 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.432 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:39:41.432 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.504608+0000", 2024-04-20T17:39:41.432 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0046146870000000001 2024-04-20T17:39:41.432 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.432 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.432 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:39:41.432 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.666734+0000", 2024-04-20T17:39:41.433 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.162125306 2024-04-20T17:39:41.433 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.433 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.433 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:39:41.433 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.666787+0000", 2024-04-20T17:39:41.433 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3316000000000002e-05 2024-04-20T17:39:41.433 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:39:41.433 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:39:41.433 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:39:41.433 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:32:56.666829+0000", 2024-04-20T17:39:41.433 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1700999999999999e-05 2024-04-20T17:39:41.433 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.433 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:41.433 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.434 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:39:41.434 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:39:41.434 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:39:41.434 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:39:41.462 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:41.481 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:39:41.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:41.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:39:41.585 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 93 16 145722 145706 97.8931 0 - 0.00944733 2024-04-20T17:39:41.585 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:39:41.611 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:41 smithi005 ceph-mon[39562]: 7.1 scrub ok 2024-04-20T17:39:41.611 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:41 smithi005 ceph-mon[39562]: 7.f scrub ok 2024-04-20T17:39:41.613 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:41.680 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:39:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:41 smithi106 ceph-mon[44423]: 7.1 scrub ok 2024-04-20T17:39:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:41 smithi106 ceph-mon[44423]: 7.f scrub ok 2024-04-20T17:39:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:41 smithi171 ceph-mon[44189]: 7.1 scrub ok 2024-04-20T17:39:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:41 smithi171 ceph-mon[44189]: 7.f scrub ok 2024-04-20T17:39:42.585 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 94 16 145722 145706 96.8519 0 - 0.00944733 2024-04-20T17:39:42.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:42 smithi171 ceph-mon[44189]: pgmap v1922: 17 pgs: 1 active+clean+scrubbing, 3 undersized+degraded+peered, 13 active+clean; 8.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 25 MiB/s wr, 405 op/s; 18059/291420 objects degraded (6.197%) 2024-04-20T17:39:42.839 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:42 smithi005 ceph-mon[39562]: pgmap v1922: 17 pgs: 1 active+clean+scrubbing, 3 undersized+degraded+peered, 13 active+clean; 8.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 25 MiB/s wr, 405 op/s; 18059/291420 objects degraded (6.197%) 2024-04-20T17:39:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:42 smithi106 ceph-mon[44423]: pgmap v1922: 17 pgs: 1 active+clean+scrubbing, 3 undersized+degraded+peered, 13 active+clean; 8.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 25 MiB/s wr, 405 op/s; 18059/291420 objects degraded (6.197%) 2024-04-20T17:39:43.173 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:39:43.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:43.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:39:43.205 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-20T17:39:43.277 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:39:43.306 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:43.326 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:39:43.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:43.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:39:43.382 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:39:43.429 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:39:43.456 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:43.478 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:39:43.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:43.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:39:43.491 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:39:43.582 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:39:43.585 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 95 16 145722 145706 95.8325 0 - 0.00944733 2024-04-20T17:39:43.613 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:43.821 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:39:43.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:43.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:39:43.837 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7 is already down. 2024-04-20T17:39:43.865 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:43 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mon-a[39539]: 2024-04-20T17:39:43.833+0000 7f73b92a1700 -1 mon.a@0(leader).osd e724 definitely_dead 0 2024-04-20T17:39:43.926 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:39:43.955 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:44.001 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:43 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3341060613' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["7"]}]: dispatch 2024-04-20T17:39:44.204 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:39:44.206 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:43 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3341060613' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["7"]}]: dispatch 2024-04-20T17:39:44.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:44.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:39:44.308 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:39:44.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:43 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3341060613' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["7"]}]: dispatch 2024-04-20T17:39:44.336 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:44.356 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:39:44.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:44.359 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:39:44.460 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:39:44.488 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:44.528 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:39:44.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:44.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:39:44.585 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 96 16 145722 145706 94.8344 0 - 0.00944733 2024-04-20T17:39:44.632 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:39:44.660 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:44.679 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:39:44.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:44.681 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:39:44.782 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:39:44.811 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:45.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:44 smithi106 ceph-mon[44423]: 7.c scrub starts 2024-04-20T17:39:45.257 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:44 smithi106 ceph-mon[44423]: pgmap v1923: 17 pgs: 2 active+clean+scrubbing, 3 undersized+degraded+peered, 12 active+clean; 8.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 11 MiB/s wr, 168 op/s; 18059/291420 objects degraded (6.197%) 2024-04-20T17:39:45.258 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:44 smithi005 ceph-mon[39562]: 7.c scrub starts 2024-04-20T17:39:45.258 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:44 smithi005 ceph-mon[39562]: pgmap v1923: 17 pgs: 2 active+clean+scrubbing, 3 undersized+degraded+peered, 12 active+clean; 8.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 11 MiB/s wr, 168 op/s; 18059/291420 objects degraded (6.197%) 2024-04-20T17:39:45.258 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:44 smithi171 ceph-mon[44189]: 7.c scrub starts 2024-04-20T17:39:45.259 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:44 smithi171 ceph-mon[44189]: pgmap v1923: 17 pgs: 2 active+clean+scrubbing, 3 undersized+degraded+peered, 12 active+clean; 8.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 11 MiB/s wr, 168 op/s; 18059/291420 objects degraded (6.197%) 2024-04-20T17:39:45.382 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:39:45.382 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:39:45.382 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-04-20T17:39:45.383 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:39:45.585 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 97 16 145722 145706 93.8569 0 - 0.00944733 2024-04-20T17:39:45.633 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 2, 3, 4, 5, 8, 9, 10, 11, 7] 2024-04-20T17:39:45.633 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd out 7 2024-04-20T17:39:45.892 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:45 smithi005 ceph-mon[39562]: 7.0 scrub starts 2024-04-20T17:39:45.892 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:45 smithi005 ceph-mon[39562]: 7.c scrub ok 2024-04-20T17:39:45.892 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:45 smithi005 ceph-mon[39562]: 7.b scrub starts 2024-04-20T17:39:45.892 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:45 smithi005 ceph-mon[39562]: 7.0 scrub ok 2024-04-20T17:39:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:45 smithi106 ceph-mon[44423]: 7.0 scrub starts 2024-04-20T17:39:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:45 smithi106 ceph-mon[44423]: 7.c scrub ok 2024-04-20T17:39:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:45 smithi106 ceph-mon[44423]: 7.b scrub starts 2024-04-20T17:39:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:45 smithi106 ceph-mon[44423]: 7.0 scrub ok 2024-04-20T17:39:46.037 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:39:46.063 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:39:46.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:46.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:39:46.167 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:39:46.204 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:46.217 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-20T17:39:46.225 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:39:46.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:46.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:39:46.328 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:39:46.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:45 smithi171 ceph-mon[44189]: 7.0 scrub starts 2024-04-20T17:39:46.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:45 smithi171 ceph-mon[44189]: 7.c scrub ok 2024-04-20T17:39:46.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:45 smithi171 ceph-mon[44189]: 7.b scrub starts 2024-04-20T17:39:46.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:45 smithi171 ceph-mon[44189]: 7.0 scrub ok 2024-04-20T17:39:46.358 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:46.483 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:39:46.490 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:39:46.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:46.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:39:46.585 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 98 16 145722 145706 92.8994 0 - 0.00944733 2024-04-20T17:39:46.595 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:39:46.622 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:46.719 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:39:46.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:46.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:39:46.826 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:39:46.852 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:46.890 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:39:46.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:46.892 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:39:46.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:46 smithi005 ceph-mon[39562]: 7.b scrub ok 2024-04-20T17:39:46.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:46 smithi005 ceph-mon[39562]: pgmap v1924: 17 pgs: 1 active+clean+scrubbing, 3 undersized+degraded+peered, 13 active+clean; 8.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 4.3 MiB/s wr, 68 op/s; 18059/291420 objects degraded (6.197%) 2024-04-20T17:39:46.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:46 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 18059/291420 objects degraded (6.197%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:39:46.993 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:39:47.020 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:47.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:46 smithi106 ceph-mon[44423]: 7.b scrub ok 2024-04-20T17:39:47.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:46 smithi106 ceph-mon[44423]: pgmap v1924: 17 pgs: 1 active+clean+scrubbing, 3 undersized+degraded+peered, 13 active+clean; 8.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 4.3 MiB/s wr, 68 op/s; 18059/291420 objects degraded (6.197%) 2024-04-20T17:39:47.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:46 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 18059/291420 objects degraded (6.197%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:39:47.268 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:39:47.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:47.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:39:47.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:46 smithi171 ceph-mon[44189]: 7.b scrub ok 2024-04-20T17:39:47.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:46 smithi171 ceph-mon[44189]: pgmap v1924: 17 pgs: 1 active+clean+scrubbing, 3 undersized+degraded+peered, 13 active+clean; 8.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 4.3 MiB/s wr, 68 op/s; 18059/291420 objects degraded (6.197%) 2024-04-20T17:39:47.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:46 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 18059/291420 objects degraded (6.197%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:39:47.372 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:39:47.403 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:47.527 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:39:47.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:47.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:39:47.607 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:39:47.621 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 99 16 145722 145706 91.9612 0 - 0.00944733 2024-04-20T17:39:47.632 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:39:47.664 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:47.686 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:39:47.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:47.689 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:39:47.790 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:39:47.820 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:47.861 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:39:47.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:47.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:39:47.909 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:39:47.966 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:39:48.000 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:48.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:47 smithi171 ceph-mon[44189]: 7.a scrub starts 2024-04-20T17:39:48.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:47 smithi171 ceph-mon[44189]: 7.a scrub ok 2024-04-20T17:39:48.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:47 smithi171 ceph-mon[44189]: 7.d scrub starts 2024-04-20T17:39:48.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:47 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1702401636' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:39:48.104 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:47 smithi106 ceph-mon[44423]: 7.a scrub starts 2024-04-20T17:39:48.104 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:47 smithi106 ceph-mon[44423]: 7.a scrub ok 2024-04-20T17:39:48.104 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:47 smithi106 ceph-mon[44423]: 7.d scrub starts 2024-04-20T17:39:48.104 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:47 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1702401636' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:39:48.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:47 smithi005 ceph-mon[39562]: 7.a scrub starts 2024-04-20T17:39:48.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:47 smithi005 ceph-mon[39562]: 7.a scrub ok 2024-04-20T17:39:48.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:47 smithi005 ceph-mon[39562]: 7.d scrub starts 2024-04-20T17:39:48.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:47 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1702401636' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:39:48.412 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:39:48.412 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:39:48.412 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-04-20T17:39:48.412 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-04-20T17:39:48.412 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:39:48.621 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:39:48.622445 min lat: 0.00100541 max lat: 0.210579 avg lat: 0.00944733 2024-04-20T17:39:48.622 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:39:48.622 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 100 16 145722 145706 91.0091 0 - 0.00944733 2024-04-20T17:39:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:48 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:39:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:48 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1702401636' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:39:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:48 smithi106 ceph-mon[44423]: osdmap e725: 12 total, 11 up, 10 in 2024-04-20T17:39:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:48 smithi106 ceph-mon[44423]: 7.d scrub ok 2024-04-20T17:39:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:48 smithi106 ceph-mon[44423]: pgmap v1926: 17 pgs: 1 active+clean+scrubbing, 3 undersized+degraded+peered, 13 active+clean; 8.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 1.8 MiB/s wr, 28 op/s; 18059/291420 objects degraded (6.197%) 2024-04-20T17:39:49.140 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-20T17:39:49.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:48 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:39:49.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:48 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1702401636' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:39:49.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:48 smithi005 ceph-mon[39562]: osdmap e725: 12 total, 11 up, 10 in 2024-04-20T17:39:49.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:48 smithi005 ceph-mon[39562]: 7.d scrub ok 2024-04-20T17:39:49.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:48 smithi005 ceph-mon[39562]: pgmap v1926: 17 pgs: 1 active+clean+scrubbing, 3 undersized+degraded+peered, 13 active+clean; 8.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 1.8 MiB/s wr, 28 op/s; 18059/291420 objects degraded (6.197%) 2024-04-20T17:39:49.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:48 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:39:49.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:48 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1702401636' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:39:49.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:48 smithi171 ceph-mon[44189]: osdmap e725: 12 total, 11 up, 10 in 2024-04-20T17:39:49.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:48 smithi171 ceph-mon[44189]: 7.d scrub ok 2024-04-20T17:39:49.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:48 smithi171 ceph-mon[44189]: pgmap v1926: 17 pgs: 1 active+clean+scrubbing, 3 undersized+degraded+peered, 13 active+clean; 8.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 1.8 MiB/s wr, 28 op/s; 18059/291420 objects degraded (6.197%) 2024-04-20T17:39:49.403 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:39:49.441 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:39:49.461 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:39:49.480 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:39:49.500 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:39:49.521 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:39:49.541 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:39:49.561 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:39:49.586 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:39:49.618 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:39:49.621 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 101 16 145722 145706 90.1084 0 - 0.00944733 2024-04-20T17:39:49.645 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:39:49.667 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:39:49.667 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:39:49.667 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:39:49.667 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:39:49.667 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:39:49.668 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:39:49.668 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:39:49.668 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:39:49.668 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:39:49.668 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:39:49.668 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:39:49.668 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:39:49.668 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:39:49.941 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:49 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3811247245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:39:49.941 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:49.941 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:49.941 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:49.941 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:49 smithi106 ceph-mon[44423]: 7.4 scrub starts 2024-04-20T17:39:49.941 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:49.941 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:49.941 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:49.941 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:50.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:49 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3811247245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:39:50.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:50.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:50.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:50.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:49 smithi005 ceph-mon[39562]: 7.4 scrub starts 2024-04-20T17:39:50.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:50.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:50.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:50.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:49 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3811247245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:39:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:49 smithi171 ceph-mon[44189]: 7.4 scrub starts 2024-04-20T17:39:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:50.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:50.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:50.622 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 102 16 145722 145706 89.2254 0 - 0.00944733 2024-04-20T17:39:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:50 smithi106 ceph-mon[44423]: 7.4 scrub ok 2024-04-20T17:39:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:50 smithi106 ceph-mon[44423]: pgmap v1927: 17 pgs: 1 active+clean+scrubbing, 3 undersized+degraded+peered, 13 active+clean; 8.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 18059/291420 objects degraded (6.197%) 2024-04-20T17:39:51.058 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:39:51.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:51.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:39:51.164 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:39:51.194 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:51.194 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:51.195 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:51.195 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:51.195 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:50 smithi171 ceph-mon[44189]: 7.4 scrub ok 2024-04-20T17:39:51.195 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:50 smithi171 ceph-mon[44189]: pgmap v1927: 17 pgs: 1 active+clean+scrubbing, 3 undersized+degraded+peered, 13 active+clean; 8.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 18059/291420 objects degraded (6.197%) 2024-04-20T17:39:51.196 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:51.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:51.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:51.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:51.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:39:51.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:50 smithi005 ceph-mon[39562]: 7.4 scrub ok 2024-04-20T17:39:51.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:50 smithi005 ceph-mon[39562]: pgmap v1927: 17 pgs: 1 active+clean+scrubbing, 3 undersized+degraded+peered, 13 active+clean; 8.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 18059/291420 objects degraded (6.197%) 2024-04-20T17:39:51.218 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:39:51.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:51.222 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:39:51.323 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:39:51.337 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:39:51.337 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:39:51.357 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:51.510 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:39:51.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:51.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:39:51.614 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:39:51.622 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 103 16 145722 145706 88.3595 0 - 0.00944733 2024-04-20T17:39:51.645 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:52.053 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-20T17:39:52.055 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:39:52.056 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:39:52.496 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:39:52.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:52.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:39:52.600 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:39:52.622 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 104 16 145722 145706 87.5103 0 - 0.00944733 2024-04-20T17:39:52.633 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:52.650 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:39:52.653 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:39:52.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:52.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:39:52.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:52 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:39:52.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:52 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:39:52.758 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:39:52.789 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:52 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:39:52.789 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:52 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:39:52.791 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:52.812 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:39:52.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:52.815 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:39:52.916 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:39:52.944 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:52 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:39:52.944 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:52 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:39:52.946 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:52.966 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:39:52.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:52.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:39:53.071 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:39:53.100 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:53.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:52 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mon-a[39539]: 2024-04-20T17:39:52.849+0000 7f73b92a1700 -1 mon.a@0(leader).osd e726 definitely_dead 0 2024-04-20T17:39:53.467 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:39:53.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:53.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:39:53.530 INFO:teuthology.orchestra.run.smithi005.stderr:marked out osd.7. 2024-04-20T17:39:53.572 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:39:53.606 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:53.622 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 105 16 145722 145706 86.6772 0 - 0.00944733 2024-04-20T17:39:53.627 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:39:53.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:53.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:39:53.731 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:39:53.760 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:53.780 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:39:53.782 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:53 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:39:53.782 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:53 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:39:53.782 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:53 smithi106 ceph-mon[44423]: pgmap v1928: 17 pgs: 1 active+clean+scrubbing, 3 undersized+degraded+peered, 13 active+clean; 8.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 18059/291420 objects degraded (6.197%) 2024-04-20T17:39:53.782 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:53 smithi106 ceph-mon[44423]: osdmap e726: 12 total, 11 up, 10 in 2024-04-20T17:39:53.782 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:53 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2388043359' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-04-20T17:39:53.782 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:53 smithi106 ceph-mon[44423]: from='client.27323 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-04-20T17:39:53.782 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:53 smithi106 ceph-mon[44423]: Client client.admin marked osd.7 out, after it was down for 18 seconds 2024-04-20T17:39:53.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:53.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:39:53.831 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:53 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:39:53.831 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:53 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:39:53.831 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:53 smithi005 ceph-mon[39562]: pgmap v1928: 17 pgs: 1 active+clean+scrubbing, 3 undersized+degraded+peered, 13 active+clean; 8.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 18059/291420 objects degraded (6.197%) 2024-04-20T17:39:53.831 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:53 smithi005 ceph-mon[39562]: osdmap e726: 12 total, 11 up, 10 in 2024-04-20T17:39:53.831 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:53 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2388043359' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-04-20T17:39:53.831 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:53 smithi005 ceph-mon[39562]: from='client.27323 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-04-20T17:39:53.831 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:53 smithi005 ceph-mon[39562]: Client client.admin marked osd.7 out, after it was down for 18 seconds 2024-04-20T17:39:53.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:53 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:39:53.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:53 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:39:53.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:53 smithi171 ceph-mon[44189]: pgmap v1928: 17 pgs: 1 active+clean+scrubbing, 3 undersized+degraded+peered, 13 active+clean; 8.9 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 18059/291420 objects degraded (6.197%) 2024-04-20T17:39:53.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:53 smithi171 ceph-mon[44189]: osdmap e726: 12 total, 11 up, 10 in 2024-04-20T17:39:53.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:53 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2388043359' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-04-20T17:39:53.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:53 smithi171 ceph-mon[44189]: from='client.27323 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-04-20T17:39:53.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:53 smithi171 ceph-mon[44189]: Client client.admin marked osd.7 out, after it was down for 18 seconds 2024-04-20T17:39:53.884 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:39:53.913 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:53.933 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:39:53.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:53.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:39:54.037 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:39:54.057 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:39:54.069 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:54.091 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:39:54.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:54.094 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:39:54.195 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:39:54.225 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:54.245 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:39:54.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:54.248 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:39:54.349 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:39:54.377 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:54.398 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:39:54.450 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:39:54.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:54.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:39:54.554 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:39:54.583 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:54.604 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:39:54.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:54.607 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:39:54.622 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 106 16 145722 145706 85.86 0 - 0.00944733 2024-04-20T17:39:54.708 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:39:54.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:54 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:39:54.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:54 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:39:54.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:54 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:39:54.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:54 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:39:54.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:54 smithi005 ceph-mon[39562]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T17:39:54.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:54 smithi005 ceph-mon[39562]: from='client.27323 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-04-20T17:39:54.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:54 smithi005 ceph-mon[39562]: osdmap e727: 12 total, 11 up, 9 in 2024-04-20T17:39:54.736 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:54.755 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:39:54.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:54.758 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:39:54.781 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.7 2024-04-20T17:39:54.782 INFO:tasks.thrashosds.thrasher:remote for osd 7 is ubuntu@smithi106.front.sepia.ceph.com 2024-04-20T17:39:54.782 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.7 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op list-pgs 2024-04-20T17:39:54.812 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:54 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:39:54.812 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:54 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:39:54.813 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:54 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:39:54.813 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:54 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:39:54.813 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:54 smithi106 ceph-mon[44423]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T17:39:54.813 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:54 smithi106 ceph-mon[44423]: from='client.27323 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-04-20T17:39:54.813 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:54 smithi106 ceph-mon[44423]: osdmap e727: 12 total, 11 up, 9 in 2024-04-20T17:39:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:54 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:39:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:54 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:39:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:54 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:39:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:54 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:39:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:54 smithi171 ceph-mon[44189]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T17:39:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:54 smithi171 ceph-mon[44189]: from='client.27323 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-04-20T17:39:54.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:54 smithi171 ceph-mon[44189]: osdmap e727: 12 total, 11 up, 9 in 2024-04-20T17:39:54.859 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:39:54.887 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:54.913 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:39:54.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:54.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:39:55.014 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.7/config 2024-04-20T17:39:55.017 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:39:55.120 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:55.160 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:39:55.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:55.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:39:55.274 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:39:55.306 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:55.324 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:39:55.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:55.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:39:55.409 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:39:55.409 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:39:55.429 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:39:55.460 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:55.622 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 107 15 145723 145708 85.0591 0.00625 0.00508759 0.00958736 2024-04-20T17:39:55.657 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:39:55.660 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:39:55.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:55 smithi005 ceph-mon[39562]: pgmap v1931: 17 pgs: 1 active+clean+scrubbing, 3 undersized+degraded+peered, 13 active+clean; 8.9 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 18059/291420 objects degraded (6.197%) 2024-04-20T17:39:55.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:55 smithi005 ceph-mon[39562]: osdmap e728: 12 total, 11 up, 9 in 2024-04-20T17:39:55.809 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:39:55.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:55.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:39:55.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:55 smithi171 ceph-mon[44189]: pgmap v1931: 17 pgs: 1 active+clean+scrubbing, 3 undersized+degraded+peered, 13 active+clean; 8.9 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 18059/291420 objects degraded (6.197%) 2024-04-20T17:39:55.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:55 smithi171 ceph-mon[44189]: osdmap e728: 12 total, 11 up, 9 in 2024-04-20T17:39:55.915 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:39:55.948 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:55.969 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:39:55.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:55.972 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:39:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:55 smithi106 ceph-mon[44423]: pgmap v1931: 17 pgs: 1 active+clean+scrubbing, 3 undersized+degraded+peered, 13 active+clean; 8.9 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 18059/291420 objects degraded (6.197%) 2024-04-20T17:39:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:55 smithi106 ceph-mon[44423]: osdmap e728: 12 total, 11 up, 9 in 2024-04-20T17:39:56.073 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:39:56.102 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:56.122 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:39:56.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:56.125 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:39:56.226 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:39:56.260 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:56.478 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-20T17:39:56.540 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:56 smithi005 ceph-mon[39562]: osdmap e729: 12 total, 11 up, 9 in 2024-04-20T17:39:56.540 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:56 smithi005 ceph-mon[39562]: 7.8 starting backfill to osd.4 from (0'0,0'0] MAX to 721'12632 2024-04-20T17:39:56.545 INFO:teuthology.orchestra.run.smithi106.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:39:56.622 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 108 15 145723 145708 84.2719 0 - 0.00958736 2024-04-20T17:39:56.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:56 smithi106 ceph-mon[44423]: osdmap e729: 12 total, 11 up, 9 in 2024-04-20T17:39:56.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:56 smithi106 ceph-mon[44423]: 7.8 starting backfill to osd.4 from (0'0,0'0] MAX to 721'12632 2024-04-20T17:39:56.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:56 smithi171 ceph-mon[44189]: osdmap e729: 12 total, 11 up, 9 in 2024-04-20T17:39:56.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:56 smithi171 ceph-mon[44189]: 7.8 starting backfill to osd.4 from (0'0,0'0] MAX to 721'12632 2024-04-20T17:39:56.914 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:39:56.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:56.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:39:57.026 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:39:57.057 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:57.181 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:39:57.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:39:57.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:39:57.285 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:39:57.318 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:39:57.623 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 109 15 145723 145708 83.4991 0 - 0.00958736 2024-04-20T17:39:57.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:57 smithi005 ceph-mon[39562]: pgmap v1934: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 8.9 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 144 KiB/s wr, 2 op/s; 18054/291438 objects degraded (6.195%); 80 KiB/s, 1 objects/s recovering 2024-04-20T17:39:57.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:57 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 18054/291438 objects degraded (6.195%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:39:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:57 smithi171 ceph-mon[44189]: pgmap v1934: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 8.9 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 144 KiB/s wr, 2 op/s; 18054/291438 objects degraded (6.195%); 80 KiB/s, 1 objects/s recovering 2024-04-20T17:39:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:57 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 18054/291438 objects degraded (6.195%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:39:58.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:57 smithi106 ceph-mon[44423]: pgmap v1934: 17 pgs: 1 active+recovering+degraded, 2 undersized+degraded+peered, 14 active+clean; 8.9 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 144 KiB/s wr, 2 op/s; 18054/291438 objects degraded (6.195%); 80 KiB/s, 1 objects/s recovering 2024-04-20T17:39:58.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:57 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 18054/291438 objects degraded (6.195%), 3 pgs degraded (PG_DEGRADED) 2024-04-20T17:39:58.503 INFO:teuthology.orchestra.run.smithi106.stdout:7.8 2024-04-20T17:39:58.503 INFO:teuthology.orchestra.run.smithi106.stdout:1.0 2024-04-20T17:39:58.503 INFO:teuthology.orchestra.run.smithi106.stdout:7.6 2024-04-20T17:39:58.623 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 110 15 145723 145708 82.7403 0 - 0.00958736 2024-04-20T17:39:59.623 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 111 15 145723 145708 81.9952 0 - 0.00958736 2024-04-20T17:39:59.680 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.7 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op export --pgid 1.0 --file /var/log/ceph/exp.1.0.7 2024-04-20T17:39:59.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:39:59 smithi171 ceph-mon[44189]: pgmap v1935: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 14 active+clean; 8.9 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 97 KiB/s wr, 1 op/s; 17174/291438 objects degraded (5.893%); 9.3 MiB/s, 148 objects/s recovering 2024-04-20T17:39:59.906 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.7/config 2024-04-20T17:39:59.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:39:59 smithi005 ceph-mon[39562]: pgmap v1935: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 14 active+clean; 8.9 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 97 KiB/s wr, 1 op/s; 17174/291438 objects degraded (5.893%); 9.3 MiB/s, 148 objects/s recovering 2024-04-20T17:40:00.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:39:59 smithi106 ceph-mon[44423]: pgmap v1935: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+peered, 1 undersized+degraded+remapped+backfilling+peered, 14 active+clean; 8.9 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 97 KiB/s wr, 1 op/s; 17174/291438 objects degraded (5.893%); 9.3 MiB/s, 148 objects/s recovering 2024-04-20T17:40:00.623 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 112 15 145723 145708 81.2635 0 - 0.00958736 2024-04-20T17:40:00.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:00 smithi106 ceph-mon[44423]: Health detail: HEALTH_WARN Degraded data redundancy: 17174/291438 objects degraded (5.893%), 3 pgs degraded 2024-04-20T17:40:00.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:00 smithi106 ceph-mon[44423]: [WRN] PG_DEGRADED: Degraded data redundancy: 17174/291438 objects degraded (5.893%), 3 pgs degraded 2024-04-20T17:40:00.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:00 smithi106 ceph-mon[44423]: pg 1.0 is undersized+degraded+peered, acting [11] 2024-04-20T17:40:00.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:00 smithi106 ceph-mon[44423]: pg 7.6 is active+recovering+degraded, acting [0,3] 2024-04-20T17:40:00.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:00 smithi106 ceph-mon[44423]: pg 7.8 is undersized+degraded+remapped+backfilling+peered, acting [2] 2024-04-20T17:40:00.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:00 smithi171 ceph-mon[44189]: Health detail: HEALTH_WARN Degraded data redundancy: 17174/291438 objects degraded (5.893%), 3 pgs degraded 2024-04-20T17:40:00.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:00 smithi171 ceph-mon[44189]: [WRN] PG_DEGRADED: Degraded data redundancy: 17174/291438 objects degraded (5.893%), 3 pgs degraded 2024-04-20T17:40:00.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:00 smithi171 ceph-mon[44189]: pg 1.0 is undersized+degraded+peered, acting [11] 2024-04-20T17:40:00.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:00 smithi171 ceph-mon[44189]: pg 7.6 is active+recovering+degraded, acting [0,3] 2024-04-20T17:40:00.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:00 smithi171 ceph-mon[44189]: pg 7.8 is undersized+degraded+remapped+backfilling+peered, acting [2] 2024-04-20T17:40:00.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:00 smithi005 ceph-mon[39562]: Health detail: HEALTH_WARN Degraded data redundancy: 17174/291438 objects degraded (5.893%), 3 pgs degraded 2024-04-20T17:40:00.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:00 smithi005 ceph-mon[39562]: [WRN] PG_DEGRADED: Degraded data redundancy: 17174/291438 objects degraded (5.893%), 3 pgs degraded 2024-04-20T17:40:00.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:00 smithi005 ceph-mon[39562]: pg 1.0 is undersized+degraded+peered, acting [11] 2024-04-20T17:40:00.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:00 smithi005 ceph-mon[39562]: pg 7.6 is active+recovering+degraded, acting [0,3] 2024-04-20T17:40:00.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:00 smithi005 ceph-mon[39562]: pg 7.8 is undersized+degraded+remapped+backfilling+peered, acting [2] 2024-04-20T17:40:01.503 INFO:teuthology.orchestra.run.smithi106.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:40:01.623 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 113 15 145723 145708 80.5446 0 - 0.00958736 2024-04-20T17:40:01.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:01 smithi005 ceph-mon[39562]: pgmap v1936: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 8.9 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 83 KiB/s wr, 1 op/s; 16929/291438 objects degraded (5.809%); 10 MiB/s, 162 objects/s recovering 2024-04-20T17:40:01.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:01 smithi106 ceph-mon[44423]: pgmap v1936: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 8.9 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 83 KiB/s wr, 1 op/s; 16929/291438 objects degraded (5.809%); 10 MiB/s, 162 objects/s recovering 2024-04-20T17:40:01.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:01 smithi171 ceph-mon[44189]: pgmap v1936: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 8.9 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 83 KiB/s wr, 1 op/s; 16929/291438 objects degraded (5.809%); 10 MiB/s, 162 objects/s recovering 2024-04-20T17:40:02.623 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 114 15 145723 145708 79.8384 0 - 0.00958736 2024-04-20T17:40:02.758 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:40:02.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:02.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:40:02.864 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:40:02.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:02 smithi005 ceph-mon[39562]: pgmap v1937: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 8.9 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 72 KiB/s wr, 1 op/s; 13435/291438 objects degraded (4.610%); 36 MiB/s, 578 objects/s recovering 2024-04-20T17:40:02.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:02 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 13435/291438 objects degraded (4.610%), 2 pgs degraded (PG_DEGRADED) 2024-04-20T17:40:02.897 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:02 smithi106 ceph-mon[44423]: pgmap v1937: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 8.9 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 72 KiB/s wr, 1 op/s; 13435/291438 objects degraded (4.610%); 36 MiB/s, 578 objects/s recovering 2024-04-20T17:40:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:02 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 13435/291438 objects degraded (4.610%), 2 pgs degraded (PG_DEGRADED) 2024-04-20T17:40:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:02 smithi171 ceph-mon[44189]: pgmap v1937: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 8.9 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 72 KiB/s wr, 1 op/s; 13435/291438 objects degraded (4.610%); 36 MiB/s, 578 objects/s recovering 2024-04-20T17:40:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:02 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 13435/291438 objects degraded (4.610%), 2 pgs degraded (PG_DEGRADED) 2024-04-20T17:40:03.616 INFO:teuthology.orchestra.run.smithi106.stderr:Exporting 1.0 info 1.0( v 15'137 (0'0,15'137] local-lis/les=414/415 n=2 ec=14/14 lis/c=414/414 les/c/f=415/415/0 sis=414) 2024-04-20T17:40:03.623 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 115 15 145723 145708 79.1444 0 - 0.00958736 2024-04-20T17:40:03.659 INFO:teuthology.orchestra.run.smithi106.stderr:Read #1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head# 2024-04-20T17:40:03.660 INFO:teuthology.orchestra.run.smithi106.stderr:Read #1:30306672:devicehealth::main.db.0000000000000000:head# 2024-04-20T17:40:03.661 INFO:teuthology.orchestra.run.smithi106.stderr:Export successful 2024-04-20T17:40:04.365 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:40:04.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:04.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:40:04.473 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:40:04.508 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:04.531 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:40:04.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:04.534 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:40:04.624 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 116 15 145723 145708 78.4624 0 - 0.00958736 2024-04-20T17:40:04.635 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:40:04.667 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:04.741 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:40:04.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:04.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:40:04.764 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.7 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --force --op remove --pgid 1.0 2024-04-20T17:40:04.846 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:40:04.878 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:04.898 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:40:04.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:04.901 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:40:05.002 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:40:05.030 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.7/config 2024-04-20T17:40:05.031 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:05.057 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:40:05.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:05.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:40:05.160 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:40:05.190 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:05.210 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:40:05.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:05.213 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:40:05.314 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:40:05.345 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:05.368 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:40:05.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:05.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:40:05.422 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:05 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:40:05.422 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:05 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:40:05.422 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:05 smithi005 ceph-mon[39562]: osdmap e730: 12 total, 11 up, 9 in 2024-04-20T17:40:05.471 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:40:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:05 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:40:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:05 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:40:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:05 smithi171 ceph-mon[44189]: osdmap e730: 12 total, 11 up, 9 in 2024-04-20T17:40:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:05 smithi171 ceph-mon[44189]: pgmap v1939: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 8.9 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 13435/291438 objects degraded (4.610%); 32 MiB/s, 515 objects/s recovering 2024-04-20T17:40:05.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:05 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:40:05.584 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:05.624 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 117 15 145723 145708 77.7921 0 - 0.00958736 2024-04-20T17:40:05.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:05 smithi005 ceph-mon[39562]: pgmap v1939: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 8.9 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 13435/291438 objects degraded (4.610%); 32 MiB/s, 515 objects/s recovering 2024-04-20T17:40:05.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:05 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:40:05.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:05 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:40:05.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:05 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:40:05.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:05 smithi106 ceph-mon[44423]: osdmap e730: 12 total, 11 up, 9 in 2024-04-20T17:40:05.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:05 smithi106 ceph-mon[44423]: pgmap v1939: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 8.9 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 13435/291438 objects degraded (4.610%); 32 MiB/s, 515 objects/s recovering 2024-04-20T17:40:05.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:05 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:40:06.441 INFO:teuthology.orchestra.run.smithi106.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:40:06.624 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 118 15 145723 145708 77.1331 0 - 0.00958736 2024-04-20T17:40:07.624 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 119 15 145723 145708 76.4852 0 - 0.00958736 2024-04-20T17:40:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:07 smithi171 ceph-mon[44189]: pgmap v1940: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 8.9 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 13181/291438 objects degraded (4.523%); 30 MiB/s, 487 objects/s recovering 2024-04-20T17:40:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:07 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:40:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:07 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:40:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:07 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:40:07.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:07 smithi005 ceph-mon[39562]: pgmap v1940: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 8.9 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 13181/291438 objects degraded (4.523%); 30 MiB/s, 487 objects/s recovering 2024-04-20T17:40:07.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:07 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:40:07.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:07 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:40:07.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:07 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:40:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:07 smithi106 ceph-mon[44423]: pgmap v1940: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 8.9 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 13181/291438 objects degraded (4.523%); 30 MiB/s, 487 objects/s recovering 2024-04-20T17:40:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:07 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:40:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:07 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:40:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:07 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:40:08.624 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:40:08.625270 min lat: 0.00100541 max lat: 20.4176 avg lat: 0.00958736 2024-04-20T17:40:08.624 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:40:08.625 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 120 15 145723 145708 75.8481 0 - 0.00958736 2024-04-20T17:40:08.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:08 smithi005 ceph-mon[39562]: pgmap v1941: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 8.9 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 9606/291438 objects degraded (3.296%); 47 MiB/s, 756 objects/s recovering 2024-04-20T17:40:08.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:08 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 9606/291438 objects degraded (3.296%), 2 pgs degraded (PG_DEGRADED) 2024-04-20T17:40:08.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:08 smithi005 ceph-mon[39562]: osdmap e731: 12 total, 11 up, 9 in 2024-04-20T17:40:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:08 smithi106 ceph-mon[44423]: pgmap v1941: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 8.9 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 9606/291438 objects degraded (3.296%); 47 MiB/s, 756 objects/s recovering 2024-04-20T17:40:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:08 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 9606/291438 objects degraded (3.296%), 2 pgs degraded (PG_DEGRADED) 2024-04-20T17:40:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:08 smithi106 ceph-mon[44423]: osdmap e731: 12 total, 11 up, 9 in 2024-04-20T17:40:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:08 smithi171 ceph-mon[44189]: pgmap v1941: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 8.9 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 9606/291438 objects degraded (3.296%); 47 MiB/s, 756 objects/s recovering 2024-04-20T17:40:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:08 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 9606/291438 objects degraded (3.296%), 2 pgs degraded (PG_DEGRADED) 2024-04-20T17:40:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:08 smithi171 ceph-mon[44189]: osdmap e731: 12 total, 11 up, 9 in 2024-04-20T17:40:09.185 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:40:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:40:09.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:09.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:40:09.201 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:40:09.223 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:40:09.243 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:40:09.272 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:40:09.293 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:40:09.295 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:40:09.320 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:40:09.325 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:09.342 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:40:09.345 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:40:09.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:09.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:40:09.366 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:40:09.389 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:40:09.410 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:40:09.411 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:40:09.411 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:40:09.412 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:40:09.412 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:40:09.412 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:40:09.412 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:40:09.412 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:40:09.412 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:40:09.412 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:40:09.412 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:40:09.412 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:40:09.412 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:40:09.412 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:40:09.449 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:40:09.482 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:09.491 INFO:teuthology.orchestra.run.smithi106.stdout: marking collection for removal 2024-04-20T17:40:09.491 INFO:teuthology.orchestra.run.smithi106.stdout:setting '_remove' omap key 2024-04-20T17:40:09.491 INFO:teuthology.orchestra.run.smithi106.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-04-20T17:40:09.503 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:40:09.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:09.506 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:40:09.563 INFO:teuthology.orchestra.run.smithi106.stdout:Remove successful 2024-04-20T17:40:09.607 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:40:09.624 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Total time run: 121.02 2024-04-20T17:40:09.625 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Total writes made: 145723 2024-04-20T17:40:09.625 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Write size: 65536 2024-04-20T17:40:09.625 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Object size: 65536 2024-04-20T17:40:09.625 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Bandwidth (MB/sec): 75.2574 2024-04-20T17:40:09.625 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Stddev Bandwidth: 66.9577 2024-04-20T17:40:09.625 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Max bandwidth (MB/sec): 301.875 2024-04-20T17:40:09.625 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Min bandwidth (MB/sec): 0 2024-04-20T17:40:09.625 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Average IOPS: 1204 2024-04-20T17:40:09.625 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Stddev IOPS: 1071.32 2024-04-20T17:40:09.625 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Max IOPS: 4830 2024-04-20T17:40:09.625 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Min IOPS: 0 2024-04-20T17:40:09.625 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Average Latency(s): 0.0131864 2024-04-20T17:40:09.626 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Stddev Latency(s): 0.358894 2024-04-20T17:40:09.626 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Max latency(s): 35.0356 2024-04-20T17:40:09.626 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Min latency(s): 0.00100541 2024-04-20T17:40:09.629 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Cleaning up (deleting benchmark objects) 2024-04-20T17:40:09.651 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:09.672 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:40:09.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:09.680 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:40:09.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:09 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1522628168' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:40:09.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:09.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:09 smithi005 ceph-mon[39562]: osdmap e732: 12 total, 11 up, 9 in 2024-04-20T17:40:09.780 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:40:09.809 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:09.827 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:40:09.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:09.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:40:09.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:09 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1522628168' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:40:09.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:09.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:09 smithi171 ceph-mon[44189]: osdmap e732: 12 total, 11 up, 9 in 2024-04-20T17:40:09.933 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:40:09.964 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:09.988 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:40:09.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:09.991 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:40:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:09 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1522628168' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:40:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:09 smithi106 ceph-mon[44423]: osdmap e732: 12 total, 11 up, 9 in 2024-04-20T17:40:10.092 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:40:10.120 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:10.140 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:40:10.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:10.147 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:40:10.248 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:40:10.276 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:10.308 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:40:10.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:10.311 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:40:10.412 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:40:10.441 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:10.616 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:40:10.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:10.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:10.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:10.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:10.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:10.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:10.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:10.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:10.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:10.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:10.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:10 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/265352754' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:40:10.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:10 smithi106 ceph-mon[44423]: pgmap v1944: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 8.9 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 168 B/s rd, 168 B/s wr, 24 op/s; 9371/291142 objects degraded (3.219%); 42 MiB/s, 668 objects/s recovering 2024-04-20T17:40:10.829 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.7 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.1.0.7 2024-04-20T17:40:10.895 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:40:10.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:10.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:40:10.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:10.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:10.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:10.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:10.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:10.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:10.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:10.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:10.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:10.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:10.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:10 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/265352754' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:40:10.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:10 smithi005 ceph-mon[39562]: pgmap v1944: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 8.9 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 168 B/s rd, 168 B/s wr, 24 op/s; 9371/291142 objects degraded (3.219%); 42 MiB/s, 668 objects/s recovering 2024-04-20T17:40:10.998 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:40:11.026 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:11.046 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:40:11.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:11.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:40:11.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:10 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/265352754' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:40:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:10 smithi171 ceph-mon[44189]: pgmap v1944: 17 pgs: 1 active+recovering+degraded, 1 undersized+degraded+remapped+backfilling+peered, 15 active+clean; 8.9 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 168 B/s rd, 168 B/s wr, 24 op/s; 9371/291142 objects degraded (3.219%); 42 MiB/s, 668 objects/s recovering 2024-04-20T17:40:11.101 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.7/config 2024-04-20T17:40:11.150 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:40:11.207 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:11.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:11 smithi005 ceph-mon[39562]: 7.8 scrub starts 2024-04-20T17:40:11.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:11 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:40:11.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:11 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/265352754' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:40:11.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:11 smithi005 ceph-mon[39562]: osdmap e733: 12 total, 11 up, 9 in 2024-04-20T17:40:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:11 smithi106 ceph-mon[44423]: 7.8 scrub starts 2024-04-20T17:40:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:11 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:40:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:11 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/265352754' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:40:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:11 smithi106 ceph-mon[44423]: osdmap e733: 12 total, 11 up, 9 in 2024-04-20T17:40:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:11 smithi171 ceph-mon[44189]: 7.8 scrub starts 2024-04-20T17:40:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:11 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:40:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:11 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/265352754' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:40:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:11 smithi171 ceph-mon[44189]: osdmap e733: 12 total, 11 up, 9 in 2024-04-20T17:40:12.574 INFO:teuthology.orchestra.run.smithi106.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:40:12.857 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:12 smithi106 ceph-mon[44423]: pgmap v1946: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 8.9 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 170 B/s rd, 64 KiB/s wr, 102 op/s; 8258/290234 objects degraded (2.845%); 51 MiB/s, 819 objects/s recovering 2024-04-20T17:40:12.900 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:12 smithi005 ceph-mon[39562]: pgmap v1946: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 8.9 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 170 B/s rd, 64 KiB/s wr, 102 op/s; 8258/290234 objects degraded (2.845%); 51 MiB/s, 819 objects/s recovering 2024-04-20T17:40:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:12 smithi171 ceph-mon[44189]: pgmap v1946: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 8.9 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 170 B/s rd, 64 KiB/s wr, 102 op/s; 8258/290234 objects degraded (2.845%); 51 MiB/s, 819 objects/s recovering 2024-04-20T17:40:13.330 INFO:teuthology.orchestra.run.smithi106.stdout:get_pg_num_history pg_num_history pg_num_history(e721 pg_nums {1={14=1},6={619=16},7={694=16}} deleted_pools 689,6) 2024-04-20T17:40:13.330 INFO:teuthology.orchestra.run.smithi106.stdout:Importing pgid 1.0 2024-04-20T17:40:13.330 INFO:teuthology.orchestra.run.smithi106.stdout:Write #1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head# 2024-04-20T17:40:13.330 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:40:13.331 INFO:teuthology.orchestra.run.smithi106.stdout:Write #1:30306672:devicehealth::main.db.0000000000000000:head# 2024-04-20T17:40:13.331 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-04-20T17:40:13.331 INFO:teuthology.orchestra.run.smithi106.stdout:write_pg epoch 712 info 1.0( v 15'137 (0'0,15'137] local-lis/les=414/415 n=2 ec=14/14 lis/c=414/414 les/c/f=415/415/0 sis=414) 2024-04-20T17:40:13.332 INFO:teuthology.orchestra.run.smithi106.stdout:Import successful 2024-04-20T17:40:13.409 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:40:13.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:13.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:40:13.519 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:40:13.551 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:13.573 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:40:13.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:13.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:40:13.676 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:40:13.756 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:13.884 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:40:13.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:13.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:40:13.935 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:40:13.936 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:40:13.990 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:40:14.019 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:14.168 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:40:14.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:14.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:40:14.272 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:40:14.303 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:14.346 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:40:14.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:14.350 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:40:14.451 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:40:14.477 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:14.635 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:40:14.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:14.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:40:14.652 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/exp.1.0.7 2024-04-20T17:40:14.687 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:40:14.687 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:40:14.687 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:40:14.687 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:40:14.687 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:40:14.720 INFO:tasks.thrashosds.thrasher:Waiting for clean again 2024-04-20T17:40:14.720 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-20T17:40:14.720 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:40:14.739 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:40:14.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:14 smithi106 ceph-mon[44423]: pgmap v1947: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 8.8 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 170 B/s rd, 64 KiB/s wr, 363 op/s; 8258/287108 objects degraded (2.876%); 14 MiB/s, 224 objects/s recovering 2024-04-20T17:40:14.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:14 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 8258/287108 objects degraded (2.876%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:40:14.852 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:14.935 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:40:14.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:14.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:40:14.948 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:14 smithi005 ceph-mon[39562]: pgmap v1947: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 8.8 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 170 B/s rd, 64 KiB/s wr, 363 op/s; 8258/287108 objects degraded (2.876%); 14 MiB/s, 224 objects/s recovering 2024-04-20T17:40:14.948 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:14 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 8258/287108 objects degraded (2.876%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:40:15.047 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:40:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:14 smithi171 ceph-mon[44189]: pgmap v1947: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 8.8 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 170 B/s rd, 64 KiB/s wr, 363 op/s; 8258/287108 objects degraded (2.876%); 14 MiB/s, 224 objects/s recovering 2024-04-20T17:40:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:14 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 8258/287108 objects degraded (2.876%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:40:15.086 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:40:15.144 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:15.937 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:40:16.987 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-20T17:40:17.062 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:40:17.072 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:40:17.080 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:40:17.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:17.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:40:17.108 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:40:17.193 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:40:17.221 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:17.241 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:40:17.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:17.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:40:17.346 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:40:17.423 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:17.664 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:17 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:40:17.664 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:17 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:40:17.664 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:17 smithi005 ceph-mon[39562]: pgmap v1948: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 8.6 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 128 B/s rd, 48 KiB/s wr, 588 op/s; 8191/282118 objects degraded (2.903%); 11 MiB/s, 177 objects/s recovering 2024-04-20T17:40:17.664 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:17 smithi005 ceph-mon[39562]: osdmap e734: 12 total, 11 up, 9 in 2024-04-20T17:40:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:17 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:40:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:17 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:40:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:17 smithi171 ceph-mon[44189]: pgmap v1948: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 8.6 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 128 B/s rd, 48 KiB/s wr, 588 op/s; 8191/282118 objects degraded (2.903%); 11 MiB/s, 177 objects/s recovering 2024-04-20T17:40:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:17 smithi171 ceph-mon[44189]: osdmap e734: 12 total, 11 up, 9 in 2024-04-20T17:40:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:17 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:40:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:17 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:40:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:17 smithi106 ceph-mon[44423]: pgmap v1948: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 8.6 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 128 B/s rd, 48 KiB/s wr, 588 op/s; 8191/282118 objects degraded (2.903%); 11 MiB/s, 177 objects/s recovering 2024-04-20T17:40:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:17 smithi106 ceph-mon[44423]: osdmap e734: 12 total, 11 up, 9 in 2024-04-20T17:40:18.299 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:40:18.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:18.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:40:18.411 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:40:18.493 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:18.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:18 smithi005 ceph-mon[39562]: pgmap v1950: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 8.6 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 48 KiB/s wr, 637 op/s; 8191/280958 objects degraded (2.915%); 9.2 MiB/s, 147 objects/s recovering 2024-04-20T17:40:18.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:18 smithi005 ceph-mon[39562]: osdmap e735: 12 total, 11 up, 9 in 2024-04-20T17:40:19.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:18 smithi106 ceph-mon[44423]: pgmap v1950: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 8.6 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 48 KiB/s wr, 637 op/s; 8191/280958 objects degraded (2.915%); 9.2 MiB/s, 147 objects/s recovering 2024-04-20T17:40:19.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:18 smithi106 ceph-mon[44423]: osdmap e735: 12 total, 11 up, 9 in 2024-04-20T17:40:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:18 smithi171 ceph-mon[44189]: pgmap v1950: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 8.6 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 48 KiB/s wr, 637 op/s; 8191/280958 objects degraded (2.915%); 9.2 MiB/s, 147 objects/s recovering 2024-04-20T17:40:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:18 smithi171 ceph-mon[44189]: osdmap e735: 12 total, 11 up, 9 in 2024-04-20T17:40:19.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:19 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:40:19.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:19 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:40:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:19 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:40:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:19 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:40:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:19 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:40:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:19 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:40:20.601 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:40:20.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:20.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:40:20.705 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:40:20.732 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:20.751 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:40:20.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:20.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:40:20.855 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:40:20.897 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:20.898 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:20 smithi005 ceph-mon[39562]: pgmap v1952: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 8.4 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 982 op/s; 8122/274504 objects degraded (2.959%); 1.1 MiB/s, 16 objects/s recovering 2024-04-20T17:40:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:20 smithi106 ceph-mon[44423]: pgmap v1952: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 8.4 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 982 op/s; 8122/274504 objects degraded (2.959%); 1.1 MiB/s, 16 objects/s recovering 2024-04-20T17:40:21.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:20 smithi171 ceph-mon[44189]: pgmap v1952: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 8.4 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 982 op/s; 8122/274504 objects degraded (2.959%); 1.1 MiB/s, 16 objects/s recovering 2024-04-20T17:40:21.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:21 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 8122/274504 objects degraded (2.959%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:40:22.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:21 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 8122/274504 objects degraded (2.959%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:40:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:21 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 8122/274504 objects degraded (2.959%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:40:22.485 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:40:22.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:22.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:40:22.593 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:40:22.628 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:22.679 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:40:22.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:22.692 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:40:22.793 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:40:22.845 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:22 smithi005 ceph-mon[39562]: pgmap v1953: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 8.3 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 972 op/s; 8122/271538 objects degraded (2.991%); 1.1 MiB/s, 16 objects/s recovering 2024-04-20T17:40:22.849 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:23.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:22 smithi106 ceph-mon[44423]: pgmap v1953: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 8.3 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 972 op/s; 8122/271538 objects degraded (2.991%); 1.1 MiB/s, 16 objects/s recovering 2024-04-20T17:40:23.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:22 smithi171 ceph-mon[44189]: pgmap v1953: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 8.3 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 972 op/s; 8122/271538 objects degraded (2.991%); 1.1 MiB/s, 16 objects/s recovering 2024-04-20T17:40:23.088 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:40:23.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:23.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:40:23.193 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:40:23.265 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:24.550 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:40:24.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:24.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:40:24.662 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:40:24.696 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:24.719 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:40:24.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:24.722 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:40:24.804 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:40:24.808 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:40:24.819 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:24 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:40:24.819 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:24 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:40:24.819 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:24 smithi005 ceph-mon[39562]: osdmap e736: 12 total, 11 up, 9 in 2024-04-20T17:40:24.820 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:24 smithi005 ceph-mon[39562]: pgmap v1955: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 8.2 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 886 op/s; 8122/268016 objects degraded (3.030%); 555 KiB/s, 8 objects/s recovering 2024-04-20T17:40:24.822 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:40:24.852 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:40:24.902 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:25.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:24 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:40:25.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:24 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:40:25.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:24 smithi106 ceph-mon[44423]: osdmap e736: 12 total, 11 up, 9 in 2024-04-20T17:40:25.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:24 smithi106 ceph-mon[44423]: pgmap v1955: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 8.2 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 886 op/s; 8122/268016 objects degraded (3.030%); 555 KiB/s, 8 objects/s recovering 2024-04-20T17:40:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:24 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:40:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:24 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:40:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:24 smithi171 ceph-mon[44189]: osdmap e736: 12 total, 11 up, 9 in 2024-04-20T17:40:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:24 smithi171 ceph-mon[44189]: pgmap v1955: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 8.2 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 0 B/s wr, 886 op/s; 8122/268016 objects degraded (3.030%); 555 KiB/s, 8 objects/s recovering 2024-04-20T17:40:25.108 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:40:25.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:25.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:40:25.218 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:40:25.245 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:25.264 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:40:25.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:25.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:40:25.368 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:40:25.400 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:25.425 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:40:25.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:25.429 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:40:25.530 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:40:25.558 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:25.577 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:40:25.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:25.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:40:25.681 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:40:25.704 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:25 smithi171 ceph-mon[44189]: from='client.27232 v1:172.21.15.5:0/4250149433' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:40:25.704 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:25 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3580667785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:40:25.711 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:25.715 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:40:25.737 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:40:25.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:25.744 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:40:25.763 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:40:25.841 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:40:25.841 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:40:25.844 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:40:25.853 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:40:25.883 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:40:25.898 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:25.918 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:40:25.961 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:40:25.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:25 smithi005 ceph-mon[39562]: from='client.27232 v1:172.21.15.5:0/4250149433' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:40:25.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:25 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3580667785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:40:25.993 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:40:26.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:25 smithi106 ceph-mon[44423]: from='client.27232 v1:172.21.15.5:0/4250149433' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:40:26.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:25 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3580667785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:40:26.024 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:40:26.056 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:40:26.057 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:40:26.057 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:40:26.057 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:40:26.057 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:40:26.057 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:40:26.057 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:40:26.057 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:40:26.057 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:40:26.057 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:40:26.057 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:40:26.057 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:40:26.057 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:40:26.887 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:40:26.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:26.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:40:26.994 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:40:27.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:26 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:27.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:26 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:27.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:26 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:27.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:26 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:27.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:26 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:27.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:26 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:27.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:26 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:27.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:26 smithi106 ceph-mon[44423]: pgmap v1956: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 8.0 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 1.09k op/s; 8052/263484 objects degraded (3.056%); 1.1 MiB/s, 17 objects/s recovering 2024-04-20T17:40:27.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:26 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 8122/268016 objects degraded (3.030%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:40:27.021 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:26 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:27.021 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:26 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:27.021 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:26 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:27.021 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:26 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:27.022 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:26 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:27.022 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:26 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:27.022 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:26 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:27.022 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:26 smithi171 ceph-mon[44189]: pgmap v1956: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 8.0 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 1.09k op/s; 8052/263484 objects degraded (3.056%); 1.1 MiB/s, 17 objects/s recovering 2024-04-20T17:40:27.022 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:26 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 8122/268016 objects degraded (3.030%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:40:27.023 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:27.044 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:40:27.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:27.048 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:40:27.149 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:40:27.178 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:27.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:26 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:27.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:26 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:27.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:26 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:27.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:26 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:27.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:26 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:27.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:26 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:27.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:26 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:27.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:26 smithi005 ceph-mon[39562]: pgmap v1956: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 8.0 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 1.09k op/s; 8052/263484 objects degraded (3.056%); 1.1 MiB/s, 17 objects/s recovering 2024-04-20T17:40:27.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:26 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 8122/268016 objects degraded (3.030%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:40:27.198 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:40:27.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:27.202 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:40:27.303 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:40:27.329 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:27.352 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:40:27.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:27.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:40:27.457 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:40:27.534 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:27.572 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:40:27.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:27.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:40:27.679 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:40:27.725 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:27.758 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:40:27.787 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:27.788 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:27.788 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:27.788 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:27.788 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:27 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4020590823' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:40:27.788 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:27 smithi005 ceph-mon[39562]: from='client.27244 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:40:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:27 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4020590823' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:40:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:27 smithi106 ceph-mon[44423]: from='client.27244 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:40:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:27 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4020590823' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:40:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:27 smithi171 ceph-mon[44189]: from='client.27244 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:40:28.819 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:40:28.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:28.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:40:28.931 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:40:28.966 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:28.987 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":1955,"stamp":"2024-04-20T17:40:24.494608+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8782742072,"num_objects":134008,"num_object_clones":0,"num_object_copies":268016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":134008,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":157623,"num_write_kb":9329211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9945,"num_bytes_recovered":653744800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157716,"ondisk_log_size":157716,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":21115524,"kb_used_data":16707124,"kb_used_omap":217,"kb_used_meta":4408102,"kb_avail":822590844,"statfs":{"total":863955320832,"available":842333024256,"internally_reserved":0,"allocated":17108094976,"data_stored":17104802518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":222986,"internal_metadata":4513896694},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,3,7,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-462094336,"num_objects":-7051,"num_object_clones":0,"num_object_copies":-14102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-69,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7051,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":4521984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.949927"},"pg_stats":[{"pgid":"7.f","version":"735'9794","reported_seq":9984,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-04-20T17:40:20.206450+0000","last_change":"2024-04-20T17:39:40.325470+0000","last_active":"2024-04-20T17:40:20.206450+0000","last_peered":"2024-04-20T17:40:20.206450+0000","last_clean":"2024-04-20T17:40:20.206450+0000","last_became_active":"2024-04-20T17:37:56.835750+0000","last_became_peered":"2024-04-20T17:37:56.835750+0000","last_unstale":"2024-04-20T17:40:20.206450+0000","last_undegraded":"2024-04-20T17:40:20.206450+0000","last_fullsized":"2024-04-20T17:40:20.206450+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9175","last_scrub_stamp":"2024-04-20T17:39:40.325340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:40.325340+0000","objects_scrubbed":9175,"log_size":9794,"log_dups_size":0,"ondisk_log_size":9794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:41:08.457253+0000","scrub_duration":741,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":560726016,"num_objects":8556,"num_object_clones":0,"num_object_copies":17112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9794,"num_write_kb":587200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"735'9686","reported_seq":9858,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-04-20T17:40:21.835520+0000","last_change":"2024-04-20T17:39:07.189704+0000","last_active":"2024-04-20T17:40:21.835520+0000","last_peered":"2024-04-20T17:40:21.835520+0000","last_clean":"2024-04-20T17:40:21.835520+0000","last_became_active":"2024-04-20T17:37:56.833703+0000","last_became_peered":"2024-04-20T17:37:56.833703+0000","last_unstale":"2024-04-20T17:40:21.835520+0000","last_undegraded":"2024-04-20T17:40:21.835520+0000","last_fullsized":"2024-04-20T17:40:21.835520+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"716'7277","last_scrub_stamp":"2024-04-20T17:39:07.189514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:07.189514+0000","objects_scrubbed":7254,"log_size":9686,"log_dups_size":0,"ondisk_log_size":9686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:40:26.002383+0000","scrub_duration":1393,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":539623424,"num_objects":8234,"num_object_clones":0,"num_object_copies":16468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9686,"num_write_kb":573440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"735'11853","reported_seq":12148,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-04-20T17:40:20.210895+0000","last_change":"2024-04-20T17:39:48.311967+0000","last_active":"2024-04-20T17:40:20.210895+0000","last_peered":"2024-04-20T17:40:20.210895+0000","last_clean":"2024-04-20T17:40:20.210895+0000","last_became_active":"2024-04-20T17:37:56.836496+0000","last_became_peered":"2024-04-20T17:37:56.836496+0000","last_unstale":"2024-04-20T17:40:20.210895+0000","last_undegraded":"2024-04-20T17:40:20.210895+0000","last_fullsized":"2024-04-20T17:40:20.210895+0000","mapping_epoch":694,"log_start":"701'1800","ondisk_log_start":"701'1800","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'11204","last_scrub_stamp":"2024-04-20T17:39:48.311825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:48.311825+0000","objects_scrubbed":9436,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:40:50.621890+0000","scrub_duration":785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575930392,"num_objects":8789,"num_object_clones":0,"num_object_copies":17578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8789,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10085,"num_write_kb":603905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.c","version":"736'9894","reported_seq":10086,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-04-20T17:40:24.085488+0000","last_change":"2024-04-20T17:39:44.683378+0000","last_active":"2024-04-20T17:40:24.085488+0000","last_peered":"2024-04-20T17:40:24.085488+0000","last_clean":"2024-04-20T17:40:24.085488+0000","last_became_active":"2024-04-20T17:37:56.834181+0000","last_became_peered":"2024-04-20T17:37:56.834181+0000","last_unstale":"2024-04-20T17:40:24.085488+0000","last_undegraded":"2024-04-20T17:40:24.085488+0000","last_fullsized":"2024-04-20T17:40:24.085488+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9063","last_scrub_stamp":"2024-04-20T17:39:44.683201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:44.683201+0000","objects_scrubbed":9063,"log_size":9894,"log_dups_size":0,"ondisk_log_size":9894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:41:08.819063+0000","scrub_duration":746,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":539492352,"num_objects":8232,"num_object_clones":0,"num_object_copies":16464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9894,"num_write_kb":580032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"736'10016","reported_seq":10210,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-04-20T17:40:23.937042+0000","last_change":"2024-04-20T17:39:45.722632+0000","last_active":"2024-04-20T17:40:23.937042+0000","last_peered":"2024-04-20T17:40:23.937042+0000","last_clean":"2024-04-20T17:40:23.937042+0000","last_became_active":"2024-04-20T17:37:56.830487+0000","last_became_peered":"2024-04-20T17:37:56.830487+0000","last_unstale":"2024-04-20T17:40:23.937042+0000","last_undegraded":"2024-04-20T17:40:23.937042+0000","last_fullsized":"2024-04-20T17:40:23.937042+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9204","last_scrub_stamp":"2024-04-20T17:39:45.722537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:45.722537+0000","objects_scrubbed":9204,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:40:52.089051+0000","scrub_duration":769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":549978112,"num_objects":8392,"num_object_clones":0,"num_object_copies":16784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10016,"num_write_kb":589056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.a","version":"735'9651","reported_seq":9840,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-04-20T17:40:20.811635+0000","last_change":"2024-04-20T17:39:47.268931+0000","last_active":"2024-04-20T17:40:20.811635+0000","last_peered":"2024-04-20T17:40:20.811635+0000","last_clean":"2024-04-20T17:40:20.811635+0000","last_became_active":"2024-04-20T17:37:56.835944+0000","last_became_peered":"2024-04-20T17:37:56.835944+0000","last_unstale":"2024-04-20T17:40:20.811635+0000","last_undegraded":"2024-04-20T17:40:20.811635+0000","last_fullsized":"2024-04-20T17:40:20.811635+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'8992","last_scrub_stamp":"2024-04-20T17:39:47.268805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:47.268805+0000","objects_scrubbed":8992,"log_size":9651,"log_dups_size":0,"ondisk_log_size":9651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:41:05.514178+0000","scrub_duration":714,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":546111488,"num_objects":8333,"num_object_clones":0,"num_object_copies":16666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9651,"num_write_kb":575488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"735'9726","reported_seq":9899,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-04-20T17:40:20.211628+0000","last_change":"2024-04-20T17:39:03.675435+0000","last_active":"2024-04-20T17:40:20.211628+0000","last_peered":"2024-04-20T17:40:20.211628+0000","last_clean":"2024-04-20T17:40:20.211628+0000","last_became_active":"2024-04-20T17:37:56.835700+0000","last_became_peered":"2024-04-20T17:37:56.835700+0000","last_unstale":"2024-04-20T17:40:20.211628+0000","last_undegraded":"2024-04-20T17:40:20.211628+0000","last_fullsized":"2024-04-20T17:40:20.211628+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'7175","last_scrub_stamp":"2024-04-20T17:39:03.675211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:03.675211+0000","objects_scrubbed":7133,"log_size":9726,"log_dups_size":0,"ondisk_log_size":9726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":550764544,"num_objects":8404,"num_object_clones":0,"num_object_copies":16808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9726,"num_write_kb":580160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11],"acting":[0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"735'13332","reported_seq":31809,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-04-20T17:40:21.823533+0000","last_change":"2024-04-20T17:40:10.667197+0000","last_active":"2024-04-20T17:40:21.823533+0000","last_peered":"2024-04-20T17:40:21.823533+0000","last_clean":"2024-04-20T17:40:21.823533+0000","last_became_active":"2024-04-20T17:40:09.546695+0000","last_became_peered":"2024-04-20T17:40:09.546695+0000","last_unstale":"2024-04-20T17:40:21.823533+0000","last_undegraded":"2024-04-20T17:40:21.823533+0000","last_fullsized":"2024-04-20T17:40:21.823533+0000","mapping_epoch":731,"log_start":"702'3301","ondisk_log_start":"702'3301","created":694,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":3399,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:40:25.989781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":548798464,"num_objects":8374,"num_object_clones":0,"num_object_copies":16748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9762,"num_write_kb":580352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9062,"num_bytes_recovered":593887232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"736'10033","reported_seq":10208,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-04-20T17:40:23.710993+0000","last_change":"2024-04-20T17:39:05.344523+0000","last_active":"2024-04-20T17:40:23.710993+0000","last_peered":"2024-04-20T17:40:23.710993+0000","last_clean":"2024-04-20T17:40:23.710993+0000","last_became_active":"2024-04-20T17:37:56.836665+0000","last_became_peered":"2024-04-20T17:37:56.836665+0000","last_unstale":"2024-04-20T17:40:23.710993+0000","last_undegraded":"2024-04-20T17:40:23.710993+0000","last_fullsized":"2024-04-20T17:40:23.710993+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'7328","last_scrub_stamp":"2024-04-20T17:39:05.344263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:05.344263+0000","objects_scrubbed":7293,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:40:25.259595+0000","scrub_duration":1327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":543621120,"num_objects":8295,"num_object_clones":0,"num_object_copies":16590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10033,"num_write_kb":586496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.5","version":"736'10018","reported_seq":10192,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-04-20T17:40:23.942366+0000","last_change":"2024-04-20T17:39:04.261938+0000","last_active":"2024-04-20T17:40:23.942366+0000","last_peered":"2024-04-20T17:40:23.942366+0000","last_clean":"2024-04-20T17:40:23.942366+0000","last_became_active":"2024-04-20T17:37:56.833600+0000","last_became_peered":"2024-04-20T17:37:56.833600+0000","last_unstale":"2024-04-20T17:40:23.942366+0000","last_undegraded":"2024-04-20T17:40:23.942366+0000","last_fullsized":"2024-04-20T17:40:23.942366+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'7212","last_scrub_stamp":"2024-04-20T17:39:04.261753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:04.261753+0000","objects_scrubbed":7172,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:40:26.425373+0000","scrub_duration":1303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":543686656,"num_objects":8296,"num_object_clones":0,"num_object_copies":16592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10018,"num_write_kb":586048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.4","version":"735'9680","reported_seq":9869,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-04-20T17:40:20.213126+0000","last_change":"2024-04-20T17:39:50.365478+0000","last_active":"2024-04-20T17:40:20.213126+0000","last_peered":"2024-04-20T17:40:20.213126+0000","last_clean":"2024-04-20T17:40:20.213126+0000","last_became_active":"2024-04-20T17:37:56.834727+0000","last_became_peered":"2024-04-20T17:37:56.834727+0000","last_unstale":"2024-04-20T17:40:20.213126+0000","last_undegraded":"2024-04-20T17:40:20.213126+0000","last_fullsized":"2024-04-20T17:40:20.213126+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9031","last_scrub_stamp":"2024-04-20T17:39:50.365339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:50.365339+0000","objects_scrubbed":9031,"log_size":9680,"log_dups_size":0,"ondisk_log_size":9680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:41:02.075665+0000","scrub_duration":876,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":549322752,"num_objects":8382,"num_object_clones":0,"num_object_copies":16764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9680,"num_write_kb":577984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"736'9844","reported_seq":10014,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-04-20T17:40:24.098166+0000","last_change":"2024-04-20T17:39:00.502710+0000","last_active":"2024-04-20T17:40:24.098166+0000","last_peered":"2024-04-20T17:40:24.098166+0000","last_clean":"2024-04-20T17:40:24.098166+0000","last_became_active":"2024-04-20T17:37:56.832911+0000","last_became_peered":"2024-04-20T17:37:56.832911+0000","last_unstale":"2024-04-20T17:40:24.098166+0000","last_undegraded":"2024-04-20T17:40:24.098166+0000","last_fullsized":"2024-04-20T17:40:24.098166+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'6831","last_scrub_stamp":"2024-04-20T17:39:00.502556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:00.502556+0000","objects_scrubbed":6782,"log_size":9844,"log_dups_size":0,"ondisk_log_size":9844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":539230208,"num_objects":8228,"num_object_clones":0,"num_object_copies":16456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9844,"num_write_kb":578304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"736'10121","reported_seq":10298,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-04-20T17:40:24.094702+0000","last_change":"2024-04-20T17:39:08.489529+0000","last_active":"2024-04-20T17:40:24.094702+0000","last_peered":"2024-04-20T17:40:24.094702+0000","last_clean":"2024-04-20T17:40:24.094702+0000","last_became_active":"2024-04-20T17:37:56.834292+0000","last_became_peered":"2024-04-20T17:37:56.834292+0000","last_unstale":"2024-04-20T17:40:24.094702+0000","last_undegraded":"2024-04-20T17:40:24.094702+0000","last_fullsized":"2024-04-20T17:40:24.094702+0000","mapping_epoch":694,"log_start":"699'100","ondisk_log_start":"699'100","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"716'7577","last_scrub_stamp":"2024-04-20T17:39:08.489308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:08.489308+0000","objects_scrubbed":7524,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:40:26.339834+0000","scrub_duration":1554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":548601856,"num_objects":8371,"num_object_clones":0,"num_object_copies":16742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10121,"num_write_kb":591744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"735'9777","reported_seq":9965,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-04-20T17:40:20.642839+0000","last_change":"2024-04-20T17:39:40.082396+0000","last_active":"2024-04-20T17:40:20.642839+0000","last_peered":"2024-04-20T17:40:20.642839+0000","last_clean":"2024-04-20T17:40:20.642839+0000","last_became_active":"2024-04-20T17:37:56.835512+0000","last_became_peered":"2024-04-20T17:37:56.835512+0000","last_unstale":"2024-04-20T17:40:20.642839+0000","last_undegraded":"2024-04-20T17:40:20.642839+0000","last_fullsized":"2024-04-20T17:40:20.642839+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9091","last_scrub_stamp":"2024-04-20T17:39:40.082255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:40.082255+0000","objects_scrubbed":9091,"log_size":9777,"log_dups_size":0,"ondisk_log_size":9777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:41:06.918395+0000","scrub_duration":743,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":550830080,"num_objects":8405,"num_object_clones":0,"num_object_copies":16810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9777,"num_write_kb":581824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"735'9702","reported_seq":9892,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-04-20T17:40:20.859857+0000","last_change":"2024-04-20T17:39:45.346458+0000","last_active":"2024-04-20T17:40:20.859857+0000","last_peered":"2024-04-20T17:40:20.859857+0000","last_clean":"2024-04-20T17:40:20.859857+0000","last_became_active":"2024-04-20T17:37:56.833972+0000","last_became_peered":"2024-04-20T17:37:56.833972+0000","last_unstale":"2024-04-20T17:40:20.859857+0000","last_undegraded":"2024-04-20T17:40:20.859857+0000","last_fullsized":"2024-04-20T17:40:20.859857+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9032","last_scrub_stamp":"2024-04-20T17:39:45.346352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:45.346352+0000","objects_scrubbed":9032,"log_size":9702,"log_dups_size":0,"ondisk_log_size":9702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:41:11.755272+0000","scrub_duration":769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":548012032,"num_objects":8362,"num_object_clones":0,"num_object_copies":16724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9702,"num_write_kb":578048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.6","version":"735'9653","reported_seq":11575,"reported_epoch":735,"state":"active+recovering+degraded","last_fresh":"2024-04-20T17:40:20.184863+0000","last_change":"2024-04-20T17:39:54.962650+0000","last_active":"2024-04-20T17:40:20.184863+0000","last_peered":"2024-04-20T17:40:20.184863+0000","last_clean":"2024-04-20T17:39:34.507147+0000","last_became_active":"2024-04-20T17:39:54.933393+0000","last_became_peered":"2024-04-20T17:39:54.933393+0000","last_unstale":"2024-04-20T17:40:20.184863+0000","last_undegraded":"2024-04-20T17:39:54.578981+0000","last_fullsized":"2024-04-20T17:40:20.184863+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'7141","last_scrub_stamp":"2024-04-20T17:39:04.435913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:04.435913+0000","objects_scrubbed":7101,"log_size":9653,"log_dups_size":0,"ondisk_log_size":9653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":547422208,"num_objects":8353,"num_object_clones":0,"num_object_copies":16706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9653,"num_write_kb":576192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":873,"num_bytes_recovered":57212928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":8122},{"shards":"0,3","objects":231}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":475,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-04-20T17:40:23.671379+0000","last_change":"2024-04-20T17:39:54.973311+0000","last_active":"2024-04-20T17:40:23.671379+0000","last_peered":"2024-04-20T17:40:23.671379+0000","last_clean":"2024-04-20T17:40:23.671379+0000","last_became_active":"2024-04-20T17:39:54.970208+0000","last_became_peered":"2024-04-20T17:39:54.970208+0000","last_unstale":"2024-04-20T17:40:23.671379+0000","last_undegraded":"2024-04-20T17:40:23.671379+0000","last_fullsized":"2024-04-20T17:40:23.671379+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:38:56.114399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:38:56.114399+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":8782151704,"num_objects":134006,"num_object_clones":0,"num_object_copies":268012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":134006,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157442,"num_write_kb":9326273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9935,"num_bytes_recovered":651100160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18207678464,"data_stored":18207670320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157579,"ondisk_log_size":157579,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991649,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2801676,"kb_used_data":2134660,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":90943476,"statfs":{"total":95995035648,"available":93126119424,"internally_reserved":0,"allocated":2185891840,"data_stored":2185524138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":683014596},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253287,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2168904,"kb_used_data":1632380,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":91576248,"statfs":{"total":95995035648,"available":93774077952,"internally_reserved":0,"allocated":1671557120,"data_stored":1671192970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":549386688},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613036,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2174284,"kb_used_data":1615556,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":91570868,"statfs":{"total":95995035648,"available":93768568832,"internally_reserved":0,"allocated":1654329344,"data_stored":1653961662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":572127680},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809267,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1365828,"kb_used_data":1068092,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":92379324,"statfs":{"total":95995035648,"available":94596427776,"internally_reserved":0,"allocated":1093726208,"data_stored":1093362078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":304871877},"hb_peers":[0,1,2,3,4,5,6,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":2297807503497,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":2830383448120,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738840,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3360648,"kb_used_data":2729728,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":90384504,"statfs":{"total":95995035648,"available":92553732096,"internally_reserved":0,"allocated":2795241472,"data_stored":2794873270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394131,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2461444,"kb_used_data":2142332,"kb_used_omap":202,"kb_used_meta":318901,"kb_avail":91283708,"statfs":{"total":95995035648,"available":93474516992,"internally_reserved":0,"allocated":2193747968,"data_stored":2193383838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":326555095},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653181,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3006600,"kb_used_data":2681280,"kb_used_omap":4,"kb_used_meta":325307,"kb_avail":90738552,"statfs":{"total":95995035648,"available":92916277248,"internally_reserved":0,"allocated":2745630720,"data_stored":2745262518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":333115079},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986629,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2061316,"kb_used_data":1615036,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":91683836,"statfs":{"total":95995035648,"available":93884248064,"internally_reserved":0,"allocated":1653796864,"data_stored":1653432734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790399,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1714824,"kb_used_data":1088060,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":92030328,"statfs":{"total":95995035648,"available":94239055872,"internally_reserved":0,"allocated":1114173440,"data_stored":1113809310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2784169984,"data_stored":2784165912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2179530752,"data_stored":2179530752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2734362624,"data_stored":2734358552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1651376128,"data_stored":1651376128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1109786624,"data_stored":1109786624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1167458304,"data_stored":1167458304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1091108864,"data_stored":1091108864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1649475584,"data_stored":1649475584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1670119424,"data_stored":1670119424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2170290176,"data_stored":2170290176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:40:28.988 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:40:28.989 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:28 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:40:28.989 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:28 smithi005 ceph-mon[39562]: from='client.27244 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:40:28.989 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:28 smithi005 ceph-mon[39562]: osdmap e737: 12 total, 11 up, 9 in 2024-04-20T17:40:28.989 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:28 smithi005 ceph-mon[39562]: pgmap v1958: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 8.0 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 755 op/s; 8052/262410 objects degraded (3.068%); 560 KiB/s, 8 objects/s recovering 2024-04-20T17:40:28.992 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:40:28.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:28.996 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:40:28.997 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:28 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:40:28.997 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:28 smithi171 ceph-mon[44189]: from='client.27244 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:40:28.997 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:28 smithi171 ceph-mon[44189]: osdmap e737: 12 total, 11 up, 9 in 2024-04-20T17:40:28.997 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:28 smithi171 ceph-mon[44189]: pgmap v1958: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 8.0 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 755 op/s; 8052/262410 objects degraded (3.068%); 560 KiB/s, 8 objects/s recovering 2024-04-20T17:40:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:28 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:40:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:28 smithi106 ceph-mon[44423]: from='client.27244 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:40:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:28 smithi106 ceph-mon[44423]: osdmap e737: 12 total, 11 up, 9 in 2024-04-20T17:40:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:28 smithi106 ceph-mon[44423]: pgmap v1958: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 8.0 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 755 op/s; 8052/262410 objects degraded (3.068%); 560 KiB/s, 8 objects/s recovering 2024-04-20T17:40:29.097 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:40:29.158 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:29.612 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:40:29.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:29.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:40:29.720 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:40:29.746 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:29.765 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:40:29.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:29.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:40:29.869 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:40:29.896 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:29.915 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:40:29.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:29.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:40:29.955 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:40:29.956 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:40:30.019 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:40:30.076 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:30.459 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:40:30.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:30.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:40:30.533 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:40:30.568 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:40:30.599 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:30.625 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:40:30.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:30.632 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:40:30.733 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:40:30.777 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:30.822 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:40:30.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:30.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:40:30.930 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:40:30.991 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:31.082 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:40:31.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:31.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:40:31.189 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:40:31.258 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:31.399 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:40:31.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:31.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:40:31.403 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:31 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:40:31.403 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:31 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:40:31.403 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:31 smithi005 ceph-mon[39562]: osdmap e738: 12 total, 11 up, 9 in 2024-04-20T17:40:31.403 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:31 smithi005 ceph-mon[39562]: pgmap v1960: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.8 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s; 7826/256862 objects degraded (3.047%); 2.7 MiB/s, 43 objects/s recovering 2024-04-20T17:40:31.503 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:40:31.532 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:31.546 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:40:31.546 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:40:31.552 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:40:31.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:31.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:40:31.555 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:40:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:31 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:40:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:31 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:40:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:31 smithi171 ceph-mon[44189]: osdmap e738: 12 total, 11 up, 9 in 2024-04-20T17:40:31.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:31 smithi171 ceph-mon[44189]: pgmap v1960: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.8 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s; 7826/256862 objects degraded (3.047%); 2.7 MiB/s, 43 objects/s recovering 2024-04-20T17:40:31.656 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:40:31.727 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:31.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:31 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:40:31.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:31 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:40:31.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:31 smithi106 ceph-mon[44423]: osdmap e738: 12 total, 11 up, 9 in 2024-04-20T17:40:31.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:31 smithi106 ceph-mon[44423]: pgmap v1960: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.8 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s; 7826/256862 objects degraded (3.047%); 2.7 MiB/s, 43 objects/s recovering 2024-04-20T17:40:31.957 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:40:32.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 7826/256862 objects degraded (3.047%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:40:32.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=59} crc {auth=2927210262,config=1460075751,health=2731299312,kv=2872500370}) 2024-04-20T17:40:32.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2245028860}) 2024-04-20T17:40:32.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3576719492}) 2024-04-20T17:40:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3091231664}) 2024-04-20T17:40:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1302997735}) 2024-04-20T17:40:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=599289947}) 2024-04-20T17:40:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=61824363}) 2024-04-20T17:40:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1003960019}) 2024-04-20T17:40:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3677781063}) 2024-04-20T17:40:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=780754769}) 2024-04-20T17:40:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1211535082}) 2024-04-20T17:40:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4204961329}) 2024-04-20T17:40:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2024440097}) 2024-04-20T17:40:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1910240381}) 2024-04-20T17:40:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=66995916}) 2024-04-20T17:40:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2099673564}) 2024-04-20T17:40:32.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1604536246}) 2024-04-20T17:40:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=441556945}) 2024-04-20T17:40:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3835202981}) 2024-04-20T17:40:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3901519597}) 2024-04-20T17:40:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3484935619}) 2024-04-20T17:40:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1178754329}) 2024-04-20T17:40:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=130654169}) 2024-04-20T17:40:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3257819545}) 2024-04-20T17:40:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2463866317}) 2024-04-20T17:40:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2859749908}) 2024-04-20T17:40:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3214604027}) 2024-04-20T17:40:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=368127160}) 2024-04-20T17:40:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=632201411}) 2024-04-20T17:40:32.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1004110372}) 2024-04-20T17:40:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=225419048}) 2024-04-20T17:40:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=183291082}) 2024-04-20T17:40:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3113044490}) 2024-04-20T17:40:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2027387351}) 2024-04-20T17:40:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4243014341}) 2024-04-20T17:40:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2296246196}) 2024-04-20T17:40:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3017257522}) 2024-04-20T17:40:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3276240512}) 2024-04-20T17:40:32.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3400770480}) 2024-04-20T17:40:32.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 7826/256862 objects degraded (3.047%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:40:32.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=59} crc {auth=2927210262,config=1460075751,health=2731299312,kv=2872500370}) 2024-04-20T17:40:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2245028860}) 2024-04-20T17:40:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3576719492}) 2024-04-20T17:40:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3091231664}) 2024-04-20T17:40:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1302997735}) 2024-04-20T17:40:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=599289947}) 2024-04-20T17:40:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=61824363}) 2024-04-20T17:40:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1003960019}) 2024-04-20T17:40:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3677781063}) 2024-04-20T17:40:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=780754769}) 2024-04-20T17:40:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1211535082}) 2024-04-20T17:40:32.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4204961329}) 2024-04-20T17:40:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2024440097}) 2024-04-20T17:40:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1910240381}) 2024-04-20T17:40:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=66995916}) 2024-04-20T17:40:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2099673564}) 2024-04-20T17:40:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1604536246}) 2024-04-20T17:40:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=441556945}) 2024-04-20T17:40:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3835202981}) 2024-04-20T17:40:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3901519597}) 2024-04-20T17:40:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3484935619}) 2024-04-20T17:40:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1178754329}) 2024-04-20T17:40:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=130654169}) 2024-04-20T17:40:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3257819545}) 2024-04-20T17:40:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2463866317}) 2024-04-20T17:40:32.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2859749908}) 2024-04-20T17:40:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3214604027}) 2024-04-20T17:40:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=368127160}) 2024-04-20T17:40:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=632201411}) 2024-04-20T17:40:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1004110372}) 2024-04-20T17:40:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=225419048}) 2024-04-20T17:40:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=183291082}) 2024-04-20T17:40:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3113044490}) 2024-04-20T17:40:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2027387351}) 2024-04-20T17:40:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4243014341}) 2024-04-20T17:40:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2296246196}) 2024-04-20T17:40:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3017257522}) 2024-04-20T17:40:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3276240512}) 2024-04-20T17:40:32.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3400770480}) 2024-04-20T17:40:32.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 7826/256862 objects degraded (3.047%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:40:32.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=59} crc {auth=2927210262,config=1460075751,health=2731299312,kv=2872500370}) 2024-04-20T17:40:32.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2245028860}) 2024-04-20T17:40:32.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3576719492}) 2024-04-20T17:40:32.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3091231664}) 2024-04-20T17:40:32.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1302997735}) 2024-04-20T17:40:32.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=599289947}) 2024-04-20T17:40:32.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=61824363}) 2024-04-20T17:40:32.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1003960019}) 2024-04-20T17:40:32.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3677781063}) 2024-04-20T17:40:32.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=780754769}) 2024-04-20T17:40:32.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1211535082}) 2024-04-20T17:40:32.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4204961329}) 2024-04-20T17:40:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2024440097}) 2024-04-20T17:40:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1910240381}) 2024-04-20T17:40:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=66995916}) 2024-04-20T17:40:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2099673564}) 2024-04-20T17:40:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1604536246}) 2024-04-20T17:40:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=441556945}) 2024-04-20T17:40:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3835202981}) 2024-04-20T17:40:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3901519597}) 2024-04-20T17:40:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3484935619}) 2024-04-20T17:40:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1178754329}) 2024-04-20T17:40:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=130654169}) 2024-04-20T17:40:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3257819545}) 2024-04-20T17:40:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2463866317}) 2024-04-20T17:40:32.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2859749908}) 2024-04-20T17:40:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3214604027}) 2024-04-20T17:40:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=368127160}) 2024-04-20T17:40:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=632201411}) 2024-04-20T17:40:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1004110372}) 2024-04-20T17:40:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=225419048}) 2024-04-20T17:40:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=183291082}) 2024-04-20T17:40:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3113044490}) 2024-04-20T17:40:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2027387351}) 2024-04-20T17:40:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4243014341}) 2024-04-20T17:40:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2296246196}) 2024-04-20T17:40:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3017257522}) 2024-04-20T17:40:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3276240512}) 2024-04-20T17:40:32.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3400770480}) 2024-04-20T17:40:33.422 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:40:33.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:33.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:40:33.527 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:40:33.557 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:33.578 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:40:33.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:33.584 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:40:33.685 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:40:33.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=519704238}) 2024-04-20T17:40:33.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2485333400}) 2024-04-20T17:40:33.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1341590561}) 2024-04-20T17:40:33.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=991132875}) 2024-04-20T17:40:33.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=177569602}) 2024-04-20T17:40:33.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2736000675}) 2024-04-20T17:40:33.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2740286668}) 2024-04-20T17:40:33.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1033876013}) 2024-04-20T17:40:33.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3761915255}) 2024-04-20T17:40:33.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3199281295}) 2024-04-20T17:40:33.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3467706601}) 2024-04-20T17:40:33.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=790994446}) 2024-04-20T17:40:33.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1019756345}) 2024-04-20T17:40:33.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1627014007}) 2024-04-20T17:40:33.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2682920901}) 2024-04-20T17:40:33.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1500066371}) 2024-04-20T17:40:33.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1955537022}) 2024-04-20T17:40:33.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=838572601}) 2024-04-20T17:40:33.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1049549521}) 2024-04-20T17:40:33.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3395866494}) 2024-04-20T17:40:33.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=921530098}) 2024-04-20T17:40:33.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2110546405}) 2024-04-20T17:40:33.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1879758954}) 2024-04-20T17:40:33.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1556517195}) 2024-04-20T17:40:33.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2874559306}) 2024-04-20T17:40:33.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1523013969}) 2024-04-20T17:40:33.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2649192464}) 2024-04-20T17:40:33.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=834894355}) 2024-04-20T17:40:33.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=360966902}) 2024-04-20T17:40:33.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2714469317}) 2024-04-20T17:40:33.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2914552523}) 2024-04-20T17:40:33.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1342690771}) 2024-04-20T17:40:33.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2033521846}) 2024-04-20T17:40:33.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=979659534}) 2024-04-20T17:40:33.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3490060101}) 2024-04-20T17:40:33.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3826767896}) 2024-04-20T17:40:33.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2483589839}) 2024-04-20T17:40:33.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1938640336}) 2024-04-20T17:40:33.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=33049532}) 2024-04-20T17:40:33.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=815528576}) 2024-04-20T17:40:33.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3172563712}) 2024-04-20T17:40:33.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1016505298}) 2024-04-20T17:40:33.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1327109792}) 2024-04-20T17:40:33.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1219856093}) 2024-04-20T17:40:33.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2020756064}) 2024-04-20T17:40:33.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1529304398}) 2024-04-20T17:40:33.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=280397512}) 2024-04-20T17:40:33.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3821369342}) 2024-04-20T17:40:33.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=2546328079,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241}) 2024-04-20T17:40:33.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=59} crc {auth=2927210262,config=1460075751,health=2731299312,kv=2872500370}) 2024-04-20T17:40:33.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2245028860}) 2024-04-20T17:40:33.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3576719492}) 2024-04-20T17:40:33.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3091231664}) 2024-04-20T17:40:33.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1302997735}) 2024-04-20T17:40:33.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=599289947}) 2024-04-20T17:40:33.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=642778375}) 2024-04-20T17:40:33.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3758090761}) 2024-04-20T17:40:33.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1075948743}) 2024-04-20T17:40:33.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3594915675}) 2024-04-20T17:40:33.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=22292920}) 2024-04-20T17:40:33.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3371825531}) 2024-04-20T17:40:33.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=630633847}) 2024-04-20T17:40:33.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2481383515}) 2024-04-20T17:40:33.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2539536142}) 2024-04-20T17:40:33.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1688349370}) 2024-04-20T17:40:33.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2435115651}) 2024-04-20T17:40:33.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3888529585}) 2024-04-20T17:40:33.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=525632982}) 2024-04-20T17:40:33.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4047359843}) 2024-04-20T17:40:33.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1710598054}) 2024-04-20T17:40:33.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=578643127}) 2024-04-20T17:40:33.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=137475321}) 2024-04-20T17:40:33.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3509541051}) 2024-04-20T17:40:33.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2747137995}) 2024-04-20T17:40:33.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=230808383}) 2024-04-20T17:40:33.719 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2074641987}) 2024-04-20T17:40:33.719 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3211751186}) 2024-04-20T17:40:33.719 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1464289765}) 2024-04-20T17:40:33.719 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=575887972}) 2024-04-20T17:40:33.719 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1181202404}) 2024-04-20T17:40:33.719 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4261504939}) 2024-04-20T17:40:33.719 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1071466465}) 2024-04-20T17:40:33.719 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1086136799}) 2024-04-20T17:40:33.719 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=605998130}) 2024-04-20T17:40:33.719 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2965670760}) 2024-04-20T17:40:33.719 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1554319229}) 2024-04-20T17:40:33.719 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4040979299}) 2024-04-20T17:40:33.719 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1687080114}) 2024-04-20T17:40:33.719 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2636892967}) 2024-04-20T17:40:33.720 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4008925009}) 2024-04-20T17:40:33.720 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2852236343}) 2024-04-20T17:40:33.720 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2275556127}) 2024-04-20T17:40:33.720 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1754112212}) 2024-04-20T17:40:33.720 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=533976486}) 2024-04-20T17:40:33.720 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2662492679}) 2024-04-20T17:40:33.720 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2354381854}) 2024-04-20T17:40:33.720 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=659248529}) 2024-04-20T17:40:33.720 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1613325388}) 2024-04-20T17:40:33.720 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1090572829}) 2024-04-20T17:40:33.720 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=956354298}) 2024-04-20T17:40:33.720 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=48923620}) 2024-04-20T17:40:33.720 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3275887445}) 2024-04-20T17:40:33.721 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2199821761}) 2024-04-20T17:40:33.721 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4145637983}) 2024-04-20T17:40:33.721 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2659166330}) 2024-04-20T17:40:33.721 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1605069063}) 2024-04-20T17:40:33.721 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3247856370}) 2024-04-20T17:40:33.721 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=38540017}) 2024-04-20T17:40:33.721 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3887463664}) 2024-04-20T17:40:33.721 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2353521175}) 2024-04-20T17:40:33.721 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2407299891}) 2024-04-20T17:40:33.721 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1736905590}) 2024-04-20T17:40:33.721 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=274979246}) 2024-04-20T17:40:33.721 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2032756258}) 2024-04-20T17:40:33.721 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077692382}) 2024-04-20T17:40:33.721 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212179151}) 2024-04-20T17:40:33.722 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2733413722}) 2024-04-20T17:40:33.722 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1258116084}) 2024-04-20T17:40:33.722 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3905486420}) 2024-04-20T17:40:33.722 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1121630868}) 2024-04-20T17:40:33.722 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2892269500}) 2024-04-20T17:40:33.722 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3871333198}) 2024-04-20T17:40:33.722 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=632550366}) 2024-04-20T17:40:33.722 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2576397044}) 2024-04-20T17:40:33.722 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3855161178}) 2024-04-20T17:40:33.722 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3102076344}) 2024-04-20T17:40:33.722 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2762478991}) 2024-04-20T17:40:33.722 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1977914096}) 2024-04-20T17:40:33.722 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3890348182}) 2024-04-20T17:40:33.722 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3474027728}) 2024-04-20T17:40:33.723 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2915973875}) 2024-04-20T17:40:33.723 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=27019309}) 2024-04-20T17:40:33.723 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2994952301}) 2024-04-20T17:40:33.723 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=994023521}) 2024-04-20T17:40:33.723 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3483840223}) 2024-04-20T17:40:33.723 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: pgmap v1961: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.8 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 854 op/s; 7826/254328 objects degraded (3.077%); 2.3 MiB/s, 36 objects/s recovering 2024-04-20T17:40:33.723 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=128924519}) 2024-04-20T17:40:33.723 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=79,mdsmap=3,mgr=18} crc {logm=2004065840,mdsmap=2840952586,mgr=1362760828}) 2024-04-20T17:40:33.723 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mgr_command_descs=1,mgr_metadata=2,mgrstat=8,mon_config_key=89} crc {mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=2579880953,mon_config_key=2494101370}) 2024-04-20T17:40:33.723 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2536147002}) 2024-04-20T17:40:33.723 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=986129753}) 2024-04-20T17:40:33.723 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=1,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=81} crc {mon_config_key=1835448079,monmap=717113318,osd_metadata=2824088611,osd_pg_creating=237960803,osdmap=1440344996}) 2024-04-20T17:40:33.723 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=40} crc {osdmap=3825669659}) 2024-04-20T17:40:33.723 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:40:33.724 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:33 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:40:33.724 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:33.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=519704238}) 2024-04-20T17:40:33.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2485333400}) 2024-04-20T17:40:33.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1341590561}) 2024-04-20T17:40:33.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=991132875}) 2024-04-20T17:40:33.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=177569602}) 2024-04-20T17:40:33.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2736000675}) 2024-04-20T17:40:33.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2740286668}) 2024-04-20T17:40:33.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1033876013}) 2024-04-20T17:40:33.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3761915255}) 2024-04-20T17:40:33.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3199281295}) 2024-04-20T17:40:33.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3467706601}) 2024-04-20T17:40:33.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=790994446}) 2024-04-20T17:40:33.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1019756345}) 2024-04-20T17:40:33.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1627014007}) 2024-04-20T17:40:33.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2682920901}) 2024-04-20T17:40:33.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1500066371}) 2024-04-20T17:40:33.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1955537022}) 2024-04-20T17:40:33.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=838572601}) 2024-04-20T17:40:33.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1049549521}) 2024-04-20T17:40:33.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3395866494}) 2024-04-20T17:40:33.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=921530098}) 2024-04-20T17:40:33.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2110546405}) 2024-04-20T17:40:33.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1879758954}) 2024-04-20T17:40:33.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1556517195}) 2024-04-20T17:40:33.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2874559306}) 2024-04-20T17:40:33.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1523013969}) 2024-04-20T17:40:33.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2649192464}) 2024-04-20T17:40:33.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=834894355}) 2024-04-20T17:40:33.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=360966902}) 2024-04-20T17:40:33.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2714469317}) 2024-04-20T17:40:33.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2914552523}) 2024-04-20T17:40:33.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1342690771}) 2024-04-20T17:40:33.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2033521846}) 2024-04-20T17:40:33.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=979659534}) 2024-04-20T17:40:33.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3490060101}) 2024-04-20T17:40:33.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3826767896}) 2024-04-20T17:40:33.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2483589839}) 2024-04-20T17:40:33.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1938640336}) 2024-04-20T17:40:33.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=33049532}) 2024-04-20T17:40:33.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=815528576}) 2024-04-20T17:40:33.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3172563712}) 2024-04-20T17:40:33.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1016505298}) 2024-04-20T17:40:33.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1327109792}) 2024-04-20T17:40:33.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1219856093}) 2024-04-20T17:40:33.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2020756064}) 2024-04-20T17:40:33.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1529304398}) 2024-04-20T17:40:33.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=280397512}) 2024-04-20T17:40:33.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3821369342}) 2024-04-20T17:40:33.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=2546328079,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241}) 2024-04-20T17:40:33.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=59} crc {auth=2927210262,config=1460075751,health=2731299312,kv=2872500370}) 2024-04-20T17:40:33.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2245028860}) 2024-04-20T17:40:33.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3576719492}) 2024-04-20T17:40:33.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3091231664}) 2024-04-20T17:40:33.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1302997735}) 2024-04-20T17:40:33.765 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=599289947}) 2024-04-20T17:40:33.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=642778375}) 2024-04-20T17:40:33.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3758090761}) 2024-04-20T17:40:33.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1075948743}) 2024-04-20T17:40:33.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3594915675}) 2024-04-20T17:40:33.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=22292920}) 2024-04-20T17:40:33.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3371825531}) 2024-04-20T17:40:33.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=630633847}) 2024-04-20T17:40:33.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2481383515}) 2024-04-20T17:40:33.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2539536142}) 2024-04-20T17:40:33.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1688349370}) 2024-04-20T17:40:33.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2435115651}) 2024-04-20T17:40:33.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3888529585}) 2024-04-20T17:40:33.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=525632982}) 2024-04-20T17:40:33.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4047359843}) 2024-04-20T17:40:33.766 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1710598054}) 2024-04-20T17:40:33.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=578643127}) 2024-04-20T17:40:33.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=137475321}) 2024-04-20T17:40:33.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3509541051}) 2024-04-20T17:40:33.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2747137995}) 2024-04-20T17:40:33.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=230808383}) 2024-04-20T17:40:33.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2074641987}) 2024-04-20T17:40:33.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3211751186}) 2024-04-20T17:40:33.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1464289765}) 2024-04-20T17:40:33.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=575887972}) 2024-04-20T17:40:33.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1181202404}) 2024-04-20T17:40:33.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4261504939}) 2024-04-20T17:40:33.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1071466465}) 2024-04-20T17:40:33.767 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1086136799}) 2024-04-20T17:40:33.768 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=605998130}) 2024-04-20T17:40:33.768 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2965670760}) 2024-04-20T17:40:33.768 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1554319229}) 2024-04-20T17:40:33.768 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4040979299}) 2024-04-20T17:40:33.768 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1687080114}) 2024-04-20T17:40:33.768 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2636892967}) 2024-04-20T17:40:33.768 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4008925009}) 2024-04-20T17:40:33.768 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2852236343}) 2024-04-20T17:40:33.768 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2275556127}) 2024-04-20T17:40:33.768 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1754112212}) 2024-04-20T17:40:33.768 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=533976486}) 2024-04-20T17:40:33.768 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2662492679}) 2024-04-20T17:40:33.768 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2354381854}) 2024-04-20T17:40:33.768 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=659248529}) 2024-04-20T17:40:33.768 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1613325388}) 2024-04-20T17:40:33.769 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1090572829}) 2024-04-20T17:40:33.769 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=956354298}) 2024-04-20T17:40:33.769 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=48923620}) 2024-04-20T17:40:33.769 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3275887445}) 2024-04-20T17:40:33.769 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2199821761}) 2024-04-20T17:40:33.769 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4145637983}) 2024-04-20T17:40:33.769 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2659166330}) 2024-04-20T17:40:33.769 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1605069063}) 2024-04-20T17:40:33.769 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3247856370}) 2024-04-20T17:40:33.769 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=38540017}) 2024-04-20T17:40:33.769 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3887463664}) 2024-04-20T17:40:33.769 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2353521175}) 2024-04-20T17:40:33.769 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2407299891}) 2024-04-20T17:40:33.769 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1736905590}) 2024-04-20T17:40:33.769 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=274979246}) 2024-04-20T17:40:33.770 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2032756258}) 2024-04-20T17:40:33.770 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077692382}) 2024-04-20T17:40:33.770 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212179151}) 2024-04-20T17:40:33.770 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2733413722}) 2024-04-20T17:40:33.770 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1258116084}) 2024-04-20T17:40:33.770 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3905486420}) 2024-04-20T17:40:33.770 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1121630868}) 2024-04-20T17:40:33.770 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2892269500}) 2024-04-20T17:40:33.770 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3871333198}) 2024-04-20T17:40:33.770 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=632550366}) 2024-04-20T17:40:33.770 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2576397044}) 2024-04-20T17:40:33.771 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3855161178}) 2024-04-20T17:40:33.771 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3102076344}) 2024-04-20T17:40:33.771 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2762478991}) 2024-04-20T17:40:33.771 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1977914096}) 2024-04-20T17:40:33.771 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3890348182}) 2024-04-20T17:40:33.771 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3474027728}) 2024-04-20T17:40:33.771 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2915973875}) 2024-04-20T17:40:33.771 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=27019309}) 2024-04-20T17:40:33.771 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2994952301}) 2024-04-20T17:40:33.771 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=994023521}) 2024-04-20T17:40:33.771 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3483840223}) 2024-04-20T17:40:33.771 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: pgmap v1961: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.8 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 854 op/s; 7826/254328 objects degraded (3.077%); 2.3 MiB/s, 36 objects/s recovering 2024-04-20T17:40:33.771 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=128924519}) 2024-04-20T17:40:33.771 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=79,mdsmap=3,mgr=18} crc {logm=2004065840,mdsmap=2840952586,mgr=1362760828}) 2024-04-20T17:40:33.771 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mgr_command_descs=1,mgr_metadata=2,mgrstat=8,mon_config_key=89} crc {mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=2579880953,mon_config_key=2494101370}) 2024-04-20T17:40:33.772 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2536147002}) 2024-04-20T17:40:33.772 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=986129753}) 2024-04-20T17:40:33.772 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=1,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=81} crc {mon_config_key=1835448079,monmap=717113318,osd_metadata=2824088611,osd_pg_creating=237960803,osdmap=1440344996}) 2024-04-20T17:40:33.772 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=40} crc {osdmap=3825669659}) 2024-04-20T17:40:33.772 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:40:33.772 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:33 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:40:33.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=519704238}) 2024-04-20T17:40:33.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2485333400}) 2024-04-20T17:40:33.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1341590561}) 2024-04-20T17:40:33.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=991132875}) 2024-04-20T17:40:33.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=177569602}) 2024-04-20T17:40:33.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2736000675}) 2024-04-20T17:40:33.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2740286668}) 2024-04-20T17:40:33.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1033876013}) 2024-04-20T17:40:33.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3761915255}) 2024-04-20T17:40:33.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3199281295}) 2024-04-20T17:40:33.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3467706601}) 2024-04-20T17:40:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=790994446}) 2024-04-20T17:40:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1019756345}) 2024-04-20T17:40:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1627014007}) 2024-04-20T17:40:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2682920901}) 2024-04-20T17:40:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1500066371}) 2024-04-20T17:40:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1955537022}) 2024-04-20T17:40:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=838572601}) 2024-04-20T17:40:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1049549521}) 2024-04-20T17:40:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3395866494}) 2024-04-20T17:40:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=921530098}) 2024-04-20T17:40:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2110546405}) 2024-04-20T17:40:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1879758954}) 2024-04-20T17:40:33.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1556517195}) 2024-04-20T17:40:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2874559306}) 2024-04-20T17:40:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1523013969}) 2024-04-20T17:40:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2649192464}) 2024-04-20T17:40:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=834894355}) 2024-04-20T17:40:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=360966902}) 2024-04-20T17:40:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2714469317}) 2024-04-20T17:40:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2914552523}) 2024-04-20T17:40:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1342690771}) 2024-04-20T17:40:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2033521846}) 2024-04-20T17:40:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=979659534}) 2024-04-20T17:40:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3490060101}) 2024-04-20T17:40:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3826767896}) 2024-04-20T17:40:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2483589839}) 2024-04-20T17:40:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1938640336}) 2024-04-20T17:40:33.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=33049532}) 2024-04-20T17:40:33.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=815528576}) 2024-04-20T17:40:33.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3172563712}) 2024-04-20T17:40:33.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1016505298}) 2024-04-20T17:40:33.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1327109792}) 2024-04-20T17:40:33.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1219856093}) 2024-04-20T17:40:33.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2020756064}) 2024-04-20T17:40:33.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1529304398}) 2024-04-20T17:40:33.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=280397512}) 2024-04-20T17:40:33.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3821369342}) 2024-04-20T17:40:33.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=78,mdsmap=3,mgr=18,mgr_command_descs=1} crc {logm=2546328079,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241}) 2024-04-20T17:40:33.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=59} crc {auth=2927210262,config=1460075751,health=2731299312,kv=2872500370}) 2024-04-20T17:40:33.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2245028860}) 2024-04-20T17:40:33.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3576719492}) 2024-04-20T17:40:33.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3091231664}) 2024-04-20T17:40:33.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1302997735}) 2024-04-20T17:40:33.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=599289947}) 2024-04-20T17:40:33.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=642778375}) 2024-04-20T17:40:33.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3758090761}) 2024-04-20T17:40:33.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1075948743}) 2024-04-20T17:40:33.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3594915675}) 2024-04-20T17:40:33.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=22292920}) 2024-04-20T17:40:33.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3371825531}) 2024-04-20T17:40:33.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=630633847}) 2024-04-20T17:40:33.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2481383515}) 2024-04-20T17:40:33.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2539536142}) 2024-04-20T17:40:33.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1688349370}) 2024-04-20T17:40:33.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2435115651}) 2024-04-20T17:40:33.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3888529585}) 2024-04-20T17:40:33.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=525632982}) 2024-04-20T17:40:33.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4047359843}) 2024-04-20T17:40:33.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1710598054}) 2024-04-20T17:40:33.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=578643127}) 2024-04-20T17:40:33.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=137475321}) 2024-04-20T17:40:33.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3509541051}) 2024-04-20T17:40:33.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2747137995}) 2024-04-20T17:40:33.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=230808383}) 2024-04-20T17:40:33.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2074641987}) 2024-04-20T17:40:33.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3211751186}) 2024-04-20T17:40:33.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1464289765}) 2024-04-20T17:40:33.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=575887972}) 2024-04-20T17:40:33.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1181202404}) 2024-04-20T17:40:33.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4261504939}) 2024-04-20T17:40:33.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1071466465}) 2024-04-20T17:40:33.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1086136799}) 2024-04-20T17:40:33.841 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=605998130}) 2024-04-20T17:40:33.841 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2965670760}) 2024-04-20T17:40:33.841 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1554319229}) 2024-04-20T17:40:33.841 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4040979299}) 2024-04-20T17:40:33.841 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1687080114}) 2024-04-20T17:40:33.841 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2636892967}) 2024-04-20T17:40:33.841 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4008925009}) 2024-04-20T17:40:33.841 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2852236343}) 2024-04-20T17:40:33.841 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2275556127}) 2024-04-20T17:40:33.841 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1754112212}) 2024-04-20T17:40:33.841 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=533976486}) 2024-04-20T17:40:33.841 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2662492679}) 2024-04-20T17:40:33.841 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2354381854}) 2024-04-20T17:40:33.841 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=659248529}) 2024-04-20T17:40:33.842 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1613325388}) 2024-04-20T17:40:33.842 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1090572829}) 2024-04-20T17:40:33.842 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=956354298}) 2024-04-20T17:40:33.842 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=48923620}) 2024-04-20T17:40:33.842 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3275887445}) 2024-04-20T17:40:33.842 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2199821761}) 2024-04-20T17:40:33.842 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4145637983}) 2024-04-20T17:40:33.842 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2659166330}) 2024-04-20T17:40:33.842 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1605069063}) 2024-04-20T17:40:33.842 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3247856370}) 2024-04-20T17:40:33.842 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=38540017}) 2024-04-20T17:40:33.842 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3887463664}) 2024-04-20T17:40:33.842 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2353521175}) 2024-04-20T17:40:33.842 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2407299891}) 2024-04-20T17:40:33.843 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1736905590}) 2024-04-20T17:40:33.843 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=274979246}) 2024-04-20T17:40:33.843 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2032756258}) 2024-04-20T17:40:33.843 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2077692382}) 2024-04-20T17:40:33.843 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2212179151}) 2024-04-20T17:40:33.843 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2733413722}) 2024-04-20T17:40:33.843 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1258116084}) 2024-04-20T17:40:33.843 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3905486420}) 2024-04-20T17:40:33.843 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1121630868}) 2024-04-20T17:40:33.843 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2892269500}) 2024-04-20T17:40:33.843 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3871333198}) 2024-04-20T17:40:33.843 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=632550366}) 2024-04-20T17:40:33.843 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2576397044}) 2024-04-20T17:40:33.843 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3855161178}) 2024-04-20T17:40:33.843 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3102076344}) 2024-04-20T17:40:33.844 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2762478991}) 2024-04-20T17:40:33.844 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1977914096}) 2024-04-20T17:40:33.844 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3890348182}) 2024-04-20T17:40:33.844 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3474027728}) 2024-04-20T17:40:33.844 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2915973875}) 2024-04-20T17:40:33.844 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=27019309}) 2024-04-20T17:40:33.844 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2994952301}) 2024-04-20T17:40:33.844 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=994023521}) 2024-04-20T17:40:33.844 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3483840223}) 2024-04-20T17:40:33.844 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: pgmap v1961: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.8 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 854 op/s; 7826/254328 objects degraded (3.077%); 2.3 MiB/s, 36 objects/s recovering 2024-04-20T17:40:33.844 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=128924519}) 2024-04-20T17:40:33.844 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=79,mdsmap=3,mgr=18} crc {logm=2004065840,mdsmap=2840952586,mgr=1362760828}) 2024-04-20T17:40:33.844 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mgr_command_descs=1,mgr_metadata=2,mgrstat=8,mon_config_key=89} crc {mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=2579880953,mon_config_key=2494101370}) 2024-04-20T17:40:33.845 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2536147002}) 2024-04-20T17:40:33.845 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=986129753}) 2024-04-20T17:40:33.845 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=1,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=81} crc {mon_config_key=1835448079,monmap=717113318,osd_metadata=2824088611,osd_pg_creating=237960803,osdmap=1440344996}) 2024-04-20T17:40:33.845 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=40} crc {osdmap=3825669659}) 2024-04-20T17:40:33.845 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:40:33.845 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:33 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:40:34.089 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:40:34.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:34.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:40:34.099 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-20T17:40:34.199 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:40:34.232 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:34.256 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:40:34.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:34.260 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:40:34.361 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:40:34.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:34 smithi005 ceph-mon[39562]: osdmap e739: 12 total, 11 up, 9 in 2024-04-20T17:40:34.392 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:34.411 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:40:34.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:34.420 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:40:34.520 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:40:34.551 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:34 smithi171 ceph-mon[44189]: osdmap e739: 12 total, 11 up, 9 in 2024-04-20T17:40:34.553 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:34.578 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:40:34.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:34.582 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:40:34.683 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:40:34.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:34 smithi106 ceph-mon[44423]: osdmap e739: 12 total, 11 up, 9 in 2024-04-20T17:40:34.823 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:40:34.824 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:40:34.824 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:40:34.824 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:35.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:35 smithi005 ceph-mon[39562]: pgmap v1963: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.7 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 806 op/s; 7826/251486 objects degraded (3.112%); 2.1 MiB/s, 33 objects/s recovering 2024-04-20T17:40:35.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:35 smithi106 ceph-mon[44423]: pgmap v1963: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.7 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 806 op/s; 7826/251486 objects degraded (3.112%); 2.1 MiB/s, 33 objects/s recovering 2024-04-20T17:40:35.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:35 smithi171 ceph-mon[44189]: pgmap v1963: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.7 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 0 B/s wr, 806 op/s; 7826/251486 objects degraded (3.112%); 2.1 MiB/s, 33 objects/s recovering 2024-04-20T17:40:35.986 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:40:35.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:35.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:40:36.094 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:40:36.143 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:37.708 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:40:37.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:37 smithi005 ceph-mon[39562]: pgmap v1964: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.5 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 975 op/s; 7593/246794 objects degraded (3.077%); 3.6 MiB/s, 57 objects/s recovering 2024-04-20T17:40:37.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:37 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 7826/251486 objects degraded (3.112%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:40:37.717 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:40:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:37 smithi171 ceph-mon[44189]: pgmap v1964: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.5 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 975 op/s; 7593/246794 objects degraded (3.077%); 3.6 MiB/s, 57 objects/s recovering 2024-04-20T17:40:37.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:37 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 7826/251486 objects degraded (3.112%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:40:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:37 smithi106 ceph-mon[44423]: pgmap v1964: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.5 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 975 op/s; 7593/246794 objects degraded (3.077%); 3.6 MiB/s, 57 objects/s recovering 2024-04-20T17:40:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:37 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 7826/251486 objects degraded (3.112%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:40:39.637 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:40:39.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:39.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:40:39.646 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:39 smithi005 ceph-mon[39562]: from='client.27356 v1:172.21.15.5:0/3610273308' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:40:39.646 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:39 smithi005 ceph-mon[39562]: pgmap v1965: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.5 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 769 op/s; 7593/246794 objects degraded (3.077%); 3.5 MiB/s, 56 objects/s recovering 2024-04-20T17:40:39.746 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:40:39.785 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:39.814 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:40:39.818 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:39 smithi171 ceph-mon[44189]: from='client.27356 v1:172.21.15.5:0/3610273308' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:40:39.818 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:39 smithi171 ceph-mon[44189]: pgmap v1965: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.5 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 769 op/s; 7593/246794 objects degraded (3.077%); 3.5 MiB/s, 56 objects/s recovering 2024-04-20T17:40:39.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:39.819 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:40:39.920 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:40:39.950 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:39.972 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:40:39.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:39.975 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:40:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:39 smithi106 ceph-mon[44423]: from='client.27356 v1:172.21.15.5:0/3610273308' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:40:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:39 smithi106 ceph-mon[44423]: pgmap v1965: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.5 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 769 op/s; 7593/246794 objects degraded (3.077%); 3.5 MiB/s, 56 objects/s recovering 2024-04-20T17:40:40.076 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:40:40.104 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:40.137 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:40:40.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:40.143 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:40:40.244 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:40:40.315 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:40.379 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:40:40.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:40.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:40:40.489 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:40:40.516 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:40.534 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:40:40.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:40.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:40:40.638 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:40:40.667 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:40.691 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:40:40.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:40.699 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:40:40.800 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:40:40.829 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:40.847 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:40:40.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:40.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:40:40.951 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:40:41.009 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:41.041 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:40:41.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:41.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:40:41.153 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:40:41.183 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:41.212 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:40:41.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:41.216 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:40:41.317 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:40:41.403 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:41.614 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:40:41.614 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:40:41.614 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:40:41.614 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:40:41.614 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:40:41.616 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:40:41.616 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:40:41.616 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:40:41.616 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:40:41.616 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:40:41.616 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:40:41.617 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:40:41.617 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:40:41.617 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:40:41.617 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:40:41.617 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:40:41.617 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:40:41.617 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:40:41.617 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:40:41.617 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:40:41.617 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:40:41.617 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:40:41.617 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:40:41.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:41 smithi005 ceph-mon[39562]: pgmap v1966: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.4 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 794 op/s; 7345/240954 objects degraded (3.048%); 3.0 MiB/s, 48 objects/s recovering 2024-04-20T17:40:41.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:41 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3328271418' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:40:41.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:41.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:41 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 7345/240954 objects degraded (3.048%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:40:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:41 smithi171 ceph-mon[44189]: pgmap v1966: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.4 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 794 op/s; 7345/240954 objects degraded (3.048%); 3.0 MiB/s, 48 objects/s recovering 2024-04-20T17:40:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:41 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3328271418' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:40:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:41 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 7345/240954 objects degraded (3.048%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:40:41.877 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":1964,"stamp":"2024-04-20T17:40:36.504066+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8087339576,"num_objects":123397,"num_object_clones":0,"num_object_copies":246794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123397,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":168234,"num_write_kb":9329211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10474,"num_bytes_recovered":688413344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161027,"ondisk_log_size":161027,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":19813816,"kb_used_data":15374808,"kb_used_omap":217,"kb_used_meta":4438630,"kb_avail":823892552,"statfs":{"total":863955320832,"available":843665973248,"internally_reserved":0,"allocated":15743803392,"data_stored":15740300344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":222986,"internal_metadata":4545157366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-511705088,"num_objects":-7808,"num_object_clones":0,"num_object_copies":-15616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-459,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7808,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":459,"num_bytes_recovered":30081024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.007907"},"pg_stats":[{"pgid":"7.f","version":"739'10598","reported_seq":10797,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-20T17:40:35.364260+0000","last_change":"2024-04-20T17:39:40.325470+0000","last_active":"2024-04-20T17:40:35.364260+0000","last_peered":"2024-04-20T17:40:35.364260+0000","last_clean":"2024-04-20T17:40:35.364260+0000","last_became_active":"2024-04-20T17:37:56.835750+0000","last_became_peered":"2024-04-20T17:37:56.835750+0000","last_unstale":"2024-04-20T17:40:35.364260+0000","last_undegraded":"2024-04-20T17:40:35.364260+0000","last_fullsized":"2024-04-20T17:40:35.364260+0000","mapping_epoch":694,"log_start":"699'500","ondisk_log_start":"699'500","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9175","last_scrub_stamp":"2024-04-20T17:39:40.325340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:40.325340+0000","objects_scrubbed":9175,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:41:08.457253+0000","scrub_duration":741,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":508035072,"num_objects":7752,"num_object_clones":0,"num_object_copies":15504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10598,"num_write_kb":587200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"739'10428","reported_seq":10608,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-20T17:40:35.950813+0000","last_change":"2024-04-20T17:39:07.189704+0000","last_active":"2024-04-20T17:40:35.950813+0000","last_peered":"2024-04-20T17:40:35.950813+0000","last_clean":"2024-04-20T17:40:35.950813+0000","last_became_active":"2024-04-20T17:37:56.833703+0000","last_became_peered":"2024-04-20T17:37:56.833703+0000","last_unstale":"2024-04-20T17:40:35.950813+0000","last_undegraded":"2024-04-20T17:40:35.950813+0000","last_fullsized":"2024-04-20T17:40:35.950813+0000","mapping_epoch":694,"log_start":"699'400","ondisk_log_start":"699'400","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"716'7277","last_scrub_stamp":"2024-04-20T17:39:07.189514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:07.189514+0000","objects_scrubbed":7254,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:40:38.067638+0000","scrub_duration":1393,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":490995712,"num_objects":7492,"num_object_clones":0,"num_object_copies":14984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10428,"num_write_kb":573440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"739'12686","reported_seq":12989,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-20T17:40:35.326733+0000","last_change":"2024-04-20T17:39:48.311967+0000","last_active":"2024-04-20T17:40:35.326733+0000","last_peered":"2024-04-20T17:40:35.326733+0000","last_clean":"2024-04-20T17:40:35.326733+0000","last_became_active":"2024-04-20T17:37:56.836496+0000","last_became_peered":"2024-04-20T17:37:56.836496+0000","last_unstale":"2024-04-20T17:40:35.326733+0000","last_undegraded":"2024-04-20T17:40:35.326733+0000","last_fullsized":"2024-04-20T17:40:35.326733+0000","mapping_epoch":694,"log_start":"702'2600","ondisk_log_start":"702'2600","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'11204","last_scrub_stamp":"2024-04-20T17:39:48.311825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:48.311825+0000","objects_scrubbed":9436,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:40:50.621890+0000","scrub_duration":785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521338904,"num_objects":7956,"num_object_clones":0,"num_object_copies":15912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7956,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10918,"num_write_kb":603905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.c","version":"739'10416","reported_seq":10614,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-20T17:40:34.130965+0000","last_change":"2024-04-20T17:39:44.683378+0000","last_active":"2024-04-20T17:40:34.130965+0000","last_peered":"2024-04-20T17:40:34.130965+0000","last_clean":"2024-04-20T17:40:34.130965+0000","last_became_active":"2024-04-20T17:37:56.834181+0000","last_became_peered":"2024-04-20T17:37:56.834181+0000","last_unstale":"2024-04-20T17:40:34.130965+0000","last_undegraded":"2024-04-20T17:40:34.130965+0000","last_fullsized":"2024-04-20T17:40:34.130965+0000","mapping_epoch":694,"log_start":"699'400","ondisk_log_start":"699'400","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9063","last_scrub_stamp":"2024-04-20T17:39:44.683201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:44.683201+0000","objects_scrubbed":9063,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:41:08.819063+0000","scrub_duration":746,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505282560,"num_objects":7710,"num_object_clones":0,"num_object_copies":15420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10416,"num_write_kb":580032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"739'10454","reported_seq":10654,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-20T17:40:34.012023+0000","last_change":"2024-04-20T17:39:45.722632+0000","last_active":"2024-04-20T17:40:34.012023+0000","last_peered":"2024-04-20T17:40:34.012023+0000","last_clean":"2024-04-20T17:40:34.012023+0000","last_became_active":"2024-04-20T17:37:56.830487+0000","last_became_peered":"2024-04-20T17:37:56.830487+0000","last_unstale":"2024-04-20T17:40:34.012023+0000","last_undegraded":"2024-04-20T17:40:34.012023+0000","last_fullsized":"2024-04-20T17:40:34.012023+0000","mapping_epoch":694,"log_start":"699'400","ondisk_log_start":"699'400","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9204","last_scrub_stamp":"2024-04-20T17:39:45.722537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:45.722537+0000","objects_scrubbed":9204,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:40:52.089051+0000","scrub_duration":769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521273344,"num_objects":7954,"num_object_clones":0,"num_object_copies":15908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10454,"num_write_kb":589056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.a","version":"739'10387","reported_seq":10584,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-20T17:40:35.857742+0000","last_change":"2024-04-20T17:39:47.268931+0000","last_active":"2024-04-20T17:40:35.857742+0000","last_peered":"2024-04-20T17:40:35.857742+0000","last_clean":"2024-04-20T17:40:35.857742+0000","last_became_active":"2024-04-20T17:37:56.835944+0000","last_became_peered":"2024-04-20T17:37:56.835944+0000","last_unstale":"2024-04-20T17:40:35.857742+0000","last_undegraded":"2024-04-20T17:40:35.857742+0000","last_fullsized":"2024-04-20T17:40:35.857742+0000","mapping_epoch":694,"log_start":"699'300","ondisk_log_start":"699'300","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'8992","last_scrub_stamp":"2024-04-20T17:39:47.268805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:47.268805+0000","objects_scrubbed":8992,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:41:05.514178+0000","scrub_duration":714,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":497876992,"num_objects":7597,"num_object_clones":0,"num_object_copies":15194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10387,"num_write_kb":575488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"739'10466","reported_seq":10647,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-20T17:40:35.340699+0000","last_change":"2024-04-20T17:39:03.675435+0000","last_active":"2024-04-20T17:40:35.340699+0000","last_peered":"2024-04-20T17:40:35.340699+0000","last_clean":"2024-04-20T17:40:35.340699+0000","last_became_active":"2024-04-20T17:37:56.835700+0000","last_became_peered":"2024-04-20T17:37:56.835700+0000","last_unstale":"2024-04-20T17:40:35.340699+0000","last_undegraded":"2024-04-20T17:40:35.340699+0000","last_fullsized":"2024-04-20T17:40:35.340699+0000","mapping_epoch":694,"log_start":"699'400","ondisk_log_start":"699'400","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'7175","last_scrub_stamp":"2024-04-20T17:39:03.675211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:03.675211+0000","objects_scrubbed":7133,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502267904,"num_objects":7664,"num_object_clones":0,"num_object_copies":15328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10466,"num_write_kb":580160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11],"acting":[0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"739'14087","reported_seq":32572,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-20T17:40:35.961894+0000","last_change":"2024-04-20T17:40:10.667197+0000","last_active":"2024-04-20T17:40:35.961894+0000","last_peered":"2024-04-20T17:40:35.961894+0000","last_clean":"2024-04-20T17:40:35.961894+0000","last_became_active":"2024-04-20T17:40:09.546695+0000","last_became_peered":"2024-04-20T17:40:09.546695+0000","last_unstale":"2024-04-20T17:40:35.961894+0000","last_undegraded":"2024-04-20T17:40:35.961894+0000","last_fullsized":"2024-04-20T17:40:35.961894+0000","mapping_epoch":731,"log_start":"704'4001","ondisk_log_start":"704'4001","created":694,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":3399,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:40:38.067604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499318784,"num_objects":7619,"num_object_clones":0,"num_object_copies":15238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10517,"num_write_kb":580352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9062,"num_bytes_recovered":593887232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"739'10526","reported_seq":10707,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-20T17:40:33.687125+0000","last_change":"2024-04-20T17:39:05.344523+0000","last_active":"2024-04-20T17:40:33.687125+0000","last_peered":"2024-04-20T17:40:33.687125+0000","last_clean":"2024-04-20T17:40:33.687125+0000","last_became_active":"2024-04-20T17:37:56.836665+0000","last_became_peered":"2024-04-20T17:37:56.836665+0000","last_unstale":"2024-04-20T17:40:33.687125+0000","last_undegraded":"2024-04-20T17:40:33.687125+0000","last_fullsized":"2024-04-20T17:40:33.687125+0000","mapping_epoch":694,"log_start":"699'500","ondisk_log_start":"699'500","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'7328","last_scrub_stamp":"2024-04-20T17:39:05.344263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:05.344263+0000","objects_scrubbed":7293,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:40:36.506021+0000","scrub_duration":1327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":511311872,"num_objects":7802,"num_object_clones":0,"num_object_copies":15604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10526,"num_write_kb":586496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.5","version":"739'10493","reported_seq":10673,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-20T17:40:34.020866+0000","last_change":"2024-04-20T17:39:04.261938+0000","last_active":"2024-04-20T17:40:34.020866+0000","last_peered":"2024-04-20T17:40:34.020866+0000","last_clean":"2024-04-20T17:40:34.020866+0000","last_became_active":"2024-04-20T17:37:56.833600+0000","last_became_peered":"2024-04-20T17:37:56.833600+0000","last_unstale":"2024-04-20T17:40:34.020866+0000","last_undegraded":"2024-04-20T17:40:34.020866+0000","last_fullsized":"2024-04-20T17:40:34.020866+0000","mapping_epoch":694,"log_start":"699'400","ondisk_log_start":"699'400","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'7212","last_scrub_stamp":"2024-04-20T17:39:04.261753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:04.261753+0000","objects_scrubbed":7172,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:40:38.224648+0000","scrub_duration":1303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":512557056,"num_objects":7821,"num_object_clones":0,"num_object_copies":15642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10493,"num_write_kb":586048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.4","version":"739'10440","reported_seq":10638,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-20T17:40:35.269116+0000","last_change":"2024-04-20T17:39:50.365478+0000","last_active":"2024-04-20T17:40:35.269116+0000","last_peered":"2024-04-20T17:40:35.269116+0000","last_clean":"2024-04-20T17:40:35.269116+0000","last_became_active":"2024-04-20T17:37:56.834727+0000","last_became_peered":"2024-04-20T17:37:56.834727+0000","last_unstale":"2024-04-20T17:40:35.269116+0000","last_undegraded":"2024-04-20T17:40:35.269116+0000","last_fullsized":"2024-04-20T17:40:35.269116+0000","mapping_epoch":694,"log_start":"699'400","ondisk_log_start":"699'400","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9031","last_scrub_stamp":"2024-04-20T17:39:50.365339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:50.365339+0000","objects_scrubbed":9031,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:41:02.075665+0000","scrub_duration":876,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499515392,"num_objects":7622,"num_object_clones":0,"num_object_copies":15244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10440,"num_write_kb":577984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"739'10369","reported_seq":10545,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-20T17:40:34.134115+0000","last_change":"2024-04-20T17:39:00.502710+0000","last_active":"2024-04-20T17:40:34.134115+0000","last_peered":"2024-04-20T17:40:34.134115+0000","last_clean":"2024-04-20T17:40:34.134115+0000","last_became_active":"2024-04-20T17:37:56.832911+0000","last_became_peered":"2024-04-20T17:37:56.832911+0000","last_unstale":"2024-04-20T17:40:34.134115+0000","last_undegraded":"2024-04-20T17:40:34.134115+0000","last_fullsized":"2024-04-20T17:40:34.134115+0000","mapping_epoch":694,"log_start":"699'300","ondisk_log_start":"699'300","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'6831","last_scrub_stamp":"2024-04-20T17:39:00.502556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:00.502556+0000","objects_scrubbed":6782,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":504823808,"num_objects":7703,"num_object_clones":0,"num_object_copies":15406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10369,"num_write_kb":578304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"739'10618","reported_seq":10801,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-20T17:40:34.097403+0000","last_change":"2024-04-20T17:39:08.489529+0000","last_active":"2024-04-20T17:40:34.097403+0000","last_peered":"2024-04-20T17:40:34.097403+0000","last_clean":"2024-04-20T17:40:34.097403+0000","last_became_active":"2024-04-20T17:37:56.834292+0000","last_became_peered":"2024-04-20T17:37:56.834292+0000","last_unstale":"2024-04-20T17:40:34.097403+0000","last_undegraded":"2024-04-20T17:40:34.097403+0000","last_fullsized":"2024-04-20T17:40:34.097403+0000","mapping_epoch":694,"log_start":"699'600","ondisk_log_start":"699'600","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"716'7577","last_scrub_stamp":"2024-04-20T17:39:08.489308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:08.489308+0000","objects_scrubbed":7524,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":516030464,"num_objects":7874,"num_object_clones":0,"num_object_copies":15748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10618,"num_write_kb":591744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"739'10525","reported_seq":10723,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-20T17:40:35.683203+0000","last_change":"2024-04-20T17:39:40.082396+0000","last_active":"2024-04-20T17:40:35.683203+0000","last_peered":"2024-04-20T17:40:35.683203+0000","last_clean":"2024-04-20T17:40:35.683203+0000","last_became_active":"2024-04-20T17:37:56.835512+0000","last_became_peered":"2024-04-20T17:37:56.835512+0000","last_unstale":"2024-04-20T17:40:35.683203+0000","last_undegraded":"2024-04-20T17:40:35.683203+0000","last_fullsized":"2024-04-20T17:40:35.683203+0000","mapping_epoch":694,"log_start":"699'500","ondisk_log_start":"699'500","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9091","last_scrub_stamp":"2024-04-20T17:39:40.082255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:40.082255+0000","objects_scrubbed":9091,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:41:06.918395+0000","scrub_duration":743,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":501809152,"num_objects":7657,"num_object_clones":0,"num_object_copies":15314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10525,"num_write_kb":581824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"739'10495","reported_seq":10693,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-20T17:40:35.869120+0000","last_change":"2024-04-20T17:39:45.346458+0000","last_active":"2024-04-20T17:40:35.869120+0000","last_peered":"2024-04-20T17:40:35.869120+0000","last_clean":"2024-04-20T17:40:35.869120+0000","last_became_active":"2024-04-20T17:37:56.833972+0000","last_became_peered":"2024-04-20T17:37:56.833972+0000","last_unstale":"2024-04-20T17:40:35.869120+0000","last_undegraded":"2024-04-20T17:40:35.869120+0000","last_fullsized":"2024-04-20T17:40:35.869120+0000","mapping_epoch":694,"log_start":"699'400","ondisk_log_start":"699'400","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9032","last_scrub_stamp":"2024-04-20T17:39:45.346352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:45.346352+0000","objects_scrubbed":9032,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:41:11.755272+0000","scrub_duration":769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":496041984,"num_objects":7569,"num_object_clones":0,"num_object_copies":15138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10495,"num_write_kb":578048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.6","version":"739'10403","reported_seq":13389,"reported_epoch":739,"state":"active+recovering+degraded","last_fresh":"2024-04-20T17:40:35.378889+0000","last_change":"2024-04-20T17:39:54.962650+0000","last_active":"2024-04-20T17:40:35.378889+0000","last_peered":"2024-04-20T17:40:35.378889+0000","last_clean":"2024-04-20T17:39:34.507147+0000","last_became_active":"2024-04-20T17:39:54.933393+0000","last_became_peered":"2024-04-20T17:39:54.933393+0000","last_unstale":"2024-04-20T17:40:35.378889+0000","last_undegraded":"2024-04-20T17:39:54.578981+0000","last_fullsized":"2024-04-20T17:40:35.378889+0000","mapping_epoch":727,"log_start":"699'400","ondisk_log_start":"699'400","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'7141","last_scrub_stamp":"2024-04-20T17:39:04.435913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:04.435913+0000","objects_scrubbed":7101,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498270208,"num_objects":7603,"num_object_clones":0,"num_object_copies":15206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10403,"num_write_kb":576192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1402,"num_bytes_recovered":91881472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":7593},{"shards":"0,3","objects":10}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":481,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-04-20T17:40:33.364922+0000","last_change":"2024-04-20T17:39:54.973311+0000","last_active":"2024-04-20T17:40:33.364922+0000","last_peered":"2024-04-20T17:40:33.364922+0000","last_clean":"2024-04-20T17:40:33.364922+0000","last_became_active":"2024-04-20T17:39:54.970208+0000","last_became_peered":"2024-04-20T17:39:54.970208+0000","last_unstale":"2024-04-20T17:40:33.364922+0000","last_undegraded":"2024-04-20T17:40:33.364922+0000","last_fullsized":"2024-04-20T17:40:33.364922+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:38:56.114399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:38:56.114399+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:40:35.144936+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":8086749208,"num_objects":123395,"num_object_clones":0,"num_object_copies":246790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123395,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168053,"num_write_kb":9326273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10464,"num_bytes_recovered":685768704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16842235904,"data_stored":16842227760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160890,"ondisk_log_size":160890,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991651,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2685808,"kb_used_data":2010472,"kb_used_omap":1,"kb_used_meta":675326,"kb_avail":91059344,"statfs":{"total":95995035648,"available":93244768256,"internally_reserved":0,"allocated":2058723328,"data_stored":2058337564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":691534276},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253290,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2026488,"kb_used_data":1481644,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":91718664,"statfs":{"total":95995035648,"available":93919911936,"internally_reserved":0,"allocated":1517203456,"data_stored":1516815240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":557906368},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613039,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2020284,"kb_used_data":1469876,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":91724868,"statfs":{"total":95995035648,"available":93926264832,"internally_reserved":0,"allocated":1505153024,"data_stored":1504761276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":563608000},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809269,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1305592,"kb_used_data":1003692,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92439560,"statfs":{"total":95995035648,"available":94658109440,"internally_reserved":0,"allocated":1027780608,"data_stored":1027392412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,2,3,4,5,6,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":2297807503499,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":2830383448122,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738844,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3109244,"kb_used_data":2482416,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":90635908,"statfs":{"total":95995035648,"available":92811169792,"internally_reserved":0,"allocated":2541993984,"data_stored":2541601716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394134,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2268664,"kb_used_data":1953708,"kb_used_omap":202,"kb_used_meta":314741,"kb_avail":91476488,"statfs":{"total":95995035648,"available":93671923712,"internally_reserved":0,"allocated":2000596992,"data_stored":2000208796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":322295255},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653183,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2847420,"kb_used_data":2512368,"kb_used_omap":4,"kb_used_meta":335035,"kb_avail":90897732,"statfs":{"total":95995035648,"available":93079277568,"internally_reserved":0,"allocated":2572664832,"data_stored":2572272564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":343076551},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986632,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1923764,"kb_used_data":1469164,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":91821388,"statfs":{"total":95995035648,"available":94025101312,"internally_reserved":0,"allocated":1504423936,"data_stored":1504035740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790402,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1626552,"kb_used_data":991468,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":92118600,"statfs":{"total":95995035648,"available":94329446400,"internally_reserved":0,"allocated":1015263232,"data_stored":1014875036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2528907264,"data_stored":2528903192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1985282048,"data_stored":1985282048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2562461696,"data_stored":2562457624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1500315648,"data_stored":1500315648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1010761728,"data_stored":1010761728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1167458304,"data_stored":1167458304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1025507328,"data_stored":1025507328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1500053504,"data_stored":1500053504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1514602496,"data_stored":1514602496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2046885888,"data_stored":2046885888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:40:41.879 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:40:41.974 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:40:41.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:41.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:40:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:41 smithi106 ceph-mon[44423]: pgmap v1966: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.4 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 794 op/s; 7345/240954 objects degraded (3.048%); 3.0 MiB/s, 48 objects/s recovering 2024-04-20T17:40:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:41 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3328271418' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:40:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:41 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 7345/240954 objects degraded (3.048%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:40:42.080 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:40:42.106 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:42.127 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:40:42.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:42.132 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:40:42.233 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:40:42.259 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:42.278 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:40:42.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:42.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:40:42.383 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:40:42.409 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:42.428 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:40:42.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:42.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:40:42.532 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:40:42.559 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:42.572 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:42.577 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:40:42.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:42.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:40:42.681 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:40:42.710 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:42.737 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:40:42.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:42.743 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:40:42.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:42.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:42.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:42.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:42.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:42.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:42.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:42.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:42.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:42.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:42.844 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:40:42.874 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:42.897 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:40:42.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:42.900 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:40:42.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:42.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:42.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:42.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:42.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:42.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:42.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:42.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:42.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:43.001 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:40:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:43.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:43.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:43.030 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:43.052 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:40:43.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:43.056 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:40:43.157 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:40:43.185 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:43.204 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:40:43.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:43.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:40:43.308 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:40:43.403 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:43.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:43 smithi005 ceph-mon[39562]: pgmap v1967: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.3 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 795 op/s; 7345/238410 objects degraded (3.081%); 3.0 MiB/s, 48 objects/s recovering 2024-04-20T17:40:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:43 smithi171 ceph-mon[44189]: pgmap v1967: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.3 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 795 op/s; 7345/238410 objects degraded (3.081%); 3.0 MiB/s, 48 objects/s recovering 2024-04-20T17:40:44.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:43 smithi106 ceph-mon[44423]: pgmap v1967: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.3 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 795 op/s; 7345/238410 objects degraded (3.081%); 3.0 MiB/s, 48 objects/s recovering 2024-04-20T17:40:44.791 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:40:44.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:44.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:40:44.825 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:40:44.825 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:40:44.899 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:40:44.951 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:40:44.951 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:40:44.951 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-04-20T17:40:44.951 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:40:44.955 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:45.654 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:40:45.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:45.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:40:45.722 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:45 smithi005 ceph-mon[39562]: pgmap v1968: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.2 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 847 op/s; 7345/235446 objects degraded (3.120%); 2.7 MiB/s, 43 objects/s recovering 2024-04-20T17:40:45.761 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:40:45.792 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:45.817 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:40:45.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:45.821 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:40:45.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:45 smithi171 ceph-mon[44189]: pgmap v1968: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.2 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 847 op/s; 7345/235446 objects degraded (3.120%); 2.7 MiB/s, 43 objects/s recovering 2024-04-20T17:40:45.922 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:40:45.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:45 smithi106 ceph-mon[44423]: pgmap v1968: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.2 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 0 B/s wr, 847 op/s; 7345/235446 objects degraded (3.120%); 2.7 MiB/s, 43 objects/s recovering 2024-04-20T17:40:45.949 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:45.969 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:40:45.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:45.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:40:46.072 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:40:46.099 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:46.120 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:40:46.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:46.123 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:40:46.224 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:40:46.319 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:46.594 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:46 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1178009269' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:40:46.594 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:46 smithi005 ceph-mon[39562]: from='client.27368 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:40:46.594 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:46 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 7345/235446 objects degraded (3.120%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:40:46.826 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:40:46.905 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:40:47.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:46 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1178009269' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:40:47.078 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:46 smithi106 ceph-mon[44423]: from='client.27368 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:40:47.078 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:46 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 7345/235446 objects degraded (3.120%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:40:47.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:46 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1178009269' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:40:47.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:46 smithi171 ceph-mon[44189]: from='client.27368 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:40:47.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:46 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 7345/235446 objects degraded (3.120%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:40:47.673 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-20T17:40:47.708 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:40:47.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:47.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:40:47.721 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:40:47.819 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:40:47.847 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:47 smithi171 ceph-mon[44189]: pgmap v1969: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 873 op/s; 7089/230504 objects degraded (3.075%); 3.8 MiB/s, 61 objects/s recovering 2024-04-20T17:40:47.847 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:47 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:40:47.847 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:47 smithi171 ceph-mon[44189]: from='client.27368 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:40:47.847 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:47 smithi171 ceph-mon[44189]: osdmap e740: 12 total, 11 up, 9 in 2024-04-20T17:40:47.847 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:47 smithi171 ceph-mon[44189]: 7.7 scrub starts 2024-04-20T17:40:47.849 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:47.875 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:40:47.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:47.889 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:40:47.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:47 smithi005 ceph-mon[39562]: pgmap v1969: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 873 op/s; 7089/230504 objects degraded (3.075%); 3.8 MiB/s, 61 objects/s recovering 2024-04-20T17:40:47.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:47 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:40:47.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:47 smithi005 ceph-mon[39562]: from='client.27368 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:40:47.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:47 smithi005 ceph-mon[39562]: osdmap e740: 12 total, 11 up, 9 in 2024-04-20T17:40:47.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:47 smithi005 ceph-mon[39562]: 7.7 scrub starts 2024-04-20T17:40:47.989 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:40:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:47 smithi106 ceph-mon[44423]: pgmap v1969: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 873 op/s; 7089/230504 objects degraded (3.075%); 3.8 MiB/s, 61 objects/s recovering 2024-04-20T17:40:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:47 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:40:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:47 smithi106 ceph-mon[44423]: from='client.27368 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:40:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:47 smithi106 ceph-mon[44423]: osdmap e740: 12 total, 11 up, 9 in 2024-04-20T17:40:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:47 smithi106 ceph-mon[44423]: 7.7 scrub starts 2024-04-20T17:40:48.016 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:48.034 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:40:48.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:48.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:40:48.139 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:40:48.167 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:48.186 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:40:48.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:48.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:40:48.290 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:40:48.360 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:48.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:48 smithi005 ceph-mon[39562]: 7.7 scrub ok 2024-04-20T17:40:48.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:48 smithi005 ceph-mon[39562]: pgmap v1971: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 813 op/s; 7089/230504 objects degraded (3.075%); 3.1 MiB/s, 50 objects/s recovering 2024-04-20T17:40:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:48 smithi106 ceph-mon[44423]: 7.7 scrub ok 2024-04-20T17:40:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:48 smithi106 ceph-mon[44423]: pgmap v1971: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 813 op/s; 7089/230504 objects degraded (3.075%); 3.1 MiB/s, 50 objects/s recovering 2024-04-20T17:40:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:48 smithi171 ceph-mon[44189]: 7.7 scrub ok 2024-04-20T17:40:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:48 smithi171 ceph-mon[44189]: pgmap v1971: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 7.0 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 813 op/s; 7089/230504 objects degraded (3.075%); 3.1 MiB/s, 50 objects/s recovering 2024-04-20T17:40:49.120 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:40:49.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:49.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:40:49.226 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:40:49.298 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:49.398 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:40:49.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:49.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:40:49.507 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:40:49.537 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:49.566 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:40:49.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:49.572 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:40:49.580 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:40:49.673 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:40:49.700 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:49.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:49 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:40:49.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:49 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:40:49.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:49 smithi005 ceph-mon[39562]: osdmap e741: 12 total, 11 up, 9 in 2024-04-20T17:40:49.719 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:40:49.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:49.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:40:49.822 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:40:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:49 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:40:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:49 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:40:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:49 smithi171 ceph-mon[44189]: osdmap e741: 12 total, 11 up, 9 in 2024-04-20T17:40:49.850 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:49.852 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:40:49.870 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:40:49.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:49.873 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:40:49.974 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:40:50.004 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:49 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:40:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:49 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:40:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:49 smithi106 ceph-mon[44423]: osdmap e741: 12 total, 11 up, 9 in 2024-04-20T17:40:50.030 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:40:50.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:50.035 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:40:50.136 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:40:50.165 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:50.190 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:40:50.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:50.195 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:40:50.296 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:40:50.342 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:50.366 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:40:50.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:50.369 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:40:50.470 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:40:50.499 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:50.518 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:40:50.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:50.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:40:50.622 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:40:50.649 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:50.670 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:40:50.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:50.672 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:40:50.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:50 smithi106 ceph-mon[44423]: pgmap v1973: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 6.9 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 685 op/s; 7089/227434 objects degraded (3.117%); 2.0 MiB/s, 31 objects/s recovering 2024-04-20T17:40:50.773 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:40:50.801 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:50.820 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:40:50.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:50.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:40:50.900 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:40:50.923 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:50 smithi005 ceph-mon[39562]: pgmap v1973: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 6.9 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 685 op/s; 7089/227434 objects degraded (3.117%); 2.0 MiB/s, 31 objects/s recovering 2024-04-20T17:40:50.923 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:40:51.027 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:50 smithi171 ceph-mon[44189]: pgmap v1973: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 6.9 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 685 op/s; 7089/227434 objects degraded (3.117%); 2.0 MiB/s, 31 objects/s recovering 2024-04-20T17:40:51.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:51 smithi005 ceph-mon[39562]: 7.8 scrub starts 2024-04-20T17:40:51.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:51 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:40:51.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:51 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:40:51.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:51 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 7089/227434 objects degraded (3.117%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:40:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:51 smithi106 ceph-mon[44423]: 7.8 scrub starts 2024-04-20T17:40:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:51 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:40:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:51 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:40:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:51 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 7089/227434 objects degraded (3.117%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:40:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:51 smithi171 ceph-mon[44189]: 7.8 scrub starts 2024-04-20T17:40:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:51 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:40:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:51 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:40:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:51 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 7089/227434 objects degraded (3.117%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:40:52.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:52 smithi005 ceph-mon[39562]: 7.8 scrub ok 2024-04-20T17:40:52.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:52 smithi005 ceph-mon[39562]: pgmap v1974: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded, 15 active+clean; 6.8 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 818 op/s; 6847/222352 objects degraded (3.079%); 3.9 MiB/s, 62 objects/s recovering 2024-04-20T17:40:53.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:52 smithi106 ceph-mon[44423]: 7.8 scrub ok 2024-04-20T17:40:53.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:52 smithi106 ceph-mon[44423]: pgmap v1974: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded, 15 active+clean; 6.8 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 818 op/s; 6847/222352 objects degraded (3.079%); 3.9 MiB/s, 62 objects/s recovering 2024-04-20T17:40:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:52 smithi171 ceph-mon[44189]: 7.8 scrub ok 2024-04-20T17:40:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:52 smithi171 ceph-mon[44189]: pgmap v1974: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded, 15 active+clean; 6.8 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 818 op/s; 6847/222352 objects degraded (3.079%); 3.9 MiB/s, 62 objects/s recovering 2024-04-20T17:40:54.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:54 smithi171 ceph-mon[44189]: 7.5 scrub starts 2024-04-20T17:40:54.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:54 smithi005 ceph-mon[39562]: 7.5 scrub starts 2024-04-20T17:40:54.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:54 smithi106 ceph-mon[44423]: 7.5 scrub starts 2024-04-20T17:40:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:55 smithi171 ceph-mon[44189]: 7.5 scrub ok 2024-04-20T17:40:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:55 smithi171 ceph-mon[44189]: pgmap v1975: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering+degraded, 14 active+clean; 6.7 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 697 op/s; 6847/219340 objects degraded (3.122%); 1.9 MiB/s, 30 objects/s recovering 2024-04-20T17:40:55.456 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:40:55.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:55.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:40:55.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:55 smithi005 ceph-mon[39562]: 7.5 scrub ok 2024-04-20T17:40:55.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:55 smithi005 ceph-mon[39562]: pgmap v1975: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering+degraded, 14 active+clean; 6.7 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 697 op/s; 6847/219340 objects degraded (3.122%); 1.9 MiB/s, 30 objects/s recovering 2024-04-20T17:40:55.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:55 smithi106 ceph-mon[44423]: 7.5 scrub ok 2024-04-20T17:40:55.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:55 smithi106 ceph-mon[44423]: pgmap v1975: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering+degraded, 14 active+clean; 6.7 GiB data, 17 GiB used, 787 GiB / 805 GiB avail; 0 B/s wr, 697 op/s; 6847/219340 objects degraded (3.122%); 1.9 MiB/s, 30 objects/s recovering 2024-04-20T17:40:55.554 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:40:55.561 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:40:55.573 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:40:55.588 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:55.607 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:40:55.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:55.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:40:55.713 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:40:55.742 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:55.766 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:40:55.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:55.771 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:40:55.872 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:40:55.899 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:55.917 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:40:55.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:55.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:40:56.021 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:40:56.094 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:56.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:56 smithi106 ceph-mon[44423]: from='client.27262 v1:172.21.15.5:0/3924517834' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:40:56.309 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:56 smithi005 ceph-mon[39562]: from='client.27262 v1:172.21.15.5:0/3924517834' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:40:56.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:56 smithi171 ceph-mon[44189]: from='client.27262 v1:172.21.15.5:0/3924517834' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:40:57.414 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:57 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:40:57.414 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:57 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:40:57.414 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:57 smithi005 ceph-mon[39562]: osdmap e742: 12 total, 11 up, 9 in 2024-04-20T17:40:57.414 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:57 smithi005 ceph-mon[39562]: pgmap v1977: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 6.5 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 6581/214308 objects degraded (3.071%); 4.0 MiB/s, 63 objects/s recovering 2024-04-20T17:40:57.414 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:57 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 6847/219340 objects degraded (3.122%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:40:57.414 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:57 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1062158244' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:40:57.414 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:57.414 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:57.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:57 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:40:57.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:57 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:40:57.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:57 smithi106 ceph-mon[44423]: osdmap e742: 12 total, 11 up, 9 in 2024-04-20T17:40:57.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:57 smithi106 ceph-mon[44423]: pgmap v1977: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 6.5 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 6581/214308 objects degraded (3.071%); 4.0 MiB/s, 63 objects/s recovering 2024-04-20T17:40:57.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:57 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 6847/219340 objects degraded (3.122%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:40:57.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:57 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1062158244' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:40:57.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:57.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:57.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:57 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:40:57.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:57 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:40:57.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:57 smithi171 ceph-mon[44189]: osdmap e742: 12 total, 11 up, 9 in 2024-04-20T17:40:57.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:57 smithi171 ceph-mon[44189]: pgmap v1977: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 6.5 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 6581/214308 objects degraded (3.071%); 4.0 MiB/s, 63 objects/s recovering 2024-04-20T17:40:57.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:57 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 6847/219340 objects degraded (3.122%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:40:57.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:57 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1062158244' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:40:57.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:57.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:57.674 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:40:57.674 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:40:57.674 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:40:57.674 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:40:57.674 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:40:57.674 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:40:57.675 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:40:57.675 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:40:57.675 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:40:57.675 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:40:57.675 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:40:57.675 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:40:57.675 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:40:57.675 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:40:57.675 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:40:57.675 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:40:57.675 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:40:57.676 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:40:57.676 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:40:57.676 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:40:57.676 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:40:57.676 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:40:57.676 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:40:58.288 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:40:58.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:58.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:40:58.391 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:40:58.399 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:58.399 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:58.399 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:58.400 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:58.400 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:58.400 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:58.400 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:58.400 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:58 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:58.423 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:58.423 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:58.423 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:58.423 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:58.423 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:58.423 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:58.423 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:58.423 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:58 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:58.425 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:58.458 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:40:58.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:58.466 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:40:58.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:58.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:58.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:58.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:58.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:58.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:58.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:58.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:58 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:40:58.567 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:40:58.629 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:58.889 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:40:58.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:58.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:40:58.920 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:40:58.920 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:40:58.995 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:40:59.017 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:40:59.018 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:40:59.023 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:59.043 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:40:59.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:59.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:40:59.147 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:40:59.201 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:59.342 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:40:59.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:59.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:40:59.451 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:40:59.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:59 smithi005 ceph-mon[39562]: 7.b scrub starts 2024-04-20T17:40:59.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:59 smithi005 ceph-mon[39562]: pgmap v1978: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 6.5 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 816 op/s; 6581/214308 objects degraded (3.071%); 3.2 MiB/s, 51 objects/s recovering 2024-04-20T17:40:59.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:40:59 smithi005 ceph-mon[39562]: 7.b scrub ok 2024-04-20T17:40:59.498 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:59.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:59 smithi106 ceph-mon[44423]: 7.b scrub starts 2024-04-20T17:40:59.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:59 smithi106 ceph-mon[44423]: pgmap v1978: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 6.5 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 816 op/s; 6581/214308 objects degraded (3.071%); 3.2 MiB/s, 51 objects/s recovering 2024-04-20T17:40:59.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:40:59 smithi106 ceph-mon[44423]: 7.b scrub ok 2024-04-20T17:40:59.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:59 smithi171 ceph-mon[44189]: 7.b scrub starts 2024-04-20T17:40:59.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:59 smithi171 ceph-mon[44189]: pgmap v1978: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 6.5 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 816 op/s; 6581/214308 objects degraded (3.071%); 3.2 MiB/s, 51 objects/s recovering 2024-04-20T17:40:59.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:40:59 smithi171 ceph-mon[44189]: 7.b scrub ok 2024-04-20T17:40:59.669 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:40:59.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:59.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:40:59.716 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-20T17:40:59.782 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:40:59.812 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:59.831 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:40:59.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:59.840 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:40:59.941 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:40:59.969 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:40:59.988 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:40:59.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:40:59.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:41:00.092 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:41:00.123 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:00.147 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:41:00.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:00.150 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:41:00.197 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":1975,"stamp":"2024-04-20T17:40:54.513058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7187726904,"num_objects":109670,"num_object_clones":0,"num_object_copies":219340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109670,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":181961,"num_write_kb":9329211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11220,"num_bytes_recovered":737303200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160854,"ondisk_log_size":160854,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":18146832,"kb_used_data":13675900,"kb_used_omap":217,"kb_used_meta":4470566,"kb_avail":825559536,"statfs":{"total":863955320832,"available":845372964864,"internally_reserved":0,"allocated":14004121600,"data_stored":14000504026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":222986,"internal_metadata":4577859830},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,5,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-365821952,"num_objects":-5582,"num_object_clones":0,"num_object_copies":-11164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5582,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":15859712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001621"},"pg_stats":[{"pgid":"7.f","version":"741'11344","reported_seq":11546,"reported_epoch":741,"state":"active+clean","last_fresh":"2024-04-20T17:40:50.590317+0000","last_change":"2024-04-20T17:39:40.325470+0000","last_active":"2024-04-20T17:40:50.590317+0000","last_peered":"2024-04-20T17:40:50.590317+0000","last_clean":"2024-04-20T17:40:50.590317+0000","last_became_active":"2024-04-20T17:37:56.835750+0000","last_became_peered":"2024-04-20T17:37:56.835750+0000","last_unstale":"2024-04-20T17:40:50.590317+0000","last_undegraded":"2024-04-20T17:40:50.590317+0000","last_fullsized":"2024-04-20T17:40:50.590317+0000","mapping_epoch":694,"log_start":"700'1300","ondisk_log_start":"700'1300","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9175","last_scrub_stamp":"2024-04-20T17:39:40.325340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:40.325340+0000","objects_scrubbed":9175,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:41:08.457253+0000","scrub_duration":741,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459145216,"num_objects":7006,"num_object_clones":0,"num_object_copies":14012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11344,"num_write_kb":587200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"741'11215","reported_seq":11399,"reported_epoch":741,"state":"active+clean","last_fresh":"2024-04-20T17:40:51.002260+0000","last_change":"2024-04-20T17:39:07.189704+0000","last_active":"2024-04-20T17:40:51.002260+0000","last_peered":"2024-04-20T17:40:51.002260+0000","last_clean":"2024-04-20T17:40:51.002260+0000","last_became_active":"2024-04-20T17:37:56.833703+0000","last_became_peered":"2024-04-20T17:37:56.833703+0000","last_unstale":"2024-04-20T17:40:51.002260+0000","last_undegraded":"2024-04-20T17:40:51.002260+0000","last_fullsized":"2024-04-20T17:40:51.002260+0000","mapping_epoch":694,"log_start":"700'1200","ondisk_log_start":"700'1200","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"716'7277","last_scrub_stamp":"2024-04-20T17:39:07.189514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:07.189514+0000","objects_scrubbed":7254,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1393,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439418880,"num_objects":6705,"num_object_clones":0,"num_object_copies":13410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11215,"num_write_kb":573440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"741'13484","reported_seq":13791,"reported_epoch":741,"state":"active+clean","last_fresh":"2024-04-20T17:40:50.529468+0000","last_change":"2024-04-20T17:39:48.311967+0000","last_active":"2024-04-20T17:40:50.529468+0000","last_peered":"2024-04-20T17:40:50.529468+0000","last_clean":"2024-04-20T17:40:50.529468+0000","last_became_active":"2024-04-20T17:37:56.836496+0000","last_became_peered":"2024-04-20T17:37:56.836496+0000","last_unstale":"2024-04-20T17:40:50.529468+0000","last_undegraded":"2024-04-20T17:40:50.529468+0000","last_fullsized":"2024-04-20T17:40:50.529468+0000","mapping_epoch":694,"log_start":"702'3400","ondisk_log_start":"702'3400","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'11204","last_scrub_stamp":"2024-04-20T17:39:48.311825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:48.311825+0000","objects_scrubbed":9436,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:40:50.621890+0000","scrub_duration":785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":469041176,"num_objects":7158,"num_object_clones":0,"num_object_copies":14316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7158,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11716,"num_write_kb":603905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.c","version":"741'11407","reported_seq":11609,"reported_epoch":741,"state":"active+clean","last_fresh":"2024-04-20T17:40:54.149909+0000","last_change":"2024-04-20T17:39:44.683378+0000","last_active":"2024-04-20T17:40:54.149909+0000","last_peered":"2024-04-20T17:40:54.149909+0000","last_clean":"2024-04-20T17:40:54.149909+0000","last_became_active":"2024-04-20T17:37:56.834181+0000","last_became_peered":"2024-04-20T17:37:56.834181+0000","last_unstale":"2024-04-20T17:40:54.149909+0000","last_undegraded":"2024-04-20T17:40:54.149909+0000","last_fullsized":"2024-04-20T17:40:54.149909+0000","mapping_epoch":694,"log_start":"701'1400","ondisk_log_start":"701'1400","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9063","last_scrub_stamp":"2024-04-20T17:39:44.683201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:44.683201+0000","objects_scrubbed":9063,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:41:08.819063+0000","scrub_duration":746,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440336384,"num_objects":6719,"num_object_clones":0,"num_object_copies":13438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11407,"num_write_kb":580032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"741'11514","reported_seq":11718,"reported_epoch":741,"state":"active+clean","last_fresh":"2024-04-20T17:40:53.993543+0000","last_change":"2024-04-20T17:39:45.722632+0000","last_active":"2024-04-20T17:40:53.993543+0000","last_peered":"2024-04-20T17:40:53.993543+0000","last_clean":"2024-04-20T17:40:53.993543+0000","last_became_active":"2024-04-20T17:37:56.830487+0000","last_became_peered":"2024-04-20T17:37:56.830487+0000","last_unstale":"2024-04-20T17:40:53.993543+0000","last_undegraded":"2024-04-20T17:40:53.993543+0000","last_fullsized":"2024-04-20T17:40:53.993543+0000","mapping_epoch":694,"log_start":"701'1500","ondisk_log_start":"701'1500","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9204","last_scrub_stamp":"2024-04-20T17:39:45.722537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:45.722537+0000","objects_scrubbed":9204,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451805184,"num_objects":6894,"num_object_clones":0,"num_object_copies":13788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11514,"num_write_kb":589056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.a","version":"741'11142","reported_seq":11343,"reported_epoch":741,"state":"active+clean","last_fresh":"2024-04-20T17:40:50.895231+0000","last_change":"2024-04-20T17:39:47.268931+0000","last_active":"2024-04-20T17:40:50.895231+0000","last_peered":"2024-04-20T17:40:50.895231+0000","last_clean":"2024-04-20T17:40:50.895231+0000","last_became_active":"2024-04-20T17:37:56.835944+0000","last_became_peered":"2024-04-20T17:37:56.835944+0000","last_unstale":"2024-04-20T17:40:50.895231+0000","last_undegraded":"2024-04-20T17:40:50.895231+0000","last_fullsized":"2024-04-20T17:40:50.895231+0000","mapping_epoch":694,"log_start":"700'1100","ondisk_log_start":"700'1100","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'8992","last_scrub_stamp":"2024-04-20T17:39:47.268805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:47.268805+0000","objects_scrubbed":8992,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:41:05.514178+0000","scrub_duration":714,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448397312,"num_objects":6842,"num_object_clones":0,"num_object_copies":13684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11142,"num_write_kb":575488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"741'11298","reported_seq":11483,"reported_epoch":741,"state":"active+clean","last_fresh":"2024-04-20T17:40:50.544123+0000","last_change":"2024-04-20T17:39:03.675435+0000","last_active":"2024-04-20T17:40:50.544123+0000","last_peered":"2024-04-20T17:40:50.544123+0000","last_clean":"2024-04-20T17:40:50.544123+0000","last_became_active":"2024-04-20T17:37:56.835700+0000","last_became_peered":"2024-04-20T17:37:56.835700+0000","last_unstale":"2024-04-20T17:40:50.544123+0000","last_undegraded":"2024-04-20T17:40:50.544123+0000","last_fullsized":"2024-04-20T17:40:50.544123+0000","mapping_epoch":694,"log_start":"700'1200","ondisk_log_start":"700'1200","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'7175","last_scrub_stamp":"2024-04-20T17:39:03.675211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:03.675211+0000","objects_scrubbed":7133,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447741952,"num_objects":6832,"num_object_clones":0,"num_object_copies":13664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11298,"num_write_kb":580160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11],"acting":[0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"741'14816","reported_seq":33357,"reported_epoch":741,"state":"active+clean+scrubbing","last_fresh":"2024-04-20T17:40:50.972133+0000","last_change":"2024-04-20T17:40:50.124644+0000","last_active":"2024-04-20T17:40:50.972133+0000","last_peered":"2024-04-20T17:40:50.972133+0000","last_clean":"2024-04-20T17:40:50.972133+0000","last_became_active":"2024-04-20T17:40:09.546695+0000","last_became_peered":"2024-04-20T17:40:09.546695+0000","last_unstale":"2024-04-20T17:40:50.972133+0000","last_undegraded":"2024-04-20T17:40:50.972133+0000","last_fullsized":"2024-04-20T17:40:50.972133+0000","mapping_epoch":731,"log_start":"706'4801","ondisk_log_start":"706'4801","created":694,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:37:55.816485+0000","objects_scrubbed":4799,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451543040,"num_objects":6890,"num_object_clones":0,"num_object_copies":13780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11246,"num_write_kb":580352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9062,"num_bytes_recovered":593887232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"741'11537","reported_seq":11802,"reported_epoch":741,"state":"active+clean","last_fresh":"2024-04-20T17:40:53.712986+0000","last_change":"2024-04-20T17:40:48.024415+0000","last_active":"2024-04-20T17:40:53.712986+0000","last_peered":"2024-04-20T17:40:53.712986+0000","last_clean":"2024-04-20T17:40:53.712986+0000","last_became_active":"2024-04-20T17:37:56.836665+0000","last_became_peered":"2024-04-20T17:37:56.836665+0000","last_unstale":"2024-04-20T17:40:53.712986+0000","last_undegraded":"2024-04-20T17:40:53.712986+0000","last_fullsized":"2024-04-20T17:40:53.712986+0000","mapping_epoch":694,"log_start":"701'1500","ondisk_log_start":"701'1500","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'11243","last_scrub_stamp":"2024-04-20T17:40:48.024149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:40:48.024149+0000","objects_scrubbed":7122,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:03.517694+0000","scrub_duration":1322,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445054976,"num_objects":6791,"num_object_clones":0,"num_object_copies":13582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11537,"num_write_kb":586496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.5","version":"741'11507","reported_seq":11756,"reported_epoch":741,"state":"active+clean+scrubbing","last_fresh":"2024-04-20T17:40:54.050365+0000","last_change":"2024-04-20T17:40:53.191967+0000","last_active":"2024-04-20T17:40:54.050365+0000","last_peered":"2024-04-20T17:40:54.050365+0000","last_clean":"2024-04-20T17:40:54.050365+0000","last_became_active":"2024-04-20T17:37:56.833600+0000","last_became_peered":"2024-04-20T17:37:56.833600+0000","last_unstale":"2024-04-20T17:40:54.050365+0000","last_undegraded":"2024-04-20T17:40:54.050365+0000","last_fullsized":"2024-04-20T17:40:54.050365+0000","mapping_epoch":694,"log_start":"701'1500","ondisk_log_start":"701'1500","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'7212","last_scrub_stamp":"2024-04-20T17:39:04.261753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:04.261753+0000","objects_scrubbed":6099,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"scrubbing for 0s","scrub_duration":1303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446103552,"num_objects":6807,"num_object_clones":0,"num_object_copies":13614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11507,"num_write_kb":586048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.4","version":"741'11210","reported_seq":11412,"reported_epoch":741,"state":"active+clean","last_fresh":"2024-04-20T17:40:50.578916+0000","last_change":"2024-04-20T17:39:50.365478+0000","last_active":"2024-04-20T17:40:50.578916+0000","last_peered":"2024-04-20T17:40:50.578916+0000","last_clean":"2024-04-20T17:40:50.578916+0000","last_became_active":"2024-04-20T17:37:56.834727+0000","last_became_peered":"2024-04-20T17:37:56.834727+0000","last_unstale":"2024-04-20T17:40:50.578916+0000","last_undegraded":"2024-04-20T17:40:50.578916+0000","last_fullsized":"2024-04-20T17:40:50.578916+0000","mapping_epoch":694,"log_start":"700'1200","ondisk_log_start":"700'1200","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9031","last_scrub_stamp":"2024-04-20T17:39:50.365339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:50.365339+0000","objects_scrubbed":9031,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:41:02.075665+0000","scrub_duration":876,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449052672,"num_objects":6852,"num_object_clones":0,"num_object_copies":13704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11210,"num_write_kb":577984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"741'11361","reported_seq":11541,"reported_epoch":741,"state":"active+clean","last_fresh":"2024-04-20T17:40:54.148454+0000","last_change":"2024-04-20T17:39:00.502710+0000","last_active":"2024-04-20T17:40:54.148454+0000","last_peered":"2024-04-20T17:40:54.148454+0000","last_clean":"2024-04-20T17:40:54.148454+0000","last_became_active":"2024-04-20T17:37:56.832911+0000","last_became_peered":"2024-04-20T17:37:56.832911+0000","last_unstale":"2024-04-20T17:40:54.148454+0000","last_undegraded":"2024-04-20T17:40:54.148454+0000","last_fullsized":"2024-04-20T17:40:54.148454+0000","mapping_epoch":694,"log_start":"700'1300","ondisk_log_start":"700'1300","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'6831","last_scrub_stamp":"2024-04-20T17:39:00.502556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:00.502556+0000","objects_scrubbed":6782,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439812096,"num_objects":6711,"num_object_clones":0,"num_object_copies":13422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11361,"num_write_kb":578304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"741'11646","reported_seq":11833,"reported_epoch":741,"state":"active+clean","last_fresh":"2024-04-20T17:40:54.152345+0000","last_change":"2024-04-20T17:39:08.489529+0000","last_active":"2024-04-20T17:40:54.152345+0000","last_peered":"2024-04-20T17:40:54.152345+0000","last_clean":"2024-04-20T17:40:54.152345+0000","last_became_active":"2024-04-20T17:37:56.834292+0000","last_became_peered":"2024-04-20T17:37:56.834292+0000","last_unstale":"2024-04-20T17:40:54.152345+0000","last_undegraded":"2024-04-20T17:40:54.152345+0000","last_fullsized":"2024-04-20T17:40:54.152345+0000","mapping_epoch":694,"log_start":"701'1600","ondisk_log_start":"701'1600","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"716'7577","last_scrub_stamp":"2024-04-20T17:39:08.489308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:08.489308+0000","objects_scrubbed":7524,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448659456,"num_objects":6846,"num_object_clones":0,"num_object_copies":13692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11646,"num_write_kb":591744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"741'11298","reported_seq":11500,"reported_epoch":741,"state":"active+clean","last_fresh":"2024-04-20T17:40:50.693359+0000","last_change":"2024-04-20T17:39:40.082396+0000","last_active":"2024-04-20T17:40:50.693359+0000","last_peered":"2024-04-20T17:40:50.693359+0000","last_clean":"2024-04-20T17:40:50.693359+0000","last_became_active":"2024-04-20T17:37:56.835512+0000","last_became_peered":"2024-04-20T17:37:56.835512+0000","last_unstale":"2024-04-20T17:40:50.693359+0000","last_undegraded":"2024-04-20T17:40:50.693359+0000","last_fullsized":"2024-04-20T17:40:50.693359+0000","mapping_epoch":694,"log_start":"700'1200","ondisk_log_start":"700'1200","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9091","last_scrub_stamp":"2024-04-20T17:39:40.082255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:40.082255+0000","objects_scrubbed":9091,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:41:06.918395+0000","scrub_duration":743,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451149824,"num_objects":6884,"num_object_clones":0,"num_object_copies":13768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11298,"num_write_kb":581824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"741'11190","reported_seq":11392,"reported_epoch":741,"state":"active+clean","last_fresh":"2024-04-20T17:40:50.888525+0000","last_change":"2024-04-20T17:39:45.346458+0000","last_active":"2024-04-20T17:40:50.888525+0000","last_peered":"2024-04-20T17:40:50.888525+0000","last_clean":"2024-04-20T17:40:50.888525+0000","last_became_active":"2024-04-20T17:37:56.833972+0000","last_became_peered":"2024-04-20T17:37:56.833972+0000","last_unstale":"2024-04-20T17:40:50.888525+0000","last_undegraded":"2024-04-20T17:40:50.888525+0000","last_fullsized":"2024-04-20T17:40:50.888525+0000","mapping_epoch":694,"log_start":"700'1100","ondisk_log_start":"700'1100","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9032","last_scrub_stamp":"2024-04-20T17:39:45.346352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:45.346352+0000","objects_scrubbed":9032,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:41:11.755272+0000","scrub_duration":769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450494464,"num_objects":6874,"num_object_clones":0,"num_object_copies":13748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11190,"num_write_kb":578048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.6","version":"741'11149","reported_seq":15630,"reported_epoch":741,"state":"active+recovering+degraded","last_fresh":"2024-04-20T17:40:50.599745+0000","last_change":"2024-04-20T17:39:54.962650+0000","last_active":"2024-04-20T17:40:50.599745+0000","last_peered":"2024-04-20T17:40:50.599745+0000","last_clean":"2024-04-20T17:39:34.507147+0000","last_became_active":"2024-04-20T17:39:54.933393+0000","last_became_peered":"2024-04-20T17:39:54.933393+0000","last_unstale":"2024-04-20T17:40:50.599745+0000","last_undegraded":"2024-04-20T17:39:54.578981+0000","last_fullsized":"2024-04-20T17:40:50.599745+0000","mapping_epoch":727,"log_start":"700'1100","ondisk_log_start":"700'1100","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'7141","last_scrub_stamp":"2024-04-20T17:39:04.435913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:04.435913+0000","objects_scrubbed":7101,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449380352,"num_objects":6857,"num_object_clones":0,"num_object_copies":13714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11149,"num_write_kb":576192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2148,"num_bytes_recovered":140771328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":6847},{"shards":"0,3","objects":10}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":493,"reported_epoch":741,"state":"active+clean","last_fresh":"2024-04-20T17:40:50.375896+0000","last_change":"2024-04-20T17:40:50.375929+0000","last_active":"2024-04-20T17:40:50.375896+0000","last_peered":"2024-04-20T17:40:50.375896+0000","last_clean":"2024-04-20T17:40:50.375896+0000","last_became_active":"2024-04-20T17:39:54.970208+0000","last_became_peered":"2024-04-20T17:39:54.970208+0000","last_unstale":"2024-04-20T17:40:50.375896+0000","last_undegraded":"2024-04-20T17:40:50.375896+0000","last_fullsized":"2024-04-20T17:40:50.375896+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:40:50.375828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:40:50.375828+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:14.500087+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":7187136536,"num_objects":109668,"num_object_clones":0,"num_object_copies":219336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109668,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181780,"num_write_kb":9326273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11210,"num_bytes_recovered":734658560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15113199616,"data_stored":15113191472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160717,"ondisk_log_size":160717,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991655,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2421588,"kb_used_data":1746252,"kb_used_omap":1,"kb_used_meta":675326,"kb_avail":91323564,"statfs":{"total":95995035648,"available":93515329536,"internally_reserved":0,"allocated":1788162048,"data_stored":1787758222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":691534276},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253293,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1873744,"kb_used_data":1333060,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":91871408,"statfs":{"total":95995035648,"available":94076321792,"internally_reserved":0,"allocated":1365053440,"data_stored":1364653166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":553646528},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613042,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1885204,"kb_used_data":1326476,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":91859948,"statfs":{"total":95995035648,"available":94064586752,"internally_reserved":0,"allocated":1358311424,"data_stored":1357907618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":572127680},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809273,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1186252,"kb_used_data":876036,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":92558900,"statfs":{"total":95995035648,"available":94780313600,"internally_reserved":0,"allocated":897060864,"data_stored":896660610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[0,1,2,3,4,5,6,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":2297807503503,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":2830383448126,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738847,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2869200,"kb_used_data":2224328,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":90875952,"statfs":{"total":95995035648,"available":93056974848,"internally_reserved":0,"allocated":2277711872,"data_stored":2277307546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394137,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2068560,"kb_used_data":1757764,"kb_used_omap":202,"kb_used_meta":310581,"kb_avail":91676592,"statfs":{"total":95995035648,"available":93876830208,"internally_reserved":0,"allocated":1799950336,"data_stored":1799550082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":318035415},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653187,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2532816,"kb_used_data":2189448,"kb_used_omap":4,"kb_used_meta":343355,"kb_avail":91212336,"statfs":{"total":95995035648,"available":93401432064,"internally_reserved":0,"allocated":2241994752,"data_stored":2241590426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":351596231},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986635,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1772428,"kb_used_data":1326148,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":91972724,"statfs":{"total":95995035648,"available":94180069376,"internally_reserved":0,"allocated":1357975552,"data_stored":1357575298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790405,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1537040,"kb_used_data":896388,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":92208112,"statfs":{"total":95995035648,"available":94421106688,"internally_reserved":0,"allocated":917901312,"data_stored":917501058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2274365440,"data_stored":2274361368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1788215296,"data_stored":1788215296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2231242752,"data_stored":2231238680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1352597504,"data_stored":1352597504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":913833984,"data_stored":913833984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1167458304,"data_stored":1167458304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":895352832,"data_stored":895352832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1353973760,"data_stored":1353973760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1361444864,"data_stored":1361444864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1774714880,"data_stored":1774714880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:41:00.198 INFO:tasks.cephadm.ceph_manager.ceph:PG 7.8 is not active+clean 2024-04-20T17:41:00.199 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '7.8', 'version': "741'14816", 'reported_seq': 33357, 'reported_epoch': 741, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-04-20T17:40:50.972133+0000', 'last_change': '2024-04-20T17:40:50.124644+0000', 'last_active': '2024-04-20T17:40:50.972133+0000', 'last_peered': '2024-04-20T17:40:50.972133+0000', 'last_clean': '2024-04-20T17:40:50.972133+0000', 'last_became_active': '2024-04-20T17:40:09.546695+0000', 'last_became_peered': '2024-04-20T17:40:09.546695+0000', 'last_unstale': '2024-04-20T17:40:50.972133+0000', 'last_undegraded': '2024-04-20T17:40:50.972133+0000', 'last_fullsized': '2024-04-20T17:40:50.972133+0000', 'mapping_epoch': 731, 'log_start': "706'4801", 'ondisk_log_start': "706'4801", 'created': 694, 'last_epoch_clean': 732, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-20T17:37:55.816485+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T17:37:55.816485+0000', 'last_clean_scrub_stamp': '2024-04-20T17:37:55.816485+0000', 'objects_scrubbed': 4799, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'scrubbing for 0s', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 451543040, 'num_objects': 6890, 'num_object_clones': 0, 'num_object_copies': 13780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6890, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11246, 'num_write_kb': 580352, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9062, 'num_bytes_recovered': 593887232, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-20T17:41:00.199 INFO:tasks.cephadm.ceph_manager.ceph:PG 7.5 is not active+clean 2024-04-20T17:41:00.199 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '7.5', 'version': "741'11507", 'reported_seq': 11756, 'reported_epoch': 741, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-04-20T17:40:54.050365+0000', 'last_change': '2024-04-20T17:40:53.191967+0000', 'last_active': '2024-04-20T17:40:54.050365+0000', 'last_peered': '2024-04-20T17:40:54.050365+0000', 'last_clean': '2024-04-20T17:40:54.050365+0000', 'last_became_active': '2024-04-20T17:37:56.833600+0000', 'last_became_peered': '2024-04-20T17:37:56.833600+0000', 'last_unstale': '2024-04-20T17:40:54.050365+0000', 'last_undegraded': '2024-04-20T17:40:54.050365+0000', 'last_fullsized': '2024-04-20T17:40:54.050365+0000', 'mapping_epoch': 694, 'log_start': "701'1500", 'ondisk_log_start': "701'1500", 'created': 694, 'last_epoch_clean': 695, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "714'7212", 'last_scrub_stamp': '2024-04-20T17:39:04.261753+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T17:37:55.816485+0000', 'last_clean_scrub_stamp': '2024-04-20T17:39:04.261753+0000', 'objects_scrubbed': 6099, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'scrubbing for 0s', 'scrub_duration': 1303, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 446103552, 'num_objects': 6807, 'num_object_clones': 0, 'num_object_copies': 13614, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6807, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11507, 'num_write_kb': 586048, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [11, 9], 'acting': [11, 9], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 11, 'acting_primary': 11, 'purged_snaps': []} 2024-04-20T17:41:00.199 INFO:tasks.cephadm.ceph_manager.ceph:PG 7.6 is not active+clean 2024-04-20T17:41:00.199 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '7.6', 'version': "741'11149", 'reported_seq': 15630, 'reported_epoch': 741, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-20T17:40:50.599745+0000', 'last_change': '2024-04-20T17:39:54.962650+0000', 'last_active': '2024-04-20T17:40:50.599745+0000', 'last_peered': '2024-04-20T17:40:50.599745+0000', 'last_clean': '2024-04-20T17:39:34.507147+0000', 'last_became_active': '2024-04-20T17:39:54.933393+0000', 'last_became_peered': '2024-04-20T17:39:54.933393+0000', 'last_unstale': '2024-04-20T17:40:50.599745+0000', 'last_undegraded': '2024-04-20T17:39:54.578981+0000', 'last_fullsized': '2024-04-20T17:40:50.599745+0000', 'mapping_epoch': 727, 'log_start': "700'1100", 'ondisk_log_start': "700'1100", 'created': 694, 'last_epoch_clean': 695, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "714'7141", 'last_scrub_stamp': '2024-04-20T17:39:04.435913+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T17:37:55.816485+0000', 'last_clean_scrub_stamp': '2024-04-20T17:39:04.435913+0000', 'objects_scrubbed': 7101, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1421, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 449380352, 'num_objects': 6857, 'num_object_clones': 0, 'num_object_copies': 13714, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 6847, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6857, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11149, 'num_write_kb': 576192, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2148, 'num_bytes_recovered': 140771328, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': ['0'], 'object_location_counts': [{'shards': '0', 'objects': 6847}, {'shards': '0,3', 'objects': 10}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-20T17:41:00.199 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph status --format=json 2024-04-20T17:41:00.235 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:41:00.251 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:41:00.278 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:00.304 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:41:00.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:00.310 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:41:00.318 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:41:00.411 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:41:00.457 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:00.490 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:41:00.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:00.496 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:41:00.597 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:41:00.624 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:00.643 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:41:00.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:00.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:41:00.749 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:41:00.794 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:00.899 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:41:00.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:00.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:41:00.921 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:41:01.003 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:41:01.045 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:01.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:01 smithi005 ceph-mon[39562]: pgmap v1979: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 6.4 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 806 op/s; 6581/211294 objects degraded (3.115%); 3.2 MiB/s, 50 objects/s recovering 2024-04-20T17:41:01.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:01 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 6581/211294 objects degraded (3.115%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:01 smithi106 ceph-mon[44423]: pgmap v1979: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 6.4 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 806 op/s; 6581/211294 objects degraded (3.115%); 3.2 MiB/s, 50 objects/s recovering 2024-04-20T17:41:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:01 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 6581/211294 objects degraded (3.115%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:01 smithi171 ceph-mon[44189]: pgmap v1979: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 6.4 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 806 op/s; 6581/211294 objects degraded (3.115%); 3.2 MiB/s, 50 objects/s recovering 2024-04-20T17:41:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:01 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 6581/211294 objects degraded (3.115%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:02.604 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:02 smithi005 ceph-mon[39562]: 7.e scrub starts 2024-04-20T17:41:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:02 smithi106 ceph-mon[44423]: 7.e scrub starts 2024-04-20T17:41:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:02 smithi171 ceph-mon[44189]: 7.e scrub starts 2024-04-20T17:41:03.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:03 smithi005 ceph-mon[39562]: 7.e scrub ok 2024-04-20T17:41:03.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:03 smithi005 ceph-mon[39562]: pgmap v1980: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded, 15 active+clean; 6.3 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 814 op/s; 6340/206044 objects degraded (3.077%); 3.2 MiB/s, 50 objects/s recovering 2024-04-20T17:41:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:03 smithi106 ceph-mon[44423]: 7.e scrub ok 2024-04-20T17:41:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:03 smithi106 ceph-mon[44423]: pgmap v1980: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded, 15 active+clean; 6.3 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 814 op/s; 6340/206044 objects degraded (3.077%); 3.2 MiB/s, 50 objects/s recovering 2024-04-20T17:41:04.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:03 smithi171 ceph-mon[44189]: 7.e scrub ok 2024-04-20T17:41:04.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:03 smithi171 ceph-mon[44189]: pgmap v1980: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded, 15 active+clean; 6.3 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 814 op/s; 6340/206044 objects degraded (3.077%); 3.2 MiB/s, 50 objects/s recovering 2024-04-20T17:41:04.236 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:41:04.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:04.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:41:04.338 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:41:04.369 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:04.389 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:41:04.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:04.394 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:41:04.495 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:41:04.525 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:04.552 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:41:04.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:04.558 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:41:04.638 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:04 smithi171 ceph-mon[44189]: pgmap v1981: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded, 15 active+clean; 6.2 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 816 op/s; 6340/202998 objects degraded (3.123%); 3.2 MiB/s, 50 objects/s recovering 2024-04-20T17:41:04.659 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:41:04.691 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:04.715 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:41:04.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:04.718 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:41:04.819 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:41:04.895 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:04 smithi005 ceph-mon[39562]: pgmap v1981: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded, 15 active+clean; 6.2 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 816 op/s; 6340/202998 objects degraded (3.123%); 3.2 MiB/s, 50 objects/s recovering 2024-04-20T17:41:04.908 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:04.949 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:41:04.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:04.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:41:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:04 smithi106 ceph-mon[44423]: pgmap v1981: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded, 15 active+clean; 6.2 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 0 B/s wr, 816 op/s; 6340/202998 objects degraded (3.123%); 3.2 MiB/s, 50 objects/s recovering 2024-04-20T17:41:05.052 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:41:05.080 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:05.103 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:41:05.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:05.106 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:41:05.207 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:41:05.234 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:05.253 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:41:05.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:05.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:41:05.357 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:41:05.384 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:05.402 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:41:05.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:05.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:41:05.506 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:41:05.533 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:05.552 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:41:05.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:05.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:41:05.656 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:41:05.729 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:06.714 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:41:06.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:06.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:41:06.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:06 smithi106 ceph-mon[44423]: 7.a scrub starts 2024-04-20T17:41:06.823 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:41:06.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:06 smithi171 ceph-mon[44189]: 7.a scrub starts 2024-04-20T17:41:06.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:06 smithi005 ceph-mon[39562]: 7.a scrub starts 2024-04-20T17:41:06.874 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:07.122 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:41:07.123 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:41:07.131 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:41:07.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:07.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:41:07.234 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:41:07.234 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:41:07.238 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:41:07.311 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:07 smithi171 ceph-mon[44189]: pgmap v1982: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 6.0 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 784 op/s; 6089/197942 objects degraded (3.076%); 2.9 MiB/s, 47 objects/s recovering 2024-04-20T17:41:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:07 smithi171 ceph-mon[44189]: 7.a scrub ok 2024-04-20T17:41:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:07 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 6340/202998 objects degraded (3.123%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:07 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:41:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:07 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:41:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:07 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:41:07.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:07 smithi005 ceph-mon[39562]: pgmap v1982: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 6.0 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 784 op/s; 6089/197942 objects degraded (3.076%); 2.9 MiB/s, 47 objects/s recovering 2024-04-20T17:41:07.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:07 smithi005 ceph-mon[39562]: 7.a scrub ok 2024-04-20T17:41:07.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:07 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 6340/202998 objects degraded (3.123%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:07.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:07 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:41:07.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:07 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:41:07.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:07 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:41:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:07 smithi106 ceph-mon[44423]: pgmap v1982: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 6.0 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 784 op/s; 6089/197942 objects degraded (3.076%); 2.9 MiB/s, 47 objects/s recovering 2024-04-20T17:41:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:07 smithi106 ceph-mon[44423]: 7.a scrub ok 2024-04-20T17:41:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:07 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 6340/202998 objects degraded (3.123%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:07 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:41:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:07 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:41:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:07 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:41:08.624 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:41:08.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:08 smithi005 ceph-mon[39562]: 7.1 scrub starts 2024-04-20T17:41:08.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:08 smithi005 ceph-mon[39562]: osdmap e743: 12 total, 11 up, 9 in 2024-04-20T17:41:08.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:08 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2652194698' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:41:08.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:08 smithi005 ceph-mon[39562]: from='client.27386 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:41:08.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:08 smithi005 ceph-mon[39562]: 7.1 scrub ok 2024-04-20T17:41:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:08 smithi106 ceph-mon[44423]: 7.1 scrub starts 2024-04-20T17:41:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:08 smithi106 ceph-mon[44423]: osdmap e743: 12 total, 11 up, 9 in 2024-04-20T17:41:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:08 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2652194698' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:41:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:08 smithi106 ceph-mon[44423]: from='client.27386 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:41:09.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:08 smithi106 ceph-mon[44423]: 7.1 scrub ok 2024-04-20T17:41:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:08 smithi171 ceph-mon[44189]: 7.1 scrub starts 2024-04-20T17:41:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:08 smithi171 ceph-mon[44189]: osdmap e743: 12 total, 11 up, 9 in 2024-04-20T17:41:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:08 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2652194698' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:41:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:08 smithi171 ceph-mon[44189]: from='client.27386 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:41:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:08 smithi171 ceph-mon[44189]: 7.1 scrub ok 2024-04-20T17:41:09.328 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:41:09.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:09.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:41:09.453 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:41:09.484 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:09.512 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:41:09.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:09.519 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:41:09.620 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:41:09.682 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:09.730 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:41:09.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:09.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:41:09.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:09 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:41:09.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:09 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:41:09.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:09 smithi171 ceph-mon[44189]: pgmap v1984: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 6.0 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 817 op/s; 6089/197942 objects degraded (3.076%); 3.1 MiB/s, 49 objects/s recovering 2024-04-20T17:41:09.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:09 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:41:09.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:09 smithi171 ceph-mon[44189]: from='client.27386 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:41:09.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:09 smithi171 ceph-mon[44189]: osdmap e744: 12 total, 11 up, 9 in 2024-04-20T17:41:09.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:09 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:41:09.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:09 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:41:09.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:09 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:41:09.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:09 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:41:09.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:09 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:41:09.837 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:41:09.892 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:09 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:41:09.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:09 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:41:09.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:09 smithi005 ceph-mon[39562]: pgmap v1984: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 6.0 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 817 op/s; 6089/197942 objects degraded (3.076%); 3.1 MiB/s, 49 objects/s recovering 2024-04-20T17:41:09.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:09 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:41:09.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:09 smithi005 ceph-mon[39562]: from='client.27386 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:41:09.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:09 smithi005 ceph-mon[39562]: osdmap e744: 12 total, 11 up, 9 in 2024-04-20T17:41:09.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:09 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:41:09.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:09 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:41:09.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:09 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:41:09.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:09 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:41:09.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:09 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:41:09.895 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:09.965 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:41:09.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:09.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:41:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:09 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:41:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:09 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:41:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:09 smithi106 ceph-mon[44423]: pgmap v1984: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 6.0 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 817 op/s; 6089/197942 objects degraded (3.076%); 3.1 MiB/s, 49 objects/s recovering 2024-04-20T17:41:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:09 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:41:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:09 smithi106 ceph-mon[44423]: from='client.27386 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:41:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:09 smithi106 ceph-mon[44423]: osdmap e744: 12 total, 11 up, 9 in 2024-04-20T17:41:10.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:09 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:41:10.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:09 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:41:10.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:09 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:41:10.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:09 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:41:10.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:09 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:41:10.070 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:41:10.102 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:10.123 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:41:10.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:10.132 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:41:10.233 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:41:10.261 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:10.280 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:41:10.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:10.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:41:10.384 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:41:10.411 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:10.429 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:41:10.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:10.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:41:10.533 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:41:10.582 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:11.463 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-20T17:41:11.571 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:41:11.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:11.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:41:11.687 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:41:11.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:11 smithi005 ceph-mon[39562]: pgmap v1986: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 5.9 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 698 op/s; 6089/194866 objects degraded (3.125%); 2.0 MiB/s, 31 objects/s recovering 2024-04-20T17:41:11.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:11 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 6089/194866 objects degraded (3.125%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:11.765 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:11 smithi106 ceph-mon[44423]: pgmap v1986: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 5.9 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 698 op/s; 6089/194866 objects degraded (3.125%); 2.0 MiB/s, 31 objects/s recovering 2024-04-20T17:41:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:11 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 6089/194866 objects degraded (3.125%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:11 smithi171 ceph-mon[44189]: pgmap v1986: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 5.9 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 698 op/s; 6089/194866 objects degraded (3.125%); 2.0 MiB/s, 31 objects/s recovering 2024-04-20T17:41:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:11 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 6089/194866 objects degraded (3.125%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:12.356 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:41:12.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:12 smithi005 ceph-mon[39562]: pgmap v1987: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 5.8 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 815 op/s; 5841/189948 objects degraded (3.075%); 3.9 MiB/s, 62 objects/s recovering 2024-04-20T17:41:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:12 smithi106 ceph-mon[44423]: pgmap v1987: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 5.8 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 815 op/s; 5841/189948 objects degraded (3.075%); 3.9 MiB/s, 62 objects/s recovering 2024-04-20T17:41:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:12 smithi171 ceph-mon[44189]: pgmap v1987: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 5.8 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 815 op/s; 5841/189948 objects degraded (3.075%); 3.9 MiB/s, 62 objects/s recovering 2024-04-20T17:41:13.289 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:41:13.343 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:41:13.396 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:41:13.397 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:41:13.439 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:41:13.467 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:41:13.522 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:41:13.557 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:41:13.601 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:41:13.637 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:41:13.675 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:41:13.675 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:41:13.675 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:41:13.675 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:41:13.675 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:41:13.675 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:41:13.675 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:41:13.675 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:41:13.676 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:41:13.676 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:41:13.676 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:41:13.676 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:41:13.676 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:41:13.949 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:41:13.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:13.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:41:13.960 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:13 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/780159931' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:41:14.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:13 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/780159931' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:41:14.060 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:41:14.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:13 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/780159931' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:41:14.113 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:14.200 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:41:14.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:14.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:41:14.308 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:41:14.352 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:14.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:14.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:14.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:14.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:14.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:14.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:14.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:14.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:14.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:14.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:14.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:14 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:14.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:14 smithi005 ceph-mon[39562]: pgmap v1988: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 5.7 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 690 op/s; 5841/186898 objects degraded (3.125%); 1.9 MiB/s, 30 objects/s recovering 2024-04-20T17:41:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:15.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:15.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:15.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:14 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:15.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:14 smithi106 ceph-mon[44423]: pgmap v1988: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 5.7 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 690 op/s; 5841/186898 objects degraded (3.125%); 1.9 MiB/s, 30 objects/s recovering 2024-04-20T17:41:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:15.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:15.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:14 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:15.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:14 smithi171 ceph-mon[44189]: pgmap v1988: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 5.7 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 690 op/s; 5841/186898 objects degraded (3.125%); 1.9 MiB/s, 30 objects/s recovering 2024-04-20T17:41:15.442 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:41:15.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:15.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:41:15.466 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:41:15.467 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:41:15.547 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:41:15.593 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:15.690 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:15 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:41:15.691 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:15 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:41:15.691 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:15 smithi005 ceph-mon[39562]: osdmap e745: 12 total, 11 up, 9 in 2024-04-20T17:41:15.769 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:41:15.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:15.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:41:15.830 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:41:15.832 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:41:15.873 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:41:15.922 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:15 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:41:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:15 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:41:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:15 smithi106 ceph-mon[44423]: osdmap e745: 12 total, 11 up, 9 in 2024-04-20T17:41:16.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:15 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:41:16.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:15 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:41:16.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:15 smithi171 ceph-mon[44189]: osdmap e745: 12 total, 11 up, 9 in 2024-04-20T17:41:16.430 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:41:16.430 INFO:teuthology.orchestra.run.smithi005.stdout:{"fsid":"13c6e76c-ff37-11ee-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5841/186898 objects degraded (3.125%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":2444,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":745,"num_osds":12,"num_up_osds":11,"osd_up_since":1713634774,"num_in_osds":9,"osd_in_since":1713634793,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovering+degraded","count":1}],"num_pgs":17,"num_pools":2,"num_objects":93449,"data_bytes":6124667448,"bytes_used":16520814592,"bytes_avail":847434506240,"bytes_total":863955320832,"degraded_objects":5841,"degraded_total":186898,"degraded_ratio":0.031252340849019254,"recovering_objects_per_sec":30,"recovering_bytes_per_sec":2031194,"recovering_keys_per_sec":0,"num_objects_recovered":248,"num_bytes_recovered":16252928,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":690},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.5:8443/"}},"servicemap":{"epoch":55,"modified":"2024-04-20T17:39:44.480882+0000","services":{}},"progress_events":{}} 2024-04-20T17:41:16.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:16 smithi005 ceph-mon[39562]: 7.0 scrub starts 2024-04-20T17:41:16.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:16 smithi005 ceph-mon[39562]: 7.0 scrub ok 2024-04-20T17:41:16.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:16 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3638749681' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:41:16.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:16 smithi005 ceph-mon[39562]: pgmap v1990: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 5.5 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 5565/181716 objects degraded (3.062%); 4.1 MiB/s, 65 objects/s recovering 2024-04-20T17:41:16.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:16 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 5841/186898 objects degraded (3.125%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:17.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:16 smithi106 ceph-mon[44423]: 7.0 scrub starts 2024-04-20T17:41:17.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:16 smithi106 ceph-mon[44423]: 7.0 scrub ok 2024-04-20T17:41:17.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:16 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3638749681' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:41:17.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:16 smithi106 ceph-mon[44423]: pgmap v1990: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 5.5 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 5565/181716 objects degraded (3.062%); 4.1 MiB/s, 65 objects/s recovering 2024-04-20T17:41:17.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:16 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 5841/186898 objects degraded (3.125%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:17.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:16 smithi171 ceph-mon[44189]: 7.0 scrub starts 2024-04-20T17:41:17.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:16 smithi171 ceph-mon[44189]: 7.0 scrub ok 2024-04-20T17:41:17.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:16 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3638749681' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:41:17.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:16 smithi171 ceph-mon[44189]: pgmap v1990: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 5.5 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.01k op/s; 5565/181716 objects degraded (3.062%); 4.1 MiB/s, 65 objects/s recovering 2024-04-20T17:41:17.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:16 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 5841/186898 objects degraded (3.125%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:17.467 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:41:18.119 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:41:18.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:18.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:41:18.224 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:41:18.299 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:18.346 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:41:18.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:18.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:41:18.466 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:41:18.492 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:18.511 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:41:18.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:18.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:41:18.615 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:41:18.641 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:18.663 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:41:18.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:18.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:41:18.767 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:41:18.821 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:18.890 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:41:18.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:18.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:41:18.999 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:41:19.029 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:19.052 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:41:19.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:19.056 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:41:19.157 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:41:19.184 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:19.203 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:41:19.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:19.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:41:19.307 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:41:19.322 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-20T17:41:19.322 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:41:19.339 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:19.364 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:41:19.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:19.369 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:41:19.470 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:41:19.499 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:19.523 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:41:19.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:19.527 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:41:19.628 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:41:19.654 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:19 smithi171 ceph-mon[44189]: pgmap v1991: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 5.5 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 816 op/s; 5565/181716 objects degraded (3.062%); 3.3 MiB/s, 52 objects/s recovering 2024-04-20T17:41:19.656 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:19.679 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:41:19.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:19.682 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:41:19.783 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:41:19.809 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:19.834 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:41:19.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:19.837 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:41:19.938 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:41:19.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:19 smithi005 ceph-mon[39562]: pgmap v1991: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 5.5 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 816 op/s; 5565/181716 objects degraded (3.062%); 3.3 MiB/s, 52 objects/s recovering 2024-04-20T17:41:19.969 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:19.991 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:41:19.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:19.994 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:41:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:19 smithi106 ceph-mon[44423]: pgmap v1991: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 5.5 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 816 op/s; 5565/181716 objects degraded (3.062%); 3.3 MiB/s, 52 objects/s recovering 2024-04-20T17:41:20.095 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:41:20.147 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:20.191 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:41:20.191 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:41:20.191 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.191 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:204605 7.6 7:6ed5a6a0:::benchmark_data_smithi171_161871_object58879:head [delete] snapc 0=[] ondisk+write+known_if_redirected e745)", 2024-04-20T17:41:20.191 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:41:19.947319+0000", 2024-04-20T17:41:20.191 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.034225685999999998, 2024-04-20T17:41:20.191 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.034259029000000003, 2024-04-20T17:41:20.191 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:41:20.191 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:41:20.192 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "waiting for degraded object", 2024-04-20T17:41:20.192 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:41:20.192 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:41:20.192 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:41:20.192 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 204605 2024-04-20T17:41:20.192 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.192 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:41:20.192 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.192 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:41:20.192 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.947319+0000", 2024-04-20T17:41:20.192 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:41:20.192 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.192 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.193 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:41:20.193 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.947319+0000", 2024-04-20T17:41:20.193 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:41:20.193 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.193 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.193 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:41:20.193 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.947322+0000", 2024-04-20T17:41:20.193 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9490000000000001e-06 2024-04-20T17:41:20.193 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.193 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.193 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:41:20.193 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.947331+0000", 2024-04-20T17:41:20.193 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.5230000000000001e-06 2024-04-20T17:41:20.193 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.194 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.194 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:41:20.194 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.947345+0000", 2024-04-20T17:41:20.194 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3478e-05 2024-04-20T17:41:20.194 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.194 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.194 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:41:20.194 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.947356+0000", 2024-04-20T17:41:20.194 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1688999999999999e-05 2024-04-20T17:41:20.194 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.194 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.194 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:41:20.194 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.967265+0000", 2024-04-20T17:41:20.195 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.019909021999999998 2024-04-20T17:41:20.195 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.195 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.195 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:41:20.195 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.972378+0000", 2024-04-20T17:41:20.195 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0051126210000000004 2024-04-20T17:41:20.195 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:41:20.195 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:41:20.195 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:41:20.195 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.195 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.195 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:204551 7.6 7:6665d6b2:::benchmark_data_smithi171_161871_object58825:head [delete] snapc 0=[] ondisk+write+known_if_redirected e745)", 2024-04-20T17:41:20.195 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:41:19.883636+0000", 2024-04-20T17:41:20.195 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.097908248000000003, 2024-04-20T17:41:20.196 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.098025620999999993, 2024-04-20T17:41:20.196 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:41:20.196 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:41:20.196 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:41:20.196 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:41:20.196 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:41:20.196 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:41:20.196 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 204551 2024-04-20T17:41:20.196 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.196 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:41:20.196 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.196 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:41:20.196 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.883636+0000", 2024-04-20T17:41:20.197 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:41:20.197 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.197 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.197 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:41:20.197 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.883636+0000", 2024-04-20T17:41:20.197 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:41:20.197 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.197 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.197 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:41:20.197 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.883638+0000", 2024-04-20T17:41:20.197 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2230000000000001e-06 2024-04-20T17:41:20.197 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.197 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.197 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:41:20.198 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.883645+0000", 2024-04-20T17:41:20.198 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.613e-06 2024-04-20T17:41:20.198 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.198 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.198 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:41:20.198 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.883658+0000", 2024-04-20T17:41:20.198 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3093e-05 2024-04-20T17:41:20.198 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.198 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.198 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:41:20.198 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.883667+0000", 2024-04-20T17:41:20.198 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9590000000000001e-06 2024-04-20T17:41:20.198 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.198 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.199 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:41:20.199 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.883729+0000", 2024-04-20T17:41:20.199 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2149000000000004e-05 2024-04-20T17:41:20.199 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.199 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.199 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:41:20.199 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.889937+0000", 2024-04-20T17:41:20.199 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0062075159999999997 2024-04-20T17:41:20.199 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.199 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.199 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:41:20.199 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.947895+0000", 2024-04-20T17:41:20.199 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057958008999999998 2024-04-20T17:41:20.199 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.200 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.200 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:41:20.200 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.947980+0000", 2024-04-20T17:41:20.200 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5309999999999995e-05 2024-04-20T17:41:20.200 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.200 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.200 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:41:20.200 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.948104+0000", 2024-04-20T17:41:20.200 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000124115 2024-04-20T17:41:20.200 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.200 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.200 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:41:20.200 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.967228+0000", 2024-04-20T17:41:20.200 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.019123689999999999 2024-04-20T17:41:20.200 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:41:20.201 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:41:20.201 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:41:20.201 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.201 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.201 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:204619 7.d 7:be623ea9:::benchmark_data_smithi171_161871_object58893:head [delete] snapc 0=[] ondisk+write+known_if_redirected e745)", 2024-04-20T17:41:20.201 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:41:19.966476+0000", 2024-04-20T17:41:20.201 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.015068715, 2024-04-20T17:41:20.201 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015282201, 2024-04-20T17:41:20.201 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:41:20.201 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:41:20.201 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:41:20.201 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:41:20.201 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:41:20.201 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:41:20.202 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 204619 2024-04-20T17:41:20.202 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.202 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:41:20.202 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.202 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:41:20.202 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.966476+0000", 2024-04-20T17:41:20.202 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:41:20.202 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.202 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.202 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:41:20.202 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.966476+0000", 2024-04-20T17:41:20.202 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:41:20.202 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.202 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.203 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:41:20.203 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.966478+0000", 2024-04-20T17:41:20.203 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.243e-06 2024-04-20T17:41:20.203 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.203 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.203 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:41:20.203 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.966484+0000", 2024-04-20T17:41:20.203 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3060000000000003e-06 2024-04-20T17:41:20.203 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.203 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.203 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:41:20.203 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.966498+0000", 2024-04-20T17:41:20.203 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3584000000000001e-05 2024-04-20T17:41:20.203 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.203 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.204 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:41:20.204 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.966505+0000", 2024-04-20T17:41:20.204 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4769999999999997e-06 2024-04-20T17:41:20.204 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.204 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.204 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:41:20.204 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.969654+0000", 2024-04-20T17:41:20.204 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0031488039999999998 2024-04-20T17:41:20.204 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.204 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.204 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:41:20.204 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.969783+0000", 2024-04-20T17:41:20.204 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000129073 2024-04-20T17:41:20.204 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.205 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.205 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:41:20.205 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.969908+0000", 2024-04-20T17:41:20.205 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012459799999999999 2024-04-20T17:41:20.205 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.205 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.205 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:41:20.205 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.980271+0000", 2024-04-20T17:41:20.205 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010363617 2024-04-20T17:41:20.205 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:41:20.205 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:41:20.205 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:41:20.205 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.206 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.206 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:204570 7.6 7:6e8a7d7d:::benchmark_data_smithi171_161871_object58844:head [delete] snapc 0=[] ondisk+write+known_if_redirected e745)", 2024-04-20T17:41:20.206 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:41:19.902275+0000", 2024-04-20T17:41:20.206 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.079269087000000002, 2024-04-20T17:41:20.206 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.079531828999999998, 2024-04-20T17:41:20.206 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:41:20.206 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:41:20.206 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "waiting for degraded object", 2024-04-20T17:41:20.206 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:41:20.206 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:41:20.206 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:41:20.206 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 204570 2024-04-20T17:41:20.206 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.206 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:41:20.207 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.207 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:41:20.207 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.902275+0000", 2024-04-20T17:41:20.207 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:41:20.207 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.207 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.207 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:41:20.207 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.902275+0000", 2024-04-20T17:41:20.207 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:41:20.207 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.207 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.207 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:41:20.207 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.902278+0000", 2024-04-20T17:41:20.207 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1160000000000002e-06 2024-04-20T17:41:20.207 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.208 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.208 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:41:20.208 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.902299+0000", 2024-04-20T17:41:20.208 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1223e-05 2024-04-20T17:41:20.208 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.208 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.208 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:41:20.208 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.902313+0000", 2024-04-20T17:41:20.208 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3801999999999999e-05 2024-04-20T17:41:20.208 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.208 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.208 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:41:20.208 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.902321+0000", 2024-04-20T17:41:20.208 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8869999999999998e-06 2024-04-20T17:41:20.209 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.209 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.209 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:41:20.209 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.959015+0000", 2024-04-20T17:41:20.209 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.056693856000000001 2024-04-20T17:41:20.209 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.209 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.209 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:41:20.209 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.959077+0000", 2024-04-20T17:41:20.209 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1563999999999995e-05 2024-04-20T17:41:20.209 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:41:20.209 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:41:20.209 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:41:20.210 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.210 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.210 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:204621 7.d 7:b6460a2b:::benchmark_data_smithi171_161871_object58895:head [delete] snapc 0=[] ondisk+write+known_if_redirected e745)", 2024-04-20T17:41:20.210 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:41:19.970004+0000", 2024-04-20T17:41:20.210 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.011540870999999999, 2024-04-20T17:41:20.210 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011843462000000001, 2024-04-20T17:41:20.210 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:41:20.210 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:41:20.210 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:41:20.210 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:41:20.210 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:41:20.210 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:41:20.210 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 204621 2024-04-20T17:41:20.210 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.211 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:41:20.211 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.211 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:41:20.211 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.970004+0000", 2024-04-20T17:41:20.211 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:41:20.211 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.211 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.211 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:41:20.211 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.970004+0000", 2024-04-20T17:41:20.211 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:41:20.211 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.211 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.211 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:41:20.211 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.970006+0000", 2024-04-20T17:41:20.212 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9290000000000001e-06 2024-04-20T17:41:20.212 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.212 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.212 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:41:20.212 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.970011+0000", 2024-04-20T17:41:20.212 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4299999999999997e-06 2024-04-20T17:41:20.212 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.212 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.212 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:41:20.212 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.970021+0000", 2024-04-20T17:41:20.212 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0373999999999999e-05 2024-04-20T17:41:20.212 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.212 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.212 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:41:20.212 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.970029+0000", 2024-04-20T17:41:20.213 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7400000000000004e-06 2024-04-20T17:41:20.213 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.213 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.213 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:41:20.213 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.974805+0000", 2024-04-20T17:41:20.213 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0047761829999999998 2024-04-20T17:41:20.213 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.213 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.213 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:41:20.213 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.974923+0000", 2024-04-20T17:41:20.213 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011773300000000001 2024-04-20T17:41:20.213 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.213 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.213 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:41:20.214 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.975032+0000", 2024-04-20T17:41:20.214 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000108681 2024-04-20T17:41:20.214 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:41:20.214 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:41:20.214 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:41:20.214 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.214 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.214 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:204625 7.9 7:9521b1db:::benchmark_data_smithi171_161871_object58899:head [delete] snapc 0=[] ondisk+write+known_if_redirected e745)", 2024-04-20T17:41:20.214 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:41:19.973810+0000", 2024-04-20T17:41:20.214 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.0077346309999999996, 2024-04-20T17:41:20.214 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0080807280000000006, 2024-04-20T17:41:20.214 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:41:20.214 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:41:20.215 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:41:20.215 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:41:20.215 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:41:20.215 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:41:20.215 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 204625 2024-04-20T17:41:20.215 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.215 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:41:20.215 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.215 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:41:20.215 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.973810+0000", 2024-04-20T17:41:20.215 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:41:20.215 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.215 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.215 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:41:20.216 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.973810+0000", 2024-04-20T17:41:20.216 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:41:20.216 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.216 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.216 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:41:20.216 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.973812+0000", 2024-04-20T17:41:20.216 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.962e-06 2024-04-20T17:41:20.216 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.216 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.216 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:41:20.216 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.973817+0000", 2024-04-20T17:41:20.216 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6130000000000003e-06 2024-04-20T17:41:20.216 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.216 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.216 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:41:20.217 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.973827+0000", 2024-04-20T17:41:20.217 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6739999999999992e-06 2024-04-20T17:41:20.217 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.217 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.217 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:41:20.217 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.973833+0000", 2024-04-20T17:41:20.217 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2750000000000001e-06 2024-04-20T17:41:20.217 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.217 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.217 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:41:20.217 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.973905+0000", 2024-04-20T17:41:20.217 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1588000000000006e-05 2024-04-20T17:41:20.217 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.217 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.217 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:41:20.218 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.974040+0000", 2024-04-20T17:41:20.218 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013487500000000001 2024-04-20T17:41:20.218 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.218 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.218 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 11", 2024-04-20T17:41:20.218 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.974791+0000", 2024-04-20T17:41:20.218 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000751445 2024-04-20T17:41:20.218 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.218 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.218 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:41:20.218 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.981545+0000", 2024-04-20T17:41:20.218 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0067537930000000001 2024-04-20T17:41:20.218 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:41:20.218 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:41:20.219 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:41:20.219 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.219 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.219 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:204627 7.4 7:22affbfe:::benchmark_data_smithi171_161871_object58901:head [delete] snapc 0=[] ondisk+write+known_if_redirected e745)", 2024-04-20T17:41:20.219 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:41:19.978692+0000", 2024-04-20T17:41:20.219 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.0028520540000000001, 2024-04-20T17:41:20.219 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0032458360000000002, 2024-04-20T17:41:20.219 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:41:20.219 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:41:20.219 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:41:20.219 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:41:20.219 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:41:20.219 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:41:20.219 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 204627 2024-04-20T17:41:20.219 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.220 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:41:20.220 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.220 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:41:20.220 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.978692+0000", 2024-04-20T17:41:20.220 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:41:20.220 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.220 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.220 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:41:20.220 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.978692+0000", 2024-04-20T17:41:20.220 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:41:20.220 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.220 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.220 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:41:20.220 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.978695+0000", 2024-04-20T17:41:20.221 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6129999999999998e-06 2024-04-20T17:41:20.221 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.221 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.221 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:41:20.221 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.978703+0000", 2024-04-20T17:41:20.221 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8180000000000004e-06 2024-04-20T17:41:20.221 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.221 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.221 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:41:20.221 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.978716+0000", 2024-04-20T17:41:20.221 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2839e-05 2024-04-20T17:41:20.221 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.221 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.221 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:41:20.222 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.978724+0000", 2024-04-20T17:41:20.222 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7530000000000007e-06 2024-04-20T17:41:20.222 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.222 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.222 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:41:20.222 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.978787+0000", 2024-04-20T17:41:20.222 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2168999999999994e-05 2024-04-20T17:41:20.222 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.222 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.222 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:41:20.222 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.978902+0000", 2024-04-20T17:41:20.222 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000114975 2024-04-20T17:41:20.222 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.222 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.223 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 4", 2024-04-20T17:41:20.223 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.979010+0000", 2024-04-20T17:41:20.223 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000108484 2024-04-20T17:41:20.223 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:41:20.223 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:41:20.223 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:41:20.223 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.223 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.223 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:204580 7.6 7:690df7e8:::benchmark_data_smithi171_161871_object58854:head [delete] snapc 0=[] ondisk+write+known_if_redirected e745)", 2024-04-20T17:41:20.223 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:41:19.917057+0000", 2024-04-20T17:41:20.223 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.064487771999999999, 2024-04-20T17:41:20.223 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.064934299000000001, 2024-04-20T17:41:20.223 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:41:20.224 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:41:20.224 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "waiting for degraded object", 2024-04-20T17:41:20.224 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:41:20.224 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:41:20.224 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:41:20.224 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 204580 2024-04-20T17:41:20.224 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.224 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:41:20.224 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.224 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:41:20.224 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.917057+0000", 2024-04-20T17:41:20.224 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:41:20.224 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.225 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.225 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:41:20.225 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.917057+0000", 2024-04-20T17:41:20.225 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:41:20.225 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.225 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.225 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:41:20.225 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.917058+0000", 2024-04-20T17:41:20.225 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.64e-06 2024-04-20T17:41:20.225 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.225 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.225 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:41:20.225 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.917064+0000", 2024-04-20T17:41:20.225 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5969999999999999e-06 2024-04-20T17:41:20.226 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.226 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.226 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:41:20.226 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.917073+0000", 2024-04-20T17:41:20.226 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0380000000000005e-06 2024-04-20T17:41:20.226 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.226 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.226 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:41:20.226 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.917079+0000", 2024-04-20T17:41:20.226 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.251e-06 2024-04-20T17:41:20.226 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.226 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.226 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:41:20.227 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.959107+0000", 2024-04-20T17:41:20.227 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.042027900999999999 2024-04-20T17:41:20.227 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.227 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.227 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:41:20.227 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.959888+0000", 2024-04-20T17:41:20.227 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00078056499999999999 2024-04-20T17:41:20.227 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:41:20.227 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:41:20.227 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:41:20.227 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.227 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.227 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:204590 7.6 7:67d5ac38:::benchmark_data_smithi171_161871_object58864:head [delete] snapc 0=[] ondisk+write+known_if_redirected e745)", 2024-04-20T17:41:20.227 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:41:19.928058+0000", 2024-04-20T17:41:20.228 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.053486578, 2024-04-20T17:41:20.228 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.053974822999999998, 2024-04-20T17:41:20.228 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:41:20.228 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:41:20.228 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "waiting for degraded object", 2024-04-20T17:41:20.228 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:41:20.228 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:41:20.228 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:41:20.228 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 204590 2024-04-20T17:41:20.228 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.228 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:41:20.228 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.228 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:41:20.228 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.928058+0000", 2024-04-20T17:41:20.229 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:41:20.229 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.229 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.229 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:41:20.229 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.928058+0000", 2024-04-20T17:41:20.229 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:41:20.229 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.229 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.229 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:41:20.229 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.928060+0000", 2024-04-20T17:41:20.229 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.778e-06 2024-04-20T17:41:20.229 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.229 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.229 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:41:20.230 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.928067+0000", 2024-04-20T17:41:20.230 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.233e-06 2024-04-20T17:41:20.230 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.230 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.230 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:41:20.230 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.928079+0000", 2024-04-20T17:41:20.230 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2285e-05 2024-04-20T17:41:20.230 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.230 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.230 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:41:20.230 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.928090+0000", 2024-04-20T17:41:20.230 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0321e-05 2024-04-20T17:41:20.230 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.231 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.231 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:41:20.231 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.959954+0000", 2024-04-20T17:41:20.231 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.031864838999999999 2024-04-20T17:41:20.231 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.231 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.231 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:41:20.231 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.960332+0000", 2024-04-20T17:41:20.231 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000377413 2024-04-20T17:41:20.231 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:41:20.231 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:41:20.231 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:41:20.231 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.231 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.231 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDPGPushReply(7.6 745/727 [PushReplyOp(7:6e8a7d7d:::benchmark_data_smithi171_161871_object58844:head)])", 2024-04-20T17:41:20.232 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:41:19.928681+0000", 2024-04-20T17:41:20.232 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.052863779, 2024-04-20T17:41:20.232 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.053391234000000003, 2024-04-20T17:41:20.232 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:41:20.232 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:41:20.232 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:41:20.232 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:41:20.232 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.232 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:41:20.232 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.928681+0000", 2024-04-20T17:41:20.232 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:41:20.232 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.232 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.232 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:41:20.232 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.928681+0000", 2024-04-20T17:41:20.233 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:41:20.233 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.233 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.233 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:41:20.233 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.928682+0000", 2024-04-20T17:41:20.233 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.767e-06 2024-04-20T17:41:20.233 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.233 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.233 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:41:20.233 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.928690+0000", 2024-04-20T17:41:20.233 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0539999999999997e-06 2024-04-20T17:41:20.233 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.233 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.233 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:41:20.233 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.928700+0000", 2024-04-20T17:41:20.234 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0159e-05 2024-04-20T17:41:20.234 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.234 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.234 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:41:20.234 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.928708+0000", 2024-04-20T17:41:20.234 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4400000000000005e-06 2024-04-20T17:41:20.234 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:41:20.234 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:41:20.234 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:41:20.234 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.234 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.234 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:204601 7.6 7:6ca75a85:::benchmark_data_smithi171_161871_object58875:head [delete] snapc 0=[] ondisk+write+known_if_redirected e745)", 2024-04-20T17:41:20.234 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:41:19.940453+0000", 2024-04-20T17:41:20.234 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.041091759999999998, 2024-04-20T17:41:20.235 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.041649565, 2024-04-20T17:41:20.235 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:41:20.235 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:41:20.235 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "waiting for degraded object", 2024-04-20T17:41:20.235 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:41:20.235 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:41:20.235 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:41:20.235 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 204601 2024-04-20T17:41:20.235 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.235 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:41:20.235 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.235 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:41:20.235 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.940453+0000", 2024-04-20T17:41:20.235 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:41:20.235 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.236 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.236 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:41:20.236 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.940453+0000", 2024-04-20T17:41:20.236 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:41:20.236 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.236 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.236 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:41:20.236 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.940456+0000", 2024-04-20T17:41:20.236 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5590000000000001e-06 2024-04-20T17:41:20.236 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.236 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.236 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:41:20.236 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.940466+0000", 2024-04-20T17:41:20.236 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2439999999999999e-06 2024-04-20T17:41:20.237 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.237 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.237 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:41:20.237 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.940484+0000", 2024-04-20T17:41:20.237 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8754e-05 2024-04-20T17:41:20.237 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.237 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.237 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:41:20.237 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.940492+0000", 2024-04-20T17:41:20.237 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7149999999999998e-06 2024-04-20T17:41:20.237 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.237 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.237 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:41:20.237 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.963512+0000", 2024-04-20T17:41:20.237 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023019891000000001 2024-04-20T17:41:20.238 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:41:20.238 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:41:20.238 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:41:20.238 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:19.963919+0000", 2024-04-20T17:41:20.238 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00040726 2024-04-20T17:41:20.238 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:41:20.238 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:41:20.238 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:41:20.238 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:41:20.238 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:41:20.238 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 11 2024-04-20T17:41:20.238 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:41:20.422 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:41:20.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:20.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:41:20.532 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:41:20.586 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:21.589 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:41:21.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:21.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:41:21.695 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:41:21.726 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:21.761 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:41:21.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:21.764 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:41:21.865 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:41:21.889 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:21 smithi106 ceph-mon[44423]: pgmap v1992: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 5.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 817 op/s; 5565/178508 objects degraded (3.118%); 3.3 MiB/s, 52 objects/s recovering 2024-04-20T17:41:21.890 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:21 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 5565/178508 objects degraded (3.118%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:21.891 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:21.910 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:41:21.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:21.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:41:21.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:21 smithi005 ceph-mon[39562]: pgmap v1992: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 5.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 817 op/s; 5565/178508 objects degraded (3.118%); 3.3 MiB/s, 52 objects/s recovering 2024-04-20T17:41:21.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:21 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 5565/178508 objects degraded (3.118%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:22.014 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:41:22.046 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:21 smithi171 ceph-mon[44189]: pgmap v1992: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 5.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 817 op/s; 5565/178508 objects degraded (3.118%); 3.3 MiB/s, 52 objects/s recovering 2024-04-20T17:41:22.046 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:21 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 5565/178508 objects degraded (3.118%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:22.048 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:22.074 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:41:22.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:22.078 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:41:22.179 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:41:22.205 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:22.224 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:41:22.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:22.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:41:22.328 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:41:22.365 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:22.655 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:41:22.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:22.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:41:22.723 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-20T17:41:22.765 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:41:22.866 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:22.945 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:41:22.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:22.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:41:23.062 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:41:23.126 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:23.175 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:41:23.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:23.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:41:23.285 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:41:23.286 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:41:23.295 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:41:23.403 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:23.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:23 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:41:23.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:23 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:41:23.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:23 smithi005 ceph-mon[39562]: pgmap v1993: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 5.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 830 op/s; 5293/173336 objects degraded (3.054%); 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:41:23.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:23 smithi005 ceph-mon[39562]: osdmap e746: 12 total, 11 up, 9 in 2024-04-20T17:41:23.691 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:41:23.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:23.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:41:23.738 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:41:23.799 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:41:23.870 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:23.970 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:41:23.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:23.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:41:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:23 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:41:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:23 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:41:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:23 smithi106 ceph-mon[44423]: pgmap v1993: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 5.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 830 op/s; 5293/173336 objects degraded (3.054%); 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:41:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:23 smithi106 ceph-mon[44423]: osdmap e746: 12 total, 11 up, 9 in 2024-04-20T17:41:24.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:23 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:41:24.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:23 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:41:24.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:23 smithi171 ceph-mon[44189]: pgmap v1993: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 5.3 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 830 op/s; 5293/173336 objects degraded (3.054%); 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:41:24.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:23 smithi171 ceph-mon[44189]: osdmap e746: 12 total, 11 up, 9 in 2024-04-20T17:41:24.084 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:41:24.119 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:24.148 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:41:24.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:24.151 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:41:24.252 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:41:24.279 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:24.300 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:41:24.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:24.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:41:24.404 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:41:24.432 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:24.454 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:41:24.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:24.460 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:41:24.561 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:41:24.590 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:24.620 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:41:24.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:24.624 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:41:24.671 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:41:24.724 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:41:24.752 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:24 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2227603294' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:41:24.752 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:24 smithi171 ceph-mon[44189]: pgmap v1995: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 5.2 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 849 op/s; 5293/170186 objects degraded (3.110%); 3.5 MiB/s, 55 objects/s recovering 2024-04-20T17:41:24.754 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:24.780 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:41:24.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:24.785 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:41:24.886 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:41:24.945 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:24 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2227603294' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:41:24.945 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:24 smithi005 ceph-mon[39562]: pgmap v1995: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 5.2 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 849 op/s; 5293/170186 objects degraded (3.110%); 3.5 MiB/s, 55 objects/s recovering 2024-04-20T17:41:24.954 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:25.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:24 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2227603294' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:41:25.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:24 smithi106 ceph-mon[44423]: pgmap v1995: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 5.2 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 849 op/s; 5293/170186 objects degraded (3.110%); 3.5 MiB/s, 55 objects/s recovering 2024-04-20T17:41:25.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:25.045 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:41:25.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:41:25.146 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:41:25.172 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:25.191 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:41:25.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:25.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:41:25.295 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:41:25.330 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:25.355 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:41:25.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:25.363 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:41:25.465 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:41:25.491 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:25.510 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:41:25.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:25.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:41:25.615 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:41:25.656 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:25.683 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:41:25.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:25.689 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:41:25.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:25 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:41:25.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:25 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2227603294' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:41:25.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:25 smithi106 ceph-mon[44423]: osdmap e747: 12 total, 11 up, 9 in 2024-04-20T17:41:25.791 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:41:25.828 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:25.853 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:41:25.855 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:25 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:41:25.855 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:25 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2227603294' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:41:25.855 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:25 smithi005 ceph-mon[39562]: osdmap e747: 12 total, 11 up, 9 in 2024-04-20T17:41:25.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:25.857 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:41:25.958 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:41:25.985 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:25 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:41:25.985 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:25 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2227603294' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:41:25.985 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:25 smithi171 ceph-mon[44189]: osdmap e747: 12 total, 11 up, 9 in 2024-04-20T17:41:25.987 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:26.011 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:41:26.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:26.014 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:41:26.115 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:41:26.197 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:26.768 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:41:26.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:26.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:41:26.785 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:26 smithi005 ceph-mon[39562]: pgmap v1997: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 5.0 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 5036/164740 objects degraded (3.057%); 4.1 MiB/s, 66 objects/s recovering 2024-04-20T17:41:26.785 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:26 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 5293/170186 objects degraded (3.110%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:26.880 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:41:26.941 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:27.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:26 smithi106 ceph-mon[44423]: pgmap v1997: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 5.0 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 5036/164740 objects degraded (3.057%); 4.1 MiB/s, 66 objects/s recovering 2024-04-20T17:41:27.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:26 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 5293/170186 objects degraded (3.110%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:26 smithi171 ceph-mon[44189]: pgmap v1997: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 5.0 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 1.06k op/s; 5036/164740 objects degraded (3.057%); 4.1 MiB/s, 66 objects/s recovering 2024-04-20T17:41:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:26 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 5293/170186 objects degraded (3.110%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:27.172 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:41:27.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:27.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:41:27.281 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:41:27.309 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:27.328 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:41:27.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:27.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:41:27.432 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:41:27.482 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:27.796 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:41:27.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:27.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:41:27.907 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:41:27.940 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:27.972 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:41:27.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:27.976 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:41:28.076 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:41:28.114 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:28.140 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:41:28.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:28.145 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:41:28.246 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:41:28.295 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:29.830 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:41:29.887 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:29 smithi005 ceph-mon[39562]: pgmap v1998: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 5.0 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 860 op/s; 5036/164740 objects degraded (3.057%); 4.1 MiB/s, 66 objects/s recovering 2024-04-20T17:41:30.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:29 smithi106 ceph-mon[44423]: pgmap v1998: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 5.0 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 860 op/s; 5036/164740 objects degraded (3.057%); 4.1 MiB/s, 66 objects/s recovering 2024-04-20T17:41:30.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:29 smithi171 ceph-mon[44189]: pgmap v1998: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 5.0 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 860 op/s; 5036/164740 objects degraded (3.057%); 4.1 MiB/s, 66 objects/s recovering 2024-04-20T17:41:30.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:30 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:41:30.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:30 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:41:30.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:30 smithi005 ceph-mon[39562]: osdmap e748: 12 total, 11 up, 9 in 2024-04-20T17:41:31.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:30 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:41:31.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:30 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:41:31.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:30 smithi106 ceph-mon[44423]: osdmap e748: 12 total, 11 up, 9 in 2024-04-20T17:41:31.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:30 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:41:31.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:30 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:41:31.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:30 smithi171 ceph-mon[44189]: osdmap e748: 12 total, 11 up, 9 in 2024-04-20T17:41:31.314 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:41:31.315 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:41:31.315 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:41:31.315 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:41:31.315 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:41:31.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:31 smithi005 ceph-mon[39562]: pgmap v2000: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 4.9 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 759 op/s; 5036/161330 objects degraded (3.122%); 2.0 MiB/s, 32 objects/s recovering 2024-04-20T17:41:31.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:31 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 5036/161330 objects degraded (3.122%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:31 smithi106 ceph-mon[44423]: pgmap v2000: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 4.9 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 759 op/s; 5036/161330 objects degraded (3.122%); 2.0 MiB/s, 32 objects/s recovering 2024-04-20T17:41:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:31 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 5036/161330 objects degraded (3.122%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:32.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:31 smithi171 ceph-mon[44189]: pgmap v2000: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 4.9 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 759 op/s; 5036/161330 objects degraded (3.122%); 2.0 MiB/s, 32 objects/s recovering 2024-04-20T17:41:32.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:31 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 5036/161330 objects degraded (3.122%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:32.643 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:32 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:41:32.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:32 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:41:32.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:32 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/426672458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:41:32.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:32 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:32.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:32 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:33.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:32 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:41:33.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:32 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:41:33.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:32 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/426672458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:41:33.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:32 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:33.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:32 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:33.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:32 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:41:33.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:32 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:41:33.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:32 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/426672458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:41:33.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:32 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:33.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:32 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:33.370 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:41:33.370 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:41:33.370 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:41:33.370 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:41:33.370 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:41:33.370 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:41:33.370 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:41:33.370 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:41:33.371 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:41:33.371 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:41:33.371 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:41:33.371 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:41:33.371 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:41:33.371 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:41:33.371 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:41:33.371 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:41:33.371 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:41:33.371 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:41:33.371 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:41:33.371 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:41:33.372 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:41:33.372 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:41:33.372 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:41:33.791 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:41:33.798 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:41:33.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:33.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:33.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:33.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:33.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:33.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:33.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:33.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:33.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:33 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:33.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:33 smithi005 ceph-mon[39562]: pgmap v2001: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 4.8 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 890 op/s; 4758/155940 objects degraded (3.051%); 4.2 MiB/s, 66 objects/s recovering 2024-04-20T17:41:33.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:33 smithi005 ceph-mon[39562]: osdmap e749: 12 total, 11 up, 9 in 2024-04-20T17:41:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:33 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:34.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:33 smithi106 ceph-mon[44423]: pgmap v2001: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 4.8 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 890 op/s; 4758/155940 objects degraded (3.051%); 4.2 MiB/s, 66 objects/s recovering 2024-04-20T17:41:34.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:33 smithi106 ceph-mon[44423]: osdmap e749: 12 total, 11 up, 9 in 2024-04-20T17:41:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:34.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:33 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:34.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:33 smithi171 ceph-mon[44189]: pgmap v2001: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 4.8 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 890 op/s; 4758/155940 objects degraded (3.051%); 4.2 MiB/s, 66 objects/s recovering 2024-04-20T17:41:34.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:33 smithi171 ceph-mon[44189]: osdmap e749: 12 total, 11 up, 9 in 2024-04-20T17:41:34.251 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:41:34.251 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:41:34.642 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:41:34.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:34.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:41:34.668 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-20T17:41:34.757 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:41:34.806 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:34.924 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:41:34.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:34.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:41:34.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:34 smithi005 ceph-mon[39562]: from='client.17259 v1:172.21.15.5:0/4050926962' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:41:34.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:34 smithi005 ceph-mon[39562]: pgmap v2003: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 4.7 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 760 op/s; 4758/152560 objects degraded (3.119%); 2.2 MiB/s, 34 objects/s recovering 2024-04-20T17:41:34.971 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:41:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:34 smithi106 ceph-mon[44423]: from='client.17259 v1:172.21.15.5:0/4050926962' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:41:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:34 smithi106 ceph-mon[44423]: pgmap v2003: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 4.7 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 760 op/s; 4758/152560 objects degraded (3.119%); 2.2 MiB/s, 34 objects/s recovering 2024-04-20T17:41:35.030 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:41:35.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:34 smithi171 ceph-mon[44189]: from='client.17259 v1:172.21.15.5:0/4050926962' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:41:35.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:34 smithi171 ceph-mon[44189]: pgmap v2003: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 4.7 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 760 op/s; 4758/152560 objects degraded (3.119%); 2.2 MiB/s, 34 objects/s recovering 2024-04-20T17:41:35.111 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:35.909 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":2002,"stamp":"2024-04-20T17:41:32.642547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5110235704,"num_objects":77970,"num_object_clones":0,"num_object_copies":155940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4758,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77970,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":213661,"num_write_kb":9329211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13309,"num_bytes_recovered":874207904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161054,"ondisk_log_size":161054,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":14289520,"kb_used_data":9744796,"kb_used_omap":217,"kb_used_meta":4544166,"kb_avail":829416848,"statfs":{"total":863955320832,"available":849322852352,"internally_reserved":0,"allocated":9978671104,"data_stored":9975300656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":222986,"internal_metadata":4653226230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,11,9,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-466812928,"num_objects":-7123,"num_object_clones":0,"num_object_copies":-14246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7123,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":535,"num_bytes_recovered":35061760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.997647"},"pg_stats":[{"pgid":"7.f","version":"748'13486","reported_seq":13703,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-04-20T17:41:30.776946+0000","last_change":"2024-04-20T17:39:40.325470+0000","last_active":"2024-04-20T17:41:30.776946+0000","last_peered":"2024-04-20T17:41:30.776946+0000","last_clean":"2024-04-20T17:41:30.776946+0000","last_became_active":"2024-04-20T17:37:56.835750+0000","last_became_peered":"2024-04-20T17:37:56.835750+0000","last_unstale":"2024-04-20T17:41:30.776946+0000","last_undegraded":"2024-04-20T17:41:30.776946+0000","last_fullsized":"2024-04-20T17:41:30.776946+0000","mapping_epoch":694,"log_start":"702'3400","ondisk_log_start":"702'3400","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9175","last_scrub_stamp":"2024-04-20T17:39:40.325340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:40.325340+0000","objects_scrubbed":9175,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":741,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318767104,"num_objects":4864,"num_object_clones":0,"num_object_copies":9728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13486,"num_write_kb":587200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"748'13206","reported_seq":13474,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-04-20T17:41:31.119128+0000","last_change":"2024-04-20T17:41:02.433089+0000","last_active":"2024-04-20T17:41:31.119128+0000","last_peered":"2024-04-20T17:41:31.119128+0000","last_clean":"2024-04-20T17:41:31.119128+0000","last_became_active":"2024-04-20T17:37:56.833703+0000","last_became_peered":"2024-04-20T17:37:56.833703+0000","last_unstale":"2024-04-20T17:41:31.119128+0000","last_undegraded":"2024-04-20T17:41:31.119128+0000","last_fullsized":"2024-04-20T17:41:31.119128+0000","mapping_epoch":694,"log_start":"702'3200","ondisk_log_start":"702'3200","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"742'11794","last_scrub_stamp":"2024-04-20T17:41:02.432782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:41:02.432782+0000","objects_scrubbed":6149,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:29.683106+0000","scrub_duration":1488,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308936704,"num_objects":4714,"num_object_clones":0,"num_object_copies":9428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13206,"num_write_kb":573440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"748'15616","reported_seq":15937,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-04-20T17:41:30.790858+0000","last_change":"2024-04-20T17:39:48.311967+0000","last_active":"2024-04-20T17:41:30.790858+0000","last_peered":"2024-04-20T17:41:30.790858+0000","last_clean":"2024-04-20T17:41:30.790858+0000","last_became_active":"2024-04-20T17:37:56.836496+0000","last_became_peered":"2024-04-20T17:37:56.836496+0000","last_unstale":"2024-04-20T17:41:30.790858+0000","last_undegraded":"2024-04-20T17:41:30.790858+0000","last_fullsized":"2024-04-20T17:41:30.790858+0000","mapping_epoch":694,"log_start":"709'5600","ondisk_log_start":"709'5600","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'11204","last_scrub_stamp":"2024-04-20T17:39:48.311825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:48.311825+0000","objects_scrubbed":9436,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329318424,"num_objects":5026,"num_object_clones":0,"num_object_copies":10052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5026,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13848,"num_write_kb":603905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.c","version":"747'13233","reported_seq":13447,"reported_epoch":747,"state":"active+clean","last_fresh":"2024-04-20T17:41:29.270568+0000","last_change":"2024-04-20T17:39:44.683378+0000","last_active":"2024-04-20T17:41:29.270568+0000","last_peered":"2024-04-20T17:41:29.270568+0000","last_clean":"2024-04-20T17:41:29.270568+0000","last_became_active":"2024-04-20T17:37:56.834181+0000","last_became_peered":"2024-04-20T17:37:56.834181+0000","last_unstale":"2024-04-20T17:41:29.270568+0000","last_undegraded":"2024-04-20T17:41:29.270568+0000","last_fullsized":"2024-04-20T17:41:29.270568+0000","mapping_epoch":694,"log_start":"702'3200","ondisk_log_start":"702'3200","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9063","last_scrub_stamp":"2024-04-20T17:39:44.683201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:44.683201+0000","objects_scrubbed":9063,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":746,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320667648,"num_objects":4893,"num_object_clones":0,"num_object_copies":9786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13233,"num_write_kb":580032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"747'13336","reported_seq":13626,"reported_epoch":747,"state":"active+clean","last_fresh":"2024-04-20T17:41:29.077333+0000","last_change":"2024-04-20T17:40:58.937747+0000","last_active":"2024-04-20T17:41:29.077333+0000","last_peered":"2024-04-20T17:41:29.077333+0000","last_clean":"2024-04-20T17:41:29.077333+0000","last_became_active":"2024-04-20T17:37:56.830487+0000","last_became_peered":"2024-04-20T17:37:56.830487+0000","last_unstale":"2024-04-20T17:41:29.077333+0000","last_undegraded":"2024-04-20T17:41:29.077333+0000","last_fullsized":"2024-04-20T17:41:29.077333+0000","mapping_epoch":694,"log_start":"702'3300","ondisk_log_start":"702'3300","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"742'11757","last_scrub_stamp":"2024-04-20T17:40:58.937604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:40:58.937604+0000","objects_scrubbed":6664,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:28.789538+0000","scrub_duration":858,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332398592,"num_objects":5072,"num_object_clones":0,"num_object_copies":10144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13336,"num_write_kb":589056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.a","version":"748'13181","reported_seq":13465,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-04-20T17:41:30.856718+0000","last_change":"2024-04-20T17:41:06.536176+0000","last_active":"2024-04-20T17:41:30.856718+0000","last_peered":"2024-04-20T17:41:30.856718+0000","last_clean":"2024-04-20T17:41:30.856718+0000","last_became_active":"2024-04-20T17:37:56.835944+0000","last_became_peered":"2024-04-20T17:37:56.835944+0000","last_unstale":"2024-04-20T17:41:30.856718+0000","last_undegraded":"2024-04-20T17:41:30.856718+0000","last_fullsized":"2024-04-20T17:41:30.856718+0000","mapping_epoch":694,"log_start":"702'3100","ondisk_log_start":"702'3100","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"742'11924","last_scrub_stamp":"2024-04-20T17:41:06.536039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:41:06.536039+0000","objects_scrubbed":6077,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:18.392067+0000","scrub_duration":857,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314769408,"num_objects":4803,"num_object_clones":0,"num_object_copies":9606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13181,"num_write_kb":575488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"748'13394","reported_seq":13593,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-04-20T17:41:30.771878+0000","last_change":"2024-04-20T17:39:03.675435+0000","last_active":"2024-04-20T17:41:30.771878+0000","last_peered":"2024-04-20T17:41:30.771878+0000","last_clean":"2024-04-20T17:41:30.771878+0000","last_became_active":"2024-04-20T17:37:56.835700+0000","last_became_peered":"2024-04-20T17:37:56.835700+0000","last_unstale":"2024-04-20T17:41:30.771878+0000","last_undegraded":"2024-04-20T17:41:30.771878+0000","last_fullsized":"2024-04-20T17:41:30.771878+0000","mapping_epoch":694,"log_start":"702'3300","ondisk_log_start":"702'3300","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'7175","last_scrub_stamp":"2024-04-20T17:39:03.675211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:03.675211+0000","objects_scrubbed":7133,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310378496,"num_objects":4736,"num_object_clones":0,"num_object_copies":9472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13394,"num_write_kb":580160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11],"acting":[0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"748'16866","reported_seq":35446,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-04-20T17:41:31.094806+0000","last_change":"2024-04-20T17:40:51.537130+0000","last_active":"2024-04-20T17:41:31.094806+0000","last_peered":"2024-04-20T17:41:31.094806+0000","last_clean":"2024-04-20T17:41:31.094806+0000","last_became_active":"2024-04-20T17:40:09.546695+0000","last_became_peered":"2024-04-20T17:40:09.546695+0000","last_unstale":"2024-04-20T17:41:31.094806+0000","last_undegraded":"2024-04-20T17:41:31.094806+0000","last_fullsized":"2024-04-20T17:41:31.094806+0000","mapping_epoch":731,"log_start":"713'6801","ondisk_log_start":"713'6801","created":694,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'14850","last_scrub_stamp":"2024-04-20T17:40:51.536812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:40:51.536812+0000","objects_scrubbed":6890,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:41:57.907978+0000","scrub_duration":1411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317194240,"num_objects":4840,"num_object_clones":0,"num_object_copies":9680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13296,"num_write_kb":580352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9062,"num_bytes_recovered":593887232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"747'13392","reported_seq":13669,"reported_epoch":747,"state":"active+clean","last_fresh":"2024-04-20T17:41:28.719074+0000","last_change":"2024-04-20T17:40:48.024415+0000","last_active":"2024-04-20T17:41:28.719074+0000","last_peered":"2024-04-20T17:41:28.719074+0000","last_clean":"2024-04-20T17:41:28.719074+0000","last_became_active":"2024-04-20T17:37:56.836665+0000","last_became_peered":"2024-04-20T17:37:56.836665+0000","last_unstale":"2024-04-20T17:41:28.719074+0000","last_undegraded":"2024-04-20T17:41:28.719074+0000","last_fullsized":"2024-04-20T17:41:28.719074+0000","mapping_epoch":694,"log_start":"702'3300","ondisk_log_start":"702'3300","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'11243","last_scrub_stamp":"2024-04-20T17:40:48.024149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:40:48.024149+0000","objects_scrubbed":7122,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:03.517694+0000","scrub_duration":1322,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323485696,"num_objects":4936,"num_object_clones":0,"num_object_copies":9872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13392,"num_write_kb":586496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.5","version":"747'13354","reported_seq":13627,"reported_epoch":747,"state":"active+clean","last_fresh":"2024-04-20T17:41:29.052580+0000","last_change":"2024-04-20T17:40:54.143175+0000","last_active":"2024-04-20T17:41:29.052580+0000","last_peered":"2024-04-20T17:41:29.052580+0000","last_clean":"2024-04-20T17:41:29.052580+0000","last_became_active":"2024-04-20T17:37:56.833600+0000","last_became_peered":"2024-04-20T17:37:56.833600+0000","last_unstale":"2024-04-20T17:41:29.052580+0000","last_undegraded":"2024-04-20T17:41:29.052580+0000","last_fullsized":"2024-04-20T17:41:29.052580+0000","mapping_epoch":694,"log_start":"702'3300","ondisk_log_start":"702'3300","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'11512","last_scrub_stamp":"2024-04-20T17:40:54.143076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:40:54.143076+0000","objects_scrubbed":6822,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:02.493494+0000","scrub_duration":950,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325058560,"num_objects":4960,"num_object_clones":0,"num_object_copies":9920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13354,"num_write_kb":586048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.4","version":"748'13220","reported_seq":13436,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-04-20T17:41:30.776946+0000","last_change":"2024-04-20T17:39:50.365478+0000","last_active":"2024-04-20T17:41:30.776946+0000","last_peered":"2024-04-20T17:41:30.776946+0000","last_clean":"2024-04-20T17:41:30.776946+0000","last_became_active":"2024-04-20T17:37:56.834727+0000","last_became_peered":"2024-04-20T17:37:56.834727+0000","last_unstale":"2024-04-20T17:41:30.776946+0000","last_undegraded":"2024-04-20T17:41:30.776946+0000","last_fullsized":"2024-04-20T17:41:30.776946+0000","mapping_epoch":694,"log_start":"702'3200","ondisk_log_start":"702'3200","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9031","last_scrub_stamp":"2024-04-20T17:39:50.365339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:50.365339+0000","objects_scrubbed":9031,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":876,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317325312,"num_objects":4842,"num_object_clones":0,"num_object_copies":9684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13220,"num_write_kb":577984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"747'13148","reported_seq":13341,"reported_epoch":747,"state":"active+clean","last_fresh":"2024-04-20T17:41:29.265712+0000","last_change":"2024-04-20T17:39:00.502710+0000","last_active":"2024-04-20T17:41:29.265712+0000","last_peered":"2024-04-20T17:41:29.265712+0000","last_clean":"2024-04-20T17:41:29.265712+0000","last_became_active":"2024-04-20T17:37:56.832911+0000","last_became_peered":"2024-04-20T17:37:56.832911+0000","last_unstale":"2024-04-20T17:41:29.265712+0000","last_undegraded":"2024-04-20T17:41:29.265712+0000","last_fullsized":"2024-04-20T17:41:29.265712+0000","mapping_epoch":694,"log_start":"702'3100","ondisk_log_start":"702'3100","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'6831","last_scrub_stamp":"2024-04-20T17:39:00.502556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:00.502556+0000","objects_scrubbed":6782,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322699264,"num_objects":4924,"num_object_clones":0,"num_object_copies":9848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13148,"num_write_kb":578304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"747'13486","reported_seq":13685,"reported_epoch":747,"state":"active+clean","last_fresh":"2024-04-20T17:41:29.282128+0000","last_change":"2024-04-20T17:39:08.489529+0000","last_active":"2024-04-20T17:41:29.282128+0000","last_peered":"2024-04-20T17:41:29.282128+0000","last_clean":"2024-04-20T17:41:29.282128+0000","last_became_active":"2024-04-20T17:37:56.834292+0000","last_became_peered":"2024-04-20T17:37:56.834292+0000","last_unstale":"2024-04-20T17:41:29.282128+0000","last_undegraded":"2024-04-20T17:41:29.282128+0000","last_fullsized":"2024-04-20T17:41:29.282128+0000","mapping_epoch":694,"log_start":"702'3400","ondisk_log_start":"702'3400","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"716'7577","last_scrub_stamp":"2024-04-20T17:39:08.489308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:08.489308+0000","objects_scrubbed":7524,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328073216,"num_objects":5006,"num_object_clones":0,"num_object_copies":10012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13486,"num_write_kb":591744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"748'13376","reported_seq":13660,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-04-20T17:41:30.656119+0000","last_change":"2024-04-20T17:41:07.795076+0000","last_active":"2024-04-20T17:41:30.656119+0000","last_peered":"2024-04-20T17:41:30.656119+0000","last_clean":"2024-04-20T17:41:30.656119+0000","last_became_active":"2024-04-20T17:37:56.835512+0000","last_became_peered":"2024-04-20T17:37:56.835512+0000","last_unstale":"2024-04-20T17:41:30.656119+0000","last_undegraded":"2024-04-20T17:41:30.656119+0000","last_fullsized":"2024-04-20T17:41:30.656119+0000","mapping_epoch":694,"log_start":"702'3300","ondisk_log_start":"702'3300","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"743'12166","last_scrub_stamp":"2024-04-20T17:41:07.794973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:41:07.794973+0000","objects_scrubbed":6033,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:12.046222+0000","scrub_duration":644,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314966016,"num_objects":4806,"num_object_clones":0,"num_object_copies":9612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13376,"num_write_kb":581824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"748'13285","reported_seq":13565,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-04-20T17:41:30.911487+0000","last_change":"2024-04-20T17:41:16.382261+0000","last_active":"2024-04-20T17:41:30.911487+0000","last_peered":"2024-04-20T17:41:30.911487+0000","last_clean":"2024-04-20T17:41:30.911487+0000","last_became_active":"2024-04-20T17:37:56.833972+0000","last_became_peered":"2024-04-20T17:37:56.833972+0000","last_unstale":"2024-04-20T17:41:30.911487+0000","last_undegraded":"2024-04-20T17:41:30.911487+0000","last_fullsized":"2024-04-20T17:41:30.911487+0000","mapping_epoch":694,"log_start":"702'3200","ondisk_log_start":"702'3200","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"745'12483","last_scrub_stamp":"2024-04-20T17:41:16.382018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:41:16.382018+0000","objects_scrubbed":5596,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:31.605235+0000","scrub_duration":653,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313196544,"num_objects":4779,"num_object_clones":0,"num_object_copies":9558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13285,"num_write_kb":578048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.6","version":"748'13239","reported_seq":21914,"reported_epoch":748,"state":"active+recovering+degraded","last_fresh":"2024-04-20T17:41:30.779543+0000","last_change":"2024-04-20T17:39:54.962650+0000","last_active":"2024-04-20T17:41:30.779543+0000","last_peered":"2024-04-20T17:41:30.779543+0000","last_clean":"2024-04-20T17:39:34.507147+0000","last_became_active":"2024-04-20T17:39:54.933393+0000","last_became_peered":"2024-04-20T17:39:54.933393+0000","last_unstale":"2024-04-20T17:41:30.779543+0000","last_undegraded":"2024-04-20T17:39:54.578981+0000","last_fullsized":"2024-04-20T17:41:30.779543+0000","mapping_epoch":727,"log_start":"702'3200","ondisk_log_start":"702'3200","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'7141","last_scrub_stamp":"2024-04-20T17:39:04.435913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:04.435913+0000","objects_scrubbed":7101,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312410112,"num_objects":4767,"num_object_clones":0,"num_object_copies":9534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4758,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13239,"num_write_kb":576192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4237,"num_bytes_recovered":277676032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":4758},{"shards":"0,3","objects":9}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":505,"reported_epoch":747,"state":"active+clean","last_fresh":"2024-04-20T17:41:24.647617+0000","last_change":"2024-04-20T17:40:50.375929+0000","last_active":"2024-04-20T17:41:24.647617+0000","last_peered":"2024-04-20T17:41:24.647617+0000","last_clean":"2024-04-20T17:41:24.647617+0000","last_became_active":"2024-04-20T17:39:54.970208+0000","last_became_peered":"2024-04-20T17:39:54.970208+0000","last_unstale":"2024-04-20T17:41:24.647617+0000","last_undegraded":"2024-04-20T17:41:24.647617+0000","last_fullsized":"2024-04-20T17:41:24.647617+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:40:50.375828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:40:50.375828+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:14.500087+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":5109645336,"num_objects":77968,"num_object_clones":0,"num_object_copies":155936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4758,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77968,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":213480,"num_write_kb":9326273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13299,"num_bytes_recovered":871563264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11082407936,"data_stored":11082399792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160917,"ondisk_log_size":160917,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991662,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1969176,"kb_used_data":1279952,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":91775976,"statfs":{"total":95995035648,"available":93978599424,"internally_reserved":0,"allocated":1310670848,"data_stored":1310297816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":705755588},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253301,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1472336,"kb_used_data":939976,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":92272816,"statfs":{"total":95995035648,"available":94487363584,"internally_reserved":0,"allocated":962535424,"data_stored":962165944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":545126848},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613050,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1483560,"kb_used_data":928988,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92261592,"statfs":{"total":95995035648,"available":94475870208,"internally_reserved":0,"allocated":951283712,"data_stored":950904696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":567867840},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809280,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":946000,"kb_used_data":635784,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":92799152,"statfs":{"total":95995035648,"available":95026331648,"internally_reserved":0,"allocated":651042816,"data_stored":650673356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[0,1,2,3,4,5,6,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":2297807503511,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":2830383448133,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738855,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2201184,"kb_used_data":1552152,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":91543968,"statfs":{"total":95995035648,"available":93741023232,"internally_reserved":0,"allocated":1589403648,"data_stored":1589024112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,6,1,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394145,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1548448,"kb_used_data":1241620,"kb_used_omap":202,"kb_used_meta":306421,"kb_avail":92196704,"statfs":{"total":95995035648,"available":94409424896,"internally_reserved":0,"allocated":1271418880,"data_stored":1271043416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":313775575},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653194,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1972116,"kb_used_data":1600972,"kb_used_omap":4,"kb_used_meta":371131,"kb_avail":91773036,"statfs":{"total":95995035648,"available":93975588864,"internally_reserved":0,"allocated":1639395328,"data_stored":1639021796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":380038855},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986643,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1425824,"kb_used_data":935124,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":92319328,"statfs":{"total":95995035648,"available":94534991872,"internally_reserved":0,"allocated":957566976,"data_stored":957191512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790413,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1270876,"kb_used_data":630228,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":92474276,"statfs":{"total":95995035648,"available":94693658624,"internally_reserved":0,"allocated":645353472,"data_stored":644978008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1588072448,"data_stored":1588068376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1256914944,"data_stored":1256914944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1628311552,"data_stored":1628307480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":951123968,"data_stored":951123968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":640548864,"data_stored":640548864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1167458304,"data_stored":1167458304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":649134080,"data_stored":649134080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":946929664,"data_stored":946929664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":958660608,"data_stored":958660608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1295253504,"data_stored":1295253504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:41:36.254 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:41:36.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:36 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 4758/152560 objects degraded (3.119%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:36.742 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:41:36.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:36.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:41:36.850 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:41:36.876 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:36 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 4758/152560 objects degraded (3.119%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:36.878 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:36.897 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:41:36.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:36.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:41:37.001 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:41:37.029 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:37.048 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:41:37.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:37.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:41:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:36 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 4758/152560 objects degraded (3.119%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:37.152 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:41:37.181 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:37.209 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:41:37.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:37.212 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:41:37.313 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:41:37.339 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:37.359 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:41:37.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:37.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:41:37.463 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:41:37.521 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:37.750 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:41:37.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:37.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:41:37.858 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:41:37.887 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:37 smithi171 ceph-mon[44189]: pgmap v2004: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 4.5 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s; 4479/147116 objects degraded (3.045%); 4.3 MiB/s, 69 objects/s recovering 2024-04-20T17:41:37.887 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:37 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:41:37.888 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:37 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:41:37.891 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:37.892 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:37 smithi005 ceph-mon[39562]: pgmap v2004: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 4.5 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s; 4479/147116 objects degraded (3.045%); 4.3 MiB/s, 69 objects/s recovering 2024-04-20T17:41:37.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:37 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:41:37.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:37 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:41:37.918 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:41:37.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:37.922 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:41:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:37 smithi106 ceph-mon[44423]: pgmap v2004: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 4.5 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 1.10k op/s; 4479/147116 objects degraded (3.045%); 4.3 MiB/s, 69 objects/s recovering 2024-04-20T17:41:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:37 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:41:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:37 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:41:38.022 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:41:38.071 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:38.337 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:41:38.338 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:41:38.352 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:41:38.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:38.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:41:38.465 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:41:38.499 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:38.519 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:41:38.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:38.523 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:41:38.624 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:41:38.631 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:38 smithi005 ceph-mon[39562]: osdmap e750: 12 total, 11 up, 9 in 2024-04-20T17:41:38.684 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:38.784 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:41:38.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:38.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:41:38.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:38 smithi171 ceph-mon[44189]: osdmap e750: 12 total, 11 up, 9 in 2024-04-20T17:41:38.888 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:41:38.912 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:41:38.920 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:38.941 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:41:38.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:38.944 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:41:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:38 smithi106 ceph-mon[44423]: osdmap e750: 12 total, 11 up, 9 in 2024-04-20T17:41:39.044 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:41:39.071 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:39.089 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:41:39.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:39.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:41:39.193 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:41:39.220 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:39.239 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:41:39.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:39.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:41:39.343 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:41:39.373 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:39.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:39 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:41:39.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:39 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:41:39.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:39 smithi005 ceph-mon[39562]: pgmap v2006: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 4.5 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 888 op/s; 4479/147116 objects degraded (3.045%); 4.4 MiB/s, 69 objects/s recovering 2024-04-20T17:41:39.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:39 smithi005 ceph-mon[39562]: osdmap e751: 12 total, 11 up, 9 in 2024-04-20T17:41:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:39 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:41:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:39 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:41:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:39 smithi106 ceph-mon[44423]: pgmap v2006: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 4.5 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 888 op/s; 4479/147116 objects degraded (3.045%); 4.4 MiB/s, 69 objects/s recovering 2024-04-20T17:41:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:39 smithi106 ceph-mon[44423]: osdmap e751: 12 total, 11 up, 9 in 2024-04-20T17:41:40.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:39 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:41:40.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:39 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:41:40.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:39 smithi171 ceph-mon[44189]: pgmap v2006: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 4.5 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 888 op/s; 4479/147116 objects degraded (3.045%); 4.4 MiB/s, 69 objects/s recovering 2024-04-20T17:41:40.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:39 smithi171 ceph-mon[44189]: osdmap e751: 12 total, 11 up, 9 in 2024-04-20T17:41:40.343 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:41:40.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:40.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:41:40.457 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:41:40.526 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:41.101 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:41:41.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:41.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:41:41.205 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:41:41.261 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:41.380 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:41:41.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:41.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:41:41.494 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:41:41.532 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:41.567 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:41:41.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:41.571 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:41:41.672 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:41:41.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:41 smithi005 ceph-mon[39562]: pgmap v2008: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 4.4 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 770 op/s; 4479/143788 objects degraded (3.115%); 2.2 MiB/s, 35 objects/s recovering 2024-04-20T17:41:41.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:41 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 4479/143788 objects degraded (3.115%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:41.747 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:41 smithi171 ceph-mon[44189]: pgmap v2008: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 4.4 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 770 op/s; 4479/143788 objects degraded (3.115%); 2.2 MiB/s, 35 objects/s recovering 2024-04-20T17:41:41.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:41 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 4479/143788 objects degraded (3.115%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:41 smithi106 ceph-mon[44423]: pgmap v2008: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 4.4 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 770 op/s; 4479/143788 objects degraded (3.115%); 2.2 MiB/s, 35 objects/s recovering 2024-04-20T17:41:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:41 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 4479/143788 objects degraded (3.115%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:42.435 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:41:42.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:42.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:41:42.543 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:41:42.572 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:42.597 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:41:42.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:42.601 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:41:42.668 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:41:42.702 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:41:42.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:42 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/198081178' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:41:42.728 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:42.749 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:41:42.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:42.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:41:42.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:42 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/198081178' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:41:42.853 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:41:42.881 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:42.899 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:41:42.900 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:42 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/198081178' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:41:42.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:42.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:41:43.003 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:41:43.030 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:43.048 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:41:43.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:43.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:41:43.152 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:41:43.178 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:43.203 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:41:43.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:43.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:41:43.307 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:41:43.360 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:43.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:43 smithi005 ceph-mon[39562]: pgmap v2009: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 4.2 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 898 op/s; 4233/138178 objects degraded (3.063%); 4.1 MiB/s, 65 objects/s recovering 2024-04-20T17:41:43.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:43 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:41:43.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:43 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/198081178' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:41:43.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:43 smithi005 ceph-mon[39562]: osdmap e752: 12 total, 11 up, 9 in 2024-04-20T17:41:44.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:43 smithi106 ceph-mon[44423]: pgmap v2009: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 4.2 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 898 op/s; 4233/138178 objects degraded (3.063%); 4.1 MiB/s, 65 objects/s recovering 2024-04-20T17:41:44.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:43 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:41:44.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:43 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/198081178' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:41:44.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:43 smithi106 ceph-mon[44423]: osdmap e752: 12 total, 11 up, 9 in 2024-04-20T17:41:44.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:43 smithi171 ceph-mon[44189]: pgmap v2009: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 4.2 GiB data, 13 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 898 op/s; 4233/138178 objects degraded (3.063%); 4.1 MiB/s, 65 objects/s recovering 2024-04-20T17:41:44.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:43 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:41:44.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:43 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/198081178' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:41:44.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:43 smithi171 ceph-mon[44189]: osdmap e752: 12 total, 11 up, 9 in 2024-04-20T17:41:44.843 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:41:44.844 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:41:45.178 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:41:45.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:45.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:41:45.295 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:41:45.341 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:45.373 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:41:45.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:45.379 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:41:45.480 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:41:45.507 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:45.528 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:41:45.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:45.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:41:45.632 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:41:45.662 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:45.688 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:41:45.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:45.692 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:41:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:45 smithi106 ceph-mon[44423]: pgmap v2011: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 4.1 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 889 op/s; 4233/134824 objects degraded (3.140%); 2.2 MiB/s, 35 objects/s recovering 2024-04-20T17:41:45.793 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:41:45.822 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:45.848 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:41:45.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:45.853 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:41:45.954 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:41:45.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:45 smithi005 ceph-mon[39562]: pgmap v2011: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 4.1 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 889 op/s; 4233/134824 objects degraded (3.140%); 2.2 MiB/s, 35 objects/s recovering 2024-04-20T17:41:46.013 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:46.069 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:41:46.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:46.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:41:46.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:45 smithi171 ceph-mon[44189]: pgmap v2011: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 4.1 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 0 B/s wr, 889 op/s; 4233/134824 objects degraded (3.140%); 2.2 MiB/s, 35 objects/s recovering 2024-04-20T17:41:46.117 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-20T17:41:46.179 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:41:46.208 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:46.235 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:41:46.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:46.240 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:41:46.341 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:41:46.369 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:46.392 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:41:46.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:46.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:41:46.404 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:41:46.500 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:41:46.527 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:46.677 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:41:46.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:46.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:41:46.781 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:41:46.811 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:41:46.862 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:46 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 4233/134824 objects degraded (3.140%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:46.863 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:46.922 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:46 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 4233/134824 objects degraded (3.140%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:47.043 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:41:47.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:47.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:41:47.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:46 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 4233/134824 objects degraded (3.140%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:47.171 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:41:47.200 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:47.229 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:41:47.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:47.234 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:41:47.336 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:41:47.367 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:47.467 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:41:47.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:47.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:41:47.577 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:41:47.610 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:47.638 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:41:47.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:47.643 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:41:47.667 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:41:47.717 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:41:47.744 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:41:47.780 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:47.792 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:41:47.807 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:41:47.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:47.810 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:41:47.825 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:41:47.861 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:41:47.862 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:47 smithi005 ceph-mon[39562]: pgmap v2012: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.9 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s; 3937/129206 objects degraded (3.047%); 4.2 MiB/s, 67 objects/s recovering 2024-04-20T17:41:47.862 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:47 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2121157401' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:41:47.896 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:41:47.911 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:41:47.940 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:47 smithi171 ceph-mon[44189]: pgmap v2012: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.9 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s; 3937/129206 objects degraded (3.047%); 4.2 MiB/s, 67 objects/s recovering 2024-04-20T17:41:47.940 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:47 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2121157401' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:41:47.941 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:41:47.941 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:47.968 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:41:47.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:47.972 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:41:47.985 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:41:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:47 smithi106 ceph-mon[44423]: pgmap v2012: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.9 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s; 3937/129206 objects degraded (3.047%); 4.2 MiB/s, 67 objects/s recovering 2024-04-20T17:41:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:47 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2121157401' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:41:48.032 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:41:48.068 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:41:48.073 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:41:48.103 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:41:48.103 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:41:48.103 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:41:48.103 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:41:48.103 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:41:48.104 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:41:48.104 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:41:48.104 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:41:48.104 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:41:48.104 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:41:48.104 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:41:48.104 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:41:48.104 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:41:48.104 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:48.127 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:41:48.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:48.133 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:41:48.234 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:41:48.266 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:48.292 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:41:48.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:48.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:41:48.396 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:41:48.541 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:48.550 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:41:48.550 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:41:48.550 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:48.550 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27088.0:229907 7.8 e752/731 7:120dd6e3:::benchmark_data_smithi171_161871_object84181:head v 752'17869, mlcod=752'17867)", 2024-04-20T17:41:48.550 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:41:48.549782+0000", 2024-04-20T17:41:48.550 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.00081516800000000001, 2024-04-20T17:41:48.550 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00083707499999999995, 2024-04-20T17:41:48.550 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:41:48.551 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:41:48.551 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-04-20T17:41:48.551 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:41:48.551 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:48.551 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:41:48.551 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:41:48.549782+0000", 2024-04-20T17:41:48.551 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:48.551 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:48.551 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:48.551 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:41:48.551 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:41:48.549782+0000", 2024-04-20T17:41:48.551 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:48.551 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:48.551 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:48.552 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:41:48.552 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:41:48.549783+0000", 2024-04-20T17:41:48.552 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2109999999999999e-06 2024-04-20T17:41:48.552 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:48.552 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:48.552 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:41:48.552 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:41:48.549789+0000", 2024-04-20T17:41:48.552 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.964e-06 2024-04-20T17:41:48.552 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:48.552 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:48.552 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:41:48.552 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:41:48.549798+0000", 2024-04-20T17:41:48.552 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8189999999999998e-06 2024-04-20T17:41:48.552 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:48.552 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:48.553 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:41:48.553 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:41:48.549803+0000", 2024-04-20T17:41:48.553 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2939999999999996e-06 2024-04-20T17:41:48.553 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:48.553 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:48.553 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:41:48.553 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:41:48.549847+0000", 2024-04-20T17:41:48.553 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4542000000000001e-05 2024-04-20T17:41:48.553 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:48.553 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:48.553 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:41:48.553 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:41:48.549864+0000", 2024-04-20T17:41:48.553 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6560000000000001e-05 2024-04-20T17:41:48.553 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:48.553 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:41:48.554 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:48.554 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:48.554 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-04-20T17:41:48.554 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 1 2024-04-20T17:41:48.554 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:41:48.863 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:41:48.926 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:48.926 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:48.926 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:48.926 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:48.926 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:48.926 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:48.926 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:48.926 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:48 smithi005 ceph-mon[39562]: pgmap v2013: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.9 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 904 op/s; 3937/129206 objects degraded (3.047%); 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:41:48.932 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:48.933 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:48.933 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:48.933 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:48.933 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:48.933 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:48.933 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:48.933 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:48 smithi106 ceph-mon[44423]: pgmap v2013: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.9 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 904 op/s; 3937/129206 objects degraded (3.047%); 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:41:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:49.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:48 smithi171 ceph-mon[44189]: pgmap v2013: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.9 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 904 op/s; 3937/129206 objects degraded (3.047%); 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:41:49.101 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:41:49.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:49.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:41:49.214 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:41:49.248 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:49.264 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-20T17:41:49.268 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:41:49.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:49.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:41:49.373 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:41:49.403 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:49.430 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:41:49.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:49.434 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:41:49.536 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:41:49.563 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:41:49.573 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:49.599 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:41:49.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:49.603 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:41:49.704 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:41:49.734 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:49.758 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:41:49.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:49.762 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:41:49.863 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:41:49.883 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:49.883 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:49.883 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:49.883 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:49.883 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:49 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:41:49.883 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:49 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:41:49.883 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:49 smithi005 ceph-mon[39562]: osdmap e753: 12 total, 11 up, 9 in 2024-04-20T17:41:49.906 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:49.922 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:41:49.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:49.925 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:41:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:49 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:41:50.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:49 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:41:50.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:49 smithi106 ceph-mon[44423]: osdmap e753: 12 total, 11 up, 9 in 2024-04-20T17:41:50.026 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:41:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:41:50.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:49 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:41:50.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:49 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:41:50.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:49 smithi171 ceph-mon[44189]: osdmap e753: 12 total, 11 up, 9 in 2024-04-20T17:41:50.099 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:50.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:50 smithi005 ceph-mon[39562]: pgmap v2015: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.8 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 782 op/s; 3937/125658 objects degraded (3.133%); 2.3 MiB/s, 36 objects/s recovering 2024-04-20T17:41:50.987 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:50 smithi106 ceph-mon[44423]: pgmap v2015: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.8 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 782 op/s; 3937/125658 objects degraded (3.133%); 2.3 MiB/s, 36 objects/s recovering 2024-04-20T17:41:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:50 smithi171 ceph-mon[44189]: pgmap v2015: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.8 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 782 op/s; 3937/125658 objects degraded (3.133%); 2.3 MiB/s, 36 objects/s recovering 2024-04-20T17:41:51.108 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:41:51.108 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:41:51.369 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:41:51.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:51.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:41:51.487 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:41:51.517 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:51.621 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:41:51.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:51.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:41:51.657 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:41:51.658 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:41:51.658 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-04-20T17:41:51.658 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-04-20T17:41:51.658 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:41:51.725 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:41:51.752 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:51.794 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:41:51.900 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:41:51.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:51.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:41:52.003 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:41:52.106 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:52.145 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:41:52.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:52.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:41:52.260 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:41:52.289 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:52.314 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:41:52.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:52.318 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:41:52.419 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:41:52.439 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-20T17:41:52.467 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:41:52.485 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:52.498 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:41:52.723 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:41:52.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:52 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 3937/125658 objects degraded (3.133%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:52.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:52 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 3937/125658 objects degraded (3.133%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:52.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:52 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 3937/125658 objects degraded (3.133%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:52.969 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:41:52.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:52.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:41:53.080 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:41:53.109 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:41:53.113 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:53.137 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:41:53.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:53.140 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:41:53.241 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:41:53.327 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:53.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:53 smithi005 ceph-mon[39562]: from='client.27307 v1:172.21.15.5:0/2804597798' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:41:53.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:53 smithi005 ceph-mon[39562]: pgmap v2016: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.7 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 916 op/s; 3652/120094 objects degraded (3.041%); 3.7 MiB/s, 58 objects/s recovering 2024-04-20T17:41:53.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:53 smithi171 ceph-mon[44189]: from='client.27307 v1:172.21.15.5:0/2804597798' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:41:53.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:53 smithi171 ceph-mon[44189]: pgmap v2016: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.7 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 916 op/s; 3652/120094 objects degraded (3.041%); 3.7 MiB/s, 58 objects/s recovering 2024-04-20T17:41:54.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:53 smithi106 ceph-mon[44423]: from='client.27307 v1:172.21.15.5:0/2804597798' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:41:54.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:53 smithi106 ceph-mon[44423]: pgmap v2016: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.7 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 916 op/s; 3652/120094 objects degraded (3.041%); 3.7 MiB/s, 58 objects/s recovering 2024-04-20T17:41:54.702 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:41:54.703 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-04-20T17:41:54.703 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-04-20T17:41:54.703 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:41:54.703 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.703 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:29 6.e 6:717b5a9e:::benchmark_data_smithi171_147972_object28:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e627)", 2024-04-20T17:41:54.703 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:33:43.456985+0000", 2024-04-20T17:41:54.703 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 491.23834641600001, 2024-04-20T17:41:54.703 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.097665752999999994, 2024-04-20T17:41:54.703 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:41:54.703 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:41:54.703 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:41:54.703 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:41:54.704 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:41:54.704 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:41:54.704 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 29 2024-04-20T17:41:54.704 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.704 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:41:54.704 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.704 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:41:54.704 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.456985+0000", 2024-04-20T17:41:54.704 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.704 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.704 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.704 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:41:54.704 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.456985+0000", 2024-04-20T17:41:54.704 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.705 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.705 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.705 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:41:54.705 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.456986+0000", 2024-04-20T17:41:54.705 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8299999999999995e-07 2024-04-20T17:41:54.705 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.705 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.705 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:41:54.705 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457050+0000", 2024-04-20T17:41:54.705 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4165999999999995e-05 2024-04-20T17:41:54.705 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.705 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.705 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:41:54.705 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457058+0000", 2024-04-20T17:41:54.706 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5549999999999997e-06 2024-04-20T17:41:54.706 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.706 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.706 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:41:54.706 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457063+0000", 2024-04-20T17:41:54.706 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.702e-06 2024-04-20T17:41:54.706 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.706 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.706 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:41:54.706 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457092+0000", 2024-04-20T17:41:54.706 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8883000000000001e-05 2024-04-20T17:41:54.706 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.706 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.707 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:41:54.707 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457095+0000", 2024-04-20T17:41:54.707 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.755e-06 2024-04-20T17:41:54.707 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.707 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.707 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:41:54.707 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.551566+0000", 2024-04-20T17:41:54.707 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.094470815 2024-04-20T17:41:54.707 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.707 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.707 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:41:54.707 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.551689+0000", 2024-04-20T17:41:54.707 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012283699999999999 2024-04-20T17:41:54.707 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.707 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.708 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2", 2024-04-20T17:41:54.708 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.551783+0000", 2024-04-20T17:41:54.708 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3807999999999994e-05 2024-04-20T17:41:54.708 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.708 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.708 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:41:54.708 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.553184+0000", 2024-04-20T17:41:54.708 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001400985 2024-04-20T17:41:54.708 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.708 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.708 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:41:54.708 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.554548+0000", 2024-04-20T17:41:54.709 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013636919999999999 2024-04-20T17:41:54.709 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.709 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.709 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:41:54.709 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.554609+0000", 2024-04-20T17:41:54.709 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1482000000000005e-05 2024-04-20T17:41:54.709 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.709 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.709 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:41:54.709 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.554651+0000", 2024-04-20T17:41:54.709 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2190000000000001e-05 2024-04-20T17:41:54.709 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.709 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:41:54.710 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.710 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.710 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.710 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:58 6.e 6:7b14fdfd:::benchmark_data_smithi171_147972_object57:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e627)", 2024-04-20T17:41:54.710 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:33:43.457071+0000", 2024-04-20T17:41:54.710 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 491.23826074099998, 2024-04-20T17:41:54.710 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.097978955000000006, 2024-04-20T17:41:54.710 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:41:54.710 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:41:54.710 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:41:54.710 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:41:54.710 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:41:54.710 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:41:54.710 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 58 2024-04-20T17:41:54.711 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.711 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:41:54.711 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.711 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:41:54.711 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457071+0000", 2024-04-20T17:41:54.711 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.711 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.711 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.711 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:41:54.711 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457071+0000", 2024-04-20T17:41:54.711 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.711 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.711 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.711 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:41:54.712 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457072+0000", 2024-04-20T17:41:54.712 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8500000000000001e-07 2024-04-20T17:41:54.712 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.712 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.712 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:41:54.712 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457125+0000", 2024-04-20T17:41:54.712 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3664e-05 2024-04-20T17:41:54.712 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.712 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.712 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:41:54.712 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457132+0000", 2024-04-20T17:41:54.712 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6900000000000003e-06 2024-04-20T17:41:54.712 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.712 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.712 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:41:54.713 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457137+0000", 2024-04-20T17:41:54.713 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3460000000000004e-06 2024-04-20T17:41:54.713 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.713 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.713 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:41:54.713 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457169+0000", 2024-04-20T17:41:54.713 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2632000000000002e-05 2024-04-20T17:41:54.713 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.713 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.713 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-04-20T17:41:54.713 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.457173+0000", 2024-04-20T17:41:54.713 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6950000000000002e-06 2024-04-20T17:41:54.713 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.713 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.713 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:41:54.714 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.552054+0000", 2024-04-20T17:41:54.714 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.094880860999999997 2024-04-20T17:41:54.714 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.714 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.714 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:41:54.714 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.552175+0000", 2024-04-20T17:41:54.714 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000120877 2024-04-20T17:41:54.714 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.714 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.714 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2", 2024-04-20T17:41:54.714 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.552331+0000", 2024-04-20T17:41:54.714 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015667799999999999 2024-04-20T17:41:54.714 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.714 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.715 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:41:54.715 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.553192+0000", 2024-04-20T17:41:54.715 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00086076100000000003 2024-04-20T17:41:54.715 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.715 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.715 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:41:54.715 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.554942+0000", 2024-04-20T17:41:54.715 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017503130000000001 2024-04-20T17:41:54.715 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.715 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.715 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:41:54.715 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.555008+0000", 2024-04-20T17:41:54.715 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5458999999999998e-05 2024-04-20T17:41:54.715 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.715 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.716 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:41:54.716 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:33:43.555050+0000", 2024-04-20T17:41:54.716 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2293999999999999e-05 2024-04-20T17:41:54.716 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.716 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:41:54.716 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.716 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.716 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.716 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:155954 6.2 6:44e5c6b4:::benchmark_data_smithi171_147972_object155953:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e646)", 2024-04-20T17:41:54.716 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:04.109965+0000", 2024-04-20T17:41:54.716 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 410.585366556, 2024-04-20T17:41:54.716 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.134973547, 2024-04-20T17:41:54.716 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:41:54.716 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:41:54.716 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:41:54.717 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:41:54.717 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:41:54.717 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:41:54.717 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 155954 2024-04-20T17:41:54.717 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.717 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:41:54.717 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.717 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:41:54.717 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.109965+0000", 2024-04-20T17:41:54.717 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.717 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.717 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.717 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:41:54.717 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.109965+0000", 2024-04-20T17:41:54.718 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.718 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.718 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.718 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:41:54.718 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.109967+0000", 2024-04-20T17:41:54.718 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.17e-06 2024-04-20T17:41:54.718 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.718 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.718 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:41:54.718 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.110007+0000", 2024-04-20T17:41:54.718 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9799999999999998e-05 2024-04-20T17:41:54.718 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.718 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:41:54.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.110019+0000", 2024-04-20T17:41:54.719 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1402e-05 2024-04-20T17:41:54.719 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.719 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:41:54.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.110026+0000", 2024-04-20T17:41:54.719 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3490000000000003e-06 2024-04-20T17:41:54.719 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.719 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:41:54.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.110072+0000", 2024-04-20T17:41:54.719 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6300000000000001e-05 2024-04-20T17:41:54.719 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.720 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.720 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:41:54.720 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.110206+0000", 2024-04-20T17:41:54.720 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013326500000000001 2024-04-20T17:41:54.720 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.720 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.720 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:41:54.720 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.110350+0000", 2024-04-20T17:41:54.720 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014443400000000001 2024-04-20T17:41:54.720 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.720 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.720 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:41:54.720 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.120041+0000", 2024-04-20T17:41:54.720 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0096908340000000006 2024-04-20T17:41:54.721 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.721 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.721 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:41:54.721 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.244862+0000", 2024-04-20T17:41:54.721 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12482156899999999 2024-04-20T17:41:54.721 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.721 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.721 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:41:54.721 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.244895+0000", 2024-04-20T17:41:54.721 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2886000000000003e-05 2024-04-20T17:41:54.721 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.721 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.721 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:41:54.721 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.244939+0000", 2024-04-20T17:41:54.721 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3538000000000002e-05 2024-04-20T17:41:54.722 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.722 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:41:54.722 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.722 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.722 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.722 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:155956 6.2 6:45a14e0f:::benchmark_data_smithi171_147972_object155955:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e646)", 2024-04-20T17:41:54.722 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:04.113314+0000", 2024-04-20T17:41:54.722 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 410.58201812599998, 2024-04-20T17:41:54.722 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13176359100000001, 2024-04-20T17:41:54.722 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:41:54.722 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:41:54.722 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:41:54.722 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:41:54.722 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:41:54.723 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:41:54.723 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 155956 2024-04-20T17:41:54.723 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.723 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:41:54.723 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.723 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:41:54.723 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.113314+0000", 2024-04-20T17:41:54.723 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.723 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.723 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.723 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:41:54.723 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.113314+0000", 2024-04-20T17:41:54.723 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.723 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.723 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.724 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:41:54.724 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.113315+0000", 2024-04-20T17:41:54.724 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4559999999999999e-06 2024-04-20T17:41:54.724 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.724 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.724 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:41:54.724 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.113379+0000", 2024-04-20T17:41:54.724 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3938999999999999e-05 2024-04-20T17:41:54.724 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.724 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.724 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:41:54.724 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.113389+0000", 2024-04-20T17:41:54.724 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6479999999999998e-06 2024-04-20T17:41:54.724 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.724 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.725 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:41:54.725 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.113395+0000", 2024-04-20T17:41:54.725 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7559999999999996e-06 2024-04-20T17:41:54.725 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.725 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.725 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:41:54.725 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114959+0000", 2024-04-20T17:41:54.725 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001564766 2024-04-20T17:41:54.725 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.725 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.725 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:41:54.725 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.115095+0000", 2024-04-20T17:41:54.725 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000135566 2024-04-20T17:41:54.725 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.725 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.726 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:41:54.726 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.115239+0000", 2024-04-20T17:41:54.726 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014454600000000001 2024-04-20T17:41:54.726 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.726 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.726 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:41:54.726 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.244987+0000", 2024-04-20T17:41:54.726 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12974762200000001 2024-04-20T17:41:54.726 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.726 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.726 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:41:54.726 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.244997+0000", 2024-04-20T17:41:54.726 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6099999999999995e-06 2024-04-20T17:41:54.726 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.727 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.727 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:41:54.727 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.245027+0000", 2024-04-20T17:41:54.727 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0741000000000003e-05 2024-04-20T17:41:54.727 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.727 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.727 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:41:54.727 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.245077+0000", 2024-04-20T17:41:54.727 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9941000000000002e-05 2024-04-20T17:41:54.727 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.727 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:41:54.727 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.727 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.727 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.728 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:155958 6.2 6:47045901:::benchmark_data_smithi171_147972_object155957:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e646)", 2024-04-20T17:41:54.728 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:04.114372+0000", 2024-04-20T17:41:54.728 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 410.58095951000001, 2024-04-20T17:41:54.728 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13086056099999999, 2024-04-20T17:41:54.728 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:41:54.728 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:41:54.728 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:41:54.728 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:41:54.728 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:41:54.728 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:41:54.728 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 155958 2024-04-20T17:41:54.728 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.728 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:41:54.728 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.728 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:41:54.728 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114372+0000", 2024-04-20T17:41:54.729 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.729 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.729 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.729 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:41:54.729 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114372+0000", 2024-04-20T17:41:54.729 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.729 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.729 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.729 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:41:54.729 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114374+0000", 2024-04-20T17:41:54.729 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5370000000000001e-06 2024-04-20T17:41:54.729 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.729 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.729 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:41:54.730 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114433+0000", 2024-04-20T17:41:54.730 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9064000000000002e-05 2024-04-20T17:41:54.730 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.730 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.730 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:41:54.730 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114443+0000", 2024-04-20T17:41:54.730 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0236e-05 2024-04-20T17:41:54.730 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.730 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.730 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:41:54.730 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114450+0000", 2024-04-20T17:41:54.730 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1940000000000001e-06 2024-04-20T17:41:54.730 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.730 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.730 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:41:54.731 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.120077+0000", 2024-04-20T17:41:54.731 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.005626458 2024-04-20T17:41:54.731 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.731 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.731 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:41:54.731 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.232545+0000", 2024-04-20T17:41:54.731 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.112467761 2024-04-20T17:41:54.731 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.731 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.731 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:41:54.731 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.232710+0000", 2024-04-20T17:41:54.731 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016564600000000001 2024-04-20T17:41:54.731 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.731 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.732 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:41:54.732 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.245088+0000", 2024-04-20T17:41:54.732 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012377384999999999 2024-04-20T17:41:54.732 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.732 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.732 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:41:54.732 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.245134+0000", 2024-04-20T17:41:54.732 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5926e-05 2024-04-20T17:41:54.732 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.732 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.732 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:41:54.732 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.245183+0000", 2024-04-20T17:41:54.732 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9815999999999999e-05 2024-04-20T17:41:54.732 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.732 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.733 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:41:54.733 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.245233+0000", 2024-04-20T17:41:54.733 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9537999999999999e-05 2024-04-20T17:41:54.733 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.733 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:41:54.733 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.733 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.733 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.733 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:155959 6.e 6:7825f00e:::benchmark_data_smithi171_147972_object155958:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e646)", 2024-04-20T17:41:54.733 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:04.114461+0000", 2024-04-20T17:41:54.733 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 410.580870602, 2024-04-20T17:41:54.733 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12842199500000001, 2024-04-20T17:41:54.733 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:41:54.733 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:41:54.734 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:41:54.734 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:41:54.734 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:41:54.734 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:41:54.734 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 155959 2024-04-20T17:41:54.734 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.734 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:41:54.734 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.734 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:41:54.734 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114461+0000", 2024-04-20T17:41:54.734 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.734 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.734 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.734 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:41:54.734 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114461+0000", 2024-04-20T17:41:54.735 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.735 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.735 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.735 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:41:54.735 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114462+0000", 2024-04-20T17:41:54.735 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7199999999999997e-07 2024-04-20T17:41:54.735 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.735 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.735 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:41:54.735 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114522+0000", 2024-04-20T17:41:54.735 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9587999999999999e-05 2024-04-20T17:41:54.735 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.735 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.735 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:41:54.735 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114528+0000", 2024-04-20T17:41:54.736 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7039999999999997e-06 2024-04-20T17:41:54.736 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.736 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.736 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:41:54.736 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114534+0000", 2024-04-20T17:41:54.736 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.981e-06 2024-04-20T17:41:54.736 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.736 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.736 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:41:54.736 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.116285+0000", 2024-04-20T17:41:54.736 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001750754 2024-04-20T17:41:54.736 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.736 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.736 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:41:54.736 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.116400+0000", 2024-04-20T17:41:54.737 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011446999999999999 2024-04-20T17:41:54.737 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.737 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.737 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2", 2024-04-20T17:41:54.737 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.116529+0000", 2024-04-20T17:41:54.737 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012922300000000001 2024-04-20T17:41:54.737 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.737 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.737 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:41:54.737 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.233387+0000", 2024-04-20T17:41:54.737 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.11685820099999999 2024-04-20T17:41:54.737 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.737 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.737 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:41:54.737 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.242762+0000", 2024-04-20T17:41:54.738 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0093748029999999993 2024-04-20T17:41:54.738 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.738 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.738 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:41:54.738 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.242820+0000", 2024-04-20T17:41:54.738 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8207e-05 2024-04-20T17:41:54.738 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.738 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.738 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:41:54.738 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.242883+0000", 2024-04-20T17:41:54.738 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3192000000000001e-05 2024-04-20T17:41:54.738 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.738 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:41:54.738 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.738 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.739 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.739 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26882.0:155961 6.6 e646/619 6:62ebcb10:::benchmark_data_smithi171_147972_object155960:head v 646'9787, mlcod=646'9785)", 2024-04-20T17:41:54.739 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:04.115125+0000", 2024-04-20T17:41:54.739 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 410.58020688599998, 2024-04-20T17:41:54.739 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.123201243, 2024-04-20T17:41:54.739 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:41:54.739 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:41:54.739 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:41:54.739 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:41:54.739 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.739 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:41:54.739 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.115125+0000", 2024-04-20T17:41:54.739 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.739 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.739 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.739 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:41:54.740 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.115125+0000", 2024-04-20T17:41:54.740 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.740 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.740 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.740 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:41:54.740 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.115127+0000", 2024-04-20T17:41:54.740 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7489999999999999e-06 2024-04-20T17:41:54.740 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.740 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.740 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:41:54.740 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.115169+0000", 2024-04-20T17:41:54.740 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2305999999999998e-05 2024-04-20T17:41:54.740 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.740 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.740 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:41:54.741 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.115178+0000", 2024-04-20T17:41:54.741 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6419999999999996e-06 2024-04-20T17:41:54.741 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.741 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.741 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:41:54.741 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.115185+0000", 2024-04-20T17:41:54.741 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3780000000000003e-06 2024-04-20T17:41:54.741 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.741 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.741 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:41:54.741 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.116276+0000", 2024-04-20T17:41:54.741 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010913470000000001 2024-04-20T17:41:54.741 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.741 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.741 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:41:54.741 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.116294+0000", 2024-04-20T17:41:54.742 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7342000000000001e-05 2024-04-20T17:41:54.742 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.742 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.742 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:41:54.742 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.238286+0000", 2024-04-20T17:41:54.742 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12199251899999999 2024-04-20T17:41:54.742 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.742 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.742 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:41:54.742 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.238326+0000", 2024-04-20T17:41:54.742 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9959999999999997e-05 2024-04-20T17:41:54.742 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.742 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:41:54.742 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.743 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.743 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.743 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.26882.0:155954 6.2 e646/619 ondisk, result = 0)", 2024-04-20T17:41:54.743 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:04.120868+0000", 2024-04-20T17:41:54.743 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 410.57446427000002, 2024-04-20T17:41:54.743 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12407834299999999, 2024-04-20T17:41:54.743 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:41:54.743 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:41:54.743 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-04-20T17:41:54.743 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:41:54.743 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.743 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:41:54.743 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.120868+0000", 2024-04-20T17:41:54.743 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.743 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.744 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.744 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:41:54.744 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.120868+0000", 2024-04-20T17:41:54.744 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.744 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.744 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.744 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:41:54.744 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.120869+0000", 2024-04-20T17:41:54.744 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4619999999999999e-06 2024-04-20T17:41:54.744 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.744 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.744 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:41:54.744 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.120876+0000", 2024-04-20T17:41:54.744 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6869999999999997e-06 2024-04-20T17:41:54.744 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.745 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.745 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:41:54.745 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.232196+0000", 2024-04-20T17:41:54.745 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.11132056999999999 2024-04-20T17:41:54.745 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.745 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.745 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:41:54.745 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.232205+0000", 2024-04-20T17:41:54.745 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1400000000000006e-06 2024-04-20T17:41:54.745 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.745 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.745 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:41:54.745 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.244854+0000", 2024-04-20T17:41:54.745 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012648995 2024-04-20T17:41:54.745 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.745 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.746 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:41:54.746 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.244859+0000", 2024-04-20T17:41:54.746 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5510000000000001e-06 2024-04-20T17:41:54.746 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.746 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.746 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:41:54.746 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.244946+0000", 2024-04-20T17:41:54.746 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6938e-05 2024-04-20T17:41:54.746 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.746 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:41:54.746 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.746 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.746 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.746 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:155974 6.e 6:7d3bdd8a:::benchmark_data_smithi171_147972_object155973:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e646)", 2024-04-20T17:41:54.747 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:04.129245+0000", 2024-04-20T17:41:54.747 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 410.56608709, 2024-04-20T17:41:54.747 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.113882792, 2024-04-20T17:41:54.747 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:41:54.747 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:41:54.747 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:41:54.747 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:41:54.747 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:41:54.747 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:41:54.747 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 155974 2024-04-20T17:41:54.747 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.747 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:41:54.747 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.747 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:41:54.747 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.129245+0000", 2024-04-20T17:41:54.748 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.748 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.748 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.748 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:41:54.748 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.129245+0000", 2024-04-20T17:41:54.748 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.748 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.748 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.748 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:41:54.748 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.129247+0000", 2024-04-20T17:41:54.748 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2790000000000001e-06 2024-04-20T17:41:54.748 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.748 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.748 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:41:54.748 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.129324+0000", 2024-04-20T17:41:54.749 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6453999999999997e-05 2024-04-20T17:41:54.749 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.749 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.749 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:41:54.749 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.232085+0000", 2024-04-20T17:41:54.749 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.102761221 2024-04-20T17:41:54.749 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.749 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.749 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:41:54.749 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.232103+0000", 2024-04-20T17:41:54.749 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8057999999999999e-05 2024-04-20T17:41:54.749 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.749 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.749 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:41:54.749 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.233436+0000", 2024-04-20T17:41:54.750 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013337119999999999 2024-04-20T17:41:54.750 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.750 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.750 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:41:54.750 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.233544+0000", 2024-04-20T17:41:54.750 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000107865 2024-04-20T17:41:54.750 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.750 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.750 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2", 2024-04-20T17:41:54.750 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.233686+0000", 2024-04-20T17:41:54.750 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000141435 2024-04-20T17:41:54.750 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.750 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.750 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:41:54.751 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.242948+0000", 2024-04-20T17:41:54.751 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0092626500000000007 2024-04-20T17:41:54.751 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.751 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.751 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:41:54.751 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.243023+0000", 2024-04-20T17:41:54.751 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4752000000000005e-05 2024-04-20T17:41:54.751 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.751 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.751 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:41:54.751 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.243069+0000", 2024-04-20T17:41:54.751 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5723000000000003e-05 2024-04-20T17:41:54.751 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.751 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.751 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:41:54.751 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.243128+0000", 2024-04-20T17:41:54.752 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8643000000000001e-05 2024-04-20T17:41:54.752 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.752 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:41:54.752 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.752 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.752 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.752 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:180908 6.2 6:4681e151:::benchmark_data_smithi171_147972_object19810:head [delete] snapc 0=[] ondisk+write+known_if_redirected e653)", 2024-04-20T17:41:54.752 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:26.376220+0000", 2024-04-20T17:41:54.752 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 388.31911151100002, 2024-04-20T17:41:54.752 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.11115101199999999, 2024-04-20T17:41:54.752 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:41:54.752 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:41:54.752 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:41:54.752 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:41:54.752 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:41:54.753 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:41:54.753 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 180908 2024-04-20T17:41:54.753 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.753 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:41:54.753 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.753 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:41:54.753 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.376220+0000", 2024-04-20T17:41:54.753 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.753 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.753 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.753 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:41:54.753 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.376220+0000", 2024-04-20T17:41:54.753 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.753 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.753 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.754 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:41:54.754 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.376224+0000", 2024-04-20T17:41:54.754 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.162e-06 2024-04-20T17:41:54.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.754 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:41:54.754 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.376232+0000", 2024-04-20T17:41:54.754 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0690000000000002e-06 2024-04-20T17:41:54.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.754 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:41:54.754 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.376244+0000", 2024-04-20T17:41:54.754 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2551000000000001e-05 2024-04-20T17:41:54.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.754 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:41:54.755 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.376255+0000", 2024-04-20T17:41:54.755 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0549e-05 2024-04-20T17:41:54.755 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.755 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.756 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:41:54.756 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.376312+0000", 2024-04-20T17:41:54.756 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7708999999999999e-05 2024-04-20T17:41:54.756 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.756 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.756 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:41:54.756 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.376417+0000", 2024-04-20T17:41:54.756 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000105049 2024-04-20T17:41:54.756 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.756 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.756 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:41:54.757 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.376493+0000", 2024-04-20T17:41:54.757 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5427999999999997e-05 2024-04-20T17:41:54.757 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.757 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.757 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:41:54.757 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.380255+0000", 2024-04-20T17:41:54.757 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003761838 2024-04-20T17:41:54.757 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.757 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.757 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:41:54.757 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.487254+0000", 2024-04-20T17:41:54.757 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.10699963899999999 2024-04-20T17:41:54.757 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.757 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.757 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:41:54.758 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.487316+0000", 2024-04-20T17:41:54.758 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1161000000000006e-05 2024-04-20T17:41:54.758 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.758 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.758 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:41:54.758 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.487371+0000", 2024-04-20T17:41:54.758 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5856999999999997e-05 2024-04-20T17:41:54.758 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.758 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:41:54.758 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.758 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.758 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.758 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:180915 6.2 6:4449404c:::benchmark_data_smithi171_147972_object19817:head [delete] snapc 0=[] ondisk+write+known_if_redirected e653)", 2024-04-20T17:41:54.758 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:26.382605+0000", 2024-04-20T17:41:54.759 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 388.31272658300003, 2024-04-20T17:41:54.759 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.115379974, 2024-04-20T17:41:54.759 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:41:54.759 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:41:54.759 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:41:54.759 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:41:54.759 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:41:54.759 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:41:54.759 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 180915 2024-04-20T17:41:54.759 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.759 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:41:54.759 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.759 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:41:54.759 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382605+0000", 2024-04-20T17:41:54.759 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.759 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.760 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.760 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:41:54.760 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382605+0000", 2024-04-20T17:41:54.760 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.760 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.760 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.760 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:41:54.760 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382607+0000", 2024-04-20T17:41:54.760 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0159999999999998e-06 2024-04-20T17:41:54.760 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.760 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.760 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:41:54.760 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382613+0000", 2024-04-20T17:41:54.760 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1659999999999999e-06 2024-04-20T17:41:54.760 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:41:54.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382639+0000", 2024-04-20T17:41:54.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5096000000000002e-05 2024-04-20T17:41:54.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:41:54.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382645+0000", 2024-04-20T17:41:54.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6470000000000001e-06 2024-04-20T17:41:54.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:41:54.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382709+0000", 2024-04-20T17:41:54.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3996000000000002e-05 2024-04-20T17:41:54.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:41:54.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382804+0000", 2024-04-20T17:41:54.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4871999999999996e-05 2024-04-20T17:41:54.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:41:54.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382873+0000", 2024-04-20T17:41:54.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8585999999999997e-05 2024-04-20T17:41:54.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:41:54.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.390622+0000", 2024-04-20T17:41:54.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0077494310000000002 2024-04-20T17:41:54.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.763 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.763 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:41:54.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.497875+0000", 2024-04-20T17:41:54.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.107252942 2024-04-20T17:41:54.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.763 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.763 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:41:54.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.497928+0000", 2024-04-20T17:41:54.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2814e-05 2024-04-20T17:41:54.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.763 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.763 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:41:54.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.497985+0000", 2024-04-20T17:41:54.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7408000000000003e-05 2024-04-20T17:41:54.763 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.764 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:41:54.764 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.764 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.764 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.764 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:180917 6.2 6:4941fea8:::benchmark_data_smithi171_147972_object19819:head [delete] snapc 0=[] ondisk+write+known_if_redirected e653)", 2024-04-20T17:41:54.764 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:26.382662+0000", 2024-04-20T17:41:54.764 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 388.31266982199998, 2024-04-20T17:41:54.764 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.117894331, 2024-04-20T17:41:54.764 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:41:54.764 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:41:54.764 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:41:54.764 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:41:54.764 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:41:54.764 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:41:54.764 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 180917 2024-04-20T17:41:54.765 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.765 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:41:54.765 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.765 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:41:54.765 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382662+0000", 2024-04-20T17:41:54.765 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.765 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.765 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.765 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:41:54.765 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382662+0000", 2024-04-20T17:41:54.765 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.765 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.765 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.765 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:41:54.765 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382663+0000", 2024-04-20T17:41:54.766 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9599999999999998e-07 2024-04-20T17:41:54.766 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.766 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.766 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:41:54.766 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382666+0000", 2024-04-20T17:41:54.766 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.275e-06 2024-04-20T17:41:54.766 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.766 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.766 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:41:54.766 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382677+0000", 2024-04-20T17:41:54.766 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0759e-05 2024-04-20T17:41:54.766 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.766 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.766 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:41:54.766 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.382681+0000", 2024-04-20T17:41:54.767 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.25e-06 2024-04-20T17:41:54.767 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.767 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.767 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:41:54.767 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.386364+0000", 2024-04-20T17:41:54.767 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0036826049999999998 2024-04-20T17:41:54.767 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.767 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.767 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:41:54.767 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.386490+0000", 2024-04-20T17:41:54.767 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000126716 2024-04-20T17:41:54.767 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.767 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.767 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:41:54.767 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.386614+0000", 2024-04-20T17:41:54.767 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012380799999999999 2024-04-20T17:41:54.768 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.768 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.768 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:41:54.768 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.394543+0000", 2024-04-20T17:41:54.768 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0079287790000000004 2024-04-20T17:41:54.768 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.768 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.768 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:41:54.768 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.500468+0000", 2024-04-20T17:41:54.768 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.10592457600000001 2024-04-20T17:41:54.768 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.768 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.768 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:41:54.768 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.500514+0000", 2024-04-20T17:41:54.768 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5905000000000002e-05 2024-04-20T17:41:54.769 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.769 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.769 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:41:54.769 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.500556+0000", 2024-04-20T17:41:54.769 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2861999999999997e-05 2024-04-20T17:41:54.769 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.769 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:41:54.769 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.769 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.769 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.769 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:180920 6.2 6:438d9d13:::benchmark_data_smithi171_147972_object19822:head [delete] snapc 0=[] ondisk+write+known_if_redirected e653)", 2024-04-20T17:41:54.769 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:26.383540+0000", 2024-04-20T17:41:54.769 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 388.31179174499999, 2024-04-20T17:41:54.769 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.11715096899999999, 2024-04-20T17:41:54.769 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:41:54.770 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:41:54.770 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:41:54.770 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:41:54.770 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:41:54.770 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:41:54.770 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 180920 2024-04-20T17:41:54.770 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.770 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:41:54.770 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.770 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:41:54.770 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.383540+0000", 2024-04-20T17:41:54.770 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.770 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.770 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.771 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:41:54.771 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.383540+0000", 2024-04-20T17:41:54.771 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.771 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.771 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.771 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:41:54.771 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.383542+0000", 2024-04-20T17:41:54.771 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4899999999999999e-06 2024-04-20T17:41:54.771 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.771 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.771 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:41:54.771 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.383549+0000", 2024-04-20T17:41:54.771 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4930000000000001e-06 2024-04-20T17:41:54.771 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.771 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.772 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:41:54.772 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.383560+0000", 2024-04-20T17:41:54.772 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1024e-05 2024-04-20T17:41:54.772 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.772 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.772 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:41:54.772 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.383567+0000", 2024-04-20T17:41:54.772 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3359999999999997e-06 2024-04-20T17:41:54.772 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.772 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.772 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:41:54.772 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.390655+0000", 2024-04-20T17:41:54.772 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0070878219999999997 2024-04-20T17:41:54.772 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.773 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.773 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:41:54.773 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.390773+0000", 2024-04-20T17:41:54.773 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000117464 2024-04-20T17:41:54.773 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.773 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.773 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:41:54.773 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.390888+0000", 2024-04-20T17:41:54.773 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011526 2024-04-20T17:41:54.773 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.773 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.773 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:41:54.773 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.394670+0000", 2024-04-20T17:41:54.773 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037814910000000001 2024-04-20T17:41:54.773 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.774 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.774 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:41:54.774 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.500601+0000", 2024-04-20T17:41:54.774 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.10593155 2024-04-20T17:41:54.774 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.774 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.774 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:41:54.774 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.500653+0000", 2024-04-20T17:41:54.774 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1504e-05 2024-04-20T17:41:54.774 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.774 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.774 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:41:54.774 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:26.500691+0000", 2024-04-20T17:41:54.774 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8535000000000003e-05 2024-04-20T17:41:54.774 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.775 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:41:54.775 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.775 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.775 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.775 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27088.0:176948 7.1 7:87bf052c:::benchmark_data_smithi171_161871_object31222:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:41:54.775 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:40:45.945786+0000", 2024-04-20T17:41:54.775 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 68.749545479999995, 2024-04-20T17:41:54.775 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21713122500000001, 2024-04-20T17:41:54.775 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:41:54.775 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:41:54.775 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:41:54.775 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:41:54.775 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27088", 2024-04-20T17:41:54.775 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:41:54.775 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 176948 2024-04-20T17:41:54.776 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.776 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:41:54.776 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.776 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:41:54.776 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.945786+0000", 2024-04-20T17:41:54.776 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.776 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.776 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.776 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:41:54.776 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.945786+0000", 2024-04-20T17:41:54.776 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.776 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.776 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.776 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:41:54.776 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.945789+0000", 2024-04-20T17:41:54.777 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0900000000000001e-06 2024-04-20T17:41:54.777 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.777 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.777 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:41:54.777 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.945807+0000", 2024-04-20T17:41:54.777 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7198e-05 2024-04-20T17:41:54.777 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.777 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.777 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:41:54.777 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.945826+0000", 2024-04-20T17:41:54.777 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8864000000000001e-05 2024-04-20T17:41:54.777 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.777 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.777 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:41:54.778 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.945850+0000", 2024-04-20T17:41:54.778 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4423999999999999e-05 2024-04-20T17:41:54.778 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.778 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.778 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:41:54.778 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.945909+0000", 2024-04-20T17:41:54.778 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9135999999999998e-05 2024-04-20T17:41:54.778 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.778 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.778 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:41:54.778 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.946022+0000", 2024-04-20T17:41:54.778 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000112977 2024-04-20T17:41:54.778 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.778 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.778 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:41:54.779 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.946103+0000", 2024-04-20T17:41:54.779 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1363e-05 2024-04-20T17:41:54.779 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.779 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.779 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:41:54.779 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.949973+0000", 2024-04-20T17:41:54.779 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0038697499999999999 2024-04-20T17:41:54.779 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.779 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.779 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:41:54.779 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.162849+0000", 2024-04-20T17:41:54.779 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.212876177 2024-04-20T17:41:54.779 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.779 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.779 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:41:54.780 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.162885+0000", 2024-04-20T17:41:54.780 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5908000000000002e-05 2024-04-20T17:41:54.780 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.780 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.780 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:41:54.780 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.162918+0000", 2024-04-20T17:41:54.780 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2338000000000001e-05 2024-04-20T17:41:54.780 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.780 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:41:54.780 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.780 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.780 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.780 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27088.0:176957 7.1 7:8429106d:::benchmark_data_smithi171_161871_object31231:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:41:54.780 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:40:45.960496+0000", 2024-04-20T17:41:54.780 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 68.734835834999998, 2024-04-20T17:41:54.781 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20636091400000001, 2024-04-20T17:41:54.781 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:41:54.781 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:41:54.781 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:41:54.781 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:41:54.781 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27088", 2024-04-20T17:41:54.781 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:41:54.781 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 176957 2024-04-20T17:41:54.781 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.781 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:41:54.781 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.781 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:41:54.781 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.960496+0000", 2024-04-20T17:41:54.781 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.781 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.781 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.782 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:41:54.782 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.960496+0000", 2024-04-20T17:41:54.782 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.782 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.782 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.782 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:41:54.782 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.960501+0000", 2024-04-20T17:41:54.782 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4710000000000001e-06 2024-04-20T17:41:54.782 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.782 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.782 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:41:54.782 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.960515+0000", 2024-04-20T17:41:54.782 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4215e-05 2024-04-20T17:41:54.782 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.783 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.783 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:41:54.783 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.960543+0000", 2024-04-20T17:41:54.783 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8041000000000001e-05 2024-04-20T17:41:54.783 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.783 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.783 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:41:54.783 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.960557+0000", 2024-04-20T17:41:54.783 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4632e-05 2024-04-20T17:41:54.783 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.783 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.783 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:41:54.783 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.960623+0000", 2024-04-20T17:41:54.783 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5154000000000007e-05 2024-04-20T17:41:54.783 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.784 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.784 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:41:54.784 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.960731+0000", 2024-04-20T17:41:54.784 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010856 2024-04-20T17:41:54.784 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.784 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.784 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:41:54.784 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.960816+0000", 2024-04-20T17:41:54.784 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5220000000000001e-05 2024-04-20T17:41:54.784 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.784 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.784 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:41:54.784 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.964987+0000", 2024-04-20T17:41:54.784 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0041703909999999999 2024-04-20T17:41:54.784 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.785 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.785 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:41:54.785 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.166798+0000", 2024-04-20T17:41:54.785 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20181174399999999 2024-04-20T17:41:54.785 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.785 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.785 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:41:54.785 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.166831+0000", 2024-04-20T17:41:54.785 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2106000000000001e-05 2024-04-20T17:41:54.785 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.785 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.785 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:41:54.785 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.166857+0000", 2024-04-20T17:41:54.785 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6380000000000002e-05 2024-04-20T17:41:54.786 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.786 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:41:54.786 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.786 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.786 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.786 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27088.0:176969 7.1 7:87ae8b31:::benchmark_data_smithi171_161871_object31243:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:41:54.786 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:40:45.972340+0000", 2024-04-20T17:41:54.786 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 68.722992176999995, 2024-04-20T17:41:54.786 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.194623088, 2024-04-20T17:41:54.786 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:41:54.786 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:41:54.786 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:41:54.786 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:41:54.786 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27088", 2024-04-20T17:41:54.787 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:41:54.787 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 176969 2024-04-20T17:41:54.787 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.787 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:41:54.787 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.787 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:41:54.787 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.972340+0000", 2024-04-20T17:41:54.787 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.787 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.787 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.787 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:41:54.787 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.972340+0000", 2024-04-20T17:41:54.787 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.787 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.787 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.787 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:41:54.788 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.972342+0000", 2024-04-20T17:41:54.788 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.469e-06 2024-04-20T17:41:54.788 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.788 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.788 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:41:54.788 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.972350+0000", 2024-04-20T17:41:54.788 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3479999999999998e-06 2024-04-20T17:41:54.788 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.788 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.788 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:41:54.788 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.972362+0000", 2024-04-20T17:41:54.788 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2376e-05 2024-04-20T17:41:54.788 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.788 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.788 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:41:54.789 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.972373+0000", 2024-04-20T17:41:54.789 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0847e-05 2024-04-20T17:41:54.789 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.789 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.789 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:41:54.789 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.972429+0000", 2024-04-20T17:41:54.789 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6762000000000003e-05 2024-04-20T17:41:54.789 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.789 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.789 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:41:54.789 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.972526+0000", 2024-04-20T17:41:54.789 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6280999999999995e-05 2024-04-20T17:41:54.789 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.789 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.789 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:41:54.790 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.972610+0000", 2024-04-20T17:41:54.790 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4726000000000002e-05 2024-04-20T17:41:54.790 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.790 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.790 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:41:54.790 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.976503+0000", 2024-04-20T17:41:54.790 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0038928159999999999 2024-04-20T17:41:54.790 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.790 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.790 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:41:54.790 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.166898+0000", 2024-04-20T17:41:54.790 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19039439999999999 2024-04-20T17:41:54.790 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.790 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.790 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:41:54.791 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.166931+0000", 2024-04-20T17:41:54.791 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3046999999999997e-05 2024-04-20T17:41:54.791 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.791 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.791 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:41:54.791 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.166963+0000", 2024-04-20T17:41:54.791 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2016e-05 2024-04-20T17:41:54.791 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.791 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:41:54.791 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.791 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.791 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.791 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27088.0:176983 7.1 7:8915abf7:::benchmark_data_smithi171_161871_object31257:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:41:54.791 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:40:45.983813+0000", 2024-04-20T17:41:54.791 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 68.711519205000002, 2024-04-20T17:41:54.792 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.18323587099999999, 2024-04-20T17:41:54.792 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:41:54.792 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:41:54.792 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:41:54.792 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:41:54.792 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27088", 2024-04-20T17:41:54.792 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:41:54.792 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 176983 2024-04-20T17:41:54.792 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.792 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:41:54.792 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.792 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:41:54.792 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.983813+0000", 2024-04-20T17:41:54.792 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.792 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.793 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.793 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:41:54.793 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.983813+0000", 2024-04-20T17:41:54.793 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.793 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.793 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.793 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:41:54.793 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.983815+0000", 2024-04-20T17:41:54.793 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1469999999999999e-06 2024-04-20T17:41:54.793 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.793 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.793 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:41:54.793 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.983821+0000", 2024-04-20T17:41:54.793 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0909999999999998e-06 2024-04-20T17:41:54.793 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.793 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.794 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:41:54.794 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.983831+0000", 2024-04-20T17:41:54.794 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0387000000000001e-05 2024-04-20T17:41:54.794 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.794 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.794 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:41:54.794 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.983837+0000", 2024-04-20T17:41:54.794 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8510000000000004e-06 2024-04-20T17:41:54.794 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.794 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.794 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:41:54.794 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.983882+0000", 2024-04-20T17:41:54.794 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4527999999999998e-05 2024-04-20T17:41:54.794 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.794 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.795 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:41:54.795 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.983984+0000", 2024-04-20T17:41:54.795 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010283199999999999 2024-04-20T17:41:54.795 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.795 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.795 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:41:54.795 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.984062+0000", 2024-04-20T17:41:54.795 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7550000000000001e-05 2024-04-20T17:41:54.795 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.795 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.795 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:41:54.795 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.987383+0000", 2024-04-20T17:41:54.795 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0033208880000000001 2024-04-20T17:41:54.795 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.795 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.796 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:41:54.796 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.166999+0000", 2024-04-20T17:41:54.796 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.17961604 2024-04-20T17:41:54.796 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.796 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.796 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:41:54.796 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.167026+0000", 2024-04-20T17:41:54.796 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7203999999999999e-05 2024-04-20T17:41:54.796 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.796 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.796 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:41:54.796 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.167049+0000", 2024-04-20T17:41:54.796 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2353000000000001e-05 2024-04-20T17:41:54.796 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.796 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:41:54.797 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.797 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.797 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.797 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27088.0:177001 7.1 7:8fadefcc:::benchmark_data_smithi171_161871_object31275:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:41:54.797 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:40:46.005497+0000", 2024-04-20T17:41:54.797 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 68.689834575999996, 2024-04-20T17:41:54.797 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.16162363699999999, 2024-04-20T17:41:54.797 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:41:54.797 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:41:54.797 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:41:54.797 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:41:54.797 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27088", 2024-04-20T17:41:54.797 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:41:54.797 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 177001 2024-04-20T17:41:54.798 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.798 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:41:54.798 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.798 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:41:54.798 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.005497+0000", 2024-04-20T17:41:54.798 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.798 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.798 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.798 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:41:54.798 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.005497+0000", 2024-04-20T17:41:54.798 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.798 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.798 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.798 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:41:54.798 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.005500+0000", 2024-04-20T17:41:54.799 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5900000000000002e-06 2024-04-20T17:41:54.799 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.799 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.799 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:41:54.799 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.005507+0000", 2024-04-20T17:41:54.799 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7499999999999997e-06 2024-04-20T17:41:54.799 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.799 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.799 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:41:54.799 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.005523+0000", 2024-04-20T17:41:54.799 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6824e-05 2024-04-20T17:41:54.799 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.799 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.799 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:41:54.800 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.005531+0000", 2024-04-20T17:41:54.800 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.255e-06 2024-04-20T17:41:54.800 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.800 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.800 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:41:54.800 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.005579+0000", 2024-04-20T17:41:54.800 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7883000000000002e-05 2024-04-20T17:41:54.800 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.800 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.800 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:41:54.800 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.005676+0000", 2024-04-20T17:41:54.800 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7600000000000001e-05 2024-04-20T17:41:54.800 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.800 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.800 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:41:54.801 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.005757+0000", 2024-04-20T17:41:54.801 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0865000000000006e-05 2024-04-20T17:41:54.801 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.801 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.801 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:41:54.801 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.009145+0000", 2024-04-20T17:41:54.801 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0033883749999999999 2024-04-20T17:41:54.801 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.801 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.801 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:41:54.801 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.167071+0000", 2024-04-20T17:41:54.801 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.157925866 2024-04-20T17:41:54.801 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.801 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.801 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:41:54.802 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.167100+0000", 2024-04-20T17:41:54.802 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8238999999999998e-05 2024-04-20T17:41:54.802 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.802 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.802 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:41:54.802 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.167121+0000", 2024-04-20T17:41:54.802 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.139e-05 2024-04-20T17:41:54.802 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.802 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:41:54.802 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.802 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.802 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.802 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27088.0:177014 7.1 7:8bbdf0df:::benchmark_data_smithi171_161871_object31288:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:41:54.802 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:40:46.021123+0000", 2024-04-20T17:41:54.803 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 68.674208848000006, 2024-04-20T17:41:54.803 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.154166049, 2024-04-20T17:41:54.803 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:41:54.803 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:41:54.803 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:41:54.803 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:41:54.803 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27088", 2024-04-20T17:41:54.803 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:41:54.803 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 177014 2024-04-20T17:41:54.803 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.803 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:41:54.803 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.803 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:41:54.803 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.021123+0000", 2024-04-20T17:41:54.803 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.804 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.804 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:41:54.804 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.021123+0000", 2024-04-20T17:41:54.804 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.804 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.804 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:41:54.804 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.021127+0000", 2024-04-20T17:41:54.804 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9530000000000001e-06 2024-04-20T17:41:54.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.804 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.804 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:41:54.804 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.021139+0000", 2024-04-20T17:41:54.804 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.205e-05 2024-04-20T17:41:54.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.805 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.805 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:41:54.805 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.021165+0000", 2024-04-20T17:41:54.805 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5936e-05 2024-04-20T17:41:54.805 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.805 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.805 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:41:54.805 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.021182+0000", 2024-04-20T17:41:54.805 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6762999999999999e-05 2024-04-20T17:41:54.805 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.805 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.805 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:41:54.805 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.021296+0000", 2024-04-20T17:41:54.805 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011462400000000001 2024-04-20T17:41:54.806 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.806 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.806 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:41:54.806 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.021466+0000", 2024-04-20T17:41:54.806 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000169464 2024-04-20T17:41:54.806 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.806 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.806 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:41:54.806 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.021541+0000", 2024-04-20T17:41:54.806 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4867e-05 2024-04-20T17:41:54.806 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.806 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.806 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:41:54.806 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.025787+0000", 2024-04-20T17:41:54.806 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0042458729999999998 2024-04-20T17:41:54.807 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.807 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.807 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:41:54.807 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.175231+0000", 2024-04-20T17:41:54.807 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14944402500000001 2024-04-20T17:41:54.807 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.807 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.807 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:41:54.807 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.175260+0000", 2024-04-20T17:41:54.807 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9801000000000001e-05 2024-04-20T17:41:54.807 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.807 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.807 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:41:54.807 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.175289+0000", 2024-04-20T17:41:54.807 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8693000000000001e-05 2024-04-20T17:41:54.808 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.808 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:41:54.808 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.808 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.808 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.808 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27088.0:177042 7.1 7:80ac16f1:::benchmark_data_smithi171_161871_object31316:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:41:54.808 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:40:46.075340+0000", 2024-04-20T17:41:54.808 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 68.619992023999998, 2024-04-20T17:41:54.808 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.104458127, 2024-04-20T17:41:54.808 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:41:54.808 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:41:54.808 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:41:54.808 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:41:54.808 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27088", 2024-04-20T17:41:54.809 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:41:54.809 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 177042 2024-04-20T17:41:54.809 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.809 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:41:54.809 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.809 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:41:54.809 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.075340+0000", 2024-04-20T17:41:54.809 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.809 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.809 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.809 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:41:54.809 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.075340+0000", 2024-04-20T17:41:54.809 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:54.809 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.810 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.810 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:41:54.810 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.075342+0000", 2024-04-20T17:41:54.810 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1090000000000001e-06 2024-04-20T17:41:54.810 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.810 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.810 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:41:54.810 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.075348+0000", 2024-04-20T17:41:54.810 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2709999999999998e-06 2024-04-20T17:41:54.810 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.810 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.810 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:41:54.810 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.075360+0000", 2024-04-20T17:41:54.810 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1542999999999999e-05 2024-04-20T17:41:54.810 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.811 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.811 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:41:54.811 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.075368+0000", 2024-04-20T17:41:54.811 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8970000000000008e-06 2024-04-20T17:41:54.811 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.811 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.811 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:41:54.811 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.075436+0000", 2024-04-20T17:41:54.811 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8497000000000005e-05 2024-04-20T17:41:54.811 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.811 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.811 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:41:54.811 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.075532+0000", 2024-04-20T17:41:54.811 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5890999999999994e-05 2024-04-20T17:41:54.811 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.812 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.812 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:41:54.812 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.075612+0000", 2024-04-20T17:41:54.812 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0389999999999994e-05 2024-04-20T17:41:54.812 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.812 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.812 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:41:54.812 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.079329+0000", 2024-04-20T17:41:54.812 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037162660000000002 2024-04-20T17:41:54.812 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.812 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.812 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:41:54.812 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.179738+0000", 2024-04-20T17:41:54.812 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.100409285 2024-04-20T17:41:54.812 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.812 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.813 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:41:54.813 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.179772+0000", 2024-04-20T17:41:54.813 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4029999999999998e-05 2024-04-20T17:41:54.813 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:54.813 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:54.813 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:41:54.813 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.179798+0000", 2024-04-20T17:41:54.813 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5948e-05 2024-04-20T17:41:54.813 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.813 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:41:54.813 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.813 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:54.813 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:41:54.813 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:41:54.841 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:41:54.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:54.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:41:54.944 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:41:54.976 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:55.000 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:41:55.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:55.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:41:55.110 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:41:55.148 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:55.179 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:41:55.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:55.189 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:41:55.290 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:41:55.321 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:55.343 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:41:55.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:55.348 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:41:55.449 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:41:55.475 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":2015,"stamp":"2024-04-20T17:41:50.533971+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4117955128,"num_objects":62829,"num_object_clones":0,"num_object_copies":125658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62829,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":228802,"num_write_kb":9329211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14130,"num_bytes_recovered":928012960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160895,"ondisk_log_size":160895,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":12404344,"kb_used_data":7808424,"kb_used_omap":217,"kb_used_meta":4595494,"kb_avail":831302024,"statfs":{"total":863955320832,"available":851253272576,"internally_reserved":0,"allocated":7995826176,"data_stored":7992741615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":222986,"internal_metadata":4705786102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-410255360,"num_objects":-6260,"num_object_clones":0,"num_object_copies":-12520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6260,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":19398656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001679"},"pg_stats":[{"pgid":"7.f","version":"752'14315","reported_seq":14540,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-04-20T17:41:45.826819+0000","last_change":"2024-04-20T17:39:40.325470+0000","last_active":"2024-04-20T17:41:45.826819+0000","last_peered":"2024-04-20T17:41:45.826819+0000","last_clean":"2024-04-20T17:41:45.826819+0000","last_became_active":"2024-04-20T17:37:56.835750+0000","last_became_peered":"2024-04-20T17:37:56.835750+0000","last_unstale":"2024-04-20T17:41:45.826819+0000","last_undegraded":"2024-04-20T17:41:45.826819+0000","last_fullsized":"2024-04-20T17:41:45.826819+0000","mapping_epoch":694,"log_start":"704'4300","ondisk_log_start":"704'4300","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9175","last_scrub_stamp":"2024-04-20T17:39:40.325340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:40.325340+0000","objects_scrubbed":9175,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":741,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264437760,"num_objects":4035,"num_object_clones":0,"num_object_copies":8070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14315,"num_write_kb":587200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"752'14014","reported_seq":14290,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-04-20T17:41:46.097542+0000","last_change":"2024-04-20T17:41:02.433089+0000","last_active":"2024-04-20T17:41:46.097542+0000","last_peered":"2024-04-20T17:41:46.097542+0000","last_clean":"2024-04-20T17:41:46.097542+0000","last_became_active":"2024-04-20T17:37:56.833703+0000","last_became_peered":"2024-04-20T17:37:56.833703+0000","last_unstale":"2024-04-20T17:41:46.097542+0000","last_undegraded":"2024-04-20T17:41:46.097542+0000","last_fullsized":"2024-04-20T17:41:46.097542+0000","mapping_epoch":694,"log_start":"704'4000","ondisk_log_start":"704'4000","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"742'11794","last_scrub_stamp":"2024-04-20T17:41:02.432782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:41:02.432782+0000","objects_scrubbed":6149,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:29.683106+0000","scrub_duration":1488,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255983616,"num_objects":3906,"num_object_clones":0,"num_object_copies":7812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14014,"num_write_kb":573440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"752'16493","reported_seq":16822,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-04-20T17:41:45.821319+0000","last_change":"2024-04-20T17:39:48.311967+0000","last_active":"2024-04-20T17:41:45.821319+0000","last_peered":"2024-04-20T17:41:45.821319+0000","last_clean":"2024-04-20T17:41:45.821319+0000","last_became_active":"2024-04-20T17:37:56.836496+0000","last_became_peered":"2024-04-20T17:37:56.836496+0000","last_unstale":"2024-04-20T17:41:45.821319+0000","last_undegraded":"2024-04-20T17:41:45.821319+0000","last_fullsized":"2024-04-20T17:41:45.821319+0000","mapping_epoch":694,"log_start":"712'6400","ondisk_log_start":"712'6400","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'11204","last_scrub_stamp":"2024-04-20T17:39:48.311825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:48.311825+0000","objects_scrubbed":9436,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271843352,"num_objects":4149,"num_object_clones":0,"num_object_copies":8298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4149,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14725,"num_write_kb":603905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.c","version":"753'14388","reported_seq":14614,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-04-20T17:41:49.295946+0000","last_change":"2024-04-20T17:39:44.683378+0000","last_active":"2024-04-20T17:41:49.295946+0000","last_peered":"2024-04-20T17:41:49.295946+0000","last_clean":"2024-04-20T17:41:49.295946+0000","last_became_active":"2024-04-20T17:37:56.834181+0000","last_became_peered":"2024-04-20T17:37:56.834181+0000","last_unstale":"2024-04-20T17:41:49.295946+0000","last_undegraded":"2024-04-20T17:41:49.295946+0000","last_fullsized":"2024-04-20T17:41:49.295946+0000","mapping_epoch":694,"log_start":"704'4300","ondisk_log_start":"704'4300","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9063","last_scrub_stamp":"2024-04-20T17:39:44.683201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:44.683201+0000","objects_scrubbed":9063,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":746,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244973568,"num_objects":3738,"num_object_clones":0,"num_object_copies":7476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14388,"num_write_kb":580032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"753'14481","reported_seq":14784,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-04-20T17:41:49.041977+0000","last_change":"2024-04-20T17:40:58.937747+0000","last_active":"2024-04-20T17:41:49.041977+0000","last_peered":"2024-04-20T17:41:49.041977+0000","last_clean":"2024-04-20T17:41:49.041977+0000","last_became_active":"2024-04-20T17:37:56.830487+0000","last_became_peered":"2024-04-20T17:37:56.830487+0000","last_unstale":"2024-04-20T17:41:49.041977+0000","last_undegraded":"2024-04-20T17:41:49.041977+0000","last_fullsized":"2024-04-20T17:41:49.041977+0000","mapping_epoch":694,"log_start":"705'4400","ondisk_log_start":"705'4400","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"742'11757","last_scrub_stamp":"2024-04-20T17:40:58.937604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:40:58.937604+0000","objects_scrubbed":6664,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:28.789538+0000","scrub_duration":858,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257359872,"num_objects":3927,"num_object_clones":0,"num_object_copies":7854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14481,"num_write_kb":589056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.a","version":"752'14045","reported_seq":14337,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-04-20T17:41:45.867928+0000","last_change":"2024-04-20T17:41:06.536176+0000","last_active":"2024-04-20T17:41:45.867928+0000","last_peered":"2024-04-20T17:41:45.867928+0000","last_clean":"2024-04-20T17:41:45.867928+0000","last_became_active":"2024-04-20T17:37:56.835944+0000","last_became_peered":"2024-04-20T17:37:56.835944+0000","last_unstale":"2024-04-20T17:41:45.867928+0000","last_undegraded":"2024-04-20T17:41:45.867928+0000","last_fullsized":"2024-04-20T17:41:45.867928+0000","mapping_epoch":694,"log_start":"704'4000","ondisk_log_start":"704'4000","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"742'11924","last_scrub_stamp":"2024-04-20T17:41:06.536039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:41:06.536039+0000","objects_scrubbed":6077,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:18.392067+0000","scrub_duration":857,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258146304,"num_objects":3939,"num_object_clones":0,"num_object_copies":7878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14045,"num_write_kb":575488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"752'14210","reported_seq":14417,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-04-20T17:41:45.781761+0000","last_change":"2024-04-20T17:39:03.675435+0000","last_active":"2024-04-20T17:41:45.781761+0000","last_peered":"2024-04-20T17:41:45.781761+0000","last_clean":"2024-04-20T17:41:45.781761+0000","last_became_active":"2024-04-20T17:37:56.835700+0000","last_became_peered":"2024-04-20T17:37:56.835700+0000","last_unstale":"2024-04-20T17:41:45.781761+0000","last_undegraded":"2024-04-20T17:41:45.781761+0000","last_fullsized":"2024-04-20T17:41:45.781761+0000","mapping_epoch":694,"log_start":"704'4200","ondisk_log_start":"704'4200","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'7175","last_scrub_stamp":"2024-04-20T17:39:03.675211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:03.675211+0000","objects_scrubbed":7133,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256901120,"num_objects":3920,"num_object_clones":0,"num_object_copies":7840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14210,"num_write_kb":580160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11],"acting":[0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"752'17724","reported_seq":36312,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-04-20T17:41:46.127381+0000","last_change":"2024-04-20T17:40:51.537130+0000","last_active":"2024-04-20T17:41:46.127381+0000","last_peered":"2024-04-20T17:41:46.127381+0000","last_clean":"2024-04-20T17:41:46.127381+0000","last_became_active":"2024-04-20T17:40:09.546695+0000","last_became_peered":"2024-04-20T17:40:09.546695+0000","last_unstale":"2024-04-20T17:41:46.127381+0000","last_undegraded":"2024-04-20T17:41:46.127381+0000","last_fullsized":"2024-04-20T17:41:46.127381+0000","mapping_epoch":731,"log_start":"714'7701","ondisk_log_start":"714'7701","created":694,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'14850","last_scrub_stamp":"2024-04-20T17:40:51.536812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:40:51.536812+0000","objects_scrubbed":6890,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:41:57.907978+0000","scrub_duration":1411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260964352,"num_objects":3982,"num_object_clones":0,"num_object_copies":7964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14154,"num_write_kb":580352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9062,"num_bytes_recovered":593887232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"753'14522","reported_seq":14810,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-04-20T17:41:48.722985+0000","last_change":"2024-04-20T17:40:48.024415+0000","last_active":"2024-04-20T17:41:48.722985+0000","last_peered":"2024-04-20T17:41:48.722985+0000","last_clean":"2024-04-20T17:41:48.722985+0000","last_became_active":"2024-04-20T17:37:56.836665+0000","last_became_peered":"2024-04-20T17:37:56.836665+0000","last_unstale":"2024-04-20T17:41:48.722985+0000","last_undegraded":"2024-04-20T17:41:48.722985+0000","last_fullsized":"2024-04-20T17:41:48.722985+0000","mapping_epoch":694,"log_start":"705'4500","ondisk_log_start":"705'4500","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'11243","last_scrub_stamp":"2024-04-20T17:40:48.024149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:40:48.024149+0000","objects_scrubbed":7122,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:03.517694+0000","scrub_duration":1322,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249430016,"num_objects":3806,"num_object_clones":0,"num_object_copies":7612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14522,"num_write_kb":586496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.5","version":"753'14442","reported_seq":14727,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-04-20T17:41:49.027849+0000","last_change":"2024-04-20T17:40:54.143175+0000","last_active":"2024-04-20T17:41:49.027849+0000","last_peered":"2024-04-20T17:41:49.027849+0000","last_clean":"2024-04-20T17:41:49.027849+0000","last_became_active":"2024-04-20T17:37:56.833600+0000","last_became_peered":"2024-04-20T17:37:56.833600+0000","last_unstale":"2024-04-20T17:41:49.027849+0000","last_undegraded":"2024-04-20T17:41:49.027849+0000","last_fullsized":"2024-04-20T17:41:49.027849+0000","mapping_epoch":694,"log_start":"705'4400","ondisk_log_start":"705'4400","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'11512","last_scrub_stamp":"2024-04-20T17:40:54.143076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:40:54.143076+0000","objects_scrubbed":6822,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:02.493494+0000","scrub_duration":950,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253755392,"num_objects":3872,"num_object_clones":0,"num_object_copies":7744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14442,"num_write_kb":586048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.4","version":"752'14066","reported_seq":14290,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-04-20T17:41:45.828804+0000","last_change":"2024-04-20T17:39:50.365478+0000","last_active":"2024-04-20T17:41:45.828804+0000","last_peered":"2024-04-20T17:41:45.828804+0000","last_clean":"2024-04-20T17:41:45.828804+0000","last_became_active":"2024-04-20T17:37:56.834727+0000","last_became_peered":"2024-04-20T17:37:56.834727+0000","last_unstale":"2024-04-20T17:41:45.828804+0000","last_undegraded":"2024-04-20T17:41:45.828804+0000","last_fullsized":"2024-04-20T17:41:45.828804+0000","mapping_epoch":694,"log_start":"704'4000","ondisk_log_start":"704'4000","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9031","last_scrub_stamp":"2024-04-20T17:39:50.365339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:50.365339+0000","objects_scrubbed":9031,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":876,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261881856,"num_objects":3996,"num_object_clones":0,"num_object_copies":7992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14066,"num_write_kb":577984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"753'14271","reported_seq":14476,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-04-20T17:41:49.244035+0000","last_change":"2024-04-20T17:39:00.502710+0000","last_active":"2024-04-20T17:41:49.244035+0000","last_peered":"2024-04-20T17:41:49.244035+0000","last_clean":"2024-04-20T17:41:49.244035+0000","last_became_active":"2024-04-20T17:37:56.832911+0000","last_became_peered":"2024-04-20T17:37:56.832911+0000","last_unstale":"2024-04-20T17:41:49.244035+0000","last_undegraded":"2024-04-20T17:41:49.244035+0000","last_fullsized":"2024-04-20T17:41:49.244035+0000","mapping_epoch":694,"log_start":"704'4200","ondisk_log_start":"704'4200","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'6831","last_scrub_stamp":"2024-04-20T17:39:00.502556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:00.502556+0000","objects_scrubbed":6782,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249102336,"num_objects":3801,"num_object_clones":0,"num_object_copies":7602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14271,"num_write_kb":578304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"753'14650","reported_seq":14861,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-04-20T17:41:49.279563+0000","last_change":"2024-04-20T17:39:08.489529+0000","last_active":"2024-04-20T17:41:49.279563+0000","last_peered":"2024-04-20T17:41:49.279563+0000","last_clean":"2024-04-20T17:41:49.279563+0000","last_became_active":"2024-04-20T17:37:56.834292+0000","last_became_peered":"2024-04-20T17:37:56.834292+0000","last_unstale":"2024-04-20T17:41:49.279563+0000","last_undegraded":"2024-04-20T17:41:49.279563+0000","last_fullsized":"2024-04-20T17:41:49.279563+0000","mapping_epoch":694,"log_start":"705'4600","ondisk_log_start":"705'4600","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"716'7577","last_scrub_stamp":"2024-04-20T17:39:08.489308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:08.489308+0000","objects_scrubbed":7524,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251789312,"num_objects":3842,"num_object_clones":0,"num_object_copies":7684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14650,"num_write_kb":591744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"752'14159","reported_seq":14451,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-04-20T17:41:45.597607+0000","last_change":"2024-04-20T17:41:07.795076+0000","last_active":"2024-04-20T17:41:45.597607+0000","last_peered":"2024-04-20T17:41:45.597607+0000","last_clean":"2024-04-20T17:41:45.597607+0000","last_became_active":"2024-04-20T17:37:56.835512+0000","last_became_peered":"2024-04-20T17:37:56.835512+0000","last_unstale":"2024-04-20T17:41:45.597607+0000","last_undegraded":"2024-04-20T17:41:45.597607+0000","last_fullsized":"2024-04-20T17:41:45.597607+0000","mapping_epoch":694,"log_start":"704'4100","ondisk_log_start":"704'4100","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"743'12166","last_scrub_stamp":"2024-04-20T17:41:07.794973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:41:07.794973+0000","objects_scrubbed":6033,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:12.046222+0000","scrub_duration":644,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263651328,"num_objects":4023,"num_object_clones":0,"num_object_copies":8046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14159,"num_write_kb":581824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"752'14119","reported_seq":14407,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-04-20T17:41:45.930915+0000","last_change":"2024-04-20T17:41:16.382261+0000","last_active":"2024-04-20T17:41:45.930915+0000","last_peered":"2024-04-20T17:41:45.930915+0000","last_clean":"2024-04-20T17:41:45.930915+0000","last_became_active":"2024-04-20T17:37:56.833972+0000","last_became_peered":"2024-04-20T17:37:56.833972+0000","last_unstale":"2024-04-20T17:41:45.930915+0000","last_undegraded":"2024-04-20T17:41:45.930915+0000","last_fullsized":"2024-04-20T17:41:45.930915+0000","mapping_epoch":694,"log_start":"704'4100","ondisk_log_start":"704'4100","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"745'12483","last_scrub_stamp":"2024-04-20T17:41:16.382018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:41:16.382018+0000","objects_scrubbed":5596,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:31.605235+0000","scrub_duration":653,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258539520,"num_objects":3945,"num_object_clones":0,"num_object_copies":7890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14119,"num_write_kb":578048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.6","version":"752'14060","reported_seq":24384,"reported_epoch":752,"state":"active+recovering+degraded","last_fresh":"2024-04-20T17:41:45.858515+0000","last_change":"2024-04-20T17:39:54.962650+0000","last_active":"2024-04-20T17:41:45.858515+0000","last_peered":"2024-04-20T17:41:45.858515+0000","last_clean":"2024-04-20T17:39:34.507147+0000","last_became_active":"2024-04-20T17:39:54.933393+0000","last_became_peered":"2024-04-20T17:39:54.933393+0000","last_unstale":"2024-04-20T17:41:45.858515+0000","last_undegraded":"2024-04-20T17:39:54.578981+0000","last_fullsized":"2024-04-20T17:41:45.858515+0000","mapping_epoch":727,"log_start":"704'4000","ondisk_log_start":"704'4000","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'7141","last_scrub_stamp":"2024-04-20T17:39:04.435913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:04.435913+0000","objects_scrubbed":7101,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258605056,"num_objects":3946,"num_object_clones":0,"num_object_copies":7892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14060,"num_write_kb":576192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5058,"num_bytes_recovered":331481088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":3937},{"shards":"0,3","objects":9}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":518,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-04-20T17:41:48.700241+0000","last_change":"2024-04-20T17:40:50.375929+0000","last_active":"2024-04-20T17:41:48.700241+0000","last_peered":"2024-04-20T17:41:48.700241+0000","last_clean":"2024-04-20T17:41:48.700241+0000","last_became_active":"2024-04-20T17:39:54.970208+0000","last_became_peered":"2024-04-20T17:39:54.970208+0000","last_unstale":"2024-04-20T17:41:48.700241+0000","last_undegraded":"2024-04-20T17:41:48.700241+0000","last_fullsized":"2024-04-20T17:41:48.700241+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:40:50.375828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:40:50.375828+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:14.500087+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4117364760,"num_objects":62827,"num_object_clones":0,"num_object_copies":125654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62827,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":228621,"num_write_kb":9326273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14120,"num_bytes_recovered":925368320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9118818304,"data_stored":9118810160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160758,"ondisk_log_size":160758,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991666,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1689388,"kb_used_data":982112,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":92055764,"statfs":{"total":95995035648,"available":94265102336,"internally_reserved":0,"allocated":1005682688,"data_stored":1005334083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":724236740},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253305,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1276260,"kb_used_data":731416,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":92468892,"statfs":{"total":95995035648,"available":94688145408,"internally_reserved":0,"allocated":748969984,"data_stored":748624931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":557906368},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613053,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1332828,"kb_used_data":769940,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":92412324,"statfs":{"total":95995035648,"available":94630219776,"internally_reserved":0,"allocated":788418560,"data_stored":788075979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":576387520},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809284,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":795604,"kb_used_data":493708,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92949548,"statfs":{"total":95995035648,"available":95180337152,"internally_reserved":0,"allocated":505556992,"data_stored":505217963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,2,3,4,5,6,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":2297807503514,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":2830383448137,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738858,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1928792,"kb_used_data":1283984,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":91816360,"statfs":{"total":95995035648,"available":94019952640,"internally_reserved":0,"allocated":1314799616,"data_stored":1314456515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394148,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1330836,"kb_used_data":1011724,"kb_used_omap":202,"kb_used_meta":318901,"kb_avail":92414316,"statfs":{"total":95995035648,"available":94632259584,"internally_reserved":0,"allocated":1036005376,"data_stored":1035666347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":326555095},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653198,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1613800,"kb_used_data":1234332,"kb_used_omap":4,"kb_used_meta":379451,"kb_avail":92131352,"statfs":{"total":95995035648,"available":94342504448,"internally_reserved":0,"allocated":1263955968,"data_stored":1263606863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":388558535},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986646,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1264532,"kb_used_data":777996,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":92480620,"statfs":{"total":95995035648,"available":94700154880,"internally_reserved":0,"allocated":796667904,"data_stored":796328875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790416,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1172304,"kb_used_data":523212,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92572848,"statfs":{"total":95995035648,"available":94794596352,"internally_reserved":0,"allocated":535769088,"data_stored":535430059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1313673216,"data_stored":1313669144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1034027008,"data_stored":1034027008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1252855808,"data_stored":1252851736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":789250048,"data_stored":789250048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":528875520,"data_stored":528875520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1167458304,"data_stored":1167458304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":503447552,"data_stored":503447552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":783613952,"data_stored":783613952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":743047168,"data_stored":743047168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1002569728,"data_stored":1002569728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:41:55.476 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:41:55.479 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:55.505 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-20T17:41:55.510 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:41:55.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:55.519 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:41:55.620 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:41:55.652 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:55.677 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:41:55.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:55.683 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:41:55.697 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:55 smithi005 ceph-mon[39562]: pgmap v2017: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.6 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 906 op/s; 3652/116694 objects degraded (3.130%); 3.6 MiB/s, 58 objects/s recovering 2024-04-20T17:41:55.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:55 smithi106 ceph-mon[44423]: pgmap v2017: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.6 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 906 op/s; 3652/116694 objects degraded (3.130%); 3.6 MiB/s, 58 objects/s recovering 2024-04-20T17:41:55.783 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:41:55.792 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:41:55.809 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:55 smithi171 ceph-mon[44189]: pgmap v2017: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.6 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 0 B/s wr, 906 op/s; 3652/116694 objects degraded (3.130%); 3.6 MiB/s, 58 objects/s recovering 2024-04-20T17:41:55.811 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:55.835 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:41:55.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:55.838 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:41:55.939 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:41:55.972 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:56.003 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:41:56.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:56.005 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:41:56.106 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:41:56.171 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:56.916 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:41:56.952 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:41:56.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:56.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:41:56.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:56 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 3652/116694 objects degraded (3.130%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:56 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 3652/116694 objects degraded (3.130%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:57.058 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:41:57.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:56 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 3652/116694 objects degraded (3.130%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:41:57.098 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:57.557 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:41:57.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:57.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:41:57.676 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:41:57.696 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:41:57.696 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:41:57.696 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:57.696 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27088.0:237994 7.b e753/694 7:dc9801a9:::benchmark_data_smithi171_161871_object92268:head v 753'14954, mlcod=753'14952)", 2024-04-20T17:41:57.696 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:41:57.694163+0000", 2024-04-20T17:41:57.696 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0016419819999999999, 2024-04-20T17:41:57.696 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0016676589999999999, 2024-04-20T17:41:57.696 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:41:57.696 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:41:57.697 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-04-20T17:41:57.697 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:41:57.697 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:57.697 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:41:57.697 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:41:57.694163+0000", 2024-04-20T17:41:57.697 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:57.697 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:57.697 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:57.697 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:41:57.697 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:41:57.694163+0000", 2024-04-20T17:41:57.697 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:41:57.697 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:57.697 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:57.697 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:41:57.698 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:41:57.694164+0000", 2024-04-20T17:41:57.698 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5719999999999999e-06 2024-04-20T17:41:57.698 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:57.698 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:57.698 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:41:57.698 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:41:57.694170+0000", 2024-04-20T17:41:57.698 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7690000000000002e-06 2024-04-20T17:41:57.698 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:57.698 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:57.698 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:41:57.698 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:41:57.694179+0000", 2024-04-20T17:41:57.698 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1330000000000005e-06 2024-04-20T17:41:57.698 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:57.698 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:57.698 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:41:57.699 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:41:57.694187+0000", 2024-04-20T17:41:57.699 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8350000000000004e-06 2024-04-20T17:41:57.699 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:57.699 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:57.699 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:41:57.699 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:41:57.694229+0000", 2024-04-20T17:41:57.699 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2352999999999999e-05 2024-04-20T17:41:57.699 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:41:57.699 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:41:57.699 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:41:57.699 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:41:57.694243+0000", 2024-04-20T17:41:57.699 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3594999999999999e-05 2024-04-20T17:41:57.699 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:57.699 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:41:57.699 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:57.699 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:41:57.700 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-04-20T17:41:57.700 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 1 2024-04-20T17:41:57.700 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:41:57.723 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:57 smithi106 ceph-mon[44423]: pgmap v2018: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.4 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 906 op/s; 3374/111056 objects degraded (3.038%); 3.5 MiB/s, 56 objects/s recovering 2024-04-20T17:41:57.752 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:57.865 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:41:57.870 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:57 smithi005 ceph-mon[39562]: pgmap v2018: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.4 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 906 op/s; 3374/111056 objects degraded (3.038%); 3.5 MiB/s, 56 objects/s recovering 2024-04-20T17:41:57.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:57.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:41:57.972 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:41:58.007 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:57 smithi171 ceph-mon[44189]: pgmap v2018: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.4 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 906 op/s; 3374/111056 objects degraded (3.038%); 3.5 MiB/s, 56 objects/s recovering 2024-04-20T17:41:58.009 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:58.033 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:41:58.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:58.038 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:41:58.139 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:41:58.170 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:58.250 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:41:58.644 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:41:58.663 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-20T17:41:58.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:58 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3795260629' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:41:58.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:58 smithi005 ceph-mon[39562]: from='client.27313 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:41:58.721 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:41:58.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:58.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:41:58.825 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:41:58.854 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:58.875 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:41:58.876 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:58 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3795260629' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:41:58.876 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:58 smithi106 ceph-mon[44423]: from='client.27313 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:41:58.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:58.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:41:58.979 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:41:58.990 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:41:59.034 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:59.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:58 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3795260629' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:41:59.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:58 smithi171 ceph-mon[44189]: from='client.27313 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:41:59.089 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:41:59.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:59.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:41:59.205 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:41:59.234 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:59.308 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:41:59.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:59.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:41:59.413 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:41:59.455 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:59.487 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:41:59.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:59.490 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:41:59.591 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:41:59.622 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:41:59.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:59 smithi171 ceph-mon[44189]: pgmap v2019: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.4 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 906 op/s; 3374/111056 objects degraded (3.038%); 3.5 MiB/s, 56 objects/s recovering 2024-04-20T17:41:59.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:59 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:41:59.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:59 smithi171 ceph-mon[44189]: from='client.27313 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:41:59.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:41:59 smithi171 ceph-mon[44189]: osdmap e754: 12 total, 11 up, 9 in 2024-04-20T17:41:59.870 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:41:59.872 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:59 smithi106 ceph-mon[44423]: pgmap v2019: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.4 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 906 op/s; 3374/111056 objects degraded (3.038%); 3.5 MiB/s, 56 objects/s recovering 2024-04-20T17:41:59.872 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:59 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:41:59.872 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:59 smithi106 ceph-mon[44423]: from='client.27313 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:41:59.872 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:41:59 smithi106 ceph-mon[44423]: osdmap e754: 12 total, 11 up, 9 in 2024-04-20T17:41:59.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:41:59.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:41:59.967 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:59 smithi005 ceph-mon[39562]: pgmap v2019: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.4 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 906 op/s; 3374/111056 objects degraded (3.038%); 3.5 MiB/s, 56 objects/s recovering 2024-04-20T17:41:59.967 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:59 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:41:59.968 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:59 smithi005 ceph-mon[39562]: from='client.27313 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:41:59.968 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:41:59 smithi005 ceph-mon[39562]: osdmap e754: 12 total, 11 up, 9 in 2024-04-20T17:41:59.974 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:42:00.002 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:00.451 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:42:00.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:00.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:42:00.557 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:42:00.621 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:00.751 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:42:00.797 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:42:00.843 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:42:00.857 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:42:00.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:00.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:42:00.886 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:42:00.915 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:42:00.958 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:42:00.965 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:42:00.996 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:42:01.035 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:42:01.052 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:01.056 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:42:01.056 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:42:01.073 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:42:01.116 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:42:01.170 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:42:01.170 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:42:01.170 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:42:01.170 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:42:01.171 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:42:01.171 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:42:01.171 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:42:01.171 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:42:01.171 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:42:01.171 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:42:01.171 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:42:01.171 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:42:01.171 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:42:01.686 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:42:01.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:01 smithi106 ceph-mon[44423]: pgmap v2021: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 896 op/s; 3374/107720 objects degraded (3.132%); 3.5 MiB/s, 56 objects/s recovering 2024-04-20T17:42:01.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:01 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3968226832' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:42:01.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:01.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:01.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:01.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:01.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:01.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:01.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:01 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 3374/107720 objects degraded (3.132%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:01.825 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-20T17:42:01.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:01 smithi005 ceph-mon[39562]: pgmap v2021: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 896 op/s; 3374/107720 objects degraded (3.132%); 3.5 MiB/s, 56 objects/s recovering 2024-04-20T17:42:01.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:01 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3968226832' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:42:01.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:01.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:01.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:01.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:01.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:01.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:01.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:01 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 3374/107720 objects degraded (3.132%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:01 smithi171 ceph-mon[44189]: pgmap v2021: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 896 op/s; 3374/107720 objects degraded (3.132%); 3.5 MiB/s, 56 objects/s recovering 2024-04-20T17:42:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:01 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3968226832' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:42:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:02.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:02.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:02.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:01 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 3374/107720 objects degraded (3.132%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:02.109 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:42:02.647 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:42:02.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:02.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:42:02.753 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:42:02.789 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:02.816 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:42:02.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:02.820 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:42:02.897 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:02.898 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:02.898 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:02.898 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:02.898 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:02.921 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:42:02.958 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:02.958 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:02.958 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:02.958 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:02.958 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:02.963 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:02.988 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:42:02.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:02.993 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:42:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:03.094 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:42:03.123 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:03.572 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:42:03.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:03.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:42:03.660 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:03 smithi005 ceph-mon[39562]: 7.8 scrub starts 2024-04-20T17:42:03.660 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:03 smithi005 ceph-mon[39562]: pgmap v2022: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.1 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 899 op/s; 3103/102076 objects degraded (3.040%); 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:42:03.660 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:03 smithi005 ceph-mon[39562]: 7.8 scrub ok 2024-04-20T17:42:03.660 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:03 smithi005 ceph-mon[39562]: 7.5 scrub starts 2024-04-20T17:42:03.675 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:42:03.704 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:03.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:03 smithi106 ceph-mon[44423]: 7.8 scrub starts 2024-04-20T17:42:03.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:03 smithi106 ceph-mon[44423]: pgmap v2022: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.1 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 899 op/s; 3103/102076 objects degraded (3.040%); 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:42:03.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:03 smithi106 ceph-mon[44423]: 7.8 scrub ok 2024-04-20T17:42:03.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:03 smithi106 ceph-mon[44423]: 7.5 scrub starts 2024-04-20T17:42:03.790 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:42:03.790 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:42:03.856 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:42:03.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:03.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:42:03.962 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:42:04.032 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:03 smithi171 ceph-mon[44189]: 7.8 scrub starts 2024-04-20T17:42:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:03 smithi171 ceph-mon[44189]: pgmap v2022: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 3.1 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 899 op/s; 3103/102076 objects degraded (3.040%); 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:42:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:03 smithi171 ceph-mon[44189]: 7.8 scrub ok 2024-04-20T17:42:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:03 smithi171 ceph-mon[44189]: 7.5 scrub starts 2024-04-20T17:42:04.099 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:42:04.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:04.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:42:04.149 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:42:04.149 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:42:04.205 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:42:04.265 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:04.385 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:42:04.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:04.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:42:04.389 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:42:04.489 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:42:04.643 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:04.671 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:42:04.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:04.674 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:42:04.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:04 smithi005 ceph-mon[39562]: 7.5 scrub ok 2024-04-20T17:42:04.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:04 smithi005 ceph-mon[39562]: 7.7 scrub starts 2024-04-20T17:42:04.775 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:42:04.849 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:04.910 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:42:04.910 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:42:04.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:04.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:42:04.922 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:42:04.976 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:04 smithi106 ceph-mon[44423]: 7.5 scrub ok 2024-04-20T17:42:04.976 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:04 smithi106 ceph-mon[44423]: 7.7 scrub starts 2024-04-20T17:42:05.006 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-20T17:42:05.014 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:42:05.034 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:04 smithi171 ceph-mon[44189]: 7.5 scrub ok 2024-04-20T17:42:05.034 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:04 smithi171 ceph-mon[44189]: 7.7 scrub starts 2024-04-20T17:42:05.080 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:05.552 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:42:05.792 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:42:05.970 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:05 smithi005 ceph-mon[39562]: 7.7 scrub ok 2024-04-20T17:42:05.971 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:05 smithi005 ceph-mon[39562]: pgmap v2023: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded, 15 active+clean; 3.0 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 899 op/s; 3103/98696 objects degraded (3.144%); 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:42:05.971 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:05 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:42:05.971 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:05 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:42:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:05 smithi106 ceph-mon[44423]: 7.7 scrub ok 2024-04-20T17:42:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:05 smithi106 ceph-mon[44423]: pgmap v2023: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded, 15 active+clean; 3.0 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 899 op/s; 3103/98696 objects degraded (3.144%); 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:42:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:05 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:42:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:05 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:42:06.037 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:05 smithi171 ceph-mon[44189]: 7.7 scrub ok 2024-04-20T17:42:06.037 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:05 smithi171 ceph-mon[44189]: pgmap v2023: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded, 15 active+clean; 3.0 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 899 op/s; 3103/98696 objects degraded (3.144%); 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:42:06.037 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:05 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:42:06.037 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:05 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:42:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:06 smithi106 ceph-mon[44423]: from='client.27319 v1:172.21.15.5:0/2245280453' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:42:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:06 smithi106 ceph-mon[44423]: osdmap e755: 12 total, 11 up, 9 in 2024-04-20T17:42:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:06 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 3103/98696 objects degraded (3.144%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:06 smithi171 ceph-mon[44189]: from='client.27319 v1:172.21.15.5:0/2245280453' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:42:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:06 smithi171 ceph-mon[44189]: osdmap e755: 12 total, 11 up, 9 in 2024-04-20T17:42:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:06 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 3103/98696 objects degraded (3.144%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:07.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:06 smithi005 ceph-mon[39562]: from='client.27319 v1:172.21.15.5:0/2245280453' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:42:07.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:06 smithi005 ceph-mon[39562]: osdmap e755: 12 total, 11 up, 9 in 2024-04-20T17:42:07.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:06 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 3103/98696 objects degraded (3.144%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:07.824 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:42:07.824 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:42:07.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:07 smithi171 ceph-mon[44189]: pgmap v2025: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 2.8 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s; 2814/93140 objects degraded (3.021%); 4.4 MiB/s, 69 objects/s recovering 2024-04-20T17:42:07.960 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:07 smithi005 ceph-mon[39562]: pgmap v2025: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 2.8 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s; 2814/93140 objects degraded (3.021%); 4.4 MiB/s, 69 objects/s recovering 2024-04-20T17:42:07.975 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:42:07.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:07.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:42:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:07 smithi106 ceph-mon[44423]: pgmap v2025: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 2.8 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.12k op/s; 2814/93140 objects degraded (3.021%); 4.4 MiB/s, 69 objects/s recovering 2024-04-20T17:42:08.088 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:42:08.158 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:08.584 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-20T17:42:08.599 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:42:08.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:08.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:42:08.651 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":2023,"stamp":"2024-04-20T17:42:04.545796+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3234464312,"num_objects":49348,"num_object_clones":0,"num_object_copies":98696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49348,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":242283,"num_write_kb":9329211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14964,"num_bytes_recovered":982669984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160776,"ondisk_log_size":160776,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":10783028,"kb_used_data":6203356,"kb_used_omap":217,"kb_used_meta":4578854,"kb_avail":832923340,"statfs":{"total":863955320832,"available":852913500160,"internally_reserved":0,"allocated":6352236544,"data_stored":6349061473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":222986,"internal_metadata":4688746742},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,0,2,4,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-589758464,"num_objects":-8999,"num_object_clones":0,"num_object_copies":-17998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8999,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":549,"num_bytes_recovered":35979264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.009043"},"pg_stats":[{"pgid":"7.f","version":"754'15183","reported_seq":15412,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-04-20T17:42:00.840691+0000","last_change":"2024-04-20T17:39:40.325470+0000","last_active":"2024-04-20T17:42:00.840691+0000","last_peered":"2024-04-20T17:42:00.840691+0000","last_clean":"2024-04-20T17:42:00.840691+0000","last_became_active":"2024-04-20T17:37:56.835750+0000","last_became_peered":"2024-04-20T17:37:56.835750+0000","last_unstale":"2024-04-20T17:42:00.840691+0000","last_undegraded":"2024-04-20T17:42:00.840691+0000","last_fullsized":"2024-04-20T17:42:00.840691+0000","mapping_epoch":694,"log_start":"707'5100","ondisk_log_start":"707'5100","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9175","last_scrub_stamp":"2024-04-20T17:39:40.325340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:40.325340+0000","objects_scrubbed":9175,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":741,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207552512,"num_objects":3167,"num_object_clones":0,"num_object_copies":6334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15183,"num_write_kb":587200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"754'14821","reported_seq":15101,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-04-20T17:42:01.163330+0000","last_change":"2024-04-20T17:41:02.433089+0000","last_active":"2024-04-20T17:42:01.163330+0000","last_peered":"2024-04-20T17:42:01.163330+0000","last_clean":"2024-04-20T17:42:01.163330+0000","last_became_active":"2024-04-20T17:37:56.833703+0000","last_became_peered":"2024-04-20T17:37:56.833703+0000","last_unstale":"2024-04-20T17:42:01.163330+0000","last_undegraded":"2024-04-20T17:42:01.163330+0000","last_fullsized":"2024-04-20T17:42:01.163330+0000","mapping_epoch":694,"log_start":"706'4800","ondisk_log_start":"706'4800","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"742'11794","last_scrub_stamp":"2024-04-20T17:41:02.432782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:41:02.432782+0000","objects_scrubbed":6149,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:29.683106+0000","scrub_duration":1488,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203096064,"num_objects":3099,"num_object_clones":0,"num_object_copies":6198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14821,"num_write_kb":573440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"754'17333","reported_seq":17667,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-04-20T17:42:00.874817+0000","last_change":"2024-04-20T17:39:48.311967+0000","last_active":"2024-04-20T17:42:00.874817+0000","last_peered":"2024-04-20T17:42:00.874817+0000","last_clean":"2024-04-20T17:42:00.874817+0000","last_became_active":"2024-04-20T17:37:56.836496+0000","last_became_peered":"2024-04-20T17:37:56.836496+0000","last_unstale":"2024-04-20T17:42:00.874817+0000","last_undegraded":"2024-04-20T17:42:00.874817+0000","last_fullsized":"2024-04-20T17:42:00.874817+0000","mapping_epoch":694,"log_start":"714'7300","ondisk_log_start":"714'7300","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'11204","last_scrub_stamp":"2024-04-20T17:39:48.311825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:48.311825+0000","objects_scrubbed":9436,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216793112,"num_objects":3309,"num_object_clones":0,"num_object_copies":6618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3309,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15565,"num_write_kb":603905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.c","version":"754'15245","reported_seq":15474,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-04-20T17:42:04.312373+0000","last_change":"2024-04-20T17:39:44.683378+0000","last_active":"2024-04-20T17:42:04.312373+0000","last_peered":"2024-04-20T17:42:04.312373+0000","last_clean":"2024-04-20T17:42:04.312373+0000","last_became_active":"2024-04-20T17:37:56.834181+0000","last_became_peered":"2024-04-20T17:37:56.834181+0000","last_unstale":"2024-04-20T17:42:04.312373+0000","last_undegraded":"2024-04-20T17:42:04.312373+0000","last_fullsized":"2024-04-20T17:42:04.312373+0000","mapping_epoch":694,"log_start":"708'5200","ondisk_log_start":"708'5200","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9063","last_scrub_stamp":"2024-04-20T17:39:44.683201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:44.683201+0000","objects_scrubbed":9063,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":746,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188809216,"num_objects":2881,"num_object_clones":0,"num_object_copies":5762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15245,"num_write_kb":580032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"754'15307","reported_seq":15612,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-04-20T17:42:04.067557+0000","last_change":"2024-04-20T17:40:58.937747+0000","last_active":"2024-04-20T17:42:04.067557+0000","last_peered":"2024-04-20T17:42:04.067557+0000","last_clean":"2024-04-20T17:42:04.067557+0000","last_became_active":"2024-04-20T17:37:56.830487+0000","last_became_peered":"2024-04-20T17:37:56.830487+0000","last_unstale":"2024-04-20T17:42:04.067557+0000","last_undegraded":"2024-04-20T17:42:04.067557+0000","last_fullsized":"2024-04-20T17:42:04.067557+0000","mapping_epoch":694,"log_start":"709'5300","ondisk_log_start":"709'5300","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"742'11757","last_scrub_stamp":"2024-04-20T17:40:58.937604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:40:58.937604+0000","objects_scrubbed":6664,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:28.789538+0000","scrub_duration":858,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203227136,"num_objects":3101,"num_object_clones":0,"num_object_copies":6202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15307,"num_write_kb":589056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.a","version":"754'14901","reported_seq":15197,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-04-20T17:42:00.888971+0000","last_change":"2024-04-20T17:41:06.536176+0000","last_active":"2024-04-20T17:42:00.888971+0000","last_peered":"2024-04-20T17:42:00.888971+0000","last_clean":"2024-04-20T17:42:00.888971+0000","last_became_active":"2024-04-20T17:37:56.835944+0000","last_became_peered":"2024-04-20T17:37:56.835944+0000","last_unstale":"2024-04-20T17:42:00.888971+0000","last_undegraded":"2024-04-20T17:42:00.888971+0000","last_fullsized":"2024-04-20T17:42:00.888971+0000","mapping_epoch":694,"log_start":"707'4900","ondisk_log_start":"707'4900","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"742'11924","last_scrub_stamp":"2024-04-20T17:41:06.536039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:41:06.536039+0000","objects_scrubbed":6077,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:18.392067+0000","scrub_duration":857,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202047488,"num_objects":3083,"num_object_clones":0,"num_object_copies":6166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14901,"num_write_kb":575488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"754'15058","reported_seq":15269,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-04-20T17:42:00.866536+0000","last_change":"2024-04-20T17:39:03.675435+0000","last_active":"2024-04-20T17:42:00.866536+0000","last_peered":"2024-04-20T17:42:00.866536+0000","last_clean":"2024-04-20T17:42:00.866536+0000","last_became_active":"2024-04-20T17:37:56.835700+0000","last_became_peered":"2024-04-20T17:37:56.835700+0000","last_unstale":"2024-04-20T17:42:00.866536+0000","last_undegraded":"2024-04-20T17:42:00.866536+0000","last_fullsized":"2024-04-20T17:42:00.866536+0000","mapping_epoch":694,"log_start":"707'5000","ondisk_log_start":"707'5000","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'7175","last_scrub_stamp":"2024-04-20T17:39:03.675211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:03.675211+0000","objects_scrubbed":7133,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201326592,"num_objects":3072,"num_object_clones":0,"num_object_copies":6144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15058,"num_write_kb":580160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11],"acting":[0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"754'18555","reported_seq":37147,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-04-20T17:42:01.141219+0000","last_change":"2024-04-20T17:40:51.537130+0000","last_active":"2024-04-20T17:42:01.141219+0000","last_peered":"2024-04-20T17:42:01.141219+0000","last_clean":"2024-04-20T17:42:01.141219+0000","last_became_active":"2024-04-20T17:40:09.546695+0000","last_became_peered":"2024-04-20T17:40:09.546695+0000","last_unstale":"2024-04-20T17:42:01.141219+0000","last_undegraded":"2024-04-20T17:42:01.141219+0000","last_fullsized":"2024-04-20T17:42:01.141219+0000","mapping_epoch":731,"log_start":"716'8501","ondisk_log_start":"716'8501","created":694,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'14850","last_scrub_stamp":"2024-04-20T17:40:51.536812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:40:51.536812+0000","objects_scrubbed":6890,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206503936,"num_objects":3151,"num_object_clones":0,"num_object_copies":6302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14985,"num_write_kb":580352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9062,"num_bytes_recovered":593887232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"754'15360","reported_seq":15660,"reported_epoch":754,"state":"active+clean+scrubbing","last_fresh":"2024-04-20T17:42:03.731023+0000","last_change":"2024-04-20T17:42:03.605604+0000","last_active":"2024-04-20T17:42:03.731023+0000","last_peered":"2024-04-20T17:42:03.731023+0000","last_clean":"2024-04-20T17:42:03.731023+0000","last_became_active":"2024-04-20T17:37:56.836665+0000","last_became_peered":"2024-04-20T17:37:56.836665+0000","last_unstale":"2024-04-20T17:42:03.731023+0000","last_undegraded":"2024-04-20T17:42:03.731023+0000","last_fullsized":"2024-04-20T17:42:03.731023+0000","mapping_epoch":694,"log_start":"709'5300","ondisk_log_start":"709'5300","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'11243","last_scrub_stamp":"2024-04-20T17:40:48.024149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:40:48.024149+0000","objects_scrubbed":599,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"scrubbing for 0s","scrub_duration":1322,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194510848,"num_objects":2968,"num_object_clones":0,"num_object_copies":5936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15360,"num_write_kb":586496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.5","version":"754'15306","reported_seq":15632,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-04-20T17:42:04.061320+0000","last_change":"2024-04-20T17:42:03.424949+0000","last_active":"2024-04-20T17:42:04.061320+0000","last_peered":"2024-04-20T17:42:04.061320+0000","last_clean":"2024-04-20T17:42:04.061320+0000","last_became_active":"2024-04-20T17:37:56.833600+0000","last_became_peered":"2024-04-20T17:37:56.833600+0000","last_unstale":"2024-04-20T17:42:04.061320+0000","last_undegraded":"2024-04-20T17:42:04.061320+0000","last_fullsized":"2024-04-20T17:42:04.061320+0000","mapping_epoch":694,"log_start":"709'5300","ondisk_log_start":"709'5300","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'15267","last_scrub_stamp":"2024-04-20T17:42:03.424811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:42:03.424811+0000","objects_scrubbed":3060,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:43:32.116339+0000","scrub_duration":489,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":3008,"num_object_clones":0,"num_object_copies":6016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15306,"num_write_kb":586048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.4","version":"754'14908","reported_seq":15137,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-04-20T17:42:00.879422+0000","last_change":"2024-04-20T17:39:50.365478+0000","last_active":"2024-04-20T17:42:00.879422+0000","last_peered":"2024-04-20T17:42:00.879422+0000","last_clean":"2024-04-20T17:42:00.879422+0000","last_became_active":"2024-04-20T17:37:56.834727+0000","last_became_peered":"2024-04-20T17:37:56.834727+0000","last_unstale":"2024-04-20T17:42:00.879422+0000","last_undegraded":"2024-04-20T17:42:00.879422+0000","last_fullsized":"2024-04-20T17:42:00.879422+0000","mapping_epoch":694,"log_start":"707'4900","ondisk_log_start":"707'4900","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9031","last_scrub_stamp":"2024-04-20T17:39:50.365339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:50.365339+0000","objects_scrubbed":9031,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":876,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206700544,"num_objects":3154,"num_object_clones":0,"num_object_copies":6308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14908,"num_write_kb":577984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"754'15072","reported_seq":15279,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-04-20T17:42:04.173430+0000","last_change":"2024-04-20T17:39:00.502710+0000","last_active":"2024-04-20T17:42:04.173430+0000","last_peered":"2024-04-20T17:42:04.173430+0000","last_clean":"2024-04-20T17:42:04.173430+0000","last_became_active":"2024-04-20T17:37:56.832911+0000","last_became_peered":"2024-04-20T17:37:56.832911+0000","last_unstale":"2024-04-20T17:42:04.173430+0000","last_undegraded":"2024-04-20T17:42:04.173430+0000","last_fullsized":"2024-04-20T17:42:04.173430+0000","mapping_epoch":694,"log_start":"707'5000","ondisk_log_start":"707'5000","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'6831","last_scrub_stamp":"2024-04-20T17:39:00.502556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:00.502556+0000","objects_scrubbed":6782,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608000,"num_objects":3000,"num_object_clones":0,"num_object_copies":6000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15072,"num_write_kb":578304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"754'15522","reported_seq":15735,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-04-20T17:42:04.312418+0000","last_change":"2024-04-20T17:39:08.489529+0000","last_active":"2024-04-20T17:42:04.312418+0000","last_peered":"2024-04-20T17:42:04.312418+0000","last_clean":"2024-04-20T17:42:04.312418+0000","last_became_active":"2024-04-20T17:37:56.834292+0000","last_became_peered":"2024-04-20T17:37:56.834292+0000","last_unstale":"2024-04-20T17:42:04.312418+0000","last_undegraded":"2024-04-20T17:42:04.312418+0000","last_fullsized":"2024-04-20T17:42:04.312418+0000","mapping_epoch":694,"log_start":"709'5500","ondisk_log_start":"709'5500","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"716'7577","last_scrub_stamp":"2024-04-20T17:39:08.489308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:08.489308+0000","objects_scrubbed":7524,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194641920,"num_objects":2970,"num_object_clones":0,"num_object_copies":5940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15522,"num_write_kb":591744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"754'15021","reported_seq":15317,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-04-20T17:42:00.627271+0000","last_change":"2024-04-20T17:41:07.795076+0000","last_active":"2024-04-20T17:42:00.627271+0000","last_peered":"2024-04-20T17:42:00.627271+0000","last_clean":"2024-04-20T17:42:00.627271+0000","last_became_active":"2024-04-20T17:37:56.835512+0000","last_became_peered":"2024-04-20T17:37:56.835512+0000","last_unstale":"2024-04-20T17:42:00.627271+0000","last_undegraded":"2024-04-20T17:42:00.627271+0000","last_fullsized":"2024-04-20T17:42:00.627271+0000","mapping_epoch":694,"log_start":"707'5000","ondisk_log_start":"707'5000","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"743'12166","last_scrub_stamp":"2024-04-20T17:41:07.794973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:41:07.794973+0000","objects_scrubbed":6033,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:12.046222+0000","scrub_duration":644,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207159296,"num_objects":3161,"num_object_clones":0,"num_object_copies":6322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15021,"num_write_kb":581824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"754'14954","reported_seq":15246,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-04-20T17:42:00.897761+0000","last_change":"2024-04-20T17:41:16.382261+0000","last_active":"2024-04-20T17:42:00.897761+0000","last_peered":"2024-04-20T17:42:00.897761+0000","last_clean":"2024-04-20T17:42:00.897761+0000","last_became_active":"2024-04-20T17:37:56.833972+0000","last_became_peered":"2024-04-20T17:37:56.833972+0000","last_unstale":"2024-04-20T17:42:00.897761+0000","last_undegraded":"2024-04-20T17:42:00.897761+0000","last_fullsized":"2024-04-20T17:42:00.897761+0000","mapping_epoch":694,"log_start":"706'4900","ondisk_log_start":"706'4900","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"745'12483","last_scrub_stamp":"2024-04-20T17:41:16.382018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:41:16.382018+0000","objects_scrubbed":5596,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:31.605235+0000","scrub_duration":653,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203816960,"num_objects":3110,"num_object_clones":0,"num_object_copies":6220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14954,"num_write_kb":578048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.6","version":"754'14894","reported_seq":26892,"reported_epoch":754,"state":"active+recovering+degraded","last_fresh":"2024-04-20T17:42:00.917680+0000","last_change":"2024-04-20T17:39:54.962650+0000","last_active":"2024-04-20T17:42:00.917680+0000","last_peered":"2024-04-20T17:42:00.917680+0000","last_clean":"2024-04-20T17:39:34.507147+0000","last_became_active":"2024-04-20T17:39:54.933393+0000","last_became_peered":"2024-04-20T17:39:54.933393+0000","last_unstale":"2024-04-20T17:42:00.917680+0000","last_undegraded":"2024-04-20T17:39:54.578981+0000","last_fullsized":"2024-04-20T17:42:00.917680+0000","mapping_epoch":727,"log_start":"706'4800","ondisk_log_start":"706'4800","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'7141","last_scrub_stamp":"2024-04-20T17:39:04.435913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:04.435913+0000","objects_scrubbed":7101,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203948032,"num_objects":3112,"num_object_clones":0,"num_object_copies":6224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14894,"num_write_kb":576192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5892,"num_bytes_recovered":386138112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":3103},{"shards":"0,3","objects":9}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":520,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-04-20T17:41:58.614510+0000","last_change":"2024-04-20T17:40:50.375929+0000","last_active":"2024-04-20T17:41:58.614510+0000","last_peered":"2024-04-20T17:41:58.614510+0000","last_clean":"2024-04-20T17:41:58.614510+0000","last_became_active":"2024-04-20T17:39:54.970208+0000","last_became_peered":"2024-04-20T17:39:54.970208+0000","last_unstale":"2024-04-20T17:41:58.614510+0000","last_undegraded":"2024-04-20T17:41:58.614510+0000","last_fullsized":"2024-04-20T17:41:58.614510+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:40:50.375828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:40:50.375828+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:14.500087+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3233873944,"num_objects":49346,"num_object_clones":0,"num_object_copies":98692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49346,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":242102,"num_write_kb":9326273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14954,"num_bytes_recovered":980025344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7463444480,"data_stored":7463436336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160639,"ondisk_log_size":160639,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991669,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1468852,"kb_used_data":765740,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":92276300,"statfs":{"total":95995035648,"available":94490931200,"internally_reserved":0,"allocated":784117760,"data_stored":783763101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":719976900},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253307,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1159536,"kb_used_data":622948,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":92585616,"statfs":{"total":95995035648,"available":94807670784,"internally_reserved":0,"allocated":637898752,"data_stored":637547645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":549386688},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613056,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1158708,"kb_used_data":604140,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92586444,"statfs":{"total":95995035648,"available":94808518656,"internally_reserved":0,"allocated":618639360,"data_stored":618284721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":567867840},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809287,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":694380,"kb_used_data":384164,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":93050772,"statfs":{"total":95995035648,"available":95283990528,"internally_reserved":0,"allocated":393383936,"data_stored":393032849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[0,1,2,3,4,5,6,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":2297807503517,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":2830383448140,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738861,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1657332,"kb_used_data":1016552,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":92087820,"statfs":{"total":95995035648,"available":94297927680,"internally_reserved":0,"allocated":1040949248,"data_stored":1040594089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,4,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394151,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1129516,"kb_used_data":814436,"kb_used_omap":202,"kb_used_meta":314741,"kb_avail":92615636,"statfs":{"total":95995035648,"available":94838411264,"internally_reserved":0,"allocated":833982464,"data_stored":833631377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":322295255},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653201,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1348976,"kb_used_data":965160,"kb_used_omap":4,"kb_used_meta":383611,"kb_avail":92396176,"statfs":{"total":95995035648,"available":94613684224,"internally_reserved":0,"allocated":988323840,"data_stored":987968681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":392818375},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986649,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1092208,"kb_used_data":613988,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":92652944,"statfs":{"total":95995035648,"available":94876614656,"internally_reserved":0,"allocated":628723712,"data_stored":628372625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":489683397},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790419,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1073520,"kb_used_data":416228,"kb_used_omap":1,"kb_used_meta":657278,"kb_avail":92671632,"statfs":{"total":95995035648,"available":94895751168,"internally_reserved":0,"allocated":426217472,"data_stored":425866385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":673053125},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1036062720,"data_stored":1036058648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":819396608,"data_stored":819396608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":976883712,"data_stored":976879640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":622592000,"data_stored":622592000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":419692544,"data_stored":419692544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1167458304,"data_stored":1167458304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":391839744,"data_stored":391839744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":615251968,"data_stored":615251968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":633602048,"data_stored":633602048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":780664832,"data_stored":780664832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:42:08.652 INFO:tasks.cephadm.ceph_manager.ceph:PG 7.7 is not active+clean 2024-04-20T17:42:08.652 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '7.7', 'version': "754'15360", 'reported_seq': 15660, 'reported_epoch': 754, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-04-20T17:42:03.731023+0000', 'last_change': '2024-04-20T17:42:03.605604+0000', 'last_active': '2024-04-20T17:42:03.731023+0000', 'last_peered': '2024-04-20T17:42:03.731023+0000', 'last_clean': '2024-04-20T17:42:03.731023+0000', 'last_became_active': '2024-04-20T17:37:56.836665+0000', 'last_became_peered': '2024-04-20T17:37:56.836665+0000', 'last_unstale': '2024-04-20T17:42:03.731023+0000', 'last_undegraded': '2024-04-20T17:42:03.731023+0000', 'last_fullsized': '2024-04-20T17:42:03.731023+0000', 'mapping_epoch': 694, 'log_start': "709'5300", 'ondisk_log_start': "709'5300", 'created': 694, 'last_epoch_clean': 695, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "740'11243", 'last_scrub_stamp': '2024-04-20T17:40:48.024149+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T17:37:55.816485+0000', 'last_clean_scrub_stamp': '2024-04-20T17:40:48.024149+0000', 'objects_scrubbed': 599, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'scrubbing for 0s', 'scrub_duration': 1322, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 194510848, 'num_objects': 2968, 'num_object_clones': 0, 'num_object_copies': 5936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2968, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15360, 'num_write_kb': 586496, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [8, 3], 'acting': [8, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 8, 'acting_primary': 8, 'purged_snaps': []} 2024-04-20T17:42:08.652 INFO:tasks.cephadm.ceph_manager.ceph:PG 7.6 is not active+clean 2024-04-20T17:42:08.652 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '7.6', 'version': "754'14894", 'reported_seq': 26892, 'reported_epoch': 754, 'state': 'active+recovering+degraded', 'last_fresh': '2024-04-20T17:42:00.917680+0000', 'last_change': '2024-04-20T17:39:54.962650+0000', 'last_active': '2024-04-20T17:42:00.917680+0000', 'last_peered': '2024-04-20T17:42:00.917680+0000', 'last_clean': '2024-04-20T17:39:34.507147+0000', 'last_became_active': '2024-04-20T17:39:54.933393+0000', 'last_became_peered': '2024-04-20T17:39:54.933393+0000', 'last_unstale': '2024-04-20T17:42:00.917680+0000', 'last_undegraded': '2024-04-20T17:39:54.578981+0000', 'last_fullsized': '2024-04-20T17:42:00.917680+0000', 'mapping_epoch': 727, 'log_start': "706'4800", 'ondisk_log_start': "706'4800", 'created': 694, 'last_epoch_clean': 695, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "714'7141", 'last_scrub_stamp': '2024-04-20T17:39:04.435913+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T17:37:55.816485+0000', 'last_clean_scrub_stamp': '2024-04-20T17:39:04.435913+0000', 'objects_scrubbed': 7101, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 1421, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 203948032, 'num_objects': 3112, 'num_object_clones': 0, 'num_object_copies': 6224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3112, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14894, 'num_write_kb': 576192, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5892, 'num_bytes_recovered': 386138112, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': ['0'], 'object_location_counts': [{'shards': '0', 'objects': 3103}, {'shards': '0,3', 'objects': 9}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-20T17:42:08.653 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph status --format=json 2024-04-20T17:42:08.706 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:42:08.742 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:08.768 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:42:08.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:08.772 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:42:08.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:08 smithi171 ceph-mon[44189]: pgmap v2026: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 2.8 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 901 op/s; 2814/93140 objects degraded (3.021%); 3.5 MiB/s, 56 objects/s recovering 2024-04-20T17:42:08.874 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:42:08.922 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:08 smithi005 ceph-mon[39562]: pgmap v2026: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 2.8 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 901 op/s; 2814/93140 objects degraded (3.021%); 3.5 MiB/s, 56 objects/s recovering 2024-04-20T17:42:08.929 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:08.992 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:42:08.993 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:42:08.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:09.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:42:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:08 smithi106 ceph-mon[44423]: pgmap v2026: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing, 15 active+clean; 2.8 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 901 op/s; 2814/93140 objects degraded (3.021%); 3.5 MiB/s, 56 objects/s recovering 2024-04-20T17:42:09.100 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:42:09.133 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:09.234 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:42:09.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:09.239 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:42:09.298 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:42:09.340 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:42:09.417 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:09.605 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:42:09.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:09.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:42:09.710 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:42:09.754 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:09.972 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:42:09.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:09.977 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:42:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:09 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:42:10.078 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:42:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:09 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:42:10.119 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:10.149 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:09 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:42:10.430 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:42:10.782 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:42:10.910 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:10 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1032155588' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:42:10.910 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:10 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:42:10.910 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:10 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:42:10.910 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:10 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:42:10.910 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:10 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:42:10.910 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:10 smithi171 ceph-mon[44189]: pgmap v2027: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 2.7 GiB data, 9.7 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 894 op/s; 2814/89818 objects degraded (3.133%); 3.5 MiB/s, 55 objects/s recovering 2024-04-20T17:42:10.911 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:42:10.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:10.917 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:42:10.927 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:10 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1032155588' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:42:10.927 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:10 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:42:10.927 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:10 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:42:10.927 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:10 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:42:10.927 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:10 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:42:10.927 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:10 smithi005 ceph-mon[39562]: pgmap v2027: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 2.7 GiB data, 9.7 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 894 op/s; 2814/89818 objects degraded (3.133%); 3.5 MiB/s, 55 objects/s recovering 2024-04-20T17:42:11.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:10 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1032155588' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:42:11.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:10 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:42:11.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:10 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:42:11.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:10 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:42:11.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:10 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:42:11.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:10 smithi106 ceph-mon[44423]: pgmap v2027: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 2.7 GiB data, 9.7 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 894 op/s; 2814/89818 objects degraded (3.133%); 3.5 MiB/s, 55 objects/s recovering 2024-04-20T17:42:11.018 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:42:11.093 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:11.168 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:42:11.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:11.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:42:11.283 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:42:11.320 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:11.538 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:42:11.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:11.543 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:42:11.566 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:42:11.566 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:42:11.645 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:42:11.715 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:11.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:11 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:42:11.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:11 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1032155588' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:42:11.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:11 smithi171 ceph-mon[44189]: osdmap e756: 12 total, 11 up, 9 in 2024-04-20T17:42:11.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:11 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:42:11.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:11 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:42:11.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:11 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:42:11.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:11 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 2814/89818 objects degraded (3.133%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:11.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:11 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:42:11.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:11 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1032155588' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:42:11.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:11 smithi005 ceph-mon[39562]: osdmap e756: 12 total, 11 up, 9 in 2024-04-20T17:42:11.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:11 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:42:11.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:11 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:42:11.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:11 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:42:11.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:11 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 2814/89818 objects degraded (3.133%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:12.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:11 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:42:12.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:11 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1032155588' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:42:12.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:11 smithi106 ceph-mon[44423]: osdmap e756: 12 total, 11 up, 9 in 2024-04-20T17:42:12.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:11 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:42:12.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:11 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:42:12.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:11 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:42:12.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:11 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 2814/89818 objects degraded (3.133%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:12.286 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-20T17:42:12.691 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:42:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:12 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/531591687' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:42:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:12 smithi171 ceph-mon[44189]: pgmap v2029: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 2.6 GiB data, 9.5 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 908 op/s; 2531/84164 objects degraded (3.007%); 4.5 MiB/s, 71 objects/s recovering 2024-04-20T17:42:13.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:12 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/531591687' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:42:13.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:13.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:13.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:13.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:13.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:13.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:12 smithi005 ceph-mon[39562]: pgmap v2029: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 2.6 GiB data, 9.5 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 908 op/s; 2531/84164 objects degraded (3.007%); 4.5 MiB/s, 71 objects/s recovering 2024-04-20T17:42:13.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:12 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/531591687' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:42:13.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:13.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:13.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:13.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:13.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:13.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:12 smithi106 ceph-mon[44423]: pgmap v2029: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 2.6 GiB data, 9.5 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 908 op/s; 2531/84164 objects degraded (3.007%); 4.5 MiB/s, 71 objects/s recovering 2024-04-20T17:42:13.439 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:42:13.439 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:42:13.440 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:42:13.440 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:42:13.440 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:42:13.440 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:42:13.440 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:42:13.440 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:42:13.440 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:42:13.440 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:42:13.440 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:42:13.440 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:42:13.440 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:42:13.440 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:42:13.440 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:42:13.441 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:42:13.441 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:42:13.441 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:42:13.441 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:42:13.441 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:42:13.441 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:42:13.441 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:42:13.441 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:42:14.024 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:13 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:14.024 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:13 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:14.024 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:13 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:14.024 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:13 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:14.024 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:13 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:14.025 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:13 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:14.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:13 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:14.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:13 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:14.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:13 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:14.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:13 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:14.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:13 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:14.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:13 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:13 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:13 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:13 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:13 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:13 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:13 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:14.319 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:42:14.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:14.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:42:14.429 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:42:14.468 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:14.960 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:14 smithi005 ceph-mon[39562]: pgmap v2030: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 2.5 GiB data, 9.3 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.00k op/s; 2531/80898 objects degraded (3.129%); 4.0 MiB/s, 64 objects/s recovering 2024-04-20T17:42:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:14 smithi171 ceph-mon[44189]: pgmap v2030: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 2.5 GiB data, 9.3 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.00k op/s; 2531/80898 objects degraded (3.129%); 4.0 MiB/s, 64 objects/s recovering 2024-04-20T17:42:15.214 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:42:15.214 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:42:15.214 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:42:15.214 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:42:15.214 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.214 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233747 6.1 6:8163dd93:::benchmark_data_smithi171_147972_object72649:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:42:15.215 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.174286+0000", 2024-04-20T17:42:15.215 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 363.47697566199997, 2024-04-20T17:42:15.215 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.206612618, 2024-04-20T17:42:15.215 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:15.215 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:15.215 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:15.215 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:15.215 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:42:15.215 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:42:15.215 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233747 2024-04-20T17:42:15.215 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.215 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:15.216 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.216 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:15.216 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.174286+0000", 2024-04-20T17:42:15.216 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.216 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.216 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.216 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:15.217 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.174286+0000", 2024-04-20T17:42:15.217 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.217 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.217 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.217 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:15.217 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.174287+0000", 2024-04-20T17:42:15.217 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.33e-06 2024-04-20T17:42:15.217 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.217 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.217 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:15.218 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.174291+0000", 2024-04-20T17:42:15.218 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8480000000000003e-06 2024-04-20T17:42:15.218 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.218 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.218 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:15.218 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.174300+0000", 2024-04-20T17:42:15.218 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.217e-06 2024-04-20T17:42:15.218 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.218 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.218 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:15.218 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.174306+0000", 2024-04-20T17:42:15.219 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.682e-06 2024-04-20T17:42:15.219 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.219 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.219 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:15.219 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.174348+0000", 2024-04-20T17:42:15.219 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2347e-05 2024-04-20T17:42:15.219 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.219 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.219 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:15.219 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.174461+0000", 2024-04-20T17:42:15.219 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000112728 2024-04-20T17:42:15.219 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.219 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.219 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:42:15.219 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.174571+0000", 2024-04-20T17:42:15.220 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010971699999999999 2024-04-20T17:42:15.220 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.220 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.220 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:15.220 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.178595+0000", 2024-04-20T17:42:15.220 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0040239050000000004 2024-04-20T17:42:15.220 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.220 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.220 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:15.220 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.380796+0000", 2024-04-20T17:42:15.220 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20220144600000001 2024-04-20T17:42:15.220 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.220 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.220 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:15.221 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.380851+0000", 2024-04-20T17:42:15.221 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4333000000000003e-05 2024-04-20T17:42:15.221 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.221 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.221 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:15.221 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.380899+0000", 2024-04-20T17:42:15.221 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8065000000000001e-05 2024-04-20T17:42:15.221 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.221 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:15.221 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.221 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.221 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.221 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233760 6.1 6:87254b5c:::benchmark_data_smithi171_147972_object72662:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:42:15.221 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.184502+0000", 2024-04-20T17:42:15.221 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 363.46675995300001, 2024-04-20T17:42:15.221 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.210398054, 2024-04-20T17:42:15.222 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:15.222 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:15.222 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:15.222 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:15.222 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:42:15.222 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:42:15.222 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233760 2024-04-20T17:42:15.222 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.222 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:15.222 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.222 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:15.222 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.184502+0000", 2024-04-20T17:42:15.222 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.222 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.223 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.223 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:15.223 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.184502+0000", 2024-04-20T17:42:15.223 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.223 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.223 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.223 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:15.223 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.184504+0000", 2024-04-20T17:42:15.223 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2869999999999999e-06 2024-04-20T17:42:15.223 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.223 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.223 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:15.223 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.184511+0000", 2024-04-20T17:42:15.223 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.55e-06 2024-04-20T17:42:15.223 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.223 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.224 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:15.224 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.184522+0000", 2024-04-20T17:42:15.224 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1722000000000001e-05 2024-04-20T17:42:15.224 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.224 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.224 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:15.224 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.184529+0000", 2024-04-20T17:42:15.224 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8889999999999996e-06 2024-04-20T17:42:15.224 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.224 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.224 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:15.224 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.184586+0000", 2024-04-20T17:42:15.224 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6449000000000001e-05 2024-04-20T17:42:15.224 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.224 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.225 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:15.225 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.184719+0000", 2024-04-20T17:42:15.225 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000133149 2024-04-20T17:42:15.225 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.225 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.225 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:42:15.225 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.184849+0000", 2024-04-20T17:42:15.225 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000129927 2024-04-20T17:42:15.225 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.225 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.225 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:15.225 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.188857+0000", 2024-04-20T17:42:15.225 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.004008541 2024-04-20T17:42:15.225 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.225 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.226 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:15.226 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.394830+0000", 2024-04-20T17:42:15.226 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20597264900000001 2024-04-20T17:42:15.226 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.226 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.226 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:15.226 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.394867+0000", 2024-04-20T17:42:15.226 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6736999999999997e-05 2024-04-20T17:42:15.226 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.226 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.226 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:15.226 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.394900+0000", 2024-04-20T17:42:15.226 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3154000000000001e-05 2024-04-20T17:42:15.226 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.226 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:15.227 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.227 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.227 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.227 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233773 6.1 6:8680cb34:::benchmark_data_smithi171_147972_object72675:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:42:15.227 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.195965+0000", 2024-04-20T17:42:15.227 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 363.45529622399999, 2024-04-20T17:42:15.227 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.21500868300000001, 2024-04-20T17:42:15.227 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:15.227 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:15.227 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:15.227 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:15.227 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:42:15.227 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:42:15.227 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233773 2024-04-20T17:42:15.227 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.227 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:15.228 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.228 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:15.228 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.195965+0000", 2024-04-20T17:42:15.228 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.228 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.228 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.228 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:15.228 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.195965+0000", 2024-04-20T17:42:15.228 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.228 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.228 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.228 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:15.228 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.195968+0000", 2024-04-20T17:42:15.228 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9359999999999999e-06 2024-04-20T17:42:15.229 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.229 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.229 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:15.229 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.195976+0000", 2024-04-20T17:42:15.229 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9459999999999998e-06 2024-04-20T17:42:15.229 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.229 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.229 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:15.229 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.195991+0000", 2024-04-20T17:42:15.229 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4728e-05 2024-04-20T17:42:15.229 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.229 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.229 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:15.229 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.195999+0000", 2024-04-20T17:42:15.229 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0150000000000003e-06 2024-04-20T17:42:15.230 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.230 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.230 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:15.230 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.196043+0000", 2024-04-20T17:42:15.230 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3757000000000001e-05 2024-04-20T17:42:15.230 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.230 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.230 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:15.230 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.196159+0000", 2024-04-20T17:42:15.230 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000115686 2024-04-20T17:42:15.230 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.230 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.230 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:42:15.230 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.196299+0000", 2024-04-20T17:42:15.230 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014027000000000001 2024-04-20T17:42:15.231 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.231 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.231 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:15.231 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.201108+0000", 2024-04-20T17:42:15.231 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0048087420000000004 2024-04-20T17:42:15.231 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.231 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.231 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:15.231 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.410861+0000", 2024-04-20T17:42:15.231 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.209753467 2024-04-20T17:42:15.231 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.231 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.231 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:15.231 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.410922+0000", 2024-04-20T17:42:15.231 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0498000000000003e-05 2024-04-20T17:42:15.232 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.232 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.232 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:15.232 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.410974+0000", 2024-04-20T17:42:15.232 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2638e-05 2024-04-20T17:42:15.232 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.232 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:15.232 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.232 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.232 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.232 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233783 6.1 6:82c033a0:::benchmark_data_smithi171_147972_object72685:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:42:15.232 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.204583+0000", 2024-04-20T17:42:15.232 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 363.44667843399998, 2024-04-20T17:42:15.232 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20653519100000001, 2024-04-20T17:42:15.232 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:15.232 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:15.233 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:15.233 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:15.233 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:42:15.233 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:42:15.233 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233783 2024-04-20T17:42:15.233 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.233 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:15.233 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.233 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:15.233 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.204583+0000", 2024-04-20T17:42:15.233 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.233 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.233 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.233 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:15.233 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.204583+0000", 2024-04-20T17:42:15.234 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.234 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.234 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.234 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:15.234 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.204586+0000", 2024-04-20T17:42:15.234 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7240000000000001e-06 2024-04-20T17:42:15.234 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.234 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.234 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:15.234 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.204592+0000", 2024-04-20T17:42:15.234 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5570000000000003e-06 2024-04-20T17:42:15.234 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.234 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.234 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:15.235 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.204603+0000", 2024-04-20T17:42:15.235 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1154e-05 2024-04-20T17:42:15.235 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.235 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.235 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:15.235 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.204609+0000", 2024-04-20T17:42:15.235 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7190000000000004e-06 2024-04-20T17:42:15.235 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.235 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.235 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:15.235 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.204652+0000", 2024-04-20T17:42:15.235 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2138000000000002e-05 2024-04-20T17:42:15.235 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.235 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.235 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:15.235 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.204752+0000", 2024-04-20T17:42:15.236 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000100317 2024-04-20T17:42:15.236 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.236 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.236 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:42:15.236 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.204823+0000", 2024-04-20T17:42:15.236 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1043000000000004e-05 2024-04-20T17:42:15.236 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.236 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.236 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:15.236 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.209034+0000", 2024-04-20T17:42:15.236 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0042115750000000004 2024-04-20T17:42:15.236 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.236 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.236 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:15.236 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.411023+0000", 2024-04-20T17:42:15.237 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20198877700000001 2024-04-20T17:42:15.237 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.237 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.237 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:15.237 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.411074+0000", 2024-04-20T17:42:15.237 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0321999999999997e-05 2024-04-20T17:42:15.237 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.237 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.237 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:15.237 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.411118+0000", 2024-04-20T17:42:15.237 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4864999999999998e-05 2024-04-20T17:42:15.237 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.237 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:15.237 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.237 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.237 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.238 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233795 6.1 6:824ee5e7:::benchmark_data_smithi171_147972_object72697:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:42:15.238 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.218816+0000", 2024-04-20T17:42:15.238 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 363.43244595099998, 2024-04-20T17:42:15.238 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19916731200000001, 2024-04-20T17:42:15.238 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:15.238 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:15.238 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:15.238 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:15.238 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:42:15.238 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:42:15.238 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233795 2024-04-20T17:42:15.238 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.238 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:15.238 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.238 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:15.239 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.218816+0000", 2024-04-20T17:42:15.239 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.239 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.239 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.239 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:15.239 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.218816+0000", 2024-04-20T17:42:15.239 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.239 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.239 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.239 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:15.239 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.218821+0000", 2024-04-20T17:42:15.239 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0200000000000002e-06 2024-04-20T17:42:15.239 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.239 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.239 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:15.240 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.218836+0000", 2024-04-20T17:42:15.240 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5142e-05 2024-04-20T17:42:15.240 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.240 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.240 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:15.240 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.218870+0000", 2024-04-20T17:42:15.240 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4483000000000002e-05 2024-04-20T17:42:15.240 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.240 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.240 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:15.240 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.218894+0000", 2024-04-20T17:42:15.240 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.336e-05 2024-04-20T17:42:15.240 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.240 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.240 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:15.240 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.218983+0000", 2024-04-20T17:42:15.241 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9497e-05 2024-04-20T17:42:15.241 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.241 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.241 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:15.241 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.219086+0000", 2024-04-20T17:42:15.241 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010312800000000001 2024-04-20T17:42:15.241 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.241 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.241 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:42:15.241 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.219191+0000", 2024-04-20T17:42:15.241 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000105009 2024-04-20T17:42:15.241 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.241 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.241 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:15.241 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.222869+0000", 2024-04-20T17:42:15.242 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0036778990000000001 2024-04-20T17:42:15.242 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.242 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.242 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:15.242 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.417920+0000", 2024-04-20T17:42:15.242 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.195051113 2024-04-20T17:42:15.242 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.242 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.242 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:15.242 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.417953+0000", 2024-04-20T17:42:15.242 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2157999999999999e-05 2024-04-20T17:42:15.242 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.242 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.242 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:15.243 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.417983+0000", 2024-04-20T17:42:15.243 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0502999999999999e-05 2024-04-20T17:42:15.243 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.243 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:15.243 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.243 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.243 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.243 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233803 6.1 6:8893042e:::benchmark_data_smithi171_147972_object72705:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:42:15.243 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.224771+0000", 2024-04-20T17:42:15.243 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 363.42649083999999, 2024-04-20T17:42:15.243 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19336389500000001, 2024-04-20T17:42:15.243 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:15.243 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:15.243 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:15.243 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:15.243 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:42:15.244 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:42:15.244 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233803 2024-04-20T17:42:15.244 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.244 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:15.244 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.244 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:15.244 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.224771+0000", 2024-04-20T17:42:15.244 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.244 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.244 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.244 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:15.244 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.224771+0000", 2024-04-20T17:42:15.244 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.244 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.244 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.245 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:15.245 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.224773+0000", 2024-04-20T17:42:15.245 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7999999999999999e-06 2024-04-20T17:42:15.245 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.245 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.245 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:15.245 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.224778+0000", 2024-04-20T17:42:15.245 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8279999999999999e-06 2024-04-20T17:42:15.245 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.245 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.245 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:15.245 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.224789+0000", 2024-04-20T17:42:15.245 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0720000000000001e-05 2024-04-20T17:42:15.245 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.245 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.246 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:15.246 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.224796+0000", 2024-04-20T17:42:15.246 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0809999999999996e-06 2024-04-20T17:42:15.246 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.246 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.246 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:15.246 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.224839+0000", 2024-04-20T17:42:15.246 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3154000000000001e-05 2024-04-20T17:42:15.246 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.246 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.246 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:15.246 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.224929+0000", 2024-04-20T17:42:15.246 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9203999999999995e-05 2024-04-20T17:42:15.246 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.247 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.247 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:42:15.247 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.225034+0000", 2024-04-20T17:42:15.247 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000105017 2024-04-20T17:42:15.247 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.247 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.247 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:15.247 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.228470+0000", 2024-04-20T17:42:15.247 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0034366959999999999 2024-04-20T17:42:15.247 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.247 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.247 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:15.247 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.418044+0000", 2024-04-20T17:42:15.247 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.18957336799999999 2024-04-20T17:42:15.247 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.248 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.248 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:15.248 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.418092+0000", 2024-04-20T17:42:15.248 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8359999999999998e-05 2024-04-20T17:42:15.248 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.248 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.248 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:15.248 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.418135+0000", 2024-04-20T17:42:15.248 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2667000000000003e-05 2024-04-20T17:42:15.248 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.248 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:15.248 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.248 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.248 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.248 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233813 6.1 6:87bd3cce:::benchmark_data_smithi171_147972_object72715:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:42:15.249 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.232901+0000", 2024-04-20T17:42:15.249 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 363.41836104399999, 2024-04-20T17:42:15.249 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.18540720599999999, 2024-04-20T17:42:15.249 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:15.249 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:15.249 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:15.249 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:15.249 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:42:15.249 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:42:15.249 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233813 2024-04-20T17:42:15.249 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.249 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:15.249 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.249 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:15.249 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.232901+0000", 2024-04-20T17:42:15.249 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.250 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.250 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.250 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:15.250 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.232901+0000", 2024-04-20T17:42:15.250 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.250 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.250 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.250 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:15.250 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.232903+0000", 2024-04-20T17:42:15.250 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1320000000000001e-06 2024-04-20T17:42:15.250 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.250 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.250 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:15.250 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.232910+0000", 2024-04-20T17:42:15.250 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9269999999999999e-06 2024-04-20T17:42:15.251 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.251 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.251 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:15.251 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.232923+0000", 2024-04-20T17:42:15.251 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3392e-05 2024-04-20T17:42:15.251 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.251 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.251 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:15.251 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.232931+0000", 2024-04-20T17:42:15.251 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0499999999999992e-06 2024-04-20T17:42:15.251 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.251 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.251 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:15.251 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.232985+0000", 2024-04-20T17:42:15.251 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3690000000000003e-05 2024-04-20T17:42:15.252 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.252 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.252 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:15.252 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.233074+0000", 2024-04-20T17:42:15.252 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8876999999999996e-05 2024-04-20T17:42:15.252 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.252 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.252 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:42:15.252 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.233150+0000", 2024-04-20T17:42:15.252 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6335999999999996e-05 2024-04-20T17:42:15.252 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.252 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.252 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:15.252 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.236856+0000", 2024-04-20T17:42:15.252 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0037059549999999999 2024-04-20T17:42:15.252 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.253 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.253 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:15.253 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.418212+0000", 2024-04-20T17:42:15.253 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.18135632600000001 2024-04-20T17:42:15.253 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.253 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.253 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:15.253 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.418265+0000", 2024-04-20T17:42:15.253 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3086e-05 2024-04-20T17:42:15.253 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.253 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.253 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:15.253 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.418308+0000", 2024-04-20T17:42:15.253 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2435000000000003e-05 2024-04-20T17:42:15.253 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.253 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:15.254 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.254 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.254 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.254 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233855 6.1 6:8faa32b1:::benchmark_data_smithi171_147972_object72757:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:42:15.254 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.278676+0000", 2024-04-20T17:42:15.254 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 363.37258560100003, 2024-04-20T17:42:15.254 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.13975268499999999, 2024-04-20T17:42:15.254 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:15.254 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:15.254 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:15.254 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:15.254 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:42:15.254 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:42:15.254 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233855 2024-04-20T17:42:15.255 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.255 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:15.255 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.255 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:15.255 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.278676+0000", 2024-04-20T17:42:15.255 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.255 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.255 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.255 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:15.255 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.278676+0000", 2024-04-20T17:42:15.255 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.255 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.255 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.255 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:15.255 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.278679+0000", 2024-04-20T17:42:15.256 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0630000000000002e-06 2024-04-20T17:42:15.256 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.256 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.256 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:15.256 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.278688+0000", 2024-04-20T17:42:15.256 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.3780000000000001e-06 2024-04-20T17:42:15.256 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.256 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.256 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:15.256 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.278703+0000", 2024-04-20T17:42:15.256 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5302999999999999e-05 2024-04-20T17:42:15.256 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.256 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.256 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:15.256 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.278713+0000", 2024-04-20T17:42:15.257 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9040000000000004e-06 2024-04-20T17:42:15.257 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.257 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.257 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:15.257 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.278767+0000", 2024-04-20T17:42:15.257 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4245e-05 2024-04-20T17:42:15.257 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.257 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.257 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:15.257 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.278869+0000", 2024-04-20T17:42:15.257 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000102377 2024-04-20T17:42:15.257 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.257 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.257 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:42:15.257 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.278951+0000", 2024-04-20T17:42:15.257 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1237999999999997e-05 2024-04-20T17:42:15.258 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.258 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.258 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:15.258 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.282731+0000", 2024-04-20T17:42:15.258 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0037807610000000001 2024-04-20T17:42:15.258 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.258 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.258 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:15.258 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.418360+0000", 2024-04-20T17:42:15.258 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.135629006 2024-04-20T17:42:15.258 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.258 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.258 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:15.258 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.418406+0000", 2024-04-20T17:42:15.259 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5343999999999998e-05 2024-04-20T17:42:15.259 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.259 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.259 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:15.259 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.418429+0000", 2024-04-20T17:42:15.259 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3065999999999999e-05 2024-04-20T17:42:15.259 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.259 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:15.259 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.259 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.259 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.259 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233864 6.1 6:866d1e67:::benchmark_data_smithi171_147972_object72766:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:42:15.259 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.286641+0000", 2024-04-20T17:42:15.259 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 363.36462042099998, 2024-04-20T17:42:15.259 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.13675062399999999, 2024-04-20T17:42:15.260 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:15.260 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:15.260 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:15.260 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:15.260 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:42:15.260 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:42:15.260 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233864 2024-04-20T17:42:15.260 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.260 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:15.260 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.260 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:15.260 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.286641+0000", 2024-04-20T17:42:15.260 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.260 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.260 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.260 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:15.261 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.286641+0000", 2024-04-20T17:42:15.261 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.261 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.261 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.261 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:15.261 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.286643+0000", 2024-04-20T17:42:15.261 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7969999999999999e-06 2024-04-20T17:42:15.261 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.261 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.261 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:15.261 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.286649+0000", 2024-04-20T17:42:15.261 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1090000000000003e-06 2024-04-20T17:42:15.261 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.261 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.261 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:15.262 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.286660+0000", 2024-04-20T17:42:15.262 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0536999999999999e-05 2024-04-20T17:42:15.262 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.262 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.262 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:15.262 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.286666+0000", 2024-04-20T17:42:15.262 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5130000000000002e-06 2024-04-20T17:42:15.262 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.262 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.262 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:15.262 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.286710+0000", 2024-04-20T17:42:15.262 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3336e-05 2024-04-20T17:42:15.262 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.262 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.262 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:15.263 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.286795+0000", 2024-04-20T17:42:15.263 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5648000000000004e-05 2024-04-20T17:42:15.263 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.263 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.263 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:42:15.263 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.286870+0000", 2024-04-20T17:42:15.263 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5222999999999995e-05 2024-04-20T17:42:15.263 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.263 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.263 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:15.263 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.290258+0000", 2024-04-20T17:42:15.263 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0033879909999999999 2024-04-20T17:42:15.263 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.263 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.263 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:15.263 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.423320+0000", 2024-04-20T17:42:15.264 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.133061771 2024-04-20T17:42:15.264 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.264 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.264 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:15.264 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.423352+0000", 2024-04-20T17:42:15.264 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2239000000000001e-05 2024-04-20T17:42:15.264 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.264 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.264 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:15.264 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.423392+0000", 2024-04-20T17:42:15.264 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9459999999999998e-05 2024-04-20T17:42:15.264 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.264 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:15.264 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.264 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.265 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.265 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:82534 7.7 7:e6749964:::benchmark_data_smithi171_161871_object82533:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e708)", 2024-04-20T17:42:15.265 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:40.968973+0000", 2024-04-20T17:42:15.265 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 213.68228916699999, 2024-04-20T17:42:15.265 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20917861900000001, 2024-04-20T17:42:15.265 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:15.265 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:15.265 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:15.265 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:15.265 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:15.265 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:15.265 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 82534 2024-04-20T17:42:15.265 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.265 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:15.265 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.265 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:15.266 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.968973+0000", 2024-04-20T17:42:15.266 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.266 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.266 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.266 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:15.266 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.968973+0000", 2024-04-20T17:42:15.266 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.266 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.266 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.266 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:15.266 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.968975+0000", 2024-04-20T17:42:15.266 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0499999999999999e-06 2024-04-20T17:42:15.266 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.266 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.267 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:15.267 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.968998+0000", 2024-04-20T17:42:15.267 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3323999999999999e-05 2024-04-20T17:42:15.267 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.267 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.267 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:15.267 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.969009+0000", 2024-04-20T17:42:15.267 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0949e-05 2024-04-20T17:42:15.267 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.267 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.267 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:15.267 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.969016+0000", 2024-04-20T17:42:15.267 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2819999999999999e-06 2024-04-20T17:42:15.267 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.267 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.268 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:15.268 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.969288+0000", 2024-04-20T17:42:15.268 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00027152799999999999 2024-04-20T17:42:15.268 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.268 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.268 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:15.268 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.969403+0000", 2024-04-20T17:42:15.268 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000115777 2024-04-20T17:42:15.268 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.268 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.268 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:42:15.268 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.969573+0000", 2024-04-20T17:42:15.268 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000169154 2024-04-20T17:42:15.268 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.268 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.269 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:15.269 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.972385+0000", 2024-04-20T17:42:15.269 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0028121550000000002 2024-04-20T17:42:15.269 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.269 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.269 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:15.269 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.178046+0000", 2024-04-20T17:42:15.269 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.205661064 2024-04-20T17:42:15.269 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.269 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.269 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:15.269 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.178107+0000", 2024-04-20T17:42:15.269 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0977000000000002e-05 2024-04-20T17:42:15.269 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.269 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.269 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:15.270 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.178151+0000", 2024-04-20T17:42:15.270 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4359e-05 2024-04-20T17:42:15.270 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.270 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:15.270 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.270 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.270 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.270 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:82538 7.7 7:ecb87a89:::benchmark_data_smithi171_161871_object82537:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e708)", 2024-04-20T17:42:15.270 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:40.969208+0000", 2024-04-20T17:42:15.270 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 213.682054072, 2024-04-20T17:42:15.270 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.21037962099999999, 2024-04-20T17:42:15.270 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:15.270 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:15.270 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:15.270 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:15.271 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:15.271 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:15.271 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 82538 2024-04-20T17:42:15.271 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.271 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:15.271 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.271 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:15.271 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.969208+0000", 2024-04-20T17:42:15.271 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.271 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.271 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.271 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:15.271 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.969208+0000", 2024-04-20T17:42:15.271 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.271 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.272 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.272 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:15.272 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.969209+0000", 2024-04-20T17:42:15.272 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4869999999999999e-06 2024-04-20T17:42:15.272 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.272 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.272 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:15.272 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.969230+0000", 2024-04-20T17:42:15.272 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0999e-05 2024-04-20T17:42:15.272 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.272 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.272 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:15.272 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.969240+0000", 2024-04-20T17:42:15.272 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9080000000000007e-06 2024-04-20T17:42:15.272 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.273 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.273 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:15.273 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.969247+0000", 2024-04-20T17:42:15.273 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0090000000000001e-06 2024-04-20T17:42:15.273 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.273 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.273 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:15.273 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.974582+0000", 2024-04-20T17:42:15.273 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.005334475 2024-04-20T17:42:15.273 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.273 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.273 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:15.273 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.974735+0000", 2024-04-20T17:42:15.273 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015318499999999999 2024-04-20T17:42:15.273 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.274 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.274 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:42:15.274 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.974884+0000", 2024-04-20T17:42:15.274 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014973499999999999 2024-04-20T17:42:15.274 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.274 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.274 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:15.274 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.980059+0000", 2024-04-20T17:42:15.274 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0051748319999999999 2024-04-20T17:42:15.274 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.275 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.275 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:15.275 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.179529+0000", 2024-04-20T17:42:15.275 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19947004300000001 2024-04-20T17:42:15.275 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.275 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.275 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:15.275 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.179563+0000", 2024-04-20T17:42:15.275 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4103999999999997e-05 2024-04-20T17:42:15.275 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.275 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.275 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:15.275 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.179587+0000", 2024-04-20T17:42:15.275 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3844000000000001e-05 2024-04-20T17:42:15.275 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.276 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:15.276 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.276 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.276 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.276 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:82540 7.7 7:e0f29cdc:::benchmark_data_smithi171_161871_object82539:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e708)", 2024-04-20T17:42:15.276 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:40.975200+0000", 2024-04-20T17:42:15.276 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 213.67606134799999, 2024-04-20T17:42:15.276 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20839459499999999, 2024-04-20T17:42:15.276 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:15.276 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:15.276 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:15.276 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:15.276 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:15.276 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:15.277 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 82540 2024-04-20T17:42:15.277 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.277 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:15.277 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.277 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:15.277 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.975200+0000", 2024-04-20T17:42:15.277 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.277 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.277 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.277 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:15.277 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.975200+0000", 2024-04-20T17:42:15.277 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.277 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.277 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.277 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:15.278 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.975210+0000", 2024-04-20T17:42:15.278 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9879999999999999e-06 2024-04-20T17:42:15.278 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.278 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.278 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:15.278 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.975238+0000", 2024-04-20T17:42:15.278 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7730999999999999e-05 2024-04-20T17:42:15.278 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.278 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.278 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:15.278 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.975255+0000", 2024-04-20T17:42:15.278 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6702999999999999e-05 2024-04-20T17:42:15.278 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.278 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.278 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:15.279 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.975265+0000", 2024-04-20T17:42:15.279 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0125e-05 2024-04-20T17:42:15.279 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.279 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.279 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:15.279 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.977747+0000", 2024-04-20T17:42:15.279 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.002481877 2024-04-20T17:42:15.279 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.279 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.279 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:15.279 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.977877+0000", 2024-04-20T17:42:15.279 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013067800000000001 2024-04-20T17:42:15.279 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.279 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.279 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:42:15.280 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.978031+0000", 2024-04-20T17:42:15.280 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015306200000000001 2024-04-20T17:42:15.280 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.280 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.280 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:15.280 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.981097+0000", 2024-04-20T17:42:15.280 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0030660010000000001 2024-04-20T17:42:15.280 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.280 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.280 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:15.280 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.183525+0000", 2024-04-20T17:42:15.280 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20242866500000001 2024-04-20T17:42:15.280 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.280 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.281 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:15.281 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.183564+0000", 2024-04-20T17:42:15.281 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8439000000000002e-05 2024-04-20T17:42:15.281 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.281 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.281 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:15.281 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.183595+0000", 2024-04-20T17:42:15.281 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1325999999999998e-05 2024-04-20T17:42:15.281 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.281 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:15.281 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.281 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.281 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.281 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:82567 7.7 7:e22bf8b5:::benchmark_data_smithi171_161871_object82566:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e708)", 2024-04-20T17:42:15.281 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:40.988647+0000", 2024-04-20T17:42:15.282 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 213.662614719, 2024-04-20T17:42:15.282 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19503699599999999, 2024-04-20T17:42:15.282 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:15.282 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:15.282 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:15.282 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:15.282 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:15.282 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:15.282 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 82567 2024-04-20T17:42:15.282 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.282 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:15.282 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.282 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:15.282 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.988647+0000", 2024-04-20T17:42:15.282 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.283 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.283 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.283 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:15.283 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.988647+0000", 2024-04-20T17:42:15.283 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.283 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.283 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.283 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:15.283 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.988649+0000", 2024-04-20T17:42:15.283 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.069e-06 2024-04-20T17:42:15.283 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.283 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.283 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:15.283 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.988669+0000", 2024-04-20T17:42:15.283 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0106999999999999e-05 2024-04-20T17:42:15.284 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.284 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.284 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:15.284 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.988685+0000", 2024-04-20T17:42:15.284 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.535e-05 2024-04-20T17:42:15.284 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.284 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.284 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:15.284 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.988693+0000", 2024-04-20T17:42:15.284 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.7649999999999999e-06 2024-04-20T17:42:15.284 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.284 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.284 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:15.285 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.988759+0000", 2024-04-20T17:42:15.285 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6152e-05 2024-04-20T17:42:15.285 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.285 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.285 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:15.285 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.988947+0000", 2024-04-20T17:42:15.285 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000188068 2024-04-20T17:42:15.285 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.285 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.285 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:42:15.285 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.989129+0000", 2024-04-20T17:42:15.285 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00018180799999999999 2024-04-20T17:42:15.285 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.285 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.285 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:15.286 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.991728+0000", 2024-04-20T17:42:15.286 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.002598852 2024-04-20T17:42:15.286 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.286 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.286 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:15.286 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.183632+0000", 2024-04-20T17:42:15.286 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19190426799999999 2024-04-20T17:42:15.286 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.286 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.286 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:15.286 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.183662+0000", 2024-04-20T17:42:15.286 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.932e-05 2024-04-20T17:42:15.286 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.286 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.287 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:15.287 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.183684+0000", 2024-04-20T17:42:15.287 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2237000000000001e-05 2024-04-20T17:42:15.287 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.287 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:15.287 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.287 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.287 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.287 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:82598 7.7 7:e029d4d2:::benchmark_data_smithi171_161871_object82597:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e708)", 2024-04-20T17:42:15.287 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:41.007885+0000", 2024-04-20T17:42:15.287 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 213.643376872, 2024-04-20T17:42:15.287 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.17593040200000001, 2024-04-20T17:42:15.287 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:15.287 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:15.287 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:15.287 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:15.288 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:15.288 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:15.288 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 82598 2024-04-20T17:42:15.288 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.288 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:15.288 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.288 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:15.288 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.007885+0000", 2024-04-20T17:42:15.288 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.288 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.288 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.288 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:15.288 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.007885+0000", 2024-04-20T17:42:15.288 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.288 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.289 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.289 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:15.289 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.007888+0000", 2024-04-20T17:42:15.289 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8949999999999998e-06 2024-04-20T17:42:15.289 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.289 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.289 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:15.289 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.007915+0000", 2024-04-20T17:42:15.289 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7611999999999999e-05 2024-04-20T17:42:15.289 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.289 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.289 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:15.289 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.007933+0000", 2024-04-20T17:42:15.289 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7869000000000001e-05 2024-04-20T17:42:15.289 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.290 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.290 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:15.290 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.007942+0000", 2024-04-20T17:42:15.290 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0380000000000005e-06 2024-04-20T17:42:15.290 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.290 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.290 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:15.290 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.008258+0000", 2024-04-20T17:42:15.290 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00031539600000000001 2024-04-20T17:42:15.290 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.290 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.290 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:15.290 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.008407+0000", 2024-04-20T17:42:15.290 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000149553 2024-04-20T17:42:15.290 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.291 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.291 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:42:15.291 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.008551+0000", 2024-04-20T17:42:15.291 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000144306 2024-04-20T17:42:15.291 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.291 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.291 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:15.291 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.013921+0000", 2024-04-20T17:42:15.291 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0053692439999999996 2024-04-20T17:42:15.291 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.291 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.291 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:15.291 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.183723+0000", 2024-04-20T17:42:15.291 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16980257400000001 2024-04-20T17:42:15.291 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.291 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.292 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:15.292 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.183771+0000", 2024-04-20T17:42:15.292 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8118000000000002e-05 2024-04-20T17:42:15.292 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.292 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.292 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:15.292 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.183815+0000", 2024-04-20T17:42:15.292 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3797000000000001e-05 2024-04-20T17:42:15.292 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.292 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:15.292 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.292 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.292 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.292 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:82606 7.7 7:e9cc89d2:::benchmark_data_smithi171_161871_object82605:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e708)", 2024-04-20T17:42:15.292 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:41.011168+0000", 2024-04-20T17:42:15.293 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 213.64009373100001, 2024-04-20T17:42:15.293 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.18523245999999999, 2024-04-20T17:42:15.293 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:15.293 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:15.293 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:15.293 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:15.293 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:15.293 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:15.293 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 82606 2024-04-20T17:42:15.293 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.293 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:15.293 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.293 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:15.293 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.011168+0000", 2024-04-20T17:42:15.293 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.294 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.294 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.294 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:15.294 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.011168+0000", 2024-04-20T17:42:15.294 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.294 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.294 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.294 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:15.294 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.011170+0000", 2024-04-20T17:42:15.294 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9920000000000002e-06 2024-04-20T17:42:15.294 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.294 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.294 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:15.294 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.011194+0000", 2024-04-20T17:42:15.294 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3745000000000001e-05 2024-04-20T17:42:15.295 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.295 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.295 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:15.295 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.011216+0000", 2024-04-20T17:42:15.295 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2075000000000001e-05 2024-04-20T17:42:15.295 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.295 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.295 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:15.295 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.011224+0000", 2024-04-20T17:42:15.295 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5890000000000003e-06 2024-04-20T17:42:15.295 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.295 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.295 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:15.295 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.011261+0000", 2024-04-20T17:42:15.295 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6606000000000002e-05 2024-04-20T17:42:15.296 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.296 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.296 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:15.296 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.011359+0000", 2024-04-20T17:42:15.296 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7948000000000005e-05 2024-04-20T17:42:15.296 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.296 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.296 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:42:15.296 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.011441+0000", 2024-04-20T17:42:15.296 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2381999999999998e-05 2024-04-20T17:42:15.296 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.296 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.296 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:15.296 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.017231+0000", 2024-04-20T17:42:15.296 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0057897199999999999 2024-04-20T17:42:15.297 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.297 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.297 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:15.297 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.196333+0000", 2024-04-20T17:42:15.297 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.17910151999999999 2024-04-20T17:42:15.297 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.297 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.297 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:15.297 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.196371+0000", 2024-04-20T17:42:15.297 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8767999999999999e-05 2024-04-20T17:42:15.297 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.297 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.297 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:15.297 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.196400+0000", 2024-04-20T17:42:15.297 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9115000000000002e-05 2024-04-20T17:42:15.298 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.298 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:15.298 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.298 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.298 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.298 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:82617 7.7 7:e0a32035:::benchmark_data_smithi171_161871_object82616:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e708)", 2024-04-20T17:42:15.298 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:41.019537+0000", 2024-04-20T17:42:15.298 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 213.631724924, 2024-04-20T17:42:15.298 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.176973201, 2024-04-20T17:42:15.298 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:15.298 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:15.298 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:15.298 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:15.298 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:15.298 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:15.299 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 82617 2024-04-20T17:42:15.299 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.299 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:15.299 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.299 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:15.299 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.019537+0000", 2024-04-20T17:42:15.299 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.299 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.299 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.299 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:15.299 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.019537+0000", 2024-04-20T17:42:15.299 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.299 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.299 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.299 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:15.300 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.019539+0000", 2024-04-20T17:42:15.300 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.902e-06 2024-04-20T17:42:15.300 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.300 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.300 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:15.300 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.019560+0000", 2024-04-20T17:42:15.300 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1610000000000001e-05 2024-04-20T17:42:15.300 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.300 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.300 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:15.300 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.019572+0000", 2024-04-20T17:42:15.300 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1946e-05 2024-04-20T17:42:15.300 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.300 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.300 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:15.301 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.019578+0000", 2024-04-20T17:42:15.301 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8930000000000002e-06 2024-04-20T17:42:15.301 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.301 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.301 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:15.301 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.019618+0000", 2024-04-20T17:42:15.301 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0086999999999998e-05 2024-04-20T17:42:15.301 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.301 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.301 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:15.301 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.019707+0000", 2024-04-20T17:42:15.301 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8913999999999996e-05 2024-04-20T17:42:15.301 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.301 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.301 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:42:15.301 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.019796+0000", 2024-04-20T17:42:15.302 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9030999999999995e-05 2024-04-20T17:42:15.302 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.302 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.302 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:15.302 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.022247+0000", 2024-04-20T17:42:15.302 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0024506469999999998 2024-04-20T17:42:15.302 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.302 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.302 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:15.302 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.196433+0000", 2024-04-20T17:42:15.302 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.17418594200000001 2024-04-20T17:42:15.302 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.302 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.302 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:15.302 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.196476+0000", 2024-04-20T17:42:15.303 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3244999999999997e-05 2024-04-20T17:42:15.303 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.303 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.303 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:15.303 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.196510+0000", 2024-04-20T17:42:15.303 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3983999999999999e-05 2024-04-20T17:42:15.303 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.303 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:15.303 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.303 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.303 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.303 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:82643 7.7 7:ec0d525f:::benchmark_data_smithi171_161871_object82642:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e708)", 2024-04-20T17:42:15.303 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:41.036081+0000", 2024-04-20T17:42:15.303 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 213.61518088400001, 2024-04-20T17:42:15.303 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16553795199999999, 2024-04-20T17:42:15.304 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:15.304 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:15.304 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:15.304 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:15.304 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:15.304 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:15.304 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 82643 2024-04-20T17:42:15.304 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.304 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:15.304 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.304 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:15.304 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036081+0000", 2024-04-20T17:42:15.304 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.304 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.304 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.305 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:15.305 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036081+0000", 2024-04-20T17:42:15.305 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.305 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.305 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.305 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:15.305 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036084+0000", 2024-04-20T17:42:15.305 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2949999999999998e-06 2024-04-20T17:42:15.305 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.305 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.305 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:15.305 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036122+0000", 2024-04-20T17:42:15.305 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8078999999999999e-05 2024-04-20T17:42:15.305 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.305 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.306 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:15.306 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036141+0000", 2024-04-20T17:42:15.306 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8828e-05 2024-04-20T17:42:15.306 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.306 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.306 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:15.306 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036151+0000", 2024-04-20T17:42:15.306 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.842e-06 2024-04-20T17:42:15.306 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.306 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.306 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:15.306 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036224+0000", 2024-04-20T17:42:15.306 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3205000000000001e-05 2024-04-20T17:42:15.306 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.306 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.307 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:15.307 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036372+0000", 2024-04-20T17:42:15.307 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014762900000000001 2024-04-20T17:42:15.307 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.307 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.307 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:42:15.307 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036540+0000", 2024-04-20T17:42:15.307 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000167827 2024-04-20T17:42:15.307 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.307 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.307 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:15.307 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.041216+0000", 2024-04-20T17:42:15.307 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00467652 2024-04-20T17:42:15.307 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.307 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.308 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:15.308 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.201518+0000", 2024-04-20T17:42:15.308 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16030146300000001 2024-04-20T17:42:15.308 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.308 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.308 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:15.308 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.201574+0000", 2024-04-20T17:42:15.308 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6100999999999997e-05 2024-04-20T17:42:15.308 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.308 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.308 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:15.308 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.201619+0000", 2024-04-20T17:42:15.308 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5163000000000001e-05 2024-04-20T17:42:15.308 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.308 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:15.309 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.309 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.309 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.309 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:82644 7.7 7:e854f808:::benchmark_data_smithi171_161871_object82643:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e708)", 2024-04-20T17:42:15.309 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:41.036171+0000", 2024-04-20T17:42:15.309 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 213.61509097000001, 2024-04-20T17:42:15.309 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16557176500000001, 2024-04-20T17:42:15.309 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:15.309 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:15.309 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:15.309 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:15.309 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:15.309 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:15.309 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 82644 2024-04-20T17:42:15.309 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.310 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:15.310 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.310 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:15.310 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036171+0000", 2024-04-20T17:42:15.310 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.310 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.310 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.310 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:15.310 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036171+0000", 2024-04-20T17:42:15.310 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.310 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.310 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.310 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:15.310 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036172+0000", 2024-04-20T17:42:15.310 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7800000000000001e-07 2024-04-20T17:42:15.311 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.311 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.311 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:15.311 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036192+0000", 2024-04-20T17:42:15.311 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0169e-05 2024-04-20T17:42:15.311 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.311 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.311 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:15.311 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036198+0000", 2024-04-20T17:42:15.311 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0260000000000004e-06 2024-04-20T17:42:15.311 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.311 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.311 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:15.311 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036211+0000", 2024-04-20T17:42:15.311 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3752e-05 2024-04-20T17:42:15.312 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.312 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.312 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:15.312 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.038883+0000", 2024-04-20T17:42:15.312 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0026713069999999999 2024-04-20T17:42:15.312 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.312 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.312 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:15.312 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.039025+0000", 2024-04-20T17:42:15.312 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000141936 2024-04-20T17:42:15.312 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.312 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.312 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:42:15.312 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.039167+0000", 2024-04-20T17:42:15.312 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014259900000000001 2024-04-20T17:42:15.313 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.313 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.313 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:15.313 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.041396+0000", 2024-04-20T17:42:15.313 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0022285310000000002 2024-04-20T17:42:15.313 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.313 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.313 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:15.313 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.201664+0000", 2024-04-20T17:42:15.313 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.160267979 2024-04-20T17:42:15.313 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.313 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.314 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:15.314 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.201713+0000", 2024-04-20T17:42:15.314 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9379000000000003e-05 2024-04-20T17:42:15.314 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.314 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.314 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:15.314 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.201742+0000", 2024-04-20T17:42:15.314 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9309e-05 2024-04-20T17:42:15.314 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.314 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:15.314 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.314 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.314 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.314 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:82651 7.7 7:e73abe37:::benchmark_data_smithi171_161871_object82650:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e708)", 2024-04-20T17:42:15.315 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:41.041427+0000", 2024-04-20T17:42:15.315 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 213.609834355, 2024-04-20T17:42:15.315 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16039520500000001, 2024-04-20T17:42:15.315 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:15.315 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:15.315 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:15.315 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:15.315 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:15.315 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:15.315 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 82651 2024-04-20T17:42:15.315 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.315 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:15.315 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.315 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:15.315 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.041427+0000", 2024-04-20T17:42:15.315 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.316 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.316 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.316 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:15.316 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.041427+0000", 2024-04-20T17:42:15.316 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.316 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.316 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.316 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:15.316 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.041430+0000", 2024-04-20T17:42:15.316 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2000000000000001e-06 2024-04-20T17:42:15.318 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.318 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.318 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:15.318 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.041456+0000", 2024-04-20T17:42:15.318 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6341000000000001e-05 2024-04-20T17:42:15.318 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.318 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.318 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:15.319 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.041469+0000", 2024-04-20T17:42:15.319 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.288e-05 2024-04-20T17:42:15.319 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.319 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.319 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:15.319 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.041477+0000", 2024-04-20T17:42:15.319 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8169999999999999e-06 2024-04-20T17:42:15.319 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.319 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.319 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:15.319 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.041538+0000", 2024-04-20T17:42:15.319 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1413000000000003e-05 2024-04-20T17:42:15.319 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.319 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.319 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:15.320 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.041665+0000", 2024-04-20T17:42:15.320 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012729399999999999 2024-04-20T17:42:15.320 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.320 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.320 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:42:15.320 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.041783+0000", 2024-04-20T17:42:15.320 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011803699999999999 2024-04-20T17:42:15.320 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.320 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.320 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:15.320 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.043544+0000", 2024-04-20T17:42:15.320 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0017604140000000001 2024-04-20T17:42:15.320 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.320 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.320 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:15.320 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.201771+0000", 2024-04-20T17:42:15.321 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15822717 2024-04-20T17:42:15.321 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.321 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.321 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:15.321 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.201804+0000", 2024-04-20T17:42:15.321 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2793000000000003e-05 2024-04-20T17:42:15.321 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.321 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.321 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:15.321 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.201823+0000", 2024-04-20T17:42:15.321 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8845999999999999e-05 2024-04-20T17:42:15.321 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.321 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:15.321 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.321 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.322 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.322 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:82693 7.7 7:ecbc6ede:::benchmark_data_smithi171_161871_object82692:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e708)", 2024-04-20T17:42:15.322 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:41.071341+0000", 2024-04-20T17:42:15.322 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 213.57992083400001, 2024-04-20T17:42:15.322 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.130543833, 2024-04-20T17:42:15.322 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:15.322 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:15.322 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:15.322 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:15.322 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:15.322 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:15.322 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 82693 2024-04-20T17:42:15.322 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.322 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:15.322 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.323 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:15.323 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.071341+0000", 2024-04-20T17:42:15.323 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.323 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.323 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.323 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:15.323 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.071341+0000", 2024-04-20T17:42:15.323 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:15.323 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.323 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.323 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:15.323 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.071344+0000", 2024-04-20T17:42:15.323 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8439999999999998e-06 2024-04-20T17:42:15.323 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.323 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.323 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:15.324 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.071379+0000", 2024-04-20T17:42:15.324 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4795000000000001e-05 2024-04-20T17:42:15.324 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.324 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.324 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:15.324 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.071394+0000", 2024-04-20T17:42:15.324 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5726000000000002e-05 2024-04-20T17:42:15.324 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.324 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.324 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:15.324 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.071403+0000", 2024-04-20T17:42:15.324 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.67e-06 2024-04-20T17:42:15.324 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.324 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.324 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:15.325 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.071451+0000", 2024-04-20T17:42:15.325 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7914999999999998e-05 2024-04-20T17:42:15.325 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.325 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.325 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:15.325 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.071561+0000", 2024-04-20T17:42:15.325 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000110608 2024-04-20T17:42:15.325 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.325 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.325 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:42:15.325 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.071658+0000", 2024-04-20T17:42:15.325 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6527999999999995e-05 2024-04-20T17:42:15.325 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.325 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.325 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:15.325 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.073830+0000", 2024-04-20T17:42:15.326 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0021715770000000001 2024-04-20T17:42:15.326 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.326 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.326 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:15.326 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.201845+0000", 2024-04-20T17:42:15.326 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.12801509 2024-04-20T17:42:15.326 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.326 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.326 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:15.326 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.201868+0000", 2024-04-20T17:42:15.326 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3156e-05 2024-04-20T17:42:15.326 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:15.326 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:15.326 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:15.326 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.201885+0000", 2024-04-20T17:42:15.327 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6923999999999999e-05 2024-04-20T17:42:15.327 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.327 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:15.327 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.327 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:15.327 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:15.327 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:42:15.328 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:14 smithi106 ceph-mon[44423]: pgmap v2030: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 2.5 GiB data, 9.3 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 1.00k op/s; 2531/80898 objects degraded (3.129%); 4.0 MiB/s, 64 objects/s recovering 2024-04-20T17:42:15.350 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:42:15.351 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:42:15.431 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:42:15.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:15.437 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:42:15.538 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:42:15.568 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:15.590 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:42:15.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:15.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:42:15.694 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:42:15.739 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:15.879 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:42:15.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:15.884 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:42:15.933 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-20T17:42:15.985 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:42:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:15 smithi106 ceph-mon[44423]: 7.1 deep-scrub starts 2024-04-20T17:42:16.033 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:15 smithi171 ceph-mon[44189]: 7.1 deep-scrub starts 2024-04-20T17:42:16.037 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:16.073 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:42:16.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:16.079 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:42:16.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:15 smithi005 ceph-mon[39562]: 7.1 deep-scrub starts 2024-04-20T17:42:16.180 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:42:16.254 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:16.284 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:42:16.287 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:42:16.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:16.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:42:16.399 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:42:16.427 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:16.447 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:42:16.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:16.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:42:16.503 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:42:16.552 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:42:16.604 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:16.637 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:42:16.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:16.646 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:42:16.749 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:42:16.782 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:16.813 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:42:16.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:16.820 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:42:16.921 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:42:16.957 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:17.058 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:42:17.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:16 smithi171 ceph-mon[44189]: pgmap v2031: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing+deep, 15 active+clean; 2.3 GiB data, 9.0 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 892 op/s; 2264/75276 objects degraded (3.008%); 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:42:17.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:16 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 2531/80898 objects degraded (3.129%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:17.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:16 smithi005 ceph-mon[39562]: pgmap v2031: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing+deep, 15 active+clean; 2.3 GiB data, 9.0 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 892 op/s; 2264/75276 objects degraded (3.008%); 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:42:17.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:16 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 2531/80898 objects degraded (3.129%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:17.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:16 smithi106 ceph-mon[44423]: pgmap v2031: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing+deep, 15 active+clean; 2.3 GiB data, 9.0 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 892 op/s; 2264/75276 objects degraded (3.008%); 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:42:17.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:16 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 2531/80898 objects degraded (3.129%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:17.352 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:42:17.894 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:42:17.894 INFO:teuthology.orchestra.run.smithi005.stdout:{"fsid":"13c6e76c-ff37-11ee-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2264/75276 objects degraded (3.008%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":2506,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":756,"num_osds":12,"num_up_osds":11,"osd_up_since":1713634774,"num_in_osds":9,"osd_in_since":1713634793,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+clean+scrubbing+deep","count":1},{"state_name":"active+recovering+degraded","count":1}],"num_pgs":17,"num_pools":2,"num_objects":37638,"data_bytes":2467037752,"bytes_used":9612292096,"bytes_avail":854343028736,"bytes_total":863955320832,"degraded_objects":2264,"degraded_total":75276,"degraded_ratio":0.030075987034380147,"recovering_objects_per_sec":54,"recovering_bytes_per_sec":3601641,"recovering_keys_per_sec":0,"num_objects_recovered":550,"num_bytes_recovered":36044800,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":892},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.5:8443/"}},"servicemap":{"epoch":57,"modified":"2024-04-20T17:41:36.531241+0000","services":{}},"progress_events":{}} 2024-04-20T17:42:18.037 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:17 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/21900151' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:42:18.114 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:42:18.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:18.119 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:42:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:17 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/21900151' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:42:18.220 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:42:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:17 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/21900151' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:42:18.272 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:18.378 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:42:18.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:18.387 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:42:18.488 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:42:18.535 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:18.736 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:42:18.736 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:42:18.736 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:18.736 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.27088.0:256679 7.5 e756/694 7:a81008e8:::benchmark_data_smithi171_161871_object110953:head v 756'16117, mlcod=756'16115)", 2024-04-20T17:42:18.736 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:42:18.734829+0000", 2024-04-20T17:42:18.736 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.0013968769999999999, 2024-04-20T17:42:18.736 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0014233819999999999, 2024-04-20T17:42:18.736 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:18.736 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:18.736 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:42:18.736 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:18.737 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:18.737 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:18.737 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:18.734829+0000", 2024-04-20T17:42:18.737 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:18.737 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:18.737 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:18.737 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:18.737 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:18.734829+0000", 2024-04-20T17:42:18.737 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:18.737 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:18.737 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:18.737 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:18.737 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:18.734830+0000", 2024-04-20T17:42:18.737 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4920000000000001e-06 2024-04-20T17:42:18.737 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:18.737 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:18.738 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:18.738 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:18.734838+0000", 2024-04-20T17:42:18.738 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7370000000000006e-06 2024-04-20T17:42:18.738 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:18.738 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:18.738 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:18.738 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:18.734849+0000", 2024-04-20T17:42:18.738 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1253999999999999e-05 2024-04-20T17:42:18.738 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:18.738 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:18.738 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:18.738 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:18.734857+0000", 2024-04-20T17:42:18.738 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1480000000000006e-06 2024-04-20T17:42:18.738 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:18.739 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:18.739 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:18.739 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:18.734927+0000", 2024-04-20T17:42:18.739 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9632e-05 2024-04-20T17:42:18.739 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:18.739 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:18.739 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:18.739 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:18.734941+0000", 2024-04-20T17:42:18.739 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4447000000000001e-05 2024-04-20T17:42:18.739 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:18.739 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:18.739 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:18.739 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:18.739 INFO:teuthology.orchestra.run.smithi171.stdout: ], 2024-04-20T17:42:18.739 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 1 2024-04-20T17:42:18.740 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:42:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:18 smithi171 ceph-mon[44189]: pgmap v2032: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing+deep, 15 active+clean; 2.3 GiB data, 9.0 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 892 op/s; 2264/75276 objects degraded (3.008%); 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:42:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:18 smithi005 ceph-mon[39562]: pgmap v2032: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing+deep, 15 active+clean; 2.3 GiB data, 9.0 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 892 op/s; 2264/75276 objects degraded (3.008%); 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:42:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:18 smithi106 ceph-mon[44423]: pgmap v2032: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing+deep, 15 active+clean; 2.3 GiB data, 9.0 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 892 op/s; 2264/75276 objects degraded (3.008%); 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:42:19.471 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-20T17:42:19.853 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:42:20.413 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:20 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:42:20.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:20 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:42:20.467 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:42:20.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:20.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:42:20.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:20 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:42:20.574 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:42:20.618 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:20.680 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:42:20.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:20.686 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:42:20.787 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:42:20.856 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:20.974 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:42:20.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:20.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:42:21.022 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-20T17:42:21.022 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:42:21.085 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:42:21.167 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:21.237 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:42:21.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:21.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:42:21.345 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:42:21.373 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:21 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:42:21.373 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:21 smithi171 ceph-mon[44189]: pgmap v2033: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing+deep, 15 active+clean; 2.2 GiB data, 8.7 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 881 op/s; 2264/72176 objects degraded (3.137%); 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:42:21.377 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:21.405 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:42:21.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:21.408 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:42:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:21 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:42:21.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:21 smithi005 ceph-mon[39562]: pgmap v2033: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing+deep, 15 active+clean; 2.2 GiB data, 8.7 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 881 op/s; 2264/72176 objects degraded (3.137%); 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:42:21.508 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:42:21.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:21 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:42:21.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:21 smithi106 ceph-mon[44423]: pgmap v2033: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing+deep, 15 active+clean; 2.2 GiB data, 8.7 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 881 op/s; 2264/72176 objects degraded (3.137%); 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:42:21.559 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:21.608 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:42:21.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:21.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:42:21.628 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:42:21.695 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:42:21.718 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:42:21.745 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:21.765 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:42:21.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:21.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:42:21.869 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:42:21.930 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:22.204 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:22 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 2264/72176 objects degraded (3.137%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:22 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 2264/72176 objects degraded (3.137%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:22.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:22 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 2264/72176 objects degraded (3.137%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:22.588 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:42:22.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:22.591 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:42:22.692 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:42:22.739 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:22.760 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:42:22.760 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:42:22.760 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-04-20T17:42:22.760 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-04-20T17:42:22.760 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:42:23.126 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:23 smithi005 ceph-mon[39562]: 7.a scrub starts 2024-04-20T17:42:23.126 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:23 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/343886311' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:42:23.126 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:23 smithi005 ceph-mon[39562]: from='client.27458 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:42:23.127 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:23 smithi005 ceph-mon[39562]: 7.a scrub ok 2024-04-20T17:42:23.127 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:23 smithi005 ceph-mon[39562]: pgmap v2034: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing+deep, 15 active+clean; 2.0 GiB data, 8.5 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 950 op/s; 1983/66828 objects degraded (2.967%); 4.4 MiB/s, 70 objects/s recovering 2024-04-20T17:42:23.137 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:42:23.480 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:23 smithi171 ceph-mon[44189]: 7.a scrub starts 2024-04-20T17:42:23.480 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:23 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/343886311' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:42:23.480 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:23 smithi171 ceph-mon[44189]: from='client.27458 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:42:23.480 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:23 smithi171 ceph-mon[44189]: 7.a scrub ok 2024-04-20T17:42:23.481 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:23 smithi171 ceph-mon[44189]: pgmap v2034: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing+deep, 15 active+clean; 2.0 GiB data, 8.5 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 950 op/s; 1983/66828 objects degraded (2.967%); 4.4 MiB/s, 70 objects/s recovering 2024-04-20T17:42:23.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:23 smithi106 ceph-mon[44423]: 7.a scrub starts 2024-04-20T17:42:23.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:23 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/343886311' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:42:23.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:23 smithi106 ceph-mon[44423]: from='client.27458 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:42:23.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:23 smithi106 ceph-mon[44423]: 7.a scrub ok 2024-04-20T17:42:23.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:23 smithi106 ceph-mon[44423]: pgmap v2034: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing+deep, 15 active+clean; 2.0 GiB data, 8.5 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 950 op/s; 1983/66828 objects degraded (2.967%); 4.4 MiB/s, 70 objects/s recovering 2024-04-20T17:42:23.522 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-20T17:42:23.894 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:42:24.425 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:24 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:42:24.426 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:24 smithi171 ceph-mon[44189]: from='client.27458 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:42:24.426 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:24 smithi171 ceph-mon[44189]: osdmap e757: 12 total, 11 up, 9 in 2024-04-20T17:42:24.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:24 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:42:24.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:24 smithi005 ceph-mon[39562]: from='client.27458 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:42:24.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:24 smithi005 ceph-mon[39562]: osdmap e757: 12 total, 11 up, 9 in 2024-04-20T17:42:24.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:24 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:42:24.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:24 smithi106 ceph-mon[44423]: from='client.27458 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:42:24.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:24 smithi106 ceph-mon[44423]: osdmap e757: 12 total, 11 up, 9 in 2024-04-20T17:42:25.203 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:42:25.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:25.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:42:25.313 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:42:25.355 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:25.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:25 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:42:25.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:25 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:42:25.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:25 smithi005 ceph-mon[39562]: osdmap e758: 12 total, 11 up, 9 in 2024-04-20T17:42:25.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:25 smithi005 ceph-mon[39562]: pgmap v2037: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing+deep, 15 active+clean; 1.9 GiB data, 8.3 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 731 op/s; 1983/63574 objects degraded (3.119%); 2.2 MiB/s, 35 objects/s recovering 2024-04-20T17:42:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:25 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:42:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:25 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:42:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:25 smithi106 ceph-mon[44423]: osdmap e758: 12 total, 11 up, 9 in 2024-04-20T17:42:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:25 smithi106 ceph-mon[44423]: pgmap v2037: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing+deep, 15 active+clean; 1.9 GiB data, 8.3 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 731 op/s; 1983/63574 objects degraded (3.119%); 2.2 MiB/s, 35 objects/s recovering 2024-04-20T17:42:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:25 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:42:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:25 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:42:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:25 smithi171 ceph-mon[44189]: osdmap e758: 12 total, 11 up, 9 in 2024-04-20T17:42:25.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:25 smithi171 ceph-mon[44189]: pgmap v2037: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing+deep, 15 active+clean; 1.9 GiB data, 8.3 GiB used, 796 GiB / 805 GiB avail; 0 B/s wr, 731 op/s; 1983/63574 objects degraded (3.119%); 2.2 MiB/s, 35 objects/s recovering 2024-04-20T17:42:25.712 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:42:25.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:25.715 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:42:25.816 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:42:25.862 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:26.227 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:42:26.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:26.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:42:26.334 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:42:26.371 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:42:26.371 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:42:26.371 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:42:26.371 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:42:26.371 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.371 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:6148 7.0 7:08ab8af0:::benchmark_data_smithi171_161871_object6147:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e699)", 2024-04-20T17:42:26.372 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:09.851521+0000", 2024-04-20T17:42:26.372 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 256.51593802299999, 2024-04-20T17:42:26.372 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043601052000000001, 2024-04-20T17:42:26.372 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:26.372 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:26.372 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:26.372 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:26.372 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:26.372 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:26.372 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 6148 2024-04-20T17:42:26.372 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.372 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:26.372 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.372 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:26.373 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:09.851521+0000", 2024-04-20T17:42:26.373 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.373 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.373 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.373 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:26.373 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:09.851521+0000", 2024-04-20T17:42:26.373 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.373 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.373 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.373 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:26.373 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:09.851522+0000", 2024-04-20T17:42:26.373 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.516e-06 2024-04-20T17:42:26.373 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.373 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.374 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:26.374 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:09.851550+0000", 2024-04-20T17:42:26.374 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7492e-05 2024-04-20T17:42:26.374 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.374 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.374 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:26.374 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:09.851559+0000", 2024-04-20T17:42:26.374 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9879999999999993e-06 2024-04-20T17:42:26.374 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.374 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.374 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:26.374 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:09.851566+0000", 2024-04-20T17:42:26.374 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.041e-06 2024-04-20T17:42:26.374 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.375 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.375 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:26.375 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:09.851624+0000", 2024-04-20T17:42:26.375 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7698999999999998e-05 2024-04-20T17:42:26.375 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.375 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.375 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:26.375 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:09.851782+0000", 2024-04-20T17:42:26.375 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000158517 2024-04-20T17:42:26.375 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.375 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.375 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:42:26.375 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:09.851923+0000", 2024-04-20T17:42:26.375 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014044099999999999 2024-04-20T17:42:26.376 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.376 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.376 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:26.376 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:09.892774+0000", 2024-04-20T17:42:26.376 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.040851493000000003 2024-04-20T17:42:26.376 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.376 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.376 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:26.376 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:09.895045+0000", 2024-04-20T17:42:26.376 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0022707420000000001 2024-04-20T17:42:26.376 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.376 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.376 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:26.376 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:09.895090+0000", 2024-04-20T17:42:26.377 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5086000000000002e-05 2024-04-20T17:42:26.377 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.377 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.377 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:26.377 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:09.895122+0000", 2024-04-20T17:42:26.377 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2036999999999998e-05 2024-04-20T17:42:26.377 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.377 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:26.377 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.377 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.377 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.377 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:6150 7.0 7:083da41e:::benchmark_data_smithi171_161871_object6149:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e699)", 2024-04-20T17:42:26.377 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:09.852187+0000", 2024-04-20T17:42:26.377 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 256.51527184600002, 2024-04-20T17:42:26.378 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043050537999999999, 2024-04-20T17:42:26.378 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:26.378 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:26.378 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:26.378 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:26.378 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:26.378 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:26.378 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 6150 2024-04-20T17:42:26.378 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.378 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:26.378 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.378 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:26.378 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:09.852187+0000", 2024-04-20T17:42:26.378 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.378 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.379 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.379 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:26.379 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:09.852187+0000", 2024-04-20T17:42:26.379 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.379 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.379 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.379 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:26.379 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:09.852189+0000", 2024-04-20T17:42:26.379 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6020000000000001e-06 2024-04-20T17:42:26.379 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.379 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.379 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:26.379 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:09.852218+0000", 2024-04-20T17:42:26.379 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9634000000000001e-05 2024-04-20T17:42:26.380 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.380 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.380 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:26.380 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:09.892362+0000", 2024-04-20T17:42:26.380 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.040143858999999997 2024-04-20T17:42:26.380 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.380 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.380 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:26.380 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:09.892372+0000", 2024-04-20T17:42:26.380 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7569999999999999e-06 2024-04-20T17:42:26.380 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.380 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.380 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:26.380 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:09.892812+0000", 2024-04-20T17:42:26.380 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000439806 2024-04-20T17:42:26.381 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.381 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.381 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:26.381 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:09.892942+0000", 2024-04-20T17:42:26.381 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012975700000000001 2024-04-20T17:42:26.381 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.381 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.381 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:42:26.381 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:09.893077+0000", 2024-04-20T17:42:26.381 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013541399999999999 2024-04-20T17:42:26.381 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.381 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.381 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:26.381 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:09.895127+0000", 2024-04-20T17:42:26.381 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0020497390000000001 2024-04-20T17:42:26.382 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.382 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.382 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:26.382 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:09.895152+0000", 2024-04-20T17:42:26.382 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5415e-05 2024-04-20T17:42:26.382 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.382 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.382 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:26.382 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:09.895195+0000", 2024-04-20T17:42:26.382 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2826e-05 2024-04-20T17:42:26.382 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.382 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.382 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:26.382 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:09.895238+0000", 2024-04-20T17:42:26.382 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2728999999999997e-05 2024-04-20T17:42:26.383 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.383 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:26.383 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.383 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.383 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.383 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:19765 7.a 7:5a0ed91e:::benchmark_data_smithi171_161871_object19764:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e700)", 2024-04-20T17:42:26.383 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:13.097092+0000", 2024-04-20T17:42:26.383 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 253.270367311, 2024-04-20T17:42:26.383 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.055692644999999999, 2024-04-20T17:42:26.383 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:26.383 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:26.383 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:26.383 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:26.383 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:26.383 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:26.384 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 19765 2024-04-20T17:42:26.384 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.384 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:26.384 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.384 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:26.384 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:13.097092+0000", 2024-04-20T17:42:26.384 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.384 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.384 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.384 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:26.384 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:13.097092+0000", 2024-04-20T17:42:26.384 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.384 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.384 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.384 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:26.385 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:13.097093+0000", 2024-04-20T17:42:26.385 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.051e-06 2024-04-20T17:42:26.385 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.385 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.385 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:26.385 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:13.097126+0000", 2024-04-20T17:42:26.385 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3216000000000002e-05 2024-04-20T17:42:26.385 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.385 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.385 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:26.385 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:13.097147+0000", 2024-04-20T17:42:26.385 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1251e-05 2024-04-20T17:42:26.385 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.385 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.385 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:26.386 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:13.097154+0000", 2024-04-20T17:42:26.386 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2130000000000002e-06 2024-04-20T17:42:26.386 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.386 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.386 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:26.386 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:13.097227+0000", 2024-04-20T17:42:26.386 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2568e-05 2024-04-20T17:42:26.386 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.386 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.386 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:26.386 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:13.097340+0000", 2024-04-20T17:42:26.386 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000113188 2024-04-20T17:42:26.386 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.386 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.387 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:42:26.387 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:13.097476+0000", 2024-04-20T17:42:26.387 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000135647 2024-04-20T17:42:26.387 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.387 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.387 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:26.387 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:13.099355+0000", 2024-04-20T17:42:26.387 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0018792290000000001 2024-04-20T17:42:26.387 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.387 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.387 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:26.387 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:13.152713+0000", 2024-04-20T17:42:26.387 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.053358134000000002 2024-04-20T17:42:26.387 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.387 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.388 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:26.388 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:13.152756+0000", 2024-04-20T17:42:26.388 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2777000000000001e-05 2024-04-20T17:42:26.388 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.388 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.388 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:26.388 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:13.152784+0000", 2024-04-20T17:42:26.388 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8371e-05 2024-04-20T17:42:26.388 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.388 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:26.388 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.388 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.388 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.388 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:22649 7.a 7:555cb4bc:::benchmark_data_smithi171_161871_object22648:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e701)", 2024-04-20T17:42:26.388 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:13.897933+0000", 2024-04-20T17:42:26.389 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 252.46952562199999, 2024-04-20T17:42:26.389 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.056757481999999998, 2024-04-20T17:42:26.389 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:26.389 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:26.389 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:26.389 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:26.389 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:26.389 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:26.389 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 22649 2024-04-20T17:42:26.389 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.389 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:26.389 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.389 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:26.389 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:13.897933+0000", 2024-04-20T17:42:26.389 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.390 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.390 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.390 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:26.390 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:13.897933+0000", 2024-04-20T17:42:26.390 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.390 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.390 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.390 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:26.390 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:13.897935+0000", 2024-04-20T17:42:26.390 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.384e-06 2024-04-20T17:42:26.390 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.390 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.390 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:26.390 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:13.897954+0000", 2024-04-20T17:42:26.391 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9550000000000001e-05 2024-04-20T17:42:26.391 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.391 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.391 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:26.391 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:13.897964+0000", 2024-04-20T17:42:26.391 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5400000000000001e-06 2024-04-20T17:42:26.391 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.391 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.391 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:26.391 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:13.897971+0000", 2024-04-20T17:42:26.391 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5580000000000003e-06 2024-04-20T17:42:26.391 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.391 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.391 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:26.392 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:13.898019+0000", 2024-04-20T17:42:26.392 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7825999999999999e-05 2024-04-20T17:42:26.392 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.392 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.392 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:26.392 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:13.898109+0000", 2024-04-20T17:42:26.392 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0204000000000006e-05 2024-04-20T17:42:26.392 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.392 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.392 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:42:26.392 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:13.898195+0000", 2024-04-20T17:42:26.392 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5682999999999999e-05 2024-04-20T17:42:26.392 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.392 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.392 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:26.393 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:13.898980+0000", 2024-04-20T17:42:26.393 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00078532299999999999 2024-04-20T17:42:26.393 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.393 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.393 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:26.393 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:13.954593+0000", 2024-04-20T17:42:26.393 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.055612531999999999 2024-04-20T17:42:26.393 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.393 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.393 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:26.393 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:13.954653+0000", 2024-04-20T17:42:26.393 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9679999999999998e-05 2024-04-20T17:42:26.393 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.393 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.393 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:26.394 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:13.954691+0000", 2024-04-20T17:42:26.394 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8201999999999998e-05 2024-04-20T17:42:26.394 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.394 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:26.394 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.394 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.394 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.394 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:26118 7.a 7:53cb78d3:::benchmark_data_smithi171_161871_object26117:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e701)", 2024-04-20T17:42:26.394 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:14.907139+0000", 2024-04-20T17:42:26.394 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 251.46031951500001, 2024-04-20T17:42:26.394 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.078899552999999997, 2024-04-20T17:42:26.394 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:26.394 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:26.394 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:26.394 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:26.395 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:26.395 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:26.395 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 26118 2024-04-20T17:42:26.395 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.395 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:26.395 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.395 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:26.395 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:14.907139+0000", 2024-04-20T17:42:26.395 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.395 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.395 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.395 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:26.395 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:14.907139+0000", 2024-04-20T17:42:26.395 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.396 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.396 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.396 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:26.396 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:14.907141+0000", 2024-04-20T17:42:26.396 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6780000000000001e-06 2024-04-20T17:42:26.396 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.396 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.396 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:26.396 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:14.907162+0000", 2024-04-20T17:42:26.396 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0582999999999999e-05 2024-04-20T17:42:26.396 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.396 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.396 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:26.396 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:14.907172+0000", 2024-04-20T17:42:26.396 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0740999999999999e-05 2024-04-20T17:42:26.397 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.397 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.397 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:26.397 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:14.907178+0000", 2024-04-20T17:42:26.397 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5500000000000002e-06 2024-04-20T17:42:26.397 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.397 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.397 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:26.397 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:14.907240+0000", 2024-04-20T17:42:26.397 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1716000000000003e-05 2024-04-20T17:42:26.397 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.397 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.397 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:26.397 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:14.907371+0000", 2024-04-20T17:42:26.397 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000130829 2024-04-20T17:42:26.398 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.398 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.398 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:42:26.398 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:14.907514+0000", 2024-04-20T17:42:26.398 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014344 2024-04-20T17:42:26.398 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.398 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.398 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:26.398 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:14.908455+0000", 2024-04-20T17:42:26.398 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00094090500000000002 2024-04-20T17:42:26.398 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.398 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.398 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:26.399 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:14.985938+0000", 2024-04-20T17:42:26.399 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.077483545000000001 2024-04-20T17:42:26.399 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.399 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.399 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:26.399 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:14.985990+0000", 2024-04-20T17:42:26.399 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1953000000000002e-05 2024-04-20T17:42:26.399 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.399 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.399 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:26.399 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:14.986039+0000", 2024-04-20T17:42:26.399 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8612999999999997e-05 2024-04-20T17:42:26.399 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.399 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:26.400 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.400 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.400 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.400 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:45350 7.a 7:5dff9a7a:::benchmark_data_smithi171_161871_object45349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e702)", 2024-04-20T17:42:26.400 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:21.835036+0000", 2024-04-20T17:42:26.400 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 244.53242338699999, 2024-04-20T17:42:26.400 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.10152404499999999, 2024-04-20T17:42:26.400 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:26.400 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:26.400 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:26.400 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:26.400 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:26.400 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:26.400 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 45350 2024-04-20T17:42:26.401 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.401 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:26.401 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.401 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:26.401 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.835036+0000", 2024-04-20T17:42:26.401 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.401 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.401 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.401 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:26.401 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.835036+0000", 2024-04-20T17:42:26.401 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.401 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.401 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.401 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:26.401 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.835038+0000", 2024-04-20T17:42:26.402 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9910000000000001e-06 2024-04-20T17:42:26.402 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.402 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.402 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:26.402 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.835072+0000", 2024-04-20T17:42:26.402 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4782e-05 2024-04-20T17:42:26.402 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.402 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.402 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:26.402 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.835093+0000", 2024-04-20T17:42:26.402 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0842000000000001e-05 2024-04-20T17:42:26.402 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.402 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.403 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:26.403 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.835102+0000", 2024-04-20T17:42:26.403 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1980000000000007e-06 2024-04-20T17:42:26.403 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.403 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.403 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:26.403 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.835154+0000", 2024-04-20T17:42:26.403 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1415999999999997e-05 2024-04-20T17:42:26.403 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.403 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.403 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:26.403 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.835281+0000", 2024-04-20T17:42:26.403 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000127317 2024-04-20T17:42:26.403 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.403 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.404 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:42:26.404 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.835412+0000", 2024-04-20T17:42:26.404 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013133099999999999 2024-04-20T17:42:26.404 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.404 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.404 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:26.404 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.838468+0000", 2024-04-20T17:42:26.404 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0030554100000000002 2024-04-20T17:42:26.404 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.404 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.404 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:26.404 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.936497+0000", 2024-04-20T17:42:26.404 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.098029158000000005 2024-04-20T17:42:26.405 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.405 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.405 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:26.405 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.936531+0000", 2024-04-20T17:42:26.405 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4239000000000002e-05 2024-04-20T17:42:26.405 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.405 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.405 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:26.405 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.936560+0000", 2024-04-20T17:42:26.405 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8360999999999998e-05 2024-04-20T17:42:26.405 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.405 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:26.405 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.405 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.405 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.406 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:45356 7.a 7:5c3b5ecc:::benchmark_data_smithi171_161871_object45355:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e702)", 2024-04-20T17:42:26.406 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:21.835124+0000", 2024-04-20T17:42:26.406 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 244.53233449800001, 2024-04-20T17:42:26.406 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.103701316, 2024-04-20T17:42:26.406 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:26.406 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:26.406 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:26.406 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:26.406 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:26.406 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:26.406 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 45356 2024-04-20T17:42:26.406 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.406 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:26.406 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.407 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:26.407 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.835124+0000", 2024-04-20T17:42:26.407 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.407 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.407 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.407 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:26.407 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.835124+0000", 2024-04-20T17:42:26.407 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.407 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.407 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.407 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:26.407 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.835125+0000", 2024-04-20T17:42:26.407 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6299999999999993e-07 2024-04-20T17:42:26.407 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.408 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.408 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:26.408 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.835147+0000", 2024-04-20T17:42:26.408 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1097000000000001e-05 2024-04-20T17:42:26.408 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.408 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.408 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:26.408 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.835154+0000", 2024-04-20T17:42:26.408 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8439999999999998e-06 2024-04-20T17:42:26.408 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.408 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.408 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:26.408 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.835161+0000", 2024-04-20T17:42:26.408 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.387e-06 2024-04-20T17:42:26.409 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.409 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.409 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:26.409 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.836966+0000", 2024-04-20T17:42:26.409 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001804749 2024-04-20T17:42:26.409 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.409 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.409 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:26.409 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.837087+0000", 2024-04-20T17:42:26.409 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012167000000000001 2024-04-20T17:42:26.409 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.409 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.409 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:42:26.409 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.837221+0000", 2024-04-20T17:42:26.410 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013347000000000001 2024-04-20T17:42:26.410 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.410 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.410 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:26.410 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.839890+0000", 2024-04-20T17:42:26.410 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0026694280000000002 2024-04-20T17:42:26.410 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.410 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.410 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:26.410 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.938759+0000", 2024-04-20T17:42:26.410 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.098868940000000002 2024-04-20T17:42:26.410 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.410 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.410 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:26.411 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.938795+0000", 2024-04-20T17:42:26.411 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5543000000000002e-05 2024-04-20T17:42:26.411 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.411 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.411 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:26.411 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.938826+0000", 2024-04-20T17:42:26.411 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1225e-05 2024-04-20T17:42:26.411 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.411 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:26.411 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.411 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.411 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.411 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:45363 7.a 7:542f472e:::benchmark_data_smithi171_161871_object45362:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e702)", 2024-04-20T17:42:26.411 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:21.838471+0000", 2024-04-20T17:42:26.412 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 244.52898816999999, 2024-04-20T17:42:26.412 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.102010401, 2024-04-20T17:42:26.412 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:26.412 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:26.412 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:26.412 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:26.412 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:26.412 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:26.412 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 45363 2024-04-20T17:42:26.412 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.412 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:26.412 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.412 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:26.412 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.838471+0000", 2024-04-20T17:42:26.412 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.413 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.413 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.413 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:26.413 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.838471+0000", 2024-04-20T17:42:26.413 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.413 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.413 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.413 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:26.413 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.838472+0000", 2024-04-20T17:42:26.413 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.09e-07 2024-04-20T17:42:26.413 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.413 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.413 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:26.413 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.838500+0000", 2024-04-20T17:42:26.413 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8016000000000001e-05 2024-04-20T17:42:26.414 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.414 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.414 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:26.414 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.838507+0000", 2024-04-20T17:42:26.414 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0040000000000002e-06 2024-04-20T17:42:26.414 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.414 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.414 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:26.414 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.838513+0000", 2024-04-20T17:42:26.414 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7000000000000002e-06 2024-04-20T17:42:26.414 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.414 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.415 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:26.416 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.838605+0000", 2024-04-20T17:42:26.416 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1735000000000002e-05 2024-04-20T17:42:26.416 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.416 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.416 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:26.416 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.838713+0000", 2024-04-20T17:42:26.416 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000108242 2024-04-20T17:42:26.416 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.416 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.416 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:42:26.417 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.838841+0000", 2024-04-20T17:42:26.417 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000128023 2024-04-20T17:42:26.417 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.417 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.417 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:26.417 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.842301+0000", 2024-04-20T17:42:26.417 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0034592920000000001 2024-04-20T17:42:26.417 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.417 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.417 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:26.417 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.940386+0000", 2024-04-20T17:42:26.418 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.098085329999999998 2024-04-20T17:42:26.418 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.418 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.418 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:26.418 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.940440+0000", 2024-04-20T17:42:26.418 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3677999999999997e-05 2024-04-20T17:42:26.418 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.418 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.418 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:26.418 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.940481+0000", 2024-04-20T17:42:26.418 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1471999999999999e-05 2024-04-20T17:42:26.418 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.418 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:26.419 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.419 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.419 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.419 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:45364 7.a 7:53927a0d:::benchmark_data_smithi171_161871_object45363:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e702)", 2024-04-20T17:42:26.419 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:21.838529+0000", 2024-04-20T17:42:26.419 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 244.528929963, 2024-04-20T17:42:26.419 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.103431878, 2024-04-20T17:42:26.419 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:26.419 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:26.419 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:26.419 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:26.419 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:26.419 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:26.419 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 45364 2024-04-20T17:42:26.419 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.420 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:26.420 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.420 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:26.420 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.838529+0000", 2024-04-20T17:42:26.420 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.420 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.420 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.420 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:26.420 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.838529+0000", 2024-04-20T17:42:26.420 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.420 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.420 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.420 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:26.420 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.838530+0000", 2024-04-20T17:42:26.420 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5499999999999997e-07 2024-04-20T17:42:26.421 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.421 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.421 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:26.421 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.838552+0000", 2024-04-20T17:42:26.421 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2033000000000001e-05 2024-04-20T17:42:26.421 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.421 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.421 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:26.421 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.838562+0000", 2024-04-20T17:42:26.421 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0567999999999999e-05 2024-04-20T17:42:26.421 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.421 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.421 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:26.421 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.838568+0000", 2024-04-20T17:42:26.421 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0619999999999997e-06 2024-04-20T17:42:26.422 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.422 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.422 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:26.422 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.839979+0000", 2024-04-20T17:42:26.422 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001410157 2024-04-20T17:42:26.422 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.422 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.422 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:26.422 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.840110+0000", 2024-04-20T17:42:26.422 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013155 2024-04-20T17:42:26.422 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.422 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.422 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:42:26.422 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.840259+0000", 2024-04-20T17:42:26.422 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000148796 2024-04-20T17:42:26.423 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.423 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.423 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:26.423 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.842304+0000", 2024-04-20T17:42:26.423 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0020453960000000001 2024-04-20T17:42:26.423 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.423 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.423 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:26.423 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.941862+0000", 2024-04-20T17:42:26.423 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.099557774000000002 2024-04-20T17:42:26.423 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.423 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.423 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:26.423 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.941924+0000", 2024-04-20T17:42:26.424 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1655999999999994e-05 2024-04-20T17:42:26.424 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.424 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.424 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:26.424 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.941961+0000", 2024-04-20T17:42:26.424 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7131000000000001e-05 2024-04-20T17:42:26.424 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.424 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:26.424 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.424 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.424 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.424 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:45369 7.a 7:519c9243:::benchmark_data_smithi171_161871_object45368:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e702)", 2024-04-20T17:42:26.424 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:21.839907+0000", 2024-04-20T17:42:26.424 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 244.52755191599999, 2024-04-20T17:42:26.424 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.10242404500000001, 2024-04-20T17:42:26.425 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:26.425 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:26.425 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:26.425 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:26.425 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:26.425 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:26.425 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 45369 2024-04-20T17:42:26.425 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.425 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:26.425 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.425 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:26.425 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.839907+0000", 2024-04-20T17:42:26.425 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.426 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.426 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.426 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:26.426 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.839907+0000", 2024-04-20T17:42:26.426 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.426 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.426 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.426 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:26.426 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.839908+0000", 2024-04-20T17:42:26.426 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.35e-06 2024-04-20T17:42:26.426 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.426 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.426 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:26.426 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.839931+0000", 2024-04-20T17:42:26.427 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2204e-05 2024-04-20T17:42:26.427 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.427 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.427 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:26.427 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.839940+0000", 2024-04-20T17:42:26.427 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7350000000000008e-06 2024-04-20T17:42:26.427 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.427 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.427 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:26.427 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.839947+0000", 2024-04-20T17:42:26.427 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0249999999999997e-06 2024-04-20T17:42:26.427 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.427 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.427 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:26.427 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.841374+0000", 2024-04-20T17:42:26.428 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0014268200000000001 2024-04-20T17:42:26.428 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.428 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.428 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:26.428 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.841457+0000", 2024-04-20T17:42:26.428 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.3244000000000005e-05 2024-04-20T17:42:26.428 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.428 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.428 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:42:26.428 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.841542+0000", 2024-04-20T17:42:26.428 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4930000000000002e-05 2024-04-20T17:42:26.428 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.428 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.428 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:26.428 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.842439+0000", 2024-04-20T17:42:26.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00089652799999999995 2024-04-20T17:42:26.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.429 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.429 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:26.429 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.942249+0000", 2024-04-20T17:42:26.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.099810057999999993 2024-04-20T17:42:26.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.429 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.429 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:26.429 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.942295+0000", 2024-04-20T17:42:26.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6505000000000003e-05 2024-04-20T17:42:26.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.429 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.429 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:26.430 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:21.942331+0000", 2024-04-20T17:42:26.430 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5645999999999997e-05 2024-04-20T17:42:26.430 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.430 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:26.430 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.430 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.430 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.430 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:54020 7.a 7:59773ea2:::benchmark_data_smithi171_161871_object54019:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e702)", 2024-04-20T17:42:26.430 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:25.792464+0000", 2024-04-20T17:42:26.430 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 240.57499536700001, 2024-04-20T17:42:26.430 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.059870001999999999, 2024-04-20T17:42:26.430 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:26.430 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:26.430 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:26.431 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:26.431 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:26.431 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:26.431 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 54020 2024-04-20T17:42:26.431 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.431 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:26.431 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.431 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:26.431 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.792464+0000", 2024-04-20T17:42:26.431 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.431 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.431 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.431 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:26.431 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.792464+0000", 2024-04-20T17:42:26.431 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.432 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.432 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.432 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:26.432 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.792466+0000", 2024-04-20T17:42:26.432 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9460000000000001e-06 2024-04-20T17:42:26.432 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.432 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.432 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:26.432 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.792497+0000", 2024-04-20T17:42:26.432 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1383000000000001e-05 2024-04-20T17:42:26.432 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.432 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.432 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:26.432 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.792507+0000", 2024-04-20T17:42:26.432 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0011e-05 2024-04-20T17:42:26.433 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.433 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.433 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:26.433 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.792514+0000", 2024-04-20T17:42:26.433 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.294e-06 2024-04-20T17:42:26.433 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.433 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.433 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:26.433 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.792556+0000", 2024-04-20T17:42:26.433 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1656000000000002e-05 2024-04-20T17:42:26.433 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.433 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.433 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:26.433 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.792643+0000", 2024-04-20T17:42:26.434 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.7016999999999997e-05 2024-04-20T17:42:26.434 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.434 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.434 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:42:26.434 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.847572+0000", 2024-04-20T17:42:26.434 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.054929450999999997 2024-04-20T17:42:26.434 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.434 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.434 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:26.434 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.852036+0000", 2024-04-20T17:42:26.434 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00446314 2024-04-20T17:42:26.434 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.434 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.434 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:26.435 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.852232+0000", 2024-04-20T17:42:26.435 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000196653 2024-04-20T17:42:26.435 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.435 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.435 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:26.435 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.852280+0000", 2024-04-20T17:42:26.435 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8282000000000003e-05 2024-04-20T17:42:26.435 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.435 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.435 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:26.435 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.852334+0000", 2024-04-20T17:42:26.435 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3168999999999999e-05 2024-04-20T17:42:26.435 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.435 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:26.436 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.436 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.436 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.436 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:54030 7.a 7:55a70d5e:::benchmark_data_smithi171_161871_object54029:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e702)", 2024-04-20T17:42:26.436 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:25.798082+0000", 2024-04-20T17:42:26.436 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 240.569377307, 2024-04-20T17:42:26.436 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.054732375, 2024-04-20T17:42:26.436 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:26.436 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:26.436 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:26.436 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:26.436 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:26.436 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:26.436 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 54030 2024-04-20T17:42:26.436 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.437 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:26.437 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.437 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:26.437 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.798082+0000", 2024-04-20T17:42:26.437 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.437 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.437 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.437 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:26.437 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.798082+0000", 2024-04-20T17:42:26.437 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.437 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.437 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.437 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:26.437 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.798083+0000", 2024-04-20T17:42:26.437 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7090000000000001e-06 2024-04-20T17:42:26.438 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.438 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.438 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:26.438 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.798103+0000", 2024-04-20T17:42:26.438 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0034000000000002e-05 2024-04-20T17:42:26.438 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.438 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.438 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:26.438 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.847415+0000", 2024-04-20T17:42:26.438 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.049311810999999997 2024-04-20T17:42:26.438 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.438 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.438 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:26.439 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.847431+0000", 2024-04-20T17:42:26.439 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5852e-05 2024-04-20T17:42:26.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.439 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.439 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:26.439 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.849970+0000", 2024-04-20T17:42:26.439 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0025392510000000002 2024-04-20T17:42:26.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.439 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.439 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:26.439 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.850105+0000", 2024-04-20T17:42:26.439 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000135134 2024-04-20T17:42:26.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.439 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.439 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:42:26.440 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.850241+0000", 2024-04-20T17:42:26.440 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000135671 2024-04-20T17:42:26.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.440 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.440 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:26.440 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.852401+0000", 2024-04-20T17:42:26.440 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.002159453 2024-04-20T17:42:26.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.440 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.440 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:26.440 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.852725+0000", 2024-04-20T17:42:26.440 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00032477999999999999 2024-04-20T17:42:26.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.440 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.440 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:26.441 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.852777+0000", 2024-04-20T17:42:26.441 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1736000000000001e-05 2024-04-20T17:42:26.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.441 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:26.441 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.852814+0000", 2024-04-20T17:42:26.441 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6943999999999997e-05 2024-04-20T17:42:26.441 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.441 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:26.441 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.441 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:154644 7.a 7:5a2f6e3e:::benchmark_data_smithi171_161871_object8918:head [delete] snapc 0=[] ondisk+write+known_if_redirected e735)", 2024-04-20T17:42:26.441 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:40:18.965836+0000", 2024-04-20T17:42:26.442 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 127.40162319, 2024-04-20T17:42:26.442 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.060048615999999999, 2024-04-20T17:42:26.442 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:26.442 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:26.442 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:26.442 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:26.442 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:26.442 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:26.442 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 154644 2024-04-20T17:42:26.442 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.442 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:26.442 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.442 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:26.442 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:18.965836+0000", 2024-04-20T17:42:26.443 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.443 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.443 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.443 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:26.443 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:18.965836+0000", 2024-04-20T17:42:26.443 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.443 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.443 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.443 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:26.443 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:18.965839+0000", 2024-04-20T17:42:26.443 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2590000000000001e-06 2024-04-20T17:42:26.443 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.443 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.443 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:26.443 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:18.965847+0000", 2024-04-20T17:42:26.444 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0739999999999993e-06 2024-04-20T17:42:26.444 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.444 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.444 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:26.444 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:18.965862+0000", 2024-04-20T17:42:26.444 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4704e-05 2024-04-20T17:42:26.444 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.444 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.444 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:26.444 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:18.965870+0000", 2024-04-20T17:42:26.444 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5699999999999993e-06 2024-04-20T17:42:26.444 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.444 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.444 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:26.444 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:18.965929+0000", 2024-04-20T17:42:26.445 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8319999999999997e-05 2024-04-20T17:42:26.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.445 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:26.445 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:18.966025+0000", 2024-04-20T17:42:26.445 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5827000000000003e-05 2024-04-20T17:42:26.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.445 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:42:26.445 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:18.966097+0000", 2024-04-20T17:42:26.445 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2893999999999997e-05 2024-04-20T17:42:26.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.445 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:26.445 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:18.971635+0000", 2024-04-20T17:42:26.446 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0055375040000000004 2024-04-20T17:42:26.446 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.446 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.446 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:26.446 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:19.025818+0000", 2024-04-20T17:42:26.446 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.054183174000000001 2024-04-20T17:42:26.446 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.446 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.446 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:26.446 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:19.025854+0000", 2024-04-20T17:42:26.446 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5744999999999997e-05 2024-04-20T17:42:26.446 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.446 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.446 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:26.446 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:19.025884+0000", 2024-04-20T17:42:26.447 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0545e-05 2024-04-20T17:42:26.447 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.447 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:26.447 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.447 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.447 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.447 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:156493 7.a 7:543f2ba6:::benchmark_data_smithi171_161871_object10767:head [delete] snapc 0=[] ondisk+write+known_if_redirected e735)", 2024-04-20T17:42:26.447 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:40:20.914534+0000", 2024-04-20T17:42:26.447 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 125.452924899, 2024-04-20T17:42:26.447 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.077758430000000003, 2024-04-20T17:42:26.447 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:26.447 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:26.447 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:26.447 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:26.448 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:26.448 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:26.448 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 156493 2024-04-20T17:42:26.448 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.448 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:26.448 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.448 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:26.448 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.914534+0000", 2024-04-20T17:42:26.448 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.448 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.448 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.448 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:26.448 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.914534+0000", 2024-04-20T17:42:26.448 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.449 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.449 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.449 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:26.449 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.914536+0000", 2024-04-20T17:42:26.449 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2500000000000001e-06 2024-04-20T17:42:26.449 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.449 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.449 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:26.449 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.914545+0000", 2024-04-20T17:42:26.449 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2160000000000006e-06 2024-04-20T17:42:26.449 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.449 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.449 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:26.449 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.914555+0000", 2024-04-20T17:42:26.449 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0336999999999999e-05 2024-04-20T17:42:26.450 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.450 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.450 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:26.450 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.914562+0000", 2024-04-20T17:42:26.450 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2439999999999996e-06 2024-04-20T17:42:26.450 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.450 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.450 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:26.450 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.914623+0000", 2024-04-20T17:42:26.450 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1253999999999993e-05 2024-04-20T17:42:26.450 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.450 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.450 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:26.450 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.914737+0000", 2024-04-20T17:42:26.450 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000113995 2024-04-20T17:42:26.451 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.451 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.451 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:42:26.451 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.914838+0000", 2024-04-20T17:42:26.451 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000100938 2024-04-20T17:42:26.451 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.451 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.451 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:26.451 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.922688+0000", 2024-04-20T17:42:26.451 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0078500319999999998 2024-04-20T17:42:26.451 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.451 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.451 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:26.451 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.992226+0000", 2024-04-20T17:42:26.451 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.069537629000000004 2024-04-20T17:42:26.452 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.452 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.452 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:26.452 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.992262+0000", 2024-04-20T17:42:26.452 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6081000000000002e-05 2024-04-20T17:42:26.452 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.452 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.452 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:26.452 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.992293+0000", 2024-04-20T17:42:26.452 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0454e-05 2024-04-20T17:42:26.452 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.452 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:26.452 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.452 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.453 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.453 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:156497 7.a 7:51a04bb7:::benchmark_data_smithi171_161871_object10771:head [delete] snapc 0=[] ondisk+write+known_if_redirected e735)", 2024-04-20T17:42:26.453 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:40:20.918494+0000", 2024-04-20T17:42:26.453 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 125.448964965, 2024-04-20T17:42:26.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.079060783999999995, 2024-04-20T17:42:26.453 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:26.453 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:26.453 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:26.453 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:26.453 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:26.453 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:26.453 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 156497 2024-04-20T17:42:26.453 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.453 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:26.454 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.454 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:26.454 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.918494+0000", 2024-04-20T17:42:26.454 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.454 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.454 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.454 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:26.454 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.918494+0000", 2024-04-20T17:42:26.454 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.454 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.454 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.454 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:26.454 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.918496+0000", 2024-04-20T17:42:26.455 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0140000000000001e-06 2024-04-20T17:42:26.455 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.455 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.455 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:26.455 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.918502+0000", 2024-04-20T17:42:26.455 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1700000000000002e-06 2024-04-20T17:42:26.455 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.455 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.455 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:26.455 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.918514+0000", 2024-04-20T17:42:26.455 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2133000000000001e-05 2024-04-20T17:42:26.455 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.455 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.455 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:26.455 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.918524+0000", 2024-04-20T17:42:26.456 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3190000000000001e-06 2024-04-20T17:42:26.456 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.456 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.456 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:26.456 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.919002+0000", 2024-04-20T17:42:26.456 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00047861699999999997 2024-04-20T17:42:26.456 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.456 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.456 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:26.456 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.919137+0000", 2024-04-20T17:42:26.456 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013506 2024-04-20T17:42:26.456 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.456 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.456 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:42:26.457 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.919222+0000", 2024-04-20T17:42:26.457 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5099000000000006e-05 2024-04-20T17:42:26.457 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.457 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:26.457 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.922856+0000", 2024-04-20T17:42:26.457 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.003633406 2024-04-20T17:42:26.457 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.457 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:26.457 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.997491+0000", 2024-04-20T17:42:26.457 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.074635624999999997 2024-04-20T17:42:26.457 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.457 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:26.458 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.997527+0000", 2024-04-20T17:42:26.458 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5457999999999998e-05 2024-04-20T17:42:26.458 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.458 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.458 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:26.458 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.997555+0000", 2024-04-20T17:42:26.458 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7883000000000001e-05 2024-04-20T17:42:26.458 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.458 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:26.458 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.458 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.458 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.458 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:156515 7.a 7:59e70d08:::benchmark_data_smithi171_161871_object10789:head [delete] snapc 0=[] ondisk+write+known_if_redirected e735)", 2024-04-20T17:42:26.458 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:40:20.948835+0000", 2024-04-20T17:42:26.459 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 125.418623877, 2024-04-20T17:42:26.459 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.048819564000000003, 2024-04-20T17:42:26.459 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:26.459 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:26.459 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:26.459 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:26.459 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:26.459 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:26.459 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 156515 2024-04-20T17:42:26.459 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.459 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:26.459 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.459 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:26.459 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.948835+0000", 2024-04-20T17:42:26.460 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.460 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.460 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.460 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:26.460 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.948835+0000", 2024-04-20T17:42:26.460 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.460 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.460 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.460 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:26.460 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.948837+0000", 2024-04-20T17:42:26.460 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0889999999999998e-06 2024-04-20T17:42:26.460 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.460 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.460 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:26.461 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.948843+0000", 2024-04-20T17:42:26.461 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9519999999999999e-06 2024-04-20T17:42:26.461 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.461 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.461 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:26.461 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.948855+0000", 2024-04-20T17:42:26.461 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.203e-05 2024-04-20T17:42:26.461 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.461 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.461 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:26.461 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.948862+0000", 2024-04-20T17:42:26.461 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1409999999999999e-06 2024-04-20T17:42:26.461 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.461 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.462 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:26.462 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.948914+0000", 2024-04-20T17:42:26.462 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1196999999999998e-05 2024-04-20T17:42:26.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.462 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:26.462 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.949004+0000", 2024-04-20T17:42:26.462 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0636000000000004e-05 2024-04-20T17:42:26.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.462 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:42:26.462 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.949076+0000", 2024-04-20T17:42:26.462 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1779000000000005e-05 2024-04-20T17:42:26.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.463 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:26.463 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.952797+0000", 2024-04-20T17:42:26.463 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0037212299999999999 2024-04-20T17:42:26.463 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.463 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.463 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:26.463 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.997586+0000", 2024-04-20T17:42:26.463 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044788659000000001 2024-04-20T17:42:26.463 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.463 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.463 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:26.463 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.997627+0000", 2024-04-20T17:42:26.463 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0757999999999998e-05 2024-04-20T17:42:26.463 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.463 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.464 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:26.464 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:40:20.997655+0000", 2024-04-20T17:42:26.464 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8093e-05 2024-04-20T17:42:26.464 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.464 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:26.464 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.464 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.464 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.464 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:259104 7.a 7:5a118956:::benchmark_data_smithi171_161871_object113378:head [delete] snapc 0=[] ondisk+write+known_if_redirected e756)", 2024-04-20T17:42:26.464 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:42:21.562589+0000", 2024-04-20T17:42:26.464 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 4.8048696379999996, 2024-04-20T17:42:26.464 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.039398014000000002, 2024-04-20T17:42:26.464 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:26.464 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:26.464 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:26.465 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:26.465 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:26.465 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:26.465 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 259104 2024-04-20T17:42:26.465 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.465 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:26.465 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.465 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:26.465 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.562589+0000", 2024-04-20T17:42:26.465 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.465 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.465 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.465 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:26.465 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.562589+0000", 2024-04-20T17:42:26.465 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.466 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.466 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.466 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:26.466 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.562591+0000", 2024-04-20T17:42:26.466 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7940000000000001e-06 2024-04-20T17:42:26.466 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.466 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.466 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:26.466 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.562596+0000", 2024-04-20T17:42:26.466 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0470000000000001e-06 2024-04-20T17:42:26.466 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.466 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.466 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:26.466 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.562607+0000", 2024-04-20T17:42:26.467 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0624e-05 2024-04-20T17:42:26.467 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.467 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.467 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:26.467 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.562613+0000", 2024-04-20T17:42:26.467 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4989999999999999e-06 2024-04-20T17:42:26.467 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.467 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.467 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:26.467 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.562686+0000", 2024-04-20T17:42:26.467 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2434999999999993e-05 2024-04-20T17:42:26.467 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.467 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.467 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:26.467 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.562826+0000", 2024-04-20T17:42:26.468 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000139982 2024-04-20T17:42:26.468 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.468 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.468 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:42:26.468 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.562956+0000", 2024-04-20T17:42:26.468 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013008400000000001 2024-04-20T17:42:26.468 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.468 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.468 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:26.468 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.573301+0000", 2024-04-20T17:42:26.468 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010345345000000001 2024-04-20T17:42:26.468 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.468 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.468 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:26.469 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.601903+0000", 2024-04-20T17:42:26.469 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028601685000000002 2024-04-20T17:42:26.469 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.469 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.469 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:26.469 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.601953+0000", 2024-04-20T17:42:26.469 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0563999999999999e-05 2024-04-20T17:42:26.469 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.469 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.469 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:26.469 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.601987+0000", 2024-04-20T17:42:26.469 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3955000000000003e-05 2024-04-20T17:42:26.469 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.469 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:26.469 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.470 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.470 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.470 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:259219 7.a 7:5fd8559a:::benchmark_data_smithi171_161871_object113493:head [delete] snapc 0=[] ondisk+write+known_if_redirected e756)", 2024-04-20T17:42:26.470 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:42:21.729525+0000", 2024-04-20T17:42:26.470 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 4.6379342719999999, 2024-04-20T17:42:26.470 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.045371458000000003, 2024-04-20T17:42:26.470 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:26.470 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:26.470 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:26.470 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:26.470 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:26.470 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:26.470 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 259219 2024-04-20T17:42:26.471 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.471 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:26.471 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.471 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:26.471 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.729525+0000", 2024-04-20T17:42:26.471 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.471 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.471 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.471 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:26.471 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.729525+0000", 2024-04-20T17:42:26.471 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.471 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.471 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.471 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:26.471 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.729526+0000", 2024-04-20T17:42:26.472 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.362e-06 2024-04-20T17:42:26.472 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.472 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.472 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:26.472 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.729531+0000", 2024-04-20T17:42:26.472 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4889999999999997e-06 2024-04-20T17:42:26.472 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.472 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.472 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:26.472 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.729543+0000", 2024-04-20T17:42:26.472 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2848e-05 2024-04-20T17:42:26.472 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.472 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.472 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:26.473 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.729551+0000", 2024-04-20T17:42:26.473 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.072e-06 2024-04-20T17:42:26.473 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.473 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.473 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:26.473 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.739747+0000", 2024-04-20T17:42:26.473 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010195890000000001 2024-04-20T17:42:26.473 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.473 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.473 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:26.473 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.739859+0000", 2024-04-20T17:42:26.473 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000111933 2024-04-20T17:42:26.473 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.473 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.474 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:42:26.474 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.739932+0000", 2024-04-20T17:42:26.474 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3198e-05 2024-04-20T17:42:26.474 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.474 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.474 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:26.474 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.744278+0000", 2024-04-20T17:42:26.474 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0043456700000000003 2024-04-20T17:42:26.474 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.474 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.474 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:26.474 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.774797+0000", 2024-04-20T17:42:26.474 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030518668999999998 2024-04-20T17:42:26.474 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.475 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.475 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:26.475 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.774844+0000", 2024-04-20T17:42:26.475 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6823000000000002e-05 2024-04-20T17:42:26.475 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.475 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.475 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:26.475 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.774896+0000", 2024-04-20T17:42:26.475 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2503999999999997e-05 2024-04-20T17:42:26.475 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.475 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:26.475 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.475 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.475 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.475 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:259412 7.a 7:55dba351:::benchmark_data_smithi171_161871_object113686:head [delete] snapc 0=[] ondisk+write+known_if_redirected e756)", 2024-04-20T17:42:26.476 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:42:21.968328+0000", 2024-04-20T17:42:26.476 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 4.3991309369999998, 2024-04-20T17:42:26.476 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.038671874000000002, 2024-04-20T17:42:26.476 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:26.476 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:26.476 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:26.476 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:26.476 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:26.476 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:26.476 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 259412 2024-04-20T17:42:26.476 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.476 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:26.476 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.476 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:26.477 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.968328+0000", 2024-04-20T17:42:26.477 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.477 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.477 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.477 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:26.477 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.968328+0000", 2024-04-20T17:42:26.477 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.477 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.477 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.477 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:26.477 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.968331+0000", 2024-04-20T17:42:26.477 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.157e-06 2024-04-20T17:42:26.477 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.477 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.477 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:26.477 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.968339+0000", 2024-04-20T17:42:26.478 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1929999999999993e-06 2024-04-20T17:42:26.478 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.478 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.478 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:26.478 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.968352+0000", 2024-04-20T17:42:26.478 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2731000000000001e-05 2024-04-20T17:42:26.478 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.478 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.478 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:26.478 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.968364+0000", 2024-04-20T17:42:26.478 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1528e-05 2024-04-20T17:42:26.478 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.478 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.478 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:26.479 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.970630+0000", 2024-04-20T17:42:26.479 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0022667719999999998 2024-04-20T17:42:26.479 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.479 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.479 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:26.479 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.970768+0000", 2024-04-20T17:42:26.479 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013746900000000001 2024-04-20T17:42:26.479 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.479 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.479 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:42:26.479 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.970908+0000", 2024-04-20T17:42:26.479 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014055200000000001 2024-04-20T17:42:26.479 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.479 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.480 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:26.480 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:21.976537+0000", 2024-04-20T17:42:26.480 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0056285520000000002 2024-04-20T17:42:26.480 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.480 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.480 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:26.480 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:22.006880+0000", 2024-04-20T17:42:26.480 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030343278000000001 2024-04-20T17:42:26.480 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.480 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.480 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:26.480 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:22.006941+0000", 2024-04-20T17:42:26.480 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1138999999999999e-05 2024-04-20T17:42:26.480 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.480 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.481 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:26.481 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:22.007000+0000", 2024-04-20T17:42:26.481 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8502999999999999e-05 2024-04-20T17:42:26.481 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.481 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:26.481 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.481 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.481 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.481 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:259506 7.a 7:55928385:::benchmark_data_smithi171_161871_object113780:head [delete] snapc 0=[] ondisk+write+known_if_redirected e756)", 2024-04-20T17:42:26.481 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:42:22.091775+0000", 2024-04-20T17:42:26.481 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 4.2756839979999999, 2024-04-20T17:42:26.481 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.037996803000000003, 2024-04-20T17:42:26.481 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:26.481 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:26.482 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:26.482 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:26.482 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:26.482 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:26.482 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 259506 2024-04-20T17:42:26.482 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.482 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:26.482 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.482 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:26.482 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:22.091775+0000", 2024-04-20T17:42:26.482 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.482 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.482 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.482 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:26.482 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:22.091775+0000", 2024-04-20T17:42:26.483 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:26.483 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.483 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.483 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:26.483 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:22.091778+0000", 2024-04-20T17:42:26.483 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2909999999999999e-06 2024-04-20T17:42:26.483 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.483 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.483 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:26.483 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:22.091789+0000", 2024-04-20T17:42:26.483 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0631e-05 2024-04-20T17:42:26.483 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.483 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.483 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:26.483 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:22.091807+0000", 2024-04-20T17:42:26.484 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8481000000000001e-05 2024-04-20T17:42:26.484 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.484 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.484 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:26.484 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:22.091819+0000", 2024-04-20T17:42:26.484 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1401e-05 2024-04-20T17:42:26.484 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.484 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.484 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:26.484 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:22.091877+0000", 2024-04-20T17:42:26.484 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7852000000000001e-05 2024-04-20T17:42:26.484 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.484 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.484 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:26.484 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:22.091967+0000", 2024-04-20T17:42:26.485 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0383000000000005e-05 2024-04-20T17:42:26.485 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.485 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.485 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:42:26.485 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:22.092053+0000", 2024-04-20T17:42:26.485 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6466999999999997e-05 2024-04-20T17:42:26.485 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.485 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.485 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:26.485 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:22.098696+0000", 2024-04-20T17:42:26.485 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0066425620000000003 2024-04-20T17:42:26.485 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.485 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.485 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:26.485 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:22.129657+0000", 2024-04-20T17:42:26.486 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030961061000000002 2024-04-20T17:42:26.486 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.486 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.486 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:26.486 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:22.129711+0000", 2024-04-20T17:42:26.486 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3900000000000002e-05 2024-04-20T17:42:26.486 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:26.486 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:26.486 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:26.486 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:22.129772+0000", 2024-04-20T17:42:26.486 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0773999999999998e-05 2024-04-20T17:42:26.486 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.486 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:26.486 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.486 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:26.487 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:26.487 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:42:26.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:26.488 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:26.488 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:42:26.488 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:42:26.589 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:42:26.625 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:26.905 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:42:26.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:26 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 1983/63574 objects degraded (3.119%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:27.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:26 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 1983/63574 objects degraded (3.119%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:27.028 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:26 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 1983/63574 objects degraded (3.119%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:27.069 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-20T17:42:27.084 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:42:27.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:27.089 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:42:27.190 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:42:27.223 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:27.248 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:42:27.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:27.251 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:42:27.352 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:42:27.396 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:27.421 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:42:27.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:27.424 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:42:27.466 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:42:27.526 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:42:27.597 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:27.657 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:42:27.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:27.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:42:27.714 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:27 smithi171 ceph-mon[44189]: 7.1 deep-scrub ok 2024-04-20T17:42:27.714 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:27 smithi171 ceph-mon[44189]: pgmap v2038: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing+deep, 15 active+clean; 1.8 GiB data, 8.0 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s; 1703/58076 objects degraded (2.932%); 4.4 MiB/s, 70 objects/s recovering 2024-04-20T17:42:27.765 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:42:27.798 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:42:27.832 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:27.865 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:42:27.876 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:27 smithi005 ceph-mon[39562]: 7.1 deep-scrub ok 2024-04-20T17:42:27.876 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:27 smithi005 ceph-mon[39562]: pgmap v2038: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing+deep, 15 active+clean; 1.8 GiB data, 8.0 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s; 1703/58076 objects degraded (2.932%); 4.4 MiB/s, 70 objects/s recovering 2024-04-20T17:42:27.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:27.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:42:27.978 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:42:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:27 smithi106 ceph-mon[44423]: 7.1 deep-scrub ok 2024-04-20T17:42:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:27 smithi106 ceph-mon[44423]: pgmap v2038: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing+deep, 15 active+clean; 1.8 GiB data, 8.0 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 1.07k op/s; 1703/58076 objects degraded (2.932%); 4.4 MiB/s, 70 objects/s recovering 2024-04-20T17:42:28.014 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:28.812 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:42:28.845 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:42:28.876 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:42:28.915 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:42:28.953 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:42:28.970 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:42:28.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:28.975 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:42:28.994 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:42:29.036 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:42:29.076 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:42:29.098 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:42:29.115 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:29.132 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:42:29.136 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:42:29.142 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:42:29.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:29.152 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:42:29.155 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:42:29.205 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:42:29.252 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:42:29.253 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:42:29.254 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:42:29.254 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:42:29.254 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:42:29.254 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:42:29.254 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:42:29.254 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:42:29.254 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:42:29.254 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:42:29.255 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:42:29.255 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:42:29.255 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:42:29.255 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:42:29.320 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:29.786 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:29 smithi171 ceph-mon[44189]: pgmap v2039: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing+deep, 15 active+clean; 1.8 GiB data, 8.0 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 881 op/s; 1703/58076 objects degraded (2.932%); 4.4 MiB/s, 70 objects/s recovering 2024-04-20T17:42:29.786 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:29 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/301521233' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:42:29.786 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:29.786 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:29.786 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:29.787 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:29.787 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:29.787 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:29.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:29 smithi005 ceph-mon[39562]: pgmap v2039: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing+deep, 15 active+clean; 1.8 GiB data, 8.0 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 881 op/s; 1703/58076 objects degraded (2.932%); 4.4 MiB/s, 70 objects/s recovering 2024-04-20T17:42:29.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:29 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/301521233' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:42:29.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:29.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:29.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:29.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:29.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:29.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:30.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:29 smithi106 ceph-mon[44423]: pgmap v2039: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing+deep, 15 active+clean; 1.8 GiB data, 8.0 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 881 op/s; 1703/58076 objects degraded (2.932%); 4.4 MiB/s, 70 objects/s recovering 2024-04-20T17:42:30.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:29 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/301521233' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:42:30.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:30.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:30.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:30.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:30.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:30.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:30.096 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:42:30.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:30.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:42:30.122 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:42:30.122 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:42:30.206 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:42:30.280 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:30.831 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:30.831 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:30.831 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:30.831 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:30 smithi171 ceph-mon[44189]: from='client.27467 v1:172.21.15.5:0/3514175165' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:42:30.832 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:30.832 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:30.832 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:30 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:42:30.832 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:30 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:42:30.867 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-20T17:42:30.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:30.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:30.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:30.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:30 smithi005 ceph-mon[39562]: from='client.27467 v1:172.21.15.5:0/3514175165' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:42:30.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:30.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:30.967 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:30 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:42:30.967 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:30 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:42:31.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:31.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:31.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:31.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:30 smithi106 ceph-mon[44423]: from='client.27467 v1:172.21.15.5:0/3514175165' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:42:31.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:31.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:31.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:30 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:42:31.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:30 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:42:31.256 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:42:31.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:31 smithi005 ceph-mon[39562]: pgmap v2040: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing+deep, 15 active+clean; 1.7 GiB data, 7.8 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 768 op/s; 1703/54534 objects degraded (3.123%); 2.2 MiB/s, 34 objects/s recovering 2024-04-20T17:42:31.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:31 smithi005 ceph-mon[39562]: osdmap e759: 12 total, 11 up, 9 in 2024-04-20T17:42:31.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:31 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 1703/54534 objects degraded (3.123%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:31.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:31 smithi171 ceph-mon[44189]: pgmap v2040: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing+deep, 15 active+clean; 1.7 GiB data, 7.8 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 768 op/s; 1703/54534 objects degraded (3.123%); 2.2 MiB/s, 34 objects/s recovering 2024-04-20T17:42:31.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:31 smithi171 ceph-mon[44189]: osdmap e759: 12 total, 11 up, 9 in 2024-04-20T17:42:31.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:31 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 1703/54534 objects degraded (3.123%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:31 smithi106 ceph-mon[44423]: pgmap v2040: 17 pgs: 1 active+recovering+degraded, 1 active+clean+scrubbing+deep, 15 active+clean; 1.7 GiB data, 7.8 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 768 op/s; 1703/54534 objects degraded (3.123%); 2.2 MiB/s, 34 objects/s recovering 2024-04-20T17:42:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:31 smithi106 ceph-mon[44423]: osdmap e759: 12 total, 11 up, 9 in 2024-04-20T17:42:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:31 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 1703/54534 objects degraded (3.123%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:32.447 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:42:32.448 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:42:32.693 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:32 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:42:32.693 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:32 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:42:32.693 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:32 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:42:32.693 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:32 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:42:32.693 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:32 smithi005 ceph-mon[39562]: osdmap e760: 12 total, 11 up, 9 in 2024-04-20T17:42:32.883 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:42:32.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:32.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:42:32.920 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":2039,"stamp":"2024-04-20T17:42:28.556672+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1903428152,"num_objects":29038,"num_object_clones":0,"num_object_copies":58076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1703,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29038,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":262593,"num_write_kb":9329211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16364,"num_bytes_recovered":1074420384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160947,"ondisk_log_size":160947,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":8369936,"kb_used_data":3664232,"kb_used_omap":217,"kb_used_meta":4705190,"kb_avail":835336432,"statfs":{"total":863955320832,"available":855384506368,"internally_reserved":0,"allocated":3752173568,"data_stored":3748799465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":222986,"internal_metadata":4818114806},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,5,11,3,1,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-462028800,"num_objects":-7050,"num_object_clones":0,"num_object_copies":-14100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7050,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":561,"num_bytes_recovered":36765696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001701"},"pg_stats":[{"pgid":"7.f","version":"758'16642","reported_seq":16877,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-04-20T17:42:25.980898+0000","last_change":"2024-04-20T17:39:40.325470+0000","last_active":"2024-04-20T17:42:25.980898+0000","last_peered":"2024-04-20T17:42:25.980898+0000","last_clean":"2024-04-20T17:42:25.980898+0000","last_became_active":"2024-04-20T17:37:56.835750+0000","last_became_peered":"2024-04-20T17:37:56.835750+0000","last_unstale":"2024-04-20T17:42:25.980898+0000","last_undegraded":"2024-04-20T17:42:25.980898+0000","last_fullsized":"2024-04-20T17:42:25.980898+0000","mapping_epoch":694,"log_start":"712'6600","ondisk_log_start":"712'6600","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9175","last_scrub_stamp":"2024-04-20T17:39:40.325340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:40.325340+0000","objects_scrubbed":9175,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":741,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111935488,"num_objects":1708,"num_object_clones":0,"num_object_copies":3416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16642,"num_write_kb":587200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"758'16218","reported_seq":16506,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-04-20T17:42:26.143625+0000","last_change":"2024-04-20T17:41:02.433089+0000","last_active":"2024-04-20T17:42:26.143625+0000","last_peered":"2024-04-20T17:42:26.143625+0000","last_clean":"2024-04-20T17:42:26.143625+0000","last_became_active":"2024-04-20T17:37:56.833703+0000","last_became_peered":"2024-04-20T17:37:56.833703+0000","last_unstale":"2024-04-20T17:42:26.143625+0000","last_undegraded":"2024-04-20T17:42:26.143625+0000","last_fullsized":"2024-04-20T17:42:26.143625+0000","mapping_epoch":694,"log_start":"712'6200","ondisk_log_start":"712'6200","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"742'11794","last_scrub_stamp":"2024-04-20T17:41:02.432782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:41:02.432782+0000","objects_scrubbed":6149,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:29.683106+0000","scrub_duration":1488,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111542272,"num_objects":1702,"num_object_clones":0,"num_object_copies":3404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16218,"num_write_kb":573440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"758'18748","reported_seq":19090,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-04-20T17:42:25.960236+0000","last_change":"2024-04-20T17:39:48.311967+0000","last_active":"2024-04-20T17:42:25.960236+0000","last_peered":"2024-04-20T17:42:25.960236+0000","last_clean":"2024-04-20T17:42:25.960236+0000","last_became_active":"2024-04-20T17:37:56.836496+0000","last_became_peered":"2024-04-20T17:37:56.836496+0000","last_unstale":"2024-04-20T17:42:25.960236+0000","last_undegraded":"2024-04-20T17:42:25.960236+0000","last_fullsized":"2024-04-20T17:42:25.960236+0000","mapping_epoch":694,"log_start":"718'8700","ondisk_log_start":"718'8700","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'11204","last_scrub_stamp":"2024-04-20T17:39:48.311825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:48.311825+0000","objects_scrubbed":9436,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124059672,"num_objects":1894,"num_object_clones":0,"num_object_copies":3788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1894,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16980,"num_write_kb":603905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.c","version":"758'16276","reported_seq":16513,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-04-20T17:42:24.247085+0000","last_change":"2024-04-20T17:39:44.683378+0000","last_active":"2024-04-20T17:42:24.247085+0000","last_peered":"2024-04-20T17:42:24.247085+0000","last_clean":"2024-04-20T17:42:24.247085+0000","last_became_active":"2024-04-20T17:37:56.834181+0000","last_became_peered":"2024-04-20T17:37:56.834181+0000","last_unstale":"2024-04-20T17:42:24.247085+0000","last_undegraded":"2024-04-20T17:42:24.247085+0000","last_fullsized":"2024-04-20T17:42:24.247085+0000","mapping_epoch":694,"log_start":"712'6200","ondisk_log_start":"712'6200","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9063","last_scrub_stamp":"2024-04-20T17:39:44.683201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:44.683201+0000","objects_scrubbed":9063,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":746,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121241600,"num_objects":1850,"num_object_clones":0,"num_object_copies":3700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16276,"num_write_kb":580032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"757'16379","reported_seq":16690,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-04-20T17:42:24.068102+0000","last_change":"2024-04-20T17:40:58.937747+0000","last_active":"2024-04-20T17:42:24.068102+0000","last_peered":"2024-04-20T17:42:24.068102+0000","last_clean":"2024-04-20T17:42:24.068102+0000","last_became_active":"2024-04-20T17:37:56.830487+0000","last_became_peered":"2024-04-20T17:37:56.830487+0000","last_unstale":"2024-04-20T17:42:24.068102+0000","last_undegraded":"2024-04-20T17:42:24.068102+0000","last_fullsized":"2024-04-20T17:42:24.068102+0000","mapping_epoch":694,"log_start":"712'6300","ondisk_log_start":"712'6300","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"742'11757","last_scrub_stamp":"2024-04-20T17:40:58.937604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:40:58.937604+0000","objects_scrubbed":6664,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:28.789538+0000","scrub_duration":858,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132972544,"num_objects":2029,"num_object_clones":0,"num_object_copies":4058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16379,"num_write_kb":589056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.a","version":"758'16290","reported_seq":16621,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-04-20T17:42:25.910521+0000","last_change":"2024-04-20T17:42:22.174131+0000","last_active":"2024-04-20T17:42:25.910521+0000","last_peered":"2024-04-20T17:42:25.910521+0000","last_clean":"2024-04-20T17:42:25.910521+0000","last_became_active":"2024-04-20T17:37:56.835944+0000","last_became_peered":"2024-04-20T17:37:56.835944+0000","last_unstale":"2024-04-20T17:42:25.910521+0000","last_undegraded":"2024-04-20T17:42:25.910521+0000","last_fullsized":"2024-04-20T17:42:25.910521+0000","mapping_epoch":694,"log_start":"712'6200","ondisk_log_start":"712'6200","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"756'16067","last_scrub_stamp":"2024-04-20T17:42:22.173861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:42:22.173861+0000","objects_scrubbed":1940,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:43:33.214516+0000","scrub_duration":489,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111017984,"num_objects":1694,"num_object_clones":0,"num_object_copies":3388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16290,"num_write_kb":575488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"758'16421","reported_seq":16641,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-04-20T17:42:25.970867+0000","last_change":"2024-04-20T17:39:03.675435+0000","last_active":"2024-04-20T17:42:25.970867+0000","last_peered":"2024-04-20T17:42:25.970867+0000","last_clean":"2024-04-20T17:42:25.970867+0000","last_became_active":"2024-04-20T17:37:56.835700+0000","last_became_peered":"2024-04-20T17:37:56.835700+0000","last_unstale":"2024-04-20T17:42:25.970867+0000","last_undegraded":"2024-04-20T17:42:25.970867+0000","last_fullsized":"2024-04-20T17:42:25.970867+0000","mapping_epoch":694,"log_start":"712'6400","ondisk_log_start":"712'6400","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'7175","last_scrub_stamp":"2024-04-20T17:39:03.675211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:03.675211+0000","objects_scrubbed":7133,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112001024,"num_objects":1709,"num_object_clones":0,"num_object_copies":3418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16421,"num_write_kb":580160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11],"acting":[0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"758'19913","reported_seq":38551,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-04-20T17:42:26.189309+0000","last_change":"2024-04-20T17:42:02.804301+0000","last_active":"2024-04-20T17:42:26.189309+0000","last_peered":"2024-04-20T17:42:26.189309+0000","last_clean":"2024-04-20T17:42:26.189309+0000","last_became_active":"2024-04-20T17:40:09.546695+0000","last_became_peered":"2024-04-20T17:40:09.546695+0000","last_unstale":"2024-04-20T17:42:26.189309+0000","last_undegraded":"2024-04-20T17:42:26.189309+0000","last_fullsized":"2024-04-20T17:42:26.189309+0000","mapping_epoch":731,"log_start":"718'9901","ondisk_log_start":"718'9901","created":694,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'18634","last_scrub_stamp":"2024-04-20T17:42:02.804073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:42:02.804073+0000","objects_scrubbed":3082,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:43:21.646311+0000","scrub_duration":620,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117506048,"num_objects":1793,"num_object_clones":0,"num_object_copies":3586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16343,"num_write_kb":580352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9062,"num_bytes_recovered":593887232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"757'16443","reported_seq":16778,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-04-20T17:42:23.693033+0000","last_change":"2024-04-20T17:42:04.144551+0000","last_active":"2024-04-20T17:42:23.693033+0000","last_peered":"2024-04-20T17:42:23.693033+0000","last_clean":"2024-04-20T17:42:23.693033+0000","last_became_active":"2024-04-20T17:37:56.836665+0000","last_became_peered":"2024-04-20T17:37:56.836665+0000","last_unstale":"2024-04-20T17:42:23.693033+0000","last_undegraded":"2024-04-20T17:42:23.693033+0000","last_fullsized":"2024-04-20T17:42:23.693033+0000","mapping_epoch":694,"log_start":"712'6400","ondisk_log_start":"712'6400","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'15376","last_scrub_stamp":"2024-04-20T17:42:04.144333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:42:04.144333+0000","objects_scrubbed":2961,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:43:15.847365+0000","scrub_duration":538,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123535360,"num_objects":1885,"num_object_clones":0,"num_object_copies":3770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16443,"num_write_kb":586496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.5","version":"757'16403","reported_seq":16735,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-04-20T17:42:24.071636+0000","last_change":"2024-04-20T17:42:03.424949+0000","last_active":"2024-04-20T17:42:24.071636+0000","last_peered":"2024-04-20T17:42:24.071636+0000","last_clean":"2024-04-20T17:42:24.071636+0000","last_became_active":"2024-04-20T17:37:56.833600+0000","last_became_peered":"2024-04-20T17:37:56.833600+0000","last_unstale":"2024-04-20T17:42:24.071636+0000","last_undegraded":"2024-04-20T17:42:24.071636+0000","last_fullsized":"2024-04-20T17:42:24.071636+0000","mapping_epoch":694,"log_start":"712'6400","ondisk_log_start":"712'6400","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'15267","last_scrub_stamp":"2024-04-20T17:42:03.424811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:42:03.424811+0000","objects_scrubbed":3060,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:43:32.116339+0000","scrub_duration":489,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125239296,"num_objects":1911,"num_object_clones":0,"num_object_copies":3822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16403,"num_write_kb":586048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.4","version":"758'16229","reported_seq":16466,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-04-20T17:42:25.927757+0000","last_change":"2024-04-20T17:39:50.365478+0000","last_active":"2024-04-20T17:42:25.927757+0000","last_peered":"2024-04-20T17:42:25.927757+0000","last_clean":"2024-04-20T17:42:25.927757+0000","last_became_active":"2024-04-20T17:37:56.834727+0000","last_became_peered":"2024-04-20T17:37:56.834727+0000","last_unstale":"2024-04-20T17:42:25.927757+0000","last_undegraded":"2024-04-20T17:42:25.927757+0000","last_fullsized":"2024-04-20T17:42:25.927757+0000","mapping_epoch":694,"log_start":"712'6200","ondisk_log_start":"712'6200","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9031","last_scrub_stamp":"2024-04-20T17:39:50.365339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:50.365339+0000","objects_scrubbed":9031,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":876,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120127488,"num_objects":1833,"num_object_clones":0,"num_object_copies":3666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16229,"num_write_kb":577984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"758'16189","reported_seq":16404,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-04-20T17:42:24.332057+0000","last_change":"2024-04-20T17:39:00.502710+0000","last_active":"2024-04-20T17:42:24.332057+0000","last_peered":"2024-04-20T17:42:24.332057+0000","last_clean":"2024-04-20T17:42:24.332057+0000","last_became_active":"2024-04-20T17:37:56.832911+0000","last_became_peered":"2024-04-20T17:37:56.832911+0000","last_unstale":"2024-04-20T17:42:24.332057+0000","last_undegraded":"2024-04-20T17:42:24.332057+0000","last_fullsized":"2024-04-20T17:42:24.332057+0000","mapping_epoch":694,"log_start":"712'6100","ondisk_log_start":"712'6100","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'6831","last_scrub_stamp":"2024-04-20T17:39:00.502556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:00.502556+0000","objects_scrubbed":6782,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123404288,"num_objects":1883,"num_object_clones":0,"num_object_copies":3766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16189,"num_write_kb":578304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"758'16593","reported_seq":16815,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-04-20T17:42:24.353209+0000","last_change":"2024-04-20T17:39:08.489529+0000","last_active":"2024-04-20T17:42:24.353209+0000","last_peered":"2024-04-20T17:42:24.353209+0000","last_clean":"2024-04-20T17:42:24.353209+0000","last_became_active":"2024-04-20T17:37:56.834292+0000","last_became_peered":"2024-04-20T17:37:56.834292+0000","last_unstale":"2024-04-20T17:42:24.353209+0000","last_undegraded":"2024-04-20T17:42:24.353209+0000","last_fullsized":"2024-04-20T17:42:24.353209+0000","mapping_epoch":694,"log_start":"712'6500","ondisk_log_start":"712'6500","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"716'7577","last_scrub_stamp":"2024-04-20T17:39:08.489308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:08.489308+0000","objects_scrubbed":7524,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124452864,"num_objects":1899,"num_object_clones":0,"num_object_copies":3798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16593,"num_write_kb":591744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"758'18456","reported_seq":18846,"reported_epoch":758,"state":"active+clean+scrubbing+deep","last_fresh":"2024-04-20T17:42:25.774319+0000","last_change":"2024-04-20T17:42:14.007395+0000","last_active":"2024-04-20T17:42:25.774319+0000","last_peered":"2024-04-20T17:42:25.774319+0000","last_clean":"2024-04-20T17:42:25.774319+0000","last_became_active":"2024-04-20T17:37:56.835512+0000","last_became_peered":"2024-04-20T17:37:56.835512+0000","last_unstale":"2024-04-20T17:42:25.774319+0000","last_undegraded":"2024-04-20T17:42:25.774319+0000","last_fullsized":"2024-04-20T17:42:25.774319+0000","mapping_epoch":694,"log_start":"720'8400","ondisk_log_start":"720'8400","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"743'12166","last_scrub_stamp":"2024-04-20T17:41:07.794973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:41:07.794973+0000","objects_scrubbed":2061,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"deep scrubbing for 11s","scrub_duration":644,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117112832,"num_objects":1787,"num_object_clones":0,"num_object_copies":3574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16395,"num_write_kb":581824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"758'16324","reported_seq":16622,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-04-20T17:42:25.963919+0000","last_change":"2024-04-20T17:41:16.382261+0000","last_active":"2024-04-20T17:42:25.963919+0000","last_peered":"2024-04-20T17:42:25.963919+0000","last_clean":"2024-04-20T17:42:25.963919+0000","last_became_active":"2024-04-20T17:37:56.833972+0000","last_became_peered":"2024-04-20T17:37:56.833972+0000","last_unstale":"2024-04-20T17:42:25.963919+0000","last_undegraded":"2024-04-20T17:42:25.963919+0000","last_fullsized":"2024-04-20T17:42:25.963919+0000","mapping_epoch":694,"log_start":"712'6300","ondisk_log_start":"712'6300","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"745'12483","last_scrub_stamp":"2024-04-20T17:41:16.382018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:41:16.382018+0000","objects_scrubbed":5596,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:31.605235+0000","scrub_duration":653,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114032640,"num_objects":1740,"num_object_clones":0,"num_object_copies":3480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16324,"num_write_kb":578048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.6","version":"758'16287","reported_seq":31094,"reported_epoch":758,"state":"active+recovering+degraded","last_fresh":"2024-04-20T17:42:25.985015+0000","last_change":"2024-04-20T17:39:54.962650+0000","last_active":"2024-04-20T17:42:25.985015+0000","last_peered":"2024-04-20T17:42:25.985015+0000","last_clean":"2024-04-20T17:39:34.507147+0000","last_became_active":"2024-04-20T17:39:54.933393+0000","last_became_peered":"2024-04-20T17:39:54.933393+0000","last_unstale":"2024-04-20T17:42:25.985015+0000","last_undegraded":"2024-04-20T17:39:54.578981+0000","last_fullsized":"2024-04-20T17:42:25.985015+0000","mapping_epoch":727,"log_start":"712'6200","ondisk_log_start":"712'6200","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'7141","last_scrub_stamp":"2024-04-20T17:39:04.435913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:04.435913+0000","objects_scrubbed":7101,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112656384,"num_objects":1719,"num_object_clones":0,"num_object_copies":3438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1703,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16287,"num_write_kb":576192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7292,"num_bytes_recovered":477888512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":1703},{"shards":"0,3","objects":16}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":534,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-04-20T17:42:23.117245+0000","last_change":"2024-04-20T17:42:20.165296+0000","last_active":"2024-04-20T17:42:23.117245+0000","last_peered":"2024-04-20T17:42:23.117245+0000","last_clean":"2024-04-20T17:42:23.117245+0000","last_became_active":"2024-04-20T17:39:54.970208+0000","last_became_peered":"2024-04-20T17:39:54.970208+0000","last_unstale":"2024-04-20T17:42:23.117245+0000","last_undegraded":"2024-04-20T17:42:23.117245+0000","last_fullsized":"2024-04-20T17:42:23.117245+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:42:20.165149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:42:20.165149+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:43:22.534703+0000","scrub_duration":180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":1902837784,"num_objects":29036,"num_object_clones":0,"num_object_copies":58072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1703,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29036,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":262412,"num_write_kb":9326273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16354,"num_bytes_recovered":1071775744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4869136384,"data_stored":4869128240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160810,"ondisk_log_size":160810,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991673,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1194204,"kb_used_data":491088,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":92550948,"statfs":{"total":95995035648,"available":94772170752,"internally_reserved":0,"allocated":502874112,"data_stored":502501341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":719976900},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253312,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":897828,"kb_used_data":357140,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":92847324,"statfs":{"total":95995035648,"available":95075659776,"internally_reserved":0,"allocated":365711360,"data_stored":365336137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":553646528},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613061,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":894824,"kb_used_data":340252,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92850328,"statfs":{"total":95995035648,"available":95078735872,"internally_reserved":0,"allocated":348418048,"data_stored":348039293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":567867840},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809291,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":563028,"kb_used_data":243080,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":93182124,"statfs":{"total":95995035648,"available":95418494976,"internally_reserved":0,"allocated":248913920,"data_stored":248544721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[0,1,2,3,4,5,6,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":2297807503522,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":2830383448144,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738866,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1240352,"kb_used_data":573208,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":92504800,"statfs":{"total":95995035648,"available":94724915200,"internally_reserved":0,"allocated":586964992,"data_stored":586585717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394156,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":782880,"kb_used_data":445716,"kb_used_omap":202,"kb_used_meta":336949,"kb_avail":92962272,"statfs":{"total":95995035648,"available":95193366528,"internally_reserved":0,"allocated":456413184,"data_stored":456037981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":345036247},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653205,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1027352,"kb_used_data":615948,"kb_used_omap":4,"kb_used_meta":411387,"kb_avail":92717800,"statfs":{"total":95995035648,"available":94943027200,"internally_reserved":0,"allocated":630730752,"data_stored":630357481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":421260999},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986654,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":879004,"kb_used_data":356372,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":92866148,"statfs":{"total":95995035648,"available":95094935552,"internally_reserved":0,"allocated":364924928,"data_stored":364549725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":535165381},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,0,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790424,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":890464,"kb_used_data":241428,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92854688,"statfs":{"total":95995035648,"available":95083200512,"internally_reserved":0,"allocated":247222272,"data_stored":246847069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":580718592,"data_stored":580714520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":455606272,"data_stored":455606272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":621416448,"data_stored":621412376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":357040128,"data_stored":357040128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":239534080,"data_stored":239534080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1167458304,"data_stored":1167458304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":244776960,"data_stored":244776960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":343539712,"data_stored":343539712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":360775680,"data_stored":360775680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":498270208,"data_stored":498270208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:42:32.989 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:42:33.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:32 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:42:33.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:32 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:42:33.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:32 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:42:33.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:32 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:42:33.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:32 smithi106 ceph-mon[44423]: osdmap e760: 12 total, 11 up, 9 in 2024-04-20T17:42:33.044 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:33.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:32 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:42:33.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:32 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:42:33.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:32 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:42:33.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:32 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:42:33.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:32 smithi171 ceph-mon[44189]: osdmap e760: 12 total, 11 up, 9 in 2024-04-20T17:42:33.137 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:42:33.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:33.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:42:33.246 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:42:33.272 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:33.292 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:42:33.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:33.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:42:33.396 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:42:33.435 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:33.463 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:42:33.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:33.467 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:42:33.567 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:42:33.629 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:33.664 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:42:33.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:33.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:42:33.769 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:42:33.818 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:33.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:33 smithi171 ceph-mon[44189]: pgmap v2043: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 1.5 GiB data, 7.5 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 925 op/s; 1444/48760 objects degraded (2.961%); 4.2 MiB/s, 67 objects/s recovering 2024-04-20T17:42:33.847 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:42:33.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:33.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:42:33.895 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:42:33.895 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:42:33.952 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:42:33.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:33 smithi005 ceph-mon[39562]: pgmap v2043: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 1.5 GiB data, 7.5 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 925 op/s; 1444/48760 objects degraded (2.961%); 4.2 MiB/s, 67 objects/s recovering 2024-04-20T17:42:33.977 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:33 smithi106 ceph-mon[44423]: pgmap v2043: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 1.5 GiB data, 7.5 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 925 op/s; 1444/48760 objects degraded (2.961%); 4.2 MiB/s, 67 objects/s recovering 2024-04-20T17:42:33.980 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:33.998 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:42:34.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:34.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:42:34.103 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:42:34.129 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:34.151 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:42:34.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:34.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:42:34.255 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:42:34.291 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:34.449 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:42:34.730 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:42:34.969 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:34 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4168726967' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:42:34.969 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:34 smithi005 ceph-mon[39562]: from='client.27355 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:42:34.995 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:34 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4168726967' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:42:34.995 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:34 smithi171 ceph-mon[44189]: from='client.27355 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:42:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:34 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4168726967' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:42:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:34 smithi106 ceph-mon[44423]: from='client.27355 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:42:35.034 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-20T17:42:35.049 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:42:35.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:35.054 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:42:35.155 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:42:35.195 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:35.229 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:42:35.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:35.232 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:42:35.333 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:42:35.373 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:35.406 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:42:35.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:35.409 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:42:35.431 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:42:35.510 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:42:35.571 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:35.614 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:42:35.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:35.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:42:35.693 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:42:35.703 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:35 smithi171 ceph-mon[44189]: pgmap v2044: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 1.4 GiB data, 7.3 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 822 op/s; 1444/44908 objects degraded (3.215%); 2.0 MiB/s, 32 objects/s recovering 2024-04-20T17:42:35.703 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:35 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:42:35.703 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:35 smithi171 ceph-mon[44189]: from='client.27355 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:42:35.703 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:35 smithi171 ceph-mon[44189]: osdmap e761: 12 total, 11 up, 9 in 2024-04-20T17:42:35.729 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:42:35.795 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:35 smithi005 ceph-mon[39562]: pgmap v2044: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 1.4 GiB data, 7.3 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 822 op/s; 1444/44908 objects degraded (3.215%); 2.0 MiB/s, 32 objects/s recovering 2024-04-20T17:42:35.795 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:35 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:42:35.795 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:35 smithi005 ceph-mon[39562]: from='client.27355 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:42:35.795 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:35 smithi005 ceph-mon[39562]: osdmap e761: 12 total, 11 up, 9 in 2024-04-20T17:42:35.799 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:35.922 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:42:35.969 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:42:35.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:35.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:42:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:35 smithi106 ceph-mon[44423]: pgmap v2044: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 1.4 GiB data, 7.3 GiB used, 797 GiB / 805 GiB avail; 0 B/s wr, 822 op/s; 1444/44908 objects degraded (3.215%); 2.0 MiB/s, 32 objects/s recovering 2024-04-20T17:42:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:35 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:42:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:35 smithi106 ceph-mon[44423]: from='client.27355 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:42:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:35 smithi106 ceph-mon[44423]: osdmap e761: 12 total, 11 up, 9 in 2024-04-20T17:42:36.076 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:42:36.113 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:36.314 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:42:36.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:36.320 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:42:36.421 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:42:36.450 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:36.812 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:42:36.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:36.817 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:42:36.919 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:42:36.952 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:36 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:42:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:36 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:42:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:36 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 1444/44908 objects degraded (3.215%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:37.061 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:42:37.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:37.066 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:42:37.067 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:36 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:42:37.067 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:36 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:42:37.067 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:36 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 1444/44908 objects degraded (3.215%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:37.167 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:42:37.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:36 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:42:37.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:36 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:42:37.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:36 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 1444/44908 objects degraded (3.215%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:37.238 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:37.394 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:42:37.394 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:42:37.460 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:42:37.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:37.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:42:37.472 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:42:37.566 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:42:37.585 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:42:37.592 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:37.614 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:42:37.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:37.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:42:37.715 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:42:37.742 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:37.764 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:42:37.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:37.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:42:37.868 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:42:37.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:37 smithi005 ceph-mon[39562]: pgmap v2046: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 1.2 GiB data, 6.9 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.31k op/s; 1162/38764 objects degraded (2.998%); 5.6 MiB/s, 90 objects/s recovering 2024-04-20T17:42:37.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:37 smithi005 ceph-mon[39562]: osdmap e762: 12 total, 11 up, 9 in 2024-04-20T17:42:37.942 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:37 smithi106 ceph-mon[44423]: pgmap v2046: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 1.2 GiB data, 6.9 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.31k op/s; 1162/38764 objects degraded (2.998%); 5.6 MiB/s, 90 objects/s recovering 2024-04-20T17:42:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:37 smithi106 ceph-mon[44423]: osdmap e762: 12 total, 11 up, 9 in 2024-04-20T17:42:38.064 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:37 smithi171 ceph-mon[44189]: pgmap v2046: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 1.2 GiB data, 6.9 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 1.31k op/s; 1162/38764 objects degraded (2.998%); 5.6 MiB/s, 90 objects/s recovering 2024-04-20T17:42:38.064 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:37 smithi171 ceph-mon[44189]: osdmap e762: 12 total, 11 up, 9 in 2024-04-20T17:42:38.104 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-20T17:42:38.452 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:42:38.981 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:38 smithi171 ceph-mon[44189]: pgmap v2048: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 1.2 GiB data, 6.9 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 725 op/s; 1162/38764 objects degraded (2.998%); 2.6 MiB/s, 40 objects/s recovering 2024-04-20T17:42:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:38 smithi106 ceph-mon[44423]: pgmap v2048: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 1.2 GiB data, 6.9 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 725 op/s; 1162/38764 objects degraded (2.998%); 2.6 MiB/s, 40 objects/s recovering 2024-04-20T17:42:39.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:38 smithi005 ceph-mon[39562]: pgmap v2048: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 1.2 GiB data, 6.9 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 725 op/s; 1162/38764 objects degraded (2.998%); 2.6 MiB/s, 40 objects/s recovering 2024-04-20T17:42:39.538 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:42:39.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:39.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:42:39.643 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:42:39.671 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:39.690 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:42:39.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:39.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:42:39.794 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:42:39.871 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:39.912 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:42:39.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:39.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:42:40.024 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:42:40.050 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:40.071 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:42:40.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:40.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:42:40.176 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:42:40.259 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:40.408 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:42:40.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:40.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:42:40.514 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:42:40.586 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:40.627 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:42:40.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:40.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:42:40.680 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:42:40.733 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:42:40.745 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:42:40.765 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:40 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:42:40.765 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:40 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:42:40.765 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:40 smithi005 ceph-mon[39562]: osdmap e763: 12 total, 11 up, 9 in 2024-04-20T17:42:40.765 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:40 smithi005 ceph-mon[39562]: pgmap v2050: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 1.1 GiB data, 6.6 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 806 op/s; 1162/35214 objects degraded (3.300%); 2.9 MiB/s, 46 objects/s recovering 2024-04-20T17:42:40.766 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:40 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/667836423' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:42:40.785 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:42:40.791 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:40.825 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:42:40.864 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:42:40.875 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:42:40.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:40.881 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:42:40.895 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:42:40.929 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:42:40.966 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:42:40.981 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:42:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:40 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:42:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:40 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:42:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:40 smithi106 ceph-mon[44423]: osdmap e763: 12 total, 11 up, 9 in 2024-04-20T17:42:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:40 smithi106 ceph-mon[44423]: pgmap v2050: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 1.1 GiB data, 6.6 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 806 op/s; 1162/35214 objects degraded (3.300%); 2.9 MiB/s, 46 objects/s recovering 2024-04-20T17:42:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:40 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/667836423' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:42:41.012 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:42:41.022 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:40 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:42:41.022 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:40 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:42:41.022 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:40 smithi171 ceph-mon[44189]: osdmap e763: 12 total, 11 up, 9 in 2024-04-20T17:42:41.022 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:40 smithi171 ceph-mon[44189]: pgmap v2050: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 1.1 GiB data, 6.6 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 806 op/s; 1162/35214 objects degraded (3.300%); 2.9 MiB/s, 46 objects/s recovering 2024-04-20T17:42:41.023 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:40 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/667836423' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:42:41.026 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:41.049 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:42:41.085 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:42:41.122 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:42:41.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:41.125 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:42:41.136 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:42:41.136 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:42:41.136 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:42:41.136 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:42:41.137 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:42:41.137 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:42:41.137 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:42:41.137 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:42:41.137 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:42:41.137 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:42:41.137 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:42:41.137 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:42:41.137 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:42:41.197 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:42:41.197 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:42:41.198 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 0 2024-04-20T17:42:41.198 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:42:41.226 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:42:41.287 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:41.378 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:42:41.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:41.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:42:41.485 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:42:41.525 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:41.809 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:41.809 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:41.809 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:41.809 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:41 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:42:41.809 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:41 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:42:41.809 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:41.809 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:41.809 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:41.809 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:41.810 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:41 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 1162/35214 objects degraded (3.300%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:42.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:42.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:42.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:42.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:41 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:42:42.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:41 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:42:42.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:42.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:42.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:42.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:42.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:41 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 1162/35214 objects degraded (3.300%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:42.245 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:42:42.246 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:42:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:41 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:42:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:41 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:42:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:42.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:42.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:41 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 1162/35214 objects degraded (3.300%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:42.379 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-20T17:42:42.388 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:42:42.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:42.394 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:42:42.495 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:42:42.542 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:42.576 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:42:42.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:42.587 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:42:42.688 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:42:42.740 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:42:42.758 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:42.854 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:42:42.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:42.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:42:42.964 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:42:43.030 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:43.030 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:43.030 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:43.030 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:43.031 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:42 smithi005 ceph-mon[39562]: osdmap e764: 12 total, 11 up, 9 in 2024-04-20T17:42:43.031 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:42 smithi005 ceph-mon[39562]: pgmap v2052: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 914 MiB data, 6.3 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 793 op/s; 867/29230 objects degraded (2.966%); 3.1 MiB/s, 49 objects/s recovering 2024-04-20T17:42:43.037 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:43.055 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:43.055 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:43.055 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:43.055 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:43.055 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:42 smithi171 ceph-mon[44189]: osdmap e764: 12 total, 11 up, 9 in 2024-04-20T17:42:43.055 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:42 smithi171 ceph-mon[44189]: pgmap v2052: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 914 MiB data, 6.3 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 793 op/s; 867/29230 objects degraded (2.966%); 3.1 MiB/s, 49 objects/s recovering 2024-04-20T17:42:43.085 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:42:43.096 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:42:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:42 smithi106 ceph-mon[44423]: osdmap e764: 12 total, 11 up, 9 in 2024-04-20T17:42:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:42 smithi106 ceph-mon[44423]: pgmap v2052: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 914 MiB data, 6.3 GiB used, 798 GiB / 805 GiB avail; 0 B/s wr, 793 op/s; 867/29230 objects degraded (2.966%); 3.1 MiB/s, 49 objects/s recovering 2024-04-20T17:42:44.005 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:43 smithi171 ceph-mon[44189]: from='client.27485 v1:172.21.15.5:0/2073915806' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:42:44.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:43 smithi005 ceph-mon[39562]: from='client.27485 v1:172.21.15.5:0/2073915806' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:42:44.246 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:42:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:43 smithi106 ceph-mon[44423]: from='client.27485 v1:172.21.15.5:0/2073915806' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:42:44.676 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:42:44.676 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:42:44.676 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-04-20T17:42:44.676 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-04-20T17:42:44.676 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:42:45.036 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:42:45.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:45.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:42:45.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:44 smithi171 ceph-mon[44189]: pgmap v2053: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 801 MiB data, 6.1 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 837 op/s; 867/25624 objects degraded (3.384%); 2.3 MiB/s, 37 objects/s recovering 2024-04-20T17:42:45.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:44 smithi005 ceph-mon[39562]: pgmap v2053: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 801 MiB data, 6.1 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 837 op/s; 867/25624 objects degraded (3.384%); 2.3 MiB/s, 37 objects/s recovering 2024-04-20T17:42:45.144 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:42:45.213 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:44 smithi106 ceph-mon[44423]: pgmap v2053: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 801 MiB data, 6.1 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 837 op/s; 867/25624 objects degraded (3.384%); 2.3 MiB/s, 37 objects/s recovering 2024-04-20T17:42:45.458 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-20T17:42:45.600 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:42:45.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:45.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:42:45.711 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:42:45.738 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:45.757 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:42:45.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:45.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:42:45.814 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:42:45.861 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:42:45.935 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:46.109 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:42:46.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:46.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:42:46.152 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":2052,"stamp":"2024-04-20T17:42:42.569507+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":958202424,"num_objects":14615,"num_object_clones":0,"num_object_copies":29230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14615,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":277016,"num_write_kb":9329211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17200,"num_bytes_recovered":1129208480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":6609688,"kb_used_data":1880504,"kb_used_omap":217,"kb_used_meta":4728806,"kb_avail":837096680,"statfs":{"total":863955320832,"available":857187000320,"internally_reserved":0,"allocated":1925636096,"data_stored":1921997817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":222986,"internal_metadata":4842297590},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-312410112,"num_objects":-4767,"num_object_clones":0,"num_object_copies":-9534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4767,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":295,"num_bytes_recovered":19333120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.006422"},"pg_stats":[{"pgid":"7.f","version":"763'17507","reported_seq":17755,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-04-20T17:42:41.019444+0000","last_change":"2024-04-20T17:39:40.325470+0000","last_active":"2024-04-20T17:42:41.019444+0000","last_peered":"2024-04-20T17:42:41.019444+0000","last_clean":"2024-04-20T17:42:41.019444+0000","last_became_active":"2024-04-20T17:37:56.835750+0000","last_became_peered":"2024-04-20T17:37:56.835750+0000","last_unstale":"2024-04-20T17:42:41.019444+0000","last_undegraded":"2024-04-20T17:42:41.019444+0000","last_fullsized":"2024-04-20T17:42:41.019444+0000","mapping_epoch":694,"log_start":"716'7500","ondisk_log_start":"716'7500","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9175","last_scrub_stamp":"2024-04-20T17:39:40.325340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:40.325340+0000","objects_scrubbed":9175,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":741,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55246848,"num_objects":843,"num_object_clones":0,"num_object_copies":1686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17507,"num_write_kb":587200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"763'17104","reported_seq":17402,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-04-20T17:42:41.230503+0000","last_change":"2024-04-20T17:41:02.433089+0000","last_active":"2024-04-20T17:42:41.230503+0000","last_peered":"2024-04-20T17:42:41.230503+0000","last_clean":"2024-04-20T17:42:41.230503+0000","last_became_active":"2024-04-20T17:37:56.833703+0000","last_became_peered":"2024-04-20T17:37:56.833703+0000","last_unstale":"2024-04-20T17:42:41.230503+0000","last_undegraded":"2024-04-20T17:42:41.230503+0000","last_fullsized":"2024-04-20T17:42:41.230503+0000","mapping_epoch":694,"log_start":"714'7100","ondisk_log_start":"714'7100","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"742'11794","last_scrub_stamp":"2024-04-20T17:41:02.432782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:41:02.432782+0000","objects_scrubbed":6149,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:45.231592+0000","scrub_duration":1488,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53477376,"num_objects":816,"num_object_clones":0,"num_object_copies":1632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17104,"num_write_kb":573440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"763'19686","reported_seq":20038,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-04-20T17:42:41.019049+0000","last_change":"2024-04-20T17:39:48.311967+0000","last_active":"2024-04-20T17:42:41.019049+0000","last_peered":"2024-04-20T17:42:41.019049+0000","last_clean":"2024-04-20T17:42:41.019049+0000","last_became_active":"2024-04-20T17:37:56.836496+0000","last_became_peered":"2024-04-20T17:37:56.836496+0000","last_unstale":"2024-04-20T17:42:41.019049+0000","last_undegraded":"2024-04-20T17:42:41.019049+0000","last_fullsized":"2024-04-20T17:42:41.019049+0000","mapping_epoch":694,"log_start":"719'9600","ondisk_log_start":"719'9600","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'11204","last_scrub_stamp":"2024-04-20T17:39:48.311825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:48.311825+0000","objects_scrubbed":9436,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62586904,"num_objects":956,"num_object_clones":0,"num_object_copies":1912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":956,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":17918,"num_write_kb":603905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.c","version":"762'17150","reported_seq":17395,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-04-20T17:42:39.365546+0000","last_change":"2024-04-20T17:39:44.683378+0000","last_active":"2024-04-20T17:42:39.365546+0000","last_peered":"2024-04-20T17:42:39.365546+0000","last_clean":"2024-04-20T17:42:39.365546+0000","last_became_active":"2024-04-20T17:37:56.834181+0000","last_became_peered":"2024-04-20T17:37:56.834181+0000","last_unstale":"2024-04-20T17:42:39.365546+0000","last_undegraded":"2024-04-20T17:42:39.365546+0000","last_fullsized":"2024-04-20T17:42:39.365546+0000","mapping_epoch":694,"log_start":"714'7100","ondisk_log_start":"714'7100","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9063","last_scrub_stamp":"2024-04-20T17:39:44.683201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:44.683201+0000","objects_scrubbed":9063,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":746,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63963136,"num_objects":976,"num_object_clones":0,"num_object_copies":1952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17150,"num_write_kb":580032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"762'17368","reported_seq":17689,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-04-20T17:42:39.105075+0000","last_change":"2024-04-20T17:40:58.937747+0000","last_active":"2024-04-20T17:42:39.105075+0000","last_peered":"2024-04-20T17:42:39.105075+0000","last_clean":"2024-04-20T17:42:39.105075+0000","last_became_active":"2024-04-20T17:37:56.830487+0000","last_became_peered":"2024-04-20T17:37:56.830487+0000","last_unstale":"2024-04-20T17:42:39.105075+0000","last_undegraded":"2024-04-20T17:42:39.105075+0000","last_fullsized":"2024-04-20T17:42:39.105075+0000","mapping_epoch":694,"log_start":"715'7300","ondisk_log_start":"715'7300","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"742'11757","last_scrub_stamp":"2024-04-20T17:40:58.937604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:40:58.937604+0000","objects_scrubbed":6664,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:40.096943+0000","scrub_duration":858,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68157440,"num_objects":1040,"num_object_clones":0,"num_object_copies":2080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17368,"num_write_kb":589056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.a","version":"763'17147","reported_seq":17488,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-04-20T17:42:40.934968+0000","last_change":"2024-04-20T17:42:22.174131+0000","last_active":"2024-04-20T17:42:40.934968+0000","last_peered":"2024-04-20T17:42:40.934968+0000","last_clean":"2024-04-20T17:42:40.934968+0000","last_became_active":"2024-04-20T17:37:56.835944+0000","last_became_peered":"2024-04-20T17:37:56.835944+0000","last_unstale":"2024-04-20T17:42:40.934968+0000","last_undegraded":"2024-04-20T17:42:40.934968+0000","last_fullsized":"2024-04-20T17:42:40.934968+0000","mapping_epoch":694,"log_start":"714'7100","ondisk_log_start":"714'7100","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"756'16067","last_scrub_stamp":"2024-04-20T17:42:22.173861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:42:22.173861+0000","objects_scrubbed":1940,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:43:33.214516+0000","scrub_duration":489,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54853632,"num_objects":837,"num_object_clones":0,"num_object_copies":1674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":575488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"763'17284","reported_seq":17514,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-04-20T17:42:41.011707+0000","last_change":"2024-04-20T17:39:03.675435+0000","last_active":"2024-04-20T17:42:41.011707+0000","last_peered":"2024-04-20T17:42:41.011707+0000","last_clean":"2024-04-20T17:42:41.011707+0000","last_became_active":"2024-04-20T17:37:56.835700+0000","last_became_peered":"2024-04-20T17:37:56.835700+0000","last_unstale":"2024-04-20T17:42:41.011707+0000","last_undegraded":"2024-04-20T17:42:41.011707+0000","last_fullsized":"2024-04-20T17:42:41.011707+0000","mapping_epoch":694,"log_start":"714'7200","ondisk_log_start":"714'7200","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'7175","last_scrub_stamp":"2024-04-20T17:39:03.675211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:03.675211+0000","objects_scrubbed":7133,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55443456,"num_objects":846,"num_object_clones":0,"num_object_copies":1692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17284,"num_write_kb":580160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11],"acting":[0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"763'20849","reported_seq":39497,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-04-20T17:42:41.187657+0000","last_change":"2024-04-20T17:42:02.804301+0000","last_active":"2024-04-20T17:42:41.187657+0000","last_peered":"2024-04-20T17:42:41.187657+0000","last_clean":"2024-04-20T17:42:41.187657+0000","last_became_active":"2024-04-20T17:40:09.546695+0000","last_became_peered":"2024-04-20T17:40:09.546695+0000","last_unstale":"2024-04-20T17:42:41.187657+0000","last_undegraded":"2024-04-20T17:42:41.187657+0000","last_fullsized":"2024-04-20T17:42:41.187657+0000","mapping_epoch":731,"log_start":"719'10801","ondisk_log_start":"719'10801","created":694,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'18634","last_scrub_stamp":"2024-04-20T17:42:02.804073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:42:02.804073+0000","objects_scrubbed":3082,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:43:21.646311+0000","scrub_duration":620,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56164352,"num_objects":857,"num_object_clones":0,"num_object_copies":1714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17279,"num_write_kb":580352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9062,"num_bytes_recovered":593887232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"762'17326","reported_seq":17671,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-04-20T17:42:38.731419+0000","last_change":"2024-04-20T17:42:04.144551+0000","last_active":"2024-04-20T17:42:38.731419+0000","last_peered":"2024-04-20T17:42:38.731419+0000","last_clean":"2024-04-20T17:42:38.731419+0000","last_became_active":"2024-04-20T17:37:56.836665+0000","last_became_peered":"2024-04-20T17:37:56.836665+0000","last_unstale":"2024-04-20T17:42:38.731419+0000","last_undegraded":"2024-04-20T17:42:38.731419+0000","last_fullsized":"2024-04-20T17:42:38.731419+0000","mapping_epoch":694,"log_start":"714'7300","ondisk_log_start":"714'7300","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'15376","last_scrub_stamp":"2024-04-20T17:42:04.144333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:42:04.144333+0000","objects_scrubbed":2961,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:43:15.847365+0000","scrub_duration":538,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65667072,"num_objects":1002,"num_object_clones":0,"num_object_copies":2004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17326,"num_write_kb":586496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.5","version":"762'17302","reported_seq":17644,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-04-20T17:42:39.092954+0000","last_change":"2024-04-20T17:42:03.424949+0000","last_active":"2024-04-20T17:42:39.092954+0000","last_peered":"2024-04-20T17:42:39.092954+0000","last_clean":"2024-04-20T17:42:39.092954+0000","last_became_active":"2024-04-20T17:37:56.833600+0000","last_became_peered":"2024-04-20T17:37:56.833600+0000","last_unstale":"2024-04-20T17:42:39.092954+0000","last_undegraded":"2024-04-20T17:42:39.092954+0000","last_fullsized":"2024-04-20T17:42:39.092954+0000","mapping_epoch":694,"log_start":"714'7300","ondisk_log_start":"714'7300","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'15267","last_scrub_stamp":"2024-04-20T17:42:03.424811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:42:03.424811+0000","objects_scrubbed":3060,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:43:32.116339+0000","scrub_duration":489,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66322432,"num_objects":1012,"num_object_clones":0,"num_object_copies":2024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17302,"num_write_kb":586048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.4","version":"763'17194","reported_seq":17441,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-04-20T17:42:41.037319+0000","last_change":"2024-04-20T17:39:50.365478+0000","last_active":"2024-04-20T17:42:41.037319+0000","last_peered":"2024-04-20T17:42:41.037319+0000","last_clean":"2024-04-20T17:42:41.037319+0000","last_became_active":"2024-04-20T17:37:56.834727+0000","last_became_peered":"2024-04-20T17:37:56.834727+0000","last_unstale":"2024-04-20T17:42:41.037319+0000","last_undegraded":"2024-04-20T17:42:41.037319+0000","last_fullsized":"2024-04-20T17:42:41.037319+0000","mapping_epoch":694,"log_start":"714'7100","ondisk_log_start":"714'7100","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9031","last_scrub_stamp":"2024-04-20T17:39:50.365339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:50.365339+0000","objects_scrubbed":9031,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":876,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56885248,"num_objects":868,"num_object_clones":0,"num_object_copies":1736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17194,"num_write_kb":577984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"762'17069","reported_seq":17292,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-04-20T17:42:39.360682+0000","last_change":"2024-04-20T17:39:00.502710+0000","last_active":"2024-04-20T17:42:39.360682+0000","last_peered":"2024-04-20T17:42:39.360682+0000","last_clean":"2024-04-20T17:42:39.360682+0000","last_became_active":"2024-04-20T17:37:56.832911+0000","last_became_peered":"2024-04-20T17:37:56.832911+0000","last_unstale":"2024-04-20T17:42:39.360682+0000","last_undegraded":"2024-04-20T17:42:39.360682+0000","last_fullsized":"2024-04-20T17:42:39.360682+0000","mapping_epoch":694,"log_start":"714'7000","ondisk_log_start":"714'7000","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'6831","last_scrub_stamp":"2024-04-20T17:39:00.502556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:00.502556+0000","objects_scrubbed":6782,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65732608,"num_objects":1003,"num_object_clones":0,"num_object_copies":2006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17069,"num_write_kb":578304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"762'17540","reported_seq":17770,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-04-20T17:42:39.388518+0000","last_change":"2024-04-20T17:39:08.489529+0000","last_active":"2024-04-20T17:42:39.388518+0000","last_peered":"2024-04-20T17:42:39.388518+0000","last_clean":"2024-04-20T17:42:39.388518+0000","last_became_active":"2024-04-20T17:37:56.834292+0000","last_became_peered":"2024-04-20T17:37:56.834292+0000","last_unstale":"2024-04-20T17:42:39.388518+0000","last_undegraded":"2024-04-20T17:42:39.388518+0000","last_fullsized":"2024-04-20T17:42:39.388518+0000","mapping_epoch":694,"log_start":"716'7500","ondisk_log_start":"716'7500","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"716'7577","last_scrub_stamp":"2024-04-20T17:39:08.489308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:08.489308+0000","objects_scrubbed":7524,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62390272,"num_objects":952,"num_object_clones":0,"num_object_copies":1904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17540,"num_write_kb":591744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"763'19393","reported_seq":19805,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-04-20T17:42:40.719482+0000","last_change":"2024-04-20T17:42:25.987544+0000","last_active":"2024-04-20T17:42:40.719482+0000","last_peered":"2024-04-20T17:42:40.719482+0000","last_clean":"2024-04-20T17:42:40.719482+0000","last_became_active":"2024-04-20T17:37:56.835512+0000","last_became_peered":"2024-04-20T17:37:56.835512+0000","last_unstale":"2024-04-20T17:42:40.719482+0000","last_undegraded":"2024-04-20T17:42:40.719482+0000","last_fullsized":"2024-04-20T17:42:40.719482+0000","mapping_epoch":694,"log_start":"733'9300","ondisk_log_start":"733'9300","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"758'18500","last_scrub_stamp":"2024-04-20T17:42:25.987445+0000","last_deep_scrub":"758'18500","last_deep_scrub_stamp":"2024-04-20T17:42:25.987445+0000","last_clean_scrub_stamp":"2024-04-20T17:42:25.987445+0000","objects_scrubbed":2091,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":12,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:43:26.213160+0000","scrub_duration":11980,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57671680,"num_objects":880,"num_object_clones":0,"num_object_copies":1760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17302,"num_write_kb":581824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"763'17221","reported_seq":17531,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-04-20T17:42:40.957213+0000","last_change":"2024-04-20T17:41:16.382261+0000","last_active":"2024-04-20T17:42:40.957213+0000","last_peered":"2024-04-20T17:42:40.957213+0000","last_clean":"2024-04-20T17:42:40.957213+0000","last_became_active":"2024-04-20T17:37:56.833972+0000","last_became_peered":"2024-04-20T17:37:56.833972+0000","last_unstale":"2024-04-20T17:42:40.957213+0000","last_undegraded":"2024-04-20T17:42:40.957213+0000","last_fullsized":"2024-04-20T17:42:40.957213+0000","mapping_epoch":694,"log_start":"714'7200","ondisk_log_start":"714'7200","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"745'12483","last_scrub_stamp":"2024-04-20T17:41:16.382018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:41:16.382018+0000","objects_scrubbed":5596,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:42:42.520100+0000","scrub_duration":653,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55246848,"num_objects":843,"num_object_clones":0,"num_object_copies":1686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17221,"num_write_kb":578048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.6","version":"763'17124","reported_seq":33613,"reported_epoch":763,"state":"active+recovering+degraded","last_fresh":"2024-04-20T17:42:41.049587+0000","last_change":"2024-04-20T17:39:54.962650+0000","last_active":"2024-04-20T17:42:41.049587+0000","last_peered":"2024-04-20T17:42:41.049587+0000","last_clean":"2024-04-20T17:39:34.507147+0000","last_became_active":"2024-04-20T17:39:54.933393+0000","last_became_peered":"2024-04-20T17:39:54.933393+0000","last_unstale":"2024-04-20T17:42:41.049587+0000","last_undegraded":"2024-04-20T17:39:54.578981+0000","last_fullsized":"2024-04-20T17:42:41.049587+0000","mapping_epoch":727,"log_start":"714'7100","ondisk_log_start":"714'7100","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'7141","last_scrub_stamp":"2024-04-20T17:39:04.435913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:04.435913+0000","objects_scrubbed":7101,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57802752,"num_objects":882,"num_object_clones":0,"num_object_copies":1764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17124,"num_write_kb":576192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8128,"num_bytes_recovered":532676608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":867},{"shards":"0,3","objects":15}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":544,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-04-20T17:42:36.722968+0000","last_change":"2024-04-20T17:42:20.165296+0000","last_active":"2024-04-20T17:42:36.722968+0000","last_peered":"2024-04-20T17:42:36.722968+0000","last_clean":"2024-04-20T17:42:36.722968+0000","last_became_active":"2024-04-20T17:39:54.970208+0000","last_became_peered":"2024-04-20T17:39:54.970208+0000","last_unstale":"2024-04-20T17:42:36.722968+0000","last_undegraded":"2024-04-20T17:42:36.722968+0000","last_fullsized":"2024-04-20T17:42:36.722968+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:42:20.165149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:42:20.165149+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:43:22.534703+0000","scrub_duration":180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":957612056,"num_objects":14613,"num_object_clones":0,"num_object_copies":29226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14613,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":276835,"num_write_kb":9326273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17190,"num_bytes_recovered":1126563840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3033800704,"data_stored":3033792560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160763,"ondisk_log_size":160763,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991676,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":974676,"kb_used_data":271564,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":92770476,"statfs":{"total":95995035648,"available":94996967424,"internally_reserved":0,"allocated":278081536,"data_stored":277678745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":719976900},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253315,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":737360,"kb_used_data":182788,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":93007792,"statfs":{"total":95995035648,"available":95239979008,"internally_reserved":0,"allocated":187174912,"data_stored":186775673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":567867840},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613064,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":722724,"kb_used_data":172312,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":93022428,"statfs":{"total":95995035648,"available":95254966272,"internally_reserved":0,"allocated":176447488,"data_stored":176038713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":563608000},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809294,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":458448,"kb_used_data":130180,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":93286704,"statfs":{"total":95995035648,"available":95525584896,"internally_reserved":0,"allocated":133304320,"data_stored":132905101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,1,2,3,4,5,6,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":2297807503525,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":2830383448147,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738869,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":951904,"kb_used_data":284820,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":92793248,"statfs":{"total":95995035648,"available":95020285952,"internally_reserved":0,"allocated":291655680,"data_stored":291246385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394159,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":550876,"kb_used_data":217872,"kb_used_omap":202,"kb_used_meta":332789,"kb_avail":93194276,"statfs":{"total":95995035648,"available":95430938624,"internally_reserved":0,"allocated":223100928,"data_stored":222695705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":340776407},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653208,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":743376,"kb_used_data":327816,"kb_used_omap":4,"kb_used_meta":415547,"kb_avail":93001776,"statfs":{"total":95995035648,"available":95233818624,"internally_reserved":0,"allocated":335683584,"data_stored":335280293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":425520839},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986657,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":713688,"kb_used_data":177168,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93031464,"statfs":{"total":95995035648,"available":95264219136,"internally_reserved":0,"allocated":181420032,"data_stored":181014809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790427,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":756636,"kb_used_data":115984,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":92988516,"statfs":{"total":95995035648,"available":95220240384,"internally_reserved":0,"allocated":118767616,"data_stored":118362393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":287707136,"data_stored":287703064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":221315072,"data_stored":221315072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":325390336,"data_stored":325386264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":173473792,"data_stored":173473792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":117440512,"data_stored":117440512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1167458304,"data_stored":1167458304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":129302528,"data_stored":129302528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":169017344,"data_stored":169017344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":180551680,"data_stored":180551680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":262144000,"data_stored":262144000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:42:46.154 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:42:46.175 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:42:46.223 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:42:46.316 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:46.440 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:42:46.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:46.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:42:46.545 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:42:46.635 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:46 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 867/25624 objects degraded (3.384%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:46.646 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:46.782 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:46 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 867/25624 objects degraded (3.384%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:47.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:46 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 867/25624 objects degraded (3.384%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:47.406 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:42:47.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:47.410 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:42:47.512 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:42:47.558 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:47.657 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:42:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:47 smithi171 ceph-mon[44189]: pgmap v2054: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 610 MiB data, 5.8 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.20k op/s; 557/19506 objects degraded (2.856%); 4.7 MiB/s, 75 objects/s recovering 2024-04-20T17:42:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:47 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/479214346' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:42:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:47 smithi171 ceph-mon[44189]: from='client.27373 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:42:47.967 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:47 smithi005 ceph-mon[39562]: pgmap v2054: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 610 MiB data, 5.8 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.20k op/s; 557/19506 objects degraded (2.856%); 4.7 MiB/s, 75 objects/s recovering 2024-04-20T17:42:47.967 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:47 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/479214346' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:42:47.967 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:47 smithi005 ceph-mon[39562]: from='client.27373 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:42:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:47 smithi106 ceph-mon[44423]: pgmap v2054: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 610 MiB data, 5.8 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 1.20k op/s; 557/19506 objects degraded (2.856%); 4.7 MiB/s, 75 objects/s recovering 2024-04-20T17:42:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:47 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/479214346' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:42:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:47 smithi106 ceph-mon[44423]: from='client.27373 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:42:48.335 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:42:48.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:48.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:42:48.444 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:42:48.491 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:48.541 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:42:48.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:48.544 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:42:48.609 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:42:48.609 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:42:48.609 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:42:48.609 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:42:48.609 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.609 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.26882.0:313715 6.8 e679/619 6:15247d69:::benchmark_data_smithi171_147972_object152617:head v 679'19417, mlcod=679'19411)", 2024-04-20T17:42:48.609 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:37:25.319461+0000", 2024-04-20T17:42:48.609 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 323.28712565400002, 2024-04-20T17:42:48.609 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.086674556999999999, 2024-04-20T17:42:48.609 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:48.610 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:48.610 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:48.610 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:48.610 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.610 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:48.610 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.319461+0000", 2024-04-20T17:42:48.610 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.610 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.610 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.610 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:48.610 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.319461+0000", 2024-04-20T17:42:48.610 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.610 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.611 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.611 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:48.611 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.319462+0000", 2024-04-20T17:42:48.611 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9999999999999997e-07 2024-04-20T17:42:48.611 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.611 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.611 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:48.611 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.319467+0000", 2024-04-20T17:42:48.611 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7520000000000004e-06 2024-04-20T17:42:48.611 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.611 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.611 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:48.611 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.319472+0000", 2024-04-20T17:42:48.611 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5210000000000002e-06 2024-04-20T17:42:48.612 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.612 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.612 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:48.612 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.319477+0000", 2024-04-20T17:42:48.612 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8250000000000004e-06 2024-04-20T17:42:48.612 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.612 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.612 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:48.612 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.319539+0000", 2024-04-20T17:42:48.612 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1933000000000005e-05 2024-04-20T17:42:48.612 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.612 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.612 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:48.612 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.319556+0000", 2024-04-20T17:42:48.613 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7609e-05 2024-04-20T17:42:48.613 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.613 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.613 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:48.613 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.406086+0000", 2024-04-20T17:42:48.613 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.086529668000000004 2024-04-20T17:42:48.613 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.613 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.613 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:48.613 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.406136+0000", 2024-04-20T17:42:48.613 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9648999999999998e-05 2024-04-20T17:42:48.613 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.613 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:48.613 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.614 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.614 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.614 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.26882.0:313740 6.8 e679/619 6:109323e2:::benchmark_data_smithi171_147972_object152642:head v 679'19418, mlcod=679'19411)", 2024-04-20T17:42:48.614 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:37:25.338547+0000", 2024-04-20T17:42:48.614 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 323.26803967799998, 2024-04-20T17:42:48.614 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.067624529000000003, 2024-04-20T17:42:48.614 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:48.614 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:48.614 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:48.614 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:48.614 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.614 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:48.614 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.338547+0000", 2024-04-20T17:42:48.615 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.615 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.615 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.615 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:48.615 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.338547+0000", 2024-04-20T17:42:48.615 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.615 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.615 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.615 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:48.615 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.338549+0000", 2024-04-20T17:42:48.615 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7129999999999999e-06 2024-04-20T17:42:48.615 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.615 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.615 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:48.615 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.338560+0000", 2024-04-20T17:42:48.616 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0908e-05 2024-04-20T17:42:48.616 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.616 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.616 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:48.616 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.338570+0000", 2024-04-20T17:42:48.616 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0659000000000001e-05 2024-04-20T17:42:48.616 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.616 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.616 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:48.616 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.338579+0000", 2024-04-20T17:42:48.616 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1019999999999997e-06 2024-04-20T17:42:48.616 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.616 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.616 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:48.616 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.338638+0000", 2024-04-20T17:42:48.617 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9231000000000003e-05 2024-04-20T17:42:48.617 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.617 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.617 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:48.617 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.338659+0000", 2024-04-20T17:42:48.617 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1506999999999999e-05 2024-04-20T17:42:48.617 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.617 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.617 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:48.617 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.406141+0000", 2024-04-20T17:42:48.617 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.067482217999999997 2024-04-20T17:42:48.617 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.617 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.617 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:48.617 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.406172+0000", 2024-04-20T17:42:48.618 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0190999999999999e-05 2024-04-20T17:42:48.618 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.618 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:48.618 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.618 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.618 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.618 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.26882.0:313762 6.8 e679/619 6:15fed4ad:::benchmark_data_smithi171_147972_object152664:head v 679'19419, mlcod=679'19415)", 2024-04-20T17:42:48.618 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:37:25.356126+0000", 2024-04-20T17:42:48.618 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 323.25046048899998, 2024-04-20T17:42:48.619 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.050072311000000001, 2024-04-20T17:42:48.619 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:48.619 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:48.619 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:48.619 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:48.619 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.619 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:48.619 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.356126+0000", 2024-04-20T17:42:48.619 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.619 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.619 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.619 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:48.619 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.356126+0000", 2024-04-20T17:42:48.619 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.619 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.620 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.620 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:48.620 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.356128+0000", 2024-04-20T17:42:48.620 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1409999999999999e-06 2024-04-20T17:42:48.620 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.620 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.620 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:48.620 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.356140+0000", 2024-04-20T17:42:48.620 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1635e-05 2024-04-20T17:42:48.620 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.620 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.620 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:48.620 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.356151+0000", 2024-04-20T17:42:48.620 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0995000000000001e-05 2024-04-20T17:42:48.621 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.621 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.621 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:48.621 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.356160+0000", 2024-04-20T17:42:48.621 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4409999999999993e-06 2024-04-20T17:42:48.621 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.621 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.621 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:48.621 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.356224+0000", 2024-04-20T17:42:48.621 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4292e-05 2024-04-20T17:42:48.621 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.621 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.621 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:48.621 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.356245+0000", 2024-04-20T17:42:48.621 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1515e-05 2024-04-20T17:42:48.622 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.622 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.622 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:48.622 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.406175+0000", 2024-04-20T17:42:48.622 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.049929603000000003 2024-04-20T17:42:48.622 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.622 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.622 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:48.622 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:37:25.406199+0000", 2024-04-20T17:42:48.622 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3689e-05 2024-04-20T17:42:48.622 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.622 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:48.622 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.622 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.623 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.623 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:53991 7.5 7:a91bcedb:::benchmark_data_smithi171_161871_object53990:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e702)", 2024-04-20T17:42:48.623 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:25.779923+0000", 2024-04-20T17:42:48.623 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 262.82666427499998, 2024-04-20T17:42:48.623 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.069622137000000001, 2024-04-20T17:42:48.623 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:48.623 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:48.623 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:48.623 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:48.623 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:48.623 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:48.623 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 53991 2024-04-20T17:42:48.623 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.623 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:48.624 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.624 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:48.624 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.779923+0000", 2024-04-20T17:42:48.624 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.624 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.624 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.624 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:48.624 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.779923+0000", 2024-04-20T17:42:48.624 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.624 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.624 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.624 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:48.624 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.779924+0000", 2024-04-20T17:42:48.624 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.457e-06 2024-04-20T17:42:48.625 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.625 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.625 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:48.625 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.779947+0000", 2024-04-20T17:42:48.625 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3433000000000001e-05 2024-04-20T17:42:48.625 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.625 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.625 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:48.625 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.779958+0000", 2024-04-20T17:42:48.625 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0385e-05 2024-04-20T17:42:48.625 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.625 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.625 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:48.625 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.779965+0000", 2024-04-20T17:42:48.626 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6029999999999999e-06 2024-04-20T17:42:48.626 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.626 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.626 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:48.626 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.781542+0000", 2024-04-20T17:42:48.626 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0015767000000000001 2024-04-20T17:42:48.626 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.626 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.626 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:48.626 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.781682+0000", 2024-04-20T17:42:48.626 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014012700000000001 2024-04-20T17:42:48.626 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.626 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.626 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:42:48.627 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.781815+0000", 2024-04-20T17:42:48.627 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013292299999999999 2024-04-20T17:42:48.627 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.627 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.627 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:48.627 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.785249+0000", 2024-04-20T17:42:48.627 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.003433827 2024-04-20T17:42:48.627 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.627 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.627 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:48.627 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.849466+0000", 2024-04-20T17:42:48.627 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.06421694 2024-04-20T17:42:48.627 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.627 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.628 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:48.628 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.849508+0000", 2024-04-20T17:42:48.628 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1569999999999997e-05 2024-04-20T17:42:48.628 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.628 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.628 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:48.628 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.849545+0000", 2024-04-20T17:42:48.628 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7172000000000003e-05 2024-04-20T17:42:48.628 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.628 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:48.628 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.628 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.628 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.628 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:53992 7.5 7:ac4011ec:::benchmark_data_smithi171_161871_object53991:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e702)", 2024-04-20T17:42:48.628 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:25.781268+0000", 2024-04-20T17:42:48.629 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 262.82531883299998, 2024-04-20T17:42:48.629 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.068393861, 2024-04-20T17:42:48.629 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:48.629 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:48.629 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:48.629 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:48.629 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:48.629 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:48.629 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 53992 2024-04-20T17:42:48.629 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.629 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:48.629 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.629 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:48.629 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.781268+0000", 2024-04-20T17:42:48.630 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.630 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.630 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.630 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:48.630 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.781268+0000", 2024-04-20T17:42:48.630 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.630 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.630 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.630 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:48.630 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.781270+0000", 2024-04-20T17:42:48.630 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0700000000000001e-06 2024-04-20T17:42:48.630 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.630 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.631 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:48.631 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.781296+0000", 2024-04-20T17:42:48.631 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6089999999999999e-05 2024-04-20T17:42:48.631 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.631 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.631 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:48.631 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.781307+0000", 2024-04-20T17:42:48.631 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1034e-05 2024-04-20T17:42:48.631 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.631 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.631 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:48.631 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.781315+0000", 2024-04-20T17:42:48.631 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2849999999999995e-06 2024-04-20T17:42:48.631 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.632 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.632 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:48.632 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.783356+0000", 2024-04-20T17:42:48.632 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0020403069999999999 2024-04-20T17:42:48.632 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.632 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.632 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:48.632 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.783427+0000", 2024-04-20T17:42:48.632 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1161000000000005e-05 2024-04-20T17:42:48.632 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.632 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.632 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:42:48.632 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.783549+0000", 2024-04-20T17:42:48.632 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000122187 2024-04-20T17:42:48.632 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.632 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.633 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:48.633 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.849585+0000", 2024-04-20T17:42:48.633 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.066036279000000003 2024-04-20T17:42:48.633 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.633 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.633 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:48.633 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.849596+0000", 2024-04-20T17:42:48.633 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0464e-05 2024-04-20T17:42:48.633 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.633 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.633 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:48.633 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.849632+0000", 2024-04-20T17:42:48.633 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5864000000000001e-05 2024-04-20T17:42:48.633 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.633 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.634 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:48.634 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.849662+0000", 2024-04-20T17:42:48.634 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.012e-05 2024-04-20T17:42:48.634 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.634 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:48.634 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.634 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.634 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.634 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:53994 7.b 7:dab5472f:::benchmark_data_smithi171_161871_object53993:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e702)", 2024-04-20T17:42:48.634 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:25.782427+0000", 2024-04-20T17:42:48.634 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 262.824160173, 2024-04-20T17:42:48.634 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.065597904999999998, 2024-04-20T17:42:48.634 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:48.634 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:48.635 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:48.635 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:48.635 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:48.635 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:48.635 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 53994 2024-04-20T17:42:48.635 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.635 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:48.635 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.635 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:48.635 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782427+0000", 2024-04-20T17:42:48.635 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.635 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.635 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.635 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:48.636 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782427+0000", 2024-04-20T17:42:48.636 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.636 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.636 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.636 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:48.636 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782428+0000", 2024-04-20T17:42:48.636 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6640000000000001e-06 2024-04-20T17:42:48.636 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.636 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.636 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:48.636 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782450+0000", 2024-04-20T17:42:48.636 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1460000000000001e-05 2024-04-20T17:42:48.636 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.636 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.636 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:48.637 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782459+0000", 2024-04-20T17:42:48.637 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6169999999999996e-06 2024-04-20T17:42:48.637 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.637 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.637 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:48.637 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782466+0000", 2024-04-20T17:42:48.637 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9539999999999998e-06 2024-04-20T17:42:48.637 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.637 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.637 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:48.637 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782569+0000", 2024-04-20T17:42:48.637 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010242300000000001 2024-04-20T17:42:48.637 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.637 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.638 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:48.638 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782682+0000", 2024-04-20T17:42:48.638 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000113454 2024-04-20T17:42:48.638 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.638 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.638 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:42:48.638 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782802+0000", 2024-04-20T17:42:48.638 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000119993 2024-04-20T17:42:48.638 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.638 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.638 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:48.638 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.847413+0000", 2024-04-20T17:42:48.638 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.064610483999999996 2024-04-20T17:42:48.638 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.639 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.639 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:48.639 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.847932+0000", 2024-04-20T17:42:48.639 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00051915299999999995 2024-04-20T17:42:48.639 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.639 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.639 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:48.639 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.847982+0000", 2024-04-20T17:42:48.639 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9793000000000003e-05 2024-04-20T17:42:48.639 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.639 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.639 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:48.639 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.848025+0000", 2024-04-20T17:42:48.639 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2910000000000001e-05 2024-04-20T17:42:48.640 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.640 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:48.640 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.640 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.640 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.640 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:53997 7.b 7:d526e560:::benchmark_data_smithi171_161871_object53996:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e702)", 2024-04-20T17:42:48.640 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:25.782484+0000", 2024-04-20T17:42:48.640 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 262.82410306700001, 2024-04-20T17:42:48.640 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.065644577999999995, 2024-04-20T17:42:48.640 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:48.640 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:48.640 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:48.640 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:48.640 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:48.641 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:48.641 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 53997 2024-04-20T17:42:48.641 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.641 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:48.641 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.641 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:48.641 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782484+0000", 2024-04-20T17:42:48.641 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.641 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.641 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.641 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:48.641 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782484+0000", 2024-04-20T17:42:48.641 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.641 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.641 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.642 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:48.642 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782485+0000", 2024-04-20T17:42:48.642 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6300000000000004e-07 2024-04-20T17:42:48.642 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.642 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.642 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:48.642 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782508+0000", 2024-04-20T17:42:48.642 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3625000000000002e-05 2024-04-20T17:42:48.642 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.642 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.642 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:48.642 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782514+0000", 2024-04-20T17:42:48.642 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1739999999999997e-06 2024-04-20T17:42:48.642 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.643 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.643 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:48.643 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.782520+0000", 2024-04-20T17:42:48.643 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2000000000000002e-06 2024-04-20T17:42:48.643 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.643 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.643 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:48.643 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.784440+0000", 2024-04-20T17:42:48.643 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001920539 2024-04-20T17:42:48.643 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.643 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.643 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:48.643 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.784611+0000", 2024-04-20T17:42:48.643 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00017075200000000001 2024-04-20T17:42:48.643 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.644 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.644 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:42:48.644 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.784752+0000", 2024-04-20T17:42:48.644 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014117200000000001 2024-04-20T17:42:48.644 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.644 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.644 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:48.644 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.847421+0000", 2024-04-20T17:42:48.644 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.062669425000000001 2024-04-20T17:42:48.644 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.644 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.644 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:48.644 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.848059+0000", 2024-04-20T17:42:48.644 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00063743100000000004 2024-04-20T17:42:48.644 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.645 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.645 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:48.645 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.848098+0000", 2024-04-20T17:42:48.645 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9391000000000003e-05 2024-04-20T17:42:48.645 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.645 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.645 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:48.645 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.848128+0000", 2024-04-20T17:42:48.645 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0105999999999999e-05 2024-04-20T17:42:48.645 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.645 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:48.645 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.645 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.645 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.645 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:53999 7.5 7:ae11ed78:::benchmark_data_smithi171_161871_object53998:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e702)", 2024-04-20T17:42:48.646 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:25.784882+0000", 2024-04-20T17:42:48.646 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 262.82170477199998, 2024-04-20T17:42:48.646 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.070871580000000003, 2024-04-20T17:42:48.646 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:48.646 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:48.646 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:48.646 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:48.646 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:48.646 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:48.646 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 53999 2024-04-20T17:42:48.646 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.646 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:48.647 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.647 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:48.647 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.784882+0000", 2024-04-20T17:42:48.647 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.647 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.647 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.647 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:48.647 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.784882+0000", 2024-04-20T17:42:48.647 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.647 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.647 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.647 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:48.647 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.784884+0000", 2024-04-20T17:42:48.647 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7090000000000001e-06 2024-04-20T17:42:48.648 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.648 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.648 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:48.648 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.784906+0000", 2024-04-20T17:42:48.648 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2039e-05 2024-04-20T17:42:48.648 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.648 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.648 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:48.648 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.784915+0000", 2024-04-20T17:42:48.648 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9840000000000007e-06 2024-04-20T17:42:48.648 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.648 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.648 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:48.648 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.784922+0000", 2024-04-20T17:42:48.648 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4030000000000001e-06 2024-04-20T17:42:48.649 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.649 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.649 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:48.649 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.785285+0000", 2024-04-20T17:42:48.649 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00036267800000000001 2024-04-20T17:42:48.649 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.649 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.649 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:48.649 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.847340+0000", 2024-04-20T17:42:48.649 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.062055297000000002 2024-04-20T17:42:48.649 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.649 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.649 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:42:48.649 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.847512+0000", 2024-04-20T17:42:48.650 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00017150499999999999 2024-04-20T17:42:48.650 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.650 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.650 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:48.650 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.853827+0000", 2024-04-20T17:42:48.650 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0063151709999999996 2024-04-20T17:42:48.650 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.650 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.650 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:48.650 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.855653+0000", 2024-04-20T17:42:48.650 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0018264519999999999 2024-04-20T17:42:48.650 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.650 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.650 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:48.651 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.855709+0000", 2024-04-20T17:42:48.651 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5624000000000001e-05 2024-04-20T17:42:48.651 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.651 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.651 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:48.651 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.855754+0000", 2024-04-20T17:42:48.651 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4718000000000001e-05 2024-04-20T17:42:48.651 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.651 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:48.651 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.651 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.651 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.651 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:54007 7.b 7:df96cf41:::benchmark_data_smithi171_161871_object54006:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e702)", 2024-04-20T17:42:48.651 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:25.785411+0000", 2024-04-20T17:42:48.652 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 262.82117563200001, 2024-04-20T17:42:48.652 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.067145148000000002, 2024-04-20T17:42:48.652 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:48.652 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:48.652 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:48.652 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:48.652 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:48.652 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:48.652 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 54007 2024-04-20T17:42:48.652 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.652 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:48.652 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.652 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:48.652 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.785411+0000", 2024-04-20T17:42:48.653 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.653 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.653 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.653 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:48.653 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.785411+0000", 2024-04-20T17:42:48.653 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.653 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.653 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.653 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:48.653 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.785413+0000", 2024-04-20T17:42:48.653 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3599999999999999e-06 2024-04-20T17:42:48.653 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.653 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.653 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:48.653 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.785435+0000", 2024-04-20T17:42:48.654 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2212e-05 2024-04-20T17:42:48.654 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.654 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.654 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:48.654 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.847536+0000", 2024-04-20T17:42:48.654 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.062101330000000003 2024-04-20T17:42:48.654 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.654 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.654 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:48.654 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.847545+0000", 2024-04-20T17:42:48.654 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8419999999999994e-06 2024-04-20T17:42:48.654 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.654 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.655 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:48.655 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.848154+0000", 2024-04-20T17:42:48.655 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00060890200000000003 2024-04-20T17:42:48.655 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.655 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.655 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:48.655 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.848282+0000", 2024-04-20T17:42:48.655 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012847299999999999 2024-04-20T17:42:48.655 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.655 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.655 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:42:48.655 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.848405+0000", 2024-04-20T17:42:48.655 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012244299999999999 2024-04-20T17:42:48.655 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.655 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.656 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:48.656 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.852027+0000", 2024-04-20T17:42:48.656 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.003622063 2024-04-20T17:42:48.656 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.656 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.656 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:48.656 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.852428+0000", 2024-04-20T17:42:48.656 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000401297 2024-04-20T17:42:48.656 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.656 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.656 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:48.656 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.852497+0000", 2024-04-20T17:42:48.656 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9077000000000003e-05 2024-04-20T17:42:48.656 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.657 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.657 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:48.657 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.852556+0000", 2024-04-20T17:42:48.657 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9148999999999999e-05 2024-04-20T17:42:48.657 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.657 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:48.657 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.657 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.657 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.657 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.27088.0:54025 7.c e702/694 7:32242b18:::benchmark_data_smithi171_161871_object54024:head v 702'3348, mlcod=702'3346)", 2024-04-20T17:42:48.657 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:25.796222+0000", 2024-04-20T17:42:48.657 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 262.81036477599997, 2024-04-20T17:42:48.657 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.056523057000000002, 2024-04-20T17:42:48.657 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:48.657 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:48.658 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:48.658 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:48.658 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.658 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:48.658 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.796222+0000", 2024-04-20T17:42:48.658 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.658 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.658 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.658 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:48.658 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.796222+0000", 2024-04-20T17:42:48.658 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.658 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.658 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.658 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:48.659 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.796226+0000", 2024-04-20T17:42:48.659 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8e-06 2024-04-20T17:42:48.659 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.659 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.659 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:48.659 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.796301+0000", 2024-04-20T17:42:48.659 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4814000000000006e-05 2024-04-20T17:42:48.659 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.659 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.659 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:48.659 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.847347+0000", 2024-04-20T17:42:48.659 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.051046455999999997 2024-04-20T17:42:48.659 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.659 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.659 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:48.660 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.847358+0000", 2024-04-20T17:42:48.660 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0852999999999999e-05 2024-04-20T17:42:48.660 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.660 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.660 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:48.660 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.847463+0000", 2024-04-20T17:42:48.660 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010525400000000001 2024-04-20T17:42:48.660 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.660 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.660 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:48.660 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.847487+0000", 2024-04-20T17:42:48.660 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4015e-05 2024-04-20T17:42:48.660 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.660 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.660 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:48.661 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.852700+0000", 2024-04-20T17:42:48.661 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0052130459999999998 2024-04-20T17:42:48.661 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.661 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.661 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:48.661 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:25.852745+0000", 2024-04-20T17:42:48.661 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4818999999999999e-05 2024-04-20T17:42:48.661 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.661 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:48.661 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.661 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.661 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.661 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:72010 7.5 7:ae102a2d:::benchmark_data_smithi171_161871_object72009:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e705)", 2024-04-20T17:42:48.661 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:34.873983+0000", 2024-04-20T17:42:48.662 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 253.732603767, 2024-04-20T17:42:48.662 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.059086393000000001, 2024-04-20T17:42:48.662 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:48.662 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:48.662 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:48.662 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:48.662 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:48.662 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:48.662 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 72010 2024-04-20T17:42:48.662 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.662 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:48.662 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.662 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:48.663 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.873983+0000", 2024-04-20T17:42:48.663 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.663 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.663 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.663 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:48.663 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.873983+0000", 2024-04-20T17:42:48.663 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.663 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.663 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.663 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:48.663 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.873985+0000", 2024-04-20T17:42:48.663 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5409999999999999e-06 2024-04-20T17:42:48.663 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.663 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.664 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:48.664 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874040+0000", 2024-04-20T17:42:48.664 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5649000000000002e-05 2024-04-20T17:42:48.664 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.664 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.664 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:48.664 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874050+0000", 2024-04-20T17:42:48.664 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3149999999999998e-06 2024-04-20T17:42:48.664 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.664 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.664 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:48.664 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874057+0000", 2024-04-20T17:42:48.664 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1019999999999999e-06 2024-04-20T17:42:48.664 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.664 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.665 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:48.665 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874114+0000", 2024-04-20T17:42:48.665 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6916999999999997e-05 2024-04-20T17:42:48.665 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.665 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.665 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:48.665 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874241+0000", 2024-04-20T17:42:48.665 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000127322 2024-04-20T17:42:48.665 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.665 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.665 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:42:48.665 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874375+0000", 2024-04-20T17:42:48.665 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000134055 2024-04-20T17:42:48.665 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.665 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.666 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:48.666 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.927913+0000", 2024-04-20T17:42:48.666 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.053538397000000001 2024-04-20T17:42:48.666 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.666 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.666 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:48.666 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.932975+0000", 2024-04-20T17:42:48.666 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0050614520000000001 2024-04-20T17:42:48.666 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.666 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.666 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:48.666 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.933024+0000", 2024-04-20T17:42:48.666 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9325e-05 2024-04-20T17:42:48.666 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.667 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.667 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:48.667 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.933069+0000", 2024-04-20T17:42:48.667 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5318000000000002e-05 2024-04-20T17:42:48.667 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.667 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:48.667 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.667 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.667 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.667 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:72011 7.b 7:dfd3059a:::benchmark_data_smithi171_161871_object72010:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e705)", 2024-04-20T17:42:48.667 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:34.874071+0000", 2024-04-20T17:42:48.667 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 253.732515721, 2024-04-20T17:42:48.667 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.058409599999999999, 2024-04-20T17:42:48.667 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:48.668 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:48.668 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:48.668 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:48.668 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:48.668 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:48.668 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 72011 2024-04-20T17:42:48.668 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.668 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:48.668 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.668 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:48.668 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874071+0000", 2024-04-20T17:42:48.668 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.668 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.668 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.668 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:48.669 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874071+0000", 2024-04-20T17:42:48.669 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.669 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.669 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.669 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:48.669 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874072+0000", 2024-04-20T17:42:48.669 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2800000000000005e-07 2024-04-20T17:42:48.669 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.669 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.669 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:48.669 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874117+0000", 2024-04-20T17:42:48.669 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5213000000000002e-05 2024-04-20T17:42:48.669 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.669 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.669 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:48.670 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874124+0000", 2024-04-20T17:42:48.670 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0720000000000002e-06 2024-04-20T17:42:48.670 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.670 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.670 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:48.670 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874129+0000", 2024-04-20T17:42:48.670 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7489999999999998e-06 2024-04-20T17:42:48.670 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.670 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.670 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:48.670 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874175+0000", 2024-04-20T17:42:48.670 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6023000000000003e-05 2024-04-20T17:42:48.670 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.670 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.670 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:48.671 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874290+0000", 2024-04-20T17:42:48.671 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000115072 2024-04-20T17:42:48.671 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.671 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.671 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:42:48.671 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874403+0000", 2024-04-20T17:42:48.671 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000113006 2024-04-20T17:42:48.671 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.671 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.671 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:48.671 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.929997+0000", 2024-04-20T17:42:48.671 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.055593548999999999 2024-04-20T17:42:48.671 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.671 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.671 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:48.672 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.932374+0000", 2024-04-20T17:42:48.672 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0023774590000000002 2024-04-20T17:42:48.672 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.672 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.672 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:48.672 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.932426+0000", 2024-04-20T17:42:48.672 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2280000000000001e-05 2024-04-20T17:42:48.672 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.672 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.672 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:48.672 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.932481+0000", 2024-04-20T17:42:48.672 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4249000000000002e-05 2024-04-20T17:42:48.672 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.672 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:48.673 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.673 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.673 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.673 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:72014 7.b 7:d5e7fe15:::benchmark_data_smithi171_161871_object72013:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e705)", 2024-04-20T17:42:48.673 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:34.874149+0000", 2024-04-20T17:42:48.673 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 253.73243798300001, 2024-04-20T17:42:48.673 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.058553926999999999, 2024-04-20T17:42:48.673 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:48.673 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:48.673 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:48.673 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:48.673 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:48.673 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:48.673 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 72014 2024-04-20T17:42:48.673 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.674 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:48.674 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.674 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:48.674 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874149+0000", 2024-04-20T17:42:48.674 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.674 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.674 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.674 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:48.674 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874149+0000", 2024-04-20T17:42:48.674 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.674 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.674 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.674 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:48.674 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874150+0000", 2024-04-20T17:42:48.675 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5199999999999995e-07 2024-04-20T17:42:48.675 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.675 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.675 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:48.675 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874188+0000", 2024-04-20T17:42:48.675 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8006999999999997e-05 2024-04-20T17:42:48.675 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.675 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.675 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:48.675 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874193+0000", 2024-04-20T17:42:48.675 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0119999999999996e-06 2024-04-20T17:42:48.675 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.675 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.675 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:48.675 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.874200+0000", 2024-04-20T17:42:48.676 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9369999999999998e-06 2024-04-20T17:42:48.676 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.676 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.676 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:48.676 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.925710+0000", 2024-04-20T17:42:48.676 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.051510565000000001 2024-04-20T17:42:48.676 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.676 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.676 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:48.676 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.927891+0000", 2024-04-20T17:42:48.676 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0021803920000000002 2024-04-20T17:42:48.676 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.676 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.676 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:42:48.677 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.927988+0000", 2024-04-20T17:42:48.677 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7677999999999996e-05 2024-04-20T17:42:48.677 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.677 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.677 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:48.677 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.932539+0000", 2024-04-20T17:42:48.677 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0045502750000000003 2024-04-20T17:42:48.677 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.677 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.677 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:48.677 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.932616+0000", 2024-04-20T17:42:48.677 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7342999999999995e-05 2024-04-20T17:42:48.677 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.677 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.678 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:48.678 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.932663+0000", 2024-04-20T17:42:48.678 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6989e-05 2024-04-20T17:42:48.678 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.678 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.678 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:48.678 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.932703+0000", 2024-04-20T17:42:48.678 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9777000000000002e-05 2024-04-20T17:42:48.678 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.678 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:48.678 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.678 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.678 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.678 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop_reply(client.27088.0:72011 7.b e705/694 ondisk, result = 0)", 2024-04-20T17:42:48.679 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:34.876315+0000", 2024-04-20T17:42:48.679 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 253.73027216, 2024-04-20T17:42:48.679 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.053690789000000003, 2024-04-20T17:42:48.679 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:48.679 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:48.679 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:42:48.679 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:48.679 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.679 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:48.679 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.876315+0000", 2024-04-20T17:42:48.679 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.679 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.679 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.679 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:48.679 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.876315+0000", 2024-04-20T17:42:48.680 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.680 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.680 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.680 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:48.680 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.876316+0000", 2024-04-20T17:42:48.680 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4419999999999999e-06 2024-04-20T17:42:48.680 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.680 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.680 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:48.680 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.876321+0000", 2024-04-20T17:42:48.680 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6460000000000001e-06 2024-04-20T17:42:48.680 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.680 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.680 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.876332+0000", 2024-04-20T17:42:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0766e-05 2024-04-20T17:42:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.876339+0000", 2024-04-20T17:42:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6539999999999999e-06 2024-04-20T17:42:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.929984+0000", 2024-04-20T17:42:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.053644758000000001 2024-04-20T17:42:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.682 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:48.682 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.929991+0000", 2024-04-20T17:42:48.682 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2400000000000001e-06 2024-04-20T17:42:48.682 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.682 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.682 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:48.682 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.930005+0000", 2024-04-20T17:42:48.682 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4283e-05 2024-04-20T17:42:48.682 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.682 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:48.682 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.682 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.682 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.682 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop_reply(client.27088.0:72010 7.5 e705/694 ondisk, result = 0)", 2024-04-20T17:42:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:34.876339+0000", 2024-04-20T17:42:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 253.73024751599999, 2024-04-20T17:42:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.051582654999999998, 2024-04-20T17:42:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:42:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.876339+0000", 2024-04-20T17:42:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.876339+0000", 2024-04-20T17:42:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.876340+0000", 2024-04-20T17:42:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.155e-06 2024-04-20T17:42:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.876345+0000", 2024-04-20T17:42:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5410000000000002e-06 2024-04-20T17:42:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.876352+0000", 2024-04-20T17:42:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1269999999999997e-06 2024-04-20T17:42:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.876358+0000", 2024-04-20T17:42:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3709999999999999e-06 2024-04-20T17:42:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.927900+0000", 2024-04-20T17:42:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.051542566999999997 2024-04-20T17:42:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:48.686 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.927908+0000", 2024-04-20T17:42:48.686 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6029999999999999e-06 2024-04-20T17:42:48.686 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.686 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.686 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:48.686 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.927922+0000", 2024-04-20T17:42:48.686 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4290999999999999e-05 2024-04-20T17:42:48.686 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.686 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:48.686 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.686 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.686 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.686 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:72021 7.5 7:a792af76:::benchmark_data_smithi171_161871_object72020:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e705)", 2024-04-20T17:42:48.686 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:34.877239+0000", 2024-04-20T17:42:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 253.72934796499999, 2024-04-20T17:42:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.060395327999999998, 2024-04-20T17:42:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 72021 2024-04-20T17:42:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.877239+0000", 2024-04-20T17:42:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.877239+0000", 2024-04-20T17:42:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.877241+0000", 2024-04-20T17:42:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2280000000000001e-06 2024-04-20T17:42:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.877281+0000", 2024-04-20T17:42:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9765000000000003e-05 2024-04-20T17:42:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.926595+0000", 2024-04-20T17:42:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.049313726000000002 2024-04-20T17:42:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.926611+0000", 2024-04-20T17:42:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6154000000000002e-05 2024-04-20T17:42:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:48.690 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.927947+0000", 2024-04-20T17:42:48.690 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001335936 2024-04-20T17:42:48.690 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.690 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.690 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:48.690 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.928080+0000", 2024-04-20T17:42:48.690 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000133268 2024-04-20T17:42:48.690 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.690 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.690 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:42:48.690 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.928221+0000", 2024-04-20T17:42:48.690 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000140798 2024-04-20T17:42:48.690 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.690 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.691 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:48.691 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.933143+0000", 2024-04-20T17:42:48.691 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0049227079999999996 2024-04-20T17:42:48.691 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.691 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.691 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:48.691 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.937539+0000", 2024-04-20T17:42:48.691 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0043956869999999997 2024-04-20T17:42:48.691 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.691 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.691 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:48.691 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.937586+0000", 2024-04-20T17:42:48.691 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6527999999999999e-05 2024-04-20T17:42:48.691 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:34.937634+0000", 2024-04-20T17:42:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8529999999999998e-05 2024-04-20T17:42:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.27088.0:87138 7.c e709/694 7:3ac78cb4:::benchmark_data_smithi171_161871_object87137:head v 709'5492, mlcod=709'5489)", 2024-04-20T17:42:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:43.842984+0000", 2024-04-20T17:42:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 244.763602392, 2024-04-20T17:42:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.156380886, 2024-04-20T17:42:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.842984+0000", 2024-04-20T17:42:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.842984+0000", 2024-04-20T17:42:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:48.694 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.842986+0000", 2024-04-20T17:42:48.694 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7129999999999999e-06 2024-04-20T17:42:48.694 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.694 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.694 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:48.694 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.843020+0000", 2024-04-20T17:42:48.694 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4038999999999997e-05 2024-04-20T17:42:48.694 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.694 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.694 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:48.694 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.843029+0000", 2024-04-20T17:42:48.694 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0950000000000002e-06 2024-04-20T17:42:48.694 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.694 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.695 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:48.695 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.843036+0000", 2024-04-20T17:42:48.695 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0539999999999997e-06 2024-04-20T17:42:48.695 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.695 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.695 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:48.695 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.843087+0000", 2024-04-20T17:42:48.695 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0528999999999997e-05 2024-04-20T17:42:48.695 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.695 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.695 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:48.695 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.843105+0000", 2024-04-20T17:42:48.695 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8394999999999999e-05 2024-04-20T17:42:48.695 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.696 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.696 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:48.696 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.845666+0000", 2024-04-20T17:42:48.696 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0025611219999999999 2024-04-20T17:42:48.696 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.696 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.696 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:48.696 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.999365+0000", 2024-04-20T17:42:48.696 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15369893900000001 2024-04-20T17:42:48.696 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.696 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:48.696 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.696 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.696 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.697 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.27088.0:87132 7.9 e709/694 7:9eb80540:::benchmark_data_smithi171_161871_object87131:head v 709'5502, mlcod=709'5498)", 2024-04-20T17:42:48.697 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:43.843723+0000", 2024-04-20T17:42:48.697 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 244.762864071, 2024-04-20T17:42:48.697 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15567329499999999, 2024-04-20T17:42:48.697 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:48.697 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:48.697 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:48.697 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:48.697 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.697 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:48.697 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.843723+0000", 2024-04-20T17:42:48.697 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.697 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.697 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.698 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:48.698 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.843723+0000", 2024-04-20T17:42:48.698 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.698 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.698 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.698 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:48.698 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.843724+0000", 2024-04-20T17:42:48.698 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4330000000000001e-06 2024-04-20T17:42:48.698 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.698 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.698 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:48.698 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.843778+0000", 2024-04-20T17:42:48.698 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3871e-05 2024-04-20T17:42:48.698 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.698 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.699 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:48.699 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.843788+0000", 2024-04-20T17:42:48.699 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.4809999999999995e-06 2024-04-20T17:42:48.699 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.699 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.699 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:48.699 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.843795+0000", 2024-04-20T17:42:48.699 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7360000000000001e-06 2024-04-20T17:42:48.699 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.699 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.699 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:48.699 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.843837+0000", 2024-04-20T17:42:48.699 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1511000000000003e-05 2024-04-20T17:42:48.700 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.700 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.700 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:48.700 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.843855+0000", 2024-04-20T17:42:48.700 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8011000000000001e-05 2024-04-20T17:42:48.700 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.700 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.700 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:48.700 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.999345+0000", 2024-04-20T17:42:48.700 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.155490355 2024-04-20T17:42:48.700 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.700 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.700 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:48.700 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.999396+0000", 2024-04-20T17:42:48.700 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0896999999999997e-05 2024-04-20T17:42:48.701 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.701 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:48.701 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.701 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.701 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.701 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:87149 7.5 7:a7f5d85a:::benchmark_data_smithi171_161871_object87148:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e709)", 2024-04-20T17:42:48.701 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:43.846493+0000", 2024-04-20T17:42:48.701 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 244.76009363599999, 2024-04-20T17:42:48.701 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16122668900000001, 2024-04-20T17:42:48.701 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:48.701 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:48.701 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:48.701 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:42:48.701 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:42:48.702 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:42:48.702 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 87149 2024-04-20T17:42:48.702 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.702 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:48.702 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.702 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:48.702 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.846493+0000", 2024-04-20T17:42:48.702 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.702 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.702 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.702 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:48.702 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.846493+0000", 2024-04-20T17:42:48.702 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.702 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.703 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.703 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:48.703 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.846496+0000", 2024-04-20T17:42:48.703 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3499999999999999e-06 2024-04-20T17:42:48.703 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.703 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.703 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:48.703 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.846518+0000", 2024-04-20T17:42:48.703 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2320999999999999e-05 2024-04-20T17:42:48.703 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.703 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.703 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.999332+0000", 2024-04-20T17:42:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15281418299999999 2024-04-20T17:42:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.999348+0000", 2024-04-20T17:42:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6354e-05 2024-04-20T17:42:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.999404+0000", 2024-04-20T17:42:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5164000000000002e-05 2024-04-20T17:42:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.999551+0000", 2024-04-20T17:42:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014728500000000001 2024-04-20T17:42:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:42:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.999688+0000", 2024-04-20T17:42:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013765000000000001 2024-04-20T17:42:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:42:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:44.007610+0000", 2024-04-20T17:42:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0079218029999999998 2024-04-20T17:42:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.706 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.706 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:42:48.706 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:44.007667+0000", 2024-04-20T17:42:48.706 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6849999999999999e-05 2024-04-20T17:42:48.706 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.706 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.706 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:48.706 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:44.007694+0000", 2024-04-20T17:42:48.706 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7243e-05 2024-04-20T17:42:48.706 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.706 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.706 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:48.706 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:44.007720+0000", 2024-04-20T17:42:48.706 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5486e-05 2024-04-20T17:42:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.27088.0:87141 7.c e709/694 7:38ce718a:::benchmark_data_smithi171_161871_object87140:head v 709'5493, mlcod=709'5490)", 2024-04-20T17:42:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:43.847640+0000", 2024-04-20T17:42:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 244.75894705600001, 2024-04-20T17:42:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16894878399999999, 2024-04-20T17:42:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:42:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:42:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:42:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:42:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:42:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.847640+0000", 2024-04-20T17:42:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:42:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.847640+0000", 2024-04-20T17:42:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:42:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:42:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.847642+0000", 2024-04-20T17:42:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8899999999999999e-06 2024-04-20T17:42:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:42:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.847697+0000", 2024-04-20T17:42:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5318999999999997e-05 2024-04-20T17:42:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:42:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.999339+0000", 2024-04-20T17:42:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15164244499999999 2024-04-20T17:42:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:42:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.999350+0000", 2024-04-20T17:42:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0628999999999999e-05 2024-04-20T17:42:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:42:48.710 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.999404+0000", 2024-04-20T17:42:48.710 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3894000000000003e-05 2024-04-20T17:42:48.710 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.710 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.710 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:42:48.710 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:43.999425+0000", 2024-04-20T17:42:48.710 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0894999999999999e-05 2024-04-20T17:42:48.710 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.710 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.710 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:42:48.710 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:44.016535+0000", 2024-04-20T17:42:48.710 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.017110112 2024-04-20T17:42:48.710 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:42:48.711 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:42:48.711 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:42:48.711 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:44.016589+0000", 2024-04-20T17:42:48.711 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3600000000000002e-05 2024-04-20T17:42:48.711 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.711 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:48.711 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.711 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:42:48.711 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:42:48.711 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:42:48.711 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:42:48.738 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:48.758 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:42:48.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:48.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:42:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:48 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:42:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:48 smithi171 ceph-mon[44189]: from='client.27373 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:42:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:48 smithi171 ceph-mon[44189]: osdmap e765: 12 total, 11 up, 9 in 2024-04-20T17:42:48.862 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:42:48.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:48 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:42:48.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:48 smithi005 ceph-mon[39562]: from='client.27373 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:42:48.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:48 smithi005 ceph-mon[39562]: osdmap e765: 12 total, 11 up, 9 in 2024-04-20T17:42:48.936 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:48 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:42:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:48 smithi106 ceph-mon[44423]: from='client.27373 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:42:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:48 smithi106 ceph-mon[44423]: osdmap e765: 12 total, 11 up, 9 in 2024-04-20T17:42:49.087 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:42:49.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:49.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:42:49.195 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:42:49.234 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:49.594 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:42:49.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:49.597 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:42:49.633 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-20T17:42:49.675 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:49 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:42:49.676 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:49 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:42:49.676 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:49 smithi005 ceph-mon[39562]: pgmap v2056: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 610 MiB data, 5.8 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 980 op/s; 557/19506 objects degraded (2.856%); 4.7 MiB/s, 75 objects/s recovering 2024-04-20T17:42:49.676 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:49 smithi005 ceph-mon[39562]: osdmap e766: 12 total, 11 up, 9 in 2024-04-20T17:42:49.698 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:42:49.726 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:49.747 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:42:49.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:49.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:42:49.851 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:42:49.880 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:49 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:42:49.880 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:49 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:42:49.880 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:49 smithi171 ceph-mon[44189]: pgmap v2056: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 610 MiB data, 5.8 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 980 op/s; 557/19506 objects degraded (2.856%); 4.7 MiB/s, 75 objects/s recovering 2024-04-20T17:42:49.881 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:49 smithi171 ceph-mon[44189]: osdmap e766: 12 total, 11 up, 9 in 2024-04-20T17:42:49.885 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:49.911 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:42:49.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:49.915 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:42:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:49 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:42:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:49 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:42:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:49 smithi106 ceph-mon[44423]: pgmap v2056: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 610 MiB data, 5.8 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 980 op/s; 557/19506 objects degraded (2.856%); 4.7 MiB/s, 75 objects/s recovering 2024-04-20T17:42:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:49 smithi106 ceph-mon[44423]: osdmap e766: 12 total, 11 up, 9 in 2024-04-20T17:42:50.016 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:42:50.045 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:50.071 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:42:50.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:50.075 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:42:50.176 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:42:50.206 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:50.229 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:42:50.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:50.232 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:42:50.324 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:42:50.332 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:42:50.423 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:50.446 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:42:50.500 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:42:50.544 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:42:50.583 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:42:50.604 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:42:50.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:50.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:42:50.626 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:42:50.654 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:42:50.680 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:42:50.691 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:50 smithi005 ceph-mon[39562]: 7.b scrub starts 2024-04-20T17:42:50.691 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:50 smithi005 ceph-mon[39562]: 7.b scrub ok 2024-04-20T17:42:50.692 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:50 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4064760748' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:42:50.712 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:42:50.726 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:42:50.770 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:42:50.803 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:42:50.815 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:50.845 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:42:50.898 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:42:50.898 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:42:50.898 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:42:50.899 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:42:50.899 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:42:50.899 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:42:50.899 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:42:50.899 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:42:50.899 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:42:50.899 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:42:50.899 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:42:50.899 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:42:50.899 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:42:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:50 smithi106 ceph-mon[44423]: 7.b scrub starts 2024-04-20T17:42:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:50 smithi106 ceph-mon[44423]: 7.b scrub ok 2024-04-20T17:42:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:50 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4064760748' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:42:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:50 smithi171 ceph-mon[44189]: 7.b scrub starts 2024-04-20T17:42:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:50 smithi171 ceph-mon[44189]: 7.b scrub ok 2024-04-20T17:42:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:50 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4064760748' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:42:51.716 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:42:51.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:51.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:51.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:51.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:51 smithi005 ceph-mon[39562]: pgmap v2058: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 497 MiB data, 5.5 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 833 op/s; 557/15882 objects degraded (3.507%); 2.4 MiB/s, 38 objects/s recovering 2024-04-20T17:42:51.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:51.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:51.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:51.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:51.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:51.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:51.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:51.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:51.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:51 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 557/15882 objects degraded (3.507%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:51.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:51 smithi005 ceph-mon[39562]: osdmap e767: 12 total, 11 up, 9 in 2024-04-20T17:42:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:51 smithi106 ceph-mon[44423]: pgmap v2058: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 497 MiB data, 5.5 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 833 op/s; 557/15882 objects degraded (3.507%); 2.4 MiB/s, 38 objects/s recovering 2024-04-20T17:42:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:52.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:52.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:52.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:52.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:52.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:52.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:51 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 557/15882 objects degraded (3.507%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:52.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:51 smithi106 ceph-mon[44423]: osdmap e767: 12 total, 11 up, 9 in 2024-04-20T17:42:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:51 smithi171 ceph-mon[44189]: pgmap v2058: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 497 MiB data, 5.5 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 833 op/s; 557/15882 objects degraded (3.507%); 2.4 MiB/s, 38 objects/s recovering 2024-04-20T17:42:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:52.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:52.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:52.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:52.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:52.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:52.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:52.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:42:52.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:51 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 557/15882 objects degraded (3.507%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:52.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:51 smithi171 ceph-mon[44189]: osdmap e767: 12 total, 11 up, 9 in 2024-04-20T17:42:52.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:52 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:42:52.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:52 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:42:52.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:52 smithi005 ceph-mon[39562]: 7.0 scrub starts 2024-04-20T17:42:52.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:52 smithi005 ceph-mon[39562]: 7.0 scrub ok 2024-04-20T17:42:53.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:52 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:42:53.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:52 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:42:53.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:52 smithi106 ceph-mon[44423]: 7.0 scrub starts 2024-04-20T17:42:53.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:52 smithi106 ceph-mon[44423]: 7.0 scrub ok 2024-04-20T17:42:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:52 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:42:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:52 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:42:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:52 smithi171 ceph-mon[44189]: 7.0 scrub starts 2024-04-20T17:42:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:52 smithi171 ceph-mon[44189]: 7.0 scrub ok 2024-04-20T17:42:53.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:53 smithi005 ceph-mon[39562]: pgmap v2060: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 315 MiB data, 5.2 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 786 op/s; 250/10056 objects degraded (2.486%); 3.2 MiB/s, 51 objects/s recovering 2024-04-20T17:42:54.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:53 smithi106 ceph-mon[44423]: pgmap v2060: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 315 MiB data, 5.2 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 786 op/s; 250/10056 objects degraded (2.486%); 3.2 MiB/s, 51 objects/s recovering 2024-04-20T17:42:54.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:53 smithi171 ceph-mon[44189]: pgmap v2060: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 315 MiB data, 5.2 GiB used, 799 GiB / 805 GiB avail; 0 B/s wr, 786 op/s; 250/10056 objects degraded (2.486%); 3.2 MiB/s, 51 objects/s recovering 2024-04-20T17:42:54.634 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:42:54.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:54.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:42:54.750 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:42:54.778 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:54.802 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:42:54.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:54.810 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:42:54.910 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:42:54.950 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:54.980 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:42:54.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:54.983 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:42:55.084 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:42:55.126 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:55.438 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Removed 145723 objects 2024-04-20T17:42:55.439 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Clean up completed and total clean up time :165.809 2024-04-20T17:42:55.564 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:42:55.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:55.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:42:55.668 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:42:55.694 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:55.713 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:42:55.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:55.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:42:55.817 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:42:55.833 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:55 smithi005 ceph-mon[39562]: pgmap v2061: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 196 MiB data, 5.0 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 953 op/s; 250/6272 objects degraded (3.986%); 2.8 MiB/s, 44 objects/s recovering 2024-04-20T17:42:55.846 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:55 smithi106 ceph-mon[44423]: pgmap v2061: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 196 MiB data, 5.0 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 953 op/s; 250/6272 objects degraded (3.986%); 2.8 MiB/s, 44 objects/s recovering 2024-04-20T17:42:56.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:55 smithi171 ceph-mon[44189]: pgmap v2061: 17 pgs: 1 active+recovering+degraded, 16 active+clean; 196 MiB data, 5.0 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 953 op/s; 250/6272 objects degraded (3.986%); 2.8 MiB/s, 44 objects/s recovering 2024-04-20T17:42:56.496 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:42:56.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:56.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:42:56.525 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:42:56.525 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:42:56.526 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:42:56.595 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:42:56.595 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:42:56.603 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:42:56.633 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:56.784 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:42:56.786 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:42:56.809 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:56 smithi005 ceph-mon[39562]: Health check update: Degraded data redundancy: 250/6272 objects degraded (3.986%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:56.810 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:56 smithi005 ceph-mon[39562]: osdmap e768: 12 total, 11 up, 9 in 2024-04-20T17:42:56.830 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:42:56.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:56.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:42:56.935 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:42:56.967 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:56.987 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:42:56.989 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:56 smithi171 ceph-mon[44189]: Health check update: Degraded data redundancy: 250/6272 objects degraded (3.986%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:56.989 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:56 smithi171 ceph-mon[44189]: osdmap e768: 12 total, 11 up, 9 in 2024-04-20T17:42:56.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:56.991 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:42:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:56 smithi106 ceph-mon[44423]: Health check update: Degraded data redundancy: 250/6272 objects degraded (3.986%), 1 pg degraded (PG_DEGRADED) 2024-04-20T17:42:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:56 smithi106 ceph-mon[44423]: osdmap e768: 12 total, 11 up, 9 in 2024-04-20T17:42:57.091 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:42:57.121 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:57.570 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:42:57.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:57.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:42:57.676 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:42:57.709 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:57.730 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:42:57.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:57.733 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:42:57.834 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:42:57.863 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:57 smithi005 ceph-mon[39562]: pgmap v2062: 17 pgs: 17 active+clean; 30 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 1.16k op/s; 4.3 MiB/s, 69 objects/s recovering 2024-04-20T17:42:57.863 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:57 smithi005 ceph-mon[39562]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 250/6272 objects degraded (3.986%), 1 pg degraded) 2024-04-20T17:42:57.863 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:57 smithi005 ceph-mon[39562]: 7.9 scrub starts 2024-04-20T17:42:57.863 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:57 smithi005 ceph-mon[39562]: 7.9 scrub ok 2024-04-20T17:42:57.863 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:57 smithi005 ceph-mon[39562]: 7.3 scrub starts 2024-04-20T17:42:57.864 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:57 smithi005 ceph-mon[39562]: 7.3 scrub ok 2024-04-20T17:42:57.866 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:58.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:57 smithi106 ceph-mon[44423]: pgmap v2062: 17 pgs: 17 active+clean; 30 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 1.16k op/s; 4.3 MiB/s, 69 objects/s recovering 2024-04-20T17:42:58.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:57 smithi106 ceph-mon[44423]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 250/6272 objects degraded (3.986%), 1 pg degraded) 2024-04-20T17:42:58.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:57 smithi106 ceph-mon[44423]: 7.9 scrub starts 2024-04-20T17:42:58.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:57 smithi106 ceph-mon[44423]: 7.9 scrub ok 2024-04-20T17:42:58.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:57 smithi106 ceph-mon[44423]: 7.3 scrub starts 2024-04-20T17:42:58.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:57 smithi106 ceph-mon[44423]: 7.3 scrub ok 2024-04-20T17:42:58.015 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:42:58.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:58.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:42:58.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:57 smithi171 ceph-mon[44189]: pgmap v2062: 17 pgs: 17 active+clean; 30 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 1.16k op/s; 4.3 MiB/s, 69 objects/s recovering 2024-04-20T17:42:58.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:57 smithi171 ceph-mon[44189]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 250/6272 objects degraded (3.986%), 1 pg degraded) 2024-04-20T17:42:58.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:57 smithi171 ceph-mon[44189]: 7.9 scrub starts 2024-04-20T17:42:58.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:57 smithi171 ceph-mon[44189]: 7.9 scrub ok 2024-04-20T17:42:58.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:57 smithi171 ceph-mon[44189]: 7.3 scrub starts 2024-04-20T17:42:58.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:57 smithi171 ceph-mon[44189]: 7.3 scrub ok 2024-04-20T17:42:58.119 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:42:58.151 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:58.172 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:42:58.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:58.176 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:42:58.277 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:42:58.307 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:58.328 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:42:58.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:58.331 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:42:58.432 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:42:58.434 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-20T17:42:58.504 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:58.526 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:42:59.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:58 smithi106 ceph-mon[44423]: from='client.17331 v1:172.21.15.5:0/1603161115' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:42:59.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:58 smithi106 ceph-mon[44423]: 7.2 scrub starts 2024-04-20T17:42:59.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:58 smithi106 ceph-mon[44423]: 7.2 scrub ok 2024-04-20T17:42:59.059 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:42:59.063 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:58 smithi005 ceph-mon[39562]: from='client.17331 v1:172.21.15.5:0/1603161115' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:42:59.063 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:58 smithi005 ceph-mon[39562]: 7.2 scrub starts 2024-04-20T17:42:59.063 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:58 smithi005 ceph-mon[39562]: 7.2 scrub ok 2024-04-20T17:42:59.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:59.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:42:59.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:58 smithi171 ceph-mon[44189]: from='client.17331 v1:172.21.15.5:0/1603161115' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:42:59.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:58 smithi171 ceph-mon[44189]: 7.2 scrub starts 2024-04-20T17:42:59.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:58 smithi171 ceph-mon[44189]: 7.2 scrub ok 2024-04-20T17:42:59.166 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:42:59.195 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:59.393 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:42:59.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:59.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:42:59.414 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":2063,"stamp":"2024-04-20T17:42:56.586313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":31916576,"num_objects":480,"num_object_clones":0,"num_object_copies":960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":480,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":291151,"num_write_kb":9329211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18067,"num_bytes_recovered":1186028192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160735,"ondisk_log_size":160735,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":4875040,"kb_used_data":108436,"kb_used_omap":217,"kb_used_meta":4766310,"kb_avail":838831328,"statfs":{"total":863955320832,"available":858963279872,"internally_reserved":0,"allocated":111038464,"data_stored":107163647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":222986,"internal_metadata":4880701686},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3],"upper_bound":4},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-607649816,"num_objects":-9273,"num_object_clones":0,"num_object_copies":-18546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-557,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9273,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":557,"num_bytes_recovered":36503552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.918587"},"pg_stats":[{"pgid":"7.f","version":"767'18350","reported_seq":18606,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-20T17:42:55.439374+0000","last_change":"2024-04-20T17:39:40.325470+0000","last_active":"2024-04-20T17:42:55.439374+0000","last_peered":"2024-04-20T17:42:55.439374+0000","last_clean":"2024-04-20T17:42:55.439374+0000","last_became_active":"2024-04-20T17:37:56.835750+0000","last_became_peered":"2024-04-20T17:37:56.835750+0000","last_unstale":"2024-04-20T17:42:55.439374+0000","last_undegraded":"2024-04-20T17:42:55.439374+0000","last_fullsized":"2024-04-20T17:42:55.439374+0000","mapping_epoch":694,"log_start":"719'8300","ondisk_log_start":"719'8300","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9175","last_scrub_stamp":"2024-04-20T17:39:40.325340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:40.325340+0000","objects_scrubbed":9175,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":741,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18350,"num_write_kb":587200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8],"acting":[0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"767'17920","reported_seq":18226,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-20T17:42:55.424484+0000","last_change":"2024-04-20T17:41:02.433089+0000","last_active":"2024-04-20T17:42:55.424484+0000","last_peered":"2024-04-20T17:42:55.424484+0000","last_clean":"2024-04-20T17:42:55.424484+0000","last_became_active":"2024-04-20T17:37:56.833703+0000","last_became_peered":"2024-04-20T17:37:56.833703+0000","last_unstale":"2024-04-20T17:42:55.424484+0000","last_undegraded":"2024-04-20T17:42:55.424484+0000","last_fullsized":"2024-04-20T17:42:55.424484+0000","mapping_epoch":694,"log_start":"718'7900","ondisk_log_start":"718'7900","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"742'11794","last_scrub_stamp":"2024-04-20T17:41:02.432782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:41:02.432782+0000","objects_scrubbed":6149,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1488,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17920,"num_write_kb":573440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10],"acting":[2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"767'20642","reported_seq":21002,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-20T17:42:55.433667+0000","last_change":"2024-04-20T17:39:48.311967+0000","last_active":"2024-04-20T17:42:55.433667+0000","last_peered":"2024-04-20T17:42:55.433667+0000","last_clean":"2024-04-20T17:42:55.433667+0000","last_became_active":"2024-04-20T17:37:56.836496+0000","last_became_peered":"2024-04-20T17:37:56.836496+0000","last_unstale":"2024-04-20T17:42:55.433667+0000","last_undegraded":"2024-04-20T17:42:55.433667+0000","last_fullsized":"2024-04-20T17:42:55.433667+0000","mapping_epoch":694,"log_start":"721'10600","ondisk_log_start":"721'10600","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'11204","last_scrub_stamp":"2024-04-20T17:39:48.311825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:48.311825+0000","objects_scrubbed":9436,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18874,"num_write_kb":603905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.c","version":"767'18061","reported_seq":18316,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-20T17:42:54.402624+0000","last_change":"2024-04-20T17:39:44.683378+0000","last_active":"2024-04-20T17:42:54.402624+0000","last_peered":"2024-04-20T17:42:54.402624+0000","last_clean":"2024-04-20T17:42:54.402624+0000","last_became_active":"2024-04-20T17:37:56.834181+0000","last_became_peered":"2024-04-20T17:37:56.834181+0000","last_unstale":"2024-04-20T17:42:54.402624+0000","last_undegraded":"2024-04-20T17:42:54.402624+0000","last_fullsized":"2024-04-20T17:42:54.402624+0000","mapping_epoch":694,"log_start":"718'8000","ondisk_log_start":"718'8000","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9063","last_scrub_stamp":"2024-04-20T17:39:44.683201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:44.683201+0000","objects_scrubbed":9063,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":746,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4259840,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18061,"num_write_kb":580032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"767'18321","reported_seq":18664,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-20T17:42:54.128981+0000","last_change":"2024-04-20T17:42:49.392928+0000","last_active":"2024-04-20T17:42:54.128981+0000","last_peered":"2024-04-20T17:42:54.128981+0000","last_clean":"2024-04-20T17:42:54.128981+0000","last_became_active":"2024-04-20T17:37:56.830487+0000","last_became_peered":"2024-04-20T17:37:56.830487+0000","last_unstale":"2024-04-20T17:42:54.128981+0000","last_undegraded":"2024-04-20T17:42:54.128981+0000","last_fullsized":"2024-04-20T17:42:54.128981+0000","mapping_epoch":694,"log_start":"720'8300","ondisk_log_start":"720'8300","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"766'18016","last_scrub_stamp":"2024-04-20T17:42:49.392721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:42:49.392721+0000","objects_scrubbed":397,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:44:05.982584+0000","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5701632,"num_objects":87,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18321,"num_write_kb":589056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.a","version":"767'17984","reported_seq":18333,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-20T17:42:55.388478+0000","last_change":"2024-04-20T17:42:22.174131+0000","last_active":"2024-04-20T17:42:55.388478+0000","last_peered":"2024-04-20T17:42:55.388478+0000","last_clean":"2024-04-20T17:42:55.388478+0000","last_became_active":"2024-04-20T17:37:56.835944+0000","last_became_peered":"2024-04-20T17:37:56.835944+0000","last_unstale":"2024-04-20T17:42:55.388478+0000","last_undegraded":"2024-04-20T17:42:55.388478+0000","last_fullsized":"2024-04-20T17:42:55.388478+0000","mapping_epoch":694,"log_start":"718'7900","ondisk_log_start":"718'7900","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"756'16067","last_scrub_stamp":"2024-04-20T17:42:22.173861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:42:22.173861+0000","objects_scrubbed":1940,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:43:33.214516+0000","scrub_duration":489,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17984,"num_write_kb":575488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"767'18130","reported_seq":18368,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-20T17:42:55.439133+0000","last_change":"2024-04-20T17:39:03.675435+0000","last_active":"2024-04-20T17:42:55.439133+0000","last_peered":"2024-04-20T17:42:55.439133+0000","last_clean":"2024-04-20T17:42:55.439133+0000","last_became_active":"2024-04-20T17:37:56.835700+0000","last_became_peered":"2024-04-20T17:37:56.835700+0000","last_unstale":"2024-04-20T17:42:55.439133+0000","last_undegraded":"2024-04-20T17:42:55.439133+0000","last_fullsized":"2024-04-20T17:42:55.439133+0000","mapping_epoch":694,"log_start":"719'8100","ondisk_log_start":"719'8100","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'7175","last_scrub_stamp":"2024-04-20T17:39:03.675211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:03.675211+0000","objects_scrubbed":7133,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18130,"num_write_kb":580160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11],"acting":[0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"767'21706","reported_seq":40362,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-20T17:42:55.428414+0000","last_change":"2024-04-20T17:42:02.804301+0000","last_active":"2024-04-20T17:42:55.428414+0000","last_peered":"2024-04-20T17:42:55.428414+0000","last_clean":"2024-04-20T17:42:55.428414+0000","last_became_active":"2024-04-20T17:40:09.546695+0000","last_became_peered":"2024-04-20T17:40:09.546695+0000","last_unstale":"2024-04-20T17:42:55.428414+0000","last_undegraded":"2024-04-20T17:42:55.428414+0000","last_fullsized":"2024-04-20T17:42:55.428414+0000","mapping_epoch":731,"log_start":"720'11701","ondisk_log_start":"720'11701","created":694,"last_epoch_clean":732,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'18634","last_scrub_stamp":"2024-04-20T17:42:02.804073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:42:02.804073+0000","objects_scrubbed":3082,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:43:21.646311+0000","scrub_duration":620,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18136,"num_write_kb":580352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9062,"num_bytes_recovered":593887232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"767'18214","reported_seq":18569,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-20T17:42:53.697364+0000","last_change":"2024-04-20T17:42:04.144551+0000","last_active":"2024-04-20T17:42:53.697364+0000","last_peered":"2024-04-20T17:42:53.697364+0000","last_clean":"2024-04-20T17:42:53.697364+0000","last_became_active":"2024-04-20T17:37:56.836665+0000","last_became_peered":"2024-04-20T17:37:56.836665+0000","last_unstale":"2024-04-20T17:42:53.697364+0000","last_undegraded":"2024-04-20T17:42:53.697364+0000","last_fullsized":"2024-04-20T17:42:53.697364+0000","mapping_epoch":694,"log_start":"719'8200","ondisk_log_start":"719'8200","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'15376","last_scrub_stamp":"2024-04-20T17:42:04.144333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:42:04.144333+0000","objects_scrubbed":2961,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:43:15.847365+0000","scrub_duration":538,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7471104,"num_objects":114,"num_object_clones":0,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18214,"num_write_kb":586496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.5","version":"767'18230","reported_seq":18582,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-20T17:42:54.123179+0000","last_change":"2024-04-20T17:42:03.424949+0000","last_active":"2024-04-20T17:42:54.123179+0000","last_peered":"2024-04-20T17:42:54.123179+0000","last_clean":"2024-04-20T17:42:54.123179+0000","last_became_active":"2024-04-20T17:37:56.833600+0000","last_became_peered":"2024-04-20T17:37:56.833600+0000","last_unstale":"2024-04-20T17:42:54.123179+0000","last_undegraded":"2024-04-20T17:42:54.123179+0000","last_fullsized":"2024-04-20T17:42:54.123179+0000","mapping_epoch":694,"log_start":"719'8200","ondisk_log_start":"719'8200","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'15267","last_scrub_stamp":"2024-04-20T17:42:03.424811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:42:03.424811+0000","objects_scrubbed":3060,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:43:32.116339+0000","scrub_duration":489,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5505024,"num_objects":84,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18230,"num_write_kb":586048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.4","version":"767'18062","reported_seq":18317,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-20T17:42:55.395880+0000","last_change":"2024-04-20T17:39:50.365478+0000","last_active":"2024-04-20T17:42:55.395880+0000","last_peered":"2024-04-20T17:42:55.395880+0000","last_clean":"2024-04-20T17:42:55.395880+0000","last_became_active":"2024-04-20T17:37:56.834727+0000","last_became_peered":"2024-04-20T17:37:56.834727+0000","last_unstale":"2024-04-20T17:42:55.395880+0000","last_undegraded":"2024-04-20T17:42:55.395880+0000","last_fullsized":"2024-04-20T17:42:55.395880+0000","mapping_epoch":694,"log_start":"719'8000","ondisk_log_start":"719'8000","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"722'9031","last_scrub_stamp":"2024-04-20T17:39:50.365339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:50.365339+0000","objects_scrubbed":9031,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":876,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18062,"num_write_kb":577984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"767'18005","reported_seq":18238,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-20T17:42:54.391351+0000","last_change":"2024-04-20T17:39:00.502710+0000","last_active":"2024-04-20T17:42:54.391351+0000","last_peered":"2024-04-20T17:42:54.391351+0000","last_clean":"2024-04-20T17:42:54.391351+0000","last_became_active":"2024-04-20T17:37:56.832911+0000","last_became_peered":"2024-04-20T17:37:56.832911+0000","last_unstale":"2024-04-20T17:42:54.391351+0000","last_undegraded":"2024-04-20T17:42:54.391351+0000","last_fullsized":"2024-04-20T17:42:54.391351+0000","mapping_epoch":694,"log_start":"719'8000","ondisk_log_start":"719'8000","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"713'6831","last_scrub_stamp":"2024-04-20T17:39:00.502556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:00.502556+0000","objects_scrubbed":6782,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4390912,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18005,"num_write_kb":578304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"767'18431","reported_seq":18671,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-20T17:42:54.404438+0000","last_change":"2024-04-20T17:39:08.489529+0000","last_active":"2024-04-20T17:42:54.404438+0000","last_peered":"2024-04-20T17:42:54.404438+0000","last_clean":"2024-04-20T17:42:54.404438+0000","last_became_active":"2024-04-20T17:37:56.834292+0000","last_became_peered":"2024-04-20T17:37:56.834292+0000","last_unstale":"2024-04-20T17:42:54.404438+0000","last_undegraded":"2024-04-20T17:42:54.404438+0000","last_fullsized":"2024-04-20T17:42:54.404438+0000","mapping_epoch":694,"log_start":"720'8400","ondisk_log_start":"720'8400","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"716'7577","last_scrub_stamp":"2024-04-20T17:39:08.489308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:08.489308+0000","objects_scrubbed":7524,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3997696,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18431,"num_write_kb":591744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10],"acting":[3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"767'20273","reported_seq":20693,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-20T17:42:55.409410+0000","last_change":"2024-04-20T17:42:25.987544+0000","last_active":"2024-04-20T17:42:55.409410+0000","last_peered":"2024-04-20T17:42:55.409410+0000","last_clean":"2024-04-20T17:42:55.409410+0000","last_became_active":"2024-04-20T17:37:56.835512+0000","last_became_peered":"2024-04-20T17:37:56.835512+0000","last_unstale":"2024-04-20T17:42:55.409410+0000","last_undegraded":"2024-04-20T17:42:55.409410+0000","last_fullsized":"2024-04-20T17:42:55.409410+0000","mapping_epoch":694,"log_start":"737'10200","ondisk_log_start":"737'10200","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"758'18500","last_scrub_stamp":"2024-04-20T17:42:25.987445+0000","last_deep_scrub":"758'18500","last_deep_scrub_stamp":"2024-04-20T17:42:25.987445+0000","last_clean_scrub_stamp":"2024-04-20T17:42:25.987445+0000","objects_scrubbed":2091,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":12,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:43:26.213160+0000","scrub_duration":11980,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18182,"num_write_kb":581824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"767'18064","reported_seq":18391,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-20T17:42:55.429868+0000","last_change":"2024-04-20T17:42:52.387729+0000","last_active":"2024-04-20T17:42:55.429868+0000","last_peered":"2024-04-20T17:42:55.429868+0000","last_clean":"2024-04-20T17:42:55.429868+0000","last_became_active":"2024-04-20T17:37:56.833972+0000","last_became_peered":"2024-04-20T17:37:56.833972+0000","last_unstale":"2024-04-20T17:42:55.429868+0000","last_undegraded":"2024-04-20T17:42:55.429868+0000","last_fullsized":"2024-04-20T17:42:55.429868+0000","mapping_epoch":694,"log_start":"718'8000","ondisk_log_start":"718'8000","created":694,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"767'17878","last_scrub_stamp":"2024-04-20T17:42:52.386974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:42:52.386974+0000","objects_scrubbed":186,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:43:53.155364+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18064,"num_write_kb":578048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.6","version":"767'18006","reported_seq":36240,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-20T17:42:55.423746+0000","last_change":"2024-04-20T17:42:55.279665+0000","last_active":"2024-04-20T17:42:55.423746+0000","last_peered":"2024-04-20T17:42:55.423746+0000","last_clean":"2024-04-20T17:42:55.423746+0000","last_became_active":"2024-04-20T17:39:54.933393+0000","last_became_peered":"2024-04-20T17:39:54.933393+0000","last_unstale":"2024-04-20T17:42:55.423746+0000","last_undegraded":"2024-04-20T17:42:55.423746+0000","last_fullsized":"2024-04-20T17:42:55.423746+0000","mapping_epoch":727,"log_start":"719'8000","ondisk_log_start":"719'8000","created":694,"last_epoch_clean":767,"parent":"0.0","parent_split_bits":0,"last_scrub":"714'7141","last_scrub_stamp":"2024-04-20T17:39:04.435913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:37:55.816485+0000","last_clean_scrub_stamp":"2024-04-20T17:39:04.435913+0000","objects_scrubbed":7101,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18006,"num_write_kb":576192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8995,"num_bytes_recovered":589496320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":554,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-20T17:42:51.597005+0000","last_change":"2024-04-20T17:42:20.165296+0000","last_active":"2024-04-20T17:42:51.597005+0000","last_peered":"2024-04-20T17:42:51.597005+0000","last_clean":"2024-04-20T17:42:51.597005+0000","last_became_active":"2024-04-20T17:39:54.970208+0000","last_became_peered":"2024-04-20T17:39:54.970208+0000","last_unstale":"2024-04-20T17:42:51.597005+0000","last_undegraded":"2024-04-20T17:42:51.597005+0000","last_fullsized":"2024-04-20T17:42:51.597005+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:42:20.165149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:42:20.165149+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:43:22.534703+0000","scrub_duration":180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":31326208,"num_objects":478,"num_object_clones":0,"num_object_copies":956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":478,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":290970,"num_write_kb":9326273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18057,"num_bytes_recovered":1183383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1235423232,"data_stored":1235419160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160598,"ondisk_log_size":160598,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991679,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":734608,"kb_used_data":35656,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":93010544,"statfs":{"total":95995035648,"available":95242797056,"internally_reserved":0,"allocated":36511744,"data_stored":36078883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":715717060},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253319,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":581896,"kb_used_data":13440,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":93163256,"statfs":{"total":95995035648,"available":95399174144,"internally_reserved":0,"allocated":13762560,"data_stored":13333251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":582089152},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613067,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":597264,"kb_used_data":2440,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93147888,"statfs":{"total":95995035648,"available":95383437312,"internally_reserved":0,"allocated":2498560,"data_stored":2065719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":609089984},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809297,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":334860,"kb_used_data":19072,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":93410292,"statfs":{"total":95995035648,"available":95652139008,"internally_reserved":0,"allocated":19529728,"data_stored":19100439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":323353029},"hb_peers":[0,1,2,3,4,5,6,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":2297807503527,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":2830383448150,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738872,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":663688,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93081464,"statfs":{"total":95995035648,"available":95315419136,"internally_reserved":0,"allocated":851968,"data_stored":422679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394162,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":329672,"kb_used_data":832,"kb_used_omap":202,"kb_used_meta":328629,"kb_avail":93415480,"statfs":{"total":95995035648,"available":95657451520,"internally_reserved":0,"allocated":851968,"data_stored":422679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":336516567},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653211,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":448140,"kb_used_data":32580,"kb_used_omap":4,"kb_used_meta":415547,"kb_avail":93297012,"statfs":{"total":95995035648,"available":95536140288,"internally_reserved":0,"allocated":33361920,"data_stored":32928559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":425520839},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986660,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":535112,"kb_used_data":2752,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":93210040,"statfs":{"total":95995035648,"available":95447080960,"internally_reserved":0,"allocated":2818048,"data_stored":2388759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":545126853},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790430,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":649800,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93095352,"statfs":{"total":95995035648,"available":95329640448,"internally_reserved":0,"allocated":851968,"data_stored":422679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":23859200,"data_stored":23855128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1167458304,"data_stored":1167458304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14155776,"data_stored":14155776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":7602176,"data_stored":7602176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":22347776,"data_stored":22347776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:42:59.416 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph status --format=json 2024-04-20T17:42:59.417 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:42:59.497 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:42:59.529 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:59.551 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:42:59.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:59.554 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:42:59.655 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:42:59.687 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:59.709 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:42:59.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:59.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:42:59.741 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:42:59.814 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:42:59.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:59 smithi171 ceph-mon[44189]: pgmap v2064: 17 pgs: 17 active+clean; 30 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 931 op/s; 4.3 MiB/s, 69 objects/s recovering 2024-04-20T17:42:59.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:42:59 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/104908690' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:42:59.844 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:42:59.864 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:42:59.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:42:59.868 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:42:59.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:59 smithi005 ceph-mon[39562]: pgmap v2064: 17 pgs: 17 active+clean; 30 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 931 op/s; 4.3 MiB/s, 69 objects/s recovering 2024-04-20T17:42:59.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:42:59 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/104908690' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:42:59.969 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:42:59.997 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:00.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:59 smithi106 ceph-mon[44423]: pgmap v2064: 17 pgs: 17 active+clean; 30 MiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 931 op/s; 4.3 MiB/s, 69 objects/s recovering 2024-04-20T17:43:00.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:42:59 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/104908690' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:43:00.900 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:00 smithi005 ceph-mon[39562]: 7.6 scrub starts 2024-04-20T17:43:00.900 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:00 smithi005 ceph-mon[39562]: 7.6 scrub ok 2024-04-20T17:43:00.900 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:00 smithi005 ceph-mon[39562]: 7.e scrub starts 2024-04-20T17:43:00.900 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:00 smithi005 ceph-mon[39562]: 7.e scrub ok 2024-04-20T17:43:00.900 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:00 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:43:00.900 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:00 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:43:00.900 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:00 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/104908690' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:43:00.900 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:00 smithi005 ceph-mon[39562]: osdmap e769: 12 total, 11 up, 9 in 2024-04-20T17:43:00.946 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:43:00.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:00.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:43:00.985 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:43:00.985 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:43:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:00 smithi106 ceph-mon[44423]: 7.6 scrub starts 2024-04-20T17:43:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:00 smithi106 ceph-mon[44423]: 7.6 scrub ok 2024-04-20T17:43:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:00 smithi106 ceph-mon[44423]: 7.e scrub starts 2024-04-20T17:43:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:00 smithi106 ceph-mon[44423]: 7.e scrub ok 2024-04-20T17:43:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:00 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:43:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:00 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:43:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:00 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/104908690' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:43:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:00 smithi106 ceph-mon[44423]: osdmap e769: 12 total, 11 up, 9 in 2024-04-20T17:43:01.052 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:43:01.081 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:00 smithi171 ceph-mon[44189]: 7.6 scrub starts 2024-04-20T17:43:01.082 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:00 smithi171 ceph-mon[44189]: 7.6 scrub ok 2024-04-20T17:43:01.082 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:00 smithi171 ceph-mon[44189]: 7.e scrub starts 2024-04-20T17:43:01.082 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:00 smithi171 ceph-mon[44189]: 7.e scrub ok 2024-04-20T17:43:01.082 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:00 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:43:01.082 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:00 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:43:01.082 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:00 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/104908690' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:43:01.082 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:00 smithi171 ceph-mon[44189]: osdmap e769: 12 total, 11 up, 9 in 2024-04-20T17:43:01.084 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:01.106 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:43:01.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:01.109 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:43:01.210 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:43:01.246 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:01.269 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:43:01.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:01.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:43:01.323 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_5 2024-04-20T17:43:01.323 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-04-20T17:43:01.373 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:43:01.404 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:01.425 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:43:01.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:01.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:43:01.528 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:43:01.563 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:01.687 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:43:01.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:01.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:43:01.792 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:43:01.820 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:01.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:01 smithi005 ceph-mon[39562]: 7.d scrub starts 2024-04-20T17:43:01.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:01 smithi005 ceph-mon[39562]: 7.d scrub ok 2024-04-20T17:43:01.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:01 smithi005 ceph-mon[39562]: pgmap v2066: 17 pgs: 17 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 627 op/s; 2.0 MiB/s, 31 objects/s recovering 2024-04-20T17:43:01.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:01 smithi005 ceph-mon[39562]: osdmap e770: 12 total, 11 up, 9 in 2024-04-20T17:43:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:01 smithi106 ceph-mon[44423]: 7.d scrub starts 2024-04-20T17:43:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:01 smithi106 ceph-mon[44423]: 7.d scrub ok 2024-04-20T17:43:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:01 smithi106 ceph-mon[44423]: pgmap v2066: 17 pgs: 17 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 627 op/s; 2.0 MiB/s, 31 objects/s recovering 2024-04-20T17:43:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:01 smithi106 ceph-mon[44423]: osdmap e770: 12 total, 11 up, 9 in 2024-04-20T17:43:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:01 smithi171 ceph-mon[44189]: 7.d scrub starts 2024-04-20T17:43:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:01 smithi171 ceph-mon[44189]: 7.d scrub ok 2024-04-20T17:43:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:01 smithi171 ceph-mon[44189]: pgmap v2066: 17 pgs: 17 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 627 op/s; 2.0 MiB/s, 31 objects/s recovering 2024-04-20T17:43:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:01 smithi171 ceph-mon[44189]: osdmap e770: 12 total, 11 up, 9 in 2024-04-20T17:43:02.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:02 smithi005 ceph-mon[39562]: 7.4 scrub starts 2024-04-20T17:43:02.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:02 smithi005 ceph-mon[39562]: 7.4 scrub ok 2024-04-20T17:43:02.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:02 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3232058673' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:43:02.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:02 smithi005 ceph-mon[39562]: pgmap v2068: 17 pgs: 17 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 79 op/s 2024-04-20T17:43:02.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:02 smithi106 ceph-mon[44423]: 7.4 scrub starts 2024-04-20T17:43:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:02 smithi106 ceph-mon[44423]: 7.4 scrub ok 2024-04-20T17:43:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:02 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3232058673' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:43:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:02 smithi106 ceph-mon[44423]: pgmap v2068: 17 pgs: 17 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 79 op/s 2024-04-20T17:43:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:02 smithi171 ceph-mon[44189]: 7.4 scrub starts 2024-04-20T17:43:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:02 smithi171 ceph-mon[44189]: 7.4 scrub ok 2024-04-20T17:43:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:02 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3232058673' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:43:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:02 smithi171 ceph-mon[44189]: pgmap v2068: 17 pgs: 17 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 79 op/s 2024-04-20T17:43:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:03.460 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:43:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:03 smithi106 ceph-mon[44423]: 7.c deep-scrub starts 2024-04-20T17:43:04.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:03 smithi106 ceph-mon[44423]: 7.c deep-scrub ok 2024-04-20T17:43:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:03 smithi171 ceph-mon[44189]: 7.c deep-scrub starts 2024-04-20T17:43:04.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:03 smithi171 ceph-mon[44189]: 7.c deep-scrub ok 2024-04-20T17:43:04.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:04.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:03 smithi005 ceph-mon[39562]: 7.c deep-scrub starts 2024-04-20T17:43:04.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:03 smithi005 ceph-mon[39562]: 7.c deep-scrub ok 2024-04-20T17:43:04.321 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:43:04.321 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:43:04.321 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:43:04.321 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:43:04.321 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:43:04.321 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:43:04.321 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:43:04.321 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:43:04.322 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:43:04.322 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:43:04.322 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:43:04.322 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:43:04.322 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:43:04.322 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:43:04.322 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:43:04.322 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:43:04.322 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:43:04.322 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:43:04.322 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:43:04.322 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:43:04.322 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:43:04.323 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:43:04.323 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:43:04.932 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:43:04.948 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:43:04.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:04.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:43:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:04 smithi106 ceph-mon[44423]: pgmap v2069: 17 pgs: 17 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 59 op/s 2024-04-20T17:43:05.054 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:43:05.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:04 smithi171 ceph-mon[44189]: pgmap v2069: 17 pgs: 17 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 59 op/s 2024-04-20T17:43:05.087 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:05.109 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:43:05.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:05.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:43:05.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:04 smithi005 ceph-mon[39562]: pgmap v2069: 17 pgs: 17 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail; 0 B/s wr, 59 op/s 2024-04-20T17:43:05.213 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:43:05.249 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:05.271 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:43:05.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:05.274 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:43:05.375 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:43:05.404 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:05.478 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:43:05.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:05.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:43:05.583 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:43:05.612 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:05.760 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:43:05.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:05.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:43:05.782 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:43:05.782 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:43:05.865 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:43:05.893 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:06.936 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:06 smithi005 ceph-mon[39562]: 7.f scrub starts 2024-04-20T17:43:06.936 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:06 smithi005 ceph-mon[39562]: 7.f scrub ok 2024-04-20T17:43:06.937 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:06 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:43:06.937 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:06 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:43:06.937 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:06 smithi005 ceph-mon[39562]: osdmap e771: 12 total, 11 up, 9 in 2024-04-20T17:43:06.937 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:06 smithi005 ceph-mon[39562]: pgmap v2071: 17 pgs: 17 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:06.937 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:06 smithi005 ceph-mon[39562]: osdmap e772: 12 total, 11 up, 9 in 2024-04-20T17:43:06.943 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:43:06.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:06.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:43:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:06 smithi106 ceph-mon[44423]: 7.f scrub starts 2024-04-20T17:43:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:06 smithi106 ceph-mon[44423]: 7.f scrub ok 2024-04-20T17:43:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:06 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:43:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:06 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:43:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:06 smithi106 ceph-mon[44423]: osdmap e771: 12 total, 11 up, 9 in 2024-04-20T17:43:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:06 smithi106 ceph-mon[44423]: pgmap v2071: 17 pgs: 17 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:06 smithi106 ceph-mon[44423]: osdmap e772: 12 total, 11 up, 9 in 2024-04-20T17:43:07.048 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:43:07.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:06 smithi171 ceph-mon[44189]: 7.f scrub starts 2024-04-20T17:43:07.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:06 smithi171 ceph-mon[44189]: 7.f scrub ok 2024-04-20T17:43:07.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:06 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:43:07.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:06 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:43:07.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:06 smithi171 ceph-mon[44189]: osdmap e771: 12 total, 11 up, 9 in 2024-04-20T17:43:07.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:06 smithi171 ceph-mon[44189]: pgmap v2071: 17 pgs: 17 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:07.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:06 smithi171 ceph-mon[44189]: osdmap e772: 12 total, 11 up, 9 in 2024-04-20T17:43:07.079 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:07.099 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:43:07.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:07.103 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:43:07.204 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:43:07.236 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:07.587 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:43:07.587 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:43:07.587 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:43:07.588 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:43:07.588 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:43:07.783 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:43:07.852 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:43:07.852 INFO:teuthology.orchestra.run.smithi005.stdout:{"fsid":"13c6e76c-ff37-11ee-bc93-c7b262605968","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":2556,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":772,"num_osds":12,"num_up_osds":11,"osd_up_since":1713634774,"num_in_osds":9,"osd_in_since":1713634793,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17}],"num_pgs":17,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":4902907904,"bytes_avail":859052412928,"bytes_total":863955320832},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.5:8443/"}},"servicemap":{"epoch":57,"modified":"2024-04-20T17:41:36.531241+0000","services":{}},"progress_events":{}} 2024-04-20T17:43:08.068 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:07 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4214460327' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:43:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:07 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4214460327' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:43:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:07 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4214460327' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:43:08.398 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-20T17:43:08.486 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:43:08.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:08.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:43:08.591 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:43:08.624 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:08.645 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:43:08.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:08.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:43:08.749 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:43:08.777 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:08.911 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'unique_pool_5' removed 2024-04-20T17:43:09.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:08 smithi005 ceph-mon[39562]: pgmap v2073: 17 pgs: 17 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:09.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:08 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2187223987' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-20T17:43:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:08 smithi106 ceph-mon[44423]: pgmap v2073: 17 pgs: 17 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:08 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2187223987' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-20T17:43:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:08 smithi171 ceph-mon[44189]: pgmap v2073: 17 pgs: 17 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:08 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2187223987' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-20T17:43:09.505 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:43:09.530 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:43:09.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:09.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:43:09.565 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-20T17:43:09.565 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:43:09.645 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:43:09.676 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:09.777 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:43:09.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:09.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:43:09.881 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:43:09.914 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:09.935 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:43:09.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:09.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:43:10.040 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:43:10.070 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:10.091 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:43:10.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:10.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:43:10.195 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:43:10.212 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:43:10.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:09 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2187223987' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-04-20T17:43:10.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:09 smithi005 ceph-mon[39562]: osdmap e773: 12 total, 11 up, 9 in 2024-04-20T17:43:10.230 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:10.256 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-20T17:43:10.256 INFO:tasks.radosbench:Beginning radosbench... 2024-04-20T17:43:10.256 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_6 2024-04-20T17:43:10.257 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pool create unique_pool_6 16 2024-04-20T17:43:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:09 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2187223987' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-04-20T17:43:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:09 smithi106 ceph-mon[44423]: osdmap e773: 12 total, 11 up, 9 in 2024-04-20T17:43:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:09 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2187223987' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-04-20T17:43:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:09 smithi171 ceph-mon[44189]: osdmap e773: 12 total, 11 up, 9 in 2024-04-20T17:43:11.007 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:10 smithi005 ceph-mon[39562]: pgmap v2075: 1 pgs: 1 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:11.103 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:43:11.107 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:43:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:10 smithi106 ceph-mon[44423]: pgmap v2075: 1 pgs: 1 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:10 smithi171 ceph-mon[44189]: pgmap v2075: 1 pgs: 1 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:11 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:43:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:11 smithi005 ceph-mon[39562]: osdmap e774: 12 total, 11 up, 9 in 2024-04-20T17:43:12.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:11 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:43:12.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:11 smithi106 ceph-mon[44423]: osdmap e774: 12 total, 11 up, 9 in 2024-04-20T17:43:12.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:11 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:43:12.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:11 smithi171 ceph-mon[44189]: osdmap e774: 12 total, 11 up, 9 in 2024-04-20T17:43:12.527 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:43:12.952 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:43:13.186 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:43:13.189 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:43:13.189 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:43:13.189 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:43:13.189 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:12 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3967540310' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:43:13.189 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:12 smithi005 ceph-mon[39562]: pgmap v2077: 1 pgs: 1 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:13.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:13.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:43:13.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:43:13.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:43:13.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:43:13.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:12 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3967540310' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:43:13.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:12 smithi106 ceph-mon[44423]: pgmap v2077: 1 pgs: 1 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:13.291 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:43:13.321 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:43:13.321 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:43:13.321 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:43:13.321 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:12 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3967540310' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:43:13.321 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:12 smithi171 ceph-mon[44189]: pgmap v2077: 1 pgs: 1 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:13.323 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:13.345 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:43:13.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:13.348 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:43:13.449 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:43:13.481 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:13.502 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:43:13.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:13.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:43:13.607 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:43:13.637 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:13.658 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:43:13.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:13.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:43:13.763 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:43:13.795 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:13.921 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:43:13.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:13.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:43:14.025 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:43:14.050 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:13 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:43:14.050 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:13 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3967540310' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:43:14.051 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:13 smithi005 ceph-mon[39562]: osdmap e775: 12 total, 11 up, 9 in 2024-04-20T17:43:14.053 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:14.251 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:43:14.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:14.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:43:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:13 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:43:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:13 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3967540310' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:43:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:13 smithi106 ceph-mon[44423]: osdmap e775: 12 total, 11 up, 9 in 2024-04-20T17:43:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:13 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:43:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:13 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3967540310' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:43:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:13 smithi171 ceph-mon[44189]: osdmap e775: 12 total, 11 up, 9 in 2024-04-20T17:43:14.356 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:43:14.387 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:14.406 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:43:14.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:14.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:43:14.511 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:43:14.543 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:14.565 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:43:14.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:14.569 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:43:14.670 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:43:14.703 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:14.724 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:43:14.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:14.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:43:14.828 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:43:14.857 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:14.877 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:43:14.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:14.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:43:14.981 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:43:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:14 smithi106 ceph-mon[44423]: pgmap v2079: 1 pgs: 1 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:15.011 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:15.149 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:14 smithi005 ceph-mon[39562]: pgmap v2079: 1 pgs: 1 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:15.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:14 smithi171 ceph-mon[44189]: pgmap v2079: 1 pgs: 1 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:15.693 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:43:15.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:15.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:43:15.798 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:43:15.829 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:16.029 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:43:16.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:16.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:43:16.134 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:43:16.167 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:15 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2403192359' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:43:16.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:15 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2403192359' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:43:16.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:15 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2403192359' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:43:16.650 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:43:16.650 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:43:16.650 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:43:16.653 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:43:16.653 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:43:16.653 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:43:16.653 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:43:16.653 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:43:16.653 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:43:16.653 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:43:16.653 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:43:16.654 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:43:16.654 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:43:16.654 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:43:16.654 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:43:16.654 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:43:16.654 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:43:16.654 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:43:16.654 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:43:16.654 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:43:16.654 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:43:16.654 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:43:16.654 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:43:17.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:16 smithi005 ceph-mon[39562]: pgmap v2080: 1 pgs: 1 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:17.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:16 smithi005 ceph-mon[39562]: osdmap e776: 12 total, 11 up, 9 in 2024-04-20T17:43:17.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:16 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/884683653' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-04-20T17:43:17.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:16 smithi106 ceph-mon[44423]: pgmap v2080: 1 pgs: 1 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:17.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:16 smithi106 ceph-mon[44423]: osdmap e776: 12 total, 11 up, 9 in 2024-04-20T17:43:17.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:16 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/884683653' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-04-20T17:43:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:17.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:16 smithi171 ceph-mon[44189]: pgmap v2080: 1 pgs: 1 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:17.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:16 smithi171 ceph-mon[44189]: osdmap e776: 12 total, 11 up, 9 in 2024-04-20T17:43:17.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:16 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/884683653' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-04-20T17:43:17.491 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:43:17.524 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:43:17.525 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:43:17.597 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'unique_pool_6' created 2024-04-20T17:43:18.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:17 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:18.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:17 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/884683653' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-04-20T17:43:18.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:17 smithi005 ceph-mon[39562]: osdmap e777: 12 total, 11 up, 9 in 2024-04-20T17:43:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:17 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:17 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/884683653' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-04-20T17:43:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:17 smithi106 ceph-mon[44423]: osdmap e777: 12 total, 11 up, 9 in 2024-04-20T17:43:18.318 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:43:18.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:18.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:43:18.333 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-20T17:43:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:17 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:17 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/884683653' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-04-20T17:43:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:17 smithi171 ceph-mon[44189]: osdmap e777: 12 total, 11 up, 9 in 2024-04-20T17:43:18.424 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:43:18.453 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:18.608 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:43:18.703 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:43:18.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:18.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:43:18.725 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:43:18.726 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:43:18.807 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:43:18.835 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:18.856 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:43:18.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:18.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:43:18.959 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:43:18.987 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:19.084 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:43:19.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:19.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:43:19.188 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:43:19.217 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:19.677 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:43:19.679 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:43:19.709 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:43:19.742 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_6 rados --yes-i-really-mean-it || true 2024-04-20T17:43:19.924 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:19 smithi106 ceph-mon[44423]: pgmap v2083: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:19.924 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:19 smithi106 ceph-mon[44423]: osdmap e778: 12 total, 11 up, 9 in 2024-04-20T17:43:19.924 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:19 smithi106 ceph-mon[44423]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-20T17:43:19.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:19 smithi005 ceph-mon[39562]: pgmap v2083: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:19.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:19 smithi005 ceph-mon[39562]: osdmap e778: 12 total, 11 up, 9 in 2024-04-20T17:43:19.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:19 smithi005 ceph-mon[39562]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-20T17:43:19.966 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:43:19.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:19.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:43:20.070 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:43:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:19 smithi171 ceph-mon[44189]: pgmap v2083: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:19 smithi171 ceph-mon[44189]: osdmap e778: 12 total, 11 up, 9 in 2024-04-20T17:43:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:19 smithi171 ceph-mon[44189]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-20T17:43:20.097 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:20.195 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:43:20.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:20.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:43:20.299 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:43:20.328 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:20.382 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:43:20.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:20.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:43:20.472 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:43:20.472 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:43:20.490 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:43:20.518 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:20.727 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:43:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:20 smithi106 ceph-mon[44423]: osdmap e779: 12 total, 11 up, 9 in 2024-04-20T17:43:20.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:20 smithi106 ceph-mon[44423]: from='client.17376 v1:172.21.15.5:0/1098176962' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:43:20.791 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:20 smithi005 ceph-mon[39562]: osdmap e779: 12 total, 11 up, 9 in 2024-04-20T17:43:20.791 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:20 smithi005 ceph-mon[39562]: from='client.17376 v1:172.21.15.5:0/1098176962' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:43:20.817 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":2085,"stamp":"2024-04-20T17:43:19.596362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":20,"acting":20,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":4,"kb":843706368,"kb_used":4824720,"kb_used_data":9500,"kb_used_omap":217,"kb_used_meta":4814886,"kb_avail":838881648,"statfs":{"total":863955320832,"available":859014807552,"internally_reserved":0,"allocated":9728000,"data_stored":5342513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":222986,"internal_metadata":4930443510},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.006821"},"pg_stats":[{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":778,"state":"active+clean","last_fresh":"2024-04-20T17:43:18.799928+0000","last_change":"2024-04-20T17:43:18.799414+0000","last_active":"2024-04-20T17:43:18.799928+0000","last_peered":"2024-04-20T17:43:18.799928+0000","last_clean":"2024-04-20T17:43:18.799928+0000","last_became_active":"2024-04-20T17:43:18.798932+0000","last_became_peered":"2024-04-20T17:43:18.798932+0000","last_unstale":"2024-04-20T17:43:18.799928+0000","last_undegraded":"2024-04-20T17:43:18.799928+0000","last_fullsized":"2024-04-20T17:43:18.799928+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:44:19.816017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-20T17:43:17.591778+0000","last_change":"2024-04-20T17:43:17.591778+0000","last_active":"2024-04-20T17:43:17.591778+0000","last_peered":"2024-04-20T17:43:17.591778+0000","last_clean":"2024-04-20T17:43:17.591778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-20T17:43:17.591778+0000","last_undegraded":"2024-04-20T17:43:17.591778+0000","last_fullsized":"2024-04-20T17:43:17.591778+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":778,"state":"active+clean","last_fresh":"2024-04-20T17:43:18.799925+0000","last_change":"2024-04-20T17:43:18.799376+0000","last_active":"2024-04-20T17:43:18.799925+0000","last_peered":"2024-04-20T17:43:18.799925+0000","last_clean":"2024-04-20T17:43:18.799925+0000","last_became_active":"2024-04-20T17:43:18.798955+0000","last_became_peered":"2024-04-20T17:43:18.798955+0000","last_unstale":"2024-04-20T17:43:18.799925+0000","last_undegraded":"2024-04-20T17:43:18.799925+0000","last_fullsized":"2024-04-20T17:43:18.799925+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:44:33.048560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-20T17:43:17.591778+0000","last_change":"2024-04-20T17:43:17.591778+0000","last_active":"2024-04-20T17:43:17.591778+0000","last_peered":"2024-04-20T17:43:17.591778+0000","last_clean":"2024-04-20T17:43:17.591778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-20T17:43:17.591778+0000","last_undegraded":"2024-04-20T17:43:17.591778+0000","last_fullsized":"2024-04-20T17:43:17.591778+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":3,"reported_epoch":778,"state":"creating+activating","last_fresh":"2024-04-20T17:43:18.601201+0000","last_change":"2024-04-20T17:43:18.601201+0000","last_active":"2024-04-20T17:43:17.591778+0000","last_peered":"2024-04-20T17:43:17.591778+0000","last_clean":"2024-04-20T17:43:17.591778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-20T17:43:18.601201+0000","last_undegraded":"2024-04-20T17:43:18.601201+0000","last_fullsized":"2024-04-20T17:43:18.601201+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":["8","4"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":3,"reported_epoch":778,"state":"creating+activating","last_fresh":"2024-04-20T17:43:18.601181+0000","last_change":"2024-04-20T17:43:18.601181+0000","last_active":"2024-04-20T17:43:17.591778+0000","last_peered":"2024-04-20T17:43:17.591778+0000","last_clean":"2024-04-20T17:43:17.591778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-20T17:43:18.601181+0000","last_undegraded":"2024-04-20T17:43:18.601181+0000","last_fullsized":"2024-04-20T17:43:18.601181+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2],"acting":[8,2],"avail_no_missing":["8","2"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":778,"state":"active+clean","last_fresh":"2024-04-20T17:43:18.610888+0000","last_change":"2024-04-20T17:43:18.610635+0000","last_active":"2024-04-20T17:43:18.610888+0000","last_peered":"2024-04-20T17:43:18.610888+0000","last_clean":"2024-04-20T17:43:18.610888+0000","last_became_active":"2024-04-20T17:43:18.610421+0000","last_became_peered":"2024-04-20T17:43:18.610421+0000","last_unstale":"2024-04-20T17:43:18.610888+0000","last_undegraded":"2024-04-20T17:43:18.610888+0000","last_fullsized":"2024-04-20T17:43:18.610888+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:44:30.178923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":778,"state":"active+clean","last_fresh":"2024-04-20T17:43:18.599830+0000","last_change":"2024-04-20T17:43:18.599491+0000","last_active":"2024-04-20T17:43:18.599830+0000","last_peered":"2024-04-20T17:43:18.599830+0000","last_clean":"2024-04-20T17:43:18.599830+0000","last_became_active":"2024-04-20T17:43:18.599234+0000","last_became_peered":"2024-04-20T17:43:18.599234+0000","last_unstale":"2024-04-20T17:43:18.599830+0000","last_undegraded":"2024-04-20T17:43:18.599830+0000","last_fullsized":"2024-04-20T17:43:18.599830+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:44:28.318878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":778,"state":"active+clean","last_fresh":"2024-04-20T17:43:18.600544+0000","last_change":"2024-04-20T17:43:18.600183+0000","last_active":"2024-04-20T17:43:18.600544+0000","last_peered":"2024-04-20T17:43:18.600544+0000","last_clean":"2024-04-20T17:43:18.600544+0000","last_became_active":"2024-04-20T17:43:18.599894+0000","last_became_peered":"2024-04-20T17:43:18.599894+0000","last_unstale":"2024-04-20T17:43:18.600544+0000","last_undegraded":"2024-04-20T17:43:18.600544+0000","last_fullsized":"2024-04-20T17:43:18.600544+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:44:44.304813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-20T17:43:17.591778+0000","last_change":"2024-04-20T17:43:17.591778+0000","last_active":"2024-04-20T17:43:17.591778+0000","last_peered":"2024-04-20T17:43:17.591778+0000","last_clean":"2024-04-20T17:43:17.591778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-20T17:43:17.591778+0000","last_undegraded":"2024-04-20T17:43:17.591778+0000","last_fullsized":"2024-04-20T17:43:17.591778+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-20T17:43:17.591778+0000","last_change":"2024-04-20T17:43:17.591778+0000","last_active":"2024-04-20T17:43:17.591778+0000","last_peered":"2024-04-20T17:43:17.591778+0000","last_clean":"2024-04-20T17:43:17.591778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-20T17:43:17.591778+0000","last_undegraded":"2024-04-20T17:43:17.591778+0000","last_fullsized":"2024-04-20T17:43:17.591778+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":778,"state":"active+clean","last_fresh":"2024-04-20T17:43:18.611143+0000","last_change":"2024-04-20T17:43:18.610814+0000","last_active":"2024-04-20T17:43:18.611143+0000","last_peered":"2024-04-20T17:43:18.611143+0000","last_clean":"2024-04-20T17:43:18.611143+0000","last_became_active":"2024-04-20T17:43:18.610492+0000","last_became_peered":"2024-04-20T17:43:18.610492+0000","last_unstale":"2024-04-20T17:43:18.611143+0000","last_undegraded":"2024-04-20T17:43:18.611143+0000","last_fullsized":"2024-04-20T17:43:18.611143+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:44:46.042371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-20T17:43:17.591778+0000","last_change":"2024-04-20T17:43:17.591778+0000","last_active":"2024-04-20T17:43:17.591778+0000","last_peered":"2024-04-20T17:43:17.591778+0000","last_clean":"2024-04-20T17:43:17.591778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-20T17:43:17.591778+0000","last_undegraded":"2024-04-20T17:43:17.591778+0000","last_fullsized":"2024-04-20T17:43:17.591778+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-20T17:43:17.591778+0000","last_change":"2024-04-20T17:43:17.591778+0000","last_active":"2024-04-20T17:43:17.591778+0000","last_peered":"2024-04-20T17:43:17.591778+0000","last_clean":"2024-04-20T17:43:17.591778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-20T17:43:17.591778+0000","last_undegraded":"2024-04-20T17:43:17.591778+0000","last_fullsized":"2024-04-20T17:43:17.591778+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":778,"state":"active+clean","last_fresh":"2024-04-20T17:43:18.610974+0000","last_change":"2024-04-20T17:43:18.610687+0000","last_active":"2024-04-20T17:43:18.610974+0000","last_peered":"2024-04-20T17:43:18.610974+0000","last_clean":"2024-04-20T17:43:18.610974+0000","last_became_active":"2024-04-20T17:43:18.610431+0000","last_became_peered":"2024-04-20T17:43:18.610431+0000","last_unstale":"2024-04-20T17:43:18.610974+0000","last_undegraded":"2024-04-20T17:43:18.610974+0000","last_fullsized":"2024-04-20T17:43:18.610974+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:44:31.345360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-20T17:43:17.591778+0000","last_change":"2024-04-20T17:43:17.591778+0000","last_active":"2024-04-20T17:43:17.591778+0000","last_peered":"2024-04-20T17:43:17.591778+0000","last_clean":"2024-04-20T17:43:17.591778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-20T17:43:17.591778+0000","last_undegraded":"2024-04-20T17:43:17.591778+0000","last_fullsized":"2024-04-20T17:43:17.591778+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":576,"reported_epoch":778,"state":"active+clean","last_fresh":"2024-04-20T17:43:18.596488+0000","last_change":"2024-04-20T17:42:20.165296+0000","last_active":"2024-04-20T17:43:18.596488+0000","last_peered":"2024-04-20T17:43:18.596488+0000","last_clean":"2024-04-20T17:43:18.596488+0000","last_became_active":"2024-04-20T17:39:54.970208+0000","last_became_peered":"2024-04-20T17:39:54.970208+0000","last_unstale":"2024-04-20T17:43:18.596488+0000","last_undegraded":"2024-04-20T17:43:18.596488+0000","last_fullsized":"2024-04-20T17:43:18.596488+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:42:20.165149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:42:20.165149+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:43:22.534703+0000","scrub_duration":180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991684,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":704592,"kb_used_data":1476,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93040560,"statfs":{"total":95995035648,"available":95273533440,"internally_reserved":0,"allocated":1511424,"data_stored":1013249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":719976900},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253323,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":569380,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":93175772,"statfs":{"total":95995035648,"available":95411990528,"internally_reserved":0,"allocated":942080,"data_stored":465990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":582089152},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613071,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":596280,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93148872,"statfs":{"total":95995035648,"available":95384444928,"internally_reserved":0,"allocated":1486848,"data_stored":1000807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":609089984},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809302,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":338952,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93406200,"statfs":{"total":95995035648,"available":95647948800,"internally_reserved":0,"allocated":978944,"data_stored":484341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,2,3,4,5,6,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":2297807503532,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":2830383448155,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":663792,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93081360,"statfs":{"total":95995035648,"available":95315312640,"internally_reserved":0,"allocated":954368,"data_stored":471879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":329780,"kb_used_data":932,"kb_used_omap":202,"kb_used_meta":328629,"kb_avail":93415372,"statfs":{"total":95995035648,"available":95657340928,"internally_reserved":0,"allocated":954368,"data_stored":471879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":336516567},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653216,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":438740,"kb_used_data":968,"kb_used_omap":4,"kb_used_meta":437755,"kb_avail":93306412,"statfs":{"total":95995035648,"available":95545765888,"internally_reserved":0,"allocated":991232,"data_stored":490610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":448261831},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986664,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":533300,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":93211852,"statfs":{"total":95995035648,"available":95448936448,"internally_reserved":0,"allocated":954368,"data_stored":471879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":545126853},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790434,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":649904,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93095248,"statfs":{"total":95995035648,"available":95329533952,"internally_reserved":0,"allocated":954368,"data_stored":471879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:43:21.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:20 smithi171 ceph-mon[44189]: osdmap e779: 12 total, 11 up, 9 in 2024-04-20T17:43:21.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:20 smithi171 ceph-mon[44189]: from='client.17376 v1:172.21.15.5:0/1098176962' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:43:21.267 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:43:21.269 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-20T17:43:21.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:21.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:43:21.371 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:43:21.404 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:21.425 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:43:21.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:21.428 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:43:21.529 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:43:21.534 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:43:21.558 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:21.579 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:43:21.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:21.582 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:43:21.683 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:43:21.862 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:21 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:43:21.862 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:21 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:43:21.862 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:21 smithi171 ceph-mon[44189]: pgmap v2086: 17 pgs: 2 creating+activating, 11 active+clean, 4 unknown; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:21.862 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:21 smithi171 ceph-mon[44189]: osdmap e780: 12 total, 11 up, 9 in 2024-04-20T17:43:21.862 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:43:21.890 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:21 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:43:21.890 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:21 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:43:21.891 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:21 smithi005 ceph-mon[39562]: pgmap v2086: 17 pgs: 2 creating+activating, 11 active+clean, 4 unknown; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:21.891 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:21 smithi005 ceph-mon[39562]: osdmap e780: 12 total, 11 up, 9 in 2024-04-20T17:43:21.893 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:21.950 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:43:21.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:21.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:43:22.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:21 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:43:22.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:21 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:43:22.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:21 smithi106 ceph-mon[44423]: pgmap v2086: 17 pgs: 2 creating+activating, 11 active+clean, 4 unknown; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:22.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:21 smithi106 ceph-mon[44423]: osdmap e780: 12 total, 11 up, 9 in 2024-04-20T17:43:22.055 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:43:22.083 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:22.155 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:43:22.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:22.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:43:22.259 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:43:22.287 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:22.306 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:43:22.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:22.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:43:22.409 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:43:22.438 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:22.544 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:43:23.288 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:43:23.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:23.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:43:23.392 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:43:23.425 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:23.428 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:43:23.428 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:43:23.446 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:43:23.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:23.449 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:43:23.550 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:43:23.580 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:23.599 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:43:23.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:23.602 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:43:23.617 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:43:23.703 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:43:23.732 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:23 smithi106 ceph-mon[44423]: pgmap v2088: 17 pgs: 2 creating+activating, 15 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:23.732 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:23 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1735080612' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:43:23.735 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:23.763 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:23 smithi005 ceph-mon[39562]: pgmap v2088: 17 pgs: 2 creating+activating, 15 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:23.763 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:23 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1735080612' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:43:23.819 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:43:23.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:23 smithi171 ceph-mon[44189]: pgmap v2088: 17 pgs: 2 creating+activating, 15 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:23.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:23 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1735080612' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:43:24.184 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:43:24.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:24.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:43:24.200 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-20T17:43:24.288 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:43:24.321 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:24.342 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:43:24.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:24.345 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:43:24.446 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:43:24.474 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:24.494 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:43:24.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:24.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:43:24.503 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:43:24.598 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:43:24.625 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:24.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:24 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:43:24.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:24 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1735080612' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:43:24.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:24 smithi005 ceph-mon[39562]: osdmap e781: 12 total, 11 up, 9 in 2024-04-20T17:43:24.742 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:24 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:43:24.742 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:24 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1735080612' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:43:24.742 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:24 smithi106 ceph-mon[44423]: osdmap e781: 12 total, 11 up, 9 in 2024-04-20T17:43:24.756 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:43:24.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:24.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:43:24.861 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:43:24.889 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:24 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:43:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:24 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1735080612' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:43:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:24 smithi171 ceph-mon[44189]: osdmap e781: 12 total, 11 up, 9 in 2024-04-20T17:43:25.338 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:43:25.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:25.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:43:25.442 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:43:25.473 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:25.502 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:43:25.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:25.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:43:25.540 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:43:25.607 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:43:25.626 INFO:teuthology.orchestra.run.smithi005.stderr:enabled application 'rados' on pool 'unique_pool_6' 2024-04-20T17:43:25.640 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:25.661 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:43:25.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:25.664 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:43:25.765 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:43:25.791 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:25 smithi005 ceph-mon[39562]: pgmap v2090: 17 pgs: 17 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:25.792 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:25 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3378094957' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:43:25.794 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:25.994 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:43:25.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:25.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:43:26.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:25 smithi106 ceph-mon[44423]: pgmap v2090: 17 pgs: 17 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:26.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:25 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3378094957' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:43:26.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:25 smithi171 ceph-mon[44189]: pgmap v2090: 17 pgs: 17 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:26.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:25 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3378094957' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:43:26.099 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:43:26.130 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:26.152 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:43:26.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:26.155 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:43:26.258 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:43:26.288 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:26.376 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:43:26.376 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:43:26.424 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:43:26.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:26 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3378094957' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-20T17:43:26.634 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:26 smithi005 ceph-mon[39562]: osdmap e782: 12 total, 11 up, 9 in 2024-04-20T17:43:26.712 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:26 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3378094957' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-20T17:43:26.712 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:26 smithi106 ceph-mon[44423]: osdmap e782: 12 total, 11 up, 9 in 2024-04-20T17:43:26.756 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:43:26.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:26.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:43:26.870 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:43:26.906 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:26 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3378094957' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-20T17:43:26.906 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:26 smithi171 ceph-mon[44189]: osdmap e782: 12 total, 11 up, 9 in 2024-04-20T17:43:26.908 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:26.931 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:43:26.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:26.934 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:43:27.034 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-20T17:43:27.035 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:43:27.063 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:27.082 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:43:27.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:27.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:43:27.187 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:43:27.216 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:27.241 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:43:27.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:27.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:43:27.346 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:43:27.379 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:27.400 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:43:27.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:27.404 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:43:27.503 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:43:27.505 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:43:27.523 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:43:27.539 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:27.543 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:43:27.558 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:43:27.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:27.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:43:27.562 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:43:27.582 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:43:27.601 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:43:27.621 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:43:27.642 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:43:27.662 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:43:27.663 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:43:27.685 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:43:27.692 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:27.703 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:43:27.704 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:43:27.704 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:43:27.704 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:43:27.704 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:43:27.704 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:43:27.704 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:43:27.705 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:43:27.705 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:43:27.705 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:43:27.705 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:43:27.705 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:43:27.705 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:43:27.710 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:43:27.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:27.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:43:27.814 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:43:27.839 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:27 smithi106 ceph-mon[44423]: pgmap v2092: 17 pgs: 17 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:27.839 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:27 smithi106 ceph-mon[44423]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-20T17:43:27.839 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:27 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3034076332' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:43:27.840 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:27.860 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:43:27.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:27.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:43:27.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:27 smithi005 ceph-mon[39562]: pgmap v2092: 17 pgs: 17 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:27.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:27 smithi005 ceph-mon[39562]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-20T17:43:27.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:27 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3034076332' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:43:27.963 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:43:27.993 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:28.013 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:43:28.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:28.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:43:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:27 smithi171 ceph-mon[44189]: pgmap v2092: 17 pgs: 17 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:27 smithi171 ceph-mon[44189]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-20T17:43:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:27 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3034076332' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:43:28.120 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:43:28.152 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:28.173 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:43:28.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:28.177 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:43:28.251 DEBUG:teuthology.orchestra.run.smithi171:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_6 bench 90 write' 2024-04-20T17:43:28.254 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-20T17:43:28.254 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-20T17:43:28.277 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:43:28.306 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:28.325 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:43:28.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:28.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:43:28.408 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:hints = 1 2024-04-20T17:43:28.408 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-20T17:43:28.408 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Object prefix: benchmark_data_smithi171_175757 2024-04-20T17:43:28.409 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:43:28.409 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 0 0 0 0 0 0 - 0 2024-04-20T17:43:28.428 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:43:28.464 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:28.510 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:43:28.511 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:43:28.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:28.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:43:28.562 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:43:28.563 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:43:28.566 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:43:28.618 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:43:28.653 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:28.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:28.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:28.792 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:43:28.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:28.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:43:28.896 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:43:28.951 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:28.952 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:28.952 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:28.952 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:28.952 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:28.952 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:28.952 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:28.952 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:28.952 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:28.952 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:28.952 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:28.954 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:28.987 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:43:28.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:28.991 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:43:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:29.092 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:43:29.121 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:29.140 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:43:29.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:29.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:43:29.245 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:43:29.289 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:29.342 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:43:29.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:29.379 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:43:29.409 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 1 16 4415 4399 274.838 274.938 0.00196964 0.00362655 2024-04-20T17:43:29.480 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:43:29.518 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:29.816 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:29 smithi005 ceph-mon[39562]: pgmap v2093: 17 pgs: 17 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:30.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:29 smithi106 ceph-mon[44423]: pgmap v2093: 17 pgs: 17 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:30.050 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:43:30.055 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:43:30.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:29 smithi171 ceph-mon[44189]: pgmap v2093: 17 pgs: 17 active+clean; 577 KiB data, 4.6 GiB used, 800 GiB / 805 GiB avail 2024-04-20T17:43:30.409 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 2 16 8318 8302 259.38 243.938 0.00392827 0.00385005 2024-04-20T17:43:30.564 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:43:30.880 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:43:30.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:30.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:43:30.989 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:43:31.044 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:31.074 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:43:31.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:31.079 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:43:31.180 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:43:31.218 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:31.246 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:43:31.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:31.250 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:43:31.351 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:43:31.394 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:31.410 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 3 16 12155 12139 252.809 239.812 0.00403634 0.00394959 2024-04-20T17:43:31.421 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:43:31.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:31.425 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:43:31.526 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:43:31.555 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:31.575 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:43:31.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:31.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:43:31.660 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:31 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:43:31.660 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:31 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:43:31.661 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:31 smithi171 ceph-mon[44189]: from='client.27581 v1:172.21.15.5:0/1179515778' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:43:31.661 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:31 smithi171 ceph-mon[44189]: pgmap v2094: 17 pgs: 17 active+clean; 180 MiB data, 4.7 GiB used, 800 GiB / 805 GiB avail; 22 MiB/s wr, 358 op/s 2024-04-20T17:43:31.661 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:31 smithi171 ceph-mon[44189]: osdmap e783: 12 total, 11 up, 9 in 2024-04-20T17:43:31.661 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:31 smithi171 ceph-mon[44189]: osdmap e784: 12 total, 11 up, 9 in 2024-04-20T17:43:31.678 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:43:31.729 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:31.755 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:43:31.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:31.760 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:43:31.861 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:43:31.886 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:31 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:43:31.886 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:31 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:43:31.886 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:31 smithi106 ceph-mon[44423]: from='client.27581 v1:172.21.15.5:0/1179515778' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:43:31.886 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:31 smithi106 ceph-mon[44423]: pgmap v2094: 17 pgs: 17 active+clean; 180 MiB data, 4.7 GiB used, 800 GiB / 805 GiB avail; 22 MiB/s wr, 358 op/s 2024-04-20T17:43:31.887 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:31 smithi106 ceph-mon[44423]: osdmap e783: 12 total, 11 up, 9 in 2024-04-20T17:43:31.887 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:31 smithi106 ceph-mon[44423]: osdmap e784: 12 total, 11 up, 9 in 2024-04-20T17:43:31.889 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:31.910 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:43:31.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:31.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:43:31.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:31 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:43:31.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:31 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:43:31.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:31 smithi005 ceph-mon[39562]: from='client.27581 v1:172.21.15.5:0/1179515778' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:43:31.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:31 smithi005 ceph-mon[39562]: pgmap v2094: 17 pgs: 17 active+clean; 180 MiB data, 4.7 GiB used, 800 GiB / 805 GiB avail; 22 MiB/s wr, 358 op/s 2024-04-20T17:43:31.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:31 smithi005 ceph-mon[39562]: osdmap e783: 12 total, 11 up, 9 in 2024-04-20T17:43:31.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:31 smithi005 ceph-mon[39562]: osdmap e784: 12 total, 11 up, 9 in 2024-04-20T17:43:32.014 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:43:32.032 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:43:32.045 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":2093,"stamp":"2024-04-20T17:43:28.592344+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":4825288,"kb_used_data":10052,"kb_used_omap":217,"kb_used_meta":4814886,"kb_avail":838881080,"statfs":{"total":863955320832,"available":859014225920,"internally_reserved":0,"allocated":10293248,"data_stored":5632930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":222986,"internal_metadata":4930443510},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.981724"},"pg_stats":[{"pgid":"8.f","version":"0'0","reported_seq":16,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-04-20T17:43:23.947708+0000","last_change":"2024-04-20T17:43:18.799414+0000","last_active":"2024-04-20T17:43:23.947708+0000","last_peered":"2024-04-20T17:43:23.947708+0000","last_clean":"2024-04-20T17:43:23.947708+0000","last_became_active":"2024-04-20T17:43:18.798932+0000","last_became_peered":"2024-04-20T17:43:18.798932+0000","last_unstale":"2024-04-20T17:43:23.947708+0000","last_undegraded":"2024-04-20T17:43:23.947708+0000","last_fullsized":"2024-04-20T17:43:23.947708+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:44:44.904613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":19,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-04-20T17:43:25.801662+0000","last_change":"2024-04-20T17:43:18.799089+0000","last_active":"2024-04-20T17:43:25.801662+0000","last_peered":"2024-04-20T17:43:25.801662+0000","last_clean":"2024-04-20T17:43:25.801662+0000","last_became_active":"2024-04-20T17:43:18.798720+0000","last_became_peered":"2024-04-20T17:43:18.798720+0000","last_unstale":"2024-04-20T17:43:25.801662+0000","last_undegraded":"2024-04-20T17:43:25.801662+0000","last_fullsized":"2024-04-20T17:43:25.801662+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:44:28.188266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":16,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-04-20T17:43:23.947661+0000","last_change":"2024-04-20T17:43:18.799376+0000","last_active":"2024-04-20T17:43:23.947661+0000","last_peered":"2024-04-20T17:43:23.947661+0000","last_clean":"2024-04-20T17:43:23.947661+0000","last_became_active":"2024-04-20T17:43:18.798955+0000","last_became_peered":"2024-04-20T17:43:18.798955+0000","last_unstale":"2024-04-20T17:43:23.947661+0000","last_undegraded":"2024-04-20T17:43:23.947661+0000","last_fullsized":"2024-04-20T17:43:23.947661+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:44:18.915884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":19,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-04-20T17:43:25.802705+0000","last_change":"2024-04-20T17:43:18.602404+0000","last_active":"2024-04-20T17:43:25.802705+0000","last_peered":"2024-04-20T17:43:25.802705+0000","last_clean":"2024-04-20T17:43:25.802705+0000","last_became_active":"2024-04-20T17:43:18.602140+0000","last_became_peered":"2024-04-20T17:43:18.602140+0000","last_unstale":"2024-04-20T17:43:25.802705+0000","last_undegraded":"2024-04-20T17:43:25.802705+0000","last_fullsized":"2024-04-20T17:43:25.802705+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:44:18.738814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":14,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-04-20T17:43:20.684060+0000","last_change":"2024-04-20T17:43:18.801639+0000","last_active":"2024-04-20T17:43:20.684060+0000","last_peered":"2024-04-20T17:43:20.684060+0000","last_clean":"2024-04-20T17:43:20.684060+0000","last_became_active":"2024-04-20T17:43:18.801242+0000","last_became_peered":"2024-04-20T17:43:18.801242+0000","last_unstale":"2024-04-20T17:43:20.684060+0000","last_undegraded":"2024-04-20T17:43:20.684060+0000","last_fullsized":"2024-04-20T17:43:20.684060+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:44:36.059018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":780,"state":"active+clean","last_fresh":"2024-04-20T17:43:20.684046+0000","last_change":"2024-04-20T17:43:18.799301+0000","last_active":"2024-04-20T17:43:20.684046+0000","last_peered":"2024-04-20T17:43:20.684046+0000","last_clean":"2024-04-20T17:43:20.684046+0000","last_became_active":"2024-04-20T17:43:18.798958+0000","last_became_peered":"2024-04-20T17:43:18.798958+0000","last_unstale":"2024-04-20T17:43:20.684046+0000","last_undegraded":"2024-04-20T17:43:20.684046+0000","last_fullsized":"2024-04-20T17:43:20.684046+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:44:19.828028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2],"acting":[8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":16,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-04-20T17:43:23.934103+0000","last_change":"2024-04-20T17:43:18.610635+0000","last_active":"2024-04-20T17:43:23.934103+0000","last_peered":"2024-04-20T17:43:23.934103+0000","last_clean":"2024-04-20T17:43:23.934103+0000","last_became_active":"2024-04-20T17:43:18.610421+0000","last_became_peered":"2024-04-20T17:43:18.610421+0000","last_unstale":"2024-04-20T17:43:23.934103+0000","last_undegraded":"2024-04-20T17:43:23.934103+0000","last_fullsized":"2024-04-20T17:43:23.934103+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:44:35.037282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":16,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-04-20T17:43:23.947718+0000","last_change":"2024-04-20T17:43:18.599491+0000","last_active":"2024-04-20T17:43:23.947718+0000","last_peered":"2024-04-20T17:43:23.947718+0000","last_clean":"2024-04-20T17:43:23.947718+0000","last_became_active":"2024-04-20T17:43:18.599234+0000","last_became_peered":"2024-04-20T17:43:18.599234+0000","last_unstale":"2024-04-20T17:43:23.947718+0000","last_undegraded":"2024-04-20T17:43:23.947718+0000","last_fullsized":"2024-04-20T17:43:23.947718+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:44:45.660100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":16,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-04-20T17:43:23.947674+0000","last_change":"2024-04-20T17:43:18.600183+0000","last_active":"2024-04-20T17:43:23.947674+0000","last_peered":"2024-04-20T17:43:23.947674+0000","last_clean":"2024-04-20T17:43:23.947674+0000","last_became_active":"2024-04-20T17:43:18.599894+0000","last_became_peered":"2024-04-20T17:43:18.599894+0000","last_unstale":"2024-04-20T17:43:23.947674+0000","last_undegraded":"2024-04-20T17:43:23.947674+0000","last_fullsized":"2024-04-20T17:43:23.947674+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:44:32.834204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":16,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-04-20T17:43:24.435298+0000","last_change":"2024-04-20T17:43:18.801577+0000","last_active":"2024-04-20T17:43:24.435298+0000","last_peered":"2024-04-20T17:43:24.435298+0000","last_clean":"2024-04-20T17:43:24.435298+0000","last_became_active":"2024-04-20T17:43:18.801263+0000","last_became_peered":"2024-04-20T17:43:18.801263+0000","last_unstale":"2024-04-20T17:43:24.435298+0000","last_undegraded":"2024-04-20T17:43:24.435298+0000","last_fullsized":"2024-04-20T17:43:24.435298+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:44:24.552192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":19,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-04-20T17:43:25.802887+0000","last_change":"2024-04-20T17:43:18.610704+0000","last_active":"2024-04-20T17:43:25.802887+0000","last_peered":"2024-04-20T17:43:25.802887+0000","last_clean":"2024-04-20T17:43:25.802887+0000","last_became_active":"2024-04-20T17:43:18.610249+0000","last_became_peered":"2024-04-20T17:43:18.610249+0000","last_unstale":"2024-04-20T17:43:25.802887+0000","last_undegraded":"2024-04-20T17:43:25.802887+0000","last_fullsized":"2024-04-20T17:43:25.802887+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:44:36.713359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":16,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-04-20T17:43:23.947680+0000","last_change":"2024-04-20T17:43:18.610814+0000","last_active":"2024-04-20T17:43:23.947680+0000","last_peered":"2024-04-20T17:43:23.947680+0000","last_clean":"2024-04-20T17:43:23.947680+0000","last_became_active":"2024-04-20T17:43:18.610492+0000","last_became_peered":"2024-04-20T17:43:18.610492+0000","last_unstale":"2024-04-20T17:43:23.947680+0000","last_undegraded":"2024-04-20T17:43:23.947680+0000","last_fullsized":"2024-04-20T17:43:23.947680+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:44:38.926756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":16,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-04-20T17:43:24.435359+0000","last_change":"2024-04-20T17:43:18.599508+0000","last_active":"2024-04-20T17:43:24.435359+0000","last_peered":"2024-04-20T17:43:24.435359+0000","last_clean":"2024-04-20T17:43:24.435359+0000","last_became_active":"2024-04-20T17:43:18.599198+0000","last_became_peered":"2024-04-20T17:43:18.599198+0000","last_unstale":"2024-04-20T17:43:24.435359+0000","last_undegraded":"2024-04-20T17:43:24.435359+0000","last_fullsized":"2024-04-20T17:43:24.435359+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:44:33.182874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":19,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-04-20T17:43:25.802530+0000","last_change":"2024-04-20T17:43:18.799305+0000","last_active":"2024-04-20T17:43:25.802530+0000","last_peered":"2024-04-20T17:43:25.802530+0000","last_clean":"2024-04-20T17:43:25.802530+0000","last_became_active":"2024-04-20T17:43:18.798942+0000","last_became_peered":"2024-04-20T17:43:18.798942+0000","last_unstale":"2024-04-20T17:43:25.802530+0000","last_undegraded":"2024-04-20T17:43:25.802530+0000","last_fullsized":"2024-04-20T17:43:25.802530+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:44:43.396567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":16,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-04-20T17:43:23.934103+0000","last_change":"2024-04-20T17:43:18.610687+0000","last_active":"2024-04-20T17:43:23.934103+0000","last_peered":"2024-04-20T17:43:23.934103+0000","last_clean":"2024-04-20T17:43:23.934103+0000","last_became_active":"2024-04-20T17:43:18.610431+0000","last_became_peered":"2024-04-20T17:43:18.610431+0000","last_unstale":"2024-04-20T17:43:23.934103+0000","last_undegraded":"2024-04-20T17:43:23.934103+0000","last_fullsized":"2024-04-20T17:43:23.934103+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:44:20.441873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":16,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-04-20T17:43:24.435082+0000","last_change":"2024-04-20T17:43:18.610645+0000","last_active":"2024-04-20T17:43:24.435082+0000","last_peered":"2024-04-20T17:43:24.435082+0000","last_clean":"2024-04-20T17:43:24.435082+0000","last_became_active":"2024-04-20T17:43:18.610315+0000","last_became_peered":"2024-04-20T17:43:18.610315+0000","last_unstale":"2024-04-20T17:43:24.435082+0000","last_undegraded":"2024-04-20T17:43:24.435082+0000","last_fullsized":"2024-04-20T17:43:24.435082+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:44:27.824102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":582,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-04-20T17:43:23.947671+0000","last_change":"2024-04-20T17:42:20.165296+0000","last_active":"2024-04-20T17:43:23.947671+0000","last_peered":"2024-04-20T17:43:23.947671+0000","last_clean":"2024-04-20T17:43:23.947671+0000","last_became_active":"2024-04-20T17:39:54.970208+0000","last_became_peered":"2024-04-20T17:39:54.970208+0000","last_unstale":"2024-04-20T17:43:23.947671+0000","last_undegraded":"2024-04-20T17:43:23.947671+0000","last_fullsized":"2024-04-20T17:43:23.947671+0000","mapping_epoch":727,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":728,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:42:20.165149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:42:20.165149+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9],"acting":[11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4231424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991685,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":704628,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93040524,"statfs":{"total":95995035648,"available":95273496576,"internally_reserved":0,"allocated":1548288,"data_stored":1032337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":719976900},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253325,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":569456,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":93175696,"statfs":{"total":95995035648,"available":95411912704,"internally_reserved":0,"allocated":1015808,"data_stored":503409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":582089152},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613073,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":596352,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93148800,"statfs":{"total":95995035648,"available":95384371200,"internally_reserved":0,"allocated":1560576,"data_stored":1038578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":609089984},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809303,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":338996,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93406156,"statfs":{"total":95995035648,"available":95647903744,"internally_reserved":0,"allocated":1015808,"data_stored":503429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,2,3,4,5,6,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":2297807503534,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":2830383448156,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738878,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":663880,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93081272,"statfs":{"total":95995035648,"available":95315222528,"internally_reserved":0,"allocated":1040384,"data_stored":516224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394168,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":329860,"kb_used_data":1016,"kb_used_omap":202,"kb_used_meta":328629,"kb_avail":93415292,"statfs":{"total":95995035648,"available":95657259008,"internally_reserved":0,"allocated":1040384,"data_stored":516224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":336516567},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653217,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":438772,"kb_used_data":992,"kb_used_omap":4,"kb_used_meta":437755,"kb_avail":93306380,"statfs":{"total":95995035648,"available":95545733120,"internally_reserved":0,"allocated":1015808,"data_stored":503429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":448261831},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986666,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":533368,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":93211784,"statfs":{"total":95995035648,"available":95448866816,"internally_reserved":0,"allocated":1028096,"data_stored":509650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":545126853},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790436,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":649976,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93095176,"statfs":{"total":95995035648,"available":95329460224,"internally_reserved":0,"allocated":1028096,"data_stored":509650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2356,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:43:32.046 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-20T17:43:32.046 INFO:tasks.thrashosds.thrasher:Waiting for trim 2024-04-20T17:43:32.064 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:32.412 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 4 16 15787 15771 246.311 227 0.00252377 0.00405662 2024-04-20T17:43:33.411 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 5 16 19285 19269 240.771 218.625 0.00371566 0.00415063 2024-04-20T17:43:33.621 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:43:33.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:33.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:43:33.728 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:43:33.776 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:33.808 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:43:33.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:33.812 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:43:33.813 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:43:33.913 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:43:33.940 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:33 smithi106 ceph-mon[44423]: pgmap v2097: 17 pgs: 17 active+clean; 355 MiB data, 5.4 GiB used, 799 GiB / 805 GiB avail; 51 MiB/s wr, 814 op/s 2024-04-20T17:43:33.940 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:33 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3194384494' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:43:33.940 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:33 smithi106 ceph-mon[44423]: from='client.27587 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:43:33.942 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:33.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:33 smithi005 ceph-mon[39562]: pgmap v2097: 17 pgs: 17 active+clean; 355 MiB data, 5.4 GiB used, 799 GiB / 805 GiB avail; 51 MiB/s wr, 814 op/s 2024-04-20T17:43:33.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:33 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3194384494' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:43:33.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:33 smithi005 ceph-mon[39562]: from='client.27587 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:43:33.962 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:43:33.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:33.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:43:34.066 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:43:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:33 smithi171 ceph-mon[44189]: pgmap v2097: 17 pgs: 17 active+clean; 355 MiB data, 5.4 GiB used, 799 GiB / 805 GiB avail; 51 MiB/s wr, 814 op/s 2024-04-20T17:43:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:33 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3194384494' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:43:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:33 smithi171 ceph-mon[44189]: from='client.27587 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:43:34.093 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:34.111 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:43:34.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:34.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:43:34.215 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:43:34.262 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:34.297 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:43:34.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:34.301 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:43:34.402 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:43:34.411 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 6 16 22217 22201 231.181 183.25 0.00695488 0.00432289 2024-04-20T17:43:34.439 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:34.475 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:43:34.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:34.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:43:34.579 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:43:34.619 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:34.751 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:43:34.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:34.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:43:34.765 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:34 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:43:34.766 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:34 smithi005 ceph-mon[39562]: from='client.27587 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:43:34.766 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:34 smithi005 ceph-mon[39562]: osdmap e785: 12 total, 11 up, 9 in 2024-04-20T17:43:34.867 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:43:34.894 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:34.915 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:43:34.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:34.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:43:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:34 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:43:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:34 smithi106 ceph-mon[44423]: from='client.27587 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:43:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:34 smithi106 ceph-mon[44423]: osdmap e785: 12 total, 11 up, 9 in 2024-04-20T17:43:35.018 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:43:35.058 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:34 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:43:35.058 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:34 smithi171 ceph-mon[44189]: from='client.27587 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:43:35.058 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:34 smithi171 ceph-mon[44189]: osdmap e785: 12 total, 11 up, 9 in 2024-04-20T17:43:35.061 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:35.093 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:43:35.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:35.098 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:43:35.199 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:43:35.234 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:35.266 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:43:35.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:35.271 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:43:35.372 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:43:35.406 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:35.411 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 7 16 25300 25284 225.663 192.688 0.0076444 0.00442867 2024-04-20T17:43:35.689 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:35 smithi005 ceph-mon[39562]: pgmap v2099: 17 pgs: 17 active+clean; 987 MiB data, 6.1 GiB used, 799 GiB / 805 GiB avail; 164 MiB/s wr, 2.63k op/s 2024-04-20T17:43:35.689 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:35 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3519730724' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:43:35.689 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:35.689 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:35 smithi106 ceph-mon[44423]: pgmap v2099: 17 pgs: 17 active+clean; 987 MiB data, 6.1 GiB used, 799 GiB / 805 GiB avail; 164 MiB/s wr, 2.63k op/s 2024-04-20T17:43:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:35 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3519730724' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:43:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:36.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:35 smithi171 ceph-mon[44189]: pgmap v2099: 17 pgs: 17 active+clean; 987 MiB data, 6.1 GiB used, 799 GiB / 805 GiB avail; 164 MiB/s wr, 2.63k op/s 2024-04-20T17:43:36.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:35 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3519730724' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:43:36.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:36.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:36.336 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:43:36.336 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:43:36.336 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-04-20T17:43:36.336 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:43:36.413 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 8 16 28364 28348 221.339 191.5 0.0062501 0.00451553 2024-04-20T17:43:36.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:36.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:36.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:36.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:36.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:36.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:36.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:36.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:36.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:36.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:36 smithi005 ceph-mon[39562]: osdmap e786: 12 total, 11 up, 9 in 2024-04-20T17:43:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:37.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:36 smithi106 ceph-mon[44423]: osdmap e786: 12 total, 11 up, 9 in 2024-04-20T17:43:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:37.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:37.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:37.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:36 smithi171 ceph-mon[44189]: osdmap e786: 12 total, 11 up, 9 in 2024-04-20T17:43:37.413 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 9 16 31147 31131 216.073 173.938 0.00572096 0.00462594 2024-04-20T17:43:37.920 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:43:37.920 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:43:37.920 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:43:37.920 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:43:37.921 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:43:37.921 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:43:37.921 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:43:37.921 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:43:37.921 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:43:37.921 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:43:37.921 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:43:37.921 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:43:37.921 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:43:37.921 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:43:37.921 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:43:37.921 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:43:37.922 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:43:37.922 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:43:37.922 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:43:37.922 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:43:37.922 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:43:37.922 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:43:37.922 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:43:37.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:37 smithi005 ceph-mon[39562]: pgmap v2100: 17 pgs: 17 active+clean; 1.4 GiB data, 7.6 GiB used, 797 GiB / 805 GiB avail; 212 MiB/s wr, 3.39k op/s 2024-04-20T17:43:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:37 smithi106 ceph-mon[44423]: pgmap v2100: 17 pgs: 17 active+clean; 1.4 GiB data, 7.6 GiB used, 797 GiB / 805 GiB avail; 212 MiB/s wr, 3.39k op/s 2024-04-20T17:43:38.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:37 smithi171 ceph-mon[44189]: pgmap v2100: 17 pgs: 17 active+clean; 1.4 GiB data, 7.6 GiB used, 797 GiB / 805 GiB avail; 212 MiB/s wr, 3.39k op/s 2024-04-20T17:43:38.414 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 10 16 33932 33916 211.872 174.062 0.00293381 0.00471641 2024-04-20T17:43:38.803 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:43:38.804 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:43:39.260 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:43:39.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:39.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:43:39.283 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-20T17:43:39.366 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:43:39.414 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 11 16 36629 36613 207.935 168.562 0.00468159 0.00480591 2024-04-20T17:43:39.419 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:39.446 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:43:39.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:39.449 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:43:39.550 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:43:39.597 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:39.623 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:43:39.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:39.627 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:43:39.671 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:43:39.700 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:39 smithi171 ceph-mon[44189]: pgmap v2102: 17 pgs: 17 active+clean; 1.4 GiB data, 7.6 GiB used, 797 GiB / 805 GiB avail; 157 MiB/s wr, 2.51k op/s 2024-04-20T17:43:39.728 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:43:39.773 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:39.804 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:43:39.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:39.808 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:43:39.909 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:43:39.943 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:39.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:39 smithi005 ceph-mon[39562]: pgmap v2102: 17 pgs: 17 active+clean; 1.4 GiB data, 7.6 GiB used, 797 GiB / 805 GiB avail; 157 MiB/s wr, 2.51k op/s 2024-04-20T17:43:39.970 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:43:39.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:39.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:43:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:39 smithi106 ceph-mon[44423]: pgmap v2102: 17 pgs: 17 active+clean; 1.4 GiB data, 7.6 GiB used, 797 GiB / 805 GiB avail; 157 MiB/s wr, 2.51k op/s 2024-04-20T17:43:40.075 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:43:40.109 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:40.283 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:43:40.414 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 12 16 39122 39106 203.591 155.812 0.0061709 0.00490984 2024-04-20T17:43:40.567 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:43:40.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:40.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:43:40.674 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:43:40.710 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:40.805 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:43:41.063 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:43:41.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:41.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:43:41.115 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:43:41.168 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:43:41.203 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:41.366 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:43:41.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:41.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:43:41.414 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 13 16 41646 41630 200.065 157.75 0.00974915 0.00499586 2024-04-20T17:43:41.473 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:43:41.501 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:41.522 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:43:41.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:41.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:43:41.626 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:43:41.718 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:41.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:41 smithi106 ceph-mon[44423]: pgmap v2103: 17 pgs: 17 active+clean; 2.1 GiB data, 8.4 GiB used, 796 GiB / 805 GiB avail; 221 MiB/s wr, 3.54k op/s 2024-04-20T17:43:41.768 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:43:41.774 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:43:41.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:41.801 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:43:41.902 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:43:41.929 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:41.949 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:43:41.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:41.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:43:41.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:41 smithi005 ceph-mon[39562]: pgmap v2103: 17 pgs: 17 active+clean; 2.1 GiB data, 8.4 GiB used, 796 GiB / 805 GiB avail; 221 MiB/s wr, 3.54k op/s 2024-04-20T17:43:42.053 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:43:42.080 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:42.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:41 smithi171 ceph-mon[44189]: pgmap v2103: 17 pgs: 17 active+clean; 2.1 GiB data, 8.4 GiB used, 796 GiB / 805 GiB avail; 221 MiB/s wr, 3.54k op/s 2024-04-20T17:43:42.101 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:43:42.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:42.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:43:42.205 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:43:42.227 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:43:42.228 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:43:42.252 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:42.278 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:43:42.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:42.283 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:43:42.384 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:43:42.414 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 14 16 43907 43891 195.868 141.312 0.00729526 0.00510328 2024-04-20T17:43:42.431 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:42.456 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:43:42.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:42.459 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:43:42.560 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:43:42.688 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:42.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:42 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:43:42.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:42 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:43:42.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:42 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:43:42.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:42 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:43:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:42 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:43:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:42 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:43:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:42 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:43:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:42 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:43:43.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:42 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:43:43.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:42 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:43:43.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:42 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:43:43.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:42 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:43:43.414 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 15 16 46240 46224 192.531 145.812 0.0109526 0.00519146 2024-04-20T17:43:44.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:43 smithi106 ceph-mon[44423]: pgmap v2104: 17 pgs: 17 active+clean; 2.3 GiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 222 MiB/s wr, 3.55k op/s 2024-04-20T17:43:44.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:43 smithi106 ceph-mon[44423]: osdmap e787: 12 total, 11 up, 9 in 2024-04-20T17:43:44.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:43 smithi171 ceph-mon[44189]: pgmap v2104: 17 pgs: 17 active+clean; 2.3 GiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 222 MiB/s wr, 3.55k op/s 2024-04-20T17:43:44.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:43 smithi171 ceph-mon[44189]: osdmap e787: 12 total, 11 up, 9 in 2024-04-20T17:43:44.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:43 smithi005 ceph-mon[39562]: pgmap v2104: 17 pgs: 17 active+clean; 2.3 GiB data, 9.4 GiB used, 795 GiB / 805 GiB avail; 222 MiB/s wr, 3.55k op/s 2024-04-20T17:43:44.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:43 smithi005 ceph-mon[39562]: osdmap e787: 12 total, 11 up, 9 in 2024-04-20T17:43:44.414 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 16 16 48461 48445 189.174 138.812 0.00475963 0.00528394 2024-04-20T17:43:45.414 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 17 16 50544 50528 185.705 130.188 0.00636311 0.00538315 2024-04-20T17:43:45.947 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:43:45.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:45.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:43:45.981 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-20T17:43:45.981 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:45 smithi005 ceph-mon[39562]: pgmap v2106: 17 pgs: 17 active+clean; 2.7 GiB data, 9.8 GiB used, 795 GiB / 805 GiB avail; 163 MiB/s wr, 2.61k op/s 2024-04-20T17:43:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:45 smithi106 ceph-mon[44423]: pgmap v2106: 17 pgs: 17 active+clean; 2.7 GiB data, 9.8 GiB used, 795 GiB / 805 GiB avail; 163 MiB/s wr, 2.61k op/s 2024-04-20T17:43:46.051 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:43:46.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:45 smithi171 ceph-mon[44189]: pgmap v2106: 17 pgs: 17 active+clean; 2.7 GiB data, 9.8 GiB used, 795 GiB / 805 GiB avail; 163 MiB/s wr, 2.61k op/s 2024-04-20T17:43:46.114 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:46.144 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:43:46.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:46.149 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:43:46.250 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:43:46.300 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:46.323 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:43:46.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:46.328 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:43:46.414 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 18 16 52673 52657 182.78 133.062 0.0029272 0.00546895 2024-04-20T17:43:46.429 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:43:46.472 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:46.504 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:43:46.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:46.510 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:43:46.612 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:43:46.646 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:47.297 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:43:47.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:47.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:43:47.321 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:43:47.403 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:43:47.414 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 19 16 54707 54691 179.85 127.125 0.00424754 0.0055587 2024-04-20T17:43:47.437 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:47.603 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:47 smithi005 ceph-mon[39562]: pgmap v2107: 17 pgs: 17 active+clean; 3.0 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 162 MiB/s wr, 2.59k op/s 2024-04-20T17:43:47.603 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:47 smithi005 ceph-mon[39562]: osdmap e788: 12 total, 11 up, 9 in 2024-04-20T17:43:47.730 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:43:47.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:47.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:43:47.839 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:43:47.879 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:47 smithi171 ceph-mon[44189]: pgmap v2107: 17 pgs: 17 active+clean; 3.0 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 162 MiB/s wr, 2.59k op/s 2024-04-20T17:43:47.879 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:47 smithi171 ceph-mon[44189]: osdmap e788: 12 total, 11 up, 9 in 2024-04-20T17:43:47.881 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:47.917 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:43:47.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:47.920 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:43:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:47 smithi106 ceph-mon[44423]: pgmap v2107: 17 pgs: 17 active+clean; 3.0 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 162 MiB/s wr, 2.59k op/s 2024-04-20T17:43:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:47 smithi106 ceph-mon[44423]: osdmap e788: 12 total, 11 up, 9 in 2024-04-20T17:43:48.021 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:43:48.050 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:48.071 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:43:48.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:48.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:43:48.124 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:43:48.148 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:43:48.172 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:43:48.175 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:43:48.199 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:43:48.221 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:48.223 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:43:48.245 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:43:48.246 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:43:48.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:48.249 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:43:48.269 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:43:48.295 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:43:48.327 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:43:48.349 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:43:48.356 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:43:48.388 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:43:48.389 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:43:48.389 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:43:48.389 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:43:48.389 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:43:48.389 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:43:48.389 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:43:48.389 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:43:48.389 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:43:48.389 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:43:48.390 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:43:48.390 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:43:48.390 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:43:48.390 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:48.414 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:43:48.415542 min lat: 0.00104535 max lat: 0.205455 avg lat: 0.0056526 2024-04-20T17:43:48.415 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:43:48.415 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 20 16 56627 56611 176.858 120 0.00346981 0.0056526 2024-04-20T17:43:48.420 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:43:48.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:48.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:43:48.525 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:43:48.563 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:48.630 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:43:48.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:48 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2243038107' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:43:48.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:48 smithi005 ceph-mon[39562]: from='client.27599 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:43:48.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:48 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/88638815' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:43:48.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:48.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:48 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2243038107' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:43:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:48 smithi106 ceph-mon[44423]: from='client.27599 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:43:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:48 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/88638815' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:43:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:49.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:48 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2243038107' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:43:49.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:48 smithi171 ceph-mon[44189]: from='client.27599 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:43:49.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:48 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/88638815' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:43:49.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:49.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:49.414 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 21 16 58495 58479 173.996 116.75 0.00303174 0.00574503 2024-04-20T17:43:49.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:49.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:49.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:49.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:49.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:49.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:49.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:49.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:49.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:49.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:49 smithi005 ceph-mon[39562]: pgmap v2109: 17 pgs: 17 active+clean; 3.0 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-04-20T17:43:49.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:49 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:43:49.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:49 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:43:49.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:49 smithi005 ceph-mon[39562]: from='client.27599 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:43:49.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:49 smithi005 ceph-mon[39562]: osdmap e789: 12 total, 11 up, 9 in 2024-04-20T17:43:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:49 smithi106 ceph-mon[44423]: pgmap v2109: 17 pgs: 17 active+clean; 3.0 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-04-20T17:43:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:49 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:43:50.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:49 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:43:50.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:49 smithi106 ceph-mon[44423]: from='client.27599 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:43:50.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:49 smithi106 ceph-mon[44423]: osdmap e789: 12 total, 11 up, 9 in 2024-04-20T17:43:50.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:50.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:50.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:50.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:50.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:50.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:43:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:49 smithi171 ceph-mon[44189]: pgmap v2109: 17 pgs: 17 active+clean; 3.0 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-04-20T17:43:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:49 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:43:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:49 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:43:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:49 smithi171 ceph-mon[44189]: from='client.27599 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:43:50.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:49 smithi171 ceph-mon[44189]: osdmap e789: 12 total, 11 up, 9 in 2024-04-20T17:43:50.218 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:43:50.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:50.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:43:50.323 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:43:50.350 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:50.371 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:43:50.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:50.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:43:50.415 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 22 16 60518 60502 171.834 126.438 0.00702041 0.00581787 2024-04-20T17:43:50.474 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:43:50.501 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:50.521 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:43:50.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:50.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:43:50.625 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:43:50.680 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:50.705 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:43:50.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:50.709 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:43:50.809 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:43:50.836 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:50.855 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:43:50.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:50.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:43:50.959 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:43:50.994 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:51.203 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:43:51.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:51.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:43:51.308 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:43:51.358 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:51.390 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:43:51.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:51.397 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:43:51.416 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 23 16 62411 62395 169.5 118.312 0.0113661 0.00589776 2024-04-20T17:43:51.498 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:43:51.553 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:51.585 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:43:51.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:51.590 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:43:51.691 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:43:51.724 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:51.802 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:43:51.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:51.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:43:51.829 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:43:51.830 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:43:51.907 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:43:51.935 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:51 smithi005 ceph-mon[39562]: pgmap v2111: 17 pgs: 17 active+clean; 3.5 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-04-20T17:43:51.936 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:51 smithi005 ceph-mon[39562]: osdmap e790: 12 total, 11 up, 9 in 2024-04-20T17:43:51.938 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:51.970 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:43:51.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:51.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:43:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:51 smithi106 ceph-mon[44423]: pgmap v2111: 17 pgs: 17 active+clean; 3.5 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-04-20T17:43:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:51 smithi106 ceph-mon[44423]: osdmap e790: 12 total, 11 up, 9 in 2024-04-20T17:43:52.079 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:43:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:51 smithi171 ceph-mon[44189]: pgmap v2111: 17 pgs: 17 active+clean; 3.5 GiB data, 11 GiB used, 793 GiB / 805 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-04-20T17:43:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:51 smithi171 ceph-mon[44189]: osdmap e790: 12 total, 11 up, 9 in 2024-04-20T17:43:52.115 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:52.142 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:43:52.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:52.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:43:52.245 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:43:52.285 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:52.319 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:43:52.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:52.332 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:43:52.416 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 24 16 64243 64227 167.209 114.5 0.00354102 0.00597844 2024-04-20T17:43:52.433 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:43:52.467 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:52.492 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:43:52.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:52.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:43:52.566 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:43:52.566 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:43:52.566 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:43:52.566 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:43:52.566 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.566 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:155961 6.6 6:62ebcb10:::benchmark_data_smithi171_147972_object155960:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e646)", 2024-04-20T17:43:52.566 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:35:04.114739+0000", 2024-04-20T17:43:52.566 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 528.44855145700001, 2024-04-20T17:43:52.566 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.124179953, 2024-04-20T17:43:52.567 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:43:52.567 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:43:52.567 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:43:52.567 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:43:52.567 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:43:52.567 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:43:52.567 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 155961 2024-04-20T17:43:52.567 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.567 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:43:52.567 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.567 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:43:52.568 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:04.114739+0000", 2024-04-20T17:43:52.568 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.568 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.568 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.568 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:43:52.568 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:04.114739+0000", 2024-04-20T17:43:52.568 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.568 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.568 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.568 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:43:52.568 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:04.114740+0000", 2024-04-20T17:43:52.568 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.046e-06 2024-04-20T17:43:52.568 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.569 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:43:52.569 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:04.114785+0000", 2024-04-20T17:43:52.569 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5170999999999998e-05 2024-04-20T17:43:52.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.569 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:43:52.569 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:04.114792+0000", 2024-04-20T17:43:52.569 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5540000000000003e-06 2024-04-20T17:43:52.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.570 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:43:52.570 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:04.114800+0000", 2024-04-20T17:43:52.570 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4549999999999995e-06 2024-04-20T17:43:52.570 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.570 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.570 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:43:52.570 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:04.114848+0000", 2024-04-20T17:43:52.570 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7352999999999999e-05 2024-04-20T17:43:52.570 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.570 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.570 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:43:52.571 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:04.114962+0000", 2024-04-20T17:43:52.571 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011466700000000001 2024-04-20T17:43:52.571 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.571 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.571 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 4", 2024-04-20T17:43:52.571 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:04.115090+0000", 2024-04-20T17:43:52.571 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000127835 2024-04-20T17:43:52.571 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.571 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.571 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:43:52.571 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:04.120513+0000", 2024-04-20T17:43:52.571 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0054230579999999997 2024-04-20T17:43:52.571 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.572 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:43:52.572 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:04.238796+0000", 2024-04-20T17:43:52.572 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.118282963 2024-04-20T17:43:52.572 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.572 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:43:52.572 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:04.238852+0000", 2024-04-20T17:43:52.572 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5179000000000001e-05 2024-04-20T17:43:52.572 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.572 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:43:52.573 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:04.238919+0000", 2024-04-20T17:43:52.573 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7671999999999999e-05 2024-04-20T17:43:52.573 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.573 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:43:52.573 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.573 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.573 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.573 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:180898 6.4 6:252deb3b:::benchmark_data_smithi171_147972_object19800:head [delete] snapc 0=[] ondisk+write+known_if_redirected e653)", 2024-04-20T17:43:52.573 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:35:26.369439+0000", 2024-04-20T17:43:52.573 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 506.19385194500001, 2024-04-20T17:43:52.574 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.122666505, 2024-04-20T17:43:52.574 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:43:52.574 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:43:52.574 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:43:52.574 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:43:52.574 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:43:52.574 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:43:52.574 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 180898 2024-04-20T17:43:52.574 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.574 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:43:52.574 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.575 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:43:52.575 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.369439+0000", 2024-04-20T17:43:52.575 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.575 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.575 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.575 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:43:52.575 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.369439+0000", 2024-04-20T17:43:52.575 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.575 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.575 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.575 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:43:52.575 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.369441+0000", 2024-04-20T17:43:52.576 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.84e-06 2024-04-20T17:43:52.576 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.576 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.576 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:43:52.576 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.369446+0000", 2024-04-20T17:43:52.576 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9740000000000001e-06 2024-04-20T17:43:52.576 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.576 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.576 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:43:52.576 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.369455+0000", 2024-04-20T17:43:52.576 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7259999999999997e-06 2024-04-20T17:43:52.576 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.577 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.577 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:43:52.577 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.369463+0000", 2024-04-20T17:43:52.577 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3339999999999996e-06 2024-04-20T17:43:52.577 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.577 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.577 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:43:52.577 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.369979+0000", 2024-04-20T17:43:52.577 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00051681600000000002 2024-04-20T17:43:52.577 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.577 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.577 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:43:52.578 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.370063+0000", 2024-04-20T17:43:52.578 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3233999999999996e-05 2024-04-20T17:43:52.578 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.578 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.578 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:43:52.578 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.370129+0000", 2024-04-20T17:43:52.578 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6097000000000001e-05 2024-04-20T17:43:52.578 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.578 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.578 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:43:52.578 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.377632+0000", 2024-04-20T17:43:52.578 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0075031760000000003 2024-04-20T17:43:52.579 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.579 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.579 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:43:52.579 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.491996+0000", 2024-04-20T17:43:52.579 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.114364389 2024-04-20T17:43:52.579 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.579 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.579 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:43:52.579 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492055+0000", 2024-04-20T17:43:52.579 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8598999999999999e-05 2024-04-20T17:43:52.579 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.579 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.579 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:43:52.580 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492105+0000", 2024-04-20T17:43:52.580 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0319999999999999e-05 2024-04-20T17:43:52.580 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.580 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:43:52.580 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.580 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.580 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.580 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:180901 6.c 6:32888237:::benchmark_data_smithi171_147972_object19803:head [delete] snapc 0=[] ondisk+write+known_if_redirected e653)", 2024-04-20T17:43:52.580 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:35:26.369556+0000", 2024-04-20T17:43:52.580 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 506.19373469999999, 2024-04-20T17:43:52.580 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.122677468, 2024-04-20T17:43:52.580 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:43:52.581 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:43:52.581 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:43:52.581 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:43:52.581 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:43:52.581 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:43:52.581 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 180901 2024-04-20T17:43:52.581 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.581 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:43:52.581 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.581 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:43:52.581 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.369556+0000", 2024-04-20T17:43:52.581 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.581 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.581 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.581 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:43:52.582 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.369556+0000", 2024-04-20T17:43:52.582 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.582 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.582 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.582 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:43:52.582 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.369557+0000", 2024-04-20T17:43:52.582 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.474e-06 2024-04-20T17:43:52.582 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.582 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.582 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:43:52.582 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.369562+0000", 2024-04-20T17:43:52.582 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9779999999999996e-06 2024-04-20T17:43:52.582 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.583 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.583 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:43:52.583 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.369573+0000", 2024-04-20T17:43:52.583 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0849000000000001e-05 2024-04-20T17:43:52.583 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.583 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.583 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:43:52.583 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.369581+0000", 2024-04-20T17:43:52.583 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0590000000000003e-06 2024-04-20T17:43:52.583 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.583 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.583 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:43:52.583 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.370067+0000", 2024-04-20T17:43:52.584 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000485198 2024-04-20T17:43:52.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.584 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:43:52.584 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.370167+0000", 2024-04-20T17:43:52.584 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000100283 2024-04-20T17:43:52.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.584 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:43:52.584 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.370269+0000", 2024-04-20T17:43:52.584 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000102538 2024-04-20T17:43:52.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.585 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:43:52.585 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.491974+0000", 2024-04-20T17:43:52.585 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.12170478899999999 2024-04-20T17:43:52.585 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.585 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.585 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:43:52.585 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492152+0000", 2024-04-20T17:43:52.585 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00017778499999999999 2024-04-20T17:43:52.585 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.585 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.585 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:43:52.585 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492203+0000", 2024-04-20T17:43:52.585 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1306e-05 2024-04-20T17:43:52.586 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.586 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.586 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:43:52.586 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492233+0000", 2024-04-20T17:43:52.586 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0209000000000002e-05 2024-04-20T17:43:52.586 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.586 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:43:52.586 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.586 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.586 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.586 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:180906 6.4 6:251cc9ba:::benchmark_data_smithi171_147972_object19808:head [delete] snapc 0=[] ondisk+write+known_if_redirected e653)", 2024-04-20T17:43:52.586 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:35:26.370225+0000", 2024-04-20T17:43:52.586 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 506.19306544199998, 2024-04-20T17:43:52.587 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.122022784, 2024-04-20T17:43:52.587 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:43:52.587 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:43:52.587 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:43:52.587 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:43:52.587 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:43:52.587 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:43:52.587 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 180906 2024-04-20T17:43:52.587 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.587 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:43:52.587 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.587 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:43:52.587 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.370225+0000", 2024-04-20T17:43:52.588 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.588 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.588 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.588 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:43:52.588 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.370225+0000", 2024-04-20T17:43:52.588 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.588 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.588 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.588 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:43:52.588 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.370226+0000", 2024-04-20T17:43:52.588 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1930000000000001e-06 2024-04-20T17:43:52.588 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.588 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.589 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:43:52.589 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.370231+0000", 2024-04-20T17:43:52.589 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4290000000000003e-06 2024-04-20T17:43:52.589 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.589 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.589 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:43:52.589 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.370240+0000", 2024-04-20T17:43:52.589 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7159999999999992e-06 2024-04-20T17:43:52.589 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.589 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.589 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:43:52.589 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.370247+0000", 2024-04-20T17:43:52.589 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0790000000000003e-06 2024-04-20T17:43:52.589 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.590 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.590 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:43:52.590 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.373809+0000", 2024-04-20T17:43:52.590 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0035624239999999998 2024-04-20T17:43:52.590 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.590 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.590 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:43:52.590 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.373896+0000", 2024-04-20T17:43:52.590 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6737999999999995e-05 2024-04-20T17:43:52.590 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.590 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.590 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:43:52.590 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.373972+0000", 2024-04-20T17:43:52.591 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6013999999999994e-05 2024-04-20T17:43:52.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.591 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.591 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:43:52.591 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492127+0000", 2024-04-20T17:43:52.591 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.118154907 2024-04-20T17:43:52.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.591 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.591 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:43:52.591 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492165+0000", 2024-04-20T17:43:52.591 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8161999999999998e-05 2024-04-20T17:43:52.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.591 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.592 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:43:52.592 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492207+0000", 2024-04-20T17:43:52.592 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2592999999999997e-05 2024-04-20T17:43:52.592 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.592 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.592 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:43:52.592 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492248+0000", 2024-04-20T17:43:52.592 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0528999999999998e-05 2024-04-20T17:43:52.592 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.592 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:43:52.592 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.592 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.592 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.592 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.26882.0:180914 6.c 6:383ce87c:::benchmark_data_smithi171_147972_object19816:head [delete] snapc 0=[] ondisk+write+known_if_redirected e653)", 2024-04-20T17:43:52.593 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:35:26.382615+0000", 2024-04-20T17:43:52.593 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 506.18067554300001, 2024-04-20T17:43:52.593 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.125150133, 2024-04-20T17:43:52.593 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:43:52.593 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:43:52.593 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:43:52.593 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:43:52.593 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.26882", 2024-04-20T17:43:52.593 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:43:52.593 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 180914 2024-04-20T17:43:52.593 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.593 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:43:52.593 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.593 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:43:52.594 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.382615+0000", 2024-04-20T17:43:52.594 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.594 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.594 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.594 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:43:52.594 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.382615+0000", 2024-04-20T17:43:52.594 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.594 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.594 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.594 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:43:52.594 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.382617+0000", 2024-04-20T17:43:52.595 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9470000000000002e-06 2024-04-20T17:43:52.595 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.595 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:43:52.595 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.382624+0000", 2024-04-20T17:43:52.595 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6189999999999996e-06 2024-04-20T17:43:52.595 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.595 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:43:52.595 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.487002+0000", 2024-04-20T17:43:52.595 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.104378027 2024-04-20T17:43:52.595 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:43:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.487022+0000", 2024-04-20T17:43:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0659999999999999e-05 2024-04-20T17:43:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:43:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492412+0000", 2024-04-20T17:43:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0053898410000000002 2024-04-20T17:43:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:43:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492527+0000", 2024-04-20T17:43:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011448 2024-04-20T17:43:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.597 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:43:52.597 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.492618+0000", 2024-04-20T17:43:52.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1150000000000007e-05 2024-04-20T17:43:52.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.597 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:43:52.597 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.507586+0000", 2024-04-20T17:43:52.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014967694 2024-04-20T17:43:52.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.597 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:43:52.598 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.507667+0000", 2024-04-20T17:43:52.598 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1562000000000003e-05 2024-04-20T17:43:52.598 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.598 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.598 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:43:52.598 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.507721+0000", 2024-04-20T17:43:52.598 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3615000000000001e-05 2024-04-20T17:43:52.598 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.598 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.598 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:43:52.598 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:35:26.507765+0000", 2024-04-20T17:43:52.598 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4537999999999999e-05 2024-04-20T17:43:52.598 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.598 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:43:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:87138 7.c 7:3ac78cb4:::benchmark_data_smithi171_161871_object87137:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e709)", 2024-04-20T17:43:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:38:43.842031+0000", 2024-04-20T17:43:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 308.72125934600001, 2024-04-20T17:43:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.157860996, 2024-04-20T17:43:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:43:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:43:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:43:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:43:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:43:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:43:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 87138 2024-04-20T17:43:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:43:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:43:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.842031+0000", 2024-04-20T17:43:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:43:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.842031+0000", 2024-04-20T17:43:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:43:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.842033+0000", 2024-04-20T17:43:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5990000000000001e-06 2024-04-20T17:43:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:43:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.842124+0000", 2024-04-20T17:43:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0934999999999996e-05 2024-04-20T17:43:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:43:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.842131+0000", 2024-04-20T17:43:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5340000000000002e-06 2024-04-20T17:43:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:43:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.842137+0000", 2024-04-20T17:43:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6559999999999997e-06 2024-04-20T17:43:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:43:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.842180+0000", 2024-04-20T17:43:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3111000000000001e-05 2024-04-20T17:43:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:43:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.842267+0000", 2024-04-20T17:43:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7250999999999995e-05 2024-04-20T17:43:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 11", 2024-04-20T17:43:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.842365+0000", 2024-04-20T17:43:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7938999999999999e-05 2024-04-20T17:43:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:43:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.848102+0000", 2024-04-20T17:43:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0057371260000000004 2024-04-20T17:43:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:43:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.999815+0000", 2024-04-20T17:43:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.15171239 2024-04-20T17:43:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:43:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.999857+0000", 2024-04-20T17:43:52.605 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2208e-05 2024-04-20T17:43:52.605 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.605 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.605 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:43:52.605 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.999892+0000", 2024-04-20T17:43:52.605 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5247000000000003e-05 2024-04-20T17:43:52.605 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.605 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:43:52.605 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.605 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.605 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.605 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:87141 7.c 7:38ce718a:::benchmark_data_smithi171_161871_object87140:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e709)", 2024-04-20T17:43:52.606 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:38:43.843661+0000", 2024-04-20T17:43:52.606 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 308.719630096, 2024-04-20T17:43:52.606 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17321147000000001, 2024-04-20T17:43:52.606 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:43:52.606 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:43:52.606 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:43:52.606 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:43:52.606 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:43:52.606 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:43:52.606 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 87141 2024-04-20T17:43:52.606 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:43:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:43:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.843661+0000", 2024-04-20T17:43:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:43:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.843661+0000", 2024-04-20T17:43:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.608 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:43:52.608 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.843662+0000", 2024-04-20T17:43:52.608 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.556e-06 2024-04-20T17:43:52.608 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.608 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.608 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:43:52.608 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.843721+0000", 2024-04-20T17:43:52.608 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8448e-05 2024-04-20T17:43:52.608 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.608 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.608 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:43:52.608 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.843737+0000", 2024-04-20T17:43:52.608 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6184999999999998e-05 2024-04-20T17:43:52.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.609 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:43:52.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.843744+0000", 2024-04-20T17:43:52.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5140000000000004e-06 2024-04-20T17:43:52.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.609 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:43:52.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.845099+0000", 2024-04-20T17:43:52.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001354926 2024-04-20T17:43:52.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.610 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:43:52.610 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.845219+0000", 2024-04-20T17:43:52.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011979799999999999 2024-04-20T17:43:52.610 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.610 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.610 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 11", 2024-04-20T17:43:52.610 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.845357+0000", 2024-04-20T17:43:52.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013789 2024-04-20T17:43:52.610 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.610 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.610 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:43:52.610 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.848603+0000", 2024-04-20T17:43:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0032464669999999998 2024-04-20T17:43:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:43:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.016772+0000", 2024-04-20T17:43:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.168168228 2024-04-20T17:43:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:43:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.016829+0000", 2024-04-20T17:43:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7497999999999997e-05 2024-04-20T17:43:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:43:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.016872+0000", 2024-04-20T17:43:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2960000000000002e-05 2024-04-20T17:43:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:43:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:87157 7.c 7:32ba4a0c:::benchmark_data_smithi171_161871_object87156:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e709)", 2024-04-20T17:43:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:38:43.854781+0000", 2024-04-20T17:43:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 308.70850985599998, 2024-04-20T17:43:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.162197069, 2024-04-20T17:43:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:43:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:43:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:43:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:43:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:43:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:43:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 87157 2024-04-20T17:43:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:43:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:43:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.854781+0000", 2024-04-20T17:43:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:43:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.854781+0000", 2024-04-20T17:43:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:43:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.854783+0000", 2024-04-20T17:43:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.697e-06 2024-04-20T17:43:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:43:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.854833+0000", 2024-04-20T17:43:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0590000000000002e-05 2024-04-20T17:43:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:43:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.854844+0000", 2024-04-20T17:43:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1025e-05 2024-04-20T17:43:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:43:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.854850+0000", 2024-04-20T17:43:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2500000000000003e-06 2024-04-20T17:43:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:43:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.854905+0000", 2024-04-20T17:43:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4215999999999997e-05 2024-04-20T17:43:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:43:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.855018+0000", 2024-04-20T17:43:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011294900000000001 2024-04-20T17:43:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 11", 2024-04-20T17:43:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.855112+0000", 2024-04-20T17:43:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4140000000000003e-05 2024-04-20T17:43:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:43:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.860376+0000", 2024-04-20T17:43:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0052640559999999996 2024-04-20T17:43:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:43:52.618 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.016911+0000", 2024-04-20T17:43:52.618 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.15653561399999999 2024-04-20T17:43:52.618 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.618 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:43:52.618 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.016942+0000", 2024-04-20T17:43:52.618 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0648999999999997e-05 2024-04-20T17:43:52.618 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.618 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:43:52.618 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.016978+0000", 2024-04-20T17:43:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5883000000000002e-05 2024-04-20T17:43:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:43:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:87160 7.c 7:39d2f99c:::benchmark_data_smithi171_161871_object87159:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e709)", 2024-04-20T17:43:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:38:43.855240+0000", 2024-04-20T17:43:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 308.70805070300003, 2024-04-20T17:43:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.161860333, 2024-04-20T17:43:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:43:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:43:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:43:52.620 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:43:52.620 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:43:52.620 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:43:52.620 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 87160 2024-04-20T17:43:52.620 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.620 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:43:52.620 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.620 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:43:52.620 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.855240+0000", 2024-04-20T17:43:52.620 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.620 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.620 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.620 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:43:52.621 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.855240+0000", 2024-04-20T17:43:52.621 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.621 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.621 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.621 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:43:52.621 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.855241+0000", 2024-04-20T17:43:52.621 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.37e-06 2024-04-20T17:43:52.621 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.621 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.621 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:43:52.621 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.855344+0000", 2024-04-20T17:43:52.621 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000102588 2024-04-20T17:43:52.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.622 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:43:52.622 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.855356+0000", 2024-04-20T17:43:52.622 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1878e-05 2024-04-20T17:43:52.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.622 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:43:52.622 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.855363+0000", 2024-04-20T17:43:52.622 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5599999999999996e-06 2024-04-20T17:43:52.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.623 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:43:52.623 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.857825+0000", 2024-04-20T17:43:52.623 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0024618130000000002 2024-04-20T17:43:52.623 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.623 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:43:52.623 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.857952+0000", 2024-04-20T17:43:52.623 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000126687 2024-04-20T17:43:52.623 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.623 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 11", 2024-04-20T17:43:52.623 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.858097+0000", 2024-04-20T17:43:52.623 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014494900000000001 2024-04-20T17:43:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:43:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.862559+0000", 2024-04-20T17:43:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0044618080000000003 2024-04-20T17:43:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:43:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.017016+0000", 2024-04-20T17:43:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.15445716800000001 2024-04-20T17:43:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:43:52.625 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.017064+0000", 2024-04-20T17:43:52.625 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.808e-05 2024-04-20T17:43:52.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.625 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.625 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:43:52.625 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.017100+0000", 2024-04-20T17:43:52.625 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6431999999999999e-05 2024-04-20T17:43:52.625 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.625 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:43:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:87165 7.c 7:3335e632:::benchmark_data_smithi171_161871_object87164:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e709)", 2024-04-20T17:43:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:38:43.858382+0000", 2024-04-20T17:43:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 308.70490877100002, 2024-04-20T17:43:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.158889949, 2024-04-20T17:43:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:43:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:43:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:43:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:43:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:43:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:43:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 87165 2024-04-20T17:43:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:43:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:43:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.858382+0000", 2024-04-20T17:43:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:43:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.858382+0000", 2024-04-20T17:43:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.628 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.628 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.628 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:43:52.628 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.858383+0000", 2024-04-20T17:43:52.628 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.336e-06 2024-04-20T17:43:52.628 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.628 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.628 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:43:52.628 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.858439+0000", 2024-04-20T17:43:52.628 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6113999999999998e-05 2024-04-20T17:43:52.628 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.628 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.629 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:43:52.629 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.858451+0000", 2024-04-20T17:43:52.629 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1439e-05 2024-04-20T17:43:52.629 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.629 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.629 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:43:52.629 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.858457+0000", 2024-04-20T17:43:52.629 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9789999999999998e-06 2024-04-20T17:43:52.629 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.629 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.629 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:43:52.629 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.860407+0000", 2024-04-20T17:43:52.630 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0019500489999999999 2024-04-20T17:43:52.630 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.630 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.630 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:43:52.630 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.860490+0000", 2024-04-20T17:43:52.630 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3089000000000004e-05 2024-04-20T17:43:52.630 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.630 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.630 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 11", 2024-04-20T17:43:52.630 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.860592+0000", 2024-04-20T17:43:52.630 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010162000000000001 2024-04-20T17:43:52.630 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.631 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.631 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:43:52.631 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.865894+0000", 2024-04-20T17:43:52.631 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0053022 2024-04-20T17:43:52.631 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.631 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.631 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:43:52.631 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.017155+0000", 2024-04-20T17:43:52.631 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.15126082299999999 2024-04-20T17:43:52.631 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.631 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.631 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:43:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.017229+0000", 2024-04-20T17:43:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4740000000000006e-05 2024-04-20T17:43:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:43:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.017272+0000", 2024-04-20T17:43:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2559999999999999e-05 2024-04-20T17:43:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:43:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.633 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:87169 7.c 7:349f0b61:::benchmark_data_smithi171_161871_object87168:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e709)", 2024-04-20T17:43:52.633 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:38:43.861700+0000", 2024-04-20T17:43:52.633 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 308.70159097800001, 2024-04-20T17:43:52.633 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.15573498599999999, 2024-04-20T17:43:52.633 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:43:52.633 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:43:52.633 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:43:52.633 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:43:52.633 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:43:52.633 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:43:52.633 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 87169 2024-04-20T17:43:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:43:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:43:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.861700+0000", 2024-04-20T17:43:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:43:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.861700+0000", 2024-04-20T17:43:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.635 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:43:52.635 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.861702+0000", 2024-04-20T17:43:52.635 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0219999999999999e-06 2024-04-20T17:43:52.635 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.635 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.635 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:43:52.635 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.861754+0000", 2024-04-20T17:43:52.635 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2176999999999999e-05 2024-04-20T17:43:52.635 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.635 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.635 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:43:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.861767+0000", 2024-04-20T17:43:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2913e-05 2024-04-20T17:43:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:43:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.861776+0000", 2024-04-20T17:43:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7080000000000003e-06 2024-04-20T17:43:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:43:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.862591+0000", 2024-04-20T17:43:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00081536800000000002 2024-04-20T17:43:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:43:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.862711+0000", 2024-04-20T17:43:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012045900000000001 2024-04-20T17:43:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 11", 2024-04-20T17:43:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.862797+0000", 2024-04-20T17:43:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5694999999999999e-05 2024-04-20T17:43:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:43:52.638 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.865905+0000", 2024-04-20T17:43:52.638 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0031081699999999999 2024-04-20T17:43:52.638 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.638 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.638 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:43:52.638 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.017346+0000", 2024-04-20T17:43:52.638 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.15144074799999999 2024-04-20T17:43:52.638 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.638 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.638 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:43:52.638 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.017396+0000", 2024-04-20T17:43:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0164000000000003e-05 2024-04-20T17:43:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:43:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.017435+0000", 2024-04-20T17:43:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8562000000000001e-05 2024-04-20T17:43:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:43:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:87171 7.c 7:32893c60:::benchmark_data_smithi171_161871_object87170:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e709)", 2024-04-20T17:43:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:38:43.862463+0000", 2024-04-20T17:43:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 308.70082776499999, 2024-04-20T17:43:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.155096917, 2024-04-20T17:43:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:43:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:43:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:43:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:43:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:43:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:43:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 87171 2024-04-20T17:43:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:43:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:43:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.862463+0000", 2024-04-20T17:43:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:43:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.862463+0000", 2024-04-20T17:43:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:43:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.862464+0000", 2024-04-20T17:43:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.516e-06 2024-04-20T17:43:52.642 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.642 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.642 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:43:52.642 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.862530+0000", 2024-04-20T17:43:52.642 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5980000000000002e-05 2024-04-20T17:43:52.642 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.642 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.642 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:43:52.642 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.862541+0000", 2024-04-20T17:43:52.642 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0900999999999999e-05 2024-04-20T17:43:52.642 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.642 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:43:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.862549+0000", 2024-04-20T17:43:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8590000000000005e-06 2024-04-20T17:43:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:43:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.864448+0000", 2024-04-20T17:43:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0018989020000000001 2024-04-20T17:43:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:43:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.864526+0000", 2024-04-20T17:43:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7631000000000003e-05 2024-04-20T17:43:52.644 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.644 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.644 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 11", 2024-04-20T17:43:52.644 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.864615+0000", 2024-04-20T17:43:52.644 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9480000000000004e-05 2024-04-20T17:43:52.644 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.644 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.644 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:43:52.644 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.866027+0000", 2024-04-20T17:43:52.644 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001411762 2024-04-20T17:43:52.644 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.644 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:43:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.017476+0000", 2024-04-20T17:43:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.151448894 2024-04-20T17:43:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:43:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.017523+0000", 2024-04-20T17:43:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7114000000000003e-05 2024-04-20T17:43:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:43:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.017560+0000", 2024-04-20T17:43:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6878000000000002e-05 2024-04-20T17:43:52.646 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.646 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:43:52.646 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.646 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.646 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.646 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:256774 7.3 7:ce830d86:::benchmark_data_smithi171_161871_object111048:head [delete] snapc 0=[] ondisk+write+known_if_redirected e756)", 2024-04-20T17:43:52.646 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:42:18.833533+0000", 2024-04-20T17:43:52.646 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 93.729757442999997, 2024-04-20T17:43:52.646 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.12193923500000001, 2024-04-20T17:43:52.646 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:43:52.646 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:43:52.646 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:43:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:43:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:43:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:43:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 256774 2024-04-20T17:43:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:43:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:43:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:18.833533+0000", 2024-04-20T17:43:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:43:52.648 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:18.833533+0000", 2024-04-20T17:43:52.648 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.648 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.648 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.648 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:43:52.648 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:18.833535+0000", 2024-04-20T17:43:52.648 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4050000000000001e-06 2024-04-20T17:43:52.648 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.648 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.648 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:43:52.648 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:18.833539+0000", 2024-04-20T17:43:52.649 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7729999999999999e-06 2024-04-20T17:43:52.649 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.649 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.649 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:43:52.649 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:18.833549+0000", 2024-04-20T17:43:52.649 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6860000000000001e-06 2024-04-20T17:43:52.649 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.649 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.649 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:43:52.649 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:18.833556+0000", 2024-04-20T17:43:52.649 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1550000000000001e-06 2024-04-20T17:43:52.649 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.650 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.650 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:43:52.650 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:18.833611+0000", 2024-04-20T17:43:52.650 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4438999999999998e-05 2024-04-20T17:43:52.650 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.650 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.650 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:43:52.650 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:18.833720+0000", 2024-04-20T17:43:52.650 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000109176 2024-04-20T17:43:52.650 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.650 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 2", 2024-04-20T17:43:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:18.833822+0000", 2024-04-20T17:43:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000102199 2024-04-20T17:43:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:43:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:18.950013+0000", 2024-04-20T17:43:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.116190796 2024-04-20T17:43:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:43:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:18.955384+0000", 2024-04-20T17:43:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0053714890000000001 2024-04-20T17:43:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:43:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:18.955429+0000", 2024-04-20T17:43:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4410999999999999e-05 2024-04-20T17:43:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:43:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:18.955472+0000", 2024-04-20T17:43:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3705999999999998e-05 2024-04-20T17:43:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:43:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27460.0:56599 8.0 8:03243b42:::benchmark_data_smithi171_175757_object56598:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e788)", 2024-04-20T17:43:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:43:48.406971+0000", 2024-04-20T17:43:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 4.156319817, 2024-04-20T17:43:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013118110000000001, 2024-04-20T17:43:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:43:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:43:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:43:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:43:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27460", 2024-04-20T17:43:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:43:52.654 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 56599 2024-04-20T17:43:52.654 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.654 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:43:52.654 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.654 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:43:52.654 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.406971+0000", 2024-04-20T17:43:52.654 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.654 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.654 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.654 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:43:52.654 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.406971+0000", 2024-04-20T17:43:52.654 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.655 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.655 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.655 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:43:52.655 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.406974+0000", 2024-04-20T17:43:52.655 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2949999999999998e-06 2024-04-20T17:43:52.655 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.655 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.655 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:43:52.655 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.407019+0000", 2024-04-20T17:43:52.655 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4481000000000003e-05 2024-04-20T17:43:52.655 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.655 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.656 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:43:52.656 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.407044+0000", 2024-04-20T17:43:52.656 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5809e-05 2024-04-20T17:43:52.656 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.656 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.656 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:43:52.656 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.407056+0000", 2024-04-20T17:43:52.656 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1313e-05 2024-04-20T17:43:52.656 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.656 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.656 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:43:52.656 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.408176+0000", 2024-04-20T17:43:52.656 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001119735 2024-04-20T17:43:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:43:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.408369+0000", 2024-04-20T17:43:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000193236 2024-04-20T17:43:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:43:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.408534+0000", 2024-04-20T17:43:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00016501100000000001 2024-04-20T17:43:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:43:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.419913+0000", 2024-04-20T17:43:52.658 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011379439999999999 2024-04-20T17:43:52.658 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.658 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.658 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:43:52.658 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.419972+0000", 2024-04-20T17:43:52.658 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8758000000000002e-05 2024-04-20T17:43:52.658 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.658 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.658 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:43:52.658 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.420044+0000", 2024-04-20T17:43:52.658 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2204000000000002e-05 2024-04-20T17:43:52.658 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.658 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.659 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:43:52.659 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.420089+0000", 2024-04-20T17:43:52.659 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4827999999999998e-05 2024-04-20T17:43:52.659 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.659 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:43:52.659 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.659 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.659 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.659 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27460.0:56605 8.0 8:0a3e5cf7:::benchmark_data_smithi171_175757_object56604:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e788)", 2024-04-20T17:43:52.659 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:43:48.407086+0000", 2024-04-20T17:43:52.659 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 4.1562045120000004, 2024-04-20T17:43:52.659 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013349158999999999, 2024-04-20T17:43:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:43:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:43:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:43:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:43:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27460", 2024-04-20T17:43:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:43:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 56605 2024-04-20T17:43:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:43:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:43:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.407086+0000", 2024-04-20T17:43:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:43:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.407086+0000", 2024-04-20T17:43:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:43:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.407087+0000", 2024-04-20T17:43:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.3200000000000003e-07 2024-04-20T17:43:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.662 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:43:52.662 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.407123+0000", 2024-04-20T17:43:52.662 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5509000000000002e-05 2024-04-20T17:43:52.662 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.662 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.662 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:43:52.662 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.407134+0000", 2024-04-20T17:43:52.662 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.173e-05 2024-04-20T17:43:52.662 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.662 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.662 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:43:52.663 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.407141+0000", 2024-04-20T17:43:52.663 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7649999999999996e-06 2024-04-20T17:43:52.663 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.663 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.663 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:43:52.663 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.410578+0000", 2024-04-20T17:43:52.663 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0034370270000000001 2024-04-20T17:43:52.663 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.663 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.663 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:43:52.663 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.410729+0000", 2024-04-20T17:43:52.663 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000150552 2024-04-20T17:43:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:43:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.410841+0000", 2024-04-20T17:43:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011205300000000001 2024-04-20T17:43:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:43:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.419920+0000", 2024-04-20T17:43:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0090788359999999999 2024-04-20T17:43:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:43:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.420312+0000", 2024-04-20T17:43:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00039261999999999999 2024-04-20T17:43:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:43:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.420368+0000", 2024-04-20T17:43:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5909999999999998e-05 2024-04-20T17:43:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:43:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.420435+0000", 2024-04-20T17:43:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7224999999999994e-05 2024-04-20T17:43:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.666 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:43:52.666 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.666 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.666 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.666 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27460.0:57095 8.0 8:087ea4e1:::benchmark_data_smithi171_175757_object57094:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e789)", 2024-04-20T17:43:52.666 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:43:48.650801+0000", 2024-04-20T17:43:52.666 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 3.9124895849999999, 2024-04-20T17:43:52.666 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014433769000000001, 2024-04-20T17:43:52.666 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:43:52.666 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:43:52.666 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:43:52.666 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:43:52.667 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27460", 2024-04-20T17:43:52.667 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:43:52.667 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 57095 2024-04-20T17:43:52.667 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.667 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:43:52.667 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.667 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:43:52.667 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.650801+0000", 2024-04-20T17:43:52.667 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.667 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.667 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.667 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:43:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.650801+0000", 2024-04-20T17:43:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:43:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.650802+0000", 2024-04-20T17:43:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.105e-06 2024-04-20T17:43:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:43:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.650894+0000", 2024-04-20T17:43:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1757999999999998e-05 2024-04-20T17:43:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:43:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.650902+0000", 2024-04-20T17:43:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8830000000000006e-06 2024-04-20T17:43:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:43:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.650908+0000", 2024-04-20T17:43:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3319999999999997e-06 2024-04-20T17:43:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.670 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:43:52.670 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.652790+0000", 2024-04-20T17:43:52.670 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00188193 2024-04-20T17:43:52.670 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.670 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.670 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:43:52.670 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.652877+0000", 2024-04-20T17:43:52.670 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6920000000000001e-05 2024-04-20T17:43:52.670 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.670 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.670 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:43:52.670 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.652957+0000", 2024-04-20T17:43:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0426000000000006e-05 2024-04-20T17:43:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:43:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.662272+0000", 2024-04-20T17:43:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0093147899999999999 2024-04-20T17:43:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:43:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.665124+0000", 2024-04-20T17:43:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0028517970000000001 2024-04-20T17:43:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:43:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.665182+0000", 2024-04-20T17:43:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7889000000000001e-05 2024-04-20T17:43:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:43:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.665235+0000", 2024-04-20T17:43:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2939000000000003e-05 2024-04-20T17:43:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:43:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27460.0:57099 8.0 8:0f42466b:::benchmark_data_smithi171_175757_object57098:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e789)", 2024-04-20T17:43:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:43:48.650926+0000", 2024-04-20T17:43:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 3.9123642199999997, 2024-04-20T17:43:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014583555999999999, 2024-04-20T17:43:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:43:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:43:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:43:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:43:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27460", 2024-04-20T17:43:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:43:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 57099 2024-04-20T17:43:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.674 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:43:52.674 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.674 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:43:52.674 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.650926+0000", 2024-04-20T17:43:52.674 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.674 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.674 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.674 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:43:52.674 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.650926+0000", 2024-04-20T17:43:52.674 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.674 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:43:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.650927+0000", 2024-04-20T17:43:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3200000000000004e-07 2024-04-20T17:43:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:43:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.650964+0000", 2024-04-20T17:43:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6325999999999997e-05 2024-04-20T17:43:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:43:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.650971+0000", 2024-04-20T17:43:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3080000000000002e-06 2024-04-20T17:43:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:43:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.650978+0000", 2024-04-20T17:43:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8820000000000003e-06 2024-04-20T17:43:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:43:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.654517+0000", 2024-04-20T17:43:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0035394379999999998 2024-04-20T17:43:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:43:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.654644+0000", 2024-04-20T17:43:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000127055 2024-04-20T17:43:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:43:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.654773+0000", 2024-04-20T17:43:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012837600000000001 2024-04-20T17:43:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:43:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.662287+0000", 2024-04-20T17:43:52.678 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0075146750000000002 2024-04-20T17:43:52.678 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.678 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.678 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:43:52.678 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.665421+0000", 2024-04-20T17:43:52.678 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0031335619999999999 2024-04-20T17:43:52.678 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.678 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.678 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:43:52.678 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.665474+0000", 2024-04-20T17:43:52.678 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3004000000000003e-05 2024-04-20T17:43:52.678 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.679 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.679 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:43:52.679 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.665510+0000", 2024-04-20T17:43:52.679 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6097999999999999e-05 2024-04-20T17:43:52.679 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.679 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:43:52.679 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.679 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.679 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.679 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27460.0:57101 8.8 8:161b143f:::benchmark_data_smithi171_175757_object57100:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e789)", 2024-04-20T17:43:52.679 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:43:48.651274+0000", 2024-04-20T17:43:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 3.9120168670000002, 2024-04-20T17:43:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014451163, 2024-04-20T17:43:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:43:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:43:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:43:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:43:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27460", 2024-04-20T17:43:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:43:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 57101 2024-04-20T17:43:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:43:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:43:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.651274+0000", 2024-04-20T17:43:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:43:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.651274+0000", 2024-04-20T17:43:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:43:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.651275+0000", 2024-04-20T17:43:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.279e-06 2024-04-20T17:43:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:43:52.682 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.651346+0000", 2024-04-20T17:43:52.682 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0532999999999997e-05 2024-04-20T17:43:52.682 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.682 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.682 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:43:52.682 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.651355+0000", 2024-04-20T17:43:52.682 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4439999999999997e-06 2024-04-20T17:43:52.682 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.682 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.682 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:43:52.682 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.651362+0000", 2024-04-20T17:43:52.682 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7039999999999997e-06 2024-04-20T17:43:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:43:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.654499+0000", 2024-04-20T17:43:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0031375750000000001 2024-04-20T17:43:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:43:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.654612+0000", 2024-04-20T17:43:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000112472 2024-04-20T17:43:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:43:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.654787+0000", 2024-04-20T17:43:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000175606 2024-04-20T17:43:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:43:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.662277+0000", 2024-04-20T17:43:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0074898530000000003 2024-04-20T17:43:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:43:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.665676+0000", 2024-04-20T17:43:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0033984250000000001 2024-04-20T17:43:52.685 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.685 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.685 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:43:52.685 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.665703+0000", 2024-04-20T17:43:52.685 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6959000000000001e-05 2024-04-20T17:43:52.685 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.685 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.685 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:43:52.685 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:48.665725+0000", 2024-04-20T17:43:52.685 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2313000000000001e-05 2024-04-20T17:43:52.685 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.685 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:43:52.686 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.686 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.686 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.686 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27460.0:61610 8.0 8:04344d25:::benchmark_data_smithi171_175757_object61609:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e789)", 2024-04-20T17:43:52.686 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:43:50.992217+0000", 2024-04-20T17:43:52.686 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 1.571074053, 2024-04-20T17:43:52.686 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012621416999999999, 2024-04-20T17:43:52.686 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:43:52.686 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:43:52.686 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:43:52.686 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:43:52.686 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27460", 2024-04-20T17:43:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:43:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 61610 2024-04-20T17:43:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:43:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:43:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:50.992217+0000", 2024-04-20T17:43:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:43:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:50.992217+0000", 2024-04-20T17:43:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.688 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.688 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:43:52.688 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:50.992217+0000", 2024-04-20T17:43:52.688 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1200000000000003e-07 2024-04-20T17:43:52.688 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.688 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.688 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:43:52.688 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:50.992250+0000", 2024-04-20T17:43:52.688 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2570999999999997e-05 2024-04-20T17:43:52.688 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.688 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.688 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:43:52.688 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:50.992256+0000", 2024-04-20T17:43:52.688 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2960000000000004e-06 2024-04-20T17:43:52.689 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.689 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.689 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:43:52.689 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:50.992260+0000", 2024-04-20T17:43:52.689 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9009999999999996e-06 2024-04-20T17:43:52.689 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.689 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.689 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:43:52.689 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:50.996394+0000", 2024-04-20T17:43:52.689 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0041335690000000001 2024-04-20T17:43:52.689 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.689 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.689 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:43:52.690 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:50.996537+0000", 2024-04-20T17:43:52.690 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000143759 2024-04-20T17:43:52.690 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.690 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.690 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:43:52.690 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:50.996702+0000", 2024-04-20T17:43:52.690 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000164718 2024-04-20T17:43:52.690 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.690 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.690 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:43:52.690 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:51.002795+0000", 2024-04-20T17:43:52.691 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0060926410000000002 2024-04-20T17:43:52.691 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.691 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.691 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:43:52.691 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:51.004761+0000", 2024-04-20T17:43:52.691 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0019666729999999999 2024-04-20T17:43:52.691 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.691 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.691 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:43:52.691 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:51.004802+0000", 2024-04-20T17:43:52.691 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0289e-05 2024-04-20T17:43:52.691 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.691 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.692 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:43:52.692 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:51.004838+0000", 2024-04-20T17:43:52.692 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6387999999999998e-05 2024-04-20T17:43:52.692 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.692 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:43:52.692 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.692 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.692 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.692 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27460.0:61613 8.0 8:01307e10:::benchmark_data_smithi171_175757_object61612:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e789)", 2024-04-20T17:43:52.692 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:43:50.996160+0000", 2024-04-20T17:43:52.692 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 1.5671304130000001, 2024-04-20T17:43:52.692 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012870919999999999, 2024-04-20T17:43:52.693 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:43:52.693 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:43:52.693 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:43:52.693 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:43:52.693 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27460", 2024-04-20T17:43:52.693 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:43:52.693 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 61613 2024-04-20T17:43:52.693 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.693 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:43:52.693 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.693 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:43:52.693 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:50.996160+0000", 2024-04-20T17:43:52.693 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.694 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.694 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.694 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:43:52.694 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:50.996160+0000", 2024-04-20T17:43:52.694 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:43:52.694 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.694 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.694 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:43:52.694 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:50.996162+0000", 2024-04-20T17:43:52.694 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9860000000000001e-06 2024-04-20T17:43:52.694 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.694 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.695 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:43:52.695 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:50.996297+0000", 2024-04-20T17:43:52.695 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000134509 2024-04-20T17:43:52.695 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.695 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.695 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:43:52.695 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:50.996308+0000", 2024-04-20T17:43:52.695 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0848e-05 2024-04-20T17:43:52.695 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.695 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.695 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:43:52.695 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:50.996317+0000", 2024-04-20T17:43:52.695 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.5990000000000008e-06 2024-04-20T17:43:52.696 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.696 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.696 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:43:52.696 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:50.998546+0000", 2024-04-20T17:43:52.696 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002229197 2024-04-20T17:43:52.696 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.696 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.696 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:43:52.696 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:50.998703+0000", 2024-04-20T17:43:52.696 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015619900000000001 2024-04-20T17:43:52.696 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.696 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.696 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:43:52.697 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:50.998873+0000", 2024-04-20T17:43:52.697 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00016998999999999999 2024-04-20T17:43:52.697 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.697 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.697 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:43:52.697 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:51.002833+0000", 2024-04-20T17:43:52.697 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0039600420000000004 2024-04-20T17:43:52.697 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.697 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.697 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:43:52.697 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:51.008923+0000", 2024-04-20T17:43:52.697 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0060901230000000002 2024-04-20T17:43:52.697 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.698 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.698 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:43:52.698 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:51.008981+0000", 2024-04-20T17:43:52.698 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7955999999999999e-05 2024-04-20T17:43:52.698 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:43:52.698 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:43:52.698 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:43:52.698 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:51.009031+0000", 2024-04-20T17:43:52.698 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0470999999999998e-05 2024-04-20T17:43:52.698 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.698 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:43:52.698 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.698 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:43:52.699 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:43:52.699 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:43:52.699 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:43:52.725 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:52.745 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:43:52.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:52.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:43:52.849 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:43:52.880 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:52.916 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:43:53.301 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-20T17:43:53.336 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:43:53.340 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:43:53.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:53.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:43:53.416 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 25 16 66090 66074 165.138 115.438 0.0135222 0.00605346 2024-04-20T17:43:53.444 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:43:53.483 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:53.520 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:43:53.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:53.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:43:53.630 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:43:53.677 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:53.706 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:43:53.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:53.713 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:43:53.814 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:43:53.831 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:43:53.842 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:53.863 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:53 smithi005 ceph-mon[39562]: pgmap v2113: 17 pgs: 17 active+clean; 3.6 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 105 MiB/s wr, 1.67k op/s 2024-04-20T17:43:53.864 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:43:53.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:53.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:43:53.969 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:43:54.008 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:53 smithi171 ceph-mon[44189]: pgmap v2113: 17 pgs: 17 active+clean; 3.6 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 105 MiB/s wr, 1.67k op/s 2024-04-20T17:43:54.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:53 smithi106 ceph-mon[44423]: pgmap v2113: 17 pgs: 17 active+clean; 3.6 GiB data, 12 GiB used, 793 GiB / 805 GiB avail; 105 MiB/s wr, 1.67k op/s 2024-04-20T17:43:54.012 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:54.045 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:43:54.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:54.053 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:43:54.154 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:43:54.187 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:54.416 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 26 16 67828 67812 162.965 108.625 0.00654206 0.00613492 2024-04-20T17:43:55.416 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 27 16 69651 69635 161.149 113.938 0.00581305 0.0062037 2024-04-20T17:43:55.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:55 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:43:55.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:55 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:43:55.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:55 smithi005 ceph-mon[39562]: pgmap v2114: 17 pgs: 17 active+clean; 3.9 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-04-20T17:43:55.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:55 smithi005 ceph-mon[39562]: osdmap e791: 12 total, 11 up, 9 in 2024-04-20T17:43:55.988 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:43:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:55 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:43:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:55 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:43:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:55 smithi106 ceph-mon[44423]: pgmap v2114: 17 pgs: 17 active+clean; 3.9 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-04-20T17:43:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:55 smithi106 ceph-mon[44423]: osdmap e791: 12 total, 11 up, 9 in 2024-04-20T17:43:56.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:55 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:43:56.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:55 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:43:56.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:55 smithi171 ceph-mon[44189]: pgmap v2114: 17 pgs: 17 active+clean; 3.9 GiB data, 12 GiB used, 792 GiB / 805 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-04-20T17:43:56.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:55 smithi171 ceph-mon[44189]: osdmap e791: 12 total, 11 up, 9 in 2024-04-20T17:43:56.150 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:43:56.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:56.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:43:56.258 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:43:56.294 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:56.327 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:43:56.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:56.334 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:43:56.416 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 28 16 71437 71421 159.38 111.625 0.00543615 0.00627232 2024-04-20T17:43:56.434 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:43:56.470 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:56.495 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:43:56.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:56.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:43:56.602 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:43:56.631 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:56.651 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:43:56.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:56.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:43:56.755 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:43:56.807 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:56.817 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:56 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:43:56.817 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:56 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:43:56.817 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:56 smithi005 ceph-mon[39562]: osdmap e792: 12 total, 11 up, 9 in 2024-04-20T17:43:56.841 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:43:56.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:56.846 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:43:56.947 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:43:56.980 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:56 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:43:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:56 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:43:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:56 smithi106 ceph-mon[44423]: osdmap e792: 12 total, 11 up, 9 in 2024-04-20T17:43:57.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:56 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:43:57.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:56 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:43:57.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:56 smithi171 ceph-mon[44189]: osdmap e792: 12 total, 11 up, 9 in 2024-04-20T17:43:57.088 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:43:57.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:57.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:43:57.193 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:43:57.236 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:57.265 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:43:57.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:57.268 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:43:57.369 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:43:57.413 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:57.417 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 29 16 73134 73118 157.536 106.062 0.00637758 0.00634606 2024-04-20T17:43:57.440 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:43:57.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:57.445 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:43:57.448 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:43:57.546 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:43:57.587 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:57.610 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:43:57.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:57.613 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:43:57.643 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:43:57.714 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:43:57.770 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:57.796 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:43:57.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:57.803 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:43:57.904 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:43:57.952 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:57 smithi171 ceph-mon[44189]: pgmap v2116: 17 pgs: 17 active+clean; 4.2 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 151 MiB/s wr, 2.41k op/s 2024-04-20T17:43:57.952 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:57 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:43:57.952 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:57 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:43:57.953 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:57 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3609341693' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:43:57.955 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:57.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:57 smithi005 ceph-mon[39562]: pgmap v2116: 17 pgs: 17 active+clean; 4.2 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 151 MiB/s wr, 2.41k op/s 2024-04-20T17:43:57.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:57 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:43:57.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:57 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:43:57.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:57 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3609341693' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:43:57.983 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:43:57.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:57.988 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:43:58.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:57 smithi106 ceph-mon[44423]: pgmap v2116: 17 pgs: 17 active+clean; 4.2 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 151 MiB/s wr, 2.41k op/s 2024-04-20T17:43:58.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:57 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:43:58.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:57 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:43:58.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:57 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3609341693' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:43:58.088 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:43:58.136 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:58.166 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:43:58.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:58.168 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:43:58.269 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:43:58.309 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:58.339 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:43:58.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:58.345 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:43:58.417 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 30 16 74712 74696 155.572 98.625 0.00849288 0.00642639 2024-04-20T17:43:58.445 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:43:58.473 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:58.493 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:43:58.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:58.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:43:58.597 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:43:58.637 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:58.666 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:43:58.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:58.670 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:43:58.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:58 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:43:58.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:58 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:43:58.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:58 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:43:58.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:58 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3609341693' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:43:58.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:58 smithi106 ceph-mon[44423]: osdmap e793: 12 total, 11 up, 9 in 2024-04-20T17:43:58.772 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:43:58.799 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:58.825 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:43:58.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:58.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:43:58.929 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:43:58.933 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:58 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:43:58.933 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:58 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:43:58.934 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:58 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:43:58.934 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:58 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3609341693' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:43:58.934 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:58 smithi005 ceph-mon[39562]: osdmap e793: 12 total, 11 up, 9 in 2024-04-20T17:43:58.966 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:58 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:43:58.966 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:58 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:43:58.966 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:58 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:43:58.966 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:58 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3609341693' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:43:58.967 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:58 smithi171 ceph-mon[44189]: osdmap e793: 12 total, 11 up, 9 in 2024-04-20T17:43:58.967 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:58.998 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:43:59.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:59.006 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:43:59.107 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:43:59.152 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:59.181 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:43:59.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:59.186 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:43:59.287 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:43:59.327 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:59.417 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 31 16 76374 76358 153.905 103.875 0.0127639 0.00649572 2024-04-20T17:43:59.453 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:43:59.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:59.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:43:59.560 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:43:59.601 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:59.630 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:43:59.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:59.634 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:43:59.735 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:43:59.765 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:43:59.894 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:43:59.895 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:43:59 smithi005 ceph-mon[39562]: pgmap v2119: 17 pgs: 17 active+clean; 4.2 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-04-20T17:43:59.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:43:59.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:43:59.999 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:44:00.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:43:59 smithi106 ceph-mon[44423]: pgmap v2119: 17 pgs: 17 active+clean; 4.2 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-04-20T17:44:00.028 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:00.047 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:44:00.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:00.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:44:00.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:43:59 smithi171 ceph-mon[44189]: pgmap v2119: 17 pgs: 17 active+clean; 4.2 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-04-20T17:44:00.151 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:44:00.191 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:00.219 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:44:00.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:00.224 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:44:00.325 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:44:00.370 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:00.398 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:44:00.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:00.404 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:44:00.417 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 32 16 77906 77890 152.087 95.75 0.00356128 0.00657393 2024-04-20T17:44:00.504 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:44:00.556 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:00.590 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:44:00.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:00.598 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:44:00.627 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:44:00.698 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:44:00.750 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:00.781 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:44:00.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:00.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:44:00.812 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:44:00.812 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:44:00.812 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-04-20T17:44:00.812 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:44:00.886 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:44:00.926 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:01.286 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:44:01.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:01.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:44:01.391 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:44:01.417 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 33 16 79529 79513 150.553 101.438 0.0038268 0.00664043 2024-04-20T17:44:01.433 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:01.457 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:44:01.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:01.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:44:01.564 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:44:01.596 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:01.618 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:44:01.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:01.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:44:01.645 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:44:01.699 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:44:01.722 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:44:01.724 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:44:01.749 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:44:01.762 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:01.781 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:44:01.795 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:44:01.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:01.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:44:01.810 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:44:01.832 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:44:01.853 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:44:01.886 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:44:01.900 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:44:01.919 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:44:01.920 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:01 smithi005 ceph-mon[39562]: pgmap v2120: 17 pgs: 17 active+clean; 4.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-04-20T17:44:01.920 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:01 smithi005 ceph-mon[39562]: osdmap e794: 12 total, 11 up, 9 in 2024-04-20T17:44:01.951 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:01.955 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:44:01.990 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:44:01.991 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:44:01.991 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:44:01.991 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:44:01.991 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:44:01.991 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:44:01.991 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:44:01.991 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:44:01.991 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:44:01.991 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:44:01.991 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:44:01.991 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:44:01.991 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:44:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:01 smithi106 ceph-mon[44423]: pgmap v2120: 17 pgs: 17 active+clean; 4.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-04-20T17:44:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:01 smithi106 ceph-mon[44423]: osdmap e794: 12 total, 11 up, 9 in 2024-04-20T17:44:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:01 smithi171 ceph-mon[44189]: pgmap v2120: 17 pgs: 17 active+clean; 4.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-04-20T17:44:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:01 smithi171 ceph-mon[44189]: osdmap e794: 12 total, 11 up, 9 in 2024-04-20T17:44:02.418 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 34 15 81135 81120 149.075 100.438 0.0150037 0.00670622 2024-04-20T17:44:02.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:02 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2253942846' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:44:02.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:02.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:02.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:02.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:02.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:02.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:02.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:02.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:02.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:02 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2253942846' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:44:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:03.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:03.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:02 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2253942846' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:44:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:03.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:03.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:03.209 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:44:03.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:03.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:44:03.223 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-20T17:44:03.315 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:44:03.356 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:03.420 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 35 16 82781 82765 147.746 102.812 0.0076896 0.00676651 2024-04-20T17:44:03.900 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:44:03.902 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:03.902 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:03.902 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:03 smithi005 ceph-mon[39562]: pgmap v2122: 17 pgs: 17 active+clean; 4.7 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-04-20T17:44:03.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:03.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:44:04.005 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:44:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:03 smithi106 ceph-mon[44423]: pgmap v2122: 17 pgs: 17 active+clean; 4.7 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-04-20T17:44:04.036 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:03 smithi171 ceph-mon[44189]: pgmap v2122: 17 pgs: 17 active+clean; 4.7 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-04-20T17:44:04.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:04.392 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:44:04.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:44:04.420 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 36 16 84404 84388 146.46 101.438 0.00679154 0.00682612 2024-04-20T17:44:04.493 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:44:04.549 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:04.584 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:44:04.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:04.589 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:44:04.690 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:44:04.719 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:04.739 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:44:04.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:04.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:44:04.843 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:44:04.865 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:44:04.870 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:04.890 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:44:04.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:04.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:44:04.896 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:44:04.897 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:44:04.995 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:44:05.027 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:05.103 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:44:05.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:05.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:44:05.211 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:44:05.249 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:05.381 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:44:05.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:05.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:44:05.421 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 37 16 85984 85968 145.168 98.75 0.0140503 0.00688693 2024-04-20T17:44:05.485 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:44:05.513 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:05.534 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:44:05.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:05.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:44:05.639 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:44:05.673 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:05.733 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:44:05.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:05.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:44:05.839 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:44:05.888 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:05.914 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:44:05.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:05.922 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:44:05.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:05 smithi005 ceph-mon[39562]: pgmap v2123: 17 pgs: 17 active+clean; 5.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-04-20T17:44:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:05 smithi106 ceph-mon[44423]: pgmap v2123: 17 pgs: 17 active+clean; 5.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-04-20T17:44:06.022 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:44:06.071 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:06.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:05 smithi171 ceph-mon[44189]: pgmap v2123: 17 pgs: 17 active+clean; 5.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-04-20T17:44:06.275 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:44:06.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:06.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:44:06.379 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:44:06.410 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:06.421 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 38 16 87545 87529 143.916 97.5625 0.00719848 0.00694632 2024-04-20T17:44:06.686 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:06 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:44:06.686 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:06 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:44:06.686 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:06 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/286594010' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:44:06.689 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:44:06.898 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:44:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:06 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:44:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:06 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:44:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:06 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/286594010' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:44:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:06 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:44:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:06 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:44:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:06 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/286594010' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:44:07.421 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 39 16 89059 89043 142.652 94.625 0.00698655 0.00700751 2024-04-20T17:44:07.761 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:44:07.763 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:07 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:44:07.763 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:07 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:44:07.763 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:07 smithi005 ceph-mon[39562]: pgmap v2124: 17 pgs: 17 active+clean; 5.2 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-04-20T17:44:07.763 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:07 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:44:07.763 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:07 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/286594010' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:44:07.763 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:07 smithi005 ceph-mon[39562]: osdmap e795: 12 total, 11 up, 9 in 2024-04-20T17:44:07.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:07.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:44:07.866 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:44:07.903 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:07.980 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:44:07.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:07.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:44:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:07 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:44:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:07 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:44:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:07 smithi106 ceph-mon[44423]: pgmap v2124: 17 pgs: 17 active+clean; 5.2 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-04-20T17:44:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:07 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:44:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:07 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/286594010' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:44:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:07 smithi106 ceph-mon[44423]: osdmap e795: 12 total, 11 up, 9 in 2024-04-20T17:44:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:07 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:44:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:07 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:44:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:07 smithi171 ceph-mon[44189]: pgmap v2124: 17 pgs: 17 active+clean; 5.2 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-04-20T17:44:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:07 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:44:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:07 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/286594010' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:44:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:07 smithi171 ceph-mon[44189]: osdmap e795: 12 total, 11 up, 9 in 2024-04-20T17:44:08.085 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:44:08.112 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:08.136 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:44:08.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:08.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:44:08.240 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:44:08.288 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:08.325 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:44:08.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:08.331 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:44:08.421 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:44:08.422222 min lat: 0.00104535 max lat: 0.216981 avg lat: 0.00706883 2024-04-20T17:44:08.421 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:44:08.421 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 40 16 90553 90537 141.42 93.375 0.0114163 0.00706883 2024-04-20T17:44:08.432 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:44:08.457 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:44:08.459 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:08.480 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:44:08.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:08.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:44:08.583 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:44:08.611 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:08.631 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:44:08.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:08.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:44:08.735 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:44:08.762 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:08.780 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:44:08.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:08.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:44:08.885 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:44:08.936 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:08.965 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:44:08.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:08.975 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:44:09.075 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:44:09.112 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:44:09.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:09.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:44:09.292 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:44:09.305 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:44:09.305 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:44:09.320 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:09.341 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:44:09.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:09.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:44:09.422 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 41 16 92057 92041 140.262 94 0.0132497 0.00712793 2024-04-20T17:44:09.444 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:44:09.486 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:09.519 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:44:09.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:09.523 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:44:09.624 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:44:09.675 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:09.707 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:44:09.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:09.711 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:44:09.812 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:44:09.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:09 smithi171 ceph-mon[44189]: pgmap v2126: 17 pgs: 17 active+clean; 5.2 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-04-20T17:44:09.851 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:09.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:09 smithi005 ceph-mon[39562]: pgmap v2126: 17 pgs: 17 active+clean; 5.2 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-04-20T17:44:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:09 smithi106 ceph-mon[44423]: pgmap v2126: 17 pgs: 17 active+clean; 5.2 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-04-20T17:44:10.422 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 42 16 93545 93529 139.136 93 0.00992556 0.00718642 2024-04-20T17:44:10.498 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:44:10.705 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:44:10.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:10.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:44:10.740 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-20T17:44:10.811 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:44:10.870 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:11.074 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:44:11.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:11.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:44:11.178 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:44:11.213 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:11.241 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:44:11.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:11.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:44:11.348 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:44:11.399 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:11.422 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 43 16 94991 94975 138.002 90.375 0.0126606 0.00724449 2024-04-20T17:44:11.426 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:44:11.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:11.432 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:44:11.490 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:44:11.501 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:44:11.533 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:44:11.574 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:11.602 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:44:11.631 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:44:11.657 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:44:11.682 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:44:11.705 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:44:11.729 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:44:11.755 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:44:11.785 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:44:11.818 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:44:11.847 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:44:11.883 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:44:11.884 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:44:11.884 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:44:11.884 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:44:11.884 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:44:11.884 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:44:11.884 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:44:11.884 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:44:11.884 INFO:teuthology.orchestra.run.smithi005.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-04-20T17:44:11.884 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:44:11.884 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:44:11.884 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:44:11.884 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:44:11.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:11 smithi005 ceph-mon[39562]: pgmap v2127: 17 pgs: 17 active+clean; 5.5 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-04-20T17:44:11.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:11 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3741438580' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:44:11.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:11 smithi005 ceph-mon[39562]: osdmap e796: 12 total, 11 up, 9 in 2024-04-20T17:44:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:11 smithi106 ceph-mon[44423]: pgmap v2127: 17 pgs: 17 active+clean; 5.5 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-04-20T17:44:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:11 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3741438580' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:44:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:11 smithi106 ceph-mon[44423]: osdmap e796: 12 total, 11 up, 9 in 2024-04-20T17:44:12.047 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2024-04-20T17:44:12.047 INFO:tasks.cephadm.osd.7:Restarting osd.7 (starting--it wasn't running)... 2024-04-20T17:44:12.047 DEBUG:teuthology.orchestra.run.smithi106:osd.7> sudo journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.7.service 2024-04-20T17:44:12.049 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl start ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.7 2024-04-20T17:44:12.078 INFO:journalctl@ceph.osd.7.smithi106.stdout:-- Logs begin at Sat 2024-04-20 16:47:44 UTC. -- 2024-04-20T17:44:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:11 smithi171 ceph-mon[44189]: pgmap v2127: 17 pgs: 17 active+clean; 5.5 GiB data, 16 GiB used, 789 GiB / 805 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-04-20T17:44:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:11 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3741438580' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:44:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:11 smithi171 ceph-mon[44189]: osdmap e796: 12 total, 11 up, 9 in 2024-04-20T17:44:12.424 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 44 16 96377 96361 136.835 86.625 0.00637955 0.00730623 2024-04-20T17:44:12.511 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:12 smithi106 systemd[1]: Starting Ceph osd.7 for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:44:12.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:12 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:44:12.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:12 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:44:12.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:12.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:12.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:12.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:12.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:12.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:12.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:12.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:12.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:12.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:12.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:12 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:44:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:12 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:44:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:12 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:44:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:13.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:13.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:13.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:13.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:12 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:44:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:12 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:44:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:12 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:44:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:12 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:44:13.422 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 45 16 97753 97737 135.705 86 0.00778886 0.00736736 2024-04-20T17:44:13.511 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:13 smithi106 bash[182050]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T17:44:13.511 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:13 smithi106 bash[182050]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-20T17:44:13.511 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:13 smithi106 bash[182050]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-20T17:44:13.511 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:13 smithi106 bash[182050]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-20T17:44:13.511 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:13 smithi106 bash[182050]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-20T17:44:13.512 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:13 smithi106 bash[182050]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-04-20T17:44:13.512 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:13 smithi106 bash[182050]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-20T17:44:13.512 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:13 smithi106 bash[182050]: --> ceph-volume raw activate successful for osd ID: 7 2024-04-20T17:44:13.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:13 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:13.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:13 smithi005 ceph-mon[39562]: pgmap v2129: 17 pgs: 17 active+clean; 5.6 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-04-20T17:44:14.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:13 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:14.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:13 smithi106 ceph-mon[44423]: pgmap v2129: 17 pgs: 17 active+clean; 5.6 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-04-20T17:44:14.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:13 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:14.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:13 smithi171 ceph-mon[44189]: pgmap v2129: 17 pgs: 17 active+clean; 5.6 GiB data, 16 GiB used, 788 GiB / 805 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-04-20T17:44:14.100 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T17:44:14.100 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T17:44:14.131 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:44:14.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:14.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:44:14.236 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:44:14.284 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:14.318 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:44:14.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:14.320 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:44:14.421 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:44:14.422 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 46 16 99166 99150 134.675 88.3125 0.00604535 0.00742371 2024-04-20T17:44:14.456 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:14.562 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:44:14.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:14.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:44:14.646 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-20T17:44:14.666 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:44:14.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:14 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:44:14.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:14 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:44:14.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:14 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:44:14.719 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:14.745 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:44:14.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:14.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:44:14.851 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:44:14.884 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:14.955 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:44:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:14 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:44:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:14 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:44:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:14 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:44:15.012 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:14 smithi106 bash[182356]: 2e85bc6c811099029735eb6e13d2c072be010391a3fbe3573761971f7d0879f2 2024-04-20T17:44:15.012 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:14 smithi106 systemd[1]: Started Ceph osd.7 for 13c6e76c-ff37-11ee-bc93-c7b262605968. 2024-04-20T17:44:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:14 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:44:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:14 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:44:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:14 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:44:15.086 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:44:15.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:15.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:44:15.191 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:44:15.243 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:15.279 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:44:15.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:15.286 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:44:15.387 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:44:15.422 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 47 16 100544 100528 133.642 86.125 0.0163601 0.00748053 2024-04-20T17:44:15.427 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:15.511 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:15 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:15.088+0000 7f0e7f0317c0 -1 Falling back to public interface 2024-04-20T17:44:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:15 smithi106 ceph-mon[44423]: pgmap v2130: 17 pgs: 17 active+clean; 5.9 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-04-20T17:44:16.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:15 smithi171 ceph-mon[44189]: pgmap v2130: 17 pgs: 17 active+clean; 5.9 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-04-20T17:44:16.101 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:44:16.135 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:15 smithi005 ceph-mon[39562]: pgmap v2130: 17 pgs: 17 active+clean; 5.9 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-04-20T17:44:16.150 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:44:16.150 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:44:16.150 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:44:16.150 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [] 2024-04-20T17:44:16.151 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:44:16.422 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 48 16 101956 101940 132.696 88.25 0.00392806 0.00753405 2024-04-20T17:44:16.491 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:44:16.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:16.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:44:16.596 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:44:16.636 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:16.668 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:44:16.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:16.676 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:44:16.777 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:44:16.826 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:16.859 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:44:16.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:16.865 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:44:16.966 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:44:17.019 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:17.053 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:44:17.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:17.057 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:44:17.158 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:44:17.169 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-20T17:44:17.199 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:17.235 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:44:17.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:17.241 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:44:17.342 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:44:17.387 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:17.414 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:44:17.414 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:44:17.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:17.419 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:44:17.422 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 49 16 103257 103241 131.648 81.3125 0.0187355 0.00759473 2024-04-20T17:44:17.520 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:44:17.556 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:17.730 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:17 smithi106 ceph-mon[44423]: pgmap v2131: 17 pgs: 17 active+clean; 6.1 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 93 MiB/s wr, 1.48k op/s 2024-04-20T17:44:17.730 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:17 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3333011270' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:44:17.730 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:17 smithi106 ceph-mon[44423]: from='client.27638 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:44:17.750 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:44:17.908 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:44:17.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:17.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:44:17.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:17 smithi005 ceph-mon[39562]: pgmap v2131: 17 pgs: 17 active+clean; 6.1 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 93 MiB/s wr, 1.48k op/s 2024-04-20T17:44:17.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:17 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3333011270' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:44:17.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:17 smithi005 ceph-mon[39562]: from='client.27638 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:44:18.012 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:44:18.046 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:17 smithi171 ceph-mon[44189]: pgmap v2131: 17 pgs: 17 active+clean; 6.1 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 93 MiB/s wr, 1.48k op/s 2024-04-20T17:44:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:17 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3333011270' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:44:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:17 smithi171 ceph-mon[44189]: from='client.27638 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:44:18.189 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:44:18.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:18.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:44:18.199 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-04-20T17:44:18.199 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-04-20T17:44:18.199 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-04-20T17:44:18.199 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-04-20T17:44:18.199 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-04-20T17:44:18.199 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-04-20T17:44:18.200 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-04-20T17:44:18.200 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-04-20T17:44:18.200 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-04-20T17:44:18.200 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-04-20T17:44:18.200 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator fragmentation block 2024-04-20T17:44:18.200 INFO:teuthology.orchestra.run.smithi106.stderr:admin_socket: invalid command 2024-04-20T17:44:18.294 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:44:18.336 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:18.372 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:44:18.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:18.375 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:44:18.423 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 50 16 104600 104584 130.694 83.9375 0.0131732 0.00764979 2024-04-20T17:44:18.476 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:44:18.511 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.227+0000 7f0e7f0317c0 -1 osd.7 721 log_to_monitors true 2024-04-20T17:44:18.536 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:18.570 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:44:18.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:18.573 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:44:18.674 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:44:18.738 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:18.739 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:18 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:44:18.739 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:18 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:44:18.739 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:18 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:44:18.739 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:18 smithi005 ceph-mon[39562]: from='client.27638 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:44:18.740 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:18 smithi005 ceph-mon[39562]: osdmap e797: 12 total, 11 up, 9 in 2024-04-20T17:44:18.740 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:18 smithi005 ceph-mon[39562]: from='osd.7 v1:172.21.15.106:6812/2531699401' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-20T17:44:18.761 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:44:18.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:18.771 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:44:18.872 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:44:18.896 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:44:18.897 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:44:18.897 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:44:18.897 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-mon[44423]: from='client.27638 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:44:18.897 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-mon[44423]: osdmap e797: 12 total, 11 up, 9 in 2024-04-20T17:44:18.897 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-mon[44423]: from='osd.7 v1:172.21.15.106:6812/2531699401' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-20T17:44:18.897 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.745+0000 7f0e6be14700 -1 osd.7 721 failed to load OSD map for epoch 745, got 0 bytes 2024-04-20T17:44:18.898 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.748+0000 7f0e5ac5e700 -1 osd.7 746 failed to load OSD map for epoch 722, got 0 bytes 2024-04-20T17:44:18.898 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.748+0000 7f0e5ac5e700 -1 osd.7 746 failed to load OSD map for epoch 723, got 0 bytes 2024-04-20T17:44:18.898 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.748+0000 7f0e5ac5e700 -1 osd.7 746 failed to load OSD map for epoch 724, got 0 bytes 2024-04-20T17:44:18.898 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ac5e700 -1 osd.7 746 failed to load OSD map for epoch 725, got 0 bytes 2024-04-20T17:44:18.898 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ac5e700 -1 osd.7 746 failed to load OSD map for epoch 726, got 0 bytes 2024-04-20T17:44:18.898 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ac5e700 -1 osd.7 746 failed to load OSD map for epoch 727, got 0 bytes 2024-04-20T17:44:18.898 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ac5e700 -1 osd.7 746 failed to load OSD map for epoch 728, got 0 bytes 2024-04-20T17:44:18.898 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ac5e700 -1 osd.7 746 failed to load OSD map for epoch 729, got 0 bytes 2024-04-20T17:44:18.898 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ac5e700 -1 osd.7 746 failed to load OSD map for epoch 730, got 0 bytes 2024-04-20T17:44:18.898 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ac5e700 -1 osd.7 746 failed to load OSD map for epoch 731, got 0 bytes 2024-04-20T17:44:18.898 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ac5e700 -1 osd.7 746 failed to load OSD map for epoch 732, got 0 bytes 2024-04-20T17:44:18.898 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ac5e700 -1 osd.7 746 failed to load OSD map for epoch 733, got 0 bytes 2024-04-20T17:44:18.898 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ac5e700 -1 osd.7 746 failed to load OSD map for epoch 734, got 0 bytes 2024-04-20T17:44:18.899 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ac5e700 -1 osd.7 746 failed to load OSD map for epoch 735, got 0 bytes 2024-04-20T17:44:18.899 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ac5e700 -1 osd.7 746 failed to load OSD map for epoch 736, got 0 bytes 2024-04-20T17:44:18.899 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ac5e700 -1 osd.7 746 failed to load OSD map for epoch 737, got 0 bytes 2024-04-20T17:44:18.899 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ac5e700 -1 osd.7 746 failed to load OSD map for epoch 738, got 0 bytes 2024-04-20T17:44:18.899 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ac5e700 -1 osd.7 746 failed to load OSD map for epoch 739, got 0 bytes 2024-04-20T17:44:18.899 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ac5e700 -1 osd.7 746 failed to load OSD map for epoch 740, got 0 bytes 2024-04-20T17:44:18.899 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ac5e700 -1 osd.7 746 failed to load OSD map for epoch 741, got 0 bytes 2024-04-20T17:44:18.899 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ac5e700 -1 osd.7 746 failed to load OSD map for epoch 742, got 0 bytes 2024-04-20T17:44:18.899 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ac5e700 -1 osd.7 746 failed to load OSD map for epoch 743, got 0 bytes 2024-04-20T17:44:18.899 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ac5e700 -1 osd.7 746 failed to load OSD map for epoch 744, got 0 bytes 2024-04-20T17:44:18.899 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ac5e700 -1 osd.7 746 failed to load OSD map for epoch 745, got 0 bytes 2024-04-20T17:44:18.899 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e57c58700 -1 osd.7 746 failed to load OSD map for epoch 722, got 0 bytes 2024-04-20T17:44:18.899 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e57c58700 -1 osd.7 746 failed to load OSD map for epoch 723, got 0 bytes 2024-04-20T17:44:18.900 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e57c58700 -1 osd.7 746 failed to load OSD map for epoch 724, got 0 bytes 2024-04-20T17:44:18.900 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e57c58700 -1 osd.7 746 failed to load OSD map for epoch 725, got 0 bytes 2024-04-20T17:44:18.900 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ec66700 -1 osd.7 746 failed to load OSD map for epoch 722, got 0 bytes 2024-04-20T17:44:18.900 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ec66700 -1 osd.7 746 failed to load OSD map for epoch 723, got 0 bytes 2024-04-20T17:44:18.900 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ec66700 -1 osd.7 746 failed to load OSD map for epoch 724, got 0 bytes 2024-04-20T17:44:18.900 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ec66700 -1 osd.7 746 failed to load OSD map for epoch 725, got 0 bytes 2024-04-20T17:44:18.900 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ec66700 -1 osd.7 746 failed to load OSD map for epoch 726, got 0 bytes 2024-04-20T17:44:18.900 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ec66700 -1 osd.7 746 failed to load OSD map for epoch 727, got 0 bytes 2024-04-20T17:44:18.900 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ec66700 -1 osd.7 746 failed to load OSD map for epoch 728, got 0 bytes 2024-04-20T17:44:18.900 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ec66700 -1 osd.7 746 failed to load OSD map for epoch 729, got 0 bytes 2024-04-20T17:44:18.900 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ec66700 -1 osd.7 746 failed to load OSD map for epoch 730, got 0 bytes 2024-04-20T17:44:18.900 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ec66700 -1 osd.7 746 failed to load OSD map for epoch 731, got 0 bytes 2024-04-20T17:44:18.900 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ec66700 -1 osd.7 746 failed to load OSD map for epoch 732, got 0 bytes 2024-04-20T17:44:18.900 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e57c58700 -1 osd.7 746 failed to load OSD map for epoch 726, got 0 bytes 2024-04-20T17:44:18.900 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e57c58700 -1 osd.7 746 failed to load OSD map for epoch 727, got 0 bytes 2024-04-20T17:44:18.901 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e57c58700 -1 osd.7 746 failed to load OSD map for epoch 728, got 0 bytes 2024-04-20T17:44:18.901 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e57c58700 -1 osd.7 746 failed to load OSD map for epoch 729, got 0 bytes 2024-04-20T17:44:18.901 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e57c58700 -1 osd.7 746 failed to load OSD map for epoch 730, got 0 bytes 2024-04-20T17:44:18.901 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e57c58700 -1 osd.7 746 failed to load OSD map for epoch 731, got 0 bytes 2024-04-20T17:44:18.901 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e57c58700 -1 osd.7 746 failed to load OSD map for epoch 732, got 0 bytes 2024-04-20T17:44:18.901 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e57c58700 -1 osd.7 746 failed to load OSD map for epoch 733, got 0 bytes 2024-04-20T17:44:18.901 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e57c58700 -1 osd.7 746 failed to load OSD map for epoch 734, got 0 bytes 2024-04-20T17:44:18.901 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e57c58700 -1 osd.7 746 failed to load OSD map for epoch 735, got 0 bytes 2024-04-20T17:44:18.901 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e57c58700 -1 osd.7 746 failed to load OSD map for epoch 736, got 0 bytes 2024-04-20T17:44:18.901 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e57c58700 -1 osd.7 746 failed to load OSD map for epoch 737, got 0 bytes 2024-04-20T17:44:18.901 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e57c58700 -1 osd.7 746 failed to load OSD map for epoch 738, got 0 bytes 2024-04-20T17:44:18.901 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e57c58700 -1 osd.7 746 failed to load OSD map for epoch 739, got 0 bytes 2024-04-20T17:44:18.901 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e57c58700 -1 osd.7 746 failed to load OSD map for epoch 740, got 0 bytes 2024-04-20T17:44:18.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e57c58700 -1 osd.7 746 failed to load OSD map for epoch 741, got 0 bytes 2024-04-20T17:44:18.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e57c58700 -1 osd.7 746 failed to load OSD map for epoch 742, got 0 bytes 2024-04-20T17:44:18.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e57c58700 -1 osd.7 746 failed to load OSD map for epoch 743, got 0 bytes 2024-04-20T17:44:18.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e57c58700 -1 osd.7 746 failed to load OSD map for epoch 744, got 0 bytes 2024-04-20T17:44:18.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e57c58700 -1 osd.7 746 failed to load OSD map for epoch 745, got 0 bytes 2024-04-20T17:44:18.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ec66700 -1 osd.7 746 failed to load OSD map for epoch 733, got 0 bytes 2024-04-20T17:44:18.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ec66700 -1 osd.7 746 failed to load OSD map for epoch 734, got 0 bytes 2024-04-20T17:44:18.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ec66700 -1 osd.7 746 failed to load OSD map for epoch 735, got 0 bytes 2024-04-20T17:44:18.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ec66700 -1 osd.7 746 failed to load OSD map for epoch 736, got 0 bytes 2024-04-20T17:44:18.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ec66700 -1 osd.7 746 failed to load OSD map for epoch 737, got 0 bytes 2024-04-20T17:44:18.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ec66700 -1 osd.7 746 failed to load OSD map for epoch 738, got 0 bytes 2024-04-20T17:44:18.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ec66700 -1 osd.7 746 failed to load OSD map for epoch 739, got 0 bytes 2024-04-20T17:44:18.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ec66700 -1 osd.7 746 failed to load OSD map for epoch 740, got 0 bytes 2024-04-20T17:44:18.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ec66700 -1 osd.7 746 failed to load OSD map for epoch 741, got 0 bytes 2024-04-20T17:44:18.902 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ec66700 -1 osd.7 746 failed to load OSD map for epoch 742, got 0 bytes 2024-04-20T17:44:18.903 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ec66700 -1 osd.7 746 failed to load OSD map for epoch 743, got 0 bytes 2024-04-20T17:44:18.903 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ec66700 -1 osd.7 746 failed to load OSD map for epoch 744, got 0 bytes 2024-04-20T17:44:18.903 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.749+0000 7f0e5ec66700 -1 osd.7 746 failed to load OSD map for epoch 745, got 0 bytes 2024-04-20T17:44:18.903 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:44:18 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:44:18.760+0000 7f0e70489700 -1 osd.7 798 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T17:44:18.921 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:18.927 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-20T17:44:18.928 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-04-20T17:44:18.960 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:44:18.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:18.964 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:44:19.065 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:44:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:18 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:44:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:18 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:44:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:18 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:44:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:18 smithi171 ceph-mon[44189]: from='client.27638 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:44:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:18 smithi171 ceph-mon[44189]: osdmap e797: 12 total, 11 up, 9 in 2024-04-20T17:44:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:18 smithi171 ceph-mon[44189]: from='osd.7 v1:172.21.15.106:6812/2531699401' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-20T17:44:19.100 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:19.423 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 51 16 105945 105929 129.778 84.0625 0.00473643 0.0077041 2024-04-20T17:44:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:19 smithi106 ceph-mon[44423]: pgmap v2133: 17 pgs: 17 active+clean; 6.1 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-20T17:44:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:19 smithi106 ceph-mon[44423]: from='osd.7 v1:172.21.15.106:6812/2531699401' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-20T17:44:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:19 smithi106 ceph-mon[44423]: osdmap e798: 12 total, 11 up, 9 in 2024-04-20T17:44:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:19 smithi106 ceph-mon[44423]: from='osd.7 v1:172.21.15.106:6812/2531699401' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:44:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:19 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-20T17:44:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:19 smithi171 ceph-mon[44189]: pgmap v2133: 17 pgs: 17 active+clean; 6.1 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-20T17:44:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:19 smithi171 ceph-mon[44189]: from='osd.7 v1:172.21.15.106:6812/2531699401' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-20T17:44:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:19 smithi171 ceph-mon[44189]: osdmap e798: 12 total, 11 up, 9 in 2024-04-20T17:44:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:19 smithi171 ceph-mon[44189]: from='osd.7 v1:172.21.15.106:6812/2531699401' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:44:20.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:19 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-20T17:44:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:19 smithi005 ceph-mon[39562]: pgmap v2133: 17 pgs: 17 active+clean; 6.1 GiB data, 17 GiB used, 788 GiB / 805 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-20T17:44:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:19 smithi005 ceph-mon[39562]: from='osd.7 v1:172.21.15.106:6812/2531699401' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-20T17:44:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:19 smithi005 ceph-mon[39562]: osdmap e798: 12 total, 11 up, 9 in 2024-04-20T17:44:20.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:19 smithi005 ceph-mon[39562]: from='osd.7 v1:172.21.15.106:6812/2531699401' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-04-20T17:44:20.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:19 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-20T17:44:20.424 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 52 16 107224 107208 128.819 79.9375 0.0427259 0.00776131 2024-04-20T17:44:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:20 smithi005 ceph-mon[39562]: osd.7 v1:172.21.15.106:6812/2531699401 boot 2024-04-20T17:44:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:20 smithi005 ceph-mon[39562]: osdmap e799: 12 total, 12 up, 9 in 2024-04-20T17:44:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:20 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-20T17:44:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:20 smithi005 ceph-mon[39562]: 8.c scrub starts 2024-04-20T17:44:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:20 smithi005 ceph-mon[39562]: pgmap v2136: 17 pgs: 17 active+clean; 6.4 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-20T17:44:21.003 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:44:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:20 smithi106 ceph-mon[44423]: osd.7 v1:172.21.15.106:6812/2531699401 boot 2024-04-20T17:44:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:20 smithi106 ceph-mon[44423]: osdmap e799: 12 total, 12 up, 9 in 2024-04-20T17:44:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:20 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-20T17:44:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:20 smithi106 ceph-mon[44423]: 8.c scrub starts 2024-04-20T17:44:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:20 smithi106 ceph-mon[44423]: pgmap v2136: 17 pgs: 17 active+clean; 6.4 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-20T17:44:21.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:20 smithi171 ceph-mon[44189]: osd.7 v1:172.21.15.106:6812/2531699401 boot 2024-04-20T17:44:21.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:20 smithi171 ceph-mon[44189]: osdmap e799: 12 total, 12 up, 9 in 2024-04-20T17:44:21.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:20 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-20T17:44:21.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:20 smithi171 ceph-mon[44189]: 8.c scrub starts 2024-04-20T17:44:21.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:20 smithi171 ceph-mon[44189]: pgmap v2136: 17 pgs: 17 active+clean; 6.4 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-20T17:44:21.211 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:44:21.235 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:44:21.259 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:44:21.260 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:44:21.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:21.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:44:21.286 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:44:21.316 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:44:21.345 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:44:21.365 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:44:21.372 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:44:21.411 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:44:21.419 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:21.423 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 53 16 108255 108239 127.605 64.4375 0.0215701 0.00783403 2024-04-20T17:44:21.439 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:44:21.456 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:44:21.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:21.463 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:44:21.472 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:44:21.507 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:44:21.537 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:44:21.540 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:44:21.540 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:44:21.541 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:44:21.541 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:44:21.541 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:44:21.541 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:44:21.541 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:44:21.541 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:44:21.541 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:44:21.541 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:44:21.541 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:44:21.541 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:44:21.564 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:44:21.597 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:21.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:21.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:21 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:44:21.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:21 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:44:21.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:21 smithi005 ceph-mon[39562]: 8.6 scrub starts 2024-04-20T17:44:21.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:21 smithi005 ceph-mon[39562]: osdmap e800: 12 total, 12 up, 9 in 2024-04-20T17:44:21.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:21 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4176252937' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:44:21.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:21.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:21.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:21 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:44:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:21 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:44:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:21 smithi171 ceph-mon[44189]: 8.6 scrub starts 2024-04-20T17:44:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:21 smithi171 ceph-mon[44189]: osdmap e800: 12 total, 12 up, 9 in 2024-04-20T17:44:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:21 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4176252937' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:44:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:22.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:22.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:21 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:44:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:21 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:44:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:21 smithi106 ceph-mon[44423]: 8.6 scrub starts 2024-04-20T17:44:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:21 smithi106 ceph-mon[44423]: osdmap e800: 12 total, 12 up, 9 in 2024-04-20T17:44:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:21 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4176252937' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:44:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:22.424 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 54 16 109488 109472 126.669 77.0625 0.0170818 0.00789254 2024-04-20T17:44:22.460 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:44:22.984 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:44:22.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:22.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:44:23.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:22 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:23.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:22 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:23.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:22 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:23.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:22 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:23.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:22 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:23.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:22 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:23.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:22 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:23.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:22 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:23.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:22 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:23.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:22 smithi171 ceph-mon[44189]: 8.6 scrub ok 2024-04-20T17:44:23.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:22 smithi171 ceph-mon[44189]: 8.c scrub ok 2024-04-20T17:44:23.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:22 smithi171 ceph-mon[44189]: 8.5 scrub starts 2024-04-20T17:44:23.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:22 smithi171 ceph-mon[44189]: pgmap v2138: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.5 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-04-20T17:44:23.090 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:44:23.118 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:22 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:23.118 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:22 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:23.118 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:22 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:23.118 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:22 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:23.118 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:22 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:23.119 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:22 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:23.119 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:22 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:23.119 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:22 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:23.119 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:22 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:23.119 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:22 smithi106 ceph-mon[44423]: 8.6 scrub ok 2024-04-20T17:44:23.119 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:22 smithi106 ceph-mon[44423]: 8.c scrub ok 2024-04-20T17:44:23.119 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:22 smithi106 ceph-mon[44423]: 8.5 scrub starts 2024-04-20T17:44:23.119 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:22 smithi106 ceph-mon[44423]: pgmap v2138: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.5 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-04-20T17:44:23.120 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:23.142 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:44:23.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:23.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:44:23.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:22 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:23.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:22 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:23.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:22 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:23.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:22 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:23.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:22 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:23.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:22 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:23.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:22 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:23.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:22 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:23.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:22 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:23.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:22 smithi005 ceph-mon[39562]: 8.6 scrub ok 2024-04-20T17:44:23.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:22 smithi005 ceph-mon[39562]: 8.c scrub ok 2024-04-20T17:44:23.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:22 smithi005 ceph-mon[39562]: 8.5 scrub starts 2024-04-20T17:44:23.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:22 smithi005 ceph-mon[39562]: pgmap v2138: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.5 GiB data, 18 GiB used, 787 GiB / 805 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-04-20T17:44:23.247 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:44:23.303 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:23.339 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:44:23.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:23.347 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:44:23.424 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 55 16 110627 110611 125.66 71.1875 0.010036 0.00795632 2024-04-20T17:44:23.447 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:44:23.503 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:23.570 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:44:23.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:23.576 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:44:23.624 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:44:23.636 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:44:23.636 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:44:23.676 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:44:23.724 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:23.753 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:44:23.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:23.756 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:44:23.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:23 smithi171 ceph-mon[44189]: 8.5 scrub ok 2024-04-20T17:44:23.857 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:44:23.884 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:23.904 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:44:23.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:23.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:44:23.922 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:23 smithi005 ceph-mon[39562]: 8.5 scrub ok 2024-04-20T17:44:23.929 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-20T17:44:24.010 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:44:24.046 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:24.072 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:44:24.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:24.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:44:24.176 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:44:24.224 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:24.245 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:44:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:23 smithi106 ceph-mon[44423]: 8.5 scrub ok 2024-04-20T17:44:24.264 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:44:24.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:24.270 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:44:24.371 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:44:24.400 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:24.427 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 56 16 111596 111580 124.491 60.5625 0.0158878 0.00803076 2024-04-20T17:44:24.499 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:44:24.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:24.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:44:24.571 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:44:24.603 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:44:24.647 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:24.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:24 smithi005 ceph-mon[39562]: 8.f scrub starts 2024-04-20T17:44:24.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:24 smithi005 ceph-mon[39562]: 8.b scrub starts 2024-04-20T17:44:24.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:24 smithi005 ceph-mon[39562]: 8.2 scrub starts 2024-04-20T17:44:24.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:24 smithi005 ceph-mon[39562]: pgmap v2139: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 6.7 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-04-20T17:44:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:24 smithi171 ceph-mon[44189]: 8.f scrub starts 2024-04-20T17:44:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:24 smithi171 ceph-mon[44189]: 8.b scrub starts 2024-04-20T17:44:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:24 smithi171 ceph-mon[44189]: 8.2 scrub starts 2024-04-20T17:44:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:24 smithi171 ceph-mon[44189]: pgmap v2139: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 6.7 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-04-20T17:44:25.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:24 smithi106 ceph-mon[44423]: 8.f scrub starts 2024-04-20T17:44:25.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:24 smithi106 ceph-mon[44423]: 8.b scrub starts 2024-04-20T17:44:25.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:24 smithi106 ceph-mon[44423]: 8.2 scrub starts 2024-04-20T17:44:25.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:24 smithi106 ceph-mon[44423]: pgmap v2139: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 6.7 GiB data, 18 GiB used, 786 GiB / 805 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-04-20T17:44:25.427 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 57 16 112655 112639 123.468 66.1875 0.00500108 0.00809692 2024-04-20T17:44:25.808 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:44:25.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:25.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:44:25.913 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:44:25.939 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:25 smithi106 ceph-mon[44423]: 8.b scrub ok 2024-04-20T17:44:25.939 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:25 smithi106 ceph-mon[44423]: 8.f scrub ok 2024-04-20T17:44:25.939 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:25 smithi106 ceph-mon[44423]: 8.2 scrub ok 2024-04-20T17:44:25.941 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:25.965 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:44:25.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:25.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:44:26.075 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:44:26.076 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:25 smithi005 ceph-mon[39562]: 8.b scrub ok 2024-04-20T17:44:26.078 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:25 smithi005 ceph-mon[39562]: 8.f scrub ok 2024-04-20T17:44:26.078 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:25 smithi005 ceph-mon[39562]: 8.2 scrub ok 2024-04-20T17:44:26.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:25 smithi171 ceph-mon[44189]: 8.b scrub ok 2024-04-20T17:44:26.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:25 smithi171 ceph-mon[44189]: 8.f scrub ok 2024-04-20T17:44:26.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:25 smithi171 ceph-mon[44189]: 8.2 scrub ok 2024-04-20T17:44:26.164 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:26.200 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:44:26.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:26.205 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:44:26.257 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:44:26.257 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:44:26.306 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:44:26.344 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:26.428 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 58 16 113794 113778 122.567 71.1875 0.00955765 0.00815693 2024-04-20T17:44:26.962 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:26 smithi106 ceph-mon[44423]: 8.d scrub starts 2024-04-20T17:44:26.962 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:26 smithi106 ceph-mon[44423]: pgmap v2140: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 6.8 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-04-20T17:44:26.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:26 smithi005 ceph-mon[39562]: 8.d scrub starts 2024-04-20T17:44:26.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:26 smithi005 ceph-mon[39562]: pgmap v2140: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 6.8 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-04-20T17:44:26.995 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:44:26.995 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-04-20T17:44:26.995 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-20T17:44:26.995 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-20T17:44:26.995 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-04-20T17:44:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:26 smithi171 ceph-mon[44189]: 8.d scrub starts 2024-04-20T17:44:27.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:26 smithi171 ceph-mon[44189]: pgmap v2140: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 6.8 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-04-20T17:44:27.298 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:44:27.430 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 59 16 115019 115003 121.78 76.5625 0.0135104 0.00820952 2024-04-20T17:44:27.540 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:44:27.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:27.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:44:27.575 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-20T17:44:27.631 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:44:27.651 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:44:27.705 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:27.737 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:44:27.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:27.746 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:44:27.802 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:27 smithi005 ceph-mon[39562]: 8.d scrub ok 2024-04-20T17:44:27.802 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:27 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3205786969' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:44:27.802 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:27 smithi005 ceph-mon[39562]: from='client.27650 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:44:27.813 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:27 smithi106 ceph-mon[44423]: 8.d scrub ok 2024-04-20T17:44:27.813 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:27 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3205786969' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:44:27.813 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:27 smithi106 ceph-mon[44423]: from='client.27650 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:44:27.817 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:44:27.847 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:44:27.887 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:27 smithi171 ceph-mon[44189]: 8.d scrub ok 2024-04-20T17:44:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:27 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3205786969' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:44:28.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:27 smithi171 ceph-mon[44189]: from='client.27650 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:44:28.431 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:44:28.431618 min lat: 0.00104535 max lat: 0.216981 avg lat: 0.00827013 2024-04-20T17:44:28.431 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:44:28.431 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 60 16 116116 116100 120.893 68.5625 0.00899592 0.00827013 2024-04-20T17:44:28.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:28 smithi005 ceph-mon[39562]: 8.a scrub starts 2024-04-20T17:44:28.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:28 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:44:28.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:28 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:44:28.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:28 smithi005 ceph-mon[39562]: from='client.27650 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:44:28.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:28 smithi005 ceph-mon[39562]: osdmap e801: 12 total, 12 up, 9 in 2024-04-20T17:44:28.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:28 smithi005 ceph-mon[39562]: pgmap v2142: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 6.8 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 57 MiB/s wr, 914 op/s 2024-04-20T17:44:28.986 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:28 smithi106 ceph-mon[44423]: 8.a scrub starts 2024-04-20T17:44:28.986 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:28 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:44:28.986 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:28 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:44:28.986 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:28 smithi106 ceph-mon[44423]: from='client.27650 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:44:28.986 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:28 smithi106 ceph-mon[44423]: osdmap e801: 12 total, 12 up, 9 in 2024-04-20T17:44:28.986 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:28 smithi106 ceph-mon[44423]: pgmap v2142: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 6.8 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 57 MiB/s wr, 914 op/s 2024-04-20T17:44:29.058 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:44:29.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:29.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:44:29.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:28 smithi171 ceph-mon[44189]: 8.a scrub starts 2024-04-20T17:44:29.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:28 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:44:29.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:28 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:44:29.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:28 smithi171 ceph-mon[44189]: from='client.27650 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:44:29.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:28 smithi171 ceph-mon[44189]: osdmap e801: 12 total, 12 up, 9 in 2024-04-20T17:44:29.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:28 smithi171 ceph-mon[44189]: pgmap v2142: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 6.8 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 57 MiB/s wr, 914 op/s 2024-04-20T17:44:29.162 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:44:29.193 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:29.343 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:44:29.343 INFO:teuthology.orchestra.run.smithi106.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-20T17:44:29.343 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:44:29.343 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:44:29.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:29.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:44:29.431 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 61 16 117322 117306 120.147 75.375 0.0308163 0.0083215 2024-04-20T17:44:29.448 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:44:29.496 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:29.530 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:44:29.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:29.533 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:44:29.634 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:44:29.640 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:44:29.666 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:29.819 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:29 smithi171 ceph-mon[44189]: 8.a scrub ok 2024-04-20T17:44:29.819 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:29 smithi171 ceph-mon[44189]: 8.9 scrub starts 2024-04-20T17:44:29.895 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:29 smithi005 ceph-mon[39562]: 8.a scrub ok 2024-04-20T17:44:29.895 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:29 smithi005 ceph-mon[39562]: 8.9 scrub starts 2024-04-20T17:44:30.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:29 smithi106 ceph-mon[44423]: 8.a scrub ok 2024-04-20T17:44:30.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:29 smithi106 ceph-mon[44423]: 8.9 scrub starts 2024-04-20T17:44:30.036 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:44:30.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:30.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:44:30.141 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:44:30.196 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:30.234 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:44:30.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:30.238 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:44:30.339 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:44:30.366 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:44:30.385 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:30.395 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:44:30.418 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:44:30.420 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:44:30.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:30.423 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:44:30.431 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 62 15 118404 118389 119.301 67.6875 0.00919206 0.00838069 2024-04-20T17:44:30.442 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:44:30.465 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:44:30.491 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:44:30.516 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:44:30.525 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:44:30.543 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:44:30.581 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:44:30.585 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:30.611 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:44:30.630 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:44:30.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:30.638 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:44:30.639 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:44:30.670 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:44:30.672 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:44:30.672 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:44:30.672 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:44:30.672 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:44:30.672 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:44:30.672 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:44:30.672 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:44:30.673 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:44:30.673 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:44:30.673 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:44:30.673 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:44:30.673 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:44:30.738 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:44:30.767 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:30.787 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:44:30.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:30.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:44:30.830 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:30 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2254841587' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:44:30.830 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:30 smithi171 ceph-mon[44189]: pgmap v2143: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.1 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 73 MiB/s wr, 1.18k op/s 2024-04-20T17:44:30.830 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:30 smithi171 ceph-mon[44189]: 8.9 scrub ok 2024-04-20T17:44:30.830 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:30.891 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:44:30.928 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:30.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:30 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2254841587' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:44:30.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:30 smithi005 ceph-mon[39562]: pgmap v2143: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.1 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 73 MiB/s wr, 1.18k op/s 2024-04-20T17:44:30.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:30 smithi005 ceph-mon[39562]: 8.9 scrub ok 2024-04-20T17:44:30.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:31.032 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:44:31.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:31.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:44:31.040 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:44:31.137 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:44:31.167 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:30 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2254841587' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:44:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:30 smithi106 ceph-mon[44423]: pgmap v2143: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.1 GiB data, 19 GiB used, 786 GiB / 805 GiB avail; 73 MiB/s wr, 1.18k op/s 2024-04-20T17:44:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:30 smithi106 ceph-mon[44423]: 8.9 scrub ok 2024-04-20T17:44:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:31.431 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 63 16 119603 119587 118.596 74.875 0.00536459 0.00843011 2024-04-20T17:44:32.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.163 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:44:32.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:32.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:44:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:32.271 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:44:32.314 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:32.353 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:44:32.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:32.358 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:44:32.431 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 64 16 120803 120787 117.915 75 0.00424663 0.00847898 2024-04-20T17:44:32.459 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:44:32.485 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:32.505 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:44:32.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:32.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:44:32.609 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:44:32.659 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:32.685 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:44:32.688 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:44:32.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:32.690 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:44:32.791 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:44:32.834 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:32 smithi005 ceph-mon[39562]: pgmap v2144: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.2 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-20T17:44:32.965 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:44:32.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:32.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:44:33.076 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:44:33.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:32 smithi171 ceph-mon[44189]: pgmap v2144: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.2 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-20T17:44:33.128 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:33.188 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:44:33.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:33.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:44:33.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:32 smithi106 ceph-mon[44423]: pgmap v2144: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.2 GiB data, 19 GiB used, 785 GiB / 805 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-20T17:44:33.296 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:44:33.323 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:33.342 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:44:33.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:33.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:44:33.431 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 65 16 121969 121953 117.222 72.875 0.00418008 0.00852862 2024-04-20T17:44:33.447 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:44:33.498 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:33.535 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:44:33.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:33.541 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:44:33.642 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:44:33.669 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:33.688 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:44:33.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:33.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:44:33.792 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:44:33.822 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:33.842 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:44:33.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:33.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:44:33.946 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:44:33.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:33 smithi005 ceph-mon[39562]: 8.e deep-scrub starts 2024-04-20T17:44:33.977 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:44:33.977 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:44:33.977 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:44:33.977 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:44:33.977 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:44:33.994 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:33 smithi106 ceph-mon[44423]: 8.e deep-scrub starts 2024-04-20T17:44:34.101 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:44:34.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:34.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:44:34.213 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:44:34.249 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:34.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:33 smithi171 ceph-mon[44189]: 8.e deep-scrub starts 2024-04-20T17:44:34.431 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 66 16 123103 123087 116.52 70.875 0.0155943 0.00858032 2024-04-20T17:44:34.691 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:44:35.060 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 8, 9, 10, 11] out_osds: [6, 1, 7] dead_osds: [] live_osds: [0, 4, 5, 8, 9, 10, 11, 2, 6, 3, 1, 7] 2024-04-20T17:44:35.060 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:44:35.061 INFO:tasks.thrashosds.thrasher:inject_pause on osd.11 2024-04-20T17:44:35.061 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-20T17:44:35.061 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-20T17:44:35.062 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 config set filestore_inject_stall 3 2024-04-20T17:44:35.109 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:34 smithi171 ceph-mon[44189]: pgmap v2145: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.4 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-20T17:44:35.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:34 smithi005 ceph-mon[39562]: pgmap v2145: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.4 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-20T17:44:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:34 smithi106 ceph-mon[44423]: pgmap v2145: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 7.4 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-20T17:44:35.431 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 67 16 124225 124209 115.828 70.125 0.00513629 0.00863117 2024-04-20T17:44:35.483 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:44:36.019 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:44:36.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:36.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:44:36.062 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-20T17:44:36.128 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:44:36.156 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:36.175 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:44:36.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:36.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:44:36.279 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:44:36.327 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:36.431 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 68 16 125256 125240 115.072 64.4375 0.0116825 0.00868811 2024-04-20T17:44:36.438 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:44:36.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:36.446 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:44:36.547 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:44:36.604 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:36.653 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:44:36.706 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:44:37.062 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:44:37.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:37.065 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:44:37.166 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:44:37.231 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:37.280 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:44:37.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:37.288 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:44:37.389 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:44:37.417 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:37.431 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 69 16 126189 126173 114.25 58.3125 0.0169173 0.00875124 2024-04-20T17:44:37.437 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:44:37.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:37.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:44:37.540 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:44:37.602 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:37.675 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:44:37.948 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:37 smithi005 ceph-mon[39562]: 8.3 scrub starts 2024-04-20T17:44:37.948 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:37 smithi005 ceph-mon[39562]: pgmap v2146: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.5 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-20T17:44:37.948 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:37 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2247508458' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:44:37.948 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:37 smithi005 ceph-mon[39562]: from='client.27662 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:44:37.948 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:37 smithi005 ceph-mon[39562]: 8.4 scrub starts 2024-04-20T17:44:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:37 smithi106 ceph-mon[44423]: 8.3 scrub starts 2024-04-20T17:44:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:37 smithi106 ceph-mon[44423]: pgmap v2146: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.5 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-20T17:44:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:37 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2247508458' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:44:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:37 smithi106 ceph-mon[44423]: from='client.27662 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:44:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:37 smithi106 ceph-mon[44423]: 8.4 scrub starts 2024-04-20T17:44:38.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:37 smithi171 ceph-mon[44189]: 8.3 scrub starts 2024-04-20T17:44:38.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:37 smithi171 ceph-mon[44189]: pgmap v2146: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.5 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-20T17:44:38.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:37 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2247508458' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:44:38.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:37 smithi171 ceph-mon[44189]: from='client.27662 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:44:38.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:37 smithi171 ceph-mon[44189]: 8.4 scrub starts 2024-04-20T17:44:38.431 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 70 16 127180 127164 113.502 61.9375 0.00916282 0.00880903 2024-04-20T17:44:38.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:38 smithi005 ceph-mon[39562]: 8.7 scrub starts 2024-04-20T17:44:38.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:38 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:44:38.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:38 smithi005 ceph-mon[39562]: from='client.27662 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:44:38.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:38 smithi005 ceph-mon[39562]: osdmap e802: 12 total, 12 up, 9 in 2024-04-20T17:44:39.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:38 smithi106 ceph-mon[44423]: 8.7 scrub starts 2024-04-20T17:44:39.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:38 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:44:39.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:38 smithi106 ceph-mon[44423]: from='client.27662 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:44:39.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:38 smithi106 ceph-mon[44423]: osdmap e802: 12 total, 12 up, 9 in 2024-04-20T17:44:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:38 smithi171 ceph-mon[44189]: 8.7 scrub starts 2024-04-20T17:44:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:38 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:44:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:38 smithi171 ceph-mon[44189]: from='client.27662 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:44:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:38 smithi171 ceph-mon[44189]: osdmap e802: 12 total, 12 up, 9 in 2024-04-20T17:44:39.432 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 71 16 128246 128230 112.842 66.625 0.00510683 0.00886056 2024-04-20T17:44:39.483 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:44:39.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:39.489 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:44:39.590 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:44:39.622 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:39.637 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:44:39.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:39.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:44:39.741 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:44:39.769 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:39.791 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:44:39.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:39.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:44:39.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:39 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:44:39.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:39 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:44:39.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:39 smithi171 ceph-mon[44189]: pgmap v2148: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.5 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-20T17:44:39.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:39 smithi171 ceph-mon[44189]: osdmap e803: 12 total, 12 up, 9 in 2024-04-20T17:44:39.894 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:44:39.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:39 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:44:39.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:39 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:44:39.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:39 smithi005 ceph-mon[39562]: pgmap v2148: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.5 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-20T17:44:39.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:39 smithi005 ceph-mon[39562]: osdmap e803: 12 total, 12 up, 9 in 2024-04-20T17:44:39.977 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:39.998 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:44:39.998 INFO:teuthology.orchestra.run.smithi171.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-20T17:44:39.998 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:44:39.998 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:44:40.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:40.000 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:44:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:39 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:44:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:39 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:44:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:39 smithi106 ceph-mon[44423]: pgmap v2148: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.5 GiB data, 20 GiB used, 785 GiB / 805 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-20T17:44:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:39 smithi106 ceph-mon[44423]: osdmap e803: 12 total, 12 up, 9 in 2024-04-20T17:44:40.101 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:44:40.145 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:40.232 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:44:40.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:40.238 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:44:40.339 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:44:40.378 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:40.432 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 72 16 129265 129249 112.16 63.6875 0.00476001 0.00891172 2024-04-20T17:44:40.740 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:44:40.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:40.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:44:40.844 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:44:40.872 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:40.891 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:44:40.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:40.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:44:40.995 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:44:41.045 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:41.069 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:44:41.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:41.077 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:44:41.178 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:44:41.226 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:41.266 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:44:41.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:41.273 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:44:41.374 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:44:41.417 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:41.433 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 73 16 130336 130320 111.538 66.9375 0.0137714 0.00896339 2024-04-20T17:44:41.452 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:44:41.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:41.459 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:44:41.560 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:44:41.623 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:41.651 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:44:41.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:41.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:44:41.695 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:44:41.700 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:41 smithi171 ceph-mon[44189]: pgmap v2150: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.8 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-04-20T17:44:41.700 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:41 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2403724956' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:44:41.726 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:44:41.754 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:44:41.754 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:44:41.754 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:44:41.754 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:44:41.754 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.754 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:174728 7.6 7:6d3f70d5:::benchmark_data_smithi171_161871_object29002:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:44:41.754 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:43.086395+0000", 2024-04-20T17:44:41.755 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 238.50406575299999, 2024-04-20T17:44:41.755 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.27433470999999998, 2024-04-20T17:44:41.755 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:44:41.755 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:44:41.755 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:41.755 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:44:41.755 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:44:41.755 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:41.755 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 174728 2024-04-20T17:44:41.755 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.755 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:44:41.755 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.755 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:44:41.755 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.086395+0000", 2024-04-20T17:44:41.755 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.756 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.756 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.756 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:44:41.756 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.086395+0000", 2024-04-20T17:44:41.756 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.756 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.756 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.756 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:44:41.756 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.086398+0000", 2024-04-20T17:44:41.756 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3240000000000001e-06 2024-04-20T17:44:41.756 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.756 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.756 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:44:41.756 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.086404+0000", 2024-04-20T17:44:41.756 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.511e-06 2024-04-20T17:44:41.757 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.757 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.757 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:44:41.757 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.086418+0000", 2024-04-20T17:44:41.757 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3489000000000001e-05 2024-04-20T17:44:41.757 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.757 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.757 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:44:41.757 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.086425+0000", 2024-04-20T17:44:41.757 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.588e-06 2024-04-20T17:44:41.757 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.757 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.757 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.757 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.168960+0000", 2024-04-20T17:44:41.758 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.082534244000000007 2024-04-20T17:44:41.758 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.758 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.758 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:44:41.758 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.169012+0000", 2024-04-20T17:44:41.758 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2469000000000002e-05 2024-04-20T17:44:41.758 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.758 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.758 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.758 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.294480+0000", 2024-04-20T17:44:41.758 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.125467459 2024-04-20T17:44:41.758 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.758 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.758 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:44:41.759 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.294584+0000", 2024-04-20T17:44:41.759 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010476000000000001 2024-04-20T17:44:41.762 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.762 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.762 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:44:41.762 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.294709+0000", 2024-04-20T17:44:41.762 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000124511 2024-04-20T17:44:41.762 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.762 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.762 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:44:41.762 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.354962+0000", 2024-04-20T17:44:41.762 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.060252992999999998 2024-04-20T17:44:41.762 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.763 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.763 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:41.763 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.358801+0000", 2024-04-20T17:44:41.763 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0038393580000000002 2024-04-20T17:44:41.763 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.763 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.763 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:44:41.763 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.358839+0000", 2024-04-20T17:44:41.763 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7851000000000001e-05 2024-04-20T17:44:41.763 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.763 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.764 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:44:41.764 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.360730+0000", 2024-04-20T17:44:41.764 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001891153 2024-04-20T17:44:41.764 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.764 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:44:41.764 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.764 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.764 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.764 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:175098 7.6 7:61cd04e5:::benchmark_data_smithi171_161871_object29372:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:44:41.764 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:43.559347+0000", 2024-04-20T17:44:41.764 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 238.03111457099999, 2024-04-20T17:44:41.764 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.26388687399999999, 2024-04-20T17:44:41.764 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:44:41.764 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:44:41.764 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:41.765 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:44:41.765 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:44:41.765 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:41.765 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 175098 2024-04-20T17:44:41.765 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.765 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:44:41.765 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.765 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:44:41.765 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.559347+0000", 2024-04-20T17:44:41.765 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.765 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.765 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.765 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:44:41.765 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.559347+0000", 2024-04-20T17:44:41.766 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.766 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.766 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.766 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:44:41.766 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.559349+0000", 2024-04-20T17:44:41.766 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0339999999999999e-06 2024-04-20T17:44:41.766 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.766 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.766 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:44:41.766 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.559354+0000", 2024-04-20T17:44:41.766 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7139999999999998e-06 2024-04-20T17:44:41.766 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.766 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.766 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:44:41.766 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.559365+0000", 2024-04-20T17:44:41.767 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0481e-05 2024-04-20T17:44:41.767 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.767 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.767 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:44:41.767 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.559373+0000", 2024-04-20T17:44:41.767 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0979999999999994e-06 2024-04-20T17:44:41.767 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.767 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.767 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.767 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.626555+0000", 2024-04-20T17:44:41.767 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.067182424000000004 2024-04-20T17:44:41.767 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.767 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.767 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:44:41.767 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.626864+0000", 2024-04-20T17:44:41.768 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00030837399999999999 2024-04-20T17:44:41.768 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.768 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.768 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.768 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.727481+0000", 2024-04-20T17:44:41.768 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.100617676 2024-04-20T17:44:41.768 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.768 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.768 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:44:41.768 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.727576+0000", 2024-04-20T17:44:41.768 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4534e-05 2024-04-20T17:44:41.768 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.768 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.768 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:44:41.768 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.727713+0000", 2024-04-20T17:44:41.769 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000137348 2024-04-20T17:44:41.769 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.769 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.769 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:44:41.769 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.750990+0000", 2024-04-20T17:44:41.769 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023276538999999999 2024-04-20T17:44:41.769 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.769 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.769 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:41.769 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.819232+0000", 2024-04-20T17:44:41.769 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.068242066000000004 2024-04-20T17:44:41.769 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.769 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.769 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:44:41.770 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.819312+0000", 2024-04-20T17:44:41.770 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0139e-05 2024-04-20T17:44:41.770 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.770 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.770 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:44:41.770 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.823234+0000", 2024-04-20T17:44:41.770 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0039214469999999998 2024-04-20T17:44:41.770 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.770 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:44:41.770 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.770 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.770 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.770 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:175104 7.6 7:6b18dfe6:::benchmark_data_smithi171_161871_object29378:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:44:41.770 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:43.566043+0000", 2024-04-20T17:44:41.771 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 238.02441834199999, 2024-04-20T17:44:41.771 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.25980681500000002, 2024-04-20T17:44:41.771 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:44:41.771 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:44:41.771 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:41.771 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:44:41.771 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:44:41.771 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:41.771 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 175104 2024-04-20T17:44:41.771 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.771 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:44:41.771 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.771 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:44:41.771 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.566043+0000", 2024-04-20T17:44:41.771 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.772 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.772 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.772 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:44:41.772 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.566043+0000", 2024-04-20T17:44:41.772 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.772 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.772 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.772 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:44:41.772 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.566045+0000", 2024-04-20T17:44:41.772 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0310000000000001e-06 2024-04-20T17:44:41.772 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.772 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.772 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:44:41.772 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.566050+0000", 2024-04-20T17:44:41.772 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3530000000000002e-06 2024-04-20T17:44:41.773 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.773 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.773 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:44:41.773 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.566061+0000", 2024-04-20T17:44:41.773 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0805000000000001e-05 2024-04-20T17:44:41.773 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.773 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.773 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:44:41.773 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.566068+0000", 2024-04-20T17:44:41.773 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3980000000000002e-06 2024-04-20T17:44:41.773 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.773 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.773 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.773 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.626905+0000", 2024-04-20T17:44:41.774 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.060836928999999998 2024-04-20T17:44:41.774 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.774 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.774 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:44:41.774 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.627159+0000", 2024-04-20T17:44:41.774 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00025357199999999998 2024-04-20T17:44:41.774 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.774 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.774 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.774 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.740635+0000", 2024-04-20T17:44:41.774 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.113475665 2024-04-20T17:44:41.774 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.774 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.774 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:44:41.775 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.740733+0000", 2024-04-20T17:44:41.775 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8189999999999993e-05 2024-04-20T17:44:41.775 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.775 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.775 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:44:41.775 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.740887+0000", 2024-04-20T17:44:41.775 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015461699999999999 2024-04-20T17:44:41.775 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.775 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.775 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:44:41.775 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.754744+0000", 2024-04-20T17:44:41.775 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013856952000000001 2024-04-20T17:44:41.775 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.775 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.776 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:41.776 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.823306+0000", 2024-04-20T17:44:41.776 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.068561936000000004 2024-04-20T17:44:41.776 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.776 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.776 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:44:41.776 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.823366+0000", 2024-04-20T17:44:41.776 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9786e-05 2024-04-20T17:44:41.776 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.776 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.776 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:44:41.776 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.825850+0000", 2024-04-20T17:44:41.776 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0024835809999999999 2024-04-20T17:44:41.776 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.776 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:44:41.777 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.777 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.777 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.777 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:175140 7.6 7:6c38f614:::benchmark_data_smithi171_161871_object29414:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:44:41.777 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:43.661848+0000", 2024-04-20T17:44:41.777 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 237.92861300199999, 2024-04-20T17:44:41.777 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.27917723, 2024-04-20T17:44:41.777 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:44:41.777 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:44:41.777 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:41.777 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:44:41.777 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:44:41.777 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:41.777 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 175140 2024-04-20T17:44:41.777 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.778 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:44:41.778 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.778 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:44:41.778 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.661848+0000", 2024-04-20T17:44:41.778 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.778 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.778 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.778 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:44:41.778 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.661848+0000", 2024-04-20T17:44:41.778 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.778 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.778 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.778 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:44:41.778 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.661849+0000", 2024-04-20T17:44:41.779 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3e-06 2024-04-20T17:44:41.779 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.779 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.779 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:44:41.779 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.661854+0000", 2024-04-20T17:44:41.779 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7960000000000003e-06 2024-04-20T17:44:41.779 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.779 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.779 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:44:41.779 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.661863+0000", 2024-04-20T17:44:41.779 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4330000000000004e-06 2024-04-20T17:44:41.779 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.779 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.779 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:44:41.779 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.661870+0000", 2024-04-20T17:44:41.779 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8970000000000002e-06 2024-04-20T17:44:41.780 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.780 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.780 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.780 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.661929+0000", 2024-04-20T17:44:41.780 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9574000000000003e-05 2024-04-20T17:44:41.780 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.780 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.780 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:44:41.780 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.661979+0000", 2024-04-20T17:44:41.780 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9489e-05 2024-04-20T17:44:41.780 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.780 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.780 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.780 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.866488+0000", 2024-04-20T17:44:41.780 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.20450923200000001 2024-04-20T17:44:41.781 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.781 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.781 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:44:41.781 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.866586+0000", 2024-04-20T17:44:41.781 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8498999999999993e-05 2024-04-20T17:44:41.781 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.781 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.781 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:44:41.781 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.866722+0000", 2024-04-20T17:44:41.781 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000135971 2024-04-20T17:44:41.781 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.781 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.782 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:44:41.782 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.891713+0000", 2024-04-20T17:44:41.782 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.024990188999999999 2024-04-20T17:44:41.782 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.782 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.782 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:41.782 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.937246+0000", 2024-04-20T17:44:41.782 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.045532975000000003 2024-04-20T17:44:41.782 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.782 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.782 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:44:41.782 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.937298+0000", 2024-04-20T17:44:41.783 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2438000000000002e-05 2024-04-20T17:44:41.783 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.783 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.783 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:44:41.783 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.941025+0000", 2024-04-20T17:44:41.783 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0037274370000000001 2024-04-20T17:44:41.783 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.783 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:44:41.783 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.783 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.784 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.784 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:175172 7.6 7:690a41fc:::benchmark_data_smithi171_161871_object29446:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:44:41.784 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:43.710664+0000", 2024-04-20T17:44:41.784 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 237.87979755699999, 2024-04-20T17:44:41.784 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.39938321900000001, 2024-04-20T17:44:41.784 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:44:41.784 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:44:41.784 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:41.784 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:44:41.784 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:44:41.784 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:41.784 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 175172 2024-04-20T17:44:41.784 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.784 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:44:41.784 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.785 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:44:41.785 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.710664+0000", 2024-04-20T17:44:41.785 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.785 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.785 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.785 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:44:41.785 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.710664+0000", 2024-04-20T17:44:41.785 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.785 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.785 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.785 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:44:41.785 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.710666+0000", 2024-04-20T17:44:41.785 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0499999999999999e-06 2024-04-20T17:44:41.785 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.786 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.786 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:44:41.786 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.710672+0000", 2024-04-20T17:44:41.786 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5880000000000003e-06 2024-04-20T17:44:41.786 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.786 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.786 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:44:41.786 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.710684+0000", 2024-04-20T17:44:41.786 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1552999999999999e-05 2024-04-20T17:44:41.786 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.786 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.786 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:44:41.786 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.710692+0000", 2024-04-20T17:44:41.786 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2069999999999995e-06 2024-04-20T17:44:41.786 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.787 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.787 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.787 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.717275+0000", 2024-04-20T17:44:41.787 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0065826540000000003 2024-04-20T17:44:41.787 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.787 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.787 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:44:41.787 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.717341+0000", 2024-04-20T17:44:41.787 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6616000000000001e-05 2024-04-20T17:44:41.787 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.787 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.787 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.787 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.968974+0000", 2024-04-20T17:44:41.787 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.25163312700000001 2024-04-20T17:44:41.787 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.788 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.788 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:44:41.788 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.969036+0000", 2024-04-20T17:44:41.788 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1395000000000004e-05 2024-04-20T17:44:41.788 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.788 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.788 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:44:41.789 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.969115+0000", 2024-04-20T17:44:41.789 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8769000000000004e-05 2024-04-20T17:44:41.789 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.789 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.789 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:44:41.789 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.018597+0000", 2024-04-20T17:44:41.789 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.049482313999999999 2024-04-20T17:44:41.789 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.789 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.789 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:41.789 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.106589+0000", 2024-04-20T17:44:41.789 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.087992206000000003 2024-04-20T17:44:41.789 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.789 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.790 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:44:41.790 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.106646+0000", 2024-04-20T17:44:41.790 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6569e-05 2024-04-20T17:44:41.790 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.790 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.790 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:44:41.790 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.110047+0000", 2024-04-20T17:44:41.790 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0034011710000000001 2024-04-20T17:44:41.790 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.790 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:44:41.790 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.790 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.790 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.790 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:175268 7.6 7:6d024280:::benchmark_data_smithi171_161871_object29542:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:44:41.791 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:43.846739+0000", 2024-04-20T17:44:41.791 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 237.74372187500001, 2024-04-20T17:44:41.791 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.40914125000000001, 2024-04-20T17:44:41.791 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:44:41.791 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:44:41.791 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:41.791 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:44:41.791 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:44:41.791 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:41.791 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 175268 2024-04-20T17:44:41.791 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.791 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:44:41.791 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.791 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:44:41.792 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.846739+0000", 2024-04-20T17:44:41.792 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.792 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.792 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.792 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:44:41.792 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.846739+0000", 2024-04-20T17:44:41.792 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.792 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.792 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.792 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:44:41.792 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.846741+0000", 2024-04-20T17:44:41.792 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1440000000000001e-06 2024-04-20T17:44:41.792 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.792 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.792 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:44:41.793 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.846747+0000", 2024-04-20T17:44:41.793 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9490000000000001e-06 2024-04-20T17:44:41.793 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.793 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.793 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:44:41.793 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.846761+0000", 2024-04-20T17:44:41.793 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3105000000000001e-05 2024-04-20T17:44:41.793 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.793 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.793 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:44:41.793 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.846769+0000", 2024-04-20T17:44:41.793 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2349999999999999e-06 2024-04-20T17:44:41.793 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.793 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.794 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.794 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.957854+0000", 2024-04-20T17:44:41.794 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.11108511 2024-04-20T17:44:41.794 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.794 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.794 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:44:41.794 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.957917+0000", 2024-04-20T17:44:41.794 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3489000000000001e-05 2024-04-20T17:44:41.794 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.794 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.794 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.794 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.154458+0000", 2024-04-20T17:44:41.794 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.196540989 2024-04-20T17:44:41.794 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.795 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.795 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:44:41.795 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.154549+0000", 2024-04-20T17:44:41.795 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0999000000000001e-05 2024-04-20T17:44:41.795 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.795 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.795 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:44:41.795 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.154678+0000", 2024-04-20T17:44:41.795 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012838699999999999 2024-04-20T17:44:41.795 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.795 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.795 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:44:41.795 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.177898+0000", 2024-04-20T17:44:41.795 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023219961000000001 2024-04-20T17:44:41.795 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.796 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.796 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:41.797 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.251484+0000", 2024-04-20T17:44:41.797 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.073586043000000004 2024-04-20T17:44:41.797 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.797 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.797 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:44:41.797 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.251550+0000", 2024-04-20T17:44:41.797 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5897999999999998e-05 2024-04-20T17:44:41.798 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.798 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.798 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:44:41.798 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.255881+0000", 2024-04-20T17:44:41.798 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0043309409999999996 2024-04-20T17:44:41.798 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.798 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:44:41.798 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.798 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.798 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.798 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDPGPushReply(7.6 739/727 [PushReplyOp(7:6d024280:::benchmark_data_smithi171_161871_object29542:head)])", 2024-04-20T17:44:41.798 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:43.870312+0000", 2024-04-20T17:44:41.798 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 237.72014952999999, 2024-04-20T17:44:41.798 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.28409749299999998, 2024-04-20T17:44:41.799 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:44:41.799 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:44:41.799 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-04-20T17:44:41.799 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:44:41.799 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.799 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:44:41.799 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.870312+0000", 2024-04-20T17:44:41.799 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.799 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.799 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.799 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:44:41.799 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.870312+0000", 2024-04-20T17:44:41.799 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.799 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.799 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.800 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:44:41.800 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.870312+0000", 2024-04-20T17:44:41.800 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1699999999999998e-07 2024-04-20T17:44:41.800 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.800 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.800 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:44:41.800 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.870315+0000", 2024-04-20T17:44:41.800 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0539999999999999e-06 2024-04-20T17:44:41.800 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.800 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.800 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:44:41.800 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.870321+0000", 2024-04-20T17:44:41.800 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9039999999999997e-06 2024-04-20T17:44:41.800 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.801 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.801 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:44:41.801 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.870326+0000", 2024-04-20T17:44:41.801 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3109999999999996e-06 2024-04-20T17:44:41.801 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.801 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.801 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.801 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.146964+0000", 2024-04-20T17:44:41.801 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.27663748500000002 2024-04-20T17:44:41.801 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.801 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.801 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:44:41.801 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.154409+0000", 2024-04-20T17:44:41.801 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0074452219999999996 2024-04-20T17:44:41.801 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.802 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:44:41.802 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.802 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.802 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.802 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:175504 7.6 7:6a999528:::benchmark_data_smithi171_161871_object29778:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:44:41.802 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:44.181312+0000", 2024-04-20T17:44:41.802 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 237.40914956399999, 2024-04-20T17:44:41.802 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.30309401499999999, 2024-04-20T17:44:41.802 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:44:41.802 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:44:41.802 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:41.802 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:44:41.802 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:44:41.802 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:41.803 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 175504 2024-04-20T17:44:41.803 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.803 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:44:41.803 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.803 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:44:41.803 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.181312+0000", 2024-04-20T17:44:41.803 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.803 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.803 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.803 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:44:41.803 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.181312+0000", 2024-04-20T17:44:41.803 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.803 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.803 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.803 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:44:41.804 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.181314+0000", 2024-04-20T17:44:41.804 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9460000000000001e-06 2024-04-20T17:44:41.804 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.804 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.804 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:44:41.804 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.181320+0000", 2024-04-20T17:44:41.804 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3249999999999996e-06 2024-04-20T17:44:41.804 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.804 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.804 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:44:41.804 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.181331+0000", 2024-04-20T17:44:41.804 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1309e-05 2024-04-20T17:44:41.804 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.804 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.804 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:44:41.805 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.181339+0000", 2024-04-20T17:44:41.805 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0199999999999994e-06 2024-04-20T17:44:41.805 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.805 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.805 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.805 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.300051+0000", 2024-04-20T17:44:41.805 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.11871208599999999 2024-04-20T17:44:41.805 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.805 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.805 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:44:41.805 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.300093+0000", 2024-04-20T17:44:41.805 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1304000000000003e-05 2024-04-20T17:44:41.805 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.805 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.805 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.806 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.309233+0000", 2024-04-20T17:44:41.806 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0091401450000000006 2024-04-20T17:44:41.806 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.806 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.806 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:44:41.806 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.309296+0000", 2024-04-20T17:44:41.806 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2829000000000004e-05 2024-04-20T17:44:41.806 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.806 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.806 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:44:41.806 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.309371+0000", 2024-04-20T17:44:41.806 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5797999999999995e-05 2024-04-20T17:44:41.806 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.806 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.807 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:44:41.807 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.338833+0000", 2024-04-20T17:44:41.807 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.02946174 2024-04-20T17:44:41.807 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.807 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.807 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:41.807 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.390289+0000", 2024-04-20T17:44:41.807 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.051455555 2024-04-20T17:44:41.807 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.807 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.807 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:44:41.807 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.480101+0000", 2024-04-20T17:44:41.807 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.089811821999999999 2024-04-20T17:44:41.807 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.807 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.808 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:44:41.808 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.484406+0000", 2024-04-20T17:44:41.808 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0043051360000000002 2024-04-20T17:44:41.808 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.808 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:44:41.808 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.808 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.808 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.808 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:175517 7.6 7:63cbdd03:::benchmark_data_smithi171_161871_object29791:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:44:41.808 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:44.194403+0000", 2024-04-20T17:44:41.808 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 237.39605796800001, 2024-04-20T17:44:41.808 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.29416489000000001, 2024-04-20T17:44:41.808 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:44:41.808 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:44:41.808 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:41.809 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:44:41.809 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:44:41.809 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:41.809 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 175517 2024-04-20T17:44:41.809 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.809 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:44:41.809 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.809 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:44:41.809 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.194403+0000", 2024-04-20T17:44:41.809 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.809 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.809 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.809 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:44:41.809 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.194403+0000", 2024-04-20T17:44:41.809 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.810 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.810 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.810 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:44:41.810 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.194405+0000", 2024-04-20T17:44:41.810 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1610000000000001e-06 2024-04-20T17:44:41.810 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.810 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.810 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:44:41.810 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.194413+0000", 2024-04-20T17:44:41.810 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8930000000000005e-06 2024-04-20T17:44:41.810 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.810 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.810 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:44:41.810 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.194424+0000", 2024-04-20T17:44:41.811 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0822000000000001e-05 2024-04-20T17:44:41.811 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.811 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.811 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:44:41.811 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.194431+0000", 2024-04-20T17:44:41.811 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2439999999999996e-06 2024-04-20T17:44:41.811 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.811 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.811 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.811 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.300133+0000", 2024-04-20T17:44:41.811 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.105701933 2024-04-20T17:44:41.811 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.811 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.811 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:44:41.811 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.300543+0000", 2024-04-20T17:44:41.811 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00041010599999999998 2024-04-20T17:44:41.812 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.812 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.812 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.812 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.338880+0000", 2024-04-20T17:44:41.812 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.038336984999999997 2024-04-20T17:44:41.812 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.812 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.812 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:44:41.812 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.338946+0000", 2024-04-20T17:44:41.812 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5179999999999996e-05 2024-04-20T17:44:41.812 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.812 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.812 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:44:41.812 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.339032+0000", 2024-04-20T17:44:41.813 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6305999999999997e-05 2024-04-20T17:44:41.813 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.813 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.813 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:44:41.813 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.361912+0000", 2024-04-20T17:44:41.813 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.022880144000000002 2024-04-20T17:44:41.813 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.813 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.813 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:41.813 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.484538+0000", 2024-04-20T17:44:41.813 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.122625494 2024-04-20T17:44:41.813 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.813 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.813 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:44:41.813 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.484613+0000", 2024-04-20T17:44:41.814 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5603000000000001e-05 2024-04-20T17:44:41.814 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.814 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.814 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:44:41.814 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.488568+0000", 2024-04-20T17:44:41.814 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0039550189999999997 2024-04-20T17:44:41.814 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.814 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:44:41.814 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.814 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.814 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.814 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:175528 7.6 7:65985805:::benchmark_data_smithi171_161871_object29802:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:44:41.814 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:44.205896+0000", 2024-04-20T17:44:41.814 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 237.38456487900001, 2024-04-20T17:44:41.815 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.28621254600000001, 2024-04-20T17:44:41.815 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:44:41.815 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:44:41.815 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:41.815 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:44:41.815 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:44:41.815 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:41.815 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 175528 2024-04-20T17:44:41.815 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.815 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:44:41.815 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.815 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:44:41.815 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.205896+0000", 2024-04-20T17:44:41.815 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.815 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.816 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.816 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:44:41.816 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.205896+0000", 2024-04-20T17:44:41.816 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.816 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.816 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.816 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:44:41.816 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.205898+0000", 2024-04-20T17:44:41.816 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4500000000000001e-06 2024-04-20T17:44:41.816 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.816 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.816 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:44:41.816 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.205903+0000", 2024-04-20T17:44:41.816 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.22e-06 2024-04-20T17:44:41.816 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.817 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.817 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:44:41.817 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.205912+0000", 2024-04-20T17:44:41.817 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2289999999999992e-06 2024-04-20T17:44:41.817 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.817 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.817 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:44:41.817 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.205918+0000", 2024-04-20T17:44:41.817 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4079999999999997e-06 2024-04-20T17:44:41.817 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.817 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.817 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.817 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.300591+0000", 2024-04-20T17:44:41.817 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.094673560000000004 2024-04-20T17:44:41.818 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.818 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.818 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:44:41.818 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.300911+0000", 2024-04-20T17:44:41.818 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00031990799999999999 2024-04-20T17:44:41.818 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.818 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.818 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.818 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.351523+0000", 2024-04-20T17:44:41.819 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.050611695999999998 2024-04-20T17:44:41.819 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.819 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.819 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:44:41.819 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.351631+0000", 2024-04-20T17:44:41.819 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000107939 2024-04-20T17:44:41.819 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.819 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.819 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:44:41.819 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.351815+0000", 2024-04-20T17:44:41.819 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00018422499999999999 2024-04-20T17:44:41.819 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.819 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.819 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:44:41.819 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.365991+0000", 2024-04-20T17:44:41.820 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014175953999999999 2024-04-20T17:44:41.820 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.820 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.820 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:41.820 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.488632+0000", 2024-04-20T17:44:41.820 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.12264077299999999 2024-04-20T17:44:41.820 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.820 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.820 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:44:41.820 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.488669+0000", 2024-04-20T17:44:41.820 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7326000000000001e-05 2024-04-20T17:44:41.820 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.820 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.820 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:44:41.820 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.492109+0000", 2024-04-20T17:44:41.821 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0034398580000000001 2024-04-20T17:44:41.821 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.821 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:44:41.821 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.821 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.821 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.821 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:178953 7.6 7:606eac2a:::benchmark_data_smithi171_161871_object33227:head [delete] snapc 0=[] ondisk+write+known_if_redirected e740)", 2024-04-20T17:44:41.821 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:48.415871+0000", 2024-04-20T17:44:41.821 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 233.17459031300001, 2024-04-20T17:44:41.821 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.25897584000000001, 2024-04-20T17:44:41.821 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:44:41.821 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:44:41.821 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:41.821 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:44:41.821 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:44:41.822 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:41.822 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 178953 2024-04-20T17:44:41.822 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.822 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:44:41.822 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.822 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:44:41.822 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.415871+0000", 2024-04-20T17:44:41.822 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.822 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.822 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.822 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:44:41.822 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.415871+0000", 2024-04-20T17:44:41.822 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.822 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.823 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.823 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:44:41.823 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.415873+0000", 2024-04-20T17:44:41.823 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0159999999999998e-06 2024-04-20T17:44:41.823 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.823 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.823 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:44:41.823 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.415880+0000", 2024-04-20T17:44:41.823 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7050000000000002e-06 2024-04-20T17:44:41.823 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.823 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.823 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:44:41.823 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.415895+0000", 2024-04-20T17:44:41.823 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5432999999999999e-05 2024-04-20T17:44:41.823 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.823 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.824 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:44:41.824 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.415905+0000", 2024-04-20T17:44:41.824 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8050000000000001e-06 2024-04-20T17:44:41.824 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.824 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.824 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.824 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.555843+0000", 2024-04-20T17:44:41.824 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.13993809600000001 2024-04-20T17:44:41.824 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.824 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.824 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:44:41.824 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.556259+0000", 2024-04-20T17:44:41.824 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00041587800000000001 2024-04-20T17:44:41.824 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.824 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.825 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.825 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.599376+0000", 2024-04-20T17:44:41.825 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.043117474000000003 2024-04-20T17:44:41.825 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.825 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.825 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:44:41.825 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.599447+0000", 2024-04-20T17:44:41.825 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0522e-05 2024-04-20T17:44:41.825 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.825 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.825 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:44:41.825 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.599537+0000", 2024-04-20T17:44:41.825 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0161000000000006e-05 2024-04-20T17:44:41.825 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.825 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.826 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:44:41.826 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.639257+0000", 2024-04-20T17:44:41.826 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.039719800999999999 2024-04-20T17:44:41.826 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.826 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.826 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:41.826 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.672742+0000", 2024-04-20T17:44:41.826 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.033484977999999999 2024-04-20T17:44:41.826 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.826 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.826 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:44:41.826 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.672773+0000", 2024-04-20T17:44:41.826 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0936000000000003e-05 2024-04-20T17:44:41.826 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.827 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.827 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:44:41.827 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.674847+0000", 2024-04-20T17:44:41.827 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0020740350000000001 2024-04-20T17:44:41.827 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.827 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:44:41.827 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.827 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.827 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.827 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:180664 7.6 7:6ef0e8cc:::benchmark_data_smithi171_161871_object34938:head [delete] snapc 0=[] ondisk+write+known_if_redirected e741)", 2024-04-20T17:44:41.827 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:50.558557+0000", 2024-04-20T17:44:41.827 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 231.031903847, 2024-04-20T17:44:41.827 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.25715610300000002, 2024-04-20T17:44:41.827 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:44:41.827 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:44:41.828 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:41.828 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:44:41.828 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:44:41.828 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:41.828 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 180664 2024-04-20T17:44:41.828 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.828 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:44:41.828 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.828 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:44:41.828 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.558557+0000", 2024-04-20T17:44:41.828 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.828 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.828 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.828 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:44:41.828 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.558557+0000", 2024-04-20T17:44:41.829 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.829 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.829 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.829 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:44:41.829 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.558560+0000", 2024-04-20T17:44:41.829 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3980000000000002e-06 2024-04-20T17:44:41.829 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.829 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.829 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:44:41.829 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.558566+0000", 2024-04-20T17:44:41.829 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2870000000000001e-06 2024-04-20T17:44:41.829 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.829 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.829 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:44:41.830 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.558575+0000", 2024-04-20T17:44:41.830 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2990000000000002e-06 2024-04-20T17:44:41.830 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.830 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.830 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:44:41.830 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.558587+0000", 2024-04-20T17:44:41.830 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1182000000000001e-05 2024-04-20T17:44:41.830 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.830 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.830 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.830 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.577424+0000", 2024-04-20T17:44:41.830 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.018837962999999999 2024-04-20T17:44:41.830 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.830 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.830 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:44:41.831 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.577488+0000", 2024-04-20T17:44:41.831 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3625000000000002e-05 2024-04-20T17:44:41.831 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.831 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.831 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.831 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.722386+0000", 2024-04-20T17:44:41.831 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.144897575 2024-04-20T17:44:41.831 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.831 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.831 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:44:41.831 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.729577+0000", 2024-04-20T17:44:41.831 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0071915729999999997 2024-04-20T17:44:41.831 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.831 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.832 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:44:41.832 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.729707+0000", 2024-04-20T17:44:41.832 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012936299999999999 2024-04-20T17:44:41.832 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.832 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.832 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:44:41.832 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.811940+0000", 2024-04-20T17:44:41.832 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.082233443000000003 2024-04-20T17:44:41.832 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.832 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.832 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:41.832 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.812042+0000", 2024-04-20T17:44:41.832 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000102364 2024-04-20T17:44:41.832 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.832 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.833 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:44:41.833 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.812099+0000", 2024-04-20T17:44:41.833 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6715000000000002e-05 2024-04-20T17:44:41.833 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.833 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.833 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:44:41.833 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.815713+0000", 2024-04-20T17:44:41.833 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0036143159999999998 2024-04-20T17:44:41.833 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.833 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:44:41.833 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.833 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.833 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.833 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:181747 7.6 7:63cd038d:::benchmark_data_smithi171_161871_object36021:head [delete] snapc 0=[] ondisk+write+known_if_redirected e741)", 2024-04-20T17:44:41.833 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:51.929299+0000", 2024-04-20T17:44:41.834 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 229.66116195999999, 2024-04-20T17:44:41.834 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.29627274799999997, 2024-04-20T17:44:41.834 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:44:41.834 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:44:41.834 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:41.834 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:44:41.834 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:44:41.834 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:41.834 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 181747 2024-04-20T17:44:41.834 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.834 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:44:41.834 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.834 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:44:41.834 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:51.929299+0000", 2024-04-20T17:44:41.834 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.835 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.835 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.835 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:44:41.835 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:51.929299+0000", 2024-04-20T17:44:41.835 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.835 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.835 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.835 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:44:41.835 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:51.929302+0000", 2024-04-20T17:44:41.835 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.582e-06 2024-04-20T17:44:41.835 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.835 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.835 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:44:41.835 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:51.929314+0000", 2024-04-20T17:44:41.836 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2160999999999999e-05 2024-04-20T17:44:41.836 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.836 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.836 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:44:41.836 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:51.929330+0000", 2024-04-20T17:44:41.836 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6144999999999999e-05 2024-04-20T17:44:41.836 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.836 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.836 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:44:41.836 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:51.929341+0000", 2024-04-20T17:44:41.836 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0392e-05 2024-04-20T17:44:41.836 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.836 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.836 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.836 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:51.999407+0000", 2024-04-20T17:44:41.837 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.070066656000000005 2024-04-20T17:44:41.837 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.837 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.837 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:44:41.837 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:51.999449+0000", 2024-04-20T17:44:41.837 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2276e-05 2024-04-20T17:44:41.837 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.837 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.837 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.837 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.103493+0000", 2024-04-20T17:44:41.837 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.104043096 2024-04-20T17:44:41.837 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.837 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.837 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:44:41.837 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.103578+0000", 2024-04-20T17:44:41.838 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5651999999999999e-05 2024-04-20T17:44:41.838 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.838 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.838 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:44:41.838 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.103704+0000", 2024-04-20T17:44:41.838 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000125592 2024-04-20T17:44:41.838 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.838 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.838 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:44:41.838 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.119570+0000", 2024-04-20T17:44:41.838 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015865707 2024-04-20T17:44:41.838 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.838 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.838 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:41.839 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.221746+0000", 2024-04-20T17:44:41.839 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.102176936 2024-04-20T17:44:41.839 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.839 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.839 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:44:41.839 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.221812+0000", 2024-04-20T17:44:41.839 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5470999999999997e-05 2024-04-20T17:44:41.839 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.839 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.839 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:44:41.839 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.225572+0000", 2024-04-20T17:44:41.839 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0037600820000000001 2024-04-20T17:44:41.839 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.839 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:44:41.840 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.840 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.840 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.840 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:182127 7.6 7:6598bc27:::benchmark_data_smithi171_161871_object36401:head [delete] snapc 0=[] ondisk+write+known_if_redirected e741)", 2024-04-20T17:44:41.840 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:52.408315+0000", 2024-04-20T17:44:41.840 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 229.18214650300001, 2024-04-20T17:44:41.840 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.27155902300000001, 2024-04-20T17:44:41.840 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:44:41.840 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:44:41.840 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:41.840 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:44:41.840 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:44:41.840 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:41.840 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 182127 2024-04-20T17:44:41.840 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.841 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:44:41.841 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.841 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:44:41.841 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.408315+0000", 2024-04-20T17:44:41.841 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.841 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.841 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.841 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:44:41.841 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.408315+0000", 2024-04-20T17:44:41.841 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.841 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.841 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.841 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:44:41.841 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.408318+0000", 2024-04-20T17:44:41.842 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2160000000000002e-06 2024-04-20T17:44:41.842 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.842 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.842 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:44:41.842 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.408326+0000", 2024-04-20T17:44:41.842 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5369999999999997e-06 2024-04-20T17:44:41.842 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.842 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.842 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:44:41.842 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.408340+0000", 2024-04-20T17:44:41.842 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.401e-05 2024-04-20T17:44:41.842 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.842 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.842 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:44:41.843 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.408351+0000", 2024-04-20T17:44:41.843 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0458e-05 2024-04-20T17:44:41.843 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.843 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.843 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.843 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.505668+0000", 2024-04-20T17:44:41.843 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.097316620000000006 2024-04-20T17:44:41.843 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.843 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.843 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:44:41.843 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.506051+0000", 2024-04-20T17:44:41.843 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00038314000000000002 2024-04-20T17:44:41.843 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.843 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.843 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.844 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.644953+0000", 2024-04-20T17:44:41.844 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.13890235100000001 2024-04-20T17:44:41.844 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.844 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.844 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:44:41.844 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.645053+0000", 2024-04-20T17:44:41.844 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9555000000000005e-05 2024-04-20T17:44:41.844 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.844 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.844 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:44:41.844 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.645184+0000", 2024-04-20T17:44:41.844 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013101599999999999 2024-04-20T17:44:41.844 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.844 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.845 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:44:41.845 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.651423+0000", 2024-04-20T17:44:41.845 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0062393400000000003 2024-04-20T17:44:41.845 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.845 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.845 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:41.845 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.676555+0000", 2024-04-20T17:44:41.845 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.025131715999999998 2024-04-20T17:44:41.845 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.845 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.845 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:44:41.845 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.676589+0000", 2024-04-20T17:44:41.845 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4586999999999999e-05 2024-04-20T17:44:41.845 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.845 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.846 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:44:41.846 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.679874+0000", 2024-04-20T17:44:41.846 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0032844770000000001 2024-04-20T17:44:41.846 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.846 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:44:41.846 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.846 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.846 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.846 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:182476 7.6 7:63007d39:::benchmark_data_smithi171_161871_object36750:head [delete] snapc 0=[] ondisk+write+known_if_redirected e741)", 2024-04-20T17:44:41.846 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:52.860619+0000", 2024-04-20T17:44:41.846 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 228.72984227000001, 2024-04-20T17:44:41.846 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.27225396400000001, 2024-04-20T17:44:41.846 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:44:41.846 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:44:41.847 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:41.847 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:44:41.847 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:44:41.847 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:41.847 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 182476 2024-04-20T17:44:41.847 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.847 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:44:41.847 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.847 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:44:41.847 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.860619+0000", 2024-04-20T17:44:41.847 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.847 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.847 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.847 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:44:41.847 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.860619+0000", 2024-04-20T17:44:41.848 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.848 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.848 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.848 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:44:41.848 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.860622+0000", 2024-04-20T17:44:41.848 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0089999999999999e-06 2024-04-20T17:44:41.848 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.848 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.848 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:44:41.848 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.860629+0000", 2024-04-20T17:44:41.848 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4499999999999998e-06 2024-04-20T17:44:41.848 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.848 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.848 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:44:41.848 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.860647+0000", 2024-04-20T17:44:41.849 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7773000000000001e-05 2024-04-20T17:44:41.849 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.849 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.849 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:44:41.849 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.860657+0000", 2024-04-20T17:44:41.849 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0006999999999999e-05 2024-04-20T17:44:41.849 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.849 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.849 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.849 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.879198+0000", 2024-04-20T17:44:41.849 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.018541067000000001 2024-04-20T17:44:41.849 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.849 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.849 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:44:41.849 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.879258+0000", 2024-04-20T17:44:41.850 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0188e-05 2024-04-20T17:44:41.850 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.850 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.850 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.850 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.021562+0000", 2024-04-20T17:44:41.850 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.14230343300000001 2024-04-20T17:44:41.850 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.850 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.850 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:44:41.850 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.021656+0000", 2024-04-20T17:44:41.850 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4246000000000005e-05 2024-04-20T17:44:41.850 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.850 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.850 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:44:41.850 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.021802+0000", 2024-04-20T17:44:41.851 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000145539 2024-04-20T17:44:41.851 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.851 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.851 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:44:41.851 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.058021+0000", 2024-04-20T17:44:41.851 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.036219121999999999 2024-04-20T17:44:41.851 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.851 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.851 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:41.851 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.128996+0000", 2024-04-20T17:44:41.851 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.070975521 2024-04-20T17:44:41.851 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.851 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.851 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:44:41.851 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.129063+0000", 2024-04-20T17:44:41.851 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6241999999999994e-05 2024-04-20T17:44:41.852 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.852 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.852 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:44:41.852 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.132873+0000", 2024-04-20T17:44:41.852 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0038103669999999998 2024-04-20T17:44:41.852 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.852 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:44:41.852 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.852 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.852 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.852 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:182477 7.6 7:6df68048:::benchmark_data_smithi171_161871_object36751:head [delete] snapc 0=[] ondisk+write+known_if_redirected e741)", 2024-04-20T17:44:41.852 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:52.861320+0000", 2024-04-20T17:44:41.852 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 228.72914139100001, 2024-04-20T17:44:41.852 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.27478792000000002, 2024-04-20T17:44:41.853 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:44:41.853 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:44:41.853 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:41.853 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:44:41.853 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:44:41.853 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:41.853 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 182477 2024-04-20T17:44:41.853 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.853 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:44:41.853 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.853 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:44:41.853 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.861320+0000", 2024-04-20T17:44:41.853 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.853 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.853 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.854 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:44:41.854 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.861320+0000", 2024-04-20T17:44:41.854 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.854 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.854 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.854 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:44:41.854 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.861321+0000", 2024-04-20T17:44:41.854 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3599999999999999e-06 2024-04-20T17:44:41.854 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.854 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.854 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:44:41.854 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.861326+0000", 2024-04-20T17:44:41.854 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7960000000000003e-06 2024-04-20T17:44:41.854 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.855 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.855 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:44:41.855 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.861340+0000", 2024-04-20T17:44:41.855 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4229999999999999e-05 2024-04-20T17:44:41.855 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.855 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.855 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:44:41.855 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.861349+0000", 2024-04-20T17:44:41.855 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0329999999999997e-06 2024-04-20T17:44:41.855 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.855 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.855 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.855 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.995885+0000", 2024-04-20T17:44:41.855 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.134535565 2024-04-20T17:44:41.855 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.856 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.856 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:44:41.856 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.997397+0000", 2024-04-20T17:44:41.856 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0015126079999999999 2024-04-20T17:44:41.856 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.856 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.856 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.856 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.047463+0000", 2024-04-20T17:44:41.856 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.050065655000000001 2024-04-20T17:44:41.856 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.856 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.856 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:44:41.856 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.047530+0000", 2024-04-20T17:44:41.856 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7113000000000006e-05 2024-04-20T17:44:41.856 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.857 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.857 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:44:41.857 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.047607+0000", 2024-04-20T17:44:41.857 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6542e-05 2024-04-20T17:44:41.857 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.857 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.857 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:44:41.857 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.068691+0000", 2024-04-20T17:44:41.857 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.021084094000000001 2024-04-20T17:44:41.857 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.857 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.857 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:41.857 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.132951+0000", 2024-04-20T17:44:41.857 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.064260566000000005 2024-04-20T17:44:41.858 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.858 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.858 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:44:41.858 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.133018+0000", 2024-04-20T17:44:41.858 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7068000000000002e-05 2024-04-20T17:44:41.858 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.858 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.858 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:44:41.858 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.136108+0000", 2024-04-20T17:44:41.858 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0030892900000000002 2024-04-20T17:44:41.858 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.858 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:44:41.858 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.858 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.858 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.859 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:182483 7.6 7:68dbf85b:::benchmark_data_smithi171_161871_object36757:head [delete] snapc 0=[] ondisk+write+known_if_redirected e741)", 2024-04-20T17:44:41.859 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:52.869238+0000", 2024-04-20T17:44:41.859 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 228.72122307800001, 2024-04-20T17:44:41.859 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.270150002, 2024-04-20T17:44:41.859 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:44:41.859 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:44:41.859 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:41.859 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:44:41.859 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:44:41.859 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:41.859 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 182483 2024-04-20T17:44:41.859 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.859 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:44:41.859 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.859 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:44:41.860 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.869238+0000", 2024-04-20T17:44:41.860 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.860 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.860 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.860 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:44:41.860 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.869238+0000", 2024-04-20T17:44:41.860 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.860 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.860 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.860 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:44:41.860 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.869241+0000", 2024-04-20T17:44:41.860 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.418e-06 2024-04-20T17:44:41.860 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.860 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.861 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:44:41.861 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.869246+0000", 2024-04-20T17:44:41.861 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.429e-06 2024-04-20T17:44:41.861 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.861 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.861 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:44:41.861 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.869261+0000", 2024-04-20T17:44:41.861 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4912000000000001e-05 2024-04-20T17:44:41.861 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.861 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.861 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:44:41.861 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.869270+0000", 2024-04-20T17:44:41.861 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1880000000000008e-06 2024-04-20T17:44:41.861 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.861 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.861 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.862 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.997440+0000", 2024-04-20T17:44:41.862 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.128169965 2024-04-20T17:44:41.862 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.862 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.862 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:44:41.862 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.997815+0000", 2024-04-20T17:44:41.862 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00037503599999999998 2024-04-20T17:44:41.862 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.862 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.862 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.862 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.058066+0000", 2024-04-20T17:44:41.863 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.060250808000000003 2024-04-20T17:44:41.863 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.863 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.863 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:44:41.863 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.058126+0000", 2024-04-20T17:44:41.863 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9808000000000001e-05 2024-04-20T17:44:41.863 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.863 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.863 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:44:41.863 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.058203+0000", 2024-04-20T17:44:41.863 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6874999999999998e-05 2024-04-20T17:44:41.863 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.863 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.863 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:44:41.863 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.078821+0000", 2024-04-20T17:44:41.864 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.020618926999999999 2024-04-20T17:44:41.864 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.864 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.864 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:41.864 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.136189+0000", 2024-04-20T17:44:41.864 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057367920000000003 2024-04-20T17:44:41.864 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.864 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.864 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:44:41.864 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.136251+0000", 2024-04-20T17:44:41.864 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1742999999999995e-05 2024-04-20T17:44:41.864 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.864 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.864 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:44:41.864 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.139388+0000", 2024-04-20T17:44:41.864 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0031369729999999999 2024-04-20T17:44:41.865 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.865 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:44:41.865 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.865 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.865 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.865 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:182484 7.6 7:67cc5f5a:::benchmark_data_smithi171_161871_object36758:head [delete] snapc 0=[] ondisk+write+known_if_redirected e741)", 2024-04-20T17:44:41.865 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:52.870902+0000", 2024-04-20T17:44:41.865 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 228.71955889, 2024-04-20T17:44:41.865 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.271107971, 2024-04-20T17:44:41.865 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:44:41.865 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:44:41.865 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:41.865 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:44:41.865 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:44:41.866 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:41.866 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 182484 2024-04-20T17:44:41.866 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.866 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:44:41.866 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.866 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:44:41.866 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.870902+0000", 2024-04-20T17:44:41.866 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.866 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.866 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.866 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:44:41.866 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.870902+0000", 2024-04-20T17:44:41.866 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.866 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.867 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.867 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:44:41.867 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.870904+0000", 2024-04-20T17:44:41.867 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4169999999999999e-06 2024-04-20T17:44:41.867 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.867 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.867 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:44:41.867 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.870908+0000", 2024-04-20T17:44:41.867 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1119999999999998e-06 2024-04-20T17:44:41.867 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.867 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.867 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:44:41.867 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.870917+0000", 2024-04-20T17:44:41.867 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7700000000000007e-06 2024-04-20T17:44:41.867 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.868 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.868 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:44:41.868 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.870924+0000", 2024-04-20T17:44:41.868 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7249999999999997e-06 2024-04-20T17:44:41.868 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.868 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.868 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.868 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.997850+0000", 2024-04-20T17:44:41.868 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.12692612 2024-04-20T17:44:41.868 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.868 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.868 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:44:41.868 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.998178+0000", 2024-04-20T17:44:41.868 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00032771599999999998 2024-04-20T17:44:41.869 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.869 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.869 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.869 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.068745+0000", 2024-04-20T17:44:41.869 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.070566738000000004 2024-04-20T17:44:41.869 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.869 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.869 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:44:41.869 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.068837+0000", 2024-04-20T17:44:41.869 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2014000000000004e-05 2024-04-20T17:44:41.869 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.869 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.870 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:44:41.870 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.068929+0000", 2024-04-20T17:44:41.870 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1618000000000003e-05 2024-04-20T17:44:41.870 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.870 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.870 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:44:41.870 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.095299+0000", 2024-04-20T17:44:41.870 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.026370050999999999 2024-04-20T17:44:41.870 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.870 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.871 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:41.871 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.139463+0000", 2024-04-20T17:44:41.871 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.044164876999999998 2024-04-20T17:44:41.871 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.871 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.871 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:44:41.871 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.139528+0000", 2024-04-20T17:44:41.871 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4930000000000003e-05 2024-04-20T17:44:41.871 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.871 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.871 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:44:41.871 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.142010+0000", 2024-04-20T17:44:41.872 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0024818829999999998 2024-04-20T17:44:41.872 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.872 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:44:41.872 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.872 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.872 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.872 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:205309 7.6 7:64cf33f4:::benchmark_data_smithi171_161871_object59583:head [delete] snapc 0=[] ondisk+write+known_if_redirected e745)", 2024-04-20T17:44:41.872 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:41:20.783215+0000", 2024-04-20T17:44:41.872 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 200.80724590700001, 2024-04-20T17:44:41.872 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.265927152, 2024-04-20T17:44:41.872 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:44:41.872 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:44:41.872 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:41.872 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:44:41.873 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:44:41.873 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:41.873 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 205309 2024-04-20T17:44:41.873 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.873 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:44:41.873 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.873 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:44:41.873 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:20.783215+0000", 2024-04-20T17:44:41.873 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.873 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.873 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.873 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:44:41.873 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:20.783215+0000", 2024-04-20T17:44:41.873 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.873 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.874 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.874 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:44:41.874 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:20.783217+0000", 2024-04-20T17:44:41.874 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9230000000000001e-06 2024-04-20T17:44:41.874 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.874 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.874 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:44:41.874 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:20.783223+0000", 2024-04-20T17:44:41.874 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1539999999999999e-06 2024-04-20T17:44:41.874 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.874 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.874 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:44:41.874 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:20.783236+0000", 2024-04-20T17:44:41.874 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2816000000000001e-05 2024-04-20T17:44:41.875 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.875 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.875 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:44:41.875 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:20.783244+0000", 2024-04-20T17:44:41.875 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5780000000000001e-06 2024-04-20T17:44:41.875 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.875 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.875 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.875 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:20.790554+0000", 2024-04-20T17:44:41.875 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0073102159999999996 2024-04-20T17:44:41.875 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.875 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.875 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:44:41.875 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:20.790594+0000", 2024-04-20T17:44:41.875 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0160000000000002e-05 2024-04-20T17:44:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:20.976126+0000", 2024-04-20T17:44:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.18553197599999999 2024-04-20T17:44:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:44:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:20.976229+0000", 2024-04-20T17:44:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000102815 2024-04-20T17:44:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:44:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:20.976381+0000", 2024-04-20T17:44:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015173199999999999 2024-04-20T17:44:41.877 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.877 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.877 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:44:41.877 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:20.990421+0000", 2024-04-20T17:44:41.877 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014040166999999999 2024-04-20T17:44:41.877 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.877 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.877 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:41.877 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:21.046014+0000", 2024-04-20T17:44:41.877 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.055593208999999998 2024-04-20T17:44:41.877 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.877 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.877 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:44:41.877 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:21.046069+0000", 2024-04-20T17:44:41.877 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4607000000000001e-05 2024-04-20T17:44:41.878 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.878 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.878 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:44:41.878 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:21.049142+0000", 2024-04-20T17:44:41.878 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0030737989999999999 2024-04-20T17:44:41.878 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.878 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:44:41.878 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.878 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.878 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.878 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:210138 7.6 7:6703f753:::benchmark_data_smithi171_161871_object64412:head [delete] snapc 0=[] ondisk+write+known_if_redirected e747)", 2024-04-20T17:44:41.878 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:41:26.392472+0000", 2024-04-20T17:44:41.878 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 195.197989608, 2024-04-20T17:44:41.878 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.26567666200000001, 2024-04-20T17:44:41.878 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:44:41.879 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:44:41.879 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:41.879 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:44:41.879 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:44:41.879 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:41.879 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 210138 2024-04-20T17:44:41.879 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.879 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:44:41.879 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.879 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:44:41.879 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.392472+0000", 2024-04-20T17:44:41.879 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.879 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.879 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.880 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:44:41.880 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.392472+0000", 2024-04-20T17:44:41.880 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:44:41.880 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.880 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.880 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:44:41.880 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.392473+0000", 2024-04-20T17:44:41.880 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.559e-06 2024-04-20T17:44:41.880 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.880 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.880 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:44:41.880 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.392478+0000", 2024-04-20T17:44:41.880 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5719999999999996e-06 2024-04-20T17:44:41.880 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.880 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.880 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:44:41.881 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.392487+0000", 2024-04-20T17:44:41.881 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0129999999999999e-06 2024-04-20T17:44:41.881 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.881 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.881 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:44:41.881 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.392492+0000", 2024-04-20T17:44:41.881 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3909999999999997e-06 2024-04-20T17:44:41.881 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.881 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.881 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.881 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.489582+0000", 2024-04-20T17:44:41.881 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.097089462000000001 2024-04-20T17:44:41.881 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.881 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.881 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:44:41.882 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.489627+0000", 2024-04-20T17:44:41.882 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5420000000000002e-05 2024-04-20T17:44:41.882 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.882 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.882 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:44:41.882 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.599066+0000", 2024-04-20T17:44:41.882 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.109438903 2024-04-20T17:44:41.882 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.882 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.882 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:44:41.882 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.599129+0000", 2024-04-20T17:44:41.882 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3441999999999993e-05 2024-04-20T17:44:41.882 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.882 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.883 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:44:41.883 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.599215+0000", 2024-04-20T17:44:41.883 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5260999999999996e-05 2024-04-20T17:44:41.883 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.883 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.883 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:44:41.883 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.612840+0000", 2024-04-20T17:44:41.883 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013625671000000001 2024-04-20T17:44:41.883 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.883 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.883 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:41.883 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.655297+0000", 2024-04-20T17:44:41.883 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.042456493999999997 2024-04-20T17:44:41.883 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.883 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.884 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:44:41.884 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.655358+0000", 2024-04-20T17:44:41.884 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1696e-05 2024-04-20T17:44:41.884 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:44:41.884 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:44:41.884 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:44:41.884 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.658148+0000", 2024-04-20T17:44:41.884 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0027897780000000001 2024-04-20T17:44:41.884 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.884 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:44:41.884 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.884 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:44:41.884 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:44:41.884 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:44:41.885 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:44:41.886 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:44:41.886 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:44:41.886 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:44:41.886 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:44:41.886 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:44:41.898 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:44:41.930 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:44:41.935 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:41.958 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:44:41.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:41 smithi005 ceph-mon[39562]: pgmap v2150: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.8 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-04-20T17:44:41.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:41 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2403724956' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:44:41.966 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:44:41.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:41.970 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:44:41.994 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:44:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:41 smithi106 ceph-mon[44423]: pgmap v2150: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.8 GiB data, 20 GiB used, 784 GiB / 805 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-04-20T17:44:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:41 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2403724956' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:44:42.026 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:44:42.027 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:44:42.027 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:44:42.027 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:44:42.027 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:44:42.028 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:44:42.028 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:44:42.028 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:44:42.028 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:44:42.028 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:44:42.028 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:44:42.028 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:44:42.028 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:44:42.071 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:44:42.108 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:42.306 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:44:42.433 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 74 16 131399 131383 110.928 66.4375 0.0109044 0.00901056 2024-04-20T17:44:42.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:42.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:42.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:42.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:42.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:42.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:42.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:42.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:42.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:42.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:42 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:43.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:42 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:43.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:43.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:43.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:43.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:43.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:43.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:43.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:43.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:43.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:43.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:42 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:43.427 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-20T17:44:43.434 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 75 16 132458 132442 110.33 66.1875 0.0154158 0.00906127 2024-04-20T17:44:43.473 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:44:43.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:43.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:44:43.580 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:44:43.626 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:43.666 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:44:43.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:43.673 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:44:43.710 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:43 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:43.710 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:43 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:43.710 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:43 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:44:43.710 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:43 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:44:43.710 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:43 smithi106 ceph-mon[44423]: pgmap v2151: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.9 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 61 MiB/s wr, 968 op/s 2024-04-20T17:44:43.710 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:43 smithi106 ceph-mon[44423]: osdmap e804: 12 total, 12 up, 9 in 2024-04-20T17:44:43.719 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:44:43.774 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:44:43.820 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:43.853 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:44:43.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:43.860 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:44:43.961 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:44:43.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:43 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:43.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:43 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:43.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:43 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:44:43.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:43 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:44:43.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:43 smithi005 ceph-mon[39562]: pgmap v2151: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.9 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 61 MiB/s wr, 968 op/s 2024-04-20T17:44:43.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:43 smithi005 ceph-mon[39562]: osdmap e804: 12 total, 12 up, 9 in 2024-04-20T17:44:43.989 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:44.029 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:44:44.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:44.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:44:44.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:43 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:44.149 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:43 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:44.150 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:43 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:44:44.150 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:43 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:44:44.150 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:43 smithi171 ceph-mon[44189]: pgmap v2151: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.9 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 61 MiB/s wr, 968 op/s 2024-04-20T17:44:44.150 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:43 smithi171 ceph-mon[44189]: osdmap e804: 12 total, 12 up, 9 in 2024-04-20T17:44:44.150 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:44:44.178 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:44.199 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:44:44.261 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:44:44.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:44.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:44:44.364 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:44:44.405 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:44.432 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:44:44.434 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 76 16 133555 133539 109.781 68.5625 0.00644375 0.0091075 2024-04-20T17:44:44.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:44.435 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:44:44.537 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:44:44.564 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:44.638 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:44:44.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:44.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:44:44.743 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:44:44.787 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:45.150 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:44:45.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:45.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:44:45.255 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:44:45.314 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:45.357 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:44:45.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:45.364 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:44:45.435 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 77 16 134634 134618 109.231 67.4375 0.0100603 0.00915217 2024-04-20T17:44:45.465 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:44:45.496 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:45.594 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:44:45.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:45.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:44:45.699 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:44:45.737 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:45.745 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 8, 9, 10, 11] out_osds: [6, 1, 7] dead_osds: [] live_osds: [0, 4, 5, 8, 9, 10, 11, 2, 6, 3, 1, 7] 2024-04-20T17:44:45.745 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:44:45.745 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-04-20T17:44:45.746 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd in 1 2024-04-20T17:44:45.765 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:44:45.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:45.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:44:45.876 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:44:45.909 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:44:45.909 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:44:45.932 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:45 smithi106 ceph-mon[44423]: pgmap v2153: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.0 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-20T17:44:45.933 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:45.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:45 smithi005 ceph-mon[39562]: pgmap v2153: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.0 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-20T17:44:45.965 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:44:45.968 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:45 smithi171 ceph-mon[44189]: pgmap v2153: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.0 GiB data, 21 GiB used, 784 GiB / 805 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-20T17:44:45.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:45.969 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:44:46.070 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:44:46.120 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:46.154 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:44:46.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:46.158 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:44:46.177 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:44:46.259 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:44:46.304 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:46.409 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:44:46.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:46.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:44:46.435 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 78 16 135689 135673 108.676 65.9375 0.0177423 0.0091997 2024-04-20T17:44:46.472 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:44:46.515 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:44:46.545 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:47.435 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 79 16 136714 136698 108.111 64.0625 0.00949217 0.00924558 2024-04-20T17:44:47.732 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:44:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:47 smithi106 ceph-mon[44423]: pgmap v2154: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.2 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-20T17:44:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:47 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4051943523' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:44:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:47 smithi106 ceph-mon[44423]: from='client.27674 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:44:48.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:47 smithi171 ceph-mon[44189]: pgmap v2154: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.2 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-20T17:44:48.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:47 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4051943523' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:44:48.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:47 smithi171 ceph-mon[44189]: from='client.27674 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:44:48.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:47 smithi005 ceph-mon[39562]: pgmap v2154: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.2 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-20T17:44:48.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:47 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4051943523' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:44:48.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:47 smithi005 ceph-mon[39562]: from='client.27674 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:44:48.286 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-20T17:44:48.297 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:44:48.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:48.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:44:48.402 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:44:48.435 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:44:48.435876 min lat: 0.00104535 max lat: 0.216981 avg lat: 0.00928903 2024-04-20T17:44:48.435 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:44:48.435 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 80 16 137833 137817 107.634 69.9375 0.0218093 0.00928903 2024-04-20T17:44:48.547 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:48.572 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:44:48.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:48.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:44:48.676 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:44:48.703 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:48.723 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:44:48.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:48.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:44:48.827 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:44:48.864 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:44:48.866 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:48 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:44:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:48 smithi106 ceph-mon[44423]: from='client.27674 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:44:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:48 smithi106 ceph-mon[44423]: osdmap e805: 12 total, 12 up, 9 in 2024-04-20T17:44:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:48 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:44:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:48 smithi171 ceph-mon[44189]: from='client.27674 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:44:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:48 smithi171 ceph-mon[44189]: osdmap e805: 12 total, 12 up, 9 in 2024-04-20T17:44:49.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:48 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:44:49.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:48 smithi005 ceph-mon[39562]: from='client.27674 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:44:49.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:48 smithi005 ceph-mon[39562]: osdmap e805: 12 total, 12 up, 9 in 2024-04-20T17:44:49.435 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 81 16 138890 138874 107.121 66.0625 0.012712 0.00933384 2024-04-20T17:44:49.772 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:44:50.008 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:49 smithi005 ceph-mon[39562]: pgmap v2156: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.2 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 51 MiB/s wr, 820 op/s 2024-04-20T17:44:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:49 smithi106 ceph-mon[44423]: pgmap v2156: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.2 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 51 MiB/s wr, 820 op/s 2024-04-20T17:44:50.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:49 smithi171 ceph-mon[44189]: pgmap v2156: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.2 GiB data, 21 GiB used, 783 GiB / 805 GiB avail; 51 MiB/s wr, 820 op/s 2024-04-20T17:44:50.257 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:44:50.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:50.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:44:50.362 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:44:50.421 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:50.436 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 82 16 140003 139987 106.662 69.5625 0.0154216 0.00937399 2024-04-20T17:44:50.447 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:44:50.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:50.451 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:44:50.552 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:44:50.582 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:50.638 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:44:50.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:50.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:44:50.742 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:44:50.796 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:50.935 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:44:50.935 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:44:51.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:50 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mon-a[39539]: 2024-04-20T17:44:50.946+0000 7f73b92a1700 -1 mon.a@0(leader).osd e805 definitely_dead 0 2024-04-20T17:44:51.437 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 83 16 141087 141071 106.192 67.75 0.0139631 0.00941443 2024-04-20T17:44:51.670 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:44:51.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:51.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:44:51.752 INFO:teuthology.orchestra.run.smithi005.stderr:marked in osd.1. 2024-04-20T17:44:51.755 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:51 smithi106 ceph-mon[44423]: pgmap v2157: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.4 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-04-20T17:44:51.755 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:51 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2745042382' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-04-20T17:44:51.775 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:44:51.803 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:44:51.809 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-20T17:44:51.815 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:51.942 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:44:51.946 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:51 smithi005 ceph-mon[39562]: pgmap v2157: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.4 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-04-20T17:44:51.946 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:51 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2745042382' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-04-20T17:44:51.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:51.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:44:52.049 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:44:52.074 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:44:52.076 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:51 smithi171 ceph-mon[44189]: pgmap v2157: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 8.4 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-04-20T17:44:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:51 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2745042382' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-04-20T17:44:52.096 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:44:52.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:52.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:44:52.201 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:44:52.228 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:52.326 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:44:52.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:52.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:44:52.372 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:44:52.405 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:44:52.411 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:44:52.430 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:44:52.431 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:44:52.437 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 84 16 141522 141506 105.251 27.1875 0.00397726 0.00942845 2024-04-20T17:44:52.451 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:44:52.461 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:52.474 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:44:52.494 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:44:52.514 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:44:52.534 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:44:52.554 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:44:52.574 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:44:52.594 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:44:52.595 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:44:52.596 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:44:52.596 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:44:52.596 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:44:52.596 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:44:52.596 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:44:52.596 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:44:52.596 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:44:52.596 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:44:52.596 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:44:52.596 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:44:52.596 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:44:53.010 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:52 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2745042382' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-04-20T17:44:53.010 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:52 smithi005 ceph-mon[39562]: osdmap e806: 12 total, 12 up, 10 in 2024-04-20T17:44:53.010 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:52 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3505029973' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:44:53.010 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:52 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:53.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:52 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2745042382' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-04-20T17:44:53.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:52 smithi106 ceph-mon[44423]: osdmap e806: 12 total, 12 up, 10 in 2024-04-20T17:44:53.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:52 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3505029973' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:44:53.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:52 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2745042382' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-04-20T17:44:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:52 smithi171 ceph-mon[44189]: osdmap e806: 12 total, 12 up, 10 in 2024-04-20T17:44:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:52 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3505029973' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:44:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:52 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:53.438 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 85 16 141534 141518 104.022 0.75 1.65197 0.00952098 2024-04-20T17:44:53.981 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:44:53.981 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-04-20T17:44:53.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-04-20T17:44:53.982 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:44:53.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.982 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:155954 6.2 6:44e5c6b4:::benchmark_data_smithi171_147972_object155953:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e646)", 2024-04-20T17:44:53.982 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:04.109965+0000", 2024-04-20T17:44:53.982 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 589.86917198599997, 2024-04-20T17:44:53.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.134973547, 2024-04-20T17:44:53.982 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:44:53.982 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:44:53.982 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:53.982 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:44:53.982 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:44:53.982 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:44:53.983 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 155954 2024-04-20T17:44:53.983 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.983 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:44:53.983 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.983 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:44:53.983 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.109965+0000", 2024-04-20T17:44:53.983 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:53.983 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.983 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.983 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:44:53.983 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.109965+0000", 2024-04-20T17:44:53.983 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:53.983 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.983 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.983 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:44:53.984 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.109967+0000", 2024-04-20T17:44:53.984 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.17e-06 2024-04-20T17:44:53.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.984 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:44:53.984 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.110007+0000", 2024-04-20T17:44:53.984 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9799999999999998e-05 2024-04-20T17:44:53.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.984 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:44:53.984 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.110019+0000", 2024-04-20T17:44:53.984 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1402e-05 2024-04-20T17:44:53.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:44:53.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.110026+0000", 2024-04-20T17:44:53.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3490000000000003e-06 2024-04-20T17:44:53.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:44:53.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.110072+0000", 2024-04-20T17:44:53.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6300000000000001e-05 2024-04-20T17:44:53.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:44:53.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.110206+0000", 2024-04-20T17:44:53.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013326500000000001 2024-04-20T17:44:53.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.986 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:44:53.986 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.110350+0000", 2024-04-20T17:44:53.986 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014443400000000001 2024-04-20T17:44:53.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.986 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.986 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:44:53.986 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.120041+0000", 2024-04-20T17:44:53.986 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0096908340000000006 2024-04-20T17:44:53.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.986 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.986 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:53.986 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.244862+0000", 2024-04-20T17:44:53.986 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12482156899999999 2024-04-20T17:44:53.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.987 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.987 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:44:53.987 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.244895+0000", 2024-04-20T17:44:53.987 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2886000000000003e-05 2024-04-20T17:44:53.987 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.987 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.987 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:44:53.987 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.244939+0000", 2024-04-20T17:44:53.987 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3538000000000002e-05 2024-04-20T17:44:53.987 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:53.987 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:44:53.987 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:53.987 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.987 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.988 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:155956 6.2 6:45a14e0f:::benchmark_data_smithi171_147972_object155955:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e646)", 2024-04-20T17:44:53.988 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:04.113314+0000", 2024-04-20T17:44:53.988 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 589.86582355600001, 2024-04-20T17:44:53.988 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13176359100000001, 2024-04-20T17:44:53.988 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:44:53.988 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:44:53.988 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:53.988 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:44:53.988 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:44:53.988 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:44:53.988 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 155956 2024-04-20T17:44:53.988 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.988 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:44:53.988 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.989 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:44:53.989 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.113314+0000", 2024-04-20T17:44:53.989 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:53.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.989 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.989 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:44:53.989 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.113314+0000", 2024-04-20T17:44:53.989 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:53.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.989 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.990 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:44:53.990 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.113315+0000", 2024-04-20T17:44:53.990 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4559999999999999e-06 2024-04-20T17:44:53.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.990 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.990 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:44:53.990 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.113379+0000", 2024-04-20T17:44:53.990 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3938999999999999e-05 2024-04-20T17:44:53.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.990 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.990 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:44:53.991 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.113389+0000", 2024-04-20T17:44:53.991 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6479999999999998e-06 2024-04-20T17:44:53.991 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.991 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.991 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:44:53.991 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.113395+0000", 2024-04-20T17:44:53.991 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7559999999999996e-06 2024-04-20T17:44:53.991 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.991 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.991 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:44:53.991 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114959+0000", 2024-04-20T17:44:53.991 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001564766 2024-04-20T17:44:53.991 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.991 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.992 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:44:53.992 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.115095+0000", 2024-04-20T17:44:53.992 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000135566 2024-04-20T17:44:53.992 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.992 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.992 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:44:53.992 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.115239+0000", 2024-04-20T17:44:53.992 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014454600000000001 2024-04-20T17:44:53.992 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.992 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.992 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:53.992 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.244987+0000", 2024-04-20T17:44:53.992 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12974762200000001 2024-04-20T17:44:53.992 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.993 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:44:53.993 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.244997+0000", 2024-04-20T17:44:53.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6099999999999995e-06 2024-04-20T17:44:53.993 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.993 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:44:53.993 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.245027+0000", 2024-04-20T17:44:53.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0741000000000003e-05 2024-04-20T17:44:53.993 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.993 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:44:53.993 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.245077+0000", 2024-04-20T17:44:53.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9941000000000002e-05 2024-04-20T17:44:53.993 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:53.994 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:44:53.994 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:53.994 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.994 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.994 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:155958 6.2 6:47045901:::benchmark_data_smithi171_147972_object155957:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e646)", 2024-04-20T17:44:53.994 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:04.114372+0000", 2024-04-20T17:44:53.994 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 589.86476493999999, 2024-04-20T17:44:53.994 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13086056099999999, 2024-04-20T17:44:53.994 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:44:53.994 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:44:53.994 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:53.994 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:44:53.994 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:44:53.994 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:44:53.995 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 155958 2024-04-20T17:44:53.995 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.995 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:44:53.995 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.995 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:44:53.995 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114372+0000", 2024-04-20T17:44:53.995 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:53.996 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.996 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.996 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:44:53.996 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114372+0000", 2024-04-20T17:44:53.996 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:53.996 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.996 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.996 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:44:53.996 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114374+0000", 2024-04-20T17:44:53.997 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5370000000000001e-06 2024-04-20T17:44:53.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.997 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:44:53.997 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114433+0000", 2024-04-20T17:44:53.997 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9064000000000002e-05 2024-04-20T17:44:53.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.997 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:44:53.997 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114443+0000", 2024-04-20T17:44:53.997 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0236e-05 2024-04-20T17:44:53.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.997 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:44:53.998 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114450+0000", 2024-04-20T17:44:53.998 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1940000000000001e-06 2024-04-20T17:44:53.998 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.998 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:44:53.998 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.120077+0000", 2024-04-20T17:44:53.998 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.005626458 2024-04-20T17:44:53.998 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.998 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:44:53.998 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.232545+0000", 2024-04-20T17:44:53.998 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.112467761 2024-04-20T17:44:53.998 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.999 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-04-20T17:44:53.999 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.232710+0000", 2024-04-20T17:44:53.999 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016564600000000001 2024-04-20T17:44:53.999 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.999 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.999 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:44:53.999 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.245088+0000", 2024-04-20T17:44:53.999 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012377384999999999 2024-04-20T17:44:53.999 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:53.999 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:53.999 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:53.999 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.245134+0000", 2024-04-20T17:44:53.999 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5926e-05 2024-04-20T17:44:53.999 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.000 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.000 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:44:54.000 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.245183+0000", 2024-04-20T17:44:54.000 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9815999999999999e-05 2024-04-20T17:44:54.000 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.000 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.000 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:44:54.000 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.245233+0000", 2024-04-20T17:44:54.000 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9537999999999999e-05 2024-04-20T17:44:54.000 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.000 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:44:54.000 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.000 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.001 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.001 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26882.0:155959 6.e 6:7825f00e:::benchmark_data_smithi171_147972_object155958:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e646)", 2024-04-20T17:44:54.001 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:35:04.114461+0000", 2024-04-20T17:44:54.001 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 589.86467603200003, 2024-04-20T17:44:54.001 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12842199500000001, 2024-04-20T17:44:54.001 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:44:54.001 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:44:54.001 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:54.001 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:44:54.001 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26882", 2024-04-20T17:44:54.001 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:44:54.001 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 155959 2024-04-20T17:44:54.001 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.002 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:44:54.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.002 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:44:54.002 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114461+0000", 2024-04-20T17:44:54.002 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.002 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:44:54.002 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114461+0000", 2024-04-20T17:44:54.002 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.002 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:44:54.002 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114462+0000", 2024-04-20T17:44:54.002 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7199999999999997e-07 2024-04-20T17:44:54.003 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.003 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:44:54.003 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114522+0000", 2024-04-20T17:44:54.003 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9587999999999999e-05 2024-04-20T17:44:54.003 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.003 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:44:54.003 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114528+0000", 2024-04-20T17:44:54.003 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7039999999999997e-06 2024-04-20T17:44:54.003 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.003 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:44:54.003 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.114534+0000", 2024-04-20T17:44:54.004 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.981e-06 2024-04-20T17:44:54.004 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.004 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.004 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:44:54.004 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.116285+0000", 2024-04-20T17:44:54.004 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001750754 2024-04-20T17:44:54.004 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.004 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.004 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:44:54.004 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.116400+0000", 2024-04-20T17:44:54.004 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011446999999999999 2024-04-20T17:44:54.004 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.004 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.004 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2", 2024-04-20T17:44:54.004 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.116529+0000", 2024-04-20T17:44:54.005 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012922300000000001 2024-04-20T17:44:54.005 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.005 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.005 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:54.005 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.233387+0000", 2024-04-20T17:44:54.005 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.11685820099999999 2024-04-20T17:44:54.005 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.005 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.005 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:44:54.005 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.242762+0000", 2024-04-20T17:44:54.005 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0093748029999999993 2024-04-20T17:44:54.005 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.005 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.005 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:44:54.005 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.242820+0000", 2024-04-20T17:44:54.006 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8207e-05 2024-04-20T17:44:54.006 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.006 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.006 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:44:54.006 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:35:04.242883+0000", 2024-04-20T17:44:54.006 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3192000000000001e-05 2024-04-20T17:44:54.006 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.006 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:44:54.006 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.006 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.006 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.006 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27088.0:176948 7.1 7:87bf052c:::benchmark_data_smithi171_161871_object31222:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:44:54.006 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:40:45.945786+0000", 2024-04-20T17:44:54.006 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 248.03335091, 2024-04-20T17:44:54.007 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21713122500000001, 2024-04-20T17:44:54.007 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:44:54.007 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:44:54.007 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:54.007 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:44:54.007 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27088", 2024-04-20T17:44:54.007 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:54.007 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 176948 2024-04-20T17:44:54.007 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.007 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:44:54.007 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.007 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:44:54.007 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.945786+0000", 2024-04-20T17:44:54.007 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.007 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.008 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.008 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:44:54.008 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.945786+0000", 2024-04-20T17:44:54.008 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.008 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.008 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:44:54.008 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.945789+0000", 2024-04-20T17:44:54.008 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0900000000000001e-06 2024-04-20T17:44:54.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.008 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.008 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:44:54.008 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.945807+0000", 2024-04-20T17:44:54.008 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7198e-05 2024-04-20T17:44:54.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.009 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.009 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:44:54.009 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.945826+0000", 2024-04-20T17:44:54.009 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8864000000000001e-05 2024-04-20T17:44:54.009 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.009 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.009 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:44:54.009 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.945850+0000", 2024-04-20T17:44:54.009 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4423999999999999e-05 2024-04-20T17:44:54.009 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.009 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.009 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:44:54.009 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.945909+0000", 2024-04-20T17:44:54.009 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9135999999999998e-05 2024-04-20T17:44:54.009 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.010 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.010 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:44:54.010 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.946022+0000", 2024-04-20T17:44:54.010 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000112977 2024-04-20T17:44:54.010 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.010 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.010 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:44:54.010 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.946103+0000", 2024-04-20T17:44:54.010 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1363e-05 2024-04-20T17:44:54.010 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.010 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.010 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:44:54.010 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.949973+0000", 2024-04-20T17:44:54.010 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0038697499999999999 2024-04-20T17:44:54.011 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.011 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.011 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:54.011 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.162849+0000", 2024-04-20T17:44:54.011 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.212876177 2024-04-20T17:44:54.011 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.011 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.011 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:44:54.011 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.162885+0000", 2024-04-20T17:44:54.011 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5908000000000002e-05 2024-04-20T17:44:54.011 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.011 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.011 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:44:54.012 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.162918+0000", 2024-04-20T17:44:54.012 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2338000000000001e-05 2024-04-20T17:44:54.012 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.012 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:44:54.012 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.012 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.012 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.012 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27088.0:176957 7.1 7:8429106d:::benchmark_data_smithi171_161871_object31231:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:44:54.012 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:40:45.960496+0000", 2024-04-20T17:44:54.012 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 248.01864126500001, 2024-04-20T17:44:54.012 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20636091400000001, 2024-04-20T17:44:54.012 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:44:54.012 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:44:54.012 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:54.012 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:44:54.013 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27088", 2024-04-20T17:44:54.013 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:54.013 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 176957 2024-04-20T17:44:54.013 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.013 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:44:54.013 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.013 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:44:54.013 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.960496+0000", 2024-04-20T17:44:54.013 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.013 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.013 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.013 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:44:54.013 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.960496+0000", 2024-04-20T17:44:54.013 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.014 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.014 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.014 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:44:54.014 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.960501+0000", 2024-04-20T17:44:54.014 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4710000000000001e-06 2024-04-20T17:44:54.014 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.014 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.014 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:44:54.014 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.960515+0000", 2024-04-20T17:44:54.014 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4215e-05 2024-04-20T17:44:54.014 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.014 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.014 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:44:54.014 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.960543+0000", 2024-04-20T17:44:54.015 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8041000000000001e-05 2024-04-20T17:44:54.015 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.015 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:44:54.015 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.960557+0000", 2024-04-20T17:44:54.015 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4632e-05 2024-04-20T17:44:54.015 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.015 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:44:54.015 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.960623+0000", 2024-04-20T17:44:54.015 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5154000000000007e-05 2024-04-20T17:44:54.015 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.015 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:44:54.016 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.960731+0000", 2024-04-20T17:44:54.016 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010856 2024-04-20T17:44:54.016 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.016 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.016 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:44:54.016 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.960816+0000", 2024-04-20T17:44:54.016 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5220000000000001e-05 2024-04-20T17:44:54.016 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.016 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.016 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:44:54.016 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.964987+0000", 2024-04-20T17:44:54.016 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0041703909999999999 2024-04-20T17:44:54.016 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.016 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.016 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:54.017 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.166798+0000", 2024-04-20T17:44:54.017 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20181174399999999 2024-04-20T17:44:54.017 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.017 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.017 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:44:54.017 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.166831+0000", 2024-04-20T17:44:54.017 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2106000000000001e-05 2024-04-20T17:44:54.017 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.017 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.017 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:44:54.017 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.166857+0000", 2024-04-20T17:44:54.017 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6380000000000002e-05 2024-04-20T17:44:54.017 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.017 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:44:54.018 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.018 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.018 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.018 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27088.0:176969 7.1 7:87ae8b31:::benchmark_data_smithi171_161871_object31243:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:44:54.018 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:40:45.972340+0000", 2024-04-20T17:44:54.018 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 248.00679760700001, 2024-04-20T17:44:54.018 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.194623088, 2024-04-20T17:44:54.018 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:44:54.018 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:44:54.018 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:54.018 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:44:54.018 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27088", 2024-04-20T17:44:54.018 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:54.018 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 176969 2024-04-20T17:44:54.019 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.019 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:44:54.019 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.019 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:44:54.019 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.972340+0000", 2024-04-20T17:44:54.019 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.019 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.019 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.019 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:44:54.019 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.972340+0000", 2024-04-20T17:44:54.019 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.019 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.019 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.019 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:44:54.020 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.972342+0000", 2024-04-20T17:44:54.020 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.469e-06 2024-04-20T17:44:54.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.020 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:44:54.020 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.972350+0000", 2024-04-20T17:44:54.020 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3479999999999998e-06 2024-04-20T17:44:54.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.020 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:44:54.020 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.972362+0000", 2024-04-20T17:44:54.020 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2376e-05 2024-04-20T17:44:54.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.021 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:44:54.021 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.972373+0000", 2024-04-20T17:44:54.021 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0847e-05 2024-04-20T17:44:54.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.021 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:44:54.021 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.972429+0000", 2024-04-20T17:44:54.021 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6762000000000003e-05 2024-04-20T17:44:54.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.021 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:44:54.021 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.972526+0000", 2024-04-20T17:44:54.021 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6280999999999995e-05 2024-04-20T17:44:54.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:44:54.022 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.972610+0000", 2024-04-20T17:44:54.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4726000000000002e-05 2024-04-20T17:44:54.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:44:54.022 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.976503+0000", 2024-04-20T17:44:54.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0038928159999999999 2024-04-20T17:44:54.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:54.022 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.166898+0000", 2024-04-20T17:44:54.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19039439999999999 2024-04-20T17:44:54.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:44:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.166931+0000", 2024-04-20T17:44:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3046999999999997e-05 2024-04-20T17:44:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:44:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.166963+0000", 2024-04-20T17:44:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2016e-05 2024-04-20T17:44:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:44:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27088.0:176983 7.1 7:8915abf7:::benchmark_data_smithi171_161871_object31257:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:44:54.023 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:40:45.983813+0000", 2024-04-20T17:44:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 247.995324635, 2024-04-20T17:44:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.18323587099999999, 2024-04-20T17:44:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:44:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:44:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:44:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27088", 2024-04-20T17:44:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 176983 2024-04-20T17:44:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:44:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:44:54.024 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.983813+0000", 2024-04-20T17:44:54.025 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.025 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.025 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:44:54.025 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.983813+0000", 2024-04-20T17:44:54.025 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.025 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.025 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:44:54.025 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.983815+0000", 2024-04-20T17:44:54.025 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1469999999999999e-06 2024-04-20T17:44:54.025 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.025 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:44:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.983821+0000", 2024-04-20T17:44:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0909999999999998e-06 2024-04-20T17:44:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:44:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.983831+0000", 2024-04-20T17:44:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0387000000000001e-05 2024-04-20T17:44:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:44:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.983837+0000", 2024-04-20T17:44:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8510000000000004e-06 2024-04-20T17:44:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:44:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.983882+0000", 2024-04-20T17:44:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4527999999999998e-05 2024-04-20T17:44:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:44:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.983984+0000", 2024-04-20T17:44:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010283199999999999 2024-04-20T17:44:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:44:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.984062+0000", 2024-04-20T17:44:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7550000000000001e-05 2024-04-20T17:44:54.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:44:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:45.987383+0000", 2024-04-20T17:44:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0033208880000000001 2024-04-20T17:44:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.166999+0000", 2024-04-20T17:44:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.17961604 2024-04-20T17:44:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:44:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.167026+0000", 2024-04-20T17:44:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7203999999999999e-05 2024-04-20T17:44:54.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.029 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:44:54.029 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.167049+0000", 2024-04-20T17:44:54.029 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2353000000000001e-05 2024-04-20T17:44:54.029 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.029 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:44:54.029 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.029 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27088.0:177001 7.1 7:8fadefcc:::benchmark_data_smithi171_161871_object31275:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:44:54.029 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:40:46.005497+0000", 2024-04-20T17:44:54.029 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 247.97364000600001, 2024-04-20T17:44:54.029 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.16162363699999999, 2024-04-20T17:44:54.029 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:44:54.030 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:44:54.030 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:54.030 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:44:54.030 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27088", 2024-04-20T17:44:54.030 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:54.030 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 177001 2024-04-20T17:44:54.030 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.030 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:44:54.030 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.030 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:44:54.030 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.005497+0000", 2024-04-20T17:44:54.030 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.030 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.030 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.030 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:44:54.031 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.005497+0000", 2024-04-20T17:44:54.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.031 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:44:54.031 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.005500+0000", 2024-04-20T17:44:54.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5900000000000002e-06 2024-04-20T17:44:54.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.031 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:44:54.031 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.005507+0000", 2024-04-20T17:44:54.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7499999999999997e-06 2024-04-20T17:44:54.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.032 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:44:54.032 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.005523+0000", 2024-04-20T17:44:54.032 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6824e-05 2024-04-20T17:44:54.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.032 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:44:54.032 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.005531+0000", 2024-04-20T17:44:54.032 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.255e-06 2024-04-20T17:44:54.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.032 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:44:54.032 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.005579+0000", 2024-04-20T17:44:54.032 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7883000000000002e-05 2024-04-20T17:44:54.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.033 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:44:54.033 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.005676+0000", 2024-04-20T17:44:54.033 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7600000000000001e-05 2024-04-20T17:44:54.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.033 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:44:54.033 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.005757+0000", 2024-04-20T17:44:54.033 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0865000000000006e-05 2024-04-20T17:44:54.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.033 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:44:54.033 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.009145+0000", 2024-04-20T17:44:54.033 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0033883749999999999 2024-04-20T17:44:54.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.034 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:54.034 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.167071+0000", 2024-04-20T17:44:54.034 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.157925866 2024-04-20T17:44:54.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.034 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:44:54.034 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.167100+0000", 2024-04-20T17:44:54.034 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8238999999999998e-05 2024-04-20T17:44:54.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.034 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:44:54.034 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.167121+0000", 2024-04-20T17:44:54.035 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.139e-05 2024-04-20T17:44:54.035 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.035 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:44:54.035 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.035 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.035 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27088.0:177014 7.1 7:8bbdf0df:::benchmark_data_smithi171_161871_object31288:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:44:54.035 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:40:46.021123+0000", 2024-04-20T17:44:54.035 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 247.95801427800001, 2024-04-20T17:44:54.035 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.154166049, 2024-04-20T17:44:54.035 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:44:54.035 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:44:54.035 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:54.035 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:44:54.036 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27088", 2024-04-20T17:44:54.036 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:54.036 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 177014 2024-04-20T17:44:54.036 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.036 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:44:54.036 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.036 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:44:54.036 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.021123+0000", 2024-04-20T17:44:54.036 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.036 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.036 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.036 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:44:54.036 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.021123+0000", 2024-04-20T17:44:54.036 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.036 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.037 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:44:54.037 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.021127+0000", 2024-04-20T17:44:54.037 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9530000000000001e-06 2024-04-20T17:44:54.037 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.037 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:44:54.037 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.021139+0000", 2024-04-20T17:44:54.037 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.205e-05 2024-04-20T17:44:54.037 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.037 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:44:54.037 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.021165+0000", 2024-04-20T17:44:54.037 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5936e-05 2024-04-20T17:44:54.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.038 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.038 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:44:54.038 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.021182+0000", 2024-04-20T17:44:54.038 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6762999999999999e-05 2024-04-20T17:44:54.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.038 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.038 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:44:54.038 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.021296+0000", 2024-04-20T17:44:54.038 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011462400000000001 2024-04-20T17:44:54.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.038 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.038 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:44:54.038 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.021466+0000", 2024-04-20T17:44:54.039 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000169464 2024-04-20T17:44:54.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.039 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:44:54.039 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.021541+0000", 2024-04-20T17:44:54.039 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4867e-05 2024-04-20T17:44:54.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.039 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:44:54.039 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.025787+0000", 2024-04-20T17:44:54.039 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0042458729999999998 2024-04-20T17:44:54.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.039 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:54.039 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.175231+0000", 2024-04-20T17:44:54.040 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14944402500000001 2024-04-20T17:44:54.040 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.040 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.040 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:44:54.040 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.175260+0000", 2024-04-20T17:44:54.040 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9801000000000001e-05 2024-04-20T17:44:54.040 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.040 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.040 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:44:54.040 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:40:46.175289+0000", 2024-04-20T17:44:54.040 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8693000000000001e-05 2024-04-20T17:44:54.040 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.040 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:44:54.040 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.040 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.041 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.041 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27088.0:254325 7.1 7:81fabaa8:::benchmark_data_smithi171_161871_object108599:head [delete] snapc 0=[] ondisk+write+known_if_redirected e756)", 2024-04-20T17:44:54.041 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:42:15.864902+0000", 2024-04-20T17:44:54.041 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 158.11423534799999, 2024-04-20T17:44:54.041 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13091608299999999, 2024-04-20T17:44:54.041 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:44:54.041 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:44:54.041 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:54.041 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:44:54.041 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27088", 2024-04-20T17:44:54.041 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:54.041 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 254325 2024-04-20T17:44:54.041 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.041 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:44:54.042 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.042 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:44:54.042 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:15.864902+0000", 2024-04-20T17:44:54.042 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.042 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.042 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.042 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:44:54.042 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:15.864902+0000", 2024-04-20T17:44:54.042 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.042 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.042 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.042 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:44:54.042 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:15.864904+0000", 2024-04-20T17:44:54.042 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7719999999999999e-06 2024-04-20T17:44:54.043 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.043 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.043 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:44:54.043 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:15.864909+0000", 2024-04-20T17:44:54.043 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4890000000000003e-06 2024-04-20T17:44:54.043 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.043 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.043 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:44:54.043 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:15.864921+0000", 2024-04-20T17:44:54.043 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1457999999999999e-05 2024-04-20T17:44:54.043 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.043 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.043 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:44:54.043 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:15.864930+0000", 2024-04-20T17:44:54.044 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5710000000000004e-06 2024-04-20T17:44:54.044 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.044 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.044 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:44:54.044 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:15.919764+0000", 2024-04-20T17:44:54.044 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.054833650999999997 2024-04-20T17:44:54.044 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.044 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.044 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:44:54.044 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:15.919841+0000", 2024-04-20T17:44:54.044 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6641e-05 2024-04-20T17:44:54.044 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.044 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.044 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:44:54.044 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:15.919914+0000", 2024-04-20T17:44:54.045 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3158000000000007e-05 2024-04-20T17:44:54.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.045 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:44:54.045 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:15.931442+0000", 2024-04-20T17:44:54.045 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011528750000000001 2024-04-20T17:44:54.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.045 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:54.045 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:15.995715+0000", 2024-04-20T17:44:54.045 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.064272177999999999 2024-04-20T17:44:54.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.045 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:44:54.046 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:15.995755+0000", 2024-04-20T17:44:54.046 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0213000000000003e-05 2024-04-20T17:44:54.046 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.046 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.046 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:44:54.046 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:15.995818+0000", 2024-04-20T17:44:54.046 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3201999999999996e-05 2024-04-20T17:44:54.046 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.046 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:44:54.046 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.046 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.046 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.046 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27088.0:256916 7.1 7:80853f24:::benchmark_data_smithi171_161871_object111190:head [delete] snapc 0=[] ondisk+write+known_if_redirected e756)", 2024-04-20T17:44:54.046 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:42:19.058808+0000", 2024-04-20T17:44:54.047 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 154.920329278, 2024-04-20T17:44:54.047 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.124168206, 2024-04-20T17:44:54.047 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:44:54.047 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:44:54.047 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:54.047 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:44:54.047 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27088", 2024-04-20T17:44:54.047 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:54.047 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 256916 2024-04-20T17:44:54.047 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.047 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:44:54.047 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.047 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:44:54.047 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.058808+0000", 2024-04-20T17:44:54.047 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.048 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.048 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.048 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:44:54.048 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.058808+0000", 2024-04-20T17:44:54.048 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.048 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.048 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.048 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:44:54.048 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.058810+0000", 2024-04-20T17:44:54.048 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2859999999999998e-06 2024-04-20T17:44:54.048 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.048 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.048 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:44:54.048 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.058817+0000", 2024-04-20T17:44:54.049 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6529999999999997e-06 2024-04-20T17:44:54.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.049 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:44:54.049 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.058829+0000", 2024-04-20T17:44:54.049 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1802e-05 2024-04-20T17:44:54.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.049 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:44:54.049 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.058837+0000", 2024-04-20T17:44:54.049 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0590000000000003e-06 2024-04-20T17:44:54.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.049 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:44:54.049 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.122995+0000", 2024-04-20T17:44:54.050 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.064157727999999997 2024-04-20T17:44:54.050 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.050 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.050 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:44:54.050 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.123086+0000", 2024-04-20T17:44:54.050 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1535999999999999e-05 2024-04-20T17:44:54.050 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.050 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.050 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:44:54.050 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.123178+0000", 2024-04-20T17:44:54.050 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2090999999999996e-05 2024-04-20T17:44:54.050 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.050 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.051 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:44:54.051 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.126883+0000", 2024-04-20T17:44:54.051 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037045609999999999 2024-04-20T17:44:54.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.051 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.051 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:54.051 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.182869+0000", 2024-04-20T17:44:54.051 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.055986538000000002 2024-04-20T17:44:54.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.051 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.051 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:44:54.051 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.182906+0000", 2024-04-20T17:44:54.051 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6955000000000001e-05 2024-04-20T17:44:54.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.052 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.052 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:44:54.052 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.182976+0000", 2024-04-20T17:44:54.052 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9997000000000001e-05 2024-04-20T17:44:54.052 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.052 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:44:54.052 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.052 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.052 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.052 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27088.0:257437 7.1 7:82b0512c:::benchmark_data_smithi171_161871_object111711:head [delete] snapc 0=[] ondisk+write+known_if_redirected e756)", 2024-04-20T17:44:54.052 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:42:19.641555+0000", 2024-04-20T17:44:54.052 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 154.33758213799999, 2024-04-20T17:44:54.052 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13640290899999999, 2024-04-20T17:44:54.053 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:44:54.053 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:44:54.053 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:54.053 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:44:54.053 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27088", 2024-04-20T17:44:54.053 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:54.053 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 257437 2024-04-20T17:44:54.053 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.053 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:44:54.053 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.053 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:44:54.053 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.641555+0000", 2024-04-20T17:44:54.053 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.054 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.054 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.054 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:44:54.054 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.641555+0000", 2024-04-20T17:44:54.054 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.054 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.054 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.054 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:44:54.054 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.641557+0000", 2024-04-20T17:44:54.054 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.844e-06 2024-04-20T17:44:54.054 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.054 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.054 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:44:54.055 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.641564+0000", 2024-04-20T17:44:54.055 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6850000000000004e-06 2024-04-20T17:44:54.055 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.055 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.055 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:44:54.055 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.641576+0000", 2024-04-20T17:44:54.055 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2051e-05 2024-04-20T17:44:54.055 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.055 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.055 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:44:54.055 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.641583+0000", 2024-04-20T17:44:54.055 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1489999999999997e-06 2024-04-20T17:44:54.055 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.055 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.056 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:44:54.056 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.700582+0000", 2024-04-20T17:44:54.056 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.058999476000000002 2024-04-20T17:44:54.056 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.056 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.056 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:44:54.056 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.700669+0000", 2024-04-20T17:44:54.056 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6456e-05 2024-04-20T17:44:54.056 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.056 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.056 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:44:54.056 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.700752+0000", 2024-04-20T17:44:54.056 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3084000000000006e-05 2024-04-20T17:44:54.056 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.057 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.057 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:44:54.057 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.711034+0000", 2024-04-20T17:44:54.057 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010282288000000001 2024-04-20T17:44:54.057 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.057 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.057 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:54.057 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.777839+0000", 2024-04-20T17:44:54.057 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.066805146999999995 2024-04-20T17:44:54.057 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.057 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.057 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:44:54.057 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.777901+0000", 2024-04-20T17:44:54.057 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1279999999999996e-05 2024-04-20T17:44:54.058 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.058 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.058 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:44:54.058 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.777958+0000", 2024-04-20T17:44:54.058 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7448999999999998e-05 2024-04-20T17:44:54.058 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.058 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:44:54.058 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.058 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.058 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.058 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27088.0:257446 7.1 7:819c0d22:::benchmark_data_smithi171_161871_object111720:head [delete] snapc 0=[] ondisk+write+known_if_redirected e756)", 2024-04-20T17:44:54.058 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:42:19.653672+0000", 2024-04-20T17:44:54.058 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 154.325465468, 2024-04-20T17:44:54.059 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12439626099999999, 2024-04-20T17:44:54.059 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:44:54.059 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:44:54.059 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:54.059 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:44:54.059 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27088", 2024-04-20T17:44:54.059 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:54.059 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 257446 2024-04-20T17:44:54.059 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.059 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:44:54.059 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.059 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:44:54.059 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.653672+0000", 2024-04-20T17:44:54.059 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.059 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.060 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.060 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:44:54.060 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.653672+0000", 2024-04-20T17:44:54.060 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.060 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.060 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.060 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:44:54.060 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.653673+0000", 2024-04-20T17:44:54.060 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.604e-06 2024-04-20T17:44:54.060 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.060 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.060 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:44:54.060 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.653679+0000", 2024-04-20T17:44:54.060 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.417e-06 2024-04-20T17:44:54.061 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.061 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.061 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:44:54.061 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.653690+0000", 2024-04-20T17:44:54.061 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0981999999999999e-05 2024-04-20T17:44:54.061 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.061 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.061 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:44:54.061 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.653697+0000", 2024-04-20T17:44:54.061 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8990000000000004e-06 2024-04-20T17:44:54.061 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.061 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.061 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:44:54.061 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.703918+0000", 2024-04-20T17:44:54.061 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.050221430999999997 2024-04-20T17:44:54.062 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.062 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.062 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:44:54.062 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.704015+0000", 2024-04-20T17:44:54.062 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6627999999999997e-05 2024-04-20T17:44:54.062 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.062 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.062 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:44:54.062 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.704096+0000", 2024-04-20T17:44:54.062 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1543000000000002e-05 2024-04-20T17:44:54.062 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.062 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.062 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:44:54.062 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.711038+0000", 2024-04-20T17:44:54.063 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.006941661 2024-04-20T17:44:54.063 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.063 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.063 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:54.063 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.777996+0000", 2024-04-20T17:44:54.063 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.066957698999999996 2024-04-20T17:44:54.063 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.063 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.063 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:44:54.063 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.778027+0000", 2024-04-20T17:44:54.063 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1699999999999998e-05 2024-04-20T17:44:54.063 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.063 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.063 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:44:54.063 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.778068+0000", 2024-04-20T17:44:54.064 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0697e-05 2024-04-20T17:44:54.064 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.064 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:44:54.064 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.064 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.064 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.064 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27088.0:257708 7.1 7:8cdba5fd:::benchmark_data_smithi171_161871_object111982:head [delete] snapc 0=[] ondisk+write+known_if_redirected e756)", 2024-04-20T17:44:54.064 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:42:19.950159+0000", 2024-04-20T17:44:54.064 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 154.02897815, 2024-04-20T17:44:54.064 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.130724381, 2024-04-20T17:44:54.064 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:44:54.064 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:44:54.064 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:54.064 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:44:54.064 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27088", 2024-04-20T17:44:54.065 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:54.065 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 257708 2024-04-20T17:44:54.065 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.065 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:44:54.065 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.065 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:44:54.065 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.950159+0000", 2024-04-20T17:44:54.065 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.065 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.065 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.066 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:44:54.066 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.950159+0000", 2024-04-20T17:44:54.066 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.066 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.066 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.066 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:44:54.066 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.950160+0000", 2024-04-20T17:44:54.066 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.08e-06 2024-04-20T17:44:54.066 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.066 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.066 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:44:54.066 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.950164+0000", 2024-04-20T17:44:54.066 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5200000000000002e-06 2024-04-20T17:44:54.066 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.067 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.067 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:44:54.067 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.950170+0000", 2024-04-20T17:44:54.067 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7020000000000004e-06 2024-04-20T17:44:54.067 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.067 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.067 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:44:54.067 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:19.950177+0000", 2024-04-20T17:44:54.067 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.071e-06 2024-04-20T17:44:54.067 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.067 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.067 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:44:54.067 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:20.015212+0000", 2024-04-20T17:44:54.067 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.065035645000000003 2024-04-20T17:44:54.068 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.068 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.068 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:44:54.068 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:20.015343+0000", 2024-04-20T17:44:54.068 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013068500000000001 2024-04-20T17:44:54.068 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.068 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.068 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:44:54.068 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:20.015476+0000", 2024-04-20T17:44:54.068 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013286299999999999 2024-04-20T17:44:54.068 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.068 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.068 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:44:54.069 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:20.023680+0000", 2024-04-20T17:44:54.069 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0082042460000000001 2024-04-20T17:44:54.069 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.069 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.069 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:54.069 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:20.080817+0000", 2024-04-20T17:44:54.069 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.057137169000000002 2024-04-20T17:44:54.069 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.069 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.069 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:44:54.069 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:20.080849+0000", 2024-04-20T17:44:54.069 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1844000000000002e-05 2024-04-20T17:44:54.069 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.069 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.070 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:44:54.070 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:20.080884+0000", 2024-04-20T17:44:54.070 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4555999999999999e-05 2024-04-20T17:44:54.070 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.070 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:44:54.070 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.070 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.070 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.070 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27088.0:257844 7.1 7:8370f84a:::benchmark_data_smithi171_161871_object112118:head [delete] snapc 0=[] ondisk+write+known_if_redirected e756)", 2024-04-20T17:44:54.070 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:42:20.106143+0000", 2024-04-20T17:44:54.070 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 153.872994339, 2024-04-20T17:44:54.070 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.126275002, 2024-04-20T17:44:54.070 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:44:54.070 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:44:54.071 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:54.071 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:44:54.071 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27088", 2024-04-20T17:44:54.071 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:54.071 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 257844 2024-04-20T17:44:54.071 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.071 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:44:54.071 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.071 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:44:54.071 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:20.106143+0000", 2024-04-20T17:44:54.071 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.071 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.072 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.072 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:44:54.072 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:20.106143+0000", 2024-04-20T17:44:54.072 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.072 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.072 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.072 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:44:54.072 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:20.106145+0000", 2024-04-20T17:44:54.072 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.796e-06 2024-04-20T17:44:54.072 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.072 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.072 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:44:54.072 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:20.106150+0000", 2024-04-20T17:44:54.072 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3519999999999997e-06 2024-04-20T17:44:54.072 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.073 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.073 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:44:54.073 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:20.106161+0000", 2024-04-20T17:44:54.073 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1338000000000001e-05 2024-04-20T17:44:54.073 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.073 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.073 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:44:54.073 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:20.106171+0000", 2024-04-20T17:44:54.073 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.251e-06 2024-04-20T17:44:54.073 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.073 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.073 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:44:54.073 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:20.158003+0000", 2024-04-20T17:44:54.073 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.051832491000000001 2024-04-20T17:44:54.074 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.074 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.074 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:44:54.074 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:20.158084+0000", 2024-04-20T17:44:54.074 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1120999999999998e-05 2024-04-20T17:44:54.074 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.074 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.074 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:44:54.074 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:20.158168+0000", 2024-04-20T17:44:54.074 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3628999999999995e-05 2024-04-20T17:44:54.074 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.074 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.074 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:44:54.074 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:20.172043+0000", 2024-04-20T17:44:54.075 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.013875492 2024-04-20T17:44:54.075 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.075 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.075 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:54.075 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:20.232337+0000", 2024-04-20T17:44:54.075 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.060293340000000001 2024-04-20T17:44:54.075 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.075 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.075 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:44:54.075 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:20.232376+0000", 2024-04-20T17:44:54.075 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9631000000000001e-05 2024-04-20T17:44:54.075 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.075 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.075 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:44:54.076 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:20.232418+0000", 2024-04-20T17:44:54.076 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1560999999999997e-05 2024-04-20T17:44:54.076 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.076 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:44:54.076 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.076 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.076 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.076 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27088.0:258898 7.1 7:81187d7a:::benchmark_data_smithi171_161871_object113172:head [delete] snapc 0=[] ondisk+write+known_if_redirected e756)", 2024-04-20T17:44:54.076 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:42:21.310010+0000", 2024-04-20T17:44:54.076 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 152.66912730000001, 2024-04-20T17:44:54.076 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.127713251, 2024-04-20T17:44:54.076 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:44:54.076 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:44:54.076 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:54.077 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:44:54.077 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27088", 2024-04-20T17:44:54.077 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:54.077 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 258898 2024-04-20T17:44:54.077 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.077 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:44:54.077 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.077 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:44:54.077 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:21.310010+0000", 2024-04-20T17:44:54.077 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.077 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.077 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.077 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:44:54.077 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:21.310010+0000", 2024-04-20T17:44:54.078 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.078 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.078 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.078 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:44:54.078 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:21.310012+0000", 2024-04-20T17:44:54.078 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0600000000000002e-06 2024-04-20T17:44:54.078 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.078 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.078 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:44:54.078 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:21.310018+0000", 2024-04-20T17:44:54.078 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3010000000000003e-06 2024-04-20T17:44:54.078 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.078 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.079 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:44:54.079 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:21.310030+0000", 2024-04-20T17:44:54.079 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1804000000000001e-05 2024-04-20T17:44:54.079 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.079 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.079 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:44:54.080 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:21.310038+0000", 2024-04-20T17:44:54.080 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4730000000000003e-06 2024-04-20T17:44:54.080 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.080 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.080 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:44:54.080 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:21.369930+0000", 2024-04-20T17:44:54.081 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.059892039000000001 2024-04-20T17:44:54.081 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.081 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.081 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:44:54.081 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:21.370070+0000", 2024-04-20T17:44:54.081 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000140326 2024-04-20T17:44:54.081 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.081 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.081 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:44:54.081 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:21.370211+0000", 2024-04-20T17:44:54.081 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014106200000000001 2024-04-20T17:44:54.081 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.081 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.081 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:44:54.082 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:21.381122+0000", 2024-04-20T17:44:54.082 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010910736000000001 2024-04-20T17:44:54.082 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.082 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.082 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:54.082 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:21.437646+0000", 2024-04-20T17:44:54.082 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.056523819000000003 2024-04-20T17:44:54.082 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.082 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.082 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:44:54.082 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:21.437682+0000", 2024-04-20T17:44:54.082 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6881999999999997e-05 2024-04-20T17:44:54.082 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.082 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.083 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:44:54.083 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:21.437723+0000", 2024-04-20T17:44:54.083 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0748999999999999e-05 2024-04-20T17:44:54.083 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.083 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:44:54.083 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.083 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.083 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.083 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27088.0:260809 7.1 7:8755b865:::benchmark_data_smithi171_161871_object115083:head [delete] snapc 0=[] ondisk+write+known_if_redirected e757)", 2024-04-20T17:44:54.083 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:42:23.554672+0000", 2024-04-20T17:44:54.083 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 150.424465368, 2024-04-20T17:44:54.083 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13638230300000001, 2024-04-20T17:44:54.083 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:44:54.083 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:44:54.084 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:54.084 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:44:54.084 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27088", 2024-04-20T17:44:54.084 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:54.084 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 260809 2024-04-20T17:44:54.084 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.084 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:44:54.084 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.084 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:44:54.084 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:23.554672+0000", 2024-04-20T17:44:54.084 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.084 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.084 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.084 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:44:54.084 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:23.554672+0000", 2024-04-20T17:44:54.085 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.085 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.085 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.085 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:44:54.085 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:23.554674+0000", 2024-04-20T17:44:54.085 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7129999999999999e-06 2024-04-20T17:44:54.085 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.085 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.085 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:44:54.085 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:23.554680+0000", 2024-04-20T17:44:54.085 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0769999999999996e-06 2024-04-20T17:44:54.085 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.085 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.085 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:44:54.086 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:23.554691+0000", 2024-04-20T17:44:54.086 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.173e-05 2024-04-20T17:44:54.086 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.086 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.086 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:44:54.086 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:23.554699+0000", 2024-04-20T17:44:54.086 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4989999999999997e-06 2024-04-20T17:44:54.086 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.086 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.086 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:44:54.086 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:23.609492+0000", 2024-04-20T17:44:54.086 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.054793466999999998 2024-04-20T17:44:54.086 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.086 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.087 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:44:54.087 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:23.609584+0000", 2024-04-20T17:44:54.087 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1779000000000003e-05 2024-04-20T17:44:54.087 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.087 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.087 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:44:54.087 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:23.609671+0000", 2024-04-20T17:44:54.087 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7053999999999997e-05 2024-04-20T17:44:54.087 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.087 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.087 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:44:54.087 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:23.612558+0000", 2024-04-20T17:44:54.087 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002886567 2024-04-20T17:44:54.087 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.087 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.088 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:54.088 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:23.690923+0000", 2024-04-20T17:44:54.088 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.078365463999999996 2024-04-20T17:44:54.088 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.088 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.088 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:44:54.088 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:23.690991+0000", 2024-04-20T17:44:54.088 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7460999999999997e-05 2024-04-20T17:44:54.088 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.088 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.088 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:44:54.088 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:23.691054+0000", 2024-04-20T17:44:54.088 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3491999999999995e-05 2024-04-20T17:44:54.088 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.089 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:44:54.089 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.089 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.089 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.089 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27088.0:261227 7.1 7:833af0ba:::benchmark_data_smithi171_161871_object115501:head [delete] snapc 0=[] ondisk+write+known_if_redirected e757)", 2024-04-20T17:44:54.089 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:42:24.038491+0000", 2024-04-20T17:44:54.089 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 149.940646173, 2024-04-20T17:44:54.089 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12501227200000001, 2024-04-20T17:44:54.089 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:44:54.089 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:44:54.089 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:54.089 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:44:54.089 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27088", 2024-04-20T17:44:54.089 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:54.089 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 261227 2024-04-20T17:44:54.090 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.090 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:44:54.090 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.090 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:44:54.090 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:24.038491+0000", 2024-04-20T17:44:54.090 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.090 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.090 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.090 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:44:54.090 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:24.038491+0000", 2024-04-20T17:44:54.090 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.090 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.090 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.090 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:44:54.090 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:24.038493+0000", 2024-04-20T17:44:54.091 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6899999999999999e-06 2024-04-20T17:44:54.091 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.091 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.091 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:44:54.091 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:24.038498+0000", 2024-04-20T17:44:54.091 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2399999999999998e-06 2024-04-20T17:44:54.091 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.091 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.091 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:44:54.091 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:24.038505+0000", 2024-04-20T17:44:54.091 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4449999999999999e-06 2024-04-20T17:44:54.091 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.091 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.091 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:44:54.091 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:24.038511+0000", 2024-04-20T17:44:54.092 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5999999999999997e-06 2024-04-20T17:44:54.092 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.092 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.092 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:44:54.092 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:24.100081+0000", 2024-04-20T17:44:54.092 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.061570038000000001 2024-04-20T17:44:54.092 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.092 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.092 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:44:54.092 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:24.100170+0000", 2024-04-20T17:44:54.092 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8665999999999994e-05 2024-04-20T17:44:54.092 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.092 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.092 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:44:54.093 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:24.100257+0000", 2024-04-20T17:44:54.093 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6711000000000004e-05 2024-04-20T17:44:54.093 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.093 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.093 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:44:54.093 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:24.103779+0000", 2024-04-20T17:44:54.093 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0035221369999999998 2024-04-20T17:44:54.093 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.093 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.093 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:54.093 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:24.163423+0000", 2024-04-20T17:44:54.093 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.059644013000000003 2024-04-20T17:44:54.093 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.093 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.094 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:44:54.094 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:24.163460+0000", 2024-04-20T17:44:54.094 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7512999999999998e-05 2024-04-20T17:44:54.094 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.094 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.094 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:44:54.094 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:24.163503+0000", 2024-04-20T17:44:54.094 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3219000000000001e-05 2024-04-20T17:44:54.094 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.094 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:44:54.094 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.094 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.094 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.094 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27088.0:261228 7.1 7:836b2558:::benchmark_data_smithi171_161871_object115502:head [delete] snapc 0=[] ondisk+write+known_if_redirected e757)", 2024-04-20T17:44:54.095 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:42:24.038833+0000", 2024-04-20T17:44:54.095 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 149.94030426800001, 2024-04-20T17:44:54.095 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12566554199999999, 2024-04-20T17:44:54.095 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:44:54.095 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:44:54.095 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:44:54.095 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:44:54.095 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27088", 2024-04-20T17:44:54.095 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:44:54.095 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 261228 2024-04-20T17:44:54.095 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.095 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:44:54.095 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.095 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:44:54.096 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:24.038833+0000", 2024-04-20T17:44:54.096 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.096 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.096 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.096 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:44:54.096 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:24.038833+0000", 2024-04-20T17:44:54.096 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:54.096 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.096 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.096 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:44:54.096 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:24.038834+0000", 2024-04-20T17:44:54.096 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3489999999999999e-06 2024-04-20T17:44:54.096 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.096 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.096 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:44:54.097 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:24.038839+0000", 2024-04-20T17:44:54.097 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6970000000000001e-06 2024-04-20T17:44:54.097 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.097 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.097 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:44:54.097 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:24.038850+0000", 2024-04-20T17:44:54.097 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0781999999999999e-05 2024-04-20T17:44:54.097 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.097 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.097 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:44:54.097 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:24.038856+0000", 2024-04-20T17:44:54.097 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7209999999999999e-06 2024-04-20T17:44:54.097 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.097 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.098 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:44:54.098 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:24.104516+0000", 2024-04-20T17:44:54.098 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.065660184999999996 2024-04-20T17:44:54.098 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.098 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.098 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:44:54.098 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:24.104639+0000", 2024-04-20T17:44:54.098 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000122984 2024-04-20T17:44:54.098 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.098 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.098 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:44:54.098 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:24.104777+0000", 2024-04-20T17:44:54.098 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013856 2024-04-20T17:44:54.098 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.099 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.099 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:44:54.099 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:24.110527+0000", 2024-04-20T17:44:54.099 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0057497879999999996 2024-04-20T17:44:54.099 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.099 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.099 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:44:54.099 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:24.164420+0000", 2024-04-20T17:44:54.099 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.053892967 2024-04-20T17:44:54.099 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.099 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.099 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:44:54.099 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:24.164457+0000", 2024-04-20T17:44:54.099 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7101999999999998e-05 2024-04-20T17:44:54.099 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:54.100 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:54.100 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:44:54.100 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:42:24.164499+0000", 2024-04-20T17:44:54.100 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1406999999999998e-05 2024-04-20T17:44:54.100 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.100 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:44:54.100 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.100 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:54.100 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:44:54.100 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:44:54.101 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:53 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.101 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:53 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.101 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:53 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.101 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:53 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.101 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:53 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.101 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:53 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.101 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:53 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.101 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:53 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.102 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:53 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.102 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:53 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.102 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:53 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.102 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:53 smithi171 ceph-mon[44189]: pgmap v2159: 17 pgs: 17 active+clean; 8.5 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 60 MiB/s wr, 956 op/s 2024-04-20T17:44:54.102 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:53 smithi171 ceph-mon[44189]: osdmap e807: 12 total, 12 up, 10 in 2024-04-20T17:44:54.102 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:53 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.102 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:53 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.102 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:53 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.102 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:53 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.103 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:53 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.103 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:53 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.103 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:53 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.103 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:53 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.103 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:53 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.103 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:53 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.103 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:53 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.103 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:53 smithi106 ceph-mon[44423]: pgmap v2159: 17 pgs: 17 active+clean; 8.5 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 60 MiB/s wr, 956 op/s 2024-04-20T17:44:54.103 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:53 smithi106 ceph-mon[44423]: osdmap e807: 12 total, 12 up, 10 in 2024-04-20T17:44:54.104 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-04-20T17:44:54.104 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:53 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.104 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:53 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.104 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:53 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.104 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:53 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.104 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:53 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.104 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:53 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.104 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:53 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.104 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:53 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.104 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:53 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.104 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:53 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.105 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:53 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:44:54.105 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:53 smithi005 ceph-mon[39562]: pgmap v2159: 17 pgs: 17 active+clean; 8.5 GiB data, 22 GiB used, 783 GiB / 805 GiB avail; 60 MiB/s wr, 956 op/s 2024-04-20T17:44:54.105 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:53 smithi005 ceph-mon[39562]: osdmap e807: 12 total, 12 up, 10 in 2024-04-20T17:44:54.110 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:44:54.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:54.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:44:54.218 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:44:54.264 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:54.303 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:44:54.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:54.306 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:44:54.407 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:44:54.438 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 86 16 142256 142240 103.337 45.125 0.00512666 0.00967421 2024-04-20T17:44:54.448 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:54.450 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:44:54.498 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:44:54.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:54.503 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:44:54.604 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:44:54.635 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:54.758 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:44:54.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:54.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:44:54.778 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-20T17:44:54.863 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:44:54.945 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:54.983 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:44:54.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:54.993 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:44:55.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:54 smithi106 ceph-mon[44423]: pgmap v2161: 17 pgs: 2 active+recovering, 15 active+clean; 8.6 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 66 MiB/s wr, 1.05k op/s; 17567/282440 objects misplaced (6.220%); 362 KiB/s, 5 objects/s recovering 2024-04-20T17:44:55.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:54 smithi171 ceph-mon[44189]: pgmap v2161: 17 pgs: 2 active+recovering, 15 active+clean; 8.6 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 66 MiB/s wr, 1.05k op/s; 17567/282440 objects misplaced (6.220%); 362 KiB/s, 5 objects/s recovering 2024-04-20T17:44:55.093 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:44:55.122 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:55.146 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:44:55.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:55.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:44:55.158 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:54 smithi005 ceph-mon[39562]: pgmap v2161: 17 pgs: 2 active+recovering, 15 active+clean; 8.6 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 66 MiB/s wr, 1.05k op/s; 17567/282440 objects misplaced (6.220%); 362 KiB/s, 5 objects/s recovering 2024-04-20T17:44:55.243 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:44:55.250 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:44:55.289 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:55.400 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:44:55.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:55.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:44:55.438 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 87 16 142999 142983 102.683 46.4375 0.0112912 0.00973649 2024-04-20T17:44:55.505 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:44:55.565 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:55.610 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:44:55.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:55.616 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:44:55.717 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:44:55.745 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:55.765 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:44:55.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:55.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:44:55.869 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:44:55.916 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:55.946 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:44:55.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:55.955 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:44:56.056 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:44:56.085 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:56.438 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 88 16 143730 143714 102.036 45.6875 0.0136967 0.00979784 2024-04-20T17:44:56.495 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:44:56.736 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:44:56.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:56.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:44:56.839 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:44:56.867 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:56.914 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:44:57.015 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:44:57.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:57.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:44:57.119 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:44:57.148 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:57.168 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:44:57.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:57.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:44:57.272 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:44:57.302 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:57.310 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:44:57.311 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:44:57.311 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:57.311 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27460.0:144342 8.4 e807/777 8:2745c185:::benchmark_data_smithi171_175757_object144341:head v 807'8967, mlcod=807'8965)", 2024-04-20T17:44:57.311 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:44:57.307609+0000", 2024-04-20T17:44:57.311 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0031288169999999999, 2024-04-20T17:44:57.311 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003164769, 2024-04-20T17:44:57.311 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:44:57.311 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:44:57.312 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-04-20T17:44:57.312 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:44:57.312 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:57.312 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:44:57.312 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:44:57.307609+0000", 2024-04-20T17:44:57.312 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:57.312 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:57.312 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:57.312 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:44:57.312 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:44:57.307609+0000", 2024-04-20T17:44:57.312 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:57.312 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:57.312 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:57.313 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:44:57.313 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:44:57.307611+0000", 2024-04-20T17:44:57.313 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6809999999999998e-06 2024-04-20T17:44:57.313 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:57.313 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:57.313 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:44:57.313 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:44:57.307646+0000", 2024-04-20T17:44:57.313 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4153000000000003e-05 2024-04-20T17:44:57.313 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:57.313 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:57.313 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:44:57.313 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:44:57.307684+0000", 2024-04-20T17:44:57.313 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8105999999999997e-05 2024-04-20T17:44:57.313 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:57.313 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:57.314 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:44:57.314 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:44:57.307706+0000", 2024-04-20T17:44:57.314 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2708999999999999e-05 2024-04-20T17:44:57.314 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:57.314 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:57.314 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:44:57.314 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:44:57.307809+0000", 2024-04-20T17:44:57.314 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000102705 2024-04-20T17:44:57.314 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:57.314 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:57.314 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:44:57.314 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:44:57.307845+0000", 2024-04-20T17:44:57.314 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6304999999999999e-05 2024-04-20T17:44:57.314 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:57.315 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:44:57.315 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:57.315 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:57.315 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:57.315 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27460.0:144343 8.9 e807/777 8:9971e565:::benchmark_data_smithi171_175757_object144342:head v 807'9120, mlcod=807'9118)", 2024-04-20T17:44:57.315 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:44:57.307693+0000", 2024-04-20T17:44:57.315 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0030440150000000002, 2024-04-20T17:44:57.315 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0031464980000000002, 2024-04-20T17:44:57.315 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:44:57.315 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:44:57.315 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-04-20T17:44:57.315 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:44:57.315 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:57.315 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:44:57.315 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:44:57.307693+0000", 2024-04-20T17:44:57.316 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:57.316 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:57.316 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:57.316 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:44:57.316 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:44:57.307693+0000", 2024-04-20T17:44:57.316 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:44:57.316 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:57.316 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:57.316 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:44:57.316 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:44:57.307695+0000", 2024-04-20T17:44:57.316 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9240000000000001e-06 2024-04-20T17:44:57.316 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:57.316 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:57.316 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:44:57.316 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:44:57.307811+0000", 2024-04-20T17:44:57.317 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000115386 2024-04-20T17:44:57.317 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:57.317 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:57.317 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:44:57.317 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:44:57.307824+0000", 2024-04-20T17:44:57.317 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3414999999999999e-05 2024-04-20T17:44:57.317 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:57.317 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:57.317 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:44:57.317 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:44:57.307830+0000", 2024-04-20T17:44:57.317 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6189999999999999e-06 2024-04-20T17:44:57.318 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:57.318 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:57.318 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:44:57.318 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:44:57.307873+0000", 2024-04-20T17:44:57.318 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3606999999999998e-05 2024-04-20T17:44:57.318 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:44:57.318 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:44:57.318 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:44:57.318 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:44:57.307887+0000", 2024-04-20T17:44:57.318 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3756e-05 2024-04-20T17:44:57.318 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:57.318 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:44:57.318 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:57.318 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:44:57.319 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-04-20T17:44:57.319 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 2 2024-04-20T17:44:57.319 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:44:57.326 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:44:57.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:57.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:44:57.429 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:44:57.438 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 89 16 144442 144426 101.389 44.5 0.0107732 0.00986109 2024-04-20T17:44:57.470 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:57.512 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:44:57.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:57.522 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:44:57.623 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:44:57.670 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:44:57.690 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:57.697 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:57 smithi005 ceph-mon[39562]: pgmap v2162: 17 pgs: 3 active+recovering, 14 active+clean; 8.7 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 67 MiB/s wr, 1.07k op/s; 26331/285118 objects misplaced (9.235%); 704 KiB/s, 10 objects/s recovering 2024-04-20T17:44:57.697 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:57 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3060003096' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:44:57.714 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:44:57.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:57.721 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:44:57.822 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:44:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:57 smithi171 ceph-mon[44189]: pgmap v2162: 17 pgs: 3 active+recovering, 14 active+clean; 8.7 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 67 MiB/s wr, 1.07k op/s; 26331/285118 objects misplaced (9.235%); 704 KiB/s, 10 objects/s recovering 2024-04-20T17:44:57.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:57 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3060003096' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:44:57.871 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:57.911 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:44:57.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:57.921 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:44:57.995 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:57 smithi106 ceph-mon[44423]: pgmap v2162: 17 pgs: 3 active+recovering, 14 active+clean; 8.7 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 67 MiB/s wr, 1.07k op/s; 26331/285118 objects misplaced (9.235%); 704 KiB/s, 10 objects/s recovering 2024-04-20T17:44:57.995 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:57 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3060003096' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:44:58.022 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:44:58.027 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-20T17:44:58.084 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:58.133 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:44:58.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:58.140 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:44:58.241 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:44:58.296 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:58.303 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:44:58.333 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:44:58.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:58.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:44:58.438 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:44:58.439 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Total time run: 90.0229 2024-04-20T17:44:58.439 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Total writes made: 145111 2024-04-20T17:44:58.439 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Write size: 65536 2024-04-20T17:44:58.439 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Object size: 65536 2024-04-20T17:44:58.439 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Bandwidth (MB/sec): 100.746 2024-04-20T17:44:58.439 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Stddev Bandwidth: 49.8698 2024-04-20T17:44:58.439 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Max bandwidth (MB/sec): 274.938 2024-04-20T17:44:58.440 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Min bandwidth (MB/sec): 0.75 2024-04-20T17:44:58.440 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Average IOPS: 1611 2024-04-20T17:44:58.440 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Stddev IOPS: 797.916 2024-04-20T17:44:58.440 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Max IOPS: 4399 2024-04-20T17:44:58.440 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Min IOPS: 12 2024-04-20T17:44:58.440 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Average Latency(s): 0.00992533 2024-04-20T17:44:58.440 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Stddev Latency(s): 0.019103 2024-04-20T17:44:58.440 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Max latency(s): 1.69299 2024-04-20T17:44:58.440 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Min latency(s): 0.00104535 2024-04-20T17:44:58.445 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Cleaning up (deleting benchmark objects) 2024-04-20T17:44:58.493 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:58.524 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:44:58.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:58.531 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:44:58.632 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:44:58.675 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:58.818 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:44:58.821 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:58 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:44:58.821 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:58 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3060003096' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:44:58.821 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:58 smithi106 ceph-mon[44423]: osdmap e808: 12 total, 12 up, 10 in 2024-04-20T17:44:58.821 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:58 smithi106 ceph-mon[44423]: 8.e scrub starts 2024-04-20T17:44:58.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:58.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:44:58.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:58 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:44:58.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:58 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3060003096' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:44:58.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:58 smithi171 ceph-mon[44189]: osdmap e808: 12 total, 12 up, 10 in 2024-04-20T17:44:58.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:58 smithi171 ceph-mon[44189]: 8.e scrub starts 2024-04-20T17:44:58.923 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:44:58.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:58 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:44:58.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:58 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3060003096' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:44:58.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:58 smithi005 ceph-mon[39562]: osdmap e808: 12 total, 12 up, 10 in 2024-04-20T17:44:58.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:58 smithi005 ceph-mon[39562]: 8.e scrub starts 2024-04-20T17:44:58.965 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:59.008 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:44:59.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:59.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:44:59.105 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 8, 9, 10, 11, 1] out_osds: [6, 7] dead_osds: [] live_osds: [0, 4, 5, 8, 9, 10, 11, 2, 6, 3, 1, 7] 2024-04-20T17:44:59.105 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:44:59.105 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-04-20T17:44:59.105 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd in 6 2024-04-20T17:44:59.114 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:44:59.223 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:59.257 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:44:59.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:59.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:44:59.367 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:44:59.425 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:59.459 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:44:59.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:59.466 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:44:59.559 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:44:59.567 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:44:59.600 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:44:59.600 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:44:59.607 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:44:59.634 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:44:59.678 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:44:59.690 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:59 smithi005 ceph-mon[39562]: pgmap v2164: 17 pgs: 3 active+recovering, 14 active+clean; 8.7 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 30 MiB/s wr, 475 op/s; 26331/285118 objects misplaced (9.235%); 821 KiB/s, 12 objects/s recovering 2024-04-20T17:44:59.690 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:44:59 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2062818948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:44:59.701 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:44:59.722 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:44:59.743 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:44:59.766 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:44:59.802 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:44:59.836 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:44:59.847 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:44:59.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:44:59.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:44:59.865 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:44:59.906 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:44:59.908 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:44:59.908 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:44:59.908 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:44:59.908 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:44:59.908 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:44:59.908 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:44:59.908 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:44:59.908 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:44:59.908 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:44:59.908 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:44:59.909 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:44:59.909 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:44:59.951 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:44:59.976 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:59 smithi106 ceph-mon[44423]: pgmap v2164: 17 pgs: 3 active+recovering, 14 active+clean; 8.7 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 30 MiB/s wr, 475 op/s; 26331/285118 objects misplaced (9.235%); 821 KiB/s, 12 objects/s recovering 2024-04-20T17:44:59.976 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:44:59 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2062818948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:44:59.979 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:59 smithi171 ceph-mon[44189]: pgmap v2164: 17 pgs: 3 active+recovering, 14 active+clean; 8.7 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 30 MiB/s wr, 475 op/s; 26331/285118 objects misplaced (9.235%); 821 KiB/s, 12 objects/s recovering 2024-04-20T17:45:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:44:59 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2062818948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:45:00.178 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:45:00.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:00.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:45:00.284 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:45:00.315 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:00.468 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:45:00.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:00.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:45:00.471 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:45:00.471 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:45:00.571 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:45:00.601 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:00.800 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:45:00.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:00.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:45:00.855 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:45:00.898 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:00 smithi106 ceph-mon[44423]: 8.e scrub ok 2024-04-20T17:45:00.898 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:00.898 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:00.898 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:00.898 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:00.898 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:00.898 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:00.898 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:00.899 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:00.899 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:00.899 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:00.899 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:00.904 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:45:00.943 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:00 smithi005 ceph-mon[39562]: 8.e scrub ok 2024-04-20T17:45:00.944 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:00.944 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:00.944 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:00.944 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:00.944 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:00.944 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:00.944 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:00.944 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:00.944 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:00.944 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:00.944 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:00.945 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:01.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:00 smithi171 ceph-mon[44189]: 8.e scrub ok 2024-04-20T17:45:01.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:01.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:01.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:01.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:01.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:01.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:01.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:01.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:01.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:01.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:01.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:01.165 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:45:01.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:01.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:45:01.271 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:45:01.317 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:01.367 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:45:01.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:01.374 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:45:01.390 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-20T17:45:01.475 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:45:01.558 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:01.590 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:45:01.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:01.595 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:45:01.675 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:45:01.695 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:45:01.702 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:45:01.740 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:01.771 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:45:01.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:01.774 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:45:01.875 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:45:01.908 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:01 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:01.908 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:01 smithi005 ceph-mon[39562]: pgmap v2165: 17 pgs: 3 active+recovering, 14 active+clean; 8.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 127 B/s rd, 42 MiB/s wr, 747 op/s; 26142/288342 objects misplaced (9.066%); 2.2 MiB/s, 34 objects/s recovering 2024-04-20T17:45:01.910 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:01.963 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:01 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:01.964 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:01 smithi106 ceph-mon[44423]: pgmap v2165: 17 pgs: 3 active+recovering, 14 active+clean; 8.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 127 B/s rd, 42 MiB/s wr, 747 op/s; 26142/288342 objects misplaced (9.066%); 2.2 MiB/s, 34 objects/s recovering 2024-04-20T17:45:01.972 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:45:01.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:01.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:45:02.080 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:45:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:01 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:01 smithi171 ceph-mon[44189]: pgmap v2165: 17 pgs: 3 active+recovering, 14 active+clean; 8.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 127 B/s rd, 42 MiB/s wr, 747 op/s; 26142/288342 objects misplaced (9.066%); 2.2 MiB/s, 34 objects/s recovering 2024-04-20T17:45:02.119 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:02.160 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:45:02.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:02.165 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:45:02.266 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:45:02.319 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:02.356 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:45:02.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:02.364 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:45:02.465 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:45:02.498 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:02.526 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:45:02.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:02.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:45:02.632 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:45:02.676 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:02.706 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:45:02.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:02.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:45:02.713 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:45:02.810 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:45:02.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:02 smithi171 ceph-mon[44189]: 8.8 scrub starts 2024-04-20T17:45:02.907 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:02.949 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:45:02.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:02.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:45:02.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:02 smithi005 ceph-mon[39562]: 8.8 scrub starts 2024-04-20T17:45:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:02 smithi106 ceph-mon[44423]: 8.8 scrub starts 2024-04-20T17:45:03.054 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:45:03.104 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:03.135 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:45:03.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:03.146 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:45:03.247 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:45:03.277 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:03.378 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:45:03.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:03.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:45:03.482 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:45:03.527 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:03.572 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:45:03.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:03.579 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:45:03.659 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:45:03.659 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:45:03.680 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:45:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:03 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mon-a[39539]: 2024-04-20T17:45:03.332+0000 7f73b92a1700 -1 mon.a@0(leader).osd e808 definitely_dead 0 2024-04-20T17:45:03.721 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:03.746 INFO:teuthology.orchestra.run.smithi005.stderr:marked in osd.6. 2024-04-20T17:45:03.802 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:45:03.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:03.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:45:03.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:03 smithi171 ceph-mon[44189]: pgmap v2166: 17 pgs: 3 active+recovering, 14 active+clean; 8.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 103 B/s rd, 37 MiB/s wr, 697 op/s; 26036/288122 objects misplaced (9.036%); 2.4 MiB/s, 38 objects/s recovering 2024-04-20T17:45:03.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:03 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3482609829' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-04-20T17:45:03.908 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:45:03.945 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:03 smithi106 ceph-mon[44423]: pgmap v2166: 17 pgs: 3 active+recovering, 14 active+clean; 8.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 103 B/s rd, 37 MiB/s wr, 697 op/s; 26036/288122 objects misplaced (9.036%); 2.4 MiB/s, 38 objects/s recovering 2024-04-20T17:45:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:03 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3482609829' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-04-20T17:45:04.016 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:45:04.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:04.021 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:03 smithi005 ceph-mon[39562]: pgmap v2166: 17 pgs: 3 active+recovering, 14 active+clean; 8.8 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 103 B/s rd, 37 MiB/s wr, 697 op/s; 26036/288122 objects misplaced (9.036%); 2.4 MiB/s, 38 objects/s recovering 2024-04-20T17:45:04.022 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:03 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3482609829' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-04-20T17:45:04.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:45:04.123 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:45:04.153 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:04.537 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-04-20T17:45:04.549 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-20T17:45:04.608 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:45:04.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:04.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:45:04.706 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:45:04.711 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:45:04.744 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:04.763 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:45:04.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:04.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:45:04.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:04 smithi171 ceph-mon[44189]: 8.8 scrub ok 2024-04-20T17:45:04.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:04 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3482609829' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-04-20T17:45:04.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:04 smithi171 ceph-mon[44189]: osdmap e809: 12 total, 12 up, 11 in 2024-04-20T17:45:04.838 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:04 smithi171 ceph-mon[44189]: 8.0 scrub starts 2024-04-20T17:45:04.868 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:45:04.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:04 smithi005 ceph-mon[39562]: 8.8 scrub ok 2024-04-20T17:45:04.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:04 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3482609829' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-04-20T17:45:04.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:04 smithi005 ceph-mon[39562]: osdmap e809: 12 total, 12 up, 11 in 2024-04-20T17:45:04.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:04 smithi005 ceph-mon[39562]: 8.0 scrub starts 2024-04-20T17:45:04.963 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:04 smithi106 ceph-mon[44423]: 8.8 scrub ok 2024-04-20T17:45:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:04 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3482609829' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-04-20T17:45:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:04 smithi106 ceph-mon[44423]: osdmap e809: 12 total, 12 up, 11 in 2024-04-20T17:45:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:04 smithi106 ceph-mon[44423]: 8.0 scrub starts 2024-04-20T17:45:05.027 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:45:05.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:05.031 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:45:05.040 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:45:05.132 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:45:05.173 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:05.764 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:45:05.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:05 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:45:05.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:05 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:45:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:05 smithi005 ceph-mon[39562]: pgmap v2168: 17 pgs: 1 active+clean+scrubbing, 1 peering, 3 active+recovering, 12 active+clean; 8.7 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 127 B/s rd, 20 MiB/s wr, 685 op/s; 25778/284374 objects misplaced (9.065%); 4.3 MiB/s, 69 objects/s recovering 2024-04-20T17:45:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:05 smithi005 ceph-mon[39562]: osdmap e810: 12 total, 12 up, 11 in 2024-04-20T17:45:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:05 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:45:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:05 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:45:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:05 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/946428062' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:45:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:05 smithi005 ceph-mon[39562]: 8.0 scrub ok 2024-04-20T17:45:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:05 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:45:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:05 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:45:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:05 smithi106 ceph-mon[44423]: pgmap v2168: 17 pgs: 1 active+clean+scrubbing, 1 peering, 3 active+recovering, 12 active+clean; 8.7 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 127 B/s rd, 20 MiB/s wr, 685 op/s; 25778/284374 objects misplaced (9.065%); 4.3 MiB/s, 69 objects/s recovering 2024-04-20T17:45:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:05 smithi106 ceph-mon[44423]: osdmap e810: 12 total, 12 up, 11 in 2024-04-20T17:45:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:05 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:45:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:05 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:45:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:05 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/946428062' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:45:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:05 smithi106 ceph-mon[44423]: 8.0 scrub ok 2024-04-20T17:45:06.082 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:05 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:45:06.082 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:05 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:45:06.082 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:05 smithi171 ceph-mon[44189]: pgmap v2168: 17 pgs: 1 active+clean+scrubbing, 1 peering, 3 active+recovering, 12 active+clean; 8.7 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 127 B/s rd, 20 MiB/s wr, 685 op/s; 25778/284374 objects misplaced (9.065%); 4.3 MiB/s, 69 objects/s recovering 2024-04-20T17:45:06.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:05 smithi171 ceph-mon[44189]: osdmap e810: 12 total, 12 up, 11 in 2024-04-20T17:45:06.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:05 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:45:06.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:05 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:45:06.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:05 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/946428062' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:45:06.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:05 smithi171 ceph-mon[44189]: 8.0 scrub ok 2024-04-20T17:45:06.638 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:45:06.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:06.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:45:06.745 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:45:06.773 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:06.792 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:45:06.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:06.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:45:06.896 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:45:06.934 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:45:06.934 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:45:06.934 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:45:06.934 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:45:06.934 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:45:06.934 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:45:06.945 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:06.966 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:45:06.992 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:45:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:06 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:45:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:06 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:45:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:06 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/946428062' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:45:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:06 smithi106 ceph-mon[44423]: osdmap e811: 12 total, 12 up, 11 in 2024-04-20T17:45:07.031 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:45:07.065 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:45:07.068 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:06 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:45:07.069 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:06 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:45:07.069 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:06 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/946428062' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:45:07.069 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:06 smithi005 ceph-mon[39562]: osdmap e811: 12 total, 12 up, 11 in 2024-04-20T17:45:07.083 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:45:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:06 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:45:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:06 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:45:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:06 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/946428062' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:45:07.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:06 smithi171 ceph-mon[44189]: osdmap e811: 12 total, 12 up, 11 in 2024-04-20T17:45:07.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:07.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:45:07.099 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:45:07.129 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:45:07.140 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:45:07.140 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:45:07.140 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:45:07.140 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:45:07.140 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:45:07.141 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:45:07.141 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:45:07.141 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:45:07.141 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:45:07.141 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:45:07.141 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:45:07.141 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:45:07.188 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:45:07.227 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:07.389 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:45:07.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:07.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:45:07.494 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:45:07.519 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:07.539 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:45:07.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:07.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:45:07.643 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:45:07.678 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:07.838 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:45:07.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:07.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:45:07.945 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:45:07.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:07 smithi005 ceph-mon[39562]: pgmap v2171: 17 pgs: 1 active+clean+scrubbing, 1 peering, 3 active+recovering, 12 active+clean; 8.6 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 4.1 MiB/s wr, 699 op/s; 25593/281506 objects misplaced (9.091%); 5.7 MiB/s, 91 objects/s recovering 2024-04-20T17:45:07.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:07 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/792216394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:45:07.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:07.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:07.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:07.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:07.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:07.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:07.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:08.065 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:07 smithi171 ceph-mon[44189]: pgmap v2171: 17 pgs: 1 active+clean+scrubbing, 1 peering, 3 active+recovering, 12 active+clean; 8.6 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 4.1 MiB/s wr, 699 op/s; 25593/281506 objects misplaced (9.091%); 5.7 MiB/s, 91 objects/s recovering 2024-04-20T17:45:08.066 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:07 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/792216394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:45:08.066 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:08.066 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:08.066 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:08.066 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:08.066 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:08.066 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:08.066 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:08.071 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:07 smithi106 ceph-mon[44423]: pgmap v2171: 17 pgs: 1 active+clean+scrubbing, 1 peering, 3 active+recovering, 12 active+clean; 8.6 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 4.1 MiB/s wr, 699 op/s; 25593/281506 objects misplaced (9.091%); 5.7 MiB/s, 91 objects/s recovering 2024-04-20T17:45:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:07 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/792216394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:45:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:08.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:08.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:08.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:08 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:45:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:08 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:45:09.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:08 smithi171 ceph-mon[44189]: osdmap e812: 12 total, 12 up, 11 in 2024-04-20T17:45:09.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:08 smithi171 ceph-mon[44189]: pgmap v2173: 17 pgs: 1 active+clean+scrubbing, 1 peering, 3 active+recovering, 12 active+clean; 8.6 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 677 op/s; 25593/281506 objects misplaced (9.091%); 5.7 MiB/s, 90 objects/s recovering 2024-04-20T17:45:09.113 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:45:09.147 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:09.147 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:09.147 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:09.147 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:09.147 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:09.147 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:08 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:45:09.147 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:08 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:45:09.147 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:08 smithi005 ceph-mon[39562]: osdmap e812: 12 total, 12 up, 11 in 2024-04-20T17:45:09.147 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:08 smithi005 ceph-mon[39562]: pgmap v2173: 17 pgs: 1 active+clean+scrubbing, 1 peering, 3 active+recovering, 12 active+clean; 8.6 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 677 op/s; 25593/281506 objects misplaced (9.091%); 5.7 MiB/s, 90 objects/s recovering 2024-04-20T17:45:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:08 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:45:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:08 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:45:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:08 smithi106 ceph-mon[44423]: osdmap e812: 12 total, 12 up, 11 in 2024-04-20T17:45:09.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:08 smithi106 ceph-mon[44423]: pgmap v2173: 17 pgs: 1 active+clean+scrubbing, 1 peering, 3 active+recovering, 12 active+clean; 8.6 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 677 op/s; 25593/281506 objects misplaced (9.091%); 5.7 MiB/s, 90 objects/s recovering 2024-04-20T17:45:09.462 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:45:09.538 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 8, 9, 10, 11, 1, 6] out_osds: [7] dead_osds: [] live_osds: [0, 4, 5, 8, 9, 10, 11, 2, 6, 3, 1, 7] 2024-04-20T17:45:09.538 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:45:09.538 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-04-20T17:45:09.538 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:45:09.878 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:45:09.910 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:09 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:45:09.910 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:09 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:45:10.071 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:45:10.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:10.079 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:45:10.079 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:09 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:45:10.079 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:09 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:45:10.180 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:45:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:09 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:45:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:09 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:45:10.294 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:10.335 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:45:10.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:10.341 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:45:10.442 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:45:10.476 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:10.799 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:10 smithi005 ceph-mon[39562]: osdmap e813: 12 total, 12 up, 11 in 2024-04-20T17:45:10.799 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:10 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:45:10.799 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:10 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:45:10.799 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:10 smithi005 ceph-mon[39562]: pgmap v2175: 17 pgs: 3 active+recovering, 14 active+clean; 8.5 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 633 op/s; 25231/276948 objects misplaced (9.110%); 5.9 MiB/s, 93 objects/s recovering 2024-04-20T17:45:10.806 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:10 smithi171 ceph-mon[44189]: osdmap e813: 12 total, 12 up, 11 in 2024-04-20T17:45:10.806 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:10 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:45:10.806 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:10 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:45:10.806 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:10 smithi171 ceph-mon[44189]: pgmap v2175: 17 pgs: 3 active+recovering, 14 active+clean; 8.5 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 633 op/s; 25231/276948 objects misplaced (9.110%); 5.9 MiB/s, 93 objects/s recovering 2024-04-20T17:45:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:10 smithi106 ceph-mon[44423]: osdmap e813: 12 total, 12 up, 11 in 2024-04-20T17:45:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:10 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:45:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:10 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:45:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:10 smithi106 ceph-mon[44423]: pgmap v2175: 17 pgs: 3 active+recovering, 14 active+clean; 8.5 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 633 op/s; 25231/276948 objects misplaced (9.110%); 5.9 MiB/s, 93 objects/s recovering 2024-04-20T17:45:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:11 smithi171 ceph-mon[44189]: osdmap e814: 12 total, 12 up, 11 in 2024-04-20T17:45:12.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:11 smithi005 ceph-mon[39562]: osdmap e814: 12 total, 12 up, 11 in 2024-04-20T17:45:12.229 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:45:12.231 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:45:12.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:12.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:45:12.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:11 smithi106 ceph-mon[44423]: osdmap e814: 12 total, 12 up, 11 in 2024-04-20T17:45:12.337 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:45:12.372 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:12 smithi171 ceph-mon[44189]: pgmap v2177: 17 pgs: 3 active+recovering, 14 active+clean; 8.4 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 507 op/s; 25042/275420 objects misplaced (9.092%); 5.8 MiB/s, 92 objects/s recovering 2024-04-20T17:45:13.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:12 smithi005 ceph-mon[39562]: pgmap v2177: 17 pgs: 3 active+recovering, 14 active+clean; 8.4 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 507 op/s; 25042/275420 objects misplaced (9.092%); 5.8 MiB/s, 92 objects/s recovering 2024-04-20T17:45:13.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:12 smithi106 ceph-mon[44423]: pgmap v2177: 17 pgs: 3 active+recovering, 14 active+clean; 8.4 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 507 op/s; 25042/275420 objects misplaced (9.092%); 5.8 MiB/s, 92 objects/s recovering 2024-04-20T17:45:14.122 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:45:14.883 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:14 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:45:15.302 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:14 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:45:15.303 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:14 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:45:15.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:15 smithi005 ceph-mon[39562]: pgmap v2178: 17 pgs: 3 active+recovering, 14 active+clean; 8.3 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 684 op/s; 24680/272126 objects misplaced (9.069%); 8.4 MiB/s, 133 objects/s recovering 2024-04-20T17:45:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:15 smithi106 ceph-mon[44423]: pgmap v2178: 17 pgs: 3 active+recovering, 14 active+clean; 8.3 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 684 op/s; 24680/272126 objects misplaced (9.069%); 8.4 MiB/s, 133 objects/s recovering 2024-04-20T17:45:16.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:15 smithi171 ceph-mon[44189]: pgmap v2178: 17 pgs: 3 active+recovering, 14 active+clean; 8.3 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 684 op/s; 24680/272126 objects misplaced (9.069%); 8.4 MiB/s, 133 objects/s recovering 2024-04-20T17:45:16.138 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:45:16.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:16.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:45:16.248 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:45:16.299 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:17.159 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:45:17.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:17.164 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:45:17.265 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:45:17.307 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:17.480 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:45:17.481 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:45:17.704 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:17 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:45:17.705 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:17 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:45:17.705 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:17 smithi171 ceph-mon[44189]: pgmap v2179: 17 pgs: 3 active+recovering, 14 active+clean; 8.2 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 754 op/s; 24488/269428 objects misplaced (9.089%); 8.7 MiB/s, 138 objects/s recovering 2024-04-20T17:45:17.705 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:17 smithi171 ceph-mon[44189]: osdmap e815: 12 total, 12 up, 11 in 2024-04-20T17:45:17.705 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:17 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1373292617' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:45:17.705 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:17 smithi171 ceph-mon[44189]: from='client.27713 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:45:17.725 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:45:17.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:17 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:45:17.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:17 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:45:17.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:17 smithi005 ceph-mon[39562]: pgmap v2179: 17 pgs: 3 active+recovering, 14 active+clean; 8.2 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 754 op/s; 24488/269428 objects misplaced (9.089%); 8.7 MiB/s, 138 objects/s recovering 2024-04-20T17:45:17.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:17 smithi005 ceph-mon[39562]: osdmap e815: 12 total, 12 up, 11 in 2024-04-20T17:45:17.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:17 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1373292617' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:45:17.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:17 smithi005 ceph-mon[39562]: from='client.27713 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:45:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:17 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:45:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:17 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:45:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:17 smithi106 ceph-mon[44423]: pgmap v2179: 17 pgs: 3 active+recovering, 14 active+clean; 8.2 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 754 op/s; 24488/269428 objects misplaced (9.089%); 8.7 MiB/s, 138 objects/s recovering 2024-04-20T17:45:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:17 smithi106 ceph-mon[44423]: osdmap e815: 12 total, 12 up, 11 in 2024-04-20T17:45:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:17 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1373292617' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:45:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:17 smithi106 ceph-mon[44423]: from='client.27713 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:45:18.200 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:45:18.200 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:45:18.200 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:18.200 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:157516 8.b 8:d966c412:::benchmark_data_smithi171_175757_object12402:head [delete] snapc 0=[] ondisk+write+known_if_redirected e816)", 2024-04-20T17:45:18.200 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:45:18.192261+0000", 2024-04-20T17:45:18.200 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.0066575519999999997, 2024-04-20T17:45:18.200 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0066910169999999996, 2024-04-20T17:45:18.200 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:45:18.200 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:45:18.200 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:45:18.201 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:45:18.201 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:45:18.201 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:45:18.201 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 157516 2024-04-20T17:45:18.201 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:18.201 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:45:18.201 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:18.201 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:45:18.201 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:18.192261+0000", 2024-04-20T17:45:18.201 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:18.201 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:18.201 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:18.201 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:45:18.201 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:18.192261+0000", 2024-04-20T17:45:18.202 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:18.202 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:18.202 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:18.202 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:45:18.202 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:18.192263+0000", 2024-04-20T17:45:18.202 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6929999999999999e-06 2024-04-20T17:45:18.202 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:18.202 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:18.202 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:45:18.202 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:18.192272+0000", 2024-04-20T17:45:18.202 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5650000000000002e-06 2024-04-20T17:45:18.202 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:18.202 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:18.203 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:45:18.203 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:18.192284+0000", 2024-04-20T17:45:18.203 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1881e-05 2024-04-20T17:45:18.203 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:18.203 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:18.203 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:45:18.203 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:18.192293+0000", 2024-04-20T17:45:18.203 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.4520000000000003e-06 2024-04-20T17:45:18.203 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:18.203 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:18.203 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:45:18.203 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:18.192357+0000", 2024-04-20T17:45:18.203 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3917999999999993e-05 2024-04-20T17:45:18.203 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:18.204 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:18.204 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:45:18.204 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:18.192481+0000", 2024-04-20T17:45:18.204 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000123662 2024-04-20T17:45:18.204 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:18.204 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:18.204 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4", 2024-04-20T17:45:18.204 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:18.192566+0000", 2024-04-20T17:45:18.204 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4646000000000002e-05 2024-04-20T17:45:18.204 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:18.204 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:18.204 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:45:18.204 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:18.196565+0000", 2024-04-20T17:45:18.204 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0039994799999999997 2024-04-20T17:45:18.204 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:18.205 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:45:18.205 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:18.205 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:18.205 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:18.205 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:157518 8.a 8:59962bed:::benchmark_data_smithi171_175757_object12404:head [delete] snapc 0=[] ondisk+write+known_if_redirected e816)", 2024-04-20T17:45:18.205 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:45:18.196553+0000", 2024-04-20T17:45:18.205 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.0023657230000000001, 2024-04-20T17:45:18.205 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0024948819999999999, 2024-04-20T17:45:18.205 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:45:18.205 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:45:18.205 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:45:18.205 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:45:18.205 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:45:18.205 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:45:18.206 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 157518 2024-04-20T17:45:18.206 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:18.206 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:45:18.206 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:18.206 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:45:18.206 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:18.196553+0000", 2024-04-20T17:45:18.206 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:18.206 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:18.206 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:18.206 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:45:18.206 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:18.196553+0000", 2024-04-20T17:45:18.206 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:18.206 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:18.206 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:18.207 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:45:18.207 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:18.196555+0000", 2024-04-20T17:45:18.207 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4849999999999999e-06 2024-04-20T17:45:18.207 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:18.207 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:18.207 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:45:18.207 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:18.196561+0000", 2024-04-20T17:45:18.207 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8529999999999997e-06 2024-04-20T17:45:18.207 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:18.207 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:18.207 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:45:18.207 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:18.196569+0000", 2024-04-20T17:45:18.207 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4509999999999992e-06 2024-04-20T17:45:18.207 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:18.207 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:18.208 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:45:18.208 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:18.196576+0000", 2024-04-20T17:45:18.208 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4679999999999997e-06 2024-04-20T17:45:18.208 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:18.208 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:18.208 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:45:18.208 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:18.196724+0000", 2024-04-20T17:45:18.208 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014803799999999999 2024-04-20T17:45:18.208 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:18.208 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:18.208 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:45:18.208 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:18.196862+0000", 2024-04-20T17:45:18.208 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013819900000000001 2024-04-20T17:45:18.208 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:18.208 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:18.209 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:45:18.209 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:18.196982+0000", 2024-04-20T17:45:18.209 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000119517 2024-04-20T17:45:18.209 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:18.209 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:45:18.209 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:18.209 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:18.209 INFO:teuthology.orchestra.run.smithi171.stdout: ], 2024-04-20T17:45:18.209 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 2 2024-04-20T17:45:18.209 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:45:18.878 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:45:18.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:18 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:45:18.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:18 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:45:18.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:18 smithi005 ceph-mon[39562]: from='client.27719 v1:172.21.15.5:0/3381963654' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:45:18.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:18 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:45:18.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:18 smithi005 ceph-mon[39562]: from='client.27713 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:45:18.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:18 smithi005 ceph-mon[39562]: osdmap e816: 12 total, 12 up, 11 in 2024-04-20T17:45:18.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:18 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:45:18.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:18 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:45:18.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:18 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:45:18.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:18.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:45:18.985 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:45:19.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:18 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:45:19.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:18 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:45:19.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:18 smithi106 ceph-mon[44423]: from='client.27719 v1:172.21.15.5:0/3381963654' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:45:19.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:18 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:45:19.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:18 smithi106 ceph-mon[44423]: from='client.27713 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:45:19.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:18 smithi106 ceph-mon[44423]: osdmap e816: 12 total, 12 up, 11 in 2024-04-20T17:45:19.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:18 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:45:19.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:18 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:45:19.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:18 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:45:19.026 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:18 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:45:19.026 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:18 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:45:19.026 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:18 smithi171 ceph-mon[44189]: from='client.27719 v1:172.21.15.5:0/3381963654' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:45:19.027 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:18 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:45:19.027 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:18 smithi171 ceph-mon[44189]: from='client.27713 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:45:19.027 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:18 smithi171 ceph-mon[44189]: osdmap e816: 12 total, 12 up, 11 in 2024-04-20T17:45:19.027 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:18 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:45:19.027 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:18 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:45:19.027 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:18 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:45:19.030 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:19.423 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:45:19.423 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:45:19.424 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:45:19.424 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:45:19.424 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:45:19.430 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:45:19.458 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:45:19.481 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:45:19.505 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:45:19.537 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:45:19.575 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:45:19.588 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":2180,"stamp":"2024-04-20T17:45:16.705392+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8829010488,"num_objects":134714,"num_object_clones":0,"num_object_copies":269428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":24488,"num_objects_unfound":0,"num_objects_dirty":134714,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":155693,"num_write_kb":9290043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1943,"num_bytes_recovered":129723584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155789,"ondisk_log_size":155789,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":37,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":22815616,"kb_used_data":17373728,"kb_used_omap":392,"kb_used_meta":5441335,"kb_avail":1008381056,"statfs":{"total":1055945392128,"available":1032582201344,"internally_reserved":0,"allocated":17790697472,"data_stored":17786522446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":402024,"internal_metadata":5571927448},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,5,1,3,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-284557312,"num_objects":-4342,"num_object_clones":0,"num_object_copies":-8684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-743,"num_objects_unfound":0,"num_objects_dirty":-4342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4342,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":743,"num_bytes_recovered":48693248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.908815"},"pg_stats":[{"pgid":"8.f","version":"814'9533","reported_seq":9693,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-04-20T17:45:14.463011+0000","last_change":"2024-04-20T17:44:24.842530+0000","last_active":"2024-04-20T17:45:14.463011+0000","last_peered":"2024-04-20T17:45:14.463011+0000","last_clean":"2024-04-20T17:45:14.463011+0000","last_became_active":"2024-04-20T17:43:18.798932+0000","last_became_peered":"2024-04-20T17:43:18.798932+0000","last_unstale":"2024-04-20T17:45:14.463011+0000","last_undegraded":"2024-04-20T17:45:14.463011+0000","last_fullsized":"2024-04-20T17:45:14.463011+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'6866","last_scrub_stamp":"2024-04-20T17:44:24.842384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:44:24.842384+0000","objects_scrubbed":6822,"log_size":9533,"log_dups_size":0,"ondisk_log_size":9533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:45:41.891634+0000","scrub_duration":1583,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":547684352,"num_objects":8357,"num_object_clones":0,"num_object_copies":16714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9533,"num_write_kb":572480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.e","version":"814'12536","reported_seq":12870,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-04-20T17:45:16.100959+0000","last_change":"2024-04-20T17:44:59.038691+0000","last_active":"2024-04-20T17:45:16.100959+0000","last_peered":"2024-04-20T17:45:16.100959+0000","last_clean":"2024-04-20T17:45:16.100959+0000","last_became_active":"2024-04-20T17:43:18.798720+0000","last_became_peered":"2024-04-20T17:43:18.798720+0000","last_unstale":"2024-04-20T17:45:16.100959+0000","last_undegraded":"2024-04-20T17:45:16.100959+0000","last_fullsized":"2024-04-20T17:45:16.100959+0000","mapping_epoch":777,"log_start":"786'2500","ondisk_log_start":"786'2500","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"808'11809","last_scrub_stamp":"2024-04-20T17:44:59.038491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:44:59.038491+0000","objects_scrubbed":9137,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:46:24.186119+0000","scrub_duration":1214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":550240256,"num_objects":8396,"num_object_clones":0,"num_object_copies":16792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9896,"num_write_kb":585344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"814'9752","reported_seq":9916,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-04-20T17:45:14.449000+0000","last_change":"2024-04-20T17:44:26.727290+0000","last_active":"2024-04-20T17:45:14.449000+0000","last_peered":"2024-04-20T17:45:14.449000+0000","last_clean":"2024-04-20T17:45:14.449000+0000","last_became_active":"2024-04-20T17:43:18.798955+0000","last_became_peered":"2024-04-20T17:43:18.798955+0000","last_unstale":"2024-04-20T17:45:14.449000+0000","last_undegraded":"2024-04-20T17:45:14.449000+0000","last_fullsized":"2024-04-20T17:45:14.449000+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'7202","last_scrub_stamp":"2024-04-20T17:44:26.727159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:44:26.727159+0000","objects_scrubbed":7157,"log_size":9752,"log_dups_size":0,"ondisk_log_size":9752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:45:48.202243+0000","scrub_duration":1434,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":552927256,"num_objects":8438,"num_object_clones":0,"num_object_copies":16876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8438,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9752,"num_write_kb":582017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"814'9850","reported_seq":10010,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-04-20T17:45:16.097614+0000","last_change":"2024-04-20T17:44:21.757085+0000","last_active":"2024-04-20T17:45:16.097614+0000","last_peered":"2024-04-20T17:45:16.097614+0000","last_clean":"2024-04-20T17:45:16.097614+0000","last_became_active":"2024-04-20T17:43:18.602140+0000","last_became_peered":"2024-04-20T17:43:18.602140+0000","last_unstale":"2024-04-20T17:45:16.097614+0000","last_undegraded":"2024-04-20T17:45:16.097614+0000","last_fullsized":"2024-04-20T17:45:16.097614+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'6852","last_scrub_stamp":"2024-04-20T17:44:21.756952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:44:21.756952+0000","objects_scrubbed":6797,"log_size":9850,"log_dups_size":0,"ondisk_log_size":9850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:45:23.728869+0000","scrub_duration":1549,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":549453824,"num_objects":8384,"num_object_clones":0,"num_object_copies":16768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9850,"num_write_kb":583488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.b","version":"814'9373","reported_seq":9532,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-04-20T17:45:13.758392+0000","last_change":"2024-04-20T17:44:24.556916+0000","last_active":"2024-04-20T17:45:13.758392+0000","last_peered":"2024-04-20T17:45:13.758392+0000","last_clean":"2024-04-20T17:45:13.758392+0000","last_became_active":"2024-04-20T17:43:18.801242+0000","last_became_peered":"2024-04-20T17:43:18.801242+0000","last_unstale":"2024-04-20T17:45:13.758392+0000","last_undegraded":"2024-04-20T17:45:13.758392+0000","last_fullsized":"2024-04-20T17:45:13.758392+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'6755","last_scrub_stamp":"2024-04-20T17:44:24.556671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:44:24.556671+0000","objects_scrubbed":6726,"log_size":9373,"log_dups_size":0,"ondisk_log_size":9373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:45:26.098192+0000","scrub_duration":1188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":539951104,"num_objects":8239,"num_object_clones":0,"num_object_copies":16478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9373,"num_write_kb":563584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.a","version":"814'9774","reported_seq":9939,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-04-20T17:45:13.743326+0000","last_change":"2024-04-20T17:44:28.659090+0000","last_active":"2024-04-20T17:45:13.743326+0000","last_peered":"2024-04-20T17:45:13.743326+0000","last_clean":"2024-04-20T17:45:13.743326+0000","last_became_active":"2024-04-20T17:43:18.798958+0000","last_became_peered":"2024-04-20T17:43:18.798958+0000","last_unstale":"2024-04-20T17:45:13.743326+0000","last_undegraded":"2024-04-20T17:45:13.743326+0000","last_fullsized":"2024-04-20T17:45:13.743326+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"801'7336","last_scrub_stamp":"2024-04-20T17:44:28.658927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:44:28.658927+0000","objects_scrubbed":7298,"log_size":9774,"log_dups_size":0,"ondisk_log_size":9774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:45:45.272200+0000","scrub_duration":1310,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":562036736,"num_objects":8576,"num_object_clones":0,"num_object_copies":17152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9774,"num_write_kb":587200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2],"acting":[8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.8","version":"814'9685","reported_seq":9864,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-04-20T17:45:14.418003+0000","last_change":"2024-04-20T17:45:03.333991+0000","last_active":"2024-04-20T17:45:14.418003+0000","last_peered":"2024-04-20T17:45:14.418003+0000","last_clean":"2024-04-20T17:45:14.418003+0000","last_became_active":"2024-04-20T17:43:18.610421+0000","last_became_peered":"2024-04-20T17:43:18.610421+0000","last_unstale":"2024-04-20T17:45:14.418003+0000","last_undegraded":"2024-04-20T17:45:14.418003+0000","last_fullsized":"2024-04-20T17:45:14.418003+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"808'9236","last_scrub_stamp":"2024-04-20T17:45:03.333793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:45:03.333793+0000","objects_scrubbed":8842,"log_size":9685,"log_dups_size":0,"ondisk_log_size":9685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:46:19.287148+0000","scrub_duration":1373,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":548077568,"num_objects":8363,"num_object_clones":0,"num_object_copies":16726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9685,"num_write_kb":577536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.7","version":"814'9595","reported_seq":10926,"reported_epoch":814,"state":"active+recovering","last_fresh":"2024-04-20T17:45:14.495115+0000","last_change":"2024-04-20T17:44:53.433139+0000","last_active":"2024-04-20T17:45:14.495115+0000","last_peered":"2024-04-20T17:45:14.495115+0000","last_clean":"2024-04-20T17:44:51.743570+0000","last_became_active":"2024-04-20T17:44:53.388589+0000","last_became_peered":"2024-04-20T17:44:53.388589+0000","last_unstale":"2024-04-20T17:45:14.495115+0000","last_undegraded":"2024-04-20T17:45:14.495115+0000","last_fullsized":"2024-04-20T17:45:14.495115+0000","mapping_epoch":806,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":1599,"log_size":9595,"log_dups_size":0,"ondisk_log_size":9595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":548601856,"num_objects":8371,"num_object_clones":0,"num_object_copies":16742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8158,"num_objects_unfound":0,"num_objects_dirty":8371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9595,"num_write_kb":574912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":613,"num_bytes_recovered":40173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":["11","10"],"object_location_counts":[{"shards":"1,11","objects":213},{"shards":"10,11","objects":8158}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.6","version":"814'9724","reported_seq":9882,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-04-20T17:45:14.471876+0000","last_change":"2024-04-20T17:44:21.667932+0000","last_active":"2024-04-20T17:45:14.471876+0000","last_peered":"2024-04-20T17:45:14.471876+0000","last_clean":"2024-04-20T17:45:14.471876+0000","last_became_active":"2024-04-20T17:43:18.599894+0000","last_became_peered":"2024-04-20T17:43:18.599894+0000","last_unstale":"2024-04-20T17:45:14.471876+0000","last_undegraded":"2024-04-20T17:45:14.471876+0000","last_fullsized":"2024-04-20T17:45:14.471876+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'6767","last_scrub_stamp":"2024-04-20T17:44:21.667758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:44:21.667758+0000","objects_scrubbed":6730,"log_size":9724,"log_dups_size":0,"ondisk_log_size":9724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:45:45.503516+0000","scrub_duration":1396,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":550240256,"num_objects":8396,"num_object_clones":0,"num_object_copies":16792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9724,"num_write_kb":579840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.5","version":"814'9813","reported_seq":9974,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-04-20T17:45:14.807822+0000","last_change":"2024-04-20T17:44:23.666647+0000","last_active":"2024-04-20T17:45:14.807822+0000","last_peered":"2024-04-20T17:45:14.807822+0000","last_clean":"2024-04-20T17:45:14.807822+0000","last_became_active":"2024-04-20T17:43:18.801263+0000","last_became_peered":"2024-04-20T17:43:18.801263+0000","last_unstale":"2024-04-20T17:45:14.807822+0000","last_undegraded":"2024-04-20T17:45:14.807822+0000","last_fullsized":"2024-04-20T17:45:14.807822+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'7080","last_scrub_stamp":"2024-04-20T17:44:23.666416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:44:23.666416+0000","objects_scrubbed":7037,"log_size":9813,"log_dups_size":0,"ondisk_log_size":9813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:45:46.719009+0000","scrub_duration":1362,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":562364416,"num_objects":8581,"num_object_clones":0,"num_object_copies":17162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9813,"num_write_kb":588608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.4","version":"814'9710","reported_seq":9827,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-04-20T17:45:16.054146+0000","last_change":"2024-04-20T17:44:37.730301+0000","last_active":"2024-04-20T17:45:16.054146+0000","last_peered":"2024-04-20T17:45:16.054146+0000","last_clean":"2024-04-20T17:45:16.054146+0000","last_became_active":"2024-04-20T17:43:18.610249+0000","last_became_peered":"2024-04-20T17:43:18.610249+0000","last_unstale":"2024-04-20T17:45:16.054146+0000","last_undegraded":"2024-04-20T17:45:16.054146+0000","last_fullsized":"2024-04-20T17:45:16.054146+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":3099,"log_size":9710,"log_dups_size":0,"ondisk_log_size":9710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":544210944,"num_objects":8304,"num_object_clones":0,"num_object_copies":16608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9710,"num_write_kb":576448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.3","version":"814'9662","reported_seq":11012,"reported_epoch":814,"state":"active+recovering","last_fresh":"2024-04-20T17:45:14.458958+0000","last_change":"2024-04-20T17:44:53.425576+0000","last_active":"2024-04-20T17:45:14.458958+0000","last_peered":"2024-04-20T17:45:14.458958+0000","last_clean":"2024-04-20T17:44:51.724944+0000","last_became_active":"2024-04-20T17:44:53.356665+0000","last_became_peered":"2024-04-20T17:44:53.356665+0000","last_unstale":"2024-04-20T17:45:14.458958+0000","last_undegraded":"2024-04-20T17:45:14.458958+0000","last_fullsized":"2024-04-20T17:45:14.458958+0000","mapping_epoch":806,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":6749,"log_size":9662,"log_dups_size":0,"ondisk_log_size":9662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":555483136,"num_objects":8476,"num_object_clones":0,"num_object_copies":16952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8238,"num_objects_unfound":0,"num_objects_dirty":8476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9662,"num_write_kb":580416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":597,"num_bytes_recovered":39124992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":["11","5"],"object_location_counts":[{"shards":"1,11","objects":238},{"shards":"5,11","objects":8238}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.2","version":"814'9699","reported_seq":9861,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-04-20T17:45:14.815911+0000","last_change":"2024-04-20T17:44:25.580233+0000","last_active":"2024-04-20T17:45:14.815911+0000","last_peered":"2024-04-20T17:45:14.815911+0000","last_clean":"2024-04-20T17:45:14.815911+0000","last_became_active":"2024-04-20T17:43:18.599198+0000","last_became_peered":"2024-04-20T17:43:18.599198+0000","last_unstale":"2024-04-20T17:45:14.815911+0000","last_undegraded":"2024-04-20T17:45:14.815911+0000","last_fullsized":"2024-04-20T17:45:14.815911+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'7034","last_scrub_stamp":"2024-04-20T17:44:25.580077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:44:25.580077+0000","objects_scrubbed":6986,"log_size":9699,"log_dups_size":0,"ondisk_log_size":9699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:45:35.138580+0000","scrub_duration":1331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":551878656,"num_objects":8421,"num_object_clones":0,"num_object_copies":16842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9699,"num_write_kb":579840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.1","version":"814'9759","reported_seq":11288,"reported_epoch":814,"state":"active+recovering","last_fresh":"2024-04-20T17:45:16.132197+0000","last_change":"2024-04-20T17:44:53.450371+0000","last_active":"2024-04-20T17:45:16.132197+0000","last_peered":"2024-04-20T17:45:16.132197+0000","last_clean":"2024-04-20T17:44:51.758441+0000","last_became_active":"2024-04-20T17:44:53.399805+0000","last_became_peered":"2024-04-20T17:44:53.399805+0000","last_unstale":"2024-04-20T17:45:16.132197+0000","last_undegraded":"2024-04-20T17:45:16.132197+0000","last_fullsized":"2024-04-20T17:45:16.132197+0000","mapping_epoch":806,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":9759,"log_dups_size":0,"ondisk_log_size":9759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":545193984,"num_objects":8319,"num_object_clones":0,"num_object_copies":16638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8092,"num_objects_unfound":0,"num_objects_dirty":8319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9759,"num_write_kb":578496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":721,"num_bytes_recovered":47251456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":["9","2"],"object_location_counts":[{"shards":"1,9","objects":227},{"shards":"2,9","objects":8092}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.0","version":"814'9846","reported_seq":10027,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-04-20T17:45:14.309222+0000","last_change":"2024-04-20T17:45:05.225565+0000","last_active":"2024-04-20T17:45:14.309222+0000","last_peered":"2024-04-20T17:45:14.309222+0000","last_clean":"2024-04-20T17:45:14.309222+0000","last_became_active":"2024-04-20T17:43:18.610431+0000","last_became_peered":"2024-04-20T17:43:18.610431+0000","last_unstale":"2024-04-20T17:45:14.309222+0000","last_undegraded":"2024-04-20T17:45:14.309222+0000","last_fullsized":"2024-04-20T17:45:14.309222+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"810'9491","last_scrub_stamp":"2024-04-20T17:45:05.225287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:45:05.225287+0000","objects_scrubbed":8954,"log_size":9846,"log_dups_size":0,"ondisk_log_size":9846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:46:14.075476+0000","scrub_duration":1261,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":562298880,"num_objects":8580,"num_object_clones":0,"num_object_copies":17160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9846,"num_write_kb":589632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"814'9841","reported_seq":10006,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-04-20T17:45:14.820395+0000","last_change":"2024-04-20T17:44:30.620339+0000","last_active":"2024-04-20T17:45:14.820395+0000","last_peered":"2024-04-20T17:45:14.820395+0000","last_clean":"2024-04-20T17:45:14.820395+0000","last_became_active":"2024-04-20T17:43:18.610315+0000","last_became_peered":"2024-04-20T17:43:18.610315+0000","last_unstale":"2024-04-20T17:45:14.820395+0000","last_undegraded":"2024-04-20T17:45:14.820395+0000","last_fullsized":"2024-04-20T17:45:14.820395+0000","mapping_epoch":777,"log_start":"0'0","ondisk_log_start":"0'0","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"801'7440","last_scrub_stamp":"2024-04-20T17:44:30.620063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:44:30.620063+0000","objects_scrubbed":7386,"log_size":9841,"log_dups_size":0,"ondisk_log_size":9841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:45:51.228457+0000","scrub_duration":1395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":557776896,"num_objects":8511,"num_object_clones":0,"num_object_copies":17022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9841,"num_write_kb":587264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":668,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-04-20T17:45:10.818507+0000","last_change":"2024-04-20T17:45:04.852684+0000","last_active":"2024-04-20T17:45:10.818507+0000","last_peered":"2024-04-20T17:45:10.818507+0000","last_clean":"2024-04-20T17:45:10.818507+0000","last_became_active":"2024-04-20T17:45:04.840979+0000","last_became_peered":"2024-04-20T17:45:04.840979+0000","last_unstale":"2024-04-20T17:45:10.818507+0000","last_undegraded":"2024-04-20T17:45:10.818507+0000","last_fullsized":"2024-04-20T17:45:10.818507+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:43:41.817800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:43:41.817800+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":261,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":8828420120,"num_objects":134712,"num_object_clones":0,"num_object_copies":269424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":24488,"num_objects_unfound":0,"num_objects_dirty":134712,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155512,"num_write_kb":9287105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1931,"num_bytes_recovered":126550016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17754365952,"data_stored":17754357808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155652,"ondisk_log_size":155652,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4321280,"data_stored":4292864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":2,"acting":2,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991708,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3526764,"kb_used_data":2693156,"kb_used_omap":1,"kb_used_meta":833598,"kb_avail":90218388,"statfs":{"total":95995035648,"available":92383629312,"internally_reserved":0,"allocated":2757791744,"data_stored":2757405942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":853604804},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253347,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3397856,"kb_used_data":2735000,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":90347296,"statfs":{"total":95995035648,"available":92515631104,"internally_reserved":0,"allocated":2800640000,"data_stored":2800261822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":678754752},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613096,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2236960,"kb_used_data":1601880,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":91508192,"statfs":{"total":95995035648,"available":93704388608,"internally_reserved":0,"allocated":1640325120,"data_stored":1639946962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":650312128},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809325,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1976228,"kb_used_data":1620184,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":91768924,"statfs":{"total":95995035648,"available":93971378176,"internally_reserved":0,"allocated":1659068416,"data_stored":1658690258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":0,"seq":3431678869516,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":535,"seq":2297807503556,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":38828,"kb_used_data":1248,"kb_used_omap":170,"kb_used_meta":37397,"kb_avail":93706324,"statfs":{"total":95995035648,"available":95955275776,"internally_reserved":0,"allocated":1277952,"data_stored":896242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":174628,"internal_metadata":38295004},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":659,"seq":2830383448178,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":125408,"kb_used_data":44440,"kb_used_omap":4,"kb_used_meta":80955,"kb_avail":93619744,"statfs":{"total":95995035648,"available":95866617856,"internally_reserved":0,"allocated":45506560,"data_stored":45128402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4410,"internal_metadata":82898630},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738900,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1783840,"kb_used_data":1076568,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":91961312,"statfs":{"total":95995035648,"available":94168383488,"internally_reserved":0,"allocated":1102405632,"data_stored":1102027474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394190,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3113572,"kb_used_data":2716700,"kb_used_omap":202,"kb_used_meta":396661,"kb_avail":90631580,"statfs":{"total":95995035648,"available":92806737920,"internally_reserved":0,"allocated":2781900800,"data_stored":2781518570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":406181335},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653239,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1538912,"kb_used_data":1087256,"kb_used_omap":4,"kb_used_meta":451643,"kb_avail":92206240,"statfs":{"total":95995035648,"available":94419189760,"internally_reserved":0,"allocated":1113350144,"data_stored":1112971986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":462483143},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986688,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1637408,"kb_used_data":1074456,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":92107744,"statfs":{"total":95995035648,"available":94318329856,"internally_reserved":0,"allocated":1100242944,"data_stored":1099864786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":576387525},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790458,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3439840,"kb_used_data":2722840,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":90305312,"statfs":{"total":95995035648,"available":92472639488,"internally_reserved":0,"allocated":2788188160,"data_stored":2787810002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":734198213},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1098776576,"data_stored":1098776576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":44630016,"data_stored":44630016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2775257088,"data_stored":2775253016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1110376448,"data_stored":1110376448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1096024064,"data_stored":1096024064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2783051776,"data_stored":2783051776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1656487936,"data_stored":1656487936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1638793216,"data_stored":1638793216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2796224512,"data_stored":2796224512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2754744320,"data_stored":2754740248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:45:19.590 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-04-20T17:45:19.590 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-04-20T17:45:19.610 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:45:19.625 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:45:19.625 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:45:19.625 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:45:19.625 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:45:19.625 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:45:19.626 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:45:19.626 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:45:19.626 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:45:19.626 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:45:19.626 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:45:19.626 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:45:19.626 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:45:19.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:19 smithi005 ceph-mon[39562]: pgmap v2182: 17 pgs: 3 active+recovering, 14 active+clean; 8.2 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 481 op/s; 24488/269428 objects misplaced (9.089%); 5.9 MiB/s, 95 objects/s recovering 2024-04-20T17:45:19.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:19 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4206759800' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:45:19.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:19 smithi106 ceph-mon[44423]: pgmap v2182: 17 pgs: 3 active+recovering, 14 active+clean; 8.2 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 481 op/s; 24488/269428 objects misplaced (9.089%); 5.9 MiB/s, 95 objects/s recovering 2024-04-20T17:45:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:19 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4206759800' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:45:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:19 smithi171 ceph-mon[44189]: pgmap v2182: 17 pgs: 3 active+recovering, 14 active+clean; 8.2 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 481 op/s; 24488/269428 objects misplaced (9.089%); 5.9 MiB/s, 95 objects/s recovering 2024-04-20T17:45:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:19 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4206759800' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:45:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:20.513 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:45:20.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:20.523 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:45:20.624 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:45:20.640 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:45:20.650 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:20.669 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:45:20.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:20.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:45:20.773 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:45:20.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:20.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:20.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:20.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:20.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:20.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:20.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:20.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:20.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:20.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:20.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:20.816 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:20.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:21.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:21.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:21.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:21.216 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:45:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:45:21.763 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:21 smithi005 ceph-mon[39562]: pgmap v2183: 17 pgs: 3 active+recovering, 14 active+clean; 8.1 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 662 op/s; 24105/264824 objects misplaced (9.102%); 7.3 MiB/s, 117 objects/s recovering 2024-04-20T17:45:22.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:21 smithi106 ceph-mon[44423]: pgmap v2183: 17 pgs: 3 active+recovering, 14 active+clean; 8.1 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 662 op/s; 24105/264824 objects misplaced (9.102%); 7.3 MiB/s, 117 objects/s recovering 2024-04-20T17:45:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:21 smithi171 ceph-mon[44189]: pgmap v2183: 17 pgs: 3 active+recovering, 14 active+clean; 8.1 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 662 op/s; 24105/264824 objects misplaced (9.102%); 7.3 MiB/s, 117 objects/s recovering 2024-04-20T17:45:22.812 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:45:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:23 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:45:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:23 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:45:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:23 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:45:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:23 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:45:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:23 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:45:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:23 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:45:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:23 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:45:24.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:23 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:45:24.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:23 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:45:24.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:23 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:45:24.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:23 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:45:24.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:23 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:45:24.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:23 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:45:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:23 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:45:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:23 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:45:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:23 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:45:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:23 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:45:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:23 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:45:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:23 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:45:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:23 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:45:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:23 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:45:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:23 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:45:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:23 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:45:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:23 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:45:24.147 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:45:24.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:23 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:45:24.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:23 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:45:24.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:23 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:45:24.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:23 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:45:24.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:23 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:45:24.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:23 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:45:24.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:23 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:45:24.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:23 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:45:24.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:23 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:45:24.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:23 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:45:24.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:23 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:45:24.148 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:23 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:45:24.746 INFO:teuthology.orchestra.run.smithi005.stderr:set pool 1 pg_num to 11 2024-04-20T17:45:24.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:24 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2752860912' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-20T17:45:25.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:24 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2752860912' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-20T17:45:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:24 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2752860912' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-20T17:45:25.924 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:45:25.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:25.936 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:45:25.968 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-20T17:45:26.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:25 smithi106 ceph-mon[44423]: pgmap v2185: 17 pgs: 3 active+recovering, 14 active+clean; 7.9 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 587 op/s; 23556/260026 objects misplaced (9.059%); 7.3 MiB/s, 116 objects/s recovering 2024-04-20T17:45:26.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:25 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2752860912' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-20T17:45:26.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:25 smithi106 ceph-mon[44423]: osdmap e817: 12 total, 12 up, 11 in 2024-04-20T17:45:26.037 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:45:26.057 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:25 smithi171 ceph-mon[44189]: pgmap v2185: 17 pgs: 3 active+recovering, 14 active+clean; 7.9 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 587 op/s; 23556/260026 objects misplaced (9.059%); 7.3 MiB/s, 116 objects/s recovering 2024-04-20T17:45:26.057 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:25 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2752860912' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-20T17:45:26.057 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:25 smithi171 ceph-mon[44189]: osdmap e817: 12 total, 12 up, 11 in 2024-04-20T17:45:26.063 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:26.081 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:45:26.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:26.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:45:26.143 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:25 smithi005 ceph-mon[39562]: pgmap v2185: 17 pgs: 3 active+recovering, 14 active+clean; 7.9 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 587 op/s; 23556/260026 objects misplaced (9.059%); 7.3 MiB/s, 116 objects/s recovering 2024-04-20T17:45:26.143 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:25 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2752860912' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-20T17:45:26.143 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:25 smithi005 ceph-mon[39562]: osdmap e817: 12 total, 12 up, 11 in 2024-04-20T17:45:26.185 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:45:26.223 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:26.463 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:45:26.678 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:45:26.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:26.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:45:26.759 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:45:26.787 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:45:26.815 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:26.835 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:45:26.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:26.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:45:26.941 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:45:26.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:26 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/870134609' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:45:26.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:26 smithi005 ceph-mon[39562]: from='client.27734 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:45:26.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:26 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-20T17:45:26.994 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:27.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:26 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/870134609' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:45:27.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:26 smithi106 ceph-mon[44423]: from='client.27734 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:45:27.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:26 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-20T17:45:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:26 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/870134609' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:45:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:26 smithi171 ceph-mon[44189]: from='client.27734 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:45:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:26 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-20T17:45:27.378 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:45:27.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:27.381 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:45:27.482 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:45:27.527 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:27 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:45:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:27 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:45:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:27 smithi106 ceph-mon[44423]: pgmap v2187: 17 pgs: 3 active+recovering, 14 active+clean; 7.9 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 673 op/s; 23373/257442 objects misplaced (9.079%); 7.8 MiB/s, 125 objects/s recovering 2024-04-20T17:45:28.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:27 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:45:28.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:27 smithi106 ceph-mon[44423]: from='client.27734 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:45:28.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:27 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-20T17:45:28.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:27 smithi106 ceph-mon[44423]: osdmap e818: 12 total, 12 up, 11 in 2024-04-20T17:45:28.067 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:45:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:27 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:45:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:27 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:45:28.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:27 smithi171 ceph-mon[44189]: pgmap v2187: 17 pgs: 3 active+recovering, 14 active+clean; 7.9 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 673 op/s; 23373/257442 objects misplaced (9.079%); 7.8 MiB/s, 125 objects/s recovering 2024-04-20T17:45:28.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:27 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:45:28.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:27 smithi171 ceph-mon[44189]: from='client.27734 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:45:28.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:27 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-20T17:45:28.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:27 smithi171 ceph-mon[44189]: osdmap e818: 12 total, 12 up, 11 in 2024-04-20T17:45:28.091 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:45:28.114 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:45:28.145 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:45:28.179 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:45:28.205 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:45:28.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:27 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:45:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:27 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:45:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:27 smithi005 ceph-mon[39562]: pgmap v2187: 17 pgs: 3 active+recovering, 14 active+clean; 7.9 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 673 op/s; 23373/257442 objects misplaced (9.079%); 7.8 MiB/s, 125 objects/s recovering 2024-04-20T17:45:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:27 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:45:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:27 smithi005 ceph-mon[39562]: from='client.27734 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:45:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:27 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-20T17:45:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:27 smithi005 ceph-mon[39562]: osdmap e818: 12 total, 12 up, 11 in 2024-04-20T17:45:28.230 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:45:28.257 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:45:28.284 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:45:28.315 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:45:28.347 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:45:28.371 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:45:28.374 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:45:28.374 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:45:28.374 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:45:28.374 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:45:28.374 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:45:28.375 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:45:28.375 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:45:28.375 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:45:28.375 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:45:28.375 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:45:28.375 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:45:28.375 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:45:29.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:28 smithi171 ceph-mon[44189]: osdmap e819: 12 total, 12 up, 11 in 2024-04-20T17:45:29.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:28 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/150784703' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:45:29.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:29.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:29.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:28 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-20T17:45:29.189 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:28 smithi005 ceph-mon[39562]: osdmap e819: 12 total, 12 up, 11 in 2024-04-20T17:45:29.189 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:28 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/150784703' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:45:29.190 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:29.190 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:29.190 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:28 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-20T17:45:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:28 smithi106 ceph-mon[44423]: osdmap e819: 12 total, 12 up, 11 in 2024-04-20T17:45:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:28 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/150784703' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:45:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:28 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-20T17:45:29.770 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:45:30.083 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:45:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:29 smithi171 ceph-mon[44189]: pgmap v2190: 27 pgs: 10 unknown, 3 active+recovering, 14 active+clean; 7.9 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 490 op/s; 23373/257442 objects misplaced (9.079%); 5.8 MiB/s, 92 objects/s recovering 2024-04-20T17:45:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:29 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-20T17:45:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:29 smithi171 ceph-mon[44189]: osdmap e820: 12 total, 12 up, 11 in 2024-04-20T17:45:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:29 smithi005 ceph-mon[39562]: pgmap v2190: 27 pgs: 10 unknown, 3 active+recovering, 14 active+clean; 7.9 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 490 op/s; 23373/257442 objects misplaced (9.079%); 5.8 MiB/s, 92 objects/s recovering 2024-04-20T17:45:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:29 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-20T17:45:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:29 smithi005 ceph-mon[39562]: osdmap e820: 12 total, 12 up, 11 in 2024-04-20T17:45:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:30.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:29 smithi106 ceph-mon[44423]: pgmap v2190: 27 pgs: 10 unknown, 3 active+recovering, 14 active+clean; 7.9 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 490 op/s; 23373/257442 objects misplaced (9.079%); 5.8 MiB/s, 92 objects/s recovering 2024-04-20T17:45:30.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:29 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-20T17:45:30.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:29 smithi106 ceph-mon[44423]: osdmap e820: 12 total, 12 up, 11 in 2024-04-20T17:45:30.719 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:45:30.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:30.728 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:45:30.829 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:45:30.865 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:30.897 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:45:30.899 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:30 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:45:30.899 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:30 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:45:30.899 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:30 smithi005 ceph-mon[39562]: osdmap e821: 12 total, 12 up, 11 in 2024-04-20T17:45:30.899 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:30 smithi005 ceph-mon[39562]: pgmap v2193: 27 pgs: 1 peering, 9 unknown, 3 active+recovering, 14 active+clean; 7.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 506 op/s; 23006/253088 objects misplaced (9.090%); 5.1 MiB/s, 89 objects/s recovering 2024-04-20T17:45:30.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:30.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:45:31.001 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:45:31.031 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:31.049 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:45:31.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:31.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:45:31.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:30 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:45:31.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:30 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:45:31.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:30 smithi171 ceph-mon[44189]: osdmap e821: 12 total, 12 up, 11 in 2024-04-20T17:45:31.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:30 smithi171 ceph-mon[44189]: pgmap v2193: 27 pgs: 1 peering, 9 unknown, 3 active+recovering, 14 active+clean; 7.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 506 op/s; 23006/253088 objects misplaced (9.090%); 5.1 MiB/s, 89 objects/s recovering 2024-04-20T17:45:31.152 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:45:31.187 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 8, 9, 10, 11, 1, 6] out_osds: [7] dead_osds: [] live_osds: [0, 4, 5, 8, 9, 10, 11, 2, 6, 3, 1, 7] 2024-04-20T17:45:31.187 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:45:31.187 INFO:tasks.thrashosds.thrasher:inject_pause on osd.8 2024-04-20T17:45:31.187 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-20T17:45:31.187 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-20T17:45:31.188 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 config set filestore_inject_stall 3 2024-04-20T17:45:31.194 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:31.226 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:45:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:30 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:45:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:30 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:45:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:30 smithi106 ceph-mon[44423]: osdmap e821: 12 total, 12 up, 11 in 2024-04-20T17:45:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:30 smithi106 ceph-mon[44423]: pgmap v2193: 27 pgs: 1 peering, 9 unknown, 3 active+recovering, 14 active+clean; 7.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 506 op/s; 23006/253088 objects misplaced (9.090%); 5.1 MiB/s, 89 objects/s recovering 2024-04-20T17:45:31.973 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:45:31.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:31.979 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:45:32.073 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:45:32.080 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:45:32.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:31 smithi171 ceph-mon[44189]: osdmap e822: 12 total, 12 up, 11 in 2024-04-20T17:45:32.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:31 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:45:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:31 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:45:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:31 smithi171 ceph-mon[44189]: osdmap e823: 12 total, 12 up, 11 in 2024-04-20T17:45:32.128 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:31 smithi005 ceph-mon[39562]: osdmap e822: 12 total, 12 up, 11 in 2024-04-20T17:45:32.129 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:31 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:45:32.129 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:31 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:45:32.129 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:31 smithi005 ceph-mon[39562]: osdmap e823: 12 total, 12 up, 11 in 2024-04-20T17:45:32.132 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:31 smithi106 ceph-mon[44423]: osdmap e822: 12 total, 12 up, 11 in 2024-04-20T17:45:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:31 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:45:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:31 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:45:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:31 smithi106 ceph-mon[44423]: osdmap e823: 12 total, 12 up, 11 in 2024-04-20T17:45:32.390 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:45:32.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:32.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:45:32.498 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:45:32.546 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:32.575 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:45:32.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:32.584 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:45:32.650 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:45:32.678 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:45:32.685 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:45:32.714 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:32.747 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:45:32.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:32.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:45:32.856 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:45:32.888 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:32.914 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:45:32.914 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:45:32.914 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-04-20T17:45:32.914 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-04-20T17:45:32.915 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:45:32.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/58956225' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:45:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: from='client.27746 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:45:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=58} crc {auth=2677080368,config=1460075751,health=179246142,kv=2354758727}) 2024-04-20T17:45:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=1310191504,logm=993565514}) 2024-04-20T17:45:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2425830205}) 2024-04-20T17:45:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4294205413}) 2024-04-20T17:45:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4045296755}) 2024-04-20T17:45:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2935496165}) 2024-04-20T17:45:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4086942721}) 2024-04-20T17:45:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1861088701}) 2024-04-20T17:45:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3721944742}) 2024-04-20T17:45:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2695590936}) 2024-04-20T17:45:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3208083200}) 2024-04-20T17:45:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=426509752}) 2024-04-20T17:45:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=845432947}) 2024-04-20T17:45:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3365957582}) 2024-04-20T17:45:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2111016058}) 2024-04-20T17:45:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=492415555}) 2024-04-20T17:45:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2444684686}) 2024-04-20T17:45:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2045683089}) 2024-04-20T17:45:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1142789499}) 2024-04-20T17:45:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1218398705}) 2024-04-20T17:45:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2605950539}) 2024-04-20T17:45:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1458456477}) 2024-04-20T17:45:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2668672706}) 2024-04-20T17:45:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1692965189}) 2024-04-20T17:45:32.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3039674431}) 2024-04-20T17:45:32.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=473230430}) 2024-04-20T17:45:32.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3343019570}) 2024-04-20T17:45:32.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2398556756}) 2024-04-20T17:45:32.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2031389019}) 2024-04-20T17:45:32.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2864224665}) 2024-04-20T17:45:32.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3249336666}) 2024-04-20T17:45:32.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3544475177}) 2024-04-20T17:45:32.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=575460961}) 2024-04-20T17:45:32.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=124785533}) 2024-04-20T17:45:32.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4030656209}) 2024-04-20T17:45:32.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1319601372}) 2024-04-20T17:45:32.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1687863863}) 2024-04-20T17:45:32.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2871747206}) 2024-04-20T17:45:32.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2537579236}) 2024-04-20T17:45:32.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1700265259}) 2024-04-20T17:45:32.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3830441113}) 2024-04-20T17:45:32.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2986151609}) 2024-04-20T17:45:32.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3187942396}) 2024-04-20T17:45:32.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2081365323}) 2024-04-20T17:45:32.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=543757466}) 2024-04-20T17:45:32.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=157728373}) 2024-04-20T17:45:32.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2958783461}) 2024-04-20T17:45:32.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4072203451}) 2024-04-20T17:45:32.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2774169608}) 2024-04-20T17:45:32.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2991150496}) 2024-04-20T17:45:32.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: pgmap v2196: 27 pgs: 1 peering, 7 unknown, 3 active+recovering, 16 active+clean; 7.7 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 699 op/s; 22815/251608 objects misplaced (9.068%); 8.2 MiB/s, 137 objects/s recovering 2024-04-20T17:45:32.958 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3897666895}) 2024-04-20T17:45:32.958 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3940108767}) 2024-04-20T17:45:32.958 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=203733332}) 2024-04-20T17:45:32.958 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2122337612}) 2024-04-20T17:45:32.958 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:45:32.958 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4267119945}) 2024-04-20T17:45:32.958 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: from='client.27746 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:45:32.958 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: osdmap e824: 12 total, 12 up, 11 in 2024-04-20T17:45:32.958 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=227523159}) 2024-04-20T17:45:32.958 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1359698913}) 2024-04-20T17:45:32.958 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=815281831}) 2024-04-20T17:45:32.958 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=541171129}) 2024-04-20T17:45:32.958 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=193720823}) 2024-04-20T17:45:32.958 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2652598185}) 2024-04-20T17:45:32.959 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1248697316}) 2024-04-20T17:45:32.959 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3712264339}) 2024-04-20T17:45:32.959 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1219109462}) 2024-04-20T17:45:32.959 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2034086587}) 2024-04-20T17:45:32.959 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=777096879}) 2024-04-20T17:45:32.959 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=770857416}) 2024-04-20T17:45:32.959 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3114204656}) 2024-04-20T17:45:32.959 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=605773684}) 2024-04-20T17:45:32.959 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1654345723}) 2024-04-20T17:45:32.959 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3044802890}) 2024-04-20T17:45:32.959 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4181747819}) 2024-04-20T17:45:32.959 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4225452756}) 2024-04-20T17:45:32.959 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3884352989}) 2024-04-20T17:45:32.959 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4059788905}) 2024-04-20T17:45:32.960 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2821273682}) 2024-04-20T17:45:32.960 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262952219}) 2024-04-20T17:45:32.960 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3668871932}) 2024-04-20T17:45:32.960 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3358275172}) 2024-04-20T17:45:32.960 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1465666986}) 2024-04-20T17:45:32.960 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=829104967}) 2024-04-20T17:45:32.960 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3770121892}) 2024-04-20T17:45:32.960 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1004618595}) 2024-04-20T17:45:32.960 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=683949072}) 2024-04-20T17:45:32.960 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3178096328}) 2024-04-20T17:45:32.960 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1361342177}) 2024-04-20T17:45:32.960 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=366898477}) 2024-04-20T17:45:32.960 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3877608970}) 2024-04-20T17:45:32.960 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=45846545}) 2024-04-20T17:45:33.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/58956225' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:45:33.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: from='client.27746 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:45:33.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=58} crc {auth=2677080368,config=1460075751,health=179246142,kv=2354758727}) 2024-04-20T17:45:33.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=1310191504,logm=993565514}) 2024-04-20T17:45:33.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2425830205}) 2024-04-20T17:45:33.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4294205413}) 2024-04-20T17:45:33.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4045296755}) 2024-04-20T17:45:33.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2935496165}) 2024-04-20T17:45:33.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4086942721}) 2024-04-20T17:45:33.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1861088701}) 2024-04-20T17:45:33.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3721944742}) 2024-04-20T17:45:33.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2695590936}) 2024-04-20T17:45:33.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3208083200}) 2024-04-20T17:45:33.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=426509752}) 2024-04-20T17:45:33.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=845432947}) 2024-04-20T17:45:33.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3365957582}) 2024-04-20T17:45:33.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2111016058}) 2024-04-20T17:45:33.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=492415555}) 2024-04-20T17:45:33.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2444684686}) 2024-04-20T17:45:33.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2045683089}) 2024-04-20T17:45:33.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1142789499}) 2024-04-20T17:45:33.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1218398705}) 2024-04-20T17:45:33.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2605950539}) 2024-04-20T17:45:33.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1458456477}) 2024-04-20T17:45:33.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2668672706}) 2024-04-20T17:45:33.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1692965189}) 2024-04-20T17:45:33.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3039674431}) 2024-04-20T17:45:33.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=473230430}) 2024-04-20T17:45:33.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3343019570}) 2024-04-20T17:45:33.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2398556756}) 2024-04-20T17:45:33.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2031389019}) 2024-04-20T17:45:33.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2864224665}) 2024-04-20T17:45:33.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3249336666}) 2024-04-20T17:45:33.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3544475177}) 2024-04-20T17:45:33.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=575460961}) 2024-04-20T17:45:33.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=124785533}) 2024-04-20T17:45:33.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4030656209}) 2024-04-20T17:45:33.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1319601372}) 2024-04-20T17:45:33.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1687863863}) 2024-04-20T17:45:33.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2871747206}) 2024-04-20T17:45:33.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2537579236}) 2024-04-20T17:45:33.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1700265259}) 2024-04-20T17:45:33.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3830441113}) 2024-04-20T17:45:33.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2986151609}) 2024-04-20T17:45:33.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3187942396}) 2024-04-20T17:45:33.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2081365323}) 2024-04-20T17:45:33.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=543757466}) 2024-04-20T17:45:33.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=157728373}) 2024-04-20T17:45:33.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2958783461}) 2024-04-20T17:45:33.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4072203451}) 2024-04-20T17:45:33.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2774169608}) 2024-04-20T17:45:33.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2991150496}) 2024-04-20T17:45:33.015 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: pgmap v2196: 27 pgs: 1 peering, 7 unknown, 3 active+recovering, 16 active+clean; 7.7 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 699 op/s; 22815/251608 objects misplaced (9.068%); 8.2 MiB/s, 137 objects/s recovering 2024-04-20T17:45:33.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3897666895}) 2024-04-20T17:45:33.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3940108767}) 2024-04-20T17:45:33.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=203733332}) 2024-04-20T17:45:33.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2122337612}) 2024-04-20T17:45:33.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:45:33.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4267119945}) 2024-04-20T17:45:33.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: from='client.27746 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:45:33.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: osdmap e824: 12 total, 12 up, 11 in 2024-04-20T17:45:33.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=227523159}) 2024-04-20T17:45:33.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1359698913}) 2024-04-20T17:45:33.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=815281831}) 2024-04-20T17:45:33.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=541171129}) 2024-04-20T17:45:33.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=193720823}) 2024-04-20T17:45:33.016 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2652598185}) 2024-04-20T17:45:33.017 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1248697316}) 2024-04-20T17:45:33.017 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3712264339}) 2024-04-20T17:45:33.017 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1219109462}) 2024-04-20T17:45:33.017 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2034086587}) 2024-04-20T17:45:33.017 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=777096879}) 2024-04-20T17:45:33.017 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=770857416}) 2024-04-20T17:45:33.017 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3114204656}) 2024-04-20T17:45:33.017 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=605773684}) 2024-04-20T17:45:33.017 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1654345723}) 2024-04-20T17:45:33.017 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3044802890}) 2024-04-20T17:45:33.017 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4181747819}) 2024-04-20T17:45:33.017 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4225452756}) 2024-04-20T17:45:33.017 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3884352989}) 2024-04-20T17:45:33.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4059788905}) 2024-04-20T17:45:33.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2821273682}) 2024-04-20T17:45:33.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262952219}) 2024-04-20T17:45:33.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3668871932}) 2024-04-20T17:45:33.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3358275172}) 2024-04-20T17:45:33.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1465666986}) 2024-04-20T17:45:33.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=829104967}) 2024-04-20T17:45:33.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3770121892}) 2024-04-20T17:45:33.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1004618595}) 2024-04-20T17:45:33.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=683949072}) 2024-04-20T17:45:33.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3178096328}) 2024-04-20T17:45:33.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1361342177}) 2024-04-20T17:45:33.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=366898477}) 2024-04-20T17:45:33.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3877608970}) 2024-04-20T17:45:33.018 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=45846545}) 2024-04-20T17:45:33.138 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:45:33.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:33.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:45:33.242 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:45:33.300 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/58956225' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:45:33.300 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: from='client.27746 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:45:33.300 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=58} crc {auth=2677080368,config=1460075751,health=179246142,kv=2354758727}) 2024-04-20T17:45:33.300 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=1310191504,logm=993565514}) 2024-04-20T17:45:33.301 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2425830205}) 2024-04-20T17:45:33.301 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4294205413}) 2024-04-20T17:45:33.301 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4045296755}) 2024-04-20T17:45:33.301 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2935496165}) 2024-04-20T17:45:33.301 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4086942721}) 2024-04-20T17:45:33.301 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1861088701}) 2024-04-20T17:45:33.301 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3721944742}) 2024-04-20T17:45:33.301 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2695590936}) 2024-04-20T17:45:33.301 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3208083200}) 2024-04-20T17:45:33.301 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=426509752}) 2024-04-20T17:45:33.301 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=845432947}) 2024-04-20T17:45:33.301 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3365957582}) 2024-04-20T17:45:33.301 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2111016058}) 2024-04-20T17:45:33.301 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=492415555}) 2024-04-20T17:45:33.302 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2444684686}) 2024-04-20T17:45:33.302 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2045683089}) 2024-04-20T17:45:33.302 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1142789499}) 2024-04-20T17:45:33.302 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1218398705}) 2024-04-20T17:45:33.302 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2605950539}) 2024-04-20T17:45:33.302 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1458456477}) 2024-04-20T17:45:33.302 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2668672706}) 2024-04-20T17:45:33.302 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1692965189}) 2024-04-20T17:45:33.302 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3039674431}) 2024-04-20T17:45:33.302 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=473230430}) 2024-04-20T17:45:33.302 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3343019570}) 2024-04-20T17:45:33.302 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2398556756}) 2024-04-20T17:45:33.302 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2031389019}) 2024-04-20T17:45:33.302 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2864224665}) 2024-04-20T17:45:33.303 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3249336666}) 2024-04-20T17:45:33.303 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3544475177}) 2024-04-20T17:45:33.303 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=575460961}) 2024-04-20T17:45:33.303 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=124785533}) 2024-04-20T17:45:33.303 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4030656209}) 2024-04-20T17:45:33.303 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1319601372}) 2024-04-20T17:45:33.303 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1687863863}) 2024-04-20T17:45:33.303 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2871747206}) 2024-04-20T17:45:33.303 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2537579236}) 2024-04-20T17:45:33.303 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1700265259}) 2024-04-20T17:45:33.303 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3830441113}) 2024-04-20T17:45:33.303 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2986151609}) 2024-04-20T17:45:33.303 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3187942396}) 2024-04-20T17:45:33.303 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2081365323}) 2024-04-20T17:45:33.303 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=543757466}) 2024-04-20T17:45:33.304 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=157728373}) 2024-04-20T17:45:33.304 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2958783461}) 2024-04-20T17:45:33.304 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4072203451}) 2024-04-20T17:45:33.304 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2774169608}) 2024-04-20T17:45:33.304 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2991150496}) 2024-04-20T17:45:33.304 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: pgmap v2196: 27 pgs: 1 peering, 7 unknown, 3 active+recovering, 16 active+clean; 7.7 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 699 op/s; 22815/251608 objects misplaced (9.068%); 8.2 MiB/s, 137 objects/s recovering 2024-04-20T17:45:33.304 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3897666895}) 2024-04-20T17:45:33.304 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3940108767}) 2024-04-20T17:45:33.304 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=203733332}) 2024-04-20T17:45:33.304 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2122337612}) 2024-04-20T17:45:33.304 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:45:33.304 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4267119945}) 2024-04-20T17:45:33.304 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: from='client.27746 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:45:33.304 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: osdmap e824: 12 total, 12 up, 11 in 2024-04-20T17:45:33.304 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=227523159}) 2024-04-20T17:45:33.305 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1359698913}) 2024-04-20T17:45:33.305 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=815281831}) 2024-04-20T17:45:33.305 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=541171129}) 2024-04-20T17:45:33.305 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=193720823}) 2024-04-20T17:45:33.305 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2652598185}) 2024-04-20T17:45:33.305 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1248697316}) 2024-04-20T17:45:33.305 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3712264339}) 2024-04-20T17:45:33.305 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1219109462}) 2024-04-20T17:45:33.305 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2034086587}) 2024-04-20T17:45:33.305 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=777096879}) 2024-04-20T17:45:33.305 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=770857416}) 2024-04-20T17:45:33.305 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3114204656}) 2024-04-20T17:45:33.305 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=605773684}) 2024-04-20T17:45:33.305 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1654345723}) 2024-04-20T17:45:33.305 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3044802890}) 2024-04-20T17:45:33.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4181747819}) 2024-04-20T17:45:33.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4225452756}) 2024-04-20T17:45:33.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3884352989}) 2024-04-20T17:45:33.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4059788905}) 2024-04-20T17:45:33.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2821273682}) 2024-04-20T17:45:33.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=262952219}) 2024-04-20T17:45:33.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3668871932}) 2024-04-20T17:45:33.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3358275172}) 2024-04-20T17:45:33.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1465666986}) 2024-04-20T17:45:33.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=829104967}) 2024-04-20T17:45:33.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3770121892}) 2024-04-20T17:45:33.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1004618595}) 2024-04-20T17:45:33.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=683949072}) 2024-04-20T17:45:33.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3178096328}) 2024-04-20T17:45:33.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1361342177}) 2024-04-20T17:45:33.307 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=366898477}) 2024-04-20T17:45:33.307 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3877608970}) 2024-04-20T17:45:33.307 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=45846545}) 2024-04-20T17:45:33.307 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:33.848 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2386447687}) 2024-04-20T17:45:33.849 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=301098995}) 2024-04-20T17:45:33.849 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2752463642}) 2024-04-20T17:45:33.849 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4098903940}) 2024-04-20T17:45:33.849 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=251233301}) 2024-04-20T17:45:33.849 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3833466683}) 2024-04-20T17:45:33.849 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2163435783}) 2024-04-20T17:45:33.849 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2246188118}) 2024-04-20T17:45:33.849 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=568971495}) 2024-04-20T17:45:33.849 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=947285433}) 2024-04-20T17:45:33.849 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=25,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=9,mon_config_key=42} crc {logm=1557355338,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=2170516910,mon_config_key=696683432}) 2024-04-20T17:45:33.849 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3989188918}) 2024-04-20T17:45:33.849 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4096526593}) 2024-04-20T17:45:33.850 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=48,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=34} crc {mon_config_key=597173981,monmap=717113318,osd_metadata=2589079959,osd_pg_creating=3603324599,osdmap=1202788847}) 2024-04-20T17:45:33.850 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=85} crc {osdmap=2034022334}) 2024-04-20T17:45:33.850 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:33 smithi005 ceph-mon[39562]: 1.5 scrub starts 2024-04-20T17:45:33.850 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:33 smithi005 ceph-mon[39562]: 1.5 scrub ok 2024-04-20T17:45:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2386447687}) 2024-04-20T17:45:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=301098995}) 2024-04-20T17:45:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2752463642}) 2024-04-20T17:45:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4098903940}) 2024-04-20T17:45:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=251233301}) 2024-04-20T17:45:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3833466683}) 2024-04-20T17:45:34.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2163435783}) 2024-04-20T17:45:34.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2246188118}) 2024-04-20T17:45:34.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=568971495}) 2024-04-20T17:45:34.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=947285433}) 2024-04-20T17:45:34.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=25,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=9,mon_config_key=42} crc {logm=1557355338,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=2170516910,mon_config_key=696683432}) 2024-04-20T17:45:34.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3989188918}) 2024-04-20T17:45:34.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4096526593}) 2024-04-20T17:45:34.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=48,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=34} crc {mon_config_key=597173981,monmap=717113318,osd_metadata=2589079959,osd_pg_creating=3603324599,osdmap=1202788847}) 2024-04-20T17:45:34.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=85} crc {osdmap=2034022334}) 2024-04-20T17:45:34.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:33 smithi106 ceph-mon[44423]: 1.5 scrub starts 2024-04-20T17:45:34.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:33 smithi106 ceph-mon[44423]: 1.5 scrub ok 2024-04-20T17:45:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2386447687}) 2024-04-20T17:45:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=301098995}) 2024-04-20T17:45:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2752463642}) 2024-04-20T17:45:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4098903940}) 2024-04-20T17:45:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=251233301}) 2024-04-20T17:45:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3833466683}) 2024-04-20T17:45:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2163435783}) 2024-04-20T17:45:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2246188118}) 2024-04-20T17:45:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=568971495}) 2024-04-20T17:45:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=947285433}) 2024-04-20T17:45:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=25,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=9,mon_config_key=42} crc {logm=1557355338,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=2170516910,mon_config_key=696683432}) 2024-04-20T17:45:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3989188918}) 2024-04-20T17:45:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4096526593}) 2024-04-20T17:45:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=48,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=34} crc {mon_config_key=597173981,monmap=717113318,osd_metadata=2589079959,osd_pg_creating=3603324599,osdmap=1202788847}) 2024-04-20T17:45:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=85} crc {osdmap=2034022334}) 2024-04-20T17:45:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:33 smithi171 ceph-mon[44189]: 1.5 scrub starts 2024-04-20T17:45:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:33 smithi171 ceph-mon[44189]: 1.5 scrub ok 2024-04-20T17:45:34.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:34 smithi005 ceph-mon[39562]: 1.2 scrub starts 2024-04-20T17:45:34.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:34 smithi005 ceph-mon[39562]: 1.8 scrub starts 2024-04-20T17:45:34.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:34 smithi005 ceph-mon[39562]: 1.2 scrub ok 2024-04-20T17:45:34.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:34 smithi005 ceph-mon[39562]: 1.8 scrub ok 2024-04-20T17:45:34.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:34 smithi005 ceph-mon[39562]: 1.6 scrub starts 2024-04-20T17:45:34.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:34 smithi005 ceph-mon[39562]: pgmap v2198: 27 pgs: 1 peering, 7 unknown, 3 active+recovering, 16 active+clean; 7.6 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 4.2 KiB/s rd, 111 KiB/s wr, 410 op/s; 22815/250324 objects misplaced (9.114%); 2.6 MiB/s, 40 objects/s recovering 2024-04-20T17:45:34.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:34 smithi005 ceph-mon[39562]: 1.6 scrub ok 2024-04-20T17:45:35.063 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:45:35.092 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:45:35.119 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:45:35.149 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:45:35.172 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:45:35.195 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:45:35.225 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:45:35.250 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:45:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:34 smithi106 ceph-mon[44423]: 1.2 scrub starts 2024-04-20T17:45:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:34 smithi106 ceph-mon[44423]: 1.8 scrub starts 2024-04-20T17:45:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:34 smithi106 ceph-mon[44423]: 1.2 scrub ok 2024-04-20T17:45:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:34 smithi106 ceph-mon[44423]: 1.8 scrub ok 2024-04-20T17:45:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:34 smithi106 ceph-mon[44423]: 1.6 scrub starts 2024-04-20T17:45:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:34 smithi106 ceph-mon[44423]: pgmap v2198: 27 pgs: 1 peering, 7 unknown, 3 active+recovering, 16 active+clean; 7.6 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 4.2 KiB/s rd, 111 KiB/s wr, 410 op/s; 22815/250324 objects misplaced (9.114%); 2.6 MiB/s, 40 objects/s recovering 2024-04-20T17:45:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:34 smithi106 ceph-mon[44423]: 1.6 scrub ok 2024-04-20T17:45:35.284 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:45:35.314 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:45:35.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:34 smithi171 ceph-mon[44189]: 1.2 scrub starts 2024-04-20T17:45:35.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:34 smithi171 ceph-mon[44189]: 1.8 scrub starts 2024-04-20T17:45:35.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:34 smithi171 ceph-mon[44189]: 1.2 scrub ok 2024-04-20T17:45:35.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:34 smithi171 ceph-mon[44189]: 1.8 scrub ok 2024-04-20T17:45:35.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:34 smithi171 ceph-mon[44189]: 1.6 scrub starts 2024-04-20T17:45:35.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:34 smithi171 ceph-mon[44189]: pgmap v2198: 27 pgs: 1 peering, 7 unknown, 3 active+recovering, 16 active+clean; 7.6 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 4.2 KiB/s rd, 111 KiB/s wr, 410 op/s; 22815/250324 objects misplaced (9.114%); 2.6 MiB/s, 40 objects/s recovering 2024-04-20T17:45:35.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:34 smithi171 ceph-mon[44189]: 1.6 scrub ok 2024-04-20T17:45:35.346 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:45:35.372 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:45:35.374 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:45:35.374 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:45:35.374 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:45:35.374 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:45:35.374 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:45:35.374 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:45:35.374 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:45:35.374 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:45:35.374 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:45:35.374 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:45:35.374 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:45:35.374 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:45:36.063 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:35 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2379242312' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:45:36.063 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:36.063 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:36.064 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:36.064 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:35 smithi005 ceph-mon[39562]: 1.3 scrub starts 2024-04-20T17:45:36.064 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:36.064 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:35 smithi005 ceph-mon[39562]: 1.3 scrub ok 2024-04-20T17:45:36.064 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:35 smithi005 ceph-mon[39562]: 1.1 scrub starts 2024-04-20T17:45:36.064 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:35 smithi005 ceph-mon[39562]: 1.1 scrub ok 2024-04-20T17:45:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:35 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2379242312' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:45:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:35 smithi106 ceph-mon[44423]: 1.3 scrub starts 2024-04-20T17:45:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:35 smithi106 ceph-mon[44423]: 1.3 scrub ok 2024-04-20T17:45:36.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:35 smithi106 ceph-mon[44423]: 1.1 scrub starts 2024-04-20T17:45:36.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:35 smithi106 ceph-mon[44423]: 1.1 scrub ok 2024-04-20T17:45:36.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:35 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2379242312' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:45:36.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:36.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:36.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:36.307 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:35 smithi171 ceph-mon[44189]: 1.3 scrub starts 2024-04-20T17:45:36.307 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:36.307 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:35 smithi171 ceph-mon[44189]: 1.3 scrub ok 2024-04-20T17:45:36.307 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:35 smithi171 ceph-mon[44189]: 1.1 scrub starts 2024-04-20T17:45:36.307 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:35 smithi171 ceph-mon[44189]: 1.1 scrub ok 2024-04-20T17:45:36.449 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:45:36.798 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:45:36.870 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:36.870 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:36.870 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:36.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:36.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:36.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:36.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:36.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:36.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:36 smithi005 ceph-mon[39562]: 1.7 scrub starts 2024-04-20T17:45:36.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:36 smithi005 ceph-mon[39562]: 1.7 scrub ok 2024-04-20T17:45:36.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:36 smithi005 ceph-mon[39562]: 8.2 scrub starts 2024-04-20T17:45:36.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:36 smithi005 ceph-mon[39562]: pgmap v2199: 27 pgs: 3 active+recovering, 24 active+clean; 7.5 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 8.3 KiB/s rd, 223 KiB/s wr, 592 op/s; 22301/246258 objects misplaced (9.056%); 7.6 MiB/s, 118 objects/s recovering 2024-04-20T17:45:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:37.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:37.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:37.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:37.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:36 smithi106 ceph-mon[44423]: 1.7 scrub starts 2024-04-20T17:45:37.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:36 smithi106 ceph-mon[44423]: 1.7 scrub ok 2024-04-20T17:45:37.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:36 smithi106 ceph-mon[44423]: 8.2 scrub starts 2024-04-20T17:45:37.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:36 smithi106 ceph-mon[44423]: pgmap v2199: 27 pgs: 3 active+recovering, 24 active+clean; 7.5 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 8.3 KiB/s rd, 223 KiB/s wr, 592 op/s; 22301/246258 objects misplaced (9.056%); 7.6 MiB/s, 118 objects/s recovering 2024-04-20T17:45:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:37.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:36 smithi171 ceph-mon[44189]: 1.7 scrub starts 2024-04-20T17:45:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:36 smithi171 ceph-mon[44189]: 1.7 scrub ok 2024-04-20T17:45:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:36 smithi171 ceph-mon[44189]: 8.2 scrub starts 2024-04-20T17:45:37.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:36 smithi171 ceph-mon[44189]: pgmap v2199: 27 pgs: 3 active+recovering, 24 active+clean; 7.5 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 8.3 KiB/s rd, 223 KiB/s wr, 592 op/s; 22301/246258 objects misplaced (9.056%); 7.6 MiB/s, 118 objects/s recovering 2024-04-20T17:45:38.103 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:45:38.776 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:45:39.678 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:45:39.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:39 smithi005 ceph-mon[39562]: 8.2 scrub ok 2024-04-20T17:45:39.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:39 smithi005 ceph-mon[39562]: pgmap v2200: 27 pgs: 3 active+recovering, 24 active+clean; 7.5 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 3.8 KiB/s rd, 103 KiB/s wr, 448 op/s; 22301/246258 objects misplaced (9.056%); 5.8 MiB/s, 90 objects/s recovering 2024-04-20T17:45:39.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:39 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2697368399' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:45:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:39 smithi106 ceph-mon[44423]: 8.2 scrub ok 2024-04-20T17:45:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:39 smithi106 ceph-mon[44423]: pgmap v2200: 27 pgs: 3 active+recovering, 24 active+clean; 7.5 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 3.8 KiB/s rd, 103 KiB/s wr, 448 op/s; 22301/246258 objects misplaced (9.056%); 5.8 MiB/s, 90 objects/s recovering 2024-04-20T17:45:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:39 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2697368399' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:45:40.191 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:39 smithi171 ceph-mon[44189]: 8.2 scrub ok 2024-04-20T17:45:40.191 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:39 smithi171 ceph-mon[44189]: pgmap v2200: 27 pgs: 3 active+recovering, 24 active+clean; 7.5 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 3.8 KiB/s rd, 103 KiB/s wr, 448 op/s; 22301/246258 objects misplaced (9.056%); 5.8 MiB/s, 90 objects/s recovering 2024-04-20T17:45:40.191 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:39 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2697368399' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:45:40.382 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:45:40.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:40.392 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:45:40.401 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-20T17:45:40.493 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:45:40.532 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:40.552 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:45:40.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:40.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:45:40.656 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:45:40.707 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:40 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:45:40.708 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:40 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:45:40.708 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:40 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2697368399' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:45:40.708 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:40 smithi171 ceph-mon[44189]: osdmap e825: 12 total, 12 up, 11 in 2024-04-20T17:45:40.719 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:40.961 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:45:40.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:40 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:45:40.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:40 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:45:40.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:40 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2697368399' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:45:40.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:40 smithi005 ceph-mon[39562]: osdmap e825: 12 total, 12 up, 11 in 2024-04-20T17:45:40.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:40.969 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:45:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:40 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:45:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:40 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:45:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:40 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2697368399' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:45:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:40 smithi106 ceph-mon[44423]: osdmap e825: 12 total, 12 up, 11 in 2024-04-20T17:45:41.070 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:45:41.105 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:41.160 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:45:41.457 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:45:41.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:41.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:45:41.561 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:45:41.619 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:41.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:41 smithi005 ceph-mon[39562]: 8.b scrub starts 2024-04-20T17:45:41.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:41 smithi005 ceph-mon[39562]: pgmap v2202: 27 pgs: 3 active+recovering, 24 active+clean; 7.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 616 op/s; 21928/241750 objects misplaced (9.071%); 7.0 MiB/s, 110 objects/s recovering 2024-04-20T17:45:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:41 smithi106 ceph-mon[44423]: 8.b scrub starts 2024-04-20T17:45:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:41 smithi106 ceph-mon[44423]: pgmap v2202: 27 pgs: 3 active+recovering, 24 active+clean; 7.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 616 op/s; 21928/241750 objects misplaced (9.071%); 7.0 MiB/s, 110 objects/s recovering 2024-04-20T17:45:42.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:41 smithi171 ceph-mon[44189]: 8.b scrub starts 2024-04-20T17:45:42.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:41 smithi171 ceph-mon[44189]: pgmap v2202: 27 pgs: 3 active+recovering, 24 active+clean; 7.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 616 op/s; 21928/241750 objects misplaced (9.071%); 7.0 MiB/s, 110 objects/s recovering 2024-04-20T17:45:42.201 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:45:42.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:42.208 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:45:42.309 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:45:42.361 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:42.502 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:45:42.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:42.515 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:45:42.616 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:45:42.703 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:42.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:42 smithi171 ceph-mon[44189]: 8.b scrub ok 2024-04-20T17:45:42.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:42 smithi005 ceph-mon[39562]: 8.b scrub ok 2024-04-20T17:45:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:42 smithi106 ceph-mon[44423]: 8.b scrub ok 2024-04-20T17:45:43.488 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:45:43.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:43.492 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:45:43.593 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:45:43.626 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:43.646 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:45:43.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:43.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:45:43.750 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:45:43.793 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:43 smithi171 ceph-mon[44189]: pgmap v2203: 27 pgs: 3 active+recovering, 24 active+clean; 7.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 564 op/s; 21757/240346 objects misplaced (9.052%); 6.7 MiB/s, 106 objects/s recovering 2024-04-20T17:45:43.796 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:43.799 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:45:43.822 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:45:43.850 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:45:43.879 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:45:43.903 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:45:43.932 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:45:43.960 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:45:43.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:43 smithi005 ceph-mon[39562]: pgmap v2203: 27 pgs: 3 active+recovering, 24 active+clean; 7.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 564 op/s; 21757/240346 objects misplaced (9.052%); 6.7 MiB/s, 106 objects/s recovering 2024-04-20T17:45:43.982 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:45:44.012 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:45:44.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:43 smithi106 ceph-mon[44423]: pgmap v2203: 27 pgs: 3 active+recovering, 24 active+clean; 7.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 564 op/s; 21757/240346 objects misplaced (9.052%); 6.7 MiB/s, 106 objects/s recovering 2024-04-20T17:45:44.044 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:45:44.075 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:45:44.101 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:45:44.106 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:45:44.106 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:45:44.106 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:45:44.107 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:45:44.107 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:45:44.107 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:45:44.107 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:45:44.107 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:45:44.107 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:45:44.107 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:45:44.107 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:45:44.107 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:45:44.850 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:45:44.850 INFO:teuthology.orchestra.run.smithi171.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-20T17:45:44.851 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:45:44.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:44 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3945611912' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:45:44.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:44 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:44.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:44 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:44.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:44 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:44.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:44 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:44.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:44 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:44.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:44 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:44.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:44 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:44.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:44 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:44.933 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:45:45.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:44 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3945611912' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:45:45.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:44 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:45.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:44 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:45.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:44 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:45.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:44 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:45.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:44 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:45.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:44 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:45.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:44 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:45.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:44 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:45.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:44 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3945611912' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:45:45.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:44 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:45.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:44 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:45.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:44 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:45.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:44 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:45.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:44 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:45.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:44 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:45.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:44 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:45.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:44 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:45.318 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:45:45.852 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:45.852 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:45.852 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:45.852 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:45.852 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:45 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:45:45.852 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:45 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:45:45.852 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:45 smithi005 ceph-mon[39562]: pgmap v2204: 27 pgs: 3 active+recovering, 24 active+clean; 7.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 573 op/s; 21757/238844 objects misplaced (9.109%); 6.6 MiB/s, 105 objects/s recovering 2024-04-20T17:45:45.852 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:45 smithi005 ceph-mon[39562]: osdmap e826: 12 total, 12 up, 11 in 2024-04-20T17:45:45.975 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:45:45.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:45.982 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:45.982 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:45.982 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:45.982 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:45.982 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:45 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:45:45.982 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:45 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:45:45.983 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:45 smithi171 ceph-mon[44189]: pgmap v2204: 27 pgs: 3 active+recovering, 24 active+clean; 7.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 573 op/s; 21757/238844 objects misplaced (9.109%); 6.6 MiB/s, 105 objects/s recovering 2024-04-20T17:45:45.983 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:45 smithi171 ceph-mon[44189]: osdmap e826: 12 total, 12 up, 11 in 2024-04-20T17:45:45.984 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:45:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:45 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:45:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:45 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:45:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:45 smithi106 ceph-mon[44423]: pgmap v2204: 27 pgs: 3 active+recovering, 24 active+clean; 7.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 573 op/s; 21757/238844 objects misplaced (9.109%); 6.6 MiB/s, 105 objects/s recovering 2024-04-20T17:45:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:45 smithi106 ceph-mon[44423]: osdmap e826: 12 total, 12 up, 11 in 2024-04-20T17:45:46.084 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:45:46.112 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:46.139 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:45:46.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:46.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:45:46.243 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:45:46.288 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:46.307 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:45:46.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:46.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:45:46.410 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:45:46.440 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:46.459 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:45:46.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:46.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:45:46.563 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:45:46.590 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:46.609 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:45:46.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:46.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:45:46.713 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:45:46.765 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:46.936 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:45:46.982 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:45:46.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:46.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:45:47.089 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:45:47.130 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:47.164 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:45:47.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:47.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:45:47.272 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:45:47.319 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:47.389 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:45:47.558 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:45:47.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:47.569 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:45:47.671 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:45:47.710 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:47.732 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:45:47.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:47.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:45:47.739 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:45:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:47 smithi171 ceph-mon[44189]: pgmap v2206: 27 pgs: 3 active+recovering, 24 active+clean; 7.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 731 op/s; 21182/234518 objects misplaced (9.032%); 8.7 MiB/s, 139 objects/s recovering 2024-04-20T17:45:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:47 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2582464331' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:45:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:47 smithi171 ceph-mon[44189]: from='client.27764 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:45:47.837 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:45:47.871 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:47.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:47 smithi005 ceph-mon[39562]: pgmap v2206: 27 pgs: 3 active+recovering, 24 active+clean; 7.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 731 op/s; 21182/234518 objects misplaced (9.032%); 8.7 MiB/s, 139 objects/s recovering 2024-04-20T17:45:47.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:47 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2582464331' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:45:47.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:47 smithi005 ceph-mon[39562]: from='client.27764 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:45:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:47 smithi106 ceph-mon[44423]: pgmap v2206: 27 pgs: 3 active+recovering, 24 active+clean; 7.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 731 op/s; 21182/234518 objects misplaced (9.032%); 8.7 MiB/s, 139 objects/s recovering 2024-04-20T17:45:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:47 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2582464331' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:45:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:47 smithi106 ceph-mon[44423]: from='client.27764 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:45:48.523 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:45:48.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:48.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:45:48.628 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:45:48.680 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:45:48.680 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:45:48.680 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:45:48.680 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:45:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233747 6.1 6:8163dd93:::benchmark_data_smithi171_147972_object72649:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:45:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.174286+0000", 2024-04-20T17:45:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 577.49860733200001, 2024-04-20T17:45:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.206612618, 2024-04-20T17:45:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:45:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:45:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:45:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:45:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:45:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:45:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233747 2024-04-20T17:45:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.681 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:45:48.682 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.682 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:45:48.682 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.174286+0000", 2024-04-20T17:45:48.682 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.682 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.682 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.682 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:45:48.682 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.174286+0000", 2024-04-20T17:45:48.682 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.682 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.682 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.682 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:45:48.682 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.174287+0000", 2024-04-20T17:45:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.33e-06 2024-04-20T17:45:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:45:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.174291+0000", 2024-04-20T17:45:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8480000000000003e-06 2024-04-20T17:45:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:45:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.174300+0000", 2024-04-20T17:45:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.217e-06 2024-04-20T17:45:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:45:48.683 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.174306+0000", 2024-04-20T17:45:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.682e-06 2024-04-20T17:45:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:45:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.174348+0000", 2024-04-20T17:45:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2347e-05 2024-04-20T17:45:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:45:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.174461+0000", 2024-04-20T17:45:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000112728 2024-04-20T17:45:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:45:48.684 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.174571+0000", 2024-04-20T17:45:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010971699999999999 2024-04-20T17:45:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:45:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.178595+0000", 2024-04-20T17:45:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0040239050000000004 2024-04-20T17:45:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:45:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.380796+0000", 2024-04-20T17:45:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20220144600000001 2024-04-20T17:45:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.685 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.686 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:45:48.686 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.380851+0000", 2024-04-20T17:45:48.686 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4333000000000003e-05 2024-04-20T17:45:48.686 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.686 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.686 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:45:48.686 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.380899+0000", 2024-04-20T17:45:48.686 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8065000000000001e-05 2024-04-20T17:45:48.686 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.686 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:45:48.686 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.686 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.686 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233760 6.1 6:87254b5c:::benchmark_data_smithi171_147972_object72662:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:45:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.184502+0000", 2024-04-20T17:45:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 577.48839162299998, 2024-04-20T17:45:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.210398054, 2024-04-20T17:45:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:45:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:45:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:45:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:45:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:45:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:45:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233760 2024-04-20T17:45:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:45:48.687 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:45:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.184502+0000", 2024-04-20T17:45:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:45:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.184502+0000", 2024-04-20T17:45:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:45:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.184504+0000", 2024-04-20T17:45:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2869999999999999e-06 2024-04-20T17:45:48.688 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:45:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.184511+0000", 2024-04-20T17:45:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.55e-06 2024-04-20T17:45:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:45:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.184522+0000", 2024-04-20T17:45:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1722000000000001e-05 2024-04-20T17:45:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:45:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.184529+0000", 2024-04-20T17:45:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8889999999999996e-06 2024-04-20T17:45:48.689 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.690 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.690 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:45:48.690 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.184586+0000", 2024-04-20T17:45:48.690 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6449000000000001e-05 2024-04-20T17:45:48.690 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.690 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.690 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:45:48.690 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.184719+0000", 2024-04-20T17:45:48.690 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000133149 2024-04-20T17:45:48.690 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.691 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.691 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:45:48.691 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.184849+0000", 2024-04-20T17:45:48.691 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000129927 2024-04-20T17:45:48.691 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.691 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.691 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:45:48.691 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.188857+0000", 2024-04-20T17:45:48.691 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.004008541 2024-04-20T17:45:48.691 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.691 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:45:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.394830+0000", 2024-04-20T17:45:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20597264900000001 2024-04-20T17:45:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:45:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.394867+0000", 2024-04-20T17:45:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6736999999999997e-05 2024-04-20T17:45:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:45:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.394900+0000", 2024-04-20T17:45:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3154000000000001e-05 2024-04-20T17:45:48.692 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:45:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233773 6.1 6:8680cb34:::benchmark_data_smithi171_147972_object72675:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:45:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.195965+0000", 2024-04-20T17:45:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 577.47692789400003, 2024-04-20T17:45:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.21500868300000001, 2024-04-20T17:45:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:45:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:45:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:45:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:45:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:45:48.693 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:45:48.694 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233773 2024-04-20T17:45:48.694 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.694 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:45:48.694 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.694 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:45:48.694 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.195965+0000", 2024-04-20T17:45:48.694 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.694 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.694 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.694 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:45:48.694 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.195965+0000", 2024-04-20T17:45:48.694 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.694 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.695 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.695 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:45:48.695 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.195968+0000", 2024-04-20T17:45:48.695 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9359999999999999e-06 2024-04-20T17:45:48.695 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.695 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.695 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:45:48.695 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.195976+0000", 2024-04-20T17:45:48.696 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9459999999999998e-06 2024-04-20T17:45:48.696 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.696 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.696 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:45:48.696 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.195991+0000", 2024-04-20T17:45:48.696 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4728e-05 2024-04-20T17:45:48.697 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.697 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.697 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:45:48.697 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.195999+0000", 2024-04-20T17:45:48.697 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0150000000000003e-06 2024-04-20T17:45:48.697 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.698 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.698 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:45:48.698 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.196043+0000", 2024-04-20T17:45:48.698 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3757000000000001e-05 2024-04-20T17:45:48.698 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.698 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.698 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:45:48.698 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.196159+0000", 2024-04-20T17:45:48.699 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000115686 2024-04-20T17:45:48.699 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.699 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.699 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:45:48.699 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.196299+0000", 2024-04-20T17:45:48.699 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014027000000000001 2024-04-20T17:45:48.699 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.699 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.699 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:45:48.700 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.201108+0000", 2024-04-20T17:45:48.700 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0048087420000000004 2024-04-20T17:45:48.700 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.700 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.700 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:45:48.700 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.410861+0000", 2024-04-20T17:45:48.700 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.209753467 2024-04-20T17:45:48.700 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.701 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.701 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:45:48.701 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.410922+0000", 2024-04-20T17:45:48.701 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0498000000000003e-05 2024-04-20T17:45:48.701 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.701 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.701 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:45:48.701 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.410974+0000", 2024-04-20T17:45:48.701 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2638e-05 2024-04-20T17:45:48.701 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.701 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:45:48.701 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.701 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.702 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.702 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233783 6.1 6:82c033a0:::benchmark_data_smithi171_147972_object72685:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:45:48.702 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.204583+0000", 2024-04-20T17:45:48.702 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 577.46831010400001, 2024-04-20T17:45:48.702 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20653519100000001, 2024-04-20T17:45:48.702 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:45:48.702 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:45:48.702 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:45:48.702 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:45:48.702 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:45:48.702 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:45:48.702 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233783 2024-04-20T17:45:48.702 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.703 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:45:48.703 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.703 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:45:48.703 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.204583+0000", 2024-04-20T17:45:48.703 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.703 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.703 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.703 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:45:48.703 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.204583+0000", 2024-04-20T17:45:48.703 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.703 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.703 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.703 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:45:48.703 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.204586+0000", 2024-04-20T17:45:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7240000000000001e-06 2024-04-20T17:45:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:45:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.204592+0000", 2024-04-20T17:45:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5570000000000003e-06 2024-04-20T17:45:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:45:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.204603+0000", 2024-04-20T17:45:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1154e-05 2024-04-20T17:45:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.704 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:45:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.204609+0000", 2024-04-20T17:45:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7190000000000004e-06 2024-04-20T17:45:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:45:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.204652+0000", 2024-04-20T17:45:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2138000000000002e-05 2024-04-20T17:45:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:45:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.204752+0000", 2024-04-20T17:45:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000100317 2024-04-20T17:45:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.705 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.706 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:45:48.706 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.204823+0000", 2024-04-20T17:45:48.706 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1043000000000004e-05 2024-04-20T17:45:48.706 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.706 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.706 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:45:48.706 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.209034+0000", 2024-04-20T17:45:48.706 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0042115750000000004 2024-04-20T17:45:48.706 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.706 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.706 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:45:48.706 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.411023+0000", 2024-04-20T17:45:48.706 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20198877700000001 2024-04-20T17:45:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:45:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.411074+0000", 2024-04-20T17:45:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0321999999999997e-05 2024-04-20T17:45:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:45:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.411118+0000", 2024-04-20T17:45:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4864999999999998e-05 2024-04-20T17:45:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:45:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.707 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233795 6.1 6:824ee5e7:::benchmark_data_smithi171_147972_object72697:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:45:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.218816+0000", 2024-04-20T17:45:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 577.45407762100001, 2024-04-20T17:45:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19916731200000001, 2024-04-20T17:45:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:45:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:45:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:45:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:45:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:45:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:45:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233795 2024-04-20T17:45:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:45:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:45:48.708 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.218816+0000", 2024-04-20T17:45:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:45:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.218816+0000", 2024-04-20T17:45:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:45:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.218821+0000", 2024-04-20T17:45:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0200000000000002e-06 2024-04-20T17:45:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.709 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.710 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:45:48.710 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.218836+0000", 2024-04-20T17:45:48.710 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5142e-05 2024-04-20T17:45:48.710 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.710 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.710 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:45:48.710 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.218870+0000", 2024-04-20T17:45:48.710 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4483000000000002e-05 2024-04-20T17:45:48.710 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.710 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.710 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:45:48.710 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.218894+0000", 2024-04-20T17:45:48.710 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.336e-05 2024-04-20T17:45:48.710 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.711 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.711 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:45:48.711 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.218983+0000", 2024-04-20T17:45:48.711 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9497e-05 2024-04-20T17:45:48.711 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.711 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.711 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:45:48.711 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.219086+0000", 2024-04-20T17:45:48.711 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010312800000000001 2024-04-20T17:45:48.711 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.711 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.711 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:45:48.711 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.219191+0000", 2024-04-20T17:45:48.711 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000105009 2024-04-20T17:45:48.711 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.712 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.712 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:45:48.712 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.222869+0000", 2024-04-20T17:45:48.712 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0036778990000000001 2024-04-20T17:45:48.712 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.712 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.712 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:45:48.712 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.417920+0000", 2024-04-20T17:45:48.712 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.195051113 2024-04-20T17:45:48.712 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.712 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.713 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:45:48.713 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.417953+0000", 2024-04-20T17:45:48.713 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2157999999999999e-05 2024-04-20T17:45:48.713 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.713 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.713 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:45:48.713 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.417983+0000", 2024-04-20T17:45:48.713 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0502999999999999e-05 2024-04-20T17:45:48.713 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.713 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:45:48.713 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.713 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.713 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.713 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233803 6.1 6:8893042e:::benchmark_data_smithi171_147972_object72705:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:45:48.713 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.224771+0000", 2024-04-20T17:45:48.714 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 577.44812250999996, 2024-04-20T17:45:48.714 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19336389500000001, 2024-04-20T17:45:48.714 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:45:48.714 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:45:48.714 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:45:48.714 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:45:48.714 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:45:48.714 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:45:48.714 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233803 2024-04-20T17:45:48.714 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.715 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:45:48.715 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.715 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:45:48.715 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.224771+0000", 2024-04-20T17:45:48.715 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.715 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.715 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.715 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:45:48.715 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.224771+0000", 2024-04-20T17:45:48.715 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.715 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.715 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.715 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:45:48.715 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.224773+0000", 2024-04-20T17:45:48.715 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7999999999999999e-06 2024-04-20T17:45:48.715 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.716 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.716 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:45:48.716 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.224778+0000", 2024-04-20T17:45:48.716 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8279999999999999e-06 2024-04-20T17:45:48.716 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.716 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.716 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:45:48.716 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.224789+0000", 2024-04-20T17:45:48.716 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0720000000000001e-05 2024-04-20T17:45:48.716 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.716 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.716 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:45:48.716 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.224796+0000", 2024-04-20T17:45:48.716 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0809999999999996e-06 2024-04-20T17:45:48.717 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.717 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.717 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:45:48.717 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.224839+0000", 2024-04-20T17:45:48.717 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3154000000000001e-05 2024-04-20T17:45:48.717 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.717 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.717 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:45:48.717 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.224929+0000", 2024-04-20T17:45:48.717 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9203999999999995e-05 2024-04-20T17:45:48.717 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.717 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.717 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:45:48.717 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.225034+0000", 2024-04-20T17:45:48.717 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000105017 2024-04-20T17:45:48.717 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.718 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.718 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:45:48.718 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.228470+0000", 2024-04-20T17:45:48.718 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0034366959999999999 2024-04-20T17:45:48.718 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.718 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.718 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:45:48.718 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.418044+0000", 2024-04-20T17:45:48.718 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.18957336799999999 2024-04-20T17:45:48.718 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.718 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.718 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:45:48.719 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.418092+0000", 2024-04-20T17:45:48.719 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8359999999999998e-05 2024-04-20T17:45:48.719 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.719 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.719 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:45:48.719 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.418135+0000", 2024-04-20T17:45:48.719 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2667000000000003e-05 2024-04-20T17:45:48.719 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.719 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:45:48.720 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.720 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.720 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.720 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233813 6.1 6:87bd3cce:::benchmark_data_smithi171_147972_object72715:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:45:48.720 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.232901+0000", 2024-04-20T17:45:48.720 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 577.43999271400003, 2024-04-20T17:45:48.720 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.18540720599999999, 2024-04-20T17:45:48.720 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:45:48.720 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:45:48.720 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:45:48.720 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:45:48.720 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:45:48.720 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:45:48.720 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233813 2024-04-20T17:45:48.721 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.721 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:45:48.721 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.721 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:45:48.721 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.232901+0000", 2024-04-20T17:45:48.721 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.721 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.721 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.721 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:45:48.721 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.232901+0000", 2024-04-20T17:45:48.721 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.721 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.722 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.722 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:45:48.722 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.232903+0000", 2024-04-20T17:45:48.722 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1320000000000001e-06 2024-04-20T17:45:48.722 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.722 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.722 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:45:48.723 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.232910+0000", 2024-04-20T17:45:48.723 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9269999999999999e-06 2024-04-20T17:45:48.723 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.723 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.723 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:45:48.723 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.232923+0000", 2024-04-20T17:45:48.724 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3392e-05 2024-04-20T17:45:48.724 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.724 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.724 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:45:48.724 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.232931+0000", 2024-04-20T17:45:48.724 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0499999999999992e-06 2024-04-20T17:45:48.725 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.725 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.725 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:45:48.725 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.232985+0000", 2024-04-20T17:45:48.725 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3690000000000003e-05 2024-04-20T17:45:48.725 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.726 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.726 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:45:48.726 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.233074+0000", 2024-04-20T17:45:48.726 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8876999999999996e-05 2024-04-20T17:45:48.726 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.726 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.727 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:45:48.727 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.233150+0000", 2024-04-20T17:45:48.727 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6335999999999996e-05 2024-04-20T17:45:48.727 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.727 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.727 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:45:48.727 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.236856+0000", 2024-04-20T17:45:48.727 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0037059549999999999 2024-04-20T17:45:48.727 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.727 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.727 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:45:48.728 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.418212+0000", 2024-04-20T17:45:48.728 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.18135632600000001 2024-04-20T17:45:48.728 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.728 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.728 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:45:48.728 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.418265+0000", 2024-04-20T17:45:48.728 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3086e-05 2024-04-20T17:45:48.728 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.728 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.728 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:45:48.728 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.418308+0000", 2024-04-20T17:45:48.728 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2435000000000003e-05 2024-04-20T17:45:48.728 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.728 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:45:48.728 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.729 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.729 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.729 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233855 6.1 6:8faa32b1:::benchmark_data_smithi171_147972_object72757:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:45:48.729 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.278676+0000", 2024-04-20T17:45:48.729 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 577.394217271, 2024-04-20T17:45:48.729 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.13975268499999999, 2024-04-20T17:45:48.729 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:45:48.729 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:45:48.730 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:45:48.730 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:45:48.730 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:45:48.730 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:45:48.730 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233855 2024-04-20T17:45:48.730 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.730 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:45:48.730 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.730 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:45:48.731 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.278676+0000", 2024-04-20T17:45:48.731 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.731 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.731 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.731 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:45:48.731 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.278676+0000", 2024-04-20T17:45:48.731 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.731 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.731 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.731 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:45:48.731 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.278679+0000", 2024-04-20T17:45:48.731 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0630000000000002e-06 2024-04-20T17:45:48.731 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.732 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.732 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:45:48.732 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.278688+0000", 2024-04-20T17:45:48.732 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.3780000000000001e-06 2024-04-20T17:45:48.732 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.732 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.732 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:45:48.732 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.278703+0000", 2024-04-20T17:45:48.732 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5302999999999999e-05 2024-04-20T17:45:48.732 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.732 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.732 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:45:48.732 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.278713+0000", 2024-04-20T17:45:48.732 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9040000000000004e-06 2024-04-20T17:45:48.733 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.733 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.733 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:45:48.733 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.278767+0000", 2024-04-20T17:45:48.733 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4245e-05 2024-04-20T17:45:48.733 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.733 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.733 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:45:48.733 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.278869+0000", 2024-04-20T17:45:48.733 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000102377 2024-04-20T17:45:48.733 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.733 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.733 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:45:48.733 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.278951+0000", 2024-04-20T17:45:48.733 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1237999999999997e-05 2024-04-20T17:45:48.734 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.734 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.734 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:45:48.734 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.282731+0000", 2024-04-20T17:45:48.734 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0037807610000000001 2024-04-20T17:45:48.734 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.734 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.734 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:45:48.734 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.418360+0000", 2024-04-20T17:45:48.734 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.135629006 2024-04-20T17:45:48.734 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.735 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.735 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:45:48.735 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.418406+0000", 2024-04-20T17:45:48.735 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5343999999999998e-05 2024-04-20T17:45:48.735 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.735 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.735 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:45:48.735 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.418429+0000", 2024-04-20T17:45:48.735 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3065999999999999e-05 2024-04-20T17:45:48.735 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.735 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:45:48.736 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.736 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.736 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.736 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26882.0:233864 6.1 6:866d1e67:::benchmark_data_smithi171_147972_object72766:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-20T17:45:48.736 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:36:11.286641+0000", 2024-04-20T17:45:48.736 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 577.38625209099996, 2024-04-20T17:45:48.736 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.13675062399999999, 2024-04-20T17:45:48.736 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:45:48.736 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:45:48.736 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:45:48.736 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:45:48.736 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26882", 2024-04-20T17:45:48.736 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/2325807671", 2024-04-20T17:45:48.736 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 233864 2024-04-20T17:45:48.736 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.737 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:45:48.737 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.737 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:45:48.737 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.286641+0000", 2024-04-20T17:45:48.737 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.737 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.737 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.737 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:45:48.737 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.286641+0000", 2024-04-20T17:45:48.737 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.738 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.738 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.738 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:45:48.738 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.286643+0000", 2024-04-20T17:45:48.738 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7969999999999999e-06 2024-04-20T17:45:48.738 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.738 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.738 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:45:48.738 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.286649+0000", 2024-04-20T17:45:48.738 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1090000000000003e-06 2024-04-20T17:45:48.738 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.738 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.739 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:45:48.739 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.286660+0000", 2024-04-20T17:45:48.739 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0536999999999999e-05 2024-04-20T17:45:48.739 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.739 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.739 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:45:48.739 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.286666+0000", 2024-04-20T17:45:48.739 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5130000000000002e-06 2024-04-20T17:45:48.739 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.739 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.739 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:45:48.739 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.286710+0000", 2024-04-20T17:45:48.739 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3336e-05 2024-04-20T17:45:48.739 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.739 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.739 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:45:48.740 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.286795+0000", 2024-04-20T17:45:48.740 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5648000000000004e-05 2024-04-20T17:45:48.740 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.740 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.740 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 9", 2024-04-20T17:45:48.740 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.286870+0000", 2024-04-20T17:45:48.740 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5222999999999995e-05 2024-04-20T17:45:48.740 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.740 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.740 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:45:48.740 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.290258+0000", 2024-04-20T17:45:48.740 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0033879909999999999 2024-04-20T17:45:48.740 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.740 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.740 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:45:48.740 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.423320+0000", 2024-04-20T17:45:48.741 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.133061771 2024-04-20T17:45:48.741 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.741 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.741 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:45:48.741 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.423352+0000", 2024-04-20T17:45:48.741 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2239000000000001e-05 2024-04-20T17:45:48.741 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.741 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.741 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:45:48.741 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:36:11.423392+0000", 2024-04-20T17:45:48.741 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9459999999999998e-05 2024-04-20T17:45:48.741 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.741 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:45:48.741 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.741 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.741 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.741 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:82534 7.7 7:e6749964:::benchmark_data_smithi171_161871_object82533:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e708)", 2024-04-20T17:45:48.742 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:40.968973+0000", 2024-04-20T17:45:48.742 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 427.703920837, 2024-04-20T17:45:48.742 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20917861900000001, 2024-04-20T17:45:48.742 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:45:48.742 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:45:48.742 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:45:48.742 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:45:48.742 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:45:48.742 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:45:48.742 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 82534 2024-04-20T17:45:48.742 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.742 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:45:48.742 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.742 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:45:48.743 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.968973+0000", 2024-04-20T17:45:48.743 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.743 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.743 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.743 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:45:48.743 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.968973+0000", 2024-04-20T17:45:48.743 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.743 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.743 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.743 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:45:48.743 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.968975+0000", 2024-04-20T17:45:48.743 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0499999999999999e-06 2024-04-20T17:45:48.743 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.743 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.743 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:45:48.743 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.968998+0000", 2024-04-20T17:45:48.743 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3323999999999999e-05 2024-04-20T17:45:48.744 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.744 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.744 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:45:48.744 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.969009+0000", 2024-04-20T17:45:48.744 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0949e-05 2024-04-20T17:45:48.744 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.744 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.744 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:45:48.744 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.969016+0000", 2024-04-20T17:45:48.744 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2819999999999999e-06 2024-04-20T17:45:48.744 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.744 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.744 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:45:48.744 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.969288+0000", 2024-04-20T17:45:48.745 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00027152799999999999 2024-04-20T17:45:48.745 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.745 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.745 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:45:48.745 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.969403+0000", 2024-04-20T17:45:48.745 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000115777 2024-04-20T17:45:48.745 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.745 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.745 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:45:48.745 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.969573+0000", 2024-04-20T17:45:48.745 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000169154 2024-04-20T17:45:48.745 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.745 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.745 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:45:48.745 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.972385+0000", 2024-04-20T17:45:48.745 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0028121550000000002 2024-04-20T17:45:48.746 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.746 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.746 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:45:48.746 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.178046+0000", 2024-04-20T17:45:48.746 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.205661064 2024-04-20T17:45:48.746 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.746 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.746 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:45:48.746 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.178107+0000", 2024-04-20T17:45:48.746 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0977000000000002e-05 2024-04-20T17:45:48.746 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.746 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.746 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:45:48.746 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.178151+0000", 2024-04-20T17:45:48.747 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4359e-05 2024-04-20T17:45:48.747 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.747 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:45:48.747 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.747 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.747 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.747 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:82538 7.7 7:ecb87a89:::benchmark_data_smithi171_161871_object82537:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e708)", 2024-04-20T17:45:48.747 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:40.969208+0000", 2024-04-20T17:45:48.747 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 427.703685742, 2024-04-20T17:45:48.747 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.21037962099999999, 2024-04-20T17:45:48.747 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:45:48.747 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:45:48.747 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:45:48.747 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:45:48.747 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:45:48.748 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:45:48.748 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 82538 2024-04-20T17:45:48.748 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.748 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:45:48.748 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.748 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:45:48.748 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.969208+0000", 2024-04-20T17:45:48.748 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.748 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.748 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.748 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:45:48.748 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.969208+0000", 2024-04-20T17:45:48.748 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.748 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.748 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.748 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:45:48.749 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.969209+0000", 2024-04-20T17:45:48.749 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4869999999999999e-06 2024-04-20T17:45:48.749 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.749 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.749 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:45:48.749 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.969230+0000", 2024-04-20T17:45:48.749 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0999e-05 2024-04-20T17:45:48.749 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.749 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.749 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:45:48.749 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.969240+0000", 2024-04-20T17:45:48.749 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9080000000000007e-06 2024-04-20T17:45:48.749 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.750 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.750 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:45:48.750 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.969247+0000", 2024-04-20T17:45:48.750 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0090000000000001e-06 2024-04-20T17:45:48.750 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.750 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.750 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:45:48.750 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.974582+0000", 2024-04-20T17:45:48.750 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.005334475 2024-04-20T17:45:48.750 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.750 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.750 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:45:48.750 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.974735+0000", 2024-04-20T17:45:48.751 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015318499999999999 2024-04-20T17:45:48.751 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.751 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.751 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:45:48.751 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.974884+0000", 2024-04-20T17:45:48.751 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014973499999999999 2024-04-20T17:45:48.751 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.751 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.751 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:45:48.751 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.980059+0000", 2024-04-20T17:45:48.751 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0051748319999999999 2024-04-20T17:45:48.751 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.751 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.751 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:45:48.751 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.179529+0000", 2024-04-20T17:45:48.751 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19947004300000001 2024-04-20T17:45:48.752 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.752 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.752 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:45:48.752 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.179563+0000", 2024-04-20T17:45:48.752 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4103999999999997e-05 2024-04-20T17:45:48.752 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.752 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.752 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:45:48.752 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.179587+0000", 2024-04-20T17:45:48.752 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3844000000000001e-05 2024-04-20T17:45:48.752 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.752 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:45:48.752 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.752 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.752 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.752 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:82540 7.7 7:e0f29cdc:::benchmark_data_smithi171_161871_object82539:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e708)", 2024-04-20T17:45:48.752 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:40.975200+0000", 2024-04-20T17:45:48.753 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 427.697693018, 2024-04-20T17:45:48.753 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20839459499999999, 2024-04-20T17:45:48.753 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:45:48.753 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:45:48.753 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:45:48.753 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:45:48.753 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:45:48.753 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:45:48.753 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 82540 2024-04-20T17:45:48.753 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.753 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:45:48.753 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.753 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:45:48.753 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.975200+0000", 2024-04-20T17:45:48.753 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.754 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.754 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.754 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:45:48.754 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.975200+0000", 2024-04-20T17:45:48.754 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.754 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.754 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.754 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:45:48.754 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.975210+0000", 2024-04-20T17:45:48.754 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9879999999999999e-06 2024-04-20T17:45:48.754 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.754 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.754 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:45:48.754 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.975238+0000", 2024-04-20T17:45:48.755 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7730999999999999e-05 2024-04-20T17:45:48.755 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.755 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.755 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:45:48.755 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.975255+0000", 2024-04-20T17:45:48.755 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6702999999999999e-05 2024-04-20T17:45:48.755 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.755 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.755 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:45:48.755 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.975265+0000", 2024-04-20T17:45:48.755 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0125e-05 2024-04-20T17:45:48.755 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.755 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.755 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:45:48.756 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.977747+0000", 2024-04-20T17:45:48.756 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.002481877 2024-04-20T17:45:48.756 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.756 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.756 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:45:48.756 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.977877+0000", 2024-04-20T17:45:48.756 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013067800000000001 2024-04-20T17:45:48.756 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.756 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.756 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:45:48.757 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.978031+0000", 2024-04-20T17:45:48.757 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015306200000000001 2024-04-20T17:45:48.757 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.757 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.757 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:45:48.757 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.981097+0000", 2024-04-20T17:45:48.757 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0030660010000000001 2024-04-20T17:45:48.757 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.757 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.757 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:45:48.757 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.183525+0000", 2024-04-20T17:45:48.757 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20242866500000001 2024-04-20T17:45:48.757 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.757 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.758 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:45:48.758 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.183564+0000", 2024-04-20T17:45:48.758 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8439000000000002e-05 2024-04-20T17:45:48.758 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.758 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.758 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:45:48.758 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.183595+0000", 2024-04-20T17:45:48.758 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1325999999999998e-05 2024-04-20T17:45:48.758 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.759 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:45:48.759 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.759 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.759 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.759 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:82567 7.7 7:e22bf8b5:::benchmark_data_smithi171_161871_object82566:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e708)", 2024-04-20T17:45:48.759 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:40.988647+0000", 2024-04-20T17:45:48.759 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 427.68424638900001, 2024-04-20T17:45:48.759 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19503699599999999, 2024-04-20T17:45:48.759 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:45:48.759 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:45:48.759 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:45:48.759 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:45:48.759 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:45:48.759 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:45:48.760 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 82567 2024-04-20T17:45:48.760 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.760 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:45:48.760 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.760 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:45:48.760 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.988647+0000", 2024-04-20T17:45:48.760 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.760 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.760 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.760 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:45:48.760 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.988647+0000", 2024-04-20T17:45:48.760 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.760 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.760 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.761 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:45:48.761 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.988649+0000", 2024-04-20T17:45:48.761 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.069e-06 2024-04-20T17:45:48.761 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.761 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.761 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:45:48.761 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.988669+0000", 2024-04-20T17:45:48.761 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0106999999999999e-05 2024-04-20T17:45:48.761 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.761 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.761 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:45:48.761 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.988685+0000", 2024-04-20T17:45:48.761 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.535e-05 2024-04-20T17:45:48.761 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.762 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.762 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:45:48.762 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.988693+0000", 2024-04-20T17:45:48.762 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.7649999999999999e-06 2024-04-20T17:45:48.762 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.762 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.762 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:45:48.762 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.988759+0000", 2024-04-20T17:45:48.762 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6152e-05 2024-04-20T17:45:48.762 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.762 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.762 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:45:48.762 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.988947+0000", 2024-04-20T17:45:48.763 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000188068 2024-04-20T17:45:48.763 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.763 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.763 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:45:48.763 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.989129+0000", 2024-04-20T17:45:48.763 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00018180799999999999 2024-04-20T17:45:48.763 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.763 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.764 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:45:48.764 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:40.991728+0000", 2024-04-20T17:45:48.764 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.002598852 2024-04-20T17:45:48.764 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.764 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.764 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:45:48.764 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.183632+0000", 2024-04-20T17:45:48.764 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19190426799999999 2024-04-20T17:45:48.764 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.764 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.764 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:45:48.764 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.183662+0000", 2024-04-20T17:45:48.764 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.932e-05 2024-04-20T17:45:48.764 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.765 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.765 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:45:48.765 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.183684+0000", 2024-04-20T17:45:48.765 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2237000000000001e-05 2024-04-20T17:45:48.765 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.765 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:45:48.765 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.765 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.765 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.765 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:82598 7.7 7:e029d4d2:::benchmark_data_smithi171_161871_object82597:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e708)", 2024-04-20T17:45:48.765 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:41.007885+0000", 2024-04-20T17:45:48.765 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 427.66500854200001, 2024-04-20T17:45:48.766 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.17593040200000001, 2024-04-20T17:45:48.766 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:45:48.766 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:45:48.766 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:45:48.766 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:45:48.766 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:45:48.766 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:45:48.766 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 82598 2024-04-20T17:45:48.766 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.766 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:45:48.766 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.766 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:45:48.766 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.007885+0000", 2024-04-20T17:45:48.766 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.767 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.767 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.767 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:45:48.767 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.007885+0000", 2024-04-20T17:45:48.767 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.767 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.767 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.767 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:45:48.767 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.007888+0000", 2024-04-20T17:45:48.767 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8949999999999998e-06 2024-04-20T17:45:48.767 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.767 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.767 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:45:48.767 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.007915+0000", 2024-04-20T17:45:48.767 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7611999999999999e-05 2024-04-20T17:45:48.768 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.768 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.768 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:45:48.768 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.007933+0000", 2024-04-20T17:45:48.768 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7869000000000001e-05 2024-04-20T17:45:48.768 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.768 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.768 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:45:48.768 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.007942+0000", 2024-04-20T17:45:48.768 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0380000000000005e-06 2024-04-20T17:45:48.768 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.768 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.768 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:45:48.768 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.008258+0000", 2024-04-20T17:45:48.769 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00031539600000000001 2024-04-20T17:45:48.769 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.769 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.769 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:45:48.769 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.008407+0000", 2024-04-20T17:45:48.769 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000149553 2024-04-20T17:45:48.769 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.769 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.769 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:45:48.769 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.008551+0000", 2024-04-20T17:45:48.769 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000144306 2024-04-20T17:45:48.769 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.769 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.769 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:45:48.769 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.013921+0000", 2024-04-20T17:45:48.770 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0053692439999999996 2024-04-20T17:45:48.770 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.770 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.770 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:45:48.770 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.183723+0000", 2024-04-20T17:45:48.770 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16980257400000001 2024-04-20T17:45:48.770 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.770 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.770 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:45:48.770 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.183771+0000", 2024-04-20T17:45:48.770 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8118000000000002e-05 2024-04-20T17:45:48.770 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.770 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.771 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:45:48.771 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.183815+0000", 2024-04-20T17:45:48.771 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3797000000000001e-05 2024-04-20T17:45:48.771 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.771 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:45:48.771 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.771 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.771 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.771 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:82606 7.7 7:e9cc89d2:::benchmark_data_smithi171_161871_object82605:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e708)", 2024-04-20T17:45:48.771 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:41.011168+0000", 2024-04-20T17:45:48.771 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 427.66172540100001, 2024-04-20T17:45:48.771 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.18523245999999999, 2024-04-20T17:45:48.771 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:45:48.772 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:45:48.772 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:45:48.772 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:45:48.772 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:45:48.772 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:45:48.772 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 82606 2024-04-20T17:45:48.772 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.772 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:45:48.772 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.772 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:45:48.773 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.011168+0000", 2024-04-20T17:45:48.773 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.773 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.773 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.773 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:45:48.773 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.011168+0000", 2024-04-20T17:45:48.773 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.773 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.773 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.773 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:45:48.773 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.011170+0000", 2024-04-20T17:45:48.773 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9920000000000002e-06 2024-04-20T17:45:48.774 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.774 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.774 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:45:48.774 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.011194+0000", 2024-04-20T17:45:48.774 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3745000000000001e-05 2024-04-20T17:45:48.774 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.774 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.774 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:45:48.774 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.011216+0000", 2024-04-20T17:45:48.774 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2075000000000001e-05 2024-04-20T17:45:48.774 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.774 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.775 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:45:48.775 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.011224+0000", 2024-04-20T17:45:48.775 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5890000000000003e-06 2024-04-20T17:45:48.775 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.775 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.775 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:45:48.775 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.011261+0000", 2024-04-20T17:45:48.775 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6606000000000002e-05 2024-04-20T17:45:48.775 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.775 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.775 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:45:48.775 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.011359+0000", 2024-04-20T17:45:48.775 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7948000000000005e-05 2024-04-20T17:45:48.776 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.776 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.776 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:45:48.776 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.011441+0000", 2024-04-20T17:45:48.776 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2381999999999998e-05 2024-04-20T17:45:48.776 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.776 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.776 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:45:48.776 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.017231+0000", 2024-04-20T17:45:48.776 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0057897199999999999 2024-04-20T17:45:48.776 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.776 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.776 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:45:48.776 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.196333+0000", 2024-04-20T17:45:48.776 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.17910151999999999 2024-04-20T17:45:48.777 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.777 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.777 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:45:48.777 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.196371+0000", 2024-04-20T17:45:48.777 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8767999999999999e-05 2024-04-20T17:45:48.777 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.777 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.777 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:45:48.777 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.196400+0000", 2024-04-20T17:45:48.777 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9115000000000002e-05 2024-04-20T17:45:48.777 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.778 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:45:48.778 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.778 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.778 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.778 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:82617 7.7 7:e0a32035:::benchmark_data_smithi171_161871_object82616:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e708)", 2024-04-20T17:45:48.778 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:41.019537+0000", 2024-04-20T17:45:48.778 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 427.653356594, 2024-04-20T17:45:48.778 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.176973201, 2024-04-20T17:45:48.778 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:45:48.778 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:45:48.778 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:45:48.778 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:45:48.778 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:45:48.778 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:45:48.778 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 82617 2024-04-20T17:45:48.779 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.779 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:45:48.779 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.779 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:45:48.779 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.019537+0000", 2024-04-20T17:45:48.779 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.779 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.779 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.779 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:45:48.779 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.019537+0000", 2024-04-20T17:45:48.779 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.779 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.779 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.779 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:45:48.780 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.019539+0000", 2024-04-20T17:45:48.780 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.902e-06 2024-04-20T17:45:48.780 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.780 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.780 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:45:48.780 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.019560+0000", 2024-04-20T17:45:48.780 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1610000000000001e-05 2024-04-20T17:45:48.780 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.780 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.780 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:45:48.780 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.019572+0000", 2024-04-20T17:45:48.780 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1946e-05 2024-04-20T17:45:48.780 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.780 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.781 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:45:48.781 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.019578+0000", 2024-04-20T17:45:48.781 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8930000000000002e-06 2024-04-20T17:45:48.781 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.781 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.781 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:45:48.781 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.019618+0000", 2024-04-20T17:45:48.781 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0086999999999998e-05 2024-04-20T17:45:48.781 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.781 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.781 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:45:48.781 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.019707+0000", 2024-04-20T17:45:48.781 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8913999999999996e-05 2024-04-20T17:45:48.782 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.782 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.782 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:45:48.782 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.019796+0000", 2024-04-20T17:45:48.782 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9030999999999995e-05 2024-04-20T17:45:48.782 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.782 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.782 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:45:48.782 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.022247+0000", 2024-04-20T17:45:48.782 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0024506469999999998 2024-04-20T17:45:48.782 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.782 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.782 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:45:48.782 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.196433+0000", 2024-04-20T17:45:48.783 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.17418594200000001 2024-04-20T17:45:48.783 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.783 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.783 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:45:48.783 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.196476+0000", 2024-04-20T17:45:48.783 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3244999999999997e-05 2024-04-20T17:45:48.783 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.783 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.783 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:45:48.783 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.196510+0000", 2024-04-20T17:45:48.783 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3983999999999999e-05 2024-04-20T17:45:48.783 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.784 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:45:48.784 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.784 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.784 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.784 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:82643 7.7 7:ec0d525f:::benchmark_data_smithi171_161871_object82642:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e708)", 2024-04-20T17:45:48.784 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:41.036081+0000", 2024-04-20T17:45:48.784 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 427.63681255400002, 2024-04-20T17:45:48.784 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16553795199999999, 2024-04-20T17:45:48.784 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:45:48.784 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:45:48.784 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:45:48.784 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:45:48.784 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:45:48.784 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:45:48.785 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 82643 2024-04-20T17:45:48.785 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.785 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:45:48.785 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.785 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:45:48.785 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036081+0000", 2024-04-20T17:45:48.785 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.785 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.785 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.785 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:45:48.785 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036081+0000", 2024-04-20T17:45:48.785 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.785 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.785 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.786 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:45:48.786 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036084+0000", 2024-04-20T17:45:48.786 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2949999999999998e-06 2024-04-20T17:45:48.786 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.786 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.786 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:45:48.786 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036122+0000", 2024-04-20T17:45:48.786 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8078999999999999e-05 2024-04-20T17:45:48.786 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.786 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.786 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:45:48.786 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036141+0000", 2024-04-20T17:45:48.786 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8828e-05 2024-04-20T17:45:48.787 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.787 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.787 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:45:48.787 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036151+0000", 2024-04-20T17:45:48.787 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.842e-06 2024-04-20T17:45:48.787 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.787 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.787 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:45:48.787 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036224+0000", 2024-04-20T17:45:48.787 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3205000000000001e-05 2024-04-20T17:45:48.787 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.787 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.787 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:45:48.787 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036372+0000", 2024-04-20T17:45:48.788 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014762900000000001 2024-04-20T17:45:48.788 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.788 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.788 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:45:48.788 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036540+0000", 2024-04-20T17:45:48.788 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000167827 2024-04-20T17:45:48.788 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.788 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.788 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:45:48.788 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.041216+0000", 2024-04-20T17:45:48.788 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00467652 2024-04-20T17:45:48.788 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.788 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.788 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:45:48.789 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.201518+0000", 2024-04-20T17:45:48.789 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16030146300000001 2024-04-20T17:45:48.789 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.789 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.789 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:45:48.789 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.201574+0000", 2024-04-20T17:45:48.789 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6100999999999997e-05 2024-04-20T17:45:48.789 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.789 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.789 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:45:48.789 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.201619+0000", 2024-04-20T17:45:48.789 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5163000000000001e-05 2024-04-20T17:45:48.789 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.789 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:45:48.789 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.790 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.790 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.790 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:82644 7.7 7:e854f808:::benchmark_data_smithi171_161871_object82643:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e708)", 2024-04-20T17:45:48.790 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:41.036171+0000", 2024-04-20T17:45:48.790 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 427.63672264000002, 2024-04-20T17:45:48.790 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16557176500000001, 2024-04-20T17:45:48.790 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:45:48.790 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:45:48.790 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:45:48.790 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:45:48.790 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:45:48.790 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:45:48.790 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 82644 2024-04-20T17:45:48.790 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.791 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:45:48.791 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.791 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:45:48.791 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036171+0000", 2024-04-20T17:45:48.791 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.791 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.791 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.791 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:45:48.791 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036171+0000", 2024-04-20T17:45:48.791 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.791 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.791 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.791 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:45:48.791 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036172+0000", 2024-04-20T17:45:48.791 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7800000000000001e-07 2024-04-20T17:45:48.792 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.792 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.792 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:45:48.792 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036192+0000", 2024-04-20T17:45:48.792 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0169e-05 2024-04-20T17:45:48.792 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.792 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.792 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:45:48.792 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036198+0000", 2024-04-20T17:45:48.792 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0260000000000004e-06 2024-04-20T17:45:48.792 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.792 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.792 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:45:48.792 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.036211+0000", 2024-04-20T17:45:48.792 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3752e-05 2024-04-20T17:45:48.793 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.793 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.793 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:45:48.793 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.038883+0000", 2024-04-20T17:45:48.793 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0026713069999999999 2024-04-20T17:45:48.793 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.793 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.793 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:45:48.793 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.039025+0000", 2024-04-20T17:45:48.793 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000141936 2024-04-20T17:45:48.793 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.793 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.793 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:45:48.793 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.039167+0000", 2024-04-20T17:45:48.794 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014259900000000001 2024-04-20T17:45:48.794 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.794 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.794 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:45:48.794 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.041396+0000", 2024-04-20T17:45:48.794 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0022285310000000002 2024-04-20T17:45:48.794 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.794 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.794 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:45:48.794 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.201664+0000", 2024-04-20T17:45:48.794 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.160267979 2024-04-20T17:45:48.794 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.794 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.794 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:45:48.795 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.201713+0000", 2024-04-20T17:45:48.795 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9379000000000003e-05 2024-04-20T17:45:48.795 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.795 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.795 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:45:48.795 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.201742+0000", 2024-04-20T17:45:48.795 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9309e-05 2024-04-20T17:45:48.795 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.795 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:45:48.795 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.795 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.795 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.795 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:82651 7.7 7:e73abe37:::benchmark_data_smithi171_161871_object82650:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e708)", 2024-04-20T17:45:48.795 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:41.041427+0000", 2024-04-20T17:45:48.796 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 427.63146602500001, 2024-04-20T17:45:48.796 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.16039520500000001, 2024-04-20T17:45:48.796 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:45:48.796 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:45:48.796 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:45:48.796 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:45:48.796 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:45:48.796 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:45:48.797 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 82651 2024-04-20T17:45:48.797 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.797 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:45:48.797 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.797 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:45:48.797 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.041427+0000", 2024-04-20T17:45:48.798 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.798 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.798 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.798 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:45:48.798 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.041427+0000", 2024-04-20T17:45:48.798 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.798 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.799 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.799 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:45:48.799 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.041430+0000", 2024-04-20T17:45:48.799 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2000000000000001e-06 2024-04-20T17:45:48.799 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.799 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.800 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:45:48.800 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.041456+0000", 2024-04-20T17:45:48.800 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6341000000000001e-05 2024-04-20T17:45:48.800 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.800 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.800 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:45:48.800 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.041469+0000", 2024-04-20T17:45:48.800 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.288e-05 2024-04-20T17:45:48.800 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.800 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.800 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:45:48.800 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.041477+0000", 2024-04-20T17:45:48.800 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8169999999999999e-06 2024-04-20T17:45:48.801 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.801 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.801 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:45:48.801 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.041538+0000", 2024-04-20T17:45:48.801 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1413000000000003e-05 2024-04-20T17:45:48.801 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.801 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.801 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:45:48.801 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.041665+0000", 2024-04-20T17:45:48.801 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012729399999999999 2024-04-20T17:45:48.801 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.801 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.801 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:45:48.801 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.041783+0000", 2024-04-20T17:45:48.801 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011803699999999999 2024-04-20T17:45:48.802 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.802 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.802 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:45:48.802 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.043544+0000", 2024-04-20T17:45:48.802 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0017604140000000001 2024-04-20T17:45:48.802 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.802 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.802 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:45:48.802 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.201771+0000", 2024-04-20T17:45:48.802 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15822717 2024-04-20T17:45:48.802 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.802 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.802 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:45:48.802 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.201804+0000", 2024-04-20T17:45:48.803 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2793000000000003e-05 2024-04-20T17:45:48.803 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.803 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.803 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:45:48.803 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.201823+0000", 2024-04-20T17:45:48.803 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8845999999999999e-05 2024-04-20T17:45:48.803 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.803 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:45:48.803 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.803 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.803 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.803 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27088.0:82693 7.7 7:ecbc6ede:::benchmark_data_smithi171_161871_object82692:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e708)", 2024-04-20T17:45:48.803 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:38:41.071341+0000", 2024-04-20T17:45:48.803 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 427.60155250399998, 2024-04-20T17:45:48.803 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.130543833, 2024-04-20T17:45:48.804 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:45:48.804 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:45:48.804 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:45:48.804 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:45:48.804 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27088", 2024-04-20T17:45:48.804 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:45:48.804 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 82693 2024-04-20T17:45:48.804 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.804 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:45:48.804 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.804 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:45:48.805 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.071341+0000", 2024-04-20T17:45:48.805 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.805 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.805 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.805 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:45:48.805 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.071341+0000", 2024-04-20T17:45:48.806 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:45:48.806 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.806 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.806 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:45:48.806 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.071344+0000", 2024-04-20T17:45:48.806 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8439999999999998e-06 2024-04-20T17:45:48.806 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.807 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.807 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:45:48.807 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.071379+0000", 2024-04-20T17:45:48.807 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4795000000000001e-05 2024-04-20T17:45:48.807 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.807 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.808 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:45:48.808 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.071394+0000", 2024-04-20T17:45:48.808 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5726000000000002e-05 2024-04-20T17:45:48.808 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.808 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.808 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:45:48.808 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.071403+0000", 2024-04-20T17:45:48.809 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.67e-06 2024-04-20T17:45:48.809 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.809 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.809 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:45:48.809 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.071451+0000", 2024-04-20T17:45:48.809 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7914999999999998e-05 2024-04-20T17:45:48.809 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.809 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.809 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:45:48.809 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.071561+0000", 2024-04-20T17:45:48.809 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000110608 2024-04-20T17:45:48.810 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.810 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.810 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3", 2024-04-20T17:45:48.810 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.071658+0000", 2024-04-20T17:45:48.810 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6527999999999995e-05 2024-04-20T17:45:48.810 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.810 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.810 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:45:48.810 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.073830+0000", 2024-04-20T17:45:48.810 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0021715770000000001 2024-04-20T17:45:48.810 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.810 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.811 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:45:48.811 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.201845+0000", 2024-04-20T17:45:48.811 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.12801509 2024-04-20T17:45:48.811 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.811 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.811 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:45:48.811 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.201868+0000", 2024-04-20T17:45:48.812 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3156e-05 2024-04-20T17:45:48.812 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:45:48.812 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:45:48.812 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:45:48.812 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:38:41.201885+0000", 2024-04-20T17:45:48.812 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6923999999999999e-05 2024-04-20T17:45:48.812 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.813 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:45:48.813 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.813 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:45:48.813 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:45:48.813 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:45:48.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:48.816 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:48.816 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:45:48.816 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:45:48.917 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:45:48.966 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:48 smithi106 ceph-mon[44423]: 8.5 scrub starts 2024-04-20T17:45:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:48 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:45:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:48 smithi106 ceph-mon[44423]: from='client.27764 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:45:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:48 smithi106 ceph-mon[44423]: osdmap e827: 12 total, 12 up, 11 in 2024-04-20T17:45:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:48 smithi171 ceph-mon[44189]: 8.5 scrub starts 2024-04-20T17:45:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:48 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:45:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:48 smithi171 ceph-mon[44189]: from='client.27764 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:45:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:48 smithi171 ceph-mon[44189]: osdmap e827: 12 total, 12 up, 11 in 2024-04-20T17:45:49.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:48 smithi005 ceph-mon[39562]: 8.5 scrub starts 2024-04-20T17:45:49.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:48 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:45:49.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:48 smithi005 ceph-mon[39562]: from='client.27764 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:45:49.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:48 smithi005 ceph-mon[39562]: osdmap e827: 12 total, 12 up, 11 in 2024-04-20T17:45:49.419 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:45:49.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:49.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:45:49.524 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:45:49.558 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:49.853 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:45:49.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:49.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:45:49.957 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:45:49.992 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:49 smithi106 ceph-mon[44423]: pgmap v2208: 27 pgs: 3 active+recovering, 24 active+clean; 7.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 451 op/s; 21182/234518 objects misplaced (9.032%); 5.8 MiB/s, 93 objects/s recovering 2024-04-20T17:45:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:49 smithi171 ceph-mon[44189]: pgmap v2208: 27 pgs: 3 active+recovering, 24 active+clean; 7.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 451 op/s; 21182/234518 objects misplaced (9.032%); 5.8 MiB/s, 93 objects/s recovering 2024-04-20T17:45:50.104 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:45:50.109 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:49 smithi005 ceph-mon[39562]: pgmap v2208: 27 pgs: 3 active+recovering, 24 active+clean; 7.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 451 op/s; 21182/234518 objects misplaced (9.032%); 5.8 MiB/s, 93 objects/s recovering 2024-04-20T17:45:50.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:50.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:45:50.211 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:45:50.238 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:50.256 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:45:50.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:50.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:45:50.360 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:45:50.413 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:50.522 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:45:50.543 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:45:50.571 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:45:50.601 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:45:50.634 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:45:50.665 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:45:50.685 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:45:50.714 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:45:50.738 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:45:50.770 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:45:50.806 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:45:50.844 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:45:50.845 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:45:50.846 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:45:50.846 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:45:50.846 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:45:50.846 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:45:50.846 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:45:50.846 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:45:50.846 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:45:50.846 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:45:50.846 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:45:50.846 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:45:50.846 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:45:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:50 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/635667367' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:45:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:50 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/635667367' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:45:51.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:50 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/635667367' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:45:51.308 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:45:51.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:51.311 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:45:51.412 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:45:51.448 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:51.476 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 8, 9, 10, 11, 1, 6] out_osds: [7] dead_osds: [] live_osds: [0, 4, 5, 8, 9, 10, 11, 2, 6, 3, 1, 7] 2024-04-20T17:45:51.476 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:45:51.476 INFO:tasks.thrashosds.thrasher:inject_pause on osd.9 2024-04-20T17:45:51.476 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-20T17:45:51.476 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-20T17:45:51.476 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 config set filestore_inject_stall 3 2024-04-20T17:45:51.528 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:45:51.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:51.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:45:51.634 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:45:51.662 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:51.681 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:45:51.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:51.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:45:51.785 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:45:51.812 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:51.831 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:45:51.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:51.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:45:51.895 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:45:51.925 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:51.925 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:51.925 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:51.925 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:51.925 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:51 smithi005 ceph-mon[39562]: pgmap v2209: 27 pgs: 3 active+recovering, 24 active+clean; 7.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 641 op/s; 20791/230080 objects misplaced (9.036%); 7.5 MiB/s, 120 objects/s recovering 2024-04-20T17:45:51.925 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:51.925 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:51.925 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:51.926 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:51.926 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:51.926 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:51.926 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:51.926 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:51.935 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:45:51.971 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:51 smithi106 ceph-mon[44423]: pgmap v2209: 27 pgs: 3 active+recovering, 24 active+clean; 7.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 641 op/s; 20791/230080 objects misplaced (9.036%); 7.5 MiB/s, 120 objects/s recovering 2024-04-20T17:45:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:52.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:52.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:52.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:52.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:52.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:51 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:52.014 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:45:52.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:52.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:45:52.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:52.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:52.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:52.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:52.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:51 smithi171 ceph-mon[44189]: pgmap v2209: 27 pgs: 3 active+recovering, 24 active+clean; 7.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 641 op/s; 20791/230080 objects misplaced (9.036%); 7.5 MiB/s, 120 objects/s recovering 2024-04-20T17:45:52.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:52.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:52.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:52.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:52.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:52.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:52.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:52.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:52.119 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:45:52.147 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:52.166 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:45:52.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:52.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:45:52.255 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:45:52.268 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:45:52.309 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:52.560 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:45:52.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:52.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:45:52.664 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:45:52.690 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:52.709 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:45:52.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:52.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:45:52.814 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:45:52.855 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:52.970 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-20T17:45:53.077 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:45:53.109 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:45:53.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:53.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:45:53.213 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:45:53.244 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:53.563 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:45:53.698 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:45:53.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:53.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:45:53.808 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:45:53.834 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:53.853 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:45:53.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:53.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:45:53.881 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:53 smithi171 ceph-mon[44189]: pgmap v2210: 27 pgs: 3 active+recovering, 24 active+clean; 7.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 644 op/s; 20598/228522 objects misplaced (9.014%); 9.1 MiB/s, 144 objects/s recovering 2024-04-20T17:45:53.920 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:53 smithi005 ceph-mon[39562]: pgmap v2210: 27 pgs: 3 active+recovering, 24 active+clean; 7.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 644 op/s; 20598/228522 objects misplaced (9.014%); 9.1 MiB/s, 144 objects/s recovering 2024-04-20T17:45:53.957 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:45:53.995 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:53 smithi106 ceph-mon[44423]: pgmap v2210: 27 pgs: 3 active+recovering, 24 active+clean; 7.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 644 op/s; 20598/228522 objects misplaced (9.014%); 9.1 MiB/s, 144 objects/s recovering 2024-04-20T17:45:54.750 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:45:54.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:54.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:45:54.855 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:45:54.911 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:55.083 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:45:55.789 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:45:56.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:55 smithi171 ceph-mon[44189]: pgmap v2211: 27 pgs: 3 active+recovering, 24 active+clean; 6.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 599 op/s; 20598/226978 objects misplaced (9.075%); 7.3 MiB/s, 117 objects/s recovering 2024-04-20T17:45:56.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:55 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2637284657' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:45:56.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:55 smithi171 ceph-mon[44189]: from='client.27776 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:45:56.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:55 smithi005 ceph-mon[39562]: pgmap v2211: 27 pgs: 3 active+recovering, 24 active+clean; 6.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 599 op/s; 20598/226978 objects misplaced (9.075%); 7.3 MiB/s, 117 objects/s recovering 2024-04-20T17:45:56.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:55 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2637284657' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:45:56.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:55 smithi005 ceph-mon[39562]: from='client.27776 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:45:56.262 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:45:56.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:55 smithi106 ceph-mon[44423]: pgmap v2211: 27 pgs: 3 active+recovering, 24 active+clean; 6.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 599 op/s; 20598/226978 objects misplaced (9.075%); 7.3 MiB/s, 117 objects/s recovering 2024-04-20T17:45:56.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:55 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2637284657' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:45:56.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:55 smithi106 ceph-mon[44423]: from='client.27776 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:45:56.278 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:45:56.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:56.282 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:45:56.383 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:45:56.416 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:56.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:56 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:45:56.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:56 smithi171 ceph-mon[44189]: from='client.27776 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:45:56.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:56 smithi171 ceph-mon[44189]: osdmap e828: 12 total, 12 up, 11 in 2024-04-20T17:45:57.108 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:56 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:45:57.109 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:56 smithi005 ceph-mon[39562]: from='client.27776 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:45:57.109 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:56 smithi005 ceph-mon[39562]: osdmap e828: 12 total, 12 up, 11 in 2024-04-20T17:45:57.111 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:45:57.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:57.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:45:57.216 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:45:57.241 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:56 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:45:57.241 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:56 smithi106 ceph-mon[44423]: from='client.27776 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:45:57.241 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:56 smithi106 ceph-mon[44423]: osdmap e828: 12 total, 12 up, 11 in 2024-04-20T17:45:57.243 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:57.264 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:45:57.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:57.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:45:57.367 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:45:57.394 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:57.412 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:45:57.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:57.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:45:57.516 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:45:57.551 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:57.761 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:45:57.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:57.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:45:57.868 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:45:57.899 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:57 smithi005 ceph-mon[39562]: pgmap v2213: 27 pgs: 3 active+recovering, 24 active+clean; 6.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 680 op/s; 20032/222426 objects misplaced (9.006%); 8.1 MiB/s, 129 objects/s recovering 2024-04-20T17:45:57.901 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:57.948 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:45:57.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:57.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:45:58.057 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:45:58.081 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:57 smithi106 ceph-mon[44423]: pgmap v2213: 27 pgs: 3 active+recovering, 24 active+clean; 6.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 680 op/s; 20032/222426 objects misplaced (9.006%); 8.1 MiB/s, 129 objects/s recovering 2024-04-20T17:45:58.083 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:58.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:57 smithi171 ceph-mon[44189]: pgmap v2213: 27 pgs: 3 active+recovering, 24 active+clean; 6.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 680 op/s; 20032/222426 objects misplaced (9.006%); 8.1 MiB/s, 129 objects/s recovering 2024-04-20T17:45:58.103 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:45:58.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:58.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:45:58.206 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:45:58.233 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:58.253 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:45:58.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:58.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:45:58.356 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:45:58.429 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:58.624 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:45:58.651 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:45:58.684 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:45:58.699 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:45:58.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:45:58.703 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:45:58.706 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:45:58.726 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:45:58.752 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:45:58.777 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:45:58.803 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:45:58.804 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:45:58.835 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:45:58.854 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:45:58.862 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:45:58.887 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:45:58.924 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:45:58.926 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:45:58.926 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:45:58.926 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:45:58.927 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:45:58.927 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:45:58.927 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:45:58.927 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:45:58.927 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:45:58.927 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:45:58.927 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:45:58.927 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:45:58.927 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:45:59.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:58 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2417745247' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:45:59.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:58 smithi171 ceph-mon[44189]: pgmap v2214: 27 pgs: 3 active+recovering, 24 active+clean; 6.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 604 op/s; 20032/222426 objects misplaced (9.006%); 7.2 MiB/s, 114 objects/s recovering 2024-04-20T17:45:59.147 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:45:59.182 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:58 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2417745247' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:45:59.182 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:58 smithi005 ceph-mon[39562]: pgmap v2214: 27 pgs: 3 active+recovering, 24 active+clean; 6.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 604 op/s; 20032/222426 objects misplaced (9.006%); 7.2 MiB/s, 114 objects/s recovering 2024-04-20T17:45:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:58 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2417745247' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:45:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:58 smithi106 ceph-mon[44423]: pgmap v2214: 27 pgs: 3 active+recovering, 24 active+clean; 6.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 604 op/s; 20032/222426 objects misplaced (9.006%); 7.2 MiB/s, 114 objects/s recovering 2024-04-20T17:45:59.845 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:45:59.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:59 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:45:59.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:59 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:45:59.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:59.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:59.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:59.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:59.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:59.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:59.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:59.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:59 smithi005 ceph-mon[39562]: osdmap e829: 12 total, 12 up, 11 in 2024-04-20T17:45:59.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:59.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:59.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:45:59.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:45:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:00.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:59 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:46:00.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:59 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:46:00.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:00.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:00.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:00.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:00.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:00.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:00.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:00.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:59 smithi171 ceph-mon[44189]: osdmap e829: 12 total, 12 up, 11 in 2024-04-20T17:46:00.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:00.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:00.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:00.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:45:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:00.231 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:46:00.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:00.235 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:46:00.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:59 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:46:00.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:59 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:46:00.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:00.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:00.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:00.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:00.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:00.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:00.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:00.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:59 smithi106 ceph-mon[44423]: osdmap e829: 12 total, 12 up, 11 in 2024-04-20T17:46:00.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:00.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:00.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:00.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:45:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:00.336 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:46:00.384 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:00.816 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:00.816 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:00 smithi171 ceph-mon[44189]: pgmap v2216: 27 pgs: 3 active+recovering, 24 active+clean; 6.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 670 op/s; 19631/217788 objects misplaced (9.014%); 7.6 MiB/s, 120 objects/s recovering 2024-04-20T17:46:01.061 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:01.062 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:00 smithi005 ceph-mon[39562]: pgmap v2216: 27 pgs: 3 active+recovering, 24 active+clean; 6.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 670 op/s; 19631/217788 objects misplaced (9.014%); 7.6 MiB/s, 120 objects/s recovering 2024-04-20T17:46:01.135 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:46:01.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:01.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:46:01.243 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:46:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:00 smithi106 ceph-mon[44423]: pgmap v2216: 27 pgs: 3 active+recovering, 24 active+clean; 6.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 670 op/s; 19631/217788 objects misplaced (9.014%); 7.6 MiB/s, 120 objects/s recovering 2024-04-20T17:46:01.274 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:01.293 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:46:01.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:01.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:46:01.397 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:46:01.424 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:01.443 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:46:01.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:01.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:46:01.548 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:46:01.585 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:01.719 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:46:01.722 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:46:01.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:01.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:46:01.824 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:46:01.887 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:01.920 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:46:01.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:01.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:46:02.031 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:46:02.077 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:02.112 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:46:02.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:02.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:46:02.134 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:46:02.165 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:46:02.217 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:46:02.244 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:02.264 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:46:02.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:02.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:46:02.368 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:46:02.442 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:02 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1727587268' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:46:02.442 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:02 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:46:02.446 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:02.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:02 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1727587268' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:46:02.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:02 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:46:02.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:02 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1727587268' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:46:02.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:02 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:46:02.513 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:46:02.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:02.524 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:46:02.626 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:46:02.701 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:02.753 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:46:02.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:02.769 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:46:02.870 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:46:02.937 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:02.954 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:46:02.954 INFO:teuthology.orchestra.run.smithi171.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-20T17:46:02.954 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:46:03.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:03 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1727587268' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:46:03.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:03 smithi171 ceph-mon[44189]: osdmap e830: 12 total, 12 up, 11 in 2024-04-20T17:46:03.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:03 smithi171 ceph-mon[44189]: 8.5 scrub starts 2024-04-20T17:46:03.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:03 smithi171 ceph-mon[44189]: pgmap v2218: 27 pgs: 3 active+recovering, 24 active+clean; 6.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 563 op/s; 19436/216192 objects misplaced (8.990%); 6.9 MiB/s, 109 objects/s recovering 2024-04-20T17:46:03.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:03 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1727587268' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:46:03.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:03 smithi005 ceph-mon[39562]: osdmap e830: 12 total, 12 up, 11 in 2024-04-20T17:46:03.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:03 smithi005 ceph-mon[39562]: 8.5 scrub starts 2024-04-20T17:46:03.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:03 smithi005 ceph-mon[39562]: pgmap v2218: 27 pgs: 3 active+recovering, 24 active+clean; 6.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 563 op/s; 19436/216192 objects misplaced (8.990%); 6.9 MiB/s, 109 objects/s recovering 2024-04-20T17:46:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:03 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1727587268' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:46:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:03 smithi106 ceph-mon[44423]: osdmap e830: 12 total, 12 up, 11 in 2024-04-20T17:46:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:03 smithi106 ceph-mon[44423]: 8.5 scrub starts 2024-04-20T17:46:03.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:03 smithi106 ceph-mon[44423]: pgmap v2218: 27 pgs: 3 active+recovering, 24 active+clean; 6.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 563 op/s; 19436/216192 objects misplaced (8.990%); 6.9 MiB/s, 109 objects/s recovering 2024-04-20T17:46:03.855 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:46:03.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:03.860 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:46:03.961 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:46:04.021 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:04.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:04 smithi005 ceph-mon[39562]: 8.a scrub starts 2024-04-20T17:46:04.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:04 smithi106 ceph-mon[44423]: 8.a scrub starts 2024-04-20T17:46:04.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:04 smithi171 ceph-mon[44189]: 8.a scrub starts 2024-04-20T17:46:05.012 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:46:05.012 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:46:05.013 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.013 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "MOSDPGPushReply(8.1 830/806 [PushReplyOp(8:8d6c5beb:::benchmark_data_smithi171_175757_object39841:head)])", 2024-04-20T17:46:05.013 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:46:04.893944+0000", 2024-04-20T17:46:05.013 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.02880485, 2024-04-20T17:46:05.013 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028828049000000001, 2024-04-20T17:46:05.013 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:05.014 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:05.014 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "reached pg", 2024-04-20T17:46:05.014 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:05.014 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.014 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:05.014 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.893944+0000", 2024-04-20T17:46:05.015 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:05.015 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.015 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.015 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:05.015 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.893944+0000", 2024-04-20T17:46:05.015 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:05.016 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.016 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.016 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:05.016 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.893945+0000", 2024-04-20T17:46:05.016 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.367e-06 2024-04-20T17:46:05.016 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.017 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.017 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:05.017 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.893955+0000", 2024-04-20T17:46:05.017 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2119999999999992e-06 2024-04-20T17:46:05.017 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.017 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.018 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:05.018 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.893967+0000", 2024-04-20T17:46:05.018 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2414e-05 2024-04-20T17:46:05.018 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.018 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.019 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:05.019 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.893978+0000", 2024-04-20T17:46:05.019 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0883999999999999e-05 2024-04-20T17:46:05.019 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.019 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.019 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:05.019 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.912734+0000", 2024-04-20T17:46:05.020 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018756239000000001 2024-04-20T17:46:05.020 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:05.020 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:05.020 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:05.020 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.020 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.021 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:184955 8.1 8:8d6c5beb:::benchmark_data_smithi171_175757_object39841:head [delete] snapc 0=[] ondisk+write+known_if_redirected e830)", 2024-04-20T17:46:05.021 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:46:04.894989+0000", 2024-04-20T17:46:05.021 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.027759611999999999, 2024-04-20T17:46:05.021 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.027825107000000002, 2024-04-20T17:46:05.021 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:05.021 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:05.021 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "waiting for degraded object", 2024-04-20T17:46:05.022 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:05.022 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:05.022 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:05.022 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 184955 2024-04-20T17:46:05.022 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.022 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:05.022 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.023 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:05.023 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.894989+0000", 2024-04-20T17:46:05.023 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:05.023 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.023 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.023 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:05.024 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.894989+0000", 2024-04-20T17:46:05.024 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:05.024 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.024 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.024 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:05.024 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.894991+0000", 2024-04-20T17:46:05.025 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.813e-06 2024-04-20T17:46:05.025 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.025 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.025 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:05.025 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.894997+0000", 2024-04-20T17:46:05.025 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5269999999999998e-06 2024-04-20T17:46:05.026 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.026 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.026 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:05.026 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.895008+0000", 2024-04-20T17:46:05.026 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1897e-05 2024-04-20T17:46:05.026 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.027 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.027 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:05.027 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.895017+0000", 2024-04-20T17:46:05.027 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4810000000000006e-06 2024-04-20T17:46:05.027 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.027 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.027 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:05.027 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.901607+0000", 2024-04-20T17:46:05.027 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0065905340000000003 2024-04-20T17:46:05.027 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.028 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.028 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for degraded object", 2024-04-20T17:46:05.028 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.901675+0000", 2024-04-20T17:46:05.028 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7330999999999996e-05 2024-04-20T17:46:05.028 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:05.028 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:05.028 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:05.028 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.028 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.028 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:184957 8.c 8:35e45615:::benchmark_data_smithi171_175757_object39843:head [delete] snapc 0=[] ondisk+write+known_if_redirected e830)", 2024-04-20T17:46:05.029 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:46:04.895087+0000", 2024-04-20T17:46:05.029 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.027661670999999999, 2024-04-20T17:46:05.029 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.027776562000000001, 2024-04-20T17:46:05.029 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:05.029 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:05.029 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:46:05.029 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:05.029 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:05.029 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:05.029 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 184957 2024-04-20T17:46:05.030 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.030 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:05.030 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.030 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:05.030 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.895087+0000", 2024-04-20T17:46:05.030 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:05.030 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.030 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.030 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:05.031 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.895087+0000", 2024-04-20T17:46:05.031 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:05.031 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.031 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.031 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:05.031 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.895089+0000", 2024-04-20T17:46:05.031 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6509999999999999e-06 2024-04-20T17:46:05.031 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.031 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.031 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:05.032 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.895095+0000", 2024-04-20T17:46:05.032 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0759999999999999e-06 2024-04-20T17:46:05.032 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.032 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.032 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:05.032 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.895108+0000", 2024-04-20T17:46:05.032 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3582e-05 2024-04-20T17:46:05.032 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.032 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.032 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:05.033 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.895117+0000", 2024-04-20T17:46:05.033 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2800000000000003e-06 2024-04-20T17:46:05.033 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.033 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.033 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:05.033 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.895168+0000", 2024-04-20T17:46:05.033 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1520000000000001e-05 2024-04-20T17:46:05.033 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.033 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.033 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:05.034 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.895318+0000", 2024-04-20T17:46:05.034 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000149841 2024-04-20T17:46:05.034 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.034 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.034 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:46:05.034 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.895449+0000", 2024-04-20T17:46:05.034 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013069500000000001 2024-04-20T17:46:05.034 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.034 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.035 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:05.035 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.900600+0000", 2024-04-20T17:46:05.035 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0051515170000000004 2024-04-20T17:46:05.035 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:05.035 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:05.035 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:05.035 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.035 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.035 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:184978 8.1 8.9119c491 (undecoded) ondisk+write+known_if_redirected e830)", 2024-04-20T17:46:05.035 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:46:04.921659+0000", 2024-04-20T17:46:05.036 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.0010894500000000001, 2024-04-20T17:46:05.036 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0012492180000000001, 2024-04-20T17:46:05.036 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:05.036 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:05.036 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "queued for pg", 2024-04-20T17:46:05.036 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:05.036 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:05.036 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:05.036 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 184978 2024-04-20T17:46:05.037 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.037 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:05.037 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.037 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:05.037 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.921659+0000", 2024-04-20T17:46:05.037 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:05.037 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.037 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.037 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:05.037 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.921659+0000", 2024-04-20T17:46:05.037 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:05.038 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.038 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.038 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:05.038 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.921663+0000", 2024-04-20T17:46:05.038 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2610000000000002e-06 2024-04-20T17:46:05.038 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.038 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.039 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:05.039 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.921671+0000", 2024-04-20T17:46:05.039 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2789999999999998e-06 2024-04-20T17:46:05.039 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.039 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.039 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:05.039 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.921685+0000", 2024-04-20T17:46:05.040 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4425e-05 2024-04-20T17:46:05.040 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:05.040 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:05.040 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:05.040 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.921696+0000", 2024-04-20T17:46:05.040 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0363999999999999e-05 2024-04-20T17:46:05.041 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:05.041 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:05.041 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:05.041 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:05.041 INFO:teuthology.orchestra.run.smithi171.stdout: ], 2024-04-20T17:46:05.041 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 4 2024-04-20T17:46:05.042 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:46:05.201 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:05 smithi005 ceph-mon[39562]: 8.5 scrub ok 2024-04-20T17:46:05.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:05 smithi005 ceph-mon[39562]: 8.a scrub ok 2024-04-20T17:46:05.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:05 smithi005 ceph-mon[39562]: pgmap v2219: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering, 23 active+clean; 6.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 484 op/s; 19436/214672 objects misplaced (9.054%); 4.7 MiB/s, 74 objects/s recovering 2024-04-20T17:46:05.322 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:46:05.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:05 smithi171 ceph-mon[44189]: 8.5 scrub ok 2024-04-20T17:46:05.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:05 smithi171 ceph-mon[44189]: 8.a scrub ok 2024-04-20T17:46:05.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:05 smithi171 ceph-mon[44189]: pgmap v2219: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering, 23 active+clean; 6.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 484 op/s; 19436/214672 objects misplaced (9.054%); 4.7 MiB/s, 74 objects/s recovering 2024-04-20T17:46:05.346 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:46:05.382 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:46:05.408 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:46:05.429 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:46:05.452 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:46:05.480 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:46:05.506 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:46:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:05 smithi106 ceph-mon[44423]: 8.5 scrub ok 2024-04-20T17:46:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:05 smithi106 ceph-mon[44423]: 8.a scrub ok 2024-04-20T17:46:05.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:05 smithi106 ceph-mon[44423]: pgmap v2219: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering, 23 active+clean; 6.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 484 op/s; 19436/214672 objects misplaced (9.054%); 4.7 MiB/s, 74 objects/s recovering 2024-04-20T17:46:05.530 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:46:05.563 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:46:05.587 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:46:05.611 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:46:05.614 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:46:05.614 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:46:05.614 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:46:05.614 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:46:05.614 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:46:05.615 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:46:05.615 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:46:05.615 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:46:05.615 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:46:05.615 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:46:05.615 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:46:05.615 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:46:05.886 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:46:06.335 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:46:06.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:06 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1028196278' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:46:06.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:06.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:06.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:06.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:06.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:06.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:06.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:06.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:06.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:06.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:06 smithi005 ceph-mon[39562]: 8.9 scrub starts 2024-04-20T17:46:06.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:06 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1028196278' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:46:06.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:06.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:06.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:06.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:06.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:06.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:06.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:06.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:06.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:06.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:06 smithi106 ceph-mon[44423]: 8.9 scrub starts 2024-04-20T17:46:06.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:06 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1028196278' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:46:06.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:06.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:06.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:06.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:06.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:06.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:06.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:06.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:06.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:06.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:06 smithi171 ceph-mon[44189]: 8.9 scrub starts 2024-04-20T17:46:07.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:07.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:07.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:07.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:07 smithi005 ceph-mon[39562]: pgmap v2220: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering, 23 active+clean; 6.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 758 op/s; 18880/210286 objects misplaced (8.978%); 9.0 MiB/s, 143 objects/s recovering 2024-04-20T17:46:07.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:07 smithi005 ceph-mon[39562]: 8.9 scrub ok 2024-04-20T17:46:07.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:07.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:07.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:07.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:07 smithi106 ceph-mon[44423]: pgmap v2220: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering, 23 active+clean; 6.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 758 op/s; 18880/210286 objects misplaced (8.978%); 9.0 MiB/s, 143 objects/s recovering 2024-04-20T17:46:07.514 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:07 smithi106 ceph-mon[44423]: 8.9 scrub ok 2024-04-20T17:46:07.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:07.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:07.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:07.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:07 smithi171 ceph-mon[44189]: pgmap v2220: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering, 23 active+clean; 6.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 758 op/s; 18880/210286 objects misplaced (8.978%); 9.0 MiB/s, 143 objects/s recovering 2024-04-20T17:46:07.585 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:07 smithi171 ceph-mon[44189]: 8.9 scrub ok 2024-04-20T17:46:07.875 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:46:07.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:07.885 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:46:07.986 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:46:08.029 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:08.136 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:46:08.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:08.139 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:46:08.240 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:46:08.294 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:46:08.314 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:08.732 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:46:09.709 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:46:09.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:09 smithi005 ceph-mon[39562]: pgmap v2221: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering, 23 active+clean; 6.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 617 op/s; 18880/210286 objects misplaced (8.978%); 7.3 MiB/s, 117 objects/s recovering 2024-04-20T17:46:09.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:09 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/43440999' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:46:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:09 smithi106 ceph-mon[44423]: pgmap v2221: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering, 23 active+clean; 6.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 617 op/s; 18880/210286 objects misplaced (8.978%); 7.3 MiB/s, 117 objects/s recovering 2024-04-20T17:46:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:09 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/43440999' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:46:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:09 smithi171 ceph-mon[44189]: pgmap v2221: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering, 23 active+clean; 6.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 617 op/s; 18880/210286 objects misplaced (8.978%); 7.3 MiB/s, 117 objects/s recovering 2024-04-20T17:46:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:09 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/43440999' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:46:10.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:10 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:46:10.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:10 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:46:10.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:10 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/43440999' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:46:10.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:10 smithi005 ceph-mon[39562]: osdmap e831: 12 total, 12 up, 11 in 2024-04-20T17:46:11.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:10 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:46:11.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:10 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:46:11.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:10 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/43440999' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:46:11.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:10 smithi106 ceph-mon[44423]: osdmap e831: 12 total, 12 up, 11 in 2024-04-20T17:46:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:10 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:46:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:10 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:46:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:10 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/43440999' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:46:11.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:10 smithi171 ceph-mon[44189]: osdmap e831: 12 total, 12 up, 11 in 2024-04-20T17:46:11.103 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:46:11.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:11.112 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:46:11.140 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-20T17:46:11.213 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:46:11.256 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:11.507 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:46:11.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:11.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:46:11.611 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:46:11.659 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:11.666 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:46:11.678 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:46:11.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:11.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:46:11.787 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:46:11.860 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:11.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:11 smithi005 ceph-mon[39562]: pgmap v2223: 27 pgs: 3 active+recovering, 24 active+clean; 6.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 617 op/s; 18477/205712 objects misplaced (8.982%); 7.1 MiB/s, 112 objects/s recovering 2024-04-20T17:46:11.972 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:11 smithi171 ceph-mon[44189]: pgmap v2223: 27 pgs: 3 active+recovering, 24 active+clean; 6.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 617 op/s; 18477/205712 objects misplaced (8.982%); 7.1 MiB/s, 112 objects/s recovering 2024-04-20T17:46:11.976 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:46:11.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:11.985 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:46:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:11 smithi106 ceph-mon[44423]: pgmap v2223: 27 pgs: 3 active+recovering, 24 active+clean; 6.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 617 op/s; 18477/205712 objects misplaced (8.982%); 7.1 MiB/s, 112 objects/s recovering 2024-04-20T17:46:12.036 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:46:12.056 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:46:12.083 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:46:12.086 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:46:12.107 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:46:12.112 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:12.132 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:46:12.133 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:46:12.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:12.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:46:12.161 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:46:12.182 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:46:12.204 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:46:12.234 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:46:12.237 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:46:12.270 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:46:12.303 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:12.308 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:46:12.348 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:46:12.353 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:46:12.353 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:46:12.353 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:46:12.353 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:46:12.353 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:46:12.353 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:46:12.353 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:46:12.353 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:46:12.353 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:46:12.354 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:46:12.354 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:46:12.354 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:46:12.596 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:46:12.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:12.603 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:46:12.704 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:46:12.731 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:12.751 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:46:12.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:12.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:46:12.784 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:46:12.854 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:46:12.882 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:12.884 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 8, 9, 10, 11, 1, 6] out_osds: [7] dead_osds: [] live_osds: [0, 4, 5, 8, 9, 10, 11, 2, 6, 3, 1, 7] 2024-04-20T17:46:12.884 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:46:12.884 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-04-20T17:46:12.885 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:46:12.901 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:46:12.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:12.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:46:13.006 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:46:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:12 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:46:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:12 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:46:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:12 smithi106 ceph-mon[44423]: osdmap e832: 12 total, 12 up, 11 in 2024-04-20T17:46:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:12 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2314129157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:46:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:13.064 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:12 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:46:13.064 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:12 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:46:13.064 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:12 smithi171 ceph-mon[44189]: osdmap e832: 12 total, 12 up, 11 in 2024-04-20T17:46:13.064 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:12 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2314129157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:46:13.064 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:13.064 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:13.064 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:13.064 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:13.065 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:13.068 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:13.098 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:12 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:46:13.099 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:12 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:46:13.099 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:12 smithi005 ceph-mon[39562]: osdmap e832: 12 total, 12 up, 11 in 2024-04-20T17:46:13.099 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:12 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2314129157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:46:13.099 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:13.099 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:13.099 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:13.099 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:13.099 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:13.136 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:46:13.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:13.142 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:46:13.186 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:46:13.242 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:46:13.292 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:13.501 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:46:13.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:13.504 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:46:13.605 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:46:13.632 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:13.650 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:46:13.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:13.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:46:13.754 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:46:13.843 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:13.911 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:46:13.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:13.913 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:46:14.010 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:46:14.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:13 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:14.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:13 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:14.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:13 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:14.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:13 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:14.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:13 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:14.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:13 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:14.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:13 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:14.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:13 smithi106 ceph-mon[44423]: pgmap v2225: 27 pgs: 3 active+recovering, 24 active+clean; 6.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 654 op/s; 18270/204204 objects misplaced (8.947%); 9.1 MiB/s, 145 objects/s recovering 2024-04-20T17:46:14.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:13 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:46:14.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:13 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:46:14.014 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:46:14.040 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:14.058 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:46:14.058 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:46:14.058 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-04-20T17:46:14.058 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-04-20T17:46:14.058 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:46:14.059 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:46:14.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:14.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:46:14.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:13 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:14.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:13 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:14.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:13 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:14.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:13 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:14.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:13 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:14.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:13 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:14.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:13 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:14.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:13 smithi171 ceph-mon[44189]: pgmap v2225: 27 pgs: 3 active+recovering, 24 active+clean; 6.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 654 op/s; 18270/204204 objects misplaced (8.947%); 9.1 MiB/s, 145 objects/s recovering 2024-04-20T17:46:14.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:13 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:46:14.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:13 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:46:14.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:13 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:14.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:13 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:14.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:13 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:14.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:13 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:14.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:13 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:14.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:13 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:14.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:13 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:14.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:13 smithi005 ceph-mon[39562]: pgmap v2225: 27 pgs: 3 active+recovering, 24 active+clean; 6.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 654 op/s; 18270/204204 objects misplaced (8.947%); 9.1 MiB/s, 145 objects/s recovering 2024-04-20T17:46:14.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:13 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:46:14.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:13 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:46:14.164 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:46:14.201 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:14.579 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:46:14.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:14.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:46:14.684 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:46:14.710 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:14.729 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:46:14.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:14.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:46:14.818 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:14 smithi005 ceph-mon[39562]: osdmap e833: 12 total, 12 up, 11 in 2024-04-20T17:46:14.832 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:46:14.934 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:14 smithi106 ceph-mon[44423]: osdmap e833: 12 total, 12 up, 11 in 2024-04-20T17:46:15.099 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:14 smithi171 ceph-mon[44189]: osdmap e833: 12 total, 12 up, 11 in 2024-04-20T17:46:15.140 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:46:15.558 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:46:15.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:15.562 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:46:15.663 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:46:15.697 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:15.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:15 smithi171 ceph-mon[44189]: pgmap v2227: 27 pgs: 3 active+recovering, 24 active+clean; 6.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 641 op/s; 18270/202582 objects misplaced (9.019%); 6.4 MiB/s, 101 objects/s recovering 2024-04-20T17:46:15.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:15 smithi005 ceph-mon[39562]: pgmap v2227: 27 pgs: 3 active+recovering, 24 active+clean; 6.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 641 op/s; 18270/202582 objects misplaced (9.019%); 6.4 MiB/s, 101 objects/s recovering 2024-04-20T17:46:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:15 smithi106 ceph-mon[44423]: pgmap v2227: 27 pgs: 3 active+recovering, 24 active+clean; 6.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 641 op/s; 18270/202582 objects misplaced (9.019%); 6.4 MiB/s, 101 objects/s recovering 2024-04-20T17:46:16.447 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:46:16.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:16.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:46:16.552 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:46:16.597 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:17.407 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:46:17.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:17.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:46:17.512 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:46:17.539 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:17.558 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:46:17.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:17.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:46:17.661 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:46:17.670 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:46:17.688 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:17.708 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:46:17.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:17.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:46:17.812 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:46:17.836 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:17 smithi106 ceph-mon[44423]: pgmap v2228: 27 pgs: 3 active+recovering, 24 active+clean; 6.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 793 op/s; 17695/197746 objects misplaced (8.948%); 11 MiB/s, 171 objects/s recovering 2024-04-20T17:46:17.836 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:17 smithi106 ceph-mon[44423]: osdmap e834: 12 total, 12 up, 11 in 2024-04-20T17:46:17.836 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:17 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:46:17.836 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:17 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:46:17.836 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:17 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1717495672' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:46:17.836 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:17 smithi106 ceph-mon[44423]: from='client.27809 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:46:17.838 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:17.857 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:46:17.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:17.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:46:17.961 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:46:17.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:17 smithi005 ceph-mon[39562]: pgmap v2228: 27 pgs: 3 active+recovering, 24 active+clean; 6.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 793 op/s; 17695/197746 objects misplaced (8.948%); 11 MiB/s, 171 objects/s recovering 2024-04-20T17:46:17.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:17 smithi005 ceph-mon[39562]: osdmap e834: 12 total, 12 up, 11 in 2024-04-20T17:46:17.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:17 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:46:17.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:17 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:46:17.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:17 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1717495672' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:46:17.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:17 smithi005 ceph-mon[39562]: from='client.27809 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:46:17.997 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:18.044 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:46:18.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:18.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:46:18.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:17 smithi171 ceph-mon[44189]: pgmap v2228: 27 pgs: 3 active+recovering, 24 active+clean; 6.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 793 op/s; 17695/197746 objects misplaced (8.948%); 11 MiB/s, 171 objects/s recovering 2024-04-20T17:46:18.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:17 smithi171 ceph-mon[44189]: osdmap e834: 12 total, 12 up, 11 in 2024-04-20T17:46:18.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:17 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:46:18.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:17 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:46:18.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:17 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1717495672' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:46:18.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:17 smithi171 ceph-mon[44189]: from='client.27809 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:46:18.149 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:46:18.363 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:18.565 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:46:18.566 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:46:18.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:18 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:46:18.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:18 smithi005 ceph-mon[39562]: from='client.27809 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:46:18.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:18 smithi005 ceph-mon[39562]: osdmap e835: 12 total, 12 up, 11 in 2024-04-20T17:46:18.957 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:46:19.007 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:18 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:46:19.007 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:18 smithi171 ceph-mon[44189]: from='client.27809 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:46:19.007 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:18 smithi171 ceph-mon[44189]: osdmap e835: 12 total, 12 up, 11 in 2024-04-20T17:46:19.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:18 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:46:19.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:18 smithi106 ceph-mon[44423]: from='client.27809 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:46:19.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:18 smithi106 ceph-mon[44423]: osdmap e835: 12 total, 12 up, 11 in 2024-04-20T17:46:19.023 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:46:19.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:19.030 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:46:19.055 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-20T17:46:19.131 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:46:19.157 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:19.177 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:46:19.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:19.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:46:19.281 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:46:19.355 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:19.398 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:46:19.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:19.405 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:46:19.506 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:46:19.508 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":2230,"stamp":"2024-04-20T17:46:17.663050+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6480087246,"num_objects":98873,"num_object_clones":0,"num_object_copies":197746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":17695,"num_objects_unfound":0,"num_objects_dirty":98873,"num_whiteouts":0,"num_read":46,"num_read_kb":40,"num_write":191417,"num_write_kb":9288173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8729,"num_bytes_recovered":573121599,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162199,"ondisk_log_size":162199,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":18837236,"kb_used_data":13234296,"kb_used_omap":392,"kb_used_meta":5602359,"kb_avail":1012359436,"statfs":{"total":1055945392128,"available":1036656062464,"internally_reserved":0,"allocated":13551919104,"data_stored":13547173068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":402024,"internal_metadata":5736816024},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,7,5,4,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-211615744,"num_objects":-3229,"num_object_clones":0,"num_object_copies":-6458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-575,"num_objects_unfound":0,"num_objects_dirty":-3229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3229,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":575,"num_bytes_recovered":37683200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.957452"},"pg_stats":[{"pgid":"1.6","version":"15'137","reported_seq":686,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-04-20T17:46:13.776568+0000","last_change":"2024-04-20T17:45:34.697653+0000","last_active":"2024-04-20T17:46:13.776568+0000","last_peered":"2024-04-20T17:46:13.776568+0000","last_clean":"2024-04-20T17:46:13.776568+0000","last_became_active":"2024-04-20T17:45:32.670941+0000","last_became_peered":"2024-04-20T17:45:32.670941+0000","last_unstale":"2024-04-20T17:46:13.776568+0000","last_undegraded":"2024-04-20T17:46:13.776568+0000","last_fullsized":"2024-04-20T17:46:13.776568+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:45:34.697489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:45:34.697489+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:46:39.034058+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"833'11790","reported_seq":11989,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-04-20T17:46:15.203595+0000","last_change":"2024-04-20T17:44:24.842530+0000","last_active":"2024-04-20T17:46:15.203595+0000","last_peered":"2024-04-20T17:46:15.203595+0000","last_clean":"2024-04-20T17:46:15.203595+0000","last_became_active":"2024-04-20T17:43:18.798932+0000","last_became_peered":"2024-04-20T17:43:18.798932+0000","last_unstale":"2024-04-20T17:46:15.203595+0000","last_undegraded":"2024-04-20T17:46:15.203595+0000","last_fullsized":"2024-04-20T17:46:15.203595+0000","mapping_epoch":777,"log_start":"785'1700","ondisk_log_start":"785'1700","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'6866","last_scrub_stamp":"2024-04-20T17:44:24.842384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:44:24.842384+0000","objects_scrubbed":6822,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1583,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399769600,"num_objects":6100,"num_object_clones":0,"num_object_copies":12200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11790,"num_write_kb":572480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.7","version":"15'137","reported_seq":691,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-04-20T17:46:16.647990+0000","last_change":"2024-04-20T17:45:36.457527+0000","last_active":"2024-04-20T17:46:16.647990+0000","last_peered":"2024-04-20T17:46:16.647990+0000","last_clean":"2024-04-20T17:46:16.647990+0000","last_became_active":"2024-04-20T17:45:29.817528+0000","last_became_peered":"2024-04-20T17:45:29.817528+0000","last_unstale":"2024-04-20T17:46:16.647990+0000","last_undegraded":"2024-04-20T17:46:16.647990+0000","last_fullsized":"2024-04-20T17:46:16.647990+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:45:36.457403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:45:36.457403+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:46:53.532649+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"833'14801","reported_seq":15173,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-04-20T17:46:16.173325+0000","last_change":"2024-04-20T17:44:59.038691+0000","last_active":"2024-04-20T17:46:16.173325+0000","last_peered":"2024-04-20T17:46:16.173325+0000","last_clean":"2024-04-20T17:46:16.173325+0000","last_became_active":"2024-04-20T17:43:18.798720+0000","last_became_peered":"2024-04-20T17:43:18.798720+0000","last_unstale":"2024-04-20T17:46:16.173325+0000","last_undegraded":"2024-04-20T17:46:16.173325+0000","last_fullsized":"2024-04-20T17:46:16.173325+0000","mapping_epoch":777,"log_start":"793'4800","ondisk_log_start":"793'4800","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"808'11809","last_scrub_stamp":"2024-04-20T17:44:59.038491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:44:59.038491+0000","objects_scrubbed":9137,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:46:24.186119+0000","scrub_duration":1214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401801216,"num_objects":6131,"num_object_clones":0,"num_object_copies":12262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12161,"num_write_kb":585344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"15'137","reported_seq":722,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-04-20T17:46:13.775199+0000","last_change":"2024-04-20T17:45:29.831148+0000","last_active":"2024-04-20T17:46:13.775199+0000","last_peered":"2024-04-20T17:46:13.775199+0000","last_clean":"2024-04-20T17:46:13.775199+0000","last_became_active":"2024-04-20T17:45:29.827577+0000","last_became_peered":"2024-04-20T17:45:29.827577+0000","last_unstale":"2024-04-20T17:46:13.775199+0000","last_undegraded":"2024-04-20T17:46:13.775199+0000","last_fullsized":"2024-04-20T17:46:13.775199+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:43:41.817800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:43:41.817800+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:46:14.405901+0000","scrub_duration":261,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":17,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":616186,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.d","version":"833'11928","reported_seq":12131,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-04-20T17:46:15.194447+0000","last_change":"2024-04-20T17:44:26.727290+0000","last_active":"2024-04-20T17:46:15.194447+0000","last_peered":"2024-04-20T17:46:15.194447+0000","last_clean":"2024-04-20T17:46:15.194447+0000","last_became_active":"2024-04-20T17:43:18.798955+0000","last_became_peered":"2024-04-20T17:43:18.798955+0000","last_unstale":"2024-04-20T17:46:15.194447+0000","last_undegraded":"2024-04-20T17:46:15.194447+0000","last_fullsized":"2024-04-20T17:46:15.194447+0000","mapping_epoch":777,"log_start":"786'1900","ondisk_log_start":"786'1900","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'7202","last_scrub_stamp":"2024-04-20T17:44:26.727159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:44:26.727159+0000","objects_scrubbed":7157,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1434,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410320920,"num_objects":6262,"num_object_clones":0,"num_object_copies":12524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6262,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11928,"num_write_kb":582017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"15'137","reported_seq":686,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-04-20T17:46:13.776775+0000","last_change":"2024-04-20T17:45:33.737437+0000","last_active":"2024-04-20T17:46:13.776775+0000","last_peered":"2024-04-20T17:46:13.776775+0000","last_clean":"2024-04-20T17:46:13.776775+0000","last_became_active":"2024-04-20T17:45:32.666523+0000","last_became_peered":"2024-04-20T17:45:32.666523+0000","last_unstale":"2024-04-20T17:46:13.776775+0000","last_undegraded":"2024-04-20T17:46:13.776775+0000","last_fullsized":"2024-04-20T17:46:13.776775+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:45:33.737326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:45:33.737326+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:46:37.734862+0000","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.c","version":"833'12120","reported_seq":12318,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-04-20T17:46:16.435101+0000","last_change":"2024-04-20T17:44:21.757085+0000","last_active":"2024-04-20T17:46:16.435101+0000","last_peered":"2024-04-20T17:46:16.435101+0000","last_clean":"2024-04-20T17:46:16.435101+0000","last_became_active":"2024-04-20T17:43:18.602140+0000","last_became_peered":"2024-04-20T17:43:18.602140+0000","last_unstale":"2024-04-20T17:46:16.435101+0000","last_undegraded":"2024-04-20T17:46:16.435101+0000","last_fullsized":"2024-04-20T17:46:16.435101+0000","mapping_epoch":777,"log_start":"786'2100","ondisk_log_start":"786'2100","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'6852","last_scrub_stamp":"2024-04-20T17:44:21.756952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:44:21.756952+0000","objects_scrubbed":6797,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1549,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400687104,"num_objects":6114,"num_object_clones":0,"num_object_copies":12228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12120,"num_write_kb":583488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.2","version":"15'137","reported_seq":685,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-04-20T17:46:13.770770+0000","last_change":"2024-04-20T17:45:33.692227+0000","last_active":"2024-04-20T17:46:13.770770+0000","last_peered":"2024-04-20T17:46:13.770770+0000","last_clean":"2024-04-20T17:46:13.770770+0000","last_became_active":"2024-04-20T17:45:32.647409+0000","last_became_peered":"2024-04-20T17:45:32.647409+0000","last_unstale":"2024-04-20T17:46:13.770770+0000","last_undegraded":"2024-04-20T17:46:13.770770+0000","last_fullsized":"2024-04-20T17:46:13.770770+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:45:33.692098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:45:33.692098+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:46:50.458506+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"833'11541","reported_seq":11819,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-04-20T17:46:13.822868+0000","last_change":"2024-04-20T17:45:41.993999+0000","last_active":"2024-04-20T17:46:13.822868+0000","last_peered":"2024-04-20T17:46:13.822868+0000","last_clean":"2024-04-20T17:46:13.822868+0000","last_became_active":"2024-04-20T17:43:18.801242+0000","last_became_peered":"2024-04-20T17:43:18.801242+0000","last_unstale":"2024-04-20T17:46:13.822868+0000","last_undegraded":"2024-04-20T17:46:13.822868+0000","last_fullsized":"2024-04-20T17:46:13.822868+0000","mapping_epoch":777,"log_start":"785'1500","ondisk_log_start":"785'1500","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"825'10367","last_scrub_stamp":"2024-04-20T17:45:41.993744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:45:41.993744+0000","objects_scrubbed":7270,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:46:47.602117+0000","scrub_duration":1432,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397869056,"num_objects":6071,"num_object_clones":0,"num_object_copies":12142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11541,"num_write_kb":563584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"15'137","reported_seq":691,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-04-20T17:46:16.647978+0000","last_change":"2024-04-20T17:45:35.351959+0000","last_active":"2024-04-20T17:46:16.647978+0000","last_peered":"2024-04-20T17:46:16.647978+0000","last_clean":"2024-04-20T17:46:16.647978+0000","last_became_active":"2024-04-20T17:45:29.815765+0000","last_became_peered":"2024-04-20T17:45:29.815765+0000","last_unstale":"2024-04-20T17:46:16.647978+0000","last_undegraded":"2024-04-20T17:46:16.647978+0000","last_fullsized":"2024-04-20T17:46:16.647978+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:45:35.351845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:45:35.351845+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:46:47.053835+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"833'12025","reported_seq":12304,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-04-20T17:46:13.817269+0000","last_change":"2024-04-20T17:46:04.194418+0000","last_active":"2024-04-20T17:46:13.817269+0000","last_peered":"2024-04-20T17:46:13.817269+0000","last_clean":"2024-04-20T17:46:13.817269+0000","last_became_active":"2024-04-20T17:43:18.798958+0000","last_became_peered":"2024-04-20T17:43:18.798958+0000","last_unstale":"2024-04-20T17:46:13.817269+0000","last_undegraded":"2024-04-20T17:46:13.817269+0000","last_fullsized":"2024-04-20T17:46:13.817269+0000","mapping_epoch":777,"log_start":"786'2000","ondisk_log_start":"786'2000","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"830'11644","last_scrub_stamp":"2024-04-20T17:46:04.193887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:46:04.193887+0000","objects_scrubbed":6734,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:47:14.177066+0000","scrub_duration":1525,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414515200,"num_objects":6325,"num_object_clones":0,"num_object_copies":12650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12025,"num_write_kb":587200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2],"acting":[8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1","version":"15'137","reported_seq":686,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-04-20T17:46:13.776341+0000","last_change":"2024-04-20T17:45:35.725288+0000","last_active":"2024-04-20T17:46:13.776341+0000","last_peered":"2024-04-20T17:46:13.776341+0000","last_clean":"2024-04-20T17:46:13.776341+0000","last_became_active":"2024-04-20T17:45:32.782706+0000","last_became_peered":"2024-04-20T17:45:32.782706+0000","last_unstale":"2024-04-20T17:46:13.776341+0000","last_undegraded":"2024-04-20T17:46:13.776341+0000","last_fullsized":"2024-04-20T17:46:13.776341+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:45:35.725170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:45:35.725170+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:47:00.727011+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.8","version":"833'11962","reported_seq":12180,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-04-20T17:46:14.517243+0000","last_change":"2024-04-20T17:45:03.333991+0000","last_active":"2024-04-20T17:46:14.517243+0000","last_peered":"2024-04-20T17:46:14.517243+0000","last_clean":"2024-04-20T17:46:14.517243+0000","last_became_active":"2024-04-20T17:43:18.610421+0000","last_became_peered":"2024-04-20T17:43:18.610421+0000","last_unstale":"2024-04-20T17:46:14.517243+0000","last_undegraded":"2024-04-20T17:46:14.517243+0000","last_fullsized":"2024-04-20T17:46:14.517243+0000","mapping_epoch":777,"log_start":"786'1900","ondisk_log_start":"786'1900","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"808'9236","last_scrub_stamp":"2024-04-20T17:45:03.333793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:45:03.333793+0000","objects_scrubbed":8842,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:46:19.287148+0000","scrub_duration":1373,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398852096,"num_objects":6086,"num_object_clones":0,"num_object_copies":12172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11962,"num_write_kb":577536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.7","version":"833'11873","reported_seq":17799,"reported_epoch":833,"state":"active+recovering","last_fresh":"2024-04-20T17:46:15.210058+0000","last_change":"2024-04-20T17:44:53.433139+0000","last_active":"2024-04-20T17:46:15.210058+0000","last_peered":"2024-04-20T17:46:15.210058+0000","last_clean":"2024-04-20T17:44:51.743570+0000","last_became_active":"2024-04-20T17:44:53.388589+0000","last_became_peered":"2024-04-20T17:44:53.388589+0000","last_unstale":"2024-04-20T17:46:15.210058+0000","last_undegraded":"2024-04-20T17:46:15.210058+0000","last_fullsized":"2024-04-20T17:46:15.210058+0000","mapping_epoch":806,"log_start":"786'1800","ondisk_log_start":"786'1800","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":1599,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399310848,"num_objects":6093,"num_object_clones":0,"num_object_copies":12186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5880,"num_objects_unfound":0,"num_objects_dirty":6093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11873,"num_write_kb":574912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2891,"num_bytes_recovered":189464576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":["11","10"],"object_location_counts":[{"shards":"1,11","objects":213},{"shards":"10,11","objects":5880}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.6","version":"833'11908","reported_seq":12105,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-04-20T17:46:15.105785+0000","last_change":"2024-04-20T17:44:21.667932+0000","last_active":"2024-04-20T17:46:15.105785+0000","last_peered":"2024-04-20T17:46:15.105785+0000","last_clean":"2024-04-20T17:46:15.105785+0000","last_became_active":"2024-04-20T17:43:18.599894+0000","last_became_peered":"2024-04-20T17:43:18.599894+0000","last_unstale":"2024-04-20T17:46:15.105785+0000","last_undegraded":"2024-04-20T17:46:15.105785+0000","last_fullsized":"2024-04-20T17:46:15.105785+0000","mapping_epoch":777,"log_start":"786'1900","ondisk_log_start":"786'1900","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'6767","last_scrub_stamp":"2024-04-20T17:44:21.667758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:44:21.667758+0000","objects_scrubbed":6730,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1396,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407109632,"num_objects":6212,"num_object_clones":0,"num_object_copies":12424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11908,"num_write_kb":579840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.5","version":"833'12105","reported_seq":12395,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-04-20T17:46:14.904804+0000","last_change":"2024-04-20T17:46:03.923919+0000","last_active":"2024-04-20T17:46:14.904804+0000","last_peered":"2024-04-20T17:46:14.904804+0000","last_clean":"2024-04-20T17:46:14.904804+0000","last_became_active":"2024-04-20T17:43:18.801263+0000","last_became_peered":"2024-04-20T17:43:18.801263+0000","last_unstale":"2024-04-20T17:46:14.904804+0000","last_undegraded":"2024-04-20T17:46:14.904804+0000","last_fullsized":"2024-04-20T17:46:14.904804+0000","mapping_epoch":777,"log_start":"786'2100","ondisk_log_start":"786'2100","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"830'11661","last_scrub_stamp":"2024-04-20T17:46:03.923727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:46:03.923727+0000","objects_scrubbed":6747,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:47:12.357612+0000","scrub_duration":1316,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412155904,"num_objects":6289,"num_object_clones":0,"num_object_copies":12578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12105,"num_write_kb":588608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.4","version":"833'11908","reported_seq":12064,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-04-20T17:46:16.482520+0000","last_change":"2024-04-20T17:44:37.730301+0000","last_active":"2024-04-20T17:46:16.482520+0000","last_peered":"2024-04-20T17:46:16.482520+0000","last_clean":"2024-04-20T17:46:16.482520+0000","last_became_active":"2024-04-20T17:43:18.610249+0000","last_became_peered":"2024-04-20T17:43:18.610249+0000","last_unstale":"2024-04-20T17:46:16.482520+0000","last_undegraded":"2024-04-20T17:46:16.482520+0000","last_fullsized":"2024-04-20T17:46:16.482520+0000","mapping_epoch":777,"log_start":"786'1900","ondisk_log_start":"786'1900","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":3099,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400162816,"num_objects":6106,"num_object_clones":0,"num_object_copies":12212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11908,"num_write_kb":576448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"15'137","reported_seq":677,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-04-20T17:46:13.780470+0000","last_change":"2024-04-20T17:45:32.669725+0000","last_active":"2024-04-20T17:46:13.780470+0000","last_peered":"2024-04-20T17:46:13.780470+0000","last_clean":"2024-04-20T17:46:13.780470+0000","last_became_active":"2024-04-20T17:45:32.669451+0000","last_became_peered":"2024-04-20T17:45:32.669451+0000","last_unstale":"2024-04-20T17:46:13.780470+0000","last_undegraded":"2024-04-20T17:46:13.780470+0000","last_fullsized":"2024-04-20T17:46:13.780470+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:43:41.817800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:43:41.817800+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:46:14.194832+0000","scrub_duration":261,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":16,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":240421,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.3","version":"833'11943","reported_seq":17888,"reported_epoch":833,"state":"active+recovering","last_fresh":"2024-04-20T17:46:15.218082+0000","last_change":"2024-04-20T17:44:53.425576+0000","last_active":"2024-04-20T17:46:15.218082+0000","last_peered":"2024-04-20T17:46:15.218082+0000","last_clean":"2024-04-20T17:44:51.724944+0000","last_became_active":"2024-04-20T17:44:53.356665+0000","last_became_peered":"2024-04-20T17:44:53.356665+0000","last_unstale":"2024-04-20T17:46:15.218082+0000","last_undegraded":"2024-04-20T17:46:15.218082+0000","last_fullsized":"2024-04-20T17:46:15.218082+0000","mapping_epoch":806,"log_start":"786'1900","ondisk_log_start":"786'1900","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":6749,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405995520,"num_objects":6195,"num_object_clones":0,"num_object_copies":12390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5959,"num_objects_unfound":0,"num_objects_dirty":6195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11943,"num_write_kb":580416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2876,"num_bytes_recovered":188481536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":["11","5"],"object_location_counts":[{"shards":"1,11","objects":236},{"shards":"5,11","objects":5959}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.2","version":"833'11893","reported_seq":12177,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-04-20T17:46:14.835331+0000","last_change":"2024-04-20T17:45:38.002227+0000","last_active":"2024-04-20T17:46:14.835331+0000","last_peered":"2024-04-20T17:46:14.835331+0000","last_clean":"2024-04-20T17:46:14.835331+0000","last_became_active":"2024-04-20T17:43:18.599198+0000","last_became_peered":"2024-04-20T17:43:18.599198+0000","last_unstale":"2024-04-20T17:46:14.835331+0000","last_undegraded":"2024-04-20T17:46:14.835331+0000","last_fullsized":"2024-04-20T17:46:14.835331+0000","mapping_epoch":777,"log_start":"786'1800","ondisk_log_start":"786'1800","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"824'10571","last_scrub_stamp":"2024-04-20T17:45:38.002031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:45:38.002031+0000","objects_scrubbed":7575,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:46:38.294419+0000","scrub_duration":1465,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408092672,"num_objects":6227,"num_object_clones":0,"num_object_copies":12454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11893,"num_write_kb":579840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.8","version":"15'137","reported_seq":685,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-04-20T17:46:13.772023+0000","last_change":"2024-04-20T17:45:33.710682+0000","last_active":"2024-04-20T17:46:13.772023+0000","last_peered":"2024-04-20T17:46:13.772023+0000","last_clean":"2024-04-20T17:46:13.772023+0000","last_became_active":"2024-04-20T17:45:32.644584+0000","last_became_peered":"2024-04-20T17:45:32.644584+0000","last_unstale":"2024-04-20T17:46:13.772023+0000","last_undegraded":"2024-04-20T17:46:13.772023+0000","last_fullsized":"2024-04-20T17:46:13.772023+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:45:33.710597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:45:33.710597+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:46:35.777802+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"833'11995","reported_seq":18035,"reported_epoch":833,"state":"active+recovering","last_fresh":"2024-04-20T17:46:16.480550+0000","last_change":"2024-04-20T17:44:53.450371+0000","last_active":"2024-04-20T17:46:16.480550+0000","last_peered":"2024-04-20T17:46:16.480550+0000","last_clean":"2024-04-20T17:44:51.758441+0000","last_became_active":"2024-04-20T17:44:53.399805+0000","last_became_peered":"2024-04-20T17:44:53.399805+0000","last_unstale":"2024-04-20T17:46:16.480550+0000","last_undegraded":"2024-04-20T17:46:16.480550+0000","last_fullsized":"2024-04-20T17:46:16.480550+0000","mapping_epoch":806,"log_start":"786'1900","ondisk_log_start":"786'1900","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398655488,"num_objects":6083,"num_object_clones":0,"num_object_copies":12166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5856,"num_objects_unfound":0,"num_objects_dirty":6083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11995,"num_write_kb":578496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2957,"num_bytes_recovered":193789952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":["9","2"],"object_location_counts":[{"shards":"1,9","objects":227},{"shards":"2,9","objects":5856}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"15'137","reported_seq":678,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-04-20T17:46:13.784133+0000","last_change":"2024-04-20T17:45:32.778482+0000","last_active":"2024-04-20T17:46:13.784133+0000","last_peered":"2024-04-20T17:46:13.784133+0000","last_clean":"2024-04-20T17:46:13.784133+0000","last_became_active":"2024-04-20T17:45:32.778099+0000","last_became_peered":"2024-04-20T17:45:32.778099+0000","last_unstale":"2024-04-20T17:46:13.784133+0000","last_undegraded":"2024-04-20T17:46:13.784133+0000","last_fullsized":"2024-04-20T17:46:13.784133+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:43:41.817800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:43:41.817800+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:46:14.194862+0000","scrub_duration":261,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":16,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":240422,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.0","version":"833'12175","reported_seq":12394,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-04-20T17:46:14.489506+0000","last_change":"2024-04-20T17:45:05.225565+0000","last_active":"2024-04-20T17:46:14.489506+0000","last_peered":"2024-04-20T17:46:14.489506+0000","last_clean":"2024-04-20T17:46:14.489506+0000","last_became_active":"2024-04-20T17:43:18.610431+0000","last_became_peered":"2024-04-20T17:43:18.610431+0000","last_unstale":"2024-04-20T17:46:14.489506+0000","last_undegraded":"2024-04-20T17:46:14.489506+0000","last_fullsized":"2024-04-20T17:46:14.489506+0000","mapping_epoch":777,"log_start":"786'2100","ondisk_log_start":"786'2100","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"810'9491","last_scrub_stamp":"2024-04-20T17:45:05.225287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:45:05.225287+0000","objects_scrubbed":8954,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1261,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409665536,"num_objects":6251,"num_object_clones":0,"num_object_copies":12502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12175,"num_write_kb":589632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"833'12025","reported_seq":12303,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-04-20T17:46:14.877366+0000","last_change":"2024-04-20T17:46:06.860120+0000","last_active":"2024-04-20T17:46:14.877366+0000","last_peered":"2024-04-20T17:46:14.877366+0000","last_clean":"2024-04-20T17:46:14.877366+0000","last_became_active":"2024-04-20T17:43:18.610315+0000","last_became_peered":"2024-04-20T17:43:18.610315+0000","last_unstale":"2024-04-20T17:46:14.877366+0000","last_undegraded":"2024-04-20T17:46:14.877366+0000","last_fullsized":"2024-04-20T17:46:14.877366+0000","mapping_epoch":777,"log_start":"786'2000","ondisk_log_start":"786'2000","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"830'11694","last_scrub_stamp":"2024-04-20T17:46:06.859946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:46:06.859946+0000","objects_scrubbed":6675,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:47:11.736411+0000","scrub_duration":1127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414646272,"num_objects":6327,"num_object_clones":0,"num_object_copies":12654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12025,"num_write_kb":587264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":715,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-04-20T17:46:13.775066+0000","last_change":"2024-04-20T17:45:27.769421+0000","last_active":"2024-04-20T17:46:13.775066+0000","last_peered":"2024-04-20T17:46:13.775066+0000","last_clean":"2024-04-20T17:46:13.775066+0000","last_became_active":"2024-04-20T17:45:27.769155+0000","last_became_peered":"2024-04-20T17:45:27.769155+0000","last_unstale":"2024-04-20T17:46:13.775066+0000","last_undegraded":"2024-04-20T17:46:13.775066+0000","last_fullsized":"2024-04-20T17:46:13.775066+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":819,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:43:41.817800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:43:41.817800+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:46:18.451310+0000","scrub_duration":261,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":16,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":288506,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":6479609880,"num_objects":98872,"num_object_clones":0,"num_object_copies":197744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":17695,"num_objects_unfound":0,"num_objects_dirty":98872,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":191352,"num_write_kb":9287105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8724,"num_bytes_recovered":571736064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13504684032,"data_stored":13504675888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160692,"ondisk_log_size":160692,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":477366,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":39,"num_write":65,"num_write_kb":1068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1385535,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1658880,"data_stored":1648224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},"log_size":1507,"ondisk_log_size":1507,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991720,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2847252,"kb_used_data":1985864,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":90897900,"statfs":{"total":95995035648,"available":93079449600,"internally_reserved":0,"allocated":2033524736,"data_stored":2033090040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":882047428},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,4,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253359,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2848016,"kb_used_data":2167108,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":90897136,"statfs":{"total":95995035648,"available":93078667264,"internally_reserved":0,"allocated":2219118592,"data_stored":2218687968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":697235904},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613108,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1806412,"kb_used_data":1175492,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":91938740,"statfs":{"total":95995035648,"available":94145269760,"internally_reserved":0,"allocated":1203703808,"data_stored":1203273192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":646052288},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809337,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1578452,"kb_used_data":1191812,"kb_used_omap":1,"kb_used_meta":386558,"kb_avail":92166700,"statfs":{"total":95995035648,"available":94378700800,"internally_reserved":0,"allocated":1220415488,"data_stored":1219984872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":395835845},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":0,"seq":3431678869528,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":535,"seq":2297807503568,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":42576,"kb_used_data":836,"kb_used_omap":170,"kb_used_meta":41557,"kb_avail":93702576,"statfs":{"total":95995035648,"available":95951437824,"internally_reserved":0,"allocated":856064,"data_stored":425448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":174628,"internal_metadata":42554844},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":659,"seq":2830383448190,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":165444,"kb_used_data":44216,"kb_used_omap":4,"kb_used_meta":121211,"kb_avail":93579708,"statfs":{"total":95995035648,"available":95825620992,"internally_reserved":0,"allocated":45277184,"data_stored":44852572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4410,"internal_metadata":124120774},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738912,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1516812,"kb_used_data":791492,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":92228340,"statfs":{"total":95995035648,"available":94441820160,"internally_reserved":0,"allocated":810487808,"data_stored":810057192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":742717893},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394202,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2541720,"kb_used_data":2140688,"kb_used_omap":202,"kb_used_meta":400821,"kb_avail":91203432,"statfs":{"total":95995035648,"available":93392314368,"internally_reserved":0,"allocated":2192064512,"data_stored":2191626272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":410441175},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653251,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1244944,"kb_used_data":793284,"kb_used_omap":4,"kb_used_meta":451643,"kb_avail":92500208,"statfs":{"total":95995035648,"available":94720212992,"internally_reserved":0,"allocated":812322816,"data_stored":811892200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":462483143},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986700,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1370640,"kb_used_data":789700,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":92374512,"statfs":{"total":95995035648,"available":94591500288,"internally_reserved":0,"allocated":808652800,"data_stored":808222184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790470,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2874968,"kb_used_data":2153804,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":90870184,"statfs":{"total":95995035648,"available":93051068416,"internally_reserved":0,"allocated":2205495296,"data_stored":2205061128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":805699584,"data_stored":805699584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":44957696,"data_stored":44957696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2189299712,"data_stored":2189295640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":808517632,"data_stored":808517632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":803405824,"data_stored":803405824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2194538496,"data_stored":2194538496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1218772992,"data_stored":1218772992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1199439872,"data_stored":1199439872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2217541632,"data_stored":2217541632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2022510592,"data_stored":2022506520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:46:19.510 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-04-20T17:46:19.517 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:46:19.534 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:19.553 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:46:19.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:19.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:46:19.657 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:46:19.666 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:19 smithi171 ceph-mon[44189]: from='client.27815 v1:172.21.15.5:0/2424384913' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:46:19.667 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:19 smithi171 ceph-mon[44189]: pgmap v2231: 27 pgs: 3 active+recovering, 24 active+clean; 6.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 538 op/s; 17695/197746 objects misplaced (8.948%); 6.0 MiB/s, 95 objects/s recovering 2024-04-20T17:46:19.667 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:19 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:46:19.667 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:19 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:46:19.735 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:19.788 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:19 smithi005 ceph-mon[39562]: from='client.27815 v1:172.21.15.5:0/2424384913' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:46:19.789 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:19 smithi005 ceph-mon[39562]: pgmap v2231: 27 pgs: 3 active+recovering, 24 active+clean; 6.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 538 op/s; 17695/197746 objects misplaced (8.948%); 6.0 MiB/s, 95 objects/s recovering 2024-04-20T17:46:19.789 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:19 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:46:19.789 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:19 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:46:19.840 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:46:19.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:19.847 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:46:19.948 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:46:19.984 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:19 smithi106 ceph-mon[44423]: from='client.27815 v1:172.21.15.5:0/2424384913' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:46:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:19 smithi106 ceph-mon[44423]: pgmap v2231: 27 pgs: 3 active+recovering, 24 active+clean; 6.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 538 op/s; 17695/197746 objects misplaced (8.948%); 6.0 MiB/s, 95 objects/s recovering 2024-04-20T17:46:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:19 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:46:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:19 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:46:20.540 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:46:20.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:20.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:46:20.646 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:46:20.707 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:20.946 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:20 smithi005 ceph-mon[39562]: osdmap e836: 12 total, 12 up, 11 in 2024-04-20T17:46:20.952 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:46:20.993 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:46:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:20 smithi106 ceph-mon[44423]: osdmap e836: 12 total, 12 up, 11 in 2024-04-20T17:46:21.025 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:20 smithi171 ceph-mon[44189]: osdmap e836: 12 total, 12 up, 11 in 2024-04-20T17:46:21.105 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:46:21.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:21.113 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:46:21.214 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:46:21.279 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:21.386 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:46:21.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:21.390 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:46:21.491 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:46:21.520 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:21.538 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:46:21.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:21.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:46:21.659 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:46:21.693 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:21.707 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:21 smithi171 ceph-mon[44189]: pgmap v2233: 27 pgs: 3 active+recovering, 24 active+clean; 5.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 811 op/s; 17288/192840 objects misplaced (8.965%); 10 MiB/s, 163 objects/s recovering 2024-04-20T17:46:21.707 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:21 smithi171 ceph-mon[44189]: osdmap e837: 12 total, 12 up, 11 in 2024-04-20T17:46:21.956 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:46:21.958 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:21 smithi005 ceph-mon[39562]: pgmap v2233: 27 pgs: 3 active+recovering, 24 active+clean; 5.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 811 op/s; 17288/192840 objects misplaced (8.965%); 10 MiB/s, 163 objects/s recovering 2024-04-20T17:46:21.958 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:21 smithi005 ceph-mon[39562]: osdmap e837: 12 total, 12 up, 11 in 2024-04-20T17:46:21.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:21.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:46:22.020 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:21 smithi106 ceph-mon[44423]: pgmap v2233: 27 pgs: 3 active+recovering, 24 active+clean; 5.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 811 op/s; 17288/192840 objects misplaced (8.965%); 10 MiB/s, 163 objects/s recovering 2024-04-20T17:46:22.020 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:21 smithi106 ceph-mon[44423]: osdmap e837: 12 total, 12 up, 11 in 2024-04-20T17:46:22.050 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:46:22.050 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:46:22.050 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:46:22.050 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:46:22.050 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.050 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:21129 8.c 8:38d359e6:::benchmark_data_smithi171_175757_object21128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e785)", 2024-04-20T17:46:22.050 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:43:33.970227+0000", 2024-04-20T17:46:22.050 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 168.07367510899999, 2024-04-20T17:46:22.050 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.204827078, 2024-04-20T17:46:22.050 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:22.050 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:22.051 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:22.051 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:22.051 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:22.051 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:22.051 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 21129 2024-04-20T17:46:22.051 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.051 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:22.051 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.051 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:22.051 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.970227+0000", 2024-04-20T17:46:22.051 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.051 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.051 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.051 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:22.052 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.970227+0000", 2024-04-20T17:46:22.052 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.052 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.052 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.052 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:22.052 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.970228+0000", 2024-04-20T17:46:22.052 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.403e-06 2024-04-20T17:46:22.052 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.052 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.052 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:22.052 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.970250+0000", 2024-04-20T17:46:22.052 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1586999999999998e-05 2024-04-20T17:46:22.052 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.052 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.052 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:22.053 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.970259+0000", 2024-04-20T17:46:22.053 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1400000000000006e-06 2024-04-20T17:46:22.053 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.053 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.053 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:22.053 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.970267+0000", 2024-04-20T17:46:22.053 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4699999999999996e-06 2024-04-20T17:46:22.053 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.053 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.053 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.053 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.970319+0000", 2024-04-20T17:46:22.053 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2228999999999997e-05 2024-04-20T17:46:22.053 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.053 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.053 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:22.053 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.970449+0000", 2024-04-20T17:46:22.054 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000130352 2024-04-20T17:46:22.054 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.054 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.054 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:46:22.054 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.970587+0000", 2024-04-20T17:46:22.054 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013777700000000001 2024-04-20T17:46:22.054 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.054 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.054 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:22.054 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.972256+0000", 2024-04-20T17:46:22.054 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0016688530000000001 2024-04-20T17:46:22.054 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.054 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.054 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:22.055 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:34.174990+0000", 2024-04-20T17:46:22.055 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20273393200000001 2024-04-20T17:46:22.055 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.055 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.055 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:22.055 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:34.175023+0000", 2024-04-20T17:46:22.055 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3129000000000001e-05 2024-04-20T17:46:22.055 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.055 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.055 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:22.055 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:34.175054+0000", 2024-04-20T17:46:22.055 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1205999999999999e-05 2024-04-20T17:46:22.055 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.055 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:22.056 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.056 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.056 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.056 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:21137 8.c 8:367eeb77:::benchmark_data_smithi171_175757_object21136:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e785)", 2024-04-20T17:46:22.056 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:43:33.972891+0000", 2024-04-20T17:46:22.056 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 168.07101078100001, 2024-04-20T17:46:22.056 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20230847900000001, 2024-04-20T17:46:22.056 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:22.056 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:22.056 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:22.056 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:22.056 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:22.056 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:22.056 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 21137 2024-04-20T17:46:22.056 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.056 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:22.057 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.057 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:22.057 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.972891+0000", 2024-04-20T17:46:22.057 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.057 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.057 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.057 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:22.057 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.972891+0000", 2024-04-20T17:46:22.057 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.057 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.057 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.057 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:22.057 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.972893+0000", 2024-04-20T17:46:22.057 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.855e-06 2024-04-20T17:46:22.057 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.058 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.058 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:22.058 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.972916+0000", 2024-04-20T17:46:22.058 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2476999999999998e-05 2024-04-20T17:46:22.058 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.058 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.058 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:22.058 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.972925+0000", 2024-04-20T17:46:22.058 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0440000000000001e-06 2024-04-20T17:46:22.058 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.058 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.058 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:22.059 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.972932+0000", 2024-04-20T17:46:22.059 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6790000000000005e-06 2024-04-20T17:46:22.059 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.059 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.059 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.059 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.972987+0000", 2024-04-20T17:46:22.059 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4440000000000001e-05 2024-04-20T17:46:22.059 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.059 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.059 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:22.059 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.973114+0000", 2024-04-20T17:46:22.059 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012759300000000001 2024-04-20T17:46:22.059 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.059 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.059 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:46:22.060 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.973261+0000", 2024-04-20T17:46:22.060 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000146191 2024-04-20T17:46:22.060 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.060 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.060 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:22.060 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.974958+0000", 2024-04-20T17:46:22.060 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001697304 2024-04-20T17:46:22.060 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.060 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.060 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:22.060 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:34.175099+0000", 2024-04-20T17:46:22.060 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20014141499999999 2024-04-20T17:46:22.060 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.060 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.060 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:22.061 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:34.175158+0000", 2024-04-20T17:46:22.061 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9049000000000003e-05 2024-04-20T17:46:22.061 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.061 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.061 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:22.061 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:34.175200+0000", 2024-04-20T17:46:22.061 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1431999999999999e-05 2024-04-20T17:46:22.061 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.061 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:22.061 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.061 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.061 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.061 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:21143 8.c 8:3470a085:::benchmark_data_smithi171_175757_object21142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e785)", 2024-04-20T17:46:22.061 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:43:33.975912+0000", 2024-04-20T17:46:22.061 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 168.06799035399999, 2024-04-20T17:46:22.062 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20128262, 2024-04-20T17:46:22.062 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:22.062 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:22.062 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:22.062 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:22.062 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:22.062 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:22.062 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 21143 2024-04-20T17:46:22.062 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.062 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:22.062 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.062 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:22.062 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.975912+0000", 2024-04-20T17:46:22.062 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.062 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.063 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.063 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:22.063 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.975912+0000", 2024-04-20T17:46:22.063 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.063 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.063 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.063 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:22.063 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.975914+0000", 2024-04-20T17:46:22.063 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.886e-06 2024-04-20T17:46:22.063 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.063 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.063 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:22.063 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.975938+0000", 2024-04-20T17:46:22.063 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4473000000000001e-05 2024-04-20T17:46:22.064 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.064 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.064 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:22.064 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.975949+0000", 2024-04-20T17:46:22.064 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0599e-05 2024-04-20T17:46:22.064 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.064 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.064 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:22.064 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.975956+0000", 2024-04-20T17:46:22.064 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6299999999999998e-06 2024-04-20T17:46:22.064 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.064 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.064 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.064 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.976013+0000", 2024-04-20T17:46:22.064 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6518000000000003e-05 2024-04-20T17:46:22.065 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.065 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.065 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:22.065 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.976163+0000", 2024-04-20T17:46:22.065 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015029900000000001 2024-04-20T17:46:22.065 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.065 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.065 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:46:22.065 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.976318+0000", 2024-04-20T17:46:22.065 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015515 2024-04-20T17:46:22.065 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.065 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.065 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:22.065 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.978753+0000", 2024-04-20T17:46:22.065 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0024346770000000001 2024-04-20T17:46:22.066 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.066 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.066 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:22.066 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:34.177090+0000", 2024-04-20T17:46:22.066 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19833748600000001 2024-04-20T17:46:22.066 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.066 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.066 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:22.066 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:34.177145+0000", 2024-04-20T17:46:22.066 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4452e-05 2024-04-20T17:46:22.066 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.066 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.066 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:22.066 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:34.177194+0000", 2024-04-20T17:46:22.067 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9450000000000003e-05 2024-04-20T17:46:22.067 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.067 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:22.067 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.067 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.067 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.067 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:21158 8.c 8:3842ead9:::benchmark_data_smithi171_175757_object21157:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e785)", 2024-04-20T17:46:22.067 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:43:33.979792+0000", 2024-04-20T17:46:22.067 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 168.06410985900001, 2024-04-20T17:46:22.067 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.198313024, 2024-04-20T17:46:22.067 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:22.067 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:22.067 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:22.067 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:22.067 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:22.068 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:22.068 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 21158 2024-04-20T17:46:22.068 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.068 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:22.068 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.068 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:22.068 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.979792+0000", 2024-04-20T17:46:22.068 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.068 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.068 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.068 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:22.068 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.979792+0000", 2024-04-20T17:46:22.068 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.068 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.068 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.069 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:22.069 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.979794+0000", 2024-04-20T17:46:22.069 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4649999999999999e-06 2024-04-20T17:46:22.069 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.069 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.069 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:22.069 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.979842+0000", 2024-04-20T17:46:22.069 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8718999999999999e-05 2024-04-20T17:46:22.069 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.069 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.069 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:22.069 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.979851+0000", 2024-04-20T17:46:22.069 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1309999999999995e-06 2024-04-20T17:46:22.069 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.070 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.070 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:22.070 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.979859+0000", 2024-04-20T17:46:22.070 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4030000000000001e-06 2024-04-20T17:46:22.070 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.070 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.070 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.070 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.979906+0000", 2024-04-20T17:46:22.070 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6919000000000003e-05 2024-04-20T17:46:22.070 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.070 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.070 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:22.070 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.980009+0000", 2024-04-20T17:46:22.070 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000103396 2024-04-20T17:46:22.071 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.071 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.071 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:46:22.071 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.980148+0000", 2024-04-20T17:46:22.071 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013908400000000001 2024-04-20T17:46:22.071 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.071 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.071 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:22.071 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.981188+0000", 2024-04-20T17:46:22.071 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001039758 2024-04-20T17:46:22.071 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.071 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.071 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:22.071 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:34.178011+0000", 2024-04-20T17:46:22.071 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.196823251 2024-04-20T17:46:22.072 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.072 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.072 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:22.072 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:34.178067+0000", 2024-04-20T17:46:22.072 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5890999999999997e-05 2024-04-20T17:46:22.072 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.072 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.072 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:22.072 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:34.178105+0000", 2024-04-20T17:46:22.072 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8006999999999997e-05 2024-04-20T17:46:22.072 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.072 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:22.072 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.072 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.072 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.073 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:21167 8.c 8:39a91475:::benchmark_data_smithi171_175757_object21166:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e785)", 2024-04-20T17:46:22.073 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:43:33.982915+0000", 2024-04-20T17:46:22.073 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 168.06098664800001, 2024-04-20T17:46:22.073 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19531495099999999, 2024-04-20T17:46:22.073 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:22.073 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:22.073 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:22.073 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:22.073 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:22.073 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:22.073 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 21167 2024-04-20T17:46:22.073 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.073 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:22.073 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.073 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:22.074 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.982915+0000", 2024-04-20T17:46:22.074 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.074 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.074 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.074 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:22.074 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.982915+0000", 2024-04-20T17:46:22.074 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.074 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.074 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.074 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:22.074 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.982917+0000", 2024-04-20T17:46:22.074 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.522e-06 2024-04-20T17:46:22.074 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.074 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.075 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:22.075 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.982964+0000", 2024-04-20T17:46:22.075 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6796999999999999e-05 2024-04-20T17:46:22.075 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.075 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.075 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:22.075 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.982974+0000", 2024-04-20T17:46:22.075 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0597e-05 2024-04-20T17:46:22.075 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.075 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.075 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:22.075 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.982982+0000", 2024-04-20T17:46:22.075 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4760000000000001e-06 2024-04-20T17:46:22.075 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.075 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.076 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.076 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.983040+0000", 2024-04-20T17:46:22.076 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8183000000000002e-05 2024-04-20T17:46:22.076 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.076 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.076 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:22.076 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.983234+0000", 2024-04-20T17:46:22.076 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00019363299999999999 2024-04-20T17:46:22.076 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.076 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.076 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:46:22.076 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.983383+0000", 2024-04-20T17:46:22.076 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000148932 2024-04-20T17:46:22.076 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.076 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.077 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:22.077 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.985243+0000", 2024-04-20T17:46:22.077 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001860937 2024-04-20T17:46:22.077 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.077 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.077 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:22.077 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:34.178159+0000", 2024-04-20T17:46:22.077 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.192915534 2024-04-20T17:46:22.077 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.077 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.077 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:22.077 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:34.178197+0000", 2024-04-20T17:46:22.077 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7849999999999998e-05 2024-04-20T17:46:22.077 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.078 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.078 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:22.078 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:34.178230+0000", 2024-04-20T17:46:22.078 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.349e-05 2024-04-20T17:46:22.078 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.078 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:22.078 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.078 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.078 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.078 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:21189 8.c 8:35a992e6:::benchmark_data_smithi171_175757_object21188:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e785)", 2024-04-20T17:46:22.078 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:43:33.988868+0000", 2024-04-20T17:46:22.078 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 168.05503427400001, 2024-04-20T17:46:22.078 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19114474100000001, 2024-04-20T17:46:22.079 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:22.079 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:22.079 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:22.079 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:22.079 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:22.079 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:22.079 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 21189 2024-04-20T17:46:22.079 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.079 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:22.079 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.079 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:22.079 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.988868+0000", 2024-04-20T17:46:22.079 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.079 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.080 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.080 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:22.080 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.988868+0000", 2024-04-20T17:46:22.080 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.080 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.080 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.080 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:22.080 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.988869+0000", 2024-04-20T17:46:22.080 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.626e-06 2024-04-20T17:46:22.080 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.080 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.080 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:22.080 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.988915+0000", 2024-04-20T17:46:22.080 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5549e-05 2024-04-20T17:46:22.080 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.081 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.081 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:22.081 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.988925+0000", 2024-04-20T17:46:22.081 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0076e-05 2024-04-20T17:46:22.081 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.081 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.081 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:22.081 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.988933+0000", 2024-04-20T17:46:22.081 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.3620000000000006e-06 2024-04-20T17:46:22.081 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.081 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.081 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.081 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.989090+0000", 2024-04-20T17:46:22.081 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015696300000000001 2024-04-20T17:46:22.081 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.082 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.082 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:22.082 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.989216+0000", 2024-04-20T17:46:22.082 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012563799999999999 2024-04-20T17:46:22.082 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.082 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.082 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 8", 2024-04-20T17:46:22.082 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.989346+0000", 2024-04-20T17:46:22.082 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012998 2024-04-20T17:46:22.082 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.082 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.082 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:22.082 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:33.990495+0000", 2024-04-20T17:46:22.082 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001148572 2024-04-20T17:46:22.083 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.083 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.083 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:22.083 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:34.179914+0000", 2024-04-20T17:46:22.083 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.18941929299999999 2024-04-20T17:46:22.083 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.083 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.083 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:22.083 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:34.179970+0000", 2024-04-20T17:46:22.083 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6421000000000001e-05 2024-04-20T17:46:22.083 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.083 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.083 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:22.083 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:34.180012+0000", 2024-04-20T17:46:22.083 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2261000000000001e-05 2024-04-20T17:46:22.084 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.084 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:22.084 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.084 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.084 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.084 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:76612 8.1 8:8088a7a0:::benchmark_data_smithi171_175757_object76611:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e793)", 2024-04-20T17:46:22.084 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:43:59.550469+0000", 2024-04-20T17:46:22.084 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 142.493433431, 2024-04-20T17:46:22.084 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.216740552, 2024-04-20T17:46:22.084 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:22.084 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:22.084 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:22.084 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:22.084 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:22.084 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:22.085 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 76612 2024-04-20T17:46:22.085 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.085 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:22.085 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.085 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:22.085 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.550469+0000", 2024-04-20T17:46:22.085 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.085 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.085 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.085 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:22.085 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.550469+0000", 2024-04-20T17:46:22.085 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.085 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.085 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.085 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:22.086 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.550469+0000", 2024-04-20T17:46:22.086 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8899999999999998e-07 2024-04-20T17:46:22.086 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.086 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.086 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:22.086 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.550491+0000", 2024-04-20T17:46:22.086 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.143e-05 2024-04-20T17:46:22.086 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.086 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.086 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:22.086 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.550498+0000", 2024-04-20T17:46:22.086 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6450000000000005e-06 2024-04-20T17:46:22.087 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.087 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.087 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:22.087 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.550505+0000", 2024-04-20T17:46:22.087 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4659999999999996e-06 2024-04-20T17:46:22.087 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.087 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.087 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.087 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.550554+0000", 2024-04-20T17:46:22.087 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8812e-05 2024-04-20T17:46:22.087 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.087 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.087 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:22.087 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.553236+0000", 2024-04-20T17:46:22.087 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0026820350000000001 2024-04-20T17:46:22.088 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.088 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.088 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:46:22.088 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.553398+0000", 2024-04-20T17:46:22.088 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00016248099999999999 2024-04-20T17:46:22.088 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.088 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.088 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:22.088 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.560024+0000", 2024-04-20T17:46:22.088 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0066262409999999997 2024-04-20T17:46:22.088 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.088 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.088 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:22.088 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.767101+0000", 2024-04-20T17:46:22.088 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20707631800000001 2024-04-20T17:46:22.089 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.089 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.089 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:22.089 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.767160+0000", 2024-04-20T17:46:22.089 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8826000000000002e-05 2024-04-20T17:46:22.089 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.089 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.089 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:22.089 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.767209+0000", 2024-04-20T17:46:22.089 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9509000000000003e-05 2024-04-20T17:46:22.089 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.089 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:22.089 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.089 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.089 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.090 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:76615 8.1 8:8cee540a:::benchmark_data_smithi171_175757_object76614:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e793)", 2024-04-20T17:46:22.090 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:43:59.550750+0000", 2024-04-20T17:46:22.090 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 142.49315165100001, 2024-04-20T17:46:22.090 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.216568396, 2024-04-20T17:46:22.091 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:22.091 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:22.091 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:22.091 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:22.091 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:22.091 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:22.091 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 76615 2024-04-20T17:46:22.091 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.091 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:22.091 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.091 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:22.091 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.550750+0000", 2024-04-20T17:46:22.091 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.092 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.092 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.092 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:22.092 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.550750+0000", 2024-04-20T17:46:22.092 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.092 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.092 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.092 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:22.092 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.550752+0000", 2024-04-20T17:46:22.092 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4500000000000001e-06 2024-04-20T17:46:22.092 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.092 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.092 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:22.092 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.550773+0000", 2024-04-20T17:46:22.092 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1460000000000001e-05 2024-04-20T17:46:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.550783+0000", 2024-04-20T17:46:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.268e-06 2024-04-20T17:46:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.550790+0000", 2024-04-20T17:46:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.622e-06 2024-04-20T17:46:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.557109+0000", 2024-04-20T17:46:22.093 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0063193299999999997 2024-04-20T17:46:22.094 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.094 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.094 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:22.094 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.557222+0000", 2024-04-20T17:46:22.094 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000112773 2024-04-20T17:46:22.094 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.094 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.094 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:46:22.094 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.557341+0000", 2024-04-20T17:46:22.094 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011915 2024-04-20T17:46:22.094 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.094 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.094 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:22.095 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.560031+0000", 2024-04-20T17:46:22.095 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0026899390000000001 2024-04-20T17:46:22.095 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.095 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.095 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:22.095 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.767250+0000", 2024-04-20T17:46:22.095 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20721911800000001 2024-04-20T17:46:22.095 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.095 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.095 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:22.095 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.767291+0000", 2024-04-20T17:46:22.095 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0617000000000001e-05 2024-04-20T17:46:22.095 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.095 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.095 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:22.096 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.767319+0000", 2024-04-20T17:46:22.096 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7668999999999999e-05 2024-04-20T17:46:22.096 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.096 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:22.096 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.096 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.096 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.096 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:76632 8.1 8:8918fa6e:::benchmark_data_smithi171_175757_object76631:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e793)", 2024-04-20T17:46:22.096 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:43:59.560494+0000", 2024-04-20T17:46:22.096 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 142.483408107, 2024-04-20T17:46:22.096 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.21032179500000001, 2024-04-20T17:46:22.096 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:22.096 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:22.096 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:22.096 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:22.097 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:22.097 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:22.097 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 76632 2024-04-20T17:46:22.097 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.097 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:22.097 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.097 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:22.097 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.560494+0000", 2024-04-20T17:46:22.097 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.097 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.097 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.097 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:22.097 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.560494+0000", 2024-04-20T17:46:22.097 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.097 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.098 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.098 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:22.098 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.560496+0000", 2024-04-20T17:46:22.098 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9319999999999999e-06 2024-04-20T17:46:22.098 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.098 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.098 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:22.098 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.560517+0000", 2024-04-20T17:46:22.098 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1591999999999999e-05 2024-04-20T17:46:22.098 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.098 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.098 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:22.098 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.560528+0000", 2024-04-20T17:46:22.099 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0336000000000001e-05 2024-04-20T17:46:22.099 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.099 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.099 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:22.099 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.560535+0000", 2024-04-20T17:46:22.099 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7330000000000003e-06 2024-04-20T17:46:22.099 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.099 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.099 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.099 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.560587+0000", 2024-04-20T17:46:22.099 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1919000000000002e-05 2024-04-20T17:46:22.099 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.099 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.099 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:22.100 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.560724+0000", 2024-04-20T17:46:22.100 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013688 2024-04-20T17:46:22.100 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.100 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.100 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:46:22.100 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.560861+0000", 2024-04-20T17:46:22.100 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000136284 2024-04-20T17:46:22.100 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.100 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.100 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:22.100 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.563095+0000", 2024-04-20T17:46:22.100 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0022348789999999999 2024-04-20T17:46:22.100 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.100 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.101 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:22.101 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.770760+0000", 2024-04-20T17:46:22.101 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.20766478799999999 2024-04-20T17:46:22.101 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.101 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.101 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:22.101 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.770792+0000", 2024-04-20T17:46:22.101 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1334000000000002e-05 2024-04-20T17:46:22.101 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.101 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.101 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:22.101 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.770816+0000", 2024-04-20T17:46:22.101 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4117999999999998e-05 2024-04-20T17:46:22.102 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.102 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:22.102 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.102 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.102 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.102 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:76650 8.1 8:84b7df75:::benchmark_data_smithi171_175757_object76649:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e793)", 2024-04-20T17:46:22.102 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:43:59.573604+0000", 2024-04-20T17:46:22.102 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 142.470297845, 2024-04-20T17:46:22.102 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19733111, 2024-04-20T17:46:22.102 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:22.102 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:22.102 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:22.102 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:22.102 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:22.103 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:22.103 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 76650 2024-04-20T17:46:22.103 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.103 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:22.103 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.103 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:22.103 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.573604+0000", 2024-04-20T17:46:22.103 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.103 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.103 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.103 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:22.103 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.573604+0000", 2024-04-20T17:46:22.103 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.103 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.103 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.104 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:22.104 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.573606+0000", 2024-04-20T17:46:22.104 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1890000000000001e-06 2024-04-20T17:46:22.104 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.104 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.104 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:22.104 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.573659+0000", 2024-04-20T17:46:22.104 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.253e-05 2024-04-20T17:46:22.104 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.104 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.104 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:22.104 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.573670+0000", 2024-04-20T17:46:22.104 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1374e-05 2024-04-20T17:46:22.104 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.105 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.105 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:22.105 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.573679+0000", 2024-04-20T17:46:22.105 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.7639999999999994e-06 2024-04-20T17:46:22.105 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.105 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.105 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.105 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.573738+0000", 2024-04-20T17:46:22.105 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8631999999999997e-05 2024-04-20T17:46:22.105 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.105 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.105 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:22.105 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.573859+0000", 2024-04-20T17:46:22.105 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012115700000000001 2024-04-20T17:46:22.105 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.106 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.106 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:46:22.106 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.574009+0000", 2024-04-20T17:46:22.106 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015032700000000001 2024-04-20T17:46:22.106 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.106 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.106 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:22.106 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.576658+0000", 2024-04-20T17:46:22.106 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0026491650000000002 2024-04-20T17:46:22.106 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.106 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.106 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:22.106 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.770852+0000", 2024-04-20T17:46:22.106 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19419330600000001 2024-04-20T17:46:22.107 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.107 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.107 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:22.107 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.770897+0000", 2024-04-20T17:46:22.107 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5766000000000002e-05 2024-04-20T17:46:22.107 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.107 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.107 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:22.107 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.770935+0000", 2024-04-20T17:46:22.107 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.79e-05 2024-04-20T17:46:22.107 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.107 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:22.107 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.107 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.108 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.108 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:76669 8.1 8:857e85c0:::benchmark_data_smithi171_175757_object76668:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e793)", 2024-04-20T17:46:22.108 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:43:59.584516+0000", 2024-04-20T17:46:22.108 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 142.45938555699999, 2024-04-20T17:46:22.108 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.19039262400000001, 2024-04-20T17:46:22.108 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:22.108 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:22.108 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:22.108 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:22.108 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:22.108 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:22.108 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 76669 2024-04-20T17:46:22.108 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.108 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:22.108 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.108 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:22.109 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.584516+0000", 2024-04-20T17:46:22.109 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.109 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.109 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.109 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:22.109 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.584516+0000", 2024-04-20T17:46:22.109 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.109 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.109 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.109 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:22.109 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.584518+0000", 2024-04-20T17:46:22.109 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2020000000000001e-06 2024-04-20T17:46:22.109 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.109 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.109 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:22.110 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.584537+0000", 2024-04-20T17:46:22.110 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9063e-05 2024-04-20T17:46:22.110 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.110 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.110 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:22.110 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.584546+0000", 2024-04-20T17:46:22.110 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.969e-06 2024-04-20T17:46:22.110 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.110 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.110 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:22.110 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.584551+0000", 2024-04-20T17:46:22.110 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5589999999999996e-06 2024-04-20T17:46:22.110 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.110 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.111 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.111 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.584595+0000", 2024-04-20T17:46:22.111 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4226e-05 2024-04-20T17:46:22.111 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.111 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.111 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:22.111 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.584682+0000", 2024-04-20T17:46:22.111 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6399999999999999e-05 2024-04-20T17:46:22.111 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.111 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.111 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:46:22.111 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.584767+0000", 2024-04-20T17:46:22.111 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5617000000000004e-05 2024-04-20T17:46:22.111 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.111 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.112 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:22.112 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.589259+0000", 2024-04-20T17:46:22.112 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.004491933 2024-04-20T17:46:22.112 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.112 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.112 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:22.112 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.774804+0000", 2024-04-20T17:46:22.112 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.185544705 2024-04-20T17:46:22.112 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.112 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.112 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:22.112 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.774863+0000", 2024-04-20T17:46:22.112 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8452999999999998e-05 2024-04-20T17:46:22.112 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.113 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.113 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:22.113 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:59.774909+0000", 2024-04-20T17:46:22.113 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6496999999999999e-05 2024-04-20T17:46:22.113 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.113 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:22.113 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.113 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.113 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.113 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:141445 8.1 8:8e6b1248:::benchmark_data_smithi171_175757_object141444:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e806)", 2024-04-20T17:46:22.113 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:44:52.811692+0000", 2024-04-20T17:46:22.113 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 89.232210440000003, 2024-04-20T17:46:22.113 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.621181133, 2024-04-20T17:46:22.113 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:22.114 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:22.114 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:22.114 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:22.114 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:22.114 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:22.114 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 141445 2024-04-20T17:46:22.114 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.114 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:22.114 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.114 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:22.114 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.811692+0000", 2024-04-20T17:46:22.114 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.114 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.115 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.115 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:22.115 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.811692+0000", 2024-04-20T17:46:22.115 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.115 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.115 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.115 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:22.115 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.811695+0000", 2024-04-20T17:46:22.115 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1269999999999999e-06 2024-04-20T17:46:22.115 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.115 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.115 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:22.115 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.811726+0000", 2024-04-20T17:46:22.115 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1581000000000001e-05 2024-04-20T17:46:22.115 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.115 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.116 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:22.116 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.811743+0000", 2024-04-20T17:46:22.116 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6246999999999999e-05 2024-04-20T17:46:22.116 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.116 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.116 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:22.116 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.811754+0000", 2024-04-20T17:46:22.116 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1404e-05 2024-04-20T17:46:22.116 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.116 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.116 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.116 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.815407+0000", 2024-04-20T17:46:22.116 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0036529330000000001 2024-04-20T17:46:22.116 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.116 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.117 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-04-20T17:46:22.117 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.815422+0000", 2024-04-20T17:46:22.117 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4694e-05 2024-04-20T17:46:22.117 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.117 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.117 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.117 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.413428+0000", 2024-04-20T17:46:22.117 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.59800629599999999 2024-04-20T17:46:22.117 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.117 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.117 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:22.117 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.413547+0000", 2024-04-20T17:46:22.117 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000118888 2024-04-20T17:46:22.117 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.117 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.118 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:46:22.118 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.413672+0000", 2024-04-20T17:46:22.118 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012481699999999999 2024-04-20T17:46:22.118 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.118 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.118 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:22.118 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.429257+0000", 2024-04-20T17:46:22.118 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015585026 2024-04-20T17:46:22.118 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.118 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.118 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:22.118 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.429268+0000", 2024-04-20T17:46:22.118 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1554e-05 2024-04-20T17:46:22.118 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.119 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.119 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:22.119 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.429306+0000", 2024-04-20T17:46:22.119 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7536000000000001e-05 2024-04-20T17:46:22.119 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.119 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.119 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:22.119 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.432873+0000", 2024-04-20T17:46:22.119 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0035670300000000001 2024-04-20T17:46:22.119 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.119 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:22.119 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.119 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.119 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.119 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:141492 8.1 8:85ea95fc:::benchmark_data_smithi171_175757_object141491:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e806)", 2024-04-20T17:46:22.120 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:44:52.811788+0000", 2024-04-20T17:46:22.120 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 89.232114330000002, 2024-04-20T17:46:22.120 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.62512437399999998, 2024-04-20T17:46:22.120 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:22.120 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:22.120 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:22.120 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:22.120 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:22.120 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:22.120 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 141492 2024-04-20T17:46:22.120 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.120 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:22.120 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.120 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:22.120 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.811788+0000", 2024-04-20T17:46:22.121 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.121 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.121 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.121 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:22.121 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.811788+0000", 2024-04-20T17:46:22.121 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.121 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.121 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.121 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:22.121 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.811788+0000", 2024-04-20T17:46:22.121 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4900000000000005e-07 2024-04-20T17:46:22.121 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.121 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.121 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:22.122 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.811808+0000", 2024-04-20T17:46:22.122 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9616999999999999e-05 2024-04-20T17:46:22.122 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.122 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.122 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:22.122 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.811815+0000", 2024-04-20T17:46:22.122 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7329999999999997e-06 2024-04-20T17:46:22.122 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.122 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.122 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:22.122 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.811820+0000", 2024-04-20T17:46:22.122 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8300000000000003e-06 2024-04-20T17:46:22.122 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.122 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.123 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.123 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.815463+0000", 2024-04-20T17:46:22.123 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0036429320000000002 2024-04-20T17:46:22.123 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.123 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.123 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-04-20T17:46:22.123 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.815473+0000", 2024-04-20T17:46:22.123 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0655e-05 2024-04-20T17:46:22.123 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.123 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.123 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.123 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.421223+0000", 2024-04-20T17:46:22.123 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.60574987199999997 2024-04-20T17:46:22.123 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.123 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.124 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:22.124 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.421335+0000", 2024-04-20T17:46:22.124 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000112277 2024-04-20T17:46:22.124 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.124 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.124 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:46:22.124 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.421490+0000", 2024-04-20T17:46:22.124 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015482699999999999 2024-04-20T17:46:22.124 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.124 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.124 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:22.124 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.432999+0000", 2024-04-20T17:46:22.124 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011508832 2024-04-20T17:46:22.124 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.124 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.124 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:22.125 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.433013+0000", 2024-04-20T17:46:22.125 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3803e-05 2024-04-20T17:46:22.125 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.125 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.125 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:22.125 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.433071+0000", 2024-04-20T17:46:22.125 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7995999999999998e-05 2024-04-20T17:46:22.125 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.125 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.125 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:22.125 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.436912+0000", 2024-04-20T17:46:22.125 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.003841251 2024-04-20T17:46:22.125 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.125 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:22.125 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.126 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.126 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.126 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:141522 8.1 8:8177863d:::benchmark_data_smithi171_175757_object141521:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e806)", 2024-04-20T17:46:22.126 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:44:52.811826+0000", 2024-04-20T17:46:22.126 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 89.232076058000004, 2024-04-20T17:46:22.126 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.63403958699999996, 2024-04-20T17:46:22.126 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:22.126 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:22.126 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:22.126 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:22.126 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:22.126 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:22.126 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 141522 2024-04-20T17:46:22.126 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.127 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:22.127 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.127 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:22.127 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.811826+0000", 2024-04-20T17:46:22.127 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.127 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.127 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.127 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:22.127 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.811826+0000", 2024-04-20T17:46:22.127 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.127 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.127 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.127 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:22.127 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.811826+0000", 2024-04-20T17:46:22.127 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0400000000000002e-07 2024-04-20T17:46:22.128 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.128 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.128 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:22.128 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.811845+0000", 2024-04-20T17:46:22.128 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.859e-05 2024-04-20T17:46:22.128 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.128 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.128 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:22.128 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.811850+0000", 2024-04-20T17:46:22.128 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3700000000000003e-06 2024-04-20T17:46:22.128 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.128 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.128 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:22.128 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.811854+0000", 2024-04-20T17:46:22.128 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0069999999999999e-06 2024-04-20T17:46:22.128 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.129 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.129 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.129 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.815514+0000", 2024-04-20T17:46:22.129 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.003660136 2024-04-20T17:46:22.129 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.129 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.129 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-04-20T17:46:22.129 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.815525+0000", 2024-04-20T17:46:22.129 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0995000000000001e-05 2024-04-20T17:46:22.129 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.129 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.129 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.129 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.424841+0000", 2024-04-20T17:46:22.129 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.60931517300000004 2024-04-20T17:46:22.129 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.130 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.130 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:22.130 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.424959+0000", 2024-04-20T17:46:22.130 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000117903 2024-04-20T17:46:22.130 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.130 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.130 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:46:22.130 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.425069+0000", 2024-04-20T17:46:22.130 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000110722 2024-04-20T17:46:22.130 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.130 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.130 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:22.131 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.436923+0000", 2024-04-20T17:46:22.131 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011854151 2024-04-20T17:46:22.131 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.131 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.131 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:22.131 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.441718+0000", 2024-04-20T17:46:22.131 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0047946270000000001 2024-04-20T17:46:22.131 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.131 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.131 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:22.131 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.441772+0000", 2024-04-20T17:46:22.131 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4348999999999998e-05 2024-04-20T17:46:22.131 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.131 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.131 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:22.132 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.445866+0000", 2024-04-20T17:46:22.132 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0040931600000000002 2024-04-20T17:46:22.132 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.132 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:22.132 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.132 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.132 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.132 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:152849 8.1 8:87433ad4:::benchmark_data_smithi171_175757_object7735:head [delete] snapc 0=[] ondisk+write+known_if_redirected e813)", 2024-04-20T17:46:22.132 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:45:10.466559+0000", 2024-04-20T17:46:22.132 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 71.577343267000003, 2024-04-20T17:46:22.132 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.187881297, 2024-04-20T17:46:22.132 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:22.132 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:22.132 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:22.132 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:22.133 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:22.133 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:22.133 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 152849 2024-04-20T17:46:22.133 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.133 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:22.133 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.133 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:22.133 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:10.466559+0000", 2024-04-20T17:46:22.133 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.133 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.133 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.133 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:22.133 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:10.466559+0000", 2024-04-20T17:46:22.133 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.133 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.134 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.134 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:22.134 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:10.466561+0000", 2024-04-20T17:46:22.134 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5239999999999999e-06 2024-04-20T17:46:22.134 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.134 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.134 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:22.134 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:10.466568+0000", 2024-04-20T17:46:22.134 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8190000000000003e-06 2024-04-20T17:46:22.134 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.134 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.134 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:22.134 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:10.466580+0000", 2024-04-20T17:46:22.134 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1810999999999999e-05 2024-04-20T17:46:22.135 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.135 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.135 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:22.135 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:10.466588+0000", 2024-04-20T17:46:22.135 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5189999999999992e-06 2024-04-20T17:46:22.135 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.135 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.135 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.135 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:10.497805+0000", 2024-04-20T17:46:22.135 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.031216166 2024-04-20T17:46:22.135 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.135 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.135 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for degraded object", 2024-04-20T17:46:22.135 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:10.498375+0000", 2024-04-20T17:46:22.136 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00057085399999999998 2024-04-20T17:46:22.136 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.136 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.136 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.136 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:10.616092+0000", 2024-04-20T17:46:22.136 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.11771638099999999 2024-04-20T17:46:22.136 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.136 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.136 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:22.136 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:10.616155+0000", 2024-04-20T17:46:22.136 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3170999999999995e-05 2024-04-20T17:46:22.136 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.136 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.136 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:46:22.137 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:10.616250+0000", 2024-04-20T17:46:22.137 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.4813000000000002e-05 2024-04-20T17:46:22.137 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.137 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.137 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:22.137 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:10.650245+0000", 2024-04-20T17:46:22.137 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.033994942 2024-04-20T17:46:22.137 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.137 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.137 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:22.137 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:10.650284+0000", 2024-04-20T17:46:22.137 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9523000000000001e-05 2024-04-20T17:46:22.137 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.137 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.137 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:22.138 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:10.650318+0000", 2024-04-20T17:46:22.138 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3698000000000002e-05 2024-04-20T17:46:22.138 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.138 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.138 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:22.138 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:10.654440+0000", 2024-04-20T17:46:22.138 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0041220759999999997 2024-04-20T17:46:22.138 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.138 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:22.138 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.138 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.138 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.138 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:153931 8.1 8:8ef7b726:::benchmark_data_smithi171_175757_object8817:head [delete] snapc 0=[] ondisk+write+known_if_redirected e814)", 2024-04-20T17:46:22.138 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:45:12.386359+0000", 2024-04-20T17:46:22.139 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 69.657542622999998, 2024-04-20T17:46:22.139 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.22675635399999999, 2024-04-20T17:46:22.139 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:22.139 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:22.139 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:22.139 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:22.139 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:22.139 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:22.139 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 153931 2024-04-20T17:46:22.139 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.139 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:22.139 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.139 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:22.139 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:12.386359+0000", 2024-04-20T17:46:22.139 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.140 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.140 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.140 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:22.140 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:12.386359+0000", 2024-04-20T17:46:22.140 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.140 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.140 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.140 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:22.140 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:12.386363+0000", 2024-04-20T17:46:22.140 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1640000000000001e-06 2024-04-20T17:46:22.140 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.140 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.140 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:22.140 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:12.386372+0000", 2024-04-20T17:46:22.140 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2299999999999997e-06 2024-04-20T17:46:22.141 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.141 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.141 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:22.141 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:12.386390+0000", 2024-04-20T17:46:22.141 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8652999999999999e-05 2024-04-20T17:46:22.141 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.141 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.141 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:22.141 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:12.386401+0000", 2024-04-20T17:46:22.141 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0661e-05 2024-04-20T17:46:22.141 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.141 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.141 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.141 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:12.437409+0000", 2024-04-20T17:46:22.141 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.051008196999999998 2024-04-20T17:46:22.142 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.142 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.142 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for degraded object", 2024-04-20T17:46:22.142 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:12.437478+0000", 2024-04-20T17:46:22.142 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8442000000000006e-05 2024-04-20T17:46:22.142 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.142 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.142 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.142 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:12.540861+0000", 2024-04-20T17:46:22.142 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.103383526 2024-04-20T17:46:22.142 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.142 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.142 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:22.143 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:12.540951+0000", 2024-04-20T17:46:22.143 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9277999999999995e-05 2024-04-20T17:46:22.143 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.143 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.143 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:46:22.143 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:12.541067+0000", 2024-04-20T17:46:22.143 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000116734 2024-04-20T17:46:22.143 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.143 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.143 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:22.143 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:12.567604+0000", 2024-04-20T17:46:22.143 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026536731000000001 2024-04-20T17:46:22.143 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.143 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.144 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:22.144 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:12.609027+0000", 2024-04-20T17:46:22.144 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.041423143000000003 2024-04-20T17:46:22.144 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.144 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.144 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:22.144 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:12.609090+0000", 2024-04-20T17:46:22.144 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3127999999999996e-05 2024-04-20T17:46:22.144 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.144 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.144 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:22.144 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:12.613116+0000", 2024-04-20T17:46:22.144 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0040254669999999996 2024-04-20T17:46:22.144 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.144 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:22.145 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.145 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.145 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.145 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:173095 8.1 8:84e7d22e:::benchmark_data_smithi171_175757_object27981:head [delete] snapc 0=[] ondisk+write+known_if_redirected e826)", 2024-04-20T17:46:22.145 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:45:45.289364+0000", 2024-04-20T17:46:22.145 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 36.754537921000001, 2024-04-20T17:46:22.145 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.21319691700000001, 2024-04-20T17:46:22.145 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:22.145 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:22.145 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:22.145 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:22.145 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:22.145 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:22.146 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 173095 2024-04-20T17:46:22.146 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.146 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:22.146 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.146 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:22.146 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.289364+0000", 2024-04-20T17:46:22.146 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.146 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.146 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.146 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:22.146 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.289364+0000", 2024-04-20T17:46:22.147 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.147 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.147 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.147 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:22.147 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.289366+0000", 2024-04-20T17:46:22.147 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9999999999999999e-06 2024-04-20T17:46:22.147 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.147 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.147 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:22.147 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.289373+0000", 2024-04-20T17:46:22.147 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6490000000000002e-06 2024-04-20T17:46:22.147 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.147 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.147 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:22.148 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.289384+0000", 2024-04-20T17:46:22.148 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1725e-05 2024-04-20T17:46:22.148 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.148 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.148 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:22.148 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.289393+0000", 2024-04-20T17:46:22.148 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.7839999999999992e-06 2024-04-20T17:46:22.148 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.148 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.148 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.148 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.354407+0000", 2024-04-20T17:46:22.148 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.065013435999999994 2024-04-20T17:46:22.148 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.148 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.148 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for degraded object", 2024-04-20T17:46:22.149 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.355073+0000", 2024-04-20T17:46:22.149 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00066591100000000004 2024-04-20T17:46:22.149 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.149 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.149 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.149 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.378824+0000", 2024-04-20T17:46:22.149 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.023751404 2024-04-20T17:46:22.149 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.149 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.149 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:22.149 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.378889+0000", 2024-04-20T17:46:22.149 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5310999999999999e-05 2024-04-20T17:46:22.149 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.149 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.150 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:46:22.150 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.378972+0000", 2024-04-20T17:46:22.150 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2454999999999996e-05 2024-04-20T17:46:22.150 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.150 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.150 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:22.150 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.499024+0000", 2024-04-20T17:46:22.150 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.120052216 2024-04-20T17:46:22.150 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.150 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.150 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:22.150 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.499075+0000", 2024-04-20T17:46:22.150 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0708999999999998e-05 2024-04-20T17:46:22.150 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.151 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.151 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:22.151 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.499134+0000", 2024-04-20T17:46:22.151 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9076999999999997e-05 2024-04-20T17:46:22.151 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.151 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.151 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:22.151 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.502561+0000", 2024-04-20T17:46:22.151 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0034272399999999998 2024-04-20T17:46:22.151 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.151 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:22.151 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.151 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.151 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.151 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:173164 8.1 8:81925e01:::benchmark_data_smithi171_175757_object28050:head [delete] snapc 0=[] ondisk+write+known_if_redirected e826)", 2024-04-20T17:46:22.152 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:45:45.392674+0000", 2024-04-20T17:46:22.152 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 36.651228424999999, 2024-04-20T17:46:22.152 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.18971637599999999, 2024-04-20T17:46:22.152 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:22.152 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:22.152 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:22.152 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:22.152 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:22.152 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:22.152 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 173164 2024-04-20T17:46:22.152 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.152 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:22.152 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.152 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:22.152 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.392674+0000", 2024-04-20T17:46:22.153 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.153 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.153 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.153 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:22.153 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.392674+0000", 2024-04-20T17:46:22.153 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.153 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.153 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.153 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:22.153 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.392676+0000", 2024-04-20T17:46:22.153 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1610000000000001e-06 2024-04-20T17:46:22.153 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.153 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.153 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:22.153 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.392682+0000", 2024-04-20T17:46:22.154 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5080000000000002e-06 2024-04-20T17:46:22.154 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.154 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.154 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:22.154 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.392691+0000", 2024-04-20T17:46:22.154 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6209999999999993e-06 2024-04-20T17:46:22.154 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.154 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.154 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:22.154 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.392697+0000", 2024-04-20T17:46:22.154 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6289999999999998e-06 2024-04-20T17:46:22.154 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.154 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.154 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.154 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.516735+0000", 2024-04-20T17:46:22.155 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.124038167 2024-04-20T17:46:22.155 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.155 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.155 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for degraded object", 2024-04-20T17:46:22.155 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.516817+0000", 2024-04-20T17:46:22.155 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2536999999999999e-05 2024-04-20T17:46:22.155 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.155 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.155 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.155 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.539016+0000", 2024-04-20T17:46:22.155 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022199180999999998 2024-04-20T17:46:22.155 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.155 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.155 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:22.155 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.539115+0000", 2024-04-20T17:46:22.156 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.8659999999999994e-05 2024-04-20T17:46:22.156 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.156 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.156 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:46:22.156 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.539259+0000", 2024-04-20T17:46:22.156 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000144301 2024-04-20T17:46:22.156 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.156 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.156 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:22.156 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.562627+0000", 2024-04-20T17:46:22.156 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.023367743999999999 2024-04-20T17:46:22.156 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.156 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.156 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:22.157 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.578427+0000", 2024-04-20T17:46:22.157 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015800380999999999 2024-04-20T17:46:22.157 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.157 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.157 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:22.157 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.578490+0000", 2024-04-20T17:46:22.157 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2984000000000005e-05 2024-04-20T17:46:22.157 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.157 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.157 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:22.157 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:45.582390+0000", 2024-04-20T17:46:22.157 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.003899502 2024-04-20T17:46:22.157 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.157 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:22.158 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.158 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.158 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.158 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:183872 8.1 8:8323b0e4:::benchmark_data_smithi171_175757_object38758:head [delete] snapc 0=[] ondisk+write+known_if_redirected e830)", 2024-04-20T17:46:22.158 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:46:03.001915+0000", 2024-04-20T17:46:22.158 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 19.041987267, 2024-04-20T17:46:22.158 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.208446875, 2024-04-20T17:46:22.158 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:22.158 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:22.158 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:22.158 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:22.158 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:22.158 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:22.159 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 183872 2024-04-20T17:46:22.159 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.159 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:22.159 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.159 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:22.159 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.001915+0000", 2024-04-20T17:46:22.159 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.159 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.159 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.159 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:22.159 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.001915+0000", 2024-04-20T17:46:22.159 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.159 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.160 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.160 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:22.160 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.001915+0000", 2024-04-20T17:46:22.160 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.51e-07 2024-04-20T17:46:22.160 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.160 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.160 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:22.160 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.001918+0000", 2024-04-20T17:46:22.160 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3649999999999998e-06 2024-04-20T17:46:22.160 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.160 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.160 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:22.160 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.001922+0000", 2024-04-20T17:46:22.160 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3200000000000001e-06 2024-04-20T17:46:22.160 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.160 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.161 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:22.161 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.001926+0000", 2024-04-20T17:46:22.161 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1119999999999998e-06 2024-04-20T17:46:22.161 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.161 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.161 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.161 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.048832+0000", 2024-04-20T17:46:22.161 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.046906390999999999 2024-04-20T17:46:22.161 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.161 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.161 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for degraded object", 2024-04-20T17:46:22.161 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.049452+0000", 2024-04-20T17:46:22.161 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00061952000000000003 2024-04-20T17:46:22.161 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.161 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.162 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.162 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.068923+0000", 2024-04-20T17:46:22.162 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019470931 2024-04-20T17:46:22.162 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.162 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.162 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:22.162 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.069032+0000", 2024-04-20T17:46:22.162 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000109368 2024-04-20T17:46:22.162 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.162 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.162 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:46:22.162 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.069184+0000", 2024-04-20T17:46:22.162 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000151223 2024-04-20T17:46:22.162 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.162 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.163 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:22.163 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.109581+0000", 2024-04-20T17:46:22.163 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.040397496999999997 2024-04-20T17:46:22.163 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.163 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.163 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:22.163 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.206849+0000", 2024-04-20T17:46:22.163 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.097267977000000005 2024-04-20T17:46:22.163 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.163 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.163 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:22.163 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.206899+0000", 2024-04-20T17:46:22.163 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9696e-05 2024-04-20T17:46:22.163 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.164 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.164 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:22.164 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.210362+0000", 2024-04-20T17:46:22.164 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.003462924 2024-04-20T17:46:22.164 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.164 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:22.164 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.164 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.164 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.164 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:184044 8.1 8:85111e11:::benchmark_data_smithi171_175757_object38930:head [delete] snapc 0=[] ondisk+write+known_if_redirected e830)", 2024-04-20T17:46:22.164 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:46:03.367604+0000", 2024-04-20T17:46:22.164 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 18.676298267, 2024-04-20T17:46:22.164 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.248470728, 2024-04-20T17:46:22.164 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:22.164 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:22.164 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:22.165 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:22.165 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:22.165 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:22.165 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 184044 2024-04-20T17:46:22.165 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.165 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:22.165 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.165 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:22.165 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.367604+0000", 2024-04-20T17:46:22.165 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.165 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.165 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.165 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:22.165 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.367604+0000", 2024-04-20T17:46:22.165 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:22.166 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.166 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.166 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:22.166 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.367605+0000", 2024-04-20T17:46:22.166 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.204e-06 2024-04-20T17:46:22.166 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.166 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.166 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:22.166 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.367609+0000", 2024-04-20T17:46:22.166 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5020000000000003e-06 2024-04-20T17:46:22.166 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.166 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.166 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:22.166 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.367620+0000", 2024-04-20T17:46:22.166 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0305999999999999e-05 2024-04-20T17:46:22.167 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.167 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.167 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:22.167 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.367626+0000", 2024-04-20T17:46:22.167 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.703e-06 2024-04-20T17:46:22.167 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.167 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.167 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.167 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.454129+0000", 2024-04-20T17:46:22.167 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.086502522999999998 2024-04-20T17:46:22.167 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.167 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.167 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for degraded object", 2024-04-20T17:46:22.167 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.454194+0000", 2024-04-20T17:46:22.167 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4769000000000003e-05 2024-04-20T17:46:22.168 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.168 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.168 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:22.168 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.558201+0000", 2024-04-20T17:46:22.168 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.10400712300000001 2024-04-20T17:46:22.168 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.168 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.168 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:22.168 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.558321+0000", 2024-04-20T17:46:22.168 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012010699999999999 2024-04-20T17:46:22.168 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.168 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.168 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:46:22.169 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.558487+0000", 2024-04-20T17:46:22.169 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00016603500000000001 2024-04-20T17:46:22.169 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.169 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.169 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:22.169 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.601028+0000", 2024-04-20T17:46:22.169 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.042541258999999998 2024-04-20T17:46:22.169 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.169 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.169 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:22.169 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.613158+0000", 2024-04-20T17:46:22.169 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.012129453 2024-04-20T17:46:22.169 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.169 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.170 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:22.170 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.613201+0000", 2024-04-20T17:46:22.170 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2948999999999998e-05 2024-04-20T17:46:22.170 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:22.170 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:22.170 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:22.170 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:03.616074+0000", 2024-04-20T17:46:22.170 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0028737950000000002 2024-04-20T17:46:22.170 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.170 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:22.170 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.170 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:22.170 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:22.170 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:46:22.171 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:46:22.207 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:22.244 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:46:22.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:22.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:46:22.304 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:46:22.332 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:46:22.347 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:46:22.362 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:46:22.387 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:22.395 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:46:22.422 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:46:22.450 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:46:22.489 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:46:22.518 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:46:22.561 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:46:22.594 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:46:22.624 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:46:22.664 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:46:22.665 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:46:22.665 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:46:22.666 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:46:22.666 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:46:22.666 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:46:22.666 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:46:22.666 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:46:22.666 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:46:22.666 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:46:22.666 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:46:22.666 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:46:22.666 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:46:22.824 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:22 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1830828413' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:46:22.824 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:22 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:22.872 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-20T17:46:22.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:22 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1830828413' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:46:22.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:22 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:23.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:22 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1830828413' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:46:23.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:22 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:23.330 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:46:23.607 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:46:23.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:23.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:23.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:23.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:23.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:23.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:23.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:23.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:23.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:23.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:23.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:23 smithi005 ceph-mon[39562]: pgmap v2235: 27 pgs: 3 active+recovering, 24 active+clean; 5.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 536 op/s; 17086/191332 objects misplaced (8.930%); 6.4 MiB/s, 101 objects/s recovering 2024-04-20T17:46:23.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:23 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:23.995 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:23.995 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:23.995 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:23.996 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:23.996 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:23.996 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:23.996 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:23.996 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:23.996 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:23.996 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:23.996 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:23 smithi171 ceph-mon[44189]: pgmap v2235: 27 pgs: 3 active+recovering, 24 active+clean; 5.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 536 op/s; 17086/191332 objects misplaced (8.930%); 6.4 MiB/s, 101 objects/s recovering 2024-04-20T17:46:23.996 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:23 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:24.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:24.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:24.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:23 smithi106 ceph-mon[44423]: pgmap v2235: 27 pgs: 3 active+recovering, 24 active+clean; 5.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 536 op/s; 17086/191332 objects misplaced (8.930%); 6.4 MiB/s, 101 objects/s recovering 2024-04-20T17:46:24.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:23 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:24.984 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:24 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:46:25.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:24 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:46:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:24 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:46:25.647 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:46:25.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:25.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:46:25.721 INFO:teuthology.orchestra.run.smithi005.stderr:set pool 1 pgp_num to 11 2024-04-20T17:46:25.725 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:25 smithi171 ceph-mon[44189]: pgmap v2236: 27 pgs: 3 active+recovering, 24 active+clean; 5.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 574 op/s; 17086/189744 objects misplaced (9.005%); 5.5 MiB/s, 87 objects/s recovering 2024-04-20T17:46:25.725 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:25 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2031987215' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-20T17:46:25.725 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:25 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:46:25.726 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:25 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:46:25.752 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:46:25.781 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:25.801 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:46:25.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:25.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:46:25.905 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:46:25.932 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:25.951 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:46:25.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:25.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:46:25.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:25 smithi005 ceph-mon[39562]: pgmap v2236: 27 pgs: 3 active+recovering, 24 active+clean; 5.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 574 op/s; 17086/189744 objects misplaced (9.005%); 5.5 MiB/s, 87 objects/s recovering 2024-04-20T17:46:25.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:25 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2031987215' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-20T17:46:25.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:25 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:46:25.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:25 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:46:26.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:25 smithi106 ceph-mon[44423]: pgmap v2236: 27 pgs: 3 active+recovering, 24 active+clean; 5.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 574 op/s; 17086/189744 objects misplaced (9.005%); 5.5 MiB/s, 87 objects/s recovering 2024-04-20T17:46:26.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:25 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2031987215' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-20T17:46:26.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:25 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:46:26.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:25 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:46:26.057 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:46:26.108 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:26.159 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:46:26.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:26.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:46:26.263 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:46:26.298 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:26.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:26 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2031987215' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-04-20T17:46:26.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:26 smithi005 ceph-mon[39562]: osdmap e838: 12 total, 12 up, 11 in 2024-04-20T17:46:26.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:26 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:46:26.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:26 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:46:26.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:26 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:46:27.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:26 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2031987215' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-04-20T17:46:27.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:26 smithi106 ceph-mon[44423]: osdmap e838: 12 total, 12 up, 11 in 2024-04-20T17:46:27.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:26 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:46:27.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:26 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:46:27.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:26 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:46:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:26 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2031987215' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-04-20T17:46:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:26 smithi171 ceph-mon[44189]: osdmap e838: 12 total, 12 up, 11 in 2024-04-20T17:46:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:26 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:46:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:26 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:46:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:26 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:46:27.188 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:46:27.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:27.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:46:27.295 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:46:27.360 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:27.671 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:46:27.737 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:27 smithi171 ceph-mon[44189]: pgmap v2238: 27 pgs: 3 active+recovering, 24 active+clean; 5.6 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 800 op/s; 16485/185004 objects misplaced (8.911%); 11 MiB/s, 173 objects/s recovering 2024-04-20T17:46:27.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:27 smithi005 ceph-mon[39562]: pgmap v2238: 27 pgs: 3 active+recovering, 24 active+clean; 5.6 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 800 op/s; 16485/185004 objects misplaced (8.911%); 11 MiB/s, 173 objects/s recovering 2024-04-20T17:46:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:27 smithi106 ceph-mon[44423]: pgmap v2238: 27 pgs: 3 active+recovering, 24 active+clean; 5.6 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 800 op/s; 16485/185004 objects misplaced (8.911%); 11 MiB/s, 173 objects/s recovering 2024-04-20T17:46:28.062 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:46:28.755 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:46:28.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:28 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3440052195' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:46:28.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:28 smithi005 ceph-mon[39562]: from='client.27833 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:46:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:28 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3440052195' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:46:29.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:28 smithi106 ceph-mon[44423]: from='client.27833 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:46:29.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:28 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3440052195' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:46:29.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:28 smithi171 ceph-mon[44189]: from='client.27833 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:46:29.432 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:46:29.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:29.439 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:46:29.540 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:46:29.702 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:29.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:29 smithi005 ceph-mon[39562]: pgmap v2239: 27 pgs: 3 active+recovering, 24 active+clean; 5.6 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 489 op/s; 16485/185004 objects misplaced (8.911%); 6.3 MiB/s, 100 objects/s recovering 2024-04-20T17:46:29.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:29 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:46:29.958 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:29 smithi005 ceph-mon[39562]: from='client.27833 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:46:29.958 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:29 smithi005 ceph-mon[39562]: osdmap e839: 12 total, 12 up, 11 in 2024-04-20T17:46:30.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:29 smithi106 ceph-mon[44423]: pgmap v2239: 27 pgs: 3 active+recovering, 24 active+clean; 5.6 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 489 op/s; 16485/185004 objects misplaced (8.911%); 6.3 MiB/s, 100 objects/s recovering 2024-04-20T17:46:30.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:29 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:46:30.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:29 smithi106 ceph-mon[44423]: from='client.27833 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:46:30.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:29 smithi106 ceph-mon[44423]: osdmap e839: 12 total, 12 up, 11 in 2024-04-20T17:46:30.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:29 smithi171 ceph-mon[44189]: pgmap v2239: 27 pgs: 3 active+recovering, 24 active+clean; 5.6 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 489 op/s; 16485/185004 objects misplaced (8.911%); 6.3 MiB/s, 100 objects/s recovering 2024-04-20T17:46:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:29 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:46:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:29 smithi171 ceph-mon[44189]: from='client.27833 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:46:30.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:29 smithi171 ceph-mon[44189]: osdmap e839: 12 total, 12 up, 11 in 2024-04-20T17:46:30.630 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:46:30.658 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:46:30.685 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:46:30.710 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:46:30.734 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:46:30.765 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:46:30.791 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:46:30.820 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:46:30.849 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:46:30.889 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:46:30.915 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:46:30.945 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:46:30.951 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:46:30.952 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:46:30.952 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:46:30.952 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:46:30.952 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:46:30.952 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:46:30.952 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:46:30.952 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:46:30.952 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:46:30.952 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:46:30.952 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:46:31.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:30 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1403709039' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:46:31.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:30 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1403709039' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:46:31.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:30 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1403709039' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:46:31.595 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:46:31.739 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 8, 9, 10, 11, 1, 6] out_osds: [7] dead_osds: [] live_osds: [0, 4, 5, 8, 9, 10, 11, 2, 6, 3, 1, 7] 2024-04-20T17:46:31.739 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:46:31.739 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-04-20T17:46:31.739 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd in 7 2024-04-20T17:46:31.944 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:46:31.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:31.948 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:46:31.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:31 smithi005 ceph-mon[39562]: pgmap v2241: 27 pgs: 3 active+recovering, 24 active+clean; 5.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 712 op/s; 16096/179922 objects misplaced (8.946%); 7.7 MiB/s, 123 objects/s recovering 2024-04-20T17:46:31.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:31.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:31.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:31.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:31.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:31.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:31.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:31.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:31.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:31.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:31.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:31 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:31.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:31 smithi005 ceph-mon[39562]: osdmap e840: 12 total, 12 up, 11 in 2024-04-20T17:46:31.995 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:46:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:31 smithi106 ceph-mon[44423]: pgmap v2241: 27 pgs: 3 active+recovering, 24 active+clean; 5.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 712 op/s; 16096/179922 objects misplaced (8.946%); 7.7 MiB/s, 123 objects/s recovering 2024-04-20T17:46:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:32.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:31 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:32.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:31 smithi106 ceph-mon[44423]: osdmap e840: 12 total, 12 up, 11 in 2024-04-20T17:46:32.049 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:46:32.076 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:31 smithi171 ceph-mon[44189]: pgmap v2241: 27 pgs: 3 active+recovering, 24 active+clean; 5.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 712 op/s; 16096/179922 objects misplaced (8.946%); 7.7 MiB/s, 123 objects/s recovering 2024-04-20T17:46:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:32.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:31 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:32.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:31 smithi171 ceph-mon[44189]: osdmap e840: 12 total, 12 up, 11 in 2024-04-20T17:46:32.095 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:46:32.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:32.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:46:32.201 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:46:32.227 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:32.249 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:46:32.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:32.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:46:32.353 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:46:32.380 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:32.401 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:46:32.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:32.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:46:32.505 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:46:32.578 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:32.688 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:46:32.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:32.702 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:46:32.803 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:46:32.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:32 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:32.867 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:46:32.867 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:32.902 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:46:32.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:32.911 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:46:33.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:32 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:33.012 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:46:33.017 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:46:33.018 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:46:33.018 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:33.018 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.27460.0:202982 8.6 e841/777 8:660dec2e:::benchmark_data_smithi171_175757_object57868:head v 841'12670, mlcod=841'12668)", 2024-04-20T17:46:33.018 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:46:33.008220+0000", 2024-04-20T17:46:33.018 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.0082001580000000008, 2024-04-20T17:46:33.018 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0082315410000000002, 2024-04-20T17:46:33.018 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:33.018 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:33.018 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:46:33.018 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:33.018 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:33.018 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:33.008220+0000", 2024-04-20T17:46:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:33.008220+0000", 2024-04-20T17:46:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:33.008222+0000", 2024-04-20T17:46:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1019999999999999e-06 2024-04-20T17:46:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:33.020 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:33.020 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:33.008233+0000", 2024-04-20T17:46:33.020 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0617e-05 2024-04-20T17:46:33.020 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:33.020 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:33.020 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:33.020 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:33.008248+0000", 2024-04-20T17:46:33.020 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5824999999999998e-05 2024-04-20T17:46:33.020 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:33.020 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:33.020 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:33.020 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:33.008258+0000", 2024-04-20T17:46:33.020 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3850000000000008e-06 2024-04-20T17:46:33.021 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:33.021 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:33.021 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:33.021 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:33.008333+0000", 2024-04-20T17:46:33.021 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5103000000000002e-05 2024-04-20T17:46:33.021 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:33.021 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:33.021 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:33.021 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:33.008355+0000", 2024-04-20T17:46:33.021 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2405999999999999e-05 2024-04-20T17:46:33.021 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:33.021 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:33.021 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:202985 8.9 8:95da18f8:::benchmark_data_smithi171_175757_object57871:head [delete] snapc 0=[] ondisk+write+known_if_redirected e841)", 2024-04-20T17:46:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:46:33.013396+0000", 2024-04-20T17:46:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.0030241339999999999, 2024-04-20T17:46:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0031053309999999998, 2024-04-20T17:46:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:46:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 202985 2024-04-20T17:46:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:33.013396+0000", 2024-04-20T17:46:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:33.013396+0000", 2024-04-20T17:46:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:33.024 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:33.013399+0000", 2024-04-20T17:46:33.024 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.366e-06 2024-04-20T17:46:33.024 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:33.024 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:33.024 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:33.024 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:33.013409+0000", 2024-04-20T17:46:33.024 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.285e-06 2024-04-20T17:46:33.024 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:33.024 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:33.024 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:33.024 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:33.013423+0000", 2024-04-20T17:46:33.024 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.42e-05 2024-04-20T17:46:33.024 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:33.025 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:33.025 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:33.025 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:33.013434+0000", 2024-04-20T17:46:33.025 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0988000000000001e-05 2024-04-20T17:46:33.025 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:33.025 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:33.025 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:33.025 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:33.013642+0000", 2024-04-20T17:46:33.025 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000207703 2024-04-20T17:46:33.025 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:33.025 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:33.025 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:33.025 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:33.013754+0000", 2024-04-20T17:46:33.025 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000112893 2024-04-20T17:46:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:46:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:33.013841+0000", 2024-04-20T17:46:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6138999999999996e-05 2024-04-20T17:46:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:202986 8.9 8.4cae2209 (undecoded) ondisk+write+known_if_redirected e841)", 2024-04-20T17:46:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:46:33.013461+0000", 2024-04-20T17:46:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.0029587659999999998, 2024-04-20T17:46:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.003084547, 2024-04-20T17:46:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "queued for pg", 2024-04-20T17:46:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 202986 2024-04-20T17:46:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:33.013461+0000", 2024-04-20T17:46:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:33.013461+0000", 2024-04-20T17:46:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:33.013462+0000", 2024-04-20T17:46:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9700000000000005e-07 2024-04-20T17:46:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:33.013466+0000", 2024-04-20T17:46:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6150000000000001e-06 2024-04-20T17:46:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:33.013472+0000", 2024-04-20T17:46:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2260000000000002e-06 2024-04-20T17:46:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:33.013479+0000", 2024-04-20T17:46:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5200000000000003e-06 2024-04-20T17:46:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: ], 2024-04-20T17:46:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 3 2024-04-20T17:46:33.029 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:46:33.038 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:33.056 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:46:33.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:33.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:46:33.117 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:32 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:33.161 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:46:33.208 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:33.238 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:46:33.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:33.244 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:46:33.344 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:46:33.412 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:33.760 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:33 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:46:33.760 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:33 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:46:33.760 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:33 smithi171 ceph-mon[44189]: pgmap v2243: 27 pgs: 3 active+recovering, 24 active+clean; 5.4 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 609 op/s; 15890/178210 objects misplaced (8.916%); 7.4 MiB/s, 118 objects/s recovering 2024-04-20T17:46:33.760 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:33 smithi171 ceph-mon[44189]: osdmap e841: 12 total, 12 up, 11 in 2024-04-20T17:46:33.774 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:46:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:33 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:46:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:33 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:46:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:33 smithi106 ceph-mon[44423]: pgmap v2243: 27 pgs: 3 active+recovering, 24 active+clean; 5.4 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 609 op/s; 15890/178210 objects misplaced (8.916%); 7.4 MiB/s, 118 objects/s recovering 2024-04-20T17:46:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:33 smithi106 ceph-mon[44423]: osdmap e841: 12 total, 12 up, 11 in 2024-04-20T17:46:34.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:33 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:46:34.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:33 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:46:34.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:33 smithi005 ceph-mon[39562]: pgmap v2243: 27 pgs: 3 active+recovering, 24 active+clean; 5.4 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 609 op/s; 15890/178210 objects misplaced (8.916%); 7.4 MiB/s, 118 objects/s recovering 2024-04-20T17:46:34.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:33 smithi005 ceph-mon[39562]: osdmap e841: 12 total, 12 up, 11 in 2024-04-20T17:46:35.611 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:46:35.612 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:46:35.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:35.620 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:46:35.721 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:46:35.774 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:46:35.830 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:36.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:35 smithi171 ceph-mon[44189]: pgmap v2245: 27 pgs: 3 active+recovering, 24 active+clean; 5.4 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 713 op/s; 15890/176442 objects misplaced (9.006%); 6.2 MiB/s, 99 objects/s recovering 2024-04-20T17:46:36.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:35 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3922331087' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:46:36.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:35 smithi005 ceph-mon[39562]: pgmap v2245: 27 pgs: 3 active+recovering, 24 active+clean; 5.4 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 713 op/s; 15890/176442 objects misplaced (9.006%); 6.2 MiB/s, 99 objects/s recovering 2024-04-20T17:46:36.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:35 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3922331087' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:46:36.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:35 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mon-a[39539]: 2024-04-20T17:46:35.942+0000 7f73b92a1700 -1 mon.a@0(leader).osd e842 definitely_dead 0 2024-04-20T17:46:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:35 smithi106 ceph-mon[44423]: pgmap v2245: 27 pgs: 3 active+recovering, 24 active+clean; 5.4 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 713 op/s; 15890/176442 objects misplaced (9.006%); 6.2 MiB/s, 99 objects/s recovering 2024-04-20T17:46:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:35 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3922331087' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:46:36.792 INFO:teuthology.orchestra.run.smithi005.stderr:marked in osd.7. 2024-04-20T17:46:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:36 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:46:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:36 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3922331087' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:46:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:36 smithi171 ceph-mon[44189]: osdmap e842: 12 total, 12 up, 11 in 2024-04-20T17:46:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:36 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2557498190' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-04-20T17:46:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:36 smithi171 ceph-mon[44189]: from='client.27848 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-04-20T17:46:36.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:36 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:46:36.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:36 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:46:36.939 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:46:36.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:36.945 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:46:36.995 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-20T17:46:37.045 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:46:37.095 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:37.140 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:46:37.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:37.144 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:46:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:36 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:46:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:36 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3922331087' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:46:37.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:36 smithi005 ceph-mon[39562]: osdmap e842: 12 total, 12 up, 11 in 2024-04-20T17:46:37.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:36 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2557498190' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-04-20T17:46:37.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:36 smithi005 ceph-mon[39562]: from='client.27848 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-04-20T17:46:37.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:36 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:46:37.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:36 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:46:37.244 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:46:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:36 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:46:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:36 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3922331087' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:46:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:36 smithi106 ceph-mon[44423]: osdmap e842: 12 total, 12 up, 11 in 2024-04-20T17:46:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:36 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2557498190' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-04-20T17:46:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:36 smithi106 ceph-mon[44423]: from='client.27848 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-04-20T17:46:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:36 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:46:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:36 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:46:37.281 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:37.302 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:46:37.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:37.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:46:37.376 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:46:37.406 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:46:37.441 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:37.645 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:46:37.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:37.649 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:46:37.750 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:46:37.777 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:37.797 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:46:37.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:37.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:46:37.901 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:46:37.928 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:37.947 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:46:37.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:37.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:46:37.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:37 smithi005 ceph-mon[39562]: pgmap v2247: 27 pgs: 3 active+recovering, 24 active+clean; 5.2 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 692 op/s; 15303/171614 objects misplaced (8.917%); 8.3 MiB/s, 132 objects/s recovering 2024-04-20T17:46:37.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:37 smithi005 ceph-mon[39562]: from='client.27848 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-04-20T17:46:37.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:37 smithi005 ceph-mon[39562]: osdmap e843: 12 total, 12 up, 12 in 2024-04-20T17:46:38.050 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:46:38.075 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:37 smithi106 ceph-mon[44423]: pgmap v2247: 27 pgs: 3 active+recovering, 24 active+clean; 5.2 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 692 op/s; 15303/171614 objects misplaced (8.917%); 8.3 MiB/s, 132 objects/s recovering 2024-04-20T17:46:38.075 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:37 smithi106 ceph-mon[44423]: from='client.27848 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-04-20T17:46:38.076 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:37 smithi106 ceph-mon[44423]: osdmap e843: 12 total, 12 up, 12 in 2024-04-20T17:46:38.077 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:38.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:37 smithi171 ceph-mon[44189]: pgmap v2247: 27 pgs: 3 active+recovering, 24 active+clean; 5.2 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 692 op/s; 15303/171614 objects misplaced (8.917%); 8.3 MiB/s, 132 objects/s recovering 2024-04-20T17:46:38.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:37 smithi171 ceph-mon[44189]: from='client.27848 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-04-20T17:46:38.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:37 smithi171 ceph-mon[44189]: osdmap e843: 12 total, 12 up, 12 in 2024-04-20T17:46:38.096 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:46:38.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:38.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:46:38.210 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:46:38.241 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:38.701 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:46:38.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:38.704 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:46:38.719 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-04-20T17:46:38.805 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:46:38.856 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:38.878 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:46:38.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:38.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:46:38.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:38 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:46:38.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:38 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:46:38.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:38 smithi005 ceph-mon[39562]: 1.6 scrub starts 2024-04-20T17:46:38.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:38 smithi005 ceph-mon[39562]: 1.6 scrub ok 2024-04-20T17:46:38.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:38 smithi005 ceph-mon[39562]: osdmap e844: 12 total, 12 up, 12 in 2024-04-20T17:46:38.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:38 smithi005 ceph-mon[39562]: osdmap e845: 12 total, 12 up, 12 in 2024-04-20T17:46:38.984 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:46:39.050 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:38 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:46:39.050 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:38 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:46:39.050 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:38 smithi106 ceph-mon[44423]: 1.6 scrub starts 2024-04-20T17:46:39.050 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:38 smithi106 ceph-mon[44423]: 1.6 scrub ok 2024-04-20T17:46:39.051 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:38 smithi106 ceph-mon[44423]: osdmap e844: 12 total, 12 up, 12 in 2024-04-20T17:46:39.051 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:38 smithi106 ceph-mon[44423]: osdmap e845: 12 total, 12 up, 12 in 2024-04-20T17:46:39.052 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:39.063 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:38 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:46:39.063 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:38 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:46:39.064 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:38 smithi171 ceph-mon[44189]: 1.6 scrub starts 2024-04-20T17:46:39.064 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:38 smithi171 ceph-mon[44189]: 1.6 scrub ok 2024-04-20T17:46:39.064 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:38 smithi171 ceph-mon[44189]: osdmap e844: 12 total, 12 up, 12 in 2024-04-20T17:46:39.064 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:38 smithi171 ceph-mon[44189]: osdmap e845: 12 total, 12 up, 12 in 2024-04-20T17:46:39.074 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:46:39.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:39.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:46:39.178 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:46:39.233 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:39.498 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:46:39.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:39.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:46:39.604 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:46:39.698 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:39.717 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:46:39.717 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:46:39.717 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-04-20T17:46:39.717 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-04-20T17:46:39.717 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:46:39.759 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:46:39.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:39.766 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:46:39.802 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:39 smithi005 ceph-mon[39562]: pgmap v2250: 27 pgs: 3 active+recovering, 24 active+clean; 5.2 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 561 op/s; 15303/171614 objects misplaced (8.917%); 6.2 MiB/s, 99 objects/s recovering 2024-04-20T17:46:39.802 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:39 smithi005 ceph-mon[39562]: 8.2 scrub starts 2024-04-20T17:46:39.802 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:39 smithi005 ceph-mon[39562]: 8.d starting backfill to osd.7 from (0'0,0'0] MAX to 842'12832 2024-04-20T17:46:39.866 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:46:39.893 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:39.914 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:46:39.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:39.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:46:39.986 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:46:40.009 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:46:40.019 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:46:40.034 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:46:40.047 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:46:40.048 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:40.062 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:46:40.067 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:46:40.069 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:39 smithi106 ceph-mon[44423]: pgmap v2250: 27 pgs: 3 active+recovering, 24 active+clean; 5.2 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 561 op/s; 15303/171614 objects misplaced (8.917%); 6.2 MiB/s, 99 objects/s recovering 2024-04-20T17:46:40.069 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:39 smithi106 ceph-mon[44423]: 8.2 scrub starts 2024-04-20T17:46:40.069 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:39 smithi106 ceph-mon[44423]: 8.d starting backfill to osd.7 from (0'0,0'0] MAX to 842'12832 2024-04-20T17:46:40.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:40.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:46:40.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:39 smithi171 ceph-mon[44189]: pgmap v2250: 27 pgs: 3 active+recovering, 24 active+clean; 5.2 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 561 op/s; 15303/171614 objects misplaced (8.917%); 6.2 MiB/s, 99 objects/s recovering 2024-04-20T17:46:40.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:39 smithi171 ceph-mon[44189]: 8.2 scrub starts 2024-04-20T17:46:40.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:39 smithi171 ceph-mon[44189]: 8.d starting backfill to osd.7 from (0'0,0'0] MAX to 842'12832 2024-04-20T17:46:40.094 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:46:40.124 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:46:40.149 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:46:40.171 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:46:40.179 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:46:40.229 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:46:40.242 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:40.266 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:46:40.291 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:46:40.327 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:46:40.334 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:46:40.334 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:46:40.334 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:46:40.334 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:46:40.334 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:46:40.334 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:46:40.334 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:46:40.334 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:46:40.334 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:46:40.334 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:46:40.335 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:46:40.335 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:46:40.460 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:46:40.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:40.469 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:46:40.501 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-20T17:46:40.570 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:46:40.598 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:40.607 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:46:40.616 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:46:40.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:40.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:46:40.722 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:46:40.755 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:40 smithi106 ceph-mon[44423]: 8.2 scrub ok 2024-04-20T17:46:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:40 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:46:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:40 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:46:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:40 smithi106 ceph-mon[44423]: osdmap e846: 12 total, 12 up, 12 in 2024-04-20T17:46:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:40 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1059652742' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:46:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:41.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:41.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:41.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:40 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:41.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:40 smithi106 ceph-mon[44423]: pgmap v2253: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 411 op/s; 19955/168324 objects misplaced (11.855%); 9.9 MiB/s, 158 objects/s recovering 2024-04-20T17:46:41.015 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:46:41.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:40 smithi171 ceph-mon[44189]: 8.2 scrub ok 2024-04-20T17:46:41.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:40 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:46:41.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:40 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:46:41.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:40 smithi171 ceph-mon[44189]: osdmap e846: 12 total, 12 up, 12 in 2024-04-20T17:46:41.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:40 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1059652742' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:46:41.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:41.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:41.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:41.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:41.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:40 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:41.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:40 smithi171 ceph-mon[44189]: pgmap v2253: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 411 op/s; 19955/168324 objects misplaced (11.855%); 9.9 MiB/s, 158 objects/s recovering 2024-04-20T17:46:41.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:40 smithi005 ceph-mon[39562]: 8.2 scrub ok 2024-04-20T17:46:41.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:40 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:46:41.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:40 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:46:41.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:40 smithi005 ceph-mon[39562]: osdmap e846: 12 total, 12 up, 12 in 2024-04-20T17:46:41.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:40 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1059652742' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:46:41.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:41.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:41.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:41.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:41.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:40 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:41.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:40 smithi005 ceph-mon[39562]: pgmap v2253: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 411 op/s; 19955/168324 objects misplaced (11.855%); 9.9 MiB/s, 158 objects/s recovering 2024-04-20T17:46:41.407 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:46:41.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:41.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:46:41.515 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:46:41.567 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:41.602 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:46:41.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:41.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:46:41.711 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:46:41.738 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:41.757 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:46:41.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:41.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:46:41.862 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:46:41.919 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:41 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:41 smithi106 ceph-mon[44423]: osdmap e847: 12 total, 12 up, 12 in 2024-04-20T17:46:42.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:41 smithi106 ceph-mon[44423]: 8.0 scrub starts 2024-04-20T17:46:42.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:41 smithi106 ceph-mon[44423]: osdmap e848: 12 total, 12 up, 12 in 2024-04-20T17:46:42.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:42.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:42.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:42.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:42.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:42.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:42.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:41 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:42.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:41 smithi171 ceph-mon[44189]: osdmap e847: 12 total, 12 up, 12 in 2024-04-20T17:46:42.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:41 smithi171 ceph-mon[44189]: 8.0 scrub starts 2024-04-20T17:46:42.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:41 smithi171 ceph-mon[44189]: osdmap e848: 12 total, 12 up, 12 in 2024-04-20T17:46:42.088 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:46:42.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:42.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:46:42.093 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:42.093 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:42.093 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:42.093 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:42.093 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:42.093 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:42.093 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:41 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:42.093 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:41 smithi005 ceph-mon[39562]: osdmap e847: 12 total, 12 up, 12 in 2024-04-20T17:46:42.093 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:41 smithi005 ceph-mon[39562]: 8.0 scrub starts 2024-04-20T17:46:42.093 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:41 smithi005 ceph-mon[39562]: osdmap e848: 12 total, 12 up, 12 in 2024-04-20T17:46:42.193 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:46:42.220 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:42.241 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:46:42.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:42.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:46:42.345 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:46:42.429 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:42.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:42 smithi005 ceph-mon[39562]: 8.e scrub starts 2024-04-20T17:46:42.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:46:42.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:46:42.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:46:42.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:46:42.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:46:42.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:46:42.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:46:42.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:42 smithi005 ceph-mon[39562]: 8.0 scrub ok 2024-04-20T17:46:42.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:42 smithi005 ceph-mon[39562]: pgmap v2256: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 543 op/s; 19743/167262 objects misplaced (11.804%); 13 MiB/s, 211 objects/s recovering 2024-04-20T17:46:43.196 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:46:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:42 smithi106 ceph-mon[44423]: 8.e scrub starts 2024-04-20T17:46:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:46:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:46:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:46:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:46:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:46:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:46:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:46:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:42 smithi106 ceph-mon[44423]: 8.0 scrub ok 2024-04-20T17:46:43.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:42 smithi106 ceph-mon[44423]: pgmap v2256: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 543 op/s; 19743/167262 objects misplaced (11.804%); 13 MiB/s, 211 objects/s recovering 2024-04-20T17:46:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:42 smithi171 ceph-mon[44189]: 8.e scrub starts 2024-04-20T17:46:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:46:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:46:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:46:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:46:43.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:46:43.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:46:43.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:46:43.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:42 smithi171 ceph-mon[44189]: 8.0 scrub ok 2024-04-20T17:46:43.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:42 smithi171 ceph-mon[44189]: pgmap v2256: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 543 op/s; 19743/167262 objects misplaced (11.804%); 13 MiB/s, 211 objects/s recovering 2024-04-20T17:46:43.720 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 8, 9, 10, 11, 1, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 4, 5, 8, 9, 10, 11, 2, 6, 3, 1, 7] 2024-04-20T17:46:43.721 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:46:43.721 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-20T17:46:43.721 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-20T17:46:43.899 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:46:43.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:43.905 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:46:43.909 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:46:44.005 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:46:44.064 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:44.109 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:46:44.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:44.118 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:43 smithi171 ceph-mon[44189]: 8.e scrub ok 2024-04-20T17:46:44.119 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:43 smithi171 ceph-mon[44189]: 8.8 scrub starts 2024-04-20T17:46:44.119 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:43 smithi171 ceph-mon[44189]: 8.8 scrub ok 2024-04-20T17:46:44.119 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:46:44.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:43 smithi005 ceph-mon[39562]: 8.e scrub ok 2024-04-20T17:46:44.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:43 smithi005 ceph-mon[39562]: 8.8 scrub starts 2024-04-20T17:46:44.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:43 smithi005 ceph-mon[39562]: 8.8 scrub ok 2024-04-20T17:46:44.220 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:46:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:43 smithi106 ceph-mon[44423]: 8.e scrub ok 2024-04-20T17:46:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:43 smithi106 ceph-mon[44423]: 8.8 scrub starts 2024-04-20T17:46:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:43 smithi106 ceph-mon[44423]: 8.8 scrub ok 2024-04-20T17:46:44.262 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:45.122 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:44 smithi171 ceph-mon[44189]: pgmap v2257: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 507 op/s; 19743/165676 objects misplaced (11.917%); 9.0 MiB/s, 144 objects/s recovering 2024-04-20T17:46:45.194 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:44 smithi005 ceph-mon[39562]: pgmap v2257: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 507 op/s; 19743/165676 objects misplaced (11.917%); 9.0 MiB/s, 144 objects/s recovering 2024-04-20T17:46:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:44 smithi106 ceph-mon[44423]: pgmap v2257: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 507 op/s; 19743/165676 objects misplaced (11.917%); 9.0 MiB/s, 144 objects/s recovering 2024-04-20T17:46:45.857 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:45 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2731679800' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:46:45.857 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:45 smithi005 ceph-mon[39562]: 1.3 scrub starts 2024-04-20T17:46:45.857 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:45 smithi005 ceph-mon[39562]: 1.3 scrub ok 2024-04-20T17:46:45.865 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:46:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:45 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2731679800' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:46:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:45 smithi106 ceph-mon[44423]: 1.3 scrub starts 2024-04-20T17:46:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:45 smithi106 ceph-mon[44423]: 1.3 scrub ok 2024-04-20T17:46:46.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:45 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2731679800' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:46:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:45 smithi171 ceph-mon[44189]: 1.3 scrub starts 2024-04-20T17:46:46.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:45 smithi171 ceph-mon[44189]: 1.3 scrub ok 2024-04-20T17:46:46.354 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:46:46.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:46 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:46:46.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:46 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:46:46.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:46 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2731679800' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:46:46.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:46 smithi005 ceph-mon[39562]: osdmap e849: 12 total, 12 up, 12 in 2024-04-20T17:46:46.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:46 smithi005 ceph-mon[39562]: pgmap v2259: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 552 op/s; 18333/161698 objects misplaced (11.338%); 15 MiB/s, 234 objects/s recovering 2024-04-20T17:46:46.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:46 smithi005 ceph-mon[39562]: osdmap e850: 12 total, 12 up, 12 in 2024-04-20T17:46:47.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:46 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:46:47.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:46 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:46:47.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:46 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2731679800' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:46:47.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:46 smithi106 ceph-mon[44423]: osdmap e849: 12 total, 12 up, 12 in 2024-04-20T17:46:47.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:46 smithi106 ceph-mon[44423]: pgmap v2259: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 552 op/s; 18333/161698 objects misplaced (11.338%); 15 MiB/s, 234 objects/s recovering 2024-04-20T17:46:47.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:46 smithi106 ceph-mon[44423]: osdmap e850: 12 total, 12 up, 12 in 2024-04-20T17:46:47.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:46 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:46:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:46 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:46:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:46 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2731679800' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:46:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:46 smithi171 ceph-mon[44189]: osdmap e849: 12 total, 12 up, 12 in 2024-04-20T17:46:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:46 smithi171 ceph-mon[44189]: pgmap v2259: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 552 op/s; 18333/161698 objects misplaced (11.338%); 15 MiB/s, 234 objects/s recovering 2024-04-20T17:46:47.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:46 smithi171 ceph-mon[44189]: osdmap e850: 12 total, 12 up, 12 in 2024-04-20T17:46:48.208 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:47 smithi005 ceph-mon[39562]: 1.7 scrub starts 2024-04-20T17:46:48.208 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:47 smithi005 ceph-mon[39562]: 1.7 scrub ok 2024-04-20T17:46:48.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:47 smithi106 ceph-mon[44423]: 1.7 scrub starts 2024-04-20T17:46:48.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:47 smithi106 ceph-mon[44423]: 1.7 scrub ok 2024-04-20T17:46:48.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:47 smithi171 ceph-mon[44189]: 1.7 scrub starts 2024-04-20T17:46:48.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:47 smithi171 ceph-mon[44189]: 1.7 scrub ok 2024-04-20T17:46:48.351 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:46:48.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:48.358 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:46:48.460 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:46:48.496 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:48.914 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:48 smithi005 ceph-mon[39562]: 1.5 scrub starts 2024-04-20T17:46:48.914 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:48 smithi005 ceph-mon[39562]: pgmap v2261: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 489 op/s; 18165/160438 objects misplaced (11.322%); 12 MiB/s, 195 objects/s recovering 2024-04-20T17:46:49.053 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:46:49.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:49.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:46:49.163 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:46:49.177 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:46:49.177 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:46:49.177 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:46:49.177 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:46:49.177 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:46:49.209 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:46:49.209 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:49.242 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:46:49.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:49.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:46:49.252 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:46:49.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:48 smithi106 ceph-mon[44423]: 1.5 scrub starts 2024-04-20T17:46:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:48 smithi106 ceph-mon[44423]: pgmap v2261: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 489 op/s; 18165/160438 objects misplaced (11.322%); 12 MiB/s, 195 objects/s recovering 2024-04-20T17:46:49.278 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:46:49.314 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:46:49.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:48 smithi171 ceph-mon[44189]: 1.5 scrub starts 2024-04-20T17:46:49.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:48 smithi171 ceph-mon[44189]: pgmap v2261: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 489 op/s; 18165/160438 objects misplaced (11.322%); 12 MiB/s, 195 objects/s recovering 2024-04-20T17:46:49.337 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:46:49.351 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:46:49.362 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:46:49.378 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:49.390 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:46:49.395 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:46:49.395 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:46:49.395 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:46:49.395 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:46:49.395 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:46:49.396 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:46:49.396 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:46:49.396 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:46:49.396 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:46:49.396 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:46:49.396 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:46:49.396 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:46:49.400 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:46:49.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:49.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:46:49.504 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:46:49.597 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:50.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:49 smithi005 ceph-mon[39562]: 1.5 scrub ok 2024-04-20T17:46:50.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:49 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2692649237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:46:50.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:50.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:50.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:50.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:50.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:50.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:50.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:49 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:50.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:49 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:46:50.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:49 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:46:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:49 smithi106 ceph-mon[44423]: 1.5 scrub ok 2024-04-20T17:46:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:49 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2692649237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:46:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:50.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:49 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:50.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:49 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:46:50.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:49 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:46:50.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:49 smithi171 ceph-mon[44189]: 1.5 scrub ok 2024-04-20T17:46:50.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:49 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2692649237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:46:50.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:50.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:50.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:50.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:50.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:50.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:50.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:49 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:50.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:49 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:46:50.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:49 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:46:50.583 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:46:50.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:50.595 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:46:50.696 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:46:50.753 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:50.815 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:46:50.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:50.828 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:46:50.869 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:46:50.869 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:46:50.869 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 851, 2024-04-20T17:46:50.869 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "13c6e76c-ff37-11ee-bc93-c7b262605968", 2024-04-20T17:46:50.869 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-04-20T16:58:25.272557+0000", 2024-04-20T17:46:50.870 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-04-20T17:46:49.912966+0000", 2024-04-20T17:46:50.870 INFO:teuthology.orchestra.run.smithi005.stdout: "last_up_change": "2024-04-20T17:44:19.738797+0000", 2024-04-20T17:46:50.870 INFO:teuthology.orchestra.run.smithi005.stdout: "last_in_change": "2024-04-20T17:46:36.769424+0000", 2024-04-20T17:46:50.870 INFO:teuthology.orchestra.run.smithi005.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-20T17:46:50.870 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_num": 5799936, 2024-04-20T17:46:50.870 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_set": [ 2024-04-20T17:46:50.870 INFO:teuthology.orchestra.run.smithi005.stdout: "pglog_hardlimit", 2024-04-20T17:46:50.870 INFO:teuthology.orchestra.run.smithi005.stdout: "purged_snapdirs", 2024-04-20T17:46:50.870 INFO:teuthology.orchestra.run.smithi005.stdout: "recovery_deletes", 2024-04-20T17:46:50.870 INFO:teuthology.orchestra.run.smithi005.stdout: "sortbitwise" 2024-04-20T17:46:50.870 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:46:50.870 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_version": 33, 2024-04-20T17:46:50.871 INFO:teuthology.orchestra.run.smithi005.stdout: "full_ratio": 0.94999998807907104, 2024-04-20T17:46:50.871 INFO:teuthology.orchestra.run.smithi005.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-20T17:46:50.871 INFO:teuthology.orchestra.run.smithi005.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-20T17:46:50.871 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_snapshot": "", 2024-04-20T17:46:50.871 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_max": 8, 2024-04-20T17:46:50.871 INFO:teuthology.orchestra.run.smithi005.stdout: "max_osd": 12, 2024-04-20T17:46:50.871 INFO:teuthology.orchestra.run.smithi005.stdout: "require_min_compat_client": "luminous", 2024-04-20T17:46:50.871 INFO:teuthology.orchestra.run.smithi005.stdout: "min_compat_client": "jewel", 2024-04-20T17:46:50.871 INFO:teuthology.orchestra.run.smithi005.stdout: "require_osd_release": "squid", 2024-04-20T17:46:50.871 INFO:teuthology.orchestra.run.smithi005.stdout: "allow_crimson": false, 2024-04-20T17:46:50.871 INFO:teuthology.orchestra.run.smithi005.stdout: "pools": [ 2024-04-20T17:46:50.871 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.871 INFO:teuthology.orchestra.run.smithi005.stdout: "pool": 1, 2024-04-20T17:46:50.872 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_name": ".mgr", 2024-04-20T17:46:50.872 INFO:teuthology.orchestra.run.smithi005.stdout: "create_time": "2024-04-20T17:01:35.909493+0000", 2024-04-20T17:46:50.872 INFO:teuthology.orchestra.run.smithi005.stdout: "flags": 1, 2024-04-20T17:46:50.872 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_names": "hashpspool", 2024-04-20T17:46:50.872 INFO:teuthology.orchestra.run.smithi005.stdout: "type": 1, 2024-04-20T17:46:50.872 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 2, 2024-04-20T17:46:50.872 INFO:teuthology.orchestra.run.smithi005.stdout: "min_size": 2, 2024-04-20T17:46:50.872 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_rule": 0, 2024-04-20T17:46:50.872 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_count": 0, 2024-04-20T17:46:50.872 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_target": 0, 2024-04-20T17:46:50.872 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T17:46:50.872 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T17:46:50.872 INFO:teuthology.orchestra.run.smithi005.stdout: "object_hash": 2, 2024-04-20T17:46:50.872 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_autoscale_mode": "off", 2024-04-20T17:46:50.873 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num": 11, 2024-04-20T17:46:50.873 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num": 11, 2024-04-20T17:46:50.873 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num_target": 11, 2024-04-20T17:46:50.873 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_target": 11, 2024-04-20T17:46:50.873 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_pending": 11, 2024-04-20T17:46:50.873 INFO:teuthology.orchestra.run.smithi005.stdout: "last_pg_merge_meta": { 2024-04-20T17:46:50.873 INFO:teuthology.orchestra.run.smithi005.stdout: "source_pgid": "0.0", 2024-04-20T17:46:50.873 INFO:teuthology.orchestra.run.smithi005.stdout: "ready_epoch": 0, 2024-04-20T17:46:50.873 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_started": 0, 2024-04-20T17:46:50.873 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_clean": 0, 2024-04-20T17:46:50.873 INFO:teuthology.orchestra.run.smithi005.stdout: "source_version": "0'0", 2024-04-20T17:46:50.873 INFO:teuthology.orchestra.run.smithi005.stdout: "target_version": "0'0" 2024-04-20T17:46:50.873 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.873 INFO:teuthology.orchestra.run.smithi005.stdout: "last_change": "838", 2024-04-20T17:46:50.874 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend": "0", 2024-04-20T17:46:50.874 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-20T17:46:50.874 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_preluminous": "818", 2024-04-20T17:46:50.874 INFO:teuthology.orchestra.run.smithi005.stdout: "auid": 0, 2024-04-20T17:46:50.874 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_mode": "selfmanaged", 2024-04-20T17:46:50.874 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_seq": 0, 2024-04-20T17:46:50.874 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_epoch": 0, 2024-04-20T17:46:50.874 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_snaps": [], 2024-04-20T17:46:50.874 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_snaps": "[]", 2024-04-20T17:46:50.874 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_bytes": 0, 2024-04-20T17:46:50.874 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_objects": 0, 2024-04-20T17:46:50.875 INFO:teuthology.orchestra.run.smithi005.stdout: "tiers": [], 2024-04-20T17:46:50.875 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_of": -1, 2024-04-20T17:46:50.875 INFO:teuthology.orchestra.run.smithi005.stdout: "read_tier": -1, 2024-04-20T17:46:50.875 INFO:teuthology.orchestra.run.smithi005.stdout: "write_tier": -1, 2024-04-20T17:46:50.875 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_mode": "none", 2024-04-20T17:46:50.875 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_bytes": 0, 2024-04-20T17:46:50.875 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_objects": 0, 2024-04-20T17:46:50.875 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T17:46:50.875 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T17:46:50.875 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T17:46:50.875 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_flush_age": 0, 2024-04-20T17:46:50.875 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_evict_age": 0, 2024-04-20T17:46:50.876 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_profile": "", 2024-04-20T17:46:50.876 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_params": { 2024-04-20T17:46:50.876 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "none" 2024-04-20T17:46:50.876 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.876 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_period": 0, 2024-04-20T17:46:50.876 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_count": 0, 2024-04-20T17:46:50.876 INFO:teuthology.orchestra.run.smithi005.stdout: "use_gmt_hitset": true, 2024-04-20T17:46:50.876 INFO:teuthology.orchestra.run.smithi005.stdout: "min_read_recency_for_promote": 0, 2024-04-20T17:46:50.876 INFO:teuthology.orchestra.run.smithi005.stdout: "min_write_recency_for_promote": 0, 2024-04-20T17:46:50.876 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T17:46:50.876 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_search_last_n": 0, 2024-04-20T17:46:50.876 INFO:teuthology.orchestra.run.smithi005.stdout: "grade_table": [], 2024-04-20T17:46:50.876 INFO:teuthology.orchestra.run.smithi005.stdout: "stripe_width": 0, 2024-04-20T17:46:50.876 INFO:teuthology.orchestra.run.smithi005.stdout: "expected_num_objects": 0, 2024-04-20T17:46:50.877 INFO:teuthology.orchestra.run.smithi005.stdout: "fast_read": false, 2024-04-20T17:46:50.877 INFO:teuthology.orchestra.run.smithi005.stdout: "options": { 2024-04-20T17:46:50.877 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_max": 32, 2024-04-20T17:46:50.877 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_min": 1 2024-04-20T17:46:50.877 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.877 INFO:teuthology.orchestra.run.smithi005.stdout: "application_metadata": { 2024-04-20T17:46:50.877 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": {} 2024-04-20T17:46:50.877 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.877 INFO:teuthology.orchestra.run.smithi005.stdout: "read_balance": { 2024-04-20T17:46:50.877 INFO:teuthology.orchestra.run.smithi005.stdout: "score_type": "Fair distribution", 2024-04-20T17:46:50.877 INFO:teuthology.orchestra.run.smithi005.stdout: "score_acting": 3, 2024-04-20T17:46:50.877 INFO:teuthology.orchestra.run.smithi005.stdout: "score_stable": 3, 2024-04-20T17:46:50.877 INFO:teuthology.orchestra.run.smithi005.stdout: "optimal_score": 0.9100000262260437, 2024-04-20T17:46:50.877 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_acting": 2.7300000190734863, 2024-04-20T17:46:50.878 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_stable": 2.7300000190734863, 2024-04-20T17:46:50.878 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity_weighted": 0.83594763278961182, 2024-04-20T17:46:50.878 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity": 0.9100000262260437, 2024-04-20T17:46:50.878 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-04-20T17:46:50.878 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.878 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.878 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.878 INFO:teuthology.orchestra.run.smithi005.stdout: "pool": 8, 2024-04-20T17:46:50.878 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_name": "unique_pool_6", 2024-04-20T17:46:50.878 INFO:teuthology.orchestra.run.smithi005.stdout: "create_time": "2024-04-20T17:43:16.604151+0000", 2024-04-20T17:46:50.878 INFO:teuthology.orchestra.run.smithi005.stdout: "flags": 1, 2024-04-20T17:46:50.878 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_names": "hashpspool", 2024-04-20T17:46:50.878 INFO:teuthology.orchestra.run.smithi005.stdout: "type": 1, 2024-04-20T17:46:50.879 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 2, 2024-04-20T17:46:50.879 INFO:teuthology.orchestra.run.smithi005.stdout: "min_size": 2, 2024-04-20T17:46:50.879 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_rule": 0, 2024-04-20T17:46:50.879 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_count": 0, 2024-04-20T17:46:50.879 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_target": 0, 2024-04-20T17:46:50.879 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T17:46:50.879 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T17:46:50.879 INFO:teuthology.orchestra.run.smithi005.stdout: "object_hash": 2, 2024-04-20T17:46:50.879 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_autoscale_mode": "off", 2024-04-20T17:46:50.879 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num": 16, 2024-04-20T17:46:50.879 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num": 16, 2024-04-20T17:46:50.879 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num_target": 16, 2024-04-20T17:46:50.879 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_target": 16, 2024-04-20T17:46:50.879 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_pending": 16, 2024-04-20T17:46:50.879 INFO:teuthology.orchestra.run.smithi005.stdout: "last_pg_merge_meta": { 2024-04-20T17:46:50.879 INFO:teuthology.orchestra.run.smithi005.stdout: "source_pgid": "0.0", 2024-04-20T17:46:50.880 INFO:teuthology.orchestra.run.smithi005.stdout: "ready_epoch": 0, 2024-04-20T17:46:50.880 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_started": 0, 2024-04-20T17:46:50.880 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_clean": 0, 2024-04-20T17:46:50.880 INFO:teuthology.orchestra.run.smithi005.stdout: "source_version": "0'0", 2024-04-20T17:46:50.880 INFO:teuthology.orchestra.run.smithi005.stdout: "target_version": "0'0" 2024-04-20T17:46:50.880 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.880 INFO:teuthology.orchestra.run.smithi005.stdout: "last_change": "782", 2024-04-20T17:46:50.880 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend": "0", 2024-04-20T17:46:50.880 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-20T17:46:50.880 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_preluminous": "0", 2024-04-20T17:46:50.880 INFO:teuthology.orchestra.run.smithi005.stdout: "auid": 0, 2024-04-20T17:46:50.880 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_mode": "selfmanaged", 2024-04-20T17:46:50.880 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_seq": 0, 2024-04-20T17:46:50.881 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_epoch": 0, 2024-04-20T17:46:50.881 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_snaps": [], 2024-04-20T17:46:50.881 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_snaps": "[]", 2024-04-20T17:46:50.881 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_bytes": 0, 2024-04-20T17:46:50.881 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_objects": 0, 2024-04-20T17:46:50.881 INFO:teuthology.orchestra.run.smithi005.stdout: "tiers": [], 2024-04-20T17:46:50.881 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_of": -1, 2024-04-20T17:46:50.881 INFO:teuthology.orchestra.run.smithi005.stdout: "read_tier": -1, 2024-04-20T17:46:50.881 INFO:teuthology.orchestra.run.smithi005.stdout: "write_tier": -1, 2024-04-20T17:46:50.881 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_mode": "none", 2024-04-20T17:46:50.881 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_bytes": 0, 2024-04-20T17:46:50.881 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_objects": 0, 2024-04-20T17:46:50.881 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T17:46:50.881 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T17:46:50.882 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T17:46:50.882 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_flush_age": 0, 2024-04-20T17:46:50.882 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_evict_age": 0, 2024-04-20T17:46:50.882 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_profile": "", 2024-04-20T17:46:50.882 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_params": { 2024-04-20T17:46:50.882 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "none" 2024-04-20T17:46:50.882 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.882 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_period": 0, 2024-04-20T17:46:50.882 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_count": 0, 2024-04-20T17:46:50.882 INFO:teuthology.orchestra.run.smithi005.stdout: "use_gmt_hitset": true, 2024-04-20T17:46:50.882 INFO:teuthology.orchestra.run.smithi005.stdout: "min_read_recency_for_promote": 0, 2024-04-20T17:46:50.882 INFO:teuthology.orchestra.run.smithi005.stdout: "min_write_recency_for_promote": 0, 2024-04-20T17:46:50.882 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T17:46:50.883 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_search_last_n": 0, 2024-04-20T17:46:50.883 INFO:teuthology.orchestra.run.smithi005.stdout: "grade_table": [], 2024-04-20T17:46:50.883 INFO:teuthology.orchestra.run.smithi005.stdout: "stripe_width": 0, 2024-04-20T17:46:50.883 INFO:teuthology.orchestra.run.smithi005.stdout: "expected_num_objects": 0, 2024-04-20T17:46:50.883 INFO:teuthology.orchestra.run.smithi005.stdout: "fast_read": false, 2024-04-20T17:46:50.883 INFO:teuthology.orchestra.run.smithi005.stdout: "options": {}, 2024-04-20T17:46:50.883 INFO:teuthology.orchestra.run.smithi005.stdout: "application_metadata": { 2024-04-20T17:46:50.883 INFO:teuthology.orchestra.run.smithi005.stdout: "rados": {} 2024-04-20T17:46:50.883 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.883 INFO:teuthology.orchestra.run.smithi005.stdout: "read_balance": { 2024-04-20T17:46:50.883 INFO:teuthology.orchestra.run.smithi005.stdout: "score_type": "Fair distribution", 2024-04-20T17:46:50.883 INFO:teuthology.orchestra.run.smithi005.stdout: "score_acting": 3.440000057220459, 2024-04-20T17:46:50.883 INFO:teuthology.orchestra.run.smithi005.stdout: "score_stable": 3.440000057220459, 2024-04-20T17:46:50.883 INFO:teuthology.orchestra.run.smithi005.stdout: "optimal_score": 1, 2024-04-20T17:46:50.883 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_acting": 3.440000057220459, 2024-04-20T17:46:50.884 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_stable": 3.440000057220459, 2024-04-20T17:46:50.884 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity_weighted": 0.83594763278961182, 2024-04-20T17:46:50.884 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity": 0.9100000262260437, 2024-04-20T17:46:50.884 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-04-20T17:46:50.884 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.884 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.884 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:46:50.884 INFO:teuthology.orchestra.run.smithi005.stdout: "osds": [ 2024-04-20T17:46:50.884 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.884 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 0, 2024-04-20T17:46:50.884 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b", 2024-04-20T17:46:50.884 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:46:50.885 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:46:50.885 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:46:50.885 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:46:50.885 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:46:50.885 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:46:50.885 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 8, 2024-04-20T17:46:50.885 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 845, 2024-04-20T17:46:50.885 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:46:50.885 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:46:50.885 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:46:50.885 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.885 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.885 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.885 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6801", 2024-04-20T17:46:50.886 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:46:50.886 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.886 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.886 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.886 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:46:50.886 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.886 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.886 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.886 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6802", 2024-04-20T17:46:50.886 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:46:50.886 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.886 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.886 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.886 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:46:50.887 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.887 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.887 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.887 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6804", 2024-04-20T17:46:50.887 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:46:50.887 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.887 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.887 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.887 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:46:50.887 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.887 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.887 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.887 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6803", 2024-04-20T17:46:50.888 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:46:50.888 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.888 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.888 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.888 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6801/3901484705", 2024-04-20T17:46:50.888 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6802/3901484705", 2024-04-20T17:46:50.888 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6804/3901484705", 2024-04-20T17:46:50.888 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6803/3901484705", 2024-04-20T17:46:50.888 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:46:50.888 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:46:50.888 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:46:50.889 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.889 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.889 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.889 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 1, 2024-04-20T17:46:50.889 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "aed0c8eb-0e81-4e85-ab8f-a60420060656", 2024-04-20T17:46:50.889 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:46:50.889 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:46:50.889 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:46:50.889 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 0.0313720703125, 2024-04-20T17:46:50.889 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 13, 2024-04-20T17:46:50.889 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 613, 2024-04-20T17:46:50.889 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 659, 2024-04-20T17:46:50.889 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 539, 2024-04-20T17:46:50.889 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 614, 2024-04-20T17:46:50.890 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:46:50.890 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:46:50.890 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.890 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.890 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.890 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6805", 2024-04-20T17:46:50.890 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2415652792 2024-04-20T17:46:50.890 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.890 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.890 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.890 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:46:50.890 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.890 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.890 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.891 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6806", 2024-04-20T17:46:50.891 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2415652792 2024-04-20T17:46:50.891 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.891 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.891 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.891 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:46:50.891 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.891 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.891 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.891 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6808", 2024-04-20T17:46:50.891 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2415652792 2024-04-20T17:46:50.891 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.892 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.892 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.892 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:46:50.892 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.892 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.892 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.892 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6807", 2024-04-20T17:46:50.892 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2415652792 2024-04-20T17:46:50.892 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.892 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.892 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.892 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6805/2415652792", 2024-04-20T17:46:50.893 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6806/2415652792", 2024-04-20T17:46:50.893 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6808/2415652792", 2024-04-20T17:46:50.893 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6807/2415652792", 2024-04-20T17:46:50.893 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:46:50.893 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:46:50.893 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:46:50.893 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.893 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.893 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.893 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 2, 2024-04-20T17:46:50.893 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "f733a6a2-19c2-4c7a-af9c-04e282062bb7", 2024-04-20T17:46:50.893 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:46:50.894 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:46:50.894 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:46:50.894 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:46:50.894 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 20, 2024-04-20T17:46:50.894 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 169, 2024-04-20T17:46:50.894 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 202, 2024-04-20T17:46:50.894 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 845, 2024-04-20T17:46:50.894 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 170, 2024-04-20T17:46:50.894 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:46:50.894 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:46:50.894 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.894 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.895 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.895 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6809", 2024-04-20T17:46:50.895 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2120275501 2024-04-20T17:46:50.895 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.895 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.895 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.895 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:46:50.895 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.895 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.895 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.895 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6810", 2024-04-20T17:46:50.895 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2120275501 2024-04-20T17:46:50.895 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.896 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.896 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.896 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:46:50.896 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.896 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.896 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.896 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6812", 2024-04-20T17:46:50.896 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2120275501 2024-04-20T17:46:50.896 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.896 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.896 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.896 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:46:50.896 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.896 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.897 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.897 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6811", 2024-04-20T17:46:50.897 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2120275501 2024-04-20T17:46:50.897 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.897 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.897 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.897 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6809/2120275501", 2024-04-20T17:46:50.897 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6810/2120275501", 2024-04-20T17:46:50.897 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6812/2120275501", 2024-04-20T17:46:50.897 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6811/2120275501", 2024-04-20T17:46:50.897 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:46:50.897 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:46:50.897 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:46:50.898 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.898 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.898 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.898 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 3, 2024-04-20T17:46:50.898 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "8c92f1df-e298-4606-88ae-b8a8e79628a1", 2024-04-20T17:46:50.898 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:46:50.898 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:46:50.898 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:46:50.898 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:46:50.898 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 25, 2024-04-20T17:46:50.898 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 538, 2024-04-20T17:46:50.898 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 561, 2024-04-20T17:46:50.898 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 777, 2024-04-20T17:46:50.898 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 539, 2024-04-20T17:46:50.899 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:46:50.899 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:46:50.899 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.899 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.899 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.899 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6813", 2024-04-20T17:46:50.899 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2272824046 2024-04-20T17:46:50.899 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.899 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.899 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.899 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:46:50.899 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.899 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.899 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.899 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6814", 2024-04-20T17:46:50.900 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2272824046 2024-04-20T17:46:50.900 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.900 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.900 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.900 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:46:50.900 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.900 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.900 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.900 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6816", 2024-04-20T17:46:50.900 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2272824046 2024-04-20T17:46:50.900 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.900 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.901 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.901 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:46:50.901 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.901 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.901 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.901 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6815", 2024-04-20T17:46:50.901 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2272824046 2024-04-20T17:46:50.901 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.901 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.901 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.901 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6813/2272824046", 2024-04-20T17:46:50.901 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6814/2272824046", 2024-04-20T17:46:50.901 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6816/2272824046", 2024-04-20T17:46:50.902 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6815/2272824046", 2024-04-20T17:46:50.902 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:46:50.902 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:46:50.902 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:46:50.902 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.902 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.902 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.902 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 4, 2024-04-20T17:46:50.902 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "46df16f1-9607-4d7b-9be8-9d4236e160d1", 2024-04-20T17:46:50.902 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:46:50.902 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:46:50.902 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:46:50.902 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:46:50.902 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:46:50.903 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:46:50.903 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 31, 2024-04-20T17:46:50.903 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 694, 2024-04-20T17:46:50.903 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:46:50.903 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:46:50.903 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:46:50.903 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.903 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.903 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.903 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6800", 2024-04-20T17:46:50.903 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:46:50.903 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.903 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.903 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.903 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:46:50.904 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.904 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.904 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.904 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6801", 2024-04-20T17:46:50.904 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:46:50.904 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.904 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.904 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.904 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:46:50.904 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.904 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.904 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.905 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6803", 2024-04-20T17:46:50.905 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:46:50.905 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.905 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.905 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.905 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:46:50.905 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.905 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.905 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.905 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6802", 2024-04-20T17:46:50.905 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:46:50.905 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.905 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.905 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.905 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6800/1839524469", 2024-04-20T17:46:50.906 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6801/1839524469", 2024-04-20T17:46:50.906 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6803/1839524469", 2024-04-20T17:46:50.906 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6802/1839524469", 2024-04-20T17:46:50.906 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:46:50.906 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:46:50.906 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:46:50.906 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.906 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.906 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.906 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 5, 2024-04-20T17:46:50.906 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "6603231d-1e5c-4575-bfaa-2eeca2c12215", 2024-04-20T17:46:50.906 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:46:50.906 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:46:50.907 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:46:50.907 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 0, 2024-04-20T17:46:50.907 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:46:50.907 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:46:50.907 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 37, 2024-04-20T17:46:50.907 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 619, 2024-04-20T17:46:50.907 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:46:50.907 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:46:50.907 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:46:50.907 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.907 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.907 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.907 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6804", 2024-04-20T17:46:50.907 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:46:50.907 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.908 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.908 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.908 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:46:50.908 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.908 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.908 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.908 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6805", 2024-04-20T17:46:50.908 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:46:50.908 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.908 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.908 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.908 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:46:50.908 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.908 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.908 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.909 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6807", 2024-04-20T17:46:50.909 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:46:50.909 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.909 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.909 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.909 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:46:50.909 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.909 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.909 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.909 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6806", 2024-04-20T17:46:50.909 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:46:50.909 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.909 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.909 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.909 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6804/2450315908", 2024-04-20T17:46:50.910 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6805/2450315908", 2024-04-20T17:46:50.910 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6807/2450315908", 2024-04-20T17:46:50.910 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6806/2450315908", 2024-04-20T17:46:50.910 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:46:50.910 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:46:50.910 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:46:50.910 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.910 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.910 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.910 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 6, 2024-04-20T17:46:50.910 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "783a55a6-f176-42be-8c8b-af8481d08dc9", 2024-04-20T17:46:50.910 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:46:50.911 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:46:50.911 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:46:50.911 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:46:50.911 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 48, 2024-04-20T17:46:50.911 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 503, 2024-04-20T17:46:50.911 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 535, 2024-04-20T17:46:50.911 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 845, 2024-04-20T17:46:50.911 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 504, 2024-04-20T17:46:50.911 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:46:50.911 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:46:50.911 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.911 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.911 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.911 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6808", 2024-04-20T17:46:50.911 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2439382182 2024-04-20T17:46:50.912 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.912 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.912 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.912 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:46:50.912 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.912 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.912 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.912 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6809", 2024-04-20T17:46:50.912 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2439382182 2024-04-20T17:46:50.912 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.912 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.912 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.912 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:46:50.912 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.913 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.913 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.913 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6811", 2024-04-20T17:46:50.913 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2439382182 2024-04-20T17:46:50.913 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.913 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.913 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.913 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:46:50.913 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.913 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.913 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.913 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6810", 2024-04-20T17:46:50.913 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2439382182 2024-04-20T17:46:50.914 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.914 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.914 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.914 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6808/2439382182", 2024-04-20T17:46:50.914 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6809/2439382182", 2024-04-20T17:46:50.914 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6811/2439382182", 2024-04-20T17:46:50.914 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6810/2439382182", 2024-04-20T17:46:50.914 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:46:50.914 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:46:50.914 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:46:50.914 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.914 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.914 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.914 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 7, 2024-04-20T17:46:50.914 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4", 2024-04-20T17:46:50.915 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:46:50.915 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:46:50.915 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:46:50.915 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:46:50.915 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 398, 2024-04-20T17:46:50.915 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 721, 2024-04-20T17:46:50.915 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 799, 2024-04-20T17:46:50.915 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 846, 2024-04-20T17:46:50.915 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 722, 2024-04-20T17:46:50.915 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:46:50.915 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:46:50.916 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.916 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.916 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.916 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6812", 2024-04-20T17:46:50.916 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2531699401 2024-04-20T17:46:50.916 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.916 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.916 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.916 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:46:50.916 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.916 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.916 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.916 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6813", 2024-04-20T17:46:50.916 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2531699401 2024-04-20T17:46:50.916 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.916 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.917 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.917 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:46:50.917 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.917 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.917 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.917 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6815", 2024-04-20T17:46:50.917 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2531699401 2024-04-20T17:46:50.917 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.917 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.917 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.917 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:46:50.917 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.917 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.917 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.917 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6814", 2024-04-20T17:46:50.918 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2531699401 2024-04-20T17:46:50.918 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.918 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.918 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.918 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6812/2531699401", 2024-04-20T17:46:50.918 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6813/2531699401", 2024-04-20T17:46:50.918 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6815/2531699401", 2024-04-20T17:46:50.918 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6814/2531699401", 2024-04-20T17:46:50.918 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:46:50.918 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:46:50.918 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:46:50.918 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.918 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.918 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.918 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 8, 2024-04-20T17:46:50.919 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "1cb232ad-1f59-4a0a-a215-e67419ad6476", 2024-04-20T17:46:50.919 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:46:50.919 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:46:50.919 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:46:50.919 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:46:50.919 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:46:50.919 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:46:50.919 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 67, 2024-04-20T17:46:50.919 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 777, 2024-04-20T17:46:50.919 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:46:50.919 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:46:50.919 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:46:50.919 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.919 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.919 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.919 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6800", 2024-04-20T17:46:50.920 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:46:50.920 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.920 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.920 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.920 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:46:50.920 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.920 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.920 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.920 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6801", 2024-04-20T17:46:50.920 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:46:50.920 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.920 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.920 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.920 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:46:50.920 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.921 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.921 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.921 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6803", 2024-04-20T17:46:50.921 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:46:50.921 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.921 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.921 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.921 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:46:50.921 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.921 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.921 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.921 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6802", 2024-04-20T17:46:50.921 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:46:50.921 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.921 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.922 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.922 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6800/4123058942", 2024-04-20T17:46:50.922 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6801/4123058942", 2024-04-20T17:46:50.922 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6803/4123058942", 2024-04-20T17:46:50.922 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6802/4123058942", 2024-04-20T17:46:50.922 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:46:50.922 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:46:50.922 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:46:50.922 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.922 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.922 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.922 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 9, 2024-04-20T17:46:50.922 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "b6c17147-b3dc-475b-bb20-f9f712fb99e8", 2024-04-20T17:46:50.922 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:46:50.923 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:46:50.923 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:46:50.923 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:46:50.923 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:46:50.923 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:46:50.923 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 73, 2024-04-20T17:46:50.923 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 823, 2024-04-20T17:46:50.923 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:46:50.923 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:46:50.923 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:46:50.923 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.923 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.923 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.924 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6804", 2024-04-20T17:46:50.924 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:46:50.924 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.924 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.924 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.924 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:46:50.924 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.924 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.924 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.924 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6805", 2024-04-20T17:46:50.924 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:46:50.924 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.924 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.924 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.924 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:46:50.925 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.925 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.925 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.925 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6807", 2024-04-20T17:46:50.925 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:46:50.925 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.925 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.925 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.925 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:46:50.925 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.925 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.925 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.925 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6806", 2024-04-20T17:46:50.925 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:46:50.926 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.926 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.926 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.926 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6804/2877589636", 2024-04-20T17:46:50.926 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6805/2877589636", 2024-04-20T17:46:50.926 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6807/2877589636", 2024-04-20T17:46:50.926 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6806/2877589636", 2024-04-20T17:46:50.926 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:46:50.926 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:46:50.926 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:46:50.926 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.926 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.926 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.927 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 10, 2024-04-20T17:46:50.927 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "1adcdf55-f002-4a39-91e7-378e5fae849c", 2024-04-20T17:46:50.927 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:46:50.927 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:46:50.927 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:46:50.927 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:46:50.927 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:46:50.927 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:46:50.927 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 84, 2024-04-20T17:46:50.927 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 823, 2024-04-20T17:46:50.927 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:46:50.927 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:46:50.927 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:46:50.928 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.928 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.928 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.928 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6808", 2024-04-20T17:46:50.928 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:46:50.928 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.928 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.928 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.928 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:46:50.928 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.928 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.928 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.928 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6809", 2024-04-20T17:46:50.928 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:46:50.928 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.929 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.929 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.929 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:46:50.929 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.929 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.929 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.929 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6811", 2024-04-20T17:46:50.929 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:46:50.929 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.929 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.929 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.929 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:46:50.930 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.930 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.930 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.930 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6810", 2024-04-20T17:46:50.930 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:46:50.930 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.930 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.930 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.930 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6808/181490621", 2024-04-20T17:46:50.930 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6809/181490621", 2024-04-20T17:46:50.930 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6811/181490621", 2024-04-20T17:46:50.930 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6810/181490621", 2024-04-20T17:46:50.930 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:46:50.930 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:46:50.931 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:46:50.931 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.931 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.931 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.931 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 11, 2024-04-20T17:46:50.931 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "737c5aec-694a-4968-819d-fccf22661f4f", 2024-04-20T17:46:50.931 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:46:50.931 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:46:50.931 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:46:50.931 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:46:50.931 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:46:50.931 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:46:50.931 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 92, 2024-04-20T17:46:50.931 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 844, 2024-04-20T17:46:50.932 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:46:50.932 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:46:50.932 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:46:50.932 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.932 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.932 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.932 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6812", 2024-04-20T17:46:50.932 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:46:50.932 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.932 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.932 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.932 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:46:50.932 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.932 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.932 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.933 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6813", 2024-04-20T17:46:50.933 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:46:50.933 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.933 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.933 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.933 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:46:50.933 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.933 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.933 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.933 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6815", 2024-04-20T17:46:50.933 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:46:50.933 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.933 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.934 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.934 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:46:50.934 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:46:50.934 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.934 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:46:50.934 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6814", 2024-04-20T17:46:50.934 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:46:50.934 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.934 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.934 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.934 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6812/3373585701", 2024-04-20T17:46:50.934 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6813/3373585701", 2024-04-20T17:46:50.934 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6815/3373585701", 2024-04-20T17:46:50.934 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6814/3373585701", 2024-04-20T17:46:50.935 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:46:50.935 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:46:50.935 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:46:50.935 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.935 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.935 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:46:50.935 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_xinfo": [ 2024-04-20T17:46:50.935 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.935 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 0, 2024-04-20T17:46:50.935 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:46:50.935 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:46:50.935 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:46:50.935 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:46:50.935 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:46:50.936 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:46:11.567649+0000", 2024-04-20T17:46:50.936 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:46:50.936 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.936 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.936 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 1, 2024-04-20T17:46:50.936 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:46:50.936 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:46:50.936 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:46:50.936 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:46:50.936 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:46:50.936 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:45:58.488187+0000", 2024-04-20T17:46:50.936 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 613 2024-04-20T17:46:50.936 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.936 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.936 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 2, 2024-04-20T17:46:50.937 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:46:50.937 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:46:50.937 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:46:50.937 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:46:50.937 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:46:50.937 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:46:37.512873+0000", 2024-04-20T17:46:50.937 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 169 2024-04-20T17:46:50.937 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.937 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.937 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 3, 2024-04-20T17:46:50.937 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:46:50.937 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:46:50.937 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:46:50.937 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:46:50.937 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:46:50.937 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:46:12.819884+0000", 2024-04-20T17:46:50.938 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 538 2024-04-20T17:46:50.938 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.938 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.938 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 4, 2024-04-20T17:46:50.938 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:46:50.938 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:46:50.938 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:46:50.938 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:46:50.938 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:46:50.938 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:46:49.692242+0000", 2024-04-20T17:46:50.938 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:46:50.938 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.938 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.938 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 5, 2024-04-20T17:46:50.938 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:46:50.938 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:46:50.939 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:46:50.939 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:46:50.939 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:46:50.939 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:46:39.754243+0000", 2024-04-20T17:46:50.939 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:46:50.939 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.939 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.939 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 6, 2024-04-20T17:46:50.939 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:46:50.939 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:46:50.939 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:46:50.939 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:46:50.939 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:46:50.939 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:45:26.351834+0000", 2024-04-20T17:46:50.939 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 503 2024-04-20T17:46:50.940 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.940 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.940 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 7, 2024-04-20T17:46:50.940 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:46:50.940 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:46:50.940 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:46:50.940 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:46:50.940 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:46:50.940 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:45:29.435677+0000", 2024-04-20T17:46:50.940 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 721 2024-04-20T17:46:50.940 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.940 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.940 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 8, 2024-04-20T17:46:50.941 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:46:50.941 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:46:50.941 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:46:50.941 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:46:50.941 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:46:50.941 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:46:18.755610+0000", 2024-04-20T17:46:50.941 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:46:50.941 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.941 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.941 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 9, 2024-04-20T17:46:50.941 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:46:50.941 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:46:50.941 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:46:50.941 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:46:50.941 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:46:50.941 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:46:36.662481+0000", 2024-04-20T17:46:50.942 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:46:50.942 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.942 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.942 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 10, 2024-04-20T17:46:50.942 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:46:50.942 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:46:50.942 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:46:50.942 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:46:50.942 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:46:50.942 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:46:16.675371+0000", 2024-04-20T17:46:50.942 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:46:50.942 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.942 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.943 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 11, 2024-04-20T17:46:50.943 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:46:50.943 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:46:50.943 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:46:50.943 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:46:50.943 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:46:50.943 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:46:32.230346+0000", 2024-04-20T17:46:50.943 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:46:50.943 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.943 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:46:50.943 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_upmap": [], 2024-04-20T17:46:50.943 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_upmap_items": [], 2024-04-20T17:46:50.943 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_upmap_primaries": [], 2024-04-20T17:46:50.943 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_temp": [ 2024-04-20T17:46:50.944 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:46:50.944 INFO:teuthology.orchestra.run.smithi005.stdout: "pgid": "8.d", 2024-04-20T17:46:50.944 INFO:teuthology.orchestra.run.smithi005.stdout: "osds": [ 2024-04-20T17:46:50.944 INFO:teuthology.orchestra.run.smithi005.stdout: 11, 2024-04-20T17:46:50.944 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-04-20T17:46:50.944 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:46:50.944 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.944 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:46:50.944 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_temp": [], 2024-04-20T17:46:50.944 INFO:teuthology.orchestra.run.smithi005.stdout: "blocklist": { 2024-04-20T17:46:50.944 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/3881726425": "2024-04-21T16:59:25.808337+0000", 2024-04-20T17:46:50.944 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/338649532": "2024-04-21T16:59:25.808337+0000", 2024-04-20T17:46:50.944 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/1227843664": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:46:50.944 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:6800/1876064631": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:46:50.945 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/160689449": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:46:50.945 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:6800/3521715286": "2024-04-21T16:59:25.808337+0000", 2024-04-20T17:46:50.945 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/3841663462": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:46:50.945 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/687690919": "2024-04-21T16:59:25.808337+0000" 2024-04-20T17:46:50.945 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.945 INFO:teuthology.orchestra.run.smithi005.stdout: "range_blocklist": {}, 2024-04-20T17:46:50.945 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_profiles": { 2024-04-20T17:46:50.945 INFO:teuthology.orchestra.run.smithi005.stdout: "default": { 2024-04-20T17:46:50.945 INFO:teuthology.orchestra.run.smithi005.stdout: "crush-failure-domain": "osd", 2024-04-20T17:46:50.945 INFO:teuthology.orchestra.run.smithi005.stdout: "k": "2", 2024-04-20T17:46:50.945 INFO:teuthology.orchestra.run.smithi005.stdout: "m": "1", 2024-04-20T17:46:50.945 INFO:teuthology.orchestra.run.smithi005.stdout: "plugin": "jerasure", 2024-04-20T17:46:50.945 INFO:teuthology.orchestra.run.smithi005.stdout: "technique": "reed_sol_van" 2024-04-20T17:46:50.945 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.946 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:46:50.946 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_snaps_queue": [], 2024-04-20T17:46:50.946 INFO:teuthology.orchestra.run.smithi005.stdout: "new_removed_snaps": [], 2024-04-20T17:46:50.946 INFO:teuthology.orchestra.run.smithi005.stdout: "new_purged_snaps": [], 2024-04-20T17:46:50.946 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_node_flags": {}, 2024-04-20T17:46:50.946 INFO:teuthology.orchestra.run.smithi005.stdout: "device_class_flags": {}, 2024-04-20T17:46:50.946 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": { 2024-04-20T17:46:50.946 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode_enabled": false, 2024-04-20T17:46:50.946 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_bucket_count": 0, 2024-04-20T17:46:50.946 INFO:teuthology.orchestra.run.smithi005.stdout: "degraded_stretch_mode": 0, 2024-04-20T17:46:50.946 INFO:teuthology.orchestra.run.smithi005.stdout: "recovering_stretch_mode": 0, 2024-04-20T17:46:50.946 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode_bucket": 0 2024-04-20T17:46:50.946 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:46:50.946 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:46:50.947 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:46:51.066 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:46:51.076 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:51.169 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:46:51.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:51.174 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:46:51.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:51.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:51.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:51.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:51.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:51.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:50 smithi005 ceph-mon[39562]: osdmap e851: 12 total, 12 up, 12 in 2024-04-20T17:46:51.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:50 smithi005 ceph-mon[39562]: pgmap v2263: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 838 op/s; 16591/155614 objects misplaced (10.662%); 29 MiB/s, 464 objects/s recovering 2024-04-20T17:46:51.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:50 smithi005 ceph-mon[39562]: 1.1 scrub starts 2024-04-20T17:46:51.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:50 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1126951718' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-20T17:46:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:50 smithi106 ceph-mon[44423]: osdmap e851: 12 total, 12 up, 12 in 2024-04-20T17:46:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:50 smithi106 ceph-mon[44423]: pgmap v2263: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 838 op/s; 16591/155614 objects misplaced (10.662%); 29 MiB/s, 464 objects/s recovering 2024-04-20T17:46:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:50 smithi106 ceph-mon[44423]: 1.1 scrub starts 2024-04-20T17:46:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:50 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1126951718' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-20T17:46:51.275 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:46:51.301 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:51.320 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:46:51.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:51.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:46:51.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:51.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:51.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:51.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:50 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:50 smithi171 ceph-mon[44189]: osdmap e851: 12 total, 12 up, 12 in 2024-04-20T17:46:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:50 smithi171 ceph-mon[44189]: pgmap v2263: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 838 op/s; 16591/155614 objects misplaced (10.662%); 29 MiB/s, 464 objects/s recovering 2024-04-20T17:46:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:50 smithi171 ceph-mon[44189]: 1.1 scrub starts 2024-04-20T17:46:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:50 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1126951718' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-20T17:46:51.424 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:46:51.529 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:51.575 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:46:51.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:51.586 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:46:51.607 INFO:tasks.thrashosds.thrasher:j is {'epoch': 851, 'fsid': '13c6e76c-ff37-11ee-bc93-c7b262605968', 'created': '2024-04-20T16:58:25.272557+0000', 'modified': '2024-04-20T17:46:49.912966+0000', 'last_up_change': '2024-04-20T17:44:19.738797+0000', 'last_in_change': '2024-04-20T17:46:36.769424+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 33, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-20T17:01:35.909493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '838', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '818', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 0.9100000262260437, 'raw_score_acting': 2.7300000190734863, 'raw_score_stable': 2.7300000190734863, 'primary_affinity_weighted': 0.8359476327896118, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 8, 'pool_name': 'unique_pool_6', 'create_time': '2024-04-20T17:43:16.604151+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '782', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.440000057220459, 'score_stable': 3.440000057220459, 'optimal_score': 1, 'raw_score_acting': 3.440000057220459, 'raw_score_stable': 3.440000057220459, 'primary_affinity_weighted': 0.8359476327896118, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.8399999737739563}}], 'osds': [{'osd': 0, 'uuid': 'ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 845, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6801', 'nonce': 3901484705}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6802', 'nonce': 3901484705}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6804', 'nonce': 3901484705}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6803', 'nonce': 3901484705}]}, 'public_addr': '172.21.15.5:6801/3901484705', 'cluster_addr': '172.21.15.5:6802/3901484705', 'heartbeat_back_addr': '172.21.15.5:6804/3901484705', 'heartbeat_front_addr': '172.21.15.5:6803/3901484705', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'aed0c8eb-0e81-4e85-ab8f-a60420060656', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.0313720703125, 'last_clean_begin': 13, 'last_clean_end': 613, 'up_from': 659, 'up_thru': 539, 'down_at': 614, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6805', 'nonce': 2415652792}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6806', 'nonce': 2415652792}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6808', 'nonce': 2415652792}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6807', 'nonce': 2415652792}]}, 'public_addr': '172.21.15.5:6805/2415652792', 'cluster_addr': '172.21.15.5:6806/2415652792', 'heartbeat_back_addr': '172.21.15.5:6808/2415652792', 'heartbeat_front_addr': '172.21.15.5:6807/2415652792', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f733a6a2-19c2-4c7a-af9c-04e282062bb7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 20, 'last_clean_end': 169, 'up_from': 202, 'up_thru': 845, 'down_at': 170, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6809', 'nonce': 2120275501}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6810', 'nonce': 2120275501}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6812', 'nonce': 2120275501}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6811', 'nonce': 2120275501}]}, 'public_addr': '172.21.15.5:6809/2120275501', 'cluster_addr': '172.21.15.5:6810/2120275501', 'heartbeat_back_addr': '172.21.15.5:6812/2120275501', 'heartbeat_front_addr': '172.21.15.5:6811/2120275501', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '8c92f1df-e298-4606-88ae-b8a8e79628a1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 25, 'last_clean_end': 538, 'up_from': 561, 'up_thru': 777, 'down_at': 539, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6813', 'nonce': 2272824046}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6814', 'nonce': 2272824046}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6816', 'nonce': 2272824046}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6815', 'nonce': 2272824046}]}, 'public_addr': '172.21.15.5:6813/2272824046', 'cluster_addr': '172.21.15.5:6814/2272824046', 'heartbeat_back_addr': '172.21.15.5:6816/2272824046', 'heartbeat_front_addr': '172.21.15.5:6815/2272824046', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '46df16f1-9607-4d7b-9be8-9d4236e160d1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 694, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6800', 'nonce': 1839524469}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6801', 'nonce': 1839524469}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6803', 'nonce': 1839524469}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6802', 'nonce': 1839524469}]}, 'public_addr': '172.21.15.106:6800/1839524469', 'cluster_addr': '172.21.15.106:6801/1839524469', 'heartbeat_back_addr': '172.21.15.106:6803/1839524469', 'heartbeat_front_addr': '172.21.15.106:6802/1839524469', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6603231d-1e5c-4575-bfaa-2eeca2c12215', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 619, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6804', 'nonce': 2450315908}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6805', 'nonce': 2450315908}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6807', 'nonce': 2450315908}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6806', 'nonce': 2450315908}]}, 'public_addr': '172.21.15.106:6804/2450315908', 'cluster_addr': '172.21.15.106:6805/2450315908', 'heartbeat_back_addr': '172.21.15.106:6807/2450315908', 'heartbeat_front_addr': '172.21.15.106:6806/2450315908', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '783a55a6-f176-42be-8c8b-af8481d08dc9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 48, 'last_clean_end': 503, 'up_from': 535, 'up_thru': 845, 'down_at': 504, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6808', 'nonce': 2439382182}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6809', 'nonce': 2439382182}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6811', 'nonce': 2439382182}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6810', 'nonce': 2439382182}]}, 'public_addr': '172.21.15.106:6808/2439382182', 'cluster_addr': '172.21.15.106:6809/2439382182', 'heartbeat_back_addr': '172.21.15.106:6811/2439382182', 'heartbeat_front_addr': '172.21.15.106:6810/2439382182', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 398, 'last_clean_end': 721, 'up_from': 799, 'up_thru': 846, 'down_at': 722, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6812', 'nonce': 2531699401}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6813', 'nonce': 2531699401}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6815', 'nonce': 2531699401}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6814', 'nonce': 2531699401}]}, 'public_addr': '172.21.15.106:6812/2531699401', 'cluster_addr': '172.21.15.106:6813/2531699401', 'heartbeat_back_addr': '172.21.15.106:6815/2531699401', 'heartbeat_front_addr': '172.21.15.106:6814/2531699401', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '1cb232ad-1f59-4a0a-a215-e67419ad6476', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 67, 'up_thru': 777, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6800', 'nonce': 4123058942}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6801', 'nonce': 4123058942}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6803', 'nonce': 4123058942}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6802', 'nonce': 4123058942}]}, 'public_addr': '172.21.15.171:6800/4123058942', 'cluster_addr': '172.21.15.171:6801/4123058942', 'heartbeat_back_addr': '172.21.15.171:6803/4123058942', 'heartbeat_front_addr': '172.21.15.171:6802/4123058942', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'b6c17147-b3dc-475b-bb20-f9f712fb99e8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 73, 'up_thru': 823, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6804', 'nonce': 2877589636}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6805', 'nonce': 2877589636}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6807', 'nonce': 2877589636}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6806', 'nonce': 2877589636}]}, 'public_addr': '172.21.15.171:6804/2877589636', 'cluster_addr': '172.21.15.171:6805/2877589636', 'heartbeat_back_addr': '172.21.15.171:6807/2877589636', 'heartbeat_front_addr': '172.21.15.171:6806/2877589636', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '1adcdf55-f002-4a39-91e7-378e5fae849c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 84, 'up_thru': 823, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6808', 'nonce': 181490621}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6809', 'nonce': 181490621}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6811', 'nonce': 181490621}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6810', 'nonce': 181490621}]}, 'public_addr': '172.21.15.171:6808/181490621', 'cluster_addr': '172.21.15.171:6809/181490621', 'heartbeat_back_addr': '172.21.15.171:6811/181490621', 'heartbeat_front_addr': '172.21.15.171:6810/181490621', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '737c5aec-694a-4968-819d-fccf22661f4f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 92, 'up_thru': 844, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6812', 'nonce': 3373585701}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6813', 'nonce': 3373585701}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6815', 'nonce': 3373585701}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6814', 'nonce': 3373585701}]}, 'public_addr': '172.21.15.171:6812/3373585701', 'cluster_addr': '172.21.15.171:6813/3373585701', 'heartbeat_back_addr': '172.21.15.171:6815/3373585701', 'heartbeat_front_addr': '172.21.15.171:6814/3373585701', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:46:11.567649+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:45:58.488187+0000', 'dead_epoch': 613}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:46:37.512873+0000', 'dead_epoch': 169}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:46:12.819884+0000', 'dead_epoch': 538}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:46:49.692242+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:46:39.754243+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:45:26.351834+0000', 'dead_epoch': 503}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:45:29.435677+0000', 'dead_epoch': 721}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:46:18.755610+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:46:36.662481+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:46:16.675371+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:46:32.230346+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '8.d', 'osds': [11, 2]}], 'primary_temp': [], 'blocklist': {'172.21.15.5:0/3881726425': '2024-04-21T16:59:25.808337+0000', '172.21.15.5:0/338649532': '2024-04-21T16:59:25.808337+0000', '172.21.15.5:0/1227843664': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:6800/1876064631': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:0/160689449': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:6800/3521715286': '2024-04-21T16:59:25.808337+0000', '172.21.15.5:0/3841663462': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:0/687690919': '2024-04-21T16:59:25.808337+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-20T17:46:51.607 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:46:51.661 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:46:51.687 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:46:51.753 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:46:51.753 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:46:51.753 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:46:51.753 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:46:51.753 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.753 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.4.0:188147 7.1 e756/694 7:87e48d08:::benchmark_data_smithi171_161871_object137531:head v 756'17194, mlcod=756'17191)", 2024-04-20T17:46:51.753 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:42:20.102271+0000", 2024-04-20T17:46:51.753 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 271.63912102400002, 2024-04-20T17:46:51.753 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.097189071000000002, 2024-04-20T17:46:51.753 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:51.753 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:51.754 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:51.754 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:51.754 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.754 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:51.754 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.102271+0000", 2024-04-20T17:46:51.754 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.754 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.754 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.754 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:51.754 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.102271+0000", 2024-04-20T17:46:51.754 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.754 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.754 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.755 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:51.755 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.102272+0000", 2024-04-20T17:46:51.755 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.733e-06 2024-04-20T17:46:51.755 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.755 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.755 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:51.755 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.102281+0000", 2024-04-20T17:46:51.755 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8580000000000006e-06 2024-04-20T17:46:51.755 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.755 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.755 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:51.755 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.102296+0000", 2024-04-20T17:46:51.755 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5258e-05 2024-04-20T17:46:51.755 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.756 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.756 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:51.756 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.102304+0000", 2024-04-20T17:46:51.756 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9119999999999998e-06 2024-04-20T17:46:51.756 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.756 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.756 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:51.756 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.102371+0000", 2024-04-20T17:46:51.756 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6272000000000005e-05 2024-04-20T17:46:51.756 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.756 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.756 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:51.756 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.102389+0000", 2024-04-20T17:46:51.756 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8266000000000001e-05 2024-04-20T17:46:51.757 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.757 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.757 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:51.757 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.199422+0000", 2024-04-20T17:46:51.757 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.097032725 2024-04-20T17:46:51.757 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.757 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.757 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:51.757 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.199460+0000", 2024-04-20T17:46:51.757 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8046999999999997e-05 2024-04-20T17:46:51.757 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.757 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:51.757 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.757 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.758 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.758 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.4.0:188148 7.1 e756/694 7:87e508f5:::benchmark_data_smithi171_161871_object135563:head v 756'17195, mlcod=756'17191)", 2024-04-20T17:46:51.758 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:42:20.104228+0000", 2024-04-20T17:46:51.758 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 271.63716373699998, 2024-04-20T17:46:51.758 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.095250046000000005, 2024-04-20T17:46:51.758 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:51.758 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:51.758 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:51.758 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:51.758 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.758 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:51.758 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.104228+0000", 2024-04-20T17:46:51.758 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.758 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.759 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.759 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:51.759 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.104228+0000", 2024-04-20T17:46:51.759 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.759 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.759 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.759 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:51.759 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.104229+0000", 2024-04-20T17:46:51.759 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5880000000000001e-06 2024-04-20T17:46:51.759 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.759 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.759 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:51.760 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.104240+0000", 2024-04-20T17:46:51.760 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0973e-05 2024-04-20T17:46:51.760 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.760 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.760 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:51.760 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.104248+0000", 2024-04-20T17:46:51.760 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4420000000000001e-06 2024-04-20T17:46:51.760 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.760 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.760 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:51.761 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.104255+0000", 2024-04-20T17:46:51.761 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1470000000000004e-06 2024-04-20T17:46:51.761 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.761 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.761 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:51.761 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.106541+0000", 2024-04-20T17:46:51.761 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00228649 2024-04-20T17:46:51.761 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.761 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.761 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:51.761 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.106561+0000", 2024-04-20T17:46:51.762 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9536999999999999e-05 2024-04-20T17:46:51.762 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.762 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.762 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:51.762 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.199463+0000", 2024-04-20T17:46:51.762 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.092902062999999993 2024-04-20T17:46:51.762 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.762 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.762 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:51.762 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.199478+0000", 2024-04-20T17:46:51.762 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4806e-05 2024-04-20T17:46:51.762 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.763 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:51.763 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.763 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.763 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.763 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.4.0:188149 7.1 e756/694 7:87e6ec78:::benchmark_data_smithi171_161871_object139853:head v 756'17196, mlcod=756'17191)", 2024-04-20T17:46:51.763 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:42:20.105775+0000", 2024-04-20T17:46:51.763 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 271.63561700499997, 2024-04-20T17:46:51.763 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.10222830199999999, 2024-04-20T17:46:51.763 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:51.763 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:51.763 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:51.763 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:51.763 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.763 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:51.763 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.105775+0000", 2024-04-20T17:46:51.764 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.764 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.764 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.764 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:51.764 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.105775+0000", 2024-04-20T17:46:51.764 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.764 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.764 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.764 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:51.764 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.105776+0000", 2024-04-20T17:46:51.764 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.311e-06 2024-04-20T17:46:51.764 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.764 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.765 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:51.765 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.105783+0000", 2024-04-20T17:46:51.765 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5510000000000002e-06 2024-04-20T17:46:51.765 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.765 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.765 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:51.765 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.105792+0000", 2024-04-20T17:46:51.765 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.3969999999999994e-06 2024-04-20T17:46:51.765 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.765 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.765 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:51.765 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.105797+0000", 2024-04-20T17:46:51.765 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8910000000000003e-06 2024-04-20T17:46:51.765 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.766 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.766 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:51.766 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.111328+0000", 2024-04-20T17:46:51.766 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0055310849999999998 2024-04-20T17:46:51.766 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.766 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.766 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:51.766 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.111351+0000", 2024-04-20T17:46:51.766 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2909999999999999e-05 2024-04-20T17:46:51.766 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.766 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.766 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:51.766 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.207958+0000", 2024-04-20T17:46:51.766 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.096607788999999999 2024-04-20T17:46:51.766 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.767 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.767 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:51.767 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.208003+0000", 2024-04-20T17:46:51.767 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4367999999999999e-05 2024-04-20T17:46:51.767 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.767 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:51.767 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.767 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.767 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.767 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.4.0:188150 7.1 e756/694 7:87e7f480:::benchmark_data_smithi171_161871_object114500:head v 756'17197, mlcod=756'17191)", 2024-04-20T17:46:51.767 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:42:20.108628+0000", 2024-04-20T17:46:51.767 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 271.63276396700002, 2024-04-20T17:46:51.768 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.099403087000000001, 2024-04-20T17:46:51.769 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:51.769 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:51.769 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:51.769 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:51.769 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.769 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:51.769 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.108628+0000", 2024-04-20T17:46:51.769 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.769 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.769 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.769 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:51.769 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.108628+0000", 2024-04-20T17:46:51.770 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.770 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.770 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.770 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:51.770 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.108629+0000", 2024-04-20T17:46:51.770 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.753e-06 2024-04-20T17:46:51.770 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.770 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.770 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:51.770 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.108640+0000", 2024-04-20T17:46:51.770 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0559e-05 2024-04-20T17:46:51.771 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.771 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.771 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:51.771 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.108654+0000", 2024-04-20T17:46:51.771 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4579000000000001e-05 2024-04-20T17:46:51.771 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.771 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.771 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:51.771 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.108663+0000", 2024-04-20T17:46:51.771 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.799e-06 2024-04-20T17:46:51.771 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.771 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.771 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:51.771 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.114989+0000", 2024-04-20T17:46:51.771 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.006326099 2024-04-20T17:46:51.772 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.772 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.772 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:51.772 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.115012+0000", 2024-04-20T17:46:51.772 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2857000000000002e-05 2024-04-20T17:46:51.772 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.772 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.772 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:51.772 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.208007+0000", 2024-04-20T17:46:51.772 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.092994798000000004 2024-04-20T17:46:51.772 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.772 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.772 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:51.772 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.208031+0000", 2024-04-20T17:46:51.772 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3643e-05 2024-04-20T17:46:51.772 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.773 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:51.773 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.773 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.773 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.773 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.4.0:188151 7.1 e756/694 7:87e816c4:::benchmark_data_smithi171_161871_object135308:head v 756'17198, mlcod=756'17191)", 2024-04-20T17:46:51.773 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:42:20.111424+0000", 2024-04-20T17:46:51.773 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 271.629967604, 2024-04-20T17:46:51.773 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.096631075999999996, 2024-04-20T17:46:51.773 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:51.773 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:51.773 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:51.773 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:51.773 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.773 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:51.774 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.111424+0000", 2024-04-20T17:46:51.774 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.774 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.774 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.774 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:51.774 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.111424+0000", 2024-04-20T17:46:51.774 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.774 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.774 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.774 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:51.774 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.111426+0000", 2024-04-20T17:46:51.774 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7850000000000001e-06 2024-04-20T17:46:51.774 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.775 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.775 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:51.775 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.111434+0000", 2024-04-20T17:46:51.775 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1939999999999998e-06 2024-04-20T17:46:51.775 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.775 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.775 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:51.775 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.111446+0000", 2024-04-20T17:46:51.775 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1945e-05 2024-04-20T17:46:51.775 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.775 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.775 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:51.775 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.111455+0000", 2024-04-20T17:46:51.775 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1300000000000007e-06 2024-04-20T17:46:51.775 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.775 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.776 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:51.776 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.119335+0000", 2024-04-20T17:46:51.776 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0078800510000000008 2024-04-20T17:46:51.776 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.776 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.776 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:51.776 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.119357+0000", 2024-04-20T17:46:51.776 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1682e-05 2024-04-20T17:46:51.776 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.776 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.776 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:51.776 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.208033+0000", 2024-04-20T17:46:51.776 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.088676594999999997 2024-04-20T17:46:51.776 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.777 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.777 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:51.777 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.208055+0000", 2024-04-20T17:46:51.777 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1693999999999999e-05 2024-04-20T17:46:51.777 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.777 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:51.777 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.777 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.777 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.777 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.4.0:188152 7.1 e756/694 7:87e8364c:::benchmark_data_smithi171_161871_object140908:head v 756'17199, mlcod=756'17191)", 2024-04-20T17:46:51.777 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:42:20.114178+0000", 2024-04-20T17:46:51.777 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 271.62721339900003, 2024-04-20T17:46:51.777 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.093901723000000006, 2024-04-20T17:46:51.778 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:51.778 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:51.778 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:51.778 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:51.778 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.778 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:51.778 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.114178+0000", 2024-04-20T17:46:51.778 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.778 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.778 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.778 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:51.778 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.114178+0000", 2024-04-20T17:46:51.778 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.779 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.779 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.779 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:51.779 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.114180+0000", 2024-04-20T17:46:51.779 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7579999999999999e-06 2024-04-20T17:46:51.779 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.779 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.779 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:51.779 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.114190+0000", 2024-04-20T17:46:51.779 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0356999999999999e-05 2024-04-20T17:46:51.779 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.779 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.779 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:51.780 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.114210+0000", 2024-04-20T17:46:51.780 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0197e-05 2024-04-20T17:46:51.780 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.780 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.780 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:51.780 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.114218+0000", 2024-04-20T17:46:51.780 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9640000000000003e-06 2024-04-20T17:46:51.780 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.780 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.780 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:51.780 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.123291+0000", 2024-04-20T17:46:51.780 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0090720880000000007 2024-04-20T17:46:51.780 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.781 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.781 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:51.781 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.123309+0000", 2024-04-20T17:46:51.781 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8165999999999999e-05 2024-04-20T17:46:51.781 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.781 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.781 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:51.781 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.208058+0000", 2024-04-20T17:46:51.781 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.084749273 2024-04-20T17:46:51.781 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.781 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.781 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:51.781 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:42:20.208080+0000", 2024-04-20T17:46:51.781 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.192e-05 2024-04-20T17:46:51.782 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.782 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:51.782 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.782 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.782 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.782 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:73314 8.2 8:48746f67:::benchmark_data_smithi171_175757_object73313:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e792)", 2024-04-20T17:46:51.782 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:43:57.527462+0000", 2024-04-20T17:46:51.782 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 174.21392960899999, 2024-04-20T17:46:51.782 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.13009863399999999, 2024-04-20T17:46:51.782 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:51.782 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:51.782 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:51.782 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:51.782 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:51.783 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:51.783 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 73314 2024-04-20T17:46:51.783 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.783 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:51.783 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.783 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:51.783 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.527462+0000", 2024-04-20T17:46:51.783 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.783 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.783 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.783 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:51.783 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.527462+0000", 2024-04-20T17:46:51.784 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.784 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.784 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.784 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:51.784 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.527464+0000", 2024-04-20T17:46:51.784 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5859999999999999e-06 2024-04-20T17:46:51.784 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.784 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.784 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:51.784 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.527484+0000", 2024-04-20T17:46:51.784 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0279e-05 2024-04-20T17:46:51.784 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.785 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.785 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:51.785 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.527496+0000", 2024-04-20T17:46:51.785 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1855000000000001e-05 2024-04-20T17:46:51.785 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.785 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.785 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:51.785 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.527502+0000", 2024-04-20T17:46:51.785 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2299999999999996e-06 2024-04-20T17:46:51.785 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.785 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.785 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:51.785 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.527552+0000", 2024-04-20T17:46:51.785 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0287000000000002e-05 2024-04-20T17:46:51.786 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.786 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.786 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:51.786 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.527685+0000", 2024-04-20T17:46:51.786 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013262699999999999 2024-04-20T17:46:51.786 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.786 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.786 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:46:51.786 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.527771+0000", 2024-04-20T17:46:51.786 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5986e-05 2024-04-20T17:46:51.786 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.786 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.786 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:51.786 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.530045+0000", 2024-04-20T17:46:51.787 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0022740009999999999 2024-04-20T17:46:51.787 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.787 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.787 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:51.787 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.657487+0000", 2024-04-20T17:46:51.787 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.127442269 2024-04-20T17:46:51.787 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.787 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.787 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:51.787 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.657527+0000", 2024-04-20T17:46:51.787 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9864000000000003e-05 2024-04-20T17:46:51.787 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.787 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.788 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:51.788 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.657561+0000", 2024-04-20T17:46:51.788 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3649999999999998e-05 2024-04-20T17:46:51.788 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.788 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:51.788 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.788 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.788 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.788 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:73321 8.2 8:4b363db9:::benchmark_data_smithi171_175757_object73320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e792)", 2024-04-20T17:46:51.788 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:43:57.530431+0000", 2024-04-20T17:46:51.788 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 174.210961051, 2024-04-20T17:46:51.788 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.13152967700000001, 2024-04-20T17:46:51.788 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:51.789 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:51.789 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:51.789 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:51.789 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:51.789 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:51.789 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 73321 2024-04-20T17:46:51.789 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.789 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:51.789 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.789 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:51.789 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.530431+0000", 2024-04-20T17:46:51.789 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.789 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.790 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.790 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:51.790 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.530431+0000", 2024-04-20T17:46:51.790 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.790 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.790 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.790 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:51.790 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.530431+0000", 2024-04-20T17:46:51.790 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4700000000000001e-07 2024-04-20T17:46:51.790 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.790 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.790 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:51.790 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.530453+0000", 2024-04-20T17:46:51.791 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1807999999999999e-05 2024-04-20T17:46:51.791 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.791 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.791 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:51.791 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.530460+0000", 2024-04-20T17:46:51.791 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6610000000000003e-06 2024-04-20T17:46:51.791 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.791 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.791 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:51.791 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.530465+0000", 2024-04-20T17:46:51.791 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2140000000000004e-06 2024-04-20T17:46:51.791 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.791 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.792 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:51.792 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.530513+0000", 2024-04-20T17:46:51.792 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8096000000000002e-05 2024-04-20T17:46:51.792 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.792 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.792 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:51.792 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.530631+0000", 2024-04-20T17:46:51.792 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011776499999999999 2024-04-20T17:46:51.792 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.792 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.792 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:46:51.792 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.530757+0000", 2024-04-20T17:46:51.792 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012592699999999999 2024-04-20T17:46:51.792 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.793 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.793 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:51.793 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.532859+0000", 2024-04-20T17:46:51.793 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.002102577 2024-04-20T17:46:51.793 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.793 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.793 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:51.793 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.661860+0000", 2024-04-20T17:46:51.793 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.129000633 2024-04-20T17:46:51.793 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.793 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.793 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:51.793 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.661914+0000", 2024-04-20T17:46:51.794 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3881999999999997e-05 2024-04-20T17:46:51.794 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.794 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.794 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:51.794 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.661960+0000", 2024-04-20T17:46:51.794 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6366999999999998e-05 2024-04-20T17:46:51.794 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.794 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:51.794 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.794 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.794 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.795 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:73350 8.2 8:45b165b0:::benchmark_data_smithi171_175757_object73349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e792)", 2024-04-20T17:46:51.795 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:43:57.547704+0000", 2024-04-20T17:46:51.795 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 174.19368773299999, 2024-04-20T17:46:51.795 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.114365699, 2024-04-20T17:46:51.795 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:51.795 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:51.795 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:51.795 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:51.795 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:51.795 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:51.795 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 73350 2024-04-20T17:46:51.795 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.796 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:51.796 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.796 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:51.796 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.547704+0000", 2024-04-20T17:46:51.796 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.796 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.796 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.796 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:51.796 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.547704+0000", 2024-04-20T17:46:51.796 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.796 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.796 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.796 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:51.797 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.547706+0000", 2024-04-20T17:46:51.797 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4210000000000002e-06 2024-04-20T17:46:51.797 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.797 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.797 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:51.797 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.547732+0000", 2024-04-20T17:46:51.797 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5624000000000001e-05 2024-04-20T17:46:51.797 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.797 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.797 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:51.797 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.547750+0000", 2024-04-20T17:46:51.797 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8244e-05 2024-04-20T17:46:51.798 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.798 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.798 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:51.798 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.547758+0000", 2024-04-20T17:46:51.798 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2970000000000004e-06 2024-04-20T17:46:51.798 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.798 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.798 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:51.798 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.547816+0000", 2024-04-20T17:46:51.798 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7722000000000001e-05 2024-04-20T17:46:51.798 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.798 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.799 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:51.799 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.547959+0000", 2024-04-20T17:46:51.799 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000142614 2024-04-20T17:46:51.799 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.799 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.799 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:46:51.799 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.548122+0000", 2024-04-20T17:46:51.799 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000163101 2024-04-20T17:46:51.799 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.799 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.799 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:51.799 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.550539+0000", 2024-04-20T17:46:51.800 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.002416793 2024-04-20T17:46:51.800 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.800 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.800 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:51.800 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.662000+0000", 2024-04-20T17:46:51.800 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.111461825 2024-04-20T17:46:51.800 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.800 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.800 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:51.800 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.662038+0000", 2024-04-20T17:46:51.800 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7064999999999998e-05 2024-04-20T17:46:51.800 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.800 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.800 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:51.801 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.662070+0000", 2024-04-20T17:46:51.801 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1992999999999997e-05 2024-04-20T17:46:51.801 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.801 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:51.801 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.801 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.801 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.801 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:73362 8.2 8:4bb3d52e:::benchmark_data_smithi171_175757_object73361:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e792)", 2024-04-20T17:46:51.801 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:43:57.559140+0000", 2024-04-20T17:46:51.801 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 174.18225167899999, 2024-04-20T17:46:51.801 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.103074659, 2024-04-20T17:46:51.801 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:51.801 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:51.802 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:51.802 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:51.802 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:51.802 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:51.802 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 73362 2024-04-20T17:46:51.802 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.802 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:51.802 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.802 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:51.802 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.559140+0000", 2024-04-20T17:46:51.802 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.802 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.802 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.802 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:51.803 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.559140+0000", 2024-04-20T17:46:51.803 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.803 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.803 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.803 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:51.803 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.559142+0000", 2024-04-20T17:46:51.803 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5100000000000001e-06 2024-04-20T17:46:51.803 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.803 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.803 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:51.803 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.559167+0000", 2024-04-20T17:46:51.803 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4275999999999999e-05 2024-04-20T17:46:51.803 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.804 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.804 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:51.804 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.559178+0000", 2024-04-20T17:46:51.804 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1636e-05 2024-04-20T17:46:51.804 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.804 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.804 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:51.804 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.559187+0000", 2024-04-20T17:46:51.804 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9309999999999997e-06 2024-04-20T17:46:51.804 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.804 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.804 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:51.804 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.559253+0000", 2024-04-20T17:46:51.804 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6099999999999994e-05 2024-04-20T17:46:51.804 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.805 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.805 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:51.805 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.559397+0000", 2024-04-20T17:46:51.805 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014334600000000001 2024-04-20T17:46:51.805 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.805 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.805 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:46:51.805 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.559562+0000", 2024-04-20T17:46:51.805 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00016495999999999999 2024-04-20T17:46:51.805 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.805 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.805 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:51.805 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.564627+0000", 2024-04-20T17:46:51.806 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0050657389999999997 2024-04-20T17:46:51.806 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.806 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.806 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:51.806 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.662117+0000", 2024-04-20T17:46:51.806 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.097489991999999998 2024-04-20T17:46:51.806 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.806 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.806 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:51.806 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.662168+0000", 2024-04-20T17:46:51.806 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0648e-05 2024-04-20T17:46:51.806 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.806 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.807 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:51.807 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.662215+0000", 2024-04-20T17:46:51.807 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6520999999999997e-05 2024-04-20T17:46:51.807 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.807 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:51.807 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.807 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.807 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.807 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:73368 8.2 8:4f04434c:::benchmark_data_smithi171_175757_object73367:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e792)", 2024-04-20T17:46:51.807 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:43:57.562200+0000", 2024-04-20T17:46:51.807 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 174.17919170799999, 2024-04-20T17:46:51.807 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.10014793800000001, 2024-04-20T17:46:51.807 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:51.807 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:51.807 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:51.807 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:51.808 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:51.808 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:51.808 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 73368 2024-04-20T17:46:51.808 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.808 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:51.808 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.808 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:51.808 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.562200+0000", 2024-04-20T17:46:51.808 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.808 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.808 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.808 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:51.808 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.562200+0000", 2024-04-20T17:46:51.808 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.809 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.809 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.809 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:51.809 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.562207+0000", 2024-04-20T17:46:51.809 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7150000000000001e-06 2024-04-20T17:46:51.809 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.809 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.809 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:51.809 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.562241+0000", 2024-04-20T17:46:51.809 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4093e-05 2024-04-20T17:46:51.809 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.809 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.809 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:51.809 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.562255+0000", 2024-04-20T17:46:51.809 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4559000000000001e-05 2024-04-20T17:46:51.809 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.810 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.810 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:51.810 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.562264+0000", 2024-04-20T17:46:51.810 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0080000000000008e-06 2024-04-20T17:46:51.810 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.810 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.810 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:51.810 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.562323+0000", 2024-04-20T17:46:51.810 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8981000000000003e-05 2024-04-20T17:46:51.810 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.810 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.810 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:51.810 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.562453+0000", 2024-04-20T17:46:51.810 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000130228 2024-04-20T17:46:51.810 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.811 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.811 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:46:51.811 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.562543+0000", 2024-04-20T17:46:51.811 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9754999999999997e-05 2024-04-20T17:46:51.811 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.811 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.811 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:51.811 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.567421+0000", 2024-04-20T17:46:51.811 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0048782469999999996 2024-04-20T17:46:51.811 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.811 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.811 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:51.811 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.662265+0000", 2024-04-20T17:46:51.812 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.094843100999999999 2024-04-20T17:46:51.812 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.812 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.812 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:51.812 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.662313+0000", 2024-04-20T17:46:51.812 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.799e-05 2024-04-20T17:46:51.812 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.812 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.812 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:51.812 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:43:57.662348+0000", 2024-04-20T17:46:51.812 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5261e-05 2024-04-20T17:46:51.812 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.812 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:51.812 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.812 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.813 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.813 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:166182 8.5 8:acaed60c:::benchmark_data_smithi171_175757_object21068:head [delete] snapc 0=[] ondisk+write+known_if_redirected e824)", 2024-04-20T17:46:51.813 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:45:33.546740+0000", 2024-04-20T17:46:51.813 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 78.194651594999996, 2024-04-20T17:46:51.813 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.18443790500000001, 2024-04-20T17:46:51.813 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:51.813 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:51.813 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:51.813 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:51.813 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:51.813 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:51.813 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 166182 2024-04-20T17:46:51.813 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.814 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:51.814 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.814 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:51.814 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.546740+0000", 2024-04-20T17:46:51.814 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.814 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.814 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.814 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:51.814 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.546740+0000", 2024-04-20T17:46:51.814 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.814 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.814 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.815 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:51.815 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.546741+0000", 2024-04-20T17:46:51.815 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3799999999999997e-07 2024-04-20T17:46:51.815 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.815 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.815 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:51.815 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.546743+0000", 2024-04-20T17:46:51.815 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8619999999999999e-06 2024-04-20T17:46:51.815 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.815 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.815 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:51.815 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.546748+0000", 2024-04-20T17:46:51.815 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3710000000000002e-06 2024-04-20T17:46:51.816 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.816 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.816 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:51.816 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.546752+0000", 2024-04-20T17:46:51.816 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3290000000000004e-06 2024-04-20T17:46:51.816 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.816 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.816 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:51.816 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.559232+0000", 2024-04-20T17:46:51.816 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.012479532999999999 2024-04-20T17:46:51.816 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.816 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.816 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:51.817 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.559364+0000", 2024-04-20T17:46:51.817 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000131972 2024-04-20T17:46:51.817 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.817 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.817 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4", 2024-04-20T17:46:51.817 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.559485+0000", 2024-04-20T17:46:51.817 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012113 2024-04-20T17:46:51.817 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.817 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.817 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:51.817 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.572560+0000", 2024-04-20T17:46:51.817 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013074776 2024-04-20T17:46:51.817 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.817 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.818 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:51.818 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.731089+0000", 2024-04-20T17:46:51.818 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.158529277 2024-04-20T17:46:51.818 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.818 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.818 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:51.818 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.731137+0000", 2024-04-20T17:46:51.818 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8418000000000003e-05 2024-04-20T17:46:51.818 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.818 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.818 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:51.818 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.731178+0000", 2024-04-20T17:46:51.818 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0599000000000002e-05 2024-04-20T17:46:51.819 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.819 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:51.819 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.819 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.819 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.819 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:166191 8.5 8:a1858a3a:::benchmark_data_smithi171_175757_object21077:head [delete] snapc 0=[] ondisk+write+known_if_redirected e824)", 2024-04-20T17:46:51.819 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:45:33.546780+0000", 2024-04-20T17:46:51.819 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 78.194611809999998, 2024-04-20T17:46:51.819 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.18456041400000001, 2024-04-20T17:46:51.819 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:51.819 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:51.819 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:51.819 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:51.819 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:51.819 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:51.820 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 166191 2024-04-20T17:46:51.820 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.820 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:51.820 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.820 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:51.820 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.546780+0000", 2024-04-20T17:46:51.820 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.820 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.820 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.820 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:51.820 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.546780+0000", 2024-04-20T17:46:51.820 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.820 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.820 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.821 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:51.821 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.546780+0000", 2024-04-20T17:46:51.821 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.63e-07 2024-04-20T17:46:51.821 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.821 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.821 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:51.821 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.546783+0000", 2024-04-20T17:46:51.821 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5519999999999999e-06 2024-04-20T17:46:51.821 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.821 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.821 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:51.821 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.546787+0000", 2024-04-20T17:46:51.821 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1300000000000003e-06 2024-04-20T17:46:51.821 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.821 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.822 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:51.822 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.546791+0000", 2024-04-20T17:46:51.822 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2640000000000002e-06 2024-04-20T17:46:51.822 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.822 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.822 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:51.822 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.567635+0000", 2024-04-20T17:46:51.822 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020843886999999998 2024-04-20T17:46:51.822 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.822 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.822 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:51.822 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.567743+0000", 2024-04-20T17:46:51.822 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010839 2024-04-20T17:46:51.822 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.822 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.823 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4", 2024-04-20T17:46:51.823 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.567848+0000", 2024-04-20T17:46:51.823 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010412 2024-04-20T17:46:51.823 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.823 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.823 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:51.823 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.576329+0000", 2024-04-20T17:46:51.823 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.008481443 2024-04-20T17:46:51.823 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.823 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.823 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:51.823 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.731186+0000", 2024-04-20T17:46:51.823 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.15485653899999999 2024-04-20T17:46:51.823 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.823 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.824 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:51.824 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.731307+0000", 2024-04-20T17:46:51.824 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000121388 2024-04-20T17:46:51.824 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.824 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.824 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:51.824 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.731340+0000", 2024-04-20T17:46:51.824 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3238000000000003e-05 2024-04-20T17:46:51.824 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.824 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:51.824 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.824 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.824 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.824 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:166223 8.2 8:4ca6df0c:::benchmark_data_smithi171_175757_object21109:head [delete] snapc 0=[] ondisk+write+known_if_redirected e824)", 2024-04-20T17:46:51.824 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:45:33.585556+0000", 2024-04-20T17:46:51.824 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 78.155835644000007, 2024-04-20T17:46:51.825 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.10085390399999999, 2024-04-20T17:46:51.825 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:51.825 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:51.825 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:51.825 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:51.825 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:51.825 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:51.825 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 166223 2024-04-20T17:46:51.825 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.825 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:51.825 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.825 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:51.825 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.585556+0000", 2024-04-20T17:46:51.825 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.825 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.826 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.826 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:51.826 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.585556+0000", 2024-04-20T17:46:51.826 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.826 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.826 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.826 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:51.826 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.585558+0000", 2024-04-20T17:46:51.826 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2560000000000001e-06 2024-04-20T17:46:51.826 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.826 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.826 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:51.826 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.585563+0000", 2024-04-20T17:46:51.826 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1660000000000002e-06 2024-04-20T17:46:51.826 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.827 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.827 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:51.827 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.585578+0000", 2024-04-20T17:46:51.827 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4524e-05 2024-04-20T17:46:51.827 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.827 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.827 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:51.827 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.585586+0000", 2024-04-20T17:46:51.827 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7849999999999992e-06 2024-04-20T17:46:51.827 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.827 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.827 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:51.827 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.585710+0000", 2024-04-20T17:46:51.828 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012420399999999999 2024-04-20T17:46:51.828 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.828 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.828 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:51.828 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.585826+0000", 2024-04-20T17:46:51.828 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011587299999999999 2024-04-20T17:46:51.828 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.828 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.828 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0", 2024-04-20T17:46:51.828 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.585930+0000", 2024-04-20T17:46:51.828 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000103924 2024-04-20T17:46:51.828 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.828 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.828 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:51.829 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.591200+0000", 2024-04-20T17:46:51.829 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0052700819999999997 2024-04-20T17:46:51.829 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.829 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.829 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:51.829 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.686313+0000", 2024-04-20T17:46:51.829 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.095113745 2024-04-20T17:46:51.829 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.829 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.829 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:51.829 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.686363+0000", 2024-04-20T17:46:51.829 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9274999999999998e-05 2024-04-20T17:46:51.829 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.830 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.830 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:51.830 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.686410+0000", 2024-04-20T17:46:51.830 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7070000000000002e-05 2024-04-20T17:46:51.830 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.830 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:51.830 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.830 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.830 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.830 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:166227 8.5 8:ada196b2:::benchmark_data_smithi171_175757_object21113:head [delete] snapc 0=[] ondisk+write+known_if_redirected e824)", 2024-04-20T17:46:51.830 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:45:33.585604+0000", 2024-04-20T17:46:51.830 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 78.155787845000006, 2024-04-20T17:46:51.830 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.14581248099999999, 2024-04-20T17:46:51.830 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:51.830 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:51.831 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:51.831 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:51.831 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:51.831 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:51.831 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 166227 2024-04-20T17:46:51.831 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.831 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:51.831 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.831 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:51.831 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.585604+0000", 2024-04-20T17:46:51.831 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.831 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.831 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.831 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:51.831 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.585604+0000", 2024-04-20T17:46:51.832 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.832 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.832 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.832 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:51.832 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.585604+0000", 2024-04-20T17:46:51.832 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4099999999999998e-07 2024-04-20T17:46:51.832 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.832 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.832 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:51.832 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.585607+0000", 2024-04-20T17:46:51.832 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.016e-06 2024-04-20T17:46:51.832 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.832 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.832 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:51.832 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.585613+0000", 2024-04-20T17:46:51.833 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7100000000000004e-06 2024-04-20T17:46:51.833 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.833 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.833 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:51.833 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.585618+0000", 2024-04-20T17:46:51.833 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0849999999999996e-06 2024-04-20T17:46:51.833 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.833 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.833 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:51.833 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.588201+0000", 2024-04-20T17:46:51.833 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0025827720000000001 2024-04-20T17:46:51.833 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.833 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.833 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:51.833 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.588319+0000", 2024-04-20T17:46:51.833 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011832000000000001 2024-04-20T17:46:51.834 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.834 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.834 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4", 2024-04-20T17:46:51.834 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.588423+0000", 2024-04-20T17:46:51.834 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010389400000000001 2024-04-20T17:46:51.834 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.834 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.834 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:51.834 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.598489+0000", 2024-04-20T17:46:51.834 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010066278999999999 2024-04-20T17:46:51.834 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.834 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.835 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:51.835 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.731349+0000", 2024-04-20T17:46:51.835 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.13286003399999999 2024-04-20T17:46:51.835 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.835 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.835 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:51.835 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.731386+0000", 2024-04-20T17:46:51.835 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6794e-05 2024-04-20T17:46:51.835 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.835 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.835 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:51.835 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.731416+0000", 2024-04-20T17:46:51.836 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9935999999999999e-05 2024-04-20T17:46:51.836 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.836 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:51.836 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.836 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.836 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.836 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:166229 8.5 8:acaf5782:::benchmark_data_smithi171_175757_object21115:head [delete] snapc 0=[] ondisk+write+known_if_redirected e824)", 2024-04-20T17:46:51.836 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:45:33.585630+0000", 2024-04-20T17:46:51.836 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 78.155761957999999, 2024-04-20T17:46:51.836 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.14586337499999999, 2024-04-20T17:46:51.836 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:51.836 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:51.836 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:51.837 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:51.837 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:51.837 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:51.837 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 166229 2024-04-20T17:46:51.837 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.837 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:51.837 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.837 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:51.837 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.585630+0000", 2024-04-20T17:46:51.837 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.837 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.837 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.838 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:51.838 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.585630+0000", 2024-04-20T17:46:51.838 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.838 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.838 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.838 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:51.838 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.585630+0000", 2024-04-20T17:46:51.838 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9299999999999998e-07 2024-04-20T17:46:51.838 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.838 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.839 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:51.839 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.585632+0000", 2024-04-20T17:46:51.839 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0770000000000002e-06 2024-04-20T17:46:51.839 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.839 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.839 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:51.839 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.585637+0000", 2024-04-20T17:46:51.839 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6290000000000001e-06 2024-04-20T17:46:51.839 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.839 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.839 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:51.839 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.585641+0000", 2024-04-20T17:46:51.840 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4610000000000001e-06 2024-04-20T17:46:51.840 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.840 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.840 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:51.840 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.593300+0000", 2024-04-20T17:46:51.840 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0076586390000000001 2024-04-20T17:46:51.840 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.840 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.840 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:51.840 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.593414+0000", 2024-04-20T17:46:51.840 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000113967 2024-04-20T17:46:51.840 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.841 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.841 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4", 2024-04-20T17:46:51.841 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.593524+0000", 2024-04-20T17:46:51.841 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000109688 2024-04-20T17:46:51.841 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.841 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.841 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:51.841 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.599380+0000", 2024-04-20T17:46:51.841 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0058563469999999996 2024-04-20T17:46:51.841 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.841 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.842 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:51.842 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.731424+0000", 2024-04-20T17:46:51.842 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.13204449200000001 2024-04-20T17:46:51.842 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.842 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.842 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:51.842 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.731460+0000", 2024-04-20T17:46:51.842 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5589000000000001e-05 2024-04-20T17:46:51.842 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.842 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.842 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:51.843 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.731493+0000", 2024-04-20T17:46:51.843 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2993000000000001e-05 2024-04-20T17:46:51.843 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.843 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:51.843 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.843 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.843 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.843 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:166240 8.5 8:ab7d9ece:::benchmark_data_smithi171_175757_object21126:head [delete] snapc 0=[] ondisk+write+known_if_redirected e824)", 2024-04-20T17:46:51.843 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:45:33.610392+0000", 2024-04-20T17:46:51.843 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 78.130999721999999, 2024-04-20T17:46:51.843 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.121172877, 2024-04-20T17:46:51.843 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:51.843 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:51.843 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:51.844 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:51.844 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:51.844 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:51.844 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 166240 2024-04-20T17:46:51.844 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.844 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:51.844 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.844 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:51.844 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.610392+0000", 2024-04-20T17:46:51.844 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.844 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.844 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.844 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:51.845 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.610392+0000", 2024-04-20T17:46:51.845 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.845 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.845 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.845 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:51.845 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.610394+0000", 2024-04-20T17:46:51.845 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3829999999999999e-06 2024-04-20T17:46:51.845 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.845 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.845 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:51.845 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.610401+0000", 2024-04-20T17:46:51.845 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1069999999999999e-06 2024-04-20T17:46:51.845 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.846 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.846 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:51.846 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.610413+0000", 2024-04-20T17:46:51.846 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2002e-05 2024-04-20T17:46:51.846 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.846 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.846 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:51.846 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.610422+0000", 2024-04-20T17:46:51.846 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1799999999999996e-06 2024-04-20T17:46:51.846 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.846 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.846 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:51.846 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.610587+0000", 2024-04-20T17:46:51.847 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000165198 2024-04-20T17:46:51.847 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.847 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.847 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:51.847 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.610708+0000", 2024-04-20T17:46:51.847 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000121641 2024-04-20T17:46:51.847 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.847 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.847 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4", 2024-04-20T17:46:51.847 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.610820+0000", 2024-04-20T17:46:51.847 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011151599999999999 2024-04-20T17:46:51.847 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.847 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.847 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:51.847 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.622056+0000", 2024-04-20T17:46:51.848 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011236265 2024-04-20T17:46:51.848 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.848 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.848 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:51.848 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.731498+0000", 2024-04-20T17:46:51.848 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.109442066 2024-04-20T17:46:51.848 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.848 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.848 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:51.848 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.731533+0000", 2024-04-20T17:46:51.848 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5117999999999998e-05 2024-04-20T17:46:51.848 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.848 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.848 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:51.848 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.731565+0000", 2024-04-20T17:46:51.848 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1401e-05 2024-04-20T17:46:51.849 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.849 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:51.849 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.849 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.849 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.849 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:166245 8.5 8:a700121f:::benchmark_data_smithi171_175757_object21131:head [delete] snapc 0=[] ondisk+write+known_if_redirected e824)", 2024-04-20T17:46:51.849 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:45:33.616336+0000", 2024-04-20T17:46:51.849 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 78.125055395000004, 2024-04-20T17:46:51.849 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.115303924, 2024-04-20T17:46:51.849 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:51.849 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:51.849 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:51.849 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:51.849 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:51.849 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:51.850 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 166245 2024-04-20T17:46:51.850 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.850 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:51.850 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.850 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:51.850 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.616336+0000", 2024-04-20T17:46:51.850 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.850 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.850 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.850 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:51.850 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.616336+0000", 2024-04-20T17:46:51.850 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.850 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.850 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.851 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:51.851 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.616338+0000", 2024-04-20T17:46:51.851 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2050000000000001e-06 2024-04-20T17:46:51.851 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.851 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.851 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:51.851 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.616345+0000", 2024-04-20T17:46:51.851 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5150000000000003e-06 2024-04-20T17:46:51.851 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.851 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.851 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:51.851 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.616357+0000", 2024-04-20T17:46:51.851 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2534000000000001e-05 2024-04-20T17:46:51.851 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.852 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.852 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:51.852 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.616366+0000", 2024-04-20T17:46:51.852 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.884e-06 2024-04-20T17:46:51.852 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.852 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.852 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:51.852 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.616422+0000", 2024-04-20T17:46:51.852 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6051000000000003e-05 2024-04-20T17:46:51.852 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.852 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.852 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:51.852 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.616535+0000", 2024-04-20T17:46:51.852 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000112977 2024-04-20T17:46:51.853 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.853 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.853 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4", 2024-04-20T17:46:51.853 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.616636+0000", 2024-04-20T17:46:51.853 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000100911 2024-04-20T17:46:51.853 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.853 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.853 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:51.853 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.622308+0000", 2024-04-20T17:46:51.853 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0056717779999999997 2024-04-20T17:46:51.853 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.853 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.853 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:51.853 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.731570+0000", 2024-04-20T17:46:51.853 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.10926177500000001 2024-04-20T17:46:51.854 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.854 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.854 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:51.854 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.731609+0000", 2024-04-20T17:46:51.854 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9557000000000001e-05 2024-04-20T17:46:51.854 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.854 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.854 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:51.854 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:45:33.731640+0000", 2024-04-20T17:46:51.854 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0737000000000001e-05 2024-04-20T17:46:51.854 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.854 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:51.855 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.855 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.855 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.855 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:213591 8.9 8:969bf2b8:::benchmark_data_smithi171_175757_object68477:head [delete] snapc 0=[] ondisk+write+known_if_redirected e851)", 2024-04-20T17:46:51.855 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:46:50.849128+0000", 2024-04-20T17:46:51.855 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.89226405399999997, 2024-04-20T17:46:51.855 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.14040171200000001, 2024-04-20T17:46:51.855 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:51.855 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:51.855 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:51.855 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:51.855 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:51.855 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:51.855 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 213591 2024-04-20T17:46:51.856 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.856 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:51.856 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.856 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:51.856 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:50.849128+0000", 2024-04-20T17:46:51.856 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.856 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.856 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.856 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:51.856 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:50.849128+0000", 2024-04-20T17:46:51.856 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.856 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.857 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.857 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:51.857 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:50.849129+0000", 2024-04-20T17:46:51.857 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.767e-06 2024-04-20T17:46:51.857 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.857 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.857 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:51.857 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:50.849133+0000", 2024-04-20T17:46:51.857 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0589999999999996e-06 2024-04-20T17:46:51.857 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.857 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.857 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:51.857 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:50.849140+0000", 2024-04-20T17:46:51.857 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0690000000000004e-06 2024-04-20T17:46:51.858 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.858 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.858 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:51.858 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:50.849155+0000", 2024-04-20T17:46:51.858 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.504e-05 2024-04-20T17:46:51.858 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.858 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.858 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:51.858 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:50.849208+0000", 2024-04-20T17:46:51.858 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2383999999999999e-05 2024-04-20T17:46:51.858 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.858 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.858 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:51.859 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:50.849311+0000", 2024-04-20T17:46:51.859 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000103065 2024-04-20T17:46:51.859 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.859 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.859 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:46:51.859 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:50.849377+0000", 2024-04-20T17:46:51.859 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6356e-05 2024-04-20T17:46:51.859 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.859 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.859 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:51.859 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:50.855574+0000", 2024-04-20T17:46:51.859 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0061969240000000004 2024-04-20T17:46:51.859 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.859 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.860 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:51.860 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:50.989430+0000", 2024-04-20T17:46:51.860 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.13385583600000001 2024-04-20T17:46:51.860 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.860 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.860 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:51.860 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:50.989486+0000", 2024-04-20T17:46:51.860 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6008000000000003e-05 2024-04-20T17:46:51.860 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.860 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.860 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:51.860 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:50.989529+0000", 2024-04-20T17:46:51.860 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3204000000000002e-05 2024-04-20T17:46:51.860 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.861 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:51.861 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.861 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.861 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.861 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:213603 8.9 8:98ab4b4f:::benchmark_data_smithi171_175757_object68489:head [delete] snapc 0=[] ondisk+write+known_if_redirected e851)", 2024-04-20T17:46:51.861 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:46:50.863893+0000", 2024-04-20T17:46:51.861 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.87749896500000002, 2024-04-20T17:46:51.861 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.12571406199999999, 2024-04-20T17:46:51.861 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:46:51.861 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:46:51.861 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:46:51.861 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:46:51.861 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:46:51.861 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:46:51.861 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 213603 2024-04-20T17:46:51.861 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.862 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:46:51.862 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.862 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:46:51.862 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:50.863893+0000", 2024-04-20T17:46:51.862 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.862 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.862 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.862 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:46:51.862 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:50.863893+0000", 2024-04-20T17:46:51.862 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:46:51.862 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.862 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.862 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:46:51.862 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:50.863894+0000", 2024-04-20T17:46:51.862 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0869999999999999e-06 2024-04-20T17:46:51.863 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.863 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.863 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:46:51.863 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:50.863898+0000", 2024-04-20T17:46:51.863 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2549999999999999e-06 2024-04-20T17:46:51.863 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.863 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.863 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:46:51.863 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:50.863906+0000", 2024-04-20T17:46:51.863 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.3170000000000002e-06 2024-04-20T17:46:51.863 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.863 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.863 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:46:51.863 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:50.863913+0000", 2024-04-20T17:46:51.864 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.092e-06 2024-04-20T17:46:51.864 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.864 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.864 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:46:51.864 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:50.863981+0000", 2024-04-20T17:46:51.864 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7255000000000006e-05 2024-04-20T17:46:51.864 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.864 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.864 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:46:51.864 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:50.864101+0000", 2024-04-20T17:46:51.864 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000119932 2024-04-20T17:46:51.864 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.864 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.864 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:46:51.864 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:50.864199+0000", 2024-04-20T17:46:51.864 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.8679999999999997e-05 2024-04-20T17:46:51.865 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.865 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.865 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:46:51.865 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:50.873876+0000", 2024-04-20T17:46:51.865 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0096763539999999999 2024-04-20T17:46:51.865 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.865 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.865 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:46:51.865 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:50.989539+0000", 2024-04-20T17:46:51.865 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.115663473 2024-04-20T17:46:51.865 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.865 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.865 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:46:51.865 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:50.989572+0000", 2024-04-20T17:46:51.865 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3148999999999997e-05 2024-04-20T17:46:51.865 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:46:51.866 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:46:51.866 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:46:51.866 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:50.989607+0000", 2024-04-20T17:46:51.866 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4468000000000003e-05 2024-04-20T17:46:51.866 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.866 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:51.866 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.866 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:46:51.866 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:46:51.866 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:46:51.867 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:52.021 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:46:52.027 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:51 smithi171 ceph-mon[44189]: 8.b scrub starts 2024-04-20T17:46:52.027 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:51 smithi171 ceph-mon[44189]: 1.1 scrub ok 2024-04-20T17:46:52.027 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:51 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:46:52.027 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:51 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:46:52.027 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:51 smithi171 ceph-mon[44189]: osdmap e852: 12 total, 12 up, 12 in 2024-04-20T17:46:52.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:52.028 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:46:52.129 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:46:52.167 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:51 smithi005 ceph-mon[39562]: 8.b scrub starts 2024-04-20T17:46:52.167 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:51 smithi005 ceph-mon[39562]: 1.1 scrub ok 2024-04-20T17:46:52.167 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:51 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:46:52.167 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:51 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:46:52.167 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:51 smithi005 ceph-mon[39562]: osdmap e852: 12 total, 12 up, 12 in 2024-04-20T17:46:52.170 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:52.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:51 smithi106 ceph-mon[44423]: 8.b scrub starts 2024-04-20T17:46:52.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:51 smithi106 ceph-mon[44423]: 1.1 scrub ok 2024-04-20T17:46:52.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:51 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:46:52.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:51 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:46:52.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:51 smithi106 ceph-mon[44423]: osdmap e852: 12 total, 12 up, 12 in 2024-04-20T17:46:52.529 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:46:52.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:52.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:46:52.637 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:46:52.702 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:52.949 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:46:53.072 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:46:53.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:52 smithi005 ceph-mon[39562]: 8.b scrub ok 2024-04-20T17:46:53.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:52 smithi005 ceph-mon[39562]: pgmap v2265: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 640 op/s; 16420/154008 objects misplaced (10.662%); 18 MiB/s, 293 objects/s recovering 2024-04-20T17:46:53.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:52 smithi106 ceph-mon[44423]: 8.b scrub ok 2024-04-20T17:46:53.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:52 smithi106 ceph-mon[44423]: pgmap v2265: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 640 op/s; 16420/154008 objects misplaced (10.662%); 18 MiB/s, 293 objects/s recovering 2024-04-20T17:46:53.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:52 smithi171 ceph-mon[44189]: 8.b scrub ok 2024-04-20T17:46:53.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:52 smithi171 ceph-mon[44189]: pgmap v2265: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 640 op/s; 16420/154008 objects misplaced (10.662%); 18 MiB/s, 293 objects/s recovering 2024-04-20T17:46:54.039 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:46:54.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:54.055 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:46:54.064 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-20T17:46:54.156 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:46:54.182 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:54.201 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:46:54.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:54.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:46:54.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:53 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:46:54.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:53 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:46:54.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:53 smithi005 ceph-mon[39562]: osdmap e853: 12 total, 12 up, 12 in 2024-04-20T17:46:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:53 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:46:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:53 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:46:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:53 smithi106 ceph-mon[44423]: osdmap e853: 12 total, 12 up, 12 in 2024-04-20T17:46:54.304 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:46:54.331 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:54.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:53 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:46:54.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:53 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:46:54.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:53 smithi171 ceph-mon[44189]: osdmap e853: 12 total, 12 up, 12 in 2024-04-20T17:46:54.352 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:46:54.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:54.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:46:54.456 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:46:54.483 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:54.505 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:46:54.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:54.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:46:54.609 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:46:54.615 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:46:54.648 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:54.906 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:46:54.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:54.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:46:54.975 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:46:55.015 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:46:55.042 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:55.061 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:46:55.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:55.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:46:55.165 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:46:55.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:54 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/666285197' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:46:55.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:54 smithi005 ceph-mon[39562]: pgmap v2267: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 675 op/s; 16420/152322 objects misplaced (10.780%); 17 MiB/s, 265 objects/s recovering 2024-04-20T17:46:55.206 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:54 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/666285197' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:46:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:54 smithi106 ceph-mon[44423]: pgmap v2267: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 675 op/s; 16420/152322 objects misplaced (10.780%); 17 MiB/s, 265 objects/s recovering 2024-04-20T17:46:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:54 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/666285197' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:46:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:54 smithi171 ceph-mon[44189]: pgmap v2267: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 675 op/s; 16420/152322 objects misplaced (10.780%); 17 MiB/s, 265 objects/s recovering 2024-04-20T17:46:55.563 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:46:55.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:55.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:46:55.669 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:46:55.729 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:55.795 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:46:56.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:55 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:46:56.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:55 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/666285197' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:46:56.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:55 smithi171 ceph-mon[44189]: osdmap e854: 12 total, 12 up, 12 in 2024-04-20T17:46:56.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:55 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:46:56.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:55 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/666285197' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:46:56.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:55 smithi005 ceph-mon[39562]: osdmap e854: 12 total, 12 up, 12 in 2024-04-20T17:46:56.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:55 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:46:56.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:55 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/666285197' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:46:56.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:55 smithi106 ceph-mon[44423]: osdmap e854: 12 total, 12 up, 12 in 2024-04-20T17:46:57.024 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:46:57.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:57.034 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:46:57.135 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:46:57.180 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:57.913 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:57 smithi005 ceph-mon[39562]: pgmap v2269: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 588 op/s; 14850/148548 objects misplaced (9.997%); 17 MiB/s, 272 objects/s recovering 2024-04-20T17:46:58.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:57 smithi106 ceph-mon[44423]: pgmap v2269: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 588 op/s; 14850/148548 objects misplaced (9.997%); 17 MiB/s, 272 objects/s recovering 2024-04-20T17:46:58.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:57 smithi171 ceph-mon[44189]: pgmap v2269: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 588 op/s; 14850/148548 objects misplaced (9.997%); 17 MiB/s, 272 objects/s recovering 2024-04-20T17:46:58.335 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:46:58.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:46:58.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:46:58.444 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:46:58.490 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:46:58.970 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:46:58.971 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:46:58.978 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:46:59.005 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:46:59.041 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:46:59.078 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:46:59.106 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:46:59.133 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:46:59.157 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:46:59.183 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:46:59.225 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:46:59.269 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:46:59.298 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:46:59.330 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:46:59.332 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:46:59.332 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:46:59.332 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:46:59.332 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:46:59.332 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:46:59.332 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:46:59.332 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:46:59.333 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:46:59.333 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:46:59.333 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:46:59.333 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:46:59.838 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":2270,"stamp":"2024-04-20T17:46:58.639352+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4831135950,"num_objects":73712,"num_object_clones":0,"num_object_copies":147424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":14658,"num_objects_unfound":0,"num_objects_dirty":73712,"num_whiteouts":0,"num_read":46,"num_read_kb":40,"num_write":216578,"num_write_kb":9288173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16584,"num_bytes_recovered":1087858818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162360,"ondisk_log_size":162360,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":16740768,"kb_used_data":11043540,"kb_used_omap":404,"kb_used_meta":5696683,"kb_avail":1108201056,"statfs":{"total":1151940427776,"available":1134797881344,"internally_reserved":0,"allocated":11308584960,"data_stored":11301929350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":414014,"internal_metadata":5833404098},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,7,4,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-215744512,"num_objects":-3292,"num_object_clones":0,"num_object_copies":-6584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1762,"num_objects_unfound":0,"num_objects_dirty":-3292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3292,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1660,"num_bytes_recovered":108724248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.969357"},"pg_stats":[{"pgid":"1.6","version":"15'137","reported_seq":735,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-04-20T17:46:52.979024+0000","last_change":"2024-04-20T17:46:37.732112+0000","last_active":"2024-04-20T17:46:52.979024+0000","last_peered":"2024-04-20T17:46:52.979024+0000","last_clean":"2024-04-20T17:46:52.979024+0000","last_became_active":"2024-04-20T17:45:32.670941+0000","last_became_peered":"2024-04-20T17:45:32.670941+0000","last_unstale":"2024-04-20T17:46:52.979024+0000","last_undegraded":"2024-04-20T17:46:52.979024+0000","last_fullsized":"2024-04-20T17:46:52.979024+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:46:37.731983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:46:37.731983+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:48:06.468892+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"854'13352","reported_seq":13594,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-04-20T17:46:55.548835+0000","last_change":"2024-04-20T17:44:24.842530+0000","last_active":"2024-04-20T17:46:55.548835+0000","last_peered":"2024-04-20T17:46:55.548835+0000","last_clean":"2024-04-20T17:46:55.548835+0000","last_became_active":"2024-04-20T17:43:18.798932+0000","last_became_peered":"2024-04-20T17:43:18.798932+0000","last_unstale":"2024-04-20T17:46:55.548835+0000","last_undegraded":"2024-04-20T17:46:55.548835+0000","last_fullsized":"2024-04-20T17:46:55.548835+0000","mapping_epoch":777,"log_start":"788'3300","ondisk_log_start":"788'3300","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'6866","last_scrub_stamp":"2024-04-20T17:44:24.842384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:44:24.842384+0000","objects_scrubbed":6822,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1583,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297402368,"num_objects":4538,"num_object_clones":0,"num_object_copies":9076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13352,"num_write_kb":572480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.7","version":"15'137","reported_seq":740,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-04-20T17:46:55.130575+0000","last_change":"2024-04-20T17:46:47.352597+0000","last_active":"2024-04-20T17:46:55.130575+0000","last_peered":"2024-04-20T17:46:55.130575+0000","last_clean":"2024-04-20T17:46:55.130575+0000","last_became_active":"2024-04-20T17:45:29.817528+0000","last_became_peered":"2024-04-20T17:45:29.817528+0000","last_unstale":"2024-04-20T17:46:55.130575+0000","last_undegraded":"2024-04-20T17:46:55.130575+0000","last_fullsized":"2024-04-20T17:46:55.130575+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:46:47.352428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:46:47.352428+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:48:13.064124+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"854'16376","reported_seq":16850,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-04-20T17:46:56.084176+0000","last_change":"2024-04-20T17:46:42.217503+0000","last_active":"2024-04-20T17:46:56.084176+0000","last_peered":"2024-04-20T17:46:56.084176+0000","last_clean":"2024-04-20T17:46:56.084176+0000","last_became_active":"2024-04-20T17:43:18.798720+0000","last_became_peered":"2024-04-20T17:43:18.798720+0000","last_unstale":"2024-04-20T17:46:56.084176+0000","last_undegraded":"2024-04-20T17:46:56.084176+0000","last_fullsized":"2024-04-20T17:46:56.084176+0000","mapping_epoch":777,"log_start":"796'6300","ondisk_log_start":"796'6300","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"848'15781","last_scrub_stamp":"2024-04-20T17:46:42.217312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:46:42.217312+0000","objects_scrubbed":5166,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:47:58.490204+0000","scrub_duration":955,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298582016,"num_objects":4556,"num_object_clones":0,"num_object_copies":9112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13736,"num_write_kb":585344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"15'137","reported_seq":765,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-04-20T17:46:54.972601+0000","last_change":"2024-04-20T17:45:29.831148+0000","last_active":"2024-04-20T17:46:54.972601+0000","last_peered":"2024-04-20T17:46:54.972601+0000","last_clean":"2024-04-20T17:46:54.972601+0000","last_became_active":"2024-04-20T17:45:29.827577+0000","last_became_peered":"2024-04-20T17:45:29.827577+0000","last_unstale":"2024-04-20T17:46:54.972601+0000","last_undegraded":"2024-04-20T17:46:54.972601+0000","last_fullsized":"2024-04-20T17:46:54.972601+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:43:41.817800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:43:41.817800+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:46:58.436472+0000","scrub_duration":261,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":17,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":616186,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.d","version":"854'13565","reported_seq":20241,"reported_epoch":854,"state":"active+remapped+backfilling","last_fresh":"2024-04-20T17:46:55.535025+0000","last_change":"2024-04-20T17:46:38.969112+0000","last_active":"2024-04-20T17:46:55.535025+0000","last_peered":"2024-04-20T17:46:55.535025+0000","last_clean":"2024-04-20T17:46:36.765437+0000","last_became_active":"2024-04-20T17:46:38.922114+0000","last_became_peered":"2024-04-20T17:46:38.922114+0000","last_unstale":"2024-04-20T17:46:55.535025+0000","last_undegraded":"2024-04-20T17:46:55.535025+0000","last_fullsized":"2024-04-20T17:46:55.535025+0000","mapping_epoch":844,"log_start":"788'3500","ondisk_log_start":"788'3500","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'7202","last_scrub_stamp":"2024-04-20T17:44:26.727159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:44:26.727159+0000","objects_scrubbed":7157,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1434,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303038488,"num_objects":4625,"num_object_clones":0,"num_object_copies":9250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1606,"num_objects_unfound":0,"num_objects_dirty":4625,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13565,"num_write_kb":582017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3213,"num_bytes_recovered":210501656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,2],"avail_no_missing":["11","2"],"object_location_counts":[{"shards":"2,11","objects":4625}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"15'137","reported_seq":735,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-04-20T17:46:52.978742+0000","last_change":"2024-04-20T17:46:47.973152+0000","last_active":"2024-04-20T17:46:52.978742+0000","last_peered":"2024-04-20T17:46:52.978742+0000","last_clean":"2024-04-20T17:46:52.978742+0000","last_became_active":"2024-04-20T17:45:32.666523+0000","last_became_peered":"2024-04-20T17:45:32.666523+0000","last_unstale":"2024-04-20T17:46:52.978742+0000","last_undegraded":"2024-04-20T17:46:52.978742+0000","last_fullsized":"2024-04-20T17:46:52.978742+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:46:47.973017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:46:47.973017+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:48:14.713397+0000","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.c","version":"854'13642","reported_seq":13883,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-04-20T17:46:56.692277+0000","last_change":"2024-04-20T17:44:21.757085+0000","last_active":"2024-04-20T17:46:56.692277+0000","last_peered":"2024-04-20T17:46:56.692277+0000","last_clean":"2024-04-20T17:46:56.692277+0000","last_became_active":"2024-04-20T17:43:18.602140+0000","last_became_peered":"2024-04-20T17:43:18.602140+0000","last_unstale":"2024-04-20T17:46:56.692277+0000","last_undegraded":"2024-04-20T17:46:56.692277+0000","last_fullsized":"2024-04-20T17:46:56.692277+0000","mapping_epoch":777,"log_start":"788'3600","ondisk_log_start":"788'3600","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'6852","last_scrub_stamp":"2024-04-20T17:44:21.756952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:44:21.756952+0000","objects_scrubbed":6797,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1549,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300941312,"num_objects":4592,"num_object_clones":0,"num_object_copies":9184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13642,"num_write_kb":583488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.2","version":"15'137","reported_seq":728,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-04-20T17:46:52.989573+0000","last_change":"2024-04-20T17:46:40.832264+0000","last_active":"2024-04-20T17:46:52.989573+0000","last_peered":"2024-04-20T17:46:52.989573+0000","last_clean":"2024-04-20T17:46:52.989573+0000","last_became_active":"2024-04-20T17:46:40.831961+0000","last_became_peered":"2024-04-20T17:46:40.831961+0000","last_unstale":"2024-04-20T17:46:52.989573+0000","last_undegraded":"2024-04-20T17:46:52.989573+0000","last_fullsized":"2024-04-20T17:46:52.989573+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:45:33.692098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:45:33.692098+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"853'13068","reported_seq":13442,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-04-20T17:46:53.833694+0000","last_change":"2024-04-20T17:46:51.661841+0000","last_active":"2024-04-20T17:46:53.833694+0000","last_peered":"2024-04-20T17:46:53.833694+0000","last_clean":"2024-04-20T17:46:53.833694+0000","last_became_active":"2024-04-20T17:43:18.801242+0000","last_became_peered":"2024-04-20T17:43:18.801242+0000","last_unstale":"2024-04-20T17:46:53.833694+0000","last_undegraded":"2024-04-20T17:46:53.833694+0000","last_fullsized":"2024-04-20T17:46:53.833694+0000","mapping_epoch":777,"log_start":"787'3000","ondisk_log_start":"787'3000","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"851'12972","last_scrub_stamp":"2024-04-20T17:46:51.661650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:46:51.661650+0000","objects_scrubbed":4659,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:48:19.802777+0000","scrub_duration":918,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795584,"num_objects":4544,"num_object_clones":0,"num_object_copies":9088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13068,"num_write_kb":563584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"15'137","reported_seq":740,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-04-20T17:46:55.130569+0000","last_change":"2024-04-20T17:46:45.259642+0000","last_active":"2024-04-20T17:46:55.130569+0000","last_peered":"2024-04-20T17:46:55.130569+0000","last_clean":"2024-04-20T17:46:55.130569+0000","last_became_active":"2024-04-20T17:45:29.815765+0000","last_became_peered":"2024-04-20T17:45:29.815765+0000","last_unstale":"2024-04-20T17:46:55.130569+0000","last_undegraded":"2024-04-20T17:46:55.130569+0000","last_fullsized":"2024-04-20T17:46:55.130569+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:46:45.259545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:46:45.259545+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:47:55.843268+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"853'13630","reported_seq":13950,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-04-20T17:46:53.858502+0000","last_change":"2024-04-20T17:46:04.194418+0000","last_active":"2024-04-20T17:46:53.858502+0000","last_peered":"2024-04-20T17:46:53.858502+0000","last_clean":"2024-04-20T17:46:53.858502+0000","last_became_active":"2024-04-20T17:43:18.798958+0000","last_became_peered":"2024-04-20T17:43:18.798958+0000","last_unstale":"2024-04-20T17:46:53.858502+0000","last_undegraded":"2024-04-20T17:46:53.858502+0000","last_fullsized":"2024-04-20T17:46:53.858502+0000","mapping_epoch":777,"log_start":"788'3600","ondisk_log_start":"788'3600","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"830'11644","last_scrub_stamp":"2024-04-20T17:46:04.193887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:46:04.193887+0000","objects_scrubbed":6734,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:47:14.177066+0000","scrub_duration":1525,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309329920,"num_objects":4720,"num_object_clones":0,"num_object_copies":9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13630,"num_write_kb":587200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2],"acting":[8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1","version":"15'137","reported_seq":735,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-04-20T17:46:52.978586+0000","last_change":"2024-04-20T17:46:51.053690+0000","last_active":"2024-04-20T17:46:52.978586+0000","last_peered":"2024-04-20T17:46:52.978586+0000","last_clean":"2024-04-20T17:46:52.978586+0000","last_became_active":"2024-04-20T17:45:32.782706+0000","last_became_peered":"2024-04-20T17:45:32.782706+0000","last_unstale":"2024-04-20T17:46:52.978586+0000","last_undegraded":"2024-04-20T17:46:52.978586+0000","last_fullsized":"2024-04-20T17:46:52.978586+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:46:51.053551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:46:51.053551+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:47:56.169166+0000","scrub_duration":191,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.8","version":"853'13528","reported_seq":13844,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-04-20T17:46:54.505037+0000","last_change":"2024-04-20T17:46:43.076442+0000","last_active":"2024-04-20T17:46:54.505037+0000","last_peered":"2024-04-20T17:46:54.505037+0000","last_clean":"2024-04-20T17:46:54.505037+0000","last_became_active":"2024-04-20T17:43:18.610421+0000","last_became_peered":"2024-04-20T17:43:18.610421+0000","last_unstale":"2024-04-20T17:46:54.505037+0000","last_undegraded":"2024-04-20T17:46:54.505037+0000","last_fullsized":"2024-04-20T17:46:54.505037+0000","mapping_epoch":777,"log_start":"788'3500","ondisk_log_start":"788'3500","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"848'13067","last_scrub_stamp":"2024-04-20T17:46:43.076274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:46:43.076274+0000","objects_scrubbed":4996,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:48:11.524040+0000","scrub_duration":739,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296222720,"num_objects":4520,"num_object_clones":0,"num_object_copies":9040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13528,"num_write_kb":577536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.7","version":"854'13403","reported_seq":22435,"reported_epoch":854,"state":"active+recovering","last_fresh":"2024-04-20T17:46:55.559665+0000","last_change":"2024-04-20T17:44:53.433139+0000","last_active":"2024-04-20T17:46:55.559665+0000","last_peered":"2024-04-20T17:46:55.559665+0000","last_clean":"2024-04-20T17:44:51.743570+0000","last_became_active":"2024-04-20T17:44:53.388589+0000","last_became_peered":"2024-04-20T17:44:53.388589+0000","last_unstale":"2024-04-20T17:46:55.559665+0000","last_undegraded":"2024-04-20T17:46:55.559665+0000","last_fullsized":"2024-04-20T17:46:55.559665+0000","mapping_epoch":806,"log_start":"788'3400","ondisk_log_start":"788'3400","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":1599,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299040768,"num_objects":4563,"num_object_clones":0,"num_object_copies":9126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4348,"num_objects_unfound":0,"num_objects_dirty":4563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13403,"num_write_kb":574912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4423,"num_bytes_recovered":289865728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":["11","10"],"object_location_counts":[{"shards":"1,11","objects":215},{"shards":"10,11","objects":4348}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.6","version":"854'13559","reported_seq":13798,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-04-20T17:46:55.498256+0000","last_change":"2024-04-20T17:44:21.667932+0000","last_active":"2024-04-20T17:46:55.498256+0000","last_peered":"2024-04-20T17:46:55.498256+0000","last_clean":"2024-04-20T17:46:55.498256+0000","last_became_active":"2024-04-20T17:43:18.599894+0000","last_became_peered":"2024-04-20T17:43:18.599894+0000","last_unstale":"2024-04-20T17:46:55.498256+0000","last_undegraded":"2024-04-20T17:46:55.498256+0000","last_fullsized":"2024-04-20T17:46:55.498256+0000","mapping_epoch":777,"log_start":"788'3500","ondisk_log_start":"788'3500","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'6767","last_scrub_stamp":"2024-04-20T17:44:21.667758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:44:21.667758+0000","objects_scrubbed":6730,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1396,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298909696,"num_objects":4561,"num_object_clones":0,"num_object_copies":9122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13559,"num_write_kb":579840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.5","version":"853'13695","reported_seq":14025,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-04-20T17:46:54.928002+0000","last_change":"2024-04-20T17:46:03.923919+0000","last_active":"2024-04-20T17:46:54.928002+0000","last_peered":"2024-04-20T17:46:54.928002+0000","last_clean":"2024-04-20T17:46:54.928002+0000","last_became_active":"2024-04-20T17:43:18.801263+0000","last_became_peered":"2024-04-20T17:43:18.801263+0000","last_unstale":"2024-04-20T17:46:54.928002+0000","last_undegraded":"2024-04-20T17:46:54.928002+0000","last_fullsized":"2024-04-20T17:46:54.928002+0000","mapping_epoch":777,"log_start":"788'3600","ondisk_log_start":"788'3600","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"830'11661","last_scrub_stamp":"2024-04-20T17:46:03.923727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:46:03.923727+0000","objects_scrubbed":6747,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:47:12.357612+0000","scrub_duration":1316,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307953664,"num_objects":4699,"num_object_clones":0,"num_object_copies":9398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13695,"num_write_kb":588608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.4","version":"854'13447","reported_seq":13644,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-04-20T17:46:56.699960+0000","last_change":"2024-04-20T17:44:37.730301+0000","last_active":"2024-04-20T17:46:56.699960+0000","last_peered":"2024-04-20T17:46:56.699960+0000","last_clean":"2024-04-20T17:46:56.699960+0000","last_became_active":"2024-04-20T17:43:18.610249+0000","last_became_peered":"2024-04-20T17:43:18.610249+0000","last_unstale":"2024-04-20T17:46:56.699960+0000","last_undegraded":"2024-04-20T17:46:56.699960+0000","last_fullsized":"2024-04-20T17:46:56.699960+0000","mapping_epoch":777,"log_start":"788'3400","ondisk_log_start":"788'3400","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":3099,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299302912,"num_objects":4567,"num_object_clones":0,"num_object_copies":9134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13447,"num_write_kb":576448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5],"acting":[9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"15'137","reported_seq":720,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-04-20T17:46:54.981795+0000","last_change":"2024-04-20T17:45:32.669725+0000","last_active":"2024-04-20T17:46:54.981795+0000","last_peered":"2024-04-20T17:46:54.981795+0000","last_clean":"2024-04-20T17:46:54.981795+0000","last_became_active":"2024-04-20T17:45:32.669451+0000","last_became_peered":"2024-04-20T17:45:32.669451+0000","last_unstale":"2024-04-20T17:46:54.981795+0000","last_undegraded":"2024-04-20T17:46:54.981795+0000","last_fullsized":"2024-04-20T17:46:54.981795+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:43:41.817800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:43:41.817800+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:46:59.117786+0000","scrub_duration":261,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":16,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":240421,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.3","version":"854'13477","reported_seq":22536,"reported_epoch":854,"state":"active+recovering","last_fresh":"2024-04-20T17:46:55.526470+0000","last_change":"2024-04-20T17:44:53.425576+0000","last_active":"2024-04-20T17:46:55.526470+0000","last_peered":"2024-04-20T17:46:55.526470+0000","last_clean":"2024-04-20T17:44:51.724944+0000","last_became_active":"2024-04-20T17:44:53.356665+0000","last_became_peered":"2024-04-20T17:44:53.356665+0000","last_unstale":"2024-04-20T17:46:55.526470+0000","last_undegraded":"2024-04-20T17:46:55.526470+0000","last_fullsized":"2024-04-20T17:46:55.526470+0000","mapping_epoch":806,"log_start":"788'3400","ondisk_log_start":"788'3400","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":6749,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305463296,"num_objects":4661,"num_object_clones":0,"num_object_copies":9322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4424,"num_objects_unfound":0,"num_objects_dirty":4661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13477,"num_write_kb":580416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4411,"num_bytes_recovered":289079296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1],"avail_no_missing":["11","5"],"object_location_counts":[{"shards":"1,11","objects":237},{"shards":"5,11","objects":4424}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.2","version":"853'13451","reported_seq":13837,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-04-20T17:46:54.940293+0000","last_change":"2024-04-20T17:46:39.594522+0000","last_active":"2024-04-20T17:46:54.940293+0000","last_peered":"2024-04-20T17:46:54.940293+0000","last_clean":"2024-04-20T17:46:54.940293+0000","last_became_active":"2024-04-20T17:43:18.599198+0000","last_became_peered":"2024-04-20T17:43:18.599198+0000","last_unstale":"2024-04-20T17:46:54.940293+0000","last_undegraded":"2024-04-20T17:46:54.940293+0000","last_fullsized":"2024-04-20T17:46:54.940293+0000","mapping_epoch":777,"log_start":"788'3400","ondisk_log_start":"788'3400","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"845'12818","last_scrub_stamp":"2024-04-20T17:46:39.594176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:46:39.594176+0000","objects_scrubbed":5320,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:47:42.127586+0000","scrub_duration":897,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305987584,"num_objects":4669,"num_object_clones":0,"num_object_copies":9338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13451,"num_write_kb":579840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0],"acting":[10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.8","version":"15'137","reported_seq":728,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-04-20T17:46:54.982910+0000","last_change":"2024-04-20T17:45:33.710682+0000","last_active":"2024-04-20T17:46:54.982910+0000","last_peered":"2024-04-20T17:46:54.982910+0000","last_clean":"2024-04-20T17:46:54.982910+0000","last_became_active":"2024-04-20T17:45:32.644584+0000","last_became_peered":"2024-04-20T17:45:32.644584+0000","last_unstale":"2024-04-20T17:46:54.982910+0000","last_undegraded":"2024-04-20T17:46:54.982910+0000","last_fullsized":"2024-04-20T17:46:54.982910+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:45:33.710597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:45:33.710597+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:47:01.382512+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"854'13546","reported_seq":22779,"reported_epoch":854,"state":"active+recovering","last_fresh":"2024-04-20T17:46:56.710477+0000","last_change":"2024-04-20T17:44:53.450371+0000","last_active":"2024-04-20T17:46:56.710477+0000","last_peered":"2024-04-20T17:46:56.710477+0000","last_clean":"2024-04-20T17:44:51.758441+0000","last_became_active":"2024-04-20T17:44:53.399805+0000","last_became_peered":"2024-04-20T17:44:53.399805+0000","last_unstale":"2024-04-20T17:46:56.710477+0000","last_undegraded":"2024-04-20T17:46:56.710477+0000","last_fullsized":"2024-04-20T17:46:56.710477+0000","mapping_epoch":806,"log_start":"788'3500","ondisk_log_start":"788'3500","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:43:17.591778+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297009152,"num_objects":4532,"num_object_clones":0,"num_object_copies":9064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4280,"num_objects_unfound":0,"num_objects_dirty":4532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13546,"num_write_kb":578496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4533,"num_bytes_recovered":297074688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1],"acting":[9,1],"avail_no_missing":["9","2"],"object_location_counts":[{"shards":"1,9","objects":252},{"shards":"2,9","objects":4280}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"15'137","reported_seq":674,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-04-20T17:46:52.989573+0000","last_change":"2024-04-20T17:46:40.881971+0000","last_active":"2024-04-20T17:46:52.989573+0000","last_peered":"2024-04-20T17:46:52.989573+0000","last_clean":"2024-04-20T17:46:52.989573+0000","last_became_active":"2024-04-20T17:46:40.881713+0000","last_became_peered":"2024-04-20T17:46:40.881713+0000","last_unstale":"2024-04-20T17:46:52.989573+0000","last_undegraded":"2024-04-20T17:46:52.989573+0000","last_fullsized":"2024-04-20T17:46:52.989573+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:43:41.817800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:43:41.817800+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:46:57.197556+0000","scrub_duration":261,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":16,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":240422,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"853'13790","reported_seq":14108,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-04-20T17:46:54.513655+0000","last_change":"2024-04-20T17:46:42.209746+0000","last_active":"2024-04-20T17:46:54.513655+0000","last_peered":"2024-04-20T17:46:54.513655+0000","last_clean":"2024-04-20T17:46:54.513655+0000","last_became_active":"2024-04-20T17:43:18.610431+0000","last_became_peered":"2024-04-20T17:43:18.610431+0000","last_unstale":"2024-04-20T17:46:54.513655+0000","last_undegraded":"2024-04-20T17:46:54.513655+0000","last_fullsized":"2024-04-20T17:46:54.513655+0000","mapping_epoch":777,"log_start":"789'3700","ondisk_log_start":"789'3700","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"848'13260","last_scrub_stamp":"2024-04-20T17:46:42.209415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:46:42.209415+0000","objects_scrubbed":5192,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:48:09.664375+0000","scrub_duration":918,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303824896,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13790,"num_write_kb":589632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"853'13624","reported_seq":13943,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-04-20T17:46:54.906231+0000","last_change":"2024-04-20T17:46:06.860120+0000","last_active":"2024-04-20T17:46:54.906231+0000","last_peered":"2024-04-20T17:46:54.906231+0000","last_clean":"2024-04-20T17:46:54.906231+0000","last_became_active":"2024-04-20T17:43:18.610315+0000","last_became_peered":"2024-04-20T17:43:18.610315+0000","last_unstale":"2024-04-20T17:46:54.906231+0000","last_undegraded":"2024-04-20T17:46:54.906231+0000","last_fullsized":"2024-04-20T17:46:54.906231+0000","mapping_epoch":777,"log_start":"789'3600","ondisk_log_start":"789'3600","created":777,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":0,"last_scrub":"830'11694","last_scrub_stamp":"2024-04-20T17:46:06.859946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:43:17.591778+0000","last_clean_scrub_stamp":"2024-04-20T17:46:06.859946+0000","objects_scrubbed":6675,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:47:11.736411+0000","scrub_duration":1127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309854208,"num_objects":4728,"num_object_clones":0,"num_object_copies":9456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13624,"num_write_kb":587264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":672,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-04-20T17:46:52.989574+0000","last_change":"2024-04-20T17:46:40.852045+0000","last_active":"2024-04-20T17:46:52.989574+0000","last_peered":"2024-04-20T17:46:52.989574+0000","last_clean":"2024-04-20T17:46:52.989574+0000","last_became_active":"2024-04-20T17:46:40.851683+0000","last_became_peered":"2024-04-20T17:46:40.851683+0000","last_unstale":"2024-04-20T17:46:52.989574+0000","last_undegraded":"2024-04-20T17:46:52.989574+0000","last_fullsized":"2024-04-20T17:46:52.989574+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:43:41.817800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:43:41.817800+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:46:57.197541+0000","scrub_duration":261,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":16,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":240421,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4830658584,"num_objects":73711,"num_object_clones":0,"num_object_copies":147422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":14658,"num_objects_unfound":0,"num_objects_dirty":73711,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":216513,"num_write_kb":9287105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16580,"num_bytes_recovered":1086521368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10703941632,"data_stored":10703929416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160853,"ondisk_log_size":160853,"up":32,"acting":32,"num_store_stats":11},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":477366,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":39,"num_write":65,"num_write_kb":1068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1337450,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},"log_size":1507,"ondisk_log_size":1507,"up":22,"acting":22,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991728,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2341648,"kb_used_data":1471940,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":91403504,"statfs":{"total":95995035648,"available":93597188096,"internally_reserved":0,"allocated":1507266560,"data_stored":1506706444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":890567108},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,4,2,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253367,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2448832,"kb_used_data":1759604,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":91296320,"statfs":{"total":95995035648,"available":93487431680,"internally_reserved":0,"allocated":1801834496,"data_stored":1801284477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":705755584},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613116,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1512716,"kb_used_data":881792,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":92232436,"statfs":{"total":95995035648,"available":94446014464,"internally_reserved":0,"allocated":902955008,"data_stored":902398972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":646052288},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809345,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1266116,"kb_used_data":896180,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92479036,"statfs":{"total":95995035648,"available":94698532864,"internally_reserved":0,"allocated":917688320,"data_stored":917138309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":799,"seq":3431678869536,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":808000,"kb_used_data":741432,"kb_used_omap":11,"kb_used_meta":66548,"kb_avail":92937152,"statfs":{"total":95995035648,"available":95167643648,"internally_reserved":0,"allocated":759226368,"data_stored":758672293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11990,"internal_metadata":68145450},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":535,"seq":2297807503576,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":42832,"kb_used_data":1088,"kb_used_omap":170,"kb_used_meta":41557,"kb_avail":93702320,"statfs":{"total":95995035648,"available":95951175680,"internally_reserved":0,"allocated":1114112,"data_stored":558076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":174628,"internal_metadata":42554844},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":659,"seq":2830383448198,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":206140,"kb_used_data":44660,"kb_used_omap":4,"kb_used_meta":161467,"kb_avail":93539012,"statfs":{"total":95995035648,"available":95783948288,"internally_reserved":0,"allocated":45731840,"data_stored":45181829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4410,"internal_metadata":165342918},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738921,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1314316,"kb_used_data":593152,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":92430836,"statfs":{"total":95995035648,"available":94649176064,"internally_reserved":0,"allocated":607387648,"data_stored":606831612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394210,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2142740,"kb_used_data":1733388,"kb_used_omap":202,"kb_used_meta":409141,"kb_avail":91602412,"statfs":{"total":95995035648,"available":93800869888,"internally_reserved":0,"allocated":1774989312,"data_stored":1774425652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":418960855},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1036096,"kb_used_data":588596,"kb_used_omap":4,"kb_used_meta":447483,"kb_avail":92709056,"statfs":{"total":95995035648,"available":94934073344,"internally_reserved":0,"allocated":602722304,"data_stored":602172293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":458223303},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986708,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1167168,"kb_used_data":590388,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92577984,"statfs":{"total":95995035648,"available":94799855616,"internally_reserved":0,"allocated":604557312,"data_stored":604007301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790478,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2454164,"kb_used_data":1741320,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":91290988,"statfs":{"total":95995035648,"available":93481971712,"internally_reserved":0,"allocated":1783111680,"data_stored":1782552092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":600375296,"data_stored":600375296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":44761088,"data_stored":44761088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1773670400,"data_stored":1773666328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":600047616,"data_stored":600047616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":598409216,"data_stored":598409216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1781399552,"data_stored":1781399552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":190320640,"data_stored":190316568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":914817024,"data_stored":914817024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":897187840,"data_stored":897187840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1799159808,"data_stored":1799159808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1503793152,"data_stored":1503789080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:46:59.840 INFO:tasks.thrashosds.thrasher:Setting 8.b to [3, 10, 9, 0] 2024-04-20T17:46:59.841 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '8.b', '3', '10', '9', '0'] 2024-04-20T17:46:59.841 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pg-upmap-items 8.b 3 10 9 0 2024-04-20T17:46:59.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:59 smithi005 ceph-mon[39562]: pgmap v2270: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 472 op/s; 14658/147424 objects misplaced (9.943%); 15 MiB/s, 238 objects/s recovering 2024-04-20T17:46:59.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:59 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/741443089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:46:59.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:59.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:59.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:59.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:46:59.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:46:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:00.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:59 smithi106 ceph-mon[44423]: pgmap v2270: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 472 op/s; 14658/147424 objects misplaced (9.943%); 15 MiB/s, 238 objects/s recovering 2024-04-20T17:47:00.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:59 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/741443089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:47:00.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:00.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:00.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:00.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:00.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:46:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:00.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:59 smithi171 ceph-mon[44189]: pgmap v2270: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 472 op/s; 14658/147424 objects misplaced (9.943%); 15 MiB/s, 238 objects/s recovering 2024-04-20T17:47:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:59 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/741443089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:47:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:00.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:46:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:00.089 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:47:00.569 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:47:00.576 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:47:00.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:00 smithi005 ceph-mon[39562]: from='client.27878 v1:172.21.15.5:0/816782897' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:47:00.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:00.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:00.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:00.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:00.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:00.871 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:00.872 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:00 smithi106 ceph-mon[44423]: from='client.27878 v1:172.21.15.5:0/816782897' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:47:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:00 smithi171 ceph-mon[44189]: from='client.27878 v1:172.21.15.5:0/816782897' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:47:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:01.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:01.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:01.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:01 smithi005 ceph-mon[39562]: pgmap v2271: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 725 op/s; 13097/142388 objects misplaced (9.198%); 25 MiB/s, 396 objects/s recovering 2024-04-20T17:47:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:01 smithi106 ceph-mon[44423]: pgmap v2271: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 725 op/s; 13097/142388 objects misplaced (9.198%); 25 MiB/s, 396 objects/s recovering 2024-04-20T17:47:02.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:01 smithi171 ceph-mon[44189]: pgmap v2271: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 725 op/s; 13097/142388 objects misplaced (9.198%); 25 MiB/s, 396 objects/s recovering 2024-04-20T17:47:02.457 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:47:02.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:02.468 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:47:02.513 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:47:02.569 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:47:02.638 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:02.750 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:47:02.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:02.758 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:47:02.859 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:47:02.916 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:02.963 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:47:02.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:02.969 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:47:03.070 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:47:03.100 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:03.124 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:47:03.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:03.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:47:03.228 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:47:03.283 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:03.454 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:47:03.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:03.460 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:47:03.485 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:47:03.561 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:47:03.589 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:03.609 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:47:03.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:03.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:47:03.713 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:47:03.763 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:03.939 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:03 smithi005 ceph-mon[39562]: pgmap v2272: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 688 op/s; 12902/140672 objects misplaced (9.172%); 22 MiB/s, 348 objects/s recovering 2024-04-20T17:47:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:03 smithi106 ceph-mon[44423]: pgmap v2272: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 688 op/s; 12902/140672 objects misplaced (9.172%); 22 MiB/s, 348 objects/s recovering 2024-04-20T17:47:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:03 smithi171 ceph-mon[44189]: pgmap v2272: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 688 op/s; 12902/140672 objects misplaced (9.172%); 22 MiB/s, 348 objects/s recovering 2024-04-20T17:47:04.332 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:47:04.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:04.335 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:47:04.436 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:47:04.465 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:04.488 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:47:04.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:04.494 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:47:04.595 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:47:04.622 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:04.641 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:47:04.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:04.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:47:04.735 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:47:04.735 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:47:04.737 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:47:04.744 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:47:04.780 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:04.806 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:47:04.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:04.809 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:47:04.910 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:47:04.964 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:04.998 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:47:05.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:05.007 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:47:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:04 smithi106 ceph-mon[44423]: osdmap e855: 12 total, 12 up, 12 in 2024-04-20T17:47:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:04 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2189883166' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:47:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:04 smithi106 ceph-mon[44423]: from='client.27890 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:47:05.015 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:04 smithi171 ceph-mon[44189]: osdmap e855: 12 total, 12 up, 12 in 2024-04-20T17:47:05.015 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:04 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2189883166' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:47:05.015 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:04 smithi171 ceph-mon[44189]: from='client.27890 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:47:05.108 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:47:05.144 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:05.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:04 smithi005 ceph-mon[39562]: osdmap e855: 12 total, 12 up, 12 in 2024-04-20T17:47:05.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:04 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2189883166' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:47:05.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:04 smithi005 ceph-mon[39562]: from='client.27890 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:47:05.753 INFO:teuthology.orchestra.run.smithi005.stderr:set 8.b pg_upmap_items mapping to [3->10,9->0] 2024-04-20T17:47:05.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:05 smithi005 ceph-mon[39562]: pgmap v2274: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s; 12902/138976 objects misplaced (9.284%); 22 MiB/s, 348 objects/s recovering 2024-04-20T17:47:05.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:05 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:47:05.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:05 smithi005 ceph-mon[39562]: from='client.27890 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:47:05.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:05 smithi005 ceph-mon[39562]: osdmap e856: 12 total, 12 up, 12 in 2024-04-20T17:47:05.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:05 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4081907744' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.b", "id": [3, 10, 9, 0]}]: dispatch 2024-04-20T17:47:05.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:05 smithi005 ceph-mon[39562]: from='client.27893 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.b", "id": [3, 10, 9, 0]}]: dispatch 2024-04-20T17:47:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:05 smithi106 ceph-mon[44423]: pgmap v2274: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s; 12902/138976 objects misplaced (9.284%); 22 MiB/s, 348 objects/s recovering 2024-04-20T17:47:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:05 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:47:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:05 smithi106 ceph-mon[44423]: from='client.27890 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:47:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:05 smithi106 ceph-mon[44423]: osdmap e856: 12 total, 12 up, 12 in 2024-04-20T17:47:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:05 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4081907744' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.b", "id": [3, 10, 9, 0]}]: dispatch 2024-04-20T17:47:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:05 smithi106 ceph-mon[44423]: from='client.27893 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.b", "id": [3, 10, 9, 0]}]: dispatch 2024-04-20T17:47:06.092 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:05 smithi171 ceph-mon[44189]: pgmap v2274: 27 pgs: 1 active+remapped+backfilling, 3 active+recovering, 23 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s; 12902/138976 objects misplaced (9.284%); 22 MiB/s, 348 objects/s recovering 2024-04-20T17:47:06.093 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:05 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:47:06.093 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:05 smithi171 ceph-mon[44189]: from='client.27890 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:47:06.093 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:05 smithi171 ceph-mon[44189]: osdmap e856: 12 total, 12 up, 12 in 2024-04-20T17:47:06.093 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:05 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4081907744' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.b", "id": [3, 10, 9, 0]}]: dispatch 2024-04-20T17:47:06.093 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:05 smithi171 ceph-mon[44189]: from='client.27893 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.b", "id": [3, 10, 9, 0]}]: dispatch 2024-04-20T17:47:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:06 smithi106 ceph-mon[44423]: from='client.27893 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "8.b", "id": [3, 10, 9, 0]}]': finished 2024-04-20T17:47:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:06 smithi106 ceph-mon[44423]: osdmap e857: 12 total, 12 up, 12 in 2024-04-20T17:47:07.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:06 smithi106 ceph-mon[44423]: osdmap e858: 12 total, 12 up, 12 in 2024-04-20T17:47:07.055 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:06 smithi005 ceph-mon[39562]: from='client.27893 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "8.b", "id": [3, 10, 9, 0]}]': finished 2024-04-20T17:47:07.055 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:06 smithi005 ceph-mon[39562]: osdmap e857: 12 total, 12 up, 12 in 2024-04-20T17:47:07.055 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:06 smithi005 ceph-mon[39562]: osdmap e858: 12 total, 12 up, 12 in 2024-04-20T17:47:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:06 smithi171 ceph-mon[44189]: from='client.27893 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "8.b", "id": [3, 10, 9, 0]}]': finished 2024-04-20T17:47:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:06 smithi171 ceph-mon[44189]: osdmap e857: 12 total, 12 up, 12 in 2024-04-20T17:47:07.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:06 smithi171 ceph-mon[44189]: osdmap e858: 12 total, 12 up, 12 in 2024-04-20T17:47:07.124 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:47:07.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:07.131 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:47:07.232 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:47:07.260 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:07.280 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:47:07.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:07.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:47:07.386 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:47:07.412 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:07.433 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:47:07.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:07.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:47:07.538 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:47:07.592 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:07.825 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:07 smithi005 ceph-mon[39562]: pgmap v2277: 27 pgs: 3 active+recovering, 24 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 558 op/s; 11980/135684 objects misplaced (8.829%); 12 MiB/s, 184 objects/s recovering 2024-04-20T17:47:07.841 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:07 smithi171 ceph-mon[44189]: pgmap v2277: 27 pgs: 3 active+recovering, 24 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 558 op/s; 11980/135684 objects misplaced (8.829%); 12 MiB/s, 184 objects/s recovering 2024-04-20T17:47:07.854 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:47:07.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:07.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:47:07.962 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:47:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:07 smithi106 ceph-mon[44423]: pgmap v2277: 27 pgs: 3 active+recovering, 24 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 558 op/s; 11980/135684 objects misplaced (8.829%); 12 MiB/s, 184 objects/s recovering 2024-04-20T17:47:08.030 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:08.081 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:47:08.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:08.087 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:47:08.097 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-20T17:47:08.187 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:47:08.225 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:08.252 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:47:08.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:08.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:47:08.291 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:47:08.313 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:47:08.339 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:47:08.356 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:47:08.365 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:47:08.365 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:47:08.398 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:08.403 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:47:08.422 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:47:08.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:08.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:47:08.437 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:47:08.462 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:47:08.493 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:47:08.526 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:47:08.529 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:47:08.571 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:47:08.584 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:08.592 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:47:08.605 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:47:08.630 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:47:08.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:08.637 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:47:08.643 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:47:08.645 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:47:08.645 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:47:08.645 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:47:08.645 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:47:08.645 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:47:08.645 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:47:08.645 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:47:08.645 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:47:08.645 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:47:08.645 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:47:08.645 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:47:08.645 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:47:08.737 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:47:08.789 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:08.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:08 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2959089805' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:47:08.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:08.919 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:47:08.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:08 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2959089805' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:47:08.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:08 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2959089805' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:47:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:09.656 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:47:09.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:09.663 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:47:09.764 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:47:09.818 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:09.962 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:09.962 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:09.962 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:09.962 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:09.962 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:09.962 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:09.962 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:09.962 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:09.963 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:09.963 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:09.963 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:09 smithi171 ceph-mon[44189]: pgmap v2279: 27 pgs: 3 active+recovering, 24 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 627 op/s; 11745/134510 objects misplaced (8.732%); 15 MiB/s, 233 objects/s recovering 2024-04-20T17:47:09.963 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:10.207 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:10.207 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:10.207 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:10.207 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:10.207 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:10.207 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:10.207 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:10.208 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:10.208 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:10.208 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:10.208 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:09 smithi005 ceph-mon[39562]: pgmap v2279: 27 pgs: 3 active+recovering, 24 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 627 op/s; 11745/134510 objects misplaced (8.732%); 15 MiB/s, 233 objects/s recovering 2024-04-20T17:47:10.208 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:10.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:10.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:09 smithi106 ceph-mon[44423]: pgmap v2279: 27 pgs: 3 active+recovering, 24 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 627 op/s; 11745/134510 objects misplaced (8.732%); 15 MiB/s, 233 objects/s recovering 2024-04-20T17:47:10.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:10.674 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:47:10.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:10.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:47:10.778 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:47:10.806 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:10.825 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:47:10.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:10.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:47:10.929 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:47:10.956 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:10.975 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:47:10.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:10.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:47:11.080 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:47:11.119 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:11.226 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:47:11.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:11.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:47:11.330 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:47:11.383 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:11.524 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:47:11.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:11.528 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:47:11.629 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:47:11.694 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:11.749 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:47:11.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:11.756 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:47:11.807 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:47:11.811 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:47:11.856 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:47:11.882 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:11.903 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:47:11.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:11.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:47:12.007 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:47:12.074 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:11 smithi171 ceph-mon[44189]: pgmap v2280: 27 pgs: 3 active+recovering, 24 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 629 op/s; 11745/131424 objects misplaced (8.937%); 12 MiB/s, 191 objects/s recovering 2024-04-20T17:47:12.075 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:11 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/752205217' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:47:12.075 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:11 smithi171 ceph-mon[44189]: from='client.27631 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:47:12.100 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:12.127 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:47:12.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:12.132 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:47:12.200 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:11 smithi005 ceph-mon[39562]: pgmap v2280: 27 pgs: 3 active+recovering, 24 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 629 op/s; 11745/131424 objects misplaced (8.937%); 12 MiB/s, 191 objects/s recovering 2024-04-20T17:47:12.200 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:11 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/752205217' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:47:12.200 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:11 smithi005 ceph-mon[39562]: from='client.27631 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:47:12.233 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:47:12.257 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:11 smithi106 ceph-mon[44423]: pgmap v2280: 27 pgs: 3 active+recovering, 24 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 629 op/s; 11745/131424 objects misplaced (8.937%); 12 MiB/s, 191 objects/s recovering 2024-04-20T17:47:12.257 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:11 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/752205217' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:47:12.257 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:11 smithi106 ceph-mon[44423]: from='client.27631 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:47:12.259 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:12.277 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:47:12.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:12.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:47:12.282 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:47:12.282 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:47:12.282 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-04-20T17:47:12.282 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-04-20T17:47:12.282 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:47:12.381 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:47:12.408 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:12.426 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:47:12.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:12.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:47:12.531 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:47:12.572 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:12.592 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 8, 9, 10, 11, 1, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 4, 5, 8, 9, 10, 11, 2, 6, 3, 1, 7] 2024-04-20T17:47:12.592 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:47:12.593 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-04-20T17:47:12.593 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-20T17:47:12.593 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-20T17:47:12.593 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-04-20T17:47:12.604 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:47:12.612 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:47:12.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:12.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:47:12.721 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:47:12.748 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:12.770 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:47:12.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:12.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:47:12.874 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:47:12.928 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:12.954 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:47:12.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:12.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:47:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:12 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:47:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:12 smithi106 ceph-mon[44423]: from='client.27631 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:47:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:12 smithi106 ceph-mon[44423]: osdmap e859: 12 total, 12 up, 12 in 2024-04-20T17:47:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:12 smithi106 ceph-mon[44423]: 8.9 scrub starts 2024-04-20T17:47:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:12 smithi106 ceph-mon[44423]: pgmap v2282: 27 pgs: 3 active+recovering, 24 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 640 op/s; 11172/127196 objects misplaced (8.783%); 7.3 MiB/s, 117 objects/s recovering 2024-04-20T17:47:13.044 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:47:13.058 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:47:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:12 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:47:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:12 smithi171 ceph-mon[44189]: from='client.27631 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:47:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:12 smithi171 ceph-mon[44189]: osdmap e859: 12 total, 12 up, 12 in 2024-04-20T17:47:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:12 smithi171 ceph-mon[44189]: 8.9 scrub starts 2024-04-20T17:47:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:12 smithi171 ceph-mon[44189]: pgmap v2282: 27 pgs: 3 active+recovering, 24 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 640 op/s; 11172/127196 objects misplaced (8.783%); 7.3 MiB/s, 117 objects/s recovering 2024-04-20T17:47:13.087 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:13.136 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:12 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:47:13.136 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:12 smithi005 ceph-mon[39562]: from='client.27631 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:47:13.137 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:12 smithi005 ceph-mon[39562]: osdmap e859: 12 total, 12 up, 12 in 2024-04-20T17:47:13.137 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:12 smithi005 ceph-mon[39562]: 8.9 scrub starts 2024-04-20T17:47:13.137 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:12 smithi005 ceph-mon[39562]: pgmap v2282: 27 pgs: 3 active+recovering, 24 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 640 op/s; 11172/127196 objects misplaced (8.783%); 7.3 MiB/s, 117 objects/s recovering 2024-04-20T17:47:13.338 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:47:13.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:13.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:47:13.442 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:47:13.482 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:13.817 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:13 smithi171 ceph-mon[44189]: 8.9 scrub ok 2024-04-20T17:47:14.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:13 smithi106 ceph-mon[44423]: 8.9 scrub ok 2024-04-20T17:47:14.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:13 smithi005 ceph-mon[39562]: 8.9 scrub ok 2024-04-20T17:47:14.536 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:47:14.618 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:47:14.645 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:47:14.668 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:47:14.697 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:47:14.726 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:47:14.747 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:47:14.750 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:47:14.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:14.753 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:47:14.780 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:47:14.811 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:47:14.844 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:47:14.854 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:47:14.875 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:47:14.911 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:14 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:47:14.911 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:14 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:47:14.911 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:14 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/675723331' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:47:14.911 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:14 smithi005 ceph-mon[39562]: pgmap v2283: 27 pgs: 3 active+recovering, 24 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 642 op/s; 11172/125394 objects misplaced (8.910%); 6.3 MiB/s, 100 objects/s recovering 2024-04-20T17:47:14.913 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:14.924 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:47:14.949 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:47:14.954 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:47:14.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:14.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:47:14.961 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:47:14.967 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:47:14.967 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:47:14.968 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:47:14.968 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:47:14.968 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:47:14.968 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:47:14.968 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:47:14.968 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:47:14.968 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:47:14.968 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:47:14.968 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:47:14.968 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:47:15.061 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:47:15.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:14 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:47:15.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:14 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:47:15.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:14 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/675723331' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:47:15.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:14 smithi171 ceph-mon[44189]: pgmap v2283: 27 pgs: 3 active+recovering, 24 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 642 op/s; 11172/125394 objects misplaced (8.910%); 6.3 MiB/s, 100 objects/s recovering 2024-04-20T17:47:15.089 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:14 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:47:15.089 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:14 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:47:15.089 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:14 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/675723331' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:47:15.089 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:14 smithi106 ceph-mon[44423]: pgmap v2283: 27 pgs: 3 active+recovering, 24 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 642 op/s; 11172/125394 objects misplaced (8.910%); 6.3 MiB/s, 100 objects/s recovering 2024-04-20T17:47:15.092 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:15.102 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:47:15.102 INFO:teuthology.orchestra.run.smithi106.stdout: "success": "" 2024-04-20T17:47:15.102 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:47:15.116 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:47:15.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:15.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:47:15.221 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:47:15.248 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:15.397 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:47:15.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:15.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:47:15.501 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:47:15.547 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:15.840 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:15 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:47:15.840 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:15 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:47:15.840 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:15.840 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:15.841 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:15.841 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:15.841 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:15.841 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:15.841 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:15.841 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:15.841 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:15 smithi005 ceph-mon[39562]: osdmap e860: 12 total, 12 up, 12 in 2024-04-20T17:47:15.841 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:15.841 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:15.841 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:15.841 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:15.841 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:15 smithi005 ceph-mon[39562]: 1.9 scrub starts 2024-04-20T17:47:15.841 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:15 smithi005 ceph-mon[39562]: 1.9 scrub ok 2024-04-20T17:47:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:15 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:47:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:15 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:47:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:16.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:16.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:16.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:16.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:16.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:15 smithi106 ceph-mon[44423]: osdmap e860: 12 total, 12 up, 12 in 2024-04-20T17:47:16.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:16.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:16.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:16.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:16.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:15 smithi106 ceph-mon[44423]: 1.9 scrub starts 2024-04-20T17:47:16.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:15 smithi106 ceph-mon[44423]: 1.9 scrub ok 2024-04-20T17:47:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:15 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:47:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:15 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:47:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:15 smithi171 ceph-mon[44189]: osdmap e860: 12 total, 12 up, 12 in 2024-04-20T17:47:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:16.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:15 smithi171 ceph-mon[44189]: 1.9 scrub starts 2024-04-20T17:47:16.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:15 smithi171 ceph-mon[44189]: 1.9 scrub ok 2024-04-20T17:47:16.845 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:16 smithi005 ceph-mon[39562]: 8.a scrub starts 2024-04-20T17:47:16.845 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:16 smithi005 ceph-mon[39562]: 1.2 scrub starts 2024-04-20T17:47:16.845 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:16 smithi005 ceph-mon[39562]: 1.2 scrub ok 2024-04-20T17:47:16.845 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:16 smithi005 ceph-mon[39562]: pgmap v2285: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering, 23 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 818 op/s; 10731/121384 objects misplaced (8.841%); 7.9 MiB/s, 126 objects/s recovering 2024-04-20T17:47:17.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:16 smithi106 ceph-mon[44423]: 8.a scrub starts 2024-04-20T17:47:17.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:16 smithi106 ceph-mon[44423]: 1.2 scrub starts 2024-04-20T17:47:17.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:16 smithi106 ceph-mon[44423]: 1.2 scrub ok 2024-04-20T17:47:17.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:16 smithi106 ceph-mon[44423]: pgmap v2285: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering, 23 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 818 op/s; 10731/121384 objects misplaced (8.841%); 7.9 MiB/s, 126 objects/s recovering 2024-04-20T17:47:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:16 smithi171 ceph-mon[44189]: 8.a scrub starts 2024-04-20T17:47:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:16 smithi171 ceph-mon[44189]: 1.2 scrub starts 2024-04-20T17:47:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:16 smithi171 ceph-mon[44189]: 1.2 scrub ok 2024-04-20T17:47:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:16 smithi171 ceph-mon[44189]: pgmap v2285: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering, 23 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 818 op/s; 10731/121384 objects misplaced (8.841%); 7.9 MiB/s, 126 objects/s recovering 2024-04-20T17:47:17.473 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-20T17:47:17.520 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:47:17.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:17.524 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:47:17.625 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:47:17.686 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:17.748 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:47:17.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:17.760 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:47:17.861 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:47:17.880 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:47:17.889 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:17.911 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:47:17.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:17.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:47:17.929 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:47:18.014 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:47:18.052 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:17 smithi171 ceph-mon[44189]: 1.8 scrub starts 2024-04-20T17:47:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:17 smithi171 ceph-mon[44189]: 1.8 scrub ok 2024-04-20T17:47:18.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:17 smithi171 ceph-mon[44189]: 8.a scrub ok 2024-04-20T17:47:18.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:17 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3566056377' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:47:18.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:17 smithi171 ceph-mon[44189]: from='client.27911 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:47:18.137 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:47:18.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:17 smithi005 ceph-mon[39562]: 1.8 scrub starts 2024-04-20T17:47:18.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:17 smithi005 ceph-mon[39562]: 1.8 scrub ok 2024-04-20T17:47:18.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:17 smithi005 ceph-mon[39562]: 8.a scrub ok 2024-04-20T17:47:18.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:17 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3566056377' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:47:18.140 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:17 smithi005 ceph-mon[39562]: from='client.27911 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:47:18.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:18.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:47:18.141 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:47:18.242 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:47:18.304 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:17 smithi106 ceph-mon[44423]: 1.8 scrub starts 2024-04-20T17:47:18.304 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:17 smithi106 ceph-mon[44423]: 1.8 scrub ok 2024-04-20T17:47:18.304 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:17 smithi106 ceph-mon[44423]: 8.a scrub ok 2024-04-20T17:47:18.305 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:17 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3566056377' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:47:18.305 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:17 smithi106 ceph-mon[44423]: from='client.27911 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:47:18.365 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:18.416 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:47:18.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:18.426 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:47:18.527 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:47:18.583 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:18.616 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:47:18.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:18.622 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:47:18.723 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:47:18.762 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:18.769 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:47:18.802 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:47:18.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:18.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:47:18.907 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:47:18.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:18 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:47:18.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:18 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:47:18.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:18 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:47:18.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:18 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:47:18.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:18 smithi005 ceph-mon[39562]: from='client.27911 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:47:18.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:18 smithi005 ceph-mon[39562]: osdmap e861: 12 total, 12 up, 12 in 2024-04-20T17:47:18.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:18 smithi005 ceph-mon[39562]: pgmap v2287: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering, 23 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 627 op/s; 10506/120032 objects misplaced (8.753%); 7.8 MiB/s, 125 objects/s recovering 2024-04-20T17:47:18.988 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:18 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:47:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:18 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:47:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:18 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:47:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:18 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:47:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:18 smithi106 ceph-mon[44423]: from='client.27911 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:47:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:18 smithi106 ceph-mon[44423]: osdmap e861: 12 total, 12 up, 12 in 2024-04-20T17:47:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:18 smithi106 ceph-mon[44423]: pgmap v2287: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering, 23 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 627 op/s; 10506/120032 objects misplaced (8.753%); 7.8 MiB/s, 125 objects/s recovering 2024-04-20T17:47:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:18 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:47:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:18 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:47:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:18 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:47:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:18 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:47:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:18 smithi171 ceph-mon[44189]: from='client.27911 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:47:19.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:18 smithi171 ceph-mon[44189]: osdmap e861: 12 total, 12 up, 12 in 2024-04-20T17:47:19.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:18 smithi171 ceph-mon[44189]: pgmap v2287: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering, 23 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 627 op/s; 10506/120032 objects misplaced (8.753%); 7.8 MiB/s, 125 objects/s recovering 2024-04-20T17:47:20.103 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:19 smithi005 ceph-mon[39562]: 8.5 scrub starts 2024-04-20T17:47:20.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:19 smithi106 ceph-mon[44423]: 8.5 scrub starts 2024-04-20T17:47:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:19 smithi171 ceph-mon[44189]: 8.5 scrub starts 2024-04-20T17:47:20.693 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:47:20.762 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:47:20.797 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:47:20.836 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:47:20.871 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:47:20.903 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:47:20.927 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:47:20.937 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 8, 9, 10, 11, 1, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 4, 5, 8, 9, 10, 11, 2, 6, 3, 1, 7] 2024-04-20T17:47:20.938 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:47:20.938 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-04-20T17:47:20.938 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-20T17:47:20.938 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-20T17:47:20.938 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-04-20T17:47:20.968 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:47:20.984 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:20 smithi005 ceph-mon[39562]: 8.5 scrub ok 2024-04-20T17:47:20.984 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:20 smithi005 ceph-mon[39562]: pgmap v2288: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering, 23 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 630 op/s; 10506/117048 objects misplaced (8.976%); 5.1 MiB/s, 83 objects/s recovering 2024-04-20T17:47:20.984 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:20 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2059025859' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:47:21.004 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:47:21.046 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:47:21.079 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:47:21.122 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:47:21.133 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:47:21.146 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:47:21.153 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:47:21.153 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:47:21.153 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:47:21.153 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:47:21.153 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:47:21.153 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:47:21.153 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:47:21.154 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:47:21.154 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:47:21.154 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:47:21.154 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:47:21.154 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:47:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:20 smithi106 ceph-mon[44423]: 8.5 scrub ok 2024-04-20T17:47:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:20 smithi106 ceph-mon[44423]: pgmap v2288: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering, 23 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 630 op/s; 10506/117048 objects misplaced (8.976%); 5.1 MiB/s, 83 objects/s recovering 2024-04-20T17:47:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:20 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2059025859' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:47:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:20 smithi171 ceph-mon[44189]: 8.5 scrub ok 2024-04-20T17:47:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:20 smithi171 ceph-mon[44189]: pgmap v2288: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering, 23 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 630 op/s; 10506/117048 objects misplaced (8.976%); 5.1 MiB/s, 83 objects/s recovering 2024-04-20T17:47:21.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:20 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2059025859' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:47:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:21 smithi005 ceph-mon[39562]: osdmap e862: 12 total, 12 up, 12 in 2024-04-20T17:47:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:21 smithi106 ceph-mon[44423]: osdmap e862: 12 total, 12 up, 12 in 2024-04-20T17:47:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:21 smithi171 ceph-mon[44189]: osdmap e862: 12 total, 12 up, 12 in 2024-04-20T17:47:22.897 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:22 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.898 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:22 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:22.898 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:22 smithi005 ceph-mon[39562]: pgmap v2290: 27 pgs: 3 active+recovering, 24 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 781 op/s; 9837/113152 objects misplaced (8.694%); 11 MiB/s, 170 objects/s recovering 2024-04-20T17:47:23.181 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:22 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:23.181 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:22 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:23.182 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:22 smithi171 ceph-mon[44189]: pgmap v2290: 27 pgs: 3 active+recovering, 24 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 781 op/s; 9837/113152 objects misplaced (8.694%); 11 MiB/s, 170 objects/s recovering 2024-04-20T17:47:23.197 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:47:23.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:23.204 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:47:23.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:22 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:23.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:22 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:23.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:22 smithi106 ceph-mon[44423]: pgmap v2290: 27 pgs: 3 active+recovering, 24 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 781 op/s; 9837/113152 objects misplaced (8.694%); 11 MiB/s, 170 objects/s recovering 2024-04-20T17:47:23.305 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:47:23.342 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:24.233 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:47:24.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:24.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:47:24.337 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:47:24.364 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:24.383 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:47:24.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:24.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:47:24.487 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:47:24.514 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:24.535 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:47:24.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:24.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:47:24.639 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:47:24.676 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:24.737 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:47:24.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:24.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:47:24.842 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:47:24.898 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:24.925 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:47:24.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:24.934 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:47:25.036 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:47:25.064 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:25.085 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:47:25.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:25.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:47:25.189 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:47:25.242 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:25.349 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:47:25.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:25.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:47:25.454 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:47:25.492 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:25.519 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:47:25.525 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:47:25.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:25.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:47:25.629 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:47:25.656 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:25.674 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:47:25.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:25.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:47:25.778 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:47:25.805 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:25.824 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:47:25.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:25.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:47:25.899 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:25 smithi005 ceph-mon[39562]: pgmap v2291: 27 pgs: 3 active+recovering, 24 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 625 op/s; 9837/111340 objects misplaced (8.835%); 7.0 MiB/s, 111 objects/s recovering 2024-04-20T17:47:25.928 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:47:25.981 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:26.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:25 smithi106 ceph-mon[44423]: pgmap v2291: 27 pgs: 3 active+recovering, 24 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 625 op/s; 9837/111340 objects misplaced (8.835%); 7.0 MiB/s, 111 objects/s recovering 2024-04-20T17:47:26.014 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:47:26.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:26.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:47:26.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:25 smithi171 ceph-mon[44189]: pgmap v2291: 27 pgs: 3 active+recovering, 24 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 625 op/s; 9837/111340 objects misplaced (8.835%); 7.0 MiB/s, 111 objects/s recovering 2024-04-20T17:47:26.121 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:47:26.148 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:26.167 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:47:26.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:26.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:47:26.271 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:47:26.318 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:26.475 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:47:26.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:26.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:47:26.583 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:47:26.616 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:26.723 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:47:26.731 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:47:26.731 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:47:26.731 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:47:26.731 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:47:26.731 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.731 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:141496 8.7 8:e9029e11:::benchmark_data_smithi171_175757_object141495:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e806)", 2024-04-20T17:47:26.731 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:44:52.815187+0000", 2024-04-20T17:47:26.731 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 153.91408119100001, 2024-04-20T17:47:26.731 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.62124218899999994, 2024-04-20T17:47:26.731 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:47:26.732 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:47:26.732 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:26.732 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:47:26.732 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:47:26.732 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:47:26.732 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 141496 2024-04-20T17:47:26.732 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.732 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:47:26.732 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.732 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:47:26.732 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.815187+0000", 2024-04-20T17:47:26.732 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.732 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.732 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.733 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:47:26.733 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.815187+0000", 2024-04-20T17:47:26.733 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.733 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.733 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.733 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:47:26.733 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.815188+0000", 2024-04-20T17:47:26.733 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9500000000000002e-07 2024-04-20T17:47:26.733 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.733 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.733 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:47:26.733 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.815221+0000", 2024-04-20T17:47:26.733 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2755000000000001e-05 2024-04-20T17:47:26.733 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.733 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.734 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:47:26.734 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.815228+0000", 2024-04-20T17:47:26.734 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5140000000000004e-06 2024-04-20T17:47:26.734 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.734 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.734 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:47:26.734 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.815234+0000", 2024-04-20T17:47:26.734 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2530000000000004e-06 2024-04-20T17:47:26.734 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.734 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.737 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.737 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.817526+0000", 2024-04-20T17:47:26.737 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0022920129999999999 2024-04-20T17:47:26.737 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.738 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.738 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-04-20T17:47:26.738 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.817533+0000", 2024-04-20T17:47:26.738 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3359999999999997e-06 2024-04-20T17:47:26.738 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.738 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.738 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.738 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.418700+0000", 2024-04-20T17:47:26.738 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.601166914 2024-04-20T17:47:26.739 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.739 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.739 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:47:26.739 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.418818+0000", 2024-04-20T17:47:26.739 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000118051 2024-04-20T17:47:26.739 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.739 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.739 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:47:26.739 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.418929+0000", 2024-04-20T17:47:26.740 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000111246 2024-04-20T17:47:26.740 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.740 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.740 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:47:26.740 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.433033+0000", 2024-04-20T17:47:26.740 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014103414999999999 2024-04-20T17:47:26.740 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.740 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.740 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:47:26.740 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.433079+0000", 2024-04-20T17:47:26.740 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6360999999999999e-05 2024-04-20T17:47:26.740 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.740 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.740 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:47:26.741 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.433138+0000", 2024-04-20T17:47:26.741 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9126000000000003e-05 2024-04-20T17:47:26.741 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.741 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.741 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:47:26.741 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.436430+0000", 2024-04-20T17:47:26.741 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0032915100000000001 2024-04-20T17:47:26.741 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.741 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:47:26.741 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.741 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.741 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.741 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:141513 8.7 8:ebe22147:::benchmark_data_smithi171_175757_object141512:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e806)", 2024-04-20T17:47:26.741 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:44:52.815241+0000", 2024-04-20T17:47:26.742 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 153.91402749700001, 2024-04-20T17:47:26.742 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.63952714799999999, 2024-04-20T17:47:26.742 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:47:26.742 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:47:26.742 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:26.742 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:47:26.742 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:47:26.742 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:47:26.742 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 141513 2024-04-20T17:47:26.742 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.742 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:47:26.742 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.742 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:47:26.742 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.815241+0000", 2024-04-20T17:47:26.743 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.743 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.743 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.743 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:47:26.743 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.815241+0000", 2024-04-20T17:47:26.743 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.743 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.743 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.743 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:47:26.743 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.815242+0000", 2024-04-20T17:47:26.743 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3499999999999996e-07 2024-04-20T17:47:26.743 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.743 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.744 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:47:26.744 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.815263+0000", 2024-04-20T17:47:26.744 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1435000000000001e-05 2024-04-20T17:47:26.744 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.744 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.744 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:47:26.744 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.815270+0000", 2024-04-20T17:47:26.744 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9739999999999996e-06 2024-04-20T17:47:26.744 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.744 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.744 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:47:26.744 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.815276+0000", 2024-04-20T17:47:26.744 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.519e-06 2024-04-20T17:47:26.745 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.745 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.745 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.745 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.817567+0000", 2024-04-20T17:47:26.745 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0022910679999999998 2024-04-20T17:47:26.745 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.745 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.745 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-04-20T17:47:26.745 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.817576+0000", 2024-04-20T17:47:26.745 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6700000000000006e-06 2024-04-20T17:47:26.745 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.745 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.745 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.745 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.425246+0000", 2024-04-20T17:47:26.746 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.60766962099999999 2024-04-20T17:47:26.746 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.746 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.746 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:47:26.746 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.425342+0000", 2024-04-20T17:47:26.746 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6226999999999999e-05 2024-04-20T17:47:26.746 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.746 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.746 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:47:26.746 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.425435+0000", 2024-04-20T17:47:26.746 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2263999999999996e-05 2024-04-20T17:47:26.746 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.746 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.747 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:47:26.747 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.436446+0000", 2024-04-20T17:47:26.747 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011011821999999999 2024-04-20T17:47:26.747 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.747 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.747 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:47:26.747 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.451152+0000", 2024-04-20T17:47:26.747 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014705269 2024-04-20T17:47:26.747 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.747 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.747 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:47:26.747 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.451209+0000", 2024-04-20T17:47:26.747 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7079e-05 2024-04-20T17:47:26.747 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.748 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.748 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:47:26.748 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.454768+0000", 2024-04-20T17:47:26.748 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0035595650000000002 2024-04-20T17:47:26.748 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.748 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:47:26.748 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.748 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.748 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.749 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:26 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1300804818' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:47:26.749 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:26 smithi005 ceph-mon[39562]: from='client.27923 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:47:26.750 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:141431 8.3 8:c0d63b4f:::benchmark_data_smithi171_175757_object141430:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e807)", 2024-04-20T17:47:26.750 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:44:52.821663+0000", 2024-04-20T17:47:26.750 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 153.90760591899999, 2024-04-20T17:47:26.750 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.61348092200000004, 2024-04-20T17:47:26.750 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:47:26.750 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:47:26.750 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "reached pg", 2024-04-20T17:47:26.750 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:47:26.751 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:47:26.751 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:47:26.751 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 141431 2024-04-20T17:47:26.751 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.751 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:47:26.751 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.751 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:47:26.751 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821663+0000", 2024-04-20T17:47:26.751 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.751 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.751 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.751 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:47:26.752 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821663+0000", 2024-04-20T17:47:26.752 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.752 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.752 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.752 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:47:26.752 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821664+0000", 2024-04-20T17:47:26.752 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6599999999999994e-07 2024-04-20T17:47:26.752 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.752 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.752 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:47:26.752 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821700+0000", 2024-04-20T17:47:26.752 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5916e-05 2024-04-20T17:47:26.752 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.753 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.753 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:47:26.753 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821708+0000", 2024-04-20T17:47:26.753 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6170000000000007e-06 2024-04-20T17:47:26.753 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.753 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.753 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:47:26.753 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821716+0000", 2024-04-20T17:47:26.753 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4819999999999997e-06 2024-04-20T17:47:26.753 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.753 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.753 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.753 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.825539+0000", 2024-04-20T17:47:26.753 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0038229929999999998 2024-04-20T17:47:26.753 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.754 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.754 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-04-20T17:47:26.754 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.825544+0000", 2024-04-20T17:47:26.754 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.604e-06 2024-04-20T17:47:26.754 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.754 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.754 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.754 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.371198+0000", 2024-04-20T17:47:26.754 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.54565325200000003 2024-04-20T17:47:26.754 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.754 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.754 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for degraded object", 2024-04-20T17:47:26.754 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.371902+0000", 2024-04-20T17:47:26.754 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00070485100000000005 2024-04-20T17:47:26.755 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.755 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.755 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.755 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.435021+0000", 2024-04-20T17:47:26.755 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.063118480000000005 2024-04-20T17:47:26.755 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.755 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.755 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:47:26.755 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.435144+0000", 2024-04-20T17:47:26.755 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012276100000000001 2024-04-20T17:47:26.755 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.755 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:47:26.755 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.755 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.755 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.756 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:141467 8.3 8:c65d4c6c:::benchmark_data_smithi171_175757_object141466:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e807)", 2024-04-20T17:47:26.756 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:44:52.821725+0000", 2024-04-20T17:47:26.756 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 153.90754322999999, 2024-04-20T17:47:26.756 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.61809995900000003, 2024-04-20T17:47:26.756 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:47:26.756 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:47:26.756 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:26.756 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:47:26.756 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:47:26.756 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:47:26.756 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 141467 2024-04-20T17:47:26.756 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.756 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:47:26.756 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.756 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:47:26.756 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821725+0000", 2024-04-20T17:47:26.757 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.757 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.757 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.757 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:47:26.757 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821725+0000", 2024-04-20T17:47:26.757 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.757 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.757 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.757 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:47:26.757 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821726+0000", 2024-04-20T17:47:26.757 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4899999999999995e-07 2024-04-20T17:47:26.757 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.757 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.757 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:47:26.757 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821755+0000", 2024-04-20T17:47:26.757 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8513e-05 2024-04-20T17:47:26.758 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.758 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.758 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:47:26.758 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821760+0000", 2024-04-20T17:47:26.758 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7729999999999996e-06 2024-04-20T17:47:26.758 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.758 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.758 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:47:26.758 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821764+0000", 2024-04-20T17:47:26.758 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5480000000000002e-06 2024-04-20T17:47:26.758 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.758 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.758 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.759 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.825583+0000", 2024-04-20T17:47:26.759 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0038187130000000001 2024-04-20T17:47:26.759 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.759 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.759 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-04-20T17:47:26.759 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.825590+0000", 2024-04-20T17:47:26.759 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1860000000000003e-06 2024-04-20T17:47:26.759 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.759 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.759 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.759 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.371971+0000", 2024-04-20T17:47:26.759 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.54638081599999999 2024-04-20T17:47:26.759 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.759 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.759 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:47:26.759 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.372138+0000", 2024-04-20T17:47:26.760 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00016759500000000001 2024-04-20T17:47:26.760 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.760 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.760 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:47:26.760 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.372337+0000", 2024-04-20T17:47:26.760 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00019848399999999999 2024-04-20T17:47:26.760 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.760 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.760 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:47:26.760 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.381629+0000", 2024-04-20T17:47:26.760 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0092923539999999992 2024-04-20T17:47:26.760 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.760 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.760 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:47:26.760 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.435190+0000", 2024-04-20T17:47:26.760 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.053560960999999997 2024-04-20T17:47:26.761 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.761 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.761 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:47:26.761 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.435248+0000", 2024-04-20T17:47:26.761 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7732999999999998e-05 2024-04-20T17:47:26.761 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.761 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.761 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:47:26.761 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.439825+0000", 2024-04-20T17:47:26.761 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0045776339999999997 2024-04-20T17:47:26.761 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.761 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:47:26.761 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.761 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.761 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.761 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:141468 8.3 8:c5481553:::benchmark_data_smithi171_175757_object141467:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e807)", 2024-04-20T17:47:26.761 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:44:52.821771+0000", 2024-04-20T17:47:26.762 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 153.90749774599999, 2024-04-20T17:47:26.762 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.623137198, 2024-04-20T17:47:26.762 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:47:26.762 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:47:26.762 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:26.762 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:47:26.762 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:47:26.762 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:47:26.762 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 141468 2024-04-20T17:47:26.762 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.762 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:47:26.762 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.762 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:47:26.762 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821771+0000", 2024-04-20T17:47:26.762 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.763 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.763 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.763 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:47:26.763 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821771+0000", 2024-04-20T17:47:26.763 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.763 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.763 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.763 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:47:26.763 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821771+0000", 2024-04-20T17:47:26.763 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7e-07 2024-04-20T17:47:26.763 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.763 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.763 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:47:26.763 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821796+0000", 2024-04-20T17:47:26.763 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.423e-05 2024-04-20T17:47:26.763 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.764 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.764 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:47:26.764 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821801+0000", 2024-04-20T17:47:26.764 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1440000000000002e-06 2024-04-20T17:47:26.764 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.764 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.764 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:47:26.764 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821805+0000", 2024-04-20T17:47:26.764 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8310000000000003e-06 2024-04-20T17:47:26.764 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.764 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.764 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.764 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.825617+0000", 2024-04-20T17:47:26.764 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0038119920000000002 2024-04-20T17:47:26.764 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.764 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.765 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-04-20T17:47:26.765 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.825620+0000", 2024-04-20T17:47:26.765 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.783e-06 2024-04-20T17:47:26.765 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.765 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.765 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.765 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.381748+0000", 2024-04-20T17:47:26.765 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.55612770300000003 2024-04-20T17:47:26.765 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.765 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.765 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:47:26.765 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.381919+0000", 2024-04-20T17:47:26.765 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00017122599999999999 2024-04-20T17:47:26.765 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.765 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.766 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:47:26.766 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.382035+0000", 2024-04-20T17:47:26.766 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011613 2024-04-20T17:47:26.766 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.766 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.766 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:47:26.766 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.404091+0000", 2024-04-20T17:47:26.766 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022055954999999999 2024-04-20T17:47:26.766 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.766 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.766 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:47:26.766 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.439906+0000", 2024-04-20T17:47:26.766 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.035814654000000001 2024-04-20T17:47:26.767 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.767 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.767 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:47:26.767 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.439967+0000", 2024-04-20T17:47:26.767 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0955000000000002e-05 2024-04-20T17:47:26.767 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.767 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.767 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:47:26.767 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.444908+0000", 2024-04-20T17:47:26.767 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0049411250000000002 2024-04-20T17:47:26.767 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.767 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:47:26.767 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.767 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.768 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.768 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:141477 8.3 8:c9bdf6fd:::benchmark_data_smithi171_175757_object141476:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e807)", 2024-04-20T17:47:26.768 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:44:52.821811+0000", 2024-04-20T17:47:26.768 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 153.90745777800001, 2024-04-20T17:47:26.768 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.62835837900000002, 2024-04-20T17:47:26.768 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:47:26.768 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:47:26.768 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:26.768 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:47:26.768 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:47:26.768 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:47:26.768 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 141477 2024-04-20T17:47:26.769 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.769 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:47:26.769 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.769 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:47:26.769 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821811+0000", 2024-04-20T17:47:26.769 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.769 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.769 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.769 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:47:26.769 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821811+0000", 2024-04-20T17:47:26.769 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.769 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.769 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.769 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:47:26.770 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821811+0000", 2024-04-20T17:47:26.770 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3099999999999998e-07 2024-04-20T17:47:26.770 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.770 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.770 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:47:26.770 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821835+0000", 2024-04-20T17:47:26.770 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4049e-05 2024-04-20T17:47:26.770 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.770 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.770 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:47:26.770 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821840+0000", 2024-04-20T17:47:26.770 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.78e-06 2024-04-20T17:47:26.770 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.771 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.771 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:47:26.771 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821844+0000", 2024-04-20T17:47:26.771 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.941e-06 2024-04-20T17:47:26.771 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.771 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.771 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.771 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.825649+0000", 2024-04-20T17:47:26.771 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0038050699999999998 2024-04-20T17:47:26.771 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.771 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.771 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-04-20T17:47:26.771 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.825653+0000", 2024-04-20T17:47:26.772 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8820000000000003e-06 2024-04-20T17:47:26.772 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.772 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.772 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.772 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.386088+0000", 2024-04-20T17:47:26.772 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.560434557 2024-04-20T17:47:26.772 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.772 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.772 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:47:26.772 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.386232+0000", 2024-04-20T17:47:26.772 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0001448 2024-04-20T17:47:26.772 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.772 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.772 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:47:26.772 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.386383+0000", 2024-04-20T17:47:26.773 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015015699999999999 2024-04-20T17:47:26.773 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.773 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.773 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:47:26.773 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.404100+0000", 2024-04-20T17:47:26.773 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.017717770000000001 2024-04-20T17:47:26.773 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.773 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.773 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:47:26.773 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.444982+0000", 2024-04-20T17:47:26.773 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.040881863999999997 2024-04-20T17:47:26.773 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.773 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.773 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:47:26.773 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.445044+0000", 2024-04-20T17:47:26.774 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1424e-05 2024-04-20T17:47:26.774 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.774 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.774 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:47:26.774 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.450169+0000", 2024-04-20T17:47:26.774 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0051256540000000003 2024-04-20T17:47:26.774 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.774 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:47:26.774 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.774 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.774 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.774 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:141497 8.3 8:c17e41ba:::benchmark_data_smithi171_175757_object141496:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e807)", 2024-04-20T17:47:26.774 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:44:52.821851+0000", 2024-04-20T17:47:26.775 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 153.90741772600001, 2024-04-20T17:47:26.775 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.63333487899999996, 2024-04-20T17:47:26.775 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:47:26.775 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:47:26.775 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:26.775 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:47:26.775 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:47:26.775 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:47:26.775 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 141497 2024-04-20T17:47:26.775 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.775 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:47:26.775 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.775 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:47:26.775 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821851+0000", 2024-04-20T17:47:26.775 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.776 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.776 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.776 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:47:26.776 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821851+0000", 2024-04-20T17:47:26.776 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.776 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.776 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.776 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:47:26.776 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821851+0000", 2024-04-20T17:47:26.776 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.34e-07 2024-04-20T17:47:26.776 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.776 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.776 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:47:26.776 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821869+0000", 2024-04-20T17:47:26.776 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8043999999999999e-05 2024-04-20T17:47:26.777 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.777 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.777 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:47:26.777 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821874+0000", 2024-04-20T17:47:26.777 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5410000000000002e-06 2024-04-20T17:47:26.777 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.777 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.777 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:47:26.777 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821878+0000", 2024-04-20T17:47:26.777 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.754e-06 2024-04-20T17:47:26.777 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.777 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.777 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.777 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.825679+0000", 2024-04-20T17:47:26.778 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0038012469999999998 2024-04-20T17:47:26.778 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.778 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.778 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-04-20T17:47:26.778 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.825683+0000", 2024-04-20T17:47:26.778 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9999999999999998e-06 2024-04-20T17:47:26.778 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.778 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.778 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.778 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.395247+0000", 2024-04-20T17:47:26.778 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.56956391100000003 2024-04-20T17:47:26.778 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.778 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.779 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:47:26.779 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.395379+0000", 2024-04-20T17:47:26.779 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013175700000000001 2024-04-20T17:47:26.779 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.779 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.779 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:47:26.779 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.395494+0000", 2024-04-20T17:47:26.779 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011512500000000001 2024-04-20T17:47:26.779 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.779 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.779 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:47:26.779 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.425512+0000", 2024-04-20T17:47:26.779 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030017814 2024-04-20T17:47:26.779 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.779 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.779 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:47:26.780 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.450243+0000", 2024-04-20T17:47:26.780 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024731281000000001 2024-04-20T17:47:26.780 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.780 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.780 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:47:26.780 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.450302+0000", 2024-04-20T17:47:26.780 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8826999999999998e-05 2024-04-20T17:47:26.780 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.780 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.780 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:47:26.780 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.455186+0000", 2024-04-20T17:47:26.780 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.004884144 2024-04-20T17:47:26.780 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.780 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:47:26.780 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.780 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.781 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.781 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:141500 8.3 8:c83e249f:::benchmark_data_smithi171_175757_object141499:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e807)", 2024-04-20T17:47:26.781 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:44:52.821882+0000", 2024-04-20T17:47:26.781 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 153.90738668099999, 2024-04-20T17:47:26.781 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.63804357, 2024-04-20T17:47:26.781 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:47:26.781 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:47:26.781 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:26.781 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:47:26.781 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:47:26.781 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:47:26.781 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 141500 2024-04-20T17:47:26.781 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.781 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:47:26.781 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.781 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:47:26.782 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821882+0000", 2024-04-20T17:47:26.782 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.782 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.782 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.782 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:47:26.782 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821882+0000", 2024-04-20T17:47:26.782 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.782 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.782 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.782 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:47:26.782 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821882+0000", 2024-04-20T17:47:26.782 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2199999999999999e-07 2024-04-20T17:47:26.782 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.782 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.782 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:47:26.782 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821926+0000", 2024-04-20T17:47:26.783 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.316e-05 2024-04-20T17:47:26.783 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.783 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.783 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:47:26.783 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821931+0000", 2024-04-20T17:47:26.783 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4859999999999997e-06 2024-04-20T17:47:26.783 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.783 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.783 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:47:26.783 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821935+0000", 2024-04-20T17:47:26.783 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1999999999999996e-06 2024-04-20T17:47:26.783 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.783 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.783 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.783 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.825715+0000", 2024-04-20T17:47:26.783 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0037800479999999998 2024-04-20T17:47:26.784 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.784 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.784 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-04-20T17:47:26.784 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.825720+0000", 2024-04-20T17:47:26.784 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6199999999999998e-06 2024-04-20T17:47:26.784 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.784 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.784 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.784 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.399158+0000", 2024-04-20T17:47:26.784 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.57343811899999997 2024-04-20T17:47:26.784 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.784 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.784 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:47:26.784 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.399346+0000", 2024-04-20T17:47:26.784 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00018777400000000001 2024-04-20T17:47:26.784 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.785 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.785 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:47:26.785 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.399530+0000", 2024-04-20T17:47:26.785 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00018449800000000001 2024-04-20T17:47:26.785 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.785 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.785 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:47:26.785 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.435130+0000", 2024-04-20T17:47:26.785 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.035600154000000002 2024-04-20T17:47:26.785 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.785 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.785 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:47:26.785 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.455269+0000", 2024-04-20T17:47:26.785 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020138624000000001 2024-04-20T17:47:26.785 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.786 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.786 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:47:26.786 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.455307+0000", 2024-04-20T17:47:26.786 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8039e-05 2024-04-20T17:47:26.786 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.786 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.786 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:47:26.786 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.459926+0000", 2024-04-20T17:47:26.786 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0046184260000000001 2024-04-20T17:47:26.786 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.786 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:47:26.786 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.786 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.786 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.786 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:141507 8.3 8:c60a4ce0:::benchmark_data_smithi171_175757_object141506:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e807)", 2024-04-20T17:47:26.787 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:44:52.821941+0000", 2024-04-20T17:47:26.787 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 153.90732736000001, 2024-04-20T17:47:26.787 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.64357269500000003, 2024-04-20T17:47:26.787 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:47:26.787 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:47:26.787 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:26.787 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:47:26.787 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:47:26.787 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:47:26.787 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 141507 2024-04-20T17:47:26.787 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.787 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:47:26.787 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.787 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:47:26.787 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821941+0000", 2024-04-20T17:47:26.788 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.788 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.788 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.788 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:47:26.788 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821941+0000", 2024-04-20T17:47:26.788 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.788 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.788 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.788 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:47:26.788 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821942+0000", 2024-04-20T17:47:26.788 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9499999999999998e-07 2024-04-20T17:47:26.788 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.788 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.788 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:47:26.788 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821975+0000", 2024-04-20T17:47:26.789 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3733999999999999e-05 2024-04-20T17:47:26.789 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.789 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.789 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:47:26.789 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821980+0000", 2024-04-20T17:47:26.789 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.977e-06 2024-04-20T17:47:26.789 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.789 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.789 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:47:26.789 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821984+0000", 2024-04-20T17:47:26.789 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8149999999999999e-06 2024-04-20T17:47:26.789 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.789 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.789 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.790 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.825748+0000", 2024-04-20T17:47:26.790 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0037634410000000002 2024-04-20T17:47:26.790 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.790 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.790 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-04-20T17:47:26.790 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.825751+0000", 2024-04-20T17:47:26.790 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6370000000000001e-06 2024-04-20T17:47:26.790 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.790 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.790 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.790 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.404145+0000", 2024-04-20T17:47:26.790 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.57839404900000002 2024-04-20T17:47:26.790 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.790 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:47:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.404283+0000", 2024-04-20T17:47:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000137534 2024-04-20T17:47:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:47:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.404421+0000", 2024-04-20T17:47:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013860000000000001 2024-04-20T17:47:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:47:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.435137+0000", 2024-04-20T17:47:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030715269 2024-04-20T17:47:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.791 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:47:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.459994+0000", 2024-04-20T17:47:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024857607 2024-04-20T17:47:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:47:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.460049+0000", 2024-04-20T17:47:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4440000000000001e-05 2024-04-20T17:47:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:47:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.465514+0000", 2024-04-20T17:47:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0054651969999999998 2024-04-20T17:47:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:47:26.792 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:141517 8.3 8:ccce9043:::benchmark_data_smithi171_175757_object141516:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e807)", 2024-04-20T17:47:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:44:52.821990+0000", 2024-04-20T17:47:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 153.90727913200001, 2024-04-20T17:47:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.64891917799999999, 2024-04-20T17:47:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:47:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:47:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:47:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:47:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:47:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 141517 2024-04-20T17:47:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.793 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:47:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:47:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821990+0000", 2024-04-20T17:47:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:47:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821990+0000", 2024-04-20T17:47:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:47:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.821990+0000", 2024-04-20T17:47:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0200000000000003e-07 2024-04-20T17:47:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:47:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.822023+0000", 2024-04-20T17:47:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3528000000000002e-05 2024-04-20T17:47:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:47:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.822028+0000", 2024-04-20T17:47:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9880000000000004e-06 2024-04-20T17:47:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:47:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.822032+0000", 2024-04-20T17:47:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8890000000000004e-06 2024-04-20T17:47:26.795 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.825770+0000", 2024-04-20T17:47:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0037380970000000001 2024-04-20T17:47:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-04-20T17:47:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:52.825774+0000", 2024-04-20T17:47:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6229999999999999e-06 2024-04-20T17:47:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.407676+0000", 2024-04-20T17:47:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.58190245200000001 2024-04-20T17:47:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.796 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:47:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.407771+0000", 2024-04-20T17:47:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.4333999999999995e-05 2024-04-20T17:47:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1", 2024-04-20T17:47:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.407883+0000", 2024-04-20T17:47:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000112451 2024-04-20T17:47:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:47:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.435142+0000", 2024-04-20T17:47:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.027259131999999998 2024-04-20T17:47:26.797 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:47:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.465588+0000", 2024-04-20T17:47:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.03044558 2024-04-20T17:47:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:47:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.465654+0000", 2024-04-20T17:47:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6353999999999995e-05 2024-04-20T17:47:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:47:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:44:53.470909+0000", 2024-04-20T17:47:26.798 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0052543479999999998 2024-04-20T17:47:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:47:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:205516 8.d 8.f59a740d (undecoded) ondisk+write+known_if_redirected e843)", 2024-04-20T17:47:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:46:36.831373+0000", 2024-04-20T17:47:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 49.897895706, 2024-04-20T17:47:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.94770686100000001, 2024-04-20T17:47:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:47:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:47:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "reached pg", 2024-04-20T17:47:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:47:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:47:26.799 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:47:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 205516 2024-04-20T17:47:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:47:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:47:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.831373+0000", 2024-04-20T17:47:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:47:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.831373+0000", 2024-04-20T17:47:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:47:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.831376+0000", 2024-04-20T17:47:26.800 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3419999999999999e-06 2024-04-20T17:47:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:47:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.831386+0000", 2024-04-20T17:47:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0163000000000001e-05 2024-04-20T17:47:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:47:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.831403+0000", 2024-04-20T17:47:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6313000000000001e-05 2024-04-20T17:47:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:47:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.831415+0000", 2024-04-20T17:47:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2207e-05 2024-04-20T17:47:26.801 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.831503+0000", 2024-04-20T17:47:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.7942000000000006e-05 2024-04-20T17:47:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-04-20T17:47:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.831515+0000", 2024-04-20T17:47:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1698999999999999e-05 2024-04-20T17:47:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.779050+0000", 2024-04-20T17:47:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.94753564400000001 2024-04-20T17:47:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.802 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:47:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.779080+0000", 2024-04-20T17:47:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9550999999999999e-05 2024-04-20T17:47:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:47:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:205519 8.d 8.1c86022d (undecoded) ondisk+write+known_if_redirected e843)", 2024-04-20T17:47:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:46:36.831668+0000", 2024-04-20T17:47:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 49.897600161, 2024-04-20T17:47:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.94745927200000002, 2024-04-20T17:47:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:47:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:47:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "reached pg", 2024-04-20T17:47:26.803 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:47:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:47:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:47:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 205519 2024-04-20T17:47:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:47:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:47:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.831668+0000", 2024-04-20T17:47:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:47:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.831668+0000", 2024-04-20T17:47:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.804 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:47:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.831669+0000", 2024-04-20T17:47:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8999999999999995e-07 2024-04-20T17:47:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:47:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.831673+0000", 2024-04-20T17:47:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6610000000000002e-06 2024-04-20T17:47:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:47:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.831679+0000", 2024-04-20T17:47:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0159999999999997e-06 2024-04-20T17:47:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.805 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:47:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.831684+0000", 2024-04-20T17:47:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5110000000000003e-06 2024-04-20T17:47:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.831781+0000", 2024-04-20T17:47:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6056000000000003e-05 2024-04-20T17:47:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-04-20T17:47:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.831789+0000", 2024-04-20T17:47:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8230000000000001e-06 2024-04-20T17:47:26.806 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.779118+0000", 2024-04-20T17:47:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.94732908199999999 2024-04-20T17:47:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:47:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.779128+0000", 2024-04-20T17:47:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3330000000000003e-06 2024-04-20T17:47:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:47:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:205521 8.d 8.edb999d (undecoded) ondisk+write+known_if_redirected e843)", 2024-04-20T17:47:26.807 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:46:36.831721+0000", 2024-04-20T17:47:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 49.897547719999999, 2024-04-20T17:47:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.94742668900000004, 2024-04-20T17:47:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:47:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:47:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "reached pg", 2024-04-20T17:47:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:47:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:47:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:47:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 205521 2024-04-20T17:47:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:47:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:47:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.831721+0000", 2024-04-20T17:47:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.808 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:47:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.831721+0000", 2024-04-20T17:47:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:47:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.831722+0000", 2024-04-20T17:47:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9699999999999996e-07 2024-04-20T17:47:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:47:26.809 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.831724+0000", 2024-04-20T17:47:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.819e-06 2024-04-20T17:47:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:47:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.831729+0000", 2024-04-20T17:47:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9269999999999996e-06 2024-04-20T17:47:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:47:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.831734+0000", 2024-04-20T17:47:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8239999999999999e-06 2024-04-20T17:47:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.810 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.831814+0000", 2024-04-20T17:47:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0346000000000006e-05 2024-04-20T17:47:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-04-20T17:47:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.831822+0000", 2024-04-20T17:47:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7519999999999996e-06 2024-04-20T17:47:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.779144+0000", 2024-04-20T17:47:26.811 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.94732202499999996 2024-04-20T17:47:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:47:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.779148+0000", 2024-04-20T17:47:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.399e-06 2024-04-20T17:47:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:47:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:205535 8.d 8.683cfe6d (undecoded) ondisk+write+known_if_redirected e843)", 2024-04-20T17:47:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:46:36.852451+0000", 2024-04-20T17:47:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 49.876817445, 2024-04-20T17:47:26.812 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.92671179599999998, 2024-04-20T17:47:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:47:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:47:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "reached pg", 2024-04-20T17:47:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:47:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:47:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:47:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 205535 2024-04-20T17:47:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:47:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:47:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.852451+0000", 2024-04-20T17:47:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.813 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:47:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.852451+0000", 2024-04-20T17:47:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:47:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.852452+0000", 2024-04-20T17:47:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.291e-06 2024-04-20T17:47:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:47:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.852457+0000", 2024-04-20T17:47:26.814 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.561e-06 2024-04-20T17:47:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:47:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.852465+0000", 2024-04-20T17:47:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2649999999999996e-06 2024-04-20T17:47:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:47:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.852471+0000", 2024-04-20T17:47:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.874e-06 2024-04-20T17:47:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.856400+0000", 2024-04-20T17:47:26.815 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0039289069999999997 2024-04-20T17:47:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-04-20T17:47:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.856413+0000", 2024-04-20T17:47:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3178e-05 2024-04-20T17:47:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.779160+0000", 2024-04-20T17:47:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.92274646100000002 2024-04-20T17:47:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:47:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.779163+0000", 2024-04-20T17:47:26.816 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2590000000000001e-06 2024-04-20T17:47:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:47:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:205562 8.d 8.68b823ed (undecoded) ondisk+write+known_if_redirected e843)", 2024-04-20T17:47:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:46:36.900626+0000", 2024-04-20T17:47:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 49.828642702000003, 2024-04-20T17:47:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.87855141199999998, 2024-04-20T17:47:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:47:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:47:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "reached pg", 2024-04-20T17:47:26.817 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:47:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:47:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:47:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 205562 2024-04-20T17:47:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:47:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:47:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.900626+0000", 2024-04-20T17:47:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:47:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.900626+0000", 2024-04-20T17:47:26.818 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:47:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.900628+0000", 2024-04-20T17:47:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.086e-06 2024-04-20T17:47:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:47:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.900634+0000", 2024-04-20T17:47:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5509999999999999e-06 2024-04-20T17:47:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:47:26.819 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.900645+0000", 2024-04-20T17:47:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1423000000000001e-05 2024-04-20T17:47:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:47:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.900652+0000", 2024-04-20T17:47:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3200000000000002e-06 2024-04-20T17:47:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.900743+0000", 2024-04-20T17:47:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0581000000000006e-05 2024-04-20T17:47:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.820 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-04-20T17:47:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.900753+0000", 2024-04-20T17:47:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0124e-05 2024-04-20T17:47:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.779175+0000", 2024-04-20T17:47:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.87842149700000005 2024-04-20T17:47:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:47:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.779177+0000", 2024-04-20T17:47:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.83e-06 2024-04-20T17:47:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.821 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:47:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:205564 8.d 8.1235e3d (undecoded) ondisk+write+known_if_redirected e843)", 2024-04-20T17:47:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:46:36.900790+0000", 2024-04-20T17:47:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 49.828478402000002, 2024-04-20T17:47:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.87840224600000005, 2024-04-20T17:47:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:47:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:47:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "reached pg", 2024-04-20T17:47:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:47:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:47:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:47:26.822 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 205564 2024-04-20T17:47:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:47:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:47:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.900790+0000", 2024-04-20T17:47:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:47:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.900790+0000", 2024-04-20T17:47:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.823 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:47:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.900791+0000", 2024-04-20T17:47:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.26e-06 2024-04-20T17:47:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:47:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.900796+0000", 2024-04-20T17:47:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5580000000000002e-06 2024-04-20T17:47:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:47:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.900805+0000", 2024-04-20T17:47:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3899999999999999e-06 2024-04-20T17:47:26.824 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:47:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.900812+0000", 2024-04-20T17:47:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1240000000000002e-06 2024-04-20T17:47:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.900857+0000", 2024-04-20T17:47:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5716000000000001e-05 2024-04-20T17:47:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-04-20T17:47:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.900865+0000", 2024-04-20T17:47:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9580000000000007e-06 2024-04-20T17:47:26.825 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.779189+0000", 2024-04-20T17:47:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.87832407999999995 2024-04-20T17:47:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:47:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.779192+0000", 2024-04-20T17:47:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1599999999999998e-06 2024-04-20T17:47:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:47:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:205605 8.d 8.31e1325d (undecoded) ondisk+write+known_if_redirected e843)", 2024-04-20T17:47:26.826 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:46:36.972904+0000", 2024-04-20T17:47:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 49.756364800999997, 2024-04-20T17:47:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.80631751100000004, 2024-04-20T17:47:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:47:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:47:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "reached pg", 2024-04-20T17:47:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:47:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:47:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:47:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 205605 2024-04-20T17:47:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:47:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:47:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.972904+0000", 2024-04-20T17:47:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.827 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:47:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.972904+0000", 2024-04-20T17:47:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:47:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.972907+0000", 2024-04-20T17:47:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8609999999999998e-06 2024-04-20T17:47:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:47:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.972916+0000", 2024-04-20T17:47:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9500000000000007e-06 2024-04-20T17:47:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.828 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:47:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.972929+0000", 2024-04-20T17:47:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3276e-05 2024-04-20T17:47:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:47:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.972938+0000", 2024-04-20T17:47:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5340000000000005e-06 2024-04-20T17:47:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.973015+0000", 2024-04-20T17:47:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6705999999999994e-05 2024-04-20T17:47:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-04-20T17:47:26.829 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.973027+0000", 2024-04-20T17:47:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2217e-05 2024-04-20T17:47:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.779215+0000", 2024-04-20T17:47:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.80618804899999996 2024-04-20T17:47:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:47:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.779221+0000", 2024-04-20T17:47:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9179999999999999e-06 2024-04-20T17:47:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:47:26.830 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:205615 8.d 8.f7736f7d (undecoded) ondisk+write+known_if_redirected e843)", 2024-04-20T17:47:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:46:36.983660+0000", 2024-04-20T17:47:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 49.745608994999998, 2024-04-20T17:47:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.79559101399999999, 2024-04-20T17:47:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:47:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:47:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "reached pg", 2024-04-20T17:47:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:47:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:47:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:47:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 205615 2024-04-20T17:47:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:47:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.831 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:47:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.983660+0000", 2024-04-20T17:47:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:47:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.983660+0000", 2024-04-20T17:47:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:47:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.983662+0000", 2024-04-20T17:47:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.906e-06 2024-04-20T17:47:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.832 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:47:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.983666+0000", 2024-04-20T17:47:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9300000000000002e-06 2024-04-20T17:47:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:47:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.983677+0000", 2024-04-20T17:47:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0637e-05 2024-04-20T17:47:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:47:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.983684+0000", 2024-04-20T17:47:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.323e-06 2024-04-20T17:47:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.983752+0000", 2024-04-20T17:47:26.833 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7267000000000005e-05 2024-04-20T17:47:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-04-20T17:47:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:36.983764+0000", 2024-04-20T17:47:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2349e-05 2024-04-20T17:47:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.779246+0000", 2024-04-20T17:47:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.79548243100000005 2024-04-20T17:47:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:47:26.834 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.779251+0000", 2024-04-20T17:47:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1710000000000004e-06 2024-04-20T17:47:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:47:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:205666 8.d 8.75b32b2d (undecoded) ondisk+write+known_if_redirected e843)", 2024-04-20T17:47:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:46:37.089434+0000", 2024-04-20T17:47:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 49.639834999999998, 2024-04-20T17:47:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.68985766900000001, 2024-04-20T17:47:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:47:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:47:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "reached pg", 2024-04-20T17:47:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:47:26.835 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:47:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:47:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 205666 2024-04-20T17:47:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:47:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:47:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.089434+0000", 2024-04-20T17:47:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:47:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.089434+0000", 2024-04-20T17:47:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.836 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:47:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.089438+0000", 2024-04-20T17:47:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8250000000000004e-06 2024-04-20T17:47:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:47:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.089450+0000", 2024-04-20T17:47:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1605e-05 2024-04-20T17:47:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:47:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.089466+0000", 2024-04-20T17:47:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6161e-05 2024-04-20T17:47:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.837 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:47:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.089478+0000", 2024-04-20T17:47:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2115e-05 2024-04-20T17:47:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.093685+0000", 2024-04-20T17:47:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0042064540000000001 2024-04-20T17:47:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-04-20T17:47:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.093694+0000", 2024-04-20T17:47:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0000000000000002e-06 2024-04-20T17:47:26.838 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.779282+0000", 2024-04-20T17:47:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.68558781499999999 2024-04-20T17:47:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:47:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.779291+0000", 2024-04-20T17:47:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6940000000000007e-06 2024-04-20T17:47:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:47:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.839 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:205687 8.d 8.d1da260d (undecoded) ondisk+write+known_if_redirected e843)", 2024-04-20T17:47:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:46:37.147892+0000", 2024-04-20T17:47:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 49.581377031000002, 2024-04-20T17:47:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.63144880199999998, 2024-04-20T17:47:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:47:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:47:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "reached pg", 2024-04-20T17:47:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:47:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:47:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:47:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 205687 2024-04-20T17:47:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:47:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:47:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.147892+0000", 2024-04-20T17:47:26.840 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:47:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.147892+0000", 2024-04-20T17:47:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:47:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:47:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.147894+0000", 2024-04-20T17:47:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8270000000000002e-06 2024-04-20T17:47:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:47:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.147902+0000", 2024-04-20T17:47:26.841 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7989999999999994e-06 2024-04-20T17:47:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:47:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.147914+0000", 2024-04-20T17:47:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1673e-05 2024-04-20T17:47:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:47:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.147921+0000", 2024-04-20T17:47:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6470000000000001e-06 2024-04-20T17:47:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.842 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.147989+0000", 2024-04-20T17:47:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8708999999999996e-05 2024-04-20T17:47:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-04-20T17:47:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.148001+0000", 2024-04-20T17:47:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2055999999999999e-05 2024-04-20T17:47:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:47:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.779331+0000", 2024-04-20T17:47:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.63132988899999998 2024-04-20T17:47:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:47:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:47:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:47:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:37.779340+0000", 2024-04-20T17:47:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2019999999999993e-06 2024-04-20T17:47:26.843 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.844 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:47:26.844 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.844 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:47:26.844 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:47:26.844 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:47:26.948 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:47:27.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:26 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1300804818' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:47:27.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:26 smithi106 ceph-mon[44423]: from='client.27923 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:47:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:26 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1300804818' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:47:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:26 smithi171 ceph-mon[44189]: from='client.27923 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:47:27.471 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:47:27.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:27.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:47:27.578 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:47:27.605 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:27.626 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:47:27.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:27.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:47:27.730 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:47:27.756 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:27.777 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:47:27.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:27.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:47:27.881 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:47:27.905 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:27 smithi106 ceph-mon[44423]: pgmap v2292: 27 pgs: 3 active+recovering, 24 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s; 9329/107010 objects misplaced (8.718%); 8.4 MiB/s, 133 objects/s recovering 2024-04-20T17:47:27.905 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:27 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:47:27.906 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:27 smithi106 ceph-mon[44423]: from='client.27923 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:47:27.906 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:27 smithi106 ceph-mon[44423]: osdmap e863: 12 total, 12 up, 12 in 2024-04-20T17:47:27.908 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:27.926 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:47:27.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:27.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:47:28.011 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:27 smithi005 ceph-mon[39562]: pgmap v2292: 27 pgs: 3 active+recovering, 24 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s; 9329/107010 objects misplaced (8.718%); 8.4 MiB/s, 133 objects/s recovering 2024-04-20T17:47:28.011 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:27 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:47:28.011 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:27 smithi005 ceph-mon[39562]: from='client.27923 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:47:28.011 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:27 smithi005 ceph-mon[39562]: osdmap e863: 12 total, 12 up, 12 in 2024-04-20T17:47:28.030 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:47:28.071 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:27 smithi171 ceph-mon[44189]: pgmap v2292: 27 pgs: 3 active+recovering, 24 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s; 9329/107010 objects misplaced (8.718%); 8.4 MiB/s, 133 objects/s recovering 2024-04-20T17:47:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:27 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:47:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:27 smithi171 ceph-mon[44189]: from='client.27923 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:47:28.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:27 smithi171 ceph-mon[44189]: osdmap e863: 12 total, 12 up, 12 in 2024-04-20T17:47:28.297 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-20T17:47:28.370 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:47:28.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:28.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:47:28.490 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:47:28.554 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:28.592 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:47:28.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:28.596 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:47:28.697 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:47:28.724 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:28.743 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:47:28.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:28.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:47:28.759 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:47:28.847 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:47:28.877 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:47:28.877 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-20T17:47:28.877 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:47:28.917 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:28.953 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:47:28.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:28.962 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:47:29.063 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:47:29.108 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:29.133 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:47:29.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:29.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:47:29.237 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:47:29.264 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:29.269 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:47:29.287 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:47:29.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:29.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:47:29.391 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:47:29.435 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:29.564 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:47:29.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:29.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:47:29.672 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:47:29.721 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:29.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:29 smithi005 ceph-mon[39562]: pgmap v2294: 27 pgs: 3 active+recovering, 24 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 719 op/s; 9108/105526 objects misplaced (8.631%); 11 MiB/s, 174 objects/s recovering 2024-04-20T17:47:30.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:29 smithi106 ceph-mon[44423]: pgmap v2294: 27 pgs: 3 active+recovering, 24 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 719 op/s; 9108/105526 objects misplaced (8.631%); 11 MiB/s, 174 objects/s recovering 2024-04-20T17:47:30.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:29 smithi171 ceph-mon[44189]: pgmap v2294: 27 pgs: 3 active+recovering, 24 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 719 op/s; 9108/105526 objects misplaced (8.631%); 11 MiB/s, 174 objects/s recovering 2024-04-20T17:47:30.417 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:47:31.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:31 smithi005 ceph-mon[39562]: pgmap v2295: 27 pgs: 3 active+recovering, 24 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 683 op/s; 9108/102184 objects misplaced (8.913%); 6.8 MiB/s, 108 objects/s recovering 2024-04-20T17:47:31.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:31 smithi005 ceph-mon[39562]: osdmap e864: 12 total, 12 up, 12 in 2024-04-20T17:47:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:31 smithi106 ceph-mon[44423]: pgmap v2295: 27 pgs: 3 active+recovering, 24 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 683 op/s; 9108/102184 objects misplaced (8.913%); 6.8 MiB/s, 108 objects/s recovering 2024-04-20T17:47:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:31 smithi106 ceph-mon[44423]: osdmap e864: 12 total, 12 up, 12 in 2024-04-20T17:47:32.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:31 smithi171 ceph-mon[44189]: pgmap v2295: 27 pgs: 3 active+recovering, 24 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 683 op/s; 9108/102184 objects misplaced (8.913%); 6.8 MiB/s, 108 objects/s recovering 2024-04-20T17:47:32.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:31 smithi171 ceph-mon[44189]: osdmap e864: 12 total, 12 up, 12 in 2024-04-20T17:47:32.388 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:47:32.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:32.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:47:32.496 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:47:32.571 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:32.621 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:47:32.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:32.627 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:47:32.728 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:47:32.767 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:32.788 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:47:32.788 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:47:32.788 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:32.788 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:243169 8.f e864/777 8:f3c582c3:::benchmark_data_smithi171_175757_object98055:head v 864'14956, mlcod=864'14954)", 2024-04-20T17:47:32.788 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:47:32.781663+0000", 2024-04-20T17:47:32.788 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.00053666599999999999, 2024-04-20T17:47:32.788 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00056079999999999997, 2024-04-20T17:47:32.788 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:47:32.788 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:47:32.788 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:47:32.788 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:47:32.788 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:32.788 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:47:32.788 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:32.781663+0000", 2024-04-20T17:47:32.789 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:32.789 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:32.789 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:32.789 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:47:32.789 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:32.781663+0000", 2024-04-20T17:47:32.789 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:32.789 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:32.789 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:32.789 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:47:32.789 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:32.781665+0000", 2024-04-20T17:47:32.789 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8899999999999999e-06 2024-04-20T17:47:32.789 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:32.789 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:32.789 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:47:32.790 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:32.781675+0000", 2024-04-20T17:47:32.790 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7100000000000002e-06 2024-04-20T17:47:32.790 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:32.790 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:32.790 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:47:32.790 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:32.781687+0000", 2024-04-20T17:47:32.790 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2520999999999999e-05 2024-04-20T17:47:32.790 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:32.790 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:32.790 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:47:32.790 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:32.781696+0000", 2024-04-20T17:47:32.790 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6540000000000005e-06 2024-04-20T17:47:32.790 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:32.790 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:32.790 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:47:32.791 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:32.781749+0000", 2024-04-20T17:47:32.791 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2695000000000003e-05 2024-04-20T17:47:32.791 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:32.791 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:32.791 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:47:32.791 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:32.781769+0000", 2024-04-20T17:47:32.791 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9695999999999999e-05 2024-04-20T17:47:32.791 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:32.791 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:32.791 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:32.791 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:32.791 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:47:32.791 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 1 2024-04-20T17:47:32.791 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:47:33.634 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:47:33.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:33.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:47:33.741 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:47:33.768 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:33.791 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:47:33.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:33.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:47:33.895 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:47:33.948 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:33 smithi005 ceph-mon[39562]: pgmap v2297: 27 pgs: 3 active+recovering, 24 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 847 op/s; 8367/97772 objects misplaced (8.558%); 11 MiB/s, 183 objects/s recovering 2024-04-20T17:47:33.951 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:33.983 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:47:33.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:33.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:47:34.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:33 smithi106 ceph-mon[44423]: pgmap v2297: 27 pgs: 3 active+recovering, 24 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 847 op/s; 8367/97772 objects misplaced (8.558%); 11 MiB/s, 183 objects/s recovering 2024-04-20T17:47:34.019 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:47:34.053 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:47:34.084 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:47:34.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:33 smithi171 ceph-mon[44189]: pgmap v2297: 27 pgs: 3 active+recovering, 24 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 847 op/s; 8367/97772 objects misplaced (8.558%); 11 MiB/s, 183 objects/s recovering 2024-04-20T17:47:34.094 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:47:34.112 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:47:34.138 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:47:34.164 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:47:34.170 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:34.188 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:47:34.210 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:47:34.212 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:47:34.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:34.221 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:47:34.247 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:47:34.283 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:47:34.320 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:47:34.322 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:47:34.349 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:34.350 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:47:34.358 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:47:34.358 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:47:34.358 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:47:34.358 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:47:34.358 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:47:34.358 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:47:34.359 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:47:34.359 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:47:34.359 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:47:34.359 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:47:34.359 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:47:34.359 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:47:34.368 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:47:34.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:34.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:47:34.402 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:47:34.451 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-20T17:47:34.471 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:47:34.526 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:34.569 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:47:34.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:34.573 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:47:34.673 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:47:34.700 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:34.719 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:47:34.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:34.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:47:34.825 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:47:34.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:34 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1891011145' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:47:34.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:34.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:34.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:34.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:34 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:34.873 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:34 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1891011145' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:47:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:35.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:34 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:35.168 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:34 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1891011145' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:47:35.168 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:35.168 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:35.168 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:35.168 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:34 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:35.335 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:47:35.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:35.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:47:35.442 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:47:35.482 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:35.536 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:47:35.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:35.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:47:35.640 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:47:35.667 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:35.687 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:47:35.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:35.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:47:35.791 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:47:35.833 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:35.856 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:47:35.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:35.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:47:35.960 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:47:35.991 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:36.007 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:36.008 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:36.008 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:36.008 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:36.008 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:36.008 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:36.008 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:36.008 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:36.008 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:35 smithi005 ceph-mon[39562]: pgmap v2298: 27 pgs: 3 active+recovering, 24 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 696 op/s; 8367/95862 objects misplaced (8.728%); 7.5 MiB/s, 120 objects/s recovering 2024-04-20T17:47:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:35 smithi106 ceph-mon[44423]: pgmap v2298: 27 pgs: 3 active+recovering, 24 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 696 op/s; 8367/95862 objects misplaced (8.728%); 7.5 MiB/s, 120 objects/s recovering 2024-04-20T17:47:36.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:36.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:36.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:36.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:36.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:36.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:36.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:36.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:36.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:35 smithi171 ceph-mon[44189]: pgmap v2298: 27 pgs: 3 active+recovering, 24 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 696 op/s; 8367/95862 objects misplaced (8.728%); 7.5 MiB/s, 120 objects/s recovering 2024-04-20T17:47:36.446 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:47:36.453 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:47:36.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:36.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:47:36.566 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:47:36.612 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:36.641 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:47:36.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:36.649 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:47:36.750 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:47:36.794 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:37.135 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 8, 9, 10, 11, 1, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 4, 5, 8, 9, 10, 11, 2, 6, 3, 1, 7] 2024-04-20T17:47:37.135 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:47:37.136 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-20T17:47:37.136 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-20T17:47:37.205 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:47:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:37 smithi106 ceph-mon[44423]: pgmap v2299: 27 pgs: 3 active+recovering, 24 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 792 op/s; 7846/91280 objects misplaced (8.596%); 9.3 MiB/s, 149 objects/s recovering 2024-04-20T17:47:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:37 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2755355286' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:47:38.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:37 smithi106 ceph-mon[44423]: from='client.27935 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:47:38.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:37 smithi171 ceph-mon[44189]: pgmap v2299: 27 pgs: 3 active+recovering, 24 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 792 op/s; 7846/91280 objects misplaced (8.596%); 9.3 MiB/s, 149 objects/s recovering 2024-04-20T17:47:38.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:37 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2755355286' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:47:38.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:37 smithi171 ceph-mon[44189]: from='client.27935 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:47:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:37 smithi005 ceph-mon[39562]: pgmap v2299: 27 pgs: 3 active+recovering, 24 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 792 op/s; 7846/91280 objects misplaced (8.596%); 9.3 MiB/s, 149 objects/s recovering 2024-04-20T17:47:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:37 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2755355286' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:47:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:37 smithi005 ceph-mon[39562]: from='client.27935 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:47:38.397 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:47:38.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:38.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:47:38.504 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:47:38.505 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:47:38.542 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:38.749 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:47:38.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:38.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:47:38.853 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:47:38.884 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:38 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:47:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:38 smithi171 ceph-mon[44189]: from='client.27935 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:47:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:38 smithi171 ceph-mon[44189]: osdmap e865: 12 total, 12 up, 12 in 2024-04-20T17:47:39.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:38 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:47:39.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:38 smithi005 ceph-mon[39562]: from='client.27935 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:47:39.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:38 smithi005 ceph-mon[39562]: osdmap e865: 12 total, 12 up, 12 in 2024-04-20T17:47:39.249 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:47:39.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:39.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:47:39.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:38 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:47:39.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:38 smithi106 ceph-mon[44423]: from='client.27935 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:47:39.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:38 smithi106 ceph-mon[44423]: osdmap e865: 12 total, 12 up, 12 in 2024-04-20T17:47:39.358 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:47:39.395 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:39.435 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:47:39.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:39.446 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:47:39.547 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:47:39.594 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:39.790 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:47:39.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:39.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:47:39.899 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:47:39.928 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:39.947 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:47:39.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:39.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:47:39.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:39 smithi005 ceph-mon[39562]: pgmap v2301: 27 pgs: 3 active+recovering, 24 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 773 op/s; 7583/89798 objects misplaced (8.445%); 12 MiB/s, 190 objects/s recovering 2024-04-20T17:47:40.050 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:47:40.074 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:39 smithi106 ceph-mon[44423]: pgmap v2301: 27 pgs: 3 active+recovering, 24 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 773 op/s; 7583/89798 objects misplaced (8.445%); 12 MiB/s, 190 objects/s recovering 2024-04-20T17:47:40.076 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:40.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:39 smithi171 ceph-mon[44189]: pgmap v2301: 27 pgs: 3 active+recovering, 24 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 773 op/s; 7583/89798 objects misplaced (8.445%); 12 MiB/s, 190 objects/s recovering 2024-04-20T17:47:40.096 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:47:40.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:40.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:47:40.201 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:47:40.228 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:40.251 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:47:40.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:40.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:47:40.354 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:47:40.383 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:40.394 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:47:40.394 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:47:40.404 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:47:40.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:40.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:47:40.507 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:47:40.564 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:40.608 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:47:40.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:40.615 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:47:40.716 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:47:40.750 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:40.887 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:47:40.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:40.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:47:40.927 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:47:40.927 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:47:40.992 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:47:41.024 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:40 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:47:41.024 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:40 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:47:41.024 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:40 smithi005 ceph-mon[39562]: osdmap e866: 12 total, 12 up, 12 in 2024-04-20T17:47:41.024 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:40 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:47:41.024 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:40 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:47:41.024 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:40 smithi005 ceph-mon[39562]: pgmap v2303: 27 pgs: 3 active+recovering, 24 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 715 op/s; 7583/86320 objects misplaced (8.785%); 6.1 MiB/s, 97 objects/s recovering 2024-04-20T17:47:41.027 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:40 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:47:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:40 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:47:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:40 smithi106 ceph-mon[44423]: osdmap e866: 12 total, 12 up, 12 in 2024-04-20T17:47:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:40 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:47:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:40 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:47:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:40 smithi106 ceph-mon[44423]: pgmap v2303: 27 pgs: 3 active+recovering, 24 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 715 op/s; 7583/86320 objects misplaced (8.785%); 6.1 MiB/s, 97 objects/s recovering 2024-04-20T17:47:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:40 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:47:41.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:40 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:47:41.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:40 smithi171 ceph-mon[44189]: osdmap e866: 12 total, 12 up, 12 in 2024-04-20T17:47:41.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:40 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:47:41.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:40 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:47:41.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:40 smithi171 ceph-mon[44189]: pgmap v2303: 27 pgs: 3 active+recovering, 24 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 715 op/s; 7583/86320 objects misplaced (8.785%); 6.1 MiB/s, 97 objects/s recovering 2024-04-20T17:47:41.841 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:47:42.103 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:47:42.105 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:41 smithi005 ceph-mon[39562]: osdmap e867: 12 total, 12 up, 12 in 2024-04-20T17:47:42.105 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:41 smithi005 ceph-mon[39562]: osdmap e868: 12 total, 12 up, 12 in 2024-04-20T17:47:42.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:42.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:47:42.207 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:47:42.232 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:41 smithi106 ceph-mon[44423]: osdmap e867: 12 total, 12 up, 12 in 2024-04-20T17:47:42.232 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:41 smithi106 ceph-mon[44423]: osdmap e868: 12 total, 12 up, 12 in 2024-04-20T17:47:42.234 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:42.256 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:47:42.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:42.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:47:42.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:41 smithi171 ceph-mon[44189]: osdmap e867: 12 total, 12 up, 12 in 2024-04-20T17:47:42.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:41 smithi171 ceph-mon[44189]: osdmap e868: 12 total, 12 up, 12 in 2024-04-20T17:47:42.360 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:47:42.415 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:42.449 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:47:42.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:42.453 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:47:42.554 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:47:42.587 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:42.611 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:47:42.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:42.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:47:42.715 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:47:42.745 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:42.765 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:47:42.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:42.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:47:42.867 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:47:42.868 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:42 smithi106 ceph-mon[44423]: pgmap v2306: 27 pgs: 3 active+recovering, 24 active+clean; 2.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 830 op/s; 6801/81684 objects misplaced (8.326%); 10 MiB/s, 160 objects/s recovering 2024-04-20T17:47:42.869 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:47:42.903 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:43.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:47:43.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:42 smithi005 ceph-mon[39562]: pgmap v2306: 27 pgs: 3 active+recovering, 24 active+clean; 2.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 830 op/s; 6801/81684 objects misplaced (8.326%); 10 MiB/s, 160 objects/s recovering 2024-04-20T17:47:43.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:47:43.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:42 smithi171 ceph-mon[44189]: pgmap v2306: 27 pgs: 3 active+recovering, 24 active+clean; 2.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 830 op/s; 6801/81684 objects misplaced (8.326%); 10 MiB/s, 160 objects/s recovering 2024-04-20T17:47:44.136 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:47:44.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:44.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:47:44.244 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:47:44.331 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:44.399 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:47:44.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:44.408 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:47:44.508 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:47:44.588 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:44.618 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:47:44.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:44.625 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:47:44.726 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:47:44.761 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:44.799 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:47:44.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:44.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:47:44.902 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:47:44.952 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:47:44.960 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:44.978 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:47:45.024 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:47:45.071 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:47:45.072 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:47:45.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:45.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:47:45.100 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:47:45.124 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:47:45.150 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:47:45.181 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:47:45.189 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:47:45.207 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:45.225 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:47:45.226 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:47:45.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:45.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:47:45.268 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:47:45.269 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:47:45.269 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:47:45.269 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 868, 2024-04-20T17:47:45.270 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "13c6e76c-ff37-11ee-bc93-c7b262605968", 2024-04-20T17:47:45.270 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-04-20T16:58:25.272557+0000", 2024-04-20T17:47:45.270 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-04-20T17:47:41.668314+0000", 2024-04-20T17:47:45.270 INFO:teuthology.orchestra.run.smithi005.stdout: "last_up_change": "2024-04-20T17:44:19.738797+0000", 2024-04-20T17:47:45.270 INFO:teuthology.orchestra.run.smithi005.stdout: "last_in_change": "2024-04-20T17:46:36.769424+0000", 2024-04-20T17:47:45.270 INFO:teuthology.orchestra.run.smithi005.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-20T17:47:45.270 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_num": 5806080, 2024-04-20T17:47:45.270 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_set": [ 2024-04-20T17:47:45.270 INFO:teuthology.orchestra.run.smithi005.stdout: "nodeep-scrub", 2024-04-20T17:47:45.270 INFO:teuthology.orchestra.run.smithi005.stdout: "noscrub", 2024-04-20T17:47:45.270 INFO:teuthology.orchestra.run.smithi005.stdout: "pglog_hardlimit", 2024-04-20T17:47:45.270 INFO:teuthology.orchestra.run.smithi005.stdout: "purged_snapdirs", 2024-04-20T17:47:45.270 INFO:teuthology.orchestra.run.smithi005.stdout: "recovery_deletes", 2024-04-20T17:47:45.271 INFO:teuthology.orchestra.run.smithi005.stdout: "sortbitwise" 2024-04-20T17:47:45.271 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:47:45.271 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_version": 33, 2024-04-20T17:47:45.271 INFO:teuthology.orchestra.run.smithi005.stdout: "full_ratio": 0.94999998807907104, 2024-04-20T17:47:45.271 INFO:teuthology.orchestra.run.smithi005.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-20T17:47:45.271 INFO:teuthology.orchestra.run.smithi005.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-20T17:47:45.271 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_snapshot": "", 2024-04-20T17:47:45.271 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_max": 8, 2024-04-20T17:47:45.271 INFO:teuthology.orchestra.run.smithi005.stdout: "max_osd": 12, 2024-04-20T17:47:45.271 INFO:teuthology.orchestra.run.smithi005.stdout: "require_min_compat_client": "luminous", 2024-04-20T17:47:45.271 INFO:teuthology.orchestra.run.smithi005.stdout: "min_compat_client": "jewel", 2024-04-20T17:47:45.271 INFO:teuthology.orchestra.run.smithi005.stdout: "require_osd_release": "squid", 2024-04-20T17:47:45.271 INFO:teuthology.orchestra.run.smithi005.stdout: "allow_crimson": false, 2024-04-20T17:47:45.271 INFO:teuthology.orchestra.run.smithi005.stdout: "pools": [ 2024-04-20T17:47:45.271 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.271 INFO:teuthology.orchestra.run.smithi005.stdout: "pool": 1, 2024-04-20T17:47:45.272 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_name": ".mgr", 2024-04-20T17:47:45.272 INFO:teuthology.orchestra.run.smithi005.stdout: "create_time": "2024-04-20T17:01:35.909493+0000", 2024-04-20T17:47:45.272 INFO:teuthology.orchestra.run.smithi005.stdout: "flags": 1, 2024-04-20T17:47:45.272 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_names": "hashpspool", 2024-04-20T17:47:45.272 INFO:teuthology.orchestra.run.smithi005.stdout: "type": 1, 2024-04-20T17:47:45.272 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 2, 2024-04-20T17:47:45.272 INFO:teuthology.orchestra.run.smithi005.stdout: "min_size": 2, 2024-04-20T17:47:45.272 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_rule": 0, 2024-04-20T17:47:45.272 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_count": 0, 2024-04-20T17:47:45.272 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_target": 0, 2024-04-20T17:47:45.272 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T17:47:45.272 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T17:47:45.272 INFO:teuthology.orchestra.run.smithi005.stdout: "object_hash": 2, 2024-04-20T17:47:45.272 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_autoscale_mode": "off", 2024-04-20T17:47:45.272 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num": 11, 2024-04-20T17:47:45.273 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num": 11, 2024-04-20T17:47:45.273 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num_target": 11, 2024-04-20T17:47:45.273 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_target": 11, 2024-04-20T17:47:45.273 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_pending": 11, 2024-04-20T17:47:45.273 INFO:teuthology.orchestra.run.smithi005.stdout: "last_pg_merge_meta": { 2024-04-20T17:47:45.273 INFO:teuthology.orchestra.run.smithi005.stdout: "source_pgid": "0.0", 2024-04-20T17:47:45.273 INFO:teuthology.orchestra.run.smithi005.stdout: "ready_epoch": 0, 2024-04-20T17:47:45.273 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_started": 0, 2024-04-20T17:47:45.273 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_clean": 0, 2024-04-20T17:47:45.273 INFO:teuthology.orchestra.run.smithi005.stdout: "source_version": "0'0", 2024-04-20T17:47:45.273 INFO:teuthology.orchestra.run.smithi005.stdout: "target_version": "0'0" 2024-04-20T17:47:45.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.274 INFO:teuthology.orchestra.run.smithi005.stdout: "last_change": "838", 2024-04-20T17:47:45.274 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend": "0", 2024-04-20T17:47:45.274 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-20T17:47:45.274 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_preluminous": "818", 2024-04-20T17:47:45.274 INFO:teuthology.orchestra.run.smithi005.stdout: "auid": 0, 2024-04-20T17:47:45.274 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_mode": "selfmanaged", 2024-04-20T17:47:45.274 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_seq": 0, 2024-04-20T17:47:45.274 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_epoch": 0, 2024-04-20T17:47:45.274 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_snaps": [], 2024-04-20T17:47:45.274 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_snaps": "[]", 2024-04-20T17:47:45.274 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_bytes": 0, 2024-04-20T17:47:45.274 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_objects": 0, 2024-04-20T17:47:45.274 INFO:teuthology.orchestra.run.smithi005.stdout: "tiers": [], 2024-04-20T17:47:45.274 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_of": -1, 2024-04-20T17:47:45.275 INFO:teuthology.orchestra.run.smithi005.stdout: "read_tier": -1, 2024-04-20T17:47:45.275 INFO:teuthology.orchestra.run.smithi005.stdout: "write_tier": -1, 2024-04-20T17:47:45.275 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_mode": "none", 2024-04-20T17:47:45.275 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_bytes": 0, 2024-04-20T17:47:45.275 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_objects": 0, 2024-04-20T17:47:45.275 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T17:47:45.275 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T17:47:45.275 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T17:47:45.275 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_flush_age": 0, 2024-04-20T17:47:45.275 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_evict_age": 0, 2024-04-20T17:47:45.275 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_profile": "", 2024-04-20T17:47:45.275 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_params": { 2024-04-20T17:47:45.275 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "none" 2024-04-20T17:47:45.276 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.276 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_period": 0, 2024-04-20T17:47:45.276 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_count": 0, 2024-04-20T17:47:45.276 INFO:teuthology.orchestra.run.smithi005.stdout: "use_gmt_hitset": true, 2024-04-20T17:47:45.276 INFO:teuthology.orchestra.run.smithi005.stdout: "min_read_recency_for_promote": 0, 2024-04-20T17:47:45.276 INFO:teuthology.orchestra.run.smithi005.stdout: "min_write_recency_for_promote": 0, 2024-04-20T17:47:45.276 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T17:47:45.276 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_search_last_n": 0, 2024-04-20T17:47:45.276 INFO:teuthology.orchestra.run.smithi005.stdout: "grade_table": [], 2024-04-20T17:47:45.276 INFO:teuthology.orchestra.run.smithi005.stdout: "stripe_width": 0, 2024-04-20T17:47:45.276 INFO:teuthology.orchestra.run.smithi005.stdout: "expected_num_objects": 0, 2024-04-20T17:47:45.276 INFO:teuthology.orchestra.run.smithi005.stdout: "fast_read": false, 2024-04-20T17:47:45.276 INFO:teuthology.orchestra.run.smithi005.stdout: "options": { 2024-04-20T17:47:45.276 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_max": 32, 2024-04-20T17:47:45.276 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_min": 1 2024-04-20T17:47:45.277 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.277 INFO:teuthology.orchestra.run.smithi005.stdout: "application_metadata": { 2024-04-20T17:47:45.277 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": {} 2024-04-20T17:47:45.277 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.277 INFO:teuthology.orchestra.run.smithi005.stdout: "read_balance": { 2024-04-20T17:47:45.277 INFO:teuthology.orchestra.run.smithi005.stdout: "score_type": "Fair distribution", 2024-04-20T17:47:45.277 INFO:teuthology.orchestra.run.smithi005.stdout: "score_acting": 3, 2024-04-20T17:47:45.277 INFO:teuthology.orchestra.run.smithi005.stdout: "score_stable": 3, 2024-04-20T17:47:45.277 INFO:teuthology.orchestra.run.smithi005.stdout: "optimal_score": 0.9100000262260437, 2024-04-20T17:47:45.277 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_acting": 2.7300000190734863, 2024-04-20T17:47:45.277 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_stable": 2.7300000190734863, 2024-04-20T17:47:45.277 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity_weighted": 0.83594763278961182, 2024-04-20T17:47:45.277 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity": 0.9100000262260437, 2024-04-20T17:47:45.277 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-04-20T17:47:45.277 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.278 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.278 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.278 INFO:teuthology.orchestra.run.smithi005.stdout: "pool": 8, 2024-04-20T17:47:45.278 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_name": "unique_pool_6", 2024-04-20T17:47:45.278 INFO:teuthology.orchestra.run.smithi005.stdout: "create_time": "2024-04-20T17:43:16.604151+0000", 2024-04-20T17:47:45.278 INFO:teuthology.orchestra.run.smithi005.stdout: "flags": 1, 2024-04-20T17:47:45.278 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_names": "hashpspool", 2024-04-20T17:47:45.278 INFO:teuthology.orchestra.run.smithi005.stdout: "type": 1, 2024-04-20T17:47:45.278 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 2, 2024-04-20T17:47:45.278 INFO:teuthology.orchestra.run.smithi005.stdout: "min_size": 2, 2024-04-20T17:47:45.278 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_rule": 0, 2024-04-20T17:47:45.278 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_count": 0, 2024-04-20T17:47:45.278 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_target": 0, 2024-04-20T17:47:45.278 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T17:47:45.278 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T17:47:45.279 INFO:teuthology.orchestra.run.smithi005.stdout: "object_hash": 2, 2024-04-20T17:47:45.279 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_autoscale_mode": "off", 2024-04-20T17:47:45.279 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num": 16, 2024-04-20T17:47:45.279 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num": 16, 2024-04-20T17:47:45.279 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num_target": 16, 2024-04-20T17:47:45.279 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_target": 16, 2024-04-20T17:47:45.279 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_pending": 16, 2024-04-20T17:47:45.279 INFO:teuthology.orchestra.run.smithi005.stdout: "last_pg_merge_meta": { 2024-04-20T17:47:45.279 INFO:teuthology.orchestra.run.smithi005.stdout: "source_pgid": "0.0", 2024-04-20T17:47:45.279 INFO:teuthology.orchestra.run.smithi005.stdout: "ready_epoch": 0, 2024-04-20T17:47:45.279 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_started": 0, 2024-04-20T17:47:45.279 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_clean": 0, 2024-04-20T17:47:45.279 INFO:teuthology.orchestra.run.smithi005.stdout: "source_version": "0'0", 2024-04-20T17:47:45.279 INFO:teuthology.orchestra.run.smithi005.stdout: "target_version": "0'0" 2024-04-20T17:47:45.280 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.280 INFO:teuthology.orchestra.run.smithi005.stdout: "last_change": "782", 2024-04-20T17:47:45.280 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend": "0", 2024-04-20T17:47:45.280 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-20T17:47:45.280 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_preluminous": "0", 2024-04-20T17:47:45.280 INFO:teuthology.orchestra.run.smithi005.stdout: "auid": 0, 2024-04-20T17:47:45.280 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_mode": "selfmanaged", 2024-04-20T17:47:45.280 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_seq": 0, 2024-04-20T17:47:45.280 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_epoch": 0, 2024-04-20T17:47:45.280 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_snaps": [], 2024-04-20T17:47:45.280 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_snaps": "[]", 2024-04-20T17:47:45.280 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_bytes": 0, 2024-04-20T17:47:45.280 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_objects": 0, 2024-04-20T17:47:45.280 INFO:teuthology.orchestra.run.smithi005.stdout: "tiers": [], 2024-04-20T17:47:45.280 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_of": -1, 2024-04-20T17:47:45.280 INFO:teuthology.orchestra.run.smithi005.stdout: "read_tier": -1, 2024-04-20T17:47:45.281 INFO:teuthology.orchestra.run.smithi005.stdout: "write_tier": -1, 2024-04-20T17:47:45.281 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_mode": "none", 2024-04-20T17:47:45.281 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_bytes": 0, 2024-04-20T17:47:45.281 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_objects": 0, 2024-04-20T17:47:45.281 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T17:47:45.281 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T17:47:45.281 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T17:47:45.281 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_flush_age": 0, 2024-04-20T17:47:45.281 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_evict_age": 0, 2024-04-20T17:47:45.281 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_profile": "", 2024-04-20T17:47:45.281 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_params": { 2024-04-20T17:47:45.281 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "none" 2024-04-20T17:47:45.281 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.281 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_period": 0, 2024-04-20T17:47:45.281 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_count": 0, 2024-04-20T17:47:45.282 INFO:teuthology.orchestra.run.smithi005.stdout: "use_gmt_hitset": true, 2024-04-20T17:47:45.282 INFO:teuthology.orchestra.run.smithi005.stdout: "min_read_recency_for_promote": 0, 2024-04-20T17:47:45.282 INFO:teuthology.orchestra.run.smithi005.stdout: "min_write_recency_for_promote": 0, 2024-04-20T17:47:45.282 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T17:47:45.282 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_search_last_n": 0, 2024-04-20T17:47:45.282 INFO:teuthology.orchestra.run.smithi005.stdout: "grade_table": [], 2024-04-20T17:47:45.282 INFO:teuthology.orchestra.run.smithi005.stdout: "stripe_width": 0, 2024-04-20T17:47:45.282 INFO:teuthology.orchestra.run.smithi005.stdout: "expected_num_objects": 0, 2024-04-20T17:47:45.282 INFO:teuthology.orchestra.run.smithi005.stdout: "fast_read": false, 2024-04-20T17:47:45.282 INFO:teuthology.orchestra.run.smithi005.stdout: "options": {}, 2024-04-20T17:47:45.282 INFO:teuthology.orchestra.run.smithi005.stdout: "application_metadata": { 2024-04-20T17:47:45.282 INFO:teuthology.orchestra.run.smithi005.stdout: "rados": {} 2024-04-20T17:47:45.282 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.282 INFO:teuthology.orchestra.run.smithi005.stdout: "read_balance": { 2024-04-20T17:47:45.282 INFO:teuthology.orchestra.run.smithi005.stdout: "score_type": "Fair distribution", 2024-04-20T17:47:45.283 INFO:teuthology.orchestra.run.smithi005.stdout: "score_acting": 3.440000057220459, 2024-04-20T17:47:45.283 INFO:teuthology.orchestra.run.smithi005.stdout: "score_stable": 3.440000057220459, 2024-04-20T17:47:45.283 INFO:teuthology.orchestra.run.smithi005.stdout: "optimal_score": 1, 2024-04-20T17:47:45.283 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_acting": 3.440000057220459, 2024-04-20T17:47:45.283 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_stable": 3.440000057220459, 2024-04-20T17:47:45.283 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity_weighted": 0.83594763278961182, 2024-04-20T17:47:45.283 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity": 0.9100000262260437, 2024-04-20T17:47:45.283 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-04-20T17:47:45.283 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.283 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.283 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:47:45.283 INFO:teuthology.orchestra.run.smithi005.stdout: "osds": [ 2024-04-20T17:47:45.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.283 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 0, 2024-04-20T17:47:45.283 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b", 2024-04-20T17:47:45.284 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:47:45.284 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:47:45.284 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:47:45.284 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:47:45.284 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:47:45.284 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:47:45.284 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 8, 2024-04-20T17:47:45.284 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 845, 2024-04-20T17:47:45.284 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:47:45.284 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:47:45.284 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:47:45.284 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.284 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.284 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.284 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6801", 2024-04-20T17:47:45.284 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:47:45.285 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.285 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.285 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.285 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:47:45.285 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.285 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.285 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.285 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6802", 2024-04-20T17:47:45.285 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:47:45.285 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.285 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.285 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.285 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:47:45.285 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.286 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.286 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.286 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6804", 2024-04-20T17:47:45.286 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:47:45.286 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.286 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.286 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.286 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:47:45.286 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.286 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.286 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.286 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6803", 2024-04-20T17:47:45.286 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:47:45.286 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.286 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.287 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.287 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6801/3901484705", 2024-04-20T17:47:45.287 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6802/3901484705", 2024-04-20T17:47:45.287 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6804/3901484705", 2024-04-20T17:47:45.287 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6803/3901484705", 2024-04-20T17:47:45.287 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:47:45.287 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:47:45.287 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:47:45.287 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.287 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.287 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.287 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 1, 2024-04-20T17:47:45.287 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "aed0c8eb-0e81-4e85-ab8f-a60420060656", 2024-04-20T17:47:45.287 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:47:45.287 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:47:45.287 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:47:45.288 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 0.0313720703125, 2024-04-20T17:47:45.288 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 13, 2024-04-20T17:47:45.288 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 613, 2024-04-20T17:47:45.288 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 659, 2024-04-20T17:47:45.288 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 539, 2024-04-20T17:47:45.288 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 614, 2024-04-20T17:47:45.288 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:47:45.288 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:47:45.288 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.288 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.288 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.288 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6805", 2024-04-20T17:47:45.288 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2415652792 2024-04-20T17:47:45.288 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.288 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.288 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.289 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:47:45.289 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.289 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.289 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.289 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6806", 2024-04-20T17:47:45.289 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2415652792 2024-04-20T17:47:45.289 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.289 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.289 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:47:45.289 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.289 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.289 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.289 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6808", 2024-04-20T17:47:45.290 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2415652792 2024-04-20T17:47:45.290 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.290 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.290 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.290 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:47:45.290 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.290 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.290 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.290 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6807", 2024-04-20T17:47:45.290 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2415652792 2024-04-20T17:47:45.290 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.290 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.290 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.290 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6805/2415652792", 2024-04-20T17:47:45.291 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6806/2415652792", 2024-04-20T17:47:45.291 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6808/2415652792", 2024-04-20T17:47:45.291 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6807/2415652792", 2024-04-20T17:47:45.291 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:47:45.291 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:47:45.291 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:47:45.291 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.291 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.291 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.291 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 2, 2024-04-20T17:47:45.291 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "f733a6a2-19c2-4c7a-af9c-04e282062bb7", 2024-04-20T17:47:45.291 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:47:45.291 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:47:45.291 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:47:45.291 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:47:45.292 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 20, 2024-04-20T17:47:45.292 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 169, 2024-04-20T17:47:45.292 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 202, 2024-04-20T17:47:45.292 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 845, 2024-04-20T17:47:45.292 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 170, 2024-04-20T17:47:45.292 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:47:45.292 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:47:45.292 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.292 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.292 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6809", 2024-04-20T17:47:45.292 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2120275501 2024-04-20T17:47:45.292 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.292 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.292 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:47:45.293 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.293 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.293 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.293 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6810", 2024-04-20T17:47:45.293 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2120275501 2024-04-20T17:47:45.293 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.293 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.293 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.293 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:47:45.293 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.293 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.293 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.293 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6812", 2024-04-20T17:47:45.293 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2120275501 2024-04-20T17:47:45.293 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.293 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.294 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.294 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:47:45.294 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.294 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.294 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.294 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6811", 2024-04-20T17:47:45.294 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2120275501 2024-04-20T17:47:45.294 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.294 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.294 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.294 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6809/2120275501", 2024-04-20T17:47:45.294 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6810/2120275501", 2024-04-20T17:47:45.294 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6812/2120275501", 2024-04-20T17:47:45.294 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6811/2120275501", 2024-04-20T17:47:45.294 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:47:45.295 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:47:45.295 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:47:45.295 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.295 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.295 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 3, 2024-04-20T17:47:45.295 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "8c92f1df-e298-4606-88ae-b8a8e79628a1", 2024-04-20T17:47:45.295 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:47:45.295 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:47:45.295 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:47:45.295 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:47:45.295 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 25, 2024-04-20T17:47:45.295 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 538, 2024-04-20T17:47:45.295 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 561, 2024-04-20T17:47:45.296 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 777, 2024-04-20T17:47:45.296 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 539, 2024-04-20T17:47:45.296 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:47:45.296 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:47:45.296 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.296 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.296 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6813", 2024-04-20T17:47:45.296 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2272824046 2024-04-20T17:47:45.296 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.296 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.296 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.296 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:47:45.296 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.296 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.297 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6814", 2024-04-20T17:47:45.297 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2272824046 2024-04-20T17:47:45.297 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.297 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.297 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:47:45.297 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.297 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.297 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.297 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6816", 2024-04-20T17:47:45.297 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2272824046 2024-04-20T17:47:45.297 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.297 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.297 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:47:45.297 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.298 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.298 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.298 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6815", 2024-04-20T17:47:45.298 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2272824046 2024-04-20T17:47:45.298 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.298 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.298 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.298 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6813/2272824046", 2024-04-20T17:47:45.298 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6814/2272824046", 2024-04-20T17:47:45.298 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6816/2272824046", 2024-04-20T17:47:45.298 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6815/2272824046", 2024-04-20T17:47:45.298 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:47:45.298 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:47:45.298 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:47:45.298 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.299 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.299 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.299 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 4, 2024-04-20T17:47:45.299 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "46df16f1-9607-4d7b-9be8-9d4236e160d1", 2024-04-20T17:47:45.299 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:47:45.299 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:47:45.299 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:47:45.299 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:47:45.299 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:47:45.299 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:47:45.299 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 31, 2024-04-20T17:47:45.299 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 694, 2024-04-20T17:47:45.299 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:47:45.299 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:47:45.299 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:47:45.300 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.300 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.300 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6800", 2024-04-20T17:47:45.300 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:47:45.300 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.300 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.300 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.300 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:47:45.300 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.300 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.300 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6801", 2024-04-20T17:47:45.300 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:47:45.300 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.300 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.301 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.301 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:47:45.301 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.301 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.301 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.301 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6803", 2024-04-20T17:47:45.301 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:47:45.301 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.301 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.301 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.301 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:47:45.301 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.301 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.301 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.301 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6802", 2024-04-20T17:47:45.301 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:47:45.301 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.302 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.302 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.302 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6800/1839524469", 2024-04-20T17:47:45.302 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6801/1839524469", 2024-04-20T17:47:45.302 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6803/1839524469", 2024-04-20T17:47:45.302 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6802/1839524469", 2024-04-20T17:47:45.302 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:47:45.302 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:47:45.302 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:47:45.302 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.302 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.302 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.302 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 5, 2024-04-20T17:47:45.302 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "6603231d-1e5c-4575-bfaa-2eeca2c12215", 2024-04-20T17:47:45.302 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:47:45.303 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:47:45.303 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:47:45.303 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 0, 2024-04-20T17:47:45.303 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:47:45.303 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:47:45.303 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 37, 2024-04-20T17:47:45.303 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 619, 2024-04-20T17:47:45.303 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:47:45.303 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:47:45.303 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:47:45.303 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.303 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.303 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.303 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6804", 2024-04-20T17:47:45.303 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:47:45.303 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.304 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.304 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.304 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:47:45.304 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.304 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.304 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.304 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6805", 2024-04-20T17:47:45.304 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:47:45.304 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.304 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.304 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.304 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:47:45.304 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.304 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.304 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.305 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6807", 2024-04-20T17:47:45.305 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:47:45.305 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.305 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.305 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:47:45.305 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.305 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.305 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.305 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6806", 2024-04-20T17:47:45.305 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:47:45.305 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.305 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.306 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6804/2450315908", 2024-04-20T17:47:45.306 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6805/2450315908", 2024-04-20T17:47:45.306 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6807/2450315908", 2024-04-20T17:47:45.306 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6806/2450315908", 2024-04-20T17:47:45.306 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:47:45.306 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:47:45.306 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:47:45.306 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.306 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.306 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 6, 2024-04-20T17:47:45.306 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "783a55a6-f176-42be-8c8b-af8481d08dc9", 2024-04-20T17:47:45.306 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:47:45.306 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:47:45.307 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:47:45.307 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:47:45.307 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 48, 2024-04-20T17:47:45.307 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 503, 2024-04-20T17:47:45.307 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 535, 2024-04-20T17:47:45.307 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 845, 2024-04-20T17:47:45.307 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 504, 2024-04-20T17:47:45.307 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:47:45.307 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:47:45.307 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.307 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.307 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.307 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6808", 2024-04-20T17:47:45.307 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2439382182 2024-04-20T17:47:45.307 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.308 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.308 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.308 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:47:45.308 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.308 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.308 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.308 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6809", 2024-04-20T17:47:45.308 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2439382182 2024-04-20T17:47:45.308 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.308 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.308 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.308 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:47:45.308 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.308 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.308 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.308 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6811", 2024-04-20T17:47:45.309 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2439382182 2024-04-20T17:47:45.309 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.309 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.309 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.309 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:47:45.309 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.309 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.309 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.309 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6810", 2024-04-20T17:47:45.309 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2439382182 2024-04-20T17:47:45.309 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.309 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.309 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.309 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6808/2439382182", 2024-04-20T17:47:45.309 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6809/2439382182", 2024-04-20T17:47:45.310 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6811/2439382182", 2024-04-20T17:47:45.310 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6810/2439382182", 2024-04-20T17:47:45.310 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:47:45.310 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:47:45.310 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:47:45.310 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.310 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.310 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.310 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 7, 2024-04-20T17:47:45.310 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4", 2024-04-20T17:47:45.310 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:47:45.310 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:47:45.310 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:47:45.310 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:47:45.311 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 398, 2024-04-20T17:47:45.311 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 721, 2024-04-20T17:47:45.311 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 799, 2024-04-20T17:47:45.311 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 846, 2024-04-20T17:47:45.311 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 722, 2024-04-20T17:47:45.311 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:47:45.311 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:47:45.311 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.311 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.311 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.311 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6812", 2024-04-20T17:47:45.311 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2531699401 2024-04-20T17:47:45.311 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.311 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.311 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.311 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:47:45.312 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.312 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.312 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.312 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6813", 2024-04-20T17:47:45.312 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2531699401 2024-04-20T17:47:45.312 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.312 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.312 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.312 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:47:45.312 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.312 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.312 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.312 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6815", 2024-04-20T17:47:45.312 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2531699401 2024-04-20T17:47:45.312 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.313 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.313 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.313 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:47:45.313 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.313 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.313 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.313 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6814", 2024-04-20T17:47:45.313 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2531699401 2024-04-20T17:47:45.313 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.313 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.313 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.313 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6812/2531699401", 2024-04-20T17:47:45.313 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6813/2531699401", 2024-04-20T17:47:45.314 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6815/2531699401", 2024-04-20T17:47:45.314 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6814/2531699401", 2024-04-20T17:47:45.314 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:47:45.314 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:47:45.314 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:47:45.314 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.314 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.314 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.314 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 8, 2024-04-20T17:47:45.314 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "1cb232ad-1f59-4a0a-a215-e67419ad6476", 2024-04-20T17:47:45.314 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:47:45.314 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:47:45.314 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:47:45.314 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:47:45.315 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:47:45.315 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:47:45.315 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 67, 2024-04-20T17:47:45.315 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 777, 2024-04-20T17:47:45.315 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:47:45.315 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:47:45.315 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:47:45.315 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.315 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.315 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.315 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6800", 2024-04-20T17:47:45.315 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:47:45.315 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.315 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.316 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.316 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:47:45.316 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.316 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.316 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.316 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6801", 2024-04-20T17:47:45.316 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:47:45.316 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.316 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.316 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.316 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:47:45.316 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.316 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.316 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.316 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6803", 2024-04-20T17:47:45.317 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:47:45.317 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.317 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.317 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.317 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:47:45.317 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.317 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.317 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.317 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6802", 2024-04-20T17:47:45.317 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:47:45.317 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.317 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.317 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.317 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6800/4123058942", 2024-04-20T17:47:45.317 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6801/4123058942", 2024-04-20T17:47:45.318 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6803/4123058942", 2024-04-20T17:47:45.318 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6802/4123058942", 2024-04-20T17:47:45.318 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:47:45.318 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:47:45.318 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:47:45.318 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.318 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.318 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.318 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 9, 2024-04-20T17:47:45.318 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "b6c17147-b3dc-475b-bb20-f9f712fb99e8", 2024-04-20T17:47:45.318 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:47:45.318 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:47:45.318 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:47:45.318 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:47:45.318 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:47:45.319 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:47:45.319 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 73, 2024-04-20T17:47:45.319 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 823, 2024-04-20T17:47:45.319 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:47:45.319 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:47:45.319 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:47:45.319 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.319 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.319 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.319 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6804", 2024-04-20T17:47:45.319 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:47:45.319 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.319 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.319 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.319 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:47:45.320 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.320 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.320 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.320 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6805", 2024-04-20T17:47:45.320 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:47:45.320 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.320 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.320 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.320 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:47:45.320 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.320 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.320 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.320 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6807", 2024-04-20T17:47:45.320 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:47:45.321 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.321 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.321 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.321 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:47:45.321 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.321 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.321 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.321 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6806", 2024-04-20T17:47:45.321 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:47:45.321 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.321 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.321 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.321 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6804/2877589636", 2024-04-20T17:47:45.321 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6805/2877589636", 2024-04-20T17:47:45.322 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6807/2877589636", 2024-04-20T17:47:45.322 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6806/2877589636", 2024-04-20T17:47:45.322 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:47:45.322 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:47:45.322 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:47:45.322 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.322 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.322 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.322 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 10, 2024-04-20T17:47:45.322 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "1adcdf55-f002-4a39-91e7-378e5fae849c", 2024-04-20T17:47:45.322 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:47:45.322 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:47:45.322 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:47:45.322 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:47:45.323 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:47:45.323 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:47:45.323 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 84, 2024-04-20T17:47:45.323 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 823, 2024-04-20T17:47:45.323 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:47:45.323 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:47:45.323 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:47:45.323 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.323 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.323 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.323 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6808", 2024-04-20T17:47:45.323 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:47:45.323 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.323 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.323 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.323 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:47:45.324 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.324 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.324 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.324 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6809", 2024-04-20T17:47:45.324 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:47:45.324 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.324 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.324 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.324 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:47:45.324 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.324 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.324 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.324 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6811", 2024-04-20T17:47:45.324 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:47:45.324 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.325 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.325 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.325 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:47:45.325 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.325 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.325 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.325 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6810", 2024-04-20T17:47:45.325 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:47:45.325 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.325 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.325 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.325 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6808/181490621", 2024-04-20T17:47:45.325 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6809/181490621", 2024-04-20T17:47:45.325 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6811/181490621", 2024-04-20T17:47:45.326 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6810/181490621", 2024-04-20T17:47:45.326 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:47:45.326 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:47:45.326 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:47:45.326 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.326 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.326 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.326 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 11, 2024-04-20T17:47:45.326 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "737c5aec-694a-4968-819d-fccf22661f4f", 2024-04-20T17:47:45.326 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:47:45.326 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:47:45.326 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:47:45.326 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:47:45.326 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:47:45.326 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:47:45.327 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 92, 2024-04-20T17:47:45.327 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 855, 2024-04-20T17:47:45.327 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:47:45.327 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:47:45.327 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:47:45.327 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.327 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.327 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.327 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6812", 2024-04-20T17:47:45.327 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:47:45.327 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.327 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.327 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.327 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:47:45.328 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.328 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.328 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.328 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6813", 2024-04-20T17:47:45.328 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:47:45.328 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.328 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.328 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.328 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:47:45.328 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.328 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.328 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.328 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6815", 2024-04-20T17:47:45.328 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:47:45.328 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.329 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.329 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.329 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:47:45.329 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:45.329 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.329 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:45.329 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6814", 2024-04-20T17:47:45.329 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:47:45.329 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.329 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.329 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.329 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6812/3373585701", 2024-04-20T17:47:45.329 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6813/3373585701", 2024-04-20T17:47:45.329 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6815/3373585701", 2024-04-20T17:47:45.329 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6814/3373585701", 2024-04-20T17:47:45.330 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:47:45.330 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:47:45.330 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:47:45.330 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:45.330 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.330 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:47:45.330 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_xinfo": [ 2024-04-20T17:47:45.330 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.330 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 0, 2024-04-20T17:47:45.330 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:47:45.330 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:47:45.330 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:47:45.330 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:47:45.330 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:47:45.331 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:47:17.386403+0000", 2024-04-20T17:47:45.331 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:47:45.331 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.331 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.331 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 1, 2024-04-20T17:47:45.331 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:47:45.331 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:47:45.331 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:47:45.331 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:47:45.331 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:47:45.331 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:47:14.265060+0000", 2024-04-20T17:47:45.331 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 613 2024-04-20T17:47:45.331 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.331 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.331 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 2, 2024-04-20T17:47:45.332 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:47:45.332 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:47:45.332 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:47:45.332 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:47:45.332 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:47:45.332 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:46:37.512873+0000", 2024-04-20T17:47:45.332 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 169 2024-04-20T17:47:45.332 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.332 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.332 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 3, 2024-04-20T17:47:45.332 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:47:45.332 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:47:45.332 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:47:45.332 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:47:45.332 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:47:45.332 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:47:40.170187+0000", 2024-04-20T17:47:45.333 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 538 2024-04-20T17:47:45.333 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.333 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.333 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 4, 2024-04-20T17:47:45.333 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:47:45.333 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:47:45.333 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:47:45.333 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:47:45.333 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:47:45.333 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:46:49.692242+0000", 2024-04-20T17:47:45.333 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:47:45.333 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.333 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.333 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 5, 2024-04-20T17:47:45.333 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:47:45.334 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:47:45.334 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:47:45.334 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:47:45.334 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:47:45.334 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:46:39.754243+0000", 2024-04-20T17:47:45.334 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:47:45.334 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.334 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.334 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 6, 2024-04-20T17:47:45.334 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:47:45.334 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:47:45.334 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:47:45.334 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:47:45.334 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:47:45.335 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:46:51.604651+0000", 2024-04-20T17:47:45.335 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 503 2024-04-20T17:47:45.335 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.335 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.335 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 7, 2024-04-20T17:47:45.335 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:47:45.335 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:47:45.335 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:47:45.335 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:47:45.335 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:47:45.335 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:46:52.336856+0000", 2024-04-20T17:47:45.335 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 721 2024-04-20T17:47:45.335 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.335 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.336 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 8, 2024-04-20T17:47:45.336 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:47:45.336 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:47:45.336 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:47:45.336 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:47:45.336 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:47:45.336 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:46:18.755610+0000", 2024-04-20T17:47:45.336 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:47:45.336 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.336 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.336 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 9, 2024-04-20T17:47:45.336 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:47:45.336 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:47:45.336 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:47:45.337 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:47:45.337 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:47:45.337 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:46:36.662481+0000", 2024-04-20T17:47:45.337 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:47:45.337 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.337 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.337 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 10, 2024-04-20T17:47:45.337 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:47:45.337 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:47:45.337 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:47:45.337 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:47:45.337 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:47:45.337 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:46:16.675371+0000", 2024-04-20T17:47:45.338 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:47:45.338 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.338 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:45.338 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 11, 2024-04-20T17:47:45.338 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:47:45.338 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:47:45.338 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:47:45.338 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:47:45.338 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:47:45.338 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:47:39.658754+0000", 2024-04-20T17:47:45.338 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:47:45.338 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.339 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:47:45.339 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_upmap": [], 2024-04-20T17:47:45.339 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_upmap_items": [], 2024-04-20T17:47:45.339 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_upmap_primaries": [], 2024-04-20T17:47:45.339 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_temp": [], 2024-04-20T17:47:45.339 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_temp": [], 2024-04-20T17:47:45.339 INFO:teuthology.orchestra.run.smithi005.stdout: "blocklist": { 2024-04-20T17:47:45.339 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/3881726425": "2024-04-21T16:59:25.808337+0000", 2024-04-20T17:47:45.339 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/338649532": "2024-04-21T16:59:25.808337+0000", 2024-04-20T17:47:45.339 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/1227843664": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:47:45.339 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:6800/1876064631": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:47:45.339 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/160689449": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:47:45.339 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:6800/3521715286": "2024-04-21T16:59:25.808337+0000", 2024-04-20T17:47:45.339 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/3841663462": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:47:45.339 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/687690919": "2024-04-21T16:59:25.808337+0000" 2024-04-20T17:47:45.340 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.340 INFO:teuthology.orchestra.run.smithi005.stdout: "range_blocklist": {}, 2024-04-20T17:47:45.340 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_profiles": { 2024-04-20T17:47:45.340 INFO:teuthology.orchestra.run.smithi005.stdout: "default": { 2024-04-20T17:47:45.340 INFO:teuthology.orchestra.run.smithi005.stdout: "crush-failure-domain": "osd", 2024-04-20T17:47:45.340 INFO:teuthology.orchestra.run.smithi005.stdout: "k": "2", 2024-04-20T17:47:45.340 INFO:teuthology.orchestra.run.smithi005.stdout: "m": "1", 2024-04-20T17:47:45.340 INFO:teuthology.orchestra.run.smithi005.stdout: "plugin": "jerasure", 2024-04-20T17:47:45.340 INFO:teuthology.orchestra.run.smithi005.stdout: "technique": "reed_sol_van" 2024-04-20T17:47:45.340 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.340 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:45.340 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_snaps_queue": [], 2024-04-20T17:47:45.340 INFO:teuthology.orchestra.run.smithi005.stdout: "new_removed_snaps": [], 2024-04-20T17:47:45.340 INFO:teuthology.orchestra.run.smithi005.stdout: "new_purged_snaps": [], 2024-04-20T17:47:45.340 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_node_flags": {}, 2024-04-20T17:47:45.340 INFO:teuthology.orchestra.run.smithi005.stdout: "device_class_flags": {}, 2024-04-20T17:47:45.340 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": { 2024-04-20T17:47:45.341 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode_enabled": false, 2024-04-20T17:47:45.341 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_bucket_count": 0, 2024-04-20T17:47:45.341 INFO:teuthology.orchestra.run.smithi005.stdout: "degraded_stretch_mode": 0, 2024-04-20T17:47:45.341 INFO:teuthology.orchestra.run.smithi005.stdout: "recovering_stretch_mode": 0, 2024-04-20T17:47:45.341 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode_bucket": 0 2024-04-20T17:47:45.341 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:45.341 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:47:45.341 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:47:45.342 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:47:45.345 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:47:45.347 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:47:45.347 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:47:45.347 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:47:45.347 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:47:45.347 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:47:45.347 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:47:45.347 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:47:45.347 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:47:45.347 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:47:45.347 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:47:45.347 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:47:45.347 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:47:45.377 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:45.405 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:47:45.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:45.411 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:47:45.512 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:47:45.539 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:45.562 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:47:45.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:45.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:47:45.666 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:47:45.692 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:45.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:47:45.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:47:45.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:45 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:47:45.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:45 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:47:45.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:45 smithi005 ceph-mon[39562]: pgmap v2307: 27 pgs: 3 active+recovering, 24 active+clean; 2.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 838 op/s; 6801/79740 objects misplaced (8.529%); 8.1 MiB/s, 130 objects/s recovering 2024-04-20T17:47:45.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:45 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3078477680' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:47:45.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:45.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:45.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:45.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:45.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:45.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:45 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/670505610' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-20T17:47:45.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:47:45.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:47:45.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:47:45.712 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:47:45.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:45.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:47:45.744 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-20T17:47:45.815 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:47:45.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:47:45.839 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:47:45.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:45 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:47:45.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:45 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:47:45.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:45 smithi171 ceph-mon[44189]: pgmap v2307: 27 pgs: 3 active+recovering, 24 active+clean; 2.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 838 op/s; 6801/79740 objects misplaced (8.529%); 8.1 MiB/s, 130 objects/s recovering 2024-04-20T17:47:45.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:45 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3078477680' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:47:45.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:45.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:45.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:45.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:45.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:45.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:45 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/670505610' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-20T17:47:45.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:47:45.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:47:45.840 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:47:45.869 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:45.964 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:47:45.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:45.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:47:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:47:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:47:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:45 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:47:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:45 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:47:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:45 smithi106 ceph-mon[44423]: pgmap v2307: 27 pgs: 3 active+recovering, 24 active+clean; 2.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 838 op/s; 6801/79740 objects misplaced (8.529%); 8.1 MiB/s, 130 objects/s recovering 2024-04-20T17:47:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:45 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3078477680' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:47:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:46.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:46.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:45 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/670505610' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-20T17:47:46.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:47:46.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:47:46.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:47:46.073 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:47:46.118 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:46.374 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:47:46.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:46.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:47:46.481 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:47:46.553 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:46.601 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:47:46.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:46.608 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:47:46.709 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:47:46.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:46.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:46.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:46.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:46.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:46.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:46.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:46.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:46 smithi005 ceph-mon[39562]: osdmap e869: 12 total, 12 up, 12 in 2024-04-20T17:47:46.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:46 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:47:46.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:46 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:47:46.735 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:46.753 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:47:46.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:46.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:47:46.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:46.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:46.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:46.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:46.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:46.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:46.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:46.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:46 smithi171 ceph-mon[44189]: osdmap e869: 12 total, 12 up, 12 in 2024-04-20T17:47:46.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:46 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:47:46.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:46 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:47:46.857 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:47:46.882 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:46.882 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:46.882 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:46.882 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:46.882 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:46.882 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:46.882 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:46.882 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:46 smithi106 ceph-mon[44423]: osdmap e869: 12 total, 12 up, 12 in 2024-04-20T17:47:46.882 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:46 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:47:46.882 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:46 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:47:46.884 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:46.888 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:47:46.904 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:47:46.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:46.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:47:46.922 INFO:tasks.thrashosds.thrasher:j is {'epoch': 868, 'fsid': '13c6e76c-ff37-11ee-bc93-c7b262605968', 'created': '2024-04-20T16:58:25.272557+0000', 'modified': '2024-04-20T17:47:41.668314+0000', 'last_up_change': '2024-04-20T17:44:19.738797+0000', 'last_in_change': '2024-04-20T17:46:36.769424+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 33, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-20T17:01:35.909493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '838', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '818', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 0.9100000262260437, 'raw_score_acting': 2.7300000190734863, 'raw_score_stable': 2.7300000190734863, 'primary_affinity_weighted': 0.8359476327896118, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 8, 'pool_name': 'unique_pool_6', 'create_time': '2024-04-20T17:43:16.604151+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '782', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.440000057220459, 'score_stable': 3.440000057220459, 'optimal_score': 1, 'raw_score_acting': 3.440000057220459, 'raw_score_stable': 3.440000057220459, 'primary_affinity_weighted': 0.8359476327896118, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.8399999737739563}}], 'osds': [{'osd': 0, 'uuid': 'ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 845, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6801', 'nonce': 3901484705}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6802', 'nonce': 3901484705}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6804', 'nonce': 3901484705}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6803', 'nonce': 3901484705}]}, 'public_addr': '172.21.15.5:6801/3901484705', 'cluster_addr': '172.21.15.5:6802/3901484705', 'heartbeat_back_addr': '172.21.15.5:6804/3901484705', 'heartbeat_front_addr': '172.21.15.5:6803/3901484705', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'aed0c8eb-0e81-4e85-ab8f-a60420060656', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.0313720703125, 'last_clean_begin': 13, 'last_clean_end': 613, 'up_from': 659, 'up_thru': 539, 'down_at': 614, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6805', 'nonce': 2415652792}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6806', 'nonce': 2415652792}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6808', 'nonce': 2415652792}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6807', 'nonce': 2415652792}]}, 'public_addr': '172.21.15.5:6805/2415652792', 'cluster_addr': '172.21.15.5:6806/2415652792', 'heartbeat_back_addr': '172.21.15.5:6808/2415652792', 'heartbeat_front_addr': '172.21.15.5:6807/2415652792', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f733a6a2-19c2-4c7a-af9c-04e282062bb7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 20, 'last_clean_end': 169, 'up_from': 202, 'up_thru': 845, 'down_at': 170, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6809', 'nonce': 2120275501}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6810', 'nonce': 2120275501}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6812', 'nonce': 2120275501}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6811', 'nonce': 2120275501}]}, 'public_addr': '172.21.15.5:6809/2120275501', 'cluster_addr': '172.21.15.5:6810/2120275501', 'heartbeat_back_addr': '172.21.15.5:6812/2120275501', 'heartbeat_front_addr': '172.21.15.5:6811/2120275501', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '8c92f1df-e298-4606-88ae-b8a8e79628a1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 25, 'last_clean_end': 538, 'up_from': 561, 'up_thru': 777, 'down_at': 539, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6813', 'nonce': 2272824046}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6814', 'nonce': 2272824046}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6816', 'nonce': 2272824046}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6815', 'nonce': 2272824046}]}, 'public_addr': '172.21.15.5:6813/2272824046', 'cluster_addr': '172.21.15.5:6814/2272824046', 'heartbeat_back_addr': '172.21.15.5:6816/2272824046', 'heartbeat_front_addr': '172.21.15.5:6815/2272824046', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '46df16f1-9607-4d7b-9be8-9d4236e160d1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 694, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6800', 'nonce': 1839524469}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6801', 'nonce': 1839524469}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6803', 'nonce': 1839524469}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6802', 'nonce': 1839524469}]}, 'public_addr': '172.21.15.106:6800/1839524469', 'cluster_addr': '172.21.15.106:6801/1839524469', 'heartbeat_back_addr': '172.21.15.106:6803/1839524469', 'heartbeat_front_addr': '172.21.15.106:6802/1839524469', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6603231d-1e5c-4575-bfaa-2eeca2c12215', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 619, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6804', 'nonce': 2450315908}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6805', 'nonce': 2450315908}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6807', 'nonce': 2450315908}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6806', 'nonce': 2450315908}]}, 'public_addr': '172.21.15.106:6804/2450315908', 'cluster_addr': '172.21.15.106:6805/2450315908', 'heartbeat_back_addr': '172.21.15.106:6807/2450315908', 'heartbeat_front_addr': '172.21.15.106:6806/2450315908', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '783a55a6-f176-42be-8c8b-af8481d08dc9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 48, 'last_clean_end': 503, 'up_from': 535, 'up_thru': 845, 'down_at': 504, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6808', 'nonce': 2439382182}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6809', 'nonce': 2439382182}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6811', 'nonce': 2439382182}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6810', 'nonce': 2439382182}]}, 'public_addr': '172.21.15.106:6808/2439382182', 'cluster_addr': '172.21.15.106:6809/2439382182', 'heartbeat_back_addr': '172.21.15.106:6811/2439382182', 'heartbeat_front_addr': '172.21.15.106:6810/2439382182', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 398, 'last_clean_end': 721, 'up_from': 799, 'up_thru': 846, 'down_at': 722, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6812', 'nonce': 2531699401}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6813', 'nonce': 2531699401}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6815', 'nonce': 2531699401}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6814', 'nonce': 2531699401}]}, 'public_addr': '172.21.15.106:6812/2531699401', 'cluster_addr': '172.21.15.106:6813/2531699401', 'heartbeat_back_addr': '172.21.15.106:6815/2531699401', 'heartbeat_front_addr': '172.21.15.106:6814/2531699401', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '1cb232ad-1f59-4a0a-a215-e67419ad6476', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 67, 'up_thru': 777, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6800', 'nonce': 4123058942}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6801', 'nonce': 4123058942}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6803', 'nonce': 4123058942}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6802', 'nonce': 4123058942}]}, 'public_addr': '172.21.15.171:6800/4123058942', 'cluster_addr': '172.21.15.171:6801/4123058942', 'heartbeat_back_addr': '172.21.15.171:6803/4123058942', 'heartbeat_front_addr': '172.21.15.171:6802/4123058942', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'b6c17147-b3dc-475b-bb20-f9f712fb99e8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 73, 'up_thru': 823, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6804', 'nonce': 2877589636}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6805', 'nonce': 2877589636}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6807', 'nonce': 2877589636}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6806', 'nonce': 2877589636}]}, 'public_addr': '172.21.15.171:6804/2877589636', 'cluster_addr': '172.21.15.171:6805/2877589636', 'heartbeat_back_addr': '172.21.15.171:6807/2877589636', 'heartbeat_front_addr': '172.21.15.171:6806/2877589636', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '1adcdf55-f002-4a39-91e7-378e5fae849c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 84, 'up_thru': 823, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6808', 'nonce': 181490621}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6809', 'nonce': 181490621}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6811', 'nonce': 181490621}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6810', 'nonce': 181490621}]}, 'public_addr': '172.21.15.171:6808/181490621', 'cluster_addr': '172.21.15.171:6809/181490621', 'heartbeat_back_addr': '172.21.15.171:6811/181490621', 'heartbeat_front_addr': '172.21.15.171:6810/181490621', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '737c5aec-694a-4968-819d-fccf22661f4f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 92, 'up_thru': 855, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6812', 'nonce': 3373585701}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6813', 'nonce': 3373585701}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6815', 'nonce': 3373585701}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6814', 'nonce': 3373585701}]}, 'public_addr': '172.21.15.171:6812/3373585701', 'cluster_addr': '172.21.15.171:6813/3373585701', 'heartbeat_back_addr': '172.21.15.171:6815/3373585701', 'heartbeat_front_addr': '172.21.15.171:6814/3373585701', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:47:17.386403+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:47:14.265060+0000', 'dead_epoch': 613}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:46:37.512873+0000', 'dead_epoch': 169}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:47:40.170187+0000', 'dead_epoch': 538}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:46:49.692242+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:46:39.754243+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:46:51.604651+0000', 'dead_epoch': 503}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:46:52.336856+0000', 'dead_epoch': 721}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:46:18.755610+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:46:36.662481+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:46:16.675371+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:47:39.658754+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.5:0/3881726425': '2024-04-21T16:59:25.808337+0000', '172.21.15.5:0/338649532': '2024-04-21T16:59:25.808337+0000', '172.21.15.5:0/1227843664': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:6800/1876064631': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:0/160689449': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:6800/3521715286': '2024-04-21T16:59:25.808337+0000', '172.21.15.5:0/3841663462': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:0/687690919': '2024-04-21T16:59:25.808337+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-20T17:47:46.922 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-04-20T17:47:47.008 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:47:47.035 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:47.055 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:47:47.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:47.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:47:47.158 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:47:47.187 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:47:47.246 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:47.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:47 smithi171 ceph-mon[44189]: osdmap e870: 12 total, 12 up, 12 in 2024-04-20T17:47:47.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:47 smithi171 ceph-mon[44189]: pgmap v2310: 27 pgs: 3 active+recovering, 24 active+clean; 2.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 960 op/s; 6304/75170 objects misplaced (8.386%); 14 MiB/s, 220 objects/s recovering 2024-04-20T17:47:47.866 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:47 smithi005 ceph-mon[39562]: osdmap e870: 12 total, 12 up, 12 in 2024-04-20T17:47:47.866 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:47 smithi005 ceph-mon[39562]: pgmap v2310: 27 pgs: 3 active+recovering, 24 active+clean; 2.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 960 op/s; 6304/75170 objects misplaced (8.386%); 14 MiB/s, 220 objects/s recovering 2024-04-20T17:47:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:47 smithi106 ceph-mon[44423]: osdmap e870: 12 total, 12 up, 12 in 2024-04-20T17:47:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:47 smithi106 ceph-mon[44423]: pgmap v2310: 27 pgs: 3 active+recovering, 24 active+clean; 2.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 960 op/s; 6304/75170 objects misplaced (8.386%); 14 MiB/s, 220 objects/s recovering 2024-04-20T17:47:48.308 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:47:48.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:48.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:47:48.394 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:47:48.425 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:47:48.452 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:48.474 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:47:48.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:48.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:47:48.578 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:47:48.606 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:48.626 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:47:48.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:48.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:47:48.730 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:47:48.789 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:48.827 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:47:48.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:48.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:47:48.932 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:47:48.964 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:48.983 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:47:48.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:48.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:47:49.087 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:47:49.121 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:49.167 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:47:49.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:49.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:47:49.274 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:47:49.315 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:49.423 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:47:49.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:49.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:47:49.532 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:47:49.593 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:49.637 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:47:49.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:49.643 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:47:49.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:49 smithi005 ceph-mon[39562]: pgmap v2311: 27 pgs: 3 active+recovering, 24 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 688 op/s; 6053/73608 objects misplaced (8.223%); 9.0 MiB/s, 144 objects/s recovering 2024-04-20T17:47:49.744 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:47:49.794 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:49.970 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:47:49.999 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:47:50.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:50.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:47:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:49 smithi106 ceph-mon[44423]: pgmap v2311: 27 pgs: 3 active+recovering, 24 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 688 op/s; 6053/73608 objects misplaced (8.223%); 9.0 MiB/s, 144 objects/s recovering 2024-04-20T17:47:50.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:49 smithi171 ceph-mon[44189]: pgmap v2311: 27 pgs: 3 active+recovering, 24 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 688 op/s; 6053/73608 objects misplaced (8.223%); 9.0 MiB/s, 144 objects/s recovering 2024-04-20T17:47:50.106 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:47:50.132 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:50.155 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:47:50.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:50.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:47:50.259 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:47:50.309 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:50.359 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:47:50.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:50.367 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:47:50.468 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:47:50.524 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:50.551 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:47:50.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:50.558 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:47:50.659 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:47:50.702 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:50.751 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:47:50.778 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:50 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3809026058' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:47:50.778 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:50 smithi005 ceph-mon[39562]: from='client.27953 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:47:50.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:50 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3809026058' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:47:50.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:50 smithi171 ceph-mon[44189]: from='client.27953 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:47:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:50 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3809026058' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:47:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:50 smithi106 ceph-mon[44423]: from='client.27953 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:47:51.922 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 8, 9, 10, 11, 1, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 4, 5, 8, 9, 10, 11, 2, 6, 3, 1, 7] 2024-04-20T17:47:51.922 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:47:51.922 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-20T17:47:51.923 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-20T17:47:51.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:51 smithi005 ceph-mon[39562]: pgmap v2312: 27 pgs: 3 active+recovering, 24 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 736 op/s; 6053/69892 objects misplaced (8.661%); 5.8 MiB/s, 93 objects/s recovering 2024-04-20T17:47:51.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:51 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:47:51.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:51 smithi005 ceph-mon[39562]: from='client.27953 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:47:51.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:51 smithi005 ceph-mon[39562]: osdmap e871: 12 total, 12 up, 12 in 2024-04-20T17:47:51.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:51 smithi005 ceph-mon[39562]: osdmap e872: 12 total, 12 up, 12 in 2024-04-20T17:47:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:51 smithi106 ceph-mon[44423]: pgmap v2312: 27 pgs: 3 active+recovering, 24 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 736 op/s; 6053/69892 objects misplaced (8.661%); 5.8 MiB/s, 93 objects/s recovering 2024-04-20T17:47:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:51 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:47:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:51 smithi106 ceph-mon[44423]: from='client.27953 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:47:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:51 smithi106 ceph-mon[44423]: osdmap e871: 12 total, 12 up, 12 in 2024-04-20T17:47:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:51 smithi106 ceph-mon[44423]: osdmap e872: 12 total, 12 up, 12 in 2024-04-20T17:47:52.062 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:47:52.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:52.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:47:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:51 smithi171 ceph-mon[44189]: pgmap v2312: 27 pgs: 3 active+recovering, 24 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 736 op/s; 6053/69892 objects misplaced (8.661%); 5.8 MiB/s, 93 objects/s recovering 2024-04-20T17:47:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:51 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:47:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:51 smithi171 ceph-mon[44189]: from='client.27953 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:47:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:51 smithi171 ceph-mon[44189]: osdmap e871: 12 total, 12 up, 12 in 2024-04-20T17:47:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:51 smithi171 ceph-mon[44189]: osdmap e872: 12 total, 12 up, 12 in 2024-04-20T17:47:52.166 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:47:52.191 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:52.220 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:47:52.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:52.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:47:52.329 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:47:52.385 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:52.441 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:47:52.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:52.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:47:52.545 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:47:52.593 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:47:52.593 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:47:52.593 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:47:52.593 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:47:52.593 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.593 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:250835 7.e 7:7a0d9bb4:::benchmark_data_smithi171_161871_object105109:head [delete] snapc 0=[] ondisk+write+known_if_redirected e756)", 2024-04-20T17:47:52.593 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:42:12.028097+0000", 2024-04-20T17:47:52.593 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 340.39074178099997, 2024-04-20T17:47:52.594 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.036592325000000002, 2024-04-20T17:47:52.594 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:47:52.594 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:47:52.594 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:52.594 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:47:52.594 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:47:52.594 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:47:52.594 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 250835 2024-04-20T17:47:52.594 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.594 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:47:52.594 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.594 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:47:52.594 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:12.028097+0000", 2024-04-20T17:47:52.594 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.595 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.595 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:47:52.595 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:12.028097+0000", 2024-04-20T17:47:52.595 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.595 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.595 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:47:52.595 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:12.028098+0000", 2024-04-20T17:47:52.595 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5499999999999998e-07 2024-04-20T17:47:52.595 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.595 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.595 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:47:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:12.028101+0000", 2024-04-20T17:47:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2210000000000002e-06 2024-04-20T17:47:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:47:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:12.028107+0000", 2024-04-20T17:47:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7640000000000002e-06 2024-04-20T17:47:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:47:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:12.028112+0000", 2024-04-20T17:47:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0549999999999999e-06 2024-04-20T17:47:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.596 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.597 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:47:52.597 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:12.038779+0000", 2024-04-20T17:47:52.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010667415 2024-04-20T17:47:52.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.597 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:47:52.597 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:12.038897+0000", 2024-04-20T17:47:52.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011803600000000001 2024-04-20T17:47:52.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.597 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.597 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:47:52.597 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:12.039018+0000", 2024-04-20T17:47:52.597 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012032 2024-04-20T17:47:52.597 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.598 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.598 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:47:52.598 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:12.049498+0000", 2024-04-20T17:47:52.598 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010480799000000001 2024-04-20T17:47:52.598 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.598 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.598 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:47:52.598 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:12.064534+0000", 2024-04-20T17:47:52.598 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015035402999999999 2024-04-20T17:47:52.598 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.598 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.598 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:47:52.598 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:12.064636+0000", 2024-04-20T17:47:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010272600000000001 2024-04-20T17:47:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:47:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:12.064689+0000", 2024-04-20T17:47:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2930999999999999e-05 2024-04-20T17:47:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:253591 7.e 7:71788685:::benchmark_data_smithi171_161871_object107865:head [delete] snapc 0=[] ondisk+write+known_if_redirected e756)", 2024-04-20T17:47:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:42:15.081571+0000", 2024-04-20T17:47:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 337.33726822400001, 2024-04-20T17:47:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.036053092000000002, 2024-04-20T17:47:52.599 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:47:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:47:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:47:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:47:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:47:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 253591 2024-04-20T17:47:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:47:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:47:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:15.081571+0000", 2024-04-20T17:47:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:47:52.600 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:15.081571+0000", 2024-04-20T17:47:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:47:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:15.081571+0000", 2024-04-20T17:47:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8199999999999999e-07 2024-04-20T17:47:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:47:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:15.081579+0000", 2024-04-20T17:47:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3839999999999999e-06 2024-04-20T17:47:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:47:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:15.081586+0000", 2024-04-20T17:47:52.601 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0859999999999996e-06 2024-04-20T17:47:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:47:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:15.081591+0000", 2024-04-20T17:47:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5280000000000003e-06 2024-04-20T17:47:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:47:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:15.087897+0000", 2024-04-20T17:47:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0063051230000000002 2024-04-20T17:47:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:47:52.602 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:15.088030+0000", 2024-04-20T17:47:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013354299999999999 2024-04-20T17:47:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:47:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:15.088156+0000", 2024-04-20T17:47:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012603700000000001 2024-04-20T17:47:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:47:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:15.107703+0000", 2024-04-20T17:47:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.019547173000000001 2024-04-20T17:47:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:47:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:15.117518+0000", 2024-04-20T17:47:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0098150749999999995 2024-04-20T17:47:52.603 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:47:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:15.117578+0000", 2024-04-20T17:47:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9449e-05 2024-04-20T17:47:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:47:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:15.117624+0000", 2024-04-20T17:47:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5911999999999997e-05 2024-04-20T17:47:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:253592 7.e 7:7251c167:::benchmark_data_smithi171_161871_object107866:head [delete] snapc 0=[] ondisk+write+known_if_redirected e756)", 2024-04-20T17:47:52.604 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:42:15.083397+0000", 2024-04-20T17:47:52.605 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 337.33544230199999, 2024-04-20T17:47:52.605 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.034369481, 2024-04-20T17:47:52.605 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:47:52.605 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:47:52.605 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:52.605 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:47:52.605 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:47:52.605 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:47:52.605 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 253592 2024-04-20T17:47:52.605 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.605 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:47:52.605 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.605 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:47:52.605 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:15.083397+0000", 2024-04-20T17:47:52.606 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.606 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.606 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.606 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:47:52.606 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:15.083397+0000", 2024-04-20T17:47:52.606 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.606 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.606 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.606 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:47:52.606 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:15.083398+0000", 2024-04-20T17:47:52.606 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8560000000000001e-06 2024-04-20T17:47:52.606 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.606 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.606 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:47:52.606 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:15.083403+0000", 2024-04-20T17:47:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1559999999999997e-06 2024-04-20T17:47:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:47:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:15.083411+0000", 2024-04-20T17:47:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2199999999999992e-06 2024-04-20T17:47:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:47:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:15.083417+0000", 2024-04-20T17:47:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1689999999999997e-06 2024-04-20T17:47:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:47:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:15.092882+0000", 2024-04-20T17:47:52.607 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0094647989999999994 2024-04-20T17:47:52.608 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.608 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.608 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:47:52.608 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:15.092994+0000", 2024-04-20T17:47:52.608 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011270299999999999 2024-04-20T17:47:52.608 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.608 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.608 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:47:52.608 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:15.093105+0000", 2024-04-20T17:47:52.608 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011079299999999999 2024-04-20T17:47:52.608 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.608 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.608 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:47:52.608 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:15.107711+0000", 2024-04-20T17:47:52.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014605511 2024-04-20T17:47:52.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.609 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:47:52.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:15.117673+0000", 2024-04-20T17:47:52.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0099619989999999992 2024-04-20T17:47:52.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.609 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:47:52.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:15.117726+0000", 2024-04-20T17:47:52.609 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2917999999999997e-05 2024-04-20T17:47:52.609 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.609 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.609 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:47:52.609 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:15.117766+0000", 2024-04-20T17:47:52.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0357e-05 2024-04-20T17:47:52.610 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.610 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:52.610 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.610 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.610 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.610 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:264025 7.e 7:7c6b0eb1:::benchmark_data_smithi171_161871_object118299:head [delete] snapc 0=[] ondisk+write+known_if_redirected e758)", 2024-04-20T17:47:52.610 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:42:26.981628+0000", 2024-04-20T17:47:52.610 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 325.43721042700002, 2024-04-20T17:47:52.610 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.039671656999999999, 2024-04-20T17:47:52.610 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:47:52.610 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:47:52.610 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:52.610 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:47:52.610 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:47:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:47:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 264025 2024-04-20T17:47:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:47:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:47:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:26.981628+0000", 2024-04-20T17:47:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:47:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:26.981628+0000", 2024-04-20T17:47:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.611 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:47:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:26.981631+0000", 2024-04-20T17:47:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3649999999999998e-06 2024-04-20T17:47:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:47:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:26.981639+0000", 2024-04-20T17:47:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.9670000000000001e-06 2024-04-20T17:47:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:47:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:26.981655+0000", 2024-04-20T17:47:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6201999999999998e-05 2024-04-20T17:47:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:47:52.612 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:26.981664+0000", 2024-04-20T17:47:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9770000000000006e-06 2024-04-20T17:47:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:47:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:26.988440+0000", 2024-04-20T17:47:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.006776575 2024-04-20T17:47:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:47:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:26.988567+0000", 2024-04-20T17:47:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000126284 2024-04-20T17:47:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:47:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:26.988686+0000", 2024-04-20T17:47:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011928 2024-04-20T17:47:52.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:47:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:27.002919+0000", 2024-04-20T17:47:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014233216999999999 2024-04-20T17:47:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:47:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:27.021193+0000", 2024-04-20T17:47:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.018273508000000001 2024-04-20T17:47:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:47:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:27.021241+0000", 2024-04-20T17:47:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8021e-05 2024-04-20T17:47:52.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:47:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:27.021300+0000", 2024-04-20T17:47:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9261000000000001e-05 2024-04-20T17:47:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:265014 7.e 7:7d4e67c4:::benchmark_data_smithi171_161871_object119288:head [delete] snapc 0=[] ondisk+write+known_if_redirected e758)", 2024-04-20T17:47:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:42:28.051136+0000", 2024-04-20T17:47:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 324.36770294799999, 2024-04-20T17:47:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.037057685, 2024-04-20T17:47:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:47:52.615 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:47:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:47:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:47:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:47:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 265014 2024-04-20T17:47:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:47:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:47:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:28.051136+0000", 2024-04-20T17:47:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:47:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:28.051136+0000", 2024-04-20T17:47:52.616 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:47:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:28.051139+0000", 2024-04-20T17:47:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7240000000000001e-06 2024-04-20T17:47:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:47:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:28.051145+0000", 2024-04-20T17:47:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5189999999999998e-06 2024-04-20T17:47:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:47:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:28.051157+0000", 2024-04-20T17:47:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1674e-05 2024-04-20T17:47:52.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.618 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:47:52.618 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:28.051165+0000", 2024-04-20T17:47:52.618 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7340000000000008e-06 2024-04-20T17:47:52.618 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.618 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:47:52.618 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:28.051214+0000", 2024-04-20T17:47:52.618 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9768000000000002e-05 2024-04-20T17:47:52.618 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.618 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:47:52.618 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:28.051366+0000", 2024-04-20T17:47:52.618 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015171800000000001 2024-04-20T17:47:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:47:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:28.051479+0000", 2024-04-20T17:47:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011259 2024-04-20T17:47:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:47:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:28.060435+0000", 2024-04-20T17:47:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0089568010000000003 2024-04-20T17:47:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:47:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:28.088121+0000", 2024-04-20T17:47:52.619 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.027685866 2024-04-20T17:47:52.620 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.620 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.620 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:47:52.620 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:28.088164+0000", 2024-04-20T17:47:52.620 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2685000000000002e-05 2024-04-20T17:47:52.620 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.620 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.620 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:47:52.620 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:28.088194+0000", 2024-04-20T17:47:52.620 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9606000000000001e-05 2024-04-20T17:47:52.620 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.620 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:52.620 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.620 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.621 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.621 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:265015 7.e 7:7412b57a:::benchmark_data_smithi171_161871_object119289:head [delete] snapc 0=[] ondisk+write+known_if_redirected e758)", 2024-04-20T17:47:52.621 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:42:28.051183+0000", 2024-04-20T17:47:52.621 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 324.36765538700001, 2024-04-20T17:47:52.621 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.037177764000000002, 2024-04-20T17:47:52.621 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:47:52.621 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:47:52.621 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:52.621 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:47:52.621 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:47:52.621 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:47:52.621 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 265015 2024-04-20T17:47:52.621 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.621 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:47:52.621 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.622 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:47:52.622 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:28.051183+0000", 2024-04-20T17:47:52.622 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.622 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:47:52.622 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:28.051183+0000", 2024-04-20T17:47:52.622 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.622 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:47:52.622 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:28.051184+0000", 2024-04-20T17:47:52.622 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.37e-07 2024-04-20T17:47:52.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.623 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:47:52.623 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:28.051187+0000", 2024-04-20T17:47:52.623 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2849999999999999e-06 2024-04-20T17:47:52.623 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.623 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:47:52.623 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:28.051193+0000", 2024-04-20T17:47:52.623 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4199999999999998e-06 2024-04-20T17:47:52.623 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.623 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:47:52.623 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:28.051198+0000", 2024-04-20T17:47:52.623 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3979999999999998e-06 2024-04-20T17:47:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:47:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:28.056366+0000", 2024-04-20T17:47:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0051680609999999998 2024-04-20T17:47:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:47:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:28.056490+0000", 2024-04-20T17:47:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012369200000000001 2024-04-20T17:47:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 10", 2024-04-20T17:47:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:28.056588+0000", 2024-04-20T17:47:52.624 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7754e-05 2024-04-20T17:47:52.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.625 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.625 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:47:52.625 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:28.065105+0000", 2024-04-20T17:47:52.625 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0085173809999999992 2024-04-20T17:47:52.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.625 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.625 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:47:52.625 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:28.088297+0000", 2024-04-20T17:47:52.625 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023191558000000001 2024-04-20T17:47:52.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.625 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.625 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:47:52.625 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:28.088323+0000", 2024-04-20T17:47:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5931e-05 2024-04-20T17:47:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:47:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:28.088361+0000", 2024-04-20T17:47:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8547000000000002e-05 2024-04-20T17:47:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:282208 7.8 7:138781e2:::benchmark_data_smithi171_161871_object136482:head [delete] snapc 0=[] ondisk+write+known_if_redirected e764)", 2024-04-20T17:47:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:42:45.881005+0000", 2024-04-20T17:47:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 306.53783362299998, 2024-04-20T17:47:52.626 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.046894993000000003, 2024-04-20T17:47:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:47:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:47:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:47:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:47:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:47:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 282208 2024-04-20T17:47:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:47:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:47:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:45.881005+0000", 2024-04-20T17:47:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.627 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.628 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.628 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:47:52.628 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:45.881005+0000", 2024-04-20T17:47:52.628 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.628 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.628 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.628 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:47:52.628 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:45.881007+0000", 2024-04-20T17:47:52.628 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3650000000000001e-06 2024-04-20T17:47:52.628 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.628 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.628 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:47:52.628 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:45.881011+0000", 2024-04-20T17:47:52.628 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8010000000000003e-06 2024-04-20T17:47:52.628 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.629 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.629 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:47:52.629 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:45.881022+0000", 2024-04-20T17:47:52.629 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0438999999999999e-05 2024-04-20T17:47:52.629 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.629 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.629 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:47:52.629 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:45.881029+0000", 2024-04-20T17:47:52.629 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1550000000000001e-06 2024-04-20T17:47:52.629 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.629 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.629 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:47:52.629 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:45.885819+0000", 2024-04-20T17:47:52.629 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0047897440000000003 2024-04-20T17:47:52.629 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.630 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.630 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:47:52.630 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:45.885926+0000", 2024-04-20T17:47:52.630 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010772000000000001 2024-04-20T17:47:52.630 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.630 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.630 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 4", 2024-04-20T17:47:52.630 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:45.886034+0000", 2024-04-20T17:47:52.630 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010774599999999999 2024-04-20T17:47:52.630 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.630 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.630 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:47:52.630 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:45.902512+0000", 2024-04-20T17:47:52.630 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.016477784999999998 2024-04-20T17:47:52.630 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.631 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.631 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:47:52.631 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:45.927818+0000", 2024-04-20T17:47:52.631 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.025306368999999999 2024-04-20T17:47:52.631 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.631 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.631 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:47:52.631 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:45.927853+0000", 2024-04-20T17:47:52.631 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5129000000000002e-05 2024-04-20T17:47:52.631 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.631 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.631 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:47:52.631 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:45.927900+0000", 2024-04-20T17:47:52.631 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6740000000000003e-05 2024-04-20T17:47:52.631 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27088.0:282568 7.a e764/694 7:5487a2a1:::benchmark_data_smithi171_161871_object136842:head v 764'17469, mlcod=764'17467)", 2024-04-20T17:47:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:42:46.241523+0000", 2024-04-20T17:47:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 306.17731596099998, 2024-04-20T17:47:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.036195758000000001, 2024-04-20T17:47:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:47:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:47:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:47:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:47:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:46.241523+0000", 2024-04-20T17:47:52.632 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.633 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.633 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.633 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:47:52.633 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:46.241523+0000", 2024-04-20T17:47:52.633 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.633 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.633 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.633 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:47:52.633 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:46.241525+0000", 2024-04-20T17:47:52.633 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1880000000000001e-06 2024-04-20T17:47:52.633 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.633 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.633 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:47:52.633 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:46.241535+0000", 2024-04-20T17:47:52.633 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0292000000000001e-05 2024-04-20T17:47:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:47:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:46.241551+0000", 2024-04-20T17:47:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5868000000000001e-05 2024-04-20T17:47:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:47:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:46.241561+0000", 2024-04-20T17:47:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4550000000000001e-06 2024-04-20T17:47:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:47:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:46.241888+0000", 2024-04-20T17:47:52.634 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00032746000000000001 2024-04-20T17:47:52.635 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.635 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.635 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:47:52.635 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:46.241907+0000", 2024-04-20T17:47:52.635 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8634000000000002e-05 2024-04-20T17:47:52.635 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.635 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.635 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:47:52.635 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:46.277679+0000", 2024-04-20T17:47:52.635 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.035771913000000002 2024-04-20T17:47:52.635 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.635 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.635 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:47:52.635 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:46.277719+0000", 2024-04-20T17:47:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9947999999999998e-05 2024-04-20T17:47:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27088.0:286493 7.a e766/694 7:5bb3490b:::benchmark_data_smithi171_161871_object140767:head v 766'17686, mlcod=766'17684)", 2024-04-20T17:47:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:42:50.385646+0000", 2024-04-20T17:47:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 302.03319282400003, 2024-04-20T17:47:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.037307922, 2024-04-20T17:47:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:47:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:47:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:47:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.636 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:47:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.385646+0000", 2024-04-20T17:47:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:47:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.385646+0000", 2024-04-20T17:47:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:47:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.385647+0000", 2024-04-20T17:47:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3799999999999999e-06 2024-04-20T17:47:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:47:52.637 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.385655+0000", 2024-04-20T17:47:52.638 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3760000000000002e-06 2024-04-20T17:47:52.638 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.638 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.638 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:47:52.638 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.385665+0000", 2024-04-20T17:47:52.638 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9839999999999996e-06 2024-04-20T17:47:52.638 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.638 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.638 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:47:52.638 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.385670+0000", 2024-04-20T17:47:52.638 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3399999999999997e-06 2024-04-20T17:47:52.638 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.638 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.638 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:47:52.638 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.385735+0000", 2024-04-20T17:47:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5011000000000005e-05 2024-04-20T17:47:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:47:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.385750+0000", 2024-04-20T17:47:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4858000000000001e-05 2024-04-20T17:47:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:47:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.422909+0000", 2024-04-20T17:47:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.037158704000000001 2024-04-20T17:47:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:47:52.639 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.422954+0000", 2024-04-20T17:47:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5269000000000003e-05 2024-04-20T17:47:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27088.0:286501 7.a e766/694 7:5d83f4cc:::benchmark_data_smithi171_161871_object140775:head v 766'17687, mlcod=766'17684)", 2024-04-20T17:47:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:42:50.396108+0000", 2024-04-20T17:47:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 302.02273072100002, 2024-04-20T17:47:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.034105666999999999, 2024-04-20T17:47:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:47:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:47:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:47:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.640 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:47:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.396108+0000", 2024-04-20T17:47:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:47:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.396108+0000", 2024-04-20T17:47:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:47:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.396110+0000", 2024-04-20T17:47:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8029999999999999e-06 2024-04-20T17:47:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:47:52.641 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.396119+0000", 2024-04-20T17:47:52.642 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9749999999999996e-06 2024-04-20T17:47:52.642 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.642 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.642 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:47:52.642 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.396130+0000", 2024-04-20T17:47:52.642 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1165000000000001e-05 2024-04-20T17:47:52.642 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.642 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.642 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:47:52.642 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.396137+0000", 2024-04-20T17:47:52.642 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3359999999999997e-06 2024-04-20T17:47:52.642 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.642 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.642 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:47:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.402061+0000", 2024-04-20T17:47:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0059235800000000003 2024-04-20T17:47:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:47:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.402082+0000", 2024-04-20T17:47:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0735000000000001e-05 2024-04-20T17:47:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:47:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.430178+0000", 2024-04-20T17:47:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.028096341 2024-04-20T17:47:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.643 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:47:52.644 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.430214+0000", 2024-04-20T17:47:52.644 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5732000000000003e-05 2024-04-20T17:47:52.644 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.644 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:52.644 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.644 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.644 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.644 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:286569 7.8 7:1b2009c3:::benchmark_data_smithi171_161871_object140843:head [delete] snapc 0=[] ondisk+write+known_if_redirected e766)", 2024-04-20T17:47:52.644 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:42:50.472946+0000", 2024-04-20T17:47:52.644 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 301.94589297700003, 2024-04-20T17:47:52.644 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.038892032999999999, 2024-04-20T17:47:52.644 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:47:52.644 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:47:52.644 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:52.644 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:47:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:47:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:47:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 286569 2024-04-20T17:47:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:47:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:47:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.472946+0000", 2024-04-20T17:47:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:47:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.472946+0000", 2024-04-20T17:47:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.645 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.646 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.646 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:47:52.646 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.472947+0000", 2024-04-20T17:47:52.646 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4e-07 2024-04-20T17:47:52.646 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.646 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.646 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:47:52.646 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.472951+0000", 2024-04-20T17:47:52.646 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7570000000000002e-06 2024-04-20T17:47:52.646 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.646 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.646 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:47:52.646 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.472960+0000", 2024-04-20T17:47:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2180000000000005e-06 2024-04-20T17:47:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:47:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.472965+0000", 2024-04-20T17:47:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.716e-06 2024-04-20T17:47:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:47:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.484359+0000", 2024-04-20T17:47:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011393195 2024-04-20T17:47:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:47:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.484442+0000", 2024-04-20T17:47:52.647 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3399000000000006e-05 2024-04-20T17:47:52.648 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.648 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.648 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 4", 2024-04-20T17:47:52.648 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.484520+0000", 2024-04-20T17:47:52.648 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7800999999999996e-05 2024-04-20T17:47:52.648 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.648 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.648 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:47:52.648 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.489247+0000", 2024-04-20T17:47:52.648 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0047275570000000003 2024-04-20T17:47:52.648 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.648 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.648 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:47:52.648 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.511743+0000", 2024-04-20T17:47:52.648 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.022495826 2024-04-20T17:47:52.649 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.649 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.649 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:47:52.649 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.511795+0000", 2024-04-20T17:47:52.649 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1434000000000003e-05 2024-04-20T17:47:52.649 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.649 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.649 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:47:52.649 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.511838+0000", 2024-04-20T17:47:52.649 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3189999999999998e-05 2024-04-20T17:47:52.649 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.649 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:52.649 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.649 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.649 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.650 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:286574 7.8 7:17d129b3:::benchmark_data_smithi171_161871_object140848:head [delete] snapc 0=[] ondisk+write+known_if_redirected e766)", 2024-04-20T17:47:52.650 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:42:50.475710+0000", 2024-04-20T17:47:52.650 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 301.94312893199998, 2024-04-20T17:47:52.650 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.036202098000000002, 2024-04-20T17:47:52.650 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:47:52.650 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:47:52.650 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:52.650 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:47:52.650 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:47:52.650 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:47:52.650 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 286574 2024-04-20T17:47:52.650 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.650 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:47:52.650 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:47:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.475710+0000", 2024-04-20T17:47:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:47:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.475710+0000", 2024-04-20T17:47:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:47:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.475712+0000", 2024-04-20T17:47:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2809999999999998e-06 2024-04-20T17:47:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.651 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:47:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.475718+0000", 2024-04-20T17:47:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5809999999999996e-06 2024-04-20T17:47:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:47:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.475735+0000", 2024-04-20T17:47:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6792000000000001e-05 2024-04-20T17:47:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:47:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.475743+0000", 2024-04-20T17:47:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6740000000000003e-06 2024-04-20T17:47:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.652 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:47:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.489350+0000", 2024-04-20T17:47:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.01360658 2024-04-20T17:47:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:47:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.489464+0000", 2024-04-20T17:47:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000113838 2024-04-20T17:47:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 4", 2024-04-20T17:47:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.489578+0000", 2024-04-20T17:47:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011407499999999999 2024-04-20T17:47:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.653 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:47:52.654 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.499589+0000", 2024-04-20T17:47:52.654 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010011557000000001 2024-04-20T17:47:52.654 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.654 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.654 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:47:52.654 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.511845+0000", 2024-04-20T17:47:52.654 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012255423999999999 2024-04-20T17:47:52.654 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.654 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.654 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:47:52.654 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.511880+0000", 2024-04-20T17:47:52.654 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5577000000000002e-05 2024-04-20T17:47:52.654 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.655 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.655 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:47:52.655 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:50.511912+0000", 2024-04-20T17:47:52.655 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1718999999999999e-05 2024-04-20T17:47:52.655 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.655 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:52.655 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.655 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.655 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.655 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:58376 8.d e789/777 8:b72e24a1:::benchmark_data_smithi171_175757_object58375:head v 789'3618, mlcod=789'3616)", 2024-04-20T17:47:52.655 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:43:49.270296+0000", 2024-04-20T17:47:52.655 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 243.14854262200001, 2024-04-20T17:47:52.655 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10571012, 2024-04-20T17:47:52.655 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:47:52.656 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:47:52.656 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:52.656 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:47:52.656 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.656 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:47:52.656 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:49.270296+0000", 2024-04-20T17:47:52.656 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.656 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.656 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.656 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:47:52.656 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:49.270296+0000", 2024-04-20T17:47:52.656 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.656 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.656 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:47:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:49.270297+0000", 2024-04-20T17:47:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0279999999999999e-06 2024-04-20T17:47:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:47:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:49.270342+0000", 2024-04-20T17:47:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4632999999999998e-05 2024-04-20T17:47:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:47:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:49.270349+0000", 2024-04-20T17:47:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4089999999999997e-06 2024-04-20T17:47:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.657 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.658 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:47:52.658 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:49.270354+0000", 2024-04-20T17:47:52.658 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8459999999999999e-06 2024-04-20T17:47:52.658 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.658 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.658 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:47:52.658 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:49.270392+0000", 2024-04-20T17:47:52.658 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7443e-05 2024-04-20T17:47:52.658 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.658 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.658 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:47:52.658 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:49.270404+0000", 2024-04-20T17:47:52.658 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2279000000000001e-05 2024-04-20T17:47:52.658 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.659 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.659 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:47:52.659 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:49.375975+0000", 2024-04-20T17:47:52.659 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10557098600000001 2024-04-20T17:47:52.659 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.659 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.659 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:47:52.659 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:49.376006+0000", 2024-04-20T17:47:52.659 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1495999999999998e-05 2024-04-20T17:47:52.659 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.659 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:52.659 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.659 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.659 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:58375 8.1 e789/777 8:87ba6d14:::benchmark_data_smithi171_175757_object58374:head v 789'3670, mlcod=789'3668)", 2024-04-20T17:47:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:43:49.270425+0000", 2024-04-20T17:47:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 243.14841353899999, 2024-04-20T17:47:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.103926237, 2024-04-20T17:47:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:47:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:47:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:47:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:47:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:49.270425+0000", 2024-04-20T17:47:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:47:52.660 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:49.270425+0000", 2024-04-20T17:47:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:47:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:49.270426+0000", 2024-04-20T17:47:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0200000000000001e-07 2024-04-20T17:47:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:47:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:49.270484+0000", 2024-04-20T17:47:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7958000000000003e-05 2024-04-20T17:47:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:47:52.661 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:49.370788+0000", 2024-04-20T17:47:52.662 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10030428500000001 2024-04-20T17:47:52.662 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.662 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.662 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:47:52.662 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:49.370800+0000", 2024-04-20T17:47:52.662 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2187e-05 2024-04-20T17:47:52.662 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.662 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.662 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:47:52.662 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:49.370859+0000", 2024-04-20T17:47:52.662 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8539999999999999e-05 2024-04-20T17:47:52.662 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.663 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.663 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:47:52.663 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:49.370876+0000", 2024-04-20T17:47:52.663 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7484999999999999e-05 2024-04-20T17:47:52.663 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.663 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.663 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:47:52.663 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:49.374300+0000", 2024-04-20T17:47:52.663 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0034231309999999998 2024-04-20T17:47:52.663 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.663 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.663 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:47:52.663 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:43:49.374352+0000", 2024-04-20T17:47:52.663 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1949e-05 2024-04-20T17:47:52.663 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:97123 8.1 e796/777 8:8676f073:::benchmark_data_smithi171_175757_object97122:head v 796'6073, mlcod=796'6071)", 2024-04-20T17:47:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:12.978836+0000", 2024-04-20T17:47:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 219.44000241500001, 2024-04-20T17:47:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.033866211, 2024-04-20T17:47:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:47:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:47:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:47:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:47:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:12.978836+0000", 2024-04-20T17:47:52.664 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:47:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:12.978836+0000", 2024-04-20T17:47:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:47:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:12.978838+0000", 2024-04-20T17:47:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2550000000000001e-06 2024-04-20T17:47:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:47:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:12.978890+0000", 2024-04-20T17:47:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2664000000000003e-05 2024-04-20T17:47:52.665 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.666 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.666 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:47:52.666 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:13.003881+0000", 2024-04-20T17:47:52.666 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.024990694000000001 2024-04-20T17:47:52.666 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.666 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.666 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:47:52.666 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:13.003893+0000", 2024-04-20T17:47:52.666 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2001000000000001e-05 2024-04-20T17:47:52.666 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.666 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.666 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:47:52.666 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:13.003949+0000", 2024-04-20T17:47:52.666 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6233999999999997e-05 2024-04-20T17:47:52.666 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.667 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.667 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:47:52.667 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:13.003972+0000", 2024-04-20T17:47:52.667 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2274999999999999e-05 2024-04-20T17:47:52.667 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.667 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.667 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:47:52.667 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:13.012657+0000", 2024-04-20T17:47:52.667 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0086853260000000002 2024-04-20T17:47:52.667 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.667 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.667 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:47:52.667 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:13.012703+0000", 2024-04-20T17:47:52.667 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5762e-05 2024-04-20T17:47:52.667 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(osd.0.0:281529 8.e e801/777 8:70733e01:::benchmark_data_smithi171_175757_object32565:head v 801'7805, mlcod=801'7801)", 2024-04-20T17:47:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:32.703052+0000", 2024-04-20T17:47:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 199.715787189, 2024-04-20T17:47:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.033298544999999999, 2024-04-20T17:47:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:47:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:47:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:47:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:47:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:32.703052+0000", 2024-04-20T17:47:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.668 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:47:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:32.703052+0000", 2024-04-20T17:47:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:47:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:32.703053+0000", 2024-04-20T17:47:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1289999999999999e-06 2024-04-20T17:47:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:47:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:32.703059+0000", 2024-04-20T17:47:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7509999999999997e-06 2024-04-20T17:47:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.669 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.670 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:47:52.670 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:32.703068+0000", 2024-04-20T17:47:52.670 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1290000000000002e-06 2024-04-20T17:47:52.670 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.670 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.670 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:47:52.670 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:32.703075+0000", 2024-04-20T17:47:52.670 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2690000000000002e-06 2024-04-20T17:47:52.670 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.670 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.670 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:47:52.670 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:32.706072+0000", 2024-04-20T17:47:52.670 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0029967539999999999 2024-04-20T17:47:52.670 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.670 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:47:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:32.706092+0000", 2024-04-20T17:47:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0690999999999999e-05 2024-04-20T17:47:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:47:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:32.736309+0000", 2024-04-20T17:47:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.030216496999999998 2024-04-20T17:47:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:47:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:32.736350+0000", 2024-04-20T17:47:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1325000000000002e-05 2024-04-20T17:47:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:52.671 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(osd.0.0:281628 8.e e801/777 8:70ab5e60:::benchmark_data_smithi171_175757_object15057:head v 801'7904, mlcod=801'7895)", 2024-04-20T17:47:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:33.089507+0000", 2024-04-20T17:47:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 199.32933209199999, 2024-04-20T17:47:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.03960731, 2024-04-20T17:47:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:47:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:47:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:47:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:47:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:33.089507+0000", 2024-04-20T17:47:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.672 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:47:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:33.089507+0000", 2024-04-20T17:47:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:47:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:33.089508+0000", 2024-04-20T17:47:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4309999999999999e-06 2024-04-20T17:47:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:47:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:33.089515+0000", 2024-04-20T17:47:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.029e-06 2024-04-20T17:47:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.673 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:47:52.674 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:33.089523+0000", 2024-04-20T17:47:52.674 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.038e-06 2024-04-20T17:47:52.674 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.674 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.674 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:47:52.674 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:33.089529+0000", 2024-04-20T17:47:52.674 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5010000000000004e-06 2024-04-20T17:47:52.674 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.674 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.674 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:47:52.674 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:33.090003+0000", 2024-04-20T17:47:52.674 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00047448899999999998 2024-04-20T17:47:52.674 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.674 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.674 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:47:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:33.090015+0000", 2024-04-20T17:47:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2140000000000001e-05 2024-04-20T17:47:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:47:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:33.129063+0000", 2024-04-20T17:47:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.039047752999999998 2024-04-20T17:47:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:47:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:33.129114+0000", 2024-04-20T17:47:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0929e-05 2024-04-20T17:47:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.675 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(osd.0.0:282429 8.e e801/777 8:7258cef3:::benchmark_data_smithi171_175757_object22974:head v 801'8705, mlcod=801'8691)", 2024-04-20T17:47:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:36.216276+0000", 2024-04-20T17:47:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 196.202562677, 2024-04-20T17:47:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.033448284000000002, 2024-04-20T17:47:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:47:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:47:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:47:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:47:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:36.216276+0000", 2024-04-20T17:47:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:47:52.676 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:36.216276+0000", 2024-04-20T17:47:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:47:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:36.216278+0000", 2024-04-20T17:47:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3599999999999999e-06 2024-04-20T17:47:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:47:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:36.216299+0000", 2024-04-20T17:47:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1452999999999999e-05 2024-04-20T17:47:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:47:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:36.216309+0000", 2024-04-20T17:47:52.677 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6010000000000001e-06 2024-04-20T17:47:52.678 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.678 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.678 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:47:52.678 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:36.216316+0000", 2024-04-20T17:47:52.678 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.729e-06 2024-04-20T17:47:52.678 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.678 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.678 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:47:52.678 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:36.224486+0000", 2024-04-20T17:47:52.678 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0081692480000000005 2024-04-20T17:47:52.678 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.678 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.678 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:47:52.678 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:36.224509+0000", 2024-04-20T17:47:52.678 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3404000000000002e-05 2024-04-20T17:47:52.679 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.679 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.679 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:47:52.679 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:36.249698+0000", 2024-04-20T17:47:52.679 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.025188977000000001 2024-04-20T17:47:52.679 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.679 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.679 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:47:52.679 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:36.249724+0000", 2024-04-20T17:47:52.679 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.6512e-05 2024-04-20T17:47:52.679 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.679 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:52.679 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.679 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.679 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(osd.0.0:282430 8.e e801/777 8:7258d9a7:::benchmark_data_smithi171_175757_object9545:head v 801'8706, mlcod=801'8691)", 2024-04-20T17:47:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:36.220028+0000", 2024-04-20T17:47:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 196.198811127, 2024-04-20T17:47:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.033175280000000001, 2024-04-20T17:47:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:47:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:47:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:47:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:47:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:36.220028+0000", 2024-04-20T17:47:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.680 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:47:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:36.220028+0000", 2024-04-20T17:47:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:47:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:36.220029+0000", 2024-04-20T17:47:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4750000000000001e-06 2024-04-20T17:47:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:47:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:36.220036+0000", 2024-04-20T17:47:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.37e-06 2024-04-20T17:47:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:47:52.681 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:36.220047+0000", 2024-04-20T17:47:52.682 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1810999999999999e-05 2024-04-20T17:47:52.682 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.682 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.682 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:47:52.682 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:36.220053+0000", 2024-04-20T17:47:52.682 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.801e-06 2024-04-20T17:47:52.682 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.682 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.682 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:47:52.682 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:36.228992+0000", 2024-04-20T17:47:52.682 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0089388269999999999 2024-04-20T17:47:52.682 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.682 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.682 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:47:52.682 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:36.229012+0000", 2024-04-20T17:47:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9729e-05 2024-04-20T17:47:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:47:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:36.253156+0000", 2024-04-20T17:47:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.024143886 2024-04-20T17:47:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:47:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:36.253203+0000", 2024-04-20T17:47:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7380999999999999e-05 2024-04-20T17:47:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.683 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(osd.0.0:283606 8.e e803/777 8:74ac8792:::benchmark_data_smithi171_175757_object64727:head v 803'9882, mlcod=803'9878)", 2024-04-20T17:47:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:41.188145+0000", 2024-04-20T17:47:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 191.230693428, 2024-04-20T17:47:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.033586939000000003, 2024-04-20T17:47:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:47:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:47:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:47:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:47:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:47:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:41.188145+0000", 2024-04-20T17:47:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:47:52.684 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:41.188145+0000", 2024-04-20T17:47:52.685 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:47:52.685 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.685 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.685 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:47:52.685 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:41.188146+0000", 2024-04-20T17:47:52.685 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.37e-07 2024-04-20T17:47:52.685 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.685 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.685 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:47:52.685 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:41.188152+0000", 2024-04-20T17:47:52.685 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7239999999999997e-06 2024-04-20T17:47:52.685 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.685 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.685 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:47:52.685 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:41.188159+0000", 2024-04-20T17:47:52.686 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6959999999999999e-06 2024-04-20T17:47:52.686 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.686 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.686 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:47:52.686 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:41.188165+0000", 2024-04-20T17:47:52.686 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5370000000000003e-06 2024-04-20T17:47:52.686 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.686 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.686 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:47:52.686 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:41.191740+0000", 2024-04-20T17:47:52.686 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0035753130000000001 2024-04-20T17:47:52.686 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.686 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.686 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:47:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:41.191769+0000", 2024-04-20T17:47:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8917000000000001e-05 2024-04-20T17:47:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:47:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:41.221684+0000", 2024-04-20T17:47:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.029914811999999999 2024-04-20T17:47:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:47:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:41.221732+0000", 2024-04-20T17:47:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8202999999999999e-05 2024-04-20T17:47:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.687 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:52.688 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:52.688 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:47:52.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:52.689 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:52.689 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:47:52.689 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:47:52.790 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:47:52.838 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:53.401 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:47:53.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:53.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:47:53.417 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-20T17:47:53.508 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:47:53.541 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:53.660 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:47:53.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:53.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:47:53.665 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:47:53.707 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:47:53.716 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:47:53.767 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:47:53.785 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:47:53.823 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:47:53.835 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:53.859 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:47:53.900 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:47:53.936 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:47:53.969 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:53 smithi005 ceph-mon[39562]: pgmap v2315: 27 pgs: 3 active+recovering, 24 active+clean; 2.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 831 op/s; 5253/65042 objects misplaced (8.076%); 11 MiB/s, 172 objects/s recovering 2024-04-20T17:47:53.980 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:47:53.981 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:53 smithi106 ceph-mon[44423]: pgmap v2315: 27 pgs: 3 active+recovering, 24 active+clean; 2.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 831 op/s; 5253/65042 objects misplaced (8.076%); 11 MiB/s, 172 objects/s recovering 2024-04-20T17:47:54.015 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:47:54.049 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:47:54.058 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:47:54.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:54.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:47:54.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:53 smithi171 ceph-mon[44189]: pgmap v2315: 27 pgs: 3 active+recovering, 24 active+clean; 2.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 831 op/s; 5253/65042 objects misplaced (8.076%); 11 MiB/s, 172 objects/s recovering 2024-04-20T17:47:54.089 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:47:54.122 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:47:54.144 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:47:54.162 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:47:54.172 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:47:54.178 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:47:54.178 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:47:54.178 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:47:54.179 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:47:54.179 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:47:54.179 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:47:54.179 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:47:54.179 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:47:54.179 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:47:54.179 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:47:54.179 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:47:54.179 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:47:54.209 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:54.740 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:54 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:47:55.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:54 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:47:55.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:54 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/197505887' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:47:55.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:54 smithi106 ceph-mon[44423]: osdmap e873: 12 total, 12 up, 12 in 2024-04-20T17:47:55.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:55.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:55.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:55.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:55.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:55.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:55.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:54 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:55.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:54 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:47:55.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:54 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:47:55.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:54 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/197505887' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:47:55.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:54 smithi171 ceph-mon[44189]: osdmap e873: 12 total, 12 up, 12 in 2024-04-20T17:47:55.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:55.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:55.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:55.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:55.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:55.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:55.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:54 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:55.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:54 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:47:55.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:54 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:47:55.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:54 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/197505887' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:47:55.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:54 smithi005 ceph-mon[39562]: osdmap e873: 12 total, 12 up, 12 in 2024-04-20T17:47:55.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:55.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:55.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:55.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:55.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:55.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:55.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:54 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:55.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:55.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:55.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:55.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:55.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:55.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:55 smithi106 ceph-mon[44423]: pgmap v2317: 27 pgs: 3 active+recovering, 24 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 883 op/s; 5253/63008 objects misplaced (8.337%); 8.3 MiB/s, 133 objects/s recovering 2024-04-20T17:47:55.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:55 smithi106 ceph-mon[44423]: 8.2 scrub starts 2024-04-20T17:47:55.869 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:47:55.869 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:47:56.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:56.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:56.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:56.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:56.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:56.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:55 smithi171 ceph-mon[44189]: pgmap v2317: 27 pgs: 3 active+recovering, 24 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 883 op/s; 5253/63008 objects misplaced (8.337%); 8.3 MiB/s, 133 objects/s recovering 2024-04-20T17:47:56.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:55 smithi171 ceph-mon[44189]: 8.2 scrub starts 2024-04-20T17:47:56.199 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:56.199 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:56.199 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:56.199 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:56.199 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:47:56.199 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:55 smithi005 ceph-mon[39562]: pgmap v2317: 27 pgs: 3 active+recovering, 24 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 883 op/s; 5253/63008 objects misplaced (8.337%); 8.3 MiB/s, 133 objects/s recovering 2024-04-20T17:47:56.200 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:55 smithi005 ceph-mon[39562]: 8.2 scrub starts 2024-04-20T17:47:56.624 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-20T17:47:56.671 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:47:56.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:56.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:47:56.780 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:47:56.826 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:56.860 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:47:56.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:56.869 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:47:56.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:56 smithi005 ceph-mon[39562]: 8.2 scrub ok 2024-04-20T17:47:56.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:56 smithi005 ceph-mon[39562]: osdmap e874: 12 total, 12 up, 12 in 2024-04-20T17:47:56.970 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:47:56.990 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:47:56.999 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:56 smithi106 ceph-mon[44423]: 8.2 scrub ok 2024-04-20T17:47:56.999 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:56 smithi106 ceph-mon[44423]: osdmap e874: 12 total, 12 up, 12 in 2024-04-20T17:47:57.001 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:56 smithi171 ceph-mon[44189]: 8.2 scrub ok 2024-04-20T17:47:57.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:56 smithi171 ceph-mon[44189]: osdmap e874: 12 total, 12 up, 12 in 2024-04-20T17:47:57.178 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:47:57.194 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:47:57.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:57.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:47:57.308 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:47:57.350 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:57.454 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:47:57.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:57.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:47:57.560 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:47:57.588 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:57.789 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:47:57.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:57.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:47:57.894 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:47:57.920 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:57 smithi106 ceph-mon[44423]: 1.3 scrub starts 2024-04-20T17:47:57.920 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:57 smithi106 ceph-mon[44423]: 1.3 scrub ok 2024-04-20T17:47:57.920 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:57 smithi106 ceph-mon[44423]: pgmap v2318: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering, 23 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 982 op/s; 4693/58098 objects misplaced (8.078%); 14 MiB/s, 226 objects/s recovering 2024-04-20T17:47:57.922 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:57.998 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:47:58.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:58.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:47:58.034 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:57 smithi005 ceph-mon[39562]: 1.3 scrub starts 2024-04-20T17:47:58.034 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:57 smithi005 ceph-mon[39562]: 1.3 scrub ok 2024-04-20T17:47:58.034 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:57 smithi005 ceph-mon[39562]: pgmap v2318: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering, 23 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 982 op/s; 4693/58098 objects misplaced (8.078%); 14 MiB/s, 226 objects/s recovering 2024-04-20T17:47:58.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:57 smithi171 ceph-mon[44189]: 1.3 scrub starts 2024-04-20T17:47:58.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:57 smithi171 ceph-mon[44189]: 1.3 scrub ok 2024-04-20T17:47:58.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:57 smithi171 ceph-mon[44189]: pgmap v2318: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering, 23 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 982 op/s; 4693/58098 objects misplaced (8.078%); 14 MiB/s, 226 objects/s recovering 2024-04-20T17:47:58.102 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:47:58.161 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:58.404 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:47:58.404 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:47:58.404 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 874, 2024-04-20T17:47:58.404 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "13c6e76c-ff37-11ee-bc93-c7b262605968", 2024-04-20T17:47:58.404 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-04-20T16:58:25.272557+0000", 2024-04-20T17:47:58.404 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-04-20T17:47:56.684948+0000", 2024-04-20T17:47:58.404 INFO:teuthology.orchestra.run.smithi005.stdout: "last_up_change": "2024-04-20T17:44:19.738797+0000", 2024-04-20T17:47:58.404 INFO:teuthology.orchestra.run.smithi005.stdout: "last_in_change": "2024-04-20T17:46:36.769424+0000", 2024-04-20T17:47:58.405 INFO:teuthology.orchestra.run.smithi005.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-20T17:47:58.405 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_num": 5804032, 2024-04-20T17:47:58.405 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_set": [ 2024-04-20T17:47:58.405 INFO:teuthology.orchestra.run.smithi005.stdout: "nodeep-scrub", 2024-04-20T17:47:58.405 INFO:teuthology.orchestra.run.smithi005.stdout: "pglog_hardlimit", 2024-04-20T17:47:58.405 INFO:teuthology.orchestra.run.smithi005.stdout: "purged_snapdirs", 2024-04-20T17:47:58.405 INFO:teuthology.orchestra.run.smithi005.stdout: "recovery_deletes", 2024-04-20T17:47:58.405 INFO:teuthology.orchestra.run.smithi005.stdout: "sortbitwise" 2024-04-20T17:47:58.405 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:47:58.405 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_version": 33, 2024-04-20T17:47:58.405 INFO:teuthology.orchestra.run.smithi005.stdout: "full_ratio": 0.94999998807907104, 2024-04-20T17:47:58.405 INFO:teuthology.orchestra.run.smithi005.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-20T17:47:58.405 INFO:teuthology.orchestra.run.smithi005.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-20T17:47:58.405 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_snapshot": "", 2024-04-20T17:47:58.406 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_max": 8, 2024-04-20T17:47:58.406 INFO:teuthology.orchestra.run.smithi005.stdout: "max_osd": 12, 2024-04-20T17:47:58.406 INFO:teuthology.orchestra.run.smithi005.stdout: "require_min_compat_client": "luminous", 2024-04-20T17:47:58.406 INFO:teuthology.orchestra.run.smithi005.stdout: "min_compat_client": "jewel", 2024-04-20T17:47:58.406 INFO:teuthology.orchestra.run.smithi005.stdout: "require_osd_release": "squid", 2024-04-20T17:47:58.406 INFO:teuthology.orchestra.run.smithi005.stdout: "allow_crimson": false, 2024-04-20T17:47:58.406 INFO:teuthology.orchestra.run.smithi005.stdout: "pools": [ 2024-04-20T17:47:58.406 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.406 INFO:teuthology.orchestra.run.smithi005.stdout: "pool": 1, 2024-04-20T17:47:58.406 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_name": ".mgr", 2024-04-20T17:47:58.406 INFO:teuthology.orchestra.run.smithi005.stdout: "create_time": "2024-04-20T17:01:35.909493+0000", 2024-04-20T17:47:58.406 INFO:teuthology.orchestra.run.smithi005.stdout: "flags": 1, 2024-04-20T17:47:58.406 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_names": "hashpspool", 2024-04-20T17:47:58.407 INFO:teuthology.orchestra.run.smithi005.stdout: "type": 1, 2024-04-20T17:47:58.407 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 2, 2024-04-20T17:47:58.407 INFO:teuthology.orchestra.run.smithi005.stdout: "min_size": 2, 2024-04-20T17:47:58.407 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_rule": 0, 2024-04-20T17:47:58.407 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_count": 0, 2024-04-20T17:47:58.407 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_target": 0, 2024-04-20T17:47:58.407 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T17:47:58.407 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T17:47:58.407 INFO:teuthology.orchestra.run.smithi005.stdout: "object_hash": 2, 2024-04-20T17:47:58.407 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_autoscale_mode": "off", 2024-04-20T17:47:58.407 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num": 11, 2024-04-20T17:47:58.407 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num": 11, 2024-04-20T17:47:58.407 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num_target": 11, 2024-04-20T17:47:58.407 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_target": 11, 2024-04-20T17:47:58.408 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_pending": 11, 2024-04-20T17:47:58.408 INFO:teuthology.orchestra.run.smithi005.stdout: "last_pg_merge_meta": { 2024-04-20T17:47:58.408 INFO:teuthology.orchestra.run.smithi005.stdout: "source_pgid": "0.0", 2024-04-20T17:47:58.408 INFO:teuthology.orchestra.run.smithi005.stdout: "ready_epoch": 0, 2024-04-20T17:47:58.408 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_started": 0, 2024-04-20T17:47:58.408 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_clean": 0, 2024-04-20T17:47:58.408 INFO:teuthology.orchestra.run.smithi005.stdout: "source_version": "0'0", 2024-04-20T17:47:58.408 INFO:teuthology.orchestra.run.smithi005.stdout: "target_version": "0'0" 2024-04-20T17:47:58.408 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.408 INFO:teuthology.orchestra.run.smithi005.stdout: "last_change": "838", 2024-04-20T17:47:58.408 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend": "0", 2024-04-20T17:47:58.408 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-20T17:47:58.408 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_preluminous": "818", 2024-04-20T17:47:58.408 INFO:teuthology.orchestra.run.smithi005.stdout: "auid": 0, 2024-04-20T17:47:58.408 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_mode": "selfmanaged", 2024-04-20T17:47:58.409 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_seq": 0, 2024-04-20T17:47:58.409 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_epoch": 0, 2024-04-20T17:47:58.409 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_snaps": [], 2024-04-20T17:47:58.409 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_snaps": "[]", 2024-04-20T17:47:58.409 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_bytes": 0, 2024-04-20T17:47:58.409 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_objects": 0, 2024-04-20T17:47:58.409 INFO:teuthology.orchestra.run.smithi005.stdout: "tiers": [], 2024-04-20T17:47:58.409 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_of": -1, 2024-04-20T17:47:58.409 INFO:teuthology.orchestra.run.smithi005.stdout: "read_tier": -1, 2024-04-20T17:47:58.409 INFO:teuthology.orchestra.run.smithi005.stdout: "write_tier": -1, 2024-04-20T17:47:58.409 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_mode": "none", 2024-04-20T17:47:58.409 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_bytes": 0, 2024-04-20T17:47:58.409 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_objects": 0, 2024-04-20T17:47:58.410 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T17:47:58.410 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T17:47:58.410 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T17:47:58.410 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_flush_age": 0, 2024-04-20T17:47:58.410 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_evict_age": 0, 2024-04-20T17:47:58.410 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_profile": "", 2024-04-20T17:47:58.410 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_params": { 2024-04-20T17:47:58.410 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "none" 2024-04-20T17:47:58.410 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.410 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_period": 0, 2024-04-20T17:47:58.410 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_count": 0, 2024-04-20T17:47:58.410 INFO:teuthology.orchestra.run.smithi005.stdout: "use_gmt_hitset": true, 2024-04-20T17:47:58.410 INFO:teuthology.orchestra.run.smithi005.stdout: "min_read_recency_for_promote": 0, 2024-04-20T17:47:58.411 INFO:teuthology.orchestra.run.smithi005.stdout: "min_write_recency_for_promote": 0, 2024-04-20T17:47:58.411 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T17:47:58.411 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_search_last_n": 0, 2024-04-20T17:47:58.411 INFO:teuthology.orchestra.run.smithi005.stdout: "grade_table": [], 2024-04-20T17:47:58.411 INFO:teuthology.orchestra.run.smithi005.stdout: "stripe_width": 0, 2024-04-20T17:47:58.411 INFO:teuthology.orchestra.run.smithi005.stdout: "expected_num_objects": 0, 2024-04-20T17:47:58.411 INFO:teuthology.orchestra.run.smithi005.stdout: "fast_read": false, 2024-04-20T17:47:58.411 INFO:teuthology.orchestra.run.smithi005.stdout: "options": { 2024-04-20T17:47:58.411 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_max": 32, 2024-04-20T17:47:58.411 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_min": 1 2024-04-20T17:47:58.411 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.411 INFO:teuthology.orchestra.run.smithi005.stdout: "application_metadata": { 2024-04-20T17:47:58.411 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": {} 2024-04-20T17:47:58.411 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.412 INFO:teuthology.orchestra.run.smithi005.stdout: "read_balance": { 2024-04-20T17:47:58.412 INFO:teuthology.orchestra.run.smithi005.stdout: "score_type": "Fair distribution", 2024-04-20T17:47:58.412 INFO:teuthology.orchestra.run.smithi005.stdout: "score_acting": 3, 2024-04-20T17:47:58.412 INFO:teuthology.orchestra.run.smithi005.stdout: "score_stable": 3, 2024-04-20T17:47:58.412 INFO:teuthology.orchestra.run.smithi005.stdout: "optimal_score": 0.9100000262260437, 2024-04-20T17:47:58.412 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_acting": 2.7300000190734863, 2024-04-20T17:47:58.412 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_stable": 2.7300000190734863, 2024-04-20T17:47:58.412 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity_weighted": 0.83594763278961182, 2024-04-20T17:47:58.412 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity": 0.9100000262260437, 2024-04-20T17:47:58.412 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-04-20T17:47:58.412 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.412 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.412 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.413 INFO:teuthology.orchestra.run.smithi005.stdout: "pool": 8, 2024-04-20T17:47:58.413 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_name": "unique_pool_6", 2024-04-20T17:47:58.413 INFO:teuthology.orchestra.run.smithi005.stdout: "create_time": "2024-04-20T17:43:16.604151+0000", 2024-04-20T17:47:58.413 INFO:teuthology.orchestra.run.smithi005.stdout: "flags": 1, 2024-04-20T17:47:58.413 INFO:teuthology.orchestra.run.smithi005.stdout: "flags_names": "hashpspool", 2024-04-20T17:47:58.413 INFO:teuthology.orchestra.run.smithi005.stdout: "type": 1, 2024-04-20T17:47:58.413 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 2, 2024-04-20T17:47:58.413 INFO:teuthology.orchestra.run.smithi005.stdout: "min_size": 2, 2024-04-20T17:47:58.413 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_rule": 0, 2024-04-20T17:47:58.413 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_count": 0, 2024-04-20T17:47:58.413 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_target": 0, 2024-04-20T17:47:58.413 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T17:47:58.413 INFO:teuthology.orchestra.run.smithi005.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T17:47:58.414 INFO:teuthology.orchestra.run.smithi005.stdout: "object_hash": 2, 2024-04-20T17:47:58.414 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_autoscale_mode": "off", 2024-04-20T17:47:58.414 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num": 16, 2024-04-20T17:47:58.414 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num": 16, 2024-04-20T17:47:58.414 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_placement_num_target": 16, 2024-04-20T17:47:58.414 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_target": 16, 2024-04-20T17:47:58.414 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_num_pending": 16, 2024-04-20T17:47:58.414 INFO:teuthology.orchestra.run.smithi005.stdout: "last_pg_merge_meta": { 2024-04-20T17:47:58.414 INFO:teuthology.orchestra.run.smithi005.stdout: "source_pgid": "0.0", 2024-04-20T17:47:58.414 INFO:teuthology.orchestra.run.smithi005.stdout: "ready_epoch": 0, 2024-04-20T17:47:58.414 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_started": 0, 2024-04-20T17:47:58.414 INFO:teuthology.orchestra.run.smithi005.stdout: "last_epoch_clean": 0, 2024-04-20T17:47:58.414 INFO:teuthology.orchestra.run.smithi005.stdout: "source_version": "0'0", 2024-04-20T17:47:58.414 INFO:teuthology.orchestra.run.smithi005.stdout: "target_version": "0'0" 2024-04-20T17:47:58.414 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.415 INFO:teuthology.orchestra.run.smithi005.stdout: "last_change": "782", 2024-04-20T17:47:58.415 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend": "0", 2024-04-20T17:47:58.415 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-20T17:47:58.415 INFO:teuthology.orchestra.run.smithi005.stdout: "last_force_op_resend_preluminous": "0", 2024-04-20T17:47:58.415 INFO:teuthology.orchestra.run.smithi005.stdout: "auid": 0, 2024-04-20T17:47:58.415 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_mode": "selfmanaged", 2024-04-20T17:47:58.415 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_seq": 0, 2024-04-20T17:47:58.415 INFO:teuthology.orchestra.run.smithi005.stdout: "snap_epoch": 0, 2024-04-20T17:47:58.415 INFO:teuthology.orchestra.run.smithi005.stdout: "pool_snaps": [], 2024-04-20T17:47:58.415 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_snaps": "[]", 2024-04-20T17:47:58.415 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_bytes": 0, 2024-04-20T17:47:58.415 INFO:teuthology.orchestra.run.smithi005.stdout: "quota_max_objects": 0, 2024-04-20T17:47:58.415 INFO:teuthology.orchestra.run.smithi005.stdout: "tiers": [], 2024-04-20T17:47:58.415 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_of": -1, 2024-04-20T17:47:58.415 INFO:teuthology.orchestra.run.smithi005.stdout: "read_tier": -1, 2024-04-20T17:47:58.416 INFO:teuthology.orchestra.run.smithi005.stdout: "write_tier": -1, 2024-04-20T17:47:58.416 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_mode": "none", 2024-04-20T17:47:58.416 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_bytes": 0, 2024-04-20T17:47:58.416 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_objects": 0, 2024-04-20T17:47:58.416 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T17:47:58.416 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T17:47:58.416 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T17:47:58.416 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_flush_age": 0, 2024-04-20T17:47:58.416 INFO:teuthology.orchestra.run.smithi005.stdout: "cache_min_evict_age": 0, 2024-04-20T17:47:58.416 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_profile": "", 2024-04-20T17:47:58.416 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_params": { 2024-04-20T17:47:58.416 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "none" 2024-04-20T17:47:58.416 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.416 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_period": 0, 2024-04-20T17:47:58.416 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_count": 0, 2024-04-20T17:47:58.417 INFO:teuthology.orchestra.run.smithi005.stdout: "use_gmt_hitset": true, 2024-04-20T17:47:58.417 INFO:teuthology.orchestra.run.smithi005.stdout: "min_read_recency_for_promote": 0, 2024-04-20T17:47:58.417 INFO:teuthology.orchestra.run.smithi005.stdout: "min_write_recency_for_promote": 0, 2024-04-20T17:47:58.417 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T17:47:58.417 INFO:teuthology.orchestra.run.smithi005.stdout: "hit_set_search_last_n": 0, 2024-04-20T17:47:58.417 INFO:teuthology.orchestra.run.smithi005.stdout: "grade_table": [], 2024-04-20T17:47:58.417 INFO:teuthology.orchestra.run.smithi005.stdout: "stripe_width": 0, 2024-04-20T17:47:58.417 INFO:teuthology.orchestra.run.smithi005.stdout: "expected_num_objects": 0, 2024-04-20T17:47:58.417 INFO:teuthology.orchestra.run.smithi005.stdout: "fast_read": false, 2024-04-20T17:47:58.417 INFO:teuthology.orchestra.run.smithi005.stdout: "options": {}, 2024-04-20T17:47:58.417 INFO:teuthology.orchestra.run.smithi005.stdout: "application_metadata": { 2024-04-20T17:47:58.417 INFO:teuthology.orchestra.run.smithi005.stdout: "rados": {} 2024-04-20T17:47:58.417 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.417 INFO:teuthology.orchestra.run.smithi005.stdout: "read_balance": { 2024-04-20T17:47:58.418 INFO:teuthology.orchestra.run.smithi005.stdout: "score_type": "Fair distribution", 2024-04-20T17:47:58.418 INFO:teuthology.orchestra.run.smithi005.stdout: "score_acting": 3.440000057220459, 2024-04-20T17:47:58.418 INFO:teuthology.orchestra.run.smithi005.stdout: "score_stable": 3.440000057220459, 2024-04-20T17:47:58.418 INFO:teuthology.orchestra.run.smithi005.stdout: "optimal_score": 1, 2024-04-20T17:47:58.418 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_acting": 3.440000057220459, 2024-04-20T17:47:58.418 INFO:teuthology.orchestra.run.smithi005.stdout: "raw_score_stable": 3.440000057220459, 2024-04-20T17:47:58.418 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity_weighted": 0.83594763278961182, 2024-04-20T17:47:58.418 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity": 0.9100000262260437, 2024-04-20T17:47:58.418 INFO:teuthology.orchestra.run.smithi005.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-04-20T17:47:58.418 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.418 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.418 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:47:58.418 INFO:teuthology.orchestra.run.smithi005.stdout: "osds": [ 2024-04-20T17:47:58.419 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.419 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 0, 2024-04-20T17:47:58.419 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b", 2024-04-20T17:47:58.419 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:47:58.419 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:47:58.419 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:47:58.419 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:47:58.419 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:47:58.419 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:47:58.419 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 8, 2024-04-20T17:47:58.419 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 845, 2024-04-20T17:47:58.419 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:47:58.419 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:47:58.420 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:47:58.420 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.420 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.420 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.420 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6801", 2024-04-20T17:47:58.420 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:47:58.420 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.420 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.420 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.420 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:47:58.420 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.420 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.420 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.420 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6802", 2024-04-20T17:47:58.421 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:47:58.421 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.421 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.421 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.421 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:47:58.421 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.421 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.421 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.421 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6804", 2024-04-20T17:47:58.421 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:47:58.421 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.421 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.421 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.421 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:47:58.422 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.422 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.422 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.422 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6803", 2024-04-20T17:47:58.422 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3901484705 2024-04-20T17:47:58.422 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.422 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.422 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.422 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6801/3901484705", 2024-04-20T17:47:58.422 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6802/3901484705", 2024-04-20T17:47:58.422 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6804/3901484705", 2024-04-20T17:47:58.422 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6803/3901484705", 2024-04-20T17:47:58.423 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:47:58.423 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:47:58.423 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:47:58.423 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.423 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.423 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.423 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 1, 2024-04-20T17:47:58.423 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "aed0c8eb-0e81-4e85-ab8f-a60420060656", 2024-04-20T17:47:58.423 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:47:58.423 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:47:58.423 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:47:58.423 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 0.0313720703125, 2024-04-20T17:47:58.423 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 13, 2024-04-20T17:47:58.423 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 613, 2024-04-20T17:47:58.424 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 659, 2024-04-20T17:47:58.424 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 539, 2024-04-20T17:47:58.424 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 614, 2024-04-20T17:47:58.424 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:47:58.424 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:47:58.424 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.424 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.424 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.424 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6805", 2024-04-20T17:47:58.424 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2415652792 2024-04-20T17:47:58.424 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.424 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.424 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.425 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:47:58.425 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.425 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.425 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.425 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6806", 2024-04-20T17:47:58.425 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2415652792 2024-04-20T17:47:58.425 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.425 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.425 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.425 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:47:58.425 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.425 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.425 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.425 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6808", 2024-04-20T17:47:58.426 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2415652792 2024-04-20T17:47:58.426 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.426 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.426 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.426 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:47:58.426 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.426 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.426 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.426 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6807", 2024-04-20T17:47:58.426 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2415652792 2024-04-20T17:47:58.426 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.426 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.426 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.426 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6805/2415652792", 2024-04-20T17:47:58.427 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6806/2415652792", 2024-04-20T17:47:58.427 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6808/2415652792", 2024-04-20T17:47:58.427 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6807/2415652792", 2024-04-20T17:47:58.427 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:47:58.427 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:47:58.427 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:47:58.427 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.427 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.427 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.427 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 2, 2024-04-20T17:47:58.427 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "f733a6a2-19c2-4c7a-af9c-04e282062bb7", 2024-04-20T17:47:58.427 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:47:58.427 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:47:58.427 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:47:58.428 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:47:58.428 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 20, 2024-04-20T17:47:58.428 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 169, 2024-04-20T17:47:58.428 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 202, 2024-04-20T17:47:58.428 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 845, 2024-04-20T17:47:58.428 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 170, 2024-04-20T17:47:58.428 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:47:58.428 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:47:58.428 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.428 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.428 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.428 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6809", 2024-04-20T17:47:58.428 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2120275501 2024-04-20T17:47:58.429 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.429 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.429 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.429 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:47:58.429 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.429 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.429 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.429 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6810", 2024-04-20T17:47:58.429 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2120275501 2024-04-20T17:47:58.429 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.429 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.429 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.429 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:47:58.429 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.430 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.430 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.430 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6812", 2024-04-20T17:47:58.430 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2120275501 2024-04-20T17:47:58.430 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.430 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.430 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.430 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:47:58.430 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.430 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.430 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.430 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6811", 2024-04-20T17:47:58.430 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2120275501 2024-04-20T17:47:58.431 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.431 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.431 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.431 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6809/2120275501", 2024-04-20T17:47:58.431 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6810/2120275501", 2024-04-20T17:47:58.431 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6812/2120275501", 2024-04-20T17:47:58.431 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6811/2120275501", 2024-04-20T17:47:58.431 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:47:58.431 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:47:58.431 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:47:58.431 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.431 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.431 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.431 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 3, 2024-04-20T17:47:58.431 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "8c92f1df-e298-4606-88ae-b8a8e79628a1", 2024-04-20T17:47:58.432 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:47:58.432 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:47:58.432 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:47:58.432 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:47:58.432 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 25, 2024-04-20T17:47:58.432 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 538, 2024-04-20T17:47:58.432 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 561, 2024-04-20T17:47:58.432 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 777, 2024-04-20T17:47:58.432 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 539, 2024-04-20T17:47:58.432 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:47:58.432 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:47:58.432 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.432 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.432 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.433 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6813", 2024-04-20T17:47:58.433 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2272824046 2024-04-20T17:47:58.433 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.433 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.433 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.433 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:47:58.433 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.433 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.433 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.433 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6814", 2024-04-20T17:47:58.433 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2272824046 2024-04-20T17:47:58.433 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.433 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.433 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.433 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:47:58.434 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.434 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.434 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.434 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6816", 2024-04-20T17:47:58.434 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2272824046 2024-04-20T17:47:58.434 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.434 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.434 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.434 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:47:58.434 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.434 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.434 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.434 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6815", 2024-04-20T17:47:58.435 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2272824046 2024-04-20T17:47:58.435 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.435 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.435 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.435 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6813/2272824046", 2024-04-20T17:47:58.435 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.5:6814/2272824046", 2024-04-20T17:47:58.435 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.5:6816/2272824046", 2024-04-20T17:47:58.435 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.5:6815/2272824046", 2024-04-20T17:47:58.435 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:47:58.435 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:47:58.435 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:47:58.435 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.435 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.435 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.435 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 4, 2024-04-20T17:47:58.436 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "46df16f1-9607-4d7b-9be8-9d4236e160d1", 2024-04-20T17:47:58.436 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:47:58.436 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:47:58.436 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:47:58.436 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:47:58.436 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:47:58.436 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:47:58.436 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 31, 2024-04-20T17:47:58.436 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 694, 2024-04-20T17:47:58.436 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:47:58.436 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:47:58.436 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:47:58.436 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.436 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.436 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.437 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6800", 2024-04-20T17:47:58.437 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:47:58.437 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.437 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.437 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.437 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:47:58.437 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.437 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.437 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.437 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6801", 2024-04-20T17:47:58.437 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:47:58.437 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.437 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.437 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.438 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:47:58.438 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.438 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.438 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.438 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6803", 2024-04-20T17:47:58.438 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:47:58.438 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.438 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.438 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.438 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:47:58.438 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.438 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.438 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.438 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6802", 2024-04-20T17:47:58.439 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 1839524469 2024-04-20T17:47:58.439 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.439 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.439 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.439 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6800/1839524469", 2024-04-20T17:47:58.439 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6801/1839524469", 2024-04-20T17:47:58.439 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6803/1839524469", 2024-04-20T17:47:58.439 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6802/1839524469", 2024-04-20T17:47:58.439 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:47:58.439 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:47:58.439 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:47:58.439 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.439 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.439 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.439 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 5, 2024-04-20T17:47:58.440 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "6603231d-1e5c-4575-bfaa-2eeca2c12215", 2024-04-20T17:47:58.440 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:47:58.440 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:47:58.440 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:47:58.440 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 0, 2024-04-20T17:47:58.440 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:47:58.440 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:47:58.440 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 37, 2024-04-20T17:47:58.440 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 619, 2024-04-20T17:47:58.440 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:47:58.440 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:47:58.440 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:47:58.440 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.440 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.441 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.441 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6804", 2024-04-20T17:47:58.441 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:47:58.441 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.441 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.441 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.441 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:47:58.441 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.441 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.441 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.441 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6805", 2024-04-20T17:47:58.441 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:47:58.441 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.441 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.441 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.442 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:47:58.442 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.442 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.442 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.442 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6807", 2024-04-20T17:47:58.442 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:47:58.442 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.442 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.442 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.442 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:47:58.442 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.442 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.442 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.442 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6806", 2024-04-20T17:47:58.443 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2450315908 2024-04-20T17:47:58.443 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.443 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.443 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.443 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6804/2450315908", 2024-04-20T17:47:58.443 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6805/2450315908", 2024-04-20T17:47:58.443 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6807/2450315908", 2024-04-20T17:47:58.443 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6806/2450315908", 2024-04-20T17:47:58.443 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:47:58.443 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:47:58.443 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:47:58.443 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.443 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.444 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.444 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 6, 2024-04-20T17:47:58.444 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "783a55a6-f176-42be-8c8b-af8481d08dc9", 2024-04-20T17:47:58.444 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:47:58.444 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:47:58.444 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:47:58.444 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:47:58.444 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 48, 2024-04-20T17:47:58.444 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 503, 2024-04-20T17:47:58.444 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 535, 2024-04-20T17:47:58.444 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 845, 2024-04-20T17:47:58.444 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 504, 2024-04-20T17:47:58.444 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:47:58.444 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:47:58.444 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.445 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.445 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.445 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6808", 2024-04-20T17:47:58.445 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2439382182 2024-04-20T17:47:58.445 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.445 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.445 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.445 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:47:58.445 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.445 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.445 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.445 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6809", 2024-04-20T17:47:58.445 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2439382182 2024-04-20T17:47:58.445 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.446 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.446 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.446 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:47:58.446 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.446 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.446 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.446 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6811", 2024-04-20T17:47:58.446 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2439382182 2024-04-20T17:47:58.446 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.446 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.446 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.446 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:47:58.446 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.446 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.447 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.447 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6810", 2024-04-20T17:47:58.447 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2439382182 2024-04-20T17:47:58.447 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.447 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.447 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.447 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6808/2439382182", 2024-04-20T17:47:58.447 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6809/2439382182", 2024-04-20T17:47:58.447 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6811/2439382182", 2024-04-20T17:47:58.447 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6810/2439382182", 2024-04-20T17:47:58.447 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:47:58.447 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:47:58.447 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:47:58.447 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.448 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.448 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.448 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 7, 2024-04-20T17:47:58.448 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4", 2024-04-20T17:47:58.448 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:47:58.448 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:47:58.448 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:47:58.448 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:47:58.448 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 398, 2024-04-20T17:47:58.448 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 721, 2024-04-20T17:47:58.448 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 799, 2024-04-20T17:47:58.448 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 846, 2024-04-20T17:47:58.448 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 722, 2024-04-20T17:47:58.448 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:47:58.448 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:47:58.448 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.449 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.449 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.449 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6812", 2024-04-20T17:47:58.449 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2531699401 2024-04-20T17:47:58.449 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.449 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.449 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.449 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:47:58.449 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.449 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.449 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.449 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6813", 2024-04-20T17:47:58.449 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2531699401 2024-04-20T17:47:58.449 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.449 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.450 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.450 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:47:58.450 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.450 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.450 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.450 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6815", 2024-04-20T17:47:58.450 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2531699401 2024-04-20T17:47:58.450 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.450 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.450 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.450 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:47:58.450 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.450 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.450 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.451 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.106:6814", 2024-04-20T17:47:58.451 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2531699401 2024-04-20T17:47:58.451 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.451 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.451 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.451 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.106:6812/2531699401", 2024-04-20T17:47:58.451 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.106:6813/2531699401", 2024-04-20T17:47:58.451 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.106:6815/2531699401", 2024-04-20T17:47:58.451 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.106:6814/2531699401", 2024-04-20T17:47:58.451 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:47:58.451 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:47:58.451 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:47:58.451 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.451 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.452 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.452 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 8, 2024-04-20T17:47:58.452 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "1cb232ad-1f59-4a0a-a215-e67419ad6476", 2024-04-20T17:47:58.452 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:47:58.452 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:47:58.452 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:47:58.452 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:47:58.452 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:47:58.452 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:47:58.452 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 67, 2024-04-20T17:47:58.452 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 777, 2024-04-20T17:47:58.452 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:47:58.452 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:47:58.452 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:47:58.452 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.453 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.453 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.453 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6800", 2024-04-20T17:47:58.453 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:47:58.453 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.453 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.453 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.453 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:47:58.453 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.453 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.453 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.453 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6801", 2024-04-20T17:47:58.453 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:47:58.453 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.453 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.454 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.454 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:47:58.454 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.454 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.454 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.454 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6803", 2024-04-20T17:47:58.454 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:47:58.454 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.454 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.454 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.454 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:47:58.454 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.454 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.454 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.455 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6802", 2024-04-20T17:47:58.455 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 4123058942 2024-04-20T17:47:58.455 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.455 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.455 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.455 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6800/4123058942", 2024-04-20T17:47:58.455 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6801/4123058942", 2024-04-20T17:47:58.455 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6803/4123058942", 2024-04-20T17:47:58.455 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6802/4123058942", 2024-04-20T17:47:58.455 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:47:58.455 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:47:58.455 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:47:58.455 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.456 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.456 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.456 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 9, 2024-04-20T17:47:58.456 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "b6c17147-b3dc-475b-bb20-f9f712fb99e8", 2024-04-20T17:47:58.456 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:47:58.456 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:47:58.456 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:47:58.456 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:47:58.456 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:47:58.456 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:47:58.456 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 73, 2024-04-20T17:47:58.456 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 823, 2024-04-20T17:47:58.457 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:47:58.457 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:47:58.457 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:47:58.457 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.457 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.457 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.457 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6804", 2024-04-20T17:47:58.457 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:47:58.457 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.457 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.457 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.457 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:47:58.457 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.458 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.458 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.458 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6805", 2024-04-20T17:47:58.458 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:47:58.458 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.458 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.458 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.458 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:47:58.458 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.458 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.458 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.458 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6807", 2024-04-20T17:47:58.458 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:47:58.458 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.459 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.459 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.459 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:47:58.459 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.459 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.459 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.459 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6806", 2024-04-20T17:47:58.459 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 2877589636 2024-04-20T17:47:58.459 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.459 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.459 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.459 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6804/2877589636", 2024-04-20T17:47:58.459 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6805/2877589636", 2024-04-20T17:47:58.459 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6807/2877589636", 2024-04-20T17:47:58.460 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6806/2877589636", 2024-04-20T17:47:58.460 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:47:58.460 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:47:58.460 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:47:58.460 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.460 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.460 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.460 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 10, 2024-04-20T17:47:58.460 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "1adcdf55-f002-4a39-91e7-378e5fae849c", 2024-04-20T17:47:58.460 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:47:58.460 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:47:58.460 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:47:58.460 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:47:58.460 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:47:58.461 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:47:58.461 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 84, 2024-04-20T17:47:58.461 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 823, 2024-04-20T17:47:58.461 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:47:58.461 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:47:58.461 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:47:58.461 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.461 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.461 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.461 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6808", 2024-04-20T17:47:58.461 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:47:58.461 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.461 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.461 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.462 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:47:58.462 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.462 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.462 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.462 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6809", 2024-04-20T17:47:58.462 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:47:58.462 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.462 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.462 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.462 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:47:58.462 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.462 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.462 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.462 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6811", 2024-04-20T17:47:58.463 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:47:58.463 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.463 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.463 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.463 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:47:58.463 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.463 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.463 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.463 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6810", 2024-04-20T17:47:58.463 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 181490621 2024-04-20T17:47:58.463 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.463 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.463 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.464 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6808/181490621", 2024-04-20T17:47:58.464 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6809/181490621", 2024-04-20T17:47:58.464 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6811/181490621", 2024-04-20T17:47:58.464 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6810/181490621", 2024-04-20T17:47:58.464 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:47:58.464 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:47:58.464 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:47:58.464 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.464 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.464 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.464 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 11, 2024-04-20T17:47:58.464 INFO:teuthology.orchestra.run.smithi005.stdout: "uuid": "737c5aec-694a-4968-819d-fccf22661f4f", 2024-04-20T17:47:58.464 INFO:teuthology.orchestra.run.smithi005.stdout: "up": 1, 2024-04-20T17:47:58.464 INFO:teuthology.orchestra.run.smithi005.stdout: "in": 1, 2024-04-20T17:47:58.465 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 1, 2024-04-20T17:47:58.465 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_affinity": 1, 2024-04-20T17:47:58.465 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_begin": 0, 2024-04-20T17:47:58.465 INFO:teuthology.orchestra.run.smithi005.stdout: "last_clean_end": 0, 2024-04-20T17:47:58.465 INFO:teuthology.orchestra.run.smithi005.stdout: "up_from": 92, 2024-04-20T17:47:58.465 INFO:teuthology.orchestra.run.smithi005.stdout: "up_thru": 855, 2024-04-20T17:47:58.465 INFO:teuthology.orchestra.run.smithi005.stdout: "down_at": 0, 2024-04-20T17:47:58.465 INFO:teuthology.orchestra.run.smithi005.stdout: "lost_at": 0, 2024-04-20T17:47:58.465 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-04-20T17:47:58.465 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.465 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.465 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.465 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6812", 2024-04-20T17:47:58.465 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:47:58.466 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.466 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.466 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.466 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addrs": { 2024-04-20T17:47:58.466 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.466 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.466 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.466 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6813", 2024-04-20T17:47:58.466 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:47:58.466 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.466 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.466 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.466 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addrs": { 2024-04-20T17:47:58.466 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.467 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.467 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.467 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6815", 2024-04-20T17:47:58.467 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:47:58.467 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.467 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.467 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.467 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addrs": { 2024-04-20T17:47:58.467 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-04-20T17:47:58.467 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.467 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-04-20T17:47:58.467 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.171:6814", 2024-04-20T17:47:58.467 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 3373585701 2024-04-20T17:47:58.468 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.468 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.468 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.468 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.171:6812/3373585701", 2024-04-20T17:47:58.468 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "172.21.15.171:6813/3373585701", 2024-04-20T17:47:58.468 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_back_addr": "172.21.15.171:6815/3373585701", 2024-04-20T17:47:58.468 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_front_addr": "172.21.15.171:6814/3373585701", 2024-04-20T17:47:58.468 INFO:teuthology.orchestra.run.smithi005.stdout: "state": [ 2024-04-20T17:47:58.468 INFO:teuthology.orchestra.run.smithi005.stdout: "exists", 2024-04-20T17:47:58.468 INFO:teuthology.orchestra.run.smithi005.stdout: "up" 2024-04-20T17:47:58.468 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:47:58.468 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.468 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:47:58.468 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_xinfo": [ 2024-04-20T17:47:58.468 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.469 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 0, 2024-04-20T17:47:58.469 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:47:58.469 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:47:58.469 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:47:58.469 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:47:58.469 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:47:58.469 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:47:17.386403+0000", 2024-04-20T17:47:58.469 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:47:58.469 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.469 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.469 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 1, 2024-04-20T17:47:58.469 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:47:58.469 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:47:58.469 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:47:58.469 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:47:58.470 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:47:58.470 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:47:14.265060+0000", 2024-04-20T17:47:58.470 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 613 2024-04-20T17:47:58.470 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.470 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.470 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 2, 2024-04-20T17:47:58.470 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:47:58.470 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:47:58.470 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:47:58.470 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:47:58.470 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:47:58.470 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:46:37.512873+0000", 2024-04-20T17:47:58.470 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 169 2024-04-20T17:47:58.471 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.471 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.471 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 3, 2024-04-20T17:47:58.471 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:47:58.471 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:47:58.471 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:47:58.471 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:47:58.471 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:47:58.471 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:47:40.170187+0000", 2024-04-20T17:47:58.471 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 538 2024-04-20T17:47:58.471 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.471 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.471 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 4, 2024-04-20T17:47:58.471 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:47:58.471 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:47:58.471 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:47:58.472 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:47:58.472 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:47:58.472 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:46:49.692242+0000", 2024-04-20T17:47:58.472 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:47:58.472 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.472 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.472 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 5, 2024-04-20T17:47:58.472 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:47:58.472 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:47:58.472 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:47:58.472 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:47:58.472 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:47:58.472 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:47:53.012607+0000", 2024-04-20T17:47:58.472 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:47:58.472 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.473 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.473 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 6, 2024-04-20T17:47:58.473 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:47:58.473 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:47:58.473 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:47:58.473 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:47:58.473 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:47:58.473 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:46:51.604651+0000", 2024-04-20T17:47:58.473 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 503 2024-04-20T17:47:58.473 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.473 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.473 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 7, 2024-04-20T17:47:58.473 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:47:58.473 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:47:58.474 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:47:58.474 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:47:58.474 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:47:58.474 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:46:52.336856+0000", 2024-04-20T17:47:58.474 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 721 2024-04-20T17:47:58.474 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.474 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.474 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 8, 2024-04-20T17:47:58.474 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:47:58.474 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:47:58.474 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:47:58.474 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:47:58.474 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:47:58.474 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:47:46.206599+0000", 2024-04-20T17:47:58.475 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:47:58.475 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.475 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.475 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 9, 2024-04-20T17:47:58.475 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:47:58.475 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:47:58.475 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:47:58.475 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:47:58.475 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:47:58.475 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:46:36.662481+0000", 2024-04-20T17:47:58.475 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:47:58.475 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.475 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.476 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 10, 2024-04-20T17:47:58.476 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:47:58.476 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:47:58.476 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:47:58.476 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:47:58.476 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:47:58.476 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:47:44.603072+0000", 2024-04-20T17:47:58.476 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:47:58.476 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.476 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:47:58.476 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": 11, 2024-04-20T17:47:58.476 INFO:teuthology.orchestra.run.smithi005.stdout: "down_stamp": "0.000000", 2024-04-20T17:47:58.476 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_probability": 0, 2024-04-20T17:47:58.476 INFO:teuthology.orchestra.run.smithi005.stdout: "laggy_interval": 0, 2024-04-20T17:47:58.477 INFO:teuthology.orchestra.run.smithi005.stdout: "features": 4540701547738038271, 2024-04-20T17:47:58.477 INFO:teuthology.orchestra.run.smithi005.stdout: "old_weight": 0, 2024-04-20T17:47:58.477 INFO:teuthology.orchestra.run.smithi005.stdout: "last_purged_snaps_scrub": "2024-04-20T17:47:39.658754+0000", 2024-04-20T17:47:58.477 INFO:teuthology.orchestra.run.smithi005.stdout: "dead_epoch": 0 2024-04-20T17:47:58.477 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.477 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:47:58.477 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_upmap": [], 2024-04-20T17:47:58.477 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_upmap_items": [], 2024-04-20T17:47:58.477 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_upmap_primaries": [], 2024-04-20T17:47:58.477 INFO:teuthology.orchestra.run.smithi005.stdout: "pg_temp": [], 2024-04-20T17:47:58.477 INFO:teuthology.orchestra.run.smithi005.stdout: "primary_temp": [], 2024-04-20T17:47:58.477 INFO:teuthology.orchestra.run.smithi005.stdout: "blocklist": { 2024-04-20T17:47:58.477 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/3881726425": "2024-04-21T16:59:25.808337+0000", 2024-04-20T17:47:58.477 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/338649532": "2024-04-21T16:59:25.808337+0000", 2024-04-20T17:47:58.478 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/1227843664": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:47:58.478 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:6800/1876064631": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:47:58.478 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/160689449": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:47:58.478 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:6800/3521715286": "2024-04-21T16:59:25.808337+0000", 2024-04-20T17:47:58.478 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/3841663462": "2024-04-21T16:58:55.392162+0000", 2024-04-20T17:47:58.478 INFO:teuthology.orchestra.run.smithi005.stdout: "172.21.15.5:0/687690919": "2024-04-21T16:59:25.808337+0000" 2024-04-20T17:47:58.478 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.478 INFO:teuthology.orchestra.run.smithi005.stdout: "range_blocklist": {}, 2024-04-20T17:47:58.478 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_profiles": { 2024-04-20T17:47:58.478 INFO:teuthology.orchestra.run.smithi005.stdout: "default": { 2024-04-20T17:47:58.478 INFO:teuthology.orchestra.run.smithi005.stdout: "crush-failure-domain": "osd", 2024-04-20T17:47:58.478 INFO:teuthology.orchestra.run.smithi005.stdout: "k": "2", 2024-04-20T17:47:58.478 INFO:teuthology.orchestra.run.smithi005.stdout: "m": "1", 2024-04-20T17:47:58.478 INFO:teuthology.orchestra.run.smithi005.stdout: "plugin": "jerasure", 2024-04-20T17:47:58.478 INFO:teuthology.orchestra.run.smithi005.stdout: "technique": "reed_sol_van" 2024-04-20T17:47:58.479 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.479 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:47:58.479 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_snaps_queue": [], 2024-04-20T17:47:58.479 INFO:teuthology.orchestra.run.smithi005.stdout: "new_removed_snaps": [], 2024-04-20T17:47:58.479 INFO:teuthology.orchestra.run.smithi005.stdout: "new_purged_snaps": [], 2024-04-20T17:47:58.479 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_node_flags": {}, 2024-04-20T17:47:58.479 INFO:teuthology.orchestra.run.smithi005.stdout: "device_class_flags": {}, 2024-04-20T17:47:58.479 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": { 2024-04-20T17:47:58.479 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode_enabled": false, 2024-04-20T17:47:58.479 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_bucket_count": 0, 2024-04-20T17:47:58.479 INFO:teuthology.orchestra.run.smithi005.stdout: "degraded_stretch_mode": 0, 2024-04-20T17:47:58.479 INFO:teuthology.orchestra.run.smithi005.stdout: "recovering_stretch_mode": 0, 2024-04-20T17:47:58.479 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode_bucket": 0 2024-04-20T17:47:58.479 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:47:58.480 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:47:58.724 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:47:58.818 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:47:58.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:58.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:47:58.876 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:58 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2878186463' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-20T17:47:58.923 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:47:58.969 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:47:59.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:58 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2878186463' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-20T17:47:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:58 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2878186463' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-20T17:47:59.487 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:47:59.487 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:47:59.939 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:47:59.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:47:59.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:48:00.048 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:48:00.075 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:59 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:48:00.076 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:59 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:48:00.076 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:59 smithi106 ceph-mon[44423]: pgmap v2320: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering, 23 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 4399/56504 objects misplaced (7.785%); 10 MiB/s, 161 objects/s recovering 2024-04-20T17:48:00.076 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:59 smithi106 ceph-mon[44423]: osdmap e875: 12 total, 12 up, 12 in 2024-04-20T17:48:00.076 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:47:59 smithi106 ceph-mon[44423]: 8.e scrub starts 2024-04-20T17:48:00.079 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:00.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:59 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:48:00.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:59 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:48:00.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:59 smithi171 ceph-mon[44189]: pgmap v2320: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering, 23 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 4399/56504 objects misplaced (7.785%); 10 MiB/s, 161 objects/s recovering 2024-04-20T17:48:00.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:59 smithi171 ceph-mon[44189]: osdmap e875: 12 total, 12 up, 12 in 2024-04-20T17:48:00.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:47:59 smithi171 ceph-mon[44189]: 8.e scrub starts 2024-04-20T17:48:00.087 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:59 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:48:00.087 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:59 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:48:00.087 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:59 smithi005 ceph-mon[39562]: pgmap v2320: 27 pgs: 1 active+clean+scrubbing, 3 active+recovering, 23 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 4399/56504 objects misplaced (7.785%); 10 MiB/s, 161 objects/s recovering 2024-04-20T17:48:00.087 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:59 smithi005 ceph-mon[39562]: osdmap e875: 12 total, 12 up, 12 in 2024-04-20T17:48:00.087 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:47:59 smithi005 ceph-mon[39562]: 8.e scrub starts 2024-04-20T17:48:00.204 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:48:00.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:00.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:48:00.233 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-20T17:48:00.311 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:48:00.351 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:00.538 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:48:00.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:00.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:48:00.578 INFO:tasks.thrashosds.thrasher:j is {'epoch': 874, 'fsid': '13c6e76c-ff37-11ee-bc93-c7b262605968', 'created': '2024-04-20T16:58:25.272557+0000', 'modified': '2024-04-20T17:47:56.684948+0000', 'last_up_change': '2024-04-20T17:44:19.738797+0000', 'last_in_change': '2024-04-20T17:46:36.769424+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 33, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-20T17:01:35.909493+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '838', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '818', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 0.9100000262260437, 'raw_score_acting': 2.7300000190734863, 'raw_score_stable': 2.7300000190734863, 'primary_affinity_weighted': 0.8359476327896118, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 8, 'pool_name': 'unique_pool_6', 'create_time': '2024-04-20T17:43:16.604151+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '782', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.440000057220459, 'score_stable': 3.440000057220459, 'optimal_score': 1, 'raw_score_acting': 3.440000057220459, 'raw_score_stable': 3.440000057220459, 'primary_affinity_weighted': 0.8359476327896118, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.8399999737739563}}], 'osds': [{'osd': 0, 'uuid': 'ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 845, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6801', 'nonce': 3901484705}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6802', 'nonce': 3901484705}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6804', 'nonce': 3901484705}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6803', 'nonce': 3901484705}]}, 'public_addr': '172.21.15.5:6801/3901484705', 'cluster_addr': '172.21.15.5:6802/3901484705', 'heartbeat_back_addr': '172.21.15.5:6804/3901484705', 'heartbeat_front_addr': '172.21.15.5:6803/3901484705', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'aed0c8eb-0e81-4e85-ab8f-a60420060656', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.0313720703125, 'last_clean_begin': 13, 'last_clean_end': 613, 'up_from': 659, 'up_thru': 539, 'down_at': 614, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6805', 'nonce': 2415652792}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6806', 'nonce': 2415652792}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6808', 'nonce': 2415652792}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6807', 'nonce': 2415652792}]}, 'public_addr': '172.21.15.5:6805/2415652792', 'cluster_addr': '172.21.15.5:6806/2415652792', 'heartbeat_back_addr': '172.21.15.5:6808/2415652792', 'heartbeat_front_addr': '172.21.15.5:6807/2415652792', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f733a6a2-19c2-4c7a-af9c-04e282062bb7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 20, 'last_clean_end': 169, 'up_from': 202, 'up_thru': 845, 'down_at': 170, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6809', 'nonce': 2120275501}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6810', 'nonce': 2120275501}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6812', 'nonce': 2120275501}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6811', 'nonce': 2120275501}]}, 'public_addr': '172.21.15.5:6809/2120275501', 'cluster_addr': '172.21.15.5:6810/2120275501', 'heartbeat_back_addr': '172.21.15.5:6812/2120275501', 'heartbeat_front_addr': '172.21.15.5:6811/2120275501', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '8c92f1df-e298-4606-88ae-b8a8e79628a1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 25, 'last_clean_end': 538, 'up_from': 561, 'up_thru': 777, 'down_at': 539, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6813', 'nonce': 2272824046}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6814', 'nonce': 2272824046}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6816', 'nonce': 2272824046}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.5:6815', 'nonce': 2272824046}]}, 'public_addr': '172.21.15.5:6813/2272824046', 'cluster_addr': '172.21.15.5:6814/2272824046', 'heartbeat_back_addr': '172.21.15.5:6816/2272824046', 'heartbeat_front_addr': '172.21.15.5:6815/2272824046', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '46df16f1-9607-4d7b-9be8-9d4236e160d1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 694, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6800', 'nonce': 1839524469}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6801', 'nonce': 1839524469}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6803', 'nonce': 1839524469}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6802', 'nonce': 1839524469}]}, 'public_addr': '172.21.15.106:6800/1839524469', 'cluster_addr': '172.21.15.106:6801/1839524469', 'heartbeat_back_addr': '172.21.15.106:6803/1839524469', 'heartbeat_front_addr': '172.21.15.106:6802/1839524469', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6603231d-1e5c-4575-bfaa-2eeca2c12215', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 619, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6804', 'nonce': 2450315908}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6805', 'nonce': 2450315908}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6807', 'nonce': 2450315908}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6806', 'nonce': 2450315908}]}, 'public_addr': '172.21.15.106:6804/2450315908', 'cluster_addr': '172.21.15.106:6805/2450315908', 'heartbeat_back_addr': '172.21.15.106:6807/2450315908', 'heartbeat_front_addr': '172.21.15.106:6806/2450315908', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '783a55a6-f176-42be-8c8b-af8481d08dc9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 48, 'last_clean_end': 503, 'up_from': 535, 'up_thru': 845, 'down_at': 504, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6808', 'nonce': 2439382182}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6809', 'nonce': 2439382182}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6811', 'nonce': 2439382182}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6810', 'nonce': 2439382182}]}, 'public_addr': '172.21.15.106:6808/2439382182', 'cluster_addr': '172.21.15.106:6809/2439382182', 'heartbeat_back_addr': '172.21.15.106:6811/2439382182', 'heartbeat_front_addr': '172.21.15.106:6810/2439382182', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 398, 'last_clean_end': 721, 'up_from': 799, 'up_thru': 846, 'down_at': 722, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6812', 'nonce': 2531699401}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6813', 'nonce': 2531699401}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6815', 'nonce': 2531699401}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.106:6814', 'nonce': 2531699401}]}, 'public_addr': '172.21.15.106:6812/2531699401', 'cluster_addr': '172.21.15.106:6813/2531699401', 'heartbeat_back_addr': '172.21.15.106:6815/2531699401', 'heartbeat_front_addr': '172.21.15.106:6814/2531699401', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '1cb232ad-1f59-4a0a-a215-e67419ad6476', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 67, 'up_thru': 777, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6800', 'nonce': 4123058942}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6801', 'nonce': 4123058942}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6803', 'nonce': 4123058942}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6802', 'nonce': 4123058942}]}, 'public_addr': '172.21.15.171:6800/4123058942', 'cluster_addr': '172.21.15.171:6801/4123058942', 'heartbeat_back_addr': '172.21.15.171:6803/4123058942', 'heartbeat_front_addr': '172.21.15.171:6802/4123058942', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'b6c17147-b3dc-475b-bb20-f9f712fb99e8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 73, 'up_thru': 823, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6804', 'nonce': 2877589636}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6805', 'nonce': 2877589636}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6807', 'nonce': 2877589636}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6806', 'nonce': 2877589636}]}, 'public_addr': '172.21.15.171:6804/2877589636', 'cluster_addr': '172.21.15.171:6805/2877589636', 'heartbeat_back_addr': '172.21.15.171:6807/2877589636', 'heartbeat_front_addr': '172.21.15.171:6806/2877589636', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '1adcdf55-f002-4a39-91e7-378e5fae849c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 84, 'up_thru': 823, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6808', 'nonce': 181490621}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6809', 'nonce': 181490621}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6811', 'nonce': 181490621}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6810', 'nonce': 181490621}]}, 'public_addr': '172.21.15.171:6808/181490621', 'cluster_addr': '172.21.15.171:6809/181490621', 'heartbeat_back_addr': '172.21.15.171:6811/181490621', 'heartbeat_front_addr': '172.21.15.171:6810/181490621', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '737c5aec-694a-4968-819d-fccf22661f4f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 92, 'up_thru': 855, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6812', 'nonce': 3373585701}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6813', 'nonce': 3373585701}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6815', 'nonce': 3373585701}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.171:6814', 'nonce': 3373585701}]}, 'public_addr': '172.21.15.171:6812/3373585701', 'cluster_addr': '172.21.15.171:6813/3373585701', 'heartbeat_back_addr': '172.21.15.171:6815/3373585701', 'heartbeat_front_addr': '172.21.15.171:6814/3373585701', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:47:17.386403+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:47:14.265060+0000', 'dead_epoch': 613}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:46:37.512873+0000', 'dead_epoch': 169}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:47:40.170187+0000', 'dead_epoch': 538}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:46:49.692242+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:47:53.012607+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:46:51.604651+0000', 'dead_epoch': 503}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:46:52.336856+0000', 'dead_epoch': 721}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:47:46.206599+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:46:36.662481+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:47:44.603072+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T17:47:39.658754+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.5:0/3881726425': '2024-04-21T16:59:25.808337+0000', '172.21.15.5:0/338649532': '2024-04-21T16:59:25.808337+0000', '172.21.15.5:0/1227843664': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:6800/1876064631': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:0/160689449': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:6800/3521715286': '2024-04-21T16:59:25.808337+0000', '172.21.15.5:0/3841663462': '2024-04-21T16:58:55.392162+0000', '172.21.15.5:0/687690919': '2024-04-21T16:59:25.808337+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-20T17:48:00.579 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-04-20T17:48:00.579 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:48:00.650 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:48:00.681 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:00.814 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:48:00.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:00.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:48:00.925 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:48:00.959 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:00.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:00 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:48:00.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:00 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:48:00.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:00 smithi005 ceph-mon[39562]: 8.e scrub ok 2024-04-20T17:48:00.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:00 smithi005 ceph-mon[39562]: osdmap e876: 12 total, 12 up, 12 in 2024-04-20T17:48:00.982 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:48:00.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:00.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:48:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:00 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:48:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:00 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:48:01.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:00 smithi171 ceph-mon[44189]: 8.e scrub ok 2024-04-20T17:48:01.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:00 smithi171 ceph-mon[44189]: osdmap e876: 12 total, 12 up, 12 in 2024-04-20T17:48:01.087 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:48:01.127 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:01.168 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:00 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:48:01.168 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:00 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:48:01.168 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:00 smithi106 ceph-mon[44423]: 8.e scrub ok 2024-04-20T17:48:01.168 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:00 smithi106 ceph-mon[44423]: osdmap e876: 12 total, 12 up, 12 in 2024-04-20T17:48:01.186 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:48:01.187 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:48:01.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:01.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:48:01.301 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:48:01.347 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:01.482 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:48:01.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:01.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:48:01.591 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:48:01.620 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:01.786 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:01 smithi005 ceph-mon[39562]: pgmap v2323: 27 pgs: 3 active+recovering, 24 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 866 op/s; 4399/52604 objects misplaced (8.362%); 8.9 MiB/s, 142 objects/s recovering 2024-04-20T17:48:01.956 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:01 smithi106 ceph-mon[44423]: pgmap v2323: 27 pgs: 3 active+recovering, 24 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 866 op/s; 4399/52604 objects misplaced (8.362%); 8.9 MiB/s, 142 objects/s recovering 2024-04-20T17:48:02.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:01 smithi171 ceph-mon[44189]: pgmap v2323: 27 pgs: 3 active+recovering, 24 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 866 op/s; 4399/52604 objects misplaced (8.362%); 8.9 MiB/s, 142 objects/s recovering 2024-04-20T17:48:02.472 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:48:02.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:02.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:48:02.579 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:48:02.633 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:02.691 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:48:02.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:02.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:48:02.799 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:48:02.804 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:48:02.861 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:02.900 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:48:02.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:02.910 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:48:02.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:02 smithi005 ceph-mon[39562]: 1.1 scrub starts 2024-04-20T17:48:02.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:02 smithi005 ceph-mon[39562]: 1.1 scrub ok 2024-04-20T17:48:02.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:02 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3996601326' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:48:02.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:02 smithi005 ceph-mon[39562]: from='client.27971 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:48:03.011 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:48:03.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:02 smithi106 ceph-mon[44423]: 1.1 scrub starts 2024-04-20T17:48:03.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:02 smithi106 ceph-mon[44423]: 1.1 scrub ok 2024-04-20T17:48:03.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:02 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3996601326' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:48:03.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:02 smithi106 ceph-mon[44423]: from='client.27971 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:48:03.034 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:48:03.034 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:48:03.070 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:02 smithi171 ceph-mon[44189]: 1.1 scrub starts 2024-04-20T17:48:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:02 smithi171 ceph-mon[44189]: 1.1 scrub ok 2024-04-20T17:48:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:02 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3996601326' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:48:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:02 smithi171 ceph-mon[44189]: from='client.27971 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:48:03.104 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:48:03.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:03.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:48:03.209 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:48:03.244 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:03.264 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:48:03.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:03.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:48:03.368 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:48:03.415 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:03.450 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:48:03.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:03.455 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:48:03.556 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:48:03.609 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:03.725 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:48:03.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:03.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:48:03.780 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-20T17:48:03.832 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:48:03.874 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:03.911 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:48:03.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:03.922 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:48:03.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:03 smithi005 ceph-mon[39562]: pgmap v2324: 27 pgs: 3 active+recovering, 24 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 872 op/s; 3483/47628 objects misplaced (7.313%); 13 MiB/s, 201 objects/s recovering 2024-04-20T17:48:03.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:03 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:48:03.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:03 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:48:03.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:03 smithi005 ceph-mon[39562]: from='client.27971 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:48:03.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:03 smithi005 ceph-mon[39562]: osdmap e877: 12 total, 12 up, 12 in 2024-04-20T17:48:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:03 smithi106 ceph-mon[44423]: pgmap v2324: 27 pgs: 3 active+recovering, 24 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 872 op/s; 3483/47628 objects misplaced (7.313%); 13 MiB/s, 201 objects/s recovering 2024-04-20T17:48:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:03 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:48:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:03 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:48:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:03 smithi106 ceph-mon[44423]: from='client.27971 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:48:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:03 smithi106 ceph-mon[44423]: osdmap e877: 12 total, 12 up, 12 in 2024-04-20T17:48:04.022 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:48:04.056 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:48:04.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:03 smithi171 ceph-mon[44189]: pgmap v2324: 27 pgs: 3 active+recovering, 24 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 872 op/s; 3483/47628 objects misplaced (7.313%); 13 MiB/s, 201 objects/s recovering 2024-04-20T17:48:04.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:03 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:48:04.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:03 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:48:04.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:03 smithi171 ceph-mon[44189]: from='client.27971 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:48:04.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:03 smithi171 ceph-mon[44189]: osdmap e877: 12 total, 12 up, 12 in 2024-04-20T17:48:04.079 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:04.090 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:48:04.120 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:48:04.133 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:48:04.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:04.140 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:48:04.156 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:48:04.174 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:48:04.186 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:48:04.217 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:48:04.241 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:48:04.245 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:48:04.270 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:04.275 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:48:04.291 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:48:04.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:04.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:48:04.307 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:48:04.338 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:48:04.374 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:48:04.395 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:48:04.402 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:48:04.404 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:48:04.404 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:48:04.404 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:48:04.404 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:48:04.404 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:48:04.404 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:48:04.404 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:48:04.404 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:48:04.404 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:48:04.405 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:48:04.405 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:48:04.405 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:48:04.444 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:04.470 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:48:04.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:04.474 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:48:04.575 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:48:04.601 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:04.626 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:48:04.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:04.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:48:04.732 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:48:04.762 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:04.783 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:48:04.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:04.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:48:04.887 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:48:04.911 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:04 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1921113366' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:48:04.911 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:04 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:04.911 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:04 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:04.911 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:04 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:04.913 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:04.938 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:48:04.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:04.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:48:05.042 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:48:05.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:04 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1921113366' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:48:05.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:04 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:05.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:04 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:05.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:04 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:05.096 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:04 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1921113366' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:48:05.096 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:04 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:05.096 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:04 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:05.096 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:04 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:05.098 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:05.580 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 8, 9, 10, 11, 1, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 4, 5, 8, 9, 10, 11, 2, 6, 3, 1, 7] 2024-04-20T17:48:05.580 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 0 1 2024-04-20T17:48:05.635 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:48:05.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:05 smithi005 ceph-mon[39562]: pgmap v2326: 27 pgs: 3 active+recovering, 24 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 930 op/s; 3483/45330 objects misplaced (7.684%); 9.5 MiB/s, 152 objects/s recovering 2024-04-20T17:48:06.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:06.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:06.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:06.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:06.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:06.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:06.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:06.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:06.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:06.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:05 smithi171 ceph-mon[44189]: pgmap v2326: 27 pgs: 3 active+recovering, 24 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 930 op/s; 3483/45330 objects misplaced (7.684%); 9.5 MiB/s, 152 objects/s recovering 2024-04-20T17:48:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:06.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:06.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:06.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:05 smithi106 ceph-mon[44423]: pgmap v2326: 27 pgs: 3 active+recovering, 24 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 930 op/s; 3483/45330 objects misplaced (7.684%); 9.5 MiB/s, 152 objects/s recovering 2024-04-20T17:48:06.356 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:48:06.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:06.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:48:06.464 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:48:06.502 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:06.587 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:48:06.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:06.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:48:06.696 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:48:06.759 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:06.790 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:48:06.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:06.796 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:48:06.897 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:48:06.968 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:06.999 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:48:07.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:07.003 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:48:07.005 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:48:07.011 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:48:07.104 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:48:07.131 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:07.150 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:48:07.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:07.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:48:07.254 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:48:07.292 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:07.815 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:07 smithi005 ceph-mon[39562]: pgmap v2327: 27 pgs: 3 active+recovering, 24 active+clean; 1.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 2908/40188 objects misplaced (7.236%); 12 MiB/s, 188 objects/s recovering 2024-04-20T17:48:07.815 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:07 smithi005 ceph-mon[39562]: osdmap e878: 12 total, 12 up, 12 in 2024-04-20T17:48:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:07 smithi106 ceph-mon[44423]: pgmap v2327: 27 pgs: 3 active+recovering, 24 active+clean; 1.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 2908/40188 objects misplaced (7.236%); 12 MiB/s, 188 objects/s recovering 2024-04-20T17:48:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:07 smithi106 ceph-mon[44423]: osdmap e878: 12 total, 12 up, 12 in 2024-04-20T17:48:08.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:07 smithi171 ceph-mon[44189]: pgmap v2327: 27 pgs: 3 active+recovering, 24 active+clean; 1.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 2908/40188 objects misplaced (7.236%); 12 MiB/s, 188 objects/s recovering 2024-04-20T17:48:08.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:07 smithi171 ceph-mon[44189]: osdmap e878: 12 total, 12 up, 12 in 2024-04-20T17:48:08.390 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:48:08.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:08 smithi005 ceph-mon[39562]: 1.6 scrub starts 2024-04-20T17:48:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:08 smithi106 ceph-mon[44423]: 1.6 scrub starts 2024-04-20T17:48:09.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:08 smithi171 ceph-mon[44189]: 1.6 scrub starts 2024-04-20T17:48:09.466 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:48:09.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:09.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:48:09.571 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:48:09.644 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:09.686 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:48:09.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:09.691 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:48:09.761 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:09 smithi005 ceph-mon[39562]: 1.6 scrub ok 2024-04-20T17:48:09.761 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:09 smithi005 ceph-mon[39562]: pgmap v2329: 27 pgs: 3 active+recovering, 24 active+clean; 1.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 877 op/s; 2549/38562 objects misplaced (6.610%); 14 MiB/s, 231 objects/s recovering 2024-04-20T17:48:09.792 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:48:09.854 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:09.892 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:48:09.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:09.898 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:48:09.899 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:09 smithi171 ceph-mon[44189]: 1.6 scrub ok 2024-04-20T17:48:09.899 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:09 smithi171 ceph-mon[44189]: pgmap v2329: 27 pgs: 3 active+recovering, 24 active+clean; 1.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 877 op/s; 2549/38562 objects misplaced (6.610%); 14 MiB/s, 231 objects/s recovering 2024-04-20T17:48:09.999 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:48:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:09 smithi106 ceph-mon[44423]: 1.6 scrub ok 2024-04-20T17:48:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:09 smithi106 ceph-mon[44423]: pgmap v2329: 27 pgs: 3 active+recovering, 24 active+clean; 1.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 877 op/s; 2549/38562 objects misplaced (6.610%); 14 MiB/s, 231 objects/s recovering 2024-04-20T17:48:10.037 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:10.165 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:48:10.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:10.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:48:10.274 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:48:10.314 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:10.910 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:48:10.910 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:48:10.910 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-04-20T17:48:10.910 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:48:11.329 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:48:11.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:11.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:48:11.439 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:48:11.489 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:11.522 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:48:11.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:11.525 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:48:11.626 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:48:11.662 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:11 smithi171 ceph-mon[44189]: pgmap v2330: 27 pgs: 3 active+recovering, 24 active+clean; 1.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 822 op/s; 2549/34470 objects misplaced (7.395%); 7.3 MiB/s, 116 objects/s recovering 2024-04-20T17:48:11.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:11 smithi171 ceph-mon[44189]: 8.0 scrub starts 2024-04-20T17:48:11.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:11 smithi005 ceph-mon[39562]: pgmap v2330: 27 pgs: 3 active+recovering, 24 active+clean; 1.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 822 op/s; 2549/34470 objects misplaced (7.395%); 7.3 MiB/s, 116 objects/s recovering 2024-04-20T17:48:11.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:11 smithi005 ceph-mon[39562]: 8.0 scrub starts 2024-04-20T17:48:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:11 smithi106 ceph-mon[44423]: pgmap v2330: 27 pgs: 3 active+recovering, 24 active+clean; 1.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 822 op/s; 2549/34470 objects misplaced (7.395%); 7.3 MiB/s, 116 objects/s recovering 2024-04-20T17:48:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:11 smithi106 ceph-mon[44423]: 8.0 scrub starts 2024-04-20T17:48:12.728 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.0 to 1 (10000) 2024-04-20T17:48:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:12 smithi106 ceph-mon[44423]: 8.0 scrub ok 2024-04-20T17:48:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:12 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2207258053' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:48:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:12 smithi106 ceph-mon[44423]: from='client.27980 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:48:13.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:12 smithi171 ceph-mon[44189]: 8.0 scrub ok 2024-04-20T17:48:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:12 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2207258053' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:48:13.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:12 smithi171 ceph-mon[44189]: from='client.27980 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:48:13.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:12 smithi005 ceph-mon[39562]: 8.0 scrub ok 2024-04-20T17:48:13.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:12 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2207258053' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:48:13.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:12 smithi005 ceph-mon[39562]: from='client.27980 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:48:13.509 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-20T17:48:13.677 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:48:13.723 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:13 smithi005 ceph-mon[39562]: pgmap v2331: 27 pgs: 3 active+recovering, 24 active+clean; 909 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 940 op/s; 1543/29076 objects misplaced (5.307%); 12 MiB/s, 196 objects/s recovering 2024-04-20T17:48:13.723 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:13 smithi005 ceph-mon[39562]: from='client.27980 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-20T17:48:13.723 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:13 smithi005 ceph-mon[39562]: osdmap e879: 12 total, 12 up, 12 in 2024-04-20T17:48:13.723 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:13 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:48:13.724 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:13 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:48:13.724 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:13 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1688689370' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:48:13.724 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:13 smithi005 ceph-mon[39562]: from='client.27986 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:48:13.746 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:48:14.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:13 smithi106 ceph-mon[44423]: pgmap v2331: 27 pgs: 3 active+recovering, 24 active+clean; 909 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 940 op/s; 1543/29076 objects misplaced (5.307%); 12 MiB/s, 196 objects/s recovering 2024-04-20T17:48:14.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:13 smithi106 ceph-mon[44423]: from='client.27980 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-20T17:48:14.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:13 smithi106 ceph-mon[44423]: osdmap e879: 12 total, 12 up, 12 in 2024-04-20T17:48:14.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:13 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:48:14.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:13 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:48:14.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:13 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1688689370' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:48:14.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:13 smithi106 ceph-mon[44423]: from='client.27986 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:48:14.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:13 smithi171 ceph-mon[44189]: pgmap v2331: 27 pgs: 3 active+recovering, 24 active+clean; 909 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 940 op/s; 1543/29076 objects misplaced (5.307%); 12 MiB/s, 196 objects/s recovering 2024-04-20T17:48:14.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:13 smithi171 ceph-mon[44189]: from='client.27980 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-20T17:48:14.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:13 smithi171 ceph-mon[44189]: osdmap e879: 12 total, 12 up, 12 in 2024-04-20T17:48:14.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:13 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:48:14.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:13 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:48:14.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:13 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1688689370' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:48:14.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:13 smithi171 ceph-mon[44189]: from='client.27986 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:48:14.562 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 2 1 2024-04-20T17:48:14.830 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:14 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:48:14.830 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:14 smithi005 ceph-mon[39562]: from='client.27986 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:48:14.830 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:14 smithi005 ceph-mon[39562]: osdmap e880: 12 total, 12 up, 12 in 2024-04-20T17:48:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:14 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:48:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:14 smithi106 ceph-mon[44423]: from='client.27986 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:48:15.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:14 smithi106 ceph-mon[44423]: osdmap e880: 12 total, 12 up, 12 in 2024-04-20T17:48:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:14 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:48:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:14 smithi171 ceph-mon[44189]: from='client.27986 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:48:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:14 smithi171 ceph-mon[44189]: osdmap e880: 12 total, 12 up, 12 in 2024-04-20T17:48:15.640 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:48:16.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:15 smithi106 ceph-mon[44423]: pgmap v2334: 27 pgs: 3 active+recovering, 24 active+clean; 833 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 850 op/s; 1543/26646 objects misplaced (5.791%); 11 MiB/s, 171 objects/s recovering 2024-04-20T17:48:16.025 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:15 smithi005 ceph-mon[39562]: pgmap v2334: 27 pgs: 3 active+recovering, 24 active+clean; 833 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 850 op/s; 1543/26646 objects misplaced (5.791%); 11 MiB/s, 171 objects/s recovering 2024-04-20T17:48:16.025 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:48:16.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:16.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:48:16.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:15 smithi171 ceph-mon[44189]: pgmap v2334: 27 pgs: 3 active+recovering, 24 active+clean; 833 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 850 op/s; 1543/26646 objects misplaced (5.791%); 11 MiB/s, 171 objects/s recovering 2024-04-20T17:48:16.131 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:48:16.191 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:16.222 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:48:16.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:16.227 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:48:16.328 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:48:16.356 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:16.378 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:48:16.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:16.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:48:16.482 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:48:16.543 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:16.579 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:48:16.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:16.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:48:16.685 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:48:16.724 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:16.881 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:48:16.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:16.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:48:16.984 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:48:17.012 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:17.036 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:48:17.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:17.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:48:17.141 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:48:17.170 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:17.191 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:48:17.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:17.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:48:17.210 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:48:17.296 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:48:17.349 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:17.375 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:48:17.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:17.380 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:48:17.481 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:48:17.507 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:17.527 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:48:17.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:17.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:48:17.632 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:48:17.694 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:17.718 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:48:17.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:17.723 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:48:17.824 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:48:17.851 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:17.869 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:48:17.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:17.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:48:17.928 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:17 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:48:17.928 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:17 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:48:17.928 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:17 smithi005 ceph-mon[39562]: pgmap v2335: 27 pgs: 3 active+recovering, 24 active+clean; 662 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 846/21192 objects misplaced (3.992%); 13 MiB/s, 212 objects/s recovering 2024-04-20T17:48:17.928 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:17 smithi005 ceph-mon[39562]: osdmap e881: 12 total, 12 up, 12 in 2024-04-20T17:48:17.973 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:48:18.025 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:17 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:48:18.025 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:17 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:48:18.026 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:17 smithi171 ceph-mon[44189]: pgmap v2335: 27 pgs: 3 active+recovering, 24 active+clean; 662 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 846/21192 objects misplaced (3.992%); 13 MiB/s, 212 objects/s recovering 2024-04-20T17:48:18.026 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:17 smithi171 ceph-mon[44189]: osdmap e881: 12 total, 12 up, 12 in 2024-04-20T17:48:18.027 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:18.059 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:48:18.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:18.069 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:48:18.170 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:48:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:17 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:48:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:17 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:48:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:17 smithi106 ceph-mon[44423]: pgmap v2335: 27 pgs: 3 active+recovering, 24 active+clean; 662 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 846/21192 objects misplaced (3.992%); 13 MiB/s, 212 objects/s recovering 2024-04-20T17:48:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:17 smithi106 ceph-mon[44423]: osdmap e881: 12 total, 12 up, 12 in 2024-04-20T17:48:18.262 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:18.306 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:48:18.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:18.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:48:18.392 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:48:18.392 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:48:18.392 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:48:18.392 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:48:18.392 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:48:18.410 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:48:18.438 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:18.461 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:48:18.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:18.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:48:18.565 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:48:18.605 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:18.643 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:48:18.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:18.654 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:48:18.755 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:48:18.784 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:18.805 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:48:18.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:18.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:48:18.910 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:48:18.949 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:18 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1355112519' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:48:18.954 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:18.982 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:48:18.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:18.990 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:48:19.091 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:48:19.116 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:18 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1355112519' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:48:19.118 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:19.137 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:48:19.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:19.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:48:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:18 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1355112519' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:48:19.225 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:48:19.225 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:48:19.225 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:48:19.225 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:48:19.225 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:48:19.225 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:48:19.225 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:48:19.226 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:48:19.226 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:48:19.226 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:48:19.226 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:48:19.226 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:48:19.226 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:48:19.226 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:48:19.226 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:48:19.226 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:48:19.226 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:48:19.226 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:48:19.226 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:48:19.227 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:48:19.227 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:48:19.227 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:48:19.227 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:48:19.227 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:48:19.241 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:48:19.276 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:19.908 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:48:19.914 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:48:19.938 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:48:19.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:19.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:48:20.044 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:48:20.070 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.070 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.070 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.070 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.070 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.070 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.070 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.070 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.070 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.070 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.070 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:19 smithi106 ceph-mon[44423]: pgmap v2337: 27 pgs: 2 active+recovering, 25 active+clean; 604 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 813 op/s; 516/19312 objects misplaced (2.672%); 11 MiB/s, 171 objects/s recovering 2024-04-20T17:48:20.070 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.071 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.072 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:19 smithi171 ceph-mon[44189]: pgmap v2337: 27 pgs: 2 active+recovering, 25 active+clean; 604 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 813 op/s; 516/19312 objects misplaced (2.672%); 11 MiB/s, 171 objects/s recovering 2024-04-20T17:48:20.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.091 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:48:20.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:20.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:48:20.195 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:48:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:19 smithi005 ceph-mon[39562]: pgmap v2337: 27 pgs: 2 active+recovering, 25 active+clean; 604 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 813 op/s; 516/19312 objects misplaced (2.672%); 11 MiB/s, 171 objects/s recovering 2024-04-20T17:48:20.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:20.242 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:20.277 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:48:20.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:20.281 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:48:20.382 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:48:20.442 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:20.472 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:48:20.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:20.480 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:48:20.581 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:48:20.608 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:20.626 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:48:20.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:20.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:48:20.730 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:48:20.770 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:20.906 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:48:20.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:20.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:48:21.010 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:48:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:20 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:48:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:20 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:48:21.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:20 smithi106 ceph-mon[44423]: osdmap e882: 12 total, 12 up, 12 in 2024-04-20T17:48:21.050 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:21.083 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:48:21.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:20 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:48:21.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:20 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:48:21.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:20 smithi171 ceph-mon[44189]: osdmap e882: 12 total, 12 up, 12 in 2024-04-20T17:48:21.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:21.088 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:48:21.189 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:48:21.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:20 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:48:21.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:20 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:48:21.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:20 smithi005 ceph-mon[39562]: osdmap e882: 12 total, 12 up, 12 in 2024-04-20T17:48:21.224 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:21.315 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:48:21.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:21.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:48:21.439 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:48:21.468 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:21.489 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:48:21.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:21.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:48:21.593 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:48:21.632 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:21.764 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:48:21.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:21.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:48:21.800 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-20T17:48:21.871 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:48:21.920 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:21 smithi005 ceph-mon[39562]: pgmap v2339: 27 pgs: 2 active+recovering, 25 active+clean; 470 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 516/15018 objects misplaced (3.436%); 9.3 MiB/s, 148 objects/s recovering 2024-04-20T17:48:21.923 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:21 smithi171 ceph-mon[44189]: pgmap v2339: 27 pgs: 2 active+recovering, 25 active+clean; 470 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 516/15018 objects misplaced (3.436%); 9.3 MiB/s, 148 objects/s recovering 2024-04-20T17:48:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:21 smithi106 ceph-mon[44423]: pgmap v2339: 27 pgs: 2 active+recovering, 25 active+clean; 470 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 516/15018 objects misplaced (3.436%); 9.3 MiB/s, 148 objects/s recovering 2024-04-20T17:48:22.286 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:48:22.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:22.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:48:22.391 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:48:22.422 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:22.930 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:48:23.305 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:48:23.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:23.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:48:23.410 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:48:23.448 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:23.993 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:23 smithi005 ceph-mon[39562]: pgmap v2340: 27 pgs: 27 active+clean; 293 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 12 MiB/s, 192 objects/s recovering 2024-04-20T17:48:24.007 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:48:24.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:24.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:48:24.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:23 smithi171 ceph-mon[44189]: pgmap v2340: 27 pgs: 27 active+clean; 293 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 12 MiB/s, 192 objects/s recovering 2024-04-20T17:48:24.115 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:48:24.153 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:24.202 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:48:24.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:24.209 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:48:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:23 smithi106 ceph-mon[44423]: pgmap v2340: 27 pgs: 27 active+clean; 293 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 12 MiB/s, 192 objects/s recovering 2024-04-20T17:48:24.310 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:48:24.338 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:24.363 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:48:24.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:24.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:48:24.468 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:48:24.496 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:24.517 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:48:24.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:24.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:48:24.621 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:48:24.659 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:24.679 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:48:24.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:24.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:48:24.783 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:48:24.831 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:24.882 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:48:24.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:24.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:48:24.907 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:48:24.986 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:48:25.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:24 smithi106 ceph-mon[44423]: pgmap v2341: 27 pgs: 27 active+clean; 213 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 899 op/s; 6.6 MiB/s, 105 objects/s recovering 2024-04-20T17:48:25.012 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:25.033 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:48:25.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:25.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:48:25.074 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Removed 145111 objects 2024-04-20T17:48:25.075 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Clean up completed and total clean up time :206.629 2024-04-20T17:48:25.141 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:48:25.166 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:24 smithi171 ceph-mon[44189]: pgmap v2341: 27 pgs: 27 active+clean; 213 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 899 op/s; 6.6 MiB/s, 105 objects/s recovering 2024-04-20T17:48:25.168 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:25.187 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:48:25.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:25.190 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:48:25.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:24 smithi005 ceph-mon[39562]: pgmap v2341: 27 pgs: 27 active+clean; 213 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 899 op/s; 6.6 MiB/s, 105 objects/s recovering 2024-04-20T17:48:25.291 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:48:25.319 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:25.468 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:48:25.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:25.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:48:25.556 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:48:25.573 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:48:25.603 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:25.903 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.2 to 1 (10000) 2024-04-20T17:48:25.903 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:48:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:25 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/506708203' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-20T17:48:26.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:25 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4166551558' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:48:26.252 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:48:26.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:26.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:48:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:25 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/506708203' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-20T17:48:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:25 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4166551558' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:48:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:25 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/506708203' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-20T17:48:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:25 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4166551558' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:48:26.356 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:48:26.389 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:26.410 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:48:26.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:26.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:48:26.514 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:48:26.544 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:26 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:48:27.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:26 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/506708203' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-20T17:48:27.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:26 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4166551558' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:48:27.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:26 smithi005 ceph-mon[39562]: osdmap e883: 12 total, 12 up, 12 in 2024-04-20T17:48:27.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:26 smithi005 ceph-mon[39562]: pgmap v2343: 27 pgs: 27 active+clean; 49 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 4.0 MiB/s, 64 objects/s recovering 2024-04-20T17:48:27.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:26 smithi005 ceph-mon[39562]: osdmap e884: 12 total, 12 up, 12 in 2024-04-20T17:48:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:26 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:48:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:26 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/506708203' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-20T17:48:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:26 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4166551558' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:48:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:26 smithi106 ceph-mon[44423]: osdmap e883: 12 total, 12 up, 12 in 2024-04-20T17:48:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:26 smithi106 ceph-mon[44423]: pgmap v2343: 27 pgs: 27 active+clean; 49 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 4.0 MiB/s, 64 objects/s recovering 2024-04-20T17:48:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:26 smithi106 ceph-mon[44423]: osdmap e884: 12 total, 12 up, 12 in 2024-04-20T17:48:27.292 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:48:27.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:27.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:48:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:26 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:48:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:26 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/506708203' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-20T17:48:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:26 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4166551558' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:48:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:26 smithi171 ceph-mon[44189]: osdmap e883: 12 total, 12 up, 12 in 2024-04-20T17:48:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:26 smithi171 ceph-mon[44189]: pgmap v2343: 27 pgs: 27 active+clean; 49 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 4.0 MiB/s, 64 objects/s recovering 2024-04-20T17:48:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:26 smithi171 ceph-mon[44189]: osdmap e884: 12 total, 12 up, 12 in 2024-04-20T17:48:27.350 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_6 2024-04-20T17:48:27.350 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pool rm unique_pool_6 unique_pool_6 --yes-i-really-really-mean-it 2024-04-20T17:48:27.396 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:48:27.429 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:27.451 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:48:27.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:27.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:48:27.555 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:48:27.587 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:27.609 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:48:27.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:27.612 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:48:27.713 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:48:27.745 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:27.767 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:48:27.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:27.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:48:27.779 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 3 1 2024-04-20T17:48:27.871 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:48:27.900 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:27.974 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:48:27.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:27.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:48:28.006 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:27 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:48:28.007 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:27 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:48:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:27 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:48:28.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:27 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:48:28.078 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:48:28.111 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:28.133 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:48:28.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:28.136 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:48:28.237 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:48:28.266 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:27 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:48:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:27 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:48:29.117 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:48:29.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:29.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:48:29.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:28 smithi005 ceph-mon[39562]: pgmap v2345: 27 pgs: 27 active+clean; 13 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 913 op/s; 4.0 MiB/s, 64 objects/s recovering 2024-04-20T17:48:29.221 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:48:29.252 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:28 smithi106 ceph-mon[44423]: pgmap v2345: 27 pgs: 27 active+clean; 13 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 913 op/s; 4.0 MiB/s, 64 objects/s recovering 2024-04-20T17:48:29.254 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:29.276 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:48:29.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:29.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:48:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:28 smithi171 ceph-mon[44189]: pgmap v2345: 27 pgs: 27 active+clean; 13 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 913 op/s; 4.0 MiB/s, 64 objects/s recovering 2024-04-20T17:48:29.380 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:48:29.410 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:29.510 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:48:29.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:29.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:48:29.614 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:48:29.646 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:29.668 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:48:29.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:29.672 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:48:29.773 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:48:29.803 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:29.823 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:48:29.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:29.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:48:29.926 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:48:29.953 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:48:29.953 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:48:29.953 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:48:29.953 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:48:29.953 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:48:29.953 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:48:29.954 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:48:29.954 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:48:29.954 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:48:29.954 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:48:29.954 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:48:29.954 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:48:29.955 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:48:29.955 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:48:29.955 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:48:29.955 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:48:29.955 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:48:29.955 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:48:29.955 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:48:29.955 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:48:29.955 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:48:29.956 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:48:29.956 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:48:29.956 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:48:29.956 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:29 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2432088802' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:48:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:30.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:29 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:29 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2432088802' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:48:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:29 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:30.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:29 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2432088802' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:48:30.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:30.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:30.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:30.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:30.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:30.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:30.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:30.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:29 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:30.605 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:48:30.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:30.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:48:30.710 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:48:30.743 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:30.765 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:48:30.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:30.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:48:30.870 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:48:30.899 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:30.920 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:48:30.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:30.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:48:31.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:31.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:31.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:31.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:30 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:31.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:30 smithi106 ceph-mon[44423]: pgmap v2346: 27 pgs: 27 active+clean; 361 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 584 op/s 2024-04-20T17:48:31.023 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:48:31.056 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:31.078 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:48:31.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:31.081 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:48:31.091 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:48:31.120 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:31.120 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:31.120 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:31.120 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:30 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:31.120 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:30 smithi005 ceph-mon[39562]: pgmap v2346: 27 pgs: 27 active+clean; 361 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 584 op/s 2024-04-20T17:48:31.160 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:48:31.161 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:48:31.183 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:48:31.212 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:31.231 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:48:31.231 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:48:31.231 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:48:31.231 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:48:31.231 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.232 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:87138 7.c 7:3ac78cb4:::benchmark_data_smithi171_161871_object87137:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e709)", 2024-04-20T17:48:31.232 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:38:43.842031+0000", 2024-04-20T17:48:31.232 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 587.38694348700005, 2024-04-20T17:48:31.232 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.157860996, 2024-04-20T17:48:31.232 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:48:31.232 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:48:31.232 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:48:31.232 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:48:31.232 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:48:31.232 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:48:31.232 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 87138 2024-04-20T17:48:31.232 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.232 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:48:31.232 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.232 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:48:31.233 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.842031+0000", 2024-04-20T17:48:31.233 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.233 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.233 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.233 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:48:31.233 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.842031+0000", 2024-04-20T17:48:31.233 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.233 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.233 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.233 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:48:31.233 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.842033+0000", 2024-04-20T17:48:31.233 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5990000000000001e-06 2024-04-20T17:48:31.233 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.233 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.234 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:48:31.234 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.842124+0000", 2024-04-20T17:48:31.234 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0934999999999996e-05 2024-04-20T17:48:31.234 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.234 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.234 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:48:31.234 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.842131+0000", 2024-04-20T17:48:31.234 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5340000000000002e-06 2024-04-20T17:48:31.234 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.234 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.234 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:48:31.234 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.842137+0000", 2024-04-20T17:48:31.234 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6559999999999997e-06 2024-04-20T17:48:31.235 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.235 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.235 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:48:31.235 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.842180+0000", 2024-04-20T17:48:31.235 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3111000000000001e-05 2024-04-20T17:48:31.235 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.235 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.235 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:48:31.235 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.842267+0000", 2024-04-20T17:48:31.235 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7250999999999995e-05 2024-04-20T17:48:31.235 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.235 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.235 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 11", 2024-04-20T17:48:31.235 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.842365+0000", 2024-04-20T17:48:31.235 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7938999999999999e-05 2024-04-20T17:48:31.235 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.236 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.236 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:48:31.236 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.848102+0000", 2024-04-20T17:48:31.236 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0057371260000000004 2024-04-20T17:48:31.236 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.236 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.236 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:48:31.236 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.999815+0000", 2024-04-20T17:48:31.236 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.15171239 2024-04-20T17:48:31.236 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.236 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.236 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:48:31.236 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.999857+0000", 2024-04-20T17:48:31.236 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2208e-05 2024-04-20T17:48:31.236 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.237 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.237 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:48:31.237 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.999892+0000", 2024-04-20T17:48:31.237 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5247000000000003e-05 2024-04-20T17:48:31.237 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.237 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:48:31.237 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.237 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.237 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.237 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:87141 7.c 7:38ce718a:::benchmark_data_smithi171_161871_object87140:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e709)", 2024-04-20T17:48:31.237 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:38:43.843661+0000", 2024-04-20T17:48:31.237 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 587.38531423699999, 2024-04-20T17:48:31.237 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.17321147000000001, 2024-04-20T17:48:31.237 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:48:31.238 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:48:31.238 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:48:31.238 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:48:31.238 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:48:31.238 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:48:31.238 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 87141 2024-04-20T17:48:31.238 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.238 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:48:31.238 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.238 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:48:31.238 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.843661+0000", 2024-04-20T17:48:31.238 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.238 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.239 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.239 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:48:31.239 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.843661+0000", 2024-04-20T17:48:31.239 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.239 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.239 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.239 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:48:31.239 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.843662+0000", 2024-04-20T17:48:31.239 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.556e-06 2024-04-20T17:48:31.239 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.239 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.239 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:48:31.239 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.843721+0000", 2024-04-20T17:48:31.239 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8448e-05 2024-04-20T17:48:31.239 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.239 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.240 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:48:31.240 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.843737+0000", 2024-04-20T17:48:31.240 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6184999999999998e-05 2024-04-20T17:48:31.240 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.240 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.240 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:48:31.240 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.843744+0000", 2024-04-20T17:48:31.240 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5140000000000004e-06 2024-04-20T17:48:31.240 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.240 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.240 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:48:31.240 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.845099+0000", 2024-04-20T17:48:31.240 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001354926 2024-04-20T17:48:31.240 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.240 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.241 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:48:31.241 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.845219+0000", 2024-04-20T17:48:31.241 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011979799999999999 2024-04-20T17:48:31.241 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.241 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.241 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 11", 2024-04-20T17:48:31.241 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.845357+0000", 2024-04-20T17:48:31.241 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013789 2024-04-20T17:48:31.241 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.241 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.241 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:48:31.241 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.848603+0000", 2024-04-20T17:48:31.241 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0032464669999999998 2024-04-20T17:48:31.241 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.241 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.242 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:48:31.242 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.016772+0000", 2024-04-20T17:48:31.242 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.168168228 2024-04-20T17:48:31.242 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.242 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.242 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:48:31.242 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.016829+0000", 2024-04-20T17:48:31.242 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7497999999999997e-05 2024-04-20T17:48:31.242 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.242 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.242 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:48:31.242 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.016872+0000", 2024-04-20T17:48:31.242 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2960000000000002e-05 2024-04-20T17:48:31.242 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.243 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:48:31.243 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.243 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.243 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.243 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:87157 7.c 7:32ba4a0c:::benchmark_data_smithi171_161871_object87156:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e709)", 2024-04-20T17:48:31.243 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:38:43.854781+0000", 2024-04-20T17:48:31.243 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 587.37419399700002, 2024-04-20T17:48:31.243 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.162197069, 2024-04-20T17:48:31.243 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:48:31.243 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:48:31.243 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:48:31.243 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:48:31.243 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:48:31.243 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:48:31.243 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 87157 2024-04-20T17:48:31.243 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.244 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:48:31.244 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.244 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:48:31.244 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.854781+0000", 2024-04-20T17:48:31.244 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.244 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.244 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.244 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:48:31.244 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.854781+0000", 2024-04-20T17:48:31.244 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.244 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.244 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.244 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:48:31.244 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.854783+0000", 2024-04-20T17:48:31.244 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.697e-06 2024-04-20T17:48:31.244 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.245 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.245 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:48:31.245 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.854833+0000", 2024-04-20T17:48:31.245 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0590000000000002e-05 2024-04-20T17:48:31.245 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.245 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.245 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:48:31.245 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.854844+0000", 2024-04-20T17:48:31.245 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1025e-05 2024-04-20T17:48:31.245 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.245 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.245 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:48:31.245 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.854850+0000", 2024-04-20T17:48:31.245 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2500000000000003e-06 2024-04-20T17:48:31.245 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.246 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.246 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:48:31.246 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.854905+0000", 2024-04-20T17:48:31.246 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4215999999999997e-05 2024-04-20T17:48:31.246 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.246 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.246 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:48:31.246 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.855018+0000", 2024-04-20T17:48:31.246 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011294900000000001 2024-04-20T17:48:31.246 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.246 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.246 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 11", 2024-04-20T17:48:31.246 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.855112+0000", 2024-04-20T17:48:31.246 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4140000000000003e-05 2024-04-20T17:48:31.246 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.247 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.247 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:48:31.247 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.860376+0000", 2024-04-20T17:48:31.247 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0052640559999999996 2024-04-20T17:48:31.247 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.247 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.247 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:48:31.247 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.016911+0000", 2024-04-20T17:48:31.247 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.15653561399999999 2024-04-20T17:48:31.247 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.247 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.247 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:48:31.247 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.016942+0000", 2024-04-20T17:48:31.247 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0648999999999997e-05 2024-04-20T17:48:31.247 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.248 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.248 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:48:31.248 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.016978+0000", 2024-04-20T17:48:31.248 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5883000000000002e-05 2024-04-20T17:48:31.248 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.248 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:48:31.248 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.248 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.248 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.248 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:87160 7.c 7:39d2f99c:::benchmark_data_smithi171_161871_object87159:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e709)", 2024-04-20T17:48:31.248 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:38:43.855240+0000", 2024-04-20T17:48:31.248 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 587.37373484399996, 2024-04-20T17:48:31.248 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.161860333, 2024-04-20T17:48:31.248 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:48:31.248 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:48:31.249 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:48:31.249 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:48:31.249 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:48:31.249 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:48:31.249 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 87160 2024-04-20T17:48:31.249 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.249 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:48:31.249 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.249 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:48:31.249 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.855240+0000", 2024-04-20T17:48:31.249 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.249 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.249 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.249 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:48:31.249 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.855240+0000", 2024-04-20T17:48:31.250 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.250 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.250 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.250 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:48:31.250 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.855241+0000", 2024-04-20T17:48:31.250 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.37e-06 2024-04-20T17:48:31.250 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.250 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.250 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:48:31.250 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.855344+0000", 2024-04-20T17:48:31.250 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000102588 2024-04-20T17:48:31.250 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.250 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.250 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:48:31.251 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.855356+0000", 2024-04-20T17:48:31.251 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1878e-05 2024-04-20T17:48:31.251 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.251 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.251 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:48:31.251 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.855363+0000", 2024-04-20T17:48:31.251 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5599999999999996e-06 2024-04-20T17:48:31.251 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.251 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.251 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:48:31.251 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.857825+0000", 2024-04-20T17:48:31.251 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0024618130000000002 2024-04-20T17:48:31.251 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.251 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.251 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:48:31.252 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.857952+0000", 2024-04-20T17:48:31.252 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000126687 2024-04-20T17:48:31.252 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.252 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.252 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 11", 2024-04-20T17:48:31.252 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.858097+0000", 2024-04-20T17:48:31.252 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014494900000000001 2024-04-20T17:48:31.252 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.252 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.252 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:48:31.252 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.862559+0000", 2024-04-20T17:48:31.252 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0044618080000000003 2024-04-20T17:48:31.252 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.252 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.252 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:48:31.253 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.017016+0000", 2024-04-20T17:48:31.253 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.15445716800000001 2024-04-20T17:48:31.253 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.253 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.253 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:48:31.253 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.017064+0000", 2024-04-20T17:48:31.253 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.808e-05 2024-04-20T17:48:31.253 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.253 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.253 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:48:31.253 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.017100+0000", 2024-04-20T17:48:31.253 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6431999999999999e-05 2024-04-20T17:48:31.253 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.253 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:48:31.254 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.254 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.254 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.254 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:87165 7.c 7:3335e632:::benchmark_data_smithi171_161871_object87164:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e709)", 2024-04-20T17:48:31.254 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:38:43.858382+0000", 2024-04-20T17:48:31.254 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 587.37059291200001, 2024-04-20T17:48:31.254 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.158889949, 2024-04-20T17:48:31.254 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:48:31.254 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:48:31.254 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:48:31.254 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:48:31.254 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:48:31.254 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:48:31.254 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 87165 2024-04-20T17:48:31.255 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.255 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:48:31.255 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.255 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:48:31.255 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.858382+0000", 2024-04-20T17:48:31.255 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.255 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.255 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.255 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:48:31.255 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.858382+0000", 2024-04-20T17:48:31.255 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.255 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.255 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.255 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:48:31.255 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.858383+0000", 2024-04-20T17:48:31.255 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.336e-06 2024-04-20T17:48:31.255 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.256 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.256 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:48:31.256 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.858439+0000", 2024-04-20T17:48:31.256 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6113999999999998e-05 2024-04-20T17:48:31.256 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.256 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.256 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:48:31.256 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.858451+0000", 2024-04-20T17:48:31.256 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1439e-05 2024-04-20T17:48:31.256 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.256 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.256 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:48:31.256 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.858457+0000", 2024-04-20T17:48:31.256 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9789999999999998e-06 2024-04-20T17:48:31.257 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.257 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.257 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:48:31.257 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.860407+0000", 2024-04-20T17:48:31.257 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0019500489999999999 2024-04-20T17:48:31.257 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.257 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.257 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:48:31.257 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.860490+0000", 2024-04-20T17:48:31.257 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.3089000000000004e-05 2024-04-20T17:48:31.257 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.257 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.257 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 11", 2024-04-20T17:48:31.257 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.860592+0000", 2024-04-20T17:48:31.258 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010162000000000001 2024-04-20T17:48:31.258 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.258 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.258 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:48:31.258 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.865894+0000", 2024-04-20T17:48:31.258 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0053022 2024-04-20T17:48:31.258 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.258 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.258 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:48:31.258 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.017155+0000", 2024-04-20T17:48:31.258 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.15126082299999999 2024-04-20T17:48:31.258 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.258 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.258 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:48:31.259 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.017229+0000", 2024-04-20T17:48:31.259 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4740000000000006e-05 2024-04-20T17:48:31.259 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.259 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.259 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:48:31.259 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.017272+0000", 2024-04-20T17:48:31.259 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2559999999999999e-05 2024-04-20T17:48:31.259 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.259 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:48:31.259 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.259 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.259 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.259 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:87169 7.c 7:349f0b61:::benchmark_data_smithi171_161871_object87168:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e709)", 2024-04-20T17:48:31.259 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:38:43.861700+0000", 2024-04-20T17:48:31.259 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 587.36727511900006, 2024-04-20T17:48:31.259 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.15573498599999999, 2024-04-20T17:48:31.260 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:48:31.260 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:48:31.260 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:48:31.260 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:48:31.260 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:48:31.260 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:48:31.260 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 87169 2024-04-20T17:48:31.260 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.260 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:48:31.260 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.260 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:48:31.260 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.861700+0000", 2024-04-20T17:48:31.260 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.260 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.260 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.260 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:48:31.261 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.861700+0000", 2024-04-20T17:48:31.261 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.261 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.261 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.261 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:48:31.261 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.861702+0000", 2024-04-20T17:48:31.261 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0219999999999999e-06 2024-04-20T17:48:31.261 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.261 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.261 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:48:31.261 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.861754+0000", 2024-04-20T17:48:31.261 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2176999999999999e-05 2024-04-20T17:48:31.261 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.261 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.261 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:48:31.261 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.861767+0000", 2024-04-20T17:48:31.262 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2913e-05 2024-04-20T17:48:31.262 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.262 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.262 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:48:31.262 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.861776+0000", 2024-04-20T17:48:31.262 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7080000000000003e-06 2024-04-20T17:48:31.262 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.262 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.262 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:48:31.262 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.862591+0000", 2024-04-20T17:48:31.262 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00081536800000000002 2024-04-20T17:48:31.262 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.262 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.262 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:48:31.262 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.862711+0000", 2024-04-20T17:48:31.263 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012045900000000001 2024-04-20T17:48:31.263 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.263 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.263 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 11", 2024-04-20T17:48:31.263 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.862797+0000", 2024-04-20T17:48:31.263 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5694999999999999e-05 2024-04-20T17:48:31.263 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.263 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.263 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:48:31.263 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.865905+0000", 2024-04-20T17:48:31.263 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0031081699999999999 2024-04-20T17:48:31.263 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.263 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.264 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:48:31.264 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.017346+0000", 2024-04-20T17:48:31.264 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.15144074799999999 2024-04-20T17:48:31.264 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.264 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.264 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:48:31.264 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.017396+0000", 2024-04-20T17:48:31.264 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0164000000000003e-05 2024-04-20T17:48:31.264 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.264 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.264 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:48:31.264 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.017435+0000", 2024-04-20T17:48:31.264 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8562000000000001e-05 2024-04-20T17:48:31.264 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.264 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:48:31.265 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.265 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.265 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.265 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:87171 7.c 7:32893c60:::benchmark_data_smithi171_161871_object87170:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e709)", 2024-04-20T17:48:31.265 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:38:43.862463+0000", 2024-04-20T17:48:31.265 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 587.36651190600003, 2024-04-20T17:48:31.265 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.155096917, 2024-04-20T17:48:31.265 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:48:31.265 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:48:31.265 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:48:31.265 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:48:31.265 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:48:31.265 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:48:31.265 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 87171 2024-04-20T17:48:31.265 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.265 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:48:31.266 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.266 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:48:31.266 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.862463+0000", 2024-04-20T17:48:31.266 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.266 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.266 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.266 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:48:31.266 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.862463+0000", 2024-04-20T17:48:31.266 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.266 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.266 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.266 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:48:31.266 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.862464+0000", 2024-04-20T17:48:31.266 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.516e-06 2024-04-20T17:48:31.267 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.267 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.267 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:48:31.267 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.862530+0000", 2024-04-20T17:48:31.267 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5980000000000002e-05 2024-04-20T17:48:31.267 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.267 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.267 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:48:31.267 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.862541+0000", 2024-04-20T17:48:31.267 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0900999999999999e-05 2024-04-20T17:48:31.267 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.267 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.267 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:48:31.267 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.862549+0000", 2024-04-20T17:48:31.267 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8590000000000005e-06 2024-04-20T17:48:31.267 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.268 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.268 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:48:31.268 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.864448+0000", 2024-04-20T17:48:31.268 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0018989020000000001 2024-04-20T17:48:31.268 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.268 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.268 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:48:31.268 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.864526+0000", 2024-04-20T17:48:31.268 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7631000000000003e-05 2024-04-20T17:48:31.268 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.268 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.268 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 11", 2024-04-20T17:48:31.268 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.864615+0000", 2024-04-20T17:48:31.268 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.9480000000000004e-05 2024-04-20T17:48:31.269 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.269 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.269 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:48:31.269 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:43.866027+0000", 2024-04-20T17:48:31.269 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001411762 2024-04-20T17:48:31.269 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.269 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.269 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:48:31.269 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.017476+0000", 2024-04-20T17:48:31.269 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.151448894 2024-04-20T17:48:31.269 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.269 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.269 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:48:31.269 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.017523+0000", 2024-04-20T17:48:31.269 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7114000000000003e-05 2024-04-20T17:48:31.269 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.270 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.270 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:48:31.270 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:38:44.017560+0000", 2024-04-20T17:48:31.270 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6878000000000002e-05 2024-04-20T17:48:31.270 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.270 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:48:31.270 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.270 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.270 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.270 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:256774 7.3 7:ce830d86:::benchmark_data_smithi171_161871_object111048:head [delete] snapc 0=[] ondisk+write+known_if_redirected e756)", 2024-04-20T17:48:31.270 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:42:18.833533+0000", 2024-04-20T17:48:31.270 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 372.39544158400003, 2024-04-20T17:48:31.270 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.12193923500000001, 2024-04-20T17:48:31.270 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:48:31.270 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:48:31.271 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:48:31.271 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:48:31.271 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:48:31.271 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:48:31.271 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 256774 2024-04-20T17:48:31.271 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.271 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:48:31.271 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.271 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:48:31.271 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:18.833533+0000", 2024-04-20T17:48:31.271 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.271 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.271 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.271 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:48:31.271 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:18.833533+0000", 2024-04-20T17:48:31.271 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.272 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.272 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.272 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:48:31.272 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:18.833535+0000", 2024-04-20T17:48:31.272 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4050000000000001e-06 2024-04-20T17:48:31.272 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.272 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.272 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:48:31.272 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:18.833539+0000", 2024-04-20T17:48:31.272 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7729999999999999e-06 2024-04-20T17:48:31.272 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.272 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.272 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:48:31.273 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:18.833549+0000", 2024-04-20T17:48:31.273 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6860000000000001e-06 2024-04-20T17:48:31.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.273 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.273 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:48:31.273 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:18.833556+0000", 2024-04-20T17:48:31.273 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1550000000000001e-06 2024-04-20T17:48:31.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.273 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.273 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:48:31.273 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:18.833611+0000", 2024-04-20T17:48:31.273 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4438999999999998e-05 2024-04-20T17:48:31.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.273 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.274 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:48:31.274 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:18.833720+0000", 2024-04-20T17:48:31.274 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000109176 2024-04-20T17:48:31.274 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.274 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.274 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 2", 2024-04-20T17:48:31.274 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:18.833822+0000", 2024-04-20T17:48:31.274 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000102199 2024-04-20T17:48:31.274 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.274 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.274 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:48:31.274 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:18.950013+0000", 2024-04-20T17:48:31.274 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.116190796 2024-04-20T17:48:31.274 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.275 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.275 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:48:31.275 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:18.955384+0000", 2024-04-20T17:48:31.275 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0053714890000000001 2024-04-20T17:48:31.275 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.275 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.275 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:48:31.275 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:18.955429+0000", 2024-04-20T17:48:31.275 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4410999999999999e-05 2024-04-20T17:48:31.275 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.275 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.275 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:48:31.275 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:42:18.955472+0000", 2024-04-20T17:48:31.275 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3705999999999998e-05 2024-04-20T17:48:31.276 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.276 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:48:31.276 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.276 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.276 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.276 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27460.0:254139 8.0 8:01a7eb1f:::benchmark_data_smithi171_175757_object109025:head [delete] snapc 0=[] ondisk+write+known_if_redirected e869)", 2024-04-20T17:48:31.276 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:47:46.415270+0000", 2024-04-20T17:48:31.276 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 44.813704448000003, 2024-04-20T17:48:31.276 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.042165986000000003, 2024-04-20T17:48:31.276 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:48:31.276 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:48:31.276 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:48:31.276 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:48:31.276 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27460", 2024-04-20T17:48:31.276 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:48:31.276 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 254139 2024-04-20T17:48:31.276 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.277 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:48:31.277 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.277 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:48:31.277 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:46.415270+0000", 2024-04-20T17:48:31.277 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.277 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.277 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.277 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:48:31.277 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:46.415270+0000", 2024-04-20T17:48:31.277 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.277 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.277 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.277 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:48:31.277 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:46.415271+0000", 2024-04-20T17:48:31.277 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7199999999999997e-07 2024-04-20T17:48:31.278 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.278 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.278 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:48:31.278 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:46.415275+0000", 2024-04-20T17:48:31.278 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.6380000000000002e-06 2024-04-20T17:48:31.278 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.278 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.278 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:48:31.278 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:46.415291+0000", 2024-04-20T17:48:31.278 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5696999999999999e-05 2024-04-20T17:48:31.278 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.278 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.278 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:48:31.278 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:46.415298+0000", 2024-04-20T17:48:31.278 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3579999999999997e-06 2024-04-20T17:48:31.279 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.279 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.279 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:48:31.279 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:46.425394+0000", 2024-04-20T17:48:31.279 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.01009553 2024-04-20T17:48:31.279 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.279 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.279 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:48:31.279 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:46.425550+0000", 2024-04-20T17:48:31.279 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000156663 2024-04-20T17:48:31.279 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.279 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.279 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:48:31.279 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:46.425700+0000", 2024-04-20T17:48:31.280 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000149369 2024-04-20T17:48:31.280 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.280 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.280 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:48:31.280 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:46.430991+0000", 2024-04-20T17:48:31.280 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0052909680000000001 2024-04-20T17:48:31.280 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.280 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.280 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:48:31.280 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:46.457325+0000", 2024-04-20T17:48:31.280 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.026334059 2024-04-20T17:48:31.280 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.280 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.280 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:48:31.280 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:46.457381+0000", 2024-04-20T17:48:31.280 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6168999999999997e-05 2024-04-20T17:48:31.281 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.281 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.281 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:48:31.281 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:46.457436+0000", 2024-04-20T17:48:31.281 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5563000000000003e-05 2024-04-20T17:48:31.281 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.281 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:48:31.281 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.281 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.281 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.281 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27460.0:271375 8.0 8:071dc12b:::benchmark_data_smithi171_175757_object126261:head [delete] snapc 0=[] ondisk+write+known_if_redirected e877)", 2024-04-20T17:48:31.281 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:48:05.984784+0000", 2024-04-20T17:48:31.281 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 25.244191076, 2024-04-20T17:48:31.281 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.116159524, 2024-04-20T17:48:31.281 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:48:31.282 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:48:31.282 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:48:31.282 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:48:31.282 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27460", 2024-04-20T17:48:31.282 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:48:31.282 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 271375 2024-04-20T17:48:31.282 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.282 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:48:31.282 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.282 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:48:31.282 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:05.984784+0000", 2024-04-20T17:48:31.282 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.282 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.282 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.283 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:48:31.283 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:05.984784+0000", 2024-04-20T17:48:31.283 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.283 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.283 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:48:31.283 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:05.984786+0000", 2024-04-20T17:48:31.283 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.481e-06 2024-04-20T17:48:31.283 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.283 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:48:31.283 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:05.984796+0000", 2024-04-20T17:48:31.283 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0139999999999999e-05 2024-04-20T17:48:31.283 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.284 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:48:31.284 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:05.984811+0000", 2024-04-20T17:48:31.284 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4188999999999999e-05 2024-04-20T17:48:31.284 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.284 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.284 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:48:31.284 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:05.984818+0000", 2024-04-20T17:48:31.284 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4229999999999999e-06 2024-04-20T17:48:31.284 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.284 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.284 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:48:31.284 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:05.984875+0000", 2024-04-20T17:48:31.284 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7386000000000002e-05 2024-04-20T17:48:31.284 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.284 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.284 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:48:31.285 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:05.984974+0000", 2024-04-20T17:48:31.285 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8449999999999994e-05 2024-04-20T17:48:31.285 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.285 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.285 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:48:31.285 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:05.985051+0000", 2024-04-20T17:48:31.285 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6990999999999995e-05 2024-04-20T17:48:31.285 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.285 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.285 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:48:31.285 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.100818+0000", 2024-04-20T17:48:31.285 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.115767256 2024-04-20T17:48:31.285 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.285 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.285 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:48:31.285 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.100836+0000", 2024-04-20T17:48:31.285 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8309e-05 2024-04-20T17:48:31.286 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.286 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.286 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:48:31.286 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.100890+0000", 2024-04-20T17:48:31.286 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3717000000000001e-05 2024-04-20T17:48:31.286 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.286 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.286 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:48:31.286 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.100943+0000", 2024-04-20T17:48:31.286 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3182e-05 2024-04-20T17:48:31.286 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.286 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:48:31.286 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.286 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.287 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.287 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(client.27460.0:271375 8.0 e877/777 ondisk, result = 0)", 2024-04-20T17:48:31.287 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:48:05.989012+0000", 2024-04-20T17:48:31.287 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 25.239963276000001, 2024-04-20T17:48:31.287 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.11181564500000001, 2024-04-20T17:48:31.287 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:48:31.287 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:48:31.287 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:48:31.287 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:48:31.287 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.287 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:48:31.287 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:05.989012+0000", 2024-04-20T17:48:31.287 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.287 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.287 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.287 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:48:31.288 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:05.989012+0000", 2024-04-20T17:48:31.288 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.288 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.288 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.288 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:48:31.288 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:05.989013+0000", 2024-04-20T17:48:31.288 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.64e-06 2024-04-20T17:48:31.288 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.288 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.288 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:48:31.288 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:05.989019+0000", 2024-04-20T17:48:31.288 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8649999999999998e-06 2024-04-20T17:48:31.288 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.288 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.288 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:48:31.288 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:05.989030+0000", 2024-04-20T17:48:31.288 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1060000000000001e-05 2024-04-20T17:48:31.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.289 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.289 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:48:31.289 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:05.989037+0000", 2024-04-20T17:48:31.289 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0979999999999996e-06 2024-04-20T17:48:31.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.289 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.289 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:48:31.289 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.100799+0000", 2024-04-20T17:48:31.289 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.11176170000000001 2024-04-20T17:48:31.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.289 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.289 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:48:31.289 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.100809+0000", 2024-04-20T17:48:31.289 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.046e-05 2024-04-20T17:48:31.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.290 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.290 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:48:31.290 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.100827+0000", 2024-04-20T17:48:31.290 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7822e-05 2024-04-20T17:48:31.290 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.290 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:48:31.290 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.290 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.290 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.290 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27460.0:271388 8.8 8:11f13751:::benchmark_data_smithi171_175757_object126274:head [delete] snapc 0=[] ondisk+write+known_if_redirected e877)", 2024-04-20T17:48:31.290 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:48:06.001004+0000", 2024-04-20T17:48:31.290 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 25.227970642999999, 2024-04-20T17:48:31.290 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.112649993, 2024-04-20T17:48:31.290 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:48:31.290 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:48:31.291 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:48:31.291 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:48:31.291 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27460", 2024-04-20T17:48:31.291 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:48:31.291 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 271388 2024-04-20T17:48:31.291 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.291 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:48:31.291 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.291 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:48:31.291 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.001004+0000", 2024-04-20T17:48:31.291 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.291 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.291 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.291 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:48:31.291 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.001004+0000", 2024-04-20T17:48:31.292 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.292 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:48:31.292 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.001007+0000", 2024-04-20T17:48:31.292 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1870000000000001e-06 2024-04-20T17:48:31.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.292 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:48:31.292 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.001017+0000", 2024-04-20T17:48:31.292 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1540000000000008e-06 2024-04-20T17:48:31.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.292 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:48:31.292 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.100769+0000", 2024-04-20T17:48:31.293 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.099752161000000006 2024-04-20T17:48:31.293 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.293 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.293 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:48:31.293 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.100785+0000", 2024-04-20T17:48:31.293 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6169999999999999e-05 2024-04-20T17:48:31.293 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.293 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.293 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:48:31.293 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.100847+0000", 2024-04-20T17:48:31.293 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2366000000000006e-05 2024-04-20T17:48:31.293 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.293 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.293 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:48:31.293 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.100970+0000", 2024-04-20T17:48:31.293 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012293800000000001 2024-04-20T17:48:31.294 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.294 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.294 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:48:31.294 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.101049+0000", 2024-04-20T17:48:31.294 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8578999999999994e-05 2024-04-20T17:48:31.294 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.294 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.294 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:48:31.294 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.113576+0000", 2024-04-20T17:48:31.294 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012526818 2024-04-20T17:48:31.294 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.294 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.294 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:48:31.294 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.113590+0000", 2024-04-20T17:48:31.295 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4295e-05 2024-04-20T17:48:31.295 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.295 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:48:31.295 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.113620+0000", 2024-04-20T17:48:31.295 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0530000000000001e-05 2024-04-20T17:48:31.295 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.295 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:48:31.295 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.113654+0000", 2024-04-20T17:48:31.295 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3794999999999997e-05 2024-04-20T17:48:31.295 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.295 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:48:31.295 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.295 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.296 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27460.0:271449 8.0 8:04b27317:::benchmark_data_smithi171_175757_object126335:head [delete] snapc 0=[] ondisk+write+known_if_redirected e877)", 2024-04-20T17:48:31.296 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:48:06.101025+0000", 2024-04-20T17:48:31.296 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 25.127949667999999, 2024-04-20T17:48:31.296 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.045790477000000003, 2024-04-20T17:48:31.296 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:48:31.296 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:48:31.296 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:48:31.296 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:48:31.296 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27460", 2024-04-20T17:48:31.296 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:48:31.296 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 271449 2024-04-20T17:48:31.296 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.296 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:48:31.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.296 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:48:31.297 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.101025+0000", 2024-04-20T17:48:31.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.297 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.297 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:48:31.297 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.101025+0000", 2024-04-20T17:48:31.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.297 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.297 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:48:31.297 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.101026+0000", 2024-04-20T17:48:31.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2100000000000002e-07 2024-04-20T17:48:31.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.297 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.297 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:48:31.298 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.101028+0000", 2024-04-20T17:48:31.298 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2409999999999998e-06 2024-04-20T17:48:31.298 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.298 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.298 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:48:31.298 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.101033+0000", 2024-04-20T17:48:31.298 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7679999999999999e-06 2024-04-20T17:48:31.298 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.298 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.298 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:48:31.298 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.101038+0000", 2024-04-20T17:48:31.298 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9609999999999996e-06 2024-04-20T17:48:31.298 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.298 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.299 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:48:31.299 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.131613+0000", 2024-04-20T17:48:31.299 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.030575727 2024-04-20T17:48:31.299 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.299 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.299 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:48:31.299 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.131725+0000", 2024-04-20T17:48:31.299 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00011210899999999999 2024-04-20T17:48:31.299 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.299 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.299 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:48:31.299 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.131843+0000", 2024-04-20T17:48:31.299 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000117855 2024-04-20T17:48:31.299 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.299 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.299 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:48:31.300 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.136942+0000", 2024-04-20T17:48:31.300 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0050990139999999998 2024-04-20T17:48:31.300 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.300 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:48:31.300 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.146692+0000", 2024-04-20T17:48:31.300 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0097501919999999995 2024-04-20T17:48:31.300 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.300 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:48:31.300 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.146755+0000", 2024-04-20T17:48:31.300 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2166000000000001e-05 2024-04-20T17:48:31.300 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.300 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:48:31.301 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:06.146816+0000", 2024-04-20T17:48:31.301 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1023000000000002e-05 2024-04-20T17:48:31.301 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.301 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:48:31.301 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.301 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.301 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.301 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27460.0:277543 8.0 8:07b035a5:::benchmark_data_smithi171_175757_object132429:head [delete] snapc 0=[] ondisk+write+known_if_redirected e878)", 2024-04-20T17:48:31.301 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:48:12.517983+0000", 2024-04-20T17:48:31.301 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 18.710991497999999, 2024-04-20T17:48:31.301 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.129015932, 2024-04-20T17:48:31.301 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:48:31.301 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:48:31.301 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:48:31.301 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:48:31.301 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27460", 2024-04-20T17:48:31.301 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:48:31.302 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 277543 2024-04-20T17:48:31.302 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.302 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:48:31.302 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.302 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:48:31.302 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.517983+0000", 2024-04-20T17:48:31.302 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.302 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.302 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.302 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:48:31.302 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.517983+0000", 2024-04-20T17:48:31.302 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.302 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.303 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.303 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:48:31.303 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.517985+0000", 2024-04-20T17:48:31.303 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.367e-06 2024-04-20T17:48:31.303 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.303 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.303 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:48:31.303 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.517989+0000", 2024-04-20T17:48:31.303 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8670000000000004e-06 2024-04-20T17:48:31.303 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.303 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.303 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:48:31.303 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.517995+0000", 2024-04-20T17:48:31.303 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.827e-06 2024-04-20T17:48:31.303 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.303 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.303 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:48:31.304 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.518000+0000", 2024-04-20T17:48:31.304 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.4179999999999999e-06 2024-04-20T17:48:31.304 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.304 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.304 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:48:31.304 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.518049+0000", 2024-04-20T17:48:31.304 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8810999999999997e-05 2024-04-20T17:48:31.304 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.304 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.304 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:48:31.304 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.518177+0000", 2024-04-20T17:48:31.304 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012818500000000001 2024-04-20T17:48:31.304 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.304 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.304 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:48:31.304 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.518305+0000", 2024-04-20T17:48:31.305 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012814099999999999 2024-04-20T17:48:31.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.305 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.305 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:48:31.305 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.641592+0000", 2024-04-20T17:48:31.305 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.12328702699999999 2024-04-20T17:48:31.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.305 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.305 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:48:31.305 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.646904+0000", 2024-04-20T17:48:31.305 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0053122630000000002 2024-04-20T17:48:31.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.305 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.305 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:48:31.305 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.646951+0000", 2024-04-20T17:48:31.306 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6925999999999998e-05 2024-04-20T17:48:31.306 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.306 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:48:31.306 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.646999+0000", 2024-04-20T17:48:31.306 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.8099999999999997e-05 2024-04-20T17:48:31.306 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.306 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:48:31.306 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.306 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.306 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27460.0:277545 8.8 8:1f9777ec:::benchmark_data_smithi171_175757_object132431:head [delete] snapc 0=[] ondisk+write+known_if_redirected e878)", 2024-04-20T17:48:31.306 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:48:12.523204+0000", 2024-04-20T17:48:31.307 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 18.705770714, 2024-04-20T17:48:31.307 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.12777819500000001, 2024-04-20T17:48:31.307 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:48:31.307 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:48:31.307 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:48:31.307 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:48:31.307 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27460", 2024-04-20T17:48:31.307 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:48:31.307 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 277545 2024-04-20T17:48:31.307 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.307 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:48:31.307 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.307 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:48:31.307 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.523204+0000", 2024-04-20T17:48:31.307 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.307 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.307 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.308 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:48:31.308 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.523204+0000", 2024-04-20T17:48:31.308 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.308 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.308 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.308 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:48:31.308 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.523206+0000", 2024-04-20T17:48:31.308 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.55e-06 2024-04-20T17:48:31.308 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.308 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.308 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:48:31.308 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.523211+0000", 2024-04-20T17:48:31.308 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3229999999999997e-06 2024-04-20T17:48:31.308 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.309 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.309 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:48:31.309 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636473+0000", 2024-04-20T17:48:31.309 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.113261801 2024-04-20T17:48:31.309 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.309 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.309 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:48:31.309 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636491+0000", 2024-04-20T17:48:31.309 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8383e-05 2024-04-20T17:48:31.309 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.309 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.309 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:48:31.309 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636586+0000", 2024-04-20T17:48:31.309 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.5161999999999995e-05 2024-04-20T17:48:31.309 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.310 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.310 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:48:31.310 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636722+0000", 2024-04-20T17:48:31.310 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000135542 2024-04-20T17:48:31.310 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.310 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.310 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:48:31.310 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636843+0000", 2024-04-20T17:48:31.310 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012078699999999999 2024-04-20T17:48:31.310 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.310 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.310 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:48:31.310 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.641595+0000", 2024-04-20T17:48:31.311 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0047521050000000004 2024-04-20T17:48:31.311 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.311 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.311 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:48:31.311 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.650874+0000", 2024-04-20T17:48:31.311 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0092789340000000008 2024-04-20T17:48:31.311 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.311 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.311 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:48:31.311 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.650932+0000", 2024-04-20T17:48:31.311 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8038000000000002e-05 2024-04-20T17:48:31.311 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.311 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.311 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:48:31.311 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.650982+0000", 2024-04-20T17:48:31.312 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0569999999999999e-05 2024-04-20T17:48:31.312 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.312 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:48:31.312 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.312 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.312 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.312 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27460.0:277602 8.8 8:14eda16b:::benchmark_data_smithi171_175757_object132488:head [delete] snapc 0=[] ondisk+write+known_if_redirected e878)", 2024-04-20T17:48:31.312 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:48:12.636641+0000", 2024-04-20T17:48:31.312 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 18.592334012999999, 2024-04-20T17:48:31.312 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.043804327999999997, 2024-04-20T17:48:31.312 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:48:31.312 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:48:31.312 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:48:31.312 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:48:31.312 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27460", 2024-04-20T17:48:31.312 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:48:31.313 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 277602 2024-04-20T17:48:31.313 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.313 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:48:31.313 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.313 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:48:31.313 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636641+0000", 2024-04-20T17:48:31.313 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.313 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.313 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.313 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:48:31.313 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636641+0000", 2024-04-20T17:48:31.313 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.313 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.313 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.314 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:48:31.314 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636641+0000", 2024-04-20T17:48:31.314 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6899999999999998e-07 2024-04-20T17:48:31.314 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.314 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.314 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:48:31.314 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636644+0000", 2024-04-20T17:48:31.314 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3640000000000002e-06 2024-04-20T17:48:31.314 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.314 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.314 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:48:31.314 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636648+0000", 2024-04-20T17:48:31.314 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2150000000000003e-06 2024-04-20T17:48:31.314 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.314 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.315 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:48:31.315 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636652+0000", 2024-04-20T17:48:31.315 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1219999999999997e-06 2024-04-20T17:48:31.315 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.315 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.315 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:48:31.315 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.656303+0000", 2024-04-20T17:48:31.315 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.019650714999999999 2024-04-20T17:48:31.315 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.315 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.315 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:48:31.315 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.656444+0000", 2024-04-20T17:48:31.315 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014138900000000001 2024-04-20T17:48:31.315 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.315 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.316 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:48:31.316 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.656584+0000", 2024-04-20T17:48:31.316 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013984499999999999 2024-04-20T17:48:31.316 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.316 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.316 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:48:31.316 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.667035+0000", 2024-04-20T17:48:31.316 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010450575 2024-04-20T17:48:31.316 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.316 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.316 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:48:31.316 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.680343+0000", 2024-04-20T17:48:31.316 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013308537 2024-04-20T17:48:31.316 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.316 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.316 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:48:31.316 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.680396+0000", 2024-04-20T17:48:31.317 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2580999999999997e-05 2024-04-20T17:48:31.317 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.317 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.317 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:48:31.317 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.680445+0000", 2024-04-20T17:48:31.317 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9515999999999998e-05 2024-04-20T17:48:31.317 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.317 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:48:31.317 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.317 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.317 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.317 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27460.0:277606 8.0 8:0b5acc5d:::benchmark_data_smithi171_175757_object132492:head [delete] snapc 0=[] ondisk+write+known_if_redirected e878)", 2024-04-20T17:48:31.317 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:48:12.636657+0000", 2024-04-20T17:48:31.317 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 18.592318062, 2024-04-20T17:48:31.318 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.043874538999999997, 2024-04-20T17:48:31.318 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:48:31.318 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:48:31.318 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:48:31.318 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:48:31.318 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27460", 2024-04-20T17:48:31.318 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:48:31.318 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 277606 2024-04-20T17:48:31.318 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.318 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:48:31.318 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.318 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:48:31.318 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636657+0000", 2024-04-20T17:48:31.319 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.319 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.319 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.319 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:48:31.319 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636657+0000", 2024-04-20T17:48:31.319 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.319 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.319 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.319 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:48:31.319 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636657+0000", 2024-04-20T17:48:31.319 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3000000000000001e-07 2024-04-20T17:48:31.319 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.319 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.319 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:48:31.319 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636660+0000", 2024-04-20T17:48:31.320 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.401e-06 2024-04-20T17:48:31.320 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.320 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.320 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:48:31.320 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636664+0000", 2024-04-20T17:48:31.320 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0069999999999999e-06 2024-04-20T17:48:31.320 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.320 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.320 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:48:31.320 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636668+0000", 2024-04-20T17:48:31.320 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.014e-06 2024-04-20T17:48:31.320 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.320 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.320 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:48:31.321 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.656294+0000", 2024-04-20T17:48:31.321 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.019626232 2024-04-20T17:48:31.321 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.321 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.321 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:48:31.321 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.656448+0000", 2024-04-20T17:48:31.321 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000154036 2024-04-20T17:48:31.321 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.321 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.321 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:48:31.321 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.656584+0000", 2024-04-20T17:48:31.321 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013594400000000001 2024-04-20T17:48:31.321 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.321 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.322 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:48:31.322 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.667037+0000", 2024-04-20T17:48:31.322 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.010453036000000001 2024-04-20T17:48:31.322 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.322 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.322 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:48:31.322 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.680457+0000", 2024-04-20T17:48:31.322 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013419614999999999 2024-04-20T17:48:31.322 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.322 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.322 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:48:31.322 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.680498+0000", 2024-04-20T17:48:31.322 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0958999999999999e-05 2024-04-20T17:48:31.322 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.323 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.323 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:48:31.323 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.680531+0000", 2024-04-20T17:48:31.323 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.3764999999999999e-05 2024-04-20T17:48:31.323 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.323 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:48:31.323 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.323 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.323 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.323 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27460.0:277617 8.0 8:0a9cae4d:::benchmark_data_smithi171_175757_object132503:head [delete] snapc 0=[] ondisk+write+known_if_redirected e878)", 2024-04-20T17:48:31.323 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:48:12.636672+0000", 2024-04-20T17:48:31.323 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 18.592302591999999, 2024-04-20T17:48:31.323 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.048211847000000002, 2024-04-20T17:48:31.323 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:48:31.324 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:48:31.324 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:48:31.324 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:48:31.324 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27460", 2024-04-20T17:48:31.324 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:48:31.324 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 277617 2024-04-20T17:48:31.324 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.324 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:48:31.324 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.324 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:48:31.324 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636672+0000", 2024-04-20T17:48:31.324 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.324 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.324 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.324 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:48:31.325 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636672+0000", 2024-04-20T17:48:31.325 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.325 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.325 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.325 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:48:31.325 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636673+0000", 2024-04-20T17:48:31.325 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9200000000000001e-07 2024-04-20T17:48:31.325 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.325 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.325 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:48:31.325 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636675+0000", 2024-04-20T17:48:31.325 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.334e-06 2024-04-20T17:48:31.325 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.325 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.326 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:48:31.326 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636682+0000", 2024-04-20T17:48:31.326 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.849e-06 2024-04-20T17:48:31.326 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.326 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.326 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:48:31.326 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636687+0000", 2024-04-20T17:48:31.326 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9969999999999998e-06 2024-04-20T17:48:31.326 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.326 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.326 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:48:31.326 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.661830+0000", 2024-04-20T17:48:31.326 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.025143057999999999 2024-04-20T17:48:31.326 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.326 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.327 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:48:31.327 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.661937+0000", 2024-04-20T17:48:31.327 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000107068 2024-04-20T17:48:31.327 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.327 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.327 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:48:31.327 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.662062+0000", 2024-04-20T17:48:31.327 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000124709 2024-04-20T17:48:31.327 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.327 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.327 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:48:31.327 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.680363+0000", 2024-04-20T17:48:31.327 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.018301339999999999 2024-04-20T17:48:31.327 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.327 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.327 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:48:31.328 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.684758+0000", 2024-04-20T17:48:31.328 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00439494 2024-04-20T17:48:31.328 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.328 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.328 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:48:31.328 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.684819+0000", 2024-04-20T17:48:31.328 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0511999999999999e-05 2024-04-20T17:48:31.328 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.328 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.328 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:48:31.328 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.684884+0000", 2024-04-20T17:48:31.328 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5548000000000003e-05 2024-04-20T17:48:31.328 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.328 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:48:31.328 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.328 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.329 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.329 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27460.0:277640 8.8 8:1af88fba:::benchmark_data_smithi171_175757_object132526:head [delete] snapc 0=[] ondisk+write+known_if_redirected e878)", 2024-04-20T17:48:31.329 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:48:12.636693+0000", 2024-04-20T17:48:31.329 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 18.592281651, 2024-04-20T17:48:31.329 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.043916085000000001, 2024-04-20T17:48:31.329 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:48:31.329 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:48:31.329 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:48:31.329 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:48:31.329 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27460", 2024-04-20T17:48:31.329 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:48:31.329 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 277640 2024-04-20T17:48:31.329 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.329 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:48:31.329 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.329 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:48:31.330 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636693+0000", 2024-04-20T17:48:31.330 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.330 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.330 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.330 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:48:31.330 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636693+0000", 2024-04-20T17:48:31.330 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.330 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.330 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.330 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:48:31.330 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636694+0000", 2024-04-20T17:48:31.330 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5700000000000002e-07 2024-04-20T17:48:31.330 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.330 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.331 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:48:31.331 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636697+0000", 2024-04-20T17:48:31.331 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9490000000000001e-06 2024-04-20T17:48:31.331 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.331 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.331 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:48:31.331 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636702+0000", 2024-04-20T17:48:31.331 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5210000000000002e-06 2024-04-20T17:48:31.331 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.331 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.331 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:48:31.331 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.636708+0000", 2024-04-20T17:48:31.331 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7370000000000003e-06 2024-04-20T17:48:31.331 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.331 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.331 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:48:31.331 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.661821+0000", 2024-04-20T17:48:31.332 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.025112760000000001 2024-04-20T17:48:31.332 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.332 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.332 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:48:31.332 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.661938+0000", 2024-04-20T17:48:31.332 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000116934 2024-04-20T17:48:31.332 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.332 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.332 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:48:31.332 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.662062+0000", 2024-04-20T17:48:31.332 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012398300000000001 2024-04-20T17:48:31.332 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.332 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.332 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:48:31.332 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.680320+0000", 2024-04-20T17:48:31.333 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.01825821 2024-04-20T17:48:31.333 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.333 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.333 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:48:31.333 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.680538+0000", 2024-04-20T17:48:31.333 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000218204 2024-04-20T17:48:31.333 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.333 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.333 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:48:31.333 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.680578+0000", 2024-04-20T17:48:31.333 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9878e-05 2024-04-20T17:48:31.333 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.333 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.333 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:48:31.333 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.680609+0000", 2024-04-20T17:48:31.333 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1251999999999998e-05 2024-04-20T17:48:31.334 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.334 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:48:31.334 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.334 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.334 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.334 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27460.0:277644 8.0 8:0a795fc1:::benchmark_data_smithi171_175757_object132530:head [delete] snapc 0=[] ondisk+write+known_if_redirected e878)", 2024-04-20T17:48:31.334 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:48:12.637207+0000", 2024-04-20T17:48:31.334 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 18.591767945000001, 2024-04-20T17:48:31.334 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.051838413, 2024-04-20T17:48:31.334 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:48:31.334 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:48:31.334 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:48:31.334 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:48:31.334 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27460", 2024-04-20T17:48:31.334 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:48:31.335 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 277644 2024-04-20T17:48:31.335 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.335 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:48:31.335 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.335 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:48:31.335 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.637207+0000", 2024-04-20T17:48:31.335 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.335 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.335 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.335 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:48:31.335 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.637207+0000", 2024-04-20T17:48:31.335 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:48:31.335 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.335 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.335 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:48:31.336 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.637208+0000", 2024-04-20T17:48:31.336 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.7999999999999993e-07 2024-04-20T17:48:31.336 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.336 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.336 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:48:31.336 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.637212+0000", 2024-04-20T17:48:31.336 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.8139999999999998e-06 2024-04-20T17:48:31.336 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.336 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.336 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:48:31.336 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.637219+0000", 2024-04-20T17:48:31.336 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2459999999999997e-06 2024-04-20T17:48:31.336 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.336 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.336 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:48:31.336 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.637223+0000", 2024-04-20T17:48:31.336 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5240000000000002e-06 2024-04-20T17:48:31.337 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.337 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.337 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:48:31.337 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.667074+0000", 2024-04-20T17:48:31.337 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.029850287999999999 2024-04-20T17:48:31.337 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.337 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.337 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:48:31.337 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.667199+0000", 2024-04-20T17:48:31.337 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012546700000000001 2024-04-20T17:48:31.337 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.337 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.337 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 5", 2024-04-20T17:48:31.337 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.667321+0000", 2024-04-20T17:48:31.337 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012163399999999999 2024-04-20T17:48:31.338 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.338 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.338 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:48:31.338 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.680396+0000", 2024-04-20T17:48:31.338 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013074806 2024-04-20T17:48:31.338 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.338 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.338 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:48:31.338 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.688920+0000", 2024-04-20T17:48:31.338 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0085240730000000001 2024-04-20T17:48:31.338 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.338 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.338 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:48:31.338 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.688988+0000", 2024-04-20T17:48:31.339 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8701000000000005e-05 2024-04-20T17:48:31.339 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:48:31.339 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:48:31.339 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:48:31.339 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:12.689045+0000", 2024-04-20T17:48:31.339 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6879999999999998e-05 2024-04-20T17:48:31.339 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.339 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:48:31.339 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.339 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:48:31.339 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:48:31.339 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:48:31.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:31.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:31.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:31.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:30 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:31.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:30 smithi171 ceph-mon[44189]: pgmap v2346: 27 pgs: 27 active+clean; 361 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 584 op/s 2024-04-20T17:48:31.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:31.341 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:48:31.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:48:31.442 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:48:31.471 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:32.631 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:48:32.907 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:48:32.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:32.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:48:33.014 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:48:33.042 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:33.191 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:48:33.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:33.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:48:33.294 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:48:33.327 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:33.349 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:48:33.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:33.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:48:33.453 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:48:33.483 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:33.684 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:48:33.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:33.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:48:33.788 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:48:33.820 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:33.842 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:48:33.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:33.846 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:48:33.946 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:48:33.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:33 smithi005 ceph-mon[39562]: pgmap v2347: 27 pgs: 27 active+clean; 361 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 424 op/s 2024-04-20T17:48:33.977 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:33 smithi106 ceph-mon[44423]: pgmap v2347: 27 pgs: 27 active+clean; 361 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 424 op/s 2024-04-20T17:48:33.980 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:34.001 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:48:34.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:34.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:48:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:33 smithi171 ceph-mon[44189]: pgmap v2347: 27 pgs: 27 active+clean; 361 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 424 op/s 2024-04-20T17:48:34.105 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:48:34.134 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:34.378 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-20T17:48:34.483 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:48:34.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:34.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:48:34.588 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:48:34.591 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:48:34.620 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:34.642 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:48:34.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:34.645 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:48:34.746 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:48:34.775 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:34.795 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:48:34.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:34.799 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:48:34.899 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:48:34.928 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:34.982 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:48:34.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:34.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:48:35.067 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:48:35.086 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:48:35.118 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:35.140 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:48:35.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:35.143 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:48:35.244 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:48:35.277 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:35.298 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:48:35.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:35.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:48:35.402 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:48:35.432 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:35.714 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'unique_pool_6' removed 2024-04-20T17:48:35.714 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.3 to 1 (10000) 2024-04-20T17:48:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:35 smithi106 ceph-mon[44423]: pgmap v2348: 27 pgs: 27 active+clean; 361 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 269 op/s 2024-04-20T17:48:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:35 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4215653448' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-20T17:48:36.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:35 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/251220250' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:48:36.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:35 smithi171 ceph-mon[44189]: pgmap v2348: 27 pgs: 27 active+clean; 361 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 269 op/s 2024-04-20T17:48:36.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:35 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4215653448' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-20T17:48:36.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:35 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/251220250' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:48:36.126 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:35 smithi005 ceph-mon[39562]: pgmap v2348: 27 pgs: 27 active+clean; 361 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 269 op/s 2024-04-20T17:48:36.126 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:35 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4215653448' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-20T17:48:36.126 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:35 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/251220250' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:48:36.566 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 4 1 2024-04-20T17:48:36.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:36 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4215653448' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-04-20T17:48:36.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:36 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/251220250' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-20T17:48:36.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:36 smithi005 ceph-mon[39562]: osdmap e885: 12 total, 12 up, 12 in 2024-04-20T17:48:36.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:36 smithi005 ceph-mon[39562]: osdmap e886: 12 total, 12 up, 12 in 2024-04-20T17:48:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:36 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4215653448' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-04-20T17:48:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:36 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/251220250' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-20T17:48:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:36 smithi106 ceph-mon[44423]: osdmap e885: 12 total, 12 up, 12 in 2024-04-20T17:48:37.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:36 smithi106 ceph-mon[44423]: osdmap e886: 12 total, 12 up, 12 in 2024-04-20T17:48:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:36 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4215653448' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-04-20T17:48:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:36 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/251220250' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-20T17:48:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:36 smithi171 ceph-mon[44189]: osdmap e885: 12 total, 12 up, 12 in 2024-04-20T17:48:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:36 smithi171 ceph-mon[44189]: osdmap e886: 12 total, 12 up, 12 in 2024-04-20T17:48:37.497 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:48:37.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:37.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:48:37.602 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:48:37.635 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:37.658 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:48:37.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:37.661 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:48:37.762 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:48:37.792 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:37.813 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:48:37.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:37.816 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:48:37.917 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:48:37.949 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:37.992 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:48:38.003 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:48:38.006 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:37 smithi005 ceph-mon[39562]: pgmap v2350: 11 pgs: 11 active+clean; 361 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:38.006 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:37 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:48:38.006 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:37 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:48:38.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:38.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:48:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:37 smithi106 ceph-mon[44423]: pgmap v2350: 11 pgs: 11 active+clean; 361 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:37 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:48:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:37 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:48:38.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:37 smithi171 ceph-mon[44189]: pgmap v2350: 11 pgs: 11 active+clean; 361 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:38.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:37 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:48:38.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:37 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:48:38.107 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:48:38.135 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:38.338 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-20T17:48:38.339 INFO:tasks.radosbench:Beginning radosbench... 2024-04-20T17:48:38.339 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_7 2024-04-20T17:48:38.339 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pool create unique_pool_7 16 2024-04-20T17:48:38.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:38 smithi005 ceph-mon[39562]: osdmap e887: 12 total, 12 up, 12 in 2024-04-20T17:48:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:38 smithi106 ceph-mon[44423]: osdmap e887: 12 total, 12 up, 12 in 2024-04-20T17:48:39.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:38 smithi171 ceph-mon[44189]: osdmap e887: 12 total, 12 up, 12 in 2024-04-20T17:48:39.086 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:48:39.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:39.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:48:39.113 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:48:39.191 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:48:39.224 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:39.246 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:48:39.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:39.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:48:39.350 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:48:39.382 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:39.742 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:48:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:39 smithi106 ceph-mon[44423]: pgmap v2353: 11 pgs: 11 active+clean; 361 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:39 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3675057600' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:48:40.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:39 smithi171 ceph-mon[44189]: pgmap v2353: 11 pgs: 11 active+clean; 361 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:40.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:39 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3675057600' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:48:40.196 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:39 smithi005 ceph-mon[39562]: pgmap v2353: 11 pgs: 11 active+clean; 361 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:40.196 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:39 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3675057600' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:48:40.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:40 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:48:40.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:40 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3675057600' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:48:40.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:40 smithi005 ceph-mon[39562]: osdmap e888: 12 total, 12 up, 12 in 2024-04-20T17:48:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:40 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:48:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:40 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3675057600' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:48:41.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:40 smithi106 ceph-mon[44423]: osdmap e888: 12 total, 12 up, 12 in 2024-04-20T17:48:41.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:40 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:48:41.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:40 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3675057600' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:48:41.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:40 smithi171 ceph-mon[44189]: osdmap e888: 12 total, 12 up, 12 in 2024-04-20T17:48:41.334 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:48:41.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:41.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:48:41.438 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:48:41.472 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:41.494 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:48:41.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:41.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:48:41.598 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:48:41.628 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:41.648 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:48:41.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:41.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:48:41.752 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:48:41.783 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:41.804 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:48:41.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:41.807 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:48:41.908 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:48:41.943 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:41 smithi106 ceph-mon[44423]: pgmap v2355: 11 pgs: 11 active+clean; 361 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:42.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:41 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:48:42.012 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:41 smithi005 ceph-mon[39562]: pgmap v2355: 11 pgs: 11 active+clean; 361 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:42.012 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:41 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:48:42.017 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:48:42.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:42.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:48:42.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:41 smithi171 ceph-mon[44189]: pgmap v2355: 11 pgs: 11 active+clean; 361 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:42.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:41 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:48:42.122 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:48:42.154 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:42.179 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:48:42.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:42.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:48:42.282 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:48:42.313 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:42.369 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:48:42.369 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:48:42.369 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-04-20T17:48:42.369 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:48:42.562 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:48:42.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:42.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:48:42.666 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:48:42.695 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:42.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:42 smithi005 ceph-mon[39562]: 1.a scrub starts 2024-04-20T17:48:42.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:42 smithi005 ceph-mon[39562]: 1.a scrub ok 2024-04-20T17:48:42.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:42 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:48:42.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:42 smithi005 ceph-mon[39562]: 1.9 scrub starts 2024-04-20T17:48:42.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:42 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/120544103' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-20T17:48:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:42 smithi106 ceph-mon[44423]: 1.a scrub starts 2024-04-20T17:48:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:42 smithi106 ceph-mon[44423]: 1.a scrub ok 2024-04-20T17:48:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:42 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:48:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:42 smithi106 ceph-mon[44423]: 1.9 scrub starts 2024-04-20T17:48:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:42 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/120544103' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-20T17:48:43.043 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:48:43.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:43.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:48:43.071 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-20T17:48:43.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:42 smithi171 ceph-mon[44189]: 1.a scrub starts 2024-04-20T17:48:43.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:42 smithi171 ceph-mon[44189]: 1.a scrub ok 2024-04-20T17:48:43.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:42 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:48:43.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:42 smithi171 ceph-mon[44189]: 1.9 scrub starts 2024-04-20T17:48:43.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:42 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/120544103' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-20T17:48:43.147 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:48:43.181 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:43.203 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:48:43.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:43.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:48:43.307 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:48:43.339 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:43.359 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:48:43.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:43.362 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:48:43.463 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:48:43.496 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:43.517 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:48:43.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:43.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:48:43.622 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:48:43.654 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:43.674 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:48:43.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:43.678 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:48:43.779 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:48:43.811 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:43.833 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:48:43.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:43.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:48:43.839 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:48:43.907 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.4 to 1 (10000) 2024-04-20T17:48:43.937 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:48:43.966 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:43 smithi171 ceph-mon[44189]: 1.9 scrub ok 2024-04-20T17:48:43.967 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:43 smithi171 ceph-mon[44189]: 1.7 scrub starts 2024-04-20T17:48:43.967 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:43 smithi171 ceph-mon[44189]: 1.7 scrub ok 2024-04-20T17:48:43.967 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:43 smithi171 ceph-mon[44189]: pgmap v2356: 11 pgs: 11 active+clean; 309 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:43.967 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:43 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/120544103' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-20T17:48:43.967 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:43 smithi171 ceph-mon[44189]: osdmap e889: 12 total, 12 up, 12 in 2024-04-20T17:48:43.969 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:43.991 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:48:43.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:43.994 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:48:44.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:43 smithi106 ceph-mon[44423]: 1.9 scrub ok 2024-04-20T17:48:44.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:43 smithi106 ceph-mon[44423]: 1.7 scrub starts 2024-04-20T17:48:44.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:43 smithi106 ceph-mon[44423]: 1.7 scrub ok 2024-04-20T17:48:44.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:43 smithi106 ceph-mon[44423]: pgmap v2356: 11 pgs: 11 active+clean; 309 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:44.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:43 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/120544103' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-20T17:48:44.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:43 smithi106 ceph-mon[44423]: osdmap e889: 12 total, 12 up, 12 in 2024-04-20T17:48:44.096 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:48:44.129 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:44.151 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:48:44.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:44.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:48:44.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:43 smithi005 ceph-mon[39562]: 1.9 scrub ok 2024-04-20T17:48:44.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:43 smithi005 ceph-mon[39562]: 1.7 scrub starts 2024-04-20T17:48:44.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:43 smithi005 ceph-mon[39562]: 1.7 scrub ok 2024-04-20T17:48:44.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:43 smithi005 ceph-mon[39562]: pgmap v2356: 11 pgs: 11 active+clean; 309 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:44.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:43 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/120544103' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-20T17:48:44.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:43 smithi005 ceph-mon[39562]: osdmap e889: 12 total, 12 up, 12 in 2024-04-20T17:48:44.254 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:48:44.285 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:44.291 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:48:44.292 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:48:44.292 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:48:44.292 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:48:44.292 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:48:44.292 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:48:44.292 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:48:44.292 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:48:44.305 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:48:44.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:44.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:48:44.312 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:48:44.333 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:48:44.353 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:48:44.373 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:48:44.375 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:48:44.375 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:48:44.375 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:48:44.375 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:48:44.375 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:48:44.375 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:48:44.375 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:48:44.375 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:48:44.375 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:48:44.375 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:48:44.375 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:48:44.376 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:48:44.408 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:48:44.436 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:44.455 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:48:44.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:44.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:48:44.559 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:48:44.589 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:45.052 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:44 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2884149606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:48:45.052 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:44 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:45.052 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:44 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:45.052 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:44 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:45.066 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:48:45.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:45.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:48:45.079 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 5 1 2024-04-20T17:48:45.082 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:48:45.082 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:48:45.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:44 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2884149606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:48:45.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:44 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:45.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:44 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:45.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:44 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:45.170 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:48:45.197 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:45.217 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:48:45.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:45.220 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:48:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:44 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2884149606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:48:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:44 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:44 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:44 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:45.321 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:48:45.354 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:45.378 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:48:45.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:45.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:48:45.483 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:48:45.510 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:45.528 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:48:45.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:45.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:48:45.550 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:48:45.632 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:48:45.659 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:45.677 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:48:45.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:45.681 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:48:45.782 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:48:45.812 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:45.833 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:48:45.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:45.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:48:45.937 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:48:45.963 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:45.982 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:48:45.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:45.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:48:46.069 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:46.070 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:46.070 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:46.070 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:46.070 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:46.070 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:46.070 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:46.070 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:46.070 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:46.070 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:45 smithi005 ceph-mon[39562]: pgmap v2358: 11 pgs: 11 active+clean; 309 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:46.070 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:48:46.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:46.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:46.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:46.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:46.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:46.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:46.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:46.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:46.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:46.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:45 smithi171 ceph-mon[44189]: pgmap v2358: 11 pgs: 11 active+clean; 309 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:46.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:48:46.086 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:48:46.116 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:46.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:46.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:46.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:48:46.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:45 smithi106 ceph-mon[44423]: pgmap v2358: 11 pgs: 11 active+clean; 309 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:46.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:48:46.698 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:48:47.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:46 smithi171 ceph-mon[44189]: 1.5 scrub starts 2024-04-20T17:48:47.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:46 smithi171 ceph-mon[44189]: 1.5 scrub ok 2024-04-20T17:48:47.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:46 smithi171 ceph-mon[44189]: 1.8 scrub starts 2024-04-20T17:48:47.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:46 smithi171 ceph-mon[44189]: 1.8 scrub ok 2024-04-20T17:48:47.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:46 smithi171 ceph-mon[44189]: 1.4 scrub starts 2024-04-20T17:48:47.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:46 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:48:47.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:46 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:48:47.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:46 smithi171 ceph-mon[44189]: 1.4 scrub ok 2024-04-20T17:48:47.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:46 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:48:47.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:46 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:48:47.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:46 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:48:47.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:46 smithi171 ceph-mon[44189]: osdmap e890: 12 total, 12 up, 12 in 2024-04-20T17:48:47.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:46 smithi005 ceph-mon[39562]: 1.5 scrub starts 2024-04-20T17:48:47.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:46 smithi005 ceph-mon[39562]: 1.5 scrub ok 2024-04-20T17:48:47.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:46 smithi005 ceph-mon[39562]: 1.8 scrub starts 2024-04-20T17:48:47.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:46 smithi005 ceph-mon[39562]: 1.8 scrub ok 2024-04-20T17:48:47.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:46 smithi005 ceph-mon[39562]: 1.4 scrub starts 2024-04-20T17:48:47.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:46 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:48:47.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:46 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:48:47.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:46 smithi005 ceph-mon[39562]: 1.4 scrub ok 2024-04-20T17:48:47.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:46 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:48:47.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:46 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:48:47.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:46 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:48:47.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:46 smithi005 ceph-mon[39562]: osdmap e890: 12 total, 12 up, 12 in 2024-04-20T17:48:47.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:46 smithi106 ceph-mon[44423]: 1.5 scrub starts 2024-04-20T17:48:47.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:46 smithi106 ceph-mon[44423]: 1.5 scrub ok 2024-04-20T17:48:47.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:46 smithi106 ceph-mon[44423]: 1.8 scrub starts 2024-04-20T17:48:47.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:46 smithi106 ceph-mon[44423]: 1.8 scrub ok 2024-04-20T17:48:47.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:46 smithi106 ceph-mon[44423]: 1.4 scrub starts 2024-04-20T17:48:47.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:46 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:48:47.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:46 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:48:47.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:46 smithi106 ceph-mon[44423]: 1.4 scrub ok 2024-04-20T17:48:47.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:46 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:48:47.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:46 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:48:47.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:46 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:48:47.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:46 smithi106 ceph-mon[44423]: osdmap e890: 12 total, 12 up, 12 in 2024-04-20T17:48:47.536 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:48:47.947 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:47 smithi005 ceph-mon[39562]: pgmap v2359: 11 pgs: 11 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:47.947 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:47 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:48:47.947 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:47 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:48:47.966 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:48:47.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:47.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:48:48.072 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:48:48.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:47 smithi171 ceph-mon[44189]: pgmap v2359: 11 pgs: 11 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:48.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:47 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:48:48.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:47 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:48:48.105 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:48.127 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:48:48.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:48.130 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:48:48.231 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:48:48.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:47 smithi106 ceph-mon[44423]: pgmap v2359: 11 pgs: 11 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:48.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:47 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:48:48.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:47 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:48:48.265 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:48.287 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:48:48.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:48.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:48:48.391 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:48:48.423 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:48.445 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:48:48.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:48.448 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:48:48.549 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:48:48.580 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:48.601 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:48:48.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:48.604 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:48:48.704 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:48:48.733 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:48.754 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:48:48.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:48.757 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:48:48.858 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:48:48.886 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:48.907 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:48:48.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:48.910 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:48:49.011 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:48:49.041 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:48 smithi106 ceph-mon[44423]: osdmap e891: 12 total, 12 up, 12 in 2024-04-20T17:48:49.043 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:49.065 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:48:49.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:49.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:48:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:48 smithi171 ceph-mon[44189]: osdmap e891: 12 total, 12 up, 12 in 2024-04-20T17:48:49.169 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:48:49.197 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:49.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:48 smithi005 ceph-mon[39562]: osdmap e891: 12 total, 12 up, 12 in 2024-04-20T17:48:49.218 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:48:49.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:49.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:48:49.322 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:48:49.350 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:49.369 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:48:49.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:49.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:48:49.473 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:48:49.502 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:49.521 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:48:49.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:49.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:48:49.625 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:48:49.656 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:49.709 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:48:49.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:49.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:48:49.785 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:48:49.813 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:48:49.842 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:49.916 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:48:49.917 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:49 smithi005 ceph-mon[39562]: pgmap v2362: 11 pgs: 11 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:49.917 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:49 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1048907369' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:48:49.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:49.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:48:50.021 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:48:50.051 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:49 smithi106 ceph-mon[44423]: pgmap v2362: 11 pgs: 11 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:50.051 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:49 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1048907369' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:48:50.053 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:50.074 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:48:50.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:50.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:48:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:49 smithi171 ceph-mon[44189]: pgmap v2362: 11 pgs: 11 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:49 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1048907369' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:48:50.178 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:48:50.206 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:50.455 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:48:50.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:50.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:48:50.560 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:48:50.592 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:50.614 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:48:50.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:50.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:48:50.718 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:48:50.750 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:50.771 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:48:50.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:50.774 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:48:50.875 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:48:50.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:50 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:48:50.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:50 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:48:50.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:50 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1048907369' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:48:50.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:50 smithi106 ceph-mon[44423]: osdmap e892: 12 total, 12 up, 12 in 2024-04-20T17:48:50.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:50 smithi106 ceph-mon[44423]: 1.2 scrub starts 2024-04-20T17:48:50.904 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:50 smithi106 ceph-mon[44423]: 1.2 scrub ok 2024-04-20T17:48:50.906 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:50.926 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:48:50.926 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:48:50.927 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:48:50.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:50.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:48:51.032 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:48:51.058 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:50 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:48:51.058 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:50 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:48:51.058 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:50 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1048907369' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:48:51.058 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:50 smithi005 ceph-mon[39562]: osdmap e892: 12 total, 12 up, 12 in 2024-04-20T17:48:51.058 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:50 smithi005 ceph-mon[39562]: 1.2 scrub starts 2024-04-20T17:48:51.058 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:50 smithi005 ceph-mon[39562]: 1.2 scrub ok 2024-04-20T17:48:51.061 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:50 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:48:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:50 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:48:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:50 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1048907369' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:48:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:50 smithi171 ceph-mon[44189]: osdmap e892: 12 total, 12 up, 12 in 2024-04-20T17:48:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:50 smithi171 ceph-mon[44189]: 1.2 scrub starts 2024-04-20T17:48:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:50 smithi171 ceph-mon[44189]: 1.2 scrub ok 2024-04-20T17:48:51.789 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.5 to 1 (10000) 2024-04-20T17:48:51.815 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:51 smithi005 ceph-mon[39562]: pgmap v2364: 11 pgs: 11 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:51.815 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:51 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2217235239' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:48:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:51 smithi171 ceph-mon[44189]: pgmap v2364: 11 pgs: 11 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:51 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2217235239' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:48:52.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:51 smithi106 ceph-mon[44423]: pgmap v2364: 11 pgs: 11 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:52.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:51 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2217235239' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:48:52.415 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:48:52.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:52.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:48:52.478 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:48:52.538 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:48:52.570 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:52.592 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:48:52.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:52.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:48:52.696 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:48:52.724 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:52.848 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:48:52.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:52.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:48:52.866 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 8 1 2024-04-20T17:48:52.951 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:48:52.982 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:52 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2217235239' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-20T17:48:52.982 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:52 smithi106 ceph-mon[44423]: osdmap e893: 12 total, 12 up, 12 in 2024-04-20T17:48:52.984 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:53.006 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:48:53.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:53.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:48:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:52 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2217235239' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-20T17:48:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:52 smithi171 ceph-mon[44189]: osdmap e893: 12 total, 12 up, 12 in 2024-04-20T17:48:53.110 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:48:53.142 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:53.163 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:48:53.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:53.167 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:48:53.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:52 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2217235239' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-20T17:48:53.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:52 smithi005 ceph-mon[39562]: osdmap e893: 12 total, 12 up, 12 in 2024-04-20T17:48:53.268 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:48:53.297 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:53.318 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:48:53.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:53.321 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:48:53.422 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:48:53.450 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:53.699 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:48:53.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:53.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:48:53.713 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-20T17:48:53.753 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:48:53.797 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:48:53.803 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:48:53.813 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'unique_pool_7' created 2024-04-20T17:48:53.834 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:53.854 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:48:53.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:53.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:48:53.958 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:48:53.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:53 smithi005 ceph-mon[39562]: pgmap v2366: 11 pgs: 11 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:53.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:53 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2950305006' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-04-20T17:48:53.987 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:54.006 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:48:54.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:54.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:48:54.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:53 smithi171 ceph-mon[44189]: pgmap v2366: 11 pgs: 11 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:54.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:53 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2950305006' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-04-20T17:48:54.109 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:48:54.141 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:54.164 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:48:54.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:54.167 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:48:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:53 smithi106 ceph-mon[44423]: pgmap v2366: 11 pgs: 11 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:53 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2950305006' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-04-20T17:48:54.268 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:48:54.296 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:54.369 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:48:54.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:54.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:48:54.473 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:48:54.506 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:54.527 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:48:54.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:54.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:48:54.632 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:48:54.661 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:54.681 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:48:54.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:54.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:48:54.785 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:48:54.822 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:54.843 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:48:54.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:54.847 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:48:54.947 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:48:54.977 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:55.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:54 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2950305006' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-04-20T17:48:55.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:54 smithi171 ceph-mon[44189]: osdmap e894: 12 total, 12 up, 12 in 2024-04-20T17:48:55.162 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:54 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2950305006' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-04-20T17:48:55.162 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:54 smithi005 ceph-mon[39562]: osdmap e894: 12 total, 12 up, 12 in 2024-04-20T17:48:55.179 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:48:55.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:55.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:48:55.187 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_7 rados --yes-i-really-mean-it || true 2024-04-20T17:48:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:54 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2950305006' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-04-20T17:48:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:54 smithi106 ceph-mon[44423]: osdmap e894: 12 total, 12 up, 12 in 2024-04-20T17:48:55.284 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:48:55.316 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:55.338 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:48:55.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:55.341 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:48:55.442 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:48:55.475 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:55.496 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:48:55.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:55.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:48:55.515 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:48:55.600 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:48:55.632 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:55.653 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:48:55.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:55.657 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:48:55.758 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:48:55.786 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:55.936 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:48:55.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:55.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:48:56.041 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:48:56.071 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:55 smithi171 ceph-mon[44189]: pgmap v2368: 27 pgs: 4 creating+peering, 12 unknown, 11 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:56.071 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:55 smithi171 ceph-mon[44189]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-20T17:48:56.071 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:55 smithi171 ceph-mon[44189]: osdmap e895: 12 total, 12 up, 12 in 2024-04-20T17:48:56.073 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:56.095 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:48:56.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:56.099 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:48:56.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:55 smithi005 ceph-mon[39562]: pgmap v2368: 27 pgs: 4 creating+peering, 12 unknown, 11 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:56.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:55 smithi005 ceph-mon[39562]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-20T17:48:56.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:55 smithi005 ceph-mon[39562]: osdmap e895: 12 total, 12 up, 12 in 2024-04-20T17:48:56.199 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:48:56.229 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:55 smithi106 ceph-mon[44423]: pgmap v2368: 27 pgs: 4 creating+peering, 12 unknown, 11 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:56.229 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:55 smithi106 ceph-mon[44423]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-20T17:48:56.229 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:55 smithi106 ceph-mon[44423]: osdmap e895: 12 total, 12 up, 12 in 2024-04-20T17:48:56.231 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:56.253 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:48:56.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:56.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:48:56.357 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:48:56.386 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:57.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:56 smithi171 ceph-mon[44189]: osdmap e896: 12 total, 12 up, 12 in 2024-04-20T17:48:57.122 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:48:57.128 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:48:57.138 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:48:57.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:56 smithi005 ceph-mon[39562]: osdmap e896: 12 total, 12 up, 12 in 2024-04-20T17:48:57.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:57.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:48:57.244 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:48:57.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:56 smithi106 ceph-mon[44423]: osdmap e896: 12 total, 12 up, 12 in 2024-04-20T17:48:57.273 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:57.347 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:48:57.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:57.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:48:57.452 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:48:57.484 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:57.506 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:48:57.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:57.509 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:48:57.610 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:48:57.639 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:57.678 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:48:57.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:57.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:48:57.799 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:48:57.833 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:57.856 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:48:57.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:57.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:48:57.960 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:48:57.992 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:58.014 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:48:58.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:58.017 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:48:58.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:57 smithi171 ceph-mon[44189]: pgmap v2371: 27 pgs: 6 creating+peering, 3 unknown, 18 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:58.118 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:48:58.145 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:57 smithi005 ceph-mon[39562]: pgmap v2371: 27 pgs: 6 creating+peering, 3 unknown, 18 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:58.147 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:58.202 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:48:58.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:58.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:48:58.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:57 smithi106 ceph-mon[44423]: pgmap v2371: 27 pgs: 6 creating+peering, 3 unknown, 18 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:48:58.307 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:48:58.341 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:58.362 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:48:58.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:58.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:48:58.466 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:48:58.497 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:58.516 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:48:58.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:58.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:48:58.621 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:48:58.650 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:58.669 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:48:58.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:58.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:48:58.773 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:48:58.802 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:59.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:58 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:48:59.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:58 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:48:59.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:58 smithi106 ceph-mon[44423]: osdmap e897: 12 total, 12 up, 12 in 2024-04-20T17:48:59.051 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:48:59.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:59.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:48:59.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:58 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:48:59.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:58 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:48:59.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:58 smithi171 ceph-mon[44189]: osdmap e897: 12 total, 12 up, 12 in 2024-04-20T17:48:59.157 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:48:59.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:58 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:48:59.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:58 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:48:59.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:58 smithi005 ceph-mon[39562]: osdmap e897: 12 total, 12 up, 12 in 2024-04-20T17:48:59.185 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:59.533 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:48:59.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:59.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:48:59.637 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:48:59.669 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:48:59.691 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:48:59.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:48:59.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:48:59.795 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:48:59.824 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:00.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:59 smithi106 ceph-mon[44423]: pgmap v2373: 27 pgs: 6 creating+peering, 21 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:49:00.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:48:59 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1852237649' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:49:00.024 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:49:00.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:00.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:49:00.079 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:59 smithi005 ceph-mon[39562]: pgmap v2373: 27 pgs: 6 creating+peering, 21 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:49:00.080 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:48:59 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1852237649' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:49:00.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:59 smithi171 ceph-mon[44189]: pgmap v2373: 27 pgs: 6 creating+peering, 21 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:49:00.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:48:59 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1852237649' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:49:00.129 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:49:00.161 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:00.183 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:49:00.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:00.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:49:00.287 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:49:00.317 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:00.517 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:49:00.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:00.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:49:00.621 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:49:00.650 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:00.748 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:49:00.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:00.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:49:00.854 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:49:00.886 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:00.907 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:49:00.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:00.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:49:01.011 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:49:01.039 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:01.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:00 smithi005 ceph-mon[39562]: pgmap v2374: 27 pgs: 27 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:49:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:00 smithi106 ceph-mon[44423]: pgmap v2374: 27 pgs: 27 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:49:01.264 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:49:01.264 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:49:01.264 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:49:01.264 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:49:01.264 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:49:01.264 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:49:01.265 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:49:01.265 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:49:01.265 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:49:01.265 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:49:01.265 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:49:01.265 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:49:01.265 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:49:01.265 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:49:01.265 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:49:01.265 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:49:01.265 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:49:01.265 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:49:01.265 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:49:01.265 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:49:01.265 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:49:01.266 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:49:01.266 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:49:01.266 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:49:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:01.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:00 smithi171 ceph-mon[44189]: pgmap v2374: 27 pgs: 27 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:49:01.717 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:49:02.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:01 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1243444991' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:49:02.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:01 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:49:02.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:01 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1243444991' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:49:02.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:01 smithi005 ceph-mon[39562]: osdmap e898: 12 total, 12 up, 12 in 2024-04-20T17:49:02.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:01 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1243444991' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:49:02.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:01 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:49:02.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:01 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1243444991' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:49:02.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:01 smithi106 ceph-mon[44423]: osdmap e898: 12 total, 12 up, 12 in 2024-04-20T17:49:02.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:01 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1243444991' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:49:02.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:01 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:49:02.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:01 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1243444991' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:49:02.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:01 smithi171 ceph-mon[44189]: osdmap e898: 12 total, 12 up, 12 in 2024-04-20T17:49:02.689 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:49:02.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:02.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:49:02.794 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:49:02.831 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:03.136 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:49:03.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:02 smithi005 ceph-mon[39562]: pgmap v2376: 27 pgs: 27 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:49:03.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:03.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:49:03.241 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:49:03.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:02 smithi106 ceph-mon[44423]: pgmap v2376: 27 pgs: 27 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:49:03.269 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:49:03.269 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:49:03.269 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:49:03.269 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:49:03.269 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.269 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:141776 8.1 e807/806 8:874c7de1:::benchmark_data_smithi171_175757_object141775:head v 807'8838, mlcod=782'5)", 2024-04-20T17:49:03.269 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:53.803994+0000", 2024-04-20T17:49:03.270 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 249.46343239399999, 2024-04-20T17:49:03.270 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.059946456000000002, 2024-04-20T17:49:03.270 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:03.270 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:03.270 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:03.270 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:03.270 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.270 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:03.270 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.803994+0000", 2024-04-20T17:49:03.270 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.270 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.270 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.270 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:03.270 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.803994+0000", 2024-04-20T17:49:03.270 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.271 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.271 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.271 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:03.271 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.803995+0000", 2024-04-20T17:49:03.271 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0210000000000001e-06 2024-04-20T17:49:03.271 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.271 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.271 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:03.271 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.804064+0000", 2024-04-20T17:49:03.271 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8670000000000005e-05 2024-04-20T17:49:03.271 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.271 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.271 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:03.272 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.804073+0000", 2024-04-20T17:49:03.272 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6889999999999999e-06 2024-04-20T17:49:03.272 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.272 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.272 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:03.272 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.804079+0000", 2024-04-20T17:49:03.272 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6509999999999998e-06 2024-04-20T17:49:03.272 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.272 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.272 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:03.272 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.806919+0000", 2024-04-20T17:49:03.272 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002840174 2024-04-20T17:49:03.272 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.272 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.272 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:03.272 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.806949+0000", 2024-04-20T17:49:03.273 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9611000000000002e-05 2024-04-20T17:49:03.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.273 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.273 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:03.273 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.863898+0000", 2024-04-20T17:49:03.273 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.056949234000000001 2024-04-20T17:49:03.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.273 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.273 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:03.273 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.863941+0000", 2024-04-20T17:49:03.273 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2406e-05 2024-04-20T17:49:03.273 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.273 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:03.273 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.274 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.274 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:141779 8.1 e807/806 8:88362553:::benchmark_data_smithi171_175757_object141778:head v 807'8839, mlcod=782'5)", 2024-04-20T17:49:03.274 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:53.808292+0000", 2024-04-20T17:49:03.274 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 249.459134062, 2024-04-20T17:49:03.274 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.072673096000000006, 2024-04-20T17:49:03.274 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:03.274 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:03.274 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:03.274 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:03.274 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.274 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:03.274 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.808292+0000", 2024-04-20T17:49:03.274 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.274 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.275 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.275 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:03.275 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.808292+0000", 2024-04-20T17:49:03.275 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.275 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.275 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.275 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:03.275 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.808294+0000", 2024-04-20T17:49:03.275 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.192e-06 2024-04-20T17:49:03.275 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.275 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.275 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:03.275 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.808343+0000", 2024-04-20T17:49:03.275 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9437000000000001e-05 2024-04-20T17:49:03.275 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.276 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.276 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:03.276 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.808353+0000", 2024-04-20T17:49:03.276 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0018e-05 2024-04-20T17:49:03.276 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.276 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.276 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:03.276 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.808359+0000", 2024-04-20T17:49:03.276 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8719999999999999e-06 2024-04-20T17:49:03.276 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.276 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.276 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:03.276 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.816072+0000", 2024-04-20T17:49:03.276 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0077126929999999996 2024-04-20T17:49:03.276 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.277 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.277 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:03.277 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.816101+0000", 2024-04-20T17:49:03.277 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9819e-05 2024-04-20T17:49:03.277 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.277 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.277 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:03.277 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.880926+0000", 2024-04-20T17:49:03.277 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.064824344000000006 2024-04-20T17:49:03.277 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.277 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.277 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:03.277 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.880965+0000", 2024-04-20T17:49:03.278 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9721000000000001e-05 2024-04-20T17:49:03.278 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.278 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:03.278 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.278 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.278 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.278 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:141780 8.1 e807/806 8:853ffce1:::benchmark_data_smithi171_175757_object141779:head v 807'8840, mlcod=782'5)", 2024-04-20T17:49:03.278 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:53.812805+0000", 2024-04-20T17:49:03.278 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 249.454621874, 2024-04-20T17:49:03.278 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.068191458999999996, 2024-04-20T17:49:03.278 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:03.278 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:03.278 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:03.278 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:03.278 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.279 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:03.279 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.812805+0000", 2024-04-20T17:49:03.279 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.279 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.279 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.279 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:03.279 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.812805+0000", 2024-04-20T17:49:03.279 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.279 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.279 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.279 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:03.279 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.812806+0000", 2024-04-20T17:49:03.279 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.781e-06 2024-04-20T17:49:03.279 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.279 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.279 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:03.280 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.812836+0000", 2024-04-20T17:49:03.280 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9433000000000001e-05 2024-04-20T17:49:03.280 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.280 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.280 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:03.280 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.812847+0000", 2024-04-20T17:49:03.280 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0943e-05 2024-04-20T17:49:03.280 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.280 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.280 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:03.280 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.812854+0000", 2024-04-20T17:49:03.280 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8310000000000003e-06 2024-04-20T17:49:03.280 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.280 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.280 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:03.280 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.824055+0000", 2024-04-20T17:49:03.280 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011201778000000001 2024-04-20T17:49:03.281 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.281 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.281 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:03.281 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.824075+0000", 2024-04-20T17:49:03.281 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9763e-05 2024-04-20T17:49:03.281 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.281 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.281 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:03.281 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.880971+0000", 2024-04-20T17:49:03.281 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.056895657000000002 2024-04-20T17:49:03.281 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.281 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.281 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:03.281 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.880996+0000", 2024-04-20T17:49:03.281 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5273e-05 2024-04-20T17:49:03.282 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.282 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:03.282 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.282 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.282 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.282 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:141781 8.1 e807/806 8:81259101:::benchmark_data_smithi171_175757_object141780:head v 807'8841, mlcod=782'5)", 2024-04-20T17:49:03.282 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:53.818147+0000", 2024-04-20T17:49:03.282 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 249.44927938399999, 2024-04-20T17:49:03.282 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.062882813999999995, 2024-04-20T17:49:03.282 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:03.282 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:03.282 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:03.282 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:03.282 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.282 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:03.283 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.818147+0000", 2024-04-20T17:49:03.283 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.283 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.283 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:03.283 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.818147+0000", 2024-04-20T17:49:03.283 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.283 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.283 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:03.283 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.818149+0000", 2024-04-20T17:49:03.283 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.827e-06 2024-04-20T17:49:03.283 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.283 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:03.284 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.818203+0000", 2024-04-20T17:49:03.284 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4391999999999997e-05 2024-04-20T17:49:03.284 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.284 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.284 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:03.284 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.818215+0000", 2024-04-20T17:49:03.284 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2159e-05 2024-04-20T17:49:03.284 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.284 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.284 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:03.284 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.818224+0000", 2024-04-20T17:49:03.284 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8990000000000007e-06 2024-04-20T17:49:03.284 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.284 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.284 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:03.284 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.830838+0000", 2024-04-20T17:49:03.285 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012613236 2024-04-20T17:49:03.285 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.285 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.285 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:03.285 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.830870+0000", 2024-04-20T17:49:03.285 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2710999999999999e-05 2024-04-20T17:49:03.285 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.285 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.285 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:03.285 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.881009+0000", 2024-04-20T17:49:03.285 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.050139184000000003 2024-04-20T17:49:03.285 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.285 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.285 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:03.285 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.881030+0000", 2024-04-20T17:49:03.285 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0406000000000001e-05 2024-04-20T17:49:03.285 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.286 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:03.286 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.286 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.286 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.286 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:143174 8.3 e807/806 8:c2e7ba4e:::benchmark_data_smithi171_175757_object143173:head v 807'8934, mlcod=782'40)", 2024-04-20T17:49:03.286 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:55.689369+0000", 2024-04-20T17:49:03.286 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 247.578057712, 2024-04-20T17:49:03.286 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.055773729000000001, 2024-04-20T17:49:03.286 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:03.286 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:03.286 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:03.286 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:03.286 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.286 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:03.287 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:55.689369+0000", 2024-04-20T17:49:03.287 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.287 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.287 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.287 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:03.287 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:55.689369+0000", 2024-04-20T17:49:03.287 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.287 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.287 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.287 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:03.287 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:55.689370+0000", 2024-04-20T17:49:03.288 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1100000000000005e-07 2024-04-20T17:49:03.288 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.288 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.288 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:03.288 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:55.689426+0000", 2024-04-20T17:49:03.288 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6876999999999998e-05 2024-04-20T17:49:03.288 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.288 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.288 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:03.288 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:55.689434+0000", 2024-04-20T17:49:03.288 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7340000000000008e-06 2024-04-20T17:49:03.288 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.288 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.288 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:03.288 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:55.689438+0000", 2024-04-20T17:49:03.289 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1869999999999999e-06 2024-04-20T17:49:03.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.289 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.289 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:03.289 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:55.702002+0000", 2024-04-20T17:49:03.289 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012563194 2024-04-20T17:49:03.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.289 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.289 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:03.289 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:55.702027+0000", 2024-04-20T17:49:03.289 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5931999999999998e-05 2024-04-20T17:49:03.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.289 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.289 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:03.289 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:55.745085+0000", 2024-04-20T17:49:03.290 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.043057722 2024-04-20T17:49:03.290 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.290 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.290 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:03.290 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:55.745142+0000", 2024-04-20T17:49:03.290 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7272000000000003e-05 2024-04-20T17:49:03.290 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.290 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:03.290 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.290 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.290 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.290 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:144039 8.7 e807/806 8:ee75ddde:::benchmark_data_smithi171_175757_object144038:head v 807'8913, mlcod=782'73)", 2024-04-20T17:49:03.290 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:56.891321+0000", 2024-04-20T17:49:03.290 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 246.376105155, 2024-04-20T17:49:03.290 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.058286162000000002, 2024-04-20T17:49:03.291 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:03.291 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:03.291 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:03.291 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:03.291 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.291 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:03.291 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:56.891321+0000", 2024-04-20T17:49:03.291 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.291 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.291 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.291 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:03.291 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:56.891321+0000", 2024-04-20T17:49:03.291 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.291 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.291 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.292 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:03.292 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:56.891323+0000", 2024-04-20T17:49:03.292 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.956e-06 2024-04-20T17:49:03.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.292 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:03.292 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:56.891368+0000", 2024-04-20T17:49:03.292 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5219999999999997e-05 2024-04-20T17:49:03.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.292 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:03.292 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:56.891383+0000", 2024-04-20T17:49:03.292 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4059999999999999e-05 2024-04-20T17:49:03.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.292 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.292 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:03.293 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:56.891391+0000", 2024-04-20T17:49:03.293 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1899999999999995e-06 2024-04-20T17:49:03.293 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.293 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.293 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:03.293 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:56.891481+0000", 2024-04-20T17:49:03.293 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0205999999999997e-05 2024-04-20T17:49:03.293 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.293 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.293 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:03.293 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:56.891499+0000", 2024-04-20T17:49:03.293 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8042000000000001e-05 2024-04-20T17:49:03.293 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.293 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.293 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:03.293 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:56.949568+0000", 2024-04-20T17:49:03.293 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.058069321 2024-04-20T17:49:03.294 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.294 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.294 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:03.294 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:56.949607+0000", 2024-04-20T17:49:03.294 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9167e-05 2024-04-20T17:49:03.294 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.294 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:03.294 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.294 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.294 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.294 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:144287 8.7 e807/806 8:ee831792:::benchmark_data_smithi171_175757_object144286:head v 807'8930, mlcod=782'79)", 2024-04-20T17:49:03.294 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:57.236564+0000", 2024-04-20T17:49:03.294 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 246.030862945, 2024-04-20T17:49:03.294 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.075174172999999997, 2024-04-20T17:49:03.294 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:03.294 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:03.295 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:03.295 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:03.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.295 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:03.295 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.236564+0000", 2024-04-20T17:49:03.295 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.295 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.295 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:03.295 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.236564+0000", 2024-04-20T17:49:03.295 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.295 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.295 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:03.296 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.236564+0000", 2024-04-20T17:49:03.296 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0399999999999996e-07 2024-04-20T17:49:03.296 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.296 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:03.296 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.236592+0000", 2024-04-20T17:49:03.296 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8288000000000001e-05 2024-04-20T17:49:03.296 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.296 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:03.296 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.236597+0000", 2024-04-20T17:49:03.296 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0499999999999999e-06 2024-04-20T17:49:03.296 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.296 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:03.296 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.236601+0000", 2024-04-20T17:49:03.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.647e-06 2024-04-20T17:49:03.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.297 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.297 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:03.297 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.239114+0000", 2024-04-20T17:49:03.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025130420000000001 2024-04-20T17:49:03.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.297 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.297 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:03.297 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.239130+0000", 2024-04-20T17:49:03.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6059999999999999e-05 2024-04-20T17:49:03.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.297 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.297 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:03.297 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.311673+0000", 2024-04-20T17:49:03.297 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.072542822000000007 2024-04-20T17:49:03.298 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.298 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.298 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:03.298 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.311738+0000", 2024-04-20T17:49:03.298 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4759999999999997e-05 2024-04-20T17:49:03.298 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.298 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:03.298 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.298 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.298 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.298 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:144288 8.7 e807/806 8:eef6a00e:::benchmark_data_smithi171_175757_object144287:head v 807'8931, mlcod=782'79)", 2024-04-20T17:49:03.298 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:57.238491+0000", 2024-04-20T17:49:03.298 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 246.02893563200001, 2024-04-20T17:49:03.298 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.073276907000000002, 2024-04-20T17:49:03.298 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:03.299 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:03.299 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:03.299 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:03.299 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.299 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:03.299 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.238491+0000", 2024-04-20T17:49:03.299 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.299 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.299 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.299 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:03.299 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.238491+0000", 2024-04-20T17:49:03.299 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.299 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.299 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.299 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:03.299 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.238493+0000", 2024-04-20T17:49:03.300 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9099999999999999e-06 2024-04-20T17:49:03.300 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.300 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:03.300 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.238796+0000", 2024-04-20T17:49:03.300 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00030377399999999998 2024-04-20T17:49:03.300 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.300 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:03.300 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.238805+0000", 2024-04-20T17:49:03.300 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2819999999999996e-06 2024-04-20T17:49:03.300 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.300 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:03.300 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.238811+0000", 2024-04-20T17:49:03.301 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0480000000000004e-06 2024-04-20T17:49:03.301 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.301 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.301 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:03.301 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.247356+0000", 2024-04-20T17:49:03.301 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0085456300000000002 2024-04-20T17:49:03.301 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.301 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.301 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:03.301 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.247378+0000", 2024-04-20T17:49:03.301 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1098999999999999e-05 2024-04-20T17:49:03.301 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.301 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.301 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:03.301 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.311743+0000", 2024-04-20T17:49:03.302 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.064365393000000007 2024-04-20T17:49:03.302 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.302 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.302 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:03.302 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.311768+0000", 2024-04-20T17:49:03.302 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4771000000000001e-05 2024-04-20T17:49:03.302 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.302 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:03.302 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.302 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.302 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.302 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:144297 8.7 e807/806 8:e477555a:::benchmark_data_smithi171_175757_object144296:head v 807'8932, mlcod=782'79)", 2024-04-20T17:49:03.302 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:57.242447+0000", 2024-04-20T17:49:03.302 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 246.02497955300001, 2024-04-20T17:49:03.302 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.069367660999999997, 2024-04-20T17:49:03.302 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:03.303 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:03.303 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:03.303 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:03.303 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.303 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:03.303 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.242447+0000", 2024-04-20T17:49:03.303 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.303 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.303 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.303 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:03.303 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.242447+0000", 2024-04-20T17:49:03.303 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.303 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.303 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.303 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:03.304 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.242449+0000", 2024-04-20T17:49:03.304 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9609999999999999e-06 2024-04-20T17:49:03.304 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.304 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.304 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:03.304 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.242483+0000", 2024-04-20T17:49:03.304 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4607999999999998e-05 2024-04-20T17:49:03.304 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.304 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.304 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:03.304 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.242498+0000", 2024-04-20T17:49:03.304 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4133e-05 2024-04-20T17:49:03.304 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.304 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.304 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:03.304 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.242508+0000", 2024-04-20T17:49:03.305 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0125e-05 2024-04-20T17:49:03.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.305 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.305 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:03.305 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.255277+0000", 2024-04-20T17:49:03.305 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012769487 2024-04-20T17:49:03.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.305 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.305 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:03.305 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.255327+0000", 2024-04-20T17:49:03.305 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9344000000000001e-05 2024-04-20T17:49:03.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.305 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.305 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:03.305 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.311773+0000", 2024-04-20T17:49:03.305 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.056446552999999997 2024-04-20T17:49:03.305 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.306 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:03.306 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.311815+0000", 2024-04-20T17:49:03.306 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1449999999999998e-05 2024-04-20T17:49:03.306 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.306 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:03.306 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.306 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.306 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.306 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:144299 8.7 e807/806 8:ebf4afb0:::benchmark_data_smithi171_175757_object144298:head v 807'8933, mlcod=782'79)", 2024-04-20T17:49:03.306 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:57.246826+0000", 2024-04-20T17:49:03.306 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 246.02060085299999, 2024-04-20T17:49:03.306 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.065020307999999999, 2024-04-20T17:49:03.306 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:03.306 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:03.306 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:03.307 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:03.307 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.307 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:03.307 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.246826+0000", 2024-04-20T17:49:03.307 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.307 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.307 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.307 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:03.307 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.246826+0000", 2024-04-20T17:49:03.307 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.307 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.307 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.307 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:03.307 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.246827+0000", 2024-04-20T17:49:03.307 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.37e-06 2024-04-20T17:49:03.308 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.308 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.308 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:03.308 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.246887+0000", 2024-04-20T17:49:03.308 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9725000000000002e-05 2024-04-20T17:49:03.308 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.308 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.308 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:03.308 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.246896+0000", 2024-04-20T17:49:03.308 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8440000000000004e-06 2024-04-20T17:49:03.308 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.308 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.308 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:03.308 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.246902+0000", 2024-04-20T17:49:03.308 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9769999999999997e-06 2024-04-20T17:49:03.309 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.309 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.309 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:03.309 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.263252+0000", 2024-04-20T17:49:03.309 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.016350166999999999 2024-04-20T17:49:03.309 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.309 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.309 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:03.309 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.263270+0000", 2024-04-20T17:49:03.309 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8722000000000001e-05 2024-04-20T17:49:03.309 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.309 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.309 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:03.309 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.311818+0000", 2024-04-20T17:49:03.310 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.048547740999999998 2024-04-20T17:49:03.310 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.310 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.310 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:03.310 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.311846+0000", 2024-04-20T17:49:03.310 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7761999999999999e-05 2024-04-20T17:49:03.310 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.310 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:03.310 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.310 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.310 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.310 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:144300 8.7 e807/806 8:e2881ae0:::benchmark_data_smithi171_175757_object144299:head v 807'8934, mlcod=782'79)", 2024-04-20T17:49:03.310 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:57.251644+0000", 2024-04-20T17:49:03.311 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 246.01578279099999, 2024-04-20T17:49:03.311 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.060234792000000002, 2024-04-20T17:49:03.311 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:03.311 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:03.311 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:03.311 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:03.311 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.311 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:03.311 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.251644+0000", 2024-04-20T17:49:03.311 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.311 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.311 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.311 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:03.311 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.251644+0000", 2024-04-20T17:49:03.311 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.312 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.312 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.312 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:03.312 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.251646+0000", 2024-04-20T17:49:03.312 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.537e-06 2024-04-20T17:49:03.312 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.312 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.312 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:03.312 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.251685+0000", 2024-04-20T17:49:03.312 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9239999999999997e-05 2024-04-20T17:49:03.312 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.312 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.312 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:03.312 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.251698+0000", 2024-04-20T17:49:03.312 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3006999999999999e-05 2024-04-20T17:49:03.313 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.313 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.313 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:03.313 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.251708+0000", 2024-04-20T17:49:03.313 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9289999999999993e-06 2024-04-20T17:49:03.313 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.313 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.313 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:03.313 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.273674+0000", 2024-04-20T17:49:03.313 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.021965340999999999 2024-04-20T17:49:03.313 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.313 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.313 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:03.313 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.273706+0000", 2024-04-20T17:49:03.313 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1965000000000003e-05 2024-04-20T17:49:03.314 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.314 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.314 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:03.314 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.311849+0000", 2024-04-20T17:49:03.314 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.038143205999999999 2024-04-20T17:49:03.314 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.314 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.314 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:03.314 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.311878+0000", 2024-04-20T17:49:03.314 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9567e-05 2024-04-20T17:49:03.314 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.314 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:03.314 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.314 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.315 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.315 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:144816 8.1 e808/806 8:8c32a843:::benchmark_data_smithi171_175757_object144815:head v 808'9022, mlcod=782'91)", 2024-04-20T17:49:03.315 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:57.976986+0000", 2024-04-20T17:49:03.315 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 245.29044073099999, 2024-04-20T17:49:03.315 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.05750425, 2024-04-20T17:49:03.315 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:03.315 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:03.315 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:03.315 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:03.315 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.315 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:03.315 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.976986+0000", 2024-04-20T17:49:03.315 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.315 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.315 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.315 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:03.316 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.976986+0000", 2024-04-20T17:49:03.316 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.316 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.316 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.316 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:03.316 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.976988+0000", 2024-04-20T17:49:03.316 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1399999999999998e-06 2024-04-20T17:49:03.316 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.316 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.316 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:03.316 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.977028+0000", 2024-04-20T17:49:03.316 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9873000000000003e-05 2024-04-20T17:49:03.316 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.316 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.316 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:03.316 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.977040+0000", 2024-04-20T17:49:03.317 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2751999999999999e-05 2024-04-20T17:49:03.317 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.317 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.317 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:03.317 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.977047+0000", 2024-04-20T17:49:03.317 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5699999999999998e-06 2024-04-20T17:49:03.317 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.317 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.317 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:03.317 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.982723+0000", 2024-04-20T17:49:03.317 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0056760300000000003 2024-04-20T17:49:03.317 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.317 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.317 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:03.317 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.982748+0000", 2024-04-20T17:49:03.317 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4491e-05 2024-04-20T17:49:03.318 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.318 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.318 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:03.318 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:58.034438+0000", 2024-04-20T17:49:03.318 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.051690738 2024-04-20T17:49:03.318 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.318 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.318 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:03.318 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:58.034490+0000", 2024-04-20T17:49:03.318 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1656000000000001e-05 2024-04-20T17:49:03.318 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.318 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:03.318 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.318 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.318 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.319 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:144830 8.1 e808/806 8:85c3cdba:::benchmark_data_smithi171_175757_object144829:head v 808'9023, mlcod=782'91)", 2024-04-20T17:49:03.319 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:57.978077+0000", 2024-04-20T17:49:03.319 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 245.28934924500001, 2024-04-20T17:49:03.319 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.056453188000000001, 2024-04-20T17:49:03.319 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:03.319 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:03.319 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:03.319 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:03.319 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.319 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:03.319 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.978077+0000", 2024-04-20T17:49:03.319 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.319 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.319 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.319 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:03.319 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.978077+0000", 2024-04-20T17:49:03.319 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.320 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.320 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.320 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:03.320 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.978078+0000", 2024-04-20T17:49:03.320 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.186e-06 2024-04-20T17:49:03.320 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.320 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.320 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:03.320 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.978133+0000", 2024-04-20T17:49:03.320 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4703000000000001e-05 2024-04-20T17:49:03.320 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.320 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.320 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:03.320 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.978142+0000", 2024-04-20T17:49:03.320 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.67e-06 2024-04-20T17:49:03.321 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.321 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.321 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:03.321 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.978148+0000", 2024-04-20T17:49:03.321 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.02e-06 2024-04-20T17:49:03.321 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.321 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.321 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:03.321 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.993664+0000", 2024-04-20T17:49:03.321 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015516336 2024-04-20T17:49:03.321 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.321 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.321 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:03.321 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.993694+0000", 2024-04-20T17:49:03.321 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0184000000000001e-05 2024-04-20T17:49:03.322 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.322 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.322 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:03.322 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:58.034495+0000", 2024-04-20T17:49:03.322 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.040800287999999997 2024-04-20T17:49:03.322 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.322 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.322 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:03.322 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:58.034530+0000", 2024-04-20T17:49:03.322 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5800999999999998e-05 2024-04-20T17:49:03.322 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.322 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:03.322 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.322 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.323 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.323 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:145634 8.1 e808/806 8:8ca600c0:::benchmark_data_smithi171_175757_object520:head v 808'9068, mlcod=782'115)", 2024-04-20T17:49:03.323 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:59.137841+0000", 2024-04-20T17:49:03.323 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 244.12958551899999, 2024-04-20T17:49:03.323 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.064744457000000005, 2024-04-20T17:49:03.323 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:03.323 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:03.323 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:03.323 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:03.323 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.323 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:03.323 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.137841+0000", 2024-04-20T17:49:03.323 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.323 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.323 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.323 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:03.324 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.137841+0000", 2024-04-20T17:49:03.324 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.324 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.324 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.324 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:03.324 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.137843+0000", 2024-04-20T17:49:03.324 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.962e-06 2024-04-20T17:49:03.324 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.324 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.324 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:03.324 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.137872+0000", 2024-04-20T17:49:03.324 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9444000000000001e-05 2024-04-20T17:49:03.325 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.325 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.325 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:03.325 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.137884+0000", 2024-04-20T17:49:03.325 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1623999999999999e-05 2024-04-20T17:49:03.325 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.325 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.325 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:03.325 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.137892+0000", 2024-04-20T17:49:03.325 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8290000000000008e-06 2024-04-20T17:49:03.325 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.325 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.325 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:03.325 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.137951+0000", 2024-04-20T17:49:03.325 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8987999999999998e-05 2024-04-20T17:49:03.326 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.326 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.326 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:03.326 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.137975+0000", 2024-04-20T17:49:03.326 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4287999999999999e-05 2024-04-20T17:49:03.326 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.326 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.326 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:03.326 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.202514+0000", 2024-04-20T17:49:03.326 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.064539223000000007 2024-04-20T17:49:03.326 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.326 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.326 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:03.326 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.202585+0000", 2024-04-20T17:49:03.326 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1099000000000005e-05 2024-04-20T17:49:03.327 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.327 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:03.327 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.327 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.327 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.327 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:145649 8.1 e808/806 8:8ffd2a1b:::benchmark_data_smithi171_175757_object535:head v 808'9070, mlcod=782'116)", 2024-04-20T17:49:03.327 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:59.155596+0000", 2024-04-20T17:49:03.327 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 244.11183082599999, 2024-04-20T17:49:03.327 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057370499999999998, 2024-04-20T17:49:03.327 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:03.327 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:03.327 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:03.327 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:03.327 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.328 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:03.328 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.155596+0000", 2024-04-20T17:49:03.328 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.328 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.328 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.328 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:03.328 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.155596+0000", 2024-04-20T17:49:03.328 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.328 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.328 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.328 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:03.328 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.155599+0000", 2024-04-20T17:49:03.328 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2480000000000001e-06 2024-04-20T17:49:03.328 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.328 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.329 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:03.329 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.155614+0000", 2024-04-20T17:49:03.329 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5126999999999999e-05 2024-04-20T17:49:03.329 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.329 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.329 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:03.329 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.155631+0000", 2024-04-20T17:49:03.329 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7212000000000001e-05 2024-04-20T17:49:03.329 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.329 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.329 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:03.329 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.155643+0000", 2024-04-20T17:49:03.329 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1657e-05 2024-04-20T17:49:03.329 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.329 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.330 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:03.330 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.159944+0000", 2024-04-20T17:49:03.330 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.004301633 2024-04-20T17:49:03.330 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.330 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.330 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:03.330 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.159968+0000", 2024-04-20T17:49:03.330 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3558999999999999e-05 2024-04-20T17:49:03.330 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.330 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.330 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:03.330 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.212921+0000", 2024-04-20T17:49:03.330 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.052953001999999999 2024-04-20T17:49:03.330 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.330 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.331 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:03.331 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.212966+0000", 2024-04-20T17:49:03.331 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5062000000000003e-05 2024-04-20T17:49:03.331 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.331 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:03.331 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.331 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.331 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.331 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:145651 8.1 e808/806 8:86aaf9bb:::benchmark_data_smithi171_175757_object537:head v 808'9071, mlcod=782'116)", 2024-04-20T17:49:03.331 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:59.155671+0000", 2024-04-20T17:49:03.331 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 244.11175552099999, 2024-04-20T17:49:03.331 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057312692999999998, 2024-04-20T17:49:03.331 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:03.331 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:03.332 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:03.332 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:03.332 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.332 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:03.332 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.155671+0000", 2024-04-20T17:49:03.332 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.332 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.332 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.332 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:03.332 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.155671+0000", 2024-04-20T17:49:03.332 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.332 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.332 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.332 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:03.332 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.155672+0000", 2024-04-20T17:49:03.332 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6899999999999997e-07 2024-04-20T17:49:03.333 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.333 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.333 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:03.333 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.155677+0000", 2024-04-20T17:49:03.333 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2290000000000002e-06 2024-04-20T17:49:03.333 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.333 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.333 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:03.333 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.155683+0000", 2024-04-20T17:49:03.333 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4799999999999998e-06 2024-04-20T17:49:03.333 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.333 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.333 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:03.333 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.155688+0000", 2024-04-20T17:49:03.333 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.074e-06 2024-04-20T17:49:03.334 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.334 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.334 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:03.334 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.171012+0000", 2024-04-20T17:49:03.334 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015323945 2024-04-20T17:49:03.334 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.334 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.334 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:03.334 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.171032+0000", 2024-04-20T17:49:03.334 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9808000000000001e-05 2024-04-20T17:49:03.334 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.334 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.334 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:03.335 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.212969+0000", 2024-04-20T17:49:03.335 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.041937161000000001 2024-04-20T17:49:03.335 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.335 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.335 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:03.335 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.212984+0000", 2024-04-20T17:49:03.335 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4327e-05 2024-04-20T17:49:03.335 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.335 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:03.335 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.335 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.335 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.335 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:240989 8.7 e863/806 8:e80f15e6:::benchmark_data_smithi171_175757_object95875:head v 863'14928, mlcod=796'5945)", 2024-04-20T17:49:03.335 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:47:30.033407+0000", 2024-04-20T17:49:03.335 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 93.234019043000004, 2024-04-20T17:49:03.336 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.107769315, 2024-04-20T17:49:03.336 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:03.336 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:03.336 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:03.336 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:03.336 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.336 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:03.336 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.033407+0000", 2024-04-20T17:49:03.336 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.336 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.336 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.336 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:03.336 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.033407+0000", 2024-04-20T17:49:03.336 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.336 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.337 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.337 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:03.337 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.033409+0000", 2024-04-20T17:49:03.337 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7489999999999999e-06 2024-04-20T17:49:03.337 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.337 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.337 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:03.337 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.033424+0000", 2024-04-20T17:49:03.337 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5211999999999999e-05 2024-04-20T17:49:03.337 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.337 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.337 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:03.337 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.033441+0000", 2024-04-20T17:49:03.337 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6954999999999999e-05 2024-04-20T17:49:03.337 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.338 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.338 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:03.338 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.033454+0000", 2024-04-20T17:49:03.338 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3052e-05 2024-04-20T17:49:03.338 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.338 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.338 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:03.338 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.033515+0000", 2024-04-20T17:49:03.338 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0470999999999998e-05 2024-04-20T17:49:03.339 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.339 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.339 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:03.339 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.033538+0000", 2024-04-20T17:49:03.339 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3448999999999999e-05 2024-04-20T17:49:03.339 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.339 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.339 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:03.339 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.041571+0000", 2024-04-20T17:49:03.339 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0080328229999999997 2024-04-20T17:49:03.339 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.339 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.339 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:03.339 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.141177+0000", 2024-04-20T17:49:03.340 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.099605604 2024-04-20T17:49:03.340 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.340 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:03.340 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.340 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.340 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.340 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDPGPush(8.7 863/806 [PushOp(8:ecaaf859:::benchmark_data_smithi171_175757_object95918:head, version: 796'5950, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:ecaaf859:::benchmark_data_smithi171_175757_object95918:head@796'5950, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-20T17:49:03.340 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:47:30.043161+0000", 2024-04-20T17:49:03.340 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 93.224265826999996, 2024-04-20T17:49:03.340 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10206235600000001, 2024-04-20T17:49:03.340 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:03.340 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:03.340 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:49:03.341 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:03.341 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.341 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:03.341 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.043161+0000", 2024-04-20T17:49:03.341 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.341 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.341 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.341 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:03.341 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.043161+0000", 2024-04-20T17:49:03.341 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.341 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.341 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.341 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:03.341 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.043162+0000", 2024-04-20T17:49:03.341 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5230000000000001e-06 2024-04-20T17:49:03.342 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.342 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.342 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:03.342 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.043191+0000", 2024-04-20T17:49:03.342 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9196e-05 2024-04-20T17:49:03.342 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.342 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.342 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:03.342 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.141148+0000", 2024-04-20T17:49:03.342 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.097956399999999999 2024-04-20T17:49:03.342 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.342 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.342 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:03.342 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.141234+0000", 2024-04-20T17:49:03.342 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6267999999999995e-05 2024-04-20T17:49:03.343 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.343 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.343 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:03.343 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.141327+0000", 2024-04-20T17:49:03.343 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.3010000000000006e-05 2024-04-20T17:49:03.343 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.343 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.343 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:03.343 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.141341+0000", 2024-04-20T17:49:03.343 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4246e-05 2024-04-20T17:49:03.343 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.343 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.343 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:03.343 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.145223+0000", 2024-04-20T17:49:03.344 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0038817130000000002 2024-04-20T17:49:03.344 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.344 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:03.344 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.344 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.344 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.344 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDPGPush(8.1 863/806 [PushOp(8:8f0cf38b:::benchmark_data_smithi171_175757_object95988:head, version: 796'5997, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:8f0cf38b:::benchmark_data_smithi171_175757_object95988:head@796'5997, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-20T17:49:03.344 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:47:30.051094+0000", 2024-04-20T17:49:03.344 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 93.216332820000005, 2024-04-20T17:49:03.344 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.093414170000000005, 2024-04-20T17:49:03.344 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:03.344 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:03.344 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:49:03.344 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:03.344 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.345 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:03.345 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.051094+0000", 2024-04-20T17:49:03.345 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.345 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.345 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.345 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:03.345 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.051094+0000", 2024-04-20T17:49:03.345 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.345 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.345 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.345 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:03.345 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.051096+0000", 2024-04-20T17:49:03.345 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3070000000000001e-06 2024-04-20T17:49:03.345 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.345 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.346 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:03.346 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.051177+0000", 2024-04-20T17:49:03.346 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1240000000000001e-05 2024-04-20T17:49:03.346 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.346 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.346 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:03.346 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.141152+0000", 2024-04-20T17:49:03.346 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.089974738999999998 2024-04-20T17:49:03.346 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.346 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.346 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:03.346 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.141233+0000", 2024-04-20T17:49:03.346 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0616000000000002e-05 2024-04-20T17:49:03.346 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.347 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.347 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:03.347 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.141332+0000", 2024-04-20T17:49:03.347 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9352999999999996e-05 2024-04-20T17:49:03.347 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.347 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.347 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:03.347 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.141345+0000", 2024-04-20T17:49:03.347 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.346e-05 2024-04-20T17:49:03.347 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.347 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.347 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:03.347 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.144508+0000", 2024-04-20T17:49:03.348 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0031624550000000002 2024-04-20T17:49:03.348 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.348 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:03.348 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.348 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.348 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.348 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:283186 8.3 e881/806 8:c4dc4640:::benchmark_data_smithi171_175757_object138072:head v 881'17694, mlcod=805'8780)", 2024-04-20T17:49:03.348 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:48:18.242274+0000", 2024-04-20T17:49:03.348 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 45.025152587000001, 2024-04-20T17:49:03.348 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.056183654999999999, 2024-04-20T17:49:03.348 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:03.348 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:03.348 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:03.348 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:03.348 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.349 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:03.349 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:18.242274+0000", 2024-04-20T17:49:03.349 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.349 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.349 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.349 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:03.349 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:18.242274+0000", 2024-04-20T17:49:03.349 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:03.349 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.349 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.349 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:03.349 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:18.242276+0000", 2024-04-20T17:49:03.349 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.815e-06 2024-04-20T17:49:03.349 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.349 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.350 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:03.350 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:18.242298+0000", 2024-04-20T17:49:03.350 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2166000000000001e-05 2024-04-20T17:49:03.350 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.350 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.350 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:03.350 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:18.242319+0000", 2024-04-20T17:49:03.350 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1067e-05 2024-04-20T17:49:03.350 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.350 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.350 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:03.350 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:18.242329+0000", 2024-04-20T17:49:03.350 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8560000000000002e-06 2024-04-20T17:49:03.350 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.350 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.351 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:03.351 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:18.245012+0000", 2024-04-20T17:49:03.351 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002683497 2024-04-20T17:49:03.351 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.351 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.351 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:03.351 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:18.245037+0000", 2024-04-20T17:49:03.351 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4702000000000002e-05 2024-04-20T17:49:03.351 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.351 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.351 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:03.351 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:18.298437+0000", 2024-04-20T17:49:03.351 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.053400446999999997 2024-04-20T17:49:03.351 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:03.351 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:03.352 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:03.352 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:18.298458+0000", 2024-04-20T17:49:03.352 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0105000000000001e-05 2024-04-20T17:49:03.352 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.352 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:03.352 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.352 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:03.352 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:03.352 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:49:03.352 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:02 smithi171 ceph-mon[44189]: pgmap v2376: 27 pgs: 27 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:49:03.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:03.354 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:03.354 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:49:03.354 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:49:03.455 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:49:03.489 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:03.511 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:49:03.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:03.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:49:03.615 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:49:03.647 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:03.669 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:49:03.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:03.672 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:49:03.773 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:49:03.804 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:03.858 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:49:03.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:03.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:49:03.909 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:49:03.960 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.8 to 1 (10000) 2024-04-20T17:49:03.963 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:49:03.989 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:03 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4037009762' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-20T17:49:04.193 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:49:04.195 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:03 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4037009762' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-20T17:49:04.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:04.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:49:04.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:03 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4037009762' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-20T17:49:04.297 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:49:04.329 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:04.351 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:49:04.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:04.355 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:49:04.455 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:49:04.484 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:04.507 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:49:04.756 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:49:04.791 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-20T17:49:04.858 INFO:teuthology.orchestra.run.smithi005.stderr:enabled application 'rados' on pool 'unique_pool_7' 2024-04-20T17:49:05.026 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:04 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4037009762' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-20T17:49:05.026 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:04 smithi005 ceph-mon[39562]: osdmap e899: 12 total, 12 up, 12 in 2024-04-20T17:49:05.026 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:04 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:49:05.026 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:04 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:49:05.026 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:04 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3507731483' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:49:05.027 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:04 smithi005 ceph-mon[39562]: pgmap v2378: 27 pgs: 27 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:49:05.033 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:49:05.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:05.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:49:05.067 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:49:05.138 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:49:05.168 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:04 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4037009762' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-20T17:49:05.169 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:04 smithi171 ceph-mon[44189]: osdmap e899: 12 total, 12 up, 12 in 2024-04-20T17:49:05.169 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:04 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:49:05.169 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:04 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:49:05.169 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:04 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3507731483' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:49:05.169 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:04 smithi171 ceph-mon[44189]: pgmap v2378: 27 pgs: 27 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:49:05.171 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:05.192 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:49:05.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:05.195 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:49:05.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:04 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4037009762' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-20T17:49:05.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:04 smithi106 ceph-mon[44423]: osdmap e899: 12 total, 12 up, 12 in 2024-04-20T17:49:05.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:04 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:49:05.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:04 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:49:05.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:04 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3507731483' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-20T17:49:05.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:04 smithi106 ceph-mon[44423]: pgmap v2378: 27 pgs: 27 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:49:05.296 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:49:05.324 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:05.345 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:49:05.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:05.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:49:05.450 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:49:05.476 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:05.824 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:49:05.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:05.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:49:05.928 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:49:05.956 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:05.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:05 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3507731483' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-20T17:49:05.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:05 smithi005 ceph-mon[39562]: osdmap e900: 12 total, 12 up, 12 in 2024-04-20T17:49:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:05 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3507731483' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-20T17:49:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:05 smithi106 ceph-mon[44423]: osdmap e900: 12 total, 12 up, 12 in 2024-04-20T17:49:06.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:05 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3507731483' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-20T17:49:06.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:05 smithi171 ceph-mon[44189]: osdmap e900: 12 total, 12 up, 12 in 2024-04-20T17:49:06.405 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:49:06.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:06.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:49:06.509 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:49:06.541 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:06.562 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:49:06.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:06.565 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:49:06.666 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:49:06.697 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:06.717 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:49:06.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:06.721 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:49:06.736 DEBUG:teuthology.orchestra.run.smithi171:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_7 bench 90 write' 2024-04-20T17:49:06.764 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-20T17:49:06.764 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-20T17:49:06.822 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:49:06.848 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:06.857 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:hints = 1 2024-04-20T17:49:06.857 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-20T17:49:06.857 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Object prefix: benchmark_data_smithi171_188319 2024-04-20T17:49:06.857 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:49:06.858 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 0 0 0 0 0 0 - 0 2024-04-20T17:49:06.875 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:49:06.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:06.879 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:49:06.980 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:49:07.027 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:07.052 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:49:07.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:07.058 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:49:07.079 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:49:07.080 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:49:07.080 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.080 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:1211 9.e 9:7dc74cf1:::benchmark_data_smithi171_188319_object1210:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e900)", 2024-04-20T17:49:07.080 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:07.069263+0000", 2024-04-20T17:49:07.080 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.002158639, 2024-04-20T17:49:07.081 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0021932029999999999, 2024-04-20T17:49:07.081 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:07.081 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:07.081 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:49:07.081 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:49:07.081 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:49:07.082 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:49:07.082 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 1211 2024-04-20T17:49:07.082 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.082 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:07.082 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.082 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:07.083 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.069263+0000", 2024-04-20T17:49:07.083 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:07.083 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.083 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.083 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:07.083 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.069263+0000", 2024-04-20T17:49:07.083 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:07.084 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.084 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.084 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:07.084 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.069264+0000", 2024-04-20T17:49:07.084 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5090000000000001e-06 2024-04-20T17:49:07.084 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.084 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.084 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:07.084 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.069293+0000", 2024-04-20T17:49:07.084 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9213e-05 2024-04-20T17:49:07.084 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.084 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.084 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:07.084 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.069302+0000", 2024-04-20T17:49:07.085 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1680000000000004e-06 2024-04-20T17:49:07.085 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.085 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.085 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:07.085 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.069308+0000", 2024-04-20T17:49:07.085 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5320000000000003e-06 2024-04-20T17:49:07.085 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.085 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.085 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:07.085 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.069532+0000", 2024-04-20T17:49:07.085 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000223408 2024-04-20T17:49:07.085 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.085 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.085 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:07.085 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.069648+0000", 2024-04-20T17:49:07.086 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000116112 2024-04-20T17:49:07.086 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.086 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.086 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:49:07.086 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.069796+0000", 2024-04-20T17:49:07.086 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000148725 2024-04-20T17:49:07.086 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.086 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.086 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:49:07.086 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.070538+0000", 2024-04-20T17:49:07.086 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00074161300000000002 2024-04-20T17:49:07.086 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:07.086 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:07.086 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:07.086 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.087 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.087 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:1220 9.d 9:b76a119a:::benchmark_data_smithi171_188319_object1219:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e900)", 2024-04-20T17:49:07.087 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:07.070833+0000", 2024-04-20T17:49:07.087 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.00058880499999999997, 2024-04-20T17:49:07.087 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00090414, 2024-04-20T17:49:07.087 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:07.087 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:07.087 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:49:07.087 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:49:07.087 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:49:07.087 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:49:07.087 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 1220 2024-04-20T17:49:07.087 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.087 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:07.088 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.088 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:07.088 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.070833+0000", 2024-04-20T17:49:07.088 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:07.088 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.088 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.088 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:07.088 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.070833+0000", 2024-04-20T17:49:07.088 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:07.088 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.088 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.088 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:07.089 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.070834+0000", 2024-04-20T17:49:07.089 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8300000000000001e-06 2024-04-20T17:49:07.089 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.089 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.089 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:07.089 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.070875+0000", 2024-04-20T17:49:07.089 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0769999999999998e-05 2024-04-20T17:49:07.089 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.089 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.089 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:07.089 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.070903+0000", 2024-04-20T17:49:07.089 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8121999999999999e-05 2024-04-20T17:49:07.089 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.089 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.089 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:07.090 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.070912+0000", 2024-04-20T17:49:07.090 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4870000000000002e-06 2024-04-20T17:49:07.090 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.090 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.090 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:07.090 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.071084+0000", 2024-04-20T17:49:07.090 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000171748 2024-04-20T17:49:07.090 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.090 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.090 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:07.091 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.071198+0000", 2024-04-20T17:49:07.091 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000114157 2024-04-20T17:49:07.091 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.091 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.091 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:49:07.091 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.071317+0000", 2024-04-20T17:49:07.091 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000119157 2024-04-20T17:49:07.091 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:07.091 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:07.091 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:07.091 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.091 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.092 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:1221 9.e 9:799f8931:::benchmark_data_smithi171_188319_object1220:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e900)", 2024-04-20T17:49:07.092 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:07.070930+0000", 2024-04-20T17:49:07.092 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.00049114199999999997, 2024-04-20T17:49:07.092 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00090273800000000004, 2024-04-20T17:49:07.092 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:07.092 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:07.092 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:49:07.092 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:49:07.092 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:49:07.092 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:49:07.092 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 1221 2024-04-20T17:49:07.092 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.092 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:07.092 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.093 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:07.093 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.070930+0000", 2024-04-20T17:49:07.093 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:07.093 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.093 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.093 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:07.093 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.070930+0000", 2024-04-20T17:49:07.093 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:07.093 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.093 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.093 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:07.093 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.070931+0000", 2024-04-20T17:49:07.093 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0839999999999999e-06 2024-04-20T17:49:07.093 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.093 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.094 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:07.094 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.070982+0000", 2024-04-20T17:49:07.094 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0451999999999997e-05 2024-04-20T17:49:07.094 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.094 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.094 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:07.094 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.070997+0000", 2024-04-20T17:49:07.094 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4958e-05 2024-04-20T17:49:07.094 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.094 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.094 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:07.094 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.071005+0000", 2024-04-20T17:49:07.094 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3229999999999998e-06 2024-04-20T17:49:07.094 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.095 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.095 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:07.095 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.071048+0000", 2024-04-20T17:49:07.095 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2987e-05 2024-04-20T17:49:07.095 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.095 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.095 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:07.095 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.071156+0000", 2024-04-20T17:49:07.095 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000107679 2024-04-20T17:49:07.095 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.095 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.095 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:49:07.095 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.071288+0000", 2024-04-20T17:49:07.096 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013194300000000001 2024-04-20T17:49:07.096 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:07.096 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:07.096 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:07.096 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.096 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.096 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:1225 9.e 9:7d135e29:::benchmark_data_smithi171_188319_object1224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e900)", 2024-04-20T17:49:07.096 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:07.071023+0000", 2024-04-20T17:49:07.096 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.00039819699999999998, 2024-04-20T17:49:07.096 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00089882099999999995, 2024-04-20T17:49:07.096 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:07.096 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:07.096 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-04-20T17:49:07.097 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:49:07.097 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:49:07.097 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:49:07.097 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 1225 2024-04-20T17:49:07.097 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.097 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:07.097 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.097 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:07.097 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.071023+0000", 2024-04-20T17:49:07.097 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:07.097 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.098 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.098 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:07.098 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.071023+0000", 2024-04-20T17:49:07.098 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:07.098 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.098 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.098 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:07.098 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.071024+0000", 2024-04-20T17:49:07.098 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8899999999999998e-07 2024-04-20T17:49:07.098 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.098 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.098 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:07.098 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.071096+0000", 2024-04-20T17:49:07.098 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1507999999999994e-05 2024-04-20T17:49:07.099 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.099 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.099 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:07.099 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.071106+0000", 2024-04-20T17:49:07.099 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9320000000000008e-06 2024-04-20T17:49:07.099 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.099 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.099 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:07.099 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.071113+0000", 2024-04-20T17:49:07.100 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.712e-06 2024-04-20T17:49:07.100 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:07.100 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:07.100 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:07.100 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:07.071840+0000", 2024-04-20T17:49:07.100 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00072634900000000005 2024-04-20T17:49:07.100 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:07.100 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:07.100 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:07.101 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:07.101 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-04-20T17:49:07.101 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 4 2024-04-20T17:49:07.101 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:49:07.158 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:49:07.200 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:07.519 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:49:07.550 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:49:07.580 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:49:07.606 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:49:07.641 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:49:07.672 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:49:07.705 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:49:07.735 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:49:07.761 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:49:07.791 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:49:07.823 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:49:07.856 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:49:07.857 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 1 16 5632 5616 350.983 351 0.00246887 0.00284416 2024-04-20T17:49:07.858 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:49:07.858 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:49:07.858 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:49:07.858 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:49:07.859 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:49:07.859 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:49:07.859 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:49:07.859 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:49:07.859 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:49:07.859 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:49:07.859 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:49:07.859 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:49:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:49:07.926 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 9 1 2024-04-20T17:49:07.967 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:07 smithi005 ceph-mon[39562]: pgmap v2380: 27 pgs: 27 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:49:07.967 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:07 smithi005 ceph-mon[39562]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-20T17:49:07.968 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:07 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2493715588' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:49:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:07 smithi106 ceph-mon[44423]: pgmap v2380: 27 pgs: 27 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:49:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:07 smithi106 ceph-mon[44423]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-20T17:49:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:07 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2493715588' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:49:08.058 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:49:08.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:08.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:49:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:07 smithi171 ceph-mon[44189]: pgmap v2380: 27 pgs: 27 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:49:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:07 smithi171 ceph-mon[44189]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-20T17:49:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:07 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2493715588' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:49:08.163 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:49:08.202 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:08.858 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 2 16 10681 10665 333.259 315.562 0.00172152 0.00299762 2024-04-20T17:49:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:08 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:49:09.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:08 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:49:09.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:08 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:08 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:49:09.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:08 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:49:09.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:08 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:08 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:49:09.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:08 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:49:09.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:08 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:09.753 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-20T17:49:09.763 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:49:09.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:09.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:49:09.811 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:49:09.858 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 3 16 15116 15100 314.562 277.188 0.00461808 0.00317509 2024-04-20T17:49:09.869 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:49:09.907 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:09 smithi106 ceph-mon[44423]: pgmap v2381: 27 pgs: 27 active+clean; 9.0 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s wr, 19 op/s 2024-04-20T17:49:10.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:09 smithi106 ceph-mon[44423]: osdmap e901: 12 total, 12 up, 12 in 2024-04-20T17:49:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:09 smithi171 ceph-mon[44189]: pgmap v2381: 27 pgs: 27 active+clean; 9.0 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s wr, 19 op/s 2024-04-20T17:49:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:09 smithi171 ceph-mon[44189]: osdmap e901: 12 total, 12 up, 12 in 2024-04-20T17:49:10.096 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:09 smithi005 ceph-mon[39562]: pgmap v2381: 27 pgs: 27 active+clean; 9.0 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s wr, 19 op/s 2024-04-20T17:49:10.096 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:09 smithi005 ceph-mon[39562]: osdmap e901: 12 total, 12 up, 12 in 2024-04-20T17:49:10.164 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:49:10.857 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:49:10.963 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 4 16 19498 19482 304.385 273.875 0.00531967 0.00328326 2024-04-20T17:49:11.002 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:10 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:49:11.003 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:10 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:49:11.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:10 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:49:11.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:10 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:49:11.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:10 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:49:11.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:10 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:49:11.858 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 5 16 23077 23061 288.241 223.688 0.0039056 0.00346748 2024-04-20T17:49:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:11 smithi106 ceph-mon[44423]: pgmap v2383: 27 pgs: 27 active+clean; 515 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s 2024-04-20T17:49:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:11 smithi106 ceph-mon[44423]: osdmap e902: 12 total, 12 up, 12 in 2024-04-20T17:49:12.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:11 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3890845955' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:49:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:11 smithi171 ceph-mon[44189]: pgmap v2383: 27 pgs: 27 active+clean; 515 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s 2024-04-20T17:49:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:11 smithi171 ceph-mon[44189]: osdmap e902: 12 total, 12 up, 12 in 2024-04-20T17:49:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:11 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3890845955' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:49:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:11 smithi005 ceph-mon[39562]: pgmap v2383: 27 pgs: 27 active+clean; 515 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s 2024-04-20T17:49:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:11 smithi005 ceph-mon[39562]: osdmap e902: 12 total, 12 up, 12 in 2024-04-20T17:49:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:11 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3890845955' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:49:12.858 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 6 16 25809 25793 268.655 170.75 0.00190697 0.0036778 2024-04-20T17:49:12.994 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:12 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:49:12.995 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:12 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3890845955' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:49:12.995 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:12 smithi005 ceph-mon[39562]: osdmap e903: 12 total, 12 up, 12 in 2024-04-20T17:49:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:12 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:49:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:12 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3890845955' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:49:13.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:12 smithi106 ceph-mon[44423]: osdmap e903: 12 total, 12 up, 12 in 2024-04-20T17:49:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:12 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:49:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:12 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3890845955' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:49:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:12 smithi171 ceph-mon[44189]: osdmap e903: 12 total, 12 up, 12 in 2024-04-20T17:49:13.464 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:49:13.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:13.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:49:13.569 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:49:13.570 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:49:13.609 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:13.673 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:49:13.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:13.678 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:49:13.779 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:49:13.819 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:13.858 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 7 16 29534 29518 263.531 232.812 0.00549457 0.00379253 2024-04-20T17:49:13.898 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:49:13.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:13.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:49:14.004 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:49:14.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:13 smithi106 ceph-mon[44423]: pgmap v2386: 27 pgs: 27 active+clean; 1.1 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 181 MiB/s wr, 2.90k op/s 2024-04-20T17:49:14.036 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:13 smithi005 ceph-mon[39562]: pgmap v2386: 27 pgs: 27 active+clean; 1.1 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 181 MiB/s wr, 2.90k op/s 2024-04-20T17:49:14.038 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:14.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:13 smithi171 ceph-mon[44189]: pgmap v2386: 27 pgs: 27 active+clean; 1.1 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 181 MiB/s wr, 2.90k op/s 2024-04-20T17:49:14.148 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:49:14.148 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:49:14.851 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:49:14.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:14.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:49:14.858 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 8 16 32879 32863 256.72 209.062 0.00432104 0.00389365 2024-04-20T17:49:14.897 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:49:14.956 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:49:15.167 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:15.858 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 9 16 35968 35952 249.647 193.062 0.00459883 0.00400309 2024-04-20T17:49:16.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:15 smithi106 ceph-mon[44423]: pgmap v2387: 27 pgs: 27 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 229 MiB/s wr, 3.67k op/s 2024-04-20T17:49:16.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:15 smithi171 ceph-mon[44189]: pgmap v2387: 27 pgs: 27 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 229 MiB/s wr, 3.67k op/s 2024-04-20T17:49:16.107 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-20T17:49:16.124 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:49:16.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:16.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:49:16.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:15 smithi005 ceph-mon[39562]: pgmap v2387: 27 pgs: 27 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 229 MiB/s wr, 3.67k op/s 2024-04-20T17:49:16.230 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:49:16.271 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:16.297 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:49:16.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:16.301 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:49:16.402 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:49:16.438 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:16.465 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:49:16.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:16.470 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:49:16.511 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:49:16.571 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:49:16.617 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:16.724 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.9 to 1 (10000) 2024-04-20T17:49:16.809 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:49:16.858 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 10 16 38855 38839 242.724 180.438 0.00414556 0.00411795 2024-04-20T17:49:16.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:16 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/80844150' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:49:16.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:16 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/80844150' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-20T17:49:16.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:16 smithi005 ceph-mon[39562]: osdmap e904: 12 total, 12 up, 12 in 2024-04-20T17:49:17.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:16 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/80844150' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:49:17.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:16 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/80844150' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-20T17:49:17.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:16 smithi106 ceph-mon[44423]: osdmap e904: 12 total, 12 up, 12 in 2024-04-20T17:49:17.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:16 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/80844150' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:49:17.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:16 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/80844150' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-20T17:49:17.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:16 smithi171 ceph-mon[44189]: osdmap e904: 12 total, 12 up, 12 in 2024-04-20T17:49:17.858 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 11 15 41749 41734 237.106 180.938 0.00251815 0.00421415 2024-04-20T17:49:17.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:17 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:49:17.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:17 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:49:17.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:17 smithi005 ceph-mon[39562]: pgmap v2388: 27 pgs: 27 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 249 MiB/s wr, 3.98k op/s 2024-04-20T17:49:18.080 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:49:18.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:18.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:49:18.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:17 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:49:18.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:17 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:49:18.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:17 smithi171 ceph-mon[44189]: pgmap v2388: 27 pgs: 27 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 249 MiB/s wr, 3.98k op/s 2024-04-20T17:49:18.184 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:49:18.241 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:17 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:49:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:17 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:49:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:17 smithi106 ceph-mon[44423]: pgmap v2388: 27 pgs: 27 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 249 MiB/s wr, 3.98k op/s 2024-04-20T17:49:18.269 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:49:18.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:18.275 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:49:18.376 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:49:18.414 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:18.519 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:49:18.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:18.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:49:18.623 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:49:18.657 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:18.690 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:49:18.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:18.696 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:49:18.798 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:49:18.835 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:18.859 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 12 16 44193 44177 230.07 152.688 0.00332065 0.0043439 2024-04-20T17:49:19.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:18 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/419970530' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:49:19.049 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:18 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/419970530' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:49:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:18 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/419970530' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:49:19.090 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 10 1 2024-04-20T17:49:19.490 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:49:19.490 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:49:19.490 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:49:19.490 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:49:19.490 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:49:19.490 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:49:19.490 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:49:19.490 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:49:19.490 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:49:19.490 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:49:19.490 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:49:19.490 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:49:19.491 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:49:19.491 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:49:19.491 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:49:19.491 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:49:19.491 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:49:19.491 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:49:19.491 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:49:19.491 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:49:19.491 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:49:19.491 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:49:19.491 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:49:19.491 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:49:19.592 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:49:19.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:19.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:49:19.697 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:49:19.729 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:19.759 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:49:19.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:19.766 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:49:19.859 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 13 15 46886 46871 225.323 168.375 0.0048373 0.00443632 2024-04-20T17:49:19.867 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:49:19.902 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:19 smithi106 ceph-mon[44423]: pgmap v2390: 27 pgs: 27 active+clean; 2.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 224 MiB/s wr, 3.58k op/s 2024-04-20T17:49:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:19.903 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:19.904 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:19.963 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:49:19.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:19.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:49:20.069 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:49:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:20.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:20.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:19 smithi171 ceph-mon[44189]: pgmap v2390: 27 pgs: 27 active+clean; 2.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 224 MiB/s wr, 3.58k op/s 2024-04-20T17:49:20.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:20.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:20.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:20.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:20.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:20.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:20.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:20.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:20.103 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:20.103 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:20.103 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:20.104 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:19 smithi005 ceph-mon[39562]: pgmap v2390: 27 pgs: 27 active+clean; 2.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 224 MiB/s wr, 3.58k op/s 2024-04-20T17:49:20.104 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:20.104 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:20.104 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:20.104 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:20.105 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:20.105 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:20.105 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:20.105 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:20.106 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:20.166 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:49:20.181 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:49:20.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:20.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:49:20.286 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:49:20.326 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:20.356 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:49:20.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:20.360 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:49:20.393 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:49:20.393 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-04-20T17:49:20.393 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-04-20T17:49:20.393 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:49:20.393 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.393 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27460.0:278165 8.d e879/855 8:b998616d:::benchmark_data_smithi171_175757_object133051:head v 879'17481, mlcod=879'17477)", 2024-04-20T17:49:20.393 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:48:13.176158+0000", 2024-04-20T17:49:20.393 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 67.214014786999996, 2024-04-20T17:49:20.393 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010872010999999999, 2024-04-20T17:49:20.393 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:20.394 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:20.394 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:20.394 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:20.394 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.394 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:20.394 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:48:13.176158+0000", 2024-04-20T17:49:20.394 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.394 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.394 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.394 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:20.394 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:48:13.176158+0000", 2024-04-20T17:49:20.394 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.394 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.395 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.395 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:20.395 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:48:13.176158+0000", 2024-04-20T17:49:20.395 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5200000000000002e-07 2024-04-20T17:49:20.395 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.395 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.395 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:20.395 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:48:13.176162+0000", 2024-04-20T17:49:20.395 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.27e-06 2024-04-20T17:49:20.395 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.395 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.396 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:20.396 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:48:13.176166+0000", 2024-04-20T17:49:20.396 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.771e-06 2024-04-20T17:49:20.396 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.396 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.396 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:20.396 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:48:13.176169+0000", 2024-04-20T17:49:20.396 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0199999999999999e-06 2024-04-20T17:49:20.396 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.396 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.396 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:20.396 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:48:13.183493+0000", 2024-04-20T17:49:20.396 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0073245890000000003 2024-04-20T17:49:20.397 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.397 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.397 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:20.397 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:48:13.183510+0000", 2024-04-20T17:49:20.397 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6671999999999999e-05 2024-04-20T17:49:20.397 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.397 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.397 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:49:20.397 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:48:13.187009+0000", 2024-04-20T17:49:20.397 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0034996390000000001 2024-04-20T17:49:20.397 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.397 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.397 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:49:20.398 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:48:13.187030+0000", 2024-04-20T17:49:20.398 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0598000000000002e-05 2024-04-20T17:49:20.398 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.398 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:20.398 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.398 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.398 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.398 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:44278 9.d 9:b63ada4a:::benchmark_data_smithi171_188319_object44277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e904)", 2024-04-20T17:49:20.398 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:18.887252+0000", 2024-04-20T17:49:20.398 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 1.5029203949999999, 2024-04-20T17:49:20.398 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.03104061, 2024-04-20T17:49:20.398 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:20.398 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:20.398 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:20.399 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:49:20.399 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:49:20.399 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:49:20.399 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 44278 2024-04-20T17:49:20.399 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.399 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:20.399 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.399 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:20.399 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.887252+0000", 2024-04-20T17:49:20.399 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.399 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.399 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.399 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:20.399 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.887252+0000", 2024-04-20T17:49:20.399 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.400 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.400 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.400 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:20.400 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.887253+0000", 2024-04-20T17:49:20.400 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.018e-06 2024-04-20T17:49:20.400 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.400 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.400 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:20.400 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.887346+0000", 2024-04-20T17:49:20.400 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2891000000000002e-05 2024-04-20T17:49:20.400 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.400 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.400 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:20.400 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.887355+0000", 2024-04-20T17:49:20.401 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.081e-06 2024-04-20T17:49:20.401 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.401 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.401 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:20.401 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.887363+0000", 2024-04-20T17:49:20.401 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2660000000000004e-06 2024-04-20T17:49:20.401 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.401 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.401 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:20.401 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.888448+0000", 2024-04-20T17:49:20.401 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010849709999999999 2024-04-20T17:49:20.401 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.401 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.401 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:20.401 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.888516+0000", 2024-04-20T17:49:20.402 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8498999999999996e-05 2024-04-20T17:49:20.402 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.402 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.402 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:49:20.402 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.888609+0000", 2024-04-20T17:49:20.402 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2643000000000001e-05 2024-04-20T17:49:20.402 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.402 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.402 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:49:20.402 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.889677+0000", 2024-04-20T17:49:20.402 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010682160000000001 2024-04-20T17:49:20.402 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.402 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.402 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:20.403 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.891417+0000", 2024-04-20T17:49:20.403 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001739738 2024-04-20T17:49:20.403 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.403 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.403 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:49:20.403 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.918258+0000", 2024-04-20T17:49:20.403 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.026840823 2024-04-20T17:49:20.403 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.403 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.403 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:49:20.403 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.918293+0000", 2024-04-20T17:49:20.403 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5463999999999998e-05 2024-04-20T17:49:20.403 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.403 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:20.403 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.404 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.404 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.404 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:44285 9.d 9:ba81a6ff:::benchmark_data_smithi171_188319_object44284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e904)", 2024-04-20T17:49:20.404 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:18.889806+0000", 2024-04-20T17:49:20.404 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 1.500366511, 2024-04-20T17:49:20.404 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.028679389, 2024-04-20T17:49:20.404 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:20.404 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:20.404 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:20.404 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:49:20.404 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:49:20.404 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:49:20.404 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 44285 2024-04-20T17:49:20.404 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.404 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:20.405 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.405 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:20.405 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.889806+0000", 2024-04-20T17:49:20.405 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.405 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.405 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.405 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:20.405 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.889806+0000", 2024-04-20T17:49:20.405 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.405 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.405 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.405 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:20.405 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.889808+0000", 2024-04-20T17:49:20.405 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.849e-06 2024-04-20T17:49:20.406 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.406 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.406 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:20.406 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.889867+0000", 2024-04-20T17:49:20.406 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9001e-05 2024-04-20T17:49:20.406 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.406 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.406 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:20.406 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.889876+0000", 2024-04-20T17:49:20.406 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1530000000000003e-06 2024-04-20T17:49:20.406 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.406 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.406 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:20.406 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.889884+0000", 2024-04-20T17:49:20.406 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5549999999999997e-06 2024-04-20T17:49:20.406 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.407 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.407 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:20.407 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.889924+0000", 2024-04-20T17:49:20.407 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0531000000000002e-05 2024-04-20T17:49:20.407 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.407 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.407 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:20.407 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.890025+0000", 2024-04-20T17:49:20.407 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000101054 2024-04-20T17:49:20.407 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.407 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.407 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:49:20.407 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.890103+0000", 2024-04-20T17:49:20.407 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7508000000000004e-05 2024-04-20T17:49:20.408 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.408 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.408 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:49:20.408 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.891253+0000", 2024-04-20T17:49:20.408 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0011504270000000001 2024-04-20T17:49:20.408 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.408 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.408 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:20.408 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.918360+0000", 2024-04-20T17:49:20.408 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.027106179000000001 2024-04-20T17:49:20.408 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.408 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.408 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:49:20.408 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.918441+0000", 2024-04-20T17:49:20.408 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1520999999999994e-05 2024-04-20T17:49:20.409 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.409 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.409 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:49:20.409 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.918486+0000", 2024-04-20T17:49:20.409 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4610999999999997e-05 2024-04-20T17:49:20.409 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.409 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:20.409 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.409 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.409 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.409 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:44286 9.e 9:734feafd:::benchmark_data_smithi171_188319_object44285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e904)", 2024-04-20T17:49:20.409 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:18.890152+0000", 2024-04-20T17:49:20.409 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 1.5000213060000001, 2024-04-20T17:49:20.409 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.028207256999999999, 2024-04-20T17:49:20.409 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:20.410 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:20.410 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:20.410 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:49:20.410 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:49:20.410 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:49:20.410 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 44286 2024-04-20T17:49:20.410 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.410 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:20.410 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.410 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:20.410 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.890152+0000", 2024-04-20T17:49:20.410 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.410 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.411 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.411 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:20.411 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.890152+0000", 2024-04-20T17:49:20.411 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.411 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.411 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.411 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:20.411 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.890153+0000", 2024-04-20T17:49:20.411 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.082e-06 2024-04-20T17:49:20.411 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.411 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.412 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:20.412 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.890225+0000", 2024-04-20T17:49:20.412 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2317999999999994e-05 2024-04-20T17:49:20.412 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.412 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.412 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:20.412 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.890235+0000", 2024-04-20T17:49:20.412 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0404999999999999e-05 2024-04-20T17:49:20.412 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.412 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.412 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:20.412 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.890242+0000", 2024-04-20T17:49:20.412 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.08e-06 2024-04-20T17:49:20.412 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.412 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.413 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:20.413 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.890288+0000", 2024-04-20T17:49:20.413 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5633000000000002e-05 2024-04-20T17:49:20.413 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.413 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.413 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:20.413 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.890389+0000", 2024-04-20T17:49:20.413 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000101168 2024-04-20T17:49:20.413 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.413 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.413 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:49:20.414 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.890506+0000", 2024-04-20T17:49:20.414 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000116552 2024-04-20T17:49:20.414 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.414 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.414 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:20.414 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.918285+0000", 2024-04-20T17:49:20.414 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.027779162999999999 2024-04-20T17:49:20.414 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.414 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.414 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:49:20.414 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.918293+0000", 2024-04-20T17:49:20.414 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6490000000000008e-06 2024-04-20T17:49:20.414 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.414 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.414 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:49:20.415 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.918320+0000", 2024-04-20T17:49:20.415 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7665e-05 2024-04-20T17:49:20.415 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.415 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.415 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:49:20.415 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.918359+0000", 2024-04-20T17:49:20.415 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8541999999999998e-05 2024-04-20T17:49:20.415 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.415 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:20.417 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.417 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.417 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.417 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.27772.0:44278 9.d e904/894 ondisk, result = 0)", 2024-04-20T17:49:20.417 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:18.891287+0000", 2024-04-20T17:49:20.417 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 1.4988860050000001, 2024-04-20T17:49:20.417 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.027036484, 2024-04-20T17:49:20.417 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:20.418 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:20.418 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-04-20T17:49:20.418 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:20.418 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.418 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:20.418 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.891287+0000", 2024-04-20T17:49:20.418 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.418 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.418 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.418 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:20.419 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.891287+0000", 2024-04-20T17:49:20.419 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.419 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.419 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.419 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:20.419 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.891287+0000", 2024-04-20T17:49:20.419 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0100000000000005e-07 2024-04-20T17:49:20.419 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.419 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.419 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:20.419 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.891290+0000", 2024-04-20T17:49:20.419 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2340000000000001e-06 2024-04-20T17:49:20.419 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.419 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.420 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:20.420 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.891294+0000", 2024-04-20T17:49:20.420 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6840000000000004e-06 2024-04-20T17:49:20.420 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.420 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.420 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:20.420 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.891299+0000", 2024-04-20T17:49:20.420 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5519999999999998e-06 2024-04-20T17:49:20.420 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.420 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.420 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:20.420 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.891405+0000", 2024-04-20T17:49:20.420 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000106646 2024-04-20T17:49:20.420 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.420 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.421 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:20.421 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.891412+0000", 2024-04-20T17:49:20.421 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.455e-06 2024-04-20T17:49:20.421 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.421 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.421 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:49:20.421 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.918323+0000", 2024-04-20T17:49:20.421 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.026911411999999999 2024-04-20T17:49:20.421 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.421 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:20.421 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.421 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.421 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.421 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.27772.0:44285 9.d e904/894 ondisk, result = 0)", 2024-04-20T17:49:20.422 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:18.891491+0000", 2024-04-20T17:49:20.422 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 1.4986819470000001, 2024-04-20T17:49:20.422 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.027001615, 2024-04-20T17:49:20.422 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:20.422 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:20.422 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-04-20T17:49:20.422 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:20.422 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.422 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:20.422 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.891491+0000", 2024-04-20T17:49:20.422 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.422 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.422 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.422 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:20.423 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.891491+0000", 2024-04-20T17:49:20.423 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.423 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.423 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.423 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:20.423 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.891492+0000", 2024-04-20T17:49:20.423 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1429999999999999e-06 2024-04-20T17:49:20.423 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.423 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.423 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:20.423 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.891497+0000", 2024-04-20T17:49:20.423 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6480000000000002e-06 2024-04-20T17:49:20.423 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.423 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.424 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:20.424 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.918215+0000", 2024-04-20T17:49:20.424 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.026718518 2024-04-20T17:49:20.424 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.424 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.424 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:20.424 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.918226+0000", 2024-04-20T17:49:20.424 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.049e-05 2024-04-20T17:49:20.424 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.424 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.424 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:20.424 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.918337+0000", 2024-04-20T17:49:20.424 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000111061 2024-04-20T17:49:20.424 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.425 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.425 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:20.425 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.918344+0000", 2024-04-20T17:49:20.425 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2749999999999998e-06 2024-04-20T17:49:20.425 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.425 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.425 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:49:20.425 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.918493+0000", 2024-04-20T17:49:20.425 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014847999999999999 2024-04-20T17:49:20.425 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.425 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:20.425 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.425 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.425 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.426 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.27772.0:44286 9.e e904/894 ondisk, result = 0)", 2024-04-20T17:49:20.426 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:18.892055+0000", 2024-04-20T17:49:20.426 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 1.4981173619999999, 2024-04-20T17:49:20.426 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.026306428999999999, 2024-04-20T17:49:20.426 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:20.426 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:20.426 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-04-20T17:49:20.426 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:20.426 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.426 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:20.426 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.892055+0000", 2024-04-20T17:49:20.426 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.426 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.426 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.427 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:20.427 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.892055+0000", 2024-04-20T17:49:20.427 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.427 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.427 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.427 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:20.427 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.892057+0000", 2024-04-20T17:49:20.427 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3349999999999999e-06 2024-04-20T17:49:20.427 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.427 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.427 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:20.427 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.892065+0000", 2024-04-20T17:49:20.427 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8259999999999993e-06 2024-04-20T17:49:20.427 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.427 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.428 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:20.428 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.918211+0000", 2024-04-20T17:49:20.428 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.026146117 2024-04-20T17:49:20.428 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.428 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.428 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:20.428 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.918225+0000", 2024-04-20T17:49:20.428 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4515000000000001e-05 2024-04-20T17:49:20.428 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.428 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.428 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:20.428 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.918275+0000", 2024-04-20T17:49:20.428 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9243999999999998e-05 2024-04-20T17:49:20.428 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.428 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.429 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:20.429 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.918281+0000", 2024-04-20T17:49:20.429 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0900000000000001e-06 2024-04-20T17:49:20.429 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.429 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.429 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:49:20.429 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.918362+0000", 2024-04-20T17:49:20.429 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1302000000000002e-05 2024-04-20T17:49:20.429 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.429 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:20.429 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.429 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.429 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.430 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:44290 9.e 9:767709a4:::benchmark_data_smithi171_188319_object44289:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e904)", 2024-04-20T17:49:20.430 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:18.892537+0000", 2024-04-20T17:49:20.430 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 1.497635796, 2024-04-20T17:49:20.430 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.031510943999999999, 2024-04-20T17:49:20.430 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:20.430 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:20.430 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:20.430 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:49:20.430 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:49:20.430 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:49:20.430 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 44290 2024-04-20T17:49:20.430 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.430 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:20.430 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.431 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:20.431 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.892537+0000", 2024-04-20T17:49:20.431 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.431 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.431 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.431 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:20.431 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.892537+0000", 2024-04-20T17:49:20.431 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.431 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.431 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.431 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:20.431 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.892541+0000", 2024-04-20T17:49:20.431 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6069999999999999e-06 2024-04-20T17:49:20.431 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.432 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.432 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:20.432 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.892683+0000", 2024-04-20T17:49:20.432 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000142433 2024-04-20T17:49:20.432 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.432 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.432 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:20.432 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.918230+0000", 2024-04-20T17:49:20.432 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.025546715000000001 2024-04-20T17:49:20.432 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.432 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.432 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:20.432 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.918242+0000", 2024-04-20T17:49:20.432 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2585999999999999e-05 2024-04-20T17:49:20.432 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.433 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.433 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:20.433 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.918384+0000", 2024-04-20T17:49:20.433 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000142013 2024-04-20T17:49:20.433 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.433 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.433 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:20.433 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.918510+0000", 2024-04-20T17:49:20.433 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012606100000000001 2024-04-20T17:49:20.433 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.433 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.433 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:49:20.433 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.918618+0000", 2024-04-20T17:49:20.433 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000107372 2024-04-20T17:49:20.434 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.434 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.434 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:20.434 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.922598+0000", 2024-04-20T17:49:20.434 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0039802190000000001 2024-04-20T17:49:20.434 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.434 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.434 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:49:20.434 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.923961+0000", 2024-04-20T17:49:20.434 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013632309999999999 2024-04-20T17:49:20.434 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.434 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.434 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:49:20.434 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.924007+0000", 2024-04-20T17:49:20.435 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5484e-05 2024-04-20T17:49:20.435 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.435 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.435 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:49:20.435 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:18.924048+0000", 2024-04-20T17:49:20.435 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1223000000000002e-05 2024-04-20T17:49:20.435 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.435 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:20.435 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.435 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.435 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.435 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:45782 9.e 9:7c5453e5:::benchmark_data_smithi171_188319_object45781:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e904)", 2024-04-20T17:49:20.435 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:19.471190+0000", 2024-04-20T17:49:20.435 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.91898254700000004, 2024-04-20T17:49:20.436 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011995525, 2024-04-20T17:49:20.436 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:20.436 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:20.436 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:20.436 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:49:20.436 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:49:20.436 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:49:20.436 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 45782 2024-04-20T17:49:20.436 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.436 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:20.436 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.436 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:20.436 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.471190+0000", 2024-04-20T17:49:20.436 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.436 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.437 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.437 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:20.437 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.471190+0000", 2024-04-20T17:49:20.437 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.437 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.437 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.437 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:20.437 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.471191+0000", 2024-04-20T17:49:20.437 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0610000000000001e-06 2024-04-20T17:49:20.437 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.437 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.437 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:20.437 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.471280+0000", 2024-04-20T17:49:20.437 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8479000000000004e-05 2024-04-20T17:49:20.438 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.438 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.438 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:20.438 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.471289+0000", 2024-04-20T17:49:20.438 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4280000000000002e-06 2024-04-20T17:49:20.438 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.438 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.438 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:20.438 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.471297+0000", 2024-04-20T17:49:20.438 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3629999999999996e-06 2024-04-20T17:49:20.438 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.438 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.438 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:20.438 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.471835+0000", 2024-04-20T17:49:20.439 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00053873099999999998 2024-04-20T17:49:20.439 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.439 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.439 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:20.439 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.471948+0000", 2024-04-20T17:49:20.439 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000112667 2024-04-20T17:49:20.439 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.439 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.439 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:49:20.439 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.472067+0000", 2024-04-20T17:49:20.439 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000118929 2024-04-20T17:49:20.439 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.439 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.440 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:49:20.440 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.474525+0000", 2024-04-20T17:49:20.440 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0024578349999999998 2024-04-20T17:49:20.440 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.440 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.440 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:20.440 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.483088+0000", 2024-04-20T17:49:20.440 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0085631790000000006 2024-04-20T17:49:20.440 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.440 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.440 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:49:20.440 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.483142+0000", 2024-04-20T17:49:20.440 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3668999999999997e-05 2024-04-20T17:49:20.440 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.441 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.441 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:49:20.441 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.483186+0000", 2024-04-20T17:49:20.441 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4184000000000003e-05 2024-04-20T17:49:20.441 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.441 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:20.441 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.441 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.441 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.441 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:45783 9.e 9:737015c9:::benchmark_data_smithi171_188319_object45782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e904)", 2024-04-20T17:49:20.441 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:19.471548+0000", 2024-04-20T17:49:20.441 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.91862488899999994, 2024-04-20T17:49:20.441 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011795659999999999, 2024-04-20T17:49:20.442 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:20.442 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:20.442 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:20.442 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:49:20.442 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:49:20.442 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:49:20.442 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 45783 2024-04-20T17:49:20.442 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.442 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:20.442 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.442 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:20.442 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.471548+0000", 2024-04-20T17:49:20.442 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.442 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.443 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.443 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:20.443 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.471548+0000", 2024-04-20T17:49:20.443 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.443 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.443 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.443 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:20.443 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.471549+0000", 2024-04-20T17:49:20.443 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.24e-06 2024-04-20T17:49:20.443 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.443 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.443 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:20.443 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.471640+0000", 2024-04-20T17:49:20.444 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0655000000000005e-05 2024-04-20T17:49:20.444 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.444 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.444 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:20.444 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.471647+0000", 2024-04-20T17:49:20.444 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4900000000000003e-06 2024-04-20T17:49:20.444 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.444 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.444 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:20.444 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.471652+0000", 2024-04-20T17:49:20.444 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.673e-06 2024-04-20T17:49:20.444 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.444 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.444 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:20.445 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.473301+0000", 2024-04-20T17:49:20.445 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0016486610000000001 2024-04-20T17:49:20.445 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.445 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.445 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:20.445 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.473431+0000", 2024-04-20T17:49:20.445 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013039 2024-04-20T17:49:20.445 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.445 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.445 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:49:20.445 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.473580+0000", 2024-04-20T17:49:20.445 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014936499999999999 2024-04-20T17:49:20.445 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.445 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.446 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:49:20.446 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.474982+0000", 2024-04-20T17:49:20.446 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001401186 2024-04-20T17:49:20.446 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.446 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.446 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:20.446 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.483227+0000", 2024-04-20T17:49:20.446 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0082450820000000008 2024-04-20T17:49:20.446 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.446 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.446 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:49:20.446 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.483289+0000", 2024-04-20T17:49:20.446 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2451000000000002e-05 2024-04-20T17:49:20.446 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.447 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.447 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:49:20.447 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.483344+0000", 2024-04-20T17:49:20.447 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4466999999999999e-05 2024-04-20T17:49:20.447 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.447 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:20.447 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.447 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.447 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.447 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:46307 9.e 9:7b26b4a3:::benchmark_data_smithi171_188319_object46306:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e904)", 2024-04-20T17:49:20.447 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:19.648373+0000", 2024-04-20T17:49:20.447 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.74179995099999996, 2024-04-20T17:49:20.447 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012474471000000001, 2024-04-20T17:49:20.447 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:20.447 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:20.448 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:20.448 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:49:20.448 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:49:20.448 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:49:20.448 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 46307 2024-04-20T17:49:20.448 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.448 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:20.448 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.448 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:20.448 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.648373+0000", 2024-04-20T17:49:20.448 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.448 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.448 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.448 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:20.448 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.648373+0000", 2024-04-20T17:49:20.449 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.449 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.449 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.449 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:20.449 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.648374+0000", 2024-04-20T17:49:20.449 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5000000000000001e-07 2024-04-20T17:49:20.449 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.449 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.449 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:20.449 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.648471+0000", 2024-04-20T17:49:20.449 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7707000000000005e-05 2024-04-20T17:49:20.449 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.449 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.449 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:20.450 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.648480+0000", 2024-04-20T17:49:20.450 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0480000000000004e-06 2024-04-20T17:49:20.450 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.450 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.450 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:20.450 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.648488+0000", 2024-04-20T17:49:20.450 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4250000000000001e-06 2024-04-20T17:49:20.450 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.450 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.450 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:20.450 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.650709+0000", 2024-04-20T17:49:20.450 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022210620000000002 2024-04-20T17:49:20.450 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.450 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.451 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:20.451 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.650836+0000", 2024-04-20T17:49:20.451 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000127322 2024-04-20T17:49:20.451 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.451 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.451 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:49:20.451 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.650960+0000", 2024-04-20T17:49:20.451 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000123492 2024-04-20T17:49:20.451 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.451 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.451 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:49:20.451 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.652213+0000", 2024-04-20T17:49:20.451 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0012528190000000001 2024-04-20T17:49:20.451 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.451 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.452 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:20.452 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.660772+0000", 2024-04-20T17:49:20.452 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0085593490000000008 2024-04-20T17:49:20.452 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.452 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.452 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:49:20.452 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.660807+0000", 2024-04-20T17:49:20.452 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4668000000000001e-05 2024-04-20T17:49:20.452 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.452 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.452 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:49:20.452 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.660847+0000", 2024-04-20T17:49:20.452 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0729000000000003e-05 2024-04-20T17:49:20.452 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.453 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:20.453 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.453 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.453 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.453 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:46522 9.d 9:bcb70870:::benchmark_data_smithi171_188319_object46521:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e904)", 2024-04-20T17:49:20.453 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:19.728169+0000", 2024-04-20T17:49:20.453 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.66200393700000004, 2024-04-20T17:49:20.453 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010898957000000001, 2024-04-20T17:49:20.453 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:20.453 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:20.453 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:20.453 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:49:20.453 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:49:20.453 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:49:20.454 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 46522 2024-04-20T17:49:20.454 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.454 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:20.454 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.454 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:20.454 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.728169+0000", 2024-04-20T17:49:20.454 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.454 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.454 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.454 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:20.454 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.728169+0000", 2024-04-20T17:49:20.454 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.455 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.455 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.455 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:20.455 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.728170+0000", 2024-04-20T17:49:20.455 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9999999999999996e-07 2024-04-20T17:49:20.455 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.455 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.455 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:20.455 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.728246+0000", 2024-04-20T17:49:20.455 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6730999999999994e-05 2024-04-20T17:49:20.455 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.455 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.455 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:20.455 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.728259+0000", 2024-04-20T17:49:20.455 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2199e-05 2024-04-20T17:49:20.456 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.456 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.456 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:20.456 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.728265+0000", 2024-04-20T17:49:20.456 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.652e-06 2024-04-20T17:49:20.456 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.456 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.456 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:20.456 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.733140+0000", 2024-04-20T17:49:20.456 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0048746040000000003 2024-04-20T17:49:20.456 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.456 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.456 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:20.456 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.733241+0000", 2024-04-20T17:49:20.457 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000101455 2024-04-20T17:49:20.457 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.457 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.457 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:49:20.457 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.733380+0000", 2024-04-20T17:49:20.457 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013833300000000001 2024-04-20T17:49:20.457 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.457 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.457 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:49:20.457 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.734661+0000", 2024-04-20T17:49:20.457 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0012814300000000001 2024-04-20T17:49:20.457 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.457 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.457 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:20.457 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.738998+0000", 2024-04-20T17:49:20.458 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0043366439999999997 2024-04-20T17:49:20.458 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.458 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.458 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:49:20.458 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.739029+0000", 2024-04-20T17:49:20.458 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1121000000000002e-05 2024-04-20T17:49:20.458 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.458 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.458 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:49:20.458 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.739068+0000", 2024-04-20T17:49:20.458 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8988e-05 2024-04-20T17:49:20.458 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.458 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:20.459 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.459 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.459 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.459 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:47838 9.d 9:bd24fbc3:::benchmark_data_smithi171_188319_object47837:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e904)", 2024-04-20T17:49:20.459 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:20.198842+0000", 2024-04-20T17:49:20.459 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.19133044599999999, 2024-04-20T17:49:20.459 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.013078766, 2024-04-20T17:49:20.459 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:20.459 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:20.459 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:20.459 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:49:20.459 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:49:20.459 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:49:20.459 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 47838 2024-04-20T17:49:20.460 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.460 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:20.460 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.460 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:20.460 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.198842+0000", 2024-04-20T17:49:20.460 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.460 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.460 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.460 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:20.460 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.198842+0000", 2024-04-20T17:49:20.460 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.460 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.460 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.460 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:20.461 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.198843+0000", 2024-04-20T17:49:20.461 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5600000000000004e-07 2024-04-20T17:49:20.461 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.461 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.461 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:20.461 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.198955+0000", 2024-04-20T17:49:20.461 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000111778 2024-04-20T17:49:20.461 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.461 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.461 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:20.461 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.198963+0000", 2024-04-20T17:49:20.461 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1780000000000003e-06 2024-04-20T17:49:20.461 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.461 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.461 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:20.462 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.198970+0000", 2024-04-20T17:49:20.462 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5119999999999997e-06 2024-04-20T17:49:20.462 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.462 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.462 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:20.462 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.199014+0000", 2024-04-20T17:49:20.462 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4313000000000001e-05 2024-04-20T17:49:20.462 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.462 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.462 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:20.462 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.199129+0000", 2024-04-20T17:49:20.462 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000114536 2024-04-20T17:49:20.462 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.462 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.463 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:49:20.463 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.199257+0000", 2024-04-20T17:49:20.463 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000128393 2024-04-20T17:49:20.463 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.463 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.463 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:49:20.463 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.200217+0000", 2024-04-20T17:49:20.463 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00096024700000000001 2024-04-20T17:49:20.463 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.463 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.463 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:20.463 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.211866+0000", 2024-04-20T17:49:20.463 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011649186000000001 2024-04-20T17:49:20.463 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.463 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.464 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:49:20.464 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.211901+0000", 2024-04-20T17:49:20.464 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4371e-05 2024-04-20T17:49:20.464 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.464 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.464 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:49:20.464 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.211921+0000", 2024-04-20T17:49:20.464 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0296e-05 2024-04-20T17:49:20.464 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.464 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:20.464 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.464 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.464 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.464 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:48161 9.e 9:7c471427:::benchmark_data_smithi171_188319_object48160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e904)", 2024-04-20T17:49:20.465 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:20.320307+0000", 2024-04-20T17:49:20.465 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.069865629999999998, 2024-04-20T17:49:20.465 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.013011264, 2024-04-20T17:49:20.465 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:20.465 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:20.465 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:20.465 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:49:20.465 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:49:20.465 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:49:20.465 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 48161 2024-04-20T17:49:20.465 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.465 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:20.465 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.466 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:20.466 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.320307+0000", 2024-04-20T17:49:20.466 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.466 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.466 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.466 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:20.466 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.320307+0000", 2024-04-20T17:49:20.466 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.466 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.466 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.466 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:20.466 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.320308+0000", 2024-04-20T17:49:20.466 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8100000000000001e-07 2024-04-20T17:49:20.466 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.467 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.467 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:20.467 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.320365+0000", 2024-04-20T17:49:20.467 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6514999999999997e-05 2024-04-20T17:49:20.467 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.467 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.467 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:20.467 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.320371+0000", 2024-04-20T17:49:20.467 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7789999999999998e-06 2024-04-20T17:49:20.467 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.467 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.467 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:20.467 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.320376+0000", 2024-04-20T17:49:20.467 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4819999999999996e-06 2024-04-20T17:49:20.468 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.468 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.468 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:20.468 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.320412+0000", 2024-04-20T17:49:20.468 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6239999999999999e-05 2024-04-20T17:49:20.468 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.468 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.468 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:20.468 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.320510+0000", 2024-04-20T17:49:20.468 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8346999999999999e-05 2024-04-20T17:49:20.468 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.468 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.468 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:49:20.468 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.320647+0000", 2024-04-20T17:49:20.469 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013644199999999999 2024-04-20T17:49:20.469 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.469 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.469 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:49:20.469 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.322321+0000", 2024-04-20T17:49:20.469 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001673791 2024-04-20T17:49:20.469 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.469 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.469 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:20.469 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.333243+0000", 2024-04-20T17:49:20.469 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010922575 2024-04-20T17:49:20.469 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.469 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.469 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:49:20.470 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.333288+0000", 2024-04-20T17:49:20.470 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4734000000000002e-05 2024-04-20T17:49:20.470 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.470 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.470 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:49:20.470 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.333318+0000", 2024-04-20T17:49:20.470 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0477999999999999e-05 2024-04-20T17:49:20.470 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.470 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:20.470 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.470 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.470 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.470 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:48170 9.e 9:7f688e8e:::benchmark_data_smithi171_188319_object48169:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e904)", 2024-04-20T17:49:20.470 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:20.324590+0000", 2024-04-20T17:49:20.471 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.065582600000000005, 2024-04-20T17:49:20.471 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.016803241999999999, 2024-04-20T17:49:20.471 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:20.471 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:20.471 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:20.471 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:49:20.471 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:49:20.471 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:49:20.471 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 48170 2024-04-20T17:49:20.471 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.471 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:20.471 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.471 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:20.471 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.324590+0000", 2024-04-20T17:49:20.471 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.472 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.472 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.472 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:20.472 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.324590+0000", 2024-04-20T17:49:20.472 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.472 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.472 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.472 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:20.472 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.324592+0000", 2024-04-20T17:49:20.472 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.686e-06 2024-04-20T17:49:20.472 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.472 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.472 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:20.473 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.324652+0000", 2024-04-20T17:49:20.473 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9620999999999997e-05 2024-04-20T17:49:20.473 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.473 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.473 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:20.473 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.324659+0000", 2024-04-20T17:49:20.473 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8599999999999993e-06 2024-04-20T17:49:20.473 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.473 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.473 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:20.473 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.324664+0000", 2024-04-20T17:49:20.473 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0849999999999996e-06 2024-04-20T17:49:20.473 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.473 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.473 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:20.474 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.324705+0000", 2024-04-20T17:49:20.474 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0584999999999999e-05 2024-04-20T17:49:20.474 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.474 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.474 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:20.474 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.324842+0000", 2024-04-20T17:49:20.474 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013679100000000001 2024-04-20T17:49:20.474 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.474 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.474 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:49:20.474 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.324943+0000", 2024-04-20T17:49:20.474 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000101378 2024-04-20T17:49:20.474 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.474 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.475 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:49:20.475 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.326606+0000", 2024-04-20T17:49:20.475 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001662724 2024-04-20T17:49:20.475 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.475 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.475 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:20.475 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.341300+0000", 2024-04-20T17:49:20.475 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.01469402 2024-04-20T17:49:20.475 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.475 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.475 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:49:20.475 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.341352+0000", 2024-04-20T17:49:20.475 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1751e-05 2024-04-20T17:49:20.475 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.475 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.476 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:49:20.476 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.341393+0000", 2024-04-20T17:49:20.476 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1740999999999999e-05 2024-04-20T17:49:20.476 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.476 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:20.476 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.476 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.476 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.476 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:48245 9.e 9:7e0d9bf1:::benchmark_data_smithi171_188319_object48244:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e904)", 2024-04-20T17:49:20.476 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:20.360206+0000", 2024-04-20T17:49:20.476 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.029966657000000001, 2024-04-20T17:49:20.476 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015435803, 2024-04-20T17:49:20.476 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:20.476 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:20.476 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:20.476 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:49:20.477 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:49:20.477 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:49:20.477 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 48245 2024-04-20T17:49:20.477 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.477 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:20.477 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.477 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:20.477 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.360206+0000", 2024-04-20T17:49:20.477 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.477 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.477 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.477 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:20.477 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.360206+0000", 2024-04-20T17:49:20.478 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.478 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.478 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.478 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:20.478 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.360208+0000", 2024-04-20T17:49:20.478 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5999999999999999e-06 2024-04-20T17:49:20.478 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.478 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.478 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:20.478 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.360284+0000", 2024-04-20T17:49:20.478 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6316999999999995e-05 2024-04-20T17:49:20.478 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.478 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.478 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:20.478 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.360301+0000", 2024-04-20T17:49:20.479 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6480000000000001e-05 2024-04-20T17:49:20.479 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.479 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.479 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:20.479 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.360307+0000", 2024-04-20T17:49:20.479 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7329999999999997e-06 2024-04-20T17:49:20.479 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.479 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.479 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:20.479 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.360359+0000", 2024-04-20T17:49:20.479 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1214999999999997e-05 2024-04-20T17:49:20.479 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.479 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.479 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:20.479 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.360449+0000", 2024-04-20T17:49:20.480 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0192999999999995e-05 2024-04-20T17:49:20.480 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.480 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.480 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:49:20.480 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.360573+0000", 2024-04-20T17:49:20.480 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012442599999999999 2024-04-20T17:49:20.480 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.480 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.480 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:49:20.480 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.366386+0000", 2024-04-20T17:49:20.480 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0058130999999999999 2024-04-20T17:49:20.480 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.480 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.480 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:20.480 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.375548+0000", 2024-04-20T17:49:20.481 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0091615919999999997 2024-04-20T17:49:20.481 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.481 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.481 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:49:20.481 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.375599+0000", 2024-04-20T17:49:20.481 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1443999999999997e-05 2024-04-20T17:49:20.481 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.481 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.481 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:49:20.481 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.375642+0000", 2024-04-20T17:49:20.481 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2703000000000001e-05 2024-04-20T17:49:20.481 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.481 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:20.481 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.482 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.482 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.482 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:48247 9.e 9:7a250cd8:::benchmark_data_smithi171_188319_object48246:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e904)", 2024-04-20T17:49:20.482 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:20.360327+0000", 2024-04-20T17:49:20.482 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.029846158000000001, 2024-04-20T17:49:20.482 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015493649999999999, 2024-04-20T17:49:20.482 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:20.482 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:20.482 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:20.482 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:49:20.482 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:49:20.482 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:49:20.482 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 48247 2024-04-20T17:49:20.482 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.482 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:20.483 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.483 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:20.483 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.360327+0000", 2024-04-20T17:49:20.483 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.483 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.483 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.483 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:20.483 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.360327+0000", 2024-04-20T17:49:20.483 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.483 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.483 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.483 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:20.483 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.360327+0000", 2024-04-20T17:49:20.483 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8999999999999996e-07 2024-04-20T17:49:20.483 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.484 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.484 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:20.484 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.360358+0000", 2024-04-20T17:49:20.484 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1115999999999998e-05 2024-04-20T17:49:20.484 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.484 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.484 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:20.484 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.360363+0000", 2024-04-20T17:49:20.484 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.42e-06 2024-04-20T17:49:20.484 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.484 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.484 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:20.484 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.360366+0000", 2024-04-20T17:49:20.484 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5949999999999999e-06 2024-04-20T17:49:20.484 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.485 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.485 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:20.485 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.362233+0000", 2024-04-20T17:49:20.485 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001866033 2024-04-20T17:49:20.485 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.485 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.485 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:20.485 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.362357+0000", 2024-04-20T17:49:20.485 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012435199999999999 2024-04-20T17:49:20.485 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.485 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.485 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:49:20.485 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.362482+0000", 2024-04-20T17:49:20.485 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012554299999999999 2024-04-20T17:49:20.485 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.486 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.486 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:49:20.486 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.366394+0000", 2024-04-20T17:49:20.486 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0039118490000000002 2024-04-20T17:49:20.486 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.486 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.486 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:20.486 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.375708+0000", 2024-04-20T17:49:20.486 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0093137169999999991 2024-04-20T17:49:20.486 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.486 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.486 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:49:20.486 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.375771+0000", 2024-04-20T17:49:20.486 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2935999999999995e-05 2024-04-20T17:49:20.487 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.487 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.487 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:49:20.487 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.375820+0000", 2024-04-20T17:49:20.487 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9398999999999999e-05 2024-04-20T17:49:20.487 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.487 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:20.487 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.487 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.487 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.487 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:48248 9.e 9:7a64dca3:::benchmark_data_smithi171_188319_object48247:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e904)", 2024-04-20T17:49:20.487 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:20.360380+0000", 2024-04-20T17:49:20.487 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.029793304, 2024-04-20T17:49:20.487 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015537346, 2024-04-20T17:49:20.488 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:20.488 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:20.488 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:20.488 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:49:20.488 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:49:20.488 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:49:20.488 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 48248 2024-04-20T17:49:20.488 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.488 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:20.488 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.488 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:20.488 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.360380+0000", 2024-04-20T17:49:20.488 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.488 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.488 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.488 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:20.489 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.360380+0000", 2024-04-20T17:49:20.489 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.489 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.489 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.489 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:20.489 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.360380+0000", 2024-04-20T17:49:20.489 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7100000000000002e-07 2024-04-20T17:49:20.489 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.489 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.489 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:20.489 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.360421+0000", 2024-04-20T17:49:20.489 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1145e-05 2024-04-20T17:49:20.489 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.489 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.490 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:20.490 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.360427+0000", 2024-04-20T17:49:20.490 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5940000000000001e-06 2024-04-20T17:49:20.490 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.490 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.490 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:20.490 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.360430+0000", 2024-04-20T17:49:20.490 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5439999999999999e-06 2024-04-20T17:49:20.490 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.490 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.490 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:20.490 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.363603+0000", 2024-04-20T17:49:20.490 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0031722709999999999 2024-04-20T17:49:20.491 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.491 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.491 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:20.491 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.363729+0000", 2024-04-20T17:49:20.491 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012665500000000001 2024-04-20T17:49:20.491 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.491 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.491 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:49:20.491 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.363885+0000", 2024-04-20T17:49:20.491 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000155854 2024-04-20T17:49:20.491 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.491 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.491 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:49:20.491 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.366400+0000", 2024-04-20T17:49:20.492 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00251512 2024-04-20T17:49:20.492 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.492 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.492 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:20.492 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.375864+0000", 2024-04-20T17:49:20.492 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0094639640000000001 2024-04-20T17:49:20.492 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.492 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.492 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:49:20.492 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.375894+0000", 2024-04-20T17:49:20.492 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.932e-05 2024-04-20T17:49:20.492 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.492 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.492 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:49:20.492 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.375917+0000", 2024-04-20T17:49:20.493 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3308000000000001e-05 2024-04-20T17:49:20.493 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.493 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:20.493 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.493 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.493 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.493 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:48254 9.e 9:7abc8c89:::benchmark_data_smithi171_188319_object48253:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e904)", 2024-04-20T17:49:20.493 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:20.360747+0000", 2024-04-20T17:49:20.493 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.029426186999999999, 2024-04-20T17:49:20.493 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015337236000000001, 2024-04-20T17:49:20.493 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:20.493 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:20.493 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:20.493 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:49:20.494 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:49:20.494 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:49:20.494 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 48254 2024-04-20T17:49:20.494 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.494 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:20.494 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.494 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:20.494 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.360747+0000", 2024-04-20T17:49:20.494 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.494 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.494 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.494 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:20.494 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.360747+0000", 2024-04-20T17:49:20.494 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.495 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.495 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.495 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:20.495 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.360748+0000", 2024-04-20T17:49:20.495 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.587e-06 2024-04-20T17:49:20.495 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.495 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.495 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:20.495 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.360867+0000", 2024-04-20T17:49:20.495 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000118472 2024-04-20T17:49:20.495 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.495 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.495 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:20.495 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.360878+0000", 2024-04-20T17:49:20.496 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0883999999999999e-05 2024-04-20T17:49:20.496 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.496 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.496 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:20.496 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.360886+0000", 2024-04-20T17:49:20.496 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9060000000000008e-06 2024-04-20T17:49:20.496 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.496 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.496 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:20.496 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.365153+0000", 2024-04-20T17:49:20.496 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.004266387 2024-04-20T17:49:20.496 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.496 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.496 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:20.497 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.365264+0000", 2024-04-20T17:49:20.497 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011089799999999999 2024-04-20T17:49:20.497 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.497 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.497 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:49:20.497 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.365353+0000", 2024-04-20T17:49:20.497 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9457999999999996e-05 2024-04-20T17:49:20.497 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.497 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.497 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:49:20.497 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.367862+0000", 2024-04-20T17:49:20.497 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0025087239999999999 2024-04-20T17:49:20.497 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.497 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.497 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:20.498 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.375979+0000", 2024-04-20T17:49:20.498 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0081171239999999999 2024-04-20T17:49:20.498 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.498 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.498 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:49:20.498 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.376040+0000", 2024-04-20T17:49:20.498 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0476999999999997e-05 2024-04-20T17:49:20.498 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.498 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.498 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:49:20.498 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.376084+0000", 2024-04-20T17:49:20.498 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4319e-05 2024-04-20T17:49:20.498 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.498 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:20.499 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.499 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.499 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.499 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:48263 9.e 9:7eee399b:::benchmark_data_smithi171_188319_object48262:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e904)", 2024-04-20T17:49:20.499 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:20.365173+0000", 2024-04-20T17:49:20.499 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.025000202999999999, 2024-04-20T17:49:20.499 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010987288, 2024-04-20T17:49:20.499 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:20.499 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:20.499 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:20.499 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:49:20.499 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:49:20.499 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:49:20.499 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 48263 2024-04-20T17:49:20.499 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.500 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:20.500 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.500 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:20.500 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.365173+0000", 2024-04-20T17:49:20.500 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.500 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.500 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.500 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:20.500 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.365173+0000", 2024-04-20T17:49:20.500 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:20.500 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.500 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.500 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:20.500 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.365175+0000", 2024-04-20T17:49:20.501 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.12e-06 2024-04-20T17:49:20.501 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.501 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.501 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:20.501 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.365221+0000", 2024-04-20T17:49:20.501 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.672e-05 2024-04-20T17:49:20.501 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.501 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.501 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:20.501 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.365232+0000", 2024-04-20T17:49:20.501 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0046e-05 2024-04-20T17:49:20.501 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.501 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.501 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:20.501 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.365241+0000", 2024-04-20T17:49:20.502 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0699999999999996e-06 2024-04-20T17:49:20.502 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.502 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.502 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:20.502 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.366431+0000", 2024-04-20T17:49:20.502 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0011908380000000001 2024-04-20T17:49:20.502 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.502 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.502 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:20.502 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.366545+0000", 2024-04-20T17:49:20.502 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011317200000000001 2024-04-20T17:49:20.502 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.502 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.502 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:49:20.502 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.366671+0000", 2024-04-20T17:49:20.503 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012676199999999999 2024-04-20T17:49:20.503 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.503 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.503 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:49:20.503 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.367944+0000", 2024-04-20T17:49:20.503 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0012728399999999999 2024-04-20T17:49:20.503 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.503 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.503 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:20.503 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.376114+0000", 2024-04-20T17:49:20.503 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0081701199999999995 2024-04-20T17:49:20.503 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.503 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.503 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:49:20.503 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.376141+0000", 2024-04-20T17:49:20.504 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6611999999999999e-05 2024-04-20T17:49:20.504 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:20.504 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:20.504 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:49:20.504 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:20.376160+0000", 2024-04-20T17:49:20.504 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8987999999999998e-05 2024-04-20T17:49:20.504 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.504 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:20.504 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.504 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:20.504 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:20.504 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:49:20.505 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:49:20.553 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:20.635 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:49:20.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:20.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:49:20.740 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:49:20.779 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:20.806 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:49:20.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:20.810 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:49:20.859 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 14 16 49603 49587 221.352 169.75 0.00564688 0.00451576 2024-04-20T17:49:20.910 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:49:20.948 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:21.021 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:49:21.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:21.121 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:21.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:21.153 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:49:21.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:21.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:49:21.163 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-20T17:49:21.257 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:49:21.300 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:21.834 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:21 smithi005 ceph-mon[39562]: pgmap v2391: 27 pgs: 27 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 192 MiB/s wr, 3.07k op/s 2024-04-20T17:49:21.859 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 15 16 52268 52252 217.699 166.562 0.00240372 0.00459071 2024-04-20T17:49:21.862 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:49:21.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:21.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:49:21.972 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:49:22.015 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:22.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:21 smithi171 ceph-mon[44189]: pgmap v2391: 27 pgs: 27 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 192 MiB/s wr, 3.07k op/s 2024-04-20T17:49:22.220 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:49:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:21 smithi106 ceph-mon[44423]: pgmap v2391: 27 pgs: 27 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 192 MiB/s wr, 3.07k op/s 2024-04-20T17:49:22.573 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:49:22.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:22.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:49:22.680 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:49:22.720 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:22.749 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:49:22.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:22.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:49:22.845 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:22 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:49:22.845 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:22 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:49:22.845 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:22 smithi005 ceph-mon[39562]: osdmap e905: 12 total, 12 up, 12 in 2024-04-20T17:49:22.852 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:49:22.859 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 16 16 54894 54878 214.349 164.125 0.00732254 0.00466375 2024-04-20T17:49:22.900 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:22.926 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:49:22.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:22.932 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:49:23.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:22 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:49:23.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:22 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:49:23.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:22 smithi106 ceph-mon[44423]: osdmap e905: 12 total, 12 up, 12 in 2024-04-20T17:49:23.032 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:49:23.078 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:23.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:22 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:49:23.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:22 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:49:23.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:22 smithi171 ceph-mon[44189]: osdmap e905: 12 total, 12 up, 12 in 2024-04-20T17:49:23.109 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:49:23.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:23.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:49:23.214 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:49:23.259 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:23.820 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:49:23.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:23.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:49:23.859 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 17 16 57432 57416 211.071 158.625 0.00589756 0.00473608 2024-04-20T17:49:23.926 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:49:23.967 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:23.971 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:49:23.994 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:49:23.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:23.998 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:49:24.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:23 smithi171 ceph-mon[44189]: pgmap v2393: 27 pgs: 27 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 200 MiB/s wr, 3.21k op/s 2024-04-20T17:49:24.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:23 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/53140576' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:49:24.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:23 smithi171 ceph-mon[44189]: from='client.28112 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:49:24.099 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:49:24.137 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:23 smithi005 ceph-mon[39562]: pgmap v2393: 27 pgs: 27 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 200 MiB/s wr, 3.21k op/s 2024-04-20T17:49:24.137 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:23 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/53140576' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:49:24.137 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:23 smithi005 ceph-mon[39562]: from='client.28112 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:49:24.140 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:24.175 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:49:24.178 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:49:24.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:24.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:49:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:23 smithi106 ceph-mon[44423]: pgmap v2393: 27 pgs: 27 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 200 MiB/s wr, 3.21k op/s 2024-04-20T17:49:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:23 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/53140576' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:49:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:23 smithi106 ceph-mon[44423]: from='client.28112 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:49:24.290 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:49:24.324 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:24.352 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:49:24.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:24.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:49:24.460 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:49:24.499 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:24.751 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:49:24.770 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:49:24.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:24.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:49:24.859 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 18 16 59883 59867 207.854 153.188 0.00710798 0.00480938 2024-04-20T17:49:24.878 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:49:24.918 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:24.965 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:49:24.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:24.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:49:25.077 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:49:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:24 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:49:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:24 smithi171 ceph-mon[44189]: from='client.28112 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:49:25.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:24 smithi171 ceph-mon[44189]: osdmap e906: 12 total, 12 up, 12 in 2024-04-20T17:49:25.112 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:24 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:49:25.112 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:24 smithi005 ceph-mon[39562]: from='client.28112 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:49:25.112 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:24 smithi005 ceph-mon[39562]: osdmap e906: 12 total, 12 up, 12 in 2024-04-20T17:49:25.114 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:25.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:24 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:49:25.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:24 smithi106 ceph-mon[44423]: from='client.28112 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:49:25.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:24 smithi106 ceph-mon[44423]: osdmap e906: 12 total, 12 up, 12 in 2024-04-20T17:49:25.859 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 19 16 62297 62281 204.855 150.875 0.00578996 0.00487957 2024-04-20T17:49:26.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:25 smithi171 ceph-mon[44189]: pgmap v2395: 27 pgs: 27 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 144 MiB/s wr, 2.31k op/s 2024-04-20T17:49:26.132 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:25 smithi005 ceph-mon[39562]: pgmap v2395: 27 pgs: 27 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 144 MiB/s wr, 2.31k op/s 2024-04-20T17:49:26.171 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:49:26.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:26.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:49:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:25 smithi106 ceph-mon[44423]: pgmap v2395: 27 pgs: 27 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 144 MiB/s wr, 2.31k op/s 2024-04-20T17:49:26.279 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:49:26.323 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:26.367 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:49:26.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:26.378 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:49:26.479 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:49:26.520 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:26.549 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:49:26.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:26.553 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:49:26.654 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:49:26.703 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:26.732 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:49:26.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:26.739 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:49:26.840 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:49:26.859 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:49:26.860148 min lat: 0.00101366 max lat: 0.200169 avg lat: 0.00495343 2024-04-20T17:49:26.859 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:49:26.859 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 20 15 64601 64586 201.815 144.062 0.00392322 0.00495343 2024-04-20T17:49:26.882 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:26.908 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:49:26.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:26.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:49:27.014 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:49:27.046 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:27.075 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:49:27.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:27.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:49:27.180 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:49:27.230 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:27.257 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:49:27.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:27.264 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:49:27.329 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:49:27.364 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:49:27.401 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:27.434 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:49:27.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:27.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:49:27.541 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:49:27.579 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:27.604 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:49:27.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:27.609 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:49:27.710 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:49:27.761 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:27.830 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:27 smithi171 ceph-mon[44189]: pgmap v2396: 27 pgs: 27 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 169 MiB/s wr, 2.71k op/s 2024-04-20T17:49:27.830 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:27 smithi171 ceph-mon[44189]: 1.6 scrub starts 2024-04-20T17:49:27.831 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:27 smithi171 ceph-mon[44189]: 1.6 scrub ok 2024-04-20T17:49:27.831 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:27 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2938965847' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:49:27.859 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 21 16 66798 66782 198.74 137.25 0.00661251 0.00503041 2024-04-20T17:49:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:27 smithi005 ceph-mon[39562]: pgmap v2396: 27 pgs: 27 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 169 MiB/s wr, 2.71k op/s 2024-04-20T17:49:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:27 smithi005 ceph-mon[39562]: 1.6 scrub starts 2024-04-20T17:49:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:27 smithi005 ceph-mon[39562]: 1.6 scrub ok 2024-04-20T17:49:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:27 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2938965847' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:49:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:27 smithi106 ceph-mon[44423]: pgmap v2396: 27 pgs: 27 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 169 MiB/s wr, 2.71k op/s 2024-04-20T17:49:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:27 smithi106 ceph-mon[44423]: 1.6 scrub starts 2024-04-20T17:49:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:27 smithi106 ceph-mon[44423]: 1.6 scrub ok 2024-04-20T17:49:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:27 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2938965847' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:49:28.316 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:49:28.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:28.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:49:28.423 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:49:28.461 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:28.490 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:49:28.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:28.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:49:28.594 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:49:28.628 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:28.651 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:49:28.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:28.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:49:28.755 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:49:28.796 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:28.827 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:49:28.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:28.832 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:49:28.859 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 22 16 68999 68983 195.958 137.562 0.0105239 0.00510111 2024-04-20T17:49:28.933 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:49:28.971 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:29.004 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:49:29.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:29.009 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:49:29.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:28 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2938965847' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-20T17:49:29.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:28 smithi171 ceph-mon[44189]: osdmap e907: 12 total, 12 up, 12 in 2024-04-20T17:49:29.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:28 smithi171 ceph-mon[44189]: pgmap v2398: 27 pgs: 27 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 151 MiB/s wr, 2.42k op/s 2024-04-20T17:49:29.095 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:28 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2938965847' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-20T17:49:29.095 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:28 smithi005 ceph-mon[39562]: osdmap e907: 12 total, 12 up, 12 in 2024-04-20T17:49:29.095 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:28 smithi005 ceph-mon[39562]: pgmap v2398: 27 pgs: 27 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 151 MiB/s wr, 2.42k op/s 2024-04-20T17:49:29.109 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:49:29.154 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:29.184 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:49:29.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:29.190 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:49:29.197 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.10 to 1 (10000) 2024-04-20T17:49:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:28 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2938965847' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-20T17:49:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:28 smithi106 ceph-mon[44423]: osdmap e907: 12 total, 12 up, 12 in 2024-04-20T17:49:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:28 smithi106 ceph-mon[44423]: pgmap v2398: 27 pgs: 27 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 151 MiB/s wr, 2.42k op/s 2024-04-20T17:49:29.290 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:49:29.331 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:29.363 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:49:29.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:29.369 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:49:29.470 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:49:29.510 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:29.546 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:49:29.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:29.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:49:29.650 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:49:29.705 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:29.860 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:49:29.860 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:49:29.860 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-04-20T17:49:29.860 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:49:29.861 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 23 16 71062 71046 193.028 128.938 0.00632381 0.00517879 2024-04-20T17:49:30.196 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:29 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:49:30.196 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:29 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:49:30.196 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:29 smithi005 ceph-mon[39562]: osdmap e908: 12 total, 12 up, 12 in 2024-04-20T17:49:30.196 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:29 smithi005 ceph-mon[39562]: 1.1 scrub starts 2024-04-20T17:49:30.236 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 11 1 2024-04-20T17:49:30.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:29 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:49:30.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:29 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:49:30.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:29 smithi106 ceph-mon[44423]: osdmap e908: 12 total, 12 up, 12 in 2024-04-20T17:49:30.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:29 smithi106 ceph-mon[44423]: 1.1 scrub starts 2024-04-20T17:49:30.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:29 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:49:30.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:29 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:49:30.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:29 smithi171 ceph-mon[44189]: osdmap e908: 12 total, 12 up, 12 in 2024-04-20T17:49:30.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:29 smithi171 ceph-mon[44189]: 1.1 scrub starts 2024-04-20T17:49:30.468 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:49:30.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:30.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:49:30.492 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:49:30.575 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:49:30.613 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:30.641 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:49:30.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:30.647 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:49:30.748 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:49:30.792 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:30.819 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:49:30.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:30.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:49:30.852 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:30 smithi005 ceph-mon[39562]: 1.1 scrub ok 2024-04-20T17:49:30.853 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:30 smithi005 ceph-mon[39562]: pgmap v2400: 27 pgs: 27 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 152 MiB/s wr, 2.44k op/s 2024-04-20T17:49:30.861 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 24 16 73114 73098 190.329 128.25 0.00807847 0.00525254 2024-04-20T17:49:30.928 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:49:30.959 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:30.982 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:49:30.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:30.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:49:31.016 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-20T17:49:31.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:30 smithi171 ceph-mon[44189]: 1.1 scrub ok 2024-04-20T17:49:31.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:30 smithi171 ceph-mon[44189]: pgmap v2400: 27 pgs: 27 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 152 MiB/s wr, 2.44k op/s 2024-04-20T17:49:31.088 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:49:31.130 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:31.159 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:49:31.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:31.162 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:49:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:30 smithi106 ceph-mon[44423]: 1.1 scrub ok 2024-04-20T17:49:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:30 smithi106 ceph-mon[44423]: pgmap v2400: 27 pgs: 27 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 152 MiB/s wr, 2.44k op/s 2024-04-20T17:49:31.262 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:49:31.298 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:31.325 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:49:31.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:31.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:49:31.433 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:49:31.506 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:31.592 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:49:31.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:31.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:49:31.648 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:49:31.681 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:49:31.699 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:49:31.709 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:49:31.745 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:49:31.754 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:31.774 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:49:31.781 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:49:31.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:31.784 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:49:31.798 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:49:31.831 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:49:31.861 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 25 16 75138 75122 187.776 126.5 0.00652865 0.00532355 2024-04-20T17:49:31.878 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:49:31.885 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:49:31.912 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:49:31.941 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:31.948 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:49:31.981 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:49:32.011 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:49:32.014 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:49:32.014 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:49:32.015 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:49:32.015 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:49:32.015 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:49:32.015 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:49:32.015 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:49:32.015 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:49:32.015 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:49:32.015 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:49:32.015 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:49:32.016 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:49:32.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:31 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1117388426' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:49:32.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:31 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1117388426' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:49:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:31 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1117388426' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:49:32.810 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:49:32.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:32.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:49:32.861 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 26 16 77230 77214 185.582 130.75 0.00880108 0.00538655 2024-04-20T17:49:32.915 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:49:32.959 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:32.990 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:49:32.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:32.994 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:49:33.095 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:49:33.129 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:32 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.129 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:32 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.129 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:32 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.129 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:32 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.130 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:32 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.130 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:32 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.130 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:32 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.130 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:32 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.130 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:32 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.130 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:32 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.130 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:32 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.130 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:32 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.130 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:32 smithi106 ceph-mon[44423]: pgmap v2401: 27 pgs: 27 active+clean; 4.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 161 MiB/s wr, 2.58k op/s 2024-04-20T17:49:33.134 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:33.158 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:49:33.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:33.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:49:33.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:32 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:32 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:32 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:32 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:32 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:32 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:32 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:32 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:32 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:32 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:32 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:32 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:32 smithi005 ceph-mon[39562]: pgmap v2401: 27 pgs: 27 active+clean; 4.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 161 MiB/s wr, 2.58k op/s 2024-04-20T17:49:33.264 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:49:33.312 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:32 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.313 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:32 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.313 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:32 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.313 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:32 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.313 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:32 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.313 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:32 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.313 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:32 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.313 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:32 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.313 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:32 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.313 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:32 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.313 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:32 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.313 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:32 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:33.313 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:32 smithi171 ceph-mon[44189]: pgmap v2401: 27 pgs: 27 active+clean; 4.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 161 MiB/s wr, 2.58k op/s 2024-04-20T17:49:33.317 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:33.345 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:49:33.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:33.350 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:49:33.451 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:49:33.477 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:33.495 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:49:33.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:33.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:49:33.599 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:49:33.637 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:33.666 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:49:33.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:33.671 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:49:33.772 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:49:33.809 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:33.862 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 27 16 79039 79023 182.896 113.062 0.00398361 0.00546579 2024-04-20T17:49:34.062 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:49:34.064 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:49:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:33 smithi171 ceph-mon[44189]: 1.3 scrub starts 2024-04-20T17:49:34.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:33 smithi171 ceph-mon[44189]: 1.3 scrub ok 2024-04-20T17:49:34.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:33 smithi005 ceph-mon[39562]: 1.3 scrub starts 2024-04-20T17:49:34.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:33 smithi005 ceph-mon[39562]: 1.3 scrub ok 2024-04-20T17:49:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:33 smithi106 ceph-mon[44423]: 1.3 scrub starts 2024-04-20T17:49:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:33 smithi106 ceph-mon[44423]: 1.3 scrub ok 2024-04-20T17:49:34.347 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:49:34.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:34.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:49:34.461 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:49:34.493 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:34.523 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:49:34.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:34.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:49:34.629 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:49:34.674 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:34.755 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:49:34.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:34.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:49:34.860 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:49:34.861 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 28 16 81067 81051 180.891 126.75 0.00299334 0.00552676 2024-04-20T17:49:34.899 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:34.926 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:49:34.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:34.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:49:34.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:34 smithi005 ceph-mon[39562]: pgmap v2402: 27 pgs: 27 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 125 MiB/s wr, 2.00k op/s 2024-04-20T17:49:35.031 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:49:35.069 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:35.099 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:49:35.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:35.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:49:35.204 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:49:35.236 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:34 smithi106 ceph-mon[44423]: pgmap v2402: 27 pgs: 27 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 125 MiB/s wr, 2.00k op/s 2024-04-20T17:49:35.237 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:35.260 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:49:35.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:35.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:49:35.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:34 smithi171 ceph-mon[44189]: pgmap v2402: 27 pgs: 27 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 125 MiB/s wr, 2.00k op/s 2024-04-20T17:49:35.366 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:49:35.422 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:35.451 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:49:35.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:35.454 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:49:35.555 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:49:35.593 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:35.618 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:49:35.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:35.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:49:35.724 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:49:35.759 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:35.784 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:49:35.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:35.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:49:35.862 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 29 16 83049 83033 178.924 123.875 0.0100363 0.00558749 2024-04-20T17:49:35.888 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:49:35.919 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:36.081 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:49:36.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:36.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:49:36.187 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:49:36.224 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:36.386 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:49:36.862 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 30 16 84963 84947 176.948 119.625 0.00362878 0.00564969 2024-04-20T17:49:37.078 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:49:37.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:37.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:49:37.184 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:49:37.229 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:37.255 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:49:37.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:37.260 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:49:37.361 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:49:37.391 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:37.422 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:49:37.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:37.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:49:37.526 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:49:37.564 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:37.651 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:49:37.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:37.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:49:37.767 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:49:37.807 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:37.862 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 31 16 86842 86826 175.028 117.438 0.00754137 0.00571194 2024-04-20T17:49:37.879 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:49:37.887 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:37 smithi005 ceph-mon[39562]: pgmap v2403: 27 pgs: 27 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s wr, 2.18k op/s 2024-04-20T17:49:37.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:37.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:49:37.989 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:49:38.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:37 smithi106 ceph-mon[44423]: pgmap v2403: 27 pgs: 27 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s wr, 2.18k op/s 2024-04-20T17:49:38.041 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:38.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:37 smithi171 ceph-mon[44189]: pgmap v2403: 27 pgs: 27 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s wr, 2.18k op/s 2024-04-20T17:49:38.601 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:49:38.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:38.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:49:38.708 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:49:38.766 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:38.798 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:49:38.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:38.802 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:49:38.862 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 32 16 88736 88720 173.257 118.375 0.00347129 0.00577058 2024-04-20T17:49:38.904 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:49:38.943 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:39.773 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.11 to 1 (10000) 2024-04-20T17:49:39.862 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 33 16 90597 90581 171.531 116.312 0.00403926 0.00582797 2024-04-20T17:49:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:39 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:49:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:39 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:49:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:39 smithi106 ceph-mon[44423]: pgmap v2404: 27 pgs: 27 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 130 MiB/s wr, 2.08k op/s 2024-04-20T17:49:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:39 smithi106 ceph-mon[44423]: osdmap e909: 12 total, 12 up, 12 in 2024-04-20T17:49:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:39 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3067455104' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:49:40.079 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:49:40.079 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:49:40.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:39 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:49:40.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:39 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:49:40.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:39 smithi171 ceph-mon[44189]: pgmap v2404: 27 pgs: 27 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 130 MiB/s wr, 2.08k op/s 2024-04-20T17:49:40.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:39 smithi171 ceph-mon[44189]: osdmap e909: 12 total, 12 up, 12 in 2024-04-20T17:49:40.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:39 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3067455104' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:49:40.105 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:39 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:49:40.105 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:39 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:49:40.105 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:39 smithi005 ceph-mon[39562]: pgmap v2404: 27 pgs: 27 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 130 MiB/s wr, 2.08k op/s 2024-04-20T17:49:40.105 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:39 smithi005 ceph-mon[39562]: osdmap e909: 12 total, 12 up, 12 in 2024-04-20T17:49:40.106 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:39 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3067455104' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:49:40.862 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 34 16 92467 92451 169.924 116.875 0.00385745 0.00588383 2024-04-20T17:49:40.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:40 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:49:40.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:40 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:49:40.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:40 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3067455104' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-20T17:49:40.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:40 smithi005 ceph-mon[39562]: osdmap e910: 12 total, 12 up, 12 in 2024-04-20T17:49:40.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:40 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2891440087' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:49:40.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:40 smithi005 ceph-mon[39562]: from='client.28136 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:49:41.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:40 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:49:41.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:40 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:49:41.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:40 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3067455104' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-20T17:49:41.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:40 smithi171 ceph-mon[44189]: osdmap e910: 12 total, 12 up, 12 in 2024-04-20T17:49:41.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:40 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2891440087' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:49:41.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:40 smithi171 ceph-mon[44189]: from='client.28136 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:49:41.165 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-20T17:49:41.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:40 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:49:41.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:40 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:49:41.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:40 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3067455104' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-20T17:49:41.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:40 smithi106 ceph-mon[44423]: osdmap e910: 12 total, 12 up, 12 in 2024-04-20T17:49:41.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:40 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2891440087' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:49:41.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:40 smithi106 ceph-mon[44423]: from='client.28136 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:49:41.589 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 1 1 2024-04-20T17:49:41.711 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:49:41.862 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 35 16 94295 94279 168.332 114.25 0.00414769 0.00593891 2024-04-20T17:49:41.908 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:49:41.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:41.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:49:41.921 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:41 smithi005 ceph-mon[39562]: pgmap v2407: 27 pgs: 27 active+clean; 5.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 129 MiB/s wr, 2.07k op/s 2024-04-20T17:49:41.921 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:41 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:49:41.921 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:41 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:49:41.921 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:41 smithi005 ceph-mon[39562]: from='client.28136 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:49:41.921 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:41 smithi005 ceph-mon[39562]: osdmap e911: 12 total, 12 up, 12 in 2024-04-20T17:49:41.921 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:41 smithi005 ceph-mon[39562]: osdmap e912: 12 total, 12 up, 12 in 2024-04-20T17:49:42.021 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:49:42.068 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:42.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:41 smithi171 ceph-mon[44189]: pgmap v2407: 27 pgs: 27 active+clean; 5.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 129 MiB/s wr, 2.07k op/s 2024-04-20T17:49:42.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:41 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:49:42.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:41 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:49:42.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:41 smithi171 ceph-mon[44189]: from='client.28136 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:49:42.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:41 smithi171 ceph-mon[44189]: osdmap e911: 12 total, 12 up, 12 in 2024-04-20T17:49:42.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:41 smithi171 ceph-mon[44189]: osdmap e912: 12 total, 12 up, 12 in 2024-04-20T17:49:42.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:41 smithi106 ceph-mon[44423]: pgmap v2407: 27 pgs: 27 active+clean; 5.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 129 MiB/s wr, 2.07k op/s 2024-04-20T17:49:42.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:41 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:49:42.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:41 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:49:42.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:41 smithi106 ceph-mon[44423]: from='client.28136 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:49:42.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:41 smithi106 ceph-mon[44423]: osdmap e911: 12 total, 12 up, 12 in 2024-04-20T17:49:42.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:41 smithi106 ceph-mon[44423]: osdmap e912: 12 total, 12 up, 12 in 2024-04-20T17:49:42.350 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:49:42.418 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:49:42.418 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:49:42.863 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 36 16 96071 96055 166.74 111 0.00977776 0.00599556 2024-04-20T17:49:42.909 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08934307888783866]}]: dispatch 2024-04-20T17:49:42.910 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08672247798618558]}]: dispatch 2024-04-20T17:49:42.910 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08781439502854103]}]: dispatch 2024-04-20T17:49:42.910 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08705005309889222]}]: dispatch 2024-04-20T17:49:42.910 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08781439502854103]}]: dispatch 2024-04-20T17:49:42.910 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08689718471296246]}]: dispatch 2024-04-20T17:49:42.910 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08705005309889222]}]: dispatch 2024-04-20T17:49:42.910 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08679527245567595]}]: dispatch 2024-04-20T17:49:42.910 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08781439502854103]}]: dispatch 2024-04-20T17:49:42.910 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08672247798618558]}]: dispatch 2024-04-20T17:49:42.910 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08662541869353177]}]: dispatch 2024-04-20T17:49:42.910 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:42 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08689718471296246]}]: dispatch 2024-04-20T17:49:43.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08934307888783866]}]: dispatch 2024-04-20T17:49:43.090 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08672247798618558]}]: dispatch 2024-04-20T17:49:43.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08781439502854103]}]: dispatch 2024-04-20T17:49:43.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08705005309889222]}]: dispatch 2024-04-20T17:49:43.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08781439502854103]}]: dispatch 2024-04-20T17:49:43.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08689718471296246]}]: dispatch 2024-04-20T17:49:43.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08705005309889222]}]: dispatch 2024-04-20T17:49:43.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08679527245567595]}]: dispatch 2024-04-20T17:49:43.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08781439502854103]}]: dispatch 2024-04-20T17:49:43.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08672247798618558]}]: dispatch 2024-04-20T17:49:43.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08662541869353177]}]: dispatch 2024-04-20T17:49:43.091 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:42 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08689718471296246]}]: dispatch 2024-04-20T17:49:43.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08934307888783866]}]: dispatch 2024-04-20T17:49:43.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08672247798618558]}]: dispatch 2024-04-20T17:49:43.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08781439502854103]}]: dispatch 2024-04-20T17:49:43.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08705005309889222]}]: dispatch 2024-04-20T17:49:43.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08781439502854103]}]: dispatch 2024-04-20T17:49:43.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08689718471296246]}]: dispatch 2024-04-20T17:49:43.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08705005309889222]}]: dispatch 2024-04-20T17:49:43.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08679527245567595]}]: dispatch 2024-04-20T17:49:43.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08781439502854103]}]: dispatch 2024-04-20T17:49:43.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08672247798618558]}]: dispatch 2024-04-20T17:49:43.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08662541869353177]}]: dispatch 2024-04-20T17:49:43.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:42 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08689718471296246]}]: dispatch 2024-04-20T17:49:43.327 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:49:43.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:43.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:49:43.432 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:49:43.472 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:43.503 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:49:43.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:43.508 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:49:43.609 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:49:43.642 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:43.783 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:43 smithi171 ceph-mon[44189]: pgmap v2410: 27 pgs: 27 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 119 MiB/s wr, 1.91k op/s 2024-04-20T17:49:43.784 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08934307888783866]}]': finished 2024-04-20T17:49:43.784 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08672247798618558]}]': finished 2024-04-20T17:49:43.784 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08781439502854103]}]': finished 2024-04-20T17:49:43.784 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08705005309889222]}]': finished 2024-04-20T17:49:43.784 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08781439502854103]}]': finished 2024-04-20T17:49:43.784 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08689718471296246]}]': finished 2024-04-20T17:49:43.784 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08705005309889222]}]': finished 2024-04-20T17:49:43.784 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08679527245567595]}]': finished 2024-04-20T17:49:43.784 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08781439502854103]}]': finished 2024-04-20T17:49:43.784 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08672247798618558]}]': finished 2024-04-20T17:49:43.784 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08662541869353177]}]': finished 2024-04-20T17:49:43.784 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08689718471296246]}]': finished 2024-04-20T17:49:43.784 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:43 smithi171 ceph-mon[44189]: osdmap e913: 12 total, 12 up, 12 in 2024-04-20T17:49:43.784 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:43 smithi171 ceph-mon[44189]: osdmap e914: 12 total, 12 up, 12 in 2024-04-20T17:49:43.863 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 37 16 97844 97828 165.228 110.812 0.00883672 0.00605057 2024-04-20T17:49:43.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:43 smithi005 ceph-mon[39562]: pgmap v2410: 27 pgs: 27 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 119 MiB/s wr, 1.91k op/s 2024-04-20T17:49:43.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08934307888783866]}]': finished 2024-04-20T17:49:43.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08672247798618558]}]': finished 2024-04-20T17:49:43.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08781439502854103]}]': finished 2024-04-20T17:49:43.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08705005309889222]}]': finished 2024-04-20T17:49:43.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08781439502854103]}]': finished 2024-04-20T17:49:43.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08689718471296246]}]': finished 2024-04-20T17:49:43.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08705005309889222]}]': finished 2024-04-20T17:49:43.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08679527245567595]}]': finished 2024-04-20T17:49:43.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08781439502854103]}]': finished 2024-04-20T17:49:43.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08672247798618558]}]': finished 2024-04-20T17:49:43.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08662541869353177]}]': finished 2024-04-20T17:49:43.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08689718471296246]}]': finished 2024-04-20T17:49:43.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:43 smithi005 ceph-mon[39562]: osdmap e913: 12 total, 12 up, 12 in 2024-04-20T17:49:43.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:43 smithi005 ceph-mon[39562]: osdmap e914: 12 total, 12 up, 12 in 2024-04-20T17:49:44.107 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:49:44.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:44.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:49:44.215 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:49:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:43 smithi106 ceph-mon[44423]: pgmap v2410: 27 pgs: 27 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 119 MiB/s wr, 1.91k op/s 2024-04-20T17:49:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.08934307888783866]}]': finished 2024-04-20T17:49:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08672247798618558]}]': finished 2024-04-20T17:49:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08781439502854103]}]': finished 2024-04-20T17:49:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08705005309889222]}]': finished 2024-04-20T17:49:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08781439502854103]}]': finished 2024-04-20T17:49:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08689718471296246]}]': finished 2024-04-20T17:49:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08705005309889222]}]': finished 2024-04-20T17:49:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08679527245567595]}]': finished 2024-04-20T17:49:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08781439502854103]}]': finished 2024-04-20T17:49:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08672247798618558]}]': finished 2024-04-20T17:49:44.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08662541869353177]}]': finished 2024-04-20T17:49:44.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08689718471296246]}]': finished 2024-04-20T17:49:44.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:43 smithi106 ceph-mon[44423]: osdmap e913: 12 total, 12 up, 12 in 2024-04-20T17:49:44.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:43 smithi106 ceph-mon[44423]: osdmap e914: 12 total, 12 up, 12 in 2024-04-20T17:49:44.271 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:44.399 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:49:44.539 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:49:44.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:44.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:49:44.647 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:49:44.679 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:44.701 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:49:44.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:44.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:49:44.807 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:49:44.845 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:44.862 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 38 16 99595 99579 163.76 109.438 0.00437584 0.00610505 2024-04-20T17:49:44.875 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:49:44.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:44.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:49:44.981 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:49:45.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:44 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:49:45.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:44 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:49:45.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:44 smithi106 ceph-mon[44423]: osdmap e915: 12 total, 12 up, 12 in 2024-04-20T17:49:45.019 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:45.049 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:49:45.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:45.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:49:45.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:44 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:49:45.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:44 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:49:45.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:44 smithi171 ceph-mon[44189]: osdmap e915: 12 total, 12 up, 12 in 2024-04-20T17:49:45.158 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:49:45.192 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:45.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:44 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:49:45.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:44 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:49:45.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:44 smithi005 ceph-mon[39562]: osdmap e915: 12 total, 12 up, 12 in 2024-04-20T17:49:45.224 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:49:45.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:45.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:49:45.328 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:49:45.363 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:45.389 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:49:45.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:45.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:49:45.495 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:49:45.545 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:45.571 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:49:45.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:45.578 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:49:45.678 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:49:45.716 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:45.775 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:49:45.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:45.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:49:45.863 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 39 16 101309 101293 162.307 107.125 0.00788536 0.00615899 2024-04-20T17:49:45.880 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:49:45.920 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:45.960 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:49:45.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:45.965 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:49:46.066 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:49:46.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:45 smithi171 ceph-mon[44189]: pgmap v2413: 27 pgs: 1 remapped+peering, 26 active+clean; 5.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 81 MiB/s wr, 1.30k op/s; 59 KiB/s, 0 objects/s recovering 2024-04-20T17:49:46.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:45 smithi171 ceph-mon[44189]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-20T17:49:46.101 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:45 smithi005 ceph-mon[39562]: pgmap v2413: 27 pgs: 1 remapped+peering, 26 active+clean; 5.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 81 MiB/s wr, 1.30k op/s; 59 KiB/s, 0 objects/s recovering 2024-04-20T17:49:46.101 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:45 smithi005 ceph-mon[39562]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-20T17:49:46.106 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:45 smithi106 ceph-mon[44423]: pgmap v2413: 27 pgs: 1 remapped+peering, 26 active+clean; 5.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 81 MiB/s wr, 1.30k op/s; 59 KiB/s, 0 objects/s recovering 2024-04-20T17:49:46.106 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:45 smithi106 ceph-mon[44423]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-20T17:49:46.108 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:46.136 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:49:46.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:46.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:49:46.243 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:49:46.284 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:46.322 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:49:46.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:46.326 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:49:46.427 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:49:46.464 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:46.487 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:49:46.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:46.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:49:46.595 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:49:46.634 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:46.738 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:49:46.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:46.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:49:46.851 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:49:46.862 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:49:46.863654 min lat: 0.00101366 max lat: 0.200169 avg lat: 0.00621896 2024-04-20T17:49:46.863 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:49:46.863 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 40 16 102907 102891 160.746 99.875 0.00369324 0.00621896 2024-04-20T17:49:46.886 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:46.921 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:49:46.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:46.930 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:49:46.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:46 smithi005 ceph-mon[39562]: osdmap e916: 12 total, 12 up, 12 in 2024-04-20T17:49:46.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:46 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:49:47.030 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:49:47.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:46 smithi171 ceph-mon[44189]: osdmap e916: 12 total, 12 up, 12 in 2024-04-20T17:49:47.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:46 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:49:47.101 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:47.159 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:49:47.159 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:49:47.159 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:49:47.159 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:49:47.159 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:49:47.159 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:49:47.159 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:49:47.159 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:49:47.167 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:49:47.167 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:49:47.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:47.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:49:47.196 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:49:47.236 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:49:47.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:46 smithi106 ceph-mon[44423]: osdmap e916: 12 total, 12 up, 12 in 2024-04-20T17:49:47.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:46 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:49:47.278 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:49:47.281 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:49:47.284 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:49:47.284 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:49:47.284 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:49:47.284 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:49:47.284 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:49:47.284 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:49:47.284 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:49:47.284 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:49:47.284 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:49:47.284 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:49:47.284 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:49:47.285 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:49:47.331 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:47.370 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:49:47.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:47.376 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:49:47.477 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:49:47.520 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:47.557 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:49:47.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:47.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:49:47.668 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:49:47.710 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:47.739 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:49:47.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:47.742 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:49:47.811 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:47 smithi106 ceph-mon[44423]: pgmap v2416: 27 pgs: 1 remapped+peering, 26 active+clean; 6.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 114 MiB/s wr, 1.83k op/s; 59 KiB/s, 0 objects/s recovering 2024-04-20T17:49:47.811 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:47 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3068160034' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:49:47.811 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:47 smithi106 ceph-mon[44423]: osdmap e917: 12 total, 12 up, 12 in 2024-04-20T17:49:47.812 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:47.812 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:47.812 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:47.812 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:47.812 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:47.812 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:47.812 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:47 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2776607820' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:49:47.812 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:47 smithi106 ceph-mon[44423]: from='client.28145 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:49:47.812 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:47.812 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:47 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:49:47.812 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:47 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:49:47.812 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:47 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:49:47.827 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.1 to 1 (10000) 2024-04-20T17:49:47.843 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:49:47.866 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 41 16 104349 104333 159.013 90.125 0.0126302 0.00628626 2024-04-20T17:49:47.887 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:47 smithi171 ceph-mon[44189]: pgmap v2416: 27 pgs: 1 remapped+peering, 26 active+clean; 6.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 114 MiB/s wr, 1.83k op/s; 59 KiB/s, 0 objects/s recovering 2024-04-20T17:49:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:47 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3068160034' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:49:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:47 smithi171 ceph-mon[44189]: osdmap e917: 12 total, 12 up, 12 in 2024-04-20T17:49:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:47 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2776607820' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:49:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:47 smithi171 ceph-mon[44189]: from='client.28145 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:49:48.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:48.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:47 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:49:48.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:47 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:49:48.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:47 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:49:48.144 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:49:48.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:47 smithi005 ceph-mon[39562]: pgmap v2416: 27 pgs: 1 remapped+peering, 26 active+clean; 6.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 114 MiB/s wr, 1.83k op/s; 59 KiB/s, 0 objects/s recovering 2024-04-20T17:49:48.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:47 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3068160034' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:49:48.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:47 smithi005 ceph-mon[39562]: osdmap e917: 12 total, 12 up, 12 in 2024-04-20T17:49:48.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:48.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:48.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:48.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:48.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:48.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:48.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:47 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2776607820' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:49:48.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:47 smithi005 ceph-mon[39562]: from='client.28145 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:49:48.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:48.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:47 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:49:48.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:47 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:49:48.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:47 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:49:48.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:48.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:49:48.257 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:49:48.300 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:48.326 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:49:48.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:48.333 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:49:48.434 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:49:48.478 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:48.504 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:49:48.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:48.508 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:49:48.609 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:49:48.646 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:48.682 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:49:48.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:48.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:49:48.787 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:49:48.824 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:48.850 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:49:48.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:48.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:49:48.865 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 42 16 105956 105940 157.619 100.438 0.00764575 0.00634292 2024-04-20T17:49:48.955 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:49:48.998 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:49.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:49.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:49.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:49.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:49.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:48 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:49.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:48 smithi106 ceph-mon[44423]: 1.3 scrub starts 2024-04-20T17:49:49.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:48 smithi106 ceph-mon[44423]: 1.3 scrub ok 2024-04-20T17:49:49.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:48 smithi106 ceph-mon[44423]: from='client.28145 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-20T17:49:49.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:48 smithi106 ceph-mon[44423]: osdmap e918: 12 total, 12 up, 12 in 2024-04-20T17:49:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:48 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:48 smithi171 ceph-mon[44189]: 1.3 scrub starts 2024-04-20T17:49:49.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:48 smithi171 ceph-mon[44189]: 1.3 scrub ok 2024-04-20T17:49:49.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:48 smithi171 ceph-mon[44189]: from='client.28145 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-20T17:49:49.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:48 smithi171 ceph-mon[44189]: osdmap e918: 12 total, 12 up, 12 in 2024-04-20T17:49:49.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:49.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:49.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:49.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:49.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:48 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:49:49.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:48 smithi005 ceph-mon[39562]: 1.3 scrub starts 2024-04-20T17:49:49.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:48 smithi005 ceph-mon[39562]: 1.3 scrub ok 2024-04-20T17:49:49.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:48 smithi005 ceph-mon[39562]: from='client.28145 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-20T17:49:49.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:48 smithi005 ceph-mon[39562]: osdmap e918: 12 total, 12 up, 12 in 2024-04-20T17:49:49.333 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 6 1 2024-04-20T17:49:49.451 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:49:49.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:49.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:49:49.556 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:49:49.592 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:49.627 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:49:49.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:49.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:49:49.665 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:49:49.665 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:49:49.665 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:49:49.665 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:49:49.665 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.665 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:174728 7.6 7:6d3f70d5:::benchmark_data_smithi171_161871_object29002:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:49:49.665 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:43.086395+0000", 2024-04-20T17:49:49.666 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 546.57497984099996, 2024-04-20T17:49:49.666 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.27433470999999998, 2024-04-20T17:49:49.666 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:49.666 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:49.666 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:49.666 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:49:49.666 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:49:49.666 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:49:49.666 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 174728 2024-04-20T17:49:49.666 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.666 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:49.666 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.666 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:49.667 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.086395+0000", 2024-04-20T17:49:49.667 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.667 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.667 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.667 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:49.667 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.086395+0000", 2024-04-20T17:49:49.667 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.667 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.667 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.667 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:49.667 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.086398+0000", 2024-04-20T17:49:49.667 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3240000000000001e-06 2024-04-20T17:49:49.667 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.667 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.668 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:49.668 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.086404+0000", 2024-04-20T17:49:49.668 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.511e-06 2024-04-20T17:49:49.668 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.668 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.668 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:49.668 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.086418+0000", 2024-04-20T17:49:49.668 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3489000000000001e-05 2024-04-20T17:49:49.668 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.668 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.668 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:49.668 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.086425+0000", 2024-04-20T17:49:49.669 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.588e-06 2024-04-20T17:49:49.669 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.669 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.669 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.669 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.168960+0000", 2024-04-20T17:49:49.669 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.082534244000000007 2024-04-20T17:49:49.669 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.669 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.669 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:49:49.669 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.169012+0000", 2024-04-20T17:49:49.669 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2469000000000002e-05 2024-04-20T17:49:49.669 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.669 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.669 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.670 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.294480+0000", 2024-04-20T17:49:49.670 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.125467459 2024-04-20T17:49:49.670 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.670 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.670 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:49.670 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.294584+0000", 2024-04-20T17:49:49.670 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00010476000000000001 2024-04-20T17:49:49.670 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.670 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.670 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:49:49.671 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.294709+0000", 2024-04-20T17:49:49.671 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000124511 2024-04-20T17:49:49.671 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.671 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.671 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:49:49.671 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.354962+0000", 2024-04-20T17:49:49.671 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.060252992999999998 2024-04-20T17:49:49.671 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.671 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.671 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:49.671 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.358801+0000", 2024-04-20T17:49:49.671 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0038393580000000002 2024-04-20T17:49:49.671 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.671 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.672 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:49.672 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.358839+0000", 2024-04-20T17:49:49.672 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7851000000000001e-05 2024-04-20T17:49:49.672 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.672 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.672 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:49.672 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.360730+0000", 2024-04-20T17:49:49.672 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001891153 2024-04-20T17:49:49.672 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.672 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:49.672 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.672 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.672 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.672 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:175098 7.6 7:61cd04e5:::benchmark_data_smithi171_161871_object29372:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:49:49.673 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:43.559347+0000", 2024-04-20T17:49:49.673 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 546.10202865899998, 2024-04-20T17:49:49.673 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.26388687399999999, 2024-04-20T17:49:49.673 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:49.673 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:49.673 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:49.673 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:49:49.673 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:49:49.673 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:49:49.673 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 175098 2024-04-20T17:49:49.673 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.673 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:49.673 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.673 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:49.673 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.559347+0000", 2024-04-20T17:49:49.674 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.674 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.674 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.674 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:49.674 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.559347+0000", 2024-04-20T17:49:49.674 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.674 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.674 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.674 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:49.674 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.559349+0000", 2024-04-20T17:49:49.674 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0339999999999999e-06 2024-04-20T17:49:49.674 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.674 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.674 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:49.675 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.559354+0000", 2024-04-20T17:49:49.675 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7139999999999998e-06 2024-04-20T17:49:49.675 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.675 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.675 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:49.675 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.559365+0000", 2024-04-20T17:49:49.675 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0481e-05 2024-04-20T17:49:49.675 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.675 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.675 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:49.675 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.559373+0000", 2024-04-20T17:49:49.675 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0979999999999994e-06 2024-04-20T17:49:49.675 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.675 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.675 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.676 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.626555+0000", 2024-04-20T17:49:49.676 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.067182424000000004 2024-04-20T17:49:49.676 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.676 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.676 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:49:49.676 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.626864+0000", 2024-04-20T17:49:49.676 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00030837399999999999 2024-04-20T17:49:49.676 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.676 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.676 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.676 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.727481+0000", 2024-04-20T17:49:49.676 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.100617676 2024-04-20T17:49:49.676 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.676 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.676 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:49.677 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.727576+0000", 2024-04-20T17:49:49.677 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4534e-05 2024-04-20T17:49:49.677 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.677 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.677 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:49:49.677 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.727713+0000", 2024-04-20T17:49:49.677 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000137348 2024-04-20T17:49:49.677 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.677 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.677 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:49:49.677 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.750990+0000", 2024-04-20T17:49:49.677 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023276538999999999 2024-04-20T17:49:49.677 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.677 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.677 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:49.678 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.819232+0000", 2024-04-20T17:49:49.678 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.068242066000000004 2024-04-20T17:49:49.678 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.678 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.678 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:49.678 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.819312+0000", 2024-04-20T17:49:49.678 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0139e-05 2024-04-20T17:49:49.678 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.678 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.678 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:49.678 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.823234+0000", 2024-04-20T17:49:49.678 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0039214469999999998 2024-04-20T17:49:49.678 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.678 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:49.679 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.679 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.679 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.679 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:175104 7.6 7:6b18dfe6:::benchmark_data_smithi171_161871_object29378:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:49:49.679 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:43.566043+0000", 2024-04-20T17:49:49.679 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 546.09533242999998, 2024-04-20T17:49:49.679 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.25980681500000002, 2024-04-20T17:49:49.679 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:49.679 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:49.679 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:49.679 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:49:49.679 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:49:49.679 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:49:49.679 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 175104 2024-04-20T17:49:49.679 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.680 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:49.680 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.680 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:49.680 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.566043+0000", 2024-04-20T17:49:49.680 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.680 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.680 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.680 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:49.680 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.566043+0000", 2024-04-20T17:49:49.680 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.680 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.680 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.680 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:49.680 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.566045+0000", 2024-04-20T17:49:49.681 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0310000000000001e-06 2024-04-20T17:49:49.681 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.681 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.681 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:49.681 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.566050+0000", 2024-04-20T17:49:49.681 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3530000000000002e-06 2024-04-20T17:49:49.681 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.681 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.681 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:49.681 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.566061+0000", 2024-04-20T17:49:49.681 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0805000000000001e-05 2024-04-20T17:49:49.681 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.681 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.681 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:49.682 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.566068+0000", 2024-04-20T17:49:49.682 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.3980000000000002e-06 2024-04-20T17:49:49.682 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.682 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.682 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.682 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.626905+0000", 2024-04-20T17:49:49.682 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.060836928999999998 2024-04-20T17:49:49.682 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.682 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.682 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:49:49.682 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.627159+0000", 2024-04-20T17:49:49.682 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00025357199999999998 2024-04-20T17:49:49.682 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.682 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.683 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.683 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.740635+0000", 2024-04-20T17:49:49.683 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.113475665 2024-04-20T17:49:49.683 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.683 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.683 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:49.683 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.740733+0000", 2024-04-20T17:49:49.683 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8189999999999993e-05 2024-04-20T17:49:49.683 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.683 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.683 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:49:49.683 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.740887+0000", 2024-04-20T17:49:49.683 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015461699999999999 2024-04-20T17:49:49.683 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.683 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.684 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:49:49.684 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.754744+0000", 2024-04-20T17:49:49.684 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013856952000000001 2024-04-20T17:49:49.684 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.684 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.684 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:49.684 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.823306+0000", 2024-04-20T17:49:49.684 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.068561936000000004 2024-04-20T17:49:49.684 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.684 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.684 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:49.684 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.823366+0000", 2024-04-20T17:49:49.684 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9786e-05 2024-04-20T17:49:49.684 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.685 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.685 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:49.685 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.825850+0000", 2024-04-20T17:49:49.685 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0024835809999999999 2024-04-20T17:49:49.685 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.685 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:49.685 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.685 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.685 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.685 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:175140 7.6 7:6c38f614:::benchmark_data_smithi171_161871_object29414:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:49:49.685 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:43.661848+0000", 2024-04-20T17:49:49.685 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 545.99952709000002, 2024-04-20T17:49:49.685 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.27917723, 2024-04-20T17:49:49.685 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:49.685 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:49.685 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:49.686 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:49:49.686 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:49:49.686 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:49:49.686 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 175140 2024-04-20T17:49:49.686 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.686 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:49.686 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.686 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:49.686 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.661848+0000", 2024-04-20T17:49:49.686 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.686 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.686 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.686 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:49.686 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.661848+0000", 2024-04-20T17:49:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.661849+0000", 2024-04-20T17:49:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3e-06 2024-04-20T17:49:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.661854+0000", 2024-04-20T17:49:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7960000000000003e-06 2024-04-20T17:49:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:49.687 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.661863+0000", 2024-04-20T17:49:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.4330000000000004e-06 2024-04-20T17:49:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.661870+0000", 2024-04-20T17:49:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8970000000000002e-06 2024-04-20T17:49:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.661929+0000", 2024-04-20T17:49:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9574000000000003e-05 2024-04-20T17:49:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:49:49.688 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.661979+0000", 2024-04-20T17:49:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9489e-05 2024-04-20T17:49:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.866488+0000", 2024-04-20T17:49:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.20450923200000001 2024-04-20T17:49:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.866586+0000", 2024-04-20T17:49:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8498999999999993e-05 2024-04-20T17:49:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:49:49.689 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.866722+0000", 2024-04-20T17:49:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000135971 2024-04-20T17:49:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:49:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.891713+0000", 2024-04-20T17:49:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.024990188999999999 2024-04-20T17:49:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.937246+0000", 2024-04-20T17:49:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.045532975000000003 2024-04-20T17:49:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.690 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.937298+0000", 2024-04-20T17:49:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2438000000000002e-05 2024-04-20T17:49:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.941025+0000", 2024-04-20T17:49:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0037274370000000001 2024-04-20T17:49:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:175172 7.6 7:690a41fc:::benchmark_data_smithi171_161871_object29446:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:49:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:43.710664+0000", 2024-04-20T17:49:49.691 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 545.95071164499996, 2024-04-20T17:49:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.39938321900000001, 2024-04-20T17:49:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:49:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:49:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:49:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 175172 2024-04-20T17:49:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.710664+0000", 2024-04-20T17:49:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.692 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.710664+0000", 2024-04-20T17:49:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.710666+0000", 2024-04-20T17:49:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0499999999999999e-06 2024-04-20T17:49:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.710672+0000", 2024-04-20T17:49:49.693 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5880000000000003e-06 2024-04-20T17:49:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.710684+0000", 2024-04-20T17:49:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1552999999999999e-05 2024-04-20T17:49:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.710692+0000", 2024-04-20T17:49:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2069999999999995e-06 2024-04-20T17:49:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.717275+0000", 2024-04-20T17:49:49.694 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0065826540000000003 2024-04-20T17:49:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:49:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.717341+0000", 2024-04-20T17:49:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6616000000000001e-05 2024-04-20T17:49:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.968974+0000", 2024-04-20T17:49:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.25163312700000001 2024-04-20T17:49:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:49.695 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.969036+0000", 2024-04-20T17:49:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1395000000000004e-05 2024-04-20T17:49:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:49:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.969115+0000", 2024-04-20T17:49:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8769000000000004e-05 2024-04-20T17:49:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:49:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.018597+0000", 2024-04-20T17:49:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.049482313999999999 2024-04-20T17:49:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:49.696 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.106589+0000", 2024-04-20T17:49:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.087992206000000003 2024-04-20T17:49:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.106646+0000", 2024-04-20T17:49:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6569e-05 2024-04-20T17:49:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.110047+0000", 2024-04-20T17:49:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0034011710000000001 2024-04-20T17:49:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.697 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:175268 7.6 7:6d024280:::benchmark_data_smithi171_161871_object29542:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:49:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:43.846739+0000", 2024-04-20T17:49:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 545.814635963, 2024-04-20T17:49:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.40914125000000001, 2024-04-20T17:49:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:49:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:49:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:49:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 175268 2024-04-20T17:49:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.698 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.846739+0000", 2024-04-20T17:49:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.846739+0000", 2024-04-20T17:49:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.846741+0000", 2024-04-20T17:49:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1440000000000001e-06 2024-04-20T17:49:49.699 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.846747+0000", 2024-04-20T17:49:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9490000000000001e-06 2024-04-20T17:49:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.846761+0000", 2024-04-20T17:49:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3105000000000001e-05 2024-04-20T17:49:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.846769+0000", 2024-04-20T17:49:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2349999999999999e-06 2024-04-20T17:49:49.700 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.957854+0000", 2024-04-20T17:49:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.11108511 2024-04-20T17:49:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:49:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.957917+0000", 2024-04-20T17:49:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3489000000000001e-05 2024-04-20T17:49:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.154458+0000", 2024-04-20T17:49:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.196540989 2024-04-20T17:49:49.701 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.702 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.702 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:49.702 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.154549+0000", 2024-04-20T17:49:49.702 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0999000000000001e-05 2024-04-20T17:49:49.702 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.702 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.702 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:49:49.702 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.154678+0000", 2024-04-20T17:49:49.702 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012838699999999999 2024-04-20T17:49:49.702 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.702 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.702 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:49:49.702 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.177898+0000", 2024-04-20T17:49:49.702 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.023219961000000001 2024-04-20T17:49:49.702 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.251484+0000", 2024-04-20T17:49:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.073586043000000004 2024-04-20T17:49:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.251550+0000", 2024-04-20T17:49:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5897999999999998e-05 2024-04-20T17:49:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.255881+0000", 2024-04-20T17:49:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0043309409999999996 2024-04-20T17:49:49.703 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDPGPushReply(7.6 739/727 [PushReplyOp(7:6d024280:::benchmark_data_smithi171_161871_object29542:head)])", 2024-04-20T17:49:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:43.870312+0000", 2024-04-20T17:49:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 545.79106361799995, 2024-04-20T17:49:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.28409749299999998, 2024-04-20T17:49:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "reached pg", 2024-04-20T17:49:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:49.704 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.870312+0000", 2024-04-20T17:49:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.870312+0000", 2024-04-20T17:49:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.870312+0000", 2024-04-20T17:49:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1699999999999998e-07 2024-04-20T17:49:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.705 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.870315+0000", 2024-04-20T17:49:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0539999999999999e-06 2024-04-20T17:49:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.870321+0000", 2024-04-20T17:49:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9039999999999997e-06 2024-04-20T17:49:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:43.870326+0000", 2024-04-20T17:49:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3109999999999996e-06 2024-04-20T17:49:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.706 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.146964+0000", 2024-04-20T17:49:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.27663748500000002 2024-04-20T17:49:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.154409+0000", 2024-04-20T17:49:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0074452219999999996 2024-04-20T17:49:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:175504 7.6 7:6a999528:::benchmark_data_smithi171_161871_object29778:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:49:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:44.181312+0000", 2024-04-20T17:49:49.707 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 545.48006365200001, 2024-04-20T17:49:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.30309401499999999, 2024-04-20T17:49:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:49:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:49:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:49:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 175504 2024-04-20T17:49:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.181312+0000", 2024-04-20T17:49:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.708 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.181312+0000", 2024-04-20T17:49:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.181314+0000", 2024-04-20T17:49:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9460000000000001e-06 2024-04-20T17:49:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:49.709 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.181320+0000", 2024-04-20T17:49:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3249999999999996e-06 2024-04-20T17:49:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.181331+0000", 2024-04-20T17:49:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1309e-05 2024-04-20T17:49:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.181339+0000", 2024-04-20T17:49:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0199999999999994e-06 2024-04-20T17:49:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.710 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.300051+0000", 2024-04-20T17:49:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.11871208599999999 2024-04-20T17:49:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:49:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.300093+0000", 2024-04-20T17:49:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1304000000000003e-05 2024-04-20T17:49:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.309233+0000", 2024-04-20T17:49:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0091401450000000006 2024-04-20T17:49:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.711 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.309296+0000", 2024-04-20T17:49:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2829000000000004e-05 2024-04-20T17:49:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:49:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.309371+0000", 2024-04-20T17:49:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5797999999999995e-05 2024-04-20T17:49:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:49:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.338833+0000", 2024-04-20T17:49:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.02946174 2024-04-20T17:49:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.712 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.390289+0000", 2024-04-20T17:49:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.051455555 2024-04-20T17:49:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.480101+0000", 2024-04-20T17:49:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.089811821999999999 2024-04-20T17:49:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.484406+0000", 2024-04-20T17:49:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0043051360000000002 2024-04-20T17:49:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.713 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:175517 7.6 7:63cbdd03:::benchmark_data_smithi171_161871_object29791:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:49:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:44.194403+0000", 2024-04-20T17:49:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 545.46697205600003, 2024-04-20T17:49:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.29416489000000001, 2024-04-20T17:49:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:49:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:49:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:49:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 175517 2024-04-20T17:49:49.714 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.194403+0000", 2024-04-20T17:49:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.194403+0000", 2024-04-20T17:49:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.194405+0000", 2024-04-20T17:49:49.715 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1610000000000001e-06 2024-04-20T17:49:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.194413+0000", 2024-04-20T17:49:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8930000000000005e-06 2024-04-20T17:49:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.194424+0000", 2024-04-20T17:49:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0822000000000001e-05 2024-04-20T17:49:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.194431+0000", 2024-04-20T17:49:49.716 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2439999999999996e-06 2024-04-20T17:49:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.300133+0000", 2024-04-20T17:49:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.105701933 2024-04-20T17:49:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:49:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.300543+0000", 2024-04-20T17:49:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00041010599999999998 2024-04-20T17:49:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.338880+0000", 2024-04-20T17:49:49.717 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.038336984999999997 2024-04-20T17:49:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.338946+0000", 2024-04-20T17:49:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5179999999999996e-05 2024-04-20T17:49:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:49:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.339032+0000", 2024-04-20T17:49:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6305999999999997e-05 2024-04-20T17:49:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:49:49.718 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.361912+0000", 2024-04-20T17:49:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.022880144000000002 2024-04-20T17:49:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.484538+0000", 2024-04-20T17:49:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.122625494 2024-04-20T17:49:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.484613+0000", 2024-04-20T17:49:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5603000000000001e-05 2024-04-20T17:49:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.719 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.488568+0000", 2024-04-20T17:49:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0039550189999999997 2024-04-20T17:49:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:175528 7.6 7:65985805:::benchmark_data_smithi171_161871_object29802:head [delete] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-20T17:49:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:44.205896+0000", 2024-04-20T17:49:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 545.45547896699998, 2024-04-20T17:49:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.28621254600000001, 2024-04-20T17:49:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:49.720 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:49:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:49:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:49:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 175528 2024-04-20T17:49:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.205896+0000", 2024-04-20T17:49:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.205896+0000", 2024-04-20T17:49:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.721 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.205898+0000", 2024-04-20T17:49:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4500000000000001e-06 2024-04-20T17:49:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.205903+0000", 2024-04-20T17:49:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.22e-06 2024-04-20T17:49:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.205912+0000", 2024-04-20T17:49:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2289999999999992e-06 2024-04-20T17:49:49.722 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.205918+0000", 2024-04-20T17:49:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4079999999999997e-06 2024-04-20T17:49:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.300591+0000", 2024-04-20T17:49:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.094673560000000004 2024-04-20T17:49:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:49:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.300911+0000", 2024-04-20T17:49:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00031990799999999999 2024-04-20T17:49:49.723 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.351523+0000", 2024-04-20T17:49:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.050611695999999998 2024-04-20T17:49:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.351631+0000", 2024-04-20T17:49:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000107939 2024-04-20T17:49:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:49:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.351815+0000", 2024-04-20T17:49:49.724 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00018422499999999999 2024-04-20T17:49:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:49:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.365991+0000", 2024-04-20T17:49:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014175953999999999 2024-04-20T17:49:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.488632+0000", 2024-04-20T17:49:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.12264077299999999 2024-04-20T17:49:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.488669+0000", 2024-04-20T17:49:49.725 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.7326000000000001e-05 2024-04-20T17:49:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:44.492109+0000", 2024-04-20T17:49:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0034398580000000001 2024-04-20T17:49:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:178953 7.6 7:606eac2a:::benchmark_data_smithi171_161871_object33227:head [delete] snapc 0=[] ondisk+write+known_if_redirected e740)", 2024-04-20T17:49:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:48.415871+0000", 2024-04-20T17:49:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 541.24550440099995, 2024-04-20T17:49:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.25897584000000001, 2024-04-20T17:49:49.726 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:49:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:49:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:49:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 178953 2024-04-20T17:49:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.415871+0000", 2024-04-20T17:49:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.415871+0000", 2024-04-20T17:49:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.415873+0000", 2024-04-20T17:49:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0159999999999998e-06 2024-04-20T17:49:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.415880+0000", 2024-04-20T17:49:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7050000000000002e-06 2024-04-20T17:49:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.728 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.415895+0000", 2024-04-20T17:49:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5432999999999999e-05 2024-04-20T17:49:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.415905+0000", 2024-04-20T17:49:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8050000000000001e-06 2024-04-20T17:49:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.555843+0000", 2024-04-20T17:49:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.13993809600000001 2024-04-20T17:49:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:49:49.729 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.556259+0000", 2024-04-20T17:49:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00041587800000000001 2024-04-20T17:49:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.599376+0000", 2024-04-20T17:49:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.043117474000000003 2024-04-20T17:49:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.599447+0000", 2024-04-20T17:49:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.0522e-05 2024-04-20T17:49:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:49:49.730 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.599537+0000", 2024-04-20T17:49:49.731 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0161000000000006e-05 2024-04-20T17:49:49.731 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.731 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.731 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:49:49.731 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.639257+0000", 2024-04-20T17:49:49.731 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.039719800999999999 2024-04-20T17:49:49.731 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.731 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.731 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:49.731 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.672742+0000", 2024-04-20T17:49:49.731 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.033484977999999999 2024-04-20T17:49:49.731 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.731 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.731 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:49.731 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.672773+0000", 2024-04-20T17:49:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0936000000000003e-05 2024-04-20T17:49:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:48.674847+0000", 2024-04-20T17:49:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0020740350000000001 2024-04-20T17:49:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:180664 7.6 7:6ef0e8cc:::benchmark_data_smithi171_161871_object34938:head [delete] snapc 0=[] ondisk+write+known_if_redirected e741)", 2024-04-20T17:49:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:50.558557+0000", 2024-04-20T17:49:49.732 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 539.10281793499996, 2024-04-20T17:49:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.25715610300000002, 2024-04-20T17:49:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:49:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:49:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:49:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 180664 2024-04-20T17:49:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.558557+0000", 2024-04-20T17:49:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.733 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.558557+0000", 2024-04-20T17:49:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.558560+0000", 2024-04-20T17:49:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3980000000000002e-06 2024-04-20T17:49:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.558566+0000", 2024-04-20T17:49:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.2870000000000001e-06 2024-04-20T17:49:49.734 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.558575+0000", 2024-04-20T17:49:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2990000000000002e-06 2024-04-20T17:49:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.558587+0000", 2024-04-20T17:49:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1182000000000001e-05 2024-04-20T17:49:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.577424+0000", 2024-04-20T17:49:49.735 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.018837962999999999 2024-04-20T17:49:49.736 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.736 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.736 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:49:49.736 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.577488+0000", 2024-04-20T17:49:49.736 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3625000000000002e-05 2024-04-20T17:49:49.736 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.736 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.736 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.736 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.722386+0000", 2024-04-20T17:49:49.736 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.144897575 2024-04-20T17:49:49.736 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.736 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.736 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:49.736 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.729577+0000", 2024-04-20T17:49:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0071915729999999997 2024-04-20T17:49:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:49:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.729707+0000", 2024-04-20T17:49:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012936299999999999 2024-04-20T17:49:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:49:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.811940+0000", 2024-04-20T17:49:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.082233443000000003 2024-04-20T17:49:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:49.737 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.812042+0000", 2024-04-20T17:49:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000102364 2024-04-20T17:49:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.812099+0000", 2024-04-20T17:49:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6715000000000002e-05 2024-04-20T17:49:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:50.815713+0000", 2024-04-20T17:49:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0036143159999999998 2024-04-20T17:49:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:49.738 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:181747 7.6 7:63cd038d:::benchmark_data_smithi171_161871_object36021:head [delete] snapc 0=[] ondisk+write+known_if_redirected e741)", 2024-04-20T17:49:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:51.929299+0000", 2024-04-20T17:49:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 537.73207604799995, 2024-04-20T17:49:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.29627274799999997, 2024-04-20T17:49:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:49:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:49:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:49:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 181747 2024-04-20T17:49:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.739 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:51.929299+0000", 2024-04-20T17:49:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:51.929299+0000", 2024-04-20T17:49:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:51.929302+0000", 2024-04-20T17:49:49.740 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.582e-06 2024-04-20T17:49:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:51.929314+0000", 2024-04-20T17:49:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2160999999999999e-05 2024-04-20T17:49:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:51.929330+0000", 2024-04-20T17:49:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6144999999999999e-05 2024-04-20T17:49:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:51.929341+0000", 2024-04-20T17:49:49.741 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0392e-05 2024-04-20T17:49:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:51.999407+0000", 2024-04-20T17:49:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.070066656000000005 2024-04-20T17:49:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:49:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:51.999449+0000", 2024-04-20T17:49:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2276e-05 2024-04-20T17:49:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.742 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.103493+0000", 2024-04-20T17:49:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.104043096 2024-04-20T17:49:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.103578+0000", 2024-04-20T17:49:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5651999999999999e-05 2024-04-20T17:49:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:49:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.103704+0000", 2024-04-20T17:49:49.743 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000125592 2024-04-20T17:49:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:49:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.119570+0000", 2024-04-20T17:49:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015865707 2024-04-20T17:49:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.221746+0000", 2024-04-20T17:49:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.102176936 2024-04-20T17:49:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.221812+0000", 2024-04-20T17:49:49.744 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5470999999999997e-05 2024-04-20T17:49:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.225572+0000", 2024-04-20T17:49:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0037600820000000001 2024-04-20T17:49:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:182127 7.6 7:6598bc27:::benchmark_data_smithi171_161871_object36401:head [delete] snapc 0=[] ondisk+write+known_if_redirected e741)", 2024-04-20T17:49:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:52.408315+0000", 2024-04-20T17:49:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 537.25306059100001, 2024-04-20T17:49:49.745 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.27155902300000001, 2024-04-20T17:49:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:49:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:49:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:49:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 182127 2024-04-20T17:49:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.408315+0000", 2024-04-20T17:49:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.746 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.408315+0000", 2024-04-20T17:49:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.408318+0000", 2024-04-20T17:49:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2160000000000002e-06 2024-04-20T17:49:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.408326+0000", 2024-04-20T17:49:49.747 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5369999999999997e-06 2024-04-20T17:49:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.408340+0000", 2024-04-20T17:49:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.401e-05 2024-04-20T17:49:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.408351+0000", 2024-04-20T17:49:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0458e-05 2024-04-20T17:49:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.748 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.505668+0000", 2024-04-20T17:49:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.097316620000000006 2024-04-20T17:49:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:49:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.506051+0000", 2024-04-20T17:49:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00038314000000000002 2024-04-20T17:49:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.644953+0000", 2024-04-20T17:49:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.13890235100000001 2024-04-20T17:49:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.749 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.645053+0000", 2024-04-20T17:49:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9555000000000005e-05 2024-04-20T17:49:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:49:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.645184+0000", 2024-04-20T17:49:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013101599999999999 2024-04-20T17:49:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:49:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.651423+0000", 2024-04-20T17:49:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0062393400000000003 2024-04-20T17:49:49.750 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.676555+0000", 2024-04-20T17:49:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.025131715999999998 2024-04-20T17:49:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.676589+0000", 2024-04-20T17:49:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4586999999999999e-05 2024-04-20T17:49:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.679874+0000", 2024-04-20T17:49:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0032844770000000001 2024-04-20T17:49:49.751 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:182476 7.6 7:63007d39:::benchmark_data_smithi171_161871_object36750:head [delete] snapc 0=[] ondisk+write+known_if_redirected e741)", 2024-04-20T17:49:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:52.860619+0000", 2024-04-20T17:49:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 536.80075635799994, 2024-04-20T17:49:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.27225396400000001, 2024-04-20T17:49:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:49:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:49:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:49:49.752 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 182476 2024-04-20T17:49:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.860619+0000", 2024-04-20T17:49:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.860619+0000", 2024-04-20T17:49:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:49.753 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.860622+0000", 2024-04-20T17:49:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0089999999999999e-06 2024-04-20T17:49:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.860629+0000", 2024-04-20T17:49:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.4499999999999998e-06 2024-04-20T17:49:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.860647+0000", 2024-04-20T17:49:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7773000000000001e-05 2024-04-20T17:49:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:49.754 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.860657+0000", 2024-04-20T17:49:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0006999999999999e-05 2024-04-20T17:49:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.879198+0000", 2024-04-20T17:49:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.018541067000000001 2024-04-20T17:49:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:49:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.879258+0000", 2024-04-20T17:49:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0188e-05 2024-04-20T17:49:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.755 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.021562+0000", 2024-04-20T17:49:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.14230343300000001 2024-04-20T17:49:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.021656+0000", 2024-04-20T17:49:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4246000000000005e-05 2024-04-20T17:49:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:49:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.021802+0000", 2024-04-20T17:49:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000145539 2024-04-20T17:49:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.756 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:49:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.058021+0000", 2024-04-20T17:49:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.036219121999999999 2024-04-20T17:49:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.128996+0000", 2024-04-20T17:49:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.070975521 2024-04-20T17:49:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.129063+0000", 2024-04-20T17:49:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6241999999999994e-05 2024-04-20T17:49:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.757 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.132873+0000", 2024-04-20T17:49:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0038103669999999998 2024-04-20T17:49:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:182477 7.6 7:6df68048:::benchmark_data_smithi171_161871_object36751:head [delete] snapc 0=[] ondisk+write+known_if_redirected e741)", 2024-04-20T17:49:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:52.861320+0000", 2024-04-20T17:49:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 536.80005547899998, 2024-04-20T17:49:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.27478792000000002, 2024-04-20T17:49:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:49:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:49:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:49:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 182477 2024-04-20T17:49:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.861320+0000", 2024-04-20T17:49:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.861320+0000", 2024-04-20T17:49:49.759 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.861321+0000", 2024-04-20T17:49:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3599999999999999e-06 2024-04-20T17:49:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.861326+0000", 2024-04-20T17:49:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.7960000000000003e-06 2024-04-20T17:49:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.861340+0000", 2024-04-20T17:49:49.760 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4229999999999999e-05 2024-04-20T17:49:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.861349+0000", 2024-04-20T17:49:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0329999999999997e-06 2024-04-20T17:49:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.995885+0000", 2024-04-20T17:49:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.134535565 2024-04-20T17:49:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:49:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.997397+0000", 2024-04-20T17:49:49.761 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0015126079999999999 2024-04-20T17:49:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.047463+0000", 2024-04-20T17:49:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.050065655000000001 2024-04-20T17:49:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.047530+0000", 2024-04-20T17:49:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7113000000000006e-05 2024-04-20T17:49:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:49:49.762 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.047607+0000", 2024-04-20T17:49:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6542e-05 2024-04-20T17:49:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:49:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.068691+0000", 2024-04-20T17:49:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.021084094000000001 2024-04-20T17:49:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.132951+0000", 2024-04-20T17:49:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.064260566000000005 2024-04-20T17:49:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.133018+0000", 2024-04-20T17:49:49.763 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.7068000000000002e-05 2024-04-20T17:49:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.136108+0000", 2024-04-20T17:49:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0030892900000000002 2024-04-20T17:49:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:182483 7.6 7:68dbf85b:::benchmark_data_smithi171_161871_object36757:head [delete] snapc 0=[] ondisk+write+known_if_redirected e741)", 2024-04-20T17:49:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:52.869238+0000", 2024-04-20T17:49:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 536.79213716599997, 2024-04-20T17:49:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.270150002, 2024-04-20T17:49:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:49.764 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:49:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:49:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:49:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 182483 2024-04-20T17:49:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.869238+0000", 2024-04-20T17:49:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.765 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.869238+0000", 2024-04-20T17:49:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.869241+0000", 2024-04-20T17:49:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.418e-06 2024-04-20T17:49:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.869246+0000", 2024-04-20T17:49:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.429e-06 2024-04-20T17:49:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.766 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.869261+0000", 2024-04-20T17:49:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4912000000000001e-05 2024-04-20T17:49:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.869270+0000", 2024-04-20T17:49:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1880000000000008e-06 2024-04-20T17:49:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.997440+0000", 2024-04-20T17:49:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.128169965 2024-04-20T17:49:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.767 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:49:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.997815+0000", 2024-04-20T17:49:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00037503599999999998 2024-04-20T17:49:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.058066+0000", 2024-04-20T17:49:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.060250808000000003 2024-04-20T17:49:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.058126+0000", 2024-04-20T17:49:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9808000000000001e-05 2024-04-20T17:49:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.768 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:49:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.058203+0000", 2024-04-20T17:49:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6874999999999998e-05 2024-04-20T17:49:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:49:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.078821+0000", 2024-04-20T17:49:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.020618926999999999 2024-04-20T17:49:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.136189+0000", 2024-04-20T17:49:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057367920000000003 2024-04-20T17:49:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.769 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.136251+0000", 2024-04-20T17:49:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1742999999999995e-05 2024-04-20T17:49:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.139388+0000", 2024-04-20T17:49:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0031369729999999999 2024-04-20T17:49:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:182484 7.6 7:67cc5f5a:::benchmark_data_smithi171_161871_object36758:head [delete] snapc 0=[] ondisk+write+known_if_redirected e741)", 2024-04-20T17:49:49.770 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:40:52.870902+0000", 2024-04-20T17:49:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 536.79047297800003, 2024-04-20T17:49:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.271107971, 2024-04-20T17:49:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:49:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:49:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:49:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 182484 2024-04-20T17:49:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.870902+0000", 2024-04-20T17:49:49.771 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.870902+0000", 2024-04-20T17:49:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.870904+0000", 2024-04-20T17:49:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4169999999999999e-06 2024-04-20T17:49:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:49.772 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.870908+0000", 2024-04-20T17:49:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1119999999999998e-06 2024-04-20T17:49:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.870917+0000", 2024-04-20T17:49:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.7700000000000007e-06 2024-04-20T17:49:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.870924+0000", 2024-04-20T17:49:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7249999999999997e-06 2024-04-20T17:49:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.997850+0000", 2024-04-20T17:49:49.773 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.12692612 2024-04-20T17:49:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:49:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:52.998178+0000", 2024-04-20T17:49:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00032771599999999998 2024-04-20T17:49:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.068745+0000", 2024-04-20T17:49:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.070566738000000004 2024-04-20T17:49:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.068837+0000", 2024-04-20T17:49:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.2014000000000004e-05 2024-04-20T17:49:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:49:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.068929+0000", 2024-04-20T17:49:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.1618000000000003e-05 2024-04-20T17:49:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:49:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.095299+0000", 2024-04-20T17:49:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.026370050999999999 2024-04-20T17:49:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:49.775 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.139463+0000", 2024-04-20T17:49:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.044164876999999998 2024-04-20T17:49:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.139528+0000", 2024-04-20T17:49:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4930000000000003e-05 2024-04-20T17:49:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:40:53.142010+0000", 2024-04-20T17:49:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0024818829999999998 2024-04-20T17:49:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.776 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:205309 7.6 7:64cf33f4:::benchmark_data_smithi171_161871_object59583:head [delete] snapc 0=[] ondisk+write+known_if_redirected e745)", 2024-04-20T17:49:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:41:20.783215+0000", 2024-04-20T17:49:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 508.87815999499998, 2024-04-20T17:49:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.265927152, 2024-04-20T17:49:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:49:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:49:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:49:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 205309 2024-04-20T17:49:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.777 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:49.778 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.778 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:49.778 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:20.783215+0000", 2024-04-20T17:49:49.778 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.778 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.778 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.778 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:49.778 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:20.783215+0000", 2024-04-20T17:49:49.778 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.778 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.778 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.778 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:49.778 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:20.783217+0000", 2024-04-20T17:49:49.778 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9230000000000001e-06 2024-04-20T17:49:49.778 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.779 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.779 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:49.779 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:20.783223+0000", 2024-04-20T17:49:49.779 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1539999999999999e-06 2024-04-20T17:49:49.779 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.779 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.779 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:49.779 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:20.783236+0000", 2024-04-20T17:49:49.779 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2816000000000001e-05 2024-04-20T17:49:49.779 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.779 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.779 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:49.779 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:20.783244+0000", 2024-04-20T17:49:49.779 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5780000000000001e-06 2024-04-20T17:49:49.779 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.779 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.780 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.780 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:20.790554+0000", 2024-04-20T17:49:49.780 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0073102159999999996 2024-04-20T17:49:49.780 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.780 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.780 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:49:49.780 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:20.790594+0000", 2024-04-20T17:49:49.780 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.0160000000000002e-05 2024-04-20T17:49:49.780 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.780 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.780 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.780 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:20.976126+0000", 2024-04-20T17:49:49.780 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.18553197599999999 2024-04-20T17:49:49.780 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.780 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.780 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:49.780 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:20.976229+0000", 2024-04-20T17:49:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000102815 2024-04-20T17:49:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:49:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:20.976381+0000", 2024-04-20T17:49:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015173199999999999 2024-04-20T17:49:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:49:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:20.990421+0000", 2024-04-20T17:49:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.014040166999999999 2024-04-20T17:49:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:21.046014+0000", 2024-04-20T17:49:49.781 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.055593208999999998 2024-04-20T17:49:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:21.046069+0000", 2024-04-20T17:49:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4607000000000001e-05 2024-04-20T17:49:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:21.049142+0000", 2024-04-20T17:49:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0030737989999999999 2024-04-20T17:49:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.782 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27088.0:210138 7.6 7:6703f753:::benchmark_data_smithi171_161871_object64412:head [delete] snapc 0=[] ondisk+write+known_if_redirected e747)", 2024-04-20T17:49:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:41:26.392472+0000", 2024-04-20T17:49:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 503.268903696, 2024-04-20T17:49:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.26567666200000001, 2024-04-20T17:49:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:49:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:49:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:49:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:49:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27088", 2024-04-20T17:49:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/799960370", 2024-04-20T17:49:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 210138 2024-04-20T17:49:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:49:49.783 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:49:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.392472+0000", 2024-04-20T17:49:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:49:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.392472+0000", 2024-04-20T17:49:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:49:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:49:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.392473+0000", 2024-04-20T17:49:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.559e-06 2024-04-20T17:49:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.784 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:49:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.392478+0000", 2024-04-20T17:49:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5719999999999996e-06 2024-04-20T17:49:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:49:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.392487+0000", 2024-04-20T17:49:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0129999999999999e-06 2024-04-20T17:49:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:49:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.392492+0000", 2024-04-20T17:49:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3909999999999997e-06 2024-04-20T17:49:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.785 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.489582+0000", 2024-04-20T17:49:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.097089462000000001 2024-04-20T17:49:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for degraded object", 2024-04-20T17:49:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.489627+0000", 2024-04-20T17:49:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5420000000000002e-05 2024-04-20T17:49:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:49:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.599066+0000", 2024-04-20T17:49:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.109438903 2024-04-20T17:49:49.786 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:49:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.599129+0000", 2024-04-20T17:49:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.3441999999999993e-05 2024-04-20T17:49:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 3", 2024-04-20T17:49:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.599215+0000", 2024-04-20T17:49:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.5260999999999996e-05 2024-04-20T17:49:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:49:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.612840+0000", 2024-04-20T17:49:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.013625671000000001 2024-04-20T17:49:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.787 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:49:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.655297+0000", 2024-04-20T17:49:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.042456493999999997 2024-04-20T17:49:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:49:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.655358+0000", 2024-04-20T17:49:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1696e-05 2024-04-20T17:49:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:49:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:49:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:49:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:41:26.658148+0000", 2024-04-20T17:49:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0027897780000000001 2024-04-20T17:49:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:49.788 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.789 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:49:49.789 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:49:49.789 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:49:49.789 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:49:49.828 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:49.866 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 43 16 107605 107589 156.35 103.062 0.0123236 0.00639396 2024-04-20T17:49:49.874 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:49:49.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:49.878 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:49:49.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:49 smithi005 ceph-mon[39562]: 1.a scrub starts 2024-04-20T17:49:49.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:49 smithi005 ceph-mon[39562]: 1.a scrub ok 2024-04-20T17:49:49.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:49 smithi005 ceph-mon[39562]: pgmap v2419: 27 pgs: 27 active+clean; 6.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 127 MiB/s wr, 2.03k op/s 2024-04-20T17:49:49.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:49 smithi005 ceph-mon[39562]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-20T17:49:49.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:49 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:49:49.979 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:49:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:49 smithi106 ceph-mon[44423]: 1.a scrub starts 2024-04-20T17:49:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:49 smithi106 ceph-mon[44423]: 1.a scrub ok 2024-04-20T17:49:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:49 smithi106 ceph-mon[44423]: pgmap v2419: 27 pgs: 27 active+clean; 6.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 127 MiB/s wr, 2.03k op/s 2024-04-20T17:49:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:49 smithi106 ceph-mon[44423]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-20T17:49:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:49 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:49:50.021 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:50.056 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:49:50.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:50.062 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:49:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:49 smithi171 ceph-mon[44189]: 1.a scrub starts 2024-04-20T17:49:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:49 smithi171 ceph-mon[44189]: 1.a scrub ok 2024-04-20T17:49:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:49 smithi171 ceph-mon[44189]: pgmap v2419: 27 pgs: 27 active+clean; 6.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 127 MiB/s wr, 2.03k op/s 2024-04-20T17:49:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:49 smithi171 ceph-mon[44189]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-20T17:49:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:49 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:49:50.163 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:49:50.194 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:50.223 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:49:50.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:50.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:49:50.328 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:49:50.379 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:50.405 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:49:50.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:50.408 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:49:50.509 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:49:50.543 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:50.571 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:49:50.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:50.575 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:49:50.663 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-20T17:49:50.676 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:49:50.721 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:50.755 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:49:50.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:50.765 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:49:50.866 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 44 16 109189 109173 155.046 99 0.00996519 0.00644836 2024-04-20T17:49:50.866 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:49:50.899 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:50 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:49:50.899 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:50 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:49:50.903 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:50.971 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:49:50.985 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:49:50.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:50.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:49:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:50 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:49:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:50 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:49:51.062 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:49:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:50 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:49:51.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:50 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:49:51.091 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:49:51.128 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:51.265 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:49:51.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:51.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:49:51.367 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:49:51.400 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:51.434 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:49:51.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:51.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:49:51.537 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:49:51.579 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:51.832 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:51 smithi005 ceph-mon[39562]: pgmap v2420: 27 pgs: 27 active+clean; 6.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 129 MiB/s wr, 2.06k op/s 2024-04-20T17:49:51.866 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 45 16 110745 110729 153.761 97.25 0.0119622 0.00650238 2024-04-20T17:49:51.903 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:49:51.934 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:49:51.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:51.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:49:52.039 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:49:52.083 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:51 smithi171 ceph-mon[44189]: pgmap v2420: 27 pgs: 27 active+clean; 6.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 129 MiB/s wr, 2.06k op/s 2024-04-20T17:49:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:51 smithi171 ceph-mon[44189]: osdmap e919: 12 total, 12 up, 12 in 2024-04-20T17:49:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:51 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:49:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:51 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:49:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:51 smithi171 ceph-mon[44189]: osdmap e920: 12 total, 12 up, 12 in 2024-04-20T17:49:52.127 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:49:52.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:52.135 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:49:52.161 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:51 smithi005 ceph-mon[39562]: osdmap e919: 12 total, 12 up, 12 in 2024-04-20T17:49:52.161 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:51 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:49:52.161 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:51 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:49:52.161 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:51 smithi005 ceph-mon[39562]: osdmap e920: 12 total, 12 up, 12 in 2024-04-20T17:49:52.236 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:49:52.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:51 smithi106 ceph-mon[44423]: pgmap v2420: 27 pgs: 27 active+clean; 6.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 129 MiB/s wr, 2.06k op/s 2024-04-20T17:49:52.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:51 smithi106 ceph-mon[44423]: osdmap e919: 12 total, 12 up, 12 in 2024-04-20T17:49:52.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:51 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:49:52.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:51 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:49:52.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:51 smithi106 ceph-mon[44423]: osdmap e920: 12 total, 12 up, 12 in 2024-04-20T17:49:52.282 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:52.866 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 46 16 112311 112295 152.546 97.875 0.00449209 0.00655391 2024-04-20T17:49:53.017 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:49:53.439 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:49:53.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:53.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:49:53.549 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:49:53.601 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:53.642 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:49:53.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:53.649 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:49:53.749 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:49:53.795 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:53.819 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:49:53.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:53.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:49:53.866 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 47 16 113863 113847 151.364 97 0.00896183 0.00660514 2024-04-20T17:49:53.928 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:49:53.966 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:54.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:53 smithi171 ceph-mon[44189]: pgmap v2423: 27 pgs: 27 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.53k op/s 2024-04-20T17:49:54.119 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:49:54.124 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:53 smithi106 ceph-mon[44423]: pgmap v2423: 27 pgs: 27 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.53k op/s 2024-04-20T17:49:54.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:54.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:49:54.126 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:53 smithi005 ceph-mon[39562]: pgmap v2423: 27 pgs: 27 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.53k op/s 2024-04-20T17:49:54.185 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:49:54.185 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:49:54.185 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.186 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.27772.0:114327 9.4 e920/894)", 2024-04-20T17:49:54.186 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:54.184960+0000", 2024-04-20T17:49:54.186 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.00027669800000000003, 2024-04-20T17:49:54.186 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00029468000000000002, 2024-04-20T17:49:54.186 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:54.186 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:54.186 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "queued for pg", 2024-04-20T17:49:54.186 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:54.186 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.186 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:54.186 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.184960+0000", 2024-04-20T17:49:54.186 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:54.186 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.186 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.187 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:54.187 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.184960+0000", 2024-04-20T17:49:54.187 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:54.187 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.187 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.187 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:54.187 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.184962+0000", 2024-04-20T17:49:54.187 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.44e-06 2024-04-20T17:49:54.187 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.187 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.187 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:54.187 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.184967+0000", 2024-04-20T17:49:54.187 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2050000000000001e-06 2024-04-20T17:49:54.187 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.188 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.188 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:54.188 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.184976+0000", 2024-04-20T17:49:54.188 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2830000000000007e-06 2024-04-20T17:49:54.188 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.188 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.188 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:54.188 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.184982+0000", 2024-04-20T17:49:54.188 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6080000000000003e-06 2024-04-20T17:49:54.188 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:54.188 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:54.188 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:54.188 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.188 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.188 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.27772.0:114330 9.4 e920/894)", 2024-04-20T17:49:54.189 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:54.184991+0000", 2024-04-20T17:49:54.189 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.00024631499999999999, 2024-04-20T17:49:54.189 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00029453500000000001, 2024-04-20T17:49:54.189 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:54.189 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:54.189 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "queued for pg", 2024-04-20T17:49:54.189 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:54.189 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.189 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:54.189 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.184991+0000", 2024-04-20T17:49:54.189 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:54.189 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.189 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.189 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:54.189 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.184991+0000", 2024-04-20T17:49:54.190 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:54.190 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.190 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.190 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:54.190 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.184991+0000", 2024-04-20T17:49:54.190 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.34e-07 2024-04-20T17:49:54.190 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.190 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.190 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:54.190 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.184993+0000", 2024-04-20T17:49:54.190 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.852e-06 2024-04-20T17:49:54.190 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.190 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.190 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:54.191 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.184996+0000", 2024-04-20T17:49:54.191 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8289999999999999e-06 2024-04-20T17:49:54.191 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.191 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.191 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:54.191 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.184998+0000", 2024-04-20T17:49:54.191 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.57e-06 2024-04-20T17:49:54.191 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:54.191 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:54.191 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:54.191 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.191 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.191 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:114327 9.4 9:225c3730:::benchmark_data_smithi171_188319_object114326:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e920)", 2024-04-20T17:49:54.191 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:54.171279+0000", 2024-04-20T17:49:54.191 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.013957762, 2024-04-20T17:49:54.192 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.014039571000000001, 2024-04-20T17:49:54.192 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:54.192 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:54.192 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:49:54.192 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:49:54.192 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:49:54.192 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:49:54.192 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 114327 2024-04-20T17:49:54.192 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.192 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:54.192 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.192 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:54.192 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.171279+0000", 2024-04-20T17:49:54.192 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:54.192 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.193 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.193 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:54.193 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.171279+0000", 2024-04-20T17:49:54.193 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:54.193 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.193 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.193 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:54.193 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.171282+0000", 2024-04-20T17:49:54.193 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5059999999999998e-06 2024-04-20T17:49:54.193 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.193 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.193 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:54.193 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.171366+0000", 2024-04-20T17:49:54.193 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4011000000000006e-05 2024-04-20T17:49:54.193 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.194 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.194 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:54.194 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.171381+0000", 2024-04-20T17:49:54.194 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5341000000000001e-05 2024-04-20T17:49:54.194 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.194 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.194 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:54.194 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.171392+0000", 2024-04-20T17:49:54.194 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1025e-05 2024-04-20T17:49:54.194 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.194 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.194 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:54.194 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.171475+0000", 2024-04-20T17:49:54.194 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3201999999999994e-05 2024-04-20T17:49:54.194 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.195 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:54.195 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.171616+0000", 2024-04-20T17:49:54.195 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000140455 2024-04-20T17:49:54.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.195 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:49:54.195 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.171755+0000", 2024-04-20T17:49:54.195 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000139369 2024-04-20T17:49:54.196 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.196 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.196 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:49:54.196 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.180837+0000", 2024-04-20T17:49:54.196 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0090818229999999993 2024-04-20T17:49:54.196 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:54.196 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:54.196 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:54.196 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.196 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.196 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:114330 9.4 9:28f713cf:::benchmark_data_smithi171_188319_object114329:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e920)", 2024-04-20T17:49:54.196 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:54.171419+0000", 2024-04-20T17:49:54.196 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.01381754, 2024-04-20T17:49:54.196 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.013950369000000001, 2024-04-20T17:49:54.196 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:54.197 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:54.197 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:49:54.197 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:49:54.197 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:49:54.197 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:49:54.197 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 114330 2024-04-20T17:49:54.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.197 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:54.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.197 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:54.197 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.171419+0000", 2024-04-20T17:49:54.197 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:54.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.197 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:54.197 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.171419+0000", 2024-04-20T17:49:54.198 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:54.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.198 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:54.198 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.171421+0000", 2024-04-20T17:49:54.198 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.091e-06 2024-04-20T17:49:54.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.198 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:54.198 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.171702+0000", 2024-04-20T17:49:54.198 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00028098799999999997 2024-04-20T17:49:54.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.199 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:54.199 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.171710+0000", 2024-04-20T17:49:54.199 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6889999999999993e-06 2024-04-20T17:49:54.199 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.199 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.199 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:54.199 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.171718+0000", 2024-04-20T17:49:54.199 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2819999999999999e-06 2024-04-20T17:49:54.199 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.199 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.199 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:54.199 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.177618+0000", 2024-04-20T17:49:54.200 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0059006079999999999 2024-04-20T17:49:54.200 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.200 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.200 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:54.200 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.177734+0000", 2024-04-20T17:49:54.200 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000116155 2024-04-20T17:49:54.200 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.200 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.200 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:49:54.200 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.177851+0000", 2024-04-20T17:49:54.200 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000117064 2024-04-20T17:49:54.200 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:54.200 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:54.200 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:54.201 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.201 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.201 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:114331 9.4 9:28a281ec:::benchmark_data_smithi171_188319_object114330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e920)", 2024-04-20T17:49:54.201 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:54.171728+0000", 2024-04-20T17:49:54.201 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.013508642, 2024-04-20T17:49:54.201 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.013685502, 2024-04-20T17:49:54.201 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:54.201 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:54.201 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:49:54.201 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:49:54.201 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:49:54.201 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:49:54.201 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 114331 2024-04-20T17:49:54.201 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.201 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:54.202 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.202 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:54.202 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.171728+0000", 2024-04-20T17:49:54.202 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:54.202 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.202 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.202 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:54.202 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.171728+0000", 2024-04-20T17:49:54.202 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:54.202 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.202 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.202 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:54.202 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.171729+0000", 2024-04-20T17:49:54.202 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9600000000000008e-07 2024-04-20T17:49:54.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.203 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:54.203 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.171756+0000", 2024-04-20T17:49:54.203 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6999e-05 2024-04-20T17:49:54.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.203 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:54.203 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.171763+0000", 2024-04-20T17:49:54.203 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4289999999999998e-06 2024-04-20T17:49:54.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.203 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:54.203 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.171768+0000", 2024-04-20T17:49:54.203 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6840000000000001e-06 2024-04-20T17:49:54.204 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.204 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:54.204 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.180874+0000", 2024-04-20T17:49:54.204 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0091054699999999992 2024-04-20T17:49:54.204 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.204 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:54.204 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.181009+0000", 2024-04-20T17:49:54.204 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013496100000000001 2024-04-20T17:49:54.204 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.204 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:49:54.204 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.181147+0000", 2024-04-20T17:49:54.204 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000137737 2024-04-20T17:49:54.205 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:54.205 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:54.205 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:54.205 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.205 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.205 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27772.0:114329 9.c e920/894 9:34373bc9:::benchmark_data_smithi171_188319_object114328:head v 920'7229, mlcod=920'7226)", 2024-04-20T17:49:54.205 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:54.174340+0000", 2024-04-20T17:49:54.205 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.010896660000000001, 2024-04-20T17:49:54.205 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011115246000000001, 2024-04-20T17:49:54.205 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:54.205 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:54.205 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-04-20T17:49:54.205 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:54.205 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.206 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:54.206 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.174340+0000", 2024-04-20T17:49:54.206 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:54.206 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.206 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.206 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:54.206 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.174340+0000", 2024-04-20T17:49:54.206 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:54.206 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.206 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.206 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:54.206 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.174342+0000", 2024-04-20T17:49:54.206 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.26e-06 2024-04-20T17:49:54.206 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.206 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.207 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:54.207 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.174397+0000", 2024-04-20T17:49:54.207 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5167000000000002e-05 2024-04-20T17:49:54.207 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.207 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.207 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:54.207 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.174408+0000", 2024-04-20T17:49:54.207 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0883999999999999e-05 2024-04-20T17:49:54.207 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.207 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.207 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:54.207 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.174415+0000", 2024-04-20T17:49:54.207 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8580000000000002e-06 2024-04-20T17:49:54.207 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.208 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.208 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:54.208 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.177720+0000", 2024-04-20T17:49:54.208 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0033058689999999999 2024-04-20T17:49:54.208 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.208 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.208 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:54.208 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.177743+0000", 2024-04-20T17:49:54.208 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3003999999999999e-05 2024-04-20T17:49:54.208 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:54.208 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:54.208 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:54.208 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.208 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.208 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:114335 9.4 9:2369f19d:::benchmark_data_smithi171_188319_object114334:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e920)", 2024-04-20T17:49:54.209 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:54.174514+0000", 2024-04-20T17:49:54.209 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.010723046999999999, 2024-04-20T17:49:54.209 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010976705, 2024-04-20T17:49:54.209 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:54.209 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:54.209 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:49:54.209 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:49:54.209 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:49:54.209 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:49:54.209 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 114335 2024-04-20T17:49:54.209 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.209 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:54.209 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.209 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:54.209 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.174514+0000", 2024-04-20T17:49:54.210 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:54.210 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.210 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:54.210 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.174514+0000", 2024-04-20T17:49:54.210 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:54.210 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.210 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:54.210 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.174516+0000", 2024-04-20T17:49:54.210 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7230000000000001e-06 2024-04-20T17:49:54.210 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.210 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:54.210 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.174576+0000", 2024-04-20T17:49:54.211 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9837999999999999e-05 2024-04-20T17:49:54.211 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.211 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.211 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:54.211 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.174589+0000", 2024-04-20T17:49:54.211 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3811999999999999e-05 2024-04-20T17:49:54.211 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.211 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.211 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:54.211 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.174597+0000", 2024-04-20T17:49:54.211 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2779999999999996e-06 2024-04-20T17:49:54.211 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.211 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.211 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:54.212 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.184290+0000", 2024-04-20T17:49:54.212 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0096936069999999999 2024-04-20T17:49:54.212 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.212 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.212 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:54.212 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.184436+0000", 2024-04-20T17:49:54.212 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014537900000000001 2024-04-20T17:49:54.212 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.212 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.212 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:49:54.212 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.184578+0000", 2024-04-20T17:49:54.212 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014241899999999999 2024-04-20T17:49:54.212 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:54.212 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:54.212 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:54.213 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.213 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.213 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27772.0:114332 9.c e920/894 9:361fa932:::benchmark_data_smithi171_188319_object114331:head v 920'7230, mlcod=920'7226)", 2024-04-20T17:49:54.213 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:54.179182+0000", 2024-04-20T17:49:54.213 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0060552169999999999, 2024-04-20T17:49:54.213 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0063516359999999999, 2024-04-20T17:49:54.213 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:54.213 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:54.213 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-04-20T17:49:54.213 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:54.213 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.213 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:54.213 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.179182+0000", 2024-04-20T17:49:54.213 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:54.213 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.214 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.214 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:54.214 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.179182+0000", 2024-04-20T17:49:54.214 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:54.214 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.214 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.214 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:54.214 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.179184+0000", 2024-04-20T17:49:54.214 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.871e-06 2024-04-20T17:49:54.214 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.214 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.214 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:54.214 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.179255+0000", 2024-04-20T17:49:54.214 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1795e-05 2024-04-20T17:49:54.214 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.215 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.215 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:54.215 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.179270+0000", 2024-04-20T17:49:54.215 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.447e-05 2024-04-20T17:49:54.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.215 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.215 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:54.215 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.179278+0000", 2024-04-20T17:49:54.215 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2740000000000007e-06 2024-04-20T17:49:54.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.215 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.215 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:49:54.215 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.180856+0000", 2024-04-20T17:49:54.215 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001577838 2024-04-20T17:49:54.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.216 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.216 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:49:54.216 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.180874+0000", 2024-04-20T17:49:54.216 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7866000000000001e-05 2024-04-20T17:49:54.216 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:54.216 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:54.216 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:54.216 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.216 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.216 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:114338 9.4 9.32785044 (undecoded) ondisk+write+known_if_redirected e920)", 2024-04-20T17:49:54.216 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:54.179678+0000", 2024-04-20T17:49:54.216 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0055588080000000002, 2024-04-20T17:49:54.216 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0058879869999999999, 2024-04-20T17:49:54.216 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:49:54.216 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:49:54.217 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "queued for pg", 2024-04-20T17:49:54.217 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:49:54.217 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:49:54.217 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:49:54.217 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 114338 2024-04-20T17:49:54.217 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.217 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:49:54.217 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.217 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:49:54.217 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.179678+0000", 2024-04-20T17:49:54.217 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:54.217 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.217 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.217 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:49:54.217 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.179678+0000", 2024-04-20T17:49:54.218 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:49:54.218 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.218 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:49:54.218 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.179680+0000", 2024-04-20T17:49:54.218 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0099999999999998e-06 2024-04-20T17:49:54.218 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.218 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:49:54.218 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.179708+0000", 2024-04-20T17:49:54.218 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7787999999999999e-05 2024-04-20T17:49:54.218 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.218 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:49:54.218 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.179722+0000", 2024-04-20T17:49:54.219 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3614e-05 2024-04-20T17:49:54.219 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:49:54.219 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:49:54.219 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:49:54.219 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.179729+0000", 2024-04-20T17:49:54.219 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8420000000000005e-06 2024-04-20T17:49:54.219 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:54.219 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:49:54.219 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:54.219 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:49:54.219 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-04-20T17:49:54.219 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 9 2024-04-20T17:49:54.219 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:49:54.227 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:49:54.255 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:49:54.278 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:54.486 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:49:54.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:54.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:49:54.593 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:49:54.636 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:54.864 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:49:54.866 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 48 16 115410 115394 150.225 96.6875 0.0135075 0.00665531 2024-04-20T17:49:55.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:54 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2595278615' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:49:55.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:54 smithi106 ceph-mon[44423]: from='client.28154 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:49:55.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:54 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2595278615' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:49:55.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:54 smithi005 ceph-mon[39562]: from='client.28154 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:49:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:54 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2595278615' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:49:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:54 smithi171 ceph-mon[44189]: from='client.28154 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:49:55.703 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:49:55.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:55.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:49:55.718 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-20T17:49:55.807 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:49:55.849 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:55.866 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 49 15 116889 116874 149.047 92.5 0.0107071 0.00670753 2024-04-20T17:49:55.867 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.6 to 1 (10000) 2024-04-20T17:49:56.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:55 smithi106 ceph-mon[44423]: pgmap v2424: 27 pgs: 27 active+clean; 6.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s 2024-04-20T17:49:56.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:55 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:49:56.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:55 smithi106 ceph-mon[44423]: from='client.28154 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:49:56.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:55 smithi106 ceph-mon[44423]: osdmap e921: 12 total, 12 up, 12 in 2024-04-20T17:49:56.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:55 smithi106 ceph-mon[44423]: from='client.28160 v1:172.21.15.5:0/1101564537' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-20T17:49:56.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:55 smithi106 ceph-mon[44423]: from='client.28160 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-20T17:49:56.114 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:49:56.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:55 smithi005 ceph-mon[39562]: pgmap v2424: 27 pgs: 27 active+clean; 6.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s 2024-04-20T17:49:56.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:55 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:49:56.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:55 smithi005 ceph-mon[39562]: from='client.28154 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:49:56.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:55 smithi005 ceph-mon[39562]: osdmap e921: 12 total, 12 up, 12 in 2024-04-20T17:49:56.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:55 smithi005 ceph-mon[39562]: from='client.28160 v1:172.21.15.5:0/1101564537' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-20T17:49:56.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:55 smithi005 ceph-mon[39562]: from='client.28160 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-20T17:49:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:55 smithi171 ceph-mon[44189]: pgmap v2424: 27 pgs: 27 active+clean; 6.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s 2024-04-20T17:49:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:55 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:49:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:55 smithi171 ceph-mon[44189]: from='client.28154 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:49:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:55 smithi171 ceph-mon[44189]: osdmap e921: 12 total, 12 up, 12 in 2024-04-20T17:49:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:55 smithi171 ceph-mon[44189]: from='client.28160 v1:172.21.15.5:0/1101564537' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-20T17:49:56.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:55 smithi171 ceph-mon[44189]: from='client.28160 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-20T17:49:56.407 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:49:56.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:56.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:49:56.511 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:49:56.545 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:56.867 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 50 16 118442 118426 148.006 97 0.0053505 0.00675458 2024-04-20T17:49:57.208 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:49:57.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:56 smithi005 ceph-mon[39562]: from='client.28160 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-20T17:49:57.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:56 smithi005 ceph-mon[39562]: osdmap e922: 12 total, 12 up, 12 in 2024-04-20T17:49:57.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:56 smithi005 ceph-mon[39562]: pgmap v2427: 27 pgs: 27 active+clean; 7.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-04-20T17:49:57.212 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:56 smithi106 ceph-mon[44423]: from='client.28160 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-20T17:49:57.212 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:56 smithi106 ceph-mon[44423]: osdmap e922: 12 total, 12 up, 12 in 2024-04-20T17:49:57.212 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:56 smithi106 ceph-mon[44423]: pgmap v2427: 27 pgs: 27 active+clean; 7.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-04-20T17:49:57.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:57.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:49:57.258 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 7 1 2024-04-20T17:49:57.314 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:49:57.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:56 smithi171 ceph-mon[44189]: from='client.28160 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-20T17:49:57.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:56 smithi171 ceph-mon[44189]: osdmap e922: 12 total, 12 up, 12 in 2024-04-20T17:49:57.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:56 smithi171 ceph-mon[44189]: pgmap v2427: 27 pgs: 27 active+clean; 7.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-04-20T17:49:57.357 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:57.394 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:49:57.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:57.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:49:57.501 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:49:57.549 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:57.576 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:49:57.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:57.581 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:49:57.682 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:49:57.722 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:57.758 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:49:57.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:57.763 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:49:57.864 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:49:57.867 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 51 16 119919 119903 146.914 92.3125 0.00787565 0.00680476 2024-04-20T17:49:57.909 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:57.938 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:49:57.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:57.951 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:49:58.052 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:49:58.091 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:58.119 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:49:58.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:58.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:49:58.224 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:49:58.279 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:58.306 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:49:58.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:58.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:49:58.411 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:49:58.454 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:58.486 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:49:58.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:58.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:49:58.592 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:49:58.634 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:58.758 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:49:58.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:58.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:49:58.794 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:49:58.803 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:49:58.803 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:49:58.803 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-04-20T17:49:58.803 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-04-20T17:49:58.804 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:49:58.865 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:49:58.867 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 52 16 121384 121368 145.849 91.5625 0.0193171 0.00685484 2024-04-20T17:49:58.900 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:59.699 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-20T17:49:59.758 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:49:59.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:59.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:49:59.865 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:49:59.866 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 53 16 122841 122825 144.816 91.0625 0.0143054 0.00690406 2024-04-20T17:49:59.907 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:49:59.938 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:49:59.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:49:59.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:49:59.943 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:49:59 smithi106 ceph-mon[44423]: pgmap v2428: 27 pgs: 27 active+clean; 7.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-20T17:50:00.044 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:50:00.082 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:49:59 smithi005 ceph-mon[39562]: pgmap v2428: 27 pgs: 27 active+clean; 7.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-20T17:50:00.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:49:59 smithi171 ceph-mon[44189]: pgmap v2428: 27 pgs: 27 active+clean; 7.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-20T17:50:00.085 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:00.087 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:50:00.277 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:50:00.648 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:50:00.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:00.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:50:00.761 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:50:00.809 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:00.834 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:50:00.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:00.838 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:50:00.867 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 54 16 124270 124254 143.788 89.3125 0.0147453 0.00695305 2024-04-20T17:50:00.940 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:50:00.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:00 smithi005 ceph-mon[39562]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-04-20T17:50:00.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:00 smithi005 ceph-mon[39562]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-04-20T17:50:00.980 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:00 smithi106 ceph-mon[44423]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-04-20T17:50:01.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:00 smithi106 ceph-mon[44423]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-04-20T17:50:01.069 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:00 smithi171 ceph-mon[44189]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-04-20T17:50:01.070 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:00 smithi171 ceph-mon[44189]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-04-20T17:50:01.088 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:50:01.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:01.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:50:01.194 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:50:01.238 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:01.639 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:50:01.664 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:50:01.695 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:50:01.696 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:50:01.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:01.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:50:01.728 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:50:01.767 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:50:01.790 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:50:01.802 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:50:01.818 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:50:01.846 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:50:01.849 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:01.867 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 55 16 125667 125651 142.761 87.3125 0.00834101 0.0070031 2024-04-20T17:50:01.881 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:50:01.881 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:50:01.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:01.886 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:50:01.916 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:50:01.945 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:50:01.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:01 smithi005 ceph-mon[39562]: pgmap v2429: 27 pgs: 27 active+clean; 7.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s wr, 1.62k op/s 2024-04-20T17:50:01.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:01 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4266130213' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:50:01.983 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:50:01.987 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:50:01.994 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:50:01.994 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:50:01.994 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:50:01.994 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:50:01.994 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:50:01.994 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:50:01.994 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:50:01.994 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:50:01.994 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:50:01.994 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:50:01.994 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:50:01.994 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:50:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:01 smithi106 ceph-mon[44423]: pgmap v2429: 27 pgs: 27 active+clean; 7.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s wr, 1.62k op/s 2024-04-20T17:50:02.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:01 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4266130213' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:50:02.039 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:01 smithi171 ceph-mon[44189]: pgmap v2429: 27 pgs: 27 active+clean; 7.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s wr, 1.62k op/s 2024-04-20T17:50:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:01 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4266130213' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:50:02.500 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:50:02.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:02.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:50:02.604 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:50:02.649 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:02.685 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:50:02.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:02.693 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:50:02.700 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:50:02.762 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:02.762 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:02.762 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:02.762 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:02.762 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:02.762 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:02.762 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:02.762 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:02.762 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:02.762 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:02.763 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:02 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:02.793 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:50:02.838 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:02.867 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 56 16 127058 127042 141.764 86.9375 0.00816272 0.00705248 2024-04-20T17:50:02.884 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:50:02.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:02.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:50:02.987 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:50:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:03.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:03.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:02 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:03.023 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:03.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:03.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:03.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:03.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:02 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:03.230 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:50:03.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:03.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:50:03.333 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:50:03.370 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:03.589 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:50:03.589 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:50:03.778 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.7 to 1 (10000) 2024-04-20T17:50:03.867 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 57 16 128486 128470 140.843 89.25 0.0152619 0.00709876 2024-04-20T17:50:03.917 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:03 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:03.918 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:03 smithi005 ceph-mon[39562]: pgmap v2430: 27 pgs: 27 active+clean; 7.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 104 MiB/s wr, 1.67k op/s 2024-04-20T17:50:03.918 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:03 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/338783277' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:50:03.918 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:03 smithi005 ceph-mon[39562]: from='client.28169 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:50:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:03 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:03 smithi106 ceph-mon[44423]: pgmap v2430: 27 pgs: 27 active+clean; 7.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 104 MiB/s wr, 1.67k op/s 2024-04-20T17:50:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:03 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/338783277' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:50:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:03 smithi106 ceph-mon[44423]: from='client.28169 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:50:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:03 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:03 smithi171 ceph-mon[44189]: pgmap v2430: 27 pgs: 27 active+clean; 7.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 104 MiB/s wr, 1.67k op/s 2024-04-20T17:50:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:03 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/338783277' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:50:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:03 smithi171 ceph-mon[44189]: from='client.28169 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:50:04.867 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 58 16 129712 129696 139.735 76.625 0.00986217 0.00715473 2024-04-20T17:50:04.926 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-20T17:50:04.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:04 smithi005 ceph-mon[39562]: from='client.28169 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-20T17:50:04.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:04 smithi005 ceph-mon[39562]: osdmap e923: 12 total, 12 up, 12 in 2024-04-20T17:50:05.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:04 smithi171 ceph-mon[44189]: from='client.28169 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-20T17:50:05.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:04 smithi171 ceph-mon[44189]: osdmap e923: 12 total, 12 up, 12 in 2024-04-20T17:50:05.133 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:50:05.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:05.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:50:05.239 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:50:05.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:04 smithi106 ceph-mon[44423]: from='client.28169 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-20T17:50:05.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:04 smithi106 ceph-mon[44423]: osdmap e923: 12 total, 12 up, 12 in 2024-04-20T17:50:05.283 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:05.289 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:50:05.640 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:50:05.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:05.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:50:05.682 INFO:tasks.thrashosds.thrasher:test_map_discontinuity 2024-04-20T17:50:05.682 INFO:tasks.thrashosds.thrasher:Waiting for recovery 2024-04-20T17:50:05.682 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-20T17:50:05.682 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-20T17:50:05.687 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:50:05.747 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:50:05.784 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:05.819 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:50:05.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:05.825 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:50:05.867 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 59 16 131085 131069 138.821 85.8125 0.00892413 0.00720112 2024-04-20T17:50:05.926 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:50:05.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:05 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:50:05.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:05 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:50:05.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:05 smithi005 ceph-mon[39562]: pgmap v2432: 27 pgs: 27 active+clean; 7.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-04-20T17:50:05.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:05 smithi005 ceph-mon[39562]: osdmap e924: 12 total, 12 up, 12 in 2024-04-20T17:50:05.967 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:06.000 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:50:06.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:06.007 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:50:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:05 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:50:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:05 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:50:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:05 smithi106 ceph-mon[44423]: pgmap v2432: 27 pgs: 27 active+clean; 7.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-04-20T17:50:06.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:05 smithi106 ceph-mon[44423]: osdmap e924: 12 total, 12 up, 12 in 2024-04-20T17:50:06.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:05 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:50:06.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:05 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:50:06.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:05 smithi171 ceph-mon[44189]: pgmap v2432: 27 pgs: 27 active+clean; 7.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-04-20T17:50:06.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:05 smithi171 ceph-mon[44189]: osdmap e924: 12 total, 12 up, 12 in 2024-04-20T17:50:06.108 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:50:06.143 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:06.378 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:50:06.386 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:50:06.600 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:50:06.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:06.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:50:06.705 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:50:06.744 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:06.795 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:50:06.867 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:50:06.868267 min lat: 0.00101366 max lat: 0.200169 avg lat: 0.00725824 2024-04-20T17:50:06.867 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:50:06.868 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 60 16 132279 132263 137.752 74.625 0.00515329 0.00725824 2024-04-20T17:50:06.898 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:06 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/516365936' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:50:07.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:06 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/516365936' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:50:07.179 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:06 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/516365936' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:50:07.867 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 61 16 133688 133672 136.937 88.0625 0.00914286 0.00730131 2024-04-20T17:50:07.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:07 smithi005 ceph-mon[39562]: pgmap v2434: 27 pgs: 27 active+clean; 7.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s 2024-04-20T17:50:07.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:07 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:50:07.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:07 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/516365936' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:50:07.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:07 smithi005 ceph-mon[39562]: osdmap e925: 12 total, 12 up, 12 in 2024-04-20T17:50:07.991 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:50:07.991 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:50:08.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:07 smithi106 ceph-mon[44423]: pgmap v2434: 27 pgs: 27 active+clean; 7.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s 2024-04-20T17:50:08.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:07 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:50:08.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:07 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/516365936' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:50:08.013 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:07 smithi106 ceph-mon[44423]: osdmap e925: 12 total, 12 up, 12 in 2024-04-20T17:50:08.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:07 smithi171 ceph-mon[44189]: pgmap v2434: 27 pgs: 27 active+clean; 7.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s 2024-04-20T17:50:08.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:07 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:50:08.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:07 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/516365936' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:50:08.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:07 smithi171 ceph-mon[44189]: osdmap e925: 12 total, 12 up, 12 in 2024-04-20T17:50:08.788 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-20T17:50:08.867 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 62 16 135096 135080 136.147 88 0.0149838 0.00734371 2024-04-20T17:50:09.167 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:50:09.509 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:50:09.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:09.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:50:09.616 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:50:09.661 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:09.867 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 63 16 136464 136448 135.344 85.5 0.00489108 0.00738706 2024-04-20T17:50:10.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:09 smithi171 ceph-mon[44189]: pgmap v2436: 27 pgs: 27 active+clean; 8.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-04-20T17:50:10.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:09 smithi005 ceph-mon[39562]: pgmap v2436: 27 pgs: 27 active+clean; 8.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-04-20T17:50:10.223 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:50:10.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:10.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:50:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:09 smithi106 ceph-mon[44423]: pgmap v2436: 27 pgs: 27 active+clean; 8.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-04-20T17:50:10.328 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:50:10.371 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:10.404 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:50:10.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:10.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:50:10.510 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:50:10.551 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:10.587 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:50:10.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:10.595 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:50:10.696 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:50:10.729 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:10.868 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 64 16 137779 137763 134.513 82.1875 0.00726912 0.007433 2024-04-20T17:50:10.899 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:50:10.924 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:50:10.953 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:50:10.987 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:50:11.042 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:50:11.059 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:50:11.073 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:50:11.107 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:50:11.147 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:50:11.188 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:50:11.223 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:50:11.249 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:50:11.249 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":925,"fsid":"13c6e76c-ff37-11ee-bc93-c7b262605968","created":"2024-04-20T16:58:25.272557+0000","modified":"2024-04-20T17:50:06.781692+0000","last_up_change":"2024-04-20T17:44:19.738797+0000","last_in_change":"2024-04-20T17:46:36.769424+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":34,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T17:01:35.909493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"838","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"818","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":0.9100000262260437,"raw_score_acting":2.7300000190734863,"raw_score_stable":2.7300000190734863,"primary_affinity_weighted":0.83594763278961182,"average_primary_affinity":0.9100000262260437,"average_primary_affinity_weighted":0.8399999737739563}},{"pool":9,"pool_name":"unique_pool_7","create_time":"2024-04-20T17:48:53.279237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"900","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.309999942779541,"score_stable":2.0699999332427979,"optimal_score":0.9100000262260437,"raw_score_acting":3.0099999904632568,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.83594763278961182,"average_primary_affinity":0.9100000262260437,"average_primary_affinity_weighted":0.8399999737739563,"error_message":"pool 9 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate"}}],"osds":[{"osd":0,"uuid":"ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":845,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6801","nonce":3901484705}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6802","nonce":3901484705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6804","nonce":3901484705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6803","nonce":3901484705}]},"public_addr":"172.21.15.5:6801/3901484705","cluster_addr":"172.21.15.5:6802/3901484705","heartbeat_back_addr":"172.21.15.5:6804/3901484705","heartbeat_front_addr":"172.21.15.5:6803/3901484705","state":["exists","up"]},{"osd":1,"uuid":"aed0c8eb-0e81-4e85-ab8f-a60420060656","up":1,"in":1,"weight":1,"primary_affinity":0.0313720703125,"last_clean_begin":13,"last_clean_end":613,"up_from":659,"up_thru":915,"down_at":614,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6805","nonce":2415652792}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6806","nonce":2415652792}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6808","nonce":2415652792}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6807","nonce":2415652792}]},"public_addr":"172.21.15.5:6805/2415652792","cluster_addr":"172.21.15.5:6806/2415652792","heartbeat_back_addr":"172.21.15.5:6808/2415652792","heartbeat_front_addr":"172.21.15.5:6807/2415652792","state":["exists","up"]},{"osd":2,"uuid":"f733a6a2-19c2-4c7a-af9c-04e282062bb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":20,"last_clean_end":169,"up_from":202,"up_thru":845,"down_at":170,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6809","nonce":2120275501}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6810","nonce":2120275501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6812","nonce":2120275501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6811","nonce":2120275501}]},"public_addr":"172.21.15.5:6809/2120275501","cluster_addr":"172.21.15.5:6810/2120275501","heartbeat_back_addr":"172.21.15.5:6812/2120275501","heartbeat_front_addr":"172.21.15.5:6811/2120275501","state":["exists","up"]},{"osd":3,"uuid":"8c92f1df-e298-4606-88ae-b8a8e79628a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":538,"up_from":561,"up_thru":916,"down_at":539,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6813","nonce":2272824046}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6814","nonce":2272824046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6816","nonce":2272824046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6815","nonce":2272824046}]},"public_addr":"172.21.15.5:6813/2272824046","cluster_addr":"172.21.15.5:6814/2272824046","heartbeat_back_addr":"172.21.15.5:6816/2272824046","heartbeat_front_addr":"172.21.15.5:6815/2272824046","state":["exists","up"]},{"osd":4,"uuid":"46df16f1-9607-4d7b-9be8-9d4236e160d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":894,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6800","nonce":1839524469}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6801","nonce":1839524469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6803","nonce":1839524469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6802","nonce":1839524469}]},"public_addr":"172.21.15.106:6800/1839524469","cluster_addr":"172.21.15.106:6801/1839524469","heartbeat_back_addr":"172.21.15.106:6803/1839524469","heartbeat_front_addr":"172.21.15.106:6802/1839524469","state":["exists","up"]},{"osd":5,"uuid":"6603231d-1e5c-4575-bfaa-2eeca2c12215","up":1,"in":1,"weight":1,"primary_affinity":0,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":894,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6804","nonce":2450315908}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6805","nonce":2450315908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6807","nonce":2450315908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6806","nonce":2450315908}]},"public_addr":"172.21.15.106:6804/2450315908","cluster_addr":"172.21.15.106:6805/2450315908","heartbeat_back_addr":"172.21.15.106:6807/2450315908","heartbeat_front_addr":"172.21.15.106:6806/2450315908","state":["exists","up"]},{"osd":6,"uuid":"783a55a6-f176-42be-8c8b-af8481d08dc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":48,"last_clean_end":503,"up_from":535,"up_thru":894,"down_at":504,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6808","nonce":2439382182}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6809","nonce":2439382182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6811","nonce":2439382182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6810","nonce":2439382182}]},"public_addr":"172.21.15.106:6808/2439382182","cluster_addr":"172.21.15.106:6809/2439382182","heartbeat_back_addr":"172.21.15.106:6811/2439382182","heartbeat_front_addr":"172.21.15.106:6810/2439382182","state":["exists","up"]},{"osd":7,"uuid":"b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":398,"last_clean_end":721,"up_from":799,"up_thru":894,"down_at":722,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6812","nonce":2531699401}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6813","nonce":2531699401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6815","nonce":2531699401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6814","nonce":2531699401}]},"public_addr":"172.21.15.106:6812/2531699401","cluster_addr":"172.21.15.106:6813/2531699401","heartbeat_back_addr":"172.21.15.106:6815/2531699401","heartbeat_front_addr":"172.21.15.106:6814/2531699401","state":["exists","up"]},{"osd":8,"uuid":"1cb232ad-1f59-4a0a-a215-e67419ad6476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":894,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6800","nonce":4123058942}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6801","nonce":4123058942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6803","nonce":4123058942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6802","nonce":4123058942}]},"public_addr":"172.21.15.171:6800/4123058942","cluster_addr":"172.21.15.171:6801/4123058942","heartbeat_back_addr":"172.21.15.171:6803/4123058942","heartbeat_front_addr":"172.21.15.171:6802/4123058942","state":["exists","up"]},{"osd":9,"uuid":"b6c17147-b3dc-475b-bb20-f9f712fb99e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":894,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6804","nonce":2877589636}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6805","nonce":2877589636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6807","nonce":2877589636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6806","nonce":2877589636}]},"public_addr":"172.21.15.171:6804/2877589636","cluster_addr":"172.21.15.171:6805/2877589636","heartbeat_back_addr":"172.21.15.171:6807/2877589636","heartbeat_front_addr":"172.21.15.171:6806/2877589636","state":["exists","up"]},{"osd":10,"uuid":"1adcdf55-f002-4a39-91e7-378e5fae849c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":894,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6808","nonce":181490621}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6809","nonce":181490621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6811","nonce":181490621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6810","nonce":181490621}]},"public_addr":"172.21.15.171:6808/181490621","cluster_addr":"172.21.15.171:6809/181490621","heartbeat_back_addr":"172.21.15.171:6811/181490621","heartbeat_front_addr":"172.21.15.171:6810/181490621","state":["exists","up"]},{"osd":11,"uuid":"737c5aec-694a-4968-819d-fccf22661f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":92,"up_thru":894,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6812","nonce":3373585701}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6813","nonce":3373585701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6815","nonce":3373585701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6814","nonce":3373585701}]},"public_addr":"172.21.15.171:6812/3373585701","cluster_addr":"172.21.15.171:6813/3373585701","heartbeat_back_addr":"172.21.15.171:6815/3373585701","heartbeat_front_addr":"172.21.15.171:6814/3373585701","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:49:43.690757+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:49:39.364022+0000","dead_epoch":613},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:49:04.038615+0000","dead_epoch":169},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:49:10.205130+0000","dead_epoch":538},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:49:38.653012+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:50:04.431220+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:49:20.960477+0000","dead_epoch":503},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:49:28.748115+0000","dead_epoch":721},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:49:50.977515+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:49:16.086269+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:49:49.894745+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:49:07.831508+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.5:0/3881726425":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/338649532":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/1227843664":"2024-04-21T16:58:55.392162+0000","172.21.15.5:6800/1876064631":"2024-04-21T16:58:55.392162+0000","172.21.15.5:0/160689449":"2024-04-21T16:58:55.392162+0000","172.21.15.5:6800/3521715286":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/3841663462":"2024-04-21T16:58:55.392162+0000","172.21.15.5:0/687690919":"2024-04-21T16:59:25.808337+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T17:50:11.255 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:50:11.295 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:50:11.306 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:50:11.306 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:50:11.306 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:50:11.306 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:50:11.306 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:50:11.306 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:50:11.306 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:50:11.306 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:50:11.306 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:50:11.307 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:50:11.307 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:50:11.307 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:50:11.488 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:50:11.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:11.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:50:11.600 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:50:11.639 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:11.868 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 65 16 139075 139059 133.689 81 0.0181827 0.00747875 2024-04-20T17:50:11.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:11 smithi005 ceph-mon[39562]: pgmap v2437: 27 pgs: 27 active+clean; 8.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-20T17:50:11.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:11 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3085299222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:50:11.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:11.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:11.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:11.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:11.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:11.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:11.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:11 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/819801216' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:50:11.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:11 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:11 smithi171 ceph-mon[44189]: pgmap v2437: 27 pgs: 27 active+clean; 8.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-20T17:50:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:11 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3085299222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:50:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:12.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:12.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:12.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:11 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/819801216' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:50:12.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:11 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:12.217 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:50:12.217 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:50:12.217 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-04-20T17:50:12.217 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-04-20T17:50:12.217 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:50:12.243 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:11 smithi106 ceph-mon[44423]: pgmap v2437: 27 pgs: 27 active+clean; 8.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-20T17:50:12.243 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:11 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3085299222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:50:12.243 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:12.243 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:12.243 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:12.243 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:12.243 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:12.243 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:12.243 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:11 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/819801216' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:50:12.244 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:11 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:12.868 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 66 15 140409 140394 132.928 83.4375 0.0147188 0.00752098 2024-04-20T17:50:12.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:12.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:12.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:12.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:12.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:12 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:12.975 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:12.976 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:12.976 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:12.976 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:12.976 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:12 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:13.013 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-20T17:50:13.034 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:50:13.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:13.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:50:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:13.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:12 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:13.143 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:50:13.184 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:13.393 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:50:13.623 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-20T17:50:13.627 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:50:13.639 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:50:13.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:13.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:50:13.745 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:50:13.781 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:13.818 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:50:13.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:13.825 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:50:13.868 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 67 16 141712 141696 132.158 81.375 0.0129938 0.00756519 2024-04-20T17:50:13.915 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:13 smithi005 ceph-mon[39562]: pgmap v2438: 27 pgs: 27 active+clean; 8.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-20T17:50:13.915 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:13 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:50:13.916 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:13 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:50:13.926 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:50:13.949 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:13 smithi106 ceph-mon[44423]: pgmap v2438: 27 pgs: 27 active+clean; 8.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-20T17:50:13.949 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:13 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:50:13.949 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:13 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:50:13.963 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:14.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:13 smithi171 ceph-mon[44189]: pgmap v2438: 27 pgs: 27 active+clean; 8.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-20T17:50:14.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:13 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:50:14.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:13 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:50:14.868 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 68 16 142929 142913 131.333 76.0625 0.0163511 0.00761301 2024-04-20T17:50:14.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:14 smithi005 ceph-mon[39562]: osdmap e926: 12 total, 12 up, 12 in 2024-04-20T17:50:15.074 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:50:15.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:14 smithi171 ceph-mon[44189]: osdmap e926: 12 total, 12 up, 12 in 2024-04-20T17:50:15.111 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:14 smithi106 ceph-mon[44423]: osdmap e926: 12 total, 12 up, 12 in 2024-04-20T17:50:15.529 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:50:15.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:15.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:50:15.634 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:50:15.681 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:15.716 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:50:15.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:15.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:50:15.746 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:50:15.824 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:50:15.869 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 69 16 144197 144181 130.577 79.25 0.00856345 0.00765665 2024-04-20T17:50:15.870 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:15.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:15 smithi005 ceph-mon[39562]: pgmap v2440: 27 pgs: 27 active+clean; 8.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-20T17:50:16.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:15 smithi171 ceph-mon[44189]: pgmap v2440: 27 pgs: 27 active+clean; 8.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-20T17:50:16.134 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:50:16.135 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:15 smithi106 ceph-mon[44423]: pgmap v2440: 27 pgs: 27 active+clean; 8.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-20T17:50:16.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:16.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:50:16.186 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:50:16.186 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-04-20T17:50:16.186 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-04-20T17:50:16.186 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:50:16.186 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.186 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:44752 9.a 9:5a35b736:::benchmark_data_smithi171_188319_object44751:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e904)", 2024-04-20T17:50:16.186 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:19.059046+0000", 2024-04-20T17:50:16.186 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 57.123546662000003, 2024-04-20T17:50:16.186 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.062133062000000003, 2024-04-20T17:50:16.186 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:50:16.186 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:50:16.186 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:50:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:50:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:50:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 44752 2024-04-20T17:50:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:50:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:50:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.059046+0000", 2024-04-20T17:50:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:50:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.059046+0000", 2024-04-20T17:50:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:50:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.059048+0000", 2024-04-20T17:50:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.635e-06 2024-04-20T17:50:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:50:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.059108+0000", 2024-04-20T17:50:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0448000000000001e-05 2024-04-20T17:50:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:50:16.189 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.059117+0000", 2024-04-20T17:50:16.189 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8969999999999997e-06 2024-04-20T17:50:16.189 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.189 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.189 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:50:16.189 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.059123+0000", 2024-04-20T17:50:16.189 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3709999999999999e-06 2024-04-20T17:50:16.189 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.189 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.189 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:50:16.189 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.059168+0000", 2024-04-20T17:50:16.189 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5349000000000002e-05 2024-04-20T17:50:16.189 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.190 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.190 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:50:16.190 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.059300+0000", 2024-04-20T17:50:16.190 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013168300000000001 2024-04-20T17:50:16.190 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.190 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.190 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:50:16.190 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.059384+0000", 2024-04-20T17:50:16.190 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4270000000000005e-05 2024-04-20T17:50:16.190 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.190 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.190 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:50:16.190 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.060733+0000", 2024-04-20T17:50:16.190 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013489660000000001 2024-04-20T17:50:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.121075+0000", 2024-04-20T17:50:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.060342086000000003 2024-04-20T17:50:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:50:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.121128+0000", 2024-04-20T17:50:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2936999999999998e-05 2024-04-20T17:50:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:50:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.121179+0000", 2024-04-20T17:50:16.192 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1419999999999999e-05 2024-04-20T17:50:16.192 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.192 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:50:16.192 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.192 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.192 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.192 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27772.0:44751 9.6 e904/894 9:6e0e1ddc:::benchmark_data_smithi171_188319_object44750:head v 904'2822, mlcod=904'2820)", 2024-04-20T17:50:16.192 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:19.061167+0000", 2024-04-20T17:50:16.192 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 57.121426417999999, 2024-04-20T17:50:16.192 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.059826198999999997, 2024-04-20T17:50:16.192 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:50:16.192 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:50:16.192 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:16.192 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:50:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:50:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.061167+0000", 2024-04-20T17:50:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:50:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.061167+0000", 2024-04-20T17:50:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:50:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.061168+0000", 2024-04-20T17:50:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1799999999999999e-06 2024-04-20T17:50:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.194 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.194 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:50:16.194 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.061231+0000", 2024-04-20T17:50:16.194 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3303e-05 2024-04-20T17:50:16.194 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.194 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.194 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:50:16.194 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.061239+0000", 2024-04-20T17:50:16.194 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0989999999999999e-06 2024-04-20T17:50:16.194 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.194 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.194 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:50:16.194 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.061244+0000", 2024-04-20T17:50:16.194 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9180000000000002e-06 2024-04-20T17:50:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:50:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.061289+0000", 2024-04-20T17:50:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5346999999999998e-05 2024-04-20T17:50:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:50:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.061307+0000", 2024-04-20T17:50:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7842999999999998e-05 2024-04-20T17:50:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:50:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.062626+0000", 2024-04-20T17:50:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013184780000000001 2024-04-20T17:50:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:50:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.120993+0000", 2024-04-20T17:50:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.058367031 2024-04-20T17:50:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:50:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:44757 9.f 9:f5517962:::benchmark_data_smithi171_188319_object44756:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e904)", 2024-04-20T17:50:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:19.061588+0000", 2024-04-20T17:50:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 57.121004909, 2024-04-20T17:50:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.059613715999999997, 2024-04-20T17:50:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:50:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:50:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:50:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:50:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:50:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 44757 2024-04-20T17:50:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:50:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:50:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.061588+0000", 2024-04-20T17:50:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.198 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:50:16.198 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.061588+0000", 2024-04-20T17:50:16.198 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.198 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:50:16.198 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.061590+0000", 2024-04-20T17:50:16.198 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.522e-06 2024-04-20T17:50:16.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.198 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:50:16.198 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.061641+0000", 2024-04-20T17:50:16.198 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1867000000000003e-05 2024-04-20T17:50:16.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:50:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.061652+0000", 2024-04-20T17:50:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0506000000000001e-05 2024-04-20T17:50:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:50:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.061658+0000", 2024-04-20T17:50:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6359999999999999e-06 2024-04-20T17:50:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:50:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.061686+0000", 2024-04-20T17:50:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8807e-05 2024-04-20T17:50:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:50:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.061786+0000", 2024-04-20T17:50:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010002500000000001 2024-04-20T17:50:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:50:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.061896+0000", 2024-04-20T17:50:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000109326 2024-04-20T17:50:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.121116+0000", 2024-04-20T17:50:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.059220042000000001 2024-04-20T17:50:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:50:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.121129+0000", 2024-04-20T17:50:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.296e-05 2024-04-20T17:50:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:50:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.121169+0000", 2024-04-20T17:50:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9986e-05 2024-04-20T17:50:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:50:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.121202+0000", 2024-04-20T17:50:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3039e-05 2024-04-20T17:50:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:50:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27772.0:44758 9.5 e904/894 9:aafd4072:::benchmark_data_smithi171_188319_object44757:head v 904'2767, mlcod=904'2765)", 2024-04-20T17:50:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:19.063619+0000", 2024-04-20T17:50:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 57.118974231000003, 2024-04-20T17:50:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.060428820000000001, 2024-04-20T17:50:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:50:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:50:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:50:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:50:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.063619+0000", 2024-04-20T17:50:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:50:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.063619+0000", 2024-04-20T17:50:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:50:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.063620+0000", 2024-04-20T17:50:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3650000000000001e-06 2024-04-20T17:50:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:50:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.063690+0000", 2024-04-20T17:50:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0112999999999998e-05 2024-04-20T17:50:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:50:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.120969+0000", 2024-04-20T17:50:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.05727873 2024-04-20T17:50:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:50:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.120984+0000", 2024-04-20T17:50:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5031999999999999e-05 2024-04-20T17:50:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:50:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.121041+0000", 2024-04-20T17:50:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7275999999999998e-05 2024-04-20T17:50:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:50:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.121064+0000", 2024-04-20T17:50:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2402e-05 2024-04-20T17:50:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:50:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.124006+0000", 2024-04-20T17:50:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002942311 2024-04-20T17:50:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:50:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.124048+0000", 2024-04-20T17:50:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1591000000000002e-05 2024-04-20T17:50:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:50:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.27772.0:44752 9.a e904/894 ondisk, result = 0)", 2024-04-20T17:50:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:19.063995+0000", 2024-04-20T17:50:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 57.118597710000003, 2024-04-20T17:50:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.057192701999999998, 2024-04-20T17:50:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:50:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:50:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-04-20T17:50:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:50:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:50:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.063995+0000", 2024-04-20T17:50:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:50:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.063995+0000", 2024-04-20T17:50:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:50:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.063996+0000", 2024-04-20T17:50:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1829999999999999e-06 2024-04-20T17:50:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:50:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.064001+0000", 2024-04-20T17:50:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1659999999999996e-06 2024-04-20T17:50:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:50:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.120972+0000", 2024-04-20T17:50:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.056970985000000002 2024-04-20T17:50:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:50:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.120983+0000", 2024-04-20T17:50:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1433999999999999e-05 2024-04-20T17:50:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:50:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.121062+0000", 2024-04-20T17:50:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8614000000000003e-05 2024-04-20T17:50:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:50:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.121069+0000", 2024-04-20T17:50:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0829999999999998e-06 2024-04-20T17:50:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:50:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.121188+0000", 2024-04-20T17:50:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000119237 2024-04-20T17:50:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:50:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:44765 9.a 9:5c7e5cc5:::benchmark_data_smithi171_188319_object44764:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e904)", 2024-04-20T17:50:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:19.065322+0000", 2024-04-20T17:50:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 57.117270959999999, 2024-04-20T17:50:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.059380413, 2024-04-20T17:50:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:50:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:50:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:50:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:50:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:50:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 44765 2024-04-20T17:50:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:50:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:50:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.065322+0000", 2024-04-20T17:50:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:50:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.065322+0000", 2024-04-20T17:50:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:50:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.065324+0000", 2024-04-20T17:50:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5600000000000001e-06 2024-04-20T17:50:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:50:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.065355+0000", 2024-04-20T17:50:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1263000000000002e-05 2024-04-20T17:50:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:50:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.120969+0000", 2024-04-20T17:50:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.055614351999999999 2024-04-20T17:50:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:50:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.120983+0000", 2024-04-20T17:50:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3594000000000001e-05 2024-04-20T17:50:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:50:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.121214+0000", 2024-04-20T17:50:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00023074200000000001 2024-04-20T17:50:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:50:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.121369+0000", 2024-04-20T17:50:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015598899999999999 2024-04-20T17:50:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:50:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.121493+0000", 2024-04-20T17:50:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012315500000000001 2024-04-20T17:50:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:50:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.124560+0000", 2024-04-20T17:50:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0030668929999999998 2024-04-20T17:50:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.124601+0000", 2024-04-20T17:50:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1322000000000002e-05 2024-04-20T17:50:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:50:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.124659+0000", 2024-04-20T17:50:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7669e-05 2024-04-20T17:50:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:50:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:19.124702+0000", 2024-04-20T17:50:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3874e-05 2024-04-20T17:50:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:50:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:103152 9.a 9:5f92df55:::benchmark_data_smithi171_188319_object103151:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e917)", 2024-04-20T17:50:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:47.053460+0000", 2024-04-20T17:50:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 29.129132457000001, 2024-04-20T17:50:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.032811530999999998, 2024-04-20T17:50:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:50:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:50:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:16.217 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:50:16.217 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:50:16.217 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:50:16.217 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 103152 2024-04-20T17:50:16.217 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.217 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:50:16.217 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.217 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:50:16.217 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:47.053460+0000", 2024-04-20T17:50:16.217 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.217 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.217 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.217 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:50:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:47.053460+0000", 2024-04-20T17:50:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:50:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:47.053462+0000", 2024-04-20T17:50:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8959999999999999e-06 2024-04-20T17:50:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:50:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:47.053516+0000", 2024-04-20T17:50:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3245999999999998e-05 2024-04-20T17:50:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:50:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:47.053526+0000", 2024-04-20T17:50:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0186999999999999e-05 2024-04-20T17:50:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:50:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:47.053532+0000", 2024-04-20T17:50:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9750000000000004e-06 2024-04-20T17:50:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:50:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:47.053570+0000", 2024-04-20T17:50:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8537000000000001e-05 2024-04-20T17:50:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:50:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:47.053676+0000", 2024-04-20T17:50:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010539099999999999 2024-04-20T17:50:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:50:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:47.053764+0000", 2024-04-20T17:50:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8364999999999998e-05 2024-04-20T17:50:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:50:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:47.056402+0000", 2024-04-20T17:50:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0026380790000000002 2024-04-20T17:50:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:47.086150+0000", 2024-04-20T17:50:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.029747429999999998 2024-04-20T17:50:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:50:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:47.086217+0000", 2024-04-20T17:50:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7156000000000006e-05 2024-04-20T17:50:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:50:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:47.086272+0000", 2024-04-20T17:50:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5269000000000002e-05 2024-04-20T17:50:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:50:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:110840 9.a 9:515330b0:::benchmark_data_smithi171_188319_object110839:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e920)", 2024-04-20T17:50:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:51.927752+0000", 2024-04-20T17:50:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 24.254840593000001, 2024-04-20T17:50:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.033146123, 2024-04-20T17:50:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:50:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:50:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:50:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:50:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:50:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 110840 2024-04-20T17:50:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:50:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:50:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:51.927752+0000", 2024-04-20T17:50:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:50:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:51.927752+0000", 2024-04-20T17:50:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:50:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:51.927753+0000", 2024-04-20T17:50:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2399999999999996e-07 2024-04-20T17:50:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:50:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:51.927866+0000", 2024-04-20T17:50:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000112446 2024-04-20T17:50:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:50:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:51.927877+0000", 2024-04-20T17:50:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0857e-05 2024-04-20T17:50:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:50:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:51.927884+0000", 2024-04-20T17:50:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7880000000000007e-06 2024-04-20T17:50:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:50:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:51.934450+0000", 2024-04-20T17:50:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0065657609999999998 2024-04-20T17:50:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:50:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:51.934597+0000", 2024-04-20T17:50:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014646599999999999 2024-04-20T17:50:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:50:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:51.934740+0000", 2024-04-20T17:50:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000142948 2024-04-20T17:50:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:50:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:51.940150+0000", 2024-04-20T17:50:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0054107189999999996 2024-04-20T17:50:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:51.960789+0000", 2024-04-20T17:50:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.020638457999999998 2024-04-20T17:50:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:50:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:51.960851+0000", 2024-04-20T17:50:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2469000000000001e-05 2024-04-20T17:50:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:50:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:51.960898+0000", 2024-04-20T17:50:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7287000000000003e-05 2024-04-20T17:50:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:50:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:111980 9.f 9:f6e6da2a:::benchmark_data_smithi171_188319_object111979:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e920)", 2024-04-20T17:50:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:52.641509+0000", 2024-04-20T17:50:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 23.541084247000001, 2024-04-20T17:50:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.043882308000000002, 2024-04-20T17:50:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:50:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:50:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:50:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:50:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:50:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 111980 2024-04-20T17:50:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:50:16.228 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.228 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:50:16.228 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.641509+0000", 2024-04-20T17:50:16.228 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.228 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.228 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.228 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:50:16.228 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.641509+0000", 2024-04-20T17:50:16.228 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.228 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.228 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.228 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:50:16.228 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.641511+0000", 2024-04-20T17:50:16.228 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.131e-06 2024-04-20T17:50:16.228 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.229 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.229 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:50:16.229 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.641582+0000", 2024-04-20T17:50:16.229 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1029e-05 2024-04-20T17:50:16.229 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.229 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.229 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:50:16.229 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.641594+0000", 2024-04-20T17:50:16.229 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2133999999999999e-05 2024-04-20T17:50:16.229 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.229 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.229 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:50:16.229 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.641603+0000", 2024-04-20T17:50:16.230 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.755e-06 2024-04-20T17:50:16.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.230 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:50:16.230 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.641653+0000", 2024-04-20T17:50:16.230 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0463000000000001e-05 2024-04-20T17:50:16.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.230 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:50:16.230 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.641748+0000", 2024-04-20T17:50:16.230 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5132999999999999e-05 2024-04-20T17:50:16.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.230 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:50:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.641832+0000", 2024-04-20T17:50:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3880000000000003e-05 2024-04-20T17:50:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:50:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.685253+0000", 2024-04-20T17:50:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.043420716999999998 2024-04-20T17:50:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.685285+0000", 2024-04-20T17:50:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1739999999999998e-05 2024-04-20T17:50:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:50:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.685340+0000", 2024-04-20T17:50:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5078999999999999e-05 2024-04-20T17:50:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:50:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.685391+0000", 2024-04-20T17:50:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1246999999999999e-05 2024-04-20T17:50:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:50:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27772.0:111979 9.6 e920/894 9:6b5beff2:::benchmark_data_smithi171_188319_object111978:head v 920'6997, mlcod=920'6995)", 2024-04-20T17:50:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:52.641865+0000", 2024-04-20T17:50:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 23.54072777, 2024-04-20T17:50:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.051120877000000002, 2024-04-20T17:50:16.233 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:50:16.233 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:50:16.233 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:16.233 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:50:16.233 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.233 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:50:16.233 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.641865+0000", 2024-04-20T17:50:16.233 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.233 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.233 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.233 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:50:16.233 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.641865+0000", 2024-04-20T17:50:16.233 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.233 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:50:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.641867+0000", 2024-04-20T17:50:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.404e-06 2024-04-20T17:50:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:50:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.641931+0000", 2024-04-20T17:50:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4515000000000001e-05 2024-04-20T17:50:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:50:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.641946+0000", 2024-04-20T17:50:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4729000000000001e-05 2024-04-20T17:50:16.235 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.235 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.235 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:50:16.235 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.641951+0000", 2024-04-20T17:50:16.235 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8679999999999998e-06 2024-04-20T17:50:16.235 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.235 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.235 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:50:16.235 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.681506+0000", 2024-04-20T17:50:16.235 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.039555649999999998 2024-04-20T17:50:16.235 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.235 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.235 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:50:16.235 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.681527+0000", 2024-04-20T17:50:16.236 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0767999999999998e-05 2024-04-20T17:50:16.236 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.236 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.236 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:50:16.236 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.692949+0000", 2024-04-20T17:50:16.236 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011421526 2024-04-20T17:50:16.236 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.236 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.236 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:50:16.236 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.692986+0000", 2024-04-20T17:50:16.236 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7416999999999998e-05 2024-04-20T17:50:16.236 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.236 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:50:16.236 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27772.0:111985 9.6 e920/894 9:6db58860:::benchmark_data_smithi171_188319_object111984:head v 920'6998, mlcod=920'6995)", 2024-04-20T17:50:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:52.645130+0000", 2024-04-20T17:50:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 23.537462522999999, 2024-04-20T17:50:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.053553204, 2024-04-20T17:50:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:50:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:50:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:50:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:50:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.645130+0000", 2024-04-20T17:50:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.238 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.238 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:50:16.238 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.645130+0000", 2024-04-20T17:50:16.238 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.238 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.238 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.238 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:50:16.238 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.645132+0000", 2024-04-20T17:50:16.238 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.426e-06 2024-04-20T17:50:16.238 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.238 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.238 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:50:16.238 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.645192+0000", 2024-04-20T17:50:16.238 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9851e-05 2024-04-20T17:50:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:50:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.681476+0000", 2024-04-20T17:50:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.036284597000000002 2024-04-20T17:50:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:50:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.681490+0000", 2024-04-20T17:50:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3499999999999999e-05 2024-04-20T17:50:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:50:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.685114+0000", 2024-04-20T17:50:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003624309 2024-04-20T17:50:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:50:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.685137+0000", 2024-04-20T17:50:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3334999999999999e-05 2024-04-20T17:50:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:50:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.698648+0000", 2024-04-20T17:50:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.01351104 2024-04-20T17:50:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:50:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.698684+0000", 2024-04-20T17:50:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5145999999999999e-05 2024-04-20T17:50:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:50:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:111996 9.f 9:f0c911d5:::benchmark_data_smithi171_188319_object111995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e920)", 2024-04-20T17:50:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:52.650394+0000", 2024-04-20T17:50:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 23.532199164000001, 2024-04-20T17:50:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.038258179000000003, 2024-04-20T17:50:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:50:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:50:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:50:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:50:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:50:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 111996 2024-04-20T17:50:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:50:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:50:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.650394+0000", 2024-04-20T17:50:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:50:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.650394+0000", 2024-04-20T17:50:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:50:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.650396+0000", 2024-04-20T17:50:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5859999999999999e-06 2024-04-20T17:50:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:50:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.650452+0000", 2024-04-20T17:50:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5779999999999998e-05 2024-04-20T17:50:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:50:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.681479+0000", 2024-04-20T17:50:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.031027319000000001 2024-04-20T17:50:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:50:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.681493+0000", 2024-04-20T17:50:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.313e-05 2024-04-20T17:50:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:50:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.681551+0000", 2024-04-20T17:50:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8748000000000001e-05 2024-04-20T17:50:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:50:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.681687+0000", 2024-04-20T17:50:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013532099999999999 2024-04-20T17:50:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.245 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.245 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:50:16.245 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.681933+0000", 2024-04-20T17:50:16.245 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000246037 2024-04-20T17:50:16.245 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.245 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.245 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:50:16.245 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.685461+0000", 2024-04-20T17:50:16.245 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0035278229999999998 2024-04-20T17:50:16.245 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.245 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.245 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:16.245 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.688577+0000", 2024-04-20T17:50:16.245 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003116962 2024-04-20T17:50:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:50:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.688612+0000", 2024-04-20T17:50:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4544e-05 2024-04-20T17:50:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:50:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:52.688652+0000", 2024-04-20T17:50:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9928999999999997e-05 2024-04-20T17:50:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:50:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27772.0:115419 9.5 e921/894 9:a4d9379b:::benchmark_data_smithi171_188319_object115418:head v 921'7205, mlcod=920'7203)", 2024-04-20T17:50:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:54.874166+0000", 2024-04-20T17:50:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 21.308427339000001, 2024-04-20T17:50:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.069262632000000005, 2024-04-20T17:50:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:50:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:50:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:50:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:50:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.874166+0000", 2024-04-20T17:50:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:50:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.874166+0000", 2024-04-20T17:50:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:50:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.874166+0000", 2024-04-20T17:50:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0399999999999996e-07 2024-04-20T17:50:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:50:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.874237+0000", 2024-04-20T17:50:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1217999999999995e-05 2024-04-20T17:50:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:50:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.874244+0000", 2024-04-20T17:50:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.759e-06 2024-04-20T17:50:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:50:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.874249+0000", 2024-04-20T17:50:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8160000000000001e-06 2024-04-20T17:50:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:50:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.874290+0000", 2024-04-20T17:50:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0531999999999998e-05 2024-04-20T17:50:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:50:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.874305+0000", 2024-04-20T17:50:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5767999999999999e-05 2024-04-20T17:50:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:50:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.943378+0000", 2024-04-20T17:50:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.069072650999999999 2024-04-20T17:50:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:50:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.943428+0000", 2024-04-20T17:50:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0284000000000002e-05 2024-04-20T17:50:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:50:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:115422 9.f 9:ff943748:::benchmark_data_smithi171_188319_object115421:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e921)", 2024-04-20T17:50:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:54.876406+0000", 2024-04-20T17:50:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 21.306186605000001, 2024-04-20T17:50:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.073846258999999997, 2024-04-20T17:50:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:50:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:50:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:50:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:50:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:50:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 115422 2024-04-20T17:50:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:50:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:50:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.876406+0000", 2024-04-20T17:50:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:50:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.876406+0000", 2024-04-20T17:50:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:50:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.876408+0000", 2024-04-20T17:50:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0090000000000002e-06 2024-04-20T17:50:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:50:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.876464+0000", 2024-04-20T17:50:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5591999999999999e-05 2024-04-20T17:50:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:50:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.876475+0000", 2024-04-20T17:50:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0917999999999999e-05 2024-04-20T17:50:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:50:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.876483+0000", 2024-04-20T17:50:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1380000000000007e-06 2024-04-20T17:50:16.254 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.254 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.254 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:50:16.254 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.876533+0000", 2024-04-20T17:50:16.254 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0170000000000002e-05 2024-04-20T17:50:16.254 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.254 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.254 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:50:16.254 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.876656+0000", 2024-04-20T17:50:16.254 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012299899999999999 2024-04-20T17:50:16.254 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.254 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.255 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:50:16.255 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.943385+0000", 2024-04-20T17:50:16.255 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.066728987000000003 2024-04-20T17:50:16.255 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.255 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.255 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:16.255 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.950164+0000", 2024-04-20T17:50:16.255 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0067791830000000003 2024-04-20T17:50:16.255 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.255 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.255 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:50:16.255 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.950172+0000", 2024-04-20T17:50:16.255 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9589999999999995e-06 2024-04-20T17:50:16.255 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:50:16.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.950203+0000", 2024-04-20T17:50:16.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0936000000000003e-05 2024-04-20T17:50:16.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:50:16.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.950253+0000", 2024-04-20T17:50:16.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9367999999999999e-05 2024-04-20T17:50:16.256 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.256 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:50:16.256 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.256 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27772.0:115423 9.6 e921/894 9:65dbabc8:::benchmark_data_smithi171_188319_object115422:head v 921'7200, mlcod=920'7198)", 2024-04-20T17:50:16.257 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:54.876912+0000", 2024-04-20T17:50:16.257 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 21.305680456000001, 2024-04-20T17:50:16.257 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.075200614999999998, 2024-04-20T17:50:16.257 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:50:16.257 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:50:16.257 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:16.257 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:50:16.257 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.257 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:50:16.257 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.876912+0000", 2024-04-20T17:50:16.257 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.257 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.257 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.257 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:50:16.257 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.876912+0000", 2024-04-20T17:50:16.258 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.258 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.258 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.258 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:50:16.258 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.876914+0000", 2024-04-20T17:50:16.258 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.192e-06 2024-04-20T17:50:16.258 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.258 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.258 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:50:16.258 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.877019+0000", 2024-04-20T17:50:16.258 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000105379 2024-04-20T17:50:16.258 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.258 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.258 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:50:16.258 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.943351+0000", 2024-04-20T17:50:16.259 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.066332033999999998 2024-04-20T17:50:16.259 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.259 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.259 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:50:16.259 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.943370+0000", 2024-04-20T17:50:16.259 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8953999999999998e-05 2024-04-20T17:50:16.259 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.259 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.259 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:50:16.259 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.943423+0000", 2024-04-20T17:50:16.259 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3356999999999998e-05 2024-04-20T17:50:16.259 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.261 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.261 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:50:16.261 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.943445+0000", 2024-04-20T17:50:16.261 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1909999999999999e-05 2024-04-20T17:50:16.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.261 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.261 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:50:16.261 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.952073+0000", 2024-04-20T17:50:16.261 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0086275299999999996 2024-04-20T17:50:16.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.262 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:50:16.262 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.952113+0000", 2024-04-20T17:50:16.262 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0259000000000002e-05 2024-04-20T17:50:16.262 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.262 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:50:16.262 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.262 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.262 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:115426 9.a 9:50026b53:::benchmark_data_smithi171_188319_object115425:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e921)", 2024-04-20T17:50:16.262 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:54.878690+0000", 2024-04-20T17:50:16.262 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 21.303903431999998, 2024-04-20T17:50:16.262 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.076047894000000005, 2024-04-20T17:50:16.262 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:50:16.263 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:50:16.263 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:16.263 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:50:16.263 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:50:16.263 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:50:16.263 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 115426 2024-04-20T17:50:16.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.263 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:50:16.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.263 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:50:16.263 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.878690+0000", 2024-04-20T17:50:16.263 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.263 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:50:16.264 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.878690+0000", 2024-04-20T17:50:16.264 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.264 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:50:16.264 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.878697+0000", 2024-04-20T17:50:16.264 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3760000000000002e-06 2024-04-20T17:50:16.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.264 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:50:16.264 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.878749+0000", 2024-04-20T17:50:16.264 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2357e-05 2024-04-20T17:50:16.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.264 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:50:16.265 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.943345+0000", 2024-04-20T17:50:16.265 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.064595572000000004 2024-04-20T17:50:16.265 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.265 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.265 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:50:16.265 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.943360+0000", 2024-04-20T17:50:16.265 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5500000000000001e-05 2024-04-20T17:50:16.265 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.265 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.265 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:50:16.265 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.943422+0000", 2024-04-20T17:50:16.265 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1999000000000001e-05 2024-04-20T17:50:16.265 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.265 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.265 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:50:16.266 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.943642+0000", 2024-04-20T17:50:16.266 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00021997999999999999 2024-04-20T17:50:16.266 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.266 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.266 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:50:16.266 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.943791+0000", 2024-04-20T17:50:16.266 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000149152 2024-04-20T17:50:16.266 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.266 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.266 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:50:16.266 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.954653+0000", 2024-04-20T17:50:16.266 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010861269 2024-04-20T17:50:16.266 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.266 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.267 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:16.267 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.954677+0000", 2024-04-20T17:50:16.267 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4255000000000001e-05 2024-04-20T17:50:16.267 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.267 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.267 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:50:16.267 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.954709+0000", 2024-04-20T17:50:16.267 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1582999999999999e-05 2024-04-20T17:50:16.267 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.267 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.267 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:50:16.267 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:54.954737+0000", 2024-04-20T17:50:16.267 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8850999999999999e-05 2024-04-20T17:50:16.267 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.267 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:50:16.268 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.268 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.268 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.268 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:140764 9.a 9:56d69c9c:::benchmark_data_smithi171_188319_object140763:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e925)", 2024-04-20T17:50:16.268 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:50:13.137202+0000", 2024-04-20T17:50:16.268 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 3.0453908209999998, 2024-04-20T17:50:16.268 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.034763686000000002, 2024-04-20T17:50:16.268 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:50:16.268 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:50:16.268 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:16.268 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:50:16.268 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:50:16.268 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:50:16.269 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 140764 2024-04-20T17:50:16.269 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.269 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:50:16.269 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.269 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:50:16.269 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:13.137202+0000", 2024-04-20T17:50:16.269 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.269 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.269 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.269 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:50:16.269 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:13.137202+0000", 2024-04-20T17:50:16.269 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.269 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.269 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.269 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:50:16.270 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:13.137203+0000", 2024-04-20T17:50:16.270 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0999999999999997e-07 2024-04-20T17:50:16.270 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.270 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.270 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:50:16.270 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:13.137270+0000", 2024-04-20T17:50:16.270 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6632999999999997e-05 2024-04-20T17:50:16.270 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.270 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.270 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:50:16.270 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:13.137289+0000", 2024-04-20T17:50:16.270 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8899e-05 2024-04-20T17:50:16.271 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.271 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.271 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:50:16.271 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:13.137299+0000", 2024-04-20T17:50:16.271 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0474e-05 2024-04-20T17:50:16.271 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.271 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.271 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:50:16.271 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:13.142221+0000", 2024-04-20T17:50:16.271 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0049223030000000003 2024-04-20T17:50:16.271 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.271 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.271 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:50:16.271 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:13.142392+0000", 2024-04-20T17:50:16.271 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000171068 2024-04-20T17:50:16.272 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.272 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.272 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:50:16.272 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:13.142550+0000", 2024-04-20T17:50:16.272 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015748100000000001 2024-04-20T17:50:16.272 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.272 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.272 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:50:16.272 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:13.150653+0000", 2024-04-20T17:50:16.272 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0081026699999999993 2024-04-20T17:50:16.272 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.272 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.272 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:16.272 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:13.171862+0000", 2024-04-20T17:50:16.273 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.021209068000000001 2024-04-20T17:50:16.273 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.273 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.273 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:50:16.273 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:13.171910+0000", 2024-04-20T17:50:16.273 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8251000000000003e-05 2024-04-20T17:50:16.273 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.273 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.273 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:50:16.273 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:13.171966+0000", 2024-04-20T17:50:16.273 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5928999999999999e-05 2024-04-20T17:50:16.273 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.273 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:50:16.273 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.274 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.274 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.274 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27772.0:142590 9.6 e926/894 9:6380f3ab:::benchmark_data_smithi171_188319_object142589:head v 926'8881, mlcod=926'8879)", 2024-04-20T17:50:16.274 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:50:14.544883+0000", 2024-04-20T17:50:16.274 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 1.6377100769999999, 2024-04-20T17:50:16.274 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13085866800000001, 2024-04-20T17:50:16.274 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:50:16.274 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:50:16.274 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:16.274 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:50:16.274 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.274 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:50:16.274 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.544883+0000", 2024-04-20T17:50:16.275 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.275 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.275 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:50:16.275 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.544883+0000", 2024-04-20T17:50:16.275 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.275 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.275 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:50:16.275 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.544885+0000", 2024-04-20T17:50:16.275 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1110000000000002e-06 2024-04-20T17:50:16.275 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.275 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:50:16.276 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.544916+0000", 2024-04-20T17:50:16.276 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0991999999999997e-05 2024-04-20T17:50:16.276 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.276 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.276 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:50:16.276 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.544927+0000", 2024-04-20T17:50:16.276 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0964999999999999e-05 2024-04-20T17:50:16.276 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.276 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.276 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:50:16.276 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.544933+0000", 2024-04-20T17:50:16.276 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6860000000000003e-06 2024-04-20T17:50:16.276 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.276 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.277 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:50:16.277 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.544989+0000", 2024-04-20T17:50:16.277 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6419000000000003e-05 2024-04-20T17:50:16.277 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.277 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.277 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:50:16.277 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.545010+0000", 2024-04-20T17:50:16.277 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1342e-05 2024-04-20T17:50:16.277 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.277 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.277 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:50:16.277 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.675693+0000", 2024-04-20T17:50:16.277 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.130682565 2024-04-20T17:50:16.277 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.277 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.278 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:50:16.278 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.675742+0000", 2024-04-20T17:50:16.278 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8588000000000003e-05 2024-04-20T17:50:16.278 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.278 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:50:16.278 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.278 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.278 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.278 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:142598 9.f 9:f5c8efc1:::benchmark_data_smithi171_188319_object142597:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e926)", 2024-04-20T17:50:16.278 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:50:14.548816+0000", 2024-04-20T17:50:16.278 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 1.6337768480000001, 2024-04-20T17:50:16.278 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.122003799, 2024-04-20T17:50:16.278 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:50:16.278 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:50:16.278 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:16.279 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:50:16.279 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:50:16.279 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:50:16.279 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 142598 2024-04-20T17:50:16.279 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.279 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:50:16.279 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.279 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:50:16.279 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.548816+0000", 2024-04-20T17:50:16.279 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.279 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.279 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.279 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:50:16.279 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.548816+0000", 2024-04-20T17:50:16.279 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.280 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.280 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:50:16.280 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.548826+0000", 2024-04-20T17:50:16.280 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0339999999999999e-05 2024-04-20T17:50:16.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.280 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.280 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:50:16.280 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.548878+0000", 2024-04-20T17:50:16.280 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1551999999999997e-05 2024-04-20T17:50:16.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.280 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.280 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:50:16.280 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.659521+0000", 2024-04-20T17:50:16.281 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.11064271000000001 2024-04-20T17:50:16.281 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.281 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:50:16.281 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.659532+0000", 2024-04-20T17:50:16.281 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1494999999999999e-05 2024-04-20T17:50:16.281 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.281 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:50:16.281 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.659585+0000", 2024-04-20T17:50:16.281 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2911999999999998e-05 2024-04-20T17:50:16.281 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.281 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:50:16.281 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.659698+0000", 2024-04-20T17:50:16.281 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000112706 2024-04-20T17:50:16.282 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.282 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.282 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:50:16.282 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.659791+0000", 2024-04-20T17:50:16.282 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2898000000000004e-05 2024-04-20T17:50:16.282 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.282 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.282 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:16.282 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.670666+0000", 2024-04-20T17:50:16.282 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010874866 2024-04-20T17:50:16.282 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.282 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.282 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:50:16.282 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.670733+0000", 2024-04-20T17:50:16.283 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7420000000000002e-05 2024-04-20T17:50:16.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.283 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.283 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:50:16.283 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.670775+0000", 2024-04-20T17:50:16.283 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2493999999999997e-05 2024-04-20T17:50:16.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.283 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.283 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:50:16.283 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.670820+0000", 2024-04-20T17:50:16.283 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4406000000000001e-05 2024-04-20T17:50:16.283 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.283 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:50:16.283 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.283 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.284 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27772.0:142592 9.6 e926/894 9:69caef56:::benchmark_data_smithi171_188319_object142591:head v 926'8882, mlcod=926'8879)", 2024-04-20T17:50:16.284 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:50:14.553703+0000", 2024-04-20T17:50:16.284 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 1.628890314, 2024-04-20T17:50:16.284 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.122068986, 2024-04-20T17:50:16.284 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:50:16.284 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:50:16.284 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:16.284 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:50:16.284 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.284 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:50:16.284 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.553703+0000", 2024-04-20T17:50:16.284 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.284 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.284 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.284 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:50:16.285 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.553703+0000", 2024-04-20T17:50:16.285 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:50:16.285 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.285 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.285 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:50:16.285 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.553704+0000", 2024-04-20T17:50:16.285 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2860000000000001e-06 2024-04-20T17:50:16.285 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.285 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.285 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:50:16.285 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.553745+0000", 2024-04-20T17:50:16.285 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1180000000000002e-05 2024-04-20T17:50:16.285 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.285 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.285 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:50:16.286 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.659511+0000", 2024-04-20T17:50:16.286 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.105765792 2024-04-20T17:50:16.286 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.286 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.286 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:50:16.286 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.659525+0000", 2024-04-20T17:50:16.286 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4421e-05 2024-04-20T17:50:16.286 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.286 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.286 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:50:16.286 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.663816+0000", 2024-04-20T17:50:16.286 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0042904149999999997 2024-04-20T17:50:16.286 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.286 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.286 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:50:16.287 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.663835+0000", 2024-04-20T17:50:16.287 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8831999999999999e-05 2024-04-20T17:50:16.287 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.287 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.287 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:50:16.287 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.675747+0000", 2024-04-20T17:50:16.287 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011911959999999999 2024-04-20T17:50:16.287 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:50:16.287 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:50:16.287 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:50:16.287 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:14.675772+0000", 2024-04-20T17:50:16.287 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.51e-05 2024-04-20T17:50:16.287 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.287 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:50:16.287 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.287 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:50:16.288 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:50:16.288 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:50:16.288 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:50:16.339 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:16.362 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:50:16.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:16.366 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:50:16.466 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:50:16.505 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:16.854 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:50:16.869 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 70 16 145491 145475 129.866 80.875 0.0230004 0.00769908 2024-04-20T17:50:16.895 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:16 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2704278415' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:50:16.895 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:16 smithi106 ceph-mon[44423]: from='client.27847 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:50:16.935 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-20T17:50:17.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:16 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2704278415' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:50:17.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:16 smithi005 ceph-mon[39562]: from='client.27847 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:50:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:16 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2704278415' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:50:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:16 smithi171 ceph-mon[44189]: from='client.27847 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:50:17.362 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:50:17.869 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 71 16 146622 146606 129.033 70.6875 0.00623735 0.00774804 2024-04-20T17:50:17.971 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:50:17.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:17.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:50:18.076 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:50:18.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:17 smithi171 ceph-mon[44189]: pgmap v2441: 27 pgs: 27 active+clean; 8.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-04-20T17:50:18.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:17 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:50:18.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:17 smithi171 ceph-mon[44189]: from='client.27847 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:50:18.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:17 smithi171 ceph-mon[44189]: osdmap e927: 12 total, 12 up, 12 in 2024-04-20T17:50:18.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:17 smithi171 ceph-mon[44189]: 9.b scrub starts 2024-04-20T17:50:18.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:17 smithi005 ceph-mon[39562]: pgmap v2441: 27 pgs: 27 active+clean; 8.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-04-20T17:50:18.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:17 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:50:18.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:17 smithi005 ceph-mon[39562]: from='client.27847 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:50:18.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:17 smithi005 ceph-mon[39562]: osdmap e927: 12 total, 12 up, 12 in 2024-04-20T17:50:18.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:17 smithi005 ceph-mon[39562]: 9.b scrub starts 2024-04-20T17:50:18.134 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:18.204 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:50:18.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:18.208 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:50:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:17 smithi106 ceph-mon[44423]: pgmap v2441: 27 pgs: 27 active+clean; 8.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-04-20T17:50:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:17 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:50:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:17 smithi106 ceph-mon[44423]: from='client.27847 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:50:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:17 smithi106 ceph-mon[44423]: osdmap e927: 12 total, 12 up, 12 in 2024-04-20T17:50:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:17 smithi106 ceph-mon[44423]: 9.b scrub starts 2024-04-20T17:50:18.309 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:50:18.341 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:18.363 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:50:18.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:18.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:50:18.474 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:50:18.508 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:18.531 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:50:18.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:18.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:50:18.635 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:50:18.689 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:18.873 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 72 16 147583 147567 128.07 60.0625 0.0257627 0.00780643 2024-04-20T17:50:18.957 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:50:18.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:18.966 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:50:19.067 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:50:19.118 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:18 smithi171 ceph-mon[44189]: 1.5 scrub starts 2024-04-20T17:50:19.120 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:18 smithi005 ceph-mon[39562]: 1.5 scrub starts 2024-04-20T17:50:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:18 smithi106 ceph-mon[44423]: 1.5 scrub starts 2024-04-20T17:50:19.301 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:50:19.327 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:50:19.353 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:50:19.358 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:50:19.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:19.367 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:50:19.381 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:50:19.405 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:50:19.427 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:50:19.456 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:50:19.468 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:50:19.480 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:50:19.496 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:19.517 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:50:19.518 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:50:19.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:19.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:50:19.548 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:50:19.582 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:50:19.614 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:50:19.616 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:50:19.616 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:50:19.616 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:50:19.616 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:50:19.616 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:50:19.616 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:50:19.616 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:50:19.616 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:50:19.616 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:50:19.616 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:50:19.617 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:50:19.617 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:50:19.623 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:50:19.656 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:19.682 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:50:19.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:19.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:50:19.786 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:50:19.827 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:19.847 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:50:19.847 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:50:19.877 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 73 16 148593 148577 127.172 63.125 0.0139168 0.00786186 2024-04-20T17:50:19.886 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:50:19.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:19.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:50:19.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:19 smithi005 ceph-mon[39562]: 9.8 scrub starts 2024-04-20T17:50:19.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:19 smithi005 ceph-mon[39562]: 1.5 scrub ok 2024-04-20T17:50:19.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:19 smithi005 ceph-mon[39562]: pgmap v2443: 27 pgs: 27 active+clean; 8.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-04-20T17:50:19.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:19 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3262138422' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:50:19.992 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:50:20.051 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:20.099 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:50:20.104 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:19 smithi171 ceph-mon[44189]: 9.8 scrub starts 2024-04-20T17:50:20.104 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:19 smithi171 ceph-mon[44189]: 1.5 scrub ok 2024-04-20T17:50:20.105 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:19 smithi171 ceph-mon[44189]: pgmap v2443: 27 pgs: 27 active+clean; 8.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-04-20T17:50:20.105 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:19 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3262138422' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:50:20.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:20.106 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:50:20.207 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:50:20.233 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:19 smithi106 ceph-mon[44423]: 9.8 scrub starts 2024-04-20T17:50:20.233 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:19 smithi106 ceph-mon[44423]: 1.5 scrub ok 2024-04-20T17:50:20.233 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:19 smithi106 ceph-mon[44423]: pgmap v2443: 27 pgs: 27 active+clean; 8.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-04-20T17:50:20.233 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:19 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3262138422' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:50:20.239 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:20.267 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:50:20.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:20.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:50:20.371 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:50:20.413 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:20.477 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:50:20.551 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:50:20.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:20.557 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:50:20.593 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-20T17:50:20.658 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:50:20.722 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:20.756 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:50:20.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:20.761 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:50:20.862 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:50:20.878 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 74 16 149751 149735 126.43 72.375 0.025388 0.00790766 2024-04-20T17:50:20.904 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:50:20.931 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:20.958 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:50:20.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:20.969 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:20.969 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:20.969 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:20.969 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:20.969 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:20.969 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:20.969 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:20.969 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:20.969 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:20.969 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:20.970 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:20.970 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:20.970 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:20 smithi171 ceph-mon[44189]: 9.b scrub ok 2024-04-20T17:50:20.970 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:20 smithi171 ceph-mon[44189]: 1.4 scrub starts 2024-04-20T17:50:20.970 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:20 smithi171 ceph-mon[44189]: 1.4 scrub ok 2024-04-20T17:50:20.970 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:50:21.037 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:50:21.072 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:50:21.135 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:21.135 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:21.135 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:21.136 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:21.136 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:21.136 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:21.136 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:21.136 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:21.136 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:21.136 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:21.136 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:21.136 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:21.136 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:20 smithi106 ceph-mon[44423]: 9.b scrub ok 2024-04-20T17:50:21.136 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:20 smithi106 ceph-mon[44423]: 1.4 scrub starts 2024-04-20T17:50:21.136 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:20 smithi106 ceph-mon[44423]: 1.4 scrub ok 2024-04-20T17:50:21.138 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:21.185 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:21.186 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:21.186 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:21.186 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:21.186 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:21.186 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:21.186 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:21.186 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:21.186 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:21.186 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:21.186 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:21.186 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:21.186 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:20 smithi005 ceph-mon[39562]: 9.b scrub ok 2024-04-20T17:50:21.187 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:20 smithi005 ceph-mon[39562]: 1.4 scrub starts 2024-04-20T17:50:21.187 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:20 smithi005 ceph-mon[39562]: 1.4 scrub ok 2024-04-20T17:50:21.190 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:50:21.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:21.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:50:21.305 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:50:21.337 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:21.864 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:21 smithi005 ceph-mon[39562]: 9.8 scrub ok 2024-04-20T17:50:21.864 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:21 smithi005 ceph-mon[39562]: 9.a scrub starts 2024-04-20T17:50:21.864 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:21 smithi005 ceph-mon[39562]: pgmap v2444: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 9.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-20T17:50:21.864 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:21 smithi005 ceph-mon[39562]: 9.2 scrub starts 2024-04-20T17:50:21.864 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:21 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:50:21.864 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:21 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:50:21.864 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:21 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:50:21.864 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:21 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:50:21.879 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 75 16 150863 150847 125.67 69.5 0.0153774 0.00795597 2024-04-20T17:50:22.099 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:50:22.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:22.109 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:50:22.153 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:21 smithi171 ceph-mon[44189]: 9.8 scrub ok 2024-04-20T17:50:22.153 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:21 smithi171 ceph-mon[44189]: 9.a scrub starts 2024-04-20T17:50:22.153 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:21 smithi171 ceph-mon[44189]: pgmap v2444: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 9.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-20T17:50:22.153 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:21 smithi171 ceph-mon[44189]: 9.2 scrub starts 2024-04-20T17:50:22.153 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:21 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:50:22.153 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:21 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:50:22.153 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:21 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:50:22.153 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:21 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:50:22.210 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:50:22.246 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:21 smithi106 ceph-mon[44423]: 9.8 scrub ok 2024-04-20T17:50:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:21 smithi106 ceph-mon[44423]: 9.a scrub starts 2024-04-20T17:50:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:21 smithi106 ceph-mon[44423]: pgmap v2444: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 9.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-20T17:50:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:21 smithi106 ceph-mon[44423]: 9.2 scrub starts 2024-04-20T17:50:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:21 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:50:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:21 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:50:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:21 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:50:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:21 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:50:22.473 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:50:22.879 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 76 16 151907 151891 124.875 65.25 0.0103933 0.00800637 2024-04-20T17:50:23.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:22 smithi171 ceph-mon[44189]: 9.4 scrub starts 2024-04-20T17:50:23.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:22 smithi171 ceph-mon[44189]: osdmap e928: 12 total, 12 up, 12 in 2024-04-20T17:50:23.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:22 smithi171 ceph-mon[44189]: 9.5 scrub starts 2024-04-20T17:50:23.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:22 smithi171 ceph-mon[44189]: pgmap v2446: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 9.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-04-20T17:50:23.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:22 smithi005 ceph-mon[39562]: 9.4 scrub starts 2024-04-20T17:50:23.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:22 smithi005 ceph-mon[39562]: osdmap e928: 12 total, 12 up, 12 in 2024-04-20T17:50:23.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:22 smithi005 ceph-mon[39562]: 9.5 scrub starts 2024-04-20T17:50:23.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:22 smithi005 ceph-mon[39562]: pgmap v2446: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 9.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-04-20T17:50:23.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:22 smithi106 ceph-mon[44423]: 9.4 scrub starts 2024-04-20T17:50:23.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:22 smithi106 ceph-mon[44423]: osdmap e928: 12 total, 12 up, 12 in 2024-04-20T17:50:23.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:22 smithi106 ceph-mon[44423]: 9.5 scrub starts 2024-04-20T17:50:23.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:22 smithi106 ceph-mon[44423]: pgmap v2446: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 9.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-04-20T17:50:23.511 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:50:23.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:23.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:50:23.598 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:50:23.615 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:50:23.649 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:23.851 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:50:23.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:23.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:50:23.879 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 77 15 152864 152849 124.031 59.875 0.014234 0.00806097 2024-04-20T17:50:23.956 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:50:23.986 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:24.019 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:50:24.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:24.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:50:24.123 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:50:24.166 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:23 smithi005 ceph-mon[39562]: 9.a scrub ok 2024-04-20T17:50:24.167 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:23 smithi005 ceph-mon[39562]: 9.f scrub starts 2024-04-20T17:50:24.167 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:23 smithi005 ceph-mon[39562]: 9.2 scrub ok 2024-04-20T17:50:24.167 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:23 smithi005 ceph-mon[39562]: 9.6 scrub starts 2024-04-20T17:50:24.167 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:23 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:50:24.167 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:23 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:50:24.167 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:23 smithi005 ceph-mon[39562]: 1.8 scrub starts 2024-04-20T17:50:24.169 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:24.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:23 smithi106 ceph-mon[44423]: 9.a scrub ok 2024-04-20T17:50:24.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:23 smithi106 ceph-mon[44423]: 9.f scrub starts 2024-04-20T17:50:24.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:23 smithi106 ceph-mon[44423]: 9.2 scrub ok 2024-04-20T17:50:24.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:23 smithi106 ceph-mon[44423]: 9.6 scrub starts 2024-04-20T17:50:24.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:23 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:50:24.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:23 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:50:24.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:23 smithi106 ceph-mon[44423]: 1.8 scrub starts 2024-04-20T17:50:24.274 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:50:24.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:24.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:50:24.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:23 smithi171 ceph-mon[44189]: 9.a scrub ok 2024-04-20T17:50:24.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:23 smithi171 ceph-mon[44189]: 9.f scrub starts 2024-04-20T17:50:24.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:23 smithi171 ceph-mon[44189]: 9.2 scrub ok 2024-04-20T17:50:24.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:23 smithi171 ceph-mon[44189]: 9.6 scrub starts 2024-04-20T17:50:24.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:23 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:50:24.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:23 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:50:24.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:23 smithi171 ceph-mon[44189]: 1.8 scrub starts 2024-04-20T17:50:24.378 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:50:24.421 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:24.585 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:50:24.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:24.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:50:24.694 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:50:24.726 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:24.879 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 78 16 154034 154018 123.378 73.0625 0.0166408 0.00810385 2024-04-20T17:50:24.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:24 smithi005 ceph-mon[39562]: 9.4 scrub ok 2024-04-20T17:50:24.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:24 smithi005 ceph-mon[39562]: 1.8 scrub ok 2024-04-20T17:50:24.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:24 smithi005 ceph-mon[39562]: 9.f scrub ok 2024-04-20T17:50:24.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:24 smithi005 ceph-mon[39562]: 9.5 scrub ok 2024-04-20T17:50:24.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:24 smithi005 ceph-mon[39562]: osdmap e929: 12 total, 12 up, 12 in 2024-04-20T17:50:24.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:24 smithi005 ceph-mon[39562]: pgmap v2448: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 9.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 993 op/s 2024-04-20T17:50:25.085 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:50:25.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:25.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:50:25.189 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:50:25.223 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:24 smithi171 ceph-mon[44189]: 9.4 scrub ok 2024-04-20T17:50:25.223 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:24 smithi171 ceph-mon[44189]: 1.8 scrub ok 2024-04-20T17:50:25.223 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:24 smithi171 ceph-mon[44189]: 9.f scrub ok 2024-04-20T17:50:25.223 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:24 smithi171 ceph-mon[44189]: 9.5 scrub ok 2024-04-20T17:50:25.223 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:24 smithi171 ceph-mon[44189]: osdmap e929: 12 total, 12 up, 12 in 2024-04-20T17:50:25.224 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:24 smithi171 ceph-mon[44189]: pgmap v2448: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 9.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 993 op/s 2024-04-20T17:50:25.232 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:25.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:24 smithi106 ceph-mon[44423]: 9.4 scrub ok 2024-04-20T17:50:25.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:24 smithi106 ceph-mon[44423]: 1.8 scrub ok 2024-04-20T17:50:25.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:24 smithi106 ceph-mon[44423]: 9.f scrub ok 2024-04-20T17:50:25.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:24 smithi106 ceph-mon[44423]: 9.5 scrub ok 2024-04-20T17:50:25.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:24 smithi106 ceph-mon[44423]: osdmap e929: 12 total, 12 up, 12 in 2024-04-20T17:50:25.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:24 smithi106 ceph-mon[44423]: pgmap v2448: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 9.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 993 op/s 2024-04-20T17:50:25.275 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:50:25.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:25.279 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:50:25.380 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:50:25.433 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:25.879 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 79 16 155018 155002 122.595 61.5 0.0184223 0.00815547 2024-04-20T17:50:25.907 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:50:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:25 smithi005 ceph-mon[39562]: 9.7 scrub starts 2024-04-20T17:50:26.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:25 smithi005 ceph-mon[39562]: 9.3 scrub starts 2024-04-20T17:50:26.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:25 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1546872234' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:50:26.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:25 smithi005 ceph-mon[39562]: 1.7 scrub starts 2024-04-20T17:50:26.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:25 smithi005 ceph-mon[39562]: 1.7 scrub ok 2024-04-20T17:50:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:25 smithi106 ceph-mon[44423]: 9.7 scrub starts 2024-04-20T17:50:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:25 smithi106 ceph-mon[44423]: 9.3 scrub starts 2024-04-20T17:50:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:25 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1546872234' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:50:26.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:25 smithi106 ceph-mon[44423]: 1.7 scrub starts 2024-04-20T17:50:26.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:25 smithi106 ceph-mon[44423]: 1.7 scrub ok 2024-04-20T17:50:26.305 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:25 smithi171 ceph-mon[44189]: 9.7 scrub starts 2024-04-20T17:50:26.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:25 smithi171 ceph-mon[44189]: 9.3 scrub starts 2024-04-20T17:50:26.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:25 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1546872234' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:50:26.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:25 smithi171 ceph-mon[44189]: 1.7 scrub starts 2024-04-20T17:50:26.306 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:25 smithi171 ceph-mon[44189]: 1.7 scrub ok 2024-04-20T17:50:26.879 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:2024-04-20 17:50:26.880191 min lat: 0.00101366 max lat: 0.200169 avg lat: 0.00821134 2024-04-20T17:50:26.880 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-20T17:50:26.880 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 80 15 155910 155895 121.76 55.8125 0.00868644 0.00821134 2024-04-20T17:50:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:26 smithi005 ceph-mon[39562]: 9.6 scrub ok 2024-04-20T17:50:27.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:26 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:50:27.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:26 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:50:27.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:26 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1546872234' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:50:27.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:26 smithi005 ceph-mon[39562]: osdmap e930: 12 total, 12 up, 12 in 2024-04-20T17:50:27.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:26 smithi005 ceph-mon[39562]: 9.9 scrub starts 2024-04-20T17:50:27.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:26 smithi005 ceph-mon[39562]: pgmap v2450: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 9.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 989 op/s 2024-04-20T17:50:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:26 smithi106 ceph-mon[44423]: 9.6 scrub ok 2024-04-20T17:50:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:26 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:50:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:26 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:50:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:26 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1546872234' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:50:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:26 smithi106 ceph-mon[44423]: osdmap e930: 12 total, 12 up, 12 in 2024-04-20T17:50:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:26 smithi106 ceph-mon[44423]: 9.9 scrub starts 2024-04-20T17:50:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:26 smithi106 ceph-mon[44423]: pgmap v2450: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 9.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 989 op/s 2024-04-20T17:50:27.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:26 smithi171 ceph-mon[44189]: 9.6 scrub ok 2024-04-20T17:50:27.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:26 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:50:27.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:26 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:50:27.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:26 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1546872234' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:50:27.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:26 smithi171 ceph-mon[44189]: osdmap e930: 12 total, 12 up, 12 in 2024-04-20T17:50:27.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:26 smithi171 ceph-mon[44189]: 9.9 scrub starts 2024-04-20T17:50:27.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:26 smithi171 ceph-mon[44189]: pgmap v2450: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 9.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 989 op/s 2024-04-20T17:50:27.879 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 81 16 156958 156942 121.065 65.4375 0.0124116 0.00825852 2024-04-20T17:50:27.902 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:50:27.902 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:50:27.902 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:50:27.902 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:50:27.902 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:50:27.902 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:50:27.903 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:50:27.903 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:50:27.903 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:50:27.903 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:50:27.903 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:50:27.919 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:50:27.920 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:50:27.920 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:50:27.920 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:50:27.921 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:50:27.921 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:50:27.921 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:50:27.921 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:50:27.921 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:50:27.921 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:50:27.921 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:50:27.921 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:50:27.921 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:50:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:27 smithi005 ceph-mon[39562]: 9.3 scrub ok 2024-04-20T17:50:28.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:27 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3740374948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:50:28.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:28.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:27 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:27 smithi106 ceph-mon[44423]: 9.3 scrub ok 2024-04-20T17:50:28.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:27 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3740374948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:50:28.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:28.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:27 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:27 smithi171 ceph-mon[44189]: 9.3 scrub ok 2024-04-20T17:50:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:27 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3740374948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:50:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:27 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:28.457 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:50:28.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:28.874 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:50:28.879 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 82 16 158014 157998 120.393 66 0.00646768 0.00830462 2024-04-20T17:50:28.975 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:50:29.006 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:29.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:28 smithi005 ceph-mon[39562]: 9.7 scrub ok 2024-04-20T17:50:29.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:28 smithi005 ceph-mon[39562]: 9.c scrub starts 2024-04-20T17:50:29.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:28 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:28 smithi005 ceph-mon[39562]: 9.0 scrub starts 2024-04-20T17:50:29.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:28 smithi005 ceph-mon[39562]: pgmap v2451: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 9.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-04-20T17:50:29.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:28 smithi106 ceph-mon[44423]: 9.7 scrub ok 2024-04-20T17:50:29.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:28 smithi106 ceph-mon[44423]: 9.c scrub starts 2024-04-20T17:50:29.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:28 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:28 smithi106 ceph-mon[44423]: 9.0 scrub starts 2024-04-20T17:50:29.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:28 smithi106 ceph-mon[44423]: pgmap v2451: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 9.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-04-20T17:50:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:28 smithi171 ceph-mon[44189]: 9.7 scrub ok 2024-04-20T17:50:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:28 smithi171 ceph-mon[44189]: 9.c scrub starts 2024-04-20T17:50:29.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:28 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:29.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:28 smithi171 ceph-mon[44189]: 9.0 scrub starts 2024-04-20T17:50:29.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:28 smithi171 ceph-mon[44189]: pgmap v2451: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 9.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-04-20T17:50:29.374 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:50:29.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:29.378 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:50:29.479 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:50:29.530 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:29.562 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:50:29.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:29.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:50:29.667 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:50:29.730 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:29.772 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:50:29.779 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:50:29.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:29.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:50:29.879 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 83 16 159073 159057 119.74 66.1875 0.010018 0.00834966 2024-04-20T17:50:29.892 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:50:29.937 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:29 smithi171 ceph-mon[44189]: 9.9 scrub ok 2024-04-20T17:50:29.937 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:29 smithi171 ceph-mon[44189]: 9.1 scrub starts 2024-04-20T17:50:29.956 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:30.006 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:50:30.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:30.011 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:29 smithi005 ceph-mon[39562]: 9.9 scrub ok 2024-04-20T17:50:30.012 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:29 smithi005 ceph-mon[39562]: 9.1 scrub starts 2024-04-20T17:50:30.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:50:30.113 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:50:30.164 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:30.182 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:50:30.182 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:50:30.183 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-04-20T17:50:30.183 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-04-20T17:50:30.183 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:50:30.200 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:50:30.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:30.204 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:50:30.242 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:50:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:29 smithi106 ceph-mon[44423]: 9.9 scrub ok 2024-04-20T17:50:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:29 smithi106 ceph-mon[44423]: 9.1 scrub starts 2024-04-20T17:50:30.305 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:50:30.359 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:30.790 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:50:30.880 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 84 16 160194 160178 119.149 70.0625 0.0106227 0.00839147 2024-04-20T17:50:31.061 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-20T17:50:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:30 smithi005 ceph-mon[39562]: 9.c scrub ok 2024-04-20T17:50:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:30 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:50:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:30 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:50:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:30 smithi005 ceph-mon[39562]: 9.0 scrub ok 2024-04-20T17:50:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:30 smithi005 ceph-mon[39562]: pgmap v2452: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 9.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-04-20T17:50:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:30 smithi106 ceph-mon[44423]: 9.c scrub ok 2024-04-20T17:50:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:30 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:50:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:30 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:50:31.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:30 smithi106 ceph-mon[44423]: 9.0 scrub ok 2024-04-20T17:50:31.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:30 smithi106 ceph-mon[44423]: pgmap v2452: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 9.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-04-20T17:50:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:30 smithi171 ceph-mon[44189]: 9.c scrub ok 2024-04-20T17:50:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:30 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:50:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:30 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:50:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:30 smithi171 ceph-mon[44189]: 9.0 scrub ok 2024-04-20T17:50:31.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:30 smithi171 ceph-mon[44189]: pgmap v2452: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 9.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-04-20T17:50:31.422 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:50:31.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:31.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:50:31.457 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:50:31.526 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:50:31.563 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:31.606 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:50:31.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:31.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:50:31.710 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:50:31.756 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:31.784 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:50:31.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:31.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:50:31.880 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 85 16 161275 161259 118.542 67.5625 0.0068182 0.00843387 2024-04-20T17:50:31.891 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:50:31.930 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:31.975 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:50:31.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:31.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:50:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:31 smithi106 ceph-mon[44423]: 9.1 scrub ok 2024-04-20T17:50:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:31 smithi106 ceph-mon[44423]: osdmap e931: 12 total, 12 up, 12 in 2024-04-20T17:50:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:31 smithi106 ceph-mon[44423]: 1.2 scrub starts 2024-04-20T17:50:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:31 smithi106 ceph-mon[44423]: 1.2 scrub ok 2024-04-20T17:50:32.015 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:50:32.067 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:31 smithi171 ceph-mon[44189]: 9.1 scrub ok 2024-04-20T17:50:32.067 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:31 smithi171 ceph-mon[44189]: osdmap e931: 12 total, 12 up, 12 in 2024-04-20T17:50:32.067 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:31 smithi171 ceph-mon[44189]: 1.2 scrub starts 2024-04-20T17:50:32.067 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:31 smithi171 ceph-mon[44189]: 1.2 scrub ok 2024-04-20T17:50:32.081 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:50:32.119 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:32.154 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:50:32.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:32.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:50:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:31 smithi005 ceph-mon[39562]: 9.1 scrub ok 2024-04-20T17:50:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:31 smithi005 ceph-mon[39562]: osdmap e931: 12 total, 12 up, 12 in 2024-04-20T17:50:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:31 smithi005 ceph-mon[39562]: 1.2 scrub starts 2024-04-20T17:50:32.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:31 smithi005 ceph-mon[39562]: 1.2 scrub ok 2024-04-20T17:50:32.269 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:50:32.299 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:32.880 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 86 16 162435 162419 118.007 72.5 0.0117073 0.00847288 2024-04-20T17:50:33.064 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:50:33.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:33.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:50:33.086 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:32 smithi005 ceph-mon[39562]: pgmap v2454: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 9.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-20T17:50:33.087 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=4052465775,config=1460075751,health=3897446751,kv=4003608488}) 2024-04-20T17:50:33.087 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=2463108999,logm=2999395231}) 2024-04-20T17:50:33.087 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4193293291}) 2024-04-20T17:50:33.087 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=636768834}) 2024-04-20T17:50:33.087 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1196664280}) 2024-04-20T17:50:33.087 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1865192766}) 2024-04-20T17:50:33.087 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3887263394}) 2024-04-20T17:50:33.087 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=49597769}) 2024-04-20T17:50:33.087 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3993859036}) 2024-04-20T17:50:33.087 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3093852987}) 2024-04-20T17:50:33.088 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3480184605}) 2024-04-20T17:50:33.088 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3882602304}) 2024-04-20T17:50:33.088 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2914389861}) 2024-04-20T17:50:33.088 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3137908861}) 2024-04-20T17:50:33.088 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=449977174}) 2024-04-20T17:50:33.088 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2034443618}) 2024-04-20T17:50:33.088 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3706487313}) 2024-04-20T17:50:33.088 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1737658243}) 2024-04-20T17:50:33.088 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3688122397}) 2024-04-20T17:50:33.088 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3160784421}) 2024-04-20T17:50:33.088 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4198125377}) 2024-04-20T17:50:33.088 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2150062672}) 2024-04-20T17:50:33.088 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1717882272}) 2024-04-20T17:50:33.088 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1723645565}) 2024-04-20T17:50:33.089 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:32 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1040971713}) 2024-04-20T17:50:33.170 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:50:33.213 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:33.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:32 smithi106 ceph-mon[44423]: pgmap v2454: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 9.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-20T17:50:33.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=4052465775,config=1460075751,health=3897446751,kv=4003608488}) 2024-04-20T17:50:33.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=2463108999,logm=2999395231}) 2024-04-20T17:50:33.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4193293291}) 2024-04-20T17:50:33.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=636768834}) 2024-04-20T17:50:33.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1196664280}) 2024-04-20T17:50:33.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1865192766}) 2024-04-20T17:50:33.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3887263394}) 2024-04-20T17:50:33.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=49597769}) 2024-04-20T17:50:33.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3993859036}) 2024-04-20T17:50:33.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3093852987}) 2024-04-20T17:50:33.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3480184605}) 2024-04-20T17:50:33.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3882602304}) 2024-04-20T17:50:33.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2914389861}) 2024-04-20T17:50:33.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3137908861}) 2024-04-20T17:50:33.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=449977174}) 2024-04-20T17:50:33.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2034443618}) 2024-04-20T17:50:33.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3706487313}) 2024-04-20T17:50:33.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1737658243}) 2024-04-20T17:50:33.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3688122397}) 2024-04-20T17:50:33.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3160784421}) 2024-04-20T17:50:33.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4198125377}) 2024-04-20T17:50:33.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2150062672}) 2024-04-20T17:50:33.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1717882272}) 2024-04-20T17:50:33.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1723645565}) 2024-04-20T17:50:33.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:32 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1040971713}) 2024-04-20T17:50:33.342 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:32 smithi171 ceph-mon[44189]: pgmap v2454: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 9.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-20T17:50:33.343 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=4052465775,config=1460075751,health=3897446751,kv=4003608488}) 2024-04-20T17:50:33.343 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=2463108999,logm=2999395231}) 2024-04-20T17:50:33.343 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4193293291}) 2024-04-20T17:50:33.343 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=636768834}) 2024-04-20T17:50:33.343 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1196664280}) 2024-04-20T17:50:33.343 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1865192766}) 2024-04-20T17:50:33.343 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3887263394}) 2024-04-20T17:50:33.343 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=49597769}) 2024-04-20T17:50:33.343 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3993859036}) 2024-04-20T17:50:33.343 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3093852987}) 2024-04-20T17:50:33.343 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3480184605}) 2024-04-20T17:50:33.343 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3882602304}) 2024-04-20T17:50:33.343 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2914389861}) 2024-04-20T17:50:33.343 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3137908861}) 2024-04-20T17:50:33.344 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=449977174}) 2024-04-20T17:50:33.344 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2034443618}) 2024-04-20T17:50:33.344 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3706487313}) 2024-04-20T17:50:33.344 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1737658243}) 2024-04-20T17:50:33.344 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3688122397}) 2024-04-20T17:50:33.344 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3160784421}) 2024-04-20T17:50:33.344 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4198125377}) 2024-04-20T17:50:33.344 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2150062672}) 2024-04-20T17:50:33.344 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1717882272}) 2024-04-20T17:50:33.344 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1723645565}) 2024-04-20T17:50:33.344 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:32 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1040971713}) 2024-04-20T17:50:33.624 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-20T17:50:33.624 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:50:33.881 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 87 16 163586 163570 117.477 71.9375 0.00564701 0.00851051 2024-04-20T17:50:34.077 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:50:34.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:34.081 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:50:34.182 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:50:34.229 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:34.257 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:50:34.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:34.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=200140472}) 2024-04-20T17:50:34.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2586830503}) 2024-04-20T17:50:34.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=702330169}) 2024-04-20T17:50:34.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3347400910}) 2024-04-20T17:50:34.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=615139296}) 2024-04-20T17:50:34.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1459177862}) 2024-04-20T17:50:34.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3608817982}) 2024-04-20T17:50:34.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3037416429}) 2024-04-20T17:50:34.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3978794581}) 2024-04-20T17:50:34.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1977121338}) 2024-04-20T17:50:34.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2477515946}) 2024-04-20T17:50:34.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=476859001}) 2024-04-20T17:50:34.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=82828139}) 2024-04-20T17:50:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4028839906}) 2024-04-20T17:50:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=441097561}) 2024-04-20T17:50:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2590740467}) 2024-04-20T17:50:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1384133379}) 2024-04-20T17:50:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3612053026}) 2024-04-20T17:50:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3761557930}) 2024-04-20T17:50:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3739045338}) 2024-04-20T17:50:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4197296035}) 2024-04-20T17:50:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3625208759}) 2024-04-20T17:50:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=492724351}) 2024-04-20T17:50:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3584042749}) 2024-04-20T17:50:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3600773743}) 2024-04-20T17:50:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3256608416}) 2024-04-20T17:50:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2764609249}) 2024-04-20T17:50:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1436236559}) 2024-04-20T17:50:34.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2343041626}) 2024-04-20T17:50:34.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2152823352}) 2024-04-20T17:50:34.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3080399990}) 2024-04-20T17:50:34.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3485974436}) 2024-04-20T17:50:34.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3813577671}) 2024-04-20T17:50:34.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=853425}) 2024-04-20T17:50:34.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1193320234}) 2024-04-20T17:50:34.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=466658268}) 2024-04-20T17:50:34.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2473223988}) 2024-04-20T17:50:34.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1468223970}) 2024-04-20T17:50:34.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=462524742}) 2024-04-20T17:50:34.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3887359116}) 2024-04-20T17:50:34.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: 1.9 scrub starts 2024-04-20T17:50:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2696463908}) 2024-04-20T17:50:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=28235703}) 2024-04-20T17:50:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3723731518}) 2024-04-20T17:50:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2215712887}) 2024-04-20T17:50:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=360944714}) 2024-04-20T17:50:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=73858805}) 2024-04-20T17:50:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1468691022}) 2024-04-20T17:50:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1409709820}) 2024-04-20T17:50:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=417643096}) 2024-04-20T17:50:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=763555463}) 2024-04-20T17:50:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1738911482}) 2024-04-20T17:50:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3769471453}) 2024-04-20T17:50:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1366190346}) 2024-04-20T17:50:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=250027142}) 2024-04-20T17:50:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785102555}) 2024-04-20T17:50:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2817459848}) 2024-04-20T17:50:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=163563553}) 2024-04-20T17:50:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1853459494}) 2024-04-20T17:50:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4010896201}) 2024-04-20T17:50:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4094085038}) 2024-04-20T17:50:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4121067412}) 2024-04-20T17:50:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=93797868}) 2024-04-20T17:50:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2605386944}) 2024-04-20T17:50:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=760611023}) 2024-04-20T17:50:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1258578923}) 2024-04-20T17:50:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2100510447}) 2024-04-20T17:50:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2138604806}) 2024-04-20T17:50:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=42573043}) 2024-04-20T17:50:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2258843547}) 2024-04-20T17:50:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=420008774}) 2024-04-20T17:50:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4172007788}) 2024-04-20T17:50:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2345764220}) 2024-04-20T17:50:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1016611783}) 2024-04-20T17:50:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=329965217}) 2024-04-20T17:50:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3138097449}) 2024-04-20T17:50:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3394378842}) 2024-04-20T17:50:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=775586749}) 2024-04-20T17:50:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1047498371}) 2024-04-20T17:50:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1511763345}) 2024-04-20T17:50:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1416831910}) 2024-04-20T17:50:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=941000905}) 2024-04-20T17:50:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3834195948}) 2024-04-20T17:50:34.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1942689640}) 2024-04-20T17:50:34.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2608065337}) 2024-04-20T17:50:34.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1997762820}) 2024-04-20T17:50:34.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=58,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=9,mon_config_key=9} crc {logm=1652231478,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=2068962153,mon_config_key=3196554281}) 2024-04-20T17:50:34.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3246587472}) 2024-04-20T17:50:34.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=18669262}) 2024-04-20T17:50:34.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=81,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=1} crc {mon_config_key=1040563310,monmap=717113318,osd_metadata=2589079959,osd_pg_creating=1182139901,osdmap=3047347455}) 2024-04-20T17:50:34.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2098762696}) 2024-04-20T17:50:34.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=22} crc {osdmap=3608087693}) 2024-04-20T17:50:34.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:33 smithi106 ceph-mon[44423]: 1.9 scrub ok 2024-04-20T17:50:34.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:50:34.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=200140472}) 2024-04-20T17:50:34.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2586830503}) 2024-04-20T17:50:34.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=702330169}) 2024-04-20T17:50:34.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3347400910}) 2024-04-20T17:50:34.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=615139296}) 2024-04-20T17:50:34.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1459177862}) 2024-04-20T17:50:34.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3608817982}) 2024-04-20T17:50:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3037416429}) 2024-04-20T17:50:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3978794581}) 2024-04-20T17:50:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1977121338}) 2024-04-20T17:50:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2477515946}) 2024-04-20T17:50:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=476859001}) 2024-04-20T17:50:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=82828139}) 2024-04-20T17:50:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4028839906}) 2024-04-20T17:50:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=441097561}) 2024-04-20T17:50:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2590740467}) 2024-04-20T17:50:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1384133379}) 2024-04-20T17:50:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3612053026}) 2024-04-20T17:50:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3761557930}) 2024-04-20T17:50:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3739045338}) 2024-04-20T17:50:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4197296035}) 2024-04-20T17:50:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3625208759}) 2024-04-20T17:50:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=492724351}) 2024-04-20T17:50:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3584042749}) 2024-04-20T17:50:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3600773743}) 2024-04-20T17:50:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3256608416}) 2024-04-20T17:50:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2764609249}) 2024-04-20T17:50:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1436236559}) 2024-04-20T17:50:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2343041626}) 2024-04-20T17:50:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2152823352}) 2024-04-20T17:50:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3080399990}) 2024-04-20T17:50:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3485974436}) 2024-04-20T17:50:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3813577671}) 2024-04-20T17:50:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=853425}) 2024-04-20T17:50:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1193320234}) 2024-04-20T17:50:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=466658268}) 2024-04-20T17:50:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2473223988}) 2024-04-20T17:50:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1468223970}) 2024-04-20T17:50:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=462524742}) 2024-04-20T17:50:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3887359116}) 2024-04-20T17:50:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: 1.9 scrub starts 2024-04-20T17:50:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2696463908}) 2024-04-20T17:50:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=28235703}) 2024-04-20T17:50:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3723731518}) 2024-04-20T17:50:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2215712887}) 2024-04-20T17:50:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=360944714}) 2024-04-20T17:50:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=73858805}) 2024-04-20T17:50:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1468691022}) 2024-04-20T17:50:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1409709820}) 2024-04-20T17:50:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=417643096}) 2024-04-20T17:50:34.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=763555463}) 2024-04-20T17:50:34.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1738911482}) 2024-04-20T17:50:34.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3769471453}) 2024-04-20T17:50:34.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1366190346}) 2024-04-20T17:50:34.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=250027142}) 2024-04-20T17:50:34.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785102555}) 2024-04-20T17:50:34.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2817459848}) 2024-04-20T17:50:34.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=163563553}) 2024-04-20T17:50:34.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1853459494}) 2024-04-20T17:50:34.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4010896201}) 2024-04-20T17:50:34.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4094085038}) 2024-04-20T17:50:34.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4121067412}) 2024-04-20T17:50:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=93797868}) 2024-04-20T17:50:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2605386944}) 2024-04-20T17:50:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=760611023}) 2024-04-20T17:50:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1258578923}) 2024-04-20T17:50:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2100510447}) 2024-04-20T17:50:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2138604806}) 2024-04-20T17:50:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=42573043}) 2024-04-20T17:50:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2258843547}) 2024-04-20T17:50:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=420008774}) 2024-04-20T17:50:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4172007788}) 2024-04-20T17:50:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2345764220}) 2024-04-20T17:50:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1016611783}) 2024-04-20T17:50:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=329965217}) 2024-04-20T17:50:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3138097449}) 2024-04-20T17:50:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3394378842}) 2024-04-20T17:50:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=775586749}) 2024-04-20T17:50:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1047498371}) 2024-04-20T17:50:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1511763345}) 2024-04-20T17:50:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1416831910}) 2024-04-20T17:50:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=941000905}) 2024-04-20T17:50:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3834195948}) 2024-04-20T17:50:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1942689640}) 2024-04-20T17:50:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2608065337}) 2024-04-20T17:50:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1997762820}) 2024-04-20T17:50:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=58,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=9,mon_config_key=9} crc {logm=1652231478,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=2068962153,mon_config_key=3196554281}) 2024-04-20T17:50:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3246587472}) 2024-04-20T17:50:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=18669262}) 2024-04-20T17:50:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=81,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=1} crc {mon_config_key=1040563310,monmap=717113318,osd_metadata=2589079959,osd_pg_creating=1182139901,osdmap=3047347455}) 2024-04-20T17:50:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2098762696}) 2024-04-20T17:50:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=22} crc {osdmap=3608087693}) 2024-04-20T17:50:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:33 smithi171 ceph-mon[44189]: 1.9 scrub ok 2024-04-20T17:50:34.372 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:50:34.405 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:34.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=200140472}) 2024-04-20T17:50:34.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2586830503}) 2024-04-20T17:50:34.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=702330169}) 2024-04-20T17:50:34.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3347400910}) 2024-04-20T17:50:34.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=615139296}) 2024-04-20T17:50:34.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1459177862}) 2024-04-20T17:50:34.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3608817982}) 2024-04-20T17:50:34.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3037416429}) 2024-04-20T17:50:34.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3978794581}) 2024-04-20T17:50:34.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1977121338}) 2024-04-20T17:50:34.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2477515946}) 2024-04-20T17:50:34.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=476859001}) 2024-04-20T17:50:34.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=82828139}) 2024-04-20T17:50:34.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4028839906}) 2024-04-20T17:50:34.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=441097561}) 2024-04-20T17:50:34.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2590740467}) 2024-04-20T17:50:34.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1384133379}) 2024-04-20T17:50:34.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3612053026}) 2024-04-20T17:50:34.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3761557930}) 2024-04-20T17:50:34.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3739045338}) 2024-04-20T17:50:34.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4197296035}) 2024-04-20T17:50:34.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3625208759}) 2024-04-20T17:50:34.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=492724351}) 2024-04-20T17:50:34.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3584042749}) 2024-04-20T17:50:34.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3600773743}) 2024-04-20T17:50:34.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3256608416}) 2024-04-20T17:50:34.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2764609249}) 2024-04-20T17:50:34.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1436236559}) 2024-04-20T17:50:34.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2343041626}) 2024-04-20T17:50:34.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2152823352}) 2024-04-20T17:50:34.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3080399990}) 2024-04-20T17:50:34.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3485974436}) 2024-04-20T17:50:34.442 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3813577671}) 2024-04-20T17:50:34.442 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=853425}) 2024-04-20T17:50:34.442 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1193320234}) 2024-04-20T17:50:34.442 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=466658268}) 2024-04-20T17:50:34.442 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2473223988}) 2024-04-20T17:50:34.442 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1468223970}) 2024-04-20T17:50:34.442 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=462524742}) 2024-04-20T17:50:34.442 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3887359116}) 2024-04-20T17:50:34.442 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: 1.9 scrub starts 2024-04-20T17:50:34.442 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2696463908}) 2024-04-20T17:50:34.442 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=28235703}) 2024-04-20T17:50:34.442 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3723731518}) 2024-04-20T17:50:34.442 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2215712887}) 2024-04-20T17:50:34.442 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=360944714}) 2024-04-20T17:50:34.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=73858805}) 2024-04-20T17:50:34.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1468691022}) 2024-04-20T17:50:34.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1409709820}) 2024-04-20T17:50:34.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=417643096}) 2024-04-20T17:50:34.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=763555463}) 2024-04-20T17:50:34.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1738911482}) 2024-04-20T17:50:34.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3769471453}) 2024-04-20T17:50:34.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1366190346}) 2024-04-20T17:50:34.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=250027142}) 2024-04-20T17:50:34.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3785102555}) 2024-04-20T17:50:34.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2817459848}) 2024-04-20T17:50:34.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=163563553}) 2024-04-20T17:50:34.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1853459494}) 2024-04-20T17:50:34.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4010896201}) 2024-04-20T17:50:34.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4094085038}) 2024-04-20T17:50:34.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4121067412}) 2024-04-20T17:50:34.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=93797868}) 2024-04-20T17:50:34.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2605386944}) 2024-04-20T17:50:34.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=760611023}) 2024-04-20T17:50:34.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1258578923}) 2024-04-20T17:50:34.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2100510447}) 2024-04-20T17:50:34.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2138604806}) 2024-04-20T17:50:34.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=42573043}) 2024-04-20T17:50:34.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2258843547}) 2024-04-20T17:50:34.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=420008774}) 2024-04-20T17:50:34.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4172007788}) 2024-04-20T17:50:34.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2345764220}) 2024-04-20T17:50:34.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1016611783}) 2024-04-20T17:50:34.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=329965217}) 2024-04-20T17:50:34.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3138097449}) 2024-04-20T17:50:34.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3394378842}) 2024-04-20T17:50:34.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=775586749}) 2024-04-20T17:50:34.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1047498371}) 2024-04-20T17:50:34.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1511763345}) 2024-04-20T17:50:34.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1416831910}) 2024-04-20T17:50:34.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=941000905}) 2024-04-20T17:50:34.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3834195948}) 2024-04-20T17:50:34.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1942689640}) 2024-04-20T17:50:34.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2608065337}) 2024-04-20T17:50:34.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1997762820}) 2024-04-20T17:50:34.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=58,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=9,mon_config_key=9} crc {logm=1652231478,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=2068962153,mon_config_key=3196554281}) 2024-04-20T17:50:34.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3246587472}) 2024-04-20T17:50:34.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=18669262}) 2024-04-20T17:50:34.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=81,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=1} crc {mon_config_key=1040563310,monmap=717113318,osd_metadata=2589079959,osd_pg_creating=1182139901,osdmap=3047347455}) 2024-04-20T17:50:34.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2098762696}) 2024-04-20T17:50:34.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=22} crc {osdmap=3608087693}) 2024-04-20T17:50:34.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:33 smithi005 ceph-mon[39562]: 1.9 scrub ok 2024-04-20T17:50:34.446 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:50:34.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:34.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:50:34.559 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:50:34.595 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:34.880 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 88 16 164658 164642 116.904 67 0.0132839 0.00855242 2024-04-20T17:50:34.914 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:50:34.915 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:50:34.915 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-04-20T17:50:34.915 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-04-20T17:50:34.915 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:50:34.915 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.915 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:184969 8.a 8:54edb4dd:::benchmark_data_smithi171_175757_object39855:head [delete] snapc 0=[] ondisk+write+known_if_redirected e830)", 2024-04-20T17:50:34.916 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:46:04.914245+0000", 2024-04-20T17:50:34.916 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 269.99633669799999, 2024-04-20T17:50:34.916 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.14490816400000001, 2024-04-20T17:50:34.916 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:50:34.916 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:50:34.916 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:34.916 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:50:34.916 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:50:34.916 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:50:34.916 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 184969 2024-04-20T17:50:34.916 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.916 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:50:34.916 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.917 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:50:34.917 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.914245+0000", 2024-04-20T17:50:34.917 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.917 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.917 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.917 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:50:34.917 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.914245+0000", 2024-04-20T17:50:34.917 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.917 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.917 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.917 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:50:34.917 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.914248+0000", 2024-04-20T17:50:34.918 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8629999999999999e-06 2024-04-20T17:50:34.918 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.918 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.918 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:50:34.918 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.914257+0000", 2024-04-20T17:50:34.918 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2649999999999996e-06 2024-04-20T17:50:34.918 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.918 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.918 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:50:34.918 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.914272+0000", 2024-04-20T17:50:34.918 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5625999999999999e-05 2024-04-20T17:50:34.918 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.918 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.919 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:50:34.919 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.914282+0000", 2024-04-20T17:50:34.919 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7450000000000007e-06 2024-04-20T17:50:34.919 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.919 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.919 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:50:34.919 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.916563+0000", 2024-04-20T17:50:34.919 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.002281136 2024-04-20T17:50:34.919 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.919 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.919 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:50:34.919 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.916664+0000", 2024-04-20T17:50:34.919 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010123099999999999 2024-04-20T17:50:34.920 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.920 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.920 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:50:34.920 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.916767+0000", 2024-04-20T17:50:34.920 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000102524 2024-04-20T17:50:34.920 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.920 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.920 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:50:34.920 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.926769+0000", 2024-04-20T17:50:34.920 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010002159 2024-04-20T17:50:34.920 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.920 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.920 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:34.920 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.926803+0000", 2024-04-20T17:50:34.921 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3776000000000003e-05 2024-04-20T17:50:34.921 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.921 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.921 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:50:34.921 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:05.059118+0000", 2024-04-20T17:50:34.921 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.13231567899999999 2024-04-20T17:50:34.921 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.921 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.921 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:50:34.921 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:05.059154+0000", 2024-04-20T17:50:34.921 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5160000000000002e-05 2024-04-20T17:50:34.921 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:34.921 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:34.922 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:34.922 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.922 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.922 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:184971 8.a 8:51a40f43:::benchmark_data_smithi171_175757_object39857:head [delete] snapc 0=[] ondisk+write+known_if_redirected e830)", 2024-04-20T17:50:34.922 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:46:04.914303+0000", 2024-04-20T17:50:34.922 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 269.996279061, 2024-04-20T17:50:34.922 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.14747222400000001, 2024-04-20T17:50:34.922 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:50:34.922 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:50:34.922 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:34.922 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:50:34.922 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:50:34.922 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:50:34.922 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 184971 2024-04-20T17:50:34.923 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.923 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:50:34.923 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.923 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:50:34.923 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.914303+0000", 2024-04-20T17:50:34.923 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.923 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.923 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.923 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:50:34.923 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.914303+0000", 2024-04-20T17:50:34.923 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.923 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.923 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.923 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:50:34.923 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.914304+0000", 2024-04-20T17:50:34.924 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1500000000000004e-07 2024-04-20T17:50:34.924 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.924 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.924 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:50:34.924 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.914307+0000", 2024-04-20T17:50:34.924 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4450000000000001e-06 2024-04-20T17:50:34.924 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.924 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.924 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:50:34.924 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.914313+0000", 2024-04-20T17:50:34.924 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9030000000000001e-06 2024-04-20T17:50:34.924 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.924 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.924 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:50:34.924 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.914318+0000", 2024-04-20T17:50:34.925 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1510000000000003e-06 2024-04-20T17:50:34.925 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.925 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.925 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:50:34.925 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.921611+0000", 2024-04-20T17:50:34.925 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0072929819999999999 2024-04-20T17:50:34.925 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.925 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.925 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:50:34.925 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.921723+0000", 2024-04-20T17:50:34.925 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000111241 2024-04-20T17:50:34.925 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.925 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.925 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:50:34.926 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.921825+0000", 2024-04-20T17:50:34.926 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000102965 2024-04-20T17:50:34.926 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.926 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.926 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:50:34.926 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:05.059168+0000", 2024-04-20T17:50:34.926 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.137342665 2024-04-20T17:50:34.926 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.926 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.926 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:34.926 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:05.061689+0000", 2024-04-20T17:50:34.926 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0025211360000000002 2024-04-20T17:50:34.926 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.926 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.927 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:50:34.927 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:05.061737+0000", 2024-04-20T17:50:34.927 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7627999999999999e-05 2024-04-20T17:50:34.927 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.927 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.927 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:50:34.927 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:05.061775+0000", 2024-04-20T17:50:34.927 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8393000000000003e-05 2024-04-20T17:50:34.927 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:34.927 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:34.927 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:34.927 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.927 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.927 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:184974 8.b 8:d9e7a912:::benchmark_data_smithi171_175757_object39860:head [delete] snapc 0=[] ondisk+write+known_if_redirected e830)", 2024-04-20T17:50:34.927 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:46:04.915278+0000", 2024-04-20T17:50:34.928 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 269.99530399700001, 2024-04-20T17:50:34.928 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.14611369299999999, 2024-04-20T17:50:34.928 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:50:34.928 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:50:34.928 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:34.928 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:50:34.928 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:50:34.928 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:50:34.928 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 184974 2024-04-20T17:50:34.928 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.928 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:50:34.928 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.928 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:50:34.928 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.915278+0000", 2024-04-20T17:50:34.929 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.929 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.929 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.929 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:50:34.929 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.915278+0000", 2024-04-20T17:50:34.929 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.929 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.929 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.929 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:50:34.929 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.915280+0000", 2024-04-20T17:50:34.929 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.663e-06 2024-04-20T17:50:34.929 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.929 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.930 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:50:34.930 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.915286+0000", 2024-04-20T17:50:34.930 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8370000000000002e-06 2024-04-20T17:50:34.930 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.930 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.930 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:50:34.930 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.915296+0000", 2024-04-20T17:50:34.930 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.058e-05 2024-04-20T17:50:34.930 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.930 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.930 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:50:34.930 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.915305+0000", 2024-04-20T17:50:34.930 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2910000000000007e-06 2024-04-20T17:50:34.930 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.931 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.931 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:50:34.931 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.915547+0000", 2024-04-20T17:50:34.931 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000242966 2024-04-20T17:50:34.931 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.931 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.931 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:50:34.931 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.915661+0000", 2024-04-20T17:50:34.931 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000114026 2024-04-20T17:50:34.931 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.931 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.931 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4", 2024-04-20T17:50:34.931 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.915767+0000", 2024-04-20T17:50:34.931 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000105475 2024-04-20T17:50:34.932 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.932 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.932 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:34.932 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:04.923877+0000", 2024-04-20T17:50:34.932 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0081096080000000008 2024-04-20T17:50:34.932 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.932 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.932 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:50:34.932 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:05.061289+0000", 2024-04-20T17:50:34.932 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.13741273900000001 2024-04-20T17:50:34.933 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.933 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.933 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:50:34.933 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:05.061340+0000", 2024-04-20T17:50:34.933 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1094000000000002e-05 2024-04-20T17:50:34.933 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.933 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.933 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:50:34.933 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:05.061392+0000", 2024-04-20T17:50:34.933 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1413999999999999e-05 2024-04-20T17:50:34.933 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:34.933 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:34.933 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:34.933 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.934 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.934 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:217335 8.b 8:d94e69a9:::benchmark_data_smithi171_175757_object72221:head [delete] snapc 0=[] ondisk+write+known_if_redirected e854)", 2024-04-20T17:50:34.934 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:46:56.596481+0000", 2024-04-20T17:50:34.934 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 218.31410150299999, 2024-04-20T17:50:34.934 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.050795687999999999, 2024-04-20T17:50:34.934 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:50:34.934 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:50:34.934 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:34.934 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:50:34.934 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:50:34.934 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:50:34.934 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 217335 2024-04-20T17:50:34.934 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.934 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:50:34.935 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.935 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:50:34.935 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:56.596481+0000", 2024-04-20T17:50:34.935 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.935 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.935 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.935 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:50:34.935 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:56.596481+0000", 2024-04-20T17:50:34.935 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.935 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.935 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.935 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:50:34.935 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:56.596483+0000", 2024-04-20T17:50:34.936 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2560000000000001e-06 2024-04-20T17:50:34.936 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.936 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.936 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:50:34.936 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:56.596490+0000", 2024-04-20T17:50:34.936 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3699999999999997e-06 2024-04-20T17:50:34.936 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.936 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.936 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:50:34.936 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:56.596504+0000", 2024-04-20T17:50:34.936 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3797e-05 2024-04-20T17:50:34.936 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.936 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.936 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:50:34.937 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:56.596513+0000", 2024-04-20T17:50:34.937 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6470000000000004e-06 2024-04-20T17:50:34.937 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.937 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.937 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:50:34.937 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:56.596594+0000", 2024-04-20T17:50:34.937 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1558999999999996e-05 2024-04-20T17:50:34.937 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.937 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.937 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:50:34.937 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:56.596723+0000", 2024-04-20T17:50:34.937 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012881599999999999 2024-04-20T17:50:34.937 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.937 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.938 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4", 2024-04-20T17:50:34.938 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:56.596840+0000", 2024-04-20T17:50:34.938 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000116607 2024-04-20T17:50:34.938 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.938 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.938 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:34.938 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:56.615227+0000", 2024-04-20T17:50:34.938 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018387582999999999 2024-04-20T17:50:34.938 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.938 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.938 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:50:34.938 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:56.647162+0000", 2024-04-20T17:50:34.938 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.031934958999999999 2024-04-20T17:50:34.939 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.939 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.939 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:50:34.939 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:56.647226+0000", 2024-04-20T17:50:34.939 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4079999999999996e-05 2024-04-20T17:50:34.939 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.939 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.939 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:50:34.939 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:56.647276+0000", 2024-04-20T17:50:34.939 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0013999999999999e-05 2024-04-20T17:50:34.939 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:34.939 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:34.939 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:34.939 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.939 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.940 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:217336 8.b 8:db9b3c45:::benchmark_data_smithi171_175757_object72222:head [delete] snapc 0=[] ondisk+write+known_if_redirected e854)", 2024-04-20T17:50:34.940 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:46:56.596535+0000", 2024-04-20T17:50:34.940 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 218.314047254, 2024-04-20T17:50:34.940 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.050883247999999999, 2024-04-20T17:50:34.940 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:50:34.940 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:50:34.940 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:34.940 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:50:34.940 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:50:34.940 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:50:34.940 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 217336 2024-04-20T17:50:34.940 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.940 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:50:34.941 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.941 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:50:34.941 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:56.596535+0000", 2024-04-20T17:50:34.941 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.941 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.941 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.941 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:50:34.941 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:56.596535+0000", 2024-04-20T17:50:34.941 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.941 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.941 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.941 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:50:34.941 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:56.596536+0000", 2024-04-20T17:50:34.941 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4199999999999995e-07 2024-04-20T17:50:34.942 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.942 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.942 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:50:34.942 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:56.596539+0000", 2024-04-20T17:50:34.942 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2650000000000001e-06 2024-04-20T17:50:34.942 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.942 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.942 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:50:34.942 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:56.596545+0000", 2024-04-20T17:50:34.942 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2940000000000002e-06 2024-04-20T17:50:34.942 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.942 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.942 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:50:34.943 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:56.596551+0000", 2024-04-20T17:50:34.943 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6370000000000004e-06 2024-04-20T17:50:34.943 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.943 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.943 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:50:34.943 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:56.602115+0000", 2024-04-20T17:50:34.943 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0055642249999999999 2024-04-20T17:50:34.943 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.943 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.943 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:50:34.943 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:56.602227+0000", 2024-04-20T17:50:34.943 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011192999999999999 2024-04-20T17:50:34.943 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.943 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.943 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4", 2024-04-20T17:50:34.944 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:56.602331+0000", 2024-04-20T17:50:34.944 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000104183 2024-04-20T17:50:34.944 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.944 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.944 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:34.944 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:56.615266+0000", 2024-04-20T17:50:34.944 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.012935149 2024-04-20T17:50:34.944 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.944 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.944 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:50:34.944 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:56.647337+0000", 2024-04-20T17:50:34.944 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032070987000000002 2024-04-20T17:50:34.944 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.944 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.944 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:50:34.945 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:56.647379+0000", 2024-04-20T17:50:34.945 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1298000000000003e-05 2024-04-20T17:50:34.945 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.945 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.945 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:50:34.945 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:46:56.647418+0000", 2024-04-20T17:50:34.945 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9538e-05 2024-04-20T17:50:34.945 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:34.945 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:34.945 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:34.945 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.945 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.945 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:259320 8.a 8:58945840:::benchmark_data_smithi171_175757_object114206:head [delete] snapc 0=[] ondisk+write+known_if_redirected e872)", 2024-04-20T17:50:34.946 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:47:52.424799+0000", 2024-04-20T17:50:34.946 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 162.48578267299999, 2024-04-20T17:50:34.946 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.064675236999999997, 2024-04-20T17:50:34.946 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:50:34.946 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:50:34.946 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:34.946 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:50:34.946 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:50:34.946 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:50:34.946 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 259320 2024-04-20T17:50:34.946 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.946 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:50:34.946 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.946 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:50:34.947 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.424799+0000", 2024-04-20T17:50:34.947 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.947 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.947 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.947 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:50:34.947 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.424799+0000", 2024-04-20T17:50:34.947 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.947 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.947 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.947 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:50:34.947 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.424800+0000", 2024-04-20T17:50:34.947 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1900000000000001e-07 2024-04-20T17:50:34.947 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.947 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.947 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:50:34.948 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.424804+0000", 2024-04-20T17:50:34.948 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8800000000000001e-06 2024-04-20T17:50:34.948 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.948 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.948 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:50:34.948 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.424810+0000", 2024-04-20T17:50:34.948 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9320000000000001e-06 2024-04-20T17:50:34.948 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.948 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.948 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:50:34.948 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.424816+0000", 2024-04-20T17:50:34.948 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6890000000000001e-06 2024-04-20T17:50:34.948 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.948 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.949 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:50:34.949 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.433578+0000", 2024-04-20T17:50:34.949 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0087618409999999994 2024-04-20T17:50:34.949 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.949 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.949 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:50:34.949 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.433686+0000", 2024-04-20T17:50:34.949 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010862900000000001 2024-04-20T17:50:34.949 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.949 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.949 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:50:34.949 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.433781+0000", 2024-04-20T17:50:34.949 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.4628000000000003e-05 2024-04-20T17:50:34.950 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.950 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.950 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:50:34.950 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.442229+0000", 2024-04-20T17:50:34.950 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0084479480000000003 2024-04-20T17:50:34.950 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.950 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.950 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:34.950 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.489345+0000", 2024-04-20T17:50:34.950 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.047115691000000001 2024-04-20T17:50:34.950 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.950 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.950 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:50:34.950 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.489401+0000", 2024-04-20T17:50:34.951 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6162999999999997e-05 2024-04-20T17:50:34.951 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.951 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.951 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:50:34.951 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.489475+0000", 2024-04-20T17:50:34.951 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3917000000000004e-05 2024-04-20T17:50:34.951 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:34.951 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:34.951 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:34.951 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.951 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.951 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:259324 8.a 8:518c4ead:::benchmark_data_smithi171_175757_object114210:head [delete] snapc 0=[] ondisk+write+known_if_redirected e872)", 2024-04-20T17:50:34.951 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:47:52.433706+0000", 2024-04-20T17:50:34.951 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 162.47687647000001, 2024-04-20T17:50:34.951 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.055898673000000003, 2024-04-20T17:50:34.952 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:50:34.952 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:50:34.952 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:34.952 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:50:34.952 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:50:34.952 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:50:34.952 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 259324 2024-04-20T17:50:34.952 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.952 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:50:34.952 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.952 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:50:34.952 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.433706+0000", 2024-04-20T17:50:34.952 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.952 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.953 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.953 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:50:34.953 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.433706+0000", 2024-04-20T17:50:34.953 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.953 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.953 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.953 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:50:34.953 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.433708+0000", 2024-04-20T17:50:34.953 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.238e-06 2024-04-20T17:50:34.953 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.953 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.953 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:50:34.953 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.433717+0000", 2024-04-20T17:50:34.953 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.431e-06 2024-04-20T17:50:34.953 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.953 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.953 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:50:34.954 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.433730+0000", 2024-04-20T17:50:34.954 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2367e-05 2024-04-20T17:50:34.954 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.954 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.954 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:50:34.954 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.433739+0000", 2024-04-20T17:50:34.954 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3850000000000008e-06 2024-04-20T17:50:34.954 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.954 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.954 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:50:34.955 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.437701+0000", 2024-04-20T17:50:34.955 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0039620319999999999 2024-04-20T17:50:34.955 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.955 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.955 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:50:34.955 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.437814+0000", 2024-04-20T17:50:34.955 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000113334 2024-04-20T17:50:34.955 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.955 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.955 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:50:34.955 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.437899+0000", 2024-04-20T17:50:34.955 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4081999999999999e-05 2024-04-20T17:50:34.955 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.956 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.956 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:50:34.956 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.442391+0000", 2024-04-20T17:50:34.956 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0044922360000000001 2024-04-20T17:50:34.956 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.956 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.956 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:34.956 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.489522+0000", 2024-04-20T17:50:34.956 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.047131179000000002 2024-04-20T17:50:34.956 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.956 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.956 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:50:34.956 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.489569+0000", 2024-04-20T17:50:34.956 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6980000000000001e-05 2024-04-20T17:50:34.956 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.957 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.957 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:50:34.957 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.489604+0000", 2024-04-20T17:50:34.957 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5408999999999999e-05 2024-04-20T17:50:34.957 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:34.957 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:34.957 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:34.957 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.957 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.957 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:259336 8.a 8:52374263:::benchmark_data_smithi171_175757_object114222:head [delete] snapc 0=[] ondisk+write+known_if_redirected e872)", 2024-04-20T17:50:34.957 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:47:52.444096+0000", 2024-04-20T17:50:34.957 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 162.46648649100001, 2024-04-20T17:50:34.957 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.045636523999999998, 2024-04-20T17:50:34.957 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:50:34.957 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:50:34.958 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:34.958 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:50:34.958 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:50:34.958 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:50:34.958 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 259336 2024-04-20T17:50:34.958 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.958 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:50:34.958 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.959 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:50:34.959 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.444096+0000", 2024-04-20T17:50:34.959 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.959 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.959 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.959 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:50:34.959 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.444096+0000", 2024-04-20T17:50:34.959 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.959 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.960 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.960 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:50:34.960 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.444097+0000", 2024-04-20T17:50:34.960 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7990000000000001e-06 2024-04-20T17:50:34.960 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.960 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.960 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:50:34.960 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.444103+0000", 2024-04-20T17:50:34.960 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2279999999999998e-06 2024-04-20T17:50:34.960 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.960 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.960 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:50:34.960 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.444115+0000", 2024-04-20T17:50:34.961 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2449000000000001e-05 2024-04-20T17:50:34.961 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.961 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.961 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:50:34.961 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.444122+0000", 2024-04-20T17:50:34.961 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3640000000000001e-06 2024-04-20T17:50:34.961 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.961 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.961 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:50:34.961 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.444170+0000", 2024-04-20T17:50:34.961 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8000000000000001e-05 2024-04-20T17:50:34.961 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.961 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.962 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:50:34.962 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.444293+0000", 2024-04-20T17:50:34.962 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012268500000000001 2024-04-20T17:50:34.962 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.962 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.962 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:50:34.962 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.444418+0000", 2024-04-20T17:50:34.962 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012457899999999999 2024-04-20T17:50:34.962 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.962 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.962 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:50:34.962 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.456037+0000", 2024-04-20T17:50:34.963 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011619443 2024-04-20T17:50:34.963 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.963 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.963 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:34.963 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.489657+0000", 2024-04-20T17:50:34.963 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.033619624000000001 2024-04-20T17:50:34.963 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.963 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.963 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:50:34.963 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.489698+0000", 2024-04-20T17:50:34.963 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1499999999999999e-05 2024-04-20T17:50:34.963 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.963 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.964 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:50:34.964 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:47:52.489732+0000", 2024-04-20T17:50:34.964 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3853000000000003e-05 2024-04-20T17:50:34.964 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:34.964 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:34.964 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:34.964 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.964 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.964 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:282435 8.a 8:5477ec2a:::benchmark_data_smithi171_175757_object137321:head [delete] snapc 0=[] ondisk+write+known_if_redirected e881)", 2024-04-20T17:50:34.964 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:48:17.460732+0000", 2024-04-20T17:50:34.964 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 137.44985020199999, 2024-04-20T17:50:34.964 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.042690862000000003, 2024-04-20T17:50:34.964 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:50:34.965 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:50:34.965 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:34.965 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:50:34.965 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:50:34.965 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:50:34.965 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 282435 2024-04-20T17:50:34.965 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.965 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:50:34.965 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.965 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:50:34.965 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:48:17.460732+0000", 2024-04-20T17:50:34.965 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.965 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.965 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.966 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:50:34.966 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:48:17.460732+0000", 2024-04-20T17:50:34.966 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.966 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.966 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.966 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:50:34.966 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:48:17.460734+0000", 2024-04-20T17:50:34.966 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6449999999999999e-06 2024-04-20T17:50:34.966 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.966 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.966 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:50:34.966 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:48:17.460739+0000", 2024-04-20T17:50:34.966 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4720000000000003e-06 2024-04-20T17:50:34.966 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.967 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.967 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:50:34.967 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:48:17.460752+0000", 2024-04-20T17:50:34.967 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2880999999999999e-05 2024-04-20T17:50:34.967 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.967 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.967 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:50:34.967 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:48:17.460759+0000", 2024-04-20T17:50:34.967 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3880000000000002e-06 2024-04-20T17:50:34.967 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.967 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.967 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:50:34.967 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:48:17.463181+0000", 2024-04-20T17:50:34.967 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0024219689999999999 2024-04-20T17:50:34.968 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.968 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.968 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:50:34.968 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:48:17.463273+0000", 2024-04-20T17:50:34.968 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1970000000000002e-05 2024-04-20T17:50:34.968 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.968 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.968 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:50:34.968 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:48:17.463369+0000", 2024-04-20T17:50:34.968 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5427999999999995e-05 2024-04-20T17:50:34.968 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.968 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.968 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:50:34.969 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:48:17.470242+0000", 2024-04-20T17:50:34.969 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0068730170000000004 2024-04-20T17:50:34.969 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.969 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.969 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:34.969 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:48:17.503321+0000", 2024-04-20T17:50:34.969 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.033078925000000002 2024-04-20T17:50:34.969 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.969 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.969 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:50:34.969 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:48:17.503377+0000", 2024-04-20T17:50:34.969 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6858999999999999e-05 2024-04-20T17:50:34.969 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.969 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.970 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:50:34.970 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:48:17.503423+0000", 2024-04-20T17:50:34.970 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5308000000000001e-05 2024-04-20T17:50:34.970 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:34.970 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:34.970 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:34.970 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.970 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.970 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27460.0:282520 8.a 8:54c63e3c:::benchmark_data_smithi171_175757_object137406:head [delete] snapc 0=[] ondisk+write+known_if_redirected e881)", 2024-04-20T17:50:34.970 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:48:17.550386+0000", 2024-04-20T17:50:34.970 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 137.360195895, 2024-04-20T17:50:34.970 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043357685999999999, 2024-04-20T17:50:34.970 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:50:34.971 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:50:34.971 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:34.971 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:50:34.971 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27460", 2024-04-20T17:50:34.971 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/538840893", 2024-04-20T17:50:34.971 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 282520 2024-04-20T17:50:34.971 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.971 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:50:34.971 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.971 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:50:34.971 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:48:17.550386+0000", 2024-04-20T17:50:34.971 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.971 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.972 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.972 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:50:34.972 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:48:17.550386+0000", 2024-04-20T17:50:34.972 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.972 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.972 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.972 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:50:34.972 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:48:17.550388+0000", 2024-04-20T17:50:34.972 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.624e-06 2024-04-20T17:50:34.972 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.972 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.972 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:50:34.972 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:48:17.550392+0000", 2024-04-20T17:50:34.972 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4580000000000003e-06 2024-04-20T17:50:34.972 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.973 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.973 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:50:34.973 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:48:17.550400+0000", 2024-04-20T17:50:34.973 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8189999999999992e-06 2024-04-20T17:50:34.973 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.973 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.973 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:50:34.973 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:48:17.550405+0000", 2024-04-20T17:50:34.973 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2750000000000003e-06 2024-04-20T17:50:34.973 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.973 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.973 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:50:34.973 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:48:17.563181+0000", 2024-04-20T17:50:34.973 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.012775396 2024-04-20T17:50:34.974 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.974 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.974 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:50:34.974 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:48:17.563298+0000", 2024-04-20T17:50:34.974 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000116766 2024-04-20T17:50:34.974 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.974 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.974 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2", 2024-04-20T17:50:34.974 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:48:17.563402+0000", 2024-04-20T17:50:34.974 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000104746 2024-04-20T17:50:34.974 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.974 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.974 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:50:34.974 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:48:17.572587+0000", 2024-04-20T17:50:34.975 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0091848620000000002 2024-04-20T17:50:34.975 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.975 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.975 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:34.975 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:48:17.593643+0000", 2024-04-20T17:50:34.975 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021055870000000001 2024-04-20T17:50:34.975 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.975 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.975 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:50:34.975 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:48:17.593698+0000", 2024-04-20T17:50:34.975 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4477e-05 2024-04-20T17:50:34.975 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.975 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.976 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:50:34.976 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:48:17.593744+0000", 2024-04-20T17:50:34.976 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6393000000000001e-05 2024-04-20T17:50:34.976 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:34.976 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:34.976 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:34.976 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.976 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.976 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27772.0:146604 9.b 9:ddc6a8cf:::benchmark_data_smithi171_188319_object146603:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e927)", 2024-04-20T17:50:34.976 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:50:17.856182+0000", 2024-04-20T17:50:34.976 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 17.054399752999998, 2024-04-20T17:50:34.976 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.038983829999999997, 2024-04-20T17:50:34.976 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:50:34.976 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:50:34.977 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:34.977 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:50:34.977 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27772", 2024-04-20T17:50:34.977 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:50:34.977 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 146604 2024-04-20T17:50:34.977 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.977 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:50:34.977 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.977 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:50:34.977 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:17.856182+0000", 2024-04-20T17:50:34.977 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.977 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.977 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.977 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:50:34.978 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:17.856182+0000", 2024-04-20T17:50:34.978 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.978 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.978 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.978 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:50:34.978 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:17.856184+0000", 2024-04-20T17:50:34.978 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.282e-06 2024-04-20T17:50:34.978 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.978 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.978 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:50:34.978 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:17.856201+0000", 2024-04-20T17:50:34.978 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6988000000000001e-05 2024-04-20T17:50:34.978 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.978 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.979 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:50:34.979 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:17.856223+0000", 2024-04-20T17:50:34.979 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2629999999999998e-05 2024-04-20T17:50:34.979 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.979 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.979 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:50:34.979 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:17.856232+0000", 2024-04-20T17:50:34.979 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2400000000000007e-06 2024-04-20T17:50:34.979 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.979 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.979 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:50:34.979 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:17.867967+0000", 2024-04-20T17:50:34.979 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011735064 2024-04-20T17:50:34.979 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.980 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.980 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:50:34.980 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:17.868099+0000", 2024-04-20T17:50:34.980 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013221199999999999 2024-04-20T17:50:34.980 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.980 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.980 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 10", 2024-04-20T17:50:34.980 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:17.868257+0000", 2024-04-20T17:50:34.980 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015788299999999999 2024-04-20T17:50:34.980 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.980 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.980 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:50:34.980 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:17.877834+0000", 2024-04-20T17:50:34.981 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0095773920000000005 2024-04-20T17:50:34.981 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.981 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.981 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:34.981 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:17.895053+0000", 2024-04-20T17:50:34.981 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.017218675999999999 2024-04-20T17:50:34.981 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.981 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.981 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:50:34.981 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:17.895105+0000", 2024-04-20T17:50:34.981 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2503000000000002e-05 2024-04-20T17:50:34.981 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.981 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.981 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:50:34.981 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:17.895166+0000", 2024-04-20T17:50:34.982 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0959999999999999e-05 2024-04-20T17:50:34.982 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:34.982 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:34.982 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:34.982 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.982 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.982 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27772.0:146750 9.b 9:d0b18961:::benchmark_data_smithi171_188319_object146749:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e927)", 2024-04-20T17:50:34.982 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:50:17.998146+0000", 2024-04-20T17:50:34.982 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 16.912436158999999, 2024-04-20T17:50:34.982 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.039417169000000002, 2024-04-20T17:50:34.982 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:50:34.983 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:50:34.983 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:34.983 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:50:34.983 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27772", 2024-04-20T17:50:34.983 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:50:34.983 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 146750 2024-04-20T17:50:34.983 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.983 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:50:34.983 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.983 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:50:34.983 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:17.998146+0000", 2024-04-20T17:50:34.983 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.983 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.983 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.983 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:50:34.984 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:17.998146+0000", 2024-04-20T17:50:34.984 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.984 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.984 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.984 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:50:34.984 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:17.998147+0000", 2024-04-20T17:50:34.984 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2399999999999996e-07 2024-04-20T17:50:34.984 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.984 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.984 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:50:34.985 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:17.998169+0000", 2024-04-20T17:50:34.985 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2155999999999999e-05 2024-04-20T17:50:34.985 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.985 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.985 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:50:34.985 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:17.998177+0000", 2024-04-20T17:50:34.985 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9670000000000001e-06 2024-04-20T17:50:34.985 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.985 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.985 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:50:34.985 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:17.998183+0000", 2024-04-20T17:50:34.985 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0689999999999998e-06 2024-04-20T17:50:34.985 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.985 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.985 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:50:34.986 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.006799+0000", 2024-04-20T17:50:34.986 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0086163969999999996 2024-04-20T17:50:34.986 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.986 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.986 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:50:34.986 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.006969+0000", 2024-04-20T17:50:34.986 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00016920600000000001 2024-04-20T17:50:34.986 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.986 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.986 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 10", 2024-04-20T17:50:34.986 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.007163+0000", 2024-04-20T17:50:34.986 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00019432100000000001 2024-04-20T17:50:34.986 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.986 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.987 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:50:34.987 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.014534+0000", 2024-04-20T17:50:34.987 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.007371145 2024-04-20T17:50:34.987 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.987 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.987 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:34.987 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.037440+0000", 2024-04-20T17:50:34.987 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022905503000000001 2024-04-20T17:50:34.987 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.987 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.987 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:50:34.987 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.037502+0000", 2024-04-20T17:50:34.987 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1948999999999999e-05 2024-04-20T17:50:34.988 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.988 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.988 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:50:34.988 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.037563+0000", 2024-04-20T17:50:34.988 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1532000000000007e-05 2024-04-20T17:50:34.988 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:34.988 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:34.988 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:34.988 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.988 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.988 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27772.0:146834 9.b 9:d8bf10d8:::benchmark_data_smithi171_188319_object146833:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e927)", 2024-04-20T17:50:34.988 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:50:18.102217+0000", 2024-04-20T17:50:34.988 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 16.808364958999999, 2024-04-20T17:50:34.988 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.051130588999999997, 2024-04-20T17:50:34.988 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:50:34.989 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:50:34.989 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:34.989 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:50:34.989 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27772", 2024-04-20T17:50:34.989 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:50:34.989 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 146834 2024-04-20T17:50:34.989 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.989 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:50:34.989 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.989 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:50:34.989 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.102217+0000", 2024-04-20T17:50:34.989 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.989 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.989 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.990 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:50:34.990 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.102217+0000", 2024-04-20T17:50:34.990 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.990 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.990 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.990 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:50:34.990 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.102220+0000", 2024-04-20T17:50:34.990 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3179999999999998e-06 2024-04-20T17:50:34.990 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.990 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.990 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:50:34.990 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.102256+0000", 2024-04-20T17:50:34.990 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5809999999999998e-05 2024-04-20T17:50:34.990 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.991 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.991 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:50:34.991 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.102277+0000", 2024-04-20T17:50:34.991 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0208999999999999e-05 2024-04-20T17:50:34.991 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.991 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.991 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:50:34.991 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.102293+0000", 2024-04-20T17:50:34.991 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6019999999999999e-05 2024-04-20T17:50:34.991 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.991 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.991 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:50:34.991 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.103478+0000", 2024-04-20T17:50:34.991 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001185349 2024-04-20T17:50:34.992 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.992 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.992 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:50:34.992 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.103602+0000", 2024-04-20T17:50:34.992 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000124413 2024-04-20T17:50:34.992 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.992 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.992 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 10", 2024-04-20T17:50:34.992 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.103735+0000", 2024-04-20T17:50:34.992 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013248599999999999 2024-04-20T17:50:34.992 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.992 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.992 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:50:34.992 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.113996+0000", 2024-04-20T17:50:34.992 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010261007000000001 2024-04-20T17:50:34.993 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.993 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.993 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:34.993 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.153218+0000", 2024-04-20T17:50:34.993 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.039221898999999998 2024-04-20T17:50:34.993 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.993 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.993 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:50:34.993 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.153284+0000", 2024-04-20T17:50:34.993 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6731999999999997e-05 2024-04-20T17:50:34.993 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.993 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.993 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:50:34.993 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.153348+0000", 2024-04-20T17:50:34.994 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3346e-05 2024-04-20T17:50:34.994 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:34.994 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:34.994 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:34.994 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.994 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.994 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27772.0:146843 9.b 9:d661f1c1:::benchmark_data_smithi171_188319_object146842:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e927)", 2024-04-20T17:50:34.994 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:50:18.109047+0000", 2024-04-20T17:50:34.994 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 16.801535631, 2024-04-20T17:50:34.994 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.04459362, 2024-04-20T17:50:34.994 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:50:34.994 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:50:34.994 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:34.994 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:50:34.995 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27772", 2024-04-20T17:50:34.995 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:50:34.995 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 146843 2024-04-20T17:50:34.995 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.995 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:50:34.995 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.995 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:50:34.995 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.109047+0000", 2024-04-20T17:50:34.995 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.995 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.995 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.995 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:50:34.995 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.109047+0000", 2024-04-20T17:50:34.996 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:34.996 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.996 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.996 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:50:34.996 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.109049+0000", 2024-04-20T17:50:34.996 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2400000000000002e-06 2024-04-20T17:50:34.996 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.996 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.996 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:50:34.996 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.109073+0000", 2024-04-20T17:50:34.996 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4113000000000001e-05 2024-04-20T17:50:34.996 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.996 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.996 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:50:34.997 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.109086+0000", 2024-04-20T17:50:34.997 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2859e-05 2024-04-20T17:50:34.997 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.997 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.997 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:50:34.997 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.109094+0000", 2024-04-20T17:50:34.997 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.3329999999999997e-06 2024-04-20T17:50:34.997 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.997 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.997 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:50:34.997 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.109195+0000", 2024-04-20T17:50:34.997 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000100949 2024-04-20T17:50:34.997 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.997 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.998 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:50:34.998 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.109352+0000", 2024-04-20T17:50:34.998 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000157057 2024-04-20T17:50:34.998 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.998 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.998 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 10", 2024-04-20T17:50:34.998 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.109503+0000", 2024-04-20T17:50:34.998 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000150447 2024-04-20T17:50:34.998 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.998 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.998 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:50:34.998 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.116600+0000", 2024-04-20T17:50:34.998 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0070976770000000002 2024-04-20T17:50:34.998 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.999 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.999 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:34.999 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.153535+0000", 2024-04-20T17:50:34.999 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036934461000000002 2024-04-20T17:50:34.999 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.999 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.999 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:50:34.999 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.153588+0000", 2024-04-20T17:50:34.999 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3278e-05 2024-04-20T17:50:34.999 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:34.999 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:34.999 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:50:34.999 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.153640+0000", 2024-04-20T17:50:34.999 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2206000000000001e-05 2024-04-20T17:50:34.999 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:35.000 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:35.000 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:35.000 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.000 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.000 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27772.0:147499 9.b 9:dc3b8651:::benchmark_data_smithi171_188319_object147498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e927)", 2024-04-20T17:50:35.000 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:50:18.785282+0000", 2024-04-20T17:50:35.000 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 16.125300221, 2024-04-20T17:50:35.000 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.039008688, 2024-04-20T17:50:35.000 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:50:35.000 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:50:35.000 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:35.000 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:50:35.000 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27772", 2024-04-20T17:50:35.000 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:50:35.001 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 147499 2024-04-20T17:50:35.001 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.001 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:50:35.001 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.001 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:50:35.001 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.785282+0000", 2024-04-20T17:50:35.001 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:35.001 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.001 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.001 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:50:35.001 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.785282+0000", 2024-04-20T17:50:35.001 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:35.001 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.001 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.002 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:50:35.002 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.785284+0000", 2024-04-20T17:50:35.002 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.463e-06 2024-04-20T17:50:35.002 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.002 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.002 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:50:35.002 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.785309+0000", 2024-04-20T17:50:35.002 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4386999999999999e-05 2024-04-20T17:50:35.002 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.002 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.002 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:50:35.002 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.785320+0000", 2024-04-20T17:50:35.002 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1208e-05 2024-04-20T17:50:35.002 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.003 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.003 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:50:35.003 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.785327+0000", 2024-04-20T17:50:35.003 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.464e-06 2024-04-20T17:50:35.003 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.003 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.003 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:50:35.003 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.795735+0000", 2024-04-20T17:50:35.003 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010407348 2024-04-20T17:50:35.003 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.003 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.003 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:50:35.003 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.795841+0000", 2024-04-20T17:50:35.003 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000105763 2024-04-20T17:50:35.004 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.004 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.004 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 10", 2024-04-20T17:50:35.004 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.795967+0000", 2024-04-20T17:50:35.004 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012648299999999999 2024-04-20T17:50:35.004 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.004 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.004 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:50:35.004 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.824110+0000", 2024-04-20T17:50:35.004 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028143204000000002 2024-04-20T17:50:35.004 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.004 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.004 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:35.004 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.824160+0000", 2024-04-20T17:50:35.004 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9931e-05 2024-04-20T17:50:35.005 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.005 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.005 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:50:35.005 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.824230+0000", 2024-04-20T17:50:35.005 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9482999999999999e-05 2024-04-20T17:50:35.005 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.005 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.005 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:50:35.005 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:18.824291+0000", 2024-04-20T17:50:35.005 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0954e-05 2024-04-20T17:50:35.005 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:35.005 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:35.005 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:35.005 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.006 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.006 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27772.0:157043 9.b 9:d1c95f95:::benchmark_data_smithi171_188319_object157042:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e930)", 2024-04-20T17:50:35.006 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:50:27.960500+0000", 2024-04-20T17:50:35.006 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 6.9500824009999995, 2024-04-20T17:50:35.006 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.135209405, 2024-04-20T17:50:35.006 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:50:35.006 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:50:35.006 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:35.006 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:50:35.006 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27772", 2024-04-20T17:50:35.006 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:50:35.006 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 157043 2024-04-20T17:50:35.006 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.006 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:50:35.006 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.007 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:50:35.007 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.960500+0000", 2024-04-20T17:50:35.007 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:35.007 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.007 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.007 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:50:35.007 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.960500+0000", 2024-04-20T17:50:35.007 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:35.007 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.007 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.007 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:50:35.007 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.960502+0000", 2024-04-20T17:50:35.007 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6469999999999998e-06 2024-04-20T17:50:35.007 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.008 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.008 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:50:35.008 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.960526+0000", 2024-04-20T17:50:35.008 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3249999999999999e-05 2024-04-20T17:50:35.008 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.008 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.008 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:50:35.008 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.960541+0000", 2024-04-20T17:50:35.008 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5084e-05 2024-04-20T17:50:35.008 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.008 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.009 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:50:35.009 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.960550+0000", 2024-04-20T17:50:35.009 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0389999999999993e-06 2024-04-20T17:50:35.009 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.009 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.009 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:50:35.009 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.960604+0000", 2024-04-20T17:50:35.009 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3826999999999998e-05 2024-04-20T17:50:35.009 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.009 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.009 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:50:35.009 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.960696+0000", 2024-04-20T17:50:35.009 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2435999999999994e-05 2024-04-20T17:50:35.009 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.010 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.010 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 10", 2024-04-20T17:50:35.010 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.960780+0000", 2024-04-20T17:50:35.010 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4308000000000007e-05 2024-04-20T17:50:35.010 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.010 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.010 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:50:35.010 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.964764+0000", 2024-04-20T17:50:35.010 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0039833469999999999 2024-04-20T17:50:35.010 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.010 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.010 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:35.010 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:28.095600+0000", 2024-04-20T17:50:35.011 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.13083647300000001 2024-04-20T17:50:35.011 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.011 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.011 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:50:35.011 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:28.095660+0000", 2024-04-20T17:50:35.011 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9611000000000002e-05 2024-04-20T17:50:35.011 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.011 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.011 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:50:35.011 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:28.095709+0000", 2024-04-20T17:50:35.011 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9382999999999998e-05 2024-04-20T17:50:35.011 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:35.011 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:35.011 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:35.011 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.012 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.012 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27772.0:157058 9.b 9:d443df96:::benchmark_data_smithi171_188319_object157057:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e930)", 2024-04-20T17:50:35.012 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:50:27.976365+0000", 2024-04-20T17:50:35.012 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 6.9342172949999998, 2024-04-20T17:50:35.012 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.11946863000000001, 2024-04-20T17:50:35.012 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:50:35.012 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:50:35.012 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:35.012 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:50:35.012 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27772", 2024-04-20T17:50:35.012 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:50:35.012 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 157058 2024-04-20T17:50:35.012 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.012 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:50:35.013 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.013 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:50:35.013 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.976365+0000", 2024-04-20T17:50:35.013 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:35.013 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.013 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.013 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:50:35.013 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.976365+0000", 2024-04-20T17:50:35.013 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:35.013 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.013 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.013 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:50:35.013 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.976367+0000", 2024-04-20T17:50:35.013 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2680000000000001e-06 2024-04-20T17:50:35.014 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.014 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.014 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:50:35.014 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.976388+0000", 2024-04-20T17:50:35.014 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0800000000000001e-05 2024-04-20T17:50:35.014 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.014 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.014 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:50:35.014 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.976401+0000", 2024-04-20T17:50:35.014 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2897999999999999e-05 2024-04-20T17:50:35.014 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.014 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.014 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:50:35.014 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.976408+0000", 2024-04-20T17:50:35.015 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8510000000000001e-06 2024-04-20T17:50:35.015 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.015 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.015 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:50:35.015 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.976455+0000", 2024-04-20T17:50:35.015 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7580999999999997e-05 2024-04-20T17:50:35.015 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.015 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.015 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:50:35.015 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.976549+0000", 2024-04-20T17:50:35.015 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3382999999999997e-05 2024-04-20T17:50:35.015 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.015 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.015 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 10", 2024-04-20T17:50:35.016 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.976657+0000", 2024-04-20T17:50:35.016 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010834 2024-04-20T17:50:35.016 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.016 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.016 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:50:35.016 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.980395+0000", 2024-04-20T17:50:35.016 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0037380069999999998 2024-04-20T17:50:35.016 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.016 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.016 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:35.016 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:28.095749+0000", 2024-04-20T17:50:35.016 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.11535408900000001 2024-04-20T17:50:35.016 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.016 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.016 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:50:35.017 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:28.095798+0000", 2024-04-20T17:50:35.017 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9064999999999999e-05 2024-04-20T17:50:35.017 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.017 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.017 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:50:35.017 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:28.095833+0000", 2024-04-20T17:50:35.017 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5348000000000001e-05 2024-04-20T17:50:35.017 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:35.017 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:35.017 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:35.017 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.017 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.017 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27772.0:157066 9.b 9:de3c1a4e:::benchmark_data_smithi171_188319_object157065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e930)", 2024-04-20T17:50:35.017 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:50:27.985158+0000", 2024-04-20T17:50:35.018 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 6.9254241099999998, 2024-04-20T17:50:35.018 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.110797547, 2024-04-20T17:50:35.018 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:50:35.018 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:50:35.018 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:35.018 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:50:35.018 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27772", 2024-04-20T17:50:35.018 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:50:35.018 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 157066 2024-04-20T17:50:35.018 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.018 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:50:35.018 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.018 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:50:35.019 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.985158+0000", 2024-04-20T17:50:35.019 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:35.019 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.019 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.019 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:50:35.019 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.985158+0000", 2024-04-20T17:50:35.019 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:35.019 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.019 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.019 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:50:35.019 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.985160+0000", 2024-04-20T17:50:35.019 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2280000000000001e-06 2024-04-20T17:50:35.019 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.019 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.020 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:50:35.020 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.985180+0000", 2024-04-20T17:50:35.020 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0047999999999998e-05 2024-04-20T17:50:35.020 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.020 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.020 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:50:35.020 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.985190+0000", 2024-04-20T17:50:35.020 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.842e-06 2024-04-20T17:50:35.020 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.020 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.020 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:50:35.020 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.985196+0000", 2024-04-20T17:50:35.020 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2999999999999998e-06 2024-04-20T17:50:35.020 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.021 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.021 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:50:35.021 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.985247+0000", 2024-04-20T17:50:35.021 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0176000000000002e-05 2024-04-20T17:50:35.021 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.021 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.021 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:50:35.021 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.985335+0000", 2024-04-20T17:50:35.021 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8661999999999999e-05 2024-04-20T17:50:35.021 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.021 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.021 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 10", 2024-04-20T17:50:35.021 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.985424+0000", 2024-04-20T17:50:35.021 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9006999999999997e-05 2024-04-20T17:50:35.022 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.022 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.022 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:50:35.022 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.993233+0000", 2024-04-20T17:50:35.022 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0078087149999999999 2024-04-20T17:50:35.022 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.022 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.022 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:35.022 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:28.095873+0000", 2024-04-20T17:50:35.022 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.10263972 2024-04-20T17:50:35.022 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.022 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.022 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:50:35.023 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:28.095920+0000", 2024-04-20T17:50:35.023 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6798000000000002e-05 2024-04-20T17:50:35.023 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.023 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.023 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:50:35.023 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:28.095956+0000", 2024-04-20T17:50:35.023 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6050999999999998e-05 2024-04-20T17:50:35.023 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:35.023 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:35.023 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:35.023 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.023 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.023 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27772.0:157068 9.b 9:d3cfd8f6:::benchmark_data_smithi171_188319_object157067:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e930)", 2024-04-20T17:50:35.023 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:50:27.987989+0000", 2024-04-20T17:50:35.024 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 6.9225930380000005, 2024-04-20T17:50:35.024 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.108375557, 2024-04-20T17:50:35.024 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:50:35.024 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:50:35.024 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:35.024 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:50:35.024 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27772", 2024-04-20T17:50:35.024 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:50:35.024 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 157068 2024-04-20T17:50:35.024 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.024 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:50:35.024 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.024 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:50:35.024 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.987989+0000", 2024-04-20T17:50:35.025 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:35.025 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.025 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.025 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:50:35.025 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.987989+0000", 2024-04-20T17:50:35.025 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:35.025 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.025 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.025 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:50:35.025 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.987991+0000", 2024-04-20T17:50:35.025 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4529999999999999e-06 2024-04-20T17:50:35.025 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.025 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.025 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:50:35.026 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.988011+0000", 2024-04-20T17:50:35.026 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0551e-05 2024-04-20T17:50:35.026 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.026 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.026 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:50:35.026 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.988021+0000", 2024-04-20T17:50:35.026 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9119999999999993e-06 2024-04-20T17:50:35.026 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.026 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.026 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:50:35.026 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.988027+0000", 2024-04-20T17:50:35.026 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7389999999999996e-06 2024-04-20T17:50:35.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:35.027 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.027 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.028 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:50:35.028 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.988840+0000", 2024-04-20T17:50:35.028 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00081303699999999996 2024-04-20T17:50:35.028 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.028 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.028 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:50:35.028 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.988964+0000", 2024-04-20T17:50:35.028 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012433799999999999 2024-04-20T17:50:35.028 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.028 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.028 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 10", 2024-04-20T17:50:35.028 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.989088+0000", 2024-04-20T17:50:35.028 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012368000000000001 2024-04-20T17:50:35.029 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.029 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.029 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:50:35.029 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.996631+0000", 2024-04-20T17:50:35.029 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0075429260000000001 2024-04-20T17:50:35.029 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.029 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.029 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:35.029 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:28.096268+0000", 2024-04-20T17:50:35.029 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.09963677 2024-04-20T17:50:35.029 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.029 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.029 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:50:35.029 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:28.096324+0000", 2024-04-20T17:50:35.029 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6113999999999998e-05 2024-04-20T17:50:35.030 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.030 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.030 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:50:35.030 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:28.096365+0000", 2024-04-20T17:50:35.030 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1037e-05 2024-04-20T17:50:35.030 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:35.030 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:35.030 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:35.030 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.030 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.030 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27772.0:157072 9.b 9:d51cc9f7:::benchmark_data_smithi171_188319_object157071:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e930)", 2024-04-20T17:50:35.030 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:50:27.989577+0000", 2024-04-20T17:50:35.030 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 6.9210051650000004, 2024-04-20T17:50:35.031 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.113149124, 2024-04-20T17:50:35.031 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:50:35.031 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:50:35.031 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:50:35.031 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:50:35.031 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27772", 2024-04-20T17:50:35.031 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:50:35.031 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 157072 2024-04-20T17:50:35.031 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.031 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:50:35.031 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.031 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:50:35.031 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.989577+0000", 2024-04-20T17:50:35.031 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:35.031 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.032 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.032 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:50:35.032 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.989577+0000", 2024-04-20T17:50:35.032 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:35.032 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.032 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.032 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:50:35.032 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.989578+0000", 2024-04-20T17:50:35.032 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.404e-06 2024-04-20T17:50:35.032 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.032 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.032 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:50:35.032 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.989602+0000", 2024-04-20T17:50:35.032 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3394e-05 2024-04-20T17:50:35.032 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.033 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.033 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:50:35.033 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.989612+0000", 2024-04-20T17:50:35.033 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0339999999999999e-05 2024-04-20T17:50:35.033 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.033 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.033 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:50:35.033 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.989619+0000", 2024-04-20T17:50:35.033 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9759999999999998e-06 2024-04-20T17:50:35.033 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.033 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.033 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:50:35.033 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.993258+0000", 2024-04-20T17:50:35.033 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0036387960000000001 2024-04-20T17:50:35.034 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.034 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.034 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:50:35.034 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.993396+0000", 2024-04-20T17:50:35.034 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013814599999999999 2024-04-20T17:50:35.034 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.034 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.034 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 10", 2024-04-20T17:50:35.034 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.993525+0000", 2024-04-20T17:50:35.034 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012866900000000001 2024-04-20T17:50:35.034 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.034 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.034 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-04-20T17:50:35.034 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:27.996817+0000", 2024-04-20T17:50:35.034 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0032918679999999999 2024-04-20T17:50:35.035 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.035 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.035 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:35.035 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:28.102597+0000", 2024-04-20T17:50:35.035 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.105780006 2024-04-20T17:50:35.035 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.035 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.035 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-04-20T17:50:35.035 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:28.102663+0000", 2024-04-20T17:50:35.035 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6239000000000001e-05 2024-04-20T17:50:35.035 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:35.035 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:35.035 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-04-20T17:50:35.035 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:28.102726+0000", 2024-04-20T17:50:35.035 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3286000000000004e-05 2024-04-20T17:50:35.036 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:35.036 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:35.036 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:35.036 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:35.036 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:35.036 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:50:35.036 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:50:35.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:50:35.037 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:50:35.138 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:50:35.185 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:35.186 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:34 smithi005 ceph-mon[39562]: 9.d scrub starts 2024-04-20T17:50:35.186 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:34 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/840483326' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:50:35.186 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:34 smithi005 ceph-mon[39562]: from='client.27862 v1:172.21.15.5:0/840483326' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:50:35.186 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:34 smithi005 ceph-mon[39562]: pgmap v2455: 27 pgs: 27 active+clean; 9.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 888 op/s 2024-04-20T17:50:35.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:34 smithi106 ceph-mon[44423]: 9.d scrub starts 2024-04-20T17:50:35.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:34 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/840483326' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:50:35.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:34 smithi106 ceph-mon[44423]: from='client.27862 v1:172.21.15.5:0/840483326' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:50:35.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:34 smithi106 ceph-mon[44423]: pgmap v2455: 27 pgs: 27 active+clean; 9.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 888 op/s 2024-04-20T17:50:35.314 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:34 smithi171 ceph-mon[44189]: 9.d scrub starts 2024-04-20T17:50:35.314 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:34 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/840483326' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:50:35.314 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:34 smithi171 ceph-mon[44189]: from='client.27862 v1:172.21.15.5:0/840483326' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:50:35.314 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:34 smithi171 ceph-mon[44189]: pgmap v2455: 27 pgs: 27 active+clean; 9.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 888 op/s 2024-04-20T17:50:35.405 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:50:35.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:35.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:50:35.511 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:50:35.539 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:35.567 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:50:35.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:35.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:50:35.675 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:50:35.689 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-20T17:50:35.728 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:35.769 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:50:35.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:35.775 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:50:35.876 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:50:35.880 INFO:tasks.radosbench.radosbench.2.smithi171.stdout: 89 16 165759 165743 116.363 68.8125 0.0292077 0.00859173 2024-04-20T17:50:35.921 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:35.957 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:50:35.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:35.960 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:50:35.997 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:35 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:50:35.997 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:35 smithi106 ceph-mon[44423]: from='client.27862 v1:172.21.15.5:0/840483326' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:50:35.997 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:35 smithi106 ceph-mon[44423]: osdmap e932: 12 total, 12 up, 12 in 2024-04-20T17:50:35.997 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:35 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:50:35.997 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:35 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:50:36.061 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:50:36.101 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:36.127 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:50:36.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:36.135 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:50:36.159 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:50:36.236 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:50:36.267 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:36.292 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:50:36.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:36.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:50:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:35 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:50:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:35 smithi171 ceph-mon[44189]: from='client.27862 v1:172.21.15.5:0/840483326' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:50:36.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:35 smithi171 ceph-mon[44189]: osdmap e932: 12 total, 12 up, 12 in 2024-04-20T17:50:36.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:35 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:50:36.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:35 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:50:36.397 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:50:36.405 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:35 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:50:36.405 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:35 smithi005 ceph-mon[39562]: from='client.27862 v1:172.21.15.5:0/840483326' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:50:36.405 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:35 smithi005 ceph-mon[39562]: osdmap e932: 12 total, 12 up, 12 in 2024-04-20T17:50:36.405 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:35 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:50:36.405 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:35 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:50:36.454 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:36.486 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:50:36.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:36.495 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:50:36.496 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:50:36.524 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:50:36.547 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:50:36.569 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:50:36.596 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:50:36.598 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:50:36.620 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:50:36.624 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:36.646 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:50:36.650 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:50:36.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:36.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:50:36.679 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:50:36.712 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:50:36.748 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:50:36.754 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:50:36.782 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:50:36.792 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:36.817 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:50:36.823 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:50:36.823 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:50:36.823 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:50:36.823 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:50:36.823 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:50:36.824 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:50:36.824 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:50:36.824 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:50:36.824 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:50:36.824 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:50:36.824 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:50:36.824 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:50:36.827 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:50:36.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:36.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:50:36.880 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Total time run: 90.0119 2024-04-20T17:50:36.880 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Total writes made: 166793 2024-04-20T17:50:36.881 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Write size: 65536 2024-04-20T17:50:36.881 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Object size: 65536 2024-04-20T17:50:36.881 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Bandwidth (MB/sec): 115.813 2024-04-20T17:50:36.881 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Stddev Bandwidth: 57.1201 2024-04-20T17:50:36.881 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Max bandwidth (MB/sec): 351 2024-04-20T17:50:36.881 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Min bandwidth (MB/sec): 55.8125 2024-04-20T17:50:36.881 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Average IOPS: 1853 2024-04-20T17:50:36.881 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Stddev IOPS: 913.921 2024-04-20T17:50:36.881 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Max IOPS: 5616 2024-04-20T17:50:36.881 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Min IOPS: 893 2024-04-20T17:50:36.881 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Average Latency(s): 0.00863394 2024-04-20T17:50:36.881 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Stddev Latency(s): 0.0071247 2024-04-20T17:50:36.881 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Max latency(s): 0.200169 2024-04-20T17:50:36.881 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Min latency(s): 0.00101366 2024-04-20T17:50:36.890 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Cleaning up (deleting benchmark objects) 2024-04-20T17:50:36.934 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:50:36.982 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:37.006 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:37 smithi106 ceph-mon[44423]: osdmap e933: 12 total, 12 up, 12 in 2024-04-20T17:50:37.006 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:37 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1359996021' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:50:37.006 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:37.006 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:37.006 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:37.006 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:37 smithi106 ceph-mon[44423]: pgmap v2458: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 10 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-04-20T17:50:37.007 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:37.072 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:50:37.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:37.077 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:50:37.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:37 smithi171 ceph-mon[44189]: osdmap e933: 12 total, 12 up, 12 in 2024-04-20T17:50:37.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:37 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1359996021' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:50:37.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:37.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:37.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:37.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:37 smithi171 ceph-mon[44189]: pgmap v2458: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 10 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-04-20T17:50:37.077 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:37.178 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:50:37.201 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:37 smithi005 ceph-mon[39562]: osdmap e933: 12 total, 12 up, 12 in 2024-04-20T17:50:37.201 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:37 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1359996021' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:50:37.201 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:37.201 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:37.201 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:37.201 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:37 smithi005 ceph-mon[39562]: pgmap v2458: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 10 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-04-20T17:50:37.201 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:37.208 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:37.241 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:50:37.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:37.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:50:37.346 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:50:37.387 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:37.647 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:50:37.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:37.655 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:50:37.756 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:50:37.794 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:37.963 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:50:37.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:37.967 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:50:38.068 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:50:38.101 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:38.191 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:50:38.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:38.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:50:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:38 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:38.296 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:50:38.328 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:38.328 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:38.328 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:38.328 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:38.328 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:38.329 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:38.329 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:38.329 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:38 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:38.331 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:38 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:38.361 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:50:38.362 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:50:38.392 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:50:38.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:38.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:50:38.496 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:50:38.537 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:38.568 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:50:38.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:38.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:50:38.675 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:50:38.734 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:39.024 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:39 smithi171 ceph-mon[44189]: pgmap v2459: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 10 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 1.02k op/s 2024-04-20T17:50:39.102 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:50:39.105 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-20T17:50:39.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:39.110 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:50:39.147 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:50:39.149 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:50:39.211 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:50:39.256 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:39.406 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:50:39.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:39.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:50:39.411 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:39 smithi005 ceph-mon[39562]: pgmap v2459: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 10 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 1.02k op/s 2024-04-20T17:50:39.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:39 smithi106 ceph-mon[44423]: pgmap v2459: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 10 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 1.02k op/s 2024-04-20T17:50:39.512 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:50:39.524 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:50:39.544 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:39.577 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:50:39.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:39.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:50:39.689 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:50:39.690 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:50:39.742 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:39.877 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:50:39.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:39.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:50:39.906 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":2459,"stamp":"2024-04-20T17:50:38.688046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":10833101344,"num_objects":165293,"num_object_clones":0,"num_object_copies":330586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":166061,"num_write_kb":10603264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162268,"ondisk_log_size":162268,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":28280964,"kb_used_data":21724420,"kb_used_omap":404,"kb_used_meta":6556011,"kb_avail":1096660860,"statfs":{"total":1151940427776,"available":1122980720640,"internally_reserved":0,"allocated":22245806080,"data_stored":22240979054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":414014,"internal_metadata":6713355970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,3,6,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":467795968,"num_objects":7138,"num_object_clones":0,"num_object_copies":14276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7908,"num_write_kb":481472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.746358"},"pg_stats":[{"pgid":"9.7","version":"932'10198","reported_seq":10387,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-20T17:50:35.086811+0000","last_change":"2024-04-20T17:50:27.211659+0000","last_active":"2024-04-20T17:50:35.086811+0000","last_peered":"2024-04-20T17:50:35.086811+0000","last_clean":"2024-04-20T17:50:35.086811+0000","last_became_active":"2024-04-20T17:48:54.814508+0000","last_became_peered":"2024-04-20T17:48:54.814508+0000","last_unstale":"2024-04-20T17:50:35.086811+0000","last_undegraded":"2024-04-20T17:50:35.086811+0000","last_fullsized":"2024-04-20T17:50:35.086811+0000","mapping_epoch":894,"log_start":"900'100","ondisk_log_start":"900'100","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'9665","last_scrub_stamp":"2024-04-20T17:50:27.211421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:27.211421+0000","objects_scrubbed":9588,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:56.034157+0000","scrub_duration":2310,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":668336128,"num_objects":10198,"num_object_clones":0,"num_object_copies":20396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10198,"num_write_kb":652672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"932'10292","reported_seq":10480,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-20T17:50:35.100968+0000","last_change":"2024-04-20T17:50:25.096410+0000","last_active":"2024-04-20T17:50:35.100968+0000","last_peered":"2024-04-20T17:50:35.100968+0000","last_clean":"2024-04-20T17:50:35.100968+0000","last_became_active":"2024-04-20T17:48:54.818033+0000","last_became_peered":"2024-04-20T17:48:54.818033+0000","last_unstale":"2024-04-20T17:50:35.100968+0000","last_undegraded":"2024-04-20T17:50:35.100968+0000","last_fullsized":"2024-04-20T17:50:35.100968+0000","mapping_epoch":894,"log_start":"900'200","ondisk_log_start":"900'200","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"929'9608","last_scrub_stamp":"2024-04-20T17:50:25.096273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:25.096273+0000","objects_scrubbed":9524,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:47.683348+0000","scrub_duration":2229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":674496512,"num_objects":10292,"num_object_clones":0,"num_object_copies":20584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10292,"num_write_kb":658688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.5","version":"933'10338","reported_seq":10529,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-04-20T17:50:36.071792+0000","last_change":"2024-04-20T17:50:23.865871+0000","last_active":"2024-04-20T17:50:36.071792+0000","last_peered":"2024-04-20T17:50:36.071792+0000","last_clean":"2024-04-20T17:50:36.071792+0000","last_became_active":"2024-04-20T17:48:54.817749+0000","last_became_peered":"2024-04-20T17:48:54.817749+0000","last_unstale":"2024-04-20T17:50:36.071792+0000","last_undegraded":"2024-04-20T17:50:36.071792+0000","last_fullsized":"2024-04-20T17:50:36.071792+0000","mapping_epoch":894,"log_start":"900'300","ondisk_log_start":"900'300","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9546","last_scrub_stamp":"2024-04-20T17:50:23.865708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:23.865708+0000","objects_scrubbed":9473,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:24.406565+0000","scrub_duration":1985,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":677511168,"num_objects":10338,"num_object_clones":0,"num_object_copies":20676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10338,"num_write_kb":661632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"932'10290","reported_seq":10475,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-20T17:50:35.844772+0000","last_change":"2024-04-20T17:50:23.032137+0000","last_active":"2024-04-20T17:50:35.844772+0000","last_peered":"2024-04-20T17:50:35.844772+0000","last_clean":"2024-04-20T17:50:35.844772+0000","last_became_active":"2024-04-20T17:48:55.061199+0000","last_became_peered":"2024-04-20T17:48:55.061199+0000","last_unstale":"2024-04-20T17:50:35.844772+0000","last_undegraded":"2024-04-20T17:50:35.844772+0000","last_fullsized":"2024-04-20T17:50:35.844772+0000","mapping_epoch":894,"log_start":"900'200","ondisk_log_start":"900'200","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9404","last_scrub_stamp":"2024-04-20T17:50:23.031924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:23.031924+0000","objects_scrubbed":9366,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:30.444210+0000","scrub_duration":1573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":674365440,"num_objects":10290,"num_object_clones":0,"num_object_copies":20580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10290,"num_write_kb":658560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"933'10306","reported_seq":10497,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-04-20T17:50:37.131888+0000","last_change":"2024-04-20T17:50:27.118878+0000","last_active":"2024-04-20T17:50:37.131888+0000","last_peered":"2024-04-20T17:50:37.131888+0000","last_clean":"2024-04-20T17:50:37.131888+0000","last_became_active":"2024-04-20T17:48:54.813808+0000","last_became_peered":"2024-04-20T17:48:54.813808+0000","last_unstale":"2024-04-20T17:50:37.131888+0000","last_undegraded":"2024-04-20T17:50:37.131888+0000","last_fullsized":"2024-04-20T17:50:37.131888+0000","mapping_epoch":894,"log_start":"900'300","ondisk_log_start":"900'300","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'9622","last_scrub_stamp":"2024-04-20T17:50:27.118695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:27.118695+0000","objects_scrubbed":9558,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:52.229368+0000","scrub_duration":2200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":673054720,"num_objects":10270,"num_object_clones":0,"num_object_copies":20540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10306,"num_write_kb":658432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.e","version":"931'10330","reported_seq":10413,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-20T17:50:34.796290+0000","last_change":"2024-04-20T17:48:54.815383+0000","last_active":"2024-04-20T17:50:34.796290+0000","last_peered":"2024-04-20T17:50:34.796290+0000","last_clean":"2024-04-20T17:50:34.796290+0000","last_became_active":"2024-04-20T17:48:54.815159+0000","last_became_peered":"2024-04-20T17:48:54.815159+0000","last_unstale":"2024-04-20T17:50:34.796290+0000","last_undegraded":"2024-04-20T17:50:34.796290+0000","last_fullsized":"2024-04-20T17:50:34.796290+0000","mapping_epoch":894,"log_start":"900'300","ondisk_log_start":"900'300","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:48:53.794927+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":676986880,"num_objects":10330,"num_object_clones":0,"num_object_copies":20660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10330,"num_write_kb":661120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"15'137","reported_seq":907,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-20T17:50:35.013980+0000","last_change":"2024-04-20T17:49:26.955123+0000","last_active":"2024-04-20T17:50:35.013980+0000","last_peered":"2024-04-20T17:50:35.013980+0000","last_clean":"2024-04-20T17:50:35.013980+0000","last_became_active":"2024-04-20T17:45:32.670941+0000","last_became_peered":"2024-04-20T17:45:32.670941+0000","last_unstale":"2024-04-20T17:50:35.013980+0000","last_undegraded":"2024-04-20T17:50:35.013980+0000","last_fullsized":"2024-04-20T17:50:35.013980+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:49:26.954998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:49:26.954998+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:50:44.799838+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"932'10352","reported_seq":10539,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-20T17:50:35.587632+0000","last_change":"2024-04-20T17:50:23.641303+0000","last_active":"2024-04-20T17:50:35.587632+0000","last_peered":"2024-04-20T17:50:35.587632+0000","last_clean":"2024-04-20T17:50:35.587632+0000","last_became_active":"2024-04-20T17:48:55.061161+0000","last_became_peered":"2024-04-20T17:48:55.061161+0000","last_unstale":"2024-04-20T17:50:35.587632+0000","last_undegraded":"2024-04-20T17:50:35.587632+0000","last_fullsized":"2024-04-20T17:50:35.587632+0000","mapping_epoch":894,"log_start":"900'300","ondisk_log_start":"900'300","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9495","last_scrub_stamp":"2024-04-20T17:50:23.641164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:23.641164+0000","objects_scrubbed":9459,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:29.146891+0000","scrub_duration":1363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":678428672,"num_objects":10352,"num_object_clones":0,"num_object_copies":20704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10352,"num_write_kb":662528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"15'137","reported_seq":915,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-04-20T17:50:36.007507+0000","last_change":"2024-04-20T17:50:25.724423+0000","last_active":"2024-04-20T17:50:36.007507+0000","last_peered":"2024-04-20T17:50:36.007507+0000","last_clean":"2024-04-20T17:50:36.007507+0000","last_became_active":"2024-04-20T17:45:29.817528+0000","last_became_peered":"2024-04-20T17:45:29.817528+0000","last_unstale":"2024-04-20T17:50:36.007507+0000","last_undegraded":"2024-04-20T17:50:36.007507+0000","last_fullsized":"2024-04-20T17:50:36.007507+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:25.724296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:25.724296+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:37.439709+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"933'10625","reported_seq":10821,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-04-20T17:50:38.529648+0000","last_change":"2024-04-20T17:50:29.927851+0000","last_active":"2024-04-20T17:50:38.529648+0000","last_peered":"2024-04-20T17:50:38.529648+0000","last_clean":"2024-04-20T17:50:38.529648+0000","last_became_active":"2024-04-20T17:48:54.817445+0000","last_became_peered":"2024-04-20T17:48:54.817445+0000","last_unstale":"2024-04-20T17:50:38.529648+0000","last_undegraded":"2024-04-20T17:50:38.529648+0000","last_fullsized":"2024-04-20T17:50:38.529648+0000","mapping_epoch":894,"log_start":"900'600","ondisk_log_start":"900'600","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'10062","last_scrub_stamp":"2024-04-20T17:50:29.927762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:29.927762+0000","objects_scrubbed":9978,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:58.752509+0000","scrub_duration":2071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":681115648,"num_objects":10393,"num_object_clones":0,"num_object_copies":20786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10625,"num_write_kb":672576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"15'137","reported_seq":941,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-04-20T17:50:36.012502+0000","last_change":"2024-04-20T17:50:20.138119+0000","last_active":"2024-04-20T17:50:36.012502+0000","last_peered":"2024-04-20T17:50:36.012502+0000","last_clean":"2024-04-20T17:50:36.012502+0000","last_became_active":"2024-04-20T17:45:29.827577+0000","last_became_peered":"2024-04-20T17:45:29.827577+0000","last_unstale":"2024-04-20T17:50:36.012502+0000","last_undegraded":"2024-04-20T17:50:36.012502+0000","last_fullsized":"2024-04-20T17:50:36.012502+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:20.137983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:20.137983+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:30.668575+0000","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"931'10257","reported_seq":10394,"reported_epoch":931,"state":"active+clean+scrubbing","last_fresh":"2024-04-20T17:50:34.799162+0000","last_change":"2024-04-20T17:50:34.011823+0000","last_active":"2024-04-20T17:50:34.799162+0000","last_peered":"2024-04-20T17:50:34.799162+0000","last_clean":"2024-04-20T17:50:34.799162+0000","last_became_active":"2024-04-20T17:48:55.061208+0000","last_became_peered":"2024-04-20T17:48:55.061208+0000","last_unstale":"2024-04-20T17:50:34.799162+0000","last_undegraded":"2024-04-20T17:50:34.799162+0000","last_fullsized":"2024-04-20T17:50:34.799162+0000","mapping_epoch":894,"log_start":"900'200","ondisk_log_start":"900'200","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:48:53.794927+0000","objects_scrubbed":5024,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672202752,"num_objects":10257,"num_object_clones":0,"num_object_copies":20514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10257,"num_write_kb":656448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"15'137","reported_seq":907,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-20T17:50:35.013913+0000","last_change":"2024-04-20T17:50:18.122040+0000","last_active":"2024-04-20T17:50:35.013913+0000","last_peered":"2024-04-20T17:50:35.013913+0000","last_clean":"2024-04-20T17:50:35.013913+0000","last_became_active":"2024-04-20T17:45:32.666523+0000","last_became_peered":"2024-04-20T17:45:32.666523+0000","last_unstale":"2024-04-20T17:50:35.013913+0000","last_undegraded":"2024-04-20T17:50:35.013913+0000","last_fullsized":"2024-04-20T17:50:35.013913+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:18.121912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:18.121912+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:25.993620+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.a","version":"932'10410","reported_seq":10601,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-20T17:50:35.550742+0000","last_change":"2024-04-20T17:50:22.180642+0000","last_active":"2024-04-20T17:50:35.550742+0000","last_peered":"2024-04-20T17:50:35.550742+0000","last_clean":"2024-04-20T17:50:35.550742+0000","last_became_active":"2024-04-20T17:48:55.061180+0000","last_became_peered":"2024-04-20T17:48:55.061180+0000","last_unstale":"2024-04-20T17:50:35.550742+0000","last_undegraded":"2024-04-20T17:50:35.550742+0000","last_fullsized":"2024-04-20T17:50:35.550742+0000","mapping_epoch":894,"log_start":"900'400","ondisk_log_start":"900'400","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9574","last_scrub_stamp":"2024-04-20T17:50:22.180501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:22.180501+0000","objects_scrubbed":9507,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:51.128747+0000","scrub_duration":1824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":682229760,"num_objects":10410,"num_object_clones":0,"num_object_copies":20820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10410,"num_write_kb":666240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"15'137","reported_seq":907,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-20T17:50:31.132969+0000","last_change":"2024-04-20T17:50:31.133012+0000","last_active":"2024-04-20T17:50:31.132969+0000","last_peered":"2024-04-20T17:50:31.132969+0000","last_clean":"2024-04-20T17:50:31.132969+0000","last_became_active":"2024-04-20T17:46:40.831961+0000","last_became_peered":"2024-04-20T17:46:40.831961+0000","last_unstale":"2024-04-20T17:50:31.132969+0000","last_undegraded":"2024-04-20T17:50:31.132969+0000","last_fullsized":"2024-04-20T17:50:31.132969+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:31.132879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:31.132879+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:31.822631+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"931'10152","reported_seq":10334,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-20T17:50:33.950472+0000","last_change":"2024-04-20T17:50:19.657299+0000","last_active":"2024-04-20T17:50:33.950472+0000","last_peered":"2024-04-20T17:50:33.950472+0000","last_clean":"2024-04-20T17:50:33.950472+0000","last_became_active":"2024-04-20T17:48:54.816644+0000","last_became_peered":"2024-04-20T17:48:54.816644+0000","last_unstale":"2024-04-20T17:50:33.950472+0000","last_undegraded":"2024-04-20T17:50:33.950472+0000","last_fullsized":"2024-04-20T17:50:33.950472+0000","mapping_epoch":894,"log_start":"900'100","ondisk_log_start":"900'100","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"927'9204","last_scrub_stamp":"2024-04-20T17:50:19.657194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:19.657194+0000","objects_scrubbed":9124,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:25.188599+0000","scrub_duration":2257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":665321472,"num_objects":10152,"num_object_clones":0,"num_object_copies":20304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10152,"num_write_kb":649728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"15'137","reported_seq":702,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-04-20T17:50:36.004826+0000","last_change":"2024-04-20T17:49:47.812349+0000","last_active":"2024-04-20T17:50:36.004826+0000","last_peered":"2024-04-20T17:50:36.004826+0000","last_clean":"2024-04-20T17:50:36.004826+0000","last_became_active":"2024-04-20T17:49:46.824559+0000","last_became_peered":"2024-04-20T17:49:46.824559+0000","last_unstale":"2024-04-20T17:50:36.004826+0000","last_undegraded":"2024-04-20T17:50:36.004826+0000","last_fullsized":"2024-04-20T17:50:36.004826+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:49:47.812234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:49:47.812234+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:50:49.678662+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"932'10508","reported_seq":10700,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-20T17:50:35.110232+0000","last_change":"2024-04-20T17:50:28.283117+0000","last_active":"2024-04-20T17:50:35.110232+0000","last_peered":"2024-04-20T17:50:35.110232+0000","last_clean":"2024-04-20T17:50:35.110232+0000","last_became_active":"2024-04-20T17:48:54.813199+0000","last_became_peered":"2024-04-20T17:48:54.813199+0000","last_unstale":"2024-04-20T17:50:35.110232+0000","last_undegraded":"2024-04-20T17:50:35.110232+0000","last_fullsized":"2024-04-20T17:50:35.110232+0000","mapping_epoch":894,"log_start":"900'500","ondisk_log_start":"900'500","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'10025","last_scrub_stamp":"2024-04-20T17:50:28.282957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:28.282957+0000","objects_scrubbed":9963,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:48.575002+0000","scrub_duration":2339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":688652288,"num_objects":10508,"num_object_clones":0,"num_object_copies":21016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10508,"num_write_kb":672512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"15'137","reported_seq":907,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-20T17:50:35.014030+0000","last_change":"2024-04-20T17:49:29.040029+0000","last_active":"2024-04-20T17:50:35.014030+0000","last_peered":"2024-04-20T17:50:35.014030+0000","last_clean":"2024-04-20T17:50:35.014030+0000","last_became_active":"2024-04-20T17:45:32.782706+0000","last_became_peered":"2024-04-20T17:45:32.782706+0000","last_unstale":"2024-04-20T17:50:35.014030+0000","last_undegraded":"2024-04-20T17:50:35.014030+0000","last_fullsized":"2024-04-20T17:50:35.014030+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:49:29.039907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:49:29.039907+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:50:49.985844+0000","scrub_duration":161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.2","version":"933'10515","reported_seq":10703,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-04-20T17:50:38.564402+0000","last_change":"2024-04-20T17:50:22.440005+0000","last_active":"2024-04-20T17:50:38.564402+0000","last_peered":"2024-04-20T17:50:38.564402+0000","last_clean":"2024-04-20T17:50:38.564402+0000","last_became_active":"2024-04-20T17:48:55.060965+0000","last_became_peered":"2024-04-20T17:48:55.060965+0000","last_unstale":"2024-04-20T17:50:38.564402+0000","last_undegraded":"2024-04-20T17:50:38.564402+0000","last_fullsized":"2024-04-20T17:50:38.564402+0000","mapping_epoch":894,"log_start":"900'500","ondisk_log_start":"900'500","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9408","last_scrub_stamp":"2024-04-20T17:50:22.439841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:22.439841+0000","objects_scrubbed":9355,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:23.199554+0000","scrub_duration":1641,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672727040,"num_objects":10265,"num_object_clones":0,"num_object_copies":20530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10515,"num_write_kb":664960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"15'137","reported_seq":895,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-04-20T17:50:36.011485+0000","last_change":"2024-04-20T17:49:48.210622+0000","last_active":"2024-04-20T17:50:36.011485+0000","last_peered":"2024-04-20T17:50:36.011485+0000","last_clean":"2024-04-20T17:50:36.011485+0000","last_became_active":"2024-04-20T17:45:32.669451+0000","last_became_peered":"2024-04-20T17:45:32.669451+0000","last_unstale":"2024-04-20T17:50:36.011485+0000","last_undegraded":"2024-04-20T17:50:36.011485+0000","last_fullsized":"2024-04-20T17:50:36.011485+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:49:48.210506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:49:48.210506+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:05.387619+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"933'10368","reported_seq":10562,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-04-20T17:50:36.707852+0000","last_change":"2024-04-20T17:50:30.239298+0000","last_active":"2024-04-20T17:50:36.707852+0000","last_peered":"2024-04-20T17:50:36.707852+0000","last_clean":"2024-04-20T17:50:36.707852+0000","last_became_active":"2024-04-20T17:48:55.061200+0000","last_became_peered":"2024-04-20T17:48:55.061200+0000","last_unstale":"2024-04-20T17:50:36.707852+0000","last_undegraded":"2024-04-20T17:50:36.707852+0000","last_fullsized":"2024-04-20T17:50:36.707852+0000","mapping_epoch":894,"log_start":"900'300","ondisk_log_start":"900'300","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'9940","last_scrub_stamp":"2024-04-20T17:50:30.239131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:30.239131+0000","objects_scrubbed":9887,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:36.933027+0000","scrub_duration":1672,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":679477248,"num_objects":10368,"num_object_clones":0,"num_object_copies":20736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10368,"num_write_kb":663552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"15'137","reported_seq":910,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-04-20T17:50:36.360940+0000","last_change":"2024-04-20T17:50:23.398315+0000","last_active":"2024-04-20T17:50:36.360940+0000","last_peered":"2024-04-20T17:50:36.360940+0000","last_clean":"2024-04-20T17:50:36.360940+0000","last_became_active":"2024-04-20T17:45:32.644584+0000","last_became_peered":"2024-04-20T17:45:32.644584+0000","last_unstale":"2024-04-20T17:50:36.360940+0000","last_undegraded":"2024-04-20T17:50:36.360940+0000","last_fullsized":"2024-04-20T17:50:36.360940+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:23.398142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:23.398142+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:31.965635+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"933'10570","reported_seq":10764,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-04-20T17:50:38.550201+0000","last_change":"2024-04-20T17:50:30.712810+0000","last_active":"2024-04-20T17:50:38.550201+0000","last_peered":"2024-04-20T17:50:38.550201+0000","last_clean":"2024-04-20T17:50:38.550201+0000","last_became_active":"2024-04-20T17:48:54.813062+0000","last_became_peered":"2024-04-20T17:48:54.813062+0000","last_unstale":"2024-04-20T17:50:38.550201+0000","last_undegraded":"2024-04-20T17:50:38.550201+0000","last_fullsized":"2024-04-20T17:50:38.550201+0000","mapping_epoch":894,"log_start":"900'500","ondisk_log_start":"900'500","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'10026","last_scrub_stamp":"2024-04-20T17:50:30.712648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:30.712648+0000","objects_scrubbed":9967,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:58.995403+0000","scrub_duration":1871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":678297600,"num_objects":10350,"num_object_clones":0,"num_object_copies":20700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10570,"num_write_kb":669440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"15'137","reported_seq":853,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-20T17:50:33.137903+0000","last_change":"2024-04-20T17:50:33.137929+0000","last_active":"2024-04-20T17:50:33.137903+0000","last_peered":"2024-04-20T17:50:33.137903+0000","last_clean":"2024-04-20T17:50:33.137903+0000","last_became_active":"2024-04-20T17:46:40.881713+0000","last_became_peered":"2024-04-20T17:46:40.881713+0000","last_unstale":"2024-04-20T17:50:33.137903+0000","last_undegraded":"2024-04-20T17:50:33.137903+0000","last_fullsized":"2024-04-20T17:50:33.137903+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:33.137831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:33.137831+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:36.471855+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.8","version":"933'10550","reported_seq":10738,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-04-20T17:50:37.118509+0000","last_change":"2024-04-20T17:50:19.940419+0000","last_active":"2024-04-20T17:50:37.118509+0000","last_peered":"2024-04-20T17:50:37.118509+0000","last_clean":"2024-04-20T17:50:37.118509+0000","last_became_active":"2024-04-20T17:48:54.813368+0000","last_became_peered":"2024-04-20T17:48:54.813368+0000","last_unstale":"2024-04-20T17:50:37.118509+0000","last_undegraded":"2024-04-20T17:50:37.118509+0000","last_fullsized":"2024-04-20T17:50:37.118509+0000","mapping_epoch":894,"log_start":"900'500","ondisk_log_start":"900'500","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"927'9391","last_scrub_stamp":"2024-04-20T17:50:19.940199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:19.940199+0000","objects_scrubbed":9334,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:34.165962+0000","scrub_duration":1977,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":689307648,"num_objects":10518,"num_object_clones":0,"num_object_copies":21036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10550,"num_write_kb":674176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":851,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-20T17:50:30.954880+0000","last_change":"2024-04-20T17:50:21.111183+0000","last_active":"2024-04-20T17:50:30.954880+0000","last_peered":"2024-04-20T17:50:30.954880+0000","last_clean":"2024-04-20T17:50:30.954880+0000","last_became_active":"2024-04-20T17:46:40.851683+0000","last_became_peered":"2024-04-20T17:46:40.851683+0000","last_unstale":"2024-04-20T17:50:30.954880+0000","last_undegraded":"2024-04-20T17:50:30.954880+0000","last_fullsized":"2024-04-20T17:50:30.954880+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:21.111059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:21.111059+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:45.968902+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":10832510976,"num_objects":165291,"num_object_clones":0,"num_object_copies":330582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":166061,"num_write_kb":10603264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21639004160,"data_stored":21639004160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160761,"ondisk_log_size":160761,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},"log_size":1507,"ondisk_log_size":1507,"up":22,"acting":22,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991772,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2933720,"kb_used_data":2009868,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":90811432,"statfs":{"total":95995035648,"available":92990906368,"internally_reserved":0,"allocated":2058104832,"data_stored":2057696646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":946010564},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253411,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4111100,"kb_used_data":3295540,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":89634052,"statfs":{"total":95995035648,"available":91785269248,"internally_reserved":0,"allocated":3374632960,"data_stored":3374236832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":835123648},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613160,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2723672,"kb_used_data":2002508,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":91021480,"statfs":{"total":95995035648,"available":93205995520,"internally_reserved":0,"allocated":2050568192,"data_stored":2050160006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":738458048},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809390,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1048636,"kb_used_data":648180,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":92696516,"statfs":{"total":95995035648,"available":94921232384,"internally_reserved":0,"allocated":663736320,"data_stored":663340192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":799,"seq":3431678869580,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2098812,"kb_used_data":1960052,"kb_used_omap":11,"kb_used_meta":138740,"kb_avail":91646340,"statfs":{"total":95995035648,"available":93845852160,"internally_reserved":0,"allocated":2007093248,"data_stored":2006697128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11990,"internal_metadata":142070058},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":535,"seq":2297807503620,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":713884,"kb_used_data":663820,"kb_used_omap":170,"kb_used_meta":49877,"kb_avail":93031268,"statfs":{"total":95995035648,"available":95264018432,"internally_reserved":0,"allocated":679751680,"data_stored":679343494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":174628,"internal_metadata":51074524},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":659,"seq":2830383448242,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4224508,"kb_used_data":3922804,"kb_used_omap":4,"kb_used_meta":301691,"kb_avail":89520644,"statfs":{"total":95995035648,"available":91669139456,"internally_reserved":0,"allocated":4016951296,"data_stored":4016555168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4410,"internal_metadata":308932294},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738965,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":744128,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":93001024,"statfs":{"total":95995035648,"available":95233048576,"internally_reserved":0,"allocated":774144,"data_stored":378016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394254,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":715412,"kb_used_data":296328,"kb_used_omap":202,"kb_used_meta":418869,"kb_avail":93029740,"statfs":{"total":95995035648,"available":95262453760,"internally_reserved":0,"allocated":303439872,"data_stored":303034138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":428922327},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653304,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3217816,"kb_used_data":2649548,"kb_used_omap":4,"kb_used_meta":568251,"kb_avail":90527336,"statfs":{"total":95995035648,"available":92699992064,"internally_reserved":0,"allocated":2713137152,"data_stored":2712728966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":581889735},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986752,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1941512,"kb_used_data":1324480,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":91803640,"statfs":{"total":95995035648,"available":94006927360,"internally_reserved":0,"allocated":1356267520,"data_stored":1355865338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790522,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3807764,"kb_used_data":2950536,"kb_used_omap":1,"kb_used_meta":857214,"kb_avail":89937388,"statfs":{"total":95995035648,"available":92095885312,"internally_reserved":0,"allocated":3021348864,"data_stored":3020943130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":877787589},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4024369152,"data_stored":4024369152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2696806400,"data_stored":2696806400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1358888960,"data_stored":1358888960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2712731648,"data_stored":2712731648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":679477248,"data_stored":679477248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2016411648,"data_stored":2016411648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":665321472,"data_stored":665321472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2046820352,"data_stored":2046820352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":3380412416,"data_stored":3380412416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2057764864,"data_stored":2057764864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:50:39.908 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:50:39.983 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:50:40.015 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:40.046 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:50:40.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:40.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:50:40.075 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:40 smithi171 ceph-mon[44189]: from='client.28211 v1:172.21.15.5:0/884364557' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:50:40.094 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:50:40.153 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:50:40.199 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:40.220 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:50:40.267 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:40 smithi005 ceph-mon[39562]: from='client.28211 v1:172.21.15.5:0/884364557' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:50:40.322 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:50:40.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:40.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:50:40.433 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:50:40.464 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:40 smithi106 ceph-mon[44423]: from='client.28211 v1:172.21.15.5:0/884364557' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:50:40.466 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:40.492 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:50:40.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:40.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:50:40.596 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:50:40.643 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:40.670 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:50:40.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:40.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:50:40.780 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:50:40.814 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:40.976 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:50:40.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:40.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:50:41.082 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:50:41.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:41 smithi171 ceph-mon[44189]: pgmap v2460: 27 pgs: 27 active+clean; 10 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 56 MiB/s wr, 1.19k op/s 2024-04-20T17:50:41.119 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:41.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:41 smithi005 ceph-mon[39562]: pgmap v2460: 27 pgs: 27 active+clean; 10 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 56 MiB/s wr, 1.19k op/s 2024-04-20T17:50:41.487 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:50:41.487 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:50:41.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:41 smithi106 ceph-mon[44423]: pgmap v2460: 27 pgs: 27 active+clean; 10 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 56 MiB/s wr, 1.19k op/s 2024-04-20T17:50:41.552 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:50:41.576 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:50:41.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:41.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:50:41.690 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:50:41.748 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:41.809 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:50:41.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:41.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:50:41.919 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:50:41.969 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:42.180 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:50:42.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:42.188 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:50:42.216 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-20T17:50:42.289 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:50:42.340 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:42.609 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:50:42.638 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:50:42.991 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:50:42.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:42.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:50:43.098 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:50:43.159 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:44.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:43 smithi171 ceph-mon[44189]: pgmap v2461: 27 pgs: 27 active+clean; 9.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 47 MiB/s wr, 1.25k op/s 2024-04-20T17:50:44.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09092738170777438]}]: dispatch 2024-04-20T17:50:44.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08634812352387042]}]: dispatch 2024-04-20T17:50:44.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08821276270418357]}]: dispatch 2024-04-20T17:50:44.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08674344910711093]}]: dispatch 2024-04-20T17:50:44.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08821276270418357]}]: dispatch 2024-04-20T17:50:44.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08659137433658654]}]: dispatch 2024-04-20T17:50:44.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08685787890906788]}]: dispatch 2024-04-20T17:50:44.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08642409407069769]}]: dispatch 2024-04-20T17:50:44.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08821276270418357]}]: dispatch 2024-04-20T17:50:44.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08629387843984285]}]: dispatch 2024-04-20T17:50:44.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08613054417466207]}]: dispatch 2024-04-20T17:50:44.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:43 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08659137433658654]}]: dispatch 2024-04-20T17:50:44.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:43 smithi005 ceph-mon[39562]: pgmap v2461: 27 pgs: 27 active+clean; 9.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 47 MiB/s wr, 1.25k op/s 2024-04-20T17:50:44.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09092738170777438]}]: dispatch 2024-04-20T17:50:44.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08634812352387042]}]: dispatch 2024-04-20T17:50:44.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08821276270418357]}]: dispatch 2024-04-20T17:50:44.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08674344910711093]}]: dispatch 2024-04-20T17:50:44.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08821276270418357]}]: dispatch 2024-04-20T17:50:44.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08659137433658654]}]: dispatch 2024-04-20T17:50:44.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08685787890906788]}]: dispatch 2024-04-20T17:50:44.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08642409407069769]}]: dispatch 2024-04-20T17:50:44.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08821276270418357]}]: dispatch 2024-04-20T17:50:44.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08629387843984285]}]: dispatch 2024-04-20T17:50:44.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08613054417466207]}]: dispatch 2024-04-20T17:50:44.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:43 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08659137433658654]}]: dispatch 2024-04-20T17:50:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:43 smithi106 ceph-mon[44423]: pgmap v2461: 27 pgs: 27 active+clean; 9.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 47 MiB/s wr, 1.25k op/s 2024-04-20T17:50:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09092738170777438]}]: dispatch 2024-04-20T17:50:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08634812352387042]}]: dispatch 2024-04-20T17:50:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08821276270418357]}]: dispatch 2024-04-20T17:50:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08674344910711093]}]: dispatch 2024-04-20T17:50:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08821276270418357]}]: dispatch 2024-04-20T17:50:44.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08659137433658654]}]: dispatch 2024-04-20T17:50:44.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08685787890906788]}]: dispatch 2024-04-20T17:50:44.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08642409407069769]}]: dispatch 2024-04-20T17:50:44.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08821276270418357]}]: dispatch 2024-04-20T17:50:44.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08629387843984285]}]: dispatch 2024-04-20T17:50:44.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08613054417466207]}]: dispatch 2024-04-20T17:50:44.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:43 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08659137433658654]}]: dispatch 2024-04-20T17:50:45.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09092738170777438]}]': finished 2024-04-20T17:50:45.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08634812352387042]}]': finished 2024-04-20T17:50:45.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08821276270418357]}]': finished 2024-04-20T17:50:45.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08674344910711093]}]': finished 2024-04-20T17:50:45.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08821276270418357]}]': finished 2024-04-20T17:50:45.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08659137433658654]}]': finished 2024-04-20T17:50:45.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08685787890906788]}]': finished 2024-04-20T17:50:45.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08642409407069769]}]': finished 2024-04-20T17:50:45.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08821276270418357]}]': finished 2024-04-20T17:50:45.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08629387843984285]}]': finished 2024-04-20T17:50:45.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08613054417466207]}]': finished 2024-04-20T17:50:45.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08659137433658654]}]': finished 2024-04-20T17:50:45.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:44 smithi171 ceph-mon[44189]: osdmap e934: 12 total, 12 up, 12 in 2024-04-20T17:50:45.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:44 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1009078089' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:50:45.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09092738170777438]}]': finished 2024-04-20T17:50:45.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08634812352387042]}]': finished 2024-04-20T17:50:45.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08821276270418357]}]': finished 2024-04-20T17:50:45.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08674344910711093]}]': finished 2024-04-20T17:50:45.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08821276270418357]}]': finished 2024-04-20T17:50:45.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08659137433658654]}]': finished 2024-04-20T17:50:45.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08685787890906788]}]': finished 2024-04-20T17:50:45.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08642409407069769]}]': finished 2024-04-20T17:50:45.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08821276270418357]}]': finished 2024-04-20T17:50:45.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08629387843984285]}]': finished 2024-04-20T17:50:45.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08613054417466207]}]': finished 2024-04-20T17:50:45.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08659137433658654]}]': finished 2024-04-20T17:50:45.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:44 smithi005 ceph-mon[39562]: osdmap e934: 12 total, 12 up, 12 in 2024-04-20T17:50:45.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:44 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1009078089' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:50:45.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09092738170777438]}]': finished 2024-04-20T17:50:45.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08634812352387042]}]': finished 2024-04-20T17:50:45.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.08821276270418357]}]': finished 2024-04-20T17:50:45.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08674344910711093]}]': finished 2024-04-20T17:50:45.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.08821276270418357]}]': finished 2024-04-20T17:50:45.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08659137433658654]}]': finished 2024-04-20T17:50:45.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08685787890906788]}]': finished 2024-04-20T17:50:45.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08642409407069769]}]': finished 2024-04-20T17:50:45.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08821276270418357]}]': finished 2024-04-20T17:50:45.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08629387843984285]}]': finished 2024-04-20T17:50:45.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08613054417466207]}]': finished 2024-04-20T17:50:45.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08659137433658654]}]': finished 2024-04-20T17:50:45.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:44 smithi106 ceph-mon[44423]: osdmap e934: 12 total, 12 up, 12 in 2024-04-20T17:50:45.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:44 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1009078089' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:50:45.424 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:50:45.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:45.427 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:50:45.528 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:50:45.561 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:45.853 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:45 smithi005 ceph-mon[39562]: pgmap v2463: 27 pgs: 1 unknown, 26 active+clean; 9.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 117 B/s rd, 0 op/s 2024-04-20T17:50:45.854 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:45 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:50:45.854 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:45 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1009078089' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:50:45.854 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:45 smithi005 ceph-mon[39562]: osdmap e935: 12 total, 12 up, 12 in 2024-04-20T17:50:46.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:45 smithi171 ceph-mon[44189]: pgmap v2463: 27 pgs: 1 unknown, 26 active+clean; 9.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 117 B/s rd, 0 op/s 2024-04-20T17:50:46.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:45 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:50:46.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:45 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1009078089' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:50:46.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:45 smithi171 ceph-mon[44189]: osdmap e935: 12 total, 12 up, 12 in 2024-04-20T17:50:46.142 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:50:46.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:46.146 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:50:46.247 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:50:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:45 smithi106 ceph-mon[44423]: pgmap v2463: 27 pgs: 1 unknown, 26 active+clean; 9.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 117 B/s rd, 0 op/s 2024-04-20T17:50:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:45 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:50:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:45 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1009078089' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:50:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:45 smithi106 ceph-mon[44423]: osdmap e935: 12 total, 12 up, 12 in 2024-04-20T17:50:46.278 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:46.301 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:50:46.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:46.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:50:46.405 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:50:46.455 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:46.462 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:50:46.653 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:50:46.653 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:50:47.011 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:46 smithi171 ceph-mon[44189]: osdmap e936: 12 total, 12 up, 12 in 2024-04-20T17:50:47.011 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:46 smithi171 ceph-mon[44189]: 9.3 starting backfill to osd.8 from (0'0,0'0] MAX to 933'10833 2024-04-20T17:50:47.011 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:46 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3041800513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:50:47.092 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:46 smithi005 ceph-mon[39562]: osdmap e936: 12 total, 12 up, 12 in 2024-04-20T17:50:47.092 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:46 smithi005 ceph-mon[39562]: 9.3 starting backfill to osd.8 from (0'0,0'0] MAX to 933'10833 2024-04-20T17:50:47.093 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:46 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3041800513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:50:47.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:46 smithi106 ceph-mon[44423]: osdmap e936: 12 total, 12 up, 12 in 2024-04-20T17:50:47.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:46 smithi106 ceph-mon[44423]: 9.3 starting backfill to osd.8 from (0'0,0'0] MAX to 933'10833 2024-04-20T17:50:47.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:46 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3041800513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:50:47.386 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-20T17:50:47.432 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:50:47.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:47.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:50:47.537 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:50:47.545 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:50:47.545 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:50:47.584 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:47.622 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:50:47.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:47.631 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:50:47.732 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:50:47.764 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:47.797 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:50:47.802 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:50:47.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:47.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:50:47.909 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:50:47.944 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:47.967 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:50:47.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:47.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:50:48.072 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:50:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:47 smithi171 ceph-mon[44189]: pgmap v2466: 27 pgs: 1 unknown, 26 active+clean; 9.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:50:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:47 smithi171 ceph-mon[44189]: from='client.17997 v1:172.21.15.5:0/2367076537' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:50:48.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:47 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.101 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.101 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.101 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.101 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.101 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:47 smithi106 ceph-mon[44423]: pgmap v2466: 27 pgs: 1 unknown, 26 active+clean; 9.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:50:48.102 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.102 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.102 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.102 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.102 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:47 smithi106 ceph-mon[44423]: from='client.17997 v1:172.21.15.5:0/2367076537' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:50:48.102 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.102 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.102 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:47 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.103 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:48.131 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:50:48.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:48.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:50:48.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:47 smithi005 ceph-mon[39562]: pgmap v2466: 27 pgs: 1 unknown, 26 active+clean; 9.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:50:48.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:47 smithi005 ceph-mon[39562]: from='client.17997 v1:172.21.15.5:0/2367076537' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:50:48.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:47 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:48.189 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":2466,"stamp":"2024-04-20T17:50:46.689626+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":9770500664,"num_objects":149080,"num_object_clones":0,"num_object_copies":298160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149080,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163934,"num_write_kb":10016321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152341,"ondisk_log_size":152341,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":27504048,"kb_used_data":20917388,"kb_used_omap":404,"kb_used_meta":6585195,"kb_avail":1097437776,"statfs":{"total":1151940427776,"available":1123776282624,"internally_reserved":0,"allocated":21419405312,"data_stored":21414967628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":414014,"internal_metadata":6743240386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1003880448,"num_objects":-15318,"num_object_clones":0,"num_object_copies":-30636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-15318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-5044,"num_write_kb":-651584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001137"},"pg_stats":[{"pgid":"9.7","version":"934'10836","reported_seq":11031,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-04-20T17:50:44.778732+0000","last_change":"2024-04-20T17:50:27.211659+0000","last_active":"2024-04-20T17:50:44.778732+0000","last_peered":"2024-04-20T17:50:44.778732+0000","last_clean":"2024-04-20T17:50:44.778732+0000","last_became_active":"2024-04-20T17:48:54.814508+0000","last_became_peered":"2024-04-20T17:48:54.814508+0000","last_unstale":"2024-04-20T17:50:44.778732+0000","last_undegraded":"2024-04-20T17:50:44.778732+0000","last_fullsized":"2024-04-20T17:50:44.778732+0000","mapping_epoch":894,"log_start":"901'800","ondisk_log_start":"901'800","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'9665","last_scrub_stamp":"2024-04-20T17:50:27.211421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:27.211421+0000","objects_scrubbed":9588,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:56.034157+0000","scrub_duration":2310,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":643956736,"num_objects":9826,"num_object_clones":0,"num_object_copies":19652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10836,"num_write_kb":661184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"934'10881","reported_seq":11075,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-04-20T17:50:44.778967+0000","last_change":"2024-04-20T17:50:25.096410+0000","last_active":"2024-04-20T17:50:44.778967+0000","last_peered":"2024-04-20T17:50:44.778967+0000","last_clean":"2024-04-20T17:50:44.778967+0000","last_became_active":"2024-04-20T17:48:54.818033+0000","last_became_peered":"2024-04-20T17:48:54.818033+0000","last_unstale":"2024-04-20T17:50:44.778967+0000","last_undegraded":"2024-04-20T17:50:44.778967+0000","last_fullsized":"2024-04-20T17:50:44.778967+0000","mapping_epoch":894,"log_start":"901'800","ondisk_log_start":"901'800","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"929'9608","last_scrub_stamp":"2024-04-20T17:50:25.096273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:25.096273+0000","objects_scrubbed":9524,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:47.683348+0000","scrub_duration":2229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":648740864,"num_objects":9899,"num_object_clones":0,"num_object_copies":19798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10881,"num_write_kb":664960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.5","version":"934'10871","reported_seq":11066,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-04-20T17:50:45.647428+0000","last_change":"2024-04-20T17:50:23.865871+0000","last_active":"2024-04-20T17:50:45.647428+0000","last_peered":"2024-04-20T17:50:45.647428+0000","last_clean":"2024-04-20T17:50:45.647428+0000","last_became_active":"2024-04-20T17:48:54.817749+0000","last_became_peered":"2024-04-20T17:48:54.817749+0000","last_unstale":"2024-04-20T17:50:45.647428+0000","last_undegraded":"2024-04-20T17:50:45.647428+0000","last_fullsized":"2024-04-20T17:50:45.647428+0000","mapping_epoch":894,"log_start":"901'800","ondisk_log_start":"901'800","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9546","last_scrub_stamp":"2024-04-20T17:50:23.865708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:23.865708+0000","objects_scrubbed":9473,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:24.406565+0000","scrub_duration":1985,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":647036928,"num_objects":9873,"num_object_clones":0,"num_object_copies":19746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10871,"num_write_kb":663808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"934'10868","reported_seq":11061,"reported_epoch":936,"state":"active+clean","last_fresh":"2024-04-20T17:50:45.779699+0000","last_change":"2024-04-20T17:50:23.032137+0000","last_active":"2024-04-20T17:50:45.779699+0000","last_peered":"2024-04-20T17:50:45.779699+0000","last_clean":"2024-04-20T17:50:45.779699+0000","last_became_active":"2024-04-20T17:48:55.061199+0000","last_became_peered":"2024-04-20T17:48:55.061199+0000","last_unstale":"2024-04-20T17:50:45.779699+0000","last_undegraded":"2024-04-20T17:50:45.779699+0000","last_fullsized":"2024-04-20T17:50:45.779699+0000","mapping_epoch":894,"log_start":"901'800","ondisk_log_start":"901'800","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9404","last_scrub_stamp":"2024-04-20T17:50:23.031924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:23.031924+0000","objects_scrubbed":9366,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:30.444210+0000","scrub_duration":1573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":645136384,"num_objects":9844,"num_object_clones":0,"num_object_copies":19688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10868,"num_write_kb":662784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":934,"state":"unknown","last_fresh":"2024-04-20T17:50:43.786019+0000","last_change":"2024-04-20T17:50:43.978277+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-20T17:50:43.786019+0000","last_undegraded":"2024-04-20T17:50:43.786019+0000","last_fullsized":"2024-04-20T17:50:43.786019+0000","mapping_epoch":934,"log_start":"0'0","ondisk_log_start":"0'0","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'9622","last_scrub_stamp":"2024-04-20T17:50:27.118695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:27.118695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.e","version":"934'10963","reported_seq":11054,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-04-20T17:50:44.777682+0000","last_change":"2024-04-20T17:48:54.815383+0000","last_active":"2024-04-20T17:50:44.777682+0000","last_peered":"2024-04-20T17:50:44.777682+0000","last_clean":"2024-04-20T17:50:44.777682+0000","last_became_active":"2024-04-20T17:48:54.815159+0000","last_became_peered":"2024-04-20T17:48:54.815159+0000","last_unstale":"2024-04-20T17:50:44.777682+0000","last_undegraded":"2024-04-20T17:50:44.777682+0000","last_fullsized":"2024-04-20T17:50:44.777682+0000","mapping_epoch":894,"log_start":"901'900","ondisk_log_start":"901'900","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:48:53.794927+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":649396224,"num_objects":9909,"num_object_clones":0,"num_object_copies":19818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10963,"num_write_kb":667904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"15'137","reported_seq":913,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-04-20T17:50:44.778856+0000","last_change":"2024-04-20T17:49:26.955123+0000","last_active":"2024-04-20T17:50:44.778856+0000","last_peered":"2024-04-20T17:50:44.778856+0000","last_clean":"2024-04-20T17:50:44.778856+0000","last_became_active":"2024-04-20T17:45:32.670941+0000","last_became_peered":"2024-04-20T17:45:32.670941+0000","last_unstale":"2024-04-20T17:50:44.778856+0000","last_undegraded":"2024-04-20T17:50:44.778856+0000","last_fullsized":"2024-04-20T17:50:44.778856+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:49:26.954998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:49:26.954998+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:50:44.799838+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"934'10961","reported_seq":11152,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-04-20T17:50:43.961970+0000","last_change":"2024-04-20T17:50:23.641303+0000","last_active":"2024-04-20T17:50:43.961970+0000","last_peered":"2024-04-20T17:50:43.961970+0000","last_clean":"2024-04-20T17:50:43.961970+0000","last_became_active":"2024-04-20T17:48:55.061161+0000","last_became_peered":"2024-04-20T17:48:55.061161+0000","last_unstale":"2024-04-20T17:50:43.961970+0000","last_undegraded":"2024-04-20T17:50:43.961970+0000","last_fullsized":"2024-04-20T17:50:43.961970+0000","mapping_epoch":894,"log_start":"901'900","ondisk_log_start":"901'900","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9495","last_scrub_stamp":"2024-04-20T17:50:23.641164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:23.641164+0000","objects_scrubbed":9459,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:29.146891+0000","scrub_duration":1363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":649789440,"num_objects":9915,"num_object_clones":0,"num_object_copies":19830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10961,"num_write_kb":668032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"15'137","reported_seq":915,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-04-20T17:50:36.007507+0000","last_change":"2024-04-20T17:50:25.724423+0000","last_active":"2024-04-20T17:50:36.007507+0000","last_peered":"2024-04-20T17:50:36.007507+0000","last_clean":"2024-04-20T17:50:36.007507+0000","last_became_active":"2024-04-20T17:45:29.817528+0000","last_became_peered":"2024-04-20T17:45:29.817528+0000","last_unstale":"2024-04-20T17:50:36.007507+0000","last_undegraded":"2024-04-20T17:50:36.007507+0000","last_fullsized":"2024-04-20T17:50:36.007507+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:25.724296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:25.724296+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:37.439709+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"933'10974","reported_seq":11170,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-04-20T17:50:43.573883+0000","last_change":"2024-04-20T17:50:29.927851+0000","last_active":"2024-04-20T17:50:43.573883+0000","last_peered":"2024-04-20T17:50:43.573883+0000","last_clean":"2024-04-20T17:50:43.573883+0000","last_became_active":"2024-04-20T17:48:54.817445+0000","last_became_peered":"2024-04-20T17:48:54.817445+0000","last_unstale":"2024-04-20T17:50:43.573883+0000","last_undegraded":"2024-04-20T17:50:43.573883+0000","last_fullsized":"2024-04-20T17:50:43.573883+0000","mapping_epoch":894,"log_start":"901'900","ondisk_log_start":"901'900","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'10062","last_scrub_stamp":"2024-04-20T17:50:29.927762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:29.927762+0000","objects_scrubbed":9978,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:58.752509+0000","scrub_duration":2071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":658243584,"num_objects":10044,"num_object_clones":0,"num_object_copies":20088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10974,"num_write_kb":672576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"15'137","reported_seq":945,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-04-20T17:50:45.647430+0000","last_change":"2024-04-20T17:50:20.138119+0000","last_active":"2024-04-20T17:50:45.647430+0000","last_peered":"2024-04-20T17:50:45.647430+0000","last_clean":"2024-04-20T17:50:45.647430+0000","last_became_active":"2024-04-20T17:45:29.827577+0000","last_became_peered":"2024-04-20T17:45:29.827577+0000","last_unstale":"2024-04-20T17:50:45.647430+0000","last_undegraded":"2024-04-20T17:50:45.647430+0000","last_fullsized":"2024-04-20T17:50:45.647430+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:20.137983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:20.137983+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:30.668575+0000","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"934'10933","reported_seq":11091,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-04-20T17:50:44.777638+0000","last_change":"2024-04-20T17:50:35.016798+0000","last_active":"2024-04-20T17:50:44.777638+0000","last_peered":"2024-04-20T17:50:44.777638+0000","last_clean":"2024-04-20T17:50:44.777638+0000","last_became_active":"2024-04-20T17:48:55.061208+0000","last_became_peered":"2024-04-20T17:48:55.061208+0000","last_unstale":"2024-04-20T17:50:44.777638+0000","last_undegraded":"2024-04-20T17:50:44.777638+0000","last_fullsized":"2024-04-20T17:50:44.777638+0000","mapping_epoch":894,"log_start":"901'900","ondisk_log_start":"901'900","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:48:53.794927+0000","objects_scrubbed":6124,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:50:45.978573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":646840344,"num_objects":9871,"num_object_clones":0,"num_object_copies":19742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9871,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10933,"num_write_kb":665665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"15'137","reported_seq":913,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-04-20T17:50:44.778732+0000","last_change":"2024-04-20T17:50:18.122040+0000","last_active":"2024-04-20T17:50:44.778732+0000","last_peered":"2024-04-20T17:50:44.778732+0000","last_clean":"2024-04-20T17:50:44.778732+0000","last_became_active":"2024-04-20T17:45:32.666523+0000","last_became_peered":"2024-04-20T17:45:32.666523+0000","last_unstale":"2024-04-20T17:50:44.778732+0000","last_undegraded":"2024-04-20T17:50:44.778732+0000","last_fullsized":"2024-04-20T17:50:44.778732+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:18.121912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:18.121912+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:25.993620+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.a","version":"934'11068","reported_seq":11263,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-04-20T17:50:44.002545+0000","last_change":"2024-04-20T17:50:22.180642+0000","last_active":"2024-04-20T17:50:44.002545+0000","last_peered":"2024-04-20T17:50:44.002545+0000","last_clean":"2024-04-20T17:50:44.002545+0000","last_became_active":"2024-04-20T17:48:55.061180+0000","last_became_peered":"2024-04-20T17:48:55.061180+0000","last_unstale":"2024-04-20T17:50:44.002545+0000","last_undegraded":"2024-04-20T17:50:44.002545+0000","last_fullsized":"2024-04-20T17:50:44.002545+0000","mapping_epoch":894,"log_start":"901'1000","ondisk_log_start":"901'1000","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9574","last_scrub_stamp":"2024-04-20T17:50:22.180501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:22.180501+0000","objects_scrubbed":9507,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:51.128747+0000","scrub_duration":1824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":649461760,"num_objects":9910,"num_object_clones":0,"num_object_copies":19820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11068,"num_write_kb":671296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"15'137","reported_seq":915,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-04-20T17:50:44.777647+0000","last_change":"2024-04-20T17:50:31.133012+0000","last_active":"2024-04-20T17:50:44.777647+0000","last_peered":"2024-04-20T17:50:44.777647+0000","last_clean":"2024-04-20T17:50:44.777647+0000","last_became_active":"2024-04-20T17:46:40.831961+0000","last_became_peered":"2024-04-20T17:46:40.831961+0000","last_unstale":"2024-04-20T17:50:44.777647+0000","last_undegraded":"2024-04-20T17:50:44.777647+0000","last_fullsized":"2024-04-20T17:50:44.777647+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:31.132879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:31.132879+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:31.822631+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"934'10864","reported_seq":11052,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-04-20T17:50:43.926171+0000","last_change":"2024-04-20T17:50:19.657299+0000","last_active":"2024-04-20T17:50:43.926171+0000","last_peered":"2024-04-20T17:50:43.926171+0000","last_clean":"2024-04-20T17:50:43.926171+0000","last_became_active":"2024-04-20T17:48:54.816644+0000","last_became_peered":"2024-04-20T17:48:54.816644+0000","last_unstale":"2024-04-20T17:50:43.926171+0000","last_undegraded":"2024-04-20T17:50:43.926171+0000","last_fullsized":"2024-04-20T17:50:43.926171+0000","mapping_epoch":894,"log_start":"901'800","ondisk_log_start":"901'800","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"927'9204","last_scrub_stamp":"2024-04-20T17:50:19.657194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:19.657194+0000","objects_scrubbed":9124,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:25.188599+0000","scrub_duration":2257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":648937472,"num_objects":9902,"num_object_clones":0,"num_object_copies":19804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10864,"num_write_kb":664512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"15'137","reported_seq":702,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-04-20T17:50:36.004826+0000","last_change":"2024-04-20T17:49:47.812349+0000","last_active":"2024-04-20T17:50:36.004826+0000","last_peered":"2024-04-20T17:50:36.004826+0000","last_clean":"2024-04-20T17:50:36.004826+0000","last_became_active":"2024-04-20T17:49:46.824559+0000","last_became_peered":"2024-04-20T17:49:46.824559+0000","last_unstale":"2024-04-20T17:50:36.004826+0000","last_undegraded":"2024-04-20T17:50:36.004826+0000","last_fullsized":"2024-04-20T17:50:36.004826+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:49:47.812234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:49:47.812234+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:50:49.678662+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"934'11224","reported_seq":11422,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-04-20T17:50:44.778692+0000","last_change":"2024-04-20T17:50:28.283117+0000","last_active":"2024-04-20T17:50:44.778692+0000","last_peered":"2024-04-20T17:50:44.778692+0000","last_clean":"2024-04-20T17:50:44.778692+0000","last_became_active":"2024-04-20T17:48:54.813199+0000","last_became_peered":"2024-04-20T17:48:54.813199+0000","last_unstale":"2024-04-20T17:50:44.778692+0000","last_undegraded":"2024-04-20T17:50:44.778692+0000","last_fullsized":"2024-04-20T17:50:44.778692+0000","mapping_epoch":894,"log_start":"901'1200","ondisk_log_start":"901'1200","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'10025","last_scrub_stamp":"2024-04-20T17:50:28.282957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:28.282957+0000","objects_scrubbed":9963,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:48.575002+0000","scrub_duration":2339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":659161088,"num_objects":10058,"num_object_clones":0,"num_object_copies":20116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11224,"num_write_kb":681024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"15'137","reported_seq":913,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-04-20T17:50:44.778829+0000","last_change":"2024-04-20T17:49:29.040029+0000","last_active":"2024-04-20T17:50:44.778829+0000","last_peered":"2024-04-20T17:50:44.778829+0000","last_clean":"2024-04-20T17:50:44.778829+0000","last_became_active":"2024-04-20T17:45:32.782706+0000","last_became_peered":"2024-04-20T17:45:32.782706+0000","last_unstale":"2024-04-20T17:50:44.778829+0000","last_undegraded":"2024-04-20T17:50:44.778829+0000","last_fullsized":"2024-04-20T17:50:44.778829+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:49:29.039907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:49:29.039907+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:50:49.985844+0000","scrub_duration":161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.2","version":"933'10870","reported_seq":11057,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-04-20T17:50:43.555690+0000","last_change":"2024-04-20T17:50:22.440005+0000","last_active":"2024-04-20T17:50:43.555690+0000","last_peered":"2024-04-20T17:50:43.555690+0000","last_clean":"2024-04-20T17:50:43.555690+0000","last_became_active":"2024-04-20T17:48:55.060965+0000","last_became_peered":"2024-04-20T17:48:55.060965+0000","last_unstale":"2024-04-20T17:50:43.555690+0000","last_undegraded":"2024-04-20T17:50:43.555690+0000","last_fullsized":"2024-04-20T17:50:43.555690+0000","mapping_epoch":894,"log_start":"901'800","ondisk_log_start":"901'800","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9408","last_scrub_stamp":"2024-04-20T17:50:22.439841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:22.439841+0000","objects_scrubbed":9355,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:23.199554+0000","scrub_duration":1641,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":649461760,"num_objects":9910,"num_object_clones":0,"num_object_copies":19820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10870,"num_write_kb":664960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"15'137","reported_seq":895,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-04-20T17:50:36.011485+0000","last_change":"2024-04-20T17:49:48.210622+0000","last_active":"2024-04-20T17:50:36.011485+0000","last_peered":"2024-04-20T17:50:36.011485+0000","last_clean":"2024-04-20T17:50:36.011485+0000","last_became_active":"2024-04-20T17:45:32.669451+0000","last_became_peered":"2024-04-20T17:45:32.669451+0000","last_unstale":"2024-04-20T17:50:36.011485+0000","last_undegraded":"2024-04-20T17:50:36.011485+0000","last_fullsized":"2024-04-20T17:50:36.011485+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:49:48.210506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:49:48.210506+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:05.387619+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"933'10730","reported_seq":10923,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-04-20T17:50:41.730083+0000","last_change":"2024-04-20T17:50:30.239298+0000","last_active":"2024-04-20T17:50:41.730083+0000","last_peered":"2024-04-20T17:50:41.730083+0000","last_clean":"2024-04-20T17:50:41.730083+0000","last_became_active":"2024-04-20T17:48:55.061200+0000","last_became_peered":"2024-04-20T17:48:55.061200+0000","last_unstale":"2024-04-20T17:50:41.730083+0000","last_undegraded":"2024-04-20T17:50:41.730083+0000","last_fullsized":"2024-04-20T17:50:41.730083+0000","mapping_epoch":894,"log_start":"901'700","ondisk_log_start":"901'700","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'9940","last_scrub_stamp":"2024-04-20T17:50:30.239131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:30.239131+0000","objects_scrubbed":9887,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:36.933027+0000","scrub_duration":1672,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":656670720,"num_objects":10020,"num_object_clones":0,"num_object_copies":20040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10730,"num_write_kb":664000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"15'137","reported_seq":916,"reported_epoch":936,"state":"active+clean","last_fresh":"2024-04-20T17:50:46.289302+0000","last_change":"2024-04-20T17:50:23.398315+0000","last_active":"2024-04-20T17:50:46.289302+0000","last_peered":"2024-04-20T17:50:46.289302+0000","last_clean":"2024-04-20T17:50:46.289302+0000","last_became_active":"2024-04-20T17:45:32.644584+0000","last_became_peered":"2024-04-20T17:45:32.644584+0000","last_unstale":"2024-04-20T17:50:46.289302+0000","last_undegraded":"2024-04-20T17:50:46.289302+0000","last_fullsized":"2024-04-20T17:50:46.289302+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:23.398142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:23.398142+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:31.965635+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"933'10944","reported_seq":11137,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-04-20T17:50:43.580670+0000","last_change":"2024-04-20T17:50:30.712810+0000","last_active":"2024-04-20T17:50:43.580670+0000","last_peered":"2024-04-20T17:50:43.580670+0000","last_clean":"2024-04-20T17:50:43.580670+0000","last_became_active":"2024-04-20T17:48:54.813062+0000","last_became_peered":"2024-04-20T17:48:54.813062+0000","last_unstale":"2024-04-20T17:50:43.580670+0000","last_undegraded":"2024-04-20T17:50:43.580670+0000","last_fullsized":"2024-04-20T17:50:43.580670+0000","mapping_epoch":894,"log_start":"901'900","ondisk_log_start":"901'900","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'10026","last_scrub_stamp":"2024-04-20T17:50:30.712648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:30.712648+0000","objects_scrubbed":9967,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:58.995403+0000","scrub_duration":1871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":653787136,"num_objects":9976,"num_object_clones":0,"num_object_copies":19952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10944,"num_write_kb":669440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"15'137","reported_seq":861,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-04-20T17:50:44.777641+0000","last_change":"2024-04-20T17:50:33.137929+0000","last_active":"2024-04-20T17:50:44.777641+0000","last_peered":"2024-04-20T17:50:44.777641+0000","last_clean":"2024-04-20T17:50:44.777641+0000","last_became_active":"2024-04-20T17:46:40.881713+0000","last_became_peered":"2024-04-20T17:46:40.881713+0000","last_unstale":"2024-04-20T17:50:44.777641+0000","last_undegraded":"2024-04-20T17:50:44.777641+0000","last_fullsized":"2024-04-20T17:50:44.777641+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:33.137831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:33.137831+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:36.471855+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.8","version":"933'10947","reported_seq":11136,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-04-20T17:50:42.112927+0000","last_change":"2024-04-20T17:50:19.940419+0000","last_active":"2024-04-20T17:50:42.112927+0000","last_peered":"2024-04-20T17:50:42.112927+0000","last_clean":"2024-04-20T17:50:42.112927+0000","last_became_active":"2024-04-20T17:48:54.813368+0000","last_became_peered":"2024-04-20T17:48:54.813368+0000","last_unstale":"2024-04-20T17:50:42.112927+0000","last_undegraded":"2024-04-20T17:50:42.112927+0000","last_fullsized":"2024-04-20T17:50:42.112927+0000","mapping_epoch":894,"log_start":"901'900","ondisk_log_start":"901'900","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"927'9391","last_scrub_stamp":"2024-04-20T17:50:19.940199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:19.940199+0000","objects_scrubbed":9334,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:34.165962+0000","scrub_duration":1977,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":663289856,"num_objects":10121,"num_object_clones":0,"num_object_copies":20242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10947,"num_write_kb":674176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":860,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-04-20T17:50:44.777839+0000","last_change":"2024-04-20T17:50:21.111183+0000","last_active":"2024-04-20T17:50:44.777839+0000","last_peered":"2024-04-20T17:50:44.777839+0000","last_clean":"2024-04-20T17:50:44.777839+0000","last_became_active":"2024-04-20T17:46:40.851683+0000","last_became_peered":"2024-04-20T17:46:40.851683+0000","last_unstale":"2024-04-20T17:50:44.777839+0000","last_undegraded":"2024-04-20T17:50:44.777839+0000","last_fullsized":"2024-04-20T17:50:44.777839+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:21.111059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:21.111059+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:45.968902+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":9769910296,"num_objects":149078,"num_object_clones":0,"num_object_copies":298156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149078,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163934,"num_write_kb":10016321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20808146944,"data_stored":20808138800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150834,"ondisk_log_size":150834,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},"log_size":1507,"ondisk_log_size":1507,"up":22,"acting":22,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991774,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2829988,"kb_used_data":1914324,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":90915164,"statfs":{"total":95995035648,"available":93097127936,"internally_reserved":0,"allocated":1960267776,"data_stored":1959921309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":937490884},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253413,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3986164,"kb_used_data":3174624,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":89758988,"statfs":{"total":95995035648,"available":91913203712,"internally_reserved":0,"allocated":3250814976,"data_stored":3250462521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":830863808},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613161,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2673812,"kb_used_data":1930444,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":91071340,"statfs":{"total":95995035648,"available":93257052160,"internally_reserved":0,"allocated":1976774656,"data_stored":1976366470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":761199040},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809392,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1038036,"kb_used_data":637580,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":92707116,"statfs":{"total":95995035648,"available":94932086784,"internally_reserved":0,"allocated":652881920,"data_stored":652473734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":799,"seq":3431678869582,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2031144,"kb_used_data":1896408,"kb_used_omap":11,"kb_used_meta":134580,"kb_avail":91714008,"statfs":{"total":95995035648,"available":93915144192,"internally_reserved":0,"allocated":1941921792,"data_stored":1941571261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11990,"internal_metadata":137810218},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":535,"seq":2297807503621,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":693268,"kb_used_data":643212,"kb_used_omap":170,"kb_used_meta":49877,"kb_avail":93051884,"statfs":{"total":95995035648,"available":95285129216,"internally_reserved":0,"allocated":658649088,"data_stored":658240902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":174628,"internal_metadata":51074524},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":659,"seq":2830383448244,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4134056,"kb_used_data":3796120,"kb_used_omap":4,"kb_used_meta":337787,"kb_avail":89611096,"statfs":{"total":95995035648,"available":91761762304,"internally_reserved":0,"allocated":3887226880,"data_stored":3886876341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4410,"internal_metadata":345894598},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738967,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":744180,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":93000972,"statfs":{"total":95995035648,"available":95232995328,"internally_reserved":0,"allocated":688128,"data_stored":335673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394256,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":696116,"kb_used_data":277032,"kb_used_omap":202,"kb_used_meta":418869,"kb_avail":93049036,"statfs":{"total":95995035648,"available":95282212864,"internally_reserved":0,"allocated":283680768,"data_stored":283324761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":428922327},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653305,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3110932,"kb_used_data":2555148,"kb_used_omap":4,"kb_used_meta":555771,"kb_avail":90634220,"statfs":{"total":95995035648,"available":92809441280,"internally_reserved":0,"allocated":2616471552,"data_stored":2616063366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":569110215},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986754,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1880356,"kb_used_data":1271508,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":91864796,"statfs":{"total":95995035648,"available":94069551104,"internally_reserved":0,"allocated":1302024192,"data_stored":1301677725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790524,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3685996,"kb_used_data":2820316,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":90059156,"statfs":{"total":95995035648,"available":92220575744,"internally_reserved":0,"allocated":2888003584,"data_stored":2887653565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":886307269},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3884388352,"data_stored":3884384280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2600796160,"data_stored":2600796160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1301348352,"data_stored":1301348352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2595028992,"data_stored":2595028992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":656605184,"data_stored":656605184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1940197376,"data_stored":1940193304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":648937472,"data_stored":648937472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1971453952,"data_stored":1971453952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":3250126848,"data_stored":3250126848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1959264256,"data_stored":1959264256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:50:48.190 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:50:48.235 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:50:48.266 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:48.281 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:50:48.281 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:50:48.281 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:50:48.281 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:50:48.281 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:50:48.281 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:50:48.281 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:50:48.281 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:50:48.281 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:50:48.281 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:50:48.281 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:50:48.281 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:50:48.282 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:50:48.282 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:50:48.282 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:50:48.282 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:50:48.282 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:50:48.282 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:50:48.282 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:50:48.282 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:50:48.282 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:50:48.282 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:50:48.282 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:50:48.282 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:50:48.295 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:50:48.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:48.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:50:48.403 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:50:48.434 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:48.463 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:50:48.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:48.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:50:48.570 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:50:48.616 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:48.667 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:50:48.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:48.675 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:50:48.776 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:50:48.817 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:48.847 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:50:48.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:48.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:50:48.951 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:50:48.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:48 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:50:48.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:48 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:50:48.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:48 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:50:48.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:48 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:50:48.993 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:49.051 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:50:49.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:49.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:50:49.085 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:50:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:48 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:50:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:48 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:50:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:48 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:50:49.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:48 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:50:49.156 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:50:49.218 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:49.243 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:50:49.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:49.251 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:50:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:48 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:50:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:48 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:50:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:48 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:50:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:48 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:50:49.352 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:50:49.397 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:49.433 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:50:49.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:49.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:50:49.454 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:50:49.538 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:50:49.567 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:49.591 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:50:49.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:49.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:50:49.696 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:50:49.794 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:49.834 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:50:49.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:49.842 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:50:49.885 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:50:49.885 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-04-20T17:50:49.885 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.885 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27772.0:179129 9.6 9:69bfaeee:::benchmark_data_smithi171_188319_object12333:head [delete] snapc 0=[] ondisk+write+known_if_redirected e937)", 2024-04-20T17:50:49.885 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:50:49.864300+0000", 2024-04-20T17:50:49.885 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.015943895999999999, 2024-04-20T17:50:49.886 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015976892999999999, 2024-04-20T17:50:49.886 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:50:49.886 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:50:49.886 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:50:49.886 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:50:49.886 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27772", 2024-04-20T17:50:49.886 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:50:49.886 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 179129 2024-04-20T17:50:49.886 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.886 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:50:49.886 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.886 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:50:49.886 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.864300+0000", 2024-04-20T17:50:49.886 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:49.886 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.887 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.887 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:50:49.887 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.864300+0000", 2024-04-20T17:50:49.887 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:49.887 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.887 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.887 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:50:49.887 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.864301+0000", 2024-04-20T17:50:49.887 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4099999999999997e-07 2024-04-20T17:50:49.887 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.887 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.887 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:50:49.887 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.864305+0000", 2024-04-20T17:50:49.887 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8990000000000003e-06 2024-04-20T17:50:49.888 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.888 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.888 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:50:49.888 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.864311+0000", 2024-04-20T17:50:49.888 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3949999999999998e-06 2024-04-20T17:50:49.888 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.888 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.888 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:50:49.888 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.864317+0000", 2024-04-20T17:50:49.888 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8680000000000004e-06 2024-04-20T17:50:49.888 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.888 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.888 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:50:49.888 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.866449+0000", 2024-04-20T17:50:49.888 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0021321249999999999 2024-04-20T17:50:49.889 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.889 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.889 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:50:49.889 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.866550+0000", 2024-04-20T17:50:49.889 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010066800000000001 2024-04-20T17:50:49.889 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.889 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.889 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:50:49.889 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.866650+0000", 2024-04-20T17:50:49.889 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000100173 2024-04-20T17:50:49.889 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.889 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.889 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:50:49.889 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.877157+0000", 2024-04-20T17:50:49.890 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010507008999999999 2024-04-20T17:50:49.890 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:49.890 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:49.890 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:49.890 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.890 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.890 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27772.0:179133 9.6 9:64c8e839:::benchmark_data_smithi171_188319_object12337:head [delete] snapc 0=[] ondisk+write+known_if_redirected e937)", 2024-04-20T17:50:49.890 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:50:49.871973+0000", 2024-04-20T17:50:49.890 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.0082706329999999995, 2024-04-20T17:50:49.890 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0083620919999999998, 2024-04-20T17:50:49.890 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:50:49.890 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:50:49.890 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:50:49.891 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:50:49.891 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27772", 2024-04-20T17:50:49.891 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:50:49.891 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 179133 2024-04-20T17:50:49.891 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.891 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:50:49.891 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.891 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:50:49.891 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.871973+0000", 2024-04-20T17:50:49.891 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:49.891 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.891 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.891 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:50:49.891 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.871973+0000", 2024-04-20T17:50:49.891 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:49.892 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.892 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.892 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:50:49.892 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.871975+0000", 2024-04-20T17:50:49.892 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8169999999999999e-06 2024-04-20T17:50:49.892 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.892 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.892 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:50:49.892 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.871980+0000", 2024-04-20T17:50:49.892 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.809e-06 2024-04-20T17:50:49.892 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.892 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.892 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:50:49.892 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.871990+0000", 2024-04-20T17:50:49.892 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9720000000000004e-06 2024-04-20T17:50:49.893 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.893 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.893 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:50:49.893 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.871997+0000", 2024-04-20T17:50:49.893 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9009999999999997e-06 2024-04-20T17:50:49.893 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.893 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.893 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:50:49.893 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.872046+0000", 2024-04-20T17:50:49.893 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9434999999999997e-05 2024-04-20T17:50:49.893 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.893 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.893 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:50:49.893 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.872177+0000", 2024-04-20T17:50:49.893 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013072600000000001 2024-04-20T17:50:49.894 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.894 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.894 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5", 2024-04-20T17:50:49.894 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.872309+0000", 2024-04-20T17:50:49.894 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000132261 2024-04-20T17:50:49.894 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:49.894 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:49.894 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:49.894 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.894 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.894 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.27772.0:179136 9.e e937/894 9:744b467d:::benchmark_data_smithi171_188319_object12340:head v 937'11190, mlcod=936'11187)", 2024-04-20T17:50:49.894 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:50:49.876774+0000", 2024-04-20T17:50:49.894 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.0034695490000000002, 2024-04-20T17:50:49.894 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.003605245, 2024-04-20T17:50:49.895 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:50:49.895 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:50:49.895 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:50:49.895 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:50:49.895 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.895 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:50:49.895 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.876774+0000", 2024-04-20T17:50:49.895 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:49.895 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.895 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.895 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:50:49.895 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.876774+0000", 2024-04-20T17:50:49.895 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:49.895 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.895 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.896 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:50:49.896 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.876775+0000", 2024-04-20T17:50:49.896 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3200000000000004e-07 2024-04-20T17:50:49.896 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.896 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.896 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:50:49.896 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.876784+0000", 2024-04-20T17:50:49.896 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0769999999999997e-06 2024-04-20T17:50:49.896 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.896 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.896 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:50:49.896 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.876790+0000", 2024-04-20T17:50:49.896 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.139e-06 2024-04-20T17:50:49.896 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.896 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.897 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:50:49.897 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.876795+0000", 2024-04-20T17:50:49.897 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8010000000000003e-06 2024-04-20T17:50:49.897 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.897 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.897 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:50:49.897 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.877184+0000", 2024-04-20T17:50:49.897 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00038934000000000001 2024-04-20T17:50:49.897 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.897 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.897 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:50:49.897 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.877200+0000", 2024-04-20T17:50:49.897 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5196e-05 2024-04-20T17:50:49.897 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:49.897 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:49.897 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:49.898 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.898 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.898 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.27772.0:179131 9.b e937/894 9:dfd33483:::benchmark_data_smithi171_188319_object12335:head v 937'11094, mlcod=937'11091)", 2024-04-20T17:50:49.898 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:50:49.876808+0000", 2024-04-20T17:50:49.898 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.0034359830000000001, 2024-04-20T17:50:49.898 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0036063319999999999, 2024-04-20T17:50:49.898 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:50:49.898 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:50:49.898 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-04-20T17:50:49.898 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:50:49.898 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.898 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:50:49.898 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.876808+0000", 2024-04-20T17:50:49.898 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:49.899 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.899 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.899 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:50:49.899 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.876808+0000", 2024-04-20T17:50:49.899 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:49.899 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.899 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.899 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:50:49.899 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.876808+0000", 2024-04-20T17:50:49.899 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4099999999999998e-07 2024-04-20T17:50:49.899 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.899 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.899 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:50:49.899 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.876812+0000", 2024-04-20T17:50:49.899 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5710000000000002e-06 2024-04-20T17:50:49.900 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.900 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.900 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:50:49.900 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.876817+0000", 2024-04-20T17:50:49.900 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3879999999999999e-06 2024-04-20T17:50:49.900 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.900 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.900 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:50:49.900 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.876821+0000", 2024-04-20T17:50:49.900 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0419999999999996e-06 2024-04-20T17:50:49.900 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.900 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.900 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-04-20T17:50:49.900 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.876937+0000", 2024-04-20T17:50:49.900 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000115048 2024-04-20T17:50:49.901 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.901 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.901 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-04-20T17:50:49.901 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.876956+0000", 2024-04-20T17:50:49.901 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9913000000000001e-05 2024-04-20T17:50:49.901 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:49.901 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:49.901 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:49.901 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.901 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.901 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.27772.0:179138 9.b e937/894)", 2024-04-20T17:50:49.901 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:50:49.876832+0000", 2024-04-20T17:50:49.901 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.0034118989999999999, 2024-04-20T17:50:49.901 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.003615297, 2024-04-20T17:50:49.901 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:50:49.902 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:50:49.902 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "queued for pg", 2024-04-20T17:50:49.902 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:50:49.902 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.902 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:50:49.902 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.876832+0000", 2024-04-20T17:50:49.902 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:49.902 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.902 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.902 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:50:49.903 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.876832+0000", 2024-04-20T17:50:49.903 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:49.903 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.903 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.903 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:50:49.903 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.876832+0000", 2024-04-20T17:50:49.903 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6699999999999999e-07 2024-04-20T17:50:49.903 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.903 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.903 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:50:49.903 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.876836+0000", 2024-04-20T17:50:49.903 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2729999999999998e-06 2024-04-20T17:50:49.903 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.903 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.903 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:50:49.903 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.876840+0000", 2024-04-20T17:50:49.904 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0400000000000003e-06 2024-04-20T17:50:49.904 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.904 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.904 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:50:49.904 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.876843+0000", 2024-04-20T17:50:49.904 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6370000000000001e-06 2024-04-20T17:50:49.904 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:49.904 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:49.904 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:49.904 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.904 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.904 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27772.0:179140 9.6 9.e2689676 (undecoded) ondisk+write+known_if_redirected e937)", 2024-04-20T17:50:49.904 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:50:49.878420+0000", 2024-04-20T17:50:49.904 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.0018234779999999999, 2024-04-20T17:50:49.904 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0020542709999999999, 2024-04-20T17:50:49.905 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:50:49.905 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:50:49.905 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "queued for pg", 2024-04-20T17:50:49.905 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:50:49.905 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27772", 2024-04-20T17:50:49.905 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:50:49.905 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 179140 2024-04-20T17:50:49.905 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.905 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:50:49.905 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.905 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:50:49.905 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.878420+0000", 2024-04-20T17:50:49.905 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:49.905 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.905 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.905 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:50:49.906 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.878420+0000", 2024-04-20T17:50:49.906 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:49.906 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.906 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.906 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:50:49.906 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.878423+0000", 2024-04-20T17:50:49.906 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2409999999999998e-06 2024-04-20T17:50:49.906 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.906 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.906 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:50:49.906 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.878428+0000", 2024-04-20T17:50:49.906 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8069999999999996e-06 2024-04-20T17:50:49.906 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.906 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.906 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:50:49.907 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.878439+0000", 2024-04-20T17:50:49.907 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0719e-05 2024-04-20T17:50:49.907 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.907 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.907 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:50:49.907 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.878446+0000", 2024-04-20T17:50:49.907 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2010000000000002e-06 2024-04-20T17:50:49.907 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:49.907 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:49.907 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:49.907 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.907 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.907 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27772.0:179141 9.6 9.735ed1c6 (undecoded) ondisk+write+known_if_redirected e937)", 2024-04-20T17:50:49.908 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-04-20T17:50:49.878459+0000", 2024-04-20T17:50:49.908 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.00178503, 2024-04-20T17:50:49.908 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0020460729999999998, 2024-04-20T17:50:49.908 INFO:teuthology.orchestra.run.smithi171.stdout: "continuous": false, 2024-04-20T17:50:49.908 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-04-20T17:50:49.908 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "queued for pg", 2024-04-20T17:50:49.908 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-04-20T17:50:49.908 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27772", 2024-04-20T17:50:49.908 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:50:49.908 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 179141 2024-04-20T17:50:49.908 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.908 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-04-20T17:50:49.908 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.909 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-04-20T17:50:49.909 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.878459+0000", 2024-04-20T17:50:49.909 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:49.909 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.909 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.909 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-04-20T17:50:49.909 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.878459+0000", 2024-04-20T17:50:49.909 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-04-20T17:50:49.909 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.909 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.909 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-04-20T17:50:49.909 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.878459+0000", 2024-04-20T17:50:49.909 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0900000000000001e-07 2024-04-20T17:50:49.909 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.909 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.910 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-04-20T17:50:49.910 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.878462+0000", 2024-04-20T17:50:49.910 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.903e-06 2024-04-20T17:50:49.910 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.910 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.910 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-04-20T17:50:49.910 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.878467+0000", 2024-04-20T17:50:49.910 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9989999999999999e-06 2024-04-20T17:50:49.910 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-20T17:50:49.910 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-04-20T17:50:49.910 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-04-20T17:50:49.910 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-04-20T17:50:49.878472+0000", 2024-04-20T17:50:49.910 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9400000000000001e-06 2024-04-20T17:50:49.910 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:49.910 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-20T17:50:49.911 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:49.911 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-20T17:50:49.911 INFO:teuthology.orchestra.run.smithi171.stdout: ], 2024-04-20T17:50:49.911 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 7 2024-04-20T17:50:49.911 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:50:49.942 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:50:49.979 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:49 smithi106 ceph-mon[44423]: pgmap v2467: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 9.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 837 op/s; 9542/315482 objects misplaced (3.025%); 1.3 MiB/s, 20 objects/s recovering 2024-04-20T17:50:50.062 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:50:50.066 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:49 smithi005 ceph-mon[39562]: pgmap v2467: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 9.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 837 op/s; 9542/315482 objects misplaced (3.025%); 1.3 MiB/s, 20 objects/s recovering 2024-04-20T17:50:50.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:50.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:50:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:49 smithi171 ceph-mon[44189]: pgmap v2467: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 9.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 837 op/s; 9542/315482 objects misplaced (3.025%); 1.3 MiB/s, 20 objects/s recovering 2024-04-20T17:50:50.168 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:50:50.201 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:50.223 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:50:50.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:50.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:50:50.328 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:50:50.363 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:50.386 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:50:50.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:50.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:50:50.493 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:50:50.524 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:50.601 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:50:50.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:50.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:50:50.680 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:50:50.707 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:50:50.712 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-20T17:50:50.755 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:50.788 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:50:50.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:50.791 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:50:50.811 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:50 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:50:50.811 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:50 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:50:50.811 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:50 smithi005 ceph-mon[39562]: osdmap e937: 12 total, 12 up, 12 in 2024-04-20T17:50:50.892 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:50:50.946 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:50 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:50:50.946 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:50 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:50:50.947 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:50 smithi171 ceph-mon[44189]: osdmap e937: 12 total, 12 up, 12 in 2024-04-20T17:50:50.959 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:50.989 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:50:50.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:50.995 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:50:51.059 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:50:51.095 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:50:51.150 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:51.164 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:50:51.183 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:50:51.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:51.189 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:50:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:50 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:50:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:50 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:50:51.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:50 smithi106 ceph-mon[44423]: osdmap e937: 12 total, 12 up, 12 in 2024-04-20T17:50:51.290 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:50:51.334 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:51.437 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:50:51.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:51.440 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:50:51.541 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:50:51.574 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:51.933 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:50:51.936 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:51 smithi005 ceph-mon[39562]: pgmap v2469: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 9.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 2.73k op/s; 9542/311348 objects misplaced (3.065%); 1.3 MiB/s, 20 objects/s recovering 2024-04-20T17:50:51.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:51.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:50:52.008 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:50:52.012 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:50:52.038 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:50:52.074 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:51 smithi171 ceph-mon[44189]: pgmap v2469: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 9.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 2.73k op/s; 9542/311348 objects misplaced (3.065%); 1.3 MiB/s, 20 objects/s recovering 2024-04-20T17:50:52.117 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:50:52.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:52.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:50:52.223 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:50:52.253 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:51 smithi106 ceph-mon[44423]: pgmap v2469: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 9.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 107 MiB/s wr, 2.73k op/s; 9542/311348 objects misplaced (3.065%); 1.3 MiB/s, 20 objects/s recovering 2024-04-20T17:50:52.256 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:52.279 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:50:52.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:52.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:50:52.389 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:50:52.417 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:52.447 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:50:52.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:52.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:50:52.553 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:50:52.594 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:52.848 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:50:52.864 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:50:52.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:52.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:50:52.969 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:50:53.014 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:53.099 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:50:53.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:53.106 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:50:53.207 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:50:53.272 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:53.276 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-20T17:50:53.276 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-04-20T17:50:53.276 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-04-20T17:50:53.276 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-04-20T17:50:53.276 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-20T17:50:53.298 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:50:53.302 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":2469,"stamp":"2024-04-20T17:50:50.690328+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":10202645048,"num_objects":155674,"num_object_clones":0,"num_object_copies":311348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9542,"num_objects_unfound":0,"num_objects_dirty":155674,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177916,"num_write_kb":10674753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":8192000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162423,"ondisk_log_size":162423,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":27182988,"kb_used_data":20548216,"kb_used_omap":404,"kb_used_meta":6633771,"kb_avail":1097758836,"statfs":{"total":1151940427776,"available":1124105048064,"internally_reserved":0,"allocated":21041373184,"data_stored":21037080374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":414014,"internal_metadata":6792982210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,10,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":275054592,"num_objects":4197,"num_object_clones":0,"num_object_copies":8394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9542,"num_objects_unfound":0,"num_objects_dirty":4197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16379,"num_write_kb":658432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":8192000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001034"},"pg_stats":[{"pgid":"9.7","version":"937'11103","reported_seq":11302,"reported_epoch":937,"state":"active+clean","last_fresh":"2024-04-20T17:50:50.127378+0000","last_change":"2024-04-20T17:50:27.211659+0000","last_active":"2024-04-20T17:50:50.127378+0000","last_peered":"2024-04-20T17:50:50.127378+0000","last_clean":"2024-04-20T17:50:50.127378+0000","last_became_active":"2024-04-20T17:48:54.814508+0000","last_became_peered":"2024-04-20T17:48:54.814508+0000","last_unstale":"2024-04-20T17:50:50.127378+0000","last_undegraded":"2024-04-20T17:50:50.127378+0000","last_fullsized":"2024-04-20T17:50:50.127378+0000","mapping_epoch":894,"log_start":"901'1100","ondisk_log_start":"901'1100","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'9665","last_scrub_stamp":"2024-04-20T17:50:27.211421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:27.211421+0000","objects_scrubbed":9588,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:56.034157+0000","scrub_duration":2310,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":626458624,"num_objects":9559,"num_object_clones":0,"num_object_copies":19118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11103,"num_write_kb":661184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"937'11182","reported_seq":11380,"reported_epoch":937,"state":"active+clean","last_fresh":"2024-04-20T17:50:50.119344+0000","last_change":"2024-04-20T17:50:25.096410+0000","last_active":"2024-04-20T17:50:50.119344+0000","last_peered":"2024-04-20T17:50:50.119344+0000","last_clean":"2024-04-20T17:50:50.119344+0000","last_became_active":"2024-04-20T17:48:54.818033+0000","last_became_peered":"2024-04-20T17:48:54.818033+0000","last_unstale":"2024-04-20T17:50:50.119344+0000","last_undegraded":"2024-04-20T17:50:50.119344+0000","last_fullsized":"2024-04-20T17:50:50.119344+0000","mapping_epoch":894,"log_start":"901'1100","ondisk_log_start":"901'1100","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"929'9608","last_scrub_stamp":"2024-04-20T17:50:25.096273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:25.096273+0000","objects_scrubbed":9524,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:47.683348+0000","scrub_duration":2229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":629014528,"num_objects":9598,"num_object_clones":0,"num_object_copies":19196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11182,"num_write_kb":664960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.5","version":"934'10871","reported_seq":11066,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-04-20T17:50:45.647428+0000","last_change":"2024-04-20T17:50:23.865871+0000","last_active":"2024-04-20T17:50:45.647428+0000","last_peered":"2024-04-20T17:50:45.647428+0000","last_clean":"2024-04-20T17:50:45.647428+0000","last_became_active":"2024-04-20T17:48:54.817749+0000","last_became_peered":"2024-04-20T17:48:54.817749+0000","last_unstale":"2024-04-20T17:50:45.647428+0000","last_undegraded":"2024-04-20T17:50:45.647428+0000","last_fullsized":"2024-04-20T17:50:45.647428+0000","mapping_epoch":894,"log_start":"901'800","ondisk_log_start":"901'800","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9546","last_scrub_stamp":"2024-04-20T17:50:23.865708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:23.865708+0000","objects_scrubbed":9473,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:24.406565+0000","scrub_duration":1985,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":647036928,"num_objects":9873,"num_object_clones":0,"num_object_copies":19746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10871,"num_write_kb":663808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"934'10868","reported_seq":11061,"reported_epoch":936,"state":"active+clean","last_fresh":"2024-04-20T17:50:45.779699+0000","last_change":"2024-04-20T17:50:23.032137+0000","last_active":"2024-04-20T17:50:45.779699+0000","last_peered":"2024-04-20T17:50:45.779699+0000","last_clean":"2024-04-20T17:50:45.779699+0000","last_became_active":"2024-04-20T17:48:55.061199+0000","last_became_peered":"2024-04-20T17:48:55.061199+0000","last_unstale":"2024-04-20T17:50:45.779699+0000","last_undegraded":"2024-04-20T17:50:45.779699+0000","last_fullsized":"2024-04-20T17:50:45.779699+0000","mapping_epoch":894,"log_start":"901'800","ondisk_log_start":"901'800","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9404","last_scrub_stamp":"2024-04-20T17:50:23.031924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:23.031924+0000","objects_scrubbed":9366,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:30.444210+0000","scrub_duration":1573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":645136384,"num_objects":9844,"num_object_clones":0,"num_object_copies":19688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10868,"num_write_kb":662784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"936'10910","reported_seq":11365,"reported_epoch":936,"state":"active+remapped+backfilling","last_fresh":"2024-04-20T17:50:47.144791+0000","last_change":"2024-04-20T17:50:46.276928+0000","last_active":"2024-04-20T17:50:47.144791+0000","last_peered":"2024-04-20T17:50:47.144791+0000","last_clean":"2024-04-20T17:50:43.748764+0000","last_became_active":"2024-04-20T17:50:46.159037+0000","last_became_peered":"2024-04-20T17:50:46.159037+0000","last_unstale":"2024-04-20T17:50:47.144791+0000","last_undegraded":"2024-04-20T17:50:47.144791+0000","last_fullsized":"2024-04-20T17:50:47.144791+0000","mapping_epoch":935,"log_start":"901'900","ondisk_log_start":"901'900","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'9622","last_scrub_stamp":"2024-04-20T17:50:27.118695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:27.118695+0000","objects_scrubbed":9558,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":633470976,"num_objects":9666,"num_object_clones":0,"num_object_copies":19332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9542,"num_objects_unfound":0,"num_objects_dirty":9666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10910,"num_write_kb":658432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":8192000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[9,3],"avail_no_missing":["9","3"],"object_location_counts":[{"shards":"3,9","objects":9666}],"blocked_by":[],"up_primary":8,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.e","version":"936'11187","reported_seq":11282,"reported_epoch":937,"state":"active+clean","last_fresh":"2024-04-20T17:50:49.799544+0000","last_change":"2024-04-20T17:48:54.815383+0000","last_active":"2024-04-20T17:50:49.799544+0000","last_peered":"2024-04-20T17:50:49.799544+0000","last_clean":"2024-04-20T17:50:49.799544+0000","last_became_active":"2024-04-20T17:48:54.815159+0000","last_became_peered":"2024-04-20T17:48:54.815159+0000","last_unstale":"2024-04-20T17:50:49.799544+0000","last_undegraded":"2024-04-20T17:50:49.799544+0000","last_fullsized":"2024-04-20T17:50:49.799544+0000","mapping_epoch":894,"log_start":"901'1100","ondisk_log_start":"901'1100","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:48:53.794927+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:50:49.969178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":634716160,"num_objects":9685,"num_object_clones":0,"num_object_copies":19370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11187,"num_write_kb":667904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"15'137","reported_seq":917,"reported_epoch":937,"state":"active+clean","last_fresh":"2024-04-20T17:50:49.812503+0000","last_change":"2024-04-20T17:49:26.955123+0000","last_active":"2024-04-20T17:50:49.812503+0000","last_peered":"2024-04-20T17:50:49.812503+0000","last_clean":"2024-04-20T17:50:49.812503+0000","last_became_active":"2024-04-20T17:45:32.670941+0000","last_became_peered":"2024-04-20T17:45:32.670941+0000","last_unstale":"2024-04-20T17:50:49.812503+0000","last_undegraded":"2024-04-20T17:50:49.812503+0000","last_fullsized":"2024-04-20T17:50:49.812503+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:49:26.954998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:49:26.954998+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:50:50.832553+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"937'11266","reported_seq":11463,"reported_epoch":937,"state":"active+clean","last_fresh":"2024-04-20T17:50:50.603838+0000","last_change":"2024-04-20T17:50:23.641303+0000","last_active":"2024-04-20T17:50:50.603838+0000","last_peered":"2024-04-20T17:50:50.603838+0000","last_clean":"2024-04-20T17:50:50.603838+0000","last_became_active":"2024-04-20T17:48:55.061161+0000","last_became_peered":"2024-04-20T17:48:55.061161+0000","last_unstale":"2024-04-20T17:50:50.603838+0000","last_undegraded":"2024-04-20T17:50:50.603838+0000","last_fullsized":"2024-04-20T17:50:50.603838+0000","mapping_epoch":894,"log_start":"902'1200","ondisk_log_start":"902'1200","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9495","last_scrub_stamp":"2024-04-20T17:50:23.641164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:23.641164+0000","objects_scrubbed":9459,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:29.146891+0000","scrub_duration":1363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":629800960,"num_objects":9610,"num_object_clones":0,"num_object_copies":19220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11266,"num_write_kb":668032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"15'137","reported_seq":921,"reported_epoch":936,"state":"active+clean","last_fresh":"2024-04-20T17:50:46.191464+0000","last_change":"2024-04-20T17:50:25.724423+0000","last_active":"2024-04-20T17:50:46.191464+0000","last_peered":"2024-04-20T17:50:46.191464+0000","last_clean":"2024-04-20T17:50:46.191464+0000","last_became_active":"2024-04-20T17:45:29.817528+0000","last_became_peered":"2024-04-20T17:45:29.817528+0000","last_unstale":"2024-04-20T17:50:46.191464+0000","last_undegraded":"2024-04-20T17:50:46.191464+0000","last_fullsized":"2024-04-20T17:50:46.191464+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:25.724296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:25.724296+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:37.439709+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"936'11174","reported_seq":11376,"reported_epoch":936,"state":"active+clean","last_fresh":"2024-04-20T17:50:48.585899+0000","last_change":"2024-04-20T17:50:29.927851+0000","last_active":"2024-04-20T17:50:48.585899+0000","last_peered":"2024-04-20T17:50:48.585899+0000","last_clean":"2024-04-20T17:50:48.585899+0000","last_became_active":"2024-04-20T17:48:54.817445+0000","last_became_peered":"2024-04-20T17:48:54.817445+0000","last_unstale":"2024-04-20T17:50:48.585899+0000","last_undegraded":"2024-04-20T17:50:48.585899+0000","last_fullsized":"2024-04-20T17:50:48.585899+0000","mapping_epoch":894,"log_start":"901'1100","ondisk_log_start":"901'1100","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'10062","last_scrub_stamp":"2024-04-20T17:50:29.927762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:29.927762+0000","objects_scrubbed":9978,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:58.752509+0000","scrub_duration":2071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":645136384,"num_objects":9844,"num_object_clones":0,"num_object_copies":19688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11174,"num_write_kb":672576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"15'137","reported_seq":945,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-04-20T17:50:45.647430+0000","last_change":"2024-04-20T17:50:20.138119+0000","last_active":"2024-04-20T17:50:45.647430+0000","last_peered":"2024-04-20T17:50:45.647430+0000","last_clean":"2024-04-20T17:50:45.647430+0000","last_became_active":"2024-04-20T17:45:29.827577+0000","last_became_peered":"2024-04-20T17:45:29.827577+0000","last_unstale":"2024-04-20T17:50:45.647430+0000","last_undegraded":"2024-04-20T17:50:45.647430+0000","last_fullsized":"2024-04-20T17:50:45.647430+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:20.137983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:20.137983+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:30.668575+0000","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"937'11178","reported_seq":11340,"reported_epoch":937,"state":"active+clean","last_fresh":"2024-04-20T17:50:49.803850+0000","last_change":"2024-04-20T17:50:35.016798+0000","last_active":"2024-04-20T17:50:49.803850+0000","last_peered":"2024-04-20T17:50:49.803850+0000","last_clean":"2024-04-20T17:50:49.803850+0000","last_became_active":"2024-04-20T17:48:55.061208+0000","last_became_peered":"2024-04-20T17:48:55.061208+0000","last_unstale":"2024-04-20T17:50:49.803850+0000","last_undegraded":"2024-04-20T17:50:49.803850+0000","last_fullsized":"2024-04-20T17:50:49.803850+0000","mapping_epoch":894,"log_start":"901'1100","ondisk_log_start":"901'1100","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:48:53.794927+0000","objects_scrubbed":6124,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:50:52.037135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":630784024,"num_objects":9626,"num_object_clones":0,"num_object_copies":19252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9626,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11178,"num_write_kb":665665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"15'137","reported_seq":917,"reported_epoch":937,"state":"active+clean","last_fresh":"2024-04-20T17:50:49.812136+0000","last_change":"2024-04-20T17:50:18.122040+0000","last_active":"2024-04-20T17:50:49.812136+0000","last_peered":"2024-04-20T17:50:49.812136+0000","last_clean":"2024-04-20T17:50:49.812136+0000","last_became_active":"2024-04-20T17:45:32.666523+0000","last_became_peered":"2024-04-20T17:45:32.666523+0000","last_unstale":"2024-04-20T17:50:49.812136+0000","last_undegraded":"2024-04-20T17:50:49.812136+0000","last_fullsized":"2024-04-20T17:50:49.812136+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:18.121912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:18.121912+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:25.993620+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.a","version":"937'11362","reported_seq":11563,"reported_epoch":937,"state":"active+clean","last_fresh":"2024-04-20T17:50:50.571733+0000","last_change":"2024-04-20T17:50:22.180642+0000","last_active":"2024-04-20T17:50:50.571733+0000","last_peered":"2024-04-20T17:50:50.571733+0000","last_clean":"2024-04-20T17:50:50.571733+0000","last_became_active":"2024-04-20T17:48:55.061180+0000","last_became_peered":"2024-04-20T17:48:55.061180+0000","last_unstale":"2024-04-20T17:50:50.571733+0000","last_undegraded":"2024-04-20T17:50:50.571733+0000","last_fullsized":"2024-04-20T17:50:50.571733+0000","mapping_epoch":894,"log_start":"902'1300","ondisk_log_start":"902'1300","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9574","last_scrub_stamp":"2024-04-20T17:50:22.180501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:22.180501+0000","objects_scrubbed":9507,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:51.128747+0000","scrub_duration":1824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":630194176,"num_objects":9616,"num_object_clones":0,"num_object_copies":19232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11362,"num_write_kb":671296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"15'137","reported_seq":919,"reported_epoch":937,"state":"active+clean","last_fresh":"2024-04-20T17:50:49.799517+0000","last_change":"2024-04-20T17:50:31.133012+0000","last_active":"2024-04-20T17:50:49.799517+0000","last_peered":"2024-04-20T17:50:49.799517+0000","last_clean":"2024-04-20T17:50:49.799517+0000","last_became_active":"2024-04-20T17:46:40.831961+0000","last_became_peered":"2024-04-20T17:46:40.831961+0000","last_unstale":"2024-04-20T17:50:49.799517+0000","last_undegraded":"2024-04-20T17:50:49.799517+0000","last_fullsized":"2024-04-20T17:50:49.799517+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:31.132879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:31.132879+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:31.822631+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"936'11041","reported_seq":11233,"reported_epoch":936,"state":"active+clean","last_fresh":"2024-04-20T17:50:48.970340+0000","last_change":"2024-04-20T17:50:19.657299+0000","last_active":"2024-04-20T17:50:48.970340+0000","last_peered":"2024-04-20T17:50:48.970340+0000","last_clean":"2024-04-20T17:50:48.970340+0000","last_became_active":"2024-04-20T17:48:54.816644+0000","last_became_peered":"2024-04-20T17:48:54.816644+0000","last_unstale":"2024-04-20T17:50:48.970340+0000","last_undegraded":"2024-04-20T17:50:48.970340+0000","last_fullsized":"2024-04-20T17:50:48.970340+0000","mapping_epoch":894,"log_start":"901'1000","ondisk_log_start":"901'1000","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"927'9204","last_scrub_stamp":"2024-04-20T17:50:19.657194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:19.657194+0000","objects_scrubbed":9124,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:25.188599+0000","scrub_duration":2257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":637337600,"num_objects":9725,"num_object_clones":0,"num_object_copies":19450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11041,"num_write_kb":664512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"15'137","reported_seq":708,"reported_epoch":936,"state":"active+clean","last_fresh":"2024-04-20T17:50:46.147027+0000","last_change":"2024-04-20T17:49:47.812349+0000","last_active":"2024-04-20T17:50:46.147027+0000","last_peered":"2024-04-20T17:50:46.147027+0000","last_clean":"2024-04-20T17:50:46.147027+0000","last_became_active":"2024-04-20T17:49:46.824559+0000","last_became_peered":"2024-04-20T17:49:46.824559+0000","last_unstale":"2024-04-20T17:50:46.147027+0000","last_undegraded":"2024-04-20T17:50:46.147027+0000","last_fullsized":"2024-04-20T17:50:46.147027+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:49:47.812234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:49:47.812234+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:50:49.678662+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"937'11478","reported_seq":11680,"reported_epoch":937,"state":"active+clean","last_fresh":"2024-04-20T17:50:50.146642+0000","last_change":"2024-04-20T17:50:28.283117+0000","last_active":"2024-04-20T17:50:50.146642+0000","last_peered":"2024-04-20T17:50:50.146642+0000","last_clean":"2024-04-20T17:50:50.146642+0000","last_became_active":"2024-04-20T17:48:54.813199+0000","last_became_peered":"2024-04-20T17:48:54.813199+0000","last_unstale":"2024-04-20T17:50:50.146642+0000","last_undegraded":"2024-04-20T17:50:50.146642+0000","last_fullsized":"2024-04-20T17:50:50.146642+0000","mapping_epoch":894,"log_start":"902'1400","ondisk_log_start":"902'1400","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'10025","last_scrub_stamp":"2024-04-20T17:50:28.282957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:28.282957+0000","objects_scrubbed":9963,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:48.575002+0000","scrub_duration":2339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":642514944,"num_objects":9804,"num_object_clones":0,"num_object_copies":19608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11478,"num_write_kb":681024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"15'137","reported_seq":917,"reported_epoch":937,"state":"active+clean","last_fresh":"2024-04-20T17:50:49.815431+0000","last_change":"2024-04-20T17:49:29.040029+0000","last_active":"2024-04-20T17:50:49.815431+0000","last_peered":"2024-04-20T17:50:49.815431+0000","last_clean":"2024-04-20T17:50:49.815431+0000","last_became_active":"2024-04-20T17:45:32.782706+0000","last_became_peered":"2024-04-20T17:45:32.782706+0000","last_unstale":"2024-04-20T17:50:49.815431+0000","last_undegraded":"2024-04-20T17:50:49.815431+0000","last_fullsized":"2024-04-20T17:50:49.815431+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:49:29.039907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:49:29.039907+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:50:49.985844+0000","scrub_duration":161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.2","version":"936'11041","reported_seq":11233,"reported_epoch":936,"state":"active+clean","last_fresh":"2024-04-20T17:50:48.587383+0000","last_change":"2024-04-20T17:50:22.440005+0000","last_active":"2024-04-20T17:50:48.587383+0000","last_peered":"2024-04-20T17:50:48.587383+0000","last_clean":"2024-04-20T17:50:48.587383+0000","last_became_active":"2024-04-20T17:48:55.060965+0000","last_became_peered":"2024-04-20T17:48:55.060965+0000","last_unstale":"2024-04-20T17:50:48.587383+0000","last_undegraded":"2024-04-20T17:50:48.587383+0000","last_fullsized":"2024-04-20T17:50:48.587383+0000","mapping_epoch":894,"log_start":"901'1000","ondisk_log_start":"901'1000","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9408","last_scrub_stamp":"2024-04-20T17:50:22.439841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:22.439841+0000","objects_scrubbed":9355,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:23.199554+0000","scrub_duration":1641,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":638255104,"num_objects":9739,"num_object_clones":0,"num_object_copies":19478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11041,"num_write_kb":664960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"15'137","reported_seq":901,"reported_epoch":936,"state":"active+clean","last_fresh":"2024-04-20T17:50:45.781503+0000","last_change":"2024-04-20T17:49:48.210622+0000","last_active":"2024-04-20T17:50:45.781503+0000","last_peered":"2024-04-20T17:50:45.781503+0000","last_clean":"2024-04-20T17:50:45.781503+0000","last_became_active":"2024-04-20T17:45:32.669451+0000","last_became_peered":"2024-04-20T17:45:32.669451+0000","last_unstale":"2024-04-20T17:50:45.781503+0000","last_undegraded":"2024-04-20T17:50:45.781503+0000","last_fullsized":"2024-04-20T17:50:45.781503+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:49:48.210506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:49:48.210506+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:05.387619+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"936'10939","reported_seq":11139,"reported_epoch":936,"state":"active+clean","last_fresh":"2024-04-20T17:50:46.715189+0000","last_change":"2024-04-20T17:50:30.239298+0000","last_active":"2024-04-20T17:50:46.715189+0000","last_peered":"2024-04-20T17:50:46.715189+0000","last_clean":"2024-04-20T17:50:46.715189+0000","last_became_active":"2024-04-20T17:48:55.061200+0000","last_became_peered":"2024-04-20T17:48:55.061200+0000","last_unstale":"2024-04-20T17:50:46.715189+0000","last_undegraded":"2024-04-20T17:50:46.715189+0000","last_fullsized":"2024-04-20T17:50:46.715189+0000","mapping_epoch":894,"log_start":"901'900","ondisk_log_start":"901'900","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'9940","last_scrub_stamp":"2024-04-20T17:50:30.239131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:30.239131+0000","objects_scrubbed":9887,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:36.933027+0000","scrub_duration":1672,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":642973696,"num_objects":9811,"num_object_clones":0,"num_object_copies":19622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10939,"num_write_kb":664000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"15'137","reported_seq":916,"reported_epoch":936,"state":"active+clean","last_fresh":"2024-04-20T17:50:46.289302+0000","last_change":"2024-04-20T17:50:23.398315+0000","last_active":"2024-04-20T17:50:46.289302+0000","last_peered":"2024-04-20T17:50:46.289302+0000","last_clean":"2024-04-20T17:50:46.289302+0000","last_became_active":"2024-04-20T17:45:32.644584+0000","last_became_peered":"2024-04-20T17:45:32.644584+0000","last_unstale":"2024-04-20T17:50:46.289302+0000","last_undegraded":"2024-04-20T17:50:46.289302+0000","last_fullsized":"2024-04-20T17:50:46.289302+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:23.398142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:23.398142+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:31.965635+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"936'11152","reported_seq":11352,"reported_epoch":936,"state":"active+clean","last_fresh":"2024-04-20T17:50:48.546194+0000","last_change":"2024-04-20T17:50:30.712810+0000","last_active":"2024-04-20T17:50:48.546194+0000","last_peered":"2024-04-20T17:50:48.546194+0000","last_clean":"2024-04-20T17:50:48.546194+0000","last_became_active":"2024-04-20T17:48:54.813062+0000","last_became_peered":"2024-04-20T17:48:54.813062+0000","last_unstale":"2024-04-20T17:50:48.546194+0000","last_undegraded":"2024-04-20T17:50:48.546194+0000","last_fullsized":"2024-04-20T17:50:48.546194+0000","mapping_epoch":894,"log_start":"901'1100","ondisk_log_start":"901'1100","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'10026","last_scrub_stamp":"2024-04-20T17:50:30.712648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:30.712648+0000","objects_scrubbed":9967,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:58.995403+0000","scrub_duration":1871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":640155648,"num_objects":9768,"num_object_clones":0,"num_object_copies":19536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11152,"num_write_kb":669440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"15'137","reported_seq":865,"reported_epoch":937,"state":"active+clean","last_fresh":"2024-04-20T17:50:49.799601+0000","last_change":"2024-04-20T17:50:33.137929+0000","last_active":"2024-04-20T17:50:49.799601+0000","last_peered":"2024-04-20T17:50:49.799601+0000","last_clean":"2024-04-20T17:50:49.799601+0000","last_became_active":"2024-04-20T17:46:40.881713+0000","last_became_peered":"2024-04-20T17:46:40.881713+0000","last_unstale":"2024-04-20T17:50:49.799601+0000","last_undegraded":"2024-04-20T17:50:49.799601+0000","last_fullsized":"2024-04-20T17:50:49.799601+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:33.137831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:33.137831+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:36.471855+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.8","version":"936'11164","reported_seq":11359,"reported_epoch":936,"state":"active+clean","last_fresh":"2024-04-20T17:50:47.137975+0000","last_change":"2024-04-20T17:50:19.940419+0000","last_active":"2024-04-20T17:50:47.137975+0000","last_peered":"2024-04-20T17:50:47.137975+0000","last_clean":"2024-04-20T17:50:47.137975+0000","last_became_active":"2024-04-20T17:48:54.813368+0000","last_became_peered":"2024-04-20T17:48:54.813368+0000","last_unstale":"2024-04-20T17:50:47.137975+0000","last_undegraded":"2024-04-20T17:50:47.137975+0000","last_fullsized":"2024-04-20T17:50:47.137975+0000","mapping_epoch":894,"log_start":"901'1100","ondisk_log_start":"901'1100","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"927'9391","last_scrub_stamp":"2024-04-20T17:50:19.940199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:19.940199+0000","objects_scrubbed":9334,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:34.165962+0000","scrub_duration":1977,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":649068544,"num_objects":9904,"num_object_clones":0,"num_object_copies":19808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11164,"num_write_kb":674176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":864,"reported_epoch":937,"state":"active+clean","last_fresh":"2024-04-20T17:50:49.799653+0000","last_change":"2024-04-20T17:50:21.111183+0000","last_active":"2024-04-20T17:50:49.799653+0000","last_peered":"2024-04-20T17:50:49.799653+0000","last_clean":"2024-04-20T17:50:49.799653+0000","last_became_active":"2024-04-20T17:46:40.851683+0000","last_became_peered":"2024-04-20T17:46:40.851683+0000","last_unstale":"2024-04-20T17:50:49.799653+0000","last_undegraded":"2024-04-20T17:50:49.799653+0000","last_fullsized":"2024-04-20T17:50:49.799653+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:21.111059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:21.111059+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:45.968902+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":10202054680,"num_objects":155672,"num_object_clones":0,"num_object_copies":311344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9542,"num_objects_unfound":0,"num_objects_dirty":155672,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177916,"num_write_kb":10674753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":8192000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20418600960,"data_stored":20418592816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160916,"ondisk_log_size":160916,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},"log_size":1507,"ondisk_log_size":1507,"up":22,"acting":22,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991774,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2829988,"kb_used_data":1914324,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":90915164,"statfs":{"total":95995035648,"available":93097127936,"internally_reserved":0,"allocated":1960267776,"data_stored":1959921309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":937490884},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253414,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3931268,"kb_used_data":3097592,"kb_used_omap":1,"kb_used_meta":833598,"kb_avail":89813884,"statfs":{"total":95995035648,"available":91969417216,"internally_reserved":0,"allocated":3171934208,"data_stored":3171569761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":853604800},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613162,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2636664,"kb_used_data":1889132,"kb_used_omap":1,"kb_used_meta":747518,"kb_avail":91108488,"statfs":{"total":95995035648,"available":93295091712,"internally_reserved":0,"allocated":1934471168,"data_stored":1934112696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":765458880},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809393,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1031092,"kb_used_data":643116,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":92714060,"statfs":{"total":95995035648,"available":94939197440,"internally_reserved":0,"allocated":658550784,"data_stored":658192312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":799,"seq":3431678869583,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1995064,"kb_used_data":1860464,"kb_used_omap":11,"kb_used_meta":134580,"kb_avail":91750088,"statfs":{"total":95995035648,"available":93952090112,"internally_reserved":0,"allocated":1905115136,"data_stored":1904752600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11990,"internal_metadata":137810218},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":535,"seq":2297807503622,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":679292,"kb_used_data":629228,"kb_used_omap":170,"kb_used_meta":49877,"kb_avail":93065860,"statfs":{"total":95995035648,"available":95299440640,"internally_reserved":0,"allocated":644329472,"data_stored":643971000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":174628,"internal_metadata":51074524},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":659,"seq":2830383448245,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4068664,"kb_used_data":3726704,"kb_used_omap":4,"kb_used_meta":341947,"kb_avail":89676488,"statfs":{"total":95995035648,"available":91828723712,"internally_reserved":0,"allocated":3816144896,"data_stored":3815782352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4410,"internal_metadata":350154438},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738967,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":744180,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":93000972,"statfs":{"total":95995035648,"available":95232995328,"internally_reserved":0,"allocated":688128,"data_stored":335673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394256,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":696116,"kb_used_data":277032,"kb_used_omap":202,"kb_used_meta":418869,"kb_avail":93049036,"statfs":{"total":95995035648,"available":95282212864,"internally_reserved":0,"allocated":283680768,"data_stored":283324761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":428922327},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653306,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3063796,"kb_used_data":2503852,"kb_used_omap":4,"kb_used_meta":559931,"kb_avail":90681356,"statfs":{"total":95995035648,"available":92857708544,"internally_reserved":0,"allocated":2563944448,"data_stored":2563585976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":573370055},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986754,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1880356,"kb_used_data":1271508,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":91864796,"statfs":{"total":95995035648,"available":94069551104,"internally_reserved":0,"allocated":1302024192,"data_stored":1301677725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790525,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3626508,"kb_used_data":2734592,"kb_used_omap":1,"kb_used_meta":891902,"kb_avail":90118644,"statfs":{"total":95995035648,"available":92281491456,"internally_reserved":0,"allocated":2800222208,"data_stored":2799854209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":913308101},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3805220864,"data_stored":3805216792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2550595584,"data_stored":2550595584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1301348352,"data_stored":1301348352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2512453632,"data_stored":2512453632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":642973696,"data_stored":642973696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1893797888,"data_stored":1893793816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":660078592,"data_stored":660078592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1929248768,"data_stored":1929248768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":3163619328,"data_stored":3163619328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1959264256,"data_stored":1959264256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:50:53.304 INFO:tasks.cephadm.ceph_manager.ceph:PG 9.3 is not active+clean 2024-04-20T17:50:53.304 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '9.3', 'version': "936'10910", 'reported_seq': 11365, 'reported_epoch': 936, 'state': 'active+remapped+backfilling', 'last_fresh': '2024-04-20T17:50:47.144791+0000', 'last_change': '2024-04-20T17:50:46.276928+0000', 'last_active': '2024-04-20T17:50:47.144791+0000', 'last_peered': '2024-04-20T17:50:47.144791+0000', 'last_clean': '2024-04-20T17:50:43.748764+0000', 'last_became_active': '2024-04-20T17:50:46.159037+0000', 'last_became_peered': '2024-04-20T17:50:46.159037+0000', 'last_unstale': '2024-04-20T17:50:47.144791+0000', 'last_undegraded': '2024-04-20T17:50:47.144791+0000', 'last_fullsized': '2024-04-20T17:50:47.144791+0000', 'mapping_epoch': 935, 'log_start': "901'900", 'ondisk_log_start': "901'900", 'created': 894, 'last_epoch_clean': 895, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "930'9622", 'last_scrub_stamp': '2024-04-20T17:50:27.118695+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T17:48:53.794927+0000', 'last_clean_scrub_stamp': '2024-04-20T17:50:27.118695+0000', 'objects_scrubbed': 9558, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2200, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 633470976, 'num_objects': 9666, 'num_object_clones': 0, 'num_object_copies': 19332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 9542, 'num_objects_unfound': 0, 'num_objects_dirty': 9666, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10910, 'num_write_kb': 658432, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 126, 'num_bytes_recovered': 8192000, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [8, 3], 'acting': [9, 3], 'avail_no_missing': ['9', '3'], 'object_location_counts': [{'shards': '3,9', 'objects': 9666}], 'blocked_by': [], 'up_primary': 8, 'acting_primary': 9, 'purged_snaps': []} 2024-04-20T17:50:53.304 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph status --format=json 2024-04-20T17:50:53.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:53.305 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:50:53.406 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:50:53.435 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:53.461 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:50:53.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:53.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:50:53.565 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:50:53.621 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:53.773 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:50:53.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:53.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:50:53.883 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:50:53.941 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:53.999 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:53 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:50:53.999 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:53 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:50:53.999 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:53 smithi171 ceph-mon[44189]: from='client.28232 v1:172.21.15.5:0/1158528307' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:50:53.999 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:53 smithi171 ceph-mon[44189]: pgmap v2470: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 9.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 2.27k op/s; 8550/308112 objects misplaced (2.775%); 6.2 MiB/s, 99 objects/s recovering 2024-04-20T17:50:54.000 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:53 smithi171 ceph-mon[44189]: osdmap e938: 12 total, 12 up, 12 in 2024-04-20T17:50:54.000 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:53 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:50:54.000 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:53 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:50:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:50:54.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:54.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:50:54.041 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-20T17:50:54.106 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:50:54.131 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:53 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:50:54.132 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:53 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:50:54.132 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:53 smithi005 ceph-mon[39562]: from='client.28232 v1:172.21.15.5:0/1158528307' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:50:54.132 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:53 smithi005 ceph-mon[39562]: pgmap v2470: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 9.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 2.27k op/s; 8550/308112 objects misplaced (2.775%); 6.2 MiB/s, 99 objects/s recovering 2024-04-20T17:50:54.132 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:53 smithi005 ceph-mon[39562]: osdmap e938: 12 total, 12 up, 12 in 2024-04-20T17:50:54.132 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:53 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:50:54.132 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:53 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:50:54.167 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:54.194 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:50:54.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:54.201 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:50:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:53 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:50:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:53 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:50:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:53 smithi106 ceph-mon[44423]: from='client.28232 v1:172.21.15.5:0/1158528307' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:50:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:53 smithi106 ceph-mon[44423]: pgmap v2470: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 9.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 2.27k op/s; 8550/308112 objects misplaced (2.775%); 6.2 MiB/s, 99 objects/s recovering 2024-04-20T17:50:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:53 smithi106 ceph-mon[44423]: osdmap e938: 12 total, 12 up, 12 in 2024-04-20T17:50:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:53 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:50:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:53 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:50:54.302 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:50:54.333 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:54.354 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:50:54.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:54.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:50:54.458 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:50:54.485 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:50:54.494 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:54.518 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:50:54.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:54.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:50:54.624 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:50:54.674 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:54.772 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:50:54.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:54.776 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:50:54.866 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:50:54.877 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:50:54.922 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:54.925 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:50:55.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:54 smithi171 ceph-mon[44189]: osdmap e939: 12 total, 12 up, 12 in 2024-04-20T17:50:55.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:54 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1665816052' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:50:55.146 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:50:55.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:55.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:50:55.157 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:54 smithi005 ceph-mon[39562]: osdmap e939: 12 total, 12 up, 12 in 2024-04-20T17:50:55.157 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:54 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1665816052' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:50:55.256 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:50:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:54 smithi106 ceph-mon[44423]: osdmap e939: 12 total, 12 up, 12 in 2024-04-20T17:50:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:54 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1665816052' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:50:55.314 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:55.376 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:50:55.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:55.384 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:50:55.485 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:50:55.526 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:55.607 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:50:55.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:55.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:50:55.714 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:50:55.757 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:56.141 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:55 smithi005 ceph-mon[39562]: pgmap v2473: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 9.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 850 op/s; 8550/305274 objects misplaced (2.801%); 6.9 MiB/s, 110 objects/s recovering 2024-04-20T17:50:56.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:55 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:50:56.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:55 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1665816052' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:50:56.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:55 smithi005 ceph-mon[39562]: osdmap e940: 12 total, 12 up, 12 in 2024-04-20T17:50:56.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:55 smithi106 ceph-mon[44423]: pgmap v2473: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 9.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 850 op/s; 8550/305274 objects misplaced (2.801%); 6.9 MiB/s, 110 objects/s recovering 2024-04-20T17:50:56.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:55 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:50:56.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:55 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1665816052' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:50:56.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:55 smithi106 ceph-mon[44423]: osdmap e940: 12 total, 12 up, 12 in 2024-04-20T17:50:56.314 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:55 smithi171 ceph-mon[44189]: pgmap v2473: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 9.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 850 op/s; 8550/305274 objects misplaced (2.801%); 6.9 MiB/s, 110 objects/s recovering 2024-04-20T17:50:56.314 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:55 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:50:56.314 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:55 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1665816052' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:50:56.315 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:55 smithi171 ceph-mon[44189]: osdmap e940: 12 total, 12 up, 12 in 2024-04-20T17:50:56.418 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:50:56.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:56.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:50:56.522 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:50:56.554 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:56.576 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:50:56.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:56.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:50:56.592 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:50:56.592 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:50:56.592 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:50:56.592 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:50:56.592 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:50:56.592 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:50:56.592 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:50:56.592 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:50:56.592 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:50:56.592 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:50:56.593 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:50:56.593 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:50:56.593 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:50:56.593 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:50:56.593 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:50:56.593 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:50:56.593 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:50:56.593 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:50:56.593 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:50:56.593 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:50:56.593 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:50:56.593 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:50:56.593 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:50:56.594 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:50:56.667 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:50:56.668 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:50:56.681 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:50:56.728 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:56.878 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:56 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1830048698' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:50:56.878 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:56.878 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:56.878 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:56.878 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:56.878 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:56.878 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:57.053 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:56 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1830048698' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:50:57.053 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:57.053 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:57.053 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:57.053 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:57.053 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:57.054 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:57.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:56 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1830048698' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:50:57.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:57.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:57.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:57.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:57.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:57.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:57.404 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:50:57.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:57.409 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:50:57.417 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-20T17:50:57.509 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:50:57.542 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:57.870 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:50:58.016 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:50:58.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:58.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:50:58.121 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:50:58.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:58.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:58.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:58.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:58.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:58.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:58.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:57 smithi005 ceph-mon[39562]: pgmap v2475: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 9.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 8550/299242 objects misplaced (2.857%); 6.9 MiB/s, 110 objects/s recovering 2024-04-20T17:50:58.155 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:58.164 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:58.164 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:58.164 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:58.164 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:58.164 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:58.164 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:58.164 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:57 smithi171 ceph-mon[44189]: pgmap v2475: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 9.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 8550/299242 objects misplaced (2.857%); 6.9 MiB/s, 110 objects/s recovering 2024-04-20T17:50:58.173 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:50:58.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:58.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:58.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:58.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:58.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:58.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:50:58.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:57 smithi106 ceph-mon[44423]: pgmap v2475: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 9.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 8550/299242 objects misplaced (2.857%); 6.9 MiB/s, 110 objects/s recovering 2024-04-20T17:50:58.620 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:50:58.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:58.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:50:58.704 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:50:58.727 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:50:58.784 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:58.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:58 smithi005 ceph-mon[39562]: 1.6 scrub starts 2024-04-20T17:50:58.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:58 smithi005 ceph-mon[39562]: 1.6 scrub ok 2024-04-20T17:50:58.882 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:58 smithi005 ceph-mon[39562]: pgmap v2476: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 9.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 7548/295016 objects misplaced (2.559%); 7.3 MiB/s, 116 objects/s recovering 2024-04-20T17:50:59.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:58 smithi171 ceph-mon[44189]: 1.6 scrub starts 2024-04-20T17:50:59.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:58 smithi171 ceph-mon[44189]: 1.6 scrub ok 2024-04-20T17:50:59.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:58 smithi171 ceph-mon[44189]: pgmap v2476: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 9.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 7548/295016 objects misplaced (2.559%); 7.3 MiB/s, 116 objects/s recovering 2024-04-20T17:50:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:58 smithi106 ceph-mon[44423]: 1.6 scrub starts 2024-04-20T17:50:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:58 smithi106 ceph-mon[44423]: 1.6 scrub ok 2024-04-20T17:50:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:58 smithi106 ceph-mon[44423]: pgmap v2476: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 9.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 7548/295016 objects misplaced (2.559%); 7.3 MiB/s, 116 objects/s recovering 2024-04-20T17:50:59.354 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:50:59.361 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:50:59.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:59.365 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:50:59.466 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:50:59.507 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:59.552 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:50:59.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:59.562 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:50:59.662 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:50:59.699 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:59.734 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:50:59.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:59.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:50:59.838 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:50:59.873 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:50:59.903 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:50:59.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:50:59.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:51:00.007 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:51:00.007 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:51:00.008 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:51:00.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:59 smithi106 ceph-mon[44423]: 1.1 scrub starts 2024-04-20T17:51:00.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:59 smithi106 ceph-mon[44423]: 1.1 scrub ok 2024-04-20T17:51:00.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:50:59 smithi106 ceph-mon[44423]: 9.d scrub starts 2024-04-20T17:51:00.036 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:59 smithi171 ceph-mon[44189]: 1.1 scrub starts 2024-04-20T17:51:00.036 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:59 smithi171 ceph-mon[44189]: 1.1 scrub ok 2024-04-20T17:51:00.036 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:50:59 smithi171 ceph-mon[44189]: 9.d scrub starts 2024-04-20T17:51:00.044 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:00.067 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:51:00.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:00.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:51:00.173 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:51:00.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:59 smithi005 ceph-mon[39562]: 1.1 scrub starts 2024-04-20T17:51:00.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:59 smithi005 ceph-mon[39562]: 1.1 scrub ok 2024-04-20T17:51:00.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:50:59 smithi005 ceph-mon[39562]: 9.d scrub starts 2024-04-20T17:51:00.228 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:00.261 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:51:00.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:00.269 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:51:00.369 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:51:00.410 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:00.762 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-20T17:51:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:00 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:51:01.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:00 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:51:01.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:00 smithi171 ceph-mon[44189]: 1.3 scrub starts 2024-04-20T17:51:01.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:00 smithi171 ceph-mon[44189]: osdmap e941: 12 total, 12 up, 12 in 2024-04-20T17:51:01.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:00 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3373017412' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:51:01.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:00 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:51:01.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:00 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:51:01.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:00 smithi171 ceph-mon[44189]: 1.3 scrub ok 2024-04-20T17:51:01.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:00 smithi171 ceph-mon[44189]: pgmap v2478: 27 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 25 active+clean; 8.8 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s; 7548/289614 objects misplaced (2.606%); 6.4 MiB/s, 102 objects/s recovering 2024-04-20T17:51:01.187 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:51:01.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:00 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:51:01.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:00 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:51:01.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:00 smithi005 ceph-mon[39562]: 1.3 scrub starts 2024-04-20T17:51:01.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:00 smithi005 ceph-mon[39562]: osdmap e941: 12 total, 12 up, 12 in 2024-04-20T17:51:01.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:00 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3373017412' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:51:01.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:00 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:51:01.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:00 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:51:01.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:00 smithi005 ceph-mon[39562]: 1.3 scrub ok 2024-04-20T17:51:01.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:00 smithi005 ceph-mon[39562]: pgmap v2478: 27 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 25 active+clean; 8.8 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s; 7548/289614 objects misplaced (2.606%); 6.4 MiB/s, 102 objects/s recovering 2024-04-20T17:51:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:00 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:51:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:00 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:51:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:00 smithi106 ceph-mon[44423]: 1.3 scrub starts 2024-04-20T17:51:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:00 smithi106 ceph-mon[44423]: osdmap e941: 12 total, 12 up, 12 in 2024-04-20T17:51:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:00 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3373017412' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:51:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:00 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:51:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:00 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:51:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:00 smithi106 ceph-mon[44423]: 1.3 scrub ok 2024-04-20T17:51:01.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:00 smithi106 ceph-mon[44423]: pgmap v2478: 27 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 25 active+clean; 8.8 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s; 7548/289614 objects misplaced (2.606%); 6.4 MiB/s, 102 objects/s recovering 2024-04-20T17:51:01.374 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:51:01.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:01.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:51:01.478 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:51:01.509 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:01.579 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:51:01.579 INFO:teuthology.orchestra.run.smithi005.stdout:{"fsid":"13c6e76c-ff37-11ee-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":3028,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":941,"num_osds":12,"num_up_osds":12,"osd_up_since":1713635059,"num_in_osds":12,"osd_in_since":1713635196,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":26},{"state_name":"active+remapped+backfilling","count":1}],"num_pgs":27,"num_pools":2,"num_objects":147508,"data_bytes":9667478072,"bytes_used":26872733696,"bytes_avail":1125067694080,"bytes_total":1151940427776,"misplaced_objects":7548,"misplaced_total":295016,"misplaced_ratio":0.025585053014073812,"recovering_objects_per_sec":116,"recovering_bytes_per_sec":7666113,"recovering_keys_per_sec":0,"num_objects_recovered":702,"num_bytes_recovered":46006272,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":1091},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.5:8443/"}},"servicemap":{"epoch":70,"modified":"2024-04-20T17:50:40.688583+0000","services":{}},"progress_events":{}} 2024-04-20T17:51:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:01 smithi171 ceph-mon[44189]: 9.d scrub ok 2024-04-20T17:51:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:01 smithi171 ceph-mon[44189]: osdmap e942: 12 total, 12 up, 12 in 2024-04-20T17:51:02.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:01 smithi171 ceph-mon[44189]: 9.e scrub starts 2024-04-20T17:51:02.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:01 smithi005 ceph-mon[39562]: 9.d scrub ok 2024-04-20T17:51:02.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:01 smithi005 ceph-mon[39562]: osdmap e942: 12 total, 12 up, 12 in 2024-04-20T17:51:02.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:01 smithi005 ceph-mon[39562]: 9.e scrub starts 2024-04-20T17:51:02.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:01 smithi106 ceph-mon[44423]: 9.d scrub ok 2024-04-20T17:51:02.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:01 smithi106 ceph-mon[44423]: osdmap e942: 12 total, 12 up, 12 in 2024-04-20T17:51:02.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:01 smithi106 ceph-mon[44423]: 9.e scrub starts 2024-04-20T17:51:02.809 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:51:02.851 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-20T17:51:02.852 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:51:02.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:02 smithi005 ceph-mon[39562]: pgmap v2480: 27 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 25 active+clean; 8.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 6618/286340 objects misplaced (2.311%); 11 MiB/s, 176 objects/s recovering 2024-04-20T17:51:03.070 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:51:03.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:03.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:51:03.177 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:51:03.219 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:02 smithi171 ceph-mon[44189]: pgmap v2480: 27 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 25 active+clean; 8.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 6618/286340 objects misplaced (2.311%); 11 MiB/s, 176 objects/s recovering 2024-04-20T17:51:03.223 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:03.258 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:51:03.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:02 smithi106 ceph-mon[44423]: pgmap v2480: 27 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 25 active+clean; 8.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 6618/286340 objects misplaced (2.311%); 11 MiB/s, 176 objects/s recovering 2024-04-20T17:51:03.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:03.264 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:51:03.264 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:51:03.264 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:51:03.364 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:51:03.399 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:03.554 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:51:03.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:03.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:51:03.658 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:51:03.693 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:03.725 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:51:03.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:03.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:51:03.829 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:51:03.872 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:03 smithi106 ceph-mon[44423]: 9.e scrub ok 2024-04-20T17:51:04.031 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:51:04.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:04.039 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:03 smithi005 ceph-mon[39562]: 9.e scrub ok 2024-04-20T17:51:04.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:51:04.104 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-20T17:51:04.140 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:51:04.177 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:04.198 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:51:04.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:04.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:51:04.303 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:51:04.334 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:04.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:03 smithi171 ceph-mon[44189]: 9.e scrub ok 2024-04-20T17:51:04.360 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:51:04.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:04.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:51:04.464 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:51:04.517 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:51:04.518 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:04.555 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:51:04.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:04.558 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:51:04.616 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:51:04.659 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:51:04.687 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:04.710 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:51:04.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:04.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:51:04.817 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:51:04.876 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:05.008 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:51:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:04 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2810576135' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:51:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:04 smithi106 ceph-mon[44423]: from='client.28253 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:51:05.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:04 smithi106 ceph-mon[44423]: pgmap v2481: 27 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 25 active+clean; 8.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 980 op/s; 6618/283556 objects misplaced (2.334%); 11 MiB/s, 172 objects/s recovering 2024-04-20T17:51:05.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:04 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2810576135' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:51:05.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:04 smithi171 ceph-mon[44189]: from='client.28253 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:51:05.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:04 smithi171 ceph-mon[44189]: pgmap v2481: 27 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 25 active+clean; 8.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 980 op/s; 6618/283556 objects misplaced (2.334%); 11 MiB/s, 172 objects/s recovering 2024-04-20T17:51:05.188 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:04 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2810576135' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:51:05.189 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:04 smithi005 ceph-mon[39562]: from='client.28253 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:51:05.189 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:04 smithi005 ceph-mon[39562]: pgmap v2481: 27 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 25 active+clean; 8.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 980 op/s; 6618/283556 objects misplaced (2.334%); 11 MiB/s, 172 objects/s recovering 2024-04-20T17:51:05.650 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:51:05.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:05.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:51:05.757 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:51:05.793 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:05.935 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:05 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:51:05.935 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:05 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:51:05.935 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:05 smithi005 ceph-mon[39562]: from='client.28253 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:51:05.935 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:05 smithi005 ceph-mon[39562]: osdmap e943: 12 total, 12 up, 12 in 2024-04-20T17:51:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:05 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:51:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:05 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:51:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:05 smithi106 ceph-mon[44423]: from='client.28253 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:51:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:05 smithi106 ceph-mon[44423]: osdmap e943: 12 total, 12 up, 12 in 2024-04-20T17:51:06.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:05 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:51:06.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:05 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:51:06.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:05 smithi171 ceph-mon[44189]: from='client.28253 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:51:06.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:05 smithi171 ceph-mon[44189]: osdmap e943: 12 total, 12 up, 12 in 2024-04-20T17:51:06.807 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:51:06.807 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:51:06.807 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:51:06.807 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:51:06.807 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:51:06.807 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:51:06.807 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:51:06.807 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:51:06.807 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:51:06.832 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:51:06.866 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:51:06.889 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:51:06.889 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:51:06.897 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:51:06.899 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:51:06.899 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:51:06.899 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:51:06.899 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:51:06.899 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:51:06.900 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:51:06.900 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:51:06.900 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:51:06.900 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:51:06.900 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:51:06.900 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:51:06.900 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:51:07.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:06 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2841203802' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:51:07.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:06 smithi005 ceph-mon[39562]: pgmap v2483: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 8.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 6618/277510 objects misplaced (2.385%); 6.2 MiB/s, 99 objects/s recovering 2024-04-20T17:51:07.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:07.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:06 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2841203802' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:51:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:06 smithi106 ceph-mon[44423]: pgmap v2483: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 8.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 6618/277510 objects misplaced (2.385%); 6.2 MiB/s, 99 objects/s recovering 2024-04-20T17:51:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:07.290 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:06 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2841203802' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:51:07.291 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:06 smithi171 ceph-mon[44189]: pgmap v2483: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 8.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 6618/277510 objects misplaced (2.385%); 6.2 MiB/s, 99 objects/s recovering 2024-04-20T17:51:07.291 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:07.291 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:07.610 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:51:07.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:07.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:51:07.720 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:51:07.742 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-20T17:51:07.751 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:07.772 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:51:07.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:07.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:51:07.876 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:51:07.917 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:07.947 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:51:07.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:07.951 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:51:07.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:07.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:07.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:07.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:07.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:07.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:07.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:07.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:07.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:07.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:07 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:08.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:08.012 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:07 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:08.052 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:51:08.096 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:08.127 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:51:08.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:08.130 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:51:08.231 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:51:08.268 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:08.269 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:08.269 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:08.269 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:08.269 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:08.269 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:08.269 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:08.269 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:08.270 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:08.270 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:07 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:08.270 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:08.296 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:51:08.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:08.307 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:51:08.409 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:51:08.448 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:08.486 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:51:08.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:08.492 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:51:08.593 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:51:08.641 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:08.990 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:51:08.991 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:51:08.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:08.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:51:09.032 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:08 smithi005 ceph-mon[39562]: pgmap v2484: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 8.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 5623/273390 objects misplaced (2.057%); 11 MiB/s, 181 objects/s recovering 2024-04-20T17:51:09.100 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:51:09.140 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:08 smithi106 ceph-mon[44423]: pgmap v2484: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 8.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 5623/273390 objects misplaced (2.057%); 11 MiB/s, 181 objects/s recovering 2024-04-20T17:51:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:08 smithi171 ceph-mon[44189]: pgmap v2484: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 8.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 5623/273390 objects misplaced (2.057%); 11 MiB/s, 181 objects/s recovering 2024-04-20T17:51:09.432 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:51:09.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:09.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:51:09.538 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:51:09.582 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:09.616 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:51:09.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:09.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:51:09.655 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:51:09.723 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:51:09.775 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:09.989 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:51:10.000 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:51:10.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:09 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:51:10.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:09 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:51:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:09 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:51:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:09 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:51:10.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:09 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:51:10.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:09 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:51:10.815 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:51:10.873 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:51:10.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:10.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:51:10.907 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":2485,"stamp":"2024-04-20T17:51:09.965251+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8892252728,"num_objects":135679,"num_object_clones":0,"num_object_copies":271358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5623,"num_objects_unfound":0,"num_objects_dirty":135679,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":197911,"num_write_kb":10674753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2940,"num_bytes_recovered":192610304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162418,"ondisk_log_size":162418,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":24833596,"kb_used_data":18090700,"kb_used_omap":404,"kb_used_meta":6742059,"kb_avail":1100108228,"statfs":{"total":1151940427776,"available":1126510825472,"internally_reserved":0,"allocated":18524876800,"data_stored":18520062821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":414014,"internal_metadata":6903869122},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,4,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-576847872,"num_objects":-8802,"num_object_clones":0,"num_object_copies":-17604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1925,"num_objects_unfound":0,"num_objects_dirty":-8802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8802,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1449,"num_bytes_recovered":94961664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.054059"},"pg_stats":[{"pgid":"9.7","version":"943'12098","reported_seq":12308,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-04-20T17:51:05.147188+0000","last_change":"2024-04-20T17:50:27.211659+0000","last_active":"2024-04-20T17:51:05.147188+0000","last_peered":"2024-04-20T17:51:05.147188+0000","last_clean":"2024-04-20T17:51:05.147188+0000","last_became_active":"2024-04-20T17:48:54.814508+0000","last_became_peered":"2024-04-20T17:48:54.814508+0000","last_unstale":"2024-04-20T17:51:05.147188+0000","last_undegraded":"2024-04-20T17:51:05.147188+0000","last_fullsized":"2024-04-20T17:51:05.147188+0000","mapping_epoch":894,"log_start":"903'2000","ondisk_log_start":"903'2000","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'9665","last_scrub_stamp":"2024-04-20T17:50:27.211421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:27.211421+0000","objects_scrubbed":9588,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:56.034157+0000","scrub_duration":2310,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561250304,"num_objects":8564,"num_object_clones":0,"num_object_copies":17128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":661184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"943'12207","reported_seq":12418,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-04-20T17:51:05.178745+0000","last_change":"2024-04-20T17:50:25.096410+0000","last_active":"2024-04-20T17:51:05.178745+0000","last_peered":"2024-04-20T17:51:05.178745+0000","last_clean":"2024-04-20T17:51:05.178745+0000","last_became_active":"2024-04-20T17:48:54.818033+0000","last_became_peered":"2024-04-20T17:48:54.818033+0000","last_unstale":"2024-04-20T17:51:05.178745+0000","last_undegraded":"2024-04-20T17:51:05.178745+0000","last_fullsized":"2024-04-20T17:51:05.178745+0000","mapping_epoch":894,"log_start":"903'2200","ondisk_log_start":"903'2200","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"929'9608","last_scrub_stamp":"2024-04-20T17:50:25.096273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:25.096273+0000","objects_scrubbed":9524,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:47.683348+0000","scrub_duration":2229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561840128,"num_objects":8573,"num_object_clones":0,"num_object_copies":17146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12207,"num_write_kb":664960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.5","version":"943'12235","reported_seq":12447,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-04-20T17:51:06.092371+0000","last_change":"2024-04-20T17:50:23.865871+0000","last_active":"2024-04-20T17:51:06.092371+0000","last_peered":"2024-04-20T17:51:06.092371+0000","last_clean":"2024-04-20T17:51:06.092371+0000","last_became_active":"2024-04-20T17:48:54.817749+0000","last_became_peered":"2024-04-20T17:48:54.817749+0000","last_unstale":"2024-04-20T17:51:06.092371+0000","last_undegraded":"2024-04-20T17:51:06.092371+0000","last_fullsized":"2024-04-20T17:51:06.092371+0000","mapping_epoch":894,"log_start":"903'2200","ondisk_log_start":"903'2200","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9546","last_scrub_stamp":"2024-04-20T17:50:23.865708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:23.865708+0000","objects_scrubbed":9473,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:24.406565+0000","scrub_duration":1985,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":557645824,"num_objects":8509,"num_object_clones":0,"num_object_copies":17018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12235,"num_write_kb":663808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"943'12149","reported_seq":12357,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-04-20T17:51:05.874284+0000","last_change":"2024-04-20T17:50:23.032137+0000","last_active":"2024-04-20T17:51:05.874284+0000","last_peered":"2024-04-20T17:51:05.874284+0000","last_clean":"2024-04-20T17:51:05.874284+0000","last_became_active":"2024-04-20T17:48:55.061199+0000","last_became_peered":"2024-04-20T17:48:55.061199+0000","last_unstale":"2024-04-20T17:51:05.874284+0000","last_undegraded":"2024-04-20T17:51:05.874284+0000","last_fullsized":"2024-04-20T17:51:05.874284+0000","mapping_epoch":894,"log_start":"903'2100","ondisk_log_start":"903'2100","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9404","last_scrub_stamp":"2024-04-20T17:50:23.031924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:23.031924+0000","objects_scrubbed":9366,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:30.444210+0000","scrub_duration":1573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561184768,"num_objects":8563,"num_object_clones":0,"num_object_copies":17126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12149,"num_write_kb":662784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"943'12235","reported_seq":18331,"reported_epoch":943,"state":"active+remapped+backfilling","last_fresh":"2024-04-20T17:51:07.184362+0000","last_change":"2024-04-20T17:50:46.276928+0000","last_active":"2024-04-20T17:51:07.184362+0000","last_peered":"2024-04-20T17:51:07.184362+0000","last_clean":"2024-04-20T17:50:43.748764+0000","last_became_active":"2024-04-20T17:50:46.159037+0000","last_became_peered":"2024-04-20T17:50:46.159037+0000","last_unstale":"2024-04-20T17:51:07.184362+0000","last_undegraded":"2024-04-20T17:51:07.184362+0000","last_fullsized":"2024-04-20T17:51:07.184362+0000","mapping_epoch":935,"log_start":"903'2200","ondisk_log_start":"903'2200","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'9622","last_scrub_stamp":"2024-04-20T17:50:27.118695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:27.118695+0000","objects_scrubbed":9558,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":546635776,"num_objects":8341,"num_object_clones":0,"num_object_copies":16682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5623,"num_objects_unfound":0,"num_objects_dirty":8341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12235,"num_write_kb":658432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2940,"num_bytes_recovered":192610304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[9,3],"avail_no_missing":["9","3"],"object_location_counts":[{"shards":"3,9","objects":8341}],"blocked_by":[],"up_primary":8,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.e","version":"943'12558","reported_seq":12763,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-04-20T17:51:09.818279+0000","last_change":"2024-04-20T17:51:03.098201+0000","last_active":"2024-04-20T17:51:09.818279+0000","last_peered":"2024-04-20T17:51:09.818279+0000","last_clean":"2024-04-20T17:51:09.818279+0000","last_became_active":"2024-04-20T17:48:54.815159+0000","last_became_peered":"2024-04-20T17:48:54.815159+0000","last_unstale":"2024-04-20T17:51:09.818279+0000","last_undegraded":"2024-04-20T17:51:09.818279+0000","last_fullsized":"2024-04-20T17:51:09.818279+0000","mapping_epoch":894,"log_start":"904'2500","ondisk_log_start":"904'2500","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"942'12102","last_scrub_stamp":"2024-04-20T17:51:03.098072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:03.098072+0000","objects_scrubbed":8857,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:30.248766+0000","scrub_duration":2003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":544866304,"num_objects":8314,"num_object_clones":0,"num_object_copies":16628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":667904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"15'137","reported_seq":937,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-04-20T17:51:04.958236+0000","last_change":"2024-04-20T17:50:57.911454+0000","last_active":"2024-04-20T17:51:04.958236+0000","last_peered":"2024-04-20T17:51:04.958236+0000","last_clean":"2024-04-20T17:51:04.958236+0000","last_became_active":"2024-04-20T17:45:32.670941+0000","last_became_peered":"2024-04-20T17:45:32.670941+0000","last_unstale":"2024-04-20T17:51:04.958236+0000","last_undegraded":"2024-04-20T17:51:04.958236+0000","last_fullsized":"2024-04-20T17:51:04.958236+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:57.911336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:57.911336+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:02.057237+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"943'12286","reported_seq":12495,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-04-20T17:51:05.627914+0000","last_change":"2024-04-20T17:50:23.641303+0000","last_active":"2024-04-20T17:51:05.627914+0000","last_peered":"2024-04-20T17:51:05.627914+0000","last_clean":"2024-04-20T17:51:05.627914+0000","last_became_active":"2024-04-20T17:48:55.061161+0000","last_became_peered":"2024-04-20T17:48:55.061161+0000","last_unstale":"2024-04-20T17:51:05.627914+0000","last_undegraded":"2024-04-20T17:51:05.627914+0000","last_fullsized":"2024-04-20T17:51:05.627914+0000","mapping_epoch":894,"log_start":"903'2200","ondisk_log_start":"903'2200","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9495","last_scrub_stamp":"2024-04-20T17:50:23.641164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:23.641164+0000","objects_scrubbed":9459,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:29.146891+0000","scrub_duration":1363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":562954240,"num_objects":8590,"num_object_clones":0,"num_object_copies":17180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12286,"num_write_kb":668032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"15'137","reported_seq":935,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-04-20T17:51:04.990704+0000","last_change":"2024-04-20T17:50:25.724423+0000","last_active":"2024-04-20T17:51:04.990704+0000","last_peered":"2024-04-20T17:51:04.990704+0000","last_clean":"2024-04-20T17:51:04.990704+0000","last_became_active":"2024-04-20T17:45:29.817528+0000","last_became_peered":"2024-04-20T17:45:29.817528+0000","last_unstale":"2024-04-20T17:51:04.990704+0000","last_undegraded":"2024-04-20T17:51:04.990704+0000","last_fullsized":"2024-04-20T17:51:04.990704+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:25.724296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:25.724296+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:37.439709+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"943'12638","reported_seq":12853,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-04-20T17:51:08.616959+0000","last_change":"2024-04-20T17:50:29.927851+0000","last_active":"2024-04-20T17:51:08.616959+0000","last_peered":"2024-04-20T17:51:08.616959+0000","last_clean":"2024-04-20T17:51:08.616959+0000","last_became_active":"2024-04-20T17:48:54.817445+0000","last_became_peered":"2024-04-20T17:48:54.817445+0000","last_unstale":"2024-04-20T17:51:08.616959+0000","last_undegraded":"2024-04-20T17:51:08.616959+0000","last_fullsized":"2024-04-20T17:51:08.616959+0000","mapping_epoch":894,"log_start":"904'2600","ondisk_log_start":"904'2600","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'10062","last_scrub_stamp":"2024-04-20T17:50:29.927762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:29.927762+0000","objects_scrubbed":9978,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:58.752509+0000","scrub_duration":2071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":549191680,"num_objects":8380,"num_object_clones":0,"num_object_copies":16760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12638,"num_write_kb":672576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"15'137","reported_seq":961,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-04-20T17:51:04.952126+0000","last_change":"2024-04-20T17:50:20.138119+0000","last_active":"2024-04-20T17:51:04.952126+0000","last_peered":"2024-04-20T17:51:04.952126+0000","last_clean":"2024-04-20T17:51:04.952126+0000","last_became_active":"2024-04-20T17:45:29.827577+0000","last_became_peered":"2024-04-20T17:45:29.827577+0000","last_unstale":"2024-04-20T17:51:04.952126+0000","last_undegraded":"2024-04-20T17:51:04.952126+0000","last_fullsized":"2024-04-20T17:51:04.952126+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:20.137983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:20.137983+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:30.668575+0000","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"943'12494","reported_seq":12767,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-04-20T17:51:09.771048+0000","last_change":"2024-04-20T17:51:00.541301+0000","last_active":"2024-04-20T17:51:09.771048+0000","last_peered":"2024-04-20T17:51:09.771048+0000","last_clean":"2024-04-20T17:51:09.771048+0000","last_became_active":"2024-04-20T17:48:55.061208+0000","last_became_peered":"2024-04-20T17:48:55.061208+0000","last_unstale":"2024-04-20T17:51:09.771048+0000","last_undegraded":"2024-04-20T17:51:09.771048+0000","last_fullsized":"2024-04-20T17:51:09.771048+0000","mapping_epoch":894,"log_start":"904'2400","ondisk_log_start":"904'2400","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"941'11888","last_scrub_stamp":"2024-04-20T17:51:00.541192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:00.541192+0000","objects_scrubbed":8973,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:11.066367+0000","scrub_duration":1502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":544538648,"num_objects":8310,"num_object_clones":0,"num_object_copies":16620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8310,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12494,"num_write_kb":665665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"15'137","reported_seq":929,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-04-20T17:51:04.954753+0000","last_change":"2024-04-20T17:50:18.122040+0000","last_active":"2024-04-20T17:51:04.954753+0000","last_peered":"2024-04-20T17:51:04.954753+0000","last_clean":"2024-04-20T17:51:04.954753+0000","last_became_active":"2024-04-20T17:45:32.666523+0000","last_became_peered":"2024-04-20T17:45:32.666523+0000","last_unstale":"2024-04-20T17:51:04.954753+0000","last_undegraded":"2024-04-20T17:51:04.954753+0000","last_fullsized":"2024-04-20T17:51:04.954753+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:18.121912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:18.121912+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:25.993620+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.a","version":"943'12357","reported_seq":12569,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-04-20T17:51:05.632637+0000","last_change":"2024-04-20T17:50:22.180642+0000","last_active":"2024-04-20T17:51:05.632637+0000","last_peered":"2024-04-20T17:51:05.632637+0000","last_clean":"2024-04-20T17:51:05.632637+0000","last_became_active":"2024-04-20T17:48:55.061180+0000","last_became_peered":"2024-04-20T17:48:55.061180+0000","last_unstale":"2024-04-20T17:51:05.632637+0000","last_undegraded":"2024-04-20T17:51:05.632637+0000","last_fullsized":"2024-04-20T17:51:05.632637+0000","mapping_epoch":894,"log_start":"903'2300","ondisk_log_start":"903'2300","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9574","last_scrub_stamp":"2024-04-20T17:50:22.180501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:22.180501+0000","objects_scrubbed":9507,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:51.128747+0000","scrub_duration":1824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":564985856,"num_objects":8621,"num_object_clones":0,"num_object_copies":17242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12357,"num_write_kb":671296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"15'137","reported_seq":932,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-04-20T17:51:04.951749+0000","last_change":"2024-04-20T17:50:31.133012+0000","last_active":"2024-04-20T17:51:04.951749+0000","last_peered":"2024-04-20T17:51:04.951749+0000","last_clean":"2024-04-20T17:51:04.951749+0000","last_became_active":"2024-04-20T17:46:40.831961+0000","last_became_peered":"2024-04-20T17:46:40.831961+0000","last_unstale":"2024-04-20T17:51:04.951749+0000","last_undegraded":"2024-04-20T17:51:04.951749+0000","last_fullsized":"2024-04-20T17:51:04.951749+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:31.132879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:31.132879+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:31.822631+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"943'12376","reported_seq":12582,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-04-20T17:51:08.988229+0000","last_change":"2024-04-20T17:50:19.657299+0000","last_active":"2024-04-20T17:51:08.988229+0000","last_peered":"2024-04-20T17:51:08.988229+0000","last_clean":"2024-04-20T17:51:08.988229+0000","last_became_active":"2024-04-20T17:48:54.816644+0000","last_became_peered":"2024-04-20T17:48:54.816644+0000","last_unstale":"2024-04-20T17:51:08.988229+0000","last_undegraded":"2024-04-20T17:51:08.988229+0000","last_fullsized":"2024-04-20T17:51:08.988229+0000","mapping_epoch":894,"log_start":"903'2300","ondisk_log_start":"903'2300","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"927'9204","last_scrub_stamp":"2024-04-20T17:50:19.657194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:19.657194+0000","objects_scrubbed":9124,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:25.188599+0000","scrub_duration":2257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":549847040,"num_objects":8390,"num_object_clones":0,"num_object_copies":16780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":664512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"15'137","reported_seq":730,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-04-20T17:51:04.948570+0000","last_change":"2024-04-20T17:51:00.057261+0000","last_active":"2024-04-20T17:51:04.948570+0000","last_peered":"2024-04-20T17:51:04.948570+0000","last_clean":"2024-04-20T17:51:04.948570+0000","last_became_active":"2024-04-20T17:49:46.824559+0000","last_became_peered":"2024-04-20T17:49:46.824559+0000","last_unstale":"2024-04-20T17:51:04.948570+0000","last_undegraded":"2024-04-20T17:51:04.948570+0000","last_fullsized":"2024-04-20T17:51:04.948570+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:00.057156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:00.057156+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:08.337431+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"943'12483","reported_seq":12698,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-04-20T17:51:05.168600+0000","last_change":"2024-04-20T17:50:28.283117+0000","last_active":"2024-04-20T17:51:05.168600+0000","last_peered":"2024-04-20T17:51:05.168600+0000","last_clean":"2024-04-20T17:51:05.168600+0000","last_became_active":"2024-04-20T17:48:54.813199+0000","last_became_peered":"2024-04-20T17:48:54.813199+0000","last_unstale":"2024-04-20T17:51:05.168600+0000","last_undegraded":"2024-04-20T17:51:05.168600+0000","last_fullsized":"2024-04-20T17:51:05.168600+0000","mapping_epoch":894,"log_start":"903'2400","ondisk_log_start":"903'2400","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'10025","last_scrub_stamp":"2024-04-20T17:50:28.282957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:28.282957+0000","objects_scrubbed":9963,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:48.575002+0000","scrub_duration":2339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":576651264,"num_objects":8799,"num_object_clones":0,"num_object_copies":17598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12483,"num_write_kb":681024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"15'137","reported_seq":937,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-04-20T17:51:04.954947+0000","last_change":"2024-04-20T17:50:58.932166+0000","last_active":"2024-04-20T17:51:04.954947+0000","last_peered":"2024-04-20T17:51:04.954947+0000","last_clean":"2024-04-20T17:51:04.954947+0000","last_became_active":"2024-04-20T17:45:32.782706+0000","last_became_peered":"2024-04-20T17:45:32.782706+0000","last_unstale":"2024-04-20T17:51:04.954947+0000","last_undegraded":"2024-04-20T17:51:04.954947+0000","last_fullsized":"2024-04-20T17:51:04.954947+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:58.932046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:58.932046+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:23.591238+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.2","version":"943'12476","reported_seq":12683,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-04-20T17:51:08.604783+0000","last_change":"2024-04-20T17:50:22.440005+0000","last_active":"2024-04-20T17:51:08.604783+0000","last_peered":"2024-04-20T17:51:08.604783+0000","last_clean":"2024-04-20T17:51:08.604783+0000","last_became_active":"2024-04-20T17:48:55.060965+0000","last_became_peered":"2024-04-20T17:48:55.060965+0000","last_unstale":"2024-04-20T17:51:08.604783+0000","last_undegraded":"2024-04-20T17:51:08.604783+0000","last_fullsized":"2024-04-20T17:51:08.604783+0000","mapping_epoch":894,"log_start":"903'2400","ondisk_log_start":"903'2400","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9408","last_scrub_stamp":"2024-04-20T17:50:22.439841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:22.439841+0000","objects_scrubbed":9355,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:23.199554+0000","scrub_duration":1641,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":544210944,"num_objects":8304,"num_object_clones":0,"num_object_copies":16608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12476,"num_write_kb":664960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"15'137","reported_seq":915,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-04-20T17:51:04.959608+0000","last_change":"2024-04-20T17:49:48.210622+0000","last_active":"2024-04-20T17:51:04.959608+0000","last_peered":"2024-04-20T17:51:04.959608+0000","last_clean":"2024-04-20T17:51:04.959608+0000","last_became_active":"2024-04-20T17:45:32.669451+0000","last_became_peered":"2024-04-20T17:45:32.669451+0000","last_unstale":"2024-04-20T17:51:04.959608+0000","last_undegraded":"2024-04-20T17:51:04.959608+0000","last_fullsized":"2024-04-20T17:51:04.959608+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:49:48.210506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:49:48.210506+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:05.387619+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"943'12259","reported_seq":12474,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-04-20T17:51:06.656776+0000","last_change":"2024-04-20T17:50:30.239298+0000","last_active":"2024-04-20T17:51:06.656776+0000","last_peered":"2024-04-20T17:51:06.656776+0000","last_clean":"2024-04-20T17:51:06.656776+0000","last_became_active":"2024-04-20T17:48:55.061200+0000","last_became_peered":"2024-04-20T17:48:55.061200+0000","last_unstale":"2024-04-20T17:51:06.656776+0000","last_undegraded":"2024-04-20T17:51:06.656776+0000","last_fullsized":"2024-04-20T17:51:06.656776+0000","mapping_epoch":894,"log_start":"903'2200","ondisk_log_start":"903'2200","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'9940","last_scrub_stamp":"2024-04-20T17:50:30.239131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:30.239131+0000","objects_scrubbed":9887,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:36.933027+0000","scrub_duration":1672,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":556466176,"num_objects":8491,"num_object_clones":0,"num_object_copies":16982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12259,"num_write_kb":664000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"15'137","reported_seq":930,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-04-20T17:51:05.015618+0000","last_change":"2024-04-20T17:50:23.398315+0000","last_active":"2024-04-20T17:51:05.015618+0000","last_peered":"2024-04-20T17:51:05.015618+0000","last_clean":"2024-04-20T17:51:05.015618+0000","last_became_active":"2024-04-20T17:45:32.644584+0000","last_became_peered":"2024-04-20T17:45:32.644584+0000","last_unstale":"2024-04-20T17:51:05.015618+0000","last_undegraded":"2024-04-20T17:51:05.015618+0000","last_fullsized":"2024-04-20T17:51:05.015618+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:23.398142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:23.398142+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:31.965635+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"943'12555","reported_seq":12770,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-04-20T17:51:08.620478+0000","last_change":"2024-04-20T17:50:30.712810+0000","last_active":"2024-04-20T17:51:08.620478+0000","last_peered":"2024-04-20T17:51:08.620478+0000","last_clean":"2024-04-20T17:51:08.620478+0000","last_became_active":"2024-04-20T17:48:54.813062+0000","last_became_peered":"2024-04-20T17:48:54.813062+0000","last_unstale":"2024-04-20T17:51:08.620478+0000","last_undegraded":"2024-04-20T17:51:08.620478+0000","last_fullsized":"2024-04-20T17:51:08.620478+0000","mapping_epoch":894,"log_start":"904'2500","ondisk_log_start":"904'2500","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'10026","last_scrub_stamp":"2024-04-20T17:50:30.712648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:30.712648+0000","objects_scrubbed":9967,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:58.995403+0000","scrub_duration":1871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":548208640,"num_objects":8365,"num_object_clones":0,"num_object_copies":16730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12555,"num_write_kb":669440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"15'137","reported_seq":878,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-04-20T17:51:04.951749+0000","last_change":"2024-04-20T17:50:33.137929+0000","last_active":"2024-04-20T17:51:04.951749+0000","last_peered":"2024-04-20T17:51:04.951749+0000","last_clean":"2024-04-20T17:51:04.951749+0000","last_became_active":"2024-04-20T17:46:40.881713+0000","last_became_peered":"2024-04-20T17:46:40.881713+0000","last_unstale":"2024-04-20T17:51:04.951749+0000","last_undegraded":"2024-04-20T17:51:04.951749+0000","last_fullsized":"2024-04-20T17:51:04.951749+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:33.137831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:33.137831+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:36.471855+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.8","version":"943'12505","reported_seq":12713,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-04-20T17:51:07.161515+0000","last_change":"2024-04-20T17:50:19.940419+0000","last_active":"2024-04-20T17:51:07.161515+0000","last_peered":"2024-04-20T17:51:07.161515+0000","last_clean":"2024-04-20T17:51:07.161515+0000","last_became_active":"2024-04-20T17:48:54.813368+0000","last_became_peered":"2024-04-20T17:48:54.813368+0000","last_unstale":"2024-04-20T17:51:07.161515+0000","last_undegraded":"2024-04-20T17:51:07.161515+0000","last_fullsized":"2024-04-20T17:51:07.161515+0000","mapping_epoch":894,"log_start":"904'2500","ondisk_log_start":"904'2500","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"927'9391","last_scrub_stamp":"2024-04-20T17:50:19.940199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:19.940199+0000","objects_scrubbed":9334,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:34.165962+0000","scrub_duration":1977,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561184768,"num_objects":8563,"num_object_clones":0,"num_object_copies":17126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12505,"num_write_kb":674176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":876,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-04-20T17:51:04.951749+0000","last_change":"2024-04-20T17:50:21.111183+0000","last_active":"2024-04-20T17:51:04.951749+0000","last_peered":"2024-04-20T17:51:04.951749+0000","last_clean":"2024-04-20T17:51:04.951749+0000","last_became_active":"2024-04-20T17:46:40.851683+0000","last_became_peered":"2024-04-20T17:46:40.851683+0000","last_unstale":"2024-04-20T17:51:04.951749+0000","last_undegraded":"2024-04-20T17:51:04.951749+0000","last_fullsized":"2024-04-20T17:51:04.951749+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:21.111059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:21.111059+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:45.968902+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":8891662360,"num_objects":135677,"num_object_clones":0,"num_object_copies":271354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5623,"num_objects_unfound":0,"num_objects_dirty":135677,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":197911,"num_write_kb":10674753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2940,"num_bytes_recovered":192610304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17936097280,"data_stored":17936089136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160911,"ondisk_log_size":160911,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},"log_size":1507,"ondisk_log_size":1507,"up":22,"acting":22,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991778,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2597580,"kb_used_data":1668160,"kb_used_omap":1,"kb_used_meta":929406,"kb_avail":91147572,"statfs":{"total":95995035648,"available":93335113728,"internally_reserved":0,"allocated":1708195840,"data_stored":1707795443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":951712196},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253417,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3607740,"kb_used_data":2774132,"kb_used_omap":1,"kb_used_meta":833598,"kb_avail":90137412,"statfs":{"total":95995035648,"available":92300709888,"internally_reserved":0,"allocated":2840711168,"data_stored":2840316796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":853604800},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613166,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2374792,"kb_used_data":1631168,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":91370360,"statfs":{"total":95995035648,"available":93563248640,"internally_reserved":0,"allocated":1670316032,"data_stored":1669915635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":761199040},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809397,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1160904,"kb_used_data":724352,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":92584248,"statfs":{"total":95995035648,"available":94806269952,"internally_reserved":0,"allocated":741736448,"data_stored":741336051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":799,"seq":3431678869587,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1736716,"kb_used_data":1602052,"kb_used_omap":11,"kb_used_meta":134580,"kb_avail":92008436,"statfs":{"total":95995035648,"available":94216638464,"internally_reserved":0,"allocated":1640501248,"data_stored":1640096787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11990,"internal_metadata":137810218},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":535,"seq":2297807503626,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":590792,"kb_used_data":549056,"kb_used_omap":170,"kb_used_meta":41557,"kb_avail":93154360,"statfs":{"total":95995035648,"available":95390064640,"internally_reserved":0,"allocated":562233344,"data_stored":561832947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":174628,"internal_metadata":42554844},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":659,"seq":2830383448249,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3566476,"kb_used_data":3210628,"kb_used_omap":4,"kb_used_meta":355835,"kb_avail":90178676,"statfs":{"total":95995035648,"available":92342964224,"internally_reserved":0,"allocated":3287683072,"data_stored":3287278603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4410,"internal_metadata":364375750},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738971,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":744140,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":93001012,"statfs":{"total":95995035648,"available":95233036288,"internally_reserved":0,"allocated":786432,"data_stored":386035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":655892,"kb_used_data":236808,"kb_used_omap":202,"kb_used_meta":418869,"kb_avail":93089260,"statfs":{"total":95995035648,"available":95323402240,"internally_reserved":0,"allocated":242491392,"data_stored":242087443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":428922327},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653310,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2728136,"kb_used_data":2145984,"kb_used_omap":4,"kb_used_meta":582139,"kb_avail":91017016,"statfs":{"total":95995035648,"available":93201424384,"internally_reserved":0,"allocated":2197487616,"data_stored":2197087219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":596111047},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986758,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1735496,"kb_used_data":1100416,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":92009656,"statfs":{"total":95995035648,"available":94217887744,"internally_reserved":0,"allocated":1126825984,"data_stored":1126425587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790528,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3334932,"kb_used_data":2447176,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":90410220,"statfs":{"total":95995035648,"available":92580065280,"internally_reserved":0,"allocated":2505908224,"data_stored":2505504275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":909048261},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3269922816,"data_stored":3269918744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2181234688,"data_stored":2181234688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1122893824,"data_stored":1122893824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2246901760,"data_stored":2246901760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":556335104,"data_stored":556335104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1627721728,"data_stored":1627717656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":742326272,"data_stored":742326272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1662976000,"data_stored":1662976000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2831679488,"data_stored":2831679488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1694105600,"data_stored":1694105600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:51:10.980 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:51:11.028 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:11.054 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:51:11.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:11.061 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:51:11.162 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:51:11.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:10 smithi005 ceph-mon[39562]: osdmap e944: 12 total, 12 up, 12 in 2024-04-20T17:51:11.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:10 smithi005 ceph-mon[39562]: from='client.27904 v1:172.21.15.5:0/2231848544' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:51:11.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:10 smithi005 ceph-mon[39562]: pgmap v2486: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 8.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 5623/267682 objects misplaced (2.101%); 6.0 MiB/s, 96 objects/s recovering 2024-04-20T17:51:11.226 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:11.251 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:51:11.258 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:10 smithi171 ceph-mon[44189]: osdmap e944: 12 total, 12 up, 12 in 2024-04-20T17:51:11.258 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:10 smithi171 ceph-mon[44189]: from='client.27904 v1:172.21.15.5:0/2231848544' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:51:11.259 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:10 smithi171 ceph-mon[44189]: pgmap v2486: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 8.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 5623/267682 objects misplaced (2.101%); 6.0 MiB/s, 96 objects/s recovering 2024-04-20T17:51:11.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:11.260 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:51:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:10 smithi106 ceph-mon[44423]: osdmap e944: 12 total, 12 up, 12 in 2024-04-20T17:51:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:10 smithi106 ceph-mon[44423]: from='client.27904 v1:172.21.15.5:0/2231848544' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:51:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:10 smithi106 ceph-mon[44423]: pgmap v2486: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 8.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 5623/267682 objects misplaced (2.101%); 6.0 MiB/s, 96 objects/s recovering 2024-04-20T17:51:11.361 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:51:11.415 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:11.439 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:51:11.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:11.443 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:51:11.488 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:51:11.544 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:51:11.587 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:11.615 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:51:11.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:11.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:51:11.723 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:51:11.752 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:11.779 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:51:11.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:11.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:51:11.883 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:51:11.926 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:11.967 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:51:11.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:11.971 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:51:12.074 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:51:12.110 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:12.132 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:51:12.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:12.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:51:12.236 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:51:12.269 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:12.300 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:51:12.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:12.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:51:12.406 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:51:12.460 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:12.501 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:51:12.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:12.513 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:51:12.613 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:51:12.642 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:12.668 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:51:12.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:12.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:51:12.774 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:51:12.804 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:12.830 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:51:12.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:12.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:51:12.935 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:51:12.988 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:13.145 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:51:13.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:13.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:51:13.256 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:51:13.296 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:13.910 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:51:14.010 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:51:14.011 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:51:14.011 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-04-20T17:51:14.011 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:51:14.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:13 smithi171 ceph-mon[44189]: pgmap v2487: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 8.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 4612/264148 objects misplaced (1.746%); 12 MiB/s, 193 objects/s recovering 2024-04-20T17:51:14.210 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:51:14.247 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:13 smithi005 ceph-mon[39562]: pgmap v2487: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 8.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 4612/264148 objects misplaced (1.746%); 12 MiB/s, 193 objects/s recovering 2024-04-20T17:51:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:13 smithi106 ceph-mon[44423]: pgmap v2487: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 8.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 4612/264148 objects misplaced (1.746%); 12 MiB/s, 193 objects/s recovering 2024-04-20T17:51:14.833 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-20T17:51:15.105 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:51:15.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:15.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:51:15.216 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:51:15.260 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:15.835 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:51:15.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:15.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:51:15.940 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:51:15.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:15 smithi005 ceph-mon[39562]: pgmap v2488: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 8.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 4612/261432 objects misplaced (1.764%); 10 MiB/s, 159 objects/s recovering 2024-04-20T17:51:15.987 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:16.019 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:51:16.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:16.023 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:51:16.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:15 smithi171 ceph-mon[44189]: pgmap v2488: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 8.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 4612/261432 objects misplaced (1.764%); 10 MiB/s, 159 objects/s recovering 2024-04-20T17:51:16.124 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:51:16.157 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:15 smithi106 ceph-mon[44423]: pgmap v2488: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 8.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 4612/261432 objects misplaced (1.764%); 10 MiB/s, 159 objects/s recovering 2024-04-20T17:51:16.162 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:16.189 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:51:16.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:16.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:51:16.296 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:51:16.324 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:16.345 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:51:16.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:16.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:51:16.449 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:51:16.486 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:16.572 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:51:16.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:16.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:51:16.575 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:51:16.676 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:51:16.710 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:16.791 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:16 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1476813229' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:51:16.792 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:16 smithi005 ceph-mon[39562]: from='client.28271 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:51:16.811 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:51:17.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:16 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1476813229' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:51:17.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:16 smithi171 ceph-mon[44189]: from='client.28271 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:51:17.164 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:51:17.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:17.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:51:17.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:16 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1476813229' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:51:17.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:16 smithi106 ceph-mon[44423]: from='client.28271 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:51:17.268 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:51:17.307 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:17.394 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:51:17.395 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:51:17.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:17.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:51:17.507 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:51:17.543 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:17.574 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:51:17.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:17.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:51:17.680 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:51:17.717 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:18.068 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:17 smithi005 ceph-mon[39562]: pgmap v2489: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 7.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 4612/255122 objects misplaced (1.808%); 9.7 MiB/s, 155 objects/s recovering 2024-04-20T17:51:18.069 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:17 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:51:18.069 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:17 smithi005 ceph-mon[39562]: from='client.28271 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:51:18.069 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:17 smithi005 ceph-mon[39562]: osdmap e945: 12 total, 12 up, 12 in 2024-04-20T17:51:18.072 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:51:18.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:18.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:51:18.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:17 smithi171 ceph-mon[44189]: pgmap v2489: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 7.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 4612/255122 objects misplaced (1.808%); 9.7 MiB/s, 155 objects/s recovering 2024-04-20T17:51:18.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:17 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:51:18.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:17 smithi171 ceph-mon[44189]: from='client.28271 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:51:18.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:17 smithi171 ceph-mon[44189]: osdmap e945: 12 total, 12 up, 12 in 2024-04-20T17:51:18.181 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:51:18.221 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:17 smithi106 ceph-mon[44423]: pgmap v2489: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 7.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 4612/255122 objects misplaced (1.808%); 9.7 MiB/s, 155 objects/s recovering 2024-04-20T17:51:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:17 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:51:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:17 smithi106 ceph-mon[44423]: from='client.28271 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:51:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:17 smithi106 ceph-mon[44423]: osdmap e945: 12 total, 12 up, 12 in 2024-04-20T17:51:19.060 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:51:19.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:19.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:51:19.096 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:51:19.128 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:51:19.159 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:51:19.167 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:51:19.185 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:51:19.217 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:51:19.229 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:19.242 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:51:19.264 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:51:19.270 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:51:19.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:19.281 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:51:19.287 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:51:19.315 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:51:19.341 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:51:19.362 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:51:19.382 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:51:19.401 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:51:19.402 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:51:19.403 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:51:19.403 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:51:19.403 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:51:19.403 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:51:19.403 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:51:19.403 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:51:19.403 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:51:19.403 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:51:19.403 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:51:19.403 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:51:19.403 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:51:19.442 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:19.470 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:51:19.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:19.477 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:51:19.578 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:51:19.626 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:19.651 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:51:19.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:19.655 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:51:19.756 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:51:19.806 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:19 smithi171 ceph-mon[44189]: pgmap v2491: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 7.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 3662/250898 objects misplaced (1.460%); 11 MiB/s, 179 objects/s recovering 2024-04-20T17:51:19.807 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:19 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1601045047' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:51:19.807 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:19.807 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:19.807 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:19.807 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:19 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:19.820 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:19.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:19.855 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:51:19.856 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:51:19.956 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:51:20.020 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:20.051 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:51:20.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:20.058 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:51:20.111 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:51:20.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:19 smithi005 ceph-mon[39562]: pgmap v2491: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 7.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 3662/250898 objects misplaced (1.460%); 11 MiB/s, 179 objects/s recovering 2024-04-20T17:51:20.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:19 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1601045047' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:51:20.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:20.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:20.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:20.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:19 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:20.159 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:51:20.190 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:19 smithi106 ceph-mon[44423]: pgmap v2491: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 7.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 3662/250898 objects misplaced (1.460%); 11 MiB/s, 179 objects/s recovering 2024-04-20T17:51:20.190 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:19 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1601045047' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:51:20.190 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:20.190 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:20.190 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:20.191 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:19 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:20.191 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:20.216 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:51:20.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:20.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:51:20.323 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:51:20.368 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:21.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:21.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:21.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:21.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:21.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:21.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:21.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:21.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:20 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:21.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:21.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:21.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:21.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:21.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:21.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:21.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:21.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:20 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:20 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:21.764 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:51:21.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:21.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:51:21.871 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:51:21.903 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:21.933 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:51:21.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:21.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:51:22.019 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:21 smithi005 ceph-mon[39562]: pgmap v2492: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 7.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 3662/245114 objects misplaced (1.494%); 9.8 MiB/s, 156 objects/s recovering 2024-04-20T17:51:22.041 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:51:22.086 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:22.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:21 smithi171 ceph-mon[44189]: pgmap v2492: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 7.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 3662/245114 objects misplaced (1.494%); 9.8 MiB/s, 156 objects/s recovering 2024-04-20T17:51:22.120 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:51:22.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:22.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:51:22.247 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:51:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:21 smithi106 ceph-mon[44423]: pgmap v2492: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 7.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 3662/245114 objects misplaced (1.494%); 9.8 MiB/s, 156 objects/s recovering 2024-04-20T17:51:22.296 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:22.334 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:51:22.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:22.338 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:51:22.439 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:51:22.471 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:22.515 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:51:22.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:22.522 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:51:22.623 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:51:22.660 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:22.690 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:51:22.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:22.694 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:51:22.794 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:51:22.823 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:22.849 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:51:22.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:22.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:51:22.953 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:51:22.996 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:23.027 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:51:23.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:23.032 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:51:23.101 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:51:23.105 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:51:23.132 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:51:23.177 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:23.343 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:51:23.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:23.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:51:23.426 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:51:23.426 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:51:23.450 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:51:23.486 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:23.510 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:51:23.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:23.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:51:23.616 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:51:23.652 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:23.692 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:51:23.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:23.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:51:23.802 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:51:23.849 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:23.885 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:51:23.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:23.889 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:51:23.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:23 smithi005 ceph-mon[39562]: pgmap v2493: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 7.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 2762/241648 objects misplaced (1.143%); 9.6 MiB/s, 154 objects/s recovering 2024-04-20T17:51:23.989 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:51:24.019 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:24.040 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:51:24.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:24.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:51:24.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:23 smithi171 ceph-mon[44189]: pgmap v2493: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 7.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 2762/241648 objects misplaced (1.143%); 9.6 MiB/s, 154 objects/s recovering 2024-04-20T17:51:24.133 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":2493,"stamp":"2024-04-20T17:51:22.696957+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7918715448,"num_objects":120824,"num_object_clones":0,"num_object_copies":241648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2762,"num_objects_unfound":0,"num_objects_dirty":120824,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":212766,"num_write_kb":10674753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5263,"num_bytes_recovered":344850432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162273,"ondisk_log_size":162273,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":22993340,"kb_used_data":16232664,"kb_used_omap":404,"kb_used_meta":6760107,"kb_avail":1101948484,"statfs":{"total":1151940427776,"available":1128395247616,"internally_reserved":0,"allocated":16622247936,"data_stored":16618042284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":414014,"internal_metadata":6922350274},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,12,9],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-737280000,"num_objects":-11250,"num_object_clones":0,"num_object_copies":-22500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1850,"num_objects_unfound":0,"num_objects_dirty":-11250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11250,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1541,"num_bytes_recovered":100990976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002086"},"pg_stats":[{"pgid":"9.7","version":"945'13168","reported_seq":13383,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-04-20T17:51:20.195044+0000","last_change":"2024-04-20T17:50:27.211659+0000","last_active":"2024-04-20T17:51:20.195044+0000","last_peered":"2024-04-20T17:51:20.195044+0000","last_clean":"2024-04-20T17:51:20.195044+0000","last_became_active":"2024-04-20T17:48:54.814508+0000","last_became_peered":"2024-04-20T17:48:54.814508+0000","last_unstale":"2024-04-20T17:51:20.195044+0000","last_undegraded":"2024-04-20T17:51:20.195044+0000","last_fullsized":"2024-04-20T17:51:20.195044+0000","mapping_epoch":894,"log_start":"904'3100","ondisk_log_start":"904'3100","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'9665","last_scrub_stamp":"2024-04-20T17:50:27.211421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:27.211421+0000","objects_scrubbed":9588,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:56.034157+0000","scrub_duration":2310,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":491126784,"num_objects":7494,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13168,"num_write_kb":661184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"945'13286","reported_seq":13499,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-04-20T17:51:20.185396+0000","last_change":"2024-04-20T17:50:25.096410+0000","last_active":"2024-04-20T17:51:20.185396+0000","last_peered":"2024-04-20T17:51:20.185396+0000","last_clean":"2024-04-20T17:51:20.185396+0000","last_became_active":"2024-04-20T17:48:54.818033+0000","last_became_peered":"2024-04-20T17:48:54.818033+0000","last_unstale":"2024-04-20T17:51:20.185396+0000","last_undegraded":"2024-04-20T17:51:20.185396+0000","last_fullsized":"2024-04-20T17:51:20.185396+0000","mapping_epoch":894,"log_start":"904'3200","ondisk_log_start":"904'3200","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"929'9608","last_scrub_stamp":"2024-04-20T17:50:25.096273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:25.096273+0000","objects_scrubbed":9524,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:47.683348+0000","scrub_duration":2229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":491126784,"num_objects":7494,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13286,"num_write_kb":664960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.5","version":"945'13275","reported_seq":13490,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-04-20T17:51:21.175847+0000","last_change":"2024-04-20T17:50:23.865871+0000","last_active":"2024-04-20T17:51:21.175847+0000","last_peered":"2024-04-20T17:51:21.175847+0000","last_clean":"2024-04-20T17:51:21.175847+0000","last_became_active":"2024-04-20T17:48:54.817749+0000","last_became_peered":"2024-04-20T17:48:54.817749+0000","last_unstale":"2024-04-20T17:51:21.175847+0000","last_undegraded":"2024-04-20T17:51:21.175847+0000","last_fullsized":"2024-04-20T17:51:21.175847+0000","mapping_epoch":894,"log_start":"904'3200","ondisk_log_start":"904'3200","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9546","last_scrub_stamp":"2024-04-20T17:50:23.865708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:23.865708+0000","objects_scrubbed":9473,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:24.406565+0000","scrub_duration":1985,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489488384,"num_objects":7469,"num_object_clones":0,"num_object_copies":14938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13275,"num_write_kb":663808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"945'13203","reported_seq":13415,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-04-20T17:51:20.849779+0000","last_change":"2024-04-20T17:50:23.032137+0000","last_active":"2024-04-20T17:51:20.849779+0000","last_peered":"2024-04-20T17:51:20.849779+0000","last_clean":"2024-04-20T17:51:20.849779+0000","last_became_active":"2024-04-20T17:48:55.061199+0000","last_became_peered":"2024-04-20T17:48:55.061199+0000","last_unstale":"2024-04-20T17:51:20.849779+0000","last_undegraded":"2024-04-20T17:51:20.849779+0000","last_fullsized":"2024-04-20T17:51:20.849779+0000","mapping_epoch":894,"log_start":"905'3200","ondisk_log_start":"905'3200","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9404","last_scrub_stamp":"2024-04-20T17:50:23.031924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:23.031924+0000","objects_scrubbed":9366,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:30.444210+0000","scrub_duration":1573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":492109824,"num_objects":7509,"num_object_clones":0,"num_object_copies":15018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13203,"num_write_kb":662784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"945'13259","reported_seq":24006,"reported_epoch":945,"state":"active+remapped+backfilling","last_fresh":"2024-04-20T17:51:22.180425+0000","last_change":"2024-04-20T17:50:46.276928+0000","last_active":"2024-04-20T17:51:22.180425+0000","last_peered":"2024-04-20T17:51:22.180425+0000","last_clean":"2024-04-20T17:50:43.748764+0000","last_became_active":"2024-04-20T17:50:46.159037+0000","last_became_peered":"2024-04-20T17:50:46.159037+0000","last_unstale":"2024-04-20T17:51:22.180425+0000","last_undegraded":"2024-04-20T17:51:22.180425+0000","last_fullsized":"2024-04-20T17:51:22.180425+0000","mapping_epoch":935,"log_start":"904'3200","ondisk_log_start":"904'3200","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'9622","last_scrub_stamp":"2024-04-20T17:50:27.118695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:27.118695+0000","objects_scrubbed":9558,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":479526912,"num_objects":7317,"num_object_clones":0,"num_object_copies":14634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2762,"num_objects_unfound":0,"num_objects_dirty":7317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13259,"num_write_kb":658432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5263,"num_bytes_recovered":344850432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[9,3],"avail_no_missing":["9","3"],"object_location_counts":[{"shards":"3,9","objects":7317}],"blocked_by":[],"up_primary":8,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.e","version":"945'13306","reported_seq":13515,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-04-20T17:51:19.820449+0000","last_change":"2024-04-20T17:51:03.098201+0000","last_active":"2024-04-20T17:51:19.820449+0000","last_peered":"2024-04-20T17:51:19.820449+0000","last_clean":"2024-04-20T17:51:19.820449+0000","last_became_active":"2024-04-20T17:48:54.815159+0000","last_became_peered":"2024-04-20T17:48:54.815159+0000","last_unstale":"2024-04-20T17:51:19.820449+0000","last_undegraded":"2024-04-20T17:51:19.820449+0000","last_fullsized":"2024-04-20T17:51:19.820449+0000","mapping_epoch":894,"log_start":"905'3300","ondisk_log_start":"905'3300","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"942'12102","last_scrub_stamp":"2024-04-20T17:51:03.098072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:03.098072+0000","objects_scrubbed":8857,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:30.248766+0000","scrub_duration":2003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":495845376,"num_objects":7566,"num_object_clones":0,"num_object_copies":15132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13306,"num_write_kb":667904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"15'137","reported_seq":941,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-04-20T17:51:16.812484+0000","last_change":"2024-04-20T17:50:57.911454+0000","last_active":"2024-04-20T17:51:16.812484+0000","last_peered":"2024-04-20T17:51:16.812484+0000","last_clean":"2024-04-20T17:51:16.812484+0000","last_became_active":"2024-04-20T17:45:32.670941+0000","last_became_peered":"2024-04-20T17:45:32.670941+0000","last_unstale":"2024-04-20T17:51:16.812484+0000","last_undegraded":"2024-04-20T17:51:16.812484+0000","last_fullsized":"2024-04-20T17:51:16.812484+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:57.911336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:57.911336+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:02.057237+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"945'13313","reported_seq":13527,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-04-20T17:51:20.618133+0000","last_change":"2024-04-20T17:50:23.641303+0000","last_active":"2024-04-20T17:51:20.618133+0000","last_peered":"2024-04-20T17:51:20.618133+0000","last_clean":"2024-04-20T17:51:20.618133+0000","last_became_active":"2024-04-20T17:48:55.061161+0000","last_became_peered":"2024-04-20T17:48:55.061161+0000","last_unstale":"2024-04-20T17:51:20.618133+0000","last_undegraded":"2024-04-20T17:51:20.618133+0000","last_fullsized":"2024-04-20T17:51:20.618133+0000","mapping_epoch":894,"log_start":"905'3300","ondisk_log_start":"905'3300","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9495","last_scrub_stamp":"2024-04-20T17:50:23.641164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:23.641164+0000","objects_scrubbed":9459,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:29.146891+0000","scrub_duration":1363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":495648768,"num_objects":7563,"num_object_clones":0,"num_object_copies":15126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13313,"num_write_kb":668032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"15'137","reported_seq":939,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-04-20T17:51:16.818172+0000","last_change":"2024-04-20T17:50:25.724423+0000","last_active":"2024-04-20T17:51:16.818172+0000","last_peered":"2024-04-20T17:51:16.818172+0000","last_clean":"2024-04-20T17:51:16.818172+0000","last_became_active":"2024-04-20T17:45:29.817528+0000","last_became_peered":"2024-04-20T17:45:29.817528+0000","last_unstale":"2024-04-20T17:51:16.818172+0000","last_undegraded":"2024-04-20T17:51:16.818172+0000","last_fullsized":"2024-04-20T17:51:16.818172+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:25.724296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:25.724296+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:37.439709+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"945'13309","reported_seq":13529,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-04-20T17:51:18.607929+0000","last_change":"2024-04-20T17:50:29.927851+0000","last_active":"2024-04-20T17:51:18.607929+0000","last_peered":"2024-04-20T17:51:18.607929+0000","last_clean":"2024-04-20T17:51:18.607929+0000","last_became_active":"2024-04-20T17:48:54.817445+0000","last_became_peered":"2024-04-20T17:48:54.817445+0000","last_unstale":"2024-04-20T17:51:18.607929+0000","last_undegraded":"2024-04-20T17:51:18.607929+0000","last_fullsized":"2024-04-20T17:51:18.607929+0000","mapping_epoch":894,"log_start":"904'3300","ondisk_log_start":"904'3300","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'10062","last_scrub_stamp":"2024-04-20T17:50:29.927762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:29.927762+0000","objects_scrubbed":9978,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:58.752509+0000","scrub_duration":2071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505217024,"num_objects":7709,"num_object_clones":0,"num_object_copies":15418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13309,"num_write_kb":672576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"15'137","reported_seq":965,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-04-20T17:51:16.817185+0000","last_change":"2024-04-20T17:50:20.138119+0000","last_active":"2024-04-20T17:51:16.817185+0000","last_peered":"2024-04-20T17:51:16.817185+0000","last_clean":"2024-04-20T17:51:16.817185+0000","last_became_active":"2024-04-20T17:45:29.827577+0000","last_became_peered":"2024-04-20T17:45:29.827577+0000","last_unstale":"2024-04-20T17:51:16.817185+0000","last_undegraded":"2024-04-20T17:51:16.817185+0000","last_fullsized":"2024-04-20T17:51:16.817185+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:20.137983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:20.137983+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:30.668575+0000","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"945'13176","reported_seq":13453,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-04-20T17:51:19.783107+0000","last_change":"2024-04-20T17:51:00.541301+0000","last_active":"2024-04-20T17:51:19.783107+0000","last_peered":"2024-04-20T17:51:19.783107+0000","last_clean":"2024-04-20T17:51:19.783107+0000","last_became_active":"2024-04-20T17:48:55.061208+0000","last_became_peered":"2024-04-20T17:48:55.061208+0000","last_unstale":"2024-04-20T17:51:19.783107+0000","last_undegraded":"2024-04-20T17:51:19.783107+0000","last_fullsized":"2024-04-20T17:51:19.783107+0000","mapping_epoch":894,"log_start":"904'3100","ondisk_log_start":"904'3100","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"941'11888","last_scrub_stamp":"2024-04-20T17:51:00.541192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:00.541192+0000","objects_scrubbed":8973,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:11.066367+0000","scrub_duration":1502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499843096,"num_objects":7628,"num_object_clones":0,"num_object_copies":15256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7628,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13176,"num_write_kb":665665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"15'137","reported_seq":933,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-04-20T17:51:16.812378+0000","last_change":"2024-04-20T17:50:18.122040+0000","last_active":"2024-04-20T17:51:16.812378+0000","last_peered":"2024-04-20T17:51:16.812378+0000","last_clean":"2024-04-20T17:51:16.812378+0000","last_became_active":"2024-04-20T17:45:32.666523+0000","last_became_peered":"2024-04-20T17:45:32.666523+0000","last_unstale":"2024-04-20T17:51:16.812378+0000","last_undegraded":"2024-04-20T17:51:16.812378+0000","last_fullsized":"2024-04-20T17:51:16.812378+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:18.121912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:18.121912+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:25.993620+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.a","version":"945'13508","reported_seq":13725,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-04-20T17:51:20.572447+0000","last_change":"2024-04-20T17:50:22.180642+0000","last_active":"2024-04-20T17:51:20.572447+0000","last_peered":"2024-04-20T17:51:20.572447+0000","last_clean":"2024-04-20T17:51:20.572447+0000","last_became_active":"2024-04-20T17:48:55.061180+0000","last_became_peered":"2024-04-20T17:48:55.061180+0000","last_unstale":"2024-04-20T17:51:20.572447+0000","last_undegraded":"2024-04-20T17:51:20.572447+0000","last_fullsized":"2024-04-20T17:51:20.572447+0000","mapping_epoch":894,"log_start":"905'3500","ondisk_log_start":"905'3500","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9574","last_scrub_stamp":"2024-04-20T17:50:22.180501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:22.180501+0000","objects_scrubbed":9507,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:51.128747+0000","scrub_duration":1824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489553920,"num_objects":7470,"num_object_clones":0,"num_object_copies":14940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13508,"num_write_kb":671296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"15'137","reported_seq":936,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-04-20T17:51:16.827003+0000","last_change":"2024-04-20T17:50:31.133012+0000","last_active":"2024-04-20T17:51:16.827003+0000","last_peered":"2024-04-20T17:51:16.827003+0000","last_clean":"2024-04-20T17:51:16.827003+0000","last_became_active":"2024-04-20T17:46:40.831961+0000","last_became_peered":"2024-04-20T17:46:40.831961+0000","last_unstale":"2024-04-20T17:51:16.827003+0000","last_undegraded":"2024-04-20T17:51:16.827003+0000","last_fullsized":"2024-04-20T17:51:16.827003+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:31.132879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:31.132879+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:31.822631+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"945'13074","reported_seq":13284,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-04-20T17:51:18.980978+0000","last_change":"2024-04-20T17:50:19.657299+0000","last_active":"2024-04-20T17:51:18.980978+0000","last_peered":"2024-04-20T17:51:18.980978+0000","last_clean":"2024-04-20T17:51:18.980978+0000","last_became_active":"2024-04-20T17:48:54.816644+0000","last_became_peered":"2024-04-20T17:48:54.816644+0000","last_unstale":"2024-04-20T17:51:18.980978+0000","last_undegraded":"2024-04-20T17:51:18.980978+0000","last_fullsized":"2024-04-20T17:51:18.980978+0000","mapping_epoch":894,"log_start":"904'3000","ondisk_log_start":"904'3000","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"927'9204","last_scrub_stamp":"2024-04-20T17:50:19.657194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:19.657194+0000","objects_scrubbed":9124,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:25.188599+0000","scrub_duration":2257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":504102912,"num_objects":7692,"num_object_clones":0,"num_object_copies":15384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13074,"num_write_kb":664512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"15'137","reported_seq":734,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-04-20T17:51:16.806883+0000","last_change":"2024-04-20T17:51:00.057261+0000","last_active":"2024-04-20T17:51:16.806883+0000","last_peered":"2024-04-20T17:51:16.806883+0000","last_clean":"2024-04-20T17:51:16.806883+0000","last_became_active":"2024-04-20T17:49:46.824559+0000","last_became_peered":"2024-04-20T17:49:46.824559+0000","last_unstale":"2024-04-20T17:51:16.806883+0000","last_undegraded":"2024-04-20T17:51:16.806883+0000","last_fullsized":"2024-04-20T17:51:16.806883+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:00.057156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:00.057156+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:08.337431+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"945'13552","reported_seq":13771,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-04-20T17:51:20.170959+0000","last_change":"2024-04-20T17:50:28.283117+0000","last_active":"2024-04-20T17:51:20.170959+0000","last_peered":"2024-04-20T17:51:20.170959+0000","last_clean":"2024-04-20T17:51:20.170959+0000","last_became_active":"2024-04-20T17:48:54.813199+0000","last_became_peered":"2024-04-20T17:48:54.813199+0000","last_unstale":"2024-04-20T17:51:20.170959+0000","last_undegraded":"2024-04-20T17:51:20.170959+0000","last_fullsized":"2024-04-20T17:51:20.170959+0000","mapping_epoch":894,"log_start":"905'3500","ondisk_log_start":"905'3500","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'10025","last_scrub_stamp":"2024-04-20T17:50:28.282957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:28.282957+0000","objects_scrubbed":9963,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:48.575002+0000","scrub_duration":2339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":506593280,"num_objects":7730,"num_object_clones":0,"num_object_copies":15460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13552,"num_write_kb":681024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"15'137","reported_seq":941,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-04-20T17:51:16.812335+0000","last_change":"2024-04-20T17:50:58.932166+0000","last_active":"2024-04-20T17:51:16.812335+0000","last_peered":"2024-04-20T17:51:16.812335+0000","last_clean":"2024-04-20T17:51:16.812335+0000","last_became_active":"2024-04-20T17:45:32.782706+0000","last_became_peered":"2024-04-20T17:45:32.782706+0000","last_unstale":"2024-04-20T17:51:16.812335+0000","last_undegraded":"2024-04-20T17:51:16.812335+0000","last_fullsized":"2024-04-20T17:51:16.812335+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:58.932046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:58.932046+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:23.591238+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.2","version":"945'13181","reported_seq":13394,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-04-20T17:51:18.590840+0000","last_change":"2024-04-20T17:50:22.440005+0000","last_active":"2024-04-20T17:51:18.590840+0000","last_peered":"2024-04-20T17:51:18.590840+0000","last_clean":"2024-04-20T17:51:18.590840+0000","last_became_active":"2024-04-20T17:48:55.060965+0000","last_became_peered":"2024-04-20T17:48:55.060965+0000","last_unstale":"2024-04-20T17:51:18.590840+0000","last_undegraded":"2024-04-20T17:51:18.590840+0000","last_fullsized":"2024-04-20T17:51:18.590840+0000","mapping_epoch":894,"log_start":"904'3100","ondisk_log_start":"904'3100","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9408","last_scrub_stamp":"2024-04-20T17:50:22.439841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:22.439841+0000","objects_scrubbed":9355,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:23.199554+0000","scrub_duration":1641,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498008064,"num_objects":7599,"num_object_clones":0,"num_object_copies":15198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13181,"num_write_kb":664960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"15'137","reported_seq":919,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-04-20T17:51:16.811426+0000","last_change":"2024-04-20T17:49:48.210622+0000","last_active":"2024-04-20T17:51:16.811426+0000","last_peered":"2024-04-20T17:51:16.811426+0000","last_clean":"2024-04-20T17:51:16.811426+0000","last_became_active":"2024-04-20T17:45:32.669451+0000","last_became_peered":"2024-04-20T17:45:32.669451+0000","last_unstale":"2024-04-20T17:51:16.811426+0000","last_undegraded":"2024-04-20T17:51:16.811426+0000","last_fullsized":"2024-04-20T17:51:16.811426+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:49:48.210506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:49:48.210506+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"945'13338","reported_seq":13555,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-04-20T17:51:21.736017+0000","last_change":"2024-04-20T17:50:30.239298+0000","last_active":"2024-04-20T17:51:21.736017+0000","last_peered":"2024-04-20T17:51:21.736017+0000","last_clean":"2024-04-20T17:51:21.736017+0000","last_became_active":"2024-04-20T17:48:55.061200+0000","last_became_peered":"2024-04-20T17:48:55.061200+0000","last_unstale":"2024-04-20T17:51:21.736017+0000","last_undegraded":"2024-04-20T17:51:21.736017+0000","last_fullsized":"2024-04-20T17:51:21.736017+0000","mapping_epoch":894,"log_start":"905'3300","ondisk_log_start":"905'3300","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'9940","last_scrub_stamp":"2024-04-20T17:50:30.239131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:30.239131+0000","objects_scrubbed":9887,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:36.933027+0000","scrub_duration":1672,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485752832,"num_objects":7412,"num_object_clones":0,"num_object_copies":14824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13338,"num_write_kb":664000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"15'137","reported_seq":934,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-04-20T17:51:17.153219+0000","last_change":"2024-04-20T17:50:23.398315+0000","last_active":"2024-04-20T17:51:17.153219+0000","last_peered":"2024-04-20T17:51:17.153219+0000","last_clean":"2024-04-20T17:51:17.153219+0000","last_became_active":"2024-04-20T17:45:32.644584+0000","last_became_peered":"2024-04-20T17:45:32.644584+0000","last_unstale":"2024-04-20T17:51:17.153219+0000","last_undegraded":"2024-04-20T17:51:17.153219+0000","last_fullsized":"2024-04-20T17:51:17.153219+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:23.398142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:23.398142+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:31.965635+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"945'13270","reported_seq":13489,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-04-20T17:51:18.606913+0000","last_change":"2024-04-20T17:50:30.712810+0000","last_active":"2024-04-20T17:51:18.606913+0000","last_peered":"2024-04-20T17:51:18.606913+0000","last_clean":"2024-04-20T17:51:18.606913+0000","last_became_active":"2024-04-20T17:48:54.813062+0000","last_became_peered":"2024-04-20T17:48:54.813062+0000","last_unstale":"2024-04-20T17:51:18.606913+0000","last_undegraded":"2024-04-20T17:51:18.606913+0000","last_fullsized":"2024-04-20T17:51:18.606913+0000","mapping_epoch":894,"log_start":"904'3200","ondisk_log_start":"904'3200","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'10026","last_scrub_stamp":"2024-04-20T17:50:30.712648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:30.712648+0000","objects_scrubbed":9967,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:58.995403+0000","scrub_duration":1871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":501350400,"num_objects":7650,"num_object_clones":0,"num_object_copies":15300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13270,"num_write_kb":669440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"15'137","reported_seq":882,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-04-20T17:51:16.825846+0000","last_change":"2024-04-20T17:50:33.137929+0000","last_active":"2024-04-20T17:51:16.825846+0000","last_peered":"2024-04-20T17:51:16.825846+0000","last_clean":"2024-04-20T17:51:16.825846+0000","last_became_active":"2024-04-20T17:46:40.881713+0000","last_became_peered":"2024-04-20T17:46:40.881713+0000","last_unstale":"2024-04-20T17:51:16.825846+0000","last_undegraded":"2024-04-20T17:51:16.825846+0000","last_fullsized":"2024-04-20T17:51:16.825846+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:33.137831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:33.137831+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:36.471855+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.8","version":"945'13548","reported_seq":13762,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-04-20T17:51:22.165394+0000","last_change":"2024-04-20T17:50:19.940419+0000","last_active":"2024-04-20T17:51:22.165394+0000","last_peered":"2024-04-20T17:51:22.165394+0000","last_clean":"2024-04-20T17:51:22.165394+0000","last_became_active":"2024-04-20T17:48:54.813368+0000","last_became_peered":"2024-04-20T17:48:54.813368+0000","last_unstale":"2024-04-20T17:51:22.165394+0000","last_undegraded":"2024-04-20T17:51:22.165394+0000","last_fullsized":"2024-04-20T17:51:22.165394+0000","mapping_epoch":894,"log_start":"905'3500","ondisk_log_start":"905'3500","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"927'9391","last_scrub_stamp":"2024-04-20T17:50:19.940199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:19.940199+0000","objects_scrubbed":9334,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:34.165962+0000","scrub_duration":1977,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":492830720,"num_objects":7520,"num_object_clones":0,"num_object_copies":15040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13548,"num_write_kb":674176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":880,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-04-20T17:51:16.825798+0000","last_change":"2024-04-20T17:50:21.111183+0000","last_active":"2024-04-20T17:51:16.825798+0000","last_peered":"2024-04-20T17:51:16.825798+0000","last_clean":"2024-04-20T17:51:16.825798+0000","last_became_active":"2024-04-20T17:46:40.851683+0000","last_became_peered":"2024-04-20T17:46:40.851683+0000","last_unstale":"2024-04-20T17:51:16.825798+0000","last_undegraded":"2024-04-20T17:51:16.825798+0000","last_fullsized":"2024-04-20T17:51:16.825798+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:21.111059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:21.111059+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:45.968902+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":7918125080,"num_objects":120822,"num_object_clones":0,"num_object_copies":241644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2762,"num_objects_unfound":0,"num_objects_dirty":120822,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":212766,"num_write_kb":10674753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5263,"num_bytes_recovered":344850432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16122191872,"data_stored":16122183728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160766,"ondisk_log_size":160766,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},"log_size":1507,"ondisk_log_size":1507,"up":22,"acting":22,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991781,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2408104,"kb_used_data":1466208,"kb_used_omap":1,"kb_used_meta":941886,"kb_avail":91337048,"statfs":{"total":95995035648,"available":93529137152,"internally_reserved":0,"allocated":1501396992,"data_stored":1501047791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":964491716},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253420,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3266344,"kb_used_data":2428576,"kb_used_omap":1,"kb_used_meta":837758,"kb_avail":90478808,"statfs":{"total":95995035648,"available":92650299392,"internally_reserved":0,"allocated":2486861824,"data_stored":2486512623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":857864640},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613169,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2166568,"kb_used_data":1427360,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":91578584,"statfs":{"total":95995035648,"available":93776470016,"internally_reserved":0,"allocated":1461616640,"data_stored":1461267439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":756939200},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809399,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1197224,"kb_used_data":760608,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":92547928,"statfs":{"total":95995035648,"available":94769078272,"internally_reserved":0,"allocated":778862592,"data_stored":778513391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":799,"seq":3431678869589,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1591212,"kb_used_data":1464932,"kb_used_omap":11,"kb_used_meta":126260,"kb_avail":92153940,"statfs":{"total":95995035648,"available":94365634560,"internally_reserved":0,"allocated":1500090368,"data_stored":1499737103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11990,"internal_metadata":129290538},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":535,"seq":2297807503629,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":521448,"kb_used_data":475552,"kb_used_omap":170,"kb_used_meta":45717,"kb_avail":93223704,"statfs":{"total":95995035648,"available":95461072896,"internally_reserved":0,"allocated":486965248,"data_stored":486616047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":174628,"internal_metadata":46814684},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":659,"seq":2830383448251,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3308652,"kb_used_data":2934756,"kb_used_omap":4,"kb_used_meta":373883,"kb_avail":90436500,"statfs":{"total":95995035648,"available":92606976000,"internally_reserved":0,"allocated":3005190144,"data_stored":3004836871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4410,"internal_metadata":382856902},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738974,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":744044,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":93001108,"statfs":{"total":95995035648,"available":95233134592,"internally_reserved":0,"allocated":688128,"data_stored":338927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394263,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":626992,"kb_used_data":207912,"kb_used_omap":202,"kb_used_meta":418869,"kb_avail":93118160,"statfs":{"total":95995035648,"available":95352995840,"internally_reserved":0,"allocated":212901888,"data_stored":212549135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":428922327},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653312,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2532136,"kb_used_data":1954144,"kb_used_omap":4,"kb_used_meta":577979,"kb_avail":91213016,"statfs":{"total":95995035648,"available":93402128384,"internally_reserved":0,"allocated":2001043456,"data_stored":2000694255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":591851207},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986761,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1594152,"kb_used_data":967392,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":92151000,"statfs":{"total":95995035648,"available":94362624000,"internally_reserved":0,"allocated":990609408,"data_stored":990260207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790531,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3036464,"kb_used_data":2144552,"kb_used_omap":1,"kb_used_meta":891902,"kb_avail":90708688,"statfs":{"total":95995035648,"available":92885696512,"internally_reserved":0,"allocated":2196021248,"data_stored":2195668495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":913308101},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2991394816,"data_stored":2991390744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1999175680,"data_stored":1999175680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":987365376,"data_stored":987365376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1966407680,"data_stored":1966407680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":485752832,"data_stored":485752832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1488064512,"data_stored":1488060440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":777715712,"data_stored":777715712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1456209920,"data_stored":1456209920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2481061888,"data_stored":2481061888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1489043456,"data_stored":1489043456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:51:24.135 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:51:24.143 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:51:24.177 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:24.201 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:51:24.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:24.207 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:51:24.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:23 smithi106 ceph-mon[44423]: pgmap v2493: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 7.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 2762/241648 objects misplaced (1.143%); 9.6 MiB/s, 154 objects/s recovering 2024-04-20T17:51:24.308 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:51:24.340 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:24.369 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:51:24.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:24.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:51:24.475 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:51:24.513 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:24.536 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:51:24.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:24.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:51:24.643 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:51:24.676 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:24.767 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:51:24.790 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:51:24.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:24.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:51:24.800 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:51:24.848 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-20T17:51:24.895 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:51:24.935 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:24.964 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:51:24.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:24.967 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:51:25.068 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:51:25.108 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:25.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:24 smithi005 ceph-mon[39562]: from='client.28280 v1:172.21.15.5:0/714300014' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:51:25.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:24 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:51:25.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:24 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:51:25.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:24 smithi005 ceph-mon[39562]: osdmap e946: 12 total, 12 up, 12 in 2024-04-20T17:51:25.141 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:51:25.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:25.144 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:51:25.147 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:24 smithi171 ceph-mon[44189]: from='client.28280 v1:172.21.15.5:0/714300014' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:51:25.147 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:24 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:51:25.147 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:24 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:51:25.147 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:24 smithi171 ceph-mon[44189]: osdmap e946: 12 total, 12 up, 12 in 2024-04-20T17:51:25.245 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:51:25.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:24 smithi106 ceph-mon[44423]: from='client.28280 v1:172.21.15.5:0/714300014' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:51:25.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:24 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:51:25.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:24 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:51:25.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:24 smithi106 ceph-mon[44423]: osdmap e946: 12 total, 12 up, 12 in 2024-04-20T17:51:25.289 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:25.317 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:51:25.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:25.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:51:25.421 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:51:25.468 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:25 smithi005 ceph-mon[39562]: pgmap v2495: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 2762/238800 objects misplaced (1.157%); 12 MiB/s, 192 objects/s recovering 2024-04-20T17:51:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:25 smithi106 ceph-mon[44423]: pgmap v2495: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 2762/238800 objects misplaced (1.157%); 12 MiB/s, 192 objects/s recovering 2024-04-20T17:51:26.326 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:51:26.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:26.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:51:26.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:25 smithi171 ceph-mon[44189]: pgmap v2495: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 7.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 2762/238800 objects misplaced (1.157%); 12 MiB/s, 192 objects/s recovering 2024-04-20T17:51:26.436 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:51:26.468 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:26.491 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:51:26.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:26.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:51:26.597 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:51:26.634 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:26.891 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:51:26.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:26.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:51:26.995 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:51:26.999 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:51:27.000 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:51:27.040 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:27.838 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:51:27.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:27 smithi005 ceph-mon[39562]: pgmap v2496: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 7.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 2762/232448 objects misplaced (1.188%); 9.7 MiB/s, 155 objects/s recovering 2024-04-20T17:51:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:27 smithi106 ceph-mon[44423]: pgmap v2496: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 7.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 2762/232448 objects misplaced (1.188%); 9.7 MiB/s, 155 objects/s recovering 2024-04-20T17:51:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:27 smithi171 ceph-mon[44189]: pgmap v2496: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 7.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 2762/232448 objects misplaced (1.188%); 9.7 MiB/s, 155 objects/s recovering 2024-04-20T17:51:28.890 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:51:28.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:28 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1602689171' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:51:28.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:28 smithi005 ceph-mon[39562]: from='client.28283 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:51:28.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:28 smithi005 ceph-mon[39562]: pgmap v2497: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 7.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 1888/228130 objects misplaced (0.828%); 9.4 MiB/s, 149 objects/s recovering 2024-04-20T17:51:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:28 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1602689171' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:51:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:28 smithi106 ceph-mon[44423]: from='client.28283 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:51:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:28 smithi106 ceph-mon[44423]: pgmap v2497: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 7.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 1888/228130 objects misplaced (0.828%); 9.4 MiB/s, 149 objects/s recovering 2024-04-20T17:51:29.296 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:51:29.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:29.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:51:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:28 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1602689171' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:51:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:28 smithi171 ceph-mon[44189]: from='client.28283 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:51:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:28 smithi171 ceph-mon[44189]: pgmap v2497: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 7.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 1888/228130 objects misplaced (0.828%); 9.4 MiB/s, 149 objects/s recovering 2024-04-20T17:51:29.409 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:51:29.446 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:29.470 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:51:29.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:29.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:51:29.574 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:51:29.626 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:29.661 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:51:29.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:29.665 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:51:29.766 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:51:29.818 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:29.853 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:51:29.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:29.858 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:51:29.959 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:51:29.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:29 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:51:29.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:29 smithi005 ceph-mon[39562]: from='client.28283 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:51:29.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:29 smithi005 ceph-mon[39562]: osdmap e947: 12 total, 12 up, 12 in 2024-04-20T17:51:30.029 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:29 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:51:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:29 smithi106 ceph-mon[44423]: from='client.28283 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:51:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:29 smithi106 ceph-mon[44423]: osdmap e947: 12 total, 12 up, 12 in 2024-04-20T17:51:30.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:29 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:51:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:29 smithi171 ceph-mon[44189]: from='client.28283 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:51:30.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:29 smithi171 ceph-mon[44189]: osdmap e947: 12 total, 12 up, 12 in 2024-04-20T17:51:30.886 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:30 smithi005 ceph-mon[39562]: pgmap v2499: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 6.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 1888/222466 objects misplaced (0.849%); 5.8 MiB/s, 93 objects/s recovering 2024-04-20T17:51:30.919 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:51:30.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:30.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:51:31.025 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:51:31.066 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:31.106 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:51:31.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:31.115 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:51:31.216 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:51:31.246 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:30 smithi106 ceph-mon[44423]: pgmap v2499: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 6.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 1888/222466 objects misplaced (0.849%); 5.8 MiB/s, 93 objects/s recovering 2024-04-20T17:51:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:30 smithi171 ceph-mon[44189]: pgmap v2499: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 6.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 1888/222466 objects misplaced (0.849%); 5.8 MiB/s, 93 objects/s recovering 2024-04-20T17:51:31.355 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:51:31.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:31.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:51:31.460 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:51:31.523 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:31.554 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:51:31.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:31.558 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:51:31.659 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:51:31.709 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:31.732 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:51:31.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:31.739 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:51:31.841 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:51:31.876 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:31.899 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:51:31.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:31.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:51:32.006 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:51:32.037 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:32.100 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:51:32.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:32.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:51:32.211 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:51:32.256 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:33.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:32 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:51:33.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:32 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:51:33.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:32 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:51:33.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:32 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:51:33.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:32 smithi005 ceph-mon[39562]: osdmap e948: 12 total, 12 up, 12 in 2024-04-20T17:51:33.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:32 smithi005 ceph-mon[39562]: pgmap v2501: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.24k op/s; 1080/218884 objects misplaced (0.493%); 11 MiB/s, 183 objects/s recovering 2024-04-20T17:51:33.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:32 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:51:33.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:32 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:51:33.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:32 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:51:33.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:32 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:51:33.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:32 smithi106 ceph-mon[44423]: osdmap e948: 12 total, 12 up, 12 in 2024-04-20T17:51:33.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:32 smithi106 ceph-mon[44423]: pgmap v2501: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.24k op/s; 1080/218884 objects misplaced (0.493%); 11 MiB/s, 183 objects/s recovering 2024-04-20T17:51:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:32 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:51:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:32 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:51:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:32 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:51:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:32 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:51:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:32 smithi171 ceph-mon[44189]: osdmap e948: 12 total, 12 up, 12 in 2024-04-20T17:51:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:32 smithi171 ceph-mon[44189]: pgmap v2501: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.24k op/s; 1080/218884 objects misplaced (0.493%); 11 MiB/s, 183 objects/s recovering 2024-04-20T17:51:33.443 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:51:33.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:33.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:51:33.549 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:51:33.574 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:51:33.574 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:51:33.596 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:33.635 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:51:33.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:33.639 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:51:33.697 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:51:33.706 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:51:33.740 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:51:33.774 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:33.912 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:33 smithi171 ceph-mon[44189]: from='client.28289 v1:172.21.15.5:0/1681103300' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:51:34.035 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:51:34.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:34.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:51:34.045 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:33 smithi005 ceph-mon[39562]: from='client.28289 v1:172.21.15.5:0/1681103300' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:51:34.144 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:51:34.170 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:33 smithi106 ceph-mon[44423]: from='client.28289 v1:172.21.15.5:0/1681103300' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:51:34.172 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:34.198 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:51:34.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:34.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:51:34.305 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:51:34.358 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:34.385 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:51:34.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:34.388 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:51:34.489 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:51:34.533 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:34.558 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:51:34.562 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:51:34.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:34.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:51:34.665 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:51:34.689 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":2501,"stamp":"2024-04-20T17:51:32.698925+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7172784696,"num_objects":109442,"num_object_clones":0,"num_object_copies":218884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1080,"num_objects_unfound":0,"num_objects_dirty":109442,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":224148,"num_write_kb":10674753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6732,"num_bytes_recovered":441122816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162255,"ondisk_log_size":162255,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":21572824,"kb_used_data":14814560,"kb_used_omap":404,"kb_used_meta":6757355,"kb_avail":1103369000,"statfs":{"total":1151940427776,"available":1129849856000,"internally_reserved":0,"allocated":15170109440,"data_stored":15165771938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":414014,"internal_metadata":6919532226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-652607488,"num_objects":-9958,"num_object_clones":0,"num_object_copies":-19916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1682,"num_objects_unfound":0,"num_objects_dirty":-9958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9958,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1469,"num_bytes_recovered":96272384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001527"},"pg_stats":[{"pgid":"9.7","version":"947'13814","reported_seq":14033,"reported_epoch":947,"state":"active+clean","last_fresh":"2024-04-20T17:51:30.215126+0000","last_change":"2024-04-20T17:50:27.211659+0000","last_active":"2024-04-20T17:51:30.215126+0000","last_peered":"2024-04-20T17:51:30.215126+0000","last_clean":"2024-04-20T17:51:30.215126+0000","last_became_active":"2024-04-20T17:48:54.814508+0000","last_became_peered":"2024-04-20T17:48:54.814508+0000","last_unstale":"2024-04-20T17:51:30.215126+0000","last_undegraded":"2024-04-20T17:51:30.215126+0000","last_fullsized":"2024-04-20T17:51:30.215126+0000","mapping_epoch":894,"log_start":"906'3800","ondisk_log_start":"906'3800","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'9665","last_scrub_stamp":"2024-04-20T17:50:27.211421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:27.211421+0000","objects_scrubbed":9588,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:56.034157+0000","scrub_duration":2310,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448790528,"num_objects":6848,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13814,"num_write_kb":661184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"947'14021","reported_seq":14240,"reported_epoch":947,"state":"active+clean","last_fresh":"2024-04-20T17:51:30.208538+0000","last_change":"2024-04-20T17:50:25.096410+0000","last_active":"2024-04-20T17:51:30.208538+0000","last_peered":"2024-04-20T17:51:30.208538+0000","last_clean":"2024-04-20T17:51:30.208538+0000","last_became_active":"2024-04-20T17:48:54.818033+0000","last_became_peered":"2024-04-20T17:48:54.818033+0000","last_unstale":"2024-04-20T17:51:30.208538+0000","last_undegraded":"2024-04-20T17:51:30.208538+0000","last_fullsized":"2024-04-20T17:51:30.208538+0000","mapping_epoch":894,"log_start":"906'4000","ondisk_log_start":"906'4000","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"929'9608","last_scrub_stamp":"2024-04-20T17:50:25.096273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:25.096273+0000","objects_scrubbed":9524,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:47.683348+0000","scrub_duration":2229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442957824,"num_objects":6759,"num_object_clones":0,"num_object_copies":13518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14021,"num_write_kb":664960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.5","version":"947'13999","reported_seq":14219,"reported_epoch":947,"state":"active+clean","last_fresh":"2024-04-20T17:51:31.180023+0000","last_change":"2024-04-20T17:50:23.865871+0000","last_active":"2024-04-20T17:51:31.180023+0000","last_peered":"2024-04-20T17:51:31.180023+0000","last_clean":"2024-04-20T17:51:31.180023+0000","last_became_active":"2024-04-20T17:48:54.817749+0000","last_became_peered":"2024-04-20T17:48:54.817749+0000","last_unstale":"2024-04-20T17:51:31.180023+0000","last_undegraded":"2024-04-20T17:51:31.180023+0000","last_fullsized":"2024-04-20T17:51:31.180023+0000","mapping_epoch":894,"log_start":"906'3900","ondisk_log_start":"906'3900","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9546","last_scrub_stamp":"2024-04-20T17:50:23.865708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:23.865708+0000","objects_scrubbed":9473,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1985,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442040320,"num_objects":6745,"num_object_clones":0,"num_object_copies":13490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13999,"num_write_kb":663808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"947'13937","reported_seq":14151,"reported_epoch":947,"state":"active+clean","last_fresh":"2024-04-20T17:51:30.895367+0000","last_change":"2024-04-20T17:50:23.032137+0000","last_active":"2024-04-20T17:51:30.895367+0000","last_peered":"2024-04-20T17:51:30.895367+0000","last_clean":"2024-04-20T17:51:30.895367+0000","last_became_active":"2024-04-20T17:48:55.061199+0000","last_became_peered":"2024-04-20T17:48:55.061199+0000","last_unstale":"2024-04-20T17:51:30.895367+0000","last_undegraded":"2024-04-20T17:51:30.895367+0000","last_fullsized":"2024-04-20T17:51:30.895367+0000","mapping_epoch":894,"log_start":"906'3900","ondisk_log_start":"906'3900","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9404","last_scrub_stamp":"2024-04-20T17:50:23.031924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:23.031924+0000","objects_scrubbed":9366,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444006400,"num_objects":6775,"num_object_clones":0,"num_object_copies":13550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13937,"num_write_kb":662784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"948'13999","reported_seq":27690,"reported_epoch":948,"state":"active+remapped+backfilling","last_fresh":"2024-04-20T17:51:32.203234+0000","last_change":"2024-04-20T17:50:46.276928+0000","last_active":"2024-04-20T17:51:32.203234+0000","last_peered":"2024-04-20T17:51:32.203234+0000","last_clean":"2024-04-20T17:50:43.748764+0000","last_became_active":"2024-04-20T17:50:46.159037+0000","last_became_peered":"2024-04-20T17:50:46.159037+0000","last_unstale":"2024-04-20T17:51:32.203234+0000","last_undegraded":"2024-04-20T17:51:32.203234+0000","last_fullsized":"2024-04-20T17:51:32.203234+0000","mapping_epoch":935,"log_start":"906'3900","ondisk_log_start":"906'3900","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'9622","last_scrub_stamp":"2024-04-20T17:50:27.118695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:27.118695+0000","objects_scrubbed":9558,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431030272,"num_objects":6577,"num_object_clones":0,"num_object_copies":13154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1080,"num_objects_unfound":0,"num_objects_dirty":6577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13999,"num_write_kb":658432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6732,"num_bytes_recovered":441122816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[9,3],"avail_no_missing":["9","3"],"object_location_counts":[{"shards":"3,9","objects":6577}],"blocked_by":[],"up_primary":8,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.e","version":"947'14020","reported_seq":14233,"reported_epoch":947,"state":"active+clean","last_fresh":"2024-04-20T17:51:29.820254+0000","last_change":"2024-04-20T17:51:03.098201+0000","last_active":"2024-04-20T17:51:29.820254+0000","last_peered":"2024-04-20T17:51:29.820254+0000","last_clean":"2024-04-20T17:51:29.820254+0000","last_became_active":"2024-04-20T17:48:54.815159+0000","last_became_peered":"2024-04-20T17:48:54.815159+0000","last_unstale":"2024-04-20T17:51:29.820254+0000","last_undegraded":"2024-04-20T17:51:29.820254+0000","last_fullsized":"2024-04-20T17:51:29.820254+0000","mapping_epoch":894,"log_start":"906'4000","ondisk_log_start":"906'4000","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"942'12102","last_scrub_stamp":"2024-04-20T17:51:03.098072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:03.098072+0000","objects_scrubbed":8857,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:30.248766+0000","scrub_duration":2003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449052672,"num_objects":6852,"num_object_clones":0,"num_object_copies":13704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14020,"num_write_kb":667904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"15'137","reported_seq":945,"reported_epoch":947,"state":"active+clean","last_fresh":"2024-04-20T17:51:28.903731+0000","last_change":"2024-04-20T17:50:57.911454+0000","last_active":"2024-04-20T17:51:28.903731+0000","last_peered":"2024-04-20T17:51:28.903731+0000","last_clean":"2024-04-20T17:51:28.903731+0000","last_became_active":"2024-04-20T17:45:32.670941+0000","last_became_peered":"2024-04-20T17:45:32.670941+0000","last_unstale":"2024-04-20T17:51:28.903731+0000","last_undegraded":"2024-04-20T17:51:28.903731+0000","last_fullsized":"2024-04-20T17:51:28.903731+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:57.911336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:57.911336+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:02.057237+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"947'14032","reported_seq":14250,"reported_epoch":947,"state":"active+clean","last_fresh":"2024-04-20T17:51:30.638475+0000","last_change":"2024-04-20T17:50:23.641303+0000","last_active":"2024-04-20T17:51:30.638475+0000","last_peered":"2024-04-20T17:51:30.638475+0000","last_clean":"2024-04-20T17:51:30.638475+0000","last_became_active":"2024-04-20T17:48:55.061161+0000","last_became_peered":"2024-04-20T17:48:55.061161+0000","last_unstale":"2024-04-20T17:51:30.638475+0000","last_undegraded":"2024-04-20T17:51:30.638475+0000","last_fullsized":"2024-04-20T17:51:30.638475+0000","mapping_epoch":894,"log_start":"906'4000","ondisk_log_start":"906'4000","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9495","last_scrub_stamp":"2024-04-20T17:50:23.641164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:23.641164+0000","objects_scrubbed":9459,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:35.240721+0000","scrub_duration":1363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448528384,"num_objects":6844,"num_object_clones":0,"num_object_copies":13688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14032,"num_write_kb":668032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"15'137","reported_seq":943,"reported_epoch":947,"state":"active+clean","last_fresh":"2024-04-20T17:51:28.908814+0000","last_change":"2024-04-20T17:50:25.724423+0000","last_active":"2024-04-20T17:51:28.908814+0000","last_peered":"2024-04-20T17:51:28.908814+0000","last_clean":"2024-04-20T17:51:28.908814+0000","last_became_active":"2024-04-20T17:45:29.817528+0000","last_became_peered":"2024-04-20T17:45:29.817528+0000","last_unstale":"2024-04-20T17:51:28.908814+0000","last_undegraded":"2024-04-20T17:51:28.908814+0000","last_fullsized":"2024-04-20T17:51:28.908814+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:25.724296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:25.724296+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:37.439709+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"946'14051","reported_seq":14273,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-04-20T17:51:28.626759+0000","last_change":"2024-04-20T17:50:29.927851+0000","last_active":"2024-04-20T17:51:28.626759+0000","last_peered":"2024-04-20T17:51:28.626759+0000","last_clean":"2024-04-20T17:51:28.626759+0000","last_became_active":"2024-04-20T17:48:54.817445+0000","last_became_peered":"2024-04-20T17:48:54.817445+0000","last_unstale":"2024-04-20T17:51:28.626759+0000","last_undegraded":"2024-04-20T17:51:28.626759+0000","last_fullsized":"2024-04-20T17:51:28.626759+0000","mapping_epoch":894,"log_start":"906'4000","ondisk_log_start":"906'4000","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'10062","last_scrub_stamp":"2024-04-20T17:50:29.927762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:29.927762+0000","objects_scrubbed":9978,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:58.752509+0000","scrub_duration":2071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456589312,"num_objects":6967,"num_object_clones":0,"num_object_copies":13934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14051,"num_write_kb":672576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"15'137","reported_seq":969,"reported_epoch":947,"state":"active+clean","last_fresh":"2024-04-20T17:51:28.898752+0000","last_change":"2024-04-20T17:50:20.138119+0000","last_active":"2024-04-20T17:51:28.898752+0000","last_peered":"2024-04-20T17:51:28.898752+0000","last_clean":"2024-04-20T17:51:28.898752+0000","last_became_active":"2024-04-20T17:45:29.827577+0000","last_became_peered":"2024-04-20T17:45:29.827577+0000","last_unstale":"2024-04-20T17:51:28.898752+0000","last_undegraded":"2024-04-20T17:51:28.898752+0000","last_fullsized":"2024-04-20T17:51:28.898752+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:20.137983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:20.137983+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:30.668575+0000","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"947'13870","reported_seq":14151,"reported_epoch":947,"state":"active+clean","last_fresh":"2024-04-20T17:51:29.816917+0000","last_change":"2024-04-20T17:51:00.541301+0000","last_active":"2024-04-20T17:51:29.816917+0000","last_peered":"2024-04-20T17:51:29.816917+0000","last_clean":"2024-04-20T17:51:29.816917+0000","last_became_active":"2024-04-20T17:48:55.061208+0000","last_became_peered":"2024-04-20T17:48:55.061208+0000","last_unstale":"2024-04-20T17:51:29.816917+0000","last_undegraded":"2024-04-20T17:51:29.816917+0000","last_fullsized":"2024-04-20T17:51:29.816917+0000","mapping_epoch":894,"log_start":"906'3800","ondisk_log_start":"906'3800","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"941'11888","last_scrub_stamp":"2024-04-20T17:51:00.541192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:00.541192+0000","objects_scrubbed":8973,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:11.066367+0000","scrub_duration":1502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454361112,"num_objects":6934,"num_object_clones":0,"num_object_copies":13868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6934,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13870,"num_write_kb":665665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"15'137","reported_seq":937,"reported_epoch":947,"state":"active+clean","last_fresh":"2024-04-20T17:51:28.901257+0000","last_change":"2024-04-20T17:50:18.122040+0000","last_active":"2024-04-20T17:51:28.901257+0000","last_peered":"2024-04-20T17:51:28.901257+0000","last_clean":"2024-04-20T17:51:28.901257+0000","last_became_active":"2024-04-20T17:45:32.666523+0000","last_became_peered":"2024-04-20T17:45:32.666523+0000","last_unstale":"2024-04-20T17:51:28.901257+0000","last_undegraded":"2024-04-20T17:51:28.901257+0000","last_fullsized":"2024-04-20T17:51:28.901257+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:18.121912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:18.121912+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:31.013365+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.a","version":"947'14215","reported_seq":14436,"reported_epoch":947,"state":"active+clean","last_fresh":"2024-04-20T17:51:30.625877+0000","last_change":"2024-04-20T17:50:22.180642+0000","last_active":"2024-04-20T17:51:30.625877+0000","last_peered":"2024-04-20T17:51:30.625877+0000","last_clean":"2024-04-20T17:51:30.625877+0000","last_became_active":"2024-04-20T17:48:55.061180+0000","last_became_peered":"2024-04-20T17:48:55.061180+0000","last_unstale":"2024-04-20T17:51:30.625877+0000","last_undegraded":"2024-04-20T17:51:30.625877+0000","last_fullsized":"2024-04-20T17:51:30.625877+0000","mapping_epoch":894,"log_start":"906'4200","ondisk_log_start":"906'4200","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9574","last_scrub_stamp":"2024-04-20T17:50:22.180501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:22.180501+0000","objects_scrubbed":9507,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:51.128747+0000","scrub_duration":1824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443219968,"num_objects":6763,"num_object_clones":0,"num_object_copies":13526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14215,"num_write_kb":671296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"15'137","reported_seq":940,"reported_epoch":947,"state":"active+clean","last_fresh":"2024-04-20T17:51:28.884447+0000","last_change":"2024-04-20T17:50:31.133012+0000","last_active":"2024-04-20T17:51:28.884447+0000","last_peered":"2024-04-20T17:51:28.884447+0000","last_clean":"2024-04-20T17:51:28.884447+0000","last_became_active":"2024-04-20T17:46:40.831961+0000","last_became_peered":"2024-04-20T17:46:40.831961+0000","last_unstale":"2024-04-20T17:51:28.884447+0000","last_undegraded":"2024-04-20T17:51:28.884447+0000","last_fullsized":"2024-04-20T17:51:28.884447+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:31.132879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:31.132879+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:31.822631+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"947'13804","reported_seq":14018,"reported_epoch":947,"state":"active+clean","last_fresh":"2024-04-20T17:51:28.986175+0000","last_change":"2024-04-20T17:50:19.657299+0000","last_active":"2024-04-20T17:51:28.986175+0000","last_peered":"2024-04-20T17:51:28.986175+0000","last_clean":"2024-04-20T17:51:28.986175+0000","last_became_active":"2024-04-20T17:48:54.816644+0000","last_became_peered":"2024-04-20T17:48:54.816644+0000","last_unstale":"2024-04-20T17:51:28.986175+0000","last_undegraded":"2024-04-20T17:51:28.986175+0000","last_fullsized":"2024-04-20T17:51:28.986175+0000","mapping_epoch":894,"log_start":"906'3800","ondisk_log_start":"906'3800","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"927'9204","last_scrub_stamp":"2024-04-20T17:50:19.657194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:19.657194+0000","objects_scrubbed":9124,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456261632,"num_objects":6962,"num_object_clones":0,"num_object_copies":13924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13804,"num_write_kb":664512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"15'137","reported_seq":736,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-04-20T17:51:23.841158+0000","last_change":"2024-04-20T17:51:00.057261+0000","last_active":"2024-04-20T17:51:23.841158+0000","last_peered":"2024-04-20T17:51:23.841158+0000","last_clean":"2024-04-20T17:51:23.841158+0000","last_became_active":"2024-04-20T17:49:46.824559+0000","last_became_peered":"2024-04-20T17:49:46.824559+0000","last_unstale":"2024-04-20T17:51:23.841158+0000","last_undegraded":"2024-04-20T17:51:23.841158+0000","last_fullsized":"2024-04-20T17:51:23.841158+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:00.057156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:00.057156+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:08.337431+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"947'14239","reported_seq":14462,"reported_epoch":947,"state":"active+clean","last_fresh":"2024-04-20T17:51:30.198435+0000","last_change":"2024-04-20T17:50:28.283117+0000","last_active":"2024-04-20T17:51:30.198435+0000","last_peered":"2024-04-20T17:51:30.198435+0000","last_clean":"2024-04-20T17:51:30.198435+0000","last_became_active":"2024-04-20T17:48:54.813199+0000","last_became_peered":"2024-04-20T17:48:54.813199+0000","last_unstale":"2024-04-20T17:51:30.198435+0000","last_undegraded":"2024-04-20T17:51:30.198435+0000","last_fullsized":"2024-04-20T17:51:30.198435+0000","mapping_epoch":894,"log_start":"906'4200","ondisk_log_start":"906'4200","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'10025","last_scrub_stamp":"2024-04-20T17:50:28.282957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:28.282957+0000","objects_scrubbed":9963,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:48.575002+0000","scrub_duration":2339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":461570048,"num_objects":7043,"num_object_clones":0,"num_object_copies":14086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14239,"num_write_kb":681024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"15'137","reported_seq":945,"reported_epoch":947,"state":"active+clean","last_fresh":"2024-04-20T17:51:28.898593+0000","last_change":"2024-04-20T17:50:58.932166+0000","last_active":"2024-04-20T17:51:28.898593+0000","last_peered":"2024-04-20T17:51:28.898593+0000","last_clean":"2024-04-20T17:51:28.898593+0000","last_became_active":"2024-04-20T17:45:32.782706+0000","last_became_peered":"2024-04-20T17:45:32.782706+0000","last_unstale":"2024-04-20T17:51:28.898593+0000","last_undegraded":"2024-04-20T17:51:28.898593+0000","last_fullsized":"2024-04-20T17:51:28.898593+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:58.932046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:58.932046+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:23.591238+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.2","version":"946'13869","reported_seq":14084,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-04-20T17:51:28.622428+0000","last_change":"2024-04-20T17:50:22.440005+0000","last_active":"2024-04-20T17:51:28.622428+0000","last_peered":"2024-04-20T17:51:28.622428+0000","last_clean":"2024-04-20T17:51:28.622428+0000","last_became_active":"2024-04-20T17:48:55.060965+0000","last_became_peered":"2024-04-20T17:48:55.060965+0000","last_unstale":"2024-04-20T17:51:28.622428+0000","last_undegraded":"2024-04-20T17:51:28.622428+0000","last_fullsized":"2024-04-20T17:51:28.622428+0000","mapping_epoch":894,"log_start":"906'3800","ondisk_log_start":"906'3800","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9408","last_scrub_stamp":"2024-04-20T17:50:22.439841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:22.439841+0000","objects_scrubbed":9355,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:32.571208+0000","scrub_duration":1641,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452919296,"num_objects":6911,"num_object_clones":0,"num_object_copies":13822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13869,"num_write_kb":664960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"15'137","reported_seq":925,"reported_epoch":948,"state":"active+clean","last_fresh":"2024-04-20T17:51:31.907982+0000","last_change":"2024-04-20T17:49:48.210622+0000","last_active":"2024-04-20T17:51:31.907982+0000","last_peered":"2024-04-20T17:51:31.907982+0000","last_clean":"2024-04-20T17:51:31.907982+0000","last_became_active":"2024-04-20T17:45:32.669451+0000","last_became_peered":"2024-04-20T17:45:32.669451+0000","last_unstale":"2024-04-20T17:51:31.907982+0000","last_undegraded":"2024-04-20T17:51:31.907982+0000","last_fullsized":"2024-04-20T17:51:31.907982+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:49:48.210506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:49:48.210506+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"947'14025","reported_seq":14248,"reported_epoch":947,"state":"active+clean","last_fresh":"2024-04-20T17:51:31.696977+0000","last_change":"2024-04-20T17:50:30.239298+0000","last_active":"2024-04-20T17:51:31.696977+0000","last_peered":"2024-04-20T17:51:31.696977+0000","last_clean":"2024-04-20T17:51:31.696977+0000","last_became_active":"2024-04-20T17:48:55.061200+0000","last_became_peered":"2024-04-20T17:48:55.061200+0000","last_unstale":"2024-04-20T17:51:31.696977+0000","last_undegraded":"2024-04-20T17:51:31.696977+0000","last_fullsized":"2024-04-20T17:51:31.696977+0000","mapping_epoch":894,"log_start":"906'4000","ondisk_log_start":"906'4000","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'9940","last_scrub_stamp":"2024-04-20T17:50:30.239131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:30.239131+0000","objects_scrubbed":9887,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:36.933027+0000","scrub_duration":1672,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440729600,"num_objects":6725,"num_object_clones":0,"num_object_copies":13450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14025,"num_write_kb":664000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"15'137","reported_seq":938,"reported_epoch":947,"state":"active+clean","last_fresh":"2024-04-20T17:51:29.065216+0000","last_change":"2024-04-20T17:50:23.398315+0000","last_active":"2024-04-20T17:51:29.065216+0000","last_peered":"2024-04-20T17:51:29.065216+0000","last_clean":"2024-04-20T17:51:29.065216+0000","last_became_active":"2024-04-20T17:45:32.644584+0000","last_became_peered":"2024-04-20T17:45:32.644584+0000","last_unstale":"2024-04-20T17:51:29.065216+0000","last_undegraded":"2024-04-20T17:51:29.065216+0000","last_fullsized":"2024-04-20T17:51:29.065216+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:23.398142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:23.398142+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:31.965635+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"946'13978","reported_seq":14199,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-04-20T17:51:28.617920+0000","last_change":"2024-04-20T17:50:30.712810+0000","last_active":"2024-04-20T17:51:28.617920+0000","last_peered":"2024-04-20T17:51:28.617920+0000","last_clean":"2024-04-20T17:51:28.617920+0000","last_became_active":"2024-04-20T17:48:54.813062+0000","last_became_peered":"2024-04-20T17:48:54.813062+0000","last_unstale":"2024-04-20T17:51:28.617920+0000","last_undegraded":"2024-04-20T17:51:28.617920+0000","last_fullsized":"2024-04-20T17:51:28.617920+0000","mapping_epoch":894,"log_start":"906'3900","ondisk_log_start":"906'3900","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'10026","last_scrub_stamp":"2024-04-20T17:50:30.712648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:30.712648+0000","objects_scrubbed":9967,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:58.995403+0000","scrub_duration":1871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454950912,"num_objects":6942,"num_object_clones":0,"num_object_copies":13884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13978,"num_write_kb":669440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"15'137","reported_seq":886,"reported_epoch":947,"state":"active+clean","last_fresh":"2024-04-20T17:51:28.884459+0000","last_change":"2024-04-20T17:50:33.137929+0000","last_active":"2024-04-20T17:51:28.884459+0000","last_peered":"2024-04-20T17:51:28.884459+0000","last_clean":"2024-04-20T17:51:28.884459+0000","last_became_active":"2024-04-20T17:46:40.881713+0000","last_became_peered":"2024-04-20T17:46:40.881713+0000","last_unstale":"2024-04-20T17:51:28.884459+0000","last_undegraded":"2024-04-20T17:51:28.884459+0000","last_fullsized":"2024-04-20T17:51:28.884459+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:33.137831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:33.137831+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:36.471855+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.8","version":"948'14275","reported_seq":14495,"reported_epoch":948,"state":"active+clean","last_fresh":"2024-04-20T17:51:32.192653+0000","last_change":"2024-04-20T17:50:19.940419+0000","last_active":"2024-04-20T17:51:32.192653+0000","last_peered":"2024-04-20T17:51:32.192653+0000","last_clean":"2024-04-20T17:51:32.192653+0000","last_became_active":"2024-04-20T17:48:54.813368+0000","last_became_peered":"2024-04-20T17:48:54.813368+0000","last_unstale":"2024-04-20T17:51:32.192653+0000","last_undegraded":"2024-04-20T17:51:32.192653+0000","last_fullsized":"2024-04-20T17:51:32.192653+0000","mapping_epoch":894,"log_start":"906'4200","ondisk_log_start":"906'4200","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"927'9391","last_scrub_stamp":"2024-04-20T17:50:19.940199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:19.940199+0000","objects_scrubbed":9334,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:34.165962+0000","scrub_duration":1977,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445186048,"num_objects":6793,"num_object_clones":0,"num_object_copies":13586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14275,"num_write_kb":674176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":884,"reported_epoch":947,"state":"active+clean","last_fresh":"2024-04-20T17:51:28.884431+0000","last_change":"2024-04-20T17:50:21.111183+0000","last_active":"2024-04-20T17:51:28.884431+0000","last_peered":"2024-04-20T17:51:28.884431+0000","last_clean":"2024-04-20T17:51:28.884431+0000","last_became_active":"2024-04-20T17:46:40.851683+0000","last_became_peered":"2024-04-20T17:46:40.851683+0000","last_unstale":"2024-04-20T17:51:28.884431+0000","last_undegraded":"2024-04-20T17:51:28.884431+0000","last_fullsized":"2024-04-20T17:51:28.884431+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:21.111059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:21.111059+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:45.968902+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":7172194328,"num_objects":109440,"num_object_clones":0,"num_object_copies":218880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1080,"num_objects_unfound":0,"num_objects_dirty":109440,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":224148,"num_write_kb":10674753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6732,"num_bytes_recovered":441122816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14706286592,"data_stored":14706278448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160748,"ondisk_log_size":160748,"up":32,"acting":32,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},"log_size":1507,"ondisk_log_size":1507,"up":22,"acting":22,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991783,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2264128,"kb_used_data":1330488,"kb_used_omap":1,"kb_used_meta":933566,"kb_avail":91481024,"statfs":{"total":95995035648,"available":93676568576,"internally_reserved":0,"allocated":1362419712,"data_stored":1362058511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":955972036},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253422,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3040388,"kb_used_data":2206520,"kb_used_omap":1,"kb_used_meta":833598,"kb_avail":90704764,"statfs":{"total":95995035648,"available":92881678336,"internally_reserved":0,"allocated":2259476480,"data_stored":2259115279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":853604800},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613171,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2037008,"kb_used_data":1288068,"kb_used_omap":1,"kb_used_meta":748926,"kb_avail":91708144,"statfs":{"total":95995035648,"available":93909139456,"internally_reserved":0,"allocated":1318981632,"data_stored":1318614506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":766900672},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809401,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1218804,"kb_used_data":782252,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":92526348,"statfs":{"total":95995035648,"available":94746980352,"internally_reserved":0,"allocated":801026048,"data_stored":800670872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":799,"seq":3431678869591,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1466824,"kb_used_data":1332156,"kb_used_omap":11,"kb_used_meta":134580,"kb_avail":92278328,"statfs":{"total":95995035648,"available":94493007872,"internally_reserved":0,"allocated":1364127744,"data_stored":1363762479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11990,"internal_metadata":137810218},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":535,"seq":2297807503631,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":477120,"kb_used_data":431224,"kb_used_omap":170,"kb_used_meta":45717,"kb_avail":93268032,"statfs":{"total":95995035648,"available":95506464768,"internally_reserved":0,"allocated":441573376,"data_stored":441212175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":174628,"internal_metadata":46814684},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":659,"seq":2830383448253,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3037112,"kb_used_data":2663216,"kb_used_omap":4,"kb_used_meta":373883,"kb_avail":90708040,"statfs":{"total":95995035648,"available":92885032960,"internally_reserved":0,"allocated":2727133184,"data_stored":2726773936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4410,"internal_metadata":382856902},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738976,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":744068,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":93001084,"statfs":{"total":95995035648,"available":95233110016,"internally_reserved":0,"allocated":712704,"data_stored":351503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394265,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":609740,"kb_used_data":190656,"kb_used_omap":202,"kb_used_meta":418869,"kb_avail":93135412,"statfs":{"total":95995035648,"available":95370661888,"internally_reserved":0,"allocated":195231744,"data_stored":194866991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":428922327},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653314,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2342708,"kb_used_data":1773036,"kb_used_omap":4,"kb_used_meta":569659,"kb_avail":91402444,"statfs":{"total":95995035648,"available":93596102656,"internally_reserved":0,"allocated":1815588864,"data_stored":1815233688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":583331527},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986763,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1508288,"kb_used_data":873208,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":92236864,"statfs":{"total":95995035648,"available":94450548736,"internally_reserved":0,"allocated":894164992,"data_stored":893803791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790533,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2826636,"kb_used_data":1943040,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":90918516,"statfs":{"total":95995035648,"available":93100560384,"internally_reserved":0,"allocated":1989672960,"data_stored":1989308207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":904788421},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2715684864,"data_stored":2715680792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1813643264,"data_stored":1813643264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":889847808,"data_stored":889847808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1776746496,"data_stored":1776746496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":440729600,"data_stored":440729600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1353715712,"data_stored":1353711640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":800456704,"data_stored":800456704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1313734656,"data_stored":1313734656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2252406784,"data_stored":2252406784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1349320704,"data_stored":1349320704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:51:34.692 INFO:tasks.cephadm.ceph_manager.ceph:PG 9.3 is not active+clean 2024-04-20T17:51:34.692 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '9.3', 'version': "948'13999", 'reported_seq': 27690, 'reported_epoch': 948, 'state': 'active+remapped+backfilling', 'last_fresh': '2024-04-20T17:51:32.203234+0000', 'last_change': '2024-04-20T17:50:46.276928+0000', 'last_active': '2024-04-20T17:51:32.203234+0000', 'last_peered': '2024-04-20T17:51:32.203234+0000', 'last_clean': '2024-04-20T17:50:43.748764+0000', 'last_became_active': '2024-04-20T17:50:46.159037+0000', 'last_became_peered': '2024-04-20T17:50:46.159037+0000', 'last_unstale': '2024-04-20T17:51:32.203234+0000', 'last_undegraded': '2024-04-20T17:51:32.203234+0000', 'last_fullsized': '2024-04-20T17:51:32.203234+0000', 'mapping_epoch': 935, 'log_start': "906'3900", 'ondisk_log_start': "906'3900", 'created': 894, 'last_epoch_clean': 895, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "930'9622", 'last_scrub_stamp': '2024-04-20T17:50:27.118695+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T17:48:53.794927+0000', 'last_clean_scrub_stamp': '2024-04-20T17:50:27.118695+0000', 'objects_scrubbed': 9558, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2200, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 431030272, 'num_objects': 6577, 'num_object_clones': 0, 'num_object_copies': 13154, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1080, 'num_objects_unfound': 0, 'num_objects_dirty': 6577, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13999, 'num_write_kb': 658432, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6732, 'num_bytes_recovered': 441122816, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [8, 3], 'acting': [9, 3], 'avail_no_missing': ['9', '3'], 'object_location_counts': [{'shards': '3,9', 'objects': 6577}], 'blocked_by': [], 'up_primary': 8, 'acting_primary': 9, 'purged_snaps': []} 2024-04-20T17:51:34.692 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph status --format=json 2024-04-20T17:51:34.710 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:34.785 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:51:34.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:34.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:51:34.893 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:51:34.927 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:34.928 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:34 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:51:34.928 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:34 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:51:34.928 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:34 smithi005 ceph-mon[39562]: pgmap v2502: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 1080/216042 objects misplaced (0.500%); 11 MiB/s, 183 objects/s recovering 2024-04-20T17:51:34.952 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:51:34.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:34.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:51:35.059 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:51:35.097 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:35.124 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:51:35.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:35.130 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:51:35.152 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-20T17:51:35.182 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:34 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:51:35.182 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:34 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:51:35.182 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:34 smithi106 ceph-mon[44423]: pgmap v2502: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 1080/216042 objects misplaced (0.500%); 11 MiB/s, 183 objects/s recovering 2024-04-20T17:51:35.225 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:51:35.225 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:51:35.225 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:51:35.225 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:51:35.226 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:51:35.226 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:51:35.226 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:51:35.226 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:51:35.231 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:51:35.238 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:51:35.278 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:51:35.292 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:35.317 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:51:35.318 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:51:35.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:34 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:51:35.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:34 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:51:35.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:34 smithi171 ceph-mon[44189]: pgmap v2502: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 1080/216042 objects misplaced (0.500%); 11 MiB/s, 183 objects/s recovering 2024-04-20T17:51:35.334 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:51:35.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:35.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:51:35.366 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:51:35.370 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:51:35.370 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:51:35.370 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:51:35.370 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:51:35.371 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:51:35.371 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:51:35.371 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:51:35.371 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:51:35.371 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:51:35.371 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:51:35.371 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:51:35.371 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:51:35.438 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:51:35.477 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:35.501 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:51:35.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:35.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:51:35.528 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:51:35.608 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:51:35.646 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:35.673 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:51:35.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:35.680 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:51:35.781 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:51:35.825 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:35.863 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:51:35.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:35.870 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:51:35.933 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:35 smithi171 ceph-mon[44189]: osdmap e949: 12 total, 12 up, 12 in 2024-04-20T17:51:35.933 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:35 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4271742173' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:51:35.934 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:35.934 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:35.970 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:51:36.009 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:36.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:35 smithi005 ceph-mon[39562]: osdmap e949: 12 total, 12 up, 12 in 2024-04-20T17:51:36.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:35 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4271742173' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:51:36.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:36.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:36.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:36.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:36.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:36.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:36.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:36.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:36.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:35 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:35 smithi106 ceph-mon[44423]: osdmap e949: 12 total, 12 up, 12 in 2024-04-20T17:51:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:35 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4271742173' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:51:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:36.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:35 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:36.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:36.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:36.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:36.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:36.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:36.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:36.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:35 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:37.173 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:37.173 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:37.173 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:37.173 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:36 smithi005 ceph-mon[39562]: pgmap v2504: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 6.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s; 1080/209556 objects misplaced (0.515%); 5.8 MiB/s, 92 objects/s recovering 2024-04-20T17:51:37.173 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:51:37.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:37.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:37.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:37.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:36 smithi106 ceph-mon[44423]: pgmap v2504: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 6.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s; 1080/209556 objects misplaced (0.515%); 5.8 MiB/s, 92 objects/s recovering 2024-04-20T17:51:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:37.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:37.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:36 smithi171 ceph-mon[44189]: pgmap v2504: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 6.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s; 1080/209556 objects misplaced (0.515%); 5.8 MiB/s, 92 objects/s recovering 2024-04-20T17:51:38.373 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:51:38.374 INFO:teuthology.orchestra.run.smithi005.stdout:{"fsid":"13c6e76c-ff37-11ee-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":3066,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":949,"num_osds":12,"num_up_osds":12,"osd_up_since":1713635059,"num_in_osds":12,"osd_in_since":1713635196,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":26},{"state_name":"active+remapped+backfilling","count":1}],"num_pgs":27,"num_pools":2,"num_objects":104778,"data_bytes":6867124792,"bytes_used":21434990592,"bytes_avail":1130505437184,"bytes_total":1151940427776,"misplaced_objects":1080,"misplaced_total":209556,"misplaced_ratio":0.0051537536505755037,"recovering_objects_per_sec":92,"recovering_bytes_per_sec":6070346,"recovering_keys_per_sec":0,"num_objects_recovered":724,"num_bytes_recovered":47448064,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":1188},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.5:8443/"}},"servicemap":{"epoch":70,"modified":"2024-04-20T17:50:40.688583+0000","services":{}},"progress_events":{}} 2024-04-20T17:51:38.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:38 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/850880537' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:51:38.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:38 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/850880537' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:51:38.843 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:38 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/850880537' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:51:38.975 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:51:38.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:38.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:51:39.080 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:51:39.128 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:39.172 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:51:39.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:39.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:51:39.214 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:51:39.280 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:51:39.313 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:39.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:39 smithi005 ceph-mon[39562]: pgmap v2505: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 6.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 327/205382 objects misplaced (0.159%); 11 MiB/s, 180 objects/s recovering 2024-04-20T17:51:39.516 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-20T17:51:39.516 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:51:39.516 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:51:39.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:39.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:51:39.621 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:51:39.648 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:39.667 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:51:39.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:39.671 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:51:39.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:39 smithi106 ceph-mon[44423]: pgmap v2505: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 6.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 327/205382 objects misplaced (0.159%); 11 MiB/s, 180 objects/s recovering 2024-04-20T17:51:39.772 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:51:39.796 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:39 smithi171 ceph-mon[44189]: pgmap v2505: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 6.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 327/205382 objects misplaced (0.159%); 11 MiB/s, 180 objects/s recovering 2024-04-20T17:51:39.798 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:39.818 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:51:39.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:39.821 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:51:39.922 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:51:39.951 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:40.004 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:51:40.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:40.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:51:40.054 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:51:40.058 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:51:40.107 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:51:40.134 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:40.136 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:51:40.136 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-04-20T17:51:40.137 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-04-20T17:51:40.137 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:51:40.155 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:51:40.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:40.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:51:40.259 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:51:40.288 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:40.447 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:51:40.487 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:51:40.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:40.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:51:40.540 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:40 smithi005 ceph-mon[39562]: osdmap e950: 12 total, 12 up, 12 in 2024-04-20T17:51:40.540 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:40 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4127472761' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:51:40.540 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:40 smithi005 ceph-mon[39562]: from='client.28304 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:51:40.591 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:51:40.638 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:40.664 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:51:40.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:40.671 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:51:40.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:40 smithi106 ceph-mon[44423]: osdmap e950: 12 total, 12 up, 12 in 2024-04-20T17:51:40.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:40 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4127472761' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:51:40.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:40 smithi106 ceph-mon[44423]: from='client.28304 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:51:40.772 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:51:40.815 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:40.816 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:40 smithi171 ceph-mon[44189]: osdmap e950: 12 total, 12 up, 12 in 2024-04-20T17:51:40.816 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:40 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4127472761' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:51:40.816 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:40 smithi171 ceph-mon[44189]: from='client.28304 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:51:40.848 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:51:40.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:40.853 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:51:40.908 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-20T17:51:40.953 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:51:40.981 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:41.003 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:51:41.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:41.011 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:51:41.112 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:51:41.148 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:41.179 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:51:41.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:41.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:51:41.287 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:51:41.299 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:51:41.325 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:41.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:41 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:51:41.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:41 smithi106 ceph-mon[44423]: from='client.28304 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:51:41.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:41 smithi106 ceph-mon[44423]: osdmap e951: 12 total, 12 up, 12 in 2024-04-20T17:51:41.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:41 smithi106 ceph-mon[44423]: 1.8 scrub starts 2024-04-20T17:51:41.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:41 smithi106 ceph-mon[44423]: pgmap v2508: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 6.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.31k op/s; 327/200286 objects misplaced (0.163%); 7.5 MiB/s, 120 objects/s recovering 2024-04-20T17:51:41.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:41 smithi106 ceph-mon[44423]: 1.8 scrub ok 2024-04-20T17:51:41.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:41 smithi106 ceph-mon[44423]: 9.5 scrub starts 2024-04-20T17:51:41.584 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:51:41.587 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:41 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:51:41.587 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:41 smithi005 ceph-mon[39562]: from='client.28304 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:51:41.587 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:41 smithi005 ceph-mon[39562]: osdmap e951: 12 total, 12 up, 12 in 2024-04-20T17:51:41.587 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:41 smithi005 ceph-mon[39562]: 1.8 scrub starts 2024-04-20T17:51:41.587 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:41 smithi005 ceph-mon[39562]: pgmap v2508: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 6.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.31k op/s; 327/200286 objects misplaced (0.163%); 7.5 MiB/s, 120 objects/s recovering 2024-04-20T17:51:41.587 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:41 smithi005 ceph-mon[39562]: 1.8 scrub ok 2024-04-20T17:51:41.587 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:41 smithi005 ceph-mon[39562]: 9.5 scrub starts 2024-04-20T17:51:41.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:41.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:51:41.689 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:51:41.722 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:41 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:51:41.722 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:41 smithi171 ceph-mon[44189]: from='client.28304 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:51:41.722 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:41 smithi171 ceph-mon[44189]: osdmap e951: 12 total, 12 up, 12 in 2024-04-20T17:51:41.722 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:41 smithi171 ceph-mon[44189]: 1.8 scrub starts 2024-04-20T17:51:41.723 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:41 smithi171 ceph-mon[44189]: pgmap v2508: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 6.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.31k op/s; 327/200286 objects misplaced (0.163%); 7.5 MiB/s, 120 objects/s recovering 2024-04-20T17:51:41.723 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:41 smithi171 ceph-mon[44189]: 1.8 scrub ok 2024-04-20T17:51:41.723 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:41 smithi171 ceph-mon[44189]: 9.5 scrub starts 2024-04-20T17:51:41.724 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:41.770 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:51:41.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:41.778 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:51:41.879 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:51:41.912 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:42.124 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:51:42.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:42.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:51:42.239 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:51:42.282 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:42.688 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:42 smithi005 ceph-mon[39562]: 1.a scrub starts 2024-04-20T17:51:42.688 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:42 smithi005 ceph-mon[39562]: 9.0 scrub starts 2024-04-20T17:51:42.688 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:42 smithi005 ceph-mon[39562]: 1.a scrub ok 2024-04-20T17:51:42.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:42 smithi106 ceph-mon[44423]: 1.a scrub starts 2024-04-20T17:51:42.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:42 smithi106 ceph-mon[44423]: 9.0 scrub starts 2024-04-20T17:51:42.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:42 smithi106 ceph-mon[44423]: 1.a scrub ok 2024-04-20T17:51:42.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:42 smithi171 ceph-mon[44189]: 1.a scrub starts 2024-04-20T17:51:42.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:42 smithi171 ceph-mon[44189]: 9.0 scrub starts 2024-04-20T17:51:42.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:42 smithi171 ceph-mon[44189]: 1.a scrub ok 2024-04-20T17:51:43.450 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:51:43.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:43.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:51:43.554 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:51:43.594 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:43.653 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:51:43.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:43.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:51:43.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:43 smithi005 ceph-mon[39562]: 9.3 scrub starts 2024-04-20T17:51:43.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:43 smithi005 ceph-mon[39562]: 9.f scrub starts 2024-04-20T17:51:43.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:43 smithi005 ceph-mon[39562]: 9.2 scrub starts 2024-04-20T17:51:43.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:43 smithi005 ceph-mon[39562]: 9.5 scrub ok 2024-04-20T17:51:43.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:43 smithi005 ceph-mon[39562]: 9.8 scrub starts 2024-04-20T17:51:43.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:43 smithi005 ceph-mon[39562]: pgmap v2509: 27 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 25 active+clean; 6.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 327/197948 objects misplaced (0.165%); 5.8 MiB/s, 92 objects/s recovering 2024-04-20T17:51:43.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:43 smithi005 ceph-mon[39562]: 1.7 scrub starts 2024-04-20T17:51:43.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:43 smithi005 ceph-mon[39562]: 9.0 scrub ok 2024-04-20T17:51:43.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:43 smithi005 ceph-mon[39562]: 1.7 scrub ok 2024-04-20T17:51:43.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:43 smithi005 ceph-mon[39562]: 1.2 scrub starts 2024-04-20T17:51:43.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:43 smithi005 ceph-mon[39562]: 1.2 scrub ok 2024-04-20T17:51:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:43 smithi106 ceph-mon[44423]: 9.3 scrub starts 2024-04-20T17:51:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:43 smithi106 ceph-mon[44423]: 9.f scrub starts 2024-04-20T17:51:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:43 smithi106 ceph-mon[44423]: 9.2 scrub starts 2024-04-20T17:51:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:43 smithi106 ceph-mon[44423]: 9.5 scrub ok 2024-04-20T17:51:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:43 smithi106 ceph-mon[44423]: 9.8 scrub starts 2024-04-20T17:51:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:43 smithi106 ceph-mon[44423]: pgmap v2509: 27 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 25 active+clean; 6.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 327/197948 objects misplaced (0.165%); 5.8 MiB/s, 92 objects/s recovering 2024-04-20T17:51:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:43 smithi106 ceph-mon[44423]: 1.7 scrub starts 2024-04-20T17:51:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:43 smithi106 ceph-mon[44423]: 9.0 scrub ok 2024-04-20T17:51:43.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:43 smithi106 ceph-mon[44423]: 1.7 scrub ok 2024-04-20T17:51:43.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:43 smithi106 ceph-mon[44423]: 1.2 scrub starts 2024-04-20T17:51:43.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:43 smithi106 ceph-mon[44423]: 1.2 scrub ok 2024-04-20T17:51:43.762 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:51:43.803 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:43 smithi171 ceph-mon[44189]: 9.3 scrub starts 2024-04-20T17:51:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:43 smithi171 ceph-mon[44189]: 9.f scrub starts 2024-04-20T17:51:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:43 smithi171 ceph-mon[44189]: 9.2 scrub starts 2024-04-20T17:51:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:43 smithi171 ceph-mon[44189]: 9.5 scrub ok 2024-04-20T17:51:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:43 smithi171 ceph-mon[44189]: 9.8 scrub starts 2024-04-20T17:51:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:43 smithi171 ceph-mon[44189]: pgmap v2509: 27 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 25 active+clean; 6.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 327/197948 objects misplaced (0.165%); 5.8 MiB/s, 92 objects/s recovering 2024-04-20T17:51:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:43 smithi171 ceph-mon[44189]: 1.7 scrub starts 2024-04-20T17:51:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:43 smithi171 ceph-mon[44189]: 9.0 scrub ok 2024-04-20T17:51:43.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:43 smithi171 ceph-mon[44189]: 1.7 scrub ok 2024-04-20T17:51:43.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:43 smithi171 ceph-mon[44189]: 1.2 scrub starts 2024-04-20T17:51:43.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:43 smithi171 ceph-mon[44189]: 1.2 scrub ok 2024-04-20T17:51:44.475 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:51:44.504 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:51:44.534 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:51:44.565 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:51:44.587 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:51:44.614 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:51:44.640 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:51:44.662 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:51:44.683 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:51:44.708 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:51:44.710 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:51:44.710 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:51:44.730 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:51:44.739 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:44 smithi106 ceph-mon[44423]: 9.3 scrub ok 2024-04-20T17:51:44.739 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:44 smithi106 ceph-mon[44423]: 9.f scrub ok 2024-04-20T17:51:44.739 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:44 smithi106 ceph-mon[44423]: 9.2 scrub ok 2024-04-20T17:51:44.739 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09951558437675598]}]: dispatch 2024-04-20T17:51:44.739 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08450765134908735]}]: dispatch 2024-04-20T17:51:44.739 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09042405487558114]}]: dispatch 2024-04-20T17:51:44.739 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08529706966127526]}]: dispatch 2024-04-20T17:51:44.739 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09042405487558114]}]: dispatch 2024-04-20T17:51:44.739 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08514700444371495]}]: dispatch 2024-04-20T17:51:44.740 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08601959369779835]}]: dispatch 2024-04-20T17:51:44.740 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08458233113995788]}]: dispatch 2024-04-20T17:51:44.740 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08838441454004171]}]: dispatch 2024-04-20T17:51:44.740 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08446284347456504]}]: dispatch 2024-04-20T17:51:44.740 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08363477984067626]}]: dispatch 2024-04-20T17:51:44.740 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08514700444371495]}]: dispatch 2024-04-20T17:51:44.740 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:44 smithi106 ceph-mon[44423]: 9.8 scrub ok 2024-04-20T17:51:44.740 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:44 smithi106 ceph-mon[44423]: 1.4 scrub starts 2024-04-20T17:51:44.740 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:44 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3933337968' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:51:44.751 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:51:44.752 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:51:44.752 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:51:44.752 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:51:44.752 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:51:44.752 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:51:44.752 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:51:44.753 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:51:44.753 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:51:44.753 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:51:44.753 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:51:44.753 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:51:44.753 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:51:44.778 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:44 smithi005 ceph-mon[39562]: 9.3 scrub ok 2024-04-20T17:51:44.778 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:44 smithi005 ceph-mon[39562]: 9.f scrub ok 2024-04-20T17:51:44.778 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:44 smithi005 ceph-mon[39562]: 9.2 scrub ok 2024-04-20T17:51:44.778 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09951558437675598]}]: dispatch 2024-04-20T17:51:44.778 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08450765134908735]}]: dispatch 2024-04-20T17:51:44.778 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09042405487558114]}]: dispatch 2024-04-20T17:51:44.778 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08529706966127526]}]: dispatch 2024-04-20T17:51:44.778 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09042405487558114]}]: dispatch 2024-04-20T17:51:44.778 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08514700444371495]}]: dispatch 2024-04-20T17:51:44.778 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08601959369779835]}]: dispatch 2024-04-20T17:51:44.778 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08458233113995788]}]: dispatch 2024-04-20T17:51:44.778 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08838441454004171]}]: dispatch 2024-04-20T17:51:44.779 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08446284347456504]}]: dispatch 2024-04-20T17:51:44.779 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08363477984067626]}]: dispatch 2024-04-20T17:51:44.779 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08514700444371495]}]: dispatch 2024-04-20T17:51:44.779 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:44 smithi005 ceph-mon[39562]: 9.8 scrub ok 2024-04-20T17:51:44.779 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:44 smithi005 ceph-mon[39562]: 1.4 scrub starts 2024-04-20T17:51:44.779 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:44 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3933337968' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:51:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:44 smithi171 ceph-mon[44189]: 9.3 scrub ok 2024-04-20T17:51:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:44 smithi171 ceph-mon[44189]: 9.f scrub ok 2024-04-20T17:51:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:44 smithi171 ceph-mon[44189]: 9.2 scrub ok 2024-04-20T17:51:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09951558437675598]}]: dispatch 2024-04-20T17:51:44.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08450765134908735]}]: dispatch 2024-04-20T17:51:44.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09042405487558114]}]: dispatch 2024-04-20T17:51:44.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08529706966127526]}]: dispatch 2024-04-20T17:51:44.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09042405487558114]}]: dispatch 2024-04-20T17:51:44.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08514700444371495]}]: dispatch 2024-04-20T17:51:44.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08601959369779835]}]: dispatch 2024-04-20T17:51:44.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08458233113995788]}]: dispatch 2024-04-20T17:51:44.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08838441454004171]}]: dispatch 2024-04-20T17:51:44.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08446284347456504]}]: dispatch 2024-04-20T17:51:44.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08363477984067626]}]: dispatch 2024-04-20T17:51:44.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08514700444371495]}]: dispatch 2024-04-20T17:51:44.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:44 smithi171 ceph-mon[44189]: 9.8 scrub ok 2024-04-20T17:51:44.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:44 smithi171 ceph-mon[44189]: 1.4 scrub starts 2024-04-20T17:51:44.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:44 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3933337968' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:51:45.071 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:51:45.420 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-20T17:51:45.685 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:51:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09951558437675598]}]': finished 2024-04-20T17:51:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08450765134908735]}]': finished 2024-04-20T17:51:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09042405487558114]}]': finished 2024-04-20T17:51:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08529706966127526]}]': finished 2024-04-20T17:51:45.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09042405487558114]}]': finished 2024-04-20T17:51:45.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08514700444371495]}]': finished 2024-04-20T17:51:45.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08601959369779835]}]': finished 2024-04-20T17:51:45.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08458233113995788]}]': finished 2024-04-20T17:51:45.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08838441454004171]}]': finished 2024-04-20T17:51:45.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08446284347456504]}]': finished 2024-04-20T17:51:45.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08363477984067626]}]': finished 2024-04-20T17:51:45.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08514700444371495]}]': finished 2024-04-20T17:51:45.762 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: osdmap e952: 12 total, 12 up, 12 in 2024-04-20T17:51:45.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: 1.4 scrub ok 2024-04-20T17:51:45.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: pgmap v2511: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 6.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 867 op/s; 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:51:45.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: 1.9 scrub starts 2024-04-20T17:51:45.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:45 smithi106 ceph-mon[44423]: 1.9 scrub ok 2024-04-20T17:51:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09951558437675598]}]': finished 2024-04-20T17:51:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08450765134908735]}]': finished 2024-04-20T17:51:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09042405487558114]}]': finished 2024-04-20T17:51:45.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08529706966127526]}]': finished 2024-04-20T17:51:45.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09042405487558114]}]': finished 2024-04-20T17:51:45.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08514700444371495]}]': finished 2024-04-20T17:51:45.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08601959369779835]}]': finished 2024-04-20T17:51:45.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08458233113995788]}]': finished 2024-04-20T17:51:45.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08838441454004171]}]': finished 2024-04-20T17:51:45.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08446284347456504]}]': finished 2024-04-20T17:51:45.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08363477984067626]}]': finished 2024-04-20T17:51:45.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08514700444371495]}]': finished 2024-04-20T17:51:45.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: osdmap e952: 12 total, 12 up, 12 in 2024-04-20T17:51:45.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: 1.4 scrub ok 2024-04-20T17:51:45.835 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: pgmap v2511: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 6.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 867 op/s; 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:51:45.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: 1.9 scrub starts 2024-04-20T17:51:45.836 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:45 smithi171 ceph-mon[44189]: 1.9 scrub ok 2024-04-20T17:51:45.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.09951558437675598]}]': finished 2024-04-20T17:51:45.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08450765134908735]}]': finished 2024-04-20T17:51:45.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09042405487558114]}]': finished 2024-04-20T17:51:45.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08529706966127526]}]': finished 2024-04-20T17:51:45.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09042405487558114]}]': finished 2024-04-20T17:51:45.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08514700444371495]}]': finished 2024-04-20T17:51:45.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08601959369779835]}]': finished 2024-04-20T17:51:45.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08458233113995788]}]': finished 2024-04-20T17:51:45.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08838441454004171]}]': finished 2024-04-20T17:51:45.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08446284347456504]}]': finished 2024-04-20T17:51:45.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08363477984067626]}]': finished 2024-04-20T17:51:45.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08514700444371495]}]': finished 2024-04-20T17:51:45.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: osdmap e952: 12 total, 12 up, 12 in 2024-04-20T17:51:45.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: 1.4 scrub ok 2024-04-20T17:51:45.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: pgmap v2511: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 6.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 867 op/s; 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:51:45.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:45.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: 1.9 scrub starts 2024-04-20T17:51:45.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:45 smithi005 ceph-mon[39562]: 1.9 scrub ok 2024-04-20T17:51:46.361 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:51:46.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:46.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:51:46.465 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:51:46.497 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:46.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:46 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:46.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:46 smithi106 ceph-mon[44423]: osdmap e953: 12 total, 12 up, 12 in 2024-04-20T17:51:46.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:46 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:51:46.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:46 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:51:46.761 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:46 smithi106 ceph-mon[44423]: 1.5 scrub starts 2024-04-20T17:51:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:46 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:46 smithi171 ceph-mon[44189]: osdmap e953: 12 total, 12 up, 12 in 2024-04-20T17:51:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:46 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:51:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:46 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:51:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:46 smithi171 ceph-mon[44189]: 1.5 scrub starts 2024-04-20T17:51:46.879 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:46 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:46.879 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:46 smithi005 ceph-mon[39562]: osdmap e953: 12 total, 12 up, 12 in 2024-04-20T17:51:46.879 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:46 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:51:46.879 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:46 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:51:46.879 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:46 smithi005 ceph-mon[39562]: 1.5 scrub starts 2024-04-20T17:51:46.948 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:51:46.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:46.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:51:47.052 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:51:47.085 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:47.120 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:51:47.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:47.280 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:51:47.381 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:51:47.429 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:47.647 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:47 smithi106 ceph-mon[44423]: 9.b scrub starts 2024-04-20T17:51:47.648 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:47 smithi106 ceph-mon[44423]: 9.b scrub ok 2024-04-20T17:51:47.648 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:47 smithi106 ceph-mon[44423]: 1.5 scrub ok 2024-04-20T17:51:47.648 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:47 smithi106 ceph-mon[44423]: 9.6 starting backfill to osd.2 from (0'0,0'0] MAX to 951'14963 2024-04-20T17:51:47.648 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:47 smithi106 ceph-mon[44423]: osdmap e954: 12 total, 12 up, 12 in 2024-04-20T17:51:47.648 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:47 smithi106 ceph-mon[44423]: pgmap v2514: 27 pgs: 1 remapped+peering, 1 active+clean+scrubbing, 25 active+clean; 5.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 850 op/s; 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:51:47.684 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:51:47.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:47.690 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:47 smithi005 ceph-mon[39562]: 9.b scrub starts 2024-04-20T17:51:47.690 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:47 smithi005 ceph-mon[39562]: 9.b scrub ok 2024-04-20T17:51:47.690 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:47 smithi005 ceph-mon[39562]: 1.5 scrub ok 2024-04-20T17:51:47.690 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:47 smithi005 ceph-mon[39562]: 9.6 starting backfill to osd.2 from (0'0,0'0] MAX to 951'14963 2024-04-20T17:51:47.690 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:47 smithi005 ceph-mon[39562]: osdmap e954: 12 total, 12 up, 12 in 2024-04-20T17:51:47.690 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:47 smithi005 ceph-mon[39562]: pgmap v2514: 27 pgs: 1 remapped+peering, 1 active+clean+scrubbing, 25 active+clean; 5.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 850 op/s; 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:51:47.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:51:47.792 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:51:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:47 smithi171 ceph-mon[44189]: 9.b scrub starts 2024-04-20T17:51:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:47 smithi171 ceph-mon[44189]: 9.b scrub ok 2024-04-20T17:51:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:47 smithi171 ceph-mon[44189]: 1.5 scrub ok 2024-04-20T17:51:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:47 smithi171 ceph-mon[44189]: 9.6 starting backfill to osd.2 from (0'0,0'0] MAX to 951'14963 2024-04-20T17:51:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:47 smithi171 ceph-mon[44189]: osdmap e954: 12 total, 12 up, 12 in 2024-04-20T17:51:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:47 smithi171 ceph-mon[44189]: pgmap v2514: 27 pgs: 1 remapped+peering, 1 active+clean+scrubbing, 25 active+clean; 5.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 850 op/s; 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:51:47.836 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:47.940 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:51:47.993 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:51:47.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:47.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:51:48.075 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:51:48.078 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:51:48.098 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:51:48.130 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:48.164 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:51:48.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:48.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:51:48.268 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:51:48.351 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:48.380 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:51:48.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:48.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:51:48.484 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:51:48.532 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:48.624 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:48 smithi106 ceph-mon[44423]: 9.4 scrub starts 2024-04-20T17:51:48.624 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:48 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:51:48.624 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:48 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:51:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:48 smithi171 ceph-mon[44189]: 9.4 scrub starts 2024-04-20T17:51:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:48 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:51:48.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:48 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:51:48.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:48 smithi005 ceph-mon[39562]: 9.4 scrub starts 2024-04-20T17:51:48.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:48 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:51:48.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:48 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:51:49.320 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:51:49.320 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:51:49.514 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:51:49.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:49.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:51:49.617 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:51:49.669 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:49.709 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:51:49.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:49 smithi005 ceph-mon[39562]: from='client.27949 v1:172.21.15.5:0/1679518753' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:51:49.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:49 smithi005 ceph-mon[39562]: pgmap v2515: 27 pgs: 1 remapped+peering, 26 active+clean; 5.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 824 op/s; 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:51:49.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:49 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:51:49.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:49 smithi005 ceph-mon[39562]: 9.4 scrub ok 2024-04-20T17:51:49.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:49 smithi005 ceph-mon[39562]: 9.9 scrub starts 2024-04-20T17:51:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:49 smithi171 ceph-mon[44189]: from='client.27949 v1:172.21.15.5:0/1679518753' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:51:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:49 smithi171 ceph-mon[44189]: pgmap v2515: 27 pgs: 1 remapped+peering, 26 active+clean; 5.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 824 op/s; 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:51:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:49 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:51:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:49 smithi171 ceph-mon[44189]: 9.4 scrub ok 2024-04-20T17:51:49.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:49 smithi171 ceph-mon[44189]: 9.9 scrub starts 2024-04-20T17:51:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:49 smithi106 ceph-mon[44423]: from='client.27949 v1:172.21.15.5:0/1679518753' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:51:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:49 smithi106 ceph-mon[44423]: pgmap v2515: 27 pgs: 1 remapped+peering, 26 active+clean; 5.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 824 op/s; 3.4 MiB/s, 54 objects/s recovering 2024-04-20T17:51:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:49 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:51:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:49 smithi106 ceph-mon[44423]: 9.4 scrub ok 2024-04-20T17:51:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:49 smithi106 ceph-mon[44423]: 9.9 scrub starts 2024-04-20T17:51:50.069 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":2514,"stamp":"2024-04-20T17:51:46.704812+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6228804152,"num_objects":95038,"num_object_clones":0,"num_object_copies":190076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95038,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":238552,"num_write_kb":10674753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7778,"num_bytes_recovered":509607936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162259,"ondisk_log_size":162259,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":19260632,"kb_used_data":12519320,"kb_used_omap":404,"kb_used_meta":6740715,"kb_avail":1105681192,"statfs":{"total":1151940427776,"available":1132217540608,"internally_reserved":0,"allocated":12819783680,"data_stored":12815106998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":414014,"internal_metadata":6902492866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,1,9,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-334561280,"num_objects":-5105,"num_object_clones":0,"num_object_copies":-10210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-327,"num_objects_unfound":0,"num_objects_dirty":-5105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5105,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":325,"num_bytes_recovered":21233664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001093"},"pg_stats":[{"pgid":"9.7","version":"952'14801","reported_seq":15030,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-04-20T17:51:44.731768+0000","last_change":"2024-04-20T17:50:27.211659+0000","last_active":"2024-04-20T17:51:44.731768+0000","last_peered":"2024-04-20T17:51:44.731768+0000","last_clean":"2024-04-20T17:51:44.731768+0000","last_became_active":"2024-04-20T17:48:54.814508+0000","last_became_peered":"2024-04-20T17:48:54.814508+0000","last_unstale":"2024-04-20T17:51:44.731768+0000","last_undegraded":"2024-04-20T17:51:44.731768+0000","last_fullsized":"2024-04-20T17:51:44.731768+0000","mapping_epoch":894,"log_start":"908'4800","ondisk_log_start":"908'4800","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'9665","last_scrub_stamp":"2024-04-20T17:50:27.211421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:27.211421+0000","objects_scrubbed":9588,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:56.034157+0000","scrub_duration":2310,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384106496,"num_objects":5861,"num_object_clones":0,"num_object_copies":11722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14801,"num_write_kb":661184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"951'14963","reported_seq":15189,"reported_epoch":953,"state":"remapped+peering","last_fresh":"2024-04-20T17:51:45.488893+0000","last_change":"2024-04-20T17:51:45.488893+0000","last_active":"2024-04-20T17:51:45.488893+0000","last_peered":"2024-04-20T17:51:44.458787+0000","last_clean":"2024-04-20T17:51:44.458787+0000","last_became_active":"2024-04-20T17:48:54.818033+0000","last_became_peered":"2024-04-20T17:48:54.818033+0000","last_unstale":"2024-04-20T17:51:45.488893+0000","last_undegraded":"2024-04-20T17:51:45.488893+0000","last_fullsized":"2024-04-20T17:51:45.488893+0000","mapping_epoch":953,"log_start":"908'4900","ondisk_log_start":"908'4900","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"929'9608","last_scrub_stamp":"2024-04-20T17:50:25.096273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:25.096273+0000","objects_scrubbed":9524,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381222912,"num_objects":5817,"num_object_clones":0,"num_object_copies":11634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14963,"num_write_kb":664960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"952'14888","reported_seq":15187,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-04-20T17:51:44.715608+0000","last_change":"2024-04-20T17:51:42.519313+0000","last_active":"2024-04-20T17:51:44.715608+0000","last_peered":"2024-04-20T17:51:44.715608+0000","last_clean":"2024-04-20T17:51:44.715608+0000","last_became_active":"2024-04-20T17:48:54.817749+0000","last_became_peered":"2024-04-20T17:48:54.817749+0000","last_unstale":"2024-04-20T17:51:44.715608+0000","last_undegraded":"2024-04-20T17:51:44.715608+0000","last_fullsized":"2024-04-20T17:51:44.715608+0000","mapping_epoch":894,"log_start":"908'4800","ondisk_log_start":"908'4800","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'14734","last_scrub_stamp":"2024-04-20T17:51:42.519071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:42.519071+0000","objects_scrubbed":6056,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:09.147760+0000","scrub_duration":1274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383778816,"num_objects":5856,"num_object_clones":0,"num_object_copies":11712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14888,"num_write_kb":663808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"952'14842","reported_seq":15070,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-04-20T17:51:45.580615+0000","last_change":"2024-04-20T17:50:23.032137+0000","last_active":"2024-04-20T17:51:45.580615+0000","last_peered":"2024-04-20T17:51:45.580615+0000","last_clean":"2024-04-20T17:51:45.580615+0000","last_became_active":"2024-04-20T17:48:55.061199+0000","last_became_peered":"2024-04-20T17:48:55.061199+0000","last_unstale":"2024-04-20T17:51:45.580615+0000","last_undegraded":"2024-04-20T17:51:45.580615+0000","last_fullsized":"2024-04-20T17:51:45.580615+0000","mapping_epoch":894,"log_start":"908'4800","ondisk_log_start":"908'4800","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9404","last_scrub_stamp":"2024-04-20T17:50:23.031924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:23.031924+0000","objects_scrubbed":9366,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384696320,"num_objects":5870,"num_object_clones":0,"num_object_copies":11740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14842,"num_write_kb":662784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"951'14749","reported_seq":30614,"reported_epoch":951,"state":"active+clean","last_fresh":"2024-04-20T17:51:44.006098+0000","last_change":"2024-04-20T17:51:42.767947+0000","last_active":"2024-04-20T17:51:44.006098+0000","last_peered":"2024-04-20T17:51:44.006098+0000","last_clean":"2024-04-20T17:51:44.006098+0000","last_became_active":"2024-04-20T17:51:40.523417+0000","last_became_peered":"2024-04-20T17:51:40.523417+0000","last_unstale":"2024-04-20T17:51:44.006098+0000","last_undegraded":"2024-04-20T17:51:44.006098+0000","last_fullsized":"2024-04-20T17:51:44.006098+0000","mapping_epoch":950,"log_start":"908'4700","ondisk_log_start":"908'4700","created":894,"last_epoch_clean":951,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'14655","last_scrub_stamp":"2024-04-20T17:51:42.767651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:42.767651+0000","objects_scrubbed":5939,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:50.590021+0000","scrub_duration":1039,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381878272,"num_objects":5827,"num_object_clones":0,"num_object_copies":11654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14749,"num_write_kb":658432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7778,"num_bytes_recovered":509607936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.e","version":"952'14979","reported_seq":15202,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-04-20T17:51:44.819857+0000","last_change":"2024-04-20T17:51:03.098201+0000","last_active":"2024-04-20T17:51:44.819857+0000","last_peered":"2024-04-20T17:51:44.819857+0000","last_clean":"2024-04-20T17:51:44.819857+0000","last_became_active":"2024-04-20T17:48:54.815159+0000","last_became_peered":"2024-04-20T17:48:54.815159+0000","last_unstale":"2024-04-20T17:51:44.819857+0000","last_undegraded":"2024-04-20T17:51:44.819857+0000","last_fullsized":"2024-04-20T17:51:44.819857+0000","mapping_epoch":894,"log_start":"908'4900","ondisk_log_start":"908'4900","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"942'12102","last_scrub_stamp":"2024-04-20T17:51:03.098072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:03.098072+0000","objects_scrubbed":8857,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:30.248766+0000","scrub_duration":2003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386203648,"num_objects":5893,"num_object_clones":0,"num_object_copies":11786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14979,"num_write_kb":667904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"15'137","reported_seq":955,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-04-20T17:51:44.478329+0000","last_change":"2024-04-20T17:50:57.911454+0000","last_active":"2024-04-20T17:51:44.478329+0000","last_peered":"2024-04-20T17:51:44.478329+0000","last_clean":"2024-04-20T17:51:44.478329+0000","last_became_active":"2024-04-20T17:45:32.670941+0000","last_became_peered":"2024-04-20T17:45:32.670941+0000","last_unstale":"2024-04-20T17:51:44.478329+0000","last_undegraded":"2024-04-20T17:51:44.478329+0000","last_fullsized":"2024-04-20T17:51:44.478329+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:57.911336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:57.911336+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:02.057237+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"952'14936","reported_seq":15235,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-04-20T17:51:45.488429+0000","last_change":"2024-04-20T17:51:43.065593+0000","last_active":"2024-04-20T17:51:45.488429+0000","last_peered":"2024-04-20T17:51:45.488429+0000","last_clean":"2024-04-20T17:51:45.488429+0000","last_became_active":"2024-04-20T17:48:55.061161+0000","last_became_peered":"2024-04-20T17:48:55.061161+0000","last_unstale":"2024-04-20T17:51:45.488429+0000","last_undegraded":"2024-04-20T17:51:45.488429+0000","last_fullsized":"2024-04-20T17:51:45.488429+0000","mapping_epoch":894,"log_start":"908'4900","ondisk_log_start":"908'4900","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'14822","last_scrub_stamp":"2024-04-20T17:51:43.065456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:43.065456+0000","objects_scrubbed":6081,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:57.084267+0000","scrub_duration":1030,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389283840,"num_objects":5940,"num_object_clones":0,"num_object_copies":11880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14936,"num_write_kb":668032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"15'137","reported_seq":951,"reported_epoch":951,"state":"active+clean","last_fresh":"2024-04-20T17:51:40.601238+0000","last_change":"2024-04-20T17:50:25.724423+0000","last_active":"2024-04-20T17:51:40.601238+0000","last_peered":"2024-04-20T17:51:40.601238+0000","last_clean":"2024-04-20T17:51:40.601238+0000","last_became_active":"2024-04-20T17:45:29.817528+0000","last_became_peered":"2024-04-20T17:45:29.817528+0000","last_unstale":"2024-04-20T17:51:40.601238+0000","last_undegraded":"2024-04-20T17:51:40.601238+0000","last_fullsized":"2024-04-20T17:51:40.601238+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:25.724296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:25.724296+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"951'15021","reported_seq":15253,"reported_epoch":951,"state":"active+clean","last_fresh":"2024-04-20T17:51:43.643444+0000","last_change":"2024-04-20T17:50:29.927851+0000","last_active":"2024-04-20T17:51:43.643444+0000","last_peered":"2024-04-20T17:51:43.643444+0000","last_clean":"2024-04-20T17:51:43.643444+0000","last_became_active":"2024-04-20T17:48:54.817445+0000","last_became_peered":"2024-04-20T17:48:54.817445+0000","last_unstale":"2024-04-20T17:51:43.643444+0000","last_undegraded":"2024-04-20T17:51:43.643444+0000","last_fullsized":"2024-04-20T17:51:43.643444+0000","mapping_epoch":894,"log_start":"908'5000","ondisk_log_start":"908'5000","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'10062","last_scrub_stamp":"2024-04-20T17:50:29.927762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:29.927762+0000","objects_scrubbed":9978,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:58.752509+0000","scrub_duration":2071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393019392,"num_objects":5997,"num_object_clones":0,"num_object_copies":11994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15021,"num_write_kb":672576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"15'137","reported_seq":988,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-04-20T17:51:44.510869+0000","last_change":"2024-04-20T17:51:44.510869+0000","last_active":"2024-04-20T17:51:44.510869+0000","last_peered":"2024-04-20T17:51:44.510869+0000","last_clean":"2024-04-20T17:51:44.510869+0000","last_became_active":"2024-04-20T17:45:29.827577+0000","last_became_peered":"2024-04-20T17:45:29.827577+0000","last_unstale":"2024-04-20T17:51:44.510869+0000","last_undegraded":"2024-04-20T17:51:44.510869+0000","last_fullsized":"2024-04-20T17:51:44.510869+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:44.510742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:44.510742+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:46.934138+0000","scrub_duration":266,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"952'14868","reported_seq":15159,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-04-20T17:51:44.657577+0000","last_change":"2024-04-20T17:51:00.541301+0000","last_active":"2024-04-20T17:51:44.657577+0000","last_peered":"2024-04-20T17:51:44.657577+0000","last_clean":"2024-04-20T17:51:44.657577+0000","last_became_active":"2024-04-20T17:48:55.061208+0000","last_became_peered":"2024-04-20T17:48:55.061208+0000","last_unstale":"2024-04-20T17:51:44.657577+0000","last_undegraded":"2024-04-20T17:51:44.657577+0000","last_fullsized":"2024-04-20T17:51:44.657577+0000","mapping_epoch":894,"log_start":"908'4800","ondisk_log_start":"908'4800","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"941'11888","last_scrub_stamp":"2024-04-20T17:51:00.541192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:00.541192+0000","objects_scrubbed":8973,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:11.066367+0000","scrub_duration":1502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388956184,"num_objects":5936,"num_object_clones":0,"num_object_copies":11872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5936,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14868,"num_write_kb":665665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"15'137","reported_seq":947,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-04-20T17:51:44.477268+0000","last_change":"2024-04-20T17:50:18.122040+0000","last_active":"2024-04-20T17:51:44.477268+0000","last_peered":"2024-04-20T17:51:44.477268+0000","last_clean":"2024-04-20T17:51:44.477268+0000","last_became_active":"2024-04-20T17:45:32.666523+0000","last_became_peered":"2024-04-20T17:45:32.666523+0000","last_unstale":"2024-04-20T17:51:44.477268+0000","last_undegraded":"2024-04-20T17:51:44.477268+0000","last_fullsized":"2024-04-20T17:51:44.477268+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:18.121912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:18.121912+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:45.028684+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.a","version":"952'15123","reported_seq":15356,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-04-20T17:51:45.488376+0000","last_change":"2024-04-20T17:50:22.180642+0000","last_active":"2024-04-20T17:51:45.488376+0000","last_peered":"2024-04-20T17:51:45.488376+0000","last_clean":"2024-04-20T17:51:45.488376+0000","last_became_active":"2024-04-20T17:48:55.061180+0000","last_became_peered":"2024-04-20T17:48:55.061180+0000","last_unstale":"2024-04-20T17:51:45.488376+0000","last_undegraded":"2024-04-20T17:51:45.488376+0000","last_fullsized":"2024-04-20T17:51:45.488376+0000","mapping_epoch":894,"log_start":"908'5100","ondisk_log_start":"908'5100","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9574","last_scrub_stamp":"2024-04-20T17:50:22.180501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:22.180501+0000","objects_scrubbed":9507,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:51.128747+0000","scrub_duration":1824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383713280,"num_objects":5855,"num_object_clones":0,"num_object_copies":11710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15123,"num_write_kb":671296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"15'137","reported_seq":958,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-04-20T17:51:44.476398+0000","last_change":"2024-04-20T17:51:43.176176+0000","last_active":"2024-04-20T17:51:44.476398+0000","last_peered":"2024-04-20T17:51:44.476398+0000","last_clean":"2024-04-20T17:51:44.476398+0000","last_became_active":"2024-04-20T17:46:40.831961+0000","last_became_peered":"2024-04-20T17:46:40.831961+0000","last_unstale":"2024-04-20T17:51:44.476398+0000","last_undegraded":"2024-04-20T17:51:44.476398+0000","last_fullsized":"2024-04-20T17:51:44.476398+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:43.176019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:43.176019+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:57.843728+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"951'14740","reported_seq":14962,"reported_epoch":951,"state":"active+clean","last_fresh":"2024-04-20T17:51:44.002600+0000","last_change":"2024-04-20T17:50:19.657299+0000","last_active":"2024-04-20T17:51:44.002600+0000","last_peered":"2024-04-20T17:51:44.002600+0000","last_clean":"2024-04-20T17:51:44.002600+0000","last_became_active":"2024-04-20T17:48:54.816644+0000","last_became_peered":"2024-04-20T17:48:54.816644+0000","last_unstale":"2024-04-20T17:51:44.002600+0000","last_undegraded":"2024-04-20T17:51:44.002600+0000","last_fullsized":"2024-04-20T17:51:44.002600+0000","mapping_epoch":894,"log_start":"908'4700","ondisk_log_start":"908'4700","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"927'9204","last_scrub_stamp":"2024-04-20T17:50:19.657194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:19.657194+0000","objects_scrubbed":9124,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:44.670679+0000","scrub_duration":2257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394919936,"num_objects":6026,"num_object_clones":0,"num_object_copies":12052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14740,"num_write_kb":664512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"15'137","reported_seq":746,"reported_epoch":951,"state":"active+clean","last_fresh":"2024-04-20T17:51:40.511058+0000","last_change":"2024-04-20T17:51:00.057261+0000","last_active":"2024-04-20T17:51:40.511058+0000","last_peered":"2024-04-20T17:51:40.511058+0000","last_clean":"2024-04-20T17:51:40.511058+0000","last_became_active":"2024-04-20T17:49:46.824559+0000","last_became_peered":"2024-04-20T17:49:46.824559+0000","last_unstale":"2024-04-20T17:51:40.511058+0000","last_undegraded":"2024-04-20T17:51:40.511058+0000","last_fullsized":"2024-04-20T17:51:40.511058+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:00.057156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:00.057156+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:08.337431+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"952'15204","reported_seq":15437,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-04-20T17:51:44.666864+0000","last_change":"2024-04-20T17:50:28.283117+0000","last_active":"2024-04-20T17:51:44.666864+0000","last_peered":"2024-04-20T17:51:44.666864+0000","last_clean":"2024-04-20T17:51:44.666864+0000","last_became_active":"2024-04-20T17:48:54.813199+0000","last_became_peered":"2024-04-20T17:48:54.813199+0000","last_unstale":"2024-04-20T17:51:44.666864+0000","last_undegraded":"2024-04-20T17:51:44.666864+0000","last_fullsized":"2024-04-20T17:51:44.666864+0000","mapping_epoch":894,"log_start":"908'5200","ondisk_log_start":"908'5200","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'10025","last_scrub_stamp":"2024-04-20T17:50:28.282957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:28.282957+0000","objects_scrubbed":9963,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:48.575002+0000","scrub_duration":2339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398327808,"num_objects":6078,"num_object_clones":0,"num_object_copies":12156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15204,"num_write_kb":681024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"15'137","reported_seq":955,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-04-20T17:51:44.477330+0000","last_change":"2024-04-20T17:50:58.932166+0000","last_active":"2024-04-20T17:51:44.477330+0000","last_peered":"2024-04-20T17:51:44.477330+0000","last_clean":"2024-04-20T17:51:44.477330+0000","last_became_active":"2024-04-20T17:45:32.782706+0000","last_became_peered":"2024-04-20T17:45:32.782706+0000","last_unstale":"2024-04-20T17:51:44.477330+0000","last_undegraded":"2024-04-20T17:51:44.477330+0000","last_fullsized":"2024-04-20T17:51:44.477330+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:58.932046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:58.932046+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:23.591238+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.2","version":"951'14854","reported_seq":15146,"reported_epoch":951,"state":"active+clean","last_fresh":"2024-04-20T17:51:43.606382+0000","last_change":"2024-04-20T17:51:43.530794+0000","last_active":"2024-04-20T17:51:43.606382+0000","last_peered":"2024-04-20T17:51:43.606382+0000","last_clean":"2024-04-20T17:51:43.606382+0000","last_became_active":"2024-04-20T17:48:55.060965+0000","last_became_peered":"2024-04-20T17:48:55.060965+0000","last_unstale":"2024-04-20T17:51:43.606382+0000","last_undegraded":"2024-04-20T17:51:43.606382+0000","last_fullsized":"2024-04-20T17:51:43.606382+0000","mapping_epoch":894,"log_start":"908'4800","ondisk_log_start":"908'4800","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'14849","last_scrub_stamp":"2024-04-20T17:51:43.530564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:43.530564+0000","objects_scrubbed":5973,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:05.542393+0000","scrub_duration":1032,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388366336,"num_objects":5926,"num_object_clones":0,"num_object_copies":11852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14854,"num_write_kb":664960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"15'137","reported_seq":939,"reported_epoch":951,"state":"active+clean","last_fresh":"2024-04-20T17:51:41.848534+0000","last_change":"2024-04-20T17:51:41.848577+0000","last_active":"2024-04-20T17:51:41.848534+0000","last_peered":"2024-04-20T17:51:41.848534+0000","last_clean":"2024-04-20T17:51:41.848534+0000","last_became_active":"2024-04-20T17:45:32.669451+0000","last_became_peered":"2024-04-20T17:45:32.669451+0000","last_unstale":"2024-04-20T17:51:41.848534+0000","last_undegraded":"2024-04-20T17:51:41.848534+0000","last_fullsized":"2024-04-20T17:51:41.848534+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:41.848440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:41.848440+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:55.264457+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"951'14668","reported_seq":14905,"reported_epoch":951,"state":"active+clean+scrubbing","last_fresh":"2024-04-20T17:51:41.739669+0000","last_change":"2024-04-20T17:51:41.712775+0000","last_active":"2024-04-20T17:51:41.739669+0000","last_peered":"2024-04-20T17:51:41.739669+0000","last_clean":"2024-04-20T17:51:41.739669+0000","last_became_active":"2024-04-20T17:48:55.061200+0000","last_became_peered":"2024-04-20T17:48:55.061200+0000","last_unstale":"2024-04-20T17:51:41.739669+0000","last_undegraded":"2024-04-20T17:51:41.739669+0000","last_fullsized":"2024-04-20T17:51:41.739669+0000","mapping_epoch":894,"log_start":"908'4600","ondisk_log_start":"908'4600","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'9940","last_scrub_stamp":"2024-04-20T17:50:30.239131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:30.239131+0000","objects_scrubbed":199,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"scrubbing for 0s","scrub_duration":1672,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398589952,"num_objects":6082,"num_object_clones":0,"num_object_copies":12164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14668,"num_write_kb":664000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"15'137","reported_seq":959,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-04-20T17:51:45.488837+0000","last_change":"2024-04-20T17:51:40.826215+0000","last_active":"2024-04-20T17:51:45.488837+0000","last_peered":"2024-04-20T17:51:45.488837+0000","last_clean":"2024-04-20T17:51:45.488837+0000","last_became_active":"2024-04-20T17:45:32.644584+0000","last_became_peered":"2024-04-20T17:45:32.644584+0000","last_unstale":"2024-04-20T17:51:45.488837+0000","last_undegraded":"2024-04-20T17:51:45.488837+0000","last_fullsized":"2024-04-20T17:51:45.488837+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:40.826081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:40.826081+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:45.293420+0000","scrub_duration":185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"951'14983","reported_seq":15214,"reported_epoch":951,"state":"active+clean","last_fresh":"2024-04-20T17:51:43.625194+0000","last_change":"2024-04-20T17:50:30.712810+0000","last_active":"2024-04-20T17:51:43.625194+0000","last_peered":"2024-04-20T17:51:43.625194+0000","last_clean":"2024-04-20T17:51:43.625194+0000","last_became_active":"2024-04-20T17:48:54.813062+0000","last_became_peered":"2024-04-20T17:48:54.813062+0000","last_unstale":"2024-04-20T17:51:43.625194+0000","last_undegraded":"2024-04-20T17:51:43.625194+0000","last_fullsized":"2024-04-20T17:51:43.625194+0000","mapping_epoch":894,"log_start":"908'4900","ondisk_log_start":"908'4900","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'10026","last_scrub_stamp":"2024-04-20T17:50:30.712648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:30.712648+0000","objects_scrubbed":9967,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:58.995403+0000","scrub_duration":1871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389087232,"num_objects":5937,"num_object_clones":0,"num_object_copies":11874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14983,"num_write_kb":669440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"15'137","reported_seq":896,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-04-20T17:51:44.476393+0000","last_change":"2024-04-20T17:50:33.137929+0000","last_active":"2024-04-20T17:51:44.476393+0000","last_peered":"2024-04-20T17:51:44.476393+0000","last_clean":"2024-04-20T17:51:44.476393+0000","last_became_active":"2024-04-20T17:46:40.881713+0000","last_became_peered":"2024-04-20T17:46:40.881713+0000","last_unstale":"2024-04-20T17:51:44.476393+0000","last_undegraded":"2024-04-20T17:51:44.476393+0000","last_fullsized":"2024-04-20T17:51:44.476393+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:33.137831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:33.137831+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.8","version":"951'14933","reported_seq":15159,"reported_epoch":951,"state":"active+clean","last_fresh":"2024-04-20T17:51:42.169354+0000","last_change":"2024-04-20T17:50:19.940419+0000","last_active":"2024-04-20T17:51:42.169354+0000","last_peered":"2024-04-20T17:51:42.169354+0000","last_clean":"2024-04-20T17:51:42.169354+0000","last_became_active":"2024-04-20T17:48:54.813368+0000","last_became_peered":"2024-04-20T17:48:54.813368+0000","last_unstale":"2024-04-20T17:51:42.169354+0000","last_undegraded":"2024-04-20T17:51:42.169354+0000","last_fullsized":"2024-04-20T17:51:42.169354+0000","mapping_epoch":894,"log_start":"908'4900","ondisk_log_start":"908'4900","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"927'9391","last_scrub_stamp":"2024-04-20T17:50:19.940199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:19.940199+0000","objects_scrubbed":9334,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1977,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402063360,"num_objects":6135,"num_object_clones":0,"num_object_copies":12270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14933,"num_write_kb":674176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":894,"reported_epoch":952,"state":"active+clean","last_fresh":"2024-04-20T17:51:44.476386+0000","last_change":"2024-04-20T17:50:21.111183+0000","last_active":"2024-04-20T17:51:44.476386+0000","last_peered":"2024-04-20T17:51:44.476386+0000","last_clean":"2024-04-20T17:51:44.476386+0000","last_became_active":"2024-04-20T17:46:40.851683+0000","last_became_peered":"2024-04-20T17:46:40.851683+0000","last_unstale":"2024-04-20T17:51:44.476386+0000","last_undegraded":"2024-04-20T17:51:44.476386+0000","last_fullsized":"2024-04-20T17:51:44.476386+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:21.111059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:21.111059+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:45.968902+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6228213784,"num_objects":95036,"num_object_clones":0,"num_object_copies":190072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95036,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":238552,"num_write_kb":10674753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7778,"num_bytes_recovered":509607936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12439265280,"data_stored":12439257136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160752,"ondisk_log_size":160752,"up":32,"acting":32,"num_store_stats":11},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},"log_size":1507,"ondisk_log_size":1507,"up":22,"acting":22,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991786,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2075072,"kb_used_data":1145652,"kb_used_omap":1,"kb_used_meta":929406,"kb_avail":91670080,"statfs":{"total":95995035648,"available":93870161920,"internally_reserved":0,"allocated":1173147648,"data_stored":1172757909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":951712196},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253425,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2730240,"kb_used_data":1896628,"kb_used_omap":1,"kb_used_meta":833598,"kb_avail":91014912,"statfs":{"total":95995035648,"available":93199269888,"internally_reserved":0,"allocated":1942147072,"data_stored":1941757333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":853604800},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613173,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1540788,"kb_used_data":783528,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92204364,"statfs":{"total":95995035648,"available":94417268736,"internally_reserved":0,"allocated":802332672,"data_stored":801947500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":775420352},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809404,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1187248,"kb_used_data":763176,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":92557904,"statfs":{"total":95995035648,"available":94779293696,"internally_reserved":0,"allocated":781492224,"data_stored":781107052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":799,"seq":3431678869594,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1271796,"kb_used_data":1141292,"kb_used_omap":11,"kb_used_meta":130420,"kb_avail":92473356,"statfs":{"total":95995035648,"available":94692716544,"internally_reserved":0,"allocated":1168683008,"data_stored":1168293772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11990,"internal_metadata":133550378},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":535,"seq":2297807503633,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":440176,"kb_used_data":390120,"kb_used_omap":170,"kb_used_meta":49877,"kb_avail":93304976,"statfs":{"total":95995035648,"available":95544295424,"internally_reserved":0,"allocated":399482880,"data_stored":399097708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":174628,"internal_metadata":51074524},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":659,"seq":2830383448256,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2663924,"kb_used_data":2285868,"kb_used_omap":4,"kb_used_meta":378043,"kb_avail":91081228,"statfs":{"total":95995035648,"available":93267177472,"internally_reserved":0,"allocated":2340728832,"data_stored":2340339588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4410,"internal_metadata":387116742},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738979,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":744144,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":93001008,"statfs":{"total":95995035648,"available":95233032192,"internally_reserved":0,"allocated":786432,"data_stored":390705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394268,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":581012,"kb_used_data":161928,"kb_used_omap":202,"kb_used_meta":418869,"kb_avail":93164140,"statfs":{"total":95995035648,"available":95400079360,"internally_reserved":0,"allocated":165814272,"data_stored":165414993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":428922327},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653317,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2091440,"kb_used_data":1521768,"kb_used_omap":4,"kb_used_meta":569659,"kb_avail":91653712,"statfs":{"total":95995035648,"available":93853401088,"internally_reserved":0,"allocated":1558290432,"data_stored":1557905260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":583331527},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986766,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1378240,"kb_used_data":755636,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":92366912,"statfs":{"total":95995035648,"available":94583717888,"internally_reserved":0,"allocated":773771264,"data_stored":773381525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790536,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2556552,"kb_used_data":1672956,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":91188600,"statfs":{"total":95995035648,"available":93377126400,"internally_reserved":0,"allocated":1713106944,"data_stored":1712713653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":904788421},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2328104960,"data_stored":2328100888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1554317312,"data_stored":1554317312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":772997120,"data_stored":772997120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1537998848,"data_stored":1537998848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":398589952,"data_stored":398589952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1159270400,"data_stored":1159266328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":776732672,"data_stored":776732672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":797835264,"data_stored":797835264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1941372928,"data_stored":1941372928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1172045824,"data_stored":1172045824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:51:50.144 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-20T17:51:50.265 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:51:50.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:50.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:51:50.375 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:51:50.407 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:50.432 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:51:50.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:50.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:51:50.536 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:51:50.596 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:50.635 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:51:50.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:50.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:51:50.680 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:51:50.723 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:51:50.739 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:51:50.779 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:50.806 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:51:50.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:50.814 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:51:50.915 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:51:50.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:50 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:51:50.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:50 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:51:50.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:50 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:51:50.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:50 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:51:50.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:50 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:51:50.963 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:50 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:51:50.963 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:50 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:51:50.963 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:50 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:51:50.963 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:50 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:51:50.963 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:50 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:51:50.965 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:50.995 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:51:50.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:50.999 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:51:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:50 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:51:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:50 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:51:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:50 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:51:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:50 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:51:51.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:50 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:51:51.101 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:51:51.149 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:51.179 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:51:51.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:51.184 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:51:51.285 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:51:51.330 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:51.367 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:51:51.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:51.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:51:51.474 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:51:51.511 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:51.536 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:51:51.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:51.544 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:51:51.645 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:51:51.684 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:51 smithi106 ceph-mon[44423]: 9.9 scrub ok 2024-04-20T17:51:51.684 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:51 smithi106 ceph-mon[44423]: pgmap v2516: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 896 op/s; 4861/183790 objects misplaced (2.645%); 6.9 MiB/s, 109 objects/s recovering 2024-04-20T17:51:51.684 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:51 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3951195774' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:51:51.684 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:51 smithi106 ceph-mon[44423]: from='client.28319 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:51:51.688 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:51 smithi005 ceph-mon[39562]: 9.9 scrub ok 2024-04-20T17:51:51.688 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:51 smithi005 ceph-mon[39562]: pgmap v2516: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 896 op/s; 4861/183790 objects misplaced (2.645%); 6.9 MiB/s, 109 objects/s recovering 2024-04-20T17:51:51.688 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:51 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3951195774' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:51:51.688 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:51 smithi005 ceph-mon[39562]: from='client.28319 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:51:51.694 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:51:51.699 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:51.737 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:51:51.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:51.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:51:51.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:51 smithi171 ceph-mon[44189]: 9.9 scrub ok 2024-04-20T17:51:51.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:51 smithi171 ceph-mon[44189]: pgmap v2516: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 896 op/s; 4861/183790 objects misplaced (2.645%); 6.9 MiB/s, 109 objects/s recovering 2024-04-20T17:51:51.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:51 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3951195774' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:51:51.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:51 smithi171 ceph-mon[44189]: from='client.28319 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:51:51.848 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:51:51.898 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:52.770 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:51:52.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:52.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:51:52.892 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:51:52.897 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:52 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:51:52.897 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:52 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:51:52.897 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:52 smithi005 ceph-mon[39562]: from='client.28319 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:51:52.897 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:52 smithi005 ceph-mon[39562]: osdmap e955: 12 total, 12 up, 12 in 2024-04-20T17:51:52.941 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:53.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:52 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:51:53.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:52 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:51:53.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:52 smithi106 ceph-mon[44423]: from='client.28319 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:51:53.014 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:52 smithi106 ceph-mon[44423]: osdmap e955: 12 total, 12 up, 12 in 2024-04-20T17:51:53.072 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:51:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:52 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:51:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:52 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:51:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:52 smithi171 ceph-mon[44189]: from='client.28319 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:51:53.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:52 smithi171 ceph-mon[44189]: osdmap e955: 12 total, 12 up, 12 in 2024-04-20T17:51:53.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:53 smithi005 ceph-mon[39562]: pgmap v2518: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 798 op/s; 4861/181158 objects misplaced (2.683%); 5.9 MiB/s, 94 objects/s recovering 2024-04-20T17:51:54.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:53 smithi106 ceph-mon[44423]: pgmap v2518: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 798 op/s; 4861/181158 objects misplaced (2.683%); 5.9 MiB/s, 94 objects/s recovering 2024-04-20T17:51:54.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:53 smithi171 ceph-mon[44189]: pgmap v2518: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 798 op/s; 4861/181158 objects misplaced (2.683%); 5.9 MiB/s, 94 objects/s recovering 2024-04-20T17:51:54.418 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:51:54.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:54.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:51:54.526 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:51:54.558 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:54.587 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:51:54.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:54.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:51:54.695 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:51:54.732 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:54.757 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:51:54.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:54.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:51:54.864 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:51:54.900 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:54.925 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:51:54.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:54.933 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:51:55.034 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:51:55.070 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:55.104 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:51:55.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:55.114 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:51:55.161 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:51:55.198 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:51:55.215 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:51:55.228 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:51:55.263 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:51:55.264 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:55.292 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:51:55.295 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:51:55.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:55.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:51:55.325 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:51:55.368 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:51:55.400 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:51:55.403 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:51:55.443 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:51:55.448 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:55.471 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:51:55.472 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:51:55.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:55.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:51:55.514 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:51:55.550 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:51:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:51:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:51:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:51:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:51:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:51:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:51:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:51:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:51:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:51:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:51:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:51:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:51:55.577 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:51:55.618 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:55.642 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:51:55.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:55.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:51:55.746 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:51:55.797 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:55.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:55 smithi005 ceph-mon[39562]: pgmap v2519: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 5.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 756 op/s; 4861/177630 objects misplaced (2.737%); 5.2 MiB/s, 83 objects/s recovering 2024-04-20T17:51:55.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:55 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4108633157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:51:55.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:55.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:55.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:55 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:55 smithi106 ceph-mon[44423]: pgmap v2519: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 5.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 756 op/s; 4861/177630 objects misplaced (2.737%); 5.2 MiB/s, 83 objects/s recovering 2024-04-20T17:51:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:55 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4108633157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:51:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:56.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:55 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:56.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:55 smithi171 ceph-mon[44189]: pgmap v2519: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 5.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 756 op/s; 4861/177630 objects misplaced (2.737%); 5.2 MiB/s, 83 objects/s recovering 2024-04-20T17:51:56.086 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:55 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4108633157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:51:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:56.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:55 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:56.462 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:51:56.468 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:51:56.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:56.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:51:56.576 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:51:56.610 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:56.639 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:51:56.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:56.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:51:56.754 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:51:56.787 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:56.817 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:51:56.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:56.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:51:56.924 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:51:56.968 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:57.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:56 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:57.069 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:51:57.072 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:51:57.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:57.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:57.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:57.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:57.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:57.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:57.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:57.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:57.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:56 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:57.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:57.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:51:57.095 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:57.095 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:57.095 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:57.096 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:57.096 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:57.096 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:57.096 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:57.096 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:57.096 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:56 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:51:57.131 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:51:57.153 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:51:57.153 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:51:57.153 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.153 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27772.0:251202 9.1 9:8e1e1d73:::benchmark_data_smithi171_188319_object84406:head [delete] snapc 0=[] ondisk+write+known_if_redirected e955)", 2024-04-20T17:51:57.153 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:51:57.135110+0000", 2024-04-20T17:51:57.153 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.017512435999999999, 2024-04-20T17:51:57.153 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.017540172, 2024-04-20T17:51:57.153 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:51:57.153 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:51:57.153 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:51:57.154 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:51:57.154 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27772", 2024-04-20T17:51:57.154 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:51:57.154 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 251202 2024-04-20T17:51:57.154 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.154 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:51:57.154 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.154 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:51:57.154 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.135110+0000", 2024-04-20T17:51:57.154 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:51:57.154 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.154 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.154 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:51:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.135110+0000", 2024-04-20T17:51:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:51:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:51:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.135112+0000", 2024-04-20T17:51:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4540000000000001e-06 2024-04-20T17:51:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:51:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.135122+0000", 2024-04-20T17:51:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4439999999999997e-06 2024-04-20T17:51:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.155 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:51:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.135136+0000", 2024-04-20T17:51:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4722999999999999e-05 2024-04-20T17:51:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:51:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.135146+0000", 2024-04-20T17:51:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6209999999999999e-06 2024-04-20T17:51:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:51:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.135320+0000", 2024-04-20T17:51:57.156 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000174122 2024-04-20T17:51:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:51:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.135450+0000", 2024-04-20T17:51:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.000129531 2024-04-20T17:51:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 9", 2024-04-20T17:51:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.135574+0000", 2024-04-20T17:51:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012425399999999999 2024-04-20T17:51:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.157 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:51:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.144928+0000", 2024-04-20T17:51:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0093541860000000004 2024-04-20T17:51:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:51:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:51:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:51:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27772.0:251208 9.1 9:814f9d86:::benchmark_data_smithi171_188319_object84412:head [delete] snapc 0=[] ondisk+write+known_if_redirected e955)", 2024-04-20T17:51:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:51:57.136678+0000", 2024-04-20T17:51:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.015944327000000001, 2024-04-20T17:51:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.016067425, 2024-04-20T17:51:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:51:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:51:57.158 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:51:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:51:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27772", 2024-04-20T17:51:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:51:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 251208 2024-04-20T17:51:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:51:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:51:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.136678+0000", 2024-04-20T17:51:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:51:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:51:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.136678+0000", 2024-04-20T17:51:57.159 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:51:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:51:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.136680+0000", 2024-04-20T17:51:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8840000000000001e-06 2024-04-20T17:51:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:51:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.136685+0000", 2024-04-20T17:51:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4299999999999997e-06 2024-04-20T17:51:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:51:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.136695+0000", 2024-04-20T17:51:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0149000000000001e-05 2024-04-20T17:51:57.160 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:51:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.136703+0000", 2024-04-20T17:51:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6620000000000005e-06 2024-04-20T17:51:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:51:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.140521+0000", 2024-04-20T17:51:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0038175240000000001 2024-04-20T17:51:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:51:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.140675+0000", 2024-04-20T17:51:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015410000000000001 2024-04-20T17:51:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.161 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 9", 2024-04-20T17:51:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.140823+0000", 2024-04-20T17:51:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014827500000000001 2024-04-20T17:51:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:51:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:51:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:51:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27772.0:251213 9.1 9:8fa57180:::benchmark_data_smithi171_188319_object84417:head [delete] snapc 0=[] ondisk+write+known_if_redirected e955)", 2024-04-20T17:51:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:51:57.143419+0000", 2024-04-20T17:51:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.0092033659999999993, 2024-04-20T17:51:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0094125719999999993, 2024-04-20T17:51:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:51:57.162 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:51:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:51:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:51:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27772", 2024-04-20T17:51:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:51:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 251213 2024-04-20T17:51:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:51:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:51:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.143419+0000", 2024-04-20T17:51:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:51:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:51:57.163 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.143419+0000", 2024-04-20T17:51:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:51:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:51:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.143421+0000", 2024-04-20T17:51:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6300000000000001e-06 2024-04-20T17:51:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:51:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.143426+0000", 2024-04-20T17:51:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5860000000000004e-06 2024-04-20T17:51:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:51:57.164 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.143437+0000", 2024-04-20T17:51:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0577999999999999e-05 2024-04-20T17:51:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:51:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.143443+0000", 2024-04-20T17:51:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5300000000000002e-06 2024-04-20T17:51:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:51:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.144967+0000", 2024-04-20T17:51:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0015235610000000001 2024-04-20T17:51:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:51:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.145119+0000", 2024-04-20T17:51:57.165 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00015181299999999999 2024-04-20T17:51:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 9", 2024-04-20T17:51:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.145265+0000", 2024-04-20T17:51:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014653499999999999 2024-04-20T17:51:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:51:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:51:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:51:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27772.0:251217 9.1 9:80c2dd41:::benchmark_data_smithi171_188319_object84421:head [delete] snapc 0=[] ondisk+write+known_if_redirected e955)", 2024-04-20T17:51:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:51:57.146766+0000", 2024-04-20T17:51:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.0058564539999999997, 2024-04-20T17:51:57.166 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0061315179999999999, 2024-04-20T17:51:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:51:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:51:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:51:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:51:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27772", 2024-04-20T17:51:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:51:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 251217 2024-04-20T17:51:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:51:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:51:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.146766+0000", 2024-04-20T17:51:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:51:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:51:57.167 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.146766+0000", 2024-04-20T17:51:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:51:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:51:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.146768+0000", 2024-04-20T17:51:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8190000000000001e-06 2024-04-20T17:51:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:51:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.146774+0000", 2024-04-20T17:51:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0469999999999999e-06 2024-04-20T17:51:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:51:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.146784+0000", 2024-04-20T17:51:57.168 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0404000000000001e-05 2024-04-20T17:51:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:51:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.146792+0000", 2024-04-20T17:51:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2609999999999993e-06 2024-04-20T17:51:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:51:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.149391+0000", 2024-04-20T17:51:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025985119999999999 2024-04-20T17:51:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:51:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.149531+0000", 2024-04-20T17:51:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014003799999999999 2024-04-20T17:51:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.169 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 9", 2024-04-20T17:51:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.149669+0000", 2024-04-20T17:51:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00013773600000000001 2024-04-20T17:51:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:51:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:51:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:51:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27772.0:251219 9.2 9:47b8be13:::benchmark_data_smithi171_188319_object84423:head [delete] snapc 0=[] ondisk+write+known_if_redirected e955)", 2024-04-20T17:51:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:51:57.147093+0000", 2024-04-20T17:51:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.0055293180000000001, 2024-04-20T17:51:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0058469610000000003, 2024-04-20T17:51:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:51:57.170 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:51:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "waiting for sub ops", 2024-04-20T17:51:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:51:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27772", 2024-04-20T17:51:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:51:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 251219 2024-04-20T17:51:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:51:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:51:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.147093+0000", 2024-04-20T17:51:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:51:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:51:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.147093+0000", 2024-04-20T17:51:57.171 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:51:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:51:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.147094+0000", 2024-04-20T17:51:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.141e-06 2024-04-20T17:51:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:51:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.147098+0000", 2024-04-20T17:51:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.946e-06 2024-04-20T17:51:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:51:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.147106+0000", 2024-04-20T17:51:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7160000000000003e-06 2024-04-20T17:51:57.172 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:51:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.147112+0000", 2024-04-20T17:51:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.1750000000000002e-06 2024-04-20T17:51:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:51:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.147172+0000", 2024-04-20T17:51:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9969000000000001e-05 2024-04-20T17:51:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:51:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.147322+0000", 2024-04-20T17:51:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00014976900000000001 2024-04-20T17:51:57.173 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.174 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.174 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "waiting for subops from 1", 2024-04-20T17:51:57.174 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.147446+0000", 2024-04-20T17:51:57.174 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00012448599999999999 2024-04-20T17:51:57.174 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.174 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.174 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "op_commit", 2024-04-20T17:51:57.174 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.152877+0000", 2024-04-20T17:51:57.174 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0054312509999999998 2024-04-20T17:51:57.174 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:51:57.174 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:51:57.174 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:51:57.174 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.174 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27772.0:251220 9.1 9.1d066b91 (undecoded) ondisk+write+known_if_redirected e955)", 2024-04-20T17:51:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:51:57.150432+0000", 2024-04-20T17:51:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.0021901260000000001, 2024-04-20T17:51:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025515889999999999, 2024-04-20T17:51:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:51:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:51:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:51:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:51:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27772", 2024-04-20T17:51:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:51:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 251220 2024-04-20T17:51:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:51:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:51:57.175 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.150432+0000", 2024-04-20T17:51:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:51:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:51:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.150432+0000", 2024-04-20T17:51:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:51:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:51:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.150434+0000", 2024-04-20T17:51:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.148e-06 2024-04-20T17:51:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:51:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.150442+0000", 2024-04-20T17:51:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0520000000000002e-06 2024-04-20T17:51:57.176 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:51:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.150459+0000", 2024-04-20T17:51:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6197000000000001e-05 2024-04-20T17:51:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:51:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.150469+0000", 2024-04-20T17:51:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0361e-05 2024-04-20T17:51:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:51:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:51:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:51:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.177 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop_reply(client.27772.0:251208 9.1 e955/894)", 2024-04-20T17:51:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:51:57.150467+0000", 2024-04-20T17:51:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.0021556919999999999, 2024-04-20T17:51:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025473430000000001, 2024-04-20T17:51:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:51:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:51:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:51:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:51:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:51:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.150467+0000", 2024-04-20T17:51:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:51:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:51:57.178 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.150467+0000", 2024-04-20T17:51:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:51:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:51:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.150468+0000", 2024-04-20T17:51:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.779e-06 2024-04-20T17:51:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:51:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.150474+0000", 2024-04-20T17:51:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9750000000000004e-06 2024-04-20T17:51:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.179 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:51:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.150485+0000", 2024-04-20T17:51:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0757999999999999e-05 2024-04-20T17:51:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:51:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.150493+0000", 2024-04-20T17:51:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8900000000000007e-06 2024-04-20T17:51:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:51:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:51:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:51:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_op(client.27772.0:251222 9.1 9.da983261 (undecoded) ondisk+write+known_if_redirected e955)", 2024-04-20T17:51:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:51:57.152138+0000", 2024-04-20T17:51:57.180 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.000484141, 2024-04-20T17:51:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00090436300000000004, 2024-04-20T17:51:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:51:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:51:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "queued for pg", 2024-04-20T17:51:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: "client_info": { 2024-04-20T17:51:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: "client": "client.27772", 2024-04-20T17:51:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:51:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: "tid": 251222 2024-04-20T17:51:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:51:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:51:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.152138+0000", 2024-04-20T17:51:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:51:57.181 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:51:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.152138+0000", 2024-04-20T17:51:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:51:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:51:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.152140+0000", 2024-04-20T17:51:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6440000000000001e-06 2024-04-20T17:51:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:51:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.152145+0000", 2024-04-20T17:51:57.182 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5459999999999999e-06 2024-04-20T17:51:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:51:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.152158+0000", 2024-04-20T17:51:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2429000000000001e-05 2024-04-20T17:51:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:51:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:51:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:51:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:51:57.152165+0000", 2024-04-20T17:51:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.2919999999999998e-06 2024-04-20T17:51:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:51:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:51:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:51:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:51:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:51:57.183 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 8 2024-04-20T17:51:57.184 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:51:57.187 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:51:57.221 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:57.249 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:51:57.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:57.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:51:57.354 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:51:57.394 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:57.423 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:51:57.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:57.429 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:51:57.530 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:51:57.562 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:57.599 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:51:57.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:57.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:51:57.703 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:51:57.754 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:57.903 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:57 smithi005 ceph-mon[39562]: pgmap v2520: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 952 op/s; 3506/171026 objects misplaced (2.050%); 11 MiB/s, 178 objects/s recovering 2024-04-20T17:51:58.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:57 smithi106 ceph-mon[44423]: pgmap v2520: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 952 op/s; 3506/171026 objects misplaced (2.050%); 11 MiB/s, 178 objects/s recovering 2024-04-20T17:51:58.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:57 smithi171 ceph-mon[44189]: pgmap v2520: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 952 op/s; 3506/171026 objects misplaced (2.050%); 11 MiB/s, 178 objects/s recovering 2024-04-20T17:51:58.214 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:51:58.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:58.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:51:58.240 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-20T17:51:58.318 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:51:58.368 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:58.403 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:51:58.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:58.409 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:51:58.510 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:51:58.514 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:51:58.549 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:58.577 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:51:58.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:58.584 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:51:58.685 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:51:58.715 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:58.739 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:51:58.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:58.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:51:58.847 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:51:58.882 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:51:58.912 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:51:58.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:51:58.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:51:59.023 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:51:59.060 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:00.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:51:59 smithi171 ceph-mon[44189]: pgmap v2521: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 3506/167246 objects misplaced (2.096%); 11 MiB/s, 178 objects/s recovering 2024-04-20T17:52:00.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:51:59 smithi005 ceph-mon[39562]: pgmap v2521: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 3506/167246 objects misplaced (2.096%); 11 MiB/s, 178 objects/s recovering 2024-04-20T17:52:00.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:51:59 smithi106 ceph-mon[44423]: pgmap v2521: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 3506/167246 objects misplaced (2.096%); 11 MiB/s, 178 objects/s recovering 2024-04-20T17:52:00.856 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:52:00.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:00.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:52:00.964 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:52:01.021 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:01.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:00 smithi171 ceph-mon[44189]: 9.c scrub starts 2024-04-20T17:52:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:00 smithi005 ceph-mon[39562]: 9.c scrub starts 2024-04-20T17:52:01.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:00 smithi106 ceph-mon[44423]: 9.c scrub starts 2024-04-20T17:52:01.701 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:52:01.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:01.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:52:01.823 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:52:01.879 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:01 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:52:01.879 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:01 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:52:01.879 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:01 smithi005 ceph-mon[39562]: pgmap v2522: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 2271/160648 objects misplaced (1.414%); 13 MiB/s, 214 objects/s recovering 2024-04-20T17:52:01.879 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:01 smithi005 ceph-mon[39562]: osdmap e956: 12 total, 12 up, 12 in 2024-04-20T17:52:01.885 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:01.889 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:52:01.903 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:52:02.102 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:52:02.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:02.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:52:02.211 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:52:02.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:01 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:52:02.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:01 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:52:02.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:01 smithi106 ceph-mon[44423]: pgmap v2522: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 2271/160648 objects misplaced (1.414%); 13 MiB/s, 214 objects/s recovering 2024-04-20T17:52:02.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:01 smithi106 ceph-mon[44423]: osdmap e956: 12 total, 12 up, 12 in 2024-04-20T17:52:02.268 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:02.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:01 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:52:02.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:01 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:52:02.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:01 smithi171 ceph-mon[44189]: pgmap v2522: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 2271/160648 objects misplaced (1.414%); 13 MiB/s, 214 objects/s recovering 2024-04-20T17:52:02.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:01 smithi171 ceph-mon[44189]: osdmap e956: 12 total, 12 up, 12 in 2024-04-20T17:52:02.354 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:52:02.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:02.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:52:02.461 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:52:02.509 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:02.543 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:52:02.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:02.549 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:52:02.650 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:52:02.688 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:02.773 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:52:02.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:02.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:52:02.814 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":2523,"stamp":"2024-04-20T17:52:00.809485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5264507448,"num_objects":80324,"num_object_clones":0,"num_object_copies":160648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2271,"num_objects_unfound":0,"num_objects_dirty":80324,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":253266,"num_write_kb":10674753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10609,"num_bytes_recovered":695140352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162173,"ondisk_log_size":162173,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":17594096,"kb_used_data":10760904,"kb_used_omap":404,"kb_used_meta":6832299,"kb_avail":1107347728,"statfs":{"total":1151940427776,"available":1133924073472,"internally_reserved":0,"allocated":11019165696,"data_stored":11015046588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":414014,"internal_metadata":6996274882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,6,6,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-719716352,"num_objects":-10982,"num_object_clones":0,"num_object_copies":-21964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2590,"num_objects_unfound":0,"num_objects_dirty":-10982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10982,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2146,"num_bytes_recovered":140640256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.122551"},"pg_stats":[{"pgid":"9.7","version":"955'15749","reported_seq":15984,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-04-20T17:52:00.213051+0000","last_change":"2024-04-20T17:50:27.211659+0000","last_active":"2024-04-20T17:52:00.213051+0000","last_peered":"2024-04-20T17:52:00.213051+0000","last_clean":"2024-04-20T17:52:00.213051+0000","last_became_active":"2024-04-20T17:48:54.814508+0000","last_became_peered":"2024-04-20T17:48:54.814508+0000","last_unstale":"2024-04-20T17:52:00.213051+0000","last_undegraded":"2024-04-20T17:52:00.213051+0000","last_fullsized":"2024-04-20T17:52:00.213051+0000","mapping_epoch":894,"log_start":"911'5700","ondisk_log_start":"911'5700","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'9665","last_scrub_stamp":"2024-04-20T17:50:27.211421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:27.211421+0000","objects_scrubbed":9588,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2310,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321978368,"num_objects":4913,"num_object_clones":0,"num_object_copies":9826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15749,"num_write_kb":661184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"955'15930","reported_seq":21828,"reported_epoch":955,"state":"active+remapped+backfilling","last_fresh":"2024-04-20T17:52:00.679313+0000","last_change":"2024-04-20T17:51:46.673164+0000","last_active":"2024-04-20T17:52:00.679313+0000","last_peered":"2024-04-20T17:52:00.679313+0000","last_clean":"2024-04-20T17:51:44.458787+0000","last_became_active":"2024-04-20T17:51:46.633967+0000","last_became_peered":"2024-04-20T17:51:46.633967+0000","last_unstale":"2024-04-20T17:52:00.679313+0000","last_undegraded":"2024-04-20T17:52:00.679313+0000","last_fullsized":"2024-04-20T17:52:00.679313+0000","mapping_epoch":953,"log_start":"912'5900","ondisk_log_start":"912'5900","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"929'9608","last_scrub_stamp":"2024-04-20T17:50:25.096273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:25.096273+0000","objects_scrubbed":9524,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317849600,"num_objects":4850,"num_object_clones":0,"num_object_copies":9700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2271,"num_objects_unfound":0,"num_objects_dirty":4850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15930,"num_write_kb":664960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2831,"num_bytes_recovered":185532416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[5,10],"avail_no_missing":["5","10"],"object_location_counts":[{"shards":"5,10","objects":4850}],"blocked_by":[],"up_primary":2,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"955'15562","reported_seq":15867,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-04-20T17:51:56.165923+0000","last_change":"2024-04-20T17:51:42.519313+0000","last_active":"2024-04-20T17:51:56.165923+0000","last_peered":"2024-04-20T17:51:56.165923+0000","last_clean":"2024-04-20T17:51:56.165923+0000","last_became_active":"2024-04-20T17:48:54.817749+0000","last_became_peered":"2024-04-20T17:48:54.817749+0000","last_unstale":"2024-04-20T17:51:56.165923+0000","last_undegraded":"2024-04-20T17:51:56.165923+0000","last_fullsized":"2024-04-20T17:51:56.165923+0000","mapping_epoch":894,"log_start":"908'5500","ondisk_log_start":"908'5500","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'14734","last_scrub_stamp":"2024-04-20T17:51:42.519071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:42.519071+0000","objects_scrubbed":6056,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:09.147760+0000","scrub_duration":1274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339607552,"num_objects":5182,"num_object_clones":0,"num_object_copies":10364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15562,"num_write_kb":663808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"955'15499","reported_seq":15797,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-04-20T17:51:55.909458+0000","last_change":"2024-04-20T17:51:48.836352+0000","last_active":"2024-04-20T17:51:55.909458+0000","last_peered":"2024-04-20T17:51:55.909458+0000","last_clean":"2024-04-20T17:51:55.909458+0000","last_became_active":"2024-04-20T17:48:55.061199+0000","last_became_peered":"2024-04-20T17:48:55.061199+0000","last_unstale":"2024-04-20T17:51:55.909458+0000","last_undegraded":"2024-04-20T17:51:55.909458+0000","last_fullsized":"2024-04-20T17:51:55.909458+0000","mapping_epoch":894,"log_start":"908'5400","ondisk_log_start":"908'5400","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"954'15006","last_scrub_stamp":"2024-04-20T17:51:48.836218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:48.836218+0000","objects_scrubbed":5762,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:51.196876+0000","scrub_duration":1412,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341639168,"num_objects":5213,"num_object_clones":0,"num_object_copies":10426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15499,"num_write_kb":662784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"955'15648","reported_seq":31521,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-04-20T17:51:58.997084+0000","last_change":"2024-04-20T17:51:42.767947+0000","last_active":"2024-04-20T17:51:58.997084+0000","last_peered":"2024-04-20T17:51:58.997084+0000","last_clean":"2024-04-20T17:51:58.997084+0000","last_became_active":"2024-04-20T17:51:40.523417+0000","last_became_peered":"2024-04-20T17:51:40.523417+0000","last_unstale":"2024-04-20T17:51:58.997084+0000","last_undegraded":"2024-04-20T17:51:58.997084+0000","last_fullsized":"2024-04-20T17:51:58.997084+0000","mapping_epoch":950,"log_start":"909'5600","ondisk_log_start":"909'5600","created":894,"last_epoch_clean":951,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'14655","last_scrub_stamp":"2024-04-20T17:51:42.767651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:42.767651+0000","objects_scrubbed":5939,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:50.590021+0000","scrub_duration":1039,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322961408,"num_objects":4928,"num_object_clones":0,"num_object_copies":9856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15648,"num_write_kb":658432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7778,"num_bytes_recovered":509607936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.e","version":"955'15948","reported_seq":16175,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-04-20T17:51:59.848059+0000","last_change":"2024-04-20T17:51:03.098201+0000","last_active":"2024-04-20T17:51:59.848059+0000","last_peered":"2024-04-20T17:51:59.848059+0000","last_clean":"2024-04-20T17:51:59.848059+0000","last_became_active":"2024-04-20T17:48:54.815159+0000","last_became_peered":"2024-04-20T17:48:54.815159+0000","last_unstale":"2024-04-20T17:51:59.848059+0000","last_undegraded":"2024-04-20T17:51:59.848059+0000","last_fullsized":"2024-04-20T17:51:59.848059+0000","mapping_epoch":894,"log_start":"911'5900","ondisk_log_start":"911'5900","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"942'12102","last_scrub_stamp":"2024-04-20T17:51:03.098072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:03.098072+0000","objects_scrubbed":8857,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:30.248766+0000","scrub_duration":2003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322699264,"num_objects":4924,"num_object_clones":0,"num_object_copies":9848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15948,"num_write_kb":667904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"15'137","reported_seq":961,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-04-20T17:51:51.689152+0000","last_change":"2024-04-20T17:50:57.911454+0000","last_active":"2024-04-20T17:51:51.689152+0000","last_peered":"2024-04-20T17:51:51.689152+0000","last_clean":"2024-04-20T17:51:51.689152+0000","last_became_active":"2024-04-20T17:45:32.670941+0000","last_became_peered":"2024-04-20T17:45:32.670941+0000","last_unstale":"2024-04-20T17:51:51.689152+0000","last_undegraded":"2024-04-20T17:51:51.689152+0000","last_fullsized":"2024-04-20T17:51:51.689152+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:57.911336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:57.911336+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:02.057237+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"955'15920","reported_seq":16223,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-04-20T17:52:00.681327+0000","last_change":"2024-04-20T17:51:43.065593+0000","last_active":"2024-04-20T17:52:00.681327+0000","last_peered":"2024-04-20T17:52:00.681327+0000","last_clean":"2024-04-20T17:52:00.681327+0000","last_became_active":"2024-04-20T17:48:55.061161+0000","last_became_peered":"2024-04-20T17:48:55.061161+0000","last_unstale":"2024-04-20T17:52:00.681327+0000","last_undegraded":"2024-04-20T17:52:00.681327+0000","last_fullsized":"2024-04-20T17:52:00.681327+0000","mapping_epoch":894,"log_start":"912'5900","ondisk_log_start":"912'5900","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'14822","last_scrub_stamp":"2024-04-20T17:51:43.065456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:43.065456+0000","objects_scrubbed":6081,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:57.084267+0000","scrub_duration":1030,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324796416,"num_objects":4956,"num_object_clones":0,"num_object_copies":9912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15920,"num_write_kb":668032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"15'137","reported_seq":967,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-04-20T17:51:51.722486+0000","last_change":"2024-04-20T17:51:42.855635+0000","last_active":"2024-04-20T17:51:51.722486+0000","last_peered":"2024-04-20T17:51:51.722486+0000","last_clean":"2024-04-20T17:51:51.722486+0000","last_became_active":"2024-04-20T17:45:29.817528+0000","last_became_peered":"2024-04-20T17:45:29.817528+0000","last_unstale":"2024-04-20T17:51:51.722486+0000","last_undegraded":"2024-04-20T17:51:51.722486+0000","last_fullsized":"2024-04-20T17:51:51.722486+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:42.855496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:42.855496+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:49.448590+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"955'15954","reported_seq":16194,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-04-20T17:51:58.623658+0000","last_change":"2024-04-20T17:50:29.927851+0000","last_active":"2024-04-20T17:51:58.623658+0000","last_peered":"2024-04-20T17:51:58.623658+0000","last_clean":"2024-04-20T17:51:58.623658+0000","last_became_active":"2024-04-20T17:48:54.817445+0000","last_became_peered":"2024-04-20T17:48:54.817445+0000","last_unstale":"2024-04-20T17:51:58.623658+0000","last_undegraded":"2024-04-20T17:51:58.623658+0000","last_fullsized":"2024-04-20T17:51:58.623658+0000","mapping_epoch":894,"log_start":"911'5900","ondisk_log_start":"911'5900","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'10062","last_scrub_stamp":"2024-04-20T17:50:29.927762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:29.927762+0000","objects_scrubbed":9978,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:58.752509+0000","scrub_duration":2071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331874304,"num_objects":5064,"num_object_clones":0,"num_object_copies":10128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15954,"num_write_kb":672576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"15'137","reported_seq":994,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-04-20T17:51:51.694222+0000","last_change":"2024-04-20T17:51:44.510869+0000","last_active":"2024-04-20T17:51:51.694222+0000","last_peered":"2024-04-20T17:51:51.694222+0000","last_clean":"2024-04-20T17:51:51.694222+0000","last_became_active":"2024-04-20T17:45:29.827577+0000","last_became_peered":"2024-04-20T17:45:29.827577+0000","last_unstale":"2024-04-20T17:51:51.694222+0000","last_undegraded":"2024-04-20T17:51:51.694222+0000","last_fullsized":"2024-04-20T17:51:51.694222+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:44.510742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:44.510742+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:46.934138+0000","scrub_duration":266,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"955'15819","reported_seq":16115,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-04-20T17:51:59.813415+0000","last_change":"2024-04-20T17:51:00.541301+0000","last_active":"2024-04-20T17:51:59.813415+0000","last_peered":"2024-04-20T17:51:59.813415+0000","last_clean":"2024-04-20T17:51:59.813415+0000","last_became_active":"2024-04-20T17:48:55.061208+0000","last_became_peered":"2024-04-20T17:48:55.061208+0000","last_unstale":"2024-04-20T17:51:59.813415+0000","last_undegraded":"2024-04-20T17:51:59.813415+0000","last_fullsized":"2024-04-20T17:51:59.813415+0000","mapping_epoch":894,"log_start":"911'5800","ondisk_log_start":"911'5800","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"941'11888","last_scrub_stamp":"2024-04-20T17:51:00.541192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:00.541192+0000","objects_scrubbed":8973,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:11.066367+0000","scrub_duration":1502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326631448,"num_objects":4985,"num_object_clones":0,"num_object_copies":9970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4985,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15819,"num_write_kb":665665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"15'137","reported_seq":962,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-04-20T17:51:51.689072+0000","last_change":"2024-04-20T17:51:46.197902+0000","last_active":"2024-04-20T17:51:51.689072+0000","last_peered":"2024-04-20T17:51:51.689072+0000","last_clean":"2024-04-20T17:51:51.689072+0000","last_became_active":"2024-04-20T17:45:32.666523+0000","last_became_peered":"2024-04-20T17:45:32.666523+0000","last_unstale":"2024-04-20T17:51:51.689072+0000","last_undegraded":"2024-04-20T17:51:51.689072+0000","last_fullsized":"2024-04-20T17:51:51.689072+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:46.197797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:46.197797+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:56.371129+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.a","version":"955'16111","reported_seq":16348,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-04-20T17:52:00.666472+0000","last_change":"2024-04-20T17:50:22.180642+0000","last_active":"2024-04-20T17:52:00.666472+0000","last_peered":"2024-04-20T17:52:00.666472+0000","last_clean":"2024-04-20T17:52:00.666472+0000","last_became_active":"2024-04-20T17:48:55.061180+0000","last_became_peered":"2024-04-20T17:48:55.061180+0000","last_unstale":"2024-04-20T17:52:00.666472+0000","last_undegraded":"2024-04-20T17:52:00.666472+0000","last_fullsized":"2024-04-20T17:52:00.666472+0000","mapping_epoch":894,"log_start":"912'6100","ondisk_log_start":"912'6100","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9574","last_scrub_stamp":"2024-04-20T17:50:22.180501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:22.180501+0000","objects_scrubbed":9507,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318963712,"num_objects":4867,"num_object_clones":0,"num_object_copies":9734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16111,"num_write_kb":671296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"15'137","reported_seq":963,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-04-20T17:51:51.688304+0000","last_change":"2024-04-20T17:51:43.176176+0000","last_active":"2024-04-20T17:51:51.688304+0000","last_peered":"2024-04-20T17:51:51.688304+0000","last_clean":"2024-04-20T17:51:51.688304+0000","last_became_active":"2024-04-20T17:46:40.831961+0000","last_became_peered":"2024-04-20T17:46:40.831961+0000","last_unstale":"2024-04-20T17:51:51.688304+0000","last_undegraded":"2024-04-20T17:51:51.688304+0000","last_fullsized":"2024-04-20T17:51:51.688304+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:43.176019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:43.176019+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:57.843728+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"955'15699","reported_seq":15996,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-04-20T17:51:58.947075+0000","last_change":"2024-04-20T17:51:46.152560+0000","last_active":"2024-04-20T17:51:58.947075+0000","last_peered":"2024-04-20T17:51:58.947075+0000","last_clean":"2024-04-20T17:51:58.947075+0000","last_became_active":"2024-04-20T17:48:54.816644+0000","last_became_peered":"2024-04-20T17:48:54.816644+0000","last_unstale":"2024-04-20T17:51:58.947075+0000","last_undegraded":"2024-04-20T17:51:58.947075+0000","last_fullsized":"2024-04-20T17:51:58.947075+0000","mapping_epoch":894,"log_start":"910'5600","ondisk_log_start":"910'5600","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"952'14792","last_scrub_stamp":"2024-04-20T17:51:46.152388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:46.152388+0000","objects_scrubbed":5974,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:04.134118+0000","scrub_duration":519,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332070912,"num_objects":5067,"num_object_clones":0,"num_object_copies":10134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15699,"num_write_kb":664512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"15'137","reported_seq":755,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-04-20T17:51:51.692402+0000","last_change":"2024-04-20T17:51:00.057261+0000","last_active":"2024-04-20T17:51:51.692402+0000","last_peered":"2024-04-20T17:51:51.692402+0000","last_clean":"2024-04-20T17:51:51.692402+0000","last_became_active":"2024-04-20T17:49:46.824559+0000","last_became_peered":"2024-04-20T17:49:46.824559+0000","last_unstale":"2024-04-20T17:51:51.692402+0000","last_undegraded":"2024-04-20T17:51:51.692402+0000","last_fullsized":"2024-04-20T17:51:51.692402+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:00.057156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:00.057156+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:08.337431+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"955'16221","reported_seq":16527,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-04-20T17:52:00.201638+0000","last_change":"2024-04-20T17:51:50.104395+0000","last_active":"2024-04-20T17:52:00.201638+0000","last_peered":"2024-04-20T17:52:00.201638+0000","last_clean":"2024-04-20T17:52:00.201638+0000","last_became_active":"2024-04-20T17:48:54.813199+0000","last_became_peered":"2024-04-20T17:48:54.813199+0000","last_unstale":"2024-04-20T17:52:00.201638+0000","last_undegraded":"2024-04-20T17:52:00.201638+0000","last_fullsized":"2024-04-20T17:52:00.201638+0000","mapping_epoch":894,"log_start":"914'6200","ondisk_log_start":"914'6200","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"954'15465","last_scrub_stamp":"2024-04-20T17:51:50.104218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:50.104218+0000","objects_scrubbed":5848,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:05.037497+0000","scrub_duration":1044,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331677696,"num_objects":5061,"num_object_clones":0,"num_object_copies":10122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16221,"num_write_kb":681024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"15'137","reported_seq":962,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-04-20T17:51:51.689055+0000","last_change":"2024-04-20T17:50:58.932166+0000","last_active":"2024-04-20T17:51:51.689055+0000","last_peered":"2024-04-20T17:51:51.689055+0000","last_clean":"2024-04-20T17:51:51.689055+0000","last_became_active":"2024-04-20T17:45:32.782706+0000","last_became_peered":"2024-04-20T17:45:32.782706+0000","last_unstale":"2024-04-20T17:51:51.689055+0000","last_undegraded":"2024-04-20T17:51:51.689055+0000","last_fullsized":"2024-04-20T17:51:51.689055+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:58.932046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:58.932046+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:23.591238+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.2","version":"955'15814","reported_seq":16114,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-04-20T17:51:58.640888+0000","last_change":"2024-04-20T17:51:43.530794+0000","last_active":"2024-04-20T17:51:58.640888+0000","last_peered":"2024-04-20T17:51:58.640888+0000","last_clean":"2024-04-20T17:51:58.640888+0000","last_became_active":"2024-04-20T17:48:55.060965+0000","last_became_peered":"2024-04-20T17:48:55.060965+0000","last_unstale":"2024-04-20T17:51:58.640888+0000","last_undegraded":"2024-04-20T17:51:58.640888+0000","last_fullsized":"2024-04-20T17:51:58.640888+0000","mapping_epoch":894,"log_start":"911'5800","ondisk_log_start":"911'5800","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'14849","last_scrub_stamp":"2024-04-20T17:51:43.530564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:43.530564+0000","objects_scrubbed":5973,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:05.542393+0000","scrub_duration":1032,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325451776,"num_objects":4966,"num_object_clones":0,"num_object_copies":9932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15814,"num_write_kb":664960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"15'137","reported_seq":947,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-04-20T17:51:51.684855+0000","last_change":"2024-04-20T17:51:41.848577+0000","last_active":"2024-04-20T17:51:51.684855+0000","last_peered":"2024-04-20T17:51:51.684855+0000","last_clean":"2024-04-20T17:51:51.684855+0000","last_became_active":"2024-04-20T17:45:32.669451+0000","last_became_peered":"2024-04-20T17:45:32.669451+0000","last_unstale":"2024-04-20T17:51:51.684855+0000","last_undegraded":"2024-04-20T17:51:51.684855+0000","last_fullsized":"2024-04-20T17:51:51.684855+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:41.848440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:41.848440+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:55.264457+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"955'15602","reported_seq":15910,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-04-20T17:51:56.735203+0000","last_change":"2024-04-20T17:51:42.714681+0000","last_active":"2024-04-20T17:51:56.735203+0000","last_peered":"2024-04-20T17:51:56.735203+0000","last_clean":"2024-04-20T17:51:56.735203+0000","last_became_active":"2024-04-20T17:48:55.061200+0000","last_became_peered":"2024-04-20T17:48:55.061200+0000","last_unstale":"2024-04-20T17:51:56.735203+0000","last_undegraded":"2024-04-20T17:51:56.735203+0000","last_fullsized":"2024-04-20T17:51:56.735203+0000","mapping_epoch":894,"log_start":"910'5600","ondisk_log_start":"910'5600","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'14728","last_scrub_stamp":"2024-04-20T17:51:42.714540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:42.714540+0000","objects_scrubbed":6058,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:00.487021+0000","scrub_duration":1002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337379328,"num_objects":5148,"num_object_clones":0,"num_object_copies":10296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15602,"num_write_kb":664000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"15'137","reported_seq":963,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-04-20T17:51:51.701683+0000","last_change":"2024-04-20T17:51:40.826215+0000","last_active":"2024-04-20T17:51:51.701683+0000","last_peered":"2024-04-20T17:51:51.701683+0000","last_clean":"2024-04-20T17:51:51.701683+0000","last_became_active":"2024-04-20T17:45:32.644584+0000","last_became_peered":"2024-04-20T17:45:32.644584+0000","last_unstale":"2024-04-20T17:51:51.701683+0000","last_undegraded":"2024-04-20T17:51:51.701683+0000","last_fullsized":"2024-04-20T17:51:51.701683+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:40.826081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:40.826081+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:45.293420+0000","scrub_duration":185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"955'15910","reported_seq":16149,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-04-20T17:51:58.623869+0000","last_change":"2024-04-20T17:50:30.712810+0000","last_active":"2024-04-20T17:51:58.623869+0000","last_peered":"2024-04-20T17:51:58.623869+0000","last_clean":"2024-04-20T17:51:58.623869+0000","last_became_active":"2024-04-20T17:48:54.813062+0000","last_became_peered":"2024-04-20T17:48:54.813062+0000","last_unstale":"2024-04-20T17:51:58.623869+0000","last_undegraded":"2024-04-20T17:51:58.623869+0000","last_fullsized":"2024-04-20T17:51:58.623869+0000","mapping_epoch":894,"log_start":"911'5900","ondisk_log_start":"911'5900","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'10026","last_scrub_stamp":"2024-04-20T17:50:30.712648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:30.712648+0000","objects_scrubbed":9967,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:51:58.995403+0000","scrub_duration":1871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328335360,"num_objects":5010,"num_object_clones":0,"num_object_copies":10020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15910,"num_write_kb":669440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"15'137","reported_seq":909,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-04-20T17:51:51.688277+0000","last_change":"2024-04-20T17:51:45.119466+0000","last_active":"2024-04-20T17:51:51.688277+0000","last_peered":"2024-04-20T17:51:51.688277+0000","last_clean":"2024-04-20T17:51:51.688277+0000","last_became_active":"2024-04-20T17:46:40.881713+0000","last_became_peered":"2024-04-20T17:46:40.881713+0000","last_unstale":"2024-04-20T17:51:51.688277+0000","last_undegraded":"2024-04-20T17:51:51.688277+0000","last_fullsized":"2024-04-20T17:51:51.688277+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:45.119332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:45.119332+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:51.157610+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.8","version":"955'15880","reported_seq":16183,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-04-20T17:51:57.209293+0000","last_change":"2024-04-20T17:51:43.840908+0000","last_active":"2024-04-20T17:51:57.209293+0000","last_peered":"2024-04-20T17:51:57.209293+0000","last_clean":"2024-04-20T17:51:57.209293+0000","last_became_active":"2024-04-20T17:48:54.813368+0000","last_became_peered":"2024-04-20T17:48:54.813368+0000","last_unstale":"2024-04-20T17:51:57.209293+0000","last_undegraded":"2024-04-20T17:51:57.209293+0000","last_fullsized":"2024-04-20T17:51:57.209293+0000","mapping_epoch":894,"log_start":"910'5800","ondisk_log_start":"910'5800","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'15051","last_scrub_stamp":"2024-04-20T17:51:43.840798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:43.840798+0000","objects_scrubbed":6061,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:52.336868+0000","scrub_duration":1160,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340000768,"num_objects":5188,"num_object_clones":0,"num_object_copies":10376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15880,"num_write_kb":674176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":907,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-04-20T17:51:51.688263+0000","last_change":"2024-04-20T17:51:48.160638+0000","last_active":"2024-04-20T17:51:51.688263+0000","last_peered":"2024-04-20T17:51:51.688263+0000","last_clean":"2024-04-20T17:51:51.688263+0000","last_became_active":"2024-04-20T17:46:40.851683+0000","last_became_peered":"2024-04-20T17:46:40.851683+0000","last_unstale":"2024-04-20T17:51:51.688263+0000","last_undegraded":"2024-04-20T17:51:51.688263+0000","last_fullsized":"2024-04-20T17:51:51.688263+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:48.160546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:48.160546+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:56.139128+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":5263917080,"num_objects":80322,"num_object_clones":0,"num_object_copies":160644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2271,"num_objects_unfound":0,"num_objects_dirty":80322,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":253266,"num_write_kb":10674753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10609,"num_bytes_recovered":695140352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10656620544,"data_stored":10656612400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160666,"ondisk_log_size":160666,"up":32,"acting":32,"num_store_stats":11},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},"log_size":1507,"ondisk_log_size":1507,"up":22,"acting":22,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991788,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1965084,"kb_used_data":1027348,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":91780068,"statfs":{"total":95995035648,"available":93982789632,"internally_reserved":0,"allocated":1052004352,"data_stored":1051662363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":960231876},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253428,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2422620,"kb_used_data":1584724,"kb_used_omap":1,"kb_used_meta":837758,"kb_avail":91322532,"statfs":{"total":95995035648,"available":93514272768,"internally_reserved":0,"allocated":1622757376,"data_stored":1622415387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":857864640},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613176,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1412316,"kb_used_data":663380,"kb_used_omap":1,"kb_used_meta":748926,"kb_avail":92332836,"statfs":{"total":95995035648,"available":94548824064,"internally_reserved":0,"allocated":679301120,"data_stored":678959131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":766900672},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809407,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1113052,"kb_used_data":644564,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":92632100,"statfs":{"total":95995035648,"available":94855270400,"internally_reserved":0,"allocated":660033536,"data_stored":659691547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":799,"seq":3431678869597,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1104352,"kb_used_data":961368,"kb_used_omap":11,"kb_used_meta":142900,"kb_avail":92640800,"statfs":{"total":95995035648,"available":94864179200,"internally_reserved":0,"allocated":984440832,"data_stored":984094779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11990,"internal_metadata":146329898},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":535,"seq":2297807503636,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":390044,"kb_used_data":334420,"kb_used_omap":170,"kb_used_meta":55445,"kb_avail":93355108,"statfs":{"total":95995035648,"available":95595630592,"internally_reserved":0,"allocated":342446080,"data_stored":342104091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":174628,"internal_metadata":56776156},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":659,"seq":2830383448259,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2309472,"kb_used_data":1927256,"kb_used_omap":4,"kb_used_meta":382203,"kb_avail":91435680,"statfs":{"total":95995035648,"available":93630136320,"internally_reserved":0,"allocated":1973510144,"data_stored":1973164083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4410,"internal_metadata":391376582},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738981,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":744164,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":93000988,"statfs":{"total":95995035648,"available":95233011712,"internally_reserved":0,"allocated":675840,"data_stored":333851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394270,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":670436,"kb_used_data":251356,"kb_used_omap":202,"kb_used_meta":418869,"kb_avail":93074716,"statfs":{"total":95995035648,"available":95308509184,"internally_reserved":0,"allocated":257388544,"data_stored":257043003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":428922327},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653320,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1870684,"kb_used_data":1288532,"kb_used_omap":4,"kb_used_meta":582139,"kb_avail":91874468,"statfs":{"total":95995035648,"available":94079455232,"internally_reserved":0,"allocated":1319456768,"data_stored":1319114779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":596111047},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986768,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1307292,"kb_used_data":676372,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":92437860,"statfs":{"total":95995035648,"available":94656368640,"internally_reserved":0,"allocated":692604928,"data_stored":692262939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790539,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2284580,"kb_used_data":1400924,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":91460572,"statfs":{"total":95995035648,"available":93655625728,"internally_reserved":0,"allocated":1434546176,"data_stored":1434200635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":904788421},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1962545152,"data_stored":1962541080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":122028032,"data_stored":122028032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1311506432,"data_stored":1311506432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":686686208,"data_stored":686686208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1278672896,"data_stored":1278672896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":337379328,"data_stored":337379328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":972886016,"data_stored":972881944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":654966784,"data_stored":654966784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":674824192,"data_stored":674824192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1619984384,"data_stored":1619984384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1035141120,"data_stored":1035141120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:52:02.816 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:52:02.878 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:52:02.912 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:02.935 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:52:02.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:02.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:52:03.042 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:52:03.071 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:03.092 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:52:03.093 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:52:03.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:03.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:52:03.099 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:52:03.199 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:52:03.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:02 smithi005 ceph-mon[39562]: 9.c scrub ok 2024-04-20T17:52:03.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:02 smithi005 ceph-mon[39562]: from='client.28328 v1:172.21.15.5:0/3448271945' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:52:03.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:02 smithi005 ceph-mon[39562]: 9.1 deep-scrub starts 2024-04-20T17:52:03.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:02 smithi106 ceph-mon[44423]: 9.c scrub ok 2024-04-20T17:52:03.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:02 smithi106 ceph-mon[44423]: from='client.28328 v1:172.21.15.5:0/3448271945' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:52:03.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:02 smithi106 ceph-mon[44423]: 9.1 deep-scrub starts 2024-04-20T17:52:03.268 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:03.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:02 smithi171 ceph-mon[44189]: 9.c scrub ok 2024-04-20T17:52:03.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:02 smithi171 ceph-mon[44189]: from='client.28328 v1:172.21.15.5:0/3448271945' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:52:03.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:02 smithi171 ceph-mon[44189]: 9.1 deep-scrub starts 2024-04-20T17:52:03.632 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:52:03.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:03.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:52:03.745 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:52:03.798 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:03.824 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:52:03.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:03.833 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:52:03.900 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:52:03.933 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:52:03.965 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:03.988 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:52:03.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:03.992 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:52:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:03 smithi171 ceph-mon[44189]: pgmap v2524: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 4.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 2271/157782 objects misplaced (1.439%); 13 MiB/s, 214 objects/s recovering 2024-04-20T17:52:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:03 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2264711472' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:52:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:03 smithi171 ceph-mon[44189]: from='client.27961 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:52:04.092 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:52:04.124 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:03 smithi005 ceph-mon[39562]: pgmap v2524: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 4.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 2271/157782 objects misplaced (1.439%); 13 MiB/s, 214 objects/s recovering 2024-04-20T17:52:04.124 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:03 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2264711472' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:52:04.124 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:03 smithi005 ceph-mon[39562]: from='client.27961 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:52:04.127 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:04.237 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:52:04.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:04.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:52:04.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:03 smithi106 ceph-mon[44423]: pgmap v2524: 27 pgs: 1 active+remapped+backfilling, 26 active+clean; 4.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 2271/157782 objects misplaced (1.439%); 13 MiB/s, 214 objects/s recovering 2024-04-20T17:52:04.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:03 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2264711472' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:52:04.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:03 smithi106 ceph-mon[44423]: from='client.27961 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:52:04.346 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:52:04.375 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:04.410 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:52:04.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:04.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:52:04.522 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:52:04.554 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:04.581 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:52:04.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:04.586 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:52:04.687 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:52:04.722 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:04.749 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:52:04.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:04.752 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:52:04.853 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:52:04.893 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:04.923 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:52:04.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:04.930 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:52:05.031 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:52:05.081 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:04 smithi005 ceph-mon[39562]: 9.7 scrub starts 2024-04-20T17:52:05.081 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:04 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:52:05.081 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:04 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:52:05.081 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:04 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:52:05.081 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:04 smithi005 ceph-mon[39562]: from='client.27961 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:52:05.081 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:04 smithi005 ceph-mon[39562]: osdmap e957: 12 total, 12 up, 12 in 2024-04-20T17:52:05.085 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:05.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:04 smithi106 ceph-mon[44423]: 9.7 scrub starts 2024-04-20T17:52:05.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:04 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:52:05.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:04 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:52:05.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:04 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:52:05.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:04 smithi106 ceph-mon[44423]: from='client.27961 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:52:05.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:04 smithi106 ceph-mon[44423]: osdmap e957: 12 total, 12 up, 12 in 2024-04-20T17:52:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:04 smithi171 ceph-mon[44189]: 9.7 scrub starts 2024-04-20T17:52:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:04 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:52:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:04 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:52:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:04 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:52:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:04 smithi171 ceph-mon[44189]: from='client.27961 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:52:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:04 smithi171 ceph-mon[44189]: osdmap e957: 12 total, 12 up, 12 in 2024-04-20T17:52:05.749 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:52:05.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:05.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:52:05.863 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:52:05.907 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:05.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:05 smithi005 ceph-mon[39562]: pgmap v2526: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 25 active+clean; 4.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 2271/154332 objects misplaced (1.472%); 8.1 MiB/s, 130 objects/s recovering 2024-04-20T17:52:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:05 smithi106 ceph-mon[44423]: pgmap v2526: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 25 active+clean; 4.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 2271/154332 objects misplaced (1.472%); 8.1 MiB/s, 130 objects/s recovering 2024-04-20T17:52:06.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:05 smithi171 ceph-mon[44189]: pgmap v2526: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 25 active+clean; 4.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 2271/154332 objects misplaced (1.472%); 8.1 MiB/s, 130 objects/s recovering 2024-04-20T17:52:07.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:06 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:52:07.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:06 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:52:07.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:06 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:52:07.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:06 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:52:07.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:06 smithi005 ceph-mon[39562]: osdmap e958: 12 total, 12 up, 12 in 2024-04-20T17:52:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:06 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:52:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:06 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:52:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:06 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:52:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:06 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:52:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:06 smithi106 ceph-mon[44423]: osdmap e958: 12 total, 12 up, 12 in 2024-04-20T17:52:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:06 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:52:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:06 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:52:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:06 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:52:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:06 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:52:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:06 smithi171 ceph-mon[44189]: osdmap e958: 12 total, 12 up, 12 in 2024-04-20T17:52:07.682 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:52:07.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:07.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:52:07.790 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:52:07.827 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:07.856 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:52:07.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:07.859 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:52:07.960 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:52:07.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:07 smithi005 ceph-mon[39562]: pgmap v2528: 27 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 25 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 1004/148472 objects misplaced (0.676%); 12 MiB/s, 188 objects/s recovering 2024-04-20T17:52:07.990 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:08.019 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:52:08.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:08.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:52:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:07 smithi171 ceph-mon[44189]: pgmap v2528: 27 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 25 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 1004/148472 objects misplaced (0.676%); 12 MiB/s, 188 objects/s recovering 2024-04-20T17:52:08.123 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:52:08.156 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:08.180 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:52:08.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:08.185 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:52:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:07 smithi106 ceph-mon[44423]: pgmap v2528: 27 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 25 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 1004/148472 objects misplaced (0.676%); 12 MiB/s, 188 objects/s recovering 2024-04-20T17:52:08.285 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:52:08.342 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:08.368 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:52:08.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:08.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:52:08.476 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:52:08.509 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:08.535 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:52:08.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:08.539 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:52:08.640 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:52:08.673 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:08.702 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:52:08.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:08.707 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:52:08.808 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:52:08.872 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:08.879 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:52:08.879 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:52:08.880 INFO:teuthology.orchestra.run.smithi005.stdout: "complaint_time": 30, 2024-04-20T17:52:08.880 INFO:teuthology.orchestra.run.smithi005.stdout: "num_blocked_ops": 0 2024-04-20T17:52:08.880 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:52:08.961 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:52:08.962 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:52:08.962 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:52:08.962 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:52:08.962 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:52:08.965 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:52:08.998 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:52:09.022 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:52:09.055 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:52:09.098 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:52:09.125 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:52:09.161 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:52:09.170 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:52:09.177 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:52:09.177 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:52:09.177 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:52:09.177 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:52:09.177 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:52:09.177 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:52:09.178 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:52:09.178 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:52:09.178 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:52:09.178 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:52:09.178 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:52:09.178 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:52:09.206 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:08 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3645273393' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:52:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:08 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3645273393' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:52:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:08 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3645273393' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:52:09.723 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:52:10.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:09 smithi005 ceph-mon[39562]: pgmap v2529: 27 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 25 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 850 op/s; 1004/147206 objects misplaced (0.682%); 8.9 MiB/s, 143 objects/s recovering 2024-04-20T17:52:10.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:10.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:10.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:10.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:10.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:10.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:10.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:10.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:10.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:09 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:09 smithi106 ceph-mon[44423]: pgmap v2529: 27 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 25 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 850 op/s; 1004/147206 objects misplaced (0.682%); 8.9 MiB/s, 143 objects/s recovering 2024-04-20T17:52:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:10.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:09 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:09 smithi171 ceph-mon[44189]: pgmap v2529: 27 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 25 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 850 op/s; 1004/147206 objects misplaced (0.682%); 8.9 MiB/s, 143 objects/s recovering 2024-04-20T17:52:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:10.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:10.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:09 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:10.834 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:52:10.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:10.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:52:10.866 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-20T17:52:10.939 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:52:10.991 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:11.090 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:11.090 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:11.090 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:10 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:11.090 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:10 smithi005 ceph-mon[39562]: osdmap e959: 12 total, 12 up, 12 in 2024-04-20T17:52:11.091 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:10 smithi005 ceph-mon[39562]: pgmap v2531: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 25 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 1004/140548 objects misplaced (0.714%); 10 MiB/s, 165 objects/s recovering 2024-04-20T17:52:11.095 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:52:11.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:11.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:52:11.210 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:52:11.237 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:11.237 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:11.237 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:10 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:11.237 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:10 smithi106 ceph-mon[44423]: osdmap e959: 12 total, 12 up, 12 in 2024-04-20T17:52:11.237 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:10 smithi106 ceph-mon[44423]: pgmap v2531: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 25 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 1004/140548 objects misplaced (0.714%); 10 MiB/s, 165 objects/s recovering 2024-04-20T17:52:11.238 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:11.269 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:52:11.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:11.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:52:11.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:11.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:11.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:10 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:11.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:10 smithi171 ceph-mon[44189]: osdmap e959: 12 total, 12 up, 12 in 2024-04-20T17:52:11.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:10 smithi171 ceph-mon[44189]: pgmap v2531: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 25 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 1004/140548 objects misplaced (0.714%); 10 MiB/s, 165 objects/s recovering 2024-04-20T17:52:11.381 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:52:11.413 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:11.435 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:52:11.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:11.442 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:52:11.543 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:52:11.585 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:11.618 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:52:11.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:11.623 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:52:11.723 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:52:11.761 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:11.787 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:52:11.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:11.790 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:52:11.891 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:52:11.944 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:11.979 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:52:11.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:11.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:52:12.083 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:52:12.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:11 smithi171 ceph-mon[44189]: osdmap e960: 12 total, 12 up, 12 in 2024-04-20T17:52:12.125 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:12.187 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:52:12.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:12.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:52:12.208 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:52:12.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:11 smithi005 ceph-mon[39562]: osdmap e960: 12 total, 12 up, 12 in 2024-04-20T17:52:12.223 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:52:12.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:11 smithi106 ceph-mon[44423]: osdmap e960: 12 total, 12 up, 12 in 2024-04-20T17:52:12.291 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:52:12.354 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:12.378 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:52:12.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:12.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:52:12.484 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:52:12.529 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:12.665 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:52:12.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:12.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:52:12.770 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:52:12.804 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:13.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:12 smithi005 ceph-mon[39562]: pgmap v2533: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 863 op/s; 8.9 MiB/s, 143 objects/s recovering 2024-04-20T17:52:13.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:12 smithi106 ceph-mon[44423]: pgmap v2533: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 863 op/s; 8.9 MiB/s, 143 objects/s recovering 2024-04-20T17:52:13.265 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:52:13.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:13.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:52:13.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:12 smithi171 ceph-mon[44189]: pgmap v2533: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 863 op/s; 8.9 MiB/s, 143 objects/s recovering 2024-04-20T17:52:13.372 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:52:13.405 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:13.433 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:52:13.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:13.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:52:13.537 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:52:13.567 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:13.604 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:52:13.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:13.608 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:52:13.656 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:52:13.708 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:52:13.763 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:13.795 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:52:13.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:13.809 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:52:13.910 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:52:13.939 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:13.959 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:52:13.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:13.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:52:14.065 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:52:14.104 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:14.669 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:52:14.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:14.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:52:14.778 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:52:14.806 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:14.830 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:52:14.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:14.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:52:14.935 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:52:14.973 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:15.209 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:52:15.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:15.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:52:15.315 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:52:15.358 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:15.567 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:52:15.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:15.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:52:15.675 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:52:15.705 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:15.730 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:52:15.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:15.734 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:52:15.835 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:52:15.884 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:15 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:52:15.884 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:15 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:52:15.884 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:15 smithi005 ceph-mon[39562]: osdmap e961: 12 total, 12 up, 12 in 2024-04-20T17:52:15.884 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:15 smithi005 ceph-mon[39562]: pgmap v2535: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 991 op/s; 10 MiB/s, 163 objects/s recovering 2024-04-20T17:52:15.927 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:15 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:52:15.927 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:15 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:52:15.927 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:15 smithi106 ceph-mon[44423]: osdmap e961: 12 total, 12 up, 12 in 2024-04-20T17:52:15.927 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:15 smithi106 ceph-mon[44423]: pgmap v2535: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 991 op/s; 10 MiB/s, 163 objects/s recovering 2024-04-20T17:52:15.930 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:15.951 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:52:15.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:15.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:52:16.059 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:52:16.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:15 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:52:16.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:15 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:52:16.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:15 smithi171 ceph-mon[44189]: osdmap e961: 12 total, 12 up, 12 in 2024-04-20T17:52:16.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:15 smithi171 ceph-mon[44189]: pgmap v2535: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 991 op/s; 10 MiB/s, 163 objects/s recovering 2024-04-20T17:52:16.106 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:16.788 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:52:16.800 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:52:17.245 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:52:17.784 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:17 smithi005 ceph-mon[39562]: pgmap v2536: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 933 op/s; 9.0 MiB/s, 143 objects/s recovering 2024-04-20T17:52:17.785 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:17 smithi005 ceph-mon[39562]: from='client.27979 v1:172.21.15.5:0/4289722918' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:52:17.785 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:17 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/13747467' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:52:17.785 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:17 smithi005 ceph-mon[39562]: from='client.27976 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:52:17.801 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:52:18.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:17 smithi171 ceph-mon[44189]: pgmap v2536: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 933 op/s; 9.0 MiB/s, 143 objects/s recovering 2024-04-20T17:52:18.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:17 smithi171 ceph-mon[44189]: from='client.27979 v1:172.21.15.5:0/4289722918' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:52:18.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:17 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/13747467' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:52:18.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:17 smithi171 ceph-mon[44189]: from='client.27976 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:52:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:17 smithi106 ceph-mon[44423]: pgmap v2536: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 933 op/s; 9.0 MiB/s, 143 objects/s recovering 2024-04-20T17:52:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:17 smithi106 ceph-mon[44423]: from='client.27979 v1:172.21.15.5:0/4289722918' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:52:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:17 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/13747467' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:52:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:17 smithi106 ceph-mon[44423]: from='client.27976 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:52:19.052 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:18 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:52:19.052 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:18 smithi005 ceph-mon[39562]: from='client.27976 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:52:19.052 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:18 smithi005 ceph-mon[39562]: osdmap e962: 12 total, 12 up, 12 in 2024-04-20T17:52:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:18 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:52:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:18 smithi171 ceph-mon[44189]: from='client.27976 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:52:19.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:18 smithi171 ceph-mon[44189]: osdmap e962: 12 total, 12 up, 12 in 2024-04-20T17:52:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:18 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:52:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:18 smithi106 ceph-mon[44423]: from='client.27976 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:52:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:18 smithi106 ceph-mon[44423]: osdmap e962: 12 total, 12 up, 12 in 2024-04-20T17:52:19.534 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":2536,"stamp":"2024-04-20T17:52:16.720343+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4267115064,"num_objects":65105,"num_object_clones":0,"num_object_copies":130210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65105,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":268485,"num_write_kb":10674753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12720,"num_bytes_recovered":833486848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162384,"ondisk_log_size":162384,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":15454400,"kb_used_data":8610312,"kb_used_omap":404,"kb_used_meta":6843435,"kb_avail":1109487424,"statfs":{"total":1151940427776,"available":1136115122176,"internally_reserved":0,"allocated":8816959488,"data_stored":8812457054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":414014,"internal_metadata":7007678146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-416677888,"num_objects":-6358,"num_object_clones":0,"num_object_copies":-12716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1004,"num_objects_unfound":0,"num_objects_dirty":-6358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6358,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":980,"num_bytes_recovered":64225280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.812894"},"pg_stats":[{"pgid":"9.7","version":"961'16659","reported_seq":16950,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-04-20T17:52:15.228301+0000","last_change":"2024-04-20T17:52:03.913973+0000","last_active":"2024-04-20T17:52:15.228301+0000","last_peered":"2024-04-20T17:52:15.228301+0000","last_clean":"2024-04-20T17:52:15.228301+0000","last_became_active":"2024-04-20T17:48:54.814508+0000","last_became_peered":"2024-04-20T17:48:54.814508+0000","last_unstale":"2024-04-20T17:52:15.228301+0000","last_undegraded":"2024-04-20T17:52:15.228301+0000","last_fullsized":"2024-04-20T17:52:15.228301+0000","mapping_epoch":894,"log_start":"918'6600","ondisk_log_start":"918'6600","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'9665","last_scrub_stamp":"2024-04-20T17:50:27.211421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:27.211421+0000","objects_scrubbed":4049,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:16.341681+0000","scrub_duration":2310,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262340608,"num_objects":4003,"num_object_clones":0,"num_object_copies":8006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16659,"num_write_kb":661184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"961'16860","reported_seq":27001,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-04-20T17:52:16.406873+0000","last_change":"2024-04-20T17:52:11.009257+0000","last_active":"2024-04-20T17:52:16.406873+0000","last_peered":"2024-04-20T17:52:16.406873+0000","last_clean":"2024-04-20T17:52:16.406873+0000","last_became_active":"2024-04-20T17:52:10.932624+0000","last_became_peered":"2024-04-20T17:52:10.932624+0000","last_unstale":"2024-04-20T17:52:16.406873+0000","last_undegraded":"2024-04-20T17:52:16.406873+0000","last_fullsized":"2024-04-20T17:52:16.406873+0000","mapping_epoch":959,"log_start":"918'6800","ondisk_log_start":"918'6800","created":894,"last_epoch_clean":960,"parent":"0.0","parent_split_bits":0,"last_scrub":"929'9608","last_scrub_stamp":"2024-04-20T17:50:25.096273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:25.096273+0000","objects_scrubbed":9524,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:16.683145+0000","scrub_duration":2229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256901120,"num_objects":3920,"num_object_clones":0,"num_object_copies":7840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16860,"num_write_kb":664960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4942,"num_bytes_recovered":323878912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.5","version":"961'16827","reported_seq":17144,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-04-20T17:52:16.142416+0000","last_change":"2024-04-20T17:51:42.519313+0000","last_active":"2024-04-20T17:52:16.142416+0000","last_peered":"2024-04-20T17:52:16.142416+0000","last_clean":"2024-04-20T17:52:16.142416+0000","last_became_active":"2024-04-20T17:48:54.817749+0000","last_became_peered":"2024-04-20T17:48:54.817749+0000","last_unstale":"2024-04-20T17:52:16.142416+0000","last_undegraded":"2024-04-20T17:52:16.142416+0000","last_fullsized":"2024-04-20T17:52:16.142416+0000","mapping_epoch":894,"log_start":"918'6800","ondisk_log_start":"918'6800","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'14734","last_scrub_stamp":"2024-04-20T17:51:42.519071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:42.519071+0000","objects_scrubbed":6056,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:09.147760+0000","scrub_duration":1274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256704512,"num_objects":3917,"num_object_clones":0,"num_object_copies":7834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16827,"num_write_kb":663808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"961'16732","reported_seq":17042,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-04-20T17:52:15.936242+0000","last_change":"2024-04-20T17:51:48.836352+0000","last_active":"2024-04-20T17:52:15.936242+0000","last_peered":"2024-04-20T17:52:15.936242+0000","last_clean":"2024-04-20T17:52:15.936242+0000","last_became_active":"2024-04-20T17:48:55.061199+0000","last_became_peered":"2024-04-20T17:48:55.061199+0000","last_unstale":"2024-04-20T17:52:15.936242+0000","last_undegraded":"2024-04-20T17:52:15.936242+0000","last_fullsized":"2024-04-20T17:52:15.936242+0000","mapping_epoch":894,"log_start":"918'6700","ondisk_log_start":"918'6700","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"954'15006","last_scrub_stamp":"2024-04-20T17:51:48.836218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:48.836218+0000","objects_scrubbed":5762,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:51.196876+0000","scrub_duration":1412,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260833280,"num_objects":3980,"num_object_clones":0,"num_object_copies":7960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16732,"num_write_kb":662784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"960'16551","reported_seq":32434,"reported_epoch":960,"state":"active+clean","last_fresh":"2024-04-20T17:52:13.999678+0000","last_change":"2024-04-20T17:51:42.767947+0000","last_active":"2024-04-20T17:52:13.999678+0000","last_peered":"2024-04-20T17:52:13.999678+0000","last_clean":"2024-04-20T17:52:13.999678+0000","last_became_active":"2024-04-20T17:51:40.523417+0000","last_became_peered":"2024-04-20T17:51:40.523417+0000","last_unstale":"2024-04-20T17:52:13.999678+0000","last_undegraded":"2024-04-20T17:52:13.999678+0000","last_fullsized":"2024-04-20T17:52:13.999678+0000","mapping_epoch":950,"log_start":"918'6500","ondisk_log_start":"918'6500","created":894,"last_epoch_clean":951,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'14655","last_scrub_stamp":"2024-04-20T17:51:42.767651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:42.767651+0000","objects_scrubbed":5939,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:50.590021+0000","scrub_duration":1039,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263782400,"num_objects":4025,"num_object_clones":0,"num_object_copies":8050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16551,"num_write_kb":658432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7778,"num_bytes_recovered":509607936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.e","version":"961'16887","reported_seq":17127,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-04-20T17:52:14.828187+0000","last_change":"2024-04-20T17:51:03.098201+0000","last_active":"2024-04-20T17:52:14.828187+0000","last_peered":"2024-04-20T17:52:14.828187+0000","last_clean":"2024-04-20T17:52:14.828187+0000","last_became_active":"2024-04-20T17:48:54.815159+0000","last_became_peered":"2024-04-20T17:48:54.815159+0000","last_unstale":"2024-04-20T17:52:14.828187+0000","last_undegraded":"2024-04-20T17:52:14.828187+0000","last_fullsized":"2024-04-20T17:52:14.828187+0000","mapping_epoch":894,"log_start":"918'6800","ondisk_log_start":"918'6800","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"942'12102","last_scrub_stamp":"2024-04-20T17:51:03.098072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:03.098072+0000","objects_scrubbed":8857,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:30.248766+0000","scrub_duration":2003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261160960,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16887,"num_write_kb":667904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"15'137","reported_seq":974,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-04-20T17:52:14.644614+0000","last_change":"2024-04-20T17:50:57.911454+0000","last_active":"2024-04-20T17:52:14.644614+0000","last_peered":"2024-04-20T17:52:14.644614+0000","last_clean":"2024-04-20T17:52:14.644614+0000","last_became_active":"2024-04-20T17:45:32.670941+0000","last_became_peered":"2024-04-20T17:45:32.670941+0000","last_unstale":"2024-04-20T17:52:14.644614+0000","last_undegraded":"2024-04-20T17:52:14.644614+0000","last_fullsized":"2024-04-20T17:52:14.644614+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:57.911336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:57.911336+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:16.341715+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"961'16781","reported_seq":17096,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-04-20T17:52:15.655704+0000","last_change":"2024-04-20T17:51:43.065593+0000","last_active":"2024-04-20T17:52:15.655704+0000","last_peered":"2024-04-20T17:52:15.655704+0000","last_clean":"2024-04-20T17:52:15.655704+0000","last_became_active":"2024-04-20T17:48:55.061161+0000","last_became_peered":"2024-04-20T17:48:55.061161+0000","last_unstale":"2024-04-20T17:52:15.655704+0000","last_undegraded":"2024-04-20T17:52:15.655704+0000","last_fullsized":"2024-04-20T17:52:15.655704+0000","mapping_epoch":894,"log_start":"918'6700","ondisk_log_start":"918'6700","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'14822","last_scrub_stamp":"2024-04-20T17:51:43.065456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:43.065456+0000","objects_scrubbed":6081,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:57.084267+0000","scrub_duration":1030,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268369920,"num_objects":4095,"num_object_clones":0,"num_object_copies":8190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16781,"num_write_kb":668032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"15'137","reported_seq":978,"reported_epoch":960,"state":"active+clean","last_fresh":"2024-04-20T17:52:11.085856+0000","last_change":"2024-04-20T17:51:42.855635+0000","last_active":"2024-04-20T17:52:11.085856+0000","last_peered":"2024-04-20T17:52:11.085856+0000","last_clean":"2024-04-20T17:52:11.085856+0000","last_became_active":"2024-04-20T17:45:29.817528+0000","last_became_peered":"2024-04-20T17:45:29.817528+0000","last_unstale":"2024-04-20T17:52:11.085856+0000","last_undegraded":"2024-04-20T17:52:11.085856+0000","last_fullsized":"2024-04-20T17:52:11.085856+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:42.855496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:42.855496+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:49.448590+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"960'16876","reported_seq":17183,"reported_epoch":960,"state":"active+clean","last_fresh":"2024-04-20T17:52:13.742673+0000","last_change":"2024-04-20T17:52:01.435065+0000","last_active":"2024-04-20T17:52:13.742673+0000","last_peered":"2024-04-20T17:52:13.742673+0000","last_clean":"2024-04-20T17:52:13.742673+0000","last_became_active":"2024-04-20T17:48:54.817445+0000","last_became_peered":"2024-04-20T17:48:54.817445+0000","last_unstale":"2024-04-20T17:52:13.742673+0000","last_undegraded":"2024-04-20T17:52:13.742673+0000","last_fullsized":"2024-04-20T17:52:13.742673+0000","mapping_epoch":894,"log_start":"918'6800","ondisk_log_start":"918'6800","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"956'16160","last_scrub_stamp":"2024-04-20T17:52:01.434805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:52:01.434805+0000","objects_scrubbed":4908,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:21.628859+0000","scrub_duration":1125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271450112,"num_objects":4142,"num_object_clones":0,"num_object_copies":8284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16876,"num_write_kb":672576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"15'137","reported_seq":1006,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-04-20T17:52:14.656711+0000","last_change":"2024-04-20T17:51:44.510869+0000","last_active":"2024-04-20T17:52:14.656711+0000","last_peered":"2024-04-20T17:52:14.656711+0000","last_clean":"2024-04-20T17:52:14.656711+0000","last_became_active":"2024-04-20T17:45:29.827577+0000","last_became_peered":"2024-04-20T17:45:29.827577+0000","last_unstale":"2024-04-20T17:52:14.656711+0000","last_undegraded":"2024-04-20T17:52:14.656711+0000","last_fullsized":"2024-04-20T17:52:14.656711+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:44.510742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:44.510742+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:46.934138+0000","scrub_duration":266,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"961'16694","reported_seq":17002,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-04-20T17:52:14.819900+0000","last_change":"2024-04-20T17:51:00.541301+0000","last_active":"2024-04-20T17:52:14.819900+0000","last_peered":"2024-04-20T17:52:14.819900+0000","last_clean":"2024-04-20T17:52:14.819900+0000","last_became_active":"2024-04-20T17:48:55.061208+0000","last_became_peered":"2024-04-20T17:48:55.061208+0000","last_unstale":"2024-04-20T17:52:14.819900+0000","last_undegraded":"2024-04-20T17:52:14.819900+0000","last_fullsized":"2024-04-20T17:52:14.819900+0000","mapping_epoch":894,"log_start":"918'6600","ondisk_log_start":"918'6600","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"941'11888","last_scrub_stamp":"2024-04-20T17:51:00.541192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:00.541192+0000","objects_scrubbed":8973,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:17.064798+0000","scrub_duration":1502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269287448,"num_objects":4110,"num_object_clones":0,"num_object_copies":8220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4110,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16694,"num_write_kb":665665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"15'137","reported_seq":974,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-04-20T17:52:14.644545+0000","last_change":"2024-04-20T17:51:46.197902+0000","last_active":"2024-04-20T17:52:14.644545+0000","last_peered":"2024-04-20T17:52:14.644545+0000","last_clean":"2024-04-20T17:52:14.644545+0000","last_became_active":"2024-04-20T17:45:32.666523+0000","last_became_peered":"2024-04-20T17:45:32.666523+0000","last_unstale":"2024-04-20T17:52:14.644545+0000","last_undegraded":"2024-04-20T17:52:14.644545+0000","last_fullsized":"2024-04-20T17:52:14.644545+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:46.197797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:46.197797+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:56.371129+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.a","version":"961'17027","reported_seq":17276,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-04-20T17:52:15.664823+0000","last_change":"2024-04-20T17:50:22.180642+0000","last_active":"2024-04-20T17:52:15.664823+0000","last_peered":"2024-04-20T17:52:15.664823+0000","last_clean":"2024-04-20T17:52:15.664823+0000","last_became_active":"2024-04-20T17:48:55.061180+0000","last_became_peered":"2024-04-20T17:48:55.061180+0000","last_unstale":"2024-04-20T17:52:15.664823+0000","last_undegraded":"2024-04-20T17:52:15.664823+0000","last_fullsized":"2024-04-20T17:52:15.664823+0000","mapping_epoch":894,"log_start":"920'7000","ondisk_log_start":"920'7000","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9574","last_scrub_stamp":"2024-04-20T17:50:22.180501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:22.180501+0000","objects_scrubbed":9507,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:20.449811+0000","scrub_duration":1824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258932736,"num_objects":3951,"num_object_clones":0,"num_object_copies":7902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17027,"num_write_kb":671296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"15'137","reported_seq":975,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-04-20T17:52:14.640948+0000","last_change":"2024-04-20T17:51:43.176176+0000","last_active":"2024-04-20T17:52:14.640948+0000","last_peered":"2024-04-20T17:52:14.640948+0000","last_clean":"2024-04-20T17:52:14.640948+0000","last_became_active":"2024-04-20T17:46:40.831961+0000","last_became_peered":"2024-04-20T17:46:40.831961+0000","last_unstale":"2024-04-20T17:52:14.640948+0000","last_undegraded":"2024-04-20T17:52:14.640948+0000","last_fullsized":"2024-04-20T17:52:14.640948+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:43.176019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:43.176019+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:57.843728+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"960'16593","reported_seq":16900,"reported_epoch":960,"state":"active+clean","last_fresh":"2024-04-20T17:52:14.004889+0000","last_change":"2024-04-20T17:51:46.152560+0000","last_active":"2024-04-20T17:52:14.004889+0000","last_peered":"2024-04-20T17:52:14.004889+0000","last_clean":"2024-04-20T17:52:14.004889+0000","last_became_active":"2024-04-20T17:48:54.816644+0000","last_became_peered":"2024-04-20T17:48:54.816644+0000","last_unstale":"2024-04-20T17:52:14.004889+0000","last_undegraded":"2024-04-20T17:52:14.004889+0000","last_fullsized":"2024-04-20T17:52:14.004889+0000","mapping_epoch":894,"log_start":"918'6500","ondisk_log_start":"918'6500","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"952'14792","last_scrub_stamp":"2024-04-20T17:51:46.152388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:46.152388+0000","objects_scrubbed":5974,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:04.134118+0000","scrub_duration":519,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273481728,"num_objects":4173,"num_object_clones":0,"num_object_copies":8346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16593,"num_write_kb":664512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"15'137","reported_seq":765,"reported_epoch":960,"state":"active+clean","last_fresh":"2024-04-20T17:52:10.922335+0000","last_change":"2024-04-20T17:51:00.057261+0000","last_active":"2024-04-20T17:52:10.922335+0000","last_peered":"2024-04-20T17:52:10.922335+0000","last_clean":"2024-04-20T17:52:10.922335+0000","last_became_active":"2024-04-20T17:49:46.824559+0000","last_became_peered":"2024-04-20T17:49:46.824559+0000","last_unstale":"2024-04-20T17:52:10.922335+0000","last_undegraded":"2024-04-20T17:52:10.922335+0000","last_fullsized":"2024-04-20T17:52:10.922335+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:00.057156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:00.057156+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:13.494913+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"961'17141","reported_seq":17458,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-04-20T17:52:15.242114+0000","last_change":"2024-04-20T17:51:50.104395+0000","last_active":"2024-04-20T17:52:15.242114+0000","last_peered":"2024-04-20T17:52:15.242114+0000","last_clean":"2024-04-20T17:52:15.242114+0000","last_became_active":"2024-04-20T17:48:54.813199+0000","last_became_peered":"2024-04-20T17:48:54.813199+0000","last_unstale":"2024-04-20T17:52:15.242114+0000","last_undegraded":"2024-04-20T17:52:15.242114+0000","last_fullsized":"2024-04-20T17:52:15.242114+0000","mapping_epoch":894,"log_start":"920'7100","ondisk_log_start":"920'7100","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"954'15465","last_scrub_stamp":"2024-04-20T17:51:50.104218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:50.104218+0000","objects_scrubbed":5848,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:05.037497+0000","scrub_duration":1044,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271384576,"num_objects":4141,"num_object_clones":0,"num_object_copies":8282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17141,"num_write_kb":681024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"15'137","reported_seq":974,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-04-20T17:52:14.644532+0000","last_change":"2024-04-20T17:50:58.932166+0000","last_active":"2024-04-20T17:52:14.644532+0000","last_peered":"2024-04-20T17:52:14.644532+0000","last_clean":"2024-04-20T17:52:14.644532+0000","last_became_active":"2024-04-20T17:45:32.782706+0000","last_became_peered":"2024-04-20T17:45:32.782706+0000","last_unstale":"2024-04-20T17:52:14.644532+0000","last_undegraded":"2024-04-20T17:52:14.644532+0000","last_fullsized":"2024-04-20T17:52:14.644532+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:58.932046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:58.932046+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:23.591238+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.2","version":"960'16717","reported_seq":17027,"reported_epoch":960,"state":"active+clean","last_fresh":"2024-04-20T17:52:13.734610+0000","last_change":"2024-04-20T17:51:43.530794+0000","last_active":"2024-04-20T17:52:13.734610+0000","last_peered":"2024-04-20T17:52:13.734610+0000","last_clean":"2024-04-20T17:52:13.734610+0000","last_became_active":"2024-04-20T17:48:55.060965+0000","last_became_peered":"2024-04-20T17:48:55.060965+0000","last_unstale":"2024-04-20T17:52:13.734610+0000","last_undegraded":"2024-04-20T17:52:13.734610+0000","last_fullsized":"2024-04-20T17:52:13.734610+0000","mapping_epoch":894,"log_start":"918'6700","ondisk_log_start":"918'6700","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'14849","last_scrub_stamp":"2024-04-20T17:51:43.530564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:43.530564+0000","objects_scrubbed":5973,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:05.542393+0000","scrub_duration":1032,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266272768,"num_objects":4063,"num_object_clones":0,"num_object_copies":8126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16717,"num_write_kb":664960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"15'137","reported_seq":958,"reported_epoch":960,"state":"active+clean","last_fresh":"2024-04-20T17:52:10.926714+0000","last_change":"2024-04-20T17:51:41.848577+0000","last_active":"2024-04-20T17:52:10.926714+0000","last_peered":"2024-04-20T17:52:10.926714+0000","last_clean":"2024-04-20T17:52:10.926714+0000","last_became_active":"2024-04-20T17:45:32.669451+0000","last_became_peered":"2024-04-20T17:45:32.669451+0000","last_unstale":"2024-04-20T17:52:10.926714+0000","last_undegraded":"2024-04-20T17:52:10.926714+0000","last_fullsized":"2024-04-20T17:52:10.926714+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:41.848440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:41.848440+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:55.264457+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"960'16487","reported_seq":16805,"reported_epoch":960,"state":"active+clean","last_fresh":"2024-04-20T17:52:11.738310+0000","last_change":"2024-04-20T17:51:42.714681+0000","last_active":"2024-04-20T17:52:11.738310+0000","last_peered":"2024-04-20T17:52:11.738310+0000","last_clean":"2024-04-20T17:52:11.738310+0000","last_became_active":"2024-04-20T17:48:55.061200+0000","last_became_peered":"2024-04-20T17:48:55.061200+0000","last_unstale":"2024-04-20T17:52:11.738310+0000","last_undegraded":"2024-04-20T17:52:11.738310+0000","last_fullsized":"2024-04-20T17:52:11.738310+0000","mapping_epoch":894,"log_start":"917'6400","ondisk_log_start":"917'6400","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'14728","last_scrub_stamp":"2024-04-20T17:51:42.714540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:42.714540+0000","objects_scrubbed":6058,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:00.487021+0000","scrub_duration":1002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279379968,"num_objects":4263,"num_object_clones":0,"num_object_copies":8526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16487,"num_write_kb":664000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"15'137","reported_seq":975,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-04-20T17:52:14.666931+0000","last_change":"2024-04-20T17:51:40.826215+0000","last_active":"2024-04-20T17:52:14.666931+0000","last_peered":"2024-04-20T17:52:14.666931+0000","last_clean":"2024-04-20T17:52:14.666931+0000","last_became_active":"2024-04-20T17:45:32.644584+0000","last_became_peered":"2024-04-20T17:45:32.644584+0000","last_unstale":"2024-04-20T17:52:14.666931+0000","last_undegraded":"2024-04-20T17:52:14.666931+0000","last_fullsized":"2024-04-20T17:52:14.666931+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:40.826081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:40.826081+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:45.293420+0000","scrub_duration":185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"960'18612","reported_seq":18939,"reported_epoch":960,"state":"active+clean+scrubbing+deep","last_fresh":"2024-04-20T17:52:13.733900+0000","last_change":"2024-04-20T17:52:02.263489+0000","last_active":"2024-04-20T17:52:13.733900+0000","last_peered":"2024-04-20T17:52:13.733900+0000","last_clean":"2024-04-20T17:52:13.733900+0000","last_became_active":"2024-04-20T17:48:54.813062+0000","last_became_peered":"2024-04-20T17:48:54.813062+0000","last_unstale":"2024-04-20T17:52:13.733900+0000","last_undegraded":"2024-04-20T17:52:13.733900+0000","last_fullsized":"2024-04-20T17:52:13.733900+0000","mapping_epoch":894,"log_start":"925'8600","ondisk_log_start":"925'8600","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'10026","last_scrub_stamp":"2024-04-20T17:50:30.712648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:30.712648+0000","objects_scrubbed":1798,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"deep scrubbing for 11s","scrub_duration":1871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268697600,"num_objects":4100,"num_object_clones":0,"num_object_copies":8200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16820,"num_write_kb":669440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"15'137","reported_seq":921,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-04-20T17:52:14.640941+0000","last_change":"2024-04-20T17:51:45.119466+0000","last_active":"2024-04-20T17:52:14.640941+0000","last_peered":"2024-04-20T17:52:14.640941+0000","last_clean":"2024-04-20T17:52:14.640941+0000","last_became_active":"2024-04-20T17:46:40.881713+0000","last_became_peered":"2024-04-20T17:46:40.881713+0000","last_unstale":"2024-04-20T17:52:14.640941+0000","last_undegraded":"2024-04-20T17:52:14.640941+0000","last_fullsized":"2024-04-20T17:52:14.640941+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:45.119332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:45.119332+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:51.157610+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.8","version":"960'16833","reported_seq":17146,"reported_epoch":960,"state":"active+clean","last_fresh":"2024-04-20T17:52:12.225473+0000","last_change":"2024-04-20T17:51:43.840908+0000","last_active":"2024-04-20T17:52:12.225473+0000","last_peered":"2024-04-20T17:52:12.225473+0000","last_clean":"2024-04-20T17:52:12.225473+0000","last_became_active":"2024-04-20T17:48:54.813368+0000","last_became_peered":"2024-04-20T17:48:54.813368+0000","last_unstale":"2024-04-20T17:52:12.225473+0000","last_undegraded":"2024-04-20T17:52:12.225473+0000","last_fullsized":"2024-04-20T17:52:12.225473+0000","mapping_epoch":894,"log_start":"918'6800","ondisk_log_start":"918'6800","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'15051","last_scrub_stamp":"2024-04-20T17:51:43.840798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:43.840798+0000","objects_scrubbed":6061,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:52.336868+0000","scrub_duration":1160,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277544960,"num_objects":4235,"num_object_clones":0,"num_object_copies":8470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16833,"num_write_kb":674176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":919,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-04-20T17:52:14.640940+0000","last_change":"2024-04-20T17:51:48.160638+0000","last_active":"2024-04-20T17:52:14.640940+0000","last_peered":"2024-04-20T17:52:14.640940+0000","last_clean":"2024-04-20T17:52:14.640940+0000","last_became_active":"2024-04-20T17:46:40.851683+0000","last_became_peered":"2024-04-20T17:46:40.851683+0000","last_unstale":"2024-04-20T17:52:14.640940+0000","last_undegraded":"2024-04-20T17:52:14.640940+0000","last_fullsized":"2024-04-20T17:52:14.640940+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:48.160546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:48.160546+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:56.139128+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":4266524696,"num_objects":65103,"num_object_clones":0,"num_object_copies":130206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65103,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":268485,"num_write_kb":10674753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12720,"num_bytes_recovered":833486848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8516870144,"data_stored":8516862000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160877,"ondisk_log_size":160877,"up":32,"acting":32,"num_store_stats":11},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},"log_size":1507,"ondisk_log_size":1507,"up":22,"acting":22,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991792,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1712036,"kb_used_data":778396,"kb_used_omap":1,"kb_used_meta":933566,"kb_avail":92033116,"statfs":{"total":95995035648,"available":94241910784,"internally_reserved":0,"allocated":797077504,"data_stored":796699565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":955972036},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253431,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1904408,"kb_used_data":1052752,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":91840744,"statfs":{"total":95995035648,"available":94044921856,"internally_reserved":0,"allocated":1078018048,"data_stored":1077646113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":872085952},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613179,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1306528,"kb_used_data":545104,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":92438624,"statfs":{"total":95995035648,"available":94657150976,"internally_reserved":0,"allocated":558186496,"data_stored":557814561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":779680192},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809410,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":999832,"kb_used_data":527184,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":92745320,"statfs":{"total":95995035648,"available":94971207680,"internally_reserved":0,"allocated":539836416,"data_stored":539464481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":799,"seq":3431678869600,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":921052,"kb_used_data":786452,"kb_used_omap":11,"kb_used_meta":134580,"kb_avail":92824100,"statfs":{"total":95995035648,"available":95051878400,"internally_reserved":0,"allocated":805326848,"data_stored":804950849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11990,"internal_metadata":137810218},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":535,"seq":2297807503639,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":336144,"kb_used_data":276356,"kb_used_omap":170,"kb_used_meta":59605,"kb_avail":93409008,"statfs":{"total":95995035648,"available":95650824192,"internally_reserved":0,"allocated":282988544,"data_stored":282622655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":174628,"internal_metadata":61035996},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":659,"seq":2830383448262,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1952540,"kb_used_data":1582740,"kb_used_omap":4,"kb_used_meta":369723,"kb_avail":91792612,"statfs":{"total":95995035648,"available":93995634688,"internally_reserved":0,"allocated":1620725760,"data_stored":1620349753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4410,"internal_metadata":378597062},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738985,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":744104,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":93001048,"statfs":{"total":95995035648,"available":95233073152,"internally_reserved":0,"allocated":749568,"data_stored":371629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394274,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":803180,"kb_used_data":357732,"kb_used_omap":202,"kb_used_meta":445237,"kb_avail":92941972,"statfs":{"total":95995035648,"available":95172579328,"internally_reserved":0,"allocated":366317568,"data_stored":365936077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":455923159},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653323,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1618264,"kb_used_data":1048592,"kb_used_omap":4,"kb_used_meta":569659,"kb_avail":92126888,"statfs":{"total":95995035648,"available":94337933312,"internally_reserved":0,"allocated":1073758208,"data_stored":1073386273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":583331527},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986772,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1138792,"kb_used_data":516188,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":92606360,"statfs":{"total":95995035648,"available":94828912640,"internally_reserved":0,"allocated":528576512,"data_stored":528198573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790542,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2017520,"kb_used_data":1138084,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":91727632,"statfs":{"total":95995035648,"available":93929095168,"internally_reserved":0,"allocated":1165398016,"data_stored":1165016525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":900528581},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1609699328,"data_stored":1609695256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":256901120,"data_stored":256901120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1070792704,"data_stored":1070792704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":523763712,"data_stored":523763712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1045430272,"data_stored":1045430272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":279379968,"data_stored":279379968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":794497024,"data_stored":794492952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":537264128,"data_stored":537264128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":552402944,"data_stored":552402944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1061158912,"data_stored":1061158912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":785580032,"data_stored":785580032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:52:19.536 INFO:tasks.cephadm.ceph_manager.ceph:PG 9.1 is not active+clean 2024-04-20T17:52:19.536 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '9.1', 'version': "960'18612", 'reported_seq': 18939, 'reported_epoch': 960, 'state': 'active+clean+scrubbing+deep', 'last_fresh': '2024-04-20T17:52:13.733900+0000', 'last_change': '2024-04-20T17:52:02.263489+0000', 'last_active': '2024-04-20T17:52:13.733900+0000', 'last_peered': '2024-04-20T17:52:13.733900+0000', 'last_clean': '2024-04-20T17:52:13.733900+0000', 'last_became_active': '2024-04-20T17:48:54.813062+0000', 'last_became_peered': '2024-04-20T17:48:54.813062+0000', 'last_unstale': '2024-04-20T17:52:13.733900+0000', 'last_undegraded': '2024-04-20T17:52:13.733900+0000', 'last_fullsized': '2024-04-20T17:52:13.733900+0000', 'mapping_epoch': 894, 'log_start': "925'8600", 'ondisk_log_start': "925'8600", 'created': 894, 'last_epoch_clean': 895, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "930'10026", 'last_scrub_stamp': '2024-04-20T17:50:30.712648+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T17:48:53.794927+0000', 'last_clean_scrub_stamp': '2024-04-20T17:50:30.712648+0000', 'objects_scrubbed': 1798, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'deep scrubbing for 11s', 'scrub_duration': 1871, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 268697600, 'num_objects': 4100, 'num_object_clones': 0, 'num_object_copies': 8200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4100, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16820, 'num_write_kb': 669440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 9], 'acting': [3, 9], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-20T17:52:19.536 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph status --format=json 2024-04-20T17:52:19.790 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:19 smithi005 ceph-mon[39562]: pgmap v2538: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 743 op/s; 7.9 MiB/s, 125 objects/s recovering 2024-04-20T17:52:19.893 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:52:19.893 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:52:20.088 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:19 smithi171 ceph-mon[44189]: pgmap v2538: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 743 op/s; 7.9 MiB/s, 125 objects/s recovering 2024-04-20T17:52:20.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:19 smithi106 ceph-mon[44423]: pgmap v2538: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 743 op/s; 7.9 MiB/s, 125 objects/s recovering 2024-04-20T17:52:20.729 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:52:20.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:20.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:52:20.836 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:52:20.891 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:20.918 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:52:20.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:20.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:52:21.023 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:52:21.053 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:21.082 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:52:21.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:21.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:52:21.093 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:52:21.128 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:52:21.170 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:52:21.186 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:52:21.208 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:52:21.250 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:21.265 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:52:21.291 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:52:21.327 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:52:21.356 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:52:21.386 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:52:21.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:21.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:52:21.407 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:52:21.413 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-20T17:52:21.439 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:52:21.482 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:52:21.493 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:52:21.495 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:52:21.526 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:21.539 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:52:21.551 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:52:21.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:21.555 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:52:21.576 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:52:21.592 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:52:21.592 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:52:21.593 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:52:21.593 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:52:21.593 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:52:21.593 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:52:21.593 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:52:21.593 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:52:21.593 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:52:21.593 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:52:21.593 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:52:21.593 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:52:21.656 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:52:21.686 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:21.715 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:52:21.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:21.719 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:52:21.820 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:52:21.851 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:21.877 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:52:21.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:21.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:52:21.982 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:52:22.042 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:22.057 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:21 smithi005 ceph-mon[39562]: pgmap v2539: 27 pgs: 27 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s 2024-04-20T17:52:22.057 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:21 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3736119501' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:52:22.057 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:22.058 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:22.058 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:21 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:21 smithi171 ceph-mon[44189]: pgmap v2539: 27 pgs: 27 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s 2024-04-20T17:52:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:21 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3736119501' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:52:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:22.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:21 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:21 smithi106 ceph-mon[44423]: pgmap v2539: 27 pgs: 27 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s 2024-04-20T17:52:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:21 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3736119501' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:52:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:21 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:22.532 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:52:22.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:22.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:52:22.636 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:52:22.674 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:22.708 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:52:22.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:22.711 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:52:22.807 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:22 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:22.807 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:22 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:22.807 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:22 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:22.807 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:22 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:22.807 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:22 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:22.807 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:22 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:22.807 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:22 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:22.807 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:22 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:22.807 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:22 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:22.812 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:52:22.864 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:23.015 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:52:23.026 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:52:23.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:23.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:52:23.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:22 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:23.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:22 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:23.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:22 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:23.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:22 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:23.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:22 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:23.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:22 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:23.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:22 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:23.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:22 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:23.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:22 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:23.134 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:52:23.176 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:23.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:22 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:23.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:22 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:23.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:22 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:23.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:22 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:23.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:22 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:23.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:22 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:23.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:22 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:23.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:22 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:23.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:22 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:23.539 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:52:23.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:23.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:52:23.598 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:52:23.654 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:52:23.687 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:23.712 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:52:23.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:23.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:52:23.806 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:23 smithi005 ceph-mon[39562]: pgmap v2540: 27 pgs: 27 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-04-20T17:52:23.819 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:52:23.862 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:23.895 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:52:23.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:23.899 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:52:24.000 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:52:24.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:23 smithi106 ceph-mon[44423]: pgmap v2540: 27 pgs: 27 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-04-20T17:52:24.044 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:24.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:23 smithi171 ceph-mon[44189]: pgmap v2540: 27 pgs: 27 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-04-20T17:52:24.526 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:52:24.726 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:52:24.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:24.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:52:24.839 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:52:24.872 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:24.895 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:52:24.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:24.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:52:25.001 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:52:25.041 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:25 smithi171 ceph-mon[44189]: pgmap v2541: 27 pgs: 27 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s 2024-04-20T17:52:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:25 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:52:26.087 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:25 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:52:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:25 smithi005 ceph-mon[39562]: pgmap v2541: 27 pgs: 27 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s 2024-04-20T17:52:26.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:25 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:52:26.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:25 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:52:26.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:25 smithi106 ceph-mon[44423]: pgmap v2541: 27 pgs: 27 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s 2024-04-20T17:52:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:25 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:52:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:25 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:52:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:26 smithi171 ceph-mon[44189]: osdmap e963: 12 total, 12 up, 12 in 2024-04-20T17:52:27.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:26 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3435385137' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:52:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:26 smithi005 ceph-mon[39562]: osdmap e963: 12 total, 12 up, 12 in 2024-04-20T17:52:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:26 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3435385137' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:52:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:26 smithi106 ceph-mon[44423]: osdmap e963: 12 total, 12 up, 12 in 2024-04-20T17:52:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:26 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3435385137' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-20T17:52:27.759 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:52:27.760 INFO:teuthology.orchestra.run.smithi005.stdout:{"fsid":"13c6e76c-ff37-11ee-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":3114,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":963,"num_osds":12,"num_up_osds":12,"osd_up_since":1713635059,"num_in_osds":12,"osd_in_since":1713635196,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":27}],"num_pgs":27,"num_pools":2,"num_objects":57006,"data_bytes":3736339000,"bytes_used":14776823808,"bytes_avail":1137163603968,"bytes_total":1151940427776,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":1063},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.5:8443/"}},"servicemap":{"epoch":70,"modified":"2024-04-20T17:50:40.688583+0000","services":{}},"progress_events":{}} 2024-04-20T17:52:27.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:27 smithi005 ceph-mon[39562]: pgmap v2543: 27 pgs: 27 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:52:28.007 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:52:28.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:28.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:52:28.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:27 smithi171 ceph-mon[44189]: pgmap v2543: 27 pgs: 27 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:52:28.117 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:52:28.164 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:28.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:27 smithi106 ceph-mon[44423]: pgmap v2543: 27 pgs: 27 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.23k op/s 2024-04-20T17:52:28.366 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:52:28.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:28.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:52:28.471 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:52:28.502 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:28.541 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:52:28.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:28.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:52:28.648 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:52:28.699 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:28.737 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:52:28.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:28.743 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:52:28.844 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:52:28.906 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:28.936 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:52:28.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:28.939 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:52:29.041 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:52:29.086 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:29.358 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:52:29.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:29.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:52:29.398 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-20T17:52:29.398 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:52:29.466 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:52:29.493 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:29.523 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:52:29.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:29.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:52:29.631 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:52:29.669 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:29.700 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:52:29.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:29.704 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:52:29.729 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:52:29.805 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:52:29.838 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:30.052 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:52:30.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:30.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:52:30.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:29 smithi171 ceph-mon[44189]: pgmap v2544: 27 pgs: 27 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s 2024-04-20T17:52:30.134 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:29 smithi005 ceph-mon[39562]: pgmap v2544: 27 pgs: 27 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s 2024-04-20T17:52:30.162 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:52:30.214 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:29 smithi106 ceph-mon[44423]: pgmap v2544: 27 pgs: 27 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s 2024-04-20T17:52:30.267 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:52:30.267 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:52:30.267 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.267 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27772.0:287056 9.2 e963/894 9:4722dcaa:::benchmark_data_smithi171_188319_object120260:head v 963'17877, mlcod=963'17874)", 2024-04-20T17:52:30.267 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:52:30.252373+0000", 2024-04-20T17:52:30.267 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.012935252, 2024-04-20T17:52:30.267 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012958440999999999, 2024-04-20T17:52:30.268 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:52:30.268 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:52:30.268 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:52:30.268 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:52:30.268 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.268 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:52:30.268 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.252373+0000", 2024-04-20T17:52:30.268 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:30.268 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.268 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.268 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:52:30.268 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.252373+0000", 2024-04-20T17:52:30.268 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:30.268 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.269 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.269 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:52:30.269 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.252374+0000", 2024-04-20T17:52:30.269 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.4600000000000003e-07 2024-04-20T17:52:30.269 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.269 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.269 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:52:30.269 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.252380+0000", 2024-04-20T17:52:30.269 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8470000000000001e-06 2024-04-20T17:52:30.269 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.269 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.269 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:52:30.269 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.252387+0000", 2024-04-20T17:52:30.269 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.109e-06 2024-04-20T17:52:30.270 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.270 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.270 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:52:30.270 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.252392+0000", 2024-04-20T17:52:30.270 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3070000000000002e-06 2024-04-20T17:52:30.270 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.270 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.270 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:52:30.270 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.254034+0000", 2024-04-20T17:52:30.270 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001642286 2024-04-20T17:52:30.270 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.270 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.270 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:52:30.270 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.254055+0000", 2024-04-20T17:52:30.271 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0295000000000001e-05 2024-04-20T17:52:30.271 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:30.271 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:52:30.271 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:30.271 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.271 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.271 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27772.0:287065 9.a e963/894 9:57da85c4:::benchmark_data_smithi171_188319_object120269:head v 963'18085, mlcod=963'18083)", 2024-04-20T17:52:30.271 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:52:30.258030+0000", 2024-04-20T17:52:30.271 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.0072779469999999999, 2024-04-20T17:52:30.271 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0073695180000000002, 2024-04-20T17:52:30.271 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:52:30.271 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:52:30.271 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:52:30.272 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:52:30.272 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.272 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:52:30.272 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.258030+0000", 2024-04-20T17:52:30.272 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:30.272 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.272 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.272 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:52:30.272 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.258030+0000", 2024-04-20T17:52:30.272 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:30.272 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.272 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.272 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:52:30.272 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.258031+0000", 2024-04-20T17:52:30.273 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1009999999999999e-06 2024-04-20T17:52:30.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.273 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.273 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:52:30.273 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.258038+0000", 2024-04-20T17:52:30.273 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.229e-06 2024-04-20T17:52:30.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.273 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.273 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:52:30.273 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.258045+0000", 2024-04-20T17:52:30.273 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.34e-06 2024-04-20T17:52:30.273 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.273 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.273 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:52:30.274 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.258050+0000", 2024-04-20T17:52:30.274 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3140000000000003e-06 2024-04-20T17:52:30.274 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.274 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.274 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:52:30.274 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.259095+0000", 2024-04-20T17:52:30.274 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.001044403 2024-04-20T17:52:30.274 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.274 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.274 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:52:30.274 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.259114+0000", 2024-04-20T17:52:30.274 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9459000000000001e-05 2024-04-20T17:52:30.274 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:30.275 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:52:30.275 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:30.275 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.275 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.275 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27772.0:287062 9.2 e963/894 9:4a1b3eb8:::benchmark_data_smithi171_188319_object120266:head v 963'17878, mlcod=963'17875)", 2024-04-20T17:52:30.275 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:52:30.259355+0000", 2024-04-20T17:52:30.275 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 0.0059527549999999997, 2024-04-20T17:52:30.275 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0061127839999999996, 2024-04-20T17:52:30.275 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:52:30.275 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:52:30.275 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:52:30.275 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:52:30.275 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.275 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:52:30.276 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.259355+0000", 2024-04-20T17:52:30.276 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:30.276 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.276 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.276 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:52:30.276 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.259355+0000", 2024-04-20T17:52:30.276 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:30.276 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.276 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.276 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:52:30.276 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.259357+0000", 2024-04-20T17:52:30.276 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2640000000000001e-06 2024-04-20T17:52:30.276 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.276 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.276 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:52:30.277 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.259364+0000", 2024-04-20T17:52:30.277 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.374e-06 2024-04-20T17:52:30.277 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.277 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.277 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:52:30.277 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.259374+0000", 2024-04-20T17:52:30.277 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.5400000000000001e-06 2024-04-20T17:52:30.277 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.277 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.277 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:52:30.277 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.259381+0000", 2024-04-20T17:52:30.277 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.131e-06 2024-04-20T17:52:30.277 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.277 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.278 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:52:30.278 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.261144+0000", 2024-04-20T17:52:30.278 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0017632170000000001 2024-04-20T17:52:30.278 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.278 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.278 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:52:30.278 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.261161+0000", 2024-04-20T17:52:30.278 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7448999999999998e-05 2024-04-20T17:52:30.278 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:30.278 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:52:30.278 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:30.278 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.278 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.279 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27772.0:287075 9.a e963/894 9:5af9215e:::benchmark_data_smithi171_188319_object120279:head v 963'18086, mlcod=963'18083)", 2024-04-20T17:52:30.279 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:52:30.265307+0000", 2024-04-20T17:52:30.279 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 1.5269999999999999e-06, 2024-04-20T17:52:30.279 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00023331099999999999, 2024-04-20T17:52:30.279 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:52:30.279 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:52:30.279 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:52:30.279 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:52:30.279 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.279 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:52:30.279 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.265307+0000", 2024-04-20T17:52:30.279 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:30.279 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.279 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.280 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:52:30.280 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.265307+0000", 2024-04-20T17:52:30.280 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:30.280 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.280 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.280 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:52:30.280 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.265308+0000", 2024-04-20T17:52:30.280 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7060000000000001e-06 2024-04-20T17:52:30.280 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.280 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.280 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:52:30.280 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.265318+0000", 2024-04-20T17:52:30.280 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.5999999999999996e-06 2024-04-20T17:52:30.280 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.281 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.281 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:52:30.281 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.265330+0000", 2024-04-20T17:52:30.281 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1898e-05 2024-04-20T17:52:30.281 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.281 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.281 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:52:30.281 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.265337+0000", 2024-04-20T17:52:30.281 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5619999999999998e-06 2024-04-20T17:52:30.281 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.281 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.281 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:52:30.281 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.265390+0000", 2024-04-20T17:52:30.281 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2791999999999999e-05 2024-04-20T17:52:30.281 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.282 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.282 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:52:30.282 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.265413+0000", 2024-04-20T17:52:30.282 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2734999999999998e-05 2024-04-20T17:52:30.282 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:30.282 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:52:30.282 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:30.282 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.282 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.282 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27772.0:287068 9.2 e963/894 9:4e67c679:::benchmark_data_smithi171_188319_object120272:head v 963'17879, mlcod=963'17875)", 2024-04-20T17:52:30.282 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:52:30.265380+0000", 2024-04-20T17:52:30.282 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 4294967295.999928, 2024-04-20T17:52:30.282 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00022499, 2024-04-20T17:52:30.283 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:52:30.283 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:52:30.283 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:52:30.283 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:52:30.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.283 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:52:30.283 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.265380+0000", 2024-04-20T17:52:30.283 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:30.283 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.283 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:52:30.283 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.265380+0000", 2024-04-20T17:52:30.283 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:30.283 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.283 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.284 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:52:30.284 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.265381+0000", 2024-04-20T17:52:30.284 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.192e-06 2024-04-20T17:52:30.284 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.284 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.284 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:52:30.284 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.265388+0000", 2024-04-20T17:52:30.284 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5259999999999999e-06 2024-04-20T17:52:30.284 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.284 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.284 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:52:30.284 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.265395+0000", 2024-04-20T17:52:30.284 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.6690000000000006e-06 2024-04-20T17:52:30.284 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.285 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.285 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:52:30.285 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.265401+0000", 2024-04-20T17:52:30.285 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5820000000000001e-06 2024-04-20T17:52:30.285 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.285 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.285 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:52:30.285 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.265455+0000", 2024-04-20T17:52:30.285 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.3618000000000001e-05 2024-04-20T17:52:30.285 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.285 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.285 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:52:30.285 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.265473+0000", 2024-04-20T17:52:30.285 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8280000000000001e-05 2024-04-20T17:52:30.286 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:30.286 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:52:30.286 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:30.286 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.286 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.286 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27772.0:287073 9.4 e963/894 9:2d2334e7:::benchmark_data_smithi171_188319_object120277:head v 963'17827, mlcod=963'17822)", 2024-04-20T17:52:30.286 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:52:30.265418+0000", 2024-04-20T17:52:30.286 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 4294967295.9998903, 2024-04-20T17:52:30.286 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00024943399999999998, 2024-04-20T17:52:30.286 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:52:30.286 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:52:30.286 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:52:30.286 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:52:30.286 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.287 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:52:30.287 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.265418+0000", 2024-04-20T17:52:30.287 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:30.287 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.287 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.287 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:52:30.287 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.265418+0000", 2024-04-20T17:52:30.287 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:30.287 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.287 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.287 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:52:30.287 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.265419+0000", 2024-04-20T17:52:30.287 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.5899999999999995e-07 2024-04-20T17:52:30.287 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.287 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.288 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:52:30.288 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.265423+0000", 2024-04-20T17:52:30.288 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.6009999999999997e-06 2024-04-20T17:52:30.288 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.288 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.288 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:52:30.288 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.265429+0000", 2024-04-20T17:52:30.288 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4809999999999997e-06 2024-04-20T17:52:30.288 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.288 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.288 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:52:30.288 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.265433+0000", 2024-04-20T17:52:30.288 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.3159999999999998e-06 2024-04-20T17:52:30.288 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.289 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.289 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:52:30.289 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.265489+0000", 2024-04-20T17:52:30.289 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.5881999999999998e-05 2024-04-20T17:52:30.289 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:30.289 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:30.289 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:52:30.289 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:52:30.265507+0000", 2024-04-20T17:52:30.289 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7708000000000001e-05 2024-04-20T17:52:30.289 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:30.289 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:52:30.289 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:30.289 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:30.289 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-04-20T17:52:30.289 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 6 2024-04-20T17:52:30.290 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:52:30.882 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:52:30.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:30.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:52:30.987 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:52:31.033 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:31.063 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:52:31.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:31.069 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:52:31.170 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:52:31.205 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:31.242 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:52:31.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:31.250 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:52:31.351 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:52:31.388 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:31.852 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:52:32.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:31 smithi005 ceph-mon[39562]: pgmap v2545: 27 pgs: 27 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s 2024-04-20T17:52:32.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:31 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1104792153' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:52:32.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:31 smithi005 ceph-mon[39562]: from='client.27994 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:52:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:31 smithi106 ceph-mon[44423]: pgmap v2545: 27 pgs: 27 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s 2024-04-20T17:52:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:31 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1104792153' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:52:32.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:31 smithi106 ceph-mon[44423]: from='client.27994 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:52:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:31 smithi171 ceph-mon[44189]: pgmap v2545: 27 pgs: 27 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s 2024-04-20T17:52:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:31 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1104792153' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:52:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:31 smithi171 ceph-mon[44189]: from='client.27994 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:52:32.727 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-20T17:52:32.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:32 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:52:32.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:32 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:52:32.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:32 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:52:32.965 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:32 smithi005 ceph-mon[39562]: from='client.27994 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:52:32.966 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:32 smithi005 ceph-mon[39562]: osdmap e964: 12 total, 12 up, 12 in 2024-04-20T17:52:32.969 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:52:32.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:32.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:52:33.081 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:52:33.133 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:32 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:52:33.133 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:32 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:52:33.133 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:32 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:52:33.133 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:32 smithi171 ceph-mon[44189]: from='client.27994 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:52:33.133 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:32 smithi171 ceph-mon[44189]: osdmap e964: 12 total, 12 up, 12 in 2024-04-20T17:52:33.136 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:33.171 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:52:33.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:33.176 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:52:33.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:32 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:52:33.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:32 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:52:33.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:32 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:52:33.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:32 smithi106 ceph-mon[44423]: from='client.27994 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:52:33.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:32 smithi106 ceph-mon[44423]: osdmap e964: 12 total, 12 up, 12 in 2024-04-20T17:52:33.277 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:52:33.330 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:33.365 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:52:33.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:33.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:52:33.475 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:52:33.522 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:33.551 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:52:33.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:33.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:52:33.655 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:52:33.707 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:33.738 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:52:33.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:33.741 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:52:33.842 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:52:33.884 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:33.919 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:52:33.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:33.928 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:52:33.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:33 smithi005 ceph-mon[39562]: pgmap v2547: 27 pgs: 27 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.30k op/s 2024-04-20T17:52:33.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:33 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:52:33.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:33 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:52:33.977 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:52:34.028 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:52:34.069 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:33 smithi171 ceph-mon[44189]: pgmap v2547: 27 pgs: 27 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.30k op/s 2024-04-20T17:52:34.069 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:33 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:52:34.069 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:33 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:52:34.070 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:34.101 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:52:34.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:34.105 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:52:34.206 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:52:34.239 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:33 smithi106 ceph-mon[44423]: pgmap v2547: 27 pgs: 27 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.30k op/s 2024-04-20T17:52:34.239 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:33 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:52:34.239 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:33 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:52:34.242 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:34.265 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:52:34.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:34.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:52:34.368 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:52:34.410 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:34.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:34 smithi005 ceph-mon[39562]: osdmap e965: 12 total, 12 up, 12 in 2024-04-20T17:52:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:34 smithi106 ceph-mon[44423]: osdmap e965: 12 total, 12 up, 12 in 2024-04-20T17:52:35.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:34 smithi171 ceph-mon[44189]: osdmap e965: 12 total, 12 up, 12 in 2024-04-20T17:52:35.689 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:52:36.182 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:52:36.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:36.302 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:35 smithi106 ceph-mon[44423]: 9.7 scrub starts 2024-04-20T17:52:36.302 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:35 smithi106 ceph-mon[44423]: pgmap v2549: 27 pgs: 27 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-04-20T17:52:36.302 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:35 smithi106 ceph-mon[44423]: 9.7 scrub ok 2024-04-20T17:52:36.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:52:36.303 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:35 smithi005 ceph-mon[39562]: 9.7 scrub starts 2024-04-20T17:52:36.303 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:35 smithi005 ceph-mon[39562]: pgmap v2549: 27 pgs: 27 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-04-20T17:52:36.303 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:35 smithi005 ceph-mon[39562]: 9.7 scrub ok 2024-04-20T17:52:36.303 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:52:36.304 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:52:36.310 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:52:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:35 smithi171 ceph-mon[44189]: 9.7 scrub starts 2024-04-20T17:52:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:35 smithi171 ceph-mon[44189]: pgmap v2549: 27 pgs: 27 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-04-20T17:52:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:35 smithi171 ceph-mon[44189]: 9.7 scrub ok 2024-04-20T17:52:36.339 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:52:36.374 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:52:36.402 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:52:36.403 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:52:36.450 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:36.456 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:52:36.492 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:52:36.530 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:52:36.564 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:52:36.597 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:52:36.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:36.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:52:36.602 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:52:36.608 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:52:36.614 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:52:36.640 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:52:36.643 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:52:36.679 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:52:36.684 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:52:36.684 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:52:36.684 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:52:36.684 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:52:36.684 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:52:36.684 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:52:36.684 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:52:36.685 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:52:36.685 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:52:36.685 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:52:36.685 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:52:36.685 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:52:36.702 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:52:36.744 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:36.766 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:52:36.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:36.773 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:52:36.874 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:52:36.913 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:36.938 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:52:36.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:36.945 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:52:36.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:36 smithi005 ceph-mon[39562]: 9.d scrub starts 2024-04-20T17:52:36.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:36 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:52:36.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:36 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:52:36.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:36 smithi005 ceph-mon[39562]: 9.d scrub ok 2024-04-20T17:52:36.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:36 smithi005 ceph-mon[39562]: osdmap e966: 12 total, 12 up, 12 in 2024-04-20T17:52:36.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:36 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2042121374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:52:36.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:36.964 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:36 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:37.046 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:52:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:36 smithi171 ceph-mon[44189]: 9.d scrub starts 2024-04-20T17:52:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:36 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:52:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:36 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:52:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:36 smithi171 ceph-mon[44189]: 9.d scrub ok 2024-04-20T17:52:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:36 smithi171 ceph-mon[44189]: osdmap e966: 12 total, 12 up, 12 in 2024-04-20T17:52:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:36 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2042121374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:52:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:37.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:36 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:37.094 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:36 smithi106 ceph-mon[44423]: 9.d scrub starts 2024-04-20T17:52:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:36 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:52:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:36 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:52:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:36 smithi106 ceph-mon[44423]: 9.d scrub ok 2024-04-20T17:52:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:36 smithi106 ceph-mon[44423]: osdmap e966: 12 total, 12 up, 12 in 2024-04-20T17:52:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:36 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2042121374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:52:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:36 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:37.858 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:52:37.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:37.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:52:37.967 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:52:38.015 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:37 smithi005 ceph-mon[39562]: 9.1 scrub starts 2024-04-20T17:52:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:37 smithi005 ceph-mon[39562]: from='client.28000 v1:172.21.15.5:0/619482710' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:52:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:37 smithi005 ceph-mon[39562]: 1.6 scrub starts 2024-04-20T17:52:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:37 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:37 smithi005 ceph-mon[39562]: 1.6 scrub ok 2024-04-20T17:52:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:37 smithi005 ceph-mon[39562]: pgmap v2551: 27 pgs: 27 active+clean; 2.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s 2024-04-20T17:52:38.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:37 smithi005 ceph-mon[39562]: 9.1 scrub ok 2024-04-20T17:52:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:37 smithi106 ceph-mon[44423]: 9.1 scrub starts 2024-04-20T17:52:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:37 smithi106 ceph-mon[44423]: from='client.28000 v1:172.21.15.5:0/619482710' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:52:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:37 smithi106 ceph-mon[44423]: 1.6 scrub starts 2024-04-20T17:52:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:37 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:37 smithi106 ceph-mon[44423]: 1.6 scrub ok 2024-04-20T17:52:38.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:37 smithi106 ceph-mon[44423]: pgmap v2551: 27 pgs: 27 active+clean; 2.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s 2024-04-20T17:52:38.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:37 smithi106 ceph-mon[44423]: 9.1 scrub ok 2024-04-20T17:52:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:37 smithi171 ceph-mon[44189]: 9.1 scrub starts 2024-04-20T17:52:38.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:37 smithi171 ceph-mon[44189]: from='client.28000 v1:172.21.15.5:0/619482710' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:52:38.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:37 smithi171 ceph-mon[44189]: 1.6 scrub starts 2024-04-20T17:52:38.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:37 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:38.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:37 smithi171 ceph-mon[44189]: 1.6 scrub ok 2024-04-20T17:52:38.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:37 smithi171 ceph-mon[44189]: pgmap v2551: 27 pgs: 27 active+clean; 2.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s 2024-04-20T17:52:38.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:37 smithi171 ceph-mon[44189]: 9.1 scrub ok 2024-04-20T17:52:40.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:39 smithi005 ceph-mon[39562]: 1.3 scrub starts 2024-04-20T17:52:40.214 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:39 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:52:40.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:39 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:52:40.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:39 smithi005 ceph-mon[39562]: 1.3 scrub ok 2024-04-20T17:52:40.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:39 smithi005 ceph-mon[39562]: pgmap v2552: 27 pgs: 27 active+clean; 2.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 943 op/s 2024-04-20T17:52:40.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:39 smithi005 ceph-mon[39562]: osdmap e967: 12 total, 12 up, 12 in 2024-04-20T17:52:40.243 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:52:40.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:40.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:52:40.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:39 smithi106 ceph-mon[44423]: 1.3 scrub starts 2024-04-20T17:52:40.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:39 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:52:40.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:39 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:52:40.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:39 smithi106 ceph-mon[44423]: 1.3 scrub ok 2024-04-20T17:52:40.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:39 smithi106 ceph-mon[44423]: pgmap v2552: 27 pgs: 27 active+clean; 2.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 943 op/s 2024-04-20T17:52:40.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:39 smithi106 ceph-mon[44423]: osdmap e967: 12 total, 12 up, 12 in 2024-04-20T17:52:40.295 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":2550,"stamp":"2024-04-20T17:52:35.898515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2779644472,"num_objects":42408,"num_object_clones":0,"num_object_copies":84816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42408,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":291182,"num_write_kb":10674753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12720,"num_bytes_recovered":833486848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162262,"ondisk_log_size":162262,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":12632804,"kb_used_data":5659764,"kb_used_omap":404,"kb_used_meta":6972523,"kb_avail":1112309020,"statfs":{"total":1151940427776,"available":1139004436480,"internally_reserved":0,"allocated":5795598336,"data_stored":5790849096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":414014,"internal_metadata":7139864258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,1,9,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-680001536,"num_objects":-10376,"num_object_clones":0,"num_object_copies":-20752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10376,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.173435"},"pg_stats":[{"pgid":"9.7","version":"965'18122","reported_seq":18455,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-20T17:52:35.289398+0000","last_change":"2024-04-20T17:52:35.055360+0000","last_active":"2024-04-20T17:52:35.289398+0000","last_peered":"2024-04-20T17:52:35.289398+0000","last_clean":"2024-04-20T17:52:35.289398+0000","last_became_active":"2024-04-20T17:48:54.814508+0000","last_became_peered":"2024-04-20T17:48:54.814508+0000","last_unstale":"2024-04-20T17:52:35.289398+0000","last_undegraded":"2024-04-20T17:52:35.289398+0000","last_fullsized":"2024-04-20T17:52:35.289398+0000","mapping_epoch":894,"log_start":"924'8100","ondisk_log_start":"924'8100","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"965'18108","last_scrub_stamp":"2024-04-20T17:52:35.055125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:52:35.055125+0000","objects_scrubbed":2572,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:54:01.516960+0000","scrub_duration":447,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166461440,"num_objects":2540,"num_object_clones":0,"num_object_copies":5080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18122,"num_write_kb":661184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"963'17992","reported_seq":28137,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-04-20T17:52:31.479541+0000","last_change":"2024-04-20T17:52:11.009257+0000","last_active":"2024-04-20T17:52:31.479541+0000","last_peered":"2024-04-20T17:52:31.479541+0000","last_clean":"2024-04-20T17:52:31.479541+0000","last_became_active":"2024-04-20T17:52:10.932624+0000","last_became_peered":"2024-04-20T17:52:10.932624+0000","last_unstale":"2024-04-20T17:52:31.479541+0000","last_undegraded":"2024-04-20T17:52:31.479541+0000","last_fullsized":"2024-04-20T17:52:31.479541+0000","mapping_epoch":959,"log_start":"922'7900","ondisk_log_start":"922'7900","created":894,"last_epoch_clean":960,"parent":"0.0","parent_split_bits":0,"last_scrub":"929'9608","last_scrub_stamp":"2024-04-20T17:50:25.096273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:25.096273+0000","objects_scrubbed":9524,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:33.789412+0000","scrub_duration":2229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182714368,"num_objects":2788,"num_object_clones":0,"num_object_copies":5576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17992,"num_write_kb":664960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4942,"num_bytes_recovered":323878912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.5","version":"963'17954","reported_seq":18275,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-04-20T17:52:31.153662+0000","last_change":"2024-04-20T17:51:42.519313+0000","last_active":"2024-04-20T17:52:31.153662+0000","last_peered":"2024-04-20T17:52:31.153662+0000","last_clean":"2024-04-20T17:52:31.153662+0000","last_became_active":"2024-04-20T17:48:54.817749+0000","last_became_peered":"2024-04-20T17:48:54.817749+0000","last_unstale":"2024-04-20T17:52:31.153662+0000","last_undegraded":"2024-04-20T17:52:31.153662+0000","last_fullsized":"2024-04-20T17:52:31.153662+0000","mapping_epoch":894,"log_start":"922'7900","ondisk_log_start":"922'7900","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'14734","last_scrub_stamp":"2024-04-20T17:51:42.519071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:42.519071+0000","objects_scrubbed":6056,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:09.147760+0000","scrub_duration":1274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182845440,"num_objects":2790,"num_object_clones":0,"num_object_copies":5580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17954,"num_write_kb":663808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"963'17876","reported_seq":18190,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-04-20T17:52:30.914649+0000","last_change":"2024-04-20T17:51:48.836352+0000","last_active":"2024-04-20T17:52:30.914649+0000","last_peered":"2024-04-20T17:52:30.914649+0000","last_clean":"2024-04-20T17:52:30.914649+0000","last_became_active":"2024-04-20T17:48:55.061199+0000","last_became_peered":"2024-04-20T17:48:55.061199+0000","last_unstale":"2024-04-20T17:52:30.914649+0000","last_undegraded":"2024-04-20T17:52:30.914649+0000","last_fullsized":"2024-04-20T17:52:30.914649+0000","mapping_epoch":894,"log_start":"922'7800","ondisk_log_start":"922'7800","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"954'15006","last_scrub_stamp":"2024-04-20T17:51:48.836218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:48.836218+0000","objects_scrubbed":5762,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:51.196876+0000","scrub_duration":1412,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185860096,"num_objects":2836,"num_object_clones":0,"num_object_copies":5672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17876,"num_write_kb":662784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"965'18008","reported_seq":33901,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-20T17:52:34.003556+0000","last_change":"2024-04-20T17:51:42.767947+0000","last_active":"2024-04-20T17:52:34.003556+0000","last_peered":"2024-04-20T17:52:34.003556+0000","last_clean":"2024-04-20T17:52:34.003556+0000","last_became_active":"2024-04-20T17:51:40.523417+0000","last_became_peered":"2024-04-20T17:51:40.523417+0000","last_unstale":"2024-04-20T17:52:34.003556+0000","last_undegraded":"2024-04-20T17:52:34.003556+0000","last_fullsized":"2024-04-20T17:52:34.003556+0000","mapping_epoch":950,"log_start":"923'8000","ondisk_log_start":"923'8000","created":894,"last_epoch_clean":951,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'14655","last_scrub_stamp":"2024-04-20T17:51:42.767651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:42.767651+0000","objects_scrubbed":5939,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:50.590021+0000","scrub_duration":1039,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168296448,"num_objects":2568,"num_object_clones":0,"num_object_copies":5136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18008,"num_write_kb":658432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7778,"num_bytes_recovered":509607936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.e","version":"965'18412","reported_seq":18660,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-20T17:52:34.836568+0000","last_change":"2024-04-20T17:51:03.098201+0000","last_active":"2024-04-20T17:52:34.836568+0000","last_peered":"2024-04-20T17:52:34.836568+0000","last_clean":"2024-04-20T17:52:34.836568+0000","last_became_active":"2024-04-20T17:48:54.815159+0000","last_became_peered":"2024-04-20T17:48:54.815159+0000","last_unstale":"2024-04-20T17:52:34.836568+0000","last_undegraded":"2024-04-20T17:52:34.836568+0000","last_fullsized":"2024-04-20T17:52:34.836568+0000","mapping_epoch":894,"log_start":"925'8400","ondisk_log_start":"925'8400","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"942'12102","last_scrub_stamp":"2024-04-20T17:51:03.098072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:03.098072+0000","objects_scrubbed":8857,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:36.160614+0000","scrub_duration":2003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161218560,"num_objects":2460,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18412,"num_write_kb":667904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"15'137","reported_seq":982,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-20T17:52:33.876214+0000","last_change":"2024-04-20T17:50:57.911454+0000","last_active":"2024-04-20T17:52:33.876214+0000","last_peered":"2024-04-20T17:52:33.876214+0000","last_clean":"2024-04-20T17:52:33.876214+0000","last_became_active":"2024-04-20T17:45:32.670941+0000","last_became_peered":"2024-04-20T17:45:32.670941+0000","last_unstale":"2024-04-20T17:52:33.876214+0000","last_undegraded":"2024-04-20T17:52:33.876214+0000","last_fullsized":"2024-04-20T17:52:33.876214+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:57.911336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:57.911336+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"965'18248","reported_seq":18571,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-20T17:52:35.670113+0000","last_change":"2024-04-20T17:51:43.065593+0000","last_active":"2024-04-20T17:52:35.670113+0000","last_peered":"2024-04-20T17:52:35.670113+0000","last_clean":"2024-04-20T17:52:35.670113+0000","last_became_active":"2024-04-20T17:48:55.061161+0000","last_became_peered":"2024-04-20T17:48:55.061161+0000","last_unstale":"2024-04-20T17:52:35.670113+0000","last_undegraded":"2024-04-20T17:52:35.670113+0000","last_fullsized":"2024-04-20T17:52:35.670113+0000","mapping_epoch":894,"log_start":"925'8200","ondisk_log_start":"925'8200","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'14822","last_scrub_stamp":"2024-04-20T17:51:43.065456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:43.065456+0000","objects_scrubbed":6081,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:57.084267+0000","scrub_duration":1030,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172228608,"num_objects":2628,"num_object_clones":0,"num_object_copies":5256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18248,"num_write_kb":668032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"15'137","reported_seq":984,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-04-20T17:52:25.827098+0000","last_change":"2024-04-20T17:51:42.855635+0000","last_active":"2024-04-20T17:52:25.827098+0000","last_peered":"2024-04-20T17:52:25.827098+0000","last_clean":"2024-04-20T17:52:25.827098+0000","last_became_active":"2024-04-20T17:45:29.817528+0000","last_became_peered":"2024-04-20T17:45:29.817528+0000","last_unstale":"2024-04-20T17:52:25.827098+0000","last_undegraded":"2024-04-20T17:52:25.827098+0000","last_fullsized":"2024-04-20T17:52:25.827098+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:42.855496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:42.855496+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:49.448590+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"964'18377","reported_seq":18692,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-04-20T17:52:33.736514+0000","last_change":"2024-04-20T17:52:01.435065+0000","last_active":"2024-04-20T17:52:33.736514+0000","last_peered":"2024-04-20T17:52:33.736514+0000","last_clean":"2024-04-20T17:52:33.736514+0000","last_became_active":"2024-04-20T17:48:54.817445+0000","last_became_peered":"2024-04-20T17:48:54.817445+0000","last_unstale":"2024-04-20T17:52:33.736514+0000","last_undegraded":"2024-04-20T17:52:33.736514+0000","last_fullsized":"2024-04-20T17:52:33.736514+0000","mapping_epoch":894,"log_start":"924'8300","ondisk_log_start":"924'8300","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"956'16160","last_scrub_stamp":"2024-04-20T17:52:01.434805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:52:01.434805+0000","objects_scrubbed":4908,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:21.628859+0000","scrub_duration":1125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173080576,"num_objects":2641,"num_object_clones":0,"num_object_copies":5282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18377,"num_write_kb":672576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"15'137","reported_seq":1010,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-04-20T17:52:25.819971+0000","last_change":"2024-04-20T17:51:44.510869+0000","last_active":"2024-04-20T17:52:25.819971+0000","last_peered":"2024-04-20T17:52:25.819971+0000","last_clean":"2024-04-20T17:52:25.819971+0000","last_became_active":"2024-04-20T17:45:29.827577+0000","last_became_peered":"2024-04-20T17:45:29.827577+0000","last_unstale":"2024-04-20T17:52:25.819971+0000","last_undegraded":"2024-04-20T17:52:25.819971+0000","last_fullsized":"2024-04-20T17:52:25.819971+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:44.510742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:44.510742+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:46.934138+0000","scrub_duration":266,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"965'18216","reported_seq":18532,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-20T17:52:34.829985+0000","last_change":"2024-04-20T17:51:00.541301+0000","last_active":"2024-04-20T17:52:34.829985+0000","last_peered":"2024-04-20T17:52:34.829985+0000","last_clean":"2024-04-20T17:52:34.829985+0000","last_became_active":"2024-04-20T17:48:55.061208+0000","last_became_peered":"2024-04-20T17:48:55.061208+0000","last_unstale":"2024-04-20T17:52:34.829985+0000","last_undegraded":"2024-04-20T17:52:34.829985+0000","last_fullsized":"2024-04-20T17:52:34.829985+0000","mapping_epoch":894,"log_start":"925'8200","ondisk_log_start":"925'8200","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"941'11888","last_scrub_stamp":"2024-04-20T17:51:00.541192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:00.541192+0000","objects_scrubbed":8973,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:35.117366+0000","scrub_duration":1502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169541656,"num_objects":2588,"num_object_clones":0,"num_object_copies":5176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2588,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18216,"num_write_kb":665665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"15'137","reported_seq":982,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-20T17:52:33.876146+0000","last_change":"2024-04-20T17:51:46.197902+0000","last_active":"2024-04-20T17:52:33.876146+0000","last_peered":"2024-04-20T17:52:33.876146+0000","last_clean":"2024-04-20T17:52:33.876146+0000","last_became_active":"2024-04-20T17:45:32.666523+0000","last_became_peered":"2024-04-20T17:45:32.666523+0000","last_unstale":"2024-04-20T17:52:33.876146+0000","last_undegraded":"2024-04-20T17:52:33.876146+0000","last_fullsized":"2024-04-20T17:52:33.876146+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:46.197797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:46.197797+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:56.371129+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.a","version":"965'18511","reported_seq":18769,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-20T17:52:35.696519+0000","last_change":"2024-04-20T17:50:22.180642+0000","last_active":"2024-04-20T17:52:35.696519+0000","last_peered":"2024-04-20T17:52:35.696519+0000","last_clean":"2024-04-20T17:52:35.696519+0000","last_became_active":"2024-04-20T17:48:55.061180+0000","last_became_peered":"2024-04-20T17:48:55.061180+0000","last_unstale":"2024-04-20T17:52:35.696519+0000","last_undegraded":"2024-04-20T17:52:35.696519+0000","last_fullsized":"2024-04-20T17:52:35.696519+0000","mapping_epoch":894,"log_start":"925'8500","ondisk_log_start":"925'8500","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'9574","last_scrub_stamp":"2024-04-20T17:50:22.180501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:22.180501+0000","objects_scrubbed":9507,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161677312,"num_objects":2467,"num_object_clones":0,"num_object_copies":4934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18511,"num_write_kb":671296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"15'137","reported_seq":983,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-20T17:52:33.876502+0000","last_change":"2024-04-20T17:51:43.176176+0000","last_active":"2024-04-20T17:52:33.876502+0000","last_peered":"2024-04-20T17:52:33.876502+0000","last_clean":"2024-04-20T17:52:33.876502+0000","last_became_active":"2024-04-20T17:46:40.831961+0000","last_became_peered":"2024-04-20T17:46:40.831961+0000","last_unstale":"2024-04-20T17:52:33.876502+0000","last_undegraded":"2024-04-20T17:52:33.876502+0000","last_fullsized":"2024-04-20T17:52:33.876502+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:43.176019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:43.176019+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:57.843728+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"965'18098","reported_seq":18413,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-20T17:52:34.038560+0000","last_change":"2024-04-20T17:51:46.152560+0000","last_active":"2024-04-20T17:52:34.038560+0000","last_peered":"2024-04-20T17:52:34.038560+0000","last_clean":"2024-04-20T17:52:34.038560+0000","last_became_active":"2024-04-20T17:48:54.816644+0000","last_became_peered":"2024-04-20T17:48:54.816644+0000","last_unstale":"2024-04-20T17:52:34.038560+0000","last_undegraded":"2024-04-20T17:52:34.038560+0000","last_fullsized":"2024-04-20T17:52:34.038560+0000","mapping_epoch":894,"log_start":"923'8000","ondisk_log_start":"923'8000","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"952'14792","last_scrub_stamp":"2024-04-20T17:51:46.152388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:46.152388+0000","objects_scrubbed":5974,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:04.134118+0000","scrub_duration":519,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174850048,"num_objects":2668,"num_object_clones":0,"num_object_copies":5336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18098,"num_write_kb":664512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"15'137","reported_seq":773,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-04-20T17:52:31.855428+0000","last_change":"2024-04-20T17:51:00.057261+0000","last_active":"2024-04-20T17:52:31.855428+0000","last_peered":"2024-04-20T17:52:31.855428+0000","last_clean":"2024-04-20T17:52:31.855428+0000","last_became_active":"2024-04-20T17:49:46.824559+0000","last_became_peered":"2024-04-20T17:49:46.824559+0000","last_unstale":"2024-04-20T17:52:31.855428+0000","last_undegraded":"2024-04-20T17:52:31.855428+0000","last_fullsized":"2024-04-20T17:52:31.855428+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:00.057156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:00.057156+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"965'18663","reported_seq":18987,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-20T17:52:35.289398+0000","last_change":"2024-04-20T17:51:50.104395+0000","last_active":"2024-04-20T17:52:35.289398+0000","last_peered":"2024-04-20T17:52:35.289398+0000","last_clean":"2024-04-20T17:52:35.289398+0000","last_became_active":"2024-04-20T17:48:54.813199+0000","last_became_peered":"2024-04-20T17:48:54.813199+0000","last_unstale":"2024-04-20T17:52:35.289398+0000","last_undegraded":"2024-04-20T17:52:35.289398+0000","last_fullsized":"2024-04-20T17:52:35.289398+0000","mapping_epoch":894,"log_start":"925'8600","ondisk_log_start":"925'8600","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"954'15465","last_scrub_stamp":"2024-04-20T17:51:50.104218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:50.104218+0000","objects_scrubbed":5848,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:05.037497+0000","scrub_duration":1044,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171638784,"num_objects":2619,"num_object_clones":0,"num_object_copies":5238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18663,"num_write_kb":681024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"15'137","reported_seq":982,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-20T17:52:33.876526+0000","last_change":"2024-04-20T17:50:58.932166+0000","last_active":"2024-04-20T17:52:33.876526+0000","last_peered":"2024-04-20T17:52:33.876526+0000","last_clean":"2024-04-20T17:52:33.876526+0000","last_became_active":"2024-04-20T17:45:32.782706+0000","last_became_peered":"2024-04-20T17:45:32.782706+0000","last_unstale":"2024-04-20T17:52:33.876526+0000","last_undegraded":"2024-04-20T17:52:33.876526+0000","last_fullsized":"2024-04-20T17:52:33.876526+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:50:58.932046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:50:58.932046+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.2","version":"964'18133","reported_seq":18451,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-04-20T17:52:33.745430+0000","last_change":"2024-04-20T17:51:43.530794+0000","last_active":"2024-04-20T17:52:33.745430+0000","last_peered":"2024-04-20T17:52:33.745430+0000","last_clean":"2024-04-20T17:52:33.745430+0000","last_became_active":"2024-04-20T17:48:55.060965+0000","last_became_peered":"2024-04-20T17:48:55.060965+0000","last_unstale":"2024-04-20T17:52:33.745430+0000","last_undegraded":"2024-04-20T17:52:33.745430+0000","last_fullsized":"2024-04-20T17:52:33.745430+0000","mapping_epoch":894,"log_start":"924'8100","ondisk_log_start":"924'8100","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'14849","last_scrub_stamp":"2024-04-20T17:51:43.530564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:43.530564+0000","objects_scrubbed":5973,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:05.542393+0000","scrub_duration":1032,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173473792,"num_objects":2647,"num_object_clones":0,"num_object_copies":5294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18133,"num_write_kb":664960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"15'137","reported_seq":966,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-04-20T17:52:31.859371+0000","last_change":"2024-04-20T17:51:41.848577+0000","last_active":"2024-04-20T17:52:31.859371+0000","last_peered":"2024-04-20T17:52:31.859371+0000","last_clean":"2024-04-20T17:52:31.859371+0000","last_became_active":"2024-04-20T17:45:32.669451+0000","last_became_peered":"2024-04-20T17:45:32.669451+0000","last_unstale":"2024-04-20T17:52:31.859371+0000","last_undegraded":"2024-04-20T17:52:31.859371+0000","last_fullsized":"2024-04-20T17:52:31.859371+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:41.848440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:41.848440+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:55.264457+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"963'17979","reported_seq":18304,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-04-20T17:52:31.739565+0000","last_change":"2024-04-20T17:51:42.714681+0000","last_active":"2024-04-20T17:52:31.739565+0000","last_peered":"2024-04-20T17:52:31.739565+0000","last_clean":"2024-04-20T17:52:31.739565+0000","last_became_active":"2024-04-20T17:48:55.061200+0000","last_became_peered":"2024-04-20T17:48:55.061200+0000","last_unstale":"2024-04-20T17:52:31.739565+0000","last_undegraded":"2024-04-20T17:52:31.739565+0000","last_fullsized":"2024-04-20T17:52:31.739565+0000","mapping_epoch":894,"log_start":"922'7900","ondisk_log_start":"922'7900","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'14728","last_scrub_stamp":"2024-04-20T17:51:42.714540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:42.714540+0000","objects_scrubbed":6058,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:00.487021+0000","scrub_duration":1002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181600256,"num_objects":2771,"num_object_clones":0,"num_object_copies":5542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17979,"num_write_kb":664000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"15'137","reported_seq":979,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-04-20T17:52:25.826887+0000","last_change":"2024-04-20T17:51:40.826215+0000","last_active":"2024-04-20T17:52:25.826887+0000","last_peered":"2024-04-20T17:52:25.826887+0000","last_clean":"2024-04-20T17:52:25.826887+0000","last_became_active":"2024-04-20T17:45:32.644584+0000","last_became_peered":"2024-04-20T17:45:32.644584+0000","last_unstale":"2024-04-20T17:52:25.826887+0000","last_undegraded":"2024-04-20T17:52:25.826887+0000","last_fullsized":"2024-04-20T17:52:25.826887+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:40.826081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:40.826081+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:45.293420+0000","scrub_duration":185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"964'20724","reported_seq":21082,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-04-20T17:52:33.776790+0000","last_change":"2024-04-20T17:52:17.934303+0000","last_active":"2024-04-20T17:52:33.776790+0000","last_peered":"2024-04-20T17:52:33.776790+0000","last_clean":"2024-04-20T17:52:33.776790+0000","last_became_active":"2024-04-20T17:48:54.813062+0000","last_became_peered":"2024-04-20T17:48:54.813062+0000","last_unstale":"2024-04-20T17:52:33.776790+0000","last_undegraded":"2024-04-20T17:52:33.776790+0000","last_fullsized":"2024-04-20T17:52:33.776790+0000","mapping_epoch":894,"log_start":"933'10700","ondisk_log_start":"933'10700","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"930'10026","last_scrub_stamp":"2024-04-20T17:50:30.712648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:30.712648+0000","objects_scrubbed":2373,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1871,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168361984,"num_objects":2569,"num_object_clones":0,"num_object_copies":5138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18351,"num_write_kb":669440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"15'137","reported_seq":929,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-20T17:52:33.876547+0000","last_change":"2024-04-20T17:51:45.119466+0000","last_active":"2024-04-20T17:52:33.876547+0000","last_peered":"2024-04-20T17:52:33.876547+0000","last_clean":"2024-04-20T17:52:33.876547+0000","last_became_active":"2024-04-20T17:46:40.881713+0000","last_became_peered":"2024-04-20T17:46:40.881713+0000","last_unstale":"2024-04-20T17:52:33.876547+0000","last_undegraded":"2024-04-20T17:52:33.876547+0000","last_fullsized":"2024-04-20T17:52:33.876547+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:45.119332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:45.119332+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:51.157610+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.8","version":"964'18242","reported_seq":18563,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-04-20T17:52:32.253687+0000","last_change":"2024-04-20T17:51:43.840908+0000","last_active":"2024-04-20T17:52:32.253687+0000","last_peered":"2024-04-20T17:52:32.253687+0000","last_clean":"2024-04-20T17:52:32.253687+0000","last_became_active":"2024-04-20T17:48:54.813368+0000","last_became_peered":"2024-04-20T17:48:54.813368+0000","last_unstale":"2024-04-20T17:52:32.253687+0000","last_undegraded":"2024-04-20T17:52:32.253687+0000","last_fullsized":"2024-04-20T17:52:32.253687+0000","mapping_epoch":894,"log_start":"924'8200","ondisk_log_start":"924'8200","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'15051","last_scrub_stamp":"2024-04-20T17:51:43.840798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:43.840798+0000","objects_scrubbed":6061,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:52.336868+0000","scrub_duration":1160,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185204736,"num_objects":2826,"num_object_clones":0,"num_object_copies":5652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18242,"num_write_kb":674176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":927,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-04-20T17:52:33.876427+0000","last_change":"2024-04-20T17:51:48.160638+0000","last_active":"2024-04-20T17:52:33.876427+0000","last_peered":"2024-04-20T17:52:33.876427+0000","last_clean":"2024-04-20T17:52:33.876427+0000","last_became_active":"2024-04-20T17:46:40.851683+0000","last_became_peered":"2024-04-20T17:46:40.851683+0000","last_unstale":"2024-04-20T17:52:33.876427+0000","last_undegraded":"2024-04-20T17:52:33.876427+0000","last_fullsized":"2024-04-20T17:52:33.876427+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:48.160546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:48.160546+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:56.139128+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":2779054104,"num_objects":42406,"num_object_clones":0,"num_object_copies":84812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42406,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":291182,"num_write_kb":10674753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12720,"num_bytes_recovered":833486848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5537996800,"data_stored":5537988656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160755,"ondisk_log_size":160755,"up":32,"acting":32,"num_store_stats":11},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},"log_size":1507,"ondisk_log_size":1507,"up":22,"acting":22,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991795,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1500092,"kb_used_data":566516,"kb_used_omap":1,"kb_used_meta":933566,"kb_avail":92245060,"statfs":{"total":95995035648,"available":94458941440,"internally_reserved":0,"allocated":580112384,"data_stored":579722387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":955972036},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253435,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1507092,"kb_used_data":663756,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":92238060,"statfs":{"total":95995035648,"available":94451773440,"internally_reserved":0,"allocated":679686144,"data_stored":679284139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":863566272},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613183,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1120764,"kb_used_data":359348,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":92624388,"statfs":{"total":95995035648,"available":94847373312,"internally_reserved":0,"allocated":367972352,"data_stored":367582355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":779680192},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809414,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":808264,"kb_used_data":339776,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":92936888,"statfs":{"total":95995035648,"available":95167373312,"internally_reserved":0,"allocated":347930624,"data_stored":347534623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":799,"seq":3431678869604,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":671576,"kb_used_data":496720,"kb_used_omap":11,"kb_used_meta":174836,"kb_avail":93073576,"statfs":{"total":95995035648,"available":95307341824,"internally_reserved":0,"allocated":508641280,"data_stored":508235211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11990,"internal_metadata":179032362},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":535,"seq":2297807503643,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":242940,"kb_used_data":178996,"kb_used_omap":170,"kb_used_meta":63765,"kb_avail":93502212,"statfs":{"total":95995035648,"available":95746265088,"internally_reserved":0,"allocated":183291904,"data_stored":182901907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":174628,"internal_metadata":65295836},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":659,"seq":2830383448266,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1422352,"kb_used_data":1022084,"kb_used_omap":4,"kb_used_meta":400251,"kb_avail":92322800,"statfs":{"total":95995035648,"available":94538547200,"internally_reserved":0,"allocated":1046614016,"data_stored":1046213943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4410,"internal_metadata":409857734},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738988,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":744128,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":93001024,"statfs":{"total":95995035648,"available":95233048576,"internally_reserved":0,"allocated":774144,"data_stored":384147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394277,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":689924,"kb_used_data":256956,"kb_used_omap":202,"kb_used_meta":432757,"kb_avail":93055228,"statfs":{"total":95995035648,"available":95288553472,"internally_reserved":0,"allocated":263122944,"data_stored":262729395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":443143639},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653327,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1287944,"kb_used_data":673856,"kb_used_omap":4,"kb_used_meta":614075,"kb_avail":92457208,"statfs":{"total":95995035648,"available":94676180992,"internally_reserved":0,"allocated":690028544,"data_stored":689632543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":628813511},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986775,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":999740,"kb_used_data":368820,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":92745412,"statfs":{"total":95995035648,"available":94971301888,"internally_reserved":0,"allocated":377671680,"data_stored":377281683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790546,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1637988,"kb_used_data":732180,"kb_used_omap":1,"kb_used_meta":905790,"kb_avail":92107164,"statfs":{"total":95995035648,"available":94317735936,"internally_reserved":0,"allocated":749752320,"data_stored":749346763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":927529413},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1029705728,"data_stored":1029701656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":182714368,"data_stored":182714368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":684064768,"data_stored":684064768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":372637696,"data_stored":372637696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":652410880,"data_stored":652410880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":181600256,"data_stored":181600256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":499126272,"data_stored":499122200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":343146496,"data_stored":343146496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":361234432,"data_stored":361234432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":665124864,"data_stored":665124864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":566231040,"data_stored":566231040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:52:40.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:39 smithi171 ceph-mon[44189]: 1.3 scrub starts 2024-04-20T17:52:40.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:39 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:52:40.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:39 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:52:40.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:39 smithi171 ceph-mon[44189]: 1.3 scrub ok 2024-04-20T17:52:40.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:39 smithi171 ceph-mon[44189]: pgmap v2552: 27 pgs: 27 active+clean; 2.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 943 op/s 2024-04-20T17:52:40.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:39 smithi171 ceph-mon[44189]: osdmap e967: 12 total, 12 up, 12 in 2024-04-20T17:52:40.349 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:52:40.402 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:40.428 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:52:40.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:40.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:52:40.534 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:52:40.564 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:40.591 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:52:40.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:40.596 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:52:40.697 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:52:40.734 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:40.996 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:52:41.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:41.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:52:41.101 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:52:41.134 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:41.170 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:52:41.175 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:40 smithi106 ceph-mon[44423]: 9.e scrub starts 2024-04-20T17:52:41.175 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:40 smithi106 ceph-mon[44423]: 9.a scrub starts 2024-04-20T17:52:41.176 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:40 smithi106 ceph-mon[44423]: 9.e scrub ok 2024-04-20T17:52:41.176 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:40 smithi106 ceph-mon[44423]: pgmap v2554: 27 pgs: 27 active+clean; 2.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.32k op/s 2024-04-20T17:52:41.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:41.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:52:41.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:40 smithi005 ceph-mon[39562]: 9.e scrub starts 2024-04-20T17:52:41.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:40 smithi005 ceph-mon[39562]: 9.a scrub starts 2024-04-20T17:52:41.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:40 smithi005 ceph-mon[39562]: 9.e scrub ok 2024-04-20T17:52:41.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:40 smithi005 ceph-mon[39562]: pgmap v2554: 27 pgs: 27 active+clean; 2.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.32k op/s 2024-04-20T17:52:41.277 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:52:41.310 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:41.330 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:52:41.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:40 smithi171 ceph-mon[44189]: 9.e scrub starts 2024-04-20T17:52:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:40 smithi171 ceph-mon[44189]: 9.a scrub starts 2024-04-20T17:52:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:40 smithi171 ceph-mon[44189]: 9.e scrub ok 2024-04-20T17:52:41.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:40 smithi171 ceph-mon[44189]: pgmap v2554: 27 pgs: 27 active+clean; 2.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.32k op/s 2024-04-20T17:52:41.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:41.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:52:41.437 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:52:41.468 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:41.504 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:52:41.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:41.510 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:52:41.611 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:52:41.661 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:41.774 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:52:41.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:41.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:52:41.887 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:52:41.941 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:41.970 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:52:41.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:41.977 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:52:42.046 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:41 smithi005 ceph-mon[39562]: 9.a scrub ok 2024-04-20T17:52:42.046 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:41 smithi005 ceph-mon[39562]: 1.1 scrub starts 2024-04-20T17:52:42.046 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:41 smithi005 ceph-mon[39562]: 1.1 scrub ok 2024-04-20T17:52:42.077 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:52:42.110 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:42.134 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:52:42.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:42.141 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:52:42.242 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:52:42.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:41 smithi106 ceph-mon[44423]: 9.a scrub ok 2024-04-20T17:52:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:41 smithi106 ceph-mon[44423]: 1.1 scrub starts 2024-04-20T17:52:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:41 smithi106 ceph-mon[44423]: 1.1 scrub ok 2024-04-20T17:52:42.277 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:42.302 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:52:42.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:42.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:52:42.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:41 smithi171 ceph-mon[44189]: 9.a scrub ok 2024-04-20T17:52:42.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:41 smithi171 ceph-mon[44189]: 1.1 scrub starts 2024-04-20T17:52:42.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:41 smithi171 ceph-mon[44189]: 1.1 scrub ok 2024-04-20T17:52:42.408 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:52:42.442 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:42.469 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:52:42.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:42.473 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:52:42.574 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:52:42.622 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:42.632 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:52:42.649 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:52:42.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:42.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:52:42.760 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:52:42.761 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:52:42.762 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:52:42.794 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:42.817 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:52:42.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:42.821 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:52:42.922 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:52:42.939 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:42 smithi005 ceph-mon[39562]: pgmap v2555: 27 pgs: 27 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s 2024-04-20T17:52:42.968 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:42.995 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:52:43.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:43.002 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:52:43.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:42 smithi171 ceph-mon[44189]: pgmap v2555: 27 pgs: 27 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s 2024-04-20T17:52:43.103 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:52:43.137 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:43.178 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:52:43.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:43.184 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:52:43.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:42 smithi106 ceph-mon[44423]: pgmap v2555: 27 pgs: 27 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s 2024-04-20T17:52:43.284 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:52:43.298 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:52:43.330 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:43.498 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:52:43.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:43.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:52:43.547 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-20T17:52:43.564 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:52:43.602 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:52:43.633 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:43.658 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:52:43.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:43.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:52:43.765 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:52:43.803 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:43.838 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:52:43.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:43.847 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:52:43.947 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:52:43.950 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:43 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2479975501' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:52:43.950 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:43 smithi005 ceph-mon[39562]: from='client.28376 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:52:43.955 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:52:43.981 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:44.008 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:52:44.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:44.015 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:52:44.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:43 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2479975501' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:52:44.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:43 smithi171 ceph-mon[44189]: from='client.28376 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:52:44.116 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:52:44.147 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:44.182 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:52:44.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:44.190 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:52:44.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:43 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2479975501' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:52:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:43 smithi106 ceph-mon[44423]: from='client.28376 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:52:44.291 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:52:44.325 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:44.351 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:52:44.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:44.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:52:44.458 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:52:44.505 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:44.531 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:52:44.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:44.536 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:52:44.636 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:52:44.689 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:44.948 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:44 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:52:44.949 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:44 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:52:44.949 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:44 smithi005 ceph-mon[39562]: from='client.28376 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:52:44.949 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:44 smithi005 ceph-mon[39562]: osdmap e968: 12 total, 12 up, 12 in 2024-04-20T17:52:44.949 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.108973345427756]}]: dispatch 2024-04-20T17:52:44.949 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08277027755001924]}]: dispatch 2024-04-20T17:52:44.949 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09066146626028872]}]: dispatch 2024-04-20T17:52:44.949 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08394217567953967]}]: dispatch 2024-04-20T17:52:44.949 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09275365394321845]}]: dispatch 2024-04-20T17:52:44.949 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08379201078565857]}]: dispatch 2024-04-20T17:52:44.949 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08524499735486163]}]: dispatch 2024-04-20T17:52:44.949 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08284500694470145]}]: dispatch 2024-04-20T17:52:44.949 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08861140948018768]}]: dispatch 2024-04-20T17:52:44.950 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08272543991320991]}]: dispatch 2024-04-20T17:52:44.950 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08143459259365003]}]: dispatch 2024-04-20T17:52:44.950 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:44 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08379201078565857]}]: dispatch 2024-04-20T17:52:44.950 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:44 smithi005 ceph-mon[39562]: pgmap v2557: 27 pgs: 27 active+clean; 2.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-04-20T17:52:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:44 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:52:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:44 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:52:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:44 smithi106 ceph-mon[44423]: from='client.28376 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:52:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:44 smithi106 ceph-mon[44423]: osdmap e968: 12 total, 12 up, 12 in 2024-04-20T17:52:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.108973345427756]}]: dispatch 2024-04-20T17:52:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08277027755001924]}]: dispatch 2024-04-20T17:52:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09066146626028872]}]: dispatch 2024-04-20T17:52:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08394217567953967]}]: dispatch 2024-04-20T17:52:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09275365394321845]}]: dispatch 2024-04-20T17:52:45.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08379201078565857]}]: dispatch 2024-04-20T17:52:45.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08524499735486163]}]: dispatch 2024-04-20T17:52:45.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08284500694470145]}]: dispatch 2024-04-20T17:52:45.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08861140948018768]}]: dispatch 2024-04-20T17:52:45.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08272543991320991]}]: dispatch 2024-04-20T17:52:45.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08143459259365003]}]: dispatch 2024-04-20T17:52:45.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:44 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08379201078565857]}]: dispatch 2024-04-20T17:52:45.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:44 smithi106 ceph-mon[44423]: pgmap v2557: 27 pgs: 27 active+clean; 2.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-04-20T17:52:45.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:44 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:52:45.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:44 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:52:45.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:44 smithi171 ceph-mon[44189]: from='client.28376 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:52:45.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:44 smithi171 ceph-mon[44189]: osdmap e968: 12 total, 12 up, 12 in 2024-04-20T17:52:45.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.108973345427756]}]: dispatch 2024-04-20T17:52:45.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08277027755001924]}]: dispatch 2024-04-20T17:52:45.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09066146626028872]}]: dispatch 2024-04-20T17:52:45.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08394217567953967]}]: dispatch 2024-04-20T17:52:45.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09275365394321845]}]: dispatch 2024-04-20T17:52:45.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08379201078565857]}]: dispatch 2024-04-20T17:52:45.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08524499735486163]}]: dispatch 2024-04-20T17:52:45.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08284500694470145]}]: dispatch 2024-04-20T17:52:45.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08861140948018768]}]: dispatch 2024-04-20T17:52:45.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08272543991320991]}]: dispatch 2024-04-20T17:52:45.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08143459259365003]}]: dispatch 2024-04-20T17:52:45.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:44 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08379201078565857]}]: dispatch 2024-04-20T17:52:45.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:44 smithi171 ceph-mon[44189]: pgmap v2557: 27 pgs: 27 active+clean; 2.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-04-20T17:52:45.453 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:52:45.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:45.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:52:45.567 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:52:45.609 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:45.869 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:52:45.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:45.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:52:45.979 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:52:46.018 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:46.168 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.108973345427756]}]': finished 2024-04-20T17:52:46.168 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08277027755001924]}]': finished 2024-04-20T17:52:46.168 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09066146626028872]}]': finished 2024-04-20T17:52:46.168 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08394217567953967]}]': finished 2024-04-20T17:52:46.168 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09275365394321845]}]': finished 2024-04-20T17:52:46.168 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08379201078565857]}]': finished 2024-04-20T17:52:46.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08524499735486163]}]': finished 2024-04-20T17:52:46.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08284500694470145]}]': finished 2024-04-20T17:52:46.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08861140948018768]}]': finished 2024-04-20T17:52:46.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08272543991320991]}]': finished 2024-04-20T17:52:46.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08143459259365003]}]': finished 2024-04-20T17:52:46.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:45 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08379201078565857]}]': finished 2024-04-20T17:52:46.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:45 smithi005 ceph-mon[39562]: osdmap e969: 12 total, 12 up, 12 in 2024-04-20T17:52:46.175 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:52:46.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:46.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:52:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.108973345427756]}]': finished 2024-04-20T17:52:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08277027755001924]}]': finished 2024-04-20T17:52:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09066146626028872]}]': finished 2024-04-20T17:52:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08394217567953967]}]': finished 2024-04-20T17:52:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09275365394321845]}]': finished 2024-04-20T17:52:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08379201078565857]}]': finished 2024-04-20T17:52:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08524499735486163]}]': finished 2024-04-20T17:52:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08284500694470145]}]': finished 2024-04-20T17:52:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08861140948018768]}]': finished 2024-04-20T17:52:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08272543991320991]}]': finished 2024-04-20T17:52:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08143459259365003]}]': finished 2024-04-20T17:52:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:45 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08379201078565857]}]': finished 2024-04-20T17:52:46.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:45 smithi106 ceph-mon[44423]: osdmap e969: 12 total, 12 up, 12 in 2024-04-20T17:52:46.262 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:52:46.275 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:52:46.289 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:52:46.315 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.0", "weight": [0.108973345427756]}]': finished 2024-04-20T17:52:46.315 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.1", "weight": [0.08277027755001924]}]': finished 2024-04-20T17:52:46.315 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.2", "weight": [0.09066146626028872]}]': finished 2024-04-20T17:52:46.315 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.3", "weight": [0.08394217567953967]}]': finished 2024-04-20T17:52:46.316 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.4", "weight": [0.09275365394321845]}]': finished 2024-04-20T17:52:46.316 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.5", "weight": [0.08379201078565857]}]': finished 2024-04-20T17:52:46.316 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.6", "weight": [0.08524499735486163]}]': finished 2024-04-20T17:52:46.316 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.7", "weight": [0.08284500694470145]}]': finished 2024-04-20T17:52:46.316 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.8", "weight": [0.08861140948018768]}]': finished 2024-04-20T17:52:46.316 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.9", "weight": [0.08272543991320991]}]': finished 2024-04-20T17:52:46.316 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.10", "weight": [0.08143459259365003]}]': finished 2024-04-20T17:52:46.316 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:45 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set reweight-compat", "format": "json", "item": "osd.11", "weight": [0.08379201078565857]}]': finished 2024-04-20T17:52:46.316 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:45 smithi171 ceph-mon[44189]: osdmap e969: 12 total, 12 up, 12 in 2024-04-20T17:52:46.317 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:46.342 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:52:46.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:46.346 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:52:46.447 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:52:46.480 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:46.504 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:52:46.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:46.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:52:46.608 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:52:46.641 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:46.666 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:52:46.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:46.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:52:46.774 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:52:46.804 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:46.827 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:52:46.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:46.833 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:52:46.934 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:52:46.983 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:47.010 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:52:47.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:47.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:52:47.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:46 smithi171 ceph-mon[44189]: 9.6 deep-scrub starts 2024-04-20T17:52:47.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:46 smithi171 ceph-mon[44189]: osdmap e970: 12 total, 12 up, 12 in 2024-04-20T17:52:47.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:46 smithi171 ceph-mon[44189]: pgmap v2560: 27 pgs: 1 active+clean+scrubbing+deep, 1 peering, 25 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s 2024-04-20T17:52:47.100 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:46 smithi005 ceph-mon[39562]: 9.6 deep-scrub starts 2024-04-20T17:52:47.100 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:46 smithi005 ceph-mon[39562]: osdmap e970: 12 total, 12 up, 12 in 2024-04-20T17:52:47.100 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:46 smithi005 ceph-mon[39562]: pgmap v2560: 27 pgs: 1 active+clean+scrubbing+deep, 1 peering, 25 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s 2024-04-20T17:52:47.116 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:52:47.143 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:47.170 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:52:47.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:47.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:52:47.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:46 smithi106 ceph-mon[44423]: 9.6 deep-scrub starts 2024-04-20T17:52:47.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:46 smithi106 ceph-mon[44423]: osdmap e970: 12 total, 12 up, 12 in 2024-04-20T17:52:47.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:46 smithi106 ceph-mon[44423]: pgmap v2560: 27 pgs: 1 active+clean+scrubbing+deep, 1 peering, 25 active+clean; 1.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s 2024-04-20T17:52:47.280 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:52:47.326 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:47.461 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:52:47.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:47.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:52:47.494 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-20T17:52:47.568 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:52:47.603 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:47.632 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:52:47.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:47.636 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:52:47.737 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:52:47.764 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:47.789 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:52:47.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:47.796 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:52:47.897 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:52:47.928 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:47.952 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:52:47.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:47.959 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:52:48.060 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:52:48.096 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:48.119 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:52:48.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:48.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:52:48.223 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:52:48.256 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:47 smithi005 ceph-mon[39562]: 1.8 scrub starts 2024-04-20T17:52:48.256 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:47 smithi005 ceph-mon[39562]: 1.8 scrub ok 2024-04-20T17:52:48.259 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:48.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:47 smithi106 ceph-mon[44423]: 1.8 scrub starts 2024-04-20T17:52:48.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:47 smithi106 ceph-mon[44423]: 1.8 scrub ok 2024-04-20T17:52:48.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:47 smithi171 ceph-mon[44189]: 1.8 scrub starts 2024-04-20T17:52:48.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:47 smithi171 ceph-mon[44189]: 1.8 scrub ok 2024-04-20T17:52:49.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:48 smithi106 ceph-mon[44423]: pgmap v2561: 27 pgs: 1 active+clean+scrubbing+deep, 1 peering, 25 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1k op/s 2024-04-20T17:52:49.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:48 smithi171 ceph-mon[44189]: pgmap v2561: 27 pgs: 1 active+clean+scrubbing+deep, 1 peering, 25 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1k op/s 2024-04-20T17:52:49.445 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:52:49.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:49.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:48 smithi005 ceph-mon[39562]: pgmap v2561: 27 pgs: 1 active+clean+scrubbing+deep, 1 peering, 25 active+clean; 1.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1k op/s 2024-04-20T17:52:49.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:52:49.556 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:52:49.587 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:49.619 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:52:49.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:49.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:52:49.723 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:52:49.751 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:49.789 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:52:49.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:49.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:52:49.896 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:52:49.929 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:49.962 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:52:49.963 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:52:49.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:49.966 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:52:49.990 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:52:50.022 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:52:50.057 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:52:50.067 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:52:50.089 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:52:50.110 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:50.121 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:52:50.150 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:52:50.186 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:52:50.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:49 smithi005 ceph-mon[39562]: 1.4 scrub starts 2024-04-20T17:52:50.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:49 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1441723447' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:52:50.212 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:52:50.246 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:52:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:49 smithi106 ceph-mon[44423]: 1.4 scrub starts 2024-04-20T17:52:50.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:49 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1441723447' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:52:50.276 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:52:50.313 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:52:50.316 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:52:50.316 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:52:50.316 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:52:50.316 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:52:50.316 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:52:50.316 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:52:50.316 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:52:50.316 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:52:50.317 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:52:50.317 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:52:50.317 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:52:50.317 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:52:50.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:49 smithi171 ceph-mon[44189]: 1.4 scrub starts 2024-04-20T17:52:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:49 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1441723447' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:52:51.041 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:52:51.045 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:52:51.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:50 smithi106 ceph-mon[44423]: 1.4 scrub ok 2024-04-20T17:52:51.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:50 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:52:51.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:50 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:52:51.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:50 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:50 smithi106 ceph-mon[44423]: pgmap v2562: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 1.5 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s 2024-04-20T17:52:51.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:50 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:52:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:51 smithi171 ceph-mon[44189]: 1.4 scrub ok 2024-04-20T17:52:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:51 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:52:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:51 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:52:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:51 smithi171 ceph-mon[44189]: pgmap v2562: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 1.5 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s 2024-04-20T17:52:51.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:51 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:52:51.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:50 smithi005 ceph-mon[39562]: 1.4 scrub ok 2024-04-20T17:52:51.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:50 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:52:51.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:50 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:52:51.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:50 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:51.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:50 smithi005 ceph-mon[39562]: pgmap v2562: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 1.5 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s 2024-04-20T17:52:51.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:50 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:52:52.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:51 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:52.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:51 smithi005 ceph-mon[39562]: 9.3 scrub starts 2024-04-20T17:52:52.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:51 smithi005 ceph-mon[39562]: osdmap e971: 12 total, 12 up, 12 in 2024-04-20T17:52:52.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:51 smithi005 ceph-mon[39562]: from='client.18135 v1:172.21.15.5:0/1585902397' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:52:52.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:51 smithi005 ceph-mon[39562]: 9.3 scrub ok 2024-04-20T17:52:52.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:51 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:52:52.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:51 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:52:52.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:51 smithi005 ceph-mon[39562]: 9.4 scrub starts 2024-04-20T17:52:52.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:51 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:52:52.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:51 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:52:52.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:51 smithi005 ceph-mon[39562]: 1.7 scrub starts 2024-04-20T17:52:52.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:51 smithi005 ceph-mon[39562]: 1.7 scrub ok 2024-04-20T17:52:52.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:51 smithi005 ceph-mon[39562]: 9.4 scrub ok 2024-04-20T17:52:52.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:52 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:52.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:52 smithi106 ceph-mon[44423]: 9.3 scrub starts 2024-04-20T17:52:52.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:52 smithi106 ceph-mon[44423]: osdmap e971: 12 total, 12 up, 12 in 2024-04-20T17:52:52.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:52 smithi106 ceph-mon[44423]: from='client.18135 v1:172.21.15.5:0/1585902397' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:52:52.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:52 smithi106 ceph-mon[44423]: 9.3 scrub ok 2024-04-20T17:52:52.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:52 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:52:52.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:52 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:52:52.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:52 smithi106 ceph-mon[44423]: 9.4 scrub starts 2024-04-20T17:52:52.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:52 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:52:52.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:52 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:52:52.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:52 smithi106 ceph-mon[44423]: 1.7 scrub starts 2024-04-20T17:52:52.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:52 smithi106 ceph-mon[44423]: 1.7 scrub ok 2024-04-20T17:52:52.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:52 smithi106 ceph-mon[44423]: 9.4 scrub ok 2024-04-20T17:52:52.282 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:52:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:51 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:51 smithi171 ceph-mon[44189]: 9.3 scrub starts 2024-04-20T17:52:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:51 smithi171 ceph-mon[44189]: osdmap e971: 12 total, 12 up, 12 in 2024-04-20T17:52:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:51 smithi171 ceph-mon[44189]: from='client.18135 v1:172.21.15.5:0/1585902397' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:52:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:51 smithi171 ceph-mon[44189]: 9.3 scrub ok 2024-04-20T17:52:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:51 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:52:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:51 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:52:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:52 smithi171 ceph-mon[44189]: 9.4 scrub starts 2024-04-20T17:52:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:52 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:52:52.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:52 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:52:52.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:52 smithi171 ceph-mon[44189]: 1.7 scrub starts 2024-04-20T17:52:52.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:52 smithi171 ceph-mon[44189]: 1.7 scrub ok 2024-04-20T17:52:52.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:52 smithi171 ceph-mon[44189]: 9.4 scrub ok 2024-04-20T17:52:52.760 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:52:52.760 INFO:teuthology.orchestra.run.smithi005.stdout: "size": 20, 2024-04-20T17:52:52.760 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 600, 2024-04-20T17:52:52.760 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [ 2024-04-20T17:52:52.760 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.760 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:141776 8.1 e807/806 8:874c7de1:::benchmark_data_smithi171_175757_object141775:head v 807'8838, mlcod=782'5)", 2024-04-20T17:52:52.760 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:53.803994+0000", 2024-04-20T17:52:52.760 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 478.95072863799999, 2024-04-20T17:52:52.760 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.059946456000000002, 2024-04-20T17:52:52.760 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:52:52.760 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:52:52.761 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:52:52.761 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:52:52.761 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.761 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:52:52.761 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.803994+0000", 2024-04-20T17:52:52.762 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.762 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.762 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.762 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:52:52.762 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.803994+0000", 2024-04-20T17:52:52.763 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.763 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.763 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.763 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:52:52.763 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.803995+0000", 2024-04-20T17:52:52.763 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0210000000000001e-06 2024-04-20T17:52:52.763 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.764 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.764 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:52:52.764 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.804064+0000", 2024-04-20T17:52:52.764 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8670000000000005e-05 2024-04-20T17:52:52.764 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.764 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.764 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:52:52.764 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.804073+0000", 2024-04-20T17:52:52.765 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.6889999999999999e-06 2024-04-20T17:52:52.765 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.765 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.766 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:52:52.766 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.804079+0000", 2024-04-20T17:52:52.766 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6509999999999998e-06 2024-04-20T17:52:52.766 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.766 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.766 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:52:52.766 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.806919+0000", 2024-04-20T17:52:52.766 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002840174 2024-04-20T17:52:52.766 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.766 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.766 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:52:52.767 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.806949+0000", 2024-04-20T17:52:52.767 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9611000000000002e-05 2024-04-20T17:52:52.767 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.767 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.767 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:52:52.767 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.863898+0000", 2024-04-20T17:52:52.767 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.056949234000000001 2024-04-20T17:52:52.767 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.767 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.767 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:52:52.767 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.863941+0000", 2024-04-20T17:52:52.767 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.2406e-05 2024-04-20T17:52:52.767 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.768 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:52:52.768 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.768 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.768 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.768 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:141779 8.1 e807/806 8:88362553:::benchmark_data_smithi171_175757_object141778:head v 807'8839, mlcod=782'5)", 2024-04-20T17:52:52.768 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:53.808292+0000", 2024-04-20T17:52:52.768 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 478.94643030600002, 2024-04-20T17:52:52.768 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.072673096000000006, 2024-04-20T17:52:52.768 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:52:52.768 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:52:52.768 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:52:52.768 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:52:52.768 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.769 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:52:52.769 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.808292+0000", 2024-04-20T17:52:52.769 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.769 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.769 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.769 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:52:52.769 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.808292+0000", 2024-04-20T17:52:52.769 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.769 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.769 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.769 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:52:52.770 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.808294+0000", 2024-04-20T17:52:52.770 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.192e-06 2024-04-20T17:52:52.770 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.770 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.770 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:52:52.770 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.808343+0000", 2024-04-20T17:52:52.770 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9437000000000001e-05 2024-04-20T17:52:52.770 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.770 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.770 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:52:52.771 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.808353+0000", 2024-04-20T17:52:52.771 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0018e-05 2024-04-20T17:52:52.771 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.771 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.771 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:52:52.771 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.808359+0000", 2024-04-20T17:52:52.771 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8719999999999999e-06 2024-04-20T17:52:52.771 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.771 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.771 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:52:52.771 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.816072+0000", 2024-04-20T17:52:52.771 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0077126929999999996 2024-04-20T17:52:52.771 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.771 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.772 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:52:52.772 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.816101+0000", 2024-04-20T17:52:52.772 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9819e-05 2024-04-20T17:52:52.772 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.772 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.772 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:52:52.772 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.880926+0000", 2024-04-20T17:52:52.772 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.064824344000000006 2024-04-20T17:52:52.772 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.772 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.772 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:52:52.772 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.880965+0000", 2024-04-20T17:52:52.772 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9721000000000001e-05 2024-04-20T17:52:52.772 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.772 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:52:52.773 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.773 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.773 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.773 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:141780 8.1 e807/806 8:853ffce1:::benchmark_data_smithi171_175757_object141779:head v 807'8840, mlcod=782'5)", 2024-04-20T17:52:52.773 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:53.812805+0000", 2024-04-20T17:52:52.773 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 478.94191811799999, 2024-04-20T17:52:52.773 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.068191458999999996, 2024-04-20T17:52:52.773 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:52:52.773 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:52:52.773 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:52:52.773 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:52:52.773 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.773 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:52:52.773 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.812805+0000", 2024-04-20T17:52:52.774 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.774 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.774 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.774 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:52:52.774 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.812805+0000", 2024-04-20T17:52:52.774 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.774 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.774 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.774 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:52:52.774 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.812806+0000", 2024-04-20T17:52:52.774 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.781e-06 2024-04-20T17:52:52.774 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.774 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.775 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:52:52.775 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.812836+0000", 2024-04-20T17:52:52.775 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9433000000000001e-05 2024-04-20T17:52:52.775 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.775 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.775 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:52:52.775 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.812847+0000", 2024-04-20T17:52:52.775 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0943e-05 2024-04-20T17:52:52.775 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.775 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.775 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:52:52.775 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.812854+0000", 2024-04-20T17:52:52.775 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.8310000000000003e-06 2024-04-20T17:52:52.775 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.775 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.776 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:52:52.776 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.824055+0000", 2024-04-20T17:52:52.776 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.011201778000000001 2024-04-20T17:52:52.776 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.776 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.776 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:52:52.776 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.824075+0000", 2024-04-20T17:52:52.776 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9763e-05 2024-04-20T17:52:52.776 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.776 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.776 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:52:52.776 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.880971+0000", 2024-04-20T17:52:52.776 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.056895657000000002 2024-04-20T17:52:52.776 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.776 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.777 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:52:52.777 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.880996+0000", 2024-04-20T17:52:52.777 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5273e-05 2024-04-20T17:52:52.777 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.777 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:52:52.777 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.777 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.777 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.777 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:141781 8.1 e807/806 8:81259101:::benchmark_data_smithi171_175757_object141780:head v 807'8841, mlcod=782'5)", 2024-04-20T17:52:52.777 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:53.818147+0000", 2024-04-20T17:52:52.777 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 478.93657562800001, 2024-04-20T17:52:52.777 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.062882813999999995, 2024-04-20T17:52:52.777 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:52:52.777 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:52:52.778 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:52:52.778 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:52:52.778 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.778 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:52:52.778 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.818147+0000", 2024-04-20T17:52:52.778 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.778 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.778 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.778 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:52:52.778 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.818147+0000", 2024-04-20T17:52:52.778 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.778 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.778 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.779 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:52:52.779 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.818149+0000", 2024-04-20T17:52:52.779 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.827e-06 2024-04-20T17:52:52.779 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.779 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.779 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:52:52.779 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.818203+0000", 2024-04-20T17:52:52.779 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4391999999999997e-05 2024-04-20T17:52:52.779 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.779 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.779 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:52:52.779 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.818215+0000", 2024-04-20T17:52:52.779 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2159e-05 2024-04-20T17:52:52.779 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.779 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.780 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:52:52.780 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.818224+0000", 2024-04-20T17:52:52.780 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8990000000000007e-06 2024-04-20T17:52:52.780 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.780 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.780 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:52:52.780 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.830838+0000", 2024-04-20T17:52:52.780 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012613236 2024-04-20T17:52:52.780 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.780 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.781 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:52:52.781 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.830870+0000", 2024-04-20T17:52:52.781 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2710999999999999e-05 2024-04-20T17:52:52.781 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.781 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.781 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:52:52.781 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.881009+0000", 2024-04-20T17:52:52.781 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.050139184000000003 2024-04-20T17:52:52.781 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.781 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.781 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:52:52.781 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:53.881030+0000", 2024-04-20T17:52:52.782 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0406000000000001e-05 2024-04-20T17:52:52.782 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.782 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:52:52.782 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.782 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.782 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.782 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:143174 8.3 e807/806 8:c2e7ba4e:::benchmark_data_smithi171_175757_object143173:head v 807'8934, mlcod=782'40)", 2024-04-20T17:52:52.782 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:55.689369+0000", 2024-04-20T17:52:52.782 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 477.06535395600002, 2024-04-20T17:52:52.782 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.055773729000000001, 2024-04-20T17:52:52.782 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:52:52.782 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:52:52.783 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:52:52.783 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:52:52.783 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.783 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:52:52.783 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:55.689369+0000", 2024-04-20T17:52:52.783 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.783 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.783 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.783 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:52:52.783 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:55.689369+0000", 2024-04-20T17:52:52.783 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.783 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.783 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.783 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:52:52.784 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:55.689370+0000", 2024-04-20T17:52:52.784 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1100000000000005e-07 2024-04-20T17:52:52.784 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.784 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.784 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:52:52.784 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:55.689426+0000", 2024-04-20T17:52:52.784 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.6876999999999998e-05 2024-04-20T17:52:52.784 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.784 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.784 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:52:52.784 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:55.689434+0000", 2024-04-20T17:52:52.784 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.7340000000000008e-06 2024-04-20T17:52:52.784 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.784 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.784 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:52:52.785 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:55.689438+0000", 2024-04-20T17:52:52.785 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1869999999999999e-06 2024-04-20T17:52:52.785 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.785 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.785 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:52:52.785 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:55.702002+0000", 2024-04-20T17:52:52.785 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012563194 2024-04-20T17:52:52.785 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.785 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.785 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:52:52.785 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:55.702027+0000", 2024-04-20T17:52:52.785 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.5931999999999998e-05 2024-04-20T17:52:52.785 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.785 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.785 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:52:52.786 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:55.745085+0000", 2024-04-20T17:52:52.786 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.043057722 2024-04-20T17:52:52.786 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.786 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.786 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:52:52.786 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:55.745142+0000", 2024-04-20T17:52:52.786 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.7272000000000003e-05 2024-04-20T17:52:52.786 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.786 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:52:52.786 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.786 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.786 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.786 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:144039 8.7 e807/806 8:ee75ddde:::benchmark_data_smithi171_175757_object144038:head v 807'8913, mlcod=782'73)", 2024-04-20T17:52:52.787 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:56.891321+0000", 2024-04-20T17:52:52.787 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 475.863401399, 2024-04-20T17:52:52.787 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.058286162000000002, 2024-04-20T17:52:52.787 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:52:52.787 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:52:52.787 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:52:52.787 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:52:52.787 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.787 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:52:52.787 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:56.891321+0000", 2024-04-20T17:52:52.787 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.787 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.787 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.787 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:52:52.788 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:56.891321+0000", 2024-04-20T17:52:52.788 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.788 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.788 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.788 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:52:52.788 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:56.891323+0000", 2024-04-20T17:52:52.788 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.956e-06 2024-04-20T17:52:52.788 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.788 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.788 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:52:52.788 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:56.891368+0000", 2024-04-20T17:52:52.788 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5219999999999997e-05 2024-04-20T17:52:52.788 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.788 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.788 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:52:52.789 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:56.891383+0000", 2024-04-20T17:52:52.789 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4059999999999999e-05 2024-04-20T17:52:52.789 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.789 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.789 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:52:52.789 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:56.891391+0000", 2024-04-20T17:52:52.789 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1899999999999995e-06 2024-04-20T17:52:52.789 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.789 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.789 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:52:52.789 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:56.891481+0000", 2024-04-20T17:52:52.789 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.0205999999999997e-05 2024-04-20T17:52:52.789 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.789 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.789 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:52:52.790 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:56.891499+0000", 2024-04-20T17:52:52.790 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8042000000000001e-05 2024-04-20T17:52:52.790 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.790 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.790 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:52:52.790 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:56.949568+0000", 2024-04-20T17:52:52.790 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.058069321 2024-04-20T17:52:52.790 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.790 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.790 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:52:52.790 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:56.949607+0000", 2024-04-20T17:52:52.790 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9167e-05 2024-04-20T17:52:52.790 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.790 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:52:52.790 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.791 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.791 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.791 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:144287 8.7 e807/806 8:ee831792:::benchmark_data_smithi171_175757_object144286:head v 807'8930, mlcod=782'79)", 2024-04-20T17:52:52.791 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:57.236564+0000", 2024-04-20T17:52:52.791 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 475.51815918900002, 2024-04-20T17:52:52.791 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.075174172999999997, 2024-04-20T17:52:52.791 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:52:52.791 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:52:52.791 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:52:52.792 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:52:52.792 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.792 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:52:52.792 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.236564+0000", 2024-04-20T17:52:52.792 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.792 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.792 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.792 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:52:52.792 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.236564+0000", 2024-04-20T17:52:52.792 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.792 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.792 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.792 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:52:52.792 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.236564+0000", 2024-04-20T17:52:52.793 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0399999999999996e-07 2024-04-20T17:52:52.793 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.793 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.793 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:52:52.793 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.236592+0000", 2024-04-20T17:52:52.793 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.8288000000000001e-05 2024-04-20T17:52:52.793 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.793 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.793 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:52:52.793 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.236597+0000", 2024-04-20T17:52:52.793 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.0499999999999999e-06 2024-04-20T17:52:52.793 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.793 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.793 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:52:52.794 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.236601+0000", 2024-04-20T17:52:52.794 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.647e-06 2024-04-20T17:52:52.794 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.794 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.794 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:52:52.794 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.239114+0000", 2024-04-20T17:52:52.794 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0025130420000000001 2024-04-20T17:52:52.794 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.794 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.794 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:52:52.794 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.239130+0000", 2024-04-20T17:52:52.794 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6059999999999999e-05 2024-04-20T17:52:52.794 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.795 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.795 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:52:52.795 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.311673+0000", 2024-04-20T17:52:52.795 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.072542822000000007 2024-04-20T17:52:52.795 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.795 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.795 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:52:52.795 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.311738+0000", 2024-04-20T17:52:52.796 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4759999999999997e-05 2024-04-20T17:52:52.796 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.796 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:52:52.796 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.796 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.796 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.796 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:144288 8.7 e807/806 8:eef6a00e:::benchmark_data_smithi171_175757_object144287:head v 807'8931, mlcod=782'79)", 2024-04-20T17:52:52.796 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:57.238491+0000", 2024-04-20T17:52:52.797 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 475.51623187600001, 2024-04-20T17:52:52.797 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.073276907000000002, 2024-04-20T17:52:52.797 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:52:52.797 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:52:52.797 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:52:52.797 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:52:52.798 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.798 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:52:52.798 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.238491+0000", 2024-04-20T17:52:52.798 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.798 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.798 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.799 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:52:52.799 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.238491+0000", 2024-04-20T17:52:52.799 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.799 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.799 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.799 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:52:52.799 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.238493+0000", 2024-04-20T17:52:52.800 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9099999999999999e-06 2024-04-20T17:52:52.800 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.800 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.800 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:52:52.800 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.238796+0000", 2024-04-20T17:52:52.800 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.00030377399999999998 2024-04-20T17:52:52.800 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.801 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.801 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:52:52.801 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.238805+0000", 2024-04-20T17:52:52.801 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.2819999999999996e-06 2024-04-20T17:52:52.801 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.801 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.801 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:52:52.801 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.238811+0000", 2024-04-20T17:52:52.802 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0480000000000004e-06 2024-04-20T17:52:52.802 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.802 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.802 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:52:52.802 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.247356+0000", 2024-04-20T17:52:52.802 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0085456300000000002 2024-04-20T17:52:52.802 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.802 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.802 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:52:52.803 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.247378+0000", 2024-04-20T17:52:52.803 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1098999999999999e-05 2024-04-20T17:52:52.803 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.803 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.803 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:52:52.803 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.311743+0000", 2024-04-20T17:52:52.803 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.064365393000000007 2024-04-20T17:52:52.803 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.803 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.803 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:52:52.803 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.311768+0000", 2024-04-20T17:52:52.804 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4771000000000001e-05 2024-04-20T17:52:52.804 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.804 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:52:52.804 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.804 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.804 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.804 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:144297 8.7 e807/806 8:e477555a:::benchmark_data_smithi171_175757_object144296:head v 807'8932, mlcod=782'79)", 2024-04-20T17:52:52.804 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:57.242447+0000", 2024-04-20T17:52:52.804 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 475.51227579699997, 2024-04-20T17:52:52.804 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.069367660999999997, 2024-04-20T17:52:52.804 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:52:52.804 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:52:52.804 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:52:52.805 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:52:52.805 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.805 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:52:52.805 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.242447+0000", 2024-04-20T17:52:52.805 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.805 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.805 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.805 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:52:52.805 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.242447+0000", 2024-04-20T17:52:52.805 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.805 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.805 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.805 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:52:52.805 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.242449+0000", 2024-04-20T17:52:52.805 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9609999999999999e-06 2024-04-20T17:52:52.806 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.806 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.806 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:52:52.806 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.242483+0000", 2024-04-20T17:52:52.806 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.4607999999999998e-05 2024-04-20T17:52:52.806 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.806 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.806 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:52:52.806 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.242498+0000", 2024-04-20T17:52:52.806 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4133e-05 2024-04-20T17:52:52.806 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.806 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.806 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:52:52.807 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.242508+0000", 2024-04-20T17:52:52.807 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.0125e-05 2024-04-20T17:52:52.807 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.807 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.807 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:52:52.807 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.255277+0000", 2024-04-20T17:52:52.807 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.012769487 2024-04-20T17:52:52.807 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.807 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.807 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:52:52.807 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.255327+0000", 2024-04-20T17:52:52.807 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.9344000000000001e-05 2024-04-20T17:52:52.807 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.807 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.807 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:52:52.808 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.311773+0000", 2024-04-20T17:52:52.808 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.056446552999999997 2024-04-20T17:52:52.808 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.808 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.808 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:52:52.808 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.311815+0000", 2024-04-20T17:52:52.808 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.1449999999999998e-05 2024-04-20T17:52:52.808 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.808 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:52:52.808 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.808 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.808 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.808 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:144299 8.7 e807/806 8:ebf4afb0:::benchmark_data_smithi171_175757_object144298:head v 807'8933, mlcod=782'79)", 2024-04-20T17:52:52.808 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:57.246826+0000", 2024-04-20T17:52:52.809 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 475.50789709700001, 2024-04-20T17:52:52.809 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.065020307999999999, 2024-04-20T17:52:52.809 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:52:52.809 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:52:52.809 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:52:52.809 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:52:52.809 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.809 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:52:52.809 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.246826+0000", 2024-04-20T17:52:52.809 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.809 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.809 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.809 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:52:52.809 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.246826+0000", 2024-04-20T17:52:52.809 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.809 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.810 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.810 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:52:52.810 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.246827+0000", 2024-04-20T17:52:52.810 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.37e-06 2024-04-20T17:52:52.810 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.810 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.810 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:52:52.810 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.246887+0000", 2024-04-20T17:52:52.810 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9725000000000002e-05 2024-04-20T17:52:52.810 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.810 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.810 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:52:52.811 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.246896+0000", 2024-04-20T17:52:52.811 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.8440000000000004e-06 2024-04-20T17:52:52.811 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.811 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.811 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:52:52.811 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.246902+0000", 2024-04-20T17:52:52.811 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.9769999999999997e-06 2024-04-20T17:52:52.811 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.811 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.811 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:52:52.811 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.263252+0000", 2024-04-20T17:52:52.811 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.016350166999999999 2024-04-20T17:52:52.811 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.812 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.812 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:52:52.812 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.263270+0000", 2024-04-20T17:52:52.812 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.8722000000000001e-05 2024-04-20T17:52:52.812 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.812 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.812 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:52:52.812 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.311818+0000", 2024-04-20T17:52:52.812 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.048547740999999998 2024-04-20T17:52:52.813 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.813 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.813 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:52:52.813 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.311846+0000", 2024-04-20T17:52:52.813 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.7761999999999999e-05 2024-04-20T17:52:52.813 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.813 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:52:52.813 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.813 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.813 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.813 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:144300 8.7 e807/806 8:e2881ae0:::benchmark_data_smithi171_175757_object144299:head v 807'8934, mlcod=782'79)", 2024-04-20T17:52:52.813 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:57.251644+0000", 2024-04-20T17:52:52.813 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 475.50307903499998, 2024-04-20T17:52:52.813 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.060234792000000002, 2024-04-20T17:52:52.814 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:52:52.814 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:52:52.814 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:52:52.814 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:52:52.814 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.814 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:52:52.814 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.251644+0000", 2024-04-20T17:52:52.814 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.814 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.814 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.814 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:52:52.814 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.251644+0000", 2024-04-20T17:52:52.815 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.815 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.815 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.815 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:52:52.815 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.251646+0000", 2024-04-20T17:52:52.815 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.537e-06 2024-04-20T17:52:52.815 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.815 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.815 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:52:52.815 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.251685+0000", 2024-04-20T17:52:52.815 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9239999999999997e-05 2024-04-20T17:52:52.815 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.815 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.815 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:52:52.816 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.251698+0000", 2024-04-20T17:52:52.816 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3006999999999999e-05 2024-04-20T17:52:52.816 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.816 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.816 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:52:52.816 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.251708+0000", 2024-04-20T17:52:52.816 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9289999999999993e-06 2024-04-20T17:52:52.816 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.816 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.816 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:52:52.816 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.273674+0000", 2024-04-20T17:52:52.817 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.021965340999999999 2024-04-20T17:52:52.817 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.817 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.817 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:52:52.817 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.273706+0000", 2024-04-20T17:52:52.817 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.1965000000000003e-05 2024-04-20T17:52:52.817 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.817 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.817 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:52:52.817 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.311849+0000", 2024-04-20T17:52:52.817 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.038143205999999999 2024-04-20T17:52:52.817 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.817 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.817 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:52:52.818 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.311878+0000", 2024-04-20T17:52:52.818 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9567e-05 2024-04-20T17:52:52.818 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.818 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:52:52.818 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.818 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.818 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.818 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:144816 8.1 e808/806 8:8c32a843:::benchmark_data_smithi171_175757_object144815:head v 808'9022, mlcod=782'91)", 2024-04-20T17:52:52.818 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:57.976986+0000", 2024-04-20T17:52:52.818 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 474.77773697499998, 2024-04-20T17:52:52.818 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.05750425, 2024-04-20T17:52:52.818 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:52:52.818 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:52:52.819 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:52:52.819 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:52:52.819 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.819 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:52:52.819 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.976986+0000", 2024-04-20T17:52:52.819 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.819 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.819 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.819 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:52:52.819 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.976986+0000", 2024-04-20T17:52:52.819 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.819 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.819 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.820 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:52:52.820 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.976988+0000", 2024-04-20T17:52:52.820 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1399999999999998e-06 2024-04-20T17:52:52.820 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.820 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.820 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:52:52.820 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.977028+0000", 2024-04-20T17:52:52.820 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.9873000000000003e-05 2024-04-20T17:52:52.820 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.820 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.820 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:52:52.821 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.977040+0000", 2024-04-20T17:52:52.821 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.2751999999999999e-05 2024-04-20T17:52:52.821 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.821 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.821 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:52:52.821 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.977047+0000", 2024-04-20T17:52:52.821 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.5699999999999998e-06 2024-04-20T17:52:52.821 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.821 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.821 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:52:52.822 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.982723+0000", 2024-04-20T17:52:52.822 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0056760300000000003 2024-04-20T17:52:52.822 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.822 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.822 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:52:52.822 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.982748+0000", 2024-04-20T17:52:52.822 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4491e-05 2024-04-20T17:52:52.822 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.822 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.823 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:52:52.823 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:58.034438+0000", 2024-04-20T17:52:52.823 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.051690738 2024-04-20T17:52:52.823 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.823 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.823 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:52:52.823 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:58.034490+0000", 2024-04-20T17:52:52.823 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.1656000000000001e-05 2024-04-20T17:52:52.824 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.824 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:52:52.824 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.824 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.824 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.824 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:144830 8.1 e808/806 8:85c3cdba:::benchmark_data_smithi171_175757_object144829:head v 808'9023, mlcod=782'91)", 2024-04-20T17:52:52.824 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:57.978077+0000", 2024-04-20T17:52:52.824 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 474.77664548899997, 2024-04-20T17:52:52.824 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.056453188000000001, 2024-04-20T17:52:52.824 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:52:52.824 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:52:52.824 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:52:52.824 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:52:52.824 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.825 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:52:52.825 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.978077+0000", 2024-04-20T17:52:52.825 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.825 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.825 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.825 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:52:52.825 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.978077+0000", 2024-04-20T17:52:52.825 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.825 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.825 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.825 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:52:52.825 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.978078+0000", 2024-04-20T17:52:52.825 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.186e-06 2024-04-20T17:52:52.825 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.825 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.826 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:52:52.826 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.978133+0000", 2024-04-20T17:52:52.826 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.4703000000000001e-05 2024-04-20T17:52:52.826 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.826 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.826 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:52:52.826 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.978142+0000", 2024-04-20T17:52:52.826 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.67e-06 2024-04-20T17:52:52.826 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.826 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.826 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:52:52.826 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.978148+0000", 2024-04-20T17:52:52.826 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.02e-06 2024-04-20T17:52:52.827 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.827 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.827 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:52:52.827 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.993664+0000", 2024-04-20T17:52:52.827 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015516336 2024-04-20T17:52:52.827 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.827 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.827 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:52:52.827 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:57.993694+0000", 2024-04-20T17:52:52.827 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.0184000000000001e-05 2024-04-20T17:52:52.827 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.827 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.827 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:52:52.827 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:58.034495+0000", 2024-04-20T17:52:52.828 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.040800287999999997 2024-04-20T17:52:52.828 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.828 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.828 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:52:52.828 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:58.034530+0000", 2024-04-20T17:52:52.828 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.5800999999999998e-05 2024-04-20T17:52:52.828 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.828 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:52:52.828 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.828 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.828 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.828 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:145634 8.1 e808/806 8:8ca600c0:::benchmark_data_smithi171_175757_object520:head v 808'9068, mlcod=782'115)", 2024-04-20T17:52:52.828 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:59.137841+0000", 2024-04-20T17:52:52.828 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 473.61688176299998, 2024-04-20T17:52:52.829 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.064744457000000005, 2024-04-20T17:52:52.829 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:52:52.829 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:52:52.829 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:52:52.829 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:52:52.829 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.829 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:52:52.829 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.137841+0000", 2024-04-20T17:52:52.829 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.829 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.829 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.829 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:52:52.829 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.137841+0000", 2024-04-20T17:52:52.829 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.829 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.830 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.830 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:52:52.830 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.137843+0000", 2024-04-20T17:52:52.830 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.962e-06 2024-04-20T17:52:52.830 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.830 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.830 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:52:52.830 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.137872+0000", 2024-04-20T17:52:52.830 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9444000000000001e-05 2024-04-20T17:52:52.830 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.830 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.830 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:52:52.830 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.137884+0000", 2024-04-20T17:52:52.831 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1623999999999999e-05 2024-04-20T17:52:52.831 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.831 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.831 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:52:52.831 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.137892+0000", 2024-04-20T17:52:52.831 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.8290000000000008e-06 2024-04-20T17:52:52.831 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.831 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.832 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:52:52.832 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.137951+0000", 2024-04-20T17:52:52.832 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.8987999999999998e-05 2024-04-20T17:52:52.832 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.832 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.832 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:52:52.833 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.137975+0000", 2024-04-20T17:52:52.833 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4287999999999999e-05 2024-04-20T17:52:52.833 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.833 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.833 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:52:52.834 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.202514+0000", 2024-04-20T17:52:52.834 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.064539223000000007 2024-04-20T17:52:52.834 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.834 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.834 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:52:52.834 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.202585+0000", 2024-04-20T17:52:52.834 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 7.1099000000000005e-05 2024-04-20T17:52:52.834 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.834 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:52:52.835 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.835 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.835 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.835 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:145649 8.1 e808/806 8:8ffd2a1b:::benchmark_data_smithi171_175757_object535:head v 808'9070, mlcod=782'116)", 2024-04-20T17:52:52.835 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:59.155596+0000", 2024-04-20T17:52:52.835 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 473.59912707000001, 2024-04-20T17:52:52.835 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057370499999999998, 2024-04-20T17:52:52.835 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:52:52.835 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:52:52.835 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:52:52.835 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:52:52.835 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.835 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:52:52.835 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.155596+0000", 2024-04-20T17:52:52.836 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.836 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.836 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.836 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:52:52.836 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.155596+0000", 2024-04-20T17:52:52.836 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.836 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.836 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.836 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:52:52.836 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.155599+0000", 2024-04-20T17:52:52.836 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 3.2480000000000001e-06 2024-04-20T17:52:52.837 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.837 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.837 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:52:52.837 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.155614+0000", 2024-04-20T17:52:52.837 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5126999999999999e-05 2024-04-20T17:52:52.837 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.837 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.837 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:52:52.837 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.155631+0000", 2024-04-20T17:52:52.837 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7212000000000001e-05 2024-04-20T17:52:52.837 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.837 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.837 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:52:52.837 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.155643+0000", 2024-04-20T17:52:52.838 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.1657e-05 2024-04-20T17:52:52.838 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.838 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.838 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:52:52.838 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.159944+0000", 2024-04-20T17:52:52.838 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.004301633 2024-04-20T17:52:52.838 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.838 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.838 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:52:52.838 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.159968+0000", 2024-04-20T17:52:52.838 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3558999999999999e-05 2024-04-20T17:52:52.838 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.838 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.838 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:52:52.839 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.212921+0000", 2024-04-20T17:52:52.839 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.052953001999999999 2024-04-20T17:52:52.839 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.839 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.839 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:52:52.839 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.212966+0000", 2024-04-20T17:52:52.839 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 4.5062000000000003e-05 2024-04-20T17:52:52.839 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.839 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:52:52.839 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.839 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.839 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.839 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:145651 8.1 e808/806 8:86aaf9bb:::benchmark_data_smithi171_175757_object537:head v 808'9071, mlcod=782'116)", 2024-04-20T17:52:52.839 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:44:59.155671+0000", 2024-04-20T17:52:52.840 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 473.59905176500001, 2024-04-20T17:52:52.840 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.057312692999999998, 2024-04-20T17:52:52.840 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:52:52.840 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:52:52.840 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:52:52.840 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:52:52.840 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.840 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:52:52.840 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.155671+0000", 2024-04-20T17:52:52.840 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.840 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.840 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.841 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:52:52.841 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.155671+0000", 2024-04-20T17:52:52.841 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.841 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.841 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.841 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:52:52.841 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.155672+0000", 2024-04-20T17:52:52.841 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.6899999999999997e-07 2024-04-20T17:52:52.841 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.841 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.841 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:52:52.841 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.155677+0000", 2024-04-20T17:52:52.841 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.2290000000000002e-06 2024-04-20T17:52:52.841 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.842 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.842 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:52:52.842 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.155683+0000", 2024-04-20T17:52:52.842 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.4799999999999998e-06 2024-04-20T17:52:52.842 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.842 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.842 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:52:52.842 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.155688+0000", 2024-04-20T17:52:52.842 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 5.074e-06 2024-04-20T17:52:52.842 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.842 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.842 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:52:52.842 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.171012+0000", 2024-04-20T17:52:52.843 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.015323945 2024-04-20T17:52:52.843 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.843 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.843 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:52:52.843 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.171032+0000", 2024-04-20T17:52:52.843 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.9808000000000001e-05 2024-04-20T17:52:52.843 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.843 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.843 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:52:52.843 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.212969+0000", 2024-04-20T17:52:52.843 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.041937161000000001 2024-04-20T17:52:52.843 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.843 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.843 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:52:52.843 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:44:59.212984+0000", 2024-04-20T17:52:52.844 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4327e-05 2024-04-20T17:52:52.844 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.844 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:52:52.844 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.844 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.844 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.844 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:240989 8.7 e863/806 8:e80f15e6:::benchmark_data_smithi171_175757_object95875:head v 863'14928, mlcod=796'5945)", 2024-04-20T17:52:52.845 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:47:30.033407+0000", 2024-04-20T17:52:52.845 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 322.72131528699998, 2024-04-20T17:52:52.845 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.107769315, 2024-04-20T17:52:52.845 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:52:52.845 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:52:52.845 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:52:52.845 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:52:52.845 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.845 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:52:52.845 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.033407+0000", 2024-04-20T17:52:52.845 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.845 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.846 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.846 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:52:52.846 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.033407+0000", 2024-04-20T17:52:52.846 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.846 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.846 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.846 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:52:52.846 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.033409+0000", 2024-04-20T17:52:52.846 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.7489999999999999e-06 2024-04-20T17:52:52.846 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.846 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.846 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:52:52.846 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.033424+0000", 2024-04-20T17:52:52.846 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5211999999999999e-05 2024-04-20T17:52:52.847 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.847 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.847 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:52:52.847 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.033441+0000", 2024-04-20T17:52:52.847 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.6954999999999999e-05 2024-04-20T17:52:52.847 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.847 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.847 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:52:52.847 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.033454+0000", 2024-04-20T17:52:52.847 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.3052e-05 2024-04-20T17:52:52.847 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.847 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.847 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:52:52.848 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.033515+0000", 2024-04-20T17:52:52.848 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 6.0470999999999998e-05 2024-04-20T17:52:52.848 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.848 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.848 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:52:52.848 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.033538+0000", 2024-04-20T17:52:52.848 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3448999999999999e-05 2024-04-20T17:52:52.848 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.848 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.848 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:52:52.848 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.041571+0000", 2024-04-20T17:52:52.848 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0080328229999999997 2024-04-20T17:52:52.849 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.849 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.849 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:52:52.849 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.141177+0000", 2024-04-20T17:52:52.849 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.099605604 2024-04-20T17:52:52.849 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.849 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:52:52.849 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.849 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.849 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.849 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDPGPush(8.7 863/806 [PushOp(8:ecaaf859:::benchmark_data_smithi171_175757_object95918:head, version: 796'5950, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:ecaaf859:::benchmark_data_smithi171_175757_object95918:head@796'5950, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-20T17:52:52.849 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:47:30.043161+0000", 2024-04-20T17:52:52.849 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 322.711562071, 2024-04-20T17:52:52.849 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.10206235600000001, 2024-04-20T17:52:52.850 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:52:52.850 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:52:52.850 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:52:52.850 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:52:52.850 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.850 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:52:52.850 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.043161+0000", 2024-04-20T17:52:52.850 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.850 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.850 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.850 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:52:52.850 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.043161+0000", 2024-04-20T17:52:52.851 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.851 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.851 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.851 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:52:52.851 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.043162+0000", 2024-04-20T17:52:52.851 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.5230000000000001e-06 2024-04-20T17:52:52.851 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.851 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.851 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:52:52.851 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.043191+0000", 2024-04-20T17:52:52.851 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.9196e-05 2024-04-20T17:52:52.851 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.851 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.851 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:52:52.851 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.141148+0000", 2024-04-20T17:52:52.852 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.097956399999999999 2024-04-20T17:52:52.852 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.852 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.852 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:52:52.852 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.141234+0000", 2024-04-20T17:52:52.852 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.6267999999999995e-05 2024-04-20T17:52:52.852 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.852 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.852 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:52:52.852 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.141327+0000", 2024-04-20T17:52:52.852 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.3010000000000006e-05 2024-04-20T17:52:52.852 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.852 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.852 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:52:52.853 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.141341+0000", 2024-04-20T17:52:52.853 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.4246e-05 2024-04-20T17:52:52.853 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.853 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.853 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:52:52.853 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.145223+0000", 2024-04-20T17:52:52.853 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0038817130000000002 2024-04-20T17:52:52.853 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.853 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:52:52.853 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.853 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.853 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.853 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "MOSDPGPush(8.1 863/806 [PushOp(8:8f0cf38b:::benchmark_data_smithi171_175757_object95988:head, version: 796'5997, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:8f0cf38b:::benchmark_data_smithi171_175757_object95988:head@796'5997, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-20T17:52:52.853 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:47:30.051094+0000", 2024-04-20T17:52:52.853 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 322.70362906399998, 2024-04-20T17:52:52.854 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.093414170000000005, 2024-04-20T17:52:52.854 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:52:52.854 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:52:52.854 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "started", 2024-04-20T17:52:52.854 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:52:52.854 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.854 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:52:52.854 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.051094+0000", 2024-04-20T17:52:52.854 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.854 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.855 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.855 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:52:52.855 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.051094+0000", 2024-04-20T17:52:52.855 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.855 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.855 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.855 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:52:52.855 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.051096+0000", 2024-04-20T17:52:52.855 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.3070000000000001e-06 2024-04-20T17:52:52.855 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.855 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.856 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:52:52.856 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.051177+0000", 2024-04-20T17:52:52.856 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.1240000000000001e-05 2024-04-20T17:52:52.856 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.856 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.856 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:52:52.856 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.141152+0000", 2024-04-20T17:52:52.856 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.089974738999999998 2024-04-20T17:52:52.856 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.856 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.856 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:52:52.856 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.141233+0000", 2024-04-20T17:52:52.856 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 8.0616000000000002e-05 2024-04-20T17:52:52.856 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.857 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.857 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:52:52.857 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.141332+0000", 2024-04-20T17:52:52.857 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.9352999999999996e-05 2024-04-20T17:52:52.857 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.857 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.857 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:52:52.857 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.141345+0000", 2024-04-20T17:52:52.857 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.346e-05 2024-04-20T17:52:52.857 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.857 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.857 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:52:52.857 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:47:30.144508+0000", 2024-04-20T17:52:52.857 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.0031624550000000002 2024-04-20T17:52:52.857 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.858 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:52:52.858 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.858 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.858 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.858 INFO:teuthology.orchestra.run.smithi005.stdout: "description": "osd_repop(client.27460.0:283186 8.3 e881/806 8:c4dc4640:::benchmark_data_smithi171_175757_object138072:head v 881'17694, mlcod=805'8780)", 2024-04-20T17:52:52.858 INFO:teuthology.orchestra.run.smithi005.stdout: "initiated_at": "2024-04-20T17:48:18.242274+0000", 2024-04-20T17:52:52.858 INFO:teuthology.orchestra.run.smithi005.stdout: "age": 274.51244883099997, 2024-04-20T17:52:52.858 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.056183654999999999, 2024-04-20T17:52:52.858 INFO:teuthology.orchestra.run.smithi005.stdout: "continuous": false, 2024-04-20T17:52:52.858 INFO:teuthology.orchestra.run.smithi005.stdout: "type_data": { 2024-04-20T17:52:52.858 INFO:teuthology.orchestra.run.smithi005.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:52:52.858 INFO:teuthology.orchestra.run.smithi005.stdout: "events": [ 2024-04-20T17:52:52.858 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.859 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "initiated", 2024-04-20T17:52:52.859 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:18.242274+0000", 2024-04-20T17:52:52.859 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.859 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.859 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.859 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "header_read", 2024-04-20T17:52:52.859 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:18.242274+0000", 2024-04-20T17:52:52.859 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0 2024-04-20T17:52:52.859 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.859 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.859 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "throttled", 2024-04-20T17:52:52.859 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:18.242276+0000", 2024-04-20T17:52:52.859 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 1.815e-06 2024-04-20T17:52:52.859 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.859 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.860 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "all_read", 2024-04-20T17:52:52.860 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:18.242298+0000", 2024-04-20T17:52:52.860 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.2166000000000001e-05 2024-04-20T17:52:52.860 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.860 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.860 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "dispatched", 2024-04-20T17:52:52.860 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:18.242319+0000", 2024-04-20T17:52:52.861 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.1067e-05 2024-04-20T17:52:52.861 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.861 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.861 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "queued_for_pg", 2024-04-20T17:52:52.861 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:18.242329+0000", 2024-04-20T17:52:52.861 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 9.8560000000000002e-06 2024-04-20T17:52:52.861 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.861 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.861 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "reached_pg", 2024-04-20T17:52:52.861 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:18.245012+0000", 2024-04-20T17:52:52.861 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.002683497 2024-04-20T17:52:52.861 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.861 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.861 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "started", 2024-04-20T17:52:52.862 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:18.245037+0000", 2024-04-20T17:52:52.862 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.4702000000000002e-05 2024-04-20T17:52:52.862 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.862 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.862 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "commit_sent", 2024-04-20T17:52:52.862 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:18.298437+0000", 2024-04-20T17:52:52.862 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 0.053400446999999997 2024-04-20T17:52:52.862 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-04-20T17:52:52.862 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-04-20T17:52:52.862 INFO:teuthology.orchestra.run.smithi005.stdout: "event": "done", 2024-04-20T17:52:52.862 INFO:teuthology.orchestra.run.smithi005.stdout: "time": "2024-04-20T17:48:18.298458+0000", 2024-04-20T17:52:52.862 INFO:teuthology.orchestra.run.smithi005.stdout: "duration": 2.0105000000000001e-05 2024-04-20T17:52:52.862 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.862 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:52:52.863 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.863 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-04-20T17:52:52.863 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-04-20T17:52:52.863 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:52:53.208 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:53 smithi005 ceph-mon[39562]: 9.8 scrub starts 2024-04-20T17:52:53.208 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:53 smithi005 ceph-mon[39562]: pgmap v2564: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 1.4 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-04-20T17:52:53.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:53 smithi171 ceph-mon[44189]: 9.8 scrub starts 2024-04-20T17:52:53.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:53 smithi171 ceph-mon[44189]: pgmap v2564: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 1.4 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-04-20T17:52:53.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:53 smithi106 ceph-mon[44423]: 9.8 scrub starts 2024-04-20T17:52:53.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:53 smithi106 ceph-mon[44423]: pgmap v2564: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 1.4 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-04-20T17:52:54.002 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:52:54.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:54.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:52:54.053 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":2563,"stamp":"2024-04-20T17:52:51.001426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1605042744,"num_objects":24485,"num_object_clones":0,"num_object_copies":48970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24485,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":309105,"num_write_kb":10674753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12720,"num_bytes_recovered":833486848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162209,"ondisk_log_size":162209,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":10227016,"kb_used_data":3291348,"kb_used_omap":404,"kb_used_meta":6935083,"kb_avail":1114714808,"statfs":{"total":1151940427776,"available":1141467963392,"internally_reserved":0,"allocated":3370340352,"data_stored":3366023898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":414014,"internal_metadata":7101525698},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-530907136,"num_objects":-8101,"num_object_clones":0,"num_object_copies":-16202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8101,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.271873"},"pg_stats":[{"pgid":"9.7","version":"970'19211","reported_seq":19555,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-20T17:52:50.277182+0000","last_change":"2024-04-20T17:52:35.055360+0000","last_active":"2024-04-20T17:52:50.277182+0000","last_peered":"2024-04-20T17:52:50.277182+0000","last_clean":"2024-04-20T17:52:50.277182+0000","last_became_active":"2024-04-20T17:48:54.814508+0000","last_became_peered":"2024-04-20T17:48:54.814508+0000","last_unstale":"2024-04-20T17:52:50.277182+0000","last_undegraded":"2024-04-20T17:52:50.277182+0000","last_fullsized":"2024-04-20T17:52:50.277182+0000","mapping_epoch":894,"log_start":"927'9200","ondisk_log_start":"927'9200","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"965'18108","last_scrub_stamp":"2024-04-20T17:52:35.055125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:52:35.055125+0000","objects_scrubbed":2572,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:54:01.516960+0000","scrub_duration":447,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95092736,"num_objects":1451,"num_object_clones":0,"num_object_copies":2902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19211,"num_write_kb":661184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"970'19218","reported_seq":29376,"reported_epoch":970,"state":"active+clean+scrubbing+deep","last_fresh":"2024-04-20T17:52:46.551963+0000","last_change":"2024-04-20T17:52:45.845413+0000","last_active":"2024-04-20T17:52:46.551963+0000","last_peered":"2024-04-20T17:52:46.551963+0000","last_clean":"2024-04-20T17:52:46.551963+0000","last_became_active":"2024-04-20T17:52:10.932624+0000","last_became_peered":"2024-04-20T17:52:10.932624+0000","last_unstale":"2024-04-20T17:52:46.551963+0000","last_undegraded":"2024-04-20T17:52:46.551963+0000","last_fullsized":"2024-04-20T17:52:46.551963+0000","mapping_epoch":959,"log_start":"927'9200","ondisk_log_start":"927'9200","created":894,"last_epoch_clean":960,"parent":"0.0","parent_split_bits":0,"last_scrub":"929'9608","last_scrub_stamp":"2024-04-20T17:50:25.096273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:50:25.096273+0000","objects_scrubbed":124,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":2229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110493696,"num_objects":1686,"num_object_clones":0,"num_object_copies":3372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19094,"num_write_kb":664960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4942,"num_bytes_recovered":323878912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.5","version":"970'19101","reported_seq":19436,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-20T17:52:46.196314+0000","last_change":"2024-04-20T17:51:42.519313+0000","last_active":"2024-04-20T17:52:46.196314+0000","last_peered":"2024-04-20T17:52:46.196314+0000","last_clean":"2024-04-20T17:52:46.196314+0000","last_became_active":"2024-04-20T17:48:54.817749+0000","last_became_peered":"2024-04-20T17:48:54.817749+0000","last_unstale":"2024-04-20T17:52:46.196314+0000","last_undegraded":"2024-04-20T17:52:46.196314+0000","last_fullsized":"2024-04-20T17:52:46.196314+0000","mapping_epoch":894,"log_start":"927'9100","ondisk_log_start":"927'9100","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'14734","last_scrub_stamp":"2024-04-20T17:51:42.519071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:42.519071+0000","objects_scrubbed":6056,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:09.147760+0000","scrub_duration":1274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107675648,"num_objects":1643,"num_object_clones":0,"num_object_copies":3286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19101,"num_write_kb":663808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"970'19301","reported_seq":19629,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-20T17:52:50.921571+0000","last_change":"2024-04-20T17:51:48.836352+0000","last_active":"2024-04-20T17:52:50.921571+0000","last_peered":"2024-04-20T17:52:50.921571+0000","last_clean":"2024-04-20T17:52:50.921571+0000","last_became_active":"2024-04-20T17:48:55.061199+0000","last_became_peered":"2024-04-20T17:48:55.061199+0000","last_unstale":"2024-04-20T17:52:50.921571+0000","last_undegraded":"2024-04-20T17:52:50.921571+0000","last_fullsized":"2024-04-20T17:52:50.921571+0000","mapping_epoch":894,"log_start":"927'9300","ondisk_log_start":"927'9300","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"954'15006","last_scrub_stamp":"2024-04-20T17:51:48.836218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:48.836218+0000","objects_scrubbed":5762,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:51.196876+0000","scrub_duration":1412,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92471296,"num_objects":1411,"num_object_clones":0,"num_object_copies":2822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19301,"num_write_kb":662784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"970'19070","reported_seq":34974,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-20T17:52:49.042605+0000","last_change":"2024-04-20T17:51:42.767947+0000","last_active":"2024-04-20T17:52:49.042605+0000","last_peered":"2024-04-20T17:52:49.042605+0000","last_clean":"2024-04-20T17:52:49.042605+0000","last_became_active":"2024-04-20T17:51:40.523417+0000","last_became_peered":"2024-04-20T17:51:40.523417+0000","last_unstale":"2024-04-20T17:52:49.042605+0000","last_undegraded":"2024-04-20T17:52:49.042605+0000","last_fullsized":"2024-04-20T17:52:49.042605+0000","mapping_epoch":950,"log_start":"927'9000","ondisk_log_start":"927'9000","created":894,"last_epoch_clean":951,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'14655","last_scrub_stamp":"2024-04-20T17:51:42.767651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:42.767651+0000","objects_scrubbed":5939,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:50.590021+0000","scrub_duration":1039,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98697216,"num_objects":1506,"num_object_clones":0,"num_object_copies":3012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19070,"num_write_kb":658432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7778,"num_bytes_recovered":509607936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.e","version":"970'19475","reported_seq":19764,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-20T17:52:49.842520+0000","last_change":"2024-04-20T17:52:39.798879+0000","last_active":"2024-04-20T17:52:49.842520+0000","last_peered":"2024-04-20T17:52:49.842520+0000","last_clean":"2024-04-20T17:52:49.842520+0000","last_became_active":"2024-04-20T17:48:54.815159+0000","last_became_peered":"2024-04-20T17:48:54.815159+0000","last_unstale":"2024-04-20T17:52:49.842520+0000","last_undegraded":"2024-04-20T17:52:49.842520+0000","last_fullsized":"2024-04-20T17:52:49.842520+0000","mapping_epoch":894,"log_start":"927'9400","ondisk_log_start":"927'9400","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"967'18791","last_scrub_stamp":"2024-04-20T17:52:39.798690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:52:39.798690+0000","objects_scrubbed":2095,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:54:07.165535+0000","scrub_duration":452,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91553792,"num_objects":1397,"num_object_clones":0,"num_object_copies":2794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19475,"num_write_kb":667904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10],"acting":[7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"15'137","reported_seq":1000,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-20T17:52:45.966941+0000","last_change":"2024-04-20T17:52:36.689482+0000","last_active":"2024-04-20T17:52:45.966941+0000","last_peered":"2024-04-20T17:52:45.966941+0000","last_clean":"2024-04-20T17:52:45.966941+0000","last_became_active":"2024-04-20T17:45:32.670941+0000","last_became_peered":"2024-04-20T17:45:32.670941+0000","last_unstale":"2024-04-20T17:52:45.966941+0000","last_undegraded":"2024-04-20T17:52:45.966941+0000","last_fullsized":"2024-04-20T17:52:45.966941+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:52:36.689379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:52:36.689379+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:56.828094+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"970'19354","reported_seq":19687,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-20T17:52:50.682774+0000","last_change":"2024-04-20T17:51:43.065593+0000","last_active":"2024-04-20T17:52:50.682774+0000","last_peered":"2024-04-20T17:52:50.682774+0000","last_clean":"2024-04-20T17:52:50.682774+0000","last_became_active":"2024-04-20T17:48:55.061161+0000","last_became_peered":"2024-04-20T17:48:55.061161+0000","last_unstale":"2024-04-20T17:52:50.682774+0000","last_undegraded":"2024-04-20T17:52:50.682774+0000","last_fullsized":"2024-04-20T17:52:50.682774+0000","mapping_epoch":894,"log_start":"927'9300","ondisk_log_start":"927'9300","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'14822","last_scrub_stamp":"2024-04-20T17:51:43.065456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:43.065456+0000","objects_scrubbed":6081,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:57.084267+0000","scrub_duration":1030,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99745792,"num_objects":1522,"num_object_clones":0,"num_object_copies":3044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19354,"num_write_kb":668032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"15'137","reported_seq":998,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-20T17:52:45.972565+0000","last_change":"2024-04-20T17:51:42.855635+0000","last_active":"2024-04-20T17:52:45.972565+0000","last_peered":"2024-04-20T17:52:45.972565+0000","last_clean":"2024-04-20T17:52:45.972565+0000","last_became_active":"2024-04-20T17:45:29.817528+0000","last_became_peered":"2024-04-20T17:45:29.817528+0000","last_unstale":"2024-04-20T17:52:45.972565+0000","last_undegraded":"2024-04-20T17:52:45.972565+0000","last_fullsized":"2024-04-20T17:52:45.972565+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:42.855496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:42.855496+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:49.448590+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"970'19476","reported_seq":19804,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-20T17:52:48.775538+0000","last_change":"2024-04-20T17:52:01.435065+0000","last_active":"2024-04-20T17:52:48.775538+0000","last_peered":"2024-04-20T17:52:48.775538+0000","last_clean":"2024-04-20T17:52:48.775538+0000","last_became_active":"2024-04-20T17:48:54.817445+0000","last_became_peered":"2024-04-20T17:48:54.817445+0000","last_unstale":"2024-04-20T17:52:48.775538+0000","last_undegraded":"2024-04-20T17:52:48.775538+0000","last_fullsized":"2024-04-20T17:52:48.775538+0000","mapping_epoch":894,"log_start":"927'9400","ondisk_log_start":"927'9400","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"956'16160","last_scrub_stamp":"2024-04-20T17:52:01.434805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:52:01.434805+0000","objects_scrubbed":4908,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:21.628859+0000","scrub_duration":1125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101056512,"num_objects":1542,"num_object_clones":0,"num_object_copies":3084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19476,"num_write_kb":672576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"15'137","reported_seq":1024,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-20T17:52:45.973981+0000","last_change":"2024-04-20T17:51:44.510869+0000","last_active":"2024-04-20T17:52:45.973981+0000","last_peered":"2024-04-20T17:52:45.973981+0000","last_clean":"2024-04-20T17:52:45.973981+0000","last_became_active":"2024-04-20T17:45:29.827577+0000","last_became_peered":"2024-04-20T17:45:29.827577+0000","last_unstale":"2024-04-20T17:52:45.973981+0000","last_undegraded":"2024-04-20T17:52:45.973981+0000","last_fullsized":"2024-04-20T17:52:45.973981+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:44.510742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:44.510742+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:46.934138+0000","scrub_duration":266,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"970'19303","reported_seq":19662,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-20T17:52:49.843257+0000","last_change":"2024-04-20T17:52:35.727967+0000","last_active":"2024-04-20T17:52:49.843257+0000","last_peered":"2024-04-20T17:52:49.843257+0000","last_clean":"2024-04-20T17:52:49.843257+0000","last_became_active":"2024-04-20T17:48:55.061208+0000","last_became_peered":"2024-04-20T17:48:55.061208+0000","last_unstale":"2024-04-20T17:52:49.843257+0000","last_undegraded":"2024-04-20T17:52:49.843257+0000","last_fullsized":"2024-04-20T17:52:49.843257+0000","mapping_epoch":894,"log_start":"927'9300","ondisk_log_start":"927'9300","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"965'18278","last_scrub_stamp":"2024-04-20T17:52:35.727850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:52:35.727850+0000","objects_scrubbed":2544,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:36.489757+0000","scrub_duration":423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98304024,"num_objects":1501,"num_object_clones":0,"num_object_copies":3002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1501,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":19303,"num_write_kb":665665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"15'137","reported_seq":992,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-20T17:52:45.964571+0000","last_change":"2024-04-20T17:51:46.197902+0000","last_active":"2024-04-20T17:52:45.964571+0000","last_peered":"2024-04-20T17:52:45.964571+0000","last_clean":"2024-04-20T17:52:45.964571+0000","last_became_active":"2024-04-20T17:45:32.666523+0000","last_became_peered":"2024-04-20T17:45:32.666523+0000","last_unstale":"2024-04-20T17:52:45.964571+0000","last_undegraded":"2024-04-20T17:52:45.964571+0000","last_fullsized":"2024-04-20T17:52:45.964571+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:46.197797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:46.197797+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:56.371129+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.a","version":"970'19614","reported_seq":19912,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-20T17:52:50.686545+0000","last_change":"2024-04-20T17:52:40.226697+0000","last_active":"2024-04-20T17:52:50.686545+0000","last_peered":"2024-04-20T17:52:50.686545+0000","last_clean":"2024-04-20T17:52:50.686545+0000","last_became_active":"2024-04-20T17:48:55.061180+0000","last_became_peered":"2024-04-20T17:48:55.061180+0000","last_unstale":"2024-04-20T17:52:50.686545+0000","last_undegraded":"2024-04-20T17:52:50.686545+0000","last_fullsized":"2024-04-20T17:52:50.686545+0000","mapping_epoch":894,"log_start":"928'9600","ondisk_log_start":"928'9600","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"967'18870","last_scrub_stamp":"2024-04-20T17:52:40.226531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:52:40.226531+0000","objects_scrubbed":2122,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:46.983591+0000","scrub_duration":435,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89391104,"num_objects":1364,"num_object_clones":0,"num_object_copies":2728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19614,"num_write_kb":671296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"15'137","reported_seq":993,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-20T17:52:45.965892+0000","last_change":"2024-04-20T17:51:43.176176+0000","last_active":"2024-04-20T17:52:45.965892+0000","last_peered":"2024-04-20T17:52:45.965892+0000","last_clean":"2024-04-20T17:52:45.965892+0000","last_became_active":"2024-04-20T17:46:40.831961+0000","last_became_peered":"2024-04-20T17:46:40.831961+0000","last_unstale":"2024-04-20T17:52:45.965892+0000","last_undegraded":"2024-04-20T17:52:45.965892+0000","last_fullsized":"2024-04-20T17:52:45.965892+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:43.176019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:43.176019+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:57.843728+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"970'19143","reported_seq":19471,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-20T17:52:49.003329+0000","last_change":"2024-04-20T17:51:46.152560+0000","last_active":"2024-04-20T17:52:49.003329+0000","last_peered":"2024-04-20T17:52:49.003329+0000","last_clean":"2024-04-20T17:52:49.003329+0000","last_became_active":"2024-04-20T17:48:54.816644+0000","last_became_peered":"2024-04-20T17:48:54.816644+0000","last_unstale":"2024-04-20T17:52:49.003329+0000","last_undegraded":"2024-04-20T17:52:49.003329+0000","last_fullsized":"2024-04-20T17:52:49.003329+0000","mapping_epoch":894,"log_start":"927'9100","ondisk_log_start":"927'9100","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"952'14792","last_scrub_stamp":"2024-04-20T17:51:46.152388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:46.152388+0000","objects_scrubbed":5974,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:04.134118+0000","scrub_duration":519,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106364928,"num_objects":1623,"num_object_clones":0,"num_object_copies":3246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19143,"num_write_kb":664512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"15'137","reported_seq":793,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-20T17:52:45.968183+0000","last_change":"2024-04-20T17:52:38.639201+0000","last_active":"2024-04-20T17:52:45.968183+0000","last_peered":"2024-04-20T17:52:45.968183+0000","last_clean":"2024-04-20T17:52:45.968183+0000","last_became_active":"2024-04-20T17:49:46.824559+0000","last_became_peered":"2024-04-20T17:49:46.824559+0000","last_unstale":"2024-04-20T17:52:45.968183+0000","last_undegraded":"2024-04-20T17:52:45.968183+0000","last_fullsized":"2024-04-20T17:52:45.968183+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:52:38.639077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:52:38.639077+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:54:06.972106+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"970'19800","reported_seq":20135,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-20T17:52:50.306401+0000","last_change":"2024-04-20T17:51:50.104395+0000","last_active":"2024-04-20T17:52:50.306401+0000","last_peered":"2024-04-20T17:52:50.306401+0000","last_clean":"2024-04-20T17:52:50.306401+0000","last_became_active":"2024-04-20T17:48:54.813199+0000","last_became_peered":"2024-04-20T17:48:54.813199+0000","last_unstale":"2024-04-20T17:52:50.306401+0000","last_undegraded":"2024-04-20T17:52:50.306401+0000","last_fullsized":"2024-04-20T17:52:50.306401+0000","mapping_epoch":894,"log_start":"928'9700","ondisk_log_start":"928'9700","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"954'15465","last_scrub_stamp":"2024-04-20T17:51:50.104218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:50.104218+0000","objects_scrubbed":5848,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:05.037497+0000","scrub_duration":1044,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97124352,"num_objects":1482,"num_object_clones":0,"num_object_copies":2964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19800,"num_write_kb":681024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11],"acting":[10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"15'137","reported_seq":1005,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-20T17:52:45.966472+0000","last_change":"2024-04-20T17:52:45.966472+0000","last_active":"2024-04-20T17:52:45.966472+0000","last_peered":"2024-04-20T17:52:45.966472+0000","last_clean":"2024-04-20T17:52:45.966472+0000","last_became_active":"2024-04-20T17:52:45.966123+0000","last_became_peered":"2024-04-20T17:52:45.966123+0000","last_unstale":"2024-04-20T17:52:45.966472+0000","last_undegraded":"2024-04-20T17:52:45.966472+0000","last_fullsized":"2024-04-20T17:52:45.966472+0000","mapping_epoch":969,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":970,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:52:40.844359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:52:40.844359+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:41.584753+0000","scrub_duration":168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.2","version":"970'19220","reported_seq":19549,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-20T17:52:48.748383+0000","last_change":"2024-04-20T17:51:43.530794+0000","last_active":"2024-04-20T17:52:48.748383+0000","last_peered":"2024-04-20T17:52:48.748383+0000","last_clean":"2024-04-20T17:52:48.748383+0000","last_became_active":"2024-04-20T17:48:55.060965+0000","last_became_peered":"2024-04-20T17:48:55.060965+0000","last_unstale":"2024-04-20T17:52:48.748383+0000","last_undegraded":"2024-04-20T17:52:48.748383+0000","last_fullsized":"2024-04-20T17:52:48.748383+0000","mapping_epoch":894,"log_start":"927'9200","ondisk_log_start":"927'9200","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'14849","last_scrub_stamp":"2024-04-20T17:51:43.530564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:43.530564+0000","objects_scrubbed":5973,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:05.542393+0000","scrub_duration":1032,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102236160,"num_objects":1560,"num_object_clones":0,"num_object_copies":3120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19220,"num_write_kb":664960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"15'137","reported_seq":978,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-20T17:52:45.975259+0000","last_change":"2024-04-20T17:51:41.848577+0000","last_active":"2024-04-20T17:52:45.975259+0000","last_peered":"2024-04-20T17:52:45.975259+0000","last_clean":"2024-04-20T17:52:45.975259+0000","last_became_active":"2024-04-20T17:45:32.669451+0000","last_became_peered":"2024-04-20T17:45:32.669451+0000","last_unstale":"2024-04-20T17:52:45.975259+0000","last_undegraded":"2024-04-20T17:52:45.975259+0000","last_fullsized":"2024-04-20T17:52:45.975259+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:41.848440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:41.848440+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:55.264457+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"970'19091","reported_seq":19429,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-20T17:52:46.751160+0000","last_change":"2024-04-20T17:51:42.714681+0000","last_active":"2024-04-20T17:52:46.751160+0000","last_peered":"2024-04-20T17:52:46.751160+0000","last_clean":"2024-04-20T17:52:46.751160+0000","last_became_active":"2024-04-20T17:48:55.061200+0000","last_became_peered":"2024-04-20T17:48:55.061200+0000","last_unstale":"2024-04-20T17:52:46.751160+0000","last_undegraded":"2024-04-20T17:52:46.751160+0000","last_fullsized":"2024-04-20T17:52:46.751160+0000","mapping_epoch":894,"log_start":"926'9000","ondisk_log_start":"926'9000","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'14728","last_scrub_stamp":"2024-04-20T17:51:42.714540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:42.714540+0000","objects_scrubbed":6058,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:00.487021+0000","scrub_duration":1002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108724224,"num_objects":1659,"num_object_clones":0,"num_object_copies":3318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19091,"num_write_kb":664000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"15'137","reported_seq":993,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-20T17:52:45.982036+0000","last_change":"2024-04-20T17:51:40.826215+0000","last_active":"2024-04-20T17:52:45.982036+0000","last_peered":"2024-04-20T17:52:45.982036+0000","last_clean":"2024-04-20T17:52:45.982036+0000","last_became_active":"2024-04-20T17:45:32.644584+0000","last_became_peered":"2024-04-20T17:45:32.644584+0000","last_unstale":"2024-04-20T17:52:45.982036+0000","last_undegraded":"2024-04-20T17:52:45.982036+0000","last_fullsized":"2024-04-20T17:52:45.982036+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:40.826081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:40.826081+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"970'21783","reported_seq":22185,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-20T17:52:48.762351+0000","last_change":"2024-04-20T17:52:37.088985+0000","last_active":"2024-04-20T17:52:48.762351+0000","last_peered":"2024-04-20T17:52:48.762351+0000","last_clean":"2024-04-20T17:52:48.762351+0000","last_became_active":"2024-04-20T17:48:54.813062+0000","last_became_peered":"2024-04-20T17:48:54.813062+0000","last_unstale":"2024-04-20T17:52:48.762351+0000","last_undegraded":"2024-04-20T17:52:48.762351+0000","last_fullsized":"2024-04-20T17:52:48.762351+0000","mapping_epoch":894,"log_start":"940'11700","ondisk_log_start":"940'11700","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"966'20945","last_scrub_stamp":"2024-04-20T17:52:37.088816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:52:37.088816+0000","objects_scrubbed":2368,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:53:44.685883+0000","scrub_duration":610,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98959360,"num_objects":1510,"num_object_clones":0,"num_object_copies":3020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19410,"num_write_kb":669440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"15'137","reported_seq":939,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-20T17:52:45.965888+0000","last_change":"2024-04-20T17:51:45.119466+0000","last_active":"2024-04-20T17:52:45.965888+0000","last_peered":"2024-04-20T17:52:45.965888+0000","last_clean":"2024-04-20T17:52:45.965888+0000","last_became_active":"2024-04-20T17:46:40.881713+0000","last_became_peered":"2024-04-20T17:46:40.881713+0000","last_unstale":"2024-04-20T17:52:45.965888+0000","last_undegraded":"2024-04-20T17:52:45.965888+0000","last_fullsized":"2024-04-20T17:52:45.965888+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:45.119332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:45.119332+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:51.157610+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.8","version":"970'19442","reported_seq":19775,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-20T17:52:47.219510+0000","last_change":"2024-04-20T17:51:43.840908+0000","last_active":"2024-04-20T17:52:47.219510+0000","last_peered":"2024-04-20T17:52:47.219510+0000","last_clean":"2024-04-20T17:52:47.219510+0000","last_became_active":"2024-04-20T17:48:54.813368+0000","last_became_peered":"2024-04-20T17:48:54.813368+0000","last_unstale":"2024-04-20T17:52:47.219510+0000","last_undegraded":"2024-04-20T17:52:47.219510+0000","last_fullsized":"2024-04-20T17:52:47.219510+0000","mapping_epoch":894,"log_start":"927'9400","ondisk_log_start":"927'9400","created":894,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'15051","last_scrub_stamp":"2024-04-20T17:51:43.840798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:48:53.794927+0000","last_clean_scrub_stamp":"2024-04-20T17:51:43.840798+0000","objects_scrubbed":6061,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:52.336868+0000","scrub_duration":1160,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106561536,"num_objects":1626,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19442,"num_write_kb":674176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":938,"reported_epoch":970,"state":"active+clean","last_fresh":"2024-04-20T17:52:45.965991+0000","last_change":"2024-04-20T17:51:48.160638+0000","last_active":"2024-04-20T17:52:45.965991+0000","last_peered":"2024-04-20T17:52:45.965991+0000","last_clean":"2024-04-20T17:52:45.965991+0000","last_became_active":"2024-04-20T17:46:40.851683+0000","last_became_peered":"2024-04-20T17:46:40.851683+0000","last_unstale":"2024-04-20T17:52:45.965991+0000","last_undegraded":"2024-04-20T17:52:45.965991+0000","last_fullsized":"2024-04-20T17:52:45.965991+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:51:48.160546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:51:48.160546+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:52:56.139128+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1604452376,"num_objects":24483,"num_object_clones":0,"num_object_copies":48966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24483,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":309105,"num_write_kb":10674753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12720,"num_bytes_recovered":833486848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3195150336,"data_stored":3195142192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":32,"num_store_stats":11},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},"log_size":1507,"ondisk_log_size":1507,"up":22,"acting":22,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991798,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1266996,"kb_used_data":337580,"kb_used_omap":1,"kb_used_meta":929406,"kb_avail":92478156,"statfs":{"total":95995035648,"available":94697631744,"internally_reserved":0,"allocated":345681920,"data_stored":345328020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":951712196},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253438,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1231232,"kb_used_data":387896,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":92513920,"statfs":{"total":95995035648,"available":94734254080,"internally_reserved":0,"allocated":397205504,"data_stored":396845558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":863566272},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613186,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":966532,"kb_used_data":213432,"kb_used_omap":1,"kb_used_meta":753086,"kb_avail":92778620,"statfs":{"total":95995035648,"available":95005306880,"internally_reserved":0,"allocated":218554368,"data_stored":218194422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":771160512},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809417,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":663744,"kb_used_data":203576,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93081408,"statfs":{"total":95995035648,"available":95315361792,"internally_reserved":0,"allocated":208461824,"data_stored":208101878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":799,"seq":3431678869607,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":457476,"kb_used_data":286780,"kb_used_omap":11,"kb_used_meta":170676,"kb_avail":93287676,"statfs":{"total":95995035648,"available":95526580224,"internally_reserved":0,"allocated":293662720,"data_stored":293298710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11990,"internal_metadata":174772522},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":535,"seq":2297807503646,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":176072,"kb_used_data":107960,"kb_used_omap":170,"kb_used_meta":67925,"kb_avail":93569080,"statfs":{"total":95995035648,"available":95814737920,"internally_reserved":0,"allocated":110551040,"data_stored":110191094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":174628,"internal_metadata":69555676},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":659,"seq":2830383448269,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":991748,"kb_used_data":599740,"kb_used_omap":4,"kb_used_meta":391931,"kb_avail":92753404,"statfs":{"total":95995035648,"available":94979485696,"internally_reserved":0,"allocated":614133760,"data_stored":613769742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4410,"internal_metadata":401338054},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738991,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":744060,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":93001092,"statfs":{"total":95995035648,"available":95233118208,"internally_reserved":0,"allocated":700416,"data_stored":346516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394280,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":598204,"kb_used_data":156916,"kb_used_omap":202,"kb_used_meta":441077,"kb_avail":93146948,"statfs":{"total":95995035648,"available":95382474752,"internally_reserved":0,"allocated":160681984,"data_stored":160324532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":451663319},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653330,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1016384,"kb_used_data":402296,"kb_used_omap":4,"kb_used_meta":614075,"kb_avail":92728768,"statfs":{"total":95995035648,"available":94954258432,"internally_reserved":0,"allocated":411951104,"data_stored":411591158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":628813511},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986779,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":810880,"kb_used_data":184120,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":92934272,"statfs":{"total":95995035648,"available":95164694528,"internally_reserved":0,"allocated":188538880,"data_stored":188178934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790549,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1303688,"kb_used_data":410368,"kb_used_omap":1,"kb_used_meta":893310,"kb_avail":92441464,"statfs":{"total":95995035648,"available":94660059136,"internally_reserved":0,"allocated":420216832,"data_stored":419853334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":914749893},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":596381696,"data_stored":596377624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110493696,"data_stored":110493696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":401866752,"data_stored":401866752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":183631872,"data_stored":183631872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":371392512,"data_stored":371392512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":108724224,"data_stored":108724224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":286920704,"data_stored":286916632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":204996608,"data_stored":204996608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":212926464,"data_stored":212926464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":382337024,"data_stored":382337024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":335478784,"data_stored":335478784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:52:54.055 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-20T17:52:54.114 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:52:54.145 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:54.183 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:52:54.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:54.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:52:54.290 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:52:54.330 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:52:54.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:54 smithi171 ceph-mon[44189]: 9.8 scrub ok 2024-04-20T17:52:54.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:54 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:52:54.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:54 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:52:54.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:54 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:52:54.343 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:54.367 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:52:54.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:54 smithi005 ceph-mon[39562]: 9.8 scrub ok 2024-04-20T17:52:54.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:54 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:52:54.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:54 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:52:54.368 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:54 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:52:54.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:54.371 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:52:54.472 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:52:54.505 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:54.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:54 smithi106 ceph-mon[44423]: 9.8 scrub ok 2024-04-20T17:52:54.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:54 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:52:54.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:54 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:52:54.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:54 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:52:54.526 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:52:54.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:54.530 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:52:54.631 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:52:54.660 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:54.685 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:52:54.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:54.692 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:52:54.793 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:52:54.841 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:55.069 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-20T17:52:55.283 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:55 smithi005 ceph-mon[39562]: pgmap v2565: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 945 op/s 2024-04-20T17:52:55.313 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:52:55.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:55.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:52:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:55 smithi171 ceph-mon[44189]: pgmap v2565: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 945 op/s 2024-04-20T17:52:55.417 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:52:55.453 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:52:55.454 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:55.482 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:52:55.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:55.487 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:52:55.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:55 smithi106 ceph-mon[44423]: pgmap v2565: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 945 op/s 2024-04-20T17:52:55.588 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:52:55.637 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:52:55.696 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:55.965 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:52:55.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:55.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:52:56.076 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-04-20T17:52:56.078 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:52:56.114 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:56.146 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:52:56.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:56.153 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:52:56.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:56 smithi005 ceph-mon[39562]: 1.9 scrub starts 2024-04-20T17:52:56.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:56 smithi005 ceph-mon[39562]: 1.9 scrub ok 2024-04-20T17:52:56.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:56 smithi005 ceph-mon[39562]: 9.6 deep-scrub ok 2024-04-20T17:52:56.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:56 smithi005 ceph-mon[39562]: 1.a scrub starts 2024-04-20T17:52:56.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:56 smithi005 ceph-mon[39562]: 1.a scrub ok 2024-04-20T17:52:56.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:56 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/477914046' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:52:56.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:56 smithi005 ceph-mon[39562]: from='client.28021 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:52:56.254 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:52:56.294 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:56.315 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:52:56.317 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:56 smithi171 ceph-mon[44189]: 1.9 scrub starts 2024-04-20T17:52:56.317 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:56 smithi171 ceph-mon[44189]: 1.9 scrub ok 2024-04-20T17:52:56.317 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:56 smithi171 ceph-mon[44189]: 9.6 deep-scrub ok 2024-04-20T17:52:56.317 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:56 smithi171 ceph-mon[44189]: 1.a scrub starts 2024-04-20T17:52:56.317 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:56 smithi171 ceph-mon[44189]: 1.a scrub ok 2024-04-20T17:52:56.318 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:56 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/477914046' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:52:56.318 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:56 smithi171 ceph-mon[44189]: from='client.28021 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:52:56.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:56.319 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:52:56.420 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:52:56.453 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:56.479 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:52:56.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:56.486 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:52:56.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:56 smithi106 ceph-mon[44423]: 1.9 scrub starts 2024-04-20T17:52:56.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:56 smithi106 ceph-mon[44423]: 1.9 scrub ok 2024-04-20T17:52:56.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:56 smithi106 ceph-mon[44423]: 9.6 deep-scrub ok 2024-04-20T17:52:56.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:56 smithi106 ceph-mon[44423]: 1.a scrub starts 2024-04-20T17:52:56.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:56 smithi106 ceph-mon[44423]: 1.a scrub ok 2024-04-20T17:52:56.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:56 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/477914046' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:52:56.512 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:56 smithi106 ceph-mon[44423]: from='client.28021 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-20T17:52:56.587 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:52:56.623 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:56.651 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:52:56.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:56.655 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:52:56.756 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:52:56.786 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:56.806 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:52:56.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:56.811 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:52:56.912 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:52:56.950 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:57.050 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:57 smithi171 ceph-mon[44189]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:52:57.050 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:57 smithi171 ceph-mon[44189]: from='client.28021 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:52:57.051 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:57 smithi171 ceph-mon[44189]: osdmap e972: 12 total, 12 up, 12 in 2024-04-20T17:52:57.051 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:57 smithi171 ceph-mon[44189]: pgmap v2567: 27 pgs: 27 active+clean; 1.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s 2024-04-20T17:52:57.204 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:52:57.205 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:57 smithi005 ceph-mon[39562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:52:57.205 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:57 smithi005 ceph-mon[39562]: from='client.28021 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:52:57.206 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:57 smithi005 ceph-mon[39562]: osdmap e972: 12 total, 12 up, 12 in 2024-04-20T17:52:57.206 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:57 smithi005 ceph-mon[39562]: pgmap v2567: 27 pgs: 27 active+clean; 1.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s 2024-04-20T17:52:57.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:57.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:52:57.315 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:52:57.354 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:57.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:57 smithi106 ceph-mon[44423]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:52:57.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:57 smithi106 ceph-mon[44423]: from='client.28021 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-20T17:52:57.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:57 smithi106 ceph-mon[44423]: osdmap e972: 12 total, 12 up, 12 in 2024-04-20T17:52:57.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:57 smithi106 ceph-mon[44423]: pgmap v2567: 27 pgs: 27 active+clean; 1.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.22k op/s 2024-04-20T17:52:57.710 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:52:57.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:57.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:52:57.816 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:52:57.860 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:57.979 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:52:57.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:57.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:52:58.087 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:52:58.138 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:58.279 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:52:58.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:58.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:52:58.326 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:52:58.353 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:52:58.378 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:52:58.383 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:52:58.410 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:52:58.435 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:58.439 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:52:58.463 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:52:58.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:58.471 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:52:58.472 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:52:58.509 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:52:58.538 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:52:58.566 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:52:58.571 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:52:58.603 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:52:58.615 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:58 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/480612874' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:52:58.621 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:58.629 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:52:58.653 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:52:58.658 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:52:58.658 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:52:58.658 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:52:58.659 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:52:58.659 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:52:58.659 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:52:58.659 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:52:58.659 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:52:58.659 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:52:58.659 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:52:58.659 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:52:58.659 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:52:58.685 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:58 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/480612874' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:52:58.837 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:58 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/480612874' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:52:59.247 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-20T17:52:59.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:59 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:52:59 smithi005 ceph-mon[39562]: pgmap v2568: 27 pgs: 27 active+clean; 1.1 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s 2024-04-20T17:52:59.483 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:52:59.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:59.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:52:59.593 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:52:59.650 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:52:59.651 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:59.680 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:52:59.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:59.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:52:59.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.763 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:59 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.764 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:52:59 smithi106 ceph-mon[44423]: pgmap v2568: 27 pgs: 27 active+clean; 1.1 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s 2024-04-20T17:52:59.785 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:52:59.830 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.830 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.830 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.830 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.830 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.831 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.831 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.831 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.831 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.831 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:59 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:52:59.831 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:52:59 smithi171 ceph-mon[44189]: pgmap v2568: 27 pgs: 27 active+clean; 1.1 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s 2024-04-20T17:52:59.837 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:52:59.866 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:52:59.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:52:59.871 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:52:59.972 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:53:00.015 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:00.038 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:53:00.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:00.042 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:53:00.143 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:53:00.190 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:00.210 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:53:00.210 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:53:00.214 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:53:00.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:00.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:53:00.324 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:53:00.361 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:00.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:00 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:00.518 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:53:00.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:00.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:53:00.630 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:53:00.656 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:00.656 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:00 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:00.658 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:00.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:00.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:00 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:00.685 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:53:00.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:00.693 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:53:00.794 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:53:00.842 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:01.041 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-20T17:53:01.056 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:53:01.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:01.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:53:01.160 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:53:01.210 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:01.237 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:53:01.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:01.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:53:01.349 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:53:01.393 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:01.408 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:53:01.417 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:53:01.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:01.423 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:53:01.524 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:53:01.561 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T17:53:01.584 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:01.625 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:53:01.629 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:01 smithi005 ceph-mon[39562]: pgmap v2569: 27 pgs: 27 active+clean; 875 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-04-20T17:53:01.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:01.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:53:01.723 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:01 smithi106 ceph-mon[44423]: pgmap v2569: 27 pgs: 27 active+clean; 875 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-04-20T17:53:01.732 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:53:01.788 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:01.827 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:53:01.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:01.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:53:01.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:01 smithi171 ceph-mon[44189]: pgmap v2569: 27 pgs: 27 active+clean; 875 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-04-20T17:53:01.931 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:53:01.989 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:02.019 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:53:02.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:02.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:53:02.042 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:53:02.122 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:53:02.153 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:02.184 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:53:02.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:02.196 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:53:02.297 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:53:02.362 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:02.394 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:53:02.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:02.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:53:02.498 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:53:02.539 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:02.562 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:53:02.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:02.571 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:53:02.672 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:53:02.724 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:02.873 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-04-20T17:53:02.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:02 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/541618483' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:53:02.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:02 smithi005 ceph-mon[39562]: from='client.28033 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:53:03.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:02 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/541618483' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:53:03.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:02 smithi106 ceph-mon[44423]: from='client.28033 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:53:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:02 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/541618483' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:53:03.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:02 smithi171 ceph-mon[44189]: from='client.28033 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:53:03.657 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:53:03.657 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:53:03.888 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:53:03.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:03.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:53:04.007 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:53:04.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:03 smithi106 ceph-mon[44423]: pgmap v2570: 27 pgs: 27 active+clean; 762 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-04-20T17:53:04.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:03 smithi106 ceph-mon[44423]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:53:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:03 smithi106 ceph-mon[44423]: from='client.28033 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:53:04.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:03 smithi106 ceph-mon[44423]: osdmap e973: 12 total, 12 up, 12 in 2024-04-20T17:53:04.049 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:03 smithi005 ceph-mon[39562]: pgmap v2570: 27 pgs: 27 active+clean; 762 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-04-20T17:53:04.049 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:03 smithi005 ceph-mon[39562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:53:04.049 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:03 smithi005 ceph-mon[39562]: from='client.28033 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:53:04.049 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:03 smithi005 ceph-mon[39562]: osdmap e973: 12 total, 12 up, 12 in 2024-04-20T17:53:04.053 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:04.213 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:53:04.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:04.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:53:04.318 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:53:04.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:03 smithi171 ceph-mon[44189]: pgmap v2570: 27 pgs: 27 active+clean; 762 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-04-20T17:53:04.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:03 smithi171 ceph-mon[44189]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:53:04.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:03 smithi171 ceph-mon[44189]: from='client.28033 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-20T17:53:04.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:03 smithi171 ceph-mon[44189]: osdmap e973: 12 total, 12 up, 12 in 2024-04-20T17:53:04.381 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:04.533 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:53:04.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:04.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:53:04.542 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-20T17:53:04.638 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:53:04.687 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:04.716 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:53:04.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:04.723 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:53:04.824 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:53:04.884 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:04.917 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:53:04.918 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:53:04.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:04.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:53:04.971 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:53:05.002 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:53:05.021 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:53:05.029 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:53:05.057 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:53:05.057 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:05.084 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:53:05.085 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:53:05.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:05.088 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:53:05.114 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:53:05.146 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:53:05.183 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:53:05.188 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:53:05.211 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:53:05.233 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:05.238 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:53:05.262 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:53:05.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:05.266 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:53:05.279 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:53:05.307 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:53:05.313 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:53:05.313 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:53:05.313 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:53:05.313 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:53:05.313 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:53:05.313 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:53:05.313 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:53:05.314 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:53:05.314 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:53:05.314 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:53:05.314 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:53:05.314 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:53:05.367 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:53:05.404 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:05.438 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:53:05.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:05.443 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:53:05.544 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:53:05.584 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:05.762 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:53:05.844 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:53:05.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:05.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:53:05.951 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:53:05.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:05 smithi005 ceph-mon[39562]: pgmap v2572: 27 pgs: 27 active+clean; 642 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-04-20T17:53:05.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:05 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/982866150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:53:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:05.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:05 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:06.000 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:06.066 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:53:06.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:06.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:53:06.174 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:53:06.230 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:06.234 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:53:06.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:05 smithi106 ceph-mon[44423]: pgmap v2572: 27 pgs: 27 active+clean; 642 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-04-20T17:53:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:05 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/982866150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:53:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:05 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:06.263 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:53:06.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:06.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:53:06.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:05 smithi171 ceph-mon[44189]: pgmap v2572: 27 pgs: 27 active+clean; 642 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-04-20T17:53:06.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:05 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/982866150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:53:06.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:06.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:06.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:06.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:06.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:06.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:06.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:05 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:06.368 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:53:06.397 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:06.433 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:53:06.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:06.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:53:06.538 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4 2024-04-20T17:53:06.573 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:06.833 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4" found: no such container 2024-04-20T17:53:06.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:06.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.4' 2024-04-20T17:53:06.937 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:53:07.010 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:07.034 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:53:07.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:07.039 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:53:07.140 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8 2024-04-20T17:53:07.173 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-04-20T17:53:07.173 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-04-20T17:53:07.173 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-04-20T17:53:07.173 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-04-20T17:53:07.173 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.173 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27772.0:118065 9.7 e922/894 9:ecbd09a5:::benchmark_data_smithi171_188319_object118064:head v 922'7307, mlcod=922'7303)", 2024-04-20T17:53:07.173 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:56.619678+0000", 2024-04-20T17:53:07.173 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 190.55173342699999, 2024-04-20T17:53:07.174 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.052183737000000001, 2024-04-20T17:53:07.174 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:53:07.174 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:53:07.174 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:53:07.174 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:53:07.174 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.174 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:53:07.174 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.619678+0000", 2024-04-20T17:53:07.174 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.174 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.174 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.174 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:53:07.174 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.619678+0000", 2024-04-20T17:53:07.174 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.175 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.175 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.175 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:53:07.175 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.619683+0000", 2024-04-20T17:53:07.175 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8879999999999996e-06 2024-04-20T17:53:07.175 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.175 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.175 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:53:07.175 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.619739+0000", 2024-04-20T17:53:07.175 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5387999999999999e-05 2024-04-20T17:53:07.175 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.175 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.175 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:53:07.175 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.619747+0000", 2024-04-20T17:53:07.176 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0679999999999997e-06 2024-04-20T17:53:07.176 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.176 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.176 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:53:07.176 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.619752+0000", 2024-04-20T17:53:07.176 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2229999999999998e-06 2024-04-20T17:53:07.176 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.176 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.176 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:53:07.176 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.662828+0000", 2024-04-20T17:53:07.176 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.043076345000000002 2024-04-20T17:53:07.176 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.176 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.176 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:53:07.177 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.662857+0000", 2024-04-20T17:53:07.177 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9128999999999998e-05 2024-04-20T17:53:07.177 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.177 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.177 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:53:07.177 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.671813+0000", 2024-04-20T17:53:07.177 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0089560160000000007 2024-04-20T17:53:07.177 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.177 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.177 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:53:07.177 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.671862+0000", 2024-04-20T17:53:07.177 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8680000000000001e-05 2024-04-20T17:53:07.177 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.177 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:53:07.177 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.178 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.178 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.178 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:118067 9.e 9:75df710d:::benchmark_data_smithi171_188319_object118066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e922)", 2024-04-20T17:53:07.178 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:56.620515+0000", 2024-04-20T17:53:07.178 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 190.55089688199999, 2024-04-20T17:53:07.178 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.046983236999999997, 2024-04-20T17:53:07.178 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:53:07.178 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:53:07.178 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:53:07.178 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:53:07.178 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:53:07.178 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:53:07.178 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 118067 2024-04-20T17:53:07.178 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.179 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:53:07.179 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.179 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:53:07.179 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.620515+0000", 2024-04-20T17:53:07.179 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.179 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.179 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.179 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:53:07.179 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.620515+0000", 2024-04-20T17:53:07.179 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.179 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.179 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.179 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:53:07.179 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.620517+0000", 2024-04-20T17:53:07.179 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0999999999999998e-06 2024-04-20T17:53:07.180 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.180 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.180 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:53:07.180 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.620611+0000", 2024-04-20T17:53:07.180 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3918000000000004e-05 2024-04-20T17:53:07.180 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.180 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.180 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:53:07.180 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.662793+0000", 2024-04-20T17:53:07.180 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.042182246 2024-04-20T17:53:07.180 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.180 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.180 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:53:07.180 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.662809+0000", 2024-04-20T17:53:07.180 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6299000000000001e-05 2024-04-20T17:53:07.181 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.181 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.181 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:53:07.181 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.662914+0000", 2024-04-20T17:53:07.181 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000104623 2024-04-20T17:53:07.181 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.181 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.181 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:53:07.181 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.663040+0000", 2024-04-20T17:53:07.181 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012611799999999999 2024-04-20T17:53:07.181 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.181 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.181 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:53:07.181 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.663165+0000", 2024-04-20T17:53:07.181 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000124664 2024-04-20T17:53:07.182 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.182 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.182 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:53:07.182 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.666382+0000", 2024-04-20T17:53:07.182 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0032171190000000001 2024-04-20T17:53:07.182 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.182 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.182 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:53:07.182 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.667435+0000", 2024-04-20T17:53:07.182 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010530579999999999 2024-04-20T17:53:07.182 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.182 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.182 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:53:07.182 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.667469+0000", 2024-04-20T17:53:07.182 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4017000000000003e-05 2024-04-20T17:53:07.183 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.183 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.183 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:53:07.183 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.667498+0000", 2024-04-20T17:53:07.183 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9074999999999999e-05 2024-04-20T17:53:07.183 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.183 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:53:07.183 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.183 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.183 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.183 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27772.0:118087 9.7 e922/894 9:e5198507:::benchmark_data_smithi171_188319_object118086:head v 922'7308, mlcod=922'7305)", 2024-04-20T17:53:07.183 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:49:56.631843+0000", 2024-04-20T17:53:07.184 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 190.53956888499999, 2024-04-20T17:53:07.184 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.043135953999999997, 2024-04-20T17:53:07.184 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:53:07.184 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:53:07.184 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:53:07.184 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:53:07.184 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.184 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:53:07.184 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.631843+0000", 2024-04-20T17:53:07.184 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.184 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.184 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.184 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:53:07.184 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.631843+0000", 2024-04-20T17:53:07.184 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.185 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.185 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.185 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:53:07.185 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.631845+0000", 2024-04-20T17:53:07.185 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0779999999999998e-06 2024-04-20T17:53:07.185 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.185 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.185 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:53:07.185 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.631960+0000", 2024-04-20T17:53:07.185 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011519900000000001 2024-04-20T17:53:07.185 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.185 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.185 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:53:07.185 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.662794+0000", 2024-04-20T17:53:07.185 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.030834318999999999 2024-04-20T17:53:07.185 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.186 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.186 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:53:07.186 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.662808+0000", 2024-04-20T17:53:07.186 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3200000000000001e-05 2024-04-20T17:53:07.186 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.186 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.186 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:53:07.186 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.665374+0000", 2024-04-20T17:53:07.186 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0025666399999999998 2024-04-20T17:53:07.186 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.186 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.186 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:53:07.186 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.665389+0000", 2024-04-20T17:53:07.186 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5174000000000001e-05 2024-04-20T17:53:07.187 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.187 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.187 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:53:07.187 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.674930+0000", 2024-04-20T17:53:07.187 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0095409280000000006 2024-04-20T17:53:07.187 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.187 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.187 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:53:07.187 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:49:56.674979+0000", 2024-04-20T17:53:07.187 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8415999999999999e-05 2024-04-20T17:53:07.187 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.187 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:53:07.187 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.187 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.187 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.188 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:144399 9.d 9:bff19b0f:::benchmark_data_smithi171_188319_object144398:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e926)", 2024-04-20T17:53:07.188 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:50:16.021298+0000", 2024-04-20T17:53:07.188 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 171.15011393899999, 2024-04-20T17:53:07.188 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.045278917000000002, 2024-04-20T17:53:07.188 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:53:07.188 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:53:07.188 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:53:07.188 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:53:07.188 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:53:07.188 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:53:07.188 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 144399 2024-04-20T17:53:07.188 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.188 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:53:07.188 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.188 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:53:07.189 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:16.021298+0000", 2024-04-20T17:53:07.189 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.189 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.189 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.189 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:53:07.189 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:16.021298+0000", 2024-04-20T17:53:07.189 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.189 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.189 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.189 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:53:07.189 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:16.021300+0000", 2024-04-20T17:53:07.189 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8190000000000001e-06 2024-04-20T17:53:07.189 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.189 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.189 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:53:07.190 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:16.021354+0000", 2024-04-20T17:53:07.190 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4688000000000002e-05 2024-04-20T17:53:07.190 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.190 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.190 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:53:07.190 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:16.021370+0000", 2024-04-20T17:53:07.190 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5350999999999999e-05 2024-04-20T17:53:07.190 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.190 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.190 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:53:07.190 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:16.021378+0000", 2024-04-20T17:53:07.190 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2509999999999994e-06 2024-04-20T17:53:07.190 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.190 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.191 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:53:07.191 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:16.025321+0000", 2024-04-20T17:53:07.191 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0039433360000000004 2024-04-20T17:53:07.191 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.191 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.191 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:53:07.191 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:16.025406+0000", 2024-04-20T17:53:07.191 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4925000000000004e-05 2024-04-20T17:53:07.191 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.191 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.191 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:53:07.191 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:16.025493+0000", 2024-04-20T17:53:07.191 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6977000000000004e-05 2024-04-20T17:53:07.191 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.191 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.192 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:53:07.192 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:16.029871+0000", 2024-04-20T17:53:07.192 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0043775960000000001 2024-04-20T17:53:07.192 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.192 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.192 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:53:07.192 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:16.066471+0000", 2024-04-20T17:53:07.192 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.036600004999999998 2024-04-20T17:53:07.192 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.192 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.192 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:53:07.192 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:16.066533+0000", 2024-04-20T17:53:07.192 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2118000000000004e-05 2024-04-20T17:53:07.192 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.193 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.193 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:53:07.193 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:16.066577+0000", 2024-04-20T17:53:07.193 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3850999999999997e-05 2024-04-20T17:53:07.193 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.193 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:53:07.193 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.193 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.193 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.193 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:147827 9.e 9:7d0ef956:::benchmark_data_smithi171_188319_object147826:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e927)", 2024-04-20T17:53:07.193 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:50:19.124821+0000", 2024-04-20T17:53:07.193 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 168.04659037799999, 2024-04-20T17:53:07.193 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.046903447000000001, 2024-04-20T17:53:07.193 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:53:07.193 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:53:07.193 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:53:07.194 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:53:07.194 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:53:07.194 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:53:07.194 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 147827 2024-04-20T17:53:07.194 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.194 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:53:07.194 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.194 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:53:07.194 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:19.124821+0000", 2024-04-20T17:53:07.194 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.194 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.194 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.194 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:53:07.194 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:19.124821+0000", 2024-04-20T17:53:07.195 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.195 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:53:07.195 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:19.124823+0000", 2024-04-20T17:53:07.195 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.251e-06 2024-04-20T17:53:07.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.195 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:53:07.195 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:19.124898+0000", 2024-04-20T17:53:07.195 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5257000000000001e-05 2024-04-20T17:53:07.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.195 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:53:07.195 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:19.124909+0000", 2024-04-20T17:53:07.195 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0753e-05 2024-04-20T17:53:07.196 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.196 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.196 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:53:07.196 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:19.124916+0000", 2024-04-20T17:53:07.196 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1919999999999999e-06 2024-04-20T17:53:07.196 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.196 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.196 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:53:07.196 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:19.124971+0000", 2024-04-20T17:53:07.196 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5223000000000003e-05 2024-04-20T17:53:07.196 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.196 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.196 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:53:07.196 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:19.125073+0000", 2024-04-20T17:53:07.196 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010215100000000001 2024-04-20T17:53:07.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.197 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:53:07.197 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:19.125163+0000", 2024-04-20T17:53:07.197 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0084e-05 2024-04-20T17:53:07.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.197 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:53:07.197 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:19.129393+0000", 2024-04-20T17:53:07.197 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0042297480000000002 2024-04-20T17:53:07.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.197 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:53:07.197 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:19.171661+0000", 2024-04-20T17:53:07.197 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.042268 2024-04-20T17:53:07.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.198 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:53:07.198 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:19.171696+0000", 2024-04-20T17:53:07.198 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4703000000000003e-05 2024-04-20T17:53:07.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.198 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:53:07.198 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:19.171725+0000", 2024-04-20T17:53:07.198 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9085e-05 2024-04-20T17:53:07.198 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.199 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:53:07.199 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.199 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.199 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.199 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:147840 9.e 9:70a3f37f:::benchmark_data_smithi171_188319_object147839:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e927)", 2024-04-20T17:53:07.199 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:50:19.135123+0000", 2024-04-20T17:53:07.199 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 168.036289106, 2024-04-20T17:53:07.199 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.050594411999999998, 2024-04-20T17:53:07.199 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:53:07.199 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:53:07.199 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:53:07.199 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:53:07.199 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:53:07.199 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:53:07.200 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 147840 2024-04-20T17:53:07.200 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.200 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:53:07.200 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.200 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:53:07.200 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:19.135123+0000", 2024-04-20T17:53:07.200 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.200 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.200 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.200 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:53:07.200 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:19.135123+0000", 2024-04-20T17:53:07.200 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.200 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.200 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.201 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:53:07.201 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:19.135124+0000", 2024-04-20T17:53:07.201 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7e-06 2024-04-20T17:53:07.201 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.201 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.201 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:53:07.201 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:19.135204+0000", 2024-04-20T17:53:07.201 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9518000000000007e-05 2024-04-20T17:53:07.201 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.201 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.201 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:53:07.201 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:19.135216+0000", 2024-04-20T17:53:07.201 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2186e-05 2024-04-20T17:53:07.201 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.201 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.202 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:53:07.202 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:19.135225+0000", 2024-04-20T17:53:07.202 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4800000000000001e-06 2024-04-20T17:53:07.202 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.202 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.202 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:53:07.202 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:19.135282+0000", 2024-04-20T17:53:07.202 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7182999999999998e-05 2024-04-20T17:53:07.202 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.202 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.202 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:53:07.202 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:19.135379+0000", 2024-04-20T17:53:07.202 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7003000000000006e-05 2024-04-20T17:53:07.202 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.203 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:53:07.203 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:19.135468+0000", 2024-04-20T17:53:07.203 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9636999999999996e-05 2024-04-20T17:53:07.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.203 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:53:07.203 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:19.139033+0000", 2024-04-20T17:53:07.203 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0035648479999999998 2024-04-20T17:53:07.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.203 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:53:07.203 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:19.185631+0000", 2024-04-20T17:53:07.203 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.046597434 2024-04-20T17:53:07.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.204 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:53:07.204 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:19.185672+0000", 2024-04-20T17:53:07.204 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.108e-05 2024-04-20T17:53:07.204 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.204 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:53:07.204 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:19.185717+0000", 2024-04-20T17:53:07.204 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5343000000000003e-05 2024-04-20T17:53:07.204 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.204 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:53:07.204 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.204 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.205 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:157052 9.e 9:7743b8ad:::benchmark_data_smithi171_188319_object157051:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e930)", 2024-04-20T17:53:07.205 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:50:27.970822+0000", 2024-04-20T17:53:07.205 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 159.200590229, 2024-04-20T17:53:07.205 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12609057000000001, 2024-04-20T17:53:07.205 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:53:07.205 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:53:07.205 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:53:07.205 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:53:07.205 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:53:07.205 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:53:07.205 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 157052 2024-04-20T17:53:07.205 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.205 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:53:07.205 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.205 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:53:07.206 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:27.970822+0000", 2024-04-20T17:53:07.206 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.206 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.206 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.206 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:53:07.206 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:27.970822+0000", 2024-04-20T17:53:07.206 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.206 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.206 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.206 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:53:07.206 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:27.970824+0000", 2024-04-20T17:53:07.206 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.644e-06 2024-04-20T17:53:07.206 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.206 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.207 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:53:07.207 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:27.970875+0000", 2024-04-20T17:53:07.207 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0510999999999998e-05 2024-04-20T17:53:07.207 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.207 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.207 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:53:07.207 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:27.970887+0000", 2024-04-20T17:53:07.207 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1929e-05 2024-04-20T17:53:07.207 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.207 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.207 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:53:07.207 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:27.970893+0000", 2024-04-20T17:53:07.207 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1879999999999999e-06 2024-04-20T17:53:07.207 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.207 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.208 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:53:07.208 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:27.970954+0000", 2024-04-20T17:53:07.208 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1278000000000005e-05 2024-04-20T17:53:07.208 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.208 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.208 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:53:07.208 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:27.971064+0000", 2024-04-20T17:53:07.208 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011010000000000001 2024-04-20T17:53:07.208 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.208 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.208 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:53:07.208 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:27.971161+0000", 2024-04-20T17:53:07.208 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7188000000000005e-05 2024-04-20T17:53:07.208 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.208 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.209 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:53:07.209 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:27.974693+0000", 2024-04-20T17:53:07.209 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0035318939999999998 2024-04-20T17:53:07.209 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.209 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.210 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:53:07.210 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:28.096844+0000", 2024-04-20T17:53:07.210 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12215093 2024-04-20T17:53:07.210 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.210 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:53:07.210 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:28.096881+0000", 2024-04-20T17:53:07.210 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6661e-05 2024-04-20T17:53:07.210 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.210 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:53:07.210 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:28.096912+0000", 2024-04-20T17:53:07.210 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1247000000000001e-05 2024-04-20T17:53:07.210 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.211 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:53:07.211 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.211 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.211 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.211 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:157063 9.e 9:74290346:::benchmark_data_smithi171_188319_object157062:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e930)", 2024-04-20T17:53:07.211 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:50:27.980411+0000", 2024-04-20T17:53:07.211 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 159.191001223, 2024-04-20T17:53:07.211 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.116593005, 2024-04-20T17:53:07.211 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:53:07.211 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:53:07.211 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:53:07.211 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:53:07.211 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:53:07.211 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:53:07.211 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 157063 2024-04-20T17:53:07.212 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.212 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:53:07.212 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.212 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:53:07.212 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:27.980411+0000", 2024-04-20T17:53:07.212 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.212 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.212 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.212 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:53:07.212 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:27.980411+0000", 2024-04-20T17:53:07.212 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.212 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.212 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.212 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:53:07.212 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:27.980417+0000", 2024-04-20T17:53:07.213 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9750000000000001e-06 2024-04-20T17:53:07.213 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.213 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.213 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:53:07.213 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:27.980474+0000", 2024-04-20T17:53:07.213 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6390000000000001e-05 2024-04-20T17:53:07.213 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.213 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.213 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:53:07.213 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:27.980483+0000", 2024-04-20T17:53:07.213 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2089999999999994e-06 2024-04-20T17:53:07.213 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.213 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.213 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:53:07.214 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:27.980489+0000", 2024-04-20T17:53:07.214 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4929999999999998e-06 2024-04-20T17:53:07.214 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.214 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.214 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:53:07.214 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:27.980537+0000", 2024-04-20T17:53:07.214 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8807000000000002e-05 2024-04-20T17:53:07.214 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.214 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.214 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:53:07.214 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:27.980663+0000", 2024-04-20T17:53:07.214 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012553 2024-04-20T17:53:07.214 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.214 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.215 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:53:07.215 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:27.980804+0000", 2024-04-20T17:53:07.215 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000141268 2024-04-20T17:53:07.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.215 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.215 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:53:07.215 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:27.984088+0000", 2024-04-20T17:53:07.215 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003284066 2024-04-20T17:53:07.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.215 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.215 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:53:07.215 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:28.096951+0000", 2024-04-20T17:53:07.215 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.112862595 2024-04-20T17:53:07.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.216 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.216 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:53:07.216 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:28.096981+0000", 2024-04-20T17:53:07.216 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0046999999999999e-05 2024-04-20T17:53:07.216 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.216 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.216 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:53:07.216 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:28.097004+0000", 2024-04-20T17:53:07.216 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2625000000000001e-05 2024-04-20T17:53:07.216 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.216 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:53:07.216 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.216 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.216 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.216 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:164193 9.e 9:785d8eed:::benchmark_data_smithi171_188319_object164192:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e931)", 2024-04-20T17:53:07.217 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:50:34.442805+0000", 2024-04-20T17:53:07.217 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 152.72860644100001, 2024-04-20T17:53:07.217 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.044316842000000002, 2024-04-20T17:53:07.217 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:53:07.217 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:53:07.217 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:53:07.217 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:53:07.217 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:53:07.217 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:53:07.217 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 164193 2024-04-20T17:53:07.217 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.217 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:53:07.217 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.217 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:53:07.218 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:34.442805+0000", 2024-04-20T17:53:07.218 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.218 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.218 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:53:07.218 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:34.442805+0000", 2024-04-20T17:53:07.218 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.218 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.219 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:53:07.219 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:34.442808+0000", 2024-04-20T17:53:07.219 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.508e-06 2024-04-20T17:53:07.219 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.219 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.219 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:53:07.219 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:34.442842+0000", 2024-04-20T17:53:07.219 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4607999999999998e-05 2024-04-20T17:53:07.219 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.219 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.220 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:53:07.220 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:34.442857+0000", 2024-04-20T17:53:07.220 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5007999999999999e-05 2024-04-20T17:53:07.220 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.220 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.221 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:53:07.221 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:34.442865+0000", 2024-04-20T17:53:07.221 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8220000000000007e-06 2024-04-20T17:53:07.221 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.221 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.221 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:53:07.222 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:34.444948+0000", 2024-04-20T17:53:07.222 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020831170000000002 2024-04-20T17:53:07.222 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.222 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.222 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:53:07.222 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:34.445088+0000", 2024-04-20T17:53:07.222 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013993799999999999 2024-04-20T17:53:07.222 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.223 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:53:07.223 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:34.445253+0000", 2024-04-20T17:53:07.223 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000164282 2024-04-20T17:53:07.223 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.223 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:53:07.223 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:34.450493+0000", 2024-04-20T17:53:07.223 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0052400490000000001 2024-04-20T17:53:07.224 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.224 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.224 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:53:07.224 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:34.487016+0000", 2024-04-20T17:53:07.224 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.036523844999999999 2024-04-20T17:53:07.224 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.224 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.224 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:53:07.224 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:34.487071+0000", 2024-04-20T17:53:07.224 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4626000000000002e-05 2024-04-20T17:53:07.225 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.225 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.225 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:53:07.225 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:34.487122+0000", 2024-04-20T17:53:07.225 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1038999999999997e-05 2024-04-20T17:53:07.225 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.225 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:53:07.225 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.225 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.225 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.226 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:181625 9.e 9:7e0bfd5d:::benchmark_data_smithi171_188319_object14829:head [delete] snapc 0=[] ondisk+write+known_if_redirected e937)", 2024-04-20T17:53:07.226 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:50:52.156247+0000", 2024-04-20T17:53:07.226 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 135.01516438300001, 2024-04-20T17:53:07.226 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.046369387999999997, 2024-04-20T17:53:07.226 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:53:07.226 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:53:07.226 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:53:07.226 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:53:07.226 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:53:07.226 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:53:07.226 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 181625 2024-04-20T17:53:07.227 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.227 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:53:07.227 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.227 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:53:07.227 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:52.156247+0000", 2024-04-20T17:53:07.227 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.227 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.227 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.227 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:53:07.227 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:52.156247+0000", 2024-04-20T17:53:07.227 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.227 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.228 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.228 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:53:07.228 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:52.156249+0000", 2024-04-20T17:53:07.228 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4339999999999999e-06 2024-04-20T17:53:07.228 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.228 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.228 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:53:07.228 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:52.156254+0000", 2024-04-20T17:53:07.228 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2440000000000001e-06 2024-04-20T17:53:07.228 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.228 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.228 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:53:07.229 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:52.156265+0000", 2024-04-20T17:53:07.229 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1277999999999999e-05 2024-04-20T17:53:07.229 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.229 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.229 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:53:07.229 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:52.156273+0000", 2024-04-20T17:53:07.229 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3509999999999996e-06 2024-04-20T17:53:07.229 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.229 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.229 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:53:07.229 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:52.156328+0000", 2024-04-20T17:53:07.229 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5049000000000001e-05 2024-04-20T17:53:07.229 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.230 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:53:07.230 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:52.156432+0000", 2024-04-20T17:53:07.230 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000104301 2024-04-20T17:53:07.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.230 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:53:07.230 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:52.156533+0000", 2024-04-20T17:53:07.230 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000100591 2024-04-20T17:53:07.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.230 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:53:07.231 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:52.164248+0000", 2024-04-20T17:53:07.231 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0077158469999999996 2024-04-20T17:53:07.231 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.231 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.231 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:53:07.231 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:52.202517+0000", 2024-04-20T17:53:07.231 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.038268798999999999 2024-04-20T17:53:07.231 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.231 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.231 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:53:07.231 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:52.202569+0000", 2024-04-20T17:53:07.231 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1450999999999999e-05 2024-04-20T17:53:07.232 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.232 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.232 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:53:07.232 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:52.202617+0000", 2024-04-20T17:53:07.232 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8043000000000001e-05 2024-04-20T17:53:07.232 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.232 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:53:07.232 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.232 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.232 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.232 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:181628 9.e 9:78e626df:::benchmark_data_smithi171_188319_object14832:head [delete] snapc 0=[] ondisk+write+known_if_redirected e937)", 2024-04-20T17:53:07.232 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:50:52.156290+0000", 2024-04-20T17:53:07.232 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 135.01512197599999, 2024-04-20T17:53:07.233 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.046487256999999997, 2024-04-20T17:53:07.233 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:53:07.233 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:53:07.233 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:53:07.233 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:53:07.233 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:53:07.233 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:53:07.233 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 181628 2024-04-20T17:53:07.233 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.233 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:53:07.233 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.233 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:53:07.233 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:52.156290+0000", 2024-04-20T17:53:07.233 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.234 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.234 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.234 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:53:07.234 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:52.156290+0000", 2024-04-20T17:53:07.234 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.234 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.234 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.234 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:53:07.234 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:52.156291+0000", 2024-04-20T17:53:07.234 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7599999999999996e-07 2024-04-20T17:53:07.234 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.234 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.234 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:53:07.235 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:52.156294+0000", 2024-04-20T17:53:07.235 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2629999999999999e-06 2024-04-20T17:53:07.235 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.235 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.235 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:53:07.235 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:52.156299+0000", 2024-04-20T17:53:07.235 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.558e-06 2024-04-20T17:53:07.235 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.235 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.235 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:53:07.235 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:52.156305+0000", 2024-04-20T17:53:07.235 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6790000000000002e-06 2024-04-20T17:53:07.235 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.235 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.236 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:53:07.236 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:52.160778+0000", 2024-04-20T17:53:07.236 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0044726979999999998 2024-04-20T17:53:07.236 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.236 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.236 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:53:07.236 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:52.160895+0000", 2024-04-20T17:53:07.236 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011695499999999999 2024-04-20T17:53:07.236 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.236 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.236 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:53:07.236 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:52.160988+0000", 2024-04-20T17:53:07.236 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3265999999999998e-05 2024-04-20T17:53:07.237 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.237 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.237 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:53:07.237 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:52.164435+0000", 2024-04-20T17:53:07.237 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0034474589999999999 2024-04-20T17:53:07.237 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.237 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.237 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:53:07.237 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:52.202681+0000", 2024-04-20T17:53:07.237 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.038245732999999997 2024-04-20T17:53:07.237 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.237 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.237 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:53:07.237 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:52.202734+0000", 2024-04-20T17:53:07.237 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2672e-05 2024-04-20T17:53:07.238 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.238 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.238 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:53:07.238 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:52.202777+0000", 2024-04-20T17:53:07.238 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3198000000000002e-05 2024-04-20T17:53:07.238 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.238 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:53:07.238 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.238 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.238 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.238 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:185023 9.e 9:7e174dfd:::benchmark_data_smithi171_188319_object18227:head [delete] snapc 0=[] ondisk+write+known_if_redirected e940)", 2024-04-20T17:53:07.238 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:50:55.303921+0000", 2024-04-20T17:53:07.238 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 131.86749032200001, 2024-04-20T17:53:07.238 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.050971081000000001, 2024-04-20T17:53:07.239 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:53:07.239 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:53:07.239 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:53:07.239 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:53:07.239 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:53:07.239 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:53:07.239 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 185023 2024-04-20T17:53:07.239 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.239 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:53:07.239 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.239 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:53:07.239 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:55.303921+0000", 2024-04-20T17:53:07.239 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.240 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.240 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.240 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:53:07.240 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:55.303921+0000", 2024-04-20T17:53:07.240 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.240 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.240 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.240 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:53:07.240 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:55.303922+0000", 2024-04-20T17:53:07.240 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1799999999999995e-07 2024-04-20T17:53:07.240 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.240 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.240 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:53:07.241 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:55.303925+0000", 2024-04-20T17:53:07.241 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9000000000000002e-06 2024-04-20T17:53:07.241 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.241 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.241 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:53:07.241 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:55.303930+0000", 2024-04-20T17:53:07.241 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3630000000000002e-06 2024-04-20T17:53:07.241 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.241 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.241 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:53:07.241 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:55.303940+0000", 2024-04-20T17:53:07.242 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6860000000000001e-06 2024-04-20T17:53:07.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.242 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:53:07.242 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:55.307418+0000", 2024-04-20T17:53:07.242 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0034778949999999999 2024-04-20T17:53:07.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.242 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:53:07.242 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:55.307537+0000", 2024-04-20T17:53:07.242 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000118972 2024-04-20T17:53:07.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.242 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:53:07.243 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:55.307615+0000", 2024-04-20T17:53:07.243 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8255000000000002e-05 2024-04-20T17:53:07.243 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.243 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.243 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:53:07.243 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:55.311102+0000", 2024-04-20T17:53:07.243 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0034872639999999999 2024-04-20T17:53:07.243 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.243 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.243 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:53:07.243 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:55.354779+0000", 2024-04-20T17:53:07.243 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.043676389000000003 2024-04-20T17:53:07.243 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.243 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.243 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:53:07.243 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:55.354845+0000", 2024-04-20T17:53:07.244 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5784999999999994e-05 2024-04-20T17:53:07.244 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.244 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.244 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:53:07.244 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:50:55.354893+0000", 2024-04-20T17:53:07.244 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7954000000000002e-05 2024-04-20T17:53:07.244 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.244 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:53:07.244 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.244 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.244 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.245 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:07.245 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:07.245 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:07.246 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:07.246 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:06 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:07.246 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:07.246 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:07.246 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:07.246 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:07.246 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:06 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:07.247 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:199217 9.e 9:793c7e57:::benchmark_data_smithi171_188319_object32421:head [delete] snapc 0=[] ondisk+write+known_if_redirected e943)", 2024-04-20T17:53:07.247 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:51:08.249679+0000", 2024-04-20T17:53:07.247 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 118.921732554, 2024-04-20T17:53:07.247 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.043963332000000001, 2024-04-20T17:53:07.247 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:53:07.247 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:53:07.248 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:53:07.248 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:53:07.248 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:53:07.248 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:53:07.248 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 199217 2024-04-20T17:53:07.248 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.248 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:53:07.248 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.248 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:53:07.248 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.249679+0000", 2024-04-20T17:53:07.248 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.248 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.249 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:53:07.249 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.249679+0000", 2024-04-20T17:53:07.249 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.249 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.249 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:53:07.249 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.249680+0000", 2024-04-20T17:53:07.249 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.68e-07 2024-04-20T17:53:07.249 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.249 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:53:07.249 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.249682+0000", 2024-04-20T17:53:07.249 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.689e-06 2024-04-20T17:53:07.249 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:53:07.250 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.249690+0000", 2024-04-20T17:53:07.250 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7789999999999996e-06 2024-04-20T17:53:07.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:53:07.250 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.249694+0000", 2024-04-20T17:53:07.250 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.563e-06 2024-04-20T17:53:07.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:53:07.250 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.256882+0000", 2024-04-20T17:53:07.250 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0071879370000000001 2024-04-20T17:53:07.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.251 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:53:07.251 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.256996+0000", 2024-04-20T17:53:07.251 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011439599999999999 2024-04-20T17:53:07.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.251 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:53:07.251 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.257077+0000", 2024-04-20T17:53:07.251 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0504e-05 2024-04-20T17:53:07.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.251 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:53:07.251 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.262891+0000", 2024-04-20T17:53:07.251 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0058146040000000001 2024-04-20T17:53:07.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.252 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:53:07.252 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.293559+0000", 2024-04-20T17:53:07.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.030667674999999998 2024-04-20T17:53:07.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.252 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:53:07.252 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.293604+0000", 2024-04-20T17:53:07.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5166000000000001e-05 2024-04-20T17:53:07.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.252 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:53:07.252 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.293643+0000", 2024-04-20T17:53:07.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8451000000000002e-05 2024-04-20T17:53:07.253 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.253 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:53:07.253 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.253 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.253 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.253 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:199234 9.e 9:76682b1c:::benchmark_data_smithi171_188319_object32438:head [delete] snapc 0=[] ondisk+write+known_if_redirected e943)", 2024-04-20T17:53:07.253 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:51:08.261698+0000", 2024-04-20T17:53:07.253 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 118.90971418399999, 2024-04-20T17:53:07.253 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.049981497, 2024-04-20T17:53:07.253 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:53:07.253 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:53:07.253 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:53:07.253 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:53:07.253 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:53:07.253 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:53:07.254 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 199234 2024-04-20T17:53:07.254 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.254 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:53:07.254 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.254 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:53:07.254 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.261698+0000", 2024-04-20T17:53:07.254 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.254 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.254 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.254 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:53:07.254 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.261698+0000", 2024-04-20T17:53:07.254 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.254 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.254 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.255 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:53:07.255 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.261699+0000", 2024-04-20T17:53:07.255 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.079e-06 2024-04-20T17:53:07.255 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.255 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.255 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:53:07.255 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.261703+0000", 2024-04-20T17:53:07.255 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0720000000000001e-06 2024-04-20T17:53:07.255 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.255 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.255 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:53:07.255 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.261713+0000", 2024-04-20T17:53:07.255 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8980000000000008e-06 2024-04-20T17:53:07.255 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.255 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.255 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:53:07.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.261719+0000", 2024-04-20T17:53:07.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8090000000000004e-06 2024-04-20T17:53:07.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:53:07.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.266350+0000", 2024-04-20T17:53:07.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0046306439999999997 2024-04-20T17:53:07.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:53:07.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.266427+0000", 2024-04-20T17:53:07.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6821000000000002e-05 2024-04-20T17:53:07.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:53:07.257 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.266502+0000", 2024-04-20T17:53:07.257 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5605999999999994e-05 2024-04-20T17:53:07.257 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.257 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.257 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:53:07.257 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.273931+0000", 2024-04-20T17:53:07.257 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0074284420000000004 2024-04-20T17:53:07.257 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.257 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.257 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:53:07.257 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.311606+0000", 2024-04-20T17:53:07.257 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.037675133999999999 2024-04-20T17:53:07.257 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.257 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.257 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:53:07.258 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.311640+0000", 2024-04-20T17:53:07.258 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4208999999999997e-05 2024-04-20T17:53:07.258 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.258 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.258 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:53:07.258 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.311679+0000", 2024-04-20T17:53:07.258 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8782999999999998e-05 2024-04-20T17:53:07.258 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.258 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:53:07.258 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.258 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.258 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.258 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:199235 9.e 9:7d78b2d2:::benchmark_data_smithi171_188319_object32439:head [delete] snapc 0=[] ondisk+write+known_if_redirected e943)", 2024-04-20T17:53:07.259 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:51:08.265501+0000", 2024-04-20T17:53:07.259 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 118.90591116900001, 2024-04-20T17:53:07.259 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.051646365, 2024-04-20T17:53:07.259 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:53:07.259 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:53:07.259 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:53:07.259 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:53:07.259 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:53:07.259 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:53:07.259 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 199235 2024-04-20T17:53:07.259 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.259 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:53:07.259 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.259 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:53:07.259 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.265501+0000", 2024-04-20T17:53:07.260 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.260 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.260 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.260 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:53:07.260 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.265501+0000", 2024-04-20T17:53:07.260 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.260 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.260 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.260 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:53:07.260 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.265502+0000", 2024-04-20T17:53:07.260 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.902e-06 2024-04-20T17:53:07.260 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.260 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.260 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:53:07.261 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.265509+0000", 2024-04-20T17:53:07.261 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3350000000000003e-06 2024-04-20T17:53:07.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.261 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.261 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:53:07.261 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.265521+0000", 2024-04-20T17:53:07.261 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1812e-05 2024-04-20T17:53:07.262 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.262 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:53:07.262 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.265530+0000", 2024-04-20T17:53:07.262 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.55e-06 2024-04-20T17:53:07.262 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.262 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:53:07.262 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.270873+0000", 2024-04-20T17:53:07.262 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0053423430000000003 2024-04-20T17:53:07.262 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.262 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:53:07.262 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.270973+0000", 2024-04-20T17:53:07.263 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010077900000000001 2024-04-20T17:53:07.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.263 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:53:07.263 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.271048+0000", 2024-04-20T17:53:07.263 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5165000000000003e-05 2024-04-20T17:53:07.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.263 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:53:07.263 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.274083+0000", 2024-04-20T17:53:07.263 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0030343890000000002 2024-04-20T17:53:07.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.263 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:53:07.263 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.317044+0000", 2024-04-20T17:53:07.264 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.042961006000000003 2024-04-20T17:53:07.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.264 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:53:07.264 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.317100+0000", 2024-04-20T17:53:07.264 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6029999999999997e-05 2024-04-20T17:53:07.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.264 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:53:07.264 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:08.317147+0000", 2024-04-20T17:53:07.264 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7054e-05 2024-04-20T17:53:07.264 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.264 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:53:07.264 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.265 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.265 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:201862 9.e 9:7895d6db:::benchmark_data_smithi171_188319_object35066:head [delete] snapc 0=[] ondisk+write+known_if_redirected e944)", 2024-04-20T17:53:07.265 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:51:10.605040+0000", 2024-04-20T17:53:07.265 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 116.56637195, 2024-04-20T17:53:07.265 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.043694482999999999, 2024-04-20T17:53:07.265 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:53:07.265 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:53:07.265 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:53:07.265 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:53:07.265 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:53:07.265 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:53:07.265 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 201862 2024-04-20T17:53:07.265 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.265 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:53:07.265 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.266 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:53:07.266 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:10.605040+0000", 2024-04-20T17:53:07.266 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.266 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.266 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.266 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:53:07.266 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:10.605040+0000", 2024-04-20T17:53:07.266 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.266 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.266 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.266 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:53:07.266 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:10.605042+0000", 2024-04-20T17:53:07.266 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9240000000000001e-06 2024-04-20T17:53:07.267 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.267 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.267 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:53:07.267 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:10.605047+0000", 2024-04-20T17:53:07.267 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3689999999999998e-06 2024-04-20T17:53:07.267 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.267 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.267 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:53:07.267 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:10.605061+0000", 2024-04-20T17:53:07.267 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.428e-05 2024-04-20T17:53:07.267 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.267 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.267 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:53:07.267 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:10.605069+0000", 2024-04-20T17:53:07.268 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8220000000000007e-06 2024-04-20T17:53:07.268 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.268 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.268 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:53:07.268 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:10.614669+0000", 2024-04-20T17:53:07.268 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0096002229999999997 2024-04-20T17:53:07.268 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.268 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.268 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:53:07.268 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:10.614783+0000", 2024-04-20T17:53:07.268 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000113848 2024-04-20T17:53:07.268 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.268 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.268 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:53:07.268 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:10.614904+0000", 2024-04-20T17:53:07.269 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000121206 2024-04-20T17:53:07.269 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.269 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.269 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:53:07.269 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:10.620730+0000", 2024-04-20T17:53:07.269 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.005825195 2024-04-20T17:53:07.269 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.269 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.269 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:53:07.269 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:10.648620+0000", 2024-04-20T17:53:07.269 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.027890361999999998 2024-04-20T17:53:07.269 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.269 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.269 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:53:07.269 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:10.648681+0000", 2024-04-20T17:53:07.269 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0529000000000003e-05 2024-04-20T17:53:07.270 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.270 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.270 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:53:07.270 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:10.648734+0000", 2024-04-20T17:53:07.270 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3724999999999998e-05 2024-04-20T17:53:07.270 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.270 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:53:07.270 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.270 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.270 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.270 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:208037 9.e 9:7cb70b29:::benchmark_data_smithi171_188319_object41241:head [delete] snapc 0=[] ondisk+write+known_if_redirected e944)", 2024-04-20T17:53:07.270 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:51:16.074054+0000", 2024-04-20T17:53:07.270 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 111.097358, 2024-04-20T17:53:07.270 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.053191319000000001, 2024-04-20T17:53:07.271 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:53:07.271 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:53:07.271 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:53:07.271 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:53:07.271 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:53:07.271 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:53:07.271 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 208037 2024-04-20T17:53:07.271 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.271 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:53:07.271 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.271 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:53:07.271 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:16.074054+0000", 2024-04-20T17:53:07.271 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.271 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.272 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.272 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:53:07.272 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:16.074054+0000", 2024-04-20T17:53:07.272 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.272 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.272 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.272 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:53:07.272 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:16.074056+0000", 2024-04-20T17:53:07.272 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0660000000000002e-06 2024-04-20T17:53:07.272 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.272 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.272 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:53:07.272 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:16.074065+0000", 2024-04-20T17:53:07.272 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1819999999999995e-06 2024-04-20T17:53:07.272 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.272 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.273 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:53:07.273 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:16.074076+0000", 2024-04-20T17:53:07.273 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.058e-05 2024-04-20T17:53:07.273 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.273 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.273 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:53:07.273 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:16.074084+0000", 2024-04-20T17:53:07.273 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0139999999999998e-06 2024-04-20T17:53:07.273 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.273 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.273 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:53:07.273 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:16.074129+0000", 2024-04-20T17:53:07.273 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5658999999999998e-05 2024-04-20T17:53:07.273 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.273 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.274 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:53:07.274 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:16.074221+0000", 2024-04-20T17:53:07.274 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1488000000000002e-05 2024-04-20T17:53:07.274 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.274 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.274 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:53:07.274 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:16.074298+0000", 2024-04-20T17:53:07.274 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6977999999999994e-05 2024-04-20T17:53:07.274 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.274 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.274 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:53:07.274 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:16.080800+0000", 2024-04-20T17:53:07.274 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0065018649999999999 2024-04-20T17:53:07.274 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.274 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.275 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:53:07.275 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:16.127172+0000", 2024-04-20T17:53:07.275 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.046371942999999999 2024-04-20T17:53:07.275 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.275 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:53:07.275 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:16.127207+0000", 2024-04-20T17:53:07.275 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5892000000000001e-05 2024-04-20T17:53:07.275 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.275 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:53:07.275 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:16.127245+0000", 2024-04-20T17:53:07.275 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7651999999999998e-05 2024-04-20T17:53:07.275 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.275 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:53:07.276 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.276 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.276 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.276 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:208038 9.e 9:7ebbb81b:::benchmark_data_smithi171_188319_object41242:head [delete] snapc 0=[] ondisk+write+known_if_redirected e944)", 2024-04-20T17:53:07.276 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:51:16.074922+0000", 2024-04-20T17:53:07.276 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 111.096490222, 2024-04-20T17:53:07.276 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.052429251000000003, 2024-04-20T17:53:07.276 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:53:07.276 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:53:07.276 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:53:07.276 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:53:07.276 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:53:07.276 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:53:07.276 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 208038 2024-04-20T17:53:07.276 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.277 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:53:07.277 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.277 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:53:07.277 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:16.074922+0000", 2024-04-20T17:53:07.277 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.277 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.277 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.277 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:53:07.277 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:16.074922+0000", 2024-04-20T17:53:07.277 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.277 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.277 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.277 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:53:07.277 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:16.074923+0000", 2024-04-20T17:53:07.277 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0210000000000001e-06 2024-04-20T17:53:07.278 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.278 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.278 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:53:07.278 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:16.074926+0000", 2024-04-20T17:53:07.278 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5149999999999998e-06 2024-04-20T17:53:07.278 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.278 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.278 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:53:07.278 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:16.074938+0000", 2024-04-20T17:53:07.278 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1695999999999999e-05 2024-04-20T17:53:07.278 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.278 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.278 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:53:07.278 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:16.074943+0000", 2024-04-20T17:53:07.278 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3360000000000002e-06 2024-04-20T17:53:07.279 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.279 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.279 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:53:07.279 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:16.077657+0000", 2024-04-20T17:53:07.279 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002713862 2024-04-20T17:53:07.279 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.279 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.279 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:53:07.279 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:16.077739+0000", 2024-04-20T17:53:07.279 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1958000000000004e-05 2024-04-20T17:53:07.279 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.279 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.279 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:53:07.279 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:16.077826+0000", 2024-04-20T17:53:07.279 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6846000000000002e-05 2024-04-20T17:53:07.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.280 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.280 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:53:07.280 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:16.080932+0000", 2024-04-20T17:53:07.280 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0031061180000000002 2024-04-20T17:53:07.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.280 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.280 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:53:07.280 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:16.127282+0000", 2024-04-20T17:53:07.280 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.046350615999999997 2024-04-20T17:53:07.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.280 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.280 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:53:07.280 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:16.127315+0000", 2024-04-20T17:53:07.280 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2765999999999998e-05 2024-04-20T17:53:07.281 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.281 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:53:07.281 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:16.127351+0000", 2024-04-20T17:53:07.281 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5516999999999999e-05 2024-04-20T17:53:07.281 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.281 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:53:07.281 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.281 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.281 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:236549 9.e 9:74c8355a:::benchmark_data_smithi171_188319_object69753:head [delete] snapc 0=[] ondisk+write+known_if_redirected e951)", 2024-04-20T17:53:07.281 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:51:42.172209+0000", 2024-04-20T17:53:07.281 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 84.999203007999995, 2024-04-20T17:53:07.281 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.046205079000000003, 2024-04-20T17:53:07.281 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:53:07.282 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:53:07.282 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:53:07.282 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:53:07.282 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:53:07.282 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:53:07.282 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 236549 2024-04-20T17:53:07.282 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.282 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:53:07.282 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.282 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:53:07.282 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:42.172209+0000", 2024-04-20T17:53:07.282 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.282 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.282 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.282 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:53:07.283 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:42.172209+0000", 2024-04-20T17:53:07.283 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.283 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.283 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:53:07.283 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:42.172211+0000", 2024-04-20T17:53:07.283 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1220000000000002e-06 2024-04-20T17:53:07.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.283 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.283 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:53:07.283 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:42.172217+0000", 2024-04-20T17:53:07.283 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8719999999999999e-06 2024-04-20T17:53:07.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.283 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.283 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:53:07.284 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:42.172229+0000", 2024-04-20T17:53:07.284 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2368000000000001e-05 2024-04-20T17:53:07.284 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.284 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.284 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:53:07.284 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:42.172238+0000", 2024-04-20T17:53:07.284 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4880000000000007e-06 2024-04-20T17:53:07.284 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.284 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.284 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:53:07.284 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:42.172645+0000", 2024-04-20T17:53:07.284 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000407174 2024-04-20T17:53:07.284 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.284 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.284 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:53:07.285 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:42.172754+0000", 2024-04-20T17:53:07.285 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010915400000000001 2024-04-20T17:53:07.285 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.285 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.285 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10", 2024-04-20T17:53:07.285 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:42.172864+0000", 2024-04-20T17:53:07.285 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010971 2024-04-20T17:53:07.285 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.285 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.285 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:53:07.285 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:42.177502+0000", 2024-04-20T17:53:07.285 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.004637965 2024-04-20T17:53:07.285 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.285 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.285 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:53:07.285 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:42.218306+0000", 2024-04-20T17:53:07.286 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.040804881000000001 2024-04-20T17:53:07.286 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.286 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.286 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:53:07.286 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:42.218365+0000", 2024-04-20T17:53:07.286 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8313000000000002e-05 2024-04-20T17:53:07.286 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.286 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.286 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:53:07.286 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:51:42.218414+0000", 2024-04-20T17:53:07.286 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9032000000000001e-05 2024-04-20T17:53:07.286 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.286 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:53:07.286 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.286 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.287 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.287 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27772.0:262619 9.d 9:b42662cb:::benchmark_data_smithi171_188319_object95823:head [delete] snapc 0=[] ondisk+write+known_if_redirected e958)", 2024-04-20T17:53:07.287 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-04-20T17:52:07.775417+0000", 2024-04-20T17:53:07.287 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 59.395994735999999, 2024-04-20T17:53:07.287 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.093588886999999996, 2024-04-20T17:53:07.287 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-04-20T17:53:07.287 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-04-20T17:53:07.287 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T17:53:07.287 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-04-20T17:53:07.287 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27772", 2024-04-20T17:53:07.287 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "v1:172.21.15.171:0/276728151", 2024-04-20T17:53:07.287 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 262619 2024-04-20T17:53:07.287 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.287 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-04-20T17:53:07.287 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.288 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-04-20T17:53:07.288 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:52:07.775417+0000", 2024-04-20T17:53:07.288 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.288 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.288 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.288 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-04-20T17:53:07.288 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:52:07.775417+0000", 2024-04-20T17:53:07.288 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-04-20T17:53:07.288 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.288 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.288 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-04-20T17:53:07.288 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:52:07.775419+0000", 2024-04-20T17:53:07.288 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9970000000000001e-06 2024-04-20T17:53:07.288 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.288 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.289 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-04-20T17:53:07.289 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:52:07.775425+0000", 2024-04-20T17:53:07.289 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6829999999999996e-06 2024-04-20T17:53:07.289 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.289 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.289 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-04-20T17:53:07.289 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:52:07.775438+0000", 2024-04-20T17:53:07.289 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3737000000000001e-05 2024-04-20T17:53:07.289 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.289 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.289 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-04-20T17:53:07.289 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:52:07.775447+0000", 2024-04-20T17:53:07.289 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8389999999999996e-06 2024-04-20T17:53:07.289 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.289 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.290 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-04-20T17:53:07.290 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:52:07.775500+0000", 2024-04-20T17:53:07.290 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3183999999999998e-05 2024-04-20T17:53:07.290 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.290 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.290 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-04-20T17:53:07.290 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:52:07.775583+0000", 2024-04-20T17:53:07.290 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2694000000000005e-05 2024-04-20T17:53:07.290 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.290 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.290 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1", 2024-04-20T17:53:07.290 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:52:07.775646+0000", 2024-04-20T17:53:07.290 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2557000000000004e-05 2024-04-20T17:53:07.290 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.290 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.291 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-04-20T17:53:07.291 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:52:07.779983+0000", 2024-04-20T17:53:07.291 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0043377859999999997 2024-04-20T17:53:07.291 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.291 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.291 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-04-20T17:53:07.291 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:52:07.868900+0000", 2024-04-20T17:53:07.291 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.088916492 2024-04-20T17:53:07.291 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.291 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.291 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-04-20T17:53:07.291 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:52:07.868963+0000", 2024-04-20T17:53:07.291 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2648000000000001e-05 2024-04-20T17:53:07.291 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-04-20T17:53:07.291 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-04-20T17:53:07.291 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-04-20T17:53:07.292 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-04-20T17:52:07.869006+0000", 2024-04-20T17:53:07.292 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3269999999999997e-05 2024-04-20T17:53:07.292 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.292 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:53:07.292 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.292 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-04-20T17:53:07.292 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-04-20T17:53:07.292 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-04-20T17:53:07.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:07.292 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:07.293 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:07.293 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:07.293 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:07.293 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:06 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:07.293 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:07.293 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8" found: no such container 2024-04-20T17:53:07.293 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.8' 2024-04-20T17:53:07.394 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:53:07.429 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:07.473 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:53:07.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:07.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:53:07.581 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:53:07.629 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:07.676 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:53:07.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:07.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:53:07.782 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:53:07.820 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:07.856 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:53:07.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:07.858 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:53:07.877 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:07 smithi005 ceph-mon[39562]: pgmap v2573: 27 pgs: 27 active+clean; 430 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:53:07.940 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-20T17:53:07.959 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0 2024-04-20T17:53:08.042 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:08.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:07 smithi171 ceph-mon[44189]: pgmap v2573: 27 pgs: 27 active+clean; 430 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:53:08.173 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T17:53:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:07 smithi106 ceph-mon[44423]: pgmap v2573: 27 pgs: 27 active+clean; 430 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s 2024-04-20T17:53:08.643 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0" found: no such container 2024-04-20T17:53:08.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:08.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.0' 2024-04-20T17:53:08.753 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:53:08.784 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:08.813 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:53:08.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:08.816 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:53:08.918 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:53:08.965 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:09.626 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:53:09.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:09.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:53:09.667 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:53:09.688 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:53:09.742 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:53:09.784 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:09.815 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:53:09.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:09.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:53:09.896 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:53:09.919 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:09 smithi005 ceph-mon[39562]: pgmap v2574: 27 pgs: 27 active+clean; 380 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s 2024-04-20T17:53:09.919 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:09 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/610094076' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:53:09.919 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:09 smithi005 ceph-mon[39562]: from='client.28415 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:53:09.921 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:53:09.953 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:09.985 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:53:09.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:09.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:53:10.090 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:53:10.133 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:09 smithi106 ceph-mon[44423]: pgmap v2574: 27 pgs: 27 active+clean; 380 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s 2024-04-20T17:53:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:09 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/610094076' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:53:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:09 smithi106 ceph-mon[44423]: from='client.28415 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:53:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:09 smithi171 ceph-mon[44189]: pgmap v2574: 27 pgs: 27 active+clean; 380 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s 2024-04-20T17:53:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:09 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/610094076' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:53:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:09 smithi171 ceph-mon[44189]: from='client.28415 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:53:10.426 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Removed 166793 objects 2024-04-20T17:53:10.426 INFO:tasks.radosbench.radosbench.2.smithi171.stdout:Clean up completed and total clean up time :153.536 2024-04-20T17:53:10.824 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_7 2024-04-20T17:53:10.824 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd pool rm unique_pool_7 unique_pool_7 --yes-i-really-really-mean-it 2024-04-20T17:53:10.893 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:53:10.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:10.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:53:10.998 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10 2024-04-20T17:53:11.026 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:11.047 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10" found: no such container 2024-04-20T17:53:11.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:11.050 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.10' 2024-04-20T17:53:11.151 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:53:11.178 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:10 smithi106 ceph-mon[44423]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:53:11.178 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:10 smithi106 ceph-mon[44423]: from='client.28415 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:53:11.178 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:10 smithi106 ceph-mon[44423]: osdmap e974: 12 total, 12 up, 12 in 2024-04-20T17:53:11.178 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:10 smithi106 ceph-mon[44423]: 9.5 scrub starts 2024-04-20T17:53:11.178 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:10 smithi106 ceph-mon[44423]: 9.5 scrub ok 2024-04-20T17:53:11.180 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:11.199 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:53:11.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:11.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:53:11.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:10 smithi005 ceph-mon[39562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:53:11.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:10 smithi005 ceph-mon[39562]: from='client.28415 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:53:11.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:10 smithi005 ceph-mon[39562]: osdmap e974: 12 total, 12 up, 12 in 2024-04-20T17:53:11.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:10 smithi005 ceph-mon[39562]: 9.5 scrub starts 2024-04-20T17:53:11.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:10 smithi005 ceph-mon[39562]: 9.5 scrub ok 2024-04-20T17:53:11.302 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:53:11.331 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:10 smithi171 ceph-mon[44189]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-20T17:53:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:10 smithi171 ceph-mon[44189]: from='client.28415 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:53:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:10 smithi171 ceph-mon[44189]: osdmap e974: 12 total, 12 up, 12 in 2024-04-20T17:53:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:10 smithi171 ceph-mon[44189]: 9.5 scrub starts 2024-04-20T17:53:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:10 smithi171 ceph-mon[44189]: 9.5 scrub ok 2024-04-20T17:53:11.350 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:53:11.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:11.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:53:11.454 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11 2024-04-20T17:53:11.482 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:11.501 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11" found: no such container 2024-04-20T17:53:11.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:11.504 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.11' 2024-04-20T17:53:11.605 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9 2024-04-20T17:53:11.632 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:11.652 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9" found: no such container 2024-04-20T17:53:11.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:11.655 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.9' 2024-04-20T17:53:11.756 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3 2024-04-20T17:53:11.786 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:12.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:11 smithi005 ceph-mon[39562]: pgmap v2576: 27 pgs: 27 active+clean; 131 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-04-20T17:53:12.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:11 smithi106 ceph-mon[44423]: pgmap v2576: 27 pgs: 27 active+clean; 131 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-04-20T17:53:12.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:11 smithi171 ceph-mon[44189]: pgmap v2576: 27 pgs: 27 active+clean; 131 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s 2024-04-20T17:53:12.637 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3" found: no such container 2024-04-20T17:53:12.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:12.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.3' 2024-04-20T17:53:12.741 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5 2024-04-20T17:53:12.773 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:12.793 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5" found: no such container 2024-04-20T17:53:12.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:12.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.5' 2024-04-20T17:53:12.897 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:53:12.926 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:13.082 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:53:13.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:12 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:53:13.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:12 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:53:13.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:12 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:53:13.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:12 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:53:13.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:12 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:53:13.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:12 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:53:13.623 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:53:13.925 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:13 smithi005 ceph-mon[39562]: 1.5 scrub starts 2024-04-20T17:53:13.925 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:13 smithi005 ceph-mon[39562]: pgmap v2577: 27 pgs: 27 active+clean; 44 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-04-20T17:53:13.925 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:13 smithi005 ceph-mon[39562]: 1.5 scrub ok 2024-04-20T17:53:13.926 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:13 smithi005 ceph-mon[39562]: osdmap e975: 12 total, 12 up, 12 in 2024-04-20T17:53:13.926 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:13 smithi005 ceph-mon[39562]: 9.b scrub starts 2024-04-20T17:53:13.926 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:13 smithi005 ceph-mon[39562]: 9.b scrub ok 2024-04-20T17:53:13.926 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:13 smithi005 ceph-mon[39562]: 9.2 scrub starts 2024-04-20T17:53:13.926 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:13 smithi005 ceph-mon[39562]: 9.2 scrub ok 2024-04-20T17:53:14.056 INFO:tasks.thrashosds.thrasher:Recovered, killing an osd 2024-04-20T17:53:14.056 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [0, 4, 5, 8, 9, 10, 11, 2, 6, 3, 1, 7] 2024-04-20T17:53:14.056 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-20T17:53:14.056 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.0 2024-04-20T17:53:14.211 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:53:14 smithi005 systemd[1]: Stopping Ceph osd.0 for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:53:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:13 smithi106 ceph-mon[44423]: 1.5 scrub starts 2024-04-20T17:53:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:13 smithi106 ceph-mon[44423]: pgmap v2577: 27 pgs: 27 active+clean; 44 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-04-20T17:53:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:13 smithi106 ceph-mon[44423]: 1.5 scrub ok 2024-04-20T17:53:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:13 smithi106 ceph-mon[44423]: osdmap e975: 12 total, 12 up, 12 in 2024-04-20T17:53:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:13 smithi106 ceph-mon[44423]: 9.b scrub starts 2024-04-20T17:53:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:13 smithi106 ceph-mon[44423]: 9.b scrub ok 2024-04-20T17:53:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:13 smithi106 ceph-mon[44423]: 9.2 scrub starts 2024-04-20T17:53:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:13 smithi106 ceph-mon[44423]: 9.2 scrub ok 2024-04-20T17:53:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:13 smithi171 ceph-mon[44189]: 1.5 scrub starts 2024-04-20T17:53:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:13 smithi171 ceph-mon[44189]: pgmap v2577: 27 pgs: 27 active+clean; 44 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s 2024-04-20T17:53:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:13 smithi171 ceph-mon[44189]: 1.5 scrub ok 2024-04-20T17:53:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:13 smithi171 ceph-mon[44189]: osdmap e975: 12 total, 12 up, 12 in 2024-04-20T17:53:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:13 smithi171 ceph-mon[44189]: 9.b scrub starts 2024-04-20T17:53:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:13 smithi171 ceph-mon[44189]: 9.b scrub ok 2024-04-20T17:53:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:13 smithi171 ceph-mon[44189]: 9.2 scrub starts 2024-04-20T17:53:14.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:13 smithi171 ceph-mon[44189]: 9.2 scrub ok 2024-04-20T17:53:15.710 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-04-20T17:53:15.710 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-04-20T17:53:15.710 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-04-20T17:53:15.710 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-04-20T17:53:15.710 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-04-20T17:53:15.710 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-04-20T17:53:15.710 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-04-20T17:53:15.710 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-04-20T17:53:15.711 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-04-20T17:53:15.711 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-04-20T17:53:15.711 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-04-20T17:53:15.711 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-04-20T17:53:15.711 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T17:53:15.711 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T17:53:15.711 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T17:53:15.711 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T17:53:15.711 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T17:53:15.711 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T17:53:15.711 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T17:53:15.711 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T17:53:15.712 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-20T17:53:15.712 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-20T17:53:15.712 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-20T17:53:15.712 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-20T17:53:15.902 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:15 smithi005 ceph-mon[39562]: pgmap v2579: 27 pgs: 27 active+clean; 6.5 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 845 op/s 2024-04-20T17:53:15.902 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:15 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4289799967' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:53:15.902 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:15.902 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:15 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:16.183 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:53:16.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:16.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:53:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:15 smithi106 ceph-mon[44423]: pgmap v2579: 27 pgs: 27 active+clean; 6.5 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 845 op/s 2024-04-20T17:53:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:15 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4289799967' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:53:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:15 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:16.290 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7 2024-04-20T17:53:16.323 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:16.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:15 smithi171 ceph-mon[44189]: pgmap v2579: 27 pgs: 27 active+clean; 6.5 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 845 op/s 2024-04-20T17:53:16.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:15 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4289799967' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-20T17:53:16.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:16.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:15 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:16.345 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7" found: no such container 2024-04-20T17:53:16.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:16.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.7' 2024-04-20T17:53:16.449 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:53:16.480 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:16.612 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:53:16.629 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:53:16.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:16.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:53:16.733 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2 2024-04-20T17:53:16.760 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:53:16 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-0[57624]: 2024-04-20T17:53:16.624+0000 7fa617533700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T17:53:16.760 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:53:16 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-0[57624]: 2024-04-20T17:53:16.624+0000 7fa617533700 -1 osd.0 975 *** Got signal Terminated *** 2024-04-20T17:53:16.760 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:53:16 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-0[57624]: 2024-04-20T17:53:16.624+0000 7fa617533700 -1 osd.0 975 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T17:53:16.763 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:16.836 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2" found: no such container 2024-04-20T17:53:16.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:16.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.2' 2024-04-20T17:53:16.908 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'unique_pool_7' removed 2024-04-20T17:53:16.940 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6 2024-04-20T17:53:16.984 INFO:teuthology.orchestra.run.smithi106.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:17.006 INFO:teuthology.orchestra.run.smithi106.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6" found: no such container 2024-04-20T17:53:17.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:17.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi106 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.6' 2024-04-20T17:53:17.110 DEBUG:teuthology.orchestra.run.smithi005:> sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1 2024-04-20T17:53:17.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:16 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:16 smithi005 ceph-mon[39562]: 9.9 scrub starts 2024-04-20T17:53:17.138 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:16 smithi005 ceph-mon[39562]: 9.9 scrub ok 2024-04-20T17:53:17.139 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:16 smithi005 ceph-mon[39562]: osd.0 marked itself down and dead 2024-04-20T17:53:17.139 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:16 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3928741064' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-20T17:53:17.140 INFO:teuthology.orchestra.run.smithi005.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-20T17:53:17.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:16 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:16 smithi106 ceph-mon[44423]: 9.9 scrub starts 2024-04-20T17:53:17.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:16 smithi106 ceph-mon[44423]: 9.9 scrub ok 2024-04-20T17:53:17.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:16 smithi106 ceph-mon[44423]: osd.0 marked itself down and dead 2024-04-20T17:53:17.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:16 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3928741064' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-20T17:53:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:16 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:53:17.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:16 smithi171 ceph-mon[44189]: 9.9 scrub starts 2024-04-20T17:53:17.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:16 smithi171 ceph-mon[44189]: 9.9 scrub ok 2024-04-20T17:53:17.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:16 smithi171 ceph-mon[44189]: osd.0 marked itself down and dead 2024-04-20T17:53:17.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:16 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3928741064' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-20T17:53:18.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:17 smithi005 ceph-mon[39562]: pgmap v2580: 27 pgs: 27 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 759 op/s 2024-04-20T17:53:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:17 smithi005 ceph-mon[39562]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T17:53:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:17 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3928741064' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-04-20T17:53:18.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:17 smithi005 ceph-mon[39562]: osdmap e976: 12 total, 11 up, 12 in 2024-04-20T17:53:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:17 smithi106 ceph-mon[44423]: pgmap v2580: 27 pgs: 27 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 759 op/s 2024-04-20T17:53:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:17 smithi106 ceph-mon[44423]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T17:53:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:17 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3928741064' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-04-20T17:53:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:17 smithi106 ceph-mon[44423]: osdmap e976: 12 total, 11 up, 12 in 2024-04-20T17:53:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:17 smithi171 ceph-mon[44189]: pgmap v2580: 27 pgs: 27 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 759 op/s 2024-04-20T17:53:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:17 smithi171 ceph-mon[44189]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-20T17:53:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:17 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3928741064' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-04-20T17:53:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:17 smithi171 ceph-mon[44189]: osdmap e976: 12 total, 11 up, 12 in 2024-04-20T17:53:18.567 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:53:18 smithi005 bash[369999]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-0 2024-04-20T17:53:18.595 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-20T17:53:18.604 INFO:tasks.thrashosds:joining thrashosds 2024-04-20T17:53:18.837 INFO:teuthology.orchestra.run.smithi005.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-20T17:53:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:18 smithi005 ceph-mon[39562]: osdmap e977: 12 total, 11 up, 12 in 2024-04-20T17:53:19.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:18 smithi106 ceph-mon[44423]: osdmap e977: 12 total, 11 up, 12 in 2024-04-20T17:53:19.321 INFO:teuthology.orchestra.run.smithi005.stderr:Error: no container with name or ID "ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1" found: no such container 2024-04-20T17:53:19.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-20T17:53:19.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi005 with status 125: 'sudo docker kill -s 1 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd.1' 2024-04-20T17:53:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:18 smithi171 ceph-mon[44189]: osdmap e977: 12 total, 11 up, 12 in 2024-04-20T17:53:19.868 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-20T17:53:19.869 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-20T17:53:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:19 smithi005 ceph-mon[39562]: pgmap v2583: 11 pgs: 11 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:20.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:19 smithi106 ceph-mon[44423]: pgmap v2583: 11 pgs: 11 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:19 smithi171 ceph-mon[44189]: pgmap v2583: 11 pgs: 11 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:21.075 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:53:21.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:20 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:53:21.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:20 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:53:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:20 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:53:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:20 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:53:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:20 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:53:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:20 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:53:21.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:21 smithi005 ceph-mon[39562]: pgmap v2584: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:21 smithi106 ceph-mon[44423]: pgmap v2584: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:21 smithi171 ceph-mon[44189]: pgmap v2584: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:22.934 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:53:23.111 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:22 smithi005 ceph-mon[39562]: pgmap v2585: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:23.112 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:22 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1407401174' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:53:23.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:22 smithi106 ceph-mon[44423]: pgmap v2585: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:23.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:22 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1407401174' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:53:23.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:22 smithi171 ceph-mon[44189]: pgmap v2585: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:23.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:22 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1407401174' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:53:23.981 INFO:teuthology.orchestra.run.smithi005.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-20T17:53:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:23 smithi005 ceph-mon[39562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:53:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:23 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1407401174' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:53:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:23 smithi005 ceph-mon[39562]: osdmap e978: 12 total, 11 up, 12 in 2024-04-20T17:53:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:23 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:53:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:23 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:53:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:23 smithi106 ceph-mon[44423]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:53:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:23 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1407401174' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:53:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:23 smithi106 ceph-mon[44423]: osdmap e978: 12 total, 11 up, 12 in 2024-04-20T17:53:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:23 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:53:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:23 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:53:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:23 smithi171 ceph-mon[44189]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-20T17:53:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:23 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1407401174' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:53:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:23 smithi171 ceph-mon[44189]: osdmap e978: 12 total, 11 up, 12 in 2024-04-20T17:53:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:23 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:53:24.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:23 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:53:25.054 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-20T17:53:25.056 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-20T17:53:25.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:24 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:53:25.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:24 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:53:25.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:24 smithi005 ceph-mon[39562]: osdmap e979: 12 total, 11 up, 12 in 2024-04-20T17:53:25.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:24 smithi005 ceph-mon[39562]: pgmap v2588: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:25.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:24 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:53:25.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:24 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:53:25.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:24 smithi106 ceph-mon[44423]: osdmap e979: 12 total, 11 up, 12 in 2024-04-20T17:53:25.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:24 smithi106 ceph-mon[44423]: pgmap v2588: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:25.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:24 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:53:25.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:24 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:53:25.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:24 smithi171 ceph-mon[44189]: osdmap e979: 12 total, 11 up, 12 in 2024-04-20T17:53:25.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:24 smithi171 ceph-mon[44189]: pgmap v2588: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:25.343 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:53:25.894 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-20T17:53:27.225 DEBUG:teuthology.orchestra.run.smithi005:> sudo pkill -f 'journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.0.service' 2024-04-20T17:53:27.254 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:53:27.277 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T17:53:27.277 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-20T17:53:27.277 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd down 0 2024-04-20T17:53:28.020 INFO:teuthology.orchestra.run.smithi005.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-20T17:53:28.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:27 smithi005 ceph-mon[39562]: pgmap v2589: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:28.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:27 smithi106 ceph-mon[44423]: pgmap v2589: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:27 smithi171 ceph-mon[44189]: pgmap v2589: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:28.626 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-20T17:53:28.627 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-20T17:53:28.906 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:53:29.937 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:53:29.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:29 smithi005 ceph-mon[39562]: pgmap v2590: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:29 smithi106 ceph-mon[44423]: pgmap v2590: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:30.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:29 smithi171 ceph-mon[44189]: pgmap v2590: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:30.858 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-04-20T17:53:31.152 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:30 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1496214676' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:53:31.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:30 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1496214676' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:53:31.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:30 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1496214676' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-20T17:53:32.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:31 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:53:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:31 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:53:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:31 smithi106 ceph-mon[44423]: pgmap v2591: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:31 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1496214676' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:53:32.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:31 smithi106 ceph-mon[44423]: osdmap e980: 12 total, 11 up, 12 in 2024-04-20T17:53:32.134 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:31 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:53:32.134 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:31 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:53:32.134 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:31 smithi005 ceph-mon[39562]: pgmap v2591: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:32.134 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:31 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1496214676' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:53:32.134 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:31 smithi005 ceph-mon[39562]: osdmap e980: 12 total, 11 up, 12 in 2024-04-20T17:53:32.157 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-20T17:53:32.282 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:53:32.282 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:53:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:31 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:53:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:31 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:53:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:31 smithi171 ceph-mon[44189]: pgmap v2591: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:31 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1496214676' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-20T17:53:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:31 smithi171 ceph-mon[44189]: osdmap e980: 12 total, 11 up, 12 in 2024-04-20T17:53:32.493 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:53:32.666 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0 is already down. 2024-04-20T17:53:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:32 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mon-a[39539]: 2024-04-20T17:53:32.663+0000 7f73b92a1700 -1 mon.a@0(leader).osd e980 definitely_dead 0 2024-04-20T17:53:32.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:53:32.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:53:32.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:53:32.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:53:32.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:53:32.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:53:32.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:32 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:53:32.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:32 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2161784065' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["0"]}]: dispatch 2024-04-20T17:53:32.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:32 smithi005 ceph-mon[39562]: from='client.28442 ' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["0"]}]: dispatch 2024-04-20T17:53:32.972 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:53:32.972 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:53:32.972 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:53:32.972 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:53:32.972 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:53:32.972 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:53:32.972 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:32 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:53:32.972 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:32 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2161784065' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["0"]}]: dispatch 2024-04-20T17:53:32.972 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:32 smithi106 ceph-mon[44423]: from='client.28442 ' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["0"]}]: dispatch 2024-04-20T17:53:32.999 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-20T17:53:33.263 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:53:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:53:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:53:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:53:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:53:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:53:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:53:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:32 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:53:33.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:32 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2161784065' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["0"]}]: dispatch 2024-04-20T17:53:33.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:32 smithi171 ceph-mon[44189]: from='client.28442 ' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["0"]}]: dispatch 2024-04-20T17:53:34.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:33 smithi005 ceph-mon[39562]: pgmap v2593: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:34.214 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:33 smithi106 ceph-mon[44423]: pgmap v2593: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:34.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:33 smithi171 ceph-mon[44189]: pgmap v2593: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:34.339 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 2, 3, 4, 5, 8, 9, 10, 11, 1, 6, 7] 2024-04-20T17:53:34.339 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd out 0 2024-04-20T17:53:34.837 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:53:35.156 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:53:35.156 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:53:35.180 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:34 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:53:35.180 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:34 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:53:35.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:34 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:53:35.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:34 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:53:35.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:34 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:53:35.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:34 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:53:35.929 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-20T17:53:35.930 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-04-20T17:53:35.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:35 smithi005 ceph-mon[39562]: pgmap v2594: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:35.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:35 smithi005 ceph-mon[39562]: osdmap e981: 12 total, 11 up, 12 in 2024-04-20T17:53:35.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:35 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3098903843' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:53:35.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:35 smithi005 ceph-mon[39562]: from='client.28081 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:53:36.197 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:53:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:35 smithi106 ceph-mon[44423]: pgmap v2594: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:35 smithi106 ceph-mon[44423]: osdmap e981: 12 total, 11 up, 12 in 2024-04-20T17:53:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:35 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3098903843' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:53:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:35 smithi106 ceph-mon[44423]: from='client.28081 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:53:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:35 smithi171 ceph-mon[44189]: pgmap v2594: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:35 smithi171 ceph-mon[44189]: osdmap e981: 12 total, 11 up, 12 in 2024-04-20T17:53:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:35 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3098903843' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:53:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:35 smithi171 ceph-mon[44189]: from='client.28081 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-20T17:53:37.048 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:36 smithi005 ceph-mon[39562]: from='client.28081 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:53:37.048 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:36 smithi005 ceph-mon[39562]: osdmap e982: 12 total, 11 up, 12 in 2024-04-20T17:53:37.048 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:36 smithi005 ceph-mon[39562]: pgmap v2597: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:37.203 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:36 smithi106 ceph-mon[44423]: from='client.28081 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:53:37.203 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:36 smithi106 ceph-mon[44423]: osdmap e982: 12 total, 11 up, 12 in 2024-04-20T17:53:37.203 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:36 smithi106 ceph-mon[44423]: pgmap v2597: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:36 smithi171 ceph-mon[44189]: from='client.28081 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-20T17:53:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:36 smithi171 ceph-mon[44189]: osdmap e982: 12 total, 11 up, 12 in 2024-04-20T17:53:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:36 smithi171 ceph-mon[44189]: pgmap v2597: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:53:37.936 INFO:teuthology.orchestra.run.smithi005.stderr:marked out osd.0. 2024-04-20T17:53:37.938 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:37 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mon-a[39539]: 2024-04-20T17:53:37.524+0000 7f73b92a1700 -1 mon.a@0(leader).osd e983 definitely_dead 0 2024-04-20T17:53:38.076 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:53:38.077 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:53:38.099 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:37 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:53:38.099 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:37 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:53:38.099 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:37 smithi106 ceph-mon[44423]: osdmap e983: 12 total, 11 up, 12 in 2024-04-20T17:53:38.099 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:37 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2653841593' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-04-20T17:53:38.099 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:37 smithi106 ceph-mon[44423]: Client client.admin marked osd.0 out, after it was down for 20 seconds 2024-04-20T17:53:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:37 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:53:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:37 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:53:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:37 smithi005 ceph-mon[39562]: osdmap e983: 12 total, 11 up, 12 in 2024-04-20T17:53:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:37 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2653841593' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-04-20T17:53:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:37 smithi005 ceph-mon[39562]: Client client.admin marked osd.0 out, after it was down for 20 seconds 2024-04-20T17:53:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:37 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:53:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:37 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:53:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:37 smithi171 ceph-mon[44189]: osdmap e983: 12 total, 11 up, 12 in 2024-04-20T17:53:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:37 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2653841593' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-04-20T17:53:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:37 smithi171 ceph-mon[44189]: Client client.admin marked osd.0 out, after it was down for 20 seconds 2024-04-20T17:53:38.702 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.0 2024-04-20T17:53:38.703 INFO:tasks.thrashosds.thrasher:remote for osd 0 is ubuntu@smithi005.front.sepia.ceph.com 2024-04-20T17:53:38.703 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.0 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op list-pgs 2024-04-20T17:53:38.926 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.0/config 2024-04-20T17:53:38.955 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-20T17:53:39.193 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:38 smithi005 ceph-mon[39562]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T17:53:39.194 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:38 smithi005 ceph-mon[39562]: Cluster is now healthy 2024-04-20T17:53:39.194 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:38 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2653841593' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-04-20T17:53:39.194 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:38 smithi005 ceph-mon[39562]: osdmap e984: 12 total, 11 up, 11 in 2024-04-20T17:53:39.194 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:38 smithi005 ceph-mon[39562]: pgmap v2600: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:39.229 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:53:39.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:38 smithi106 ceph-mon[44423]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T17:53:39.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:38 smithi106 ceph-mon[44423]: Cluster is now healthy 2024-04-20T17:53:39.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:38 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2653841593' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-04-20T17:53:39.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:38 smithi106 ceph-mon[44423]: osdmap e984: 12 total, 11 up, 11 in 2024-04-20T17:53:39.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:38 smithi106 ceph-mon[44423]: pgmap v2600: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:38 smithi171 ceph-mon[44189]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-20T17:53:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:38 smithi171 ceph-mon[44189]: Cluster is now healthy 2024-04-20T17:53:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:38 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2653841593' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-04-20T17:53:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:38 smithi171 ceph-mon[44189]: osdmap e984: 12 total, 11 up, 11 in 2024-04-20T17:53:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:38 smithi171 ceph-mon[44189]: pgmap v2600: 11 pgs: 1 undersized+peered, 10 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:40.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:39 smithi171 ceph-mon[44189]: osdmap e985: 12 total, 11 up, 11 in 2024-04-20T17:53:40.145 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:39 smithi005 ceph-mon[39562]: osdmap e985: 12 total, 11 up, 11 in 2024-04-20T17:53:40.192 INFO:teuthology.orchestra.run.smithi005.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:53:40.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:39 smithi106 ceph-mon[44423]: osdmap e985: 12 total, 11 up, 11 in 2024-04-20T17:53:41.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:40 smithi171 ceph-mon[44189]: pgmap v2602: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:41.149 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:53:41.149 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:53:41.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:40 smithi005 ceph-mon[39562]: pgmap v2602: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:41.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:40 smithi106 ceph-mon[44423]: pgmap v2602: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:41.771 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-20T17:53:42.038 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:53:42.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:41 smithi171 ceph-mon[44189]: 1.2 scrub starts 2024-04-20T17:53:42.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:41 smithi171 ceph-mon[44189]: 1.2 scrub ok 2024-04-20T17:53:42.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:41 smithi005 ceph-mon[39562]: 1.2 scrub starts 2024-04-20T17:53:42.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:41 smithi005 ceph-mon[39562]: 1.2 scrub ok 2024-04-20T17:53:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:41 smithi106 ceph-mon[44423]: 1.2 scrub starts 2024-04-20T17:53:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:41 smithi106 ceph-mon[44423]: 1.2 scrub ok 2024-04-20T17:53:43.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:42 smithi005 ceph-mon[39562]: pgmap v2603: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:43.251 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:42 smithi171 ceph-mon[44189]: pgmap v2603: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:43.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:42 smithi106 ceph-mon[44423]: pgmap v2603: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:43.305 INFO:teuthology.orchestra.run.smithi005.stdout:1.2 2024-04-20T17:53:43.803 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:53:43.803 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:53:44.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:43 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:53:44.085 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:43 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:53:44.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:43 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:53:44.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:43 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:53:44.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:43 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:53:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:43 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:53:44.392 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.0 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op export --pgid 1.2 --file /var/log/ceph/exp.1.2.0 2024-04-20T17:53:44.477 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-20T17:53:44.616 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.0/config 2024-04-20T17:53:44.744 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:53:45.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:44 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:53:45.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:44 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:53:45.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:44 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:53:45.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:44 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:53:45.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:44 smithi005 ceph-mon[39562]: osdmap e986: 12 total, 11 up, 11 in 2024-04-20T17:53:45.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:44 smithi005 ceph-mon[39562]: pgmap v2605: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:45.217 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:44 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:53:45.217 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:44 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:53:45.217 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:44 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:53:45.218 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:44 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:53:45.218 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:44 smithi171 ceph-mon[44189]: osdmap e986: 12 total, 11 up, 11 in 2024-04-20T17:53:45.218 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:44 smithi171 ceph-mon[44189]: pgmap v2605: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:44 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:53:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:44 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:53:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:44 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:53:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:44 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:53:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:44 smithi106 ceph-mon[44423]: osdmap e986: 12 total, 11 up, 11 in 2024-04-20T17:53:45.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:44 smithi106 ceph-mon[44423]: pgmap v2605: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:45.855 INFO:teuthology.orchestra.run.smithi005.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:53:46.198 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:46 smithi171 ceph-mon[44189]: 1.1 scrub starts 2024-04-20T17:53:46.198 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:46 smithi171 ceph-mon[44189]: 1.1 scrub ok 2024-04-20T17:53:46.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:46 smithi005 ceph-mon[39562]: 1.1 scrub starts 2024-04-20T17:53:46.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:46 smithi005 ceph-mon[39562]: 1.1 scrub ok 2024-04-20T17:53:46.469 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:53:46.469 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:53:46.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:46 smithi106 ceph-mon[44423]: 1.1 scrub starts 2024-04-20T17:53:46.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:46 smithi106 ceph-mon[44423]: 1.1 scrub ok 2024-04-20T17:53:47.151 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-20T17:53:47.420 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:53:47.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:47 smithi005 ceph-mon[39562]: pgmap v2606: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:47.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:47 smithi106 ceph-mon[44423]: pgmap v2606: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:47.584 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:47 smithi171 ceph-mon[44189]: pgmap v2606: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:48.925 INFO:teuthology.orchestra.run.smithi005.stderr:Exporting 1.2 info 1.2( v 15'137 (0'0,15'137] local-lis/les=846/847 n=0 ec=818/14 lis/c=846/846 les/c/f=847/847/0 sis=846) 2024-04-20T17:53:48.996 INFO:teuthology.orchestra.run.smithi005.stderr:Export successful 2024-04-20T17:53:49.121 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:53:49.122 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:53:49.756 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-20T17:53:49.899 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:49 smithi005 ceph-mon[39562]: pgmap v2607: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:49.931 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.0 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --force --op remove --pgid 1.2 2024-04-20T17:53:50.025 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:53:50.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:49 smithi171 ceph-mon[44189]: pgmap v2607: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:50.155 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.0/config 2024-04-20T17:53:50.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:49 smithi106 ceph-mon[44423]: pgmap v2607: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:51.453 INFO:teuthology.orchestra.run.smithi005.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:53:51.736 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:53:51.736 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:53:51.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:51 smithi005 ceph-mon[39562]: pgmap v2608: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:51 smithi171 ceph-mon[44189]: pgmap v2608: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:52.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:51 smithi106 ceph-mon[44423]: pgmap v2608: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:52.434 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-20T17:53:52.701 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:53:54.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:53 smithi171 ceph-mon[44189]: pgmap v2609: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:54.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:53 smithi005 ceph-mon[39562]: pgmap v2609: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:53 smithi106 ceph-mon[44423]: pgmap v2609: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:54.416 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:53:54.416 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:53:55.061 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:54 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:53:55.061 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:54 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:53:55.088 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-20T17:53:55.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:54 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:53:55.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:54 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:53:55.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:54 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:53:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:54 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:53:55.356 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:53:56.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:55 smithi005 ceph-mon[39562]: pgmap v2610: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:56.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:55 smithi005 ceph-mon[39562]: osdmap e987: 12 total, 11 up, 11 in 2024-04-20T17:53:56.211 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:55 smithi171 ceph-mon[44189]: pgmap v2610: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:56.212 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:55 smithi171 ceph-mon[44189]: osdmap e987: 12 total, 11 up, 11 in 2024-04-20T17:53:56.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:55 smithi106 ceph-mon[44423]: pgmap v2610: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:56.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:55 smithi106 ceph-mon[44423]: osdmap e987: 12 total, 11 up, 11 in 2024-04-20T17:53:57.094 INFO:teuthology.orchestra.run.smithi005.stdout: marking collection for removal 2024-04-20T17:53:57.094 INFO:teuthology.orchestra.run.smithi005.stdout:setting '_remove' omap key 2024-04-20T17:53:57.106 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:53:57.106 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:53:57.115 INFO:teuthology.orchestra.run.smithi005.stdout:finish_remove_pgs 1.2_head removing 1.2 2024-04-20T17:53:57.163 INFO:teuthology.orchestra.run.smithi005.stdout:Remove successful 2024-04-20T17:53:57.817 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-20T17:53:57.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:57 smithi005 ceph-mon[39562]: pgmap v2612: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:57.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:57 smithi005 ceph-mon[39562]: 1.6 scrub starts 2024-04-20T17:53:58.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:57 smithi171 ceph-mon[44189]: pgmap v2612: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:58.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:57 smithi171 ceph-mon[44189]: 1.6 scrub starts 2024-04-20T17:53:58.084 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:53:58.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:57 smithi106 ceph-mon[44423]: pgmap v2612: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:53:58.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:57 smithi106 ceph-mon[44423]: 1.6 scrub starts 2024-04-20T17:53:58.463 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell -n osd.0 --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.1.2.0 2024-04-20T17:53:58.685 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/osd.0/config 2024-04-20T17:53:59.184 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:58 smithi171 ceph-mon[44189]: 1.6 scrub ok 2024-04-20T17:53:59.205 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:58 smithi005 ceph-mon[39562]: 1.6 scrub ok 2024-04-20T17:53:59.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:58 smithi106 ceph-mon[44423]: 1.6 scrub ok 2024-04-20T17:53:59.868 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:53:59.868 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:53:59.922 INFO:teuthology.orchestra.run.smithi005.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-20T17:54:00.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:59 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:54:00.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:59 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:54:00.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:59 smithi005 ceph-mon[39562]: pgmap v2613: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:00.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:53:59 smithi005 ceph-mon[39562]: osdmap e988: 12 total, 11 up, 11 in 2024-04-20T17:54:00.209 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:59 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:54:00.209 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:59 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:54:00.209 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:59 smithi171 ceph-mon[44189]: pgmap v2613: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:00.209 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:53:59 smithi171 ceph-mon[44189]: osdmap e988: 12 total, 11 up, 11 in 2024-04-20T17:54:00.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:59 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:54:00.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:59 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:54:00.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:59 smithi106 ceph-mon[44423]: pgmap v2613: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:00.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:53:59 smithi106 ceph-mon[44423]: osdmap e988: 12 total, 11 up, 11 in 2024-04-20T17:54:00.534 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-20T17:54:00.638 INFO:teuthology.orchestra.run.smithi005.stdout:get_pg_num_history pg_num_history pg_num_history(e975 pg_nums {1={818=11},9={894=16}} deleted_pools ) 2024-04-20T17:54:00.638 INFO:teuthology.orchestra.run.smithi005.stdout:Importing pgid 1.2 2024-04-20T17:54:00.638 INFO:teuthology.orchestra.run.smithi005.stdout:write_pg epoch 951 info 1.2( v 15'137 (0'0,15'137] local-lis/les=846/847 n=0 ec=818/14 lis/c=846/846 les/c/f=847/847/0 sis=846) 2024-04-20T17:54:00.638 INFO:teuthology.orchestra.run.smithi005.stdout:Import successful 2024-04-20T17:54:00.800 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:54:01.801 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/exp.1.2.0 2024-04-20T17:54:01.835 INFO:tasks.thrashosds.thrasher:Waiting for clean again 2024-04-20T17:54:01.835 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-20T17:54:01.835 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:54:02.013 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:01 smithi171 ceph-mon[44189]: pgmap v2615: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:02.151 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:54:02.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:01 smithi005 ceph-mon[39562]: pgmap v2615: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:02.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:01 smithi106 ceph-mon[44423]: pgmap v2615: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:02.605 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:54:02.605 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:54:03.222 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-20T17:54:03.489 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:54:04.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:03 smithi171 ceph-mon[44189]: pgmap v2616: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:04.151 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:54:04.153 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:54:04.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:03 smithi005 ceph-mon[39562]: pgmap v2616: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:04.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:03 smithi106 ceph-mon[44423]: pgmap v2616: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:04.778 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":2616,"stamp":"2024-04-20T17:54:02.757786+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1507,"ondisk_log_size":1507,"up":22,"acting":22,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":6240736,"kb_used_data":8668,"kb_used_omap":402,"kb_used_meta":6230573,"kb_avail":1024955936,"statfs":{"total":1055945392128,"available":1049554878464,"internally_reserved":0,"allocated":8876032,"data_stored":4844743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":412419,"internal_metadata":6380107005},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001749"},"pg_stats":[{"pgid":"1.6","version":"15'137","reported_seq":1044,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-04-20T17:53:59.004407+0000","last_change":"2024-04-20T17:53:57.201156+0000","last_active":"2024-04-20T17:53:59.004407+0000","last_peered":"2024-04-20T17:53:59.004407+0000","last_clean":"2024-04-20T17:53:59.004407+0000","last_became_active":"2024-04-20T17:45:32.670941+0000","last_became_peered":"2024-04-20T17:45:32.670941+0000","last_unstale":"2024-04-20T17:53:59.004407+0000","last_undegraded":"2024-04-20T17:53:59.004407+0000","last_fullsized":"2024-04-20T17:53:59.004407+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:53:57.201032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:53:57.201032+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:55:17.206875+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"15'137","reported_seq":1041,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-04-20T17:53:59.299448+0000","last_change":"2024-04-20T17:52:51.613348+0000","last_active":"2024-04-20T17:53:59.299448+0000","last_peered":"2024-04-20T17:53:59.299448+0000","last_clean":"2024-04-20T17:53:59.299448+0000","last_became_active":"2024-04-20T17:45:29.817528+0000","last_became_peered":"2024-04-20T17:45:29.817528+0000","last_unstale":"2024-04-20T17:53:59.299448+0000","last_undegraded":"2024-04-20T17:53:59.299448+0000","last_fullsized":"2024-04-20T17:53:59.299448+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:52:51.613156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:52:51.613156+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:54:17.992704+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"15'137","reported_seq":1068,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-04-20T17:53:58.937907+0000","last_change":"2024-04-20T17:52:49.667119+0000","last_active":"2024-04-20T17:53:58.937907+0000","last_peered":"2024-04-20T17:53:58.937907+0000","last_clean":"2024-04-20T17:53:58.937907+0000","last_became_active":"2024-04-20T17:45:29.827577+0000","last_became_peered":"2024-04-20T17:45:29.827577+0000","last_unstale":"2024-04-20T17:53:58.937907+0000","last_undegraded":"2024-04-20T17:53:58.937907+0000","last_fullsized":"2024-04-20T17:53:58.937907+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:52:49.666979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:52:49.666979+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:54:07.562551+0000","scrub_duration":246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"15'137","reported_seq":1037,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-04-20T17:53:59.004405+0000","last_change":"2024-04-20T17:53:12.851930+0000","last_active":"2024-04-20T17:53:59.004405+0000","last_peered":"2024-04-20T17:53:59.004405+0000","last_clean":"2024-04-20T17:53:59.004405+0000","last_became_active":"2024-04-20T17:45:32.666523+0000","last_became_peered":"2024-04-20T17:45:32.666523+0000","last_unstale":"2024-04-20T17:53:59.004405+0000","last_undegraded":"2024-04-20T17:53:59.004405+0000","last_fullsized":"2024-04-20T17:53:59.004405+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:53:12.851797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:53:12.851797+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:54:26.388410+0000","scrub_duration":166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"15'137","reported_seq":1045,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-04-20T17:53:58.871323+0000","last_change":"2024-04-20T17:53:41.217702+0000","last_active":"2024-04-20T17:53:58.871323+0000","last_peered":"2024-04-20T17:53:58.871323+0000","last_clean":"2024-04-20T17:53:58.871323+0000","last_became_active":"2024-04-20T17:53:39.199785+0000","last_became_peered":"2024-04-20T17:53:39.199785+0000","last_unstale":"2024-04-20T17:53:58.871323+0000","last_undegraded":"2024-04-20T17:53:58.871323+0000","last_fullsized":"2024-04-20T17:53:58.871323+0000","mapping_epoch":984,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":985,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:53:41.217595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:53:41.217595+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:55:01.586495+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"15'137","reported_seq":827,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-04-20T17:53:55.592463+0000","last_change":"2024-04-20T17:52:38.639201+0000","last_active":"2024-04-20T17:53:55.592463+0000","last_peered":"2024-04-20T17:53:55.592463+0000","last_clean":"2024-04-20T17:53:55.592463+0000","last_became_active":"2024-04-20T17:49:46.824559+0000","last_became_peered":"2024-04-20T17:49:46.824559+0000","last_unstale":"2024-04-20T17:53:55.592463+0000","last_undegraded":"2024-04-20T17:53:55.592463+0000","last_fullsized":"2024-04-20T17:53:55.592463+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:52:38.639077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:52:38.639077+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:54:06.972106+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"15'137","reported_seq":1050,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-04-20T17:53:59.004399+0000","last_change":"2024-04-20T17:53:45.218447+0000","last_active":"2024-04-20T17:53:59.004399+0000","last_peered":"2024-04-20T17:53:59.004399+0000","last_clean":"2024-04-20T17:53:59.004399+0000","last_became_active":"2024-04-20T17:52:45.966123+0000","last_became_peered":"2024-04-20T17:52:45.966123+0000","last_unstale":"2024-04-20T17:53:59.004399+0000","last_undegraded":"2024-04-20T17:53:59.004399+0000","last_fullsized":"2024-04-20T17:53:59.004399+0000","mapping_epoch":969,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":970,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:53:45.218291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:53:45.218291+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:55:00.682694+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.a","version":"15'137","reported_seq":1021,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-04-20T17:53:59.299297+0000","last_change":"2024-04-20T17:52:55.793507+0000","last_active":"2024-04-20T17:53:59.299297+0000","last_peered":"2024-04-20T17:53:59.299297+0000","last_clean":"2024-04-20T17:53:59.299297+0000","last_became_active":"2024-04-20T17:45:32.669451+0000","last_became_peered":"2024-04-20T17:45:32.669451+0000","last_unstale":"2024-04-20T17:53:59.299297+0000","last_undegraded":"2024-04-20T17:53:59.299297+0000","last_fullsized":"2024-04-20T17:53:59.299297+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:52:55.793368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:52:55.793368+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:54:06.988011+0000","scrub_duration":121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"15'137","reported_seq":1037,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-04-20T17:53:59.299305+0000","last_change":"2024-04-20T17:52:47.034007+0000","last_active":"2024-04-20T17:53:59.299305+0000","last_peered":"2024-04-20T17:53:59.299305+0000","last_clean":"2024-04-20T17:53:59.299305+0000","last_became_active":"2024-04-20T17:45:32.644584+0000","last_became_peered":"2024-04-20T17:45:32.644584+0000","last_unstale":"2024-04-20T17:53:59.299305+0000","last_undegraded":"2024-04-20T17:53:59.299305+0000","last_fullsized":"2024-04-20T17:53:59.299305+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:52:47.033857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:52:47.033857+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:54:01.870825+0000","scrub_duration":188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"15'137","reported_seq":984,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-04-20T17:53:58.871323+0000","last_change":"2024-04-20T17:52:54.443377+0000","last_active":"2024-04-20T17:53:58.871323+0000","last_peered":"2024-04-20T17:53:58.871323+0000","last_clean":"2024-04-20T17:53:58.871323+0000","last_became_active":"2024-04-20T17:46:40.881713+0000","last_became_peered":"2024-04-20T17:46:40.881713+0000","last_unstale":"2024-04-20T17:53:58.871323+0000","last_undegraded":"2024-04-20T17:53:58.871323+0000","last_fullsized":"2024-04-20T17:53:58.871323+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:52:54.443251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:52:54.443251+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:54:11.199535+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":982,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-04-20T17:53:58.871337+0000","last_change":"2024-04-20T17:53:20.286031+0000","last_active":"2024-04-20T17:53:58.871337+0000","last_peered":"2024-04-20T17:53:58.871337+0000","last_clean":"2024-04-20T17:53:58.871337+0000","last_became_active":"2024-04-20T17:46:40.851683+0000","last_became_peered":"2024-04-20T17:46:40.851683+0000","last_unstale":"2024-04-20T17:53:58.871337+0000","last_undegraded":"2024-04-20T17:53:58.871337+0000","last_fullsized":"2024-04-20T17:53:58.871337+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:53:20.285898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:53:20.285898+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:54:33.054518+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},"log_size":1507,"ondisk_log_size":1507,"up":22,"acting":22,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991813,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":952732,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":951614,"kb_avail":92792420,"statfs":{"total":95995035648,"available":95019438080,"internally_reserved":0,"allocated":1007616,"data_stored":656066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":974453188},"hb_peers":[1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253452,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":852444,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92892708,"statfs":{"total":95995035648,"available":95122132992,"internally_reserved":0,"allocated":679936,"data_stored":328386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":872085952},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613201,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":749724,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":748926,"kb_avail":92995428,"statfs":{"total":95995035648,"available":95227318272,"internally_reserved":0,"allocated":679936,"data_stored":328386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":766900672},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809431,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":473744,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93271408,"statfs":{"total":95995035648,"available":95509921792,"internally_reserved":0,"allocated":1118208,"data_stored":714277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":799,"seq":3431678869621,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":171484,"kb_used_data":664,"kb_used_omap":11,"kb_used_meta":170676,"kb_avail":93573668,"statfs":{"total":95995035648,"available":95819436032,"internally_reserved":0,"allocated":679936,"data_stored":328394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11990,"internal_metadata":174772522},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":535,"seq":2297807503661,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":60572,"kb_used_data":664,"kb_used_omap":170,"kb_used_meta":59605,"kb_avail":93684580,"statfs":{"total":95995035648,"available":95933009920,"internally_reserved":0,"allocated":679936,"data_stored":328386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":174628,"internal_metadata":61035996},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":659,"seq":2830383448283,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":392720,"kb_used_data":772,"kb_used_omap":4,"kb_used_meta":391931,"kb_avail":93352432,"statfs":{"total":95995035648,"available":95592890368,"internally_reserved":0,"allocated":790528,"data_stored":386597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4410,"internal_metadata":401338054},"hb_peers":[2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394295,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":438116,"kb_used_data":864,"kb_used_omap":202,"kb_used_meta":436917,"kb_avail":93307036,"statfs":{"total":95995035648,"available":95546404864,"internally_reserved":0,"allocated":884736,"data_stored":529634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":447403479},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653344,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":623184,"kb_used_data":772,"kb_used_omap":4,"kb_used_meta":622395,"kb_avail":93121968,"statfs":{"total":95995035648,"available":95356895232,"internally_reserved":0,"allocated":790528,"data_stored":386597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":637333191},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986793,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":623388,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93121764,"statfs":{"total":95995035648,"available":95356686336,"internally_reserved":0,"allocated":679936,"data_stored":328386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790563,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":902628,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":92842524,"statfs":{"total":95995035648,"available":95070744576,"internally_reserved":0,"allocated":884736,"data_stored":529634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":923269573},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:54:04.780 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:54:04.967 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:04 smithi171 ceph-mon[44189]: 1.8 scrub starts 2024-04-20T17:54:04.967 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:04 smithi171 ceph-mon[44189]: 1.8 scrub ok 2024-04-20T17:54:05.057 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:54:05.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:04 smithi005 ceph-mon[39562]: 1.8 scrub starts 2024-04-20T17:54:05.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:04 smithi005 ceph-mon[39562]: 1.8 scrub ok 2024-04-20T17:54:05.242 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:54:05.242 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:54:05.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:04 smithi106 ceph-mon[44423]: 1.8 scrub starts 2024-04-20T17:54:05.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:04 smithi106 ceph-mon[44423]: 1.8 scrub ok 2024-04-20T17:54:05.924 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:05 smithi171 ceph-mon[44189]: from='client.18204 v1:172.21.15.5:0/4264840134' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:54:05.924 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:05 smithi171 ceph-mon[44189]: pgmap v2617: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:05.952 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-20T17:54:06.190 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:05 smithi005 ceph-mon[39562]: from='client.18204 v1:172.21.15.5:0/4264840134' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:54:06.191 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:05 smithi005 ceph-mon[39562]: pgmap v2617: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:06.217 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:54:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:05 smithi106 ceph-mon[44423]: from='client.18204 v1:172.21.15.5:0/4264840134' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:54:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:05 smithi106 ceph-mon[44423]: pgmap v2617: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:06.985 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:54:06.987 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:54:07.727 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":2618,"stamp":"2024-04-20T17:54:06.758577+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1507,"ondisk_log_size":1507,"up":22,"acting":22,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":6240648,"kb_used_data":8344,"kb_used_omap":402,"kb_used_meta":6230573,"kb_avail":1024956024,"statfs":{"total":1055945392128,"available":1049554968576,"internally_reserved":0,"allocated":8544256,"data_stored":4670110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":412419,"internal_metadata":6380107005},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002223"},"pg_stats":[{"pgid":"1.6","version":"15'137","reported_seq":1044,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-04-20T17:53:59.004407+0000","last_change":"2024-04-20T17:53:57.201156+0000","last_active":"2024-04-20T17:53:59.004407+0000","last_peered":"2024-04-20T17:53:59.004407+0000","last_clean":"2024-04-20T17:53:59.004407+0000","last_became_active":"2024-04-20T17:45:32.670941+0000","last_became_peered":"2024-04-20T17:45:32.670941+0000","last_unstale":"2024-04-20T17:53:59.004407+0000","last_undegraded":"2024-04-20T17:53:59.004407+0000","last_fullsized":"2024-04-20T17:53:59.004407+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:53:57.201032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:53:57.201032+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:55:17.206875+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"15'137","reported_seq":1041,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-04-20T17:53:59.299448+0000","last_change":"2024-04-20T17:52:51.613348+0000","last_active":"2024-04-20T17:53:59.299448+0000","last_peered":"2024-04-20T17:53:59.299448+0000","last_clean":"2024-04-20T17:53:59.299448+0000","last_became_active":"2024-04-20T17:45:29.817528+0000","last_became_peered":"2024-04-20T17:45:29.817528+0000","last_unstale":"2024-04-20T17:53:59.299448+0000","last_undegraded":"2024-04-20T17:53:59.299448+0000","last_fullsized":"2024-04-20T17:53:59.299448+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:52:51.613156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:52:51.613156+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:54:17.992704+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"15'137","reported_seq":1068,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-04-20T17:53:58.937907+0000","last_change":"2024-04-20T17:52:49.667119+0000","last_active":"2024-04-20T17:53:58.937907+0000","last_peered":"2024-04-20T17:53:58.937907+0000","last_clean":"2024-04-20T17:53:58.937907+0000","last_became_active":"2024-04-20T17:45:29.827577+0000","last_became_peered":"2024-04-20T17:45:29.827577+0000","last_unstale":"2024-04-20T17:53:58.937907+0000","last_undegraded":"2024-04-20T17:53:58.937907+0000","last_fullsized":"2024-04-20T17:53:58.937907+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:52:49.666979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:52:49.666979+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:54:07.562551+0000","scrub_duration":246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"15'137","reported_seq":1037,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-04-20T17:53:59.004405+0000","last_change":"2024-04-20T17:53:12.851930+0000","last_active":"2024-04-20T17:53:59.004405+0000","last_peered":"2024-04-20T17:53:59.004405+0000","last_clean":"2024-04-20T17:53:59.004405+0000","last_became_active":"2024-04-20T17:45:32.666523+0000","last_became_peered":"2024-04-20T17:45:32.666523+0000","last_unstale":"2024-04-20T17:53:59.004405+0000","last_undegraded":"2024-04-20T17:53:59.004405+0000","last_fullsized":"2024-04-20T17:53:59.004405+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:53:12.851797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:53:12.851797+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:54:26.388410+0000","scrub_duration":166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"15'137","reported_seq":1045,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-04-20T17:53:58.871323+0000","last_change":"2024-04-20T17:53:41.217702+0000","last_active":"2024-04-20T17:53:58.871323+0000","last_peered":"2024-04-20T17:53:58.871323+0000","last_clean":"2024-04-20T17:53:58.871323+0000","last_became_active":"2024-04-20T17:53:39.199785+0000","last_became_peered":"2024-04-20T17:53:39.199785+0000","last_unstale":"2024-04-20T17:53:58.871323+0000","last_undegraded":"2024-04-20T17:53:58.871323+0000","last_fullsized":"2024-04-20T17:53:58.871323+0000","mapping_epoch":984,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":985,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:53:41.217595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:53:41.217595+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:55:01.586495+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"15'137","reported_seq":829,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-04-20T17:53:59.298977+0000","last_change":"2024-04-20T17:52:38.639201+0000","last_active":"2024-04-20T17:53:59.298977+0000","last_peered":"2024-04-20T17:53:59.298977+0000","last_clean":"2024-04-20T17:53:59.298977+0000","last_became_active":"2024-04-20T17:49:46.824559+0000","last_became_peered":"2024-04-20T17:49:46.824559+0000","last_unstale":"2024-04-20T17:53:59.298977+0000","last_undegraded":"2024-04-20T17:53:59.298977+0000","last_fullsized":"2024-04-20T17:53:59.298977+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:52:38.639077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:52:38.639077+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:54:06.972106+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"15'137","reported_seq":1050,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-04-20T17:53:59.004399+0000","last_change":"2024-04-20T17:53:45.218447+0000","last_active":"2024-04-20T17:53:59.004399+0000","last_peered":"2024-04-20T17:53:59.004399+0000","last_clean":"2024-04-20T17:53:59.004399+0000","last_became_active":"2024-04-20T17:52:45.966123+0000","last_became_peered":"2024-04-20T17:52:45.966123+0000","last_unstale":"2024-04-20T17:53:59.004399+0000","last_undegraded":"2024-04-20T17:53:59.004399+0000","last_fullsized":"2024-04-20T17:53:59.004399+0000","mapping_epoch":969,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":970,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:53:45.218291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:53:45.218291+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:55:00.682694+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.a","version":"15'137","reported_seq":1021,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-04-20T17:53:59.299297+0000","last_change":"2024-04-20T17:52:55.793507+0000","last_active":"2024-04-20T17:53:59.299297+0000","last_peered":"2024-04-20T17:53:59.299297+0000","last_clean":"2024-04-20T17:53:59.299297+0000","last_became_active":"2024-04-20T17:45:32.669451+0000","last_became_peered":"2024-04-20T17:45:32.669451+0000","last_unstale":"2024-04-20T17:53:59.299297+0000","last_undegraded":"2024-04-20T17:53:59.299297+0000","last_fullsized":"2024-04-20T17:53:59.299297+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:52:55.793368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:52:55.793368+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:54:06.988011+0000","scrub_duration":121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"15'137","reported_seq":1045,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-04-20T17:54:04.143232+0000","last_change":"2024-04-20T17:54:04.143260+0000","last_active":"2024-04-20T17:54:04.143232+0000","last_peered":"2024-04-20T17:54:04.143232+0000","last_clean":"2024-04-20T17:54:04.143232+0000","last_became_active":"2024-04-20T17:45:32.644584+0000","last_became_peered":"2024-04-20T17:45:32.644584+0000","last_unstale":"2024-04-20T17:54:04.143232+0000","last_undegraded":"2024-04-20T17:54:04.143232+0000","last_fullsized":"2024-04-20T17:54:04.143232+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:54:04.143147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:54:04.143147+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:55:18.902933+0000","scrub_duration":165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"15'137","reported_seq":984,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-04-20T17:53:58.871323+0000","last_change":"2024-04-20T17:52:54.443377+0000","last_active":"2024-04-20T17:53:58.871323+0000","last_peered":"2024-04-20T17:53:58.871323+0000","last_clean":"2024-04-20T17:53:58.871323+0000","last_became_active":"2024-04-20T17:46:40.881713+0000","last_became_peered":"2024-04-20T17:46:40.881713+0000","last_unstale":"2024-04-20T17:53:58.871323+0000","last_undegraded":"2024-04-20T17:53:58.871323+0000","last_fullsized":"2024-04-20T17:53:58.871323+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:52:54.443251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:52:54.443251+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:54:11.199535+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":982,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-04-20T17:53:58.871337+0000","last_change":"2024-04-20T17:53:20.286031+0000","last_active":"2024-04-20T17:53:58.871337+0000","last_peered":"2024-04-20T17:53:58.871337+0000","last_clean":"2024-04-20T17:53:58.871337+0000","last_became_active":"2024-04-20T17:46:40.851683+0000","last_became_peered":"2024-04-20T17:46:40.851683+0000","last_unstale":"2024-04-20T17:53:58.871337+0000","last_undegraded":"2024-04-20T17:53:58.871337+0000","last_fullsized":"2024-04-20T17:53:58.871337+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:53:20.285898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:53:20.285898+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:54:33.054518+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},"log_size":1507,"ondisk_log_size":1507,"up":22,"acting":22,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991814,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":952732,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":951614,"kb_avail":92792420,"statfs":{"total":95995035648,"available":95019438080,"internally_reserved":0,"allocated":1007616,"data_stored":656066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":974453188},"hb_peers":[1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253453,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":852444,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92892708,"statfs":{"total":95995035648,"available":95122132992,"internally_reserved":0,"allocated":679936,"data_stored":328386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":872085952},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613201,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":749724,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":748926,"kb_avail":92995428,"statfs":{"total":95995035648,"available":95227318272,"internally_reserved":0,"allocated":679936,"data_stored":328386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":766900672},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809432,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":473756,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93271396,"statfs":{"total":95995035648,"available":95509909504,"internally_reserved":0,"allocated":1007616,"data_stored":656066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":799,"seq":3431678869622,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":171484,"kb_used_data":664,"kb_used_omap":11,"kb_used_meta":170676,"kb_avail":93573668,"statfs":{"total":95995035648,"available":95819436032,"internally_reserved":0,"allocated":679936,"data_stored":328394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11990,"internal_metadata":174772522},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":535,"seq":2297807503661,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":60572,"kb_used_data":664,"kb_used_omap":170,"kb_used_meta":59605,"kb_avail":93684580,"statfs":{"total":95995035648,"available":95933009920,"internally_reserved":0,"allocated":679936,"data_stored":328386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":174628,"internal_metadata":61035996},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":659,"seq":2830383448284,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":392732,"kb_used_data":664,"kb_used_omap":4,"kb_used_meta":391931,"kb_avail":93352420,"statfs":{"total":95995035648,"available":95592878080,"internally_reserved":0,"allocated":679936,"data_stored":328386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4410,"internal_metadata":401338054},"hb_peers":[2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394296,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":437992,"kb_used_data":864,"kb_used_omap":202,"kb_used_meta":436917,"kb_avail":93307160,"statfs":{"total":95995035648,"available":95546531840,"internally_reserved":0,"allocated":884736,"data_stored":529634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":447403479},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653345,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":623196,"kb_used_data":664,"kb_used_omap":4,"kb_used_meta":622395,"kb_avail":93121956,"statfs":{"total":95995035648,"available":95356882944,"internally_reserved":0,"allocated":679936,"data_stored":328386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":637333191},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986794,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":623388,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93121764,"statfs":{"total":95995035648,"available":95356686336,"internally_reserved":0,"allocated":679936,"data_stored":328386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790564,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":902628,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":92842524,"statfs":{"total":95995035648,"available":95070744576,"internally_reserved":0,"allocated":884736,"data_stored":529634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":923269573},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:54:07.728 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-20T17:54:07.728 INFO:tasks.thrashosds.thrasher:Waiting for trim 2024-04-20T17:54:07.940 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:54:07.940 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:54:07.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:07 smithi005 ceph-mon[39562]: pgmap v2618: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:07.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:07 smithi005 ceph-mon[39562]: 1.3 scrub starts 2024-04-20T17:54:07.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:07 smithi005 ceph-mon[39562]: 1.3 scrub ok 2024-04-20T17:54:07.964 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:07 smithi171 ceph-mon[44189]: pgmap v2618: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:07.964 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:07 smithi171 ceph-mon[44189]: 1.3 scrub starts 2024-04-20T17:54:07.964 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:07 smithi171 ceph-mon[44189]: 1.3 scrub ok 2024-04-20T17:54:08.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:07 smithi106 ceph-mon[44423]: pgmap v2618: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:07 smithi106 ceph-mon[44423]: 1.3 scrub starts 2024-04-20T17:54:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:07 smithi106 ceph-mon[44423]: 1.3 scrub ok 2024-04-20T17:54:08.691 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-20T17:54:08.960 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.c/config 2024-04-20T17:54:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:08 smithi171 ceph-mon[44189]: from='client.18210 v1:172.21.15.5:0/3803181445' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:54:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:08 smithi171 ceph-mon[44189]: 1.4 scrub starts 2024-04-20T17:54:09.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:08 smithi171 ceph-mon[44189]: 1.4 scrub ok 2024-04-20T17:54:09.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:08 smithi005 ceph-mon[39562]: from='client.18210 v1:172.21.15.5:0/3803181445' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:54:09.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:08 smithi005 ceph-mon[39562]: 1.4 scrub starts 2024-04-20T17:54:09.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:08 smithi005 ceph-mon[39562]: 1.4 scrub ok 2024-04-20T17:54:09.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:08 smithi106 ceph-mon[44423]: from='client.18210 v1:172.21.15.5:0/3803181445' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:54:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:08 smithi106 ceph-mon[44423]: 1.4 scrub starts 2024-04-20T17:54:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:08 smithi106 ceph-mon[44423]: 1.4 scrub ok 2024-04-20T17:54:10.210 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:09 smithi171 ceph-mon[44189]: pgmap v2619: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:10.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:09 smithi005 ceph-mon[39562]: pgmap v2619: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:10.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:09 smithi106 ceph-mon[44423]: pgmap v2619: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:10.834 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-04-20T17:54:10.834 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:54:11.178 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:10 smithi171 ceph-mon[44189]: 1.a scrub starts 2024-04-20T17:54:11.178 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:10 smithi171 ceph-mon[44189]: 1.a scrub ok 2024-04-20T17:54:11.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:10 smithi005 ceph-mon[39562]: 1.a scrub starts 2024-04-20T17:54:11.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:10 smithi005 ceph-mon[39562]: 1.a scrub ok 2024-04-20T17:54:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:10 smithi106 ceph-mon[44423]: 1.a scrub starts 2024-04-20T17:54:11.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:10 smithi106 ceph-mon[44423]: 1.a scrub ok 2024-04-20T17:54:11.501 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-20T17:54:11.792 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:54:12.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:11 smithi005 ceph-mon[39562]: pgmap v2620: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:12.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:11 smithi106 ceph-mon[44423]: pgmap v2620: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:12.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:11 smithi171 ceph-mon[44189]: pgmap v2620: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:12.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:12 smithi005 ceph-mon[39562]: 1.9 scrub starts 2024-04-20T17:54:13.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:12 smithi106 ceph-mon[44423]: 1.9 scrub starts 2024-04-20T17:54:13.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:12 smithi171 ceph-mon[44189]: 1.9 scrub starts 2024-04-20T17:54:13.734 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:54:13.734 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:54:14.064 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:13 smithi005 ceph-mon[39562]: 1.9 scrub ok 2024-04-20T17:54:14.064 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:13 smithi005 ceph-mon[39562]: pgmap v2621: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:14.064 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:13 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:54:14.064 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:13 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:54:14.064 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:13 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:54:14.064 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:13 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:54:14.064 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:13 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:54:14.065 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:13 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:54:14.065 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:13 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:54:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:13 smithi106 ceph-mon[44423]: 1.9 scrub ok 2024-04-20T17:54:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:13 smithi106 ceph-mon[44423]: pgmap v2621: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:13 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:54:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:13 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:54:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:13 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:54:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:13 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:54:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:13 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:54:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:13 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:54:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:13 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:54:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:13 smithi171 ceph-mon[44189]: 1.9 scrub ok 2024-04-20T17:54:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:13 smithi171 ceph-mon[44189]: pgmap v2621: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:13 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-20T17:54:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:13 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-20T17:54:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:13 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-20T17:54:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:13 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:54:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:13 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:54:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:13 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:54:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:13 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:54:14.421 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-20T17:54:14.703 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:54:16.109 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:15 smithi005 ceph-mon[39562]: pgmap v2622: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:15 smithi106 ceph-mon[44423]: pgmap v2622: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:16.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:15 smithi171 ceph-mon[44189]: pgmap v2622: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:16.348 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:54:16.348 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:54:16.970 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-20T17:54:17.247 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:54:18.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:17 smithi005 ceph-mon[39562]: pgmap v2623: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:18.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:17 smithi106 ceph-mon[44423]: pgmap v2623: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:17 smithi171 ceph-mon[44189]: pgmap v2623: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:19.221 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:54:19.221 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:54:19.955 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-20T17:54:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:19 smithi005 ceph-mon[39562]: pgmap v2624: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:20.241 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:54:20.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:19 smithi106 ceph-mon[44423]: pgmap v2624: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:20.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:19 smithi171 ceph-mon[44189]: pgmap v2624: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:22.029 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:21 smithi106 ceph-mon[44423]: pgmap v2625: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:22.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:21 smithi005 ceph-mon[39562]: pgmap v2625: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:22.304 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:54:22.305 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:54:22.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:21 smithi171 ceph-mon[44189]: pgmap v2625: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:22.970 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-20T17:54:23.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:22 smithi005 ceph-mon[39562]: 1.7 scrub starts 2024-04-20T17:54:23.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:22 smithi005 ceph-mon[39562]: 1.7 scrub ok 2024-04-20T17:54:23.234 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:54:23.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:22 smithi106 ceph-mon[44423]: 1.7 scrub starts 2024-04-20T17:54:23.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:22 smithi106 ceph-mon[44423]: 1.7 scrub ok 2024-04-20T17:54:23.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:22 smithi171 ceph-mon[44189]: 1.7 scrub starts 2024-04-20T17:54:23.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:22 smithi171 ceph-mon[44189]: 1.7 scrub ok 2024-04-20T17:54:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:23 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:54:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:23 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:54:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:23 smithi005 ceph-mon[39562]: pgmap v2626: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:23 smithi005 ceph-mon[39562]: osdmap e989: 12 total, 11 up, 11 in 2024-04-20T17:54:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:23 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:54:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:23 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:54:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:23 smithi106 ceph-mon[44423]: pgmap v2626: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:23 smithi106 ceph-mon[44423]: osdmap e989: 12 total, 11 up, 11 in 2024-04-20T17:54:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:23 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:54:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:23 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:54:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:23 smithi171 ceph-mon[44189]: pgmap v2626: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:23 smithi171 ceph-mon[44189]: osdmap e989: 12 total, 11 up, 11 in 2024-04-20T17:54:25.047 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:54:25.047 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:54:25.694 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-20T17:54:25.957 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:54:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:25 smithi005 ceph-mon[39562]: pgmap v2628: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:26.228 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:25 smithi106 ceph-mon[44423]: pgmap v2628: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:25 smithi171 ceph-mon[44189]: pgmap v2628: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:27.795 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:54:27.795 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:54:28.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:27 smithi005 ceph-mon[39562]: pgmap v2629: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:28.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:27 smithi005 ceph-mon[39562]: 1.5 scrub starts 2024-04-20T17:54:28.248 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:27 smithi106 ceph-mon[44423]: pgmap v2629: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:28.248 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:27 smithi106 ceph-mon[44423]: 1.5 scrub starts 2024-04-20T17:54:28.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:27 smithi171 ceph-mon[44189]: pgmap v2629: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:27 smithi171 ceph-mon[44189]: 1.5 scrub starts 2024-04-20T17:54:28.538 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-20T17:54:28.831 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:54:29.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:28 smithi005 ceph-mon[39562]: 1.5 scrub ok 2024-04-20T17:54:29.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:28 smithi106 ceph-mon[44423]: 1.5 scrub ok 2024-04-20T17:54:29.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:28 smithi171 ceph-mon[44189]: 1.5 scrub ok 2024-04-20T17:54:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:29 smithi005 ceph-mon[39562]: pgmap v2630: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:30.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:29 smithi106 ceph-mon[44423]: pgmap v2630: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:30.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:29 smithi171 ceph-mon[44189]: pgmap v2630: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:30.570 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:54:30.570 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:54:31.359 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-20T17:54:31.640 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:54:32.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:31 smithi005 ceph-mon[39562]: pgmap v2631: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:32.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:31 smithi106 ceph-mon[44423]: pgmap v2631: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:31 smithi171 ceph-mon[44189]: pgmap v2631: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:33.143 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:32 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:54:33.143 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:32 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:54:33.143 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:32 smithi005 ceph-mon[39562]: osdmap e990: 12 total, 11 up, 11 in 2024-04-20T17:54:33.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:32 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:54:33.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:32 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:54:33.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:32 smithi106 ceph-mon[44423]: osdmap e990: 12 total, 11 up, 11 in 2024-04-20T17:54:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:32 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:54:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:32 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:54:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:32 smithi171 ceph-mon[44189]: osdmap e990: 12 total, 11 up, 11 in 2024-04-20T17:54:33.396 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:54:33.396 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:54:34.109 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:33 smithi005 ceph-mon[39562]: pgmap v2633: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:34.140 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-20T17:54:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:33 smithi106 ceph-mon[44423]: pgmap v2633: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:34.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:33 smithi171 ceph-mon[44189]: pgmap v2633: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:34.421 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:54:34.917 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:34 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:54:34.917 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:34 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:54:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:34 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:54:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:34 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:54:35.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:34 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:54:35.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:34 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:54:35.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:35 smithi005 ceph-mon[39562]: pgmap v2634: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:36.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:35 smithi106 ceph-mon[44423]: pgmap v2634: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:36.297 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:54:36.297 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:54:36.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:35 smithi171 ceph-mon[44189]: pgmap v2634: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:36.902 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-20T17:54:37.178 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:54:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:37 smithi005 ceph-mon[39562]: pgmap v2635: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:38.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:37 smithi106 ceph-mon[44423]: pgmap v2635: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:38.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:37 smithi171 ceph-mon[44189]: pgmap v2635: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:38.916 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:54:38.916 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:54:39.684 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-20T17:54:39.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:39 smithi005 ceph-mon[39562]: pgmap v2636: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:39.964 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:54:40.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:39 smithi106 ceph-mon[44423]: pgmap v2636: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:40.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:39 smithi171 ceph-mon[44189]: pgmap v2636: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:41.784 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:54:41.784 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:54:42.128 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:41 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:54:42.129 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:41 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:54:42.129 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:41 smithi005 ceph-mon[39562]: pgmap v2637: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:42.129 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:41 smithi005 ceph-mon[39562]: osdmap e991: 12 total, 11 up, 11 in 2024-04-20T17:54:42.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:41 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:54:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:41 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:54:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:41 smithi106 ceph-mon[44423]: pgmap v2637: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:41 smithi106 ceph-mon[44423]: osdmap e991: 12 total, 11 up, 11 in 2024-04-20T17:54:42.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:41 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:54:42.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:41 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:54:42.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:41 smithi171 ceph-mon[44189]: pgmap v2637: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:42.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:41 smithi171 ceph-mon[44189]: osdmap e991: 12 total, 11 up, 11 in 2024-04-20T17:54:42.508 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-20T17:54:42.789 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:54:44.193 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:43 smithi005 ceph-mon[39562]: pgmap v2639: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:44.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:43 smithi106 ceph-mon[44423]: pgmap v2639: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:44.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:43 smithi171 ceph-mon[44189]: pgmap v2639: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:44.779 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-04-20T17:54:44.779 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:54:45.477 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-20T17:54:45.765 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:54:46.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:45 smithi005 ceph-mon[39562]: pgmap v2640: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:46.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:45 smithi106 ceph-mon[44423]: pgmap v2640: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:46.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:45 smithi171 ceph-mon[44189]: pgmap v2640: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:47.475 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:54:47.475 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:54:47.729 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-04-20T17:54:47.729 INFO:tasks.cephadm.osd.0:Restarting osd.0 (starting--it wasn't running)... 2024-04-20T17:54:47.730 DEBUG:teuthology.orchestra.run.smithi005:osd.0> sudo journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.0.service 2024-04-20T17:54:47.732 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl start ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.0 2024-04-20T17:54:47.770 INFO:journalctl@ceph.osd.0.smithi005.stdout:-- Logs begin at Sat 2024-04-20 16:47:38 UTC. -- 2024-04-20T17:54:48.171 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:54:47 smithi005 systemd[1]: Starting Ceph osd.0 for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:54:48.172 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:47 smithi005 ceph-mon[39562]: pgmap v2641: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:48.172 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:47 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:54:48.172 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:47 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:54:48.182 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:47 smithi106 ceph-mon[44423]: pgmap v2641: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:48.183 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:47 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:54:48.183 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:47 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:54:48.211 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-20T17:54:48.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:47 smithi171 ceph-mon[44189]: pgmap v2641: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:48.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:47 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:54:48.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:47 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:54:48.476 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:54:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:48 smithi106 ceph-mon[44423]: osdmap e992: 12 total, 11 up, 11 in 2024-04-20T17:54:49.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:48 smithi005 ceph-mon[39562]: osdmap e992: 12 total, 11 up, 11 in 2024-04-20T17:54:49.211 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:54:49 smithi005 bash[376278]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-20T17:54:49.211 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:54:49 smithi005 bash[376278]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-20T17:54:49.211 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:54:49 smithi005 bash[376278]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-20T17:54:49.211 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:54:49 smithi005 bash[376278]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-20T17:54:49.212 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:54:49 smithi005 bash[376278]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-20T17:54:49.212 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:54:49 smithi005 bash[376278]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-20T17:54:49.212 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:54:49 smithi005 bash[376278]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-20T17:54:49.212 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:54:49 smithi005 bash[376278]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-20T17:54:49.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:48 smithi171 ceph-mon[44189]: osdmap e992: 12 total, 11 up, 11 in 2024-04-20T17:54:50.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:49 smithi005 ceph-mon[39562]: pgmap v2643: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:50.219 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:54:50.219 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:54:50.242 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:49 smithi106 ceph-mon[44423]: pgmap v2643: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:50.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:49 smithi171 ceph-mon[44189]: pgmap v2643: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:50.360 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-20T17:54:50.641 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:54:50.711 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:54:50 smithi005 bash[376559]: 9aaf0a965ad083b5cf4f00a8c1ac64783844d2077b7967b3683efff4fb18a3ae 2024-04-20T17:54:50.711 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:54:50 smithi005 systemd[1]: Started Ceph osd.0 for 13c6e76c-ff37-11ee-bc93-c7b262605968. 2024-04-20T17:54:50.856 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-20T17:54:51.121 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.b/config 2024-04-20T17:54:51.153 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:54:50 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-0[376574]: 2024-04-20T17:54:50.796+0000 7efc37a667c0 -1 Falling back to public interface 2024-04-20T17:54:52.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:51 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:54:52.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:51 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:54:52.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:51 smithi005 ceph-mon[39562]: pgmap v2644: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:52.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:51 smithi005 ceph-mon[39562]: osdmap e993: 12 total, 11 up, 11 in 2024-04-20T17:54:52.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:51 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:54:52.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:51 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:54:52.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:51 smithi106 ceph-mon[44423]: pgmap v2644: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:52.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:51 smithi106 ceph-mon[44423]: osdmap e993: 12 total, 11 up, 11 in 2024-04-20T17:54:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:51 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:54:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:51 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:54:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:51 smithi171 ceph-mon[44189]: pgmap v2644: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:52.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:51 smithi171 ceph-mon[44189]: osdmap e993: 12 total, 11 up, 11 in 2024-04-20T17:54:52.602 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2024-04-20T17:54:52.603 INFO:teuthology.orchestra.run.smithi005.stderr:0 2024-04-20T17:54:52.603 INFO:teuthology.orchestra.run.smithi005.stderr:1 2024-04-20T17:54:52.603 INFO:teuthology.orchestra.run.smithi005.stderr:2 2024-04-20T17:54:52.603 INFO:teuthology.orchestra.run.smithi005.stderr:abort 2024-04-20T17:54:52.603 INFO:teuthology.orchestra.run.smithi005.stderr:assert 2024-04-20T17:54:52.603 INFO:teuthology.orchestra.run.smithi005.stderr:config diff 2024-04-20T17:54:52.603 INFO:teuthology.orchestra.run.smithi005.stderr:config diff get 2024-04-20T17:54:52.603 INFO:teuthology.orchestra.run.smithi005.stderr:config get 2024-04-20T17:54:52.603 INFO:teuthology.orchestra.run.smithi005.stderr:config help [] 2024-04-20T17:54:52.603 INFO:teuthology.orchestra.run.smithi005.stderr:config set ... 2024-04-20T17:54:52.603 INFO:teuthology.orchestra.run.smithi005.stderr:admin_socket: invalid command 2024-04-20T17:54:53.195 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-04-20T17:54:53.196 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T17:54:53.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:52 smithi005 ceph-mon[39562]: pgmap v2646: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:53.214 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:52 smithi106 ceph-mon[44423]: pgmap v2646: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:53.314 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-20T17:54:53.314 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-04-20T17:54:53.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:52 smithi171 ceph-mon[44189]: pgmap v2646: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:53.961 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:54:53 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-0[376574]: 2024-04-20T17:54:53.565+0000 7efc37a667c0 -1 osd.0 975 log_to_monitors true 2024-04-20T17:54:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:53 smithi106 ceph-mon[44423]: from='osd.0 v1:172.21.15.5:6801/415791178' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-20T17:54:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:53 smithi106 ceph-mon[44423]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-20T17:54:54.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:53 smithi171 ceph-mon[44189]: from='osd.0 v1:172.21.15.5:6801/415791178' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-20T17:54:54.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:53 smithi171 ceph-mon[44189]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-20T17:54:54.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:53 smithi005 ceph-mon[39562]: from='osd.0 v1:172.21.15.5:6801/415791178' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-20T17:54:54.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:53 smithi005 ceph-mon[39562]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-20T17:54:54.461 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:54:53 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-0[376574]: 2024-04-20T17:54:53.997+0000 7efc24eb6700 -1 osd.0 975 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T17:54:55.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:54 smithi106 ceph-mon[44423]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-20T17:54:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:54 smithi106 ceph-mon[44423]: osdmap e994: 12 total, 11 up, 11 in 2024-04-20T17:54:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:54 smithi106 ceph-mon[44423]: from='osd.0 v1:172.21.15.5:6801/415791178' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:54:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:54 smithi106 ceph-mon[44423]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:54:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:54 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T17:54:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:54 smithi106 ceph-mon[44423]: pgmap v2648: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:54 smithi171 ceph-mon[44189]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-20T17:54:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:54 smithi171 ceph-mon[44189]: osdmap e994: 12 total, 11 up, 11 in 2024-04-20T17:54:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:54 smithi171 ceph-mon[44189]: from='osd.0 v1:172.21.15.5:6801/415791178' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:54:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:54 smithi171 ceph-mon[44189]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:54:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:54 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T17:54:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:54 smithi171 ceph-mon[44189]: pgmap v2648: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:55.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:54 smithi005 ceph-mon[39562]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-20T17:54:55.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:54 smithi005 ceph-mon[39562]: osdmap e994: 12 total, 11 up, 11 in 2024-04-20T17:54:55.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:54 smithi005 ceph-mon[39562]: from='osd.0 v1:172.21.15.5:6801/415791178' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:54:55.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:54 smithi005 ceph-mon[39562]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-04-20T17:54:55.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:54 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T17:54:55.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:54 smithi005 ceph-mon[39562]: pgmap v2648: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:56.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:55 smithi106 ceph-mon[44423]: osd.0 v1:172.21.15.5:6801/415791178 boot 2024-04-20T17:54:56.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:55 smithi106 ceph-mon[44423]: osdmap e995: 12 total, 12 up, 11 in 2024-04-20T17:54:56.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:55 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T17:54:56.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:55 smithi171 ceph-mon[44189]: osd.0 v1:172.21.15.5:6801/415791178 boot 2024-04-20T17:54:56.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:55 smithi171 ceph-mon[44189]: osdmap e995: 12 total, 12 up, 11 in 2024-04-20T17:54:56.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:55 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T17:54:56.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:55 smithi005 ceph-mon[39562]: osd.0 v1:172.21.15.5:6801/415791178 boot 2024-04-20T17:54:56.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:55 smithi005 ceph-mon[39562]: osdmap e995: 12 total, 12 up, 11 in 2024-04-20T17:54:56.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:55 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-20T17:54:57.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:57 smithi106 ceph-mon[44423]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:54:57.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:57 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:54:57.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:57 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:54:57.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:57 smithi106 ceph-mon[44423]: osdmap e996: 12 total, 12 up, 11 in 2024-04-20T17:54:57.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:57 smithi106 ceph-mon[44423]: pgmap v2651: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:57.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:57 smithi171 ceph-mon[44189]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:54:57.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:57 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:54:57.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:57 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:54:57.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:57 smithi171 ceph-mon[44189]: osdmap e996: 12 total, 12 up, 11 in 2024-04-20T17:54:57.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:57 smithi171 ceph-mon[44189]: pgmap v2651: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:57.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:57 smithi005 ceph-mon[39562]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-20T17:54:57.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:57 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:54:57.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:57 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:54:57.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:57 smithi005 ceph-mon[39562]: osdmap e996: 12 total, 12 up, 11 in 2024-04-20T17:54:57.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:57 smithi005 ceph-mon[39562]: pgmap v2651: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:58.315 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-20T17:54:58.610 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:54:59.130 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:59 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:54:59.131 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:59 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:54:59.131 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:59 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:54:59.131 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:59 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:54:59.131 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:59 smithi005 ceph-mon[39562]: osdmap e997: 12 total, 12 up, 11 in 2024-04-20T17:54:59.131 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:59 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:54:59.131 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:59 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:54:59.131 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:54:59 smithi005 ceph-mon[39562]: pgmap v2653: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:59 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:54:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:59 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:54:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:59 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:54:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:59 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:54:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:59 smithi171 ceph-mon[44189]: osdmap e997: 12 total, 12 up, 11 in 2024-04-20T17:54:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:59 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:54:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:59 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:54:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:54:59 smithi171 ceph-mon[44189]: pgmap v2653: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:54:59.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:59 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:54:59.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:59 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:54:59.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:59 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:54:59.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:59 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:54:59.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:59 smithi106 ceph-mon[44423]: osdmap e997: 12 total, 12 up, 11 in 2024-04-20T17:54:59.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:59 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:54:59.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:59 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:54:59.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:54:59 smithi106 ceph-mon[44423]: pgmap v2653: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:55:00.152 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:00 smithi005 ceph-mon[39562]: osdmap e998: 12 total, 12 up, 11 in 2024-04-20T17:55:00.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:00 smithi171 ceph-mon[44189]: osdmap e998: 12 total, 12 up, 11 in 2024-04-20T17:55:00.447 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-04-20T17:55:00.447 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-04-20T17:55:00.447 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-04-20T17:55:00.447 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-04-20T17:55:00.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:00 smithi106 ceph-mon[44423]: osdmap e998: 12 total, 12 up, 11 in 2024-04-20T17:55:01.116 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:01 smithi005 ceph-mon[39562]: pgmap v2655: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:55:01.146 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T17:55:01.146 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-04-20T17:55:01.147 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd in 0 2024-04-20T17:55:01.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:01 smithi171 ceph-mon[44189]: pgmap v2655: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:55:01.426 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:55:01.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:01 smithi106 ceph-mon[44423]: pgmap v2655: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:55:02.915 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:02 smithi005 ceph-mon[39562]: 1.1 scrub starts 2024-04-20T17:55:02.915 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:02 smithi005 ceph-mon[39562]: 1.1 scrub ok 2024-04-20T17:55:03.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:02 smithi106 ceph-mon[44423]: 1.1 scrub starts 2024-04-20T17:55:03.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:02 smithi106 ceph-mon[44423]: 1.1 scrub ok 2024-04-20T17:55:03.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:02 smithi171 ceph-mon[44189]: 1.1 scrub starts 2024-04-20T17:55:03.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:02 smithi171 ceph-mon[44189]: 1.1 scrub ok 2024-04-20T17:55:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:03 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mon-a[39539]: 2024-04-20T17:55:03.414+0000 7f73b92a1700 -1 mon.a@0(leader).osd e998 definitely_dead 0 2024-04-20T17:55:03.880 INFO:teuthology.orchestra.run.smithi005.stderr:marked in osd.0. 2024-04-20T17:55:04.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:03 smithi005 ceph-mon[39562]: pgmap v2656: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:55:04.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:03 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3945229626' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-04-20T17:55:04.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:03 smithi106 ceph-mon[44423]: pgmap v2656: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:55:04.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:03 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3945229626' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-04-20T17:55:04.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:03 smithi171 ceph-mon[44189]: pgmap v2656: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:55:04.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:03 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3945229626' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-04-20T17:55:04.613 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-04-20T17:55:05.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:04 smithi005 ceph-mon[39562]: 1.2 scrub starts 2024-04-20T17:55:05.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:04 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3945229626' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-04-20T17:55:05.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:04 smithi005 ceph-mon[39562]: osdmap e999: 12 total, 12 up, 12 in 2024-04-20T17:55:05.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:04 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:55:05.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:04 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:55:05.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:04 smithi106 ceph-mon[44423]: 1.2 scrub starts 2024-04-20T17:55:05.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:04 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3945229626' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-04-20T17:55:05.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:04 smithi106 ceph-mon[44423]: osdmap e999: 12 total, 12 up, 12 in 2024-04-20T17:55:05.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:04 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:55:05.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:04 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:55:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:04 smithi171 ceph-mon[44189]: 1.2 scrub starts 2024-04-20T17:55:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:04 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3945229626' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-04-20T17:55:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:04 smithi171 ceph-mon[44189]: osdmap e999: 12 total, 12 up, 12 in 2024-04-20T17:55:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:04 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:55:05.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:04 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:55:06.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:05 smithi005 ceph-mon[39562]: pgmap v2658: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:55:06.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:05 smithi005 ceph-mon[39562]: osdmap e1000: 12 total, 12 up, 12 in 2024-04-20T17:55:06.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:05 smithi106 ceph-mon[44423]: pgmap v2658: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:55:06.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:05 smithi106 ceph-mon[44423]: osdmap e1000: 12 total, 12 up, 12 in 2024-04-20T17:55:06.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:05 smithi171 ceph-mon[44189]: pgmap v2658: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 977 GiB / 983 GiB avail 2024-04-20T17:55:06.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:05 smithi171 ceph-mon[44189]: osdmap e1000: 12 total, 12 up, 12 in 2024-04-20T17:55:08.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:07 smithi005 ceph-mon[39562]: pgmap v2660: 11 pgs: 1 activating, 10 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:08.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:07 smithi005 ceph-mon[39562]: 1.2 scrub starts 2024-04-20T17:55:08.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:07 smithi005 ceph-mon[39562]: 1.2 scrub ok 2024-04-20T17:55:08.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:07 smithi106 ceph-mon[44423]: pgmap v2660: 11 pgs: 1 activating, 10 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:07 smithi106 ceph-mon[44423]: 1.2 scrub starts 2024-04-20T17:55:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:07 smithi106 ceph-mon[44423]: 1.2 scrub ok 2024-04-20T17:55:08.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:07 smithi171 ceph-mon[44189]: pgmap v2660: 11 pgs: 1 activating, 10 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:07 smithi171 ceph-mon[44189]: 1.2 scrub starts 2024-04-20T17:55:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:07 smithi171 ceph-mon[44189]: 1.2 scrub ok 2024-04-20T17:55:09.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:08 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:55:09.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:08 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:55:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:08 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:55:09.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:08 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:55:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:08 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:55:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:08 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:55:09.614 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-04-20T17:55:09.908 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:55:09.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:09 smithi005 ceph-mon[39562]: pgmap v2661: 11 pgs: 1 activating, 10 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:09.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:09 smithi005 ceph-mon[39562]: osdmap e1001: 12 total, 12 up, 12 in 2024-04-20T17:55:10.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:09 smithi106 ceph-mon[44423]: pgmap v2661: 11 pgs: 1 activating, 10 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:10.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:09 smithi106 ceph-mon[44423]: osdmap e1001: 12 total, 12 up, 12 in 2024-04-20T17:55:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:09 smithi171 ceph-mon[44189]: pgmap v2661: 11 pgs: 1 activating, 10 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:10.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:09 smithi171 ceph-mon[44189]: osdmap e1001: 12 total, 12 up, 12 in 2024-04-20T17:55:11.754 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a: {} 2024-04-20T17:55:11.765 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c: {} 2024-04-20T17:55:11.776 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b: {} 2024-04-20T17:55:11.777 INFO:teuthology.orchestra.run.smithi005.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-20T17:55:11.777 INFO:teuthology.orchestra.run.smithi005.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-20T17:55:11.777 INFO:teuthology.orchestra.run.smithi005.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-20T17:55:12.112 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:11 smithi005 ceph-mon[39562]: pgmap v2663: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:12.112 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:11 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/234504646' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T17:55:12.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:11 smithi106 ceph-mon[44423]: pgmap v2663: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:12.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:11 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/234504646' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T17:55:12.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:11 smithi171 ceph-mon[44189]: pgmap v2663: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:12.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:11 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/234504646' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-20T17:55:12.522 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 4 1 2024-04-20T17:55:12.801 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:55:14.068 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:13 smithi005 ceph-mon[39562]: pgmap v2664: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:14.068 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:13 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:55:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:13 smithi106 ceph-mon[44423]: pgmap v2664: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:14.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:13 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:55:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:13 smithi171 ceph-mon[44189]: pgmap v2664: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:14.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:13 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:55:15.209 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.4 to 1 (10000) 2024-04-20T17:55:15.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:14 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:55:15.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:14 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:55:15.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:14 smithi005 ceph-mon[39562]: osdmap e1002: 12 total, 12 up, 12 in 2024-04-20T17:55:15.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:14 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:55:15.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:14 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:55:15.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:14 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:55:15.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:14 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2595754916' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-20T17:55:15.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:14 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:55:15.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:14 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:55:15.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:14 smithi106 ceph-mon[44423]: osdmap e1002: 12 total, 12 up, 12 in 2024-04-20T17:55:15.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:14 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:55:15.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:14 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:55:15.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:14 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:55:15.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:14 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2595754916' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-20T17:55:15.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:14 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:55:15.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:14 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:55:15.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:14 smithi171 ceph-mon[44189]: osdmap e1002: 12 total, 12 up, 12 in 2024-04-20T17:55:15.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:14 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:55:15.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:14 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:55:15.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:14 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:55:15.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:14 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2595754916' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-20T17:55:16.000 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd primary-affinity 4 1 2024-04-20T17:55:16.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:15 smithi005 ceph-mon[39562]: pgmap v2666: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:16.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:15 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2595754916' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-20T17:55:16.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:15 smithi005 ceph-mon[39562]: osdmap e1003: 12 total, 12 up, 12 in 2024-04-20T17:55:16.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:15 smithi106 ceph-mon[44423]: pgmap v2666: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:15 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2595754916' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-20T17:55:16.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:15 smithi106 ceph-mon[44423]: osdmap e1003: 12 total, 12 up, 12 in 2024-04-20T17:55:16.281 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:55:16.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:15 smithi171 ceph-mon[44189]: pgmap v2666: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:16.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:15 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2595754916' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-20T17:55:16.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:15 smithi171 ceph-mon[44189]: osdmap e1003: 12 total, 12 up, 12 in 2024-04-20T17:55:18.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:17 smithi005 ceph-mon[39562]: pgmap v2668: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:18.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:17 smithi106 ceph-mon[44423]: pgmap v2668: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:18.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:17 smithi171 ceph-mon[44189]: pgmap v2668: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:18.922 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-04-20T17:55:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:18 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3562614331' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-20T17:55:19.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:18 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3562614331' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-20T17:55:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:18 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3562614331' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-20T17:55:19.876 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 5 1 2024-04-20T17:55:20.154 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:55:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:19 smithi005 ceph-mon[39562]: pgmap v2669: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:19 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3562614331' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-20T17:55:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:19 smithi005 ceph-mon[39562]: osdmap e1004: 12 total, 12 up, 12 in 2024-04-20T17:55:20.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:19 smithi106 ceph-mon[44423]: pgmap v2669: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:20.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:19 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3562614331' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-20T17:55:20.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:19 smithi106 ceph-mon[44423]: osdmap e1004: 12 total, 12 up, 12 in 2024-04-20T17:55:20.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:19 smithi171 ceph-mon[44189]: pgmap v2669: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:19 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3562614331' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-20T17:55:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:19 smithi171 ceph-mon[44189]: osdmap e1004: 12 total, 12 up, 12 in 2024-04-20T17:55:21.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:20 smithi005 ceph-mon[39562]: 1.6 scrub starts 2024-04-20T17:55:21.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:20 smithi005 ceph-mon[39562]: 1.6 scrub ok 2024-04-20T17:55:21.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:20 smithi005 ceph-mon[39562]: 1.8 scrub starts 2024-04-20T17:55:21.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:20 smithi005 ceph-mon[39562]: 1.8 scrub ok 2024-04-20T17:55:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:20 smithi106 ceph-mon[44423]: 1.6 scrub starts 2024-04-20T17:55:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:20 smithi106 ceph-mon[44423]: 1.6 scrub ok 2024-04-20T17:55:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:20 smithi106 ceph-mon[44423]: 1.8 scrub starts 2024-04-20T17:55:21.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:20 smithi106 ceph-mon[44423]: 1.8 scrub ok 2024-04-20T17:55:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:20 smithi171 ceph-mon[44189]: 1.6 scrub starts 2024-04-20T17:55:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:20 smithi171 ceph-mon[44189]: 1.6 scrub ok 2024-04-20T17:55:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:20 smithi171 ceph-mon[44189]: 1.8 scrub starts 2024-04-20T17:55:21.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:20 smithi171 ceph-mon[44189]: 1.8 scrub ok 2024-04-20T17:55:21.928 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:21 smithi005 ceph-mon[39562]: pgmap v2671: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:22.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:21 smithi106 ceph-mon[44423]: pgmap v2671: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:22.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:21 smithi171 ceph-mon[44189]: pgmap v2671: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:22.942 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.5 to 1 (10000) 2024-04-20T17:55:23.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:22 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1424566976' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:55:23.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:22 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1424566976' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:55:23.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:22 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1424566976' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:55:23.601 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd primary-affinity 5 1 2024-04-20T17:55:23.883 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:55:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:23 smithi005 ceph-mon[39562]: pgmap v2672: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:23 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1424566976' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-20T17:55:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:23 smithi005 ceph-mon[39562]: osdmap e1005: 12 total, 12 up, 12 in 2024-04-20T17:55:24.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:23 smithi106 ceph-mon[44423]: pgmap v2672: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:23 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1424566976' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-20T17:55:24.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:23 smithi106 ceph-mon[44423]: osdmap e1005: 12 total, 12 up, 12 in 2024-04-20T17:55:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:23 smithi171 ceph-mon[44189]: pgmap v2672: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:23 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1424566976' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-20T17:55:24.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:23 smithi171 ceph-mon[44189]: osdmap e1005: 12 total, 12 up, 12 in 2024-04-20T17:55:25.103 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:24 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:55:25.103 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:24 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:55:25.103 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:24 smithi005 ceph-mon[39562]: pgmap v2674: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:25.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:24 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:55:25.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:24 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:55:25.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:24 smithi106 ceph-mon[44423]: pgmap v2674: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:25.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:24 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:55:25.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:24 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:55:25.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:24 smithi171 ceph-mon[44189]: pgmap v2674: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:25.946 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-04-20T17:55:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:25 smithi005 ceph-mon[39562]: osdmap e1006: 12 total, 12 up, 12 in 2024-04-20T17:55:26.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:25 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1725180668' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:55:26.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:25 smithi106 ceph-mon[44423]: osdmap e1006: 12 total, 12 up, 12 in 2024-04-20T17:55:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:25 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1725180668' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:55:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:25 smithi171 ceph-mon[44189]: osdmap e1006: 12 total, 12 up, 12 in 2024-04-20T17:55:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:25 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1725180668' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-20T17:55:26.625 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 8 1 2024-04-20T17:55:26.910 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:55:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:26 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1725180668' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-20T17:55:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:26 smithi005 ceph-mon[39562]: osdmap e1007: 12 total, 12 up, 12 in 2024-04-20T17:55:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:26 smithi005 ceph-mon[39562]: pgmap v2677: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:27.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:26 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1725180668' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-20T17:55:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:26 smithi106 ceph-mon[44423]: osdmap e1007: 12 total, 12 up, 12 in 2024-04-20T17:55:27.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:26 smithi106 ceph-mon[44423]: pgmap v2677: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:27.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:26 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1725180668' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-20T17:55:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:26 smithi171 ceph-mon[44189]: osdmap e1007: 12 total, 12 up, 12 in 2024-04-20T17:55:27.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:26 smithi171 ceph-mon[44189]: pgmap v2677: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:28.174 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:27 smithi005 ceph-mon[39562]: osdmap e1008: 12 total, 12 up, 12 in 2024-04-20T17:55:28.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:27 smithi106 ceph-mon[44423]: osdmap e1008: 12 total, 12 up, 12 in 2024-04-20T17:55:28.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:27 smithi171 ceph-mon[44189]: osdmap e1008: 12 total, 12 up, 12 in 2024-04-20T17:55:28.959 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.8 to 1 (10000) 2024-04-20T17:55:29.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:28 smithi005 ceph-mon[39562]: 1.a deep-scrub starts 2024-04-20T17:55:29.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:28 smithi005 ceph-mon[39562]: 1.a deep-scrub ok 2024-04-20T17:55:29.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:28 smithi005 ceph-mon[39562]: 1.8 scrub starts 2024-04-20T17:55:29.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:28 smithi005 ceph-mon[39562]: 1.8 scrub ok 2024-04-20T17:55:29.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:28 smithi005 ceph-mon[39562]: pgmap v2679: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:29.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:28 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3668868109' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-20T17:55:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:28 smithi106 ceph-mon[44423]: 1.a deep-scrub starts 2024-04-20T17:55:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:28 smithi106 ceph-mon[44423]: 1.a deep-scrub ok 2024-04-20T17:55:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:28 smithi106 ceph-mon[44423]: 1.8 scrub starts 2024-04-20T17:55:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:28 smithi106 ceph-mon[44423]: 1.8 scrub ok 2024-04-20T17:55:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:28 smithi106 ceph-mon[44423]: pgmap v2679: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:29.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:28 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3668868109' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-20T17:55:29.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:28 smithi171 ceph-mon[44189]: 1.a deep-scrub starts 2024-04-20T17:55:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:28 smithi171 ceph-mon[44189]: 1.a deep-scrub ok 2024-04-20T17:55:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:28 smithi171 ceph-mon[44189]: 1.8 scrub starts 2024-04-20T17:55:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:28 smithi171 ceph-mon[44189]: 1.8 scrub ok 2024-04-20T17:55:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:28 smithi171 ceph-mon[44189]: pgmap v2679: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:29.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:28 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3668868109' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-20T17:55:29.577 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd primary-affinity 8 1 2024-04-20T17:55:29.859 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:55:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:29 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3668868109' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-20T17:55:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:29 smithi005 ceph-mon[39562]: osdmap e1009: 12 total, 12 up, 12 in 2024-04-20T17:55:30.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:29 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3668868109' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-20T17:55:30.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:29 smithi106 ceph-mon[44423]: osdmap e1009: 12 total, 12 up, 12 in 2024-04-20T17:55:30.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:29 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3668868109' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-20T17:55:30.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:29 smithi171 ceph-mon[44189]: osdmap e1009: 12 total, 12 up, 12 in 2024-04-20T17:55:31.072 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:30 smithi005 ceph-mon[39562]: pgmap v2681: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:31.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:30 smithi106 ceph-mon[44423]: pgmap v2681: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:31.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:30 smithi171 ceph-mon[44189]: pgmap v2681: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:31.974 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-04-20T17:55:32.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:31 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3758768805' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-04-20T17:55:32.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:31 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3758768805' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-04-20T17:55:32.339 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:31 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3758768805' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-04-20T17:55:32.748 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 9 1 2024-04-20T17:55:33.026 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:55:33.259 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:32 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3758768805' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-04-20T17:55:33.260 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:32 smithi005 ceph-mon[39562]: osdmap e1010: 12 total, 12 up, 12 in 2024-04-20T17:55:33.260 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:32 smithi005 ceph-mon[39562]: pgmap v2683: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:33.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:32 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3758768805' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-04-20T17:55:33.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:32 smithi106 ceph-mon[44423]: osdmap e1010: 12 total, 12 up, 12 in 2024-04-20T17:55:33.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:32 smithi106 ceph-mon[44423]: pgmap v2683: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:32 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3758768805' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-04-20T17:55:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:32 smithi171 ceph-mon[44189]: osdmap e1010: 12 total, 12 up, 12 in 2024-04-20T17:55:33.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:32 smithi171 ceph-mon[44189]: pgmap v2683: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:34.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: 1.5 scrub starts 2024-04-20T17:55:34.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: 1.5 scrub ok 2024-04-20T17:55:34.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=3284060828,config=1460075751,health=3897446751,kv=763203449}) 2024-04-20T17:55:34.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=393863248,logm=3442722627}) 2024-04-20T17:55:34.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2172999083}) 2024-04-20T17:55:34.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2025852485}) 2024-04-20T17:55:34.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1479361809}) 2024-04-20T17:55:34.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2856358649}) 2024-04-20T17:55:34.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2826000029}) 2024-04-20T17:55:34.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3881677693}) 2024-04-20T17:55:34.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334331375}) 2024-04-20T17:55:34.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1003941451}) 2024-04-20T17:55:34.216 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3867627396}) 2024-04-20T17:55:34.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1593506920}) 2024-04-20T17:55:34.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3815971744}) 2024-04-20T17:55:34.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=469053104}) 2024-04-20T17:55:34.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=451696845}) 2024-04-20T17:55:34.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3800303367}) 2024-04-20T17:55:34.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3190763330}) 2024-04-20T17:55:34.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2984184053}) 2024-04-20T17:55:34.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3771005569}) 2024-04-20T17:55:34.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2083580823}) 2024-04-20T17:55:34.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1241767031}) 2024-04-20T17:55:34.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2075228232}) 2024-04-20T17:55:34.217 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2397319583}) 2024-04-20T17:55:34.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2444132625}) 2024-04-20T17:55:34.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2432362745}) 2024-04-20T17:55:34.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1922123321}) 2024-04-20T17:55:34.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1962165378}) 2024-04-20T17:55:34.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2495267043}) 2024-04-20T17:55:34.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3334779589}) 2024-04-20T17:55:34.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3830045170}) 2024-04-20T17:55:34.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3637070965}) 2024-04-20T17:55:34.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=116123633}) 2024-04-20T17:55:34.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2798723975}) 2024-04-20T17:55:34.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2492444470}) 2024-04-20T17:55:34.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1191795430}) 2024-04-20T17:55:34.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1087736149}) 2024-04-20T17:55:34.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1729839522}) 2024-04-20T17:55:34.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2894111541}) 2024-04-20T17:55:34.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4121111169}) 2024-04-20T17:55:34.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3718152527}) 2024-04-20T17:55:34.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1711820141}) 2024-04-20T17:55:34.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4199675399}) 2024-04-20T17:55:34.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1487968812}) 2024-04-20T17:55:34.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1575009939}) 2024-04-20T17:55:34.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3245703826}) 2024-04-20T17:55:34.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2091849073}) 2024-04-20T17:55:34.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=318939679}) 2024-04-20T17:55:34.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1966011081}) 2024-04-20T17:55:34.220 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1281011035}) 2024-04-20T17:55:34.220 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3184399794}) 2024-04-20T17:55:34.220 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2399634170}) 2024-04-20T17:55:34.220 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=953876783}) 2024-04-20T17:55:34.220 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1272797228}) 2024-04-20T17:55:34.220 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4259203009}) 2024-04-20T17:55:34.220 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2394577521}) 2024-04-20T17:55:34.220 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1288872439}) 2024-04-20T17:55:34.220 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3235927704}) 2024-04-20T17:55:34.220 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4285800615}) 2024-04-20T17:55:34.221 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2129448983}) 2024-04-20T17:55:34.221 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1413748065}) 2024-04-20T17:55:34.221 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=647330850}) 2024-04-20T17:55:34.221 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3138807896}) 2024-04-20T17:55:34.221 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3276715662}) 2024-04-20T17:55:34.221 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3899921710}) 2024-04-20T17:55:34.221 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1838431892}) 2024-04-20T17:55:34.221 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1239659472}) 2024-04-20T17:55:34.221 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1612233855}) 2024-04-20T17:55:34.221 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2377673189}) 2024-04-20T17:55:34.221 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1135402562}) 2024-04-20T17:55:34.221 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=788807333}) 2024-04-20T17:55:34.221 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3459422758}) 2024-04-20T17:55:34.221 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=667189958}) 2024-04-20T17:55:34.221 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2527302754}) 2024-04-20T17:55:34.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=981530270}) 2024-04-20T17:55:34.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=348744220}) 2024-04-20T17:55:34.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=455424073}) 2024-04-20T17:55:34.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=417953523}) 2024-04-20T17:55:34.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1288248423}) 2024-04-20T17:55:34.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=620985211}) 2024-04-20T17:55:34.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1465020916}) 2024-04-20T17:55:34.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1862204332}) 2024-04-20T17:55:34.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=664255252}) 2024-04-20T17:55:34.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=638360428}) 2024-04-20T17:55:34.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3050159219}) 2024-04-20T17:55:34.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2851781812}) 2024-04-20T17:55:34.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4242826142}) 2024-04-20T17:55:34.222 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3826825005}) 2024-04-20T17:55:34.223 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=125553550}) 2024-04-20T17:55:34.223 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=612236174}) 2024-04-20T17:55:34.223 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3809960629}) 2024-04-20T17:55:34.223 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4222043744}) 2024-04-20T17:55:34.223 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=834147034}) 2024-04-20T17:55:34.223 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=686967928}) 2024-04-20T17:55:34.223 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=844744611}) 2024-04-20T17:55:34.223 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1483022784}) 2024-04-20T17:55:34.223 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3271236493}) 2024-04-20T17:55:34.223 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2242518312}) 2024-04-20T17:55:34.223 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1317675121}) 2024-04-20T17:55:34.223 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=742365664}) 2024-04-20T17:55:34.223 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3550596079}) 2024-04-20T17:55:34.223 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1723623193}) 2024-04-20T17:55:34.223 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3997426345}) 2024-04-20T17:55:34.224 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3028717885}) 2024-04-20T17:55:34.224 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2282074727}) 2024-04-20T17:55:34.224 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1504560273}) 2024-04-20T17:55:34.224 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2067349592}) 2024-04-20T17:55:34.224 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=36731499}) 2024-04-20T17:55:34.224 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1393541856}) 2024-04-20T17:55:34.224 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=922717774}) 2024-04-20T17:55:34.224 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3324414756}) 2024-04-20T17:55:34.224 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2769295035}) 2024-04-20T17:55:34.225 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2162494100}) 2024-04-20T17:55:34.225 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1318332946}) 2024-04-20T17:55:34.225 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=227447583}) 2024-04-20T17:55:34.225 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2882987797}) 2024-04-20T17:55:34.225 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1263637176}) 2024-04-20T17:55:34.225 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4222560873}) 2024-04-20T17:55:34.225 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2640851781}) 2024-04-20T17:55:34.225 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3471724017}) 2024-04-20T17:55:34.225 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1209995332}) 2024-04-20T17:55:34.225 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {logm=35,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=9,mon_config_key=32} crc {logm=713966708,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=587883768,mon_config_key=3654313052}) 2024-04-20T17:55:34.225 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=408941724}) 2024-04-20T17:55:34.225 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1771371597}) 2024-04-20T17:55:34.225 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=58,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=24} crc {mon_config_key=3539527497,monmap=717113318,osd_metadata=3393198279,osd_pg_creating=2244894951,osdmap=743872458}) 2024-04-20T17:55:34.225 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:33 smithi005 ceph-mon[39562]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=83} crc {osdmap=3955813019}) 2024-04-20T17:55:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: 1.5 scrub starts 2024-04-20T17:55:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: 1.5 scrub ok 2024-04-20T17:55:34.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=3284060828,config=1460075751,health=3897446751,kv=763203449}) 2024-04-20T17:55:34.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=393863248,logm=3442722627}) 2024-04-20T17:55:34.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2172999083}) 2024-04-20T17:55:34.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2025852485}) 2024-04-20T17:55:34.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1479361809}) 2024-04-20T17:55:34.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2856358649}) 2024-04-20T17:55:34.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2826000029}) 2024-04-20T17:55:34.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3881677693}) 2024-04-20T17:55:34.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334331375}) 2024-04-20T17:55:34.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1003941451}) 2024-04-20T17:55:34.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3867627396}) 2024-04-20T17:55:34.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1593506920}) 2024-04-20T17:55:34.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3815971744}) 2024-04-20T17:55:34.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=469053104}) 2024-04-20T17:55:34.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=451696845}) 2024-04-20T17:55:34.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3800303367}) 2024-04-20T17:55:34.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3190763330}) 2024-04-20T17:55:34.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2984184053}) 2024-04-20T17:55:34.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3771005569}) 2024-04-20T17:55:34.263 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2083580823}) 2024-04-20T17:55:34.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1241767031}) 2024-04-20T17:55:34.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2075228232}) 2024-04-20T17:55:34.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2397319583}) 2024-04-20T17:55:34.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2444132625}) 2024-04-20T17:55:34.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2432362745}) 2024-04-20T17:55:34.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1922123321}) 2024-04-20T17:55:34.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1962165378}) 2024-04-20T17:55:34.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2495267043}) 2024-04-20T17:55:34.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3334779589}) 2024-04-20T17:55:34.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3830045170}) 2024-04-20T17:55:34.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3637070965}) 2024-04-20T17:55:34.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=116123633}) 2024-04-20T17:55:34.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2798723975}) 2024-04-20T17:55:34.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2492444470}) 2024-04-20T17:55:34.264 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1191795430}) 2024-04-20T17:55:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1087736149}) 2024-04-20T17:55:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1729839522}) 2024-04-20T17:55:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2894111541}) 2024-04-20T17:55:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4121111169}) 2024-04-20T17:55:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3718152527}) 2024-04-20T17:55:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1711820141}) 2024-04-20T17:55:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4199675399}) 2024-04-20T17:55:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1487968812}) 2024-04-20T17:55:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1575009939}) 2024-04-20T17:55:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3245703826}) 2024-04-20T17:55:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2091849073}) 2024-04-20T17:55:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=318939679}) 2024-04-20T17:55:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1966011081}) 2024-04-20T17:55:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1281011035}) 2024-04-20T17:55:34.265 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3184399794}) 2024-04-20T17:55:34.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2399634170}) 2024-04-20T17:55:34.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=953876783}) 2024-04-20T17:55:34.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1272797228}) 2024-04-20T17:55:34.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4259203009}) 2024-04-20T17:55:34.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2394577521}) 2024-04-20T17:55:34.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1288872439}) 2024-04-20T17:55:34.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3235927704}) 2024-04-20T17:55:34.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4285800615}) 2024-04-20T17:55:34.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2129448983}) 2024-04-20T17:55:34.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1413748065}) 2024-04-20T17:55:34.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=647330850}) 2024-04-20T17:55:34.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3138807896}) 2024-04-20T17:55:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3276715662}) 2024-04-20T17:55:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3899921710}) 2024-04-20T17:55:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1838431892}) 2024-04-20T17:55:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1239659472}) 2024-04-20T17:55:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1612233855}) 2024-04-20T17:55:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2377673189}) 2024-04-20T17:55:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1135402562}) 2024-04-20T17:55:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=788807333}) 2024-04-20T17:55:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3459422758}) 2024-04-20T17:55:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=667189958}) 2024-04-20T17:55:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2527302754}) 2024-04-20T17:55:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=981530270}) 2024-04-20T17:55:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=348744220}) 2024-04-20T17:55:34.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=455424073}) 2024-04-20T17:55:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=417953523}) 2024-04-20T17:55:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1288248423}) 2024-04-20T17:55:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=620985211}) 2024-04-20T17:55:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1465020916}) 2024-04-20T17:55:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1862204332}) 2024-04-20T17:55:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=664255252}) 2024-04-20T17:55:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=638360428}) 2024-04-20T17:55:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3050159219}) 2024-04-20T17:55:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2851781812}) 2024-04-20T17:55:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4242826142}) 2024-04-20T17:55:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3826825005}) 2024-04-20T17:55:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=125553550}) 2024-04-20T17:55:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=612236174}) 2024-04-20T17:55:34.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3809960629}) 2024-04-20T17:55:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4222043744}) 2024-04-20T17:55:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=834147034}) 2024-04-20T17:55:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=686967928}) 2024-04-20T17:55:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=844744611}) 2024-04-20T17:55:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1483022784}) 2024-04-20T17:55:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3271236493}) 2024-04-20T17:55:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2242518312}) 2024-04-20T17:55:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1317675121}) 2024-04-20T17:55:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=742365664}) 2024-04-20T17:55:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3550596079}) 2024-04-20T17:55:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1723623193}) 2024-04-20T17:55:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3997426345}) 2024-04-20T17:55:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3028717885}) 2024-04-20T17:55:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2282074727}) 2024-04-20T17:55:34.269 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1504560273}) 2024-04-20T17:55:34.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2067349592}) 2024-04-20T17:55:34.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=36731499}) 2024-04-20T17:55:34.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1393541856}) 2024-04-20T17:55:34.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=922717774}) 2024-04-20T17:55:34.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3324414756}) 2024-04-20T17:55:34.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2769295035}) 2024-04-20T17:55:34.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2162494100}) 2024-04-20T17:55:34.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1318332946}) 2024-04-20T17:55:34.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=227447583}) 2024-04-20T17:55:34.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2882987797}) 2024-04-20T17:55:34.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1263637176}) 2024-04-20T17:55:34.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4222560873}) 2024-04-20T17:55:34.270 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2640851781}) 2024-04-20T17:55:34.271 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3471724017}) 2024-04-20T17:55:34.271 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1209995332}) 2024-04-20T17:55:34.271 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {logm=35,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=9,mon_config_key=32} crc {logm=713966708,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=587883768,mon_config_key=3654313052}) 2024-04-20T17:55:34.271 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=408941724}) 2024-04-20T17:55:34.271 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1771371597}) 2024-04-20T17:55:34.271 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=58,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=24} crc {mon_config_key=3539527497,monmap=717113318,osd_metadata=3393198279,osd_pg_creating=2244894951,osdmap=743872458}) 2024-04-20T17:55:34.271 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:33 smithi106 ceph-mon[44423]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=83} crc {osdmap=3955813019}) 2024-04-20T17:55:34.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: 1.5 scrub starts 2024-04-20T17:55:34.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: 1.5 scrub ok 2024-04-20T17:55:34.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=3284060828,config=1460075751,health=3897446751,kv=763203449}) 2024-04-20T17:55:34.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=393863248,logm=3442722627}) 2024-04-20T17:55:34.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2172999083}) 2024-04-20T17:55:34.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2025852485}) 2024-04-20T17:55:34.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1479361809}) 2024-04-20T17:55:34.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2856358649}) 2024-04-20T17:55:34.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2826000029}) 2024-04-20T17:55:34.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3881677693}) 2024-04-20T17:55:34.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=334331375}) 2024-04-20T17:55:34.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1003941451}) 2024-04-20T17:55:34.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3867627396}) 2024-04-20T17:55:34.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1593506920}) 2024-04-20T17:55:34.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3815971744}) 2024-04-20T17:55:34.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=469053104}) 2024-04-20T17:55:34.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=451696845}) 2024-04-20T17:55:34.335 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3800303367}) 2024-04-20T17:55:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3190763330}) 2024-04-20T17:55:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2984184053}) 2024-04-20T17:55:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3771005569}) 2024-04-20T17:55:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2083580823}) 2024-04-20T17:55:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1241767031}) 2024-04-20T17:55:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2075228232}) 2024-04-20T17:55:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2397319583}) 2024-04-20T17:55:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2444132625}) 2024-04-20T17:55:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2432362745}) 2024-04-20T17:55:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1922123321}) 2024-04-20T17:55:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1962165378}) 2024-04-20T17:55:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2495267043}) 2024-04-20T17:55:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3334779589}) 2024-04-20T17:55:34.336 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3830045170}) 2024-04-20T17:55:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3637070965}) 2024-04-20T17:55:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=116123633}) 2024-04-20T17:55:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2798723975}) 2024-04-20T17:55:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2492444470}) 2024-04-20T17:55:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1191795430}) 2024-04-20T17:55:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1087736149}) 2024-04-20T17:55:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1729839522}) 2024-04-20T17:55:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2894111541}) 2024-04-20T17:55:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4121111169}) 2024-04-20T17:55:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3718152527}) 2024-04-20T17:55:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1711820141}) 2024-04-20T17:55:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4199675399}) 2024-04-20T17:55:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1487968812}) 2024-04-20T17:55:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1575009939}) 2024-04-20T17:55:34.337 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3245703826}) 2024-04-20T17:55:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2091849073}) 2024-04-20T17:55:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=318939679}) 2024-04-20T17:55:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1966011081}) 2024-04-20T17:55:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1281011035}) 2024-04-20T17:55:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3184399794}) 2024-04-20T17:55:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2399634170}) 2024-04-20T17:55:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=953876783}) 2024-04-20T17:55:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1272797228}) 2024-04-20T17:55:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4259203009}) 2024-04-20T17:55:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2394577521}) 2024-04-20T17:55:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1288872439}) 2024-04-20T17:55:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3235927704}) 2024-04-20T17:55:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4285800615}) 2024-04-20T17:55:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2129448983}) 2024-04-20T17:55:34.338 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1413748065}) 2024-04-20T17:55:34.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=647330850}) 2024-04-20T17:55:34.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3138807896}) 2024-04-20T17:55:34.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3276715662}) 2024-04-20T17:55:34.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3899921710}) 2024-04-20T17:55:34.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1838431892}) 2024-04-20T17:55:34.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1239659472}) 2024-04-20T17:55:34.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1612233855}) 2024-04-20T17:55:34.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2377673189}) 2024-04-20T17:55:34.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1135402562}) 2024-04-20T17:55:34.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=788807333}) 2024-04-20T17:55:34.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3459422758}) 2024-04-20T17:55:34.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=667189958}) 2024-04-20T17:55:34.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2527302754}) 2024-04-20T17:55:34.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=981530270}) 2024-04-20T17:55:34.339 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=348744220}) 2024-04-20T17:55:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=455424073}) 2024-04-20T17:55:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=417953523}) 2024-04-20T17:55:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1288248423}) 2024-04-20T17:55:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=620985211}) 2024-04-20T17:55:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1465020916}) 2024-04-20T17:55:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1862204332}) 2024-04-20T17:55:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=664255252}) 2024-04-20T17:55:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=638360428}) 2024-04-20T17:55:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3050159219}) 2024-04-20T17:55:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2851781812}) 2024-04-20T17:55:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4242826142}) 2024-04-20T17:55:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3826825005}) 2024-04-20T17:55:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=125553550}) 2024-04-20T17:55:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=612236174}) 2024-04-20T17:55:34.340 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3809960629}) 2024-04-20T17:55:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4222043744}) 2024-04-20T17:55:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=834147034}) 2024-04-20T17:55:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=686967928}) 2024-04-20T17:55:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=844744611}) 2024-04-20T17:55:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1483022784}) 2024-04-20T17:55:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3271236493}) 2024-04-20T17:55:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2242518312}) 2024-04-20T17:55:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1317675121}) 2024-04-20T17:55:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=742365664}) 2024-04-20T17:55:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3550596079}) 2024-04-20T17:55:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1723623193}) 2024-04-20T17:55:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3997426345}) 2024-04-20T17:55:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3028717885}) 2024-04-20T17:55:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2282074727}) 2024-04-20T17:55:34.341 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1504560273}) 2024-04-20T17:55:34.342 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2067349592}) 2024-04-20T17:55:34.342 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=36731499}) 2024-04-20T17:55:34.342 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1393541856}) 2024-04-20T17:55:34.342 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=922717774}) 2024-04-20T17:55:34.342 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3324414756}) 2024-04-20T17:55:34.342 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2769295035}) 2024-04-20T17:55:34.342 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2162494100}) 2024-04-20T17:55:34.342 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1318332946}) 2024-04-20T17:55:34.342 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=227447583}) 2024-04-20T17:55:34.342 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2882987797}) 2024-04-20T17:55:34.342 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1263637176}) 2024-04-20T17:55:34.342 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4222560873}) 2024-04-20T17:55:34.342 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2640851781}) 2024-04-20T17:55:34.342 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3471724017}) 2024-04-20T17:55:34.343 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1209995332}) 2024-04-20T17:55:34.343 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {logm=35,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=9,mon_config_key=32} crc {logm=713966708,mdsmap=2840952586,mgr=1362760828,mgr_command_descs=4152233241,mgr_metadata=2104953944,mgrstat=587883768,mon_config_key=3654313052}) 2024-04-20T17:55:34.343 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=408941724}) 2024-04-20T17:55:34.343 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1771371597}) 2024-04-20T17:55:34.343 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=58,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=24} crc {mon_config_key=3539527497,monmap=717113318,osd_metadata=3393198279,osd_pg_creating=2244894951,osdmap=743872458}) 2024-04-20T17:55:34.343 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:33 smithi171 ceph-mon[44189]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=83} crc {osdmap=3955813019}) 2024-04-20T17:55:35.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:34 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:55:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:34 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:55:35.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:34 smithi106 ceph-mon[44423]: pgmap v2684: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:35.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:34 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:55:35.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:34 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:55:35.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:34 smithi171 ceph-mon[44189]: pgmap v2684: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:34 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:55:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:34 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:55:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:34 smithi005 ceph-mon[39562]: pgmap v2684: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:35.990 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.9 to 1 (10000) 2024-04-20T17:55:36.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:35 smithi106 ceph-mon[44423]: osdmap e1011: 12 total, 12 up, 12 in 2024-04-20T17:55:36.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:35 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2352960112' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:55:36.279 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:35 smithi005 ceph-mon[39562]: osdmap e1011: 12 total, 12 up, 12 in 2024-04-20T17:55:36.279 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:35 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2352960112' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:55:36.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:35 smithi171 ceph-mon[44189]: osdmap e1011: 12 total, 12 up, 12 in 2024-04-20T17:55:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:35 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2352960112' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:55:36.713 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd primary-affinity 9 1 2024-04-20T17:55:37.010 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:55:37.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:36 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2352960112' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-20T17:55:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:36 smithi106 ceph-mon[44423]: osdmap e1012: 12 total, 12 up, 12 in 2024-04-20T17:55:37.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:36 smithi106 ceph-mon[44423]: pgmap v2687: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:37.271 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:36 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2352960112' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-20T17:55:37.271 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:36 smithi005 ceph-mon[39562]: osdmap e1012: 12 total, 12 up, 12 in 2024-04-20T17:55:37.271 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:36 smithi005 ceph-mon[39562]: pgmap v2687: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:37.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:36 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2352960112' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-20T17:55:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:36 smithi171 ceph-mon[44189]: osdmap e1012: 12 total, 12 up, 12 in 2024-04-20T17:55:37.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:36 smithi171 ceph-mon[44189]: pgmap v2687: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:38.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:38 smithi005 ceph-mon[39562]: 1.4 scrub starts 2024-04-20T17:55:38.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:38 smithi005 ceph-mon[39562]: 1.4 scrub ok 2024-04-20T17:55:38.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:38 smithi171 ceph-mon[44189]: 1.4 scrub starts 2024-04-20T17:55:38.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:38 smithi171 ceph-mon[44189]: 1.4 scrub ok 2024-04-20T17:55:39.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:38 smithi106 ceph-mon[44423]: 1.4 scrub starts 2024-04-20T17:55:39.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:38 smithi106 ceph-mon[44423]: 1.4 scrub ok 2024-04-20T17:55:39.545 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-04-20T17:55:39.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:39 smithi171 ceph-mon[44189]: pgmap v2688: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:39.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:39 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3494829743' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:55:39.902 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:39 smithi005 ceph-mon[39562]: pgmap v2688: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:39.902 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:39 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3494829743' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:55:40.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:39 smithi106 ceph-mon[44423]: pgmap v2688: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:40.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:39 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3494829743' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-04-20T17:55:40.276 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 10 1 2024-04-20T17:55:40.541 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:40 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3494829743' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-04-20T17:55:40.541 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:40 smithi005 ceph-mon[39562]: osdmap e1013: 12 total, 12 up, 12 in 2024-04-20T17:55:40.561 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:55:40.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:40 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3494829743' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-04-20T17:55:40.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:40 smithi171 ceph-mon[44189]: osdmap e1013: 12 total, 12 up, 12 in 2024-04-20T17:55:41.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:40 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3494829743' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-04-20T17:55:41.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:40 smithi106 ceph-mon[44423]: osdmap e1013: 12 total, 12 up, 12 in 2024-04-20T17:55:41.806 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:41 smithi005 ceph-mon[39562]: pgmap v2690: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:41.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:41 smithi171 ceph-mon[44189]: pgmap v2690: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:42.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:41 smithi106 ceph-mon[44423]: pgmap v2690: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:42.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:42 smithi171 ceph-mon[44189]: 1.9 deep-scrub starts 2024-04-20T17:55:42.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:42 smithi171 ceph-mon[44189]: 1.9 deep-scrub ok 2024-04-20T17:55:42.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:42 smithi005 ceph-mon[39562]: 1.9 deep-scrub starts 2024-04-20T17:55:42.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:42 smithi005 ceph-mon[39562]: 1.9 deep-scrub ok 2024-04-20T17:55:43.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:42 smithi106 ceph-mon[44423]: 1.9 deep-scrub starts 2024-04-20T17:55:43.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:42 smithi106 ceph-mon[44423]: 1.9 deep-scrub ok 2024-04-20T17:55:43.555 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.10 to 1 (10000) 2024-04-20T17:55:43.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:43 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1934588665' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:55:43.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:43 smithi171 ceph-mon[44189]: pgmap v2691: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:43.948 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:43 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1934588665' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:55:43.948 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:43 smithi005 ceph-mon[39562]: pgmap v2691: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:44.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:43 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1934588665' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:55:44.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:43 smithi106 ceph-mon[44423]: pgmap v2691: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:44.348 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd primary-affinity 10 1 2024-04-20T17:55:44.632 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:55:44.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:44 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1934588665' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-20T17:55:44.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:44 smithi005 ceph-mon[39562]: osdmap e1014: 12 total, 12 up, 12 in 2024-04-20T17:55:44.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:44 smithi005 ceph-mon[39562]: 1.3 scrub starts 2024-04-20T17:55:44.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:44 smithi005 ceph-mon[39562]: 1.3 scrub ok 2024-04-20T17:55:44.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:44 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1934588665' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-20T17:55:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:44 smithi171 ceph-mon[44189]: osdmap e1014: 12 total, 12 up, 12 in 2024-04-20T17:55:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:44 smithi171 ceph-mon[44189]: 1.3 scrub starts 2024-04-20T17:55:44.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:44 smithi171 ceph-mon[44189]: 1.3 scrub ok 2024-04-20T17:55:45.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:44 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1934588665' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-20T17:55:45.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:44 smithi106 ceph-mon[44423]: osdmap e1014: 12 total, 12 up, 12 in 2024-04-20T17:55:45.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:44 smithi106 ceph-mon[44423]: 1.3 scrub starts 2024-04-20T17:55:45.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:44 smithi106 ceph-mon[44423]: 1.3 scrub ok 2024-04-20T17:55:45.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:45 smithi171 ceph-mon[44189]: pgmap v2693: 11 pgs: 1 active+clean+scrubbing, 10 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:45.862 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:45 smithi005 ceph-mon[39562]: pgmap v2693: 11 pgs: 1 active+clean+scrubbing, 10 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:46.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:45 smithi106 ceph-mon[44423]: pgmap v2693: 11 pgs: 1 active+clean+scrubbing, 10 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:46.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:46 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:55:46.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:46 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:55:46.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:46 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:55:46.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:46 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:55:47.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:46 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:55:47.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:46 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:55:47.569 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-04-20T17:55:47.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:47 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/153191322' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:55:47.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:47 smithi171 ceph-mon[44189]: pgmap v2694: 11 pgs: 1 active+clean+scrubbing, 10 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:47.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:47 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/153191322' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:55:47.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:47 smithi005 ceph-mon[39562]: pgmap v2694: 11 pgs: 1 active+clean+scrubbing, 10 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:47 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/153191322' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-04-20T17:55:48.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:47 smithi106 ceph-mon[44423]: pgmap v2694: 11 pgs: 1 active+clean+scrubbing, 10 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:48.343 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 11 1 2024-04-20T17:55:48.626 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:55:48.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:48 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/153191322' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-04-20T17:55:48.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:48 smithi171 ceph-mon[44189]: osdmap e1015: 12 total, 12 up, 12 in 2024-04-20T17:55:48.866 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:48 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/153191322' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-04-20T17:55:48.866 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:48 smithi005 ceph-mon[39562]: osdmap e1015: 12 total, 12 up, 12 in 2024-04-20T17:55:49.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:48 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/153191322' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-04-20T17:55:49.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:48 smithi106 ceph-mon[44423]: osdmap e1015: 12 total, 12 up, 12 in 2024-04-20T17:55:49.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:49 smithi171 ceph-mon[44189]: pgmap v2696: 11 pgs: 1 active+clean+scrubbing, 10 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:49.851 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:49 smithi005 ceph-mon[39562]: pgmap v2696: 11 pgs: 1 active+clean+scrubbing, 10 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:50.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:49 smithi106 ceph-mon[44423]: pgmap v2696: 11 pgs: 1 active+clean+scrubbing, 10 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:50.698 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.11 to 1 (10000) 2024-04-20T17:55:50.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:50 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2709848298' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:55:51.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:50 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2709848298' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:55:51.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:50 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2709848298' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:55:51.667 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd primary-affinity 11 1 2024-04-20T17:55:51.951 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:55:51.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:51 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2709848298' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-20T17:55:51.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:51 smithi005 ceph-mon[39562]: osdmap e1016: 12 total, 12 up, 12 in 2024-04-20T17:55:51.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:51 smithi005 ceph-mon[39562]: pgmap v2698: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:52.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:51 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2709848298' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-20T17:55:52.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:51 smithi106 ceph-mon[44423]: osdmap e1016: 12 total, 12 up, 12 in 2024-04-20T17:55:52.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:51 smithi106 ceph-mon[44423]: pgmap v2698: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:52.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:51 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2709848298' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-20T17:55:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:51 smithi171 ceph-mon[44189]: osdmap e1016: 12 total, 12 up, 12 in 2024-04-20T17:55:52.084 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:51 smithi171 ceph-mon[44189]: pgmap v2698: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:53.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:52 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:55:53.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:52 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:55:53.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:52 smithi005 ceph-mon[39562]: osdmap e1017: 12 total, 12 up, 12 in 2024-04-20T17:55:53.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:52 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:55:53.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:52 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:55:53.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:52 smithi106 ceph-mon[44423]: osdmap e1017: 12 total, 12 up, 12 in 2024-04-20T17:55:53.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:52 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:55:53.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:52 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:55:53.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:52 smithi171 ceph-mon[44189]: osdmap e1017: 12 total, 12 up, 12 in 2024-04-20T17:55:54.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:53 smithi005 ceph-mon[39562]: pgmap v2700: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:54.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:53 smithi106 ceph-mon[44423]: pgmap v2700: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:54.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:53 smithi171 ceph-mon[44189]: pgmap v2700: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:54.889 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-04-20T17:55:55.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:54 smithi005 ceph-mon[39562]: 1.7 scrub starts 2024-04-20T17:55:55.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:54 smithi005 ceph-mon[39562]: 1.7 scrub ok 2024-04-20T17:55:55.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:54 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/90621708' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:55:55.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:54 smithi106 ceph-mon[44423]: 1.7 scrub starts 2024-04-20T17:55:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:54 smithi106 ceph-mon[44423]: 1.7 scrub ok 2024-04-20T17:55:55.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:54 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/90621708' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:55:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:54 smithi171 ceph-mon[44189]: 1.7 scrub starts 2024-04-20T17:55:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:54 smithi171 ceph-mon[44189]: 1.7 scrub ok 2024-04-20T17:55:55.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:54 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/90621708' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-04-20T17:55:55.621 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 2 1 2024-04-20T17:55:55.891 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:55 smithi005 ceph-mon[39562]: pgmap v2701: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:55.891 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:55 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/90621708' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-04-20T17:55:55.891 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:55 smithi005 ceph-mon[39562]: osdmap e1018: 12 total, 12 up, 12 in 2024-04-20T17:55:55.903 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:55:56.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:55 smithi106 ceph-mon[44423]: pgmap v2701: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:56.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:55 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/90621708' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-04-20T17:55:56.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:55 smithi106 ceph-mon[44423]: osdmap e1018: 12 total, 12 up, 12 in 2024-04-20T17:55:56.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:55 smithi171 ceph-mon[44189]: pgmap v2701: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:56.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:55 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/90621708' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-04-20T17:55:56.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:55 smithi171 ceph-mon[44189]: osdmap e1018: 12 total, 12 up, 12 in 2024-04-20T17:55:58.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:57 smithi005 ceph-mon[39562]: pgmap v2703: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:58.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:57 smithi106 ceph-mon[44423]: pgmap v2703: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:58.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:57 smithi171 ceph-mon[44189]: pgmap v2703: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:55:58.913 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.2 to 1 (10000) 2024-04-20T17:55:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:58 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:55:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:58 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:55:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:58 smithi005 ceph-mon[39562]: osdmap e1019: 12 total, 12 up, 12 in 2024-04-20T17:55:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:58 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/854435644' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-20T17:55:59.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:58 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:55:59.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:58 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:55:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:58 smithi106 ceph-mon[44423]: osdmap e1019: 12 total, 12 up, 12 in 2024-04-20T17:55:59.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:58 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/854435644' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-20T17:55:59.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:58 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:55:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:58 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:55:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:58 smithi171 ceph-mon[44189]: osdmap e1019: 12 total, 12 up, 12 in 2024-04-20T17:55:59.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:58 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/854435644' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-20T17:55:59.763 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd primary-affinity 2 1 2024-04-20T17:56:00.045 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:56:00.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:59 smithi005 ceph-mon[39562]: pgmap v2705: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:00.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:59 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/854435644' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-20T17:56:00.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:55:59 smithi005 ceph-mon[39562]: osdmap e1020: 12 total, 12 up, 12 in 2024-04-20T17:56:00.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:59 smithi106 ceph-mon[44423]: pgmap v2705: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:00.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:59 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/854435644' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-20T17:56:00.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:55:59 smithi106 ceph-mon[44423]: osdmap e1020: 12 total, 12 up, 12 in 2024-04-20T17:56:00.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:59 smithi171 ceph-mon[44189]: pgmap v2705: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:00.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:59 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/854435644' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-20T17:56:00.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:55:59 smithi171 ceph-mon[44189]: osdmap e1020: 12 total, 12 up, 12 in 2024-04-20T17:56:02.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:01 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:56:02.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:01 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:56:02.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:01 smithi005 ceph-mon[39562]: pgmap v2707: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:02.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:01 smithi005 ceph-mon[39562]: osdmap e1021: 12 total, 12 up, 12 in 2024-04-20T17:56:02.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:01 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:56:02.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:01 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:56:02.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:01 smithi106 ceph-mon[44423]: pgmap v2707: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:02.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:01 smithi106 ceph-mon[44423]: osdmap e1021: 12 total, 12 up, 12 in 2024-04-20T17:56:02.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:01 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:56:02.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:01 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:56:02.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:01 smithi171 ceph-mon[44189]: pgmap v2707: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:02.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:01 smithi171 ceph-mon[44189]: osdmap e1021: 12 total, 12 up, 12 in 2024-04-20T17:56:02.929 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-04-20T17:56:03.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:02 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:56:03.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:02 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:56:03.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:02 smithi005 ceph-mon[39562]: osdmap e1022: 12 total, 12 up, 12 in 2024-04-20T17:56:03.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:02 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1416941511' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-20T17:56:03.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:02 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:56:03.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:02 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:56:03.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:02 smithi106 ceph-mon[44423]: osdmap e1022: 12 total, 12 up, 12 in 2024-04-20T17:56:03.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:02 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1416941511' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-20T17:56:03.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:02 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:56:03.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:02 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:56:03.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:02 smithi171 ceph-mon[44189]: osdmap e1022: 12 total, 12 up, 12 in 2024-04-20T17:56:03.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:02 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1416941511' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-20T17:56:03.702 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 6 1 2024-04-20T17:56:03.985 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:56:04.199 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:03 smithi005 ceph-mon[39562]: pgmap v2710: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:04.199 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:03 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1416941511' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-20T17:56:04.199 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:03 smithi005 ceph-mon[39562]: osdmap e1023: 12 total, 12 up, 12 in 2024-04-20T17:56:04.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:03 smithi106 ceph-mon[44423]: pgmap v2710: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:04.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:03 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1416941511' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-20T17:56:04.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:03 smithi106 ceph-mon[44423]: osdmap e1023: 12 total, 12 up, 12 in 2024-04-20T17:56:04.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:03 smithi171 ceph-mon[44189]: pgmap v2710: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:04.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:03 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1416941511' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-20T17:56:04.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:03 smithi171 ceph-mon[44189]: osdmap e1023: 12 total, 12 up, 12 in 2024-04-20T17:56:05.936 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.6 to 1 (10000) 2024-04-20T17:56:06.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:05 smithi005 ceph-mon[39562]: pgmap v2712: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:06.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:05 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1552380932' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-20T17:56:06.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:05 smithi106 ceph-mon[44423]: pgmap v2712: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:06.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:05 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1552380932' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-20T17:56:06.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:05 smithi171 ceph-mon[44189]: pgmap v2712: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:06.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:05 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1552380932' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-20T17:56:06.795 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd primary-affinity 6 1 2024-04-20T17:56:07.084 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:56:07.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:06 smithi005 ceph-mon[39562]: 1.1 scrub starts 2024-04-20T17:56:07.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:06 smithi005 ceph-mon[39562]: 1.1 scrub ok 2024-04-20T17:56:07.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:06 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1552380932' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-20T17:56:07.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:06 smithi005 ceph-mon[39562]: osdmap e1024: 12 total, 12 up, 12 in 2024-04-20T17:56:07.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:06 smithi106 ceph-mon[44423]: 1.1 scrub starts 2024-04-20T17:56:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:06 smithi106 ceph-mon[44423]: 1.1 scrub ok 2024-04-20T17:56:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:06 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1552380932' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-20T17:56:07.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:06 smithi106 ceph-mon[44423]: osdmap e1024: 12 total, 12 up, 12 in 2024-04-20T17:56:07.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:06 smithi171 ceph-mon[44189]: 1.1 scrub starts 2024-04-20T17:56:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:06 smithi171 ceph-mon[44189]: 1.1 scrub ok 2024-04-20T17:56:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:06 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1552380932' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-20T17:56:07.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:06 smithi171 ceph-mon[44189]: osdmap e1024: 12 total, 12 up, 12 in 2024-04-20T17:56:08.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:07 smithi005 ceph-mon[39562]: pgmap v2714: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:08.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:07 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:56:08.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:07 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:56:08.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:07 smithi106 ceph-mon[44423]: pgmap v2714: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:08.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:07 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:56:08.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:07 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:56:08.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:07 smithi171 ceph-mon[44189]: pgmap v2714: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:08.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:07 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:56:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:07 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:56:09.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:08 smithi005 ceph-mon[39562]: osdmap e1025: 12 total, 12 up, 12 in 2024-04-20T17:56:09.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:08 smithi005 ceph-mon[39562]: pgmap v2716: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:09.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:08 smithi106 ceph-mon[44423]: osdmap e1025: 12 total, 12 up, 12 in 2024-04-20T17:56:09.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:08 smithi106 ceph-mon[44423]: pgmap v2716: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:09.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:08 smithi171 ceph-mon[44189]: osdmap e1025: 12 total, 12 up, 12 in 2024-04-20T17:56:09.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:08 smithi171 ceph-mon[44189]: pgmap v2716: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:09.964 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-04-20T17:56:10.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:09 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2868175703' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-20T17:56:10.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:09 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2868175703' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-20T17:56:10.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:09 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2868175703' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-20T17:56:10.722 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 3 1 2024-04-20T17:56:11.006 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:56:11.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:10 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2868175703' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-20T17:56:11.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:10 smithi005 ceph-mon[39562]: osdmap e1026: 12 total, 12 up, 12 in 2024-04-20T17:56:11.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:10 smithi005 ceph-mon[39562]: pgmap v2718: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:11.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:10 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2868175703' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-20T17:56:11.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:10 smithi106 ceph-mon[44423]: osdmap e1026: 12 total, 12 up, 12 in 2024-04-20T17:56:11.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:10 smithi106 ceph-mon[44423]: pgmap v2718: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:11.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:10 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2868175703' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-20T17:56:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:10 smithi171 ceph-mon[44189]: osdmap e1026: 12 total, 12 up, 12 in 2024-04-20T17:56:11.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:10 smithi171 ceph-mon[44189]: pgmap v2718: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:12.153 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:11 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:56:12.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:11 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:56:12.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:11 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:56:12.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:11 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:56:12.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:11 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:56:12.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:11 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:56:13.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:12 smithi106 ceph-mon[44423]: osdmap e1027: 12 total, 12 up, 12 in 2024-04-20T17:56:13.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:12 smithi106 ceph-mon[44423]: pgmap v2720: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:13.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:12 smithi171 ceph-mon[44189]: osdmap e1027: 12 total, 12 up, 12 in 2024-04-20T17:56:13.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:12 smithi171 ceph-mon[44189]: pgmap v2720: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:13.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:12 smithi005 ceph-mon[39562]: osdmap e1027: 12 total, 12 up, 12 in 2024-04-20T17:56:13.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:12 smithi005 ceph-mon[39562]: pgmap v2720: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:13.986 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.3 to 1 (10000) 2024-04-20T17:56:14.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:13 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2649559999' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:56:14.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:13 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2649559999' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:56:14.363 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:13 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2649559999' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:56:14.785 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd primary-affinity 3 1 2024-04-20T17:56:15.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:14 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2649559999' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-20T17:56:15.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:14 smithi005 ceph-mon[39562]: osdmap e1028: 12 total, 12 up, 12 in 2024-04-20T17:56:15.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:14 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:56:15.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:14 smithi005 ceph-mon[39562]: pgmap v2722: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:15.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:14 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2649559999' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-20T17:56:15.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:14 smithi106 ceph-mon[44423]: osdmap e1028: 12 total, 12 up, 12 in 2024-04-20T17:56:15.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:14 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:56:15.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:14 smithi106 ceph-mon[44423]: pgmap v2722: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:15.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:14 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2649559999' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-20T17:56:15.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:14 smithi171 ceph-mon[44189]: osdmap e1028: 12 total, 12 up, 12 in 2024-04-20T17:56:15.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:14 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:56:15.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:14 smithi171 ceph-mon[44189]: pgmap v2722: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:15.650 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:56:17.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:17 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:56:17.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:17 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:56:17.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:17 smithi005 ceph-mon[39562]: pgmap v2723: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:17.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:17 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:56:17.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:17 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:56:17.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:17 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:56:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:17 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:56:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:17 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:56:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:17 smithi171 ceph-mon[44189]: pgmap v2723: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:17 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:56:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:17 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:56:17.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:17 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:56:18.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:17 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:56:18.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:17 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:56:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:17 smithi106 ceph-mon[44423]: pgmap v2723: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:17 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:56:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:17 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:56:18.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:17 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:56:18.544 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-04-20T17:56:18.833 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:18 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3732664441' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:56:18.882 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:18 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3732664441' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:56:19.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:18 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3732664441' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-20T17:56:19.332 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 1 1 2024-04-20T17:56:19.631 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:56:19.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:19 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3732664441' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-20T17:56:19.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:19 smithi005 ceph-mon[39562]: osdmap e1029: 12 total, 12 up, 12 in 2024-04-20T17:56:19.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:19 smithi005 ceph-mon[39562]: pgmap v2725: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:19 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3732664441' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-20T17:56:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:19 smithi171 ceph-mon[44189]: osdmap e1029: 12 total, 12 up, 12 in 2024-04-20T17:56:19.834 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:19 smithi171 ceph-mon[44189]: pgmap v2725: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:19 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3732664441' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-20T17:56:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:19 smithi106 ceph-mon[44423]: osdmap e1029: 12 total, 12 up, 12 in 2024-04-20T17:56:20.011 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:19 smithi106 ceph-mon[44423]: pgmap v2725: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:21.857 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.1 to 1 (10000) 2024-04-20T17:56:22.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:21 smithi005 ceph-mon[39562]: pgmap v2726: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:22.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:21 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:56:22.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:21 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:56:22.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:21 smithi005 ceph-mon[39562]: osdmap e1030: 12 total, 12 up, 12 in 2024-04-20T17:56:22.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:21 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1712553886' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:56:22.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:21 smithi106 ceph-mon[44423]: pgmap v2726: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:22.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:21 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:56:22.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:21 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:56:22.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:21 smithi106 ceph-mon[44423]: osdmap e1030: 12 total, 12 up, 12 in 2024-04-20T17:56:22.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:21 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1712553886' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:56:22.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:21 smithi171 ceph-mon[44189]: pgmap v2726: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:22.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:21 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:56:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:21 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:56:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:21 smithi171 ceph-mon[44189]: osdmap e1030: 12 total, 12 up, 12 in 2024-04-20T17:56:22.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:21 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1712553886' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:56:22.545 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd primary-affinity 1 1 2024-04-20T17:56:22.825 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:56:23.108 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:22 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:56:23.108 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:22 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:56:23.108 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:22 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1712553886' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-20T17:56:23.108 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:22 smithi005 ceph-mon[39562]: osdmap e1031: 12 total, 12 up, 12 in 2024-04-20T17:56:23.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:22 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:56:23.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:22 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:56:23.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:22 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1712553886' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-20T17:56:23.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:22 smithi106 ceph-mon[44423]: osdmap e1031: 12 total, 12 up, 12 in 2024-04-20T17:56:23.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:22 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:56:23.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:22 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:56:23.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:22 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1712553886' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-20T17:56:23.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:22 smithi171 ceph-mon[44189]: osdmap e1031: 12 total, 12 up, 12 in 2024-04-20T17:56:24.068 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:23 smithi005 ceph-mon[39562]: pgmap v2729: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:24.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:23 smithi106 ceph-mon[44423]: pgmap v2729: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:24.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:23 smithi171 ceph-mon[44189]: pgmap v2729: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:24.901 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-04-20T17:56:25.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:24 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1600277365' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:56:25.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:24 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1600277365' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:56:25.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:24 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1600277365' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-20T17:56:25.633 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 7 1 2024-04-20T17:56:25.914 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:56:26.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:25 smithi005 ceph-mon[39562]: pgmap v2730: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:26.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:25 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:56:26.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:25 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:56:26.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:25 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1600277365' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-20T17:56:26.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:25 smithi005 ceph-mon[39562]: osdmap e1032: 12 total, 12 up, 12 in 2024-04-20T17:56:26.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:25 smithi106 ceph-mon[44423]: pgmap v2730: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:25 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:56:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:25 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:56:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:25 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1600277365' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-20T17:56:26.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:25 smithi106 ceph-mon[44423]: osdmap e1032: 12 total, 12 up, 12 in 2024-04-20T17:56:26.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:25 smithi171 ceph-mon[44189]: pgmap v2730: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:26.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:25 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:56:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:25 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:56:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:25 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1600277365' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-20T17:56:26.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:25 smithi171 ceph-mon[44189]: osdmap e1032: 12 total, 12 up, 12 in 2024-04-20T17:56:28.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:27 smithi005 ceph-mon[39562]: 1.6 deep-scrub starts 2024-04-20T17:56:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:27 smithi005 ceph-mon[39562]: 1.6 deep-scrub ok 2024-04-20T17:56:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:27 smithi005 ceph-mon[39562]: pgmap v2732: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:27 smithi005 ceph-mon[39562]: 1.2 scrub starts 2024-04-20T17:56:28.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:27 smithi005 ceph-mon[39562]: 1.2 scrub ok 2024-04-20T17:56:28.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:27 smithi106 ceph-mon[44423]: 1.6 deep-scrub starts 2024-04-20T17:56:28.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:27 smithi106 ceph-mon[44423]: 1.6 deep-scrub ok 2024-04-20T17:56:28.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:27 smithi106 ceph-mon[44423]: pgmap v2732: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:27 smithi106 ceph-mon[44423]: 1.2 scrub starts 2024-04-20T17:56:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:27 smithi106 ceph-mon[44423]: 1.2 scrub ok 2024-04-20T17:56:28.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:27 smithi171 ceph-mon[44189]: 1.6 deep-scrub starts 2024-04-20T17:56:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:27 smithi171 ceph-mon[44189]: 1.6 deep-scrub ok 2024-04-20T17:56:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:27 smithi171 ceph-mon[44189]: pgmap v2732: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:27 smithi171 ceph-mon[44189]: 1.2 scrub starts 2024-04-20T17:56:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:27 smithi171 ceph-mon[44189]: 1.2 scrub ok 2024-04-20T17:56:28.914 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.7 to 1 (10000) 2024-04-20T17:56:29.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:28 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1997600078' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:56:29.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:28 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1997600078' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:56:29.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:28 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1997600078' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:56:29.789 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd primary-affinity 7 1 2024-04-20T17:56:30.073 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:56:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:29 smithi005 ceph-mon[39562]: pgmap v2733: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:29 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1997600078' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-20T17:56:30.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:29 smithi005 ceph-mon[39562]: osdmap e1033: 12 total, 12 up, 12 in 2024-04-20T17:56:30.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:29 smithi106 ceph-mon[44423]: pgmap v2733: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:30.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:29 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1997600078' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-20T17:56:30.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:29 smithi106 ceph-mon[44423]: osdmap e1033: 12 total, 12 up, 12 in 2024-04-20T17:56:30.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:29 smithi171 ceph-mon[44189]: pgmap v2733: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:30.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:29 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1997600078' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-20T17:56:30.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:29 smithi171 ceph-mon[44189]: osdmap e1033: 12 total, 12 up, 12 in 2024-04-20T17:56:32.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:31 smithi005 ceph-mon[39562]: pgmap v2735: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:32.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:31 smithi106 ceph-mon[44423]: pgmap v2735: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:32.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:31 smithi171 ceph-mon[44189]: pgmap v2735: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:32.921 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-04-20T17:56:33.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:32 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2259418138' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:56:33.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:32 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2259418138' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:56:33.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:32 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2259418138' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-20T17:56:33.712 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd reweight 0 1 2024-04-20T17:56:33.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:33 smithi005 ceph-mon[39562]: pgmap v2736: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:33.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:33 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2259418138' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-20T17:56:33.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:33 smithi005 ceph-mon[39562]: osdmap e1034: 12 total, 12 up, 12 in 2024-04-20T17:56:33.993 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:56:34.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:33 smithi106 ceph-mon[44423]: pgmap v2736: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:33 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2259418138' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-20T17:56:34.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:33 smithi106 ceph-mon[44423]: osdmap e1034: 12 total, 12 up, 12 in 2024-04-20T17:56:34.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:33 smithi171 ceph-mon[44189]: pgmap v2736: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:34.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:33 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2259418138' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-20T17:56:34.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:33 smithi171 ceph-mon[44189]: osdmap e1034: 12 total, 12 up, 12 in 2024-04-20T17:56:35.931 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.0 to 1 (10000) 2024-04-20T17:56:36.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:35 smithi005 ceph-mon[39562]: pgmap v2738: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:36.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:35 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:56:36.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:35 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:56:36.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:35 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2251446272' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:56:36.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:35 smithi106 ceph-mon[44423]: pgmap v2738: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:36.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:35 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:56:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:35 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:56:36.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:35 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2251446272' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:56:36.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:35 smithi171 ceph-mon[44189]: pgmap v2738: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:36.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:35 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:56:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:35 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:56:36.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:35 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2251446272' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:56:36.693 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd primary-affinity 0 1 2024-04-20T17:56:36.974 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:56:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:36 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2251446272' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-20T17:56:37.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:36 smithi005 ceph-mon[39562]: osdmap e1035: 12 total, 12 up, 12 in 2024-04-20T17:56:37.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:36 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2251446272' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-20T17:56:37.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:36 smithi106 ceph-mon[44423]: osdmap e1035: 12 total, 12 up, 12 in 2024-04-20T17:56:37.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:36 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2251446272' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-20T17:56:37.348 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:36 smithi171 ceph-mon[44189]: osdmap e1035: 12 total, 12 up, 12 in 2024-04-20T17:56:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:37 smithi005 ceph-mon[39562]: pgmap v2740: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:38.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:37 smithi106 ceph-mon[44423]: pgmap v2740: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:38.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:37 smithi171 ceph-mon[44189]: pgmap v2740: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:39.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:38 smithi005 ceph-mon[39562]: 1.5 scrub starts 2024-04-20T17:56:39.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:38 smithi005 ceph-mon[39562]: 1.5 scrub ok 2024-04-20T17:56:39.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:38 smithi106 ceph-mon[44423]: 1.5 scrub starts 2024-04-20T17:56:39.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:38 smithi106 ceph-mon[44423]: 1.5 scrub ok 2024-04-20T17:56:39.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:38 smithi171 ceph-mon[44189]: 1.5 scrub starts 2024-04-20T17:56:39.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:38 smithi171 ceph-mon[44189]: 1.5 scrub ok 2024-04-20T17:56:39.944 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-04-20T17:56:40.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:39 smithi005 ceph-mon[39562]: pgmap v2741: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:40.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:39 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2160972230' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:56:40.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:39 smithi106 ceph-mon[44423]: pgmap v2741: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:40.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:39 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2160972230' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:56:40.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:39 smithi171 ceph-mon[44189]: pgmap v2741: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:40.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:39 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2160972230' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-20T17:56:40.776 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-04-20T17:56:40.777 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-20T17:56:40.777 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-04-20T17:56:40.777 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-04-20T17:56:40.777 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-20T17:56:40.777 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-20T17:56:41.062 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:56:41.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:40 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2160972230' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-20T17:56:41.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:40 smithi005 ceph-mon[39562]: osdmap e1036: 12 total, 12 up, 12 in 2024-04-20T17:56:41.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:40 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2160972230' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-20T17:56:41.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:40 smithi106 ceph-mon[44423]: osdmap e1036: 12 total, 12 up, 12 in 2024-04-20T17:56:41.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:40 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2160972230' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-20T17:56:41.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:40 smithi171 ceph-mon[44189]: osdmap e1036: 12 total, 12 up, 12 in 2024-04-20T17:56:42.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:41 smithi005 ceph-mon[39562]: 1.a deep-scrub starts 2024-04-20T17:56:42.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:41 smithi005 ceph-mon[39562]: 1.a deep-scrub ok 2024-04-20T17:56:42.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:41 smithi005 ceph-mon[39562]: pgmap v2743: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:42.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:41 smithi106 ceph-mon[44423]: 1.a deep-scrub starts 2024-04-20T17:56:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:41 smithi106 ceph-mon[44423]: 1.a deep-scrub ok 2024-04-20T17:56:42.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:41 smithi106 ceph-mon[44423]: pgmap v2743: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:42.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:41 smithi171 ceph-mon[44189]: 1.a deep-scrub starts 2024-04-20T17:56:42.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:41 smithi171 ceph-mon[44189]: 1.a deep-scrub ok 2024-04-20T17:56:42.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:41 smithi171 ceph-mon[44189]: pgmap v2743: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:43.109 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:56:43.110 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":1037,"fsid":"13c6e76c-ff37-11ee-bc93-c7b262605968","created":"2024-04-20T16:58:25.272557+0000","modified":"2024-04-20T17:56:42.953244+0000","last_up_change":"2024-04-20T17:54:54.989339+0000","last_in_change":"2024-04-20T17:55:03.875085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":38,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T17:01:35.909493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"838","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"818","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.2899999618530273,"score_stable":3.2899999618530273,"optimal_score":0.82999998331069946,"raw_score_acting":2.7300000190734863,"raw_score_stable":2.7300000190734863,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":975,"up_from":995,"up_thru":845,"down_at":976,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6801","nonce":415791178}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6802","nonce":415791178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6804","nonce":415791178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6803","nonce":415791178}]},"public_addr":"172.21.15.5:6801/415791178","cluster_addr":"172.21.15.5:6802/415791178","heartbeat_back_addr":"172.21.15.5:6804/415791178","heartbeat_front_addr":"172.21.15.5:6803/415791178","state":["exists","up"]},{"osd":1,"uuid":"aed0c8eb-0e81-4e85-ab8f-a60420060656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":613,"up_from":659,"up_thru":915,"down_at":614,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6805","nonce":2415652792}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6806","nonce":2415652792}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6808","nonce":2415652792}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6807","nonce":2415652792}]},"public_addr":"172.21.15.5:6805/2415652792","cluster_addr":"172.21.15.5:6806/2415652792","heartbeat_back_addr":"172.21.15.5:6808/2415652792","heartbeat_front_addr":"172.21.15.5:6807/2415652792","state":["exists","up"]},{"osd":2,"uuid":"f733a6a2-19c2-4c7a-af9c-04e282062bb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":20,"last_clean_end":169,"up_from":202,"up_thru":959,"down_at":170,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6809","nonce":2120275501}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6810","nonce":2120275501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6812","nonce":2120275501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6811","nonce":2120275501}]},"public_addr":"172.21.15.5:6809/2120275501","cluster_addr":"172.21.15.5:6810/2120275501","heartbeat_back_addr":"172.21.15.5:6812/2120275501","heartbeat_front_addr":"172.21.15.5:6811/2120275501","state":["exists","up"]},{"osd":3,"uuid":"8c92f1df-e298-4606-88ae-b8a8e79628a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":538,"up_from":561,"up_thru":916,"down_at":539,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6813","nonce":2272824046}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6814","nonce":2272824046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6816","nonce":2272824046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6815","nonce":2272824046}]},"public_addr":"172.21.15.5:6813/2272824046","cluster_addr":"172.21.15.5:6814/2272824046","heartbeat_back_addr":"172.21.15.5:6816/2272824046","heartbeat_front_addr":"172.21.15.5:6815/2272824046","state":["exists","up"]},{"osd":4,"uuid":"46df16f1-9607-4d7b-9be8-9d4236e160d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":894,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6800","nonce":1839524469}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6801","nonce":1839524469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6803","nonce":1839524469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6802","nonce":1839524469}]},"public_addr":"172.21.15.106:6800/1839524469","cluster_addr":"172.21.15.106:6801/1839524469","heartbeat_back_addr":"172.21.15.106:6803/1839524469","heartbeat_front_addr":"172.21.15.106:6802/1839524469","state":["exists","up"]},{"osd":5,"uuid":"6603231d-1e5c-4575-bfaa-2eeca2c12215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":1007,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6804","nonce":2450315908}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6805","nonce":2450315908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6807","nonce":2450315908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6806","nonce":2450315908}]},"public_addr":"172.21.15.106:6804/2450315908","cluster_addr":"172.21.15.106:6805/2450315908","heartbeat_back_addr":"172.21.15.106:6807/2450315908","heartbeat_front_addr":"172.21.15.106:6806/2450315908","state":["exists","up"]},{"osd":6,"uuid":"783a55a6-f176-42be-8c8b-af8481d08dc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":48,"last_clean_end":503,"up_from":535,"up_thru":894,"down_at":504,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6808","nonce":2439382182}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6809","nonce":2439382182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6811","nonce":2439382182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6810","nonce":2439382182}]},"public_addr":"172.21.15.106:6808/2439382182","cluster_addr":"172.21.15.106:6809/2439382182","heartbeat_back_addr":"172.21.15.106:6811/2439382182","heartbeat_front_addr":"172.21.15.106:6810/2439382182","state":["exists","up"]},{"osd":7,"uuid":"b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":398,"last_clean_end":721,"up_from":799,"up_thru":999,"down_at":722,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6812","nonce":2531699401}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6813","nonce":2531699401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6815","nonce":2531699401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6814","nonce":2531699401}]},"public_addr":"172.21.15.106:6812/2531699401","cluster_addr":"172.21.15.106:6813/2531699401","heartbeat_back_addr":"172.21.15.106:6815/2531699401","heartbeat_front_addr":"172.21.15.106:6814/2531699401","state":["exists","up"]},{"osd":8,"uuid":"1cb232ad-1f59-4a0a-a215-e67419ad6476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":950,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6800","nonce":4123058942}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6801","nonce":4123058942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6803","nonce":4123058942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6802","nonce":4123058942}]},"public_addr":"172.21.15.171:6800/4123058942","cluster_addr":"172.21.15.171:6801/4123058942","heartbeat_back_addr":"172.21.15.171:6803/4123058942","heartbeat_front_addr":"172.21.15.171:6802/4123058942","state":["exists","up"]},{"osd":9,"uuid":"b6c17147-b3dc-475b-bb20-f9f712fb99e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":949,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6804","nonce":2877589636}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6805","nonce":2877589636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6807","nonce":2877589636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6806","nonce":2877589636}]},"public_addr":"172.21.15.171:6804/2877589636","cluster_addr":"172.21.15.171:6805/2877589636","heartbeat_back_addr":"172.21.15.171:6807/2877589636","heartbeat_front_addr":"172.21.15.171:6806/2877589636","state":["exists","up"]},{"osd":10,"uuid":"1adcdf55-f002-4a39-91e7-378e5fae849c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":969,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6808","nonce":181490621}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6809","nonce":181490621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6811","nonce":181490621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6810","nonce":181490621}]},"public_addr":"172.21.15.171:6808/181490621","cluster_addr":"172.21.15.171:6809/181490621","heartbeat_back_addr":"172.21.15.171:6811/181490621","heartbeat_front_addr":"172.21.15.171:6810/181490621","state":["exists","up"]},{"osd":11,"uuid":"737c5aec-694a-4968-819d-fccf22661f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":92,"up_thru":894,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6812","nonce":3373585701}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6813","nonce":3373585701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6815","nonce":3373585701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6814","nonce":3373585701}]},"public_addr":"172.21.15.171:6812/3373585701","cluster_addr":"172.21.15.171:6813/3373585701","heartbeat_back_addr":"172.21.15.171:6815/3373585701","heartbeat_front_addr":"172.21.15.171:6814/3373585701","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:56:11.320565+0000","dead_epoch":975},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:56:21.315407+0000","dead_epoch":613},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:56:20.796037+0000","dead_epoch":169},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:56:07.176134+0000","dead_epoch":538},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:55:57.493755+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:56:24.849093+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:55:34.576330+0000","dead_epoch":503},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:56:35.145537+0000","dead_epoch":721},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:55:51.793731+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:56:01.782603+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:56:00.510272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:56:42.118644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.5:0/3881726425":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/338649532":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/1227843664":"2024-04-21T16:58:55.392162+0000","172.21.15.5:6800/1876064631":"2024-04-21T16:58:55.392162+0000","172.21.15.5:0/160689449":"2024-04-21T16:58:55.392162+0000","172.21.15.5:6800/3521715286":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/3841663462":"2024-04-21T16:58:55.392162+0000","172.21.15.5:0/687690919":"2024-04-21T16:59:25.808337+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T17:56:43.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:42 smithi005 ceph-mon[39562]: 1.8 scrub starts 2024-04-20T17:56:43.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:42 smithi005 ceph-mon[39562]: 1.8 scrub ok 2024-04-20T17:56:43.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:42 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:56:43.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:42 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:56:43.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:42 smithi005 ceph-mon[39562]: pgmap v2744: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:43.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:42 smithi106 ceph-mon[44423]: 1.8 scrub starts 2024-04-20T17:56:43.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:42 smithi106 ceph-mon[44423]: 1.8 scrub ok 2024-04-20T17:56:43.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:42 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:56:43.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:42 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:56:43.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:42 smithi106 ceph-mon[44423]: pgmap v2744: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:43.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:42 smithi171 ceph-mon[44189]: 1.8 scrub starts 2024-04-20T17:56:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:42 smithi171 ceph-mon[44189]: 1.8 scrub ok 2024-04-20T17:56:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:42 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:56:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:42 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:56:43.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:42 smithi171 ceph-mon[44189]: pgmap v2744: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:43.821 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-20T17:56:43.821 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-20T17:56:44.106 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:56:44.210 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:43 smithi005 ceph-mon[39562]: osdmap e1037: 12 total, 12 up, 12 in 2024-04-20T17:56:44.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:43 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3664392446' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:56:44.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:43 smithi106 ceph-mon[44423]: osdmap e1037: 12 total, 12 up, 12 in 2024-04-20T17:56:44.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:43 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3664392446' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:56:44.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:43 smithi171 ceph-mon[44189]: osdmap e1037: 12 total, 12 up, 12 in 2024-04-20T17:56:44.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:43 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3664392446' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:56:45.236 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:44 smithi005 ceph-mon[39562]: pgmap v2746: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:45.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:44 smithi106 ceph-mon[44423]: pgmap v2746: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:45.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:44 smithi171 ceph-mon[44189]: pgmap v2746: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:45.997 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:56:45.997 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":1037,"fsid":"13c6e76c-ff37-11ee-bc93-c7b262605968","created":"2024-04-20T16:58:25.272557+0000","modified":"2024-04-20T17:56:42.953244+0000","last_up_change":"2024-04-20T17:54:54.989339+0000","last_in_change":"2024-04-20T17:55:03.875085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":38,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T17:01:35.909493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"838","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"818","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.2899999618530273,"score_stable":3.2899999618530273,"optimal_score":0.82999998331069946,"raw_score_acting":2.7300000190734863,"raw_score_stable":2.7300000190734863,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ca04e7fd-01ed-4e82-82a1-4969fe9d2e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":975,"up_from":995,"up_thru":845,"down_at":976,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6801","nonce":415791178}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6802","nonce":415791178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6804","nonce":415791178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6803","nonce":415791178}]},"public_addr":"172.21.15.5:6801/415791178","cluster_addr":"172.21.15.5:6802/415791178","heartbeat_back_addr":"172.21.15.5:6804/415791178","heartbeat_front_addr":"172.21.15.5:6803/415791178","state":["exists","up"]},{"osd":1,"uuid":"aed0c8eb-0e81-4e85-ab8f-a60420060656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":613,"up_from":659,"up_thru":915,"down_at":614,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6805","nonce":2415652792}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6806","nonce":2415652792}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6808","nonce":2415652792}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6807","nonce":2415652792}]},"public_addr":"172.21.15.5:6805/2415652792","cluster_addr":"172.21.15.5:6806/2415652792","heartbeat_back_addr":"172.21.15.5:6808/2415652792","heartbeat_front_addr":"172.21.15.5:6807/2415652792","state":["exists","up"]},{"osd":2,"uuid":"f733a6a2-19c2-4c7a-af9c-04e282062bb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":20,"last_clean_end":169,"up_from":202,"up_thru":959,"down_at":170,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6809","nonce":2120275501}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6810","nonce":2120275501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6812","nonce":2120275501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6811","nonce":2120275501}]},"public_addr":"172.21.15.5:6809/2120275501","cluster_addr":"172.21.15.5:6810/2120275501","heartbeat_back_addr":"172.21.15.5:6812/2120275501","heartbeat_front_addr":"172.21.15.5:6811/2120275501","state":["exists","up"]},{"osd":3,"uuid":"8c92f1df-e298-4606-88ae-b8a8e79628a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":538,"up_from":561,"up_thru":916,"down_at":539,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6813","nonce":2272824046}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6814","nonce":2272824046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6816","nonce":2272824046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.5:6815","nonce":2272824046}]},"public_addr":"172.21.15.5:6813/2272824046","cluster_addr":"172.21.15.5:6814/2272824046","heartbeat_back_addr":"172.21.15.5:6816/2272824046","heartbeat_front_addr":"172.21.15.5:6815/2272824046","state":["exists","up"]},{"osd":4,"uuid":"46df16f1-9607-4d7b-9be8-9d4236e160d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":894,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6800","nonce":1839524469}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6801","nonce":1839524469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6803","nonce":1839524469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6802","nonce":1839524469}]},"public_addr":"172.21.15.106:6800/1839524469","cluster_addr":"172.21.15.106:6801/1839524469","heartbeat_back_addr":"172.21.15.106:6803/1839524469","heartbeat_front_addr":"172.21.15.106:6802/1839524469","state":["exists","up"]},{"osd":5,"uuid":"6603231d-1e5c-4575-bfaa-2eeca2c12215","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":1007,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6804","nonce":2450315908}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6805","nonce":2450315908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6807","nonce":2450315908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6806","nonce":2450315908}]},"public_addr":"172.21.15.106:6804/2450315908","cluster_addr":"172.21.15.106:6805/2450315908","heartbeat_back_addr":"172.21.15.106:6807/2450315908","heartbeat_front_addr":"172.21.15.106:6806/2450315908","state":["exists","up"]},{"osd":6,"uuid":"783a55a6-f176-42be-8c8b-af8481d08dc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":48,"last_clean_end":503,"up_from":535,"up_thru":894,"down_at":504,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6808","nonce":2439382182}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6809","nonce":2439382182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6811","nonce":2439382182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6810","nonce":2439382182}]},"public_addr":"172.21.15.106:6808/2439382182","cluster_addr":"172.21.15.106:6809/2439382182","heartbeat_back_addr":"172.21.15.106:6811/2439382182","heartbeat_front_addr":"172.21.15.106:6810/2439382182","state":["exists","up"]},{"osd":7,"uuid":"b3b92f46-4f19-47d5-b05b-c1ae9fadf7b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":398,"last_clean_end":721,"up_from":799,"up_thru":999,"down_at":722,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6812","nonce":2531699401}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6813","nonce":2531699401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6815","nonce":2531699401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.106:6814","nonce":2531699401}]},"public_addr":"172.21.15.106:6812/2531699401","cluster_addr":"172.21.15.106:6813/2531699401","heartbeat_back_addr":"172.21.15.106:6815/2531699401","heartbeat_front_addr":"172.21.15.106:6814/2531699401","state":["exists","up"]},{"osd":8,"uuid":"1cb232ad-1f59-4a0a-a215-e67419ad6476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":950,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6800","nonce":4123058942}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6801","nonce":4123058942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6803","nonce":4123058942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6802","nonce":4123058942}]},"public_addr":"172.21.15.171:6800/4123058942","cluster_addr":"172.21.15.171:6801/4123058942","heartbeat_back_addr":"172.21.15.171:6803/4123058942","heartbeat_front_addr":"172.21.15.171:6802/4123058942","state":["exists","up"]},{"osd":9,"uuid":"b6c17147-b3dc-475b-bb20-f9f712fb99e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":949,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6804","nonce":2877589636}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6805","nonce":2877589636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6807","nonce":2877589636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6806","nonce":2877589636}]},"public_addr":"172.21.15.171:6804/2877589636","cluster_addr":"172.21.15.171:6805/2877589636","heartbeat_back_addr":"172.21.15.171:6807/2877589636","heartbeat_front_addr":"172.21.15.171:6806/2877589636","state":["exists","up"]},{"osd":10,"uuid":"1adcdf55-f002-4a39-91e7-378e5fae849c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":969,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6808","nonce":181490621}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6809","nonce":181490621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6811","nonce":181490621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6810","nonce":181490621}]},"public_addr":"172.21.15.171:6808/181490621","cluster_addr":"172.21.15.171:6809/181490621","heartbeat_back_addr":"172.21.15.171:6811/181490621","heartbeat_front_addr":"172.21.15.171:6810/181490621","state":["exists","up"]},{"osd":11,"uuid":"737c5aec-694a-4968-819d-fccf22661f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":92,"up_thru":894,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6812","nonce":3373585701}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6813","nonce":3373585701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6815","nonce":3373585701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6814","nonce":3373585701}]},"public_addr":"172.21.15.171:6812/3373585701","cluster_addr":"172.21.15.171:6813/3373585701","heartbeat_back_addr":"172.21.15.171:6815/3373585701","heartbeat_front_addr":"172.21.15.171:6814/3373585701","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:56:11.320565+0000","dead_epoch":975},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:56:21.315407+0000","dead_epoch":613},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:56:20.796037+0000","dead_epoch":169},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:56:07.176134+0000","dead_epoch":538},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:55:57.493755+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:56:24.849093+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:55:34.576330+0000","dead_epoch":503},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:56:35.145537+0000","dead_epoch":721},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:55:51.793731+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:56:01.782603+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:56:00.510272+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T17:56:42.118644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.5:0/3881726425":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/338649532":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/1227843664":"2024-04-21T16:58:55.392162+0000","172.21.15.5:6800/1876064631":"2024-04-21T16:58:55.392162+0000","172.21.15.5:0/160689449":"2024-04-21T16:58:55.392162+0000","172.21.15.5:6800/3521715286":"2024-04-21T16:59:25.808337+0000","172.21.15.5:0/3841663462":"2024-04-21T16:58:55.392162+0000","172.21.15.5:0/687690919":"2024-04-21T16:59:25.808337+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T17:56:46.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:46 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/211765323' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:56:46.370 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:46 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/211765323' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:56:46.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:46 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/211765323' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-20T17:56:46.758 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-20T17:56:46.758 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-20T17:56:46.759 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-20T17:56:46.759 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-20T17:56:46.759 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-20T17:56:46.760 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-20T17:56:46.760 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-20T17:56:46.761 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-20T17:56:46.761 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-20T17:56:46.761 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-20T17:56:46.762 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-20T17:56:46.762 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-20T17:56:47.547 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:56:47.556 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:56:47.560 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:56:48.198 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:47 smithi005 ceph-mon[39562]: pgmap v2747: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:48.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:47 smithi106 ceph-mon[44423]: pgmap v2747: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:48.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:47 smithi171 ceph-mon[44189]: pgmap v2747: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:49.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:49 smithi005 ceph-mon[39562]: pgmap v2748: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:50.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:49 smithi106 ceph-mon[44423]: pgmap v2748: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:50.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:49 smithi171 ceph-mon[44189]: pgmap v2748: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:50.488 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:56:50.490 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:56:50.496 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:56:50.496 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:56:50.497 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:56:50.500 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:56:50.504 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:56:50.509 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:56:50.510 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:56:51.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:50 smithi005 ceph-mon[39562]: 1.9 scrub ok 2024-04-20T17:56:51.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:50 smithi106 ceph-mon[44423]: 1.9 scrub ok 2024-04-20T17:56:51.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:50 smithi171 ceph-mon[44189]: 1.9 scrub ok 2024-04-20T17:56:52.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:51 smithi005 ceph-mon[39562]: pgmap v2749: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:52.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:51 smithi106 ceph-mon[44423]: pgmap v2749: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:52.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:51 smithi171 ceph-mon[44189]: pgmap v2749: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:53.937 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:53 smithi005 ceph-mon[39562]: 1.3 deep-scrub starts 2024-04-20T17:56:53.937 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:53 smithi005 ceph-mon[39562]: pgmap v2750: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:53.937 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:53 smithi005 ceph-mon[39562]: 1.3 deep-scrub ok 2024-04-20T17:56:54.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:53 smithi106 ceph-mon[44423]: 1.3 deep-scrub starts 2024-04-20T17:56:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:53 smithi106 ceph-mon[44423]: pgmap v2750: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:54.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:53 smithi106 ceph-mon[44423]: 1.3 deep-scrub ok 2024-04-20T17:56:54.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:53 smithi171 ceph-mon[44189]: 1.3 deep-scrub starts 2024-04-20T17:56:54.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:53 smithi171 ceph-mon[44189]: pgmap v2750: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:54.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:53 smithi171 ceph-mon[44189]: 1.3 deep-scrub ok 2024-04-20T17:56:55.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:54 smithi005 ceph-mon[39562]: 1.0 scrub starts 2024-04-20T17:56:55.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:54 smithi005 ceph-mon[39562]: 1.0 scrub ok 2024-04-20T17:56:55.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:54 smithi106 ceph-mon[44423]: 1.0 scrub starts 2024-04-20T17:56:55.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:54 smithi106 ceph-mon[44423]: 1.0 scrub ok 2024-04-20T17:56:55.315 INFO:teuthology.orchestra.run.smithi005.stdout:4273492459545 2024-04-20T17:56:55.316 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-20T17:56:55.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:54 smithi171 ceph-mon[44189]: 1.0 scrub starts 2024-04-20T17:56:55.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:54 smithi171 ceph-mon[44189]: 1.0 scrub ok 2024-04-20T17:56:56.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:55 smithi005 ceph-mon[39562]: pgmap v2751: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:56.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:55 smithi106 ceph-mon[44423]: pgmap v2751: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:56.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:55 smithi171 ceph-mon[44189]: pgmap v2751: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:57.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:56 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:56:57.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:56 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:56:57.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:56 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:56:57.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:56 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:56:57.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:56 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:56:57.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:56 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:56:58.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:57 smithi005 ceph-mon[39562]: pgmap v2752: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:58.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:57 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:56:58.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:57 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:56:58.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:57 smithi005 ceph-mon[39562]: osdmap e1038: 12 total, 12 up, 12 in 2024-04-20T17:56:58.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:57 smithi106 ceph-mon[44423]: pgmap v2752: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:58.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:57 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:56:58.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:57 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:56:58.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:57 smithi106 ceph-mon[44423]: osdmap e1038: 12 total, 12 up, 12 in 2024-04-20T17:56:58.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:57 smithi171 ceph-mon[44189]: pgmap v2752: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:58.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:57 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:56:58.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:57 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:56:58.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:57 smithi171 ceph-mon[44189]: osdmap e1038: 12 total, 12 up, 12 in 2024-04-20T17:56:59.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:56:58 smithi005 ceph-mon[39562]: pgmap v2754: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:59.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:56:58 smithi106 ceph-mon[44423]: pgmap v2754: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:56:59.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:56:58 smithi171 ceph-mon[44189]: pgmap v2754: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:02.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:01 smithi005 ceph-mon[39562]: pgmap v2755: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:02.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:01 smithi106 ceph-mon[44423]: pgmap v2755: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:02.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:01 smithi171 ceph-mon[44189]: pgmap v2755: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:02.803 INFO:teuthology.orchestra.run.smithi005.stdout:395136991849 2024-04-20T17:57:02.804 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-20T17:57:03.142 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:02 smithi005 ceph-mon[39562]: 1.4 scrub starts 2024-04-20T17:57:03.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:02 smithi106 ceph-mon[44423]: 1.4 scrub starts 2024-04-20T17:57:03.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:02 smithi171 ceph-mon[44189]: 1.4 scrub starts 2024-04-20T17:57:03.551 INFO:teuthology.orchestra.run.smithi005.stdout:133143986831 2024-04-20T17:57:03.552 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-20T17:57:04.191 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:03 smithi005 ceph-mon[39562]: 1.4 scrub ok 2024-04-20T17:57:04.191 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:03 smithi005 ceph-mon[39562]: pgmap v2756: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:04.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:03 smithi106 ceph-mon[44423]: 1.4 scrub ok 2024-04-20T17:57:04.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:03 smithi106 ceph-mon[44423]: pgmap v2756: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:04.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:03 smithi171 ceph-mon[44189]: 1.4 scrub ok 2024-04-20T17:57:04.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:03 smithi171 ceph-mon[44189]: pgmap v2756: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:04.516 INFO:teuthology.orchestra.run.smithi005.stdout:360777253487 2024-04-20T17:57:04.517 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-20T17:57:05.328 INFO:teuthology.orchestra.run.smithi005.stdout:287762809468 2024-04-20T17:57:05.329 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-20T17:57:06.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:05 smithi005 ceph-mon[39562]: pgmap v2757: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:06.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:05 smithi106 ceph-mon[44423]: pgmap v2757: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:06.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:05 smithi171 ceph-mon[44189]: pgmap v2757: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:08.185 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:07 smithi005 ceph-mon[39562]: pgmap v2758: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:08.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:07 smithi106 ceph-mon[44423]: pgmap v2758: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:08.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:07 smithi171 ceph-mon[44189]: pgmap v2758: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:08.409 INFO:teuthology.orchestra.run.smithi005.stdout:2409476653382 2024-04-20T17:57:08.410 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-20T17:57:10.210 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:09 smithi005 ceph-mon[39562]: pgmap v2759: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:10.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:09 smithi106 ceph-mon[44423]: pgmap v2759: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:10.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:09 smithi171 ceph-mon[44189]: pgmap v2759: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:10.669 INFO:teuthology.orchestra.run.smithi005.stdout:2830383448321 2024-04-20T17:57:10.669 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-20T17:57:11.607 INFO:teuthology.orchestra.run.smithi005.stdout:158913790599 2024-04-20T17:57:11.607 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-20T17:57:12.036 INFO:teuthology.orchestra.run.smithi005.stdout:867583394333 2024-04-20T17:57:12.036 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-20T17:57:12.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:11 smithi005 ceph-mon[39562]: 1.1 scrub starts 2024-04-20T17:57:12.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:11 smithi005 ceph-mon[39562]: 1.1 scrub ok 2024-04-20T17:57:12.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:11 smithi005 ceph-mon[39562]: pgmap v2760: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:12.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:11 smithi106 ceph-mon[44423]: 1.1 scrub starts 2024-04-20T17:57:12.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:11 smithi106 ceph-mon[44423]: 1.1 scrub ok 2024-04-20T17:57:12.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:11 smithi106 ceph-mon[44423]: pgmap v2760: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:12.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:11 smithi171 ceph-mon[44189]: 1.1 scrub starts 2024-04-20T17:57:12.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:11 smithi171 ceph-mon[44189]: 1.1 scrub ok 2024-04-20T17:57:12.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:11 smithi171 ceph-mon[44189]: pgmap v2760: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:12.404 INFO:teuthology.orchestra.run.smithi005.stdout:313532613238 2024-04-20T17:57:12.404 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-20T17:57:12.851 INFO:teuthology.orchestra.run.smithi005.stdout:3431678869659 2024-04-20T17:57:12.852 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-20T17:57:13.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:12 smithi005 ceph-mon[39562]: 1.7 scrub starts 2024-04-20T17:57:13.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:12 smithi005 ceph-mon[39562]: 1.7 scrub ok 2024-04-20T17:57:13.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:12 smithi106 ceph-mon[44423]: 1.7 scrub starts 2024-04-20T17:57:13.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:12 smithi106 ceph-mon[44423]: 1.7 scrub ok 2024-04-20T17:57:13.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:12 smithi171 ceph-mon[44189]: 1.7 scrub starts 2024-04-20T17:57:13.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:12 smithi171 ceph-mon[44189]: 1.7 scrub ok 2024-04-20T17:57:13.396 INFO:teuthology.orchestra.run.smithi005.stdout:2297807503699 2024-04-20T17:57:13.396 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-20T17:57:13.453 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:57:13.455 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:57:13.461 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:57:13.461 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:57:13.546 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:57:13.780 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:57:13.788 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:57:13.793 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:57:13.796 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:57:14.036 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:13 smithi005 ceph-mon[39562]: pgmap v2761: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:14.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:13 smithi106 ceph-mon[44423]: pgmap v2761: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:14.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:13 smithi171 ceph-mon[44189]: pgmap v2761: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:16.207 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:15 smithi005 ceph-mon[39562]: pgmap v2762: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:16.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:15 smithi106 ceph-mon[44423]: pgmap v2762: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:16.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:15 smithi171 ceph-mon[44189]: pgmap v2762: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:17.164 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:16 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:57:17.164 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:16 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:57:17.164 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:16 smithi005 ceph-mon[39562]: osdmap e1039: 12 total, 12 up, 12 in 2024-04-20T17:57:17.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:16 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:57:17.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:16 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:57:17.262 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:16 smithi106 ceph-mon[44423]: osdmap e1039: 12 total, 12 up, 12 in 2024-04-20T17:57:17.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:16 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:57:17.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:16 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:57:17.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:16 smithi171 ceph-mon[44189]: osdmap e1039: 12 total, 12 up, 12 in 2024-04-20T17:57:18.210 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:17 smithi106 ceph-mon[44423]: pgmap v2764: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:18.210 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:17 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:57:18.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:17 smithi005 ceph-mon[39562]: pgmap v2764: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:18.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:17 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:57:18.216 INFO:teuthology.orchestra.run.smithi005.stdout:4273492459550 2024-04-20T17:57:18.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:17 smithi171 ceph-mon[44189]: pgmap v2764: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:18.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:17 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:57:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:18 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/134659765' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-20T17:57:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:18 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:57:19.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:18 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:57:19.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:18 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/134659765' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-20T17:57:19.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:18 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:57:19.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:18 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:57:19.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:18 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/134659765' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-20T17:57:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:18 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:57:19.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:18 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:57:20.116 INFO:teuthology.orchestra.run.smithi005.stdout:158913790603 2024-04-20T17:57:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:19 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:57:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:19 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:57:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:19 smithi005 ceph-mon[39562]: pgmap v2765: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:19 smithi005 ceph-mon[39562]: osdmap e1040: 12 total, 12 up, 12 in 2024-04-20T17:57:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:19 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:57:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:19 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:57:20.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:19 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:57:20.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:19 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:57:20.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:19 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:57:20.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:19 smithi106 ceph-mon[44423]: pgmap v2765: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:20.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:19 smithi106 ceph-mon[44423]: osdmap e1040: 12 total, 12 up, 12 in 2024-04-20T17:57:20.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:19 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:57:20.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:19 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:57:20.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:19 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:57:20.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:19 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:57:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:19 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:57:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:19 smithi171 ceph-mon[44189]: pgmap v2765: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:19 smithi171 ceph-mon[44189]: osdmap e1040: 12 total, 12 up, 12 in 2024-04-20T17:57:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:19 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-20T17:57:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:19 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-20T17:57:20.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:19 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' 2024-04-20T17:57:21.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:20 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2054609394' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-20T17:57:21.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:20 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2054609394' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-20T17:57:21.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:20 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2054609394' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-20T17:57:22.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:21 smithi005 ceph-mon[39562]: pgmap v2767: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:22.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:21 smithi106 ceph-mon[44423]: pgmap v2767: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:22.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:21 smithi171 ceph-mon[44189]: pgmap v2767: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:23.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:23 smithi106 ceph-mon[44423]: pgmap v2768: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:23.324 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:23 smithi005 ceph-mon[39562]: pgmap v2768: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:23.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:23 smithi171 ceph-mon[44189]: pgmap v2768: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:24.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:24 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3456918261' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-20T17:57:24.328 INFO:tasks.cephadm.ceph_manager.ceph:need seq 4273492459545 got 4273492459550 for osd.0 2024-04-20T17:57:24.328 DEBUG:teuthology.parallel:result is None 2024-04-20T17:57:24.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:24 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3456918261' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-20T17:57:24.574 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:57:24.575 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:57:24.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:24 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3456918261' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-20T17:57:25.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:25 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2838934378' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-20T17:57:25.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:25 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:57:25.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:25 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:57:25.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:25 smithi005 ceph-mon[39562]: pgmap v2769: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:25.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:25 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2838934378' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-20T17:57:25.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:25 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:57:25.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:25 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:57:25.511 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:25 smithi106 ceph-mon[44423]: pgmap v2769: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:25.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:25 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2838934378' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-20T17:57:25.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:25 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:57:25.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:25 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:57:25.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:25 smithi171 ceph-mon[44189]: pgmap v2769: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:26.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:26 smithi005 ceph-mon[39562]: osdmap e1041: 12 total, 12 up, 12 in 2024-04-20T17:57:26.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:26 smithi106 ceph-mon[44423]: osdmap e1041: 12 total, 12 up, 12 in 2024-04-20T17:57:26.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:26 smithi171 ceph-mon[44189]: osdmap e1041: 12 total, 12 up, 12 in 2024-04-20T17:57:27.681 INFO:teuthology.orchestra.run.smithi005.stdout:2830383448325 2024-04-20T17:57:28.099 INFO:teuthology.orchestra.run.smithi005.stdout:3431678869662 2024-04-20T17:57:28.210 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:27 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:57:28.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:27 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:57:28.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:27 smithi005 ceph-mon[39562]: pgmap v2771: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:28.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:27 smithi005 ceph-mon[39562]: osdmap e1042: 12 total, 12 up, 12 in 2024-04-20T17:57:28.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:27 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/148046168' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-20T17:57:28.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:27 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:57:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:27 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:57:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:27 smithi106 ceph-mon[44423]: pgmap v2771: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:27 smithi106 ceph-mon[44423]: osdmap e1042: 12 total, 12 up, 12 in 2024-04-20T17:57:28.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:27 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/148046168' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-20T17:57:28.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:27 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:57:28.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:27 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:57:28.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:27 smithi171 ceph-mon[44189]: pgmap v2771: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:27 smithi171 ceph-mon[44189]: osdmap e1042: 12 total, 12 up, 12 in 2024-04-20T17:57:28.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:27 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/148046168' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-20T17:57:29.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:28 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/1357062809' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-20T17:57:29.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:28 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/1357062809' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-20T17:57:29.322 INFO:teuthology.orchestra.run.smithi005.stdout:867583394337 2024-04-20T17:57:29.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:28 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/1357062809' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-20T17:57:30.204 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:29 smithi005 ceph-mon[39562]: pgmap v2773: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:30.204 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:29 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:57:30.204 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:29 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:57:30.204 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:29 smithi005 ceph-mon[39562]: osdmap e1043: 12 total, 12 up, 12 in 2024-04-20T17:57:30.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:29 smithi106 ceph-mon[44423]: pgmap v2773: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:30.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:29 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:57:30.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:29 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:57:30.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:29 smithi106 ceph-mon[44423]: osdmap e1043: 12 total, 12 up, 12 in 2024-04-20T17:57:30.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:29 smithi171 ceph-mon[44189]: pgmap v2773: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:30.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:29 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:57:30.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:29 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:57:30.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:29 smithi171 ceph-mon[44189]: osdmap e1043: 12 total, 12 up, 12 in 2024-04-20T17:57:30.345 INFO:teuthology.orchestra.run.smithi005.stdout:287762809472 2024-04-20T17:57:30.995 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913790599 got 158913790603 for osd.5 2024-04-20T17:57:30.995 DEBUG:teuthology.parallel:result is None 2024-04-20T17:57:31.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:30 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/4022255471' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-20T17:57:31.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:30 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/4022255471' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-20T17:57:31.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:30 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/4022255471' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-20T17:57:31.344 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2830383448321 got 2830383448325 for osd.1 2024-04-20T17:57:31.344 DEBUG:teuthology.parallel:result is None 2024-04-20T17:57:31.788 INFO:tasks.cephadm.ceph_manager.ceph:need seq 3431678869659 got 3431678869662 for osd.7 2024-04-20T17:57:31.788 DEBUG:teuthology.parallel:result is None 2024-04-20T17:57:31.873 INFO:teuthology.orchestra.run.smithi005.stdout:395136991855 2024-04-20T17:57:32.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:31 smithi005 ceph-mon[39562]: pgmap v2775: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:32.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:31 smithi005 ceph-mon[39562]: 1.2 scrub starts 2024-04-20T17:57:32.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:31 smithi005 ceph-mon[39562]: 1.2 scrub ok 2024-04-20T17:57:32.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:31 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/278252003' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-20T17:57:32.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:31 smithi106 ceph-mon[44423]: pgmap v2775: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:32.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:31 smithi106 ceph-mon[44423]: 1.2 scrub starts 2024-04-20T17:57:32.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:31 smithi106 ceph-mon[44423]: 1.2 scrub ok 2024-04-20T17:57:32.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:31 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/278252003' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-20T17:57:32.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:31 smithi171 ceph-mon[44189]: pgmap v2775: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:32.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:31 smithi171 ceph-mon[44189]: 1.2 scrub starts 2024-04-20T17:57:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:31 smithi171 ceph-mon[44189]: 1.2 scrub ok 2024-04-20T17:57:32.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:31 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/278252003' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-20T17:57:32.440 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762809468 got 287762809472 for osd.8 2024-04-20T17:57:32.440 DEBUG:teuthology.parallel:result is None 2024-04-20T17:57:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:32 smithi005 ceph-mon[39562]: purged_snaps scrub starts 2024-04-20T17:57:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:32 smithi005 ceph-mon[39562]: purged_snaps scrub ok 2024-04-20T17:57:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:32 smithi005 ceph-mon[39562]: osdmap e1044: 12 total, 12 up, 12 in 2024-04-20T17:57:33.052 INFO:tasks.cephadm.ceph_manager.ceph:need seq 867583394333 got 867583394337 for osd.2 2024-04-20T17:57:33.052 DEBUG:teuthology.parallel:result is None 2024-04-20T17:57:33.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:32 smithi106 ceph-mon[44423]: purged_snaps scrub starts 2024-04-20T17:57:33.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:32 smithi106 ceph-mon[44423]: purged_snaps scrub ok 2024-04-20T17:57:33.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:32 smithi106 ceph-mon[44423]: osdmap e1044: 12 total, 12 up, 12 in 2024-04-20T17:57:33.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:32 smithi171 ceph-mon[44189]: purged_snaps scrub starts 2024-04-20T17:57:33.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:32 smithi171 ceph-mon[44189]: purged_snaps scrub ok 2024-04-20T17:57:33.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:32 smithi171 ceph-mon[44189]: osdmap e1044: 12 total, 12 up, 12 in 2024-04-20T17:57:33.725 INFO:tasks.cephadm.ceph_manager.ceph:need seq 395136991849 got 395136991855 for osd.11 2024-04-20T17:57:33.725 DEBUG:teuthology.parallel:result is None 2024-04-20T17:57:34.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:33 smithi005 ceph-mon[39562]: pgmap v2777: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:34.154 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:33 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3971473364' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-20T17:57:34.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:33 smithi106 ceph-mon[44423]: pgmap v2777: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:34.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:33 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3971473364' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-20T17:57:34.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:33 smithi171 ceph-mon[44189]: pgmap v2777: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:34.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:33 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3971473364' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-20T17:57:34.366 INFO:teuthology.orchestra.run.smithi005.stdout:313532613244 2024-04-20T17:57:34.903 INFO:teuthology.orchestra.run.smithi005.stdout:360777253495 2024-04-20T17:57:35.110 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:57:35.196 INFO:teuthology.orchestra.run.smithi005.stdout:133143986837 2024-04-20T17:57:35.210 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:34 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3520208651' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-20T17:57:35.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:34 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3520208651' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-20T17:57:35.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:34 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3520208651' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-20T17:57:35.871 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986831 got 133143986837 for osd.4 2024-04-20T17:57:35.871 DEBUG:teuthology.parallel:result is None 2024-04-20T17:57:36.210 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:35 smithi005 ceph-mon[39562]: pgmap v2778: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:36.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:35 smithi106 ceph-mon[44423]: pgmap v2778: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:36.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:35 smithi171 ceph-mon[44189]: pgmap v2778: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:36.395 INFO:tasks.cephadm.ceph_manager.ceph:need seq 360777253487 got 360777253495 for osd.10 2024-04-20T17:57:36.396 DEBUG:teuthology.parallel:result is None 2024-04-20T17:57:37.149 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532613238 got 313532613244 for osd.9 2024-04-20T17:57:37.149 DEBUG:teuthology.parallel:result is None 2024-04-20T17:57:37.539 INFO:teuthology.orchestra.run.smithi005.stdout:2409476653388 2024-04-20T17:57:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:37 smithi005 ceph-mon[39562]: pgmap v2779: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:38.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:37 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/3238229464' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-20T17:57:38.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:37 smithi106 ceph-mon[44423]: pgmap v2779: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:38.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:37 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/3238229464' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-20T17:57:38.330 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2409476653382 got 2409476653388 for osd.3 2024-04-20T17:57:38.331 DEBUG:teuthology.parallel:result is None 2024-04-20T17:57:38.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:37 smithi171 ceph-mon[44189]: pgmap v2779: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:38.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:37 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/3238229464' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-20T17:57:39.522 INFO:teuthology.orchestra.run.smithi005.stdout:2297807503705 2024-04-20T17:57:40.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:39 smithi005 ceph-mon[39562]: 1.5 scrub starts 2024-04-20T17:57:40.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:39 smithi005 ceph-mon[39562]: 1.5 scrub ok 2024-04-20T17:57:40.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:39 smithi005 ceph-mon[39562]: pgmap v2780: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:40.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:39 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/889189857' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-20T17:57:40.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:39 smithi106 ceph-mon[44423]: 1.5 scrub starts 2024-04-20T17:57:40.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:39 smithi106 ceph-mon[44423]: 1.5 scrub ok 2024-04-20T17:57:40.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:39 smithi106 ceph-mon[44423]: pgmap v2780: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:40.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:39 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/889189857' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-20T17:57:40.265 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2297807503699 got 2297807503705 for osd.6 2024-04-20T17:57:40.265 DEBUG:teuthology.parallel:result is None 2024-04-20T17:57:40.265 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-04-20T17:57:40.265 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:57:40.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:39 smithi171 ceph-mon[44189]: 1.5 scrub starts 2024-04-20T17:57:40.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:39 smithi171 ceph-mon[44189]: 1.5 scrub ok 2024-04-20T17:57:40.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:39 smithi171 ceph-mon[44189]: pgmap v2780: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:40.334 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:39 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/889189857' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-20T17:57:40.551 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:57:42.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:41 smithi005 ceph-mon[39562]: pgmap v2781: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:42.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:41 smithi106 ceph-mon[44423]: pgmap v2781: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:42.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:41 smithi171 ceph-mon[44189]: pgmap v2781: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:42.433 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:57:42.435 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:57:43.158 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:42 smithi005 ceph-mon[39562]: 1.6 scrub starts 2024-04-20T17:57:43.158 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:42 smithi005 ceph-mon[39562]: 1.6 scrub ok 2024-04-20T17:57:43.188 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":2781,"stamp":"2024-04-20T17:57:40.804470+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1507,"ondisk_log_size":1507,"up":22,"acting":22,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1124941824,"kb_used":6277788,"kb_used_data":9296,"kb_used_omap":407,"kb_used_meta":6267944,"kb_avail":1118664036,"statfs":{"total":1151940427776,"available":1145511972864,"internally_reserved":0,"allocated":9519104,"data_stored":4945084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":416829,"internal_metadata":6418375619},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001862"},"pg_stats":[{"pgid":"1.6","version":"15'137","reported_seq":1173,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-04-20T17:57:33.516456+0000","last_change":"2024-04-20T17:56:26.416161+0000","last_active":"2024-04-20T17:57:33.516456+0000","last_peered":"2024-04-20T17:57:33.516456+0000","last_clean":"2024-04-20T17:57:33.516456+0000","last_became_active":"2024-04-20T17:45:32.670941+0000","last_became_peered":"2024-04-20T17:45:32.670941+0000","last_unstale":"2024-04-20T17:57:33.516456+0000","last_undegraded":"2024-04-20T17:57:33.516456+0000","last_fullsized":"2024-04-20T17:57:33.516456+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:56:26.416049+0000","last_deep_scrub":"15'137","last_deep_scrub_stamp":"2024-04-20T17:56:26.416049+0000","last_clean_scrub_stamp":"2024-04-20T17:56:26.416049+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:57:39.103097+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"15'137","reported_seq":1177,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-04-20T17:57:32.646344+0000","last_change":"2024-04-20T17:57:11.800492+0000","last_active":"2024-04-20T17:57:32.646344+0000","last_peered":"2024-04-20T17:57:32.646344+0000","last_clean":"2024-04-20T17:57:32.646344+0000","last_became_active":"2024-04-20T17:45:29.817528+0000","last_became_peered":"2024-04-20T17:45:29.817528+0000","last_unstale":"2024-04-20T17:57:32.646344+0000","last_undegraded":"2024-04-20T17:57:32.646344+0000","last_fullsized":"2024-04-20T17:57:32.646344+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:57:11.800366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:57:11.800366+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:58:20.401510+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"15'137","reported_seq":1202,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-04-20T17:57:33.375147+0000","last_change":"2024-04-20T17:57:02.533162+0000","last_active":"2024-04-20T17:57:33.375147+0000","last_peered":"2024-04-20T17:57:33.375147+0000","last_clean":"2024-04-20T17:57:33.375147+0000","last_became_active":"2024-04-20T17:45:29.827577+0000","last_became_peered":"2024-04-20T17:45:29.827577+0000","last_unstale":"2024-04-20T17:57:33.375147+0000","last_undegraded":"2024-04-20T17:57:33.375147+0000","last_fullsized":"2024-04-20T17:57:33.375147+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:57:02.533056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:57:02.533056+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:58:05.714284+0000","scrub_duration":221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"15'137","reported_seq":1181,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-04-20T17:57:38.693833+0000","last_change":"2024-04-20T17:57:38.693860+0000","last_active":"2024-04-20T17:57:38.693833+0000","last_peered":"2024-04-20T17:57:38.693833+0000","last_clean":"2024-04-20T17:57:38.693833+0000","last_became_active":"2024-04-20T17:45:32.666523+0000","last_became_peered":"2024-04-20T17:45:32.666523+0000","last_unstale":"2024-04-20T17:57:38.693833+0000","last_undegraded":"2024-04-20T17:57:38.693833+0000","last_fullsized":"2024-04-20T17:57:38.693833+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:57:38.693737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:57:38.693737+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:58:47.924899+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"15'137","reported_seq":1191,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-04-20T17:57:33.633183+0000","last_change":"2024-04-20T17:57:30.907394+0000","last_active":"2024-04-20T17:57:33.633183+0000","last_peered":"2024-04-20T17:57:33.633183+0000","last_clean":"2024-04-20T17:57:33.633183+0000","last_became_active":"2024-04-20T17:55:05.652626+0000","last_became_peered":"2024-04-20T17:55:05.652626+0000","last_unstale":"2024-04-20T17:57:33.633183+0000","last_undegraded":"2024-04-20T17:57:33.633183+0000","last_fullsized":"2024-04-20T17:57:33.633183+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":1000,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:57:30.907243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:57:30.907243+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:58:32.719294+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"15'137","reported_seq":964,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-04-20T17:57:31.934491+0000","last_change":"2024-04-20T17:56:52.897948+0000","last_active":"2024-04-20T17:57:31.934491+0000","last_peered":"2024-04-20T17:57:31.934491+0000","last_clean":"2024-04-20T17:57:31.934491+0000","last_became_active":"2024-04-20T17:49:46.824559+0000","last_became_peered":"2024-04-20T17:49:46.824559+0000","last_unstale":"2024-04-20T17:57:31.934491+0000","last_undegraded":"2024-04-20T17:57:31.934491+0000","last_fullsized":"2024-04-20T17:57:31.934491+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:56:52.897769+0000","last_deep_scrub":"15'137","last_deep_scrub_stamp":"2024-04-20T17:56:52.897769+0000","last_clean_scrub_stamp":"2024-04-20T17:56:52.897769+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:57:52.915321+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"15'137","reported_seq":1186,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-04-20T17:57:33.516471+0000","last_change":"2024-04-20T17:57:10.651998+0000","last_active":"2024-04-20T17:57:33.516471+0000","last_peered":"2024-04-20T17:57:33.516471+0000","last_clean":"2024-04-20T17:57:33.516471+0000","last_became_active":"2024-04-20T17:52:45.966123+0000","last_became_peered":"2024-04-20T17:52:45.966123+0000","last_unstale":"2024-04-20T17:57:33.516471+0000","last_undegraded":"2024-04-20T17:57:33.516471+0000","last_fullsized":"2024-04-20T17:57:33.516471+0000","mapping_epoch":969,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":970,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:57:10.651884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:57:10.651884+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:58:29.855898+0000","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.a","version":"15'137","reported_seq":1158,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-04-20T17:57:31.937791+0000","last_change":"2024-04-20T17:56:40.253862+0000","last_active":"2024-04-20T17:57:31.937791+0000","last_peered":"2024-04-20T17:57:31.937791+0000","last_clean":"2024-04-20T17:57:31.937791+0000","last_became_active":"2024-04-20T17:45:32.669451+0000","last_became_peered":"2024-04-20T17:45:32.669451+0000","last_unstale":"2024-04-20T17:57:31.937791+0000","last_undegraded":"2024-04-20T17:57:31.937791+0000","last_fullsized":"2024-04-20T17:57:31.937791+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:56:40.253735+0000","last_deep_scrub":"15'137","last_deep_scrub_stamp":"2024-04-20T17:56:40.253735+0000","last_clean_scrub_stamp":"2024-04-20T17:56:40.253735+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:57:58.014864+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"15'137","reported_seq":747,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-04-20T17:57:33.633234+0000","last_change":"2024-04-20T17:56:41.323728+0000","last_active":"2024-04-20T17:57:33.633234+0000","last_peered":"2024-04-20T17:57:33.633234+0000","last_clean":"2024-04-20T17:57:33.633234+0000","last_became_active":"2024-04-20T17:55:26.959760+0000","last_became_peered":"2024-04-20T17:55:26.959760+0000","last_unstale":"2024-04-20T17:57:33.633234+0000","last_undegraded":"2024-04-20T17:57:33.633234+0000","last_fullsized":"2024-04-20T17:57:33.633234+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:56:41.323593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:56:41.323593+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:58:02.296736+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"15'137","reported_seq":1122,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-04-20T17:57:33.633270+0000","last_change":"2024-04-20T17:56:50.097507+0000","last_active":"2024-04-20T17:57:33.633270+0000","last_peered":"2024-04-20T17:57:33.633270+0000","last_clean":"2024-04-20T17:57:33.633270+0000","last_became_active":"2024-04-20T17:46:40.881713+0000","last_became_peered":"2024-04-20T17:46:40.881713+0000","last_unstale":"2024-04-20T17:57:33.633270+0000","last_undegraded":"2024-04-20T17:57:33.633270+0000","last_fullsized":"2024-04-20T17:57:33.633270+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:56:50.097384+0000","last_deep_scrub":"15'137","last_deep_scrub_stamp":"2024-04-20T17:55:41.921529+0000","last_clean_scrub_stamp":"2024-04-20T17:56:50.097384+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:58:04.763791+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1120,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-04-20T17:57:33.633175+0000","last_change":"2024-04-20T17:56:54.089973+0000","last_active":"2024-04-20T17:57:33.633175+0000","last_peered":"2024-04-20T17:57:33.633175+0000","last_clean":"2024-04-20T17:57:33.633175+0000","last_became_active":"2024-04-20T17:46:40.851683+0000","last_became_peered":"2024-04-20T17:46:40.851683+0000","last_unstale":"2024-04-20T17:57:33.633175+0000","last_undegraded":"2024-04-20T17:57:33.633175+0000","last_fullsized":"2024-04-20T17:57:33.633175+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:56:54.089836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:56:54.089836+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:58:11.566887+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},"log_size":1507,"ondisk_log_size":1507,"up":22,"acting":22,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991857,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":952636,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":951614,"kb_avail":92792516,"statfs":{"total":95995035648,"available":95019536384,"internally_reserved":0,"allocated":1032192,"data_stored":651615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":974453188},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253497,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":852344,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92892808,"statfs":{"total":95995035648,"available":95122235392,"internally_reserved":0,"allocated":704512,"data_stored":323935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":872085952},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613245,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":749628,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":748926,"kb_avail":92995524,"statfs":{"total":95995035648,"available":95227416576,"internally_reserved":0,"allocated":704512,"data_stored":323935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":766900672},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809476,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":473660,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93271492,"statfs":{"total":95995035648,"available":95510007808,"internally_reserved":0,"allocated":1032192,"data_stored":651615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":799,"seq":3431678869666,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":171388,"kb_used_data":688,"kb_used_omap":11,"kb_used_meta":170676,"kb_avail":93573764,"statfs":{"total":95995035648,"available":95819534336,"internally_reserved":0,"allocated":704512,"data_stored":323943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11990,"internal_metadata":174772522},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":535,"seq":2297807503705,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":60476,"kb_used_data":688,"kb_used_omap":170,"kb_used_meta":59605,"kb_avail":93684676,"statfs":{"total":95995035648,"available":95933108224,"internally_reserved":0,"allocated":704512,"data_stored":323935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":174628,"internal_metadata":61035996},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":659,"seq":2830383448328,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":392636,"kb_used_data":688,"kb_used_omap":4,"kb_used_meta":391931,"kb_avail":93352516,"statfs":{"total":95995035648,"available":95592976384,"internally_reserved":0,"allocated":704512,"data_stored":323935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4410,"internal_metadata":401338054},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":995,"seq":4273492459555,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":38076,"kb_used_data":688,"kb_used_omap":4,"kb_used_meta":37371,"kb_avail":93707076,"statfs":{"total":95995035648,"available":95956045824,"internally_reserved":0,"allocated":704512,"data_stored":323935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4410,"internal_metadata":38268614},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394339,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":438020,"kb_used_data":888,"kb_used_omap":202,"kb_used_meta":436917,"kb_avail":93307132,"statfs":{"total":95995035648,"available":95546503168,"internally_reserved":0,"allocated":909312,"data_stored":525183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":447403479},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653389,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":623100,"kb_used_data":688,"kb_used_omap":4,"kb_used_meta":622395,"kb_avail":93122052,"statfs":{"total":95995035648,"available":95356981248,"internally_reserved":0,"allocated":704512,"data_stored":323935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":637333191},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986838,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":623292,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93121860,"statfs":{"total":95995035648,"available":95356784640,"internally_reserved":0,"allocated":704512,"data_stored":323935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790608,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":902532,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":92842620,"statfs":{"total":95995035648,"available":95070842880,"internally_reserved":0,"allocated":909312,"data_stored":525183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":923269573},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:57:43.190 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-20T17:57:43.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:42 smithi106 ceph-mon[44423]: 1.6 scrub starts 2024-04-20T17:57:43.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:42 smithi106 ceph-mon[44423]: 1.6 scrub ok 2024-04-20T17:57:43.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:42 smithi171 ceph-mon[44189]: 1.6 scrub starts 2024-04-20T17:57:43.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:42 smithi171 ceph-mon[44189]: 1.6 scrub ok 2024-04-20T17:57:43.473 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:57:44.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:43 smithi005 ceph-mon[39562]: from='client.18540 v1:172.21.15.5:0/3155037588' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:57:44.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:43 smithi005 ceph-mon[39562]: pgmap v2782: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:44.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:43 smithi106 ceph-mon[44423]: from='client.18540 v1:172.21.15.5:0/3155037588' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:57:44.261 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:43 smithi106 ceph-mon[44423]: pgmap v2782: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:44.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:43 smithi171 ceph-mon[44189]: from='client.18540 v1:172.21.15.5:0/3155037588' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:57:44.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:43 smithi171 ceph-mon[44189]: pgmap v2782: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:45.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:44 smithi005 ceph-mon[39562]: pgmap v2783: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:45.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:44 smithi106 ceph-mon[44423]: pgmap v2783: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:45.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:44 smithi171 ceph-mon[44189]: pgmap v2783: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:45.623 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-04-20T17:57:45.625 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-04-20T17:57:46.245 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:45 smithi005 ceph-mon[39562]: from='client.18546 v1:172.21.15.5:0/3382230891' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:57:46.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:45 smithi106 ceph-mon[44423]: from='client.18546 v1:172.21.15.5:0/3382230891' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:57:46.276 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":2783,"stamp":"2024-04-20T17:57:44.805295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1507,"ondisk_log_size":1507,"up":22,"acting":22,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1124941824,"kb_used":6277788,"kb_used_data":9296,"kb_used_omap":407,"kb_used_meta":6267944,"kb_avail":1118664036,"statfs":{"total":1151940427776,"available":1145511972864,"internally_reserved":0,"allocated":9519104,"data_stored":4945084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":416829,"internal_metadata":6418375619},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002177"},"pg_stats":[{"pgid":"1.6","version":"15'137","reported_seq":1173,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-04-20T17:57:33.516456+0000","last_change":"2024-04-20T17:56:26.416161+0000","last_active":"2024-04-20T17:57:33.516456+0000","last_peered":"2024-04-20T17:57:33.516456+0000","last_clean":"2024-04-20T17:57:33.516456+0000","last_became_active":"2024-04-20T17:45:32.670941+0000","last_became_peered":"2024-04-20T17:45:32.670941+0000","last_unstale":"2024-04-20T17:57:33.516456+0000","last_undegraded":"2024-04-20T17:57:33.516456+0000","last_fullsized":"2024-04-20T17:57:33.516456+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:56:26.416049+0000","last_deep_scrub":"15'137","last_deep_scrub_stamp":"2024-04-20T17:56:26.416049+0000","last_clean_scrub_stamp":"2024-04-20T17:56:26.416049+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:57:39.103097+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"15'137","reported_seq":1177,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-04-20T17:57:32.646344+0000","last_change":"2024-04-20T17:57:11.800492+0000","last_active":"2024-04-20T17:57:32.646344+0000","last_peered":"2024-04-20T17:57:32.646344+0000","last_clean":"2024-04-20T17:57:32.646344+0000","last_became_active":"2024-04-20T17:45:29.817528+0000","last_became_peered":"2024-04-20T17:45:29.817528+0000","last_unstale":"2024-04-20T17:57:32.646344+0000","last_undegraded":"2024-04-20T17:57:32.646344+0000","last_fullsized":"2024-04-20T17:57:32.646344+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:57:11.800366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:57:11.800366+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:58:20.401510+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"15'137","reported_seq":1202,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-04-20T17:57:33.375147+0000","last_change":"2024-04-20T17:57:02.533162+0000","last_active":"2024-04-20T17:57:33.375147+0000","last_peered":"2024-04-20T17:57:33.375147+0000","last_clean":"2024-04-20T17:57:33.375147+0000","last_became_active":"2024-04-20T17:45:29.827577+0000","last_became_peered":"2024-04-20T17:45:29.827577+0000","last_unstale":"2024-04-20T17:57:33.375147+0000","last_undegraded":"2024-04-20T17:57:33.375147+0000","last_fullsized":"2024-04-20T17:57:33.375147+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:57:02.533056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:57:02.533056+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:58:05.714284+0000","scrub_duration":221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"15'137","reported_seq":1181,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-04-20T17:57:38.693833+0000","last_change":"2024-04-20T17:57:38.693860+0000","last_active":"2024-04-20T17:57:38.693833+0000","last_peered":"2024-04-20T17:57:38.693833+0000","last_clean":"2024-04-20T17:57:38.693833+0000","last_became_active":"2024-04-20T17:45:32.666523+0000","last_became_peered":"2024-04-20T17:45:32.666523+0000","last_unstale":"2024-04-20T17:57:38.693833+0000","last_undegraded":"2024-04-20T17:57:38.693833+0000","last_fullsized":"2024-04-20T17:57:38.693833+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:57:38.693737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:57:38.693737+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:58:47.924899+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"15'137","reported_seq":1191,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-04-20T17:57:33.633183+0000","last_change":"2024-04-20T17:57:30.907394+0000","last_active":"2024-04-20T17:57:33.633183+0000","last_peered":"2024-04-20T17:57:33.633183+0000","last_clean":"2024-04-20T17:57:33.633183+0000","last_became_active":"2024-04-20T17:55:05.652626+0000","last_became_peered":"2024-04-20T17:55:05.652626+0000","last_unstale":"2024-04-20T17:57:33.633183+0000","last_undegraded":"2024-04-20T17:57:33.633183+0000","last_fullsized":"2024-04-20T17:57:33.633183+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":1000,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:57:30.907243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:57:30.907243+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:58:32.719294+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"15'137","reported_seq":964,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-04-20T17:57:31.934491+0000","last_change":"2024-04-20T17:56:52.897948+0000","last_active":"2024-04-20T17:57:31.934491+0000","last_peered":"2024-04-20T17:57:31.934491+0000","last_clean":"2024-04-20T17:57:31.934491+0000","last_became_active":"2024-04-20T17:49:46.824559+0000","last_became_peered":"2024-04-20T17:49:46.824559+0000","last_unstale":"2024-04-20T17:57:31.934491+0000","last_undegraded":"2024-04-20T17:57:31.934491+0000","last_fullsized":"2024-04-20T17:57:31.934491+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":3,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:56:52.897769+0000","last_deep_scrub":"15'137","last_deep_scrub_stamp":"2024-04-20T17:56:52.897769+0000","last_clean_scrub_stamp":"2024-04-20T17:56:52.897769+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:57:52.915321+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"15'137","reported_seq":1186,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-04-20T17:57:33.516471+0000","last_change":"2024-04-20T17:57:10.651998+0000","last_active":"2024-04-20T17:57:33.516471+0000","last_peered":"2024-04-20T17:57:33.516471+0000","last_clean":"2024-04-20T17:57:33.516471+0000","last_became_active":"2024-04-20T17:52:45.966123+0000","last_became_peered":"2024-04-20T17:52:45.966123+0000","last_unstale":"2024-04-20T17:57:33.516471+0000","last_undegraded":"2024-04-20T17:57:33.516471+0000","last_fullsized":"2024-04-20T17:57:33.516471+0000","mapping_epoch":969,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":970,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:57:10.651884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:57:10.651884+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:58:29.855898+0000","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.a","version":"15'137","reported_seq":1158,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-04-20T17:57:31.937791+0000","last_change":"2024-04-20T17:56:40.253862+0000","last_active":"2024-04-20T17:57:31.937791+0000","last_peered":"2024-04-20T17:57:31.937791+0000","last_clean":"2024-04-20T17:57:31.937791+0000","last_became_active":"2024-04-20T17:45:32.669451+0000","last_became_peered":"2024-04-20T17:45:32.669451+0000","last_unstale":"2024-04-20T17:57:31.937791+0000","last_undegraded":"2024-04-20T17:57:31.937791+0000","last_fullsized":"2024-04-20T17:57:31.937791+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:56:40.253735+0000","last_deep_scrub":"15'137","last_deep_scrub_stamp":"2024-04-20T17:56:40.253735+0000","last_clean_scrub_stamp":"2024-04-20T17:56:40.253735+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:57:58.014864+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"15'137","reported_seq":747,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-04-20T17:57:33.633234+0000","last_change":"2024-04-20T17:56:41.323728+0000","last_active":"2024-04-20T17:57:33.633234+0000","last_peered":"2024-04-20T17:57:33.633234+0000","last_clean":"2024-04-20T17:57:33.633234+0000","last_became_active":"2024-04-20T17:55:26.959760+0000","last_became_peered":"2024-04-20T17:55:26.959760+0000","last_unstale":"2024-04-20T17:57:33.633234+0000","last_undegraded":"2024-04-20T17:57:33.633234+0000","last_fullsized":"2024-04-20T17:57:33.633234+0000","mapping_epoch":1007,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":1008,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:56:41.323593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:56:41.323593+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:58:02.296736+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"15'137","reported_seq":1122,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-04-20T17:57:33.633270+0000","last_change":"2024-04-20T17:56:50.097507+0000","last_active":"2024-04-20T17:57:33.633270+0000","last_peered":"2024-04-20T17:57:33.633270+0000","last_clean":"2024-04-20T17:57:33.633270+0000","last_became_active":"2024-04-20T17:46:40.881713+0000","last_became_peered":"2024-04-20T17:46:40.881713+0000","last_unstale":"2024-04-20T17:57:33.633270+0000","last_undegraded":"2024-04-20T17:57:33.633270+0000","last_fullsized":"2024-04-20T17:57:33.633270+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":818,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":4,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:56:50.097384+0000","last_deep_scrub":"15'137","last_deep_scrub_stamp":"2024-04-20T17:55:41.921529+0000","last_clean_scrub_stamp":"2024-04-20T17:56:50.097384+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:58:04.763791+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9],"acting":[7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'137","reported_seq":1120,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-04-20T17:57:33.633175+0000","last_change":"2024-04-20T17:56:54.089973+0000","last_active":"2024-04-20T17:57:33.633175+0000","last_peered":"2024-04-20T17:57:33.633175+0000","last_clean":"2024-04-20T17:57:33.633175+0000","last_became_active":"2024-04-20T17:46:40.851683+0000","last_became_peered":"2024-04-20T17:46:40.851683+0000","last_unstale":"2024-04-20T17:57:33.633175+0000","last_undegraded":"2024-04-20T17:57:33.633175+0000","last_fullsized":"2024-04-20T17:57:33.633175+0000","mapping_epoch":846,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":847,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'137","last_scrub_stamp":"2024-04-20T17:56:54.089836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T17:01:36.651409+0000","last_clean_scrub_stamp":"2024-04-20T17:56:54.089836+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T17:58:11.566887+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},"log_size":1507,"ondisk_log_size":1507,"up":22,"acting":22,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":92,"seq":395136991858,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":952636,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":951614,"kb_avail":92792516,"statfs":{"total":95995035648,"available":95019536384,"internally_reserved":0,"allocated":1032192,"data_stored":651615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":974453188},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":84,"seq":360777253497,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":852344,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92892808,"statfs":{"total":95995035648,"available":95122235392,"internally_reserved":0,"allocated":704512,"data_stored":323935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":872085952},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":73,"seq":313532613246,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":749628,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":748926,"kb_avail":92995524,"statfs":{"total":95995035648,"available":95227416576,"internally_reserved":0,"allocated":704512,"data_stored":323935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":766900672},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":67,"seq":287762809477,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":473660,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93271492,"statfs":{"total":95995035648,"available":95510007808,"internally_reserved":0,"allocated":1032192,"data_stored":651615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":799,"seq":3431678869666,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":171388,"kb_used_data":688,"kb_used_omap":11,"kb_used_meta":170676,"kb_avail":93573764,"statfs":{"total":95995035648,"available":95819534336,"internally_reserved":0,"allocated":704512,"data_stored":323943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11990,"internal_metadata":174772522},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":535,"seq":2297807503706,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":60476,"kb_used_data":688,"kb_used_omap":170,"kb_used_meta":59605,"kb_avail":93684676,"statfs":{"total":95995035648,"available":95933108224,"internally_reserved":0,"allocated":704512,"data_stored":323935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":174628,"internal_metadata":61035996},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":659,"seq":2830383448329,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":392636,"kb_used_data":688,"kb_used_omap":4,"kb_used_meta":391931,"kb_avail":93352516,"statfs":{"total":95995035648,"available":95592976384,"internally_reserved":0,"allocated":704512,"data_stored":323935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4410,"internal_metadata":401338054},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":995,"seq":4273492459555,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":38076,"kb_used_data":688,"kb_used_omap":4,"kb_used_meta":37371,"kb_avail":93707076,"statfs":{"total":95995035648,"available":95956045824,"internally_reserved":0,"allocated":704512,"data_stored":323935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4410,"internal_metadata":38268614},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":202,"seq":867583394340,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":438020,"kb_used_data":888,"kb_used_omap":202,"kb_used_meta":436917,"kb_avail":93307132,"statfs":{"total":95995035648,"available":95546503168,"internally_reserved":0,"allocated":909312,"data_stored":525183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":207401,"internal_metadata":447403479},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":561,"seq":2409476653390,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":623100,"kb_used_data":688,"kb_used_omap":4,"kb_used_meta":622395,"kb_avail":93122052,"statfs":{"total":95995035648,"available":95356981248,"internally_reserved":0,"allocated":704512,"data_stored":323935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4409,"internal_metadata":637333191},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986839,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":623292,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93121860,"statfs":{"total":95995035648,"available":95356784640,"internally_reserved":0,"allocated":704512,"data_stored":323935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790608,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":902532,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":92842620,"statfs":{"total":95995035648,"available":95070842880,"internally_reserved":0,"allocated":909312,"data_stored":525183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":923269573},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7579,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T17:57:46.277 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-04-20T17:57:46.277 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:36c371567dddacf6207ea36f2535396ab31415fc shell --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-04-20T17:57:46.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:45 smithi171 ceph-mon[44189]: from='client.18546 v1:172.21.15.5:0/3382230891' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-20T17:57:46.558 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/mon.a/config 2024-04-20T17:57:47.058 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:46 smithi005 ceph-mon[39562]: pgmap v2784: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:47.260 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:46 smithi106 ceph-mon[44423]: pgmap v2784: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:47.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:46 smithi171 ceph-mon[44189]: pgmap v2784: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:48.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:48 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2227286868' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-20T17:57:48.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:48 smithi005 ceph-mon[39562]: from='client.? v1:172.21.15.5:0/2227286868' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-20T17:57:48.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:48 smithi005 ceph-mon[39562]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:57:49.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:48 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2227286868' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-20T17:57:49.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:48 smithi106 ceph-mon[44423]: from='client.? v1:172.21.15.5:0/2227286868' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-20T17:57:49.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:48 smithi106 ceph-mon[44423]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:57:49.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:48 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2227286868' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-20T17:57:49.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:48 smithi171 ceph-mon[44189]: from='client.? v1:172.21.15.5:0/2227286868' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-20T17:57:49.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:48 smithi171 ceph-mon[44189]: from='mgr.14148 v1:172.21.15.5:0/2850149640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-20T17:57:49.403 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-20T17:57:49.413 INFO:tasks.cephadm:Teardown begin 2024-04-20T17:57:49.414 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-20T17:57:49.451 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-20T17:57:49.527 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-20T17:57:49.620 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-20T17:57:49.620 DEBUG:teuthology.orchestra.run.smithi005:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-20T17:57:49.655 DEBUG:teuthology.orchestra.run.smithi106:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-20T17:57:49.708 DEBUG:teuthology.orchestra.run.smithi171:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-20T17:57:49.736 INFO:tasks.cephadm:Stopping all daemons... 2024-04-20T17:57:49.736 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-20T17:57:49.737 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@mon.a 2024-04-20T17:57:49.760 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:49 smithi106 ceph-mon[44423]: pgmap v2785: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:49.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:49 smithi005 ceph-mon[39562]: pgmap v2785: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:49.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:49 smithi005 systemd[1]: Stopping Ceph mon.a for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:57:50.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:49 smithi171 ceph-mon[44189]: pgmap v2785: 11 pgs: 11 active+clean; 577 KiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-20T17:57:50.332 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:50 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mon-a[39539]: 2024-04-20T17:57:50.039+0000 7f73beaac700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T17:57:50.332 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:50 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mon-a[39539]: 2024-04-20T17:57:50.039+0000 7f73beaac700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-20T17:57:50.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Apr 20 17:57:50 smithi005 bash[390289]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mon-a 2024-04-20T17:57:50.835 DEBUG:teuthology.orchestra.run.smithi005:> sudo pkill -f 'journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@mon.a.service' 2024-04-20T17:57:50.875 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T17:57:50.876 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-20T17:57:50.876 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-20T17:57:50.876 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@mon.b 2024-04-20T17:57:51.162 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:50 smithi106 systemd[1]: Stopping Ceph mon.b for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:57:51.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:51 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mon-b[44400]: 2024-04-20T17:57:51.161+0000 7f94ff25d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T17:57:51.510 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:51 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mon-b[44400]: 2024-04-20T17:57:51.161+0000 7f94ff25d700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-20T17:57:52.010 INFO:journalctl@ceph.mon.b.smithi106.stdout:Apr 20 17:57:51 smithi106 bash[207117]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mon-b 2024-04-20T17:57:52.258 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@mon.b.service' 2024-04-20T17:57:52.299 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T17:57:52.299 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-20T17:57:52.299 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-20T17:57:52.299 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@mon.c 2024-04-20T17:57:52.583 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:52 smithi171 systemd[1]: Stopping Ceph mon.c for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:57:53.006 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:52 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mon-c[44138]: 2024-04-20T17:57:52.612+0000 7f8c05d69700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T17:57:53.006 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:52 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mon-c[44138]: 2024-04-20T17:57:52.612+0000 7f8c05d69700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-20T17:57:53.333 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 20 17:57:53 smithi171 bash[201972]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mon-c 2024-04-20T17:57:53.654 DEBUG:teuthology.orchestra.run.smithi171:> sudo pkill -f 'journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@mon.c.service' 2024-04-20T17:57:53.703 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T17:57:53.703 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-20T17:57:53.703 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-20T17:57:53.703 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@mgr.y 2024-04-20T17:57:53.986 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 17:57:53 smithi005 systemd[1]: Stopping Ceph mgr.y for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:57:54.288 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 17:57:53 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y[39811]: 2024-04-20T17:57:53.985+0000 7f0d4f6f5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T17:57:54.710 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Apr 20 17:57:54 smithi005 bash[390470]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-y 2024-04-20T17:57:54.798 DEBUG:teuthology.orchestra.run.smithi005:> sudo pkill -f 'journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@mgr.y.service' 2024-04-20T17:57:54.834 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T17:57:54.835 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-20T17:57:54.835 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-20T17:57:54.835 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@mgr.x 2024-04-20T17:57:55.261 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:57:54 smithi106 systemd[1]: Stopping Ceph mgr.x for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:57:55.760 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:57:55 smithi106 bash[207279]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-mgr-x 2024-04-20T17:57:55.943 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:57:55 smithi106 systemd[1]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-04-20T17:57:55.943 INFO:journalctl@ceph.mgr.x.smithi106.stdout:Apr 20 17:57:55 smithi106 systemd[1]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-04-20T17:57:55.945 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@mgr.x.service' 2024-04-20T17:57:55.980 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T17:57:55.980 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-20T17:57:55.980 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-20T17:57:55.980 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.0 2024-04-20T17:57:56.265 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:57:56 smithi005 systemd[1]: Stopping Ceph osd.0 for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:57:56.711 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:57:56 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-0[376574]: 2024-04-20T17:57:56.264+0000 7efc2e0cd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T17:57:56.711 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:57:56 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-0[376574]: 2024-04-20T17:57:56.265+0000 7efc2e0cd700 -1 osd.0 1044 *** Got signal Terminated *** 2024-04-20T17:57:56.711 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:57:56 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-0[376574]: 2024-04-20T17:57:56.265+0000 7efc2e0cd700 -1 osd.0 1044 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T17:58:02.461 INFO:journalctl@ceph.osd.0.smithi005.stdout:Apr 20 17:58:02 smithi005 bash[390615]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-0 2024-04-20T17:58:03.954 DEBUG:teuthology.orchestra.run.smithi005:> sudo pkill -f 'journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.0.service' 2024-04-20T17:58:03.999 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T17:58:03.999 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-20T17:58:04.000 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-20T17:58:04.000 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.1 2024-04-20T17:58:04.214 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:58:04 smithi005 systemd[1]: Stopping Ceph osd.1 for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:58:04.711 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:58:04 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-1[272232]: 2024-04-20T17:58:04.263+0000 7fdfbdc1f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T17:58:04.711 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:58:04 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-1[272232]: 2024-04-20T17:58:04.263+0000 7fdfbdc1f700 -1 osd.1 1044 *** Got signal Terminated *** 2024-04-20T17:58:04.711 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:58:04 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-1[272232]: 2024-04-20T17:58:04.263+0000 7fdfbdc1f700 -1 osd.1 1044 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T17:58:10.211 INFO:journalctl@ceph.osd.1.smithi005.stdout:Apr 20 17:58:10 smithi005 bash[390884]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-1 2024-04-20T17:58:11.931 DEBUG:teuthology.orchestra.run.smithi005:> sudo pkill -f 'journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.1.service' 2024-04-20T17:58:11.975 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T17:58:11.975 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-20T17:58:11.975 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-20T17:58:11.976 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.2 2024-04-20T17:58:12.211 INFO:journalctl@ceph.osd.2.smithi005.stdout:Apr 20 17:58:12 smithi005 systemd[1]: Stopping Ceph osd.2 for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:58:12.711 INFO:journalctl@ceph.osd.2.smithi005.stdout:Apr 20 17:58:12 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-2[119106]: 2024-04-20T17:58:12.321+0000 7faa4c1a5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T17:58:12.711 INFO:journalctl@ceph.osd.2.smithi005.stdout:Apr 20 17:58:12 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-2[119106]: 2024-04-20T17:58:12.321+0000 7faa4c1a5700 -1 osd.2 1044 *** Got signal Terminated *** 2024-04-20T17:58:12.711 INFO:journalctl@ceph.osd.2.smithi005.stdout:Apr 20 17:58:12 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-2[119106]: 2024-04-20T17:58:12.321+0000 7faa4c1a5700 -1 osd.2 1044 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T17:58:18.461 INFO:journalctl@ceph.osd.2.smithi005.stdout:Apr 20 17:58:18 smithi005 bash[391151]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-2 2024-04-20T17:58:20.009 DEBUG:teuthology.orchestra.run.smithi005:> sudo pkill -f 'journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.2.service' 2024-04-20T17:58:20.054 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T17:58:20.055 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-20T17:58:20.055 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-20T17:58:20.055 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.3 2024-04-20T17:58:20.366 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:58:20 smithi005 systemd[1]: Stopping Ceph osd.3 for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:58:20.711 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:58:20 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-3[233833]: 2024-04-20T17:58:20.366+0000 7fee1224f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T17:58:20.711 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:58:20 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-3[233833]: 2024-04-20T17:58:20.366+0000 7fee1224f700 -1 osd.3 1044 *** Got signal Terminated *** 2024-04-20T17:58:20.711 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:58:20 smithi005 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-3[233833]: 2024-04-20T17:58:20.366+0000 7fee1224f700 -1 osd.3 1044 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T17:58:23.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:23 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:23.190+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:24.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:24 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:24.192+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:25.333 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:25 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:25.208+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:25.333 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:25 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:25.034+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:26.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:25 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:25.642+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:26.333 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:26 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:26.078+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:26.333 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:26 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:26.240+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:26.457 INFO:journalctl@ceph.osd.3.smithi005.stdout:Apr 20 17:58:26 smithi005 bash[391426]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-3 2024-04-20T17:58:27.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:26 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:26.671+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:27.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:27 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:27.249+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:27.583 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:27 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:27.089+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:27.993 DEBUG:teuthology.orchestra.run.smithi005:> sudo pkill -f 'journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.3.service' 2024-04-20T17:58:28.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:27 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:27.664+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:28.035 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T17:58:28.035 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-20T17:58:28.035 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-20T17:58:28.035 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.4 2024-04-20T17:58:28.333 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:28 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:28.259+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:28.333 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:28 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:28.060+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:28.510 INFO:journalctl@ceph.osd.4.smithi106.stdout:Apr 20 17:58:28 smithi106 systemd[1]: Stopping Ceph osd.4 for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:58:28.511 INFO:journalctl@ceph.osd.4.smithi106.stdout:Apr 20 17:58:28 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-4[49872]: 2024-04-20T17:58:28.316+0000 7fb31abe9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T17:58:28.511 INFO:journalctl@ceph.osd.4.smithi106.stdout:Apr 20 17:58:28 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-4[49872]: 2024-04-20T17:58:28.316+0000 7fb31abe9700 -1 osd.4 1044 *** Got signal Terminated *** 2024-04-20T17:58:28.511 INFO:journalctl@ceph.osd.4.smithi106.stdout:Apr 20 17:58:28 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-4[49872]: 2024-04-20T17:58:28.316+0000 7fb31abe9700 -1 osd.4 1044 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T17:58:29.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:28 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:28.625+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:29.583 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:29 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:29.091+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:29.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:29 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:29.301+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:30.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:29 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:29.602+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:30.583 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:30 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:30.138+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:30.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:30 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:30.268+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:30.640 INFO:journalctl@ceph.osd.5.smithi106.stdout:Apr 20 17:58:30 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-5[57661]: 2024-04-20T17:58:30.272+0000 7fbd03cce700 -1 osd.5 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:04.935289+0000 front 2024-04-20T17:58:04.935448+0000 (oldest deadline 2024-04-20T17:58:30.235292+0000) 2024-04-20T17:58:31.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:30 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:30.640+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:31.431 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:31 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:31.153+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:31.431 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:31 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:31.289+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:31.510 INFO:journalctl@ceph.osd.5.smithi106.stdout:Apr 20 17:58:31 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-5[57661]: 2024-04-20T17:58:31.242+0000 7fbd03cce700 -1 osd.5 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:04.935289+0000 front 2024-04-20T17:58:04.935448+0000 (oldest deadline 2024-04-20T17:58:30.235292+0000) 2024-04-20T17:58:31.833 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:31 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:31.432+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:32.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:31 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:31.640+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:32.010 INFO:journalctl@ceph.osd.4.smithi106.stdout:Apr 20 17:58:31 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-4[49872]: 2024-04-20T17:58:31.755+0000 7fb3171dd700 -1 osd.4 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.545927+0000 front 2024-04-20T17:58:07.545862+0000 (oldest deadline 2024-04-20T17:58:31.045889+0000) 2024-04-20T17:58:32.145 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:31 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:31.843+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:32.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:32 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:32.261+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:32.583 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:32 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:32.146+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:32.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:32 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:32.390+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:32.592 INFO:journalctl@ceph.osd.5.smithi106.stdout:Apr 20 17:58:32 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-5[57661]: 2024-04-20T17:58:32.288+0000 7fbd03cce700 -1 osd.5 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:04.935289+0000 front 2024-04-20T17:58:04.935448+0000 (oldest deadline 2024-04-20T17:58:30.235292+0000) 2024-04-20T17:58:33.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:32 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:32.593+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:33.010 INFO:journalctl@ceph.osd.4.smithi106.stdout:Apr 20 17:58:32 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-4[49872]: 2024-04-20T17:58:32.755+0000 7fb3171dd700 -1 osd.4 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.545927+0000 front 2024-04-20T17:58:07.545862+0000 (oldest deadline 2024-04-20T17:58:31.045889+0000) 2024-04-20T17:58:33.177 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:32 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:32.871+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:33.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:33 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:33.227+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:33.583 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:33 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:33.178+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:33.583 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:33 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:33.389+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:33.598 INFO:journalctl@ceph.osd.5.smithi106.stdout:Apr 20 17:58:33 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-5[57661]: 2024-04-20T17:58:33.276+0000 7fbd03cce700 -1 osd.5 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:04.935289+0000 front 2024-04-20T17:58:04.935448+0000 (oldest deadline 2024-04-20T17:58:30.235292+0000) 2024-04-20T17:58:34.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:33 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:33.598+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:34.159 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:33 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:33.861+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:34.384 INFO:journalctl@ceph.osd.5.smithi106.stdout:Apr 20 17:58:34 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-5[57661]: 2024-04-20T17:58:34.254+0000 7fbd03cce700 -1 osd.5 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:04.935289+0000 front 2024-04-20T17:58:04.935448+0000 (oldest deadline 2024-04-20T17:58:30.235292+0000) 2024-04-20T17:58:34.385 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:34 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:34.018+0000 7f4b0bcb2700 -1 osd.6 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.854093+0000 front 2024-04-20T17:58:07.854284+0000 (oldest deadline 2024-04-20T17:58:33.754032+0000) 2024-04-20T17:58:34.413 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:34 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:34.160+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:34.414 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:34 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:34.226+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:34.760 INFO:journalctl@ceph.osd.4.smithi106.stdout:Apr 20 17:58:34 smithi106 bash[207441]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-4 2024-04-20T17:58:34.760 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:34 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:34.578+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:34.833 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:34 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:34.414+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:35.128 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:34 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:34.861+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:35.310 INFO:journalctl@ceph.osd.5.smithi106.stdout:Apr 20 17:58:35 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-5[57661]: 2024-04-20T17:58:35.297+0000 7fbd03cce700 -1 osd.5 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:04.935289+0000 front 2024-04-20T17:58:04.935448+0000 (oldest deadline 2024-04-20T17:58:30.235292+0000) 2024-04-20T17:58:35.311 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:35 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:35.056+0000 7f4b0bcb2700 -1 osd.6 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.854093+0000 front 2024-04-20T17:58:07.854284+0000 (oldest deadline 2024-04-20T17:58:33.754032+0000) 2024-04-20T17:58:35.389 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:35 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:35.128+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:35.389 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:35 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:35.180+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:35.814 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:35 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:35.390+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:36.009 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:35 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:35.532+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:36.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:35 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:35.815+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:36.260 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:36 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:36.085+0000 7f4b0bcb2700 -1 osd.6 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.854093+0000 front 2024-04-20T17:58:07.854284+0000 (oldest deadline 2024-04-20T17:58:33.754032+0000) 2024-04-20T17:58:36.344 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.4.service' 2024-04-20T17:58:36.383 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T17:58:36.383 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-20T17:58:36.384 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-20T17:58:36.384 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.5 2024-04-20T17:58:36.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:36 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:36.170+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:36.584 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:36 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:36.153+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:36.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:36 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:36.386+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:36.686 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:36 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:36.524+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:36.687 INFO:journalctl@ceph.osd.5.smithi106.stdout:Apr 20 17:58:36 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-5[57661]: 2024-04-20T17:58:36.337+0000 7fbd03cce700 -1 osd.5 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:04.935289+0000 front 2024-04-20T17:58:04.935448+0000 (oldest deadline 2024-04-20T17:58:30.235292+0000) 2024-04-20T17:58:36.687 INFO:journalctl@ceph.osd.5.smithi106.stdout:Apr 20 17:58:36 smithi106 systemd[1]: Stopping Ceph osd.5 for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:58:36.687 INFO:journalctl@ceph.osd.5.smithi106.stdout:Apr 20 17:58:36 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-5[57661]: 2024-04-20T17:58:36.686+0000 7fbd076da700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T17:58:36.687 INFO:journalctl@ceph.osd.5.smithi106.stdout:Apr 20 17:58:36 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-5[57661]: 2024-04-20T17:58:36.686+0000 7fbd076da700 -1 osd.5 1044 *** Got signal Terminated *** 2024-04-20T17:58:37.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:36 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:36.790+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:37.342 INFO:journalctl@ceph.osd.5.smithi106.stdout:Apr 20 17:58:36 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-5[57661]: 2024-04-20T17:58:36.686+0000 7fbd076da700 -1 osd.5 1044 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T17:58:37.343 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:37 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:37.058+0000 7f4b0bcb2700 -1 osd.6 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.854093+0000 front 2024-04-20T17:58:07.854284+0000 (oldest deadline 2024-04-20T17:58:33.754032+0000) 2024-04-20T17:58:37.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:37 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:37.209+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:37.583 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:37 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:37.173+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:37.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:37 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:37.405+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:37.760 INFO:journalctl@ceph.osd.5.smithi106.stdout:Apr 20 17:58:37 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-5[57661]: 2024-04-20T17:58:37.343+0000 7fbd03cce700 -1 osd.5 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:04.935289+0000 front 2024-04-20T17:58:04.935448+0000 (oldest deadline 2024-04-20T17:58:30.235292+0000) 2024-04-20T17:58:37.760 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:37 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:37.504+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:38.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:37 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:37.747+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:38.425 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:38 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:38.185+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:38.425 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:38 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:38.170+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:38.510 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:38 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:38.102+0000 7f4b0bcb2700 -1 osd.6 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.854093+0000 front 2024-04-20T17:58:07.854284+0000 (oldest deadline 2024-04-20T17:58:33.754032+0000) 2024-04-20T17:58:38.510 INFO:journalctl@ceph.osd.5.smithi106.stdout:Apr 20 17:58:38 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-5[57661]: 2024-04-20T17:58:38.349+0000 7fbd03cce700 -1 osd.5 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:04.935289+0000 front 2024-04-20T17:58:04.935448+0000 (oldest deadline 2024-04-20T17:58:30.235292+0000) 2024-04-20T17:58:38.796 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:38 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:38.425+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:39.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:38 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:38.521+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:39.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:38 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:38.797+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:39.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:38 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:38.797+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:58:39.453 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:39 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:39.193+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:39.453 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:39 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:39.193+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.771365+0000 front 2024-04-20T17:58:15.771279+0000 (oldest deadline 2024-04-20T17:58:38.671320+0000) 2024-04-20T17:58:39.453 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:39 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:39.169+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:39.510 INFO:journalctl@ceph.osd.5.smithi106.stdout:Apr 20 17:58:39 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-5[57661]: 2024-04-20T17:58:39.322+0000 7fbd03cce700 -1 osd.5 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:04.935289+0000 front 2024-04-20T17:58:04.935448+0000 (oldest deadline 2024-04-20T17:58:30.235292+0000) 2024-04-20T17:58:39.510 INFO:journalctl@ceph.osd.5.smithi106.stdout:Apr 20 17:58:39 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-5[57661]: 2024-04-20T17:58:39.322+0000 7fbd03cce700 -1 osd.5 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:13.736372+0000 front 2024-04-20T17:58:13.736643+0000 (oldest deadline 2024-04-20T17:58:39.036378+0000) 2024-04-20T17:58:39.510 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:39 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:39.152+0000 7f4b0bcb2700 -1 osd.6 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.854093+0000 front 2024-04-20T17:58:07.854284+0000 (oldest deadline 2024-04-20T17:58:33.754032+0000) 2024-04-20T17:58:39.750 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:39 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:39.453+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:39.750 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:39 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:39.453+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:58:40.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:39 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:39.541+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:40.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:39 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:39.541+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:12.765228+0000 front 2024-04-20T17:58:12.765133+0000 (oldest deadline 2024-04-20T17:58:39.165982+0000) 2024-04-20T17:58:40.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:39 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:39.750+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:40.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:39 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:39.750+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:58:40.443 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:40 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:40.207+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:40.443 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:40 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:40.207+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.771365+0000 front 2024-04-20T17:58:15.771279+0000 (oldest deadline 2024-04-20T17:58:38.671320+0000) 2024-04-20T17:58:40.443 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:40 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:40.153+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:40.510 INFO:journalctl@ceph.osd.5.smithi106.stdout:Apr 20 17:58:40 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-5[57661]: 2024-04-20T17:58:40.282+0000 7fbd03cce700 -1 osd.5 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:04.935289+0000 front 2024-04-20T17:58:04.935448+0000 (oldest deadline 2024-04-20T17:58:30.235292+0000) 2024-04-20T17:58:40.510 INFO:journalctl@ceph.osd.5.smithi106.stdout:Apr 20 17:58:40 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-5[57661]: 2024-04-20T17:58:40.282+0000 7fbd03cce700 -1 osd.5 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:13.736372+0000 front 2024-04-20T17:58:13.736643+0000 (oldest deadline 2024-04-20T17:58:39.036378+0000) 2024-04-20T17:58:40.510 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:40 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:40.114+0000 7f4b0bcb2700 -1 osd.6 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.854093+0000 front 2024-04-20T17:58:07.854284+0000 (oldest deadline 2024-04-20T17:58:33.754032+0000) 2024-04-20T17:58:40.761 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:40 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:40.444+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:40.762 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:40 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:40.444+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:58:41.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:40 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:40.579+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:41.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:40 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:40.579+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:12.765228+0000 front 2024-04-20T17:58:12.765133+0000 (oldest deadline 2024-04-20T17:58:39.165982+0000) 2024-04-20T17:58:41.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:40 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:40.762+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:41.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:40 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:40.762+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:58:41.457 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:41 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:41.210+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:41.457 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:41 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:41.210+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.771365+0000 front 2024-04-20T17:58:15.771279+0000 (oldest deadline 2024-04-20T17:58:38.671320+0000) 2024-04-20T17:58:41.457 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:41 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:41.183+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:41.510 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:41 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:41.109+0000 7f4b0bcb2700 -1 osd.6 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.854093+0000 front 2024-04-20T17:58:07.854284+0000 (oldest deadline 2024-04-20T17:58:33.754032+0000) 2024-04-20T17:58:41.511 INFO:journalctl@ceph.osd.5.smithi106.stdout:Apr 20 17:58:41 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-5[57661]: 2024-04-20T17:58:41.332+0000 7fbd03cce700 -1 osd.5 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:04.935289+0000 front 2024-04-20T17:58:04.935448+0000 (oldest deadline 2024-04-20T17:58:30.235292+0000) 2024-04-20T17:58:41.511 INFO:journalctl@ceph.osd.5.smithi106.stdout:Apr 20 17:58:41 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-5[57661]: 2024-04-20T17:58:41.332+0000 7fbd03cce700 -1 osd.5 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:13.736372+0000 front 2024-04-20T17:58:13.736643+0000 (oldest deadline 2024-04-20T17:58:39.036378+0000) 2024-04-20T17:58:41.777 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:41 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:41.458+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:41.777 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:41 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:41.458+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:58:42.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:41 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:41.595+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:42.011 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:41 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:41.595+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:12.765228+0000 front 2024-04-20T17:58:12.765133+0000 (oldest deadline 2024-04-20T17:58:39.165982+0000) 2024-04-20T17:58:42.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:41 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:41.778+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:42.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:41 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:41.778+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:58:42.510 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:42 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:42.059+0000 7f4b0bcb2700 -1 osd.6 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.854093+0000 front 2024-04-20T17:58:07.854284+0000 (oldest deadline 2024-04-20T17:58:33.754032+0000) 2024-04-20T17:58:42.510 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:42 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:42.059+0000 7f4b0bcb2700 -1 osd.6 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:17.056298+0000 front 2024-04-20T17:58:17.056429+0000 (oldest deadline 2024-04-20T17:58:41.156429+0000) 2024-04-20T17:58:42.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:42 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:42.208+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:42.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:42 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:42.208+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:58:42.583 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:42 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:42.197+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:42.584 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:42 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:42.197+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.771365+0000 front 2024-04-20T17:58:15.771279+0000 (oldest deadline 2024-04-20T17:58:38.671320+0000) 2024-04-20T17:58:42.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:42 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:42.445+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:42.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:42 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:42.445+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:58:42.895 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:42 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:42.622+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:42.896 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:42 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:42.622+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:12.765228+0000 front 2024-04-20T17:58:12.765133+0000 (oldest deadline 2024-04-20T17:58:39.165982+0000) 2024-04-20T17:58:43.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:42 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:42.736+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:43.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:42 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:42.736+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:58:43.260 INFO:journalctl@ceph.osd.5.smithi106.stdout:Apr 20 17:58:42 smithi106 bash[207703]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-5 2024-04-20T17:58:43.260 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:43 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:43.079+0000 7f4b0bcb2700 -1 osd.6 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.854093+0000 front 2024-04-20T17:58:07.854284+0000 (oldest deadline 2024-04-20T17:58:33.754032+0000) 2024-04-20T17:58:43.260 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:43 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:43.079+0000 7f4b0bcb2700 -1 osd.6 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:17.056298+0000 front 2024-04-20T17:58:17.056429+0000 (oldest deadline 2024-04-20T17:58:41.156429+0000) 2024-04-20T17:58:43.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:43 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:43.195+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:43.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:43 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:43.195+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:58:43.584 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:43 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:43.217+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:43.584 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:43 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:43.217+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.771365+0000 front 2024-04-20T17:58:15.771279+0000 (oldest deadline 2024-04-20T17:58:38.671320+0000) 2024-04-20T17:58:43.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:43 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:43.408+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:43.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:43 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:43.408+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:58:43.994 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:43 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:43.638+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:43.994 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:43 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:43.638+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:12.765228+0000 front 2024-04-20T17:58:12.765133+0000 (oldest deadline 2024-04-20T17:58:39.165982+0000) 2024-04-20T17:58:44.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:43 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:43.715+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:44.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:43 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:43.715+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:58:44.260 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:44 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:44.090+0000 7f4b0bcb2700 -1 osd.6 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.854093+0000 front 2024-04-20T17:58:07.854284+0000 (oldest deadline 2024-04-20T17:58:33.754032+0000) 2024-04-20T17:58:44.260 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:44 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:44.090+0000 7f4b0bcb2700 -1 osd.6 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:17.056298+0000 front 2024-04-20T17:58:17.056429+0000 (oldest deadline 2024-04-20T17:58:41.156429+0000) 2024-04-20T17:58:44.456 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:44 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:44.182+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:44.457 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:44 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:44.182+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.771365+0000 front 2024-04-20T17:58:15.771279+0000 (oldest deadline 2024-04-20T17:58:38.671320+0000) 2024-04-20T17:58:44.457 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:44 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:44.159+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:44.457 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:44 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:44.159+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:58:44.655 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:44 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:44.619+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:44.655 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:44 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:44.619+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:12.765228+0000 front 2024-04-20T17:58:12.765133+0000 (oldest deadline 2024-04-20T17:58:39.165982+0000) 2024-04-20T17:58:44.663 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.5.service' 2024-04-20T17:58:44.705 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T17:58:44.705 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-20T17:58:44.705 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-20T17:58:44.705 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.6 2024-04-20T17:58:44.757 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:44 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:44.457+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:44.758 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:44 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:44.457+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:58:45.010 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:44 smithi106 systemd[1]: Stopping Ceph osd.6 for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:58:45.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:44 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:44.757+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:45.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:44 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:44.757+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:58:45.440 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:45 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:45.192+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:45.440 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:45 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:45.192+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.771365+0000 front 2024-04-20T17:58:15.771279+0000 (oldest deadline 2024-04-20T17:58:38.671320+0000) 2024-04-20T17:58:45.441 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:45 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:45.171+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:45.441 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:45 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:45.171+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:58:45.510 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:45 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:45.022+0000 7f4b136c6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T17:58:45.510 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:45 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:45.022+0000 7f4b136c6700 -1 osd.6 1044 *** Got signal Terminated *** 2024-04-20T17:58:45.510 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:45 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:45.022+0000 7f4b136c6700 -1 osd.6 1044 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T17:58:45.511 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:45 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:45.061+0000 7f4b0bcb2700 -1 osd.6 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.854093+0000 front 2024-04-20T17:58:07.854284+0000 (oldest deadline 2024-04-20T17:58:33.754032+0000) 2024-04-20T17:58:45.511 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:45 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:45.061+0000 7f4b0bcb2700 -1 osd.6 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:17.056298+0000 front 2024-04-20T17:58:17.056429+0000 (oldest deadline 2024-04-20T17:58:41.156429+0000) 2024-04-20T17:58:45.773 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:45 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:45.441+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:45.773 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:45 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:45.441+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:58:46.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:45 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:45.641+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:46.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:45 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:45.641+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:12.765228+0000 front 2024-04-20T17:58:12.765133+0000 (oldest deadline 2024-04-20T17:58:39.165982+0000) 2024-04-20T17:58:46.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:45 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:45.774+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:46.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:45 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:45.774+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:58:46.398 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:46 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:46.177+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:46.398 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:46 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:46.177+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.771365+0000 front 2024-04-20T17:58:15.771279+0000 (oldest deadline 2024-04-20T17:58:38.671320+0000) 2024-04-20T17:58:46.399 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:46 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:46.146+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:46.399 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:46 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:46.146+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:58:46.510 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:46 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:46.101+0000 7f4b0bcb2700 -1 osd.6 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.854093+0000 front 2024-04-20T17:58:07.854284+0000 (oldest deadline 2024-04-20T17:58:33.754032+0000) 2024-04-20T17:58:46.510 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:46 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:46.101+0000 7f4b0bcb2700 -1 osd.6 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:17.056298+0000 front 2024-04-20T17:58:17.056429+0000 (oldest deadline 2024-04-20T17:58:41.156429+0000) 2024-04-20T17:58:46.793 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:46 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:46.399+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:46.793 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:46 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:46.399+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:58:47.013 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:46 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:46.688+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:47.013 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:46 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:46.688+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:12.765228+0000 front 2024-04-20T17:58:12.765133+0000 (oldest deadline 2024-04-20T17:58:39.165982+0000) 2024-04-20T17:58:47.014 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:46 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:46.688+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:21.967785+0000 front 2024-04-20T17:58:21.967771+0000 (oldest deadline 2024-04-20T17:58:46.067760+0000) 2024-04-20T17:58:47.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:46 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:46.792+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:47.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:46 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:46.792+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:58:47.385 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:47 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:47.211+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:47.386 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:47 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:47.211+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.771365+0000 front 2024-04-20T17:58:15.771279+0000 (oldest deadline 2024-04-20T17:58:38.671320+0000) 2024-04-20T17:58:47.386 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:47 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:47.100+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:47.386 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:47 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:47.100+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:58:47.510 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:47 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:47.065+0000 7f4b0bcb2700 -1 osd.6 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.854093+0000 front 2024-04-20T17:58:07.854284+0000 (oldest deadline 2024-04-20T17:58:33.754032+0000) 2024-04-20T17:58:47.510 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:47 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:47.065+0000 7f4b0bcb2700 -1 osd.6 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:17.056298+0000 front 2024-04-20T17:58:17.056429+0000 (oldest deadline 2024-04-20T17:58:41.156429+0000) 2024-04-20T17:58:47.795 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:47 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:47.386+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:47.796 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:47 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:47.386+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:58:48.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:47 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:47.732+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:48.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:47 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:47.732+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:12.765228+0000 front 2024-04-20T17:58:12.765133+0000 (oldest deadline 2024-04-20T17:58:39.165982+0000) 2024-04-20T17:58:48.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:47 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:47.732+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:21.967785+0000 front 2024-04-20T17:58:21.967771+0000 (oldest deadline 2024-04-20T17:58:46.067760+0000) 2024-04-20T17:58:48.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:47 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:47.795+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:48.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:47 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:47.795+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:58:48.510 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:48 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:48.108+0000 7f4b0bcb2700 -1 osd.6 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.854093+0000 front 2024-04-20T17:58:07.854284+0000 (oldest deadline 2024-04-20T17:58:33.754032+0000) 2024-04-20T17:58:48.510 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:48 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:48.108+0000 7f4b0bcb2700 -1 osd.6 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:17.056298+0000 front 2024-04-20T17:58:17.056429+0000 (oldest deadline 2024-04-20T17:58:41.156429+0000) 2024-04-20T17:58:48.510 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:48 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:48.108+0000 7f4b0bcb2700 -1 osd.6 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.558026+0000 front 2024-04-20T17:58:24.558240+0000 (oldest deadline 2024-04-20T17:58:47.458066+0000) 2024-04-20T17:58:48.583 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:48 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:48.345+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:48.583 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:48 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:48.345+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:58:48.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:48 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:48.134+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:48.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:48 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:48.134+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:58:48.584 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:48 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:48.193+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:48.584 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:48 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:48.193+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.771365+0000 front 2024-04-20T17:58:15.771279+0000 (oldest deadline 2024-04-20T17:58:38.671320+0000) 2024-04-20T17:58:49.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:48 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:48.707+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:49.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:48 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:48.707+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:12.765228+0000 front 2024-04-20T17:58:12.765133+0000 (oldest deadline 2024-04-20T17:58:39.165982+0000) 2024-04-20T17:58:49.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:48 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:48.707+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:21.967785+0000 front 2024-04-20T17:58:21.967771+0000 (oldest deadline 2024-04-20T17:58:46.067760+0000) 2024-04-20T17:58:49.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:48 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:48.821+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:49.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:48 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:48.821+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:58:49.510 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:49 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:49.109+0000 7f4b0bcb2700 -1 osd.6 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.854093+0000 front 2024-04-20T17:58:07.854284+0000 (oldest deadline 2024-04-20T17:58:33.754032+0000) 2024-04-20T17:58:49.510 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:49 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:49.109+0000 7f4b0bcb2700 -1 osd.6 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:17.056298+0000 front 2024-04-20T17:58:17.056429+0000 (oldest deadline 2024-04-20T17:58:41.156429+0000) 2024-04-20T17:58:49.510 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:49 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6[139378]: 2024-04-20T17:58:49.109+0000 7f4b0bcb2700 -1 osd.6 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.558026+0000 front 2024-04-20T17:58:24.558240+0000 (oldest deadline 2024-04-20T17:58:47.458066+0000) 2024-04-20T17:58:49.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:49 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:49.179+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:49.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:49 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:49.179+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:58:49.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:49 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:49.179+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:58:49.583 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:49 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:49.241+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:49.584 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:49 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:49.241+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.771365+0000 front 2024-04-20T17:58:15.771279+0000 (oldest deadline 2024-04-20T17:58:38.671320+0000) 2024-04-20T17:58:49.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:49 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:49.336+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:49.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:49 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:49.336+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:58:49.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:49 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:49.336+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:58:50.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:49 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:49.727+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:50.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:49 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:49.727+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:12.765228+0000 front 2024-04-20T17:58:12.765133+0000 (oldest deadline 2024-04-20T17:58:39.165982+0000) 2024-04-20T17:58:50.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:49 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:49.727+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:21.967785+0000 front 2024-04-20T17:58:21.967771+0000 (oldest deadline 2024-04-20T17:58:46.067760+0000) 2024-04-20T17:58:50.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:49 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:49.815+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:50.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:49 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:49.815+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:58:50.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:49 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:49.815+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.996354+0000 front 2024-04-20T17:58:23.996416+0000 (oldest deadline 2024-04-20T17:58:49.796949+0000) 2024-04-20T17:58:50.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:50 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:50.184+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:50.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:50 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:50.184+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:58:50.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:50 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:50.184+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:58:50.584 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:50 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:50.280+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:50.584 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:50 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:50.280+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.771365+0000 front 2024-04-20T17:58:15.771279+0000 (oldest deadline 2024-04-20T17:58:38.671320+0000) 2024-04-20T17:58:50.584 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:50 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:50.280+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.372470+0000 front 2024-04-20T17:58:23.372501+0000 (oldest deadline 2024-04-20T17:58:49.272554+0000) 2024-04-20T17:58:50.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:50 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:50.321+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:50.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:50 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:50.321+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:58:50.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:50 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:50.321+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:58:51.010 INFO:journalctl@ceph.osd.6.smithi106.stdout:Apr 20 17:58:50 smithi106 bash[207966]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-6 2024-04-20T17:58:51.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:50 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:50.696+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:51.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:50 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:50.696+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:12.765228+0000 front 2024-04-20T17:58:12.765133+0000 (oldest deadline 2024-04-20T17:58:39.165982+0000) 2024-04-20T17:58:51.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:50 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:50.696+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:21.967785+0000 front 2024-04-20T17:58:21.967771+0000 (oldest deadline 2024-04-20T17:58:46.067760+0000) 2024-04-20T17:58:51.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:50 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:50.804+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:51.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:50 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:50.804+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:58:51.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:50 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:50.804+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.996354+0000 front 2024-04-20T17:58:23.996416+0000 (oldest deadline 2024-04-20T17:58:49.796949+0000) 2024-04-20T17:58:51.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:51 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:51.166+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:51.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:51 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:51.166+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:58:51.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:51 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:51.166+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:58:51.584 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:51 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:51.238+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:51.584 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:51 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:51.238+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.771365+0000 front 2024-04-20T17:58:15.771279+0000 (oldest deadline 2024-04-20T17:58:38.671320+0000) 2024-04-20T17:58:51.584 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:51 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:51.238+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.372470+0000 front 2024-04-20T17:58:23.372501+0000 (oldest deadline 2024-04-20T17:58:49.272554+0000) 2024-04-20T17:58:51.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:51 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:51.355+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:51.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:51 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:51.355+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:58:51.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:51 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:51.355+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:58:51.760 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:51 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:51.673+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:51.760 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:51 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:51.673+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:12.765228+0000 front 2024-04-20T17:58:12.765133+0000 (oldest deadline 2024-04-20T17:58:39.165982+0000) 2024-04-20T17:58:51.760 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:51 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:51.673+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:21.967785+0000 front 2024-04-20T17:58:21.967771+0000 (oldest deadline 2024-04-20T17:58:46.067760+0000) 2024-04-20T17:58:52.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:51 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:51.763+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:52.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:51 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:51.763+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:58:52.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:51 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:51.763+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.996354+0000 front 2024-04-20T17:58:23.996416+0000 (oldest deadline 2024-04-20T17:58:49.796949+0000) 2024-04-20T17:58:52.524 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.6.service' 2024-04-20T17:58:52.563 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T17:58:52.564 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-20T17:58:52.564 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-20T17:58:52.564 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.7 2024-04-20T17:58:52.583 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:52 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:52.255+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:52.583 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:52 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:52.255+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.771365+0000 front 2024-04-20T17:58:15.771279+0000 (oldest deadline 2024-04-20T17:58:38.671320+0000) 2024-04-20T17:58:52.583 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:52 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:52.255+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.372470+0000 front 2024-04-20T17:58:23.372501+0000 (oldest deadline 2024-04-20T17:58:49.272554+0000) 2024-04-20T17:58:52.583 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:52 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:52.385+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:52.583 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:52 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:52.385+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:58:52.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:52 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:52.385+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:58:52.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:52 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:52.148+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:52.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:52 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:52.148+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:58:52.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:52 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:52.148+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:58:52.856 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:52 smithi106 systemd[1]: Stopping Ceph osd.7 for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:58:52.856 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:52 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:52.642+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:52.856 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:52 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:52.643+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:12.765228+0000 front 2024-04-20T17:58:12.765133+0000 (oldest deadline 2024-04-20T17:58:39.165982+0000) 2024-04-20T17:58:52.856 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:52 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:52.643+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:21.967785+0000 front 2024-04-20T17:58:21.967771+0000 (oldest deadline 2024-04-20T17:58:46.067760+0000) 2024-04-20T17:58:53.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:52 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:52.760+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:53.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:52 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:52.760+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:58:53.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:52 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:52.760+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.996354+0000 front 2024-04-20T17:58:23.996416+0000 (oldest deadline 2024-04-20T17:58:49.796949+0000) 2024-04-20T17:58:53.260 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:52 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:52.856+0000 7f0e75698700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T17:58:53.260 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:52 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:52.856+0000 7f0e75698700 -1 osd.7 1044 *** Got signal Terminated *** 2024-04-20T17:58:53.260 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:52 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:52.856+0000 7f0e75698700 -1 osd.7 1044 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T17:58:53.399 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:53 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:53.213+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:53.399 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:53 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:53.213+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.771365+0000 front 2024-04-20T17:58:15.771279+0000 (oldest deadline 2024-04-20T17:58:38.671320+0000) 2024-04-20T17:58:53.399 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:53 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:53.213+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.372470+0000 front 2024-04-20T17:58:23.372501+0000 (oldest deadline 2024-04-20T17:58:49.272554+0000) 2024-04-20T17:58:53.399 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:53 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:53.127+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:53.399 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:53 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:53.127+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:58:53.399 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:53 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:53.127+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:58:53.724 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:53 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:53.399+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:53.724 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:53 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:53.399+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:58:53.724 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:53 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:53.399+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:58:54.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:53 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:53.630+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:54.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:53 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:53.630+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:12.765228+0000 front 2024-04-20T17:58:12.765133+0000 (oldest deadline 2024-04-20T17:58:39.165982+0000) 2024-04-20T17:58:54.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:53 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:53.630+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:21.967785+0000 front 2024-04-20T17:58:21.967771+0000 (oldest deadline 2024-04-20T17:58:46.067760+0000) 2024-04-20T17:58:54.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:53 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:53.724+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:54.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:53 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:53.724+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:58:54.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:53 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:53.724+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.996354+0000 front 2024-04-20T17:58:23.996416+0000 (oldest deadline 2024-04-20T17:58:49.796949+0000) 2024-04-20T17:58:54.397 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:54 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:54.178+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:54.398 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:54 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:54.178+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.771365+0000 front 2024-04-20T17:58:15.771279+0000 (oldest deadline 2024-04-20T17:58:38.671320+0000) 2024-04-20T17:58:54.398 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:54 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:54.178+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.372470+0000 front 2024-04-20T17:58:23.372501+0000 (oldest deadline 2024-04-20T17:58:49.272554+0000) 2024-04-20T17:58:54.398 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:54 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:54.122+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:54.398 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:54 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:54.122+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:58:54.398 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:54 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:54.122+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:58:54.742 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:54 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:54.398+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:54.742 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:54 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:54.398+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:58:54.742 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:54 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:54.398+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:58:55.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:54 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:54.679+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:55.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:54 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:54.680+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:12.765228+0000 front 2024-04-20T17:58:12.765133+0000 (oldest deadline 2024-04-20T17:58:39.165982+0000) 2024-04-20T17:58:55.011 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:54 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:54.680+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:21.967785+0000 front 2024-04-20T17:58:21.967771+0000 (oldest deadline 2024-04-20T17:58:46.067760+0000) 2024-04-20T17:58:55.011 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:54 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:54.680+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:31.869389+0000 front 2024-04-20T17:58:31.869415+0000 (oldest deadline 2024-04-20T17:58:54.169567+0000) 2024-04-20T17:58:55.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:54 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:54.743+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:55.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:54 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:54.743+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:58:55.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:54 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:54.743+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.996354+0000 front 2024-04-20T17:58:23.996416+0000 (oldest deadline 2024-04-20T17:58:49.796949+0000) 2024-04-20T17:58:55.411 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:55 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:55.191+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:55.411 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:55 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:55.191+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.771365+0000 front 2024-04-20T17:58:15.771279+0000 (oldest deadline 2024-04-20T17:58:38.671320+0000) 2024-04-20T17:58:55.412 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:55 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:55.191+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.372470+0000 front 2024-04-20T17:58:23.372501+0000 (oldest deadline 2024-04-20T17:58:49.272554+0000) 2024-04-20T17:58:55.412 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:55 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:55.151+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:55.412 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:55 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:55.151+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:58:55.412 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:55 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:55.151+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:58:55.412 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:55 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:55.151+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:58:55.790 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:55 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:55.411+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:55.791 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:55 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:55.411+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:58:55.791 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:55 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:55.411+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:58:56.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:55 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:55.724+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:56.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:55 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:55.724+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:12.765228+0000 front 2024-04-20T17:58:12.765133+0000 (oldest deadline 2024-04-20T17:58:39.165982+0000) 2024-04-20T17:58:56.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:55 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:55.724+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:21.967785+0000 front 2024-04-20T17:58:21.967771+0000 (oldest deadline 2024-04-20T17:58:46.067760+0000) 2024-04-20T17:58:56.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:55 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:55.724+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:31.869389+0000 front 2024-04-20T17:58:31.869415+0000 (oldest deadline 2024-04-20T17:58:54.169567+0000) 2024-04-20T17:58:56.086 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:55 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:55.791+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:56.086 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:55 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:55.791+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:58:56.086 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:55 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:55.791+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.996354+0000 front 2024-04-20T17:58:23.996416+0000 (oldest deadline 2024-04-20T17:58:49.796949+0000) 2024-04-20T17:58:56.086 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:55 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:55.791+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:33.298244+0000 front 2024-04-20T17:58:33.298202+0000 (oldest deadline 2024-04-20T17:58:55.598003+0000) 2024-04-20T17:58:56.583 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:56 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:56.408+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:56.583 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:56 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:56.408+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:58:56.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:56 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:56.408+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:58:56.584 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:56 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:56.164+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:56.584 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:56 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:56.164+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.771365+0000 front 2024-04-20T17:58:15.771279+0000 (oldest deadline 2024-04-20T17:58:38.671320+0000) 2024-04-20T17:58:56.584 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:56 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:56.164+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.372470+0000 front 2024-04-20T17:58:23.372501+0000 (oldest deadline 2024-04-20T17:58:49.272554+0000) 2024-04-20T17:58:56.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:56 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:56.188+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:56.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:56 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:56.188+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:58:56.585 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:56 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:56.188+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:58:56.585 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:56 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:56.188+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:58:57.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:56 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:56.727+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:57.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:56 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:56.727+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:12.765228+0000 front 2024-04-20T17:58:12.765133+0000 (oldest deadline 2024-04-20T17:58:39.165982+0000) 2024-04-20T17:58:57.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:56 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:56.727+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:21.967785+0000 front 2024-04-20T17:58:21.967771+0000 (oldest deadline 2024-04-20T17:58:46.067760+0000) 2024-04-20T17:58:57.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:56 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:56.727+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:31.869389+0000 front 2024-04-20T17:58:31.869415+0000 (oldest deadline 2024-04-20T17:58:54.169567+0000) 2024-04-20T17:58:57.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:56 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:56.826+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:57.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:56 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:56.826+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:58:57.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:56 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:56.826+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.996354+0000 front 2024-04-20T17:58:23.996416+0000 (oldest deadline 2024-04-20T17:58:49.796949+0000) 2024-04-20T17:58:57.084 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:56 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:56.826+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:33.298244+0000 front 2024-04-20T17:58:33.298202+0000 (oldest deadline 2024-04-20T17:58:55.598003+0000) 2024-04-20T17:58:57.438 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:57 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:57.163+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:57.438 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:57 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:57.163+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:58:57.438 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:57 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:57.163+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:58:57.438 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:57 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:57.163+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:58:57.438 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:57 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:57.155+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:57.438 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:57 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:57.155+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.771365+0000 front 2024-04-20T17:58:15.771279+0000 (oldest deadline 2024-04-20T17:58:38.671320+0000) 2024-04-20T17:58:57.438 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:57 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:57.155+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.372470+0000 front 2024-04-20T17:58:23.372501+0000 (oldest deadline 2024-04-20T17:58:49.272554+0000) 2024-04-20T17:58:57.791 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:57 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:57.438+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:57.791 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:57 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:57.438+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:58:57.791 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:57 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:57.438+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:58:58.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:57 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:57.691+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.063187+0000 front 2024-04-20T17:58:01.063250+0000 (oldest deadline 2024-04-20T17:58:25.163248+0000) 2024-04-20T17:58:58.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:57 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:57.691+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:12.765228+0000 front 2024-04-20T17:58:12.765133+0000 (oldest deadline 2024-04-20T17:58:39.165982+0000) 2024-04-20T17:58:58.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:57 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:57.691+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:21.967785+0000 front 2024-04-20T17:58:21.967771+0000 (oldest deadline 2024-04-20T17:58:46.067760+0000) 2024-04-20T17:58:58.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:57 smithi106 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7[182370]: 2024-04-20T17:58:57.691+0000 7f0e71c8c700 -1 osd.7 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:31.869389+0000 front 2024-04-20T17:58:31.869415+0000 (oldest deadline 2024-04-20T17:58:54.169567+0000) 2024-04-20T17:58:58.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:57 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:57.792+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:58.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:57 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:57.792+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:58:58.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:57 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:57.792+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.996354+0000 front 2024-04-20T17:58:23.996416+0000 (oldest deadline 2024-04-20T17:58:49.796949+0000) 2024-04-20T17:58:58.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:57 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:57.792+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:33.298244+0000 front 2024-04-20T17:58:33.298202+0000 (oldest deadline 2024-04-20T17:58:55.598003+0000) 2024-04-20T17:58:58.430 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:58 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:58.106+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:58.430 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:58 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:58.106+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.771365+0000 front 2024-04-20T17:58:15.771279+0000 (oldest deadline 2024-04-20T17:58:38.671320+0000) 2024-04-20T17:58:58.430 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:58 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:58.106+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.372470+0000 front 2024-04-20T17:58:23.372501+0000 (oldest deadline 2024-04-20T17:58:49.272554+0000) 2024-04-20T17:58:58.430 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:58 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:58.106+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:33.274196+0000 front 2024-04-20T17:58:33.274347+0000 (oldest deadline 2024-04-20T17:58:57.374094+0000) 2024-04-20T17:58:58.430 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:58 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:58.197+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:58.430 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:58 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:58.197+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:58:58.430 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:58 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:58.197+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:58:58.431 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:58 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:58.197+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:58:58.833 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:58 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:58.431+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:58.833 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:58 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:58.431+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:58:58.833 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:58 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:58.431+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:58:58.833 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:58 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:58.431+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:32.958702+0000 front 2024-04-20T17:58:32.958682+0000 (oldest deadline 2024-04-20T17:58:57.658626+0000) 2024-04-20T17:58:59.010 INFO:journalctl@ceph.osd.7.smithi106.stdout:Apr 20 17:58:58 smithi106 bash[208227]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-7 2024-04-20T17:58:59.210 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:59 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:59.069+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:58:59.210 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:59 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:59.069+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.771365+0000 front 2024-04-20T17:58:15.771279+0000 (oldest deadline 2024-04-20T17:58:38.671320+0000) 2024-04-20T17:58:59.210 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:59 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:59.069+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.372470+0000 front 2024-04-20T17:58:23.372501+0000 (oldest deadline 2024-04-20T17:58:49.272554+0000) 2024-04-20T17:58:59.210 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:58:59 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:58:59.069+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:33.274196+0000 front 2024-04-20T17:58:33.274347+0000 (oldest deadline 2024-04-20T17:58:57.374094+0000) 2024-04-20T17:58:59.211 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:58 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:58.836+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:58:59.211 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:58 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:58.836+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:58:59.211 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:58 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:58.836+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.996354+0000 front 2024-04-20T17:58:23.996416+0000 (oldest deadline 2024-04-20T17:58:49.796949+0000) 2024-04-20T17:58:59.211 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:58 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:58.836+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:33.298244+0000 front 2024-04-20T17:58:33.298202+0000 (oldest deadline 2024-04-20T17:58:55.598003+0000) 2024-04-20T17:58:59.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:59 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:59.211+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:58:59.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:59 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:59.211+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:58:59.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:59 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:59.211+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:58:59.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:58:59 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:58:59.211+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:58:59.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:59 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:59.415+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:58:59.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:59 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:59.415+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:58:59.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:59 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:59.415+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:58:59.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:58:59 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:58:59.415+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:32.958702+0000 front 2024-04-20T17:58:32.958682+0000 (oldest deadline 2024-04-20T17:58:57.658626+0000) 2024-04-20T17:59:00.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:59 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:59.804+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:59:00.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:59 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:59.804+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:59:00.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:59 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:59.804+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.996354+0000 front 2024-04-20T17:58:23.996416+0000 (oldest deadline 2024-04-20T17:58:49.796949+0000) 2024-04-20T17:59:00.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:58:59 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:58:59.804+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:33.298244+0000 front 2024-04-20T17:58:33.298202+0000 (oldest deadline 2024-04-20T17:58:55.598003+0000) 2024-04-20T17:59:00.418 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:00 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:00.091+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:59:00.419 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:00 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:00.091+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.771365+0000 front 2024-04-20T17:58:15.771279+0000 (oldest deadline 2024-04-20T17:58:38.671320+0000) 2024-04-20T17:59:00.419 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:00 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:00.091+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.372470+0000 front 2024-04-20T17:58:23.372501+0000 (oldest deadline 2024-04-20T17:58:49.272554+0000) 2024-04-20T17:59:00.419 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:00 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:00.091+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:33.274196+0000 front 2024-04-20T17:58:33.274347+0000 (oldest deadline 2024-04-20T17:58:57.374094+0000) 2024-04-20T17:59:00.419 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:00 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:00.248+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:00.419 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:00 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:00.248+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:00.419 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:00 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:00.248+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:00.419 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:00 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:00.248+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:00.486 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.7.service' 2024-04-20T17:59:00.526 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T17:59:00.526 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-20T17:59:00.526 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-20T17:59:00.526 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.8 2024-04-20T17:59:00.782 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:00 smithi171 systemd[1]: Stopping Ceph osd.8 for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:59:00.783 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:00 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:00.419+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:59:00.783 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:00 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:00.419+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:59:00.783 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:00 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:00.419+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:59:00.783 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:00 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:00.419+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:32.958702+0000 front 2024-04-20T17:58:32.958682+0000 (oldest deadline 2024-04-20T17:58:57.658626+0000) 2024-04-20T17:59:01.083 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:00 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:00.837+0000 7f72bddb0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T17:59:01.084 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:00 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:00.837+0000 7f72bddb0700 -1 osd.8 1044 *** Got signal Terminated *** 2024-04-20T17:59:01.084 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:00 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:00.837+0000 7f72bddb0700 -1 osd.8 1044 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T17:59:01.084 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:00 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:00.782+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:59:01.084 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:00 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:00.782+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:59:01.084 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:00 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:00.782+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.996354+0000 front 2024-04-20T17:58:23.996416+0000 (oldest deadline 2024-04-20T17:58:49.796949+0000) 2024-04-20T17:59:01.084 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:00 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:00.782+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:33.298244+0000 front 2024-04-20T17:58:33.298202+0000 (oldest deadline 2024-04-20T17:58:55.598003+0000) 2024-04-20T17:59:01.418 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:01 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:01.094+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:59:01.418 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:01 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:01.094+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.771365+0000 front 2024-04-20T17:58:15.771279+0000 (oldest deadline 2024-04-20T17:58:38.671320+0000) 2024-04-20T17:59:01.418 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:01 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:01.094+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.372470+0000 front 2024-04-20T17:58:23.372501+0000 (oldest deadline 2024-04-20T17:58:49.272554+0000) 2024-04-20T17:59:01.418 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:01 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:01.094+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:33.274196+0000 front 2024-04-20T17:58:33.274347+0000 (oldest deadline 2024-04-20T17:58:57.374094+0000) 2024-04-20T17:59:01.418 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:01 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:01.289+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:01.418 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:01 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:01.289+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:01.418 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:01 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:01.289+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:01.419 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:01 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:01.289+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:01.755 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:01 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:01.417+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:59:01.755 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:01 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:01.417+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:59:01.755 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:01 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:01.418+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:59:01.756 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:01 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:01.418+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:32.958702+0000 front 2024-04-20T17:58:32.958682+0000 (oldest deadline 2024-04-20T17:58:57.658626+0000) 2024-04-20T17:59:02.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:01 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:01.756+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:59:02.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:01 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:01.756+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:59:02.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:01 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:01.756+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.996354+0000 front 2024-04-20T17:58:23.996416+0000 (oldest deadline 2024-04-20T17:58:49.796949+0000) 2024-04-20T17:59:02.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:01 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:01.756+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:33.298244+0000 front 2024-04-20T17:58:33.298202+0000 (oldest deadline 2024-04-20T17:58:55.598003+0000) 2024-04-20T17:59:02.377 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:02 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:02.124+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:59:02.377 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:02 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:02.124+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.771365+0000 front 2024-04-20T17:58:15.771279+0000 (oldest deadline 2024-04-20T17:58:38.671320+0000) 2024-04-20T17:59:02.377 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:02 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:02.124+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.372470+0000 front 2024-04-20T17:58:23.372501+0000 (oldest deadline 2024-04-20T17:58:49.272554+0000) 2024-04-20T17:59:02.377 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:02 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:02.124+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:33.274196+0000 front 2024-04-20T17:58:33.274347+0000 (oldest deadline 2024-04-20T17:58:57.374094+0000) 2024-04-20T17:59:02.378 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:02 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:02.243+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:02.378 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:02 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:02.243+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:02.378 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:02 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:02.243+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:02.378 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:02 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:02.243+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:02.797 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:02 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:02.378+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:59:02.797 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:02 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:02.378+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:59:02.797 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:02 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:02.378+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:59:02.797 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:02 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:02.378+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:32.958702+0000 front 2024-04-20T17:58:32.958682+0000 (oldest deadline 2024-04-20T17:58:57.658626+0000) 2024-04-20T17:59:03.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:02 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:02.798+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:59:03.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:02 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:02.798+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:59:03.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:02 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:02.798+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.996354+0000 front 2024-04-20T17:58:23.996416+0000 (oldest deadline 2024-04-20T17:58:49.796949+0000) 2024-04-20T17:59:03.084 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:02 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:02.798+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:33.298244+0000 front 2024-04-20T17:58:33.298202+0000 (oldest deadline 2024-04-20T17:58:55.598003+0000) 2024-04-20T17:59:03.383 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:03 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:03.084+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:59:03.383 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:03 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:03.084+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.771365+0000 front 2024-04-20T17:58:15.771279+0000 (oldest deadline 2024-04-20T17:58:38.671320+0000) 2024-04-20T17:59:03.383 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:03 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:03.084+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.372470+0000 front 2024-04-20T17:58:23.372501+0000 (oldest deadline 2024-04-20T17:58:49.272554+0000) 2024-04-20T17:59:03.383 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:03 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:03.084+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:33.274196+0000 front 2024-04-20T17:58:33.274347+0000 (oldest deadline 2024-04-20T17:58:57.374094+0000) 2024-04-20T17:59:03.383 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:03 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:03.283+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:03.384 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:03 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:03.283+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:03.384 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:03 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:03.283+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:03.384 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:03 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:03.283+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:03.775 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:03 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:03.383+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:59:03.775 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:03 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:03.383+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:59:03.775 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:03 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:03.383+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:59:03.775 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:03 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:03.383+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:32.958702+0000 front 2024-04-20T17:58:32.958682+0000 (oldest deadline 2024-04-20T17:58:57.658626+0000) 2024-04-20T17:59:03.775 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:03 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:03.383+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:37.659168+0000 front 2024-04-20T17:58:37.659326+0000 (oldest deadline 2024-04-20T17:59:02.959081+0000) 2024-04-20T17:59:04.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:03 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:03.776+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:59:04.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:03 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:03.776+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:59:04.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:03 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:03.776+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.996354+0000 front 2024-04-20T17:58:23.996416+0000 (oldest deadline 2024-04-20T17:58:49.796949+0000) 2024-04-20T17:59:04.084 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:03 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:03.776+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:33.298244+0000 front 2024-04-20T17:58:33.298202+0000 (oldest deadline 2024-04-20T17:58:55.598003+0000) 2024-04-20T17:59:04.084 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:03 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:03.776+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.899352+0000 front 2024-04-20T17:58:40.899282+0000 (oldest deadline 2024-04-20T17:59:03.199107+0000) 2024-04-20T17:59:04.414 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:04 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:04.319+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:04.415 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:04 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:04.319+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:04.415 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:04 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:04.319+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:04.415 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:04 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:04.319+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:04.415 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:04 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:04.105+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:59:04.415 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:04 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:04.105+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.771365+0000 front 2024-04-20T17:58:15.771279+0000 (oldest deadline 2024-04-20T17:58:38.671320+0000) 2024-04-20T17:59:04.415 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:04 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:04.105+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.372470+0000 front 2024-04-20T17:58:23.372501+0000 (oldest deadline 2024-04-20T17:58:49.272554+0000) 2024-04-20T17:59:04.415 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:04 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:04.105+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:33.274196+0000 front 2024-04-20T17:58:33.274347+0000 (oldest deadline 2024-04-20T17:58:57.374094+0000) 2024-04-20T17:59:04.415 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:04 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:04.105+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:37.374913+0000 front 2024-04-20T17:58:37.374757+0000 (oldest deadline 2024-04-20T17:59:03.274677+0000) 2024-04-20T17:59:04.825 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:04 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:04.415+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:59:04.825 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:04 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:04.415+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:59:04.825 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:04 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:04.415+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:59:04.825 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:04 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:04.415+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:32.958702+0000 front 2024-04-20T17:58:32.958682+0000 (oldest deadline 2024-04-20T17:58:57.658626+0000) 2024-04-20T17:59:04.825 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:04 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:04.415+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:37.659168+0000 front 2024-04-20T17:58:37.659326+0000 (oldest deadline 2024-04-20T17:59:02.959081+0000) 2024-04-20T17:59:05.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:04 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:04.826+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:59:05.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:04 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:04.826+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:59:05.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:04 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:04.826+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.996354+0000 front 2024-04-20T17:58:23.996416+0000 (oldest deadline 2024-04-20T17:58:49.796949+0000) 2024-04-20T17:59:05.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:04 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:04.826+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:33.298244+0000 front 2024-04-20T17:58:33.298202+0000 (oldest deadline 2024-04-20T17:58:55.598003+0000) 2024-04-20T17:59:05.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:04 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:04.826+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.899352+0000 front 2024-04-20T17:58:40.899282+0000 (oldest deadline 2024-04-20T17:59:03.199107+0000) 2024-04-20T17:59:05.400 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:05 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:05.297+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:05.400 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:05 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:05.297+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:05.400 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:05 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:05.297+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:05.400 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:05 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:05.297+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:05.400 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:05 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:05.133+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:57:58.868570+0000 front 2024-04-20T17:57:58.868664+0000 (oldest deadline 2024-04-20T17:58:24.168621+0000) 2024-04-20T17:59:05.400 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:05 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:05.133+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.771365+0000 front 2024-04-20T17:58:15.771279+0000 (oldest deadline 2024-04-20T17:58:38.671320+0000) 2024-04-20T17:59:05.401 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:05 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:05.133+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.372470+0000 front 2024-04-20T17:58:23.372501+0000 (oldest deadline 2024-04-20T17:58:49.272554+0000) 2024-04-20T17:59:05.401 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:05 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:05.133+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:33.274196+0000 front 2024-04-20T17:58:33.274347+0000 (oldest deadline 2024-04-20T17:58:57.374094+0000) 2024-04-20T17:59:05.401 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:05 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8[49829]: 2024-04-20T17:59:05.133+0000 7f72ba3a4700 -1 osd.8 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:37.374913+0000 front 2024-04-20T17:58:37.374757+0000 (oldest deadline 2024-04-20T17:59:03.274677+0000) 2024-04-20T17:59:05.812 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:05 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:05.401+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:59:05.812 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:05 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:05.401+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:59:05.813 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:05 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:05.401+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:59:05.813 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:05 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:05.401+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:32.958702+0000 front 2024-04-20T17:58:32.958682+0000 (oldest deadline 2024-04-20T17:58:57.658626+0000) 2024-04-20T17:59:05.813 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:05 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:05.401+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:37.659168+0000 front 2024-04-20T17:58:37.659326+0000 (oldest deadline 2024-04-20T17:59:02.959081+0000) 2024-04-20T17:59:06.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:05 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:05.813+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:59:06.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:05 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:05.813+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:59:06.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:05 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:05.813+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.996354+0000 front 2024-04-20T17:58:23.996416+0000 (oldest deadline 2024-04-20T17:58:49.796949+0000) 2024-04-20T17:59:06.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:05 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:05.813+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:33.298244+0000 front 2024-04-20T17:58:33.298202+0000 (oldest deadline 2024-04-20T17:58:55.598003+0000) 2024-04-20T17:59:06.084 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:05 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:05.813+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.899352+0000 front 2024-04-20T17:58:40.899282+0000 (oldest deadline 2024-04-20T17:59:03.199107+0000) 2024-04-20T17:59:06.554 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:06 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:06.252+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:06.554 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:06 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:06.252+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:06.554 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:06 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:06.252+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:06.554 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:06 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:06.252+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:06.555 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:06 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:06.412+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:59:06.555 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:06 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:06.412+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:59:06.555 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:06 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:06.412+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:59:06.555 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:06 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:06.412+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:32.958702+0000 front 2024-04-20T17:58:32.958682+0000 (oldest deadline 2024-04-20T17:58:57.658626+0000) 2024-04-20T17:59:06.555 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:06 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:06.412+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:37.659168+0000 front 2024-04-20T17:58:37.659326+0000 (oldest deadline 2024-04-20T17:59:02.959081+0000) 2024-04-20T17:59:06.833 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 20 17:59:06 smithi171 bash[202147]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-8 2024-04-20T17:59:07.265 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:06 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:06.858+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:59:07.265 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:06 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:06.858+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:59:07.265 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:06 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:06.858+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.996354+0000 front 2024-04-20T17:58:23.996416+0000 (oldest deadline 2024-04-20T17:58:49.796949+0000) 2024-04-20T17:59:07.265 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:06 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:06.858+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:33.298244+0000 front 2024-04-20T17:58:33.298202+0000 (oldest deadline 2024-04-20T17:58:55.598003+0000) 2024-04-20T17:59:07.265 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:06 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:06.858+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.899352+0000 front 2024-04-20T17:58:40.899282+0000 (oldest deadline 2024-04-20T17:59:03.199107+0000) 2024-04-20T17:59:07.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:07 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:07.265+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:07.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:07 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:07.265+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:07.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:07 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:07.265+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:07.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:07 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:07.265+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:07.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:07 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:07.265+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.588179+0000 front 2024-04-20T17:58:40.588216+0000 (oldest deadline 2024-04-20T17:59:06.488042+0000) 2024-04-20T17:59:07.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:07 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:07.446+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:59:07.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:07 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:07.446+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:59:07.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:07 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:07.446+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:59:07.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:07 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:07.446+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:32.958702+0000 front 2024-04-20T17:58:32.958682+0000 (oldest deadline 2024-04-20T17:58:57.658626+0000) 2024-04-20T17:59:07.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:07 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:07.446+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:37.659168+0000 front 2024-04-20T17:58:37.659326+0000 (oldest deadline 2024-04-20T17:59:02.959081+0000) 2024-04-20T17:59:08.084 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:07 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:07.823+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:59:08.084 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:07 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:07.823+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:59:08.084 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:07 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:07.823+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.996354+0000 front 2024-04-20T17:58:23.996416+0000 (oldest deadline 2024-04-20T17:58:49.796949+0000) 2024-04-20T17:59:08.084 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:07 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:07.823+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:33.298244+0000 front 2024-04-20T17:58:33.298202+0000 (oldest deadline 2024-04-20T17:58:55.598003+0000) 2024-04-20T17:59:08.084 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:07 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:07.823+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.899352+0000 front 2024-04-20T17:58:40.899282+0000 (oldest deadline 2024-04-20T17:59:03.199107+0000) 2024-04-20T17:59:08.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:08 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:08.254+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:08.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:08 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:08.254+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:08.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:08 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:08.254+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:08.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:08 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:08.254+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:08.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:08 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:08.254+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.588179+0000 front 2024-04-20T17:58:40.588216+0000 (oldest deadline 2024-04-20T17:59:06.488042+0000) 2024-04-20T17:59:08.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:08 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:08.488+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:59:08.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:08 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:08.488+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:59:08.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:08 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:08.488+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:59:08.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:08 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:08.488+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:32.958702+0000 front 2024-04-20T17:58:32.958682+0000 (oldest deadline 2024-04-20T17:58:57.658626+0000) 2024-04-20T17:59:08.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:08 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:08.488+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:37.659168+0000 front 2024-04-20T17:58:37.659326+0000 (oldest deadline 2024-04-20T17:59:02.959081+0000) 2024-04-20T17:59:08.829 DEBUG:teuthology.orchestra.run.smithi171:> sudo pkill -f 'journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.8.service' 2024-04-20T17:59:08.871 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T17:59:08.872 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-20T17:59:08.872 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-20T17:59:08.872 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.9 2024-04-20T17:59:09.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:08 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:08.786+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:59:09.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:08 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:08.786+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:59:09.084 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:08 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:08.786+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.996354+0000 front 2024-04-20T17:58:23.996416+0000 (oldest deadline 2024-04-20T17:58:49.796949+0000) 2024-04-20T17:59:09.084 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:08 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:08.786+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:33.298244+0000 front 2024-04-20T17:58:33.298202+0000 (oldest deadline 2024-04-20T17:58:55.598003+0000) 2024-04-20T17:59:09.084 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:08 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:08.786+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.899352+0000 front 2024-04-20T17:58:40.899282+0000 (oldest deadline 2024-04-20T17:59:03.199107+0000) 2024-04-20T17:59:09.084 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:08 smithi171 systemd[1]: Stopping Ceph osd.9 for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:59:09.510 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:09 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:09.204+0000 7fbb04c78700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T17:59:09.510 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:09 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:09.204+0000 7fbb04c78700 -1 osd.9 1044 *** Got signal Terminated *** 2024-04-20T17:59:09.510 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:09 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:09.204+0000 7fbb04c78700 -1 osd.9 1044 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T17:59:09.510 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:09 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:09.293+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:09.510 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:09 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:09.293+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:09.510 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:09 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:09.293+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:09.510 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:09 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:09.293+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:09.510 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:09 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:09.293+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.588179+0000 front 2024-04-20T17:58:40.588216+0000 (oldest deadline 2024-04-20T17:59:06.488042+0000) 2024-04-20T17:59:09.833 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:09 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:09.745+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:59:09.833 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:09 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:09.745+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:59:09.833 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:09 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:09.745+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.996354+0000 front 2024-04-20T17:58:23.996416+0000 (oldest deadline 2024-04-20T17:58:49.796949+0000) 2024-04-20T17:59:09.833 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:09 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:09.745+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:33.298244+0000 front 2024-04-20T17:58:33.298202+0000 (oldest deadline 2024-04-20T17:58:55.598003+0000) 2024-04-20T17:59:09.833 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:09 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:09.745+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.899352+0000 front 2024-04-20T17:58:40.899282+0000 (oldest deadline 2024-04-20T17:59:03.199107+0000) 2024-04-20T17:59:09.834 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:09 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:09.510+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:59:09.834 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:09 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:09.510+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:59:09.834 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:09 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:09.510+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:59:09.834 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:09 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:09.510+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:32.958702+0000 front 2024-04-20T17:58:32.958682+0000 (oldest deadline 2024-04-20T17:58:57.658626+0000) 2024-04-20T17:59:09.834 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:09 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:09.510+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:37.659168+0000 front 2024-04-20T17:58:37.659326+0000 (oldest deadline 2024-04-20T17:59:02.959081+0000) 2024-04-20T17:59:10.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:10 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:10.286+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:10.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:10 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:10.286+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:10.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:10 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:10.286+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:10.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:10 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:10.286+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:10.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:10 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:10.286+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.588179+0000 front 2024-04-20T17:58:40.588216+0000 (oldest deadline 2024-04-20T17:59:06.488042+0000) 2024-04-20T17:59:10.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:10 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:10.528+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:59:10.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:10 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:10.528+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:59:10.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:10 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:10.528+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:59:10.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:10 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:10.528+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:32.958702+0000 front 2024-04-20T17:58:32.958682+0000 (oldest deadline 2024-04-20T17:58:57.658626+0000) 2024-04-20T17:59:10.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:10 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:10.528+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:37.659168+0000 front 2024-04-20T17:58:37.659326+0000 (oldest deadline 2024-04-20T17:59:02.959081+0000) 2024-04-20T17:59:11.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:10 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:10.743+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:59:11.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:10 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:10.743+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:59:11.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:10 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:10.743+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.996354+0000 front 2024-04-20T17:58:23.996416+0000 (oldest deadline 2024-04-20T17:58:49.796949+0000) 2024-04-20T17:59:11.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:10 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:10.743+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:33.298244+0000 front 2024-04-20T17:58:33.298202+0000 (oldest deadline 2024-04-20T17:58:55.598003+0000) 2024-04-20T17:59:11.084 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:10 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:10.743+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.899352+0000 front 2024-04-20T17:58:40.899282+0000 (oldest deadline 2024-04-20T17:59:03.199107+0000) 2024-04-20T17:59:11.521 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:11 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:11.259+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:11.521 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:11 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:11.259+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:11.521 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:11 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:11.259+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:11.521 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:11 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:11.259+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:11.521 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:11 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:11.259+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.588179+0000 front 2024-04-20T17:58:40.588216+0000 (oldest deadline 2024-04-20T17:59:06.488042+0000) 2024-04-20T17:59:11.521 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:11 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:11.259+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:49.190161+0000 front 2024-04-20T17:58:49.190253+0000 (oldest deadline 2024-04-20T17:59:10.890056+0000) 2024-04-20T17:59:11.833 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:11 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:11.744+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:59:11.833 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:11 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:11.744+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:59:11.833 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:11 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:11.744+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.996354+0000 front 2024-04-20T17:58:23.996416+0000 (oldest deadline 2024-04-20T17:58:49.796949+0000) 2024-04-20T17:59:11.833 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:11 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:11.744+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:33.298244+0000 front 2024-04-20T17:58:33.298202+0000 (oldest deadline 2024-04-20T17:58:55.598003+0000) 2024-04-20T17:59:11.834 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:11 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:11.744+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.899352+0000 front 2024-04-20T17:58:40.899282+0000 (oldest deadline 2024-04-20T17:59:03.199107+0000) 2024-04-20T17:59:11.834 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:11 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:11.521+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:59:11.834 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:11 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:11.521+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:59:11.834 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:11 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:11.521+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:59:11.834 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:11 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:11.521+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:32.958702+0000 front 2024-04-20T17:58:32.958682+0000 (oldest deadline 2024-04-20T17:58:57.658626+0000) 2024-04-20T17:59:11.834 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:11 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:11.521+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:37.659168+0000 front 2024-04-20T17:58:37.659326+0000 (oldest deadline 2024-04-20T17:59:02.959081+0000) 2024-04-20T17:59:12.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:12 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:12.245+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:12.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:12 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:12.245+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:12.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:12 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:12.245+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:12.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:12 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:12.245+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:12.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:12 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:12.245+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.588179+0000 front 2024-04-20T17:58:40.588216+0000 (oldest deadline 2024-04-20T17:59:06.488042+0000) 2024-04-20T17:59:12.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:12 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:12.245+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:49.190161+0000 front 2024-04-20T17:58:49.190253+0000 (oldest deadline 2024-04-20T17:59:10.890056+0000) 2024-04-20T17:59:12.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:12 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:12.479+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:59:12.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:12 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:12.479+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:59:12.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:12 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:12.479+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:59:12.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:12 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:12.479+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:32.958702+0000 front 2024-04-20T17:58:32.958682+0000 (oldest deadline 2024-04-20T17:58:57.658626+0000) 2024-04-20T17:59:12.585 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:12 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:12.479+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:37.659168+0000 front 2024-04-20T17:58:37.659326+0000 (oldest deadline 2024-04-20T17:59:02.959081+0000) 2024-04-20T17:59:13.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:12 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:12.740+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:59:13.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:12 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:12.740+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:59:13.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:12 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:12.740+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.996354+0000 front 2024-04-20T17:58:23.996416+0000 (oldest deadline 2024-04-20T17:58:49.796949+0000) 2024-04-20T17:59:13.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:12 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:12.740+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:33.298244+0000 front 2024-04-20T17:58:33.298202+0000 (oldest deadline 2024-04-20T17:58:55.598003+0000) 2024-04-20T17:59:13.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:12 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:12.740+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.899352+0000 front 2024-04-20T17:58:40.899282+0000 (oldest deadline 2024-04-20T17:59:03.199107+0000) 2024-04-20T17:59:13.084 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:12 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:12.740+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:46.700177+0000 front 2024-04-20T17:58:46.700232+0000 (oldest deadline 2024-04-20T17:59:11.999982+0000) 2024-04-20T17:59:13.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:13 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:13.237+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:13.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:13 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:13.237+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:13.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:13 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:13.237+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:13.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:13 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:13.237+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:13.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:13 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:13.237+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.588179+0000 front 2024-04-20T17:58:40.588216+0000 (oldest deadline 2024-04-20T17:59:06.488042+0000) 2024-04-20T17:59:13.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:13 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:13.237+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:49.190161+0000 front 2024-04-20T17:58:49.190253+0000 (oldest deadline 2024-04-20T17:59:10.890056+0000) 2024-04-20T17:59:13.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:13 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:13.484+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:59:13.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:13 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:13.484+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:59:13.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:13 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:13.484+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:59:13.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:13 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:13.484+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:32.958702+0000 front 2024-04-20T17:58:32.958682+0000 (oldest deadline 2024-04-20T17:58:57.658626+0000) 2024-04-20T17:59:13.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:13 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:13.484+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:37.659168+0000 front 2024-04-20T17:58:37.659326+0000 (oldest deadline 2024-04-20T17:59:02.959081+0000) 2024-04-20T17:59:14.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:13 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:13.754+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.094053+0000 front 2024-04-20T17:58:07.094379+0000 (oldest deadline 2024-04-20T17:58:31.793879+0000) 2024-04-20T17:59:14.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:13 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:13.754+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.195399+0000 front 2024-04-20T17:58:15.195325+0000 (oldest deadline 2024-04-20T17:58:38.095472+0000) 2024-04-20T17:59:14.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:13 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:13.754+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.996354+0000 front 2024-04-20T17:58:23.996416+0000 (oldest deadline 2024-04-20T17:58:49.796949+0000) 2024-04-20T17:59:14.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:13 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:13.754+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:33.298244+0000 front 2024-04-20T17:58:33.298202+0000 (oldest deadline 2024-04-20T17:58:55.598003+0000) 2024-04-20T17:59:14.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:13 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:13.754+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.899352+0000 front 2024-04-20T17:58:40.899282+0000 (oldest deadline 2024-04-20T17:59:03.199107+0000) 2024-04-20T17:59:14.083 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:13 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9[57611]: 2024-04-20T17:59:13.754+0000 7fbafd264700 -1 osd.9 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:46.700177+0000 front 2024-04-20T17:58:46.700232+0000 (oldest deadline 2024-04-20T17:59:11.999982+0000) 2024-04-20T17:59:14.525 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:14 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:14.209+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:14.526 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:14 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:14.209+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:14.526 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:14 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:14.209+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:14.526 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:14 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:14.209+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:14.526 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:14 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:14.209+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.588179+0000 front 2024-04-20T17:58:40.588216+0000 (oldest deadline 2024-04-20T17:59:06.488042+0000) 2024-04-20T17:59:14.526 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:14 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:14.209+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:49.190161+0000 front 2024-04-20T17:58:49.190253+0000 (oldest deadline 2024-04-20T17:59:10.890056+0000) 2024-04-20T17:59:14.833 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:14 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:14.525+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:59:14.833 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:14 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:14.525+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:59:14.833 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:14 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:14.525+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:59:14.833 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:14 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:14.525+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:32.958702+0000 front 2024-04-20T17:58:32.958682+0000 (oldest deadline 2024-04-20T17:58:57.658626+0000) 2024-04-20T17:59:14.833 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:14 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:14.525+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:37.659168+0000 front 2024-04-20T17:58:37.659326+0000 (oldest deadline 2024-04-20T17:59:02.959081+0000) 2024-04-20T17:59:15.333 INFO:journalctl@ceph.osd.9.smithi171.stdout:Apr 20 17:59:15 smithi171 bash[202422]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-9 2024-04-20T17:59:15.333 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:15 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:15.173+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:15.333 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:15 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:15.173+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:15.333 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:15 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:15.173+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:15.333 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:15 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:15.173+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:15.334 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:15 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:15.173+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.588179+0000 front 2024-04-20T17:58:40.588216+0000 (oldest deadline 2024-04-20T17:59:06.488042+0000) 2024-04-20T17:59:15.334 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:15 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:15.173+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:49.190161+0000 front 2024-04-20T17:58:49.190253+0000 (oldest deadline 2024-04-20T17:59:10.890056+0000) 2024-04-20T17:59:15.833 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:15 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:15.495+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:59:15.833 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:15 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:15.496+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:59:15.833 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:15 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:15.496+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:59:15.834 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:15 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:15.496+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:32.958702+0000 front 2024-04-20T17:58:32.958682+0000 (oldest deadline 2024-04-20T17:58:57.658626+0000) 2024-04-20T17:59:15.834 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:15 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:15.496+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:37.659168+0000 front 2024-04-20T17:58:37.659326+0000 (oldest deadline 2024-04-20T17:59:02.959081+0000) 2024-04-20T17:59:16.333 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:16 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:16.168+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:16.334 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:16 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:16.168+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:16.334 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:16 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:16.168+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:16.334 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:16 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:16.168+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:16.334 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:16 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:16.168+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.588179+0000 front 2024-04-20T17:58:40.588216+0000 (oldest deadline 2024-04-20T17:59:06.488042+0000) 2024-04-20T17:59:16.334 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:16 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:16.168+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:49.190161+0000 front 2024-04-20T17:58:49.190253+0000 (oldest deadline 2024-04-20T17:59:10.890056+0000) 2024-04-20T17:59:16.723 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:16 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:16.480+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:59:16.723 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:16 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:16.480+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:59:16.723 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:16 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:16.480+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:59:16.723 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:16 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:16.480+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:32.958702+0000 front 2024-04-20T17:58:32.958682+0000 (oldest deadline 2024-04-20T17:58:57.658626+0000) 2024-04-20T17:59:16.723 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:16 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:16.480+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:37.659168+0000 front 2024-04-20T17:58:37.659326+0000 (oldest deadline 2024-04-20T17:59:02.959081+0000) 2024-04-20T17:59:16.723 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:16 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:16.480+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:47.461402+0000 front 2024-04-20T17:58:47.461465+0000 (oldest deadline 2024-04-20T17:59:15.661764+0000) 2024-04-20T17:59:17.015 DEBUG:teuthology.orchestra.run.smithi171:> sudo pkill -f 'journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.9.service' 2024-04-20T17:59:17.056 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T17:59:17.057 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-20T17:59:17.057 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-20T17:59:17.057 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.10 2024-04-20T17:59:17.333 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:17 smithi171 systemd[1]: Stopping Ceph osd.10 for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:59:17.333 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:17 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:17.214+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:17.333 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:17 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:17.214+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:17.334 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:17 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:17.214+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:17.334 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:17 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:17.214+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:17.334 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:17 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:17.214+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.588179+0000 front 2024-04-20T17:58:40.588216+0000 (oldest deadline 2024-04-20T17:59:06.488042+0000) 2024-04-20T17:59:17.334 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:17 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:17.214+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:49.190161+0000 front 2024-04-20T17:58:49.190253+0000 (oldest deadline 2024-04-20T17:59:10.890056+0000) 2024-04-20T17:59:17.833 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:17 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:17.344+0000 7f5cd308b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T17:59:17.833 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:17 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:17.344+0000 7f5cd308b700 -1 osd.10 1044 *** Got signal Terminated *** 2024-04-20T17:59:17.833 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:17 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:17.344+0000 7f5cd308b700 -1 osd.10 1044 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T17:59:17.833 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:17 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:17.514+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:59:17.833 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:17 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:17.514+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:59:17.834 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:17 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:17.514+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:59:17.834 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:17 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:17.514+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:32.958702+0000 front 2024-04-20T17:58:32.958682+0000 (oldest deadline 2024-04-20T17:58:57.658626+0000) 2024-04-20T17:59:17.834 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:17 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:17.514+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:37.659168+0000 front 2024-04-20T17:58:37.659326+0000 (oldest deadline 2024-04-20T17:59:02.959081+0000) 2024-04-20T17:59:17.834 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:17 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:17.514+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:47.461402+0000 front 2024-04-20T17:58:47.461465+0000 (oldest deadline 2024-04-20T17:59:15.661764+0000) 2024-04-20T17:59:18.473 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:18 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:18.203+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:18.473 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:18 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:18.203+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:18.473 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:18 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:18.203+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:18.473 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:18 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:18.203+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:18.473 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:18 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:18.203+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.588179+0000 front 2024-04-20T17:58:40.588216+0000 (oldest deadline 2024-04-20T17:59:06.488042+0000) 2024-04-20T17:59:18.473 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:18 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:18.203+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:49.190161+0000 front 2024-04-20T17:58:49.190253+0000 (oldest deadline 2024-04-20T17:59:10.890056+0000) 2024-04-20T17:59:18.833 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:18 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:18.474+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:59:18.833 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:18 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:18.474+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:59:18.833 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:18 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:18.474+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:59:18.833 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:18 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:18.474+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:32.958702+0000 front 2024-04-20T17:58:32.958682+0000 (oldest deadline 2024-04-20T17:58:57.658626+0000) 2024-04-20T17:59:18.833 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:18 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:18.474+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:37.659168+0000 front 2024-04-20T17:58:37.659326+0000 (oldest deadline 2024-04-20T17:59:02.959081+0000) 2024-04-20T17:59:18.834 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:18 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:18.474+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:47.461402+0000 front 2024-04-20T17:58:47.461465+0000 (oldest deadline 2024-04-20T17:59:15.661764+0000) 2024-04-20T17:59:19.504 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:19 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:19.213+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:19.504 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:19 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:19.213+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:19.504 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:19 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:19.213+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:19.504 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:19 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:19.213+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:19.504 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:19 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:19.213+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.588179+0000 front 2024-04-20T17:58:40.588216+0000 (oldest deadline 2024-04-20T17:59:06.488042+0000) 2024-04-20T17:59:19.504 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:19 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:19.213+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:49.190161+0000 front 2024-04-20T17:58:49.190253+0000 (oldest deadline 2024-04-20T17:59:10.890056+0000) 2024-04-20T17:59:19.833 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:19 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:19.504+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:59:19.833 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:19 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:19.504+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:59:19.833 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:19 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:19.504+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:59:19.834 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:19 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:19.504+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:32.958702+0000 front 2024-04-20T17:58:32.958682+0000 (oldest deadline 2024-04-20T17:58:57.658626+0000) 2024-04-20T17:59:19.834 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:19 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:19.504+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:37.659168+0000 front 2024-04-20T17:58:37.659326+0000 (oldest deadline 2024-04-20T17:59:02.959081+0000) 2024-04-20T17:59:19.834 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:19 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:19.504+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:47.461402+0000 front 2024-04-20T17:58:47.461465+0000 (oldest deadline 2024-04-20T17:59:15.661764+0000) 2024-04-20T17:59:20.583 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:20 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:20.467+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:59:20.583 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:20 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:20.467+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:59:20.583 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:20 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:20.467+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:59:20.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:20 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:20.467+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:32.958702+0000 front 2024-04-20T17:58:32.958682+0000 (oldest deadline 2024-04-20T17:58:57.658626+0000) 2024-04-20T17:59:20.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:20 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:20.467+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:37.659168+0000 front 2024-04-20T17:58:37.659326+0000 (oldest deadline 2024-04-20T17:59:02.959081+0000) 2024-04-20T17:59:20.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:20 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:20.467+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:47.461402+0000 front 2024-04-20T17:58:47.461465+0000 (oldest deadline 2024-04-20T17:59:15.661764+0000) 2024-04-20T17:59:20.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:20 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:20.467+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6814 osd.7 since back 2024-04-20T17:58:57.362942+0000 front 2024-04-20T17:58:57.362895+0000 (oldest deadline 2024-04-20T17:59:20.262900+0000) 2024-04-20T17:59:20.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:20 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:20.246+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:20.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:20 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:20.246+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:20.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:20 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:20.246+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:20.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:20 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:20.246+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:20.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:20 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:20.246+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.588179+0000 front 2024-04-20T17:58:40.588216+0000 (oldest deadline 2024-04-20T17:59:06.488042+0000) 2024-04-20T17:59:20.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:20 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:20.246+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:49.190161+0000 front 2024-04-20T17:58:49.190253+0000 (oldest deadline 2024-04-20T17:59:10.890056+0000) 2024-04-20T17:59:20.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:20 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:20.246+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6814 osd.7 since back 2024-04-20T17:58:56.691794+0000 front 2024-04-20T17:58:56.691780+0000 (oldest deadline 2024-04-20T17:59:20.191659+0000) 2024-04-20T17:59:21.583 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:21 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:21.429+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6807 osd.1 since back 2024-04-20T17:58:07.953980+0000 front 2024-04-20T17:58:07.954132+0000 (oldest deadline 2024-04-20T17:58:30.853777+0000) 2024-04-20T17:59:21.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:21 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:21.429+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:15.455793+0000 front 2024-04-20T17:58:15.456027+0000 (oldest deadline 2024-04-20T17:58:38.955832+0000) 2024-04-20T17:59:21.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:21 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:21.429+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:23.056773+0000 front 2024-04-20T17:58:23.056747+0000 (oldest deadline 2024-04-20T17:58:48.956937+0000) 2024-04-20T17:59:21.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:21 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:21.429+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:32.958702+0000 front 2024-04-20T17:58:32.958682+0000 (oldest deadline 2024-04-20T17:58:57.658626+0000) 2024-04-20T17:59:21.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:21 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:21.429+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:37.659168+0000 front 2024-04-20T17:58:37.659326+0000 (oldest deadline 2024-04-20T17:59:02.959081+0000) 2024-04-20T17:59:21.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:21 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:21.429+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:47.461402+0000 front 2024-04-20T17:58:47.461465+0000 (oldest deadline 2024-04-20T17:59:15.661764+0000) 2024-04-20T17:59:21.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:21 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10[65557]: 2024-04-20T17:59:21.429+0000 7f5ccb677700 -1 osd.10 1044 heartbeat_check: no reply from 172.21.15.106:6814 osd.7 since back 2024-04-20T17:58:57.362942+0000 front 2024-04-20T17:58:57.362895+0000 (oldest deadline 2024-04-20T17:59:20.262900+0000) 2024-04-20T17:59:21.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:21 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:21.250+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:21.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:21 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:21.250+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:21.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:21 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:21.250+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:21.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:21 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:21.250+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:21.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:21 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:21.250+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.588179+0000 front 2024-04-20T17:58:40.588216+0000 (oldest deadline 2024-04-20T17:59:06.488042+0000) 2024-04-20T17:59:21.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:21 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:21.250+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:49.190161+0000 front 2024-04-20T17:58:49.190253+0000 (oldest deadline 2024-04-20T17:59:10.890056+0000) 2024-04-20T17:59:21.585 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:21 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:21.250+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6814 osd.7 since back 2024-04-20T17:58:56.691794+0000 front 2024-04-20T17:58:56.691780+0000 (oldest deadline 2024-04-20T17:59:20.191659+0000) 2024-04-20T17:59:22.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:22 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:22.260+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:22.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:22 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:22.260+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:22.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:22 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:22.260+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:22.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:22 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:22.260+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:22.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:22 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:22.260+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.588179+0000 front 2024-04-20T17:58:40.588216+0000 (oldest deadline 2024-04-20T17:59:06.488042+0000) 2024-04-20T17:59:22.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:22 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:22.260+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:49.190161+0000 front 2024-04-20T17:58:49.190253+0000 (oldest deadline 2024-04-20T17:59:10.890056+0000) 2024-04-20T17:59:22.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:22 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:22.260+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6814 osd.7 since back 2024-04-20T17:58:56.691794+0000 front 2024-04-20T17:58:56.691780+0000 (oldest deadline 2024-04-20T17:59:20.191659+0000) 2024-04-20T17:59:23.584 INFO:journalctl@ceph.osd.10.smithi171.stdout:Apr 20 17:59:23 smithi171 bash[202685]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-10 2024-04-20T17:59:23.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:23 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:23.247+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:23.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:23 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:23.247+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:23.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:23 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:23.247+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:23.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:23 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:23.247+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:23.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:23 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:23.247+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.588179+0000 front 2024-04-20T17:58:40.588216+0000 (oldest deadline 2024-04-20T17:59:06.488042+0000) 2024-04-20T17:59:23.585 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:23 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:23.247+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:49.190161+0000 front 2024-04-20T17:58:49.190253+0000 (oldest deadline 2024-04-20T17:59:10.890056+0000) 2024-04-20T17:59:23.585 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:23 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:23.247+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6814 osd.7 since back 2024-04-20T17:58:56.691794+0000 front 2024-04-20T17:58:56.691780+0000 (oldest deadline 2024-04-20T17:59:20.191659+0000) 2024-04-20T17:59:24.333 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:24 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:24.220+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:24.333 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:24 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:24.220+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:24.333 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:24 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:24.220+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:24.333 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:24 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:24.220+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:24.333 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:24 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:24.220+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.588179+0000 front 2024-04-20T17:58:40.588216+0000 (oldest deadline 2024-04-20T17:59:06.488042+0000) 2024-04-20T17:59:24.334 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:24 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:24.220+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:49.190161+0000 front 2024-04-20T17:58:49.190253+0000 (oldest deadline 2024-04-20T17:59:10.890056+0000) 2024-04-20T17:59:24.334 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:24 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:24.220+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6814 osd.7 since back 2024-04-20T17:58:56.691794+0000 front 2024-04-20T17:58:56.691780+0000 (oldest deadline 2024-04-20T17:59:20.191659+0000) 2024-04-20T17:59:24.999 DEBUG:teuthology.orchestra.run.smithi171:> sudo pkill -f 'journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.10.service' 2024-04-20T17:59:25.040 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T17:59:25.041 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-20T17:59:25.041 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-20T17:59:25.041 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.11 2024-04-20T17:59:25.333 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:25 smithi171 systemd[1]: Stopping Ceph osd.11 for 13c6e76c-ff37-11ee-bc93-c7b262605968... 2024-04-20T17:59:25.333 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:25 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:25.224+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:25.334 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:25 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:25.224+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:25.334 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:25 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:25.224+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:25.334 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:25 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:25.224+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:25.334 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:25 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:25.224+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.588179+0000 front 2024-04-20T17:58:40.588216+0000 (oldest deadline 2024-04-20T17:59:06.488042+0000) 2024-04-20T17:59:25.334 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:25 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:25.224+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:49.190161+0000 front 2024-04-20T17:58:49.190253+0000 (oldest deadline 2024-04-20T17:59:10.890056+0000) 2024-04-20T17:59:25.334 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:25 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:25.224+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6814 osd.7 since back 2024-04-20T17:58:56.691794+0000 front 2024-04-20T17:58:56.691780+0000 (oldest deadline 2024-04-20T17:59:20.191659+0000) 2024-04-20T17:59:25.833 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:25 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:25.341+0000 7f113e198700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-20T17:59:25.833 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:25 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:25.341+0000 7f113e198700 -1 osd.11 1044 *** Got signal Terminated *** 2024-04-20T17:59:25.833 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:25 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:25.341+0000 7f113e198700 -1 osd.11 1044 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-20T17:59:26.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:26 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:26.228+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:26.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:26 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:26.228+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:26.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:26 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:26.228+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:26.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:26 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:26.228+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:26.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:26 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:26.228+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.588179+0000 front 2024-04-20T17:58:40.588216+0000 (oldest deadline 2024-04-20T17:59:06.488042+0000) 2024-04-20T17:59:26.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:26 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:26.228+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:49.190161+0000 front 2024-04-20T17:58:49.190253+0000 (oldest deadline 2024-04-20T17:59:10.890056+0000) 2024-04-20T17:59:26.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:26 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:26.228+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6814 osd.7 since back 2024-04-20T17:58:56.691794+0000 front 2024-04-20T17:58:56.691780+0000 (oldest deadline 2024-04-20T17:59:20.191659+0000) 2024-04-20T17:59:27.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:27 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:27.254+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:27.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:27 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:27.254+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:27.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:27 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:27.254+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:27.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:27 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:27.254+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:27.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:27 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:27.254+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.588179+0000 front 2024-04-20T17:58:40.588216+0000 (oldest deadline 2024-04-20T17:59:06.488042+0000) 2024-04-20T17:59:27.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:27 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:27.254+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:49.190161+0000 front 2024-04-20T17:58:49.190253+0000 (oldest deadline 2024-04-20T17:59:10.890056+0000) 2024-04-20T17:59:27.584 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:27 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:27.254+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6814 osd.7 since back 2024-04-20T17:58:56.691794+0000 front 2024-04-20T17:58:56.691780+0000 (oldest deadline 2024-04-20T17:59:20.191659+0000) 2024-04-20T17:59:28.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:28 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:28.219+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:28.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:28 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:28.219+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:28.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:28 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:28.219+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:28.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:28 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:28.219+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:28.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:28 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:28.219+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.588179+0000 front 2024-04-20T17:58:40.588216+0000 (oldest deadline 2024-04-20T17:59:06.488042+0000) 2024-04-20T17:59:28.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:28 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:28.219+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:49.190161+0000 front 2024-04-20T17:58:49.190253+0000 (oldest deadline 2024-04-20T17:59:10.890056+0000) 2024-04-20T17:59:28.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:28 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:28.219+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6814 osd.7 since back 2024-04-20T17:58:56.691794+0000 front 2024-04-20T17:58:56.691780+0000 (oldest deadline 2024-04-20T17:59:20.191659+0000) 2024-04-20T17:59:29.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:29 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:29.195+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:29.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:29 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:29.195+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:29.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:29 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:29.195+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:29.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:29 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:29.195+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:29.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:29 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:29.195+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.588179+0000 front 2024-04-20T17:58:40.588216+0000 (oldest deadline 2024-04-20T17:59:06.488042+0000) 2024-04-20T17:59:29.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:29 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:29.195+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:49.190161+0000 front 2024-04-20T17:58:49.190253+0000 (oldest deadline 2024-04-20T17:59:10.890056+0000) 2024-04-20T17:59:29.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:29 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:29.195+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6814 osd.7 since back 2024-04-20T17:58:56.691794+0000 front 2024-04-20T17:58:56.691780+0000 (oldest deadline 2024-04-20T17:59:20.191659+0000) 2024-04-20T17:59:30.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:30 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:30.176+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6803 osd.0 since back 2024-04-20T17:58:01.179840+0000 front 2024-04-20T17:58:01.179909+0000 (oldest deadline 2024-04-20T17:58:22.879838+0000) 2024-04-20T17:59:30.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:30 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:30.176+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6811 osd.2 since back 2024-04-20T17:58:16.083680+0000 front 2024-04-20T17:58:16.083773+0000 (oldest deadline 2024-04-20T17:58:41.983831+0000) 2024-04-20T17:59:30.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:30 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:30.176+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.5:6815 osd.3 since back 2024-04-20T17:58:24.785508+0000 front 2024-04-20T17:58:24.785594+0000 (oldest deadline 2024-04-20T17:58:48.285342+0000) 2024-04-20T17:59:30.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:30 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:30.176+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6802 osd.4 since back 2024-04-20T17:58:28.286173+0000 front 2024-04-20T17:58:28.286071+0000 (oldest deadline 2024-04-20T17:58:54.185996+0000) 2024-04-20T17:59:30.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:30 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:30.176+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6806 osd.5 since back 2024-04-20T17:58:40.588179+0000 front 2024-04-20T17:58:40.588216+0000 (oldest deadline 2024-04-20T17:59:06.488042+0000) 2024-04-20T17:59:30.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:30 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:30.176+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6810 osd.6 since back 2024-04-20T17:58:49.190161+0000 front 2024-04-20T17:58:49.190253+0000 (oldest deadline 2024-04-20T17:59:10.890056+0000) 2024-04-20T17:59:30.583 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:30 smithi171 ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11[72621]: 2024-04-20T17:59:30.176+0000 7f1136784700 -1 osd.11 1044 heartbeat_check: no reply from 172.21.15.106:6814 osd.7 since back 2024-04-20T17:58:56.691794+0000 front 2024-04-20T17:58:56.691780+0000 (oldest deadline 2024-04-20T17:59:20.191659+0000) 2024-04-20T17:59:31.499 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 20 17:59:31 smithi171 bash[202946]: ceph-13c6e76c-ff37-11ee-bc93-c7b262605968-osd-11 2024-04-20T17:59:33.073 DEBUG:teuthology.orchestra.run.smithi171:> sudo pkill -f 'journalctl -f -n 0 -u ceph-13c6e76c-ff37-11ee-bc93-c7b262605968@osd.11.service' 2024-04-20T17:59:33.112 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-20T17:59:33.113 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-20T17:59:33.113 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 --force --keep-logs 2024-04-20T17:59:33.356 INFO:teuthology.orchestra.run.smithi005.stdout:Deleting cluster with fsid: 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T17:59:34.562 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 --force --keep-logs 2024-04-20T17:59:34.791 INFO:teuthology.orchestra.run.smithi106.stdout:Deleting cluster with fsid: 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T17:59:35.899 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 --force --keep-logs 2024-04-20T17:59:36.126 INFO:teuthology.orchestra.run.smithi171.stdout:Deleting cluster with fsid: 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T17:59:37.122 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-20T17:59:37.159 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-20T17:59:37.195 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-20T17:59:37.224 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-20T17:59:37.227 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/crash to /home/teuthworker/archive/yuriw-2024-04-20_15:32:38-rados-wip-yuriw-testing-20240419.185239-main-distro-default-smithi/7664810/remote/smithi005/crash 2024-04-20T17:59:37.228 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/crash -- . 2024-04-20T17:59:37.290 INFO:teuthology.orchestra.run.smithi005.stderr:tar: /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-20T17:59:37.290 INFO:teuthology.orchestra.run.smithi005.stderr:tar: Error is not recoverable: exiting now 2024-04-20T17:59:37.291 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/crash to /home/teuthworker/archive/yuriw-2024-04-20_15:32:38-rados-wip-yuriw-testing-20240419.185239-main-distro-default-smithi/7664810/remote/smithi106/crash 2024-04-20T17:59:37.293 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/crash -- . 2024-04-20T17:59:37.359 INFO:teuthology.orchestra.run.smithi106.stderr:tar: /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-20T17:59:37.360 INFO:teuthology.orchestra.run.smithi106.stderr:tar: Error is not recoverable: exiting now 2024-04-20T17:59:37.362 DEBUG:teuthology.misc:Transferring archived files from smithi171:/var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/crash to /home/teuthworker/archive/yuriw-2024-04-20_15:32:38-rados-wip-yuriw-testing-20240419.185239-main-distro-default-smithi/7664810/remote/smithi171/crash 2024-04-20T17:59:37.363 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar c -f - -C /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/crash -- . 2024-04-20T17:59:37.432 INFO:teuthology.orchestra.run.smithi171.stderr:tar: /var/lib/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-20T17:59:37.433 INFO:teuthology.orchestra.run.smithi171.stderr:tar: Error is not recoverable: exiting now 2024-04-20T17:59:37.434 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-20T17:59:37.434 DEBUG:teuthology.orchestra.run.smithi005:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-04-20T17:59:37.537 INFO:teuthology.orchestra.run.smithi005.stdout:2024-04-20T17:20:00.000101+0000 mon.a (mon.0) 1604 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set 2024-04-20T17:59:37.538 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-20T17:59:37.538 DEBUG:teuthology.orchestra.run.smithi005:> sudo egrep '\[SEC\]' /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-04-20T17:59:37.579 DEBUG:teuthology.orchestra.run.smithi005:> sudo egrep '\[ERR\]' /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-04-20T17:59:37.659 DEBUG:teuthology.orchestra.run.smithi005:> sudo egrep '\[WRN\]' /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-04-20T17:59:37.738 INFO:teuthology.orchestra.run.smithi005.stdout:2024-04-20T17:20:00.000101+0000 mon.a (mon.0) 1604 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set 2024-04-20T17:59:37.738 INFO:tasks.cephadm:Compressing logs... 2024-04-20T17:59:37.738 DEBUG:teuthology.orchestra.run.smithi005:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-20T17:59:37.781 DEBUG:teuthology.orchestra.run.smithi106:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-20T17:59:37.784 DEBUG:teuthology.orchestra.run.smithi171:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-20T17:59:37.820 INFO:teuthology.orchestra.run.smithi005.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- : No such file or directory/var/log/ceph/cephadm.log 2024-04-20T17:59:37.820 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-04-20T17:59:37.823 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-volume.log 2024-04-20T17:59:37.823 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.2.log 2024-04-20T17:59:37.825 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-mgr.y.log 2024-04-20T17:59:37.825 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.audit.log 2024-04-20T17:59:37.826 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.cephadm.log 2024-04-20T17:59:37.826 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.3.log 2024-04-20T17:59:37.826 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.0.log 2024-04-20T17:59:37.826 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/objectstore_tool.6.log 2024-04-20T17:59:37.826 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/objectstore_tool.7.log 2024-04-20T17:59:37.826 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/objectstore_tool.6.log: gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-mon.a.log 2024-04-20T17:59:37.826 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/objectstore_tool.7.log: gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.log 2024-04-20T17:59:37.826 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.1.log 2024-04-20T17:59:37.827 INFO:teuthology.orchestra.run.smithi171.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-20T17:59:37.827 INFO:teuthology.orchestra.run.smithi171.stderr:: No such file or directory 2024-04-20T17:59:37.829 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.11.log 2024-04-20T17:59:37.830 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-volume.log 2024-04-20T17:59:37.830 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.audit.log 2024-04-20T17:59:37.831 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.cephadm.log 2024-04-20T17:59:37.831 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.10.log 2024-04-20T17:59:37.831 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.8.log 2024-04-20T17:59:37.831 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-mon.c.log 2024-04-20T17:59:37.832 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.9.log 2024-04-20T17:59:37.832 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.log 2024-04-20T17:59:37.841 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.log: /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.1.log: 87.7% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.log.gz 2024-04-20T17:59:37.858 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.9.log: /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.log: 82.1% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-20T17:59:37.925 INFO:teuthology.orchestra.run.smithi106.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-20T17:59:37.925 INFO:teuthology.orchestra.run.smithi106.stderr:: No such file or directory 2024-04-20T17:59:37.940 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.6.log 2024-04-20T17:59:37.940 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-volume.log 2024-04-20T17:59:37.940 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.audit.log 2024-04-20T17:59:37.941 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.cephadm.log 2024-04-20T17:59:37.942 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.7.log 2024-04-20T17:59:37.942 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-mgr.x.log 2024-04-20T17:59:37.942 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.cephadm.log: /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/objectstore_tool.7.log 2024-04-20T17:59:37.943 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-mon.b.log 2024-04-20T17:59:37.943 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-mgr.x.log: /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/objectstore_tool.7.log: gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.log 2024-04-20T17:59:37.943 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.4.log 2024-04-20T17:59:37.943 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.5.log 2024-04-20T17:59:37.972 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.4.log: /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.5.log: 93.3% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-20T17:59:38.002 INFO:teuthology.orchestra.run.smithi005.stderr: 82.6% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-20T17:59:38.015 INFO:teuthology.orchestra.run.smithi005.stderr: 96.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-20T17:59:38.061 INFO:teuthology.orchestra.run.smithi106.stderr: 95.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-20T17:59:38.177 INFO:teuthology.orchestra.run.smithi005.stderr: 89.5% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.audit.log.gz 2024-04-20T17:59:38.226 INFO:teuthology.orchestra.run.smithi005.stderr: 94.3% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/objectstore_tool.6.log.gz 2024-04-20T17:59:38.327 INFO:teuthology.orchestra.run.smithi106.stderr: 82.1% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-20T17:59:38.341 INFO:teuthology.orchestra.run.smithi106.stderr: 91.3% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-volume.log.gz 2024-04-20T17:59:38.431 INFO:teuthology.orchestra.run.smithi005.stderr: 94.6% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/objectstore_tool.7.log.gz 2024-04-20T17:59:38.467 INFO:teuthology.orchestra.run.smithi106.stderr: 89.6% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.audit.log.gz 2024-04-20T17:59:38.467 INFO:teuthology.orchestra.run.smithi005.stderr: 91.4% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-volume.log.gz 2024-04-20T17:59:38.499 INFO:teuthology.orchestra.run.smithi106.stderr: 94.8% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/objectstore_tool.7.log.gz 2024-04-20T17:59:38.522 INFO:teuthology.orchestra.run.smithi106.stderr: 87.8% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.log.gz 2024-04-20T17:59:38.727 INFO:teuthology.orchestra.run.smithi171.stderr: 89.6% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.audit.log.gz 2024-04-20T17:59:39.072 INFO:teuthology.orchestra.run.smithi171.stderr: 91.4% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-volume.log.gz 2024-04-20T17:59:39.106 INFO:teuthology.orchestra.run.smithi171.stderr: 95.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-20T17:59:39.157 INFO:teuthology.orchestra.run.smithi171.stderr: 87.8% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph.log.gz 2024-04-20T17:59:39.990 INFO:teuthology.orchestra.run.smithi106.stderr: 90.6% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-20T17:59:40.089 INFO:teuthology.orchestra.run.smithi005.stderr: 91.1% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-20T17:59:41.598 INFO:teuthology.orchestra.run.smithi171.stderr: 90.8% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-20T17:59:45.276 INFO:teuthology.orchestra.run.smithi005.stderr: 89.7% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-20T18:00:24.743 INFO:teuthology.orchestra.run.smithi106.stderr: 94.9% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-20T18:01:08.925 INFO:teuthology.orchestra.run.smithi106.stderr: 94.3% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-20T18:01:16.432 INFO:teuthology.orchestra.run.smithi106.stderr: 94.4% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-20T18:01:24.823 INFO:teuthology.orchestra.run.smithi106.stderr: 94.0% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-20T18:01:24.825 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-04-20T18:01:24.825 INFO:teuthology.orchestra.run.smithi106.stderr:real 1m47.023s 2024-04-20T18:01:24.825 INFO:teuthology.orchestra.run.smithi106.stderr:user 3m48.717s 2024-04-20T18:01:24.825 INFO:teuthology.orchestra.run.smithi106.stderr:sys 0m9.568s 2024-04-20T18:01:49.142 INFO:teuthology.orchestra.run.smithi005.stderr: 94.3% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-20T18:02:46.257 INFO:teuthology.orchestra.run.smithi005.stderr: 93.2% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-20T18:03:08.416 INFO:teuthology.orchestra.run.smithi171.stderr: 94.5% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-20T18:03:37.653 INFO:teuthology.orchestra.run.smithi171.stderr: 94.1% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-20T18:03:51.320 INFO:teuthology.orchestra.run.smithi005.stderr: 94.0% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-20T18:04:26.906 INFO:teuthology.orchestra.run.smithi005.stderr: 92.8% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-20T18:04:26.908 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-04-20T18:04:26.909 INFO:teuthology.orchestra.run.smithi005.stderr:real 4m49.112s 2024-04-20T18:04:26.909 INFO:teuthology.orchestra.run.smithi005.stderr:user 6m45.454s 2024-04-20T18:04:26.909 INFO:teuthology.orchestra.run.smithi005.stderr:sys 0m15.532s 2024-04-20T18:04:41.604 INFO:teuthology.orchestra.run.smithi171.stderr: 92.9% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-20T18:06:03.492 INFO:teuthology.orchestra.run.smithi171.stderr: 92.6% -- replaced with /var/log/ceph/13c6e76c-ff37-11ee-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-20T18:06:03.494 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-04-20T18:06:03.494 INFO:teuthology.orchestra.run.smithi171.stderr:real 6m25.693s 2024-04-20T18:06:03.494 INFO:teuthology.orchestra.run.smithi171.stderr:user 7m16.398s 2024-04-20T18:06:03.494 INFO:teuthology.orchestra.run.smithi171.stderr:sys 0m18.888s 2024-04-20T18:06:03.495 INFO:tasks.cephadm:Archiving logs... 2024-04-20T18:06:03.495 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-20_15:32:38-rados-wip-yuriw-testing-20240419.185239-main-distro-default-smithi/7664810/remote/smithi005/log 2024-04-20T18:06:03.496 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-20T18:07:05.660 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-20_15:32:38-rados-wip-yuriw-testing-20240419.185239-main-distro-default-smithi/7664810/remote/smithi106/log 2024-04-20T18:07:05.661 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-20T18:07:39.961 DEBUG:teuthology.misc:Transferring archived files from smithi171:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-20_15:32:38-rados-wip-yuriw-testing-20240419.185239-main-distro-default-smithi/7664810/remote/smithi171/log 2024-04-20T18:07:39.962 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-20T18:08:54.371 INFO:tasks.cephadm:Removing cluster... 2024-04-20T18:08:54.372 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 --force 2024-04-20T18:08:54.681 INFO:teuthology.orchestra.run.smithi005.stdout:Deleting cluster with fsid: 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T18:08:55.340 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 --force 2024-04-20T18:08:55.761 INFO:teuthology.orchestra.run.smithi106.stdout:Deleting cluster with fsid: 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T18:08:56.274 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 13c6e76c-ff37-11ee-bc93-c7b262605968 --force 2024-04-20T18:08:56.789 INFO:teuthology.orchestra.run.smithi171.stdout:Deleting cluster with fsid: 13c6e76c-ff37-11ee-bc93-c7b262605968 2024-04-20T18:08:57.538 INFO:tasks.cephadm:Removing cephadm ... 2024-04-20T18:08:57.538 DEBUG:teuthology.orchestra.run.smithi005:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-20T18:08:57.582 DEBUG:teuthology.orchestra.run.smithi106:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-20T18:08:57.606 DEBUG:teuthology.orchestra.run.smithi171:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-20T18:08:57.636 INFO:tasks.cephadm:Teardown complete 2024-04-20T18:08:57.636 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-20T18:08:57.645 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-20T18:08:57.645 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-20T18:08:57.648 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-20T18:08:57.650 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-20T18:08:57.864 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-20T18:08:57.873 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-20T18:08:57.873 DEBUG:teuthology.orchestra.run.smithi005:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-20T18:08:57.876 DEBUG:teuthology.orchestra.run.smithi106:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-20T18:08:57.879 DEBUG:teuthology.orchestra.run.smithi171:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-20T18:08:57.896 INFO:teuthology.orchestra.run.smithi005.stderr:bash: ntpq: command not found 2024-04-20T18:08:57.897 INFO:teuthology.orchestra.run.smithi171.stderr:bash: ntpq: command not found 2024-04-20T18:08:57.897 INFO:teuthology.orchestra.run.smithi106.stderr:bash: ntpq: command not found 2024-04-20T18:08:57.913 INFO:teuthology.orchestra.run.smithi171.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-20T18:08:57.914 INFO:teuthology.orchestra.run.smithi171.stdout:=============================================================================== 2024-04-20T18:08:57.914 INFO:teuthology.orchestra.run.smithi171.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 16 +1472us[+1472us] +/- 74ms 2024-04-20T18:08:57.914 INFO:teuthology.orchestra.run.smithi171.stdout:^* hv02.front.sepia.ceph.com 3 6 377 20 -1000us[-1163us] +/- 53ms 2024-04-20T18:08:57.914 INFO:teuthology.orchestra.run.smithi171.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 -145us[ -145us] +/- 65ms 2024-04-20T18:08:57.914 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T18:08:57.919 INFO:teuthology.orchestra.run.smithi106.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-20T18:08:57.919 INFO:teuthology.orchestra.run.smithi106.stdout:=============================================================================== 2024-04-20T18:08:57.920 INFO:teuthology.orchestra.run.smithi106.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 8 +1518us[+1518us] +/- 74ms 2024-04-20T18:08:57.920 INFO:teuthology.orchestra.run.smithi106.stdout:^* hv02.front.sepia.ceph.com 3 6 377 12 -1001us[-1129us] +/- 53ms 2024-04-20T18:08:57.920 INFO:teuthology.orchestra.run.smithi106.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 13 -96us[ -224us] +/- 65ms 2024-04-20T18:08:57.920 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T18:08:57.927 INFO:teuthology.orchestra.run.smithi005.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-20T18:08:57.927 INFO:teuthology.orchestra.run.smithi005.stdout:=============================================================================== 2024-04-20T18:08:57.927 INFO:teuthology.orchestra.run.smithi005.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 9 +1517us[+1517us] +/- 74ms 2024-04-20T18:08:57.927 INFO:teuthology.orchestra.run.smithi005.stdout:^* hv02.front.sepia.ceph.com 3 6 377 13 -982us[-1108us] +/- 53ms 2024-04-20T18:08:57.928 INFO:teuthology.orchestra.run.smithi005.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 13 -94us[ -94us] +/- 65ms 2024-04-20T18:08:57.928 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T18:08:57.929 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-20T18:08:57.939 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-20T18:08:57.939 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-20T18:08:57.948 DEBUG:teuthology.orchestra.run.smithi005:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-20T18:08:58.079 DEBUG:teuthology.orchestra.run.smithi106:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-20T18:08:58.170 DEBUG:teuthology.orchestra.run.smithi171:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-20T18:08:58.253 DEBUG:teuthology.orchestra.run.smithi005:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-20T18:08:58.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T18:08:58.289 DEBUG:teuthology.orchestra.run.smithi106:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-20T18:08:58.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T18:08:58.320 DEBUG:teuthology.orchestra.run.smithi171:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-20T18:08:58.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T18:08:58.350 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-20T18:08:58.359 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-20T18:08:58.367 INFO:teuthology.task.internal:Duration was 4848.302929 seconds 2024-04-20T18:08:58.367 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-20T18:08:58.377 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-20T18:08:58.377 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-20T18:08:58.380 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-20T18:08:58.382 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-20T18:08:58.480 INFO:teuthology.orchestra.run.smithi106.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-20T18:08:58.486 INFO:teuthology.orchestra.run.smithi005.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-20T18:08:58.503 INFO:teuthology.orchestra.run.smithi171.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-20T18:08:58.785 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-20T18:08:58.785 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi005.front.sepia.ceph.com 2024-04-20T18:08:58.785 DEBUG:teuthology.orchestra.run.smithi005:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-20T18:08:58.813 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi106.front.sepia.ceph.com 2024-04-20T18:08:58.814 DEBUG:teuthology.orchestra.run.smithi106:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-20T18:08:58.843 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi171.front.sepia.ceph.com 2024-04-20T18:08:58.843 DEBUG:teuthology.orchestra.run.smithi171:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-20T18:08:58.875 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-20T18:08:58.875 DEBUG:teuthology.orchestra.run.smithi005:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-20T18:08:58.878 DEBUG:teuthology.orchestra.run.smithi106:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-20T18:08:58.885 DEBUG:teuthology.orchestra.run.smithi171:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-20T18:08:59.037 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-20T18:08:59.037 DEBUG:teuthology.orchestra.run.smithi005:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-20T18:08:59.039 DEBUG:teuthology.orchestra.run.smithi106:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-20T18:08:59.041 DEBUG:teuthology.orchestra.run.smithi171:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-20T18:08:59.348 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-20T18:08:59.358 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-20T18:08:59.358 DEBUG:teuthology.orchestra.run.smithi005:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-20T18:08:59.400 DEBUG:teuthology.orchestra.run.smithi106:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-20T18:08:59.433 DEBUG:teuthology.orchestra.run.smithi171:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-20T18:08:59.473 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-20T18:08:59.484 DEBUG:teuthology.orchestra.run.smithi005:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-20T18:08:59.486 DEBUG:teuthology.orchestra.run.smithi106:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-20T18:08:59.489 DEBUG:teuthology.orchestra.run.smithi171:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-20T18:08:59.520 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern = core 2024-04-20T18:08:59.530 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern = core 2024-04-20T18:08:59.555 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern = core 2024-04-20T18:08:59.569 DEBUG:teuthology.orchestra.run.smithi005:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-20T18:08:59.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T18:08:59.609 DEBUG:teuthology.orchestra.run.smithi106:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-20T18:08:59.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T18:08:59.626 DEBUG:teuthology.orchestra.run.smithi171:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-20T18:08:59.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T18:08:59.643 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-20T18:08:59.653 INFO:teuthology.task.internal:Transferring archived files... 2024-04-20T18:08:59.654 DEBUG:teuthology.misc:Transferring archived files from smithi005:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-20_15:32:38-rados-wip-yuriw-testing-20240419.185239-main-distro-default-smithi/7664810/remote/smithi005 2024-04-20T18:08:59.655 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-20T18:08:59.717 DEBUG:teuthology.misc:Transferring archived files from smithi106:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-20_15:32:38-rados-wip-yuriw-testing-20240419.185239-main-distro-default-smithi/7664810/remote/smithi106 2024-04-20T18:08:59.718 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-20T18:08:59.766 DEBUG:teuthology.misc:Transferring archived files from smithi171:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-20_15:32:38-rados-wip-yuriw-testing-20240419.185239-main-distro-default-smithi/7664810/remote/smithi171 2024-04-20T18:08:59.766 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-20T18:08:59.815 INFO:teuthology.task.internal:Removing archive directory... 2024-04-20T18:08:59.815 DEBUG:teuthology.orchestra.run.smithi005:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-20T18:08:59.817 DEBUG:teuthology.orchestra.run.smithi106:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-20T18:08:59.821 DEBUG:teuthology.orchestra.run.smithi171:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-20T18:08:59.864 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-20T18:08:59.877 INFO:teuthology.task.internal:Not uploading archives. 2024-04-20T18:08:59.877 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-20T18:08:59.887 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-20T18:08:59.887 DEBUG:teuthology.orchestra.run.smithi005:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-20T18:08:59.890 DEBUG:teuthology.orchestra.run.smithi106:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-20T18:08:59.892 DEBUG:teuthology.orchestra.run.smithi171:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-20T18:08:59.906 INFO:teuthology.orchestra.run.smithi005.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 20 18:08 /home/ubuntu/cephtest 2024-04-20T18:08:59.907 INFO:teuthology.orchestra.run.smithi106.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 20 18:08 /home/ubuntu/cephtest 2024-04-20T18:08:59.922 INFO:teuthology.orchestra.run.smithi171.stdout: 525832 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 20 18:08 /home/ubuntu/cephtest 2024-04-20T18:08:59.923 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-20T18:08:59.934 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-20T18:08:59.954 INFO:teuthology.nuke:Checking targets against current locks 2024-04-20T18:08:59.984 DEBUG:teuthology.nuke:shortname: smithi005 2024-04-20T18:08:59.984 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-20T18:09:00.012 DEBUG:teuthology.nuke:shortname: smithi106 2024-04-20T18:09:00.012 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-20T18:09:00.017 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi005.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-20_15:32:38-rados-wip-yuriw-testing-20240419.185239-main-distro-default-smithi/7664810', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-20 16:39:55.568854', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJfMFwSPgQGEdAO4p0iQci9eIYTRPcXnPfeUnyUQDTzKQGpqpb6bqvXpPOdoZpD4EQK3+wMhMU292jcNygMRYLA='} 2024-04-20T18:09:00.034 DEBUG:teuthology.nuke:shortname: smithi171 2024-04-20T18:09:00.034 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-20T18:09:00.039 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi106.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-20_15:32:38-rados-wip-yuriw-testing-20240419.185239-main-distro-default-smithi/7664810', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-20 16:39:55.570207', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHbMICTpir/0oEL5S4+LyKYMdSx5m/gasVlEaOPGrfpQCtBJyThqhcPoCqqv5hdQBNOEqldwehx0hojv7O3QO4c='} 2024-04-20T18:09:00.054 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi171.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-20_15:32:38-rados-wip-yuriw-testing-20240419.185239-main-distro-default-smithi/7664810', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-20 16:39:55.571172', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMDFTDumLh8k/iLaIxuZqDoERBk9s1/MtgV73jCR0fPJzKr9Cjh/Q6E4+9kzar6GnzAjr7VdItI/eS8o3x3xzCE='} 2024-04-20T18:09:00.069 INFO:teuthology.orchestra.console.smithi005:Power off 2024-04-20T18:09:00.069 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-20T18:09:00.087 INFO:teuthology.orchestra.console.smithi106:Power off 2024-04-20T18:09:00.087 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-20T18:09:00.100 DEBUG:teuthology.orchestra.console.smithi005:power off output: Chassis Power Control: Down/Off 2024-04-20T18:09:00.100 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T18:09:00.111 DEBUG:teuthology.orchestra.console.smithi106:power off output: Chassis Power Control: Down/Off 2024-04-20T18:09:00.111 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T18:09:00.127 INFO:teuthology.orchestra.console.smithi171:Power off 2024-04-20T18:09:00.128 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-20T18:09:00.137 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is on 2024-04-20T18:09:00.137 DEBUG:teuthology.orchestra.console.smithi005:check power output: Chassis Power is on 2024-04-20T18:09:00.150 DEBUG:teuthology.orchestra.console.smithi171:power off output: Chassis Power Control: Down/Off 2024-04-20T18:09:00.150 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T18:09:00.173 DEBUG:teuthology.orchestra.console.smithi171:check power output: Chassis Power is on 2024-04-20T18:09:04.138 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T18:09:04.146 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T18:09:04.174 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T18:09:04.249 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is on 2024-04-20T18:09:04.256 DEBUG:teuthology.orchestra.console.smithi005:check power output: Chassis Power is on 2024-04-20T18:09:04.285 DEBUG:teuthology.orchestra.console.smithi171:check power output: Chassis Power is on 2024-04-20T18:09:08.250 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T18:09:08.260 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T18:09:08.286 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T18:09:08.362 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is on 2024-04-20T18:09:08.371 DEBUG:teuthology.orchestra.console.smithi005:check power output: Chassis Power is off 2024-04-20T18:09:08.396 DEBUG:teuthology.orchestra.console.smithi171:check power output: Chassis Power is on 2024-04-20T18:09:08.473 INFO:teuthology.orchestra.console.smithi005:Power off completed 2024-04-20T18:09:12.367 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T18:09:12.397 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-20T18:09:12.477 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is off 2024-04-20T18:09:12.507 DEBUG:teuthology.orchestra.console.smithi171:check power output: Chassis Power is off 2024-04-20T18:09:12.578 INFO:teuthology.orchestra.console.smithi106:Power off completed 2024-04-20T18:09:12.608 INFO:teuthology.orchestra.console.smithi171:Power off completed 2024-04-20T18:09:12.715 INFO:teuthology.run:Summary data: description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench} duration: 4848.302928686142 failure_reason: '"2024-04-20T17:20:00.000101+0000 mon.a (mon.0) 1604 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set" in cluster log' flavor: default owner: scheduled_yuriw@teuthology success: false 2024-04-20T18:09:12.716 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-20T18:09:12.797 INFO:teuthology.run:FAIL